2023-10-19T12:31:56.748 INFO:root:teuthology version: 0.0.1.dev201+g202b180 2023-10-19T12:31:56.749 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --owner scheduled_pdonnell@teuthology --archive /home/teuthworker/archive/pdonnell-2023-10-19_12:04:12-fs-wip-batrick-testing-20231016.203825-distro-default-smithi/7432290 --name pdonnell-2023-10-19_12:04:12-fs-wip-batrick-testing-20231016.203825-distro-default-smithi --description fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} -- /home/teuthworker/archive/pdonnell-2023-10-19_12:04:12-fs-wip-batrick-testing-20231016.203825-distro-default-smithi/7432290/orig.config.yaml 2023-10-19T12:31:56.806 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-19T12:31:56.904 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2023-10-19_12:04:12-fs-wip-batrick-testing-20231016.203825-distro-default-smithi/7432290 branch: wip-batrick-testing-20231016.203825 description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} email: pdonnell@redhat.com first_in_suite: false job_id: '7432290' kernel: kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: pdonnell-2023-10-19_12:04:12-fs-wip-batrick-testing-20231016.203825-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-batrick-testing-20231016.203825 ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd cluster-conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mon: mon op complaint time: 120 osd: osd op complaint time: 180 conf: client: pid file: '' mgr: debug client: 20 debug finisher: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding log-whitelist: - OSD full dropping all updates - OSD near full - pausewr flag - failsafe engaged, dropping updates - failsafe disengaged, no longer dropping - is full \(reached quota - POOL_FULL - POOL_BACKFILLFULL sha1: f7420d7f8e55ceea786dcbf536c338ce920c21eb ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: f7420d7f8e55ceea786dcbf536c338ce920c21eb thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-batrick-testing-20231016.203825 sha1: f7420d7f8e55ceea786dcbf536c338ce920c21eb owner: scheduled_pdonnell@teuthology priority: 65 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 7703 sha1: f7420d7f8e55ceea786dcbf536c338ce920c21eb sleep_before_teardown: 0 subset: 153/256 suite: fs suite_branch: wip-batrick-testing-20231016.203825 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_f7420d7f8e55ceea786dcbf536c338ce920c21eb/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: f7420d7f8e55ceea786dcbf536c338ce920c21eb syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi057.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLIN80GzytgnsPmhQxnO98RJ6hrAia1j+6pBY3TcjQx0CYaH+EBbVFNcBgcBJT8k6V/pnOvlWsn8miLAG0PNdlU= smithi179.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHzD4QfGTQMro3MfFhf2wmDWesRzz0Us4uWFCUG2uJYCokdZYdTLrJb2/y+R7emIU5CcjUKUzVhjYy6iyLxqsbk= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - ceph-fuse: null - cephfs_test_runner: modules: - tasks.cephfs.test_snap_schedules teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: 202b180cb047e798fb131047314a862593f45403 timestamp: 2023-10-19_12:04:12 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2830 2023-10-19T12:31:56.904 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_f7420d7f8e55ceea786dcbf536c338ce920c21eb/qa; will attempt to use it 2023-10-19T12:31:56.906 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_f7420d7f8e55ceea786dcbf536c338ce920c21eb/qa/tasks 2023-10-19T12:31:56.906 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-10-19T12:31:56.909 INFO:teuthology.task.internal:Checking packages... 2023-10-19T12:31:56.938 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'f7420d7f8e55ceea786dcbf536c338ce920c21eb' 2023-10-19T12:31:56.938 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-10-19T12:31:56.939 INFO:teuthology.packaging:ref: None 2023-10-19T12:31:56.939 INFO:teuthology.packaging:tag: None 2023-10-19T12:31:56.940 INFO:teuthology.packaging:branch: wip-batrick-testing-20231016.203825 2023-10-19T12:31:56.940 INFO:teuthology.packaging:sha1: f7420d7f8e55ceea786dcbf536c338ce920c21eb 2023-10-19T12:31:56.940 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-batrick-testing-20231016.203825 2023-10-19T12:31:57.109 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-6786.gf7420d7f 2023-10-19T12:31:57.109 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-10-19T12:31:57.116 INFO:teuthology.task.internal:no buildpackages task found 2023-10-19T12:31:57.116 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-10-19T12:31:57.160 INFO:teuthology.task.internal:Saving configuration 2023-10-19T12:31:57.178 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-10-19T12:31:57.184 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-19T12:31:57.208 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi057.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-10-19_12:04:12-fs-wip-batrick-testing-20231016.203825-distro-default-smithi/7432290', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-19 12:26:53.268577', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLIN80GzytgnsPmhQxnO98RJ6hrAia1j+6pBY3TcjQx0CYaH+EBbVFNcBgcBJT8k6V/pnOvlWsn8miLAG0PNdlU='} 2023-10-19T12:31:57.229 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi179.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-10-19_12:04:12-fs-wip-batrick-testing-20231016.203825-distro-default-smithi/7432290', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-19 12:26:53.266642', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHzD4QfGTQMro3MfFhf2wmDWesRzz0Us4uWFCUG2uJYCokdZYdTLrJb2/y+R7emIU5CcjUKUzVhjYy6iyLxqsbk='} 2023-10-19T12:31:57.230 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-10-19T12:31:57.236 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2023-10-19T12:31:57.237 INFO:teuthology.task.internal:roles: ubuntu@smithi179.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2023-10-19T12:31:57.237 INFO:teuthology.run_tasks:Running task console_log... 2023-10-19T12:31:57.334 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7ffb53a7eb80>, signals=[15]) 2023-10-19T12:31:57.335 INFO:teuthology.run_tasks:Running task internal.connect... 2023-10-19T12:31:57.341 INFO:teuthology.task.internal:Opening connections... 2023-10-19T12:31:57.342 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi057.front.sepia.ceph.com 2023-10-19T12:31:57.344 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi057.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-19T12:31:57.446 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi179.front.sepia.ceph.com 2023-10-19T12:31:57.447 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi179.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-19T12:31:57.530 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-10-19T12:31:57.537 DEBUG:teuthology.orchestra.run.smithi057:> uname -m 2023-10-19T12:31:57.555 INFO:teuthology.orchestra.run.smithi057.stdout:x86_64 2023-10-19T12:31:57.555 DEBUG:teuthology.orchestra.run.smithi057:> cat /etc/os-release 2023-10-19T12:31:57.612 INFO:teuthology.orchestra.run.smithi057.stdout:NAME="CentOS Stream" 2023-10-19T12:31:57.612 INFO:teuthology.orchestra.run.smithi057.stdout:VERSION="9" 2023-10-19T12:31:57.612 INFO:teuthology.orchestra.run.smithi057.stdout:ID="centos" 2023-10-19T12:31:57.612 INFO:teuthology.orchestra.run.smithi057.stdout:ID_LIKE="rhel fedora" 2023-10-19T12:31:57.613 INFO:teuthology.orchestra.run.smithi057.stdout:VERSION_ID="9" 2023-10-19T12:31:57.613 INFO:teuthology.orchestra.run.smithi057.stdout:PLATFORM_ID="platform:el9" 2023-10-19T12:31:57.613 INFO:teuthology.orchestra.run.smithi057.stdout:PRETTY_NAME="CentOS Stream 9" 2023-10-19T12:31:57.613 INFO:teuthology.orchestra.run.smithi057.stdout:ANSI_COLOR="0;31" 2023-10-19T12:31:57.613 INFO:teuthology.orchestra.run.smithi057.stdout:LOGO="fedora-logo-icon" 2023-10-19T12:31:57.613 INFO:teuthology.orchestra.run.smithi057.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-10-19T12:31:57.614 INFO:teuthology.orchestra.run.smithi057.stdout:HOME_URL="https://centos.org/" 2023-10-19T12:31:57.614 INFO:teuthology.orchestra.run.smithi057.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-19T12:31:57.614 INFO:teuthology.orchestra.run.smithi057.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-10-19T12:31:57.614 INFO:teuthology.orchestra.run.smithi057.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-10-19T12:31:57.615 INFO:teuthology.lock.ops:Updating smithi057.front.sepia.ceph.com on lock server 2023-10-19T12:31:57.636 DEBUG:teuthology.orchestra.run.smithi179:> uname -m 2023-10-19T12:31:57.653 INFO:teuthology.orchestra.run.smithi179.stdout:x86_64 2023-10-19T12:31:57.654 DEBUG:teuthology.orchestra.run.smithi179:> cat /etc/os-release 2023-10-19T12:31:57.710 INFO:teuthology.orchestra.run.smithi179.stdout:NAME="CentOS Stream" 2023-10-19T12:31:57.710 INFO:teuthology.orchestra.run.smithi179.stdout:VERSION="9" 2023-10-19T12:31:57.711 INFO:teuthology.orchestra.run.smithi179.stdout:ID="centos" 2023-10-19T12:31:57.711 INFO:teuthology.orchestra.run.smithi179.stdout:ID_LIKE="rhel fedora" 2023-10-19T12:31:57.711 INFO:teuthology.orchestra.run.smithi179.stdout:VERSION_ID="9" 2023-10-19T12:31:57.711 INFO:teuthology.orchestra.run.smithi179.stdout:PLATFORM_ID="platform:el9" 2023-10-19T12:31:57.711 INFO:teuthology.orchestra.run.smithi179.stdout:PRETTY_NAME="CentOS Stream 9" 2023-10-19T12:31:57.712 INFO:teuthology.orchestra.run.smithi179.stdout:ANSI_COLOR="0;31" 2023-10-19T12:31:57.712 INFO:teuthology.orchestra.run.smithi179.stdout:LOGO="fedora-logo-icon" 2023-10-19T12:31:57.712 INFO:teuthology.orchestra.run.smithi179.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-10-19T12:31:57.712 INFO:teuthology.orchestra.run.smithi179.stdout:HOME_URL="https://centos.org/" 2023-10-19T12:31:57.712 INFO:teuthology.orchestra.run.smithi179.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-19T12:31:57.713 INFO:teuthology.orchestra.run.smithi179.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-10-19T12:31:57.713 INFO:teuthology.orchestra.run.smithi179.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-10-19T12:31:57.713 INFO:teuthology.lock.ops:Updating smithi179.front.sepia.ceph.com on lock server 2023-10-19T12:31:57.737 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-10-19T12:31:57.746 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-10-19T12:31:57.768 INFO:teuthology.task.internal:Checking for old test directory... 2023-10-19T12:31:57.769 DEBUG:teuthology.orchestra.run.smithi057:> test '!' -e /home/ubuntu/cephtest 2023-10-19T12:31:57.772 DEBUG:teuthology.orchestra.run.smithi179:> test '!' -e /home/ubuntu/cephtest 2023-10-19T12:31:57.787 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-10-19T12:31:57.793 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-10-19T12:31:57.793 DEBUG:teuthology.orchestra.run.smithi057:> test -z $(ls -A /var/lib/ceph) 2023-10-19T12:31:57.829 DEBUG:teuthology.orchestra.run.smithi179:> test -z $(ls -A /var/lib/ceph) 2023-10-19T12:31:57.860 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-10-19T12:31:57.914 INFO:teuthology.run_tasks:Running task kernel... 2023-10-19T12:31:57.929 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-10-19T12:31:57.929 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-10-19T12:31:57.930 DEBUG:teuthology.orchestra.run.smithi057:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-19T12:31:57.931 DEBUG:teuthology.orchestra.run.smithi179:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-19T12:31:57.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-19T12:31:57.946 DEBUG:teuthology.orchestra.run.smithi179:> uname -r 2023-10-19T12:31:57.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-19T12:31:57.948 DEBUG:teuthology.orchestra.run.smithi057:> uname -r 2023-10-19T12:31:58.002 INFO:teuthology.orchestra.run.smithi179.stdout:5.14.0-361.el9.x86_64 2023-10-19T12:31:58.003 INFO:teuthology.task.kernel:Running kernel on smithi179: 5.14.0-361.el9.x86_64 2023-10-19T12:31:58.003 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum install -y kernel 2023-10-19T12:31:58.006 INFO:teuthology.orchestra.run.smithi057.stdout:5.14.0-361.el9.x86_64 2023-10-19T12:31:58.006 INFO:teuthology.task.kernel:Running kernel on smithi057: 5.14.0-361.el9.x86_64 2023-10-19T12:31:58.006 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum install -y kernel 2023-10-19T12:32:00.553 INFO:teuthology.orchestra.run.smithi057.stdout:CentOS-9 - AppStream 6.7 MB/s | 6.3 MB 00:00 2023-10-19T12:32:00.554 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS-9 - AppStream 6.7 MB/s | 6.3 MB 00:00 2023-10-19T12:32:02.027 INFO:teuthology.orchestra.run.smithi057.stdout:CentOS-9 - BaseOS 4.9 MB/s | 2.0 MB 00:00 2023-10-19T12:32:02.029 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS-9 - BaseOS 4.7 MB/s | 2.0 MB 00:00 2023-10-19T12:32:03.943 INFO:teuthology.orchestra.run.smithi057.stdout:CentOS Stream 9 - BaseOS 5.0 MB/s | 7.9 MB 00:01 2023-10-19T12:32:05.420 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS Stream 9 - BaseOS 2.6 MB/s | 7.9 MB 00:03 2023-10-19T12:32:11.304 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS Stream 9 - AppStream 3.7 MB/s | 18 MB 00:04 2023-10-19T12:32:12.168 INFO:teuthology.orchestra.run.smithi057.stdout:CentOS Stream 9 - AppStream 2.5 MB/s | 18 MB 00:07 2023-10-19T12:32:16.888 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS Stream 9 - CRB 2.6 MB/s | 5.7 MB 00:02 2023-10-19T12:32:17.269 INFO:teuthology.orchestra.run.smithi057.stdout:CentOS Stream 9 - CRB 3.9 MB/s | 5.7 MB 00:01 2023-10-19T12:32:18.447 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS Stream 9 - Extras packages 30 kB/s | 14 kB 00:00 2023-10-19T12:32:18.703 INFO:teuthology.orchestra.run.smithi057.stdout:CentOS Stream 9 - Extras packages 60 kB/s | 14 kB 00:00 2023-10-19T12:32:20.768 INFO:teuthology.orchestra.run.smithi179.stdout:Extra Packages for Enterprise Linux 8.6 MB/s | 19 MB 00:02 2023-10-19T12:32:20.989 INFO:teuthology.orchestra.run.smithi057.stdout:Extra Packages for Enterprise Linux 8.8 MB/s | 19 MB 00:02 2023-10-19T12:32:25.399 INFO:teuthology.orchestra.run.smithi179.stdout:lab-extras 9.9 kB/s | 1.7 kB 00:00 2023-10-19T12:32:25.778 INFO:teuthology.orchestra.run.smithi057.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2023-10-19T12:32:27.464 INFO:teuthology.orchestra.run.smithi179.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-10-19T12:32:27.465 INFO:teuthology.orchestra.run.smithi179.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-10-19T12:32:27.551 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2023-10-19T12:32:27.559 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2023-10-19T12:32:27.560 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repository Size 2023-10-19T12:32:27.560 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2023-10-19T12:32:27.560 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2023-10-19T12:32:27.560 INFO:teuthology.orchestra.run.smithi179.stdout: kernel x86_64 5.14.0-376.el9 CentOS-BaseOS 5.3 M 2023-10-19T12:32:27.561 INFO:teuthology.orchestra.run.smithi179.stdout:Installing dependencies: 2023-10-19T12:32:27.561 INFO:teuthology.orchestra.run.smithi179.stdout: kernel-core x86_64 5.14.0-376.el9 CentOS-BaseOS 20 M 2023-10-19T12:32:27.561 INFO:teuthology.orchestra.run.smithi179.stdout: kernel-modules x86_64 5.14.0-376.el9 CentOS-BaseOS 38 M 2023-10-19T12:32:27.561 INFO:teuthology.orchestra.run.smithi179.stdout: kernel-modules-core x86_64 5.14.0-376.el9 CentOS-BaseOS 32 M 2023-10-19T12:32:27.561 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-10-19T12:32:27.562 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2023-10-19T12:32:27.562 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2023-10-19T12:32:27.562 INFO:teuthology.orchestra.run.smithi179.stdout:Install 4 Packages 2023-10-19T12:32:27.562 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-10-19T12:32:27.562 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 95 M 2023-10-19T12:32:27.563 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 120 M 2023-10-19T12:32:27.563 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2023-10-19T12:32:27.843 INFO:teuthology.orchestra.run.smithi057.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-10-19T12:32:27.844 INFO:teuthology.orchestra.run.smithi057.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-10-19T12:32:27.942 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2023-10-19T12:32:27.948 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2023-10-19T12:32:27.949 INFO:teuthology.orchestra.run.smithi057.stdout: Package Arch Version Repository Size 2023-10-19T12:32:27.949 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2023-10-19T12:32:27.949 INFO:teuthology.orchestra.run.smithi057.stdout:Installing: 2023-10-19T12:32:27.949 INFO:teuthology.orchestra.run.smithi057.stdout: kernel x86_64 5.14.0-376.el9 CentOS-BaseOS 5.3 M 2023-10-19T12:32:27.950 INFO:teuthology.orchestra.run.smithi057.stdout:Installing dependencies: 2023-10-19T12:32:27.950 INFO:teuthology.orchestra.run.smithi057.stdout: kernel-core x86_64 5.14.0-376.el9 CentOS-BaseOS 20 M 2023-10-19T12:32:27.950 INFO:teuthology.orchestra.run.smithi057.stdout: kernel-modules x86_64 5.14.0-376.el9 CentOS-BaseOS 38 M 2023-10-19T12:32:27.950 INFO:teuthology.orchestra.run.smithi057.stdout: kernel-modules-core x86_64 5.14.0-376.el9 CentOS-BaseOS 32 M 2023-10-19T12:32:27.951 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:32:27.951 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction Summary 2023-10-19T12:32:27.951 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2023-10-19T12:32:27.951 INFO:teuthology.orchestra.run.smithi057.stdout:Install 4 Packages 2023-10-19T12:32:27.951 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:32:27.952 INFO:teuthology.orchestra.run.smithi057.stdout:Total download size: 95 M 2023-10-19T12:32:27.952 INFO:teuthology.orchestra.run.smithi057.stdout:Installed size: 120 M 2023-10-19T12:32:27.952 INFO:teuthology.orchestra.run.smithi057.stdout:Downloading Packages: 2023-10-19T12:32:28.367 INFO:teuthology.orchestra.run.smithi179.stdout:(1/4): kernel-5.14.0-376.el9.x86_64.rpm 6.6 MB/s | 5.3 MB 00:00 2023-10-19T12:32:29.276 INFO:teuthology.orchestra.run.smithi179.stdout:(2/4): kernel-core-5.14.0-376.el9.x86_64.rpm 12 MB/s | 20 MB 00:01 2023-10-19T12:32:29.646 INFO:teuthology.orchestra.run.smithi057.stdout:(1/4): kernel-5.14.0-376.el9.x86_64.rpm 3.1 MB/s | 5.3 MB 00:01 2023-10-19T12:32:30.051 INFO:teuthology.orchestra.run.smithi179.stdout:(3/4): kernel-modules-5.14.0-376.el9.x86_64.rpm 15 MB/s | 38 MB 00:02 2023-10-19T12:32:30.471 INFO:teuthology.orchestra.run.smithi057.stdout:(2/4): kernel-modules-5.14.0-376.el9.x86_64.rpm 15 MB/s | 38 MB 00:02 2023-10-19T12:32:30.584 INFO:teuthology.orchestra.run.smithi179.stdout:(4/4): kernel-modules-core-5.14.0-376.el9.x86_6 14 MB/s | 32 MB 00:02 2023-10-19T12:32:30.588 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2023-10-19T12:32:30.588 INFO:teuthology.orchestra.run.smithi179.stdout:Total 31 MB/s | 95 MB 00:03 2023-10-19T12:32:30.589 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2023-10-19T12:32:30.663 INFO:teuthology.orchestra.run.smithi057.stdout:(3/4): kernel-core-5.14.0-376.el9.x86_64.rpm 7.4 MB/s | 20 MB 00:02 2023-10-19T12:32:30.964 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2023-10-19T12:32:30.964 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2023-10-19T12:32:31.155 INFO:teuthology.orchestra.run.smithi057.stdout:(4/4): kernel-modules-core-5.14.0-376.el9.x86_6 21 MB/s | 32 MB 00:01 2023-10-19T12:32:31.158 INFO:teuthology.orchestra.run.smithi057.stdout:-------------------------------------------------------------------------------- 2023-10-19T12:32:31.159 INFO:teuthology.orchestra.run.smithi057.stdout:Total 30 MB/s | 95 MB 00:03 2023-10-19T12:32:31.159 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction check 2023-10-19T12:32:31.563 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction check succeeded. 2023-10-19T12:32:31.564 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction test 2023-10-19T12:32:31.973 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2023-10-19T12:32:31.973 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2023-10-19T12:32:32.589 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction test succeeded. 2023-10-19T12:32:32.590 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction 2023-10-19T12:32:34.326 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2023-10-19T12:32:34.971 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : kernel-modules-core-5.14.0-376.el9.x86_64 1/4 2023-10-19T12:32:35.080 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : kernel-core-5.14.0-376.el9.x86_64 2/4 2023-10-19T12:32:35.084 INFO:teuthology.orchestra.run.smithi057.stdout: Preparing : 1/1 2023-10-19T12:32:35.737 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : kernel-modules-core-5.14.0-376.el9.x86_64 1/4 2023-10-19T12:32:35.871 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : kernel-core-5.14.0-376.el9.x86_64 2/4 2023-10-19T12:32:37.801 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: kernel-core-5.14.0-376.el9.x86_64 2/4 2023-10-19T12:32:37.980 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : kernel-modules-5.14.0-376.el9.x86_64 3/4 2023-10-19T12:32:38.757 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: kernel-core-5.14.0-376.el9.x86_64 2/4 2023-10-19T12:32:38.940 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : kernel-modules-5.14.0-376.el9.x86_64 3/4 2023-10-19T12:32:42.527 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: kernel-modules-5.14.0-376.el9.x86_64 3/4 2023-10-19T12:32:42.624 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : kernel-5.14.0-376.el9.x86_64 4/4 2023-10-19T12:32:43.906 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: kernel-modules-5.14.0-376.el9.x86_64 3/4 2023-10-19T12:32:44.009 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : kernel-5.14.0-376.el9.x86_64 4/4 2023-10-19T12:32:47.245 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: kernel-modules-core-5.14.0-376.el9.x86_64 4/4 2023-10-19T12:32:49.005 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: kernel-modules-core-5.14.0-376.el9.x86_64 4/4 2023-10-19T12:33:16.098 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: kernel-core-5.14.0-376.el9.x86_64 4/4 2023-10-19T12:33:16.139 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: kernel-modules-5.14.0-376.el9.x86_64 4/4 2023-10-19T12:33:16.516 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: kernel-5.14.0-376.el9.x86_64 4/4 2023-10-19T12:33:16.517 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : kernel-5.14.0-376.el9.x86_64 1/4 2023-10-19T12:33:16.517 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : kernel-core-5.14.0-376.el9.x86_64 2/4 2023-10-19T12:33:16.517 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : kernel-modules-5.14.0-376.el9.x86_64 3/4 2023-10-19T12:33:16.927 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : kernel-modules-core-5.14.0-376.el9.x86_64 4/4 2023-10-19T12:33:16.927 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-10-19T12:33:16.928 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2023-10-19T12:33:16.928 INFO:teuthology.orchestra.run.smithi179.stdout: kernel-5.14.0-376.el9.x86_64 2023-10-19T12:33:16.928 INFO:teuthology.orchestra.run.smithi179.stdout: kernel-core-5.14.0-376.el9.x86_64 2023-10-19T12:33:16.928 INFO:teuthology.orchestra.run.smithi179.stdout: kernel-modules-5.14.0-376.el9.x86_64 2023-10-19T12:33:16.929 INFO:teuthology.orchestra.run.smithi179.stdout: kernel-modules-core-5.14.0-376.el9.x86_64 2023-10-19T12:33:16.929 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-10-19T12:33:16.929 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2023-10-19T12:33:17.445 DEBUG:teuthology.orchestra.run.smithi179:> rpm -q kernel | sort -rV | head -n 1 2023-10-19T12:33:17.567 INFO:teuthology.orchestra.run.smithi179.stdout:kernel-5.14.0-376.el9.x86_64 2023-10-19T12:33:17.568 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-376.el9.x86_64 2023-10-19T12:33:17.568 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-376.el9.x86_64 2023-10-19T12:33:17.568 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-19T12:33:17.568 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-19T12:33:17.569 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2023-10-19T12:33:17.569 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi179.front.sepia.ceph.com, path=None, version=distro) 2023-10-19T12:33:17.569 DEBUG:teuthology.orchestra.run.smithi179:> rpm -q kernel | sort -rV | head -n 1 2023-10-19T12:33:17.674 INFO:teuthology.orchestra.run.smithi179.stdout:kernel-5.14.0-376.el9.x86_64 2023-10-19T12:33:17.674 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-376.el9.x86_64 2023-10-19T12:33:17.675 DEBUG:teuthology.orchestra.run.smithi179:> sudo rpm -qi grub2-tools 2023-10-19T12:33:17.713 INFO:teuthology.orchestra.run.smithi179.stdout:Name : grub2-tools 2023-10-19T12:33:17.713 INFO:teuthology.orchestra.run.smithi179.stdout:Epoch : 1 2023-10-19T12:33:17.713 INFO:teuthology.orchestra.run.smithi179.stdout:Version : 2.06 2023-10-19T12:33:17.714 INFO:teuthology.orchestra.run.smithi179.stdout:Release : 68.el9 2023-10-19T12:33:17.714 INFO:teuthology.orchestra.run.smithi179.stdout:Architecture: x86_64 2023-10-19T12:33:17.714 INFO:teuthology.orchestra.run.smithi179.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-10-19T12:33:17.714 INFO:teuthology.orchestra.run.smithi179.stdout:Group : Unspecified 2023-10-19T12:33:17.714 INFO:teuthology.orchestra.run.smithi179.stdout:Size : 8264254 2023-10-19T12:33:17.715 INFO:teuthology.orchestra.run.smithi179.stdout:License : GPLv3+ 2023-10-19T12:33:17.715 INFO:teuthology.orchestra.run.smithi179.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-10-19T12:33:17.715 INFO:teuthology.orchestra.run.smithi179.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-10-19T12:33:17.715 INFO:teuthology.orchestra.run.smithi179.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-10-19T12:33:17.715 INFO:teuthology.orchestra.run.smithi179.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-10-19T12:33:17.715 INFO:teuthology.orchestra.run.smithi179.stdout:Packager : builder@centos.org 2023-10-19T12:33:17.716 INFO:teuthology.orchestra.run.smithi179.stdout:Vendor : CentOS 2023-10-19T12:33:17.716 INFO:teuthology.orchestra.run.smithi179.stdout:URL : http://www.gnu.org/software/grub/ 2023-10-19T12:33:17.716 INFO:teuthology.orchestra.run.smithi179.stdout:Summary : Support tools for GRUB. 2023-10-19T12:33:17.716 INFO:teuthology.orchestra.run.smithi179.stdout:Description : 2023-10-19T12:33:17.716 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-10-19T12:33:17.717 INFO:teuthology.orchestra.run.smithi179.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-10-19T12:33:17.717 INFO:teuthology.orchestra.run.smithi179.stdout:customizable bootloader with modular architecture. It supports a rich 2023-10-19T12:33:17.717 INFO:teuthology.orchestra.run.smithi179.stdout:variety of kernel formats, file systems, computer architectures and 2023-10-19T12:33:17.717 INFO:teuthology.orchestra.run.smithi179.stdout:hardware devices. 2023-10-19T12:33:17.717 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-10-19T12:33:17.718 INFO:teuthology.orchestra.run.smithi179.stdout:This subpackage provides tools for support of all platforms. 2023-10-19T12:33:17.719 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-10-19T12:33:17.719 INFO:teuthology.task.kernel:Updating grub on smithi179 to boot 5.14.0-376.el9.x86_64 2023-10-19T12:33:17.719 DEBUG:teuthology.orchestra.run.smithi179:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-10-19T12:33:18.357 INFO:teuthology.orchestra.run.smithi179.stderr:Generating grub configuration file ... 2023-10-19T12:33:19.681 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: kernel-core-5.14.0-376.el9.x86_64 4/4 2023-10-19T12:33:19.696 INFO:teuthology.orchestra.run.smithi179.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-10-19T12:33:19.725 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: kernel-modules-5.14.0-376.el9.x86_64 4/4 2023-10-19T12:33:19.741 INFO:teuthology.orchestra.run.smithi179.stderr:done 2023-10-19T12:33:19.744 DEBUG:teuthology.orchestra.run.smithi179:> mktemp 2023-10-19T12:33:19.759 INFO:teuthology.orchestra.run.smithi179.stdout:/tmp/tmp.HUJh5qfg7b 2023-10-19T12:33:19.760 DEBUG:teuthology.orchestra.run.smithi179:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.HUJh5qfg7b 2023-10-19T12:33:19.828 DEBUG:teuthology.orchestra.run.smithi179:> sudo chmod 0666 /tmp/tmp.HUJh5qfg7b 2023-10-19T12:33:19.978 DEBUG:teuthology.orchestra.remote:smithi179:/tmp/tmp.HUJh5qfg7b is 6KB 2023-10-19T12:33:20.025 DEBUG:teuthology.orchestra.run.smithi179:> rm -fr /tmp/tmp.HUJh5qfg7b 2023-10-19T12:33:20.040 DEBUG:teuthology.orchestra.run.smithi179:> sudo /bin/ls /boot/loader/entries || true 2023-10-19T12:33:20.094 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: kernel-5.14.0-376.el9.x86_64 4/4 2023-10-19T12:33:20.095 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : kernel-5.14.0-376.el9.x86_64 1/4 2023-10-19T12:33:20.095 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : kernel-core-5.14.0-376.el9.x86_64 2/4 2023-10-19T12:33:20.095 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : kernel-modules-5.14.0-376.el9.x86_64 3/4 2023-10-19T12:33:20.104 INFO:teuthology.orchestra.run.smithi179.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-10-19T12:33:20.104 INFO:teuthology.orchestra.run.smithi179.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-10-19T12:33:20.104 INFO:teuthology.orchestra.run.smithi179.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-10-19T12:33:20.105 INFO:teuthology.orchestra.run.smithi179.stdout:15949e3c88704300882482eddfd7baae-5.14.0-376.el9.x86_64.conf 2023-10-19T12:33:20.106 DEBUG:teuthology.orchestra.run.smithi179:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-376.el9.x86_64 2023-10-19T12:33:20.255 DEBUG:teuthology.orchestra.run.smithi179:> sudo shutdown -r now 2023-10-19T12:33:20.534 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : kernel-modules-core-5.14.0-376.el9.x86_64 4/4 2023-10-19T12:33:20.534 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:33:20.534 INFO:teuthology.orchestra.run.smithi057.stdout:Installed: 2023-10-19T12:33:20.535 INFO:teuthology.orchestra.run.smithi057.stdout: kernel-5.14.0-376.el9.x86_64 2023-10-19T12:33:20.535 INFO:teuthology.orchestra.run.smithi057.stdout: kernel-core-5.14.0-376.el9.x86_64 2023-10-19T12:33:20.535 INFO:teuthology.orchestra.run.smithi057.stdout: kernel-modules-5.14.0-376.el9.x86_64 2023-10-19T12:33:20.535 INFO:teuthology.orchestra.run.smithi057.stdout: kernel-modules-core-5.14.0-376.el9.x86_64 2023-10-19T12:33:20.535 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:33:20.535 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2023-10-19T12:33:21.018 DEBUG:teuthology.orchestra.run.smithi057:> rpm -q kernel | sort -rV | head -n 1 2023-10-19T12:33:21.148 INFO:teuthology.orchestra.run.smithi057.stdout:kernel-5.14.0-376.el9.x86_64 2023-10-19T12:33:21.149 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-376.el9.x86_64 2023-10-19T12:33:21.149 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-376.el9.x86_64 2023-10-19T12:33:21.150 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-19T12:33:21.150 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-19T12:33:21.150 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-10-19T12:33:21.151 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi057.front.sepia.ceph.com, path=None, version=distro) 2023-10-19T12:33:21.151 DEBUG:teuthology.orchestra.run.smithi057:> rpm -q kernel | sort -rV | head -n 1 2023-10-19T12:33:21.261 INFO:teuthology.orchestra.run.smithi057.stdout:kernel-5.14.0-376.el9.x86_64 2023-10-19T12:33:21.261 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-376.el9.x86_64 2023-10-19T12:33:21.262 DEBUG:teuthology.orchestra.run.smithi057:> sudo rpm -qi grub2-tools 2023-10-19T12:33:21.307 INFO:teuthology.orchestra.run.smithi057.stdout:Name : grub2-tools 2023-10-19T12:33:21.307 INFO:teuthology.orchestra.run.smithi057.stdout:Epoch : 1 2023-10-19T12:33:21.307 INFO:teuthology.orchestra.run.smithi057.stdout:Version : 2.06 2023-10-19T12:33:21.307 INFO:teuthology.orchestra.run.smithi057.stdout:Release : 68.el9 2023-10-19T12:33:21.308 INFO:teuthology.orchestra.run.smithi057.stdout:Architecture: x86_64 2023-10-19T12:33:21.308 INFO:teuthology.orchestra.run.smithi057.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-10-19T12:33:21.308 INFO:teuthology.orchestra.run.smithi057.stdout:Group : Unspecified 2023-10-19T12:33:21.308 INFO:teuthology.orchestra.run.smithi057.stdout:Size : 8264254 2023-10-19T12:33:21.309 INFO:teuthology.orchestra.run.smithi057.stdout:License : GPLv3+ 2023-10-19T12:33:21.309 INFO:teuthology.orchestra.run.smithi057.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-10-19T12:33:21.309 INFO:teuthology.orchestra.run.smithi057.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-10-19T12:33:21.309 INFO:teuthology.orchestra.run.smithi057.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-10-19T12:33:21.309 INFO:teuthology.orchestra.run.smithi057.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-10-19T12:33:21.310 INFO:teuthology.orchestra.run.smithi057.stdout:Packager : builder@centos.org 2023-10-19T12:33:21.310 INFO:teuthology.orchestra.run.smithi057.stdout:Vendor : CentOS 2023-10-19T12:33:21.310 INFO:teuthology.orchestra.run.smithi057.stdout:URL : http://www.gnu.org/software/grub/ 2023-10-19T12:33:21.310 INFO:teuthology.orchestra.run.smithi057.stdout:Summary : Support tools for GRUB. 2023-10-19T12:33:21.310 INFO:teuthology.orchestra.run.smithi057.stdout:Description : 2023-10-19T12:33:21.311 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:33:21.311 INFO:teuthology.orchestra.run.smithi057.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-10-19T12:33:21.311 INFO:teuthology.orchestra.run.smithi057.stdout:customizable bootloader with modular architecture. It supports a rich 2023-10-19T12:33:21.311 INFO:teuthology.orchestra.run.smithi057.stdout:variety of kernel formats, file systems, computer architectures and 2023-10-19T12:33:21.311 INFO:teuthology.orchestra.run.smithi057.stdout:hardware devices. 2023-10-19T12:33:21.312 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:33:21.312 INFO:teuthology.orchestra.run.smithi057.stdout:This subpackage provides tools for support of all platforms. 2023-10-19T12:33:21.313 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-10-19T12:33:21.313 INFO:teuthology.task.kernel:Updating grub on smithi057 to boot 5.14.0-376.el9.x86_64 2023-10-19T12:33:21.313 DEBUG:teuthology.orchestra.run.smithi057:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-10-19T12:33:21.853 INFO:teuthology.orchestra.run.smithi057.stderr:Generating grub configuration file ... 2023-10-19T12:33:23.270 INFO:teuthology.orchestra.run.smithi057.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-10-19T12:33:23.300 INFO:teuthology.orchestra.run.smithi057.stderr:done 2023-10-19T12:33:23.302 DEBUG:teuthology.orchestra.run.smithi057:> mktemp 2023-10-19T12:33:23.318 INFO:teuthology.orchestra.run.smithi057.stdout:/tmp/tmp.AqtYoItaAH 2023-10-19T12:33:23.318 DEBUG:teuthology.orchestra.run.smithi057:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.AqtYoItaAH 2023-10-19T12:33:23.405 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod 0666 /tmp/tmp.AqtYoItaAH 2023-10-19T12:33:23.501 DEBUG:teuthology.orchestra.remote:smithi057:/tmp/tmp.AqtYoItaAH is 6KB 2023-10-19T12:33:23.549 DEBUG:teuthology.orchestra.run.smithi057:> rm -fr /tmp/tmp.AqtYoItaAH 2023-10-19T12:33:23.563 DEBUG:teuthology.orchestra.run.smithi057:> sudo /bin/ls /boot/loader/entries || true 2023-10-19T12:33:23.638 INFO:teuthology.orchestra.run.smithi057.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-10-19T12:33:23.639 INFO:teuthology.orchestra.run.smithi057.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-10-19T12:33:23.639 INFO:teuthology.orchestra.run.smithi057.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-10-19T12:33:23.639 INFO:teuthology.orchestra.run.smithi057.stdout:15949e3c88704300882482eddfd7baae-5.14.0-376.el9.x86_64.conf 2023-10-19T12:33:23.640 DEBUG:teuthology.orchestra.run.smithi057:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-376.el9.x86_64 2023-10-19T12:33:23.794 DEBUG:teuthology.orchestra.run.smithi057:> sudo shutdown -r now 2023-10-19T12:33:50.282 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2023-10-19T12:33:50.283 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi179.front.sepia.ceph.com' 2023-10-19T12:33:50.284 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi179.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-19T12:33:53.798 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-10-19T12:33:53.799 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com' 2023-10-19T12:33:53.799 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi057.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-19T12:34:08.634 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.179 2023-10-19T12:34:12.218 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.57 2023-10-19T12:34:17.642 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi179.front.sepia.ceph.com' 2023-10-19T12:34:17.643 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi179.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-19T12:34:21.222 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com' 2023-10-19T12:34:21.223 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi057.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-19T12:34:24.282 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.57 2023-10-19T12:34:36.291 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com' 2023-10-19T12:34:36.292 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi057.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-19T12:35:17.687 DEBUG:teuthology.orchestra.remote:timed out 2023-10-19T12:35:29.698 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi179.front.sepia.ceph.com' 2023-10-19T12:35:29.699 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi179.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-19T12:35:30.022 DEBUG:teuthology.orchestra.run.smithi179:> true 2023-10-19T12:35:30.396 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi179.front.sepia.ceph.com' 2023-10-19T12:35:30.396 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-376.el9.x86_64"... 2023-10-19T12:35:30.397 DEBUG:teuthology.orchestra.run.smithi179:> uname -r 2023-10-19T12:35:30.412 INFO:teuthology.orchestra.run.smithi179.stdout:5.14.0-376.el9.x86_64 2023-10-19T12:35:30.413 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-376.el9.x86_64 vs 5.14.0-376.el9.x86_64 2023-10-19T12:35:30.413 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-19T12:35:30.413 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-10-19T12:35:31.414 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2023-10-19T12:35:31.415 DEBUG:teuthology.orchestra.run.smithi179:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-10-19T12:35:31.533 INFO:teuthology.orchestra.run.smithi179.stdout:ttyS1 2023-10-19T12:35:31.565 DEBUG:teuthology.parallel:result is None 2023-10-19T12:35:36.295 DEBUG:teuthology.orchestra.remote:timed out 2023-10-19T12:35:51.296 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com' 2023-10-19T12:35:51.297 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi057.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-19T12:35:51.716 DEBUG:teuthology.orchestra.run.smithi057:> true 2023-10-19T12:35:52.086 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi057.front.sepia.ceph.com' 2023-10-19T12:35:52.086 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-376.el9.x86_64"... 2023-10-19T12:35:52.087 DEBUG:teuthology.orchestra.run.smithi057:> uname -r 2023-10-19T12:35:52.102 INFO:teuthology.orchestra.run.smithi057.stdout:5.14.0-376.el9.x86_64 2023-10-19T12:35:52.103 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-376.el9.x86_64 vs 5.14.0-376.el9.x86_64 2023-10-19T12:35:52.103 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-19T12:35:52.103 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-10-19T12:35:53.104 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-10-19T12:35:53.105 DEBUG:teuthology.orchestra.run.smithi057:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-10-19T12:35:53.177 INFO:teuthology.orchestra.run.smithi057.stdout:ttyS1 2023-10-19T12:35:53.215 DEBUG:teuthology.parallel:result is None 2023-10-19T12:35:53.216 INFO:teuthology.run_tasks:Running task internal.base... 2023-10-19T12:35:53.225 INFO:teuthology.task.internal:Creating test directory... 2023-10-19T12:35:53.225 DEBUG:teuthology.orchestra.run.smithi057:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-19T12:35:53.227 DEBUG:teuthology.orchestra.run.smithi179:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-19T12:35:53.245 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-10-19T12:35:53.276 INFO:teuthology.run_tasks:Running task internal.archive... 2023-10-19T12:35:53.320 INFO:teuthology.task.internal:Creating archive directory... 2023-10-19T12:35:53.320 DEBUG:teuthology.orchestra.run.smithi057:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-19T12:35:53.323 DEBUG:teuthology.orchestra.run.smithi179:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-19T12:35:53.360 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-10-19T12:35:53.371 INFO:teuthology.task.internal:Enabling coredump saving... 2023-10-19T12:35:53.372 DEBUG:teuthology.orchestra.run.smithi057:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-19T12:35:53.388 DEBUG:teuthology.orchestra.run.smithi179:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-19T12:35:53.428 INFO:teuthology.orchestra.run.smithi057.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-19T12:35:53.442 INFO:teuthology.orchestra.run.smithi179.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-19T12:35:53.446 INFO:teuthology.orchestra.run.smithi057.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-19T12:35:53.456 INFO:teuthology.orchestra.run.smithi179.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-19T12:35:53.458 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-10-19T12:35:53.466 INFO:teuthology.task.internal:Configuring sudo... 2023-10-19T12:35:53.467 DEBUG:teuthology.orchestra.run.smithi057:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-19T12:35:53.490 DEBUG:teuthology.orchestra.run.smithi179:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-19T12:35:53.521 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-10-19T12:35:53.534 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-10-19T12:35:53.534 DEBUG:teuthology.orchestra.run.smithi057:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-19T12:35:53.563 DEBUG:teuthology.orchestra.run.smithi179:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-19T12:35:53.580 DEBUG:teuthology.orchestra.run.smithi057:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-19T12:35:53.662 DEBUG:teuthology.orchestra.run.smithi057:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-19T12:35:53.710 DEBUG:teuthology.orchestra.run.smithi057:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-19T12:35:53.794 DEBUG:teuthology.orchestra.run.smithi057:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-19T12:35:53.838 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T12:35:53.839 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-19T12:35:53.909 DEBUG:teuthology.orchestra.run.smithi179:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-19T12:35:53.953 DEBUG:teuthology.orchestra.run.smithi179:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-19T12:35:54.004 DEBUG:teuthology.orchestra.run.smithi179:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-19T12:35:54.088 DEBUG:teuthology.orchestra.run.smithi179:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-19T12:35:54.131 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-19T12:35:54.132 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-19T12:35:54.200 DEBUG:teuthology.orchestra.run.smithi057:> sudo service rsyslog restart 2023-10-19T12:35:54.203 DEBUG:teuthology.orchestra.run.smithi179:> sudo service rsyslog restart 2023-10-19T12:35:54.245 INFO:teuthology.orchestra.run.smithi057.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-19T12:35:54.284 INFO:teuthology.orchestra.run.smithi179.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-19T12:35:54.713 INFO:teuthology.run_tasks:Running task internal.timer... 2023-10-19T12:35:54.722 INFO:teuthology.task.internal:Starting timer... 2023-10-19T12:35:54.722 INFO:teuthology.run_tasks:Running task pcp... 2023-10-19T12:35:54.777 INFO:teuthology.run_tasks:Running task selinux... 2023-10-19T12:35:54.820 DEBUG:teuthology.orchestra.run.smithi057:> sudo service auditd rotate 2023-10-19T12:35:54.886 INFO:teuthology.orchestra.run.smithi057.stdout:Rotating logs: 2023-10-19T12:35:54.888 DEBUG:teuthology.orchestra.run.smithi179:> sudo service auditd rotate 2023-10-19T12:35:54.952 INFO:teuthology.orchestra.run.smithi179.stdout:Rotating logs: 2023-10-19T12:35:54.953 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-10-19T12:35:54.954 DEBUG:teuthology.orchestra.run.smithi057:> /usr/sbin/getenforce 2023-10-19T12:35:54.975 INFO:teuthology.orchestra.run.smithi057.stdout:Permissive 2023-10-19T12:35:54.975 DEBUG:teuthology.orchestra.run.smithi179:> /usr/sbin/getenforce 2023-10-19T12:35:55.005 INFO:teuthology.orchestra.run.smithi179.stdout:Permissive 2023-10-19T12:35:55.005 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi057.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi179.front.sepia.ceph.com': 'permissive'} 2023-10-19T12:35:55.005 DEBUG:teuthology.orchestra.run.smithi057:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-19T12:35:55.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-19T12:35:55.046 DEBUG:teuthology.orchestra.run.smithi179:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-19T12:35:55.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-19T12:35:55.074 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-10-19T12:35:55.075 DEBUG:teuthology.orchestra.run.smithi057:> sudo /usr/sbin/setenforce permissive 2023-10-19T12:35:55.127 DEBUG:teuthology.orchestra.run.smithi179:> sudo /usr/sbin/setenforce permissive 2023-10-19T12:35:55.157 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-10-19T12:35:55.167 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-10-19T12:35:55.173 INFO:teuthology.repo_utils:Fetching main from origin 2023-10-19T12:35:55.321 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-10-19T12:35:55.335 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-10-19T12:35:55.337 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi057.front.sepia.ceph.com,smithi179.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-10-19T12:41:26.268 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi057.front.sepia.ceph.com'), Remote(name='ubuntu@smithi179.front.sepia.ceph.com')] 2023-10-19T12:41:26.269 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com' 2023-10-19T12:41:26.270 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi057.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-19T12:41:26.370 DEBUG:teuthology.orchestra.run.smithi057:> true 2023-10-19T12:41:26.438 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi057.front.sepia.ceph.com' 2023-10-19T12:41:26.439 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi179.front.sepia.ceph.com' 2023-10-19T12:41:26.439 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi179.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-19T12:41:26.521 DEBUG:teuthology.orchestra.run.smithi179:> true 2023-10-19T12:41:26.589 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi179.front.sepia.ceph.com' 2023-10-19T12:41:26.590 INFO:teuthology.run_tasks:Running task clock... 2023-10-19T12:41:26.600 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-10-19T12:41:26.600 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-19T12:41:26.601 DEBUG:teuthology.orchestra.run.smithi057:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-19T12:41:26.603 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-19T12:41:26.604 DEBUG:teuthology.orchestra.run.smithi179:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-19T12:41:26.637 INFO:teuthology.orchestra.run.smithi057.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-10-19T12:41:26.658 INFO:teuthology.orchestra.run.smithi057.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-10-19T12:41:26.661 INFO:teuthology.orchestra.run.smithi179.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-10-19T12:41:26.678 INFO:teuthology.orchestra.run.smithi179.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-10-19T12:41:26.695 INFO:teuthology.orchestra.run.smithi057.stderr:sudo: ntpd: command not found 2023-10-19T12:41:26.708 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: ntpd: command not found 2023-10-19T12:41:26.712 INFO:teuthology.orchestra.run.smithi057.stdout:506 Cannot talk to daemon 2023-10-19T12:41:26.722 INFO:teuthology.orchestra.run.smithi179.stdout:506 Cannot talk to daemon 2023-10-19T12:41:26.732 INFO:teuthology.orchestra.run.smithi057.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-10-19T12:41:26.737 INFO:teuthology.orchestra.run.smithi179.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-10-19T12:41:26.752 INFO:teuthology.orchestra.run.smithi057.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-10-19T12:41:26.753 INFO:teuthology.orchestra.run.smithi179.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-10-19T12:41:26.797 INFO:teuthology.orchestra.run.smithi179.stderr:bash: line 1: ntpq: command not found 2023-10-19T12:41:26.799 INFO:teuthology.orchestra.run.smithi057.stderr:bash: line 1: ntpq: command not found 2023-10-19T12:41:26.799 INFO:teuthology.orchestra.run.smithi179.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-19T12:41:26.800 INFO:teuthology.orchestra.run.smithi179.stdout:=============================================================================== 2023-10-19T12:41:26.800 INFO:teuthology.orchestra.run.smithi179.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-19T12:41:26.800 INFO:teuthology.orchestra.run.smithi179.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-19T12:41:26.800 INFO:teuthology.orchestra.run.smithi179.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-19T12:41:26.800 INFO:teuthology.orchestra.run.smithi179.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-19T12:41:26.802 INFO:teuthology.orchestra.run.smithi057.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-19T12:41:26.802 INFO:teuthology.orchestra.run.smithi057.stdout:=============================================================================== 2023-10-19T12:41:26.803 INFO:teuthology.orchestra.run.smithi057.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-19T12:41:26.803 INFO:teuthology.orchestra.run.smithi057.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-19T12:41:26.803 INFO:teuthology.orchestra.run.smithi057.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-19T12:41:26.803 INFO:teuthology.orchestra.run.smithi057.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-19T12:41:26.804 INFO:teuthology.run_tasks:Running task install... 2023-10-19T12:41:26.819 DEBUG:teuthology.task.install:project ceph 2023-10-19T12:41:26.819 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'f7420d7f8e55ceea786dcbf536c338ce920c21eb'}} 2023-10-19T12:41:26.820 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'f7420d7f8e55ceea786dcbf536c338ce920c21eb'} 2023-10-19T12:41:26.821 INFO:teuthology.task.install:Using flavor: default 2023-10-19T12:41:26.828 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-10-19T12:41:26.828 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2023-10-19T12:41:26.829 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'f7420d7f8e55ceea786dcbf536c338ce920c21eb', 'tag': None, 'wait_for_package': False} 2023-10-19T12:41:26.829 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=f7420d7f8e55ceea786dcbf536c338ce920c21eb 2023-10-19T12:41:26.831 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'f7420d7f8e55ceea786dcbf536c338ce920c21eb', 'tag': None, 'wait_for_package': False} 2023-10-19T12:41:26.832 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=f7420d7f8e55ceea786dcbf536c338ce920c21eb 2023-10-19T12:41:27.002 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20231016.203825/f7420d7f8e55ceea786dcbf536c338ce920c21eb/centos/9/flavors/default/ 2023-10-19T12:41:27.002 INFO:teuthology.task.install.rpm:Package version is 18.0.0-6786.gf7420d7f 2023-10-19T12:41:27.072 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20231016.203825/f7420d7f8e55ceea786dcbf536c338ce920c21eb/centos/9/flavors/default/ 2023-10-19T12:41:27.073 INFO:teuthology.task.install.rpm:Package version is 18.0.0-6786.gf7420d7f 2023-10-19T12:41:27.195 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20231016.203825/f7420d7f8e55ceea786dcbf536c338ce920c21eb/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20231016.203825/f7420d7f8e55ceea786dcbf536c338ce920c21eb/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20231016.203825/f7420d7f8e55ceea786dcbf536c338ce920c21eb/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-10-19T12:41:27.196 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-19T12:41:27.196 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-10-19T12:41:27.238 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-10-19T12:41:27.239 DEBUG:teuthology.orchestra.run.smithi179:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/f7420d7f8e55ceea786dcbf536c338ce920c21eb/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-10-19T12:41:27.264 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20231016.203825/f7420d7f8e55ceea786dcbf536c338ce920c21eb/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20231016.203825/f7420d7f8e55ceea786dcbf536c338ce920c21eb/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20231016.203825/f7420d7f8e55ceea786dcbf536c338ce920c21eb/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-10-19T12:41:27.265 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T12:41:27.265 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-10-19T12:41:27.301 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-10-19T12:41:27.302 DEBUG:teuthology.orchestra.run.smithi057:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/f7420d7f8e55ceea786dcbf536c338ce920c21eb/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-10-19T12:41:27.313 DEBUG:teuthology.orchestra.run.smithi179:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-10-19T12:41:27.391 DEBUG:teuthology.orchestra.run.smithi057:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-10-19T12:41:27.403 DEBUG:teuthology.orchestra.run.smithi179:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-10-19T12:41:27.476 INFO:teuthology.orchestra.run.smithi179.stdout:check_obsoletes = 1 2023-10-19T12:41:27.479 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum clean all 2023-10-19T12:41:27.485 DEBUG:teuthology.orchestra.run.smithi057:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-10-19T12:41:27.516 INFO:teuthology.orchestra.run.smithi057.stdout:check_obsoletes = 1 2023-10-19T12:41:27.517 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum clean all 2023-10-19T12:41:27.739 INFO:teuthology.orchestra.run.smithi179.stdout:72 files removed 2023-10-19T12:41:27.767 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-10-19T12:41:27.774 INFO:teuthology.orchestra.run.smithi057.stdout:72 files removed 2023-10-19T12:41:27.801 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-10-19T12:41:28.498 INFO:teuthology.orchestra.run.smithi057.stdout:ceph packages for x86_64 217 kB/s | 78 kB 00:00 2023-10-19T12:41:28.521 INFO:teuthology.orchestra.run.smithi179.stdout:ceph packages for x86_64 181 kB/s | 78 kB 00:00 2023-10-19T12:41:28.847 INFO:teuthology.orchestra.run.smithi179.stdout:ceph noarch packages 34 kB/s | 9.4 kB 00:00 2023-10-19T12:41:28.874 INFO:teuthology.orchestra.run.smithi057.stdout:ceph noarch packages 29 kB/s | 9.4 kB 00:00 2023-10-19T12:41:29.206 INFO:teuthology.orchestra.run.smithi179.stdout:ceph source packages 5.8 kB/s | 1.9 kB 00:00 2023-10-19T12:41:29.250 INFO:teuthology.orchestra.run.smithi057.stdout:ceph source packages 5.6 kB/s | 1.9 kB 00:00 2023-10-19T12:41:29.592 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS-9 - AppStream 18 MB/s | 6.3 MB 00:00 2023-10-19T12:41:29.711 INFO:teuthology.orchestra.run.smithi057.stdout:CentOS-9 - AppStream 15 MB/s | 6.3 MB 00:00 2023-10-19T12:41:30.925 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS-9 - BaseOS 7.1 MB/s | 2.0 MB 00:00 2023-10-19T12:41:31.111 INFO:teuthology.orchestra.run.smithi057.stdout:CentOS-9 - BaseOS 6.6 MB/s | 2.0 MB 00:00 2023-10-19T12:41:31.868 INFO:teuthology.orchestra.run.smithi057.stdout:CentOS Stream 9 - BaseOS 18 MB/s | 7.9 MB 00:00 2023-10-19T12:41:33.691 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS Stream 9 - BaseOS 3.2 MB/s | 7.9 MB 00:02 2023-10-19T12:41:35.442 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS Stream 9 - AppStream 25 MB/s | 18 MB 00:00 2023-10-19T12:41:40.385 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS Stream 9 - CRB 3.5 MB/s | 5.7 MB 00:01 2023-10-19T12:41:41.744 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS Stream 9 - Extras packages 60 kB/s | 14 kB 00:00 2023-10-19T12:41:42.631 INFO:teuthology.orchestra.run.smithi179.stdout:Extra Packages for Enterprise Linux 24 MB/s | 19 MB 00:00 2023-10-19T12:41:45.677 INFO:teuthology.orchestra.run.smithi057.stdout:CentOS Stream 9 - AppStream 1.4 MB/s | 18 MB 00:12 2023-10-19T12:41:47.087 INFO:teuthology.orchestra.run.smithi179.stdout:lab-extras 26 kB/s | 1.7 kB 00:00 2023-10-19T12:41:49.382 INFO:teuthology.orchestra.run.smithi179.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2023-10-19T12:41:49.383 INFO:teuthology.orchestra.run.smithi179.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2023-10-19T12:41:49.394 INFO:teuthology.orchestra.run.smithi179.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2023-10-19T12:41:49.395 INFO:teuthology.orchestra.run.smithi179.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2023-10-19T12:41:49.396 INFO:teuthology.orchestra.run.smithi179.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2023-10-19T12:41:49.397 INFO:teuthology.orchestra.run.smithi179.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2023-10-19T12:41:49.399 INFO:teuthology.orchestra.run.smithi179.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2023-10-19T12:41:49.402 INFO:teuthology.orchestra.run.smithi179.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2023-10-19T12:41:49.403 INFO:teuthology.orchestra.run.smithi179.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2023-10-19T12:41:49.472 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2023-10-19T12:41:49.486 INFO:teuthology.orchestra.run.smithi179.stdout:============================================================================================ 2023-10-19T12:41:49.486 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repository Size 2023-10-19T12:41:49.487 INFO:teuthology.orchestra.run.smithi179.stdout:============================================================================================ 2023-10-19T12:41:49.487 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2023-10-19T12:41:49.487 INFO:teuthology.orchestra.run.smithi179.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2023-10-19T12:41:49.487 INFO:teuthology.orchestra.run.smithi179.stdout: ceph x86_64 2:18.0.0-6786.gf7420d7f.el9 ceph 6.5 k 2023-10-19T12:41:49.488 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-base x86_64 2:18.0.0-6786.gf7420d7f.el9 ceph 5.2 M 2023-10-19T12:41:49.488 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-fuse x86_64 2:18.0.0-6786.gf7420d7f.el9 ceph 857 k 2023-10-19T12:41:49.488 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-6786.gf7420d7f.el9 ceph 142 k 2023-10-19T12:41:49.488 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr x86_64 2:18.0.0-6786.gf7420d7f.el9 ceph 1.5 M 2023-10-19T12:41:49.488 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-cephadm noarch 2:18.0.0-6786.gf7420d7f.el9 ceph-noarch 132 k 2023-10-19T12:41:49.489 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-dashboard noarch 2:18.0.0-6786.gf7420d7f.el9 ceph-noarch 1.8 M 2023-10-19T12:41:49.489 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-6786.gf7420d7f.el9 ceph-noarch 7.4 M 2023-10-19T12:41:49.489 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-rook noarch 2:18.0.0-6786.gf7420d7f.el9 ceph-noarch 47 k 2023-10-19T12:41:49.489 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-radosgw x86_64 2:18.0.0-6786.gf7420d7f.el9 ceph 8.6 M 2023-10-19T12:41:49.489 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-test x86_64 2:18.0.0-6786.gf7420d7f.el9 ceph 41 M 2023-10-19T12:41:49.490 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-volume noarch 2:18.0.0-6786.gf7420d7f.el9 ceph-noarch 262 k 2023-10-19T12:41:49.490 INFO:teuthology.orchestra.run.smithi179.stdout: cephadm noarch 2:18.0.0-6786.gf7420d7f.el9 ceph-noarch 270 k 2023-10-19T12:41:49.490 INFO:teuthology.orchestra.run.smithi179.stdout: cephfs-mirror x86_64 2:18.0.0-6786.gf7420d7f.el9 ceph 218 k 2023-10-19T12:41:49.490 INFO:teuthology.orchestra.run.smithi179.stdout: cephfs-top noarch 2:18.0.0-6786.gf7420d7f.el9 ceph-noarch 18 k 2023-10-19T12:41:49.491 INFO:teuthology.orchestra.run.smithi179.stdout: elfutils-libelf-devel x86_64 0.189-3.el9 appstream 22 k 2023-10-19T12:41:49.491 INFO:teuthology.orchestra.run.smithi179.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2023-10-19T12:41:49.491 INFO:teuthology.orchestra.run.smithi179.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2023-10-19T12:41:49.491 INFO:teuthology.orchestra.run.smithi179.stdout: libcephfs-devel x86_64 2:18.0.0-6786.gf7420d7f.el9 ceph 31 k 2023-10-19T12:41:49.491 INFO:teuthology.orchestra.run.smithi179.stdout: libcephfs2 x86_64 2:18.0.0-6786.gf7420d7f.el9 ceph 716 k 2023-10-19T12:41:49.492 INFO:teuthology.orchestra.run.smithi179.stdout: librados-devel x86_64 2:18.0.0-6786.gf7420d7f.el9 ceph 127 k 2023-10-19T12:41:49.492 INFO:teuthology.orchestra.run.smithi179.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2023-10-19T12:41:49.492 INFO:teuthology.orchestra.run.smithi179.stdout: libuuid-devel x86_64 2.37.4-15.el9 appstream 26 k 2023-10-19T12:41:49.492 INFO:teuthology.orchestra.run.smithi179.stdout: openssl-devel x86_64 1:3.0.7-24.el9 appstream 4.1 M 2023-10-19T12:41:49.493 INFO:teuthology.orchestra.run.smithi179.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2023-10-19T12:41:49.493 INFO:teuthology.orchestra.run.smithi179.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2023-10-19T12:41:49.493 INFO:teuthology.orchestra.run.smithi179.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2023-10-19T12:41:49.493 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cephfs x86_64 2:18.0.0-6786.gf7420d7f.el9 ceph 162 k 2023-10-19T12:41:49.494 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rados x86_64 2:18.0.0-6786.gf7420d7f.el9 ceph 321 k 2023-10-19T12:41:49.494 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rbd x86_64 2:18.0.0-6786.gf7420d7f.el9 ceph 297 k 2023-10-19T12:41:49.494 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rgw x86_64 2:18.0.0-6786.gf7420d7f.el9 ceph 99 k 2023-10-19T12:41:49.494 INFO:teuthology.orchestra.run.smithi179.stdout: rbd-fuse x86_64 2:18.0.0-6786.gf7420d7f.el9 ceph 86 k 2023-10-19T12:41:49.494 INFO:teuthology.orchestra.run.smithi179.stdout: rbd-mirror x86_64 2:18.0.0-6786.gf7420d7f.el9 ceph 3.0 M 2023-10-19T12:41:49.495 INFO:teuthology.orchestra.run.smithi179.stdout: rbd-nbd x86_64 2:18.0.0-6786.gf7420d7f.el9 ceph 169 k 2023-10-19T12:41:49.495 INFO:teuthology.orchestra.run.smithi179.stdout: xfsprogs-devel x86_64 5.19.0-4.el9 appstream 86 k 2023-10-19T12:41:49.495 INFO:teuthology.orchestra.run.smithi179.stdout:Upgrading: 2023-10-19T12:41:49.495 INFO:teuthology.orchestra.run.smithi179.stdout: NetworkManager x86_64 1:1.45.4-1.el9 CentOS-BaseOS 2.2 M 2023-10-19T12:41:49.496 INFO:teuthology.orchestra.run.smithi179.stdout: NetworkManager-libnm x86_64 1:1.45.4-1.el9 CentOS-BaseOS 1.8 M 2023-10-19T12:41:49.496 INFO:teuthology.orchestra.run.smithi179.stdout: NetworkManager-team x86_64 1:1.45.4-1.el9 CentOS-BaseOS 41 k 2023-10-19T12:41:49.496 INFO:teuthology.orchestra.run.smithi179.stdout: NetworkManager-tui x86_64 1:1.45.4-1.el9 CentOS-BaseOS 244 k 2023-10-19T12:41:49.496 INFO:teuthology.orchestra.run.smithi179.stdout: acl x86_64 2.3.1-4.el9 CentOS-BaseOS 69 k 2023-10-19T12:41:49.496 INFO:teuthology.orchestra.run.smithi179.stdout: libacl x86_64 2.3.1-4.el9 CentOS-BaseOS 22 k 2023-10-19T12:41:49.497 INFO:teuthology.orchestra.run.smithi179.stdout: libacl-devel x86_64 2.3.1-4.el9 CentOS-AppStream 68 k 2023-10-19T12:41:49.497 INFO:teuthology.orchestra.run.smithi179.stdout: librados2 x86_64 2:18.0.0-6786.gf7420d7f.el9 ceph 3.3 M 2023-10-19T12:41:49.497 INFO:teuthology.orchestra.run.smithi179.stdout: librbd1 x86_64 2:18.0.0-6786.gf7420d7f.el9 ceph 3.0 M 2023-10-19T12:41:49.497 INFO:teuthology.orchestra.run.smithi179.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2023-10-19T12:41:49.497 INFO:teuthology.orchestra.run.smithi179.stdout:Installing dependencies: 2023-10-19T12:41:49.498 INFO:teuthology.orchestra.run.smithi179.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2023-10-19T12:41:49.498 INFO:teuthology.orchestra.run.smithi179.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-10-19T12:41:49.498 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-common x86_64 2:18.0.0-6786.gf7420d7f.el9 ceph 19 M 2023-10-19T12:41:49.498 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-grafana-dashboards noarch 2:18.0.0-6786.gf7420d7f.el9 ceph-noarch 23 k 2023-10-19T12:41:49.498 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mds x86_64 2:18.0.0-6786.gf7420d7f.el9 ceph 2.1 M 2023-10-19T12:41:49.499 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-modules-core noarch 2:18.0.0-6786.gf7420d7f.el9 ceph-noarch 242 k 2023-10-19T12:41:49.499 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mon x86_64 2:18.0.0-6786.gf7420d7f.el9 ceph 4.4 M 2023-10-19T12:41:49.499 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-osd x86_64 2:18.0.0-6786.gf7420d7f.el9 ceph 18 M 2023-10-19T12:41:49.499 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-prometheus-alerts noarch 2:18.0.0-6786.gf7420d7f.el9 ceph-noarch 15 k 2023-10-19T12:41:49.500 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-selinux x86_64 2:18.0.0-6786.gf7420d7f.el9 ceph 24 k 2023-10-19T12:41:49.500 INFO:teuthology.orchestra.run.smithi179.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-10-19T12:41:49.500 INFO:teuthology.orchestra.run.smithi179.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-10-19T12:41:49.500 INFO:teuthology.orchestra.run.smithi179.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-10-19T12:41:49.501 INFO:teuthology.orchestra.run.smithi179.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2023-10-19T12:41:49.501 INFO:teuthology.orchestra.run.smithi179.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-10-19T12:41:49.501 INFO:teuthology.orchestra.run.smithi179.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-10-19T12:41:49.501 INFO:teuthology.orchestra.run.smithi179.stdout: jq x86_64 1.6-15.el9 appstream 187 k 2023-10-19T12:41:49.501 INFO:teuthology.orchestra.run.smithi179.stdout: libarrow x86_64 9.0.0-9.el9 epel 4.4 M 2023-10-19T12:41:49.501 INFO:teuthology.orchestra.run.smithi179.stdout: libarrow-doc noarch 9.0.0-9.el9 epel 24 k 2023-10-19T12:41:49.502 INFO:teuthology.orchestra.run.smithi179.stdout: libcephsqlite x86_64 2:18.0.0-6786.gf7420d7f.el9 ceph 164 k 2023-10-19T12:41:49.502 INFO:teuthology.orchestra.run.smithi179.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-10-19T12:41:49.502 INFO:teuthology.orchestra.run.smithi179.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-10-19T12:41:49.502 INFO:teuthology.orchestra.run.smithi179.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-10-19T12:41:49.503 INFO:teuthology.orchestra.run.smithi179.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-10-19T12:41:49.503 INFO:teuthology.orchestra.run.smithi179.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2023-10-19T12:41:49.503 INFO:teuthology.orchestra.run.smithi179.stdout: libradosstriper1 x86_64 2:18.0.0-6786.gf7420d7f.el9 ceph 472 k 2023-10-19T12:41:49.503 INFO:teuthology.orchestra.run.smithi179.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-10-19T12:41:49.504 INFO:teuthology.orchestra.run.smithi179.stdout: librgw2 x86_64 2:18.0.0-6786.gf7420d7f.el9 ceph 4.8 M 2023-10-19T12:41:49.504 INFO:teuthology.orchestra.run.smithi179.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-10-19T12:41:49.504 INFO:teuthology.orchestra.run.smithi179.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2023-10-19T12:41:49.505 INFO:teuthology.orchestra.run.smithi179.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-10-19T12:41:49.505 INFO:teuthology.orchestra.run.smithi179.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-10-19T12:41:49.505 INFO:teuthology.orchestra.run.smithi179.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-10-19T12:41:49.505 INFO:teuthology.orchestra.run.smithi179.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-10-19T12:41:49.505 INFO:teuthology.orchestra.run.smithi179.stdout: parquet-libs x86_64 9.0.0-9.el9 epel 837 k 2023-10-19T12:41:49.506 INFO:teuthology.orchestra.run.smithi179.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2023-10-19T12:41:49.506 INFO:teuthology.orchestra.run.smithi179.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-10-19T12:41:49.506 INFO:teuthology.orchestra.run.smithi179.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-10-19T12:41:49.506 INFO:teuthology.orchestra.run.smithi179.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-10-19T12:41:49.507 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-10-19T12:41:49.507 INFO:teuthology.orchestra.run.smithi179.stdout: python3-ceph-argparse x86_64 2:18.0.0-6786.gf7420d7f.el9 ceph 45 k 2023-10-19T12:41:49.507 INFO:teuthology.orchestra.run.smithi179.stdout: python3-ceph-common x86_64 2:18.0.0-6786.gf7420d7f.el9 ceph 127 k 2023-10-19T12:41:49.507 INFO:teuthology.orchestra.run.smithi179.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-10-19T12:41:49.507 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-10-19T12:41:49.508 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-10-19T12:41:49.508 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-10-19T12:41:49.508 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2023-10-19T12:41:49.508 INFO:teuthology.orchestra.run.smithi179.stdout: python3-devel x86_64 3.9.17-2.el9 CentOS-AppStream 205 k 2023-10-19T12:41:49.509 INFO:teuthology.orchestra.run.smithi179.stdout: python3-google-auth noarch 1:2.23.3-1.el9 epel 197 k 2023-10-19T12:41:49.509 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-10-19T12:41:49.509 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-10-19T12:41:49.510 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-10-19T12:41:49.510 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-10-19T12:41:49.510 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-10-19T12:41:49.510 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-10-19T12:41:49.510 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-10-19T12:41:49.511 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-10-19T12:41:49.511 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-10-19T12:41:49.511 INFO:teuthology.orchestra.run.smithi179.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-10-19T12:41:49.511 INFO:teuthology.orchestra.run.smithi179.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-10-19T12:41:49.512 INFO:teuthology.orchestra.run.smithi179.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-10-19T12:41:49.512 INFO:teuthology.orchestra.run.smithi179.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-10-19T12:41:49.512 INFO:teuthology.orchestra.run.smithi179.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-10-19T12:41:49.512 INFO:teuthology.orchestra.run.smithi179.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-10-19T12:41:49.512 INFO:teuthology.orchestra.run.smithi179.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-10-19T12:41:49.513 INFO:teuthology.orchestra.run.smithi179.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-10-19T12:41:49.513 INFO:teuthology.orchestra.run.smithi179.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-10-19T12:41:49.513 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-10-19T12:41:49.514 INFO:teuthology.orchestra.run.smithi179.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-10-19T12:41:49.514 INFO:teuthology.orchestra.run.smithi179.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-10-19T12:41:49.514 INFO:teuthology.orchestra.run.smithi179.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-10-19T12:41:49.515 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-10-19T12:41:49.515 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-10-19T12:41:49.515 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-10-19T12:41:49.516 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-10-19T12:41:49.516 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2023-10-19T12:41:49.516 INFO:teuthology.orchestra.run.smithi179.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-10-19T12:41:49.517 INFO:teuthology.orchestra.run.smithi179.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-10-19T12:41:49.517 INFO:teuthology.orchestra.run.smithi179.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-10-19T12:41:49.518 INFO:teuthology.orchestra.run.smithi179.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-10-19T12:41:49.518 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-10-19T12:41:49.518 INFO:teuthology.orchestra.run.smithi179.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-10-19T12:41:49.519 INFO:teuthology.orchestra.run.smithi179.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-10-19T12:41:49.519 INFO:teuthology.orchestra.run.smithi179.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-10-19T12:41:49.520 INFO:teuthology.orchestra.run.smithi179.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-10-19T12:41:49.520 INFO:teuthology.orchestra.run.smithi179.stdout: python3-urllib3 noarch 1.26.5-3.el9 CentOS-BaseOS 188 k 2023-10-19T12:41:49.520 INFO:teuthology.orchestra.run.smithi179.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-10-19T12:41:49.521 INFO:teuthology.orchestra.run.smithi179.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-10-19T12:41:49.521 INFO:teuthology.orchestra.run.smithi179.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-10-19T12:41:49.521 INFO:teuthology.orchestra.run.smithi179.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-10-19T12:41:49.521 INFO:teuthology.orchestra.run.smithi179.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-10-19T12:41:49.521 INFO:teuthology.orchestra.run.smithi179.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-10-19T12:41:49.522 INFO:teuthology.orchestra.run.smithi179.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-10-19T12:41:49.522 INFO:teuthology.orchestra.run.smithi179.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2023-10-19T12:41:49.522 INFO:teuthology.orchestra.run.smithi179.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-10-19T12:41:49.522 INFO:teuthology.orchestra.run.smithi179.stdout: zlib-devel x86_64 1.2.11-40.el9 appstream 45 k 2023-10-19T12:41:49.523 INFO:teuthology.orchestra.run.smithi179.stdout:Installing weak dependencies: 2023-10-19T12:41:49.523 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-10-19T12:41:49.523 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-10-19T12:41:49.523 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2023-10-19T12:41:49.523 INFO:teuthology.orchestra.run.smithi179.stdout:============================================================================================ 2023-10-19T12:41:49.524 INFO:teuthology.orchestra.run.smithi179.stdout:Install 133 Packages 2023-10-19T12:41:49.524 INFO:teuthology.orchestra.run.smithi179.stdout:Upgrade 10 Packages 2023-10-19T12:41:49.524 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-10-19T12:41:49.525 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 200 M 2023-10-19T12:41:49.525 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2023-10-19T12:41:50.097 INFO:teuthology.orchestra.run.smithi179.stdout:(1/143): ceph-18.0.0-6786.gf7420d7f.el9.x86_64. 40 kB/s | 6.5 kB 00:00 2023-10-19T12:41:50.322 INFO:teuthology.orchestra.run.smithi179.stdout:(2/143): ceph-fuse-18.0.0-6786.gf7420d7f.el9.x8 3.7 MB/s | 857 kB 00:00 2023-10-19T12:41:50.389 INFO:teuthology.orchestra.run.smithi179.stdout:(3/143): ceph-immutable-object-cache-18.0.0-678 2.1 MB/s | 142 kB 00:00 2023-10-19T12:41:50.697 INFO:teuthology.orchestra.run.smithi179.stdout:(4/143): ceph-base-18.0.0-6786.gf7420d7f.el9.x8 6.8 MB/s | 5.2 MB 00:00 2023-10-19T12:41:50.736 INFO:teuthology.orchestra.run.smithi057.stdout:CentOS Stream 9 - CRB 3.7 MB/s | 5.7 MB 00:01 2023-10-19T12:41:50.739 INFO:teuthology.orchestra.run.smithi179.stdout:(5/143): ceph-mds-18.0.0-6786.gf7420d7f.el9.x86 6.0 MB/s | 2.1 MB 00:00 2023-10-19T12:41:50.865 INFO:teuthology.orchestra.run.smithi179.stdout:(6/143): ceph-mgr-18.0.0-6786.gf7420d7f.el9.x86 8.8 MB/s | 1.5 MB 00:00 2023-10-19T12:41:51.348 INFO:teuthology.orchestra.run.smithi179.stdout:(7/143): ceph-mon-18.0.0-6786.gf7420d7f.el9.x86 7.2 MB/s | 4.4 MB 00:00 2023-10-19T12:41:51.923 INFO:teuthology.orchestra.run.smithi179.stdout:(8/143): ceph-common-18.0.0-6786.gf7420d7f.el9. 9.6 MB/s | 19 MB 00:01 2023-10-19T12:41:51.999 INFO:teuthology.orchestra.run.smithi179.stdout:(9/143): ceph-selinux-18.0.0-6786.gf7420d7f.el9 322 kB/s | 24 kB 00:00 2023-10-19T12:41:52.345 INFO:teuthology.orchestra.run.smithi057.stdout:CentOS Stream 9 - Extras packages 31 kB/s | 14 kB 00:00 2023-10-19T12:41:52.507 INFO:teuthology.orchestra.run.smithi179.stdout:(10/143): ceph-radosgw-18.0.0-6786.gf7420d7f.el 7.4 MB/s | 8.6 MB 00:01 2023-10-19T12:41:52.683 INFO:teuthology.orchestra.run.smithi179.stdout:(11/143): ceph-osd-18.0.0-6786.gf7420d7f.el9.x8 9.7 MB/s | 18 MB 00:01 2023-10-19T12:41:52.708 INFO:teuthology.orchestra.run.smithi179.stdout:(12/143): cephfs-mirror-18.0.0-6786.gf7420d7f.e 1.1 MB/s | 218 kB 00:00 2023-10-19T12:41:52.750 INFO:teuthology.orchestra.run.smithi179.stdout:(13/143): libcephfs-devel-18.0.0-6786.gf7420d7f 468 kB/s | 31 kB 00:00 2023-10-19T12:41:52.801 INFO:teuthology.orchestra.run.smithi179.stdout:(14/143): libcephfs2-18.0.0-6786.gf7420d7f.el9. 7.6 MB/s | 716 kB 00:00 2023-10-19T12:41:52.826 INFO:teuthology.orchestra.run.smithi179.stdout:(15/143): libcephsqlite-18.0.0-6786.gf7420d7f.e 2.1 MB/s | 164 kB 00:00 2023-10-19T12:41:52.859 INFO:teuthology.orchestra.run.smithi179.stdout:(16/143): librados-devel-18.0.0-6786.gf7420d7f. 2.1 MB/s | 127 kB 00:00 2023-10-19T12:41:52.902 INFO:teuthology.orchestra.run.smithi179.stdout:(17/143): libradosstriper1-18.0.0-6786.gf7420d7 6.1 MB/s | 472 kB 00:00 2023-10-19T12:41:52.976 INFO:teuthology.orchestra.run.smithi179.stdout:(18/143): python3-ceph-argparse-18.0.0-6786.gf7 601 kB/s | 45 kB 00:00 2023-10-19T12:41:53.051 INFO:teuthology.orchestra.run.smithi179.stdout:(19/143): python3-ceph-common-18.0.0-6786.gf742 1.7 MB/s | 127 kB 00:00 2023-10-19T12:41:53.127 INFO:teuthology.orchestra.run.smithi179.stdout:(20/143): python3-cephfs-18.0.0-6786.gf7420d7f. 2.1 MB/s | 162 kB 00:00 2023-10-19T12:41:53.210 INFO:teuthology.orchestra.run.smithi179.stdout:(21/143): python3-rados-18.0.0-6786.gf7420d7f.e 3.8 MB/s | 321 kB 00:00 2023-10-19T12:41:53.294 INFO:teuthology.orchestra.run.smithi179.stdout:(22/143): python3-rbd-18.0.0-6786.gf7420d7f.el9 3.5 MB/s | 297 kB 00:00 2023-10-19T12:41:53.365 INFO:teuthology.orchestra.run.smithi057.stdout:Extra Packages for Enterprise Linux 21 MB/s | 19 MB 00:00 2023-10-19T12:41:53.394 INFO:teuthology.orchestra.run.smithi179.stdout:(23/143): librgw2-18.0.0-6786.gf7420d7f.el9.x86 8.9 MB/s | 4.8 MB 00:00 2023-10-19T12:41:53.419 INFO:teuthology.orchestra.run.smithi179.stdout:(24/143): python3-rgw-18.0.0-6786.gf7420d7f.el9 796 kB/s | 99 kB 00:00 2023-10-19T12:41:53.453 INFO:teuthology.orchestra.run.smithi179.stdout:(25/143): rbd-fuse-18.0.0-6786.gf7420d7f.el9.x8 1.4 MB/s | 86 kB 00:00 2023-10-19T12:41:53.528 INFO:teuthology.orchestra.run.smithi179.stdout:(26/143): rbd-nbd-18.0.0-6786.gf7420d7f.el9.x86 2.2 MB/s | 169 kB 00:00 2023-10-19T12:41:53.603 INFO:teuthology.orchestra.run.smithi179.stdout:(27/143): ceph-grafana-dashboards-18.0.0-6786.g 309 kB/s | 23 kB 00:00 2023-10-19T12:41:53.687 INFO:teuthology.orchestra.run.smithi179.stdout:(28/143): ceph-mgr-cephadm-18.0.0-6786.gf7420d7 1.6 MB/s | 132 kB 00:00 2023-10-19T12:41:53.754 INFO:teuthology.orchestra.run.smithi179.stdout:(29/143): rbd-mirror-18.0.0-6786.gf7420d7f.el9. 9.0 MB/s | 3.0 MB 00:00 2023-10-19T12:41:53.946 INFO:teuthology.orchestra.run.smithi179.stdout:(30/143): ceph-mgr-dashboard-18.0.0-6786.gf7420 6.8 MB/s | 1.8 MB 00:00 2023-10-19T12:41:54.038 INFO:teuthology.orchestra.run.smithi179.stdout:(31/143): ceph-mgr-modules-core-18.0.0-6786.gf7 2.6 MB/s | 242 kB 00:00 2023-10-19T12:41:54.121 INFO:teuthology.orchestra.run.smithi179.stdout:(32/143): ceph-mgr-rook-18.0.0-6786.gf7420d7f.e 571 kB/s | 47 kB 00:00 2023-10-19T12:41:54.197 INFO:teuthology.orchestra.run.smithi179.stdout:(33/143): ceph-prometheus-alerts-18.0.0-6786.gf 201 kB/s | 15 kB 00:00 2023-10-19T12:41:54.289 INFO:teuthology.orchestra.run.smithi179.stdout:(34/143): ceph-volume-18.0.0-6786.gf7420d7f.el9 2.8 MB/s | 262 kB 00:00 2023-10-19T12:41:54.381 INFO:teuthology.orchestra.run.smithi179.stdout:(35/143): cephadm-18.0.0-6786.gf7420d7f.el9.noa 2.9 MB/s | 270 kB 00:00 2023-10-19T12:41:54.464 INFO:teuthology.orchestra.run.smithi179.stdout:(36/143): cephfs-top-18.0.0-6786.gf7420d7f.el9. 219 kB/s | 18 kB 00:00 2023-10-19T12:41:54.614 INFO:teuthology.orchestra.run.smithi179.stdout:(37/143): bison-3.7.4-5.el9.x86_64.rpm 6.0 MB/s | 920 kB 00:00 2023-10-19T12:41:54.648 INFO:teuthology.orchestra.run.smithi179.stdout:(38/143): boost-program-options-1.75.0-8.el9.x8 3.1 MB/s | 106 kB 00:00 2023-10-19T12:41:54.698 INFO:teuthology.orchestra.run.smithi179.stdout:(39/143): flex-2.6.4-9.el9.x86_64.rpm 6.0 MB/s | 307 kB 00:00 2023-10-19T12:41:54.790 INFO:teuthology.orchestra.run.smithi179.stdout:(40/143): ceph-mgr-diskprediction-local-18.0.0- 7.1 MB/s | 7.4 MB 00:01 2023-10-19T12:41:54.807 INFO:teuthology.orchestra.run.smithi179.stdout:(41/143): flexiblas-3.0.4-8.el9.x86_64.rpm 290 kB/s | 32 kB 00:00 2023-10-19T12:41:55.208 INFO:teuthology.orchestra.run.smithi179.stdout:(42/143): ceph-test-18.0.0-6786.gf7420d7f.el9.x 13 MB/s | 41 MB 00:03 2023-10-19T12:41:55.258 INFO:teuthology.orchestra.run.smithi179.stdout:(43/143): flexiblas-netlib-3.0.4-8.el9.x86_64.r 6.4 MB/s | 3.0 MB 00:00 2023-10-19T12:41:55.275 INFO:teuthology.orchestra.run.smithi179.stdout:(44/143): flexiblas-openblas-openmp-3.0.4-8.el9 37 kB/s | 17 kB 00:00 2023-10-19T12:41:55.292 INFO:teuthology.orchestra.run.smithi179.stdout:(45/143): libaio-devel-0.3.111-13.el9.x86_64.rp 124 kB/s | 10 kB 00:00 2023-10-19T12:41:55.317 INFO:teuthology.orchestra.run.smithi179.stdout:(46/143): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.8 MB/s | 159 kB 00:00 2023-10-19T12:41:55.351 INFO:teuthology.orchestra.run.smithi179.stdout:(47/143): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 661 kB 00:00 2023-10-19T12:41:55.376 INFO:teuthology.orchestra.run.smithi179.stdout:(48/143): libtool-2.4.6-45.el9.x86_64.rpm 6.9 MB/s | 578 kB 00:00 2023-10-19T12:41:55.401 INFO:teuthology.orchestra.run.smithi179.stdout:(49/143): libzstd-devel-1.5.1-2.el9.x86_64.rpm 562 kB/s | 46 kB 00:00 2023-10-19T12:41:55.435 INFO:teuthology.orchestra.run.smithi179.stdout:(50/143): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.7 MB/s | 282 kB 00:00 2023-10-19T12:41:55.460 INFO:teuthology.orchestra.run.smithi179.stdout:(51/143): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.5 MB/s | 217 kB 00:00 2023-10-19T12:41:55.485 INFO:teuthology.orchestra.run.smithi179.stdout:(52/143): openblas-0.3.21-2.el9.x86_64.rpm 414 kB/s | 34 kB 00:00 2023-10-19T12:41:55.561 INFO:teuthology.orchestra.run.smithi179.stdout:(53/143): postgresql-13.11-1.el9.x86_64.rpm 15 MB/s | 1.5 MB 00:00 2023-10-19T12:41:55.636 INFO:teuthology.orchestra.run.smithi179.stdout:(54/143): openblas-openmp-0.3.21-2.el9.x86_64.r 27 MB/s | 4.7 MB 00:00 2023-10-19T12:41:55.669 INFO:teuthology.orchestra.run.smithi179.stdout:(55/143): postgresql-contrib-13.11-1.el9.x86_64 4.4 MB/s | 811 kB 00:00 2023-10-19T12:41:55.695 INFO:teuthology.orchestra.run.smithi179.stdout:(56/143): postgresql-private-libs-13.11-1.el9.x 1.0 MB/s | 134 kB 00:00 2023-10-19T12:41:55.970 INFO:teuthology.orchestra.run.smithi179.stdout:(57/143): postgresql-server-13.11-1.el9.x86_64. 19 MB/s | 5.7 MB 00:00 2023-10-19T12:41:56.045 INFO:teuthology.orchestra.run.smithi179.stdout:(58/143): python3-babel-2.9.1-2.el9.noarch.rpm 15 MB/s | 5.8 MB 00:00 2023-10-19T12:41:56.070 INFO:teuthology.orchestra.run.smithi179.stdout:(59/143): python3-devel-3.9.17-2.el9.x86_64.rpm 550 kB/s | 205 kB 00:00 2023-10-19T12:41:56.096 INFO:teuthology.orchestra.run.smithi179.stdout:(60/143): python3-jinja2-2.11.3-4.el9.noarch.rp 1.8 MB/s | 228 kB 00:00 2023-10-19T12:41:56.121 INFO:teuthology.orchestra.run.smithi179.stdout:(61/143): python3-jsonpatch-1.21-16.el9.noarch. 490 kB/s | 24 kB 00:00 2023-10-19T12:41:56.146 INFO:teuthology.orchestra.run.smithi179.stdout:(62/143): python3-jsonpointer-2.0-4.el9.noarch. 241 kB/s | 18 kB 00:00 2023-10-19T12:41:56.176 INFO:teuthology.orchestra.run.smithi179.stdout:(63/143): python3-mako-1.1.4-6.el9.noarch.rpm 1.8 MB/s | 144 kB 00:00 2023-10-19T12:41:56.192 INFO:teuthology.orchestra.run.smithi179.stdout:(64/143): python3-markupsafe-1.1.1-12.el9.x86_6 688 kB/s | 32 kB 00:00 2023-10-19T12:41:56.318 INFO:teuthology.orchestra.run.smithi179.stdout:(65/143): python3-numpy-1.20.1-5.el9.x86_64.rpm 28 MB/s | 4.8 MB 00:00 2023-10-19T12:41:56.343 INFO:teuthology.orchestra.run.smithi179.stdout:(66/143): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.3 MB/s | 217 kB 00:00 2023-10-19T12:41:56.369 INFO:teuthology.orchestra.run.smithi179.stdout:(67/143): python3-oauthlib-3.1.1-5.el9.noarch.r 998 kB/s | 174 kB 00:00 2023-10-19T12:41:56.394 INFO:teuthology.orchestra.run.smithi179.stdout:(68/143): python3-prettytable-0.7.2-27.el9.noar 547 kB/s | 41 kB 00:00 2023-10-19T12:41:56.419 INFO:teuthology.orchestra.run.smithi179.stdout:(69/143): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 132 kB 00:00 2023-10-19T12:41:56.445 INFO:teuthology.orchestra.run.smithi179.stdout:(70/143): python3-pyasn1-modules-0.4.8-6.el9.no 2.7 MB/s | 211 kB 00:00 2023-10-19T12:41:56.470 INFO:teuthology.orchestra.run.smithi179.stdout:(71/143): python3-requests-oauthlib-1.3.0-12.el 572 kB/s | 43 kB 00:00 2023-10-19T12:41:56.529 INFO:teuthology.orchestra.run.smithi179.stdout:(72/143): python3-toml-0.10.2-6.el9.noarch.rpm 463 kB/s | 39 kB 00:00 2023-10-19T12:41:56.671 INFO:teuthology.orchestra.run.smithi179.stdout:(73/143): socat-1.7.4.1-5.el9.x86_64.rpm 1.5 MB/s | 300 kB 00:00 2023-10-19T12:41:56.821 INFO:teuthology.orchestra.run.smithi179.stdout:(74/143): python3-scipy-1.6.2-8.el9.x86_64.rpm 42 MB/s | 16 MB 00:00 2023-10-19T12:41:56.846 INFO:teuthology.orchestra.run.smithi179.stdout:(75/143): uuid-1.6.2-55.el9.x86_64.rpm 179 kB/s | 56 kB 00:00 2023-10-19T12:41:56.871 INFO:teuthology.orchestra.run.smithi179.stdout:(76/143): xmlstarlet-1.6.1-20.el9.x86_64.rpm 316 kB/s | 63 kB 00:00 2023-10-19T12:41:56.905 INFO:teuthology.orchestra.run.smithi179.stdout:(77/143): fuse-2.9.9-15.el9.x86_64.rpm 1.3 MB/s | 79 kB 00:00 2023-10-19T12:41:56.930 INFO:teuthology.orchestra.run.smithi179.stdout:(78/143): libgfortran-11.4.1-2.3.el9.x86_64.rpm 9.4 MB/s | 803 kB 00:00 2023-10-19T12:41:56.955 INFO:teuthology.orchestra.run.smithi179.stdout:(79/143): libquadmath-11.4.1-2.3.el9.x86_64.rpm 2.3 MB/s | 195 kB 00:00 2023-10-19T12:41:56.981 INFO:teuthology.orchestra.run.smithi179.stdout:(80/143): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2023-10-19T12:41:56.997 INFO:teuthology.orchestra.run.smithi179.stdout:(81/143): python3-ply-3.11-14.el9.noarch.rpm 1.5 MB/s | 103 kB 00:00 2023-10-19T12:41:57.022 INFO:teuthology.orchestra.run.smithi179.stdout:(82/143): python3-pycparser-2.20-6.el9.noarch.r 1.8 MB/s | 124 kB 00:00 2023-10-19T12:41:57.048 INFO:teuthology.orchestra.run.smithi179.stdout:(83/143): python3-requests-2.25.1-7.el9.noarch. 2.2 MB/s | 113 kB 00:00 2023-10-19T12:41:57.073 INFO:teuthology.orchestra.run.smithi179.stdout:(84/143): python3-urllib3-1.26.5-3.el9.noarch.r 2.5 MB/s | 188 kB 00:00 2023-10-19T12:41:57.173 INFO:teuthology.orchestra.run.smithi179.stdout:(85/143): elfutils-libelf-devel-0.189-3.el9.x86 224 kB/s | 22 kB 00:00 2023-10-19T12:41:57.282 INFO:teuthology.orchestra.run.smithi179.stdout:(86/143): jq-1.6-15.el9.x86_64.rpm 1.7 MB/s | 187 kB 00:00 2023-10-19T12:41:57.315 INFO:teuthology.orchestra.run.smithi179.stdout:(87/143): automake-1.16.2-8.el9.noarch.rpm 2.8 MB/s | 689 kB 00:00 2023-10-19T12:41:57.341 INFO:teuthology.orchestra.run.smithi179.stdout:(88/143): librabbitmq-0.11.0-7.el9.x86_64.rpm 774 kB/s | 45 kB 00:00 2023-10-19T12:41:57.383 INFO:teuthology.orchestra.run.smithi179.stdout:(89/143): python3-cryptography-36.0.1-4.el9.x86 3.4 MB/s | 1.2 MB 00:00 2023-10-19T12:41:57.449 INFO:teuthology.orchestra.run.smithi179.stdout:(90/143): libuuid-devel-2.37.4-15.el9.x86_64.rp 191 kB/s | 26 kB 00:00 2023-10-19T12:41:57.500 INFO:teuthology.orchestra.run.smithi179.stdout:(91/143): python3-pytz-2021.1-5.el9.noarch.rpm 437 kB/s | 51 kB 00:00 2023-10-19T12:41:57.567 INFO:teuthology.orchestra.run.smithi179.stdout:(92/143): openssl-devel-3.0.7-24.el9.x86_64.rpm 18 MB/s | 4.1 MB 00:00 2023-10-19T12:41:57.592 INFO:teuthology.orchestra.run.smithi179.stdout:(93/143): zlib-devel-1.2.11-40.el9.x86_64.rpm 490 kB/s | 45 kB 00:00 2023-10-19T12:41:57.617 INFO:teuthology.orchestra.run.smithi179.stdout:(94/143): xfsprogs-devel-5.19.0-4.el9.x86_64.rp 515 kB/s | 86 kB 00:00 2023-10-19T12:41:57.784 INFO:teuthology.orchestra.run.smithi179.stdout:(95/143): fmt-8.1.1-5.el9.x86_64.rpm 510 kB/s | 111 kB 00:00 2023-10-19T12:41:57.843 INFO:teuthology.orchestra.run.smithi179.stdout:(96/143): gperftools-libs-2.9.1-2.el9.x86_64.rp 1.2 MB/s | 309 kB 00:00 2023-10-19T12:41:57.868 INFO:teuthology.orchestra.run.smithi179.stdout:(97/143): libarrow-doc-9.0.0-9.el9.noarch.rpm 291 kB/s | 24 kB 00:00 2023-10-19T12:41:57.910 INFO:teuthology.orchestra.run.smithi179.stdout:(98/143): liboath-2.6.7-2.el9.x86_64.rpm 728 kB/s | 49 kB 00:00 2023-10-19T12:41:57.956 INFO:teuthology.orchestra.run.smithi057.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2023-10-19T12:41:57.956 INFO:teuthology.orchestra.run.smithi179.stdout:(99/143): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2023-10-19T12:41:58.069 INFO:teuthology.orchestra.run.smithi179.stdout:(100/143): libarrow-9.0.0-9.el9.x86_64.rpm 9.8 MB/s | 4.4 MB 00:00 2023-10-19T12:41:58.094 INFO:teuthology.orchestra.run.smithi179.stdout:(101/143): python3-asyncssh-2.13.2-3.el9.noarch 3.4 MB/s | 548 kB 00:00 2023-10-19T12:41:58.128 INFO:teuthology.orchestra.run.smithi179.stdout:(102/143): parquet-libs-9.0.0-9.el9.x86_64.rpm 3.8 MB/s | 837 kB 00:00 2023-10-19T12:41:58.145 INFO:teuthology.orchestra.run.smithi179.stdout:(103/143): python3-bcrypt-3.2.2-1.el9.x86_64.rp 575 kB/s | 43 kB 00:00 2023-10-19T12:41:58.162 INFO:teuthology.orchestra.run.smithi179.stdout:(104/143): python3-cachetools-4.2.4-1.el9.noarc 480 kB/s | 32 kB 00:00 2023-10-19T12:41:58.178 INFO:teuthology.orchestra.run.smithi179.stdout:(105/143): python3-certifi-2021.10.8-2.el9.noar 298 kB/s | 15 kB 00:00 2023-10-19T12:41:58.221 INFO:teuthology.orchestra.run.smithi179.stdout:(106/143): python3-cheroot-8.6.0-4.el9.noarch.r 2.9 MB/s | 172 kB 00:00 2023-10-19T12:41:58.246 INFO:teuthology.orchestra.run.smithi179.stdout:(107/143): python3-google-auth-2.23.3-1.el9.noa 2.9 MB/s | 197 kB 00:00 2023-10-19T12:41:58.271 INFO:teuthology.orchestra.run.smithi179.stdout:(108/143): python3-cherrypy-18.6.1-2.el9.noarch 3.2 MB/s | 358 kB 00:00 2023-10-19T12:41:58.296 INFO:teuthology.orchestra.run.smithi179.stdout:(109/143): python3-jaraco-8.2.1-3.el9.noarch.rp 142 kB/s | 11 kB 00:00 2023-10-19T12:41:58.322 INFO:teuthology.orchestra.run.smithi179.stdout:(110/143): python3-jaraco-classes-3.2.1-5.el9.n 234 kB/s | 18 kB 00:00 2023-10-19T12:41:58.347 INFO:teuthology.orchestra.run.smithi179.stdout:(111/143): python3-jaraco-collections-3.0.0-8.e 307 kB/s | 23 kB 00:00 2023-10-19T12:41:58.372 INFO:teuthology.orchestra.run.smithi179.stdout:(112/143): python3-jaraco-functools-3.5.0-2.el9 257 kB/s | 19 kB 00:00 2023-10-19T12:41:58.397 INFO:teuthology.orchestra.run.smithi179.stdout:(113/143): python3-jaraco-text-3.2.0-6.el9.noar 260 kB/s | 20 kB 00:00 2023-10-19T12:41:58.423 INFO:teuthology.orchestra.run.smithi179.stdout:(114/143): python3-jwt+crypto-2.4.0-1.el9.noarc 120 kB/s | 9.0 kB 00:00 2023-10-19T12:41:58.448 INFO:teuthology.orchestra.run.smithi179.stdout:(115/143): python3-jwt-2.4.0-1.el9.noarch.rpm 542 kB/s | 41 kB 00:00 2023-10-19T12:41:58.482 INFO:teuthology.orchestra.run.smithi179.stdout:(116/143): python3-logutils-0.3.5-21.el9.noarch 790 kB/s | 46 kB 00:00 2023-10-19T12:41:58.507 INFO:teuthology.orchestra.run.smithi179.stdout:(117/143): python3-more-itertools-8.12.0-2.el9. 1.3 MB/s | 79 kB 00:00 2023-10-19T12:41:58.540 INFO:teuthology.orchestra.run.smithi179.stdout:(118/143): python3-kubernetes-26.1.0-2.el9.noar 7.2 MB/s | 1.0 MB 00:00 2023-10-19T12:41:58.566 INFO:teuthology.orchestra.run.smithi179.stdout:(119/143): python3-natsort-7.1.1-5.el9.noarch.r 688 kB/s | 58 kB 00:00 2023-10-19T12:41:58.594 INFO:teuthology.orchestra.run.smithi179.stdout:(120/143): python3-pecan-1.4.2-3.el9.noarch.rpm 3.1 MB/s | 272 kB 00:00 2023-10-19T12:41:58.611 INFO:teuthology.orchestra.run.smithi179.stdout:(121/143): python3-portend-3.1.0-2.el9.noarch.r 234 kB/s | 16 kB 00:00 2023-10-19T12:41:58.636 INFO:teuthology.orchestra.run.smithi179.stdout:(122/143): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.3 MB/s | 90 kB 00:00 2023-10-19T12:41:58.653 INFO:teuthology.orchestra.run.smithi179.stdout:(123/143): python3-repoze-lru-0.7-16.el9.noarch 522 kB/s | 31 kB 00:00 2023-10-19T12:41:58.678 INFO:teuthology.orchestra.run.smithi179.stdout:(124/143): python3-routes-2.5.1-5.el9.noarch.rp 2.7 MB/s | 188 kB 00:00 2023-10-19T12:41:58.704 INFO:teuthology.orchestra.run.smithi179.stdout:(125/143): python3-rsa-4.9-2.el9.noarch.rpm 879 kB/s | 59 kB 00:00 2023-10-19T12:41:58.721 INFO:teuthology.orchestra.run.smithi179.stdout:(126/143): python3-tempora-5.0.0-2.el9.noarch.r 533 kB/s | 36 kB 00:00 2023-10-19T12:41:58.746 INFO:teuthology.orchestra.run.smithi179.stdout:(127/143): python3-typing-extensions-4.4.0-2.el 760 kB/s | 51 kB 00:00 2023-10-19T12:41:58.779 INFO:teuthology.orchestra.run.smithi179.stdout:(128/143): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2023-10-19T12:41:58.805 INFO:teuthology.orchestra.run.smithi179.stdout:(129/143): python3-websocket-client-1.2.3-2.el9 1.0 MB/s | 90 kB 00:00 2023-10-19T12:41:58.830 INFO:teuthology.orchestra.run.smithi179.stdout:(130/143): python3-werkzeug-2.0.3-3.el9.1.noarc 5.0 MB/s | 427 kB 00:00 2023-10-19T12:41:58.855 INFO:teuthology.orchestra.run.smithi179.stdout:(131/143): python3-zc-lockfile-2.0-10.el9.noarc 265 kB/s | 20 kB 00:00 2023-10-19T12:41:58.881 INFO:teuthology.orchestra.run.smithi179.stdout:(132/143): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2023-10-19T12:41:58.948 INFO:teuthology.orchestra.run.smithi179.stdout:(133/143): thrift-0.15.0-2.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2023-10-19T12:41:58.990 INFO:teuthology.orchestra.run.smithi179.stdout:(134/143): libacl-devel-2.3.1-4.el9.x86_64.rpm 1.6 MB/s | 68 kB 00:00 2023-10-19T12:41:59.082 INFO:teuthology.orchestra.run.smithi179.stdout:(135/143): NetworkManager-1.45.4-1.el9.x86_64.r 24 MB/s | 2.2 MB 00:00 2023-10-19T12:41:59.157 INFO:teuthology.orchestra.run.smithi179.stdout:(136/143): NetworkManager-libnm-1.45.4-1.el9.x8 24 MB/s | 1.8 MB 00:00 2023-10-19T12:41:59.190 INFO:teuthology.orchestra.run.smithi179.stdout:(137/143): NetworkManager-team-1.45.4-1.el9.x86 1.2 MB/s | 41 kB 00:00 2023-10-19T12:41:59.249 INFO:teuthology.orchestra.run.smithi179.stdout:(138/143): librados2-18.0.0-6786.gf7420d7f.el9. 8.4 MB/s | 3.3 MB 00:00 2023-10-19T12:41:59.274 INFO:teuthology.orchestra.run.smithi179.stdout:(139/143): NetworkManager-tui-1.45.4-1.el9.x86_ 2.9 MB/s | 244 kB 00:00 2023-10-19T12:41:59.299 INFO:teuthology.orchestra.run.smithi179.stdout:(140/143): acl-2.3.1-4.el9.x86_64.rpm 1.4 MB/s | 69 kB 00:00 2023-10-19T12:41:59.325 INFO:teuthology.orchestra.run.smithi179.stdout:(141/143): libacl-2.3.1-4.el9.x86_64.rpm 438 kB/s | 22 kB 00:00 2023-10-19T12:41:59.350 INFO:teuthology.orchestra.run.smithi179.stdout:(142/143): xfsdump-3.1.12-4.el9.x86_64.rpm 6.3 MB/s | 321 kB 00:00 2023-10-19T12:41:59.400 INFO:teuthology.orchestra.run.smithi179.stdout:(143/143): librbd1-18.0.0-6786.gf7420d7f.el9.x8 5.8 MB/s | 3.0 MB 00:00 2023-10-19T12:41:59.404 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2023-10-19T12:41:59.404 INFO:teuthology.orchestra.run.smithi179.stdout:Total 20 MB/s | 200 MB 00:09 2023-10-19T12:41:59.547 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2023-10-19T12:41:59.673 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2023-10-19T12:41:59.674 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2023-10-19T12:42:00.338 INFO:teuthology.orchestra.run.smithi057.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2023-10-19T12:42:00.339 INFO:teuthology.orchestra.run.smithi057.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2023-10-19T12:42:00.350 INFO:teuthology.orchestra.run.smithi057.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2023-10-19T12:42:00.351 INFO:teuthology.orchestra.run.smithi057.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2023-10-19T12:42:00.352 INFO:teuthology.orchestra.run.smithi057.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2023-10-19T12:42:00.353 INFO:teuthology.orchestra.run.smithi057.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2023-10-19T12:42:00.355 INFO:teuthology.orchestra.run.smithi057.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2023-10-19T12:42:00.358 INFO:teuthology.orchestra.run.smithi057.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2023-10-19T12:42:00.358 INFO:teuthology.orchestra.run.smithi057.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2023-10-19T12:42:00.435 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2023-10-19T12:42:00.446 INFO:teuthology.orchestra.run.smithi057.stdout:============================================================================================ 2023-10-19T12:42:00.446 INFO:teuthology.orchestra.run.smithi057.stdout: Package Arch Version Repository Size 2023-10-19T12:42:00.447 INFO:teuthology.orchestra.run.smithi057.stdout:============================================================================================ 2023-10-19T12:42:00.447 INFO:teuthology.orchestra.run.smithi057.stdout:Installing: 2023-10-19T12:42:00.447 INFO:teuthology.orchestra.run.smithi057.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2023-10-19T12:42:00.447 INFO:teuthology.orchestra.run.smithi057.stdout: ceph x86_64 2:18.0.0-6786.gf7420d7f.el9 ceph 6.5 k 2023-10-19T12:42:00.447 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-base x86_64 2:18.0.0-6786.gf7420d7f.el9 ceph 5.2 M 2023-10-19T12:42:00.448 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-fuse x86_64 2:18.0.0-6786.gf7420d7f.el9 ceph 857 k 2023-10-19T12:42:00.449 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-6786.gf7420d7f.el9 ceph 142 k 2023-10-19T12:42:00.449 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr x86_64 2:18.0.0-6786.gf7420d7f.el9 ceph 1.5 M 2023-10-19T12:42:00.449 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-cephadm noarch 2:18.0.0-6786.gf7420d7f.el9 ceph-noarch 132 k 2023-10-19T12:42:00.450 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-dashboard noarch 2:18.0.0-6786.gf7420d7f.el9 ceph-noarch 1.8 M 2023-10-19T12:42:00.450 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-6786.gf7420d7f.el9 ceph-noarch 7.4 M 2023-10-19T12:42:00.450 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-rook noarch 2:18.0.0-6786.gf7420d7f.el9 ceph-noarch 47 k 2023-10-19T12:42:00.450 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-radosgw x86_64 2:18.0.0-6786.gf7420d7f.el9 ceph 8.6 M 2023-10-19T12:42:00.451 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-test x86_64 2:18.0.0-6786.gf7420d7f.el9 ceph 41 M 2023-10-19T12:42:00.451 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-volume noarch 2:18.0.0-6786.gf7420d7f.el9 ceph-noarch 262 k 2023-10-19T12:42:00.451 INFO:teuthology.orchestra.run.smithi057.stdout: cephadm noarch 2:18.0.0-6786.gf7420d7f.el9 ceph-noarch 270 k 2023-10-19T12:42:00.451 INFO:teuthology.orchestra.run.smithi057.stdout: cephfs-mirror x86_64 2:18.0.0-6786.gf7420d7f.el9 ceph 218 k 2023-10-19T12:42:00.451 INFO:teuthology.orchestra.run.smithi057.stdout: cephfs-top noarch 2:18.0.0-6786.gf7420d7f.el9 ceph-noarch 18 k 2023-10-19T12:42:00.451 INFO:teuthology.orchestra.run.smithi057.stdout: elfutils-libelf-devel x86_64 0.189-3.el9 appstream 22 k 2023-10-19T12:42:00.452 INFO:teuthology.orchestra.run.smithi057.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2023-10-19T12:42:00.452 INFO:teuthology.orchestra.run.smithi057.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2023-10-19T12:42:00.452 INFO:teuthology.orchestra.run.smithi057.stdout: libcephfs-devel x86_64 2:18.0.0-6786.gf7420d7f.el9 ceph 31 k 2023-10-19T12:42:00.452 INFO:teuthology.orchestra.run.smithi057.stdout: libcephfs2 x86_64 2:18.0.0-6786.gf7420d7f.el9 ceph 716 k 2023-10-19T12:42:00.452 INFO:teuthology.orchestra.run.smithi057.stdout: librados-devel x86_64 2:18.0.0-6786.gf7420d7f.el9 ceph 127 k 2023-10-19T12:42:00.453 INFO:teuthology.orchestra.run.smithi057.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2023-10-19T12:42:00.453 INFO:teuthology.orchestra.run.smithi057.stdout: libuuid-devel x86_64 2.37.4-15.el9 appstream 26 k 2023-10-19T12:42:00.453 INFO:teuthology.orchestra.run.smithi057.stdout: openssl-devel x86_64 1:3.0.7-24.el9 appstream 4.1 M 2023-10-19T12:42:00.453 INFO:teuthology.orchestra.run.smithi057.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2023-10-19T12:42:00.453 INFO:teuthology.orchestra.run.smithi057.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2023-10-19T12:42:00.454 INFO:teuthology.orchestra.run.smithi057.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2023-10-19T12:42:00.454 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cephfs x86_64 2:18.0.0-6786.gf7420d7f.el9 ceph 162 k 2023-10-19T12:42:00.454 INFO:teuthology.orchestra.run.smithi057.stdout: python3-rados x86_64 2:18.0.0-6786.gf7420d7f.el9 ceph 321 k 2023-10-19T12:42:00.454 INFO:teuthology.orchestra.run.smithi057.stdout: python3-rbd x86_64 2:18.0.0-6786.gf7420d7f.el9 ceph 297 k 2023-10-19T12:42:00.454 INFO:teuthology.orchestra.run.smithi057.stdout: python3-rgw x86_64 2:18.0.0-6786.gf7420d7f.el9 ceph 99 k 2023-10-19T12:42:00.455 INFO:teuthology.orchestra.run.smithi057.stdout: rbd-fuse x86_64 2:18.0.0-6786.gf7420d7f.el9 ceph 86 k 2023-10-19T12:42:00.455 INFO:teuthology.orchestra.run.smithi057.stdout: rbd-mirror x86_64 2:18.0.0-6786.gf7420d7f.el9 ceph 3.0 M 2023-10-19T12:42:00.455 INFO:teuthology.orchestra.run.smithi057.stdout: rbd-nbd x86_64 2:18.0.0-6786.gf7420d7f.el9 ceph 169 k 2023-10-19T12:42:00.455 INFO:teuthology.orchestra.run.smithi057.stdout: xfsprogs-devel x86_64 5.19.0-4.el9 appstream 86 k 2023-10-19T12:42:00.455 INFO:teuthology.orchestra.run.smithi057.stdout:Upgrading: 2023-10-19T12:42:00.456 INFO:teuthology.orchestra.run.smithi057.stdout: NetworkManager x86_64 1:1.45.4-1.el9 CentOS-BaseOS 2.2 M 2023-10-19T12:42:00.456 INFO:teuthology.orchestra.run.smithi057.stdout: NetworkManager-libnm x86_64 1:1.45.4-1.el9 CentOS-BaseOS 1.8 M 2023-10-19T12:42:00.456 INFO:teuthology.orchestra.run.smithi057.stdout: NetworkManager-team x86_64 1:1.45.4-1.el9 CentOS-BaseOS 41 k 2023-10-19T12:42:00.456 INFO:teuthology.orchestra.run.smithi057.stdout: NetworkManager-tui x86_64 1:1.45.4-1.el9 CentOS-BaseOS 244 k 2023-10-19T12:42:00.456 INFO:teuthology.orchestra.run.smithi057.stdout: acl x86_64 2.3.1-4.el9 CentOS-BaseOS 69 k 2023-10-19T12:42:00.457 INFO:teuthology.orchestra.run.smithi057.stdout: libacl x86_64 2.3.1-4.el9 CentOS-BaseOS 22 k 2023-10-19T12:42:00.457 INFO:teuthology.orchestra.run.smithi057.stdout: libacl-devel x86_64 2.3.1-4.el9 CentOS-AppStream 68 k 2023-10-19T12:42:00.457 INFO:teuthology.orchestra.run.smithi057.stdout: librados2 x86_64 2:18.0.0-6786.gf7420d7f.el9 ceph 3.3 M 2023-10-19T12:42:00.457 INFO:teuthology.orchestra.run.smithi057.stdout: librbd1 x86_64 2:18.0.0-6786.gf7420d7f.el9 ceph 3.0 M 2023-10-19T12:42:00.457 INFO:teuthology.orchestra.run.smithi057.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2023-10-19T12:42:00.458 INFO:teuthology.orchestra.run.smithi057.stdout:Installing dependencies: 2023-10-19T12:42:00.458 INFO:teuthology.orchestra.run.smithi057.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2023-10-19T12:42:00.458 INFO:teuthology.orchestra.run.smithi057.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-10-19T12:42:00.458 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-common x86_64 2:18.0.0-6786.gf7420d7f.el9 ceph 19 M 2023-10-19T12:42:00.458 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-grafana-dashboards noarch 2:18.0.0-6786.gf7420d7f.el9 ceph-noarch 23 k 2023-10-19T12:42:00.459 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mds x86_64 2:18.0.0-6786.gf7420d7f.el9 ceph 2.1 M 2023-10-19T12:42:00.459 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-modules-core noarch 2:18.0.0-6786.gf7420d7f.el9 ceph-noarch 242 k 2023-10-19T12:42:00.459 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mon x86_64 2:18.0.0-6786.gf7420d7f.el9 ceph 4.4 M 2023-10-19T12:42:00.459 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-osd x86_64 2:18.0.0-6786.gf7420d7f.el9 ceph 18 M 2023-10-19T12:42:00.459 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-prometheus-alerts noarch 2:18.0.0-6786.gf7420d7f.el9 ceph-noarch 15 k 2023-10-19T12:42:00.460 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-selinux x86_64 2:18.0.0-6786.gf7420d7f.el9 ceph 24 k 2023-10-19T12:42:00.460 INFO:teuthology.orchestra.run.smithi057.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-10-19T12:42:00.460 INFO:teuthology.orchestra.run.smithi057.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-10-19T12:42:00.460 INFO:teuthology.orchestra.run.smithi057.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-10-19T12:42:00.460 INFO:teuthology.orchestra.run.smithi057.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2023-10-19T12:42:00.461 INFO:teuthology.orchestra.run.smithi057.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-10-19T12:42:00.461 INFO:teuthology.orchestra.run.smithi057.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-10-19T12:42:00.461 INFO:teuthology.orchestra.run.smithi057.stdout: jq x86_64 1.6-15.el9 appstream 187 k 2023-10-19T12:42:00.461 INFO:teuthology.orchestra.run.smithi057.stdout: libarrow x86_64 9.0.0-9.el9 epel 4.4 M 2023-10-19T12:42:00.461 INFO:teuthology.orchestra.run.smithi057.stdout: libarrow-doc noarch 9.0.0-9.el9 epel 24 k 2023-10-19T12:42:00.461 INFO:teuthology.orchestra.run.smithi057.stdout: libcephsqlite x86_64 2:18.0.0-6786.gf7420d7f.el9 ceph 164 k 2023-10-19T12:42:00.462 INFO:teuthology.orchestra.run.smithi057.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-10-19T12:42:00.462 INFO:teuthology.orchestra.run.smithi057.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-10-19T12:42:00.462 INFO:teuthology.orchestra.run.smithi057.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-10-19T12:42:00.462 INFO:teuthology.orchestra.run.smithi057.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-10-19T12:42:00.462 INFO:teuthology.orchestra.run.smithi057.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2023-10-19T12:42:00.463 INFO:teuthology.orchestra.run.smithi057.stdout: libradosstriper1 x86_64 2:18.0.0-6786.gf7420d7f.el9 ceph 472 k 2023-10-19T12:42:00.463 INFO:teuthology.orchestra.run.smithi057.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-10-19T12:42:00.463 INFO:teuthology.orchestra.run.smithi057.stdout: librgw2 x86_64 2:18.0.0-6786.gf7420d7f.el9 ceph 4.8 M 2023-10-19T12:42:00.463 INFO:teuthology.orchestra.run.smithi057.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-10-19T12:42:00.463 INFO:teuthology.orchestra.run.smithi057.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2023-10-19T12:42:00.464 INFO:teuthology.orchestra.run.smithi057.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-10-19T12:42:00.464 INFO:teuthology.orchestra.run.smithi057.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-10-19T12:42:00.464 INFO:teuthology.orchestra.run.smithi057.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-10-19T12:42:00.464 INFO:teuthology.orchestra.run.smithi057.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-10-19T12:42:00.464 INFO:teuthology.orchestra.run.smithi057.stdout: parquet-libs x86_64 9.0.0-9.el9 epel 837 k 2023-10-19T12:42:00.464 INFO:teuthology.orchestra.run.smithi057.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2023-10-19T12:42:00.465 INFO:teuthology.orchestra.run.smithi057.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-10-19T12:42:00.465 INFO:teuthology.orchestra.run.smithi057.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-10-19T12:42:00.465 INFO:teuthology.orchestra.run.smithi057.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-10-19T12:42:00.465 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-10-19T12:42:00.466 INFO:teuthology.orchestra.run.smithi057.stdout: python3-ceph-argparse x86_64 2:18.0.0-6786.gf7420d7f.el9 ceph 45 k 2023-10-19T12:42:00.466 INFO:teuthology.orchestra.run.smithi057.stdout: python3-ceph-common x86_64 2:18.0.0-6786.gf7420d7f.el9 ceph 127 k 2023-10-19T12:42:00.466 INFO:teuthology.orchestra.run.smithi057.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-10-19T12:42:00.466 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-10-19T12:42:00.466 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-10-19T12:42:00.467 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-10-19T12:42:00.467 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2023-10-19T12:42:00.467 INFO:teuthology.orchestra.run.smithi057.stdout: python3-devel x86_64 3.9.17-2.el9 CentOS-AppStream 205 k 2023-10-19T12:42:00.467 INFO:teuthology.orchestra.run.smithi057.stdout: python3-google-auth noarch 1:2.23.3-1.el9 epel 197 k 2023-10-19T12:42:00.467 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-10-19T12:42:00.468 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-10-19T12:42:00.468 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-10-19T12:42:00.468 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-10-19T12:42:00.468 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-10-19T12:42:00.468 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-10-19T12:42:00.468 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-10-19T12:42:00.469 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-10-19T12:42:00.469 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-10-19T12:42:00.469 INFO:teuthology.orchestra.run.smithi057.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-10-19T12:42:00.469 INFO:teuthology.orchestra.run.smithi057.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-10-19T12:42:00.469 INFO:teuthology.orchestra.run.smithi057.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-10-19T12:42:00.470 INFO:teuthology.orchestra.run.smithi057.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-10-19T12:42:00.470 INFO:teuthology.orchestra.run.smithi057.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-10-19T12:42:00.470 INFO:teuthology.orchestra.run.smithi057.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-10-19T12:42:00.470 INFO:teuthology.orchestra.run.smithi057.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-10-19T12:42:00.470 INFO:teuthology.orchestra.run.smithi057.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-10-19T12:42:00.471 INFO:teuthology.orchestra.run.smithi057.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-10-19T12:42:00.471 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-10-19T12:42:00.471 INFO:teuthology.orchestra.run.smithi057.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-10-19T12:42:00.471 INFO:teuthology.orchestra.run.smithi057.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-10-19T12:42:00.471 INFO:teuthology.orchestra.run.smithi057.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-10-19T12:42:00.471 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-10-19T12:42:00.472 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-10-19T12:42:00.472 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-10-19T12:42:00.472 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-10-19T12:42:00.472 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2023-10-19T12:42:00.472 INFO:teuthology.orchestra.run.smithi057.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-10-19T12:42:00.473 INFO:teuthology.orchestra.run.smithi057.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-10-19T12:42:00.473 INFO:teuthology.orchestra.run.smithi057.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-10-19T12:42:00.473 INFO:teuthology.orchestra.run.smithi057.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-10-19T12:42:00.473 INFO:teuthology.orchestra.run.smithi057.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-10-19T12:42:00.473 INFO:teuthology.orchestra.run.smithi057.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-10-19T12:42:00.474 INFO:teuthology.orchestra.run.smithi057.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-10-19T12:42:00.474 INFO:teuthology.orchestra.run.smithi057.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-10-19T12:42:00.474 INFO:teuthology.orchestra.run.smithi057.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-10-19T12:42:00.474 INFO:teuthology.orchestra.run.smithi057.stdout: python3-urllib3 noarch 1.26.5-3.el9 CentOS-BaseOS 188 k 2023-10-19T12:42:00.474 INFO:teuthology.orchestra.run.smithi057.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-10-19T12:42:00.475 INFO:teuthology.orchestra.run.smithi057.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-10-19T12:42:00.475 INFO:teuthology.orchestra.run.smithi057.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-10-19T12:42:00.475 INFO:teuthology.orchestra.run.smithi057.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-10-19T12:42:00.475 INFO:teuthology.orchestra.run.smithi057.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-10-19T12:42:00.476 INFO:teuthology.orchestra.run.smithi057.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-10-19T12:42:00.476 INFO:teuthology.orchestra.run.smithi057.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-10-19T12:42:00.476 INFO:teuthology.orchestra.run.smithi057.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2023-10-19T12:42:00.476 INFO:teuthology.orchestra.run.smithi057.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-10-19T12:42:00.476 INFO:teuthology.orchestra.run.smithi057.stdout: zlib-devel x86_64 1.2.11-40.el9 appstream 45 k 2023-10-19T12:42:00.476 INFO:teuthology.orchestra.run.smithi057.stdout:Installing weak dependencies: 2023-10-19T12:42:00.477 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-10-19T12:42:00.477 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:42:00.477 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction Summary 2023-10-19T12:42:00.477 INFO:teuthology.orchestra.run.smithi057.stdout:============================================================================================ 2023-10-19T12:42:00.477 INFO:teuthology.orchestra.run.smithi057.stdout:Install 133 Packages 2023-10-19T12:42:00.478 INFO:teuthology.orchestra.run.smithi057.stdout:Upgrade 10 Packages 2023-10-19T12:42:00.478 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:42:00.479 INFO:teuthology.orchestra.run.smithi057.stdout:Total download size: 200 M 2023-10-19T12:42:00.479 INFO:teuthology.orchestra.run.smithi057.stdout:Downloading Packages: 2023-10-19T12:42:01.165 INFO:teuthology.orchestra.run.smithi057.stdout:(1/143): ceph-18.0.0-6786.gf7420d7f.el9.x86_64. 27 kB/s | 6.5 kB 00:00 2023-10-19T12:42:01.470 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2023-10-19T12:42:01.471 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2023-10-19T12:42:01.598 INFO:teuthology.orchestra.run.smithi057.stdout:(2/143): ceph-base-18.0.0-6786.gf7420d7f.el9.x8 7.7 MB/s | 5.2 MB 00:00 2023-10-19T12:42:01.657 INFO:teuthology.orchestra.run.smithi057.stdout:(3/143): ceph-immutable-object-cache-18.0.0-678 2.4 MB/s | 142 kB 00:00 2023-10-19T12:42:01.874 INFO:teuthology.orchestra.run.smithi057.stdout:(4/143): ceph-mds-18.0.0-6786.gf7420d7f.el9.x86 9.6 MB/s | 2.1 MB 00:00 2023-10-19T12:42:01.957 INFO:teuthology.orchestra.run.smithi057.stdout:(5/143): ceph-fuse-18.0.0-6786.gf7420d7f.el9.x8 1.1 MB/s | 857 kB 00:00 2023-10-19T12:42:02.049 INFO:teuthology.orchestra.run.smithi057.stdout:(6/143): ceph-mgr-18.0.0-6786.gf7420d7f.el9.x86 8.3 MB/s | 1.5 MB 00:00 2023-10-19T12:42:02.483 INFO:teuthology.orchestra.run.smithi057.stdout:(7/143): ceph-common-18.0.0-6786.gf7420d7f.el9. 12 MB/s | 19 MB 00:01 2023-10-19T12:42:03.142 INFO:teuthology.orchestra.run.smithi057.stdout:(8/143): ceph-radosgw-18.0.0-6786.gf7420d7f.el9 13 MB/s | 8.6 MB 00:00 2023-10-19T12:42:03.200 INFO:teuthology.orchestra.run.smithi057.stdout:(9/143): ceph-selinux-18.0.0-6786.gf7420d7f.el9 415 kB/s | 24 kB 00:00 2023-10-19T12:42:03.442 INFO:teuthology.orchestra.run.smithi057.stdout:(10/143): ceph-osd-18.0.0-6786.gf7420d7f.el9.x8 13 MB/s | 18 MB 00:01 2023-10-19T12:42:03.609 INFO:teuthology.orchestra.run.smithi057.stdout:(11/143): cephfs-mirror-18.0.0-6786.gf7420d7f.e 1.3 MB/s | 218 kB 00:00 2023-10-19T12:42:03.659 INFO:teuthology.orchestra.run.smithi057.stdout:(12/143): libcephfs-devel-18.0.0-6786.gf7420d7f 626 kB/s | 31 kB 00:00 2023-10-19T12:42:03.760 INFO:teuthology.orchestra.run.smithi057.stdout:(13/143): libcephfs2-18.0.0-6786.gf7420d7f.el9. 7.0 MB/s | 716 kB 00:00 2023-10-19T12:42:03.835 INFO:teuthology.orchestra.run.smithi057.stdout:(14/143): libcephsqlite-18.0.0-6786.gf7420d7f.e 2.1 MB/s | 164 kB 00:00 2023-10-19T12:42:03.910 INFO:teuthology.orchestra.run.smithi057.stdout:(15/143): librados-devel-18.0.0-6786.gf7420d7f. 1.7 MB/s | 127 kB 00:00 2023-10-19T12:42:04.002 INFO:teuthology.orchestra.run.smithi057.stdout:(16/143): libradosstriper1-18.0.0-6786.gf7420d7 5.0 MB/s | 472 kB 00:00 2023-10-19T12:42:04.486 INFO:teuthology.orchestra.run.smithi057.stdout:(17/143): librgw2-18.0.0-6786.gf7420d7f.el9.x86 9.9 MB/s | 4.8 MB 00:00 2023-10-19T12:42:04.586 INFO:teuthology.orchestra.run.smithi057.stdout:(18/143): ceph-mon-18.0.0-6786.gf7420d7f.el9.x8 1.7 MB/s | 4.4 MB 00:02 2023-10-19T12:42:04.611 INFO:teuthology.orchestra.run.smithi057.stdout:(19/143): python3-ceph-argparse-18.0.0-6786.gf7 361 kB/s | 45 kB 00:00 2023-10-19T12:42:04.687 INFO:teuthology.orchestra.run.smithi057.stdout:(20/143): python3-ceph-common-18.0.0-6786.gf742 1.2 MB/s | 127 kB 00:00 2023-10-19T12:42:04.712 INFO:teuthology.orchestra.run.smithi057.stdout:(21/143): python3-cephfs-18.0.0-6786.gf7420d7f. 1.6 MB/s | 162 kB 00:00 2023-10-19T12:42:04.796 INFO:teuthology.orchestra.run.smithi057.stdout:(22/143): python3-rbd-18.0.0-6786.gf7420d7f.el9 3.5 MB/s | 297 kB 00:00 2023-10-19T12:42:04.871 INFO:teuthology.orchestra.run.smithi057.stdout:(23/143): python3-rgw-18.0.0-6786.gf7420d7f.el9 1.3 MB/s | 99 kB 00:00 2023-10-19T12:42:04.938 INFO:teuthology.orchestra.run.smithi057.stdout:(24/143): python3-rados-18.0.0-6786.gf7420d7f.e 1.2 MB/s | 321 kB 00:00 2023-10-19T12:42:04.955 INFO:teuthology.orchestra.run.smithi057.stdout:(25/143): rbd-fuse-18.0.0-6786.gf7420d7f.el9.x8 1.0 MB/s | 86 kB 00:00 2023-10-19T12:42:05.030 INFO:teuthology.orchestra.run.smithi057.stdout:(26/143): rbd-nbd-18.0.0-6786.gf7420d7f.el9.x86 2.2 MB/s | 169 kB 00:00 2023-10-19T12:42:05.439 INFO:teuthology.orchestra.run.smithi057.stdout:(27/143): ceph-test-18.0.0-6786.gf7420d7f.el9.x 18 MB/s | 41 MB 00:02 2023-10-19T12:42:05.455 INFO:teuthology.orchestra.run.smithi057.stdout:(28/143): ceph-grafana-dashboards-18.0.0-6786.g 55 kB/s | 23 kB 00:00 2023-10-19T12:42:05.489 INFO:teuthology.orchestra.run.smithi057.stdout:(29/143): ceph-mgr-cephadm-18.0.0-6786.gf7420d7 2.6 MB/s | 132 kB 00:00 2023-10-19T12:42:05.598 INFO:teuthology.orchestra.run.smithi057.stdout:(30/143): ceph-mgr-dashboard-18.0.0-6786.gf7420 12 MB/s | 1.8 MB 00:00 2023-10-19T12:42:05.682 INFO:teuthology.orchestra.run.smithi057.stdout:(31/143): ceph-mgr-modules-core-18.0.0-6786.gf7 2.8 MB/s | 242 kB 00:00 2023-10-19T12:42:05.757 INFO:teuthology.orchestra.run.smithi057.stdout:(32/143): ceph-mgr-rook-18.0.0-6786.gf7420d7f.e 635 kB/s | 47 kB 00:00 2023-10-19T12:42:05.833 INFO:teuthology.orchestra.run.smithi057.stdout:(33/143): ceph-prometheus-alerts-18.0.0-6786.gf 201 kB/s | 15 kB 00:00 2023-10-19T12:42:05.925 INFO:teuthology.orchestra.run.smithi057.stdout:(34/143): ceph-mgr-diskprediction-local-18.0.0- 17 MB/s | 7.4 MB 00:00 2023-10-19T12:42:05.950 INFO:teuthology.orchestra.run.smithi057.stdout:(35/143): ceph-volume-18.0.0-6786.gf7420d7f.el9 2.2 MB/s | 262 kB 00:00 2023-10-19T12:42:05.975 INFO:teuthology.orchestra.run.smithi057.stdout:(36/143): cephadm-18.0.0-6786.gf7420d7f.el9.noa 5.3 MB/s | 270 kB 00:00 2023-10-19T12:42:06.000 INFO:teuthology.orchestra.run.smithi057.stdout:(37/143): cephfs-top-18.0.0-6786.gf7420d7f.el9. 363 kB/s | 18 kB 00:00 2023-10-19T12:42:06.209 INFO:teuthology.orchestra.run.smithi057.stdout:(38/143): boost-program-options-1.75.0-8.el9.x8 509 kB/s | 106 kB 00:00 2023-10-19T12:42:06.301 INFO:teuthology.orchestra.run.smithi057.stdout:(39/143): bison-3.7.4-5.el9.x86_64.rpm 2.8 MB/s | 920 kB 00:00 2023-10-19T12:42:06.327 INFO:teuthology.orchestra.run.smithi057.stdout:(40/143): flex-2.6.4-9.el9.x86_64.rpm 2.6 MB/s | 307 kB 00:00 2023-10-19T12:42:06.368 INFO:teuthology.orchestra.run.smithi057.stdout:(41/143): flexiblas-3.0.4-8.el9.x86_64.rpm 478 kB/s | 32 kB 00:00 2023-10-19T12:42:06.552 INFO:teuthology.orchestra.run.smithi057.stdout:(42/143): flexiblas-netlib-3.0.4-8.el9.x86_64.r 13 MB/s | 3.0 MB 00:00 2023-10-19T12:42:06.577 INFO:teuthology.orchestra.run.smithi057.stdout:(43/143): flexiblas-openblas-openmp-3.0.4-8.el9 82 kB/s | 17 kB 00:00 2023-10-19T12:42:06.619 INFO:teuthology.orchestra.run.smithi057.stdout:(44/143): rbd-mirror-18.0.0-6786.gf7420d7f.el9. 1.8 MB/s | 3.0 MB 00:01 2023-10-19T12:42:06.644 INFO:teuthology.orchestra.run.smithi057.stdout:(45/143): libaio-devel-0.3.111-13.el9.x86_64.rp 150 kB/s | 10 kB 00:00 2023-10-19T12:42:06.669 INFO:teuthology.orchestra.run.smithi057.stdout:(46/143): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.7 MB/s | 159 kB 00:00 2023-10-19T12:42:06.695 INFO:teuthology.orchestra.run.smithi057.stdout:(47/143): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 661 kB 00:00 2023-10-19T12:42:06.720 INFO:teuthology.orchestra.run.smithi057.stdout:(48/143): libzstd-devel-1.5.1-2.el9.x86_64.rpm 954 kB/s | 46 kB 00:00 2023-10-19T12:42:06.751 INFO:teuthology.orchestra.run.smithi057.stdout:(49/143): libtool-2.4.6-45.el9.x86_64.rpm 7.1 MB/s | 578 kB 00:00 2023-10-19T12:42:06.776 INFO:teuthology.orchestra.run.smithi057.stdout:(50/143): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.4 MB/s | 282 kB 00:00 2023-10-19T12:42:06.802 INFO:teuthology.orchestra.run.smithi057.stdout:(51/143): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.7 MB/s | 217 kB 00:00 2023-10-19T12:42:06.827 INFO:teuthology.orchestra.run.smithi057.stdout:(52/143): openblas-0.3.21-2.el9.x86_64.rpm 688 kB/s | 34 kB 00:00 2023-10-19T12:42:06.970 INFO:teuthology.orchestra.run.smithi057.stdout:(53/143): openblas-openmp-0.3.21-2.el9.x86_64.r 25 MB/s | 4.7 MB 00:00 2023-10-19T12:42:07.012 INFO:teuthology.orchestra.run.smithi057.stdout:(54/143): postgresql-13.11-1.el9.x86_64.rpm 7.3 MB/s | 1.5 MB 00:00 2023-10-19T12:42:07.037 INFO:teuthology.orchestra.run.smithi057.stdout:(55/143): postgresql-contrib-13.11-1.el9.x86_64 3.9 MB/s | 811 kB 00:00 2023-10-19T12:42:07.062 INFO:teuthology.orchestra.run.smithi057.stdout:(56/143): postgresql-private-libs-13.11-1.el9.x 1.5 MB/s | 134 kB 00:00 2023-10-19T12:42:07.063 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2023-10-19T12:42:07.136 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : thrift-0.15.0-2.el9.x86_64 1/153 2023-10-19T12:42:07.161 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 2/153 2023-10-19T12:42:07.187 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 3/153 2023-10-19T12:42:07.243 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/153 2023-10-19T12:42:07.296 INFO:teuthology.orchestra.run.smithi057.stdout:(57/143): postgresql-server-13.11-1.el9.x86_64. 22 MB/s | 5.7 MB 00:00 2023-10-19T12:42:07.371 INFO:teuthology.orchestra.run.smithi057.stdout:(58/143): python3-babel-2.9.1-2.el9.noarch.rpm 17 MB/s | 5.8 MB 00:00 2023-10-19T12:42:07.397 INFO:teuthology.orchestra.run.smithi057.stdout:(59/143): python3-devel-3.9.17-2.el9.x86_64.rpm 615 kB/s | 205 kB 00:00 2023-10-19T12:42:07.421 INFO:teuthology.orchestra.run.smithi057.stdout:(60/143): python3-jinja2-2.11.3-4.el9.noarch.rp 1.8 MB/s | 228 kB 00:00 2023-10-19T12:42:07.446 INFO:teuthology.orchestra.run.smithi057.stdout:(61/143): python3-jsonpatch-1.21-16.el9.noarch. 492 kB/s | 24 kB 00:00 2023-10-19T12:42:07.472 INFO:teuthology.orchestra.run.smithi057.stdout:(62/143): python3-jsonpointer-2.0-4.el9.noarch. 242 kB/s | 18 kB 00:00 2023-10-19T12:42:07.497 INFO:teuthology.orchestra.run.smithi057.stdout:(63/143): python3-mako-1.1.4-6.el9.noarch.rpm 1.9 MB/s | 144 kB 00:00 2023-10-19T12:42:07.522 INFO:teuthology.orchestra.run.smithi057.stdout:(64/143): python3-markupsafe-1.1.1-12.el9.x86_6 635 kB/s | 32 kB 00:00 2023-10-19T12:42:07.576 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 5/153 2023-10-19T12:42:07.638 INFO:teuthology.orchestra.run.smithi179.stdout: Upgrading : librados2-2:18.0.0-6786.gf7420d7f.el9.x86_64 6/153 2023-10-19T12:42:07.656 INFO:teuthology.orchestra.run.smithi057.stdout:(65/143): python3-numpy-1.20.1-5.el9.x86_64.rpm 26 MB/s | 4.8 MB 00:00 2023-10-19T12:42:07.681 INFO:teuthology.orchestra.run.smithi057.stdout:(66/143): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.2 MB/s | 217 kB 00:00 2023-10-19T12:42:07.706 INFO:teuthology.orchestra.run.smithi057.stdout:(67/143): python3-oauthlib-3.1.1-5.el9.noarch.r 967 kB/s | 174 kB 00:00 2023-10-19T12:42:07.732 INFO:teuthology.orchestra.run.smithi057.stdout:(68/143): python3-prettytable-0.7.2-27.el9.noar 553 kB/s | 41 kB 00:00 2023-10-19T12:42:07.748 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: librados2-2:18.0.0-6786.gf7420d7f.el9.x86_64 6/153 2023-10-19T12:42:07.757 INFO:teuthology.orchestra.run.smithi057.stdout:(69/143): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 132 kB 00:00 2023-10-19T12:42:07.782 INFO:teuthology.orchestra.run.smithi057.stdout:(70/143): python3-pyasn1-modules-0.4.8-6.el9.no 2.7 MB/s | 211 kB 00:00 2023-10-19T12:42:07.797 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libcephfs2-2:18.0.0-6786.gf7420d7f.el9.x86_64 7/153 2023-10-19T12:42:07.807 INFO:teuthology.orchestra.run.smithi057.stdout:(71/143): python3-requests-oauthlib-1.3.0-12.el 569 kB/s | 43 kB 00:00 2023-10-19T12:42:07.832 INFO:teuthology.orchestra.run.smithi057.stdout:(72/143): python3-toml-0.10.2-6.el9.noarch.rpm 776 kB/s | 39 kB 00:00 2023-10-19T12:42:07.894 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: libcephfs2-2:18.0.0-6786.gf7420d7f.el9.x86_64 7/153 2023-10-19T12:42:07.939 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-rados-2:18.0.0-6786.gf7420d7f.el9.x86_64 8/153 2023-10-19T12:42:07.972 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 9/153 2023-10-19T12:42:08.066 INFO:teuthology.orchestra.run.smithi057.stdout:(73/143): socat-1.7.4.1-5.el9.x86_64.rpm 1.1 MB/s | 300 kB 00:00 2023-10-19T12:42:08.129 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 10/153 2023-10-19T12:42:08.183 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : postgresql-13.11-1.el9.x86_64 11/153 2023-10-19T12:42:08.210 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libcephsqlite-2:18.0.0-6786.gf7420d7f.el9.x86_64 12/153 2023-10-19T12:42:08.233 INFO:teuthology.orchestra.run.smithi057.stdout:(74/143): python3-scipy-1.6.2-8.el9.x86_64.rpm 35 MB/s | 16 MB 00:00 2023-10-19T12:42:08.250 INFO:teuthology.orchestra.run.smithi057.stdout:(75/143): uuid-1.6.2-55.el9.x86_64.rpm 136 kB/s | 56 kB 00:00 2023-10-19T12:42:08.267 INFO:teuthology.orchestra.run.smithi057.stdout:(76/143): xmlstarlet-1.6.1-20.el9.x86_64.rpm 321 kB/s | 63 kB 00:00 2023-10-19T12:42:08.292 INFO:teuthology.orchestra.run.smithi057.stdout:(77/143): fuse-2.9.9-15.el9.x86_64.rpm 1.9 MB/s | 79 kB 00:00 2023-10-19T12:42:08.311 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: libcephsqlite-2:18.0.0-6786.gf7420d7f.el9.x86_64 12/153 2023-10-19T12:42:08.318 INFO:teuthology.orchestra.run.smithi057.stdout:(78/143): libgfortran-11.4.1-2.3.el9.x86_64.rpm 12 MB/s | 803 kB 00:00 2023-10-19T12:42:08.336 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libradosstriper1-2:18.0.0-6786.gf7420d7f.el9.x86 13/153 2023-10-19T12:42:08.343 INFO:teuthology.orchestra.run.smithi057.stdout:(79/143): libquadmath-11.4.1-2.3.el9.x86_64.rpm 2.5 MB/s | 195 kB 00:00 2023-10-19T12:42:08.368 INFO:teuthology.orchestra.run.smithi057.stdout:(80/143): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.2 MB/s | 241 kB 00:00 2023-10-19T12:42:08.380 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: libradosstriper1-2:18.0.0-6786.gf7420d7f.el9.x86 13/153 2023-10-19T12:42:08.393 INFO:teuthology.orchestra.run.smithi057.stdout:(81/143): python3-ply-3.11-14.el9.noarch.rpm 2.0 MB/s | 103 kB 00:00 2023-10-19T12:42:08.418 INFO:teuthology.orchestra.run.smithi057.stdout:(82/143): python3-pycparser-2.20-6.el9.noarch.r 1.6 MB/s | 124 kB 00:00 2023-10-19T12:42:08.435 INFO:teuthology.orchestra.run.smithi057.stdout:(83/143): python3-requests-2.25.1-7.el9.noarch. 1.6 MB/s | 113 kB 00:00 2023-10-19T12:42:08.461 INFO:teuthology.orchestra.run.smithi057.stdout:(84/143): python3-urllib3-1.26.5-3.el9.noarch.r 4.4 MB/s | 188 kB 00:00 2023-10-19T12:42:08.565 INFO:teuthology.orchestra.run.smithi179.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 14/153 2023-10-19T12:42:08.674 INFO:teuthology.orchestra.run.smithi179.stdout: Upgrading : NetworkManager-libnm-1:1.45.4-1.el9.x86_64 15/153 2023-10-19T12:42:08.733 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: NetworkManager-libnm-1:1.45.4-1.el9.x86_64 15/153 2023-10-19T12:42:08.736 INFO:teuthology.orchestra.run.smithi057.stdout:(85/143): python3-cryptography-36.0.1-4.el9.x86 3.9 MB/s | 1.2 MB 00:00 2023-10-19T12:42:08.839 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: NetworkManager-1:1.45.4-1.el9.x86_64 16/153 2023-10-19T12:42:08.893 INFO:teuthology.orchestra.run.smithi179.stdout: Upgrading : NetworkManager-1:1.45.4-1.el9.x86_64 16/153 2023-10-19T12:42:09.167 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: NetworkManager-1:1.45.4-1.el9.x86_64 16/153 2023-10-19T12:42:09.214 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 17/153 2023-10-19T12:42:09.249 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : liboath-2.6.7-2.el9.x86_64 18/153 2023-10-19T12:42:09.320 INFO:teuthology.orchestra.run.smithi057.stdout:(86/143): elfutils-libelf-devel-0.189-3.el9.x86 26 kB/s | 22 kB 00:00 2023-10-19T12:42:09.327 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 19/153 2023-10-19T12:42:09.397 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-urllib3-1.26.5-3.el9.noarch 20/153 2023-10-19T12:42:09.427 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 21/153 2023-10-19T12:42:09.488 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 22/153 2023-10-19T12:42:09.602 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 23/153 2023-10-19T12:42:09.645 INFO:teuthology.orchestra.run.smithi057.stdout:(87/143): jq-1.6-15.el9.x86_64.rpm 206 kB/s | 187 kB 00:00 2023-10-19T12:42:09.658 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/153 2023-10-19T12:42:09.689 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/153 2023-10-19T12:42:09.745 INFO:teuthology.orchestra.run.smithi057.stdout:(88/143): automake-1.16.2-8.el9.noarch.rpm 526 kB/s | 689 kB 00:01 2023-10-19T12:42:09.779 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/153 2023-10-19T12:42:09.816 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-ceph-common-2:18.0.0-6786.gf7420d7f.el9. 27/153 2023-10-19T12:42:09.852 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-ceph-argparse-2:18.0.0-6786.gf7420d7f.el 28/153 2023-10-19T12:42:09.926 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-cephfs-2:18.0.0-6786.gf7420d7f.el9.x86_6 29/153 2023-10-19T12:42:10.072 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 30/153 2023-10-19T12:42:10.129 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 31/153 2023-10-19T12:42:10.262 INFO:teuthology.orchestra.run.smithi057.stdout:(89/143): librabbitmq-0.11.0-7.el9.x86_64.rpm 48 kB/s | 45 kB 00:00 2023-10-19T12:42:10.362 INFO:teuthology.orchestra.run.smithi057.stdout:(90/143): libuuid-devel-2.37.4-15.el9.x86_64.rp 37 kB/s | 26 kB 00:00 2023-10-19T12:42:10.662 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-rsa-4.9-2.el9.noarch 32/153 2023-10-19T12:42:10.742 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 33/153 2023-10-19T12:42:10.779 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 34/153 2023-10-19T12:42:10.817 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : librados-devel-2:18.0.0-6786.gf7420d7f.el9.x86_6 35/153 2023-10-19T12:42:10.847 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 36/153 2023-10-19T12:42:10.953 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : re2-1:20211101-3.el9.x86_64 37/153 2023-10-19T12:42:11.001 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libarrow-9.0.0-9.el9.x86_64 38/153 2023-10-19T12:42:11.056 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 39/153 2023-10-19T12:42:11.063 INFO:teuthology.orchestra.run.smithi057.stdout:(91/143): python3-pytz-2021.1-5.el9.noarch.rpm 64 kB/s | 51 kB 00:00 2023-10-19T12:42:11.137 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 40/153 2023-10-19T12:42:11.168 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 41/153 2023-10-19T12:42:11.205 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 42/153 2023-10-19T12:42:11.238 INFO:teuthology.orchestra.run.smithi057.stdout:(92/143): xfsprogs-devel-5.19.0-4.el9.x86_64.rp 98 kB/s | 86 kB 00:00 2023-10-19T12:42:11.251 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 43/153 2023-10-19T12:42:11.296 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 44/153 2023-10-19T12:42:11.340 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 45/153 2023-10-19T12:42:11.380 INFO:teuthology.orchestra.run.smithi057.stdout:(93/143): fmt-8.1.1-5.el9.x86_64.rpm 784 kB/s | 111 kB 00:00 2023-10-19T12:42:11.438 INFO:teuthology.orchestra.run.smithi057.stdout:(94/143): gperftools-libs-2.9.1-2.el9.x86_64.rp 5.2 MB/s | 309 kB 00:00 2023-10-19T12:42:11.484 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 46/153 2023-10-19T12:42:11.524 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 47/153 2023-10-19T12:42:11.556 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 48/153 2023-10-19T12:42:11.641 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 49/153 2023-10-19T12:42:11.672 INFO:teuthology.orchestra.run.smithi057.stdout:(95/143): libarrow-9.0.0-9.el9.x86_64.rpm 19 MB/s | 4.4 MB 00:00 2023-10-19T12:42:11.682 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-google-auth-1:2.23.3-1.el9.noarch 50/153 2023-10-19T12:42:11.714 INFO:teuthology.orchestra.run.smithi057.stdout:(96/143): libarrow-doc-9.0.0-9.el9.noarch.rpm 590 kB/s | 24 kB 00:00 2023-10-19T12:42:11.723 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 51/153 2023-10-19T12:42:11.755 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 52/153 2023-10-19T12:42:11.756 INFO:teuthology.orchestra.run.smithi057.stdout:(97/143): liboath-2.6.7-2.el9.x86_64.rpm 1.1 MB/s | 49 kB 00:00 2023-10-19T12:42:11.783 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libarrow-doc-9.0.0-9.el9.noarch 53/153 2023-10-19T12:42:11.798 INFO:teuthology.orchestra.run.smithi057.stdout:(98/143): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2023-10-19T12:42:11.816 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : fmt-8.1.1-5.el9.x86_64 54/153 2023-10-19T12:42:11.823 INFO:teuthology.orchestra.run.smithi057.stdout:(99/143): zlib-devel-1.2.11-40.el9.x86_64.rpm 59 kB/s | 45 kB 00:00 2023-10-19T12:42:11.844 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 55/153 2023-10-19T12:42:11.873 INFO:teuthology.orchestra.run.smithi057.stdout:(100/143): parquet-libs-9.0.0-9.el9.x86_64.rpm 11 MB/s | 837 kB 00:00 2023-10-19T12:42:11.916 INFO:teuthology.orchestra.run.smithi057.stdout:(101/143): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.0 MB/s | 43 kB 00:00 2023-10-19T12:42:11.931 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libuuid-devel-2.37.4-15.el9.x86_64 56/153 2023-10-19T12:42:11.957 INFO:teuthology.orchestra.run.smithi057.stdout:(102/143): python3-cachetools-4.2.4-1.el9.noarc 776 kB/s | 32 kB 00:00 2023-10-19T12:42:11.968 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : automake-1.16.2-8.el9.noarch 57/153 2023-10-19T12:42:12.008 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-ply-3.11-14.el9.noarch 58/153 2023-10-19T12:42:12.033 INFO:teuthology.orchestra.run.smithi057.stdout:(103/143): python3-asyncssh-2.13.2-3.el9.noarch 2.6 MB/s | 548 kB 00:00 2023-10-19T12:42:12.050 INFO:teuthology.orchestra.run.smithi057.stdout:(104/143): python3-certifi-2021.10.8-2.el9.noar 163 kB/s | 15 kB 00:00 2023-10-19T12:42:12.065 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 59/153 2023-10-19T12:42:12.075 INFO:teuthology.orchestra.run.smithi057.stdout:(105/143): python3-cheroot-8.6.0-4.el9.noarch.r 4.0 MB/s | 172 kB 00:00 2023-10-19T12:42:12.114 INFO:teuthology.orchestra.run.smithi057.stdout:(106/143): python3-cherrypy-18.6.1-2.el9.noarch 5.5 MB/s | 358 kB 00:00 2023-10-19T12:42:12.139 INFO:teuthology.orchestra.run.smithi057.stdout:(107/143): python3-google-auth-2.23.3-1.el9.noa 3.0 MB/s | 197 kB 00:00 2023-10-19T12:42:12.164 INFO:teuthology.orchestra.run.smithi057.stdout:(108/143): python3-jaraco-8.2.1-3.el9.noarch.rp 212 kB/s | 11 kB 00:00 2023-10-19T12:42:12.190 INFO:teuthology.orchestra.run.smithi057.stdout:(109/143): python3-jaraco-classes-3.2.1-5.el9.n 352 kB/s | 18 kB 00:00 2023-10-19T12:42:12.215 INFO:teuthology.orchestra.run.smithi057.stdout:(110/143): python3-jaraco-collections-3.0.0-8.e 462 kB/s | 23 kB 00:00 2023-10-19T12:42:12.240 INFO:teuthology.orchestra.run.smithi057.stdout:(111/143): python3-jaraco-functools-3.5.0-2.el9 387 kB/s | 19 kB 00:00 2023-10-19T12:42:12.253 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 60/153 2023-10-19T12:42:12.266 INFO:teuthology.orchestra.run.smithi057.stdout:(112/143): python3-jaraco-text-3.2.0-6.el9.noar 391 kB/s | 20 kB 00:00 2023-10-19T12:42:12.291 INFO:teuthology.orchestra.run.smithi057.stdout:(113/143): python3-jwt+crypto-2.4.0-1.el9.noarc 180 kB/s | 9.0 kB 00:00 2023-10-19T12:42:12.297 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 61/153 2023-10-19T12:42:12.316 INFO:teuthology.orchestra.run.smithi057.stdout:(114/143): python3-jwt-2.4.0-1.el9.noarch.rpm 815 kB/s | 41 kB 00:00 2023-10-19T12:42:12.358 INFO:teuthology.orchestra.run.smithi057.stdout:(115/143): python3-logutils-0.3.5-21.el9.noarch 1.1 MB/s | 46 kB 00:00 2023-10-19T12:42:12.392 INFO:teuthology.orchestra.run.smithi057.stdout:(116/143): python3-kubernetes-26.1.0-2.el9.noar 10 MB/s | 1.0 MB 00:00 2023-10-19T12:42:12.417 INFO:teuthology.orchestra.run.smithi057.stdout:(117/143): python3-more-itertools-8.12.0-2.el9. 1.3 MB/s | 79 kB 00:00 2023-10-19T12:42:12.426 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 62/153 2023-10-19T12:42:12.442 INFO:teuthology.orchestra.run.smithi057.stdout:(118/143): python3-natsort-7.1.1-5.el9.noarch.r 1.1 MB/s | 58 kB 00:00 2023-10-19T12:42:12.468 INFO:teuthology.orchestra.run.smithi057.stdout:(119/143): python3-pecan-1.4.2-3.el9.noarch.rpm 5.3 MB/s | 272 kB 00:00 2023-10-19T12:42:12.493 INFO:teuthology.orchestra.run.smithi057.stdout:(120/143): python3-portend-3.1.0-2.el9.noarch.r 327 kB/s | 16 kB 00:00 2023-10-19T12:42:12.518 INFO:teuthology.orchestra.run.smithi057.stdout:(121/143): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.7 MB/s | 90 kB 00:00 2023-10-19T12:42:12.544 INFO:teuthology.orchestra.run.smithi057.stdout:(122/143): python3-repoze-lru-0.7-16.el9.noarch 612 kB/s | 31 kB 00:00 2023-10-19T12:42:12.569 INFO:teuthology.orchestra.run.smithi057.stdout:(123/143): python3-routes-2.5.1-5.el9.noarch.rp 3.7 MB/s | 188 kB 00:00 2023-10-19T12:42:12.586 INFO:teuthology.orchestra.run.smithi057.stdout:(124/143): python3-rsa-4.9-2.el9.noarch.rpm 1.4 MB/s | 59 kB 00:00 2023-10-19T12:42:12.620 INFO:teuthology.orchestra.run.smithi057.stdout:(125/143): python3-tempora-5.0.0-2.el9.noarch.r 711 kB/s | 36 kB 00:00 2023-10-19T12:42:12.637 INFO:teuthology.orchestra.run.smithi057.stdout:(126/143): python3-typing-extensions-4.4.0-2.el 1.0 MB/s | 51 kB 00:00 2023-10-19T12:42:12.670 INFO:teuthology.orchestra.run.smithi057.stdout:(127/143): python3-webob-1.8.7-6.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2023-10-19T12:42:12.674 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 63/153 2023-10-19T12:42:12.695 INFO:teuthology.orchestra.run.smithi057.stdout:(128/143): python3-websocket-client-1.2.3-2.el9 1.5 MB/s | 90 kB 00:00 2023-10-19T12:42:12.705 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 64/153 2023-10-19T12:42:12.738 INFO:teuthology.orchestra.run.smithi057.stdout:(129/143): python3-werkzeug-2.0.3-3.el9.1.noarc 6.2 MB/s | 427 kB 00:00 2023-10-19T12:42:12.739 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 65/153 2023-10-19T12:42:12.763 INFO:teuthology.orchestra.run.smithi057.stdout:(130/143): python3-zc-lockfile-2.0-10.el9.noarc 298 kB/s | 20 kB 00:00 2023-10-19T12:42:12.771 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 66/153 2023-10-19T12:42:12.804 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : fuse-2.9.9-15.el9.x86_64 67/153 2023-10-19T12:42:12.813 INFO:teuthology.orchestra.run.smithi057.stdout:(131/143): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2023-10-19T12:42:12.849 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/153 2023-10-19T12:42:12.899 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/153 2023-10-19T12:42:12.935 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 70/153 2023-10-19T12:42:12.963 INFO:teuthology.orchestra.run.smithi057.stdout:(132/143): thrift-0.15.0-2.el9.x86_64.rpm 7.9 MB/s | 1.6 MB 00:00 2023-10-19T12:42:12.964 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/153 2023-10-19T12:42:13.034 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/153 2023-10-19T12:42:13.066 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 73/153 2023-10-19T12:42:13.091 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 74/153 2023-10-19T12:42:13.132 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 75/153 2023-10-19T12:42:13.167 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 76/153 2023-10-19T12:42:13.239 INFO:teuthology.orchestra.run.smithi057.stdout:(133/143): librbd1-18.0.0-6786.gf7420d7f.el9.x8 11 MB/s | 3.0 MB 00:00 2023-10-19T12:42:13.256 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 77/153 2023-10-19T12:42:13.286 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 78/153 2023-10-19T12:42:13.306 INFO:teuthology.orchestra.run.smithi057.stdout:(134/143): libacl-devel-2.3.1-4.el9.x86_64.rpm 1.0 MB/s | 68 kB 00:00 2023-10-19T12:42:13.388 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/153 2023-10-19T12:42:13.437 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 80/153 2023-10-19T12:42:13.531 INFO:teuthology.orchestra.run.smithi057.stdout:(135/143): NetworkManager-1.45.4-1.el9.x86_64.r 10 MB/s | 2.2 MB 00:00 2023-10-19T12:42:13.807 INFO:teuthology.orchestra.run.smithi057.stdout:(136/143): NetworkManager-libnm-1.45.4-1.el9.x8 6.5 MB/s | 1.8 MB 00:00 2023-10-19T12:42:13.882 INFO:teuthology.orchestra.run.smithi057.stdout:(137/143): NetworkManager-team-1.45.4-1.el9.x86 554 kB/s | 41 kB 00:00 2023-10-19T12:42:14.074 INFO:teuthology.orchestra.run.smithi057.stdout:(138/143): NetworkManager-tui-1.45.4-1.el9.x86_ 1.2 MB/s | 244 kB 00:00 2023-10-19T12:42:14.174 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 81/153 2023-10-19T12:42:14.199 INFO:teuthology.orchestra.run.smithi057.stdout:(139/143): acl-2.3.1-4.el9.x86_64.rpm 550 kB/s | 69 kB 00:00 2023-10-19T12:42:14.217 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 82/153 2023-10-19T12:42:14.249 INFO:teuthology.orchestra.run.smithi057.stdout:(140/143): libacl-2.3.1-4.el9.x86_64.rpm 439 kB/s | 22 kB 00:00 2023-10-19T12:42:14.251 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 83/153 2023-10-19T12:42:14.341 INFO:teuthology.orchestra.run.smithi057.stdout:(141/143): xfsdump-3.1.12-4.el9.x86_64.rpm 3.4 MB/s | 321 kB 00:00 2023-10-19T12:42:14.352 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 84/153 2023-10-19T12:42:14.393 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 85/153 2023-10-19T12:42:14.570 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : openblas-0.3.21-2.el9.x86_64 86/153 2023-10-19T12:42:14.616 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 87/153 2023-10-19T12:42:14.694 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 88/153 2023-10-19T12:42:15.083 INFO:teuthology.orchestra.run.smithi057.stdout:(142/143): librados2-18.0.0-6786.gf7420d7f.el9. 1.5 MB/s | 3.3 MB 00:02 2023-10-19T12:42:15.345 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 89/153 2023-10-19T12:42:15.456 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 90/153 2023-10-19T12:42:16.167 INFO:teuthology.orchestra.run.smithi057.stdout:(143/143): openssl-devel-3.0.7-24.el9.x86_64.rp 658 kB/s | 4.1 MB 00:06 2023-10-19T12:42:16.171 INFO:teuthology.orchestra.run.smithi057.stdout:-------------------------------------------------------------------------------- 2023-10-19T12:42:16.171 INFO:teuthology.orchestra.run.smithi057.stdout:Total 13 MB/s | 200 MB 00:15 2023-10-19T12:42:16.317 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction check 2023-10-19T12:42:16.458 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction check succeeded. 2023-10-19T12:42:16.459 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction test 2023-10-19T12:42:16.693 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 91/153 2023-10-19T12:42:16.776 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 92/153 2023-10-19T12:42:16.807 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 93/153 2023-10-19T12:42:16.838 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : jq-1.6-15.el9.x86_64 94/153 2023-10-19T12:42:16.882 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 95/153 2023-10-19T12:42:17.184 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 96/153 2023-10-19T12:42:17.212 INFO:teuthology.orchestra.run.smithi179.stdout: Upgrading : librbd1-2:18.0.0-6786.gf7420d7f.el9.x86_64 97/153 2023-10-19T12:42:17.308 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: librbd1-2:18.0.0-6786.gf7420d7f.el9.x86_64 97/153 2023-10-19T12:42:17.333 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-rbd-2:18.0.0-6786.gf7420d7f.el9.x86_64 98/153 2023-10-19T12:42:17.369 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 99/153 2023-10-19T12:42:17.794 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : parquet-libs-9.0.0-9.el9.x86_64 100/153 2023-10-19T12:42:17.823 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : librgw2-2:18.0.0-6786.gf7420d7f.el9.x86_64 101/153 2023-10-19T12:42:17.894 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: librgw2-2:18.0.0-6786.gf7420d7f.el9.x86_64 101/153 2023-10-19T12:42:17.916 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-rgw-2:18.0.0-6786.gf7420d7f.el9.x86_64 102/153 2023-10-19T12:42:18.317 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction test succeeded. 2023-10-19T12:42:18.317 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction 2023-10-19T12:42:20.574 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-common-2:18.0.0-6786.gf7420d7f.el9.x86_64 103/153 2023-10-19T12:42:20.621 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-common-2:18.0.0-6786.gf7420d7f.el9.x86_64 103/153 2023-10-19T12:42:21.166 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-common-2:18.0.0-6786.gf7420d7f.el9.x86_64 103/153 2023-10-19T12:42:21.200 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-base-2:18.0.0-6786.gf7420d7f.el9.x86_64 104/153 2023-10-19T12:42:21.286 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-base-2:18.0.0-6786.gf7420d7f.el9.x86_64 104/153 2023-10-19T12:42:21.287 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-10-19T12:42:21.287 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-10-19T12:42:21.287 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-10-19T12:42:21.317 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-selinux-2:18.0.0-6786.gf7420d7f.el9.x86_64 105/153 2023-10-19T12:42:24.230 INFO:teuthology.orchestra.run.smithi057.stdout: Preparing : 1/1 2023-10-19T12:42:24.311 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : thrift-0.15.0-2.el9.x86_64 1/153 2023-10-19T12:42:24.344 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 2/153 2023-10-19T12:42:24.370 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 3/153 2023-10-19T12:42:24.425 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/153 2023-10-19T12:42:24.775 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 5/153 2023-10-19T12:42:24.831 INFO:teuthology.orchestra.run.smithi057.stdout: Upgrading : librados2-2:18.0.0-6786.gf7420d7f.el9.x86_64 6/153 2023-10-19T12:42:24.953 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: librados2-2:18.0.0-6786.gf7420d7f.el9.x86_64 6/153 2023-10-19T12:42:25.015 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libcephfs2-2:18.0.0-6786.gf7420d7f.el9.x86_64 7/153 2023-10-19T12:42:25.112 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: libcephfs2-2:18.0.0-6786.gf7420d7f.el9.x86_64 7/153 2023-10-19T12:42:25.161 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-rados-2:18.0.0-6786.gf7420d7f.el9.x86_64 8/153 2023-10-19T12:42:25.196 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 9/153 2023-10-19T12:42:25.363 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 10/153 2023-10-19T12:42:25.409 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : postgresql-13.11-1.el9.x86_64 11/153 2023-10-19T12:42:25.436 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libcephsqlite-2:18.0.0-6786.gf7420d7f.el9.x86_64 12/153 2023-10-19T12:42:25.539 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: libcephsqlite-2:18.0.0-6786.gf7420d7f.el9.x86_64 12/153 2023-10-19T12:42:25.562 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libradosstriper1-2:18.0.0-6786.gf7420d7f.el9.x86 13/153 2023-10-19T12:42:25.657 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: libradosstriper1-2:18.0.0-6786.gf7420d7f.el9.x86 13/153 2023-10-19T12:42:25.845 INFO:teuthology.orchestra.run.smithi057.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 14/153 2023-10-19T12:42:25.975 INFO:teuthology.orchestra.run.smithi057.stdout: Upgrading : NetworkManager-libnm-1:1.45.4-1.el9.x86_64 15/153 2023-10-19T12:42:26.068 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: NetworkManager-libnm-1:1.45.4-1.el9.x86_64 15/153 2023-10-19T12:42:26.180 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: NetworkManager-1:1.45.4-1.el9.x86_64 16/153 2023-10-19T12:42:26.220 INFO:teuthology.orchestra.run.smithi057.stdout: Upgrading : NetworkManager-1:1.45.4-1.el9.x86_64 16/153 2023-10-19T12:42:26.487 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: NetworkManager-1:1.45.4-1.el9.x86_64 16/153 2023-10-19T12:42:26.523 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 17/153 2023-10-19T12:42:26.559 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : liboath-2.6.7-2.el9.x86_64 18/153 2023-10-19T12:42:26.638 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 19/153 2023-10-19T12:42:26.690 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-urllib3-1.26.5-3.el9.noarch 20/153 2023-10-19T12:42:26.728 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 21/153 2023-10-19T12:42:26.772 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 22/153 2023-10-19T12:42:26.845 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 23/153 2023-10-19T12:42:26.885 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/153 2023-10-19T12:42:26.912 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/153 2023-10-19T12:42:26.984 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/153 2023-10-19T12:42:27.017 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-ceph-common-2:18.0.0-6786.gf7420d7f.el9. 27/153 2023-10-19T12:42:27.062 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-ceph-argparse-2:18.0.0-6786.gf7420d7f.el 28/153 2023-10-19T12:42:27.137 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-cephfs-2:18.0.0-6786.gf7420d7f.el9.x86_6 29/153 2023-10-19T12:42:27.278 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 30/153 2023-10-19T12:42:27.341 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 31/153 2023-10-19T12:42:27.912 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-rsa-4.9-2.el9.noarch 32/153 2023-10-19T12:42:27.994 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 33/153 2023-10-19T12:42:28.039 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 34/153 2023-10-19T12:42:28.076 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : librados-devel-2:18.0.0-6786.gf7420d7f.el9.x86_6 35/153 2023-10-19T12:42:28.107 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 36/153 2023-10-19T12:42:28.218 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : re2-1:20211101-3.el9.x86_64 37/153 2023-10-19T12:42:28.253 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libarrow-9.0.0-9.el9.x86_64 38/153 2023-10-19T12:42:28.308 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 39/153 2023-10-19T12:42:28.382 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 40/153 2023-10-19T12:42:28.419 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 41/153 2023-10-19T12:42:28.456 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 42/153 2023-10-19T12:42:28.503 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 43/153 2023-10-19T12:42:28.550 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 44/153 2023-10-19T12:42:28.590 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 45/153 2023-10-19T12:42:28.733 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 46/153 2023-10-19T12:42:28.775 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 47/153 2023-10-19T12:42:28.817 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 48/153 2023-10-19T12:42:28.915 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 49/153 2023-10-19T12:42:28.951 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-google-auth-1:2.23.3-1.el9.noarch 50/153 2023-10-19T12:42:28.991 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 51/153 2023-10-19T12:42:29.023 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 52/153 2023-10-19T12:42:29.050 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libarrow-doc-9.0.0-9.el9.noarch 53/153 2023-10-19T12:42:29.083 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : fmt-8.1.1-5.el9.x86_64 54/153 2023-10-19T12:42:29.112 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 55/153 2023-10-19T12:42:29.206 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libuuid-devel-2.37.4-15.el9.x86_64 56/153 2023-10-19T12:42:29.253 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : automake-1.16.2-8.el9.noarch 57/153 2023-10-19T12:42:29.311 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-ply-3.11-14.el9.noarch 58/153 2023-10-19T12:42:29.376 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 59/153 2023-10-19T12:42:29.575 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 60/153 2023-10-19T12:42:29.624 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 61/153 2023-10-19T12:42:29.771 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 62/153 2023-10-19T12:42:29.818 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 63/153 2023-10-19T12:42:29.846 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 64/153 2023-10-19T12:42:29.880 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 65/153 2023-10-19T12:42:29.913 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 66/153 2023-10-19T12:42:29.946 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : fuse-2.9.9-15.el9.x86_64 67/153 2023-10-19T12:42:29.986 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/153 2023-10-19T12:42:30.025 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/153 2023-10-19T12:42:30.060 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 70/153 2023-10-19T12:42:30.089 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/153 2023-10-19T12:42:30.462 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/153 2023-10-19T12:42:30.500 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 73/153 2023-10-19T12:42:30.533 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 74/153 2023-10-19T12:42:30.576 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 75/153 2023-10-19T12:42:30.609 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 76/153 2023-10-19T12:42:30.709 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 77/153 2023-10-19T12:42:30.753 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 78/153 2023-10-19T12:42:30.863 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/153 2023-10-19T12:42:30.922 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 80/153 2023-10-19T12:42:31.709 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 81/153 2023-10-19T12:42:31.759 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 82/153 2023-10-19T12:42:31.793 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 83/153 2023-10-19T12:42:31.897 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 84/153 2023-10-19T12:42:31.940 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 85/153 2023-10-19T12:42:32.125 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : openblas-0.3.21-2.el9.x86_64 86/153 2023-10-19T12:42:32.155 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 87/153 2023-10-19T12:42:32.242 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 88/153 2023-10-19T12:42:32.625 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-selinux-2:18.0.0-6786.gf7420d7f.el9.x86_64 105/153 2023-10-19T12:42:32.625 INFO:teuthology.orchestra.run.smithi179.stdout:skipping the directory /sys 2023-10-19T12:42:32.626 INFO:teuthology.orchestra.run.smithi179.stdout:skipping the directory /proc 2023-10-19T12:42:32.626 INFO:teuthology.orchestra.run.smithi179.stdout:skipping the directory /mnt 2023-10-19T12:42:32.626 INFO:teuthology.orchestra.run.smithi179.stdout:skipping the directory /var/tmp 2023-10-19T12:42:32.626 INFO:teuthology.orchestra.run.smithi179.stdout:skipping the directory /home 2023-10-19T12:42:32.627 INFO:teuthology.orchestra.run.smithi179.stdout:skipping the directory /root 2023-10-19T12:42:32.627 INFO:teuthology.orchestra.run.smithi179.stdout:skipping the directory /tmp 2023-10-19T12:42:32.627 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-10-19T12:42:32.787 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-volume-2:18.0.0-6786.gf7420d7f.el9.noarch 106/153 2023-10-19T12:42:32.802 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-volume-2:18.0.0-6786.gf7420d7f.el9.noarch 106/153 2023-10-19T12:42:32.802 INFO:teuthology.orchestra.run.smithi179.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-19T12:42:32.802 INFO:teuthology.orchestra.run.smithi179.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-10-19T12:42:32.802 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-10-19T12:42:32.945 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 89/153 2023-10-19T12:42:33.058 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 90/153 2023-10-19T12:42:34.355 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-osd-2:18.0.0-6786.gf7420d7f.el9.x86_64 107/153 2023-10-19T12:42:34.381 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-osd-2:18.0.0-6786.gf7420d7f.el9.x86_64 107/153 2023-10-19T12:42:34.381 INFO:teuthology.orchestra.run.smithi179.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-19T12:42:34.381 INFO:teuthology.orchestra.run.smithi179.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-10-19T12:42:34.381 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-10-19T12:42:34.382 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-10-19T12:42:34.382 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-10-19T12:42:34.383 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 91/153 2023-10-19T12:42:34.454 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 92/153 2023-10-19T12:42:34.485 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 93/153 2023-10-19T12:42:34.516 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : jq-1.6-15.el9.x86_64 94/153 2023-10-19T12:42:34.543 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 95/153 2023-10-19T12:42:34.581 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-mds-2:18.0.0-6786.gf7420d7f.el9.x86_64 108/153 2023-10-19T12:42:34.604 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-mds-2:18.0.0-6786.gf7420d7f.el9.x86_64 108/153 2023-10-19T12:42:34.605 INFO:teuthology.orchestra.run.smithi179.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-19T12:42:34.605 INFO:teuthology.orchestra.run.smithi179.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-10-19T12:42:34.605 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-10-19T12:42:34.605 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-10-19T12:42:34.605 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-10-19T12:42:34.861 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 96/153 2023-10-19T12:42:34.889 INFO:teuthology.orchestra.run.smithi057.stdout: Upgrading : librbd1-2:18.0.0-6786.gf7420d7f.el9.x86_64 97/153 2023-10-19T12:42:34.964 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: librbd1-2:18.0.0-6786.gf7420d7f.el9.x86_64 97/153 2023-10-19T12:42:35.002 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-rbd-2:18.0.0-6786.gf7420d7f.el9.x86_64 98/153 2023-10-19T12:42:35.040 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 99/153 2023-10-19T12:42:35.067 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-mon-2:18.0.0-6786.gf7420d7f.el9.x86_64 109/153 2023-10-19T12:42:35.089 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-mon-2:18.0.0-6786.gf7420d7f.el9.x86_64 109/153 2023-10-19T12:42:35.090 INFO:teuthology.orchestra.run.smithi179.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-19T12:42:35.090 INFO:teuthology.orchestra.run.smithi179.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-10-19T12:42:35.090 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-10-19T12:42:35.090 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-10-19T12:42:35.091 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-10-19T12:42:35.492 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : parquet-libs-9.0.0-9.el9.x86_64 100/153 2023-10-19T12:42:35.517 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : librgw2-2:18.0.0-6786.gf7420d7f.el9.x86_64 101/153 2023-10-19T12:42:35.532 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: cephadm-2:18.0.0-6786.gf7420d7f.el9.noarch 110/153 2023-10-19T12:42:35.557 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : cephadm-2:18.0.0-6786.gf7420d7f.el9.noarch 110/153 2023-10-19T12:42:35.580 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: librgw2-2:18.0.0-6786.gf7420d7f.el9.x86_64 101/153 2023-10-19T12:42:35.586 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-6786.gf7420d7f.e 111/153 2023-10-19T12:42:35.610 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-rgw-2:18.0.0-6786.gf7420d7f.el9.x86_64 102/153 2023-10-19T12:42:35.647 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-6786.gf7420d7f. 112/153 2023-10-19T12:42:35.672 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-6786.gf7420d7f.el9.noa 113/153 2023-10-19T12:42:35.895 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-6786.gf7420d7f.el9.noa 113/153 2023-10-19T12:42:35.923 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-6786.gf7420d7f.el9.n 114/153 2023-10-19T12:42:36.888 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-6786.gf7420d7f.el9.n 114/153 2023-10-19T12:42:36.923 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-6786.gf74 115/153 2023-10-19T12:42:37.024 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-6786.gf74 115/153 2023-10-19T12:42:37.181 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-6786.gf7420d7f.el 116/153 2023-10-19T12:42:37.217 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-mgr-2:18.0.0-6786.gf7420d7f.el9.x86_64 117/153 2023-10-19T12:42:37.240 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-mgr-2:18.0.0-6786.gf7420d7f.el9.x86_64 117/153 2023-10-19T12:42:37.240 INFO:teuthology.orchestra.run.smithi179.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-19T12:42:37.240 INFO:teuthology.orchestra.run.smithi179.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-10-19T12:42:37.240 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-10-19T12:42:37.241 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-10-19T12:42:37.241 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-10-19T12:42:37.284 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-mgr-rook-2:18.0.0-6786.gf7420d7f.el9.noarch 118/153 2023-10-19T12:42:37.295 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-6786.gf7420d7f.el9.noarch 118/153 2023-10-19T12:42:38.070 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-2:18.0.0-6786.gf7420d7f.el9.x86_64 119/153 2023-10-19T12:42:38.111 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-radosgw-2:18.0.0-6786.gf7420d7f.el9.x86_64 120/153 2023-10-19T12:42:38.131 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-common-2:18.0.0-6786.gf7420d7f.el9.x86_64 103/153 2023-10-19T12:42:38.133 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-6786.gf7420d7f.el9.x86_64 120/153 2023-10-19T12:42:38.133 INFO:teuthology.orchestra.run.smithi179.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-19T12:42:38.133 INFO:teuthology.orchestra.run.smithi179.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-10-19T12:42:38.134 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-10-19T12:42:38.134 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-10-19T12:42:38.134 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-10-19T12:42:38.164 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-common-2:18.0.0-6786.gf7420d7f.el9.x86_64 103/153 2023-10-19T12:42:38.178 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-6786.gf7420 121/153 2023-10-19T12:42:38.200 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-6786.gf7420 121/153 2023-10-19T12:42:38.200 INFO:teuthology.orchestra.run.smithi179.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-19T12:42:38.200 INFO:teuthology.orchestra.run.smithi179.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-10-19T12:42:38.201 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-10-19T12:42:38.245 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : cephfs-mirror-2:18.0.0-6786.gf7420d7f.el9.x86_64 122/153 2023-10-19T12:42:38.267 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-6786.gf7420d7f.el9.x86_64 122/153 2023-10-19T12:42:38.268 INFO:teuthology.orchestra.run.smithi179.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-19T12:42:38.268 INFO:teuthology.orchestra.run.smithi179.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2023-10-19T12:42:38.268 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2023-10-19T12:42:38.268 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2023-10-19T12:42:38.269 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-10-19T12:42:38.554 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : rbd-mirror-2:18.0.0-6786.gf7420d7f.el9.x86_64 123/153 2023-10-19T12:42:38.576 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: rbd-mirror-2:18.0.0-6786.gf7420d7f.el9.x86_64 123/153 2023-10-19T12:42:38.576 INFO:teuthology.orchestra.run.smithi179.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-19T12:42:38.577 INFO:teuthology.orchestra.run.smithi179.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-10-19T12:42:38.577 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-10-19T12:42:38.577 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-10-19T12:42:38.577 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-10-19T12:42:38.735 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-common-2:18.0.0-6786.gf7420d7f.el9.x86_64 103/153 2023-10-19T12:42:38.768 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-base-2:18.0.0-6786.gf7420d7f.el9.x86_64 104/153 2023-10-19T12:42:38.858 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-base-2:18.0.0-6786.gf7420d7f.el9.x86_64 104/153 2023-10-19T12:42:38.858 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-10-19T12:42:38.858 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-10-19T12:42:38.858 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:42:38.894 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-selinux-2:18.0.0-6786.gf7420d7f.el9.x86_64 105/153 2023-10-19T12:42:42.374 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-test-2:18.0.0-6786.gf7420d7f.el9.x86_64 124/153 2023-10-19T12:42:42.415 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : rbd-fuse-2:18.0.0-6786.gf7420d7f.el9.x86_64 125/153 2023-10-19T12:42:42.446 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : rbd-nbd-2:18.0.0-6786.gf7420d7f.el9.x86_64 126/153 2023-10-19T12:42:42.636 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : elfutils-libelf-devel-0.189-3.el9.x86_64 127/153 2023-10-19T12:42:42.760 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 128/153 2023-10-19T12:42:42.815 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-fuse-2:18.0.0-6786.gf7420d7f.el9.x86_64 129/153 2023-10-19T12:42:42.854 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libtool-2.4.6-45.el9.x86_64 130/153 2023-10-19T12:42:42.896 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : xfsprogs-devel-5.19.0-4.el9.x86_64 131/153 2023-10-19T12:42:42.918 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libcephfs-devel-2:18.0.0-6786.gf7420d7f.el9.x86_ 132/153 2023-10-19T12:42:42.948 INFO:teuthology.orchestra.run.smithi179.stdout: Upgrading : NetworkManager-team-1:1.45.4-1.el9.x86_64 133/153 2023-10-19T12:42:43.021 INFO:teuthology.orchestra.run.smithi179.stdout: Upgrading : NetworkManager-tui-1:1.45.4-1.el9.x86_64 134/153 2023-10-19T12:42:43.100 INFO:teuthology.orchestra.run.smithi179.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 135/153 2023-10-19T12:42:43.138 INFO:teuthology.orchestra.run.smithi179.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 136/153 2023-10-19T12:42:43.759 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 137/153 2023-10-19T12:42:43.797 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 137/153 2023-10-19T12:42:43.816 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 137/153 2023-10-19T12:42:43.864 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : cephfs-top-2:18.0.0-6786.gf7420d7f.el9.noarch 138/153 2023-10-19T12:42:44.693 INFO:teuthology.orchestra.run.smithi179.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 139/153 2023-10-19T12:42:44.769 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : openssl-devel-1:3.0.7-24.el9.x86_64 140/153 2023-10-19T12:42:44.818 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 141/153 2023-10-19T12:42:44.916 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : flex-2.6.4-9.el9.x86_64 142/153 2023-10-19T12:42:44.947 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : bison-3.7.4-5.el9.x86_64 143/153 2023-10-19T12:42:44.973 INFO:teuthology.orchestra.run.smithi179.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 144/153 2023-10-19T12:42:44.973 INFO:teuthology.orchestra.run.smithi179.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 145/153 2023-10-19T12:42:45.066 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 145/153 2023-10-19T12:42:45.090 INFO:teuthology.orchestra.run.smithi179.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 146/153 2023-10-19T12:42:45.115 INFO:teuthology.orchestra.run.smithi179.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 147/153 2023-10-19T12:42:45.141 INFO:teuthology.orchestra.run.smithi179.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 148/153 2023-10-19T12:42:45.147 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 149/153 2023-10-19T12:42:45.147 INFO:teuthology.orchestra.run.smithi179.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 149/153 2023-10-19T12:42:45.335 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 149/153 2023-10-19T12:42:45.335 INFO:teuthology.orchestra.run.smithi179.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 150/153 2023-10-19T12:42:45.418 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 150/153 2023-10-19T12:42:45.444 INFO:teuthology.orchestra.run.smithi179.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 151/153 2023-10-19T12:42:45.444 INFO:teuthology.orchestra.run.smithi179.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 152/153 2023-10-19T12:42:45.537 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 152/153 2023-10-19T12:42:45.602 INFO:teuthology.orchestra.run.smithi179.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 153/153 2023-10-19T12:42:50.154 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: xfsdump-3.1.12-3.el9.x86_64 153/153 2023-10-19T12:42:50.155 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-2:18.0.0-6786.gf7420d7f.el9.x86_64 1/153 2023-10-19T12:42:50.155 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-base-2:18.0.0-6786.gf7420d7f.el9.x86_64 2/153 2023-10-19T12:42:50.155 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-common-2:18.0.0-6786.gf7420d7f.el9.x86_64 3/153 2023-10-19T12:42:50.155 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-fuse-2:18.0.0-6786.gf7420d7f.el9.x86_64 4/153 2023-10-19T12:42:50.156 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-6786.gf7420 5/153 2023-10-19T12:42:50.156 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-mds-2:18.0.0-6786.gf7420d7f.el9.x86_64 6/153 2023-10-19T12:42:50.156 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-mgr-2:18.0.0-6786.gf7420d7f.el9.x86_64 7/153 2023-10-19T12:42:50.156 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-mon-2:18.0.0-6786.gf7420d7f.el9.x86_64 8/153 2023-10-19T12:42:50.156 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-osd-2:18.0.0-6786.gf7420d7f.el9.x86_64 9/153 2023-10-19T12:42:50.157 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-radosgw-2:18.0.0-6786.gf7420d7f.el9.x86_64 10/153 2023-10-19T12:42:50.157 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-selinux-2:18.0.0-6786.gf7420d7f.el9.x86_64 11/153 2023-10-19T12:42:50.157 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-test-2:18.0.0-6786.gf7420d7f.el9.x86_64 12/153 2023-10-19T12:42:50.157 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : cephfs-mirror-2:18.0.0-6786.gf7420d7f.el9.x86_64 13/153 2023-10-19T12:42:50.157 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libcephfs-devel-2:18.0.0-6786.gf7420d7f.el9.x86_ 14/153 2023-10-19T12:42:50.158 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libcephfs2-2:18.0.0-6786.gf7420d7f.el9.x86_64 15/153 2023-10-19T12:42:50.158 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libcephsqlite-2:18.0.0-6786.gf7420d7f.el9.x86_64 16/153 2023-10-19T12:42:50.158 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : librados-devel-2:18.0.0-6786.gf7420d7f.el9.x86_6 17/153 2023-10-19T12:42:50.158 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libradosstriper1-2:18.0.0-6786.gf7420d7f.el9.x86 18/153 2023-10-19T12:42:50.158 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : librgw2-2:18.0.0-6786.gf7420d7f.el9.x86_64 19/153 2023-10-19T12:42:50.158 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-ceph-argparse-2:18.0.0-6786.gf7420d7f.el 20/153 2023-10-19T12:42:50.159 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-ceph-common-2:18.0.0-6786.gf7420d7f.el9. 21/153 2023-10-19T12:42:50.159 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-cephfs-2:18.0.0-6786.gf7420d7f.el9.x86_6 22/153 2023-10-19T12:42:50.159 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-rados-2:18.0.0-6786.gf7420d7f.el9.x86_64 23/153 2023-10-19T12:42:50.159 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-rbd-2:18.0.0-6786.gf7420d7f.el9.x86_64 24/153 2023-10-19T12:42:50.159 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-rgw-2:18.0.0-6786.gf7420d7f.el9.x86_64 25/153 2023-10-19T12:42:50.160 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : rbd-fuse-2:18.0.0-6786.gf7420d7f.el9.x86_64 26/153 2023-10-19T12:42:50.160 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : rbd-mirror-2:18.0.0-6786.gf7420d7f.el9.x86_64 27/153 2023-10-19T12:42:50.160 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : rbd-nbd-2:18.0.0-6786.gf7420d7f.el9.x86_64 28/153 2023-10-19T12:42:50.160 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-6786.gf7420d7f. 29/153 2023-10-19T12:42:50.160 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-6786.gf7420d7f.el9.noa 30/153 2023-10-19T12:42:50.161 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-6786.gf7420d7f.el9.n 31/153 2023-10-19T12:42:50.161 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-6786.gf74 32/153 2023-10-19T12:42:50.161 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-6786.gf7420d7f.el 33/153 2023-10-19T12:42:50.161 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-mgr-rook-2:18.0.0-6786.gf7420d7f.el9.noarch 34/153 2023-10-19T12:42:50.161 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-6786.gf7420d7f.e 35/153 2023-10-19T12:42:50.161 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-volume-2:18.0.0-6786.gf7420d7f.el9.noarch 36/153 2023-10-19T12:42:50.162 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : cephadm-2:18.0.0-6786.gf7420d7f.el9.noarch 37/153 2023-10-19T12:42:50.162 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : cephfs-top-2:18.0.0-6786.gf7420d7f.el9.noarch 38/153 2023-10-19T12:42:50.162 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : bison-3.7.4-5.el9.x86_64 39/153 2023-10-19T12:42:50.162 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 40/153 2023-10-19T12:42:50.162 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : flex-2.6.4-9.el9.x86_64 41/153 2023-10-19T12:42:50.163 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 42/153 2023-10-19T12:42:50.163 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 43/153 2023-10-19T12:42:50.163 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 44/153 2023-10-19T12:42:50.163 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 45/153 2023-10-19T12:42:50.163 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 46/153 2023-10-19T12:42:50.163 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 47/153 2023-10-19T12:42:50.164 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 48/153 2023-10-19T12:42:50.164 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 49/153 2023-10-19T12:42:50.164 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 50/153 2023-10-19T12:42:50.164 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 51/153 2023-10-19T12:42:50.164 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 52/153 2023-10-19T12:42:50.165 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 53/153 2023-10-19T12:42:50.165 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : postgresql-13.11-1.el9.x86_64 54/153 2023-10-19T12:42:50.165 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 55/153 2023-10-19T12:42:50.165 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 56/153 2023-10-19T12:42:50.165 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 57/153 2023-10-19T12:42:50.165 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 58/153 2023-10-19T12:42:50.166 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 59/153 2023-10-19T12:42:50.166 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 60/153 2023-10-19T12:42:50.166 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 61/153 2023-10-19T12:42:50.166 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 62/153 2023-10-19T12:42:50.166 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 63/153 2023-10-19T12:42:50.167 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 64/153 2023-10-19T12:42:50.167 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 65/153 2023-10-19T12:42:50.167 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 66/153 2023-10-19T12:42:50.167 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 67/153 2023-10-19T12:42:50.167 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 68/153 2023-10-19T12:42:50.168 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 69/153 2023-10-19T12:42:50.168 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 70/153 2023-10-19T12:42:50.168 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 71/153 2023-10-19T12:42:50.168 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 72/153 2023-10-19T12:42:50.168 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 73/153 2023-10-19T12:42:50.168 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 74/153 2023-10-19T12:42:50.169 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 75/153 2023-10-19T12:42:50.169 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 76/153 2023-10-19T12:42:50.169 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 77/153 2023-10-19T12:42:50.169 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 78/153 2023-10-19T12:42:50.169 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 79/153 2023-10-19T12:42:50.170 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 80/153 2023-10-19T12:42:50.170 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-ply-3.11-14.el9.noarch 81/153 2023-10-19T12:42:50.170 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 82/153 2023-10-19T12:42:50.170 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 83/153 2023-10-19T12:42:50.170 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-urllib3-1.26.5-3.el9.noarch 84/153 2023-10-19T12:42:50.171 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 85/153 2023-10-19T12:42:50.171 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : automake-1.16.2-8.el9.noarch 86/153 2023-10-19T12:42:50.171 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : elfutils-libelf-devel-0.189-3.el9.x86_64 87/153 2023-10-19T12:42:50.171 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : jq-1.6-15.el9.x86_64 88/153 2023-10-19T12:42:50.171 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 89/153 2023-10-19T12:42:50.171 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libuuid-devel-2.37.4-15.el9.x86_64 90/153 2023-10-19T12:42:50.172 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : openssl-devel-1:3.0.7-24.el9.x86_64 91/153 2023-10-19T12:42:50.172 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 92/153 2023-10-19T12:42:50.172 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : xfsprogs-devel-5.19.0-4.el9.x86_64 93/153 2023-10-19T12:42:50.172 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 94/153 2023-10-19T12:42:50.172 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 95/153 2023-10-19T12:42:50.173 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 96/153 2023-10-19T12:42:50.173 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libarrow-9.0.0-9.el9.x86_64 97/153 2023-10-19T12:42:50.173 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libarrow-doc-9.0.0-9.el9.noarch 98/153 2023-10-19T12:42:50.173 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 99/153 2023-10-19T12:42:50.173 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 100/153 2023-10-19T12:42:50.173 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : parquet-libs-9.0.0-9.el9.x86_64 101/153 2023-10-19T12:42:50.174 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 102/153 2023-10-19T12:42:50.174 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 103/153 2023-10-19T12:42:50.174 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 104/153 2023-10-19T12:42:50.174 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 105/153 2023-10-19T12:42:50.174 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 106/153 2023-10-19T12:42:50.175 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 107/153 2023-10-19T12:42:50.175 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-google-auth-1:2.23.3-1.el9.noarch 108/153 2023-10-19T12:42:50.175 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 109/153 2023-10-19T12:42:50.175 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 110/153 2023-10-19T12:42:50.175 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 111/153 2023-10-19T12:42:50.175 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 112/153 2023-10-19T12:42:50.176 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 113/153 2023-10-19T12:42:50.176 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 114/153 2023-10-19T12:42:50.176 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 115/153 2023-10-19T12:42:50.176 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 116/153 2023-10-19T12:42:50.176 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 117/153 2023-10-19T12:42:50.176 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 118/153 2023-10-19T12:42:50.177 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 119/153 2023-10-19T12:42:50.177 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 120/153 2023-10-19T12:42:50.177 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 121/153 2023-10-19T12:42:50.177 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 122/153 2023-10-19T12:42:50.177 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 123/153 2023-10-19T12:42:50.178 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 124/153 2023-10-19T12:42:50.178 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 125/153 2023-10-19T12:42:50.178 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 126/153 2023-10-19T12:42:50.178 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 127/153 2023-10-19T12:42:50.178 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 128/153 2023-10-19T12:42:50.178 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 129/153 2023-10-19T12:42:50.179 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 130/153 2023-10-19T12:42:50.179 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 131/153 2023-10-19T12:42:50.179 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : re2-1:20211101-3.el9.x86_64 132/153 2023-10-19T12:42:50.179 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 133/153 2023-10-19T12:42:50.179 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : librados2-2:18.0.0-6786.gf7420d7f.el9.x86_64 134/153 2023-10-19T12:42:50.180 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 135/153 2023-10-19T12:42:50.180 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : librbd1-2:18.0.0-6786.gf7420d7f.el9.x86_64 136/153 2023-10-19T12:42:50.180 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 137/153 2023-10-19T12:42:50.180 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 138/153 2023-10-19T12:42:50.180 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 139/153 2023-10-19T12:42:50.180 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : NetworkManager-1:1.45.4-1.el9.x86_64 140/153 2023-10-19T12:42:50.181 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 141/153 2023-10-19T12:42:50.181 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : NetworkManager-libnm-1:1.45.4-1.el9.x86_64 142/153 2023-10-19T12:42:50.181 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 143/153 2023-10-19T12:42:50.181 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : NetworkManager-team-1:1.45.4-1.el9.x86_64 144/153 2023-10-19T12:42:50.181 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 145/153 2023-10-19T12:42:50.181 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : NetworkManager-tui-1:1.45.4-1.el9.x86_64 146/153 2023-10-19T12:42:50.181 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 147/153 2023-10-19T12:42:50.182 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : acl-2.3.1-4.el9.x86_64 148/153 2023-10-19T12:42:50.182 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : acl-2.3.1-3.el9.x86_64 149/153 2023-10-19T12:42:50.182 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 150/153 2023-10-19T12:42:50.182 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 151/153 2023-10-19T12:42:50.182 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 152/153 2023-10-19T12:42:50.754 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-selinux-2:18.0.0-6786.gf7420d7f.el9.x86_64 105/153 2023-10-19T12:42:50.754 INFO:teuthology.orchestra.run.smithi057.stdout:skipping the directory /sys 2023-10-19T12:42:50.754 INFO:teuthology.orchestra.run.smithi057.stdout:skipping the directory /proc 2023-10-19T12:42:50.754 INFO:teuthology.orchestra.run.smithi057.stdout:skipping the directory /mnt 2023-10-19T12:42:50.755 INFO:teuthology.orchestra.run.smithi057.stdout:skipping the directory /var/tmp 2023-10-19T12:42:50.755 INFO:teuthology.orchestra.run.smithi057.stdout:skipping the directory /home 2023-10-19T12:42:50.755 INFO:teuthology.orchestra.run.smithi057.stdout:skipping the directory /root 2023-10-19T12:42:50.755 INFO:teuthology.orchestra.run.smithi057.stdout:skipping the directory /tmp 2023-10-19T12:42:50.755 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:42:50.920 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-volume-2:18.0.0-6786.gf7420d7f.el9.noarch 106/153 2023-10-19T12:42:50.936 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-volume-2:18.0.0-6786.gf7420d7f.el9.noarch 106/153 2023-10-19T12:42:50.936 INFO:teuthology.orchestra.run.smithi057.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-19T12:42:50.936 INFO:teuthology.orchestra.run.smithi057.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-10-19T12:42:50.937 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:42:52.556 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-osd-2:18.0.0-6786.gf7420d7f.el9.x86_64 107/153 2023-10-19T12:42:52.583 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-osd-2:18.0.0-6786.gf7420d7f.el9.x86_64 107/153 2023-10-19T12:42:52.583 INFO:teuthology.orchestra.run.smithi057.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-19T12:42:52.583 INFO:teuthology.orchestra.run.smithi057.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-10-19T12:42:52.583 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-10-19T12:42:52.583 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-10-19T12:42:52.584 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:42:52.749 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 153/153 2023-10-19T12:42:52.749 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-10-19T12:42:52.750 INFO:teuthology.orchestra.run.smithi179.stdout:Upgraded: 2023-10-19T12:42:52.750 INFO:teuthology.orchestra.run.smithi179.stdout: NetworkManager-1:1.45.4-1.el9.x86_64 2023-10-19T12:42:52.750 INFO:teuthology.orchestra.run.smithi179.stdout: NetworkManager-libnm-1:1.45.4-1.el9.x86_64 2023-10-19T12:42:52.750 INFO:teuthology.orchestra.run.smithi179.stdout: NetworkManager-team-1:1.45.4-1.el9.x86_64 2023-10-19T12:42:52.751 INFO:teuthology.orchestra.run.smithi179.stdout: NetworkManager-tui-1:1.45.4-1.el9.x86_64 2023-10-19T12:42:52.751 INFO:teuthology.orchestra.run.smithi179.stdout: acl-2.3.1-4.el9.x86_64 2023-10-19T12:42:52.751 INFO:teuthology.orchestra.run.smithi179.stdout: libacl-2.3.1-4.el9.x86_64 2023-10-19T12:42:52.751 INFO:teuthology.orchestra.run.smithi179.stdout: libacl-devel-2.3.1-4.el9.x86_64 2023-10-19T12:42:52.751 INFO:teuthology.orchestra.run.smithi179.stdout: librados2-2:18.0.0-6786.gf7420d7f.el9.x86_64 2023-10-19T12:42:52.752 INFO:teuthology.orchestra.run.smithi179.stdout: librbd1-2:18.0.0-6786.gf7420d7f.el9.x86_64 2023-10-19T12:42:52.752 INFO:teuthology.orchestra.run.smithi179.stdout: xfsdump-3.1.12-4.el9.x86_64 2023-10-19T12:42:52.752 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2023-10-19T12:42:52.752 INFO:teuthology.orchestra.run.smithi179.stdout: automake-1.16.2-8.el9.noarch 2023-10-19T12:42:52.752 INFO:teuthology.orchestra.run.smithi179.stdout: bison-3.7.4-5.el9.x86_64 2023-10-19T12:42:52.753 INFO:teuthology.orchestra.run.smithi179.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-10-19T12:42:52.753 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-2:18.0.0-6786.gf7420d7f.el9.x86_64 2023-10-19T12:42:52.753 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-base-2:18.0.0-6786.gf7420d7f.el9.x86_64 2023-10-19T12:42:52.753 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-common-2:18.0.0-6786.gf7420d7f.el9.x86_64 2023-10-19T12:42:52.753 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-fuse-2:18.0.0-6786.gf7420d7f.el9.x86_64 2023-10-19T12:42:52.754 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-grafana-dashboards-2:18.0.0-6786.gf7420d7f.el9.noarch 2023-10-19T12:42:52.754 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-immutable-object-cache-2:18.0.0-6786.gf7420d7f.el9.x86_64 2023-10-19T12:42:52.754 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mds-2:18.0.0-6786.gf7420d7f.el9.x86_64 2023-10-19T12:42:52.754 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-2:18.0.0-6786.gf7420d7f.el9.x86_64 2023-10-19T12:42:52.754 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-cephadm-2:18.0.0-6786.gf7420d7f.el9.noarch 2023-10-19T12:42:52.755 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-dashboard-2:18.0.0-6786.gf7420d7f.el9.noarch 2023-10-19T12:42:52.755 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-diskprediction-local-2:18.0.0-6786.gf7420d7f.el9.noarch 2023-10-19T12:42:52.755 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-modules-core-2:18.0.0-6786.gf7420d7f.el9.noarch 2023-10-19T12:42:52.755 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-rook-2:18.0.0-6786.gf7420d7f.el9.noarch 2023-10-19T12:42:52.755 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mon-2:18.0.0-6786.gf7420d7f.el9.x86_64 2023-10-19T12:42:52.756 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-osd-2:18.0.0-6786.gf7420d7f.el9.x86_64 2023-10-19T12:42:52.756 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-prometheus-alerts-2:18.0.0-6786.gf7420d7f.el9.noarch 2023-10-19T12:42:52.756 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-radosgw-2:18.0.0-6786.gf7420d7f.el9.x86_64 2023-10-19T12:42:52.756 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-selinux-2:18.0.0-6786.gf7420d7f.el9.x86_64 2023-10-19T12:42:52.756 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-test-2:18.0.0-6786.gf7420d7f.el9.x86_64 2023-10-19T12:42:52.757 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-volume-2:18.0.0-6786.gf7420d7f.el9.noarch 2023-10-19T12:42:52.757 INFO:teuthology.orchestra.run.smithi179.stdout: cephadm-2:18.0.0-6786.gf7420d7f.el9.noarch 2023-10-19T12:42:52.757 INFO:teuthology.orchestra.run.smithi179.stdout: cephfs-mirror-2:18.0.0-6786.gf7420d7f.el9.x86_64 2023-10-19T12:42:52.757 INFO:teuthology.orchestra.run.smithi179.stdout: cephfs-top-2:18.0.0-6786.gf7420d7f.el9.noarch 2023-10-19T12:42:52.757 INFO:teuthology.orchestra.run.smithi179.stdout: elfutils-libelf-devel-0.189-3.el9.x86_64 2023-10-19T12:42:52.758 INFO:teuthology.orchestra.run.smithi179.stdout: flex-2.6.4-9.el9.x86_64 2023-10-19T12:42:52.758 INFO:teuthology.orchestra.run.smithi179.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-10-19T12:42:52.758 INFO:teuthology.orchestra.run.smithi179.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-10-19T12:42:52.758 INFO:teuthology.orchestra.run.smithi179.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-10-19T12:42:52.758 INFO:teuthology.orchestra.run.smithi179.stdout: fmt-8.1.1-5.el9.x86_64 2023-10-19T12:42:52.759 INFO:teuthology.orchestra.run.smithi179.stdout: fuse-2.9.9-15.el9.x86_64 2023-10-19T12:42:52.759 INFO:teuthology.orchestra.run.smithi179.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-10-19T12:42:52.759 INFO:teuthology.orchestra.run.smithi179.stdout: jq-1.6-15.el9.x86_64 2023-10-19T12:42:52.759 INFO:teuthology.orchestra.run.smithi179.stdout: libaio-devel-0.3.111-13.el9.x86_64 2023-10-19T12:42:52.759 INFO:teuthology.orchestra.run.smithi179.stdout: libarrow-9.0.0-9.el9.x86_64 2023-10-19T12:42:52.760 INFO:teuthology.orchestra.run.smithi179.stdout: libarrow-doc-9.0.0-9.el9.noarch 2023-10-19T12:42:52.760 INFO:teuthology.orchestra.run.smithi179.stdout: libcephfs-devel-2:18.0.0-6786.gf7420d7f.el9.x86_64 2023-10-19T12:42:52.760 INFO:teuthology.orchestra.run.smithi179.stdout: libcephfs2-2:18.0.0-6786.gf7420d7f.el9.x86_64 2023-10-19T12:42:52.760 INFO:teuthology.orchestra.run.smithi179.stdout: libcephsqlite-2:18.0.0-6786.gf7420d7f.el9.x86_64 2023-10-19T12:42:52.760 INFO:teuthology.orchestra.run.smithi179.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-10-19T12:42:52.761 INFO:teuthology.orchestra.run.smithi179.stdout: liboath-2.6.7-2.el9.x86_64 2023-10-19T12:42:52.761 INFO:teuthology.orchestra.run.smithi179.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-10-19T12:42:52.761 INFO:teuthology.orchestra.run.smithi179.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-10-19T12:42:52.761 INFO:teuthology.orchestra.run.smithi179.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-10-19T12:42:52.761 INFO:teuthology.orchestra.run.smithi179.stdout: librados-devel-2:18.0.0-6786.gf7420d7f.el9.x86_64 2023-10-19T12:42:52.761 INFO:teuthology.orchestra.run.smithi179.stdout: libradosstriper1-2:18.0.0-6786.gf7420d7f.el9.x86_64 2023-10-19T12:42:52.762 INFO:teuthology.orchestra.run.smithi179.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-10-19T12:42:52.762 INFO:teuthology.orchestra.run.smithi179.stdout: librgw2-2:18.0.0-6786.gf7420d7f.el9.x86_64 2023-10-19T12:42:52.762 INFO:teuthology.orchestra.run.smithi179.stdout: libtool-2.4.6-45.el9.x86_64 2023-10-19T12:42:52.762 INFO:teuthology.orchestra.run.smithi179.stdout: libunwind-1.6.2-1.el9.x86_64 2023-10-19T12:42:52.762 INFO:teuthology.orchestra.run.smithi179.stdout: libuuid-devel-2.37.4-15.el9.x86_64 2023-10-19T12:42:52.763 INFO:teuthology.orchestra.run.smithi179.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2023-10-19T12:42:52.763 INFO:teuthology.orchestra.run.smithi179.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-10-19T12:42:52.763 INFO:teuthology.orchestra.run.smithi179.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-10-19T12:42:52.763 INFO:teuthology.orchestra.run.smithi179.stdout: openblas-0.3.21-2.el9.x86_64 2023-10-19T12:42:52.763 INFO:teuthology.orchestra.run.smithi179.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-10-19T12:42:52.764 INFO:teuthology.orchestra.run.smithi179.stdout: openssl-devel-1:3.0.7-24.el9.x86_64 2023-10-19T12:42:52.764 INFO:teuthology.orchestra.run.smithi179.stdout: parquet-libs-9.0.0-9.el9.x86_64 2023-10-19T12:42:52.764 INFO:teuthology.orchestra.run.smithi179.stdout: postgresql-13.11-1.el9.x86_64 2023-10-19T12:42:52.764 INFO:teuthology.orchestra.run.smithi179.stdout: postgresql-contrib-13.11-1.el9.x86_64 2023-10-19T12:42:52.764 INFO:teuthology.orchestra.run.smithi179.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2023-10-19T12:42:52.765 INFO:teuthology.orchestra.run.smithi179.stdout: postgresql-server-13.11-1.el9.x86_64 2023-10-19T12:42:52.765 INFO:teuthology.orchestra.run.smithi179.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-10-19T12:42:52.765 INFO:teuthology.orchestra.run.smithi179.stdout: python3-babel-2.9.1-2.el9.noarch 2023-10-19T12:42:52.765 INFO:teuthology.orchestra.run.smithi179.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-10-19T12:42:52.765 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-10-19T12:42:52.766 INFO:teuthology.orchestra.run.smithi179.stdout: python3-ceph-argparse-2:18.0.0-6786.gf7420d7f.el9.x86_64 2023-10-19T12:42:52.766 INFO:teuthology.orchestra.run.smithi179.stdout: python3-ceph-common-2:18.0.0-6786.gf7420d7f.el9.x86_64 2023-10-19T12:42:52.766 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cephfs-2:18.0.0-6786.gf7420d7f.el9.x86_64 2023-10-19T12:42:52.766 INFO:teuthology.orchestra.run.smithi179.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-10-19T12:42:52.766 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-10-19T12:42:52.767 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-10-19T12:42:52.767 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-10-19T12:42:52.767 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-10-19T12:42:52.767 INFO:teuthology.orchestra.run.smithi179.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-10-19T12:42:52.767 INFO:teuthology.orchestra.run.smithi179.stdout: python3-google-auth-1:2.23.3-1.el9.noarch 2023-10-19T12:42:52.767 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-10-19T12:42:52.768 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-10-19T12:42:52.768 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-10-19T12:42:52.768 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-10-19T12:42:52.768 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-10-19T12:42:52.768 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-10-19T12:42:52.769 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-10-19T12:42:52.769 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-10-19T12:42:52.769 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-10-19T12:42:52.769 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-10-19T12:42:52.769 INFO:teuthology.orchestra.run.smithi179.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-10-19T12:42:52.769 INFO:teuthology.orchestra.run.smithi179.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-10-19T12:42:52.770 INFO:teuthology.orchestra.run.smithi179.stdout: python3-mako-1.1.4-6.el9.noarch 2023-10-19T12:42:52.770 INFO:teuthology.orchestra.run.smithi179.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-10-19T12:42:52.770 INFO:teuthology.orchestra.run.smithi179.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-10-19T12:42:52.770 INFO:teuthology.orchestra.run.smithi179.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-10-19T12:42:52.771 INFO:teuthology.orchestra.run.smithi179.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-10-19T12:42:52.771 INFO:teuthology.orchestra.run.smithi179.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-10-19T12:42:52.771 INFO:teuthology.orchestra.run.smithi179.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-10-19T12:42:52.771 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-10-19T12:42:52.771 INFO:teuthology.orchestra.run.smithi179.stdout: python3-ply-3.11-14.el9.noarch 2023-10-19T12:42:52.772 INFO:teuthology.orchestra.run.smithi179.stdout: python3-portend-3.1.0-2.el9.noarch 2023-10-19T12:42:52.772 INFO:teuthology.orchestra.run.smithi179.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-10-19T12:42:52.772 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-10-19T12:42:52.772 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-10-19T12:42:52.772 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-10-19T12:42:52.773 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pycparser-2.20-6.el9.noarch 2023-10-19T12:42:52.773 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pytz-2021.1-5.el9.noarch 2023-10-19T12:42:52.773 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rados-2:18.0.0-6786.gf7420d7f.el9.x86_64 2023-10-19T12:42:52.773 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rbd-2:18.0.0-6786.gf7420d7f.el9.x86_64 2023-10-19T12:42:52.773 INFO:teuthology.orchestra.run.smithi179.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-10-19T12:42:52.773 INFO:teuthology.orchestra.run.smithi179.stdout: python3-requests-2.25.1-7.el9.noarch 2023-10-19T12:42:52.774 INFO:teuthology.orchestra.run.smithi179.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-10-19T12:42:52.774 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rgw-2:18.0.0-6786.gf7420d7f.el9.x86_64 2023-10-19T12:42:52.774 INFO:teuthology.orchestra.run.smithi179.stdout: python3-routes-2.5.1-5.el9.noarch 2023-10-19T12:42:52.774 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rsa-4.9-2.el9.noarch 2023-10-19T12:42:52.774 INFO:teuthology.orchestra.run.smithi179.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-10-19T12:42:52.775 INFO:teuthology.orchestra.run.smithi179.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-10-19T12:42:52.775 INFO:teuthology.orchestra.run.smithi179.stdout: python3-toml-0.10.2-6.el9.noarch 2023-10-19T12:42:52.775 INFO:teuthology.orchestra.run.smithi179.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-10-19T12:42:52.775 INFO:teuthology.orchestra.run.smithi179.stdout: python3-urllib3-1.26.5-3.el9.noarch 2023-10-19T12:42:52.775 INFO:teuthology.orchestra.run.smithi179.stdout: python3-webob-1.8.7-6.el9.noarch 2023-10-19T12:42:52.775 INFO:teuthology.orchestra.run.smithi179.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-10-19T12:42:52.776 INFO:teuthology.orchestra.run.smithi179.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-10-19T12:42:52.776 INFO:teuthology.orchestra.run.smithi179.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-10-19T12:42:52.776 INFO:teuthology.orchestra.run.smithi179.stdout: rbd-fuse-2:18.0.0-6786.gf7420d7f.el9.x86_64 2023-10-19T12:42:52.776 INFO:teuthology.orchestra.run.smithi179.stdout: rbd-mirror-2:18.0.0-6786.gf7420d7f.el9.x86_64 2023-10-19T12:42:52.776 INFO:teuthology.orchestra.run.smithi179.stdout: rbd-nbd-2:18.0.0-6786.gf7420d7f.el9.x86_64 2023-10-19T12:42:52.777 INFO:teuthology.orchestra.run.smithi179.stdout: re2-1:20211101-3.el9.x86_64 2023-10-19T12:42:52.777 INFO:teuthology.orchestra.run.smithi179.stdout: socat-1.7.4.1-5.el9.x86_64 2023-10-19T12:42:52.777 INFO:teuthology.orchestra.run.smithi179.stdout: thrift-0.15.0-2.el9.x86_64 2023-10-19T12:42:52.777 INFO:teuthology.orchestra.run.smithi179.stdout: uuid-1.6.2-55.el9.x86_64 2023-10-19T12:42:52.777 INFO:teuthology.orchestra.run.smithi179.stdout: xfsprogs-devel-5.19.0-4.el9.x86_64 2023-10-19T12:42:52.777 INFO:teuthology.orchestra.run.smithi179.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-10-19T12:42:52.778 INFO:teuthology.orchestra.run.smithi179.stdout: zlib-devel-1.2.11-40.el9.x86_64 2023-10-19T12:42:52.778 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-10-19T12:42:52.779 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2023-10-19T12:42:52.790 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-mds-2:18.0.0-6786.gf7420d7f.el9.x86_64 108/153 2023-10-19T12:42:52.814 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-mds-2:18.0.0-6786.gf7420d7f.el9.x86_64 108/153 2023-10-19T12:42:52.814 INFO:teuthology.orchestra.run.smithi057.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-19T12:42:52.814 INFO:teuthology.orchestra.run.smithi057.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-10-19T12:42:52.814 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-10-19T12:42:52.815 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-10-19T12:42:52.815 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:42:53.141 DEBUG:teuthology.parallel:result is None 2023-10-19T12:42:53.299 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-mon-2:18.0.0-6786.gf7420d7f.el9.x86_64 109/153 2023-10-19T12:42:53.322 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-mon-2:18.0.0-6786.gf7420d7f.el9.x86_64 109/153 2023-10-19T12:42:53.322 INFO:teuthology.orchestra.run.smithi057.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-19T12:42:53.322 INFO:teuthology.orchestra.run.smithi057.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-10-19T12:42:53.323 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-10-19T12:42:53.323 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-10-19T12:42:53.323 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:42:53.749 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: cephadm-2:18.0.0-6786.gf7420d7f.el9.noarch 110/153 2023-10-19T12:42:53.781 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : cephadm-2:18.0.0-6786.gf7420d7f.el9.noarch 110/153 2023-10-19T12:42:53.810 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-6786.gf7420d7f.e 111/153 2023-10-19T12:42:53.874 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-6786.gf7420d7f. 112/153 2023-10-19T12:42:53.904 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-6786.gf7420d7f.el9.noa 113/153 2023-10-19T12:42:54.138 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-6786.gf7420d7f.el9.noa 113/153 2023-10-19T12:42:54.173 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-6786.gf7420d7f.el9.n 114/153 2023-10-19T12:42:55.190 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-6786.gf7420d7f.el9.n 114/153 2023-10-19T12:42:55.213 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-6786.gf74 115/153 2023-10-19T12:42:55.316 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-6786.gf74 115/153 2023-10-19T12:42:55.478 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-6786.gf7420d7f.el 116/153 2023-10-19T12:42:55.507 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-mgr-2:18.0.0-6786.gf7420d7f.el9.x86_64 117/153 2023-10-19T12:42:55.531 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-mgr-2:18.0.0-6786.gf7420d7f.el9.x86_64 117/153 2023-10-19T12:42:55.532 INFO:teuthology.orchestra.run.smithi057.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-19T12:42:55.532 INFO:teuthology.orchestra.run.smithi057.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-10-19T12:42:55.532 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-10-19T12:42:55.532 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-10-19T12:42:55.533 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:42:55.574 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-mgr-rook-2:18.0.0-6786.gf7420d7f.el9.noarch 118/153 2023-10-19T12:42:55.585 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-6786.gf7420d7f.el9.noarch 118/153 2023-10-19T12:42:56.410 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-2:18.0.0-6786.gf7420d7f.el9.x86_64 119/153 2023-10-19T12:42:56.442 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-radosgw-2:18.0.0-6786.gf7420d7f.el9.x86_64 120/153 2023-10-19T12:42:56.465 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-6786.gf7420d7f.el9.x86_64 120/153 2023-10-19T12:42:56.466 INFO:teuthology.orchestra.run.smithi057.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-19T12:42:56.466 INFO:teuthology.orchestra.run.smithi057.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-10-19T12:42:56.466 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-10-19T12:42:56.466 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-10-19T12:42:56.467 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:42:56.509 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-6786.gf7420 121/153 2023-10-19T12:42:56.532 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-6786.gf7420 121/153 2023-10-19T12:42:56.533 INFO:teuthology.orchestra.run.smithi057.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-19T12:42:56.533 INFO:teuthology.orchestra.run.smithi057.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-10-19T12:42:56.533 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:42:56.576 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : cephfs-mirror-2:18.0.0-6786.gf7420d7f.el9.x86_64 122/153 2023-10-19T12:42:56.600 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-6786.gf7420d7f.el9.x86_64 122/153 2023-10-19T12:42:56.600 INFO:teuthology.orchestra.run.smithi057.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-19T12:42:56.601 INFO:teuthology.orchestra.run.smithi057.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2023-10-19T12:42:56.601 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2023-10-19T12:42:56.601 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2023-10-19T12:42:56.601 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:42:56.919 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : rbd-mirror-2:18.0.0-6786.gf7420d7f.el9.x86_64 123/153 2023-10-19T12:42:56.942 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: rbd-mirror-2:18.0.0-6786.gf7420d7f.el9.x86_64 123/153 2023-10-19T12:42:56.943 INFO:teuthology.orchestra.run.smithi057.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-19T12:42:56.943 INFO:teuthology.orchestra.run.smithi057.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-10-19T12:42:56.943 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-10-19T12:42:56.943 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-10-19T12:42:56.943 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:43:00.980 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-test-2:18.0.0-6786.gf7420d7f.el9.x86_64 124/153 2023-10-19T12:43:01.022 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : rbd-fuse-2:18.0.0-6786.gf7420d7f.el9.x86_64 125/153 2023-10-19T12:43:01.052 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : rbd-nbd-2:18.0.0-6786.gf7420d7f.el9.x86_64 126/153 2023-10-19T12:43:01.243 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : elfutils-libelf-devel-0.189-3.el9.x86_64 127/153 2023-10-19T12:43:01.362 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 128/153 2023-10-19T12:43:01.426 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-fuse-2:18.0.0-6786.gf7420d7f.el9.x86_64 129/153 2023-10-19T12:43:01.469 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libtool-2.4.6-45.el9.x86_64 130/153 2023-10-19T12:43:01.501 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : xfsprogs-devel-5.19.0-4.el9.x86_64 131/153 2023-10-19T12:43:01.532 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libcephfs-devel-2:18.0.0-6786.gf7420d7f.el9.x86_ 132/153 2023-10-19T12:43:01.562 INFO:teuthology.orchestra.run.smithi057.stdout: Upgrading : NetworkManager-team-1:1.45.4-1.el9.x86_64 133/153 2023-10-19T12:43:01.633 INFO:teuthology.orchestra.run.smithi057.stdout: Upgrading : NetworkManager-tui-1:1.45.4-1.el9.x86_64 134/153 2023-10-19T12:43:01.697 INFO:teuthology.orchestra.run.smithi057.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 135/153 2023-10-19T12:43:01.735 INFO:teuthology.orchestra.run.smithi057.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 136/153 2023-10-19T12:43:02.394 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 137/153 2023-10-19T12:43:02.425 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 137/153 2023-10-19T12:43:02.446 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 137/153 2023-10-19T12:43:02.495 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : cephfs-top-2:18.0.0-6786.gf7420d7f.el9.noarch 138/153 2023-10-19T12:43:03.418 INFO:teuthology.orchestra.run.smithi057.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 139/153 2023-10-19T12:43:03.504 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : openssl-devel-1:3.0.7-24.el9.x86_64 140/153 2023-10-19T12:43:03.547 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 141/153 2023-10-19T12:43:03.645 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : flex-2.6.4-9.el9.x86_64 142/153 2023-10-19T12:43:03.674 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : bison-3.7.4-5.el9.x86_64 143/153 2023-10-19T12:43:03.700 INFO:teuthology.orchestra.run.smithi057.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 144/153 2023-10-19T12:43:03.700 INFO:teuthology.orchestra.run.smithi057.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 145/153 2023-10-19T12:43:03.768 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 145/153 2023-10-19T12:43:03.792 INFO:teuthology.orchestra.run.smithi057.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 146/153 2023-10-19T12:43:03.817 INFO:teuthology.orchestra.run.smithi057.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 147/153 2023-10-19T12:43:03.843 INFO:teuthology.orchestra.run.smithi057.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 148/153 2023-10-19T12:43:03.849 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 149/153 2023-10-19T12:43:03.850 INFO:teuthology.orchestra.run.smithi057.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 149/153 2023-10-19T12:43:04.037 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 149/153 2023-10-19T12:43:04.038 INFO:teuthology.orchestra.run.smithi057.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 150/153 2023-10-19T12:43:04.120 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 150/153 2023-10-19T12:43:04.144 INFO:teuthology.orchestra.run.smithi057.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 151/153 2023-10-19T12:43:04.145 INFO:teuthology.orchestra.run.smithi057.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 152/153 2023-10-19T12:43:04.233 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 152/153 2023-10-19T12:43:04.306 INFO:teuthology.orchestra.run.smithi057.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 153/153 2023-10-19T12:43:08.737 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: xfsdump-3.1.12-3.el9.x86_64 153/153 2023-10-19T12:43:08.737 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-2:18.0.0-6786.gf7420d7f.el9.x86_64 1/153 2023-10-19T12:43:08.737 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-base-2:18.0.0-6786.gf7420d7f.el9.x86_64 2/153 2023-10-19T12:43:08.738 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-common-2:18.0.0-6786.gf7420d7f.el9.x86_64 3/153 2023-10-19T12:43:08.738 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-fuse-2:18.0.0-6786.gf7420d7f.el9.x86_64 4/153 2023-10-19T12:43:08.738 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-6786.gf7420 5/153 2023-10-19T12:43:08.738 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-mds-2:18.0.0-6786.gf7420d7f.el9.x86_64 6/153 2023-10-19T12:43:08.738 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-mgr-2:18.0.0-6786.gf7420d7f.el9.x86_64 7/153 2023-10-19T12:43:08.738 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-mon-2:18.0.0-6786.gf7420d7f.el9.x86_64 8/153 2023-10-19T12:43:08.739 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-osd-2:18.0.0-6786.gf7420d7f.el9.x86_64 9/153 2023-10-19T12:43:08.739 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-radosgw-2:18.0.0-6786.gf7420d7f.el9.x86_64 10/153 2023-10-19T12:43:08.739 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-selinux-2:18.0.0-6786.gf7420d7f.el9.x86_64 11/153 2023-10-19T12:43:08.739 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-test-2:18.0.0-6786.gf7420d7f.el9.x86_64 12/153 2023-10-19T12:43:08.739 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : cephfs-mirror-2:18.0.0-6786.gf7420d7f.el9.x86_64 13/153 2023-10-19T12:43:08.740 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libcephfs-devel-2:18.0.0-6786.gf7420d7f.el9.x86_ 14/153 2023-10-19T12:43:08.740 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libcephfs2-2:18.0.0-6786.gf7420d7f.el9.x86_64 15/153 2023-10-19T12:43:08.740 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libcephsqlite-2:18.0.0-6786.gf7420d7f.el9.x86_64 16/153 2023-10-19T12:43:08.740 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : librados-devel-2:18.0.0-6786.gf7420d7f.el9.x86_6 17/153 2023-10-19T12:43:08.740 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libradosstriper1-2:18.0.0-6786.gf7420d7f.el9.x86 18/153 2023-10-19T12:43:08.740 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : librgw2-2:18.0.0-6786.gf7420d7f.el9.x86_64 19/153 2023-10-19T12:43:08.741 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-ceph-argparse-2:18.0.0-6786.gf7420d7f.el 20/153 2023-10-19T12:43:08.741 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-ceph-common-2:18.0.0-6786.gf7420d7f.el9. 21/153 2023-10-19T12:43:08.741 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-cephfs-2:18.0.0-6786.gf7420d7f.el9.x86_6 22/153 2023-10-19T12:43:08.741 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-rados-2:18.0.0-6786.gf7420d7f.el9.x86_64 23/153 2023-10-19T12:43:08.741 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-rbd-2:18.0.0-6786.gf7420d7f.el9.x86_64 24/153 2023-10-19T12:43:08.741 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-rgw-2:18.0.0-6786.gf7420d7f.el9.x86_64 25/153 2023-10-19T12:43:08.742 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : rbd-fuse-2:18.0.0-6786.gf7420d7f.el9.x86_64 26/153 2023-10-19T12:43:08.742 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : rbd-mirror-2:18.0.0-6786.gf7420d7f.el9.x86_64 27/153 2023-10-19T12:43:08.742 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : rbd-nbd-2:18.0.0-6786.gf7420d7f.el9.x86_64 28/153 2023-10-19T12:43:08.742 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-6786.gf7420d7f. 29/153 2023-10-19T12:43:08.742 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-6786.gf7420d7f.el9.noa 30/153 2023-10-19T12:43:08.743 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-6786.gf7420d7f.el9.n 31/153 2023-10-19T12:43:08.743 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-6786.gf74 32/153 2023-10-19T12:43:08.743 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-6786.gf7420d7f.el 33/153 2023-10-19T12:43:08.743 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-mgr-rook-2:18.0.0-6786.gf7420d7f.el9.noarch 34/153 2023-10-19T12:43:08.743 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-6786.gf7420d7f.e 35/153 2023-10-19T12:43:08.743 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-volume-2:18.0.0-6786.gf7420d7f.el9.noarch 36/153 2023-10-19T12:43:08.744 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : cephadm-2:18.0.0-6786.gf7420d7f.el9.noarch 37/153 2023-10-19T12:43:08.744 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : cephfs-top-2:18.0.0-6786.gf7420d7f.el9.noarch 38/153 2023-10-19T12:43:08.744 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : bison-3.7.4-5.el9.x86_64 39/153 2023-10-19T12:43:08.744 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 40/153 2023-10-19T12:43:08.744 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : flex-2.6.4-9.el9.x86_64 41/153 2023-10-19T12:43:08.744 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 42/153 2023-10-19T12:43:08.745 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 43/153 2023-10-19T12:43:08.745 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 44/153 2023-10-19T12:43:08.745 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 45/153 2023-10-19T12:43:08.745 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 46/153 2023-10-19T12:43:08.745 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 47/153 2023-10-19T12:43:08.746 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 48/153 2023-10-19T12:43:08.746 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 49/153 2023-10-19T12:43:08.746 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 50/153 2023-10-19T12:43:08.746 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 51/153 2023-10-19T12:43:08.746 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 52/153 2023-10-19T12:43:08.746 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 53/153 2023-10-19T12:43:08.747 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : postgresql-13.11-1.el9.x86_64 54/153 2023-10-19T12:43:08.747 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 55/153 2023-10-19T12:43:08.747 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 56/153 2023-10-19T12:43:08.747 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 57/153 2023-10-19T12:43:08.747 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 58/153 2023-10-19T12:43:08.748 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 59/153 2023-10-19T12:43:08.748 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 60/153 2023-10-19T12:43:08.748 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 61/153 2023-10-19T12:43:08.748 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 62/153 2023-10-19T12:43:08.748 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 63/153 2023-10-19T12:43:08.749 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 64/153 2023-10-19T12:43:08.749 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 65/153 2023-10-19T12:43:08.749 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 66/153 2023-10-19T12:43:08.749 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 67/153 2023-10-19T12:43:08.749 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 68/153 2023-10-19T12:43:08.749 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 69/153 2023-10-19T12:43:08.750 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 70/153 2023-10-19T12:43:08.750 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 71/153 2023-10-19T12:43:08.750 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 72/153 2023-10-19T12:43:08.750 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 73/153 2023-10-19T12:43:08.750 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 74/153 2023-10-19T12:43:08.751 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 75/153 2023-10-19T12:43:08.751 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 76/153 2023-10-19T12:43:08.751 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 77/153 2023-10-19T12:43:08.751 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 78/153 2023-10-19T12:43:08.751 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 79/153 2023-10-19T12:43:08.752 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 80/153 2023-10-19T12:43:08.752 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-ply-3.11-14.el9.noarch 81/153 2023-10-19T12:43:08.752 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 82/153 2023-10-19T12:43:08.752 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 83/153 2023-10-19T12:43:08.752 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-urllib3-1.26.5-3.el9.noarch 84/153 2023-10-19T12:43:08.752 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 85/153 2023-10-19T12:43:08.753 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : automake-1.16.2-8.el9.noarch 86/153 2023-10-19T12:43:08.753 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : elfutils-libelf-devel-0.189-3.el9.x86_64 87/153 2023-10-19T12:43:08.753 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : jq-1.6-15.el9.x86_64 88/153 2023-10-19T12:43:08.753 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 89/153 2023-10-19T12:43:08.753 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libuuid-devel-2.37.4-15.el9.x86_64 90/153 2023-10-19T12:43:08.754 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : openssl-devel-1:3.0.7-24.el9.x86_64 91/153 2023-10-19T12:43:08.754 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 92/153 2023-10-19T12:43:08.754 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : xfsprogs-devel-5.19.0-4.el9.x86_64 93/153 2023-10-19T12:43:08.754 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 94/153 2023-10-19T12:43:08.754 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 95/153 2023-10-19T12:43:08.755 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 96/153 2023-10-19T12:43:08.755 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libarrow-9.0.0-9.el9.x86_64 97/153 2023-10-19T12:43:08.755 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libarrow-doc-9.0.0-9.el9.noarch 98/153 2023-10-19T12:43:08.755 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 99/153 2023-10-19T12:43:08.755 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 100/153 2023-10-19T12:43:08.756 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : parquet-libs-9.0.0-9.el9.x86_64 101/153 2023-10-19T12:43:08.756 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 102/153 2023-10-19T12:43:08.756 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 103/153 2023-10-19T12:43:08.756 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 104/153 2023-10-19T12:43:08.756 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 105/153 2023-10-19T12:43:08.756 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 106/153 2023-10-19T12:43:08.757 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 107/153 2023-10-19T12:43:08.757 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-google-auth-1:2.23.3-1.el9.noarch 108/153 2023-10-19T12:43:08.757 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 109/153 2023-10-19T12:43:08.757 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 110/153 2023-10-19T12:43:08.757 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 111/153 2023-10-19T12:43:08.757 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 112/153 2023-10-19T12:43:08.758 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 113/153 2023-10-19T12:43:08.758 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 114/153 2023-10-19T12:43:08.758 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 115/153 2023-10-19T12:43:08.758 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 116/153 2023-10-19T12:43:08.758 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 117/153 2023-10-19T12:43:08.759 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 118/153 2023-10-19T12:43:08.759 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 119/153 2023-10-19T12:43:08.759 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 120/153 2023-10-19T12:43:08.759 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 121/153 2023-10-19T12:43:08.759 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 122/153 2023-10-19T12:43:08.760 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 123/153 2023-10-19T12:43:08.760 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 124/153 2023-10-19T12:43:08.760 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 125/153 2023-10-19T12:43:08.760 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 126/153 2023-10-19T12:43:08.760 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 127/153 2023-10-19T12:43:08.761 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 128/153 2023-10-19T12:43:08.761 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 129/153 2023-10-19T12:43:08.761 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 130/153 2023-10-19T12:43:08.761 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 131/153 2023-10-19T12:43:08.761 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : re2-1:20211101-3.el9.x86_64 132/153 2023-10-19T12:43:08.761 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 133/153 2023-10-19T12:43:08.762 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : librados2-2:18.0.0-6786.gf7420d7f.el9.x86_64 134/153 2023-10-19T12:43:08.762 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 135/153 2023-10-19T12:43:08.762 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : librbd1-2:18.0.0-6786.gf7420d7f.el9.x86_64 136/153 2023-10-19T12:43:08.762 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 137/153 2023-10-19T12:43:08.762 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 138/153 2023-10-19T12:43:08.763 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 139/153 2023-10-19T12:43:08.763 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : NetworkManager-1:1.45.4-1.el9.x86_64 140/153 2023-10-19T12:43:08.763 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 141/153 2023-10-19T12:43:08.763 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : NetworkManager-libnm-1:1.45.4-1.el9.x86_64 142/153 2023-10-19T12:43:08.763 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 143/153 2023-10-19T12:43:08.764 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : NetworkManager-team-1:1.45.4-1.el9.x86_64 144/153 2023-10-19T12:43:08.764 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 145/153 2023-10-19T12:43:08.764 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : NetworkManager-tui-1:1.45.4-1.el9.x86_64 146/153 2023-10-19T12:43:08.764 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 147/153 2023-10-19T12:43:08.764 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : acl-2.3.1-4.el9.x86_64 148/153 2023-10-19T12:43:08.765 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : acl-2.3.1-3.el9.x86_64 149/153 2023-10-19T12:43:08.765 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 150/153 2023-10-19T12:43:08.765 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 151/153 2023-10-19T12:43:08.765 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 152/153 2023-10-19T12:43:11.073 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 153/153 2023-10-19T12:43:11.073 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:43:11.074 INFO:teuthology.orchestra.run.smithi057.stdout:Upgraded: 2023-10-19T12:43:11.074 INFO:teuthology.orchestra.run.smithi057.stdout: NetworkManager-1:1.45.4-1.el9.x86_64 2023-10-19T12:43:11.074 INFO:teuthology.orchestra.run.smithi057.stdout: NetworkManager-libnm-1:1.45.4-1.el9.x86_64 2023-10-19T12:43:11.074 INFO:teuthology.orchestra.run.smithi057.stdout: NetworkManager-team-1:1.45.4-1.el9.x86_64 2023-10-19T12:43:11.074 INFO:teuthology.orchestra.run.smithi057.stdout: NetworkManager-tui-1:1.45.4-1.el9.x86_64 2023-10-19T12:43:11.075 INFO:teuthology.orchestra.run.smithi057.stdout: acl-2.3.1-4.el9.x86_64 2023-10-19T12:43:11.075 INFO:teuthology.orchestra.run.smithi057.stdout: libacl-2.3.1-4.el9.x86_64 2023-10-19T12:43:11.075 INFO:teuthology.orchestra.run.smithi057.stdout: libacl-devel-2.3.1-4.el9.x86_64 2023-10-19T12:43:11.075 INFO:teuthology.orchestra.run.smithi057.stdout: librados2-2:18.0.0-6786.gf7420d7f.el9.x86_64 2023-10-19T12:43:11.075 INFO:teuthology.orchestra.run.smithi057.stdout: librbd1-2:18.0.0-6786.gf7420d7f.el9.x86_64 2023-10-19T12:43:11.076 INFO:teuthology.orchestra.run.smithi057.stdout: xfsdump-3.1.12-4.el9.x86_64 2023-10-19T12:43:11.076 INFO:teuthology.orchestra.run.smithi057.stdout:Installed: 2023-10-19T12:43:11.076 INFO:teuthology.orchestra.run.smithi057.stdout: automake-1.16.2-8.el9.noarch 2023-10-19T12:43:11.076 INFO:teuthology.orchestra.run.smithi057.stdout: bison-3.7.4-5.el9.x86_64 2023-10-19T12:43:11.076 INFO:teuthology.orchestra.run.smithi057.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-10-19T12:43:11.076 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-2:18.0.0-6786.gf7420d7f.el9.x86_64 2023-10-19T12:43:11.077 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-base-2:18.0.0-6786.gf7420d7f.el9.x86_64 2023-10-19T12:43:11.077 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-common-2:18.0.0-6786.gf7420d7f.el9.x86_64 2023-10-19T12:43:11.077 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-fuse-2:18.0.0-6786.gf7420d7f.el9.x86_64 2023-10-19T12:43:11.077 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-grafana-dashboards-2:18.0.0-6786.gf7420d7f.el9.noarch 2023-10-19T12:43:11.077 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-immutable-object-cache-2:18.0.0-6786.gf7420d7f.el9.x86_64 2023-10-19T12:43:11.078 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mds-2:18.0.0-6786.gf7420d7f.el9.x86_64 2023-10-19T12:43:11.078 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-2:18.0.0-6786.gf7420d7f.el9.x86_64 2023-10-19T12:43:11.078 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-cephadm-2:18.0.0-6786.gf7420d7f.el9.noarch 2023-10-19T12:43:11.078 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-dashboard-2:18.0.0-6786.gf7420d7f.el9.noarch 2023-10-19T12:43:11.078 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-diskprediction-local-2:18.0.0-6786.gf7420d7f.el9.noarch 2023-10-19T12:43:11.079 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-modules-core-2:18.0.0-6786.gf7420d7f.el9.noarch 2023-10-19T12:43:11.079 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-rook-2:18.0.0-6786.gf7420d7f.el9.noarch 2023-10-19T12:43:11.079 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mon-2:18.0.0-6786.gf7420d7f.el9.x86_64 2023-10-19T12:43:11.079 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-osd-2:18.0.0-6786.gf7420d7f.el9.x86_64 2023-10-19T12:43:11.079 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-prometheus-alerts-2:18.0.0-6786.gf7420d7f.el9.noarch 2023-10-19T12:43:11.080 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-radosgw-2:18.0.0-6786.gf7420d7f.el9.x86_64 2023-10-19T12:43:11.080 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-selinux-2:18.0.0-6786.gf7420d7f.el9.x86_64 2023-10-19T12:43:11.080 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-test-2:18.0.0-6786.gf7420d7f.el9.x86_64 2023-10-19T12:43:11.080 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-volume-2:18.0.0-6786.gf7420d7f.el9.noarch 2023-10-19T12:43:11.080 INFO:teuthology.orchestra.run.smithi057.stdout: cephadm-2:18.0.0-6786.gf7420d7f.el9.noarch 2023-10-19T12:43:11.081 INFO:teuthology.orchestra.run.smithi057.stdout: cephfs-mirror-2:18.0.0-6786.gf7420d7f.el9.x86_64 2023-10-19T12:43:11.081 INFO:teuthology.orchestra.run.smithi057.stdout: cephfs-top-2:18.0.0-6786.gf7420d7f.el9.noarch 2023-10-19T12:43:11.081 INFO:teuthology.orchestra.run.smithi057.stdout: elfutils-libelf-devel-0.189-3.el9.x86_64 2023-10-19T12:43:11.081 INFO:teuthology.orchestra.run.smithi057.stdout: flex-2.6.4-9.el9.x86_64 2023-10-19T12:43:11.081 INFO:teuthology.orchestra.run.smithi057.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-10-19T12:43:11.082 INFO:teuthology.orchestra.run.smithi057.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-10-19T12:43:11.082 INFO:teuthology.orchestra.run.smithi057.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-10-19T12:43:11.082 INFO:teuthology.orchestra.run.smithi057.stdout: fmt-8.1.1-5.el9.x86_64 2023-10-19T12:43:11.082 INFO:teuthology.orchestra.run.smithi057.stdout: fuse-2.9.9-15.el9.x86_64 2023-10-19T12:43:11.082 INFO:teuthology.orchestra.run.smithi057.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-10-19T12:43:11.083 INFO:teuthology.orchestra.run.smithi057.stdout: jq-1.6-15.el9.x86_64 2023-10-19T12:43:11.083 INFO:teuthology.orchestra.run.smithi057.stdout: libaio-devel-0.3.111-13.el9.x86_64 2023-10-19T12:43:11.083 INFO:teuthology.orchestra.run.smithi057.stdout: libarrow-9.0.0-9.el9.x86_64 2023-10-19T12:43:11.083 INFO:teuthology.orchestra.run.smithi057.stdout: libarrow-doc-9.0.0-9.el9.noarch 2023-10-19T12:43:11.083 INFO:teuthology.orchestra.run.smithi057.stdout: libcephfs-devel-2:18.0.0-6786.gf7420d7f.el9.x86_64 2023-10-19T12:43:11.084 INFO:teuthology.orchestra.run.smithi057.stdout: libcephfs2-2:18.0.0-6786.gf7420d7f.el9.x86_64 2023-10-19T12:43:11.084 INFO:teuthology.orchestra.run.smithi057.stdout: libcephsqlite-2:18.0.0-6786.gf7420d7f.el9.x86_64 2023-10-19T12:43:11.084 INFO:teuthology.orchestra.run.smithi057.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-10-19T12:43:11.084 INFO:teuthology.orchestra.run.smithi057.stdout: liboath-2.6.7-2.el9.x86_64 2023-10-19T12:43:11.084 INFO:teuthology.orchestra.run.smithi057.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-10-19T12:43:11.085 INFO:teuthology.orchestra.run.smithi057.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-10-19T12:43:11.085 INFO:teuthology.orchestra.run.smithi057.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-10-19T12:43:11.085 INFO:teuthology.orchestra.run.smithi057.stdout: librados-devel-2:18.0.0-6786.gf7420d7f.el9.x86_64 2023-10-19T12:43:11.085 INFO:teuthology.orchestra.run.smithi057.stdout: libradosstriper1-2:18.0.0-6786.gf7420d7f.el9.x86_64 2023-10-19T12:43:11.085 INFO:teuthology.orchestra.run.smithi057.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-10-19T12:43:11.085 INFO:teuthology.orchestra.run.smithi057.stdout: librgw2-2:18.0.0-6786.gf7420d7f.el9.x86_64 2023-10-19T12:43:11.086 INFO:teuthology.orchestra.run.smithi057.stdout: libtool-2.4.6-45.el9.x86_64 2023-10-19T12:43:11.086 INFO:teuthology.orchestra.run.smithi057.stdout: libunwind-1.6.2-1.el9.x86_64 2023-10-19T12:43:11.086 INFO:teuthology.orchestra.run.smithi057.stdout: libuuid-devel-2.37.4-15.el9.x86_64 2023-10-19T12:43:11.086 INFO:teuthology.orchestra.run.smithi057.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2023-10-19T12:43:11.087 INFO:teuthology.orchestra.run.smithi057.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-10-19T12:43:11.087 INFO:teuthology.orchestra.run.smithi057.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-10-19T12:43:11.087 INFO:teuthology.orchestra.run.smithi057.stdout: openblas-0.3.21-2.el9.x86_64 2023-10-19T12:43:11.087 INFO:teuthology.orchestra.run.smithi057.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-10-19T12:43:11.087 INFO:teuthology.orchestra.run.smithi057.stdout: openssl-devel-1:3.0.7-24.el9.x86_64 2023-10-19T12:43:11.087 INFO:teuthology.orchestra.run.smithi057.stdout: parquet-libs-9.0.0-9.el9.x86_64 2023-10-19T12:43:11.088 INFO:teuthology.orchestra.run.smithi057.stdout: postgresql-13.11-1.el9.x86_64 2023-10-19T12:43:11.088 INFO:teuthology.orchestra.run.smithi057.stdout: postgresql-contrib-13.11-1.el9.x86_64 2023-10-19T12:43:11.088 INFO:teuthology.orchestra.run.smithi057.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2023-10-19T12:43:11.088 INFO:teuthology.orchestra.run.smithi057.stdout: postgresql-server-13.11-1.el9.x86_64 2023-10-19T12:43:11.088 INFO:teuthology.orchestra.run.smithi057.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-10-19T12:43:11.089 INFO:teuthology.orchestra.run.smithi057.stdout: python3-babel-2.9.1-2.el9.noarch 2023-10-19T12:43:11.089 INFO:teuthology.orchestra.run.smithi057.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-10-19T12:43:11.089 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-10-19T12:43:11.089 INFO:teuthology.orchestra.run.smithi057.stdout: python3-ceph-argparse-2:18.0.0-6786.gf7420d7f.el9.x86_64 2023-10-19T12:43:11.089 INFO:teuthology.orchestra.run.smithi057.stdout: python3-ceph-common-2:18.0.0-6786.gf7420d7f.el9.x86_64 2023-10-19T12:43:11.090 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cephfs-2:18.0.0-6786.gf7420d7f.el9.x86_64 2023-10-19T12:43:11.090 INFO:teuthology.orchestra.run.smithi057.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-10-19T12:43:11.090 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-10-19T12:43:11.090 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-10-19T12:43:11.091 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-10-19T12:43:11.091 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-10-19T12:43:11.091 INFO:teuthology.orchestra.run.smithi057.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-10-19T12:43:11.091 INFO:teuthology.orchestra.run.smithi057.stdout: python3-google-auth-1:2.23.3-1.el9.noarch 2023-10-19T12:43:11.091 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-10-19T12:43:11.092 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-10-19T12:43:11.092 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-10-19T12:43:11.092 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-10-19T12:43:11.092 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-10-19T12:43:11.092 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-10-19T12:43:11.092 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-10-19T12:43:11.093 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-10-19T12:43:11.093 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-10-19T12:43:11.093 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-10-19T12:43:11.093 INFO:teuthology.orchestra.run.smithi057.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-10-19T12:43:11.093 INFO:teuthology.orchestra.run.smithi057.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-10-19T12:43:11.094 INFO:teuthology.orchestra.run.smithi057.stdout: python3-mako-1.1.4-6.el9.noarch 2023-10-19T12:43:11.094 INFO:teuthology.orchestra.run.smithi057.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-10-19T12:43:11.094 INFO:teuthology.orchestra.run.smithi057.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-10-19T12:43:11.094 INFO:teuthology.orchestra.run.smithi057.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-10-19T12:43:11.094 INFO:teuthology.orchestra.run.smithi057.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-10-19T12:43:11.095 INFO:teuthology.orchestra.run.smithi057.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-10-19T12:43:11.095 INFO:teuthology.orchestra.run.smithi057.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-10-19T12:43:11.095 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-10-19T12:43:11.095 INFO:teuthology.orchestra.run.smithi057.stdout: python3-ply-3.11-14.el9.noarch 2023-10-19T12:43:11.095 INFO:teuthology.orchestra.run.smithi057.stdout: python3-portend-3.1.0-2.el9.noarch 2023-10-19T12:43:11.096 INFO:teuthology.orchestra.run.smithi057.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-10-19T12:43:11.096 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-10-19T12:43:11.096 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-10-19T12:43:11.096 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-10-19T12:43:11.096 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pycparser-2.20-6.el9.noarch 2023-10-19T12:43:11.097 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pytz-2021.1-5.el9.noarch 2023-10-19T12:43:11.097 INFO:teuthology.orchestra.run.smithi057.stdout: python3-rados-2:18.0.0-6786.gf7420d7f.el9.x86_64 2023-10-19T12:43:11.097 INFO:teuthology.orchestra.run.smithi057.stdout: python3-rbd-2:18.0.0-6786.gf7420d7f.el9.x86_64 2023-10-19T12:43:11.097 INFO:teuthology.orchestra.run.smithi057.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-10-19T12:43:11.097 INFO:teuthology.orchestra.run.smithi057.stdout: python3-requests-2.25.1-7.el9.noarch 2023-10-19T12:43:11.098 INFO:teuthology.orchestra.run.smithi057.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-10-19T12:43:11.098 INFO:teuthology.orchestra.run.smithi057.stdout: python3-rgw-2:18.0.0-6786.gf7420d7f.el9.x86_64 2023-10-19T12:43:11.098 INFO:teuthology.orchestra.run.smithi057.stdout: python3-routes-2.5.1-5.el9.noarch 2023-10-19T12:43:11.098 INFO:teuthology.orchestra.run.smithi057.stdout: python3-rsa-4.9-2.el9.noarch 2023-10-19T12:43:11.098 INFO:teuthology.orchestra.run.smithi057.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-10-19T12:43:11.099 INFO:teuthology.orchestra.run.smithi057.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-10-19T12:43:11.099 INFO:teuthology.orchestra.run.smithi057.stdout: python3-toml-0.10.2-6.el9.noarch 2023-10-19T12:43:11.099 INFO:teuthology.orchestra.run.smithi057.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-10-19T12:43:11.099 INFO:teuthology.orchestra.run.smithi057.stdout: python3-urllib3-1.26.5-3.el9.noarch 2023-10-19T12:43:11.099 INFO:teuthology.orchestra.run.smithi057.stdout: python3-webob-1.8.7-6.el9.noarch 2023-10-19T12:43:11.100 INFO:teuthology.orchestra.run.smithi057.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-10-19T12:43:11.100 INFO:teuthology.orchestra.run.smithi057.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-10-19T12:43:11.100 INFO:teuthology.orchestra.run.smithi057.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-10-19T12:43:11.100 INFO:teuthology.orchestra.run.smithi057.stdout: rbd-fuse-2:18.0.0-6786.gf7420d7f.el9.x86_64 2023-10-19T12:43:11.100 INFO:teuthology.orchestra.run.smithi057.stdout: rbd-mirror-2:18.0.0-6786.gf7420d7f.el9.x86_64 2023-10-19T12:43:11.100 INFO:teuthology.orchestra.run.smithi057.stdout: rbd-nbd-2:18.0.0-6786.gf7420d7f.el9.x86_64 2023-10-19T12:43:11.101 INFO:teuthology.orchestra.run.smithi057.stdout: re2-1:20211101-3.el9.x86_64 2023-10-19T12:43:11.101 INFO:teuthology.orchestra.run.smithi057.stdout: socat-1.7.4.1-5.el9.x86_64 2023-10-19T12:43:11.101 INFO:teuthology.orchestra.run.smithi057.stdout: thrift-0.15.0-2.el9.x86_64 2023-10-19T12:43:11.101 INFO:teuthology.orchestra.run.smithi057.stdout: uuid-1.6.2-55.el9.x86_64 2023-10-19T12:43:11.101 INFO:teuthology.orchestra.run.smithi057.stdout: xfsprogs-devel-5.19.0-4.el9.x86_64 2023-10-19T12:43:11.102 INFO:teuthology.orchestra.run.smithi057.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-10-19T12:43:11.102 INFO:teuthology.orchestra.run.smithi057.stdout: zlib-devel-1.2.11-40.el9.x86_64 2023-10-19T12:43:11.102 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:43:11.103 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2023-10-19T12:43:11.536 DEBUG:teuthology.parallel:result is None 2023-10-19T12:43:11.537 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=f7420d7f8e55ceea786dcbf536c338ce920c21eb 2023-10-19T12:43:11.780 DEBUG:teuthology.orchestra.run.smithi057:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-10-19T12:43:11.803 INFO:teuthology.orchestra.run.smithi057.stdout:18.0.0-6786.gf7420d7f.el9 2023-10-19T12:43:11.803 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6786.gf7420d7f.el9 2023-10-19T12:43:11.804 INFO:teuthology.task.install:The correct ceph version 18.0.0-6786.gf7420d7f is installed. 2023-10-19T12:43:11.804 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=f7420d7f8e55ceea786dcbf536c338ce920c21eb 2023-10-19T12:43:11.953 DEBUG:teuthology.orchestra.run.smithi179:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-10-19T12:43:11.978 INFO:teuthology.orchestra.run.smithi179.stdout:18.0.0-6786.gf7420d7f.el9 2023-10-19T12:43:11.978 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6786.gf7420d7f.el9 2023-10-19T12:43:11.979 INFO:teuthology.task.install:The correct ceph version 18.0.0-6786.gf7420d7f is installed. 2023-10-19T12:43:11.979 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-10-19T12:43:11.980 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T12:43:11.980 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-19T12:43:12.015 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-19T12:43:12.015 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-19T12:43:12.052 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-10-19T12:43:12.052 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T12:43:12.052 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/usr/bin/daemon-helper 2023-10-19T12:43:12.086 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-19T12:43:12.156 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-19T12:43:12.156 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/usr/bin/daemon-helper 2023-10-19T12:43:12.186 DEBUG:teuthology.orchestra.run.smithi179:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-19T12:43:12.256 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-10-19T12:43:12.256 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T12:43:12.256 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-19T12:43:12.287 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-19T12:43:12.358 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-19T12:43:12.358 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-19T12:43:12.387 DEBUG:teuthology.orchestra.run.smithi179:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-19T12:43:12.455 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-10-19T12:43:12.455 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T12:43:12.456 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/usr/bin/stdin-killer 2023-10-19T12:43:12.486 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-19T12:43:12.557 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-19T12:43:12.557 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/usr/bin/stdin-killer 2023-10-19T12:43:12.585 DEBUG:teuthology.orchestra.run.smithi179:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-19T12:43:12.653 INFO:teuthology.run_tasks:Running task ceph... 2023-10-19T12:43:12.716 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-10-19T12:43:12.716 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod 777 /var/log/ceph 2023-10-19T12:43:12.719 DEBUG:teuthology.orchestra.run.smithi179:> sudo chmod 777 /var/log/ceph 2023-10-19T12:43:12.748 INFO:tasks.ceph:Disabling ceph logrotate... 2023-10-19T12:43:12.748 DEBUG:teuthology.orchestra.run.smithi057:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-19T12:43:12.790 DEBUG:teuthology.orchestra.run.smithi179:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-19T12:43:12.818 INFO:tasks.ceph:Creating extra log directories... 2023-10-19T12:43:12.821 DEBUG:teuthology.orchestra.run.smithi057:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-19T12:43:12.861 DEBUG:teuthology.orchestra.run.smithi179:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-19T12:43:12.895 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2023-10-19T12:43:12.895 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2023-10-19T12:43:12.896 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2023-10-19T12:43:12.896 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T12:43:12.896 DEBUG:teuthology.orchestra.run.smithi057:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-10-19T12:43:12.944 DEBUG:teuthology.orchestra.run.smithi057:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-10-19T12:43:13.076 DEBUG:teuthology.orchestra.run.smithi057:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2023-10-19T12:43:13.106 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-19T12:43:13.107 DEBUG:teuthology.orchestra.run.smithi179:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-10-19T12:43:13.122 DEBUG:teuthology.orchestra.run.smithi179:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-10-19T12:43:13.239 DEBUG:teuthology.orchestra.run.smithi179:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2023-10-19T12:43:13.271 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-10-19T12:43:13.271 INFO:tasks.ceph:config {'conf': {'client': {'pid file': ''}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-10-19T12:43:13.271 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/pdonnell-2023-10-19_12:04:12-fs-wip-batrick-testing-20231016.203825-distro-default-smithi/7432290', 'branch': 'wip-batrick-testing-20231016.203825', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule}', 'email': 'pdonnell@redhat.com', 'first_in_suite': False, 'job_id': '7432290', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'pdonnell-2023-10-19_12:04:12-fs-wip-batrick-testing-20231016.203825-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-batrick-testing-20231016.203825'}, 'ceph': {'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'cluster-conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon': {'mon op complaint time': 120}, 'osd': {'osd op complaint time': 180}}, 'conf': {'client': {'pid file': ''}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'log-whitelist': ['OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'sha1': 'f7420d7f8e55ceea786dcbf536c338ce920c21eb'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'f7420d7f8e55ceea786dcbf536c338ce920c21eb'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-batrick-testing-20231016.203825', 'sha1': 'f7420d7f8e55ceea786dcbf536c338ce920c21eb'}}, 'owner': 'scheduled_pdonnell@teuthology', 'priority': 65, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 7703, 'sha1': 'f7420d7f8e55ceea786dcbf536c338ce920c21eb', 'sleep_before_teardown': 0, 'subset': '153/256', 'suite': 'fs', 'suite_branch': 'wip-batrick-testing-20231016.203825', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_f7420d7f8e55ceea786dcbf536c338ce920c21eb/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': 'f7420d7f8e55ceea786dcbf536c338ce920c21eb', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi057.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLIN80GzytgnsPmhQxnO98RJ6hrAia1j+6pBY3TcjQx0CYaH+EBbVFNcBgcBJT8k6V/pnOvlWsn8miLAG0PNdlU=', 'smithi179.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHzD4QfGTQMro3MfFhf2wmDWesRzz0Us4uWFCUG2uJYCokdZYdTLrJb2/y+R7emIU5CcjUKUzVhjYy6iyLxqsbk='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'f7420d7f8e55ceea786dcbf536c338ce920c21eb'}}, {'ceph': {'create_rbd_pool': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'cluster-conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon': {'mon op complaint time': 120}, 'osd': {'osd op complaint time': 180}}, 'conf': {'client': {'pid file': ''}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'log-whitelist': ['OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'sha1': 'f7420d7f8e55ceea786dcbf536c338ce920c21eb', 'cluster': 'ceph'}}, {'ceph-fuse': None}, {'cephfs_test_runner': {'modules': ['tasks.cephfs.test_snap_schedules']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ['local function is_kupstream() return false end', 'local function is_kdistro() return false end', 'local function is_fuse() return true end', "local function syntax_version() return '' end"]}, 'teuthology_branch': 'main', 'teuthology_sha1': '202b180cb047e798fb131047314a862593f45403', 'timestamp': '2023-10-19_12:04:12', 'tube': 'smithi', 'user': 'pdonnell', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2830'} 2023-10-19T12:43:13.272 DEBUG:teuthology.orchestra.run.smithi057:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-10-19T12:43:13.275 DEBUG:teuthology.orchestra.run.smithi179:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-10-19T12:43:13.319 DEBUG:teuthology.orchestra.run.smithi057:> sudo install -d -m0777 -- /var/run/ceph 2023-10-19T12:43:13.330 DEBUG:teuthology.orchestra.run.smithi179:> sudo install -d -m0777 -- /var/run/ceph 2023-10-19T12:43:13.385 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T12:43:13.386 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/scratch_devs of=/dev/stdout 2023-10-19T12:43:13.413 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-19T12:43:13.413 DEBUG:teuthology.orchestra.run.smithi057:> stat /dev/vg_nvme/lv_1 2023-10-19T12:43:13.468 INFO:teuthology.orchestra.run.smithi057.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-19T12:43:13.468 INFO:teuthology.orchestra.run.smithi057.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-19T12:43:13.469 INFO:teuthology.orchestra.run.smithi057.stdout:Device: 5h/5d Inode: 662 Links: 1 2023-10-19T12:43:13.469 INFO:teuthology.orchestra.run.smithi057.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-19T12:43:13.469 INFO:teuthology.orchestra.run.smithi057.stdout:Context: system_u:object_r:device_t:s0 2023-10-19T12:43:13.469 INFO:teuthology.orchestra.run.smithi057.stdout:Access: 2023-10-19 12:43:04.645762650 +0000 2023-10-19T12:43:13.469 INFO:teuthology.orchestra.run.smithi057.stdout:Modify: 2023-10-19 12:41:01.364489198 +0000 2023-10-19T12:43:13.470 INFO:teuthology.orchestra.run.smithi057.stdout:Change: 2023-10-19 12:41:01.364489198 +0000 2023-10-19T12:43:13.470 INFO:teuthology.orchestra.run.smithi057.stdout: Birth: 2023-10-19 12:41:01.364489198 +0000 2023-10-19T12:43:13.471 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-19T12:43:13.538 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records in 2023-10-19T12:43:13.538 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records out 2023-10-19T12:43:13.538 INFO:teuthology.orchestra.run.smithi057.stderr:512 bytes copied, 0.000279776 s, 1.8 MB/s 2023-10-19T12:43:13.539 DEBUG:teuthology.orchestra.run.smithi057:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-19T12:43:13.595 DEBUG:teuthology.orchestra.run.smithi057:> stat /dev/vg_nvme/lv_2 2023-10-19T12:43:13.651 INFO:teuthology.orchestra.run.smithi057.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-19T12:43:13.652 INFO:teuthology.orchestra.run.smithi057.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-19T12:43:13.652 INFO:teuthology.orchestra.run.smithi057.stdout:Device: 5h/5d Inode: 684 Links: 1 2023-10-19T12:43:13.652 INFO:teuthology.orchestra.run.smithi057.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-19T12:43:13.652 INFO:teuthology.orchestra.run.smithi057.stdout:Context: system_u:object_r:device_t:s0 2023-10-19T12:43:13.652 INFO:teuthology.orchestra.run.smithi057.stdout:Access: 2023-10-19 12:43:04.645762650 +0000 2023-10-19T12:43:13.652 INFO:teuthology.orchestra.run.smithi057.stdout:Modify: 2023-10-19 12:41:01.861481691 +0000 2023-10-19T12:43:13.653 INFO:teuthology.orchestra.run.smithi057.stdout:Change: 2023-10-19 12:41:01.861481691 +0000 2023-10-19T12:43:13.653 INFO:teuthology.orchestra.run.smithi057.stdout: Birth: 2023-10-19 12:41:01.861481691 +0000 2023-10-19T12:43:13.653 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-19T12:43:13.720 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records in 2023-10-19T12:43:13.720 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records out 2023-10-19T12:43:13.720 INFO:teuthology.orchestra.run.smithi057.stderr:512 bytes copied, 0.000248669 s, 2.1 MB/s 2023-10-19T12:43:13.721 DEBUG:teuthology.orchestra.run.smithi057:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-19T12:43:13.777 DEBUG:teuthology.orchestra.run.smithi057:> stat /dev/vg_nvme/lv_3 2023-10-19T12:43:13.833 INFO:teuthology.orchestra.run.smithi057.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-19T12:43:13.833 INFO:teuthology.orchestra.run.smithi057.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-19T12:43:13.833 INFO:teuthology.orchestra.run.smithi057.stdout:Device: 5h/5d Inode: 702 Links: 1 2023-10-19T12:43:13.833 INFO:teuthology.orchestra.run.smithi057.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-19T12:43:13.834 INFO:teuthology.orchestra.run.smithi057.stdout:Context: system_u:object_r:device_t:s0 2023-10-19T12:43:13.834 INFO:teuthology.orchestra.run.smithi057.stdout:Access: 2023-10-19 12:43:04.646762635 +0000 2023-10-19T12:43:13.834 INFO:teuthology.orchestra.run.smithi057.stdout:Modify: 2023-10-19 12:41:02.386473760 +0000 2023-10-19T12:43:13.834 INFO:teuthology.orchestra.run.smithi057.stdout:Change: 2023-10-19 12:41:02.386473760 +0000 2023-10-19T12:43:13.834 INFO:teuthology.orchestra.run.smithi057.stdout: Birth: 2023-10-19 12:41:02.386473760 +0000 2023-10-19T12:43:13.835 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-19T12:43:13.901 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records in 2023-10-19T12:43:13.902 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records out 2023-10-19T12:43:13.902 INFO:teuthology.orchestra.run.smithi057.stderr:512 bytes copied, 0.000286727 s, 1.8 MB/s 2023-10-19T12:43:13.903 DEBUG:teuthology.orchestra.run.smithi057:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-19T12:43:13.958 DEBUG:teuthology.orchestra.run.smithi057:> stat /dev/vg_nvme/lv_4 2023-10-19T12:43:14.014 INFO:teuthology.orchestra.run.smithi057.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-19T12:43:14.014 INFO:teuthology.orchestra.run.smithi057.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-19T12:43:14.014 INFO:teuthology.orchestra.run.smithi057.stdout:Device: 5h/5d Inode: 722 Links: 1 2023-10-19T12:43:14.014 INFO:teuthology.orchestra.run.smithi057.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-19T12:43:14.014 INFO:teuthology.orchestra.run.smithi057.stdout:Context: system_u:object_r:device_t:s0 2023-10-19T12:43:14.015 INFO:teuthology.orchestra.run.smithi057.stdout:Access: 2023-10-19 12:43:04.646762635 +0000 2023-10-19T12:43:14.015 INFO:teuthology.orchestra.run.smithi057.stdout:Modify: 2023-10-19 12:41:02.907465890 +0000 2023-10-19T12:43:14.015 INFO:teuthology.orchestra.run.smithi057.stdout:Change: 2023-10-19 12:41:02.907465890 +0000 2023-10-19T12:43:14.015 INFO:teuthology.orchestra.run.smithi057.stdout: Birth: 2023-10-19 12:41:02.907465890 +0000 2023-10-19T12:43:14.015 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-19T12:43:14.082 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records in 2023-10-19T12:43:14.083 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records out 2023-10-19T12:43:14.083 INFO:teuthology.orchestra.run.smithi057.stderr:512 bytes copied, 0.000280932 s, 1.8 MB/s 2023-10-19T12:43:14.084 DEBUG:teuthology.orchestra.run.smithi057:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-19T12:43:14.139 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-19T12:43:14.139 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-19T12:43:14.139 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/scratch_devs of=/dev/stdout 2023-10-19T12:43:14.157 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-19T12:43:14.157 DEBUG:teuthology.orchestra.run.smithi179:> stat /dev/vg_nvme/lv_1 2023-10-19T12:43:14.213 INFO:teuthology.orchestra.run.smithi179.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-19T12:43:14.213 INFO:teuthology.orchestra.run.smithi179.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-19T12:43:14.213 INFO:teuthology.orchestra.run.smithi179.stdout:Device: 5h/5d Inode: 663 Links: 1 2023-10-19T12:43:14.213 INFO:teuthology.orchestra.run.smithi179.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-19T12:43:14.214 INFO:teuthology.orchestra.run.smithi179.stdout:Context: system_u:object_r:device_t:s0 2023-10-19T12:43:14.214 INFO:teuthology.orchestra.run.smithi179.stdout:Access: 2023-10-19 12:42:45.931273272 +0000 2023-10-19T12:43:14.214 INFO:teuthology.orchestra.run.smithi179.stdout:Modify: 2023-10-19 12:40:53.566141441 +0000 2023-10-19T12:43:14.214 INFO:teuthology.orchestra.run.smithi179.stdout:Change: 2023-10-19 12:40:53.566141441 +0000 2023-10-19T12:43:14.214 INFO:teuthology.orchestra.run.smithi179.stdout: Birth: 2023-10-19 12:40:53.566141441 +0000 2023-10-19T12:43:14.215 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-19T12:43:14.280 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records in 2023-10-19T12:43:14.280 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records out 2023-10-19T12:43:14.280 INFO:teuthology.orchestra.run.smithi179.stderr:512 bytes copied, 0.000192314 s, 2.7 MB/s 2023-10-19T12:43:14.281 DEBUG:teuthology.orchestra.run.smithi179:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-19T12:43:14.338 DEBUG:teuthology.orchestra.run.smithi179:> stat /dev/vg_nvme/lv_2 2023-10-19T12:43:14.394 INFO:teuthology.orchestra.run.smithi179.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-19T12:43:14.394 INFO:teuthology.orchestra.run.smithi179.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-19T12:43:14.394 INFO:teuthology.orchestra.run.smithi179.stdout:Device: 5h/5d Inode: 681 Links: 1 2023-10-19T12:43:14.394 INFO:teuthology.orchestra.run.smithi179.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-19T12:43:14.395 INFO:teuthology.orchestra.run.smithi179.stdout:Context: system_u:object_r:device_t:s0 2023-10-19T12:43:14.395 INFO:teuthology.orchestra.run.smithi179.stdout:Access: 2023-10-19 12:42:45.931273272 +0000 2023-10-19T12:43:14.395 INFO:teuthology.orchestra.run.smithi179.stdout:Modify: 2023-10-19 12:40:54.034131147 +0000 2023-10-19T12:43:14.395 INFO:teuthology.orchestra.run.smithi179.stdout:Change: 2023-10-19 12:40:54.034131147 +0000 2023-10-19T12:43:14.395 INFO:teuthology.orchestra.run.smithi179.stdout: Birth: 2023-10-19 12:40:54.034131147 +0000 2023-10-19T12:43:14.396 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-19T12:43:14.461 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records in 2023-10-19T12:43:14.461 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records out 2023-10-19T12:43:14.461 INFO:teuthology.orchestra.run.smithi179.stderr:512 bytes copied, 0.000219154 s, 2.3 MB/s 2023-10-19T12:43:14.462 DEBUG:teuthology.orchestra.run.smithi179:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-19T12:43:14.518 DEBUG:teuthology.orchestra.run.smithi179:> stat /dev/vg_nvme/lv_3 2023-10-19T12:43:14.573 INFO:teuthology.orchestra.run.smithi179.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-19T12:43:14.573 INFO:teuthology.orchestra.run.smithi179.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-19T12:43:14.573 INFO:teuthology.orchestra.run.smithi179.stdout:Device: 5h/5d Inode: 701 Links: 1 2023-10-19T12:43:14.574 INFO:teuthology.orchestra.run.smithi179.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-19T12:43:14.574 INFO:teuthology.orchestra.run.smithi179.stdout:Context: system_u:object_r:device_t:s0 2023-10-19T12:43:14.574 INFO:teuthology.orchestra.run.smithi179.stdout:Access: 2023-10-19 12:42:45.931273272 +0000 2023-10-19T12:43:14.574 INFO:teuthology.orchestra.run.smithi179.stdout:Modify: 2023-10-19 12:40:54.512120631 +0000 2023-10-19T12:43:14.574 INFO:teuthology.orchestra.run.smithi179.stdout:Change: 2023-10-19 12:40:54.512120631 +0000 2023-10-19T12:43:14.575 INFO:teuthology.orchestra.run.smithi179.stdout: Birth: 2023-10-19 12:40:54.512120631 +0000 2023-10-19T12:43:14.575 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-19T12:43:14.639 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records in 2023-10-19T12:43:14.639 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records out 2023-10-19T12:43:14.640 INFO:teuthology.orchestra.run.smithi179.stderr:512 bytes copied, 0.000243639 s, 2.1 MB/s 2023-10-19T12:43:14.641 DEBUG:teuthology.orchestra.run.smithi179:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-19T12:43:14.696 DEBUG:teuthology.orchestra.run.smithi179:> stat /dev/vg_nvme/lv_4 2023-10-19T12:43:14.751 INFO:teuthology.orchestra.run.smithi179.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-19T12:43:14.752 INFO:teuthology.orchestra.run.smithi179.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-19T12:43:14.752 INFO:teuthology.orchestra.run.smithi179.stdout:Device: 5h/5d Inode: 719 Links: 1 2023-10-19T12:43:14.752 INFO:teuthology.orchestra.run.smithi179.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-19T12:43:14.752 INFO:teuthology.orchestra.run.smithi179.stdout:Context: system_u:object_r:device_t:s0 2023-10-19T12:43:14.752 INFO:teuthology.orchestra.run.smithi179.stdout:Access: 2023-10-19 12:42:45.932273250 +0000 2023-10-19T12:43:14.753 INFO:teuthology.orchestra.run.smithi179.stdout:Modify: 2023-10-19 12:40:54.993110049 +0000 2023-10-19T12:43:14.753 INFO:teuthology.orchestra.run.smithi179.stdout:Change: 2023-10-19 12:40:54.993110049 +0000 2023-10-19T12:43:14.753 INFO:teuthology.orchestra.run.smithi179.stdout: Birth: 2023-10-19 12:40:54.993110049 +0000 2023-10-19T12:43:14.753 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-19T12:43:14.817 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records in 2023-10-19T12:43:14.817 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records out 2023-10-19T12:43:14.817 INFO:teuthology.orchestra.run.smithi179.stderr:512 bytes copied, 0.000232701 s, 2.2 MB/s 2023-10-19T12:43:14.818 DEBUG:teuthology.orchestra.run.smithi179:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-19T12:43:14.876 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-19T12:43:14.876 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi057.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi179.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-10-19T12:43:14.877 INFO:tasks.ceph:Generating config... 2023-10-19T12:43:14.878 INFO:tasks.ceph:[client] pid file = 2023-10-19T12:43:14.878 INFO:tasks.ceph:[mgr] debug client = 20 2023-10-19T12:43:14.878 INFO:tasks.ceph:[mgr] debug finisher = 20 2023-10-19T12:43:14.879 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-10-19T12:43:14.879 INFO:tasks.ceph:[mgr] debug ms = 1 2023-10-19T12:43:14.879 INFO:tasks.ceph:[mon] debug mon = 20 2023-10-19T12:43:14.879 INFO:tasks.ceph:[mon] debug ms = 1 2023-10-19T12:43:14.880 INFO:tasks.ceph:[mon] debug paxos = 20 2023-10-19T12:43:14.880 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-10-19T12:43:14.880 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-10-19T12:43:14.880 INFO:tasks.ceph:[osd] debug bluefs = 20 2023-10-19T12:43:14.881 INFO:tasks.ceph:[osd] debug bluestore = 20 2023-10-19T12:43:14.881 INFO:tasks.ceph:[osd] debug ms = 1 2023-10-19T12:43:14.881 INFO:tasks.ceph:[osd] debug osd = 20 2023-10-19T12:43:14.881 INFO:tasks.ceph:[osd] debug rocksdb = 10 2023-10-19T12:43:14.881 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-10-19T12:43:14.882 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-10-19T12:43:14.882 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-10-19T12:43:14.882 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-10-19T12:43:14.882 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-10-19T12:43:14.883 INFO:tasks.ceph:Setting up mon.a... 2023-10-19T12:43:14.883 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-10-19T12:43:14.937 INFO:teuthology.orchestra.run.smithi057.stdout:creating /etc/ceph/ceph.keyring 2023-10-19T12:43:14.940 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-10-19T12:43:14.990 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-19T12:43:15.020 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.57'), ('mon.b', '172.21.15.179'), ('mon.c', '[v2:172.21.15.179:3301,v1:172.21.15.179:6790]')] 2023-10-19T12:43:15.021 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.57,172.21.15.179,[v2:172.21.15.179:3301,v1:172.21.15.179:6790]'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20, 'debug finisher': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'pid file': ''}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}} 2023-10-19T12:43:15.022 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T12:43:15.022 DEBUG:teuthology.orchestra.run.smithi057:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-10-19T12:43:15.076 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.57 --add b 172.21.15.179 --addv c '[v2:172.21.15.179:3301,v1:172.21.15.179:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2023-10-19T12:43:15.162 INFO:teuthology.orchestra.run.smithi057.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-10-19T12:43:15.163 INFO:teuthology.orchestra.run.smithi057.stdout:monmaptool: generated fsid 1e9beb19-3260-4dd7-85ef-5b399ce774ac 2023-10-19T12:43:15.163 INFO:teuthology.orchestra.run.smithi057.stdout:setting min_mon_release = pacific 2023-10-19T12:43:15.163 INFO:teuthology.orchestra.run.smithi057.stdout:epoch 0 2023-10-19T12:43:15.163 INFO:teuthology.orchestra.run.smithi057.stdout:fsid 1e9beb19-3260-4dd7-85ef-5b399ce774ac 2023-10-19T12:43:15.164 INFO:teuthology.orchestra.run.smithi057.stdout:last_changed 2023-10-19T12:43:15.160947+0000 2023-10-19T12:43:15.164 INFO:teuthology.orchestra.run.smithi057.stdout:created 2023-10-19T12:43:15.160947+0000 2023-10-19T12:43:15.164 INFO:teuthology.orchestra.run.smithi057.stdout:min_mon_release 16 (pacific) 2023-10-19T12:43:15.164 INFO:teuthology.orchestra.run.smithi057.stdout:election_strategy: 1 2023-10-19T12:43:15.164 INFO:teuthology.orchestra.run.smithi057.stdout:0: [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] mon.a 2023-10-19T12:43:15.164 INFO:teuthology.orchestra.run.smithi057.stdout:1: [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] mon.b 2023-10-19T12:43:15.165 INFO:teuthology.orchestra.run.smithi057.stdout:2: [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] mon.c 2023-10-19T12:43:15.165 INFO:teuthology.orchestra.run.smithi057.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-10-19T12:43:15.166 DEBUG:teuthology.orchestra.run.smithi057:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-10-19T12:43:15.181 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 1e9beb19-3260-4dd7-85ef-5b399ce774ac... 2023-10-19T12:43:15.182 DEBUG:teuthology.orchestra.run.smithi057:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-19T12:43:15.223 DEBUG:teuthology.orchestra.run.smithi179:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-19T12:43:15.274 INFO:teuthology.orchestra.run.smithi179.stdout:[global] 2023-10-19T12:43:15.274 INFO:teuthology.orchestra.run.smithi179.stdout: chdir = "" 2023-10-19T12:43:15.274 INFO:teuthology.orchestra.run.smithi179.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-19T12:43:15.275 INFO:teuthology.orchestra.run.smithi179.stdout: auth supported = cephx 2023-10-19T12:43:15.275 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-10-19T12:43:15.275 INFO:teuthology.orchestra.run.smithi179.stdout: filestore xattr use omap = true 2023-10-19T12:43:15.275 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-10-19T12:43:15.276 INFO:teuthology.orchestra.run.smithi179.stdout: mon clock drift allowed = 1.000 2023-10-19T12:43:15.276 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-10-19T12:43:15.276 INFO:teuthology.orchestra.run.smithi179.stdout: osd crush chooseleaf type = 0 2023-10-19T12:43:15.276 INFO:teuthology.orchestra.run.smithi179.stdout: auth debug = true 2023-10-19T12:43:15.276 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-10-19T12:43:15.277 INFO:teuthology.orchestra.run.smithi179.stdout: ms die on old message = true 2023-10-19T12:43:15.277 INFO:teuthology.orchestra.run.smithi179.stdout: ms die on bug = true 2023-10-19T12:43:15.277 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-10-19T12:43:15.277 INFO:teuthology.orchestra.run.smithi179.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-19T12:43:15.277 INFO:teuthology.orchestra.run.smithi179.stdout: mon pg warn max object skew = 0 2023-10-19T12:43:15.278 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-10-19T12:43:15.278 INFO:teuthology.orchestra.run.smithi179.stdout: # disable pg_autoscaler by default for new pools 2023-10-19T12:43:15.278 INFO:teuthology.orchestra.run.smithi179.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-19T12:43:15.278 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-10-19T12:43:15.279 INFO:teuthology.orchestra.run.smithi179.stdout: osd pool default size = 2 2023-10-19T12:43:15.279 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-10-19T12:43:15.279 INFO:teuthology.orchestra.run.smithi179.stdout: mon osd allow primary affinity = true 2023-10-19T12:43:15.279 INFO:teuthology.orchestra.run.smithi179.stdout: mon osd allow pg remap = true 2023-10-19T12:43:15.279 INFO:teuthology.orchestra.run.smithi179.stdout: mon warn on legacy crush tunables = false 2023-10-19T12:43:15.280 INFO:teuthology.orchestra.run.smithi179.stdout: mon warn on crush straw calc version zero = false 2023-10-19T12:43:15.280 INFO:teuthology.orchestra.run.smithi179.stdout: mon warn on no sortbitwise = false 2023-10-19T12:43:15.280 INFO:teuthology.orchestra.run.smithi179.stdout: mon warn on osd down out interval zero = false 2023-10-19T12:43:15.280 INFO:teuthology.orchestra.run.smithi179.stdout: mon warn on too few osds = false 2023-10-19T12:43:15.280 INFO:teuthology.orchestra.run.smithi179.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-19T12:43:15.281 INFO:teuthology.orchestra.run.smithi179.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-19T12:43:15.281 INFO:teuthology.orchestra.run.smithi179.stdout: mon_allow_pool_size_one = true 2023-10-19T12:43:15.282 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-10-19T12:43:15.282 INFO:teuthology.orchestra.run.smithi179.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-19T12:43:15.282 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-10-19T12:43:15.282 INFO:teuthology.orchestra.run.smithi179.stdout: osd default data pool replay window = 5 2023-10-19T12:43:15.283 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-10-19T12:43:15.283 INFO:teuthology.orchestra.run.smithi179.stdout: mon allow pool delete = true 2023-10-19T12:43:15.283 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-10-19T12:43:15.283 INFO:teuthology.orchestra.run.smithi179.stdout: mon cluster log file level = debug 2023-10-19T12:43:15.283 INFO:teuthology.orchestra.run.smithi179.stdout: debug asserts on shutdown = true 2023-10-19T12:43:15.283 INFO:teuthology.orchestra.run.smithi179.stdout: mon health detail to clog = false 2023-10-19T12:43:15.284 INFO:teuthology.orchestra.run.smithi179.stdout: mon host = "172.21.15.57,172.21.15.179,[v2:172.21.15.179:3301,v1:172.21.15.179:6790]" 2023-10-19T12:43:15.284 INFO:teuthology.orchestra.run.smithi179.stdout: fsid = 1e9beb19-3260-4dd7-85ef-5b399ce774ac 2023-10-19T12:43:15.284 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-10-19T12:43:15.284 INFO:teuthology.orchestra.run.smithi179.stdout:[osd] 2023-10-19T12:43:15.284 INFO:teuthology.orchestra.run.smithi179.stdout: osd journal size = 100 2023-10-19T12:43:15.285 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-10-19T12:43:15.285 INFO:teuthology.orchestra.run.smithi179.stdout: osd scrub load threshold = 5.0 2023-10-19T12:43:15.285 INFO:teuthology.orchestra.run.smithi179.stdout: osd scrub max interval = 600 2023-10-19T12:43:15.285 INFO:teuthology.orchestra.run.smithi179.stdout: osd mclock profile = high_recovery_ops 2023-10-19T12:43:15.285 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-10-19T12:43:15.286 INFO:teuthology.orchestra.run.smithi179.stdout: osd recover clone overlap = true 2023-10-19T12:43:15.286 INFO:teuthology.orchestra.run.smithi179.stdout: osd recovery max chunk = 1048576 2023-10-19T12:43:15.286 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-10-19T12:43:15.286 INFO:teuthology.orchestra.run.smithi179.stdout: osd debug shutdown = true 2023-10-19T12:43:15.286 INFO:teuthology.orchestra.run.smithi179.stdout: osd debug op order = true 2023-10-19T12:43:15.286 INFO:teuthology.orchestra.run.smithi179.stdout: osd debug verify stray on activate = true 2023-10-19T12:43:15.287 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-10-19T12:43:15.287 INFO:teuthology.orchestra.run.smithi179.stdout: osd open classes on start = true 2023-10-19T12:43:15.287 INFO:teuthology.orchestra.run.smithi179.stdout: osd debug pg log writeout = true 2023-10-19T12:43:15.287 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-10-19T12:43:15.288 INFO:teuthology.orchestra.run.smithi179.stdout: osd deep scrub update digest min age = 30 2023-10-19T12:43:15.288 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-10-19T12:43:15.288 INFO:teuthology.orchestra.run.smithi179.stdout: osd map max advance = 10 2023-10-19T12:43:15.288 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-10-19T12:43:15.288 INFO:teuthology.orchestra.run.smithi179.stdout: journal zero on create = true 2023-10-19T12:43:15.288 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-10-19T12:43:15.289 INFO:teuthology.orchestra.run.smithi179.stdout: filestore ondisk finisher threads = 3 2023-10-19T12:43:15.289 INFO:teuthology.orchestra.run.smithi179.stdout: filestore apply finisher threads = 3 2023-10-19T12:43:15.289 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-10-19T12:43:15.289 INFO:teuthology.orchestra.run.smithi179.stdout: bdev debug aio = true 2023-10-19T12:43:15.289 INFO:teuthology.orchestra.run.smithi179.stdout: osd debug misdirected ops = true 2023-10-19T12:43:15.290 INFO:teuthology.orchestra.run.smithi179.stdout: bluestore block size = 96636764160 2023-10-19T12:43:15.290 INFO:teuthology.orchestra.run.smithi179.stdout: bluestore fsck on mount = True 2023-10-19T12:43:15.290 INFO:teuthology.orchestra.run.smithi179.stdout: debug bluefs = 20 2023-10-19T12:43:15.290 INFO:teuthology.orchestra.run.smithi179.stdout: debug bluestore = 20 2023-10-19T12:43:15.290 INFO:teuthology.orchestra.run.smithi179.stdout: debug ms = 1 2023-10-19T12:43:15.291 INFO:teuthology.orchestra.run.smithi179.stdout: debug osd = 20 2023-10-19T12:43:15.291 INFO:teuthology.orchestra.run.smithi179.stdout: debug rocksdb = 10 2023-10-19T12:43:15.291 INFO:teuthology.orchestra.run.smithi179.stdout: mon osd backfillfull_ratio = 0.85 2023-10-19T12:43:15.291 INFO:teuthology.orchestra.run.smithi179.stdout: mon osd full ratio = 0.9 2023-10-19T12:43:15.291 INFO:teuthology.orchestra.run.smithi179.stdout: mon osd nearfull ratio = 0.8 2023-10-19T12:43:15.291 INFO:teuthology.orchestra.run.smithi179.stdout: osd failsafe full ratio = 0.95 2023-10-19T12:43:15.292 INFO:teuthology.orchestra.run.smithi179.stdout: osd objectstore = bluestore 2023-10-19T12:43:15.292 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-10-19T12:43:15.292 INFO:teuthology.orchestra.run.smithi179.stdout:[mgr] 2023-10-19T12:43:15.292 INFO:teuthology.orchestra.run.smithi179.stdout: debug ms = 1 2023-10-19T12:43:15.292 INFO:teuthology.orchestra.run.smithi179.stdout: debug mgr = 20 2023-10-19T12:43:15.292 INFO:teuthology.orchestra.run.smithi179.stdout: debug mon = 20 2023-10-19T12:43:15.293 INFO:teuthology.orchestra.run.smithi179.stdout: debug auth = 20 2023-10-19T12:43:15.293 INFO:teuthology.orchestra.run.smithi179.stdout: mon reweight min pgs per osd = 4 2023-10-19T12:43:15.293 INFO:teuthology.orchestra.run.smithi179.stdout: mon reweight min bytes per osd = 10 2023-10-19T12:43:15.293 INFO:teuthology.orchestra.run.smithi179.stdout: mgr/telemetry/nag = false 2023-10-19T12:43:15.293 INFO:teuthology.orchestra.run.smithi179.stdout: debug client = 20 2023-10-19T12:43:15.293 INFO:teuthology.orchestra.run.smithi179.stdout: debug finisher = 20 2023-10-19T12:43:15.294 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-10-19T12:43:15.294 INFO:teuthology.orchestra.run.smithi179.stdout:[mon] 2023-10-19T12:43:15.294 INFO:teuthology.orchestra.run.smithi179.stdout: debug ms = 1 2023-10-19T12:43:15.294 INFO:teuthology.orchestra.run.smithi179.stdout: debug mon = 20 2023-10-19T12:43:15.294 INFO:teuthology.orchestra.run.smithi179.stdout: debug paxos = 20 2023-10-19T12:43:15.295 INFO:teuthology.orchestra.run.smithi179.stdout: debug auth = 20 2023-10-19T12:43:15.295 INFO:teuthology.orchestra.run.smithi179.stdout: mon data avail warn = 5 2023-10-19T12:43:15.295 INFO:teuthology.orchestra.run.smithi179.stdout: mon mgr mkfs grace = 240 2023-10-19T12:43:15.295 INFO:teuthology.orchestra.run.smithi179.stdout: mon reweight min pgs per osd = 4 2023-10-19T12:43:15.295 INFO:teuthology.orchestra.run.smithi179.stdout: mon osd reporter subtree level = osd 2023-10-19T12:43:15.295 INFO:teuthology.orchestra.run.smithi179.stdout: mon osd prime pg temp = true 2023-10-19T12:43:15.296 INFO:teuthology.orchestra.run.smithi179.stdout: mon reweight min bytes per osd = 10 2023-10-19T12:43:15.296 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-10-19T12:43:15.296 INFO:teuthology.orchestra.run.smithi179.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-19T12:43:15.296 INFO:teuthology.orchestra.run.smithi179.stdout: auth mon ticket ttl = 660 # 11m 2023-10-19T12:43:15.296 INFO:teuthology.orchestra.run.smithi179.stdout: auth service ticket ttl = 240 # 4m 2023-10-19T12:43:15.296 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-10-19T12:43:15.297 INFO:teuthology.orchestra.run.smithi179.stdout: # don't complain about insecure global_id in the test suite 2023-10-19T12:43:15.297 INFO:teuthology.orchestra.run.smithi179.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-19T12:43:15.297 INFO:teuthology.orchestra.run.smithi179.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-19T12:43:15.297 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-10-19T12:43:15.297 INFO:teuthology.orchestra.run.smithi179.stdout: # 1m isn't quite enough 2023-10-19T12:43:15.297 INFO:teuthology.orchestra.run.smithi179.stdout: mon_down_mkfs_grace = 2m 2023-10-19T12:43:15.298 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-10-19T12:43:15.298 INFO:teuthology.orchestra.run.smithi179.stdout: mon_warn_on_filestore_osds = false 2023-10-19T12:43:15.298 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-10-19T12:43:15.298 INFO:teuthology.orchestra.run.smithi179.stdout:[client] 2023-10-19T12:43:15.298 INFO:teuthology.orchestra.run.smithi179.stdout: rgw cache enabled = true 2023-10-19T12:43:15.299 INFO:teuthology.orchestra.run.smithi179.stdout: rgw enable ops log = true 2023-10-19T12:43:15.299 INFO:teuthology.orchestra.run.smithi179.stdout: rgw enable usage log = true 2023-10-19T12:43:15.299 INFO:teuthology.orchestra.run.smithi179.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-19T12:43:15.299 INFO:teuthology.orchestra.run.smithi179.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-19T12:43:15.299 INFO:teuthology.orchestra.run.smithi179.stdout: pid file = "" 2023-10-19T12:43:15.299 INFO:teuthology.orchestra.run.smithi179.stdout:[mon.a] 2023-10-19T12:43:15.299 INFO:teuthology.orchestra.run.smithi179.stdout:[mon.b] 2023-10-19T12:43:15.300 INFO:teuthology.orchestra.run.smithi179.stdout:[mon.c] 2023-10-19T12:43:15.300 INFO:teuthology.orchestra.run.smithi179.stdout:[mds.a] 2023-10-19T12:43:15.300 INFO:teuthology.orchestra.run.smithi179.stdout:[mds.c] 2023-10-19T12:43:15.300 INFO:teuthology.orchestra.run.smithi179.stdout:[mds.b] 2023-10-19T12:43:15.300 INFO:teuthology.orchestra.run.smithi179.stdout:[mds.d] 2023-10-19T12:43:15.302 INFO:teuthology.orchestra.run.smithi057.stdout:[global] 2023-10-19T12:43:15.302 INFO:teuthology.orchestra.run.smithi057.stdout: chdir = "" 2023-10-19T12:43:15.302 INFO:teuthology.orchestra.run.smithi057.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-19T12:43:15.302 INFO:teuthology.orchestra.run.smithi057.stdout: auth supported = cephx 2023-10-19T12:43:15.303 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:43:15.303 INFO:teuthology.orchestra.run.smithi057.stdout: filestore xattr use omap = true 2023-10-19T12:43:15.303 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:43:15.303 INFO:teuthology.orchestra.run.smithi057.stdout: mon clock drift allowed = 1.000 2023-10-19T12:43:15.303 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:43:15.303 INFO:teuthology.orchestra.run.smithi057.stdout: osd crush chooseleaf type = 0 2023-10-19T12:43:15.304 INFO:teuthology.orchestra.run.smithi057.stdout: auth debug = true 2023-10-19T12:43:15.304 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:43:15.304 INFO:teuthology.orchestra.run.smithi057.stdout: ms die on old message = true 2023-10-19T12:43:15.304 INFO:teuthology.orchestra.run.smithi057.stdout: ms die on bug = true 2023-10-19T12:43:15.304 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:43:15.305 INFO:teuthology.orchestra.run.smithi057.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-19T12:43:15.305 INFO:teuthology.orchestra.run.smithi057.stdout: mon pg warn max object skew = 0 2023-10-19T12:43:15.305 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:43:15.305 INFO:teuthology.orchestra.run.smithi057.stdout: # disable pg_autoscaler by default for new pools 2023-10-19T12:43:15.305 INFO:teuthology.orchestra.run.smithi057.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-19T12:43:15.306 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:43:15.306 INFO:teuthology.orchestra.run.smithi057.stdout: osd pool default size = 2 2023-10-19T12:43:15.306 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:43:15.306 INFO:teuthology.orchestra.run.smithi057.stdout: mon osd allow primary affinity = true 2023-10-19T12:43:15.306 INFO:teuthology.orchestra.run.smithi057.stdout: mon osd allow pg remap = true 2023-10-19T12:43:15.307 INFO:teuthology.orchestra.run.smithi057.stdout: mon warn on legacy crush tunables = false 2023-10-19T12:43:15.307 INFO:teuthology.orchestra.run.smithi057.stdout: mon warn on crush straw calc version zero = false 2023-10-19T12:43:15.307 INFO:teuthology.orchestra.run.smithi057.stdout: mon warn on no sortbitwise = false 2023-10-19T12:43:15.307 INFO:teuthology.orchestra.run.smithi057.stdout: mon warn on osd down out interval zero = false 2023-10-19T12:43:15.307 INFO:teuthology.orchestra.run.smithi057.stdout: mon warn on too few osds = false 2023-10-19T12:43:15.308 INFO:teuthology.orchestra.run.smithi057.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-19T12:43:15.308 INFO:teuthology.orchestra.run.smithi057.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-19T12:43:15.308 INFO:teuthology.orchestra.run.smithi057.stdout: mon_allow_pool_size_one = true 2023-10-19T12:43:15.308 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:43:15.308 INFO:teuthology.orchestra.run.smithi057.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-19T12:43:15.309 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:43:15.309 INFO:teuthology.orchestra.run.smithi057.stdout: osd default data pool replay window = 5 2023-10-19T12:43:15.309 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:43:15.309 INFO:teuthology.orchestra.run.smithi057.stdout: mon allow pool delete = true 2023-10-19T12:43:15.309 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:43:15.309 INFO:teuthology.orchestra.run.smithi057.stdout: mon cluster log file level = debug 2023-10-19T12:43:15.310 INFO:teuthology.orchestra.run.smithi057.stdout: debug asserts on shutdown = true 2023-10-19T12:43:15.310 INFO:teuthology.orchestra.run.smithi057.stdout: mon health detail to clog = false 2023-10-19T12:43:15.310 INFO:teuthology.orchestra.run.smithi057.stdout: mon host = "172.21.15.57,172.21.15.179,[v2:172.21.15.179:3301,v1:172.21.15.179:6790]" 2023-10-19T12:43:15.310 INFO:teuthology.orchestra.run.smithi057.stdout: fsid = 1e9beb19-3260-4dd7-85ef-5b399ce774ac 2023-10-19T12:43:15.310 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:43:15.311 INFO:teuthology.orchestra.run.smithi057.stdout:[osd] 2023-10-19T12:43:15.311 INFO:teuthology.orchestra.run.smithi057.stdout: osd journal size = 100 2023-10-19T12:43:15.311 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:43:15.311 INFO:teuthology.orchestra.run.smithi057.stdout: osd scrub load threshold = 5.0 2023-10-19T12:43:15.311 INFO:teuthology.orchestra.run.smithi057.stdout: osd scrub max interval = 600 2023-10-19T12:43:15.311 INFO:teuthology.orchestra.run.smithi057.stdout: osd mclock profile = high_recovery_ops 2023-10-19T12:43:15.312 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:43:15.312 INFO:teuthology.orchestra.run.smithi057.stdout: osd recover clone overlap = true 2023-10-19T12:43:15.312 INFO:teuthology.orchestra.run.smithi057.stdout: osd recovery max chunk = 1048576 2023-10-19T12:43:15.312 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:43:15.312 INFO:teuthology.orchestra.run.smithi057.stdout: osd debug shutdown = true 2023-10-19T12:43:15.312 INFO:teuthology.orchestra.run.smithi057.stdout: osd debug op order = true 2023-10-19T12:43:15.313 INFO:teuthology.orchestra.run.smithi057.stdout: osd debug verify stray on activate = true 2023-10-19T12:43:15.313 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:43:15.313 INFO:teuthology.orchestra.run.smithi057.stdout: osd open classes on start = true 2023-10-19T12:43:15.313 INFO:teuthology.orchestra.run.smithi057.stdout: osd debug pg log writeout = true 2023-10-19T12:43:15.313 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:43:15.314 INFO:teuthology.orchestra.run.smithi057.stdout: osd deep scrub update digest min age = 30 2023-10-19T12:43:15.314 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:43:15.314 INFO:teuthology.orchestra.run.smithi057.stdout: osd map max advance = 10 2023-10-19T12:43:15.314 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:43:15.314 INFO:teuthology.orchestra.run.smithi057.stdout: journal zero on create = true 2023-10-19T12:43:15.315 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:43:15.315 INFO:teuthology.orchestra.run.smithi057.stdout: filestore ondisk finisher threads = 3 2023-10-19T12:43:15.315 INFO:teuthology.orchestra.run.smithi057.stdout: filestore apply finisher threads = 3 2023-10-19T12:43:15.315 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:43:15.315 INFO:teuthology.orchestra.run.smithi057.stdout: bdev debug aio = true 2023-10-19T12:43:15.315 INFO:teuthology.orchestra.run.smithi057.stdout: osd debug misdirected ops = true 2023-10-19T12:43:15.316 INFO:teuthology.orchestra.run.smithi057.stdout: bluestore block size = 96636764160 2023-10-19T12:43:15.316 INFO:teuthology.orchestra.run.smithi057.stdout: bluestore fsck on mount = True 2023-10-19T12:43:15.316 INFO:teuthology.orchestra.run.smithi057.stdout: debug bluefs = 20 2023-10-19T12:43:15.316 INFO:teuthology.orchestra.run.smithi057.stdout: debug bluestore = 20 2023-10-19T12:43:15.316 INFO:teuthology.orchestra.run.smithi057.stdout: debug ms = 1 2023-10-19T12:43:15.317 INFO:teuthology.orchestra.run.smithi057.stdout: debug osd = 20 2023-10-19T12:43:15.317 INFO:teuthology.orchestra.run.smithi057.stdout: debug rocksdb = 10 2023-10-19T12:43:15.317 INFO:teuthology.orchestra.run.smithi057.stdout: mon osd backfillfull_ratio = 0.85 2023-10-19T12:43:15.317 INFO:teuthology.orchestra.run.smithi057.stdout: mon osd full ratio = 0.9 2023-10-19T12:43:15.317 INFO:teuthology.orchestra.run.smithi057.stdout: mon osd nearfull ratio = 0.8 2023-10-19T12:43:15.317 INFO:teuthology.orchestra.run.smithi057.stdout: osd failsafe full ratio = 0.95 2023-10-19T12:43:15.318 INFO:teuthology.orchestra.run.smithi057.stdout: osd objectstore = bluestore 2023-10-19T12:43:15.318 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:43:15.318 INFO:teuthology.orchestra.run.smithi057.stdout:[mgr] 2023-10-19T12:43:15.318 INFO:teuthology.orchestra.run.smithi057.stdout: debug ms = 1 2023-10-19T12:43:15.318 INFO:teuthology.orchestra.run.smithi057.stdout: debug mgr = 20 2023-10-19T12:43:15.319 INFO:teuthology.orchestra.run.smithi057.stdout: debug mon = 20 2023-10-19T12:43:15.319 INFO:teuthology.orchestra.run.smithi057.stdout: debug auth = 20 2023-10-19T12:43:15.319 INFO:teuthology.orchestra.run.smithi057.stdout: mon reweight min pgs per osd = 4 2023-10-19T12:43:15.319 INFO:teuthology.orchestra.run.smithi057.stdout: mon reweight min bytes per osd = 10 2023-10-19T12:43:15.319 INFO:teuthology.orchestra.run.smithi057.stdout: mgr/telemetry/nag = false 2023-10-19T12:43:15.320 INFO:teuthology.orchestra.run.smithi057.stdout: debug client = 20 2023-10-19T12:43:15.320 INFO:teuthology.orchestra.run.smithi057.stdout: debug finisher = 20 2023-10-19T12:43:15.320 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:43:15.320 INFO:teuthology.orchestra.run.smithi057.stdout:[mon] 2023-10-19T12:43:15.320 INFO:teuthology.orchestra.run.smithi057.stdout: debug ms = 1 2023-10-19T12:43:15.320 INFO:teuthology.orchestra.run.smithi057.stdout: debug mon = 20 2023-10-19T12:43:15.321 INFO:teuthology.orchestra.run.smithi057.stdout: debug paxos = 20 2023-10-19T12:43:15.321 INFO:teuthology.orchestra.run.smithi057.stdout: debug auth = 20 2023-10-19T12:43:15.321 INFO:teuthology.orchestra.run.smithi057.stdout: mon data avail warn = 5 2023-10-19T12:43:15.321 INFO:teuthology.orchestra.run.smithi057.stdout: mon mgr mkfs grace = 240 2023-10-19T12:43:15.321 INFO:teuthology.orchestra.run.smithi057.stdout: mon reweight min pgs per osd = 4 2023-10-19T12:43:15.321 INFO:teuthology.orchestra.run.smithi057.stdout: mon osd reporter subtree level = osd 2023-10-19T12:43:15.322 INFO:teuthology.orchestra.run.smithi057.stdout: mon osd prime pg temp = true 2023-10-19T12:43:15.322 INFO:teuthology.orchestra.run.smithi057.stdout: mon reweight min bytes per osd = 10 2023-10-19T12:43:15.322 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:43:15.322 INFO:teuthology.orchestra.run.smithi057.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-19T12:43:15.322 INFO:teuthology.orchestra.run.smithi057.stdout: auth mon ticket ttl = 660 # 11m 2023-10-19T12:43:15.322 INFO:teuthology.orchestra.run.smithi057.stdout: auth service ticket ttl = 240 # 4m 2023-10-19T12:43:15.323 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:43:15.323 INFO:teuthology.orchestra.run.smithi057.stdout: # don't complain about insecure global_id in the test suite 2023-10-19T12:43:15.323 INFO:teuthology.orchestra.run.smithi057.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-19T12:43:15.323 INFO:teuthology.orchestra.run.smithi057.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-19T12:43:15.323 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:43:15.323 INFO:teuthology.orchestra.run.smithi057.stdout: # 1m isn't quite enough 2023-10-19T12:43:15.324 INFO:teuthology.orchestra.run.smithi057.stdout: mon_down_mkfs_grace = 2m 2023-10-19T12:43:15.324 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:43:15.324 INFO:teuthology.orchestra.run.smithi057.stdout: mon_warn_on_filestore_osds = false 2023-10-19T12:43:15.324 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:43:15.324 INFO:teuthology.orchestra.run.smithi057.stdout:[client] 2023-10-19T12:43:15.325 INFO:teuthology.orchestra.run.smithi057.stdout: rgw cache enabled = true 2023-10-19T12:43:15.325 INFO:teuthology.orchestra.run.smithi057.stdout: rgw enable ops log = true 2023-10-19T12:43:15.325 INFO:teuthology.orchestra.run.smithi057.stdout: rgw enable usage log = true 2023-10-19T12:43:15.325 INFO:teuthology.orchestra.run.smithi057.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-19T12:43:15.325 INFO:teuthology.orchestra.run.smithi057.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-19T12:43:15.325 INFO:teuthology.orchestra.run.smithi057.stdout: pid file = "" 2023-10-19T12:43:15.326 INFO:teuthology.orchestra.run.smithi057.stdout:[mon.a] 2023-10-19T12:43:15.326 INFO:teuthology.orchestra.run.smithi057.stdout:[mon.b] 2023-10-19T12:43:15.326 INFO:teuthology.orchestra.run.smithi057.stdout:[mon.c] 2023-10-19T12:43:15.326 INFO:teuthology.orchestra.run.smithi057.stdout:[mds.a] 2023-10-19T12:43:15.326 INFO:teuthology.orchestra.run.smithi057.stdout:[mds.c] 2023-10-19T12:43:15.326 INFO:teuthology.orchestra.run.smithi057.stdout:[mds.b] 2023-10-19T12:43:15.327 INFO:teuthology.orchestra.run.smithi057.stdout:[mds.d] 2023-10-19T12:43:15.327 INFO:tasks.ceph:Creating admin key on mon.a... 2023-10-19T12:43:15.327 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-10-19T12:43:15.401 INFO:tasks.ceph:Copying monmap to all nodes... 2023-10-19T12:43:15.401 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T12:43:15.401 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-10-19T12:43:15.416 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T12:43:15.416 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-10-19T12:43:15.472 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi057.front.sepia.ceph.com 2023-10-19T12:43:15.472 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T12:43:15.472 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/etc/ceph/ceph.keyring 2023-10-19T12:43:15.473 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-19T12:43:15.557 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T12:43:15.558 DEBUG:teuthology.orchestra.run.smithi057:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-10-19T12:43:15.612 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi179.front.sepia.ceph.com 2023-10-19T12:43:15.613 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-19T12:43:15.613 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/etc/ceph/ceph.keyring 2023-10-19T12:43:15.613 DEBUG:teuthology.orchestra.run.smithi179:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-19T12:43:15.653 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-19T12:43:15.653 DEBUG:teuthology.orchestra.run.smithi179:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-10-19T12:43:15.707 INFO:tasks.ceph:Setting up mon nodes... 2023-10-19T12:43:15.708 INFO:tasks.ceph:Setting up mgr nodes... 2023-10-19T12:43:15.708 DEBUG:teuthology.orchestra.run.smithi057:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-10-19T12:43:15.770 INFO:teuthology.orchestra.run.smithi057.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-10-19T12:43:15.772 DEBUG:teuthology.orchestra.run.smithi179:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-10-19T12:43:15.847 INFO:teuthology.orchestra.run.smithi179.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-10-19T12:43:15.849 INFO:tasks.ceph:Setting up mds nodes... 2023-10-19T12:43:15.850 DEBUG:teuthology.orchestra.run.smithi057:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2023-10-19T12:43:15.912 INFO:teuthology.orchestra.run.smithi057.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2023-10-19T12:43:15.914 DEBUG:teuthology.orchestra.run.smithi057:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2023-10-19T12:43:15.944 DEBUG:teuthology.orchestra.run.smithi057:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2023-10-19T12:43:16.046 INFO:teuthology.orchestra.run.smithi057.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2023-10-19T12:43:16.048 DEBUG:teuthology.orchestra.run.smithi057:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2023-10-19T12:43:16.078 DEBUG:teuthology.orchestra.run.smithi179:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2023-10-19T12:43:16.132 INFO:teuthology.orchestra.run.smithi179.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2023-10-19T12:43:16.135 DEBUG:teuthology.orchestra.run.smithi179:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2023-10-19T12:43:16.161 DEBUG:teuthology.orchestra.run.smithi179:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2023-10-19T12:43:16.256 INFO:teuthology.orchestra.run.smithi179.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2023-10-19T12:43:16.258 DEBUG:teuthology.orchestra.run.smithi179:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2023-10-19T12:43:16.285 INFO:tasks.ceph_client:Setting up client nodes... 2023-10-19T12:43:16.285 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-10-19T12:43:16.332 INFO:teuthology.orchestra.run.smithi057.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-10-19T12:43:16.350 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2023-10-19T12:43:16.396 INFO:teuthology.orchestra.run.smithi057.stdout:creating /etc/ceph/ceph.client.2.keyring 2023-10-19T12:43:16.414 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2023-10-19T12:43:16.456 INFO:teuthology.orchestra.run.smithi179.stdout:creating /etc/ceph/ceph.client.1.keyring 2023-10-19T12:43:16.471 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2023-10-19T12:43:16.553 INFO:teuthology.orchestra.run.smithi179.stdout:creating /etc/ceph/ceph.client.3.keyring 2023-10-19T12:43:16.568 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-10-19T12:43:16.568 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi057.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi179.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-10-19T12:43:16.569 DEBUG:teuthology.orchestra.run.smithi057:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-10-19T12:43:16.600 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-19T12:43:16.600 INFO:tasks.ceph:role: osd.0 2023-10-19T12:43:16.601 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi057.front.sepia.ceph.com 2023-10-19T12:43:16.601 DEBUG:teuthology.orchestra.run.smithi057:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-19T12:43:16.698 INFO:teuthology.orchestra.run.smithi057.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-19T12:43:16.698 INFO:teuthology.orchestra.run.smithi057.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-19T12:43:16.698 INFO:teuthology.orchestra.run.smithi057.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-19T12:43:16.699 INFO:teuthology.orchestra.run.smithi057.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-19T12:43:16.699 INFO:teuthology.orchestra.run.smithi057.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-19T12:43:16.699 INFO:teuthology.orchestra.run.smithi057.stdout: = sunit=0 swidth=0 blks 2023-10-19T12:43:16.699 INFO:teuthology.orchestra.run.smithi057.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-19T12:43:16.699 INFO:teuthology.orchestra.run.smithi057.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-19T12:43:16.699 INFO:teuthology.orchestra.run.smithi057.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-19T12:43:16.700 INFO:teuthology.orchestra.run.smithi057.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-19T12:43:18.019 INFO:teuthology.orchestra.run.smithi057.stdout:Discarding blocks...Done. 2023-10-19T12:43:18.021 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi057.front.sepia.ceph.com -o noatime 2023-10-19T12:43:18.022 DEBUG:teuthology.orchestra.run.smithi057:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-10-19T12:43:18.064 DEBUG:teuthology.orchestra.run.smithi057:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-10-19T12:43:18.143 DEBUG:teuthology.orchestra.run.smithi057:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-10-19T12:43:18.217 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-19T12:43:18.218 INFO:tasks.ceph:role: osd.1 2023-10-19T12:43:18.218 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi057.front.sepia.ceph.com 2023-10-19T12:43:18.218 DEBUG:teuthology.orchestra.run.smithi057:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-19T12:43:18.318 INFO:teuthology.orchestra.run.smithi057.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-19T12:43:18.319 INFO:teuthology.orchestra.run.smithi057.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-19T12:43:18.319 INFO:teuthology.orchestra.run.smithi057.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-19T12:43:18.319 INFO:teuthology.orchestra.run.smithi057.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-19T12:43:18.319 INFO:teuthology.orchestra.run.smithi057.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-19T12:43:18.319 INFO:teuthology.orchestra.run.smithi057.stdout: = sunit=0 swidth=0 blks 2023-10-19T12:43:18.320 INFO:teuthology.orchestra.run.smithi057.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-19T12:43:18.320 INFO:teuthology.orchestra.run.smithi057.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-19T12:43:18.320 INFO:teuthology.orchestra.run.smithi057.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-19T12:43:18.320 INFO:teuthology.orchestra.run.smithi057.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-19T12:43:19.646 INFO:teuthology.orchestra.run.smithi057.stdout:Discarding blocks...Done. 2023-10-19T12:43:19.648 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi057.front.sepia.ceph.com -o noatime 2023-10-19T12:43:19.649 DEBUG:teuthology.orchestra.run.smithi057:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-10-19T12:43:19.692 DEBUG:teuthology.orchestra.run.smithi057:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-10-19T12:43:19.771 DEBUG:teuthology.orchestra.run.smithi057:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-10-19T12:43:19.845 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-19T12:43:19.845 INFO:tasks.ceph:role: osd.2 2023-10-19T12:43:19.846 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi057.front.sepia.ceph.com 2023-10-19T12:43:19.846 DEBUG:teuthology.orchestra.run.smithi057:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-19T12:43:19.946 INFO:teuthology.orchestra.run.smithi057.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-19T12:43:19.946 INFO:teuthology.orchestra.run.smithi057.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-19T12:43:19.946 INFO:teuthology.orchestra.run.smithi057.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-19T12:43:19.946 INFO:teuthology.orchestra.run.smithi057.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-19T12:43:19.947 INFO:teuthology.orchestra.run.smithi057.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-19T12:43:19.947 INFO:teuthology.orchestra.run.smithi057.stdout: = sunit=0 swidth=0 blks 2023-10-19T12:43:19.947 INFO:teuthology.orchestra.run.smithi057.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-19T12:43:19.947 INFO:teuthology.orchestra.run.smithi057.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-19T12:43:19.947 INFO:teuthology.orchestra.run.smithi057.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-19T12:43:19.948 INFO:teuthology.orchestra.run.smithi057.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-19T12:43:21.287 INFO:teuthology.orchestra.run.smithi057.stdout:Discarding blocks...Done. 2023-10-19T12:43:21.290 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi057.front.sepia.ceph.com -o noatime 2023-10-19T12:43:21.290 DEBUG:teuthology.orchestra.run.smithi057:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-10-19T12:43:21.331 DEBUG:teuthology.orchestra.run.smithi057:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-10-19T12:43:21.413 DEBUG:teuthology.orchestra.run.smithi057:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-10-19T12:43:21.484 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-19T12:43:21.484 INFO:tasks.ceph:role: osd.3 2023-10-19T12:43:21.485 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi057.front.sepia.ceph.com 2023-10-19T12:43:21.485 DEBUG:teuthology.orchestra.run.smithi057:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-10-19T12:43:21.584 INFO:teuthology.orchestra.run.smithi057.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-10-19T12:43:21.584 INFO:teuthology.orchestra.run.smithi057.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-19T12:43:21.584 INFO:teuthology.orchestra.run.smithi057.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-19T12:43:21.585 INFO:teuthology.orchestra.run.smithi057.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-19T12:43:21.585 INFO:teuthology.orchestra.run.smithi057.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-19T12:43:21.585 INFO:teuthology.orchestra.run.smithi057.stdout: = sunit=0 swidth=0 blks 2023-10-19T12:43:21.585 INFO:teuthology.orchestra.run.smithi057.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-19T12:43:21.585 INFO:teuthology.orchestra.run.smithi057.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-19T12:43:21.585 INFO:teuthology.orchestra.run.smithi057.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-19T12:43:21.586 INFO:teuthology.orchestra.run.smithi057.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-19T12:43:22.899 INFO:teuthology.orchestra.run.smithi057.stdout:Discarding blocks...Done. 2023-10-19T12:43:22.902 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi057.front.sepia.ceph.com -o noatime 2023-10-19T12:43:22.902 DEBUG:teuthology.orchestra.run.smithi057:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2023-10-19T12:43:22.944 DEBUG:teuthology.orchestra.run.smithi057:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-10-19T12:43:23.024 DEBUG:teuthology.orchestra.run.smithi057:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-19T12:43:23.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:23.123+0000 7f24d44f8500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-10-19T12:43:23.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:23.123+0000 7f24d44f8500 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-10-19T12:43:23.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:23.123+0000 7f24d44f8500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-19T12:43:23.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:23.123+0000 7f24d44f8500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-19T12:43:23.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:23.123+0000 7f24d44f8500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-19T12:43:23.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:23.125+0000 7f24d44f8500 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-10-19T12:43:26.005 DEBUG:teuthology.orchestra.run.smithi057:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-10-19T12:43:26.040 DEBUG:teuthology.orchestra.run.smithi057:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-19T12:43:26.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:26.133+0000 7f9f5c31a500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-10-19T12:43:26.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:26.134+0000 7f9f5c31a500 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-10-19T12:43:26.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:26.134+0000 7f9f5c31a500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-19T12:43:26.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:26.134+0000 7f9f5c31a500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-19T12:43:26.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:26.134+0000 7f9f5c31a500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-19T12:43:26.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:26.135+0000 7f9f5c31a500 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-10-19T12:43:29.003 DEBUG:teuthology.orchestra.run.smithi057:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-10-19T12:43:29.041 DEBUG:teuthology.orchestra.run.smithi057:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-19T12:43:29.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:29.133+0000 7f508d863500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-10-19T12:43:29.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:29.134+0000 7f508d863500 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-10-19T12:43:29.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:29.134+0000 7f508d863500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-19T12:43:29.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:29.134+0000 7f508d863500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-19T12:43:29.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:29.134+0000 7f508d863500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-19T12:43:29.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:29.135+0000 7f508d863500 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-10-19T12:43:32.009 DEBUG:teuthology.orchestra.run.smithi057:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-10-19T12:43:32.045 DEBUG:teuthology.orchestra.run.smithi057:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-19T12:43:32.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:32.140+0000 7fadf699b500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-10-19T12:43:32.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:32.140+0000 7fadf699b500 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-10-19T12:43:32.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:32.140+0000 7fadf699b500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-10-19T12:43:32.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:32.140+0000 7fadf699b500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-10-19T12:43:32.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:32.140+0000 7fadf699b500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-10-19T12:43:32.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:32.141+0000 7fadf699b500 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-10-19T12:43:35.025 DEBUG:teuthology.orchestra.run.smithi057:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-10-19T12:43:35.060 DEBUG:teuthology.orchestra.run.smithi179:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-10-19T12:43:35.091 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-19T12:43:35.091 INFO:tasks.ceph:role: osd.4 2023-10-19T12:43:35.092 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi179.front.sepia.ceph.com 2023-10-19T12:43:35.092 DEBUG:teuthology.orchestra.run.smithi179:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-19T12:43:35.170 INFO:teuthology.orchestra.run.smithi179.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-19T12:43:35.203 INFO:teuthology.orchestra.run.smithi179.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-19T12:43:35.204 INFO:teuthology.orchestra.run.smithi179.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-19T12:43:35.204 INFO:teuthology.orchestra.run.smithi179.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-19T12:43:35.204 INFO:teuthology.orchestra.run.smithi179.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-19T12:43:35.204 INFO:teuthology.orchestra.run.smithi179.stdout: = sunit=0 swidth=0 blks 2023-10-19T12:43:35.204 INFO:teuthology.orchestra.run.smithi179.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-19T12:43:35.205 INFO:teuthology.orchestra.run.smithi179.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-19T12:43:35.205 INFO:teuthology.orchestra.run.smithi179.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-19T12:43:35.205 INFO:teuthology.orchestra.run.smithi179.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-19T12:43:35.680 INFO:teuthology.orchestra.run.smithi179.stdout:Discarding blocks...Done. 2023-10-19T12:43:35.682 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi179.front.sepia.ceph.com -o noatime 2023-10-19T12:43:35.683 DEBUG:teuthology.orchestra.run.smithi179:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2023-10-19T12:43:35.822 DEBUG:teuthology.orchestra.run.smithi179:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-10-19T12:43:35.858 DEBUG:teuthology.orchestra.run.smithi179:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-10-19T12:43:35.927 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-19T12:43:35.927 INFO:tasks.ceph:role: osd.5 2023-10-19T12:43:35.927 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi179.front.sepia.ceph.com 2023-10-19T12:43:35.928 DEBUG:teuthology.orchestra.run.smithi179:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-19T12:43:36.002 INFO:teuthology.orchestra.run.smithi179.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-19T12:43:36.003 INFO:teuthology.orchestra.run.smithi179.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-19T12:43:36.003 INFO:teuthology.orchestra.run.smithi179.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-19T12:43:36.003 INFO:teuthology.orchestra.run.smithi179.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-19T12:43:36.003 INFO:teuthology.orchestra.run.smithi179.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-19T12:43:36.004 INFO:teuthology.orchestra.run.smithi179.stdout: = sunit=0 swidth=0 blks 2023-10-19T12:43:36.004 INFO:teuthology.orchestra.run.smithi179.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-19T12:43:36.004 INFO:teuthology.orchestra.run.smithi179.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-19T12:43:36.004 INFO:teuthology.orchestra.run.smithi179.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-19T12:43:36.004 INFO:teuthology.orchestra.run.smithi179.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-19T12:43:36.444 INFO:teuthology.orchestra.run.smithi179.stdout:Discarding blocks...Done. 2023-10-19T12:43:36.446 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi179.front.sepia.ceph.com -o noatime 2023-10-19T12:43:36.446 DEBUG:teuthology.orchestra.run.smithi179:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2023-10-19T12:43:36.540 DEBUG:teuthology.orchestra.run.smithi179:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-10-19T12:43:36.571 DEBUG:teuthology.orchestra.run.smithi179:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-10-19T12:43:36.638 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-19T12:43:36.639 INFO:tasks.ceph:role: osd.6 2023-10-19T12:43:36.639 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi179.front.sepia.ceph.com 2023-10-19T12:43:36.640 DEBUG:teuthology.orchestra.run.smithi179:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-19T12:43:36.713 INFO:teuthology.orchestra.run.smithi179.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-19T12:43:36.713 INFO:teuthology.orchestra.run.smithi179.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-19T12:43:36.714 INFO:teuthology.orchestra.run.smithi179.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-19T12:43:36.714 INFO:teuthology.orchestra.run.smithi179.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-19T12:43:36.714 INFO:teuthology.orchestra.run.smithi179.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-19T12:43:36.714 INFO:teuthology.orchestra.run.smithi179.stdout: = sunit=0 swidth=0 blks 2023-10-19T12:43:36.714 INFO:teuthology.orchestra.run.smithi179.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-19T12:43:36.715 INFO:teuthology.orchestra.run.smithi179.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-19T12:43:36.715 INFO:teuthology.orchestra.run.smithi179.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-19T12:43:36.715 INFO:teuthology.orchestra.run.smithi179.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-19T12:43:37.202 INFO:teuthology.orchestra.run.smithi179.stdout:Discarding blocks...Done. 2023-10-19T12:43:37.204 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi179.front.sepia.ceph.com -o noatime 2023-10-19T12:43:37.204 DEBUG:teuthology.orchestra.run.smithi179:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2023-10-19T12:43:37.342 DEBUG:teuthology.orchestra.run.smithi179:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2023-10-19T12:43:37.374 DEBUG:teuthology.orchestra.run.smithi179:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-10-19T12:43:37.441 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-19T12:43:37.441 INFO:tasks.ceph:role: osd.7 2023-10-19T12:43:37.442 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi179.front.sepia.ceph.com 2023-10-19T12:43:37.442 DEBUG:teuthology.orchestra.run.smithi179:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-10-19T12:43:37.515 INFO:teuthology.orchestra.run.smithi179.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-10-19T12:43:37.516 INFO:teuthology.orchestra.run.smithi179.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-19T12:43:37.516 INFO:teuthology.orchestra.run.smithi179.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-19T12:43:37.516 INFO:teuthology.orchestra.run.smithi179.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-19T12:43:37.516 INFO:teuthology.orchestra.run.smithi179.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-19T12:43:37.516 INFO:teuthology.orchestra.run.smithi179.stdout: = sunit=0 swidth=0 blks 2023-10-19T12:43:37.517 INFO:teuthology.orchestra.run.smithi179.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-19T12:43:37.517 INFO:teuthology.orchestra.run.smithi179.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-19T12:43:37.517 INFO:teuthology.orchestra.run.smithi179.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-19T12:43:37.517 INFO:teuthology.orchestra.run.smithi179.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-19T12:43:38.008 INFO:teuthology.orchestra.run.smithi179.stdout:Discarding blocks...Done. 2023-10-19T12:43:38.010 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi179.front.sepia.ceph.com -o noatime 2023-10-19T12:43:38.010 DEBUG:teuthology.orchestra.run.smithi179:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2023-10-19T12:43:38.148 DEBUG:teuthology.orchestra.run.smithi179:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2023-10-19T12:43:38.180 DEBUG:teuthology.orchestra.run.smithi179:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-19T12:43:38.273 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:38.270+0000 7f71eb907500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-10-19T12:43:38.274 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:38.270+0000 7f71eb907500 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-10-19T12:43:38.274 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:38.270+0000 7f71eb907500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-10-19T12:43:38.274 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:38.270+0000 7f71eb907500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-10-19T12:43:38.274 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:38.270+0000 7f71eb907500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-10-19T12:43:38.275 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:38.271+0000 7f71eb907500 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-10-19T12:43:41.119 DEBUG:teuthology.orchestra.run.smithi179:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-10-19T12:43:41.149 DEBUG:teuthology.orchestra.run.smithi179:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-19T12:43:41.238 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:41.235+0000 7f0a39419500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-10-19T12:43:41.238 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:41.235+0000 7f0a39419500 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-10-19T12:43:41.238 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:41.236+0000 7f0a39419500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-10-19T12:43:41.238 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:41.236+0000 7f0a39419500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-10-19T12:43:41.238 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:41.236+0000 7f0a39419500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-10-19T12:43:41.239 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:41.236+0000 7f0a39419500 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-10-19T12:43:43.274 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T12:43:43.277 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T12:43:44.080 DEBUG:teuthology.orchestra.run.smithi179:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-10-19T12:43:44.108 DEBUG:teuthology.orchestra.run.smithi179:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-19T12:43:44.198 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:44.195+0000 7fe33aeab500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-10-19T12:43:44.199 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:44.195+0000 7fe33aeab500 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-10-19T12:43:44.199 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:44.195+0000 7fe33aeab500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-10-19T12:43:44.199 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:44.195+0000 7fe33aeab500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-10-19T12:43:44.199 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:44.195+0000 7fe33aeab500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-10-19T12:43:44.200 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:44.196+0000 7fe33aeab500 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2023-10-19T12:43:47.052 DEBUG:teuthology.orchestra.run.smithi179:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-10-19T12:43:47.080 DEBUG:teuthology.orchestra.run.smithi179:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-19T12:43:47.167 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:47.164+0000 7f423c394500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-10-19T12:43:47.168 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:47.164+0000 7f423c394500 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-10-19T12:43:47.168 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:47.164+0000 7f423c394500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-10-19T12:43:47.168 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:47.164+0000 7f423c394500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-10-19T12:43:47.168 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:47.164+0000 7f423c394500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-10-19T12:43:47.169 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:47.165+0000 7f423c394500 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2023-10-19T12:43:50.016 DEBUG:teuthology.orchestra.run.smithi179:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-10-19T12:43:50.046 INFO:tasks.ceph:Reading keys from all nodes... 2023-10-19T12:43:50.047 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T12:43:50.047 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-10-19T12:43:50.084 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T12:43:50.085 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2023-10-19T12:43:50.158 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T12:43:50.158 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2023-10-19T12:43:50.230 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T12:43:50.230 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-10-19T12:43:50.301 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T12:43:50.302 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-10-19T12:43:50.373 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T12:43:50.373 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-10-19T12:43:50.444 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T12:43:50.444 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-10-19T12:43:50.514 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-19T12:43:50.515 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-10-19T12:43:50.546 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-19T12:43:50.547 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2023-10-19T12:43:50.617 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-19T12:43:50.617 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2023-10-19T12:43:50.686 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-19T12:43:50.687 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-10-19T12:43:50.756 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-19T12:43:50.757 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-10-19T12:43:50.825 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-19T12:43:50.825 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-10-19T12:43:50.893 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-19T12:43:50.893 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-10-19T12:43:50.960 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T12:43:50.961 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-10-19T12:43:50.978 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T12:43:50.978 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2023-10-19T12:43:51.034 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-19T12:43:51.034 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2023-10-19T12:43:51.049 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-19T12:43:51.050 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2023-10-19T12:43:51.105 INFO:tasks.ceph:Adding keys to all mons... 2023-10-19T12:43:51.106 DEBUG:teuthology.orchestra.run.smithi057:> sudo tee -a /etc/ceph/ceph.keyring 2023-10-19T12:43:51.108 DEBUG:teuthology.orchestra.run.smithi179:> sudo tee -a /etc/ceph/ceph.keyring 2023-10-19T12:43:51.150 INFO:teuthology.orchestra.run.smithi057.stdout:[mgr.y] 2023-10-19T12:43:51.150 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQBjJDFlxRzJLRAAGjG58o7qXAHvIEIRYwl6QA== 2023-10-19T12:43:51.151 INFO:teuthology.orchestra.run.smithi057.stdout:[mds.a] 2023-10-19T12:43:51.151 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQBjJDFlmbVANhAAP/ynHlU/OxaRq5lQoAa8Ng== 2023-10-19T12:43:51.151 INFO:teuthology.orchestra.run.smithi057.stdout:[mds.c] 2023-10-19T12:43:51.151 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQBkJDFlFkaoAhAA/PBizkJ4bm958oP4CmPuIQ== 2023-10-19T12:43:51.151 INFO:teuthology.orchestra.run.smithi057.stdout:[osd.0] 2023-10-19T12:43:51.151 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQBrJDFlCo9sBxAAQN2cID4nCQgRwGaGFEV8LQ== 2023-10-19T12:43:51.152 INFO:teuthology.orchestra.run.smithi057.stdout:[osd.1] 2023-10-19T12:43:51.152 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQBuJDFl4fsLCBAAp3Pi80yZtc5sju0GisEuVA== 2023-10-19T12:43:51.152 INFO:teuthology.orchestra.run.smithi057.stdout:[osd.2] 2023-10-19T12:43:51.152 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQBxJDFl+KAKCBAAd4N9cluARxPVfBLU+7Hd7A== 2023-10-19T12:43:51.152 INFO:teuthology.orchestra.run.smithi057.stdout:[osd.3] 2023-10-19T12:43:51.153 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQB0JDFlU/toCBAAP0Ou1zcHKNx5SfbybJkh4Q== 2023-10-19T12:43:51.153 INFO:teuthology.orchestra.run.smithi057.stdout:[mgr.x] 2023-10-19T12:43:51.153 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQBjJDFllFBfMhAA5+KPXWmYZqcESmLObaUbBA== 2023-10-19T12:43:51.153 INFO:teuthology.orchestra.run.smithi057.stdout:[mds.b] 2023-10-19T12:43:51.153 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQBkJDFl9l7RBxAAC0geI2tJ1SIZZr1++kS/BA== 2023-10-19T12:43:51.154 INFO:teuthology.orchestra.run.smithi057.stdout:[mds.d] 2023-10-19T12:43:51.154 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQBkJDFlx8skDxAAAfCHTYtYJgJik0WHSWeQfQ== 2023-10-19T12:43:51.154 INFO:teuthology.orchestra.run.smithi057.stdout:[osd.4] 2023-10-19T12:43:51.154 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQB6JDFlc6stEBAALkgNyxoiCTzZnKpBSPxI6w== 2023-10-19T12:43:51.155 INFO:teuthology.orchestra.run.smithi057.stdout:[osd.5] 2023-10-19T12:43:51.155 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQB9JDFlI3cNDhAAuG3tHJ6RvPwAzPE76wTIVQ== 2023-10-19T12:43:51.155 INFO:teuthology.orchestra.run.smithi057.stdout:[osd.6] 2023-10-19T12:43:51.155 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQCAJDFlLPumCxAA/iy08t/2N9rXfn71KtbSQw== 2023-10-19T12:43:51.155 INFO:teuthology.orchestra.run.smithi057.stdout:[osd.7] 2023-10-19T12:43:51.155 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQCDJDFlHmrbCRAA+hpWygGIJydfU6cHWlfx/Q== 2023-10-19T12:43:51.156 INFO:teuthology.orchestra.run.smithi057.stdout:[client.0] 2023-10-19T12:43:51.156 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQBkJDFlg9yuExAATxiWDfS+p07G1F0pORbJEg== 2023-10-19T12:43:51.156 INFO:teuthology.orchestra.run.smithi057.stdout:[client.2] 2023-10-19T12:43:51.156 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQBkJDFlMd+AFxAAS/UTESDt76GmrhhN2o/Fdw== 2023-10-19T12:43:51.156 INFO:teuthology.orchestra.run.smithi057.stdout:[client.1] 2023-10-19T12:43:51.157 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQBkJDFl9IwYGxAAHPituiJSPkHDuD0QyG4Hww== 2023-10-19T12:43:51.157 INFO:teuthology.orchestra.run.smithi057.stdout:[client.3] 2023-10-19T12:43:51.157 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQBkJDFlHZ3fIBAA42B7n8opLS5smXIwJyVYzw== 2023-10-19T12:43:51.171 INFO:teuthology.orchestra.run.smithi179.stdout:[mgr.y] 2023-10-19T12:43:51.171 INFO:teuthology.orchestra.run.smithi179.stdout: key = AQBjJDFlxRzJLRAAGjG58o7qXAHvIEIRYwl6QA== 2023-10-19T12:43:51.171 INFO:teuthology.orchestra.run.smithi179.stdout:[mds.a] 2023-10-19T12:43:51.171 INFO:teuthology.orchestra.run.smithi179.stdout: key = AQBjJDFlmbVANhAAP/ynHlU/OxaRq5lQoAa8Ng== 2023-10-19T12:43:51.172 INFO:teuthology.orchestra.run.smithi179.stdout:[mds.c] 2023-10-19T12:43:51.172 INFO:teuthology.orchestra.run.smithi179.stdout: key = AQBkJDFlFkaoAhAA/PBizkJ4bm958oP4CmPuIQ== 2023-10-19T12:43:51.172 INFO:teuthology.orchestra.run.smithi179.stdout:[osd.0] 2023-10-19T12:43:51.173 INFO:teuthology.orchestra.run.smithi179.stdout: key = AQBrJDFlCo9sBxAAQN2cID4nCQgRwGaGFEV8LQ== 2023-10-19T12:43:51.173 INFO:teuthology.orchestra.run.smithi179.stdout:[osd.1] 2023-10-19T12:43:51.173 INFO:teuthology.orchestra.run.smithi179.stdout: key = AQBuJDFl4fsLCBAAp3Pi80yZtc5sju0GisEuVA== 2023-10-19T12:43:51.173 INFO:teuthology.orchestra.run.smithi179.stdout:[osd.2] 2023-10-19T12:43:51.174 INFO:teuthology.orchestra.run.smithi179.stdout: key = AQBxJDFl+KAKCBAAd4N9cluARxPVfBLU+7Hd7A== 2023-10-19T12:43:51.174 INFO:teuthology.orchestra.run.smithi179.stdout:[osd.3] 2023-10-19T12:43:51.175 INFO:teuthology.orchestra.run.smithi179.stdout: key = AQB0JDFlU/toCBAAP0Ou1zcHKNx5SfbybJkh4Q== 2023-10-19T12:43:51.175 INFO:teuthology.orchestra.run.smithi179.stdout:[mgr.x] 2023-10-19T12:43:51.175 INFO:teuthology.orchestra.run.smithi179.stdout: key = AQBjJDFllFBfMhAA5+KPXWmYZqcESmLObaUbBA== 2023-10-19T12:43:51.175 INFO:teuthology.orchestra.run.smithi179.stdout:[mds.b] 2023-10-19T12:43:51.175 INFO:teuthology.orchestra.run.smithi179.stdout: key = AQBkJDFl9l7RBxAAC0geI2tJ1SIZZr1++kS/BA== 2023-10-19T12:43:51.175 INFO:teuthology.orchestra.run.smithi179.stdout:[mds.d] 2023-10-19T12:43:51.176 INFO:teuthology.orchestra.run.smithi179.stdout: key = AQBkJDFlx8skDxAAAfCHTYtYJgJik0WHSWeQfQ== 2023-10-19T12:43:51.176 INFO:teuthology.orchestra.run.smithi179.stdout:[osd.4] 2023-10-19T12:43:51.176 INFO:teuthology.orchestra.run.smithi179.stdout: key = AQB6JDFlc6stEBAALkgNyxoiCTzZnKpBSPxI6w== 2023-10-19T12:43:51.176 INFO:teuthology.orchestra.run.smithi179.stdout:[osd.5] 2023-10-19T12:43:51.176 INFO:teuthology.orchestra.run.smithi179.stdout: key = AQB9JDFlI3cNDhAAuG3tHJ6RvPwAzPE76wTIVQ== 2023-10-19T12:43:51.177 INFO:teuthology.orchestra.run.smithi179.stdout:[osd.6] 2023-10-19T12:43:51.177 INFO:teuthology.orchestra.run.smithi179.stdout: key = AQCAJDFlLPumCxAA/iy08t/2N9rXfn71KtbSQw== 2023-10-19T12:43:51.177 INFO:teuthology.orchestra.run.smithi179.stdout:[osd.7] 2023-10-19T12:43:51.177 INFO:teuthology.orchestra.run.smithi179.stdout: key = AQCDJDFlHmrbCRAA+hpWygGIJydfU6cHWlfx/Q== 2023-10-19T12:43:51.177 INFO:teuthology.orchestra.run.smithi179.stdout:[client.0] 2023-10-19T12:43:51.177 INFO:teuthology.orchestra.run.smithi179.stdout: key = AQBkJDFlg9yuExAATxiWDfS+p07G1F0pORbJEg== 2023-10-19T12:43:51.178 INFO:teuthology.orchestra.run.smithi179.stdout:[client.2] 2023-10-19T12:43:51.178 INFO:teuthology.orchestra.run.smithi179.stdout: key = AQBkJDFlMd+AFxAAS/UTESDt76GmrhhN2o/Fdw== 2023-10-19T12:43:51.178 INFO:teuthology.orchestra.run.smithi179.stdout:[client.1] 2023-10-19T12:43:51.178 INFO:teuthology.orchestra.run.smithi179.stdout: key = AQBkJDFl9IwYGxAAHPituiJSPkHDuD0QyG4Hww== 2023-10-19T12:43:51.178 INFO:teuthology.orchestra.run.smithi179.stdout:[client.3] 2023-10-19T12:43:51.179 INFO:teuthology.orchestra.run.smithi179.stdout: key = AQBkJDFlHZ3fIBAA42B7n8opLS5smXIwJyVYzw== 2023-10-19T12:43:51.180 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-19T12:43:51.200 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-19T12:43:51.272 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-19T12:43:51.275 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-19T12:43:51.332 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-19T12:43:51.334 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-19T12:43:51.391 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-19T12:43:51.393 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-19T12:43:51.447 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-19T12:43:51.449 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-19T12:43:51.503 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-19T12:43:51.505 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-19T12:43:51.559 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-19T12:43:51.561 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-19T12:43:51.615 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-19T12:43:51.617 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-19T12:43:51.671 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-19T12:43:51.674 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-19T12:43:51.731 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-19T12:43:51.733 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-19T12:43:51.790 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-19T12:43:51.792 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-19T12:43:51.846 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-19T12:43:51.849 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-19T12:43:51.902 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-19T12:43:51.905 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-19T12:43:51.958 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-19T12:43:51.961 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-19T12:43:52.014 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-19T12:43:52.016 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-19T12:43:52.073 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-19T12:43:52.075 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-19T12:43:52.132 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-19T12:43:52.135 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-19T12:43:52.192 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-19T12:43:52.195 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-19T12:43:52.251 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-10-19T12:43:52.251 DEBUG:teuthology.orchestra.run.smithi057:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-10-19T12:43:52.281 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-19T12:43:52.388 DEBUG:teuthology.orchestra.run.smithi057:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-10-19T12:43:52.419 DEBUG:teuthology.orchestra.run.smithi179:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-10-19T12:43:52.446 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-19T12:43:52.542 DEBUG:teuthology.orchestra.run.smithi179:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-10-19T12:43:52.568 DEBUG:teuthology.orchestra.run.smithi179:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-10-19T12:43:52.635 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-19T12:43:52.731 DEBUG:teuthology.orchestra.run.smithi179:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-10-19T12:43:52.759 DEBUG:teuthology.orchestra.run.smithi057:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-10-19T12:43:52.761 DEBUG:teuthology.orchestra.run.smithi179:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-10-19T12:43:52.813 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-10-19T12:43:52.814 INFO:tasks.ceph.mon.a:Restarting daemon 2023-10-19T12:43:52.814 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-10-19T12:43:52.817 INFO:tasks.ceph.mon.a:Started 2023-10-19T12:43:52.817 INFO:tasks.ceph.mon.b:Restarting daemon 2023-10-19T12:43:52.818 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-10-19T12:43:52.855 INFO:tasks.ceph.mon.b:Started 2023-10-19T12:43:52.856 INFO:tasks.ceph.mon.c:Restarting daemon 2023-10-19T12:43:52.856 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-10-19T12:43:52.858 INFO:tasks.ceph.mon.c:Started 2023-10-19T12:43:52.858 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-10-19T12:43:52.859 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-10-19T12:43:52.859 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2023-10-19T12:43:52.861 INFO:tasks.ceph.mgr.y:Started 2023-10-19T12:43:52.862 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-10-19T12:43:52.862 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-10-19T12:43:52.864 INFO:tasks.ceph.mgr.x:Started 2023-10-19T12:43:52.865 INFO:tasks.ceph:setting config [client] client_mount_timeout = 600 2023-10-19T12:43:52.865 DEBUG:teuthology.orchestra.run.smithi057:> sudo ceph --cluster ceph config set client client_mount_timeout 600 2023-10-19T12:43:52.867 INFO:tasks.ceph:setting config [client] debug_client = 20 2023-10-19T12:43:52.867 DEBUG:teuthology.orchestra.run.smithi057:> sudo ceph --cluster ceph config set client debug_client 20 2023-10-19T12:43:52.911 INFO:tasks.ceph:setting config [client] debug_ms = 1 2023-10-19T12:43:52.911 DEBUG:teuthology.orchestra.run.smithi057:> sudo ceph --cluster ceph config set client debug_ms 1 2023-10-19T12:43:52.955 INFO:tasks.ceph:setting config [client] rados_mon_op_timeout = 900 2023-10-19T12:43:52.955 DEBUG:teuthology.orchestra.run.smithi057:> sudo ceph --cluster ceph config set client rados_mon_op_timeout 900 2023-10-19T12:43:52.998 INFO:tasks.ceph:setting config [client] rados_osd_op_timeout = 900 2023-10-19T12:43:52.998 DEBUG:teuthology.orchestra.run.smithi057:> sudo ceph --cluster ceph config set client rados_osd_op_timeout 900 2023-10-19T12:43:53.042 INFO:tasks.ceph:setting config [mds] debug_mds = 20 2023-10-19T12:43:53.042 DEBUG:teuthology.orchestra.run.smithi057:> sudo ceph --cluster ceph config set mds debug_mds 20 2023-10-19T12:43:53.085 INFO:tasks.ceph:setting config [mds] debug_mds_balancer = 20 2023-10-19T12:43:53.086 DEBUG:teuthology.orchestra.run.smithi057:> sudo ceph --cluster ceph config set mds debug_mds_balancer 20 2023-10-19T12:43:53.128 INFO:tasks.ceph:setting config [mds] debug_ms = 1 2023-10-19T12:43:53.129 DEBUG:teuthology.orchestra.run.smithi057:> sudo ceph --cluster ceph config set mds debug_ms 1 2023-10-19T12:43:53.171 INFO:tasks.ceph:setting config [mds] mds_bal_fragment_size_max = 10000 2023-10-19T12:43:53.172 DEBUG:teuthology.orchestra.run.smithi057:> sudo ceph --cluster ceph config set mds mds_bal_fragment_size_max 10000 2023-10-19T12:43:53.214 INFO:tasks.ceph:setting config [mds] mds_bal_merge_size = 5 2023-10-19T12:43:53.215 DEBUG:teuthology.orchestra.run.smithi057:> sudo ceph --cluster ceph config set mds mds_bal_merge_size 5 2023-10-19T12:43:53.260 INFO:tasks.ceph:setting config [mds] mds_bal_split_bits = 3 2023-10-19T12:43:53.261 DEBUG:teuthology.orchestra.run.smithi057:> sudo ceph --cluster ceph config set mds mds_bal_split_bits 3 2023-10-19T12:43:53.304 INFO:tasks.ceph:setting config [mds] mds_bal_split_size = 100 2023-10-19T12:43:53.304 DEBUG:teuthology.orchestra.run.smithi057:> sudo ceph --cluster ceph config set mds mds_bal_split_size 100 2023-10-19T12:43:53.348 INFO:tasks.ceph:setting config [mds] mds_debug_frag = True 2023-10-19T12:43:53.348 DEBUG:teuthology.orchestra.run.smithi057:> sudo ceph --cluster ceph config set mds mds_debug_frag True 2023-10-19T12:43:53.393 INFO:tasks.ceph:setting config [mds] mds_debug_scatterstat = True 2023-10-19T12:43:53.393 DEBUG:teuthology.orchestra.run.smithi057:> sudo ceph --cluster ceph config set mds mds_debug_scatterstat True 2023-10-19T12:43:53.413 INFO:tasks.ceph:setting config [mds] mds_op_complaint_time = 180 2023-10-19T12:43:53.414 DEBUG:teuthology.orchestra.run.smithi057:> sudo ceph --cluster ceph config set mds mds_op_complaint_time 180 2023-10-19T12:43:53.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.415+0000 7f609f890640 1 -- 172.21.15.57:0/3741609729 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f60980b3cd0 msgr2=0x7f60980c4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:53.419 INFO:tasks.ceph:setting config [mds] mds_verify_scatter = True 2023-10-19T12:43:53.419 DEBUG:teuthology.orchestra.run.smithi057:> sudo ceph --cluster ceph config set mds mds_verify_scatter True 2023-10-19T12:43:53.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.418+0000 7f609f890640 1 --2- 172.21.15.57:0/3741609729 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f60980b3cd0 0x7f60980c4f50 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f6088006ef0 tx=0x7f60880017e0 comp rx=0 tx=0).stop 2023-10-19T12:43:53.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.418+0000 7f609f890640 1 -- 172.21.15.57:0/3741609729 shutdown_connections 2023-10-19T12:43:53.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.418+0000 7fea0b7fe640 1 -- 172.21.15.57:0/1960099229 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fea1801c760 con 0x7fea2406c510 2023-10-19T12:43:53.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.418+0000 7fea0b7fe640 1 -- 172.21.15.57:0/1960099229 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fea180118c0 con 0x7fea2406c510 2023-10-19T12:43:53.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.419+0000 7f49325a5640 1 -- 172.21.15.57:0/2530393221 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f492c167bc0 msgr2=0x7f492c0b1560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:53.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.419+0000 7f609f890640 1 --2- 172.21.15.57:0/3741609729 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6098152f90 0x7f60981f1f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:53.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.419+0000 7f609f890640 1 --2- 172.21.15.57:0/3741609729 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f60980b3cd0 0x7f60980c4f50 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:53.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.420+0000 7f609f890640 1 --2- 172.21.15.57:0/3741609729 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6084004d50 0x7f60980c25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:53.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.420+0000 7f609f890640 1 -- 172.21.15.57:0/3741609729 >> 172.21.15.57:0/3741609729 conn(0x7f60980b0670 msgr2=0x7f60980ccc30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:43:53.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.420+0000 7f609f890640 1 -- 172.21.15.57:0/3741609729 shutdown_connections 2023-10-19T12:43:53.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.420+0000 7f609f890640 1 -- 172.21.15.57:0/3741609729 wait complete. 2023-10-19T12:43:53.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.419+0000 7f49325a5640 1 --2- 172.21.15.57:0/2530393221 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f492c167bc0 0x7f492c0b1560 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f4928009830 tx=0x7f492802bcd0 comp rx=0 tx=0).stop 2023-10-19T12:43:53.425 INFO:tasks.ceph:setting config [mds] osd_op_complaint_time = 180 2023-10-19T12:43:53.425 DEBUG:teuthology.orchestra.run.smithi057:> sudo ceph --cluster ceph config set mds osd_op_complaint_time 180 2023-10-19T12:43:53.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.421+0000 7f49325a5640 1 -- 172.21.15.57:0/2530393221 shutdown_connections 2023-10-19T12:43:53.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.421+0000 7f49325a5640 1 --2- 172.21.15.57:0/2530393221 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f492c167bc0 0x7f492c0b1560 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:53.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.421+0000 7f49325a5640 1 -- 172.21.15.57:0/2530393221 >> 172.21.15.57:0/2530393221 conn(0x7f492c069720 msgr2=0x7f492c069b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:43:53.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.435+0000 7fea29644640 1 -- 172.21.15.57:0/1960099229 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea04005040 con 0x7fea2406c510 2023-10-19T12:43:53.438 INFO:tasks.ceph:setting config [mds] rados_mon_op_timeout = 900 2023-10-19T12:43:53.438 DEBUG:teuthology.orchestra.run.smithi057:> sudo ceph --cluster ceph config set mds rados_mon_op_timeout 900 2023-10-19T12:43:53.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.440+0000 7fa309db0640 1 -- 172.21.15.57:0/3662337753 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_command([{prefix=config set, name=rados_mon_op_timeout}] v 0) v1 -- 0x7fa2dc0050c0 con 0x7fa30415a460 2023-10-19T12:43:53.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.421+0000 7f49325a5640 1 -- 172.21.15.57:0/2530393221 shutdown_connections 2023-10-19T12:43:53.445 INFO:tasks.ceph:setting config [mds] rados_osd_op_timeout = 900 2023-10-19T12:43:53.445 DEBUG:teuthology.orchestra.run.smithi057:> sudo ceph --cluster ceph config set mds rados_osd_op_timeout 900 2023-10-19T12:43:53.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.441+0000 7fea0b7fe640 1 -- 172.21.15.57:0/1960099229 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7fea18011460 con 0x7fea2406c510 2023-10-19T12:43:53.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.421+0000 7f49325a5640 1 -- 172.21.15.57:0/2530393221 wait complete. 2023-10-19T12:43:53.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.433+0000 7f49325a5640 1 Processor -- start 2023-10-19T12:43:53.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.434+0000 7f49325a5640 1 -- start start 2023-10-19T12:43:53.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.435+0000 7f49325a5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f492c0709b0 0x7f492c1570a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:43:53.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.435+0000 7f49325a5640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f492c1686b0 0x7f492c0aa670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:43:53.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.435+0000 7f49325a5640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f492c0aabb0 0x7f492c0aed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:43:53.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.435+0000 7f49325a5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f492c0ab4a0 con 0x7f492c0709b0 2023-10-19T12:43:53.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.435+0000 7f49325a5640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f492c0a6c80 con 0x7f492c1686b0 2023-10-19T12:43:53.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.435+0000 7f49325a5640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f492c157c10 con 0x7f492c0aabb0 2023-10-19T12:43:53.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.439+0000 7f4930da2640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f492c1686b0 0x7f492c0aa670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:43:53.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.439+0000 7f4930da2640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f492c1686b0 0x7f492c0aa670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.57:33446/0 (socket says 172.21.15.57:33446) 2023-10-19T12:43:53.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.439+0000 7f4930da2640 1 -- 172.21.15.57:0/3698341320 learned_addr learned my addr 172.21.15.57:0/3698341320 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:43:53.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.439+0000 7f49315a3640 1 --2- 172.21.15.57:0/3698341320 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f492c0709b0 0x7f492c1570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:43:53.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.440+0000 7f49315a3640 1 -- 172.21.15.57:0/3698341320 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f492c0aabb0 msgr2=0x7f492c0aed70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T12:43:53.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.440+0000 7f49315a3640 1 --2- 172.21.15.57:0/3698341320 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f492c0aabb0 0x7f492c0aed70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:53.452 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.440+0000 7f49315a3640 1 -- 172.21.15.57:0/3698341320 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f492c1686b0 msgr2=0x7f492c0aa670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:53.452 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.440+0000 7f49315a3640 1 --2- 172.21.15.57:0/3698341320 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f492c1686b0 0x7f492c0aa670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:53.452 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.440+0000 7f49315a3640 1 -- 172.21.15.57:0/3698341320 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4928009490 con 0x7f492c0709b0 2023-10-19T12:43:53.452 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.440+0000 7f49315a3640 1 --2- 172.21.15.57:0/3698341320 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f492c0709b0 0x7f492c1570a0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f4928033000 tx=0x7f4928034d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:43:53.452 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.446+0000 7f4921ffb640 1 -- 172.21.15.57:0/3698341320 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4928004050 con 0x7f492c0709b0 2023-10-19T12:43:53.453 INFO:tasks.ceph:setting config [mon] mon_op_complaint_time = 120 2023-10-19T12:43:53.455 DEBUG:teuthology.orchestra.run.smithi057:> sudo ceph --cluster ceph config set mon mon_op_complaint_time 120 2023-10-19T12:43:53.456 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.452+0000 7fea0b7fe640 1 -- 172.21.15.57:0/1960099229 <== mon.0 v2:172.21.15.57:3300/0 7 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fea18011bc0 con 0x7fea2406c510 2023-10-19T12:43:53.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.446+0000 7f49325a5640 1 -- 172.21.15.57:0/3698341320 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f492c0ab1c0 con 0x7f492c0709b0 2023-10-19T12:43:53.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.446+0000 7f49325a5640 1 -- 172.21.15.57:0/3698341320 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f492c0a6910 con 0x7f492c0709b0 2023-10-19T12:43:53.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.453+0000 7f4921ffb640 1 -- 172.21.15.57:0/3698341320 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(4 keys) v1 ==== 110+0+0 (secure 0 0 0) 0x7f4928004280 con 0x7f492c0709b0 2023-10-19T12:43:53.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.453+0000 7f4921ffb640 1 -- 172.21.15.57:0/3698341320 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4928005020 con 0x7f492c0709b0 2023-10-19T12:43:53.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.454+0000 7f4921ffb640 1 -- 172.21.15.57:0/3698341320 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f4928005540 con 0x7f492c0709b0 2023-10-19T12:43:53.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.454+0000 7f4921ffb640 1 -- 172.21.15.57:0/3698341320 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f49280035f0 con 0x7f492c0709b0 2023-10-19T12:43:53.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.454+0000 7f5d9a7fc640 1 -- 172.21.15.57:0/3800803281 <== mon.1 v2:172.21.15.179:3300/0 11 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f5d9c052430 con 0x7f5da4158be0 2023-10-19T12:43:53.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.454+0000 7f4921ffb640 1 -- 172.21.15.57:0/3698341320 <== mon.0 v2:172.21.15.57:3300/0 6 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f4928004eb0 con 0x7f492c0709b0 2023-10-19T12:43:53.461 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.458+0000 7fcb1e7fc640 1 -- 172.21.15.57:0/85951075 <== mon.2 v2:172.21.15.179:3301/0 11 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fcb180356e0 con 0x7fcb2806e0b0 2023-10-19T12:43:53.461 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.457+0000 7f49325a5640 1 -- 172.21.15.57:0/3698341320 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4904005250 con 0x7f492c0709b0 2023-10-19T12:43:53.463 INFO:tasks.ceph:setting config [osd] osd_op_complaint_time = 180 2023-10-19T12:43:53.463 DEBUG:teuthology.orchestra.run.smithi057:> sudo ceph --cluster ceph config set osd osd_op_complaint_time 180 2023-10-19T12:43:53.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.457+0000 7fa2ebfff640 1 -- 172.21.15.57:0/3662337753 <== mon.2 v2:172.21.15.179:3301/0 11 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fa2fc011650 con 0x7fa30415a460 2023-10-19T12:43:53.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.457+0000 7fa2ebfff640 1 -- 172.21.15.57:0/3662337753 <== mon.2 v2:172.21.15.179:3301/0 12 ==== mon_command_ack([{prefix=config set, name=rados_mon_op_timeout}]=0 v6)=0 v6) v1 ==== 123+0+0 (secure 0 0 0) 0x7fa2fc0119a0 con 0x7fa30415a460 2023-10-19T12:43:53.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.461+0000 7fa2e9ffb640 1 -- 172.21.15.57:0/3662337753 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa30415a460 msgr2=0x7fa304138030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:53.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.461+0000 7fa2e9ffb640 1 --2- 172.21.15.57:0/3662337753 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa30415a460 0x7fa304138030 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa3041539d0 tx=0x7fa2fc009a80 comp rx=0 tx=0).stop 2023-10-19T12:43:53.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.463+0000 7fa2e9ffb640 1 -- 172.21.15.57:0/3662337753 shutdown_connections 2023-10-19T12:43:53.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.463+0000 7fa2e9ffb640 1 --2- 172.21.15.57:0/3662337753 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa30415a460 0x7fa304138030 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:53.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.463+0000 7fa2e9ffb640 1 --2- 172.21.15.57:0/3662337753 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa3041ed100 0x7fa30413da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:53.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.463+0000 7fa2e9ffb640 1 --2- 172.21.15.57:0/3662337753 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa2ec0039a0 0x7fa304150460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:53.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.463+0000 7fa2e9ffb640 1 -- 172.21.15.57:0/3662337753 >> 172.21.15.57:0/3662337753 conn(0x7fa3040af730 msgr2=0x7fa3040afb10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:43:53.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.463+0000 7fa2e9ffb640 1 -- 172.21.15.57:0/3662337753 shutdown_connections 2023-10-19T12:43:53.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.463+0000 7fa2e9ffb640 1 -- 172.21.15.57:0/3662337753 wait complete. 2023-10-19T12:43:53.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.462+0000 7f4921ffb640 1 -- 172.21.15.57:0/3698341320 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f4928042030 con 0x7f492c0709b0 2023-10-19T12:43:53.470 DEBUG:tasks.ceph:set 21 configs 2023-10-19T12:43:53.470 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'client', 'client_mount_timeout', '600'], check_status=True, hostname='smithi057') 2023-10-19T12:43:53.479 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'client', 'debug_client', '20'], check_status=True, hostname='smithi057') 2023-10-19T12:43:53.480 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'client', 'debug_ms', '1'], check_status=True, hostname='smithi057') 2023-10-19T12:43:53.480 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'client', 'rados_mon_op_timeout', '900'], check_status=True, hostname='smithi057') 2023-10-19T12:43:53.509 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'client', 'rados_osd_op_timeout', '900'], check_status=True, hostname='smithi057') 2023-10-19T12:43:53.509 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'debug_mds', '20'], check_status=True, hostname='smithi057') 2023-10-19T12:43:53.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.532+0000 7f5dac14c640 1 -- 172.21.15.57:0/3800803281 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command([{prefix=config set, name=debug_mds}] v 0) v1 -- 0x7f5d800052b0 con 0x7f5da4158be0 2023-10-19T12:43:53.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.549+0000 7f5d9a7fc640 1 -- 172.21.15.57:0/3800803281 <== mon.1 v2:172.21.15.179:3300/0 12 ==== mon_command_ack([{prefix=config set, name=debug_mds}]=0 v7)=0 v7) v1 ==== 108+0+0 (secure 0 0 0) 0x7f5d9c0570b0 con 0x7f5da4158be0 2023-10-19T12:43:53.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.551+0000 7f5d87fff640 1 -- 172.21.15.57:0/3800803281 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5da4158be0 msgr2=0x7f5da4070830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:53.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.551+0000 7f5d87fff640 1 --2- 172.21.15.57:0/3800803281 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5da4158be0 0x7f5da4070830 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f5d9c009000 tx=0x7f5d9c004a40 comp rx=0 tx=0).stop 2023-10-19T12:43:53.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.551+0000 7f5d87fff640 1 -- 172.21.15.57:0/3800803281 shutdown_connections 2023-10-19T12:43:53.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.551+0000 7f5d87fff640 1 --2- 172.21.15.57:0/3800803281 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5da4158be0 0x7f5da4070830 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:53.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.551+0000 7f5d87fff640 1 --2- 172.21.15.57:0/3800803281 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5da40b18f0 0x7f5da40a7290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:53.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.551+0000 7f5d87fff640 1 --2- 172.21.15.57:0/3800803281 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5da406ca10 0x7f5da4156820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:53.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.551+0000 7f5d87fff640 1 -- 172.21.15.57:0/3800803281 >> 172.21.15.57:0/3800803281 conn(0x7f5da40aec40 msgr2=0x7f5da406b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:43:53.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.552+0000 7f5d87fff640 1 -- 172.21.15.57:0/3800803281 shutdown_connections 2023-10-19T12:43:53.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.552+0000 7f5d87fff640 1 -- 172.21.15.57:0/3800803281 wait complete. 2023-10-19T12:43:53.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.564+0000 7fc1816a9640 1 -- 172.21.15.57:0/1005844253 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc17c069720 msgr2=0x7fc17c0cd580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:53.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.564+0000 7fc1816a9640 1 --2- 172.21.15.57:0/1005844253 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc17c069720 0x7fc17c0cd580 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fc170009860 tx=0x7fc17002be60 comp rx=0 tx=0).stop 2023-10-19T12:43:53.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.569+0000 7fc1816a9640 1 -- 172.21.15.57:0/1005844253 shutdown_connections 2023-10-19T12:43:53.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.569+0000 7fc1816a9640 1 --2- 172.21.15.57:0/1005844253 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc17c069720 0x7fc17c0cd580 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:53.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.569+0000 7fc1816a9640 1 -- 172.21.15.57:0/1005844253 >> 172.21.15.57:0/1005844253 conn(0x7fc17c0b0cd0 msgr2=0x7fc17c0b1f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:43:53.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.569+0000 7fc1816a9640 1 -- 172.21.15.57:0/1005844253 shutdown_connections 2023-10-19T12:43:53.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.569+0000 7fc1816a9640 1 -- 172.21.15.57:0/1005844253 wait complete. 2023-10-19T12:43:53.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.571+0000 7fc1816a9640 1 Processor -- start 2023-10-19T12:43:53.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.571+0000 7fc1816a9640 1 -- start start 2023-10-19T12:43:53.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.572+0000 7fc1816a9640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc17c069720 0x7fc17c1e3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:43:53.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.572+0000 7fc1816a9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc17c0ac7a0 0x7fc17c20d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:43:53.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.572+0000 7fc1816a9640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc17c15afe0 0x7fc17c15d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:43:53.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.572+0000 7fc1816a9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc17c20df20 con 0x7fc17c0ac7a0 2023-10-19T12:43:53.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.572+0000 7fc1816a9640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc17c20e200 con 0x7fc17c15afe0 2023-10-19T12:43:53.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.572+0000 7fc1816a9640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc17c20e4e0 con 0x7fc17c069720 2023-10-19T12:43:53.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.573+0000 7fc17a7fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc17c0ac7a0 0x7fc17c20d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:43:53.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.573+0000 7fc17a7fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc17c0ac7a0 0x7fc17c20d610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50904/0 (socket says 172.21.15.57:50904) 2023-10-19T12:43:53.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.573+0000 7fc17a7fc640 1 -- 172.21.15.57:0/2765127877 learned_addr learned my addr 172.21.15.57:0/2765127877 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:43:53.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.573+0000 7fc17a7fc640 1 -- 172.21.15.57:0/2765127877 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc17c069720 msgr2=0x7fc17c1e3cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-19T12:43:53.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.573+0000 7fc17a7fc640 1 --2- 172.21.15.57:0/2765127877 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc17c069720 0x7fc17c1e3cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:53.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.573+0000 7fc17a7fc640 1 -- 172.21.15.57:0/2765127877 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc17c15afe0 msgr2=0x7fc17c15d4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-19T12:43:53.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.573+0000 7fc17a7fc640 1 --2- 172.21.15.57:0/2765127877 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc17c15afe0 0x7fc17c15d4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:53.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.573+0000 7fc17a7fc640 1 -- 172.21.15.57:0/2765127877 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc170009490 con 0x7fc17c0ac7a0 2023-10-19T12:43:53.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.574+0000 7fc17a7fc640 1 --2- 172.21.15.57:0/2765127877 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc17c0ac7a0 0x7fc17c20d610 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fc16c00c780 tx=0x7fc16c00cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:43:53.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.574+0000 7fc1637fe640 1 -- 172.21.15.57:0/2765127877 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc16c00a640 con 0x7fc17c0ac7a0 2023-10-19T12:43:53.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.574+0000 7fc1637fe640 1 -- 172.21.15.57:0/2765127877 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fc16c00a890 con 0x7fc17c0ac7a0 2023-10-19T12:43:53.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.574+0000 7fc1637fe640 1 -- 172.21.15.57:0/2765127877 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc16c00ac60 con 0x7fc17c0ac7a0 2023-10-19T12:43:53.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.575+0000 7fc1816a9640 1 -- 172.21.15.57:0/2765127877 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc17c20e7c0 con 0x7fc17c0ac7a0 2023-10-19T12:43:53.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.575+0000 7fc1816a9640 1 -- 172.21.15.57:0/2765127877 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc17c15da00 con 0x7fc17c0ac7a0 2023-10-19T12:43:53.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.576+0000 7fc1637fe640 1 -- 172.21.15.57:0/2765127877 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fc16c012720 con 0x7fc17c0ac7a0 2023-10-19T12:43:53.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.576+0000 7fc1637fe640 1 -- 172.21.15.57:0/2765127877 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fc16c01b670 con 0x7fc17c0ac7a0 2023-10-19T12:43:53.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.581+0000 7fc1816a9640 1 -- 172.21.15.57:0/2765127877 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc15c005250 con 0x7fc17c0ac7a0 2023-10-19T12:43:53.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.585+0000 7fc1637fe640 1 -- 172.21.15.57:0/2765127877 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7fc16c016020 con 0x7fc17c0ac7a0 2023-10-19T12:43:53.597 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'debug_mds_balancer', '20'], check_status=True, hostname='smithi057') 2023-10-19T12:43:53.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.624+0000 7fabf31e2640 1 -- 172.21.15.57:0/4008376285 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fabd8004120 msgr2=0x7fabd800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:53.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.624+0000 7fabf31e2640 1 --2- 172.21.15.57:0/4008376285 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fabd8004120 0x7fabd800b980 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fabdc00c290 tx=0x7fabd80057f0 comp rx=0 tx=0).stop 2023-10-19T12:43:53.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.624+0000 7fabf29e1640 1 -- 172.21.15.57:0/4008376285 reap_dead start 2023-10-19T12:43:53.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.624+0000 7fabf31e2640 1 -- 172.21.15.57:0/4008376285 shutdown_connections 2023-10-19T12:43:53.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.624+0000 7fabf31e2640 1 -- 172.21.15.57:0/4008376285 >> 172.21.15.57:0/4008376285 conn(0x7fabe4009f80 msgr2=0x7fabe400a360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:43:53.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.636+0000 7fabf31e2640 1 -- 172.21.15.57:0/4008376285 shutdown_connections 2023-10-19T12:43:53.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.636+0000 7fabf31e2640 1 -- 172.21.15.57:0/4008376285 wait complete. 2023-10-19T12:43:53.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.638+0000 7fabf31e2640 1 Processor -- start 2023-10-19T12:43:53.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.638+0000 7fabf31e2640 1 -- start start 2023-10-19T12:43:53.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.639+0000 7fabf31e2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fabe4035c10 0x7fabe40db130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:43:53.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.639+0000 7fabf31e2640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fabe4035ff0 0x7fabe40db670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:43:53.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.639+0000 7fabf31e2640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fabe404c9c0 0x7fabe40d7cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:43:53.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.639+0000 7fabf31e2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabe404bc00 con 0x7fabe4035c10 2023-10-19T12:43:53.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.639+0000 7fabf31e2640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabe404bee0 con 0x7fabe404c9c0 2023-10-19T12:43:53.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.639+0000 7fabf31e2640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabe404c1c0 con 0x7fabe4035ff0 2023-10-19T12:43:53.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.639+0000 7fabf29e1640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fabe404c9c0 0x7fabe40d7cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:43:53.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.639+0000 7fabf29e1640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fabe404c9c0 0x7fabe40d7cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.57:33460/0 (socket says 172.21.15.57:33460) 2023-10-19T12:43:53.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.639+0000 7fabf29e1640 1 -- 172.21.15.57:0/2612779313 learned_addr learned my addr 172.21.15.57:0/2612779313 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:43:53.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.640+0000 7fabf19df640 1 --2- 172.21.15.57:0/2612779313 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fabe4035ff0 0x7fabe40db670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:43:53.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.640+0000 7fabf29e1640 1 -- 172.21.15.57:0/2612779313 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fabe4035ff0 msgr2=0x7fabe40db670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:53.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.640+0000 7fabf29e1640 1 --2- 172.21.15.57:0/2612779313 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fabe4035ff0 0x7fabe40db670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:53.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.640+0000 7fabf29e1640 1 -- 172.21.15.57:0/2612779313 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fabe4035c10 msgr2=0x7fabe40db130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T12:43:53.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.640+0000 7fabf29e1640 1 --2- 172.21.15.57:0/2612779313 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fabe4035c10 0x7fabe40db130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:53.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.640+0000 7fabf29e1640 1 -- 172.21.15.57:0/2612779313 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabe404c4a0 con 0x7fabe404c9c0 2023-10-19T12:43:53.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.641+0000 7fabf29e1640 1 --2- 172.21.15.57:0/2612779313 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fabe404c9c0 0x7fabe40d7cd0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fabdc00c3b0 tx=0x7fabdc001d20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:43:53.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.644+0000 7fabe2ffd640 1 -- 172.21.15.57:0/2612779313 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabdc03b070 con 0x7fabe404c9c0 2023-10-19T12:43:53.648 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:43:53.645+0000 7fdc67c3a040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-19T12:43:53.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.644+0000 7fabf31e2640 1 -- 172.21.15.57:0/2612779313 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabe40d8cd0 con 0x7fabe404c9c0 2023-10-19T12:43:53.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.644+0000 7fabf31e2640 1 -- 172.21.15.57:0/2612779313 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabe40d8210 con 0x7fabe404c9c0 2023-10-19T12:43:53.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.645+0000 7fabe2ffd640 1 -- 172.21.15.57:0/2612779313 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fabdc02f040 con 0x7fabe404c9c0 2023-10-19T12:43:53.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.646+0000 7fabe2ffd640 1 -- 172.21.15.57:0/2612779313 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabdc006280 con 0x7fabe404c9c0 2023-10-19T12:43:53.650 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.647+0000 7fabe2ffd640 1 -- 172.21.15.57:0/2612779313 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fabdc03e070 con 0x7fabe404c9c0 2023-10-19T12:43:53.650 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.647+0000 7fabe2ffd640 1 -- 172.21.15.57:0/2612779313 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fabdc034770 con 0x7fabe404c9c0 2023-10-19T12:43:53.650 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.647+0000 7fabf31e2640 1 -- 172.21.15.57:0/2612779313 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabd4005250 con 0x7fabe404c9c0 2023-10-19T12:43:53.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.649+0000 7fabe2ffd640 1 -- 172.21.15.57:0/2612779313 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7fabdc045060 con 0x7fabe404c9c0 2023-10-19T12:43:53.700 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:43:53.697+0000 7fdc67c3a040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-19T12:43:53.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.704+0000 7f252d743640 1 -- 172.21.15.57:0/1455267542 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f252815a7f0 msgr2=0x7f252806add0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:53.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.704+0000 7f252d743640 1 --2- 172.21.15.57:0/1455267542 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f252815a7f0 0x7f252806add0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f251c0074e0 tx=0x7f251c02cbb0 comp rx=0 tx=0).stop 2023-10-19T12:43:53.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.708+0000 7f252d743640 1 -- 172.21.15.57:0/1455267542 shutdown_connections 2023-10-19T12:43:53.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.708+0000 7f252d743640 1 --2- 172.21.15.57:0/1455267542 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f252815a7f0 0x7f252806add0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:53.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.708+0000 7f252d743640 1 -- 172.21.15.57:0/1455267542 >> 172.21.15.57:0/1455267542 conn(0x7f25280ae460 msgr2=0x7f25280ae840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:43:53.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.712+0000 7f252d743640 1 -- 172.21.15.57:0/1455267542 shutdown_connections 2023-10-19T12:43:53.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.712+0000 7f252d743640 1 -- 172.21.15.57:0/1455267542 wait complete. 2023-10-19T12:43:53.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.714+0000 7f252d743640 1 Processor -- start 2023-10-19T12:43:53.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.714+0000 7f252d743640 1 -- start start 2023-10-19T12:43:53.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.715+0000 7f252d743640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25280af760 0x7f252813fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:43:53.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.716+0000 7f252d743640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f25280b0f90 0x7f252812a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:43:53.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.716+0000 7f252d743640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f252815a7f0 0x7f25280cabd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:43:53.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.716+0000 7f252d743640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f252812aee0 con 0x7f25280af760 2023-10-19T12:43:53.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.716+0000 7f252d743640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25280cb110 con 0x7f252815a7f0 2023-10-19T12:43:53.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.716+0000 7f252d743640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f252812b140 con 0x7f25280b0f90 2023-10-19T12:43:53.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.716+0000 7f2527fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25280af760 0x7f252813fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:43:53.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.716+0000 7f2527fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25280af760 0x7f252813fa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50926/0 (socket says 172.21.15.57:50926) 2023-10-19T12:43:53.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.716+0000 7f2527fff640 1 -- 172.21.15.57:0/900337064 learned_addr learned my addr 172.21.15.57:0/900337064 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:43:53.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.717+0000 7f2527fff640 1 -- 172.21.15.57:0/900337064 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f25280b0f90 msgr2=0x7f252812a9a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T12:43:53.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.717+0000 7f2527fff640 1 --2- 172.21.15.57:0/900337064 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f25280b0f90 0x7f252812a9a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:53.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.717+0000 7f2527fff640 1 -- 172.21.15.57:0/900337064 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f252815a7f0 msgr2=0x7f25280cabd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T12:43:53.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.717+0000 7f2527fff640 1 --2- 172.21.15.57:0/900337064 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f252815a7f0 0x7f25280cabd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:53.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.717+0000 7f2527fff640 1 -- 172.21.15.57:0/900337064 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f251c0071c0 con 0x7f25280af760 2023-10-19T12:43:53.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.718+0000 7f2527fff640 1 --2- 172.21.15.57:0/900337064 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25280af760 0x7f252813fa70 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f252000caf0 tx=0x7f252000cef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:43:53.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.718+0000 7f2524ff9640 1 -- 172.21.15.57:0/900337064 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2520003b70 con 0x7f25280af760 2023-10-19T12:43:53.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.718+0000 7f252d743640 1 -- 172.21.15.57:0/900337064 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f252812b420 con 0x7f25280af760 2023-10-19T12:43:53.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.719+0000 7f252d743640 1 -- 172.21.15.57:0/900337064 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f252812b9e0 con 0x7f25280af760 2023-10-19T12:43:53.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.721+0000 7f2524ff9640 1 -- 172.21.15.57:0/900337064 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f2520003cf0 con 0x7f25280af760 2023-10-19T12:43:53.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.721+0000 7f2524ff9640 1 -- 172.21.15.57:0/900337064 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2520003ea0 con 0x7f25280af760 2023-10-19T12:43:53.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.722+0000 7f2524ff9640 1 -- 172.21.15.57:0/900337064 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f25200050d0 con 0x7f25280af760 2023-10-19T12:43:53.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.723+0000 7f250e7fc640 1 -- 172.21.15.57:0/900337064 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2514005250 con 0x7f25280af760 2023-10-19T12:43:53.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.723+0000 7f2524ff9640 1 -- 172.21.15.57:0/900337064 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f2520012a70 con 0x7f25280af760 2023-10-19T12:43:53.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.727+0000 7f2524ff9640 1 -- 172.21.15.57:0/900337064 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f2520005380 con 0x7f25280af760 2023-10-19T12:43:53.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.739+0000 7fcb0bfff640 1 -- 172.21.15.57:0/85951075 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_command([{prefix=config set, name=debug_mds_balancer}] v 0) v1 -- 0x7fcb040052b0 con 0x7fcb2806e0b0 2023-10-19T12:43:53.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.747+0000 7fcb1e7fc640 1 -- 172.21.15.57:0/85951075 <== mon.2 v2:172.21.15.179:3301/0 12 ==== mon_command_ack([{prefix=config set, name=debug_mds_balancer}]=0 v8)=0 v8) v1 ==== 117+0+0 (secure 0 0 0) 0x7fcb18035850 con 0x7fcb2806e0b0 2023-10-19T12:43:53.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.750+0000 7fcb0bfff640 1 -- 172.21.15.57:0/85951075 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcb2806e0b0 msgr2=0x7fcb2814d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:53.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.750+0000 7fcb0bfff640 1 --2- 172.21.15.57:0/85951075 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcb2806e0b0 0x7fcb2814d110 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fcb180209e0 tx=0x7fcb18020ea0 comp rx=0 tx=0).stop 2023-10-19T12:43:53.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.750+0000 7fcb0bfff640 1 -- 172.21.15.57:0/85951075 shutdown_connections 2023-10-19T12:43:53.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.750+0000 7fcb0bfff640 1 --2- 172.21.15.57:0/85951075 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcb28152120 0x7fcb280ca5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:53.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.750+0000 7fcb0bfff640 1 --2- 172.21.15.57:0/85951075 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcb2806e0b0 0x7fcb2814d110 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:53.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.750+0000 7fcb0bfff640 1 --2- 172.21.15.57:0/85951075 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcb28069950 0x7fcb280b23a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:53.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.751+0000 7fcb0bfff640 1 -- 172.21.15.57:0/85951075 >> 172.21.15.57:0/85951075 conn(0x7fcb280af730 msgr2=0x7fcb281516a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:43:53.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.751+0000 7f4116a13640 1 -- 172.21.15.57:0/3647911798 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f41080338f0 msgr2=0x7f4108053cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:53.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.751+0000 7f4116a13640 1 --2- 172.21.15.57:0/3647911798 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f41080338f0 0x7f4108053cf0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f4110068940 tx=0x7f4110093e20 comp rx=0 tx=0).stop 2023-10-19T12:43:53.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.751+0000 7f4116a13640 1 -- 172.21.15.57:0/3647911798 shutdown_connections 2023-10-19T12:43:53.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.751+0000 7f4116a13640 1 --2- 172.21.15.57:0/3647911798 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f41080338f0 0x7f4108053cf0 secure :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f4110068940 tx=0x7f4110093e20 comp rx=0 tx=0).stop 2023-10-19T12:43:53.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.751+0000 7f4116a13640 1 --2- 172.21.15.57:0/3647911798 >> v2:172.21.15.179:3300/0 conn(0x7f4108100fe0 0x7f4108037c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:53.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.751+0000 7f4116a13640 1 -- 172.21.15.57:0/3647911798 >> 172.21.15.57:0/3647911798 conn(0x7f4108009f80 msgr2=0x7f410800a360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:43:53.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.754+0000 7fea29644640 1 -- 172.21.15.57:0/1960099229 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command([{prefix=config set, name=debug_ms}] v 0) v1 -- 0x7fea040052f0 con 0x7fea2406c510 2023-10-19T12:43:53.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.755+0000 7f4116a13640 1 -- 172.21.15.57:0/3647911798 shutdown_connections 2023-10-19T12:43:53.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.755+0000 7f4116a13640 1 -- 172.21.15.57:0/3647911798 wait complete. 2023-10-19T12:43:53.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.762+0000 7fd8bb59e640 1 -- 172.21.15.57:0/1072407661 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd8b40df1e0 msgr2=0x7fd8b4038040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:53.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.762+0000 7fd8b959a640 1 -- 172.21.15.57:0/1072407661 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8bc0a48a0 con 0x7fd8b40df1e0 2023-10-19T12:43:53.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.762+0000 7fd8bb59e640 1 --2- 172.21.15.57:0/1072407661 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd8b40df1e0 0x7fd8b4038040 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd8bc068a70 tx=0x7fd8bc0a3c40 comp rx=0 tx=0).stop 2023-10-19T12:43:53.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.764+0000 7fd8bb59e640 1 -- 172.21.15.57:0/1072407661 shutdown_connections 2023-10-19T12:43:53.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.764+0000 7fd8bb59e640 1 --2- 172.21.15.57:0/1072407661 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd8b40df1e0 0x7fd8b4038040 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:53.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.764+0000 7fd8bb59e640 1 -- 172.21.15.57:0/1072407661 >> 172.21.15.57:0/1072407661 conn(0x7fd8b4009f70 msgr2=0x7fd8b400a350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:43:53.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.767+0000 7fd8bb59e640 1 -- 172.21.15.57:0/1072407661 shutdown_connections 2023-10-19T12:43:53.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.768+0000 7fea0b7fe640 1 -- 172.21.15.57:0/1960099229 <== mon.0 v2:172.21.15.57:3300/0 8 ==== mon_command_ack([{prefix=config set, name=debug_ms}]=0 v9)=0 v9) v1 ==== 106+0+0 (secure 0 0 0) 0x7fea18011e00 con 0x7fea2406c510 2023-10-19T12:43:53.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.771+0000 7fcb0bfff640 1 -- 172.21.15.57:0/85951075 shutdown_connections 2023-10-19T12:43:53.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.775+0000 7fcb0bfff640 1 -- 172.21.15.57:0/85951075 wait complete. 2023-10-19T12:43:53.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.773+0000 7fd8bb59e640 1 -- 172.21.15.57:0/1072407661 wait complete. 2023-10-19T12:43:53.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.775+0000 7fd8bb59e640 1 Processor -- start 2023-10-19T12:43:53.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.773+0000 7f850972c640 1 -- 172.21.15.57:0/2311821668 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f84ec005180 msgr2=0x7f84ec00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:53.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.773+0000 7f850972c640 1 --2- 172.21.15.57:0/2311821668 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f84ec005180 0x7f84ec00b980 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f84fc00c4c0 tx=0x7f84ec0057f0 comp rx=0 tx=0).stop 2023-10-19T12:43:53.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.773+0000 7f8508f2b640 1 -- 172.21.15.57:0/2311821668 reap_dead start 2023-10-19T12:43:53.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.773+0000 7f850972c640 1 -- 172.21.15.57:0/2311821668 shutdown_connections 2023-10-19T12:43:53.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.773+0000 7f850972c640 1 -- 172.21.15.57:0/2311821668 >> 172.21.15.57:0/2311821668 conn(0x7f8504069720 msgr2=0x7f8504069b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:43:53.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.757+0000 7f4116a13640 1 Processor -- start 2023-10-19T12:43:53.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.757+0000 7f4116a13640 1 -- start start 2023-10-19T12:43:53.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.758+0000 7f4116a13640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4108054230 0x7f410804b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:43:53.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.758+0000 7f4116a13640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4108100fe0 0x7f41080ee040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:43:53.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.758+0000 7f4116a13640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4108035bf0 0x7f41080f9730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:43:53.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.758+0000 7f4116a13640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4108036ab0 con 0x7f4108100fe0 2023-10-19T12:43:53.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.758+0000 7f4116a13640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4108035760 con 0x7f4108054230 2023-10-19T12:43:53.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.758+0000 7f4116a13640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4108034b50 con 0x7f4108035bf0 2023-10-19T12:43:53.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.759+0000 7f4115a11640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4108054230 0x7f410804b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:43:53.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.759+0000 7f4116212640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4108035bf0 0x7f41080f9730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:43:53.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.759+0000 7f4115a11640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4108054230 0x7f410804b4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.57:33470/0 (socket says 172.21.15.57:33470) 2023-10-19T12:43:53.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.759+0000 7f4115a11640 1 -- 172.21.15.57:0/858111068 learned_addr learned my addr 172.21.15.57:0/858111068 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:43:53.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.759+0000 7f4115a11640 1 -- 172.21.15.57:0/858111068 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4108035bf0 msgr2=0x7f41080f9730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:53.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.759+0000 7f4115a11640 1 --2- 172.21.15.57:0/858111068 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4108035bf0 0x7f41080f9730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:53.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.759+0000 7f4115a11640 1 -- 172.21.15.57:0/858111068 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4108100fe0 msgr2=0x7f41080ee040 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T12:43:53.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.759+0000 7f4115a11640 1 --2- 172.21.15.57:0/858111068 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4108100fe0 0x7f41080ee040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:53.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.759+0000 7f4115a11640 1 -- 172.21.15.57:0/858111068 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f411004e960 con 0x7f4108054230 2023-10-19T12:43:53.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.760+0000 7f4115a11640 1 --2- 172.21.15.57:0/858111068 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4108054230 0x7f410804b4f0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f410c00e710 tx=0x7f410c00ebd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:43:53.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.763+0000 7f41067fc640 1 -- 172.21.15.57:0/858111068 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f410c017dd0 con 0x7f4108054230 2023-10-19T12:43:53.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.763+0000 7f41067fc640 1 -- 172.21.15.57:0/858111068 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f410c00c9a0 con 0x7f4108054230 2023-10-19T12:43:53.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.764+0000 7f41067fc640 1 -- 172.21.15.57:0/858111068 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f410c00cd90 con 0x7f4108054230 2023-10-19T12:43:53.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.772+0000 7f4116a13640 1 -- 172.21.15.57:0/858111068 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4108034e30 con 0x7f4108054230 2023-10-19T12:43:53.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.772+0000 7f4116a13640 1 -- 172.21.15.57:0/858111068 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41080353f0 con 0x7f4108054230 2023-10-19T12:43:53.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.773+0000 7f41067fc640 1 -- 172.21.15.57:0/858111068 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f410c004020 con 0x7f4108054230 2023-10-19T12:43:53.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.773+0000 7f41067fc640 1 -- 172.21.15.57:0/858111068 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f410c01f650 con 0x7f4108054230 2023-10-19T12:43:53.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.775+0000 7fd8bb59e640 1 -- start start 2023-10-19T12:43:53.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.777+0000 7f8eae7ae640 1 -- 172.21.15.57:0/2119089592 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8ea80b10f0 msgr2=0x7f8ea80cc140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:53.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.777+0000 7f8eae7ae640 1 --2- 172.21.15.57:0/2119089592 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8ea80b10f0 0x7f8ea80cc140 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f8e98005fd0 tx=0x7f8e9802ceb0 comp rx=0 tx=0).stop 2023-10-19T12:43:53.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.777+0000 7f8eae7ae640 1 -- 172.21.15.57:0/2119089592 shutdown_connections 2023-10-19T12:43:53.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.777+0000 7f8eae7ae640 1 --2- 172.21.15.57:0/2119089592 >> v2:172.21.15.179:3300/0 conn(0x7f8ea806ef40 0x7f8ea806f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:53.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.777+0000 7f8eae7ae640 1 --2- 172.21.15.57:0/2119089592 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8ea80b10f0 0x7f8ea80cc140 secure :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f8e98005fd0 tx=0x7f8e9802ceb0 comp rx=0 tx=0).stop 2023-10-19T12:43:53.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.777+0000 7f8eae7ae640 1 -- 172.21.15.57:0/2119089592 >> 172.21.15.57:0/2119089592 conn(0x7f8ea80ae5d0 msgr2=0x7f8ea80ae9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:43:53.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.776+0000 7fd8bb59e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd8b4050160 0x7fd8b40cca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:43:53.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.776+0000 7fd8bb59e640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd8b4053c30 0x7fd8b40de1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:43:53.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.776+0000 7fd8bb59e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd8b40df1e0 0x7fd8b40d6630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:43:53.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.776+0000 7fd8bb59e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b40ccf90 con 0x7fd8b40df1e0 2023-10-19T12:43:53.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.776+0000 7fd8bb59e640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b404a9f0 con 0x7fd8b4050160 2023-10-19T12:43:53.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.776+0000 7fd8bb59e640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b404acd0 con 0x7fd8b4053c30 2023-10-19T12:43:53.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.778+0000 7f850972c640 1 -- 172.21.15.57:0/2311821668 shutdown_connections 2023-10-19T12:43:53.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.778+0000 7fea097fa640 1 -- 172.21.15.57:0/1960099229 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fea2406c510 msgr2=0x7fea24152c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:53.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.778+0000 7fea097fa640 1 --2- 172.21.15.57:0/1960099229 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fea2406c510 0x7fea24152c60 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fea24072650 tx=0x7fea1800e8e0 comp rx=0 tx=0).stop 2023-10-19T12:43:53.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.778+0000 7fea097fa640 1 -- 172.21.15.57:0/1960099229 shutdown_connections 2023-10-19T12:43:53.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.778+0000 7fd8bad9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd8b40df1e0 0x7fd8b40d6630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:43:53.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.778+0000 7fea097fa640 1 --2- 172.21.15.57:0/1960099229 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fea2406c510 0x7fea24152c60 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:53.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.778+0000 7fea097fa640 1 --2- 172.21.15.57:0/1960099229 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fea241544c0 0x7fea2406bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:53.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.778+0000 7fea097fa640 1 --2- 172.21.15.57:0/1960099229 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fea24144340 0x7fea24153f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:53.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.778+0000 7fea097fa640 1 -- 172.21.15.57:0/1960099229 >> 172.21.15.57:0/1960099229 conn(0x7fea240ae720 msgr2=0x7fea2406ae90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:43:53.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.778+0000 7f850972c640 1 -- 172.21.15.57:0/2311821668 wait complete. 2023-10-19T12:43:53.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.778+0000 7fd8bad9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd8b40df1e0 0x7fd8b40d6630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50972/0 (socket says 172.21.15.57:50972) 2023-10-19T12:43:53.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.778+0000 7fd8bad9d640 1 -- 172.21.15.57:0/3084431538 learned_addr learned my addr 172.21.15.57:0/3084431538 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:43:53.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.778+0000 7fea097fa640 1 -- 172.21.15.57:0/1960099229 shutdown_connections 2023-10-19T12:43:53.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.778+0000 7fea097fa640 1 -- 172.21.15.57:0/1960099229 wait complete. 2023-10-19T12:43:53.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.778+0000 7fd8bad9d640 1 -- 172.21.15.57:0/3084431538 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd8b4053c30 msgr2=0x7fd8b40de1b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T12:43:53.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.778+0000 7fd8bad9d640 1 --2- 172.21.15.57:0/3084431538 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd8b4053c30 0x7fd8b40de1b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:53.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.778+0000 7fd8bad9d640 1 -- 172.21.15.57:0/3084431538 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd8b4050160 msgr2=0x7fd8b40cca50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T12:43:53.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.778+0000 7fd8bad9d640 1 --2- 172.21.15.57:0/3084431538 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd8b4050160 0x7fd8b40cca50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:53.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.778+0000 7fd8bad9d640 1 -- 172.21.15.57:0/3084431538 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8b404afb0 con 0x7fd8b40df1e0 2023-10-19T12:43:53.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.779+0000 7fd8bad9d640 1 --2- 172.21.15.57:0/3084431538 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd8b40df1e0 0x7fd8b40d6630 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd8ac00b780 tx=0x7fd8ac00bc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:43:53.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.780+0000 7f8eae7ae640 1 -- 172.21.15.57:0/2119089592 shutdown_connections 2023-10-19T12:43:53.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.780+0000 7fd8aaffd640 1 -- 172.21.15.57:0/3084431538 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8ac00c720 con 0x7fd8b40df1e0 2023-10-19T12:43:53.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.780+0000 7fd8aaffd640 1 -- 172.21.15.57:0/3084431538 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fd8ac00c990 con 0x7fd8b40df1e0 2023-10-19T12:43:53.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.780+0000 7fd8bb59e640 1 -- 172.21.15.57:0/3084431538 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8b404b290 con 0x7fd8b40df1e0 2023-10-19T12:43:53.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.780+0000 7fd8bb59e640 1 -- 172.21.15.57:0/3084431538 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8b40de6f0 con 0x7fd8b40df1e0 2023-10-19T12:43:53.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.780+0000 7fd8aaffd640 1 -- 172.21.15.57:0/3084431538 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8ac00cda0 con 0x7fd8b40df1e0 2023-10-19T12:43:53.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.780+0000 7f850972c640 1 Processor -- start 2023-10-19T12:43:53.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.781+0000 7f850972c640 1 -- start start 2023-10-19T12:43:53.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.782+0000 7f850972c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85041fd8d0 0x7f8504154790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:43:53.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.782+0000 7f850972c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f85041fdcb0 0x7f85040cc1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:43:53.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.782+0000 7f49325a5640 1 -- 172.21.15.57:0/3698341320 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command([{prefix=config set, name=mds_bal_fragment_size_max}] v 0) v1 -- 0x7f49040057d0 con 0x7f492c0709b0 2023-10-19T12:43:53.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.776+0000 7f40f3fff640 1 -- 172.21.15.57:0/858111068 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41080089a0 con 0x7f4108054230 2023-10-19T12:43:53.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.782+0000 7f8503fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85041fd8d0 0x7f8504154790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:43:53.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.782+0000 7f850972c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8504163e10 0x7f85040cc700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:43:53.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.782+0000 7f850972c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8504154cd0 con 0x7f85041fd8d0 2023-10-19T12:43:53.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.782+0000 7f8503fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85041fd8d0 0x7f8504154790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50974/0 (socket says 172.21.15.57:50974) 2023-10-19T12:43:53.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.782+0000 7f850972c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85041643b0 con 0x7f8504163e10 2023-10-19T12:43:53.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.782+0000 7f8503fff640 1 -- 172.21.15.57:0/1659934438 learned_addr learned my addr 172.21.15.57:0/1659934438 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:43:53.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.782+0000 7f850972c640 1 -- 172.21.15.57:0/1659934438 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8504164690 con 0x7f85041fdcb0 2023-10-19T12:43:53.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.781+0000 7f8eae7ae640 1 -- 172.21.15.57:0/2119089592 wait complete. 2023-10-19T12:43:53.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.783+0000 7f8eae7ae640 1 Processor -- start 2023-10-19T12:43:53.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.783+0000 7f8eae7ae640 1 -- start start 2023-10-19T12:43:53.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.784+0000 7f8eae7ae640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8ea806ef40 0x7f8ea81691a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:43:53.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.784+0000 7f8eae7ae640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8ea80c8e50 0x7f8ea815f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:43:53.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.785+0000 7f8eae7ae640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ea815d420 0x7f8ea8163090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:43:53.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.785+0000 7f8eae7ae640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea81696e0 con 0x7f8ea815d420 2023-10-19T12:43:53.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.785+0000 7f8eae7ae640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea815fb10 con 0x7f8ea806ef40 2023-10-19T12:43:53.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.785+0000 7f8eae7ae640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea815fdb0 con 0x7f8ea80c8e50 2023-10-19T12:43:53.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.785+0000 7fd8aaffd640 1 -- 172.21.15.57:0/3084431538 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fd8ac01e740 con 0x7fd8b40df1e0 2023-10-19T12:43:53.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.785+0000 7fd8aaffd640 1 -- 172.21.15.57:0/3084431538 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fd8ac014a20 con 0x7fd8b40df1e0 2023-10-19T12:43:53.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.785+0000 7f8eacfab640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8ea80c8e50 0x7f8ea815f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:43:53.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.785+0000 7f8ead7ac640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8ea806ef40 0x7f8ea81691a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:43:53.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.785+0000 7f8eacfab640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8ea80c8e50 0x7f8ea815f5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3301/0 says I am v2:172.21.15.57:54018/0 (socket says 172.21.15.57:54018) 2023-10-19T12:43:53.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.785+0000 7f8eacfab640 1 -- 172.21.15.57:0/3557275618 learned_addr learned my addr 172.21.15.57:0/3557275618 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:43:53.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.786+0000 7f8eacfab640 1 -- 172.21.15.57:0/3557275618 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8ea806ef40 msgr2=0x7f8ea81691a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:53.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.786+0000 7f8eacfab640 1 --2- 172.21.15.57:0/3557275618 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8ea806ef40 0x7f8ea81691a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:53.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.786+0000 7f8eacfab640 1 -- 172.21.15.57:0/3557275618 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ea815d420 msgr2=0x7f8ea8163090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:53.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.786+0000 7f8eacfab640 1 --2- 172.21.15.57:0/3557275618 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ea815d420 0x7f8ea8163090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:53.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.786+0000 7f8eacfab640 1 -- 172.21.15.57:0/3557275618 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e98009560 con 0x7f8ea80c8e50 2023-10-19T12:43:53.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.786+0000 7f8eacfab640 1 --2- 172.21.15.57:0/3557275618 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8ea80c8e50 0x7f8ea815f5d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8e98037040 tx=0x7f8e98003960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:43:53.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.786+0000 7f8e9dffb640 1 -- 172.21.15.57:0/3557275618 <== mon.2 v2:172.21.15.179:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e9803e470 con 0x7f8ea80c8e50 2023-10-19T12:43:53.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.786+0000 7fd8bb59e640 1 -- 172.21.15.57:0/3084431538 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8a0005250 con 0x7fd8b40df1e0 2023-10-19T12:43:53.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.788+0000 7f41067fc640 1 -- 172.21.15.57:0/858111068 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f410c024070 con 0x7f4108054230 2023-10-19T12:43:53.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.786+0000 7f8e9dffb640 1 -- 172.21.15.57:0/3557275618 <== mon.2 v2:172.21.15.179:3301/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f8e9803e6c0 con 0x7f8ea80c8e50 2023-10-19T12:43:53.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.786+0000 7f8e9dffb640 1 -- 172.21.15.57:0/3557275618 <== mon.2 v2:172.21.15.179:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e9803ea50 con 0x7f8ea80c8e50 2023-10-19T12:43:53.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.787+0000 7f8eae7ae640 1 -- 172.21.15.57:0/3557275618 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ea8160090 con 0x7f8ea80c8e50 2023-10-19T12:43:53.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.789+0000 7f2f1e432640 1 -- 172.21.15.57:0/1804482748 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2f10007580 msgr2=0x7f2f10037bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:53.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.789+0000 7f2f1e432640 1 --2- 172.21.15.57:0/1804482748 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2f10007580 0x7f2f10037bf0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f2f18068940 tx=0x7f2f180a3db0 comp rx=0 tx=0).stop 2023-10-19T12:43:53.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.787+0000 7f8eae7ae640 1 -- 172.21.15.57:0/3557275618 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ea8160650 con 0x7f8ea80c8e50 2023-10-19T12:43:53.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.789+0000 7f8e9dffb640 1 -- 172.21.15.57:0/3557275618 <== mon.2 v2:172.21.15.179:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f8e9803ebb0 con 0x7f8ea80c8e50 2023-10-19T12:43:53.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.789+0000 7f8e9dffb640 1 -- 172.21.15.57:0/3557275618 <== mon.2 v2:172.21.15.179:3301/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f8e980489f0 con 0x7f8ea80c8e50 2023-10-19T12:43:53.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.790+0000 7f8e937fe640 1 -- 172.21.15.57:0/3557275618 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e88005250 con 0x7f8ea80c8e50 2023-10-19T12:43:53.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.792+0000 7efee0f1b640 1 -- 172.21.15.57:0/180008447 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efedc070900 msgr2=0x7efedc06fe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:53.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.792+0000 7efee0f1b640 1 --2- 172.21.15.57:0/180008447 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efedc070900 0x7efedc06fe80 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7efedc066e70 tx=0x7efecc02be60 comp rx=0 tx=0).stop 2023-10-19T12:43:53.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.792+0000 7efee0f1b640 1 -- 172.21.15.57:0/180008447 shutdown_connections 2023-10-19T12:43:53.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.792+0000 7efee0f1b640 1 --2- 172.21.15.57:0/180008447 >> v2:172.21.15.57:3300/0 conn(0x7efedc070ce0 0x7efedc0afbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:53.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.792+0000 7efee0f1b640 1 --2- 172.21.15.57:0/180008447 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efedc070900 0x7efedc06fe80 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:53.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.792+0000 7efee0f1b640 1 -- 172.21.15.57:0/180008447 >> 172.21.15.57:0/180008447 conn(0x7efedc069720 msgr2=0x7efedc069b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:43:53.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.792+0000 7efee0f1b640 1 -- 172.21.15.57:0/180008447 shutdown_connections 2023-10-19T12:43:53.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.792+0000 7efee0f1b640 1 -- 172.21.15.57:0/180008447 wait complete. 2023-10-19T12:43:53.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.793+0000 7f2f1e432640 1 -- 172.21.15.57:0/1804482748 shutdown_connections 2023-10-19T12:43:53.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.793+0000 7f2f1e432640 1 --2- 172.21.15.57:0/1804482748 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2f10007580 0x7f2f10037bf0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:53.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.793+0000 7f2f1e432640 1 -- 172.21.15.57:0/1804482748 >> 172.21.15.57:0/1804482748 conn(0x7f2f10009f80 msgr2=0x7f2f1000a360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:43:53.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.795+0000 7fd8aaffd640 1 -- 172.21.15.57:0/3084431538 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7fd8ac023060 con 0x7fd8b40df1e0 2023-10-19T12:43:53.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.796+0000 7efee0f1b640 1 Processor -- start 2023-10-19T12:43:53.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.793+0000 7f2f1e432640 1 -- 172.21.15.57:0/1804482748 shutdown_connections 2023-10-19T12:43:53.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.793+0000 7f2f1e432640 1 -- 172.21.15.57:0/1804482748 wait complete. 2023-10-19T12:43:53.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.795+0000 7f2f1e432640 1 Processor -- start 2023-10-19T12:43:53.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.795+0000 7f2f1e432640 1 -- start start 2023-10-19T12:43:53.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.796+0000 7f2f1e432640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2f10007580 0x7f2f10168520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:43:53.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.796+0000 7f2f1e432640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2f1004f260 0x7f2f101957f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:43:53.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.796+0000 7f2f1e432640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2f100e31a0 0x7f2f1019ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:43:53.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.796+0000 7f2f1e432640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f10168a60 con 0x7f2f1004f260 2023-10-19T12:43:53.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.796+0000 7f2f1e432640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f10033860 con 0x7f2f10007580 2023-10-19T12:43:53.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.796+0000 7f2f1e432640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f10195d30 con 0x7f2f100e31a0 2023-10-19T12:43:53.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.796+0000 7f2f1dc31640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2f100e31a0 0x7f2f1019ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:43:53.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.797+0000 7f2f1dc31640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2f100e31a0 0x7f2f1019ee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3301/0 says I am v2:172.21.15.57:54024/0 (socket says 172.21.15.57:54024) 2023-10-19T12:43:53.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.797+0000 7f2f1dc31640 1 -- 172.21.15.57:0/1378396156 learned_addr learned my addr 172.21.15.57:0/1378396156 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:43:53.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.797+0000 7f2f1cc2f640 1 --2- 172.21.15.57:0/1378396156 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2f1004f260 0x7f2f101957f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:43:53.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.797+0000 7f2f1d430640 1 --2- 172.21.15.57:0/1378396156 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2f10007580 0x7f2f10168520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:43:53.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.797+0000 7f2f1dc31640 1 -- 172.21.15.57:0/1378396156 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2f10007580 msgr2=0x7f2f10168520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:53.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.797+0000 7f2f1dc31640 1 --2- 172.21.15.57:0/1378396156 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2f10007580 0x7f2f10168520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:53.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.797+0000 7f2f1dc31640 1 -- 172.21.15.57:0/1378396156 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2f1004f260 msgr2=0x7f2f101957f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:53.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.797+0000 7f2f1dc31640 1 --2- 172.21.15.57:0/1378396156 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2f1004f260 0x7f2f101957f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:53.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.797+0000 7f2f1dc31640 1 -- 172.21.15.57:0/1378396156 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f1804e960 con 0x7f2f100e31a0 2023-10-19T12:43:53.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.797+0000 7f2f1cc2f640 1 --2- 172.21.15.57:0/1378396156 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2f1004f260 0x7f2f101957f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:43:53.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.798+0000 7f2f1dc31640 1 --2- 172.21.15.57:0/1378396156 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2f100e31a0 0x7f2f1019ee80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f2f18067d70 tx=0x7f2f180ab5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:43:53.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.799+0000 7f2f0dffb640 1 -- 172.21.15.57:0/1378396156 <== mon.2 v2:172.21.15.179:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f180ab940 con 0x7f2f100e31a0 2023-10-19T12:43:53.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.799+0000 7f2f1e432640 1 -- 172.21.15.57:0/1378396156 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f100330d0 con 0x7f2f100e31a0 2023-10-19T12:43:53.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.800+0000 7f2f1e432640 1 -- 172.21.15.57:0/1378396156 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f100df2f0 con 0x7f2f100e31a0 2023-10-19T12:43:53.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.800+0000 7f2f0dffb640 1 -- 172.21.15.57:0/1378396156 <== mon.2 v2:172.21.15.179:3301/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f2f180abaa0 con 0x7f2f100e31a0 2023-10-19T12:43:53.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.800+0000 7f2f0dffb640 1 -- 172.21.15.57:0/1378396156 <== mon.2 v2:172.21.15.179:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f180abed0 con 0x7f2f100e31a0 2023-10-19T12:43:53.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.800+0000 7f2f0dffb640 1 -- 172.21.15.57:0/1378396156 <== mon.2 v2:172.21.15.179:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f2f18080070 con 0x7f2f100e31a0 2023-10-19T12:43:53.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.800+0000 7f2f0dffb640 1 -- 172.21.15.57:0/1378396156 <== mon.2 v2:172.21.15.179:3301/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f2f18082990 con 0x7f2f100e31a0 2023-10-19T12:43:53.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.801+0000 7f4921ffb640 1 -- 172.21.15.57:0/3698341320 <== mon.0 v2:172.21.15.57:3300/0 8 ==== mon_command_ack([{prefix=config set, name=mds_bal_fragment_size_max}]=0 v10)=0 v10) v1 ==== 127+0+0 (secure 0 0 0) 0x7f4928039070 con 0x7f492c0709b0 2023-10-19T12:43:53.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.802+0000 7fba10ef2640 1 -- 172.21.15.57:0/1068418311 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fba04006390 msgr2=0x7fba040067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:53.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.802+0000 7fba10ef2640 1 --2- 172.21.15.57:0/1068418311 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fba04006390 0x7fba040067e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb9fc009830 tx=0x7fb9fc02be20 comp rx=0 tx=0).stop 2023-10-19T12:43:53.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.802+0000 7fba10ef2640 1 -- 172.21.15.57:0/1068418311 shutdown_connections 2023-10-19T12:43:53.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.802+0000 7fba10ef2640 1 --2- 172.21.15.57:0/1068418311 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fba04006390 0x7fba040067e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:53.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.802+0000 7fba10ef2640 1 --2- 172.21.15.57:0/1068418311 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fba0404f250 0x7fba04005e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:53.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.802+0000 7fba10ef2640 1 -- 172.21.15.57:0/1068418311 >> 172.21.15.57:0/1068418311 conn(0x7fba04009f70 msgr2=0x7fba0400a350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:43:53.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.802+0000 7fba10ef2640 1 -- 172.21.15.57:0/1068418311 shutdown_connections 2023-10-19T12:43:53.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.804+0000 7f2f1e432640 1 -- 172.21.15.57:0/1378396156 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ef8005250 con 0x7f2f100e31a0 2023-10-19T12:43:53.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.802+0000 7efee0f1b640 1 -- start start 2023-10-19T12:43:53.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.803+0000 7efee0f1b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efedc070900 0x7efedc153340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:43:53.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.803+0000 7efee0f1b640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efedc070ce0 0x7efedc13ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:43:53.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.804+0000 7efee0f1b640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7efedc0b0100 0x7efedc143340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:43:53.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.804+0000 7efee0f1b640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efedc1447a0 con 0x7efedc070900 2023-10-19T12:43:53.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.804+0000 7efee0f1b640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efedc143b90 con 0x7efedc070ce0 2023-10-19T12:43:53.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.804+0000 7efee0f1b640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efedc143e70 con 0x7efedc0b0100 2023-10-19T12:43:53.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.802+0000 7fba10ef2640 1 -- 172.21.15.57:0/1068418311 wait complete. 2023-10-19T12:43:53.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.804+0000 7fba10ef2640 1 Processor -- start 2023-10-19T12:43:53.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.783+0000 7f8503fff640 1 -- 172.21.15.57:0/1659934438 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f85041fdcb0 msgr2=0x7f85040cc1c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-19T12:43:53.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.783+0000 7f8503fff640 1 --2- 172.21.15.57:0/1659934438 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f85041fdcb0 0x7f85040cc1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:53.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.783+0000 7f8503fff640 1 -- 172.21.15.57:0/1659934438 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8504163e10 msgr2=0x7f85040cc700 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-19T12:43:53.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.783+0000 7f8503fff640 1 --2- 172.21.15.57:0/1659934438 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8504163e10 0x7f85040cc700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:53.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.783+0000 7f8503fff640 1 -- 172.21.15.57:0/1659934438 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8504164970 con 0x7f85041fd8d0 2023-10-19T12:43:53.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.784+0000 7f8503fff640 1 --2- 172.21.15.57:0/1659934438 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85041fd8d0 0x7f8504154790 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f84f400e940 tx=0x7f84f400ee00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:43:53.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.805+0000 7fba10ef2640 1 -- start start 2023-10-19T12:43:53.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.806+0000 7fba10ef2640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fba0404f250 0x7fba0404e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:43:53.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.806+0000 7fba10ef2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fba040e6f70 0x7fba040d9a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:43:53.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.806+0000 7fba10ef2640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fba040d9f60 0x7fba040e0840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:43:53.833 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'debug_ms', '1'], check_status=True, hostname='smithi057') 2023-10-19T12:43:53.833 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'mds_bal_fragment_size_max', '10000'], check_status=True, hostname='smithi057') 2023-10-19T12:43:53.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.806+0000 7fba10ef2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba040da3f0 con 0x7fba040e6f70 2023-10-19T12:43:53.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.806+0000 7fba10ef2640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba0404d560 con 0x7fba040d9f60 2023-10-19T12:43:53.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.806+0000 7fba10ef2640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba0404d840 con 0x7fba0404f250 2023-10-19T12:43:53.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.806+0000 7fba0a59c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fba0404f250 0x7fba0404e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:43:53.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.807+0000 7fdd15ed7640 1 -- 172.21.15.57:0/2192254349 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd100ac6e0 msgr2=0x7fdd10159800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:53.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.807+0000 7fdd15ed7640 1 --2- 172.21.15.57:0/2192254349 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd100ac6e0 0x7fdd10159800 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fdd04009880 tx=0x7fdd0403cdd0 comp rx=0 tx=0).stop 2023-10-19T12:43:53.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.807+0000 7efedaffd640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efedc070ce0 0x7efedc13ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:43:53.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.807+0000 7efedaffd640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efedc070ce0 0x7efedc13ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.57:33532/0 (socket says 172.21.15.57:33532) 2023-10-19T12:43:53.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.807+0000 7efedaffd640 1 -- 172.21.15.57:0/3850040185 learned_addr learned my addr 172.21.15.57:0/3850040185 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:43:53.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.807+0000 7efedb7fe640 1 --2- 172.21.15.57:0/3850040185 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efedc070900 0x7efedc153340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:43:53.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.807+0000 7efedbfff640 1 --2- 172.21.15.57:0/3850040185 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7efedc0b0100 0x7efedc143340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:43:53.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.806+0000 7fba0a59c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fba0404f250 0x7fba0404e600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3301/0 says I am v2:172.21.15.57:54066/0 (socket says 172.21.15.57:54066) 2023-10-19T12:43:53.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.806+0000 7fba0a59c640 1 -- 172.21.15.57:0/1340831843 learned_addr learned my addr 172.21.15.57:0/1340831843 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:43:53.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.807+0000 7fba0a59c640 1 -- 172.21.15.57:0/1340831843 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fba040d9f60 msgr2=0x7fba040e0840 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T12:43:53.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.807+0000 7fba0a59c640 1 --2- 172.21.15.57:0/1340831843 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fba040d9f60 0x7fba040e0840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:53.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.807+0000 7fba0a59c640 1 -- 172.21.15.57:0/1340831843 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fba040e6f70 msgr2=0x7fba040d9a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:53.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.807+0000 7fba0a59c640 1 --2- 172.21.15.57:0/1340831843 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fba040e6f70 0x7fba040d9a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:53.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.807+0000 7fba0a59c640 1 -- 172.21.15.57:0/1340831843 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9fc009490 con 0x7fba0404f250 2023-10-19T12:43:53.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.807+0000 7fba0a59c640 1 --2- 172.21.15.57:0/1340831843 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fba0404f250 0x7fba0404e600 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fba0000c3c0 tx=0x7fba0000c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:43:53.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.808+0000 7efedaffd640 1 -- 172.21.15.57:0/3850040185 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7efedc0b0100 msgr2=0x7efedc143340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:53.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.808+0000 7efedaffd640 1 --2- 172.21.15.57:0/3850040185 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7efedc0b0100 0x7efedc143340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:53.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.808+0000 7efedaffd640 1 -- 172.21.15.57:0/3850040185 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efedc070900 msgr2=0x7efedc153340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:53.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.808+0000 7efedaffd640 1 --2- 172.21.15.57:0/3850040185 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efedc070900 0x7efedc153340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:53.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.808+0000 7efedaffd640 1 -- 172.21.15.57:0/3850040185 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efecc009560 con 0x7efedc070ce0 2023-10-19T12:43:53.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.808+0000 7f49177fe640 1 -- 172.21.15.57:0/3698341320 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f492c0709b0 msgr2=0x7f492c1570a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:53.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.808+0000 7f49177fe640 1 --2- 172.21.15.57:0/3698341320 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f492c0709b0 0x7f492c1570a0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f4928033000 tx=0x7f4928034d20 comp rx=0 tx=0).stop 2023-10-19T12:43:53.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.808+0000 7f49177fe640 1 -- 172.21.15.57:0/3698341320 shutdown_connections 2023-10-19T12:43:53.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.808+0000 7f49177fe640 1 --2- 172.21.15.57:0/3698341320 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f492c0aabb0 0x7f492c0aed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:53.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.808+0000 7f49177fe640 1 --2- 172.21.15.57:0/3698341320 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f492c1686b0 0x7f492c0aa670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:53.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.808+0000 7f49177fe640 1 --2- 172.21.15.57:0/3698341320 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f492c0709b0 0x7f492c1570a0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:53.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.808+0000 7f49177fe640 1 -- 172.21.15.57:0/3698341320 >> 172.21.15.57:0/3698341320 conn(0x7f492c069720 msgr2=0x7f492c0ac880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:43:53.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.808+0000 7f8e9dffb640 1 -- 172.21.15.57:0/3557275618 <== mon.2 v2:172.21.15.179:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f8ea80cc680 con 0x7f8ea80c8e50 2023-10-19T12:43:53.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.808+0000 7f49177fe640 1 -- 172.21.15.57:0/3698341320 shutdown_connections 2023-10-19T12:43:53.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.809+0000 7f49177fe640 1 -- 172.21.15.57:0/3698341320 wait complete. 2023-10-19T12:43:53.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.809+0000 7efedaffd640 1 --2- 172.21.15.57:0/3850040185 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efedc070ce0 0x7efedc13ee60 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7efed000b4a0 tx=0x7efed000b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:43:53.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.812+0000 7f8500ff9640 1 -- 172.21.15.57:0/1659934438 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84f400ccc0 con 0x7f85041fd8d0 2023-10-19T12:43:53.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.812+0000 7f8500ff9640 1 -- 172.21.15.57:0/1659934438 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f84f400ce20 con 0x7f85041fd8d0 2023-10-19T12:43:53.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.812+0000 7f850972c640 1 -- 172.21.15.57:0/1659934438 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8504164c50 con 0x7f85041fd8d0 2023-10-19T12:43:53.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.812+0000 7f850972c640 1 -- 172.21.15.57:0/1659934438 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8504165450 con 0x7f85041fd8d0 2023-10-19T12:43:53.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.812+0000 7f8500ff9640 1 -- 172.21.15.57:0/1659934438 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84f4005280 con 0x7f85041fd8d0 2023-10-19T12:43:53.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.813+0000 7f8500ff9640 1 -- 172.21.15.57:0/1659934438 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f84f4005460 con 0x7f85041fd8d0 2023-10-19T12:43:53.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.813+0000 7f8500ff9640 1 -- 172.21.15.57:0/1659934438 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f84f4014070 con 0x7f85041fd8d0 2023-10-19T12:43:53.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.816+0000 7f850972c640 1 -- 172.21.15.57:0/1659934438 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84e4005250 con 0x7f85041fd8d0 2023-10-19T12:43:53.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.813+0000 7fdd15ed7640 1 -- 172.21.15.57:0/2192254349 shutdown_connections 2023-10-19T12:43:53.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.813+0000 7fdd15ed7640 1 --2- 172.21.15.57:0/2192254349 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd100ac6e0 0x7fdd10159800 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:53.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.813+0000 7fdd15ed7640 1 -- 172.21.15.57:0/2192254349 >> 172.21.15.57:0/2192254349 conn(0x7fdd100b0cd0 msgr2=0x7fdd100b1f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:43:53.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.814+0000 7fdd15ed7640 1 -- 172.21.15.57:0/2192254349 shutdown_connections 2023-10-19T12:43:53.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.814+0000 7fdd15ed7640 1 -- 172.21.15.57:0/2192254349 wait complete. 2023-10-19T12:43:53.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.815+0000 7fdd15ed7640 1 Processor -- start 2023-10-19T12:43:53.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.816+0000 7fdd15ed7640 1 -- start start 2023-10-19T12:43:53.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.817+0000 7fdd15ed7640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdd10069720 0x7fdd10154a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:43:53.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.817+0000 7fdd15ed7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd100ac300 0x7fdd1015aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:43:53.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.817+0000 7fdd15ed7640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdd100ac6e0 0x7fdd1014fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:43:53.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.817+0000 7fdd15ed7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd1015af70 con 0x7fdd100ac300 2023-10-19T12:43:53.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.817+0000 7fdd15ed7640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd10155150 con 0x7fdd100ac6e0 2023-10-19T12:43:53.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.817+0000 7fdd15ed7640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd10155430 con 0x7fdd10069720 2023-10-19T12:43:53.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.820+0000 7efec3fff640 1 -- 172.21.15.57:0/3850040185 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efed0010610 con 0x7efedc070ce0 2023-10-19T12:43:53.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.820+0000 7efec3fff640 1 -- 172.21.15.57:0/3850040185 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7efed0010860 con 0x7efedc070ce0 2023-10-19T12:43:53.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.820+0000 7efec3fff640 1 -- 172.21.15.57:0/3850040185 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efed0010c70 con 0x7efedc070ce0 2023-10-19T12:43:53.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.820+0000 7fdd0ffff640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdd100ac6e0 0x7fdd1014fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:43:53.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.820+0000 7fdd0ffff640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdd100ac6e0 0x7fdd1014fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.57:33534/0 (socket says 172.21.15.57:33534) 2023-10-19T12:43:53.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.820+0000 7fdd0ffff640 1 -- 172.21.15.57:0/329755988 learned_addr learned my addr 172.21.15.57:0/329755988 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:43:53.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.820+0000 7fdd0f7fe640 1 --2- 172.21.15.57:0/329755988 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdd10069720 0x7fdd10154a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:43:53.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.820+0000 7fdd0effd640 1 --2- 172.21.15.57:0/329755988 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd100ac300 0x7fdd1015aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:43:53.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.820+0000 7efee0f1b640 1 -- 172.21.15.57:0/3850040185 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efedc144150 con 0x7efedc070ce0 2023-10-19T12:43:53.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.821+0000 7fb9faffd640 1 -- 172.21.15.57:0/1340831843 <== mon.2 v2:172.21.15.179:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba00011790 con 0x7fba0404f250 2023-10-19T12:43:53.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.821+0000 7f8500ff9640 1 -- 172.21.15.57:0/1659934438 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f84f40047a0 con 0x7f85041fd8d0 2023-10-19T12:43:53.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.821+0000 7f2f0dffb640 1 -- 172.21.15.57:0/1378396156 <== mon.2 v2:172.21.15.179:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f2f180ba060 con 0x7f2f100e31a0 2023-10-19T12:43:53.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.821+0000 7efee0f1b640 1 -- 172.21.15.57:0/3850040185 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efedc13f3a0 con 0x7efedc070ce0 2023-10-19T12:43:53.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.821+0000 7fdd0ffff640 1 -- 172.21.15.57:0/329755988 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdd10069720 msgr2=0x7fdd10154a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:53.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.821+0000 7fdd0ffff640 1 --2- 172.21.15.57:0/329755988 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdd10069720 0x7fdd10154a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:53.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.821+0000 7fdd0ffff640 1 -- 172.21.15.57:0/329755988 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd100ac300 msgr2=0x7fdd1015aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:53.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.821+0000 7fdd0ffff640 1 --2- 172.21.15.57:0/329755988 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd100ac300 0x7fdd1015aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:53.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.821+0000 7fdd0ffff640 1 -- 172.21.15.57:0/329755988 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd04009490 con 0x7fdd100ac6e0 2023-10-19T12:43:53.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.821+0000 7fba10ef2640 1 -- 172.21.15.57:0/1340831843 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba0404db20 con 0x7fba0404f250 2023-10-19T12:43:53.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.821+0000 7fba10ef2640 1 -- 172.21.15.57:0/1340831843 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba0404e0e0 con 0x7fba0404f250 2023-10-19T12:43:53.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.822+0000 7fb9faffd640 1 -- 172.21.15.57:0/1340831843 <== mon.2 v2:172.21.15.179:3301/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fba000119e0 con 0x7fba0404f250 2023-10-19T12:43:53.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.822+0000 7fba10ef2640 1 -- 172.21.15.57:0/1340831843 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba0404e300 con 0x7fba0404f250 2023-10-19T12:43:53.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.824+0000 7fb9faffd640 1 -- 172.21.15.57:0/1340831843 <== mon.2 v2:172.21.15.179:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba00011e30 con 0x7fba0404f250 2023-10-19T12:43:53.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.824+0000 7fb9faffd640 1 -- 172.21.15.57:0/1340831843 <== mon.2 v2:172.21.15.179:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fba0001c760 con 0x7fba0404f250 2023-10-19T12:43:53.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.824+0000 7fb9faffd640 1 -- 172.21.15.57:0/1340831843 <== mon.2 v2:172.21.15.179:3301/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fba00026830 con 0x7fba0404f250 2023-10-19T12:43:53.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.825+0000 7efec3fff640 1 -- 172.21.15.57:0/3850040185 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7efed0019750 con 0x7efedc070ce0 2023-10-19T12:43:53.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.824+0000 7fdd0effd640 1 --2- 172.21.15.57:0/329755988 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd100ac300 0x7fdd1015aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:43:53.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.825+0000 7fdd0ffff640 1 --2- 172.21.15.57:0/329755988 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdd100ac6e0 0x7fdd1014fc50 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fdd0403d310 tx=0x7fdd040459d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:43:53.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.825+0000 7efee0f1b640 1 -- 172.21.15.57:0/3850040185 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efedc144950 con 0x7efedc070ce0 2023-10-19T12:43:53.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.826+0000 7fb9faffd640 1 -- 172.21.15.57:0/1340831843 <== mon.2 v2:172.21.15.179:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7fba00026bb0 con 0x7fba0404f250 2023-10-19T12:43:53.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.831+0000 7efec3fff640 1 -- 172.21.15.57:0/3850040185 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7efed0017b40 con 0x7efedc070ce0 2023-10-19T12:43:53.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.827+0000 7fc9acc4f640 1 -- 172.21.15.57:0/4204185459 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc9a003a630 msgr2=0x7fc9a00dffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:53.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.827+0000 7fc9acc4f640 1 --2- 172.21.15.57:0/4204185459 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc9a003a630 0x7fc9a00dffd0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc9a80685a0 tx=0x7fc9a80a3cf0 comp rx=0 tx=0).stop 2023-10-19T12:43:53.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.827+0000 7fc9acc4f640 1 -- 172.21.15.57:0/4204185459 shutdown_connections 2023-10-19T12:43:53.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.827+0000 7fc9acc4f640 1 --2- 172.21.15.57:0/4204185459 >> v2:172.21.15.179:3300/0 conn(0x7fc9a0039c80 0x7fc9a0052aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:53.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.827+0000 7fc9acc4f640 1 --2- 172.21.15.57:0/4204185459 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc9a003a630 0x7fc9a00dffd0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:53.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.827+0000 7fc9acc4f640 1 -- 172.21.15.57:0/4204185459 >> 172.21.15.57:0/4204185459 conn(0x7fc9a0054ed0 msgr2=0x7fc9a00552e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:43:53.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.836+0000 7efec3fff640 1 -- 172.21.15.57:0/3850040185 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7efed0037b40 con 0x7efedc070ce0 2023-10-19T12:43:53.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.834+0000 7fdcf7fff640 1 -- 172.21.15.57:0/329755988 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd04005220 con 0x7fdd100ac6e0 2023-10-19T12:43:53.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.834+0000 7fdcf7fff640 1 -- 172.21.15.57:0/329755988 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fdd04005470 con 0x7fdd100ac6e0 2023-10-19T12:43:53.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.834+0000 7fdcf7fff640 1 -- 172.21.15.57:0/329755988 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd04004a00 con 0x7fdd100ac6e0 2023-10-19T12:43:53.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.836+0000 7fdd15ed7640 1 -- 172.21.15.57:0/329755988 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd10155710 con 0x7fdd100ac6e0 2023-10-19T12:43:53.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.836+0000 7fdd15ed7640 1 -- 172.21.15.57:0/329755988 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd10155cd0 con 0x7fdd100ac6e0 2023-10-19T12:43:53.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.837+0000 7fdd15ed7640 1 -- 172.21.15.57:0/329755988 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd10155ef0 con 0x7fdd100ac6e0 2023-10-19T12:43:53.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.837+0000 7fdcf7fff640 1 -- 172.21.15.57:0/329755988 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fdd0404c050 con 0x7fdd100ac6e0 2023-10-19T12:43:53.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.838+0000 7fdcf7fff640 1 -- 172.21.15.57:0/329755988 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fdd04003b40 con 0x7fdd100ac6e0 2023-10-19T12:43:53.864 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'mds_bal_merge_size', '5'], check_status=True, hostname='smithi057') 2023-10-19T12:43:53.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.836+0000 7fc9acc4f640 1 -- 172.21.15.57:0/4204185459 shutdown_connections 2023-10-19T12:43:53.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.842+0000 7fc9acc4f640 1 -- 172.21.15.57:0/4204185459 wait complete. 2023-10-19T12:43:53.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.844+0000 7fc9acc4f640 1 Processor -- start 2023-10-19T12:43:53.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.844+0000 7fc9acc4f640 1 -- start start 2023-10-19T12:43:53.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.845+0000 7fc9acc4f640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc9a0039c80 0x7fc9a01769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:43:53.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.845+0000 7fc9acc4f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc9a0052fe0 0x7fc9a00d6720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:43:53.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.845+0000 7fc9acc4f640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc9a00d6c60 0x7fc9a004e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:43:53.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.845+0000 7fc9acc4f640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9a0176f00 con 0x7fc9a0052fe0 2023-10-19T12:43:53.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.845+0000 7fc9acc4f640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9a00d71e0 con 0x7fc9a00d6c60 2023-10-19T12:43:53.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.845+0000 7fc9acc4f640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9a00e09f0 con 0x7fc9a0039c80 2023-10-19T12:43:53.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.845+0000 7fc9a6d9d640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc9a00d6c60 0x7fc9a004e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:43:53.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.845+0000 7fc9a6d9d640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc9a00d6c60 0x7fc9a004e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.57:33550/0 (socket says 172.21.15.57:33550) 2023-10-19T12:43:53.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.845+0000 7fc9a6d9d640 1 -- 172.21.15.57:0/2776583341 learned_addr learned my addr 172.21.15.57:0/2776583341 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:43:53.868 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.846+0000 7fc9a659c640 1 --2- 172.21.15.57:0/2776583341 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc9a0039c80 0x7fc9a01769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:43:53.868 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.846+0000 7fc9a5d9b640 1 --2- 172.21.15.57:0/2776583341 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc9a0052fe0 0x7fc9a00d6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:43:53.868 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.846+0000 7fc9a6d9d640 1 -- 172.21.15.57:0/2776583341 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc9a0039c80 msgr2=0x7fc9a01769c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:53.868 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.846+0000 7fc9a6d9d640 1 --2- 172.21.15.57:0/2776583341 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc9a0039c80 0x7fc9a01769c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:53.868 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.846+0000 7fc9a6d9d640 1 -- 172.21.15.57:0/2776583341 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc9a0052fe0 msgr2=0x7fc9a00d6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:53.869 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.846+0000 7fc9a6d9d640 1 --2- 172.21.15.57:0/2776583341 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc9a0052fe0 0x7fc9a00d6720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:53.869 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.846+0000 7fc9a6d9d640 1 -- 172.21.15.57:0/2776583341 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9a00e0cd0 con 0x7fc9a00d6c60 2023-10-19T12:43:53.869 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.846+0000 7fc9a5d9b640 1 --2- 172.21.15.57:0/2776583341 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc9a0052fe0 0x7fc9a00d6720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:43:53.869 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.846+0000 7fc9a6d9d640 1 --2- 172.21.15.57:0/2776583341 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc9a00d6c60 0x7fc9a004e3e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc99800c390 tx=0x7fc99800c850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:43:53.869 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.846+0000 7fc996ffd640 1 -- 172.21.15.57:0/2776583341 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc998004230 con 0x7fc9a00d6c60 2023-10-19T12:43:53.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.847+0000 7fc9acc4f640 1 -- 172.21.15.57:0/2776583341 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9a00e0fb0 con 0x7fc9a00d6c60 2023-10-19T12:43:53.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.847+0000 7fc9acc4f640 1 -- 172.21.15.57:0/2776583341 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9a00e3760 con 0x7fc9a00d6c60 2023-10-19T12:43:53.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.848+0000 7fdcf7fff640 1 -- 172.21.15.57:0/329755988 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7fdd04075b40 con 0x7fdd100ac6e0 2023-10-19T12:43:53.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.848+0000 7fc996ffd640 1 -- 172.21.15.57:0/2776583341 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fc9980044a0 con 0x7fc9a00d6c60 2023-10-19T12:43:53.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.848+0000 7fc996ffd640 1 -- 172.21.15.57:0/2776583341 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc998003830 con 0x7fc9a00d6c60 2023-10-19T12:43:53.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.848+0000 7fc996ffd640 1 -- 172.21.15.57:0/2776583341 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fc998003d50 con 0x7fc9a00d6c60 2023-10-19T12:43:53.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.848+0000 7fc996ffd640 1 -- 172.21.15.57:0/2776583341 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fc99801c780 con 0x7fc9a00d6c60 2023-10-19T12:43:53.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.848+0000 7fc9acc4f640 1 -- 172.21.15.57:0/2776583341 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9a0008de0 con 0x7fc9a00d6c60 2023-10-19T12:43:53.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.853+0000 7fc996ffd640 1 -- 172.21.15.57:0/2776583341 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7fc998017070 con 0x7fc9a00d6c60 2023-10-19T12:43:53.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.871+0000 7fabf31e2640 1 -- 172.21.15.57:0/2612779313 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command([{prefix=config set, name=mds_bal_split_bits}] v 0) v1 -- 0x7fabd4005870 con 0x7fabe404c9c0 2023-10-19T12:43:53.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.875+0000 7fabe2ffd640 1 -- 172.21.15.57:0/2612779313 <== mon.1 v2:172.21.15.179:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_split_bits}]=0 v11)=0 v11) v1 ==== 116+0+0 (secure 0 0 0) 0x7fabdc006c30 con 0x7fabe404c9c0 2023-10-19T12:43:53.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.875+0000 7fc1816a9640 1 -- 172.21.15.57:0/2765127877 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command([{prefix=config set, name=mds_bal_merge_size}] v 0) v1 -- 0x7fc15c0050c0 con 0x7fc17c0ac7a0 2023-10-19T12:43:53.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.883+0000 7fc1637fe640 1 -- 172.21.15.57:0/2765127877 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_merge_size}]=0 v12)=0 v12) v1 ==== 116+0+0 (secure 0 0 0) 0x7fc16c02a0b0 con 0x7fc17c0ac7a0 2023-10-19T12:43:53.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.887+0000 7fc1816a9640 1 -- 172.21.15.57:0/2765127877 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc17c0ac7a0 msgr2=0x7fc17c20d610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:53.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.887+0000 7fc1816a9640 1 --2- 172.21.15.57:0/2765127877 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc17c0ac7a0 0x7fc17c20d610 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fc16c00c780 tx=0x7fc16c00cb80 comp rx=0 tx=0).stop 2023-10-19T12:43:53.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.887+0000 7fc1816a9640 1 -- 172.21.15.57:0/2765127877 shutdown_connections 2023-10-19T12:43:53.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.887+0000 7fc1816a9640 1 --2- 172.21.15.57:0/2765127877 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc17c15afe0 0x7fc17c15d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:53.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.887+0000 7fc1816a9640 1 --2- 172.21.15.57:0/2765127877 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc17c0ac7a0 0x7fc17c20d610 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:53.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.887+0000 7fc1816a9640 1 --2- 172.21.15.57:0/2765127877 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc17c069720 0x7fc17c1e3cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:53.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.887+0000 7fc1816a9640 1 -- 172.21.15.57:0/2765127877 >> 172.21.15.57:0/2765127877 conn(0x7fc17c0b0cd0 msgr2=0x7fc17c159650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:43:53.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.889+0000 7fc1816a9640 1 -- 172.21.15.57:0/2765127877 shutdown_connections 2023-10-19T12:43:53.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.889+0000 7fc1816a9640 1 -- 172.21.15.57:0/2765127877 wait complete. 2023-10-19T12:43:53.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.890+0000 7fabf31e2640 1 -- 172.21.15.57:0/2612779313 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fabe404c9c0 msgr2=0x7fabe40d7cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:53.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.890+0000 7fabf31e2640 1 --2- 172.21.15.57:0/2612779313 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fabe404c9c0 0x7fabe40d7cd0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fabdc00c3b0 tx=0x7fabdc001d20 comp rx=0 tx=0).stop 2023-10-19T12:43:53.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.890+0000 7fabf31e2640 1 -- 172.21.15.57:0/2612779313 shutdown_connections 2023-10-19T12:43:53.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.890+0000 7fabf31e2640 1 --2- 172.21.15.57:0/2612779313 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fabe404c9c0 0x7fabe40d7cd0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:53.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.890+0000 7fabf31e2640 1 --2- 172.21.15.57:0/2612779313 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fabe4035ff0 0x7fabe40db670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:53.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.890+0000 7fabf31e2640 1 --2- 172.21.15.57:0/2612779313 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fabe4035c10 0x7fabe40db130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:53.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.890+0000 7fabf31e2640 1 -- 172.21.15.57:0/2612779313 >> 172.21.15.57:0/2612779313 conn(0x7fabe4009f80 msgr2=0x7fabe40e0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:43:53.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.890+0000 7fabf31e2640 1 -- 172.21.15.57:0/2612779313 shutdown_connections 2023-10-19T12:43:53.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.890+0000 7fabf31e2640 1 -- 172.21.15.57:0/2612779313 wait complete. 2023-10-19T12:43:53.907 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'mds_bal_split_bits', '3'], check_status=True, hostname='smithi057') 2023-10-19T12:43:53.908 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'mds_bal_split_size', '100'], check_status=True, hostname='smithi057') 2023-10-19T12:43:53.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.936+0000 7f250e7fc640 1 -- 172.21.15.57:0/900337064 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command([{prefix=config set, name=mds_bal_split_size}] v 0) v1 -- 0x7f25140050c0 con 0x7f25280af760 2023-10-19T12:43:53.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.941+0000 7f2524ff9640 1 -- 172.21.15.57:0/900337064 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_split_size}]=0 v13)=0 v13) v1 ==== 118+0+0 (secure 0 0 0) 0x7f25200290b0 con 0x7f25280af760 2023-10-19T12:43:53.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.953+0000 7f250e7fc640 1 -- 172.21.15.57:0/900337064 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25280af760 msgr2=0x7f252813fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:53.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.953+0000 7f250e7fc640 1 --2- 172.21.15.57:0/900337064 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25280af760 0x7f252813fa70 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f252000caf0 tx=0x7f252000cef0 comp rx=0 tx=0).stop 2023-10-19T12:43:53.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.953+0000 7f250e7fc640 1 -- 172.21.15.57:0/900337064 shutdown_connections 2023-10-19T12:43:53.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.953+0000 7f250e7fc640 1 --2- 172.21.15.57:0/900337064 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f252815a7f0 0x7f25280cabd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:53.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.953+0000 7f250e7fc640 1 --2- 172.21.15.57:0/900337064 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f25280b0f90 0x7f252812a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:53.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.953+0000 7f250e7fc640 1 --2- 172.21.15.57:0/900337064 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25280af760 0x7f252813fa70 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:53.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.953+0000 7f250e7fc640 1 -- 172.21.15.57:0/900337064 >> 172.21.15.57:0/900337064 conn(0x7f25280ae460 msgr2=0x7f252806ed70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:43:53.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.953+0000 7f250e7fc640 1 -- 172.21.15.57:0/900337064 shutdown_connections 2023-10-19T12:43:53.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.953+0000 7f250e7fc640 1 -- 172.21.15.57:0/900337064 wait complete. 2023-10-19T12:43:53.970 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'mds_debug_frag', 'True'], check_status=True, hostname='smithi057') 2023-10-19T12:43:53.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.983+0000 7f850972c640 1 -- 172.21.15.57:0/1659934438 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command([{prefix=config set, name=mds_debug_scatterstat}] v 0) v1 -- 0x7f84e40057d0 con 0x7f85041fd8d0 2023-10-19T12:43:53.992 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:53.990+0000 7f8500ff9640 1 -- 172.21.15.57:0/1659934438 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_debug_scatterstat}]=0 v14)=0 v14) v1 ==== 122+0+0 (secure 0 0 0) 0x7f84f4004940 con 0x7f85041fd8d0 2023-10-19T12:43:54.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.000+0000 7f84ea7fc640 1 -- 172.21.15.57:0/1659934438 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85041fd8d0 msgr2=0x7f8504154790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:54.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.000+0000 7f84ea7fc640 1 --2- 172.21.15.57:0/1659934438 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85041fd8d0 0x7f8504154790 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f84f400e940 tx=0x7f84f400ee00 comp rx=0 tx=0).stop 2023-10-19T12:43:54.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.000+0000 7f84ea7fc640 1 -- 172.21.15.57:0/1659934438 shutdown_connections 2023-10-19T12:43:54.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.000+0000 7f84ea7fc640 1 --2- 172.21.15.57:0/1659934438 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8504163e10 0x7f85040cc700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:54.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.000+0000 7f84ea7fc640 1 --2- 172.21.15.57:0/1659934438 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f85041fdcb0 0x7f85040cc1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:54.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.000+0000 7f84ea7fc640 1 --2- 172.21.15.57:0/1659934438 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85041fd8d0 0x7f8504154790 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:54.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.000+0000 7f84ea7fc640 1 -- 172.21.15.57:0/1659934438 >> 172.21.15.57:0/1659934438 conn(0x7f8504069720 msgr2=0x7f85040ae850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:43:54.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.000+0000 7f84ea7fc640 1 -- 172.21.15.57:0/1659934438 shutdown_connections 2023-10-19T12:43:54.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.000+0000 7f84ea7fc640 1 -- 172.21.15.57:0/1659934438 wait complete. 2023-10-19T12:43:54.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.002+0000 7fba10ef2640 1 -- 172.21.15.57:0/1340831843 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_command([{prefix=config set, name=mds_op_complaint_time}] v 0) v1 -- 0x7fba04008990 con 0x7fba0404f250 2023-10-19T12:43:54.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.003+0000 7fd8bb59e640 1 -- 172.21.15.57:0/3084431538 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command([{prefix=config set, name=mds_debug_frag}] v 0) v1 -- 0x7fd8a0005870 con 0x7fd8b40df1e0 2023-10-19T12:43:54.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.012+0000 7fd8aaffd640 1 -- 172.21.15.57:0/3084431538 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_debug_frag}]=0 v16)=0 v16) v1 ==== 115+0+0 (secure 0 0 0) 0x7fd8ac019070 con 0x7fd8b40df1e0 2023-10-19T12:43:54.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.017+0000 7fb9faffd640 1 -- 172.21.15.57:0/1340831843 <== mon.2 v2:172.21.15.179:3301/0 7 ==== mon_command_ack([{prefix=config set, name=mds_op_complaint_time}]=0 v15)=0 v15) v1 ==== 121+0+0 (secure 0 0 0) 0x7fba00026da0 con 0x7fba0404f250 2023-10-19T12:43:54.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.019+0000 7fba10ef2640 1 -- 172.21.15.57:0/1340831843 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fba0404f250 msgr2=0x7fba0404e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:54.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.019+0000 7fba10ef2640 1 --2- 172.21.15.57:0/1340831843 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fba0404f250 0x7fba0404e600 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fba0000c3c0 tx=0x7fba0000c880 comp rx=0 tx=0).stop 2023-10-19T12:43:54.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.019+0000 7fba10ef2640 1 -- 172.21.15.57:0/1340831843 shutdown_connections 2023-10-19T12:43:54.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.019+0000 7fba10ef2640 1 --2- 172.21.15.57:0/1340831843 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fba040d9f60 0x7fba040e0840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:54.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.019+0000 7fba10ef2640 1 --2- 172.21.15.57:0/1340831843 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fba040e6f70 0x7fba040d9a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:54.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.019+0000 7fba10ef2640 1 --2- 172.21.15.57:0/1340831843 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fba0404f250 0x7fba0404e600 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:54.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.019+0000 7fba10ef2640 1 -- 172.21.15.57:0/1340831843 >> 172.21.15.57:0/1340831843 conn(0x7fba04009f70 msgr2=0x7fba04035280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:43:54.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.019+0000 7fba10ef2640 1 -- 172.21.15.57:0/1340831843 shutdown_connections 2023-10-19T12:43:54.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.019+0000 7fba10ef2640 1 -- 172.21.15.57:0/1340831843 wait complete. 2023-10-19T12:43:54.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.020+0000 7fc9acc4f640 1 -- 172.21.15.57:0/2776583341 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command([{prefix=config set, name=rados_osd_op_timeout}] v 0) v1 -- 0x7fc9a00057a0 con 0x7fc9a00d6c60 2023-10-19T12:43:54.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.022+0000 7fd8bb59e640 1 -- 172.21.15.57:0/3084431538 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd8b40df1e0 msgr2=0x7fd8b40d6630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:54.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.022+0000 7fd8bb59e640 1 --2- 172.21.15.57:0/3084431538 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd8b40df1e0 0x7fd8b40d6630 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd8ac00b780 tx=0x7fd8ac00bc40 comp rx=0 tx=0).stop 2023-10-19T12:43:54.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.022+0000 7fd8bb59e640 1 -- 172.21.15.57:0/3084431538 shutdown_connections 2023-10-19T12:43:54.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.022+0000 7fd8bb59e640 1 --2- 172.21.15.57:0/3084431538 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd8b40df1e0 0x7fd8b40d6630 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:54.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.022+0000 7fd8bb59e640 1 --2- 172.21.15.57:0/3084431538 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd8b4053c30 0x7fd8b40de1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:54.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.022+0000 7fd8bb59e640 1 --2- 172.21.15.57:0/3084431538 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd8b4050160 0x7fd8b40cca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:54.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.022+0000 7fd8bb59e640 1 -- 172.21.15.57:0/3084431538 >> 172.21.15.57:0/3084431538 conn(0x7fd8b4009f70 msgr2=0x7fd8b40dfd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:43:54.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.023+0000 7fd8bb59e640 1 -- 172.21.15.57:0/3084431538 shutdown_connections 2023-10-19T12:43:54.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.023+0000 7fd8bb59e640 1 -- 172.21.15.57:0/3084431538 wait complete. 2023-10-19T12:43:54.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.024+0000 7fdd15ed7640 1 -- 172.21.15.57:0/329755988 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command([{prefix=config set, name=mon_op_complaint_time}] v 0) v1 -- 0x7fdd100ae5c0 con 0x7fdd100ac6e0 2023-10-19T12:43:54.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.026+0000 7f40f3fff640 1 -- 172.21.15.57:0/858111068 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command([{prefix=config set, name=osd_op_complaint_time}] v 0) v1 -- 0x7f4108005360 con 0x7f4108054230 2023-10-19T12:43:54.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.027+0000 7fc996ffd640 1 -- 172.21.15.57:0/2776583341 <== mon.1 v2:172.21.15.179:3300/0 7 ==== mon_command_ack([{prefix=config set, name=rados_osd_op_timeout}]=0 v17)=0 v17) v1 ==== 120+0+0 (secure 0 0 0) 0x7fc9a00e0510 con 0x7fc9a00d6c60 2023-10-19T12:43:54.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.029+0000 7fc994ff9640 1 -- 172.21.15.57:0/2776583341 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc9a00d6c60 msgr2=0x7fc9a004e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:54.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.029+0000 7fc994ff9640 1 --2- 172.21.15.57:0/2776583341 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc9a00d6c60 0x7fc9a004e3e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc99800c390 tx=0x7fc99800c850 comp rx=0 tx=0).stop 2023-10-19T12:43:54.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.029+0000 7fc994ff9640 1 -- 172.21.15.57:0/2776583341 shutdown_connections 2023-10-19T12:43:54.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.029+0000 7fc994ff9640 1 --2- 172.21.15.57:0/2776583341 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc9a00d6c60 0x7fc9a004e3e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:54.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.029+0000 7fc994ff9640 1 --2- 172.21.15.57:0/2776583341 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc9a0052fe0 0x7fc9a00d6720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:54.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.029+0000 7fc994ff9640 1 --2- 172.21.15.57:0/2776583341 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc9a0039c80 0x7fc9a01769c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:54.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.029+0000 7fc994ff9640 1 -- 172.21.15.57:0/2776583341 >> 172.21.15.57:0/2776583341 conn(0x7fc9a0054ed0 msgr2=0x7fc9a0006b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:43:54.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.029+0000 7fc994ff9640 1 -- 172.21.15.57:0/2776583341 shutdown_connections 2023-10-19T12:43:54.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.029+0000 7fc994ff9640 1 -- 172.21.15.57:0/2776583341 wait complete. 2023-10-19T12:43:54.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.031+0000 7fdcf7fff640 1 -- 172.21.15.57:0/329755988 <== mon.1 v2:172.21.15.179:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_op_complaint_time}]=0 v18)=0 v18) v1 ==== 121+0+0 (secure 0 0 0) 0x7fdd04064030 con 0x7fdd100ac6e0 2023-10-19T12:43:54.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.032+0000 7fdcf5ffb640 1 -- 172.21.15.57:0/329755988 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdd100ac6e0 msgr2=0x7fdd1014fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:54.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.032+0000 7fdcf5ffb640 1 --2- 172.21.15.57:0/329755988 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdd100ac6e0 0x7fdd1014fc50 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fdd0403d310 tx=0x7fdd040459d0 comp rx=0 tx=0).stop 2023-10-19T12:43:54.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.032+0000 7fdcf5ffb640 1 -- 172.21.15.57:0/329755988 shutdown_connections 2023-10-19T12:43:54.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.032+0000 7fdcf5ffb640 1 --2- 172.21.15.57:0/329755988 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdd100ac6e0 0x7fdd1014fc50 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:54.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.032+0000 7fdcf5ffb640 1 --2- 172.21.15.57:0/329755988 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd100ac300 0x7fdd1015aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:54.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.032+0000 7fdcf5ffb640 1 --2- 172.21.15.57:0/329755988 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdd10069720 0x7fdd10154a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:54.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.032+0000 7fdcf5ffb640 1 -- 172.21.15.57:0/329755988 >> 172.21.15.57:0/329755988 conn(0x7fdd100b0cd0 msgr2=0x7fdd100aa290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:43:54.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.033+0000 7fdcf5ffb640 1 -- 172.21.15.57:0/329755988 shutdown_connections 2023-10-19T12:43:54.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.033+0000 7fdcf5ffb640 1 -- 172.21.15.57:0/329755988 wait complete. 2023-10-19T12:43:54.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.036+0000 7f2f1e432640 1 -- 172.21.15.57:0/1378396156 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_command([{prefix=config set, name=mds_verify_scatter}] v 0) v1 -- 0x7f2ef8005500 con 0x7f2f100e31a0 2023-10-19T12:43:54.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.038+0000 7efee0f1b640 1 -- 172.21.15.57:0/3850040185 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command([{prefix=config set, name=rados_mon_op_timeout}] v 0) v1 -- 0x7efedc04da70 con 0x7efedc070ce0 2023-10-19T12:43:54.042 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'mds_debug_scatterstat', 'True'], check_status=True, hostname='smithi057') 2023-10-19T12:43:54.043 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'mds_op_complaint_time', '180'], check_status=True, hostname='smithi057') 2023-10-19T12:43:54.043 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'mds_verify_scatter', 'True'], check_status=True, hostname='smithi057') 2023-10-19T12:43:54.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.041+0000 7f41067fc640 1 -- 172.21.15.57:0/858111068 <== mon.1 v2:172.21.15.179:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_op_complaint_time}]=0 v19)=0 v19) v1 ==== 121+0+0 (secure 0 0 0) 0x7f410c004460 con 0x7f4108054230 2023-10-19T12:43:54.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.042+0000 7f40f3fff640 1 -- 172.21.15.57:0/858111068 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4108054230 msgr2=0x7f410804b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:54.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.042+0000 7f40f3fff640 1 --2- 172.21.15.57:0/858111068 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4108054230 0x7f410804b4f0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f410c00e710 tx=0x7f410c00ebd0 comp rx=0 tx=0).stop 2023-10-19T12:43:54.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.042+0000 7f40f3fff640 1 -- 172.21.15.57:0/858111068 shutdown_connections 2023-10-19T12:43:54.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.042+0000 7f40f3fff640 1 --2- 172.21.15.57:0/858111068 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4108035bf0 0x7f41080f9730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:54.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.042+0000 7f40f3fff640 1 --2- 172.21.15.57:0/858111068 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4108100fe0 0x7f41080ee040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:54.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.042+0000 7f40f3fff640 1 --2- 172.21.15.57:0/858111068 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4108054230 0x7f410804b4f0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:54.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.042+0000 7f40f3fff640 1 -- 172.21.15.57:0/858111068 >> 172.21.15.57:0/858111068 conn(0x7f4108009f80 msgr2=0x7f41080ffda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:43:54.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.043+0000 7f40f3fff640 1 -- 172.21.15.57:0/858111068 shutdown_connections 2023-10-19T12:43:54.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.043+0000 7f40f3fff640 1 -- 172.21.15.57:0/858111068 wait complete. 2023-10-19T12:43:54.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.044+0000 7efec3fff640 1 -- 172.21.15.57:0/3850040185 <== mon.1 v2:172.21.15.179:3300/0 7 ==== mon_command_ack([{prefix=config set, name=rados_mon_op_timeout}]=0 v20)=0 v20) v1 ==== 120+0+0 (secure 0 0 0) 0x7efed0026020 con 0x7efedc070ce0 2023-10-19T12:43:54.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.045+0000 7f8e937fe640 1 -- 172.21.15.57:0/3557275618 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_command([{prefix=config set, name=osd_op_complaint_time}] v 0) v1 -- 0x7f8e880050c0 con 0x7f8ea80c8e50 2023-10-19T12:43:54.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.046+0000 7efee0f1b640 1 -- 172.21.15.57:0/3850040185 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efedc070ce0 msgr2=0x7efedc13ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:54.049 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.046+0000 7efee0f1b640 1 --2- 172.21.15.57:0/3850040185 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efedc070ce0 0x7efedc13ee60 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7efed000b4a0 tx=0x7efed000b960 comp rx=0 tx=0).stop 2023-10-19T12:43:54.049 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.046+0000 7efee0f1b640 1 -- 172.21.15.57:0/3850040185 shutdown_connections 2023-10-19T12:43:54.049 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.046+0000 7efee0f1b640 1 --2- 172.21.15.57:0/3850040185 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7efedc0b0100 0x7efedc143340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:54.049 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.046+0000 7efee0f1b640 1 --2- 172.21.15.57:0/3850040185 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efedc070ce0 0x7efedc13ee60 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:54.049 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.046+0000 7efee0f1b640 1 --2- 172.21.15.57:0/3850040185 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efedc070900 0x7efedc153340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:54.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.046+0000 7efee0f1b640 1 -- 172.21.15.57:0/3850040185 >> 172.21.15.57:0/3850040185 conn(0x7efedc069720 msgr2=0x7efedc146030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:43:54.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.046+0000 7efee0f1b640 1 -- 172.21.15.57:0/3850040185 shutdown_connections 2023-10-19T12:43:54.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.046+0000 7efee0f1b640 1 -- 172.21.15.57:0/3850040185 wait complete. 2023-10-19T12:43:54.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.047+0000 7f2f0dffb640 1 -- 172.21.15.57:0/1378396156 <== mon.2 v2:172.21.15.179:3301/0 7 ==== mon_command_ack([{prefix=config set, name=mds_verify_scatter}]=0 v21)=0 v21) v1 ==== 119+0+0 (secure 0 0 0) 0x7f2f18087070 con 0x7f2f100e31a0 2023-10-19T12:43:54.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.049+0000 7f2f1e432640 1 -- 172.21.15.57:0/1378396156 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2f100e31a0 msgr2=0x7f2f1019ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:54.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.049+0000 7f2f1e432640 1 --2- 172.21.15.57:0/1378396156 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2f100e31a0 0x7f2f1019ee80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f2f18067d70 tx=0x7f2f180ab5d0 comp rx=0 tx=0).stop 2023-10-19T12:43:54.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.049+0000 7f2f1e432640 1 -- 172.21.15.57:0/1378396156 shutdown_connections 2023-10-19T12:43:54.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.049+0000 7f2f1e432640 1 --2- 172.21.15.57:0/1378396156 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2f100e31a0 0x7f2f1019ee80 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:54.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.049+0000 7f2f1e432640 1 --2- 172.21.15.57:0/1378396156 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2f1004f260 0x7f2f101957f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:54.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.049+0000 7f2f1e432640 1 --2- 172.21.15.57:0/1378396156 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2f10007580 0x7f2f10168520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:54.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.049+0000 7f2f1e432640 1 -- 172.21.15.57:0/1378396156 >> 172.21.15.57:0/1378396156 conn(0x7f2f10009f80 msgr2=0x7f2f10035260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:43:54.054 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.050+0000 7f2f1e432640 1 -- 172.21.15.57:0/1378396156 shutdown_connections 2023-10-19T12:43:54.054 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.050+0000 7f2f1e432640 1 -- 172.21.15.57:0/1378396156 wait complete. 2023-10-19T12:43:54.055 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.052+0000 7f8e9dffb640 1 -- 172.21.15.57:0/3557275618 <== mon.2 v2:172.21.15.179:3301/0 7 ==== mon_command_ack([{prefix=config set, name=osd_op_complaint_time}]=0 v22)=0 v22) v1 ==== 121+0+0 (secure 0 0 0) 0x7f8e980570c0 con 0x7f8ea80c8e50 2023-10-19T12:43:54.056 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.053+0000 7f8e937fe640 1 -- 172.21.15.57:0/3557275618 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8ea80c8e50 msgr2=0x7f8ea815f5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:54.056 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.053+0000 7f8e937fe640 1 --2- 172.21.15.57:0/3557275618 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8ea80c8e50 0x7f8ea815f5d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8e98037040 tx=0x7f8e98003960 comp rx=0 tx=0).stop 2023-10-19T12:43:54.056 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.053+0000 7f8e937fe640 1 -- 172.21.15.57:0/3557275618 shutdown_connections 2023-10-19T12:43:54.056 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.053+0000 7f8e937fe640 1 --2- 172.21.15.57:0/3557275618 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ea815d420 0x7f8ea8163090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:54.056 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.053+0000 7f8e937fe640 1 --2- 172.21.15.57:0/3557275618 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8ea80c8e50 0x7f8ea815f5d0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:54.057 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.053+0000 7f8e937fe640 1 --2- 172.21.15.57:0/3557275618 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8ea806ef40 0x7f8ea81691a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:54.057 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.053+0000 7f8e937fe640 1 -- 172.21.15.57:0/3557275618 >> 172.21.15.57:0/3557275618 conn(0x7f8ea80ae5d0 msgr2=0x7f8ea806fee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:43:54.057 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.053+0000 7f8e937fe640 1 -- 172.21.15.57:0/3557275618 shutdown_connections 2023-10-19T12:43:54.057 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.053+0000 7f8e937fe640 1 -- 172.21.15.57:0/3557275618 wait complete. 2023-10-19T12:43:54.068 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'osd_op_complaint_time', '180'], check_status=True, hostname='smithi057') 2023-10-19T12:43:54.069 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'rados_mon_op_timeout', '900'], check_status=True, hostname='smithi057') 2023-10-19T12:43:54.069 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'rados_osd_op_timeout', '900'], check_status=True, hostname='smithi057') 2023-10-19T12:43:54.069 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mon', 'mon_op_complaint_time', '120'], check_status=True, hostname='smithi057') 2023-10-19T12:43:54.070 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'osd', 'osd_op_complaint_time', '180'], check_status=True, hostname='smithi057') 2023-10-19T12:43:54.072 INFO:tasks.ceph:Setting crush tunables to default 2023-10-19T12:43:54.072 DEBUG:teuthology.orchestra.run.smithi057:> sudo ceph --cluster ceph osd crush tunables default 2023-10-19T12:43:54.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.218+0000 7f83cc2bb640 1 -- 172.21.15.57:0/3672172054 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f83c4142800 msgr2=0x7f83c40ca410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:54.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.218+0000 7f83cc2bb640 1 --2- 172.21.15.57:0/3672172054 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f83c4142800 0x7f83c40ca410 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f83c406bfd0 tx=0x7f83b402bd70 comp rx=0 tx=0).stop 2023-10-19T12:43:54.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.218+0000 7f83cc2bb640 1 -- 172.21.15.57:0/3672172054 shutdown_connections 2023-10-19T12:43:54.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.218+0000 7f83cc2bb640 1 --2- 172.21.15.57:0/3672172054 >> v2:172.21.15.179:3300/0 conn(0x7f83c4069b80 0x7f83c406ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:54.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.218+0000 7f83cc2bb640 1 --2- 172.21.15.57:0/3672172054 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f83c4142800 0x7f83c40ca410 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:54.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.218+0000 7f83cc2bb640 1 -- 172.21.15.57:0/3672172054 >> 172.21.15.57:0/3672172054 conn(0x7f83c40cda00 msgr2=0x7f83c40cdde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:43:54.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.218+0000 7f83cc2bb640 1 -- 172.21.15.57:0/3672172054 shutdown_connections 2023-10-19T12:43:54.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.218+0000 7f83cc2bb640 1 -- 172.21.15.57:0/3672172054 wait complete. 2023-10-19T12:43:54.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.219+0000 7f83cc2bb640 1 Processor -- start 2023-10-19T12:43:54.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.219+0000 7f83cc2bb640 1 -- start start 2023-10-19T12:43:54.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.220+0000 7f83cc2bb640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f83c4069720 0x7f83c4158820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:43:54.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.220+0000 7f83cc2bb640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f83c4069b80 0x7f83c413ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:43:54.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.220+0000 7f83cc2bb640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f83c4142800 0x7f83c4141120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:43:54.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.220+0000 7f83cc2bb640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83c4140740 con 0x7f83c4069720 2023-10-19T12:43:54.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.220+0000 7f83cc2bb640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83c413f6a0 con 0x7f83c4142800 2023-10-19T12:43:54.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.220+0000 7f83cc2bb640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83c413f980 con 0x7f83c4069b80 2023-10-19T12:43:54.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.220+0000 7f83ca057640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f83c4069720 0x7f83c4158820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:43:54.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.220+0000 7f83ca057640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f83c4069720 0x7f83c4158820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51072/0 (socket says 172.21.15.57:51072) 2023-10-19T12:43:54.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.220+0000 7f83ca057640 1 -- 172.21.15.57:0/1994114236 learned_addr learned my addr 172.21.15.57:0/1994114236 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:43:54.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.220+0000 7f83ca858640 1 --2- 172.21.15.57:0/1994114236 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f83c4142800 0x7f83c4141120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:43:54.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.220+0000 7f83c9856640 1 --2- 172.21.15.57:0/1994114236 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f83c4069b80 0x7f83c413ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:43:54.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.220+0000 7f83ca057640 1 -- 172.21.15.57:0/1994114236 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f83c4069b80 msgr2=0x7f83c413ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:54.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.220+0000 7f83ca057640 1 --2- 172.21.15.57:0/1994114236 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f83c4069b80 0x7f83c413ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:54.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.220+0000 7f83ca057640 1 -- 172.21.15.57:0/1994114236 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f83c4142800 msgr2=0x7f83c4141120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:54.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.220+0000 7f83ca057640 1 --2- 172.21.15.57:0/1994114236 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f83c4142800 0x7f83c4141120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:54.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.220+0000 7f83ca057640 1 -- 172.21.15.57:0/1994114236 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83b4009490 con 0x7f83c4069720 2023-10-19T12:43:54.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.220+0000 7f83ca057640 1 --2- 172.21.15.57:0/1994114236 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f83c4069720 0x7f83c4158820 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f83b4000c00 tx=0x7f83b40038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:43:54.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.221+0000 7f83baffd640 1 -- 172.21.15.57:0/1994114236 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83b4003ea0 con 0x7f83c4069720 2023-10-19T12:43:54.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.221+0000 7f83baffd640 1 -- 172.21.15.57:0/1994114236 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f83b4034440 con 0x7f83c4069720 2023-10-19T12:43:54.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.221+0000 7f83cc2bb640 1 -- 172.21.15.57:0/1994114236 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83c413fc60 con 0x7f83c4069720 2023-10-19T12:43:54.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.221+0000 7f83baffd640 1 -- 172.21.15.57:0/1994114236 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83b4034850 con 0x7f83c4069720 2023-10-19T12:43:54.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.221+0000 7f83cc2bb640 1 -- 172.21.15.57:0/1994114236 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83c4140220 con 0x7f83c4069720 2023-10-19T12:43:54.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.221+0000 7f83baffd640 1 -- 172.21.15.57:0/1994114236 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f83b403b040 con 0x7f83c4069720 2023-10-19T12:43:54.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.221+0000 7f83baffd640 1 -- 172.21.15.57:0/1994114236 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f83b403da00 con 0x7f83c4069720 2023-10-19T12:43:54.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.222+0000 7f83cc2bb640 1 -- 172.21.15.57:0/1994114236 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83c40675f0 con 0x7f83c4069720 2023-10-19T12:43:54.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.224+0000 7f83baffd640 1 -- 172.21.15.57:0/1994114236 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f83b4041020 con 0x7f83c4069720 2023-10-19T12:43:54.246 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:43:54.243+0000 7fdc67c3a040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-19T12:43:54.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.289+0000 7f83cc2bb640 1 -- 172.21.15.57:0/1994114236 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f83c40710d0 con 0x7f83c4069720 2023-10-19T12:43:54.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.291+0000 7f83baffd640 1 -- 172.21.15.57:0/1994114236 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f83b4039070 con 0x7f83c4069720 2023-10-19T12:43:54.293 INFO:teuthology.orchestra.run.smithi057.stderr:adjusted tunables profile to default 2023-10-19T12:43:54.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.292+0000 7f83cc2bb640 1 -- 172.21.15.57:0/1994114236 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f83c4069720 msgr2=0x7f83c4158820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:54.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.292+0000 7f83cc2bb640 1 --2- 172.21.15.57:0/1994114236 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f83c4069720 0x7f83c4158820 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f83b4000c00 tx=0x7f83b40038a0 comp rx=0 tx=0).stop 2023-10-19T12:43:54.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.292+0000 7f83cc2bb640 1 -- 172.21.15.57:0/1994114236 shutdown_connections 2023-10-19T12:43:54.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.292+0000 7f83cc2bb640 1 --2- 172.21.15.57:0/1994114236 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f83c4142800 0x7f83c4141120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:54.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.292+0000 7f83cc2bb640 1 --2- 172.21.15.57:0/1994114236 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f83c4069b80 0x7f83c413ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:54.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.292+0000 7f83cc2bb640 1 --2- 172.21.15.57:0/1994114236 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f83c4069720 0x7f83c4158820 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:54.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.292+0000 7f83cc2bb640 1 -- 172.21.15.57:0/1994114236 >> 172.21.15.57:0/1994114236 conn(0x7f83c40cda00 msgr2=0x7f83c406d4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:43:54.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.292+0000 7f83cc2bb640 1 -- 172.21.15.57:0/1994114236 shutdown_connections 2023-10-19T12:43:54.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:54.292+0000 7f83cc2bb640 1 -- 172.21.15.57:0/1994114236 wait complete. 2023-10-19T12:43:54.297 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:43:54.294+0000 7fdc67c3a040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-19T12:43:54.304 INFO:tasks.ceph:check_enable_crimson: False 2023-10-19T12:43:54.305 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-10-19T12:43:54.305 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T12:43:54.305 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-10-19T12:43:54.315 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:43:54.313+0000 7fc7ad00f040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-19T12:43:54.335 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T12:43:54.336 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-10-19T12:43:54.341 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:43:54.338+0000 7fdc67c3a040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-19T12:43:54.370 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:43:54.368+0000 7fc7ad00f040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-19T12:43:54.394 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:43:54.391+0000 7fdc67c3a040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-19T12:43:54.400 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T12:43:54.401 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-10-19T12:43:54.436 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:43:54.432+0000 7fdc67c3a040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-19T12:43:54.471 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T12:43:54.471 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-10-19T12:43:54.535 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:43:54.532+0000 7fdc67c3a040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-19T12:43:54.541 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-19T12:43:54.542 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-10-19T12:43:54.569 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-19T12:43:54.569 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-10-19T12:43:54.608 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:43:54.604+0000 7fdc67c3a040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-19T12:43:54.634 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-19T12:43:54.635 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-10-19T12:43:54.655 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:43:54.652+0000 7fdc67c3a040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-19T12:43:54.682 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-19T12:43:54.682 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-10-19T12:43:54.710 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:43:54.707+0000 7fdc67c3a040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-19T12:43:54.736 DEBUG:teuthology.orchestra.run.smithi179:> sudo ceph --cluster ceph osd new c0b0bea8-67a6-4909-9801-a10b4d135776 0 2023-10-19T12:43:54.759 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:43:54.756+0000 7fdc67c3a040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-19T12:43:54.860 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:54.857+0000 7fefc7632640 1 -- 172.21.15.179:0/2609530987 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fefc0159060 msgr2=0x7fefc006d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:54.861 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:54.857+0000 7fefc7632640 1 --2- 172.21.15.179:0/2609530987 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fefc0159060 0x7fefc006d260 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fefc00714a0 tx=0x7fefbc02bd10 comp rx=0 tx=0).stop 2023-10-19T12:43:54.861 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:54.857+0000 7fefc7632640 1 -- 172.21.15.179:0/2609530987 shutdown_connections 2023-10-19T12:43:54.862 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:54.857+0000 7fefc7632640 1 --2- 172.21.15.179:0/2609530987 >> v2:172.21.15.57:3300/0 conn(0x7fefc015ccf0 0x7fefc015d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:54.862 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:54.857+0000 7fefc7632640 1 --2- 172.21.15.179:0/2609530987 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fefc00cdb30 0x7fefc00cdfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:54.862 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:54.857+0000 7fefc7632640 1 --2- 172.21.15.179:0/2609530987 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fefc0159060 0x7fefc006d260 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:54.862 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:54.857+0000 7fefc7632640 1 -- 172.21.15.179:0/2609530987 >> 172.21.15.179:0/2609530987 conn(0x7fefc00b0670 msgr2=0x7fefc00b0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:43:54.862 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:54.857+0000 7fefc7632640 1 -- 172.21.15.179:0/2609530987 shutdown_connections 2023-10-19T12:43:54.863 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:54.857+0000 7fefc7632640 1 -- 172.21.15.179:0/2609530987 wait complete. 2023-10-19T12:43:54.865 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:54.858+0000 7fefc7632640 1 Processor -- start 2023-10-19T12:43:54.865 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:54.858+0000 7fefc7632640 1 -- start start 2023-10-19T12:43:54.865 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:54.859+0000 7fefc7632640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fefc00cdb30 0x7fefc015ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:43:54.866 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:54.859+0000 7fefc7632640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fefc0159060 0x7fefc00c4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:43:54.866 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:54.859+0000 7fefc7632640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fefc015ccf0 0x7fefc014ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:43:54.866 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:54.859+0000 7fefc7632640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefc015c0e0 con 0x7fefc0159060 2023-10-19T12:43:54.867 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:54.859+0000 7fefc7632640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefc00c54b0 con 0x7fefc015ccf0 2023-10-19T12:43:54.867 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:54.859+0000 7fefc7632640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefc015b460 con 0x7fefc00cdb30 2023-10-19T12:43:54.867 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:54.859+0000 7fefc53ce640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fefc00cdb30 0x7fefc015ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:43:54.867 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:54.859+0000 7fefc5bcf640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fefc015ccf0 0x7fefc014ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:43:54.867 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:54.859+0000 7fefc53ce640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fefc00cdb30 0x7fefc015ad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3301/0 says I am v2:172.21.15.179:44390/0 (socket says 172.21.15.179:44390) 2023-10-19T12:43:54.868 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:54.859+0000 7fefc53ce640 1 -- 172.21.15.179:0/4173080641 learned_addr learned my addr 172.21.15.179:0/4173080641 (peer_addr_for_me v2:172.21.15.179:0/0) 2023-10-19T12:43:54.868 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:54.859+0000 7fefc4bcd640 1 --2- 172.21.15.179:0/4173080641 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fefc0159060 0x7fefc00c4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:43:54.868 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:54.859+0000 7fefc53ce640 1 -- 172.21.15.179:0/4173080641 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fefc015ccf0 msgr2=0x7fefc014ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:54.868 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:54.859+0000 7fefc53ce640 1 --2- 172.21.15.179:0/4173080641 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fefc015ccf0 0x7fefc014ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:54.868 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:54.859+0000 7fefc53ce640 1 -- 172.21.15.179:0/4173080641 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fefc0159060 msgr2=0x7fefc00c4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:54.869 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:54.859+0000 7fefc53ce640 1 --2- 172.21.15.179:0/4173080641 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fefc0159060 0x7fefc00c4f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:54.869 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:54.859+0000 7fefc53ce640 1 -- 172.21.15.179:0/4173080641 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefbc009560 con 0x7fefc00cdb30 2023-10-19T12:43:54.869 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:54.859+0000 7fefc5bcf640 1 --2- 172.21.15.179:0/4173080641 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fefc015ccf0 0x7fefc014ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T12:43:54.869 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:54.859+0000 7fefc53ce640 1 --2- 172.21.15.179:0/4173080641 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fefc00cdb30 0x7fefc015ad60 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fefbc036040 tx=0x7fefbc02d9e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:43:54.870 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:54.860+0000 7fefb1ffb640 1 -- 172.21.15.179:0/4173080641 <== mon.2 v2:172.21.15.179:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefbc02dde0 con 0x7fefc00cdb30 2023-10-19T12:43:54.870 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:54.860+0000 7fefb1ffb640 1 -- 172.21.15.179:0/4173080641 <== mon.2 v2:172.21.15.179:3301/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fefbc0036e0 con 0x7fefc00cdb30 2023-10-19T12:43:54.870 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:54.860+0000 7fefc7632640 1 -- 172.21.15.179:0/4173080641 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefc015b740 con 0x7fefc00cdb30 2023-10-19T12:43:54.870 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:54.860+0000 7fefb1ffb640 1 -- 172.21.15.179:0/4173080641 <== mon.2 v2:172.21.15.179:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefbc003ab0 con 0x7fefc00cdb30 2023-10-19T12:43:54.871 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:54.860+0000 7fefc7632640 1 -- 172.21.15.179:0/4173080641 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefc015bd00 con 0x7fefc00cdb30 2023-10-19T12:43:54.871 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:54.860+0000 7fefb1ffb640 1 -- 172.21.15.179:0/4173080641 <== mon.2 v2:172.21.15.179:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fefbc03e450 con 0x7fefc00cdb30 2023-10-19T12:43:54.871 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:54.860+0000 7fefb1ffb640 1 -- 172.21.15.179:0/4173080641 <== mon.2 v2:172.21.15.179:3301/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7fefbc034440 con 0x7fefc00cdb30 2023-10-19T12:43:54.871 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:54.861+0000 7fefc7632640 1 -- 172.21.15.179:0/4173080641 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefc00675f0 con 0x7fefc00cdb30 2023-10-19T12:43:54.872 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:54.864+0000 7fefb1ffb640 1 -- 172.21.15.179:0/4173080641 <== mon.2 v2:172.21.15.179:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7fefbc042020 con 0x7fefc00cdb30 2023-10-19T12:43:54.927 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:43:54.924+0000 7fc7ad00f040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-19T12:43:54.939 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:54.936+0000 7fefc7632640 1 -- 172.21.15.179:0/4173080641 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_command({"prefix": "osd new", "uuid": "c0b0bea8-67a6-4909-9801-a10b4d135776", "id": 0} v 0) v1 -- 0x7fefc00c5970 con 0x7fefc00cdb30 2023-10-19T12:43:54.944 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:54.941+0000 7fefb1ffb640 1 -- 172.21.15.179:0/4173080641 <== mon.2 v2:172.21.15.179:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c0b0bea8-67a6-4909-9801-a10b4d135776", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7fefbc03e9b0 con 0x7fefc00cdb30 2023-10-19T12:43:54.944 INFO:teuthology.orchestra.run.smithi179.stdout:0 2023-10-19T12:43:54.945 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:54.942+0000 7fefc7632640 1 -- 172.21.15.179:0/4173080641 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fefc00cdb30 msgr2=0x7fefc015ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:54.945 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:54.942+0000 7fefc7632640 1 --2- 172.21.15.179:0/4173080641 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fefc00cdb30 0x7fefc015ad60 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fefbc036040 tx=0x7fefbc02d9e0 comp rx=0 tx=0).stop 2023-10-19T12:43:54.946 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:54.942+0000 7fefc7632640 1 -- 172.21.15.179:0/4173080641 shutdown_connections 2023-10-19T12:43:54.946 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:54.942+0000 7fefc7632640 1 --2- 172.21.15.179:0/4173080641 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fefc015ccf0 0x7fefc014ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:54.946 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:54.942+0000 7fefc7632640 1 --2- 172.21.15.179:0/4173080641 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fefc0159060 0x7fefc00c4f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:54.946 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:54.942+0000 7fefc7632640 1 --2- 172.21.15.179:0/4173080641 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fefc00cdb30 0x7fefc015ad60 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:54.947 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:54.942+0000 7fefc7632640 1 -- 172.21.15.179:0/4173080641 >> 172.21.15.179:0/4173080641 conn(0x7fefc00b0670 msgr2=0x7fefc00ca310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:43:54.947 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:54.942+0000 7fefc7632640 1 -- 172.21.15.179:0/4173080641 shutdown_connections 2023-10-19T12:43:54.947 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:54.942+0000 7fefc7632640 1 -- 172.21.15.179:0/4173080641 wait complete. 2023-10-19T12:43:54.955 DEBUG:teuthology.orchestra.run.smithi179:> sudo ceph --cluster ceph osd new c8734f82-652c-4579-bf85-443fad32f442 1 2023-10-19T12:43:54.982 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:43:54.979+0000 7fc7ad00f040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-19T12:43:55.028 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:43:55.026+0000 7fc7ad00f040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-19T12:43:55.058 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.055+0000 7f751c7a9640 1 -- 172.21.15.179:0/3913345601 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7500003b00 msgr2=0x7f7500003f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:55.058 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.055+0000 7f751c7a9640 1 --2- 172.21.15.179:0/3913345601 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7500003b00 0x7f7500003f00 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f7510001800 tx=0x7f75000057f0 comp rx=0 tx=0).stop 2023-10-19T12:43:55.059 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.055+0000 7f751c7a9640 1 -- 172.21.15.179:0/3913345601 shutdown_connections 2023-10-19T12:43:55.059 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.055+0000 7f751c7a9640 1 --2- 172.21.15.179:0/3913345601 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7500003b00 0x7f7500003f00 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:55.060 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.055+0000 7f751c7a9640 1 -- 172.21.15.179:0/3913345601 >> 172.21.15.179:0/3913345601 conn(0x7f75140af730 msgr2=0x7f75140afb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:43:55.060 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.055+0000 7f751c7a9640 1 -- 172.21.15.179:0/3913345601 shutdown_connections 2023-10-19T12:43:55.060 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.055+0000 7f751c7a9640 1 -- 172.21.15.179:0/3913345601 wait complete. 2023-10-19T12:43:55.060 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.056+0000 7f751c7a9640 1 Processor -- start 2023-10-19T12:43:55.060 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.056+0000 7f751c7a9640 1 -- start start 2023-10-19T12:43:55.063 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.056+0000 7f751c7a9640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7500003b00 0x7f751413ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:43:55.063 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.056+0000 7f751c7a9640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7514157490 0x7f75141eccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:43:55.063 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.057+0000 7f751c7a9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f751415bb30 0x7f751413f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:43:55.063 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.057+0000 7f751c7a9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f751413a700 con 0x7f751415bb30 2023-10-19T12:43:55.063 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.057+0000 7f751c7a9640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75141ed5b0 con 0x7f7514157490 2023-10-19T12:43:55.064 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.057+0000 7f751c7a9640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75141ed890 con 0x7f7500003b00 2023-10-19T12:43:55.064 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.057+0000 7f7519d44640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7514157490 0x7f75141eccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:43:55.064 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.057+0000 7f751a545640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7500003b00 0x7f751413ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:43:55.064 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.057+0000 7f7519d44640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7514157490 0x7f75141eccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.179:55674/0 (socket says 172.21.15.179:55674) 2023-10-19T12:43:55.064 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.057+0000 7f751a545640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7500003b00 0x7f751413ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3301/0 says I am v2:172.21.15.179:44404/0 (socket says 172.21.15.179:44404) 2023-10-19T12:43:55.065 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.057+0000 7f7519d44640 1 -- 172.21.15.179:0/939105296 learned_addr learned my addr 172.21.15.179:0/939105296 (peer_addr_for_me v2:172.21.15.179:0/0) 2023-10-19T12:43:55.065 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.057+0000 7f751ad46640 1 --2- 172.21.15.179:0/939105296 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f751415bb30 0x7f751413f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:43:55.065 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.057+0000 7f7519d44640 1 -- 172.21.15.179:0/939105296 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7500003b00 msgr2=0x7f751413ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:55.065 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.057+0000 7f7519d44640 1 --2- 172.21.15.179:0/939105296 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7500003b00 0x7f751413ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:55.066 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.057+0000 7f7519d44640 1 -- 172.21.15.179:0/939105296 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f751415bb30 msgr2=0x7f751413f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:55.066 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.057+0000 7f7519d44640 1 --2- 172.21.15.179:0/939105296 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f751415bb30 0x7f751413f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:55.066 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.057+0000 7f7519d44640 1 -- 172.21.15.179:0/939105296 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f751413a390 con 0x7f7514157490 2023-10-19T12:43:55.068 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.057+0000 7f751a545640 1 --2- 172.21.15.179:0/939105296 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7500003b00 0x7f751413ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T12:43:55.068 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.057+0000 7f7519d44640 1 --2- 172.21.15.179:0/939105296 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7514157490 0x7f75141eccf0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f75080066b0 tx=0x7f75080039c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:43:55.068 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.057+0000 7f7506ffd640 1 -- 172.21.15.179:0/939105296 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7508011900 con 0x7f7514157490 2023-10-19T12:43:55.068 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.057+0000 7f7506ffd640 1 -- 172.21.15.179:0/939105296 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f7508011a80 con 0x7f7514157490 2023-10-19T12:43:55.069 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.057+0000 7f7506ffd640 1 -- 172.21.15.179:0/939105296 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7508011e90 con 0x7f7514157490 2023-10-19T12:43:55.069 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.057+0000 7f751c7a9640 1 -- 172.21.15.179:0/939105296 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75141edb70 con 0x7f7514157490 2023-10-19T12:43:55.069 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.057+0000 7f751c7a9640 1 -- 172.21.15.179:0/939105296 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75141ed230 con 0x7f7514157490 2023-10-19T12:43:55.069 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.057+0000 7f7506ffd640 1 -- 172.21.15.179:0/939105296 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f7508010420 con 0x7f7514157490 2023-10-19T12:43:55.069 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.057+0000 7f7506ffd640 1 -- 172.21.15.179:0/939105296 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7f75080239a0 con 0x7f7514157490 2023-10-19T12:43:55.069 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.058+0000 7f751c7a9640 1 -- 172.21.15.179:0/939105296 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f751413a870 con 0x7f7514157490 2023-10-19T12:43:55.070 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.060+0000 7f7506ffd640 1 -- 172.21.15.179:0/939105296 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f750801f060 con 0x7f7514157490 2023-10-19T12:43:55.084 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:43:55.082+0000 7fc7ad00f040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-19T12:43:55.129 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:43:55.127+0000 7fc7ad00f040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-19T12:43:55.132 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.129+0000 7f751c7a9640 1 -- 172.21.15.179:0/939105296 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "osd new", "uuid": "c8734f82-652c-4579-bf85-443fad32f442", "id": 1} v 0) v1 -- 0x7f75140675f0 con 0x7f7514157490 2023-10-19T12:43:55.137 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.134+0000 7f7506ffd640 1 -- 172.21.15.179:0/939105296 <== mon.1 v2:172.21.15.179:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c8734f82-652c-4579-bf85-443fad32f442", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f7508015070 con 0x7f7514157490 2023-10-19T12:43:55.137 INFO:teuthology.orchestra.run.smithi179.stdout:1 2023-10-19T12:43:55.138 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.135+0000 7f751c7a9640 1 -- 172.21.15.179:0/939105296 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7514157490 msgr2=0x7f75141eccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:55.139 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.135+0000 7f751c7a9640 1 --2- 172.21.15.179:0/939105296 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7514157490 0x7f75141eccf0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f75080066b0 tx=0x7f75080039c0 comp rx=0 tx=0).stop 2023-10-19T12:43:55.139 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.135+0000 7f751c7a9640 1 -- 172.21.15.179:0/939105296 shutdown_connections 2023-10-19T12:43:55.139 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.135+0000 7f751c7a9640 1 --2- 172.21.15.179:0/939105296 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f751415bb30 0x7f751413f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:55.139 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.135+0000 7f751c7a9640 1 --2- 172.21.15.179:0/939105296 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7514157490 0x7f75141eccf0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:55.139 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.135+0000 7f751c7a9640 1 --2- 172.21.15.179:0/939105296 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7500003b00 0x7f751413ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:55.140 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.135+0000 7f751c7a9640 1 -- 172.21.15.179:0/939105296 >> 172.21.15.179:0/939105296 conn(0x7f75140af730 msgr2=0x7f751413d3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:43:55.140 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.135+0000 7f751c7a9640 1 -- 172.21.15.179:0/939105296 shutdown_connections 2023-10-19T12:43:55.140 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.135+0000 7f751c7a9640 1 -- 172.21.15.179:0/939105296 wait complete. 2023-10-19T12:43:55.149 DEBUG:teuthology.orchestra.run.smithi179:> sudo ceph --cluster ceph osd new e9ff64fe-d989-42fe-97ee-1aed1b5d98d1 2 2023-10-19T12:43:55.155 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:43:55.152+0000 7fdc67c3a040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-19T12:43:55.234 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:43:55.232+0000 7fc7ad00f040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-19T12:43:55.253 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.249+0000 7f06a3032640 1 -- 172.21.15.179:0/2085217451 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f069c15d400 msgr2=0x7f069c0cdfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:55.253 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.249+0000 7f06a3032640 1 --2- 172.21.15.179:0/2085217451 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f069c15d400 0x7f069c0cdfe0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f0694009ca0 tx=0x7f069402c230 comp rx=0 tx=0).stop 2023-10-19T12:43:55.253 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.249+0000 7f06a3032640 1 -- 172.21.15.179:0/2085217451 shutdown_connections 2023-10-19T12:43:55.254 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.249+0000 7f06a3032640 1 --2- 172.21.15.179:0/2085217451 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f069c15d400 0x7f069c0cdfe0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:55.254 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.249+0000 7f06a3032640 1 --2- 172.21.15.179:0/2085217451 >> v2:172.21.15.57:3300/0 conn(0x7f069c159060 0x7f069c06d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:55.254 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.249+0000 7f06a3032640 1 -- 172.21.15.179:0/2085217451 >> 172.21.15.179:0/2085217451 conn(0x7f069c0b0670 msgr2=0x7f069c0b0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:43:55.254 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.249+0000 7f06a3032640 1 -- 172.21.15.179:0/2085217451 shutdown_connections 2023-10-19T12:43:55.255 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.249+0000 7f06a3032640 1 -- 172.21.15.179:0/2085217451 wait complete. 2023-10-19T12:43:55.257 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.250+0000 7f06a3032640 1 Processor -- start 2023-10-19T12:43:55.258 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.250+0000 7f06a3032640 1 -- start start 2023-10-19T12:43:55.258 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.251+0000 7f06a3032640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f069c0cdb30 0x7f069c1507b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:43:55.258 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.251+0000 7f06a3032640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f069c159060 0x7f069c0b1f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:43:55.258 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.251+0000 7f06a3032640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f069c15d400 0x7f069c0b3b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:43:55.259 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.251+0000 7f06a3032640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f069c150cf0 con 0x7f069c15d400 2023-10-19T12:43:55.259 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.251+0000 7f06a3032640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f069c0c2fc0 con 0x7f069c159060 2023-10-19T12:43:55.259 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.251+0000 7f06a3032640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f069c0c3210 con 0x7f069c0cdb30 2023-10-19T12:43:55.259 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.251+0000 7f06a0dce640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f069c0cdb30 0x7f069c1507b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:43:55.259 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.251+0000 7f069bfff640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f069c159060 0x7f069c0b1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:43:55.260 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.251+0000 7f06a0dce640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f069c0cdb30 0x7f069c1507b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3301/0 says I am v2:172.21.15.179:44422/0 (socket says 172.21.15.179:44422) 2023-10-19T12:43:55.260 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.251+0000 7f06a0dce640 1 -- 172.21.15.179:0/3431469217 learned_addr learned my addr 172.21.15.179:0/3431469217 (peer_addr_for_me v2:172.21.15.179:0/0) 2023-10-19T12:43:55.260 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.251+0000 7f06a15cf640 1 --2- 172.21.15.179:0/3431469217 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f069c15d400 0x7f069c0b3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:43:55.260 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.252+0000 7f069bfff640 1 -- 172.21.15.179:0/3431469217 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f069c0cdb30 msgr2=0x7f069c1507b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:55.260 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.252+0000 7f069bfff640 1 --2- 172.21.15.179:0/3431469217 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f069c0cdb30 0x7f069c1507b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:55.261 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.252+0000 7f069bfff640 1 -- 172.21.15.179:0/3431469217 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f069c15d400 msgr2=0x7f069c0b3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:55.261 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.252+0000 7f069bfff640 1 --2- 172.21.15.179:0/3431469217 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f069c15d400 0x7f069c0b3b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:55.261 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.252+0000 7f069bfff640 1 -- 172.21.15.179:0/3431469217 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0694009900 con 0x7f069c159060 2023-10-19T12:43:55.261 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.252+0000 7f06a0dce640 1 --2- 172.21.15.179:0/3431469217 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f069c0cdb30 0x7f069c1507b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T12:43:55.261 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.252+0000 7f069bfff640 1 --2- 172.21.15.179:0/3431469217 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f069c159060 0x7f069c0b1f50 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f068800c3c0 tx=0x7f068800c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:43:55.262 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.252+0000 7f06997fa640 1 -- 172.21.15.179:0/3431469217 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06880052d0 con 0x7f069c159060 2023-10-19T12:43:55.262 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.252+0000 7f06997fa640 1 -- 172.21.15.179:0/3431469217 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f06880035f0 con 0x7f069c159060 2023-10-19T12:43:55.262 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.252+0000 7f06997fa640 1 -- 172.21.15.179:0/3431469217 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0688003a00 con 0x7f069c159060 2023-10-19T12:43:55.262 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.252+0000 7f06a3032640 1 -- 172.21.15.179:0/3431469217 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f069c0c34f0 con 0x7f069c159060 2023-10-19T12:43:55.262 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.252+0000 7f06a3032640 1 -- 172.21.15.179:0/3431469217 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f069c0c3ab0 con 0x7f069c159060 2023-10-19T12:43:55.263 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.252+0000 7f06997fa640 1 -- 172.21.15.179:0/3431469217 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f0688011590 con 0x7f069c159060 2023-10-19T12:43:55.263 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.252+0000 7f06997fa640 1 -- 172.21.15.179:0/3431469217 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7f0688012840 con 0x7f069c159060 2023-10-19T12:43:55.263 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.253+0000 7f06a3032640 1 -- 172.21.15.179:0/3431469217 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f069c0cdfa0 con 0x7f069c159060 2023-10-19T12:43:55.264 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.255+0000 7f06997fa640 1 -- 172.21.15.179:0/3431469217 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f0688017070 con 0x7f069c159060 2023-10-19T12:43:55.296 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:43:55.293+0000 7fdc67c3a040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-19T12:43:55.308 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:43:55.306+0000 7fc7ad00f040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-19T12:43:55.329 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.326+0000 7f06a3032640 1 -- 172.21.15.179:0/3431469217 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "osd new", "uuid": "e9ff64fe-d989-42fe-97ee-1aed1b5d98d1", "id": 2} v 0) v1 -- 0x7f069c0675f0 con 0x7f069c159060 2023-10-19T12:43:55.333 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.330+0000 7f06997fa640 1 -- 172.21.15.179:0/3431469217 <== mon.1 v2:172.21.15.179:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e9ff64fe-d989-42fe-97ee-1aed1b5d98d1", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f0688012420 con 0x7f069c159060 2023-10-19T12:43:55.333 INFO:teuthology.orchestra.run.smithi179.stdout:2 2023-10-19T12:43:55.334 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.331+0000 7f06a3032640 1 -- 172.21.15.179:0/3431469217 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f069c159060 msgr2=0x7f069c0b1f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:55.335 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.331+0000 7f06a3032640 1 --2- 172.21.15.179:0/3431469217 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f069c159060 0x7f069c0b1f50 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f068800c3c0 tx=0x7f068800c880 comp rx=0 tx=0).stop 2023-10-19T12:43:55.335 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.331+0000 7f06a3032640 1 -- 172.21.15.179:0/3431469217 shutdown_connections 2023-10-19T12:43:55.335 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.331+0000 7f06a3032640 1 --2- 172.21.15.179:0/3431469217 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f069c15d400 0x7f069c0b3b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:55.335 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.331+0000 7f06a3032640 1 --2- 172.21.15.179:0/3431469217 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f069c159060 0x7f069c0b1f50 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:55.336 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.331+0000 7f06a3032640 1 --2- 172.21.15.179:0/3431469217 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f069c0cdb30 0x7f069c1507b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:55.336 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.331+0000 7f06a3032640 1 -- 172.21.15.179:0/3431469217 >> 172.21.15.179:0/3431469217 conn(0x7f069c0b0670 msgr2=0x7f069c0af450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:43:55.336 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.331+0000 7f06a3032640 1 -- 172.21.15.179:0/3431469217 shutdown_connections 2023-10-19T12:43:55.336 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.331+0000 7f06a3032640 1 -- 172.21.15.179:0/3431469217 wait complete. 2023-10-19T12:43:55.344 DEBUG:teuthology.orchestra.run.smithi179:> sudo ceph --cluster ceph osd new 19f034f4-5964-4284-a429-229d154a2130 3 2023-10-19T12:43:55.356 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:43:55.354+0000 7fc7ad00f040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-19T12:43:55.410 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:43:55.409+0000 7fc7ad00f040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-19T12:43:55.422 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:43:55.419+0000 7fdc67c3a040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-19T12:43:55.459 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:43:55.456+0000 7fc7ad00f040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-19T12:43:55.469 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:43:55.466+0000 7fdc67c3a040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-19T12:43:55.486 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.483+0000 7f9c9b807640 1 -- 172.21.15.179:0/690477879 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9c941416f0 msgr2=0x7f9c940b1440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:55.487 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.483+0000 7f9c9b807640 1 --2- 172.21.15.179:0/690477879 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9c941416f0 0x7f9c940b1440 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9c940711c0 tx=0x7f9c8402be40 comp rx=0 tx=0).stop 2023-10-19T12:43:55.487 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.483+0000 7f9c9b807640 1 -- 172.21.15.179:0/690477879 shutdown_connections 2023-10-19T12:43:55.487 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.483+0000 7f9c9b807640 1 --2- 172.21.15.179:0/690477879 >> v2:172.21.15.57:3300/0 conn(0x7f9c94070790 0x7f9c940cc200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:55.487 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.483+0000 7f9c9b807640 1 --2- 172.21.15.179:0/690477879 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9c941416f0 0x7f9c940b1440 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:55.488 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.483+0000 7f9c9b807640 1 -- 172.21.15.179:0/690477879 >> 172.21.15.179:0/690477879 conn(0x7f9c940af730 msgr2=0x7f9c940afb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:43:55.488 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.483+0000 7f9c9b807640 1 -- 172.21.15.179:0/690477879 shutdown_connections 2023-10-19T12:43:55.488 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.483+0000 7f9c9b807640 1 -- 172.21.15.179:0/690477879 wait complete. 2023-10-19T12:43:55.490 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.484+0000 7f9c9b807640 1 Processor -- start 2023-10-19T12:43:55.491 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.484+0000 7f9c9b807640 1 -- start start 2023-10-19T12:43:55.491 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.485+0000 7f9c9b807640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9c94070790 0x7f9c941548c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:43:55.491 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.485+0000 7f9c9b807640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9c940b1980 0x7f9c9415b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:43:55.491 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.485+0000 7f9c9b807640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9c941416f0 0x7f9c94138ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:43:55.492 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.485+0000 7f9c9b807640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c9413b090 con 0x7f9c941416f0 2023-10-19T12:43:55.492 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.485+0000 7f9c9b807640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c9415ba70 con 0x7f9c940b1980 2023-10-19T12:43:55.492 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.485+0000 7f9c9b807640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c9415bd50 con 0x7f9c94070790 2023-10-19T12:43:55.492 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.485+0000 7f9c98da2640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9c940b1980 0x7f9c9415b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:43:55.492 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.485+0000 7f9c995a3640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9c94070790 0x7f9c941548c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:43:55.493 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.485+0000 7f9c995a3640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9c94070790 0x7f9c941548c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3301/0 says I am v2:172.21.15.179:44438/0 (socket says 172.21.15.179:44438) 2023-10-19T12:43:55.493 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.485+0000 7f9c995a3640 1 -- 172.21.15.179:0/1496165254 learned_addr learned my addr 172.21.15.179:0/1496165254 (peer_addr_for_me v2:172.21.15.179:0/0) 2023-10-19T12:43:55.493 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.485+0000 7f9c99da4640 1 --2- 172.21.15.179:0/1496165254 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9c941416f0 0x7f9c94138ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:43:55.493 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.486+0000 7f9c98da2640 1 -- 172.21.15.179:0/1496165254 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9c94070790 msgr2=0x7f9c941548c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:55.493 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.486+0000 7f9c98da2640 1 --2- 172.21.15.179:0/1496165254 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9c94070790 0x7f9c941548c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:55.494 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.486+0000 7f9c98da2640 1 -- 172.21.15.179:0/1496165254 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9c941416f0 msgr2=0x7f9c94138ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:55.494 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.486+0000 7f9c98da2640 1 --2- 172.21.15.179:0/1496165254 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9c941416f0 0x7f9c94138ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:55.494 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.486+0000 7f9c98da2640 1 -- 172.21.15.179:0/1496165254 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c84009560 con 0x7f9c940b1980 2023-10-19T12:43:55.494 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.486+0000 7f9c995a3640 1 --2- 172.21.15.179:0/1496165254 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9c94070790 0x7f9c941548c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T12:43:55.494 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.486+0000 7f9c98da2640 1 --2- 172.21.15.179:0/1496165254 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9c940b1980 0x7f9c9415b170 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9c9000c390 tx=0x7f9c9000c850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:43:55.495 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.486+0000 7f9c89ffb640 1 -- 172.21.15.179:0/1496165254 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c900052d0 con 0x7f9c940b1980 2023-10-19T12:43:55.495 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.486+0000 7f9c89ffb640 1 -- 172.21.15.179:0/1496165254 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f9c900035f0 con 0x7f9c940b1980 2023-10-19T12:43:55.495 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.486+0000 7f9c9b807640 1 -- 172.21.15.179:0/1496165254 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c9415c030 con 0x7f9c940b1980 2023-10-19T12:43:55.495 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.486+0000 7f9c89ffb640 1 -- 172.21.15.179:0/1496165254 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c90003a00 con 0x7f9c940b1980 2023-10-19T12:43:55.495 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.486+0000 7f9c9b807640 1 -- 172.21.15.179:0/1496165254 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c9415c310 con 0x7f9c940b1980 2023-10-19T12:43:55.496 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.486+0000 7f9c89ffb640 1 -- 172.21.15.179:0/1496165254 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f9c90011590 con 0x7f9c940b1980 2023-10-19T12:43:55.496 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.486+0000 7f9c89ffb640 1 -- 172.21.15.179:0/1496165254 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7f9c900128b0 con 0x7f9c940b1980 2023-10-19T12:43:55.496 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.487+0000 7f9c9b807640 1 -- 172.21.15.179:0/1496165254 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c9413aef0 con 0x7f9c940b1980 2023-10-19T12:43:55.496 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.489+0000 7f9c89ffb640 1 -- 172.21.15.179:0/1496165254 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f9c9413aef0 con 0x7f9c940b1980 2023-10-19T12:43:55.564 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.561+0000 7f9c9b807640 1 -- 172.21.15.179:0/1496165254 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "osd new", "uuid": "19f034f4-5964-4284-a429-229d154a2130", "id": 3} v 0) v1 -- 0x7f9c940675f0 con 0x7f9c940b1980 2023-10-19T12:43:55.567 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.564+0000 7f9c89ffb640 1 -- 172.21.15.179:0/1496165254 <== mon.1 v2:172.21.15.179:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "19f034f4-5964-4284-a429-229d154a2130", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f9c90017070 con 0x7f9c940b1980 2023-10-19T12:43:55.567 INFO:teuthology.orchestra.run.smithi179.stdout:3 2023-10-19T12:43:55.568 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.565+0000 7f9c9b807640 1 -- 172.21.15.179:0/1496165254 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9c940b1980 msgr2=0x7f9c9415b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:55.569 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.565+0000 7f9c9b807640 1 --2- 172.21.15.179:0/1496165254 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9c940b1980 0x7f9c9415b170 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9c9000c390 tx=0x7f9c9000c850 comp rx=0 tx=0).stop 2023-10-19T12:43:55.569 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.565+0000 7f9c9b807640 1 -- 172.21.15.179:0/1496165254 shutdown_connections 2023-10-19T12:43:55.569 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.565+0000 7f9c9b807640 1 --2- 172.21.15.179:0/1496165254 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9c941416f0 0x7f9c94138ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:55.569 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.565+0000 7f9c9b807640 1 --2- 172.21.15.179:0/1496165254 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9c940b1980 0x7f9c9415b170 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:55.570 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.565+0000 7f9c9b807640 1 --2- 172.21.15.179:0/1496165254 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9c94070790 0x7f9c941548c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:55.570 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.565+0000 7f9c9b807640 1 -- 172.21.15.179:0/1496165254 >> 172.21.15.179:0/1496165254 conn(0x7f9c940af730 msgr2=0x7f9c94156b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:43:55.570 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.565+0000 7f9c9b807640 1 -- 172.21.15.179:0/1496165254 shutdown_connections 2023-10-19T12:43:55.570 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.565+0000 7f9c9b807640 1 -- 172.21.15.179:0/1496165254 wait complete. 2023-10-19T12:43:55.571 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:43:55.568+0000 7fdc67c3a040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-19T12:43:55.578 DEBUG:teuthology.orchestra.run.smithi179:> sudo ceph --cluster ceph osd new 082fed67-b8da-4207-bab9-27f01a29ab02 4 2023-10-19T12:43:55.670 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:43:55.667+0000 7fdc67c3a040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-19T12:43:55.722 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.718+0000 7f60d2d68640 1 -- 172.21.15.179:0/421130548 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f60cc13bd70 msgr2=0x7f60cc070ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:55.722 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.718+0000 7f60d2d68640 1 --2- 172.21.15.179:0/421130548 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f60cc13bd70 0x7f60cc070ce0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f60c0009860 tx=0x7f60c002bde0 comp rx=0 tx=0).stop 2023-10-19T12:43:55.722 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.718+0000 7f60d2d68640 1 -- 172.21.15.179:0/421130548 shutdown_connections 2023-10-19T12:43:55.722 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.718+0000 7f60d2d68640 1 --2- 172.21.15.179:0/421130548 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f60cc13bd70 0x7f60cc070ce0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:55.723 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.718+0000 7f60d2d68640 1 -- 172.21.15.179:0/421130548 >> 172.21.15.179:0/421130548 conn(0x7f60cc0af730 msgr2=0x7f60cc0afb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:43:55.723 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.718+0000 7f60d2d68640 1 -- 172.21.15.179:0/421130548 shutdown_connections 2023-10-19T12:43:55.723 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.718+0000 7f60d2d68640 1 -- 172.21.15.179:0/421130548 wait complete. 2023-10-19T12:43:55.725 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.719+0000 7f60d2d68640 1 Processor -- start 2023-10-19T12:43:55.726 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.719+0000 7f60d2d68640 1 -- start start 2023-10-19T12:43:55.726 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.720+0000 7f60d2d68640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f60cc069720 0x7f60cc146d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:43:55.726 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.720+0000 7f60d2d68640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f60cc13bd70 0x7f60cc138c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:43:55.726 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.720+0000 7f60d2d68640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f60cc13ebe0 0x7f60cc0b17b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:43:55.727 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.720+0000 7f60d2d68640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60cc13a310 con 0x7f60cc13bd70 2023-10-19T12:43:55.727 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.720+0000 7f60d2d68640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60cc13a5f0 con 0x7f60cc069720 2023-10-19T12:43:55.727 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.720+0000 7f60d2d68640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60cc139690 con 0x7f60cc13ebe0 2023-10-19T12:43:55.727 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.720+0000 7f60d1305640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f60cc13ebe0 0x7f60cc0b17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:43:55.727 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.720+0000 7f60d0b04640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f60cc069720 0x7f60cc146d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:43:55.728 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.720+0000 7f60d1305640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f60cc13ebe0 0x7f60cc0b17b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3301/0 says I am v2:172.21.15.179:44464/0 (socket says 172.21.15.179:44464) 2023-10-19T12:43:55.728 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.720+0000 7f60d1305640 1 -- 172.21.15.179:0/1828920962 learned_addr learned my addr 172.21.15.179:0/1828920962 (peer_addr_for_me v2:172.21.15.179:0/0) 2023-10-19T12:43:55.728 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.720+0000 7f60d0b04640 1 --2- 172.21.15.179:0/1828920962 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f60cc069720 0x7f60cc146d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.179:55706/0 (socket says 172.21.15.179:55706) 2023-10-19T12:43:55.728 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.720+0000 7f60cbfff640 1 --2- 172.21.15.179:0/1828920962 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f60cc13bd70 0x7f60cc138c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:43:55.728 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.720+0000 7f60d1305640 1 -- 172.21.15.179:0/1828920962 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f60cc069720 msgr2=0x7f60cc146d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:55.728 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.720+0000 7f60d1305640 1 --2- 172.21.15.179:0/1828920962 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f60cc069720 0x7f60cc146d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:55.729 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.720+0000 7f60d1305640 1 -- 172.21.15.179:0/1828920962 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f60cc13bd70 msgr2=0x7f60cc138c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:55.729 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.720+0000 7f60d1305640 1 --2- 172.21.15.179:0/1828920962 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f60cc13bd70 0x7f60cc138c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:55.729 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.720+0000 7f60d1305640 1 -- 172.21.15.179:0/1828920962 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60c0009490 con 0x7f60cc13ebe0 2023-10-19T12:43:55.729 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.721+0000 7f60d1305640 1 --2- 172.21.15.179:0/1828920962 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f60cc13ebe0 0x7f60cc0b17b0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f60c400c6e0 tx=0x7f60c400cba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:43:55.730 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.721+0000 7f60c97fa640 1 -- 172.21.15.179:0/1828920962 <== mon.2 v2:172.21.15.179:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60c4003b70 con 0x7f60cc13ebe0 2023-10-19T12:43:55.730 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.721+0000 7f60c97fa640 1 -- 172.21.15.179:0/1828920962 <== mon.2 v2:172.21.15.179:3301/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f60c4003cf0 con 0x7f60cc13ebe0 2023-10-19T12:43:55.730 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.721+0000 7f60d2d68640 1 -- 172.21.15.179:0/1828920962 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60cc139970 con 0x7f60cc13ebe0 2023-10-19T12:43:55.730 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.721+0000 7f60c97fa640 1 -- 172.21.15.179:0/1828920962 <== mon.2 v2:172.21.15.179:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60c4003ea0 con 0x7f60cc13ebe0 2023-10-19T12:43:55.730 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.721+0000 7f60d2d68640 1 -- 172.21.15.179:0/1828920962 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60cc139f30 con 0x7f60cc13ebe0 2023-10-19T12:43:55.731 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.721+0000 7f60c97fa640 1 -- 172.21.15.179:0/1828920962 <== mon.2 v2:172.21.15.179:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f60c4005630 con 0x7f60cc13ebe0 2023-10-19T12:43:55.731 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.721+0000 7f60c97fa640 1 -- 172.21.15.179:0/1828920962 <== mon.2 v2:172.21.15.179:3301/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7f60c4012930 con 0x7f60cc13ebe0 2023-10-19T12:43:55.731 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.722+0000 7f60d2d68640 1 -- 172.21.15.179:0/1828920962 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60cc13a450 con 0x7f60cc13ebe0 2023-10-19T12:43:55.732 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.724+0000 7f60c97fa640 1 -- 172.21.15.179:0/1828920962 <== mon.2 v2:172.21.15.179:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f60cc13a450 con 0x7f60cc13ebe0 2023-10-19T12:43:55.797 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.794+0000 7f60d2d68640 1 -- 172.21.15.179:0/1828920962 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_command({"prefix": "osd new", "uuid": "082fed67-b8da-4207-bab9-27f01a29ab02", "id": 4} v 0) v1 -- 0x7f60cc0675f0 con 0x7f60cc13ebe0 2023-10-19T12:43:55.801 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.798+0000 7f60c97fa640 1 -- 172.21.15.179:0/1828920962 <== mon.2 v2:172.21.15.179:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "082fed67-b8da-4207-bab9-27f01a29ab02", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f60c4012420 con 0x7f60cc13ebe0 2023-10-19T12:43:55.801 INFO:teuthology.orchestra.run.smithi179.stdout:4 2023-10-19T12:43:55.802 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.799+0000 7f60d2d68640 1 -- 172.21.15.179:0/1828920962 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f60cc13ebe0 msgr2=0x7f60cc0b17b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:55.803 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.799+0000 7f60d2d68640 1 --2- 172.21.15.179:0/1828920962 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f60cc13ebe0 0x7f60cc0b17b0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f60c400c6e0 tx=0x7f60c400cba0 comp rx=0 tx=0).stop 2023-10-19T12:43:55.803 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.799+0000 7f60d2d68640 1 -- 172.21.15.179:0/1828920962 shutdown_connections 2023-10-19T12:43:55.803 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.799+0000 7f60d2d68640 1 --2- 172.21.15.179:0/1828920962 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f60cc13ebe0 0x7f60cc0b17b0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:55.803 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.799+0000 7f60d2d68640 1 --2- 172.21.15.179:0/1828920962 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f60cc13bd70 0x7f60cc138c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:55.803 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.799+0000 7f60d2d68640 1 --2- 172.21.15.179:0/1828920962 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f60cc069720 0x7f60cc146d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:55.804 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.799+0000 7f60d2d68640 1 -- 172.21.15.179:0/1828920962 >> 172.21.15.179:0/1828920962 conn(0x7f60cc0af730 msgr2=0x7f60cc15d380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:43:55.804 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.799+0000 7f60d2d68640 1 -- 172.21.15.179:0/1828920962 shutdown_connections 2023-10-19T12:43:55.804 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.799+0000 7f60d2d68640 1 -- 172.21.15.179:0/1828920962 wait complete. 2023-10-19T12:43:55.812 DEBUG:teuthology.orchestra.run.smithi179:> sudo ceph --cluster ceph osd new 8ecee277-1594-48b1-89b6-01629c88636a 5 2023-10-19T12:43:55.815 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:43:55.812+0000 7fdc67c3a040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-19T12:43:55.858 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:43:55.855+0000 7fc7ad00f040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-19T12:43:55.860 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:43:55.857+0000 7fdc67c3a040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-19T12:43:55.915 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.911+0000 7f6bf42bb640 1 -- 172.21.15.179:0/2139909951 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6bec070080 msgr2=0x7f6bec0704e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:55.915 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.911+0000 7f6bf42bb640 1 --2- 172.21.15.179:0/2139909951 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6bec070080 0x7f6bec0704e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6bdc009860 tx=0x7f6bdc02bde0 comp rx=0 tx=0).stop 2023-10-19T12:43:55.916 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.911+0000 7f6bf42bb640 1 -- 172.21.15.179:0/2139909951 shutdown_connections 2023-10-19T12:43:55.916 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.911+0000 7f6bf42bb640 1 --2- 172.21.15.179:0/2139909951 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6bec070080 0x7f6bec0704e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:55.916 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.911+0000 7f6bf42bb640 1 -- 172.21.15.179:0/2139909951 >> 172.21.15.179:0/2139909951 conn(0x7f6bec0af730 msgr2=0x7f6bec0afb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:43:55.917 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.911+0000 7f6bf42bb640 1 -- 172.21.15.179:0/2139909951 shutdown_connections 2023-10-19T12:43:55.917 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.911+0000 7f6bf42bb640 1 -- 172.21.15.179:0/2139909951 wait complete. 2023-10-19T12:43:55.922 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.912+0000 7f6bf42bb640 1 Processor -- start 2023-10-19T12:43:55.922 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.912+0000 7f6bf42bb640 1 -- start start 2023-10-19T12:43:55.923 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.913+0000 7f6bf42bb640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6bec070080 0x7f6bec13fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:43:55.923 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.913+0000 7f6bf42bb640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6bec0cc200 0x7f6bec13b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:43:55.924 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.913+0000 7f6bf42bb640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6bec1416f0 0x7f6bec1377e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:43:55.924 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.913+0000 7f6bf42bb640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bec13b5a0 con 0x7f6bec070080 2023-10-19T12:43:55.924 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.913+0000 7f6bf42bb640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bec140290 con 0x7f6bec1416f0 2023-10-19T12:43:55.925 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.913+0000 7f6bf42bb640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bec140570 con 0x7f6bec0cc200 2023-10-19T12:43:55.925 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.913+0000 7f6bf2858640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6bec1416f0 0x7f6bec1377e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:43:55.926 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.913+0000 7f6bf1856640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6bec0cc200 0x7f6bec13b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:43:55.926 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.913+0000 7f6bf2858640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6bec1416f0 0x7f6bec1377e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.179:55710/0 (socket says 172.21.15.179:55710) 2023-10-19T12:43:55.926 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.913+0000 7f6bf2858640 1 -- 172.21.15.179:0/2806285885 learned_addr learned my addr 172.21.15.179:0/2806285885 (peer_addr_for_me v2:172.21.15.179:0/0) 2023-10-19T12:43:55.927 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.913+0000 7f6bf2057640 1 --2- 172.21.15.179:0/2806285885 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6bec070080 0x7f6bec13fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:43:55.927 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.913+0000 7f6bf2858640 1 -- 172.21.15.179:0/2806285885 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6bec0cc200 msgr2=0x7f6bec13b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:55.928 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.913+0000 7f6bf2858640 1 --2- 172.21.15.179:0/2806285885 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6bec0cc200 0x7f6bec13b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:55.928 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.913+0000 7f6bf2858640 1 -- 172.21.15.179:0/2806285885 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6bec070080 msgr2=0x7f6bec13fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:55.928 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.913+0000 7f6bf2858640 1 --2- 172.21.15.179:0/2806285885 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6bec070080 0x7f6bec13fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:55.929 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.913+0000 7f6bf2858640 1 -- 172.21.15.179:0/2806285885 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bdc009490 con 0x7f6bec1416f0 2023-10-19T12:43:55.929 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.913+0000 7f6bf1856640 1 --2- 172.21.15.179:0/2806285885 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6bec0cc200 0x7f6bec13b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T12:43:55.929 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.914+0000 7f6bf2858640 1 --2- 172.21.15.179:0/2806285885 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6bec1416f0 0x7f6bec1377e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6be400cbc0 tx=0x7f6be400a430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:43:55.930 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.914+0000 7f6be2ffd640 1 -- 172.21.15.179:0/2806285885 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6be400ad40 con 0x7f6bec1416f0 2023-10-19T12:43:55.930 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.914+0000 7f6be2ffd640 1 -- 172.21.15.179:0/2806285885 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f6be4012420 con 0x7f6bec1416f0 2023-10-19T12:43:55.931 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.914+0000 7f6bf42bb640 1 -- 172.21.15.179:0/2806285885 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bec0b1cd0 con 0x7f6bec1416f0 2023-10-19T12:43:55.931 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.914+0000 7f6be2ffd640 1 -- 172.21.15.179:0/2806285885 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6be4012850 con 0x7f6bec1416f0 2023-10-19T12:43:55.931 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.914+0000 7f6bf42bb640 1 -- 172.21.15.179:0/2806285885 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bec140b30 con 0x7f6bec1416f0 2023-10-19T12:43:55.932 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.914+0000 7f6be2ffd640 1 -- 172.21.15.179:0/2806285885 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f6be4012cb0 con 0x7f6bec1416f0 2023-10-19T12:43:55.932 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.914+0000 7f6be2ffd640 1 -- 172.21.15.179:0/2806285885 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7f6be401bdb0 con 0x7f6bec1416f0 2023-10-19T12:43:55.933 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.915+0000 7f6bf42bb640 1 -- 172.21.15.179:0/2806285885 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bec0675f0 con 0x7f6bec1416f0 2023-10-19T12:43:55.933 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.917+0000 7f6be2ffd640 1 -- 172.21.15.179:0/2806285885 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f6be4016020 con 0x7f6bec1416f0 2023-10-19T12:43:55.991 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.988+0000 7f6bf42bb640 1 -- 172.21.15.179:0/2806285885 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "osd new", "uuid": "8ecee277-1594-48b1-89b6-01629c88636a", "id": 5} v 0) v1 -- 0x7f6bec0b2150 con 0x7f6bec1416f0 2023-10-19T12:43:55.996 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.993+0000 7f6be2ffd640 1 -- 172.21.15.179:0/2806285885 <== mon.1 v2:172.21.15.179:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "8ecee277-1594-48b1-89b6-01629c88636a", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f6be401a560 con 0x7f6bec1416f0 2023-10-19T12:43:55.996 INFO:teuthology.orchestra.run.smithi179.stdout:5 2023-10-19T12:43:55.997 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.993+0000 7f6bf42bb640 1 -- 172.21.15.179:0/2806285885 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6bec1416f0 msgr2=0x7f6bec1377e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:55.997 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.993+0000 7f6bf42bb640 1 --2- 172.21.15.179:0/2806285885 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6bec1416f0 0x7f6bec1377e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6be400cbc0 tx=0x7f6be400a430 comp rx=0 tx=0).stop 2023-10-19T12:43:55.998 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.993+0000 7f6bf42bb640 1 -- 172.21.15.179:0/2806285885 shutdown_connections 2023-10-19T12:43:55.998 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.993+0000 7f6bf42bb640 1 --2- 172.21.15.179:0/2806285885 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6bec1416f0 0x7f6bec1377e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:55.998 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.993+0000 7f6bf42bb640 1 --2- 172.21.15.179:0/2806285885 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6bec0cc200 0x7f6bec13b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:55.998 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.993+0000 7f6bf42bb640 1 --2- 172.21.15.179:0/2806285885 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6bec070080 0x7f6bec13fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:55.998 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.993+0000 7f6bf42bb640 1 -- 172.21.15.179:0/2806285885 >> 172.21.15.179:0/2806285885 conn(0x7f6bec0af730 msgr2=0x7f6bec0ce2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:43:55.999 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.994+0000 7f6bf42bb640 1 -- 172.21.15.179:0/2806285885 shutdown_connections 2023-10-19T12:43:55.999 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:55.994+0000 7f6bf42bb640 1 -- 172.21.15.179:0/2806285885 wait complete. 2023-10-19T12:43:55.999 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:43:55.996+0000 7fc7ad00f040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-19T12:43:56.007 DEBUG:teuthology.orchestra.run.smithi179:> sudo ceph --cluster ceph osd new 3b7b20a2-ea0e-40ca-be09-29a46bb1a997 6 2023-10-19T12:43:56.007 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:43:56.004+0000 7fdc67c3a040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-19T12:43:56.130 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:43:56.127+0000 7fc7ad00f040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-19T12:43:56.149 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.146+0000 7f6e90c75640 1 -- 172.21.15.179:0/128583228 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6e8c0cdb30 msgr2=0x7f6e8c0cdfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:56.150 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.146+0000 7f6e90c75640 1 --2- 172.21.15.179:0/128583228 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6e8c0cdb30 0x7f6e8c0cdfe0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6e74005fd0 tx=0x7f6e7402bd90 comp rx=0 tx=0).stop 2023-10-19T12:43:56.150 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.146+0000 7f6e90c75640 1 -- 172.21.15.179:0/128583228 shutdown_connections 2023-10-19T12:43:56.150 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.146+0000 7f6e90c75640 1 --2- 172.21.15.179:0/128583228 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6e8c0cdb30 0x7f6e8c0cdfe0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:56.150 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.146+0000 7f6e90c75640 1 --2- 172.21.15.179:0/128583228 >> v2:172.21.15.57:3300/0 conn(0x7f6e8c159060 0x7f6e8c06d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:56.151 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.146+0000 7f6e90c75640 1 -- 172.21.15.179:0/128583228 >> 172.21.15.179:0/128583228 conn(0x7f6e8c0b0670 msgr2=0x7f6e8c0b0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:43:56.151 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.146+0000 7f6e90c75640 1 -- 172.21.15.179:0/128583228 shutdown_connections 2023-10-19T12:43:56.151 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.146+0000 7f6e90c75640 1 -- 172.21.15.179:0/128583228 wait complete. 2023-10-19T12:43:56.154 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.147+0000 7f6e90c75640 1 Processor -- start 2023-10-19T12:43:56.154 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.147+0000 7f6e90c75640 1 -- start start 2023-10-19T12:43:56.154 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.148+0000 7f6e90c75640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6e8c0cdb30 0x7f6e8c14bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:43:56.154 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.148+0000 7f6e90c75640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6e8c159060 0x7f6e8c15ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:43:56.155 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.148+0000 7f6e90c75640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6e8c15ccf0 0x7f6e8c156620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:43:56.155 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.148+0000 7f6e90c75640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e8c15a810 con 0x7f6e8c159060 2023-10-19T12:43:56.155 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.148+0000 7f6e90c75640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e8c15c6d0 con 0x7f6e8c0cdb30 2023-10-19T12:43:56.155 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.148+0000 7f6e90c75640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e8c15b630 con 0x7f6e8c15ccf0 2023-10-19T12:43:56.155 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.148+0000 7f6e8a59c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6e8c0cdb30 0x7f6e8c14bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:43:56.156 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.148+0000 7f6e8ad9d640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6e8c15ccf0 0x7f6e8c156620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:43:56.156 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.148+0000 7f6e8a59c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6e8c0cdb30 0x7f6e8c14bdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.179:55728/0 (socket says 172.21.15.179:55728) 2023-10-19T12:43:56.156 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.148+0000 7f6e8ad9d640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6e8c15ccf0 0x7f6e8c156620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3301/0 says I am v2:172.21.15.179:44502/0 (socket says 172.21.15.179:44502) 2023-10-19T12:43:56.156 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.148+0000 7f6e8a59c640 1 -- 172.21.15.179:0/4102996805 learned_addr learned my addr 172.21.15.179:0/4102996805 (peer_addr_for_me v2:172.21.15.179:0/0) 2023-10-19T12:43:56.156 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.148+0000 7f6e83fff640 1 --2- 172.21.15.179:0/4102996805 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6e8c159060 0x7f6e8c15ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:43:56.157 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.149+0000 7f6e8a59c640 1 -- 172.21.15.179:0/4102996805 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6e8c15ccf0 msgr2=0x7f6e8c156620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:56.157 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.149+0000 7f6e8a59c640 1 --2- 172.21.15.179:0/4102996805 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6e8c15ccf0 0x7f6e8c156620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:56.157 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.149+0000 7f6e8a59c640 1 -- 172.21.15.179:0/4102996805 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6e8c159060 msgr2=0x7f6e8c15ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:56.157 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.149+0000 7f6e8a59c640 1 --2- 172.21.15.179:0/4102996805 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6e8c159060 0x7f6e8c15ad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:56.158 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.149+0000 7f6e8a59c640 1 -- 172.21.15.179:0/4102996805 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e74009560 con 0x7f6e8c0cdb30 2023-10-19T12:43:56.158 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.149+0000 7f6e8a59c640 1 --2- 172.21.15.179:0/4102996805 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6e8c0cdb30 0x7f6e8c14bdd0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6e7c00d950 tx=0x7f6e7c00de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:43:56.158 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.149+0000 7f6e82ffd640 1 -- 172.21.15.179:0/4102996805 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e7c009c30 con 0x7f6e8c0cdb30 2023-10-19T12:43:56.158 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.149+0000 7f6e82ffd640 1 -- 172.21.15.179:0/4102996805 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f6e7c009e80 con 0x7f6e8c0cdb30 2023-10-19T12:43:56.158 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.149+0000 7f6e90c75640 1 -- 172.21.15.179:0/4102996805 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e8c15b910 con 0x7f6e8c0cdb30 2023-10-19T12:43:56.159 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.149+0000 7f6e82ffd640 1 -- 172.21.15.179:0/4102996805 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e7c0106c0 con 0x7f6e8c0cdb30 2023-10-19T12:43:56.159 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.149+0000 7f6e90c75640 1 -- 172.21.15.179:0/4102996805 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e8c15bed0 con 0x7f6e8c0cdb30 2023-10-19T12:43:56.159 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.149+0000 7f6e82ffd640 1 -- 172.21.15.179:0/4102996805 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f6e7c010b20 con 0x7f6e8c0cdb30 2023-10-19T12:43:56.159 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.149+0000 7f6e82ffd640 1 -- 172.21.15.179:0/4102996805 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7f6e7c01bae0 con 0x7f6e8c0cdb30 2023-10-19T12:43:56.159 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.150+0000 7f6e90c75640 1 -- 172.21.15.179:0/4102996805 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e8c0675f0 con 0x7f6e8c0cdb30 2023-10-19T12:43:56.160 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.152+0000 7f6e82ffd640 1 -- 172.21.15.179:0/4102996805 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f6e7c020060 con 0x7f6e8c0cdb30 2023-10-19T12:43:56.177 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:43:56.174+0000 7fc7ad00f040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-19T12:43:56.227 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.224+0000 7f6e90c75640 1 -- 172.21.15.179:0/4102996805 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "osd new", "uuid": "3b7b20a2-ea0e-40ca-be09-29a46bb1a997", "id": 6} v 0) v1 -- 0x7f6e8c0ce620 con 0x7f6e8c0cdb30 2023-10-19T12:43:56.230 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.227+0000 7f6e82ffd640 1 -- 172.21.15.179:0/4102996805 <== mon.1 v2:172.21.15.179:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "3b7b20a2-ea0e-40ca-be09-29a46bb1a997", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f6e7c017070 con 0x7f6e8c0cdb30 2023-10-19T12:43:56.231 INFO:teuthology.orchestra.run.smithi179.stdout:6 2023-10-19T12:43:56.232 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.228+0000 7f6e90c75640 1 -- 172.21.15.179:0/4102996805 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6e8c0cdb30 msgr2=0x7f6e8c14bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:56.232 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.228+0000 7f6e90c75640 1 --2- 172.21.15.179:0/4102996805 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6e8c0cdb30 0x7f6e8c14bdd0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6e7c00d950 tx=0x7f6e7c00de10 comp rx=0 tx=0).stop 2023-10-19T12:43:56.233 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.228+0000 7f6e90c75640 1 -- 172.21.15.179:0/4102996805 shutdown_connections 2023-10-19T12:43:56.233 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.228+0000 7f6e90c75640 1 --2- 172.21.15.179:0/4102996805 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6e8c15ccf0 0x7f6e8c156620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:56.233 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.228+0000 7f6e90c75640 1 --2- 172.21.15.179:0/4102996805 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6e8c159060 0x7f6e8c15ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:56.234 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.228+0000 7f6e90c75640 1 --2- 172.21.15.179:0/4102996805 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6e8c0cdb30 0x7f6e8c14bdd0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:56.234 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.228+0000 7f6e90c75640 1 -- 172.21.15.179:0/4102996805 >> 172.21.15.179:0/4102996805 conn(0x7f6e8c0b0670 msgr2=0x7f6e8c15d120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:43:56.234 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.228+0000 7f6e90c75640 1 -- 172.21.15.179:0/4102996805 shutdown_connections 2023-10-19T12:43:56.235 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.228+0000 7f6e90c75640 1 -- 172.21.15.179:0/4102996805 wait complete. 2023-10-19T12:43:56.242 DEBUG:teuthology.orchestra.run.smithi179:> sudo ceph --cluster ceph osd new 2ec733c5-528d-4f91-b32f-42dc4ee0525f 7 2023-10-19T12:43:56.279 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:43:56.276+0000 7fc7ad00f040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-19T12:43:56.344 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.341+0000 7fad9d456640 1 -- 172.21.15.179:0/478454979 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fad98158cf0 msgr2=0x7fad9815c0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:56.344 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.341+0000 7fad9d456640 1 --2- 172.21.15.179:0/478454979 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fad98158cf0 0x7fad9815c0e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fad8c009830 tx=0x7fad8c02bdf0 comp rx=0 tx=0).stop 2023-10-19T12:43:56.345 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.341+0000 7fad9d456640 1 -- 172.21.15.179:0/478454979 shutdown_connections 2023-10-19T12:43:56.345 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.341+0000 7fad9d456640 1 --2- 172.21.15.179:0/478454979 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fad98158cf0 0x7fad9815c0e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:56.345 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.341+0000 7fad9d456640 1 -- 172.21.15.179:0/478454979 >> 172.21.15.179:0/478454979 conn(0x7fad980af7a0 msgr2=0x7fad980afb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:43:56.345 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.341+0000 7fad9d456640 1 -- 172.21.15.179:0/478454979 shutdown_connections 2023-10-19T12:43:56.345 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.341+0000 7fad9d456640 1 -- 172.21.15.179:0/478454979 wait complete. 2023-10-19T12:43:56.348 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.342+0000 7fad9d456640 1 Processor -- start 2023-10-19T12:43:56.348 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.342+0000 7fad9d456640 1 -- start start 2023-10-19T12:43:56.349 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.343+0000 7fad9d456640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fad980cd9e0 0x7fad981517d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:43:56.349 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.343+0000 7fad9d456640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fad98158910 0x7fad981f1340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:43:56.349 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.343+0000 7fad9d456640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fad98158cf0 0x7fad980ac200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:43:56.350 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.343+0000 7fad9d456640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad980ab3e0 con 0x7fad98158910 2023-10-19T12:43:56.350 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.343+0000 7fad9d456640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad980b09c0 con 0x7fad980cd9e0 2023-10-19T12:43:56.350 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.343+0000 7fad9d456640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad980b0ca0 con 0x7fad98158cf0 2023-10-19T12:43:56.350 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.343+0000 7fad96ffd640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fad980cd9e0 0x7fad981517d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:43:56.351 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.343+0000 7fad96ffd640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fad980cd9e0 0x7fad981517d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.179:55740/0 (socket says 172.21.15.179:55740) 2023-10-19T12:43:56.351 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.343+0000 7fad96ffd640 1 -- 172.21.15.179:0/1273505703 learned_addr learned my addr 172.21.15.179:0/1273505703 (peer_addr_for_me v2:172.21.15.179:0/0) 2023-10-19T12:43:56.351 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.343+0000 7fad977fe640 1 --2- 172.21.15.179:0/1273505703 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fad98158cf0 0x7fad980ac200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:43:56.351 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.343+0000 7fad967fc640 1 --2- 172.21.15.179:0/1273505703 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fad98158910 0x7fad981f1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:43:56.351 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.343+0000 7fad96ffd640 1 -- 172.21.15.179:0/1273505703 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fad98158cf0 msgr2=0x7fad980ac200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:56.351 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.343+0000 7fad96ffd640 1 --2- 172.21.15.179:0/1273505703 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fad98158cf0 0x7fad980ac200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:56.352 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.343+0000 7fad96ffd640 1 -- 172.21.15.179:0/1273505703 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fad98158910 msgr2=0x7fad981f1340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:56.352 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.343+0000 7fad96ffd640 1 --2- 172.21.15.179:0/1273505703 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fad98158910 0x7fad981f1340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:56.352 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.343+0000 7fad96ffd640 1 -- 172.21.15.179:0/1273505703 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad8c009490 con 0x7fad980cd9e0 2023-10-19T12:43:56.352 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.343+0000 7fad967fc640 1 --2- 172.21.15.179:0/1273505703 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fad98158910 0x7fad981f1340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T12:43:56.352 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.343+0000 7fad977fe640 1 --2- 172.21.15.179:0/1273505703 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fad98158cf0 0x7fad980ac200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T12:43:56.353 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.343+0000 7fad96ffd640 1 --2- 172.21.15.179:0/1273505703 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fad980cd9e0 0x7fad981517d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fad8800c490 tx=0x7fad8800c950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:43:56.353 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.343+0000 7fad7f7fe640 1 -- 172.21.15.179:0/1273505703 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad88012830 con 0x7fad980cd9e0 2023-10-19T12:43:56.353 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.343+0000 7fad7f7fe640 1 -- 172.21.15.179:0/1273505703 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fad88012a80 con 0x7fad980cd9e0 2023-10-19T12:43:56.353 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.343+0000 7fad9d456640 1 -- 172.21.15.179:0/1273505703 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad980b0f80 con 0x7fad980cd9e0 2023-10-19T12:43:56.353 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.343+0000 7fad7f7fe640 1 -- 172.21.15.179:0/1273505703 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad88011380 con 0x7fad980cd9e0 2023-10-19T12:43:56.354 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.344+0000 7fad9d456640 1 -- 172.21.15.179:0/1273505703 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad980b1540 con 0x7fad980cd9e0 2023-10-19T12:43:56.354 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.344+0000 7fad7f7fe640 1 -- 172.21.15.179:0/1273505703 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fad880117e0 con 0x7fad980cd9e0 2023-10-19T12:43:56.354 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.344+0000 7fad7f7fe640 1 -- 172.21.15.179:0/1273505703 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7fad8801a3b0 con 0x7fad980cd9e0 2023-10-19T12:43:56.354 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.344+0000 7fad9d456640 1 -- 172.21.15.179:0/1273505703 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad98151d10 con 0x7fad980cd9e0 2023-10-19T12:43:56.355 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.346+0000 7fad7f7fe640 1 -- 172.21.15.179:0/1273505703 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7fad88016020 con 0x7fad980cd9e0 2023-10-19T12:43:56.378 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:43:56.375+0000 7fc7ad00f040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-19T12:43:56.420 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.417+0000 7fad9d456640 1 -- 172.21.15.179:0/1273505703 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "osd new", "uuid": "2ec733c5-528d-4f91-b32f-42dc4ee0525f", "id": 7} v 0) v1 -- 0x7fad980b1730 con 0x7fad980cd9e0 2023-10-19T12:43:56.424 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.421+0000 7fad7f7fe640 1 -- 172.21.15.179:0/1273505703 <== mon.1 v2:172.21.15.179:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "2ec733c5-528d-4f91-b32f-42dc4ee0525f", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7fad8802a0b0 con 0x7fad980cd9e0 2023-10-19T12:43:56.424 INFO:teuthology.orchestra.run.smithi179.stdout:7 2023-10-19T12:43:56.425 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.422+0000 7fad9d456640 1 -- 172.21.15.179:0/1273505703 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fad980cd9e0 msgr2=0x7fad981517d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:56.426 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.422+0000 7fad9d456640 1 --2- 172.21.15.179:0/1273505703 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fad980cd9e0 0x7fad981517d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fad8800c490 tx=0x7fad8800c950 comp rx=0 tx=0).stop 2023-10-19T12:43:56.426 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.422+0000 7fad9d456640 1 -- 172.21.15.179:0/1273505703 shutdown_connections 2023-10-19T12:43:56.426 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.422+0000 7fad9d456640 1 --2- 172.21.15.179:0/1273505703 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fad98158cf0 0x7fad980ac200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:56.426 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.422+0000 7fad9d456640 1 --2- 172.21.15.179:0/1273505703 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fad98158910 0x7fad981f1340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:56.426 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.422+0000 7fad9d456640 1 --2- 172.21.15.179:0/1273505703 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fad980cd9e0 0x7fad981517d0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:56.427 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.422+0000 7fad9d456640 1 -- 172.21.15.179:0/1273505703 >> 172.21.15.179:0/1273505703 conn(0x7fad980af7a0 msgr2=0x7fad980cb950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:43:56.427 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.422+0000 7fad9d456640 1 -- 172.21.15.179:0/1273505703 shutdown_connections 2023-10-19T12:43:56.427 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:43:56.422+0000 7fad9d456640 1 -- 172.21.15.179:0/1273505703 wait complete. 2023-10-19T12:43:56.435 INFO:tasks.ceph.osd.0:Restarting daemon 2023-10-19T12:43:56.436 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-10-19T12:43:56.439 INFO:tasks.ceph.osd.0:Started 2023-10-19T12:43:56.439 INFO:tasks.ceph.osd.1:Restarting daemon 2023-10-19T12:43:56.440 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-10-19T12:43:56.442 INFO:tasks.ceph.osd.1:Started 2023-10-19T12:43:56.443 INFO:tasks.ceph.osd.2:Restarting daemon 2023-10-19T12:43:56.443 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-10-19T12:43:56.445 INFO:tasks.ceph.osd.2:Started 2023-10-19T12:43:56.446 INFO:tasks.ceph.osd.3:Restarting daemon 2023-10-19T12:43:56.446 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-10-19T12:43:56.449 INFO:tasks.ceph.osd.3:Started 2023-10-19T12:43:56.449 INFO:tasks.ceph.osd.4:Restarting daemon 2023-10-19T12:43:56.449 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2023-10-19T12:43:56.452 INFO:tasks.ceph.osd.4:Started 2023-10-19T12:43:56.453 INFO:tasks.ceph.osd.5:Restarting daemon 2023-10-19T12:43:56.453 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2023-10-19T12:43:56.455 INFO:tasks.ceph.osd.5:Started 2023-10-19T12:43:56.456 INFO:tasks.ceph.osd.6:Restarting daemon 2023-10-19T12:43:56.456 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2023-10-19T12:43:56.458 INFO:tasks.ceph.osd.6:Started 2023-10-19T12:43:56.458 INFO:tasks.ceph.osd.7:Restarting daemon 2023-10-19T12:43:56.459 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2023-10-19T12:43:56.461 INFO:tasks.ceph.osd.7:Started 2023-10-19T12:43:56.462 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T12:43:56.565 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:43:56.562+0000 7fc7ad00f040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-19T12:43:56.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:56.592+0000 7f49a1a4b640 1 -- 172.21.15.57:0/3496729027 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f499c0cf340 msgr2=0x7f499c0d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:56.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:56.592+0000 7f49a1a4b640 1 --2- 172.21.15.57:0/3496729027 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f499c0cf340 0x7f499c0d1910 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f499c08a120 tx=0x7f498c02bd10 comp rx=0 tx=0).stop 2023-10-19T12:43:56.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:56.592+0000 7f49a1a4b640 1 -- 172.21.15.57:0/3496729027 shutdown_connections 2023-10-19T12:43:56.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:56.592+0000 7f49a1a4b640 1 --2- 172.21.15.57:0/3496729027 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f499c0ceb40 0x7f499c0b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:56.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:56.592+0000 7f49a1a4b640 1 --2- 172.21.15.57:0/3496729027 >> v2:172.21.15.179:3300/0 conn(0x7f499c15b2a0 0x7f499c0cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:56.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:56.592+0000 7f49a1a4b640 1 --2- 172.21.15.57:0/3496729027 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f499c0cf340 0x7f499c0d1910 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:56.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:56.592+0000 7f49a1a4b640 1 -- 172.21.15.57:0/3496729027 >> 172.21.15.57:0/3496729027 conn(0x7f499c0d0d60 msgr2=0x7f499c0d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:43:56.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:56.592+0000 7f49a1a4b640 1 -- 172.21.15.57:0/3496729027 shutdown_connections 2023-10-19T12:43:56.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:56.592+0000 7f49a1a4b640 1 -- 172.21.15.57:0/3496729027 wait complete. 2023-10-19T12:43:56.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:56.594+0000 7f49a1a4b640 1 Processor -- start 2023-10-19T12:43:56.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:56.594+0000 7f49a1a4b640 1 -- start start 2023-10-19T12:43:56.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:56.594+0000 7f49a1a4b640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f499c0ceb40 0x7f499c14a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:43:56.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:56.594+0000 7f49a1a4b640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f499c0cf340 0x7f499c14e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:43:56.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:56.595+0000 7f49a1a4b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f499c15b2a0 0x7f499c0ca160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:43:56.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:56.595+0000 7f49a1a4b640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f499c14a960 con 0x7f499c15b2a0 2023-10-19T12:43:56.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:56.595+0000 7f49a1a4b640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f499c0c7e30 con 0x7f499c0cf340 2023-10-19T12:43:56.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:56.595+0000 7f49a1a4b640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f499c14ea80 con 0x7f499c0ceb40 2023-10-19T12:43:56.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:56.595+0000 7f499b7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f499c15b2a0 0x7f499c0ca160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:43:56.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:56.595+0000 7f499b7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f499c15b2a0 0x7f499c0ca160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51096/0 (socket says 172.21.15.57:51096) 2023-10-19T12:43:56.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:56.595+0000 7f499b7fe640 1 -- 172.21.15.57:0/3786875885 learned_addr learned my addr 172.21.15.57:0/3786875885 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:43:56.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:56.595+0000 7f499a7fc640 1 --2- 172.21.15.57:0/3786875885 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f499c0cf340 0x7f499c14e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:43:56.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:56.595+0000 7f499affd640 1 --2- 172.21.15.57:0/3786875885 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f499c0ceb40 0x7f499c14a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:43:56.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:56.595+0000 7f499b7fe640 1 -- 172.21.15.57:0/3786875885 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f499c0ceb40 msgr2=0x7f499c14a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:56.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:56.595+0000 7f499b7fe640 1 --2- 172.21.15.57:0/3786875885 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f499c0ceb40 0x7f499c14a420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:56.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:56.595+0000 7f499b7fe640 1 -- 172.21.15.57:0/3786875885 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f499c0cf340 msgr2=0x7f499c14e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:56.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:56.595+0000 7f499b7fe640 1 --2- 172.21.15.57:0/3786875885 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f499c0cf340 0x7f499c14e0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:56.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:56.595+0000 7f499b7fe640 1 -- 172.21.15.57:0/3786875885 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f498c009560 con 0x7f499c15b2a0 2023-10-19T12:43:56.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:56.595+0000 7f499b7fe640 1 --2- 172.21.15.57:0/3786875885 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f499c15b2a0 0x7f499c0ca160 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f499400eaa0 tx=0x7f499400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:43:56.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:56.595+0000 7f4993fff640 1 -- 172.21.15.57:0/3786875885 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f499400ce70 con 0x7f499c15b2a0 2023-10-19T12:43:56.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:56.595+0000 7f4993fff640 1 -- 172.21.15.57:0/3786875885 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f4994004c10 con 0x7f499c15b2a0 2023-10-19T12:43:56.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:56.595+0000 7f49a1a4b640 1 -- 172.21.15.57:0/3786875885 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f499c14ed60 con 0x7f499c15b2a0 2023-10-19T12:43:56.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:56.595+0000 7f4993fff640 1 -- 172.21.15.57:0/3786875885 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4994005020 con 0x7f499c15b2a0 2023-10-19T12:43:56.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:56.595+0000 7f49a1a4b640 1 -- 172.21.15.57:0/3786875885 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f499c14f320 con 0x7f499c15b2a0 2023-10-19T12:43:56.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:56.596+0000 7f4993fff640 1 -- 172.21.15.57:0/3786875885 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f49940051a0 con 0x7f499c15b2a0 2023-10-19T12:43:56.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:56.596+0000 7f4993fff640 1 -- 172.21.15.57:0/3786875885 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f4994014070 con 0x7f499c15b2a0 2023-10-19T12:43:56.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:56.596+0000 7f49a1a4b640 1 -- 172.21.15.57:0/3786875885 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f499c0c8050 con 0x7f499c15b2a0 2023-10-19T12:43:56.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:56.598+0000 7f4993fff640 1 -- 172.21.15.57:0/3786875885 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f499c0c8050 con 0x7f499c15b2a0 2023-10-19T12:43:56.611 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:43:56.608+0000 7fc7ad00f040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-19T12:43:56.662 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:56.659+0000 7f49a1a4b640 1 -- 172.21.15.57:0/3786875885 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f499c0d07d0 con 0x7f499c15b2a0 2023-10-19T12:43:56.663 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:56.660+0000 7f4993fff640 1 -- 172.21.15.57:0/3786875885 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f4994005450 con 0x7f499c15b2a0 2023-10-19T12:43:56.663 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:43:56.663 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":10,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T12:43:56.419827+0000","last_up_change":"0.000000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T12:43:56.664 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:56.660+0000 7f49a1a4b640 1 -- 172.21.15.57:0/3786875885 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f499c15b2a0 msgr2=0x7f499c0ca160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:56.665 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:56.660+0000 7f49a1a4b640 1 --2- 172.21.15.57:0/3786875885 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f499c15b2a0 0x7f499c0ca160 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f499400eaa0 tx=0x7f499400ef60 comp rx=0 tx=0).stop 2023-10-19T12:43:56.665 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:56.660+0000 7f49a1a4b640 1 -- 172.21.15.57:0/3786875885 shutdown_connections 2023-10-19T12:43:56.665 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:56.660+0000 7f49a1a4b640 1 --2- 172.21.15.57:0/3786875885 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f499c15b2a0 0x7f499c0ca160 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:56.665 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:56.660+0000 7f49a1a4b640 1 --2- 172.21.15.57:0/3786875885 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f499c0cf340 0x7f499c14e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:56.665 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:56.660+0000 7f49a1a4b640 1 --2- 172.21.15.57:0/3786875885 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f499c0ceb40 0x7f499c14a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:56.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:56.660+0000 7f49a1a4b640 1 -- 172.21.15.57:0/3786875885 >> 172.21.15.57:0/3786875885 conn(0x7f499c0d0d60 msgr2=0x7f499c0cd260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:43:56.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:56.661+0000 7f49a1a4b640 1 -- 172.21.15.57:0/3786875885 shutdown_connections 2023-10-19T12:43:56.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:43:56.661+0000 7f49a1a4b640 1 -- 172.21.15.57:0/3786875885 wait complete. 2023-10-19T12:43:56.674 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-10-19T12:43:56.674 INFO:tasks.ceph:Waiting for OSDs to come up 2023-10-19T12:43:56.759 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:43:56.756+0000 7fc7ad00f040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-19T12:43:56.866 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:43:56.863+0000 7fdc67c3a040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-19T12:43:57.050 INFO:tasks.ceph.osd.5.smithi179.stderr:2023-10-19T12:43:57.046+0000 7f7c90010500 -1 Falling back to public interface 2023-10-19T12:43:57.051 INFO:tasks.ceph.osd.2.smithi057.stderr:2023-10-19T12:43:57.046+0000 7fec76f22500 -1 Falling back to public interface 2023-10-19T12:43:57.051 INFO:tasks.ceph.osd.4.smithi179.stderr:2023-10-19T12:43:57.047+0000 7f7e6e414500 -1 Falling back to public interface 2023-10-19T12:43:57.054 INFO:tasks.ceph.osd.7.smithi179.stderr:2023-10-19T12:43:57.051+0000 7fe0bc39b500 -1 Falling back to public interface 2023-10-19T12:43:57.058 INFO:tasks.ceph.osd.6.smithi179.stderr:2023-10-19T12:43:57.055+0000 7f1991018500 -1 Falling back to public interface 2023-10-19T12:43:57.067 INFO:tasks.ceph.osd.1.smithi057.stderr:2023-10-19T12:43:57.065+0000 7f579a416500 -1 Falling back to public interface 2023-10-19T12:43:57.075 INFO:tasks.ceph.osd.3.smithi057.stderr:2023-10-19T12:43:57.072+0000 7fdc89069500 -1 Falling back to public interface 2023-10-19T12:43:57.078 INFO:tasks.ceph.osd.0.smithi057.stderr:2023-10-19T12:43:57.075+0000 7f92b220e500 -1 Falling back to public interface 2023-10-19T12:43:57.140 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:43:57.137+0000 7fdc67c3a040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-19T12:43:57.190 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:43:57.187+0000 7fdc67c3a040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-19T12:43:57.292 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:43:57.289+0000 7fdc67c3a040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-19T12:43:57.480 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-19T12:43:57.576 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:43:57.572+0000 7fed8a632640 1 -- 172.21.15.57:0/703182574 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fed84070080 msgr2=0x7fed84159f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:57.577 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:43:57.572+0000 7fed8a632640 1 --2- 172.21.15.57:0/703182574 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fed84070080 0x7fed84159f40 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fed78009830 tx=0x7fed7802bd60 comp rx=0 tx=0).stop 2023-10-19T12:43:57.577 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:43:57.572+0000 7fed8a632640 1 -- 172.21.15.57:0/703182574 shutdown_connections 2023-10-19T12:43:57.577 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:43:57.572+0000 7fed8a632640 1 --2- 172.21.15.57:0/703182574 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fed84070080 0x7fed84159f40 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:57.577 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:43:57.572+0000 7fed8a632640 1 --2- 172.21.15.57:0/703182574 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fed841424c0 0x7fed8406c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:57.578 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:43:57.572+0000 7fed8a632640 1 -- 172.21.15.57:0/703182574 >> 172.21.15.57:0/703182574 conn(0x7fed840af730 msgr2=0x7fed840afb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:43:57.578 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:43:57.572+0000 7fed8a632640 1 -- 172.21.15.57:0/703182574 shutdown_connections 2023-10-19T12:43:57.578 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:43:57.572+0000 7fed8a632640 1 -- 172.21.15.57:0/703182574 wait complete. 2023-10-19T12:43:57.578 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:43:57.573+0000 7fed8a632640 1 Processor -- start 2023-10-19T12:43:57.578 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:43:57.573+0000 7fed8a632640 1 -- start start 2023-10-19T12:43:57.581 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:43:57.574+0000 7fed8a632640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fed84069720 0x7fed841eed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:43:57.581 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:43:57.574+0000 7fed8a632640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fed84070080 0x7fed840b0b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:43:57.581 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:43:57.574+0000 7fed8a632640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fed841424c0 0x7fed84158100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:43:57.582 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:43:57.574+0000 7fed8a632640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed840b1510 con 0x7fed841424c0 2023-10-19T12:43:57.582 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:43:57.574+0000 7fed8a632640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed840b17f0 con 0x7fed84069720 2023-10-19T12:43:57.582 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:43:57.574+0000 7fed8a632640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed841f2c50 con 0x7fed84070080 2023-10-19T12:43:57.582 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:43:57.574+0000 7fed88bcf640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fed841424c0 0x7fed84158100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:43:57.582 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:43:57.574+0000 7fed88bcf640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fed841424c0 0x7fed84158100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:36152/0 (socket says 172.21.15.57:36152) 2023-10-19T12:43:57.583 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:43:57.574+0000 7fed88bcf640 1 -- 172.21.15.57:0/3888509383 learned_addr learned my addr 172.21.15.57:0/3888509383 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:43:57.583 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:43:57.574+0000 7fed83fff640 1 --2- 172.21.15.57:0/3888509383 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fed84069720 0x7fed841eed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:43:57.583 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:43:57.574+0000 7fed837fe640 1 --2- 172.21.15.57:0/3888509383 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fed84070080 0x7fed840b0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:43:57.583 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:43:57.574+0000 7fed88bcf640 1 -- 172.21.15.57:0/3888509383 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fed84070080 msgr2=0x7fed840b0b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:57.583 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:43:57.574+0000 7fed88bcf640 1 --2- 172.21.15.57:0/3888509383 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fed84070080 0x7fed840b0b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:57.584 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:43:57.574+0000 7fed88bcf640 1 -- 172.21.15.57:0/3888509383 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fed84069720 msgr2=0x7fed841eed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:57.584 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:43:57.574+0000 7fed88bcf640 1 --2- 172.21.15.57:0/3888509383 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fed84069720 0x7fed841eed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:57.584 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:43:57.574+0000 7fed88bcf640 1 -- 172.21.15.57:0/3888509383 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed78009490 con 0x7fed841424c0 2023-10-19T12:43:57.584 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:43:57.574+0000 7fed83fff640 1 --2- 172.21.15.57:0/3888509383 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fed84069720 0x7fed841eed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:43:57.584 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:43:57.575+0000 7fed88bcf640 1 --2- 172.21.15.57:0/3888509383 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fed841424c0 0x7fed84158100 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fed7c00cb40 tx=0x7fed7c00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:43:57.585 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:43:57.575+0000 7fed80ff9640 1 -- 172.21.15.57:0/3888509383 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed7c00aca0 con 0x7fed841424c0 2023-10-19T12:43:57.585 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:43:57.575+0000 7fed80ff9640 1 -- 172.21.15.57:0/3888509383 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fed7c012420 con 0x7fed841424c0 2023-10-19T12:43:57.585 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:43:57.575+0000 7fed8a632640 1 -- 172.21.15.57:0/3888509383 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed841f2f30 con 0x7fed841424c0 2023-10-19T12:43:57.585 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:43:57.575+0000 7fed80ff9640 1 -- 172.21.15.57:0/3888509383 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed7c012850 con 0x7fed841424c0 2023-10-19T12:43:57.586 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:43:57.575+0000 7fed8a632640 1 -- 172.21.15.57:0/3888509383 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed840b1070 con 0x7fed841424c0 2023-10-19T12:43:57.586 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:43:57.576+0000 7fed8a632640 1 -- 172.21.15.57:0/3888509383 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed840b1260 con 0x7fed841424c0 2023-10-19T12:43:57.586 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:43:57.576+0000 7fed80ff9640 1 -- 172.21.15.57:0/3888509383 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 2) v1 ==== 44566+0+0 (secure 0 0 0) 0x7fed7c0129b0 con 0x7fed841424c0 2023-10-19T12:43:57.586 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:43:57.576+0000 7fed80ff9640 1 -- 172.21.15.57:0/3888509383 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fed7c049db0 con 0x7fed841424c0 2023-10-19T12:43:57.586 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:43:57.578+0000 7fed80ff9640 1 -- 172.21.15.57:0/3888509383 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7fed7c021d10 con 0x7fed841424c0 2023-10-19T12:43:57.606 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:43:57.603+0000 7fc7ad00f040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-19T12:43:57.643 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:43:57.640+0000 7fed8a632640 1 -- 172.21.15.57:0/3888509383 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fed840675f0 con 0x7fed841424c0 2023-10-19T12:43:57.644 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:43:57.640+0000 7fed80ff9640 1 -- 172.21.15.57:0/3888509383 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7fed7c0212b0 con 0x7fed841424c0 2023-10-19T12:43:57.644 INFO:teuthology.misc.health.smithi057.stdout: 2023-10-19T12:43:57.644 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":10,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T12:43:56.419827+0000","last_up_change":"0.000000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T12:43:57.645 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:43:57.641+0000 7fed8a632640 1 -- 172.21.15.57:0/3888509383 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fed841424c0 msgr2=0x7fed84158100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:43:57.645 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:43:57.641+0000 7fed8a632640 1 --2- 172.21.15.57:0/3888509383 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fed841424c0 0x7fed84158100 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fed7c00cb40 tx=0x7fed7c00a060 comp rx=0 tx=0).stop 2023-10-19T12:43:57.646 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:43:57.641+0000 7fed8a632640 1 -- 172.21.15.57:0/3888509383 shutdown_connections 2023-10-19T12:43:57.646 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:43:57.641+0000 7fed8a632640 1 --2- 172.21.15.57:0/3888509383 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fed841424c0 0x7fed84158100 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:57.646 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:43:57.641+0000 7fed8a632640 1 --2- 172.21.15.57:0/3888509383 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fed84070080 0x7fed840b0b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:57.646 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:43:57.641+0000 7fed8a632640 1 --2- 172.21.15.57:0/3888509383 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fed84069720 0x7fed841eed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:43:57.646 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:43:57.641+0000 7fed8a632640 1 -- 172.21.15.57:0/3888509383 >> 172.21.15.57:0/3888509383 conn(0x7fed840af730 msgr2=0x7fed8413a270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:43:57.647 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:43:57.642+0000 7fed8a632640 1 -- 172.21.15.57:0/3888509383 shutdown_connections 2023-10-19T12:43:57.647 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:43:57.642+0000 7fed8a632640 1 -- 172.21.15.57:0/3888509383 wait complete. 2023-10-19T12:43:57.654 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-10-19T12:43:57.893 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:43:57.890+0000 7fc7ad00f040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-19T12:43:57.947 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:43:57.944+0000 7fc7ad00f040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-19T12:43:58.052 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:43:58.049+0000 7fc7ad00f040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-19T12:43:58.319 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:43:58.316+0000 7fdc09051640 -1 mgr.server handle_report got status from non-daemon mon.b 2023-10-19T12:43:58.319 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:43:58.316+0000 7fdc09051640 -1 mgr.server handle_report got status from non-daemon mon.c 2023-10-19T12:44:00.539 INFO:tasks.ceph.osd.2.smithi057.stderr:2023-10-19T12:44:00.536+0000 7fec76f22500 -1 osd.2 0 log_to_monitors true 2023-10-19T12:44:00.551 INFO:tasks.ceph.osd.5.smithi179.stderr:2023-10-19T12:44:00.548+0000 7f7c90010500 -1 osd.5 0 log_to_monitors true 2023-10-19T12:44:00.552 INFO:tasks.ceph.osd.4.smithi179.stderr:2023-10-19T12:44:00.549+0000 7f7e6e414500 -1 osd.4 0 log_to_monitors true 2023-10-19T12:44:00.555 INFO:tasks.ceph.osd.1.smithi057.stderr:2023-10-19T12:44:00.552+0000 7f579a416500 -1 osd.1 0 log_to_monitors true 2023-10-19T12:44:00.588 INFO:tasks.ceph.osd.6.smithi179.stderr:2023-10-19T12:44:00.585+0000 7f1991018500 -1 osd.6 0 log_to_monitors true 2023-10-19T12:44:00.588 INFO:tasks.ceph.osd.7.smithi179.stderr:2023-10-19T12:44:00.585+0000 7fe0bc39b500 -1 osd.7 0 log_to_monitors true 2023-10-19T12:44:00.600 INFO:tasks.ceph.osd.3.smithi057.stderr:2023-10-19T12:44:00.597+0000 7fdc89069500 -1 osd.3 0 log_to_monitors true 2023-10-19T12:44:00.604 INFO:tasks.ceph.osd.0.smithi057.stderr:2023-10-19T12:44:00.602+0000 7f92b220e500 -1 osd.0 0 log_to_monitors true 2023-10-19T12:44:04.387 INFO:tasks.ceph.osd.6.smithi179.stderr:2023-10-19T12:44:04.384+0000 7f198d219640 -1 osd.6 0 waiting for initial osdmap 2023-10-19T12:44:04.400 INFO:tasks.ceph.osd.6.smithi179.stderr:2023-10-19T12:44:04.397+0000 7f1984823640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-19T12:44:04.461 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-19T12:44:04.488 INFO:tasks.ceph.osd.7.smithi179.stderr:2023-10-19T12:44:04.485+0000 7fe0b8329640 -1 osd.7 0 waiting for initial osdmap 2023-10-19T12:44:04.499 INFO:tasks.ceph.osd.4.smithi179.stderr:2023-10-19T12:44:04.496+0000 7f7e69d75640 -1 osd.4 0 waiting for initial osdmap 2023-10-19T12:44:04.502 INFO:tasks.ceph.osd.5.smithi179.stderr:2023-10-19T12:44:04.499+0000 7f7c8c212640 -1 osd.5 0 waiting for initial osdmap 2023-10-19T12:44:04.504 INFO:tasks.ceph.osd.7.smithi179.stderr:2023-10-19T12:44:04.501+0000 7fe0b0149640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-19T12:44:04.514 INFO:tasks.ceph.osd.4.smithi179.stderr:2023-10-19T12:44:04.511+0000 7f7e61b95640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-19T12:44:04.515 INFO:tasks.ceph.osd.5.smithi179.stderr:2023-10-19T12:44:04.513+0000 7f7c8381c640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-19T12:44:04.573 INFO:tasks.ceph.osd.3.smithi057.stderr:2023-10-19T12:44:04.565+0000 7fdc84ff5640 -1 osd.3 0 waiting for initial osdmap 2023-10-19T12:44:04.584 INFO:tasks.ceph.osd.0.smithi057.stderr:2023-10-19T12:44:04.581+0000 7f92adb40640 -1 osd.0 0 waiting for initial osdmap 2023-10-19T12:44:04.588 INFO:tasks.ceph.osd.3.smithi057.stderr:2023-10-19T12:44:04.585+0000 7fdc7ce15640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-19T12:44:04.602 INFO:tasks.ceph.osd.0.smithi057.stderr:2023-10-19T12:44:04.599+0000 7f92a5960640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-19T12:44:04.616 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:04.613+0000 7fe899a56640 1 -- 172.21.15.57:0/1170873331 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe89415b260 msgr2=0x7fe8940cf310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:04.616 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:04.613+0000 7fe899a56640 1 --2- 172.21.15.57:0/1170873331 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe89415b260 0x7fe8940cf310 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe884005fd0 tx=0x7fe88402be10 comp rx=0 tx=0).stop 2023-10-19T12:44:04.617 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:04.613+0000 7fe899a56640 1 -- 172.21.15.57:0/1170873331 shutdown_connections 2023-10-19T12:44:04.617 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:04.613+0000 7fe899a56640 1 --2- 172.21.15.57:0/1170873331 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe8940cf850 0x7fe8940cc9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:04.617 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:04.613+0000 7fe899a56640 1 --2- 172.21.15.57:0/1170873331 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe89415b260 0x7fe8940cf310 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:04.617 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:04.613+0000 7fe899a56640 1 -- 172.21.15.57:0/1170873331 >> 172.21.15.57:0/1170873331 conn(0x7fe89409a030 msgr2=0x7fe8940d0d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:04.618 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:04.613+0000 7fe899a56640 1 -- 172.21.15.57:0/1170873331 shutdown_connections 2023-10-19T12:44:04.618 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:04.613+0000 7fe899a56640 1 -- 172.21.15.57:0/1170873331 wait complete. 2023-10-19T12:44:04.621 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:04.614+0000 7fe899a56640 1 Processor -- start 2023-10-19T12:44:04.621 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:04.614+0000 7fe899a56640 1 -- start start 2023-10-19T12:44:04.621 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:04.615+0000 7fe899a56640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe8940cf850 0x7fe8940c97d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:04.622 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:04.615+0000 7fe899a56640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe89415b260 0x7fe89415d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:04.622 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:04.615+0000 7fe899a56640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe89415b640 0x7fe8941522d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:04.622 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:04.615+0000 7fe899a56640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8940c8800 con 0x7fe89415b260 2023-10-19T12:44:04.622 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:04.615+0000 7fe899a56640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8940c8a10 con 0x7fe8940cf850 2023-10-19T12:44:04.623 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:04.615+0000 7fe899a56640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8940c8cf0 con 0x7fe89415b640 2023-10-19T12:44:04.623 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:04.615+0000 7fe8927fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe89415b260 0x7fe89415d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:04.623 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:04.615+0000 7fe892ffd640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe8940cf850 0x7fe8940c97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:04.623 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:04.615+0000 7fe892ffd640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe8940cf850 0x7fe8940c97d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.57:40162/0 (socket says 172.21.15.57:40162) 2023-10-19T12:44:04.624 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:04.615+0000 7fe892ffd640 1 -- 172.21.15.57:0/2117837335 learned_addr learned my addr 172.21.15.57:0/2117837335 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:04.624 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:04.615+0000 7fe8937fe640 1 --2- 172.21.15.57:0/2117837335 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe89415b640 0x7fe8941522d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:04.624 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:04.615+0000 7fe892ffd640 1 -- 172.21.15.57:0/2117837335 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe89415b640 msgr2=0x7fe8941522d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:04.624 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:04.615+0000 7fe892ffd640 1 --2- 172.21.15.57:0/2117837335 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe89415b640 0x7fe8941522d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:04.624 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:04.615+0000 7fe892ffd640 1 -- 172.21.15.57:0/2117837335 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe89415b260 msgr2=0x7fe89415d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:04.625 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:04.616+0000 7fe892ffd640 1 --2- 172.21.15.57:0/2117837335 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe89415b260 0x7fe89415d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:04.625 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:04.616+0000 7fe892ffd640 1 -- 172.21.15.57:0/2117837335 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe884009560 con 0x7fe8940cf850 2023-10-19T12:44:04.625 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:04.616+0000 7fe8927fc640 1 --2- 172.21.15.57:0/2117837335 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe89415b260 0x7fe89415d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:44:04.625 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:04.616+0000 7fe892ffd640 1 --2- 172.21.15.57:0/2117837335 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe8940cf850 0x7fe8940c97d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe884009850 tx=0x7fe8840042d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:04.626 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:04.616+0000 7fe87bfff640 1 -- 172.21.15.57:0/2117837335 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe88402cbc0 con 0x7fe8940cf850 2023-10-19T12:44:04.626 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:04.616+0000 7fe87bfff640 1 -- 172.21.15.57:0/2117837335 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fe88402ce10 con 0x7fe8940cf850 2023-10-19T12:44:04.626 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:04.616+0000 7fe87bfff640 1 -- 172.21.15.57:0/2117837335 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe884003830 con 0x7fe8940cf850 2023-10-19T12:44:04.626 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:04.616+0000 7fe899a56640 1 -- 172.21.15.57:0/2117837335 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8940c8fd0 con 0x7fe8940cf850 2023-10-19T12:44:04.626 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:04.616+0000 7fe899a56640 1 -- 172.21.15.57:0/2117837335 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8940c9d10 con 0x7fe8940cf850 2023-10-19T12:44:04.627 INFO:tasks.ceph.osd.2.smithi057.stderr:2023-10-19T12:44:04.617+0000 7fec736c4640 -1 osd.2 0 waiting for initial osdmap 2023-10-19T12:44:04.628 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:04.617+0000 7fe899a56640 1 -- 172.21.15.57:0/2117837335 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8940675f0 con 0x7fe8940cf850 2023-10-19T12:44:04.628 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:04.618+0000 7fe87bfff640 1 -- 172.21.15.57:0/2117837335 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 4) v1 ==== 89173+0+0 (secure 0 0 0) 0x7fe884003990 con 0x7fe8940cf850 2023-10-19T12:44:04.628 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:04.619+0000 7fe87bfff640 1 --2- 172.21.15.57:0/2117837335 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fe88006adb0 0x7fe88006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:04.628 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:04.619+0000 7fe87bfff640 1 -- 172.21.15.57:0/2117837335 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2231+0+0 (secure 0 0 0) 0x7fe8840b2e10 con 0x7fe8940cf850 2023-10-19T12:44:04.629 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:04.619+0000 7fe8927fc640 1 --2- 172.21.15.57:0/2117837335 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fe88006adb0 0x7fe88006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:04.629 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:04.620+0000 7fe8927fc640 1 --2- 172.21.15.57:0/2117837335 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fe88006adb0 0x7fe88006d270 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe888007fd0 tx=0x7fe888009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:04.629 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:04.624+0000 7fe87bfff640 1 -- 172.21.15.57:0/2117837335 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe88407ed30 con 0x7fe8940cf850 2023-10-19T12:44:04.630 INFO:tasks.ceph.osd.1.smithi057.stderr:2023-10-19T12:44:04.627+0000 7f5795e74640 -1 osd.1 0 waiting for initial osdmap 2023-10-19T12:44:04.634 INFO:tasks.ceph.osd.2.smithi057.stderr:2023-10-19T12:44:04.632+0000 7fec6acce640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-19T12:44:04.642 INFO:tasks.ceph.osd.1.smithi057.stderr:2023-10-19T12:44:04.640+0000 7f578dc94640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-19T12:44:04.806 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:04.803+0000 7fe899a56640 1 -- 172.21.15.57:0/2117837335 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe8940cd8c0 con 0x7fe8940cf850 2023-10-19T12:44:04.807 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:04.804+0000 7fe87bfff640 1 -- 172.21.15.57:0/2117837335 <== mon.1 v2:172.21.15.179:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7057 (secure 0 0 0) 0x7fe8840b4020 con 0x7fe8940cf850 2023-10-19T12:44:04.807 INFO:teuthology.misc.health.smithi057.stdout: 2023-10-19T12:44:04.808 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":12,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T12:44:02.328068+0000","last_up_change":"0.000000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T12:44:04.809 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:04.806+0000 7fe899a56640 1 -- 172.21.15.57:0/2117837335 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fe88006adb0 msgr2=0x7fe88006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:04.809 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:04.806+0000 7fe899a56640 1 --2- 172.21.15.57:0/2117837335 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fe88006adb0 0x7fe88006d270 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe888007fd0 tx=0x7fe888009040 comp rx=0 tx=0).stop 2023-10-19T12:44:04.810 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:04.806+0000 7fe899a56640 1 -- 172.21.15.57:0/2117837335 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe8940cf850 msgr2=0x7fe8940c97d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:04.810 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:04.806+0000 7fe899a56640 1 --2- 172.21.15.57:0/2117837335 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe8940cf850 0x7fe8940c97d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe884009850 tx=0x7fe8840042d0 comp rx=0 tx=0).stop 2023-10-19T12:44:04.810 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:04.806+0000 7fe899a56640 1 -- 172.21.15.57:0/2117837335 shutdown_connections 2023-10-19T12:44:04.811 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:04.806+0000 7fe899a56640 1 --2- 172.21.15.57:0/2117837335 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fe88006adb0 0x7fe88006d270 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:04.811 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:04.806+0000 7fe899a56640 1 --2- 172.21.15.57:0/2117837335 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe89415b640 0x7fe8941522d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:04.811 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:04.806+0000 7fe899a56640 1 --2- 172.21.15.57:0/2117837335 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe89415b260 0x7fe89415d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:04.811 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:04.806+0000 7fe899a56640 1 --2- 172.21.15.57:0/2117837335 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe8940cf850 0x7fe8940c97d0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:04.811 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:04.806+0000 7fe899a56640 1 -- 172.21.15.57:0/2117837335 >> 172.21.15.57:0/2117837335 conn(0x7fe89409a030 msgr2=0x7fe8940ad990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:04.812 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:04.806+0000 7fe899a56640 1 -- 172.21.15.57:0/2117837335 shutdown_connections 2023-10-19T12:44:04.812 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:04.806+0000 7fe899a56640 1 -- 172.21.15.57:0/2117837335 wait complete. 2023-10-19T12:44:04.818 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-10-19T12:44:11.625 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-19T12:44:11.731 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:11.728+0000 7f0cf6840640 1 -- 172.21.15.57:0/3779750479 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0cf015b000 msgr2=0x7f0cf00d1810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:11.731 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:11.728+0000 7f0cf6840640 1 --2- 172.21.15.57:0/3779750479 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0cf015b000 0x7f0cf00d1810 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0cf00b37f0 tx=0x7f0ce002bda0 comp rx=0 tx=0).stop 2023-10-19T12:44:11.732 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:11.728+0000 7f0cf6840640 1 -- 172.21.15.57:0/3779750479 shutdown_connections 2023-10-19T12:44:11.732 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:11.728+0000 7f0cf6840640 1 --2- 172.21.15.57:0/3779750479 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0cf00afdc0 0x7f0cf00cf810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:11.732 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:11.728+0000 7f0cf6840640 1 --2- 172.21.15.57:0/3779750479 >> v2:172.21.15.179:3300/0 conn(0x7f0cf015b3e0 0x7f0cf00cf2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:11.732 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:11.728+0000 7f0cf6840640 1 --2- 172.21.15.57:0/3779750479 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0cf015b000 0x7f0cf00d1810 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:11.732 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:11.728+0000 7f0cf6840640 1 -- 172.21.15.57:0/3779750479 >> 172.21.15.57:0/3779750479 conn(0x7f0cf00d0cf0 msgr2=0x7f0cf00d1100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:11.733 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:11.728+0000 7f0cf6840640 1 -- 172.21.15.57:0/3779750479 shutdown_connections 2023-10-19T12:44:11.733 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:11.728+0000 7f0cf6840640 1 -- 172.21.15.57:0/3779750479 wait complete. 2023-10-19T12:44:11.735 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:11.729+0000 7f0cf6840640 1 Processor -- start 2023-10-19T12:44:11.735 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:11.729+0000 7f0cf6840640 1 -- start start 2023-10-19T12:44:11.735 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:11.730+0000 7f0cf6840640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0cf00afdc0 0x7f0cf01e2c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:11.736 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:11.730+0000 7f0cf6840640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0cf015b000 0x7f0cf01e15f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:11.736 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:11.730+0000 7f0cf6840640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0cf015b3e0 0x7f0cf01e1b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:11.736 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:11.730+0000 7f0cf6840640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf00b3dd0 con 0x7f0cf015b3e0 2023-10-19T12:44:11.736 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:11.730+0000 7f0cf6840640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf014a070 con 0x7f0cf015b000 2023-10-19T12:44:11.736 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:11.730+0000 7f0cf6840640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf00b6bd0 con 0x7f0cf00afdc0 2023-10-19T12:44:11.737 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:11.730+0000 7f0cf4ddd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0cf015b3e0 0x7f0cf01e1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:11.737 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:11.730+0000 7f0cf4ddd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0cf015b3e0 0x7f0cf01e1b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40798/0 (socket says 172.21.15.57:40798) 2023-10-19T12:44:11.737 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:11.730+0000 7f0cf4ddd640 1 -- 172.21.15.57:0/2126901160 learned_addr learned my addr 172.21.15.57:0/2126901160 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:11.737 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:11.730+0000 7f0ceffff640 1 --2- 172.21.15.57:0/2126901160 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0cf00afdc0 0x7f0cf01e2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:11.737 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:11.730+0000 7f0cef7fe640 1 --2- 172.21.15.57:0/2126901160 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0cf015b000 0x7f0cf01e15f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:11.737 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:11.731+0000 7f0cf4ddd640 1 -- 172.21.15.57:0/2126901160 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0cf00afdc0 msgr2=0x7f0cf01e2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:11.738 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:11.731+0000 7f0cf4ddd640 1 --2- 172.21.15.57:0/2126901160 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0cf00afdc0 0x7f0cf01e2c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:11.738 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:11.731+0000 7f0cf4ddd640 1 -- 172.21.15.57:0/2126901160 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0cf015b000 msgr2=0x7f0cf01e15f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:11.738 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:11.731+0000 7f0cf4ddd640 1 --2- 172.21.15.57:0/2126901160 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0cf015b000 0x7f0cf01e15f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:11.738 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:11.731+0000 7f0cf4ddd640 1 -- 172.21.15.57:0/2126901160 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ce0009560 con 0x7f0cf015b3e0 2023-10-19T12:44:11.738 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:11.731+0000 7f0cf4ddd640 1 --2- 172.21.15.57:0/2126901160 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0cf015b3e0 0x7f0cf01e1b30 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0ce800caa0 tx=0x7f0ce800cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:11.739 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:11.731+0000 7f0cecff9640 1 -- 172.21.15.57:0/2126901160 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce800adf0 con 0x7f0cf015b3e0 2023-10-19T12:44:11.739 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:11.731+0000 7f0cecff9640 1 -- 172.21.15.57:0/2126901160 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f0ce8004c10 con 0x7f0cf015b3e0 2023-10-19T12:44:11.739 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:11.731+0000 7f0cf6840640 1 -- 172.21.15.57:0/2126901160 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cf00b6610 con 0x7f0cf015b3e0 2023-10-19T12:44:11.739 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:11.731+0000 7f0cecff9640 1 -- 172.21.15.57:0/2126901160 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce8005020 con 0x7f0cf015b3e0 2023-10-19T12:44:11.739 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:11.731+0000 7f0cf6840640 1 -- 172.21.15.57:0/2126901160 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cf0149d00 con 0x7f0cf015b3e0 2023-10-19T12:44:11.740 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:11.732+0000 7f0cf6840640 1 -- 172.21.15.57:0/2126901160 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cf00690e0 con 0x7f0cf015b3e0 2023-10-19T12:44:11.741 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:11.733+0000 7f0cecff9640 1 -- 172.21.15.57:0/2126901160 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f0ce8005180 con 0x7f0cf015b3e0 2023-10-19T12:44:11.741 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:11.733+0000 7f0cecff9640 1 --2- 172.21.15.57:0/2126901160 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f0cd006ae40 0x7f0cd006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:11.741 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:11.733+0000 7f0cecff9640 1 -- 172.21.15.57:0/2126901160 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f0ce8089a70 con 0x7f0cf015b3e0 2023-10-19T12:44:11.741 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:11.733+0000 7f0ceffff640 1 --2- 172.21.15.57:0/2126901160 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f0cd006ae40 0x7f0cd006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:11.742 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:11.734+0000 7f0ceffff640 1 --2- 172.21.15.57:0/2126901160 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f0cd006ae40 0x7f0cd006d300 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0ce0035000 tx=0x7f0ce0004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:11.742 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:11.738+0000 7f0cecff9640 1 -- 172.21.15.57:0/2126901160 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0ce808be60 con 0x7f0cf015b3e0 2023-10-19T12:44:11.917 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:11.914+0000 7f0cf6840640 1 -- 172.21.15.57:0/2126901160 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0cf014a1b0 con 0x7f0cf015b3e0 2023-10-19T12:44:11.917 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:11.915+0000 7f0cecff9640 1 -- 172.21.15.57:0/2126901160 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+12862 (secure 0 0 0) 0x7f0ce801d070 con 0x7f0cf015b3e0 2023-10-19T12:44:11.918 INFO:teuthology.misc.health.smithi057.stdout: 2023-10-19T12:44:11.918 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":17,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T12:44:09.423856+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T12:44:11.920 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:11.916+0000 7f0cf6840640 1 -- 172.21.15.57:0/2126901160 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f0cd006ae40 msgr2=0x7f0cd006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:11.920 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:11.916+0000 7f0cf6840640 1 --2- 172.21.15.57:0/2126901160 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f0cd006ae40 0x7f0cd006d300 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0ce0035000 tx=0x7f0ce0004730 comp rx=0 tx=0).stop 2023-10-19T12:44:11.920 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:11.916+0000 7f0cf6840640 1 -- 172.21.15.57:0/2126901160 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0cf015b3e0 msgr2=0x7f0cf01e1b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:11.920 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:11.916+0000 7f0cf6840640 1 --2- 172.21.15.57:0/2126901160 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0cf015b3e0 0x7f0cf01e1b30 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0ce800caa0 tx=0x7f0ce800cf60 comp rx=0 tx=0).stop 2023-10-19T12:44:11.921 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:11.916+0000 7f0cf6840640 1 -- 172.21.15.57:0/2126901160 shutdown_connections 2023-10-19T12:44:11.921 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:11.916+0000 7f0cf6840640 1 --2- 172.21.15.57:0/2126901160 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f0cd006ae40 0x7f0cd006d300 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:11.921 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:11.916+0000 7f0cf6840640 1 --2- 172.21.15.57:0/2126901160 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0cf015b3e0 0x7f0cf01e1b30 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:11.921 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:11.916+0000 7f0cf6840640 1 --2- 172.21.15.57:0/2126901160 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0cf015b000 0x7f0cf01e15f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:11.921 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:11.916+0000 7f0cf6840640 1 --2- 172.21.15.57:0/2126901160 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0cf00afdc0 0x7f0cf01e2c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:11.922 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:11.917+0000 7f0cf6840640 1 -- 172.21.15.57:0/2126901160 >> 172.21.15.57:0/2126901160 conn(0x7f0cf00d0cf0 msgr2=0x7f0cf00d10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:11.922 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:11.917+0000 7f0cf6840640 1 -- 172.21.15.57:0/2126901160 shutdown_connections 2023-10-19T12:44:11.922 INFO:teuthology.misc.health.smithi057.stderr:2023-10-19T12:44:11.917+0000 7f0cf6840640 1 -- 172.21.15.57:0/2126901160 wait complete. 2023-10-19T12:44:11.929 DEBUG:teuthology.misc:8 of 8 OSDs are up 2023-10-19T12:44:11.929 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-10-19T12:44:11.930 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-19T12:44:11.930 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-19T12:44:11.933 INFO:tasks.ceph.mds.a:Started 2023-10-19T12:44:11.933 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-19T12:44:11.933 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-19T12:44:11.935 INFO:tasks.ceph.mds.c:Started 2023-10-19T12:44:11.936 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-19T12:44:11.936 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-19T12:44:11.939 INFO:tasks.ceph.mds.b:Started 2023-10-19T12:44:11.940 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-19T12:44:11.940 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-19T12:44:11.942 INFO:tasks.ceph.mds.d:Started 2023-10-19T12:44:11.943 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2023-10-19T12:44:11.943 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T12:44:12.014 INFO:tasks.ceph.mds.a.smithi057.stdout:starting mds.a at 2023-10-19T12:44:12.015 INFO:tasks.ceph.mds.c.smithi057.stdout:starting mds.c at 2023-10-19T12:44:12.023 INFO:tasks.ceph.mds.b.smithi179.stdout:starting mds.b at 2023-10-19T12:44:12.024 INFO:tasks.ceph.mds.d.smithi179.stdout:starting mds.d at 2023-10-19T12:44:12.057 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.054+0000 7f0e906b4640 1 -- 172.21.15.57:0/2114281466 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0e880cf340 msgr2=0x7f0e880d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:12.058 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.054+0000 7f0e906b4640 1 --2- 172.21.15.57:0/2114281466 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0e880cf340 0x7f0e880d1910 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f0e8808a120 tx=0x7f0e8402bcf0 comp rx=0 tx=0).stop 2023-10-19T12:44:12.058 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.054+0000 7f0e906b4640 1 -- 172.21.15.57:0/2114281466 shutdown_connections 2023-10-19T12:44:12.058 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.054+0000 7f0e906b4640 1 --2- 172.21.15.57:0/2114281466 >> v2:172.21.15.179:3300/0 conn(0x7f0e8815b2a0 0x7f0e880cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:12.058 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.054+0000 7f0e906b4640 1 --2- 172.21.15.57:0/2114281466 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0e880cf340 0x7f0e880d1910 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:12.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.054+0000 7f0e906b4640 1 -- 172.21.15.57:0/2114281466 >> 172.21.15.57:0/2114281466 conn(0x7f0e880d0d60 msgr2=0x7f0e880d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:12.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.054+0000 7f0e906b4640 1 -- 172.21.15.57:0/2114281466 shutdown_connections 2023-10-19T12:44:12.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.054+0000 7f0e906b4640 1 -- 172.21.15.57:0/2114281466 wait complete. 2023-10-19T12:44:12.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.055+0000 7f0e906b4640 1 Processor -- start 2023-10-19T12:44:12.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.055+0000 7f0e906b4640 1 -- start start 2023-10-19T12:44:12.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.056+0000 7f0e906b4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0e880ceb40 0x7f0e880cad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:12.062 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.056+0000 7f0e906b4640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0e880cf340 0x7f0e88157610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:12.062 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.056+0000 7f0e906b4640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0e8815b2a0 0x7f0e880b5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:12.062 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.056+0000 7f0e8e450640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0e880ceb40 0x7f0e880cad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:12.062 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.056+0000 7f0e8e450640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0e880ceb40 0x7f0e880cad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40838/0 (socket says 172.21.15.57:40838) 2023-10-19T12:44:12.062 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.056+0000 7f0e8e450640 1 -- 172.21.15.57:0/2182398832 learned_addr learned my addr 172.21.15.57:0/2182398832 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:12.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.056+0000 7f0e906b4640 1 -- 172.21.15.57:0/2182398832 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e880b57a0 con 0x7f0e880ceb40 2023-10-19T12:44:12.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.056+0000 7f0e906b4640 1 -- 172.21.15.57:0/2182398832 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e880b59b0 con 0x7f0e8815b2a0 2023-10-19T12:44:12.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.056+0000 7f0e906b4640 1 -- 172.21.15.57:0/2182398832 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e880b5c90 con 0x7f0e880cf340 2023-10-19T12:44:12.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.056+0000 7f0e8dc4f640 1 --2- 172.21.15.57:0/2182398832 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0e880cf340 0x7f0e88157610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:12.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.057+0000 7f0e8ec51640 1 --2- 172.21.15.57:0/2182398832 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0e8815b2a0 0x7f0e880b5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:12.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.057+0000 7f0e8e450640 1 -- 172.21.15.57:0/2182398832 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0e880cf340 msgr2=0x7f0e88157610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:12.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.057+0000 7f0e8e450640 1 --2- 172.21.15.57:0/2182398832 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0e880cf340 0x7f0e88157610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:12.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.057+0000 7f0e8e450640 1 -- 172.21.15.57:0/2182398832 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0e8815b2a0 msgr2=0x7f0e880b5260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:12.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.057+0000 7f0e8e450640 1 --2- 172.21.15.57:0/2182398832 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0e8815b2a0 0x7f0e880b5260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:12.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.057+0000 7f0e8e450640 1 -- 172.21.15.57:0/2182398832 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e84009490 con 0x7f0e880ceb40 2023-10-19T12:44:12.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.057+0000 7f0e8e450640 1 --2- 172.21.15.57:0/2182398832 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0e880ceb40 0x7f0e880cad90 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0e84009780 tx=0x7f0e840038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:12.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.057+0000 7f0e7affd640 1 -- 172.21.15.57:0/2182398832 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e84003ea0 con 0x7f0e880ceb40 2023-10-19T12:44:12.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.057+0000 7f0e7affd640 1 -- 172.21.15.57:0/2182398832 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f0e84005020 con 0x7f0e880ceb40 2023-10-19T12:44:12.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.057+0000 7f0e906b4640 1 -- 172.21.15.57:0/2182398832 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e880b5f70 con 0x7f0e880ceb40 2023-10-19T12:44:12.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.057+0000 7f0e7affd640 1 -- 172.21.15.57:0/2182398832 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e84005410 con 0x7f0e880ceb40 2023-10-19T12:44:12.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.057+0000 7f0e906b4640 1 -- 172.21.15.57:0/2182398832 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e880b6530 con 0x7f0e880ceb40 2023-10-19T12:44:12.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.058+0000 7f0e906b4640 1 -- 172.21.15.57:0/2182398832 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e880690e0 con 0x7f0e880ceb40 2023-10-19T12:44:12.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.058+0000 7f0e7affd640 1 -- 172.21.15.57:0/2182398832 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f0e840047a0 con 0x7f0e880ceb40 2023-10-19T12:44:12.067 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.059+0000 7f0e7affd640 1 --2- 172.21.15.57:0/2182398832 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f0e6c06ae40 0x7f0e6c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:12.067 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.059+0000 7f0e7affd640 1 -- 172.21.15.57:0/2182398832 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f0e840ab570 con 0x7f0e880ceb40 2023-10-19T12:44:12.067 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.059+0000 7f0e8dc4f640 1 --2- 172.21.15.57:0/2182398832 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f0e6c06ae40 0x7f0e6c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:12.067 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.060+0000 7f0e8dc4f640 1 --2- 172.21.15.57:0/2182398832 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f0e6c06ae40 0x7f0e6c06d300 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0e880b1630 tx=0x7f0e7c005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:12.067 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.063+0000 7f0e7affd640 1 -- 172.21.15.57:0/2182398832 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0e84076a20 con 0x7f0e880ceb40 2023-10-19T12:44:12.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.241+0000 7f0e906b4640 1 -- 172.21.15.57:0/2182398832 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0e880cb980 con 0x7f0e880ceb40 2023-10-19T12:44:12.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.241+0000 7f0e7affd640 1 -- 172.21.15.57:0/2182398832 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+12862 (secure 0 0 0) 0x7f0e84039090 con 0x7f0e880ceb40 2023-10-19T12:44:12.245 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:44:12.245 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":17,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T12:44:09.423856+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T12:44:12.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.243+0000 7f0e906b4640 1 -- 172.21.15.57:0/2182398832 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f0e6c06ae40 msgr2=0x7f0e6c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:12.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.243+0000 7f0e906b4640 1 --2- 172.21.15.57:0/2182398832 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f0e6c06ae40 0x7f0e6c06d300 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0e880b1630 tx=0x7f0e7c005f90 comp rx=0 tx=0).stop 2023-10-19T12:44:12.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.243+0000 7f0e906b4640 1 -- 172.21.15.57:0/2182398832 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0e880ceb40 msgr2=0x7f0e880cad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:12.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.243+0000 7f0e906b4640 1 --2- 172.21.15.57:0/2182398832 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0e880ceb40 0x7f0e880cad90 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0e84009780 tx=0x7f0e840038a0 comp rx=0 tx=0).stop 2023-10-19T12:44:12.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.244+0000 7f0e906b4640 1 -- 172.21.15.57:0/2182398832 shutdown_connections 2023-10-19T12:44:12.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.244+0000 7f0e906b4640 1 --2- 172.21.15.57:0/2182398832 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f0e6c06ae40 0x7f0e6c06d300 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:12.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.244+0000 7f0e906b4640 1 --2- 172.21.15.57:0/2182398832 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0e8815b2a0 0x7f0e880b5260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:12.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.244+0000 7f0e906b4640 1 --2- 172.21.15.57:0/2182398832 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0e880cf340 0x7f0e88157610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:12.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.244+0000 7f0e906b4640 1 --2- 172.21.15.57:0/2182398832 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0e880ceb40 0x7f0e880cad90 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:12.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.244+0000 7f0e906b4640 1 -- 172.21.15.57:0/2182398832 >> 172.21.15.57:0/2182398832 conn(0x7f0e880d0d60 msgr2=0x7f0e880ad8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:12.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.244+0000 7f0e906b4640 1 -- 172.21.15.57:0/2182398832 shutdown_connections 2023-10-19T12:44:12.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.244+0000 7f0e906b4640 1 -- 172.21.15.57:0/2182398832 wait complete. 2023-10-19T12:44:12.257 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-19T12:44:07.371186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-19T12:44:12.258 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-19T12:44:12.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.366+0000 7f2739b39640 1 -- 172.21.15.57:0/644968413 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f273415b2a0 msgr2=0x7f27340d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:12.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.366+0000 7f2739b39640 1 --2- 172.21.15.57:0/644968413 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f273415b2a0 0x7f27340d1910 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2728005fd0 tx=0x7f272802bd90 comp rx=0 tx=0).stop 2023-10-19T12:44:12.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.366+0000 7f2739b39640 1 -- 172.21.15.57:0/644968413 shutdown_connections 2023-10-19T12:44:12.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.366+0000 7f2739b39640 1 --2- 172.21.15.57:0/644968413 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f27340ceb40 0x7f27340b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:12.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.366+0000 7f2739b39640 1 --2- 172.21.15.57:0/644968413 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f273415b2a0 0x7f27340d1910 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:12.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.366+0000 7f2739b39640 1 -- 172.21.15.57:0/644968413 >> 172.21.15.57:0/644968413 conn(0x7f27340d0d60 msgr2=0x7f27340d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:12.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.366+0000 7f2739b39640 1 -- 172.21.15.57:0/644968413 shutdown_connections 2023-10-19T12:44:12.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.366+0000 7f2739b39640 1 -- 172.21.15.57:0/644968413 wait complete. 2023-10-19T12:44:12.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.367+0000 7f2739b39640 1 Processor -- start 2023-10-19T12:44:12.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.368+0000 7f2739b39640 1 -- start start 2023-10-19T12:44:12.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.368+0000 7f2739b39640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f27340ceb40 0x7f27341582e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:12.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.368+0000 7f2739b39640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f27340cf340 0x7f273415b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:12.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.368+0000 7f2739b39640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f273415b2a0 0x7f27341e5db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:12.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.368+0000 7f2739b39640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27341e5990 con 0x7f27340ceb40 2023-10-19T12:44:12.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.368+0000 7f2739b39640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27341e4610 con 0x7f273415b2a0 2023-10-19T12:44:12.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.368+0000 7f2739b39640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27341e52d0 con 0x7f27340cf340 2023-10-19T12:44:12.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.368+0000 7f27337fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f27340ceb40 0x7f27341582e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:12.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.368+0000 7f27337fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f27340ceb40 0x7f27341582e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40856/0 (socket says 172.21.15.57:40856) 2023-10-19T12:44:12.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.368+0000 7f27337fe640 1 -- 172.21.15.57:0/1572940823 learned_addr learned my addr 172.21.15.57:0/1572940823 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:12.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.369+0000 7f2732ffd640 1 --2- 172.21.15.57:0/1572940823 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f27340cf340 0x7f273415b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:12.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.369+0000 7f27337fe640 1 -- 172.21.15.57:0/1572940823 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f27340cf340 msgr2=0x7f273415b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:12.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.369+0000 7f2733fff640 1 --2- 172.21.15.57:0/1572940823 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f273415b2a0 0x7f27341e5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:12.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.369+0000 7f27337fe640 1 --2- 172.21.15.57:0/1572940823 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f27340cf340 0x7f273415b6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:12.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.369+0000 7f27337fe640 1 -- 172.21.15.57:0/1572940823 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f273415b2a0 msgr2=0x7f27341e5db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:12.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.369+0000 7f27337fe640 1 --2- 172.21.15.57:0/1572940823 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f273415b2a0 0x7f27341e5db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:12.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.369+0000 7f27337fe640 1 -- 172.21.15.57:0/1572940823 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2728009560 con 0x7f27340ceb40 2023-10-19T12:44:12.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.369+0000 7f2732ffd640 1 --2- 172.21.15.57:0/1572940823 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f27340cf340 0x7f273415b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:44:12.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.369+0000 7f2733fff640 1 --2- 172.21.15.57:0/1572940823 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f273415b2a0 0x7f27341e5db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:44:12.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.369+0000 7f27337fe640 1 --2- 172.21.15.57:0/1572940823 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f27340ceb40 0x7f27341582e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f272400c400 tx=0x7f272400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:12.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.369+0000 7f271bfff640 1 -- 172.21.15.57:0/1572940823 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27240052d0 con 0x7f27340ceb40 2023-10-19T12:44:12.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.369+0000 7f271bfff640 1 -- 172.21.15.57:0/1572940823 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f2724007020 con 0x7f27340ceb40 2023-10-19T12:44:12.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.369+0000 7f2739b39640 1 -- 172.21.15.57:0/1572940823 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27340b0d50 con 0x7f27340ceb40 2023-10-19T12:44:12.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.369+0000 7f271bfff640 1 -- 172.21.15.57:0/1572940823 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2724007470 con 0x7f27340ceb40 2023-10-19T12:44:12.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.369+0000 7f2739b39640 1 -- 172.21.15.57:0/1572940823 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2734158820 con 0x7f27340ceb40 2023-10-19T12:44:12.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.370+0000 7f2739b39640 1 -- 172.21.15.57:0/1572940823 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27341e5130 con 0x7f27340ceb40 2023-10-19T12:44:12.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.371+0000 7f271bfff640 1 -- 172.21.15.57:0/1572940823 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f27240067d0 con 0x7f27340ceb40 2023-10-19T12:44:12.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.371+0000 7f271bfff640 1 --2- 172.21.15.57:0/1572940823 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f272006ae40 0x7f272006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:12.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.371+0000 7f271bfff640 1 -- 172.21.15.57:0/1572940823 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f27240888c0 con 0x7f27340ceb40 2023-10-19T12:44:12.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.371+0000 7f2732ffd640 1 --2- 172.21.15.57:0/1572940823 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f272006ae40 0x7f272006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:12.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.372+0000 7f2732ffd640 1 --2- 172.21.15.57:0/1572940823 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f272006ae40 0x7f272006d300 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f2728035000 tx=0x7f2728004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:12.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.375+0000 7f271bfff640 1 -- 172.21.15.57:0/1572940823 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f27341e5130 con 0x7f27340ceb40 2023-10-19T12:44:12.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.551+0000 7f2739b39640 1 -- 172.21.15.57:0/1572940823 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f27341e5130 con 0x7f27340ceb40 2023-10-19T12:44:12.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.552+0000 7f271bfff640 1 -- 172.21.15.57:0/1572940823 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v17) v1 ==== 93+0+10 (secure 0 0 0) 0x7f27341e5130 con 0x7f27340ceb40 2023-10-19T12:44:12.555 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-19T12:44:12.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.554+0000 7f2739b39640 1 -- 172.21.15.57:0/1572940823 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f272006ae40 msgr2=0x7f272006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:12.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.554+0000 7f2739b39640 1 --2- 172.21.15.57:0/1572940823 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f272006ae40 0x7f272006d300 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f2728035000 tx=0x7f2728004730 comp rx=0 tx=0).stop 2023-10-19T12:44:12.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.554+0000 7f2739b39640 1 -- 172.21.15.57:0/1572940823 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f27340ceb40 msgr2=0x7f27341582e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:12.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.554+0000 7f2739b39640 1 --2- 172.21.15.57:0/1572940823 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f27340ceb40 0x7f27341582e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f272400c400 tx=0x7f272400c8c0 comp rx=0 tx=0).stop 2023-10-19T12:44:12.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.554+0000 7f2739b39640 1 -- 172.21.15.57:0/1572940823 shutdown_connections 2023-10-19T12:44:12.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.554+0000 7f2739b39640 1 --2- 172.21.15.57:0/1572940823 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f272006ae40 0x7f272006d300 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:12.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.554+0000 7f2739b39640 1 --2- 172.21.15.57:0/1572940823 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f273415b2a0 0x7f27341e5db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:12.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.554+0000 7f2739b39640 1 --2- 172.21.15.57:0/1572940823 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f27340cf340 0x7f273415b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:12.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.554+0000 7f2739b39640 1 --2- 172.21.15.57:0/1572940823 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f27340ceb40 0x7f27341582e0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:12.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.554+0000 7f2739b39640 1 -- 172.21.15.57:0/1572940823 >> 172.21.15.57:0/1572940823 conn(0x7f27340d0d60 msgr2=0x7f273406b7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:12.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.554+0000 7f2739b39640 1 -- 172.21.15.57:0/1572940823 shutdown_connections 2023-10-19T12:44:12.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.554+0000 7f2739b39640 1 -- 172.21.15.57:0/1572940823 wait complete. 2023-10-19T12:44:12.567 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T12:44:12.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.672+0000 7f073656e640 1 -- 172.21.15.57:0/286079884 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f07300ceb40 msgr2=0x7f07300b0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:12.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.672+0000 7f073656e640 1 --2- 172.21.15.57:0/286079884 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f07300ceb40 0x7f07300b0650 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0728005fd0 tx=0x7f072802c050 comp rx=0 tx=0).stop 2023-10-19T12:44:12.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.672+0000 7f073656e640 1 -- 172.21.15.57:0/286079884 shutdown_connections 2023-10-19T12:44:12.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.672+0000 7f073656e640 1 --2- 172.21.15.57:0/286079884 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f073015b2a0 0x7f07300d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:12.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.672+0000 7f073656e640 1 --2- 172.21.15.57:0/286079884 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f07300ceb40 0x7f07300b0650 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:12.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.672+0000 7f073656e640 1 -- 172.21.15.57:0/286079884 >> 172.21.15.57:0/286079884 conn(0x7f07300d0d60 msgr2=0x7f07300d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:12.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.672+0000 7f073656e640 1 -- 172.21.15.57:0/286079884 shutdown_connections 2023-10-19T12:44:12.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.672+0000 7f073656e640 1 -- 172.21.15.57:0/286079884 wait complete. 2023-10-19T12:44:12.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.673+0000 7f073656e640 1 Processor -- start 2023-10-19T12:44:12.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.673+0000 7f073656e640 1 -- start start 2023-10-19T12:44:12.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.674+0000 7f073656e640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f07300ceb40 0x7f07300b38e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:12.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.674+0000 7f073656e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f07300cf340 0x7f07300c6b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:12.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.674+0000 7f073656e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f073015b2a0 0x7f07300b5270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:12.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.674+0000 7f073656e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07300b3e20 con 0x7f073015b2a0 2023-10-19T12:44:12.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.674+0000 7f073656e640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07300c70d0 con 0x7f07300cf340 2023-10-19T12:44:12.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.674+0000 7f073656e640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07300b6020 con 0x7f07300ceb40 2023-10-19T12:44:12.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.674+0000 7f0734b0b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f073015b2a0 0x7f07300b5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:12.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.674+0000 7f0734b0b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f073015b2a0 0x7f07300b5270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40876/0 (socket says 172.21.15.57:40876) 2023-10-19T12:44:12.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.674+0000 7f0734b0b640 1 -- 172.21.15.57:0/4177084612 learned_addr learned my addr 172.21.15.57:0/4177084612 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:12.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.674+0000 7f072ffff640 1 --2- 172.21.15.57:0/4177084612 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f07300ceb40 0x7f07300b38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:12.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.675+0000 7f072f7fe640 1 --2- 172.21.15.57:0/4177084612 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f07300cf340 0x7f07300c6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:12.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.675+0000 7f0734b0b640 1 -- 172.21.15.57:0/4177084612 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f07300ceb40 msgr2=0x7f07300b38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:12.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.675+0000 7f0734b0b640 1 --2- 172.21.15.57:0/4177084612 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f07300ceb40 0x7f07300b38e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:12.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.675+0000 7f0734b0b640 1 -- 172.21.15.57:0/4177084612 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f07300cf340 msgr2=0x7f07300c6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:12.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.675+0000 7f0734b0b640 1 --2- 172.21.15.57:0/4177084612 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f07300cf340 0x7f07300c6b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:12.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.675+0000 7f0734b0b640 1 -- 172.21.15.57:0/4177084612 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0728009840 con 0x7f073015b2a0 2023-10-19T12:44:12.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.675+0000 7f072f7fe640 1 --2- 172.21.15.57:0/4177084612 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f07300cf340 0x7f07300c6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:44:12.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.675+0000 7f0734b0b640 1 --2- 172.21.15.57:0/4177084612 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f073015b2a0 0x7f07300b5270 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0728009b30 tx=0x7f07280042e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:12.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.675+0000 7f072cff9640 1 -- 172.21.15.57:0/4177084612 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f072803e460 con 0x7f073015b2a0 2023-10-19T12:44:12.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.675+0000 7f072cff9640 1 -- 172.21.15.57:0/4177084612 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f072803e6b0 con 0x7f073015b2a0 2023-10-19T12:44:12.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.675+0000 7f072cff9640 1 -- 172.21.15.57:0/4177084612 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f072803ea40 con 0x7f073015b2a0 2023-10-19T12:44:12.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.675+0000 7f073656e640 1 -- 172.21.15.57:0/4177084612 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07300b6300 con 0x7f073015b2a0 2023-10-19T12:44:12.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.675+0000 7f073656e640 1 -- 172.21.15.57:0/4177084612 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07300b68c0 con 0x7f073015b2a0 2023-10-19T12:44:12.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.676+0000 7f073656e640 1 -- 172.21.15.57:0/4177084612 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07300677b0 con 0x7f073015b2a0 2023-10-19T12:44:12.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.676+0000 7f072cff9640 1 -- 172.21.15.57:0/4177084612 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f072803eba0 con 0x7f073015b2a0 2023-10-19T12:44:12.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.677+0000 7f072cff9640 1 --2- 172.21.15.57:0/4177084612 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f071006ae70 0x7f071006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:12.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.677+0000 7f072cff9640 1 -- 172.21.15.57:0/4177084612 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f07280ac430 con 0x7f073015b2a0 2023-10-19T12:44:12.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.677+0000 7f072ffff640 1 --2- 172.21.15.57:0/4177084612 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f071006ae70 0x7f071006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:12.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.678+0000 7f072ffff640 1 --2- 172.21.15.57:0/4177084612 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f071006ae70 0x7f071006d330 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0718000f60 tx=0x7f0718009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:12.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.681+0000 7f072cff9640 1 -- 172.21.15.57:0/4177084612 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f07280778c0 con 0x7f073015b2a0 2023-10-19T12:44:12.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.892+0000 7f073656e640 1 -- 172.21.15.57:0/4177084612 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f07300d04f0 con 0x7f073015b2a0 2023-10-19T12:44:12.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.893+0000 7f072cff9640 1 -- 172.21.15.57:0/4177084612 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1324 (secure 0 0 0) 0x7f0728042090 con 0x7f073015b2a0 2023-10-19T12:44:12.896 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:44:12.897 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4357,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/4228712742","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":4228712742},{"type":"v1","addr":"172.21.15.57:6835","nonce":4228712742}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2023-10-19T12:44:12.898 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 2 2023-10-19T12:44:12.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.895+0000 7f073656e640 1 -- 172.21.15.57:0/4177084612 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f071006ae70 msgr2=0x7f071006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:12.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.895+0000 7f073656e640 1 --2- 172.21.15.57:0/4177084612 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f071006ae70 0x7f071006d330 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0718000f60 tx=0x7f0718009040 comp rx=0 tx=0).stop 2023-10-19T12:44:12.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.895+0000 7f073656e640 1 -- 172.21.15.57:0/4177084612 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f073015b2a0 msgr2=0x7f07300b5270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:12.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.895+0000 7f073656e640 1 --2- 172.21.15.57:0/4177084612 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f073015b2a0 0x7f07300b5270 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0728009b30 tx=0x7f07280042e0 comp rx=0 tx=0).stop 2023-10-19T12:44:12.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.895+0000 7f073656e640 1 -- 172.21.15.57:0/4177084612 shutdown_connections 2023-10-19T12:44:12.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.895+0000 7f073656e640 1 --2- 172.21.15.57:0/4177084612 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f071006ae70 0x7f071006d330 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:12.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.895+0000 7f073656e640 1 --2- 172.21.15.57:0/4177084612 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f073015b2a0 0x7f07300b5270 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:12.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.895+0000 7f073656e640 1 --2- 172.21.15.57:0/4177084612 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f07300cf340 0x7f07300c6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:12.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.895+0000 7f073656e640 1 --2- 172.21.15.57:0/4177084612 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f07300ceb40 0x7f07300b38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:12.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.895+0000 7f073656e640 1 -- 172.21.15.57:0/4177084612 >> 172.21.15.57:0/4177084612 conn(0x7f07300d0d60 msgr2=0x7f07301512b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:12.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.895+0000 7f073656e640 1 -- 172.21.15.57:0/4177084612 shutdown_connections 2023-10-19T12:44:12.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:12.895+0000 7f073656e640 1 -- 172.21.15.57:0/4177084612 wait complete. 2023-10-19T12:44:13.359 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T12:44:13.362 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T12:44:14.908 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T12:44:15.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.027+0000 7f40ba044640 1 -- 172.21.15.57:0/3834960072 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f40b40b24d0 msgr2=0x7f40b40b28b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:15.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.027+0000 7f40ba044640 1 --2- 172.21.15.57:0/3834960072 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f40b40b24d0 0x7f40b40b28b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f40a4009830 tx=0x7f40a402bd60 comp rx=0 tx=0).stop 2023-10-19T12:44:15.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.027+0000 7f40ba044640 1 -- 172.21.15.57:0/3834960072 shutdown_connections 2023-10-19T12:44:15.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.027+0000 7f40ba044640 1 --2- 172.21.15.57:0/3834960072 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f40b40b24d0 0x7f40b40b28b0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:15.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.027+0000 7f40ba044640 1 -- 172.21.15.57:0/3834960072 >> 172.21.15.57:0/3834960072 conn(0x7f40b40b3f00 msgr2=0x7f40b40b4310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:15.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.027+0000 7f40ba044640 1 -- 172.21.15.57:0/3834960072 shutdown_connections 2023-10-19T12:44:15.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.027+0000 7f40ba044640 1 -- 172.21.15.57:0/3834960072 wait complete. 2023-10-19T12:44:15.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.028+0000 7f40ba044640 1 Processor -- start 2023-10-19T12:44:15.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.028+0000 7f40ba044640 1 -- start start 2023-10-19T12:44:15.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.029+0000 7f40ba044640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f40b40acd40 0x7f40b414db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:15.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.029+0000 7f40ba044640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f40b40b24d0 0x7f40b40ae650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:15.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.029+0000 7f40ba044640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f40b40b3070 0x7f40b4148910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:15.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.029+0000 7f40ba044640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40b414e040 con 0x7f40b40b3070 2023-10-19T12:44:15.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.029+0000 7f40ba044640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40b40af6d0 con 0x7f40b40b24d0 2023-10-19T12:44:15.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.029+0000 7f40ba044640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40b40aeb90 con 0x7f40b40acd40 2023-10-19T12:44:15.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.029+0000 7f40b3fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f40b40b3070 0x7f40b4148910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:15.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.029+0000 7f40b3fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f40b40b3070 0x7f40b4148910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40890/0 (socket says 172.21.15.57:40890) 2023-10-19T12:44:15.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.029+0000 7f40b3fff640 1 -- 172.21.15.57:0/1115425713 learned_addr learned my addr 172.21.15.57:0/1115425713 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:15.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.029+0000 7f40b2ffd640 1 --2- 172.21.15.57:0/1115425713 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f40b40b24d0 0x7f40b40ae650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:15.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.029+0000 7f40b37fe640 1 --2- 172.21.15.57:0/1115425713 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f40b40acd40 0x7f40b414db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:15.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.029+0000 7f40b3fff640 1 -- 172.21.15.57:0/1115425713 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f40b40acd40 msgr2=0x7f40b414db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:15.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.029+0000 7f40b3fff640 1 --2- 172.21.15.57:0/1115425713 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f40b40acd40 0x7f40b414db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:15.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.029+0000 7f40b3fff640 1 -- 172.21.15.57:0/1115425713 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f40b40b24d0 msgr2=0x7f40b40ae650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:15.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.029+0000 7f40b3fff640 1 --2- 172.21.15.57:0/1115425713 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f40b40b24d0 0x7f40b40ae650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:15.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.029+0000 7f40b3fff640 1 -- 172.21.15.57:0/1115425713 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40a4009490 con 0x7f40b40b3070 2023-10-19T12:44:15.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.029+0000 7f40b2ffd640 1 --2- 172.21.15.57:0/1115425713 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f40b40b24d0 0x7f40b40ae650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:44:15.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.030+0000 7f40b3fff640 1 --2- 172.21.15.57:0/1115425713 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f40b40b3070 0x7f40b4148910 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f40ac001560 tx=0x7f40ac00cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:15.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.036+0000 7f409bfff640 1 -- 172.21.15.57:0/1115425713 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40ac003da0 con 0x7f40b40b3070 2023-10-19T12:44:15.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.037+0000 7f409bfff640 1 -- 172.21.15.57:0/1115425713 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f40ac005070 con 0x7f40b40b3070 2023-10-19T12:44:15.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.037+0000 7f40ba044640 1 -- 172.21.15.57:0/1115425713 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40b4148e50 con 0x7f40b40b3070 2023-10-19T12:44:15.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.037+0000 7f409bfff640 1 -- 172.21.15.57:0/1115425713 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40ac005460 con 0x7f40b40b3070 2023-10-19T12:44:15.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.037+0000 7f40ba044640 1 -- 172.21.15.57:0/1115425713 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40b40af110 con 0x7f40b40b3070 2023-10-19T12:44:15.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.038+0000 7f40ba044640 1 -- 172.21.15.57:0/1115425713 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40b4159cc0 con 0x7f40b40b3070 2023-10-19T12:44:15.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.038+0000 7f409bfff640 1 -- 172.21.15.57:0/1115425713 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f40ac0055c0 con 0x7f40b40b3070 2023-10-19T12:44:15.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.038+0000 7f409bfff640 1 --2- 172.21.15.57:0/1115425713 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f40a006ae40 0x7f40a006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:15.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.039+0000 7f409bfff640 1 -- 172.21.15.57:0/1115425713 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f40ac088e00 con 0x7f40b40b3070 2023-10-19T12:44:15.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.039+0000 7f40b37fe640 1 --2- 172.21.15.57:0/1115425713 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f40a006ae40 0x7f40a006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:15.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.039+0000 7f40b37fe640 1 --2- 172.21.15.57:0/1115425713 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f40a006ae40 0x7f40a006d300 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f40a4036000 tx=0x7f40a4004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:15.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.043+0000 7f409bfff640 1 -- 172.21.15.57:0/1115425713 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f40ac054210 con 0x7f40b40b3070 2023-10-19T12:44:15.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.253+0000 7f40ba044640 1 -- 172.21.15.57:0/1115425713 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f40b40690e0 con 0x7f40b40b3070 2023-10-19T12:44:15.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.253+0000 7f409bfff640 1 -- 172.21.15.57:0/1115425713 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+3691 (secure 0 0 0) 0x7f40ac017020 con 0x7f40b40b3070 2023-10-19T12:44:15.257 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:44:15.257 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4357,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/4228712742","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":4228712742},{"type":"v1","addr":"172.21.15.57:6835","nonce":4228712742}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4360,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/971892630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":971892630},{"type":"v1","addr":"172.21.15.57:6834","nonce":971892630}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4372,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/104495983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":104495983},{"type":"v1","addr":"172.21.15.179:6836","nonce":104495983}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4374,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/622550365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":622550365},{"type":"v1","addr":"172.21.15.179:6837","nonce":622550365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2023-10-19T12:44:15.258 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 3 2023-10-19T12:44:15.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.255+0000 7f40ba044640 1 -- 172.21.15.57:0/1115425713 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f40a006ae40 msgr2=0x7f40a006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:15.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.255+0000 7f40ba044640 1 --2- 172.21.15.57:0/1115425713 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f40a006ae40 0x7f40a006d300 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f40a4036000 tx=0x7f40a4004730 comp rx=0 tx=0).stop 2023-10-19T12:44:15.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.255+0000 7f40ba044640 1 -- 172.21.15.57:0/1115425713 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f40b40b3070 msgr2=0x7f40b4148910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:15.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.255+0000 7f40ba044640 1 --2- 172.21.15.57:0/1115425713 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f40b40b3070 0x7f40b4148910 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f40ac001560 tx=0x7f40ac00cc70 comp rx=0 tx=0).stop 2023-10-19T12:44:15.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.255+0000 7f40ba044640 1 -- 172.21.15.57:0/1115425713 shutdown_connections 2023-10-19T12:44:15.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.255+0000 7f40ba044640 1 --2- 172.21.15.57:0/1115425713 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f40a006ae40 0x7f40a006d300 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:15.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.255+0000 7f40ba044640 1 --2- 172.21.15.57:0/1115425713 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f40b40b3070 0x7f40b4148910 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:15.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.255+0000 7f40ba044640 1 --2- 172.21.15.57:0/1115425713 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f40b40b24d0 0x7f40b40ae650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:15.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.255+0000 7f40ba044640 1 --2- 172.21.15.57:0/1115425713 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f40b40acd40 0x7f40b414db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:15.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.255+0000 7f40ba044640 1 -- 172.21.15.57:0/1115425713 >> 172.21.15.57:0/1115425713 conn(0x7f40b40b3f00 msgr2=0x7f40b415bcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:15.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.255+0000 7f40ba044640 1 -- 172.21.15.57:0/1115425713 shutdown_connections 2023-10-19T12:44:15.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.255+0000 7f40ba044640 1 -- 172.21.15.57:0/1115425713 wait complete. 2023-10-19T12:44:15.269 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T12:44:15.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.374+0000 7f8c7e039640 1 -- 172.21.15.57:0/496971390 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8c7806d2d0 msgr2=0x7f8c7806d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:15.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.374+0000 7f8c7e039640 1 --2- 172.21.15.57:0/496971390 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8c7806d2d0 0x7f8c7806d6b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8c68009830 tx=0x7f8c6802bd60 comp rx=0 tx=0).stop 2023-10-19T12:44:15.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.374+0000 7f8c7e039640 1 -- 172.21.15.57:0/496971390 shutdown_connections 2023-10-19T12:44:15.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.374+0000 7f8c7e039640 1 --2- 172.21.15.57:0/496971390 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8c780ad370 0x7f8c780ad800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:15.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.374+0000 7f8c7e039640 1 --2- 172.21.15.57:0/496971390 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8c7806d2d0 0x7f8c7806d6b0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:15.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.374+0000 7f8c7e039640 1 -- 172.21.15.57:0/496971390 >> 172.21.15.57:0/496971390 conn(0x7f8c780b0670 msgr2=0x7f8c780b0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:15.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.374+0000 7f8c7e039640 1 -- 172.21.15.57:0/496971390 shutdown_connections 2023-10-19T12:44:15.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.374+0000 7f8c7e039640 1 -- 172.21.15.57:0/496971390 wait complete. 2023-10-19T12:44:15.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.375+0000 7f8c7e039640 1 Processor -- start 2023-10-19T12:44:15.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.376+0000 7f8c7e039640 1 -- start start 2023-10-19T12:44:15.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.376+0000 7f8c7e039640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8c7806d2d0 0x7f8c781eef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:15.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.376+0000 7f8c7e039640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8c7806f0a0 0x7f8c780c81b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:15.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.376+0000 7f8c7e039640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8c780ad370 0x7f8c780c86f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:15.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.376+0000 7f8c7e039640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c780c99c0 con 0x7f8c780ad370 2023-10-19T12:44:15.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.376+0000 7f8c7e039640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c780c9ca0 con 0x7f8c7806d2d0 2023-10-19T12:44:15.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.376+0000 7f8c7e039640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c780c9f80 con 0x7f8c7806f0a0 2023-10-19T12:44:15.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.376+0000 7f8c77fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8c780ad370 0x7f8c780c86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:15.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.377+0000 7f8c77fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8c780ad370 0x7f8c780c86f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40900/0 (socket says 172.21.15.57:40900) 2023-10-19T12:44:15.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.377+0000 7f8c77fff640 1 -- 172.21.15.57:0/3664917706 learned_addr learned my addr 172.21.15.57:0/3664917706 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:15.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.377+0000 7f8c777fe640 1 --2- 172.21.15.57:0/3664917706 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8c7806d2d0 0x7f8c781eef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:15.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.377+0000 7f8c76ffd640 1 --2- 172.21.15.57:0/3664917706 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8c7806f0a0 0x7f8c780c81b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:15.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.377+0000 7f8c77fff640 1 -- 172.21.15.57:0/3664917706 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8c7806f0a0 msgr2=0x7f8c780c81b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:15.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.377+0000 7f8c77fff640 1 --2- 172.21.15.57:0/3664917706 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8c7806f0a0 0x7f8c780c81b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:15.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.377+0000 7f8c77fff640 1 -- 172.21.15.57:0/3664917706 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8c7806d2d0 msgr2=0x7f8c781eef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:15.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.377+0000 7f8c77fff640 1 --2- 172.21.15.57:0/3664917706 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8c7806d2d0 0x7f8c781eef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:15.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.377+0000 7f8c77fff640 1 -- 172.21.15.57:0/3664917706 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c70009840 con 0x7f8c780ad370 2023-10-19T12:44:15.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.377+0000 7f8c77fff640 1 --2- 172.21.15.57:0/3664917706 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8c780ad370 0x7f8c780c86f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8c7000ec20 tx=0x7f8c7000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:15.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.377+0000 7f8c5ffff640 1 -- 172.21.15.57:0/3664917706 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c70009aa0 con 0x7f8c780ad370 2023-10-19T12:44:15.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.377+0000 7f8c5ffff640 1 -- 172.21.15.57:0/3664917706 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f8c70003c60 con 0x7f8c780ad370 2023-10-19T12:44:15.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.377+0000 7f8c7e039640 1 -- 172.21.15.57:0/3664917706 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c68009490 con 0x7f8c780ad370 2023-10-19T12:44:15.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.377+0000 7f8c5ffff640 1 -- 172.21.15.57:0/3664917706 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c70004020 con 0x7f8c780ad370 2023-10-19T12:44:15.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.377+0000 7f8c7e039640 1 -- 172.21.15.57:0/3664917706 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c780cd310 con 0x7f8c780ad370 2023-10-19T12:44:15.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.378+0000 7f8c7e039640 1 -- 172.21.15.57:0/3664917706 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c780ce1d0 con 0x7f8c780ad370 2023-10-19T12:44:15.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.379+0000 7f8c5ffff640 1 -- 172.21.15.57:0/3664917706 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f8c700041a0 con 0x7f8c780ad370 2023-10-19T12:44:15.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.379+0000 7f8c5ffff640 1 --2- 172.21.15.57:0/3664917706 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f8c6406ae40 0x7f8c6406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:15.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.379+0000 7f8c5ffff640 1 -- 172.21.15.57:0/3664917706 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f8c70014070 con 0x7f8c780ad370 2023-10-19T12:44:15.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.379+0000 7f8c777fe640 1 --2- 172.21.15.57:0/3664917706 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f8c6406ae40 0x7f8c6406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:15.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.380+0000 7f8c777fe640 1 --2- 172.21.15.57:0/3664917706 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f8c6406ae40 0x7f8c6406d300 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8c68036000 tx=0x7f8c68004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:15.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.383+0000 7f8c5ffff640 1 -- 172.21.15.57:0/3664917706 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8c700556c0 con 0x7f8c780ad370 2023-10-19T12:44:15.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.563+0000 7f8c7e039640 1 -- 172.21.15.57:0/3664917706 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8c780690e0 con 0x7f8c780ad370 2023-10-19T12:44:15.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.563+0000 7f8c5ffff640 1 -- 172.21.15.57:0/3664917706 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+12862 (secure 0 0 0) 0x7f8c70058ce0 con 0x7f8c780ad370 2023-10-19T12:44:15.567 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:44:15.567 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":17,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T12:44:09.423856+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T12:44:15.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.565+0000 7f8c7e039640 1 -- 172.21.15.57:0/3664917706 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f8c6406ae40 msgr2=0x7f8c6406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:15.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.565+0000 7f8c7e039640 1 --2- 172.21.15.57:0/3664917706 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f8c6406ae40 0x7f8c6406d300 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8c68036000 tx=0x7f8c68004730 comp rx=0 tx=0).stop 2023-10-19T12:44:15.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.565+0000 7f8c7e039640 1 -- 172.21.15.57:0/3664917706 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8c780ad370 msgr2=0x7f8c780c86f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:15.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.565+0000 7f8c7e039640 1 --2- 172.21.15.57:0/3664917706 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8c780ad370 0x7f8c780c86f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8c7000ec20 tx=0x7f8c7000c6a0 comp rx=0 tx=0).stop 2023-10-19T12:44:15.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.565+0000 7f8c7e039640 1 -- 172.21.15.57:0/3664917706 shutdown_connections 2023-10-19T12:44:15.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.565+0000 7f8c7e039640 1 --2- 172.21.15.57:0/3664917706 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f8c6406ae40 0x7f8c6406d300 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:15.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.565+0000 7f8c7e039640 1 --2- 172.21.15.57:0/3664917706 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8c780ad370 0x7f8c780c86f0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:15.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.565+0000 7f8c7e039640 1 --2- 172.21.15.57:0/3664917706 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8c7806f0a0 0x7f8c780c81b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:15.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.565+0000 7f8c7e039640 1 --2- 172.21.15.57:0/3664917706 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8c7806d2d0 0x7f8c781eef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:15.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.565+0000 7f8c7e039640 1 -- 172.21.15.57:0/3664917706 >> 172.21.15.57:0/3664917706 conn(0x7f8c780b0670 msgr2=0x7f8c78150020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:15.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.565+0000 7f8c7e039640 1 -- 172.21.15.57:0/3664917706 shutdown_connections 2023-10-19T12:44:15.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.565+0000 7f8c7e039640 1 -- 172.21.15.57:0/3664917706 wait complete. 2023-10-19T12:44:15.579 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-19T12:44:07.371186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-19T12:44:15.580 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-19T12:44:15.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.688+0000 7f2cf23d0640 1 -- 172.21.15.57:0/502149063 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2cec15b3e0 msgr2=0x7f2cec0cf2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:15.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.688+0000 7f2cf23d0640 1 --2- 172.21.15.57:0/502149063 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2cec15b3e0 0x7f2cec0cf2d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2cdc005fd0 tx=0x7f2cdc02bd90 comp rx=0 tx=0).stop 2023-10-19T12:44:15.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.688+0000 7f2cf23d0640 1 -- 172.21.15.57:0/502149063 shutdown_connections 2023-10-19T12:44:15.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.688+0000 7f2cf23d0640 1 --2- 172.21.15.57:0/502149063 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2cec15b3e0 0x7f2cec0cf2d0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:15.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.688+0000 7f2cf23d0640 1 --2- 172.21.15.57:0/502149063 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2cec15b000 0x7f2cec0d1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:15.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.688+0000 7f2cf23d0640 1 -- 172.21.15.57:0/502149063 >> 172.21.15.57:0/502149063 conn(0x7f2cec0d0cf0 msgr2=0x7f2cec0d1100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:15.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.688+0000 7f2cf23d0640 1 -- 172.21.15.57:0/502149063 shutdown_connections 2023-10-19T12:44:15.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.688+0000 7f2cf23d0640 1 -- 172.21.15.57:0/502149063 wait complete. 2023-10-19T12:44:15.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.689+0000 7f2cf23d0640 1 Processor -- start 2023-10-19T12:44:15.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.689+0000 7f2cf23d0640 1 -- start start 2023-10-19T12:44:15.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.690+0000 7f2cf23d0640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2cec0afdc0 0x7f2cec158290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:15.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.690+0000 7f2cf23d0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2cec15b000 0x7f2cec06b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:15.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.690+0000 7f2cf23d0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2cec15b3e0 0x7f2cec06b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:15.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.690+0000 7f2cf23d0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cec1587d0 con 0x7f2cec15b000 2023-10-19T12:44:15.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.690+0000 7f2cf23d0640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cec0c8ab0 con 0x7f2cec15b3e0 2023-10-19T12:44:15.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.690+0000 7f2cf23d0640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cec0c8d90 con 0x7f2cec0afdc0 2023-10-19T12:44:15.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.690+0000 7f2ceb7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2cec15b000 0x7f2cec06b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:15.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.690+0000 7f2ceb7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2cec15b000 0x7f2cec06b0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40908/0 (socket says 172.21.15.57:40908) 2023-10-19T12:44:15.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.690+0000 7f2ceb7fe640 1 -- 172.21.15.57:0/2108819968 learned_addr learned my addr 172.21.15.57:0/2108819968 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:15.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.690+0000 7f2cebfff640 1 --2- 172.21.15.57:0/2108819968 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2cec0afdc0 0x7f2cec158290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:15.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.690+0000 7f2cf096d640 1 --2- 172.21.15.57:0/2108819968 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2cec15b3e0 0x7f2cec06b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:15.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.690+0000 7f2ceb7fe640 1 -- 172.21.15.57:0/2108819968 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2cec0afdc0 msgr2=0x7f2cec158290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:15.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.690+0000 7f2ceb7fe640 1 --2- 172.21.15.57:0/2108819968 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2cec0afdc0 0x7f2cec158290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:15.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.690+0000 7f2ceb7fe640 1 -- 172.21.15.57:0/2108819968 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2cec15b3e0 msgr2=0x7f2cec06b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:15.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.690+0000 7f2ceb7fe640 1 --2- 172.21.15.57:0/2108819968 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2cec15b3e0 0x7f2cec06b5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:15.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.690+0000 7f2ceb7fe640 1 -- 172.21.15.57:0/2108819968 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cdc009560 con 0x7f2cec15b000 2023-10-19T12:44:15.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.690+0000 7f2cf096d640 1 --2- 172.21.15.57:0/2108819968 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2cec15b3e0 0x7f2cec06b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:44:15.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.690+0000 7f2ceb7fe640 1 --2- 172.21.15.57:0/2108819968 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2cec15b000 0x7f2cec06b0a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2cdc036040 tx=0x7f2cdc02db30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:15.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.691+0000 7f2ce8ff9640 1 -- 172.21.15.57:0/2108819968 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cdc0035f0 con 0x7f2cec15b000 2023-10-19T12:44:15.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.691+0000 7f2ce8ff9640 1 -- 172.21.15.57:0/2108819968 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f2cdc003840 con 0x7f2cec15b000 2023-10-19T12:44:15.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.691+0000 7f2cf23d0640 1 -- 172.21.15.57:0/2108819968 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cec0c9070 con 0x7f2cec15b000 2023-10-19T12:44:15.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.691+0000 7f2ce8ff9640 1 -- 172.21.15.57:0/2108819968 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cdc003bf0 con 0x7f2cec15b000 2023-10-19T12:44:15.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.691+0000 7f2cf23d0640 1 -- 172.21.15.57:0/2108819968 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cec0c9630 con 0x7f2cec15b000 2023-10-19T12:44:15.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.692+0000 7f2cf23d0640 1 -- 172.21.15.57:0/2108819968 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cec0c8ed0 con 0x7f2cec15b000 2023-10-19T12:44:15.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.692+0000 7f2ce8ff9640 1 -- 172.21.15.57:0/2108819968 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f2cdc003d50 con 0x7f2cec15b000 2023-10-19T12:44:15.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.692+0000 7f2ce8ff9640 1 --2- 172.21.15.57:0/2108819968 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f2ccc06ae40 0x7f2ccc06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:15.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.692+0000 7f2ce8ff9640 1 -- 172.21.15.57:0/2108819968 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f2cdc0abe30 con 0x7f2cec15b000 2023-10-19T12:44:15.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.693+0000 7f2cebfff640 1 --2- 172.21.15.57:0/2108819968 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f2ccc06ae40 0x7f2ccc06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:15.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.693+0000 7f2cebfff640 1 --2- 172.21.15.57:0/2108819968 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f2ccc06ae40 0x7f2ccc06d300 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2cd4005fd0 tx=0x7f2cd4005850 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:15.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.696+0000 7f2ce8ff9640 1 -- 172.21.15.57:0/2108819968 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2cec0c8ed0 con 0x7f2cec15b000 2023-10-19T12:44:15.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.871+0000 7f2cf23d0640 1 -- 172.21.15.57:0/2108819968 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2cec0c8ed0 con 0x7f2cec15b000 2023-10-19T12:44:15.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.871+0000 7f2ce8ff9640 1 -- 172.21.15.57:0/2108819968 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v17) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2cec0c8ed0 con 0x7f2cec15b000 2023-10-19T12:44:15.875 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-19T12:44:15.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.873+0000 7f2cf23d0640 1 -- 172.21.15.57:0/2108819968 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f2ccc06ae40 msgr2=0x7f2ccc06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:15.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.873+0000 7f2cf23d0640 1 --2- 172.21.15.57:0/2108819968 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f2ccc06ae40 0x7f2ccc06d300 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2cd4005fd0 tx=0x7f2cd4005850 comp rx=0 tx=0).stop 2023-10-19T12:44:15.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.873+0000 7f2cf23d0640 1 -- 172.21.15.57:0/2108819968 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2cec15b000 msgr2=0x7f2cec06b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:15.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.873+0000 7f2cf23d0640 1 --2- 172.21.15.57:0/2108819968 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2cec15b000 0x7f2cec06b0a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2cdc036040 tx=0x7f2cdc02db30 comp rx=0 tx=0).stop 2023-10-19T12:44:15.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.873+0000 7f2cf23d0640 1 -- 172.21.15.57:0/2108819968 shutdown_connections 2023-10-19T12:44:15.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.873+0000 7f2cf23d0640 1 --2- 172.21.15.57:0/2108819968 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f2ccc06ae40 0x7f2ccc06d300 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:15.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.873+0000 7f2cf23d0640 1 --2- 172.21.15.57:0/2108819968 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2cec15b3e0 0x7f2cec06b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:15.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.873+0000 7f2cf23d0640 1 --2- 172.21.15.57:0/2108819968 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2cec15b000 0x7f2cec06b0a0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:15.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.873+0000 7f2cf23d0640 1 --2- 172.21.15.57:0/2108819968 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2cec0afdc0 0x7f2cec158290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:15.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.873+0000 7f2cf23d0640 1 -- 172.21.15.57:0/2108819968 >> 172.21.15.57:0/2108819968 conn(0x7f2cec0d0cf0 msgr2=0x7f2cec15a010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:15.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.873+0000 7f2cf23d0640 1 -- 172.21.15.57:0/2108819968 shutdown_connections 2023-10-19T12:44:15.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.873+0000 7f2cf23d0640 1 -- 172.21.15.57:0/2108819968 wait complete. 2023-10-19T12:44:15.886 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-19T12:44:15.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.991+0000 7f7962d97640 1 -- 172.21.15.57:0/1011083555 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f795c0cd9e0 msgr2=0x7f795c0cde70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:15.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.991+0000 7f7962d97640 1 --2- 172.21.15.57:0/1011083555 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f795c0cd9e0 0x7f795c0cde70 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f795c073460 tx=0x7f795402c250 comp rx=0 tx=0).stop 2023-10-19T12:44:15.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.991+0000 7f7962d97640 1 -- 172.21.15.57:0/1011083555 shutdown_connections 2023-10-19T12:44:15.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.991+0000 7f7962d97640 1 --2- 172.21.15.57:0/1011083555 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f795c0cd9e0 0x7f795c0cde70 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:15.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.991+0000 7f7962d97640 1 --2- 172.21.15.57:0/1011083555 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f795c0cd1c0 0x7f795c15c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:15.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.991+0000 7f7962d97640 1 --2- 172.21.15.57:0/1011083555 >> v2:172.21.15.179:3300/0 conn(0x7f795c158910 0x7f795c158cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:15.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.991+0000 7f7962d97640 1 -- 172.21.15.57:0/1011083555 >> 172.21.15.57:0/1011083555 conn(0x7f795c0af7a0 msgr2=0x7f795c0afb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:15.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.992+0000 7f7962d97640 1 -- 172.21.15.57:0/1011083555 shutdown_connections 2023-10-19T12:44:15.997 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.992+0000 7f7962d97640 1 -- 172.21.15.57:0/1011083555 wait complete. 2023-10-19T12:44:15.999 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.993+0000 7f7962d97640 1 Processor -- start 2023-10-19T12:44:15.999 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.993+0000 7f7962d97640 1 -- start start 2023-10-19T12:44:15.999 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.993+0000 7f7962d97640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f795c0cd1c0 0x7f795c144670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:15.999 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.994+0000 7f7962d97640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f795c0cd9e0 0x7f795c15a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:15.999 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.994+0000 7f7962d97640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f795c158910 0x7f795c15a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:16.000 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.994+0000 7f7962d97640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f795c0c2a10 con 0x7f795c158910 2023-10-19T12:44:16.000 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.994+0000 7f7962d97640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f795c0c2c60 con 0x7f795c0cd9e0 2023-10-19T12:44:16.000 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.994+0000 7f7962d97640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f795c0c2f40 con 0x7f795c0cd1c0 2023-10-19T12:44:16.000 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.994+0000 7f7961334640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f795c158910 0x7f795c15a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:16.000 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.994+0000 7f7961334640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f795c158910 0x7f795c15a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40922/0 (socket says 172.21.15.57:40922) 2023-10-19T12:44:16.001 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.994+0000 7f7961334640 1 -- 172.21.15.57:0/3310188639 learned_addr learned my addr 172.21.15.57:0/3310188639 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:16.001 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.994+0000 7f7953fff640 1 --2- 172.21.15.57:0/3310188639 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f795c0cd9e0 0x7f795c15a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:16.001 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.994+0000 7f7960b33640 1 --2- 172.21.15.57:0/3310188639 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f795c0cd1c0 0x7f795c144670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:16.001 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.994+0000 7f7961334640 1 -- 172.21.15.57:0/3310188639 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f795c0cd1c0 msgr2=0x7f795c144670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:16.001 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.994+0000 7f7961334640 1 --2- 172.21.15.57:0/3310188639 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f795c0cd1c0 0x7f795c144670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:16.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.994+0000 7f7961334640 1 -- 172.21.15.57:0/3310188639 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f795c0cd9e0 msgr2=0x7f795c15a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:16.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.994+0000 7f7961334640 1 --2- 172.21.15.57:0/3310188639 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f795c0cd9e0 0x7f795c15a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:16.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.994+0000 7f7961334640 1 -- 172.21.15.57:0/3310188639 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7954009900 con 0x7f795c158910 2023-10-19T12:44:16.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.994+0000 7f7961334640 1 --2- 172.21.15.57:0/3310188639 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f795c158910 0x7f795c15a620 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7954037040 tx=0x7f7954035db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:16.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.994+0000 7f79517fa640 1 -- 172.21.15.57:0/3310188639 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7954034440 con 0x7f795c158910 2023-10-19T12:44:16.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.994+0000 7f79517fa640 1 -- 172.21.15.57:0/3310188639 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f7954004250 con 0x7f795c158910 2023-10-19T12:44:16.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.994+0000 7f7962d97640 1 -- 172.21.15.57:0/3310188639 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f795c0c3220 con 0x7f795c158910 2023-10-19T12:44:16.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.994+0000 7f79517fa640 1 -- 172.21.15.57:0/3310188639 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7954004640 con 0x7f795c158910 2023-10-19T12:44:16.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.994+0000 7f7962d97640 1 -- 172.21.15.57:0/3310188639 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f795c0c37e0 con 0x7f795c158910 2023-10-19T12:44:16.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.995+0000 7f7962d97640 1 -- 172.21.15.57:0/3310188639 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f795c0c39d0 con 0x7f795c158910 2023-10-19T12:44:16.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.996+0000 7f79517fa640 1 -- 172.21.15.57:0/3310188639 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f795403b040 con 0x7f795c158910 2023-10-19T12:44:16.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.996+0000 7f79517fa640 1 --2- 172.21.15.57:0/3310188639 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f794806ae40 0x7f794806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:16.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.996+0000 7f79517fa640 1 -- 172.21.15.57:0/3310188639 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f79540abed0 con 0x7f795c158910 2023-10-19T12:44:16.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.996+0000 7f7960b33640 1 --2- 172.21.15.57:0/3310188639 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f794806ae40 0x7f794806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:16.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:15.997+0000 7f7960b33640 1 --2- 172.21.15.57:0/3310188639 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f794806ae40 0x7f794806d300 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f795800bb00 tx=0x7f7958005e30 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:16.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:16.001+0000 7f79517fa640 1 -- 172.21.15.57:0/3310188639 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f79540772c0 con 0x7f795c158910 2023-10-19T12:44:16.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:16.177+0000 7f7962d97640 1 -- 172.21.15.57:0/3310188639 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f795c0690e0 con 0x7f795c158910 2023-10-19T12:44:16.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:16.178+0000 7f79517fa640 1 -- 172.21.15.57:0/3310188639 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v17) v1 ==== 84+0+65 (secure 0 0 0) 0x7f795407a8e0 con 0x7f795c158910 2023-10-19T12:44:16.181 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:44:16.182 INFO:teuthology.orchestra.run.smithi057.stdout:[ 2023-10-19T12:44:16.182 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T12:44:16.182 INFO:teuthology.orchestra.run.smithi057.stdout: "poolnum": 1, 2023-10-19T12:44:16.182 INFO:teuthology.orchestra.run.smithi057.stdout: "poolname": ".mgr" 2023-10-19T12:44:16.182 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T12:44:16.183 INFO:teuthology.orchestra.run.smithi057.stdout:] 2023-10-19T12:44:16.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:16.179+0000 7f7962d97640 1 -- 172.21.15.57:0/3310188639 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f794806ae40 msgr2=0x7f794806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:16.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:16.179+0000 7f7962d97640 1 --2- 172.21.15.57:0/3310188639 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f794806ae40 0x7f794806d300 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f795800bb00 tx=0x7f7958005e30 comp rx=0 tx=0).stop 2023-10-19T12:44:16.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:16.179+0000 7f7962d97640 1 -- 172.21.15.57:0/3310188639 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f795c158910 msgr2=0x7f795c15a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:16.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:16.179+0000 7f7962d97640 1 --2- 172.21.15.57:0/3310188639 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f795c158910 0x7f795c15a620 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7954037040 tx=0x7f7954035db0 comp rx=0 tx=0).stop 2023-10-19T12:44:16.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:16.180+0000 7f7962d97640 1 -- 172.21.15.57:0/3310188639 shutdown_connections 2023-10-19T12:44:16.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:16.180+0000 7f7962d97640 1 --2- 172.21.15.57:0/3310188639 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f794806ae40 0x7f794806d300 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:16.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:16.180+0000 7f7962d97640 1 --2- 172.21.15.57:0/3310188639 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f795c158910 0x7f795c15a620 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:16.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:16.180+0000 7f7962d97640 1 --2- 172.21.15.57:0/3310188639 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f795c0cd9e0 0x7f795c15a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:16.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:16.180+0000 7f7962d97640 1 --2- 172.21.15.57:0/3310188639 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f795c0cd1c0 0x7f795c144670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:16.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:16.180+0000 7f7962d97640 1 -- 172.21.15.57:0/3310188639 >> 172.21.15.57:0/3310188639 conn(0x7f795c0af7a0 msgr2=0x7f795c155610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:16.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:16.180+0000 7f7962d97640 1 -- 172.21.15.57:0/3310188639 shutdown_connections 2023-10-19T12:44:16.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:16.180+0000 7f7962d97640 1 -- 172.21.15.57:0/3310188639 wait complete. 2023-10-19T12:44:16.193 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-19T12:44:16.193 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-19T12:44:16.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:16.302+0000 7ffa684a9640 1 -- 172.21.15.57:0/70145022 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffa6015b000 msgr2=0x7ffa600d1810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:16.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:16.302+0000 7ffa684a9640 1 --2- 172.21.15.57:0/70145022 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffa6015b000 0x7ffa600d1810 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ffa5c009830 tx=0x7ffa5c02bd60 comp rx=0 tx=0).stop 2023-10-19T12:44:16.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:16.302+0000 7ffa684a9640 1 -- 172.21.15.57:0/70145022 shutdown_connections 2023-10-19T12:44:16.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:16.302+0000 7ffa684a9640 1 --2- 172.21.15.57:0/70145022 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffa6015b000 0x7ffa600d1810 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:16.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:16.302+0000 7ffa684a9640 1 -- 172.21.15.57:0/70145022 >> 172.21.15.57:0/70145022 conn(0x7ffa600d0cf0 msgr2=0x7ffa600d1100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:16.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:16.302+0000 7ffa684a9640 1 -- 172.21.15.57:0/70145022 shutdown_connections 2023-10-19T12:44:16.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:16.302+0000 7ffa684a9640 1 -- 172.21.15.57:0/70145022 wait complete. 2023-10-19T12:44:16.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:16.303+0000 7ffa684a9640 1 Processor -- start 2023-10-19T12:44:16.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:16.304+0000 7ffa684a9640 1 -- start start 2023-10-19T12:44:16.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:16.304+0000 7ffa684a9640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffa600cf9e0 0x7ffa601e2fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:16.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:16.304+0000 7ffa684a9640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ffa6015b000 0x7ffa600b1aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:16.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:16.304+0000 7ffa684a9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffa6015b3e0 0x7ffa600b1fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:16.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:16.304+0000 7ffa684a9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa600ae290 con 0x7ffa6015b3e0 2023-10-19T12:44:16.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:16.304+0000 7ffa684a9640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa600ae570 con 0x7ffa600cf9e0 2023-10-19T12:44:16.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:16.304+0000 7ffa684a9640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa600ae850 con 0x7ffa6015b000 2023-10-19T12:44:16.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:16.304+0000 7ffa66a46640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffa6015b3e0 0x7ffa600b1fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:16.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:16.304+0000 7ffa66a46640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffa6015b3e0 0x7ffa600b1fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40936/0 (socket says 172.21.15.57:40936) 2023-10-19T12:44:16.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:16.304+0000 7ffa66a46640 1 -- 172.21.15.57:0/3038809121 learned_addr learned my addr 172.21.15.57:0/3038809121 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:16.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:16.305+0000 7ffa66245640 1 --2- 172.21.15.57:0/3038809121 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffa600cf9e0 0x7ffa601e2fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:16.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:16.305+0000 7ffa65a44640 1 --2- 172.21.15.57:0/3038809121 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ffa6015b000 0x7ffa600b1aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:16.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:16.305+0000 7ffa66a46640 1 -- 172.21.15.57:0/3038809121 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ffa6015b000 msgr2=0x7ffa600b1aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:16.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:16.305+0000 7ffa66a46640 1 --2- 172.21.15.57:0/3038809121 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ffa6015b000 0x7ffa600b1aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:16.312 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:16.305+0000 7ffa66a46640 1 -- 172.21.15.57:0/3038809121 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffa600cf9e0 msgr2=0x7ffa601e2fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:16.312 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:16.305+0000 7ffa66a46640 1 --2- 172.21.15.57:0/3038809121 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffa600cf9e0 0x7ffa601e2fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:16.312 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:16.305+0000 7ffa66a46640 1 -- 172.21.15.57:0/3038809121 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa5c009490 con 0x7ffa6015b3e0 2023-10-19T12:44:16.312 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:16.305+0000 7ffa66245640 1 --2- 172.21.15.57:0/3038809121 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffa600cf9e0 0x7ffa601e2fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:44:16.312 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:16.305+0000 7ffa65a44640 1 --2- 172.21.15.57:0/3038809121 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ffa6015b000 0x7ffa600b1aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:44:16.312 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:16.305+0000 7ffa66a46640 1 --2- 172.21.15.57:0/3038809121 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffa6015b3e0 0x7ffa600b1fe0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ffa5800c780 tx=0x7ffa5800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:16.313 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:16.305+0000 7ffa56ffd640 1 -- 172.21.15.57:0/3038809121 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa58003d80 con 0x7ffa6015b3e0 2023-10-19T12:44:16.313 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:16.305+0000 7ffa56ffd640 1 -- 172.21.15.57:0/3038809121 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7ffa5800ce30 con 0x7ffa6015b3e0 2023-10-19T12:44:16.313 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:16.305+0000 7ffa684a9640 1 -- 172.21.15.57:0/3038809121 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa600aeb30 con 0x7ffa6015b3e0 2023-10-19T12:44:16.313 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:16.305+0000 7ffa56ffd640 1 -- 172.21.15.57:0/3038809121 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa580052b0 con 0x7ffa6015b3e0 2023-10-19T12:44:16.313 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:16.305+0000 7ffa684a9640 1 -- 172.21.15.57:0/3038809121 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa600b53c0 con 0x7ffa6015b3e0 2023-10-19T12:44:16.314 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:16.306+0000 7ffa684a9640 1 -- 172.21.15.57:0/3038809121 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa601e10b0 con 0x7ffa6015b3e0 2023-10-19T12:44:16.315 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:16.306+0000 7ffa56ffd640 1 -- 172.21.15.57:0/3038809121 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7ffa58005410 con 0x7ffa6015b3e0 2023-10-19T12:44:16.315 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:16.307+0000 7ffa56ffd640 1 --2- 172.21.15.57:0/3038809121 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7ffa4406ae40 0x7ffa4406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:16.315 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:16.307+0000 7ffa56ffd640 1 -- 172.21.15.57:0/3038809121 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7ffa58088ce0 con 0x7ffa6015b3e0 2023-10-19T12:44:16.315 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:16.307+0000 7ffa66245640 1 --2- 172.21.15.57:0/3038809121 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7ffa4406ae40 0x7ffa4406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:16.315 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:16.308+0000 7ffa66245640 1 --2- 172.21.15.57:0/3038809121 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7ffa4406ae40 0x7ffa4406d300 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ffa5c036000 tx=0x7ffa5c004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:16.315 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:16.311+0000 7ffa56ffd640 1 -- 172.21.15.57:0/3038809121 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ffa58054170 con 0x7ffa6015b3e0 2023-10-19T12:44:16.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:16.486+0000 7ffa684a9640 1 -- 172.21.15.57:0/3038809121 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7ffa600690e0 con 0x7ffa6015b3e0 2023-10-19T12:44:17.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:17.044+0000 7ffa56ffd640 1 -- 172.21.15.57:0/3038809121 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v18) v1 ==== 138+0+0 (secure 0 0 0) 0x7ffa58017020 con 0x7ffa6015b3e0 2023-10-19T12:44:17.048 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'cephfs_metadata' created 2023-10-19T12:44:17.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:17.047+0000 7ffa684a9640 1 -- 172.21.15.57:0/3038809121 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7ffa4406ae40 msgr2=0x7ffa4406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:17.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:17.047+0000 7ffa684a9640 1 --2- 172.21.15.57:0/3038809121 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7ffa4406ae40 0x7ffa4406d300 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ffa5c036000 tx=0x7ffa5c004730 comp rx=0 tx=0).stop 2023-10-19T12:44:17.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:17.047+0000 7ffa684a9640 1 -- 172.21.15.57:0/3038809121 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffa6015b3e0 msgr2=0x7ffa600b1fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:17.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:17.047+0000 7ffa684a9640 1 --2- 172.21.15.57:0/3038809121 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffa6015b3e0 0x7ffa600b1fe0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ffa5800c780 tx=0x7ffa5800cc40 comp rx=0 tx=0).stop 2023-10-19T12:44:17.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:17.047+0000 7ffa684a9640 1 -- 172.21.15.57:0/3038809121 shutdown_connections 2023-10-19T12:44:17.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:17.047+0000 7ffa684a9640 1 --2- 172.21.15.57:0/3038809121 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7ffa4406ae40 0x7ffa4406d300 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:17.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:17.047+0000 7ffa684a9640 1 --2- 172.21.15.57:0/3038809121 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffa6015b3e0 0x7ffa600b1fe0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:17.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:17.047+0000 7ffa684a9640 1 --2- 172.21.15.57:0/3038809121 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ffa6015b000 0x7ffa600b1aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:17.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:17.047+0000 7ffa684a9640 1 --2- 172.21.15.57:0/3038809121 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffa600cf9e0 0x7ffa601e2fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:17.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:17.047+0000 7ffa684a9640 1 -- 172.21.15.57:0/3038809121 >> 172.21.15.57:0/3038809121 conn(0x7ffa600d0cf0 msgr2=0x7ffa6006b080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:17.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:17.047+0000 7ffa684a9640 1 -- 172.21.15.57:0/3038809121 shutdown_connections 2023-10-19T12:44:17.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:17.047+0000 7ffa684a9640 1 -- 172.21.15.57:0/3038809121 wait complete. 2023-10-19T12:44:17.067 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2023-10-19T12:44:17.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:17.182+0000 7f15c600e640 1 -- 172.21.15.57:0/1433401747 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f15c0159060 msgr2=0x7f15c006d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:17.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:17.182+0000 7f15c600e640 1 --2- 172.21.15.57:0/1433401747 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f15c0159060 0x7f15c006d260 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f15b4009830 tx=0x7f15b402bcd0 comp rx=0 tx=0).stop 2023-10-19T12:44:17.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:17.182+0000 7f15c600e640 1 -- 172.21.15.57:0/1433401747 shutdown_connections 2023-10-19T12:44:17.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:17.182+0000 7f15c600e640 1 --2- 172.21.15.57:0/1433401747 >> v2:172.21.15.179:3300/0 conn(0x7f15c00cdb30 0x7f15c00cdfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:17.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:17.182+0000 7f15c600e640 1 --2- 172.21.15.57:0/1433401747 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f15c0159060 0x7f15c006d260 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:17.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:17.182+0000 7f15c600e640 1 -- 172.21.15.57:0/1433401747 >> 172.21.15.57:0/1433401747 conn(0x7f15c00b0670 msgr2=0x7f15c00b0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:17.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:17.182+0000 7f15c600e640 1 -- 172.21.15.57:0/1433401747 shutdown_connections 2023-10-19T12:44:17.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:17.182+0000 7f15c600e640 1 -- 172.21.15.57:0/1433401747 wait complete. 2023-10-19T12:44:17.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:17.183+0000 7f15c600e640 1 Processor -- start 2023-10-19T12:44:17.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:17.183+0000 7f15c600e640 1 -- start start 2023-10-19T12:44:17.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:17.184+0000 7f15c600e640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f15c00cdb30 0x7f15c00c5bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:17.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:17.184+0000 7f15c600e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f15c0159060 0x7f15c0150fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:17.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:17.184+0000 7f15c600e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f15c015ccf0 0x7f15c00b15b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:17.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:17.184+0000 7f15c600e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15c00c3950 con 0x7f15c015ccf0 2023-10-19T12:44:17.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:17.184+0000 7f15c600e640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15c00c3060 con 0x7f15c0159060 2023-10-19T12:44:17.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:17.184+0000 7f15c600e640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15c00c3300 con 0x7f15c00cdb30 2023-10-19T12:44:17.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:17.184+0000 7f15bffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f15c015ccf0 0x7f15c00b15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:17.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:17.184+0000 7f15bffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f15c015ccf0 0x7f15c00b15b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:52976/0 (socket says 172.21.15.57:52976) 2023-10-19T12:44:17.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:17.184+0000 7f15bffff640 1 -- 172.21.15.57:0/737073458 learned_addr learned my addr 172.21.15.57:0/737073458 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:17.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:17.184+0000 7f15beffd640 1 --2- 172.21.15.57:0/737073458 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f15c0159060 0x7f15c0150fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:17.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:17.184+0000 7f15bf7fe640 1 --2- 172.21.15.57:0/737073458 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f15c00cdb30 0x7f15c00c5bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:17.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:17.185+0000 7f15bffff640 1 -- 172.21.15.57:0/737073458 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f15c00cdb30 msgr2=0x7f15c00c5bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:17.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:17.185+0000 7f15bffff640 1 --2- 172.21.15.57:0/737073458 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f15c00cdb30 0x7f15c00c5bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:17.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:17.185+0000 7f15bffff640 1 -- 172.21.15.57:0/737073458 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f15c0159060 msgr2=0x7f15c0150fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:17.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:17.185+0000 7f15bffff640 1 --2- 172.21.15.57:0/737073458 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f15c0159060 0x7f15c0150fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:17.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:17.185+0000 7f15bffff640 1 -- 172.21.15.57:0/737073458 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15b0009490 con 0x7f15c015ccf0 2023-10-19T12:44:17.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:17.185+0000 7f15bffff640 1 --2- 172.21.15.57:0/737073458 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f15c015ccf0 0x7f15c00b15b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f15b800caa0 tx=0x7f15b800cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:17.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:17.185+0000 7f15a7fff640 1 -- 172.21.15.57:0/737073458 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15b800ac00 con 0x7f15c015ccf0 2023-10-19T12:44:17.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:17.185+0000 7f15a7fff640 1 -- 172.21.15.57:0/737073458 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f15b800ae70 con 0x7f15c015ccf0 2023-10-19T12:44:17.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:17.185+0000 7f15c600e640 1 -- 172.21.15.57:0/737073458 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15b4009490 con 0x7f15c015ccf0 2023-10-19T12:44:17.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:17.185+0000 7f15a7fff640 1 -- 172.21.15.57:0/737073458 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15b8003bb0 con 0x7f15c015ccf0 2023-10-19T12:44:17.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:17.185+0000 7f15c600e640 1 -- 172.21.15.57:0/737073458 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15c00b2270 con 0x7f15c015ccf0 2023-10-19T12:44:17.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:17.186+0000 7f15c600e640 1 -- 172.21.15.57:0/737073458 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15c00c3ac0 con 0x7f15c015ccf0 2023-10-19T12:44:17.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:17.187+0000 7f15a7fff640 1 -- 172.21.15.57:0/737073458 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f15b8003d10 con 0x7f15c015ccf0 2023-10-19T12:44:17.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:17.187+0000 7f15a7fff640 1 --2- 172.21.15.57:0/737073458 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f15ac06ae40 0x7f15ac06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:17.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:17.187+0000 7f15a7fff640 1 -- 172.21.15.57:0/737073458 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f15b8089cb0 con 0x7f15c015ccf0 2023-10-19T12:44:17.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:17.187+0000 7f15bf7fe640 1 --2- 172.21.15.57:0/737073458 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f15ac06ae40 0x7f15ac06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:17.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:17.188+0000 7f15bf7fe640 1 --2- 172.21.15.57:0/737073458 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f15ac06ae40 0x7f15ac06d300 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f15b4036000 tx=0x7f15b4004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:17.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:17.192+0000 7f15a7fff640 1 -- 172.21.15.57:0/737073458 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f15b8058a90 con 0x7f15c015ccf0 2023-10-19T12:44:17.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:17.367+0000 7f15c600e640 1 -- 172.21.15.57:0/737073458 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) v1 -- 0x7f15c00675f0 con 0x7f15c015ccf0 2023-10-19T12:44:18.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:18.049+0000 7f15a7fff640 1 -- 172.21.15.57:0/737073458 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v19) v1 ==== 171+0+0 (secure 0 0 0) 0x7f15b8077020 con 0x7f15c015ccf0 2023-10-19T12:44:18.052 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'cephfs_data' created 2023-10-19T12:44:18.055 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:18.052+0000 7f15c600e640 1 -- 172.21.15.57:0/737073458 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f15ac06ae40 msgr2=0x7f15ac06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:18.055 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:18.052+0000 7f15c600e640 1 --2- 172.21.15.57:0/737073458 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f15ac06ae40 0x7f15ac06d300 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f15b4036000 tx=0x7f15b4004730 comp rx=0 tx=0).stop 2023-10-19T12:44:18.056 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:18.052+0000 7f15c600e640 1 -- 172.21.15.57:0/737073458 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f15c015ccf0 msgr2=0x7f15c00b15b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:18.056 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:18.052+0000 7f15c600e640 1 --2- 172.21.15.57:0/737073458 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f15c015ccf0 0x7f15c00b15b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f15b800caa0 tx=0x7f15b800cf60 comp rx=0 tx=0).stop 2023-10-19T12:44:18.056 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:18.053+0000 7f15c600e640 1 -- 172.21.15.57:0/737073458 shutdown_connections 2023-10-19T12:44:18.057 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:18.053+0000 7f15c600e640 1 --2- 172.21.15.57:0/737073458 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f15ac06ae40 0x7f15ac06d300 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:18.057 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:18.053+0000 7f15c600e640 1 --2- 172.21.15.57:0/737073458 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f15c015ccf0 0x7f15c00b15b0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:18.057 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:18.053+0000 7f15c600e640 1 --2- 172.21.15.57:0/737073458 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f15c0159060 0x7f15c0150fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:18.057 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:18.053+0000 7f15c600e640 1 --2- 172.21.15.57:0/737073458 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f15c00cdb30 0x7f15c00c5bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:18.057 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:18.053+0000 7f15c600e640 1 -- 172.21.15.57:0/737073458 >> 172.21.15.57:0/737073458 conn(0x7f15c00b0670 msgr2=0x7f15c00cca20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:18.058 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:18.054+0000 7f15c600e640 1 -- 172.21.15.57:0/737073458 shutdown_connections 2023-10-19T12:44:18.058 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:18.054+0000 7f15c600e640 1 -- 172.21.15.57:0/737073458 wait complete. 2023-10-19T12:44:18.082 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-19T12:44:18.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:18.196+0000 7f18f0fb6640 1 -- 172.21.15.57:0/2141144289 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f18ec15c510 msgr2=0x7f18ec06d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:18.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:18.196+0000 7f18f0fb6640 1 --2- 172.21.15.57:0/2141144289 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f18ec15c510 0x7f18ec06d2d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f18ec0c2280 tx=0x7f18dc02bda0 comp rx=0 tx=0).stop 2023-10-19T12:44:18.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:18.196+0000 7f18f0fb6640 1 -- 172.21.15.57:0/2141144289 shutdown_connections 2023-10-19T12:44:18.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:18.196+0000 7f18f0fb6640 1 --2- 172.21.15.57:0/2141144289 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f18ec06dd50 0x7f18ec0ab950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:18.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:18.196+0000 7f18f0fb6640 1 --2- 172.21.15.57:0/2141144289 >> v2:172.21.15.179:3300/0 conn(0x7f18ec06f0a0 0x7f18ec06d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:18.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:18.196+0000 7f18f0fb6640 1 --2- 172.21.15.57:0/2141144289 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f18ec15c510 0x7f18ec06d2d0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:18.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:18.196+0000 7f18f0fb6640 1 -- 172.21.15.57:0/2141144289 >> 172.21.15.57:0/2141144289 conn(0x7f18ec0b0670 msgr2=0x7f18ec0b0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:18.201 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:18.196+0000 7f18f0fb6640 1 -- 172.21.15.57:0/2141144289 shutdown_connections 2023-10-19T12:44:18.201 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:18.196+0000 7f18f0fb6640 1 -- 172.21.15.57:0/2141144289 wait complete. 2023-10-19T12:44:18.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:18.197+0000 7f18f0fb6640 1 Processor -- start 2023-10-19T12:44:18.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:18.197+0000 7f18f0fb6640 1 -- start start 2023-10-19T12:44:18.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:18.198+0000 7f18f0fb6640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f18ec06dd50 0x7f18ec06bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:18.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:18.198+0000 7f18f0fb6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f18ec06f0a0 0x7f18ec06a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:18.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:18.198+0000 7f18f0fb6640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f18ec15c510 0x7f18ec06ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:18.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:18.198+0000 7f18f0fb6640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18ec06c220 con 0x7f18ec06f0a0 2023-10-19T12:44:18.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:18.198+0000 7f18f0fb6640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18ec0b03e0 con 0x7f18ec15c510 2023-10-19T12:44:18.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:18.198+0000 7f18f0fb6640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18ec0ad580 con 0x7f18ec06dd50 2023-10-19T12:44:18.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:18.198+0000 7f18e3fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f18ec06f0a0 0x7f18ec06a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:18.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:18.198+0000 7f18e3fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f18ec06f0a0 0x7f18ec06a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:52988/0 (socket says 172.21.15.57:52988) 2023-10-19T12:44:18.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:18.198+0000 7f18e3fff640 1 -- 172.21.15.57:0/2194342832 learned_addr learned my addr 172.21.15.57:0/2194342832 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:18.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:18.198+0000 7f18ea59c640 1 --2- 172.21.15.57:0/2194342832 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f18ec06dd50 0x7f18ec06bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:18.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:18.198+0000 7f18ead9d640 1 --2- 172.21.15.57:0/2194342832 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f18ec15c510 0x7f18ec06ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:18.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:18.198+0000 7f18e3fff640 1 -- 172.21.15.57:0/2194342832 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f18ec06dd50 msgr2=0x7f18ec06bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:18.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:18.198+0000 7f18e3fff640 1 --2- 172.21.15.57:0/2194342832 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f18ec06dd50 0x7f18ec06bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:18.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:18.198+0000 7f18e3fff640 1 -- 172.21.15.57:0/2194342832 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f18ec15c510 msgr2=0x7f18ec06ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:18.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:18.198+0000 7f18e3fff640 1 --2- 172.21.15.57:0/2194342832 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f18ec15c510 0x7f18ec06ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:18.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:18.198+0000 7f18e3fff640 1 -- 172.21.15.57:0/2194342832 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18dc009560 con 0x7f18ec06f0a0 2023-10-19T12:44:18.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:18.198+0000 7f18e3fff640 1 --2- 172.21.15.57:0/2194342832 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f18ec06f0a0 0x7f18ec06a830 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f18d400d980 tx=0x7f18d400de40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:18.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:18.199+0000 7f18e2ffd640 1 -- 172.21.15.57:0/2194342832 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18d4009c30 con 0x7f18ec06f0a0 2023-10-19T12:44:18.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:18.199+0000 7f18e2ffd640 1 -- 172.21.15.57:0/2194342832 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f18d4009e80 con 0x7f18ec06f0a0 2023-10-19T12:44:18.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:18.199+0000 7f18f0fb6640 1 -- 172.21.15.57:0/2194342832 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18ec0ad820 con 0x7f18ec06f0a0 2023-10-19T12:44:18.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:18.199+0000 7f18e2ffd640 1 -- 172.21.15.57:0/2194342832 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18d40106a0 con 0x7f18ec06f0a0 2023-10-19T12:44:18.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:18.199+0000 7f18f0fb6640 1 -- 172.21.15.57:0/2194342832 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18ec0adde0 con 0x7f18ec06f0a0 2023-10-19T12:44:18.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:18.200+0000 7f18f0fb6640 1 -- 172.21.15.57:0/2194342832 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18ec0690e0 con 0x7f18ec06f0a0 2023-10-19T12:44:18.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:18.200+0000 7f18e2ffd640 1 -- 172.21.15.57:0/2194342832 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f18d4010800 con 0x7f18ec06f0a0 2023-10-19T12:44:18.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:18.200+0000 7f18e2ffd640 1 --2- 172.21.15.57:0/2194342832 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f18cc06ae70 0x7f18cc06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:18.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:18.201+0000 7f18e2ffd640 1 -- 172.21.15.57:0/2194342832 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f18d4089910 con 0x7f18ec06f0a0 2023-10-19T12:44:18.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:18.201+0000 7f18ea59c640 1 --2- 172.21.15.57:0/2194342832 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f18cc06ae70 0x7f18cc06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:18.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:18.201+0000 7f18ea59c640 1 --2- 172.21.15.57:0/2194342832 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f18cc06ae70 0x7f18cc06d330 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f18dc035000 tx=0x7f18dc004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:18.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:18.205+0000 7f18e2ffd640 1 -- 172.21.15.57:0/2194342832 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f18d4054ad0 con 0x7f18ec06f0a0 2023-10-19T12:44:18.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:18.417+0000 7f18f0fb6640 1 -- 172.21.15.57:0/2194342832 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f18ec0c1c80 con 0x7f18ec06f0a0 2023-10-19T12:44:19.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:19.081+0000 7f18e2ffd640 1 -- 172.21.15.57:0/2194342832 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7f18d40170c0 con 0x7f18ec06f0a0 2023-10-19T12:44:19.085 INFO:teuthology.orchestra.run.smithi057.stderr:new fs with metadata pool 2 and data pool 3 2023-10-19T12:44:19.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:19.092+0000 7f18f0fb6640 1 -- 172.21.15.57:0/2194342832 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f18cc06ae70 msgr2=0x7f18cc06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:19.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:19.092+0000 7f18f0fb6640 1 --2- 172.21.15.57:0/2194342832 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f18cc06ae70 0x7f18cc06d330 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f18dc035000 tx=0x7f18dc004730 comp rx=0 tx=0).stop 2023-10-19T12:44:19.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:19.092+0000 7f18f0fb6640 1 -- 172.21.15.57:0/2194342832 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f18ec06f0a0 msgr2=0x7f18ec06a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:19.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:19.092+0000 7f18f0fb6640 1 --2- 172.21.15.57:0/2194342832 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f18ec06f0a0 0x7f18ec06a830 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f18d400d980 tx=0x7f18d400de40 comp rx=0 tx=0).stop 2023-10-19T12:44:19.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:19.092+0000 7f18f0fb6640 1 -- 172.21.15.57:0/2194342832 shutdown_connections 2023-10-19T12:44:19.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:19.092+0000 7f18f0fb6640 1 --2- 172.21.15.57:0/2194342832 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f18cc06ae70 0x7f18cc06d330 secure :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f18dc035000 tx=0x7f18dc004730 comp rx=0 tx=0).stop 2023-10-19T12:44:19.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:19.092+0000 7f18f0fb6640 1 --2- 172.21.15.57:0/2194342832 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f18ec15c510 0x7f18ec06ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:19.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:19.092+0000 7f18f0fb6640 1 --2- 172.21.15.57:0/2194342832 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f18ec06f0a0 0x7f18ec06a830 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:19.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:19.092+0000 7f18f0fb6640 1 --2- 172.21.15.57:0/2194342832 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f18ec06dd50 0x7f18ec06bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:19.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:19.092+0000 7f18f0fb6640 1 -- 172.21.15.57:0/2194342832 >> 172.21.15.57:0/2194342832 conn(0x7f18ec0b0670 msgr2=0x7f18ec0c4730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:19.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:19.092+0000 7f18f0fb6640 1 -- 172.21.15.57:0/2194342832 shutdown_connections 2023-10-19T12:44:19.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:19.092+0000 7f18f0fb6640 1 -- 172.21.15.57:0/2194342832 wait complete. 2023-10-19T12:44:19.114 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2023-10-19T12:44:19.114 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2023-10-19T12:44:19.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:19.231+0000 7f8c518f2640 1 -- 172.21.15.57:0/1206380584 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8c380052a0 msgr2=0x7f8c3800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:19.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:19.231+0000 7f8c518f2640 1 --2- 172.21.15.57:0/1206380584 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8c380052a0 0x7f8c3800b980 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8c44001d50 tx=0x7f8c380057f0 comp rx=0 tx=0).stop 2023-10-19T12:44:19.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:19.231+0000 7f8c518f2640 1 -- 172.21.15.57:0/1206380584 shutdown_connections 2023-10-19T12:44:19.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:19.231+0000 7f8c518f2640 1 --2- 172.21.15.57:0/1206380584 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8c380052a0 0x7f8c3800b980 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:19.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:19.231+0000 7f8c518f2640 1 -- 172.21.15.57:0/1206380584 >> 172.21.15.57:0/1206380584 conn(0x7f8c4c0b0670 msgr2=0x7f8c4c0b0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:19.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:19.231+0000 7f8c518f2640 1 -- 172.21.15.57:0/1206380584 shutdown_connections 2023-10-19T12:44:19.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:19.231+0000 7f8c518f2640 1 -- 172.21.15.57:0/1206380584 wait complete. 2023-10-19T12:44:19.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:19.232+0000 7f8c518f2640 1 Processor -- start 2023-10-19T12:44:19.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:19.232+0000 7f8c518f2640 1 -- start start 2023-10-19T12:44:19.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:19.233+0000 7f8c518f2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8c380052a0 0x7f8c4c0c4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:19.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:19.233+0000 7f8c518f2640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8c4c151050 0x7f8c4c15aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:19.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:19.233+0000 7f8c518f2640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8c4c1e2af0 0x7f8c4c1e2f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:19.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:19.233+0000 7f8c518f2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c4c15bf00 con 0x7f8c380052a0 2023-10-19T12:44:19.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:19.233+0000 7f8c518f2640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c4c15b2f0 con 0x7f8c4c1e2af0 2023-10-19T12:44:19.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:19.233+0000 7f8c518f2640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c4c15b5d0 con 0x7f8c4c151050 2023-10-19T12:44:19.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:19.233+0000 7f8c4affd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8c380052a0 0x7f8c4c0c4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:19.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:19.233+0000 7f8c4affd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8c380052a0 0x7f8c4c0c4ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:53010/0 (socket says 172.21.15.57:53010) 2023-10-19T12:44:19.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:19.233+0000 7f8c4affd640 1 -- 172.21.15.57:0/2758824650 learned_addr learned my addr 172.21.15.57:0/2758824650 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:19.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:19.233+0000 7f8c4a7fc640 1 --2- 172.21.15.57:0/2758824650 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8c4c151050 0x7f8c4c15aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:19.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:19.233+0000 7f8c4b7fe640 1 --2- 172.21.15.57:0/2758824650 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8c4c1e2af0 0x7f8c4c1e2f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:19.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:19.233+0000 7f8c4affd640 1 -- 172.21.15.57:0/2758824650 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8c4c151050 msgr2=0x7f8c4c15aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:19.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:19.233+0000 7f8c4affd640 1 --2- 172.21.15.57:0/2758824650 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8c4c151050 0x7f8c4c15aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:19.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:19.233+0000 7f8c4affd640 1 -- 172.21.15.57:0/2758824650 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8c4c1e2af0 msgr2=0x7f8c4c1e2f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:19.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:19.233+0000 7f8c4affd640 1 --2- 172.21.15.57:0/2758824650 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8c4c1e2af0 0x7f8c4c1e2f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:19.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:19.233+0000 7f8c4affd640 1 -- 172.21.15.57:0/2758824650 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c4c15b8b0 con 0x7f8c380052a0 2023-10-19T12:44:19.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:19.233+0000 7f8c4affd640 1 --2- 172.21.15.57:0/2758824650 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8c380052a0 0x7f8c4c0c4ef0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c006590 tx=0x7f8c3c0039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:19.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:19.234+0000 7f8c43fff640 1 -- 172.21.15.57:0/2758824650 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c3c011900 con 0x7f8c380052a0 2023-10-19T12:44:19.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:19.234+0000 7f8c43fff640 1 -- 172.21.15.57:0/2758824650 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f8c3c011a60 con 0x7f8c380052a0 2023-10-19T12:44:19.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:19.234+0000 7f8c518f2640 1 -- 172.21.15.57:0/2758824650 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c4c15bb90 con 0x7f8c380052a0 2023-10-19T12:44:19.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:19.234+0000 7f8c43fff640 1 -- 172.21.15.57:0/2758824650 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c3c011e70 con 0x7f8c380052a0 2023-10-19T12:44:19.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:19.234+0000 7f8c518f2640 1 -- 172.21.15.57:0/2758824650 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c4c15c390 con 0x7f8c380052a0 2023-10-19T12:44:19.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:19.235+0000 7f8c518f2640 1 -- 172.21.15.57:0/2758824650 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c4c15b710 con 0x7f8c380052a0 2023-10-19T12:44:19.243 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:19.235+0000 7f8c43fff640 1 -- 172.21.15.57:0/2758824650 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f8c3c010420 con 0x7f8c380052a0 2023-10-19T12:44:19.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:19.236+0000 7f8c43fff640 1 --2- 172.21.15.57:0/2758824650 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f8c2c06ae40 0x7f8c2c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:19.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:19.236+0000 7f8c43fff640 1 -- 172.21.15.57:0/2758824650 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8c3c08a710 con 0x7f8c380052a0 2023-10-19T12:44:19.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:19.236+0000 7f8c4a7fc640 1 --2- 172.21.15.57:0/2758824650 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f8c2c06ae40 0x7f8c2c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:19.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:19.236+0000 7f8c4a7fc640 1 --2- 172.21.15.57:0/2758824650 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f8c2c06ae40 0x7f8c2c06d300 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8c4c0c4a70 tx=0x7f8c3400b260 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:19.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:19.240+0000 7f8c43fff640 1 -- 172.21.15.57:0/2758824650 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8c4c15b710 con 0x7f8c380052a0 2023-10-19T12:44:19.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:19.423+0000 7f8c518f2640 1 -- 172.21.15.57:0/2758824650 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) v1 -- 0x7f8c4c15b710 con 0x7f8c380052a0 2023-10-19T12:44:20.073 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:20.070+0000 7f8c43fff640 1 -- 172.21.15.57:0/2758824650 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v21) v1 ==== 157+0+0 (secure 0 0 0) 0x7f8c4c15b710 con 0x7f8c380052a0 2023-10-19T12:44:20.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:20.073+0000 7f8c518f2640 1 -- 172.21.15.57:0/2758824650 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f8c2c06ae40 msgr2=0x7f8c2c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:20.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:20.073+0000 7f8c518f2640 1 --2- 172.21.15.57:0/2758824650 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f8c2c06ae40 0x7f8c2c06d300 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8c4c0c4a70 tx=0x7f8c3400b260 comp rx=0 tx=0).stop 2023-10-19T12:44:20.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:20.073+0000 7f8c518f2640 1 -- 172.21.15.57:0/2758824650 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8c380052a0 msgr2=0x7f8c4c0c4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:20.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:20.073+0000 7f8c518f2640 1 --2- 172.21.15.57:0/2758824650 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8c380052a0 0x7f8c4c0c4ef0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c006590 tx=0x7f8c3c0039c0 comp rx=0 tx=0).stop 2023-10-19T12:44:20.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:20.073+0000 7f8c518f2640 1 -- 172.21.15.57:0/2758824650 shutdown_connections 2023-10-19T12:44:20.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:20.073+0000 7f8c518f2640 1 --2- 172.21.15.57:0/2758824650 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f8c2c06ae40 0x7f8c2c06d300 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:20.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:20.073+0000 7f8c518f2640 1 --2- 172.21.15.57:0/2758824650 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8c4c1e2af0 0x7f8c4c1e2f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:20.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:20.073+0000 7f8c518f2640 1 --2- 172.21.15.57:0/2758824650 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8c4c151050 0x7f8c4c15aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:20.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:20.073+0000 7f8c518f2640 1 --2- 172.21.15.57:0/2758824650 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8c380052a0 0x7f8c4c0c4ef0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:20.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:20.073+0000 7f8c518f2640 1 -- 172.21.15.57:0/2758824650 >> 172.21.15.57:0/2758824650 conn(0x7f8c4c0b0670 msgr2=0x7f8c4c0ae620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:20.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:20.073+0000 7f8c518f2640 1 -- 172.21.15.57:0/2758824650 shutdown_connections 2023-10-19T12:44:20.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:20.073+0000 7f8c518f2640 1 -- 172.21.15.57:0/2758824650 wait complete. 2023-10-19T12:44:20.090 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2023-10-19T12:44:20.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:20.210+0000 7f0a31d05640 1 -- 172.21.15.57:0/3822960308 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0a2c069720 msgr2=0x7f0a2c069bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:20.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:20.210+0000 7f0a31d05640 1 --2- 172.21.15.57:0/3822960308 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0a2c069720 0x7f0a2c069bd0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c068fa0 tx=0x7f0a2402c260 comp rx=0 tx=0).stop 2023-10-19T12:44:20.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:20.210+0000 7f0a31d05640 1 -- 172.21.15.57:0/3822960308 shutdown_connections 2023-10-19T12:44:20.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:20.210+0000 7f0a31d05640 1 --2- 172.21.15.57:0/3822960308 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0a2c069720 0x7f0a2c069bd0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:20.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:20.210+0000 7f0a31d05640 1 --2- 172.21.15.57:0/3822960308 >> v2:172.21.15.179:3300/0 conn(0x7f0a2c158d40 0x7f0a2c06c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:20.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:20.210+0000 7f0a31d05640 1 -- 172.21.15.57:0/3822960308 >> 172.21.15.57:0/3822960308 conn(0x7f0a2c070340 msgr2=0x7f0a2c070750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:20.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:20.210+0000 7f0a31d05640 1 -- 172.21.15.57:0/3822960308 shutdown_connections 2023-10-19T12:44:20.216 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:20.210+0000 7f0a31d05640 1 -- 172.21.15.57:0/3822960308 wait complete. 2023-10-19T12:44:20.216 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:20.211+0000 7f0a31d05640 1 Processor -- start 2023-10-19T12:44:20.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:20.211+0000 7f0a31d05640 1 -- start start 2023-10-19T12:44:20.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:20.212+0000 7f0a31d05640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0a2c069720 0x7f0a2c150bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:20.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:20.212+0000 7f0a31d05640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0a2c09bca0 0x7f0a2c159d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:20.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:20.212+0000 7f0a31d05640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0a2c158d40 0x7f0a2c156e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:20.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:20.212+0000 7f0a31d05640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a2c15a2d0 con 0x7f0a2c069720 2023-10-19T12:44:20.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:20.212+0000 7f0a31d05640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a2c1563d0 con 0x7f0a2c158d40 2023-10-19T12:44:20.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:20.212+0000 7f0a31d05640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a2c15bf80 con 0x7f0a2c09bca0 2023-10-19T12:44:20.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:20.212+0000 7f0a2b7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0a2c069720 0x7f0a2c150bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:20.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:20.212+0000 7f0a2b7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0a2c069720 0x7f0a2c150bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:53030/0 (socket says 172.21.15.57:53030) 2023-10-19T12:44:20.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:20.212+0000 7f0a2b7fe640 1 -- 172.21.15.57:0/1522802658 learned_addr learned my addr 172.21.15.57:0/1522802658 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:20.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:20.212+0000 7f0a2bfff640 1 --2- 172.21.15.57:0/1522802658 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0a2c158d40 0x7f0a2c156e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:20.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:20.212+0000 7f0a2affd640 1 --2- 172.21.15.57:0/1522802658 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0a2c09bca0 0x7f0a2c159d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:20.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:20.212+0000 7f0a2b7fe640 1 -- 172.21.15.57:0/1522802658 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0a2c09bca0 msgr2=0x7f0a2c159d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:20.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:20.212+0000 7f0a2b7fe640 1 --2- 172.21.15.57:0/1522802658 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0a2c09bca0 0x7f0a2c159d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:20.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:20.212+0000 7f0a2b7fe640 1 -- 172.21.15.57:0/1522802658 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0a2c158d40 msgr2=0x7f0a2c156e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:20.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:20.212+0000 7f0a2b7fe640 1 --2- 172.21.15.57:0/1522802658 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0a2c158d40 0x7f0a2c156e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:20.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:20.212+0000 7f0a2b7fe640 1 -- 172.21.15.57:0/1522802658 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a24009900 con 0x7f0a2c069720 2023-10-19T12:44:20.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:20.212+0000 7f0a2bfff640 1 --2- 172.21.15.57:0/1522802658 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0a2c158d40 0x7f0a2c156e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:44:20.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:20.213+0000 7f0a2b7fe640 1 --2- 172.21.15.57:0/1522802658 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0a2c069720 0x7f0a2c150bc0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f0a1c00b410 tx=0x7f0a1c00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:20.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:20.213+0000 7f0a13fff640 1 -- 172.21.15.57:0/1522802658 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a1c011830 con 0x7f0a2c069720 2023-10-19T12:44:20.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:20.213+0000 7f0a13fff640 1 -- 172.21.15.57:0/1522802658 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f0a1c011aa0 con 0x7f0a2c069720 2023-10-19T12:44:20.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:20.213+0000 7f0a13fff640 1 -- 172.21.15.57:0/1522802658 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a1c010380 con 0x7f0a2c069720 2023-10-19T12:44:20.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:20.213+0000 7f0a31d05640 1 -- 172.21.15.57:0/1522802658 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a2c159a40 con 0x7f0a2c069720 2023-10-19T12:44:20.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:20.213+0000 7f0a31d05640 1 -- 172.21.15.57:0/1522802658 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a2c156950 con 0x7f0a2c069720 2023-10-19T12:44:20.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:20.214+0000 7f0a31d05640 1 -- 172.21.15.57:0/1522802658 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a2c156b70 con 0x7f0a2c069720 2023-10-19T12:44:20.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:20.214+0000 7f0a13fff640 1 -- 172.21.15.57:0/1522802658 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f0a1c0104e0 con 0x7f0a2c069720 2023-10-19T12:44:20.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:20.215+0000 7f0a13fff640 1 --2- 172.21.15.57:0/1522802658 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f0a1806ae40 0x7f0a1806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:20.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:20.215+0000 7f0a13fff640 1 -- 172.21.15.57:0/1522802658 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5935+0+0 (secure 0 0 0) 0x7f0a1c089b80 con 0x7f0a2c069720 2023-10-19T12:44:20.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:20.215+0000 7f0a2affd640 1 --2- 172.21.15.57:0/1522802658 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f0a1806ae40 0x7f0a1806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:20.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:20.215+0000 7f0a2affd640 1 --2- 172.21.15.57:0/1522802658 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f0a1806ae40 0x7f0a1806d300 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c158750 tx=0x7f0a20009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:20.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:20.219+0000 7f0a13fff640 1 -- 172.21.15.57:0/1522802658 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0a1c054ad0 con 0x7f0a2c069720 2023-10-19T12:44:20.398 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:20.395+0000 7f0a31d05640 1 -- 172.21.15.57:0/1522802658 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f0a2c068e50 con 0x7f0a2c069720 2023-10-19T12:44:22.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:22.098+0000 7f0a13fff640 1 -- 172.21.15.57:0/1522802658 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v23) v1 ==== 228+0+0 (secure 0 0 0) 0x7f0a1c0580f0 con 0x7f0a2c069720 2023-10-19T12:44:22.102 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'cephfs_data_ec' created 2023-10-19T12:44:22.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:22.101+0000 7f0a31d05640 1 -- 172.21.15.57:0/1522802658 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f0a1806ae40 msgr2=0x7f0a1806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:22.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:22.101+0000 7f0a31d05640 1 --2- 172.21.15.57:0/1522802658 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f0a1806ae40 0x7f0a1806d300 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c158750 tx=0x7f0a20009040 comp rx=0 tx=0).stop 2023-10-19T12:44:22.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:22.101+0000 7f0a31d05640 1 -- 172.21.15.57:0/1522802658 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0a2c069720 msgr2=0x7f0a2c150bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:22.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:22.101+0000 7f0a31d05640 1 --2- 172.21.15.57:0/1522802658 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0a2c069720 0x7f0a2c150bc0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f0a1c00b410 tx=0x7f0a1c00b8d0 comp rx=0 tx=0).stop 2023-10-19T12:44:22.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:22.101+0000 7f0a31d05640 1 -- 172.21.15.57:0/1522802658 shutdown_connections 2023-10-19T12:44:22.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:22.101+0000 7f0a31d05640 1 --2- 172.21.15.57:0/1522802658 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f0a1806ae40 0x7f0a1806d300 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:22.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:22.101+0000 7f0a31d05640 1 --2- 172.21.15.57:0/1522802658 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0a2c158d40 0x7f0a2c156e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:22.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:22.101+0000 7f0a31d05640 1 --2- 172.21.15.57:0/1522802658 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0a2c09bca0 0x7f0a2c159d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:22.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:22.101+0000 7f0a31d05640 1 --2- 172.21.15.57:0/1522802658 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0a2c069720 0x7f0a2c150bc0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:22.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:22.101+0000 7f0a31d05640 1 -- 172.21.15.57:0/1522802658 >> 172.21.15.57:0/1522802658 conn(0x7f0a2c070340 msgr2=0x7f0a2c1e4760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:22.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:22.101+0000 7f0a31d05640 1 -- 172.21.15.57:0/1522802658 shutdown_connections 2023-10-19T12:44:22.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:22.101+0000 7f0a31d05640 1 -- 172.21.15.57:0/1522802658 wait complete. 2023-10-19T12:44:22.123 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2023-10-19T12:44:22.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:22.239+0000 7f3221933640 1 -- 172.21.15.57:0/3147802745 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f321c0cf8b0 msgr2=0x7f321c0ce9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:22.243 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:22.239+0000 7f3221933640 1 --2- 172.21.15.57:0/3147802745 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f321c0cf8b0 0x7f321c0ce9a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3204005fd0 tx=0x7f320402bd90 comp rx=0 tx=0).stop 2023-10-19T12:44:22.243 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:22.239+0000 7f3221933640 1 -- 172.21.15.57:0/3147802745 shutdown_connections 2023-10-19T12:44:22.243 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:22.239+0000 7f3221933640 1 --2- 172.21.15.57:0/3147802745 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f321c0b1ca0 0x7f321c0b42d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:22.243 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:22.239+0000 7f3221933640 1 --2- 172.21.15.57:0/3147802745 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f321c0cf8b0 0x7f321c0ce9a0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:22.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:22.239+0000 7f3221933640 1 -- 172.21.15.57:0/3147802745 >> 172.21.15.57:0/3147802745 conn(0x7f321c0d0c50 msgr2=0x7f321c0d1060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:22.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:22.239+0000 7f3221933640 1 -- 172.21.15.57:0/3147802745 shutdown_connections 2023-10-19T12:44:22.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:22.239+0000 7f3221933640 1 -- 172.21.15.57:0/3147802745 wait complete. 2023-10-19T12:44:22.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:22.240+0000 7f3221933640 1 Processor -- start 2023-10-19T12:44:22.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:22.240+0000 7f3221933640 1 -- start start 2023-10-19T12:44:22.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:22.241+0000 7f3221933640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f321c0b1ca0 0x7f321c0b2940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:22.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:22.241+0000 7f3221933640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f321c0cf8b0 0x7f321c1f1690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:22.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:22.241+0000 7f3221933640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f321c15e5e0 0x7f321c149e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:22.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:22.241+0000 7f3221933640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f321c1f21a0 con 0x7f321c0cf8b0 2023-10-19T12:44:22.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:22.241+0000 7f3221933640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f321c1f2480 con 0x7f321c0b1ca0 2023-10-19T12:44:22.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:22.241+0000 7f3221933640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f321c0afb80 con 0x7f321c15e5e0 2023-10-19T12:44:22.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:22.241+0000 7f321a7fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f321c0cf8b0 0x7f321c1f1690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:22.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:22.241+0000 7f321a7fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f321c0cf8b0 0x7f321c1f1690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:53044/0 (socket says 172.21.15.57:53044) 2023-10-19T12:44:22.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:22.241+0000 7f321a7fc640 1 -- 172.21.15.57:0/2261080733 learned_addr learned my addr 172.21.15.57:0/2261080733 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:22.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:22.241+0000 7f321b7fe640 1 --2- 172.21.15.57:0/2261080733 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f321c15e5e0 0x7f321c149e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:22.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:22.241+0000 7f321affd640 1 --2- 172.21.15.57:0/2261080733 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f321c0b1ca0 0x7f321c0b2940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:22.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:22.241+0000 7f321a7fc640 1 -- 172.21.15.57:0/2261080733 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f321c15e5e0 msgr2=0x7f321c149e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:22.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:22.241+0000 7f321a7fc640 1 --2- 172.21.15.57:0/2261080733 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f321c15e5e0 0x7f321c149e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:22.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:22.242+0000 7f321a7fc640 1 -- 172.21.15.57:0/2261080733 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f321c0b1ca0 msgr2=0x7f321c0b2940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:22.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:22.242+0000 7f321a7fc640 1 --2- 172.21.15.57:0/2261080733 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f321c0b1ca0 0x7f321c0b2940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:22.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:22.242+0000 7f321a7fc640 1 -- 172.21.15.57:0/2261080733 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3204009560 con 0x7f321c0cf8b0 2023-10-19T12:44:22.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:22.242+0000 7f321a7fc640 1 --2- 172.21.15.57:0/2261080733 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f321c0cf8b0 0x7f321c1f1690 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f3204006010 tx=0x7f320402db30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:22.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:22.242+0000 7f3213fff640 1 -- 172.21.15.57:0/2261080733 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32040035f0 con 0x7f321c0cf8b0 2023-10-19T12:44:22.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:22.242+0000 7f3213fff640 1 -- 172.21.15.57:0/2261080733 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f3204003840 con 0x7f321c0cf8b0 2023-10-19T12:44:22.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:22.242+0000 7f3221933640 1 -- 172.21.15.57:0/2261080733 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f321c0afe60 con 0x7f321c0cf8b0 2023-10-19T12:44:22.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:22.242+0000 7f3213fff640 1 -- 172.21.15.57:0/2261080733 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3204003c50 con 0x7f321c0cf8b0 2023-10-19T12:44:22.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:22.242+0000 7f3221933640 1 -- 172.21.15.57:0/2261080733 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f321c0b5970 con 0x7f321c0cf8b0 2023-10-19T12:44:22.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:22.243+0000 7f3221933640 1 -- 172.21.15.57:0/2261080733 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f321c0690e0 con 0x7f321c0cf8b0 2023-10-19T12:44:22.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:22.243+0000 7f3213fff640 1 -- 172.21.15.57:0/2261080733 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f320403e450 con 0x7f321c0cf8b0 2023-10-19T12:44:22.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:22.244+0000 7f3213fff640 1 --2- 172.21.15.57:0/2261080733 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f31fc06ae40 0x7f31fc06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:22.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:22.244+0000 7f3213fff640 1 -- 172.21.15.57:0/2261080733 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6428+0+0 (secure 0 0 0) 0x7f32040ac3c0 con 0x7f321c0cf8b0 2023-10-19T12:44:22.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:22.244+0000 7f321affd640 1 --2- 172.21.15.57:0/2261080733 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f31fc06ae40 0x7f31fc06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:22.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:22.244+0000 7f321affd640 1 --2- 172.21.15.57:0/2261080733 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f31fc06ae40 0x7f31fc06d300 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3214006fd0 tx=0x7f3214008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:22.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:22.248+0000 7f3213fff640 1 -- 172.21.15.57:0/2261080733 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f32040772e0 con 0x7f321c0cf8b0 2023-10-19T12:44:22.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:22.422+0000 7f3221933640 1 -- 172.21.15.57:0/2261080733 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) v1 -- 0x7f321c069ca0 con 0x7f321c0cf8b0 2023-10-19T12:44:23.031 INFO:tasks.ceph.mds.d.smithi179.stderr:2023-10-19T12:44:23.029+0000 7f7aa057d640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-19T12:44:23.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:23.100+0000 7f3213fff640 1 -- 172.21.15.57:0/2261080733 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v24) v1 ==== 169+0+0 (secure 0 0 0) 0x7f3204003db0 con 0x7f321c0cf8b0 2023-10-19T12:44:23.103 INFO:teuthology.orchestra.run.smithi057.stderr:set pool 4 allow_ec_overwrites to true 2023-10-19T12:44:23.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:23.102+0000 7f3221933640 1 -- 172.21.15.57:0/2261080733 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f31fc06ae40 msgr2=0x7f31fc06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:23.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:23.102+0000 7f3221933640 1 --2- 172.21.15.57:0/2261080733 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f31fc06ae40 0x7f31fc06d300 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3214006fd0 tx=0x7f3214008040 comp rx=0 tx=0).stop 2023-10-19T12:44:23.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:23.102+0000 7f3221933640 1 -- 172.21.15.57:0/2261080733 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f321c0cf8b0 msgr2=0x7f321c1f1690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:23.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:23.102+0000 7f3221933640 1 --2- 172.21.15.57:0/2261080733 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f321c0cf8b0 0x7f321c1f1690 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f3204006010 tx=0x7f320402db30 comp rx=0 tx=0).stop 2023-10-19T12:44:23.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:23.103+0000 7f3221933640 1 -- 172.21.15.57:0/2261080733 shutdown_connections 2023-10-19T12:44:23.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:23.103+0000 7f3221933640 1 --2- 172.21.15.57:0/2261080733 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f31fc06ae40 0x7f31fc06d300 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:23.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:23.103+0000 7f3221933640 1 --2- 172.21.15.57:0/2261080733 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f321c15e5e0 0x7f321c149e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:23.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:23.103+0000 7f3221933640 1 --2- 172.21.15.57:0/2261080733 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f321c0cf8b0 0x7f321c1f1690 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:23.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:23.103+0000 7f3221933640 1 --2- 172.21.15.57:0/2261080733 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f321c0b1ca0 0x7f321c0b2940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:23.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:23.103+0000 7f3221933640 1 -- 172.21.15.57:0/2261080733 >> 172.21.15.57:0/2261080733 conn(0x7f321c0d0c50 msgr2=0x7f321c0cf1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:23.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:23.103+0000 7f3221933640 1 -- 172.21.15.57:0/2261080733 shutdown_connections 2023-10-19T12:44:23.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:23.103+0000 7f3221933640 1 -- 172.21.15.57:0/2261080733 wait complete. 2023-10-19T12:44:23.127 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2023-10-19T12:44:23.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:23.243+0000 7fcb34fb6640 1 -- 172.21.15.57:0/3272448156 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcb300ad370 msgr2=0x7fcb300ad800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:23.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:23.243+0000 7fcb34fb6640 1 --2- 172.21.15.57:0/3272448156 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcb300ad370 0x7fcb300ad800 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fcb3006fc50 tx=0x7fcb2802c2f0 comp rx=0 tx=0).stop 2023-10-19T12:44:23.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:23.243+0000 7fcb34fb6640 1 -- 172.21.15.57:0/3272448156 shutdown_connections 2023-10-19T12:44:23.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:23.243+0000 7fcb34fb6640 1 --2- 172.21.15.57:0/3272448156 >> v2:172.21.15.179:3300/0 conn(0x7fcb3006f0a0 0x7fcb3006ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:23.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:23.243+0000 7fcb34fb6640 1 --2- 172.21.15.57:0/3272448156 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcb300ad370 0x7fcb300ad800 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:23.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:23.243+0000 7fcb34fb6640 1 -- 172.21.15.57:0/3272448156 >> 172.21.15.57:0/3272448156 conn(0x7fcb300b0670 msgr2=0x7fcb300b0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:23.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:23.243+0000 7fcb34fb6640 1 -- 172.21.15.57:0/3272448156 shutdown_connections 2023-10-19T12:44:23.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:23.243+0000 7fcb34fb6640 1 -- 172.21.15.57:0/3272448156 wait complete. 2023-10-19T12:44:23.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:23.244+0000 7fcb34fb6640 1 Processor -- start 2023-10-19T12:44:23.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:23.244+0000 7fcb34fb6640 1 -- start start 2023-10-19T12:44:23.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:23.245+0000 7fcb34fb6640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcb3006d2d0 0x7fcb301568f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:23.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:23.245+0000 7fcb34fb6640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcb3006f0a0 0x7fcb3015b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:23.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:23.245+0000 7fcb34fb6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcb300ad370 0x7fcb3015b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:23.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:23.245+0000 7fcb34fb6640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb30156e30 con 0x7fcb300ad370 2023-10-19T12:44:23.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:23.245+0000 7fcb34fb6640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb3014d010 con 0x7fcb3006f0a0 2023-10-19T12:44:23.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:23.245+0000 7fcb34fb6640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb30155260 con 0x7fcb3006d2d0 2023-10-19T12:44:23.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:23.245+0000 7fcb2ed9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcb300ad370 0x7fcb3015b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:23.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:23.245+0000 7fcb2ed9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcb300ad370 0x7fcb3015b790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:53074/0 (socket says 172.21.15.57:53074) 2023-10-19T12:44:23.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:23.245+0000 7fcb2ed9d640 1 -- 172.21.15.57:0/966504986 learned_addr learned my addr 172.21.15.57:0/966504986 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:23.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:23.245+0000 7fcb2e59c640 1 --2- 172.21.15.57:0/966504986 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcb3006d2d0 0x7fcb301568f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:23.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:23.245+0000 7fcb2dd9b640 1 --2- 172.21.15.57:0/966504986 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcb3006f0a0 0x7fcb3015b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:23.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:23.245+0000 7fcb2ed9d640 1 -- 172.21.15.57:0/966504986 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcb3006d2d0 msgr2=0x7fcb301568f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:23.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:23.245+0000 7fcb2ed9d640 1 --2- 172.21.15.57:0/966504986 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcb3006d2d0 0x7fcb301568f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:23.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:23.245+0000 7fcb2ed9d640 1 -- 172.21.15.57:0/966504986 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcb3006f0a0 msgr2=0x7fcb3015b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:23.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:23.245+0000 7fcb2ed9d640 1 --2- 172.21.15.57:0/966504986 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcb3006f0a0 0x7fcb3015b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:23.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:23.245+0000 7fcb2ed9d640 1 -- 172.21.15.57:0/966504986 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb28009900 con 0x7fcb300ad370 2023-10-19T12:44:23.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:23.245+0000 7fcb2e59c640 1 --2- 172.21.15.57:0/966504986 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcb3006d2d0 0x7fcb301568f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:44:23.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:23.245+0000 7fcb2dd9b640 1 --2- 172.21.15.57:0/966504986 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcb3006f0a0 0x7fcb3015b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:44:23.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:23.245+0000 7fcb2ed9d640 1 --2- 172.21.15.57:0/966504986 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcb300ad370 0x7fcb3015b790 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fcb2802c900 tx=0x7fcb28035ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:23.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:23.245+0000 7fcb1effd640 1 -- 172.21.15.57:0/966504986 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb28003e60 con 0x7fcb300ad370 2023-10-19T12:44:23.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:23.246+0000 7fcb1effd640 1 -- 172.21.15.57:0/966504986 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fcb28004340 con 0x7fcb300ad370 2023-10-19T12:44:23.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:23.246+0000 7fcb34fb6640 1 -- 172.21.15.57:0/966504986 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb30155540 con 0x7fcb300ad370 2023-10-19T12:44:23.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:23.246+0000 7fcb1effd640 1 -- 172.21.15.57:0/966504986 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb28004750 con 0x7fcb300ad370 2023-10-19T12:44:23.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:23.246+0000 7fcb34fb6640 1 -- 172.21.15.57:0/966504986 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb30155b00 con 0x7fcb300ad370 2023-10-19T12:44:23.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:23.247+0000 7fcb34fb6640 1 -- 172.21.15.57:0/966504986 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb30155cc0 con 0x7fcb300ad370 2023-10-19T12:44:23.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:23.247+0000 7fcb1effd640 1 -- 172.21.15.57:0/966504986 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fcb2800a040 con 0x7fcb300ad370 2023-10-19T12:44:23.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:23.247+0000 7fcb1effd640 1 --2- 172.21.15.57:0/966504986 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fcb1006ae40 0x7fcb1006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:23.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:23.248+0000 7fcb1effd640 1 -- 172.21.15.57:0/966504986 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6428+0+0 (secure 0 0 0) 0x7fcb280ac610 con 0x7fcb300ad370 2023-10-19T12:44:23.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:23.248+0000 7fcb2e59c640 1 --2- 172.21.15.57:0/966504986 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fcb1006ae40 0x7fcb1006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:23.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:23.248+0000 7fcb2e59c640 1 --2- 172.21.15.57:0/966504986 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fcb1006ae40 0x7fcb1006d300 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fcb20000f60 tx=0x7fcb20009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:23.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:23.252+0000 7fcb1effd640 1 -- 172.21.15.57:0/966504986 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fcb28077480 con 0x7fcb300ad370 2023-10-19T12:44:23.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:23.471+0000 7fcb34fb6640 1 -- 172.21.15.57:0/966504986 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) v1 -- 0x7fcb300ce1d0 con 0x7fcb300ad370 2023-10-19T12:44:24.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.124+0000 7fcb1effd640 1 -- 172.21.15.57:0/966504986 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v7) v1 ==== 137+0+0 (secure 0 0 0) 0x7fcb2807aaa0 con 0x7fcb300ad370 2023-10-19T12:44:24.128 INFO:teuthology.orchestra.run.smithi057.stderr:added data pool 4 to fsmap 2023-10-19T12:44:24.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.127+0000 7fcb34fb6640 1 -- 172.21.15.57:0/966504986 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fcb1006ae40 msgr2=0x7fcb1006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:24.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.127+0000 7fcb34fb6640 1 --2- 172.21.15.57:0/966504986 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fcb1006ae40 0x7fcb1006d300 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fcb20000f60 tx=0x7fcb20009040 comp rx=0 tx=0).stop 2023-10-19T12:44:24.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.127+0000 7fcb34fb6640 1 -- 172.21.15.57:0/966504986 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcb300ad370 msgr2=0x7fcb3015b790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:24.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.127+0000 7fcb34fb6640 1 --2- 172.21.15.57:0/966504986 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcb300ad370 0x7fcb3015b790 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fcb2802c900 tx=0x7fcb28035ed0 comp rx=0 tx=0).stop 2023-10-19T12:44:24.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.127+0000 7fcb34fb6640 1 -- 172.21.15.57:0/966504986 shutdown_connections 2023-10-19T12:44:24.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.127+0000 7fcb34fb6640 1 --2- 172.21.15.57:0/966504986 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fcb1006ae40 0x7fcb1006d300 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:24.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.127+0000 7fcb34fb6640 1 --2- 172.21.15.57:0/966504986 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcb300ad370 0x7fcb3015b790 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:24.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.127+0000 7fcb34fb6640 1 --2- 172.21.15.57:0/966504986 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcb3006f0a0 0x7fcb3015b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:24.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.127+0000 7fcb34fb6640 1 --2- 172.21.15.57:0/966504986 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcb3006d2d0 0x7fcb301568f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:24.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.127+0000 7fcb34fb6640 1 -- 172.21.15.57:0/966504986 >> 172.21.15.57:0/966504986 conn(0x7fcb300b0670 msgr2=0x7fcb301577b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:24.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.127+0000 7fcb34fb6640 1 -- 172.21.15.57:0/966504986 shutdown_connections 2023-10-19T12:44:24.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.127+0000 7fcb34fb6640 1 -- 172.21.15.57:0/966504986 wait complete. 2023-10-19T12:44:24.143 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T12:44:24.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.255+0000 7f1e88a6b640 1 -- 172.21.15.57:0/432791880 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1e84141690 msgr2=0x7f1e840703e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:24.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.255+0000 7f1e88a6b640 1 --2- 172.21.15.57:0/432791880 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1e84141690 0x7f1e840703e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f1e840cbd00 tx=0x7f1e6c02bdb0 comp rx=0 tx=0).stop 2023-10-19T12:44:24.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.255+0000 7f1e88a6b640 1 -- 172.21.15.57:0/432791880 shutdown_connections 2023-10-19T12:44:24.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.255+0000 7f1e88a6b640 1 --2- 172.21.15.57:0/432791880 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1e84141690 0x7f1e840703e0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:24.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.255+0000 7f1e88a6b640 1 --2- 172.21.15.57:0/432791880 >> v2:172.21.15.179:3300/0 conn(0x7f1e8413e520 0x7f1e8406bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:24.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.255+0000 7f1e88a6b640 1 -- 172.21.15.57:0/432791880 >> 172.21.15.57:0/432791880 conn(0x7f1e84069930 msgr2=0x7f1e84069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:24.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.255+0000 7f1e88a6b640 1 -- 172.21.15.57:0/432791880 shutdown_connections 2023-10-19T12:44:24.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.255+0000 7f1e88a6b640 1 -- 172.21.15.57:0/432791880 wait complete. 2023-10-19T12:44:24.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.256+0000 7f1e88a6b640 1 Processor -- start 2023-10-19T12:44:24.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.256+0000 7f1e88a6b640 1 -- start start 2023-10-19T12:44:24.262 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.258+0000 7f1e88a6b640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1e8406c0a0 0x7f1e841e5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:24.263 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.258+0000 7f1e88a6b640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1e8413e520 0x7f1e841df9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:24.263 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.258+0000 7f1e88a6b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1e84141690 0x7f1e841dcc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:24.263 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.258+0000 7f1e88a6b640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e841e5540 con 0x7f1e84141690 2023-10-19T12:44:24.263 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.258+0000 7f1e88a6b640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e841e3ff0 con 0x7f1e8406c0a0 2023-10-19T12:44:24.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.258+0000 7f1e88a6b640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e841e4b70 con 0x7f1e8413e520 2023-10-19T12:44:24.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.259+0000 7f1e82d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1e84141690 0x7f1e841dcc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:24.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.259+0000 7f1e8259c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1e8406c0a0 0x7f1e841e5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:24.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.259+0000 7f1e8259c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1e8406c0a0 0x7f1e841e5000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.57:45802/0 (socket says 172.21.15.57:45802) 2023-10-19T12:44:24.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.259+0000 7f1e82d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1e84141690 0x7f1e841dcc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:53096/0 (socket says 172.21.15.57:53096) 2023-10-19T12:44:24.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.259+0000 7f1e8259c640 1 -- 172.21.15.57:0/1609006021 learned_addr learned my addr 172.21.15.57:0/1609006021 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:24.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.259+0000 7f1e7bfff640 1 --2- 172.21.15.57:0/1609006021 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1e8413e520 0x7f1e841df9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:24.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.259+0000 7f1e82d9d640 1 -- 172.21.15.57:0/1609006021 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1e8413e520 msgr2=0x7f1e841df9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:24.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.259+0000 7f1e82d9d640 1 --2- 172.21.15.57:0/1609006021 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1e8413e520 0x7f1e841df9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:24.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.259+0000 7f1e82d9d640 1 -- 172.21.15.57:0/1609006021 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1e8406c0a0 msgr2=0x7f1e841e5000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:24.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.259+0000 7f1e82d9d640 1 --2- 172.21.15.57:0/1609006021 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1e8406c0a0 0x7f1e841e5000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:24.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.259+0000 7f1e82d9d640 1 -- 172.21.15.57:0/1609006021 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e6c009560 con 0x7f1e84141690 2023-10-19T12:44:24.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.259+0000 7f1e8259c640 1 --2- 172.21.15.57:0/1609006021 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1e8406c0a0 0x7f1e841e5000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T12:44:24.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.259+0000 7f1e82d9d640 1 --2- 172.21.15.57:0/1609006021 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1e84141690 0x7f1e841dcc20 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f1e7c00c760 tx=0x7f1e7c00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:24.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.259+0000 7f1e7affd640 1 -- 172.21.15.57:0/1609006021 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e7c006d50 con 0x7f1e84141690 2023-10-19T12:44:24.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.259+0000 7f1e7affd640 1 -- 172.21.15.57:0/1609006021 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f1e7c00ce10 con 0x7f1e84141690 2023-10-19T12:44:24.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.259+0000 7f1e88a6b640 1 -- 172.21.15.57:0/1609006021 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e841e2ca0 con 0x7f1e84141690 2023-10-19T12:44:24.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.259+0000 7f1e7affd640 1 -- 172.21.15.57:0/1609006021 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e7c003bd0 con 0x7f1e84141690 2023-10-19T12:44:24.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.259+0000 7f1e88a6b640 1 -- 172.21.15.57:0/1609006021 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e841e4520 con 0x7f1e84141690 2023-10-19T12:44:24.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.260+0000 7f1e88a6b640 1 -- 172.21.15.57:0/1609006021 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e840690e0 con 0x7f1e84141690 2023-10-19T12:44:24.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.261+0000 7f1e7affd640 1 -- 172.21.15.57:0/1609006021 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f1e7c003d30 con 0x7f1e84141690 2023-10-19T12:44:24.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.261+0000 7f1e7affd640 1 --2- 172.21.15.57:0/1609006021 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f1e6406ae40 0x7f1e6406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:24.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.261+0000 7f1e7affd640 1 -- 172.21.15.57:0/1609006021 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f1e7c089500 con 0x7f1e84141690 2023-10-19T12:44:24.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.261+0000 7f1e8259c640 1 --2- 172.21.15.57:0/1609006021 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f1e6406ae40 0x7f1e6406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:24.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.262+0000 7f1e8259c640 1 --2- 172.21.15.57:0/1609006021 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f1e6406ae40 0x7f1e6406d300 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1e74005fd0 tx=0x7f1e7400b480 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:24.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.265+0000 7f1e7affd640 1 -- 172.21.15.57:0/1609006021 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f1e7c054380 con 0x7f1e84141690 2023-10-19T12:44:24.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.476+0000 7f1e88a6b640 1 -- 172.21.15.57:0/1609006021 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1e841e4cb0 con 0x7f1e84141690 2023-10-19T12:44:24.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.476+0000 7f1e7affd640 1 -- 172.21.15.57:0/1609006021 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4906 (secure 0 0 0) 0x7f1e7c0579a0 con 0x7f1e84141690 2023-10-19T12:44:24.480 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:44:24.481 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4357,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/4228712742","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":4228712742},{"type":"v1","addr":"172.21.15.57:6835","nonce":4228712742}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4360,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/971892630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":971892630},{"type":"v1","addr":"172.21.15.57:6834","nonce":971892630}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4372,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/104495983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":104495983},{"type":"v1","addr":"172.21.15.179:6836","nonce":104495983}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T12:44:18.419669+0000","modified":"2023-10-19T12:44:23.472554+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4374},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4374":{"gid":4374,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.179:6837/622550365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":622550365},{"type":"v1","addr":"172.21.15.179:6837","nonce":622550365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2023-10-19T12:44:24.482 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 7 2023-10-19T12:44:24.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.478+0000 7f1e88a6b640 1 -- 172.21.15.57:0/1609006021 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f1e6406ae40 msgr2=0x7f1e6406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:24.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.478+0000 7f1e88a6b640 1 --2- 172.21.15.57:0/1609006021 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f1e6406ae40 0x7f1e6406d300 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1e74005fd0 tx=0x7f1e7400b480 comp rx=0 tx=0).stop 2023-10-19T12:44:24.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.478+0000 7f1e88a6b640 1 -- 172.21.15.57:0/1609006021 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1e84141690 msgr2=0x7f1e841dcc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:24.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.478+0000 7f1e88a6b640 1 --2- 172.21.15.57:0/1609006021 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1e84141690 0x7f1e841dcc20 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f1e7c00c760 tx=0x7f1e7c00cc20 comp rx=0 tx=0).stop 2023-10-19T12:44:24.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.478+0000 7f1e88a6b640 1 -- 172.21.15.57:0/1609006021 shutdown_connections 2023-10-19T12:44:24.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.478+0000 7f1e88a6b640 1 --2- 172.21.15.57:0/1609006021 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f1e6406ae40 0x7f1e6406d300 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:24.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.478+0000 7f1e88a6b640 1 --2- 172.21.15.57:0/1609006021 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1e84141690 0x7f1e841dcc20 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:24.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.478+0000 7f1e88a6b640 1 --2- 172.21.15.57:0/1609006021 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1e8413e520 0x7f1e841df9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:24.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.478+0000 7f1e88a6b640 1 --2- 172.21.15.57:0/1609006021 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1e8406c0a0 0x7f1e841e5000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:24.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.478+0000 7f1e88a6b640 1 -- 172.21.15.57:0/1609006021 >> 172.21.15.57:0/1609006021 conn(0x7f1e84069930 msgr2=0x7f1e840c9810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:24.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.478+0000 7f1e88a6b640 1 -- 172.21.15.57:0/1609006021 shutdown_connections 2023-10-19T12:44:24.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.478+0000 7f1e88a6b640 1 -- 172.21.15.57:0/1609006021 wait complete. 2023-10-19T12:44:24.492 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T12:44:24.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.598+0000 7f8f0e5b6640 1 -- 172.21.15.57:0/3262306846 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8f0815b260 msgr2=0x7f8f080cf310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:24.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.598+0000 7f8f0e5b6640 1 --2- 172.21.15.57:0/3262306846 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8f0815b260 0x7f8f080cf310 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f8ef8009830 tx=0x7f8ef802bd60 comp rx=0 tx=0).stop 2023-10-19T12:44:24.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.598+0000 7f8f0e5b6640 1 -- 172.21.15.57:0/3262306846 shutdown_connections 2023-10-19T12:44:24.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.598+0000 7f8f0e5b6640 1 --2- 172.21.15.57:0/3262306846 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8f0815b260 0x7f8f080cf310 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:24.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.598+0000 7f8f0e5b6640 1 -- 172.21.15.57:0/3262306846 >> 172.21.15.57:0/3262306846 conn(0x7f8f0809a030 msgr2=0x7f8f080d0d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:24.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.598+0000 7f8f0e5b6640 1 -- 172.21.15.57:0/3262306846 shutdown_connections 2023-10-19T12:44:24.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.598+0000 7f8f0e5b6640 1 -- 172.21.15.57:0/3262306846 wait complete. 2023-10-19T12:44:24.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.599+0000 7f8f0e5b6640 1 Processor -- start 2023-10-19T12:44:24.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.600+0000 7f8f0e5b6640 1 -- start start 2023-10-19T12:44:24.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.600+0000 7f8f0e5b6640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8f080cf850 0x7f8f0809e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:24.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.600+0000 7f8f0e5b6640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8f0815b260 0x7f8f080b62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:24.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.600+0000 7f8f0e5b6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8f0815b640 0x7f8f080b4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:24.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.600+0000 7f8f0e5b6640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f080b6830 con 0x7f8f0815b640 2023-10-19T12:44:24.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.600+0000 7f8f0e5b6640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f080b4f70 con 0x7f8f080cf850 2023-10-19T12:44:24.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.600+0000 7f8f0e5b6640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f080b5250 con 0x7f8f0815b260 2023-10-19T12:44:24.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.600+0000 7f8f0cb53640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8f0815b640 0x7f8f080b4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:24.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.601+0000 7f8f0cb53640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8f0815b640 0x7f8f080b4350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:53104/0 (socket says 172.21.15.57:53104) 2023-10-19T12:44:24.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.601+0000 7f8f0cb53640 1 -- 172.21.15.57:0/1964602705 learned_addr learned my addr 172.21.15.57:0/1964602705 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:24.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.601+0000 7f8f07fff640 1 --2- 172.21.15.57:0/1964602705 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8f080cf850 0x7f8f0809e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:24.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.601+0000 7f8f077fe640 1 --2- 172.21.15.57:0/1964602705 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8f0815b260 0x7f8f080b62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:24.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.601+0000 7f8f0cb53640 1 -- 172.21.15.57:0/1964602705 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8f0815b260 msgr2=0x7f8f080b62f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:24.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.601+0000 7f8f0cb53640 1 --2- 172.21.15.57:0/1964602705 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8f0815b260 0x7f8f080b62f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:24.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.601+0000 7f8f0cb53640 1 -- 172.21.15.57:0/1964602705 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8f080cf850 msgr2=0x7f8f0809e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:24.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.601+0000 7f8f0cb53640 1 --2- 172.21.15.57:0/1964602705 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8f080cf850 0x7f8f0809e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:24.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.601+0000 7f8f0cb53640 1 -- 172.21.15.57:0/1964602705 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ef8009490 con 0x7f8f0815b640 2023-10-19T12:44:24.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.601+0000 7f8f07fff640 1 --2- 172.21.15.57:0/1964602705 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8f080cf850 0x7f8f0809e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:44:24.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.601+0000 7f8f077fe640 1 --2- 172.21.15.57:0/1964602705 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8f0815b260 0x7f8f080b62f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:44:24.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.601+0000 7f8f0cb53640 1 --2- 172.21.15.57:0/1964602705 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8f0815b640 0x7f8f080b4350 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f8f0000c780 tx=0x7f8f0000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:24.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.601+0000 7f8f04ff9640 1 -- 172.21.15.57:0/1964602705 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f00004b60 con 0x7f8f0815b640 2023-10-19T12:44:24.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.601+0000 7f8f04ff9640 1 -- 172.21.15.57:0/1964602705 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f8f0000ce90 con 0x7f8f0815b640 2023-10-19T12:44:24.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.601+0000 7f8f0e5b6640 1 -- 172.21.15.57:0/1964602705 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f080b5530 con 0x7f8f0815b640 2023-10-19T12:44:24.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.601+0000 7f8f04ff9640 1 -- 172.21.15.57:0/1964602705 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f0000aab0 con 0x7f8f0815b640 2023-10-19T12:44:24.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.601+0000 7f8f0e5b6640 1 -- 172.21.15.57:0/1964602705 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f080b5af0 con 0x7f8f0815b640 2023-10-19T12:44:24.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.602+0000 7f8f0e5b6640 1 -- 172.21.15.57:0/1964602705 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f080675f0 con 0x7f8f0815b640 2023-10-19T12:44:24.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.603+0000 7f8f04ff9640 1 -- 172.21.15.57:0/1964602705 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f8f0000ac10 con 0x7f8f0815b640 2023-10-19T12:44:24.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.603+0000 7f8f04ff9640 1 --2- 172.21.15.57:0/1964602705 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f8ef406ae40 0x7f8ef406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:24.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.603+0000 7f8f04ff9640 1 -- 172.21.15.57:0/1964602705 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f8f00089d40 con 0x7f8f0815b640 2023-10-19T12:44:24.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.603+0000 7f8f07fff640 1 --2- 172.21.15.57:0/1964602705 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f8ef406ae40 0x7f8ef406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:24.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.604+0000 7f8f07fff640 1 --2- 172.21.15.57:0/1964602705 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f8ef406ae40 0x7f8ef406d300 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8ef8036000 tx=0x7f8ef8004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:24.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.607+0000 7f8f04ff9640 1 -- 172.21.15.57:0/1964602705 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8f00054b00 con 0x7f8f0815b640 2023-10-19T12:44:24.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.785+0000 7f8f0e5b6640 1 -- 172.21.15.57:0/1964602705 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8f080cb300 con 0x7f8f0815b640 2023-10-19T12:44:24.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.786+0000 7f8f04ff9640 1 -- 172.21.15.57:0/1964602705 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18358 (secure 0 0 0) 0x7f8f00058120 con 0x7f8f0815b640 2023-10-19T12:44:24.790 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:44:24.790 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":25,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T12:44:23.472399+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-19T12:44:16.517510+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-19T12:44:17.398297+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-19T12:44:21.129403+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T12:44:24.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.788+0000 7f8f0e5b6640 1 -- 172.21.15.57:0/1964602705 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f8ef406ae40 msgr2=0x7f8ef406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:24.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.788+0000 7f8f0e5b6640 1 --2- 172.21.15.57:0/1964602705 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f8ef406ae40 0x7f8ef406d300 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8ef8036000 tx=0x7f8ef8004730 comp rx=0 tx=0).stop 2023-10-19T12:44:24.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.788+0000 7f8f0e5b6640 1 -- 172.21.15.57:0/1964602705 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8f0815b640 msgr2=0x7f8f080b4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:24.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.788+0000 7f8f0e5b6640 1 --2- 172.21.15.57:0/1964602705 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8f0815b640 0x7f8f080b4350 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f8f0000c780 tx=0x7f8f0000cc40 comp rx=0 tx=0).stop 2023-10-19T12:44:24.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.788+0000 7f8f0e5b6640 1 -- 172.21.15.57:0/1964602705 shutdown_connections 2023-10-19T12:44:24.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.788+0000 7f8f0e5b6640 1 --2- 172.21.15.57:0/1964602705 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f8ef406ae40 0x7f8ef406d300 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:24.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.788+0000 7f8f0e5b6640 1 --2- 172.21.15.57:0/1964602705 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8f0815b640 0x7f8f080b4350 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:24.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.788+0000 7f8f0e5b6640 1 --2- 172.21.15.57:0/1964602705 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8f0815b260 0x7f8f080b62f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:24.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.788+0000 7f8f0e5b6640 1 --2- 172.21.15.57:0/1964602705 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8f080cf850 0x7f8f0809e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:24.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.788+0000 7f8f0e5b6640 1 -- 172.21.15.57:0/1964602705 >> 172.21.15.57:0/1964602705 conn(0x7f8f0809a030 msgr2=0x7f8f080af250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:24.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.788+0000 7f8f0e5b6640 1 -- 172.21.15.57:0/1964602705 shutdown_connections 2023-10-19T12:44:24.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.788+0000 7f8f0e5b6640 1 -- 172.21.15.57:0/1964602705 wait complete. 2023-10-19T12:44:24.802 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T12:44:24.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.907+0000 7f110c3ce640 1 -- 172.21.15.57:0/343655626 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f110406af00 msgr2=0x7f110406b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:24.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.907+0000 7f110c3ce640 1 --2- 172.21.15.57:0/343655626 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f110406af00 0x7f110406b2e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f1100009900 tx=0x7f110002be00 comp rx=0 tx=0).stop 2023-10-19T12:44:24.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.908+0000 7f110c3ce640 1 -- 172.21.15.57:0/343655626 shutdown_connections 2023-10-19T12:44:24.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.908+0000 7f110c3ce640 1 --2- 172.21.15.57:0/343655626 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f11040c96b0 0x7f11040c9b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:24.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.908+0000 7f110c3ce640 1 --2- 172.21.15.57:0/343655626 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f110406af00 0x7f110406b2e0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:24.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.908+0000 7f110c3ce640 1 -- 172.21.15.57:0/343655626 >> 172.21.15.57:0/343655626 conn(0x7f1104069930 msgr2=0x7f1104069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:24.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.908+0000 7f110c3ce640 1 -- 172.21.15.57:0/343655626 shutdown_connections 2023-10-19T12:44:24.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.908+0000 7f110c3ce640 1 -- 172.21.15.57:0/343655626 wait complete. 2023-10-19T12:44:24.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.909+0000 7f110c3ce640 1 Processor -- start 2023-10-19T12:44:24.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.909+0000 7f110c3ce640 1 -- start start 2023-10-19T12:44:24.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.909+0000 7f110c3ce640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f110406af00 0x7f1104152e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:24.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.909+0000 7f110c3ce640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f11040ab500 0x7f1104139de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:24.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.909+0000 7f110c3ce640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f11040c96b0 0x7f11040b0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:24.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.909+0000 7f110c3ce640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11041ef130 con 0x7f11040ab500 2023-10-19T12:44:24.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.909+0000 7f110c3ce640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11040b0ee0 con 0x7f110406af00 2023-10-19T12:44:24.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.910+0000 7f110c3ce640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11040b11c0 con 0x7f11040c96b0 2023-10-19T12:44:24.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.910+0000 7f1109969640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f11040ab500 0x7f1104139de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:24.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.910+0000 7f1109969640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f11040ab500 0x7f1104139de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:53120/0 (socket says 172.21.15.57:53120) 2023-10-19T12:44:24.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.910+0000 7f1109969640 1 -- 172.21.15.57:0/1018409557 learned_addr learned my addr 172.21.15.57:0/1018409557 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:24.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.910+0000 7f110a16a640 1 --2- 172.21.15.57:0/1018409557 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f110406af00 0x7f1104152e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:24.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.910+0000 7f110a96b640 1 --2- 172.21.15.57:0/1018409557 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f11040c96b0 0x7f11040b0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:24.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.910+0000 7f1109969640 1 -- 172.21.15.57:0/1018409557 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f11040c96b0 msgr2=0x7f11040b0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:24.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.910+0000 7f1109969640 1 --2- 172.21.15.57:0/1018409557 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f11040c96b0 0x7f11040b0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:24.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.910+0000 7f1109969640 1 -- 172.21.15.57:0/1018409557 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f110406af00 msgr2=0x7f1104152e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:24.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.910+0000 7f1109969640 1 --2- 172.21.15.57:0/1018409557 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f110406af00 0x7f1104152e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:24.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.910+0000 7f1109969640 1 -- 172.21.15.57:0/1018409557 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1100009560 con 0x7f11040ab500 2023-10-19T12:44:24.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.910+0000 7f1109969640 1 --2- 172.21.15.57:0/1018409557 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f11040ab500 0x7f1104139de0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f10f400d740 tx=0x7f10f400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:24.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.910+0000 7f10faffd640 1 -- 172.21.15.57:0/1018409557 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f4009c30 con 0x7f11040ab500 2023-10-19T12:44:24.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.910+0000 7f10faffd640 1 -- 172.21.15.57:0/1018409557 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f10f4009e80 con 0x7f11040ab500 2023-10-19T12:44:24.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.910+0000 7f10faffd640 1 -- 172.21.15.57:0/1018409557 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f40106c0 con 0x7f11040ab500 2023-10-19T12:44:24.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.910+0000 7f110c3ce640 1 -- 172.21.15.57:0/1018409557 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11040b14a0 con 0x7f11040ab500 2023-10-19T12:44:24.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.910+0000 7f110c3ce640 1 -- 172.21.15.57:0/1018409557 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11040b0bc0 con 0x7f11040ab500 2023-10-19T12:44:24.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.911+0000 7f110c3ce640 1 -- 172.21.15.57:0/1018409557 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1104067930 con 0x7f11040ab500 2023-10-19T12:44:24.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.912+0000 7f10faffd640 1 -- 172.21.15.57:0/1018409557 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f10f4010820 con 0x7f11040ab500 2023-10-19T12:44:24.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.912+0000 7f10faffd640 1 --2- 172.21.15.57:0/1018409557 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f10e806ae40 0x7f10e806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:24.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.912+0000 7f10faffd640 1 -- 172.21.15.57:0/1018409557 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f10f4089cf0 con 0x7f11040ab500 2023-10-19T12:44:24.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.912+0000 7f110a16a640 1 --2- 172.21.15.57:0/1018409557 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f10e806ae40 0x7f10e806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:24.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.913+0000 7f110a16a640 1 --2- 172.21.15.57:0/1018409557 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f10e806ae40 0x7f10e806d300 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f1100035000 tx=0x7f1100004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:24.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:24.916+0000 7f10faffd640 1 -- 172.21.15.57:0/1018409557 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f10f408e070 con 0x7f11040ab500 2023-10-19T12:44:25.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.092+0000 7f110c3ce640 1 -- 172.21.15.57:0/1018409557 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f11040690e0 con 0x7f11040ab500 2023-10-19T12:44:25.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.093+0000 7f10faffd640 1 -- 172.21.15.57:0/1018409557 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18358 (secure 0 0 0) 0x7f10f4021090 con 0x7f11040ab500 2023-10-19T12:44:25.097 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:44:25.097 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":25,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T12:44:23.472399+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-19T12:44:16.517510+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-19T12:44:17.398297+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-19T12:44:21.129403+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T12:44:25.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.095+0000 7f110c3ce640 1 -- 172.21.15.57:0/1018409557 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f10e806ae40 msgr2=0x7f10e806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:25.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.095+0000 7f110c3ce640 1 --2- 172.21.15.57:0/1018409557 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f10e806ae40 0x7f10e806d300 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f1100035000 tx=0x7f1100004730 comp rx=0 tx=0).stop 2023-10-19T12:44:25.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.095+0000 7f110c3ce640 1 -- 172.21.15.57:0/1018409557 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f11040ab500 msgr2=0x7f1104139de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:25.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.095+0000 7f110c3ce640 1 --2- 172.21.15.57:0/1018409557 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f11040ab500 0x7f1104139de0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f10f400d740 tx=0x7f10f400dc00 comp rx=0 tx=0).stop 2023-10-19T12:44:25.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.095+0000 7f110c3ce640 1 -- 172.21.15.57:0/1018409557 shutdown_connections 2023-10-19T12:44:25.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.095+0000 7f110c3ce640 1 --2- 172.21.15.57:0/1018409557 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f10e806ae40 0x7f10e806d300 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:25.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.095+0000 7f110c3ce640 1 --2- 172.21.15.57:0/1018409557 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f11040c96b0 0x7f11040b0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:25.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.095+0000 7f110c3ce640 1 --2- 172.21.15.57:0/1018409557 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f11040ab500 0x7f1104139de0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:25.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.095+0000 7f110c3ce640 1 --2- 172.21.15.57:0/1018409557 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f110406af00 0x7f1104152e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:25.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.095+0000 7f110c3ce640 1 -- 172.21.15.57:0/1018409557 >> 172.21.15.57:0/1018409557 conn(0x7f1104069930 msgr2=0x7f11040ab990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:25.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.095+0000 7f110c3ce640 1 -- 172.21.15.57:0/1018409557 shutdown_connections 2023-10-19T12:44:25.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.095+0000 7f110c3ce640 1 -- 172.21.15.57:0/1018409557 wait complete. 2023-10-19T12:44:25.124 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T12:44:25.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.229+0000 7fb71e7f5640 1 -- 172.21.15.57:0/176355750 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb7180afa20 msgr2=0x7fb718158300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:25.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.229+0000 7fb71e7f5640 1 --2- 172.21.15.57:0/176355750 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb7180afa20 0x7fb718158300 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb708005fd0 tx=0x7fb70802d1f0 comp rx=0 tx=0).stop 2023-10-19T12:44:25.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.229+0000 7fb71e7f5640 1 -- 172.21.15.57:0/176355750 shutdown_connections 2023-10-19T12:44:25.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.229+0000 7fb71e7f5640 1 --2- 172.21.15.57:0/176355750 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb7180afa20 0x7fb718158300 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:25.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.229+0000 7fb71e7f5640 1 -- 172.21.15.57:0/176355750 >> 172.21.15.57:0/176355750 conn(0x7fb718069930 msgr2=0x7fb718069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:25.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.230+0000 7fb71e7f5640 1 -- 172.21.15.57:0/176355750 shutdown_connections 2023-10-19T12:44:25.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.230+0000 7fb71e7f5640 1 -- 172.21.15.57:0/176355750 wait complete. 2023-10-19T12:44:25.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.231+0000 7fb71e7f5640 1 Processor -- start 2023-10-19T12:44:25.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.231+0000 7fb71e7f5640 1 -- start start 2023-10-19T12:44:25.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.232+0000 7fb71e7f5640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb7180afa20 0x7fb718149040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:25.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.232+0000 7fb71e7f5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb7180b0530 0x7fb71814ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:25.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.232+0000 7fb71e7f5640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb7180cbd20 0x7fb7180aeae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:25.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.232+0000 7fb71e7f5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb71814d2e0 con 0x7fb7180b0530 2023-10-19T12:44:25.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.232+0000 7fb71e7f5640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb718149580 con 0x7fb7180cbd20 2023-10-19T12:44:25.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.232+0000 7fb71e7f5640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb71814d4f0 con 0x7fb7180afa20 2023-10-19T12:44:25.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.232+0000 7fb7177fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb7180b0530 0x7fb71814ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:25.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.232+0000 7fb7177fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb7180b0530 0x7fb71814ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:53124/0 (socket says 172.21.15.57:53124) 2023-10-19T12:44:25.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.232+0000 7fb7177fe640 1 -- 172.21.15.57:0/2728399107 learned_addr learned my addr 172.21.15.57:0/2728399107 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:25.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.232+0000 7fb717fff640 1 --2- 172.21.15.57:0/2728399107 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb7180afa20 0x7fb718149040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:25.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.232+0000 7fb71cd92640 1 --2- 172.21.15.57:0/2728399107 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb7180cbd20 0x7fb7180aeae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:25.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.232+0000 7fb7177fe640 1 -- 172.21.15.57:0/2728399107 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb7180afa20 msgr2=0x7fb718149040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:25.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.232+0000 7fb7177fe640 1 --2- 172.21.15.57:0/2728399107 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb7180afa20 0x7fb718149040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:25.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.232+0000 7fb7177fe640 1 -- 172.21.15.57:0/2728399107 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb7180cbd20 msgr2=0x7fb7180aeae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:25.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.232+0000 7fb7177fe640 1 --2- 172.21.15.57:0/2728399107 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb7180cbd20 0x7fb7180aeae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:25.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.232+0000 7fb7177fe640 1 -- 172.21.15.57:0/2728399107 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb70800a670 con 0x7fb7180b0530 2023-10-19T12:44:25.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.232+0000 7fb71cd92640 1 --2- 172.21.15.57:0/2728399107 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb7180cbd20 0x7fb7180aeae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:44:25.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.232+0000 7fb717fff640 1 --2- 172.21.15.57:0/2728399107 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb7180afa20 0x7fb718149040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:44:25.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.232+0000 7fb7177fe640 1 --2- 172.21.15.57:0/2728399107 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb7180b0530 0x7fb71814ef80 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fb708006010 tx=0x7fb708004a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:25.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.232+0000 7fb714ff9640 1 -- 172.21.15.57:0/2728399107 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7080035f0 con 0x7fb7180b0530 2023-10-19T12:44:25.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.232+0000 7fb714ff9640 1 -- 172.21.15.57:0/2728399107 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fb708003840 con 0x7fb7180b0530 2023-10-19T12:44:25.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.232+0000 7fb71e7f5640 1 -- 172.21.15.57:0/2728399107 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb71814d7d0 con 0x7fb7180b0530 2023-10-19T12:44:25.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.232+0000 7fb714ff9640 1 -- 172.21.15.57:0/2728399107 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb708003c50 con 0x7fb7180b0530 2023-10-19T12:44:25.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.233+0000 7fb71e7f5640 1 -- 172.21.15.57:0/2728399107 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb71814dd90 con 0x7fb7180b0530 2023-10-19T12:44:25.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.233+0000 7fb71e7f5640 1 -- 172.21.15.57:0/2728399107 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7180690e0 con 0x7fb7180b0530 2023-10-19T12:44:25.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.234+0000 7fb714ff9640 1 -- 172.21.15.57:0/2728399107 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fb708035440 con 0x7fb7180b0530 2023-10-19T12:44:25.243 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.234+0000 7fb714ff9640 1 --2- 172.21.15.57:0/2728399107 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fb70406ae40 0x7fb70406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:25.243 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.234+0000 7fb714ff9640 1 -- 172.21.15.57:0/2728399107 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fb7080acc40 con 0x7fb7180b0530 2023-10-19T12:44:25.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.234+0000 7fb717fff640 1 --2- 172.21.15.57:0/2728399107 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fb70406ae40 0x7fb70406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:25.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.235+0000 7fb717fff640 1 --2- 172.21.15.57:0/2728399107 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fb70406ae40 0x7fb70406d300 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb70c000f60 tx=0x7fb70c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:25.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.238+0000 7fb714ff9640 1 -- 172.21.15.57:0/2728399107 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb708077ac0 con 0x7fb7180b0530 2023-10-19T12:44:25.453 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.450+0000 7fb71e7f5640 1 -- 172.21.15.57:0/2728399107 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb7180ae490 con 0x7fb7180b0530 2023-10-19T12:44:25.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.450+0000 7fb714ff9640 1 -- 172.21.15.57:0/2728399107 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4906 (secure 0 0 0) 0x7fb708042090 con 0x7fb7180b0530 2023-10-19T12:44:25.454 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:44:25.454 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4357,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/4228712742","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":4228712742},{"type":"v1","addr":"172.21.15.57:6835","nonce":4228712742}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4360,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/971892630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":971892630},{"type":"v1","addr":"172.21.15.57:6834","nonce":971892630}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4372,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/104495983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":104495983},{"type":"v1","addr":"172.21.15.179:6836","nonce":104495983}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T12:44:18.419669+0000","modified":"2023-10-19T12:44:23.472554+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4374},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4374":{"gid":4374,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.179:6837/622550365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":622550365},{"type":"v1","addr":"172.21.15.179:6837","nonce":622550365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2023-10-19T12:44:25.455 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 7 2023-10-19T12:44:25.456 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.452+0000 7fb71e7f5640 1 -- 172.21.15.57:0/2728399107 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fb70406ae40 msgr2=0x7fb70406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:25.456 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.452+0000 7fb71e7f5640 1 --2- 172.21.15.57:0/2728399107 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fb70406ae40 0x7fb70406d300 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb70c000f60 tx=0x7fb70c009040 comp rx=0 tx=0).stop 2023-10-19T12:44:25.456 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.452+0000 7fb71e7f5640 1 -- 172.21.15.57:0/2728399107 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb7180b0530 msgr2=0x7fb71814ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:25.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.452+0000 7fb71e7f5640 1 --2- 172.21.15.57:0/2728399107 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb7180b0530 0x7fb71814ef80 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fb708006010 tx=0x7fb708004a50 comp rx=0 tx=0).stop 2023-10-19T12:44:25.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.453+0000 7fb71e7f5640 1 -- 172.21.15.57:0/2728399107 shutdown_connections 2023-10-19T12:44:25.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.453+0000 7fb71e7f5640 1 --2- 172.21.15.57:0/2728399107 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fb70406ae40 0x7fb70406d300 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:25.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.453+0000 7fb71e7f5640 1 --2- 172.21.15.57:0/2728399107 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb7180cbd20 0x7fb7180aeae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:25.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.453+0000 7fb71e7f5640 1 --2- 172.21.15.57:0/2728399107 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb7180b0530 0x7fb71814ef80 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:25.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.453+0000 7fb71e7f5640 1 --2- 172.21.15.57:0/2728399107 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb7180afa20 0x7fb718149040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:25.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.453+0000 7fb71e7f5640 1 -- 172.21.15.57:0/2728399107 >> 172.21.15.57:0/2728399107 conn(0x7fb718069930 msgr2=0x7fb718069d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:25.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.453+0000 7fb71e7f5640 1 -- 172.21.15.57:0/2728399107 shutdown_connections 2023-10-19T12:44:25.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.453+0000 7fb71e7f5640 1 -- 172.21.15.57:0/2728399107 wait complete. 2023-10-19T12:44:25.466 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T12:44:25.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.571+0000 7f44b4bd0640 1 -- 172.21.15.57:0/934974453 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f44b00afa20 msgr2=0x7f44b00705a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:25.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.571+0000 7f44b4bd0640 1 --2- 172.21.15.57:0/934974453 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f44b00afa20 0x7f44b00705a0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f4498005fd0 tx=0x7f449802d0d0 comp rx=0 tx=0).stop 2023-10-19T12:44:25.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.571+0000 7f44b4bd0640 1 -- 172.21.15.57:0/934974453 shutdown_connections 2023-10-19T12:44:25.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.571+0000 7f44b4bd0640 1 --2- 172.21.15.57:0/934974453 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f44b00afa20 0x7f44b00705a0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:25.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.571+0000 7f44b4bd0640 1 --2- 172.21.15.57:0/934974453 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f44b00cbd20 0x7f44b0158300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:25.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.571+0000 7f44b4bd0640 1 -- 172.21.15.57:0/934974453 >> 172.21.15.57:0/934974453 conn(0x7f44b0069930 msgr2=0x7f44b0069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:25.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.571+0000 7f44b4bd0640 1 -- 172.21.15.57:0/934974453 shutdown_connections 2023-10-19T12:44:25.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.571+0000 7f44b4bd0640 1 -- 172.21.15.57:0/934974453 wait complete. 2023-10-19T12:44:25.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.573+0000 7f44b4bd0640 1 Processor -- start 2023-10-19T12:44:25.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.573+0000 7f44b4bd0640 1 -- start start 2023-10-19T12:44:25.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.574+0000 7f44b4bd0640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f44b00afa20 0x7f44b0143ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:25.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.574+0000 7f44b4bd0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f44b00b0530 0x7f44b01300b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:25.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.574+0000 7f44b4bd0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f44b00cbd20 0x7f44b01305f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:25.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.574+0000 7f44b4bd0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44b0130b30 con 0x7f44b00cbd20 2023-10-19T12:44:25.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.574+0000 7f44b4bd0640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44b0130d90 con 0x7f44b00b0530 2023-10-19T12:44:25.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.574+0000 7f44b4bd0640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44b0131070 con 0x7f44b00afa20 2023-10-19T12:44:25.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.574+0000 7f44aed9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f44b00cbd20 0x7f44b01305f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:25.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.574+0000 7f44aed9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f44b00cbd20 0x7f44b01305f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:53140/0 (socket says 172.21.15.57:53140) 2023-10-19T12:44:25.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.574+0000 7f44aed9d640 1 -- 172.21.15.57:0/1528853455 learned_addr learned my addr 172.21.15.57:0/1528853455 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:25.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.574+0000 7f44ae59c640 1 --2- 172.21.15.57:0/1528853455 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f44b00afa20 0x7f44b0143ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:25.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.574+0000 7f44a5d9b640 1 --2- 172.21.15.57:0/1528853455 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f44b00b0530 0x7f44b01300b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:25.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.574+0000 7f44aed9d640 1 -- 172.21.15.57:0/1528853455 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f44b00afa20 msgr2=0x7f44b0143ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:25.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.574+0000 7f44aed9d640 1 --2- 172.21.15.57:0/1528853455 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f44b00afa20 0x7f44b0143ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:25.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.574+0000 7f44aed9d640 1 -- 172.21.15.57:0/1528853455 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f44b00b0530 msgr2=0x7f44b01300b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:25.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.574+0000 7f44aed9d640 1 --2- 172.21.15.57:0/1528853455 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f44b00b0530 0x7f44b01300b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:25.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.574+0000 7f44aed9d640 1 -- 172.21.15.57:0/1528853455 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f449800a670 con 0x7f44b00cbd20 2023-10-19T12:44:25.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.574+0000 7f44aed9d640 1 --2- 172.21.15.57:0/1528853455 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f44b00cbd20 0x7f44b01305f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f44a800c780 tx=0x7f44a800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:25.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.574+0000 7f44a77fe640 1 -- 172.21.15.57:0/1528853455 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44a8004b60 con 0x7f44b00cbd20 2023-10-19T12:44:25.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.574+0000 7f44a77fe640 1 -- 172.21.15.57:0/1528853455 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f44a800ce90 con 0x7f44b00cbd20 2023-10-19T12:44:25.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.575+0000 7f44b4bd0640 1 -- 172.21.15.57:0/1528853455 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44b0070cf0 con 0x7f44b00cbd20 2023-10-19T12:44:25.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.575+0000 7f44a77fe640 1 -- 172.21.15.57:0/1528853455 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44a800aab0 con 0x7f44b00cbd20 2023-10-19T12:44:25.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.575+0000 7f44b4bd0640 1 -- 172.21.15.57:0/1528853455 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44b0131630 con 0x7f44b00cbd20 2023-10-19T12:44:25.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.575+0000 7f44b4bd0640 1 -- 172.21.15.57:0/1528853455 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44b006f970 con 0x7f44b00cbd20 2023-10-19T12:44:25.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.576+0000 7f44a77fe640 1 -- 172.21.15.57:0/1528853455 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f44a800ac10 con 0x7f44b00cbd20 2023-10-19T12:44:25.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.576+0000 7f44a77fe640 1 --2- 172.21.15.57:0/1528853455 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f449006ae70 0x7f449006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:25.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.577+0000 7f44a77fe640 1 -- 172.21.15.57:0/1528853455 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f44a8089c70 con 0x7f44b00cbd20 2023-10-19T12:44:25.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.577+0000 7f44ae59c640 1 --2- 172.21.15.57:0/1528853455 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f449006ae70 0x7f449006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:25.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.577+0000 7f44ae59c640 1 --2- 172.21.15.57:0/1528853455 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f449006ae70 0x7f449006d330 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f44a00058c0 tx=0x7f44a0005f60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:25.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.581+0000 7f44a77fe640 1 -- 172.21.15.57:0/1528853455 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f44a8054a40 con 0x7f44b00cbd20 2023-10-19T12:44:25.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.758+0000 7f44b4bd0640 1 -- 172.21.15.57:0/1528853455 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f44b00cc180 con 0x7f44b00cbd20 2023-10-19T12:44:25.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.759+0000 7f44a77fe640 1 -- 172.21.15.57:0/1528853455 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18358 (secure 0 0 0) 0x7f44a8058060 con 0x7f44b00cbd20 2023-10-19T12:44:25.762 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:44:25.763 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":25,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T12:44:23.472399+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-19T12:44:16.517510+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-19T12:44:17.398297+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-19T12:44:21.129403+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T12:44:25.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.761+0000 7f44b4bd0640 1 -- 172.21.15.57:0/1528853455 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f449006ae70 msgr2=0x7f449006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:25.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.761+0000 7f44b4bd0640 1 --2- 172.21.15.57:0/1528853455 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f449006ae70 0x7f449006d330 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f44a00058c0 tx=0x7f44a0005f60 comp rx=0 tx=0).stop 2023-10-19T12:44:25.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.761+0000 7f44b4bd0640 1 -- 172.21.15.57:0/1528853455 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f44b00cbd20 msgr2=0x7f44b01305f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:25.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.761+0000 7f44b4bd0640 1 --2- 172.21.15.57:0/1528853455 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f44b00cbd20 0x7f44b01305f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f44a800c780 tx=0x7f44a800cc40 comp rx=0 tx=0).stop 2023-10-19T12:44:25.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.761+0000 7f44b4bd0640 1 -- 172.21.15.57:0/1528853455 shutdown_connections 2023-10-19T12:44:25.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.761+0000 7f44b4bd0640 1 --2- 172.21.15.57:0/1528853455 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f449006ae70 0x7f449006d330 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:25.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.761+0000 7f44b4bd0640 1 --2- 172.21.15.57:0/1528853455 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f44b00cbd20 0x7f44b01305f0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:25.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.761+0000 7f44b4bd0640 1 --2- 172.21.15.57:0/1528853455 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f44b00b0530 0x7f44b01300b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:25.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.761+0000 7f44b4bd0640 1 --2- 172.21.15.57:0/1528853455 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f44b00afa20 0x7f44b0143ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:25.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.761+0000 7f44b4bd0640 1 -- 172.21.15.57:0/1528853455 >> 172.21.15.57:0/1528853455 conn(0x7f44b0069930 msgr2=0x7f44b014d2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:25.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.761+0000 7f44b4bd0640 1 -- 172.21.15.57:0/1528853455 shutdown_connections 2023-10-19T12:44:25.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:25.761+0000 7f44b4bd0640 1 -- 172.21.15.57:0/1528853455 wait complete. 2023-10-19T12:44:26.176 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 7, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-19T12:44:18.419669+0000', 'modified': '2023-10-19T12:44:23.472554+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4374}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4374': {'gid': 4374, 'name': 'd', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.179:6837/622550365', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6835', 'nonce': 622550365}, {'type': 'v1', 'addr': '172.21.15.179:6837', 'nonce': 622550365}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1} 2023-10-19T12:44:26.177 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-19T12:44:26.177 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2023-10-19T12:44:26.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.290+0000 7fa18e16c640 1 -- 172.21.15.57:0/284919300 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa170003970 msgr2=0x7fa170003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:26.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.290+0000 7fa18e16c640 1 --2- 172.21.15.57:0/284919300 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa170003970 0x7fa170003d90 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fa178009b50 tx=0x7fa1700057f0 comp rx=0 tx=0).stop 2023-10-19T12:44:26.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.290+0000 7fa187fff640 1 -- 172.21.15.57:0/284919300 reap_dead start 2023-10-19T12:44:26.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.290+0000 7fa18e16c640 1 -- 172.21.15.57:0/284919300 shutdown_connections 2023-10-19T12:44:26.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.290+0000 7fa18e16c640 1 -- 172.21.15.57:0/284919300 >> 172.21.15.57:0/284919300 conn(0x7fa1880b0670 msgr2=0x7fa1880b0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:26.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.291+0000 7fa18e16c640 1 -- 172.21.15.57:0/284919300 shutdown_connections 2023-10-19T12:44:26.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.291+0000 7fa18e16c640 1 -- 172.21.15.57:0/284919300 wait complete. 2023-10-19T12:44:26.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.292+0000 7fa18e16c640 1 Processor -- start 2023-10-19T12:44:26.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.292+0000 7fa18e16c640 1 -- start start 2023-10-19T12:44:26.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.293+0000 7fa18e16c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa170003970 0x7fa1880c9580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:26.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.293+0000 7fa18e16c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa1880cb4f0 0x7fa1881f0c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:26.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.293+0000 7fa18e16c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa1880cb970 0x7fa1881f1140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:26.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.293+0000 7fa18e16c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1880b2770 con 0x7fa170003970 2023-10-19T12:44:26.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.293+0000 7fa18e16c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1881f1680 con 0x7fa1880cb970 2023-10-19T12:44:26.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.293+0000 7fa18e16c640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1880b1af0 con 0x7fa1880cb4f0 2023-10-19T12:44:26.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.293+0000 7fa1877fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa170003970 0x7fa1880c9580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:26.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.293+0000 7fa1877fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa170003970 0x7fa1880c9580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:53152/0 (socket says 172.21.15.57:53152) 2023-10-19T12:44:26.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.293+0000 7fa1877fe640 1 -- 172.21.15.57:0/3472563532 learned_addr learned my addr 172.21.15.57:0/3472563532 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:26.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.293+0000 7fa186ffd640 1 --2- 172.21.15.57:0/3472563532 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa1880cb4f0 0x7fa1881f0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:26.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.293+0000 7fa187fff640 1 --2- 172.21.15.57:0/3472563532 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa1880cb970 0x7fa1881f1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:26.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.293+0000 7fa1877fe640 1 -- 172.21.15.57:0/3472563532 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa1880cb4f0 msgr2=0x7fa1881f0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:26.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.293+0000 7fa1877fe640 1 --2- 172.21.15.57:0/3472563532 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa1880cb4f0 0x7fa1881f0c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:26.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.293+0000 7fa1877fe640 1 -- 172.21.15.57:0/3472563532 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa1880cb970 msgr2=0x7fa1881f1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:26.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.293+0000 7fa1877fe640 1 --2- 172.21.15.57:0/3472563532 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa1880cb970 0x7fa1881f1140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:26.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.293+0000 7fa1877fe640 1 -- 172.21.15.57:0/3472563532 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1880b1dd0 con 0x7fa170003970 2023-10-19T12:44:26.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.293+0000 7fa1877fe640 1 --2- 172.21.15.57:0/3472563532 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa170003970 0x7fa1880c9580 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fa178009c70 tx=0x7fa178033c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:26.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.293+0000 7fa16ffff640 1 -- 172.21.15.57:0/3472563532 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa178004230 con 0x7fa170003970 2023-10-19T12:44:26.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.293+0000 7fa16ffff640 1 -- 172.21.15.57:0/3472563532 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fa178004390 con 0x7fa170003970 2023-10-19T12:44:26.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.293+0000 7fa18e16c640 1 -- 172.21.15.57:0/3472563532 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1880b20b0 con 0x7fa170003970 2023-10-19T12:44:26.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.293+0000 7fa16ffff640 1 -- 172.21.15.57:0/3472563532 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa178003850 con 0x7fa170003970 2023-10-19T12:44:26.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.293+0000 7fa18e16c640 1 -- 172.21.15.57:0/3472563532 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1880c9ac0 con 0x7fa170003970 2023-10-19T12:44:26.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.295+0000 7fa16ffff640 1 -- 172.21.15.57:0/3472563532 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fa1780039b0 con 0x7fa170003970 2023-10-19T12:44:26.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.295+0000 7fa16ffff640 1 --2- 172.21.15.57:0/3472563532 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fa17406ae40 0x7fa17406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:26.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.295+0000 7fa16ffff640 1 -- 172.21.15.57:0/3472563532 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fa1780ac440 con 0x7fa170003970 2023-10-19T12:44:26.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.295+0000 7fa186ffd640 1 --2- 172.21.15.57:0/3472563532 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fa17406ae40 0x7fa17406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:26.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.296+0000 7fa186ffd640 1 --2- 172.21.15.57:0/3472563532 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fa17406ae40 0x7fa17406d300 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa1881f4c60 tx=0x7fa17c005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:26.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.296+0000 7fa190dd4740 1 Processor -- start 2023-10-19T12:44:26.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.296+0000 7fa190dd4740 1 -- start start 2023-10-19T12:44:26.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.297+0000 7fa190dd4740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5636c8ba2b40 0x5636c8b75ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:26.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.297+0000 7fa190dd4740 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x5636c8ba2f20 0x5636c8ba58d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:26.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.297+0000 7fa190dd4740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5636c8b76020 0x5636c8ba5e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:26.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.297+0000 7fa190dd4740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636c8a87730 con 0x5636c8ba2b40 2023-10-19T12:44:26.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.297+0000 7fa190dd4740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636c8ae8780 con 0x5636c8b76020 2023-10-19T12:44:26.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.297+0000 7fa190dd4740 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x5636c8ae5e70 con 0x5636c8ba2f20 2023-10-19T12:44:26.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.297+0000 7fa1877fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5636c8ba2b40 0x5636c8b75ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:26.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.297+0000 7fa1877fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5636c8ba2b40 0x5636c8b75ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:53156/0 (socket says 172.21.15.57:53156) 2023-10-19T12:44:26.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.297+0000 7fa1877fe640 1 -- 172.21.15.57:0/2920880868 learned_addr learned my addr 172.21.15.57:0/2920880868 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:26.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.297+0000 7fa187fff640 1 --2- 172.21.15.57:0/2920880868 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5636c8b76020 0x5636c8ba5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:26.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.297+0000 7fa186ffd640 1 --2- 172.21.15.57:0/2920880868 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x5636c8ba2f20 0x5636c8ba58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:26.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.297+0000 7fa1877fe640 1 -- 172.21.15.57:0/2920880868 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x5636c8ba2f20 msgr2=0x5636c8ba58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T12:44:26.312 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.297+0000 7fa1877fe640 1 --2- 172.21.15.57:0/2920880868 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x5636c8ba2f20 0x5636c8ba58d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:26.312 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.297+0000 7fa1877fe640 1 -- 172.21.15.57:0/2920880868 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5636c8b76020 msgr2=0x5636c8ba5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T12:44:26.312 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.297+0000 7fa1877fe640 1 --2- 172.21.15.57:0/2920880868 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5636c8b76020 0x5636c8ba5e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:26.312 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.297+0000 7fa1877fe640 1 -- 172.21.15.57:0/2920880868 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5636c8ae63d0 con 0x5636c8ba2b40 2023-10-19T12:44:26.312 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.298+0000 7fa1877fe640 1 --2- 172.21.15.57:0/2920880868 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5636c8ba2b40 0x5636c8b75ae0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fa178077500 tx=0x7fa17807ac70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aaaf8e012c6dd58 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:26.313 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.298+0000 7fa16d7fa640 1 -- 172.21.15.57:0/2920880868 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa178083250 con 0x5636c8ba2b40 2023-10-19T12:44:26.313 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.298+0000 7fa16d7fa640 1 -- 172.21.15.57:0/2920880868 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fa1780793a0 con 0x5636c8ba2b40 2023-10-19T12:44:26.313 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.298+0000 7fa16d7fa640 1 -- 172.21.15.57:0/2920880868 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1780810c0 con 0x5636c8ba2b40 2023-10-19T12:44:26.313 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.298+0000 7fa190dd4740 1 -- 172.21.15.57:0/2920880868 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5636c8ba2b40 msgr2=0x5636c8b75ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:26.313 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.298+0000 7fa190dd4740 1 --2- 172.21.15.57:0/2920880868 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5636c8ba2b40 0x5636c8b75ae0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fa178077500 tx=0x7fa17807ac70 comp rx=0 tx=0).stop 2023-10-19T12:44:26.313 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.298+0000 7fa190dd4740 1 -- 172.21.15.57:0/2920880868 shutdown_connections 2023-10-19T12:44:26.314 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.298+0000 7fa190dd4740 1 --2- 172.21.15.57:0/2920880868 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5636c8b76020 0x5636c8ba5e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:26.314 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.298+0000 7fa190dd4740 1 --2- 172.21.15.57:0/2920880868 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x5636c8ba2f20 0x5636c8ba58d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:26.314 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.298+0000 7fa190dd4740 1 --2- 172.21.15.57:0/2920880868 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5636c8ba2b40 0x5636c8b75ae0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:26.314 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.298+0000 7fa190dd4740 1 -- 172.21.15.57:0/2920880868 >> 172.21.15.57:0/2920880868 conn(0x5636c8b77d50 msgr2=0x5636c8bc08d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:26.314 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.298+0000 7fa190dd4740 1 -- 172.21.15.57:0/2920880868 shutdown_connections 2023-10-19T12:44:26.315 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.298+0000 7fa190dd4740 1 -- 172.21.15.57:0/2920880868 wait complete. 2023-10-19T12:44:26.315 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.299+0000 7fa190dd4740 1 -- start start 2023-10-19T12:44:26.315 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.299+0000 7fa190dd4740 1 Processor -- start 2023-10-19T12:44:26.315 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.300+0000 7fa190dd4740 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x5636c8ba2b40 0x5636c8ba8d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:26.315 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.300+0000 7fa190dd4740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5636c8ba2f20 0x5636c8bba260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:26.316 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.300+0000 7fa190dd4740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5636c8b92f60 0x5636c8bba7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:26.316 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.300+0000 7fa190dd4740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636c8ae1760 con 0x5636c8b92f60 2023-10-19T12:44:26.316 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.300+0000 7fa190dd4740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636c8a9c8f0 con 0x5636c8ba2f20 2023-10-19T12:44:26.316 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.300+0000 7fa190dd4740 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x5636c8a9c590 con 0x5636c8ba2b40 2023-10-19T12:44:26.316 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.300+0000 7fa190dd4740 10 client.0 _ll_register_callbacks cb 0x5636c8ad8ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-19T12:44:26.316 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.300+0000 7fa187fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5636c8b92f60 0x5636c8bba7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:26.317 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.300+0000 7fa187fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5636c8b92f60 0x5636c8bba7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:53170/0 (socket says 172.21.15.57:53170) 2023-10-19T12:44:26.317 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.300+0000 7fa187fff640 1 -- 172.21.15.57:0/1723130354 learned_addr learned my addr 172.21.15.57:0/1723130354 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:26.317 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.300+0000 7fa186ffd640 1 --2- 172.21.15.57:0/1723130354 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5636c8ba2f20 0x5636c8bba260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:26.317 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.300+0000 7fa1877fe640 1 --2- 172.21.15.57:0/1723130354 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x5636c8ba2b40 0x5636c8ba8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:26.317 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.300+0000 7fa187fff640 1 -- 172.21.15.57:0/1723130354 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x5636c8ba2b40 msgr2=0x5636c8ba8d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T12:44:26.318 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.300+0000 7fa187fff640 1 --2- 172.21.15.57:0/1723130354 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x5636c8ba2b40 0x5636c8ba8d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:26.318 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.300+0000 7fa187fff640 1 -- 172.21.15.57:0/1723130354 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5636c8ba2f20 msgr2=0x5636c8bba260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T12:44:26.318 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.300+0000 7fa187fff640 1 --2- 172.21.15.57:0/1723130354 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5636c8ba2f20 0x5636c8bba260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:26.318 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.300+0000 7fa187fff640 1 -- 172.21.15.57:0/1723130354 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1880c9ac0 con 0x5636c8b92f60 2023-10-19T12:44:26.318 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.300+0000 7fa187fff640 1 --2- 172.21.15.57:0/1723130354 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5636c8b92f60 0x5636c8bba7a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fa1880abd30 tx=0x7fa1800064d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90031f5a39f47d47 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:26.318 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.300+0000 7fa1667fc640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2023-10-19T12:44:26.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.301+0000 7fa1667fc640 1 -- 172.21.15.57:0/1723130354 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa180016070 con 0x5636c8b92f60 2023-10-19T12:44:26.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.301+0000 7fa190dd4740 1 -- 172.21.15.57:0/1723130354 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5636c8ada740 con 0x5636c8b92f60 2023-10-19T12:44:26.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.301+0000 7fa1667fc640 1 -- 172.21.15.57:0/1723130354 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fa180011700 con 0x5636c8b92f60 2023-10-19T12:44:26.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.301+0000 7fa1667fc640 1 -- 172.21.15.57:0/1723130354 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa1800118c0 con 0x5636c8b92f60 2023-10-19T12:44:26.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.301+0000 7fa190dd4740 10 client.4524 fetch_fsmap learned FSMap version 7 2023-10-19T12:44:26.320 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.301+0000 7fa190dd4740 1 -- 172.21.15.57:0/1723130354 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) v3 -- 0x5636c8adfde0 con 0x5636c8b92f60 2023-10-19T12:44:26.320 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.301+0000 7fa1667fc640 1 -- 172.21.15.57:0/1723130354 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x5636c8ada740 con 0x5636c8b92f60 2023-10-19T12:44:26.320 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.301+0000 7fa1667fc640 1 -- 172.21.15.57:0/1723130354 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 7) v1 ==== 3302+0+0 (secure 0 0 0) 0x7fa180011a20 con 0x5636c8b92f60 2023-10-19T12:44:26.320 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.301+0000 7fa190dd4740 10 client.4524 fetch_fsmap finished waiting for FSMap version 7 2023-10-19T12:44:26.320 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.301+0000 7fa190dd4740 10 client.4524 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-19T12:44:26.321 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.301+0000 7fa190dd4740 20 client.4524 populate_metadata read hostname 'smithi057' 2023-10-19T12:44:26.321 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.301+0000 7fa190dd4740 1 --2- 172.21.15.57:0/1723130354 >> [v2:172.21.15.179:6835/622550365,v1:172.21.15.179:6837/622550365] conn(0x5636c8ba7910 0x5636c8ba7cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:26.321 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.301+0000 7fa190dd4740 4 client.4524 mds_command: new command op to 4374 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-19T12:44:26.321 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.301+0000 7fa190dd4740 1 -- 172.21.15.57:0/1723130354 --> [v2:172.21.15.179:6835/622550365,v1:172.21.15.179:6837/622550365] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5636c8ada460 con 0x5636c8ba7910 2023-10-19T12:44:26.321 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.301+0000 7fa1877fe640 1 --2- 172.21.15.57:0/1723130354 >> [v2:172.21.15.179:6835/622550365,v1:172.21.15.179:6837/622550365] conn(0x5636c8ba7910 0x5636c8ba7cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:26.321 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.302+0000 7fa1877fe640 1 --2- 172.21.15.57:0/1723130354 >> [v2:172.21.15.179:6835/622550365,v1:172.21.15.179:6837/622550365] conn(0x5636c8ba7910 0x5636c8ba7cf0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d31a950e1ff8a913 server_cookie=4409564971dbda22 in_seq=0 out_seq=0 2023-10-19T12:44:26.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.302+0000 7fa1667fc640 10 client.4524 ms_handle_connect on v2:172.21.15.179:6835/622550365 2023-10-19T12:44:26.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.302+0000 7fa1667fc640 1 -- 172.21.15.57:0/1723130354 <== mds.0 v2:172.21.15.179:6835/622550365 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5636c8ada460 con 0x5636c8ba7910 2023-10-19T12:44:26.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.302+0000 7fa1667fc640 10 client.4524 handle_command_reply: tid=0 2023-10-19T12:44:26.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.302+0000 7fa190dd4740 1 client.4524 shutdown 2023-10-19T12:44:26.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.302+0000 7fa190dd4740 1 -- 172.21.15.57:0/1723130354 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5636c8b92f60 msgr2=0x5636c8bba7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:26.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.302+0000 7fa190dd4740 1 --2- 172.21.15.57:0/1723130354 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5636c8b92f60 0x5636c8bba7a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fa1880abd30 tx=0x7fa1800064d0 comp rx=0 tx=0).stop 2023-10-19T12:44:26.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.302+0000 7fa190dd4740 1 -- 172.21.15.57:0/1723130354 shutdown_connections 2023-10-19T12:44:26.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.302+0000 7fa190dd4740 1 --2- 172.21.15.57:0/1723130354 >> [v2:172.21.15.179:6835/622550365,v1:172.21.15.179:6837/622550365] conn(0x5636c8ba7910 0x5636c8ba7cf0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:26.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.302+0000 7fa190dd4740 1 --2- 172.21.15.57:0/1723130354 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5636c8b92f60 0x5636c8bba7a0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:26.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.302+0000 7fa190dd4740 1 --2- 172.21.15.57:0/1723130354 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5636c8ba2f20 0x5636c8bba260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:26.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.302+0000 7fa190dd4740 1 --2- 172.21.15.57:0/1723130354 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x5636c8ba2b40 0x5636c8ba8d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:26.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.302+0000 7fa190dd4740 1 -- 172.21.15.57:0/1723130354 >> 172.21.15.57:0/1723130354 conn(0x5636c8b76020 msgr2=0x5636c8b79b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:26.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.302+0000 7fa1667fc640 0 client.4524 ms_handle_reset on v2:172.21.15.179:6835/622550365 2023-10-19T12:44:26.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.302+0000 7fa190dd4740 1 -- 172.21.15.57:0/1723130354 shutdown_connections 2023-10-19T12:44:26.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.302+0000 7fa190dd4740 1 -- 172.21.15.57:0/1723130354 wait complete. 2023-10-19T12:44:26.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.303+0000 7fa190dd4740 20 client.4524 trim_cache size 0 max 16384 2023-10-19T12:44:26.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.318+0000 7fa190dd4740 1 Processor -- start 2023-10-19T12:44:26.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.318+0000 7fa190dd4740 1 -- start start 2023-10-19T12:44:26.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.319+0000 7fa190dd4740 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x5636c8b92f60 0x5636c8bc2f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:26.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.319+0000 7fa190dd4740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5636c8ba2b40 0x5636c8b79640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:26.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.319+0000 7fa190dd4740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5636c8ba2f20 0x5636c8b79b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:26.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.319+0000 7fa190dd4740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636c8a62270 con 0x5636c8ba2b40 2023-10-19T12:44:26.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.319+0000 7fa190dd4740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636c8a97f10 con 0x5636c8ba2f20 2023-10-19T12:44:26.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.319+0000 7fa190dd4740 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x5636c8ae36e0 con 0x5636c8b92f60 2023-10-19T12:44:26.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.319+0000 7fa186ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5636c8ba2b40 0x5636c8b79640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:26.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.319+0000 7fa186ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5636c8ba2b40 0x5636c8b79640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:53172/0 (socket says 172.21.15.57:53172) 2023-10-19T12:44:26.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.319+0000 7fa186ffd640 1 -- 172.21.15.57:0/2209705714 learned_addr learned my addr 172.21.15.57:0/2209705714 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:26.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.319+0000 7fa187fff640 1 --2- 172.21.15.57:0/2209705714 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5636c8ba2f20 0x5636c8b79b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:26.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.319+0000 7fa186ffd640 1 -- 172.21.15.57:0/2209705714 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x5636c8b92f60 msgr2=0x5636c8bc2f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T12:44:26.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.319+0000 7fa186ffd640 1 --2- 172.21.15.57:0/2209705714 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x5636c8b92f60 0x5636c8bc2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:26.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.319+0000 7fa186ffd640 1 -- 172.21.15.57:0/2209705714 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5636c8ba2f20 msgr2=0x5636c8b79b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T12:44:26.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.319+0000 7fa186ffd640 1 --2- 172.21.15.57:0/2209705714 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5636c8ba2f20 0x5636c8b79b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:26.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.319+0000 7fa186ffd640 1 -- 172.21.15.57:0/2209705714 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5636c8a7ed00 con 0x5636c8ba2b40 2023-10-19T12:44:26.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.319+0000 7fa187fff640 1 --2- 172.21.15.57:0/2209705714 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5636c8ba2f20 0x5636c8b79b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:44:26.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.319+0000 7fa186ffd640 1 --2- 172.21.15.57:0/2209705714 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5636c8ba2b40 0x5636c8b79640 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fa17c01d5a0 tx=0x7fa17c01b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c6bce9501e1ad87 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:26.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.319+0000 7fa16dffb640 1 -- 172.21.15.57:0/2209705714 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa17c025050 con 0x5636c8ba2b40 2023-10-19T12:44:26.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.319+0000 7fa16dffb640 1 -- 172.21.15.57:0/2209705714 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fa17c026040 con 0x5636c8ba2b40 2023-10-19T12:44:26.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.319+0000 7fa16dffb640 1 -- 172.21.15.57:0/2209705714 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa17c017070 con 0x5636c8ba2b40 2023-10-19T12:44:26.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.320+0000 7fa190dd4740 1 -- 172.21.15.57:0/2209705714 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5636c8ba2b40 msgr2=0x5636c8b79640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:26.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.320+0000 7fa190dd4740 1 --2- 172.21.15.57:0/2209705714 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5636c8ba2b40 0x5636c8b79640 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fa17c01d5a0 tx=0x7fa17c01b000 comp rx=0 tx=0).stop 2023-10-19T12:44:26.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.320+0000 7fa190dd4740 1 -- 172.21.15.57:0/2209705714 shutdown_connections 2023-10-19T12:44:26.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.320+0000 7fa190dd4740 1 --2- 172.21.15.57:0/2209705714 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5636c8ba2f20 0x5636c8b79b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:26.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.320+0000 7fa190dd4740 1 --2- 172.21.15.57:0/2209705714 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5636c8ba2b40 0x5636c8b79640 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:26.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.320+0000 7fa190dd4740 1 --2- 172.21.15.57:0/2209705714 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x5636c8b92f60 0x5636c8bc2f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:26.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.320+0000 7fa190dd4740 1 -- 172.21.15.57:0/2209705714 >> 172.21.15.57:0/2209705714 conn(0x5636c8b76020 msgr2=0x5636c8bb7710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:26.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.320+0000 7fa190dd4740 1 -- 172.21.15.57:0/2209705714 shutdown_connections 2023-10-19T12:44:26.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.320+0000 7fa190dd4740 1 -- 172.21.15.57:0/2209705714 wait complete. 2023-10-19T12:44:26.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.321+0000 7fa190dd4740 1 -- start start 2023-10-19T12:44:26.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.321+0000 7fa190dd4740 1 Processor -- start 2023-10-19T12:44:26.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.322+0000 7fa190dd4740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5636c8b92f60 0x5636c8c11980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:26.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.322+0000 7fa190dd4740 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x5636c8ba2b40 0x5636c8b79540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:26.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.322+0000 7fa190dd4740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5636c8ba2f20 0x5636c8b79a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:26.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.322+0000 7fa190dd4740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636c8c2cc90 con 0x5636c8ba2f20 2023-10-19T12:44:26.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.322+0000 7fa190dd4740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636c8bba500 con 0x5636c8b92f60 2023-10-19T12:44:26.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.322+0000 7fa190dd4740 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x5636c8bba750 con 0x5636c8ba2b40 2023-10-19T12:44:26.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.322+0000 7fa190dd4740 10 client.0 _ll_register_callbacks cb 0x7fa1800018c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-19T12:44:26.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.322+0000 7fa187fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5636c8ba2f20 0x5636c8b79a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:26.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.322+0000 7fa187fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5636c8ba2f20 0x5636c8b79a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:53176/0 (socket says 172.21.15.57:53176) 2023-10-19T12:44:26.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.322+0000 7fa187fff640 1 -- 172.21.15.57:0/3685854992 learned_addr learned my addr 172.21.15.57:0/3685854992 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:26.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.322+0000 7fa1877fe640 1 --2- 172.21.15.57:0/3685854992 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5636c8b92f60 0x5636c8c11980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:26.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.322+0000 7fa186ffd640 1 --2- 172.21.15.57:0/3685854992 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x5636c8ba2b40 0x5636c8b79540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:26.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.322+0000 7fa187fff640 1 -- 172.21.15.57:0/3685854992 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x5636c8ba2b40 msgr2=0x5636c8b79540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T12:44:26.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.322+0000 7fa187fff640 1 --2- 172.21.15.57:0/3685854992 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x5636c8ba2b40 0x5636c8b79540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:26.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.322+0000 7fa187fff640 1 -- 172.21.15.57:0/3685854992 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5636c8b92f60 msgr2=0x5636c8c11980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T12:44:26.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.322+0000 7fa187fff640 1 --2- 172.21.15.57:0/3685854992 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5636c8b92f60 0x5636c8c11980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:26.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.322+0000 7fa187fff640 1 -- 172.21.15.57:0/3685854992 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa17c012db0 con 0x5636c8ba2f20 2023-10-19T12:44:26.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.322+0000 7fa187fff640 1 --2- 172.21.15.57:0/3685854992 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5636c8ba2f20 0x5636c8b79a80 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa1880abd30 tx=0x7fa180011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df853421c65f94 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:26.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.322+0000 7fa1667fc640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2023-10-19T12:44:26.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.322+0000 7fa1667fc640 1 -- 172.21.15.57:0/3685854992 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa18001b860 con 0x5636c8ba2f20 2023-10-19T12:44:26.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.322+0000 7fa1667fc640 1 -- 172.21.15.57:0/3685854992 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fa18001b9c0 con 0x5636c8ba2f20 2023-10-19T12:44:26.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.322+0000 7fa190dd4740 1 -- 172.21.15.57:0/3685854992 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa1780793a0 con 0x5636c8ba2f20 2023-10-19T12:44:26.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.322+0000 7fa1667fc640 1 -- 172.21.15.57:0/3685854992 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa18001bb80 con 0x5636c8ba2f20 2023-10-19T12:44:26.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.323+0000 7fa190dd4740 10 client.4530 fetch_fsmap learned FSMap version 7 2023-10-19T12:44:26.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.323+0000 7fa190dd4740 1 -- 172.21.15.57:0/3685854992 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) v3 -- 0x5636c8aa44f0 con 0x5636c8ba2f20 2023-10-19T12:44:26.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.323+0000 7fa1667fc640 1 -- 172.21.15.57:0/3685854992 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fa180011e90 con 0x5636c8ba2f20 2023-10-19T12:44:26.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.323+0000 7fa1667fc640 1 -- 172.21.15.57:0/3685854992 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 7) v1 ==== 3302+0+0 (secure 0 0 0) 0x7fa18002b020 con 0x5636c8ba2f20 2023-10-19T12:44:26.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.323+0000 7fa190dd4740 10 client.4530 fetch_fsmap finished waiting for FSMap version 7 2023-10-19T12:44:26.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.323+0000 7fa190dd4740 10 client.4530 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-19T12:44:26.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.323+0000 7fa190dd4740 20 client.4530 populate_metadata read hostname 'smithi057' 2023-10-19T12:44:26.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.323+0000 7fa190dd4740 1 --2- 172.21.15.57:0/3685854992 >> [v2:172.21.15.179:6835/622550365,v1:172.21.15.179:6837/622550365] conn(0x5636c8bc1ff0 0x5636c8bc23d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:26.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.323+0000 7fa190dd4740 4 client.4530 mds_command: new command op to 4374 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-19T12:44:26.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.323+0000 7fa190dd4740 1 -- 172.21.15.57:0/3685854992 --> [v2:172.21.15.179:6835/622550365,v1:172.21.15.179:6837/622550365] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5636c87e36c0 con 0x5636c8bc1ff0 2023-10-19T12:44:26.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.323+0000 7fa1877fe640 1 --2- 172.21.15.57:0/3685854992 >> [v2:172.21.15.179:6835/622550365,v1:172.21.15.179:6837/622550365] conn(0x5636c8bc1ff0 0x5636c8bc23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:26.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.323+0000 7fa1667fc640 10 client.4530 ms_handle_connect on v2:172.21.15.179:6835/622550365 2023-10-19T12:44:26.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.323+0000 7fa1877fe640 1 --2- 172.21.15.57:0/3685854992 >> [v2:172.21.15.179:6835/622550365,v1:172.21.15.179:6837/622550365] conn(0x5636c8bc1ff0 0x5636c8bc23d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2eec2e0bebd626e6 server_cookie=654601ee993b4a91 in_seq=0 out_seq=0 2023-10-19T12:44:26.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.324+0000 7fa1667fc640 1 -- 172.21.15.57:0/3685854992 <== mds.0 v2:172.21.15.179:6835/622550365 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+323 (crc 0 0 0) 0x5636c87e36c0 con 0x5636c8bc1ff0 2023-10-19T12:44:26.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.324+0000 7fa1667fc640 10 client.4530 handle_command_reply: tid=0 2023-10-19T12:44:26.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.324+0000 7fa190dd4740 1 client.4530 shutdown 2023-10-19T12:44:26.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.324+0000 7fa190dd4740 1 -- 172.21.15.57:0/3685854992 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5636c8ba2f20 msgr2=0x5636c8b79a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:26.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.324+0000 7fa190dd4740 1 --2- 172.21.15.57:0/3685854992 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5636c8ba2f20 0x5636c8b79a80 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa1880abd30 tx=0x7fa180011b60 comp rx=0 tx=0).stop 2023-10-19T12:44:26.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.324+0000 7fa190dd4740 1 -- 172.21.15.57:0/3685854992 shutdown_connections 2023-10-19T12:44:26.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.324+0000 7fa190dd4740 1 --2- 172.21.15.57:0/3685854992 >> [v2:172.21.15.179:6835/622550365,v1:172.21.15.179:6837/622550365] conn(0x5636c8bc1ff0 0x5636c8bc23d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:26.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.324+0000 7fa190dd4740 1 --2- 172.21.15.57:0/3685854992 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5636c8ba2f20 0x5636c8b79a80 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:26.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.324+0000 7fa190dd4740 1 --2- 172.21.15.57:0/3685854992 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x5636c8ba2b40 0x5636c8b79540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:26.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.324+0000 7fa190dd4740 1 --2- 172.21.15.57:0/3685854992 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5636c8b92f60 0x5636c8c11980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:26.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.324+0000 7fa1667fc640 0 client.4530 ms_handle_reset on v2:172.21.15.179:6835/622550365 2023-10-19T12:44:26.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.324+0000 7fa190dd4740 1 -- 172.21.15.57:0/3685854992 >> 172.21.15.57:0/3685854992 conn(0x5636c8b76020 msgr2=0x5636c8b7c8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:26.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.324+0000 7fa190dd4740 1 -- 172.21.15.57:0/3685854992 shutdown_connections 2023-10-19T12:44:26.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.324+0000 7fa190dd4740 1 -- 172.21.15.57:0/3685854992 wait complete. 2023-10-19T12:44:26.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.324+0000 7fa190dd4740 20 client.4530 trim_cache size 0 max 16384 2023-10-19T12:44:26.346 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2023-10-19T12:44:26.346 INFO:teuthology.orchestra.run.smithi057.stdout: "cluster_fsid": "1e9beb19-3260-4dd7-85ef-5b399ce774ac", 2023-10-19T12:44:26.346 INFO:teuthology.orchestra.run.smithi057.stdout: "whoami": 0, 2023-10-19T12:44:26.346 INFO:teuthology.orchestra.run.smithi057.stdout: "id": 4374, 2023-10-19T12:44:26.347 INFO:teuthology.orchestra.run.smithi057.stdout: "want_state": "up:active", 2023-10-19T12:44:26.347 INFO:teuthology.orchestra.run.smithi057.stdout: "state": "up:active", 2023-10-19T12:44:26.347 INFO:teuthology.orchestra.run.smithi057.stdout: "fs_name": "cephfs", 2023-10-19T12:44:26.347 INFO:teuthology.orchestra.run.smithi057.stdout: "rank_uptime": 7.245152611, 2023-10-19T12:44:26.347 INFO:teuthology.orchestra.run.smithi057.stdout: "mdsmap_epoch": 7, 2023-10-19T12:44:26.348 INFO:teuthology.orchestra.run.smithi057.stdout: "osdmap_epoch": 25, 2023-10-19T12:44:26.348 INFO:teuthology.orchestra.run.smithi057.stdout: "osdmap_epoch_barrier": 25, 2023-10-19T12:44:26.348 INFO:teuthology.orchestra.run.smithi057.stdout: "uptime": 14.301177944000001 2023-10-19T12:44:26.348 INFO:teuthology.orchestra.run.smithi057.stdout:} 2023-10-19T12:44:26.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.324+0000 7fa18e16c640 1 -- 172.21.15.57:0/3472563532 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fa17406ae40 msgr2=0x7fa17406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:26.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.324+0000 7fa18e16c640 1 --2- 172.21.15.57:0/3472563532 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fa17406ae40 0x7fa17406d300 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa1881f4c60 tx=0x7fa17c005f90 comp rx=0 tx=0).stop 2023-10-19T12:44:26.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.324+0000 7fa18e16c640 1 -- 172.21.15.57:0/3472563532 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa170003970 msgr2=0x7fa1880c9580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:26.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.324+0000 7fa18e16c640 1 --2- 172.21.15.57:0/3472563532 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa170003970 0x7fa1880c9580 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fa178009c70 tx=0x7fa178033c30 comp rx=0 tx=0).stop 2023-10-19T12:44:26.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.324+0000 7fa18e16c640 1 -- 172.21.15.57:0/3472563532 shutdown_connections 2023-10-19T12:44:26.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.324+0000 7fa18e16c640 1 --2- 172.21.15.57:0/3472563532 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fa17406ae40 0x7fa17406d300 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:26.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.324+0000 7fa18e16c640 1 --2- 172.21.15.57:0/3472563532 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa1880cb970 0x7fa1881f1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:26.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.324+0000 7fa18e16c640 1 --2- 172.21.15.57:0/3472563532 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa1880cb4f0 0x7fa1881f0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:26.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.324+0000 7fa18e16c640 1 --2- 172.21.15.57:0/3472563532 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa170003970 0x7fa1880c9580 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:26.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.324+0000 7fa18e16c640 1 -- 172.21.15.57:0/3472563532 >> 172.21.15.57:0/3472563532 conn(0x7fa1880b0670 msgr2=0x7fa18815a610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:26.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.324+0000 7fa18e16c640 1 -- 172.21.15.57:0/3472563532 shutdown_connections 2023-10-19T12:44:26.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.325+0000 7fa18e16c640 1 -- 172.21.15.57:0/3472563532 wait complete. 2023-10-19T12:44:26.352 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-10-19T12:44:26.353 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2023-10-19T12:44:26.354 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-19T12:44:26.462 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.459+0000 7faca007c640 1 -- 172.21.15.57:0/1347929680 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac98141690 msgr2=0x7fac980703e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:26.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.459+0000 7faca007c640 1 --2- 172.21.15.57:0/1347929680 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac98141690 0x7fac980703e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fac980ae6b0 tx=0x7fac8802bc80 comp rx=0 tx=0).stop 2023-10-19T12:44:26.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.459+0000 7faca007c640 1 -- 172.21.15.57:0/1347929680 shutdown_connections 2023-10-19T12:44:26.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.459+0000 7faca007c640 1 --2- 172.21.15.57:0/1347929680 >> v2:172.21.15.179:3300/0 conn(0x7fac9806c0a0 0x7fac98070920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:26.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.459+0000 7faca007c640 1 --2- 172.21.15.57:0/1347929680 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac98141690 0x7fac980703e0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:26.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.459+0000 7faca007c640 1 --2- 172.21.15.57:0/1347929680 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fac9813e520 0x7fac9806bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:26.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.459+0000 7faca007c640 1 -- 172.21.15.57:0/1347929680 >> 172.21.15.57:0/1347929680 conn(0x7fac98069930 msgr2=0x7fac98069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:26.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.459+0000 7faca007c640 1 -- 172.21.15.57:0/1347929680 shutdown_connections 2023-10-19T12:44:26.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.459+0000 7faca007c640 1 -- 172.21.15.57:0/1347929680 wait complete. 2023-10-19T12:44:26.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.460+0000 7faca007c640 1 Processor -- start 2023-10-19T12:44:26.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.460+0000 7faca007c640 1 -- start start 2023-10-19T12:44:26.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.461+0000 7faca007c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac9806c0a0 0x7fac981d9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:26.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.461+0000 7faca007c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fac9813e520 0x7fac9813adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:26.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.461+0000 7faca007c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fac98141690 0x7fac980b0db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:26.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.461+0000 7faca007c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac981e1280 con 0x7fac9806c0a0 2023-10-19T12:44:26.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.461+0000 7faca007c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac981da680 con 0x7fac98141690 2023-10-19T12:44:26.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.461+0000 7faca007c640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac981d8c50 con 0x7fac9813e520 2023-10-19T12:44:26.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.461+0000 7fac9de18640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac9806c0a0 0x7fac981d9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:26.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.461+0000 7fac9de18640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac9806c0a0 0x7fac981d9ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:53202/0 (socket says 172.21.15.57:53202) 2023-10-19T12:44:26.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.461+0000 7fac9de18640 1 -- 172.21.15.57:0/1902309892 learned_addr learned my addr 172.21.15.57:0/1902309892 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:26.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.461+0000 7fac9d617640 1 --2- 172.21.15.57:0/1902309892 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fac9813e520 0x7fac9813adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:26.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.461+0000 7fac9e619640 1 --2- 172.21.15.57:0/1902309892 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fac98141690 0x7fac980b0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:26.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.461+0000 7fac9de18640 1 -- 172.21.15.57:0/1902309892 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fac9813e520 msgr2=0x7fac9813adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:26.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.461+0000 7fac9de18640 1 --2- 172.21.15.57:0/1902309892 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fac9813e520 0x7fac9813adf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:26.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.461+0000 7fac9de18640 1 -- 172.21.15.57:0/1902309892 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fac98141690 msgr2=0x7fac980b0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:26.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.461+0000 7fac9de18640 1 --2- 172.21.15.57:0/1902309892 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fac98141690 0x7fac980b0db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:26.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.461+0000 7fac9de18640 1 -- 172.21.15.57:0/1902309892 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac88009560 con 0x7fac9806c0a0 2023-10-19T12:44:26.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.461+0000 7fac9e619640 1 --2- 172.21.15.57:0/1902309892 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fac98141690 0x7fac980b0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:44:26.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.461+0000 7fac9d617640 1 --2- 172.21.15.57:0/1902309892 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fac9813e520 0x7fac9813adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:44:26.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.461+0000 7fac9de18640 1 --2- 172.21.15.57:0/1902309892 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac9806c0a0 0x7fac981d9ff0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fac9400d950 tx=0x7fac9400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:26.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.462+0000 7fac8e7fc640 1 -- 172.21.15.57:0/1902309892 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac94009c30 con 0x7fac9806c0a0 2023-10-19T12:44:26.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.462+0000 7fac8e7fc640 1 -- 172.21.15.57:0/1902309892 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fac94009e80 con 0x7fac9806c0a0 2023-10-19T12:44:26.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.462+0000 7faca007c640 1 -- 172.21.15.57:0/1902309892 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac981d8f30 con 0x7fac9806c0a0 2023-10-19T12:44:26.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.462+0000 7fac8e7fc640 1 -- 172.21.15.57:0/1902309892 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac94004280 con 0x7fac9806c0a0 2023-10-19T12:44:26.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.462+0000 7faca007c640 1 -- 172.21.15.57:0/1902309892 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac981dc920 con 0x7fac9806c0a0 2023-10-19T12:44:26.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.463+0000 7faca007c640 1 -- 172.21.15.57:0/1902309892 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac981d9450 con 0x7fac9806c0a0 2023-10-19T12:44:26.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.463+0000 7fac8e7fc640 1 -- 172.21.15.57:0/1902309892 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fac940035f0 con 0x7fac9806c0a0 2023-10-19T12:44:26.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.464+0000 7fac8e7fc640 1 --2- 172.21.15.57:0/1902309892 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fac7c06ae70 0x7fac7c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:26.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.464+0000 7fac8e7fc640 1 -- 172.21.15.57:0/1902309892 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fac94089b20 con 0x7fac9806c0a0 2023-10-19T12:44:26.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.464+0000 7fac9d617640 1 --2- 172.21.15.57:0/1902309892 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fac7c06ae70 0x7fac7c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:26.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.464+0000 7fac9d617640 1 --2- 172.21.15.57:0/1902309892 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fac7c06ae70 0x7fac7c06d330 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fac88035000 tx=0x7fac88004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:26.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.468+0000 7fac8e7fc640 1 -- 172.21.15.57:0/1902309892 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fac940548d0 con 0x7fac9806c0a0 2023-10-19T12:44:26.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.681+0000 7faca007c640 1 -- 172.21.15.57:0/1902309892 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fac980690e0 con 0x7fac9806c0a0 2023-10-19T12:44:26.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.681+0000 7fac8e7fc640 1 -- 172.21.15.57:0/1902309892 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) v1 ==== 53+0+88 (secure 0 0 0) 0x7fac94017070 con 0x7fac9806c0a0 2023-10-19T12:44:26.685 INFO:teuthology.orchestra.run.smithi057.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-10-19T12:44:26.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.683+0000 7faca007c640 1 -- 172.21.15.57:0/1902309892 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fac7c06ae70 msgr2=0x7fac7c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:26.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.683+0000 7faca007c640 1 --2- 172.21.15.57:0/1902309892 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fac7c06ae70 0x7fac7c06d330 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fac88035000 tx=0x7fac88004730 comp rx=0 tx=0).stop 2023-10-19T12:44:26.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.683+0000 7faca007c640 1 -- 172.21.15.57:0/1902309892 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac9806c0a0 msgr2=0x7fac981d9ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:26.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.683+0000 7faca007c640 1 --2- 172.21.15.57:0/1902309892 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac9806c0a0 0x7fac981d9ff0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fac9400d950 tx=0x7fac9400de10 comp rx=0 tx=0).stop 2023-10-19T12:44:26.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.683+0000 7faca007c640 1 -- 172.21.15.57:0/1902309892 shutdown_connections 2023-10-19T12:44:26.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.683+0000 7faca007c640 1 --2- 172.21.15.57:0/1902309892 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fac7c06ae70 0x7fac7c06d330 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:26.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.683+0000 7faca007c640 1 --2- 172.21.15.57:0/1902309892 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fac98141690 0x7fac980b0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:26.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.683+0000 7faca007c640 1 --2- 172.21.15.57:0/1902309892 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fac9813e520 0x7fac9813adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:26.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.683+0000 7faca007c640 1 --2- 172.21.15.57:0/1902309892 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac9806c0a0 0x7fac981d9ff0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:26.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.683+0000 7faca007c640 1 -- 172.21.15.57:0/1902309892 >> 172.21.15.57:0/1902309892 conn(0x7fac98069930 msgr2=0x7fac981403a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:26.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.683+0000 7faca007c640 1 -- 172.21.15.57:0/1902309892 shutdown_connections 2023-10-19T12:44:26.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:26.683+0000 7faca007c640 1 -- 172.21.15.57:0/1902309892 wait complete. 2023-10-19T12:44:26.697 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-19T12:44:26.697 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-19T12:44:26.697 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi057.front.sepia.ceph.com 2023-10-19T12:44:26.698 INFO:tasks.cephfs.mount:self.client.name = client.admin 2023-10-19T12:44:26.698 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2023-10-19T12:44:26.698 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-19T12:44:26.699 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-10-19T12:44:26.699 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-19T12:44:26.699 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2023-10-19T12:44:26.700 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:44:26.700 DEBUG:teuthology.orchestra.run.smithi057:> ip addr 2023-10-19T12:44:26.715 INFO:teuthology.orchestra.run.smithi057.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-19T12:44:26.716 INFO:teuthology.orchestra.run.smithi057.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-19T12:44:26.716 INFO:teuthology.orchestra.run.smithi057.stdout: inet 127.0.0.1/8 scope host lo 2023-10-19T12:44:26.716 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T12:44:26.716 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 ::1/128 scope host 2023-10-19T12:44:26.717 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T12:44:26.717 INFO:teuthology.orchestra.run.smithi057.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-19T12:44:26.717 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:6b:ff:86 brd ff:ff:ff:ff:ff:ff 2023-10-19T12:44:26.717 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp7s0f0 2023-10-19T12:44:26.717 INFO:teuthology.orchestra.run.smithi057.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-19T12:44:26.718 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:6b:ff:87 brd ff:ff:ff:ff:ff:ff 2023-10-19T12:44:26.718 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp7s0f1 2023-10-19T12:44:26.718 INFO:teuthology.orchestra.run.smithi057.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-19T12:44:26.718 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:bd:15:62 brd ff:ff:ff:ff:ff:ff 2023-10-19T12:44:26.718 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp1s0f0 2023-10-19T12:44:26.719 INFO:teuthology.orchestra.run.smithi057.stdout: inet 172.21.15.57/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-19T12:44:26.719 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft 42644sec preferred_lft 42644sec 2023-10-19T12:44:26.719 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 fe80::ec4:7aff:febd:1562/64 scope link noprefixroute 2023-10-19T12:44:26.719 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T12:44:26.719 INFO:teuthology.orchestra.run.smithi057.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-19T12:44:26.719 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:bd:15:63 brd ff:ff:ff:ff:ff:ff 2023-10-19T12:44:26.720 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp1s0f1 2023-10-19T12:44:26.720 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-10-19T12:44:26.721 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:44:26.721 DEBUG:teuthology.orchestra.run.smithi057:> (cd / && exec bash -c ' 2023-10-19T12:44:26.721 DEBUG:teuthology.orchestra.run.smithi057:> set -e 2023-10-19T12:44:26.722 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip link add name ceph-brx type bridge 2023-10-19T12:44:26.722 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip addr flush dev ceph-brx 2023-10-19T12:44:26.722 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip link set ceph-brx up 2023-10-19T12:44:26.723 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-10-19T12:44:26.723 DEBUG:teuthology.orchestra.run.smithi057:> ') 2023-10-19T12:44:26.884 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:44:26.884 DEBUG:teuthology.orchestra.run.smithi057:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-19T12:44:26.917 INFO:teuthology.orchestra.run.smithi057.stdout:1 2023-10-19T12:44:26.917 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:44:26.917 DEBUG:teuthology.orchestra.run.smithi057:> route 2023-10-19T12:44:26.989 INFO:teuthology.orchestra.run.smithi057.stdout:Kernel IP routing table 2023-10-19T12:44:26.990 INFO:teuthology.orchestra.run.smithi057.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-19T12:44:26.990 INFO:teuthology.orchestra.run.smithi057.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-19T12:44:26.990 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-19T12:44:26.990 INFO:teuthology.orchestra.run.smithi057.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-19T12:44:26.991 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:44:26.991 DEBUG:teuthology.orchestra.run.smithi057:> (cd / && exec bash -c ' 2023-10-19T12:44:26.991 DEBUG:teuthology.orchestra.run.smithi057:> set -e 2023-10-19T12:44:26.992 DEBUG:teuthology.orchestra.run.smithi057:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-19T12:44:26.992 DEBUG:teuthology.orchestra.run.smithi057:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-19T12:44:26.992 DEBUG:teuthology.orchestra.run.smithi057:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-19T12:44:26.993 DEBUG:teuthology.orchestra.run.smithi057:> ') 2023-10-19T12:44:27.295 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:44:27.295 DEBUG:teuthology.orchestra.run.smithi057:> ip netns list 2023-10-19T12:44:27.311 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:44:27.312 DEBUG:teuthology.orchestra.run.smithi057:> ip netns list-id 2023-10-19T12:44:27.366 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:44:27.366 DEBUG:teuthology.orchestra.run.smithi057:> (cd / && exec bash -c ' 2023-10-19T12:44:27.367 DEBUG:teuthology.orchestra.run.smithi057:> set -e 2023-10-19T12:44:27.367 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2023-10-19T12:44:27.368 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2023-10-19T12:44:27.368 DEBUG:teuthology.orchestra.run.smithi057:> ') 2023-10-19T12:44:27.455 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2023-10-19T12:44:27.455 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:44:27.456 DEBUG:teuthology.orchestra.run.smithi057:> (cd / && exec bash -c ' 2023-10-19T12:44:27.456 DEBUG:teuthology.orchestra.run.smithi057:> set -e 2023-10-19T12:44:27.457 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2023-10-19T12:44:27.457 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-10-19T12:44:27.458 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2023-10-19T12:44:27.458 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2023-10-19T12:44:27.458 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2023-10-19T12:44:27.459 DEBUG:teuthology.orchestra.run.smithi057:> ') 2023-10-19T12:44:27.574 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:44:27.575 DEBUG:teuthology.orchestra.run.smithi057:> (cd / && exec bash -c ' 2023-10-19T12:44:27.575 DEBUG:teuthology.orchestra.run.smithi057:> set -e 2023-10-19T12:44:27.576 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip link set brx.0 up 2023-10-19T12:44:27.576 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip link set dev brx.0 master ceph-brx 2023-10-19T12:44:27.576 DEBUG:teuthology.orchestra.run.smithi057:> ') 2023-10-19T12:44:27.625 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2023-10-19T12:44:27.626 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-19T12:44:27.626 DEBUG:teuthology.orchestra.run.smithi057:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2023-10-19T12:44:27.641 INFO:teuthology.orchestra.run.smithi057.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2023-10-19T12:44:27.642 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-19T12:44:27.642 DEBUG:teuthology.orchestra.run.smithi057:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2023-10-19T12:44:27.696 DEBUG:teuthology.orchestra.run.smithi057:> sudo modprobe fuse 2023-10-19T12:44:27.768 DEBUG:teuthology.orchestra.run.smithi057:> cat /proc/self/mounts | awk '{print $2}' 2023-10-19T12:44:27.824 INFO:teuthology.orchestra.run.smithi057.stdout:/proc 2023-10-19T12:44:27.824 INFO:teuthology.orchestra.run.smithi057.stdout:/sys 2023-10-19T12:44:27.824 INFO:teuthology.orchestra.run.smithi057.stdout:/dev 2023-10-19T12:44:27.825 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/security 2023-10-19T12:44:27.825 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/shm 2023-10-19T12:44:27.825 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/pts 2023-10-19T12:44:27.825 INFO:teuthology.orchestra.run.smithi057.stdout:/run 2023-10-19T12:44:27.825 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/cgroup 2023-10-19T12:44:27.826 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/pstore 2023-10-19T12:44:27.826 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/bpf 2023-10-19T12:44:27.826 INFO:teuthology.orchestra.run.smithi057.stdout:/ 2023-10-19T12:44:27.826 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/selinux 2023-10-19T12:44:27.826 INFO:teuthology.orchestra.run.smithi057.stdout:/proc/sys/fs/binfmt_misc 2023-10-19T12:44:27.827 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/hugepages 2023-10-19T12:44:27.827 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/mqueue 2023-10-19T12:44:27.827 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/debug 2023-10-19T12:44:27.827 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/tracing 2023-10-19T12:44:27.827 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/fuse/connections 2023-10-19T12:44:27.828 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/config 2023-10-19T12:44:27.828 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-sysctl.service 2023-10-19T12:44:27.828 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-10-19T12:44:27.828 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-10-19T12:44:27.828 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/nfs/rpc_pipefs 2023-10-19T12:44:27.829 INFO:teuthology.orchestra.run.smithi057.stdout:/run/user/1000 2023-10-19T12:44:27.829 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph 2023-10-19T12:44:27.829 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-0 2023-10-19T12:44:27.829 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-1 2023-10-19T12:44:27.829 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-2 2023-10-19T12:44:27.830 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-3 2023-10-19T12:44:27.830 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns 2023-10-19T12:44:27.830 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2023-10-19T12:44:27.830 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2023-10-19T12:44:27.831 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:44:27.831 DEBUG:teuthology.orchestra.run.smithi057:> ls /sys/fs/fuse/connections 2023-10-19T12:44:27.878 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-10-19T12:44:27.879 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2023-10-19T12:44:27.921 DEBUG:teuthology.orchestra.run.smithi057:> sudo modprobe fuse 2023-10-19T12:44:27.952 DEBUG:teuthology.orchestra.run.smithi057:> cat /proc/self/mounts | awk '{print $2}' 2023-10-19T12:44:28.009 INFO:teuthology.orchestra.run.smithi057.stdout:/proc 2023-10-19T12:44:28.010 INFO:teuthology.orchestra.run.smithi057.stdout:/sys 2023-10-19T12:44:28.010 INFO:teuthology.orchestra.run.smithi057.stdout:/dev 2023-10-19T12:44:28.010 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/security 2023-10-19T12:44:28.010 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/shm 2023-10-19T12:44:28.011 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/pts 2023-10-19T12:44:28.011 INFO:teuthology.orchestra.run.smithi057.stdout:/run 2023-10-19T12:44:28.011 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/cgroup 2023-10-19T12:44:28.011 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/pstore 2023-10-19T12:44:28.011 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/bpf 2023-10-19T12:44:28.011 INFO:teuthology.orchestra.run.smithi057.stdout:/ 2023-10-19T12:44:28.012 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/selinux 2023-10-19T12:44:28.012 INFO:teuthology.orchestra.run.smithi057.stdout:/proc/sys/fs/binfmt_misc 2023-10-19T12:44:28.012 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/hugepages 2023-10-19T12:44:28.012 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/mqueue 2023-10-19T12:44:28.012 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/debug 2023-10-19T12:44:28.013 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/tracing 2023-10-19T12:44:28.013 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/fuse/connections 2023-10-19T12:44:28.013 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/config 2023-10-19T12:44:28.013 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-sysctl.service 2023-10-19T12:44:28.014 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-10-19T12:44:28.014 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-10-19T12:44:28.014 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/nfs/rpc_pipefs 2023-10-19T12:44:28.014 INFO:teuthology.orchestra.run.smithi057.stdout:/run/user/1000 2023-10-19T12:44:28.014 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph 2023-10-19T12:44:28.015 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-0 2023-10-19T12:44:28.015 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-1 2023-10-19T12:44:28.015 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-2 2023-10-19T12:44:28.015 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-3 2023-10-19T12:44:28.016 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns 2023-10-19T12:44:28.016 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2023-10-19T12:44:28.016 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2023-10-19T12:44:28.016 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:44:28.017 DEBUG:teuthology.orchestra.run.smithi057:> ls /sys/fs/fuse/connections 2023-10-19T12:44:28.045 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi057.stderr:2023-10-19T12:44:28.042+0000 7f3104a52480 -1 init, newargv = 0x55f5aa1117b0 newargc=15 2023-10-19T12:44:28.046 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi057.stderr:ceph-fuse[95612]: starting ceph client 2023-10-19T12:44:28.051 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi057.stderr:ceph-fuse[95612]: starting fuse 2023-10-19T12:44:28.061 INFO:teuthology.orchestra.run.smithi057.stdout:43 2023-10-19T12:44:28.061 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2023-10-19T12:44:28.062 DEBUG:teuthology.orchestra.run.smithi057:> sudo stdin-killer -- python3 -c ' 2023-10-19T12:44:28.062 DEBUG:teuthology.orchestra.run.smithi057:> import glob 2023-10-19T12:44:28.062 DEBUG:teuthology.orchestra.run.smithi057:> import re 2023-10-19T12:44:28.063 DEBUG:teuthology.orchestra.run.smithi057:> import os 2023-10-19T12:44:28.063 DEBUG:teuthology.orchestra.run.smithi057:> import subprocess 2023-10-19T12:44:28.064 DEBUG:teuthology.orchestra.run.smithi057:> 2023-10-19T12:44:28.064 DEBUG:teuthology.orchestra.run.smithi057:> def _find_admin_socket(client_name): 2023-10-19T12:44:28.064 DEBUG:teuthology.orchestra.run.smithi057:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2023-10-19T12:44:28.065 DEBUG:teuthology.orchestra.run.smithi057:> files = glob.glob(asok_path) 2023-10-19T12:44:28.065 DEBUG:teuthology.orchestra.run.smithi057:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2023-10-19T12:44:28.065 DEBUG:teuthology.orchestra.run.smithi057:> 2023-10-19T12:44:28.065 DEBUG:teuthology.orchestra.run.smithi057:> # Given a non-glob path, it better be there 2023-10-19T12:44:28.066 DEBUG:teuthology.orchestra.run.smithi057:> if "*" not in asok_path: 2023-10-19T12:44:28.066 DEBUG:teuthology.orchestra.run.smithi057:> assert(len(files) == 1) 2023-10-19T12:44:28.067 DEBUG:teuthology.orchestra.run.smithi057:> return files[0] 2023-10-19T12:44:28.067 DEBUG:teuthology.orchestra.run.smithi057:> 2023-10-19T12:44:28.067 DEBUG:teuthology.orchestra.run.smithi057:> for f in files: 2023-10-19T12:44:28.068 DEBUG:teuthology.orchestra.run.smithi057:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-10-19T12:44:28.068 DEBUG:teuthology.orchestra.run.smithi057:> if os.path.exists("/proc/{0}".format(pid)): 2023-10-19T12:44:28.069 DEBUG:teuthology.orchestra.run.smithi057:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-10-19T12:44:28.069 DEBUG:teuthology.orchestra.run.smithi057:> contents = proc_f.read() 2023-10-19T12:44:28.069 DEBUG:teuthology.orchestra.run.smithi057:> if mountpoint in contents: 2023-10-19T12:44:28.070 DEBUG:teuthology.orchestra.run.smithi057:> return f 2023-10-19T12:44:28.070 DEBUG:teuthology.orchestra.run.smithi057:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-10-19T12:44:28.071 DEBUG:teuthology.orchestra.run.smithi057:> 2023-10-19T12:44:28.071 DEBUG:teuthology.orchestra.run.smithi057:> print(_find_admin_socket("client.admin")) 2023-10-19T12:44:28.071 DEBUG:teuthology.orchestra.run.smithi057:> ' 2023-10-19T12:44:28.187 INFO:teuthology.orchestra.run.smithi057.stdout:/var/run/ceph/ceph-client.admin.95612.asok 2023-10-19T12:44:28.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2023-10-19T12:44:28.197 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.95612.asok 2023-10-19T12:44:28.197 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:44:28.198 DEBUG:teuthology.orchestra.run.smithi057:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.95612.asok status 2023-10-19T12:44:28.337 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2023-10-19T12:44:28.338 INFO:teuthology.orchestra.run.smithi057.stdout: "metadata": { 2023-10-19T12:44:28.338 INFO:teuthology.orchestra.run.smithi057.stdout: "ceph_sha1": "f7420d7f8e55ceea786dcbf536c338ce920c21eb", 2023-10-19T12:44:28.338 INFO:teuthology.orchestra.run.smithi057.stdout: "ceph_version": "ceph version 18.0.0-6786-gf7420d7f (f7420d7f8e55ceea786dcbf536c338ce920c21eb) reef (dev)", 2023-10-19T12:44:28.338 INFO:teuthology.orchestra.run.smithi057.stdout: "entity_id": "admin", 2023-10-19T12:44:28.339 INFO:teuthology.orchestra.run.smithi057.stdout: "hostname": "smithi057", 2023-10-19T12:44:28.339 INFO:teuthology.orchestra.run.smithi057.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2023-10-19T12:44:28.339 INFO:teuthology.orchestra.run.smithi057.stdout: "pid": "95612", 2023-10-19T12:44:28.339 INFO:teuthology.orchestra.run.smithi057.stdout: "root": "/" 2023-10-19T12:44:28.339 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T12:44:28.340 INFO:teuthology.orchestra.run.smithi057.stdout: "dentry_count": 0, 2023-10-19T12:44:28.340 INFO:teuthology.orchestra.run.smithi057.stdout: "dentry_pinned_count": 0, 2023-10-19T12:44:28.340 INFO:teuthology.orchestra.run.smithi057.stdout: "id": 4542, 2023-10-19T12:44:28.340 INFO:teuthology.orchestra.run.smithi057.stdout: "inst": { 2023-10-19T12:44:28.340 INFO:teuthology.orchestra.run.smithi057.stdout: "name": { 2023-10-19T12:44:28.341 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "client", 2023-10-19T12:44:28.341 INFO:teuthology.orchestra.run.smithi057.stdout: "num": 4542 2023-10-19T12:44:28.341 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T12:44:28.341 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": { 2023-10-19T12:44:28.341 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "v1", 2023-10-19T12:44:28.342 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": "192.168.0.1:0", 2023-10-19T12:44:28.342 INFO:teuthology.orchestra.run.smithi057.stdout: "nonce": 3459281386 2023-10-19T12:44:28.342 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T12:44:28.342 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T12:44:28.342 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": { 2023-10-19T12:44:28.343 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "v1", 2023-10-19T12:44:28.343 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": "192.168.0.1:0", 2023-10-19T12:44:28.343 INFO:teuthology.orchestra.run.smithi057.stdout: "nonce": 3459281386 2023-10-19T12:44:28.343 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T12:44:28.343 INFO:teuthology.orchestra.run.smithi057.stdout: "inst_str": "client.4542 192.168.0.1:0/3459281386", 2023-10-19T12:44:28.343 INFO:teuthology.orchestra.run.smithi057.stdout: "addr_str": "192.168.0.1:0/3459281386", 2023-10-19T12:44:28.344 INFO:teuthology.orchestra.run.smithi057.stdout: "inode_count": 1, 2023-10-19T12:44:28.344 INFO:teuthology.orchestra.run.smithi057.stdout: "mds_epoch": 7, 2023-10-19T12:44:28.344 INFO:teuthology.orchestra.run.smithi057.stdout: "osd_epoch": 25, 2023-10-19T12:44:28.344 INFO:teuthology.orchestra.run.smithi057.stdout: "osd_epoch_barrier": 0, 2023-10-19T12:44:28.344 INFO:teuthology.orchestra.run.smithi057.stdout: "blocklisted": false, 2023-10-19T12:44:28.345 INFO:teuthology.orchestra.run.smithi057.stdout: "fs_name": "cephfs" 2023-10-19T12:44:28.345 INFO:teuthology.orchestra.run.smithi057.stdout:} 2023-10-19T12:44:28.346 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:44:28.346 DEBUG:teuthology.orchestra.run.smithi057:> stat --file-system '--printf=%T 2023-10-19T12:44:28.347 DEBUG:teuthology.orchestra.run.smithi057:> ' -- /home/ubuntu/cephtest/mnt.admin 2023-10-19T12:44:28.362 INFO:teuthology.orchestra.run.smithi057.stdout:fuseblk 2023-10-19T12:44:28.362 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2023-10-19T12:44:28.363 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:44:28.363 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2023-10-19T12:44:28.434 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:44:28.434 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.admin && exec bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2023-10-19T12:44:28.509 INFO:teuthology.orchestra.run.smithi057.stdout:# file: . 2023-10-19T12:44:28.510 INFO:teuthology.orchestra.run.smithi057.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2023-10-19T12:44:28.510 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:44:28.510 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T12:44:28.510 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T12:44:28.566 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T12:44:28.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:28.712+0000 7f4b79989640 1 -- 172.21.15.57:0/3222871467 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4b740cfa20 msgr2=0x7f4b740cb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:28.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:28.712+0000 7f4b79989640 1 --2- 172.21.15.57:0/3222871467 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4b740cfa20 0x7f4b740cb9e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f4b5c009830 tx=0x7f4b5c02bd60 comp rx=0 tx=0).stop 2023-10-19T12:44:28.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:28.712+0000 7f4b79989640 1 -- 172.21.15.57:0/3222871467 shutdown_connections 2023-10-19T12:44:28.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:28.712+0000 7f4b79989640 1 --2- 172.21.15.57:0/3222871467 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4b740cfa20 0x7f4b740cb9e0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:28.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:28.712+0000 7f4b79989640 1 -- 172.21.15.57:0/3222871467 >> 172.21.15.57:0/3222871467 conn(0x7f4b7409a030 msgr2=0x7f4b740d0d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:28.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:28.712+0000 7f4b79989640 1 -- 172.21.15.57:0/3222871467 shutdown_connections 2023-10-19T12:44:28.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:28.712+0000 7f4b79989640 1 -- 172.21.15.57:0/3222871467 wait complete. 2023-10-19T12:44:28.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:28.713+0000 7f4b79989640 1 Processor -- start 2023-10-19T12:44:28.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:28.714+0000 7f4b79989640 1 -- start start 2023-10-19T12:44:28.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:28.714+0000 7f4b79989640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4b740cfa20 0x7f4b740c7460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:28.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:28.714+0000 7f4b79989640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4b7415b260 0x7f4b741f4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:28.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:28.714+0000 7f4b79989640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4b7415b640 0x7f4b741f5020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:28.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:28.714+0000 7f4b79989640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b740c79a0 con 0x7f4b7415b260 2023-10-19T12:44:28.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:28.714+0000 7f4b79989640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b741f27a0 con 0x7f4b7415b640 2023-10-19T12:44:28.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:28.714+0000 7f4b79989640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b740b60d0 con 0x7f4b740cfa20 2023-10-19T12:44:28.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:28.714+0000 7f4b727fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4b7415b260 0x7f4b741f4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:28.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:28.714+0000 7f4b727fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4b7415b260 0x7f4b741f4ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43306/0 (socket says 172.21.15.57:43306) 2023-10-19T12:44:28.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:28.714+0000 7f4b727fc640 1 -- 172.21.15.57:0/3564095235 learned_addr learned my addr 172.21.15.57:0/3564095235 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:28.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:28.715+0000 7f4b727fc640 1 -- 172.21.15.57:0/3564095235 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4b740cfa20 msgr2=0x7f4b740c7460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:28.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:28.715+0000 7f4b737fe640 1 --2- 172.21.15.57:0/3564095235 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4b7415b640 0x7f4b741f5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:28.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:28.715+0000 7f4b72ffd640 1 --2- 172.21.15.57:0/3564095235 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4b740cfa20 0x7f4b740c7460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:28.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:28.715+0000 7f4b727fc640 1 --2- 172.21.15.57:0/3564095235 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4b740cfa20 0x7f4b740c7460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:28.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:28.715+0000 7f4b727fc640 1 -- 172.21.15.57:0/3564095235 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4b7415b640 msgr2=0x7f4b741f5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:28.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:28.715+0000 7f4b727fc640 1 --2- 172.21.15.57:0/3564095235 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4b7415b640 0x7f4b741f5020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:28.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:28.715+0000 7f4b727fc640 1 -- 172.21.15.57:0/3564095235 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b5c009490 con 0x7f4b7415b260 2023-10-19T12:44:28.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:28.715+0000 7f4b737fe640 1 --2- 172.21.15.57:0/3564095235 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4b7415b640 0x7f4b741f5020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:44:28.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:28.715+0000 7f4b727fc640 1 --2- 172.21.15.57:0/3564095235 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4b7415b260 0x7f4b741f4ae0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f4b5c009950 tx=0x7f4b5c0038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:28.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:28.715+0000 7f4b6bfff640 1 -- 172.21.15.57:0/3564095235 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b5c003ea0 con 0x7f4b7415b260 2023-10-19T12:44:28.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:28.715+0000 7f4b6bfff640 1 -- 172.21.15.57:0/3564095235 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f4b5c005020 con 0x7f4b7415b260 2023-10-19T12:44:28.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:28.715+0000 7f4b6bfff640 1 -- 172.21.15.57:0/3564095235 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b5c005410 con 0x7f4b7415b260 2023-10-19T12:44:28.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:28.715+0000 7f4b79989640 1 -- 172.21.15.57:0/3564095235 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b741f2a60 con 0x7f4b7415b260 2023-10-19T12:44:28.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:28.715+0000 7f4b79989640 1 -- 172.21.15.57:0/3564095235 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b740b5350 con 0x7f4b7415b260 2023-10-19T12:44:28.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:28.716+0000 7f4b79989640 1 -- 172.21.15.57:0/3564095235 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b741f2600 con 0x7f4b7415b260 2023-10-19T12:44:28.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:28.717+0000 7f4b6bfff640 1 -- 172.21.15.57:0/3564095235 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f4b5c0047a0 con 0x7f4b7415b260 2023-10-19T12:44:28.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:28.717+0000 7f4b6bfff640 1 --2- 172.21.15.57:0/3564095235 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f4b5406ae40 0x7f4b5406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:28.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:28.717+0000 7f4b6bfff640 1 -- 172.21.15.57:0/3564095235 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f4b5c0ac640 con 0x7f4b7415b260 2023-10-19T12:44:28.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:28.717+0000 7f4b72ffd640 1 --2- 172.21.15.57:0/3564095235 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f4b5406ae40 0x7f4b5406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:28.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:28.718+0000 7f4b72ffd640 1 --2- 172.21.15.57:0/3564095235 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f4b5406ae40 0x7f4b5406d300 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c003d90 tx=0x7f4b6c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:28.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:28.721+0000 7f4b6bfff640 1 -- 172.21.15.57:0/3564095235 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4b741f2600 con 0x7f4b7415b260 2023-10-19T12:44:28.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:28.899+0000 7f4b79989640 1 -- 172.21.15.57:0/3564095235 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4b7409eb70 con 0x7f4b7415b260 2023-10-19T12:44:28.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:28.900+0000 7f4b6bfff640 1 -- 172.21.15.57:0/3564095235 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18358 (secure 0 0 0) 0x7f4b7409eb70 con 0x7f4b7415b260 2023-10-19T12:44:28.904 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:44:28.904 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":25,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T12:44:23.472399+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-19T12:44:16.517510+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-19T12:44:17.398297+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-19T12:44:21.129403+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T12:44:28.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:28.902+0000 7f4b79989640 1 -- 172.21.15.57:0/3564095235 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f4b5406ae40 msgr2=0x7f4b5406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:28.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:28.902+0000 7f4b79989640 1 --2- 172.21.15.57:0/3564095235 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f4b5406ae40 0x7f4b5406d300 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c003d90 tx=0x7f4b6c009040 comp rx=0 tx=0).stop 2023-10-19T12:44:28.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:28.902+0000 7f4b79989640 1 -- 172.21.15.57:0/3564095235 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4b7415b260 msgr2=0x7f4b741f4ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:28.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:28.902+0000 7f4b79989640 1 --2- 172.21.15.57:0/3564095235 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4b7415b260 0x7f4b741f4ae0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f4b5c009950 tx=0x7f4b5c0038a0 comp rx=0 tx=0).stop 2023-10-19T12:44:28.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:28.902+0000 7f4b79989640 1 -- 172.21.15.57:0/3564095235 shutdown_connections 2023-10-19T12:44:28.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:28.902+0000 7f4b79989640 1 --2- 172.21.15.57:0/3564095235 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f4b5406ae40 0x7f4b5406d300 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:28.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:28.902+0000 7f4b79989640 1 --2- 172.21.15.57:0/3564095235 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4b7415b640 0x7f4b741f5020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:28.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:28.902+0000 7f4b79989640 1 --2- 172.21.15.57:0/3564095235 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4b7415b260 0x7f4b741f4ae0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:28.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:28.902+0000 7f4b79989640 1 --2- 172.21.15.57:0/3564095235 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4b740cfa20 0x7f4b740c7460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:28.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:28.902+0000 7f4b79989640 1 -- 172.21.15.57:0/3564095235 >> 172.21.15.57:0/3564095235 conn(0x7f4b7409a030 msgr2=0x7f4b740b2650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:28.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:28.902+0000 7f4b79989640 1 -- 172.21.15.57:0/3564095235 shutdown_connections 2023-10-19T12:44:28.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:28.903+0000 7f4b79989640 1 -- 172.21.15.57:0/3564095235 wait complete. 2023-10-19T12:44:28.916 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-19T12:44:07.371186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-19T12:44:16.517510+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-19T12:44:17.398297+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-19T12:44:21.129403+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-19T12:44:28.917 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-19T12:44:29.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.025+0000 7f1716cda640 1 -- 172.21.15.57:0/2365344258 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f171015d400 msgr2=0x7f17100cdfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:29.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.025+0000 7f1716cda640 1 --2- 172.21.15.57:0/2365344258 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f171015d400 0x7f17100cdfe0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f16f8005fd0 tx=0x7f16f802bd90 comp rx=0 tx=0).stop 2023-10-19T12:44:29.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.025+0000 7f1716cda640 1 -- 172.21.15.57:0/2365344258 shutdown_connections 2023-10-19T12:44:29.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.025+0000 7f1716cda640 1 --2- 172.21.15.57:0/2365344258 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f171015d400 0x7f17100cdfe0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:29.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.025+0000 7f1716cda640 1 --2- 172.21.15.57:0/2365344258 >> v2:172.21.15.179:3300/0 conn(0x7f1710159060 0x7f171006d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:29.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.025+0000 7f1716cda640 1 -- 172.21.15.57:0/2365344258 >> 172.21.15.57:0/2365344258 conn(0x7f17100b0670 msgr2=0x7f17100b0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:29.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.025+0000 7f1716cda640 1 -- 172.21.15.57:0/2365344258 shutdown_connections 2023-10-19T12:44:29.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.025+0000 7f1716cda640 1 -- 172.21.15.57:0/2365344258 wait complete. 2023-10-19T12:44:29.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.026+0000 7f1716cda640 1 Processor -- start 2023-10-19T12:44:29.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.027+0000 7f1716cda640 1 -- start start 2023-10-19T12:44:29.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.027+0000 7f1716cda640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f17100cdb30 0x7f17100c5c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:29.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.027+0000 7f1716cda640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1710159060 0x7f17100b1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:29.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.027+0000 7f1716cda640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f171015d400 0x7f17100c1360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:29.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.027+0000 7f1716cda640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17100b3c00 con 0x7f17100cdb30 2023-10-19T12:44:29.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.027+0000 7f1716cda640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17100b4740 con 0x7f1710159060 2023-10-19T12:44:29.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.027+0000 7f1716cda640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17100b3e10 con 0x7f171015d400 2023-10-19T12:44:29.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.027+0000 7f1714a76640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f17100cdb30 0x7f17100c5c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:29.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.027+0000 7f1714a76640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f17100cdb30 0x7f17100c5c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43320/0 (socket says 172.21.15.57:43320) 2023-10-19T12:44:29.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.028+0000 7f1714a76640 1 -- 172.21.15.57:0/4035174497 learned_addr learned my addr 172.21.15.57:0/4035174497 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:29.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.028+0000 7f1714a76640 1 -- 172.21.15.57:0/4035174497 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f171015d400 msgr2=0x7f17100c1360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:29.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.028+0000 7f1714a76640 1 --2- 172.21.15.57:0/4035174497 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f171015d400 0x7f17100c1360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:29.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.028+0000 7f170ffff640 1 --2- 172.21.15.57:0/4035174497 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1710159060 0x7f17100b1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:29.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.028+0000 7f1714a76640 1 -- 172.21.15.57:0/4035174497 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1710159060 msgr2=0x7f17100b1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:29.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.028+0000 7f1714a76640 1 --2- 172.21.15.57:0/4035174497 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1710159060 0x7f17100b1760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:29.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.028+0000 7f1714a76640 1 -- 172.21.15.57:0/4035174497 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16f8009560 con 0x7f17100cdb30 2023-10-19T12:44:29.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.028+0000 7f1714a76640 1 --2- 172.21.15.57:0/4035174497 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f17100cdb30 0x7f17100c5c30 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f170800ebc0 tx=0x7f170800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:29.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.028+0000 7f170d7fa640 1 -- 172.21.15.57:0/4035174497 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17080099b0 con 0x7f17100cdb30 2023-10-19T12:44:29.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.028+0000 7f170d7fa640 1 -- 172.21.15.57:0/4035174497 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f1708010720 con 0x7f17100cdb30 2023-10-19T12:44:29.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.028+0000 7f1716cda640 1 -- 172.21.15.57:0/4035174497 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17100b40f0 con 0x7f17100cdb30 2023-10-19T12:44:29.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.028+0000 7f170d7fa640 1 -- 172.21.15.57:0/4035174497 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1708010b30 con 0x7f17100cdb30 2023-10-19T12:44:29.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.028+0000 7f1716cda640 1 -- 172.21.15.57:0/4035174497 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f171006be70 con 0x7f17100cdb30 2023-10-19T12:44:29.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.029+0000 7f1716cda640 1 -- 172.21.15.57:0/4035174497 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17100675f0 con 0x7f17100cdb30 2023-10-19T12:44:29.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.030+0000 7f170d7fa640 1 -- 172.21.15.57:0/4035174497 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f1708010c90 con 0x7f17100cdb30 2023-10-19T12:44:29.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.030+0000 7f170d7fa640 1 --2- 172.21.15.57:0/4035174497 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f16f006ae40 0x7f16f006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:29.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.030+0000 7f170d7fa640 1 -- 172.21.15.57:0/4035174497 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f1708014070 con 0x7f17100cdb30 2023-10-19T12:44:29.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.030+0000 7f170ffff640 1 --2- 172.21.15.57:0/4035174497 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f16f006ae40 0x7f16f006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:29.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.031+0000 7f170ffff640 1 --2- 172.21.15.57:0/4035174497 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f16f006ae40 0x7f16f006d300 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1710154f10 tx=0x7f1700009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:29.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.035+0000 7f170d7fa640 1 -- 172.21.15.57:0/4035174497 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f1708051ca0 con 0x7f17100cdb30 2023-10-19T12:44:29.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.211+0000 7f1716cda640 1 -- 172.21.15.57:0/4035174497 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f17101e5dd0 con 0x7f17100cdb30 2023-10-19T12:44:29.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.212+0000 7f170d7fa640 1 -- 172.21.15.57:0/4035174497 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1708088e60 con 0x7f17100cdb30 2023-10-19T12:44:29.215 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-19T12:44:29.217 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.214+0000 7f1716cda640 1 -- 172.21.15.57:0/4035174497 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f16f006ae40 msgr2=0x7f16f006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:29.217 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.214+0000 7f1716cda640 1 --2- 172.21.15.57:0/4035174497 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f16f006ae40 0x7f16f006d300 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1710154f10 tx=0x7f1700009040 comp rx=0 tx=0).stop 2023-10-19T12:44:29.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.214+0000 7f1716cda640 1 -- 172.21.15.57:0/4035174497 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f17100cdb30 msgr2=0x7f17100c5c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:29.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.214+0000 7f1716cda640 1 --2- 172.21.15.57:0/4035174497 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f17100cdb30 0x7f17100c5c30 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f170800ebc0 tx=0x7f170800c6a0 comp rx=0 tx=0).stop 2023-10-19T12:44:29.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.214+0000 7f1716cda640 1 -- 172.21.15.57:0/4035174497 shutdown_connections 2023-10-19T12:44:29.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.214+0000 7f1716cda640 1 --2- 172.21.15.57:0/4035174497 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f16f006ae40 0x7f16f006d300 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:29.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.214+0000 7f1716cda640 1 --2- 172.21.15.57:0/4035174497 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f171015d400 0x7f17100c1360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:29.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.214+0000 7f1716cda640 1 --2- 172.21.15.57:0/4035174497 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1710159060 0x7f17100b1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:29.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.214+0000 7f1716cda640 1 --2- 172.21.15.57:0/4035174497 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f17100cdb30 0x7f17100c5c30 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:29.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.214+0000 7f1716cda640 1 -- 172.21.15.57:0/4035174497 >> 172.21.15.57:0/4035174497 conn(0x7f17100b0670 msgr2=0x7f17100ca330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:29.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.214+0000 7f1716cda640 1 -- 172.21.15.57:0/4035174497 shutdown_connections 2023-10-19T12:44:29.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.214+0000 7f1716cda640 1 -- 172.21.15.57:0/4035174497 wait complete. 2023-10-19T12:44:29.227 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-19T12:44:29.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.336+0000 7f1840db6640 1 -- 172.21.15.57:0/2878313395 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f183c0ceb40 msgr2=0x7f183c0b0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:29.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.336+0000 7f1840db6640 1 --2- 172.21.15.57:0/2878313395 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f183c0ceb40 0x7f183c0b0650 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f1834005fd0 tx=0x7f183402c1b0 comp rx=0 tx=0).stop 2023-10-19T12:44:29.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.336+0000 7f1840db6640 1 -- 172.21.15.57:0/2878313395 shutdown_connections 2023-10-19T12:44:29.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.336+0000 7f1840db6640 1 --2- 172.21.15.57:0/2878313395 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f183c0ceb40 0x7f183c0b0650 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:29.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.336+0000 7f1840db6640 1 --2- 172.21.15.57:0/2878313395 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f183c15b2a0 0x7f183c0d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:29.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.336+0000 7f1840db6640 1 -- 172.21.15.57:0/2878313395 >> 172.21.15.57:0/2878313395 conn(0x7f183c0d0d60 msgr2=0x7f183c0d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:29.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.336+0000 7f1840db6640 1 -- 172.21.15.57:0/2878313395 shutdown_connections 2023-10-19T12:44:29.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.336+0000 7f1840db6640 1 -- 172.21.15.57:0/2878313395 wait complete. 2023-10-19T12:44:29.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.337+0000 7f1840db6640 1 Processor -- start 2023-10-19T12:44:29.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.338+0000 7f1840db6640 1 -- start start 2023-10-19T12:44:29.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.338+0000 7f1840db6640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f183c0ceb40 0x7f183c0ca900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:29.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.338+0000 7f1840db6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f183c0cf340 0x7f183c1f14c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:29.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.338+0000 7f1840db6640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f183c15b2a0 0x7f183c1f1a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:29.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.338+0000 7f1840db6640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f183c0cae40 con 0x7f183c0cf340 2023-10-19T12:44:29.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.338+0000 7f1840db6640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f183c0c93d0 con 0x7f183c0ceb40 2023-10-19T12:44:29.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.338+0000 7f1840db6640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f183c0c9ce0 con 0x7f183c15b2a0 2023-10-19T12:44:29.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.338+0000 7f1833fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f183c0cf340 0x7f183c1f14c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:29.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.338+0000 7f1833fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f183c0cf340 0x7f183c1f14c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43334/0 (socket says 172.21.15.57:43334) 2023-10-19T12:44:29.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.338+0000 7f1833fff640 1 -- 172.21.15.57:0/4070762270 learned_addr learned my addr 172.21.15.57:0/4070762270 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:29.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.339+0000 7f183ad9d640 1 --2- 172.21.15.57:0/4070762270 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f183c15b2a0 0x7f183c1f1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:29.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.339+0000 7f183a59c640 1 --2- 172.21.15.57:0/4070762270 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f183c0ceb40 0x7f183c0ca900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:29.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.339+0000 7f1833fff640 1 -- 172.21.15.57:0/4070762270 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f183c15b2a0 msgr2=0x7f183c1f1a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:29.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.339+0000 7f1833fff640 1 --2- 172.21.15.57:0/4070762270 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f183c15b2a0 0x7f183c1f1a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:29.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.339+0000 7f1833fff640 1 -- 172.21.15.57:0/4070762270 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f183c0ceb40 msgr2=0x7f183c0ca900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:29.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.339+0000 7f1833fff640 1 --2- 172.21.15.57:0/4070762270 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f183c0ceb40 0x7f183c0ca900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:29.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.339+0000 7f1833fff640 1 -- 172.21.15.57:0/4070762270 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1834009900 con 0x7f183c0cf340 2023-10-19T12:44:29.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.339+0000 7f183a59c640 1 --2- 172.21.15.57:0/4070762270 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f183c0ceb40 0x7f183c0ca900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:44:29.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.339+0000 7f1833fff640 1 --2- 172.21.15.57:0/4070762270 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f183c0cf340 0x7f183c1f14c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f182400b410 tx=0x7f182400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:29.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.339+0000 7f1832ffd640 1 -- 172.21.15.57:0/4070762270 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1824011830 con 0x7f183c0cf340 2023-10-19T12:44:29.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.339+0000 7f1832ffd640 1 -- 172.21.15.57:0/4070762270 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f1824011aa0 con 0x7f183c0cf340 2023-10-19T12:44:29.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.339+0000 7f1840db6640 1 -- 172.21.15.57:0/4070762270 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f183c0c8330 con 0x7f183c0cf340 2023-10-19T12:44:29.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.339+0000 7f1832ffd640 1 -- 172.21.15.57:0/4070762270 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1824010380 con 0x7f183c0cf340 2023-10-19T12:44:29.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.339+0000 7f1840db6640 1 -- 172.21.15.57:0/4070762270 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f183c0c88f0 con 0x7f183c0cf340 2023-10-19T12:44:29.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.340+0000 7f1840db6640 1 -- 172.21.15.57:0/4070762270 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f183c0690e0 con 0x7f183c0cf340 2023-10-19T12:44:29.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.340+0000 7f1832ffd640 1 -- 172.21.15.57:0/4070762270 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f18240104e0 con 0x7f183c0cf340 2023-10-19T12:44:29.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.341+0000 7f1832ffd640 1 --2- 172.21.15.57:0/4070762270 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f181c06ae40 0x7f181c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:29.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.341+0000 7f1832ffd640 1 -- 172.21.15.57:0/4070762270 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f1824089b70 con 0x7f183c0cf340 2023-10-19T12:44:29.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.341+0000 7f183a59c640 1 --2- 172.21.15.57:0/4070762270 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f181c06ae40 0x7f181c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:29.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.341+0000 7f183a59c640 1 --2- 172.21.15.57:0/4070762270 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f181c06ae40 0x7f181c06d300 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f182c003eb0 tx=0x7f182c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:29.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.345+0000 7f1832ffd640 1 -- 172.21.15.57:0/4070762270 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f18240549c0 con 0x7f183c0cf340 2023-10-19T12:44:29.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.522+0000 7f1840db6640 1 -- 172.21.15.57:0/4070762270 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f183c0cb9b0 con 0x7f183c0cf340 2023-10-19T12:44:29.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.523+0000 7f1832ffd640 1 -- 172.21.15.57:0/4070762270 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) v1 ==== 104+0+11 (secure 0 0 0) 0x7f1824015020 con 0x7f183c0cf340 2023-10-19T12:44:29.526 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2023-10-19T12:44:29.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.525+0000 7f1840db6640 1 -- 172.21.15.57:0/4070762270 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f181c06ae40 msgr2=0x7f181c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:29.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.525+0000 7f1840db6640 1 --2- 172.21.15.57:0/4070762270 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f181c06ae40 0x7f181c06d300 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f182c003eb0 tx=0x7f182c009040 comp rx=0 tx=0).stop 2023-10-19T12:44:29.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.525+0000 7f1840db6640 1 -- 172.21.15.57:0/4070762270 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f183c0cf340 msgr2=0x7f183c1f14c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:29.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.525+0000 7f1840db6640 1 --2- 172.21.15.57:0/4070762270 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f183c0cf340 0x7f183c1f14c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f182400b410 tx=0x7f182400b8d0 comp rx=0 tx=0).stop 2023-10-19T12:44:29.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.525+0000 7f1840db6640 1 -- 172.21.15.57:0/4070762270 shutdown_connections 2023-10-19T12:44:29.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.525+0000 7f1840db6640 1 --2- 172.21.15.57:0/4070762270 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f181c06ae40 0x7f181c06d300 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:29.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.525+0000 7f1840db6640 1 --2- 172.21.15.57:0/4070762270 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f183c15b2a0 0x7f183c1f1a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:29.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.525+0000 7f1840db6640 1 --2- 172.21.15.57:0/4070762270 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f183c0cf340 0x7f183c1f14c0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:29.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.525+0000 7f1840db6640 1 --2- 172.21.15.57:0/4070762270 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f183c0ceb40 0x7f183c0ca900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:29.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.525+0000 7f1840db6640 1 -- 172.21.15.57:0/4070762270 >> 172.21.15.57:0/4070762270 conn(0x7f183c0d0d60 msgr2=0x7f183c0af030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:29.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.525+0000 7f1840db6640 1 -- 172.21.15.57:0/4070762270 shutdown_connections 2023-10-19T12:44:29.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.525+0000 7f1840db6640 1 -- 172.21.15.57:0/4070762270 wait complete. 2023-10-19T12:44:29.538 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-19T12:44:29.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.647+0000 7f28b37d7640 1 -- 172.21.15.57:0/1865934027 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f28980047c0 msgr2=0x7f2898004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:29.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.647+0000 7f28b37d7640 1 --2- 172.21.15.57:0/1865934027 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f28980047c0 0x7f2898004c40 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f289c001680 tx=0x7f28980057f0 comp rx=0 tx=0).stop 2023-10-19T12:44:29.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.648+0000 7f28b37d7640 1 -- 172.21.15.57:0/1865934027 shutdown_connections 2023-10-19T12:44:29.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.648+0000 7f28b37d7640 1 --2- 172.21.15.57:0/1865934027 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f28980047c0 0x7f2898004c40 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:29.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.648+0000 7f28b37d7640 1 -- 172.21.15.57:0/1865934027 >> 172.21.15.57:0/1865934027 conn(0x7f28ac0af7a0 msgr2=0x7f28ac0afb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:29.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.648+0000 7f28b37d7640 1 -- 172.21.15.57:0/1865934027 shutdown_connections 2023-10-19T12:44:29.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.648+0000 7f28b37d7640 1 -- 172.21.15.57:0/1865934027 wait complete. 2023-10-19T12:44:29.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.649+0000 7f28b37d7640 1 Processor -- start 2023-10-19T12:44:29.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.649+0000 7f28b37d7640 1 -- start start 2023-10-19T12:44:29.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.650+0000 7f28b37d7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f28980047c0 0x7f28ac14cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:29.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.650+0000 7f28b37d7640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f28ac0b18e0 0x7f28ac0b1d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:29.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.650+0000 7f28b37d7640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f28ac0c45e0 0x7f28ac0c4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:29.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.650+0000 7f28b37d7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28ac154400 con 0x7f28980047c0 2023-10-19T12:44:29.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.650+0000 7f28b37d7640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28ac155c30 con 0x7f28ac0b18e0 2023-10-19T12:44:29.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.650+0000 7f28b37d7640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28ac0b2280 con 0x7f28ac0c45e0 2023-10-19T12:44:29.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.650+0000 7f28b1573640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f28980047c0 0x7f28ac14cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:29.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.650+0000 7f28b1573640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f28980047c0 0x7f28ac14cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43344/0 (socket says 172.21.15.57:43344) 2023-10-19T12:44:29.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.650+0000 7f28b1573640 1 -- 172.21.15.57:0/188283563 learned_addr learned my addr 172.21.15.57:0/188283563 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:29.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.650+0000 7f28b0d72640 1 --2- 172.21.15.57:0/188283563 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f28ac0b18e0 0x7f28ac0b1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:29.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.650+0000 7f28b1d74640 1 --2- 172.21.15.57:0/188283563 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f28ac0c45e0 0x7f28ac0c4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:29.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.650+0000 7f28b1573640 1 -- 172.21.15.57:0/188283563 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f28ac0c45e0 msgr2=0x7f28ac0c4a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:29.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.650+0000 7f28b1573640 1 --2- 172.21.15.57:0/188283563 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f28ac0c45e0 0x7f28ac0c4a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:29.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.650+0000 7f28b1573640 1 -- 172.21.15.57:0/188283563 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f28ac0b18e0 msgr2=0x7f28ac0b1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:29.658 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.650+0000 7f28b1573640 1 --2- 172.21.15.57:0/188283563 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f28ac0b18e0 0x7f28ac0b1d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:29.658 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.650+0000 7f28b1573640 1 -- 172.21.15.57:0/188283563 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28ac14d430 con 0x7f28980047c0 2023-10-19T12:44:29.658 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.650+0000 7f28b1573640 1 --2- 172.21.15.57:0/188283563 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f28980047c0 0x7f28ac14cef0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f28a8003f00 tx=0x7f28a800b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:29.658 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.650+0000 7f28a1ffb640 1 -- 172.21.15.57:0/188283563 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28a8011760 con 0x7f28980047c0 2023-10-19T12:44:29.658 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.650+0000 7f28a1ffb640 1 -- 172.21.15.57:0/188283563 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f28a8015070 con 0x7f28980047c0 2023-10-19T12:44:29.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.650+0000 7f28a1ffb640 1 -- 172.21.15.57:0/188283563 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28a8011b20 con 0x7f28980047c0 2023-10-19T12:44:29.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.650+0000 7f28b37d7640 1 -- 172.21.15.57:0/188283563 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28ac153780 con 0x7f28980047c0 2023-10-19T12:44:29.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.651+0000 7f28b37d7640 1 -- 172.21.15.57:0/188283563 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28ac153d40 con 0x7f28980047c0 2023-10-19T12:44:29.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.652+0000 7f28b37d7640 1 -- 172.21.15.57:0/188283563 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28ac0690e0 con 0x7f28980047c0 2023-10-19T12:44:29.660 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.652+0000 7f28a1ffb640 1 -- 172.21.15.57:0/188283563 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f28a8017030 con 0x7f28980047c0 2023-10-19T12:44:29.660 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.652+0000 7f28a1ffb640 1 --2- 172.21.15.57:0/188283563 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f289006ae40 0x7f289006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:29.661 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.653+0000 7f28a1ffb640 1 -- 172.21.15.57:0/188283563 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f28a8089fd0 con 0x7f28980047c0 2023-10-19T12:44:29.661 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.653+0000 7f28b0d72640 1 --2- 172.21.15.57:0/188283563 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f289006ae40 0x7f289006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:29.661 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.653+0000 7f28b0d72640 1 --2- 172.21.15.57:0/188283563 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f289006ae40 0x7f289006d300 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f289c034000 tx=0x7f289c001190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:29.661 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.657+0000 7f28a1ffb640 1 -- 172.21.15.57:0/188283563 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f28a8058790 con 0x7f28980047c0 2023-10-19T12:44:29.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.830+0000 7f28b37d7640 1 -- 172.21.15.57:0/188283563 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f28ac14d430 con 0x7f28980047c0 2023-10-19T12:44:29.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.831+0000 7f28a1ffb640 1 -- 172.21.15.57:0/188283563 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) v1 ==== 100+0+11 (secure 0 0 0) 0x7f28ac14d430 con 0x7f28980047c0 2023-10-19T12:44:29.835 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 64 2023-10-19T12:44:29.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.833+0000 7f28b37d7640 1 -- 172.21.15.57:0/188283563 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f289006ae40 msgr2=0x7f289006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:29.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.833+0000 7f28b37d7640 1 --2- 172.21.15.57:0/188283563 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f289006ae40 0x7f289006d300 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f289c034000 tx=0x7f289c001190 comp rx=0 tx=0).stop 2023-10-19T12:44:29.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.833+0000 7f28b37d7640 1 -- 172.21.15.57:0/188283563 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f28980047c0 msgr2=0x7f28ac14cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:29.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.833+0000 7f28b37d7640 1 --2- 172.21.15.57:0/188283563 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f28980047c0 0x7f28ac14cef0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f28a8003f00 tx=0x7f28a800b980 comp rx=0 tx=0).stop 2023-10-19T12:44:29.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.833+0000 7f28b37d7640 1 -- 172.21.15.57:0/188283563 shutdown_connections 2023-10-19T12:44:29.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.833+0000 7f28b37d7640 1 --2- 172.21.15.57:0/188283563 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f289006ae40 0x7f289006d300 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:29.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.833+0000 7f28b37d7640 1 --2- 172.21.15.57:0/188283563 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f28ac0c45e0 0x7f28ac0c4a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:29.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.833+0000 7f28b37d7640 1 --2- 172.21.15.57:0/188283563 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f28ac0b18e0 0x7f28ac0b1d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:29.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.833+0000 7f28b37d7640 1 --2- 172.21.15.57:0/188283563 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f28980047c0 0x7f28ac14cef0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:29.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.833+0000 7f28b37d7640 1 -- 172.21.15.57:0/188283563 >> 172.21.15.57:0/188283563 conn(0x7f28ac0af7a0 msgr2=0x7f28ac157640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:29.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.833+0000 7f28b37d7640 1 -- 172.21.15.57:0/188283563 shutdown_connections 2023-10-19T12:44:29.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.833+0000 7f28b37d7640 1 -- 172.21.15.57:0/188283563 wait complete. 2023-10-19T12:44:29.847 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-19T12:44:29.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.957+0000 7fc883bc9640 1 -- 172.21.15.57:0/2475181447 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc8680039a0 msgr2=0x7fc868003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:29.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.957+0000 7fc883bc9640 1 --2- 172.21.15.57:0/2475181447 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc8680039a0 0x7fc868003dc0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fc878001680 tx=0x7fc8680057f0 comp rx=0 tx=0).stop 2023-10-19T12:44:29.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.957+0000 7fc883bc9640 1 -- 172.21.15.57:0/2475181447 shutdown_connections 2023-10-19T12:44:29.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.957+0000 7fc883bc9640 1 --2- 172.21.15.57:0/2475181447 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc8680039a0 0x7fc868003dc0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:29.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.957+0000 7fc883bc9640 1 -- 172.21.15.57:0/2475181447 >> 172.21.15.57:0/2475181447 conn(0x7fc87c0b3f00 msgr2=0x7fc87c0b4310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:29.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.957+0000 7fc883bc9640 1 -- 172.21.15.57:0/2475181447 shutdown_connections 2023-10-19T12:44:29.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.957+0000 7fc883bc9640 1 -- 172.21.15.57:0/2475181447 wait complete. 2023-10-19T12:44:29.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.958+0000 7fc883bc9640 1 Processor -- start 2023-10-19T12:44:29.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.958+0000 7fc883bc9640 1 -- start start 2023-10-19T12:44:29.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.959+0000 7fc883bc9640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc8680039a0 0x7fc87c06b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:29.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.959+0000 7fc883bc9640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc87c0c96f0 0x7fc87c1528f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:29.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.959+0000 7fc883bc9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc87c1544a0 0x7fc87c154920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:29.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.959+0000 7fc883bc9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc87c0ca7c0 con 0x7fc87c1544a0 2023-10-19T12:44:29.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.959+0000 7fc883bc9640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc87c154e60 con 0x7fc8680039a0 2023-10-19T12:44:29.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.959+0000 7fc883bc9640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc87c0998a0 con 0x7fc87c0c96f0 2023-10-19T12:44:29.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.959+0000 7fc882166640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc87c1544a0 0x7fc87c154920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:29.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.959+0000 7fc882166640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc87c1544a0 0x7fc87c154920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43372/0 (socket says 172.21.15.57:43372) 2023-10-19T12:44:29.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.959+0000 7fc882166640 1 -- 172.21.15.57:0/896625386 learned_addr learned my addr 172.21.15.57:0/896625386 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:29.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.959+0000 7fc881965640 1 --2- 172.21.15.57:0/896625386 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc8680039a0 0x7fc87c06b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:29.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.959+0000 7fc881164640 1 --2- 172.21.15.57:0/896625386 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc87c0c96f0 0x7fc87c1528f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:29.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.959+0000 7fc882166640 1 -- 172.21.15.57:0/896625386 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc87c0c96f0 msgr2=0x7fc87c1528f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:29.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.960+0000 7fc882166640 1 --2- 172.21.15.57:0/896625386 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc87c0c96f0 0x7fc87c1528f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:29.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.960+0000 7fc882166640 1 -- 172.21.15.57:0/896625386 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc8680039a0 msgr2=0x7fc87c06b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:29.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.960+0000 7fc882166640 1 --2- 172.21.15.57:0/896625386 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc8680039a0 0x7fc87c06b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:29.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.960+0000 7fc882166640 1 -- 172.21.15.57:0/896625386 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc87c0c9c40 con 0x7fc87c1544a0 2023-10-19T12:44:29.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.960+0000 7fc881965640 1 --2- 172.21.15.57:0/896625386 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc8680039a0 0x7fc87c06b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:44:29.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.960+0000 7fc882166640 1 --2- 172.21.15.57:0/896625386 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc87c1544a0 0x7fc87c154920 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fc87c0b3450 tx=0x7fc874006720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:29.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.960+0000 7fc8727fc640 1 -- 172.21.15.57:0/896625386 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc87400fa80 con 0x7fc87c1544a0 2023-10-19T12:44:29.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.960+0000 7fc8727fc640 1 -- 172.21.15.57:0/896625386 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fc87400fbe0 con 0x7fc87c1544a0 2023-10-19T12:44:29.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.960+0000 7fc883bc9640 1 -- 172.21.15.57:0/896625386 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc87c0c9f20 con 0x7fc87c1544a0 2023-10-19T12:44:29.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.960+0000 7fc8727fc640 1 -- 172.21.15.57:0/896625386 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc874011430 con 0x7fc87c1544a0 2023-10-19T12:44:29.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.960+0000 7fc883bc9640 1 -- 172.21.15.57:0/896625386 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc87c0ca4e0 con 0x7fc87c1544a0 2023-10-19T12:44:29.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.961+0000 7fc883bc9640 1 -- 172.21.15.57:0/896625386 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc87c0690e0 con 0x7fc87c1544a0 2023-10-19T12:44:29.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.961+0000 7fc8727fc640 1 -- 172.21.15.57:0/896625386 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fc874011590 con 0x7fc87c1544a0 2023-10-19T12:44:29.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.962+0000 7fc8727fc640 1 --2- 172.21.15.57:0/896625386 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fc86006ae40 0x7fc86006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:29.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.962+0000 7fc8727fc640 1 -- 172.21.15.57:0/896625386 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fc87408a3d0 con 0x7fc87c1544a0 2023-10-19T12:44:29.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.962+0000 7fc881965640 1 --2- 172.21.15.57:0/896625386 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fc86006ae40 0x7fc86006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:29.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.962+0000 7fc881965640 1 --2- 172.21.15.57:0/896625386 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fc86006ae40 0x7fc86006d300 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc878034000 tx=0x7fc878039040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:29.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:29.966+0000 7fc8727fc640 1 -- 172.21.15.57:0/896625386 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc8740550f0 con 0x7fc87c1544a0 2023-10-19T12:44:30.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.140+0000 7fc883bc9640 1 -- 172.21.15.57:0/896625386 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fc87c15c160 con 0x7fc87c1544a0 2023-10-19T12:44:30.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.141+0000 7fc8727fc640 1 -- 172.21.15.57:0/896625386 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) v1 ==== 103+0+11 (secure 0 0 0) 0x7fc874058710 con 0x7fc87c1544a0 2023-10-19T12:44:30.144 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2023-10-19T12:44:30.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.143+0000 7fc883bc9640 1 -- 172.21.15.57:0/896625386 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fc86006ae40 msgr2=0x7fc86006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:30.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.143+0000 7fc883bc9640 1 --2- 172.21.15.57:0/896625386 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fc86006ae40 0x7fc86006d300 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc878034000 tx=0x7fc878039040 comp rx=0 tx=0).stop 2023-10-19T12:44:30.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.143+0000 7fc883bc9640 1 -- 172.21.15.57:0/896625386 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc87c1544a0 msgr2=0x7fc87c154920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:30.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.143+0000 7fc883bc9640 1 --2- 172.21.15.57:0/896625386 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc87c1544a0 0x7fc87c154920 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fc87c0b3450 tx=0x7fc874006720 comp rx=0 tx=0).stop 2023-10-19T12:44:30.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.143+0000 7fc883bc9640 1 -- 172.21.15.57:0/896625386 shutdown_connections 2023-10-19T12:44:30.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.143+0000 7fc883bc9640 1 --2- 172.21.15.57:0/896625386 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fc86006ae40 0x7fc86006d300 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:30.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.143+0000 7fc883bc9640 1 --2- 172.21.15.57:0/896625386 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc87c1544a0 0x7fc87c154920 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:30.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.143+0000 7fc883bc9640 1 --2- 172.21.15.57:0/896625386 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc87c0c96f0 0x7fc87c1528f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:30.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.143+0000 7fc883bc9640 1 --2- 172.21.15.57:0/896625386 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc8680039a0 0x7fc87c06b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:30.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.143+0000 7fc883bc9640 1 -- 172.21.15.57:0/896625386 >> 172.21.15.57:0/896625386 conn(0x7fc87c0b3f00 msgr2=0x7fc87c156790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:30.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.143+0000 7fc883bc9640 1 -- 172.21.15.57:0/896625386 shutdown_connections 2023-10-19T12:44:30.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.143+0000 7fc883bc9640 1 -- 172.21.15.57:0/896625386 wait complete. 2023-10-19T12:44:30.157 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-19T12:44:30.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.266+0000 7fbc58858640 1 -- 172.21.15.57:0/25924349 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc40004d00 msgr2=0x7fbc40005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:30.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.266+0000 7fbc58858640 1 --2- 172.21.15.57:0/25924349 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc40004d00 0x7fbc40005180 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fbc4c009b50 tx=0x7fbc400057f0 comp rx=0 tx=0).stop 2023-10-19T12:44:30.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.266+0000 7fbc52d9d640 1 -- 172.21.15.57:0/25924349 reap_dead start 2023-10-19T12:44:30.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.266+0000 7fbc58858640 1 -- 172.21.15.57:0/25924349 shutdown_connections 2023-10-19T12:44:30.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.266+0000 7fbc58858640 1 -- 172.21.15.57:0/25924349 >> 172.21.15.57:0/25924349 conn(0x7fbc540d0c50 msgr2=0x7fbc540d1060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:30.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.266+0000 7fbc58858640 1 -- 172.21.15.57:0/25924349 shutdown_connections 2023-10-19T12:44:30.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.266+0000 7fbc58858640 1 -- 172.21.15.57:0/25924349 wait complete. 2023-10-19T12:44:30.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.267+0000 7fbc58858640 1 Processor -- start 2023-10-19T12:44:30.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.267+0000 7fbc58858640 1 -- start start 2023-10-19T12:44:30.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.268+0000 7fbc58858640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbc40004d00 0x7fbc5414abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:30.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.268+0000 7fbc58858640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbc540c96d0 0x7fbc540c7c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:30.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.268+0000 7fbc58858640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc5415cb00 0x7fbc5415cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:30.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.268+0000 7fbc58858640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc5415d4f0 con 0x7fbc5415cb00 2023-10-19T12:44:30.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.268+0000 7fbc58858640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc540c8630 con 0x7fbc540c96d0 2023-10-19T12:44:30.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.268+0000 7fbc58858640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc540c8910 con 0x7fbc40004d00 2023-10-19T12:44:30.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.268+0000 7fbc52d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc5415cb00 0x7fbc5415cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:30.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.268+0000 7fbc52d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc5415cb00 0x7fbc5415cfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43400/0 (socket says 172.21.15.57:43400) 2023-10-19T12:44:30.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.268+0000 7fbc52d9d640 1 -- 172.21.15.57:0/3222013558 learned_addr learned my addr 172.21.15.57:0/3222013558 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:30.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.268+0000 7fbc5259c640 1 --2- 172.21.15.57:0/3222013558 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbc40004d00 0x7fbc5414abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:30.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.268+0000 7fbc51d9b640 1 --2- 172.21.15.57:0/3222013558 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbc540c96d0 0x7fbc540c7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:30.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.268+0000 7fbc52d9d640 1 -- 172.21.15.57:0/3222013558 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbc40004d00 msgr2=0x7fbc5414abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:30.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.268+0000 7fbc52d9d640 1 --2- 172.21.15.57:0/3222013558 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbc40004d00 0x7fbc5414abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:30.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.268+0000 7fbc52d9d640 1 -- 172.21.15.57:0/3222013558 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbc540c96d0 msgr2=0x7fbc540c7c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:30.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.268+0000 7fbc52d9d640 1 --2- 172.21.15.57:0/3222013558 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbc540c96d0 0x7fbc540c7c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:30.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.268+0000 7fbc52d9d640 1 -- 172.21.15.57:0/3222013558 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc540c8bf0 con 0x7fbc5415cb00 2023-10-19T12:44:30.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.269+0000 7fbc52d9d640 1 --2- 172.21.15.57:0/3222013558 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc5415cb00 0x7fbc5415cfb0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fbc4c009b50 tx=0x7fbc4c035c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:30.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.269+0000 7fbc4affd640 1 -- 172.21.15.57:0/3222013558 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc4c035270 con 0x7fbc5415cb00 2023-10-19T12:44:30.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.269+0000 7fbc4affd640 1 -- 172.21.15.57:0/3222013558 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fbc4c004030 con 0x7fbc5415cb00 2023-10-19T12:44:30.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.269+0000 7fbc58858640 1 -- 172.21.15.57:0/3222013558 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc540c8ed0 con 0x7fbc5415cb00 2023-10-19T12:44:30.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.269+0000 7fbc4affd640 1 -- 172.21.15.57:0/3222013558 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc4c004400 con 0x7fbc5415cb00 2023-10-19T12:44:30.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.269+0000 7fbc58858640 1 -- 172.21.15.57:0/3222013558 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc540c6a80 con 0x7fbc5415cb00 2023-10-19T12:44:30.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.270+0000 7fbc58858640 1 -- 172.21.15.57:0/3222013558 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc540c93f0 con 0x7fbc5415cb00 2023-10-19T12:44:30.278 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.270+0000 7fbc4affd640 1 -- 172.21.15.57:0/3222013558 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fbc4c0035f0 con 0x7fbc5415cb00 2023-10-19T12:44:30.279 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.271+0000 7fbc4affd640 1 --2- 172.21.15.57:0/3222013558 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fbc3406ae70 0x7fbc3406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:30.279 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.271+0000 7fbc4affd640 1 -- 172.21.15.57:0/3222013558 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fbc4c000e40 con 0x7fbc5415cb00 2023-10-19T12:44:30.279 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.271+0000 7fbc5259c640 1 --2- 172.21.15.57:0/3222013558 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fbc3406ae70 0x7fbc3406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:30.279 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.271+0000 7fbc5259c640 1 --2- 172.21.15.57:0/3222013558 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fbc3406ae70 0x7fbc3406d330 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fbc440013d0 tx=0x7fbc44000f40 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:30.280 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.275+0000 7fbc4affd640 1 -- 172.21.15.57:0/3222013558 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fbc540c93f0 con 0x7fbc5415cb00 2023-10-19T12:44:30.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.465+0000 7fbc58858640 1 -- 172.21.15.57:0/3222013558 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fbc540690e0 con 0x7fbc5415cb00 2023-10-19T12:44:30.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.465+0000 7fbc4affd640 1 -- 172.21.15.57:0/3222013558 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v25) v1 ==== 80+0+0 (secure 0 0 0) 0x7fbc4c077250 con 0x7fbc5415cb00 2023-10-19T12:44:30.469 INFO:teuthology.orchestra.run.smithi057.stderr:listed 0 entries 2023-10-19T12:44:30.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.467+0000 7fbc58858640 1 -- 172.21.15.57:0/3222013558 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fbc3406ae70 msgr2=0x7fbc3406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:30.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.467+0000 7fbc58858640 1 --2- 172.21.15.57:0/3222013558 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fbc3406ae70 0x7fbc3406d330 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fbc440013d0 tx=0x7fbc44000f40 comp rx=0 tx=0).stop 2023-10-19T12:44:30.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.467+0000 7fbc58858640 1 -- 172.21.15.57:0/3222013558 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc5415cb00 msgr2=0x7fbc5415cfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:30.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.467+0000 7fbc58858640 1 --2- 172.21.15.57:0/3222013558 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc5415cb00 0x7fbc5415cfb0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fbc4c009b50 tx=0x7fbc4c035c10 comp rx=0 tx=0).stop 2023-10-19T12:44:30.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.467+0000 7fbc58858640 1 -- 172.21.15.57:0/3222013558 shutdown_connections 2023-10-19T12:44:30.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.467+0000 7fbc58858640 1 --2- 172.21.15.57:0/3222013558 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fbc3406ae70 0x7fbc3406d330 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:30.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.467+0000 7fbc58858640 1 --2- 172.21.15.57:0/3222013558 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc5415cb00 0x7fbc5415cfb0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:30.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.467+0000 7fbc58858640 1 --2- 172.21.15.57:0/3222013558 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbc540c96d0 0x7fbc540c7c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:30.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.467+0000 7fbc58858640 1 --2- 172.21.15.57:0/3222013558 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbc40004d00 0x7fbc5414abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:30.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.467+0000 7fbc58858640 1 -- 172.21.15.57:0/3222013558 >> 172.21.15.57:0/3222013558 conn(0x7fbc540d0c50 msgr2=0x7fbc540b4ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:30.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.467+0000 7fbc58858640 1 -- 172.21.15.57:0/3222013558 shutdown_connections 2023-10-19T12:44:30.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.467+0000 7fbc58858640 1 -- 172.21.15.57:0/3222013558 wait complete. 2023-10-19T12:44:30.481 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T12:44:30.482 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T12:44:30.497 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T12:44:30.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.643+0000 7f2aad355640 1 -- 172.21.15.57:0/62512643 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2aa80b1ca0 msgr2=0x7f2aa80b42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:30.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.643+0000 7f2aad355640 1 --2- 172.21.15.57:0/62512643 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2aa80b1ca0 0x7f2aa80b42d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f2a98005fd0 tx=0x7f2a9802c2e0 comp rx=0 tx=0).stop 2023-10-19T12:44:30.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.643+0000 7f2aad355640 1 -- 172.21.15.57:0/62512643 shutdown_connections 2023-10-19T12:44:30.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.643+0000 7f2aad355640 1 --2- 172.21.15.57:0/62512643 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2aa80b1ca0 0x7f2aa80b42d0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:30.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.643+0000 7f2aad355640 1 --2- 172.21.15.57:0/62512643 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2aa80cf8b0 0x7f2aa80ce9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:30.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.643+0000 7f2aad355640 1 -- 172.21.15.57:0/62512643 >> 172.21.15.57:0/62512643 conn(0x7f2aa80d0c50 msgr2=0x7f2aa80d1060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:30.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.643+0000 7f2aad355640 1 -- 172.21.15.57:0/62512643 shutdown_connections 2023-10-19T12:44:30.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.643+0000 7f2aad355640 1 -- 172.21.15.57:0/62512643 wait complete. 2023-10-19T12:44:30.650 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.644+0000 7f2aad355640 1 Processor -- start 2023-10-19T12:44:30.650 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.644+0000 7f2aad355640 1 -- start start 2023-10-19T12:44:30.650 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.645+0000 7f2aad355640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2aa80b1ca0 0x7f2aa8155850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:30.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.645+0000 7f2aad355640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2aa80cf8b0 0x7f2aa80c9a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:30.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.645+0000 7f2aad355640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2aa815e5e0 0x7f2aa809bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:30.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.645+0000 7f2aad355640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa81596e0 con 0x7f2aa80b1ca0 2023-10-19T12:44:30.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.645+0000 7f2aad355640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa8159980 con 0x7f2aa815e5e0 2023-10-19T12:44:30.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.645+0000 7f2aad355640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa8159c60 con 0x7f2aa80cf8b0 2023-10-19T12:44:30.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.645+0000 7f2aa6ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2aa80b1ca0 0x7f2aa8155850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:30.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.645+0000 7f2aa6ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2aa80b1ca0 0x7f2aa8155850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43416/0 (socket says 172.21.15.57:43416) 2023-10-19T12:44:30.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.645+0000 7f2aa6ffd640 1 -- 172.21.15.57:0/967671158 learned_addr learned my addr 172.21.15.57:0/967671158 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:30.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.645+0000 7f2aa77fe640 1 --2- 172.21.15.57:0/967671158 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2aa815e5e0 0x7f2aa809bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:30.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.645+0000 7f2aa67fc640 1 --2- 172.21.15.57:0/967671158 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2aa80cf8b0 0x7f2aa80c9a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:30.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.645+0000 7f2aa6ffd640 1 -- 172.21.15.57:0/967671158 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2aa80cf8b0 msgr2=0x7f2aa80c9a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:30.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.645+0000 7f2aa6ffd640 1 --2- 172.21.15.57:0/967671158 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2aa80cf8b0 0x7f2aa80c9a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:30.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.645+0000 7f2aa6ffd640 1 -- 172.21.15.57:0/967671158 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2aa815e5e0 msgr2=0x7f2aa809bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:30.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.645+0000 7f2aa6ffd640 1 --2- 172.21.15.57:0/967671158 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2aa815e5e0 0x7f2aa809bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:30.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.645+0000 7f2aa6ffd640 1 -- 172.21.15.57:0/967671158 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a98009900 con 0x7f2aa80b1ca0 2023-10-19T12:44:30.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.645+0000 7f2aa67fc640 1 --2- 172.21.15.57:0/967671158 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2aa80cf8b0 0x7f2aa80c9a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:44:30.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.646+0000 7f2aa6ffd640 1 --2- 172.21.15.57:0/967671158 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2aa80b1ca0 0x7f2aa8155850 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f2aa000c3c0 tx=0x7f2aa000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:30.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.646+0000 7f2a9f7fe640 1 -- 172.21.15.57:0/967671158 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aa00052d0 con 0x7f2aa80b1ca0 2023-10-19T12:44:30.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.646+0000 7f2a9f7fe640 1 -- 172.21.15.57:0/967671158 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f2aa0007020 con 0x7f2aa80b1ca0 2023-10-19T12:44:30.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.646+0000 7f2a9f7fe640 1 -- 172.21.15.57:0/967671158 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aa0007470 con 0x7f2aa80b1ca0 2023-10-19T12:44:30.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.646+0000 7f2aad355640 1 -- 172.21.15.57:0/967671158 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2aa8159f40 con 0x7f2aa80b1ca0 2023-10-19T12:44:30.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.646+0000 7f2aad355640 1 -- 172.21.15.57:0/967671158 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2aa8155d90 con 0x7f2aa80b1ca0 2023-10-19T12:44:30.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.647+0000 7f2aad355640 1 -- 172.21.15.57:0/967671158 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2aa815a460 con 0x7f2aa80b1ca0 2023-10-19T12:44:30.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.648+0000 7f2a9f7fe640 1 -- 172.21.15.57:0/967671158 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f2aa00067d0 con 0x7f2aa80b1ca0 2023-10-19T12:44:30.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.648+0000 7f2a9f7fe640 1 --2- 172.21.15.57:0/967671158 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f2a8806ae70 0x7f2a8806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:30.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.648+0000 7f2a9f7fe640 1 -- 172.21.15.57:0/967671158 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f2aa0088d70 con 0x7f2aa80b1ca0 2023-10-19T12:44:30.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.648+0000 7f2aa67fc640 1 --2- 172.21.15.57:0/967671158 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f2a8806ae70 0x7f2a8806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:30.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.649+0000 7f2aa67fc640 1 --2- 172.21.15.57:0/967671158 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f2a8806ae70 0x7f2a8806d330 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2aa806b530 tx=0x7f2a90005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:30.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.652+0000 7f2a9f7fe640 1 -- 172.21.15.57:0/967671158 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2aa815a460 con 0x7f2aa80b1ca0 2023-10-19T12:44:30.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.833+0000 7f2aad355640 1 -- 172.21.15.57:0/967671158 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2aa80cb340 con 0x7f2aa80b1ca0 2023-10-19T12:44:30.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.834+0000 7f2a9f7fe640 1 -- 172.21.15.57:0/967671158 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18358 (secure 0 0 0) 0x7f2aa0053c70 con 0x7f2aa80b1ca0 2023-10-19T12:44:30.837 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:44:30.838 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":25,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T12:44:23.472399+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-19T12:44:16.517510+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-19T12:44:17.398297+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-19T12:44:21.129403+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T12:44:30.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.836+0000 7f2aad355640 1 -- 172.21.15.57:0/967671158 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f2a8806ae70 msgr2=0x7f2a8806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:30.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.836+0000 7f2aad355640 1 --2- 172.21.15.57:0/967671158 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f2a8806ae70 0x7f2a8806d330 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2aa806b530 tx=0x7f2a90005f90 comp rx=0 tx=0).stop 2023-10-19T12:44:30.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.836+0000 7f2aad355640 1 -- 172.21.15.57:0/967671158 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2aa80b1ca0 msgr2=0x7f2aa8155850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:30.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.836+0000 7f2aad355640 1 --2- 172.21.15.57:0/967671158 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2aa80b1ca0 0x7f2aa8155850 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f2aa000c3c0 tx=0x7f2aa000c880 comp rx=0 tx=0).stop 2023-10-19T12:44:30.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.836+0000 7f2aad355640 1 -- 172.21.15.57:0/967671158 shutdown_connections 2023-10-19T12:44:30.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.836+0000 7f2aad355640 1 --2- 172.21.15.57:0/967671158 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f2a8806ae70 0x7f2a8806d330 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:30.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.836+0000 7f2aad355640 1 --2- 172.21.15.57:0/967671158 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2aa815e5e0 0x7f2aa809bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:30.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.836+0000 7f2aad355640 1 --2- 172.21.15.57:0/967671158 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2aa80cf8b0 0x7f2aa80c9a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:30.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.836+0000 7f2aad355640 1 --2- 172.21.15.57:0/967671158 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2aa80b1ca0 0x7f2aa8155850 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:30.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.836+0000 7f2aad355640 1 -- 172.21.15.57:0/967671158 >> 172.21.15.57:0/967671158 conn(0x7f2aa80d0c50 msgr2=0x7f2aa815adf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:30.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.836+0000 7f2aad355640 1 -- 172.21.15.57:0/967671158 shutdown_connections 2023-10-19T12:44:30.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:30.836+0000 7f2aad355640 1 -- 172.21.15.57:0/967671158 wait complete. 2023-10-19T12:44:30.850 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-19T12:44:07.371186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-19T12:44:16.517510+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-19T12:44:17.398297+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-19T12:44:21.129403+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-19T12:44:30.851 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-19T12:44:31.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.017+0000 7f10c30d7640 1 -- 172.21.15.57:0/2455233596 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f10a4005180 msgr2=0x7f10a4005950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:31.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.017+0000 7f10c30d7640 1 --2- 172.21.15.57:0/2455233596 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f10a4005180 0x7f10a4005950 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f10bc066ec0 tx=0x7f10a40057f0 comp rx=0 tx=0).stop 2023-10-19T12:44:31.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.017+0000 7f10c1674640 1 -- 172.21.15.57:0/2455233596 reap_dead start 2023-10-19T12:44:31.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.017+0000 7f10c30d7640 1 -- 172.21.15.57:0/2455233596 shutdown_connections 2023-10-19T12:44:31.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.017+0000 7f10c30d7640 1 -- 172.21.15.57:0/2455233596 >> 172.21.15.57:0/2455233596 conn(0x7f10bc0ae720 msgr2=0x7f10bc0aeb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:31.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.017+0000 7f10c30d7640 1 -- 172.21.15.57:0/2455233596 shutdown_connections 2023-10-19T12:44:31.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.017+0000 7f10c30d7640 1 -- 172.21.15.57:0/2455233596 wait complete. 2023-10-19T12:44:31.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.018+0000 7f10c30d7640 1 Processor -- start 2023-10-19T12:44:31.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.019+0000 7f10c30d7640 1 -- start start 2023-10-19T12:44:31.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.019+0000 7f10c30d7640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f10bc087990 0x7f10bc162980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:31.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.019+0000 7f10c30d7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f10bc162ec0 0x7f10bc163320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:31.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.019+0000 7f10c30d7640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f10bc163860 0x7f10bc0aa8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:31.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.019+0000 7f10c30d7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10bc163d10 con 0x7f10bc162ec0 2023-10-19T12:44:31.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.019+0000 7f10c30d7640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10bc161bc0 con 0x7f10bc087990 2023-10-19T12:44:31.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.019+0000 7f10c30d7640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10bc161ea0 con 0x7f10bc163860 2023-10-19T12:44:31.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.020+0000 7f10b3fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f10bc162ec0 0x7f10bc163320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:31.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.020+0000 7f10b3fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f10bc162ec0 0x7f10bc163320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43428/0 (socket says 172.21.15.57:43428) 2023-10-19T12:44:31.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.020+0000 7f10b3fff640 1 -- 172.21.15.57:0/1699036530 learned_addr learned my addr 172.21.15.57:0/1699036530 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:31.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.020+0000 7f10c1674640 1 --2- 172.21.15.57:0/1699036530 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f10bc163860 0x7f10bc0aa8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:31.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.020+0000 7f10c0e73640 1 --2- 172.21.15.57:0/1699036530 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f10bc087990 0x7f10bc162980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:31.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.020+0000 7f10b3fff640 1 -- 172.21.15.57:0/1699036530 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f10bc163860 msgr2=0x7f10bc0aa8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:31.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.020+0000 7f10b3fff640 1 --2- 172.21.15.57:0/1699036530 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f10bc163860 0x7f10bc0aa8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:31.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.020+0000 7f10b3fff640 1 -- 172.21.15.57:0/1699036530 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f10bc087990 msgr2=0x7f10bc162980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:31.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.020+0000 7f10b3fff640 1 --2- 172.21.15.57:0/1699036530 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f10bc087990 0x7f10bc162980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:31.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.020+0000 7f10b3fff640 1 -- 172.21.15.57:0/1699036530 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10bc162180 con 0x7f10bc162ec0 2023-10-19T12:44:31.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.020+0000 7f10c0e73640 1 --2- 172.21.15.57:0/1699036530 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f10bc087990 0x7f10bc162980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:44:31.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.020+0000 7f10b3fff640 1 --2- 172.21.15.57:0/1699036530 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f10bc162ec0 0x7f10bc163320 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f10ac003670 tx=0x7f10ac007430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:31.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.020+0000 7f10b17fa640 1 -- 172.21.15.57:0/1699036530 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10ac00e9a0 con 0x7f10bc162ec0 2023-10-19T12:44:31.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.020+0000 7f10b17fa640 1 -- 172.21.15.57:0/1699036530 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f10ac00ebf0 con 0x7f10bc162ec0 2023-10-19T12:44:31.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.020+0000 7f10c30d7640 1 -- 172.21.15.57:0/1699036530 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10bc162460 con 0x7f10bc162ec0 2023-10-19T12:44:31.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.020+0000 7f10b17fa640 1 -- 172.21.15.57:0/1699036530 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10ac011450 con 0x7f10bc162ec0 2023-10-19T12:44:31.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.020+0000 7f10c30d7640 1 -- 172.21.15.57:0/1699036530 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10bc087d70 con 0x7f10bc162ec0 2023-10-19T12:44:31.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.021+0000 7f10c30d7640 1 -- 172.21.15.57:0/1699036530 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10bc162680 con 0x7f10bc162ec0 2023-10-19T12:44:31.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.022+0000 7f10b17fa640 1 -- 172.21.15.57:0/1699036530 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f10ac0115b0 con 0x7f10bc162ec0 2023-10-19T12:44:31.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.022+0000 7f10b17fa640 1 --2- 172.21.15.57:0/1699036530 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f10a806ae40 0x7f10a806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:31.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.022+0000 7f10b17fa640 1 -- 172.21.15.57:0/1699036530 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f10ac08a240 con 0x7f10bc162ec0 2023-10-19T12:44:31.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.022+0000 7f10c0e73640 1 --2- 172.21.15.57:0/1699036530 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f10a806ae40 0x7f10a806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:31.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.023+0000 7f10c0e73640 1 --2- 172.21.15.57:0/1699036530 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f10a806ae40 0x7f10a806d300 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f10b8000c00 tx=0x7f10b8006b30 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:31.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.026+0000 7f10b17fa640 1 -- 172.21.15.57:0/1699036530 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f10ac055000 con 0x7f10bc162ec0 2023-10-19T12:44:31.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.205+0000 7f10c30d7640 1 -- 172.21.15.57:0/1699036530 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f10bc0690e0 con 0x7f10bc162ec0 2023-10-19T12:44:31.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.206+0000 7f10b17fa640 1 -- 172.21.15.57:0/1699036530 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) v1 ==== 93+0+10 (secure 0 0 0) 0x7f10ac058620 con 0x7f10bc162ec0 2023-10-19T12:44:31.209 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-19T12:44:31.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.208+0000 7f10c30d7640 1 -- 172.21.15.57:0/1699036530 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f10a806ae40 msgr2=0x7f10a806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:31.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.208+0000 7f10c30d7640 1 --2- 172.21.15.57:0/1699036530 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f10a806ae40 0x7f10a806d300 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f10b8000c00 tx=0x7f10b8006b30 comp rx=0 tx=0).stop 2023-10-19T12:44:31.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.208+0000 7f10c30d7640 1 -- 172.21.15.57:0/1699036530 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f10bc162ec0 msgr2=0x7f10bc163320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:31.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.208+0000 7f10c30d7640 1 --2- 172.21.15.57:0/1699036530 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f10bc162ec0 0x7f10bc163320 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f10ac003670 tx=0x7f10ac007430 comp rx=0 tx=0).stop 2023-10-19T12:44:31.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.208+0000 7f10c30d7640 1 -- 172.21.15.57:0/1699036530 shutdown_connections 2023-10-19T12:44:31.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.208+0000 7f10c30d7640 1 --2- 172.21.15.57:0/1699036530 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f10a806ae40 0x7f10a806d300 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:31.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.208+0000 7f10c30d7640 1 --2- 172.21.15.57:0/1699036530 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f10bc163860 0x7f10bc0aa8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:31.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.208+0000 7f10c30d7640 1 --2- 172.21.15.57:0/1699036530 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f10bc162ec0 0x7f10bc163320 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:31.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.208+0000 7f10c30d7640 1 --2- 172.21.15.57:0/1699036530 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f10bc087990 0x7f10bc162980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:31.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.208+0000 7f10c30d7640 1 -- 172.21.15.57:0/1699036530 >> 172.21.15.57:0/1699036530 conn(0x7f10bc0ae720 msgr2=0x7f10bc1675d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:31.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.208+0000 7f10c30d7640 1 -- 172.21.15.57:0/1699036530 shutdown_connections 2023-10-19T12:44:31.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.208+0000 7f10c30d7640 1 -- 172.21.15.57:0/1699036530 wait complete. 2023-10-19T12:44:31.221 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-19T12:44:31.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.331+0000 7f5445be2640 1 -- 172.21.15.57:0/2947343421 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f54400cf340 msgr2=0x7f54400d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:31.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.331+0000 7f5445be2640 1 --2- 172.21.15.57:0/2947343421 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f54400cf340 0x7f54400d1910 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f5430009830 tx=0x7f543002bcd0 comp rx=0 tx=0).stop 2023-10-19T12:44:31.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.331+0000 7f5445be2640 1 -- 172.21.15.57:0/2947343421 shutdown_connections 2023-10-19T12:44:31.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.331+0000 7f5445be2640 1 --2- 172.21.15.57:0/2947343421 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f54400cf340 0x7f54400d1910 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:31.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.331+0000 7f5445be2640 1 -- 172.21.15.57:0/2947343421 >> 172.21.15.57:0/2947343421 conn(0x7f54400d0d60 msgr2=0x7f54400d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:31.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.331+0000 7f5445be2640 1 -- 172.21.15.57:0/2947343421 shutdown_connections 2023-10-19T12:44:31.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.331+0000 7f5445be2640 1 -- 172.21.15.57:0/2947343421 wait complete. 2023-10-19T12:44:31.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.332+0000 7f5445be2640 1 Processor -- start 2023-10-19T12:44:31.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.333+0000 7f5445be2640 1 -- start start 2023-10-19T12:44:31.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.333+0000 7f5445be2640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f54400ceb40 0x7f54400c9170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:31.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.333+0000 7f5445be2640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f54400cf340 0x7f54400b5a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:31.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.333+0000 7f5445be2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f544015b2a0 0x7f54400caa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:31.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.333+0000 7f5445be2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54400b5030 con 0x7f544015b2a0 2023-10-19T12:44:31.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.333+0000 7f5445be2640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54400c8ce0 con 0x7f54400ceb40 2023-10-19T12:44:31.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.333+0000 7f5445be2640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54400b4a70 con 0x7f54400cf340 2023-10-19T12:44:31.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.333+0000 7f543ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f544015b2a0 0x7f54400caa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:31.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.333+0000 7f543ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f544015b2a0 0x7f54400caa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43450/0 (socket says 172.21.15.57:43450) 2023-10-19T12:44:31.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.333+0000 7f543ffff640 1 -- 172.21.15.57:0/1573903266 learned_addr learned my addr 172.21.15.57:0/1573903266 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:31.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.334+0000 7f543ffff640 1 -- 172.21.15.57:0/1573903266 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f54400cf340 msgr2=0x7f54400b5a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:31.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.334+0000 7f5436ffd640 1 --2- 172.21.15.57:0/1573903266 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f54400cf340 0x7f54400b5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:31.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.334+0000 7f543f7fe640 1 --2- 172.21.15.57:0/1573903266 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f54400ceb40 0x7f54400c9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:31.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.334+0000 7f543ffff640 1 --2- 172.21.15.57:0/1573903266 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f54400cf340 0x7f54400b5a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:31.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.334+0000 7f543ffff640 1 -- 172.21.15.57:0/1573903266 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f54400ceb40 msgr2=0x7f54400c9170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:31.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.334+0000 7f543ffff640 1 --2- 172.21.15.57:0/1573903266 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f54400ceb40 0x7f54400c9170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:31.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.334+0000 7f543ffff640 1 -- 172.21.15.57:0/1573903266 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5430009490 con 0x7f544015b2a0 2023-10-19T12:44:31.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.334+0000 7f543ffff640 1 --2- 172.21.15.57:0/1573903266 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f544015b2a0 0x7f54400caa30 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f543800cb70 tx=0x7f543800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:31.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.334+0000 7f543cff9640 1 -- 172.21.15.57:0/1573903266 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f543800ad40 con 0x7f544015b2a0 2023-10-19T12:44:31.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.334+0000 7f543cff9640 1 -- 172.21.15.57:0/1573903266 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f5438012420 con 0x7f544015b2a0 2023-10-19T12:44:31.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.334+0000 7f543cff9640 1 -- 172.21.15.57:0/1573903266 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5438012810 con 0x7f544015b2a0 2023-10-19T12:44:31.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.334+0000 7f5445be2640 1 -- 172.21.15.57:0/1573903266 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54400b4d50 con 0x7f544015b2a0 2023-10-19T12:44:31.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.334+0000 7f5445be2640 1 -- 172.21.15.57:0/1573903266 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5440158a60 con 0x7f544015b2a0 2023-10-19T12:44:31.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.335+0000 7f5445be2640 1 -- 172.21.15.57:0/1573903266 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54400c8e70 con 0x7f544015b2a0 2023-10-19T12:44:31.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.335+0000 7f543cff9640 1 -- 172.21.15.57:0/1573903266 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f543801a3b0 con 0x7f544015b2a0 2023-10-19T12:44:31.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.336+0000 7f543cff9640 1 --2- 172.21.15.57:0/1573903266 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f542006ae40 0x7f542006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:31.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.336+0000 7f543cff9640 1 -- 172.21.15.57:0/1573903266 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f543808a030 con 0x7f544015b2a0 2023-10-19T12:44:31.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.336+0000 7f543f7fe640 1 --2- 172.21.15.57:0/1573903266 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f542006ae40 0x7f542006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:31.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.337+0000 7f543f7fe640 1 --2- 172.21.15.57:0/1573903266 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f542006ae40 0x7f542006d300 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5430036000 tx=0x7f5430004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:31.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.340+0000 7f543cff9640 1 -- 172.21.15.57:0/1573903266 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f5438054e80 con 0x7f544015b2a0 2023-10-19T12:44:31.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.514+0000 7f5445be2640 1 -- 172.21.15.57:0/1573903266 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f54400690e0 con 0x7f544015b2a0 2023-10-19T12:44:31.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.515+0000 7f543cff9640 1 -- 172.21.15.57:0/1573903266 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) v1 ==== 104+0+11 (secure 0 0 0) 0x7f54380584a0 con 0x7f544015b2a0 2023-10-19T12:44:31.518 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2023-10-19T12:44:31.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.516+0000 7f5445be2640 1 -- 172.21.15.57:0/1573903266 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f542006ae40 msgr2=0x7f542006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:31.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.516+0000 7f5445be2640 1 --2- 172.21.15.57:0/1573903266 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f542006ae40 0x7f542006d300 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5430036000 tx=0x7f5430004730 comp rx=0 tx=0).stop 2023-10-19T12:44:31.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.516+0000 7f5445be2640 1 -- 172.21.15.57:0/1573903266 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f544015b2a0 msgr2=0x7f54400caa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:31.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.516+0000 7f5445be2640 1 --2- 172.21.15.57:0/1573903266 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f544015b2a0 0x7f54400caa30 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f543800cb70 tx=0x7f543800a430 comp rx=0 tx=0).stop 2023-10-19T12:44:31.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.517+0000 7f5445be2640 1 -- 172.21.15.57:0/1573903266 shutdown_connections 2023-10-19T12:44:31.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.517+0000 7f5445be2640 1 --2- 172.21.15.57:0/1573903266 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f542006ae40 0x7f542006d300 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:31.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.517+0000 7f5445be2640 1 --2- 172.21.15.57:0/1573903266 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f544015b2a0 0x7f54400caa30 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:31.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.517+0000 7f5445be2640 1 --2- 172.21.15.57:0/1573903266 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f54400cf340 0x7f54400b5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:31.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.517+0000 7f5445be2640 1 --2- 172.21.15.57:0/1573903266 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f54400ceb40 0x7f54400c9170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:31.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.517+0000 7f5445be2640 1 -- 172.21.15.57:0/1573903266 >> 172.21.15.57:0/1573903266 conn(0x7f54400d0d60 msgr2=0x7f544006b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:31.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.517+0000 7f5445be2640 1 -- 172.21.15.57:0/1573903266 shutdown_connections 2023-10-19T12:44:31.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.517+0000 7f5445be2640 1 -- 172.21.15.57:0/1573903266 wait complete. 2023-10-19T12:44:31.530 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-19T12:44:31.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.639+0000 7fddb4a32640 1 -- 172.21.15.57:0/806451778 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fddb013d350 msgr2=0x7fddb00cbd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:31.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.639+0000 7fddb4a32640 1 --2- 172.21.15.57:0/806451778 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fddb013d350 0x7fddb00cbd60 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fdda0009830 tx=0x7fdda002bd60 comp rx=0 tx=0).stop 2023-10-19T12:44:31.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.639+0000 7fddb4a32640 1 -- 172.21.15.57:0/806451778 shutdown_connections 2023-10-19T12:44:31.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.639+0000 7fddb4a32640 1 --2- 172.21.15.57:0/806451778 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fddb013d350 0x7fddb00cbd60 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:31.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.639+0000 7fddb4a32640 1 -- 172.21.15.57:0/806451778 >> 172.21.15.57:0/806451778 conn(0x7fddb0069930 msgr2=0x7fddb0069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:31.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.639+0000 7fddb4a32640 1 -- 172.21.15.57:0/806451778 shutdown_connections 2023-10-19T12:44:31.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.639+0000 7fddb4a32640 1 -- 172.21.15.57:0/806451778 wait complete. 2023-10-19T12:44:31.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.640+0000 7fddb4a32640 1 Processor -- start 2023-10-19T12:44:31.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.640+0000 7fddb4a32640 1 -- start start 2023-10-19T12:44:31.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.641+0000 7fddb4a32640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fddb0070350 0x7fddb0135f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:31.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.641+0000 7fddb4a32640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fddb00cb980 0x7fddb00cb3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:31.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.641+0000 7fddb4a32640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fddb01513e0 0x7fddb01341d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:31.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.641+0000 7fddb4a32640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddb00caf90 con 0x7fddb00cb980 2023-10-19T12:44:31.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.641+0000 7fddb4a32640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddb0136ad0 con 0x7fddb0070350 2023-10-19T12:44:31.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.641+0000 7fddb4a32640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddb0134710 con 0x7fddb01513e0 2023-10-19T12:44:31.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.641+0000 7fddadd9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fddb00cb980 0x7fddb00cb3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:31.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.641+0000 7fddadd9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fddb00cb980 0x7fddb00cb3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43466/0 (socket says 172.21.15.57:43466) 2023-10-19T12:44:31.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.641+0000 7fddadd9b640 1 -- 172.21.15.57:0/464645844 learned_addr learned my addr 172.21.15.57:0/464645844 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:31.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.641+0000 7fddaed9d640 1 --2- 172.21.15.57:0/464645844 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fddb01513e0 0x7fddb01341d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:31.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.641+0000 7fddae59c640 1 --2- 172.21.15.57:0/464645844 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fddb0070350 0x7fddb0135f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:31.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.641+0000 7fddadd9b640 1 -- 172.21.15.57:0/464645844 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fddb01513e0 msgr2=0x7fddb01341d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:31.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.641+0000 7fddadd9b640 1 --2- 172.21.15.57:0/464645844 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fddb01513e0 0x7fddb01341d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:31.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.641+0000 7fddadd9b640 1 -- 172.21.15.57:0/464645844 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fddb0070350 msgr2=0x7fddb0135f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:31.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.641+0000 7fddadd9b640 1 --2- 172.21.15.57:0/464645844 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fddb0070350 0x7fddb0135f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:31.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.641+0000 7fddadd9b640 1 -- 172.21.15.57:0/464645844 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdda0009490 con 0x7fddb00cb980 2023-10-19T12:44:31.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.642+0000 7fddadd9b640 1 --2- 172.21.15.57:0/464645844 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fddb00cb980 0x7fddb00cb3b0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fdda0009780 tx=0x7fdda00038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:31.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.642+0000 7fdda6ffd640 1 -- 172.21.15.57:0/464645844 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdda0003ea0 con 0x7fddb00cb980 2023-10-19T12:44:31.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.642+0000 7fddb4a32640 1 -- 172.21.15.57:0/464645844 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddb0136510 con 0x7fddb00cb980 2023-10-19T12:44:31.650 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.642+0000 7fdda6ffd640 1 -- 172.21.15.57:0/464645844 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fdda0034550 con 0x7fddb00cb980 2023-10-19T12:44:31.650 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.642+0000 7fdda6ffd640 1 -- 172.21.15.57:0/464645844 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdda0034960 con 0x7fddb00cb980 2023-10-19T12:44:31.650 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.642+0000 7fddb4a32640 1 -- 172.21.15.57:0/464645844 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddb00cabb0 con 0x7fddb00cb980 2023-10-19T12:44:31.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.643+0000 7fddb4a32640 1 -- 172.21.15.57:0/464645844 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddb00cb0d0 con 0x7fddb00cb980 2023-10-19T12:44:31.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.643+0000 7fdda6ffd640 1 -- 172.21.15.57:0/464645844 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fdda003b040 con 0x7fddb00cb980 2023-10-19T12:44:31.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.644+0000 7fdda6ffd640 1 --2- 172.21.15.57:0/464645844 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fdd9006ae40 0x7fdd9006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:31.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.644+0000 7fdda6ffd640 1 -- 172.21.15.57:0/464645844 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fdda00abfe0 con 0x7fddb00cb980 2023-10-19T12:44:31.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.644+0000 7fddae59c640 1 --2- 172.21.15.57:0/464645844 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fdd9006ae40 0x7fdd9006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:31.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.644+0000 7fddae59c640 1 --2- 172.21.15.57:0/464645844 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fdd9006ae40 0x7fdd9006d300 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fdd98004ea0 tx=0x7fdd98009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:31.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.648+0000 7fdda6ffd640 1 -- 172.21.15.57:0/464645844 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fddb00cb0d0 con 0x7fddb00cb980 2023-10-19T12:44:31.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.821+0000 7fddb4a32640 1 -- 172.21.15.57:0/464645844 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fddb00cb0d0 con 0x7fddb00cb980 2023-10-19T12:44:31.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.821+0000 7fdda6ffd640 1 -- 172.21.15.57:0/464645844 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) v1 ==== 100+0+11 (secure 0 0 0) 0x7fddb00cb0d0 con 0x7fddb00cb980 2023-10-19T12:44:31.825 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 64 2023-10-19T12:44:31.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.823+0000 7fddb4a32640 1 -- 172.21.15.57:0/464645844 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fdd9006ae40 msgr2=0x7fdd9006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:31.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.823+0000 7fddb4a32640 1 --2- 172.21.15.57:0/464645844 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fdd9006ae40 0x7fdd9006d300 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fdd98004ea0 tx=0x7fdd98009040 comp rx=0 tx=0).stop 2023-10-19T12:44:31.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.823+0000 7fddb4a32640 1 -- 172.21.15.57:0/464645844 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fddb00cb980 msgr2=0x7fddb00cb3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:31.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.823+0000 7fddb4a32640 1 --2- 172.21.15.57:0/464645844 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fddb00cb980 0x7fddb00cb3b0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fdda0009780 tx=0x7fdda00038a0 comp rx=0 tx=0).stop 2023-10-19T12:44:31.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.823+0000 7fddb4a32640 1 -- 172.21.15.57:0/464645844 shutdown_connections 2023-10-19T12:44:31.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.823+0000 7fddb4a32640 1 --2- 172.21.15.57:0/464645844 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fdd9006ae40 0x7fdd9006d300 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:31.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.823+0000 7fddb4a32640 1 --2- 172.21.15.57:0/464645844 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fddb01513e0 0x7fddb01341d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:31.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.823+0000 7fddb4a32640 1 --2- 172.21.15.57:0/464645844 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fddb00cb980 0x7fddb00cb3b0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:31.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.823+0000 7fddb4a32640 1 --2- 172.21.15.57:0/464645844 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fddb0070350 0x7fddb0135f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:31.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.823+0000 7fddb4a32640 1 -- 172.21.15.57:0/464645844 >> 172.21.15.57:0/464645844 conn(0x7fddb0069930 msgr2=0x7fddb0070780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:31.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.823+0000 7fddb4a32640 1 -- 172.21.15.57:0/464645844 shutdown_connections 2023-10-19T12:44:31.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.823+0000 7fddb4a32640 1 -- 172.21.15.57:0/464645844 wait complete. 2023-10-19T12:44:31.836 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-19T12:44:31.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.945+0000 7faaa03b6640 1 -- 172.21.15.57:0/2726178191 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa980cf9e0 msgr2=0x7faa980d1810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:31.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.945+0000 7faaa03b6640 1 --2- 172.21.15.57:0/2726178191 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa980cf9e0 0x7faa980d1810 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7faa8c009830 tx=0x7faa8c02bdf0 comp rx=0 tx=0).stop 2023-10-19T12:44:31.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.945+0000 7faaa03b6640 1 -- 172.21.15.57:0/2726178191 shutdown_connections 2023-10-19T12:44:31.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.945+0000 7faaa03b6640 1 --2- 172.21.15.57:0/2726178191 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa980cf9e0 0x7faa980d1810 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:31.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.945+0000 7faaa03b6640 1 -- 172.21.15.57:0/2726178191 >> 172.21.15.57:0/2726178191 conn(0x7faa980d0cf0 msgr2=0x7faa980d1100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:31.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.945+0000 7faaa03b6640 1 -- 172.21.15.57:0/2726178191 shutdown_connections 2023-10-19T12:44:31.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.945+0000 7faaa03b6640 1 -- 172.21.15.57:0/2726178191 wait complete. 2023-10-19T12:44:31.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.946+0000 7faaa03b6640 1 Processor -- start 2023-10-19T12:44:31.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.946+0000 7faaa03b6640 1 -- start start 2023-10-19T12:44:31.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.947+0000 7faaa03b6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa980cf9e0 0x7faa98153340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:31.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.947+0000 7faaa03b6640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7faa9815b000 0x7faa981e0a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:31.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.947+0000 7faaa03b6640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faa9815b3e0 0x7faa981e0f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:31.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.947+0000 7faaa03b6640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa98153880 con 0x7faa980cf9e0 2023-10-19T12:44:31.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.947+0000 7faaa03b6640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa980c5870 con 0x7faa9815b3e0 2023-10-19T12:44:31.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.947+0000 7faaa03b6640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa981e29a0 con 0x7faa9815b000 2023-10-19T12:44:31.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.947+0000 7faa9e152640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa980cf9e0 0x7faa98153340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:31.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.947+0000 7faa9e152640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa980cf9e0 0x7faa98153340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43488/0 (socket says 172.21.15.57:43488) 2023-10-19T12:44:31.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.947+0000 7faa9e152640 1 -- 172.21.15.57:0/1962957459 learned_addr learned my addr 172.21.15.57:0/1962957459 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:31.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.947+0000 7faa9e953640 1 --2- 172.21.15.57:0/1962957459 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faa9815b3e0 0x7faa981e0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:31.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.947+0000 7faa9d951640 1 --2- 172.21.15.57:0/1962957459 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7faa9815b000 0x7faa981e0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:31.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.947+0000 7faa9e152640 1 -- 172.21.15.57:0/1962957459 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7faa9815b000 msgr2=0x7faa981e0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:31.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.947+0000 7faa9e152640 1 --2- 172.21.15.57:0/1962957459 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7faa9815b000 0x7faa981e0a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:31.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.947+0000 7faa9e152640 1 -- 172.21.15.57:0/1962957459 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faa9815b3e0 msgr2=0x7faa981e0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:31.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.947+0000 7faa9e152640 1 --2- 172.21.15.57:0/1962957459 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faa9815b3e0 0x7faa981e0f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:31.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.947+0000 7faa9e152640 1 -- 172.21.15.57:0/1962957459 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa8c009490 con 0x7faa980cf9e0 2023-10-19T12:44:31.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.947+0000 7faa9e953640 1 --2- 172.21.15.57:0/1962957459 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faa9815b3e0 0x7faa981e0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:44:31.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.947+0000 7faa9e152640 1 --2- 172.21.15.57:0/1962957459 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa980cf9e0 0x7faa98153340 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7faa9000c490 tx=0x7faa9000c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:31.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.948+0000 7faa96ffd640 1 -- 172.21.15.57:0/1962957459 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa90005480 con 0x7faa980cf9e0 2023-10-19T12:44:31.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.948+0000 7faa96ffd640 1 -- 172.21.15.57:0/1962957459 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7faa900035f0 con 0x7faa980cf9e0 2023-10-19T12:44:31.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.948+0000 7faa96ffd640 1 -- 172.21.15.57:0/1962957459 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa90003a00 con 0x7faa980cf9e0 2023-10-19T12:44:31.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.948+0000 7faaa03b6640 1 -- 172.21.15.57:0/1962957459 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa981e2c80 con 0x7faa980cf9e0 2023-10-19T12:44:31.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.948+0000 7faaa03b6640 1 -- 172.21.15.57:0/1962957459 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa981e3240 con 0x7faa980cf9e0 2023-10-19T12:44:31.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.949+0000 7faaa03b6640 1 -- 172.21.15.57:0/1962957459 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa980690e0 con 0x7faa980cf9e0 2023-10-19T12:44:31.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.949+0000 7faa96ffd640 1 -- 172.21.15.57:0/1962957459 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7faa90003b60 con 0x7faa980cf9e0 2023-10-19T12:44:31.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.950+0000 7faa96ffd640 1 --2- 172.21.15.57:0/1962957459 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7faa8406ae40 0x7faa8406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:31.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.950+0000 7faa96ffd640 1 -- 172.21.15.57:0/1962957459 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7faa90089560 con 0x7faa980cf9e0 2023-10-19T12:44:31.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.950+0000 7faa9d951640 1 --2- 172.21.15.57:0/1962957459 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7faa8406ae40 0x7faa8406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:31.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.950+0000 7faa9d951640 1 --2- 172.21.15.57:0/1962957459 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7faa8406ae40 0x7faa8406d300 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7faa8c036000 tx=0x7faa8c004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:31.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:31.954+0000 7faa96ffd640 1 -- 172.21.15.57:0/1962957459 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7faa900543b0 con 0x7faa980cf9e0 2023-10-19T12:44:32.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:32.129+0000 7faaa03b6640 1 -- 172.21.15.57:0/1962957459 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7faa980d1810 con 0x7faa980cf9e0 2023-10-19T12:44:32.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:32.129+0000 7faa96ffd640 1 -- 172.21.15.57:0/1962957459 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) v1 ==== 103+0+11 (secure 0 0 0) 0x7faa900579d0 con 0x7faa980cf9e0 2023-10-19T12:44:32.132 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2023-10-19T12:44:32.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:32.131+0000 7faaa03b6640 1 -- 172.21.15.57:0/1962957459 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7faa8406ae40 msgr2=0x7faa8406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:32.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:32.131+0000 7faaa03b6640 1 --2- 172.21.15.57:0/1962957459 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7faa8406ae40 0x7faa8406d300 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7faa8c036000 tx=0x7faa8c004730 comp rx=0 tx=0).stop 2023-10-19T12:44:32.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:32.131+0000 7faaa03b6640 1 -- 172.21.15.57:0/1962957459 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa980cf9e0 msgr2=0x7faa98153340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:32.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:32.131+0000 7faaa03b6640 1 --2- 172.21.15.57:0/1962957459 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa980cf9e0 0x7faa98153340 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7faa9000c490 tx=0x7faa9000c950 comp rx=0 tx=0).stop 2023-10-19T12:44:32.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:32.131+0000 7faaa03b6640 1 -- 172.21.15.57:0/1962957459 shutdown_connections 2023-10-19T12:44:32.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:32.131+0000 7faaa03b6640 1 --2- 172.21.15.57:0/1962957459 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7faa8406ae40 0x7faa8406d300 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:32.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:32.131+0000 7faaa03b6640 1 --2- 172.21.15.57:0/1962957459 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faa9815b3e0 0x7faa981e0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:32.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:32.131+0000 7faaa03b6640 1 --2- 172.21.15.57:0/1962957459 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7faa9815b000 0x7faa981e0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:32.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:32.131+0000 7faaa03b6640 1 --2- 172.21.15.57:0/1962957459 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa980cf9e0 0x7faa98153340 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:32.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:32.131+0000 7faaa03b6640 1 -- 172.21.15.57:0/1962957459 >> 172.21.15.57:0/1962957459 conn(0x7faa980d0cf0 msgr2=0x7faa98153f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:32.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:32.131+0000 7faaa03b6640 1 -- 172.21.15.57:0/1962957459 shutdown_connections 2023-10-19T12:44:32.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:32.131+0000 7faaa03b6640 1 -- 172.21.15.57:0/1962957459 wait complete. 2023-10-19T12:44:32.145 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-19T12:44:32.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:32.252+0000 7f594da32640 1 -- 172.21.15.57:0/991200396 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f594806f0a0 msgr2=0x7f594806dbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:32.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:32.252+0000 7f594da32640 1 --2- 172.21.15.57:0/991200396 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f594806f0a0 0x7f594806dbf0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f5948066970 tx=0x7f593c02be40 comp rx=0 tx=0).stop 2023-10-19T12:44:32.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:32.252+0000 7f594da32640 1 -- 172.21.15.57:0/991200396 shutdown_connections 2023-10-19T12:44:32.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:32.252+0000 7f594da32640 1 --2- 172.21.15.57:0/991200396 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f594806f0a0 0x7f594806dbf0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:32.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:32.252+0000 7f594da32640 1 --2- 172.21.15.57:0/991200396 >> v2:172.21.15.179:3300/0 conn(0x7f594806d2d0 0x7f594806d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:32.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:32.252+0000 7f594da32640 1 -- 172.21.15.57:0/991200396 >> 172.21.15.57:0/991200396 conn(0x7f59480b0670 msgr2=0x7f59480b0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:32.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:32.252+0000 7f594da32640 1 -- 172.21.15.57:0/991200396 shutdown_connections 2023-10-19T12:44:32.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:32.252+0000 7f594da32640 1 -- 172.21.15.57:0/991200396 wait complete. 2023-10-19T12:44:32.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:32.253+0000 7f594da32640 1 Processor -- start 2023-10-19T12:44:32.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:32.254+0000 7f594da32640 1 -- start start 2023-10-19T12:44:32.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:32.254+0000 7f594da32640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f594806d2d0 0x7f59480cbeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:32.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:32.254+0000 7f594da32640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f594806f0a0 0x7f59481ee9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:32.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:32.254+0000 7f594da32640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f59480ab950 0x7f59481eef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:32.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:32.254+0000 7f594da32640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59480c9ab0 con 0x7f594806d2d0 2023-10-19T12:44:32.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:32.254+0000 7f594da32640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59480b4470 con 0x7f594806f0a0 2023-10-19T12:44:32.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:32.254+0000 7f594da32640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59481ef460 con 0x7f59480ab950 2023-10-19T12:44:32.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:32.254+0000 7f5946ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f594806d2d0 0x7f59480cbeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:32.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:32.254+0000 7f5946ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f594806d2d0 0x7f59480cbeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43500/0 (socket says 172.21.15.57:43500) 2023-10-19T12:44:32.262 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:32.255+0000 7f5946ffd640 1 -- 172.21.15.57:0/98338842 learned_addr learned my addr 172.21.15.57:0/98338842 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:32.262 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:32.255+0000 7f5946ffd640 1 -- 172.21.15.57:0/98338842 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f59480ab950 msgr2=0x7f59481eef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:32.262 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:32.255+0000 7f59467fc640 1 --2- 172.21.15.57:0/98338842 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f594806f0a0 0x7f59481ee9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:32.262 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:32.255+0000 7f59477fe640 1 --2- 172.21.15.57:0/98338842 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f59480ab950 0x7f59481eef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:32.262 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:32.255+0000 7f5946ffd640 1 --2- 172.21.15.57:0/98338842 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f59480ab950 0x7f59481eef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:32.263 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:32.255+0000 7f5946ffd640 1 -- 172.21.15.57:0/98338842 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f594806f0a0 msgr2=0x7f59481ee9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:32.263 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:32.255+0000 7f5946ffd640 1 --2- 172.21.15.57:0/98338842 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f594806f0a0 0x7f59481ee9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:32.263 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:32.255+0000 7f5946ffd640 1 -- 172.21.15.57:0/98338842 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f593c009560 con 0x7f594806d2d0 2023-10-19T12:44:32.263 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:32.255+0000 7f5946ffd640 1 --2- 172.21.15.57:0/98338842 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f594806d2d0 0x7f59480cbeb0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f594000eab0 tx=0x7f594000ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:32.263 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:32.255+0000 7f592ffff640 1 -- 172.21.15.57:0/98338842 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f594000ce40 con 0x7f594806d2d0 2023-10-19T12:44:32.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:32.255+0000 7f592ffff640 1 -- 172.21.15.57:0/98338842 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f5940004c10 con 0x7f594806d2d0 2023-10-19T12:44:32.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:32.255+0000 7f594da32640 1 -- 172.21.15.57:0/98338842 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59480b3eb0 con 0x7f594806d2d0 2023-10-19T12:44:32.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:32.255+0000 7f592ffff640 1 -- 172.21.15.57:0/98338842 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5940005020 con 0x7f594806d2d0 2023-10-19T12:44:32.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:32.255+0000 7f594da32640 1 -- 172.21.15.57:0/98338842 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59480c96d0 con 0x7f594806d2d0 2023-10-19T12:44:32.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:32.256+0000 7f594da32640 1 -- 172.21.15.57:0/98338842 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59480ce1d0 con 0x7f594806d2d0 2023-10-19T12:44:32.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:32.257+0000 7f592ffff640 1 -- 172.21.15.57:0/98338842 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f5940005180 con 0x7f594806d2d0 2023-10-19T12:44:32.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:32.257+0000 7f592ffff640 1 --2- 172.21.15.57:0/98338842 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f593406ae40 0x7f593406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:32.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:32.257+0000 7f592ffff640 1 -- 172.21.15.57:0/98338842 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f594008a2f0 con 0x7f594806d2d0 2023-10-19T12:44:32.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:32.257+0000 7f59467fc640 1 --2- 172.21.15.57:0/98338842 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f593406ae40 0x7f593406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:32.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:32.258+0000 7f59467fc640 1 --2- 172.21.15.57:0/98338842 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f593406ae40 0x7f593406d300 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f593c035000 tx=0x7f593c004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:32.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:32.261+0000 7f592ffff640 1 -- 172.21.15.57:0/98338842 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f5940055140 con 0x7f594806d2d0 2023-10-19T12:44:32.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:32.450+0000 7f594da32640 1 -- 172.21.15.57:0/98338842 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f59480690e0 con 0x7f594806d2d0 2023-10-19T12:44:32.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:32.451+0000 7f592ffff640 1 -- 172.21.15.57:0/98338842 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v25) v1 ==== 80+0+0 (secure 0 0 0) 0x7f5940058760 con 0x7f594806d2d0 2023-10-19T12:44:32.454 INFO:teuthology.orchestra.run.smithi057.stderr:listed 0 entries 2023-10-19T12:44:32.456 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:32.453+0000 7f594da32640 1 -- 172.21.15.57:0/98338842 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f593406ae40 msgr2=0x7f593406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:32.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:32.453+0000 7f594da32640 1 --2- 172.21.15.57:0/98338842 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f593406ae40 0x7f593406d300 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f593c035000 tx=0x7f593c004730 comp rx=0 tx=0).stop 2023-10-19T12:44:32.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:32.453+0000 7f594da32640 1 -- 172.21.15.57:0/98338842 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f594806d2d0 msgr2=0x7f59480cbeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:32.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:32.453+0000 7f594da32640 1 --2- 172.21.15.57:0/98338842 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f594806d2d0 0x7f59480cbeb0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f594000eab0 tx=0x7f594000ef70 comp rx=0 tx=0).stop 2023-10-19T12:44:32.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:32.453+0000 7f594da32640 1 -- 172.21.15.57:0/98338842 shutdown_connections 2023-10-19T12:44:32.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:32.453+0000 7f594da32640 1 --2- 172.21.15.57:0/98338842 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f593406ae40 0x7f593406d300 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:32.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:32.453+0000 7f594da32640 1 --2- 172.21.15.57:0/98338842 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f59480ab950 0x7f59481eef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:32.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:32.453+0000 7f594da32640 1 --2- 172.21.15.57:0/98338842 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f594806f0a0 0x7f59481ee9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:32.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:32.453+0000 7f594da32640 1 --2- 172.21.15.57:0/98338842 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f594806d2d0 0x7f59480cbeb0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:32.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:32.453+0000 7f594da32640 1 -- 172.21.15.57:0/98338842 >> 172.21.15.57:0/98338842 conn(0x7f59480b0670 msgr2=0x7f5948159d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:32.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:32.453+0000 7f594da32640 1 -- 172.21.15.57:0/98338842 shutdown_connections 2023-10-19T12:44:32.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:32.453+0000 7f594da32640 1 -- 172.21.15.57:0/98338842 wait complete. 2023-10-19T12:44:32.467 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi057.front.sepia.ceph.com... 2023-10-19T12:44:32.467 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:44:32.467 DEBUG:teuthology.orchestra.run.smithi057:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2023-10-19T12:44:32.496 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi057.stderr:ceph-fuse[95612]: fuse finished with error 0 and tester_r 0 2023-10-19T12:44:32.498 INFO:teuthology.orchestra.run:waiting for 300 2023-10-19T12:44:38.599 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi057.front.sepia.ceph.com 2023-10-19T12:44:38.600 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:44:38.600 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2023-10-19T12:44:38.633 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T12:44:38.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:38.791+0000 7f51cf490640 1 -- 172.21.15.57:0/2236942910 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f51c80cf340 msgr2=0x7f51c80d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:38.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:38.791+0000 7f51cf490640 1 --2- 172.21.15.57:0/2236942910 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f51c80cf340 0x7f51c80d1910 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f51c0009830 tx=0x7f51c002bd60 comp rx=0 tx=0).stop 2023-10-19T12:44:38.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:38.791+0000 7f51cf490640 1 -- 172.21.15.57:0/2236942910 shutdown_connections 2023-10-19T12:44:38.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:38.791+0000 7f51cf490640 1 --2- 172.21.15.57:0/2236942910 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f51c80cf340 0x7f51c80d1910 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:38.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:38.791+0000 7f51cf490640 1 -- 172.21.15.57:0/2236942910 >> 172.21.15.57:0/2236942910 conn(0x7f51c80d0d60 msgr2=0x7f51c80d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:38.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:38.791+0000 7f51cf490640 1 -- 172.21.15.57:0/2236942910 shutdown_connections 2023-10-19T12:44:38.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:38.791+0000 7f51cf490640 1 -- 172.21.15.57:0/2236942910 wait complete. 2023-10-19T12:44:38.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:38.792+0000 7f51cf490640 1 Processor -- start 2023-10-19T12:44:38.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:38.792+0000 7f51cf490640 1 -- start start 2023-10-19T12:44:38.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:38.793+0000 7f51cf490640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51c80ceb40 0x7f51c81578b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:38.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:38.793+0000 7f51cf490640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f51c80cf340 0x7f51c80c93c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:38.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:38.793+0000 7f51cf490640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f51c815b2a0 0x7f51c80b0ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:38.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:38.793+0000 7f51cf490640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51c80ca990 con 0x7f51c80ceb40 2023-10-19T12:44:38.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:38.793+0000 7f51cf490640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51c80c9d80 con 0x7f51c815b2a0 2023-10-19T12:44:38.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:38.793+0000 7f51cf490640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51c80ca060 con 0x7f51c80cf340 2023-10-19T12:44:38.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:38.793+0000 7f51cd22c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51c80ceb40 0x7f51c81578b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:38.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:38.793+0000 7f51cd22c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51c80ceb40 0x7f51c81578b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50378/0 (socket says 172.21.15.57:50378) 2023-10-19T12:44:38.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:38.793+0000 7f51cd22c640 1 -- 172.21.15.57:0/1950699806 learned_addr learned my addr 172.21.15.57:0/1950699806 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:38.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:38.793+0000 7f51cca2b640 1 --2- 172.21.15.57:0/1950699806 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f51c80cf340 0x7f51c80c93c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:38.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:38.793+0000 7f51cda2d640 1 --2- 172.21.15.57:0/1950699806 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f51c815b2a0 0x7f51c80b0ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:38.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:38.793+0000 7f51cd22c640 1 -- 172.21.15.57:0/1950699806 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f51c80cf340 msgr2=0x7f51c80c93c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:38.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:38.793+0000 7f51cd22c640 1 --2- 172.21.15.57:0/1950699806 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f51c80cf340 0x7f51c80c93c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:38.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:38.793+0000 7f51cd22c640 1 -- 172.21.15.57:0/1950699806 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f51c815b2a0 msgr2=0x7f51c80b0ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:38.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:38.793+0000 7f51cd22c640 1 --2- 172.21.15.57:0/1950699806 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f51c815b2a0 0x7f51c80b0ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:38.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:38.793+0000 7f51cd22c640 1 -- 172.21.15.57:0/1950699806 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51c0009490 con 0x7f51c80ceb40 2023-10-19T12:44:38.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:38.793+0000 7f51cda2d640 1 --2- 172.21.15.57:0/1950699806 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f51c815b2a0 0x7f51c80b0ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:44:38.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:38.794+0000 7f51cd22c640 1 --2- 172.21.15.57:0/1950699806 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51c80ceb40 0x7f51c81578b0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f51c0009780 tx=0x7f51c00038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:38.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:38.794+0000 7f51c5ffb640 1 -- 172.21.15.57:0/1950699806 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51c0003ea0 con 0x7f51c80ceb40 2023-10-19T12:44:38.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:38.794+0000 7f51c5ffb640 1 -- 172.21.15.57:0/1950699806 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f51c0034440 con 0x7f51c80ceb40 2023-10-19T12:44:38.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:38.794+0000 7f51cf490640 1 -- 172.21.15.57:0/1950699806 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51c80ca340 con 0x7f51c80ceb40 2023-10-19T12:44:38.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:38.794+0000 7f51c5ffb640 1 -- 172.21.15.57:0/1950699806 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51c0034850 con 0x7f51c80ceb40 2023-10-19T12:44:38.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:38.794+0000 7f51cf490640 1 -- 172.21.15.57:0/1950699806 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51c8157df0 con 0x7f51c80ceb40 2023-10-19T12:44:38.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:38.795+0000 7f51cf490640 1 -- 172.21.15.57:0/1950699806 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51c80690e0 con 0x7f51c80ceb40 2023-10-19T12:44:38.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:38.796+0000 7f51c5ffb640 1 -- 172.21.15.57:0/1950699806 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f51c003b040 con 0x7f51c80ceb40 2023-10-19T12:44:38.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:38.796+0000 7f51c5ffb640 1 --2- 172.21.15.57:0/1950699806 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f51b406ae40 0x7f51b406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:38.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:38.796+0000 7f51c5ffb640 1 -- 172.21.15.57:0/1950699806 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f51c00abc40 con 0x7f51c80ceb40 2023-10-19T12:44:38.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:38.796+0000 7f51cca2b640 1 --2- 172.21.15.57:0/1950699806 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f51b406ae40 0x7f51b406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:38.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:38.797+0000 7f51cca2b640 1 --2- 172.21.15.57:0/1950699806 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f51b406ae40 0x7f51b406d300 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f51c80b6dc0 tx=0x7f51bc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:38.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:38.800+0000 7f51c5ffb640 1 -- 172.21.15.57:0/1950699806 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f51c0076a10 con 0x7f51c80ceb40 2023-10-19T12:44:38.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:38.982+0000 7f51cf490640 1 -- 172.21.15.57:0/1950699806 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f51c80d0b40 con 0x7f51c80ceb40 2023-10-19T12:44:38.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:38.983+0000 7f51c5ffb640 1 -- 172.21.15.57:0/1950699806 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18358 (secure 0 0 0) 0x7f51c007a030 con 0x7f51c80ceb40 2023-10-19T12:44:38.987 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:44:38.987 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":25,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T12:44:23.472399+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-19T12:44:16.517510+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-19T12:44:17.398297+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-19T12:44:21.129403+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T12:44:38.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:38.985+0000 7f51cf490640 1 -- 172.21.15.57:0/1950699806 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f51b406ae40 msgr2=0x7f51b406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:38.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:38.985+0000 7f51cf490640 1 --2- 172.21.15.57:0/1950699806 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f51b406ae40 0x7f51b406d300 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f51c80b6dc0 tx=0x7f51bc009040 comp rx=0 tx=0).stop 2023-10-19T12:44:38.989 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:38.985+0000 7f51cf490640 1 -- 172.21.15.57:0/1950699806 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51c80ceb40 msgr2=0x7f51c81578b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:38.989 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:38.985+0000 7f51cf490640 1 --2- 172.21.15.57:0/1950699806 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51c80ceb40 0x7f51c81578b0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f51c0009780 tx=0x7f51c00038a0 comp rx=0 tx=0).stop 2023-10-19T12:44:38.989 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:38.985+0000 7f51cf490640 1 -- 172.21.15.57:0/1950699806 shutdown_connections 2023-10-19T12:44:38.989 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:38.985+0000 7f51cf490640 1 --2- 172.21.15.57:0/1950699806 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f51b406ae40 0x7f51b406d300 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:38.989 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:38.985+0000 7f51cf490640 1 --2- 172.21.15.57:0/1950699806 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f51c815b2a0 0x7f51c80b0ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:38.990 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:38.985+0000 7f51cf490640 1 --2- 172.21.15.57:0/1950699806 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f51c80cf340 0x7f51c80c93c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:38.990 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:38.985+0000 7f51cf490640 1 --2- 172.21.15.57:0/1950699806 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51c80ceb40 0x7f51c81578b0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:38.990 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:38.985+0000 7f51cf490640 1 -- 172.21.15.57:0/1950699806 >> 172.21.15.57:0/1950699806 conn(0x7f51c80d0d60 msgr2=0x7f51c806b600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:38.990 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:38.985+0000 7f51cf490640 1 -- 172.21.15.57:0/1950699806 shutdown_connections 2023-10-19T12:44:38.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:38.985+0000 7f51cf490640 1 -- 172.21.15.57:0/1950699806 wait complete. 2023-10-19T12:44:38.999 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T12:44:39.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.105+0000 7f9895b7d640 1 -- 172.21.15.57:0/2583663060 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f989006d2d0 msgr2=0x7f989006d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:39.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.105+0000 7f9895b7d640 1 --2- 172.21.15.57:0/2583663060 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f989006d2d0 0x7f989006d6b0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f9880005fd0 tx=0x7f988002bed0 comp rx=0 tx=0).stop 2023-10-19T12:44:39.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.105+0000 7f9895b7d640 1 -- 172.21.15.57:0/2583663060 shutdown_connections 2023-10-19T12:44:39.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.105+0000 7f9895b7d640 1 --2- 172.21.15.57:0/2583663060 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f989006d2d0 0x7f989006d6b0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:39.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.105+0000 7f9895b7d640 1 -- 172.21.15.57:0/2583663060 >> 172.21.15.57:0/2583663060 conn(0x7f98900b0670 msgr2=0x7f98900b0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:39.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.105+0000 7f9895b7d640 1 -- 172.21.15.57:0/2583663060 shutdown_connections 2023-10-19T12:44:39.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.105+0000 7f9895b7d640 1 -- 172.21.15.57:0/2583663060 wait complete. 2023-10-19T12:44:39.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.106+0000 7f9895b7d640 1 Processor -- start 2023-10-19T12:44:39.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.106+0000 7f9895b7d640 1 -- start start 2023-10-19T12:44:39.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.107+0000 7f9895b7d640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f989006d2d0 0x7f9890158870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:39.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.107+0000 7f9895b7d640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f989006f0a0 0x7f98900c9de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:39.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.107+0000 7f9895b7d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f98900ad370 0x7f98900b3910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:39.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.107+0000 7f9895b7d640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f989014e8b0 con 0x7f98900ad370 2023-10-19T12:44:39.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.107+0000 7f9895b7d640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98901583e0 con 0x7f989006f0a0 2023-10-19T12:44:39.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.107+0000 7f9895b7d640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9890158db0 con 0x7f989006d2d0 2023-10-19T12:44:39.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.107+0000 7f988ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f98900ad370 0x7f98900b3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:39.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.107+0000 7f988ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f98900ad370 0x7f98900b3910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50394/0 (socket says 172.21.15.57:50394) 2023-10-19T12:44:39.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.107+0000 7f988ffff640 1 -- 172.21.15.57:0/3447716733 learned_addr learned my addr 172.21.15.57:0/3447716733 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:39.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.107+0000 7f988effd640 1 --2- 172.21.15.57:0/3447716733 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f989006f0a0 0x7f98900c9de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:39.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.107+0000 7f988f7fe640 1 --2- 172.21.15.57:0/3447716733 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f989006d2d0 0x7f9890158870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:39.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.107+0000 7f988ffff640 1 -- 172.21.15.57:0/3447716733 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f989006d2d0 msgr2=0x7f9890158870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:39.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.107+0000 7f988ffff640 1 --2- 172.21.15.57:0/3447716733 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f989006d2d0 0x7f9890158870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:39.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.107+0000 7f988ffff640 1 -- 172.21.15.57:0/3447716733 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f989006f0a0 msgr2=0x7f98900c9de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:39.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.107+0000 7f988ffff640 1 --2- 172.21.15.57:0/3447716733 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f989006f0a0 0x7f98900c9de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:39.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.107+0000 7f988ffff640 1 -- 172.21.15.57:0/3447716733 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9880009560 con 0x7f98900ad370 2023-10-19T12:44:39.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.107+0000 7f988effd640 1 --2- 172.21.15.57:0/3447716733 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f989006f0a0 0x7f98900c9de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:44:39.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.107+0000 7f988f7fe640 1 --2- 172.21.15.57:0/3447716733 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f989006d2d0 0x7f9890158870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:44:39.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.108+0000 7f988ffff640 1 --2- 172.21.15.57:0/3447716733 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f98900ad370 0x7f98900b3910 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f988800c6b0 tx=0x7f988800cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:39.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.108+0000 7f9877fff640 1 -- 172.21.15.57:0/3447716733 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9888003b70 con 0x7f98900ad370 2023-10-19T12:44:39.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.108+0000 7f9895b7d640 1 -- 172.21.15.57:0/3447716733 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9890157830 con 0x7f98900ad370 2023-10-19T12:44:39.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.108+0000 7f9877fff640 1 -- 172.21.15.57:0/3447716733 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f9888003cf0 con 0x7f98900ad370 2023-10-19T12:44:39.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.108+0000 7f9877fff640 1 -- 172.21.15.57:0/3447716733 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9888003ea0 con 0x7f98900ad370 2023-10-19T12:44:39.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.108+0000 7f9895b7d640 1 -- 172.21.15.57:0/3447716733 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9890158070 con 0x7f98900ad370 2023-10-19T12:44:39.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.109+0000 7f9895b7d640 1 -- 172.21.15.57:0/3447716733 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98900ce1d0 con 0x7f98900ad370 2023-10-19T12:44:39.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.109+0000 7f9877fff640 1 -- 172.21.15.57:0/3447716733 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f9888005330 con 0x7f98900ad370 2023-10-19T12:44:39.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.110+0000 7f9877fff640 1 --2- 172.21.15.57:0/3447716733 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f987c06ae40 0x7f987c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:39.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.110+0000 7f9877fff640 1 -- 172.21.15.57:0/3447716733 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f9888089810 con 0x7f98900ad370 2023-10-19T12:44:39.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.110+0000 7f988f7fe640 1 --2- 172.21.15.57:0/3447716733 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f987c06ae40 0x7f987c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:39.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.110+0000 7f988f7fe640 1 --2- 172.21.15.57:0/3447716733 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f987c06ae40 0x7f987c06d300 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f9880035000 tx=0x7f988002e040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:39.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.114+0000 7f9877fff640 1 -- 172.21.15.57:0/3447716733 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f98880545e0 con 0x7f98900ad370 2023-10-19T12:44:39.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.293+0000 7f9895b7d640 1 -- 172.21.15.57:0/3447716733 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f98900690e0 con 0x7f98900ad370 2023-10-19T12:44:39.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.294+0000 7f9877fff640 1 -- 172.21.15.57:0/3447716733 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18358 (secure 0 0 0) 0x7f9888057c00 con 0x7f98900ad370 2023-10-19T12:44:39.297 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:44:39.298 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":25,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T12:44:23.472399+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-19T12:44:16.517510+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-19T12:44:17.398297+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-19T12:44:21.129403+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T12:44:39.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.296+0000 7f9895b7d640 1 -- 172.21.15.57:0/3447716733 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f987c06ae40 msgr2=0x7f987c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:39.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.296+0000 7f9895b7d640 1 --2- 172.21.15.57:0/3447716733 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f987c06ae40 0x7f987c06d300 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f9880035000 tx=0x7f988002e040 comp rx=0 tx=0).stop 2023-10-19T12:44:39.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.296+0000 7f9895b7d640 1 -- 172.21.15.57:0/3447716733 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f98900ad370 msgr2=0x7f98900b3910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:39.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.296+0000 7f9895b7d640 1 --2- 172.21.15.57:0/3447716733 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f98900ad370 0x7f98900b3910 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f988800c6b0 tx=0x7f988800cb70 comp rx=0 tx=0).stop 2023-10-19T12:44:39.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.296+0000 7f9895b7d640 1 -- 172.21.15.57:0/3447716733 shutdown_connections 2023-10-19T12:44:39.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.296+0000 7f9895b7d640 1 --2- 172.21.15.57:0/3447716733 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f987c06ae40 0x7f987c06d300 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:39.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.296+0000 7f9895b7d640 1 --2- 172.21.15.57:0/3447716733 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f98900ad370 0x7f98900b3910 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:39.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.296+0000 7f9895b7d640 1 --2- 172.21.15.57:0/3447716733 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f989006f0a0 0x7f98900c9de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:39.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.296+0000 7f9895b7d640 1 --2- 172.21.15.57:0/3447716733 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f989006d2d0 0x7f9890158870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:39.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.296+0000 7f9895b7d640 1 -- 172.21.15.57:0/3447716733 >> 172.21.15.57:0/3447716733 conn(0x7f98900b0670 msgr2=0x7f98901500d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:39.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.296+0000 7f9895b7d640 1 -- 172.21.15.57:0/3447716733 shutdown_connections 2023-10-19T12:44:39.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.296+0000 7f9895b7d640 1 -- 172.21.15.57:0/3447716733 wait complete. 2023-10-19T12:44:39.310 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-19T12:44:39.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.419+0000 7f044e021640 1 -- 172.21.15.57:0/2865659324 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0430003f20 msgr2=0x7f04300047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:39.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.419+0000 7f044e021640 1 --2- 172.21.15.57:0/2865659324 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0430003f20 0x7f04300047c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f043c009b50 tx=0x7f04300057f0 comp rx=0 tx=0).stop 2023-10-19T12:44:39.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.419+0000 7f0447fff640 1 -- 172.21.15.57:0/2865659324 reap_dead start 2023-10-19T12:44:39.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.419+0000 7f044e021640 1 -- 172.21.15.57:0/2865659324 shutdown_connections 2023-10-19T12:44:39.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.419+0000 7f044e021640 1 -- 172.21.15.57:0/2865659324 >> 172.21.15.57:0/2865659324 conn(0x7f04480b3f00 msgr2=0x7f04480b4310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:39.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.419+0000 7f044e021640 1 -- 172.21.15.57:0/2865659324 shutdown_connections 2023-10-19T12:44:39.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.419+0000 7f044e021640 1 -- 172.21.15.57:0/2865659324 wait complete. 2023-10-19T12:44:39.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.420+0000 7f044e021640 1 Processor -- start 2023-10-19T12:44:39.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.420+0000 7f044e021640 1 -- start start 2023-10-19T12:44:39.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.421+0000 7f044e021640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0430003f20 0x7f044814a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:39.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.421+0000 7f044e021640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f04481e14d0 0x7f04481e1950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:39.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.421+0000 7f044e021640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f04481482a0 0x7f0448148720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:39.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.421+0000 7f044e021640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04480af980 con 0x7f04481e14d0 2023-10-19T12:44:39.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.421+0000 7f044e021640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04480af370 con 0x7f0430003f20 2023-10-19T12:44:39.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.421+0000 7f044e021640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04481e2510 con 0x7f04481482a0 2023-10-19T12:44:39.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.421+0000 7f0446ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f04481e14d0 0x7f04481e1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:39.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.421+0000 7f0446ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f04481e14d0 0x7f04481e1950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50408/0 (socket says 172.21.15.57:50408) 2023-10-19T12:44:39.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.421+0000 7f0446ffd640 1 -- 172.21.15.57:0/2187636248 learned_addr learned my addr 172.21.15.57:0/2187636248 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:39.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.421+0000 7f0447fff640 1 --2- 172.21.15.57:0/2187636248 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f04481482a0 0x7f0448148720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:39.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.421+0000 7f04477fe640 1 --2- 172.21.15.57:0/2187636248 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0430003f20 0x7f044814a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:39.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.421+0000 7f0446ffd640 1 -- 172.21.15.57:0/2187636248 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f04481482a0 msgr2=0x7f0448148720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:39.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.421+0000 7f0446ffd640 1 --2- 172.21.15.57:0/2187636248 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f04481482a0 0x7f0448148720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:39.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.421+0000 7f0446ffd640 1 -- 172.21.15.57:0/2187636248 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0430003f20 msgr2=0x7f044814a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:39.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.421+0000 7f0446ffd640 1 --2- 172.21.15.57:0/2187636248 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0430003f20 0x7f044814a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:39.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.421+0000 7f0446ffd640 1 -- 172.21.15.57:0/2187636248 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0448148c60 con 0x7f04481e14d0 2023-10-19T12:44:39.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.421+0000 7f0446ffd640 1 --2- 172.21.15.57:0/2187636248 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f04481e14d0 0x7f04481e1950 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f043c02cc40 tx=0x7f043c033c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:39.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.421+0000 7f042ffff640 1 -- 172.21.15.57:0/2187636248 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f043c03d8e0 con 0x7f04481e14d0 2023-10-19T12:44:39.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.422+0000 7f042ffff640 1 -- 172.21.15.57:0/2187636248 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f043c03da40 con 0x7f04481e14d0 2023-10-19T12:44:39.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.422+0000 7f044e021640 1 -- 172.21.15.57:0/2187636248 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04481e1f50 con 0x7f04481e14d0 2023-10-19T12:44:39.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.422+0000 7f042ffff640 1 -- 172.21.15.57:0/2187636248 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f043c03de30 con 0x7f04481e14d0 2023-10-19T12:44:39.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.422+0000 7f044e021640 1 -- 172.21.15.57:0/2187636248 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04480af5a0 con 0x7f04481e14d0 2023-10-19T12:44:39.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.423+0000 7f044e021640 1 -- 172.21.15.57:0/2187636248 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04480afbf0 con 0x7f04481e14d0 2023-10-19T12:44:39.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.423+0000 7f042ffff640 1 -- 172.21.15.57:0/2187636248 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f043c04e3a0 con 0x7f04481e14d0 2023-10-19T12:44:39.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.423+0000 7f042ffff640 1 --2- 172.21.15.57:0/2187636248 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f043406ae40 0x7f043406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:39.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.423+0000 7f042ffff640 1 -- 172.21.15.57:0/2187636248 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f043c033e40 con 0x7f04481e14d0 2023-10-19T12:44:39.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.424+0000 7f04477fe640 1 --2- 172.21.15.57:0/2187636248 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f043406ae40 0x7f043406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:39.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.424+0000 7f04477fe640 1 --2- 172.21.15.57:0/2187636248 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f043406ae40 0x7f043406d300 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0438001050 tx=0x7f0438000f80 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:39.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.428+0000 7f042ffff640 1 -- 172.21.15.57:0/2187636248 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f043c077750 con 0x7f04481e14d0 2023-10-19T12:44:39.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:39.642+0000 7f044e021640 1 -- 172.21.15.57:0/2187636248 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f04480690e0 con 0x7f04481e14d0 2023-10-19T12:44:40.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:40.191+0000 7f042ffff640 1 -- 172.21.15.57:0/2187636248 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v8) v1 ==== 118+0+0 (secure 0 0 0) 0x7f043c07ad70 con 0x7f04481e14d0 2023-10-19T12:44:40.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:40.193+0000 7f044e021640 1 -- 172.21.15.57:0/2187636248 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f043406ae40 msgr2=0x7f043406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:40.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:40.193+0000 7f044e021640 1 --2- 172.21.15.57:0/2187636248 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f043406ae40 0x7f043406d300 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0438001050 tx=0x7f0438000f80 comp rx=0 tx=0).stop 2023-10-19T12:44:40.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:40.194+0000 7f044e021640 1 -- 172.21.15.57:0/2187636248 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f04481e14d0 msgr2=0x7f04481e1950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:40.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:40.194+0000 7f044e021640 1 --2- 172.21.15.57:0/2187636248 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f04481e14d0 0x7f04481e1950 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f043c02cc40 tx=0x7f043c033c50 comp rx=0 tx=0).stop 2023-10-19T12:44:40.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:40.194+0000 7f044e021640 1 -- 172.21.15.57:0/2187636248 shutdown_connections 2023-10-19T12:44:40.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:40.194+0000 7f044e021640 1 --2- 172.21.15.57:0/2187636248 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f043406ae40 0x7f043406d300 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:40.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:40.194+0000 7f044e021640 1 --2- 172.21.15.57:0/2187636248 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f04481482a0 0x7f0448148720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:40.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:40.194+0000 7f044e021640 1 --2- 172.21.15.57:0/2187636248 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f04481e14d0 0x7f04481e1950 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:40.201 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:40.194+0000 7f044e021640 1 --2- 172.21.15.57:0/2187636248 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0430003f20 0x7f044814a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:40.201 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:40.194+0000 7f044e021640 1 -- 172.21.15.57:0/2187636248 >> 172.21.15.57:0/2187636248 conn(0x7f04480b3f00 msgr2=0x7f044814c920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:40.201 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:40.194+0000 7f044e021640 1 -- 172.21.15.57:0/2187636248 shutdown_connections 2023-10-19T12:44:40.202 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:40.194+0000 7f044e021640 1 -- 172.21.15.57:0/2187636248 wait complete. 2023-10-19T12:44:40.210 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']} 2023-10-19T12:44:40.210 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-19T12:44:40.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:40.325+0000 7f7730983640 1 -- 172.21.15.57:0/3188835175 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f772c13d350 msgr2=0x7f772c0cbd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:40.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:40.325+0000 7f7730983640 1 --2- 172.21.15.57:0/3188835175 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f772c13d350 0x7f772c0cbd60 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f7714009830 tx=0x7f771402bd60 comp rx=0 tx=0).stop 2023-10-19T12:44:40.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:40.325+0000 7f7730983640 1 -- 172.21.15.57:0/3188835175 shutdown_connections 2023-10-19T12:44:40.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:40.325+0000 7f7730983640 1 --2- 172.21.15.57:0/3188835175 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f772c13d350 0x7f772c0cbd60 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:40.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:40.325+0000 7f7730983640 1 -- 172.21.15.57:0/3188835175 >> 172.21.15.57:0/3188835175 conn(0x7f772c069930 msgr2=0x7f772c069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:40.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:40.326+0000 7f7730983640 1 -- 172.21.15.57:0/3188835175 shutdown_connections 2023-10-19T12:44:40.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:40.326+0000 7f7730983640 1 -- 172.21.15.57:0/3188835175 wait complete. 2023-10-19T12:44:40.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:40.327+0000 7f7730983640 1 Processor -- start 2023-10-19T12:44:40.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:40.327+0000 7f7730983640 1 -- start start 2023-10-19T12:44:40.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:40.328+0000 7f7730983640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f772c070350 0x7f772c0b04a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:40.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:40.328+0000 7f7730983640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f772c0cb980 0x7f772c0af5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:40.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:40.328+0000 7f7730983640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f772c136180 0x7f772c06c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:40.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:40.328+0000 7f7730983640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f772c153b70 con 0x7f772c070350 2023-10-19T12:44:40.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:40.328+0000 7f7730983640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f772c0afb10 con 0x7f772c0cb980 2023-10-19T12:44:40.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:40.328+0000 7f7730983640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f772c0b0a80 con 0x7f772c136180 2023-10-19T12:44:40.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:40.328+0000 7f772a59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f772c070350 0x7f772c0b04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:40.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:40.328+0000 7f772a59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f772c070350 0x7f772c0b04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50422/0 (socket says 172.21.15.57:50422) 2023-10-19T12:44:40.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:40.328+0000 7f772a59c640 1 -- 172.21.15.57:0/690457928 learned_addr learned my addr 172.21.15.57:0/690457928 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:40.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:40.328+0000 7f7721d9b640 1 --2- 172.21.15.57:0/690457928 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f772c0cb980 0x7f772c0af5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:40.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:40.328+0000 7f772ad9d640 1 --2- 172.21.15.57:0/690457928 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f772c136180 0x7f772c06c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:40.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:40.328+0000 7f772a59c640 1 -- 172.21.15.57:0/690457928 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f772c136180 msgr2=0x7f772c06c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:40.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:40.328+0000 7f772a59c640 1 --2- 172.21.15.57:0/690457928 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f772c136180 0x7f772c06c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:40.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:40.328+0000 7f772a59c640 1 -- 172.21.15.57:0/690457928 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f772c0cb980 msgr2=0x7f772c0af5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:40.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:40.328+0000 7f772a59c640 1 --2- 172.21.15.57:0/690457928 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f772c0cb980 0x7f772c0af5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:40.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:40.328+0000 7f772a59c640 1 -- 172.21.15.57:0/690457928 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7714009490 con 0x7f772c070350 2023-10-19T12:44:40.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:40.328+0000 7f7721d9b640 1 --2- 172.21.15.57:0/690457928 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f772c0cb980 0x7f772c0af5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:44:40.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:40.328+0000 7f772ad9d640 1 --2- 172.21.15.57:0/690457928 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f772c136180 0x7f772c06c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:44:40.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:40.328+0000 7f772a59c640 1 --2- 172.21.15.57:0/690457928 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f772c070350 0x7f772c0b04a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f771c00c500 tx=0x7f771c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:40.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:40.329+0000 7f77237fe640 1 -- 172.21.15.57:0/690457928 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f771c012830 con 0x7f772c070350 2023-10-19T12:44:40.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:40.329+0000 7f77237fe640 1 -- 172.21.15.57:0/690457928 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f771c012a80 con 0x7f772c070350 2023-10-19T12:44:40.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:40.329+0000 7f7730983640 1 -- 172.21.15.57:0/690457928 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f772c0b0d60 con 0x7f772c070350 2023-10-19T12:44:40.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:40.329+0000 7f77237fe640 1 -- 172.21.15.57:0/690457928 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f771c012e90 con 0x7f772c070350 2023-10-19T12:44:40.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:40.329+0000 7f7730983640 1 -- 172.21.15.57:0/690457928 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f772c0b1320 con 0x7f772c070350 2023-10-19T12:44:40.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:40.330+0000 7f7730983640 1 -- 172.21.15.57:0/690457928 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f772c0690e0 con 0x7f772c070350 2023-10-19T12:44:40.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:40.330+0000 7f77237fe640 1 -- 172.21.15.57:0/690457928 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f771c01a3b0 con 0x7f772c070350 2023-10-19T12:44:40.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:40.330+0000 7f77237fe640 1 --2- 172.21.15.57:0/690457928 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f770c06ae40 0x7f770c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:40.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:40.331+0000 7f77237fe640 1 -- 172.21.15.57:0/690457928 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f771c0897a0 con 0x7f772c070350 2023-10-19T12:44:40.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:40.331+0000 7f7721d9b640 1 --2- 172.21.15.57:0/690457928 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f770c06ae40 0x7f770c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:40.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:40.331+0000 7f7721d9b640 1 --2- 172.21.15.57:0/690457928 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f770c06ae40 0x7f770c06d300 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f7714036000 tx=0x7f7714004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:40.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:40.335+0000 7f77237fe640 1 -- 172.21.15.57:0/690457928 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f771c054570 con 0x7f772c070350 2023-10-19T12:44:40.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:40.544+0000 7f7730983640 1 -- 172.21.15.57:0/690457928 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f772c06ff80 con 0x7f772c070350 2023-10-19T12:44:41.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.193+0000 7f77237fe640 1 -- 172.21.15.57:0/690457928 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v9) v1 ==== 122+0+0 (secure 0 0 0) 0x7f771c057b90 con 0x7f772c070350 2023-10-19T12:44:41.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.195+0000 7f7730983640 1 -- 172.21.15.57:0/690457928 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f770c06ae40 msgr2=0x7f770c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:41.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.195+0000 7f7730983640 1 --2- 172.21.15.57:0/690457928 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f770c06ae40 0x7f770c06d300 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f7714036000 tx=0x7f7714004730 comp rx=0 tx=0).stop 2023-10-19T12:44:41.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.195+0000 7f7730983640 1 -- 172.21.15.57:0/690457928 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f772c070350 msgr2=0x7f772c0b04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:41.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.195+0000 7f7730983640 1 --2- 172.21.15.57:0/690457928 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f772c070350 0x7f772c0b04a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f771c00c500 tx=0x7f771c00c9c0 comp rx=0 tx=0).stop 2023-10-19T12:44:41.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.195+0000 7f7730983640 1 -- 172.21.15.57:0/690457928 shutdown_connections 2023-10-19T12:44:41.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.195+0000 7f7730983640 1 --2- 172.21.15.57:0/690457928 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f770c06ae40 0x7f770c06d300 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:41.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.195+0000 7f7730983640 1 --2- 172.21.15.57:0/690457928 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f772c136180 0x7f772c06c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:41.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.195+0000 7f7730983640 1 --2- 172.21.15.57:0/690457928 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f772c0cb980 0x7f772c0af5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:41.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.195+0000 7f7730983640 1 --2- 172.21.15.57:0/690457928 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f772c070350 0x7f772c0b04a0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:41.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.195+0000 7f7730983640 1 -- 172.21.15.57:0/690457928 >> 172.21.15.57:0/690457928 conn(0x7f772c069930 msgr2=0x7f772c159a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:41.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.195+0000 7f7730983640 1 -- 172.21.15.57:0/690457928 shutdown_connections 2023-10-19T12:44:41.201 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.195+0000 7f7730983640 1 -- 172.21.15.57:0/690457928 wait complete. 2023-10-19T12:44:41.212 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-19T12:44:41.212 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T12:44:41.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.324+0000 7fad94f0c640 1 -- 172.21.15.57:0/2056425008 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fad90070af0 msgr2=0x7fad900affe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:41.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.324+0000 7fad94f0c640 1 --2- 172.21.15.57:0/2056425008 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fad90070af0 0x7fad900affe0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fad900cbd00 tx=0x7fad8002c1d0 comp rx=0 tx=0).stop 2023-10-19T12:44:41.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.324+0000 7fad94f0c640 1 -- 172.21.15.57:0/2056425008 shutdown_connections 2023-10-19T12:44:41.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.324+0000 7fad94f0c640 1 --2- 172.21.15.57:0/2056425008 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fad90070af0 0x7fad900affe0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:41.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.324+0000 7fad94f0c640 1 --2- 172.21.15.57:0/2056425008 >> v2:172.21.15.179:3300/0 conn(0x7fad9013e520 0x7fad9006bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:41.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.324+0000 7fad94f0c640 1 -- 172.21.15.57:0/2056425008 >> 172.21.15.57:0/2056425008 conn(0x7fad90069930 msgr2=0x7fad90069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:41.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.324+0000 7fad94f0c640 1 -- 172.21.15.57:0/2056425008 shutdown_connections 2023-10-19T12:44:41.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.324+0000 7fad94f0c640 1 -- 172.21.15.57:0/2056425008 wait complete. 2023-10-19T12:44:41.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.325+0000 7fad94f0c640 1 Processor -- start 2023-10-19T12:44:41.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.325+0000 7fad94f0c640 1 -- start start 2023-10-19T12:44:41.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.326+0000 7fad94f0c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fad90070af0 0x7fad90159100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:41.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.326+0000 7fad94f0c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fad9013e520 0x7fad900ad870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:41.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.326+0000 7fad94f0c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fad90141690 0x7fad900addb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:41.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.326+0000 7fad94f0c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad901e3440 con 0x7fad90141690 2023-10-19T12:44:41.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.326+0000 7fad94f0c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad901e28c0 con 0x7fad90070af0 2023-10-19T12:44:41.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.326+0000 7fad94f0c640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad901e2ba0 con 0x7fad9013e520 2023-10-19T12:44:41.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.326+0000 7fad8ed9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fad90141690 0x7fad900addb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:41.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.326+0000 7fad8ed9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fad90141690 0x7fad900addb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50444/0 (socket says 172.21.15.57:50444) 2023-10-19T12:44:41.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.326+0000 7fad8ed9d640 1 -- 172.21.15.57:0/3066832655 learned_addr learned my addr 172.21.15.57:0/3066832655 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:41.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.326+0000 7fad8dd9b640 1 --2- 172.21.15.57:0/3066832655 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fad9013e520 0x7fad900ad870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:41.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.326+0000 7fad8ed9d640 1 -- 172.21.15.57:0/3066832655 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fad9013e520 msgr2=0x7fad900ad870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:41.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.326+0000 7fad8ed9d640 1 --2- 172.21.15.57:0/3066832655 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fad9013e520 0x7fad900ad870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:41.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.326+0000 7fad8ed9d640 1 -- 172.21.15.57:0/3066832655 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fad90070af0 msgr2=0x7fad90159100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:41.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.326+0000 7fad8e59c640 1 --2- 172.21.15.57:0/3066832655 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fad90070af0 0x7fad90159100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:41.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.326+0000 7fad8ed9d640 1 --2- 172.21.15.57:0/3066832655 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fad90070af0 0x7fad90159100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:41.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.327+0000 7fad8ed9d640 1 -- 172.21.15.57:0/3066832655 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad80009900 con 0x7fad90141690 2023-10-19T12:44:41.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.327+0000 7fad8dd9b640 1 --2- 172.21.15.57:0/3066832655 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fad9013e520 0x7fad900ad870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:44:41.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.327+0000 7fad8ed9d640 1 --2- 172.21.15.57:0/3066832655 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fad90141690 0x7fad900addb0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fad80037040 tx=0x7fad80035db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:41.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.327+0000 7fad86ffd640 1 -- 172.21.15.57:0/3066832655 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad80034440 con 0x7fad90141690 2023-10-19T12:44:41.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.327+0000 7fad86ffd640 1 -- 172.21.15.57:0/3066832655 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fad80004450 con 0x7fad90141690 2023-10-19T12:44:41.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.327+0000 7fad94f0c640 1 -- 172.21.15.57:0/3066832655 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad901e2e80 con 0x7fad90141690 2023-10-19T12:44:41.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.327+0000 7fad86ffd640 1 -- 172.21.15.57:0/3066832655 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad80004840 con 0x7fad90141690 2023-10-19T12:44:41.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.327+0000 7fad94f0c640 1 -- 172.21.15.57:0/3066832655 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad90159640 con 0x7fad90141690 2023-10-19T12:44:41.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.328+0000 7fad94f0c640 1 -- 172.21.15.57:0/3066832655 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad900690e0 con 0x7fad90141690 2023-10-19T12:44:41.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.328+0000 7fad86ffd640 1 -- 172.21.15.57:0/3066832655 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fad8003b040 con 0x7fad90141690 2023-10-19T12:44:41.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.329+0000 7fad86ffd640 1 --2- 172.21.15.57:0/3066832655 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fad7006ae40 0x7fad7006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:41.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.329+0000 7fad86ffd640 1 -- 172.21.15.57:0/3066832655 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fad800ac7c0 con 0x7fad90141690 2023-10-19T12:44:41.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.329+0000 7fad8e59c640 1 --2- 172.21.15.57:0/3066832655 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fad7006ae40 0x7fad7006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:41.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.330+0000 7fad8e59c640 1 --2- 172.21.15.57:0/3066832655 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fad7006ae40 0x7fad7006d300 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fad8800a810 tx=0x7fad88008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:41.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.333+0000 7fad86ffd640 1 -- 172.21.15.57:0/3066832655 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fad80077580 con 0x7fad90141690 2023-10-19T12:44:41.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.546+0000 7fad94f0c640 1 -- 172.21.15.57:0/3066832655 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fad900a64b0 con 0x7fad90141690 2023-10-19T12:44:41.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.546+0000 7fad86ffd640 1 -- 172.21.15.57:0/3066832655 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4906 (secure 0 0 0) 0x7fad8007aba0 con 0x7fad90141690 2023-10-19T12:44:41.550 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:44:41.550 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4357,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/4228712742","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":4228712742},{"type":"v1","addr":"172.21.15.57:6835","nonce":4228712742}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4360,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/971892630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":971892630},{"type":"v1","addr":"172.21.15.57:6834","nonce":971892630}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4372,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/104495983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":104495983},{"type":"v1","addr":"172.21.15.179:6836","nonce":104495983}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T12:44:18.419669+0000","modified":"2023-10-19T12:44:40.545928+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4374},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4374":{"gid":4374,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.179:6837/622550365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":622550365},{"type":"v1","addr":"172.21.15.179:6837","nonce":622550365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-19T12:44:41.551 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 9 2023-10-19T12:44:41.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.548+0000 7fad94f0c640 1 -- 172.21.15.57:0/3066832655 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fad7006ae40 msgr2=0x7fad7006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:41.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.548+0000 7fad94f0c640 1 --2- 172.21.15.57:0/3066832655 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fad7006ae40 0x7fad7006d300 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fad8800a810 tx=0x7fad88008040 comp rx=0 tx=0).stop 2023-10-19T12:44:41.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.548+0000 7fad94f0c640 1 -- 172.21.15.57:0/3066832655 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fad90141690 msgr2=0x7fad900addb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:41.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.548+0000 7fad94f0c640 1 --2- 172.21.15.57:0/3066832655 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fad90141690 0x7fad900addb0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fad80037040 tx=0x7fad80035db0 comp rx=0 tx=0).stop 2023-10-19T12:44:41.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.548+0000 7fad94f0c640 1 -- 172.21.15.57:0/3066832655 shutdown_connections 2023-10-19T12:44:41.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.548+0000 7fad94f0c640 1 --2- 172.21.15.57:0/3066832655 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fad7006ae40 0x7fad7006d300 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:41.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.548+0000 7fad94f0c640 1 --2- 172.21.15.57:0/3066832655 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fad90141690 0x7fad900addb0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:41.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.548+0000 7fad94f0c640 1 --2- 172.21.15.57:0/3066832655 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fad9013e520 0x7fad900ad870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:41.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.548+0000 7fad94f0c640 1 --2- 172.21.15.57:0/3066832655 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fad90070af0 0x7fad90159100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:41.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.548+0000 7fad94f0c640 1 -- 172.21.15.57:0/3066832655 >> 172.21.15.57:0/3066832655 conn(0x7fad90069930 msgr2=0x7fad900ae700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:41.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.548+0000 7fad94f0c640 1 -- 172.21.15.57:0/3066832655 shutdown_connections 2023-10-19T12:44:41.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.548+0000 7fad94f0c640 1 -- 172.21.15.57:0/3066832655 wait complete. 2023-10-19T12:44:41.562 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T12:44:41.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.668+0000 7f59db239640 1 -- 172.21.15.57:0/3921524290 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f59d415b640 msgr2=0x7f59d40cf850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:41.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.668+0000 7f59db239640 1 --2- 172.21.15.57:0/3921524290 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f59d415b640 0x7f59d40cf850 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f59d40894b0 tx=0x7f59c802bda0 comp rx=0 tx=0).stop 2023-10-19T12:44:41.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.668+0000 7f59db239640 1 -- 172.21.15.57:0/3921524290 shutdown_connections 2023-10-19T12:44:41.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.668+0000 7f59db239640 1 --2- 172.21.15.57:0/3921524290 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f59d415b640 0x7f59d40cf850 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:41.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.668+0000 7f59db239640 1 --2- 172.21.15.57:0/3921524290 >> v2:172.21.15.179:3300/0 conn(0x7f59d415b260 0x7f59d40cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:41.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.668+0000 7f59db239640 1 -- 172.21.15.57:0/3921524290 >> 172.21.15.57:0/3921524290 conn(0x7f59d409a030 msgr2=0x7f59d40d0d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:41.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.668+0000 7f59db239640 1 -- 172.21.15.57:0/3921524290 shutdown_connections 2023-10-19T12:44:41.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.668+0000 7f59db239640 1 -- 172.21.15.57:0/3921524290 wait complete. 2023-10-19T12:44:41.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.669+0000 7f59db239640 1 Processor -- start 2023-10-19T12:44:41.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.669+0000 7f59db239640 1 -- start start 2023-10-19T12:44:41.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.670+0000 7f59db239640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f59d40b0b10 0x7f59d40b3940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:41.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.670+0000 7f59db239640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f59d415b260 0x7f59d41418e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:41.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.670+0000 7f59db239640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f59d415b640 0x7f59d415cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:41.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.670+0000 7f59db239640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59d40b6c10 con 0x7f59d40b0b10 2023-10-19T12:44:41.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.670+0000 7f59db239640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59d4147cb0 con 0x7f59d415b640 2023-10-19T12:44:41.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.670+0000 7f59db239640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59d40b6650 con 0x7f59d415b260 2023-10-19T12:44:41.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.670+0000 7f59d8fd5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f59d40b0b10 0x7f59d40b3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:41.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.670+0000 7f59d8fd5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f59d40b0b10 0x7f59d40b3940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50468/0 (socket says 172.21.15.57:50468) 2023-10-19T12:44:41.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.670+0000 7f59d8fd5640 1 -- 172.21.15.57:0/1781802896 learned_addr learned my addr 172.21.15.57:0/1781802896 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:41.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.671+0000 7f59d8fd5640 1 -- 172.21.15.57:0/1781802896 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f59d415b260 msgr2=0x7f59d41418e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:41.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.671+0000 7f59d3fff640 1 --2- 172.21.15.57:0/1781802896 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f59d415b260 0x7f59d41418e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:41.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.671+0000 7f59d97d6640 1 --2- 172.21.15.57:0/1781802896 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f59d415b640 0x7f59d415cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:41.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.671+0000 7f59d8fd5640 1 --2- 172.21.15.57:0/1781802896 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f59d415b260 0x7f59d41418e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:41.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.671+0000 7f59d8fd5640 1 -- 172.21.15.57:0/1781802896 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f59d415b640 msgr2=0x7f59d415cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:41.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.671+0000 7f59d8fd5640 1 --2- 172.21.15.57:0/1781802896 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f59d415b640 0x7f59d415cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:41.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.671+0000 7f59d8fd5640 1 -- 172.21.15.57:0/1781802896 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59c8009560 con 0x7f59d40b0b10 2023-10-19T12:44:41.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.671+0000 7f59d3fff640 1 --2- 172.21.15.57:0/1781802896 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f59d415b260 0x7f59d41418e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:44:41.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.671+0000 7f59d97d6640 1 --2- 172.21.15.57:0/1781802896 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f59d415b640 0x7f59d415cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:44:41.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.671+0000 7f59d8fd5640 1 --2- 172.21.15.57:0/1781802896 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f59d40b0b10 0x7f59d40b3940 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f59cc00cce0 tx=0x7f59cc00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:41.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.671+0000 7f59d17fa640 1 -- 172.21.15.57:0/1781802896 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59cc00f650 con 0x7f59d40b0b10 2023-10-19T12:44:41.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.671+0000 7f59d17fa640 1 -- 172.21.15.57:0/1781802896 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f59cc00ce90 con 0x7f59d40b0b10 2023-10-19T12:44:41.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.671+0000 7f59db239640 1 -- 172.21.15.57:0/1781802896 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59d40b6930 con 0x7f59d40b0b10 2023-10-19T12:44:41.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.671+0000 7f59d17fa640 1 -- 172.21.15.57:0/1781802896 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59cc00fb40 con 0x7f59d40b0b10 2023-10-19T12:44:41.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.671+0000 7f59db239640 1 -- 172.21.15.57:0/1781802896 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59d4141e20 con 0x7f59d40b0b10 2023-10-19T12:44:41.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.672+0000 7f59db239640 1 -- 172.21.15.57:0/1781802896 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59d415cc60 con 0x7f59d40b0b10 2023-10-19T12:44:41.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.673+0000 7f59d17fa640 1 -- 172.21.15.57:0/1781802896 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f59cc00fca0 con 0x7f59d40b0b10 2023-10-19T12:44:41.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.673+0000 7f59d17fa640 1 --2- 172.21.15.57:0/1781802896 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f59c006ae40 0x7f59c006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:41.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.673+0000 7f59d17fa640 1 -- 172.21.15.57:0/1781802896 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f59cc08a5f0 con 0x7f59d40b0b10 2023-10-19T12:44:41.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.673+0000 7f59d3fff640 1 --2- 172.21.15.57:0/1781802896 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f59c006ae40 0x7f59c006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:41.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.674+0000 7f59d3fff640 1 --2- 172.21.15.57:0/1781802896 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f59c006ae40 0x7f59c006d300 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f59c8035000 tx=0x7f59c8004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:41.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.677+0000 7f59d17fa640 1 -- 172.21.15.57:0/1781802896 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f59d415cc60 con 0x7f59d40b0b10 2023-10-19T12:44:41.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.854+0000 7f59db239640 1 -- 172.21.15.57:0/1781802896 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f59d415cc60 con 0x7f59d40b0b10 2023-10-19T12:44:41.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.855+0000 7f59d17fa640 1 -- 172.21.15.57:0/1781802896 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18358 (secure 0 0 0) 0x7f59d415cc60 con 0x7f59d40b0b10 2023-10-19T12:44:41.859 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:44:41.859 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":25,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T12:44:23.472399+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-19T12:44:16.517510+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-19T12:44:17.398297+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-19T12:44:21.129403+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T12:44:41.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.857+0000 7f59db239640 1 -- 172.21.15.57:0/1781802896 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f59c006ae40 msgr2=0x7f59c006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:41.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.857+0000 7f59db239640 1 --2- 172.21.15.57:0/1781802896 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f59c006ae40 0x7f59c006d300 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f59c8035000 tx=0x7f59c8004730 comp rx=0 tx=0).stop 2023-10-19T12:44:41.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.857+0000 7f59db239640 1 -- 172.21.15.57:0/1781802896 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f59d40b0b10 msgr2=0x7f59d40b3940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:41.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.857+0000 7f59db239640 1 --2- 172.21.15.57:0/1781802896 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f59d40b0b10 0x7f59d40b3940 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f59cc00cce0 tx=0x7f59cc00a620 comp rx=0 tx=0).stop 2023-10-19T12:44:41.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.857+0000 7f59db239640 1 -- 172.21.15.57:0/1781802896 shutdown_connections 2023-10-19T12:44:41.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.857+0000 7f59db239640 1 --2- 172.21.15.57:0/1781802896 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f59c006ae40 0x7f59c006d300 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:41.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.857+0000 7f59db239640 1 --2- 172.21.15.57:0/1781802896 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f59d415b640 0x7f59d415cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:41.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.857+0000 7f59db239640 1 --2- 172.21.15.57:0/1781802896 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f59d415b260 0x7f59d41418e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:41.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.857+0000 7f59db239640 1 --2- 172.21.15.57:0/1781802896 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f59d40b0b10 0x7f59d40b3940 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:41.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.857+0000 7f59db239640 1 -- 172.21.15.57:0/1781802896 >> 172.21.15.57:0/1781802896 conn(0x7f59d409a030 msgr2=0x7f59d40aef50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:41.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.857+0000 7f59db239640 1 -- 172.21.15.57:0/1781802896 shutdown_connections 2023-10-19T12:44:41.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.857+0000 7f59db239640 1 -- 172.21.15.57:0/1781802896 wait complete. 2023-10-19T12:44:41.872 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-19T12:44:41.872 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-19T12:44:41.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.979+0000 7f5b44e75640 1 -- 172.21.15.57:0/1987459634 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5b4006ae40 msgr2=0x7f5b400cccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:41.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.979+0000 7f5b44e75640 1 --2- 172.21.15.57:0/1987459634 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5b4006ae40 0x7f5b400cccc0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f5b30005fd0 tx=0x7f5b3002c0e0 comp rx=0 tx=0).stop 2023-10-19T12:44:41.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.979+0000 7f5b44e75640 1 -- 172.21.15.57:0/1987459634 shutdown_connections 2023-10-19T12:44:41.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.979+0000 7f5b44e75640 1 --2- 172.21.15.57:0/1987459634 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5b4006ae40 0x7f5b400cccc0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:41.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.979+0000 7f5b44e75640 1 -- 172.21.15.57:0/1987459634 >> 172.21.15.57:0/1987459634 conn(0x7f5b4006fae0 msgr2=0x7f5b4006fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:41.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.979+0000 7f5b44e75640 1 -- 172.21.15.57:0/1987459634 shutdown_connections 2023-10-19T12:44:41.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.979+0000 7f5b44e75640 1 -- 172.21.15.57:0/1987459634 wait complete. 2023-10-19T12:44:41.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.980+0000 7f5b44e75640 1 Processor -- start 2023-10-19T12:44:41.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.980+0000 7f5b44e75640 1 -- start start 2023-10-19T12:44:41.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.981+0000 7f5b44e75640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5b4006ae40 0x7f5b400b1560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:41.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.981+0000 7f5b44e75640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5b4014fd70 0x7f5b400aab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:41.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.981+0000 7f5b44e75640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5b40159d40 0x7f5b401463f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:41.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.981+0000 7f5b44e75640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b400b1220 con 0x7f5b4006ae40 2023-10-19T12:44:41.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.981+0000 7f5b44e75640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b400add40 con 0x7f5b4014fd70 2023-10-19T12:44:41.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.981+0000 7f5b44e75640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b400ae5e0 con 0x7f5b40159d40 2023-10-19T12:44:41.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.981+0000 7f5b3e59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5b4006ae40 0x7f5b400b1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:41.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.981+0000 7f5b3e59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5b4006ae40 0x7f5b400b1560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50492/0 (socket says 172.21.15.57:50492) 2023-10-19T12:44:41.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.981+0000 7f5b3e59c640 1 -- 172.21.15.57:0/1737218849 learned_addr learned my addr 172.21.15.57:0/1737218849 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:41.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.981+0000 7f5b3ed9d640 1 --2- 172.21.15.57:0/1737218849 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5b40159d40 0x7f5b401463f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:41.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.981+0000 7f5b3dd9b640 1 --2- 172.21.15.57:0/1737218849 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5b4014fd70 0x7f5b400aab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:41.989 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.981+0000 7f5b3e59c640 1 -- 172.21.15.57:0/1737218849 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5b40159d40 msgr2=0x7f5b401463f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:41.989 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.981+0000 7f5b3e59c640 1 --2- 172.21.15.57:0/1737218849 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5b40159d40 0x7f5b401463f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:41.989 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.981+0000 7f5b3e59c640 1 -- 172.21.15.57:0/1737218849 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5b4014fd70 msgr2=0x7f5b400aab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:41.989 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.981+0000 7f5b3e59c640 1 --2- 172.21.15.57:0/1737218849 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5b4014fd70 0x7f5b400aab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:41.989 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.981+0000 7f5b3e59c640 1 -- 172.21.15.57:0/1737218849 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b30009840 con 0x7f5b4006ae40 2023-10-19T12:44:41.990 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.981+0000 7f5b3dd9b640 1 --2- 172.21.15.57:0/1737218849 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5b4014fd70 0x7f5b400aab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:44:41.990 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.982+0000 7f5b3e59c640 1 --2- 172.21.15.57:0/1737218849 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5b4006ae40 0x7f5b400b1560 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f5b3800c460 tx=0x7f5b3800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:41.990 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.982+0000 7f5b2effd640 1 -- 172.21.15.57:0/1737218849 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b38012830 con 0x7f5b4006ae40 2023-10-19T12:44:41.990 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.982+0000 7f5b2effd640 1 -- 172.21.15.57:0/1737218849 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f5b38012ac0 con 0x7f5b4006ae40 2023-10-19T12:44:41.990 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.982+0000 7f5b2effd640 1 -- 172.21.15.57:0/1737218849 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b38011380 con 0x7f5b4006ae40 2023-10-19T12:44:41.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.982+0000 7f5b44e75640 1 -- 172.21.15.57:0/1737218849 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b400b1bf0 con 0x7f5b4006ae40 2023-10-19T12:44:41.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.982+0000 7f5b44e75640 1 -- 172.21.15.57:0/1737218849 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b400ae270 con 0x7f5b4006ae40 2023-10-19T12:44:41.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.983+0000 7f5b44e75640 1 -- 172.21.15.57:0/1737218849 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b40066f20 con 0x7f5b4006ae40 2023-10-19T12:44:41.992 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.983+0000 7f5b2effd640 1 -- 172.21.15.57:0/1737218849 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f5b380114e0 con 0x7f5b4006ae40 2023-10-19T12:44:41.992 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.984+0000 7f5b2effd640 1 --2- 172.21.15.57:0/1737218849 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f5b2006ae70 0x7f5b2006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:41.992 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.984+0000 7f5b2effd640 1 -- 172.21.15.57:0/1737218849 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f5b38089720 con 0x7f5b4006ae40 2023-10-19T12:44:41.993 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.984+0000 7f5b3dd9b640 1 --2- 172.21.15.57:0/1737218849 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f5b2006ae70 0x7f5b2006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:41.993 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.984+0000 7f5b3dd9b640 1 --2- 172.21.15.57:0/1737218849 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f5b2006ae70 0x7f5b2006d330 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f5b40159030 tx=0x7f5b34009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:41.993 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:41.988+0000 7f5b2effd640 1 -- 172.21.15.57:0/1737218849 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f5b38054570 con 0x7f5b4006ae40 2023-10-19T12:44:42.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.164+0000 7f5b44e75640 1 -- 172.21.15.57:0/1737218849 --> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5b400ae720 con 0x7f5b2006ae70 2023-10-19T12:44:42.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.174+0000 7f5b2effd640 1 -- 172.21.15.57:0/1737218849 <== mgr.4100 v2:172.21.15.179:6832/2609646285 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+345304 (secure 0 0 0) 0x7f5b3400d4f0 con 0x7f5b2006ae70 2023-10-19T12:44:42.180 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:44:42.182 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2023-10-19T12:44:42.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.177+0000 7f5b44e75640 1 -- 172.21.15.57:0/1737218849 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f5b2006ae70 msgr2=0x7f5b2006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:42.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.177+0000 7f5b44e75640 1 --2- 172.21.15.57:0/1737218849 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f5b2006ae70 0x7f5b2006d330 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f5b40159030 tx=0x7f5b34009040 comp rx=0 tx=0).stop 2023-10-19T12:44:42.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.177+0000 7f5b44e75640 1 -- 172.21.15.57:0/1737218849 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5b4006ae40 msgr2=0x7f5b400b1560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:42.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.177+0000 7f5b44e75640 1 --2- 172.21.15.57:0/1737218849 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5b4006ae40 0x7f5b400b1560 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f5b3800c460 tx=0x7f5b3800c920 comp rx=0 tx=0).stop 2023-10-19T12:44:42.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.177+0000 7f5b44e75640 1 -- 172.21.15.57:0/1737218849 shutdown_connections 2023-10-19T12:44:42.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.177+0000 7f5b44e75640 1 --2- 172.21.15.57:0/1737218849 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f5b2006ae70 0x7f5b2006d330 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:42.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.177+0000 7f5b44e75640 1 --2- 172.21.15.57:0/1737218849 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5b40159d40 0x7f5b401463f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:42.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.177+0000 7f5b44e75640 1 --2- 172.21.15.57:0/1737218849 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5b4014fd70 0x7f5b400aab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:42.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.177+0000 7f5b44e75640 1 --2- 172.21.15.57:0/1737218849 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5b4006ae40 0x7f5b400b1560 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:42.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.177+0000 7f5b44e75640 1 -- 172.21.15.57:0/1737218849 >> 172.21.15.57:0/1737218849 conn(0x7f5b4006fae0 msgr2=0x7f5b4006c9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:42.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.177+0000 7f5b44e75640 1 -- 172.21.15.57:0/1737218849 shutdown_connections 2023-10-19T12:44:42.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.177+0000 7f5b44e75640 1 -- 172.21.15.57:0/1737218849 wait complete. 2023-10-19T12:44:42.191 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":36,"stamp":"2023-10-19T12:44:41.319901+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595007,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236140,"kb_used_data":3300,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738580,"statfs":{"total":773094113280,"available":772852305920,"internally_reserved":0,"allocated":3379200,"data_stored":2025003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":238276153},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2229,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003613"},"pg_stats":[{"pgid":"2.1f","version":"20'4","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.113796+0000","last_change":"2023-10-19T12:44:18.071366+0000","last_active":"2023-10-19T12:44:23.113796+0000","last_peered":"2023-10-19T12:44:23.113796+0000","last_clean":"2023-10-19T12:44:23.113796+0000","last_became_active":"2023-10-19T12:44:18.071117+0000","last_became_peered":"2023-10-19T12:44:18.071117+0000","last_unstale":"2023-10-19T12:44:23.113796+0000","last_undegraded":"2023-10-19T12:44:23.113796+0000","last_fullsized":"2023-10-19T12:44:23.113796+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:24:31.294273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.107571+0000","last_change":"2023-10-19T12:44:19.080917+0000","last_active":"2023-10-19T12:44:23.107571+0000","last_peered":"2023-10-19T12:44:23.107571+0000","last_clean":"2023-10-19T12:44:23.107571+0000","last_became_active":"2023-10-19T12:44:19.080408+0000","last_became_peered":"2023-10-19T12:44:19.080408+0000","last_unstale":"2023-10-19T12:44:23.107571+0000","last_undegraded":"2023-10-19T12:44:23.107571+0000","last_fullsized":"2023-10-19T12:44:23.107571+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:29:52.686703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.258880+0000","last_change":"2023-10-19T12:44:23.262149+0000","last_active":"2023-10-19T12:44:24.258880+0000","last_peered":"2023-10-19T12:44:24.258880+0000","last_clean":"2023-10-19T12:44:24.258880+0000","last_became_active":"2023-10-19T12:44:23.261764+0000","last_became_peered":"2023-10-19T12:44:23.261764+0000","last_unstale":"2023-10-19T12:44:24.258880+0000","last_undegraded":"2023-10-19T12:44:24.258880+0000","last_fullsized":"2023-10-19T12:44:24.258880+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:33:24.282692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.107679+0000","last_change":"2023-10-19T12:44:18.064762+0000","last_active":"2023-10-19T12:44:23.107679+0000","last_peered":"2023-10-19T12:44:23.107679+0000","last_clean":"2023-10-19T12:44:23.107679+0000","last_became_active":"2023-10-19T12:44:18.064478+0000","last_became_peered":"2023-10-19T12:44:18.064478+0000","last_unstale":"2023-10-19T12:44:23.107679+0000","last_undegraded":"2023-10-19T12:44:23.107679+0000","last_fullsized":"2023-10-19T12:44:23.107679+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:46:29.885859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.113623+0000","last_change":"2023-10-19T12:44:19.083619+0000","last_active":"2023-10-19T12:44:23.113623+0000","last_peered":"2023-10-19T12:44:23.113623+0000","last_clean":"2023-10-19T12:44:23.113623+0000","last_became_active":"2023-10-19T12:44:19.083330+0000","last_became_peered":"2023-10-19T12:44:19.083330+0000","last_unstale":"2023-10-19T12:44:23.113623+0000","last_undegraded":"2023-10-19T12:44:23.113623+0000","last_fullsized":"2023-10-19T12:44:23.113623+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:00:15.982705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.123057+0000","last_change":"2023-10-19T12:44:23.132517+0000","last_active":"2023-10-19T12:44:24.123057+0000","last_peered":"2023-10-19T12:44:24.123057+0000","last_clean":"2023-10-19T12:44:24.123057+0000","last_became_active":"2023-10-19T12:44:23.132066+0000","last_became_peered":"2023-10-19T12:44:23.132066+0000","last_unstale":"2023-10-19T12:44:24.123057+0000","last_undegraded":"2023-10-19T12:44:24.123057+0000","last_fullsized":"2023-10-19T12:44:24.123057+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:46:59.402450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.260017+0000","last_change":"2023-10-19T12:44:18.070873+0000","last_active":"2023-10-19T12:44:23.260017+0000","last_peered":"2023-10-19T12:44:23.260017+0000","last_clean":"2023-10-19T12:44:23.260017+0000","last_became_active":"2023-10-19T12:44:18.070602+0000","last_became_peered":"2023-10-19T12:44:18.070602+0000","last_unstale":"2023-10-19T12:44:23.260017+0000","last_undegraded":"2023-10-19T12:44:23.260017+0000","last_fullsized":"2023-10-19T12:44:23.260017+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:15:18.575278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.106941+0000","last_change":"2023-10-19T12:44:19.075787+0000","last_active":"2023-10-19T12:44:23.106941+0000","last_peered":"2023-10-19T12:44:23.106941+0000","last_clean":"2023-10-19T12:44:23.106941+0000","last_became_active":"2023-10-19T12:44:19.074529+0000","last_became_peered":"2023-10-19T12:44:19.074529+0000","last_unstale":"2023-10-19T12:44:23.106941+0000","last_undegraded":"2023-10-19T12:44:23.106941+0000","last_fullsized":"2023-10-19T12:44:23.106941+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:29:52.686703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.124069+0000","last_change":"2023-10-19T12:44:23.132792+0000","last_active":"2023-10-19T12:44:24.124069+0000","last_peered":"2023-10-19T12:44:24.124069+0000","last_clean":"2023-10-19T12:44:24.124069+0000","last_became_active":"2023-10-19T12:44:23.132099+0000","last_became_peered":"2023-10-19T12:44:23.132099+0000","last_unstale":"2023-10-19T12:44:24.124069+0000","last_undegraded":"2023-10-19T12:44:24.124069+0000","last_fullsized":"2023-10-19T12:44:24.124069+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:01:13.019006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.259778+0000","last_change":"2023-10-19T12:44:18.068582+0000","last_active":"2023-10-19T12:44:23.259778+0000","last_peered":"2023-10-19T12:44:23.259778+0000","last_clean":"2023-10-19T12:44:23.259778+0000","last_became_active":"2023-10-19T12:44:18.068099+0000","last_became_peered":"2023-10-19T12:44:18.068099+0000","last_unstale":"2023-10-19T12:44:23.259778+0000","last_undegraded":"2023-10-19T12:44:23.259778+0000","last_fullsized":"2023-10-19T12:44:23.259778+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:24:31.294273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.107389+0000","last_change":"2023-10-19T12:44:19.076968+0000","last_active":"2023-10-19T12:44:23.107389+0000","last_peered":"2023-10-19T12:44:23.107389+0000","last_clean":"2023-10-19T12:44:23.107389+0000","last_became_active":"2023-10-19T12:44:19.076105+0000","last_became_peered":"2023-10-19T12:44:19.076105+0000","last_unstale":"2023-10-19T12:44:23.107389+0000","last_undegraded":"2023-10-19T12:44:23.107389+0000","last_fullsized":"2023-10-19T12:44:23.107389+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:59:45.926427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.123560+0000","last_change":"2023-10-19T12:44:23.125353+0000","last_active":"2023-10-19T12:44:24.123560+0000","last_peered":"2023-10-19T12:44:24.123560+0000","last_clean":"2023-10-19T12:44:24.123560+0000","last_became_active":"2023-10-19T12:44:23.124845+0000","last_became_peered":"2023-10-19T12:44:23.124845+0000","last_unstale":"2023-10-19T12:44:24.123560+0000","last_undegraded":"2023-10-19T12:44:24.123560+0000","last_fullsized":"2023-10-19T12:44:24.123560+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:59:49.972984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.116065+0000","last_change":"2023-10-19T12:44:19.084588+0000","last_active":"2023-10-19T12:44:23.116065+0000","last_peered":"2023-10-19T12:44:23.116065+0000","last_clean":"2023-10-19T12:44:23.116065+0000","last_became_active":"2023-10-19T12:44:19.084315+0000","last_became_peered":"2023-10-19T12:44:19.084315+0000","last_unstale":"2023-10-19T12:44:23.116065+0000","last_undegraded":"2023-10-19T12:44:23.116065+0000","last_fullsized":"2023-10-19T12:44:23.116065+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:54:38.037190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.259794+0000","last_change":"2023-10-19T12:44:18.069120+0000","last_active":"2023-10-19T12:44:23.259794+0000","last_peered":"2023-10-19T12:44:23.259794+0000","last_clean":"2023-10-19T12:44:23.259794+0000","last_became_active":"2023-10-19T12:44:18.068509+0000","last_became_peered":"2023-10-19T12:44:18.068509+0000","last_unstale":"2023-10-19T12:44:23.259794+0000","last_undegraded":"2023-10-19T12:44:23.259794+0000","last_fullsized":"2023-10-19T12:44:23.259794+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:01:31.282481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.124541+0000","last_change":"2023-10-19T12:44:23.133194+0000","last_active":"2023-10-19T12:44:24.124541+0000","last_peered":"2023-10-19T12:44:24.124541+0000","last_clean":"2023-10-19T12:44:24.124541+0000","last_became_active":"2023-10-19T12:44:23.132109+0000","last_became_peered":"2023-10-19T12:44:23.132109+0000","last_unstale":"2023-10-19T12:44:24.124541+0000","last_undegraded":"2023-10-19T12:44:24.124541+0000","last_fullsized":"2023-10-19T12:44:24.124541+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:37:44.936303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.108577+0000","last_change":"2023-10-19T12:44:19.082038+0000","last_active":"2023-10-19T12:44:23.108577+0000","last_peered":"2023-10-19T12:44:23.108577+0000","last_clean":"2023-10-19T12:44:23.108577+0000","last_became_active":"2023-10-19T12:44:19.080765+0000","last_became_peered":"2023-10-19T12:44:19.080765+0000","last_unstale":"2023-10-19T12:44:23.108577+0000","last_undegraded":"2023-10-19T12:44:23.108577+0000","last_fullsized":"2023-10-19T12:44:23.108577+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:01:32.286452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.123463+0000","last_change":"2023-10-19T12:44:18.070227+0000","last_active":"2023-10-19T12:44:23.123463+0000","last_peered":"2023-10-19T12:44:23.123463+0000","last_clean":"2023-10-19T12:44:23.123463+0000","last_became_active":"2023-10-19T12:44:18.069958+0000","last_became_peered":"2023-10-19T12:44:18.069958+0000","last_unstale":"2023-10-19T12:44:23.123463+0000","last_undegraded":"2023-10-19T12:44:23.123463+0000","last_fullsized":"2023-10-19T12:44:23.123463+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:27:49.242393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.123413+0000","last_change":"2023-10-19T12:44:23.126070+0000","last_active":"2023-10-19T12:44:24.123413+0000","last_peered":"2023-10-19T12:44:24.123413+0000","last_clean":"2023-10-19T12:44:24.123413+0000","last_became_active":"2023-10-19T12:44:23.125651+0000","last_became_peered":"2023-10-19T12:44:23.125651+0000","last_unstale":"2023-10-19T12:44:24.123413+0000","last_undegraded":"2023-10-19T12:44:24.123413+0000","last_fullsized":"2023-10-19T12:44:24.123413+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:29:56.733260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.109518+0000","last_change":"2023-10-19T12:44:19.085767+0000","last_active":"2023-10-19T12:44:23.109518+0000","last_peered":"2023-10-19T12:44:23.109518+0000","last_clean":"2023-10-19T12:44:23.109518+0000","last_became_active":"2023-10-19T12:44:19.085473+0000","last_became_peered":"2023-10-19T12:44:19.085473+0000","last_unstale":"2023-10-19T12:44:23.109518+0000","last_undegraded":"2023-10-19T12:44:23.109518+0000","last_fullsized":"2023-10-19T12:44:23.109518+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:50:42.817437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.123002+0000","last_change":"2023-10-19T12:44:18.074055+0000","last_active":"2023-10-19T12:44:23.123002+0000","last_peered":"2023-10-19T12:44:23.123002+0000","last_clean":"2023-10-19T12:44:23.123002+0000","last_became_active":"2023-10-19T12:44:18.073821+0000","last_became_peered":"2023-10-19T12:44:18.073821+0000","last_unstale":"2023-10-19T12:44:23.123002+0000","last_undegraded":"2023-10-19T12:44:23.123002+0000","last_fullsized":"2023-10-19T12:44:23.123002+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:24:31.294273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.125580+0000","last_change":"2023-10-19T12:44:23.263639+0000","last_active":"2023-10-19T12:44:24.125580+0000","last_peered":"2023-10-19T12:44:24.125580+0000","last_clean":"2023-10-19T12:44:24.125580+0000","last_became_active":"2023-10-19T12:44:23.263261+0000","last_became_peered":"2023-10-19T12:44:23.263261+0000","last_unstale":"2023-10-19T12:44:24.125580+0000","last_undegraded":"2023-10-19T12:44:24.125580+0000","last_fullsized":"2023-10-19T12:44:24.125580+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:53:36.534082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.114649+0000","last_change":"2023-10-19T12:44:19.080014+0000","last_active":"2023-10-19T12:44:23.114649+0000","last_peered":"2023-10-19T12:44:23.114649+0000","last_clean":"2023-10-19T12:44:23.114649+0000","last_became_active":"2023-10-19T12:44:19.079754+0000","last_became_peered":"2023-10-19T12:44:19.079754+0000","last_unstale":"2023-10-19T12:44:23.114649+0000","last_undegraded":"2023-10-19T12:44:23.114649+0000","last_fullsized":"2023-10-19T12:44:23.114649+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:27:39.900179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.259411+0000","last_change":"2023-10-19T12:44:18.069208+0000","last_active":"2023-10-19T12:44:23.259411+0000","last_peered":"2023-10-19T12:44:23.259411+0000","last_clean":"2023-10-19T12:44:23.259411+0000","last_became_active":"2023-10-19T12:44:18.068303+0000","last_became_peered":"2023-10-19T12:44:18.068303+0000","last_unstale":"2023-10-19T12:44:23.259411+0000","last_undegraded":"2023-10-19T12:44:23.259411+0000","last_fullsized":"2023-10-19T12:44:23.259411+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:11:21.842323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.123471+0000","last_change":"2023-10-19T12:44:23.131978+0000","last_active":"2023-10-19T12:44:24.123471+0000","last_peered":"2023-10-19T12:44:24.123471+0000","last_clean":"2023-10-19T12:44:24.123471+0000","last_became_active":"2023-10-19T12:44:23.131330+0000","last_became_peered":"2023-10-19T12:44:23.131330+0000","last_unstale":"2023-10-19T12:44:24.123471+0000","last_undegraded":"2023-10-19T12:44:24.123471+0000","last_fullsized":"2023-10-19T12:44:24.123471+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:49:30.221056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.113302+0000","last_change":"2023-10-19T12:44:19.087545+0000","last_active":"2023-10-19T12:44:23.113302+0000","last_peered":"2023-10-19T12:44:23.113302+0000","last_clean":"2023-10-19T12:44:23.113302+0000","last_became_active":"2023-10-19T12:44:19.087254+0000","last_became_peered":"2023-10-19T12:44:19.087254+0000","last_unstale":"2023-10-19T12:44:23.113302+0000","last_undegraded":"2023-10-19T12:44:23.113302+0000","last_fullsized":"2023-10-19T12:44:23.113302+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:22:48.368572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.116983+0000","last_change":"2023-10-19T12:44:18.074011+0000","last_active":"2023-10-19T12:44:23.116983+0000","last_peered":"2023-10-19T12:44:23.116983+0000","last_clean":"2023-10-19T12:44:23.116983+0000","last_became_active":"2023-10-19T12:44:18.073749+0000","last_became_peered":"2023-10-19T12:44:18.073749+0000","last_unstale":"2023-10-19T12:44:23.116983+0000","last_undegraded":"2023-10-19T12:44:23.116983+0000","last_fullsized":"2023-10-19T12:44:23.116983+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:46:29.885859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.123298+0000","last_change":"2023-10-19T12:44:23.264194+0000","last_active":"2023-10-19T12:44:24.123298+0000","last_peered":"2023-10-19T12:44:24.123298+0000","last_clean":"2023-10-19T12:44:24.123298+0000","last_became_active":"2023-10-19T12:44:23.263028+0000","last_became_peered":"2023-10-19T12:44:23.263028+0000","last_unstale":"2023-10-19T12:44:24.123298+0000","last_undegraded":"2023-10-19T12:44:24.123298+0000","last_fullsized":"2023-10-19T12:44:24.123298+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:05:19.882281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.107858+0000","last_change":"2023-10-19T12:44:19.094059+0000","last_active":"2023-10-19T12:44:23.107858+0000","last_peered":"2023-10-19T12:44:23.107858+0000","last_clean":"2023-10-19T12:44:23.107858+0000","last_became_active":"2023-10-19T12:44:19.093724+0000","last_became_peered":"2023-10-19T12:44:19.093724+0000","last_unstale":"2023-10-19T12:44:23.107858+0000","last_undegraded":"2023-10-19T12:44:23.107858+0000","last_fullsized":"2023-10-19T12:44:23.107858+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:49:26.174499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.123329+0000","last_change":"2023-10-19T12:44:18.071840+0000","last_active":"2023-10-19T12:44:23.123329+0000","last_peered":"2023-10-19T12:44:23.123329+0000","last_clean":"2023-10-19T12:44:23.123329+0000","last_became_active":"2023-10-19T12:44:18.071447+0000","last_became_peered":"2023-10-19T12:44:18.071447+0000","last_unstale":"2023-10-19T12:44:23.123329+0000","last_undegraded":"2023-10-19T12:44:23.123329+0000","last_fullsized":"2023-10-19T12:44:23.123329+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:46:38.630648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.258316+0000","last_change":"2023-10-19T12:44:23.263517+0000","last_active":"2023-10-19T12:44:24.258316+0000","last_peered":"2023-10-19T12:44:24.258316+0000","last_clean":"2023-10-19T12:44:24.258316+0000","last_became_active":"2023-10-19T12:44:23.263105+0000","last_became_peered":"2023-10-19T12:44:23.263105+0000","last_unstale":"2023-10-19T12:44:24.258316+0000","last_undegraded":"2023-10-19T12:44:24.258316+0000","last_fullsized":"2023-10-19T12:44:24.258316+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:56:44.960048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.122932+0000","last_change":"2023-10-19T12:44:19.088058+0000","last_active":"2023-10-19T12:44:23.122932+0000","last_peered":"2023-10-19T12:44:23.122932+0000","last_clean":"2023-10-19T12:44:23.122932+0000","last_became_active":"2023-10-19T12:44:19.087755+0000","last_became_peered":"2023-10-19T12:44:19.087755+0000","last_unstale":"2023-10-19T12:44:23.122932+0000","last_undegraded":"2023-10-19T12:44:23.122932+0000","last_fullsized":"2023-10-19T12:44:23.122932+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:54:38.037190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.260168+0000","last_change":"2023-10-19T12:44:18.070041+0000","last_active":"2023-10-19T12:44:23.260168+0000","last_peered":"2023-10-19T12:44:23.260168+0000","last_clean":"2023-10-19T12:44:23.260168+0000","last_became_active":"2023-10-19T12:44:18.069723+0000","last_became_peered":"2023-10-19T12:44:18.069723+0000","last_unstale":"2023-10-19T12:44:23.260168+0000","last_undegraded":"2023-10-19T12:44:23.260168+0000","last_fullsized":"2023-10-19T12:44:23.260168+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:10:07.046736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.258633+0000","last_change":"2023-10-19T12:44:23.260957+0000","last_active":"2023-10-19T12:44:24.258633+0000","last_peered":"2023-10-19T12:44:24.258633+0000","last_clean":"2023-10-19T12:44:24.258633+0000","last_became_active":"2023-10-19T12:44:23.260559+0000","last_became_peered":"2023-10-19T12:44:23.260559+0000","last_unstale":"2023-10-19T12:44:24.258633+0000","last_undegraded":"2023-10-19T12:44:24.258633+0000","last_fullsized":"2023-10-19T12:44:24.258633+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:30:09.977362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.107403+0000","last_change":"2023-10-19T12:44:19.080613+0000","last_active":"2023-10-19T12:44:23.107403+0000","last_peered":"2023-10-19T12:44:23.107403+0000","last_clean":"2023-10-19T12:44:23.107403+0000","last_became_active":"2023-10-19T12:44:19.080260+0000","last_became_peered":"2023-10-19T12:44:19.080260+0000","last_unstale":"2023-10-19T12:44:23.107403+0000","last_undegraded":"2023-10-19T12:44:23.107403+0000","last_fullsized":"2023-10-19T12:44:23.107403+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:49:26.174499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.123051+0000","last_change":"2023-10-19T12:44:18.072668+0000","last_active":"2023-10-19T12:44:23.123051+0000","last_peered":"2023-10-19T12:44:23.123051+0000","last_clean":"2023-10-19T12:44:23.123051+0000","last_became_active":"2023-10-19T12:44:18.072383+0000","last_became_peered":"2023-10-19T12:44:18.072383+0000","last_unstale":"2023-10-19T12:44:23.123051+0000","last_undegraded":"2023-10-19T12:44:23.123051+0000","last_fullsized":"2023-10-19T12:44:23.123051+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:07:40.511877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.124753+0000","last_change":"2023-10-19T12:44:23.130828+0000","last_active":"2023-10-19T12:44:24.124753+0000","last_peered":"2023-10-19T12:44:24.124753+0000","last_clean":"2023-10-19T12:44:24.124753+0000","last_became_active":"2023-10-19T12:44:23.130454+0000","last_became_peered":"2023-10-19T12:44:23.130454+0000","last_unstale":"2023-10-19T12:44:24.124753+0000","last_undegraded":"2023-10-19T12:44:24.124753+0000","last_fullsized":"2023-10-19T12:44:24.124753+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:30:09.977362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.114746+0000","last_change":"2023-10-19T12:44:19.075066+0000","last_active":"2023-10-19T12:44:23.114746+0000","last_peered":"2023-10-19T12:44:23.114746+0000","last_clean":"2023-10-19T12:44:23.114746+0000","last_became_active":"2023-10-19T12:44:19.074770+0000","last_became_peered":"2023-10-19T12:44:19.074770+0000","last_unstale":"2023-10-19T12:44:23.114746+0000","last_undegraded":"2023-10-19T12:44:23.114746+0000","last_fullsized":"2023-10-19T12:44:23.114746+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:11:22.846295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.106419+0000","last_change":"2023-10-19T12:44:18.072600+0000","last_active":"2023-10-19T12:44:23.106419+0000","last_peered":"2023-10-19T12:44:23.106419+0000","last_clean":"2023-10-19T12:44:23.106419+0000","last_became_active":"2023-10-19T12:44:18.072278+0000","last_became_peered":"2023-10-19T12:44:18.072278+0000","last_unstale":"2023-10-19T12:44:23.106419+0000","last_undegraded":"2023-10-19T12:44:23.106419+0000","last_fullsized":"2023-10-19T12:44:23.106419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:50:41.813466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.125020+0000","last_change":"2023-10-19T12:44:23.132395+0000","last_active":"2023-10-19T12:44:24.125020+0000","last_peered":"2023-10-19T12:44:24.125020+0000","last_clean":"2023-10-19T12:44:24.125020+0000","last_became_active":"2023-10-19T12:44:23.131961+0000","last_became_peered":"2023-10-19T12:44:23.131961+0000","last_unstale":"2023-10-19T12:44:24.125020+0000","last_undegraded":"2023-10-19T12:44:24.125020+0000","last_fullsized":"2023-10-19T12:44:24.125020+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:37:44.936303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.122772+0000","last_change":"2023-10-19T12:44:19.084119+0000","last_active":"2023-10-19T12:44:23.122772+0000","last_peered":"2023-10-19T12:44:23.122772+0000","last_clean":"2023-10-19T12:44:23.122772+0000","last_became_active":"2023-10-19T12:44:19.083803+0000","last_became_peered":"2023-10-19T12:44:19.083803+0000","last_unstale":"2023-10-19T12:44:23.122772+0000","last_undegraded":"2023-10-19T12:44:23.122772+0000","last_fullsized":"2023-10-19T12:44:23.122772+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:01:32.286452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.107952+0000","last_change":"2023-10-19T12:44:18.066014+0000","last_active":"2023-10-19T12:44:23.107952+0000","last_peered":"2023-10-19T12:44:23.107952+0000","last_clean":"2023-10-19T12:44:23.107952+0000","last_became_active":"2023-10-19T12:44:18.065652+0000","last_became_peered":"2023-10-19T12:44:18.065652+0000","last_unstale":"2023-10-19T12:44:23.107952+0000","last_undegraded":"2023-10-19T12:44:23.107952+0000","last_fullsized":"2023-10-19T12:44:23.107952+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:46:36.490219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.125536+0000","last_change":"2023-10-19T12:44:23.135917+0000","last_active":"2023-10-19T12:44:24.125536+0000","last_peered":"2023-10-19T12:44:24.125536+0000","last_clean":"2023-10-19T12:44:24.125536+0000","last_became_active":"2023-10-19T12:44:23.135459+0000","last_became_peered":"2023-10-19T12:44:23.135459+0000","last_unstale":"2023-10-19T12:44:24.125536+0000","last_undegraded":"2023-10-19T12:44:24.125536+0000","last_fullsized":"2023-10-19T12:44:24.125536+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:24:52.469571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.114317+0000","last_change":"2023-10-19T12:44:19.071520+0000","last_active":"2023-10-19T12:44:23.114317+0000","last_peered":"2023-10-19T12:44:23.114317+0000","last_clean":"2023-10-19T12:44:23.114317+0000","last_became_active":"2023-10-19T12:44:19.071008+0000","last_became_peered":"2023-10-19T12:44:19.071008+0000","last_unstale":"2023-10-19T12:44:23.114317+0000","last_undegraded":"2023-10-19T12:44:23.114317+0000","last_fullsized":"2023-10-19T12:44:23.114317+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:27:50.246364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.259411+0000","last_change":"2023-10-19T12:44:18.071276+0000","last_active":"2023-10-19T12:44:23.259411+0000","last_peered":"2023-10-19T12:44:23.259411+0000","last_clean":"2023-10-19T12:44:23.259411+0000","last_became_active":"2023-10-19T12:44:18.069565+0000","last_became_peered":"2023-10-19T12:44:18.069565+0000","last_unstale":"2023-10-19T12:44:23.259411+0000","last_undegraded":"2023-10-19T12:44:23.259411+0000","last_fullsized":"2023-10-19T12:44:23.259411+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:46:38.630648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.259572+0000","last_change":"2023-10-19T12:44:23.261271+0000","last_active":"2023-10-19T12:44:24.259572+0000","last_peered":"2023-10-19T12:44:24.259572+0000","last_clean":"2023-10-19T12:44:24.259572+0000","last_became_active":"2023-10-19T12:44:23.260853+0000","last_became_peered":"2023-10-19T12:44:23.260853+0000","last_unstale":"2023-10-19T12:44:24.259572+0000","last_undegraded":"2023-10-19T12:44:24.259572+0000","last_fullsized":"2023-10-19T12:44:24.259572+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:53:36.534082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.108153+0000","last_change":"2023-10-19T12:44:19.086009+0000","last_active":"2023-10-19T12:44:23.108153+0000","last_peered":"2023-10-19T12:44:23.108153+0000","last_clean":"2023-10-19T12:44:23.108153+0000","last_became_active":"2023-10-19T12:44:19.085688+0000","last_became_peered":"2023-10-19T12:44:19.085688+0000","last_unstale":"2023-10-19T12:44:23.108153+0000","last_undegraded":"2023-10-19T12:44:23.108153+0000","last_fullsized":"2023-10-19T12:44:23.108153+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:00:15.982705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.106386+0000","last_change":"2023-10-19T12:44:18.067051+0000","last_active":"2023-10-19T12:44:23.106386+0000","last_peered":"2023-10-19T12:44:23.106386+0000","last_clean":"2023-10-19T12:44:23.106386+0000","last_became_active":"2023-10-19T12:44:18.066784+0000","last_became_peered":"2023-10-19T12:44:18.066784+0000","last_unstale":"2023-10-19T12:44:23.106386+0000","last_undegraded":"2023-10-19T12:44:23.106386+0000","last_fullsized":"2023-10-19T12:44:23.106386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:43:47.410870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":168,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.259530+0000","last_change":"2023-10-19T12:44:08.636613+0000","last_active":"2023-10-19T12:44:23.259530+0000","last_peered":"2023-10-19T12:44:23.259530+0000","last_clean":"2023-10-19T12:44:23.259530+0000","last_became_active":"2023-10-19T12:44:08.635697+0000","last_became_peered":"2023-10-19T12:44:08.635697+0000","last_unstale":"2023-10-19T12:44:23.259530+0000","last_undegraded":"2023-10-19T12:44:23.259530+0000","last_fullsized":"2023-10-19T12:44:23.259530+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_clean_scrub_stamp":"2023-10-19T12:44:07.399738+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:15:23.685485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.258583+0000","last_change":"2023-10-19T12:44:23.261893+0000","last_active":"2023-10-19T12:44:24.258583+0000","last_peered":"2023-10-19T12:44:24.258583+0000","last_clean":"2023-10-19T12:44:24.258583+0000","last_became_active":"2023-10-19T12:44:23.261440+0000","last_became_peered":"2023-10-19T12:44:23.261440+0000","last_unstale":"2023-10-19T12:44:24.258583+0000","last_undegraded":"2023-10-19T12:44:24.258583+0000","last_fullsized":"2023-10-19T12:44:24.258583+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:30:09.977362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.259893+0000","last_change":"2023-10-19T12:44:19.093167+0000","last_active":"2023-10-19T12:44:23.259893+0000","last_peered":"2023-10-19T12:44:23.259893+0000","last_clean":"2023-10-19T12:44:23.259893+0000","last_became_active":"2023-10-19T12:44:19.092842+0000","last_became_peered":"2023-10-19T12:44:19.092842+0000","last_unstale":"2023-10-19T12:44:23.259893+0000","last_undegraded":"2023-10-19T12:44:23.259893+0000","last_fullsized":"2023-10-19T12:44:23.259893+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:01:08.972450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.108142+0000","last_change":"2023-10-19T12:44:18.071274+0000","last_active":"2023-10-19T12:44:23.108142+0000","last_peered":"2023-10-19T12:44:23.108142+0000","last_clean":"2023-10-19T12:44:23.108142+0000","last_became_active":"2023-10-19T12:44:18.070997+0000","last_became_peered":"2023-10-19T12:44:18.070997+0000","last_unstale":"2023-10-19T12:44:23.108142+0000","last_undegraded":"2023-10-19T12:44:23.108142+0000","last_fullsized":"2023-10-19T12:44:23.108142+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:27:38.896207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.258302+0000","last_change":"2023-10-19T12:44:23.261162+0000","last_active":"2023-10-19T12:44:24.258302+0000","last_peered":"2023-10-19T12:44:24.258302+0000","last_clean":"2023-10-19T12:44:24.258302+0000","last_became_active":"2023-10-19T12:44:23.260701+0000","last_became_peered":"2023-10-19T12:44:23.260701+0000","last_unstale":"2023-10-19T12:44:24.258302+0000","last_undegraded":"2023-10-19T12:44:24.258302+0000","last_fullsized":"2023-10-19T12:44:24.258302+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:01:13.019006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.107087+0000","last_change":"2023-10-19T12:44:19.090016+0000","last_active":"2023-10-19T12:44:23.107087+0000","last_peered":"2023-10-19T12:44:23.107087+0000","last_clean":"2023-10-19T12:44:23.107087+0000","last_became_active":"2023-10-19T12:44:19.086367+0000","last_became_peered":"2023-10-19T12:44:19.086367+0000","last_unstale":"2023-10-19T12:44:23.107087+0000","last_undegraded":"2023-10-19T12:44:23.107087+0000","last_fullsized":"2023-10-19T12:44:23.107087+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:01:08.972450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.123668+0000","last_change":"2023-10-19T12:44:18.069871+0000","last_active":"2023-10-19T12:44:23.123668+0000","last_peered":"2023-10-19T12:44:23.123668+0000","last_clean":"2023-10-19T12:44:23.123668+0000","last_became_active":"2023-10-19T12:44:18.069558+0000","last_became_peered":"2023-10-19T12:44:18.069558+0000","last_unstale":"2023-10-19T12:44:23.123668+0000","last_undegraded":"2023-10-19T12:44:23.123668+0000","last_fullsized":"2023-10-19T12:44:23.123668+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:15:18.575278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.258925+0000","last_change":"2023-10-19T12:44:23.131775+0000","last_active":"2023-10-19T12:44:24.258925+0000","last_peered":"2023-10-19T12:44:24.258925+0000","last_clean":"2023-10-19T12:44:24.258925+0000","last_became_active":"2023-10-19T12:44:23.131215+0000","last_became_peered":"2023-10-19T12:44:23.131215+0000","last_unstale":"2023-10-19T12:44:24.258925+0000","last_undegraded":"2023-10-19T12:44:24.258925+0000","last_fullsized":"2023-10-19T12:44:24.258925+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:37:44.936303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.260452+0000","last_change":"2023-10-19T12:44:19.089818+0000","last_active":"2023-10-19T12:44:23.260452+0000","last_peered":"2023-10-19T12:44:23.260452+0000","last_clean":"2023-10-19T12:44:23.260452+0000","last_became_active":"2023-10-19T12:44:19.089336+0000","last_became_peered":"2023-10-19T12:44:19.089336+0000","last_unstale":"2023-10-19T12:44:23.260452+0000","last_undegraded":"2023-10-19T12:44:23.260452+0000","last_fullsized":"2023-10-19T12:44:23.260452+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:53:32.487525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.108384+0000","last_change":"2023-10-19T12:44:18.068828+0000","last_active":"2023-10-19T12:44:23.108384+0000","last_peered":"2023-10-19T12:44:23.108384+0000","last_clean":"2023-10-19T12:44:23.108384+0000","last_became_active":"2023-10-19T12:44:18.068553+0000","last_became_peered":"2023-10-19T12:44:18.068553+0000","last_unstale":"2023-10-19T12:44:23.108384+0000","last_undegraded":"2023-10-19T12:44:23.108384+0000","last_fullsized":"2023-10-19T12:44:23.108384+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:11:21.842323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.123595+0000","last_change":"2023-10-19T12:44:23.261245+0000","last_active":"2023-10-19T12:44:24.123595+0000","last_peered":"2023-10-19T12:44:24.123595+0000","last_clean":"2023-10-19T12:44:24.123595+0000","last_became_active":"2023-10-19T12:44:23.260818+0000","last_became_peered":"2023-10-19T12:44:23.260818+0000","last_unstale":"2023-10-19T12:44:24.123595+0000","last_undegraded":"2023-10-19T12:44:24.123595+0000","last_fullsized":"2023-10-19T12:44:24.123595+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:55:55.981359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.260228+0000","last_change":"2023-10-19T12:44:19.087785+0000","last_active":"2023-10-19T12:44:23.260228+0000","last_peered":"2023-10-19T12:44:23.260228+0000","last_clean":"2023-10-19T12:44:23.260228+0000","last_became_active":"2023-10-19T12:44:19.087394+0000","last_became_peered":"2023-10-19T12:44:19.087394+0000","last_unstale":"2023-10-19T12:44:23.260228+0000","last_undegraded":"2023-10-19T12:44:23.260228+0000","last_fullsized":"2023-10-19T12:44:23.260228+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:37:40.889746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.115032+0000","last_change":"2023-10-19T12:44:18.068944+0000","last_active":"2023-10-19T12:44:23.115032+0000","last_peered":"2023-10-19T12:44:23.115032+0000","last_clean":"2023-10-19T12:44:23.115032+0000","last_became_active":"2023-10-19T12:44:18.068674+0000","last_became_peered":"2023-10-19T12:44:18.068674+0000","last_unstale":"2023-10-19T12:44:23.115032+0000","last_undegraded":"2023-10-19T12:44:23.115032+0000","last_fullsized":"2023-10-19T12:44:23.115032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:16:09.533399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.123008+0000","last_change":"2023-10-19T12:44:23.131983+0000","last_active":"2023-10-19T12:44:24.123008+0000","last_peered":"2023-10-19T12:44:24.123008+0000","last_clean":"2023-10-19T12:44:24.123008+0000","last_became_active":"2023-10-19T12:44:23.131304+0000","last_became_peered":"2023-10-19T12:44:23.131304+0000","last_unstale":"2023-10-19T12:44:24.123008+0000","last_undegraded":"2023-10-19T12:44:24.123008+0000","last_fullsized":"2023-10-19T12:44:24.123008+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:01:36.333009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.259741+0000","last_change":"2023-10-19T12:44:19.088843+0000","last_active":"2023-10-19T12:44:23.259741+0000","last_peered":"2023-10-19T12:44:23.259741+0000","last_clean":"2023-10-19T12:44:23.259741+0000","last_became_active":"2023-10-19T12:44:19.088578+0000","last_became_peered":"2023-10-19T12:44:19.088578+0000","last_unstale":"2023-10-19T12:44:23.259741+0000","last_undegraded":"2023-10-19T12:44:23.259741+0000","last_fullsized":"2023-10-19T12:44:23.259741+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:46:55.355893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.108073+0000","last_change":"2023-10-19T12:44:18.069371+0000","last_active":"2023-10-19T12:44:23.108073+0000","last_peered":"2023-10-19T12:44:23.108073+0000","last_clean":"2023-10-19T12:44:23.108073+0000","last_became_active":"2023-10-19T12:44:18.069118+0000","last_became_peered":"2023-10-19T12:44:18.069118+0000","last_unstale":"2023-10-19T12:44:23.108073+0000","last_undegraded":"2023-10-19T12:44:23.108073+0000","last_fullsized":"2023-10-19T12:44:23.108073+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:12:58.093890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.259448+0000","last_change":"2023-10-19T12:44:23.126162+0000","last_active":"2023-10-19T12:44:24.259448+0000","last_peered":"2023-10-19T12:44:24.259448+0000","last_clean":"2023-10-19T12:44:24.259448+0000","last_became_active":"2023-10-19T12:44:23.125364+0000","last_became_peered":"2023-10-19T12:44:23.125364+0000","last_unstale":"2023-10-19T12:44:24.259448+0000","last_undegraded":"2023-10-19T12:44:24.259448+0000","last_fullsized":"2023-10-19T12:44:24.259448+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:24:52.469571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.114403+0000","last_change":"2023-10-19T12:44:19.079421+0000","last_active":"2023-10-19T12:44:23.114403+0000","last_peered":"2023-10-19T12:44:23.114403+0000","last_clean":"2023-10-19T12:44:23.114403+0000","last_became_active":"2023-10-19T12:44:19.079151+0000","last_became_peered":"2023-10-19T12:44:19.079151+0000","last_unstale":"2023-10-19T12:44:23.114403+0000","last_undegraded":"2023-10-19T12:44:23.114403+0000","last_fullsized":"2023-10-19T12:44:23.114403+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:22:48.368572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.114676+0000","last_change":"2023-10-19T12:44:18.070653+0000","last_active":"2023-10-19T12:44:23.114676+0000","last_peered":"2023-10-19T12:44:23.114676+0000","last_clean":"2023-10-19T12:44:23.114676+0000","last_became_active":"2023-10-19T12:44:18.070404+0000","last_became_peered":"2023-10-19T12:44:18.070404+0000","last_unstale":"2023-10-19T12:44:23.114676+0000","last_undegraded":"2023-10-19T12:44:23.114676+0000","last_fullsized":"2023-10-19T12:44:23.114676+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:27:38.896207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.259106+0000","last_change":"2023-10-19T12:44:23.262410+0000","last_active":"2023-10-19T12:44:24.259106+0000","last_peered":"2023-10-19T12:44:24.259106+0000","last_clean":"2023-10-19T12:44:24.259106+0000","last_became_active":"2023-10-19T12:44:23.261667+0000","last_became_peered":"2023-10-19T12:44:23.261667+0000","last_unstale":"2023-10-19T12:44:24.259106+0000","last_undegraded":"2023-10-19T12:44:24.259106+0000","last_fullsized":"2023-10-19T12:44:24.259106+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:23:00.780721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.259656+0000","last_change":"2023-10-19T12:44:19.092039+0000","last_active":"2023-10-19T12:44:23.259656+0000","last_peered":"2023-10-19T12:44:23.259656+0000","last_clean":"2023-10-19T12:44:23.259656+0000","last_became_active":"2023-10-19T12:44:19.090421+0000","last_became_peered":"2023-10-19T12:44:19.090421+0000","last_unstale":"2023-10-19T12:44:23.259656+0000","last_undegraded":"2023-10-19T12:44:23.259656+0000","last_fullsized":"2023-10-19T12:44:23.259656+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:49:26.174499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.106406+0000","last_change":"2023-10-19T12:44:18.066260+0000","last_active":"2023-10-19T12:44:23.106406+0000","last_peered":"2023-10-19T12:44:23.106406+0000","last_clean":"2023-10-19T12:44:23.106406+0000","last_became_active":"2023-10-19T12:44:18.065913+0000","last_became_peered":"2023-10-19T12:44:18.065913+0000","last_unstale":"2023-10-19T12:44:23.106406+0000","last_undegraded":"2023-10-19T12:44:23.106406+0000","last_fullsized":"2023-10-19T12:44:23.106406+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:22:47.364601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.260099+0000","last_change":"2023-10-19T12:44:23.261486+0000","last_active":"2023-10-19T12:44:24.260099+0000","last_peered":"2023-10-19T12:44:24.260099+0000","last_clean":"2023-10-19T12:44:24.260099+0000","last_became_active":"2023-10-19T12:44:23.261044+0000","last_became_peered":"2023-10-19T12:44:23.261044+0000","last_unstale":"2023-10-19T12:44:24.260099+0000","last_undegraded":"2023-10-19T12:44:24.260099+0000","last_fullsized":"2023-10-19T12:44:24.260099+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:27:47.598684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.113455+0000","last_change":"2023-10-19T12:44:19.090117+0000","last_active":"2023-10-19T12:44:23.113455+0000","last_peered":"2023-10-19T12:44:23.113455+0000","last_clean":"2023-10-19T12:44:23.113455+0000","last_became_active":"2023-10-19T12:44:19.089829+0000","last_became_peered":"2023-10-19T12:44:19.089829+0000","last_unstale":"2023-10-19T12:44:23.113455+0000","last_undegraded":"2023-10-19T12:44:23.113455+0000","last_fullsized":"2023-10-19T12:44:23.113455+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:07:41.515848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.113821+0000","last_change":"2023-10-19T12:44:18.067372+0000","last_active":"2023-10-19T12:44:23.113821+0000","last_peered":"2023-10-19T12:44:23.113821+0000","last_clean":"2023-10-19T12:44:23.113821+0000","last_became_active":"2023-10-19T12:44:18.066991+0000","last_became_peered":"2023-10-19T12:44:18.066991+0000","last_unstale":"2023-10-19T12:44:23.113821+0000","last_undegraded":"2023-10-19T12:44:23.113821+0000","last_fullsized":"2023-10-19T12:44:23.113821+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:44:58.807931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.124117+0000","last_change":"2023-10-19T12:44:23.262285+0000","last_active":"2023-10-19T12:44:24.124117+0000","last_peered":"2023-10-19T12:44:24.124117+0000","last_clean":"2023-10-19T12:44:24.124117+0000","last_became_active":"2023-10-19T12:44:23.261716+0000","last_became_peered":"2023-10-19T12:44:23.261716+0000","last_unstale":"2023-10-19T12:44:24.124117+0000","last_undegraded":"2023-10-19T12:44:24.124117+0000","last_fullsized":"2023-10-19T12:44:24.124117+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:33:24.282692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.259803+0000","last_change":"2023-10-19T12:44:19.090802+0000","last_active":"2023-10-19T12:44:23.259803+0000","last_peered":"2023-10-19T12:44:23.259803+0000","last_clean":"2023-10-19T12:44:23.259803+0000","last_became_active":"2023-10-19T12:44:19.087107+0000","last_became_peered":"2023-10-19T12:44:19.087107+0000","last_unstale":"2023-10-19T12:44:23.259803+0000","last_undegraded":"2023-10-19T12:44:23.259803+0000","last_fullsized":"2023-10-19T12:44:23.259803+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:59:45.926427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"25'9","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:28.505371+0000","last_change":"2023-10-19T12:44:18.067624+0000","last_active":"2023-10-19T12:44:28.505371+0000","last_peered":"2023-10-19T12:44:28.505371+0000","last_clean":"2023-10-19T12:44:28.505371+0000","last_became_active":"2023-10-19T12:44:18.067343+0000","last_became_peered":"2023-10-19T12:44:18.067343+0000","last_unstale":"2023-10-19T12:44:28.505371+0000","last_undegraded":"2023-10-19T12:44:28.505371+0000","last_fullsized":"2023-10-19T12:44:28.505371+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:43:47.410870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3841,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.125150+0000","last_change":"2023-10-19T12:44:23.126083+0000","last_active":"2023-10-19T12:44:24.125150+0000","last_peered":"2023-10-19T12:44:24.125150+0000","last_clean":"2023-10-19T12:44:24.125150+0000","last_became_active":"2023-10-19T12:44:23.125365+0000","last_became_peered":"2023-10-19T12:44:23.125365+0000","last_unstale":"2023-10-19T12:44:24.125150+0000","last_undegraded":"2023-10-19T12:44:24.125150+0000","last_fullsized":"2023-10-19T12:44:24.125150+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:40:13.396617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.114883+0000","last_change":"2023-10-19T12:44:19.091229+0000","last_active":"2023-10-19T12:44:23.114883+0000","last_peered":"2023-10-19T12:44:23.114883+0000","last_clean":"2023-10-19T12:44:23.114883+0000","last_became_active":"2023-10-19T12:44:19.090912+0000","last_became_peered":"2023-10-19T12:44:19.090912+0000","last_unstale":"2023-10-19T12:44:23.114883+0000","last_undegraded":"2023-10-19T12:44:23.114883+0000","last_fullsized":"2023-10-19T12:44:23.114883+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:46:39.634619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.123572+0000","last_change":"2023-10-19T12:44:18.072463+0000","last_active":"2023-10-19T12:44:23.123572+0000","last_peered":"2023-10-19T12:44:23.123572+0000","last_clean":"2023-10-19T12:44:23.123572+0000","last_became_active":"2023-10-19T12:44:18.072156+0000","last_became_peered":"2023-10-19T12:44:18.072156+0000","last_unstale":"2023-10-19T12:44:23.123572+0000","last_undegraded":"2023-10-19T12:44:23.123572+0000","last_fullsized":"2023-10-19T12:44:23.123572+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:44:58.807931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.124138+0000","last_change":"2023-10-19T12:44:23.130655+0000","last_active":"2023-10-19T12:44:24.124138+0000","last_peered":"2023-10-19T12:44:24.124138+0000","last_clean":"2023-10-19T12:44:24.124138+0000","last_became_active":"2023-10-19T12:44:23.130142+0000","last_became_peered":"2023-10-19T12:44:23.130142+0000","last_unstale":"2023-10-19T12:44:24.124138+0000","last_undegraded":"2023-10-19T12:44:24.124138+0000","last_fullsized":"2023-10-19T12:44:24.124138+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:30:09.977362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.113707+0000","last_change":"2023-10-19T12:44:19.084143+0000","last_active":"2023-10-19T12:44:23.113707+0000","last_peered":"2023-10-19T12:44:23.113707+0000","last_clean":"2023-10-19T12:44:23.113707+0000","last_became_active":"2023-10-19T12:44:19.083824+0000","last_became_peered":"2023-10-19T12:44:19.083824+0000","last_unstale":"2023-10-19T12:44:23.113707+0000","last_undegraded":"2023-10-19T12:44:23.113707+0000","last_fullsized":"2023-10-19T12:44:23.113707+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:01:32.286452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.106975+0000","last_change":"2023-10-19T12:44:18.070592+0000","last_active":"2023-10-19T12:44:23.106975+0000","last_peered":"2023-10-19T12:44:23.106975+0000","last_clean":"2023-10-19T12:44:23.106975+0000","last_became_active":"2023-10-19T12:44:18.070297+0000","last_became_peered":"2023-10-19T12:44:18.070297+0000","last_unstale":"2023-10-19T12:44:23.106975+0000","last_undegraded":"2023-10-19T12:44:23.106975+0000","last_fullsized":"2023-10-19T12:44:23.106975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:24:31.294273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.258600+0000","last_change":"2023-10-19T12:44:23.131766+0000","last_active":"2023-10-19T12:44:24.258600+0000","last_peered":"2023-10-19T12:44:24.258600+0000","last_clean":"2023-10-19T12:44:24.258600+0000","last_became_active":"2023-10-19T12:44:23.131349+0000","last_became_peered":"2023-10-19T12:44:23.131349+0000","last_unstale":"2023-10-19T12:44:24.258600+0000","last_undegraded":"2023-10-19T12:44:24.258600+0000","last_fullsized":"2023-10-19T12:44:24.258600+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:55:55.981359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.106934+0000","last_change":"2023-10-19T12:44:19.089065+0000","last_active":"2023-10-19T12:44:23.106934+0000","last_peered":"2023-10-19T12:44:23.106934+0000","last_clean":"2023-10-19T12:44:23.106934+0000","last_became_active":"2023-10-19T12:44:19.088295+0000","last_became_peered":"2023-10-19T12:44:19.088295+0000","last_unstale":"2023-10-19T12:44:23.106934+0000","last_undegraded":"2023-10-19T12:44:23.106934+0000","last_fullsized":"2023-10-19T12:44:23.106934+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:30:05.930806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.116850+0000","last_change":"2023-10-19T12:44:18.071630+0000","last_active":"2023-10-19T12:44:23.116850+0000","last_peered":"2023-10-19T12:44:23.116850+0000","last_clean":"2023-10-19T12:44:23.116850+0000","last_became_active":"2023-10-19T12:44:18.071374+0000","last_became_peered":"2023-10-19T12:44:18.071374+0000","last_unstale":"2023-10-19T12:44:23.116850+0000","last_undegraded":"2023-10-19T12:44:23.116850+0000","last_fullsized":"2023-10-19T12:44:23.116850+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:11:21.842323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.259022+0000","last_change":"2023-10-19T12:44:23.261083+0000","last_active":"2023-10-19T12:44:24.259022+0000","last_peered":"2023-10-19T12:44:24.259022+0000","last_clean":"2023-10-19T12:44:24.259022+0000","last_became_active":"2023-10-19T12:44:23.260643+0000","last_became_peered":"2023-10-19T12:44:23.260643+0000","last_unstale":"2023-10-19T12:44:24.259022+0000","last_undegraded":"2023-10-19T12:44:24.259022+0000","last_fullsized":"2023-10-19T12:44:24.259022+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:25:26.010997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.259319+0000","last_change":"2023-10-19T12:44:23.263298+0000","last_active":"2023-10-19T12:44:24.259319+0000","last_peered":"2023-10-19T12:44:24.259319+0000","last_clean":"2023-10-19T12:44:24.259319+0000","last_became_active":"2023-10-19T12:44:23.262936+0000","last_became_peered":"2023-10-19T12:44:23.262936+0000","last_unstale":"2023-10-19T12:44:24.259319+0000","last_undegraded":"2023-10-19T12:44:24.259319+0000","last_fullsized":"2023-10-19T12:44:24.259319+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:40:13.396617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.122817+0000","last_change":"2023-10-19T12:44:19.089929+0000","last_active":"2023-10-19T12:44:23.122817+0000","last_peered":"2023-10-19T12:44:23.122817+0000","last_clean":"2023-10-19T12:44:23.122817+0000","last_became_active":"2023-10-19T12:44:19.089601+0000","last_became_peered":"2023-10-19T12:44:19.089601+0000","last_unstale":"2023-10-19T12:44:23.122817+0000","last_undegraded":"2023-10-19T12:44:23.122817+0000","last_fullsized":"2023-10-19T12:44:23.122817+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:22:48.368572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.107187+0000","last_change":"2023-10-19T12:44:18.069653+0000","last_active":"2023-10-19T12:44:23.107187+0000","last_peered":"2023-10-19T12:44:23.107187+0000","last_clean":"2023-10-19T12:44:23.107187+0000","last_became_active":"2023-10-19T12:44:18.068221+0000","last_became_peered":"2023-10-19T12:44:18.068221+0000","last_unstale":"2023-10-19T12:44:23.107187+0000","last_undegraded":"2023-10-19T12:44:23.107187+0000","last_fullsized":"2023-10-19T12:44:23.107187+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:54:37.033219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.258743+0000","last_change":"2023-10-19T12:44:23.131662+0000","last_active":"2023-10-19T12:44:24.258743+0000","last_peered":"2023-10-19T12:44:24.258743+0000","last_clean":"2023-10-19T12:44:24.258743+0000","last_became_active":"2023-10-19T12:44:23.131182+0000","last_became_peered":"2023-10-19T12:44:23.131182+0000","last_unstale":"2023-10-19T12:44:24.258743+0000","last_undegraded":"2023-10-19T12:44:24.258743+0000","last_fullsized":"2023-10-19T12:44:24.258743+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:37:44.936303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.109433+0000","last_change":"2023-10-19T12:44:19.081926+0000","last_active":"2023-10-19T12:44:23.109433+0000","last_peered":"2023-10-19T12:44:23.109433+0000","last_clean":"2023-10-19T12:44:23.109433+0000","last_became_active":"2023-10-19T12:44:19.081619+0000","last_became_peered":"2023-10-19T12:44:19.081619+0000","last_unstale":"2023-10-19T12:44:23.109433+0000","last_undegraded":"2023-10-19T12:44:23.109433+0000","last_fullsized":"2023-10-19T12:44:23.109433+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:22:48.368572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.108872+0000","last_change":"2023-10-19T12:44:18.071670+0000","last_active":"2023-10-19T12:44:23.108872+0000","last_peered":"2023-10-19T12:44:23.108872+0000","last_clean":"2023-10-19T12:44:23.108872+0000","last_became_active":"2023-10-19T12:44:18.071370+0000","last_became_peered":"2023-10-19T12:44:18.071370+0000","last_unstale":"2023-10-19T12:44:23.108872+0000","last_undegraded":"2023-10-19T12:44:23.108872+0000","last_fullsized":"2023-10-19T12:44:23.108872+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:15:18.575278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.116496+0000","last_change":"2023-10-19T12:44:18.068200+0000","last_active":"2023-10-19T12:44:23.116496+0000","last_peered":"2023-10-19T12:44:23.116496+0000","last_clean":"2023-10-19T12:44:23.116496+0000","last_became_active":"2023-10-19T12:44:18.067903+0000","last_became_peered":"2023-10-19T12:44:18.067903+0000","last_unstale":"2023-10-19T12:44:23.116496+0000","last_undegraded":"2023-10-19T12:44:23.116496+0000","last_fullsized":"2023-10-19T12:44:23.116496+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:27:49.242393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.113730+0000","last_change":"2023-10-19T12:44:19.073697+0000","last_active":"2023-10-19T12:44:23.113730+0000","last_peered":"2023-10-19T12:44:23.113730+0000","last_clean":"2023-10-19T12:44:23.113730+0000","last_became_active":"2023-10-19T12:44:19.073366+0000","last_became_peered":"2023-10-19T12:44:19.073366+0000","last_unstale":"2023-10-19T12:44:23.113730+0000","last_undegraded":"2023-10-19T12:44:23.113730+0000","last_fullsized":"2023-10-19T12:44:23.113730+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:46:55.355893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.258799+0000","last_change":"2023-10-19T12:44:23.131575+0000","last_active":"2023-10-19T12:44:24.258799+0000","last_peered":"2023-10-19T12:44:24.258799+0000","last_clean":"2023-10-19T12:44:24.258799+0000","last_became_active":"2023-10-19T12:44:23.131125+0000","last_became_peered":"2023-10-19T12:44:23.131125+0000","last_unstale":"2023-10-19T12:44:24.258799+0000","last_undegraded":"2023-10-19T12:44:24.258799+0000","last_fullsized":"2023-10-19T12:44:24.258799+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:15:23.625806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.108702+0000","last_change":"2023-10-19T12:44:18.071298+0000","last_active":"2023-10-19T12:44:23.108702+0000","last_peered":"2023-10-19T12:44:23.108702+0000","last_clean":"2023-10-19T12:44:23.108702+0000","last_became_active":"2023-10-19T12:44:18.070913+0000","last_became_peered":"2023-10-19T12:44:18.070913+0000","last_unstale":"2023-10-19T12:44:23.108702+0000","last_undegraded":"2023-10-19T12:44:23.108702+0000","last_fullsized":"2023-10-19T12:44:23.108702+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:27:49.242393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.115487+0000","last_change":"2023-10-19T12:44:19.085944+0000","last_active":"2023-10-19T12:44:23.115487+0000","last_peered":"2023-10-19T12:44:23.115487+0000","last_clean":"2023-10-19T12:44:23.115487+0000","last_became_active":"2023-10-19T12:44:19.085621+0000","last_became_peered":"2023-10-19T12:44:19.085621+0000","last_unstale":"2023-10-19T12:44:23.115487+0000","last_undegraded":"2023-10-19T12:44:23.115487+0000","last_fullsized":"2023-10-19T12:44:23.115487+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:12:59.097861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.260011+0000","last_change":"2023-10-19T12:44:23.126882+0000","last_active":"2023-10-19T12:44:24.260011+0000","last_peered":"2023-10-19T12:44:24.260011+0000","last_clean":"2023-10-19T12:44:24.260011+0000","last_became_active":"2023-10-19T12:44:23.126421+0000","last_became_peered":"2023-10-19T12:44:23.126421+0000","last_unstale":"2023-10-19T12:44:24.260011+0000","last_undegraded":"2023-10-19T12:44:24.260011+0000","last_fullsized":"2023-10-19T12:44:24.260011+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:32:54.650601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.114291+0000","last_change":"2023-10-19T12:44:19.080194+0000","last_active":"2023-10-19T12:44:23.114291+0000","last_peered":"2023-10-19T12:44:23.114291+0000","last_clean":"2023-10-19T12:44:23.114291+0000","last_became_active":"2023-10-19T12:44:19.079941+0000","last_became_peered":"2023-10-19T12:44:19.079941+0000","last_unstale":"2023-10-19T12:44:23.114291+0000","last_undegraded":"2023-10-19T12:44:23.114291+0000","last_fullsized":"2023-10-19T12:44:23.114291+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:50:42.817437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.107859+0000","last_change":"2023-10-19T12:44:19.098110+0000","last_active":"2023-10-19T12:44:23.107859+0000","last_peered":"2023-10-19T12:44:23.107859+0000","last_clean":"2023-10-19T12:44:23.107859+0000","last_became_active":"2023-10-19T12:44:19.097774+0000","last_became_peered":"2023-10-19T12:44:19.097774+0000","last_unstale":"2023-10-19T12:44:23.107859+0000","last_undegraded":"2023-10-19T12:44:23.107859+0000","last_fullsized":"2023-10-19T12:44:23.107859+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:24:32.298244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.106333+0000","last_change":"2023-10-19T12:44:19.083631+0000","last_active":"2023-10-19T12:44:23.106333+0000","last_peered":"2023-10-19T12:44:23.106333+0000","last_clean":"2023-10-19T12:44:23.106333+0000","last_became_active":"2023-10-19T12:44:19.082464+0000","last_became_peered":"2023-10-19T12:44:19.082464+0000","last_unstale":"2023-10-19T12:44:23.106333+0000","last_undegraded":"2023-10-19T12:44:23.106333+0000","last_fullsized":"2023-10-19T12:44:23.106333+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:40:09.350060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.113640+0000","last_change":"2023-10-19T12:44:19.083390+0000","last_active":"2023-10-19T12:44:23.113640+0000","last_peered":"2023-10-19T12:44:23.113640+0000","last_clean":"2023-10-19T12:44:23.113640+0000","last_became_active":"2023-10-19T12:44:19.082423+0000","last_became_peered":"2023-10-19T12:44:19.082423+0000","last_unstale":"2023-10-19T12:44:23.113640+0000","last_undegraded":"2023-10-19T12:44:23.113640+0000","last_fullsized":"2023-10-19T12:44:23.113640+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:54:38.037190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.110543+0000","last_change":"2023-10-19T12:44:19.090250+0000","last_active":"2023-10-19T12:44:23.110543+0000","last_peered":"2023-10-19T12:44:23.110543+0000","last_clean":"2023-10-19T12:44:23.110543+0000","last_became_active":"2023-10-19T12:44:19.089587+0000","last_became_peered":"2023-10-19T12:44:19.089587+0000","last_unstale":"2023-10-19T12:44:23.110543+0000","last_undegraded":"2023-10-19T12:44:23.110543+0000","last_fullsized":"2023-10-19T12:44:23.110543+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:46:55.355893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.122832+0000","last_change":"2023-10-19T12:44:19.088635+0000","last_active":"2023-10-19T12:44:23.122832+0000","last_peered":"2023-10-19T12:44:23.122832+0000","last_clean":"2023-10-19T12:44:23.122832+0000","last_became_active":"2023-10-19T12:44:19.088378+0000","last_became_peered":"2023-10-19T12:44:19.088378+0000","last_unstale":"2023-10-19T12:44:23.122832+0000","last_undegraded":"2023-10-19T12:44:23.122832+0000","last_fullsized":"2023-10-19T12:44:23.122832+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:00:15.982705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.115903+0000","last_change":"2023-10-19T12:44:19.089688+0000","last_active":"2023-10-19T12:44:23.115903+0000","last_peered":"2023-10-19T12:44:23.115903+0000","last_clean":"2023-10-19T12:44:23.115903+0000","last_became_active":"2023-10-19T12:44:19.089438+0000","last_became_peered":"2023-10-19T12:44:19.089438+0000","last_unstale":"2023-10-19T12:44:23.115903+0000","last_undegraded":"2023-10-19T12:44:23.115903+0000","last_fullsized":"2023-10-19T12:44:23.115903+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:46:55.355893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.107774+0000","last_change":"2023-10-19T12:44:19.087749+0000","last_active":"2023-10-19T12:44:23.107774+0000","last_peered":"2023-10-19T12:44:23.107774+0000","last_clean":"2023-10-19T12:44:23.107774+0000","last_became_active":"2023-10-19T12:44:19.087020+0000","last_became_peered":"2023-10-19T12:44:19.087020+0000","last_unstale":"2023-10-19T12:44:23.107774+0000","last_undegraded":"2023-10-19T12:44:23.107774+0000","last_fullsized":"2023-10-19T12:44:23.107774+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:05:15.835724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.106909+0000","last_change":"2023-10-19T12:44:19.085297+0000","last_active":"2023-10-19T12:44:23.106909+0000","last_peered":"2023-10-19T12:44:23.106909+0000","last_clean":"2023-10-19T12:44:23.106909+0000","last_became_active":"2023-10-19T12:44:19.084652+0000","last_became_peered":"2023-10-19T12:44:19.084652+0000","last_unstale":"2023-10-19T12:44:23.106909+0000","last_undegraded":"2023-10-19T12:44:23.106909+0000","last_fullsized":"2023-10-19T12:44:23.106909+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:05:15.835724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.108790+0000","last_change":"2023-10-19T12:44:19.084748+0000","last_active":"2023-10-19T12:44:23.108790+0000","last_peered":"2023-10-19T12:44:23.108790+0000","last_clean":"2023-10-19T12:44:23.108790+0000","last_became_active":"2023-10-19T12:44:19.084460+0000","last_became_peered":"2023-10-19T12:44:19.084460+0000","last_unstale":"2023-10-19T12:44:23.108790+0000","last_undegraded":"2023-10-19T12:44:23.108790+0000","last_fullsized":"2023-10-19T12:44:23.108790+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:54:38.037190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.106299+0000","last_change":"2023-10-19T12:44:19.087125+0000","last_active":"2023-10-19T12:44:23.106299+0000","last_peered":"2023-10-19T12:44:23.106299+0000","last_clean":"2023-10-19T12:44:23.106299+0000","last_became_active":"2023-10-19T12:44:19.086845+0000","last_became_peered":"2023-10-19T12:44:19.086845+0000","last_unstale":"2023-10-19T12:44:23.106299+0000","last_undegraded":"2023-10-19T12:44:23.106299+0000","last_fullsized":"2023-10-19T12:44:23.106299+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:37:40.889746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.113534+0000","last_change":"2023-10-19T12:44:19.089064+0000","last_active":"2023-10-19T12:44:23.113534+0000","last_peered":"2023-10-19T12:44:23.113534+0000","last_clean":"2023-10-19T12:44:23.113534+0000","last_became_active":"2023-10-19T12:44:19.088780+0000","last_became_peered":"2023-10-19T12:44:19.088780+0000","last_unstale":"2023-10-19T12:44:23.113534+0000","last_undegraded":"2023-10-19T12:44:23.113534+0000","last_fullsized":"2023-10-19T12:44:23.113534+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:10:08.050707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.115561+0000","last_change":"2023-10-19T12:44:19.087124+0000","last_active":"2023-10-19T12:44:23.115561+0000","last_peered":"2023-10-19T12:44:23.115561+0000","last_clean":"2023-10-19T12:44:23.115561+0000","last_became_active":"2023-10-19T12:44:19.086851+0000","last_became_peered":"2023-10-19T12:44:19.086851+0000","last_unstale":"2023-10-19T12:44:23.115561+0000","last_undegraded":"2023-10-19T12:44:23.115561+0000","last_fullsized":"2023-10-19T12:44:23.115561+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:10:08.050707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.260204+0000","last_change":"2023-10-19T12:44:19.091368+0000","last_active":"2023-10-19T12:44:23.260204+0000","last_peered":"2023-10-19T12:44:23.260204+0000","last_clean":"2023-10-19T12:44:23.260204+0000","last_became_active":"2023-10-19T12:44:19.091052+0000","last_became_peered":"2023-10-19T12:44:19.091052+0000","last_unstale":"2023-10-19T12:44:23.260204+0000","last_undegraded":"2023-10-19T12:44:23.260204+0000","last_fullsized":"2023-10-19T12:44:23.260204+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:40:09.350060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.108300+0000","last_change":"2023-10-19T12:44:19.080430+0000","last_active":"2023-10-19T12:44:23.108300+0000","last_peered":"2023-10-19T12:44:23.108300+0000","last_clean":"2023-10-19T12:44:23.108300+0000","last_became_active":"2023-10-19T12:44:19.079493+0000","last_became_peered":"2023-10-19T12:44:19.079493+0000","last_unstale":"2023-10-19T12:44:23.108300+0000","last_undegraded":"2023-10-19T12:44:23.108300+0000","last_fullsized":"2023-10-19T12:44:23.108300+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:46:39.634619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.107179+0000","last_change":"2023-10-19T12:44:19.088751+0000","last_active":"2023-10-19T12:44:23.107179+0000","last_peered":"2023-10-19T12:44:23.107179+0000","last_clean":"2023-10-19T12:44:23.107179+0000","last_became_active":"2023-10-19T12:44:19.088430+0000","last_became_peered":"2023-10-19T12:44:19.088430+0000","last_unstale":"2023-10-19T12:44:23.107179+0000","last_undegraded":"2023-10-19T12:44:23.107179+0000","last_fullsized":"2023-10-19T12:44:23.107179+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:24:48.423014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.106786+0000","last_change":"2023-10-19T12:44:19.082935+0000","last_active":"2023-10-19T12:44:23.106786+0000","last_peered":"2023-10-19T12:44:23.106786+0000","last_clean":"2023-10-19T12:44:23.106786+0000","last_became_active":"2023-10-19T12:44:19.082692+0000","last_became_peered":"2023-10-19T12:44:19.082692+0000","last_unstale":"2023-10-19T12:44:23.106786+0000","last_undegraded":"2023-10-19T12:44:23.106786+0000","last_fullsized":"2023-10-19T12:44:23.106786+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:10:08.050707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.122913+0000","last_change":"2023-10-19T12:44:19.087441+0000","last_active":"2023-10-19T12:44:23.122913+0000","last_peered":"2023-10-19T12:44:23.122913+0000","last_clean":"2023-10-19T12:44:23.122913+0000","last_became_active":"2023-10-19T12:44:19.087147+0000","last_became_peered":"2023-10-19T12:44:19.087147+0000","last_unstale":"2023-10-19T12:44:23.122913+0000","last_undegraded":"2023-10-19T12:44:23.122913+0000","last_fullsized":"2023-10-19T12:44:23.122913+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:50:42.817437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.113370+0000","last_change":"2023-10-19T12:44:19.085136+0000","last_active":"2023-10-19T12:44:23.113370+0000","last_peered":"2023-10-19T12:44:23.113370+0000","last_clean":"2023-10-19T12:44:23.113370+0000","last_became_active":"2023-10-19T12:44:19.084856+0000","last_became_peered":"2023-10-19T12:44:19.084856+0000","last_unstale":"2023-10-19T12:44:23.113370+0000","last_undegraded":"2023-10-19T12:44:23.113370+0000","last_fullsized":"2023-10-19T12:44:23.113370+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:46:30.889830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.109109+0000","last_change":"2023-10-19T12:44:19.080419+0000","last_active":"2023-10-19T12:44:23.109109+0000","last_peered":"2023-10-19T12:44:23.109109+0000","last_clean":"2023-10-19T12:44:23.109109+0000","last_became_active":"2023-10-19T12:44:19.080123+0000","last_became_peered":"2023-10-19T12:44:19.080123+0000","last_unstale":"2023-10-19T12:44:23.109109+0000","last_undegraded":"2023-10-19T12:44:23.109109+0000","last_fullsized":"2023-10-19T12:44:23.109109+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:10:08.050707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.114806+0000","last_change":"2023-10-19T12:44:19.080702+0000","last_active":"2023-10-19T12:44:23.114806+0000","last_peered":"2023-10-19T12:44:23.114806+0000","last_clean":"2023-10-19T12:44:23.114806+0000","last_became_active":"2023-10-19T12:44:19.079347+0000","last_became_peered":"2023-10-19T12:44:19.079347+0000","last_unstale":"2023-10-19T12:44:23.114806+0000","last_undegraded":"2023-10-19T12:44:23.114806+0000","last_fullsized":"2023-10-19T12:44:23.114806+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:43:48.414841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.115372+0000","last_change":"2023-10-19T12:44:19.081326+0000","last_active":"2023-10-19T12:44:23.115372+0000","last_peered":"2023-10-19T12:44:23.115372+0000","last_clean":"2023-10-19T12:44:23.115372+0000","last_became_active":"2023-10-19T12:44:19.081063+0000","last_became_peered":"2023-10-19T12:44:19.081063+0000","last_unstale":"2023-10-19T12:44:23.115372+0000","last_undegraded":"2023-10-19T12:44:23.115372+0000","last_fullsized":"2023-10-19T12:44:23.115372+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:00:15.982705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.115814+0000","last_change":"2023-10-19T12:44:19.094478+0000","last_active":"2023-10-19T12:44:23.115814+0000","last_peered":"2023-10-19T12:44:23.115814+0000","last_clean":"2023-10-19T12:44:23.115814+0000","last_became_active":"2023-10-19T12:44:19.094217+0000","last_became_peered":"2023-10-19T12:44:19.094217+0000","last_unstale":"2023-10-19T12:44:23.115814+0000","last_undegraded":"2023-10-19T12:44:23.115814+0000","last_fullsized":"2023-10-19T12:44:23.115814+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:01:32.286452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.260360+0000","last_change":"2023-10-19T12:44:19.093197+0000","last_active":"2023-10-19T12:44:23.260360+0000","last_peered":"2023-10-19T12:44:23.260360+0000","last_clean":"2023-10-19T12:44:23.260360+0000","last_became_active":"2023-10-19T12:44:19.092926+0000","last_became_peered":"2023-10-19T12:44:19.092926+0000","last_unstale":"2023-10-19T12:44:23.260360+0000","last_undegraded":"2023-10-19T12:44:23.260360+0000","last_fullsized":"2023-10-19T12:44:23.260360+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:30:05.930806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.109339+0000","last_change":"2023-10-19T12:44:19.095093+0000","last_active":"2023-10-19T12:44:23.109339+0000","last_peered":"2023-10-19T12:44:23.109339+0000","last_clean":"2023-10-19T12:44:23.109339+0000","last_became_active":"2023-10-19T12:44:19.094816+0000","last_became_peered":"2023-10-19T12:44:19.094816+0000","last_unstale":"2023-10-19T12:44:23.109339+0000","last_undegraded":"2023-10-19T12:44:23.109339+0000","last_fullsized":"2023-10-19T12:44:23.109339+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:07:41.515848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.259584+0000","last_change":"2023-10-19T12:44:19.086048+0000","last_active":"2023-10-19T12:44:23.259584+0000","last_peered":"2023-10-19T12:44:23.259584+0000","last_clean":"2023-10-19T12:44:23.259584+0000","last_became_active":"2023-10-19T12:44:19.085086+0000","last_became_peered":"2023-10-19T12:44:19.085086+0000","last_unstale":"2023-10-19T12:44:23.259584+0000","last_undegraded":"2023-10-19T12:44:23.259584+0000","last_fullsized":"2023-10-19T12:44:23.259584+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:05:15.835724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.106214+0000","last_change":"2023-10-19T12:44:19.086341+0000","last_active":"2023-10-19T12:44:23.106214+0000","last_peered":"2023-10-19T12:44:23.106214+0000","last_clean":"2023-10-19T12:44:23.106214+0000","last_became_active":"2023-10-19T12:44:19.083365+0000","last_became_peered":"2023-10-19T12:44:19.083365+0000","last_unstale":"2023-10-19T12:44:23.106214+0000","last_undegraded":"2023-10-19T12:44:23.106214+0000","last_fullsized":"2023-10-19T12:44:23.106214+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:46:55.355893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.259588+0000","last_change":"2023-10-19T12:44:19.091228+0000","last_active":"2023-10-19T12:44:23.259588+0000","last_peered":"2023-10-19T12:44:23.259588+0000","last_clean":"2023-10-19T12:44:23.259588+0000","last_became_active":"2023-10-19T12:44:19.087203+0000","last_became_peered":"2023-10-19T12:44:19.087203+0000","last_unstale":"2023-10-19T12:44:23.259588+0000","last_undegraded":"2023-10-19T12:44:23.259588+0000","last_fullsized":"2023-10-19T12:44:23.259588+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:29:52.686703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.106856+0000","last_change":"2023-10-19T12:44:19.086112+0000","last_active":"2023-10-19T12:44:23.106856+0000","last_peered":"2023-10-19T12:44:23.106856+0000","last_clean":"2023-10-19T12:44:23.106856+0000","last_became_active":"2023-10-19T12:44:19.085822+0000","last_became_peered":"2023-10-19T12:44:19.085822+0000","last_unstale":"2023-10-19T12:44:23.106856+0000","last_undegraded":"2023-10-19T12:44:23.106856+0000","last_fullsized":"2023-10-19T12:44:23.106856+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:01:32.286452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.113429+0000","last_change":"2023-10-19T12:44:19.078205+0000","last_active":"2023-10-19T12:44:23.113429+0000","last_peered":"2023-10-19T12:44:23.113429+0000","last_clean":"2023-10-19T12:44:23.113429+0000","last_became_active":"2023-10-19T12:44:19.077824+0000","last_became_peered":"2023-10-19T12:44:19.077824+0000","last_unstale":"2023-10-19T12:44:23.113429+0000","last_undegraded":"2023-10-19T12:44:23.113429+0000","last_fullsized":"2023-10-19T12:44:23.113429+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:50:42.817437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.123225+0000","last_change":"2023-10-19T12:44:19.084212+0000","last_active":"2023-10-19T12:44:23.123225+0000","last_peered":"2023-10-19T12:44:23.123225+0000","last_clean":"2023-10-19T12:44:23.123225+0000","last_became_active":"2023-10-19T12:44:19.083928+0000","last_became_peered":"2023-10-19T12:44:19.083928+0000","last_unstale":"2023-10-19T12:44:23.123225+0000","last_undegraded":"2023-10-19T12:44:23.123225+0000","last_fullsized":"2023-10-19T12:44:23.123225+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:10:08.050707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.260753+0000","last_change":"2023-10-19T12:44:19.087938+0000","last_active":"2023-10-19T12:44:23.260753+0000","last_peered":"2023-10-19T12:44:23.260753+0000","last_clean":"2023-10-19T12:44:23.260753+0000","last_became_active":"2023-10-19T12:44:19.087552+0000","last_became_peered":"2023-10-19T12:44:19.087552+0000","last_unstale":"2023-10-19T12:44:23.260753+0000","last_undegraded":"2023-10-19T12:44:23.260753+0000","last_fullsized":"2023-10-19T12:44:23.260753+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:24:48.423014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4639,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574857,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27868,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343972,"statfs":{"total":96636764160,"available":96608227328,"internally_reserved":0,"allocated":864256,"data_stored":695401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55100000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.497}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52500000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38200000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60699999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45400000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":108292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66100000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.622}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51700000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67700000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42099999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39900000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574857,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":270336,"data_stored":104541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74399999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68799999999999994}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60699999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56000000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79300000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]}]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":278528,"data_stored":104577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66700000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76900000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73799999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53800000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52100000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44500000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64100000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31448,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340392,"statfs":{"total":96636764160,"available":96604561408,"internally_reserved":0,"allocated":278528,"data_stored":108292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36099999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39500000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47799999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61599999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83199999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85199999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71899999999999997}]}]},{"osd":2,"up_from":13,"seq":55834574858,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31448,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340392,"statfs":{"total":96636764160,"available":96604561408,"internally_reserved":0,"allocated":266240,"data_stored":104451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34300000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30299999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66000000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.504}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59499999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56399999999999995}]}]},{"osd":1,"up_from":13,"seq":55834574857,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31460,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340380,"statfs":{"total":96636764160,"available":96604549120,"internally_reserved":0,"allocated":274432,"data_stored":104611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35999999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41599999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47099999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.873}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82399999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74299999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84199999999999997}]}]},{"osd":0,"up_from":13,"seq":55834574858,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32032,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339808,"statfs":{"total":96636764160,"available":96603963392,"internally_reserved":0,"allocated":868352,"data_stored":694838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34799999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.379}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57899999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45500000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53200000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-19T12:44:42.199 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-19T12:44:42.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.337+0000 7f7f22ee5640 1 -- 172.21.15.57:0/2389404380 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7f0c004d00 msgr2=0x7f7f0c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:42.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.337+0000 7f7f22ee5640 1 --2- 172.21.15.57:0/2389404380 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7f0c004d00 0x7f7f0c005180 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f7f14009ed0 tx=0x7f7f0c0057f0 comp rx=0 tx=0).stop 2023-10-19T12:44:42.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.337+0000 7f7f21482640 1 -- 172.21.15.57:0/2389404380 reap_dead start 2023-10-19T12:44:42.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.337+0000 7f7f22ee5640 1 -- 172.21.15.57:0/2389404380 shutdown_connections 2023-10-19T12:44:42.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.337+0000 7f7f22ee5640 1 -- 172.21.15.57:0/2389404380 >> 172.21.15.57:0/2389404380 conn(0x7f7f1c069930 msgr2=0x7f7f1c069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:42.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.337+0000 7f7f22ee5640 1 -- 172.21.15.57:0/2389404380 shutdown_connections 2023-10-19T12:44:42.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.337+0000 7f7f22ee5640 1 -- 172.21.15.57:0/2389404380 wait complete. 2023-10-19T12:44:42.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.338+0000 7f7f22ee5640 1 Processor -- start 2023-10-19T12:44:42.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.339+0000 7f7f22ee5640 1 -- start start 2023-10-19T12:44:42.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.339+0000 7f7f22ee5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7f0c004d00 0x7f7f1c070b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:42.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.339+0000 7f7f22ee5640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7f1c0b18b0 0x7f7f1c13c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:42.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.339+0000 7f7f22ee5640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7f1c0aa8c0 0x7f7f1c0aad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:42.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.339+0000 7f7f22ee5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f1c0ab280 con 0x7f7f0c004d00 2023-10-19T12:44:42.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.339+0000 7f7f22ee5640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f1c06f080 con 0x7f7f1c0aa8c0 2023-10-19T12:44:42.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.339+0000 7f7f22ee5640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f1c13c660 con 0x7f7f1c0b18b0 2023-10-19T12:44:42.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.340+0000 7f7f20c81640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7f0c004d00 0x7f7f1c070b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:42.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.340+0000 7f7f20c81640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7f0c004d00 0x7f7f1c070b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50514/0 (socket says 172.21.15.57:50514) 2023-10-19T12:44:42.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.340+0000 7f7f20c81640 1 -- 172.21.15.57:0/198567213 learned_addr learned my addr 172.21.15.57:0/198567213 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:42.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.340+0000 7f7f1bfff640 1 --2- 172.21.15.57:0/198567213 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7f1c0b18b0 0x7f7f1c13c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:42.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.340+0000 7f7f21482640 1 --2- 172.21.15.57:0/198567213 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7f1c0aa8c0 0x7f7f1c0aad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:42.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.340+0000 7f7f20c81640 1 -- 172.21.15.57:0/198567213 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7f1c0b18b0 msgr2=0x7f7f1c13c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:42.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.340+0000 7f7f20c81640 1 --2- 172.21.15.57:0/198567213 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7f1c0b18b0 0x7f7f1c13c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:42.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.340+0000 7f7f20c81640 1 -- 172.21.15.57:0/198567213 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7f1c0aa8c0 msgr2=0x7f7f1c0aad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:42.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.340+0000 7f7f20c81640 1 --2- 172.21.15.57:0/198567213 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7f1c0aa8c0 0x7f7f1c0aad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:42.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.340+0000 7f7f20c81640 1 -- 172.21.15.57:0/198567213 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f1c070090 con 0x7f7f0c004d00 2023-10-19T12:44:42.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.340+0000 7f7f21482640 1 --2- 172.21.15.57:0/198567213 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7f1c0aa8c0 0x7f7f1c0aad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:44:42.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.340+0000 7f7f20c81640 1 --2- 172.21.15.57:0/198567213 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7f0c004d00 0x7f7f1c070b00 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f7f10011b40 tx=0x7f7f10009e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:42.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.340+0000 7f7f197fa640 1 -- 172.21.15.57:0/198567213 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f1000fe70 con 0x7f7f0c004d00 2023-10-19T12:44:42.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.340+0000 7f7f197fa640 1 -- 172.21.15.57:0/198567213 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f7f10004050 con 0x7f7f0c004d00 2023-10-19T12:44:42.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.340+0000 7f7f22ee5640 1 -- 172.21.15.57:0/198567213 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f1c06f2d0 con 0x7f7f0c004d00 2023-10-19T12:44:42.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.340+0000 7f7f197fa640 1 -- 172.21.15.57:0/198567213 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f10004460 con 0x7f7f0c004d00 2023-10-19T12:44:42.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.340+0000 7f7f22ee5640 1 -- 172.21.15.57:0/198567213 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f1c06f890 con 0x7f7f0c004d00 2023-10-19T12:44:42.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.341+0000 7f7f22ee5640 1 -- 172.21.15.57:0/198567213 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f1c06c7e0 con 0x7f7f0c004d00 2023-10-19T12:44:42.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.342+0000 7f7f197fa640 1 -- 172.21.15.57:0/198567213 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f7f100035f0 con 0x7f7f0c004d00 2023-10-19T12:44:42.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.342+0000 7f7f197fa640 1 --2- 172.21.15.57:0/198567213 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f7f0006ae40 0x7f7f0006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:42.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.342+0000 7f7f197fa640 1 -- 172.21.15.57:0/198567213 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f7f1008a620 con 0x7f7f0c004d00 2023-10-19T12:44:42.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.342+0000 7f7f1bfff640 1 --2- 172.21.15.57:0/198567213 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f7f0006ae40 0x7f7f0006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:42.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.343+0000 7f7f1bfff640 1 --2- 172.21.15.57:0/198567213 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f7f0006ae40 0x7f7f0006d300 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f7f080017a0 tx=0x7f7f08001550 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:42.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.346+0000 7f7f197fa640 1 -- 172.21.15.57:0/198567213 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7f10055550 con 0x7f7f0c004d00 2023-10-19T12:44:42.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.517+0000 7f7f22ee5640 1 -- 172.21.15.57:0/198567213 --> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7f1c159c50 con 0x7f7f0006ae40 2023-10-19T12:44:42.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.526+0000 7f7f197fa640 1 -- 172.21.15.57:0/198567213 <== mgr.4100 v2:172.21.15.179:6832/2609646285 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+345304 (secure 0 0 0) 0x7f7f08016070 con 0x7f7f0006ae40 2023-10-19T12:44:42.533 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:44:42.535 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2023-10-19T12:44:42.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.529+0000 7f7f22ee5640 1 -- 172.21.15.57:0/198567213 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f7f0006ae40 msgr2=0x7f7f0006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:42.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.529+0000 7f7f22ee5640 1 --2- 172.21.15.57:0/198567213 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f7f0006ae40 0x7f7f0006d300 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f7f080017a0 tx=0x7f7f08001550 comp rx=0 tx=0).stop 2023-10-19T12:44:42.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.529+0000 7f7f22ee5640 1 -- 172.21.15.57:0/198567213 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7f0c004d00 msgr2=0x7f7f1c070b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:42.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.529+0000 7f7f22ee5640 1 --2- 172.21.15.57:0/198567213 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7f0c004d00 0x7f7f1c070b00 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f7f10011b40 tx=0x7f7f10009e90 comp rx=0 tx=0).stop 2023-10-19T12:44:42.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.529+0000 7f7f22ee5640 1 -- 172.21.15.57:0/198567213 shutdown_connections 2023-10-19T12:44:42.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.529+0000 7f7f22ee5640 1 --2- 172.21.15.57:0/198567213 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f7f0006ae40 0x7f7f0006d300 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:42.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.529+0000 7f7f22ee5640 1 --2- 172.21.15.57:0/198567213 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7f1c0aa8c0 0x7f7f1c0aad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:42.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.529+0000 7f7f22ee5640 1 --2- 172.21.15.57:0/198567213 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7f1c0b18b0 0x7f7f1c13c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:42.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.529+0000 7f7f22ee5640 1 --2- 172.21.15.57:0/198567213 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7f0c004d00 0x7f7f1c070b00 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:42.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.529+0000 7f7f22ee5640 1 -- 172.21.15.57:0/198567213 >> 172.21.15.57:0/198567213 conn(0x7f7f1c069930 msgr2=0x7f7f1c15bc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:42.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.530+0000 7f7f22ee5640 1 -- 172.21.15.57:0/198567213 shutdown_connections 2023-10-19T12:44:42.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.530+0000 7f7f22ee5640 1 -- 172.21.15.57:0/198567213 wait complete. 2023-10-19T12:44:42.543 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":36,"stamp":"2023-10-19T12:44:41.319901+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595007,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236140,"kb_used_data":3300,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738580,"statfs":{"total":773094113280,"available":772852305920,"internally_reserved":0,"allocated":3379200,"data_stored":2025003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":238276153},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2229,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003613"},"pg_stats":[{"pgid":"2.1f","version":"20'4","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.113796+0000","last_change":"2023-10-19T12:44:18.071366+0000","last_active":"2023-10-19T12:44:23.113796+0000","last_peered":"2023-10-19T12:44:23.113796+0000","last_clean":"2023-10-19T12:44:23.113796+0000","last_became_active":"2023-10-19T12:44:18.071117+0000","last_became_peered":"2023-10-19T12:44:18.071117+0000","last_unstale":"2023-10-19T12:44:23.113796+0000","last_undegraded":"2023-10-19T12:44:23.113796+0000","last_fullsized":"2023-10-19T12:44:23.113796+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:24:31.294273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.107571+0000","last_change":"2023-10-19T12:44:19.080917+0000","last_active":"2023-10-19T12:44:23.107571+0000","last_peered":"2023-10-19T12:44:23.107571+0000","last_clean":"2023-10-19T12:44:23.107571+0000","last_became_active":"2023-10-19T12:44:19.080408+0000","last_became_peered":"2023-10-19T12:44:19.080408+0000","last_unstale":"2023-10-19T12:44:23.107571+0000","last_undegraded":"2023-10-19T12:44:23.107571+0000","last_fullsized":"2023-10-19T12:44:23.107571+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:29:52.686703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.258880+0000","last_change":"2023-10-19T12:44:23.262149+0000","last_active":"2023-10-19T12:44:24.258880+0000","last_peered":"2023-10-19T12:44:24.258880+0000","last_clean":"2023-10-19T12:44:24.258880+0000","last_became_active":"2023-10-19T12:44:23.261764+0000","last_became_peered":"2023-10-19T12:44:23.261764+0000","last_unstale":"2023-10-19T12:44:24.258880+0000","last_undegraded":"2023-10-19T12:44:24.258880+0000","last_fullsized":"2023-10-19T12:44:24.258880+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:33:24.282692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.107679+0000","last_change":"2023-10-19T12:44:18.064762+0000","last_active":"2023-10-19T12:44:23.107679+0000","last_peered":"2023-10-19T12:44:23.107679+0000","last_clean":"2023-10-19T12:44:23.107679+0000","last_became_active":"2023-10-19T12:44:18.064478+0000","last_became_peered":"2023-10-19T12:44:18.064478+0000","last_unstale":"2023-10-19T12:44:23.107679+0000","last_undegraded":"2023-10-19T12:44:23.107679+0000","last_fullsized":"2023-10-19T12:44:23.107679+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:46:29.885859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.113623+0000","last_change":"2023-10-19T12:44:19.083619+0000","last_active":"2023-10-19T12:44:23.113623+0000","last_peered":"2023-10-19T12:44:23.113623+0000","last_clean":"2023-10-19T12:44:23.113623+0000","last_became_active":"2023-10-19T12:44:19.083330+0000","last_became_peered":"2023-10-19T12:44:19.083330+0000","last_unstale":"2023-10-19T12:44:23.113623+0000","last_undegraded":"2023-10-19T12:44:23.113623+0000","last_fullsized":"2023-10-19T12:44:23.113623+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:00:15.982705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.123057+0000","last_change":"2023-10-19T12:44:23.132517+0000","last_active":"2023-10-19T12:44:24.123057+0000","last_peered":"2023-10-19T12:44:24.123057+0000","last_clean":"2023-10-19T12:44:24.123057+0000","last_became_active":"2023-10-19T12:44:23.132066+0000","last_became_peered":"2023-10-19T12:44:23.132066+0000","last_unstale":"2023-10-19T12:44:24.123057+0000","last_undegraded":"2023-10-19T12:44:24.123057+0000","last_fullsized":"2023-10-19T12:44:24.123057+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:46:59.402450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.260017+0000","last_change":"2023-10-19T12:44:18.070873+0000","last_active":"2023-10-19T12:44:23.260017+0000","last_peered":"2023-10-19T12:44:23.260017+0000","last_clean":"2023-10-19T12:44:23.260017+0000","last_became_active":"2023-10-19T12:44:18.070602+0000","last_became_peered":"2023-10-19T12:44:18.070602+0000","last_unstale":"2023-10-19T12:44:23.260017+0000","last_undegraded":"2023-10-19T12:44:23.260017+0000","last_fullsized":"2023-10-19T12:44:23.260017+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:15:18.575278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.106941+0000","last_change":"2023-10-19T12:44:19.075787+0000","last_active":"2023-10-19T12:44:23.106941+0000","last_peered":"2023-10-19T12:44:23.106941+0000","last_clean":"2023-10-19T12:44:23.106941+0000","last_became_active":"2023-10-19T12:44:19.074529+0000","last_became_peered":"2023-10-19T12:44:19.074529+0000","last_unstale":"2023-10-19T12:44:23.106941+0000","last_undegraded":"2023-10-19T12:44:23.106941+0000","last_fullsized":"2023-10-19T12:44:23.106941+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:29:52.686703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.124069+0000","last_change":"2023-10-19T12:44:23.132792+0000","last_active":"2023-10-19T12:44:24.124069+0000","last_peered":"2023-10-19T12:44:24.124069+0000","last_clean":"2023-10-19T12:44:24.124069+0000","last_became_active":"2023-10-19T12:44:23.132099+0000","last_became_peered":"2023-10-19T12:44:23.132099+0000","last_unstale":"2023-10-19T12:44:24.124069+0000","last_undegraded":"2023-10-19T12:44:24.124069+0000","last_fullsized":"2023-10-19T12:44:24.124069+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:01:13.019006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.259778+0000","last_change":"2023-10-19T12:44:18.068582+0000","last_active":"2023-10-19T12:44:23.259778+0000","last_peered":"2023-10-19T12:44:23.259778+0000","last_clean":"2023-10-19T12:44:23.259778+0000","last_became_active":"2023-10-19T12:44:18.068099+0000","last_became_peered":"2023-10-19T12:44:18.068099+0000","last_unstale":"2023-10-19T12:44:23.259778+0000","last_undegraded":"2023-10-19T12:44:23.259778+0000","last_fullsized":"2023-10-19T12:44:23.259778+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:24:31.294273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.107389+0000","last_change":"2023-10-19T12:44:19.076968+0000","last_active":"2023-10-19T12:44:23.107389+0000","last_peered":"2023-10-19T12:44:23.107389+0000","last_clean":"2023-10-19T12:44:23.107389+0000","last_became_active":"2023-10-19T12:44:19.076105+0000","last_became_peered":"2023-10-19T12:44:19.076105+0000","last_unstale":"2023-10-19T12:44:23.107389+0000","last_undegraded":"2023-10-19T12:44:23.107389+0000","last_fullsized":"2023-10-19T12:44:23.107389+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:59:45.926427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.123560+0000","last_change":"2023-10-19T12:44:23.125353+0000","last_active":"2023-10-19T12:44:24.123560+0000","last_peered":"2023-10-19T12:44:24.123560+0000","last_clean":"2023-10-19T12:44:24.123560+0000","last_became_active":"2023-10-19T12:44:23.124845+0000","last_became_peered":"2023-10-19T12:44:23.124845+0000","last_unstale":"2023-10-19T12:44:24.123560+0000","last_undegraded":"2023-10-19T12:44:24.123560+0000","last_fullsized":"2023-10-19T12:44:24.123560+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:59:49.972984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.116065+0000","last_change":"2023-10-19T12:44:19.084588+0000","last_active":"2023-10-19T12:44:23.116065+0000","last_peered":"2023-10-19T12:44:23.116065+0000","last_clean":"2023-10-19T12:44:23.116065+0000","last_became_active":"2023-10-19T12:44:19.084315+0000","last_became_peered":"2023-10-19T12:44:19.084315+0000","last_unstale":"2023-10-19T12:44:23.116065+0000","last_undegraded":"2023-10-19T12:44:23.116065+0000","last_fullsized":"2023-10-19T12:44:23.116065+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:54:38.037190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.259794+0000","last_change":"2023-10-19T12:44:18.069120+0000","last_active":"2023-10-19T12:44:23.259794+0000","last_peered":"2023-10-19T12:44:23.259794+0000","last_clean":"2023-10-19T12:44:23.259794+0000","last_became_active":"2023-10-19T12:44:18.068509+0000","last_became_peered":"2023-10-19T12:44:18.068509+0000","last_unstale":"2023-10-19T12:44:23.259794+0000","last_undegraded":"2023-10-19T12:44:23.259794+0000","last_fullsized":"2023-10-19T12:44:23.259794+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:01:31.282481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.124541+0000","last_change":"2023-10-19T12:44:23.133194+0000","last_active":"2023-10-19T12:44:24.124541+0000","last_peered":"2023-10-19T12:44:24.124541+0000","last_clean":"2023-10-19T12:44:24.124541+0000","last_became_active":"2023-10-19T12:44:23.132109+0000","last_became_peered":"2023-10-19T12:44:23.132109+0000","last_unstale":"2023-10-19T12:44:24.124541+0000","last_undegraded":"2023-10-19T12:44:24.124541+0000","last_fullsized":"2023-10-19T12:44:24.124541+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:37:44.936303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.108577+0000","last_change":"2023-10-19T12:44:19.082038+0000","last_active":"2023-10-19T12:44:23.108577+0000","last_peered":"2023-10-19T12:44:23.108577+0000","last_clean":"2023-10-19T12:44:23.108577+0000","last_became_active":"2023-10-19T12:44:19.080765+0000","last_became_peered":"2023-10-19T12:44:19.080765+0000","last_unstale":"2023-10-19T12:44:23.108577+0000","last_undegraded":"2023-10-19T12:44:23.108577+0000","last_fullsized":"2023-10-19T12:44:23.108577+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:01:32.286452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.123463+0000","last_change":"2023-10-19T12:44:18.070227+0000","last_active":"2023-10-19T12:44:23.123463+0000","last_peered":"2023-10-19T12:44:23.123463+0000","last_clean":"2023-10-19T12:44:23.123463+0000","last_became_active":"2023-10-19T12:44:18.069958+0000","last_became_peered":"2023-10-19T12:44:18.069958+0000","last_unstale":"2023-10-19T12:44:23.123463+0000","last_undegraded":"2023-10-19T12:44:23.123463+0000","last_fullsized":"2023-10-19T12:44:23.123463+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:27:49.242393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.123413+0000","last_change":"2023-10-19T12:44:23.126070+0000","last_active":"2023-10-19T12:44:24.123413+0000","last_peered":"2023-10-19T12:44:24.123413+0000","last_clean":"2023-10-19T12:44:24.123413+0000","last_became_active":"2023-10-19T12:44:23.125651+0000","last_became_peered":"2023-10-19T12:44:23.125651+0000","last_unstale":"2023-10-19T12:44:24.123413+0000","last_undegraded":"2023-10-19T12:44:24.123413+0000","last_fullsized":"2023-10-19T12:44:24.123413+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:29:56.733260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.109518+0000","last_change":"2023-10-19T12:44:19.085767+0000","last_active":"2023-10-19T12:44:23.109518+0000","last_peered":"2023-10-19T12:44:23.109518+0000","last_clean":"2023-10-19T12:44:23.109518+0000","last_became_active":"2023-10-19T12:44:19.085473+0000","last_became_peered":"2023-10-19T12:44:19.085473+0000","last_unstale":"2023-10-19T12:44:23.109518+0000","last_undegraded":"2023-10-19T12:44:23.109518+0000","last_fullsized":"2023-10-19T12:44:23.109518+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:50:42.817437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.123002+0000","last_change":"2023-10-19T12:44:18.074055+0000","last_active":"2023-10-19T12:44:23.123002+0000","last_peered":"2023-10-19T12:44:23.123002+0000","last_clean":"2023-10-19T12:44:23.123002+0000","last_became_active":"2023-10-19T12:44:18.073821+0000","last_became_peered":"2023-10-19T12:44:18.073821+0000","last_unstale":"2023-10-19T12:44:23.123002+0000","last_undegraded":"2023-10-19T12:44:23.123002+0000","last_fullsized":"2023-10-19T12:44:23.123002+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:24:31.294273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.125580+0000","last_change":"2023-10-19T12:44:23.263639+0000","last_active":"2023-10-19T12:44:24.125580+0000","last_peered":"2023-10-19T12:44:24.125580+0000","last_clean":"2023-10-19T12:44:24.125580+0000","last_became_active":"2023-10-19T12:44:23.263261+0000","last_became_peered":"2023-10-19T12:44:23.263261+0000","last_unstale":"2023-10-19T12:44:24.125580+0000","last_undegraded":"2023-10-19T12:44:24.125580+0000","last_fullsized":"2023-10-19T12:44:24.125580+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:53:36.534082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.114649+0000","last_change":"2023-10-19T12:44:19.080014+0000","last_active":"2023-10-19T12:44:23.114649+0000","last_peered":"2023-10-19T12:44:23.114649+0000","last_clean":"2023-10-19T12:44:23.114649+0000","last_became_active":"2023-10-19T12:44:19.079754+0000","last_became_peered":"2023-10-19T12:44:19.079754+0000","last_unstale":"2023-10-19T12:44:23.114649+0000","last_undegraded":"2023-10-19T12:44:23.114649+0000","last_fullsized":"2023-10-19T12:44:23.114649+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:27:39.900179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.259411+0000","last_change":"2023-10-19T12:44:18.069208+0000","last_active":"2023-10-19T12:44:23.259411+0000","last_peered":"2023-10-19T12:44:23.259411+0000","last_clean":"2023-10-19T12:44:23.259411+0000","last_became_active":"2023-10-19T12:44:18.068303+0000","last_became_peered":"2023-10-19T12:44:18.068303+0000","last_unstale":"2023-10-19T12:44:23.259411+0000","last_undegraded":"2023-10-19T12:44:23.259411+0000","last_fullsized":"2023-10-19T12:44:23.259411+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:11:21.842323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.123471+0000","last_change":"2023-10-19T12:44:23.131978+0000","last_active":"2023-10-19T12:44:24.123471+0000","last_peered":"2023-10-19T12:44:24.123471+0000","last_clean":"2023-10-19T12:44:24.123471+0000","last_became_active":"2023-10-19T12:44:23.131330+0000","last_became_peered":"2023-10-19T12:44:23.131330+0000","last_unstale":"2023-10-19T12:44:24.123471+0000","last_undegraded":"2023-10-19T12:44:24.123471+0000","last_fullsized":"2023-10-19T12:44:24.123471+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:49:30.221056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.113302+0000","last_change":"2023-10-19T12:44:19.087545+0000","last_active":"2023-10-19T12:44:23.113302+0000","last_peered":"2023-10-19T12:44:23.113302+0000","last_clean":"2023-10-19T12:44:23.113302+0000","last_became_active":"2023-10-19T12:44:19.087254+0000","last_became_peered":"2023-10-19T12:44:19.087254+0000","last_unstale":"2023-10-19T12:44:23.113302+0000","last_undegraded":"2023-10-19T12:44:23.113302+0000","last_fullsized":"2023-10-19T12:44:23.113302+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:22:48.368572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.116983+0000","last_change":"2023-10-19T12:44:18.074011+0000","last_active":"2023-10-19T12:44:23.116983+0000","last_peered":"2023-10-19T12:44:23.116983+0000","last_clean":"2023-10-19T12:44:23.116983+0000","last_became_active":"2023-10-19T12:44:18.073749+0000","last_became_peered":"2023-10-19T12:44:18.073749+0000","last_unstale":"2023-10-19T12:44:23.116983+0000","last_undegraded":"2023-10-19T12:44:23.116983+0000","last_fullsized":"2023-10-19T12:44:23.116983+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:46:29.885859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.123298+0000","last_change":"2023-10-19T12:44:23.264194+0000","last_active":"2023-10-19T12:44:24.123298+0000","last_peered":"2023-10-19T12:44:24.123298+0000","last_clean":"2023-10-19T12:44:24.123298+0000","last_became_active":"2023-10-19T12:44:23.263028+0000","last_became_peered":"2023-10-19T12:44:23.263028+0000","last_unstale":"2023-10-19T12:44:24.123298+0000","last_undegraded":"2023-10-19T12:44:24.123298+0000","last_fullsized":"2023-10-19T12:44:24.123298+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:05:19.882281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.107858+0000","last_change":"2023-10-19T12:44:19.094059+0000","last_active":"2023-10-19T12:44:23.107858+0000","last_peered":"2023-10-19T12:44:23.107858+0000","last_clean":"2023-10-19T12:44:23.107858+0000","last_became_active":"2023-10-19T12:44:19.093724+0000","last_became_peered":"2023-10-19T12:44:19.093724+0000","last_unstale":"2023-10-19T12:44:23.107858+0000","last_undegraded":"2023-10-19T12:44:23.107858+0000","last_fullsized":"2023-10-19T12:44:23.107858+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:49:26.174499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.123329+0000","last_change":"2023-10-19T12:44:18.071840+0000","last_active":"2023-10-19T12:44:23.123329+0000","last_peered":"2023-10-19T12:44:23.123329+0000","last_clean":"2023-10-19T12:44:23.123329+0000","last_became_active":"2023-10-19T12:44:18.071447+0000","last_became_peered":"2023-10-19T12:44:18.071447+0000","last_unstale":"2023-10-19T12:44:23.123329+0000","last_undegraded":"2023-10-19T12:44:23.123329+0000","last_fullsized":"2023-10-19T12:44:23.123329+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:46:38.630648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.258316+0000","last_change":"2023-10-19T12:44:23.263517+0000","last_active":"2023-10-19T12:44:24.258316+0000","last_peered":"2023-10-19T12:44:24.258316+0000","last_clean":"2023-10-19T12:44:24.258316+0000","last_became_active":"2023-10-19T12:44:23.263105+0000","last_became_peered":"2023-10-19T12:44:23.263105+0000","last_unstale":"2023-10-19T12:44:24.258316+0000","last_undegraded":"2023-10-19T12:44:24.258316+0000","last_fullsized":"2023-10-19T12:44:24.258316+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:56:44.960048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.122932+0000","last_change":"2023-10-19T12:44:19.088058+0000","last_active":"2023-10-19T12:44:23.122932+0000","last_peered":"2023-10-19T12:44:23.122932+0000","last_clean":"2023-10-19T12:44:23.122932+0000","last_became_active":"2023-10-19T12:44:19.087755+0000","last_became_peered":"2023-10-19T12:44:19.087755+0000","last_unstale":"2023-10-19T12:44:23.122932+0000","last_undegraded":"2023-10-19T12:44:23.122932+0000","last_fullsized":"2023-10-19T12:44:23.122932+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:54:38.037190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.260168+0000","last_change":"2023-10-19T12:44:18.070041+0000","last_active":"2023-10-19T12:44:23.260168+0000","last_peered":"2023-10-19T12:44:23.260168+0000","last_clean":"2023-10-19T12:44:23.260168+0000","last_became_active":"2023-10-19T12:44:18.069723+0000","last_became_peered":"2023-10-19T12:44:18.069723+0000","last_unstale":"2023-10-19T12:44:23.260168+0000","last_undegraded":"2023-10-19T12:44:23.260168+0000","last_fullsized":"2023-10-19T12:44:23.260168+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:10:07.046736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.258633+0000","last_change":"2023-10-19T12:44:23.260957+0000","last_active":"2023-10-19T12:44:24.258633+0000","last_peered":"2023-10-19T12:44:24.258633+0000","last_clean":"2023-10-19T12:44:24.258633+0000","last_became_active":"2023-10-19T12:44:23.260559+0000","last_became_peered":"2023-10-19T12:44:23.260559+0000","last_unstale":"2023-10-19T12:44:24.258633+0000","last_undegraded":"2023-10-19T12:44:24.258633+0000","last_fullsized":"2023-10-19T12:44:24.258633+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:30:09.977362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.107403+0000","last_change":"2023-10-19T12:44:19.080613+0000","last_active":"2023-10-19T12:44:23.107403+0000","last_peered":"2023-10-19T12:44:23.107403+0000","last_clean":"2023-10-19T12:44:23.107403+0000","last_became_active":"2023-10-19T12:44:19.080260+0000","last_became_peered":"2023-10-19T12:44:19.080260+0000","last_unstale":"2023-10-19T12:44:23.107403+0000","last_undegraded":"2023-10-19T12:44:23.107403+0000","last_fullsized":"2023-10-19T12:44:23.107403+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:49:26.174499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.123051+0000","last_change":"2023-10-19T12:44:18.072668+0000","last_active":"2023-10-19T12:44:23.123051+0000","last_peered":"2023-10-19T12:44:23.123051+0000","last_clean":"2023-10-19T12:44:23.123051+0000","last_became_active":"2023-10-19T12:44:18.072383+0000","last_became_peered":"2023-10-19T12:44:18.072383+0000","last_unstale":"2023-10-19T12:44:23.123051+0000","last_undegraded":"2023-10-19T12:44:23.123051+0000","last_fullsized":"2023-10-19T12:44:23.123051+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:07:40.511877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.124753+0000","last_change":"2023-10-19T12:44:23.130828+0000","last_active":"2023-10-19T12:44:24.124753+0000","last_peered":"2023-10-19T12:44:24.124753+0000","last_clean":"2023-10-19T12:44:24.124753+0000","last_became_active":"2023-10-19T12:44:23.130454+0000","last_became_peered":"2023-10-19T12:44:23.130454+0000","last_unstale":"2023-10-19T12:44:24.124753+0000","last_undegraded":"2023-10-19T12:44:24.124753+0000","last_fullsized":"2023-10-19T12:44:24.124753+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:30:09.977362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.114746+0000","last_change":"2023-10-19T12:44:19.075066+0000","last_active":"2023-10-19T12:44:23.114746+0000","last_peered":"2023-10-19T12:44:23.114746+0000","last_clean":"2023-10-19T12:44:23.114746+0000","last_became_active":"2023-10-19T12:44:19.074770+0000","last_became_peered":"2023-10-19T12:44:19.074770+0000","last_unstale":"2023-10-19T12:44:23.114746+0000","last_undegraded":"2023-10-19T12:44:23.114746+0000","last_fullsized":"2023-10-19T12:44:23.114746+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:11:22.846295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.106419+0000","last_change":"2023-10-19T12:44:18.072600+0000","last_active":"2023-10-19T12:44:23.106419+0000","last_peered":"2023-10-19T12:44:23.106419+0000","last_clean":"2023-10-19T12:44:23.106419+0000","last_became_active":"2023-10-19T12:44:18.072278+0000","last_became_peered":"2023-10-19T12:44:18.072278+0000","last_unstale":"2023-10-19T12:44:23.106419+0000","last_undegraded":"2023-10-19T12:44:23.106419+0000","last_fullsized":"2023-10-19T12:44:23.106419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:50:41.813466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.125020+0000","last_change":"2023-10-19T12:44:23.132395+0000","last_active":"2023-10-19T12:44:24.125020+0000","last_peered":"2023-10-19T12:44:24.125020+0000","last_clean":"2023-10-19T12:44:24.125020+0000","last_became_active":"2023-10-19T12:44:23.131961+0000","last_became_peered":"2023-10-19T12:44:23.131961+0000","last_unstale":"2023-10-19T12:44:24.125020+0000","last_undegraded":"2023-10-19T12:44:24.125020+0000","last_fullsized":"2023-10-19T12:44:24.125020+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:37:44.936303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.122772+0000","last_change":"2023-10-19T12:44:19.084119+0000","last_active":"2023-10-19T12:44:23.122772+0000","last_peered":"2023-10-19T12:44:23.122772+0000","last_clean":"2023-10-19T12:44:23.122772+0000","last_became_active":"2023-10-19T12:44:19.083803+0000","last_became_peered":"2023-10-19T12:44:19.083803+0000","last_unstale":"2023-10-19T12:44:23.122772+0000","last_undegraded":"2023-10-19T12:44:23.122772+0000","last_fullsized":"2023-10-19T12:44:23.122772+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:01:32.286452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.107952+0000","last_change":"2023-10-19T12:44:18.066014+0000","last_active":"2023-10-19T12:44:23.107952+0000","last_peered":"2023-10-19T12:44:23.107952+0000","last_clean":"2023-10-19T12:44:23.107952+0000","last_became_active":"2023-10-19T12:44:18.065652+0000","last_became_peered":"2023-10-19T12:44:18.065652+0000","last_unstale":"2023-10-19T12:44:23.107952+0000","last_undegraded":"2023-10-19T12:44:23.107952+0000","last_fullsized":"2023-10-19T12:44:23.107952+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:46:36.490219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.125536+0000","last_change":"2023-10-19T12:44:23.135917+0000","last_active":"2023-10-19T12:44:24.125536+0000","last_peered":"2023-10-19T12:44:24.125536+0000","last_clean":"2023-10-19T12:44:24.125536+0000","last_became_active":"2023-10-19T12:44:23.135459+0000","last_became_peered":"2023-10-19T12:44:23.135459+0000","last_unstale":"2023-10-19T12:44:24.125536+0000","last_undegraded":"2023-10-19T12:44:24.125536+0000","last_fullsized":"2023-10-19T12:44:24.125536+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:24:52.469571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.114317+0000","last_change":"2023-10-19T12:44:19.071520+0000","last_active":"2023-10-19T12:44:23.114317+0000","last_peered":"2023-10-19T12:44:23.114317+0000","last_clean":"2023-10-19T12:44:23.114317+0000","last_became_active":"2023-10-19T12:44:19.071008+0000","last_became_peered":"2023-10-19T12:44:19.071008+0000","last_unstale":"2023-10-19T12:44:23.114317+0000","last_undegraded":"2023-10-19T12:44:23.114317+0000","last_fullsized":"2023-10-19T12:44:23.114317+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:27:50.246364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.259411+0000","last_change":"2023-10-19T12:44:18.071276+0000","last_active":"2023-10-19T12:44:23.259411+0000","last_peered":"2023-10-19T12:44:23.259411+0000","last_clean":"2023-10-19T12:44:23.259411+0000","last_became_active":"2023-10-19T12:44:18.069565+0000","last_became_peered":"2023-10-19T12:44:18.069565+0000","last_unstale":"2023-10-19T12:44:23.259411+0000","last_undegraded":"2023-10-19T12:44:23.259411+0000","last_fullsized":"2023-10-19T12:44:23.259411+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:46:38.630648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.259572+0000","last_change":"2023-10-19T12:44:23.261271+0000","last_active":"2023-10-19T12:44:24.259572+0000","last_peered":"2023-10-19T12:44:24.259572+0000","last_clean":"2023-10-19T12:44:24.259572+0000","last_became_active":"2023-10-19T12:44:23.260853+0000","last_became_peered":"2023-10-19T12:44:23.260853+0000","last_unstale":"2023-10-19T12:44:24.259572+0000","last_undegraded":"2023-10-19T12:44:24.259572+0000","last_fullsized":"2023-10-19T12:44:24.259572+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:53:36.534082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.108153+0000","last_change":"2023-10-19T12:44:19.086009+0000","last_active":"2023-10-19T12:44:23.108153+0000","last_peered":"2023-10-19T12:44:23.108153+0000","last_clean":"2023-10-19T12:44:23.108153+0000","last_became_active":"2023-10-19T12:44:19.085688+0000","last_became_peered":"2023-10-19T12:44:19.085688+0000","last_unstale":"2023-10-19T12:44:23.108153+0000","last_undegraded":"2023-10-19T12:44:23.108153+0000","last_fullsized":"2023-10-19T12:44:23.108153+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:00:15.982705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.106386+0000","last_change":"2023-10-19T12:44:18.067051+0000","last_active":"2023-10-19T12:44:23.106386+0000","last_peered":"2023-10-19T12:44:23.106386+0000","last_clean":"2023-10-19T12:44:23.106386+0000","last_became_active":"2023-10-19T12:44:18.066784+0000","last_became_peered":"2023-10-19T12:44:18.066784+0000","last_unstale":"2023-10-19T12:44:23.106386+0000","last_undegraded":"2023-10-19T12:44:23.106386+0000","last_fullsized":"2023-10-19T12:44:23.106386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:43:47.410870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":168,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.259530+0000","last_change":"2023-10-19T12:44:08.636613+0000","last_active":"2023-10-19T12:44:23.259530+0000","last_peered":"2023-10-19T12:44:23.259530+0000","last_clean":"2023-10-19T12:44:23.259530+0000","last_became_active":"2023-10-19T12:44:08.635697+0000","last_became_peered":"2023-10-19T12:44:08.635697+0000","last_unstale":"2023-10-19T12:44:23.259530+0000","last_undegraded":"2023-10-19T12:44:23.259530+0000","last_fullsized":"2023-10-19T12:44:23.259530+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_clean_scrub_stamp":"2023-10-19T12:44:07.399738+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:15:23.685485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.258583+0000","last_change":"2023-10-19T12:44:23.261893+0000","last_active":"2023-10-19T12:44:24.258583+0000","last_peered":"2023-10-19T12:44:24.258583+0000","last_clean":"2023-10-19T12:44:24.258583+0000","last_became_active":"2023-10-19T12:44:23.261440+0000","last_became_peered":"2023-10-19T12:44:23.261440+0000","last_unstale":"2023-10-19T12:44:24.258583+0000","last_undegraded":"2023-10-19T12:44:24.258583+0000","last_fullsized":"2023-10-19T12:44:24.258583+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:30:09.977362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.259893+0000","last_change":"2023-10-19T12:44:19.093167+0000","last_active":"2023-10-19T12:44:23.259893+0000","last_peered":"2023-10-19T12:44:23.259893+0000","last_clean":"2023-10-19T12:44:23.259893+0000","last_became_active":"2023-10-19T12:44:19.092842+0000","last_became_peered":"2023-10-19T12:44:19.092842+0000","last_unstale":"2023-10-19T12:44:23.259893+0000","last_undegraded":"2023-10-19T12:44:23.259893+0000","last_fullsized":"2023-10-19T12:44:23.259893+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:01:08.972450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.108142+0000","last_change":"2023-10-19T12:44:18.071274+0000","last_active":"2023-10-19T12:44:23.108142+0000","last_peered":"2023-10-19T12:44:23.108142+0000","last_clean":"2023-10-19T12:44:23.108142+0000","last_became_active":"2023-10-19T12:44:18.070997+0000","last_became_peered":"2023-10-19T12:44:18.070997+0000","last_unstale":"2023-10-19T12:44:23.108142+0000","last_undegraded":"2023-10-19T12:44:23.108142+0000","last_fullsized":"2023-10-19T12:44:23.108142+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:27:38.896207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.258302+0000","last_change":"2023-10-19T12:44:23.261162+0000","last_active":"2023-10-19T12:44:24.258302+0000","last_peered":"2023-10-19T12:44:24.258302+0000","last_clean":"2023-10-19T12:44:24.258302+0000","last_became_active":"2023-10-19T12:44:23.260701+0000","last_became_peered":"2023-10-19T12:44:23.260701+0000","last_unstale":"2023-10-19T12:44:24.258302+0000","last_undegraded":"2023-10-19T12:44:24.258302+0000","last_fullsized":"2023-10-19T12:44:24.258302+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:01:13.019006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.107087+0000","last_change":"2023-10-19T12:44:19.090016+0000","last_active":"2023-10-19T12:44:23.107087+0000","last_peered":"2023-10-19T12:44:23.107087+0000","last_clean":"2023-10-19T12:44:23.107087+0000","last_became_active":"2023-10-19T12:44:19.086367+0000","last_became_peered":"2023-10-19T12:44:19.086367+0000","last_unstale":"2023-10-19T12:44:23.107087+0000","last_undegraded":"2023-10-19T12:44:23.107087+0000","last_fullsized":"2023-10-19T12:44:23.107087+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:01:08.972450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.123668+0000","last_change":"2023-10-19T12:44:18.069871+0000","last_active":"2023-10-19T12:44:23.123668+0000","last_peered":"2023-10-19T12:44:23.123668+0000","last_clean":"2023-10-19T12:44:23.123668+0000","last_became_active":"2023-10-19T12:44:18.069558+0000","last_became_peered":"2023-10-19T12:44:18.069558+0000","last_unstale":"2023-10-19T12:44:23.123668+0000","last_undegraded":"2023-10-19T12:44:23.123668+0000","last_fullsized":"2023-10-19T12:44:23.123668+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:15:18.575278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.258925+0000","last_change":"2023-10-19T12:44:23.131775+0000","last_active":"2023-10-19T12:44:24.258925+0000","last_peered":"2023-10-19T12:44:24.258925+0000","last_clean":"2023-10-19T12:44:24.258925+0000","last_became_active":"2023-10-19T12:44:23.131215+0000","last_became_peered":"2023-10-19T12:44:23.131215+0000","last_unstale":"2023-10-19T12:44:24.258925+0000","last_undegraded":"2023-10-19T12:44:24.258925+0000","last_fullsized":"2023-10-19T12:44:24.258925+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:37:44.936303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.260452+0000","last_change":"2023-10-19T12:44:19.089818+0000","last_active":"2023-10-19T12:44:23.260452+0000","last_peered":"2023-10-19T12:44:23.260452+0000","last_clean":"2023-10-19T12:44:23.260452+0000","last_became_active":"2023-10-19T12:44:19.089336+0000","last_became_peered":"2023-10-19T12:44:19.089336+0000","last_unstale":"2023-10-19T12:44:23.260452+0000","last_undegraded":"2023-10-19T12:44:23.260452+0000","last_fullsized":"2023-10-19T12:44:23.260452+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:53:32.487525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.108384+0000","last_change":"2023-10-19T12:44:18.068828+0000","last_active":"2023-10-19T12:44:23.108384+0000","last_peered":"2023-10-19T12:44:23.108384+0000","last_clean":"2023-10-19T12:44:23.108384+0000","last_became_active":"2023-10-19T12:44:18.068553+0000","last_became_peered":"2023-10-19T12:44:18.068553+0000","last_unstale":"2023-10-19T12:44:23.108384+0000","last_undegraded":"2023-10-19T12:44:23.108384+0000","last_fullsized":"2023-10-19T12:44:23.108384+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:11:21.842323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.123595+0000","last_change":"2023-10-19T12:44:23.261245+0000","last_active":"2023-10-19T12:44:24.123595+0000","last_peered":"2023-10-19T12:44:24.123595+0000","last_clean":"2023-10-19T12:44:24.123595+0000","last_became_active":"2023-10-19T12:44:23.260818+0000","last_became_peered":"2023-10-19T12:44:23.260818+0000","last_unstale":"2023-10-19T12:44:24.123595+0000","last_undegraded":"2023-10-19T12:44:24.123595+0000","last_fullsized":"2023-10-19T12:44:24.123595+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:55:55.981359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.260228+0000","last_change":"2023-10-19T12:44:19.087785+0000","last_active":"2023-10-19T12:44:23.260228+0000","last_peered":"2023-10-19T12:44:23.260228+0000","last_clean":"2023-10-19T12:44:23.260228+0000","last_became_active":"2023-10-19T12:44:19.087394+0000","last_became_peered":"2023-10-19T12:44:19.087394+0000","last_unstale":"2023-10-19T12:44:23.260228+0000","last_undegraded":"2023-10-19T12:44:23.260228+0000","last_fullsized":"2023-10-19T12:44:23.260228+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:37:40.889746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.115032+0000","last_change":"2023-10-19T12:44:18.068944+0000","last_active":"2023-10-19T12:44:23.115032+0000","last_peered":"2023-10-19T12:44:23.115032+0000","last_clean":"2023-10-19T12:44:23.115032+0000","last_became_active":"2023-10-19T12:44:18.068674+0000","last_became_peered":"2023-10-19T12:44:18.068674+0000","last_unstale":"2023-10-19T12:44:23.115032+0000","last_undegraded":"2023-10-19T12:44:23.115032+0000","last_fullsized":"2023-10-19T12:44:23.115032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:16:09.533399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.123008+0000","last_change":"2023-10-19T12:44:23.131983+0000","last_active":"2023-10-19T12:44:24.123008+0000","last_peered":"2023-10-19T12:44:24.123008+0000","last_clean":"2023-10-19T12:44:24.123008+0000","last_became_active":"2023-10-19T12:44:23.131304+0000","last_became_peered":"2023-10-19T12:44:23.131304+0000","last_unstale":"2023-10-19T12:44:24.123008+0000","last_undegraded":"2023-10-19T12:44:24.123008+0000","last_fullsized":"2023-10-19T12:44:24.123008+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:01:36.333009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.259741+0000","last_change":"2023-10-19T12:44:19.088843+0000","last_active":"2023-10-19T12:44:23.259741+0000","last_peered":"2023-10-19T12:44:23.259741+0000","last_clean":"2023-10-19T12:44:23.259741+0000","last_became_active":"2023-10-19T12:44:19.088578+0000","last_became_peered":"2023-10-19T12:44:19.088578+0000","last_unstale":"2023-10-19T12:44:23.259741+0000","last_undegraded":"2023-10-19T12:44:23.259741+0000","last_fullsized":"2023-10-19T12:44:23.259741+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:46:55.355893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.108073+0000","last_change":"2023-10-19T12:44:18.069371+0000","last_active":"2023-10-19T12:44:23.108073+0000","last_peered":"2023-10-19T12:44:23.108073+0000","last_clean":"2023-10-19T12:44:23.108073+0000","last_became_active":"2023-10-19T12:44:18.069118+0000","last_became_peered":"2023-10-19T12:44:18.069118+0000","last_unstale":"2023-10-19T12:44:23.108073+0000","last_undegraded":"2023-10-19T12:44:23.108073+0000","last_fullsized":"2023-10-19T12:44:23.108073+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:12:58.093890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.259448+0000","last_change":"2023-10-19T12:44:23.126162+0000","last_active":"2023-10-19T12:44:24.259448+0000","last_peered":"2023-10-19T12:44:24.259448+0000","last_clean":"2023-10-19T12:44:24.259448+0000","last_became_active":"2023-10-19T12:44:23.125364+0000","last_became_peered":"2023-10-19T12:44:23.125364+0000","last_unstale":"2023-10-19T12:44:24.259448+0000","last_undegraded":"2023-10-19T12:44:24.259448+0000","last_fullsized":"2023-10-19T12:44:24.259448+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:24:52.469571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.114403+0000","last_change":"2023-10-19T12:44:19.079421+0000","last_active":"2023-10-19T12:44:23.114403+0000","last_peered":"2023-10-19T12:44:23.114403+0000","last_clean":"2023-10-19T12:44:23.114403+0000","last_became_active":"2023-10-19T12:44:19.079151+0000","last_became_peered":"2023-10-19T12:44:19.079151+0000","last_unstale":"2023-10-19T12:44:23.114403+0000","last_undegraded":"2023-10-19T12:44:23.114403+0000","last_fullsized":"2023-10-19T12:44:23.114403+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:22:48.368572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.114676+0000","last_change":"2023-10-19T12:44:18.070653+0000","last_active":"2023-10-19T12:44:23.114676+0000","last_peered":"2023-10-19T12:44:23.114676+0000","last_clean":"2023-10-19T12:44:23.114676+0000","last_became_active":"2023-10-19T12:44:18.070404+0000","last_became_peered":"2023-10-19T12:44:18.070404+0000","last_unstale":"2023-10-19T12:44:23.114676+0000","last_undegraded":"2023-10-19T12:44:23.114676+0000","last_fullsized":"2023-10-19T12:44:23.114676+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:27:38.896207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.259106+0000","last_change":"2023-10-19T12:44:23.262410+0000","last_active":"2023-10-19T12:44:24.259106+0000","last_peered":"2023-10-19T12:44:24.259106+0000","last_clean":"2023-10-19T12:44:24.259106+0000","last_became_active":"2023-10-19T12:44:23.261667+0000","last_became_peered":"2023-10-19T12:44:23.261667+0000","last_unstale":"2023-10-19T12:44:24.259106+0000","last_undegraded":"2023-10-19T12:44:24.259106+0000","last_fullsized":"2023-10-19T12:44:24.259106+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:23:00.780721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.259656+0000","last_change":"2023-10-19T12:44:19.092039+0000","last_active":"2023-10-19T12:44:23.259656+0000","last_peered":"2023-10-19T12:44:23.259656+0000","last_clean":"2023-10-19T12:44:23.259656+0000","last_became_active":"2023-10-19T12:44:19.090421+0000","last_became_peered":"2023-10-19T12:44:19.090421+0000","last_unstale":"2023-10-19T12:44:23.259656+0000","last_undegraded":"2023-10-19T12:44:23.259656+0000","last_fullsized":"2023-10-19T12:44:23.259656+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:49:26.174499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.106406+0000","last_change":"2023-10-19T12:44:18.066260+0000","last_active":"2023-10-19T12:44:23.106406+0000","last_peered":"2023-10-19T12:44:23.106406+0000","last_clean":"2023-10-19T12:44:23.106406+0000","last_became_active":"2023-10-19T12:44:18.065913+0000","last_became_peered":"2023-10-19T12:44:18.065913+0000","last_unstale":"2023-10-19T12:44:23.106406+0000","last_undegraded":"2023-10-19T12:44:23.106406+0000","last_fullsized":"2023-10-19T12:44:23.106406+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:22:47.364601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.260099+0000","last_change":"2023-10-19T12:44:23.261486+0000","last_active":"2023-10-19T12:44:24.260099+0000","last_peered":"2023-10-19T12:44:24.260099+0000","last_clean":"2023-10-19T12:44:24.260099+0000","last_became_active":"2023-10-19T12:44:23.261044+0000","last_became_peered":"2023-10-19T12:44:23.261044+0000","last_unstale":"2023-10-19T12:44:24.260099+0000","last_undegraded":"2023-10-19T12:44:24.260099+0000","last_fullsized":"2023-10-19T12:44:24.260099+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:27:47.598684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.113455+0000","last_change":"2023-10-19T12:44:19.090117+0000","last_active":"2023-10-19T12:44:23.113455+0000","last_peered":"2023-10-19T12:44:23.113455+0000","last_clean":"2023-10-19T12:44:23.113455+0000","last_became_active":"2023-10-19T12:44:19.089829+0000","last_became_peered":"2023-10-19T12:44:19.089829+0000","last_unstale":"2023-10-19T12:44:23.113455+0000","last_undegraded":"2023-10-19T12:44:23.113455+0000","last_fullsized":"2023-10-19T12:44:23.113455+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:07:41.515848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.113821+0000","last_change":"2023-10-19T12:44:18.067372+0000","last_active":"2023-10-19T12:44:23.113821+0000","last_peered":"2023-10-19T12:44:23.113821+0000","last_clean":"2023-10-19T12:44:23.113821+0000","last_became_active":"2023-10-19T12:44:18.066991+0000","last_became_peered":"2023-10-19T12:44:18.066991+0000","last_unstale":"2023-10-19T12:44:23.113821+0000","last_undegraded":"2023-10-19T12:44:23.113821+0000","last_fullsized":"2023-10-19T12:44:23.113821+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:44:58.807931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.124117+0000","last_change":"2023-10-19T12:44:23.262285+0000","last_active":"2023-10-19T12:44:24.124117+0000","last_peered":"2023-10-19T12:44:24.124117+0000","last_clean":"2023-10-19T12:44:24.124117+0000","last_became_active":"2023-10-19T12:44:23.261716+0000","last_became_peered":"2023-10-19T12:44:23.261716+0000","last_unstale":"2023-10-19T12:44:24.124117+0000","last_undegraded":"2023-10-19T12:44:24.124117+0000","last_fullsized":"2023-10-19T12:44:24.124117+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:33:24.282692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.259803+0000","last_change":"2023-10-19T12:44:19.090802+0000","last_active":"2023-10-19T12:44:23.259803+0000","last_peered":"2023-10-19T12:44:23.259803+0000","last_clean":"2023-10-19T12:44:23.259803+0000","last_became_active":"2023-10-19T12:44:19.087107+0000","last_became_peered":"2023-10-19T12:44:19.087107+0000","last_unstale":"2023-10-19T12:44:23.259803+0000","last_undegraded":"2023-10-19T12:44:23.259803+0000","last_fullsized":"2023-10-19T12:44:23.259803+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:59:45.926427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"25'9","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:28.505371+0000","last_change":"2023-10-19T12:44:18.067624+0000","last_active":"2023-10-19T12:44:28.505371+0000","last_peered":"2023-10-19T12:44:28.505371+0000","last_clean":"2023-10-19T12:44:28.505371+0000","last_became_active":"2023-10-19T12:44:18.067343+0000","last_became_peered":"2023-10-19T12:44:18.067343+0000","last_unstale":"2023-10-19T12:44:28.505371+0000","last_undegraded":"2023-10-19T12:44:28.505371+0000","last_fullsized":"2023-10-19T12:44:28.505371+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:43:47.410870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3841,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.125150+0000","last_change":"2023-10-19T12:44:23.126083+0000","last_active":"2023-10-19T12:44:24.125150+0000","last_peered":"2023-10-19T12:44:24.125150+0000","last_clean":"2023-10-19T12:44:24.125150+0000","last_became_active":"2023-10-19T12:44:23.125365+0000","last_became_peered":"2023-10-19T12:44:23.125365+0000","last_unstale":"2023-10-19T12:44:24.125150+0000","last_undegraded":"2023-10-19T12:44:24.125150+0000","last_fullsized":"2023-10-19T12:44:24.125150+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:40:13.396617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.114883+0000","last_change":"2023-10-19T12:44:19.091229+0000","last_active":"2023-10-19T12:44:23.114883+0000","last_peered":"2023-10-19T12:44:23.114883+0000","last_clean":"2023-10-19T12:44:23.114883+0000","last_became_active":"2023-10-19T12:44:19.090912+0000","last_became_peered":"2023-10-19T12:44:19.090912+0000","last_unstale":"2023-10-19T12:44:23.114883+0000","last_undegraded":"2023-10-19T12:44:23.114883+0000","last_fullsized":"2023-10-19T12:44:23.114883+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:46:39.634619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.123572+0000","last_change":"2023-10-19T12:44:18.072463+0000","last_active":"2023-10-19T12:44:23.123572+0000","last_peered":"2023-10-19T12:44:23.123572+0000","last_clean":"2023-10-19T12:44:23.123572+0000","last_became_active":"2023-10-19T12:44:18.072156+0000","last_became_peered":"2023-10-19T12:44:18.072156+0000","last_unstale":"2023-10-19T12:44:23.123572+0000","last_undegraded":"2023-10-19T12:44:23.123572+0000","last_fullsized":"2023-10-19T12:44:23.123572+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:44:58.807931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.124138+0000","last_change":"2023-10-19T12:44:23.130655+0000","last_active":"2023-10-19T12:44:24.124138+0000","last_peered":"2023-10-19T12:44:24.124138+0000","last_clean":"2023-10-19T12:44:24.124138+0000","last_became_active":"2023-10-19T12:44:23.130142+0000","last_became_peered":"2023-10-19T12:44:23.130142+0000","last_unstale":"2023-10-19T12:44:24.124138+0000","last_undegraded":"2023-10-19T12:44:24.124138+0000","last_fullsized":"2023-10-19T12:44:24.124138+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:30:09.977362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.113707+0000","last_change":"2023-10-19T12:44:19.084143+0000","last_active":"2023-10-19T12:44:23.113707+0000","last_peered":"2023-10-19T12:44:23.113707+0000","last_clean":"2023-10-19T12:44:23.113707+0000","last_became_active":"2023-10-19T12:44:19.083824+0000","last_became_peered":"2023-10-19T12:44:19.083824+0000","last_unstale":"2023-10-19T12:44:23.113707+0000","last_undegraded":"2023-10-19T12:44:23.113707+0000","last_fullsized":"2023-10-19T12:44:23.113707+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:01:32.286452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.106975+0000","last_change":"2023-10-19T12:44:18.070592+0000","last_active":"2023-10-19T12:44:23.106975+0000","last_peered":"2023-10-19T12:44:23.106975+0000","last_clean":"2023-10-19T12:44:23.106975+0000","last_became_active":"2023-10-19T12:44:18.070297+0000","last_became_peered":"2023-10-19T12:44:18.070297+0000","last_unstale":"2023-10-19T12:44:23.106975+0000","last_undegraded":"2023-10-19T12:44:23.106975+0000","last_fullsized":"2023-10-19T12:44:23.106975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:24:31.294273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.258600+0000","last_change":"2023-10-19T12:44:23.131766+0000","last_active":"2023-10-19T12:44:24.258600+0000","last_peered":"2023-10-19T12:44:24.258600+0000","last_clean":"2023-10-19T12:44:24.258600+0000","last_became_active":"2023-10-19T12:44:23.131349+0000","last_became_peered":"2023-10-19T12:44:23.131349+0000","last_unstale":"2023-10-19T12:44:24.258600+0000","last_undegraded":"2023-10-19T12:44:24.258600+0000","last_fullsized":"2023-10-19T12:44:24.258600+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:55:55.981359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.106934+0000","last_change":"2023-10-19T12:44:19.089065+0000","last_active":"2023-10-19T12:44:23.106934+0000","last_peered":"2023-10-19T12:44:23.106934+0000","last_clean":"2023-10-19T12:44:23.106934+0000","last_became_active":"2023-10-19T12:44:19.088295+0000","last_became_peered":"2023-10-19T12:44:19.088295+0000","last_unstale":"2023-10-19T12:44:23.106934+0000","last_undegraded":"2023-10-19T12:44:23.106934+0000","last_fullsized":"2023-10-19T12:44:23.106934+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:30:05.930806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.116850+0000","last_change":"2023-10-19T12:44:18.071630+0000","last_active":"2023-10-19T12:44:23.116850+0000","last_peered":"2023-10-19T12:44:23.116850+0000","last_clean":"2023-10-19T12:44:23.116850+0000","last_became_active":"2023-10-19T12:44:18.071374+0000","last_became_peered":"2023-10-19T12:44:18.071374+0000","last_unstale":"2023-10-19T12:44:23.116850+0000","last_undegraded":"2023-10-19T12:44:23.116850+0000","last_fullsized":"2023-10-19T12:44:23.116850+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:11:21.842323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.259022+0000","last_change":"2023-10-19T12:44:23.261083+0000","last_active":"2023-10-19T12:44:24.259022+0000","last_peered":"2023-10-19T12:44:24.259022+0000","last_clean":"2023-10-19T12:44:24.259022+0000","last_became_active":"2023-10-19T12:44:23.260643+0000","last_became_peered":"2023-10-19T12:44:23.260643+0000","last_unstale":"2023-10-19T12:44:24.259022+0000","last_undegraded":"2023-10-19T12:44:24.259022+0000","last_fullsized":"2023-10-19T12:44:24.259022+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:25:26.010997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.259319+0000","last_change":"2023-10-19T12:44:23.263298+0000","last_active":"2023-10-19T12:44:24.259319+0000","last_peered":"2023-10-19T12:44:24.259319+0000","last_clean":"2023-10-19T12:44:24.259319+0000","last_became_active":"2023-10-19T12:44:23.262936+0000","last_became_peered":"2023-10-19T12:44:23.262936+0000","last_unstale":"2023-10-19T12:44:24.259319+0000","last_undegraded":"2023-10-19T12:44:24.259319+0000","last_fullsized":"2023-10-19T12:44:24.259319+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:40:13.396617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.122817+0000","last_change":"2023-10-19T12:44:19.089929+0000","last_active":"2023-10-19T12:44:23.122817+0000","last_peered":"2023-10-19T12:44:23.122817+0000","last_clean":"2023-10-19T12:44:23.122817+0000","last_became_active":"2023-10-19T12:44:19.089601+0000","last_became_peered":"2023-10-19T12:44:19.089601+0000","last_unstale":"2023-10-19T12:44:23.122817+0000","last_undegraded":"2023-10-19T12:44:23.122817+0000","last_fullsized":"2023-10-19T12:44:23.122817+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:22:48.368572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.107187+0000","last_change":"2023-10-19T12:44:18.069653+0000","last_active":"2023-10-19T12:44:23.107187+0000","last_peered":"2023-10-19T12:44:23.107187+0000","last_clean":"2023-10-19T12:44:23.107187+0000","last_became_active":"2023-10-19T12:44:18.068221+0000","last_became_peered":"2023-10-19T12:44:18.068221+0000","last_unstale":"2023-10-19T12:44:23.107187+0000","last_undegraded":"2023-10-19T12:44:23.107187+0000","last_fullsized":"2023-10-19T12:44:23.107187+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:54:37.033219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.258743+0000","last_change":"2023-10-19T12:44:23.131662+0000","last_active":"2023-10-19T12:44:24.258743+0000","last_peered":"2023-10-19T12:44:24.258743+0000","last_clean":"2023-10-19T12:44:24.258743+0000","last_became_active":"2023-10-19T12:44:23.131182+0000","last_became_peered":"2023-10-19T12:44:23.131182+0000","last_unstale":"2023-10-19T12:44:24.258743+0000","last_undegraded":"2023-10-19T12:44:24.258743+0000","last_fullsized":"2023-10-19T12:44:24.258743+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:37:44.936303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.109433+0000","last_change":"2023-10-19T12:44:19.081926+0000","last_active":"2023-10-19T12:44:23.109433+0000","last_peered":"2023-10-19T12:44:23.109433+0000","last_clean":"2023-10-19T12:44:23.109433+0000","last_became_active":"2023-10-19T12:44:19.081619+0000","last_became_peered":"2023-10-19T12:44:19.081619+0000","last_unstale":"2023-10-19T12:44:23.109433+0000","last_undegraded":"2023-10-19T12:44:23.109433+0000","last_fullsized":"2023-10-19T12:44:23.109433+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:22:48.368572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.108872+0000","last_change":"2023-10-19T12:44:18.071670+0000","last_active":"2023-10-19T12:44:23.108872+0000","last_peered":"2023-10-19T12:44:23.108872+0000","last_clean":"2023-10-19T12:44:23.108872+0000","last_became_active":"2023-10-19T12:44:18.071370+0000","last_became_peered":"2023-10-19T12:44:18.071370+0000","last_unstale":"2023-10-19T12:44:23.108872+0000","last_undegraded":"2023-10-19T12:44:23.108872+0000","last_fullsized":"2023-10-19T12:44:23.108872+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:15:18.575278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.116496+0000","last_change":"2023-10-19T12:44:18.068200+0000","last_active":"2023-10-19T12:44:23.116496+0000","last_peered":"2023-10-19T12:44:23.116496+0000","last_clean":"2023-10-19T12:44:23.116496+0000","last_became_active":"2023-10-19T12:44:18.067903+0000","last_became_peered":"2023-10-19T12:44:18.067903+0000","last_unstale":"2023-10-19T12:44:23.116496+0000","last_undegraded":"2023-10-19T12:44:23.116496+0000","last_fullsized":"2023-10-19T12:44:23.116496+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:27:49.242393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.113730+0000","last_change":"2023-10-19T12:44:19.073697+0000","last_active":"2023-10-19T12:44:23.113730+0000","last_peered":"2023-10-19T12:44:23.113730+0000","last_clean":"2023-10-19T12:44:23.113730+0000","last_became_active":"2023-10-19T12:44:19.073366+0000","last_became_peered":"2023-10-19T12:44:19.073366+0000","last_unstale":"2023-10-19T12:44:23.113730+0000","last_undegraded":"2023-10-19T12:44:23.113730+0000","last_fullsized":"2023-10-19T12:44:23.113730+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:46:55.355893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.258799+0000","last_change":"2023-10-19T12:44:23.131575+0000","last_active":"2023-10-19T12:44:24.258799+0000","last_peered":"2023-10-19T12:44:24.258799+0000","last_clean":"2023-10-19T12:44:24.258799+0000","last_became_active":"2023-10-19T12:44:23.131125+0000","last_became_peered":"2023-10-19T12:44:23.131125+0000","last_unstale":"2023-10-19T12:44:24.258799+0000","last_undegraded":"2023-10-19T12:44:24.258799+0000","last_fullsized":"2023-10-19T12:44:24.258799+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:15:23.625806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.108702+0000","last_change":"2023-10-19T12:44:18.071298+0000","last_active":"2023-10-19T12:44:23.108702+0000","last_peered":"2023-10-19T12:44:23.108702+0000","last_clean":"2023-10-19T12:44:23.108702+0000","last_became_active":"2023-10-19T12:44:18.070913+0000","last_became_peered":"2023-10-19T12:44:18.070913+0000","last_unstale":"2023-10-19T12:44:23.108702+0000","last_undegraded":"2023-10-19T12:44:23.108702+0000","last_fullsized":"2023-10-19T12:44:23.108702+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:27:49.242393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.115487+0000","last_change":"2023-10-19T12:44:19.085944+0000","last_active":"2023-10-19T12:44:23.115487+0000","last_peered":"2023-10-19T12:44:23.115487+0000","last_clean":"2023-10-19T12:44:23.115487+0000","last_became_active":"2023-10-19T12:44:19.085621+0000","last_became_peered":"2023-10-19T12:44:19.085621+0000","last_unstale":"2023-10-19T12:44:23.115487+0000","last_undegraded":"2023-10-19T12:44:23.115487+0000","last_fullsized":"2023-10-19T12:44:23.115487+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:12:59.097861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.260011+0000","last_change":"2023-10-19T12:44:23.126882+0000","last_active":"2023-10-19T12:44:24.260011+0000","last_peered":"2023-10-19T12:44:24.260011+0000","last_clean":"2023-10-19T12:44:24.260011+0000","last_became_active":"2023-10-19T12:44:23.126421+0000","last_became_peered":"2023-10-19T12:44:23.126421+0000","last_unstale":"2023-10-19T12:44:24.260011+0000","last_undegraded":"2023-10-19T12:44:24.260011+0000","last_fullsized":"2023-10-19T12:44:24.260011+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:32:54.650601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.114291+0000","last_change":"2023-10-19T12:44:19.080194+0000","last_active":"2023-10-19T12:44:23.114291+0000","last_peered":"2023-10-19T12:44:23.114291+0000","last_clean":"2023-10-19T12:44:23.114291+0000","last_became_active":"2023-10-19T12:44:19.079941+0000","last_became_peered":"2023-10-19T12:44:19.079941+0000","last_unstale":"2023-10-19T12:44:23.114291+0000","last_undegraded":"2023-10-19T12:44:23.114291+0000","last_fullsized":"2023-10-19T12:44:23.114291+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:50:42.817437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.107859+0000","last_change":"2023-10-19T12:44:19.098110+0000","last_active":"2023-10-19T12:44:23.107859+0000","last_peered":"2023-10-19T12:44:23.107859+0000","last_clean":"2023-10-19T12:44:23.107859+0000","last_became_active":"2023-10-19T12:44:19.097774+0000","last_became_peered":"2023-10-19T12:44:19.097774+0000","last_unstale":"2023-10-19T12:44:23.107859+0000","last_undegraded":"2023-10-19T12:44:23.107859+0000","last_fullsized":"2023-10-19T12:44:23.107859+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:24:32.298244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.106333+0000","last_change":"2023-10-19T12:44:19.083631+0000","last_active":"2023-10-19T12:44:23.106333+0000","last_peered":"2023-10-19T12:44:23.106333+0000","last_clean":"2023-10-19T12:44:23.106333+0000","last_became_active":"2023-10-19T12:44:19.082464+0000","last_became_peered":"2023-10-19T12:44:19.082464+0000","last_unstale":"2023-10-19T12:44:23.106333+0000","last_undegraded":"2023-10-19T12:44:23.106333+0000","last_fullsized":"2023-10-19T12:44:23.106333+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:40:09.350060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.113640+0000","last_change":"2023-10-19T12:44:19.083390+0000","last_active":"2023-10-19T12:44:23.113640+0000","last_peered":"2023-10-19T12:44:23.113640+0000","last_clean":"2023-10-19T12:44:23.113640+0000","last_became_active":"2023-10-19T12:44:19.082423+0000","last_became_peered":"2023-10-19T12:44:19.082423+0000","last_unstale":"2023-10-19T12:44:23.113640+0000","last_undegraded":"2023-10-19T12:44:23.113640+0000","last_fullsized":"2023-10-19T12:44:23.113640+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:54:38.037190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.110543+0000","last_change":"2023-10-19T12:44:19.090250+0000","last_active":"2023-10-19T12:44:23.110543+0000","last_peered":"2023-10-19T12:44:23.110543+0000","last_clean":"2023-10-19T12:44:23.110543+0000","last_became_active":"2023-10-19T12:44:19.089587+0000","last_became_peered":"2023-10-19T12:44:19.089587+0000","last_unstale":"2023-10-19T12:44:23.110543+0000","last_undegraded":"2023-10-19T12:44:23.110543+0000","last_fullsized":"2023-10-19T12:44:23.110543+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:46:55.355893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.122832+0000","last_change":"2023-10-19T12:44:19.088635+0000","last_active":"2023-10-19T12:44:23.122832+0000","last_peered":"2023-10-19T12:44:23.122832+0000","last_clean":"2023-10-19T12:44:23.122832+0000","last_became_active":"2023-10-19T12:44:19.088378+0000","last_became_peered":"2023-10-19T12:44:19.088378+0000","last_unstale":"2023-10-19T12:44:23.122832+0000","last_undegraded":"2023-10-19T12:44:23.122832+0000","last_fullsized":"2023-10-19T12:44:23.122832+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:00:15.982705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.115903+0000","last_change":"2023-10-19T12:44:19.089688+0000","last_active":"2023-10-19T12:44:23.115903+0000","last_peered":"2023-10-19T12:44:23.115903+0000","last_clean":"2023-10-19T12:44:23.115903+0000","last_became_active":"2023-10-19T12:44:19.089438+0000","last_became_peered":"2023-10-19T12:44:19.089438+0000","last_unstale":"2023-10-19T12:44:23.115903+0000","last_undegraded":"2023-10-19T12:44:23.115903+0000","last_fullsized":"2023-10-19T12:44:23.115903+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:46:55.355893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.107774+0000","last_change":"2023-10-19T12:44:19.087749+0000","last_active":"2023-10-19T12:44:23.107774+0000","last_peered":"2023-10-19T12:44:23.107774+0000","last_clean":"2023-10-19T12:44:23.107774+0000","last_became_active":"2023-10-19T12:44:19.087020+0000","last_became_peered":"2023-10-19T12:44:19.087020+0000","last_unstale":"2023-10-19T12:44:23.107774+0000","last_undegraded":"2023-10-19T12:44:23.107774+0000","last_fullsized":"2023-10-19T12:44:23.107774+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:05:15.835724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.106909+0000","last_change":"2023-10-19T12:44:19.085297+0000","last_active":"2023-10-19T12:44:23.106909+0000","last_peered":"2023-10-19T12:44:23.106909+0000","last_clean":"2023-10-19T12:44:23.106909+0000","last_became_active":"2023-10-19T12:44:19.084652+0000","last_became_peered":"2023-10-19T12:44:19.084652+0000","last_unstale":"2023-10-19T12:44:23.106909+0000","last_undegraded":"2023-10-19T12:44:23.106909+0000","last_fullsized":"2023-10-19T12:44:23.106909+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:05:15.835724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.108790+0000","last_change":"2023-10-19T12:44:19.084748+0000","last_active":"2023-10-19T12:44:23.108790+0000","last_peered":"2023-10-19T12:44:23.108790+0000","last_clean":"2023-10-19T12:44:23.108790+0000","last_became_active":"2023-10-19T12:44:19.084460+0000","last_became_peered":"2023-10-19T12:44:19.084460+0000","last_unstale":"2023-10-19T12:44:23.108790+0000","last_undegraded":"2023-10-19T12:44:23.108790+0000","last_fullsized":"2023-10-19T12:44:23.108790+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:54:38.037190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.106299+0000","last_change":"2023-10-19T12:44:19.087125+0000","last_active":"2023-10-19T12:44:23.106299+0000","last_peered":"2023-10-19T12:44:23.106299+0000","last_clean":"2023-10-19T12:44:23.106299+0000","last_became_active":"2023-10-19T12:44:19.086845+0000","last_became_peered":"2023-10-19T12:44:19.086845+0000","last_unstale":"2023-10-19T12:44:23.106299+0000","last_undegraded":"2023-10-19T12:44:23.106299+0000","last_fullsized":"2023-10-19T12:44:23.106299+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:37:40.889746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.113534+0000","last_change":"2023-10-19T12:44:19.089064+0000","last_active":"2023-10-19T12:44:23.113534+0000","last_peered":"2023-10-19T12:44:23.113534+0000","last_clean":"2023-10-19T12:44:23.113534+0000","last_became_active":"2023-10-19T12:44:19.088780+0000","last_became_peered":"2023-10-19T12:44:19.088780+0000","last_unstale":"2023-10-19T12:44:23.113534+0000","last_undegraded":"2023-10-19T12:44:23.113534+0000","last_fullsized":"2023-10-19T12:44:23.113534+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:10:08.050707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.115561+0000","last_change":"2023-10-19T12:44:19.087124+0000","last_active":"2023-10-19T12:44:23.115561+0000","last_peered":"2023-10-19T12:44:23.115561+0000","last_clean":"2023-10-19T12:44:23.115561+0000","last_became_active":"2023-10-19T12:44:19.086851+0000","last_became_peered":"2023-10-19T12:44:19.086851+0000","last_unstale":"2023-10-19T12:44:23.115561+0000","last_undegraded":"2023-10-19T12:44:23.115561+0000","last_fullsized":"2023-10-19T12:44:23.115561+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:10:08.050707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.260204+0000","last_change":"2023-10-19T12:44:19.091368+0000","last_active":"2023-10-19T12:44:23.260204+0000","last_peered":"2023-10-19T12:44:23.260204+0000","last_clean":"2023-10-19T12:44:23.260204+0000","last_became_active":"2023-10-19T12:44:19.091052+0000","last_became_peered":"2023-10-19T12:44:19.091052+0000","last_unstale":"2023-10-19T12:44:23.260204+0000","last_undegraded":"2023-10-19T12:44:23.260204+0000","last_fullsized":"2023-10-19T12:44:23.260204+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:40:09.350060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.108300+0000","last_change":"2023-10-19T12:44:19.080430+0000","last_active":"2023-10-19T12:44:23.108300+0000","last_peered":"2023-10-19T12:44:23.108300+0000","last_clean":"2023-10-19T12:44:23.108300+0000","last_became_active":"2023-10-19T12:44:19.079493+0000","last_became_peered":"2023-10-19T12:44:19.079493+0000","last_unstale":"2023-10-19T12:44:23.108300+0000","last_undegraded":"2023-10-19T12:44:23.108300+0000","last_fullsized":"2023-10-19T12:44:23.108300+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:46:39.634619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.107179+0000","last_change":"2023-10-19T12:44:19.088751+0000","last_active":"2023-10-19T12:44:23.107179+0000","last_peered":"2023-10-19T12:44:23.107179+0000","last_clean":"2023-10-19T12:44:23.107179+0000","last_became_active":"2023-10-19T12:44:19.088430+0000","last_became_peered":"2023-10-19T12:44:19.088430+0000","last_unstale":"2023-10-19T12:44:23.107179+0000","last_undegraded":"2023-10-19T12:44:23.107179+0000","last_fullsized":"2023-10-19T12:44:23.107179+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:24:48.423014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.106786+0000","last_change":"2023-10-19T12:44:19.082935+0000","last_active":"2023-10-19T12:44:23.106786+0000","last_peered":"2023-10-19T12:44:23.106786+0000","last_clean":"2023-10-19T12:44:23.106786+0000","last_became_active":"2023-10-19T12:44:19.082692+0000","last_became_peered":"2023-10-19T12:44:19.082692+0000","last_unstale":"2023-10-19T12:44:23.106786+0000","last_undegraded":"2023-10-19T12:44:23.106786+0000","last_fullsized":"2023-10-19T12:44:23.106786+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:10:08.050707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.122913+0000","last_change":"2023-10-19T12:44:19.087441+0000","last_active":"2023-10-19T12:44:23.122913+0000","last_peered":"2023-10-19T12:44:23.122913+0000","last_clean":"2023-10-19T12:44:23.122913+0000","last_became_active":"2023-10-19T12:44:19.087147+0000","last_became_peered":"2023-10-19T12:44:19.087147+0000","last_unstale":"2023-10-19T12:44:23.122913+0000","last_undegraded":"2023-10-19T12:44:23.122913+0000","last_fullsized":"2023-10-19T12:44:23.122913+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:50:42.817437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.113370+0000","last_change":"2023-10-19T12:44:19.085136+0000","last_active":"2023-10-19T12:44:23.113370+0000","last_peered":"2023-10-19T12:44:23.113370+0000","last_clean":"2023-10-19T12:44:23.113370+0000","last_became_active":"2023-10-19T12:44:19.084856+0000","last_became_peered":"2023-10-19T12:44:19.084856+0000","last_unstale":"2023-10-19T12:44:23.113370+0000","last_undegraded":"2023-10-19T12:44:23.113370+0000","last_fullsized":"2023-10-19T12:44:23.113370+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:46:30.889830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.109109+0000","last_change":"2023-10-19T12:44:19.080419+0000","last_active":"2023-10-19T12:44:23.109109+0000","last_peered":"2023-10-19T12:44:23.109109+0000","last_clean":"2023-10-19T12:44:23.109109+0000","last_became_active":"2023-10-19T12:44:19.080123+0000","last_became_peered":"2023-10-19T12:44:19.080123+0000","last_unstale":"2023-10-19T12:44:23.109109+0000","last_undegraded":"2023-10-19T12:44:23.109109+0000","last_fullsized":"2023-10-19T12:44:23.109109+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:10:08.050707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.114806+0000","last_change":"2023-10-19T12:44:19.080702+0000","last_active":"2023-10-19T12:44:23.114806+0000","last_peered":"2023-10-19T12:44:23.114806+0000","last_clean":"2023-10-19T12:44:23.114806+0000","last_became_active":"2023-10-19T12:44:19.079347+0000","last_became_peered":"2023-10-19T12:44:19.079347+0000","last_unstale":"2023-10-19T12:44:23.114806+0000","last_undegraded":"2023-10-19T12:44:23.114806+0000","last_fullsized":"2023-10-19T12:44:23.114806+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:43:48.414841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.115372+0000","last_change":"2023-10-19T12:44:19.081326+0000","last_active":"2023-10-19T12:44:23.115372+0000","last_peered":"2023-10-19T12:44:23.115372+0000","last_clean":"2023-10-19T12:44:23.115372+0000","last_became_active":"2023-10-19T12:44:19.081063+0000","last_became_peered":"2023-10-19T12:44:19.081063+0000","last_unstale":"2023-10-19T12:44:23.115372+0000","last_undegraded":"2023-10-19T12:44:23.115372+0000","last_fullsized":"2023-10-19T12:44:23.115372+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:00:15.982705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.115814+0000","last_change":"2023-10-19T12:44:19.094478+0000","last_active":"2023-10-19T12:44:23.115814+0000","last_peered":"2023-10-19T12:44:23.115814+0000","last_clean":"2023-10-19T12:44:23.115814+0000","last_became_active":"2023-10-19T12:44:19.094217+0000","last_became_peered":"2023-10-19T12:44:19.094217+0000","last_unstale":"2023-10-19T12:44:23.115814+0000","last_undegraded":"2023-10-19T12:44:23.115814+0000","last_fullsized":"2023-10-19T12:44:23.115814+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:01:32.286452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.260360+0000","last_change":"2023-10-19T12:44:19.093197+0000","last_active":"2023-10-19T12:44:23.260360+0000","last_peered":"2023-10-19T12:44:23.260360+0000","last_clean":"2023-10-19T12:44:23.260360+0000","last_became_active":"2023-10-19T12:44:19.092926+0000","last_became_peered":"2023-10-19T12:44:19.092926+0000","last_unstale":"2023-10-19T12:44:23.260360+0000","last_undegraded":"2023-10-19T12:44:23.260360+0000","last_fullsized":"2023-10-19T12:44:23.260360+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:30:05.930806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.109339+0000","last_change":"2023-10-19T12:44:19.095093+0000","last_active":"2023-10-19T12:44:23.109339+0000","last_peered":"2023-10-19T12:44:23.109339+0000","last_clean":"2023-10-19T12:44:23.109339+0000","last_became_active":"2023-10-19T12:44:19.094816+0000","last_became_peered":"2023-10-19T12:44:19.094816+0000","last_unstale":"2023-10-19T12:44:23.109339+0000","last_undegraded":"2023-10-19T12:44:23.109339+0000","last_fullsized":"2023-10-19T12:44:23.109339+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:07:41.515848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.259584+0000","last_change":"2023-10-19T12:44:19.086048+0000","last_active":"2023-10-19T12:44:23.259584+0000","last_peered":"2023-10-19T12:44:23.259584+0000","last_clean":"2023-10-19T12:44:23.259584+0000","last_became_active":"2023-10-19T12:44:19.085086+0000","last_became_peered":"2023-10-19T12:44:19.085086+0000","last_unstale":"2023-10-19T12:44:23.259584+0000","last_undegraded":"2023-10-19T12:44:23.259584+0000","last_fullsized":"2023-10-19T12:44:23.259584+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:05:15.835724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.106214+0000","last_change":"2023-10-19T12:44:19.086341+0000","last_active":"2023-10-19T12:44:23.106214+0000","last_peered":"2023-10-19T12:44:23.106214+0000","last_clean":"2023-10-19T12:44:23.106214+0000","last_became_active":"2023-10-19T12:44:19.083365+0000","last_became_peered":"2023-10-19T12:44:19.083365+0000","last_unstale":"2023-10-19T12:44:23.106214+0000","last_undegraded":"2023-10-19T12:44:23.106214+0000","last_fullsized":"2023-10-19T12:44:23.106214+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:46:55.355893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.259588+0000","last_change":"2023-10-19T12:44:19.091228+0000","last_active":"2023-10-19T12:44:23.259588+0000","last_peered":"2023-10-19T12:44:23.259588+0000","last_clean":"2023-10-19T12:44:23.259588+0000","last_became_active":"2023-10-19T12:44:19.087203+0000","last_became_peered":"2023-10-19T12:44:19.087203+0000","last_unstale":"2023-10-19T12:44:23.259588+0000","last_undegraded":"2023-10-19T12:44:23.259588+0000","last_fullsized":"2023-10-19T12:44:23.259588+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:29:52.686703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.106856+0000","last_change":"2023-10-19T12:44:19.086112+0000","last_active":"2023-10-19T12:44:23.106856+0000","last_peered":"2023-10-19T12:44:23.106856+0000","last_clean":"2023-10-19T12:44:23.106856+0000","last_became_active":"2023-10-19T12:44:19.085822+0000","last_became_peered":"2023-10-19T12:44:19.085822+0000","last_unstale":"2023-10-19T12:44:23.106856+0000","last_undegraded":"2023-10-19T12:44:23.106856+0000","last_fullsized":"2023-10-19T12:44:23.106856+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:01:32.286452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.113429+0000","last_change":"2023-10-19T12:44:19.078205+0000","last_active":"2023-10-19T12:44:23.113429+0000","last_peered":"2023-10-19T12:44:23.113429+0000","last_clean":"2023-10-19T12:44:23.113429+0000","last_became_active":"2023-10-19T12:44:19.077824+0000","last_became_peered":"2023-10-19T12:44:19.077824+0000","last_unstale":"2023-10-19T12:44:23.113429+0000","last_undegraded":"2023-10-19T12:44:23.113429+0000","last_fullsized":"2023-10-19T12:44:23.113429+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:50:42.817437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.123225+0000","last_change":"2023-10-19T12:44:19.084212+0000","last_active":"2023-10-19T12:44:23.123225+0000","last_peered":"2023-10-19T12:44:23.123225+0000","last_clean":"2023-10-19T12:44:23.123225+0000","last_became_active":"2023-10-19T12:44:19.083928+0000","last_became_peered":"2023-10-19T12:44:19.083928+0000","last_unstale":"2023-10-19T12:44:23.123225+0000","last_undegraded":"2023-10-19T12:44:23.123225+0000","last_fullsized":"2023-10-19T12:44:23.123225+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:10:08.050707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.260753+0000","last_change":"2023-10-19T12:44:19.087938+0000","last_active":"2023-10-19T12:44:23.260753+0000","last_peered":"2023-10-19T12:44:23.260753+0000","last_clean":"2023-10-19T12:44:23.260753+0000","last_became_active":"2023-10-19T12:44:19.087552+0000","last_became_peered":"2023-10-19T12:44:19.087552+0000","last_unstale":"2023-10-19T12:44:23.260753+0000","last_undegraded":"2023-10-19T12:44:23.260753+0000","last_fullsized":"2023-10-19T12:44:23.260753+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:24:48.423014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4639,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574857,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27868,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343972,"statfs":{"total":96636764160,"available":96608227328,"internally_reserved":0,"allocated":864256,"data_stored":695401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55100000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.497}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52500000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38200000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60699999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45400000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":108292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66100000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.622}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51700000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67700000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42099999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39900000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574857,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":270336,"data_stored":104541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74399999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68799999999999994}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60699999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56000000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79300000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]}]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":278528,"data_stored":104577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66700000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76900000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73799999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53800000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52100000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44500000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64100000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31448,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340392,"statfs":{"total":96636764160,"available":96604561408,"internally_reserved":0,"allocated":278528,"data_stored":108292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36099999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39500000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47799999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61599999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83199999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85199999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71899999999999997}]}]},{"osd":2,"up_from":13,"seq":55834574858,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31448,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340392,"statfs":{"total":96636764160,"available":96604561408,"internally_reserved":0,"allocated":266240,"data_stored":104451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34300000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30299999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66000000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.504}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59499999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56399999999999995}]}]},{"osd":1,"up_from":13,"seq":55834574857,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31460,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340380,"statfs":{"total":96636764160,"available":96604549120,"internally_reserved":0,"allocated":274432,"data_stored":104611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35999999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41599999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47099999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.873}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82399999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74299999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84199999999999997}]}]},{"osd":0,"up_from":13,"seq":55834574858,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32032,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339808,"statfs":{"total":96636764160,"available":96603963392,"internally_reserved":0,"allocated":868352,"data_stored":694838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34799999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.379}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57899999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45500000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53200000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-19T12:44:42.550 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-19T12:44:42.550 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2023-10-19T12:44:42.551 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-10-19T12:44:42.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.690+0000 7f1c224d7640 1 -- 172.21.15.57:0/697473786 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1c1c15b2a0 msgr2=0x7f1c1c0d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:42.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.690+0000 7f1c224d7640 1 --2- 172.21.15.57:0/697473786 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1c1c15b2a0 0x7f1c1c0d1910 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f1c10005fd0 tx=0x7f1c1002bd90 comp rx=0 tx=0).stop 2023-10-19T12:44:42.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.690+0000 7f1c224d7640 1 -- 172.21.15.57:0/697473786 shutdown_connections 2023-10-19T12:44:42.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.690+0000 7f1c224d7640 1 --2- 172.21.15.57:0/697473786 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1c1c0ceb40 0x7f1c1c0b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:42.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.690+0000 7f1c224d7640 1 --2- 172.21.15.57:0/697473786 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1c1c15b2a0 0x7f1c1c0d1910 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:42.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.690+0000 7f1c224d7640 1 -- 172.21.15.57:0/697473786 >> 172.21.15.57:0/697473786 conn(0x7f1c1c0d0d60 msgr2=0x7f1c1c0d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:42.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.690+0000 7f1c224d7640 1 -- 172.21.15.57:0/697473786 shutdown_connections 2023-10-19T12:44:42.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.690+0000 7f1c224d7640 1 -- 172.21.15.57:0/697473786 wait complete. 2023-10-19T12:44:42.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.691+0000 7f1c224d7640 1 Processor -- start 2023-10-19T12:44:42.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.692+0000 7f1c224d7640 1 -- start start 2023-10-19T12:44:42.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.692+0000 7f1c224d7640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1c1c0ceb40 0x7f1c1c15e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:42.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.692+0000 7f1c224d7640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1c1c0cf340 0x7f1c1c155d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:42.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.692+0000 7f1c224d7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1c1c15b2a0 0x7f1c1c0b60c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:42.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.692+0000 7f1c224d7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c1c0c7f60 con 0x7f1c1c15b2a0 2023-10-19T12:44:42.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.692+0000 7f1c224d7640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c1c1570d0 con 0x7f1c1c0cf340 2023-10-19T12:44:42.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.692+0000 7f1c224d7640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c1c1573b0 con 0x7f1c1c0ceb40 2023-10-19T12:44:42.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.693+0000 7f1c20a74640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1c1c15b2a0 0x7f1c1c0b60c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:42.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.693+0000 7f1c20a74640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1c1c15b2a0 0x7f1c1c0b60c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50542/0 (socket says 172.21.15.57:50542) 2023-10-19T12:44:42.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.693+0000 7f1c20a74640 1 -- 172.21.15.57:0/2736295532 learned_addr learned my addr 172.21.15.57:0/2736295532 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:42.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.693+0000 7f1c1b7fe640 1 --2- 172.21.15.57:0/2736295532 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1c1c0cf340 0x7f1c1c155d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:42.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.693+0000 7f1c1bfff640 1 --2- 172.21.15.57:0/2736295532 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1c1c0ceb40 0x7f1c1c15e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:42.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.693+0000 7f1c20a74640 1 -- 172.21.15.57:0/2736295532 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1c1c0ceb40 msgr2=0x7f1c1c15e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:42.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.693+0000 7f1c20a74640 1 --2- 172.21.15.57:0/2736295532 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1c1c0ceb40 0x7f1c1c15e220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:42.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.693+0000 7f1c20a74640 1 -- 172.21.15.57:0/2736295532 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1c1c0cf340 msgr2=0x7f1c1c155d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:42.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.693+0000 7f1c20a74640 1 --2- 172.21.15.57:0/2736295532 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1c1c0cf340 0x7f1c1c155d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:42.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.693+0000 7f1c20a74640 1 -- 172.21.15.57:0/2736295532 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c10009560 con 0x7f1c1c15b2a0 2023-10-19T12:44:42.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.693+0000 7f1c20a74640 1 --2- 172.21.15.57:0/2736295532 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1c1c15b2a0 0x7f1c1c0b60c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f1c1400eaa0 tx=0x7f1c1400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:42.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.693+0000 7f1c18ff9640 1 -- 172.21.15.57:0/2736295532 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c1400cdb0 con 0x7f1c1c15b2a0 2023-10-19T12:44:42.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.693+0000 7f1c18ff9640 1 -- 172.21.15.57:0/2736295532 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f1c14004b20 con 0x7f1c1c15b2a0 2023-10-19T12:44:42.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.693+0000 7f1c18ff9640 1 -- 172.21.15.57:0/2736295532 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c14005020 con 0x7f1c1c15b2a0 2023-10-19T12:44:42.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.693+0000 7f1c224d7640 1 -- 172.21.15.57:0/2736295532 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c1c0b0d50 con 0x7f1c1c15b2a0 2023-10-19T12:44:42.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.693+0000 7f1c224d7640 1 -- 172.21.15.57:0/2736295532 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c1c0b6600 con 0x7f1c1c15b2a0 2023-10-19T12:44:42.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.694+0000 7f1c224d7640 1 -- 172.21.15.57:0/2736295532 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c1c0c80d0 con 0x7f1c1c15b2a0 2023-10-19T12:44:42.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.695+0000 7f1c18ff9640 1 -- 172.21.15.57:0/2736295532 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f1c14005180 con 0x7f1c1c15b2a0 2023-10-19T12:44:42.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.695+0000 7f1c18ff9640 1 --2- 172.21.15.57:0/2736295532 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f1c0806ae40 0x7f1c0806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:42.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.695+0000 7f1c18ff9640 1 -- 172.21.15.57:0/2736295532 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f1c1408a3c0 con 0x7f1c1c15b2a0 2023-10-19T12:44:42.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.695+0000 7f1c1bfff640 1 --2- 172.21.15.57:0/2736295532 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f1c0806ae40 0x7f1c0806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:42.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.696+0000 7f1c1bfff640 1 --2- 172.21.15.57:0/2736295532 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f1c0806ae40 0x7f1c0806d300 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c000f60 tx=0x7f1c0c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:42.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.699+0000 7f1c18ff9640 1 -- 172.21.15.57:0/2736295532 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f1c140550e0 con 0x7f1c1c15b2a0 2023-10-19T12:44:42.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.875+0000 7f1c224d7640 1 -- 172.21.15.57:0/2736295532 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "config log", "num": 1, "format": "json"} v 0) v1 -- 0x7f1c1c0690e0 con 0x7f1c1c15b2a0 2023-10-19T12:44:42.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.875+0000 7f1c18ff9640 1 -- 172.21.15.57:0/2736295532 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "config log", "num": 1, "format": "json"}]=0 v30) v1 ==== 86+0+160 (secure 0 0 0) 0x7f1c1401f0b0 con 0x7f1c1c15b2a0 2023-10-19T12:44:42.879 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:44:42.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.877+0000 7f1c224d7640 1 -- 172.21.15.57:0/2736295532 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f1c0806ae40 msgr2=0x7f1c0806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:42.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.877+0000 7f1c224d7640 1 --2- 172.21.15.57:0/2736295532 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f1c0806ae40 0x7f1c0806d300 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c000f60 tx=0x7f1c0c009040 comp rx=0 tx=0).stop 2023-10-19T12:44:42.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.877+0000 7f1c224d7640 1 -- 172.21.15.57:0/2736295532 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1c1c15b2a0 msgr2=0x7f1c1c0b60c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:42.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.877+0000 7f1c224d7640 1 --2- 172.21.15.57:0/2736295532 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1c1c15b2a0 0x7f1c1c0b60c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f1c1400eaa0 tx=0x7f1c1400ef60 comp rx=0 tx=0).stop 2023-10-19T12:44:42.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.877+0000 7f1c224d7640 1 -- 172.21.15.57:0/2736295532 shutdown_connections 2023-10-19T12:44:42.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.877+0000 7f1c224d7640 1 --2- 172.21.15.57:0/2736295532 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f1c0806ae40 0x7f1c0806d300 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:42.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.877+0000 7f1c224d7640 1 --2- 172.21.15.57:0/2736295532 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1c1c15b2a0 0x7f1c1c0b60c0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:42.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.877+0000 7f1c224d7640 1 --2- 172.21.15.57:0/2736295532 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1c1c0cf340 0x7f1c1c155d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:42.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.877+0000 7f1c224d7640 1 --2- 172.21.15.57:0/2736295532 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1c1c0ceb40 0x7f1c1c15e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:42.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.877+0000 7f1c224d7640 1 -- 172.21.15.57:0/2736295532 >> 172.21.15.57:0/2736295532 conn(0x7f1c1c0d0d60 msgr2=0x7f1c1c0ad8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:42.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.877+0000 7f1c224d7640 1 -- 172.21.15.57:0/2736295532 shutdown_connections 2023-10-19T12:44:42.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.878+0000 7f1c224d7640 1 -- 172.21.15.57:0/2736295532 wait complete. 2023-10-19T12:44:42.891 INFO:teuthology.orchestra.run.smithi057.stdout:[{"version":30,"timestamp":"2023-10-19T12:44:04.628793+0000","name":"","changes":[{"name":"osd.1/osd_mclock_max_capacity_iops_ssd","new_value":"4208.118410"}]}] 2023-10-19T12:44:42.891 INFO:tasks.ceph_manager:config epoch is 30 2023-10-19T12:44:42.892 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-10-19T12:44:42.892 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-10-19T12:44:42.892 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-10-19T12:44:43.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.999+0000 7f8880d95640 1 -- 172.21.15.57:0/3369608220 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f887c0b0530 msgr2=0x7f887c0b09c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:43.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.999+0000 7f8880d95640 1 --2- 172.21.15.57:0/3369608220 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f887c0b0530 0x7f887c0b09c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f8874005fd0 tx=0x7f887402c0e0 comp rx=0 tx=0).stop 2023-10-19T12:44:43.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.999+0000 7f8880d95640 1 -- 172.21.15.57:0/3369608220 shutdown_connections 2023-10-19T12:44:43.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.999+0000 7f8880d95640 1 --2- 172.21.15.57:0/3369608220 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f887c0b0530 0x7f887c0b09c0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:43.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.999+0000 7f8880d95640 1 --2- 172.21.15.57:0/3369608220 >> v2:172.21.15.179:3300/0 conn(0x7f887c0cbd20 0x7f887c158300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:43.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.999+0000 7f8880d95640 1 -- 172.21.15.57:0/3369608220 >> 172.21.15.57:0/3369608220 conn(0x7f887c069930 msgr2=0x7f887c069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:43.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.999+0000 7f8880d95640 1 -- 172.21.15.57:0/3369608220 shutdown_connections 2023-10-19T12:44:43.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:42.999+0000 7f8880d95640 1 -- 172.21.15.57:0/3369608220 wait complete. 2023-10-19T12:44:43.008 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.000+0000 7f8880d95640 1 Processor -- start 2023-10-19T12:44:43.008 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.000+0000 7f8880d95640 1 -- start start 2023-10-19T12:44:43.008 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.001+0000 7f8880d95640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f887c0afa20 0x7f887c150220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:43.008 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.001+0000 7f8880d95640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f887c0b0530 0x7f887c149420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:43.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.001+0000 7f8880d95640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f887c0cbd20 0x7f887c1562c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:43.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.001+0000 7f8880d95640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f887c14ea50 con 0x7f887c0b0530 2023-10-19T12:44:43.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.001+0000 7f8880d95640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f887c14ecf0 con 0x7f887c0cbd20 2023-10-19T12:44:43.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.001+0000 7f8880d95640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f887c14efd0 con 0x7f887c0afa20 2023-10-19T12:44:43.010 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.001+0000 7f8879d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f887c0b0530 0x7f887c149420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:43.010 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.001+0000 7f8879d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f887c0b0530 0x7f887c149420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50558/0 (socket says 172.21.15.57:50558) 2023-10-19T12:44:43.010 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.001+0000 7f8879d9b640 1 -- 172.21.15.57:0/3601047770 learned_addr learned my addr 172.21.15.57:0/3601047770 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:43.010 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.001+0000 7f887a59c640 1 --2- 172.21.15.57:0/3601047770 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f887c0afa20 0x7f887c150220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:43.010 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.001+0000 7f887ad9d640 1 --2- 172.21.15.57:0/3601047770 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f887c0cbd20 0x7f887c1562c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:43.011 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.001+0000 7f8879d9b640 1 -- 172.21.15.57:0/3601047770 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f887c0afa20 msgr2=0x7f887c150220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:43.011 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.001+0000 7f8879d9b640 1 --2- 172.21.15.57:0/3601047770 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f887c0afa20 0x7f887c150220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:43.011 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.001+0000 7f8879d9b640 1 -- 172.21.15.57:0/3601047770 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f887c0cbd20 msgr2=0x7f887c1562c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:43.011 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.001+0000 7f8879d9b640 1 --2- 172.21.15.57:0/3601047770 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f887c0cbd20 0x7f887c1562c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:43.011 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.001+0000 7f8879d9b640 1 -- 172.21.15.57:0/3601047770 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f886c00a5a0 con 0x7f887c0b0530 2023-10-19T12:44:43.012 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.001+0000 7f887a59c640 1 --2- 172.21.15.57:0/3601047770 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f887c0afa20 0x7f887c150220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:44:43.012 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.001+0000 7f887ad9d640 1 --2- 172.21.15.57:0/3601047770 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f887c0cbd20 0x7f887c1562c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:44:43.012 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.002+0000 7f8879d9b640 1 --2- 172.21.15.57:0/3601047770 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f887c0b0530 0x7f887c149420 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f887000c400 tx=0x7f887000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:43.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.002+0000 7f886affd640 1 -- 172.21.15.57:0/3601047770 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8870004230 con 0x7f887c0b0530 2023-10-19T12:44:43.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.002+0000 7f886affd640 1 -- 172.21.15.57:0/3601047770 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f88700044a0 con 0x7f887c0b0530 2023-10-19T12:44:43.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.002+0000 7f8880d95640 1 -- 172.21.15.57:0/3601047770 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8874009840 con 0x7f887c0b0530 2023-10-19T12:44:43.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.002+0000 7f886affd640 1 -- 172.21.15.57:0/3601047770 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8870003850 con 0x7f887c0b0530 2023-10-19T12:44:43.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.002+0000 7f8880d95640 1 -- 172.21.15.57:0/3601047770 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f887c15a960 con 0x7f887c0b0530 2023-10-19T12:44:43.014 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.003+0000 7f8880d95640 1 -- 172.21.15.57:0/3601047770 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f887c14f4f0 con 0x7f887c0b0530 2023-10-19T12:44:43.014 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.003+0000 7f886affd640 1 -- 172.21.15.57:0/3601047770 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f88700039b0 con 0x7f887c0b0530 2023-10-19T12:44:43.014 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.003+0000 7f886affd640 1 --2- 172.21.15.57:0/3601047770 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f885c06ae40 0x7f885c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:43.014 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.004+0000 7f886affd640 1 -- 172.21.15.57:0/3601047770 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f8870089120 con 0x7f887c0b0530 2023-10-19T12:44:43.014 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.004+0000 7f887a59c640 1 --2- 172.21.15.57:0/3601047770 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f885c06ae40 0x7f885c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:43.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.004+0000 7f887a59c640 1 --2- 172.21.15.57:0/3601047770 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f885c06ae40 0x7f885c06d300 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f887c1585e0 tx=0x7f886c005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:43.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.008+0000 7f886affd640 1 -- 172.21.15.57:0/3601047770 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f887c14f4f0 con 0x7f887c0b0530 2023-10-19T12:44:43.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.234+0000 7f8880d95640 1 -- 172.21.15.57:0/3601047770 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f887c14f4f0 con 0x7f887c0b0530 2023-10-19T12:44:43.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.237+0000 7f886affd640 1 -- 172.21.15.57:0/3601047770 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+171036 (secure 0 0 0) 0x7f887c14f4f0 con 0x7f887c0b0530 2023-10-19T12:44:43.243 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:44:43.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.239+0000 7f8880d95640 1 -- 172.21.15.57:0/3601047770 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f885c06ae40 msgr2=0x7f885c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:43.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.239+0000 7f8880d95640 1 --2- 172.21.15.57:0/3601047770 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f885c06ae40 0x7f885c06d300 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f887c1585e0 tx=0x7f886c005f90 comp rx=0 tx=0).stop 2023-10-19T12:44:43.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.239+0000 7f8880d95640 1 -- 172.21.15.57:0/3601047770 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f887c0b0530 msgr2=0x7f887c149420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:43.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.239+0000 7f8880d95640 1 --2- 172.21.15.57:0/3601047770 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f887c0b0530 0x7f887c149420 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f887000c400 tx=0x7f887000c8c0 comp rx=0 tx=0).stop 2023-10-19T12:44:43.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.240+0000 7f8880d95640 1 -- 172.21.15.57:0/3601047770 shutdown_connections 2023-10-19T12:44:43.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.240+0000 7f8880d95640 1 --2- 172.21.15.57:0/3601047770 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f885c06ae40 0x7f885c06d300 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:43.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.240+0000 7f8880d95640 1 --2- 172.21.15.57:0/3601047770 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f887c0cbd20 0x7f887c1562c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:43.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.240+0000 7f8880d95640 1 --2- 172.21.15.57:0/3601047770 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f887c0b0530 0x7f887c149420 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:43.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.240+0000 7f8880d95640 1 --2- 172.21.15.57:0/3601047770 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f887c0afa20 0x7f887c150220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:43.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.240+0000 7f8880d95640 1 -- 172.21.15.57:0/3601047770 >> 172.21.15.57:0/3601047770 conn(0x7f887c069930 msgr2=0x7f887c0ac590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:43.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.240+0000 7f8880d95640 1 -- 172.21.15.57:0/3601047770 shutdown_connections 2023-10-19T12:44:43.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.240+0000 7f8880d95640 1 -- 172.21.15.57:0/3601047770 wait complete. 2023-10-19T12:44:43.253 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":5,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":2609646285},{"type":"v1","addr":"172.21.15.179:6833","nonce":2609646285}]},"active_addr":"172.21.15.179:6833/2609646285","active_change":"2023-10-19T12:43:57.303706+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":4106,"name":"y","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.179:0","nonce":515590766}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.179:0","nonce":2223195228}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.179:0","nonce":1377585784}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.179:0","nonce":2313691653}]}]} 2023-10-19T12:44:43.257 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-10-19T12:44:43.257 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-10-19T12:44:43.258 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T12:44:43.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.364+0000 7fa8a57a9640 1 -- 172.21.15.57:0/341626530 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa8a00ab500 msgr2=0x7fa8a00ab990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:43.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.364+0000 7fa8a57a9640 1 --2- 172.21.15.57:0/341626530 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa8a00ab500 0x7fa8a00ab990 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa8a00ca260 tx=0x7fa89002c110 comp rx=0 tx=0).stop 2023-10-19T12:44:43.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.364+0000 7fa8a57a9640 1 -- 172.21.15.57:0/341626530 shutdown_connections 2023-10-19T12:44:43.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.364+0000 7fa8a57a9640 1 --2- 172.21.15.57:0/341626530 >> v2:172.21.15.179:3300/0 conn(0x7fa8a00c9dc0 0x7fa8a006b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:43.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.364+0000 7fa8a57a9640 1 --2- 172.21.15.57:0/341626530 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa8a00ab500 0x7fa8a00ab990 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:43.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.364+0000 7fa8a57a9640 1 -- 172.21.15.57:0/341626530 >> 172.21.15.57:0/341626530 conn(0x7fa8a0069930 msgr2=0x7fa8a0069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:43.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.364+0000 7fa8a57a9640 1 -- 172.21.15.57:0/341626530 shutdown_connections 2023-10-19T12:44:43.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.364+0000 7fa8a57a9640 1 -- 172.21.15.57:0/341626530 wait complete. 2023-10-19T12:44:43.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.365+0000 7fa8a57a9640 1 Processor -- start 2023-10-19T12:44:43.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.365+0000 7fa8a57a9640 1 -- start start 2023-10-19T12:44:43.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.366+0000 7fa8a57a9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa8a006af00 0x7fa8a00a64b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:43.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.366+0000 7fa8a57a9640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa8a00ab500 0x7fa8a014ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:43.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.366+0000 7fa8a57a9640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa8a00c9dc0 0x7fa8a0152500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:43.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.366+0000 7fa8a57a9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a00af5f0 con 0x7fa8a006af00 2023-10-19T12:44:43.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.366+0000 7fa8a57a9640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a00a69f0 con 0x7fa8a00ab500 2023-10-19T12:44:43.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.366+0000 7fa8a57a9640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a00ae700 con 0x7fa8a00c9dc0 2023-10-19T12:44:43.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.366+0000 7fa89effd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa8a006af00 0x7fa8a00a64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:43.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.366+0000 7fa89effd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa8a006af00 0x7fa8a00a64b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50578/0 (socket says 172.21.15.57:50578) 2023-10-19T12:44:43.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.366+0000 7fa89effd640 1 -- 172.21.15.57:0/1010548440 learned_addr learned my addr 172.21.15.57:0/1010548440 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:43.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.366+0000 7fa89f7fe640 1 --2- 172.21.15.57:0/1010548440 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa8a00c9dc0 0x7fa8a0152500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:43.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.366+0000 7fa89e7fc640 1 --2- 172.21.15.57:0/1010548440 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa8a00ab500 0x7fa8a014ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:43.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.366+0000 7fa89effd640 1 -- 172.21.15.57:0/1010548440 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa8a00c9dc0 msgr2=0x7fa8a0152500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:43.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.367+0000 7fa89effd640 1 --2- 172.21.15.57:0/1010548440 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa8a00c9dc0 0x7fa8a0152500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:43.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.367+0000 7fa89effd640 1 -- 172.21.15.57:0/1010548440 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa8a00ab500 msgr2=0x7fa8a014ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:43.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.367+0000 7fa89effd640 1 --2- 172.21.15.57:0/1010548440 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa8a00ab500 0x7fa8a014ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:43.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.367+0000 7fa89effd640 1 -- 172.21.15.57:0/1010548440 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa890009840 con 0x7fa8a006af00 2023-10-19T12:44:43.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.367+0000 7fa89e7fc640 1 --2- 172.21.15.57:0/1010548440 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa8a00ab500 0x7fa8a014ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:44:43.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.367+0000 7fa89effd640 1 --2- 172.21.15.57:0/1010548440 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa8a006af00 0x7fa8a00a64b0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fa89800c430 tx=0x7fa89800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:43.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.367+0000 7fa8977fe640 1 -- 172.21.15.57:0/1010548440 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa898004230 con 0x7fa8a006af00 2023-10-19T12:44:43.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.367+0000 7fa8977fe640 1 -- 172.21.15.57:0/1010548440 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fa8980044a0 con 0x7fa8a006af00 2023-10-19T12:44:43.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.367+0000 7fa8a57a9640 1 -- 172.21.15.57:0/1010548440 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8a00ae9e0 con 0x7fa8a006af00 2023-10-19T12:44:43.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.367+0000 7fa8977fe640 1 -- 172.21.15.57:0/1010548440 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa898003850 con 0x7fa8a006af00 2023-10-19T12:44:43.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.367+0000 7fa8a57a9640 1 -- 172.21.15.57:0/1010548440 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8a00aefa0 con 0x7fa8a006af00 2023-10-19T12:44:43.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.368+0000 7fa8a57a9640 1 -- 172.21.15.57:0/1010548440 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8a00690e0 con 0x7fa8a006af00 2023-10-19T12:44:43.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.368+0000 7fa8977fe640 1 -- 172.21.15.57:0/1010548440 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fa8980039b0 con 0x7fa8a006af00 2023-10-19T12:44:43.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.369+0000 7fa8977fe640 1 --2- 172.21.15.57:0/1010548440 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fa88006ae40 0x7fa88006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:43.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.369+0000 7fa8977fe640 1 -- 172.21.15.57:0/1010548440 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fa898089110 con 0x7fa8a006af00 2023-10-19T12:44:43.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.369+0000 7fa89e7fc640 1 --2- 172.21.15.57:0/1010548440 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fa88006ae40 0x7fa88006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:43.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.369+0000 7fa89e7fc640 1 --2- 172.21.15.57:0/1010548440 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fa88006ae40 0x7fa88006d300 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fa8a0130ff0 tx=0x7fa888005c90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:43.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.373+0000 7fa8977fe640 1 -- 172.21.15.57:0/1010548440 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa898053ed0 con 0x7fa8a006af00 2023-10-19T12:44:43.442 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T12:44:43.445 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T12:44:43.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.550+0000 7fa8a57a9640 1 -- 172.21.15.57:0/1010548440 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa8a00cc420 con 0x7fa8a006af00 2023-10-19T12:44:43.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.551+0000 7fa8977fe640 1 -- 172.21.15.57:0/1010548440 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18358 (secure 0 0 0) 0x7fa898017090 con 0x7fa8a006af00 2023-10-19T12:44:43.555 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:44:43.555 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":25,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T12:44:23.472399+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-19T12:44:16.517510+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-19T12:44:17.398297+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-19T12:44:21.129403+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T12:44:43.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.553+0000 7fa8a57a9640 1 -- 172.21.15.57:0/1010548440 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fa88006ae40 msgr2=0x7fa88006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:43.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.553+0000 7fa8a57a9640 1 --2- 172.21.15.57:0/1010548440 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fa88006ae40 0x7fa88006d300 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fa8a0130ff0 tx=0x7fa888005c90 comp rx=0 tx=0).stop 2023-10-19T12:44:43.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.553+0000 7fa8a57a9640 1 -- 172.21.15.57:0/1010548440 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa8a006af00 msgr2=0x7fa8a00a64b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:43.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.553+0000 7fa8a57a9640 1 --2- 172.21.15.57:0/1010548440 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa8a006af00 0x7fa8a00a64b0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fa89800c430 tx=0x7fa89800c8f0 comp rx=0 tx=0).stop 2023-10-19T12:44:43.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.553+0000 7fa8a57a9640 1 -- 172.21.15.57:0/1010548440 shutdown_connections 2023-10-19T12:44:43.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.553+0000 7fa8a57a9640 1 --2- 172.21.15.57:0/1010548440 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fa88006ae40 0x7fa88006d300 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:43.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.553+0000 7fa8a57a9640 1 --2- 172.21.15.57:0/1010548440 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa8a00c9dc0 0x7fa8a0152500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:43.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.553+0000 7fa8a57a9640 1 --2- 172.21.15.57:0/1010548440 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa8a00ab500 0x7fa8a014ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:43.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.553+0000 7fa8a57a9640 1 --2- 172.21.15.57:0/1010548440 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa8a006af00 0x7fa8a00a64b0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:43.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.553+0000 7fa8a57a9640 1 -- 172.21.15.57:0/1010548440 >> 172.21.15.57:0/1010548440 conn(0x7fa8a0069930 msgr2=0x7fa8a00cb020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:43.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.553+0000 7fa8a57a9640 1 -- 172.21.15.57:0/1010548440 shutdown_connections 2023-10-19T12:44:43.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.553+0000 7fa8a57a9640 1 -- 172.21.15.57:0/1010548440 wait complete. 2023-10-19T12:44:43.567 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-10-19T12:44:43.568 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T12:44:43.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.674+0000 7f01c3010640 1 -- 172.21.15.57:0/1787010906 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f01bc0cf340 msgr2=0x7f01bc0d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:43.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.674+0000 7f01c3010640 1 --2- 172.21.15.57:0/1787010906 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f01bc0cf340 0x7f01bc0d1910 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f01bc08a120 tx=0x7f01b402c260 comp rx=0 tx=0).stop 2023-10-19T12:44:43.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.674+0000 7f01c3010640 1 -- 172.21.15.57:0/1787010906 shutdown_connections 2023-10-19T12:44:43.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.674+0000 7f01c3010640 1 --2- 172.21.15.57:0/1787010906 >> v2:172.21.15.179:3300/0 conn(0x7f01bc15b2a0 0x7f01bc0cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:43.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.674+0000 7f01c3010640 1 --2- 172.21.15.57:0/1787010906 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f01bc0cf340 0x7f01bc0d1910 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:43.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.674+0000 7f01c3010640 1 -- 172.21.15.57:0/1787010906 >> 172.21.15.57:0/1787010906 conn(0x7f01bc0d0d60 msgr2=0x7f01bc0d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:43.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.674+0000 7f01c3010640 1 -- 172.21.15.57:0/1787010906 shutdown_connections 2023-10-19T12:44:43.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.674+0000 7f01c3010640 1 -- 172.21.15.57:0/1787010906 wait complete. 2023-10-19T12:44:43.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.675+0000 7f01c3010640 1 Processor -- start 2023-10-19T12:44:43.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.675+0000 7f01c3010640 1 -- start start 2023-10-19T12:44:43.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.676+0000 7f01c3010640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f01bc0ceb40 0x7f01bc155c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:43.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.676+0000 7f01c3010640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f01bc0cf340 0x7f01bc0c7e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:43.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.676+0000 7f01c3010640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f01bc15b2a0 0x7f01bc0b6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:43.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.676+0000 7f01c3010640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01bc0ca900 con 0x7f01bc0cf340 2023-10-19T12:44:43.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.676+0000 7f01c3010640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01bc156190 con 0x7f01bc0ceb40 2023-10-19T12:44:43.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.676+0000 7f01c3010640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01bc0c9fe0 con 0x7f01bc15b2a0 2023-10-19T12:44:43.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.676+0000 7f01bbfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f01bc0cf340 0x7f01bc0c7e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:43.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.676+0000 7f01bbfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f01bc0cf340 0x7f01bc0c7e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50588/0 (socket says 172.21.15.57:50588) 2023-10-19T12:44:43.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.676+0000 7f01bbfff640 1 -- 172.21.15.57:0/3046064230 learned_addr learned my addr 172.21.15.57:0/3046064230 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:43.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.677+0000 7f01c15ad640 1 --2- 172.21.15.57:0/3046064230 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f01bc15b2a0 0x7f01bc0b6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:43.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.677+0000 7f01c0dac640 1 --2- 172.21.15.57:0/3046064230 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f01bc0ceb40 0x7f01bc155c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:43.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.677+0000 7f01bbfff640 1 -- 172.21.15.57:0/3046064230 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f01bc15b2a0 msgr2=0x7f01bc0b6100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:43.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.677+0000 7f01bbfff640 1 --2- 172.21.15.57:0/3046064230 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f01bc15b2a0 0x7f01bc0b6100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:43.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.677+0000 7f01bbfff640 1 -- 172.21.15.57:0/3046064230 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f01bc0ceb40 msgr2=0x7f01bc155c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:43.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.677+0000 7f01bbfff640 1 --2- 172.21.15.57:0/3046064230 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f01bc0ceb40 0x7f01bc155c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:43.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.677+0000 7f01bbfff640 1 -- 172.21.15.57:0/3046064230 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01b4009900 con 0x7f01bc0cf340 2023-10-19T12:44:43.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.677+0000 7f01bbfff640 1 --2- 172.21.15.57:0/3046064230 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f01bc0cf340 0x7f01bc0c7e30 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f01b000d6d0 tx=0x7f01b000db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:43.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.677+0000 7f01b97fa640 1 -- 172.21.15.57:0/3046064230 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b0005220 con 0x7f01bc0cf340 2023-10-19T12:44:43.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.677+0000 7f01b97fa640 1 -- 172.21.15.57:0/3046064230 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f01b0005490 con 0x7f01bc0cf340 2023-10-19T12:44:43.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.677+0000 7f01b97fa640 1 -- 172.21.15.57:0/3046064230 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b0009a80 con 0x7f01bc0cf340 2023-10-19T12:44:43.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.677+0000 7f01c3010640 1 -- 172.21.15.57:0/3046064230 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01bc0ca280 con 0x7f01bc0cf340 2023-10-19T12:44:43.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.677+0000 7f01c3010640 1 -- 172.21.15.57:0/3046064230 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01bc0c8860 con 0x7f01bc0cf340 2023-10-19T12:44:43.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.678+0000 7f01c3010640 1 -- 172.21.15.57:0/3046064230 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01bc0d04f0 con 0x7f01bc0cf340 2023-10-19T12:44:43.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.679+0000 7f01b97fa640 1 -- 172.21.15.57:0/3046064230 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f01b00047a0 con 0x7f01bc0cf340 2023-10-19T12:44:43.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.679+0000 7f01b97fa640 1 --2- 172.21.15.57:0/3046064230 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f01a806ae40 0x7f01a806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:43.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.679+0000 7f01b97fa640 1 -- 172.21.15.57:0/3046064230 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f01b0089c70 con 0x7f01bc0cf340 2023-10-19T12:44:43.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.679+0000 7f01c0dac640 1 --2- 172.21.15.57:0/3046064230 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f01a806ae40 0x7f01a806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:43.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.680+0000 7f01c0dac640 1 --2- 172.21.15.57:0/3046064230 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f01a806ae40 0x7f01a806d300 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f01b4036000 tx=0x7f01b4004ab0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:43.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.683+0000 7f01b97fa640 1 -- 172.21.15.57:0/3046064230 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f01b0054a20 con 0x7f01bc0cf340 2023-10-19T12:44:43.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.859+0000 7f01c3010640 1 -- 172.21.15.57:0/3046064230 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f01bc0677b0 con 0x7f01bc0cf340 2023-10-19T12:44:43.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.860+0000 7f01b97fa640 1 -- 172.21.15.57:0/3046064230 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18358 (secure 0 0 0) 0x7f01b0017020 con 0x7f01bc0cf340 2023-10-19T12:44:43.864 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:44:43.864 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":25,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T12:44:23.472399+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-19T12:44:16.517510+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-19T12:44:17.398297+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-19T12:44:21.129403+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T12:44:43.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.862+0000 7f01c3010640 1 -- 172.21.15.57:0/3046064230 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f01a806ae40 msgr2=0x7f01a806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:43.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.862+0000 7f01c3010640 1 --2- 172.21.15.57:0/3046064230 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f01a806ae40 0x7f01a806d300 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f01b4036000 tx=0x7f01b4004ab0 comp rx=0 tx=0).stop 2023-10-19T12:44:43.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.862+0000 7f01c3010640 1 -- 172.21.15.57:0/3046064230 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f01bc0cf340 msgr2=0x7f01bc0c7e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:43.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.862+0000 7f01c3010640 1 --2- 172.21.15.57:0/3046064230 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f01bc0cf340 0x7f01bc0c7e30 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f01b000d6d0 tx=0x7f01b000db90 comp rx=0 tx=0).stop 2023-10-19T12:44:43.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.862+0000 7f01c3010640 1 -- 172.21.15.57:0/3046064230 shutdown_connections 2023-10-19T12:44:43.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.862+0000 7f01c3010640 1 --2- 172.21.15.57:0/3046064230 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f01a806ae40 0x7f01a806d300 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:43.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.862+0000 7f01c3010640 1 --2- 172.21.15.57:0/3046064230 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f01bc15b2a0 0x7f01bc0b6100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:43.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.862+0000 7f01c3010640 1 --2- 172.21.15.57:0/3046064230 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f01bc0cf340 0x7f01bc0c7e30 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:43.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.862+0000 7f01c3010640 1 --2- 172.21.15.57:0/3046064230 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f01bc0ceb40 0x7f01bc155c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:43.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.862+0000 7f01c3010640 1 -- 172.21.15.57:0/3046064230 >> 172.21.15.57:0/3046064230 conn(0x7f01bc0d0d60 msgr2=0x7f01bc0b2810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:43.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.862+0000 7f01c3010640 1 -- 172.21.15.57:0/3046064230 shutdown_connections 2023-10-19T12:44:43.868 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:43.862+0000 7f01c3010640 1 -- 172.21.15.57:0/3046064230 wait complete. 2023-10-19T12:44:43.876 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-10-19T12:44:43.877 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-10-19T12:44:43.878 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-10-19T12:44:43.878 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-10-19T12:44:43.879 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-10-19T12:44:43.879 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-10-19T12:44:43.880 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2023-10-19T12:44:43.881 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2023-10-19T12:44:44.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.062+0000 7f435b41a640 1 -- 172.21.15.57:0/833388640 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f434c054230 msgr2=0x7f434c04f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:44.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.062+0000 7f435b41a640 1 --2- 172.21.15.57:0/833388640 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f434c054230 0x7f434c04f260 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f4354077310 tx=0x7f4354092d50 comp rx=0 tx=0).stop 2023-10-19T12:44:44.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.062+0000 7f435b41a640 1 -- 172.21.15.57:0/833388640 shutdown_connections 2023-10-19T12:44:44.067 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.062+0000 7f435b41a640 1 --2- 172.21.15.57:0/833388640 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f434c0338f0 0x7f434c053cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.067 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.062+0000 7f435b41a640 1 --2- 172.21.15.57:0/833388640 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f434c054230 0x7f434c04f260 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.067 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.062+0000 7f435b41a640 1 -- 172.21.15.57:0/833388640 >> 172.21.15.57:0/833388640 conn(0x7f434c009f80 msgr2=0x7f434c00a360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:44.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.063+0000 7f435b41a640 1 -- 172.21.15.57:0/833388640 shutdown_connections 2023-10-19T12:44:44.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.063+0000 7f435b41a640 1 -- 172.21.15.57:0/833388640 wait complete. 2023-10-19T12:44:44.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.065+0000 7f435b41a640 1 Processor -- start 2023-10-19T12:44:44.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.065+0000 7f435b41a640 1 -- start start 2023-10-19T12:44:44.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.067+0000 7f435b41a640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f434c0338f0 0x7f434c04cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:44.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.067+0000 7f435b41a640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f434c054230 0x7f434c036ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:44.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.067+0000 7fdca9763640 1 -- 172.21.15.57:0/4201237734 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdca40b15a0 msgr2=0x7fdca40af760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:44.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.067+0000 7fdca9763640 1 --2- 172.21.15.57:0/4201237734 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdca40b15a0 0x7fdca40af760 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fdc9c005fd0 tx=0x7fdc9c02c170 comp rx=0 tx=0).stop 2023-10-19T12:44:44.072 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.067+0000 7f435b41a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f434c100fe0 0x7f434c04e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:44.072 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.067+0000 7fdca9763640 1 -- 172.21.15.57:0/4201237734 shutdown_connections 2023-10-19T12:44:44.072 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.067+0000 7fdca9763640 1 --2- 172.21.15.57:0/4201237734 >> v2:172.21.15.179:3300/0 conn(0x7fdca415b880 0x7fdca4157680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.073 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.067+0000 7fdca9763640 1 --2- 172.21.15.57:0/4201237734 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdca40b15a0 0x7fdca40af760 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.073 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.067+0000 7fdca9763640 1 -- 172.21.15.57:0/4201237734 >> 172.21.15.57:0/4201237734 conn(0x7fdca40ae460 msgr2=0x7fdca40ae840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:44.073 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.067+0000 7f435a418640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f434c0338f0 0x7f434c04cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:44.074 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.067+0000 7fac86f6c640 1 -- 172.21.15.57:0/116929757 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fac8006a9a0 msgr2=0x7fac800cb720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:44.074 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.067+0000 7fac86f6c640 1 --2- 172.21.15.57:0/116929757 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fac8006a9a0 0x7fac800cb720 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fac80066c90 tx=0x7fac7802c0a0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.067+0000 7f6aa1b4c640 1 -- 172.21.15.57:0/3236260661 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6a9c158af0 msgr2=0x7f6a9c06c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:44.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.067+0000 7f6aa1b4c640 1 --2- 172.21.15.57:0/3236260661 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6a9c158af0 0x7f6a9c06c230 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6a9c06a700 tx=0x7f6a8c02bcf0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.067+0000 7f6aa1b4c640 1 -- 172.21.15.57:0/3236260661 shutdown_connections 2023-10-19T12:44:44.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.067+0000 7f6aa1b4c640 1 --2- 172.21.15.57:0/3236260661 >> v2:172.21.15.57:3300/0 conn(0x7f6a9c06c770 0x7f6a9c0c9770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.067+0000 7f6aa1b4c640 1 --2- 172.21.15.57:0/3236260661 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6a9c158af0 0x7f6a9c06c230 secure :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6a9c06a700 tx=0x7f6a8c02bcf0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.067+0000 7f6aa1b4c640 1 --2- 172.21.15.57:0/3236260661 >> v2:172.21.15.179:3300/0 conn(0x7f6a9c13d450 0x7f6a9c1585b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.067+0000 7f6aa1b4c640 1 -- 172.21.15.57:0/3236260661 >> 172.21.15.57:0/3236260661 conn(0x7f6a9c069930 msgr2=0x7f6a9c069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:44.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.067+0000 7fdca9763640 1 -- 172.21.15.57:0/4201237734 shutdown_connections 2023-10-19T12:44:44.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.067+0000 7fac86f6c640 1 -- 172.21.15.57:0/116929757 shutdown_connections 2023-10-19T12:44:44.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.067+0000 7fac86f6c640 1 --2- 172.21.15.57:0/116929757 >> v2:172.21.15.57:3300/0 conn(0x7fac8006ad80 0x7fac8015e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.067+0000 7fac86f6c640 1 --2- 172.21.15.57:0/116929757 >> v2:172.21.15.179:3300/0 conn(0x7fac800c8990 0x7fac800cbc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.067+0000 7fac86f6c640 1 --2- 172.21.15.57:0/116929757 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fac8006a9a0 0x7fac800cb720 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.067+0000 7fac86f6c640 1 -- 172.21.15.57:0/116929757 >> 172.21.15.57:0/116929757 conn(0x7fac800c9a50 msgr2=0x7fac800c9e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:44.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.067+0000 7f435a418640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f434c0338f0 0x7f434c04cc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3301/0 says I am v2:172.21.15.57:48016/0 (socket says 172.21.15.57:48016) 2023-10-19T12:44:44.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.067+0000 7f435a418640 1 -- 172.21.15.57:0/86828474 learned_addr learned my addr 172.21.15.57:0/86828474 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:44.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.068+0000 7fac86f6c640 1 -- 172.21.15.57:0/116929757 shutdown_connections 2023-10-19T12:44:44.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.067+0000 7f435b41a640 1 -- 172.21.15.57:0/86828474 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f434c04bc00 con 0x7f434c100fe0 2023-10-19T12:44:44.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.067+0000 7f435b41a640 1 -- 172.21.15.57:0/86828474 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f434c04bee0 con 0x7f434c054230 2023-10-19T12:44:44.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.068+0000 7fc55cc5d640 1 -- 172.21.15.57:0/3552890274 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc5580cd1a0 msgr2=0x7fc5580b17e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:44.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.068+0000 7fc55cc5d640 1 --2- 172.21.15.57:0/3552890274 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc5580cd1a0 0x7fc5580b17e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fc5580cc5d0 tx=0x7fc55002cce0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.068+0000 7f6aa1b4c640 1 -- 172.21.15.57:0/3236260661 shutdown_connections 2023-10-19T12:44:44.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.068+0000 7f6aa1b4c640 1 -- 172.21.15.57:0/3236260661 wait complete. 2023-10-19T12:44:44.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.067+0000 7f435b41a640 1 -- 172.21.15.57:0/86828474 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f434c04c1c0 con 0x7f434c0338f0 2023-10-19T12:44:44.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.068+0000 7f435a418640 1 -- 172.21.15.57:0/86828474 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f434c054230 msgr2=0x7f434c036ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:44.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.068+0000 7f435a418640 1 --2- 172.21.15.57:0/86828474 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f434c054230 0x7f434c036ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.068+0000 7f435a418640 1 -- 172.21.15.57:0/86828474 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f434c100fe0 msgr2=0x7f434c04e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:44.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.068+0000 7f435a418640 1 --2- 172.21.15.57:0/86828474 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f434c100fe0 0x7f434c04e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.068+0000 7f435a418640 1 -- 172.21.15.57:0/86828474 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f435404e960 con 0x7f434c0338f0 2023-10-19T12:44:44.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.068+0000 7f435a418640 1 --2- 172.21.15.57:0/86828474 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f434c0338f0 0x7f434c04cc80 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f435000c3c0 tx=0x7f435000c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:44.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.068+0000 7fac86f6c640 1 -- 172.21.15.57:0/116929757 wait complete. 2023-10-19T12:44:44.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.069+0000 7fc55cc5d640 1 -- 172.21.15.57:0/3552890274 shutdown_connections 2023-10-19T12:44:44.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.069+0000 7fc55cc5d640 1 --2- 172.21.15.57:0/3552890274 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc5580cd1a0 0x7fc5580b17e0 secure :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fc5580cc5d0 tx=0x7fc55002cce0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.069+0000 7fc55cc5d640 1 --2- 172.21.15.57:0/3552890274 >> v2:172.21.15.57:3300/0 conn(0x7fc558156e30 0x7fc55806f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.069+0000 7fc55cc5d640 1 -- 172.21.15.57:0/3552890274 >> 172.21.15.57:0/3552890274 conn(0x7fc558069900 msgr2=0x7fc558069ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:44.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.069+0000 7fddc359e640 1 -- 172.21.15.57:0/949091027 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fddac003970 msgr2=0x7fddac003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:44.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.069+0000 7fddc359e640 1 --2- 172.21.15.57:0/949091027 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fddac003970 0x7fddac003d90 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fddb80099d0 tx=0x7fddac0057f0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.069+0000 7f434affd640 1 -- 172.21.15.57:0/86828474 <== mon.2 v2:172.21.15.179:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4350011790 con 0x7f434c0338f0 2023-10-19T12:44:44.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.069+0000 7f5c7c765640 1 -- 172.21.15.57:0/3596742939 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5c740aee00 msgr2=0x7f5c7406cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:44.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.069+0000 7f5c7c765640 1 --2- 172.21.15.57:0/3596742939 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5c740aee00 0x7f5c7406cef0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f5c740ce1f0 tx=0x7f5c6402bd90 comp rx=0 tx=0).stop 2023-10-19T12:44:44.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.069+0000 7fc55cc5d640 1 -- 172.21.15.57:0/3552890274 shutdown_connections 2023-10-19T12:44:44.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.067+0000 7fdca9763640 1 -- 172.21.15.57:0/4201237734 wait complete. 2023-10-19T12:44:44.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.069+0000 7fddc359e640 1 -- 172.21.15.57:0/949091027 shutdown_connections 2023-10-19T12:44:44.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.069+0000 7fddc359e640 1 --2- 172.21.15.57:0/949091027 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fddac003970 0x7fddac003d90 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.069+0000 7fddc359e640 1 -- 172.21.15.57:0/949091027 >> 172.21.15.57:0/949091027 conn(0x7fddbc0559b0 msgr2=0x7fddbc055dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:44.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.069+0000 7f435b41a640 1 -- 172.21.15.57:0/86828474 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f434c04c4a0 con 0x7f434c0338f0 2023-10-19T12:44:44.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.069+0000 7f435b41a640 1 -- 172.21.15.57:0/86828474 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f434c0370e0 con 0x7f434c0338f0 2023-10-19T12:44:44.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.069+0000 7fdca9763640 1 Processor -- start 2023-10-19T12:44:44.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.069+0000 7f5c7c765640 1 -- 172.21.15.57:0/3596742939 shutdown_connections 2023-10-19T12:44:44.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.069+0000 7f5c7c765640 1 --2- 172.21.15.57:0/3596742939 >> v2:172.21.15.57:3300/0 conn(0x7f5c7406d430 0x7f5c7415e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.069+0000 7f5c7c765640 1 --2- 172.21.15.57:0/3596742939 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5c740aee00 0x7f5c7406cef0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.069+0000 7f434affd640 1 -- 172.21.15.57:0/86828474 <== mon.2 v2:172.21.15.179:3301/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f43500119e0 con 0x7f434c0338f0 2023-10-19T12:44:44.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.069+0000 7f434affd640 1 -- 172.21.15.57:0/86828474 <== mon.2 v2:172.21.15.179:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4350011e10 con 0x7f434c0338f0 2023-10-19T12:44:44.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.069+0000 7f5c7c765640 1 --2- 172.21.15.57:0/3596742939 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5c7415ae10 0x7f5c74069720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.069+0000 7f5c7c765640 1 -- 172.21.15.57:0/3596742939 >> 172.21.15.57:0/3596742939 conn(0x7f5c740b1f60 msgr2=0x7f5c740b2620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:44.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.069+0000 7fc55cc5d640 1 -- 172.21.15.57:0/3552890274 wait complete. 2023-10-19T12:44:44.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.069+0000 7fac86f6c640 1 Processor -- start 2023-10-19T12:44:44.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.069+0000 7fddc359e640 1 -- 172.21.15.57:0/949091027 shutdown_connections 2023-10-19T12:44:44.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.069+0000 7fac86f6c640 1 -- start start 2023-10-19T12:44:44.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.070+0000 7fdca9763640 1 -- start start 2023-10-19T12:44:44.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.069+0000 7fddc359e640 1 -- 172.21.15.57:0/949091027 wait complete. 2023-10-19T12:44:44.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.070+0000 7f6aa1b4c640 1 Processor -- start 2023-10-19T12:44:44.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.070+0000 7f5c7c765640 1 -- 172.21.15.57:0/3596742939 shutdown_connections 2023-10-19T12:44:44.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.071+0000 7f5c7c765640 1 -- 172.21.15.57:0/3596742939 wait complete. 2023-10-19T12:44:44.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.071+0000 7fac86f6c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac8006a9a0 0x7fac80158230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:44.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.071+0000 7f4f8c58e640 1 -- 172.21.15.57:0/417607619 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4f8415f440 msgr2=0x7f4f840af210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:44.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.071+0000 7f4f8c58e640 1 --2- 172.21.15.57:0/417607619 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4f8415f440 0x7f4f840af210 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f4f74009830 tx=0x7f4f7402bd50 comp rx=0 tx=0).stop 2023-10-19T12:44:44.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.071+0000 7fc55cc5d640 1 Processor -- start 2023-10-19T12:44:44.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.071+0000 7fac86f6c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fac8006ad80 0x7fac80098460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:44.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.071+0000 7fdca9763640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdca40b15a0 0x7fdca4140f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:44.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.071+0000 7fc55cc5d640 1 -- start start 2023-10-19T12:44:44.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.071+0000 7fdca9763640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdca4157220 0x7fdca40a6ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:44.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.071+0000 7fac86f6c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fac800c8990 0x7fac8009d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:44.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.071+0000 7fac86f6c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac80158770 con 0x7fac8006a9a0 2023-10-19T12:44:44.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.071+0000 7fac86f6c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac800c0c30 con 0x7fac8006ad80 2023-10-19T12:44:44.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.071+0000 7fac86f6c640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac800c0ed0 con 0x7fac800c8990 2023-10-19T12:44:44.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.071+0000 7fdca9763640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdca415b880 0x7fdca40acc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:44.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.071+0000 7f435b41a640 1 -- 172.21.15.57:0/86828474 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f433c000fc0 con 0x7f434c0338f0 2023-10-19T12:44:44.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.071+0000 7fdca9763640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdca40a6a10 con 0x7fdca4157220 2023-10-19T12:44:44.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.071+0000 7fdca9763640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdca40cb9f0 con 0x7fdca415b880 2023-10-19T12:44:44.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.071+0000 7fdca9763640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdca4141460 con 0x7fdca40b15a0 2023-10-19T12:44:44.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.071+0000 7fdca3fff640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdca40b15a0 0x7fdca4140f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:44.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.070+0000 7f6aa1b4c640 1 -- start start 2023-10-19T12:44:44.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.071+0000 7f6aa1b4c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6a9c06c770 0x7f6a9c162b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:44.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.071+0000 7f6aa1b4c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6a9c13d450 0x7f6a9c156290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:44.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.071+0000 7f6aa1b4c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6a9c1619d0 0x7f6a9c13ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:44.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.071+0000 7f6aa1b4c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a9c13c360 con 0x7f6a9c1619d0 2023-10-19T12:44:44.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.071+0000 7f6aa1b4c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a9c13a6c0 con 0x7f6a9c13d450 2023-10-19T12:44:44.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.071+0000 7f6aa1b4c640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a9c13c5a0 con 0x7f6a9c06c770 2023-10-19T12:44:44.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.072+0000 7fdca3fff640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdca40b15a0 0x7fdca4140f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3301/0 says I am v2:172.21.15.57:48018/0 (socket says 172.21.15.57:48018) 2023-10-19T12:44:44.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.072+0000 7fdca3fff640 1 -- 172.21.15.57:0/3424334509 learned_addr learned my addr 172.21.15.57:0/3424334509 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:44.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.072+0000 7fddc359e640 1 Processor -- start 2023-10-19T12:44:44.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.072+0000 7fddc359e640 1 -- start start 2023-10-19T12:44:44.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.072+0000 7f6a9bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6a9c1619d0 0x7f6a9c13ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:44.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.072+0000 7fac8676b640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fac800c8990 0x7fac8009d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:44.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.072+0000 7fac85769640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fac8006ad80 0x7fac80098460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:44.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.072+0000 7fac8676b640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fac800c8990 0x7fac8009d5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3301/0 says I am v2:172.21.15.57:48026/0 (socket says 172.21.15.57:48026) 2023-10-19T12:44:44.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.072+0000 7fac8676b640 1 -- 172.21.15.57:0/4105396125 learned_addr learned my addr 172.21.15.57:0/4105396125 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:44.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.072+0000 7f6a9b7fe640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6a9c06c770 0x7f6a9c162b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:44.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.072+0000 7fdca37fe640 1 --2- 172.21.15.57:0/3424334509 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdca4157220 0x7fdca40a6ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:44.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.072+0000 7fdca8f62640 1 --2- 172.21.15.57:0/3424334509 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdca415b880 0x7fdca40acc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:44.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.072+0000 7f6a9affd640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6a9c13d450 0x7f6a9c156290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:44.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.072+0000 7f6a9affd640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6a9c13d450 0x7f6a9c156290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.57:43152/0 (socket says 172.21.15.57:43152) 2023-10-19T12:44:44.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.072+0000 7f6a9affd640 1 -- 172.21.15.57:0/2246721187 learned_addr learned my addr 172.21.15.57:0/2246721187 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:44.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.072+0000 7f6a9affd640 1 -- 172.21.15.57:0/2246721187 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6a9c06c770 msgr2=0x7f6a9c162b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:44.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.072+0000 7fc55cc5d640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc558156a50 0x7fc558159e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:44.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.072+0000 7fc55cc5d640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc558156e30 0x7fc55814f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:44.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.072+0000 7f6a9affd640 1 --2- 172.21.15.57:0/2246721187 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6a9c06c770 0x7f6a9c162b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.073+0000 7fc55cc5d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5581eb550 0x7fc55806f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:44.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.072+0000 7f6a9affd640 1 -- 172.21.15.57:0/2246721187 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6a9c1619d0 msgr2=0x7f6a9c13ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:44.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.073+0000 7fc55cc5d640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5581f6030 con 0x7fc5581eb550 2023-10-19T12:44:44.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.073+0000 7f6a9b7fe640 1 --2- 172.21.15.57:0/2246721187 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6a9c06c770 0x7f6a9c162b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:44:44.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.073+0000 7f6a9affd640 1 --2- 172.21.15.57:0/2246721187 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6a9c1619d0 0x7f6a9c13ab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.073+0000 7f6a9affd640 1 -- 172.21.15.57:0/2246721187 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a8c009490 con 0x7f6a9c13d450 2023-10-19T12:44:44.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.073+0000 7f5c7c765640 1 Processor -- start 2023-10-19T12:44:44.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.073+0000 7f6a9bfff640 1 --2- 172.21.15.57:0/2246721187 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6a9c1619d0 0x7f6a9c13ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:44:44.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.073+0000 7f5c7c765640 1 -- start start 2023-10-19T12:44:44.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.073+0000 7fddc359e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fddac003970 0x7fddbc0e4380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:44.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.073+0000 7fc555d9b640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc558156e30 0x7fc55814f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:44.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.073+0000 7fc555d9b640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc558156e30 0x7fc55814f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3301/0 says I am v2:172.21.15.57:48034/0 (socket says 172.21.15.57:48034) 2023-10-19T12:44:44.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.073+0000 7fddc359e640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fddbc0d2b60 0x7fddbc0e5d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:44.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.073+0000 7fddc359e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fddbc0dffa0 0x7fddbc052140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:44.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.073+0000 7f434affd640 1 -- 172.21.15.57:0/86828474 <== mon.2 v2:172.21.15.179:3301/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f4350006890 con 0x7f434c0338f0 2023-10-19T12:44:44.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.073+0000 7fc555d9b640 1 -- 172.21.15.57:0/305170354 learned_addr learned my addr 172.21.15.57:0/305170354 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:44.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.073+0000 7fddc259c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fddac003970 0x7fddbc0e4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:44.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.073+0000 7fddc259c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fddac003970 0x7fddbc0e4380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50692/0 (socket says 172.21.15.57:50692) 2023-10-19T12:44:44.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.073+0000 7f4f8c58e640 1 -- 172.21.15.57:0/417607619 shutdown_connections 2023-10-19T12:44:44.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.073+0000 7f4f8c58e640 1 --2- 172.21.15.57:0/417607619 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4f8415f440 0x7f4f840af210 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.073+0000 7f4f8c58e640 1 --2- 172.21.15.57:0/417607619 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4f8415ae60 0x7f4f8415eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.073+0000 7f4f8c58e640 1 -- 172.21.15.57:0/417607619 >> 172.21.15.57:0/417607619 conn(0x7f4f840b1fd0 msgr2=0x7f4f840b2690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:44.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.073+0000 7fddc259c640 1 -- 172.21.15.57:0/3683187965 learned_addr learned my addr 172.21.15.57:0/3683187965 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:44.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.073+0000 7fc55cc5d640 1 -- 172.21.15.57:0/305170354 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc55814f5f0 con 0x7fc558156a50 2023-10-19T12:44:44.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.073+0000 7fc55cc5d640 1 -- 172.21.15.57:0/305170354 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5581ebd60 con 0x7fc558156e30 2023-10-19T12:44:44.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.073+0000 7f6a9affd640 1 --2- 172.21.15.57:0/2246721187 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6a9c13d450 0x7f6a9c156290 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f6a8c02c280 tx=0x7f6a8c003820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:44.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.073+0000 7fddc1d9b640 1 --2- 172.21.15.57:0/3683187965 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fddbc0d2b60 0x7fddbc0e5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:44.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.073+0000 7f434affd640 1 --2- 172.21.15.57:0/86828474 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f433806aec0 0x7f433806d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:44.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.073+0000 7fddc2d9d640 1 --2- 172.21.15.57:0/3683187965 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fddbc0dffa0 0x7fddbc052140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:44.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.074+0000 7fddc359e640 1 -- 172.21.15.57:0/3683187965 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddbc0df550 con 0x7fddac003970 2023-10-19T12:44:44.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.074+0000 7fddc359e640 1 -- 172.21.15.57:0/3683187965 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddbc0e48c0 con 0x7fddbc0dffa0 2023-10-19T12:44:44.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.073+0000 7fdca37fe640 1 -- 172.21.15.57:0/3424334509 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdca40b15a0 msgr2=0x7fdca4140f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:44.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.073+0000 7fdca37fe640 1 --2- 172.21.15.57:0/3424334509 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdca40b15a0 0x7fdca4140f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.073+0000 7fdca37fe640 1 -- 172.21.15.57:0/3424334509 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdca415b880 msgr2=0x7fdca40acc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:44.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.073+0000 7fdca37fe640 1 --2- 172.21.15.57:0/3424334509 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdca415b880 0x7fdca40acc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.073+0000 7fdca37fe640 1 -- 172.21.15.57:0/3424334509 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc9c009840 con 0x7fdca4157220 2023-10-19T12:44:44.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.074+0000 7fddc359e640 1 -- 172.21.15.57:0/3683187965 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddbc0e6240 con 0x7fddbc0d2b60 2023-10-19T12:44:44.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.074+0000 7fc55659c640 1 --2- 172.21.15.57:0/305170354 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc558156a50 0x7fc558159e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:44.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.073+0000 7f4f8c58e640 1 -- 172.21.15.57:0/417607619 shutdown_connections 2023-10-19T12:44:44.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.074+0000 7f434affd640 1 -- 172.21.15.57:0/86828474 <== mon.2 v2:172.21.15.179:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f4350089a80 con 0x7f434c0338f0 2023-10-19T12:44:44.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.074+0000 7f434affd640 1 --2- 172.21.15.57:0/86828474 >> [v2:172.21.15.57:6808/3729172421,v1:172.21.15.57:6809/3729172421] conn(0x7f4338074500 0x7f4338076940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:44.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.074+0000 7f434affd640 1 -- 172.21.15.57:0/86828474 --> [v2:172.21.15.57:6808/3729172421,v1:172.21.15.57:6809/3729172421] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4338077010 con 0x7f4338074500 2023-10-19T12:44:44.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.074+0000 7f434affd640 1 -- 172.21.15.57:0/86828474 <== mon.2 v2:172.21.15.179:3301/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4350089ea0 con 0x7f434c0338f0 2023-10-19T12:44:44.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.074+0000 7fac85769640 1 -- 172.21.15.57:0/4105396125 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fac800c8990 msgr2=0x7fac8009d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:44.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.074+0000 7fac85769640 1 --2- 172.21.15.57:0/4105396125 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fac800c8990 0x7fac8009d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.074+0000 7fac85769640 1 -- 172.21.15.57:0/4105396125 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac8006a9a0 msgr2=0x7fac80158230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:44.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.074+0000 7fac85769640 1 --2- 172.21.15.57:0/4105396125 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac8006a9a0 0x7fac80158230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.074+0000 7fac85769640 1 -- 172.21.15.57:0/4105396125 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac78009840 con 0x7fac8006ad80 2023-10-19T12:44:44.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.074+0000 7f6a83fff640 1 -- 172.21.15.57:0/2246721187 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a8c003c20 con 0x7f6a9c13d450 2023-10-19T12:44:44.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.074+0000 7f6a83fff640 1 -- 172.21.15.57:0/2246721187 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f6a8c003e70 con 0x7f6a9c13d450 2023-10-19T12:44:44.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.074+0000 7f6a83fff640 1 -- 172.21.15.57:0/2246721187 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a8c005260 con 0x7f6a9c13d450 2023-10-19T12:44:44.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.074+0000 7f4f8c58e640 1 -- 172.21.15.57:0/417607619 wait complete. 2023-10-19T12:44:44.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.074+0000 7fac85769640 1 --2- 172.21.15.57:0/4105396125 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fac8006ad80 0x7fac80098460 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fac7000e6e0 tx=0x7fac7000eba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:44.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.074+0000 7f6aa1b4c640 1 -- 172.21.15.57:0/2246721187 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a9c163070 con 0x7f6a9c13d450 2023-10-19T12:44:44.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.074+0000 7f6aa1b4c640 1 -- 172.21.15.57:0/2246721187 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a9c13c810 con 0x7f6a9c13d450 2023-10-19T12:44:44.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.074+0000 7fddc259c640 1 -- 172.21.15.57:0/3683187965 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fddbc0d2b60 msgr2=0x7fddbc0e5d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:44.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.074+0000 7fddc259c640 1 --2- 172.21.15.57:0/3683187965 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fddbc0d2b60 0x7fddbc0e5d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.074+0000 7fddc259c640 1 -- 172.21.15.57:0/3683187965 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fddbc0dffa0 msgr2=0x7fddbc052140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:44.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.075+0000 7fddc259c640 1 --2- 172.21.15.57:0/3683187965 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fddbc0dffa0 0x7fddbc052140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.075+0000 7fddc259c640 1 -- 172.21.15.57:0/3683187965 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddbc0e6510 con 0x7fddac003970 2023-10-19T12:44:44.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.075+0000 7fddc2d9d640 1 --2- 172.21.15.57:0/3683187965 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fddbc0dffa0 0x7fddbc052140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T12:44:44.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.074+0000 7f4359c17640 1 --2- 172.21.15.57:0/86828474 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f433806aec0 0x7f433806d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:44.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.074+0000 7fdca37fe640 1 --2- 172.21.15.57:0/3424334509 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdca4157220 0x7fdca40a6ec0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fdc9800c3a0 tx=0x7fdc9800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:44.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.074+0000 7fdca0ff9640 1 -- 172.21.15.57:0/3424334509 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc98011790 con 0x7fdca4157220 2023-10-19T12:44:44.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.075+0000 7fdca0ff9640 1 -- 172.21.15.57:0/3424334509 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fdc980119e0 con 0x7fdca4157220 2023-10-19T12:44:44.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.075+0000 7fdca0ff9640 1 -- 172.21.15.57:0/3424334509 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc98011df0 con 0x7fdca4157220 2023-10-19T12:44:44.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.074+0000 7f5c7c765640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5c7406d430 0x7f5c74150ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:44.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.074+0000 7f5c7c765640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5c740aee00 0x7f5c7415bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:44.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.074+0000 7f5c7c765640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5c7415ae10 0x7f5c7415c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:44.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.074+0000 7f5c7c765640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c7414bb40 con 0x7f5c7415ae10 2023-10-19T12:44:44.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.074+0000 7f5c7c765640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c7414f120 con 0x7f5c7406d430 2023-10-19T12:44:44.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.074+0000 7f5c7c765640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c7414f400 con 0x7f5c740aee00 2023-10-19T12:44:44.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.074+0000 7fc556d9d640 1 --2- 172.21.15.57:0/305170354 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5581eb550 0x7fc55806f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:44.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.074+0000 7fc555d9b640 1 -- 172.21.15.57:0/305170354 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc558156a50 msgr2=0x7fc558159e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:44.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.074+0000 7fc555d9b640 1 --2- 172.21.15.57:0/305170354 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc558156a50 0x7fc558159e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.074+0000 7fc555d9b640 1 -- 172.21.15.57:0/305170354 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5581eb550 msgr2=0x7fc55806f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:44.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.074+0000 7fc555d9b640 1 --2- 172.21.15.57:0/305170354 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5581eb550 0x7fc55806f8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.074+0000 7fc555d9b640 1 -- 172.21.15.57:0/305170354 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5500073b0 con 0x7fc558156e30 2023-10-19T12:44:44.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.074+0000 7fc556d9d640 1 --2- 172.21.15.57:0/305170354 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5581eb550 0x7fc55806f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:44:44.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.075+0000 7fc555d9b640 1 --2- 172.21.15.57:0/305170354 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc558156e30 0x7fc55814f0b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc55002d2f0 tx=0x7fc550008ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:44.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.074+0000 7f5c79d00640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5c740aee00 0x7f5c7415bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:44.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.074+0000 7f5c7ad02640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5c7415ae10 0x7f5c7415c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:44.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.074+0000 7f5c79d00640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5c740aee00 0x7f5c7415bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3301/0 says I am v2:172.21.15.57:48052/0 (socket says 172.21.15.57:48052) 2023-10-19T12:44:44.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.074+0000 7f5c79d00640 1 -- 172.21.15.57:0/1545656769 learned_addr learned my addr 172.21.15.57:0/1545656769 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:44.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.075+0000 7fddc259c640 1 --2- 172.21.15.57:0/3683187965 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fddac003970 0x7fddbc0e4380 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fddb80098f0 tx=0x7fddb802cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:44.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.075+0000 7fac767fc640 1 -- 172.21.15.57:0/4105396125 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac7000cab0 con 0x7fac8006ad80 2023-10-19T12:44:44.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.075+0000 7fac86f6c640 1 -- 172.21.15.57:0/4105396125 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac800c11b0 con 0x7fac8006ad80 2023-10-19T12:44:44.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.075+0000 7fac767fc640 1 -- 172.21.15.57:0/4105396125 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fac7000cd00 con 0x7fac8006ad80 2023-10-19T12:44:44.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.075+0000 7fac86f6c640 1 -- 172.21.15.57:0/4105396125 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac80098e20 con 0x7fac8006ad80 2023-10-19T12:44:44.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.075+0000 7f5c7a501640 1 --2- 172.21.15.57:0/1545656769 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5c7406d430 0x7f5c74150ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:44.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.075+0000 7f5c7ad02640 1 -- 172.21.15.57:0/1545656769 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5c740aee00 msgr2=0x7f5c7415bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:44.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.075+0000 7f5c7ad02640 1 --2- 172.21.15.57:0/1545656769 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5c740aee00 0x7f5c7415bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.075+0000 7f5c7ad02640 1 -- 172.21.15.57:0/1545656769 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5c7406d430 msgr2=0x7f5c74150ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:44.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.075+0000 7f5c7ad02640 1 --2- 172.21.15.57:0/1545656769 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5c7406d430 0x7f5c74150ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.076+0000 7f5c7ad02640 1 -- 172.21.15.57:0/1545656769 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c64009560 con 0x7f5c7415ae10 2023-10-19T12:44:44.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.076+0000 7f5c7a501640 1 --2- 172.21.15.57:0/1545656769 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5c7406d430 0x7f5c74150ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:44:44.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.076+0000 7f4f8c58e640 1 Processor -- start 2023-10-19T12:44:44.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.076+0000 7fddb2ffd640 1 -- 172.21.15.57:0/3683187965 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddb8034950 con 0x7fddac003970 2023-10-19T12:44:44.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.076+0000 7fdca9763640 1 -- 172.21.15.57:0/3424334509 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdca40cbc40 con 0x7fdca4157220 2023-10-19T12:44:44.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.076+0000 7f6aa1b4c640 1 -- 172.21.15.57:0/2246721187 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6a9c0690e0 con 0x7f6a9c13d450 2023-10-19T12:44:44.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.076+0000 7fddb2ffd640 1 -- 172.21.15.57:0/3683187965 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fddb8034ab0 con 0x7fddac003970 2023-10-19T12:44:44.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.076+0000 7fdca9763640 1 -- 172.21.15.57:0/3424334509 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdca40cc200 con 0x7fdca4157220 2023-10-19T12:44:44.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.076+0000 7fac767fc640 1 -- 172.21.15.57:0/4105396125 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac70010690 con 0x7fac8006ad80 2023-10-19T12:44:44.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.076+0000 7f5c7ad02640 1 --2- 172.21.15.57:0/1545656769 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5c7415ae10 0x7f5c7415c1d0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f5c6c00ea80 tx=0x7f5c6c00ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:44.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.076+0000 7fac86f6c640 1 -- 172.21.15.57:0/4105396125 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fac8004da70 con 0x7fac8006ad80 2023-10-19T12:44:44.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.076+0000 7fc546ffd640 1 -- 172.21.15.57:0/305170354 <== mon.2 v2:172.21.15.179:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc550004450 con 0x7fc558156e30 2023-10-19T12:44:44.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.076+0000 7fc546ffd640 1 -- 172.21.15.57:0/305170354 <== mon.2 v2:172.21.15.179:3301/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fc5500046a0 con 0x7fc558156e30 2023-10-19T12:44:44.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.076+0000 7fc546ffd640 1 -- 172.21.15.57:0/305170354 <== mon.2 v2:172.21.15.179:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc55003f630 con 0x7fc558156e30 2023-10-19T12:44:44.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.076+0000 7f4f8c58e640 1 -- start start 2023-10-19T12:44:44.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.076+0000 7f435ac19640 1 --2- 172.21.15.57:0/86828474 >> [v2:172.21.15.57:6808/3729172421,v1:172.21.15.57:6809/3729172421] conn(0x7f4338074500 0x7f4338076940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:44.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.076+0000 7f4359c17640 1 --2- 172.21.15.57:0/86828474 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f433806aec0 0x7f433806d380 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f434c0ffb80 tx=0x7f434400c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:44.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.076+0000 7f435ac19640 1 --2- 172.21.15.57:0/86828474 >> [v2:172.21.15.57:6808/3729172421,v1:172.21.15.57:6809/3729172421] conn(0x7f4338074500 0x7f4338076940 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:44.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.077+0000 7fddc359e640 1 -- 172.21.15.57:0/3683187965 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddbc0e67f0 con 0x7fddac003970 2023-10-19T12:44:44.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.077+0000 7fddc359e640 1 -- 172.21.15.57:0/3683187965 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddbc0dfa80 con 0x7fddac003970 2023-10-19T12:44:44.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.077+0000 7fdc8a7fc640 1 -- 172.21.15.57:0/3424334509 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fdca404da70 con 0x7fdca4157220 2023-10-19T12:44:44.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.077+0000 7fddb2ffd640 1 -- 172.21.15.57:0/3683187965 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddb803f440 con 0x7fddac003970 2023-10-19T12:44:44.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.077+0000 7f434affd640 1 -- 172.21.15.57:0/86828474 <== osd.1 v2:172.21.15.57:6808/3729172421 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f4338077010 con 0x7f4338074500 2023-10-19T12:44:44.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.077+0000 7fc55cc5d640 1 -- 172.21.15.57:0/305170354 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5581f59e0 con 0x7fc558156e30 2023-10-19T12:44:44.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.078+0000 7f4f8c58e640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4f8415ae60 0x7f4f840972d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:44.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.078+0000 7fc55cc5d640 1 -- 172.21.15.57:0/305170354 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5581f56c0 con 0x7fc558156e30 2023-10-19T12:44:44.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.078+0000 7f4f8c58e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4f8415efe0 0x7f4f84098110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:44.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.078+0000 7f4f8c58e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4f8415f440 0x7f4f840ab4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:44.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.078+0000 7f5c6affd640 1 -- 172.21.15.57:0/1545656769 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c6c00ce40 con 0x7f5c7415ae10 2023-10-19T12:44:44.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.078+0000 7fdca0ff9640 1 -- 172.21.15.57:0/3424334509 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fdc9801c460 con 0x7fdca4157220 2023-10-19T12:44:44.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.078+0000 7f5c6affd640 1 -- 172.21.15.57:0/1545656769 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f5c6c004b20 con 0x7f5c7415ae10 2023-10-19T12:44:44.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.078+0000 7f5c6affd640 1 -- 172.21.15.57:0/1545656769 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c6c005020 con 0x7f5c7415ae10 2023-10-19T12:44:44.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.078+0000 7f6a83fff640 1 -- 172.21.15.57:0/2246721187 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f6a8c0053c0 con 0x7f6a9c13d450 2023-10-19T12:44:44.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.079+0000 7fdca0ff9640 1 --2- 172.21.15.57:0/3424334509 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fdc9006af10 0x7fdc9006d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:44.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.078+0000 7fac767fc640 1 -- 172.21.15.57:0/4105396125 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fac700107f0 con 0x7fac8006ad80 2023-10-19T12:44:44.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.078+0000 7fac767fc640 1 --2- 172.21.15.57:0/4105396125 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fac6406ae40 0x7fac6406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:44.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.079+0000 7fac85f6a640 1 --2- 172.21.15.57:0/4105396125 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fac6406ae40 0x7fac6406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:44.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.079+0000 7fac767fc640 1 -- 172.21.15.57:0/4105396125 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fac70014070 con 0x7fac8006ad80 2023-10-19T12:44:44.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.079+0000 7fac767fc640 1 --2- 172.21.15.57:0/4105396125 >> [v2:172.21.15.179:6805/4179727948,v1:172.21.15.179:6807/4179727948] conn(0x7fac64074290 0x7fac640766d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:44.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.079+0000 7fac767fc640 1 -- 172.21.15.57:0/4105396125 --> [v2:172.21.15.179:6805/4179727948,v1:172.21.15.179:6807/4179727948] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fac64076da0 con 0x7fac64074290 2023-10-19T12:44:44.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.078+0000 7f4f8ab2b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4f8415f440 0x7f4f840ab4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:44.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.079+0000 7f4f8ab2b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4f8415f440 0x7f4f840ab4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50734/0 (socket says 172.21.15.57:50734) 2023-10-19T12:44:44.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.079+0000 7f4f8ab2b640 1 -- 172.21.15.57:0/2792373613 learned_addr learned my addr 172.21.15.57:0/2792373613 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:44.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.079+0000 7f6a83fff640 1 --2- 172.21.15.57:0/2246721187 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f6a8806af10 0x7f6a8806d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:44.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.079+0000 7f6a83fff640 1 -- 172.21.15.57:0/2246721187 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f6a8c0ac470 con 0x7f6a9c13d450 2023-10-19T12:44:44.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.079+0000 7f6a83fff640 1 --2- 172.21.15.57:0/2246721187 >> [v2:172.21.15.57:6800/2805338151,v1:172.21.15.57:6801/2805338151] conn(0x7f6a88074550 0x7f6a88076990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:44.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.079+0000 7f6a83fff640 1 -- 172.21.15.57:0/2246721187 --> [v2:172.21.15.57:6800/2805338151,v1:172.21.15.57:6801/2805338151] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6a88077060 con 0x7f6a88074550 2023-10-19T12:44:44.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.079+0000 7f6a83fff640 1 -- 172.21.15.57:0/2246721187 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6a8c0ac820 con 0x7f6a9c13d450 2023-10-19T12:44:44.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.080+0000 7f6a9b7fe640 1 --2- 172.21.15.57:0/2246721187 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f6a8806af10 0x7f6a8806d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:44.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.079+0000 7fdca3fff640 1 --2- 172.21.15.57:0/3424334509 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fdc9006af10 0x7fdc9006d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:44.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.080+0000 7fdca0ff9640 1 -- 172.21.15.57:0/3424334509 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fdc9808a330 con 0x7fdca4157220 2023-10-19T12:44:44.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.080+0000 7fdca0ff9640 1 --2- 172.21.15.57:0/3424334509 >> [v2:172.21.15.57:6816/3386449871,v1:172.21.15.57:6817/3386449871] conn(0x7fdc900744d0 0x7fdc90076910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:44.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.080+0000 7fdca0ff9640 1 -- 172.21.15.57:0/3424334509 --> [v2:172.21.15.57:6816/3386449871,v1:172.21.15.57:6817/3386449871] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdc90076fe0 con 0x7fdc900744d0 2023-10-19T12:44:44.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.079+0000 7fac85f6a640 1 --2- 172.21.15.57:0/4105396125 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fac6406ae40 0x7fac6406d300 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fac78035000 tx=0x7fac78004a40 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:44.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.080+0000 7fac8676b640 1 --2- 172.21.15.57:0/4105396125 >> [v2:172.21.15.179:6805/4179727948,v1:172.21.15.179:6807/4179727948] conn(0x7fac64074290 0x7fac640766d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:44.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.080+0000 7fac767fc640 1 -- 172.21.15.57:0/4105396125 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fac70055610 con 0x7fac8006ad80 2023-10-19T12:44:44.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.078+0000 7f5c7c765640 1 -- 172.21.15.57:0/1545656769 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c7414f6e0 con 0x7f5c7415ae10 2023-10-19T12:44:44.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.078+0000 7f5c7c765640 1 -- 172.21.15.57:0/1545656769 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c7414fee0 con 0x7f5c7415ae10 2023-10-19T12:44:44.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.079+0000 7f5c7c765640 1 -- 172.21.15.57:0/1545656769 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5c7415ef10 con 0x7f5c7415ae10 2023-10-19T12:44:44.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.080+0000 7fc55cc5d640 1 -- 172.21.15.57:0/305170354 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc55804da70 con 0x7fc558156e30 2023-10-19T12:44:44.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.079+0000 7f4f8c58e640 1 -- 172.21.15.57:0/2792373613 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f84097810 con 0x7f4f8415f440 2023-10-19T12:44:44.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.079+0000 7f4f8c58e640 1 -- 172.21.15.57:0/2792373613 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f8406c5e0 con 0x7f4f8415efe0 2023-10-19T12:44:44.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.079+0000 7f4f8c58e640 1 -- 172.21.15.57:0/2792373613 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f8406c8c0 con 0x7f4f8415ae60 2023-10-19T12:44:44.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.080+0000 7f4f8a32a640 1 --2- 172.21.15.57:0/2792373613 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4f8415ae60 0x7f4f840972d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:44.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.080+0000 7f4f89b29640 1 --2- 172.21.15.57:0/2792373613 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4f8415efe0 0x7f4f84098110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:44.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.080+0000 7f4f8ab2b640 1 -- 172.21.15.57:0/2792373613 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4f8415ae60 msgr2=0x7f4f840972d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:44.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.080+0000 7f4f8ab2b640 1 --2- 172.21.15.57:0/2792373613 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4f8415ae60 0x7f4f840972d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.080+0000 7f4f8ab2b640 1 -- 172.21.15.57:0/2792373613 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4f8415efe0 msgr2=0x7f4f84098110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:44.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.080+0000 7f4f8ab2b640 1 --2- 172.21.15.57:0/2792373613 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4f8415efe0 0x7f4f84098110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.080+0000 7f4f8ab2b640 1 -- 172.21.15.57:0/2792373613 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f74009490 con 0x7f4f8415f440 2023-10-19T12:44:44.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.080+0000 7f4f8a32a640 1 --2- 172.21.15.57:0/2792373613 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4f8415ae60 0x7f4f840972d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T12:44:44.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.080+0000 7fdca8f62640 1 --2- 172.21.15.57:0/3424334509 >> [v2:172.21.15.57:6816/3386449871,v1:172.21.15.57:6817/3386449871] conn(0x7fdc900744d0 0x7fdc90076910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:44.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.079+0000 7fddc359e640 1 -- 172.21.15.57:0/3683187965 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fddbc0057b0 con 0x7fddac003970 2023-10-19T12:44:44.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.080+0000 7fddb2ffd640 1 -- 172.21.15.57:0/3683187965 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fddb803b040 con 0x7fddac003970 2023-10-19T12:44:44.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.081+0000 7fc546ffd640 1 -- 172.21.15.57:0/305170354 <== mon.2 v2:172.21.15.179:3301/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fc55003f790 con 0x7fc558156e30 2023-10-19T12:44:44.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.081+0000 7fdca0ff9640 1 -- 172.21.15.57:0/3424334509 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdc98001120 con 0x7fdca4157220 2023-10-19T12:44:44.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.081+0000 7f4f8ab2b640 1 --2- 172.21.15.57:0/2792373613 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4f8415f440 0x7f4f840ab4d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f4f74009780 tx=0x7f4f74005cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:44.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.081+0000 7f6a9bfff640 1 --2- 172.21.15.57:0/2246721187 >> [v2:172.21.15.57:6800/2805338151,v1:172.21.15.57:6801/2805338151] conn(0x7f6a88074550 0x7f6a88076990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:44.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.081+0000 7fdca3fff640 1 --2- 172.21.15.57:0/3424334509 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fdc9006af10 0x7fdc9006d3d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fdca40abce0 tx=0x7fdc9c004ab0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:44.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.081+0000 7f6a9b7fe640 1 --2- 172.21.15.57:0/2246721187 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f6a8806af10 0x7f6a8806d3d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6a90009bb0 tx=0x7f6a90005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:44.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.081+0000 7fc546ffd640 1 --2- 172.21.15.57:0/305170354 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fc53806aec0 0x7fc53806d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:44.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.081+0000 7f4f7affd640 1 -- 172.21.15.57:0/2792373613 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f7402c8f0 con 0x7f4f8415f440 2023-10-19T12:44:44.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.081+0000 7fac8676b640 1 --2- 172.21.15.57:0/4105396125 >> [v2:172.21.15.179:6805/4179727948,v1:172.21.15.179:6807/4179727948] conn(0x7fac64074290 0x7fac640766d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:44.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.081+0000 7fc546ffd640 1 -- 172.21.15.57:0/305170354 <== mon.2 v2:172.21.15.179:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fc5500ac3e0 con 0x7fc558156e30 2023-10-19T12:44:44.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.081+0000 7fc55659c640 1 --2- 172.21.15.57:0/305170354 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fc53806aec0 0x7fc53806d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:44.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.081+0000 7f4f7affd640 1 -- 172.21.15.57:0/2792373613 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f4f74005020 con 0x7f4f8415f440 2023-10-19T12:44:44.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.081+0000 7f4f7affd640 1 -- 172.21.15.57:0/2792373613 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f74005410 con 0x7f4f8415f440 2023-10-19T12:44:44.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.081+0000 7fc546ffd640 1 --2- 172.21.15.57:0/305170354 >> [v2:172.21.15.179:6824/2898099196,v1:172.21.15.179:6825/2898099196] conn(0x7fc538074480 0x7fc5380768c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:44.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.082+0000 7f4f8c58e640 1 -- 172.21.15.57:0/2792373613 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f8406cba0 con 0x7f4f8415f440 2023-10-19T12:44:44.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.082+0000 7fdca8f62640 1 --2- 172.21.15.57:0/3424334509 >> [v2:172.21.15.57:6816/3386449871,v1:172.21.15.57:6817/3386449871] conn(0x7fdc900744d0 0x7fdc90076910 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:44.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.082+0000 7f4f8c58e640 1 -- 172.21.15.57:0/2792373613 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f84099b50 con 0x7f4f8415f440 2023-10-19T12:44:44.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.082+0000 7fc55659c640 1 --2- 172.21.15.57:0/305170354 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fc53806aec0 0x7fc53806d380 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fc5581567b0 tx=0x7fc548009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:44.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.082+0000 7fc556d9d640 1 --2- 172.21.15.57:0/305170354 >> [v2:172.21.15.179:6824/2898099196,v1:172.21.15.179:6825/2898099196] conn(0x7fc538074480 0x7fc5380768c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:44.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.082+0000 7f6a9bfff640 1 --2- 172.21.15.57:0/2246721187 >> [v2:172.21.15.57:6800/2805338151,v1:172.21.15.57:6801/2805338151] conn(0x7f6a88074550 0x7f6a88076990 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:44.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.082+0000 7fac767fc640 1 -- 172.21.15.57:0/4105396125 <== osd.4 v2:172.21.15.179:6805/4179727948 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7fac64076da0 con 0x7fac64074290 2023-10-19T12:44:44.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.081+0000 7fddb2ffd640 1 --2- 172.21.15.57:0/3683187965 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fdda406af40 0x7fdda406d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:44.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.081+0000 7fddb2ffd640 1 -- 172.21.15.57:0/3683187965 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fddb80ab530 con 0x7fddac003970 2023-10-19T12:44:44.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.081+0000 7fddb2ffd640 1 --2- 172.21.15.57:0/3683187965 >> [v2:172.21.15.57:6824/1015489591,v1:172.21.15.57:6825/1015489591] conn(0x7fdda4074580 0x7fdda40769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:44.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.081+0000 7fddb2ffd640 1 -- 172.21.15.57:0/3683187965 --> [v2:172.21.15.57:6824/1015489591,v1:172.21.15.57:6825/1015489591] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdda4077090 con 0x7fdda4074580 2023-10-19T12:44:44.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.081+0000 7fddb2ffd640 1 -- 172.21.15.57:0/3683187965 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fddb80adb50 con 0x7fddac003970 2023-10-19T12:44:44.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.081+0000 7fddc2d9d640 1 --2- 172.21.15.57:0/3683187965 >> [v2:172.21.15.57:6824/1015489591,v1:172.21.15.57:6825/1015489591] conn(0x7fdda4074580 0x7fdda40769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:44.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.081+0000 7fddc1d9b640 1 --2- 172.21.15.57:0/3683187965 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fdda406af40 0x7fdda406d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:44.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.082+0000 7fddc2d9d640 1 --2- 172.21.15.57:0/3683187965 >> [v2:172.21.15.57:6824/1015489591,v1:172.21.15.57:6825/1015489591] conn(0x7fdda4074580 0x7fdda40769c0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:44.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.082+0000 7fddc1d9b640 1 --2- 172.21.15.57:0/3683187965 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fdda406af40 0x7fdda406d400 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fddb4001800 tx=0x7fddb4006040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:44.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.083+0000 7fddb2ffd640 1 -- 172.21.15.57:0/3683187965 <== osd.0 v2:172.21.15.57:6824/1015489591 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7fdda4077090 con 0x7fdda4074580 2023-10-19T12:44:44.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.083+0000 7fc556d9d640 1 --2- 172.21.15.57:0/305170354 >> [v2:172.21.15.179:6824/2898099196,v1:172.21.15.179:6825/2898099196] conn(0x7fc538074480 0x7fc5380768c0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:44.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.083+0000 7f5c6affd640 1 -- 172.21.15.57:0/1545656769 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f5c6c005180 con 0x7f5c7415ae10 2023-10-19T12:44:44.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.084+0000 7f5c6affd640 1 --2- 172.21.15.57:0/1545656769 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f5c5806af40 0x7f5c5806d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:44.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.083+0000 7f4f78ff9640 1 -- 172.21.15.57:0/2792373613 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4f60000fc0 con 0x7f4f8415f440 2023-10-19T12:44:44.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.084+0000 7fdca0ff9640 1 -- 172.21.15.57:0/3424334509 <== osd.3 v2:172.21.15.57:6816/3386449871 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7fdc90076fe0 con 0x7fdc900744d0 2023-10-19T12:44:44.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.084+0000 7f5c6affd640 1 -- 172.21.15.57:0/1545656769 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f5c6c014070 con 0x7f5c7415ae10 2023-10-19T12:44:44.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.084+0000 7fc546ffd640 1 -- 172.21.15.57:0/305170354 --> [v2:172.21.15.179:6824/2898099196,v1:172.21.15.179:6825/2898099196] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc538076f90 con 0x7fc538074480 2023-10-19T12:44:44.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.084+0000 7fc546ffd640 1 -- 172.21.15.57:0/305170354 <== mon.2 v2:172.21.15.179:3301/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc550077190 con 0x7fc558156e30 2023-10-19T12:44:44.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.083+0000 7f6a83fff640 1 -- 172.21.15.57:0/2246721187 <== osd.2 v2:172.21.15.57:6800/2805338151 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f6a88077060 con 0x7f6a88074550 2023-10-19T12:44:44.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.084+0000 7f5c6affd640 1 --2- 172.21.15.57:0/1545656769 >> [v2:172.21.15.179:6800/1129921230,v1:172.21.15.179:6801/1129921230] conn(0x7f5c58074580 0x7f5c580769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:44.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.084+0000 7f5c6affd640 1 -- 172.21.15.57:0/1545656769 --> [v2:172.21.15.179:6800/1129921230,v1:172.21.15.179:6801/1129921230] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5c58077090 con 0x7f5c58074580 2023-10-19T12:44:44.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.084+0000 7f5c6affd640 1 -- 172.21.15.57:0/1545656769 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5c6c08acb0 con 0x7f5c7415ae10 2023-10-19T12:44:44.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.084+0000 7f5c7a501640 1 --2- 172.21.15.57:0/1545656769 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f5c5806af40 0x7f5c5806d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:44.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.084+0000 7f5c79d00640 1 --2- 172.21.15.57:0/1545656769 >> [v2:172.21.15.179:6800/1129921230,v1:172.21.15.179:6801/1129921230] conn(0x7f5c58074580 0x7f5c580769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:44.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.085+0000 7f5c7a501640 1 --2- 172.21.15.57:0/1545656769 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f5c5806af40 0x7f5c5806d400 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f5c74065c80 tx=0x7f5c70005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:44.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.085+0000 7f4f7affd640 1 -- 172.21.15.57:0/2792373613 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f4f740047a0 con 0x7f4f8415f440 2023-10-19T12:44:44.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.085+0000 7f5c79d00640 1 --2- 172.21.15.57:0/1545656769 >> [v2:172.21.15.179:6800/1129921230,v1:172.21.15.179:6801/1129921230] conn(0x7f5c58074580 0x7f5c580769c0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:44.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.085+0000 7fc546ffd640 1 -- 172.21.15.57:0/305170354 <== osd.6 v2:172.21.15.179:6824/2898099196 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7fc538076f90 con 0x7fc538074480 2023-10-19T12:44:44.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.085+0000 7f4f7affd640 1 --2- 172.21.15.57:0/2792373613 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f4f6806ae40 0x7f4f6806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:44.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.085+0000 7f4f7affd640 1 -- 172.21.15.57:0/2792373613 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f4f740ac740 con 0x7f4f8415f440 2023-10-19T12:44:44.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.085+0000 7f4f8a32a640 1 --2- 172.21.15.57:0/2792373613 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f4f6806ae40 0x7f4f6806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:44.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.085+0000 7f4f7affd640 1 --2- 172.21.15.57:0/2792373613 >> [v2:172.21.15.179:6816/3758126853,v1:172.21.15.179:6817/3758126853] conn(0x7f4f68074480 0x7f4f680768c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:44.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.085+0000 7f4f7affd640 1 -- 172.21.15.57:0/2792373613 --> [v2:172.21.15.179:6816/3758126853,v1:172.21.15.179:6817/3758126853] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4f68076f90 con 0x7f4f68074480 2023-10-19T12:44:44.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.085+0000 7f4f7affd640 1 -- 172.21.15.57:0/2792373613 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4f740acaf0 con 0x7f4f8415f440 2023-10-19T12:44:44.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.086+0000 7f5c6affd640 1 -- 172.21.15.57:0/1545656769 <== osd.5 v2:172.21.15.179:6800/1129921230 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f5c58077090 con 0x7f5c58074580 2023-10-19T12:44:44.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.086+0000 7f4f89b29640 1 --2- 172.21.15.57:0/2792373613 >> [v2:172.21.15.179:6816/3758126853,v1:172.21.15.179:6817/3758126853] conn(0x7f4f68074480 0x7f4f680768c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:44.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.086+0000 7f4f8a32a640 1 --2- 172.21.15.57:0/2792373613 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f4f6806ae40 0x7f4f6806d300 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f4f7c009fd0 tx=0x7f4f7c009680 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:44.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.087+0000 7f4f89b29640 1 --2- 172.21.15.57:0/2792373613 >> [v2:172.21.15.179:6816/3758126853,v1:172.21.15.179:6817/3758126853] conn(0x7f4f68074480 0x7f4f680768c0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:44.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.087+0000 7f4f7affd640 1 -- 172.21.15.57:0/2792373613 <== osd.7 v2:172.21.15.179:6816/3758126853 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f4f68076f90 con 0x7f4f68074480 2023-10-19T12:44:44.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.107+0000 7f435b41a640 1 -- 172.21.15.57:0/86828474 --> [v2:172.21.15.57:6808/3729172421,v1:172.21.15.57:6809/3729172421] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f433c002cb0 con 0x7f4338074500 2023-10-19T12:44:44.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.107+0000 7f434affd640 1 -- 172.21.15.57:0/86828474 <== osd.1 v2:172.21.15.57:6808/3729172421 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f433c002cb0 con 0x7f4338074500 2023-10-19T12:44:44.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.108+0000 7f4348ff9640 1 -- 172.21.15.57:0/86828474 >> [v2:172.21.15.57:6808/3729172421,v1:172.21.15.57:6809/3729172421] conn(0x7f4338074500 msgr2=0x7f4338076940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:44.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.108+0000 7f4348ff9640 1 --2- 172.21.15.57:0/86828474 >> [v2:172.21.15.57:6808/3729172421,v1:172.21.15.57:6809/3729172421] conn(0x7f4338074500 0x7f4338076940 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.108+0000 7f4348ff9640 1 -- 172.21.15.57:0/86828474 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f433806aec0 msgr2=0x7f433806d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:44.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.108+0000 7f4348ff9640 1 --2- 172.21.15.57:0/86828474 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f433806aec0 0x7f433806d380 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f434c0ffb80 tx=0x7f434400c040 comp rx=0 tx=0).stop 2023-10-19T12:44:44.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.108+0000 7f4348ff9640 1 -- 172.21.15.57:0/86828474 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f434c0338f0 msgr2=0x7f434c04cc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:44.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.109+0000 7f4348ff9640 1 --2- 172.21.15.57:0/86828474 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f434c0338f0 0x7f434c04cc80 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f435000c3c0 tx=0x7f435000c880 comp rx=0 tx=0).stop 2023-10-19T12:44:44.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.109+0000 7f435ac19640 1 -- 172.21.15.57:0/86828474 reap_dead start 2023-10-19T12:44:44.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.109+0000 7f4348ff9640 1 -- 172.21.15.57:0/86828474 shutdown_connections 2023-10-19T12:44:44.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.109+0000 7f4348ff9640 1 -- 172.21.15.57:0/86828474 >> 172.21.15.57:0/86828474 conn(0x7f434c009f80 msgr2=0x7f434c0f96f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:44.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.109+0000 7f4348ff9640 1 -- 172.21.15.57:0/86828474 shutdown_connections 2023-10-19T12:44:44.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.109+0000 7f4348ff9640 1 -- 172.21.15.57:0/86828474 wait complete. 2023-10-19T12:44:44.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.116+0000 7fac86f6c640 1 -- 172.21.15.57:0/4105396125 --> [v2:172.21.15.179:6805/4179727948,v1:172.21.15.179:6807/4179727948] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fac800c16d0 con 0x7fac64074290 2023-10-19T12:44:44.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.116+0000 7fddc359e640 1 -- 172.21.15.57:0/3683187965 --> [v2:172.21.15.57:6824/1015489591,v1:172.21.15.57:6825/1015489591] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fddbc0dfc40 con 0x7fdda4074580 2023-10-19T12:44:44.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.117+0000 7fdc8a7fc640 1 -- 172.21.15.57:0/3424334509 --> [v2:172.21.15.57:6816/3386449871,v1:172.21.15.57:6817/3386449871] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fdca404da70 con 0x7fdc900744d0 2023-10-19T12:44:44.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.117+0000 7f6aa1b4c640 1 -- 172.21.15.57:0/2246721187 --> [v2:172.21.15.57:6800/2805338151,v1:172.21.15.57:6801/2805338151] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f6a9c13a800 con 0x7f6a88074550 2023-10-19T12:44:44.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.117+0000 7fac767fc640 1 -- 172.21.15.57:0/4105396125 <== osd.4 v2:172.21.15.179:6805/4179727948 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fac800c16d0 con 0x7fac64074290 2023-10-19T12:44:44.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.117+0000 7fddb2ffd640 1 -- 172.21.15.57:0/3683187965 <== osd.0 v2:172.21.15.57:6824/1015489591 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fddbc0dfc40 con 0x7fdda4074580 2023-10-19T12:44:44.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.117+0000 7f6a83fff640 1 -- 172.21.15.57:0/2246721187 <== osd.2 v2:172.21.15.57:6800/2805338151 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f6a9c13a800 con 0x7f6a88074550 2023-10-19T12:44:44.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.118+0000 7fdca0ff9640 1 -- 172.21.15.57:0/3424334509 <== osd.3 v2:172.21.15.57:6816/3386449871 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fdca404da70 con 0x7fdc900744d0 2023-10-19T12:44:44.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.118+0000 7f4f78ff9640 1 -- 172.21.15.57:0/2792373613 --> [v2:172.21.15.179:6816/3758126853,v1:172.21.15.179:6817/3758126853] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f4f60002cb0 con 0x7f4f68074480 2023-10-19T12:44:44.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.118+0000 7fc55cc5d640 1 -- 172.21.15.57:0/305170354 --> [v2:172.21.15.179:6824/2898099196,v1:172.21.15.179:6825/2898099196] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc5581f61e0 con 0x7fc538074480 2023-10-19T12:44:44.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.118+0000 7fac63fff640 1 -- 172.21.15.57:0/4105396125 >> [v2:172.21.15.179:6805/4179727948,v1:172.21.15.179:6807/4179727948] conn(0x7fac64074290 msgr2=0x7fac640766d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:44.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.118+0000 7fac63fff640 1 --2- 172.21.15.57:0/4105396125 >> [v2:172.21.15.179:6805/4179727948,v1:172.21.15.179:6807/4179727948] conn(0x7fac64074290 0x7fac640766d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.118+0000 7fddc359e640 1 -- 172.21.15.57:0/3683187965 >> [v2:172.21.15.57:6824/1015489591,v1:172.21.15.57:6825/1015489591] conn(0x7fdda4074580 msgr2=0x7fdda40769c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:44.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.118+0000 7fddc359e640 1 --2- 172.21.15.57:0/3683187965 >> [v2:172.21.15.57:6824/1015489591,v1:172.21.15.57:6825/1015489591] conn(0x7fdda4074580 0x7fdda40769c0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.118+0000 7fdc8a7fc640 1 -- 172.21.15.57:0/3424334509 >> [v2:172.21.15.57:6816/3386449871,v1:172.21.15.57:6817/3386449871] conn(0x7fdc900744d0 msgr2=0x7fdc90076910 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:44.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.118+0000 7fac63fff640 1 -- 172.21.15.57:0/4105396125 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fac6406ae40 msgr2=0x7fac6406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:44.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.118+0000 7fac63fff640 1 --2- 172.21.15.57:0/4105396125 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fac6406ae40 0x7fac6406d300 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fac78035000 tx=0x7fac78004a40 comp rx=0 tx=0).stop 2023-10-19T12:44:44.201 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.118+0000 7fac63fff640 1 -- 172.21.15.57:0/4105396125 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fac8006ad80 msgr2=0x7fac80098460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:44.201 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.118+0000 7fdc8a7fc640 1 --2- 172.21.15.57:0/3424334509 >> [v2:172.21.15.57:6816/3386449871,v1:172.21.15.57:6817/3386449871] conn(0x7fdc900744d0 0x7fdc90076910 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.201 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.118+0000 7fac63fff640 1 --2- 172.21.15.57:0/4105396125 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fac8006ad80 0x7fac80098460 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fac7000e6e0 tx=0x7fac7000eba0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.202 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.118+0000 7f6aa1b4c640 1 -- 172.21.15.57:0/2246721187 >> [v2:172.21.15.57:6800/2805338151,v1:172.21.15.57:6801/2805338151] conn(0x7f6a88074550 msgr2=0x7f6a88076990 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:44.202 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.118+0000 7f6aa1b4c640 1 --2- 172.21.15.57:0/2246721187 >> [v2:172.21.15.57:6800/2805338151,v1:172.21.15.57:6801/2805338151] conn(0x7f6a88074550 0x7f6a88076990 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.118+0000 7fac8676b640 1 -- 172.21.15.57:0/4105396125 reap_dead start 2023-10-19T12:44:44.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.118+0000 7f6aa1b4c640 1 -- 172.21.15.57:0/2246721187 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f6a8806af10 msgr2=0x7f6a8806d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:44.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.118+0000 7f6aa1b4c640 1 --2- 172.21.15.57:0/2246721187 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f6a8806af10 0x7f6a8806d3d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6a90009bb0 tx=0x7f6a90005f90 comp rx=0 tx=0).stop 2023-10-19T12:44:44.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.118+0000 7fdc8a7fc640 1 -- 172.21.15.57:0/3424334509 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fdc9006af10 msgr2=0x7fdc9006d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:44.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.118+0000 7fdc8a7fc640 1 --2- 172.21.15.57:0/3424334509 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fdc9006af10 0x7fdc9006d3d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fdca40abce0 tx=0x7fdc9c004ab0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.118+0000 7f4f7affd640 1 -- 172.21.15.57:0/2792373613 <== osd.7 v2:172.21.15.179:6816/3758126853 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f4f60002cb0 con 0x7f4f68074480 2023-10-19T12:44:44.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.118+0000 7f6aa1b4c640 1 -- 172.21.15.57:0/2246721187 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6a9c13d450 msgr2=0x7f6a9c156290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:44.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.118+0000 7fdc8a7fc640 1 -- 172.21.15.57:0/3424334509 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdca4157220 msgr2=0x7fdca40a6ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:44.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.118+0000 7f6aa1b4c640 1 --2- 172.21.15.57:0/2246721187 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6a9c13d450 0x7f6a9c156290 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f6a8c02c280 tx=0x7f6a8c003820 comp rx=0 tx=0).stop 2023-10-19T12:44:44.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.118+0000 7fdc8a7fc640 1 --2- 172.21.15.57:0/3424334509 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdca4157220 0x7fdca40a6ec0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fdc9800c3a0 tx=0x7fdc9800c860 comp rx=0 tx=0).stop 2023-10-19T12:44:44.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.118+0000 7fddc359e640 1 -- 172.21.15.57:0/3683187965 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fdda406af40 msgr2=0x7fdda406d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:44.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.118+0000 7fac63fff640 1 -- 172.21.15.57:0/4105396125 shutdown_connections 2023-10-19T12:44:44.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.118+0000 7fac63fff640 1 -- 172.21.15.57:0/4105396125 >> 172.21.15.57:0/4105396125 conn(0x7fac800c9a50 msgr2=0x7fac800c7600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:44.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.119+0000 7fddc359e640 1 --2- 172.21.15.57:0/3683187965 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fdda406af40 0x7fdda406d400 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fddb4001800 tx=0x7fddb4006040 comp rx=0 tx=0).stop 2023-10-19T12:44:44.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.119+0000 7f6a9bfff640 1 -- 172.21.15.57:0/2246721187 reap_dead start 2023-10-19T12:44:44.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.119+0000 7f5c7c765640 1 -- 172.21.15.57:0/1545656769 --> [v2:172.21.15.179:6800/1129921230,v1:172.21.15.179:6801/1129921230] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f5c7414f570 con 0x7f5c58074580 2023-10-19T12:44:44.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.119+0000 7fddc359e640 1 -- 172.21.15.57:0/3683187965 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fddac003970 msgr2=0x7fddbc0e4380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:44.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.119+0000 7f6aa1b4c640 1 -- 172.21.15.57:0/2246721187 shutdown_connections 2023-10-19T12:44:44.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.119+0000 7f6aa1b4c640 1 -- 172.21.15.57:0/2246721187 >> 172.21.15.57:0/2246721187 conn(0x7f6a9c069930 msgr2=0x7f6a9c0ad910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:44.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.119+0000 7fdca8f62640 1 -- 172.21.15.57:0/3424334509 reap_dead start 2023-10-19T12:44:44.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.119+0000 7fc546ffd640 1 -- 172.21.15.57:0/305170354 <== osd.6 v2:172.21.15.179:6824/2898099196 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fc5581f61e0 con 0x7fc538074480 2023-10-19T12:44:44.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.119+0000 7fddc359e640 1 --2- 172.21.15.57:0/3683187965 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fddac003970 0x7fddbc0e4380 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fddb80098f0 tx=0x7fddb802cf90 comp rx=0 tx=0).stop 2023-10-19T12:44:44.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.119+0000 7fddc2d9d640 1 -- 172.21.15.57:0/3683187965 reap_dead start 2023-10-19T12:44:44.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.119+0000 7fddc359e640 1 -- 172.21.15.57:0/3683187965 shutdown_connections 2023-10-19T12:44:44.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.119+0000 7fddc359e640 1 -- 172.21.15.57:0/3683187965 >> 172.21.15.57:0/3683187965 conn(0x7fddbc0559b0 msgr2=0x7fddbc03db50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:44.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.119+0000 7fdc8a7fc640 1 -- 172.21.15.57:0/3424334509 shutdown_connections 2023-10-19T12:44:44.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.119+0000 7fdc8a7fc640 1 -- 172.21.15.57:0/3424334509 >> 172.21.15.57:0/3424334509 conn(0x7fdca40ae460 msgr2=0x7fdca406b610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:44.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.119+0000 7fddc359e640 1 -- 172.21.15.57:0/3683187965 shutdown_connections 2023-10-19T12:44:44.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.119+0000 7f6aa1b4c640 1 -- 172.21.15.57:0/2246721187 shutdown_connections 2023-10-19T12:44:44.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.119+0000 7fac63fff640 1 -- 172.21.15.57:0/4105396125 shutdown_connections 2023-10-19T12:44:44.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.119+0000 7fac63fff640 1 -- 172.21.15.57:0/4105396125 wait complete. 2023-10-19T12:44:44.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.119+0000 7f6aa1b4c640 1 -- 172.21.15.57:0/2246721187 wait complete. 2023-10-19T12:44:44.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.119+0000 7f4f78ff9640 1 -- 172.21.15.57:0/2792373613 >> [v2:172.21.15.179:6816/3758126853,v1:172.21.15.179:6817/3758126853] conn(0x7f4f68074480 msgr2=0x7f4f680768c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:44.216 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.119+0000 7fddc359e640 1 -- 172.21.15.57:0/3683187965 wait complete. 2023-10-19T12:44:44.216 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.119+0000 7f4f78ff9640 1 --2- 172.21.15.57:0/2792373613 >> [v2:172.21.15.179:6816/3758126853,v1:172.21.15.179:6817/3758126853] conn(0x7f4f68074480 0x7f4f680768c0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.217 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.119+0000 7fdc8a7fc640 1 -- 172.21.15.57:0/3424334509 shutdown_connections 2023-10-19T12:44:44.217 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.119+0000 7fdc8a7fc640 1 -- 172.21.15.57:0/3424334509 wait complete. 2023-10-19T12:44:44.217 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.119+0000 7f5c6affd640 1 -- 172.21.15.57:0/1545656769 <== osd.5 v2:172.21.15.179:6800/1129921230 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f5c7414f570 con 0x7f5c58074580 2023-10-19T12:44:44.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.119+0000 7fc55cc5d640 1 -- 172.21.15.57:0/305170354 >> [v2:172.21.15.179:6824/2898099196,v1:172.21.15.179:6825/2898099196] conn(0x7fc538074480 msgr2=0x7fc5380768c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:44.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.119+0000 7fc55cc5d640 1 --2- 172.21.15.57:0/305170354 >> [v2:172.21.15.179:6824/2898099196,v1:172.21.15.179:6825/2898099196] conn(0x7fc538074480 0x7fc5380768c0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.119+0000 7fc55cc5d640 1 -- 172.21.15.57:0/305170354 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fc53806aec0 msgr2=0x7fc53806d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:44.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.119+0000 7fc55cc5d640 1 --2- 172.21.15.57:0/305170354 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fc53806aec0 0x7fc53806d380 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fc5581567b0 tx=0x7fc548009040 comp rx=0 tx=0).stop 2023-10-19T12:44:44.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.120+0000 7fc55cc5d640 1 -- 172.21.15.57:0/305170354 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc558156e30 msgr2=0x7fc55814f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:44.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.120+0000 7fc55cc5d640 1 --2- 172.21.15.57:0/305170354 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc558156e30 0x7fc55814f0b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc55002d2f0 tx=0x7fc550008ad0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.120+0000 7f4f78ff9640 1 -- 172.21.15.57:0/2792373613 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f4f6806ae40 msgr2=0x7f4f6806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:44.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.120+0000 7f4f78ff9640 1 --2- 172.21.15.57:0/2792373613 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f4f6806ae40 0x7f4f6806d300 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f4f7c009fd0 tx=0x7f4f7c009680 comp rx=0 tx=0).stop 2023-10-19T12:44:44.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.120+0000 7f4f78ff9640 1 -- 172.21.15.57:0/2792373613 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4f8415f440 msgr2=0x7f4f840ab4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:44.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.120+0000 7f4f78ff9640 1 --2- 172.21.15.57:0/2792373613 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4f8415f440 0x7f4f840ab4d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f4f74009780 tx=0x7f4f74005cf0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.120+0000 7f4f8ab2b640 1 -- 172.21.15.57:0/2792373613 reap_dead start 2023-10-19T12:44:44.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.120+0000 7f5c7c765640 1 -- 172.21.15.57:0/1545656769 >> [v2:172.21.15.179:6800/1129921230,v1:172.21.15.179:6801/1129921230] conn(0x7f5c58074580 msgr2=0x7f5c580769c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:44.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.120+0000 7f5c7c765640 1 --2- 172.21.15.57:0/1545656769 >> [v2:172.21.15.179:6800/1129921230,v1:172.21.15.179:6801/1129921230] conn(0x7f5c58074580 0x7f5c580769c0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.120+0000 7fc556d9d640 1 -- 172.21.15.57:0/305170354 reap_dead start 2023-10-19T12:44:44.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.120+0000 7fc55cc5d640 1 -- 172.21.15.57:0/305170354 shutdown_connections 2023-10-19T12:44:44.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.120+0000 7fc55cc5d640 1 -- 172.21.15.57:0/305170354 >> 172.21.15.57:0/305170354 conn(0x7fc558069900 msgr2=0x7fc558153820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:44.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.120+0000 7f4f78ff9640 1 -- 172.21.15.57:0/2792373613 shutdown_connections 2023-10-19T12:44:44.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.120+0000 7f4f78ff9640 1 -- 172.21.15.57:0/2792373613 >> 172.21.15.57:0/2792373613 conn(0x7f4f840b1fd0 msgr2=0x7f4f840ae1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:44.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.120+0000 7f4f78ff9640 1 -- 172.21.15.57:0/2792373613 shutdown_connections 2023-10-19T12:44:44.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.120+0000 7f4f78ff9640 1 -- 172.21.15.57:0/2792373613 wait complete. 2023-10-19T12:44:44.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.120+0000 7f5c7c765640 1 -- 172.21.15.57:0/1545656769 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f5c5806af40 msgr2=0x7f5c5806d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:44.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.120+0000 7f5c7c765640 1 --2- 172.21.15.57:0/1545656769 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f5c5806af40 0x7f5c5806d400 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f5c74065c80 tx=0x7f5c70005f90 comp rx=0 tx=0).stop 2023-10-19T12:44:44.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.120+0000 7f5c7c765640 1 -- 172.21.15.57:0/1545656769 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5c7415ae10 msgr2=0x7f5c7415c1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:44.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.120+0000 7f5c7c765640 1 --2- 172.21.15.57:0/1545656769 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5c7415ae10 0x7f5c7415c1d0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f5c6c00ea80 tx=0x7f5c6c00ef40 comp rx=0 tx=0).stop 2023-10-19T12:44:44.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.120+0000 7f5c7ad02640 1 -- 172.21.15.57:0/1545656769 reap_dead start 2023-10-19T12:44:44.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.121+0000 7f5c7c765640 1 -- 172.21.15.57:0/1545656769 shutdown_connections 2023-10-19T12:44:44.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.121+0000 7f5c7c765640 1 -- 172.21.15.57:0/1545656769 >> 172.21.15.57:0/1545656769 conn(0x7f5c740b1f60 msgr2=0x7f5c740aaec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:44.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.121+0000 7fc55cc5d640 1 -- 172.21.15.57:0/305170354 shutdown_connections 2023-10-19T12:44:44.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.121+0000 7fc55cc5d640 1 -- 172.21.15.57:0/305170354 wait complete. 2023-10-19T12:44:44.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.121+0000 7f5c7c765640 1 -- 172.21.15.57:0/1545656769 shutdown_connections 2023-10-19T12:44:44.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.121+0000 7f5c7c765640 1 -- 172.21.15.57:0/1545656769 wait complete. 2023-10-19T12:44:44.228 INFO:teuthology.orchestra.run.smithi057.stdout:55834574859 2023-10-19T12:44:44.228 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-19T12:44:44.230 INFO:teuthology.orchestra.run.smithi057.stdout:55834574859 2023-10-19T12:44:44.230 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-10-19T12:44:44.231 INFO:teuthology.orchestra.run.smithi057.stdout:55834574859 2023-10-19T12:44:44.231 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-10-19T12:44:44.232 INFO:teuthology.orchestra.run.smithi057.stdout:55834574860 2023-10-19T12:44:44.233 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-19T12:44:44.234 INFO:teuthology.orchestra.run.smithi057.stdout:55834574858 2023-10-19T12:44:44.234 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-10-19T12:44:44.235 INFO:teuthology.orchestra.run.smithi057.stdout:55834574860 2023-10-19T12:44:44.236 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-19T12:44:44.237 INFO:teuthology.orchestra.run.smithi057.stdout:55834574859 2023-10-19T12:44:44.237 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-10-19T12:44:44.238 INFO:teuthology.orchestra.run.smithi057.stdout:55834574859 2023-10-19T12:44:44.238 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-10-19T12:44:44.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.456+0000 7f05349ed640 1 -- 172.21.15.57:0/719348323 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f05300cd1c0 msgr2=0x7f05300cd650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:44.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.456+0000 7f28ed700640 1 -- 172.21.15.57:0/4230478659 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f28e81557f0 msgr2=0x7f28e80aad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:44.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.456+0000 7f28ed700640 1 --2- 172.21.15.57:0/4230478659 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f28e81557f0 0x7f28e80aad90 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f28e8072070 tx=0x7f28d802bdb0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.456+0000 7f05349ed640 1 --2- 172.21.15.57:0/719348323 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f05300cd1c0 0x7f05300cd650 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f0528005fd0 tx=0x7f052802c080 comp rx=0 tx=0).stop 2023-10-19T12:44:44.461 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.456+0000 7f28ed700640 1 -- 172.21.15.57:0/4230478659 shutdown_connections 2023-10-19T12:44:44.461 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.456+0000 7f28ed700640 1 --2- 172.21.15.57:0/4230478659 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f28e80b2cb0 0x7f28e80b2290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.462 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.456+0000 7f28ed700640 1 --2- 172.21.15.57:0/4230478659 >> v2:172.21.15.57:3300/0 conn(0x7f28e80ab2d0 0x7f28e80b1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.462 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.457+0000 7f05349ed640 1 -- 172.21.15.57:0/719348323 shutdown_connections 2023-10-19T12:44:44.462 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.457+0000 7f05349ed640 1 --2- 172.21.15.57:0/719348323 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f05300cd1c0 0x7f05300cd650 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.462 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.457+0000 7f05349ed640 1 --2- 172.21.15.57:0/719348323 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0530158eb0 0x7f053006d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.457+0000 7f05349ed640 1 -- 172.21.15.57:0/719348323 >> 172.21.15.57:0/719348323 conn(0x7f05300ce870 msgr2=0x7f05300cec80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:44.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.456+0000 7f7636ffd640 1 -- 172.21.15.57:0/826035052 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f762c0047a0 con 0x7f7630050170 2023-10-19T12:44:44.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.457+0000 7fe40ca32640 1 -- 172.21.15.57:0/3048212815 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe4000554f0 msgr2=0x7fe400054240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:44.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.456+0000 7f28ed700640 1 --2- 172.21.15.57:0/4230478659 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f28e81557f0 0x7f28e80aad90 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.456+0000 7f28ed700640 1 -- 172.21.15.57:0/4230478659 >> 172.21.15.57:0/4230478659 conn(0x7f28e80701b0 msgr2=0x7f28e80705c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:44.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.457+0000 7f28ed700640 1 -- 172.21.15.57:0/4230478659 shutdown_connections 2023-10-19T12:44:44.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.457+0000 7fe40ca32640 1 --2- 172.21.15.57:0/3048212815 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe4000554f0 0x7fe400054240 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe3f8009830 tx=0x7fe3f802bc30 comp rx=0 tx=0).stop 2023-10-19T12:44:44.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.457+0000 7f05349ed640 1 -- 172.21.15.57:0/719348323 shutdown_connections 2023-10-19T12:44:44.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.457+0000 7f763d1e2640 1 -- 172.21.15.57:0/826035052 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7630050170 msgr2=0x7f7630038050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:44.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.457+0000 7f763d1e2640 1 --2- 172.21.15.57:0/826035052 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7630050170 0x7f7630038050 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f763000b3c0 tx=0x7f762c02ca30 comp rx=0 tx=0).stop 2023-10-19T12:44:44.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.457+0000 7f763d1e2640 1 -- 172.21.15.57:0/826035052 shutdown_connections 2023-10-19T12:44:44.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.457+0000 7f763d1e2640 1 --2- 172.21.15.57:0/826035052 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7630050170 0x7f7630038050 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.457+0000 7f763d1e2640 1 --2- 172.21.15.57:0/826035052 >> v2:172.21.15.57:3300/0 conn(0x7f7630053c40 0x7f76300069e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.457+0000 7f763d1e2640 1 -- 172.21.15.57:0/826035052 >> 172.21.15.57:0/826035052 conn(0x7f7630009f80 msgr2=0x7f763000a360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:44.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.457+0000 7f763d1e2640 1 -- 172.21.15.57:0/826035052 shutdown_connections 2023-10-19T12:44:44.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.458+0000 7fe00239a640 1 -- 172.21.15.57:0/442807380 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdfe8005180 msgr2=0x7fdfe800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:44.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.458+0000 7fe00239a640 1 --2- 172.21.15.57:0/442807380 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdfe8005180 0x7fdfe800b980 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fdffc068340 tx=0x7fdfe80057f0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.458+0000 7fe00239a640 1 -- 172.21.15.57:0/442807380 shutdown_connections 2023-10-19T12:44:44.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.458+0000 7fe00239a640 1 --2- 172.21.15.57:0/442807380 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdfe8005180 0x7fdfe800b980 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.458+0000 7fe00239a640 1 -- 172.21.15.57:0/442807380 >> 172.21.15.57:0/442807380 conn(0x7fdff4009f80 msgr2=0x7fdff400a360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:44.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.458+0000 7f763d1e2640 1 -- 172.21.15.57:0/826035052 wait complete. 2023-10-19T12:44:44.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.458+0000 7f255f59e640 1 -- 172.21.15.57:0/1313840138 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2548004bb0 msgr2=0x7f2548004fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:44.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.458+0000 7fe00239a640 1 -- 172.21.15.57:0/442807380 shutdown_connections 2023-10-19T12:44:44.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.458+0000 7fe00239a640 1 -- 172.21.15.57:0/442807380 wait complete. 2023-10-19T12:44:44.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.458+0000 7f255f59e640 1 --2- 172.21.15.57:0/1313840138 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2548004bb0 0x7f2548004fb0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2554006910 tx=0x7f25480057f0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.458+0000 7f255f59e640 1 -- 172.21.15.57:0/1313840138 shutdown_connections 2023-10-19T12:44:44.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.458+0000 7f255f59e640 1 --2- 172.21.15.57:0/1313840138 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2548004bb0 0x7f2548004fb0 secure :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2554006910 tx=0x7f25480057f0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.458+0000 7f255f59e640 1 -- 172.21.15.57:0/1313840138 >> 172.21.15.57:0/1313840138 conn(0x7f256006a830 msgr2=0x7f256006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:44.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.457+0000 7fe40ca32640 1 -- 172.21.15.57:0/3048212815 shutdown_connections 2023-10-19T12:44:44.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.458+0000 7f28ed700640 1 -- 172.21.15.57:0/4230478659 wait complete. 2023-10-19T12:44:44.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.458+0000 7fe40ca32640 1 --2- 172.21.15.57:0/3048212815 >> v2:172.21.15.179:3300/0 conn(0x7fe400035ea0 0x7fe400007bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.458+0000 7fe40ca32640 1 --2- 172.21.15.57:0/3048212815 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe4000554f0 0x7fe400054240 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.458+0000 7fe40ca32640 1 -- 172.21.15.57:0/3048212815 >> 172.21.15.57:0/3048212815 conn(0x7fe400053720 msgr2=0x7fe400053b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:44.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.458+0000 7f255f59e640 1 -- 172.21.15.57:0/1313840138 shutdown_connections 2023-10-19T12:44:44.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.459+0000 7fe40ca32640 1 -- 172.21.15.57:0/3048212815 shutdown_connections 2023-10-19T12:44:44.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.459+0000 7fe40ca32640 1 -- 172.21.15.57:0/3048212815 wait complete. 2023-10-19T12:44:44.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.457+0000 7f05349ed640 1 -- 172.21.15.57:0/719348323 wait complete. 2023-10-19T12:44:44.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.459+0000 7f05349ed640 1 Processor -- start 2023-10-19T12:44:44.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.459+0000 7f2731852640 1 -- 172.21.15.57:0/3350934110 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f27240549a0 msgr2=0x7f2724050090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:44.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.459+0000 7f2731852640 1 --2- 172.21.15.57:0/3350934110 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f27240549a0 0x7f2724050090 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2724008960 tx=0x7f272002beb0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.458+0000 7fde353cc640 1 -- 172.21.15.57:0/2443329519 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde20004a00 con 0x7fde301411e0 2023-10-19T12:44:44.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.459+0000 7f2731852640 1 -- 172.21.15.57:0/3350934110 shutdown_connections 2023-10-19T12:44:44.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.459+0000 7f2731852640 1 --2- 172.21.15.57:0/3350934110 >> v2:172.21.15.57:3300/0 conn(0x7f2724052ab0 0x7f2724052f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.459+0000 7f2731852640 1 --2- 172.21.15.57:0/3350934110 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f27240549a0 0x7f2724050090 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.459+0000 7f2731852640 1 --2- 172.21.15.57:0/3350934110 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f272404fcb0 0x7f27240540a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.459+0000 7f2731852640 1 -- 172.21.15.57:0/3350934110 >> 172.21.15.57:0/3350934110 conn(0x7f2724054ee0 msgr2=0x7f27240552f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:44.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.459+0000 7fde373d0640 1 -- 172.21.15.57:0/2443329519 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fde301411e0 msgr2=0x7fde3006d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:44.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.459+0000 7f05349ed640 1 -- start start 2023-10-19T12:44:44.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.459+0000 7f763d1e2640 1 Processor -- start 2023-10-19T12:44:44.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.460+0000 7f763d1e2640 1 -- start start 2023-10-19T12:44:44.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.461+0000 7f763d1e2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7630050170 0x7f7630173f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:44.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.461+0000 7f7637fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7630050170 0x7f7630173f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:44.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.461+0000 7f7637fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7630050170 0x7f7630173f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50818/0 (socket says 172.21.15.57:50818) 2023-10-19T12:44:44.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.461+0000 7f7637fff640 1 -- 172.21.15.57:0/3141884811 learned_addr learned my addr 172.21.15.57:0/3141884811 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:44.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.461+0000 7f763d1e2640 1 --2- 172.21.15.57:0/3141884811 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7630053c40 0x7f76300e2030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:44.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.462+0000 7f763d1e2640 1 --2- 172.21.15.57:0/3141884811 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f76300df1f0 0x7f7630180c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:44.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.462+0000 7f76377fe640 1 --2- 172.21.15.57:0/3141884811 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7630053c40 0x7f76300e2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:44.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.462+0000 7f763d1e2640 1 -- 172.21.15.57:0/3141884811 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7630041100 con 0x7f7630050170 2023-10-19T12:44:44.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.462+0000 7f763d1e2640 1 -- 172.21.15.57:0/3141884811 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76300dfc70 con 0x7f76300df1f0 2023-10-19T12:44:44.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.462+0000 7f763d1e2640 1 -- 172.21.15.57:0/3141884811 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76300dff50 con 0x7f7630053c40 2023-10-19T12:44:44.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.463+0000 7f7637fff640 1 -- 172.21.15.57:0/3141884811 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7630053c40 msgr2=0x7f76300e2030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:44.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.463+0000 7f7637fff640 1 --2- 172.21.15.57:0/3141884811 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7630053c40 0x7f76300e2030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.463+0000 7f7637fff640 1 -- 172.21.15.57:0/3141884811 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f76300df1f0 msgr2=0x7f7630180c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T12:44:44.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.463+0000 7f7637fff640 1 --2- 172.21.15.57:0/3141884811 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f76300df1f0 0x7f7630180c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.463+0000 7f7637fff640 1 -- 172.21.15.57:0/3141884811 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f762c0071c0 con 0x7f7630050170 2023-10-19T12:44:44.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.463+0000 7f76377fe640 1 --2- 172.21.15.57:0/3141884811 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7630053c40 0x7f76300e2030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:44:44.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.463+0000 7f7637fff640 1 --2- 172.21.15.57:0/3141884811 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7630050170 0x7f7630173f10 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f762400c750 tx=0x7f762400cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:44.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.461+0000 7f05349ed640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f05300cafa0 0x7f05302011c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:44.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.461+0000 7f05349ed640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f05300cd1c0 0x7f0530099e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:44.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.461+0000 7f05349ed640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0530158eb0 0x7f05300c47e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:44.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.461+0000 7f05349ed640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0530202160 con 0x7f05300cafa0 2023-10-19T12:44:44.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.461+0000 7f052effd640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f05300cd1c0 0x7f0530099e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:44.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.461+0000 7f052effd640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f05300cd1c0 0x7f0530099e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3301/0 says I am v2:172.21.15.57:48112/0 (socket says 172.21.15.57:48112) 2023-10-19T12:44:44.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.461+0000 7f052effd640 1 -- 172.21.15.57:0/175308918 learned_addr learned my addr 172.21.15.57:0/175308918 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:44.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.461+0000 7f052ffff640 1 --2- 172.21.15.57:0/175308918 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0530158eb0 0x7f05300c47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:44.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.461+0000 7f05349ed640 1 -- 172.21.15.57:0/175308918 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f053009a370 con 0x7f0530158eb0 2023-10-19T12:44:44.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.461+0000 7f05349ed640 1 -- 172.21.15.57:0/175308918 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05300c5b70 con 0x7f05300cd1c0 2023-10-19T12:44:44.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.462+0000 7f052ffff640 1 -- 172.21.15.57:0/175308918 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f05300cd1c0 msgr2=0x7f0530099e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:44.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.462+0000 7f052ffff640 1 --2- 172.21.15.57:0/175308918 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f05300cd1c0 0x7f0530099e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.462+0000 7f052ffff640 1 -- 172.21.15.57:0/175308918 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f05300cafa0 msgr2=0x7f05302011c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T12:44:44.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.462+0000 7f052ffff640 1 --2- 172.21.15.57:0/175308918 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f05300cafa0 0x7f05302011c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.462+0000 7f052ffff640 1 -- 172.21.15.57:0/175308918 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0528009840 con 0x7f0530158eb0 2023-10-19T12:44:44.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.461+0000 7fe40ca32640 1 Processor -- start 2023-10-19T12:44:44.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.461+0000 7fe40ca32640 1 -- start start 2023-10-19T12:44:44.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.462+0000 7fe40ca32640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe400035ea0 0x7fe40017bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:44.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.462+0000 7fe40ca32640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe4000554f0 0x7fe40004b650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:44.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.462+0000 7fe40ca32640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe4000558d0 0x7fe40004ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:44.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.462+0000 7fe40ca32640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe40017c040 con 0x7fe4000558d0 2023-10-19T12:44:44.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.462+0000 7fe40ca32640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe40004bb90 con 0x7fe400035ea0 2023-10-19T12:44:44.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.462+0000 7fe40ca32640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe40004be30 con 0x7fe4000554f0 2023-10-19T12:44:44.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.462+0000 7fe405d9b640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe4000554f0 0x7fe40004b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:44.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.463+0000 7fe405d9b640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe4000554f0 0x7fe40004b650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3301/0 says I am v2:172.21.15.57:48142/0 (socket says 172.21.15.57:48142) 2023-10-19T12:44:44.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.463+0000 7fe405d9b640 1 -- 172.21.15.57:0/1853330748 learned_addr learned my addr 172.21.15.57:0/1853330748 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:44.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.463+0000 7fe406d9d640 1 --2- 172.21.15.57:0/1853330748 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe4000558d0 0x7fe40004ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:44.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.463+0000 7fe406d9d640 1 -- 172.21.15.57:0/1853330748 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe4000554f0 msgr2=0x7fe40004b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:44.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.463+0000 7fe406d9d640 1 --2- 172.21.15.57:0/1853330748 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe4000554f0 0x7fe40004b650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.463+0000 7fe406d9d640 1 -- 172.21.15.57:0/1853330748 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe400035ea0 msgr2=0x7fe40017bb00 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-19T12:44:44.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.463+0000 7fe406d9d640 1 --2- 172.21.15.57:0/1853330748 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe400035ea0 0x7fe40017bb00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.463+0000 7fe406d9d640 1 -- 172.21.15.57:0/1853330748 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3f8009490 con 0x7fe4000558d0 2023-10-19T12:44:44.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.460+0000 7fe00239a640 1 Processor -- start 2023-10-19T12:44:44.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.460+0000 7fe00239a640 1 -- start start 2023-10-19T12:44:44.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.461+0000 7fe00239a640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdfe8005180 0x7fdff40dfaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:44.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.461+0000 7fe00239a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdff4033820 0x7fdff404cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:44.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.461+0000 7fe00239a640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdff40ddf40 0x7fdff404d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:44.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.462+0000 7fe001398640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdfe8005180 0x7fdff40dfaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:44.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.462+0000 7fe001398640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdfe8005180 0x7fdff40dfaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3301/0 says I am v2:172.21.15.57:48126/0 (socket says 172.21.15.57:48126) 2023-10-19T12:44:44.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.462+0000 7fe000b97640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdff4033820 0x7fdff404cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:44.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.462+0000 7fe00239a640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdff404c950 con 0x7fdff4033820 2023-10-19T12:44:44.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.462+0000 7fe001398640 1 -- 172.21.15.57:0/192370102 learned_addr learned my addr 172.21.15.57:0/192370102 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:44.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.462+0000 7fe00239a640 1 -- 172.21.15.57:0/192370102 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdff40dffe0 con 0x7fdff40ddf40 2023-10-19T12:44:44.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.462+0000 7fe00239a640 1 -- 172.21.15.57:0/192370102 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdff404a9f0 con 0x7fdfe8005180 2023-10-19T12:44:44.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.462+0000 7fe001b99640 1 --2- 172.21.15.57:0/192370102 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdff40ddf40 0x7fdff404d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:44.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.462+0000 7fe001b99640 1 -- 172.21.15.57:0/192370102 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdfe8005180 msgr2=0x7fdff40dfaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:44.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.462+0000 7fe001b99640 1 --2- 172.21.15.57:0/192370102 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdfe8005180 0x7fdff40dfaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.462+0000 7fe001b99640 1 -- 172.21.15.57:0/192370102 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdff4033820 msgr2=0x7fdff404cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:44.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.462+0000 7fe001b99640 1 --2- 172.21.15.57:0/192370102 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdff4033820 0x7fdff404cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.463+0000 7fe001b99640 1 -- 172.21.15.57:0/192370102 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdff404acd0 con 0x7fdff40ddf40 2023-10-19T12:44:44.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.463+0000 7fe000b97640 1 --2- 172.21.15.57:0/192370102 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdff4033820 0x7fdff404cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T12:44:44.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.463+0000 7fe001398640 1 --2- 172.21.15.57:0/192370102 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdfe8005180 0x7fdff40dfaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:44:44.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.463+0000 7fe001b99640 1 --2- 172.21.15.57:0/192370102 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdff40ddf40 0x7fdff404d2b0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fdffc063540 tx=0x7fdffc0ab5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:44.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.459+0000 7f255f59e640 1 -- 172.21.15.57:0/1313840138 wait complete. 2023-10-19T12:44:44.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.461+0000 7f255f59e640 1 Processor -- start 2023-10-19T12:44:44.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.461+0000 7f255f59e640 1 -- start start 2023-10-19T12:44:44.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.463+0000 7f255f59e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2560165100 0x7f2560143850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:44.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.463+0000 7f255f59e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25601654e0 0x7f2560165a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:44.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.463+0000 7f255f59e640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2560165fb0 0x7f2560168980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:44.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.463+0000 7f255f59e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2560166570 con 0x7f25601654e0 2023-10-19T12:44:44.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.463+0000 7f255f59e640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2560142a90 con 0x7f2560165100 2023-10-19T12:44:44.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.463+0000 7f255f59e640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2560142d70 con 0x7f2560165fb0 2023-10-19T12:44:44.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.463+0000 7f255e59c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2560165100 0x7f2560143850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:44.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.463+0000 7f255e59c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2560165100 0x7f2560143850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.57:43310/0 (socket says 172.21.15.57:43310) 2023-10-19T12:44:44.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.463+0000 7f255e59c640 1 -- 172.21.15.57:0/542508038 learned_addr learned my addr 172.21.15.57:0/542508038 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:44.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.463+0000 7f255dd9b640 1 --2- 172.21.15.57:0/542508038 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25601654e0 0x7f2560165a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:44.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.461+0000 7f28ed700640 1 Processor -- start 2023-10-19T12:44:44.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.461+0000 7f28ed700640 1 -- start start 2023-10-19T12:44:44.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.463+0000 7f28ed700640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f28e80ab2d0 0x7f28e809a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:44.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.463+0000 7f28ed700640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f28e80b2cb0 0x7f28e81f0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:44.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.463+0000 7f28e6ffd640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f28e80ab2d0 0x7f28e809a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:44.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.463+0000 7f28e6ffd640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f28e80ab2d0 0x7f28e809a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3301/0 says I am v2:172.21.15.57:48150/0 (socket says 172.21.15.57:48150) 2023-10-19T12:44:44.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.463+0000 7f28e6ffd640 1 -- 172.21.15.57:0/1801367852 learned_addr learned my addr 172.21.15.57:0/1801367852 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:44.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.463+0000 7f28ed700640 1 --2- 172.21.15.57:0/1801367852 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f28e81557f0 0x7f28e81f0b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:44.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.463+0000 7f28ed700640 1 -- 172.21.15.57:0/1801367852 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28e809ae00 con 0x7f28e80b2cb0 2023-10-19T12:44:44.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.463+0000 7f28ed700640 1 -- 172.21.15.57:0/1801367852 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28e809b0e0 con 0x7f28e81557f0 2023-10-19T12:44:44.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.463+0000 7f28ed700640 1 -- 172.21.15.57:0/1801367852 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28e809b3c0 con 0x7f28e80ab2d0 2023-10-19T12:44:44.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.459+0000 7f2731852640 1 -- 172.21.15.57:0/3350934110 shutdown_connections 2023-10-19T12:44:44.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.459+0000 7f2731852640 1 -- 172.21.15.57:0/3350934110 wait complete. 2023-10-19T12:44:44.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.462+0000 7f2731852640 1 Processor -- start 2023-10-19T12:44:44.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.462+0000 7f2731852640 1 -- start start 2023-10-19T12:44:44.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.459+0000 7fde373d0640 1 --2- 172.21.15.57:0/2443329519 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fde301411e0 0x7fde3006d4c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fde300b1da0 tx=0x7fde2002bd90 comp rx=0 tx=0).stop 2023-10-19T12:44:44.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.460+0000 7fde373d0640 1 -- 172.21.15.57:0/2443329519 shutdown_connections 2023-10-19T12:44:44.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.460+0000 7fde373d0640 1 --2- 172.21.15.57:0/2443329519 >> v2:172.21.15.57:3300/0 conn(0x7fde300b09e0 0x7fde300b0e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.460+0000 7fde373d0640 1 --2- 172.21.15.57:0/2443329519 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fde3015c0c0 0x7fde30070850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.460+0000 7fde373d0640 1 --2- 172.21.15.57:0/2443329519 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fde301411e0 0x7fde3006d4c0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.460+0000 7fde373d0640 1 -- 172.21.15.57:0/2443329519 >> 172.21.15.57:0/2443329519 conn(0x7fde300ae5d0 msgr2=0x7fde300ae9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:44.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.460+0000 7fde373d0640 1 -- 172.21.15.57:0/2443329519 shutdown_connections 2023-10-19T12:44:44.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.460+0000 7fde373d0640 1 -- 172.21.15.57:0/2443329519 wait complete. 2023-10-19T12:44:44.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.462+0000 7fde373d0640 1 Processor -- start 2023-10-19T12:44:44.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.462+0000 7fde373d0640 1 -- start start 2023-10-19T12:44:44.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.463+0000 7fe406d9d640 1 --2- 172.21.15.57:0/1853330748 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe4000558d0 0x7fe40004ced0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fe408068a70 tx=0x7fe408069570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:44.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.463+0000 7fde373d0640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fde300b09e0 0x7fde301f0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:44.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.463+0000 7fde373d0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fde301411e0 0x7fde30136700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:44.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.463+0000 7fde373d0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde3015c0c0 0x7fde30157cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:44.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.463+0000 7f7634ff9640 1 -- 172.21.15.57:0/3141884811 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76240050d0 con 0x7f7630050170 2023-10-19T12:44:44.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.464+0000 7f7634ff9640 1 -- 172.21.15.57:0/3141884811 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f7624005250 con 0x7f7630050170 2023-10-19T12:44:44.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.464+0000 7f7634ff9640 1 -- 172.21.15.57:0/3141884811 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f762400a640 con 0x7f7630050170 2023-10-19T12:44:44.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.464+0000 7fde363ce640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fde300b09e0 0x7fde301f0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:44.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.464+0000 7fde363ce640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fde300b09e0 0x7fde301f0ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3301/0 says I am v2:172.21.15.57:48158/0 (socket says 172.21.15.57:48158) 2023-10-19T12:44:44.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.463+0000 7f28e7fff640 1 --2- 172.21.15.57:0/1801367852 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f28e81557f0 0x7f28e81f0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:44.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.464+0000 7fde363ce640 1 -- 172.21.15.57:0/4126217862 learned_addr learned my addr 172.21.15.57:0/4126217862 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:44.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.464+0000 7fde373d0640 1 -- 172.21.15.57:0/4126217862 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde301f1400 con 0x7fde3015c0c0 2023-10-19T12:44:44.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.464+0000 7fde373d0640 1 -- 172.21.15.57:0/4126217862 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde30136c40 con 0x7fde301411e0 2023-10-19T12:44:44.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.464+0000 7fde373d0640 1 -- 172.21.15.57:0/4126217862 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde301f1660 con 0x7fde300b09e0 2023-10-19T12:44:44.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.464+0000 7f763d1e2640 1 -- 172.21.15.57:0/3141884811 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76300e0230 con 0x7f7630050170 2023-10-19T12:44:44.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.464+0000 7f763d1e2640 1 -- 172.21.15.57:0/3141884811 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7630174450 con 0x7f7630050170 2023-10-19T12:44:44.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.463+0000 7f28e67fc640 1 --2- 172.21.15.57:0/1801367852 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f28e80b2cb0 0x7f28e81f0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:44.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.463+0000 7fdff1ffb640 1 -- 172.21.15.57:0/192370102 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdffc04df90 con 0x7fdff40ddf40 2023-10-19T12:44:44.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.463+0000 7fdff1ffb640 1 -- 172.21.15.57:0/192370102 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fdffc0ab8d0 con 0x7fdff40ddf40 2023-10-19T12:44:44.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.464+0000 7fdff1ffb640 1 -- 172.21.15.57:0/192370102 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdffc0abc80 con 0x7fdff40ddf40 2023-10-19T12:44:44.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.464+0000 7f28e7fff640 1 -- 172.21.15.57:0/1801367852 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f28e80ab2d0 msgr2=0x7f28e809a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:44.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.464+0000 7f28e7fff640 1 --2- 172.21.15.57:0/1801367852 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f28e80ab2d0 0x7f28e809a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.464+0000 7f28e7fff640 1 -- 172.21.15.57:0/1801367852 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f28e80b2cb0 msgr2=0x7f28e81f0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:44.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.464+0000 7f28e7fff640 1 --2- 172.21.15.57:0/1801367852 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f28e80b2cb0 0x7f28e81f0610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.464+0000 7f28e7fff640 1 -- 172.21.15.57:0/1801367852 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28d8009560 con 0x7f28e81557f0 2023-10-19T12:44:44.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.464+0000 7f28e6ffd640 1 --2- 172.21.15.57:0/1801367852 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f28e80ab2d0 0x7f28e809a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:44:44.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.464+0000 7f2731852640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f272404fcb0 0x7f2724036d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:44.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.464+0000 7fde36bcf640 1 --2- 172.21.15.57:0/4126217862 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde3015c0c0 0x7fde30157cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:44.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.464+0000 7fde35bcd640 1 --2- 172.21.15.57:0/4126217862 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fde301411e0 0x7fde30136700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:44.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.464+0000 7fe00239a640 1 -- 172.21.15.57:0/192370102 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdff404afb0 con 0x7fdff40ddf40 2023-10-19T12:44:44.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.464+0000 7f255dd9b640 1 -- 172.21.15.57:0/542508038 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2560165fb0 msgr2=0x7f2560168980 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-19T12:44:44.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.464+0000 7f255dd9b640 1 --2- 172.21.15.57:0/542508038 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2560165fb0 0x7f2560168980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.464+0000 7f255dd9b640 1 -- 172.21.15.57:0/542508038 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2560165100 msgr2=0x7f2560143850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:44.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.464+0000 7f255dd9b640 1 --2- 172.21.15.57:0/542508038 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2560165100 0x7f2560143850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.464+0000 7f052ffff640 1 --2- 172.21.15.57:0/175308918 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0530158eb0 0x7f05300c47e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f05300705b0 tx=0x7f0528009320 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:44.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.464+0000 7f255dd9b640 1 -- 172.21.15.57:0/542508038 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2560143050 con 0x7f25601654e0 2023-10-19T12:44:44.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.465+0000 7fde363ce640 1 -- 172.21.15.57:0/4126217862 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fde301411e0 msgr2=0x7fde30136700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:44.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.465+0000 7fde363ce640 1 --2- 172.21.15.57:0/4126217862 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fde301411e0 0x7fde30136700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.465+0000 7fde363ce640 1 -- 172.21.15.57:0/4126217862 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde3015c0c0 msgr2=0x7fde30157cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:44.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.465+0000 7fde363ce640 1 --2- 172.21.15.57:0/4126217862 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde3015c0c0 0x7fde30157cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.465+0000 7fde363ce640 1 -- 172.21.15.57:0/4126217862 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde28009840 con 0x7fde300b09e0 2023-10-19T12:44:44.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.465+0000 7fde36bcf640 1 --2- 172.21.15.57:0/4126217862 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde3015c0c0 0x7fde30157cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T12:44:44.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.465+0000 7f255dd9b640 1 --2- 172.21.15.57:0/542508038 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25601654e0 0x7f2560165a70 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f2558010bb0 tx=0x7f255800e6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:44.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.465+0000 7fde35bcd640 1 --2- 172.21.15.57:0/4126217862 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fde301411e0 0x7fde30136700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:44:44.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.465+0000 7f0517fff640 1 -- 172.21.15.57:0/175308918 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f052803d600 con 0x7f0530158eb0 2023-10-19T12:44:44.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.464+0000 7fe3f6ffd640 1 -- 172.21.15.57:0/1853330748 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe40808d070 con 0x7fe4000558d0 2023-10-19T12:44:44.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.464+0000 7fe3f6ffd640 1 -- 172.21.15.57:0/1853330748 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fe408066ee0 con 0x7fe4000558d0 2023-10-19T12:44:44.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.464+0000 7fe3f6ffd640 1 -- 172.21.15.57:0/1853330748 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe408072220 con 0x7fe4000558d0 2023-10-19T12:44:44.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.464+0000 7fe40ca32640 1 -- 172.21.15.57:0/1853330748 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe40004c110 con 0x7fe4000558d0 2023-10-19T12:44:44.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.464+0000 7fe40ca32640 1 -- 172.21.15.57:0/1853330748 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe40004c6d0 con 0x7fe4000558d0 2023-10-19T12:44:44.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.464+0000 7f28e7fff640 1 --2- 172.21.15.57:0/1801367852 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f28e81557f0 0x7f28e81f0b50 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f28e000be20 tx=0x7f28e000bf10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:44.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.465+0000 7f28cf7fe640 1 -- 172.21.15.57:0/1801367852 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28e000ce70 con 0x7f28e81557f0 2023-10-19T12:44:44.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.465+0000 7fde363ce640 1 --2- 172.21.15.57:0/4126217862 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fde300b09e0 0x7fde301f0ec0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fde20009850 tx=0x7fde2002d970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:44.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.465+0000 7f76227fc640 1 -- 172.21.15.57:0/3141884811 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7610005250 con 0x7f7630050170 2023-10-19T12:44:44.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.465+0000 7fe00239a640 1 -- 172.21.15.57:0/192370102 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdff4186430 con 0x7fdff40ddf40 2023-10-19T12:44:44.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.465+0000 7f28cf7fe640 1 -- 172.21.15.57:0/1801367852 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f28e0014440 con 0x7f28e81557f0 2023-10-19T12:44:44.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.465+0000 7f28ed700640 1 -- 172.21.15.57:0/1801367852 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28e809b6a0 con 0x7f28e81557f0 2023-10-19T12:44:44.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.465+0000 7f05349ed640 1 -- 172.21.15.57:0/175308918 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05302018c0 con 0x7f0530158eb0 2023-10-19T12:44:44.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.465+0000 7f05349ed640 1 -- 172.21.15.57:0/175308918 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0530201e80 con 0x7f0530158eb0 2023-10-19T12:44:44.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.466+0000 7f28ed700640 1 -- 172.21.15.57:0/1801367852 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28e8097c50 con 0x7f28e81557f0 2023-10-19T12:44:44.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.466+0000 7f0517fff640 1 -- 172.21.15.57:0/175308918 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f052803d850 con 0x7f0530158eb0 2023-10-19T12:44:44.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.466+0000 7f0517fff640 1 -- 172.21.15.57:0/175308918 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f052803dc00 con 0x7f0530158eb0 2023-10-19T12:44:44.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.466+0000 7f28cf7fe640 1 -- 172.21.15.57:0/1801367852 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28e0014890 con 0x7f28e81557f0 2023-10-19T12:44:44.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.465+0000 7f2731852640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2724052ab0 0x7f272404b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:44.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.465+0000 7f2731852640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f27240549a0 0x7f2724174a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:44.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.465+0000 7f2731852640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27241797f0 con 0x7f272404fcb0 2023-10-19T12:44:44.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.465+0000 7f2731852640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f272404b8d0 con 0x7f2724052ab0 2023-10-19T12:44:44.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.465+0000 7f2731852640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2724037370 con 0x7f27240549a0 2023-10-19T12:44:44.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.466+0000 7f2730850640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f272404fcb0 0x7f2724036d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:44.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.466+0000 7f2730850640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f272404fcb0 0x7f2724036d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50878/0 (socket says 172.21.15.57:50878) 2023-10-19T12:44:44.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.466+0000 7fe40ca32640 1 -- 172.21.15.57:0/1853330748 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4000057b0 con 0x7fe4000558d0 2023-10-19T12:44:44.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.466+0000 7f2730850640 1 -- 172.21.15.57:0/1088338452 learned_addr learned my addr 172.21.15.57:0/1088338452 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:44.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.466+0000 7fde26ffd640 1 -- 172.21.15.57:0/4126217862 <== mon.2 v2:172.21.15.179:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde2002dd70 con 0x7fde300b09e0 2023-10-19T12:44:44.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.466+0000 7fde26ffd640 1 -- 172.21.15.57:0/4126217862 <== mon.2 v2:172.21.15.179:3301/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fde200035f0 con 0x7fde300b09e0 2023-10-19T12:44:44.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.466+0000 7fde26ffd640 1 -- 172.21.15.57:0/4126217862 <== mon.2 v2:172.21.15.179:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde20003980 con 0x7fde300b09e0 2023-10-19T12:44:44.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.466+0000 7fde373d0640 1 -- 172.21.15.57:0/4126217862 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde20009560 con 0x7fde300b09e0 2023-10-19T12:44:44.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.466+0000 7fde373d0640 1 -- 172.21.15.57:0/4126217862 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde301f1c20 con 0x7fde300b09e0 2023-10-19T12:44:44.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.466+0000 7f2731051640 1 --2- 172.21.15.57:0/1088338452 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f27240549a0 0x7f2724174a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:44.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.467+0000 7f272bfff640 1 --2- 172.21.15.57:0/1088338452 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2724052ab0 0x7f272404b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:44.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.467+0000 7f2730850640 1 -- 172.21.15.57:0/1088338452 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f27240549a0 msgr2=0x7f2724174a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:44.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.467+0000 7f2730850640 1 --2- 172.21.15.57:0/1088338452 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f27240549a0 0x7f2724174a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.467+0000 7f2730850640 1 -- 172.21.15.57:0/1088338452 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2724052ab0 msgr2=0x7f272404b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:44.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.467+0000 7f2730850640 1 --2- 172.21.15.57:0/1088338452 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2724052ab0 0x7f272404b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.467+0000 7f2730850640 1 -- 172.21.15.57:0/1088338452 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2720009df0 con 0x7f272404fcb0 2023-10-19T12:44:44.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.467+0000 7f2730850640 1 --2- 172.21.15.57:0/1088338452 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f272404fcb0 0x7f2724036d70 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f272c04f370 tx=0x7f272c0679c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:44.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.468+0000 7f2731051640 1 --2- 172.21.15.57:0/1088338452 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f27240549a0 0x7f2724174a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:44:44.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.468+0000 7fdff1ffb640 1 -- 172.21.15.57:0/192370102 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fdffc0b6400 con 0x7fdff40ddf40 2023-10-19T12:44:44.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.468+0000 7f0517fff640 1 -- 172.21.15.57:0/175308918 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f0528049460 con 0x7f0530158eb0 2023-10-19T12:44:44.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.469+0000 7fde24ff9640 1 -- 172.21.15.57:0/4126217862 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde3004da70 con 0x7fde300b09e0 2023-10-19T12:44:44.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.467+0000 7f7634ff9640 1 -- 172.21.15.57:0/3141884811 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f762400a7a0 con 0x7f7630050170 2023-10-19T12:44:44.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.468+0000 7f7634ff9640 1 --2- 172.21.15.57:0/3141884811 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f761c06af10 0x7f761c06d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:44.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.468+0000 7f7634ff9640 1 -- 172.21.15.57:0/3141884811 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f7624089570 con 0x7f7630050170 2023-10-19T12:44:44.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.468+0000 7f254effd640 1 -- 172.21.15.57:0/542508038 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2558004d20 con 0x7f25601654e0 2023-10-19T12:44:44.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.468+0000 7f254effd640 1 -- 172.21.15.57:0/542508038 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f2558004e80 con 0x7f25601654e0 2023-10-19T12:44:44.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.468+0000 7f254effd640 1 -- 172.21.15.57:0/542508038 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2558003bd0 con 0x7f25601654e0 2023-10-19T12:44:44.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.468+0000 7fdff1ffb640 1 --2- 172.21.15.57:0/192370102 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fdfe006af10 0x7fdfe006d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:44.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.468+0000 7fdff1ffb640 1 -- 172.21.15.57:0/192370102 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fdffc122d30 con 0x7fdff40ddf40 2023-10-19T12:44:44.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.469+0000 7f255f59e640 1 -- 172.21.15.57:0/542508038 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2560143330 con 0x7f25601654e0 2023-10-19T12:44:44.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.469+0000 7f255f59e640 1 -- 172.21.15.57:0/542508038 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2560168ec0 con 0x7f25601654e0 2023-10-19T12:44:44.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.469+0000 7fdfdf7fe640 1 -- 172.21.15.57:0/192370102 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdff404ca90 con 0x7fdff40ddf40 2023-10-19T12:44:44.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.468+0000 7f28cf7fe640 1 -- 172.21.15.57:0/1801367852 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f28e0014af0 con 0x7f28e81557f0 2023-10-19T12:44:44.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.469+0000 7f28cf7fe640 1 --2- 172.21.15.57:0/1801367852 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f28d406af10 0x7f28d406d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:44.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.469+0000 7f28cf7fe640 1 -- 172.21.15.57:0/1801367852 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f28e0089c90 con 0x7f28e81557f0 2023-10-19T12:44:44.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.469+0000 7f28ed700640 1 -- 172.21.15.57:0/1801367852 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28c8005250 con 0x7f28e81557f0 2023-10-19T12:44:44.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.469+0000 7fe001398640 1 --2- 172.21.15.57:0/192370102 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fdfe006af10 0x7fdfe006d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:44.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.469+0000 7f28e6ffd640 1 --2- 172.21.15.57:0/1801367852 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f28d406af10 0x7f28d406d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:44.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.469+0000 7f0517fff640 1 --2- 172.21.15.57:0/175308918 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f051c06af10 0x7f051c06d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:44.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.469+0000 7f05349ed640 1 -- 172.21.15.57:0/175308918 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0510005250 con 0x7f0530158eb0 2023-10-19T12:44:44.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.470+0000 7f0517fff640 1 -- 172.21.15.57:0/175308918 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f05280ac3f0 con 0x7f0530158eb0 2023-10-19T12:44:44.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.469+0000 7f76377fe640 1 --2- 172.21.15.57:0/3141884811 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f761c06af10 0x7f761c06d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:44.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.470+0000 7f052f7fe640 1 --2- 172.21.15.57:0/175308918 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f051c06af10 0x7f051c06d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:44.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.470+0000 7fe001398640 1 --2- 172.21.15.57:0/192370102 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fdfe006af10 0x7fdfe006d3d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fdff8009d50 tx=0x7fdff8009320 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:44.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.470+0000 7fe3f6ffd640 1 -- 172.21.15.57:0/1853330748 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fe408072380 con 0x7fe4000558d0 2023-10-19T12:44:44.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.470+0000 7f255f59e640 1 -- 172.21.15.57:0/542508038 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f256004da70 con 0x7f25601654e0 2023-10-19T12:44:44.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.470+0000 7f28e6ffd640 1 --2- 172.21.15.57:0/1801367852 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f28d406af10 0x7f28d406d3d0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f28d80057c0 tx=0x7f28d80046c0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:44.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.470+0000 7f76377fe640 1 --2- 172.21.15.57:0/3141884811 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f761c06af10 0x7f761c06d3d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f762c035000 tx=0x7f762c004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:44.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.470+0000 7f052f7fe640 1 --2- 172.21.15.57:0/175308918 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f051c06af10 0x7f051c06d3d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f0520005fd0 tx=0x7f052000b2f0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:44.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.471+0000 7fde26ffd640 1 -- 172.21.15.57:0/4126217862 <== mon.2 v2:172.21.15.179:3301/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fde20003ae0 con 0x7fde300b09e0 2023-10-19T12:44:44.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.471+0000 7fe3f6ffd640 1 --2- 172.21.15.57:0/1853330748 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fe3e806ae90 0x7fe3e806d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:44.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.471+0000 7fe3f6ffd640 1 -- 172.21.15.57:0/1853330748 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fe408101630 con 0x7fe4000558d0 2023-10-19T12:44:44.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.471+0000 7fe40659c640 1 --2- 172.21.15.57:0/1853330748 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fe3e806ae90 0x7fe3e806d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:44.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.471+0000 7fde26ffd640 1 --2- 172.21.15.57:0/4126217862 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fde1406af10 0x7fde1406d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:44.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.471+0000 7fde26ffd640 1 -- 172.21.15.57:0/4126217862 <== mon.2 v2:172.21.15.179:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fde200ac090 con 0x7fde300b09e0 2023-10-19T12:44:44.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.471+0000 7fe40659c640 1 --2- 172.21.15.57:0/1853330748 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fe3e806ae90 0x7fe3e806d350 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fe3f8036000 tx=0x7fe3f80046c0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:44.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.471+0000 7fde35bcd640 1 --2- 172.21.15.57:0/4126217862 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fde1406af10 0x7fde1406d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:44.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.472+0000 7fde35bcd640 1 --2- 172.21.15.57:0/4126217862 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fde1406af10 0x7fde1406d3d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fde300aeb20 tx=0x7fde2c005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:44.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.473+0000 7fe3f6ffd640 1 -- 172.21.15.57:0/1853330748 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe4080cc3d0 con 0x7fe4000558d0 2023-10-19T12:44:44.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.477+0000 7f7634ff9640 1 -- 172.21.15.57:0/3141884811 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7624054310 con 0x7f7630050170 2023-10-19T12:44:44.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.477+0000 7f28cf7fe640 1 -- 172.21.15.57:0/1801367852 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f28e0054ac0 con 0x7f28e81557f0 2023-10-19T12:44:44.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.478+0000 7f27297fa640 1 -- 172.21.15.57:0/1088338452 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f272c0721e0 con 0x7f272404fcb0 2023-10-19T12:44:44.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.478+0000 7f27297fa640 1 -- 172.21.15.57:0/1088338452 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f272c072430 con 0x7f272404fcb0 2023-10-19T12:44:44.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.478+0000 7f27297fa640 1 -- 172.21.15.57:0/1088338452 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f272c0725a0 con 0x7f272404fcb0 2023-10-19T12:44:44.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.479+0000 7f2731852640 1 -- 172.21.15.57:0/1088338452 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2724006e00 con 0x7f272404fcb0 2023-10-19T12:44:44.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.479+0000 7f2731852640 1 -- 172.21.15.57:0/1088338452 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2724037930 con 0x7f272404fcb0 2023-10-19T12:44:44.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.479+0000 7f254effd640 1 -- 172.21.15.57:0/542508038 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f255801a070 con 0x7f25601654e0 2023-10-19T12:44:44.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.480+0000 7f254effd640 1 --2- 172.21.15.57:0/542508038 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f254006af40 0x7f254006d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:44.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.480+0000 7f254effd640 1 -- 172.21.15.57:0/542508038 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f255808aaa0 con 0x7f25601654e0 2023-10-19T12:44:44.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.480+0000 7f2731852640 1 -- 172.21.15.57:0/1088338452 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2724008df0 con 0x7f272404fcb0 2023-10-19T12:44:44.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.480+0000 7f255e59c640 1 --2- 172.21.15.57:0/542508038 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f254006af40 0x7f254006d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:44.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.481+0000 7f255e59c640 1 --2- 172.21.15.57:0/542508038 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f254006af40 0x7f254006d400 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f2554001270 tx=0x7f25540014d0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:44.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.481+0000 7fde26ffd640 1 -- 172.21.15.57:0/4126217862 <== mon.2 v2:172.21.15.179:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fde20076ec0 con 0x7fde300b09e0 2023-10-19T12:44:44.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.484+0000 7f254effd640 1 -- 172.21.15.57:0/542508038 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f25580558f0 con 0x7f25601654e0 2023-10-19T12:44:44.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.484+0000 7fdff1ffb640 1 -- 172.21.15.57:0/192370102 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fdff404ca90 con 0x7fdff40ddf40 2023-10-19T12:44:44.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.485+0000 7f27297fa640 1 -- 172.21.15.57:0/1088338452 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f272c0729b0 con 0x7f272404fcb0 2023-10-19T12:44:44.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.486+0000 7f27297fa640 1 --2- 172.21.15.57:0/1088338452 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f271006af10 0x7f271006d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:44.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.486+0000 7f27297fa640 1 -- 172.21.15.57:0/1088338452 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f272c101740 con 0x7f272404fcb0 2023-10-19T12:44:44.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.487+0000 7f272bfff640 1 --2- 172.21.15.57:0/1088338452 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f271006af10 0x7f271006d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:44.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.487+0000 7f272bfff640 1 --2- 172.21.15.57:0/1088338452 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f271006af10 0x7f271006d3d0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f2720035000 tx=0x7f27200046c0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:44.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.491+0000 7f0517fff640 1 -- 172.21.15.57:0/175308918 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0528077340 con 0x7f0530158eb0 2023-10-19T12:44:44.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.493+0000 7f27297fa640 1 -- 172.21.15.57:0/1088338452 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f272c0cc620 con 0x7f272404fcb0 2023-10-19T12:44:44.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.872+0000 7fe40ca32640 1 -- 172.21.15.57:0/1853330748 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fe40003cc20 con 0x7fe4000558d0 2023-10-19T12:44:44.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.873+0000 7fe3f6ffd640 1 -- 172.21.15.57:0/1853330748 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe4080cf9f0 con 0x7fe4000558d0 2023-10-19T12:44:44.876 INFO:teuthology.orchestra.run.smithi057.stdout:55834574858 2023-10-19T12:44:44.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.876+0000 7fe3f4ff9640 1 -- 172.21.15.57:0/1853330748 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fe3e806ae90 msgr2=0x7fe3e806d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:44.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.876+0000 7fe3f4ff9640 1 --2- 172.21.15.57:0/1853330748 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fe3e806ae90 0x7fe3e806d350 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fe3f8036000 tx=0x7fe3f80046c0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.876+0000 7fe3f4ff9640 1 -- 172.21.15.57:0/1853330748 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe4000558d0 msgr2=0x7fe40004ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:44.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.876+0000 7fe3f4ff9640 1 --2- 172.21.15.57:0/1853330748 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe4000558d0 0x7fe40004ced0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fe408068a70 tx=0x7fe408069570 comp rx=0 tx=0).stop 2023-10-19T12:44:44.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.877+0000 7fe3f4ff9640 1 -- 172.21.15.57:0/1853330748 shutdown_connections 2023-10-19T12:44:44.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.877+0000 7fe3f4ff9640 1 --2- 172.21.15.57:0/1853330748 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fe3e806ae90 0x7fe3e806d350 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.877+0000 7fe3f4ff9640 1 --2- 172.21.15.57:0/1853330748 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe4000558d0 0x7fe40004ced0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.877+0000 7fe3f4ff9640 1 --2- 172.21.15.57:0/1853330748 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe4000554f0 0x7fe40004b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.877+0000 7fe3f4ff9640 1 --2- 172.21.15.57:0/1853330748 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe400035ea0 0x7fe40017bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.877+0000 7fe3f4ff9640 1 -- 172.21.15.57:0/1853330748 >> 172.21.15.57:0/1853330748 conn(0x7fe400053720 msgr2=0x7fe4000367b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:44.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.877+0000 7fe3f4ff9640 1 -- 172.21.15.57:0/1853330748 shutdown_connections 2023-10-19T12:44:44.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.877+0000 7fe3f4ff9640 1 -- 172.21.15.57:0/1853330748 wait complete. 2023-10-19T12:44:44.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.881+0000 7f76227fc640 1 -- 172.21.15.57:0/3141884811 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f7610005500 con 0x7f7630050170 2023-10-19T12:44:44.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.882+0000 7f7634ff9640 1 -- 172.21.15.57:0/3141884811 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f762405ce60 con 0x7f7630050170 2023-10-19T12:44:44.885 INFO:teuthology.orchestra.run.smithi057.stdout:55834574858 2023-10-19T12:44:44.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.885+0000 7f763d1e2640 1 -- 172.21.15.57:0/3141884811 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f761c06af10 msgr2=0x7f761c06d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:44.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.885+0000 7f763d1e2640 1 --2- 172.21.15.57:0/3141884811 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f761c06af10 0x7f761c06d3d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f762c035000 tx=0x7f762c004730 comp rx=0 tx=0).stop 2023-10-19T12:44:44.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.885+0000 7f763d1e2640 1 -- 172.21.15.57:0/3141884811 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7630050170 msgr2=0x7f7630173f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:44.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.885+0000 7f763d1e2640 1 --2- 172.21.15.57:0/3141884811 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7630050170 0x7f7630173f10 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f762400c750 tx=0x7f762400cc10 comp rx=0 tx=0).stop 2023-10-19T12:44:44.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.886+0000 7f763d1e2640 1 -- 172.21.15.57:0/3141884811 shutdown_connections 2023-10-19T12:44:44.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.886+0000 7f763d1e2640 1 --2- 172.21.15.57:0/3141884811 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f761c06af10 0x7f761c06d3d0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.886+0000 7f763d1e2640 1 --2- 172.21.15.57:0/3141884811 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f76300df1f0 0x7f7630180c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.886+0000 7f763d1e2640 1 --2- 172.21.15.57:0/3141884811 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7630053c40 0x7f76300e2030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.886+0000 7f763d1e2640 1 --2- 172.21.15.57:0/3141884811 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7630050170 0x7f7630173f10 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.886+0000 7f763d1e2640 1 -- 172.21.15.57:0/3141884811 >> 172.21.15.57:0/3141884811 conn(0x7f7630009f80 msgr2=0x7f7630047a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:44.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.886+0000 7f763d1e2640 1 -- 172.21.15.57:0/3141884811 shutdown_connections 2023-10-19T12:44:44.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.886+0000 7f763d1e2640 1 -- 172.21.15.57:0/3141884811 wait complete. 2023-10-19T12:44:44.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.892+0000 7fdfdf7fe640 1 -- 172.21.15.57:0/192370102 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fdff404ca90 con 0x7fdff40ddf40 2023-10-19T12:44:44.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.893+0000 7fdff1ffb640 1 -- 172.21.15.57:0/192370102 <== mon.1 v2:172.21.15.179:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fdff404ca90 con 0x7fdff40ddf40 2023-10-19T12:44:44.896 INFO:teuthology.orchestra.run.smithi057.stdout:55834574859 2023-10-19T12:44:44.897 INFO:teuthology.orchestra.run.smithi057.stdout:55834574859 2023-10-19T12:44:44.898 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574858 for osd.4 2023-10-19T12:44:44.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.893+0000 7f28ed700640 1 -- 172.21.15.57:0/1801367852 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f28c80050c0 con 0x7f28e81557f0 2023-10-19T12:44:44.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.893+0000 7f28cf7fe640 1 -- 172.21.15.57:0/1801367852 <== mon.1 v2:172.21.15.179:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f28e0019030 con 0x7f28e81557f0 2023-10-19T12:44:44.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.895+0000 7fde24ff9640 1 -- 172.21.15.57:0/4126217862 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fde30141670 con 0x7fde300b09e0 2023-10-19T12:44:44.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.896+0000 7f28ed700640 1 -- 172.21.15.57:0/1801367852 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f28d406af10 msgr2=0x7f28d406d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:44.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.896+0000 7f28ed700640 1 --2- 172.21.15.57:0/1801367852 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f28d406af10 0x7f28d406d3d0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f28d80057c0 tx=0x7f28d80046c0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.896+0000 7f28ed700640 1 -- 172.21.15.57:0/1801367852 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f28e81557f0 msgr2=0x7f28e81f0b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:44.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.896+0000 7f28ed700640 1 --2- 172.21.15.57:0/1801367852 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f28e81557f0 0x7f28e81f0b50 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f28e000be20 tx=0x7f28e000bf10 comp rx=0 tx=0).stop 2023-10-19T12:44:44.900 INFO:teuthology.orchestra.run.smithi057.stdout:55834574858 2023-10-19T12:44:44.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.896+0000 7fde26ffd640 1 -- 172.21.15.57:0/4126217862 <== mon.2 v2:172.21.15.179:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fde2007a4e0 con 0x7fde300b09e0 2023-10-19T12:44:44.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.896+0000 7f28ed700640 1 -- 172.21.15.57:0/1801367852 shutdown_connections 2023-10-19T12:44:44.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.896+0000 7f28ed700640 1 --2- 172.21.15.57:0/1801367852 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f28d406af10 0x7f28d406d3d0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.896+0000 7f28ed700640 1 --2- 172.21.15.57:0/1801367852 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f28e81557f0 0x7f28e81f0b50 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.896+0000 7f28ed700640 1 --2- 172.21.15.57:0/1801367852 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f28e80b2cb0 0x7f28e81f0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.896+0000 7f28ed700640 1 --2- 172.21.15.57:0/1801367852 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f28e80ab2d0 0x7f28e809a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.896+0000 7f28ed700640 1 -- 172.21.15.57:0/1801367852 >> 172.21.15.57:0/1801367852 conn(0x7f28e80701b0 msgr2=0x7f28e8070590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:44.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.896+0000 7fdfdf7fe640 1 -- 172.21.15.57:0/192370102 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fdfe006af10 msgr2=0x7fdfe006d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:44.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.896+0000 7fdfdf7fe640 1 --2- 172.21.15.57:0/192370102 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fdfe006af10 0x7fdfe006d3d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fdff8009d50 tx=0x7fdff8009320 comp rx=0 tx=0).stop 2023-10-19T12:44:44.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.896+0000 7fdfdf7fe640 1 -- 172.21.15.57:0/192370102 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdff40ddf40 msgr2=0x7fdff404d2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:44.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.896+0000 7fdfdf7fe640 1 --2- 172.21.15.57:0/192370102 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdff40ddf40 0x7fdff404d2b0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fdffc063540 tx=0x7fdffc0ab5d0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.897+0000 7f28ed700640 1 -- 172.21.15.57:0/1801367852 shutdown_connections 2023-10-19T12:44:44.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.897+0000 7f28ed700640 1 -- 172.21.15.57:0/1801367852 wait complete. 2023-10-19T12:44:44.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.897+0000 7fdfdf7fe640 1 -- 172.21.15.57:0/192370102 shutdown_connections 2023-10-19T12:44:44.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.897+0000 7fdfdf7fe640 1 --2- 172.21.15.57:0/192370102 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fdfe006af10 0x7fdfe006d3d0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.897+0000 7fdfdf7fe640 1 --2- 172.21.15.57:0/192370102 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdff40ddf40 0x7fdff404d2b0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.897+0000 7fdfdf7fe640 1 --2- 172.21.15.57:0/192370102 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdff4033820 0x7fdff404cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.897+0000 7fdfdf7fe640 1 --2- 172.21.15.57:0/192370102 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdfe8005180 0x7fdff40dfaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.897+0000 7fdfdf7fe640 1 -- 172.21.15.57:0/192370102 >> 172.21.15.57:0/192370102 conn(0x7fdff4009f80 msgr2=0x7fdff4006950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:44.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.897+0000 7fdfdf7fe640 1 -- 172.21.15.57:0/192370102 shutdown_connections 2023-10-19T12:44:44.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.897+0000 7fdfdf7fe640 1 -- 172.21.15.57:0/192370102 wait complete. 2023-10-19T12:44:44.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.898+0000 7fde24ff9640 1 -- 172.21.15.57:0/4126217862 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fde1406af10 msgr2=0x7fde1406d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:44.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.898+0000 7fde24ff9640 1 --2- 172.21.15.57:0/4126217862 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fde1406af10 0x7fde1406d3d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fde300aeb20 tx=0x7fde2c005f90 comp rx=0 tx=0).stop 2023-10-19T12:44:44.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.898+0000 7fde24ff9640 1 -- 172.21.15.57:0/4126217862 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fde300b09e0 msgr2=0x7fde301f0ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:44.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.898+0000 7fde24ff9640 1 --2- 172.21.15.57:0/4126217862 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fde300b09e0 0x7fde301f0ec0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fde20009850 tx=0x7fde2002d970 comp rx=0 tx=0).stop 2023-10-19T12:44:44.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.899+0000 7f05349ed640 1 -- 172.21.15.57:0/175308918 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f05100050c0 con 0x7f0530158eb0 2023-10-19T12:44:44.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.899+0000 7f255f59e640 1 -- 172.21.15.57:0/542508038 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f25600ad1c0 con 0x7f25601654e0 2023-10-19T12:44:44.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.899+0000 7fde24ff9640 1 -- 172.21.15.57:0/4126217862 shutdown_connections 2023-10-19T12:44:44.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.899+0000 7fde24ff9640 1 --2- 172.21.15.57:0/4126217862 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fde1406af10 0x7fde1406d3d0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.899+0000 7fde24ff9640 1 --2- 172.21.15.57:0/4126217862 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde3015c0c0 0x7fde30157cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.899+0000 7fde24ff9640 1 --2- 172.21.15.57:0/4126217862 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fde301411e0 0x7fde30136700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.899+0000 7fde24ff9640 1 --2- 172.21.15.57:0/4126217862 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fde300b09e0 0x7fde301f0ec0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.899+0000 7fde24ff9640 1 -- 172.21.15.57:0/4126217862 >> 172.21.15.57:0/4126217862 conn(0x7fde300ae5d0 msgr2=0x7fde30158f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:44.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.899+0000 7f0517fff640 1 -- 172.21.15.57:0/175308918 <== mon.1 v2:172.21.15.179:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f052807a960 con 0x7f0530158eb0 2023-10-19T12:44:44.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.899+0000 7fde24ff9640 1 -- 172.21.15.57:0/4126217862 shutdown_connections 2023-10-19T12:44:44.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.899+0000 7fde24ff9640 1 -- 172.21.15.57:0/4126217862 wait complete. 2023-10-19T12:44:44.912 INFO:teuthology.orchestra.run.smithi057.stdout:55834574858 2023-10-19T12:44:44.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.899+0000 7f254effd640 1 -- 172.21.15.57:0/542508038 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2558021090 con 0x7f25601654e0 2023-10-19T12:44:44.913 INFO:teuthology.orchestra.run.smithi057.stdout:55834574857 2023-10-19T12:44:44.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.902+0000 7f05349ed640 1 -- 172.21.15.57:0/175308918 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f051c06af10 msgr2=0x7f051c06d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:44.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.902+0000 7f05349ed640 1 --2- 172.21.15.57:0/175308918 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f051c06af10 0x7f051c06d3d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f0520005fd0 tx=0x7f052000b2f0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.902+0000 7f05349ed640 1 -- 172.21.15.57:0/175308918 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0530158eb0 msgr2=0x7f05300c47e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:44.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.902+0000 7f05349ed640 1 --2- 172.21.15.57:0/175308918 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0530158eb0 0x7f05300c47e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f05300705b0 tx=0x7f0528009320 comp rx=0 tx=0).stop 2023-10-19T12:44:44.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.902+0000 7f05349ed640 1 -- 172.21.15.57:0/175308918 shutdown_connections 2023-10-19T12:44:44.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.902+0000 7f05349ed640 1 --2- 172.21.15.57:0/175308918 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f051c06af10 0x7f051c06d3d0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.902+0000 7f05349ed640 1 --2- 172.21.15.57:0/175308918 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0530158eb0 0x7f05300c47e0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.915 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574858 for osd.1 2023-10-19T12:44:44.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.902+0000 7f05349ed640 1 --2- 172.21.15.57:0/175308918 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f05300cd1c0 0x7f0530099e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.902+0000 7f05349ed640 1 --2- 172.21.15.57:0/175308918 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f05300cafa0 0x7f05302011c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.902+0000 7f05349ed640 1 -- 172.21.15.57:0/175308918 >> 172.21.15.57:0/175308918 conn(0x7f05300ce870 msgr2=0x7f05300abab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:44.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.903+0000 7f05349ed640 1 -- 172.21.15.57:0/175308918 shutdown_connections 2023-10-19T12:44:44.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.903+0000 7f05349ed640 1 -- 172.21.15.57:0/175308918 wait complete. 2023-10-19T12:44:44.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.903+0000 7f255f59e640 1 -- 172.21.15.57:0/542508038 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f254006af40 msgr2=0x7f254006d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:44.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.903+0000 7f255f59e640 1 --2- 172.21.15.57:0/542508038 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f254006af40 0x7f254006d400 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f2554001270 tx=0x7f25540014d0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.903+0000 7f255f59e640 1 -- 172.21.15.57:0/542508038 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25601654e0 msgr2=0x7f2560165a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:44.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.903+0000 7f255f59e640 1 --2- 172.21.15.57:0/542508038 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25601654e0 0x7f2560165a70 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f2558010bb0 tx=0x7f255800e6c0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.903+0000 7f255f59e640 1 -- 172.21.15.57:0/542508038 shutdown_connections 2023-10-19T12:44:44.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.903+0000 7f255f59e640 1 --2- 172.21.15.57:0/542508038 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f254006af40 0x7f254006d400 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.903+0000 7f255f59e640 1 --2- 172.21.15.57:0/542508038 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2560165fb0 0x7f2560168980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.903+0000 7f255f59e640 1 --2- 172.21.15.57:0/542508038 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25601654e0 0x7f2560165a70 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.903+0000 7f255f59e640 1 --2- 172.21.15.57:0/542508038 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2560165100 0x7f2560143850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.903+0000 7f255f59e640 1 -- 172.21.15.57:0/542508038 >> 172.21.15.57:0/542508038 conn(0x7f256006a830 msgr2=0x7f256015a640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:44.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.903+0000 7f255f59e640 1 -- 172.21.15.57:0/542508038 shutdown_connections 2023-10-19T12:44:44.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.903+0000 7f255f59e640 1 -- 172.21.15.57:0/542508038 wait complete. 2023-10-19T12:44:44.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.907+0000 7f2731852640 1 -- 172.21.15.57:0/1088338452 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f2724037c70 con 0x7f272404fcb0 2023-10-19T12:44:44.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.908+0000 7f27297fa640 1 -- 172.21.15.57:0/1088338452 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f272c08e090 con 0x7f272404fcb0 2023-10-19T12:44:44.921 INFO:teuthology.orchestra.run.smithi057.stdout:55834574858 2023-10-19T12:44:44.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.911+0000 7f2731852640 1 -- 172.21.15.57:0/1088338452 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f271006af10 msgr2=0x7f271006d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:44.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.911+0000 7f2731852640 1 --2- 172.21.15.57:0/1088338452 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f271006af10 0x7f271006d3d0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f2720035000 tx=0x7f27200046c0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.911+0000 7f2731852640 1 -- 172.21.15.57:0/1088338452 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f272404fcb0 msgr2=0x7f2724036d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:44.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.911+0000 7f2731852640 1 --2- 172.21.15.57:0/1088338452 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f272404fcb0 0x7f2724036d70 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f272c04f370 tx=0x7f272c0679c0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.912+0000 7f2731852640 1 -- 172.21.15.57:0/1088338452 shutdown_connections 2023-10-19T12:44:44.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.912+0000 7f2731852640 1 --2- 172.21.15.57:0/1088338452 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f271006af10 0x7f271006d3d0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.912+0000 7f2731852640 1 --2- 172.21.15.57:0/1088338452 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f27240549a0 0x7f2724174a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.912+0000 7f2731852640 1 --2- 172.21.15.57:0/1088338452 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2724052ab0 0x7f272404b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.912+0000 7f2731852640 1 --2- 172.21.15.57:0/1088338452 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f272404fcb0 0x7f2724036d70 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:44.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.912+0000 7f2731852640 1 -- 172.21.15.57:0/1088338452 >> 172.21.15.57:0/1088338452 conn(0x7f2724054ee0 msgr2=0x7f27240de050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:44.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.912+0000 7f2731852640 1 -- 172.21.15.57:0/1088338452 shutdown_connections 2023-10-19T12:44:44.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:44.912+0000 7f2731852640 1 -- 172.21.15.57:0/1088338452 wait complete. 2023-10-19T12:44:44.925 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574858 for osd.6 2023-10-19T12:44:44.926 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574860 got 55834574859 for osd.0 2023-10-19T12:44:44.927 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574860 got 55834574859 for osd.2 2023-10-19T12:44:44.928 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574858 for osd.7 2023-10-19T12:44:44.928 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574857 for osd.3 2023-10-19T12:44:44.930 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574858 for osd.5 2023-10-19T12:44:45.899 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-10-19T12:44:45.917 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-19T12:44:45.927 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-10-19T12:44:45.927 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-19T12:44:45.928 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-19T12:44:45.928 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-10-19T12:44:45.930 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-10-19T12:44:45.930 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-10-19T12:44:46.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.079+0000 7f3ef085d640 1 -- 172.21.15.57:0/3768538387 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3eec06fdc0 msgr2=0x7f3eec0aa830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:46.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.079+0000 7f3ef085d640 1 --2- 172.21.15.57:0/3768538387 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3eec06fdc0 0x7f3eec0aa830 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f3eec0ce1a0 tx=0x7f3ee402d120 comp rx=0 tx=0).stop 2023-10-19T12:44:46.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.080+0000 7f3ef085d640 1 -- 172.21.15.57:0/3768538387 shutdown_connections 2023-10-19T12:44:46.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.080+0000 7f3ef085d640 1 --2- 172.21.15.57:0/3768538387 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3eec06fdc0 0x7f3eec0aa830 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:46.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.080+0000 7f3ef085d640 1 --2- 172.21.15.57:0/3768538387 >> v2:172.21.15.179:3300/0 conn(0x7f3eec0af4b0 0x7f3eec0aa2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:46.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.080+0000 7f3ef085d640 1 -- 172.21.15.57:0/3768538387 >> 172.21.15.57:0/3768538387 conn(0x7f3eec0b0cd0 msgr2=0x7f3eec0b1f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:46.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.080+0000 7f3ef085d640 1 -- 172.21.15.57:0/3768538387 shutdown_connections 2023-10-19T12:44:46.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.080+0000 7f3ef085d640 1 -- 172.21.15.57:0/3768538387 wait complete. 2023-10-19T12:44:46.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.082+0000 7f3ef085d640 1 Processor -- start 2023-10-19T12:44:46.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.083+0000 7f3ef085d640 1 -- start start 2023-10-19T12:44:46.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.084+0000 7f3ef085d640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3eec06fdc0 0x7f3eec153e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:46.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.084+0000 7f3ef085d640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3eec0af4b0 0x7f3eec1fdd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:46.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.084+0000 7f3ef085d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3eec1573f0 0x7f3eec159ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:46.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.084+0000 7f3eea59c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3eec06fdc0 0x7f3eec153e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:46.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.084+0000 7f3eea59c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3eec06fdc0 0x7f3eec153e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3301/0 says I am v2:172.21.15.57:48186/0 (socket says 172.21.15.57:48186) 2023-10-19T12:44:46.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.084+0000 7f3ee9d9b640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3eec0af4b0 0x7f3eec1fdd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:46.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.084+0000 7f3ee9d9b640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3eec0af4b0 0x7f3eec1fdd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.57:43360/0 (socket says 172.21.15.57:43360) 2023-10-19T12:44:46.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.085+0000 7f3eea59c640 1 -- 172.21.15.57:0/1770060322 learned_addr learned my addr 172.21.15.57:0/1770060322 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:46.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.085+0000 7f3eead9d640 1 --2- 172.21.15.57:0/1770060322 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3eec1573f0 0x7f3eec159ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:46.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.085+0000 7f3ef085d640 1 -- 172.21.15.57:0/1770060322 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eec097e90 con 0x7f3eec1573f0 2023-10-19T12:44:46.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.085+0000 7f3ef085d640 1 -- 172.21.15.57:0/1770060322 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eec154390 con 0x7f3eec0af4b0 2023-10-19T12:44:46.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.085+0000 7f3ef085d640 1 -- 172.21.15.57:0/1770060322 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3eec1fe2c0 con 0x7f3eec06fdc0 2023-10-19T12:44:46.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.086+0000 7f3eead9d640 1 -- 172.21.15.57:0/1770060322 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3eec06fdc0 msgr2=0x7f3eec153e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:46.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.086+0000 7f3eead9d640 1 --2- 172.21.15.57:0/1770060322 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3eec06fdc0 0x7f3eec153e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:46.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.086+0000 7f3eead9d640 1 -- 172.21.15.57:0/1770060322 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3eec0af4b0 msgr2=0x7f3eec1fdd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:46.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.086+0000 7f3eead9d640 1 --2- 172.21.15.57:0/1770060322 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3eec0af4b0 0x7f3eec1fdd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:46.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.086+0000 7f3eead9d640 1 -- 172.21.15.57:0/1770060322 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ee4009900 con 0x7f3eec1573f0 2023-10-19T12:44:46.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.087+0000 7f3eead9d640 1 --2- 172.21.15.57:0/1770060322 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3eec1573f0 0x7f3eec159ae0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f3ee4034000 tx=0x7f3ee4003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:46.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.088+0000 7f3edaffd640 1 -- 172.21.15.57:0/1770060322 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ee4004300 con 0x7f3eec1573f0 2023-10-19T12:44:46.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.088+0000 7f3ef085d640 1 -- 172.21.15.57:0/1770060322 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3eec0997f0 con 0x7f3eec1573f0 2023-10-19T12:44:46.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.088+0000 7f3ef085d640 1 -- 172.21.15.57:0/1770060322 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3eec099db0 con 0x7f3eec1573f0 2023-10-19T12:44:46.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.092+0000 7f3edaffd640 1 -- 172.21.15.57:0/1770060322 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f3ee4004550 con 0x7f3eec1573f0 2023-10-19T12:44:46.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.092+0000 7f3edaffd640 1 -- 172.21.15.57:0/1770060322 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ee4040470 con 0x7f3eec1573f0 2023-10-19T12:44:46.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.092+0000 7f3edaffd640 1 -- 172.21.15.57:0/1770060322 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f3ee403c070 con 0x7f3eec1573f0 2023-10-19T12:44:46.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.093+0000 7f3edaffd640 1 --2- 172.21.15.57:0/1770060322 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f3ecc06af90 0x7f3ecc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:46.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.093+0000 7f3edaffd640 1 -- 172.21.15.57:0/1770060322 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f3ee40acd80 con 0x7f3eec1573f0 2023-10-19T12:44:46.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.094+0000 7f3eea59c640 1 --2- 172.21.15.57:0/1770060322 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f3ecc06af90 0x7f3ecc06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:46.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.095+0000 7f3ef085d640 1 -- 172.21.15.57:0/1770060322 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ed0005250 con 0x7f3eec1573f0 2023-10-19T12:44:46.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.105+0000 7f3edaffd640 1 -- 172.21.15.57:0/1770060322 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3ee403a080 con 0x7f3eec1573f0 2023-10-19T12:44:46.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.110+0000 7f3eea59c640 1 --2- 172.21.15.57:0/1770060322 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f3ecc06af90 0x7f3ecc06d450 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f3ee0000f60 tx=0x7f3ee0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:46.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.132+0000 7f2d51651640 1 -- 172.21.15.57:0/2257166549 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2d44008cb0 msgr2=0x7f2d44007da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:46.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.132+0000 7f2d51651640 1 --2- 172.21.15.57:0/2257166549 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2d44008cb0 0x7f2d44007da0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f2d3c005fd0 tx=0x7f2d3c02beb0 comp rx=0 tx=0).stop 2023-10-19T12:44:46.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.133+0000 7f2d51651640 1 -- 172.21.15.57:0/2257166549 shutdown_connections 2023-10-19T12:44:46.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.133+0000 7f2d51651640 1 --2- 172.21.15.57:0/2257166549 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2d44008cb0 0x7f2d44007da0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:46.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.133+0000 7f2d51651640 1 -- 172.21.15.57:0/2257166549 >> 172.21.15.57:0/2257166549 conn(0x7f2d44020010 msgr2=0x7f2d44056d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:46.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.133+0000 7f2d51651640 1 -- 172.21.15.57:0/2257166549 shutdown_connections 2023-10-19T12:44:46.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.134+0000 7f2d51651640 1 -- 172.21.15.57:0/2257166549 wait complete. 2023-10-19T12:44:46.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.134+0000 7f61fff95640 1 -- 172.21.15.57:0/779608123 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f61f00063d0 msgr2=0x7f61f0006820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:46.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.134+0000 7fd31f7fe640 1 -- 172.21.15.57:0/176780367 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd318034840 con 0x7fd3200cfa20 2023-10-19T12:44:46.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.134+0000 7f61fff95640 1 --2- 172.21.15.57:0/779608123 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f61f00063d0 0x7f61f0006820 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f61f8077310 tx=0x7f61f80a3d60 comp rx=0 tx=0).stop 2023-10-19T12:44:46.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.134+0000 7f61fff95640 1 -- 172.21.15.57:0/779608123 shutdown_connections 2023-10-19T12:44:46.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.134+0000 7f61fff95640 1 --2- 172.21.15.57:0/779608123 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f61f00063d0 0x7f61f0006820 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:46.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.134+0000 7f61fff95640 1 --2- 172.21.15.57:0/779608123 >> v2:172.21.15.57:3300/0 conn(0x7f61f004f290 0x7f61f0005e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:46.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.134+0000 7f61fff95640 1 -- 172.21.15.57:0/779608123 >> 172.21.15.57:0/779608123 conn(0x7f61f0009fb0 msgr2=0x7f61f000a390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:46.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.135+0000 7fd327000640 1 -- 172.21.15.57:0/176780367 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd3200cfa20 msgr2=0x7fd3200cb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:46.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.135+0000 7fd327000640 1 --2- 172.21.15.57:0/176780367 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd3200cfa20 0x7fd3200cb9e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fd318005fd0 tx=0x7fd31802c110 comp rx=0 tx=0).stop 2023-10-19T12:44:46.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.135+0000 7fd327000640 1 -- 172.21.15.57:0/176780367 shutdown_connections 2023-10-19T12:44:46.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.135+0000 7fd327000640 1 --2- 172.21.15.57:0/176780367 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd32015b640 0x7fd3200cbf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:46.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.135+0000 7fd327000640 1 --2- 172.21.15.57:0/176780367 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd3200cfa20 0x7fd3200cb9e0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:46.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.135+0000 7fd327000640 1 -- 172.21.15.57:0/176780367 >> 172.21.15.57:0/176780367 conn(0x7fd32009a030 msgr2=0x7fd3200d0d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:46.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.135+0000 7fd327000640 1 -- 172.21.15.57:0/176780367 shutdown_connections 2023-10-19T12:44:46.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.135+0000 7fd327000640 1 -- 172.21.15.57:0/176780367 wait complete. 2023-10-19T12:44:46.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.135+0000 7fb4960d7640 1 -- 172.21.15.57:0/478933313 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb4900c9d00 msgr2=0x7fb49006b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:46.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.135+0000 7fb4960d7640 1 --2- 172.21.15.57:0/478933313 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb4900c9d00 0x7fb49006b910 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb49006f010 tx=0x7fb48402c8f0 comp rx=0 tx=0).stop 2023-10-19T12:44:46.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.136+0000 7f2d51651640 1 Processor -- start 2023-10-19T12:44:46.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.136+0000 7fb4960d7640 1 -- 172.21.15.57:0/478933313 shutdown_connections 2023-10-19T12:44:46.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.136+0000 7fb4960d7640 1 --2- 172.21.15.57:0/478933313 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb4900c9d00 0x7fb49006b910 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:46.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.136+0000 7fb4960d7640 1 --2- 172.21.15.57:0/478933313 >> v2:172.21.15.57:3300/0 conn(0x7fb490140660 0x7fb4900c97c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:46.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.136+0000 7f61fff95640 1 -- 172.21.15.57:0/779608123 shutdown_connections 2023-10-19T12:44:46.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.136+0000 7fb4960d7640 1 -- 172.21.15.57:0/478933313 >> 172.21.15.57:0/478933313 conn(0x7fb490069930 msgr2=0x7fb490069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:46.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.136+0000 7f2d51651640 1 -- start start 2023-10-19T12:44:46.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.136+0000 7f61fff95640 1 -- 172.21.15.57:0/779608123 wait complete. 2023-10-19T12:44:46.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.136+0000 7fb4960d7640 1 -- 172.21.15.57:0/478933313 shutdown_connections 2023-10-19T12:44:46.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.136+0000 7fb4960d7640 1 -- 172.21.15.57:0/478933313 wait complete. 2023-10-19T12:44:46.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.137+0000 7fd327000640 1 Processor -- start 2023-10-19T12:44:46.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.137+0000 7fd327000640 1 -- start start 2023-10-19T12:44:46.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.138+0000 7f2d51651640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2d44008cb0 0x7f2d440dae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:46.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.138+0000 7f61fff95640 1 Processor -- start 2023-10-19T12:44:46.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.138+0000 7f61fff95640 1 -- start start 2023-10-19T12:44:46.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.138+0000 7f2d4affd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2d44008cb0 0x7f2d440dae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:46.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.138+0000 7f8d3e002640 1 -- 172.21.15.57:0/3154864261 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8d30057d60 msgr2=0x7f8d300de6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:46.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.138+0000 7f8d3e002640 1 --2- 172.21.15.57:0/3154864261 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8d30057d60 0x7f8d300de6f0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f8d380775d0 tx=0x7f8d380a3e20 comp rx=0 tx=0).stop 2023-10-19T12:44:46.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.138+0000 7f8d3e002640 1 -- 172.21.15.57:0/3154864261 shutdown_connections 2023-10-19T12:44:46.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.138+0000 7f8d3e002640 1 --2- 172.21.15.57:0/3154864261 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8d30054210 0x7f8d300dec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:46.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.138+0000 7f8d3e002640 1 --2- 172.21.15.57:0/3154864261 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8d30057d60 0x7f8d300de6f0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:46.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.138+0000 7fb4960d7640 1 Processor -- start 2023-10-19T12:44:46.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.138+0000 7f8d3e002640 1 -- 172.21.15.57:0/3154864261 >> 172.21.15.57:0/3154864261 conn(0x7f8d300559b0 msgr2=0x7f8d30055dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:46.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.138+0000 7f2d4affd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2d44008cb0 0x7f2d440dae90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50954/0 (socket says 172.21.15.57:50954) 2023-10-19T12:44:46.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.138+0000 7f2d4affd640 1 -- 172.21.15.57:0/4281622001 learned_addr learned my addr 172.21.15.57:0/4281622001 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:46.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.138+0000 7f8d3e002640 1 -- 172.21.15.57:0/3154864261 shutdown_connections 2023-10-19T12:44:46.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.138+0000 7f8d3e002640 1 -- 172.21.15.57:0/3154864261 wait complete. 2023-10-19T12:44:46.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.138+0000 7f2d51651640 1 --2- 172.21.15.57:0/4281622001 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2d44053dc0 0x7f2d440de140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:46.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.138+0000 7f2d51651640 1 --2- 172.21.15.57:0/4281622001 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2d44056010 0x7f2d440d6950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:46.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.139+0000 7f2d51651640 1 -- 172.21.15.57:0/4281622001 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d440df7a0 con 0x7f2d44008cb0 2023-10-19T12:44:46.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.139+0000 7f2d51651640 1 -- 172.21.15.57:0/4281622001 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d440deb20 con 0x7f2d44053dc0 2023-10-19T12:44:46.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.139+0000 7f2d4a7fc640 1 --2- 172.21.15.57:0/4281622001 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2d44053dc0 0x7f2d440de140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:46.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.139+0000 7f2d4b7fe640 1 --2- 172.21.15.57:0/4281622001 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2d44056010 0x7f2d440d6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:46.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.139+0000 7f2d51651640 1 -- 172.21.15.57:0/4281622001 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d440dee00 con 0x7f2d44056010 2023-10-19T12:44:46.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.139+0000 7fd327000640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd3200cfa20 0x7fd3200ca510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:46.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.139+0000 7fd327000640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd32015b260 0x7fd32015d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:46.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.139+0000 7fd324d9c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd3200cfa20 0x7fd3200ca510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:46.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.140+0000 7fd324d9c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd3200cfa20 0x7fd3200ca510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.57:43406/0 (socket says 172.21.15.57:43406) 2023-10-19T12:44:46.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.140+0000 7f2d4b7fe640 1 -- 172.21.15.57:0/4281622001 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2d44053dc0 msgr2=0x7f2d440de140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:46.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.140+0000 7f2d4b7fe640 1 --2- 172.21.15.57:0/4281622001 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2d44053dc0 0x7f2d440de140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:46.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.140+0000 7f2d4b7fe640 1 -- 172.21.15.57:0/4281622001 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2d44008cb0 msgr2=0x7f2d440dae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:46.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.140+0000 7f61fff95640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f61f00063d0 0x7f61f00d6cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:46.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.140+0000 7f61fff95640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f61f004f290 0x7f61f00d6650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:46.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.140+0000 7fb4960d7640 1 -- start start 2023-10-19T12:44:46.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.140+0000 7f61fff95640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f61f00e6fb0 0x7f61f00dd250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:46.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.140+0000 7f61fff95640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61f00d6b90 con 0x7f61f00e6fb0 2023-10-19T12:44:46.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.140+0000 7f61fff95640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61f004bd10 con 0x7f61f004f290 2023-10-19T12:44:46.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.140+0000 7f61fff95640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61f004bfd0 con 0x7f61f00063d0 2023-10-19T12:44:46.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.140+0000 7f2d4b7fe640 1 --2- 172.21.15.57:0/4281622001 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2d44008cb0 0x7f2d440dae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:46.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.140+0000 7f2d4b7fe640 1 -- 172.21.15.57:0/4281622001 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d3c009560 con 0x7f2d44056010 2023-10-19T12:44:46.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.140+0000 7fd324d9c640 1 -- 172.21.15.57:0/1403553607 learned_addr learned my addr 172.21.15.57:0/1403553607 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:46.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.140+0000 7fd327000640 1 --2- 172.21.15.57:0/1403553607 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd32015b640 0x7fd32015d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:46.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.140+0000 7f2d4affd640 1 --2- 172.21.15.57:0/4281622001 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2d44008cb0 0x7f2d440dae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T12:44:46.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.140+0000 7f61fe532640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f61f00e6fb0 0x7f61f00dd250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:46.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.140+0000 7f61fe532640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f61f00e6fb0 0x7f61f00dd250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50982/0 (socket says 172.21.15.57:50982) 2023-10-19T12:44:46.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.140+0000 7f61fe532640 1 -- 172.21.15.57:0/830658675 learned_addr learned my addr 172.21.15.57:0/830658675 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:46.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.140+0000 7fd327000640 1 -- 172.21.15.57:0/1403553607 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3201e9430 con 0x7fd32015b260 2023-10-19T12:44:46.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.140+0000 7fd327000640 1 -- 172.21.15.57:0/1403553607 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3201ea720 con 0x7fd3200cfa20 2023-10-19T12:44:46.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.140+0000 7fd327000640 1 -- 172.21.15.57:0/1403553607 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3201e9680 con 0x7fd32015b640 2023-10-19T12:44:46.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.140+0000 7fd32559d640 1 --2- 172.21.15.57:0/1403553607 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd32015b640 0x7fd32015d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:46.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.141+0000 7f8d3e002640 1 Processor -- start 2023-10-19T12:44:46.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.141+0000 7f61fe532640 1 -- 172.21.15.57:0/830658675 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f61f00063d0 msgr2=0x7f61f00d6cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-19T12:44:46.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.141+0000 7f61fe532640 1 --2- 172.21.15.57:0/830658675 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f61f00063d0 0x7f61f00d6cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:46.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.141+0000 7f61fe532640 1 -- 172.21.15.57:0/830658675 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f61f004f290 msgr2=0x7f61f00d6650 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-19T12:44:46.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.141+0000 7f61fd530640 1 --2- 172.21.15.57:0/830658675 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f61f004f290 0x7f61f00d6650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:46.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.141+0000 7fd31ffff640 1 --2- 172.21.15.57:0/1403553607 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd32015b260 0x7fd32015d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:46.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.141+0000 7f61fe532640 1 --2- 172.21.15.57:0/830658675 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f61f004f290 0x7f61f00d6650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:46.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.141+0000 7f61fe532640 1 -- 172.21.15.57:0/830658675 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61f804e960 con 0x7f61f00e6fb0 2023-10-19T12:44:46.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.141+0000 7f8d3e002640 1 -- start start 2023-10-19T12:44:46.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.141+0000 7f2d4b7fe640 1 --2- 172.21.15.57:0/4281622001 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2d44056010 0x7f2d440d6950 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2d4000c390 tx=0x7f2d4000c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:46.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.141+0000 7f61fe532640 1 --2- 172.21.15.57:0/830658675 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f61f00e6fb0 0x7f61f00dd250 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f61f80630a0 tx=0x7f61f80acb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:46.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.141+0000 7fd324d9c640 1 -- 172.21.15.57:0/1403553607 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd32015b640 msgr2=0x7fd32015d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:46.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.141+0000 7fd324d9c640 1 --2- 172.21.15.57:0/1403553607 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd32015b640 0x7fd32015d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:46.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.141+0000 7fd324d9c640 1 -- 172.21.15.57:0/1403553607 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd32015b260 msgr2=0x7fd32015d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:46.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.141+0000 7fd324d9c640 1 --2- 172.21.15.57:0/1403553607 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd32015b260 0x7fd32015d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:46.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.141+0000 7fd324d9c640 1 -- 172.21.15.57:0/1403553607 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd310009490 con 0x7fd3200cfa20 2023-10-19T12:44:46.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.142+0000 7f8d3e002640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8d30053e30 0x7f8d300d7320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:46.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.141+0000 7fd324d9c640 1 --2- 172.21.15.57:0/1403553607 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd3200cfa20 0x7fd3200ca510 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd31400c490 tx=0x7fd31400c890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:46.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.142+0000 7f8d3e002640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8d30054210 0x7f8d30052bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:46.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.143+0000 7fb4960d7640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb4900c9d00 0x7fb490156610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:46.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.143+0000 7f8d3e002640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8d30057d60 0x7f8d30038990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:46.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.141+0000 7f2d337fe640 1 -- 172.21.15.57:0/4281622001 <== mon.2 v2:172.21.15.179:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d400052d0 con 0x7f2d44056010 2023-10-19T12:44:46.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.141+0000 7f2d337fe640 1 -- 172.21.15.57:0/4281622001 <== mon.2 v2:172.21.15.179:3301/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f2d40005430 con 0x7f2d44056010 2023-10-19T12:44:46.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.141+0000 7f2d337fe640 1 -- 172.21.15.57:0/4281622001 <== mon.2 v2:172.21.15.179:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d400118b0 con 0x7f2d44056010 2023-10-19T12:44:46.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.142+0000 7f2d51651640 1 -- 172.21.15.57:0/4281622001 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d440df0e0 con 0x7f2d44056010 2023-10-19T12:44:46.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.142+0000 7f2d51651640 1 -- 172.21.15.57:0/4281622001 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d440db3d0 con 0x7f2d44056010 2023-10-19T12:44:46.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.143+0000 7f8d36ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8d30054210 0x7f8d30052bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:46.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.143+0000 7fb4960d7640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb49013d360 0x7fb49013b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:46.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.143+0000 7f61ee7fc640 1 -- 172.21.15.57:0/830658675 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61f80b5430 con 0x7f61f00e6fb0 2023-10-19T12:44:46.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.143+0000 7f8d36ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8d30054210 0x7f8d30052bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50994/0 (socket says 172.21.15.57:50994) 2023-10-19T12:44:46.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.143+0000 7fb4960d7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb490140660 0x7fb4900a64b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:46.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.143+0000 7fb4960d7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb49013c650 con 0x7fb490140660 2023-10-19T12:44:46.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.143+0000 7f61fff95640 1 -- 172.21.15.57:0/830658675 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61f004c2b0 con 0x7f61f00e6fb0 2023-10-19T12:44:46.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.143+0000 7f8d377fe640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8d30053e30 0x7f8d300d7320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:46.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.143+0000 7f8d377fe640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8d30053e30 0x7f8d300d7320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.57:43432/0 (socket says 172.21.15.57:43432) 2023-10-19T12:44:46.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.143+0000 7f8d3e002640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d3000aa90 con 0x7f8d30054210 2023-10-19T12:44:46.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.143+0000 7f8d3e002640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d30051590 con 0x7f8d30053e30 2023-10-19T12:44:46.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.143+0000 7f8d3e002640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d300517e0 con 0x7f8d30057d60 2023-10-19T12:44:46.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.143+0000 7fb4960d7640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4900acfa0 con 0x7fb49013d360 2023-10-19T12:44:46.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.143+0000 7fb48f7fe640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb4900c9d00 0x7fb490156610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:46.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.143+0000 7fb48f7fe640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb4900c9d00 0x7fb490156610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3301/0 says I am v2:172.21.15.57:48266/0 (socket says 172.21.15.57:48266) 2023-10-19T12:44:46.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.143+0000 7fb48f7fe640 1 -- 172.21.15.57:0/1505010627 learned_addr learned my addr 172.21.15.57:0/1505010627 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:46.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.143+0000 7fb4960d7640 1 -- 172.21.15.57:0/1505010627 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4900adfb0 con 0x7fb4900c9d00 2023-10-19T12:44:46.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.143+0000 7f61fff95640 1 -- 172.21.15.57:0/830658675 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61f00d7390 con 0x7f61f00e6fb0 2023-10-19T12:44:46.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.143+0000 7fd31d7fa640 1 -- 172.21.15.57:0/1403553607 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd314011850 con 0x7fd3200cfa20 2023-10-19T12:44:46.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.143+0000 7fd31d7fa640 1 -- 172.21.15.57:0/1403553607 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fd314011aa0 con 0x7fd3200cfa20 2023-10-19T12:44:46.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.143+0000 7fd31d7fa640 1 -- 172.21.15.57:0/1403553607 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd314012420 con 0x7fd3200cfa20 2023-10-19T12:44:46.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.143+0000 7fd327000640 1 -- 172.21.15.57:0/1403553607 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd318009840 con 0x7fd3200cfa20 2023-10-19T12:44:46.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.143+0000 7f8d37fff640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8d30057d60 0x7f8d30038990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:46.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.143+0000 7f8d36ffd640 1 -- 172.21.15.57:0/220851218 learned_addr learned my addr 172.21.15.57:0/220851218 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:46.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.143+0000 7f2d51651640 1 -- 172.21.15.57:0/4281622001 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d440057b0 con 0x7f2d44056010 2023-10-19T12:44:46.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.143+0000 7fd327000640 1 -- 172.21.15.57:0/1403553607 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3201e9c40 con 0x7fd3200cfa20 2023-10-19T12:44:46.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.144+0000 7fb48effd640 1 --2- 172.21.15.57:0/1505010627 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb49013d360 0x7fb49013b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:46.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.143+0000 7f61ee7fc640 1 -- 172.21.15.57:0/830658675 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f61f80b5680 con 0x7f61f00e6fb0 2023-10-19T12:44:46.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.144+0000 7f61ee7fc640 1 -- 172.21.15.57:0/830658675 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61f80b5a90 con 0x7f61f00e6fb0 2023-10-19T12:44:46.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.144+0000 7f8d377fe640 1 -- 172.21.15.57:0/220851218 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8d30057d60 msgr2=0x7f8d30038990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:46.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.144+0000 7f8d377fe640 1 --2- 172.21.15.57:0/220851218 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8d30057d60 0x7f8d30038990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:46.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.144+0000 7f8d377fe640 1 -- 172.21.15.57:0/220851218 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8d30054210 msgr2=0x7f8d30052bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:46.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.144+0000 7f8d377fe640 1 --2- 172.21.15.57:0/220851218 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8d30054210 0x7f8d30052bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:46.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.144+0000 7f8d377fe640 1 -- 172.21.15.57:0/220851218 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d3804e960 con 0x7f8d30053e30 2023-10-19T12:44:46.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.144+0000 7fb48ffff640 1 --2- 172.21.15.57:0/1505010627 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb490140660 0x7fb4900a64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:46.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.144+0000 7fb48f7fe640 1 -- 172.21.15.57:0/1505010627 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb49013d360 msgr2=0x7fb49013b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:46.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.144+0000 7fb48f7fe640 1 --2- 172.21.15.57:0/1505010627 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb49013d360 0x7fb49013b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:46.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.144+0000 7fb48f7fe640 1 -- 172.21.15.57:0/1505010627 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb490140660 msgr2=0x7fb4900a64b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:46.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.144+0000 7fb48f7fe640 1 --2- 172.21.15.57:0/1505010627 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb490140660 0x7fb4900a64b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:46.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.144+0000 7fb48f7fe640 1 -- 172.21.15.57:0/1505010627 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb488009840 con 0x7fb4900c9d00 2023-10-19T12:44:46.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.144+0000 7fb48ffff640 1 --2- 172.21.15.57:0/1505010627 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb490140660 0x7fb4900a64b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:44:46.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.145+0000 7fb48f7fe640 1 --2- 172.21.15.57:0/1505010627 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb4900c9d00 0x7fb490156610 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb48000c7f0 tx=0x7fb48000ccb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:46.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.145+0000 7fd327000640 1 -- 172.21.15.57:0/1403553607 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3200675f0 con 0x7fd3200cfa20 2023-10-19T12:44:46.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.146+0000 7fde6e7fc640 1 -- 172.21.15.57:0/3296213895 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde6800b5d0 con 0x7fde70140660 2023-10-19T12:44:46.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.145+0000 7f8d377fe640 1 --2- 172.21.15.57:0/220851218 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8d30053e30 0x7f8d300d7320 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f8d2c000f60 tx=0x7f8d2c00c890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:46.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.145+0000 7f8d23fff640 1 -- 172.21.15.57:0/220851218 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d2c0052d0 con 0x7f8d30053e30 2023-10-19T12:44:46.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.145+0000 7f8d23fff640 1 -- 172.21.15.57:0/220851218 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f8d2c0035f0 con 0x7f8d30053e30 2023-10-19T12:44:46.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.145+0000 7f8d23fff640 1 -- 172.21.15.57:0/220851218 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d2c003a20 con 0x7f8d30053e30 2023-10-19T12:44:46.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.146+0000 7f8d3e002640 1 -- 172.21.15.57:0/220851218 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d30051ac0 con 0x7f8d30053e30 2023-10-19T12:44:46.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.146+0000 7f8d3e002640 1 -- 172.21.15.57:0/220851218 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d30052080 con 0x7f8d30053e30 2023-10-19T12:44:46.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.146+0000 7fde75e90640 1 -- 172.21.15.57:0/3296213895 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fde70140660 msgr2=0x7fde7015a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:46.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.146+0000 7fde75e90640 1 --2- 172.21.15.57:0/3296213895 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fde70140660 0x7fde7015a9f0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fde68005fd0 tx=0x7fde68030050 comp rx=0 tx=0).stop 2023-10-19T12:44:46.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.146+0000 7fde75e90640 1 -- 172.21.15.57:0/3296213895 shutdown_connections 2023-10-19T12:44:46.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.146+0000 7fde75e90640 1 --2- 172.21.15.57:0/3296213895 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fde70140660 0x7fde7015a9f0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:46.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.146+0000 7fde75e90640 1 --2- 172.21.15.57:0/3296213895 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fde7006d170 0x7fde7013d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:46.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.146+0000 7fde75e90640 1 -- 172.21.15.57:0/3296213895 >> 172.21.15.57:0/3296213895 conn(0x7fde70069930 msgr2=0x7fde70069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:46.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.145+0000 7f61dbfff640 1 -- 172.21.15.57:0/830658675 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61f004c7d0 con 0x7f61f00e6fb0 2023-10-19T12:44:46.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.146+0000 7f61ee7fc640 1 -- 172.21.15.57:0/830658675 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f61f80b3040 con 0x7f61f00e6fb0 2023-10-19T12:44:46.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.146+0000 7f61ee7fc640 1 --2- 172.21.15.57:0/830658675 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f61dc06af10 0x7f61dc06d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:46.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.146+0000 7f2d337fe640 1 -- 172.21.15.57:0/4281622001 <== mon.2 v2:172.21.15.179:3301/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f2d40011a10 con 0x7f2d44056010 2023-10-19T12:44:46.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.146+0000 7f61ee7fc640 1 -- 172.21.15.57:0/830658675 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f61f8123cb0 con 0x7f61f00e6fb0 2023-10-19T12:44:46.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.147+0000 7f61fdd31640 1 --2- 172.21.15.57:0/830658675 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f61dc06af10 0x7f61dc06d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:46.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.147+0000 7f2d337fe640 1 --2- 172.21.15.57:0/4281622001 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f2d3806af10 0x7f2d3806d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:46.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.147+0000 7f2d337fe640 1 -- 172.21.15.57:0/4281622001 <== mon.2 v2:172.21.15.179:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f2d40089290 con 0x7f2d44056010 2023-10-19T12:44:46.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.147+0000 7fde75e90640 1 -- 172.21.15.57:0/3296213895 shutdown_connections 2023-10-19T12:44:46.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.147+0000 7fde75e90640 1 -- 172.21.15.57:0/3296213895 wait complete. 2023-10-19T12:44:46.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.147+0000 7f2d4affd640 1 --2- 172.21.15.57:0/4281622001 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f2d3806af10 0x7f2d3806d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:46.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.147+0000 7f61fdd31640 1 --2- 172.21.15.57:0/830658675 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f61dc06af10 0x7f61dc06d3d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f61f00def80 tx=0x7f61f4007a30 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:46.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.147+0000 7f8d3e002640 1 -- 172.21.15.57:0/220851218 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d30052270 con 0x7f8d30053e30 2023-10-19T12:44:46.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.148+0000 7f2d4affd640 1 --2- 172.21.15.57:0/4281622001 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f2d3806af10 0x7f2d3806d3d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f2d4c0775d0 tx=0x7f2d4c0681c0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:46.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.148+0000 7fd31d7fa640 1 -- 172.21.15.57:0/1403553607 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fd314012580 con 0x7fd3200cfa20 2023-10-19T12:44:46.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.148+0000 7fd31d7fa640 1 --2- 172.21.15.57:0/1403553607 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fd30c06ae90 0x7fd30c06d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:46.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.148+0000 7fde75e90640 1 Processor -- start 2023-10-19T12:44:46.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.148+0000 7fd31d7fa640 1 -- 172.21.15.57:0/1403553607 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fd3140898a0 con 0x7fd3200cfa20 2023-10-19T12:44:46.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.148+0000 7f8d23fff640 1 -- 172.21.15.57:0/220851218 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f8d2c003b80 con 0x7f8d30053e30 2023-10-19T12:44:46.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.149+0000 7fd31ffff640 1 --2- 172.21.15.57:0/1403553607 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fd30c06ae90 0x7fd30c06d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:46.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.148+0000 7fde75e90640 1 -- start start 2023-10-19T12:44:46.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.149+0000 7f8d23fff640 1 --2- 172.21.15.57:0/220851218 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f8d1c06ae40 0x7f8d1c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:46.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.149+0000 7f8d23fff640 1 -- 172.21.15.57:0/220851218 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f8d2c089030 con 0x7f8d30053e30 2023-10-19T12:44:46.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.149+0000 7f8d36ffd640 1 --2- 172.21.15.57:0/220851218 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f8d1c06ae40 0x7f8d1c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:46.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.149+0000 7fde75e90640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde7006d170 0x7fde701ed2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:46.201 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.149+0000 7fde75e90640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fde700ca870 0x7fde70139320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:46.201 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.149+0000 7fde75e90640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fde70140660 0x7fde700ae8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:46.201 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.149+0000 7fde75e90640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde70139860 con 0x7fde7006d170 2023-10-19T12:44:46.201 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.149+0000 7fde75e90640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde700ac630 con 0x7fde700ca870 2023-10-19T12:44:46.201 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.149+0000 7fde75e90640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde700ac910 con 0x7fde70140660 2023-10-19T12:44:46.202 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.149+0000 7fd31ffff640 1 --2- 172.21.15.57:0/1403553607 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fd30c06ae90 0x7fd30c06d350 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fd318035000 tx=0x7fd31802e040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:46.202 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.149+0000 7fde6f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde7006d170 0x7fde701ed2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:46.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.149+0000 7fde6f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde7006d170 0x7fde701ed2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51022/0 (socket says 172.21.15.57:51022) 2023-10-19T12:44:46.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.149+0000 7fde6f7fe640 1 -- 172.21.15.57:0/2384518273 learned_addr learned my addr 172.21.15.57:0/2384518273 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:46.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.149+0000 7fde6ffff640 1 --2- 172.21.15.57:0/2384518273 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fde70140660 0x7fde700ae8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:46.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.150+0000 7fde6effd640 1 --2- 172.21.15.57:0/2384518273 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fde700ca870 0x7fde70139320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:46.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.150+0000 7fb477fff640 1 -- 172.21.15.57:0/1505010627 <== mon.2 v2:172.21.15.179:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb480005480 con 0x7fb4900c9d00 2023-10-19T12:44:46.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.150+0000 7f8d36ffd640 1 --2- 172.21.15.57:0/220851218 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f8d1c06ae40 0x7f8d1c06d300 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f8d38067d70 tx=0x7f8d380a6040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:46.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.150+0000 7f2d337fe640 1 -- 172.21.15.57:0/4281622001 <== mon.2 v2:172.21.15.179:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2d40054170 con 0x7f2d44056010 2023-10-19T12:44:46.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.150+0000 7fb477fff640 1 -- 172.21.15.57:0/1505010627 <== mon.2 v2:172.21.15.179:3301/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fb4800035f0 con 0x7fb4900c9d00 2023-10-19T12:44:46.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.150+0000 7fb4960d7640 1 -- 172.21.15.57:0/1505010627 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4840071c0 con 0x7fb4900c9d00 2023-10-19T12:44:46.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.150+0000 7fb4960d7640 1 -- 172.21.15.57:0/1505010627 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4900ad4d0 con 0x7fb4900c9d00 2023-10-19T12:44:46.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.150+0000 7fb477fff640 1 -- 172.21.15.57:0/1505010627 <== mon.2 v2:172.21.15.179:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb480003a20 con 0x7fb4900c9d00 2023-10-19T12:44:46.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.150+0000 7fde6f7fe640 1 -- 172.21.15.57:0/2384518273 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fde70140660 msgr2=0x7fde700ae8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:46.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.150+0000 7fde6f7fe640 1 --2- 172.21.15.57:0/2384518273 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fde70140660 0x7fde700ae8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:46.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.150+0000 7fde6f7fe640 1 -- 172.21.15.57:0/2384518273 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fde700ca870 msgr2=0x7fde70139320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:46.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.150+0000 7fde6f7fe640 1 --2- 172.21.15.57:0/2384518273 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fde700ca870 0x7fde70139320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:46.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.150+0000 7fde6f7fe640 1 -- 172.21.15.57:0/2384518273 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde6800dc50 con 0x7fde7006d170 2023-10-19T12:44:46.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.150+0000 7fde6f7fe640 1 --2- 172.21.15.57:0/2384518273 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde7006d170 0x7fde701ed2d0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fde6000be40 tx=0x7fde6000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:46.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.151+0000 7fb4960d7640 1 -- 172.21.15.57:0/1505010627 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb470005250 con 0x7fb4900c9d00 2023-10-19T12:44:46.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.152+0000 7fb477fff640 1 -- 172.21.15.57:0/1505010627 <== mon.2 v2:172.21.15.179:3301/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fb480003b80 con 0x7fb4900c9d00 2023-10-19T12:44:46.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.152+0000 7fb477fff640 1 --2- 172.21.15.57:0/1505010627 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fb47c06ae40 0x7fb47c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:46.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.152+0000 7fb477fff640 1 -- 172.21.15.57:0/1505010627 <== mon.2 v2:172.21.15.179:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fb480089190 con 0x7fb4900c9d00 2023-10-19T12:44:46.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.152+0000 7fb48effd640 1 --2- 172.21.15.57:0/1505010627 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fb47c06ae40 0x7fb47c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:46.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.153+0000 7fb48effd640 1 --2- 172.21.15.57:0/1505010627 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fb47c06ae40 0x7fb47c06d300 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb49006bc60 tx=0x7fb488009500 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:46.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.153+0000 7fde57fff640 1 -- 172.21.15.57:0/2384518273 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde6001e640 con 0x7fde7006d170 2023-10-19T12:44:46.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.153+0000 7fde75e90640 1 -- 172.21.15.57:0/2384518273 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde7015b0a0 con 0x7fde7006d170 2023-10-19T12:44:46.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.153+0000 7f61ee7fc640 1 -- 172.21.15.57:0/830658675 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f61f004c7d0 con 0x7f61f00e6fb0 2023-10-19T12:44:46.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.153+0000 7fde75e90640 1 -- 172.21.15.57:0/2384518273 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde700aced0 con 0x7fde7006d170 2023-10-19T12:44:46.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.153+0000 7fde57fff640 1 -- 172.21.15.57:0/2384518273 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fde6001e7a0 con 0x7fde7006d170 2023-10-19T12:44:46.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.153+0000 7fde57fff640 1 -- 172.21.15.57:0/2384518273 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde6001eb30 con 0x7fde7006d170 2023-10-19T12:44:46.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.153+0000 7fd31d7fa640 1 -- 172.21.15.57:0/1403553607 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd3140546d0 con 0x7fd3200cfa20 2023-10-19T12:44:46.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.154+0000 7fde75e90640 1 -- 172.21.15.57:0/2384518273 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde50005250 con 0x7fde7006d170 2023-10-19T12:44:46.216 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.154+0000 7fc3771de640 1 -- 172.21.15.57:0/2048252412 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc3680531c0 msgr2=0x7fc368050040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:46.216 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.154+0000 7fc3771de640 1 --2- 172.21.15.57:0/2048252412 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc3680531c0 0x7fc368050040 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fc36800c140 tx=0x7fc36002bd90 comp rx=0 tx=0).stop 2023-10-19T12:44:46.216 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.154+0000 7fc3771de640 1 -- 172.21.15.57:0/2048252412 shutdown_connections 2023-10-19T12:44:46.216 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.154+0000 7fc3771de640 1 --2- 172.21.15.57:0/2048252412 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc3680531c0 0x7fc368050040 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:46.217 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.154+0000 7fc3771de640 1 --2- 172.21.15.57:0/2048252412 >> v2:172.21.15.179:3300/0 conn(0x7fc3680dffe0 0x7fc36803a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:46.217 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.154+0000 7fc3771de640 1 -- 172.21.15.57:0/2048252412 >> 172.21.15.57:0/2048252412 conn(0x7fc368054ee0 msgr2=0x7fc3680552f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:46.217 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.155+0000 7fc3771de640 1 -- 172.21.15.57:0/2048252412 shutdown_connections 2023-10-19T12:44:46.217 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.155+0000 7fc3771de640 1 -- 172.21.15.57:0/2048252412 wait complete. 2023-10-19T12:44:46.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.155+0000 7fde57fff640 1 -- 172.21.15.57:0/2384518273 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fde6001ec90 con 0x7fde7006d170 2023-10-19T12:44:46.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.156+0000 7fde57fff640 1 --2- 172.21.15.57:0/2384518273 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fde5c06ae40 0x7fde5c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:46.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.156+0000 7fde57fff640 1 -- 172.21.15.57:0/2384518273 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fde60092d20 con 0x7fde7006d170 2023-10-19T12:44:46.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.156+0000 7fde6effd640 1 --2- 172.21.15.57:0/2384518273 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fde5c06ae40 0x7fde5c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:46.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.157+0000 7fc3771de640 1 Processor -- start 2023-10-19T12:44:46.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.157+0000 7fc3771de640 1 -- start start 2023-10-19T12:44:46.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.157+0000 7fde6effd640 1 --2- 172.21.15.57:0/2384518273 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fde5c06ae40 0x7fde5c06d300 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fde68039000 tx=0x7fde68032040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:46.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.157+0000 7fb477fff640 1 -- 172.21.15.57:0/1505010627 <== mon.2 v2:172.21.15.179:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb480057a70 con 0x7fb4900c9d00 2023-10-19T12:44:46.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.158+0000 7fc3771de640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc368039c90 0x7fc3680df470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:46.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.158+0000 7fc3771de640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc3680531c0 0x7fc3680e0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:46.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.159+0000 7fc3759db640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc3680531c0 0x7fc3680e0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:46.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.159+0000 7fc3759db640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc3680531c0 0x7fc3680e0680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51034/0 (socket says 172.21.15.57:51034) 2023-10-19T12:44:46.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.159+0000 7fc3759db640 1 -- 172.21.15.57:0/2565820549 learned_addr learned my addr 172.21.15.57:0/2565820549 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:46.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.159+0000 7fc3771de640 1 --2- 172.21.15.57:0/2565820549 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc3680dffe0 0x7fc3680e0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:46.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.159+0000 7fc3771de640 1 -- 172.21.15.57:0/2565820549 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc36804c3c0 con 0x7fc3680531c0 2023-10-19T12:44:46.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.159+0000 7fc3771de640 1 -- 172.21.15.57:0/2565820549 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc36804c050 con 0x7fc3680dffe0 2023-10-19T12:44:46.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.160+0000 7fc3769dd640 1 --2- 172.21.15.57:0/2565820549 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc3680dffe0 0x7fc3680e0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:46.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.160+0000 7fc3771de640 1 -- 172.21.15.57:0/2565820549 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3680e2660 con 0x7fc368039c90 2023-10-19T12:44:46.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.160+0000 7fc3759db640 1 -- 172.21.15.57:0/2565820549 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc368039c90 msgr2=0x7fc3680df470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T12:44:46.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.160+0000 7fc3759db640 1 --2- 172.21.15.57:0/2565820549 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc368039c90 0x7fc3680df470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:46.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.160+0000 7fc3759db640 1 -- 172.21.15.57:0/2565820549 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc3680dffe0 msgr2=0x7fc3680e0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:46.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.160+0000 7fc3759db640 1 --2- 172.21.15.57:0/2565820549 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc3680dffe0 0x7fc3680e0bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:46.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.160+0000 7fc3759db640 1 -- 172.21.15.57:0/2565820549 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc360009560 con 0x7fc3680531c0 2023-10-19T12:44:46.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.161+0000 7fc3759db640 1 --2- 172.21.15.57:0/2565820549 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc3680531c0 0x7fc3680e0680 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fc36c0019d0 tx=0x7fc36c00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:46.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.157+0000 7f8d23fff640 1 -- 172.21.15.57:0/220851218 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8d2c0bbca0 con 0x7f8d30053e30 2023-10-19T12:44:46.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.164+0000 7fde57fff640 1 -- 172.21.15.57:0/2384518273 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fde6005db90 con 0x7fde7006d170 2023-10-19T12:44:46.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.164+0000 7fc366ffd640 1 -- 172.21.15.57:0/2565820549 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc36c0052d0 con 0x7fc3680531c0 2023-10-19T12:44:46.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.164+0000 7fc366ffd640 1 -- 172.21.15.57:0/2565820549 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fc36c00d070 con 0x7fc3680531c0 2023-10-19T12:44:46.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.164+0000 7fc366ffd640 1 -- 172.21.15.57:0/2565820549 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc36c010360 con 0x7fc3680531c0 2023-10-19T12:44:46.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.164+0000 7fc3771de640 1 -- 172.21.15.57:0/2565820549 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3680e2940 con 0x7fc3680531c0 2023-10-19T12:44:46.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.164+0000 7fc3771de640 1 -- 172.21.15.57:0/2565820549 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3680df9b0 con 0x7fc3680531c0 2023-10-19T12:44:46.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.168+0000 7fc366ffd640 1 -- 172.21.15.57:0/2565820549 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fc36c00a890 con 0x7fc3680531c0 2023-10-19T12:44:46.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.168+0000 7fc366ffd640 1 --2- 172.21.15.57:0/2565820549 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fc35406af10 0x7fc35406d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:46.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.168+0000 7fc366ffd640 1 -- 172.21.15.57:0/2565820549 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fc36c0915f0 con 0x7fc3680531c0 2023-10-19T12:44:46.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.169+0000 7fc3771de640 1 -- 172.21.15.57:0/2565820549 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc358005250 con 0x7fc3680531c0 2023-10-19T12:44:46.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.170+0000 7fc3761dc640 1 --2- 172.21.15.57:0/2565820549 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fc35406af10 0x7fc35406d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:46.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.178+0000 7fc366ffd640 1 -- 172.21.15.57:0/2565820549 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc36c05c420 con 0x7fc3680531c0 2023-10-19T12:44:46.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.186+0000 7fc3761dc640 1 --2- 172.21.15.57:0/2565820549 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fc35406af10 0x7fc35406d3d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fc3700775d0 tx=0x7fc370067270 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:46.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.512+0000 7f3ef085d640 1 -- 172.21.15.57:0/1770060322 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f3ed00050c0 con 0x7f3eec1573f0 2023-10-19T12:44:46.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.513+0000 7f3edaffd640 1 -- 172.21.15.57:0/1770060322 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3ee4077d80 con 0x7f3eec1573f0 2023-10-19T12:44:46.516 INFO:teuthology.orchestra.run.smithi057.stdout:55834574859 2023-10-19T12:44:46.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.516+0000 7f3ed8ff9640 1 -- 172.21.15.57:0/1770060322 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f3ecc06af90 msgr2=0x7f3ecc06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:46.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.516+0000 7f3ed8ff9640 1 --2- 172.21.15.57:0/1770060322 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f3ecc06af90 0x7f3ecc06d450 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f3ee0000f60 tx=0x7f3ee0009040 comp rx=0 tx=0).stop 2023-10-19T12:44:46.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.516+0000 7f3ed8ff9640 1 -- 172.21.15.57:0/1770060322 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3eec1573f0 msgr2=0x7f3eec159ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:46.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.516+0000 7f3ed8ff9640 1 --2- 172.21.15.57:0/1770060322 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3eec1573f0 0x7f3eec159ae0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f3ee4034000 tx=0x7f3ee4003f10 comp rx=0 tx=0).stop 2023-10-19T12:44:46.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.517+0000 7f3ed8ff9640 1 -- 172.21.15.57:0/1770060322 shutdown_connections 2023-10-19T12:44:46.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.517+0000 7f3ed8ff9640 1 --2- 172.21.15.57:0/1770060322 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f3ecc06af90 0x7f3ecc06d450 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:46.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.517+0000 7f3ed8ff9640 1 --2- 172.21.15.57:0/1770060322 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3eec1573f0 0x7f3eec159ae0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:46.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.517+0000 7f3ed8ff9640 1 --2- 172.21.15.57:0/1770060322 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3eec0af4b0 0x7f3eec1fdd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:46.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.517+0000 7f3ed8ff9640 1 --2- 172.21.15.57:0/1770060322 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3eec06fdc0 0x7f3eec153e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:46.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.517+0000 7f3ed8ff9640 1 -- 172.21.15.57:0/1770060322 >> 172.21.15.57:0/1770060322 conn(0x7f3eec0b0cd0 msgr2=0x7f3eec152190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:46.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.517+0000 7f3ed8ff9640 1 -- 172.21.15.57:0/1770060322 shutdown_connections 2023-10-19T12:44:46.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.517+0000 7f3ed8ff9640 1 -- 172.21.15.57:0/1770060322 wait complete. 2023-10-19T12:44:46.537 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574859 for osd.4 2023-10-19T12:44:46.537 DEBUG:teuthology.parallel:result is None 2023-10-19T12:44:46.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.544+0000 7f2d51651640 1 -- 172.21.15.57:0/4281622001 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f2d440def40 con 0x7f2d44056010 2023-10-19T12:44:46.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.545+0000 7f2d337fe640 1 -- 172.21.15.57:0/4281622001 <== mon.2 v2:172.21.15.179:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2d440def40 con 0x7f2d44056010 2023-10-19T12:44:46.548 INFO:teuthology.orchestra.run.smithi057.stdout:55834574858 2023-10-19T12:44:46.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.547+0000 7f2d51651640 1 -- 172.21.15.57:0/4281622001 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f2d3806af10 msgr2=0x7f2d3806d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:46.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.547+0000 7f2d51651640 1 --2- 172.21.15.57:0/4281622001 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f2d3806af10 0x7f2d3806d3d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f2d4c0775d0 tx=0x7f2d4c0681c0 comp rx=0 tx=0).stop 2023-10-19T12:44:46.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.547+0000 7f2d51651640 1 -- 172.21.15.57:0/4281622001 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2d44056010 msgr2=0x7f2d440d6950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:46.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.547+0000 7f2d51651640 1 --2- 172.21.15.57:0/4281622001 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2d44056010 0x7f2d440d6950 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2d4000c390 tx=0x7f2d4000c850 comp rx=0 tx=0).stop 2023-10-19T12:44:46.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.547+0000 7f2d51651640 1 -- 172.21.15.57:0/4281622001 shutdown_connections 2023-10-19T12:44:46.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.547+0000 7f2d51651640 1 --2- 172.21.15.57:0/4281622001 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f2d3806af10 0x7f2d3806d3d0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:46.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.547+0000 7f2d51651640 1 --2- 172.21.15.57:0/4281622001 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2d44056010 0x7f2d440d6950 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:46.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.547+0000 7f2d51651640 1 --2- 172.21.15.57:0/4281622001 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2d44053dc0 0x7f2d440de140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:46.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.547+0000 7f2d51651640 1 --2- 172.21.15.57:0/4281622001 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2d44008cb0 0x7f2d440dae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:46.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.547+0000 7f2d51651640 1 -- 172.21.15.57:0/4281622001 >> 172.21.15.57:0/4281622001 conn(0x7f2d44020010 msgr2=0x7f2d440e61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:46.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.547+0000 7f2d51651640 1 -- 172.21.15.57:0/4281622001 shutdown_connections 2023-10-19T12:44:46.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.547+0000 7f2d51651640 1 -- 172.21.15.57:0/4281622001 wait complete. 2023-10-19T12:44:46.561 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574858 for osd.3 2023-10-19T12:44:46.562 DEBUG:teuthology.parallel:result is None 2023-10-19T12:44:46.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.566+0000 7fd327000640 1 -- 172.21.15.57:0/1403553607 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fd32006b650 con 0x7fd3200cfa20 2023-10-19T12:44:46.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.566+0000 7fd31d7fa640 1 -- 172.21.15.57:0/1403553607 <== mon.1 v2:172.21.15.179:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd314011ed0 con 0x7fd3200cfa20 2023-10-19T12:44:46.570 INFO:teuthology.orchestra.run.smithi057.stdout:55834574859 2023-10-19T12:44:46.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.567+0000 7f8d3e002640 1 -- 172.21.15.57:0/220851218 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f8d300057b0 con 0x7f8d30053e30 2023-10-19T12:44:46.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.567+0000 7fb4960d7640 1 -- 172.21.15.57:0/1505010627 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fb4700050c0 con 0x7fb4900c9d00 2023-10-19T12:44:46.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.567+0000 7f8d23fff640 1 -- 172.21.15.57:0/220851218 <== mon.1 v2:172.21.15.179:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8d2c017090 con 0x7f8d30053e30 2023-10-19T12:44:46.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.567+0000 7fb477fff640 1 -- 172.21.15.57:0/1505010627 <== mon.2 v2:172.21.15.179:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb480057600 con 0x7fb4900c9d00 2023-10-19T12:44:46.572 INFO:teuthology.orchestra.run.smithi057.stdout:55834574860 2023-10-19T12:44:46.572 INFO:teuthology.orchestra.run.smithi057.stdout:55834574860 2023-10-19T12:44:46.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.569+0000 7fd327000640 1 -- 172.21.15.57:0/1403553607 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fd30c06ae90 msgr2=0x7fd30c06d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:46.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.569+0000 7fd327000640 1 --2- 172.21.15.57:0/1403553607 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fd30c06ae90 0x7fd30c06d350 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fd318035000 tx=0x7fd31802e040 comp rx=0 tx=0).stop 2023-10-19T12:44:46.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.569+0000 7fd327000640 1 -- 172.21.15.57:0/1403553607 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd3200cfa20 msgr2=0x7fd3200ca510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:46.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.569+0000 7fd327000640 1 --2- 172.21.15.57:0/1403553607 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd3200cfa20 0x7fd3200ca510 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd31400c490 tx=0x7fd31400c890 comp rx=0 tx=0).stop 2023-10-19T12:44:46.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.569+0000 7fd327000640 1 -- 172.21.15.57:0/1403553607 shutdown_connections 2023-10-19T12:44:46.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.569+0000 7fd327000640 1 --2- 172.21.15.57:0/1403553607 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fd30c06ae90 0x7fd30c06d350 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:46.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.569+0000 7fd327000640 1 --2- 172.21.15.57:0/1403553607 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd32015b640 0x7fd32015d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:46.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.569+0000 7fd327000640 1 --2- 172.21.15.57:0/1403553607 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd32015b260 0x7fd32015d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:46.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.569+0000 7fd327000640 1 --2- 172.21.15.57:0/1403553607 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd3200cfa20 0x7fd3200ca510 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:46.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.569+0000 7fd327000640 1 -- 172.21.15.57:0/1403553607 >> 172.21.15.57:0/1403553607 conn(0x7fd32009a030 msgr2=0x7fd32006a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:46.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.569+0000 7fd327000640 1 -- 172.21.15.57:0/1403553607 shutdown_connections 2023-10-19T12:44:46.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.570+0000 7fd327000640 1 -- 172.21.15.57:0/1403553607 wait complete. 2023-10-19T12:44:46.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.570+0000 7f8d3e002640 1 -- 172.21.15.57:0/220851218 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f8d1c06ae40 msgr2=0x7f8d1c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:46.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.570+0000 7f8d3e002640 1 --2- 172.21.15.57:0/220851218 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f8d1c06ae40 0x7f8d1c06d300 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f8d38067d70 tx=0x7f8d380a6040 comp rx=0 tx=0).stop 2023-10-19T12:44:46.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.570+0000 7f8d3e002640 1 -- 172.21.15.57:0/220851218 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8d30053e30 msgr2=0x7f8d300d7320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:46.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.570+0000 7f8d3e002640 1 --2- 172.21.15.57:0/220851218 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8d30053e30 0x7f8d300d7320 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f8d2c000f60 tx=0x7f8d2c00c890 comp rx=0 tx=0).stop 2023-10-19T12:44:46.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.570+0000 7f8d3e002640 1 -- 172.21.15.57:0/220851218 shutdown_connections 2023-10-19T12:44:46.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.570+0000 7f8d3e002640 1 --2- 172.21.15.57:0/220851218 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f8d1c06ae40 0x7f8d1c06d300 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:46.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.570+0000 7f8d3e002640 1 --2- 172.21.15.57:0/220851218 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8d30057d60 0x7f8d30038990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:46.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.570+0000 7f8d3e002640 1 --2- 172.21.15.57:0/220851218 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8d30054210 0x7f8d30052bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:46.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.570+0000 7f8d3e002640 1 --2- 172.21.15.57:0/220851218 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8d30053e30 0x7f8d300d7320 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:46.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.570+0000 7f8d3e002640 1 -- 172.21.15.57:0/220851218 >> 172.21.15.57:0/220851218 conn(0x7f8d300559b0 msgr2=0x7f8d300d95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:46.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.571+0000 7f8d3e002640 1 -- 172.21.15.57:0/220851218 shutdown_connections 2023-10-19T12:44:46.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.571+0000 7f8d3e002640 1 -- 172.21.15.57:0/220851218 wait complete. 2023-10-19T12:44:46.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.571+0000 7fb4960d7640 1 -- 172.21.15.57:0/1505010627 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fb47c06ae40 msgr2=0x7fb47c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:46.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.571+0000 7fb4960d7640 1 --2- 172.21.15.57:0/1505010627 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fb47c06ae40 0x7fb47c06d300 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb49006bc60 tx=0x7fb488009500 comp rx=0 tx=0).stop 2023-10-19T12:44:46.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.571+0000 7fb4960d7640 1 -- 172.21.15.57:0/1505010627 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb4900c9d00 msgr2=0x7fb490156610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:46.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.571+0000 7fb4960d7640 1 --2- 172.21.15.57:0/1505010627 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb4900c9d00 0x7fb490156610 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb48000c7f0 tx=0x7fb48000ccb0 comp rx=0 tx=0).stop 2023-10-19T12:44:46.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.571+0000 7fb4960d7640 1 -- 172.21.15.57:0/1505010627 shutdown_connections 2023-10-19T12:44:46.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.571+0000 7fb4960d7640 1 --2- 172.21.15.57:0/1505010627 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fb47c06ae40 0x7fb47c06d300 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:46.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.571+0000 7fb4960d7640 1 --2- 172.21.15.57:0/1505010627 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb490140660 0x7fb4900a64b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:46.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.571+0000 7fb4960d7640 1 --2- 172.21.15.57:0/1505010627 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb49013d360 0x7fb49013b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:46.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.571+0000 7fb4960d7640 1 --2- 172.21.15.57:0/1505010627 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb4900c9d00 0x7fb490156610 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:46.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.571+0000 7fb4960d7640 1 -- 172.21.15.57:0/1505010627 >> 172.21.15.57:0/1505010627 conn(0x7fb490069930 msgr2=0x7fb49013db20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:46.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.571+0000 7f61dbfff640 1 -- 172.21.15.57:0/830658675 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f61f004c110 con 0x7f61f00e6fb0 2023-10-19T12:44:46.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.571+0000 7fb4960d7640 1 -- 172.21.15.57:0/1505010627 shutdown_connections 2023-10-19T12:44:46.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.571+0000 7fb4960d7640 1 -- 172.21.15.57:0/1505010627 wait complete. 2023-10-19T12:44:46.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.572+0000 7f61ee7fc640 1 -- 172.21.15.57:0/830658675 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f61f80b1020 con 0x7f61f00e6fb0 2023-10-19T12:44:46.585 INFO:teuthology.orchestra.run.smithi057.stdout:55834574859 2023-10-19T12:44:46.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.576+0000 7f61dbfff640 1 -- 172.21.15.57:0/830658675 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f61dc06af10 msgr2=0x7f61dc06d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:46.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.576+0000 7f61dbfff640 1 --2- 172.21.15.57:0/830658675 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f61dc06af10 0x7f61dc06d3d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f61f00def80 tx=0x7f61f4007a30 comp rx=0 tx=0).stop 2023-10-19T12:44:46.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.576+0000 7f61dbfff640 1 -- 172.21.15.57:0/830658675 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f61f00e6fb0 msgr2=0x7f61f00dd250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:46.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.576+0000 7f61dbfff640 1 --2- 172.21.15.57:0/830658675 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f61f00e6fb0 0x7f61f00dd250 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f61f80630a0 tx=0x7f61f80acb20 comp rx=0 tx=0).stop 2023-10-19T12:44:46.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.576+0000 7f61dbfff640 1 -- 172.21.15.57:0/830658675 shutdown_connections 2023-10-19T12:44:46.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.576+0000 7f61dbfff640 1 --2- 172.21.15.57:0/830658675 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f61dc06af10 0x7f61dc06d3d0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:46.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.576+0000 7f61dbfff640 1 --2- 172.21.15.57:0/830658675 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f61f00e6fb0 0x7f61f00dd250 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:46.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.576+0000 7f61dbfff640 1 --2- 172.21.15.57:0/830658675 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f61f004f290 0x7f61f00d6650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:46.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.576+0000 7f61dbfff640 1 --2- 172.21.15.57:0/830658675 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f61f00063d0 0x7f61f00d6cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:46.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.576+0000 7f61dbfff640 1 -- 172.21.15.57:0/830658675 >> 172.21.15.57:0/830658675 conn(0x7f61f0009fb0 msgr2=0x7f61f000a420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:46.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.577+0000 7f61dbfff640 1 -- 172.21.15.57:0/830658675 shutdown_connections 2023-10-19T12:44:46.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.577+0000 7f61dbfff640 1 -- 172.21.15.57:0/830658675 wait complete. 2023-10-19T12:44:46.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.578+0000 7fde75e90640 1 -- 172.21.15.57:0/2384518273 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fde500050c0 con 0x7fde7006d170 2023-10-19T12:44:46.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.579+0000 7fde57fff640 1 -- 172.21.15.57:0/2384518273 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fde60023020 con 0x7fde7006d170 2023-10-19T12:44:46.593 INFO:teuthology.orchestra.run.smithi057.stdout:55834574859 2023-10-19T12:44:46.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.582+0000 7fde75e90640 1 -- 172.21.15.57:0/2384518273 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fde5c06ae40 msgr2=0x7fde5c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:46.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.582+0000 7fde75e90640 1 --2- 172.21.15.57:0/2384518273 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fde5c06ae40 0x7fde5c06d300 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fde68039000 tx=0x7fde68032040 comp rx=0 tx=0).stop 2023-10-19T12:44:46.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.582+0000 7fde75e90640 1 -- 172.21.15.57:0/2384518273 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde7006d170 msgr2=0x7fde701ed2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:46.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.582+0000 7fde75e90640 1 --2- 172.21.15.57:0/2384518273 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde7006d170 0x7fde701ed2d0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fde6000be40 tx=0x7fde6000a620 comp rx=0 tx=0).stop 2023-10-19T12:44:46.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.582+0000 7fde75e90640 1 -- 172.21.15.57:0/2384518273 shutdown_connections 2023-10-19T12:44:46.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.582+0000 7fde75e90640 1 --2- 172.21.15.57:0/2384518273 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fde5c06ae40 0x7fde5c06d300 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:46.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.582+0000 7fde75e90640 1 --2- 172.21.15.57:0/2384518273 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fde70140660 0x7fde700ae8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:46.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.582+0000 7fde75e90640 1 --2- 172.21.15.57:0/2384518273 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fde700ca870 0x7fde70139320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:46.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.582+0000 7fde75e90640 1 --2- 172.21.15.57:0/2384518273 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde7006d170 0x7fde701ed2d0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:46.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.582+0000 7fde75e90640 1 -- 172.21.15.57:0/2384518273 >> 172.21.15.57:0/2384518273 conn(0x7fde70069930 msgr2=0x7fde70065520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:46.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.582+0000 7fde75e90640 1 -- 172.21.15.57:0/2384518273 shutdown_connections 2023-10-19T12:44:46.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.583+0000 7fde75e90640 1 -- 172.21.15.57:0/2384518273 wait complete. 2023-10-19T12:44:46.599 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574859 for osd.6 2023-10-19T12:44:46.600 DEBUG:teuthology.parallel:result is None 2023-10-19T12:44:46.601 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574860 got 55834574860 for osd.0 2023-10-19T12:44:46.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.588+0000 7fc3771de640 1 -- 172.21.15.57:0/2565820549 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fc3580050c0 con 0x7fc3680531c0 2023-10-19T12:44:46.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.589+0000 7fc366ffd640 1 -- 172.21.15.57:0/2565820549 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc36c05fa40 con 0x7fc3680531c0 2023-10-19T12:44:46.602 DEBUG:teuthology.parallel:result is None 2023-10-19T12:44:46.603 INFO:teuthology.orchestra.run.smithi057.stdout:55834574859 2023-10-19T12:44:46.604 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574860 got 55834574860 for osd.2 2023-10-19T12:44:46.604 DEBUG:teuthology.parallel:result is None 2023-10-19T12:44:46.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.591+0000 7fc3771de640 1 -- 172.21.15.57:0/2565820549 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fc35406af10 msgr2=0x7fc35406d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:46.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.591+0000 7fc3771de640 1 --2- 172.21.15.57:0/2565820549 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fc35406af10 0x7fc35406d3d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fc3700775d0 tx=0x7fc370067270 comp rx=0 tx=0).stop 2023-10-19T12:44:46.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.591+0000 7fc3771de640 1 -- 172.21.15.57:0/2565820549 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc3680531c0 msgr2=0x7fc3680e0680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:46.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.591+0000 7fc3771de640 1 --2- 172.21.15.57:0/2565820549 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc3680531c0 0x7fc3680e0680 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fc36c0019d0 tx=0x7fc36c00a060 comp rx=0 tx=0).stop 2023-10-19T12:44:46.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.592+0000 7fc3771de640 1 -- 172.21.15.57:0/2565820549 shutdown_connections 2023-10-19T12:44:46.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.592+0000 7fc3771de640 1 --2- 172.21.15.57:0/2565820549 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fc35406af10 0x7fc35406d3d0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:46.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.592+0000 7fc3771de640 1 --2- 172.21.15.57:0/2565820549 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc3680dffe0 0x7fc3680e0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:46.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.592+0000 7fc3771de640 1 --2- 172.21.15.57:0/2565820549 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc3680531c0 0x7fc3680e0680 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:46.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.592+0000 7fc3771de640 1 --2- 172.21.15.57:0/2565820549 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc368039c90 0x7fc3680df470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:46.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.592+0000 7fc3771de640 1 -- 172.21.15.57:0/2565820549 >> 172.21.15.57:0/2565820549 conn(0x7fc368054ee0 msgr2=0x7fc368004fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:46.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.592+0000 7fc3771de640 1 -- 172.21.15.57:0/2565820549 shutdown_connections 2023-10-19T12:44:46.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.592+0000 7fc3771de640 1 -- 172.21.15.57:0/2565820549 wait complete. 2023-10-19T12:44:46.608 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574859 for osd.1 2023-10-19T12:44:46.608 DEBUG:teuthology.parallel:result is None 2023-10-19T12:44:46.609 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574859 for osd.7 2023-10-19T12:44:46.610 DEBUG:teuthology.parallel:result is None 2023-10-19T12:44:46.610 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574859 for osd.5 2023-10-19T12:44:46.610 DEBUG:teuthology.parallel:result is None 2023-10-19T12:44:46.611 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-19T12:44:46.611 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-19T12:44:46.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.751+0000 7fa196e32640 1 -- 172.21.15.57:0/3043327770 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa1900ac870 msgr2=0x7fa19006b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:46.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.751+0000 7fa196e32640 1 --2- 172.21.15.57:0/3043327770 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa1900ac870 0x7fa19006b980 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fa190072e80 tx=0x7fa18002be20 comp rx=0 tx=0).stop 2023-10-19T12:44:46.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.751+0000 7fa196e32640 1 -- 172.21.15.57:0/3043327770 shutdown_connections 2023-10-19T12:44:46.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.751+0000 7fa196e32640 1 --2- 172.21.15.57:0/3043327770 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa1900ac870 0x7fa19006b980 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:46.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.751+0000 7fa196e32640 1 --2- 172.21.15.57:0/3043327770 >> v2:172.21.15.179:3300/0 conn(0x7fa190156790 0x7fa1900cc990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:46.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.751+0000 7fa196e32640 1 -- 172.21.15.57:0/3043327770 >> 172.21.15.57:0/3043327770 conn(0x7fa19006fae0 msgr2=0x7fa19006fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:46.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.751+0000 7fa196e32640 1 -- 172.21.15.57:0/3043327770 shutdown_connections 2023-10-19T12:44:46.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.751+0000 7fa196e32640 1 -- 172.21.15.57:0/3043327770 wait complete. 2023-10-19T12:44:46.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.752+0000 7fa196e32640 1 Processor -- start 2023-10-19T12:44:46.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.752+0000 7fa196e32640 1 -- start start 2023-10-19T12:44:46.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.753+0000 7fa196e32640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa19006bec0 0x7fa190143e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:46.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.753+0000 7fa196e32640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa1900ac870 0x7fa1900aa8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:46.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.753+0000 7fa196e32640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa190156790 0x7fa1900aae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:46.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.753+0000 7fa196e32640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1901f4c00 con 0x7fa19006bec0 2023-10-19T12:44:46.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.753+0000 7fa196e32640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1901f4ee0 con 0x7fa1900ac870 2023-10-19T12:44:46.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.753+0000 7fa196e32640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1901f51c0 con 0x7fa190156790 2023-10-19T12:44:46.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.753+0000 7fa194bce640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa19006bec0 0x7fa190143e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:46.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.753+0000 7fa194bce640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa19006bec0 0x7fa190143e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59122/0 (socket says 172.21.15.57:59122) 2023-10-19T12:44:46.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.753+0000 7fa194bce640 1 -- 172.21.15.57:0/1898984220 learned_addr learned my addr 172.21.15.57:0/1898984220 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:46.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.753+0000 7fa187fff640 1 --2- 172.21.15.57:0/1898984220 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa1900ac870 0x7fa1900aa8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:46.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.753+0000 7fa1953cf640 1 --2- 172.21.15.57:0/1898984220 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa190156790 0x7fa1900aae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:46.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.753+0000 7fa194bce640 1 -- 172.21.15.57:0/1898984220 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa190156790 msgr2=0x7fa1900aae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:46.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.753+0000 7fa194bce640 1 --2- 172.21.15.57:0/1898984220 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa190156790 0x7fa1900aae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:46.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.753+0000 7fa194bce640 1 -- 172.21.15.57:0/1898984220 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa1900ac870 msgr2=0x7fa1900aa8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:46.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.753+0000 7fa194bce640 1 --2- 172.21.15.57:0/1898984220 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa1900ac870 0x7fa1900aa8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:46.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.753+0000 7fa194bce640 1 -- 172.21.15.57:0/1898984220 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa180009560 con 0x7fa19006bec0 2023-10-19T12:44:46.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.753+0000 7fa1953cf640 1 --2- 172.21.15.57:0/1898984220 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa190156790 0x7fa1900aae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:44:46.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.753+0000 7fa194bce640 1 --2- 172.21.15.57:0/1898984220 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa19006bec0 0x7fa190143e90 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fa18c00d950 tx=0x7fa18c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:46.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.754+0000 7fa1857fa640 1 -- 172.21.15.57:0/1898984220 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa18c009c30 con 0x7fa19006bec0 2023-10-19T12:44:46.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.754+0000 7fa1857fa640 1 -- 172.21.15.57:0/1898984220 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fa18c009e80 con 0x7fa19006bec0 2023-10-19T12:44:46.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.754+0000 7fa196e32640 1 -- 172.21.15.57:0/1898984220 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1901f54a0 con 0x7fa19006bec0 2023-10-19T12:44:46.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.754+0000 7fa1857fa640 1 -- 172.21.15.57:0/1898984220 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa18c0106c0 con 0x7fa19006bec0 2023-10-19T12:44:46.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.754+0000 7fa196e32640 1 -- 172.21.15.57:0/1898984220 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa19015c370 con 0x7fa19006bec0 2023-10-19T12:44:46.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.755+0000 7fa196e32640 1 -- 172.21.15.57:0/1898984220 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1900690e0 con 0x7fa19006bec0 2023-10-19T12:44:46.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.755+0000 7fa1857fa640 1 -- 172.21.15.57:0/1898984220 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fa18c010820 con 0x7fa19006bec0 2023-10-19T12:44:46.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.755+0000 7fa1857fa640 1 --2- 172.21.15.57:0/1898984220 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fa17c06ae40 0x7fa17c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:46.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.755+0000 7fa1857fa640 1 -- 172.21.15.57:0/1898984220 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fa18c089dc0 con 0x7fa19006bec0 2023-10-19T12:44:46.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.756+0000 7fa187fff640 1 --2- 172.21.15.57:0/1898984220 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fa17c06ae40 0x7fa17c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:46.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.756+0000 7fa187fff640 1 --2- 172.21.15.57:0/1898984220 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fa17c06ae40 0x7fa17c06d300 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fa180035000 tx=0x7fa180004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:46.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.760+0000 7fa1857fa640 1 -- 172.21.15.57:0/1898984220 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa18c054bb0 con 0x7fa19006bec0 2023-10-19T12:44:46.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.930+0000 7fa196e32640 1 -- 172.21.15.57:0/1898984220 --> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa1901f5050 con 0x7fa17c06ae40 2023-10-19T12:44:46.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.939+0000 7fa1857fa640 1 -- 172.21.15.57:0/1898984220 <== mgr.4100 v2:172.21.15.179:6832/2609646285 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+345217 (secure 0 0 0) 0x7fa1901f5050 con 0x7fa17c06ae40 2023-10-19T12:44:46.943 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:44:46.950 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2023-10-19T12:44:46.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.942+0000 7fa196e32640 1 -- 172.21.15.57:0/1898984220 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fa17c06ae40 msgr2=0x7fa17c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:46.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.942+0000 7fa196e32640 1 --2- 172.21.15.57:0/1898984220 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fa17c06ae40 0x7fa17c06d300 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fa180035000 tx=0x7fa180004730 comp rx=0 tx=0).stop 2023-10-19T12:44:46.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.942+0000 7fa196e32640 1 -- 172.21.15.57:0/1898984220 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa19006bec0 msgr2=0x7fa190143e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:46.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.942+0000 7fa196e32640 1 --2- 172.21.15.57:0/1898984220 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa19006bec0 0x7fa190143e90 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fa18c00d950 tx=0x7fa18c00de10 comp rx=0 tx=0).stop 2023-10-19T12:44:46.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.942+0000 7fa196e32640 1 -- 172.21.15.57:0/1898984220 shutdown_connections 2023-10-19T12:44:46.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.942+0000 7fa196e32640 1 --2- 172.21.15.57:0/1898984220 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fa17c06ae40 0x7fa17c06d300 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:46.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.942+0000 7fa196e32640 1 --2- 172.21.15.57:0/1898984220 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa190156790 0x7fa1900aae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:46.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.942+0000 7fa196e32640 1 --2- 172.21.15.57:0/1898984220 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa1900ac870 0x7fa1900aa8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:46.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.942+0000 7fa196e32640 1 --2- 172.21.15.57:0/1898984220 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa19006bec0 0x7fa190143e90 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:46.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.942+0000 7fa196e32640 1 -- 172.21.15.57:0/1898984220 >> 172.21.15.57:0/1898984220 conn(0x7fa19006fae0 msgr2=0x7fa19006c2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:46.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.942+0000 7fa196e32640 1 -- 172.21.15.57:0/1898984220 shutdown_connections 2023-10-19T12:44:46.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:46.942+0000 7fa196e32640 1 -- 172.21.15.57:0/1898984220 wait complete. 2023-10-19T12:44:46.956 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":38,"stamp":"2023-10-19T12:44:45.320998+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595007,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236140,"kb_used_data":3300,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738580,"statfs":{"total":773094113280,"available":772852305920,"internally_reserved":0,"allocated":3379200,"data_stored":2025003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":238276153},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":141,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003664"},"pg_stats":[{"pgid":"2.1f","version":"20'4","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.113796+0000","last_change":"2023-10-19T12:44:18.071366+0000","last_active":"2023-10-19T12:44:23.113796+0000","last_peered":"2023-10-19T12:44:23.113796+0000","last_clean":"2023-10-19T12:44:23.113796+0000","last_became_active":"2023-10-19T12:44:18.071117+0000","last_became_peered":"2023-10-19T12:44:18.071117+0000","last_unstale":"2023-10-19T12:44:23.113796+0000","last_undegraded":"2023-10-19T12:44:23.113796+0000","last_fullsized":"2023-10-19T12:44:23.113796+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:24:31.294273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.107571+0000","last_change":"2023-10-19T12:44:19.080917+0000","last_active":"2023-10-19T12:44:23.107571+0000","last_peered":"2023-10-19T12:44:23.107571+0000","last_clean":"2023-10-19T12:44:23.107571+0000","last_became_active":"2023-10-19T12:44:19.080408+0000","last_became_peered":"2023-10-19T12:44:19.080408+0000","last_unstale":"2023-10-19T12:44:23.107571+0000","last_undegraded":"2023-10-19T12:44:23.107571+0000","last_fullsized":"2023-10-19T12:44:23.107571+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:29:52.686703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.258880+0000","last_change":"2023-10-19T12:44:23.262149+0000","last_active":"2023-10-19T12:44:24.258880+0000","last_peered":"2023-10-19T12:44:24.258880+0000","last_clean":"2023-10-19T12:44:24.258880+0000","last_became_active":"2023-10-19T12:44:23.261764+0000","last_became_peered":"2023-10-19T12:44:23.261764+0000","last_unstale":"2023-10-19T12:44:24.258880+0000","last_undegraded":"2023-10-19T12:44:24.258880+0000","last_fullsized":"2023-10-19T12:44:24.258880+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:33:24.282692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.107679+0000","last_change":"2023-10-19T12:44:18.064762+0000","last_active":"2023-10-19T12:44:23.107679+0000","last_peered":"2023-10-19T12:44:23.107679+0000","last_clean":"2023-10-19T12:44:23.107679+0000","last_became_active":"2023-10-19T12:44:18.064478+0000","last_became_peered":"2023-10-19T12:44:18.064478+0000","last_unstale":"2023-10-19T12:44:23.107679+0000","last_undegraded":"2023-10-19T12:44:23.107679+0000","last_fullsized":"2023-10-19T12:44:23.107679+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:46:29.885859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.113623+0000","last_change":"2023-10-19T12:44:19.083619+0000","last_active":"2023-10-19T12:44:23.113623+0000","last_peered":"2023-10-19T12:44:23.113623+0000","last_clean":"2023-10-19T12:44:23.113623+0000","last_became_active":"2023-10-19T12:44:19.083330+0000","last_became_peered":"2023-10-19T12:44:19.083330+0000","last_unstale":"2023-10-19T12:44:23.113623+0000","last_undegraded":"2023-10-19T12:44:23.113623+0000","last_fullsized":"2023-10-19T12:44:23.113623+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:00:15.982705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.123057+0000","last_change":"2023-10-19T12:44:23.132517+0000","last_active":"2023-10-19T12:44:24.123057+0000","last_peered":"2023-10-19T12:44:24.123057+0000","last_clean":"2023-10-19T12:44:24.123057+0000","last_became_active":"2023-10-19T12:44:23.132066+0000","last_became_peered":"2023-10-19T12:44:23.132066+0000","last_unstale":"2023-10-19T12:44:24.123057+0000","last_undegraded":"2023-10-19T12:44:24.123057+0000","last_fullsized":"2023-10-19T12:44:24.123057+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:46:59.402450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.260017+0000","last_change":"2023-10-19T12:44:18.070873+0000","last_active":"2023-10-19T12:44:23.260017+0000","last_peered":"2023-10-19T12:44:23.260017+0000","last_clean":"2023-10-19T12:44:23.260017+0000","last_became_active":"2023-10-19T12:44:18.070602+0000","last_became_peered":"2023-10-19T12:44:18.070602+0000","last_unstale":"2023-10-19T12:44:23.260017+0000","last_undegraded":"2023-10-19T12:44:23.260017+0000","last_fullsized":"2023-10-19T12:44:23.260017+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:15:18.575278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.106941+0000","last_change":"2023-10-19T12:44:19.075787+0000","last_active":"2023-10-19T12:44:23.106941+0000","last_peered":"2023-10-19T12:44:23.106941+0000","last_clean":"2023-10-19T12:44:23.106941+0000","last_became_active":"2023-10-19T12:44:19.074529+0000","last_became_peered":"2023-10-19T12:44:19.074529+0000","last_unstale":"2023-10-19T12:44:23.106941+0000","last_undegraded":"2023-10-19T12:44:23.106941+0000","last_fullsized":"2023-10-19T12:44:23.106941+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:29:52.686703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.124069+0000","last_change":"2023-10-19T12:44:23.132792+0000","last_active":"2023-10-19T12:44:24.124069+0000","last_peered":"2023-10-19T12:44:24.124069+0000","last_clean":"2023-10-19T12:44:24.124069+0000","last_became_active":"2023-10-19T12:44:23.132099+0000","last_became_peered":"2023-10-19T12:44:23.132099+0000","last_unstale":"2023-10-19T12:44:24.124069+0000","last_undegraded":"2023-10-19T12:44:24.124069+0000","last_fullsized":"2023-10-19T12:44:24.124069+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:01:13.019006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.259778+0000","last_change":"2023-10-19T12:44:18.068582+0000","last_active":"2023-10-19T12:44:23.259778+0000","last_peered":"2023-10-19T12:44:23.259778+0000","last_clean":"2023-10-19T12:44:23.259778+0000","last_became_active":"2023-10-19T12:44:18.068099+0000","last_became_peered":"2023-10-19T12:44:18.068099+0000","last_unstale":"2023-10-19T12:44:23.259778+0000","last_undegraded":"2023-10-19T12:44:23.259778+0000","last_fullsized":"2023-10-19T12:44:23.259778+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:24:31.294273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.107389+0000","last_change":"2023-10-19T12:44:19.076968+0000","last_active":"2023-10-19T12:44:23.107389+0000","last_peered":"2023-10-19T12:44:23.107389+0000","last_clean":"2023-10-19T12:44:23.107389+0000","last_became_active":"2023-10-19T12:44:19.076105+0000","last_became_peered":"2023-10-19T12:44:19.076105+0000","last_unstale":"2023-10-19T12:44:23.107389+0000","last_undegraded":"2023-10-19T12:44:23.107389+0000","last_fullsized":"2023-10-19T12:44:23.107389+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:59:45.926427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.123560+0000","last_change":"2023-10-19T12:44:23.125353+0000","last_active":"2023-10-19T12:44:24.123560+0000","last_peered":"2023-10-19T12:44:24.123560+0000","last_clean":"2023-10-19T12:44:24.123560+0000","last_became_active":"2023-10-19T12:44:23.124845+0000","last_became_peered":"2023-10-19T12:44:23.124845+0000","last_unstale":"2023-10-19T12:44:24.123560+0000","last_undegraded":"2023-10-19T12:44:24.123560+0000","last_fullsized":"2023-10-19T12:44:24.123560+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:59:49.972984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.116065+0000","last_change":"2023-10-19T12:44:19.084588+0000","last_active":"2023-10-19T12:44:23.116065+0000","last_peered":"2023-10-19T12:44:23.116065+0000","last_clean":"2023-10-19T12:44:23.116065+0000","last_became_active":"2023-10-19T12:44:19.084315+0000","last_became_peered":"2023-10-19T12:44:19.084315+0000","last_unstale":"2023-10-19T12:44:23.116065+0000","last_undegraded":"2023-10-19T12:44:23.116065+0000","last_fullsized":"2023-10-19T12:44:23.116065+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:54:38.037190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.259794+0000","last_change":"2023-10-19T12:44:18.069120+0000","last_active":"2023-10-19T12:44:23.259794+0000","last_peered":"2023-10-19T12:44:23.259794+0000","last_clean":"2023-10-19T12:44:23.259794+0000","last_became_active":"2023-10-19T12:44:18.068509+0000","last_became_peered":"2023-10-19T12:44:18.068509+0000","last_unstale":"2023-10-19T12:44:23.259794+0000","last_undegraded":"2023-10-19T12:44:23.259794+0000","last_fullsized":"2023-10-19T12:44:23.259794+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:01:31.282481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.124541+0000","last_change":"2023-10-19T12:44:23.133194+0000","last_active":"2023-10-19T12:44:24.124541+0000","last_peered":"2023-10-19T12:44:24.124541+0000","last_clean":"2023-10-19T12:44:24.124541+0000","last_became_active":"2023-10-19T12:44:23.132109+0000","last_became_peered":"2023-10-19T12:44:23.132109+0000","last_unstale":"2023-10-19T12:44:24.124541+0000","last_undegraded":"2023-10-19T12:44:24.124541+0000","last_fullsized":"2023-10-19T12:44:24.124541+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:37:44.936303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.108577+0000","last_change":"2023-10-19T12:44:19.082038+0000","last_active":"2023-10-19T12:44:23.108577+0000","last_peered":"2023-10-19T12:44:23.108577+0000","last_clean":"2023-10-19T12:44:23.108577+0000","last_became_active":"2023-10-19T12:44:19.080765+0000","last_became_peered":"2023-10-19T12:44:19.080765+0000","last_unstale":"2023-10-19T12:44:23.108577+0000","last_undegraded":"2023-10-19T12:44:23.108577+0000","last_fullsized":"2023-10-19T12:44:23.108577+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:01:32.286452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.123463+0000","last_change":"2023-10-19T12:44:18.070227+0000","last_active":"2023-10-19T12:44:23.123463+0000","last_peered":"2023-10-19T12:44:23.123463+0000","last_clean":"2023-10-19T12:44:23.123463+0000","last_became_active":"2023-10-19T12:44:18.069958+0000","last_became_peered":"2023-10-19T12:44:18.069958+0000","last_unstale":"2023-10-19T12:44:23.123463+0000","last_undegraded":"2023-10-19T12:44:23.123463+0000","last_fullsized":"2023-10-19T12:44:23.123463+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:27:49.242393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.123413+0000","last_change":"2023-10-19T12:44:23.126070+0000","last_active":"2023-10-19T12:44:24.123413+0000","last_peered":"2023-10-19T12:44:24.123413+0000","last_clean":"2023-10-19T12:44:24.123413+0000","last_became_active":"2023-10-19T12:44:23.125651+0000","last_became_peered":"2023-10-19T12:44:23.125651+0000","last_unstale":"2023-10-19T12:44:24.123413+0000","last_undegraded":"2023-10-19T12:44:24.123413+0000","last_fullsized":"2023-10-19T12:44:24.123413+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:29:56.733260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.109518+0000","last_change":"2023-10-19T12:44:19.085767+0000","last_active":"2023-10-19T12:44:23.109518+0000","last_peered":"2023-10-19T12:44:23.109518+0000","last_clean":"2023-10-19T12:44:23.109518+0000","last_became_active":"2023-10-19T12:44:19.085473+0000","last_became_peered":"2023-10-19T12:44:19.085473+0000","last_unstale":"2023-10-19T12:44:23.109518+0000","last_undegraded":"2023-10-19T12:44:23.109518+0000","last_fullsized":"2023-10-19T12:44:23.109518+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:50:42.817437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.123002+0000","last_change":"2023-10-19T12:44:18.074055+0000","last_active":"2023-10-19T12:44:23.123002+0000","last_peered":"2023-10-19T12:44:23.123002+0000","last_clean":"2023-10-19T12:44:23.123002+0000","last_became_active":"2023-10-19T12:44:18.073821+0000","last_became_peered":"2023-10-19T12:44:18.073821+0000","last_unstale":"2023-10-19T12:44:23.123002+0000","last_undegraded":"2023-10-19T12:44:23.123002+0000","last_fullsized":"2023-10-19T12:44:23.123002+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:24:31.294273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.125580+0000","last_change":"2023-10-19T12:44:23.263639+0000","last_active":"2023-10-19T12:44:24.125580+0000","last_peered":"2023-10-19T12:44:24.125580+0000","last_clean":"2023-10-19T12:44:24.125580+0000","last_became_active":"2023-10-19T12:44:23.263261+0000","last_became_peered":"2023-10-19T12:44:23.263261+0000","last_unstale":"2023-10-19T12:44:24.125580+0000","last_undegraded":"2023-10-19T12:44:24.125580+0000","last_fullsized":"2023-10-19T12:44:24.125580+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:53:36.534082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.114649+0000","last_change":"2023-10-19T12:44:19.080014+0000","last_active":"2023-10-19T12:44:23.114649+0000","last_peered":"2023-10-19T12:44:23.114649+0000","last_clean":"2023-10-19T12:44:23.114649+0000","last_became_active":"2023-10-19T12:44:19.079754+0000","last_became_peered":"2023-10-19T12:44:19.079754+0000","last_unstale":"2023-10-19T12:44:23.114649+0000","last_undegraded":"2023-10-19T12:44:23.114649+0000","last_fullsized":"2023-10-19T12:44:23.114649+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:27:39.900179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.259411+0000","last_change":"2023-10-19T12:44:18.069208+0000","last_active":"2023-10-19T12:44:23.259411+0000","last_peered":"2023-10-19T12:44:23.259411+0000","last_clean":"2023-10-19T12:44:23.259411+0000","last_became_active":"2023-10-19T12:44:18.068303+0000","last_became_peered":"2023-10-19T12:44:18.068303+0000","last_unstale":"2023-10-19T12:44:23.259411+0000","last_undegraded":"2023-10-19T12:44:23.259411+0000","last_fullsized":"2023-10-19T12:44:23.259411+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:11:21.842323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.123471+0000","last_change":"2023-10-19T12:44:23.131978+0000","last_active":"2023-10-19T12:44:24.123471+0000","last_peered":"2023-10-19T12:44:24.123471+0000","last_clean":"2023-10-19T12:44:24.123471+0000","last_became_active":"2023-10-19T12:44:23.131330+0000","last_became_peered":"2023-10-19T12:44:23.131330+0000","last_unstale":"2023-10-19T12:44:24.123471+0000","last_undegraded":"2023-10-19T12:44:24.123471+0000","last_fullsized":"2023-10-19T12:44:24.123471+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:49:30.221056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.113302+0000","last_change":"2023-10-19T12:44:19.087545+0000","last_active":"2023-10-19T12:44:23.113302+0000","last_peered":"2023-10-19T12:44:23.113302+0000","last_clean":"2023-10-19T12:44:23.113302+0000","last_became_active":"2023-10-19T12:44:19.087254+0000","last_became_peered":"2023-10-19T12:44:19.087254+0000","last_unstale":"2023-10-19T12:44:23.113302+0000","last_undegraded":"2023-10-19T12:44:23.113302+0000","last_fullsized":"2023-10-19T12:44:23.113302+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:22:48.368572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.116983+0000","last_change":"2023-10-19T12:44:18.074011+0000","last_active":"2023-10-19T12:44:23.116983+0000","last_peered":"2023-10-19T12:44:23.116983+0000","last_clean":"2023-10-19T12:44:23.116983+0000","last_became_active":"2023-10-19T12:44:18.073749+0000","last_became_peered":"2023-10-19T12:44:18.073749+0000","last_unstale":"2023-10-19T12:44:23.116983+0000","last_undegraded":"2023-10-19T12:44:23.116983+0000","last_fullsized":"2023-10-19T12:44:23.116983+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:46:29.885859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.123298+0000","last_change":"2023-10-19T12:44:23.264194+0000","last_active":"2023-10-19T12:44:24.123298+0000","last_peered":"2023-10-19T12:44:24.123298+0000","last_clean":"2023-10-19T12:44:24.123298+0000","last_became_active":"2023-10-19T12:44:23.263028+0000","last_became_peered":"2023-10-19T12:44:23.263028+0000","last_unstale":"2023-10-19T12:44:24.123298+0000","last_undegraded":"2023-10-19T12:44:24.123298+0000","last_fullsized":"2023-10-19T12:44:24.123298+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:05:19.882281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.107858+0000","last_change":"2023-10-19T12:44:19.094059+0000","last_active":"2023-10-19T12:44:23.107858+0000","last_peered":"2023-10-19T12:44:23.107858+0000","last_clean":"2023-10-19T12:44:23.107858+0000","last_became_active":"2023-10-19T12:44:19.093724+0000","last_became_peered":"2023-10-19T12:44:19.093724+0000","last_unstale":"2023-10-19T12:44:23.107858+0000","last_undegraded":"2023-10-19T12:44:23.107858+0000","last_fullsized":"2023-10-19T12:44:23.107858+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:49:26.174499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.123329+0000","last_change":"2023-10-19T12:44:18.071840+0000","last_active":"2023-10-19T12:44:23.123329+0000","last_peered":"2023-10-19T12:44:23.123329+0000","last_clean":"2023-10-19T12:44:23.123329+0000","last_became_active":"2023-10-19T12:44:18.071447+0000","last_became_peered":"2023-10-19T12:44:18.071447+0000","last_unstale":"2023-10-19T12:44:23.123329+0000","last_undegraded":"2023-10-19T12:44:23.123329+0000","last_fullsized":"2023-10-19T12:44:23.123329+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:46:38.630648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.258316+0000","last_change":"2023-10-19T12:44:23.263517+0000","last_active":"2023-10-19T12:44:24.258316+0000","last_peered":"2023-10-19T12:44:24.258316+0000","last_clean":"2023-10-19T12:44:24.258316+0000","last_became_active":"2023-10-19T12:44:23.263105+0000","last_became_peered":"2023-10-19T12:44:23.263105+0000","last_unstale":"2023-10-19T12:44:24.258316+0000","last_undegraded":"2023-10-19T12:44:24.258316+0000","last_fullsized":"2023-10-19T12:44:24.258316+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:56:44.960048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.122932+0000","last_change":"2023-10-19T12:44:19.088058+0000","last_active":"2023-10-19T12:44:23.122932+0000","last_peered":"2023-10-19T12:44:23.122932+0000","last_clean":"2023-10-19T12:44:23.122932+0000","last_became_active":"2023-10-19T12:44:19.087755+0000","last_became_peered":"2023-10-19T12:44:19.087755+0000","last_unstale":"2023-10-19T12:44:23.122932+0000","last_undegraded":"2023-10-19T12:44:23.122932+0000","last_fullsized":"2023-10-19T12:44:23.122932+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:54:38.037190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.260168+0000","last_change":"2023-10-19T12:44:18.070041+0000","last_active":"2023-10-19T12:44:23.260168+0000","last_peered":"2023-10-19T12:44:23.260168+0000","last_clean":"2023-10-19T12:44:23.260168+0000","last_became_active":"2023-10-19T12:44:18.069723+0000","last_became_peered":"2023-10-19T12:44:18.069723+0000","last_unstale":"2023-10-19T12:44:23.260168+0000","last_undegraded":"2023-10-19T12:44:23.260168+0000","last_fullsized":"2023-10-19T12:44:23.260168+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:10:07.046736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.258633+0000","last_change":"2023-10-19T12:44:23.260957+0000","last_active":"2023-10-19T12:44:24.258633+0000","last_peered":"2023-10-19T12:44:24.258633+0000","last_clean":"2023-10-19T12:44:24.258633+0000","last_became_active":"2023-10-19T12:44:23.260559+0000","last_became_peered":"2023-10-19T12:44:23.260559+0000","last_unstale":"2023-10-19T12:44:24.258633+0000","last_undegraded":"2023-10-19T12:44:24.258633+0000","last_fullsized":"2023-10-19T12:44:24.258633+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:30:09.977362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.107403+0000","last_change":"2023-10-19T12:44:19.080613+0000","last_active":"2023-10-19T12:44:23.107403+0000","last_peered":"2023-10-19T12:44:23.107403+0000","last_clean":"2023-10-19T12:44:23.107403+0000","last_became_active":"2023-10-19T12:44:19.080260+0000","last_became_peered":"2023-10-19T12:44:19.080260+0000","last_unstale":"2023-10-19T12:44:23.107403+0000","last_undegraded":"2023-10-19T12:44:23.107403+0000","last_fullsized":"2023-10-19T12:44:23.107403+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:49:26.174499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.123051+0000","last_change":"2023-10-19T12:44:18.072668+0000","last_active":"2023-10-19T12:44:23.123051+0000","last_peered":"2023-10-19T12:44:23.123051+0000","last_clean":"2023-10-19T12:44:23.123051+0000","last_became_active":"2023-10-19T12:44:18.072383+0000","last_became_peered":"2023-10-19T12:44:18.072383+0000","last_unstale":"2023-10-19T12:44:23.123051+0000","last_undegraded":"2023-10-19T12:44:23.123051+0000","last_fullsized":"2023-10-19T12:44:23.123051+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:07:40.511877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.124753+0000","last_change":"2023-10-19T12:44:23.130828+0000","last_active":"2023-10-19T12:44:24.124753+0000","last_peered":"2023-10-19T12:44:24.124753+0000","last_clean":"2023-10-19T12:44:24.124753+0000","last_became_active":"2023-10-19T12:44:23.130454+0000","last_became_peered":"2023-10-19T12:44:23.130454+0000","last_unstale":"2023-10-19T12:44:24.124753+0000","last_undegraded":"2023-10-19T12:44:24.124753+0000","last_fullsized":"2023-10-19T12:44:24.124753+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:30:09.977362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.114746+0000","last_change":"2023-10-19T12:44:19.075066+0000","last_active":"2023-10-19T12:44:23.114746+0000","last_peered":"2023-10-19T12:44:23.114746+0000","last_clean":"2023-10-19T12:44:23.114746+0000","last_became_active":"2023-10-19T12:44:19.074770+0000","last_became_peered":"2023-10-19T12:44:19.074770+0000","last_unstale":"2023-10-19T12:44:23.114746+0000","last_undegraded":"2023-10-19T12:44:23.114746+0000","last_fullsized":"2023-10-19T12:44:23.114746+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:11:22.846295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.106419+0000","last_change":"2023-10-19T12:44:18.072600+0000","last_active":"2023-10-19T12:44:23.106419+0000","last_peered":"2023-10-19T12:44:23.106419+0000","last_clean":"2023-10-19T12:44:23.106419+0000","last_became_active":"2023-10-19T12:44:18.072278+0000","last_became_peered":"2023-10-19T12:44:18.072278+0000","last_unstale":"2023-10-19T12:44:23.106419+0000","last_undegraded":"2023-10-19T12:44:23.106419+0000","last_fullsized":"2023-10-19T12:44:23.106419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:50:41.813466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.125020+0000","last_change":"2023-10-19T12:44:23.132395+0000","last_active":"2023-10-19T12:44:24.125020+0000","last_peered":"2023-10-19T12:44:24.125020+0000","last_clean":"2023-10-19T12:44:24.125020+0000","last_became_active":"2023-10-19T12:44:23.131961+0000","last_became_peered":"2023-10-19T12:44:23.131961+0000","last_unstale":"2023-10-19T12:44:24.125020+0000","last_undegraded":"2023-10-19T12:44:24.125020+0000","last_fullsized":"2023-10-19T12:44:24.125020+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:37:44.936303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.122772+0000","last_change":"2023-10-19T12:44:19.084119+0000","last_active":"2023-10-19T12:44:23.122772+0000","last_peered":"2023-10-19T12:44:23.122772+0000","last_clean":"2023-10-19T12:44:23.122772+0000","last_became_active":"2023-10-19T12:44:19.083803+0000","last_became_peered":"2023-10-19T12:44:19.083803+0000","last_unstale":"2023-10-19T12:44:23.122772+0000","last_undegraded":"2023-10-19T12:44:23.122772+0000","last_fullsized":"2023-10-19T12:44:23.122772+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:01:32.286452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.107952+0000","last_change":"2023-10-19T12:44:18.066014+0000","last_active":"2023-10-19T12:44:23.107952+0000","last_peered":"2023-10-19T12:44:23.107952+0000","last_clean":"2023-10-19T12:44:23.107952+0000","last_became_active":"2023-10-19T12:44:18.065652+0000","last_became_peered":"2023-10-19T12:44:18.065652+0000","last_unstale":"2023-10-19T12:44:23.107952+0000","last_undegraded":"2023-10-19T12:44:23.107952+0000","last_fullsized":"2023-10-19T12:44:23.107952+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:46:36.490219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.125536+0000","last_change":"2023-10-19T12:44:23.135917+0000","last_active":"2023-10-19T12:44:24.125536+0000","last_peered":"2023-10-19T12:44:24.125536+0000","last_clean":"2023-10-19T12:44:24.125536+0000","last_became_active":"2023-10-19T12:44:23.135459+0000","last_became_peered":"2023-10-19T12:44:23.135459+0000","last_unstale":"2023-10-19T12:44:24.125536+0000","last_undegraded":"2023-10-19T12:44:24.125536+0000","last_fullsized":"2023-10-19T12:44:24.125536+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:24:52.469571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.114317+0000","last_change":"2023-10-19T12:44:19.071520+0000","last_active":"2023-10-19T12:44:23.114317+0000","last_peered":"2023-10-19T12:44:23.114317+0000","last_clean":"2023-10-19T12:44:23.114317+0000","last_became_active":"2023-10-19T12:44:19.071008+0000","last_became_peered":"2023-10-19T12:44:19.071008+0000","last_unstale":"2023-10-19T12:44:23.114317+0000","last_undegraded":"2023-10-19T12:44:23.114317+0000","last_fullsized":"2023-10-19T12:44:23.114317+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:27:50.246364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.259411+0000","last_change":"2023-10-19T12:44:18.071276+0000","last_active":"2023-10-19T12:44:23.259411+0000","last_peered":"2023-10-19T12:44:23.259411+0000","last_clean":"2023-10-19T12:44:23.259411+0000","last_became_active":"2023-10-19T12:44:18.069565+0000","last_became_peered":"2023-10-19T12:44:18.069565+0000","last_unstale":"2023-10-19T12:44:23.259411+0000","last_undegraded":"2023-10-19T12:44:23.259411+0000","last_fullsized":"2023-10-19T12:44:23.259411+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:46:38.630648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.259572+0000","last_change":"2023-10-19T12:44:23.261271+0000","last_active":"2023-10-19T12:44:24.259572+0000","last_peered":"2023-10-19T12:44:24.259572+0000","last_clean":"2023-10-19T12:44:24.259572+0000","last_became_active":"2023-10-19T12:44:23.260853+0000","last_became_peered":"2023-10-19T12:44:23.260853+0000","last_unstale":"2023-10-19T12:44:24.259572+0000","last_undegraded":"2023-10-19T12:44:24.259572+0000","last_fullsized":"2023-10-19T12:44:24.259572+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:53:36.534082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.108153+0000","last_change":"2023-10-19T12:44:19.086009+0000","last_active":"2023-10-19T12:44:23.108153+0000","last_peered":"2023-10-19T12:44:23.108153+0000","last_clean":"2023-10-19T12:44:23.108153+0000","last_became_active":"2023-10-19T12:44:19.085688+0000","last_became_peered":"2023-10-19T12:44:19.085688+0000","last_unstale":"2023-10-19T12:44:23.108153+0000","last_undegraded":"2023-10-19T12:44:23.108153+0000","last_fullsized":"2023-10-19T12:44:23.108153+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:00:15.982705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.106386+0000","last_change":"2023-10-19T12:44:18.067051+0000","last_active":"2023-10-19T12:44:23.106386+0000","last_peered":"2023-10-19T12:44:23.106386+0000","last_clean":"2023-10-19T12:44:23.106386+0000","last_became_active":"2023-10-19T12:44:18.066784+0000","last_became_peered":"2023-10-19T12:44:18.066784+0000","last_unstale":"2023-10-19T12:44:23.106386+0000","last_undegraded":"2023-10-19T12:44:23.106386+0000","last_fullsized":"2023-10-19T12:44:23.106386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:43:47.410870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":168,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.259530+0000","last_change":"2023-10-19T12:44:08.636613+0000","last_active":"2023-10-19T12:44:23.259530+0000","last_peered":"2023-10-19T12:44:23.259530+0000","last_clean":"2023-10-19T12:44:23.259530+0000","last_became_active":"2023-10-19T12:44:08.635697+0000","last_became_peered":"2023-10-19T12:44:08.635697+0000","last_unstale":"2023-10-19T12:44:23.259530+0000","last_undegraded":"2023-10-19T12:44:23.259530+0000","last_fullsized":"2023-10-19T12:44:23.259530+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_clean_scrub_stamp":"2023-10-19T12:44:07.399738+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:15:23.685485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.258583+0000","last_change":"2023-10-19T12:44:23.261893+0000","last_active":"2023-10-19T12:44:24.258583+0000","last_peered":"2023-10-19T12:44:24.258583+0000","last_clean":"2023-10-19T12:44:24.258583+0000","last_became_active":"2023-10-19T12:44:23.261440+0000","last_became_peered":"2023-10-19T12:44:23.261440+0000","last_unstale":"2023-10-19T12:44:24.258583+0000","last_undegraded":"2023-10-19T12:44:24.258583+0000","last_fullsized":"2023-10-19T12:44:24.258583+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:30:09.977362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.259893+0000","last_change":"2023-10-19T12:44:19.093167+0000","last_active":"2023-10-19T12:44:23.259893+0000","last_peered":"2023-10-19T12:44:23.259893+0000","last_clean":"2023-10-19T12:44:23.259893+0000","last_became_active":"2023-10-19T12:44:19.092842+0000","last_became_peered":"2023-10-19T12:44:19.092842+0000","last_unstale":"2023-10-19T12:44:23.259893+0000","last_undegraded":"2023-10-19T12:44:23.259893+0000","last_fullsized":"2023-10-19T12:44:23.259893+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:01:08.972450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.108142+0000","last_change":"2023-10-19T12:44:18.071274+0000","last_active":"2023-10-19T12:44:23.108142+0000","last_peered":"2023-10-19T12:44:23.108142+0000","last_clean":"2023-10-19T12:44:23.108142+0000","last_became_active":"2023-10-19T12:44:18.070997+0000","last_became_peered":"2023-10-19T12:44:18.070997+0000","last_unstale":"2023-10-19T12:44:23.108142+0000","last_undegraded":"2023-10-19T12:44:23.108142+0000","last_fullsized":"2023-10-19T12:44:23.108142+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:27:38.896207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.258302+0000","last_change":"2023-10-19T12:44:23.261162+0000","last_active":"2023-10-19T12:44:24.258302+0000","last_peered":"2023-10-19T12:44:24.258302+0000","last_clean":"2023-10-19T12:44:24.258302+0000","last_became_active":"2023-10-19T12:44:23.260701+0000","last_became_peered":"2023-10-19T12:44:23.260701+0000","last_unstale":"2023-10-19T12:44:24.258302+0000","last_undegraded":"2023-10-19T12:44:24.258302+0000","last_fullsized":"2023-10-19T12:44:24.258302+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:01:13.019006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.107087+0000","last_change":"2023-10-19T12:44:19.090016+0000","last_active":"2023-10-19T12:44:23.107087+0000","last_peered":"2023-10-19T12:44:23.107087+0000","last_clean":"2023-10-19T12:44:23.107087+0000","last_became_active":"2023-10-19T12:44:19.086367+0000","last_became_peered":"2023-10-19T12:44:19.086367+0000","last_unstale":"2023-10-19T12:44:23.107087+0000","last_undegraded":"2023-10-19T12:44:23.107087+0000","last_fullsized":"2023-10-19T12:44:23.107087+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:01:08.972450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.123668+0000","last_change":"2023-10-19T12:44:18.069871+0000","last_active":"2023-10-19T12:44:23.123668+0000","last_peered":"2023-10-19T12:44:23.123668+0000","last_clean":"2023-10-19T12:44:23.123668+0000","last_became_active":"2023-10-19T12:44:18.069558+0000","last_became_peered":"2023-10-19T12:44:18.069558+0000","last_unstale":"2023-10-19T12:44:23.123668+0000","last_undegraded":"2023-10-19T12:44:23.123668+0000","last_fullsized":"2023-10-19T12:44:23.123668+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:15:18.575278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.258925+0000","last_change":"2023-10-19T12:44:23.131775+0000","last_active":"2023-10-19T12:44:24.258925+0000","last_peered":"2023-10-19T12:44:24.258925+0000","last_clean":"2023-10-19T12:44:24.258925+0000","last_became_active":"2023-10-19T12:44:23.131215+0000","last_became_peered":"2023-10-19T12:44:23.131215+0000","last_unstale":"2023-10-19T12:44:24.258925+0000","last_undegraded":"2023-10-19T12:44:24.258925+0000","last_fullsized":"2023-10-19T12:44:24.258925+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:37:44.936303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.260452+0000","last_change":"2023-10-19T12:44:19.089818+0000","last_active":"2023-10-19T12:44:23.260452+0000","last_peered":"2023-10-19T12:44:23.260452+0000","last_clean":"2023-10-19T12:44:23.260452+0000","last_became_active":"2023-10-19T12:44:19.089336+0000","last_became_peered":"2023-10-19T12:44:19.089336+0000","last_unstale":"2023-10-19T12:44:23.260452+0000","last_undegraded":"2023-10-19T12:44:23.260452+0000","last_fullsized":"2023-10-19T12:44:23.260452+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:53:32.487525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.108384+0000","last_change":"2023-10-19T12:44:18.068828+0000","last_active":"2023-10-19T12:44:23.108384+0000","last_peered":"2023-10-19T12:44:23.108384+0000","last_clean":"2023-10-19T12:44:23.108384+0000","last_became_active":"2023-10-19T12:44:18.068553+0000","last_became_peered":"2023-10-19T12:44:18.068553+0000","last_unstale":"2023-10-19T12:44:23.108384+0000","last_undegraded":"2023-10-19T12:44:23.108384+0000","last_fullsized":"2023-10-19T12:44:23.108384+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:11:21.842323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.123595+0000","last_change":"2023-10-19T12:44:23.261245+0000","last_active":"2023-10-19T12:44:24.123595+0000","last_peered":"2023-10-19T12:44:24.123595+0000","last_clean":"2023-10-19T12:44:24.123595+0000","last_became_active":"2023-10-19T12:44:23.260818+0000","last_became_peered":"2023-10-19T12:44:23.260818+0000","last_unstale":"2023-10-19T12:44:24.123595+0000","last_undegraded":"2023-10-19T12:44:24.123595+0000","last_fullsized":"2023-10-19T12:44:24.123595+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:55:55.981359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.260228+0000","last_change":"2023-10-19T12:44:19.087785+0000","last_active":"2023-10-19T12:44:23.260228+0000","last_peered":"2023-10-19T12:44:23.260228+0000","last_clean":"2023-10-19T12:44:23.260228+0000","last_became_active":"2023-10-19T12:44:19.087394+0000","last_became_peered":"2023-10-19T12:44:19.087394+0000","last_unstale":"2023-10-19T12:44:23.260228+0000","last_undegraded":"2023-10-19T12:44:23.260228+0000","last_fullsized":"2023-10-19T12:44:23.260228+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:37:40.889746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.115032+0000","last_change":"2023-10-19T12:44:18.068944+0000","last_active":"2023-10-19T12:44:23.115032+0000","last_peered":"2023-10-19T12:44:23.115032+0000","last_clean":"2023-10-19T12:44:23.115032+0000","last_became_active":"2023-10-19T12:44:18.068674+0000","last_became_peered":"2023-10-19T12:44:18.068674+0000","last_unstale":"2023-10-19T12:44:23.115032+0000","last_undegraded":"2023-10-19T12:44:23.115032+0000","last_fullsized":"2023-10-19T12:44:23.115032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:16:09.533399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.123008+0000","last_change":"2023-10-19T12:44:23.131983+0000","last_active":"2023-10-19T12:44:24.123008+0000","last_peered":"2023-10-19T12:44:24.123008+0000","last_clean":"2023-10-19T12:44:24.123008+0000","last_became_active":"2023-10-19T12:44:23.131304+0000","last_became_peered":"2023-10-19T12:44:23.131304+0000","last_unstale":"2023-10-19T12:44:24.123008+0000","last_undegraded":"2023-10-19T12:44:24.123008+0000","last_fullsized":"2023-10-19T12:44:24.123008+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:01:36.333009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.259741+0000","last_change":"2023-10-19T12:44:19.088843+0000","last_active":"2023-10-19T12:44:23.259741+0000","last_peered":"2023-10-19T12:44:23.259741+0000","last_clean":"2023-10-19T12:44:23.259741+0000","last_became_active":"2023-10-19T12:44:19.088578+0000","last_became_peered":"2023-10-19T12:44:19.088578+0000","last_unstale":"2023-10-19T12:44:23.259741+0000","last_undegraded":"2023-10-19T12:44:23.259741+0000","last_fullsized":"2023-10-19T12:44:23.259741+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:46:55.355893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.108073+0000","last_change":"2023-10-19T12:44:18.069371+0000","last_active":"2023-10-19T12:44:23.108073+0000","last_peered":"2023-10-19T12:44:23.108073+0000","last_clean":"2023-10-19T12:44:23.108073+0000","last_became_active":"2023-10-19T12:44:18.069118+0000","last_became_peered":"2023-10-19T12:44:18.069118+0000","last_unstale":"2023-10-19T12:44:23.108073+0000","last_undegraded":"2023-10-19T12:44:23.108073+0000","last_fullsized":"2023-10-19T12:44:23.108073+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:12:58.093890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.259448+0000","last_change":"2023-10-19T12:44:23.126162+0000","last_active":"2023-10-19T12:44:24.259448+0000","last_peered":"2023-10-19T12:44:24.259448+0000","last_clean":"2023-10-19T12:44:24.259448+0000","last_became_active":"2023-10-19T12:44:23.125364+0000","last_became_peered":"2023-10-19T12:44:23.125364+0000","last_unstale":"2023-10-19T12:44:24.259448+0000","last_undegraded":"2023-10-19T12:44:24.259448+0000","last_fullsized":"2023-10-19T12:44:24.259448+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:24:52.469571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.114403+0000","last_change":"2023-10-19T12:44:19.079421+0000","last_active":"2023-10-19T12:44:23.114403+0000","last_peered":"2023-10-19T12:44:23.114403+0000","last_clean":"2023-10-19T12:44:23.114403+0000","last_became_active":"2023-10-19T12:44:19.079151+0000","last_became_peered":"2023-10-19T12:44:19.079151+0000","last_unstale":"2023-10-19T12:44:23.114403+0000","last_undegraded":"2023-10-19T12:44:23.114403+0000","last_fullsized":"2023-10-19T12:44:23.114403+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:22:48.368572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.114676+0000","last_change":"2023-10-19T12:44:18.070653+0000","last_active":"2023-10-19T12:44:23.114676+0000","last_peered":"2023-10-19T12:44:23.114676+0000","last_clean":"2023-10-19T12:44:23.114676+0000","last_became_active":"2023-10-19T12:44:18.070404+0000","last_became_peered":"2023-10-19T12:44:18.070404+0000","last_unstale":"2023-10-19T12:44:23.114676+0000","last_undegraded":"2023-10-19T12:44:23.114676+0000","last_fullsized":"2023-10-19T12:44:23.114676+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:27:38.896207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.259106+0000","last_change":"2023-10-19T12:44:23.262410+0000","last_active":"2023-10-19T12:44:24.259106+0000","last_peered":"2023-10-19T12:44:24.259106+0000","last_clean":"2023-10-19T12:44:24.259106+0000","last_became_active":"2023-10-19T12:44:23.261667+0000","last_became_peered":"2023-10-19T12:44:23.261667+0000","last_unstale":"2023-10-19T12:44:24.259106+0000","last_undegraded":"2023-10-19T12:44:24.259106+0000","last_fullsized":"2023-10-19T12:44:24.259106+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:23:00.780721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.259656+0000","last_change":"2023-10-19T12:44:19.092039+0000","last_active":"2023-10-19T12:44:23.259656+0000","last_peered":"2023-10-19T12:44:23.259656+0000","last_clean":"2023-10-19T12:44:23.259656+0000","last_became_active":"2023-10-19T12:44:19.090421+0000","last_became_peered":"2023-10-19T12:44:19.090421+0000","last_unstale":"2023-10-19T12:44:23.259656+0000","last_undegraded":"2023-10-19T12:44:23.259656+0000","last_fullsized":"2023-10-19T12:44:23.259656+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:49:26.174499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.106406+0000","last_change":"2023-10-19T12:44:18.066260+0000","last_active":"2023-10-19T12:44:23.106406+0000","last_peered":"2023-10-19T12:44:23.106406+0000","last_clean":"2023-10-19T12:44:23.106406+0000","last_became_active":"2023-10-19T12:44:18.065913+0000","last_became_peered":"2023-10-19T12:44:18.065913+0000","last_unstale":"2023-10-19T12:44:23.106406+0000","last_undegraded":"2023-10-19T12:44:23.106406+0000","last_fullsized":"2023-10-19T12:44:23.106406+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:22:47.364601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.260099+0000","last_change":"2023-10-19T12:44:23.261486+0000","last_active":"2023-10-19T12:44:24.260099+0000","last_peered":"2023-10-19T12:44:24.260099+0000","last_clean":"2023-10-19T12:44:24.260099+0000","last_became_active":"2023-10-19T12:44:23.261044+0000","last_became_peered":"2023-10-19T12:44:23.261044+0000","last_unstale":"2023-10-19T12:44:24.260099+0000","last_undegraded":"2023-10-19T12:44:24.260099+0000","last_fullsized":"2023-10-19T12:44:24.260099+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:27:47.598684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.113455+0000","last_change":"2023-10-19T12:44:19.090117+0000","last_active":"2023-10-19T12:44:23.113455+0000","last_peered":"2023-10-19T12:44:23.113455+0000","last_clean":"2023-10-19T12:44:23.113455+0000","last_became_active":"2023-10-19T12:44:19.089829+0000","last_became_peered":"2023-10-19T12:44:19.089829+0000","last_unstale":"2023-10-19T12:44:23.113455+0000","last_undegraded":"2023-10-19T12:44:23.113455+0000","last_fullsized":"2023-10-19T12:44:23.113455+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:07:41.515848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.113821+0000","last_change":"2023-10-19T12:44:18.067372+0000","last_active":"2023-10-19T12:44:23.113821+0000","last_peered":"2023-10-19T12:44:23.113821+0000","last_clean":"2023-10-19T12:44:23.113821+0000","last_became_active":"2023-10-19T12:44:18.066991+0000","last_became_peered":"2023-10-19T12:44:18.066991+0000","last_unstale":"2023-10-19T12:44:23.113821+0000","last_undegraded":"2023-10-19T12:44:23.113821+0000","last_fullsized":"2023-10-19T12:44:23.113821+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:44:58.807931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.124117+0000","last_change":"2023-10-19T12:44:23.262285+0000","last_active":"2023-10-19T12:44:24.124117+0000","last_peered":"2023-10-19T12:44:24.124117+0000","last_clean":"2023-10-19T12:44:24.124117+0000","last_became_active":"2023-10-19T12:44:23.261716+0000","last_became_peered":"2023-10-19T12:44:23.261716+0000","last_unstale":"2023-10-19T12:44:24.124117+0000","last_undegraded":"2023-10-19T12:44:24.124117+0000","last_fullsized":"2023-10-19T12:44:24.124117+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:33:24.282692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.259803+0000","last_change":"2023-10-19T12:44:19.090802+0000","last_active":"2023-10-19T12:44:23.259803+0000","last_peered":"2023-10-19T12:44:23.259803+0000","last_clean":"2023-10-19T12:44:23.259803+0000","last_became_active":"2023-10-19T12:44:19.087107+0000","last_became_peered":"2023-10-19T12:44:19.087107+0000","last_unstale":"2023-10-19T12:44:23.259803+0000","last_undegraded":"2023-10-19T12:44:23.259803+0000","last_fullsized":"2023-10-19T12:44:23.259803+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:59:45.926427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"25'9","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:28.505371+0000","last_change":"2023-10-19T12:44:18.067624+0000","last_active":"2023-10-19T12:44:28.505371+0000","last_peered":"2023-10-19T12:44:28.505371+0000","last_clean":"2023-10-19T12:44:28.505371+0000","last_became_active":"2023-10-19T12:44:18.067343+0000","last_became_peered":"2023-10-19T12:44:18.067343+0000","last_unstale":"2023-10-19T12:44:28.505371+0000","last_undegraded":"2023-10-19T12:44:28.505371+0000","last_fullsized":"2023-10-19T12:44:28.505371+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:43:47.410870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3841,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.125150+0000","last_change":"2023-10-19T12:44:23.126083+0000","last_active":"2023-10-19T12:44:24.125150+0000","last_peered":"2023-10-19T12:44:24.125150+0000","last_clean":"2023-10-19T12:44:24.125150+0000","last_became_active":"2023-10-19T12:44:23.125365+0000","last_became_peered":"2023-10-19T12:44:23.125365+0000","last_unstale":"2023-10-19T12:44:24.125150+0000","last_undegraded":"2023-10-19T12:44:24.125150+0000","last_fullsized":"2023-10-19T12:44:24.125150+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:40:13.396617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.114883+0000","last_change":"2023-10-19T12:44:19.091229+0000","last_active":"2023-10-19T12:44:23.114883+0000","last_peered":"2023-10-19T12:44:23.114883+0000","last_clean":"2023-10-19T12:44:23.114883+0000","last_became_active":"2023-10-19T12:44:19.090912+0000","last_became_peered":"2023-10-19T12:44:19.090912+0000","last_unstale":"2023-10-19T12:44:23.114883+0000","last_undegraded":"2023-10-19T12:44:23.114883+0000","last_fullsized":"2023-10-19T12:44:23.114883+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:46:39.634619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.123572+0000","last_change":"2023-10-19T12:44:18.072463+0000","last_active":"2023-10-19T12:44:23.123572+0000","last_peered":"2023-10-19T12:44:23.123572+0000","last_clean":"2023-10-19T12:44:23.123572+0000","last_became_active":"2023-10-19T12:44:18.072156+0000","last_became_peered":"2023-10-19T12:44:18.072156+0000","last_unstale":"2023-10-19T12:44:23.123572+0000","last_undegraded":"2023-10-19T12:44:23.123572+0000","last_fullsized":"2023-10-19T12:44:23.123572+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:44:58.807931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.124138+0000","last_change":"2023-10-19T12:44:23.130655+0000","last_active":"2023-10-19T12:44:24.124138+0000","last_peered":"2023-10-19T12:44:24.124138+0000","last_clean":"2023-10-19T12:44:24.124138+0000","last_became_active":"2023-10-19T12:44:23.130142+0000","last_became_peered":"2023-10-19T12:44:23.130142+0000","last_unstale":"2023-10-19T12:44:24.124138+0000","last_undegraded":"2023-10-19T12:44:24.124138+0000","last_fullsized":"2023-10-19T12:44:24.124138+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:30:09.977362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.113707+0000","last_change":"2023-10-19T12:44:19.084143+0000","last_active":"2023-10-19T12:44:23.113707+0000","last_peered":"2023-10-19T12:44:23.113707+0000","last_clean":"2023-10-19T12:44:23.113707+0000","last_became_active":"2023-10-19T12:44:19.083824+0000","last_became_peered":"2023-10-19T12:44:19.083824+0000","last_unstale":"2023-10-19T12:44:23.113707+0000","last_undegraded":"2023-10-19T12:44:23.113707+0000","last_fullsized":"2023-10-19T12:44:23.113707+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:01:32.286452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.106975+0000","last_change":"2023-10-19T12:44:18.070592+0000","last_active":"2023-10-19T12:44:23.106975+0000","last_peered":"2023-10-19T12:44:23.106975+0000","last_clean":"2023-10-19T12:44:23.106975+0000","last_became_active":"2023-10-19T12:44:18.070297+0000","last_became_peered":"2023-10-19T12:44:18.070297+0000","last_unstale":"2023-10-19T12:44:23.106975+0000","last_undegraded":"2023-10-19T12:44:23.106975+0000","last_fullsized":"2023-10-19T12:44:23.106975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:24:31.294273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.258600+0000","last_change":"2023-10-19T12:44:23.131766+0000","last_active":"2023-10-19T12:44:24.258600+0000","last_peered":"2023-10-19T12:44:24.258600+0000","last_clean":"2023-10-19T12:44:24.258600+0000","last_became_active":"2023-10-19T12:44:23.131349+0000","last_became_peered":"2023-10-19T12:44:23.131349+0000","last_unstale":"2023-10-19T12:44:24.258600+0000","last_undegraded":"2023-10-19T12:44:24.258600+0000","last_fullsized":"2023-10-19T12:44:24.258600+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:55:55.981359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.106934+0000","last_change":"2023-10-19T12:44:19.089065+0000","last_active":"2023-10-19T12:44:23.106934+0000","last_peered":"2023-10-19T12:44:23.106934+0000","last_clean":"2023-10-19T12:44:23.106934+0000","last_became_active":"2023-10-19T12:44:19.088295+0000","last_became_peered":"2023-10-19T12:44:19.088295+0000","last_unstale":"2023-10-19T12:44:23.106934+0000","last_undegraded":"2023-10-19T12:44:23.106934+0000","last_fullsized":"2023-10-19T12:44:23.106934+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:30:05.930806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.116850+0000","last_change":"2023-10-19T12:44:18.071630+0000","last_active":"2023-10-19T12:44:23.116850+0000","last_peered":"2023-10-19T12:44:23.116850+0000","last_clean":"2023-10-19T12:44:23.116850+0000","last_became_active":"2023-10-19T12:44:18.071374+0000","last_became_peered":"2023-10-19T12:44:18.071374+0000","last_unstale":"2023-10-19T12:44:23.116850+0000","last_undegraded":"2023-10-19T12:44:23.116850+0000","last_fullsized":"2023-10-19T12:44:23.116850+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:11:21.842323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.259022+0000","last_change":"2023-10-19T12:44:23.261083+0000","last_active":"2023-10-19T12:44:24.259022+0000","last_peered":"2023-10-19T12:44:24.259022+0000","last_clean":"2023-10-19T12:44:24.259022+0000","last_became_active":"2023-10-19T12:44:23.260643+0000","last_became_peered":"2023-10-19T12:44:23.260643+0000","last_unstale":"2023-10-19T12:44:24.259022+0000","last_undegraded":"2023-10-19T12:44:24.259022+0000","last_fullsized":"2023-10-19T12:44:24.259022+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:25:26.010997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.259319+0000","last_change":"2023-10-19T12:44:23.263298+0000","last_active":"2023-10-19T12:44:24.259319+0000","last_peered":"2023-10-19T12:44:24.259319+0000","last_clean":"2023-10-19T12:44:24.259319+0000","last_became_active":"2023-10-19T12:44:23.262936+0000","last_became_peered":"2023-10-19T12:44:23.262936+0000","last_unstale":"2023-10-19T12:44:24.259319+0000","last_undegraded":"2023-10-19T12:44:24.259319+0000","last_fullsized":"2023-10-19T12:44:24.259319+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:40:13.396617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.122817+0000","last_change":"2023-10-19T12:44:19.089929+0000","last_active":"2023-10-19T12:44:23.122817+0000","last_peered":"2023-10-19T12:44:23.122817+0000","last_clean":"2023-10-19T12:44:23.122817+0000","last_became_active":"2023-10-19T12:44:19.089601+0000","last_became_peered":"2023-10-19T12:44:19.089601+0000","last_unstale":"2023-10-19T12:44:23.122817+0000","last_undegraded":"2023-10-19T12:44:23.122817+0000","last_fullsized":"2023-10-19T12:44:23.122817+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:22:48.368572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.107187+0000","last_change":"2023-10-19T12:44:18.069653+0000","last_active":"2023-10-19T12:44:23.107187+0000","last_peered":"2023-10-19T12:44:23.107187+0000","last_clean":"2023-10-19T12:44:23.107187+0000","last_became_active":"2023-10-19T12:44:18.068221+0000","last_became_peered":"2023-10-19T12:44:18.068221+0000","last_unstale":"2023-10-19T12:44:23.107187+0000","last_undegraded":"2023-10-19T12:44:23.107187+0000","last_fullsized":"2023-10-19T12:44:23.107187+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:54:37.033219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.258743+0000","last_change":"2023-10-19T12:44:23.131662+0000","last_active":"2023-10-19T12:44:24.258743+0000","last_peered":"2023-10-19T12:44:24.258743+0000","last_clean":"2023-10-19T12:44:24.258743+0000","last_became_active":"2023-10-19T12:44:23.131182+0000","last_became_peered":"2023-10-19T12:44:23.131182+0000","last_unstale":"2023-10-19T12:44:24.258743+0000","last_undegraded":"2023-10-19T12:44:24.258743+0000","last_fullsized":"2023-10-19T12:44:24.258743+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:37:44.936303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.109433+0000","last_change":"2023-10-19T12:44:19.081926+0000","last_active":"2023-10-19T12:44:23.109433+0000","last_peered":"2023-10-19T12:44:23.109433+0000","last_clean":"2023-10-19T12:44:23.109433+0000","last_became_active":"2023-10-19T12:44:19.081619+0000","last_became_peered":"2023-10-19T12:44:19.081619+0000","last_unstale":"2023-10-19T12:44:23.109433+0000","last_undegraded":"2023-10-19T12:44:23.109433+0000","last_fullsized":"2023-10-19T12:44:23.109433+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:22:48.368572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.108872+0000","last_change":"2023-10-19T12:44:18.071670+0000","last_active":"2023-10-19T12:44:23.108872+0000","last_peered":"2023-10-19T12:44:23.108872+0000","last_clean":"2023-10-19T12:44:23.108872+0000","last_became_active":"2023-10-19T12:44:18.071370+0000","last_became_peered":"2023-10-19T12:44:18.071370+0000","last_unstale":"2023-10-19T12:44:23.108872+0000","last_undegraded":"2023-10-19T12:44:23.108872+0000","last_fullsized":"2023-10-19T12:44:23.108872+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:15:18.575278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.116496+0000","last_change":"2023-10-19T12:44:18.068200+0000","last_active":"2023-10-19T12:44:23.116496+0000","last_peered":"2023-10-19T12:44:23.116496+0000","last_clean":"2023-10-19T12:44:23.116496+0000","last_became_active":"2023-10-19T12:44:18.067903+0000","last_became_peered":"2023-10-19T12:44:18.067903+0000","last_unstale":"2023-10-19T12:44:23.116496+0000","last_undegraded":"2023-10-19T12:44:23.116496+0000","last_fullsized":"2023-10-19T12:44:23.116496+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:27:49.242393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.113730+0000","last_change":"2023-10-19T12:44:19.073697+0000","last_active":"2023-10-19T12:44:23.113730+0000","last_peered":"2023-10-19T12:44:23.113730+0000","last_clean":"2023-10-19T12:44:23.113730+0000","last_became_active":"2023-10-19T12:44:19.073366+0000","last_became_peered":"2023-10-19T12:44:19.073366+0000","last_unstale":"2023-10-19T12:44:23.113730+0000","last_undegraded":"2023-10-19T12:44:23.113730+0000","last_fullsized":"2023-10-19T12:44:23.113730+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:46:55.355893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.258799+0000","last_change":"2023-10-19T12:44:23.131575+0000","last_active":"2023-10-19T12:44:24.258799+0000","last_peered":"2023-10-19T12:44:24.258799+0000","last_clean":"2023-10-19T12:44:24.258799+0000","last_became_active":"2023-10-19T12:44:23.131125+0000","last_became_peered":"2023-10-19T12:44:23.131125+0000","last_unstale":"2023-10-19T12:44:24.258799+0000","last_undegraded":"2023-10-19T12:44:24.258799+0000","last_fullsized":"2023-10-19T12:44:24.258799+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:15:23.625806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.108702+0000","last_change":"2023-10-19T12:44:18.071298+0000","last_active":"2023-10-19T12:44:23.108702+0000","last_peered":"2023-10-19T12:44:23.108702+0000","last_clean":"2023-10-19T12:44:23.108702+0000","last_became_active":"2023-10-19T12:44:18.070913+0000","last_became_peered":"2023-10-19T12:44:18.070913+0000","last_unstale":"2023-10-19T12:44:23.108702+0000","last_undegraded":"2023-10-19T12:44:23.108702+0000","last_fullsized":"2023-10-19T12:44:23.108702+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:27:49.242393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.115487+0000","last_change":"2023-10-19T12:44:19.085944+0000","last_active":"2023-10-19T12:44:23.115487+0000","last_peered":"2023-10-19T12:44:23.115487+0000","last_clean":"2023-10-19T12:44:23.115487+0000","last_became_active":"2023-10-19T12:44:19.085621+0000","last_became_peered":"2023-10-19T12:44:19.085621+0000","last_unstale":"2023-10-19T12:44:23.115487+0000","last_undegraded":"2023-10-19T12:44:23.115487+0000","last_fullsized":"2023-10-19T12:44:23.115487+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:12:59.097861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.260011+0000","last_change":"2023-10-19T12:44:23.126882+0000","last_active":"2023-10-19T12:44:24.260011+0000","last_peered":"2023-10-19T12:44:24.260011+0000","last_clean":"2023-10-19T12:44:24.260011+0000","last_became_active":"2023-10-19T12:44:23.126421+0000","last_became_peered":"2023-10-19T12:44:23.126421+0000","last_unstale":"2023-10-19T12:44:24.260011+0000","last_undegraded":"2023-10-19T12:44:24.260011+0000","last_fullsized":"2023-10-19T12:44:24.260011+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:32:54.650601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.114291+0000","last_change":"2023-10-19T12:44:19.080194+0000","last_active":"2023-10-19T12:44:23.114291+0000","last_peered":"2023-10-19T12:44:23.114291+0000","last_clean":"2023-10-19T12:44:23.114291+0000","last_became_active":"2023-10-19T12:44:19.079941+0000","last_became_peered":"2023-10-19T12:44:19.079941+0000","last_unstale":"2023-10-19T12:44:23.114291+0000","last_undegraded":"2023-10-19T12:44:23.114291+0000","last_fullsized":"2023-10-19T12:44:23.114291+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:50:42.817437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.107859+0000","last_change":"2023-10-19T12:44:19.098110+0000","last_active":"2023-10-19T12:44:23.107859+0000","last_peered":"2023-10-19T12:44:23.107859+0000","last_clean":"2023-10-19T12:44:23.107859+0000","last_became_active":"2023-10-19T12:44:19.097774+0000","last_became_peered":"2023-10-19T12:44:19.097774+0000","last_unstale":"2023-10-19T12:44:23.107859+0000","last_undegraded":"2023-10-19T12:44:23.107859+0000","last_fullsized":"2023-10-19T12:44:23.107859+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:24:32.298244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.106333+0000","last_change":"2023-10-19T12:44:19.083631+0000","last_active":"2023-10-19T12:44:23.106333+0000","last_peered":"2023-10-19T12:44:23.106333+0000","last_clean":"2023-10-19T12:44:23.106333+0000","last_became_active":"2023-10-19T12:44:19.082464+0000","last_became_peered":"2023-10-19T12:44:19.082464+0000","last_unstale":"2023-10-19T12:44:23.106333+0000","last_undegraded":"2023-10-19T12:44:23.106333+0000","last_fullsized":"2023-10-19T12:44:23.106333+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:40:09.350060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.113640+0000","last_change":"2023-10-19T12:44:19.083390+0000","last_active":"2023-10-19T12:44:23.113640+0000","last_peered":"2023-10-19T12:44:23.113640+0000","last_clean":"2023-10-19T12:44:23.113640+0000","last_became_active":"2023-10-19T12:44:19.082423+0000","last_became_peered":"2023-10-19T12:44:19.082423+0000","last_unstale":"2023-10-19T12:44:23.113640+0000","last_undegraded":"2023-10-19T12:44:23.113640+0000","last_fullsized":"2023-10-19T12:44:23.113640+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:54:38.037190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.110543+0000","last_change":"2023-10-19T12:44:19.090250+0000","last_active":"2023-10-19T12:44:23.110543+0000","last_peered":"2023-10-19T12:44:23.110543+0000","last_clean":"2023-10-19T12:44:23.110543+0000","last_became_active":"2023-10-19T12:44:19.089587+0000","last_became_peered":"2023-10-19T12:44:19.089587+0000","last_unstale":"2023-10-19T12:44:23.110543+0000","last_undegraded":"2023-10-19T12:44:23.110543+0000","last_fullsized":"2023-10-19T12:44:23.110543+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:46:55.355893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.122832+0000","last_change":"2023-10-19T12:44:19.088635+0000","last_active":"2023-10-19T12:44:23.122832+0000","last_peered":"2023-10-19T12:44:23.122832+0000","last_clean":"2023-10-19T12:44:23.122832+0000","last_became_active":"2023-10-19T12:44:19.088378+0000","last_became_peered":"2023-10-19T12:44:19.088378+0000","last_unstale":"2023-10-19T12:44:23.122832+0000","last_undegraded":"2023-10-19T12:44:23.122832+0000","last_fullsized":"2023-10-19T12:44:23.122832+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:00:15.982705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.115903+0000","last_change":"2023-10-19T12:44:19.089688+0000","last_active":"2023-10-19T12:44:23.115903+0000","last_peered":"2023-10-19T12:44:23.115903+0000","last_clean":"2023-10-19T12:44:23.115903+0000","last_became_active":"2023-10-19T12:44:19.089438+0000","last_became_peered":"2023-10-19T12:44:19.089438+0000","last_unstale":"2023-10-19T12:44:23.115903+0000","last_undegraded":"2023-10-19T12:44:23.115903+0000","last_fullsized":"2023-10-19T12:44:23.115903+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:46:55.355893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.107774+0000","last_change":"2023-10-19T12:44:19.087749+0000","last_active":"2023-10-19T12:44:23.107774+0000","last_peered":"2023-10-19T12:44:23.107774+0000","last_clean":"2023-10-19T12:44:23.107774+0000","last_became_active":"2023-10-19T12:44:19.087020+0000","last_became_peered":"2023-10-19T12:44:19.087020+0000","last_unstale":"2023-10-19T12:44:23.107774+0000","last_undegraded":"2023-10-19T12:44:23.107774+0000","last_fullsized":"2023-10-19T12:44:23.107774+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:05:15.835724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.106909+0000","last_change":"2023-10-19T12:44:19.085297+0000","last_active":"2023-10-19T12:44:23.106909+0000","last_peered":"2023-10-19T12:44:23.106909+0000","last_clean":"2023-10-19T12:44:23.106909+0000","last_became_active":"2023-10-19T12:44:19.084652+0000","last_became_peered":"2023-10-19T12:44:19.084652+0000","last_unstale":"2023-10-19T12:44:23.106909+0000","last_undegraded":"2023-10-19T12:44:23.106909+0000","last_fullsized":"2023-10-19T12:44:23.106909+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:05:15.835724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.108790+0000","last_change":"2023-10-19T12:44:19.084748+0000","last_active":"2023-10-19T12:44:23.108790+0000","last_peered":"2023-10-19T12:44:23.108790+0000","last_clean":"2023-10-19T12:44:23.108790+0000","last_became_active":"2023-10-19T12:44:19.084460+0000","last_became_peered":"2023-10-19T12:44:19.084460+0000","last_unstale":"2023-10-19T12:44:23.108790+0000","last_undegraded":"2023-10-19T12:44:23.108790+0000","last_fullsized":"2023-10-19T12:44:23.108790+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:54:38.037190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.106299+0000","last_change":"2023-10-19T12:44:19.087125+0000","last_active":"2023-10-19T12:44:23.106299+0000","last_peered":"2023-10-19T12:44:23.106299+0000","last_clean":"2023-10-19T12:44:23.106299+0000","last_became_active":"2023-10-19T12:44:19.086845+0000","last_became_peered":"2023-10-19T12:44:19.086845+0000","last_unstale":"2023-10-19T12:44:23.106299+0000","last_undegraded":"2023-10-19T12:44:23.106299+0000","last_fullsized":"2023-10-19T12:44:23.106299+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:37:40.889746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.113534+0000","last_change":"2023-10-19T12:44:19.089064+0000","last_active":"2023-10-19T12:44:23.113534+0000","last_peered":"2023-10-19T12:44:23.113534+0000","last_clean":"2023-10-19T12:44:23.113534+0000","last_became_active":"2023-10-19T12:44:19.088780+0000","last_became_peered":"2023-10-19T12:44:19.088780+0000","last_unstale":"2023-10-19T12:44:23.113534+0000","last_undegraded":"2023-10-19T12:44:23.113534+0000","last_fullsized":"2023-10-19T12:44:23.113534+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:10:08.050707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.115561+0000","last_change":"2023-10-19T12:44:19.087124+0000","last_active":"2023-10-19T12:44:23.115561+0000","last_peered":"2023-10-19T12:44:23.115561+0000","last_clean":"2023-10-19T12:44:23.115561+0000","last_became_active":"2023-10-19T12:44:19.086851+0000","last_became_peered":"2023-10-19T12:44:19.086851+0000","last_unstale":"2023-10-19T12:44:23.115561+0000","last_undegraded":"2023-10-19T12:44:23.115561+0000","last_fullsized":"2023-10-19T12:44:23.115561+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:10:08.050707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.260204+0000","last_change":"2023-10-19T12:44:19.091368+0000","last_active":"2023-10-19T12:44:23.260204+0000","last_peered":"2023-10-19T12:44:23.260204+0000","last_clean":"2023-10-19T12:44:23.260204+0000","last_became_active":"2023-10-19T12:44:19.091052+0000","last_became_peered":"2023-10-19T12:44:19.091052+0000","last_unstale":"2023-10-19T12:44:23.260204+0000","last_undegraded":"2023-10-19T12:44:23.260204+0000","last_fullsized":"2023-10-19T12:44:23.260204+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:40:09.350060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.108300+0000","last_change":"2023-10-19T12:44:19.080430+0000","last_active":"2023-10-19T12:44:23.108300+0000","last_peered":"2023-10-19T12:44:23.108300+0000","last_clean":"2023-10-19T12:44:23.108300+0000","last_became_active":"2023-10-19T12:44:19.079493+0000","last_became_peered":"2023-10-19T12:44:19.079493+0000","last_unstale":"2023-10-19T12:44:23.108300+0000","last_undegraded":"2023-10-19T12:44:23.108300+0000","last_fullsized":"2023-10-19T12:44:23.108300+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:46:39.634619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.107179+0000","last_change":"2023-10-19T12:44:19.088751+0000","last_active":"2023-10-19T12:44:23.107179+0000","last_peered":"2023-10-19T12:44:23.107179+0000","last_clean":"2023-10-19T12:44:23.107179+0000","last_became_active":"2023-10-19T12:44:19.088430+0000","last_became_peered":"2023-10-19T12:44:19.088430+0000","last_unstale":"2023-10-19T12:44:23.107179+0000","last_undegraded":"2023-10-19T12:44:23.107179+0000","last_fullsized":"2023-10-19T12:44:23.107179+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:24:48.423014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.106786+0000","last_change":"2023-10-19T12:44:19.082935+0000","last_active":"2023-10-19T12:44:23.106786+0000","last_peered":"2023-10-19T12:44:23.106786+0000","last_clean":"2023-10-19T12:44:23.106786+0000","last_became_active":"2023-10-19T12:44:19.082692+0000","last_became_peered":"2023-10-19T12:44:19.082692+0000","last_unstale":"2023-10-19T12:44:23.106786+0000","last_undegraded":"2023-10-19T12:44:23.106786+0000","last_fullsized":"2023-10-19T12:44:23.106786+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:10:08.050707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.122913+0000","last_change":"2023-10-19T12:44:19.087441+0000","last_active":"2023-10-19T12:44:23.122913+0000","last_peered":"2023-10-19T12:44:23.122913+0000","last_clean":"2023-10-19T12:44:23.122913+0000","last_became_active":"2023-10-19T12:44:19.087147+0000","last_became_peered":"2023-10-19T12:44:19.087147+0000","last_unstale":"2023-10-19T12:44:23.122913+0000","last_undegraded":"2023-10-19T12:44:23.122913+0000","last_fullsized":"2023-10-19T12:44:23.122913+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:50:42.817437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.113370+0000","last_change":"2023-10-19T12:44:19.085136+0000","last_active":"2023-10-19T12:44:23.113370+0000","last_peered":"2023-10-19T12:44:23.113370+0000","last_clean":"2023-10-19T12:44:23.113370+0000","last_became_active":"2023-10-19T12:44:19.084856+0000","last_became_peered":"2023-10-19T12:44:19.084856+0000","last_unstale":"2023-10-19T12:44:23.113370+0000","last_undegraded":"2023-10-19T12:44:23.113370+0000","last_fullsized":"2023-10-19T12:44:23.113370+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:46:30.889830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.109109+0000","last_change":"2023-10-19T12:44:19.080419+0000","last_active":"2023-10-19T12:44:23.109109+0000","last_peered":"2023-10-19T12:44:23.109109+0000","last_clean":"2023-10-19T12:44:23.109109+0000","last_became_active":"2023-10-19T12:44:19.080123+0000","last_became_peered":"2023-10-19T12:44:19.080123+0000","last_unstale":"2023-10-19T12:44:23.109109+0000","last_undegraded":"2023-10-19T12:44:23.109109+0000","last_fullsized":"2023-10-19T12:44:23.109109+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:10:08.050707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.114806+0000","last_change":"2023-10-19T12:44:19.080702+0000","last_active":"2023-10-19T12:44:23.114806+0000","last_peered":"2023-10-19T12:44:23.114806+0000","last_clean":"2023-10-19T12:44:23.114806+0000","last_became_active":"2023-10-19T12:44:19.079347+0000","last_became_peered":"2023-10-19T12:44:19.079347+0000","last_unstale":"2023-10-19T12:44:23.114806+0000","last_undegraded":"2023-10-19T12:44:23.114806+0000","last_fullsized":"2023-10-19T12:44:23.114806+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:43:48.414841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.115372+0000","last_change":"2023-10-19T12:44:19.081326+0000","last_active":"2023-10-19T12:44:23.115372+0000","last_peered":"2023-10-19T12:44:23.115372+0000","last_clean":"2023-10-19T12:44:23.115372+0000","last_became_active":"2023-10-19T12:44:19.081063+0000","last_became_peered":"2023-10-19T12:44:19.081063+0000","last_unstale":"2023-10-19T12:44:23.115372+0000","last_undegraded":"2023-10-19T12:44:23.115372+0000","last_fullsized":"2023-10-19T12:44:23.115372+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:00:15.982705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.115814+0000","last_change":"2023-10-19T12:44:19.094478+0000","last_active":"2023-10-19T12:44:23.115814+0000","last_peered":"2023-10-19T12:44:23.115814+0000","last_clean":"2023-10-19T12:44:23.115814+0000","last_became_active":"2023-10-19T12:44:19.094217+0000","last_became_peered":"2023-10-19T12:44:19.094217+0000","last_unstale":"2023-10-19T12:44:23.115814+0000","last_undegraded":"2023-10-19T12:44:23.115814+0000","last_fullsized":"2023-10-19T12:44:23.115814+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:01:32.286452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.260360+0000","last_change":"2023-10-19T12:44:19.093197+0000","last_active":"2023-10-19T12:44:23.260360+0000","last_peered":"2023-10-19T12:44:23.260360+0000","last_clean":"2023-10-19T12:44:23.260360+0000","last_became_active":"2023-10-19T12:44:19.092926+0000","last_became_peered":"2023-10-19T12:44:19.092926+0000","last_unstale":"2023-10-19T12:44:23.260360+0000","last_undegraded":"2023-10-19T12:44:23.260360+0000","last_fullsized":"2023-10-19T12:44:23.260360+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:30:05.930806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.109339+0000","last_change":"2023-10-19T12:44:19.095093+0000","last_active":"2023-10-19T12:44:23.109339+0000","last_peered":"2023-10-19T12:44:23.109339+0000","last_clean":"2023-10-19T12:44:23.109339+0000","last_became_active":"2023-10-19T12:44:19.094816+0000","last_became_peered":"2023-10-19T12:44:19.094816+0000","last_unstale":"2023-10-19T12:44:23.109339+0000","last_undegraded":"2023-10-19T12:44:23.109339+0000","last_fullsized":"2023-10-19T12:44:23.109339+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:07:41.515848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.259584+0000","last_change":"2023-10-19T12:44:19.086048+0000","last_active":"2023-10-19T12:44:23.259584+0000","last_peered":"2023-10-19T12:44:23.259584+0000","last_clean":"2023-10-19T12:44:23.259584+0000","last_became_active":"2023-10-19T12:44:19.085086+0000","last_became_peered":"2023-10-19T12:44:19.085086+0000","last_unstale":"2023-10-19T12:44:23.259584+0000","last_undegraded":"2023-10-19T12:44:23.259584+0000","last_fullsized":"2023-10-19T12:44:23.259584+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:05:15.835724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.106214+0000","last_change":"2023-10-19T12:44:19.086341+0000","last_active":"2023-10-19T12:44:23.106214+0000","last_peered":"2023-10-19T12:44:23.106214+0000","last_clean":"2023-10-19T12:44:23.106214+0000","last_became_active":"2023-10-19T12:44:19.083365+0000","last_became_peered":"2023-10-19T12:44:19.083365+0000","last_unstale":"2023-10-19T12:44:23.106214+0000","last_undegraded":"2023-10-19T12:44:23.106214+0000","last_fullsized":"2023-10-19T12:44:23.106214+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:46:55.355893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.259588+0000","last_change":"2023-10-19T12:44:19.091228+0000","last_active":"2023-10-19T12:44:23.259588+0000","last_peered":"2023-10-19T12:44:23.259588+0000","last_clean":"2023-10-19T12:44:23.259588+0000","last_became_active":"2023-10-19T12:44:19.087203+0000","last_became_peered":"2023-10-19T12:44:19.087203+0000","last_unstale":"2023-10-19T12:44:23.259588+0000","last_undegraded":"2023-10-19T12:44:23.259588+0000","last_fullsized":"2023-10-19T12:44:23.259588+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:29:52.686703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.106856+0000","last_change":"2023-10-19T12:44:19.086112+0000","last_active":"2023-10-19T12:44:23.106856+0000","last_peered":"2023-10-19T12:44:23.106856+0000","last_clean":"2023-10-19T12:44:23.106856+0000","last_became_active":"2023-10-19T12:44:19.085822+0000","last_became_peered":"2023-10-19T12:44:19.085822+0000","last_unstale":"2023-10-19T12:44:23.106856+0000","last_undegraded":"2023-10-19T12:44:23.106856+0000","last_fullsized":"2023-10-19T12:44:23.106856+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:01:32.286452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.113429+0000","last_change":"2023-10-19T12:44:19.078205+0000","last_active":"2023-10-19T12:44:23.113429+0000","last_peered":"2023-10-19T12:44:23.113429+0000","last_clean":"2023-10-19T12:44:23.113429+0000","last_became_active":"2023-10-19T12:44:19.077824+0000","last_became_peered":"2023-10-19T12:44:19.077824+0000","last_unstale":"2023-10-19T12:44:23.113429+0000","last_undegraded":"2023-10-19T12:44:23.113429+0000","last_fullsized":"2023-10-19T12:44:23.113429+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:50:42.817437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.123225+0000","last_change":"2023-10-19T12:44:19.084212+0000","last_active":"2023-10-19T12:44:23.123225+0000","last_peered":"2023-10-19T12:44:23.123225+0000","last_clean":"2023-10-19T12:44:23.123225+0000","last_became_active":"2023-10-19T12:44:19.083928+0000","last_became_peered":"2023-10-19T12:44:19.083928+0000","last_unstale":"2023-10-19T12:44:23.123225+0000","last_undegraded":"2023-10-19T12:44:23.123225+0000","last_fullsized":"2023-10-19T12:44:23.123225+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:10:08.050707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.260753+0000","last_change":"2023-10-19T12:44:19.087938+0000","last_active":"2023-10-19T12:44:23.260753+0000","last_peered":"2023-10-19T12:44:23.260753+0000","last_clean":"2023-10-19T12:44:23.260753+0000","last_became_active":"2023-10-19T12:44:19.087552+0000","last_became_peered":"2023-10-19T12:44:19.087552+0000","last_unstale":"2023-10-19T12:44:23.260753+0000","last_undegraded":"2023-10-19T12:44:23.260753+0000","last_fullsized":"2023-10-19T12:44:23.260753+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:24:48.423014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4639,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574859,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27868,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343972,"statfs":{"total":96636764160,"available":96608227328,"internally_reserved":0,"allocated":864256,"data_stored":695401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78500000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71299999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69899999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76600000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39500000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42999999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53800000000000003}]}]},{"osd":6,"up_from":13,"seq":55834574859,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":108292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54600000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71999999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57399999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67600000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.628}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.495}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73799999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574859,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":270336,"data_stored":104541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39200000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47099999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65900000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60299999999999998}]}]},{"osd":4,"up_from":13,"seq":55834574859,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":278528,"data_stored":104577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56000000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60999999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57999999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45300000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41399999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.43099999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574858,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31448,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340392,"statfs":{"total":96636764160,"available":96604561408,"internally_reserved":0,"allocated":278528,"data_stored":108292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27400000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33500000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39900000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59699999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61299999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68799999999999994}]}]},{"osd":2,"up_from":13,"seq":55834574860,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31448,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340392,"statfs":{"total":96636764160,"available":96604561408,"internally_reserved":0,"allocated":266240,"data_stored":104451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39100000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36899999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.435}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.504}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55700000000000005}]}]},{"osd":1,"up_from":13,"seq":55834574859,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31460,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340380,"statfs":{"total":96636764160,"available":96604549120,"internally_reserved":0,"allocated":274432,"data_stored":104611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.219}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.255}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35199999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70199999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52600000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64200000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71599999999999997}]}]},{"osd":0,"up_from":13,"seq":55834574860,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32032,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339808,"statfs":{"total":96636764160,"available":96603963392,"internally_reserved":0,"allocated":868352,"data_stored":694838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.219}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.318}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30099999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57899999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54300000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.502}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-19T12:44:46.964 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-19T12:44:47.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.104+0000 7f4e43421640 1 -- 172.21.15.57:0/1611597881 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e3c0c9dc0 msgr2=0x7f4e3c06b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:47.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.104+0000 7f4e43421640 1 --2- 172.21.15.57:0/1611597881 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e3c0c9dc0 0x7f4e3c06b2e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f4e30009830 tx=0x7f4e3002bd60 comp rx=0 tx=0).stop 2023-10-19T12:44:47.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.104+0000 7f4e43421640 1 -- 172.21.15.57:0/1611597881 shutdown_connections 2023-10-19T12:44:47.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.104+0000 7f4e43421640 1 --2- 172.21.15.57:0/1611597881 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4e3c0ab500 0x7f4e3c0ab990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:47.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.104+0000 7f4e43421640 1 --2- 172.21.15.57:0/1611597881 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e3c0c9dc0 0x7f4e3c06b2e0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:47.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.104+0000 7f4e43421640 1 -- 172.21.15.57:0/1611597881 >> 172.21.15.57:0/1611597881 conn(0x7f4e3c069930 msgr2=0x7f4e3c069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:47.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.104+0000 7f4e43421640 1 -- 172.21.15.57:0/1611597881 shutdown_connections 2023-10-19T12:44:47.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.104+0000 7f4e43421640 1 -- 172.21.15.57:0/1611597881 wait complete. 2023-10-19T12:44:47.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.105+0000 7f4e43421640 1 Processor -- start 2023-10-19T12:44:47.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.105+0000 7f4e43421640 1 -- start start 2023-10-19T12:44:47.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.106+0000 7f4e43421640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4e3c06af00 0x7f4e3c0b12a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:47.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.106+0000 7f4e43421640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4e3c0ab500 0x7f4e3c1f1350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:47.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.106+0000 7f4e43421640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e3c0c9dc0 0x7f4e3c1f4260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:47.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.106+0000 7f4e43421640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e3c13ada0 con 0x7f4e3c0c9dc0 2023-10-19T12:44:47.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.106+0000 7f4e43421640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e3c1503d0 con 0x7f4e3c06af00 2023-10-19T12:44:47.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.106+0000 7f4e43421640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e3c070650 con 0x7f4e3c0ab500 2023-10-19T12:44:47.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.106+0000 7f4e419be640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e3c0c9dc0 0x7f4e3c1f4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:47.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.106+0000 7f4e419be640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e3c0c9dc0 0x7f4e3c1f4260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59146/0 (socket says 172.21.15.57:59146) 2023-10-19T12:44:47.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.106+0000 7f4e419be640 1 -- 172.21.15.57:0/2470261561 learned_addr learned my addr 172.21.15.57:0/2470261561 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:47.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.106+0000 7f4e409bc640 1 --2- 172.21.15.57:0/2470261561 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4e3c0ab500 0x7f4e3c1f1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:47.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.106+0000 7f4e411bd640 1 --2- 172.21.15.57:0/2470261561 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4e3c06af00 0x7f4e3c0b12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:47.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.106+0000 7f4e419be640 1 -- 172.21.15.57:0/2470261561 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4e3c0ab500 msgr2=0x7f4e3c1f1350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:47.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.106+0000 7f4e419be640 1 --2- 172.21.15.57:0/2470261561 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4e3c0ab500 0x7f4e3c1f1350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:47.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.106+0000 7f4e419be640 1 -- 172.21.15.57:0/2470261561 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4e3c06af00 msgr2=0x7f4e3c0b12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:47.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.106+0000 7f4e419be640 1 --2- 172.21.15.57:0/2470261561 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4e3c06af00 0x7f4e3c0b12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:47.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.106+0000 7f4e419be640 1 -- 172.21.15.57:0/2470261561 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e30009490 con 0x7f4e3c0c9dc0 2023-10-19T12:44:47.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.107+0000 7f4e419be640 1 --2- 172.21.15.57:0/2470261561 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e3c0c9dc0 0x7f4e3c1f4260 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c00eaa0 tx=0x7f4e2c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:47.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.107+0000 7f4e39ffb640 1 -- 172.21.15.57:0/2470261561 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e2c00cdb0 con 0x7f4e3c0c9dc0 2023-10-19T12:44:47.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.107+0000 7f4e39ffb640 1 -- 172.21.15.57:0/2470261561 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f4e2c004b20 con 0x7f4e3c0c9dc0 2023-10-19T12:44:47.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.107+0000 7f4e43421640 1 -- 172.21.15.57:0/2470261561 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e3c070370 con 0x7f4e3c0c9dc0 2023-10-19T12:44:47.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.107+0000 7f4e39ffb640 1 -- 172.21.15.57:0/2470261561 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e2c005020 con 0x7f4e3c0c9dc0 2023-10-19T12:44:47.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.107+0000 7f4e43421640 1 -- 172.21.15.57:0/2470261561 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e3c150950 con 0x7f4e3c0c9dc0 2023-10-19T12:44:47.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.108+0000 7f4e43421640 1 -- 172.21.15.57:0/2470261561 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e3c0690e0 con 0x7f4e3c0c9dc0 2023-10-19T12:44:47.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.108+0000 7f4e39ffb640 1 -- 172.21.15.57:0/2470261561 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f4e2c005180 con 0x7f4e3c0c9dc0 2023-10-19T12:44:47.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.109+0000 7f4e39ffb640 1 --2- 172.21.15.57:0/2470261561 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f4e2806ae40 0x7f4e2806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:47.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.109+0000 7f4e39ffb640 1 -- 172.21.15.57:0/2470261561 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f4e2c08a3b0 con 0x7f4e3c0c9dc0 2023-10-19T12:44:47.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.109+0000 7f4e411bd640 1 --2- 172.21.15.57:0/2470261561 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f4e2806ae40 0x7f4e2806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:47.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.110+0000 7f4e411bd640 1 --2- 172.21.15.57:0/2470261561 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f4e2806ae40 0x7f4e2806d300 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f4e340046b0 tx=0x7f4e34009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:47.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.113+0000 7f4e39ffb640 1 -- 172.21.15.57:0/2470261561 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4e2c055160 con 0x7f4e3c0c9dc0 2023-10-19T12:44:47.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.289+0000 7f4e43421640 1 -- 172.21.15.57:0/2470261561 --> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4e3c0abed0 con 0x7f4e2806ae40 2023-10-19T12:44:47.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.298+0000 7f4e39ffb640 1 -- 172.21.15.57:0/2470261561 <== mgr.4100 v2:172.21.15.179:6832/2609646285 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+345217 (secure 0 0 0) 0x7f4e34001c70 con 0x7f4e2806ae40 2023-10-19T12:44:47.304 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:44:47.306 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2023-10-19T12:44:47.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.301+0000 7f4e43421640 1 -- 172.21.15.57:0/2470261561 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f4e2806ae40 msgr2=0x7f4e2806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:47.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.301+0000 7f4e43421640 1 --2- 172.21.15.57:0/2470261561 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f4e2806ae40 0x7f4e2806d300 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f4e340046b0 tx=0x7f4e34009040 comp rx=0 tx=0).stop 2023-10-19T12:44:47.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.301+0000 7f4e43421640 1 -- 172.21.15.57:0/2470261561 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e3c0c9dc0 msgr2=0x7f4e3c1f4260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:47.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.301+0000 7f4e43421640 1 --2- 172.21.15.57:0/2470261561 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e3c0c9dc0 0x7f4e3c1f4260 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c00eaa0 tx=0x7f4e2c00ef60 comp rx=0 tx=0).stop 2023-10-19T12:44:47.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.301+0000 7f4e43421640 1 -- 172.21.15.57:0/2470261561 shutdown_connections 2023-10-19T12:44:47.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.301+0000 7f4e43421640 1 --2- 172.21.15.57:0/2470261561 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f4e2806ae40 0x7f4e2806d300 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:47.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.301+0000 7f4e43421640 1 --2- 172.21.15.57:0/2470261561 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e3c0c9dc0 0x7f4e3c1f4260 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:47.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.301+0000 7f4e43421640 1 --2- 172.21.15.57:0/2470261561 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4e3c0ab500 0x7f4e3c1f1350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:47.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.301+0000 7f4e43421640 1 --2- 172.21.15.57:0/2470261561 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4e3c06af00 0x7f4e3c0b12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:47.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.301+0000 7f4e43421640 1 -- 172.21.15.57:0/2470261561 >> 172.21.15.57:0/2470261561 conn(0x7f4e3c069930 msgr2=0x7f4e3c0c9680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:47.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.301+0000 7f4e43421640 1 -- 172.21.15.57:0/2470261561 shutdown_connections 2023-10-19T12:44:47.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.301+0000 7f4e43421640 1 -- 172.21.15.57:0/2470261561 wait complete. 2023-10-19T12:44:47.315 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":38,"stamp":"2023-10-19T12:44:45.320998+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595007,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236140,"kb_used_data":3300,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738580,"statfs":{"total":773094113280,"available":772852305920,"internally_reserved":0,"allocated":3379200,"data_stored":2025003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":238276153},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":141,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003664"},"pg_stats":[{"pgid":"2.1f","version":"20'4","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.113796+0000","last_change":"2023-10-19T12:44:18.071366+0000","last_active":"2023-10-19T12:44:23.113796+0000","last_peered":"2023-10-19T12:44:23.113796+0000","last_clean":"2023-10-19T12:44:23.113796+0000","last_became_active":"2023-10-19T12:44:18.071117+0000","last_became_peered":"2023-10-19T12:44:18.071117+0000","last_unstale":"2023-10-19T12:44:23.113796+0000","last_undegraded":"2023-10-19T12:44:23.113796+0000","last_fullsized":"2023-10-19T12:44:23.113796+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:24:31.294273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.107571+0000","last_change":"2023-10-19T12:44:19.080917+0000","last_active":"2023-10-19T12:44:23.107571+0000","last_peered":"2023-10-19T12:44:23.107571+0000","last_clean":"2023-10-19T12:44:23.107571+0000","last_became_active":"2023-10-19T12:44:19.080408+0000","last_became_peered":"2023-10-19T12:44:19.080408+0000","last_unstale":"2023-10-19T12:44:23.107571+0000","last_undegraded":"2023-10-19T12:44:23.107571+0000","last_fullsized":"2023-10-19T12:44:23.107571+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:29:52.686703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.258880+0000","last_change":"2023-10-19T12:44:23.262149+0000","last_active":"2023-10-19T12:44:24.258880+0000","last_peered":"2023-10-19T12:44:24.258880+0000","last_clean":"2023-10-19T12:44:24.258880+0000","last_became_active":"2023-10-19T12:44:23.261764+0000","last_became_peered":"2023-10-19T12:44:23.261764+0000","last_unstale":"2023-10-19T12:44:24.258880+0000","last_undegraded":"2023-10-19T12:44:24.258880+0000","last_fullsized":"2023-10-19T12:44:24.258880+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:33:24.282692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.107679+0000","last_change":"2023-10-19T12:44:18.064762+0000","last_active":"2023-10-19T12:44:23.107679+0000","last_peered":"2023-10-19T12:44:23.107679+0000","last_clean":"2023-10-19T12:44:23.107679+0000","last_became_active":"2023-10-19T12:44:18.064478+0000","last_became_peered":"2023-10-19T12:44:18.064478+0000","last_unstale":"2023-10-19T12:44:23.107679+0000","last_undegraded":"2023-10-19T12:44:23.107679+0000","last_fullsized":"2023-10-19T12:44:23.107679+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:46:29.885859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.113623+0000","last_change":"2023-10-19T12:44:19.083619+0000","last_active":"2023-10-19T12:44:23.113623+0000","last_peered":"2023-10-19T12:44:23.113623+0000","last_clean":"2023-10-19T12:44:23.113623+0000","last_became_active":"2023-10-19T12:44:19.083330+0000","last_became_peered":"2023-10-19T12:44:19.083330+0000","last_unstale":"2023-10-19T12:44:23.113623+0000","last_undegraded":"2023-10-19T12:44:23.113623+0000","last_fullsized":"2023-10-19T12:44:23.113623+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:00:15.982705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.123057+0000","last_change":"2023-10-19T12:44:23.132517+0000","last_active":"2023-10-19T12:44:24.123057+0000","last_peered":"2023-10-19T12:44:24.123057+0000","last_clean":"2023-10-19T12:44:24.123057+0000","last_became_active":"2023-10-19T12:44:23.132066+0000","last_became_peered":"2023-10-19T12:44:23.132066+0000","last_unstale":"2023-10-19T12:44:24.123057+0000","last_undegraded":"2023-10-19T12:44:24.123057+0000","last_fullsized":"2023-10-19T12:44:24.123057+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:46:59.402450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.260017+0000","last_change":"2023-10-19T12:44:18.070873+0000","last_active":"2023-10-19T12:44:23.260017+0000","last_peered":"2023-10-19T12:44:23.260017+0000","last_clean":"2023-10-19T12:44:23.260017+0000","last_became_active":"2023-10-19T12:44:18.070602+0000","last_became_peered":"2023-10-19T12:44:18.070602+0000","last_unstale":"2023-10-19T12:44:23.260017+0000","last_undegraded":"2023-10-19T12:44:23.260017+0000","last_fullsized":"2023-10-19T12:44:23.260017+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:15:18.575278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.106941+0000","last_change":"2023-10-19T12:44:19.075787+0000","last_active":"2023-10-19T12:44:23.106941+0000","last_peered":"2023-10-19T12:44:23.106941+0000","last_clean":"2023-10-19T12:44:23.106941+0000","last_became_active":"2023-10-19T12:44:19.074529+0000","last_became_peered":"2023-10-19T12:44:19.074529+0000","last_unstale":"2023-10-19T12:44:23.106941+0000","last_undegraded":"2023-10-19T12:44:23.106941+0000","last_fullsized":"2023-10-19T12:44:23.106941+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:29:52.686703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.124069+0000","last_change":"2023-10-19T12:44:23.132792+0000","last_active":"2023-10-19T12:44:24.124069+0000","last_peered":"2023-10-19T12:44:24.124069+0000","last_clean":"2023-10-19T12:44:24.124069+0000","last_became_active":"2023-10-19T12:44:23.132099+0000","last_became_peered":"2023-10-19T12:44:23.132099+0000","last_unstale":"2023-10-19T12:44:24.124069+0000","last_undegraded":"2023-10-19T12:44:24.124069+0000","last_fullsized":"2023-10-19T12:44:24.124069+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:01:13.019006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.259778+0000","last_change":"2023-10-19T12:44:18.068582+0000","last_active":"2023-10-19T12:44:23.259778+0000","last_peered":"2023-10-19T12:44:23.259778+0000","last_clean":"2023-10-19T12:44:23.259778+0000","last_became_active":"2023-10-19T12:44:18.068099+0000","last_became_peered":"2023-10-19T12:44:18.068099+0000","last_unstale":"2023-10-19T12:44:23.259778+0000","last_undegraded":"2023-10-19T12:44:23.259778+0000","last_fullsized":"2023-10-19T12:44:23.259778+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:24:31.294273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.107389+0000","last_change":"2023-10-19T12:44:19.076968+0000","last_active":"2023-10-19T12:44:23.107389+0000","last_peered":"2023-10-19T12:44:23.107389+0000","last_clean":"2023-10-19T12:44:23.107389+0000","last_became_active":"2023-10-19T12:44:19.076105+0000","last_became_peered":"2023-10-19T12:44:19.076105+0000","last_unstale":"2023-10-19T12:44:23.107389+0000","last_undegraded":"2023-10-19T12:44:23.107389+0000","last_fullsized":"2023-10-19T12:44:23.107389+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:59:45.926427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.123560+0000","last_change":"2023-10-19T12:44:23.125353+0000","last_active":"2023-10-19T12:44:24.123560+0000","last_peered":"2023-10-19T12:44:24.123560+0000","last_clean":"2023-10-19T12:44:24.123560+0000","last_became_active":"2023-10-19T12:44:23.124845+0000","last_became_peered":"2023-10-19T12:44:23.124845+0000","last_unstale":"2023-10-19T12:44:24.123560+0000","last_undegraded":"2023-10-19T12:44:24.123560+0000","last_fullsized":"2023-10-19T12:44:24.123560+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:59:49.972984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.116065+0000","last_change":"2023-10-19T12:44:19.084588+0000","last_active":"2023-10-19T12:44:23.116065+0000","last_peered":"2023-10-19T12:44:23.116065+0000","last_clean":"2023-10-19T12:44:23.116065+0000","last_became_active":"2023-10-19T12:44:19.084315+0000","last_became_peered":"2023-10-19T12:44:19.084315+0000","last_unstale":"2023-10-19T12:44:23.116065+0000","last_undegraded":"2023-10-19T12:44:23.116065+0000","last_fullsized":"2023-10-19T12:44:23.116065+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:54:38.037190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.259794+0000","last_change":"2023-10-19T12:44:18.069120+0000","last_active":"2023-10-19T12:44:23.259794+0000","last_peered":"2023-10-19T12:44:23.259794+0000","last_clean":"2023-10-19T12:44:23.259794+0000","last_became_active":"2023-10-19T12:44:18.068509+0000","last_became_peered":"2023-10-19T12:44:18.068509+0000","last_unstale":"2023-10-19T12:44:23.259794+0000","last_undegraded":"2023-10-19T12:44:23.259794+0000","last_fullsized":"2023-10-19T12:44:23.259794+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:01:31.282481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.124541+0000","last_change":"2023-10-19T12:44:23.133194+0000","last_active":"2023-10-19T12:44:24.124541+0000","last_peered":"2023-10-19T12:44:24.124541+0000","last_clean":"2023-10-19T12:44:24.124541+0000","last_became_active":"2023-10-19T12:44:23.132109+0000","last_became_peered":"2023-10-19T12:44:23.132109+0000","last_unstale":"2023-10-19T12:44:24.124541+0000","last_undegraded":"2023-10-19T12:44:24.124541+0000","last_fullsized":"2023-10-19T12:44:24.124541+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:37:44.936303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.108577+0000","last_change":"2023-10-19T12:44:19.082038+0000","last_active":"2023-10-19T12:44:23.108577+0000","last_peered":"2023-10-19T12:44:23.108577+0000","last_clean":"2023-10-19T12:44:23.108577+0000","last_became_active":"2023-10-19T12:44:19.080765+0000","last_became_peered":"2023-10-19T12:44:19.080765+0000","last_unstale":"2023-10-19T12:44:23.108577+0000","last_undegraded":"2023-10-19T12:44:23.108577+0000","last_fullsized":"2023-10-19T12:44:23.108577+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:01:32.286452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.123463+0000","last_change":"2023-10-19T12:44:18.070227+0000","last_active":"2023-10-19T12:44:23.123463+0000","last_peered":"2023-10-19T12:44:23.123463+0000","last_clean":"2023-10-19T12:44:23.123463+0000","last_became_active":"2023-10-19T12:44:18.069958+0000","last_became_peered":"2023-10-19T12:44:18.069958+0000","last_unstale":"2023-10-19T12:44:23.123463+0000","last_undegraded":"2023-10-19T12:44:23.123463+0000","last_fullsized":"2023-10-19T12:44:23.123463+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:27:49.242393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.123413+0000","last_change":"2023-10-19T12:44:23.126070+0000","last_active":"2023-10-19T12:44:24.123413+0000","last_peered":"2023-10-19T12:44:24.123413+0000","last_clean":"2023-10-19T12:44:24.123413+0000","last_became_active":"2023-10-19T12:44:23.125651+0000","last_became_peered":"2023-10-19T12:44:23.125651+0000","last_unstale":"2023-10-19T12:44:24.123413+0000","last_undegraded":"2023-10-19T12:44:24.123413+0000","last_fullsized":"2023-10-19T12:44:24.123413+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:29:56.733260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.109518+0000","last_change":"2023-10-19T12:44:19.085767+0000","last_active":"2023-10-19T12:44:23.109518+0000","last_peered":"2023-10-19T12:44:23.109518+0000","last_clean":"2023-10-19T12:44:23.109518+0000","last_became_active":"2023-10-19T12:44:19.085473+0000","last_became_peered":"2023-10-19T12:44:19.085473+0000","last_unstale":"2023-10-19T12:44:23.109518+0000","last_undegraded":"2023-10-19T12:44:23.109518+0000","last_fullsized":"2023-10-19T12:44:23.109518+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:50:42.817437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.123002+0000","last_change":"2023-10-19T12:44:18.074055+0000","last_active":"2023-10-19T12:44:23.123002+0000","last_peered":"2023-10-19T12:44:23.123002+0000","last_clean":"2023-10-19T12:44:23.123002+0000","last_became_active":"2023-10-19T12:44:18.073821+0000","last_became_peered":"2023-10-19T12:44:18.073821+0000","last_unstale":"2023-10-19T12:44:23.123002+0000","last_undegraded":"2023-10-19T12:44:23.123002+0000","last_fullsized":"2023-10-19T12:44:23.123002+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:24:31.294273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.125580+0000","last_change":"2023-10-19T12:44:23.263639+0000","last_active":"2023-10-19T12:44:24.125580+0000","last_peered":"2023-10-19T12:44:24.125580+0000","last_clean":"2023-10-19T12:44:24.125580+0000","last_became_active":"2023-10-19T12:44:23.263261+0000","last_became_peered":"2023-10-19T12:44:23.263261+0000","last_unstale":"2023-10-19T12:44:24.125580+0000","last_undegraded":"2023-10-19T12:44:24.125580+0000","last_fullsized":"2023-10-19T12:44:24.125580+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:53:36.534082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.114649+0000","last_change":"2023-10-19T12:44:19.080014+0000","last_active":"2023-10-19T12:44:23.114649+0000","last_peered":"2023-10-19T12:44:23.114649+0000","last_clean":"2023-10-19T12:44:23.114649+0000","last_became_active":"2023-10-19T12:44:19.079754+0000","last_became_peered":"2023-10-19T12:44:19.079754+0000","last_unstale":"2023-10-19T12:44:23.114649+0000","last_undegraded":"2023-10-19T12:44:23.114649+0000","last_fullsized":"2023-10-19T12:44:23.114649+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:27:39.900179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.259411+0000","last_change":"2023-10-19T12:44:18.069208+0000","last_active":"2023-10-19T12:44:23.259411+0000","last_peered":"2023-10-19T12:44:23.259411+0000","last_clean":"2023-10-19T12:44:23.259411+0000","last_became_active":"2023-10-19T12:44:18.068303+0000","last_became_peered":"2023-10-19T12:44:18.068303+0000","last_unstale":"2023-10-19T12:44:23.259411+0000","last_undegraded":"2023-10-19T12:44:23.259411+0000","last_fullsized":"2023-10-19T12:44:23.259411+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:11:21.842323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.123471+0000","last_change":"2023-10-19T12:44:23.131978+0000","last_active":"2023-10-19T12:44:24.123471+0000","last_peered":"2023-10-19T12:44:24.123471+0000","last_clean":"2023-10-19T12:44:24.123471+0000","last_became_active":"2023-10-19T12:44:23.131330+0000","last_became_peered":"2023-10-19T12:44:23.131330+0000","last_unstale":"2023-10-19T12:44:24.123471+0000","last_undegraded":"2023-10-19T12:44:24.123471+0000","last_fullsized":"2023-10-19T12:44:24.123471+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:49:30.221056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.113302+0000","last_change":"2023-10-19T12:44:19.087545+0000","last_active":"2023-10-19T12:44:23.113302+0000","last_peered":"2023-10-19T12:44:23.113302+0000","last_clean":"2023-10-19T12:44:23.113302+0000","last_became_active":"2023-10-19T12:44:19.087254+0000","last_became_peered":"2023-10-19T12:44:19.087254+0000","last_unstale":"2023-10-19T12:44:23.113302+0000","last_undegraded":"2023-10-19T12:44:23.113302+0000","last_fullsized":"2023-10-19T12:44:23.113302+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:22:48.368572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.116983+0000","last_change":"2023-10-19T12:44:18.074011+0000","last_active":"2023-10-19T12:44:23.116983+0000","last_peered":"2023-10-19T12:44:23.116983+0000","last_clean":"2023-10-19T12:44:23.116983+0000","last_became_active":"2023-10-19T12:44:18.073749+0000","last_became_peered":"2023-10-19T12:44:18.073749+0000","last_unstale":"2023-10-19T12:44:23.116983+0000","last_undegraded":"2023-10-19T12:44:23.116983+0000","last_fullsized":"2023-10-19T12:44:23.116983+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:46:29.885859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.123298+0000","last_change":"2023-10-19T12:44:23.264194+0000","last_active":"2023-10-19T12:44:24.123298+0000","last_peered":"2023-10-19T12:44:24.123298+0000","last_clean":"2023-10-19T12:44:24.123298+0000","last_became_active":"2023-10-19T12:44:23.263028+0000","last_became_peered":"2023-10-19T12:44:23.263028+0000","last_unstale":"2023-10-19T12:44:24.123298+0000","last_undegraded":"2023-10-19T12:44:24.123298+0000","last_fullsized":"2023-10-19T12:44:24.123298+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:05:19.882281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.107858+0000","last_change":"2023-10-19T12:44:19.094059+0000","last_active":"2023-10-19T12:44:23.107858+0000","last_peered":"2023-10-19T12:44:23.107858+0000","last_clean":"2023-10-19T12:44:23.107858+0000","last_became_active":"2023-10-19T12:44:19.093724+0000","last_became_peered":"2023-10-19T12:44:19.093724+0000","last_unstale":"2023-10-19T12:44:23.107858+0000","last_undegraded":"2023-10-19T12:44:23.107858+0000","last_fullsized":"2023-10-19T12:44:23.107858+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:49:26.174499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.123329+0000","last_change":"2023-10-19T12:44:18.071840+0000","last_active":"2023-10-19T12:44:23.123329+0000","last_peered":"2023-10-19T12:44:23.123329+0000","last_clean":"2023-10-19T12:44:23.123329+0000","last_became_active":"2023-10-19T12:44:18.071447+0000","last_became_peered":"2023-10-19T12:44:18.071447+0000","last_unstale":"2023-10-19T12:44:23.123329+0000","last_undegraded":"2023-10-19T12:44:23.123329+0000","last_fullsized":"2023-10-19T12:44:23.123329+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:46:38.630648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.258316+0000","last_change":"2023-10-19T12:44:23.263517+0000","last_active":"2023-10-19T12:44:24.258316+0000","last_peered":"2023-10-19T12:44:24.258316+0000","last_clean":"2023-10-19T12:44:24.258316+0000","last_became_active":"2023-10-19T12:44:23.263105+0000","last_became_peered":"2023-10-19T12:44:23.263105+0000","last_unstale":"2023-10-19T12:44:24.258316+0000","last_undegraded":"2023-10-19T12:44:24.258316+0000","last_fullsized":"2023-10-19T12:44:24.258316+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:56:44.960048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.122932+0000","last_change":"2023-10-19T12:44:19.088058+0000","last_active":"2023-10-19T12:44:23.122932+0000","last_peered":"2023-10-19T12:44:23.122932+0000","last_clean":"2023-10-19T12:44:23.122932+0000","last_became_active":"2023-10-19T12:44:19.087755+0000","last_became_peered":"2023-10-19T12:44:19.087755+0000","last_unstale":"2023-10-19T12:44:23.122932+0000","last_undegraded":"2023-10-19T12:44:23.122932+0000","last_fullsized":"2023-10-19T12:44:23.122932+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:54:38.037190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.260168+0000","last_change":"2023-10-19T12:44:18.070041+0000","last_active":"2023-10-19T12:44:23.260168+0000","last_peered":"2023-10-19T12:44:23.260168+0000","last_clean":"2023-10-19T12:44:23.260168+0000","last_became_active":"2023-10-19T12:44:18.069723+0000","last_became_peered":"2023-10-19T12:44:18.069723+0000","last_unstale":"2023-10-19T12:44:23.260168+0000","last_undegraded":"2023-10-19T12:44:23.260168+0000","last_fullsized":"2023-10-19T12:44:23.260168+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:10:07.046736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.258633+0000","last_change":"2023-10-19T12:44:23.260957+0000","last_active":"2023-10-19T12:44:24.258633+0000","last_peered":"2023-10-19T12:44:24.258633+0000","last_clean":"2023-10-19T12:44:24.258633+0000","last_became_active":"2023-10-19T12:44:23.260559+0000","last_became_peered":"2023-10-19T12:44:23.260559+0000","last_unstale":"2023-10-19T12:44:24.258633+0000","last_undegraded":"2023-10-19T12:44:24.258633+0000","last_fullsized":"2023-10-19T12:44:24.258633+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:30:09.977362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.107403+0000","last_change":"2023-10-19T12:44:19.080613+0000","last_active":"2023-10-19T12:44:23.107403+0000","last_peered":"2023-10-19T12:44:23.107403+0000","last_clean":"2023-10-19T12:44:23.107403+0000","last_became_active":"2023-10-19T12:44:19.080260+0000","last_became_peered":"2023-10-19T12:44:19.080260+0000","last_unstale":"2023-10-19T12:44:23.107403+0000","last_undegraded":"2023-10-19T12:44:23.107403+0000","last_fullsized":"2023-10-19T12:44:23.107403+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:49:26.174499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.123051+0000","last_change":"2023-10-19T12:44:18.072668+0000","last_active":"2023-10-19T12:44:23.123051+0000","last_peered":"2023-10-19T12:44:23.123051+0000","last_clean":"2023-10-19T12:44:23.123051+0000","last_became_active":"2023-10-19T12:44:18.072383+0000","last_became_peered":"2023-10-19T12:44:18.072383+0000","last_unstale":"2023-10-19T12:44:23.123051+0000","last_undegraded":"2023-10-19T12:44:23.123051+0000","last_fullsized":"2023-10-19T12:44:23.123051+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:07:40.511877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.124753+0000","last_change":"2023-10-19T12:44:23.130828+0000","last_active":"2023-10-19T12:44:24.124753+0000","last_peered":"2023-10-19T12:44:24.124753+0000","last_clean":"2023-10-19T12:44:24.124753+0000","last_became_active":"2023-10-19T12:44:23.130454+0000","last_became_peered":"2023-10-19T12:44:23.130454+0000","last_unstale":"2023-10-19T12:44:24.124753+0000","last_undegraded":"2023-10-19T12:44:24.124753+0000","last_fullsized":"2023-10-19T12:44:24.124753+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:30:09.977362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.114746+0000","last_change":"2023-10-19T12:44:19.075066+0000","last_active":"2023-10-19T12:44:23.114746+0000","last_peered":"2023-10-19T12:44:23.114746+0000","last_clean":"2023-10-19T12:44:23.114746+0000","last_became_active":"2023-10-19T12:44:19.074770+0000","last_became_peered":"2023-10-19T12:44:19.074770+0000","last_unstale":"2023-10-19T12:44:23.114746+0000","last_undegraded":"2023-10-19T12:44:23.114746+0000","last_fullsized":"2023-10-19T12:44:23.114746+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:11:22.846295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.106419+0000","last_change":"2023-10-19T12:44:18.072600+0000","last_active":"2023-10-19T12:44:23.106419+0000","last_peered":"2023-10-19T12:44:23.106419+0000","last_clean":"2023-10-19T12:44:23.106419+0000","last_became_active":"2023-10-19T12:44:18.072278+0000","last_became_peered":"2023-10-19T12:44:18.072278+0000","last_unstale":"2023-10-19T12:44:23.106419+0000","last_undegraded":"2023-10-19T12:44:23.106419+0000","last_fullsized":"2023-10-19T12:44:23.106419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:50:41.813466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.125020+0000","last_change":"2023-10-19T12:44:23.132395+0000","last_active":"2023-10-19T12:44:24.125020+0000","last_peered":"2023-10-19T12:44:24.125020+0000","last_clean":"2023-10-19T12:44:24.125020+0000","last_became_active":"2023-10-19T12:44:23.131961+0000","last_became_peered":"2023-10-19T12:44:23.131961+0000","last_unstale":"2023-10-19T12:44:24.125020+0000","last_undegraded":"2023-10-19T12:44:24.125020+0000","last_fullsized":"2023-10-19T12:44:24.125020+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:37:44.936303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.122772+0000","last_change":"2023-10-19T12:44:19.084119+0000","last_active":"2023-10-19T12:44:23.122772+0000","last_peered":"2023-10-19T12:44:23.122772+0000","last_clean":"2023-10-19T12:44:23.122772+0000","last_became_active":"2023-10-19T12:44:19.083803+0000","last_became_peered":"2023-10-19T12:44:19.083803+0000","last_unstale":"2023-10-19T12:44:23.122772+0000","last_undegraded":"2023-10-19T12:44:23.122772+0000","last_fullsized":"2023-10-19T12:44:23.122772+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:01:32.286452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.107952+0000","last_change":"2023-10-19T12:44:18.066014+0000","last_active":"2023-10-19T12:44:23.107952+0000","last_peered":"2023-10-19T12:44:23.107952+0000","last_clean":"2023-10-19T12:44:23.107952+0000","last_became_active":"2023-10-19T12:44:18.065652+0000","last_became_peered":"2023-10-19T12:44:18.065652+0000","last_unstale":"2023-10-19T12:44:23.107952+0000","last_undegraded":"2023-10-19T12:44:23.107952+0000","last_fullsized":"2023-10-19T12:44:23.107952+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:46:36.490219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.125536+0000","last_change":"2023-10-19T12:44:23.135917+0000","last_active":"2023-10-19T12:44:24.125536+0000","last_peered":"2023-10-19T12:44:24.125536+0000","last_clean":"2023-10-19T12:44:24.125536+0000","last_became_active":"2023-10-19T12:44:23.135459+0000","last_became_peered":"2023-10-19T12:44:23.135459+0000","last_unstale":"2023-10-19T12:44:24.125536+0000","last_undegraded":"2023-10-19T12:44:24.125536+0000","last_fullsized":"2023-10-19T12:44:24.125536+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:24:52.469571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.114317+0000","last_change":"2023-10-19T12:44:19.071520+0000","last_active":"2023-10-19T12:44:23.114317+0000","last_peered":"2023-10-19T12:44:23.114317+0000","last_clean":"2023-10-19T12:44:23.114317+0000","last_became_active":"2023-10-19T12:44:19.071008+0000","last_became_peered":"2023-10-19T12:44:19.071008+0000","last_unstale":"2023-10-19T12:44:23.114317+0000","last_undegraded":"2023-10-19T12:44:23.114317+0000","last_fullsized":"2023-10-19T12:44:23.114317+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:27:50.246364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.259411+0000","last_change":"2023-10-19T12:44:18.071276+0000","last_active":"2023-10-19T12:44:23.259411+0000","last_peered":"2023-10-19T12:44:23.259411+0000","last_clean":"2023-10-19T12:44:23.259411+0000","last_became_active":"2023-10-19T12:44:18.069565+0000","last_became_peered":"2023-10-19T12:44:18.069565+0000","last_unstale":"2023-10-19T12:44:23.259411+0000","last_undegraded":"2023-10-19T12:44:23.259411+0000","last_fullsized":"2023-10-19T12:44:23.259411+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:46:38.630648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.259572+0000","last_change":"2023-10-19T12:44:23.261271+0000","last_active":"2023-10-19T12:44:24.259572+0000","last_peered":"2023-10-19T12:44:24.259572+0000","last_clean":"2023-10-19T12:44:24.259572+0000","last_became_active":"2023-10-19T12:44:23.260853+0000","last_became_peered":"2023-10-19T12:44:23.260853+0000","last_unstale":"2023-10-19T12:44:24.259572+0000","last_undegraded":"2023-10-19T12:44:24.259572+0000","last_fullsized":"2023-10-19T12:44:24.259572+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:53:36.534082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.108153+0000","last_change":"2023-10-19T12:44:19.086009+0000","last_active":"2023-10-19T12:44:23.108153+0000","last_peered":"2023-10-19T12:44:23.108153+0000","last_clean":"2023-10-19T12:44:23.108153+0000","last_became_active":"2023-10-19T12:44:19.085688+0000","last_became_peered":"2023-10-19T12:44:19.085688+0000","last_unstale":"2023-10-19T12:44:23.108153+0000","last_undegraded":"2023-10-19T12:44:23.108153+0000","last_fullsized":"2023-10-19T12:44:23.108153+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:00:15.982705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.106386+0000","last_change":"2023-10-19T12:44:18.067051+0000","last_active":"2023-10-19T12:44:23.106386+0000","last_peered":"2023-10-19T12:44:23.106386+0000","last_clean":"2023-10-19T12:44:23.106386+0000","last_became_active":"2023-10-19T12:44:18.066784+0000","last_became_peered":"2023-10-19T12:44:18.066784+0000","last_unstale":"2023-10-19T12:44:23.106386+0000","last_undegraded":"2023-10-19T12:44:23.106386+0000","last_fullsized":"2023-10-19T12:44:23.106386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:43:47.410870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":168,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.259530+0000","last_change":"2023-10-19T12:44:08.636613+0000","last_active":"2023-10-19T12:44:23.259530+0000","last_peered":"2023-10-19T12:44:23.259530+0000","last_clean":"2023-10-19T12:44:23.259530+0000","last_became_active":"2023-10-19T12:44:08.635697+0000","last_became_peered":"2023-10-19T12:44:08.635697+0000","last_unstale":"2023-10-19T12:44:23.259530+0000","last_undegraded":"2023-10-19T12:44:23.259530+0000","last_fullsized":"2023-10-19T12:44:23.259530+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_clean_scrub_stamp":"2023-10-19T12:44:07.399738+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:15:23.685485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.258583+0000","last_change":"2023-10-19T12:44:23.261893+0000","last_active":"2023-10-19T12:44:24.258583+0000","last_peered":"2023-10-19T12:44:24.258583+0000","last_clean":"2023-10-19T12:44:24.258583+0000","last_became_active":"2023-10-19T12:44:23.261440+0000","last_became_peered":"2023-10-19T12:44:23.261440+0000","last_unstale":"2023-10-19T12:44:24.258583+0000","last_undegraded":"2023-10-19T12:44:24.258583+0000","last_fullsized":"2023-10-19T12:44:24.258583+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:30:09.977362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.259893+0000","last_change":"2023-10-19T12:44:19.093167+0000","last_active":"2023-10-19T12:44:23.259893+0000","last_peered":"2023-10-19T12:44:23.259893+0000","last_clean":"2023-10-19T12:44:23.259893+0000","last_became_active":"2023-10-19T12:44:19.092842+0000","last_became_peered":"2023-10-19T12:44:19.092842+0000","last_unstale":"2023-10-19T12:44:23.259893+0000","last_undegraded":"2023-10-19T12:44:23.259893+0000","last_fullsized":"2023-10-19T12:44:23.259893+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:01:08.972450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.108142+0000","last_change":"2023-10-19T12:44:18.071274+0000","last_active":"2023-10-19T12:44:23.108142+0000","last_peered":"2023-10-19T12:44:23.108142+0000","last_clean":"2023-10-19T12:44:23.108142+0000","last_became_active":"2023-10-19T12:44:18.070997+0000","last_became_peered":"2023-10-19T12:44:18.070997+0000","last_unstale":"2023-10-19T12:44:23.108142+0000","last_undegraded":"2023-10-19T12:44:23.108142+0000","last_fullsized":"2023-10-19T12:44:23.108142+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:27:38.896207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.258302+0000","last_change":"2023-10-19T12:44:23.261162+0000","last_active":"2023-10-19T12:44:24.258302+0000","last_peered":"2023-10-19T12:44:24.258302+0000","last_clean":"2023-10-19T12:44:24.258302+0000","last_became_active":"2023-10-19T12:44:23.260701+0000","last_became_peered":"2023-10-19T12:44:23.260701+0000","last_unstale":"2023-10-19T12:44:24.258302+0000","last_undegraded":"2023-10-19T12:44:24.258302+0000","last_fullsized":"2023-10-19T12:44:24.258302+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:01:13.019006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.107087+0000","last_change":"2023-10-19T12:44:19.090016+0000","last_active":"2023-10-19T12:44:23.107087+0000","last_peered":"2023-10-19T12:44:23.107087+0000","last_clean":"2023-10-19T12:44:23.107087+0000","last_became_active":"2023-10-19T12:44:19.086367+0000","last_became_peered":"2023-10-19T12:44:19.086367+0000","last_unstale":"2023-10-19T12:44:23.107087+0000","last_undegraded":"2023-10-19T12:44:23.107087+0000","last_fullsized":"2023-10-19T12:44:23.107087+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:01:08.972450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.123668+0000","last_change":"2023-10-19T12:44:18.069871+0000","last_active":"2023-10-19T12:44:23.123668+0000","last_peered":"2023-10-19T12:44:23.123668+0000","last_clean":"2023-10-19T12:44:23.123668+0000","last_became_active":"2023-10-19T12:44:18.069558+0000","last_became_peered":"2023-10-19T12:44:18.069558+0000","last_unstale":"2023-10-19T12:44:23.123668+0000","last_undegraded":"2023-10-19T12:44:23.123668+0000","last_fullsized":"2023-10-19T12:44:23.123668+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:15:18.575278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.258925+0000","last_change":"2023-10-19T12:44:23.131775+0000","last_active":"2023-10-19T12:44:24.258925+0000","last_peered":"2023-10-19T12:44:24.258925+0000","last_clean":"2023-10-19T12:44:24.258925+0000","last_became_active":"2023-10-19T12:44:23.131215+0000","last_became_peered":"2023-10-19T12:44:23.131215+0000","last_unstale":"2023-10-19T12:44:24.258925+0000","last_undegraded":"2023-10-19T12:44:24.258925+0000","last_fullsized":"2023-10-19T12:44:24.258925+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:37:44.936303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.260452+0000","last_change":"2023-10-19T12:44:19.089818+0000","last_active":"2023-10-19T12:44:23.260452+0000","last_peered":"2023-10-19T12:44:23.260452+0000","last_clean":"2023-10-19T12:44:23.260452+0000","last_became_active":"2023-10-19T12:44:19.089336+0000","last_became_peered":"2023-10-19T12:44:19.089336+0000","last_unstale":"2023-10-19T12:44:23.260452+0000","last_undegraded":"2023-10-19T12:44:23.260452+0000","last_fullsized":"2023-10-19T12:44:23.260452+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:53:32.487525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.108384+0000","last_change":"2023-10-19T12:44:18.068828+0000","last_active":"2023-10-19T12:44:23.108384+0000","last_peered":"2023-10-19T12:44:23.108384+0000","last_clean":"2023-10-19T12:44:23.108384+0000","last_became_active":"2023-10-19T12:44:18.068553+0000","last_became_peered":"2023-10-19T12:44:18.068553+0000","last_unstale":"2023-10-19T12:44:23.108384+0000","last_undegraded":"2023-10-19T12:44:23.108384+0000","last_fullsized":"2023-10-19T12:44:23.108384+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:11:21.842323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.123595+0000","last_change":"2023-10-19T12:44:23.261245+0000","last_active":"2023-10-19T12:44:24.123595+0000","last_peered":"2023-10-19T12:44:24.123595+0000","last_clean":"2023-10-19T12:44:24.123595+0000","last_became_active":"2023-10-19T12:44:23.260818+0000","last_became_peered":"2023-10-19T12:44:23.260818+0000","last_unstale":"2023-10-19T12:44:24.123595+0000","last_undegraded":"2023-10-19T12:44:24.123595+0000","last_fullsized":"2023-10-19T12:44:24.123595+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:55:55.981359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.260228+0000","last_change":"2023-10-19T12:44:19.087785+0000","last_active":"2023-10-19T12:44:23.260228+0000","last_peered":"2023-10-19T12:44:23.260228+0000","last_clean":"2023-10-19T12:44:23.260228+0000","last_became_active":"2023-10-19T12:44:19.087394+0000","last_became_peered":"2023-10-19T12:44:19.087394+0000","last_unstale":"2023-10-19T12:44:23.260228+0000","last_undegraded":"2023-10-19T12:44:23.260228+0000","last_fullsized":"2023-10-19T12:44:23.260228+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:37:40.889746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.115032+0000","last_change":"2023-10-19T12:44:18.068944+0000","last_active":"2023-10-19T12:44:23.115032+0000","last_peered":"2023-10-19T12:44:23.115032+0000","last_clean":"2023-10-19T12:44:23.115032+0000","last_became_active":"2023-10-19T12:44:18.068674+0000","last_became_peered":"2023-10-19T12:44:18.068674+0000","last_unstale":"2023-10-19T12:44:23.115032+0000","last_undegraded":"2023-10-19T12:44:23.115032+0000","last_fullsized":"2023-10-19T12:44:23.115032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:16:09.533399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.123008+0000","last_change":"2023-10-19T12:44:23.131983+0000","last_active":"2023-10-19T12:44:24.123008+0000","last_peered":"2023-10-19T12:44:24.123008+0000","last_clean":"2023-10-19T12:44:24.123008+0000","last_became_active":"2023-10-19T12:44:23.131304+0000","last_became_peered":"2023-10-19T12:44:23.131304+0000","last_unstale":"2023-10-19T12:44:24.123008+0000","last_undegraded":"2023-10-19T12:44:24.123008+0000","last_fullsized":"2023-10-19T12:44:24.123008+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:01:36.333009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.259741+0000","last_change":"2023-10-19T12:44:19.088843+0000","last_active":"2023-10-19T12:44:23.259741+0000","last_peered":"2023-10-19T12:44:23.259741+0000","last_clean":"2023-10-19T12:44:23.259741+0000","last_became_active":"2023-10-19T12:44:19.088578+0000","last_became_peered":"2023-10-19T12:44:19.088578+0000","last_unstale":"2023-10-19T12:44:23.259741+0000","last_undegraded":"2023-10-19T12:44:23.259741+0000","last_fullsized":"2023-10-19T12:44:23.259741+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:46:55.355893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.108073+0000","last_change":"2023-10-19T12:44:18.069371+0000","last_active":"2023-10-19T12:44:23.108073+0000","last_peered":"2023-10-19T12:44:23.108073+0000","last_clean":"2023-10-19T12:44:23.108073+0000","last_became_active":"2023-10-19T12:44:18.069118+0000","last_became_peered":"2023-10-19T12:44:18.069118+0000","last_unstale":"2023-10-19T12:44:23.108073+0000","last_undegraded":"2023-10-19T12:44:23.108073+0000","last_fullsized":"2023-10-19T12:44:23.108073+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:12:58.093890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.259448+0000","last_change":"2023-10-19T12:44:23.126162+0000","last_active":"2023-10-19T12:44:24.259448+0000","last_peered":"2023-10-19T12:44:24.259448+0000","last_clean":"2023-10-19T12:44:24.259448+0000","last_became_active":"2023-10-19T12:44:23.125364+0000","last_became_peered":"2023-10-19T12:44:23.125364+0000","last_unstale":"2023-10-19T12:44:24.259448+0000","last_undegraded":"2023-10-19T12:44:24.259448+0000","last_fullsized":"2023-10-19T12:44:24.259448+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:24:52.469571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.114403+0000","last_change":"2023-10-19T12:44:19.079421+0000","last_active":"2023-10-19T12:44:23.114403+0000","last_peered":"2023-10-19T12:44:23.114403+0000","last_clean":"2023-10-19T12:44:23.114403+0000","last_became_active":"2023-10-19T12:44:19.079151+0000","last_became_peered":"2023-10-19T12:44:19.079151+0000","last_unstale":"2023-10-19T12:44:23.114403+0000","last_undegraded":"2023-10-19T12:44:23.114403+0000","last_fullsized":"2023-10-19T12:44:23.114403+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:22:48.368572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.114676+0000","last_change":"2023-10-19T12:44:18.070653+0000","last_active":"2023-10-19T12:44:23.114676+0000","last_peered":"2023-10-19T12:44:23.114676+0000","last_clean":"2023-10-19T12:44:23.114676+0000","last_became_active":"2023-10-19T12:44:18.070404+0000","last_became_peered":"2023-10-19T12:44:18.070404+0000","last_unstale":"2023-10-19T12:44:23.114676+0000","last_undegraded":"2023-10-19T12:44:23.114676+0000","last_fullsized":"2023-10-19T12:44:23.114676+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:27:38.896207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.259106+0000","last_change":"2023-10-19T12:44:23.262410+0000","last_active":"2023-10-19T12:44:24.259106+0000","last_peered":"2023-10-19T12:44:24.259106+0000","last_clean":"2023-10-19T12:44:24.259106+0000","last_became_active":"2023-10-19T12:44:23.261667+0000","last_became_peered":"2023-10-19T12:44:23.261667+0000","last_unstale":"2023-10-19T12:44:24.259106+0000","last_undegraded":"2023-10-19T12:44:24.259106+0000","last_fullsized":"2023-10-19T12:44:24.259106+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:23:00.780721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.259656+0000","last_change":"2023-10-19T12:44:19.092039+0000","last_active":"2023-10-19T12:44:23.259656+0000","last_peered":"2023-10-19T12:44:23.259656+0000","last_clean":"2023-10-19T12:44:23.259656+0000","last_became_active":"2023-10-19T12:44:19.090421+0000","last_became_peered":"2023-10-19T12:44:19.090421+0000","last_unstale":"2023-10-19T12:44:23.259656+0000","last_undegraded":"2023-10-19T12:44:23.259656+0000","last_fullsized":"2023-10-19T12:44:23.259656+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:49:26.174499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.106406+0000","last_change":"2023-10-19T12:44:18.066260+0000","last_active":"2023-10-19T12:44:23.106406+0000","last_peered":"2023-10-19T12:44:23.106406+0000","last_clean":"2023-10-19T12:44:23.106406+0000","last_became_active":"2023-10-19T12:44:18.065913+0000","last_became_peered":"2023-10-19T12:44:18.065913+0000","last_unstale":"2023-10-19T12:44:23.106406+0000","last_undegraded":"2023-10-19T12:44:23.106406+0000","last_fullsized":"2023-10-19T12:44:23.106406+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:22:47.364601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.260099+0000","last_change":"2023-10-19T12:44:23.261486+0000","last_active":"2023-10-19T12:44:24.260099+0000","last_peered":"2023-10-19T12:44:24.260099+0000","last_clean":"2023-10-19T12:44:24.260099+0000","last_became_active":"2023-10-19T12:44:23.261044+0000","last_became_peered":"2023-10-19T12:44:23.261044+0000","last_unstale":"2023-10-19T12:44:24.260099+0000","last_undegraded":"2023-10-19T12:44:24.260099+0000","last_fullsized":"2023-10-19T12:44:24.260099+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:27:47.598684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.113455+0000","last_change":"2023-10-19T12:44:19.090117+0000","last_active":"2023-10-19T12:44:23.113455+0000","last_peered":"2023-10-19T12:44:23.113455+0000","last_clean":"2023-10-19T12:44:23.113455+0000","last_became_active":"2023-10-19T12:44:19.089829+0000","last_became_peered":"2023-10-19T12:44:19.089829+0000","last_unstale":"2023-10-19T12:44:23.113455+0000","last_undegraded":"2023-10-19T12:44:23.113455+0000","last_fullsized":"2023-10-19T12:44:23.113455+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:07:41.515848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.113821+0000","last_change":"2023-10-19T12:44:18.067372+0000","last_active":"2023-10-19T12:44:23.113821+0000","last_peered":"2023-10-19T12:44:23.113821+0000","last_clean":"2023-10-19T12:44:23.113821+0000","last_became_active":"2023-10-19T12:44:18.066991+0000","last_became_peered":"2023-10-19T12:44:18.066991+0000","last_unstale":"2023-10-19T12:44:23.113821+0000","last_undegraded":"2023-10-19T12:44:23.113821+0000","last_fullsized":"2023-10-19T12:44:23.113821+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:44:58.807931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.124117+0000","last_change":"2023-10-19T12:44:23.262285+0000","last_active":"2023-10-19T12:44:24.124117+0000","last_peered":"2023-10-19T12:44:24.124117+0000","last_clean":"2023-10-19T12:44:24.124117+0000","last_became_active":"2023-10-19T12:44:23.261716+0000","last_became_peered":"2023-10-19T12:44:23.261716+0000","last_unstale":"2023-10-19T12:44:24.124117+0000","last_undegraded":"2023-10-19T12:44:24.124117+0000","last_fullsized":"2023-10-19T12:44:24.124117+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:33:24.282692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.259803+0000","last_change":"2023-10-19T12:44:19.090802+0000","last_active":"2023-10-19T12:44:23.259803+0000","last_peered":"2023-10-19T12:44:23.259803+0000","last_clean":"2023-10-19T12:44:23.259803+0000","last_became_active":"2023-10-19T12:44:19.087107+0000","last_became_peered":"2023-10-19T12:44:19.087107+0000","last_unstale":"2023-10-19T12:44:23.259803+0000","last_undegraded":"2023-10-19T12:44:23.259803+0000","last_fullsized":"2023-10-19T12:44:23.259803+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:59:45.926427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"25'9","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:28.505371+0000","last_change":"2023-10-19T12:44:18.067624+0000","last_active":"2023-10-19T12:44:28.505371+0000","last_peered":"2023-10-19T12:44:28.505371+0000","last_clean":"2023-10-19T12:44:28.505371+0000","last_became_active":"2023-10-19T12:44:18.067343+0000","last_became_peered":"2023-10-19T12:44:18.067343+0000","last_unstale":"2023-10-19T12:44:28.505371+0000","last_undegraded":"2023-10-19T12:44:28.505371+0000","last_fullsized":"2023-10-19T12:44:28.505371+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:43:47.410870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3841,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.125150+0000","last_change":"2023-10-19T12:44:23.126083+0000","last_active":"2023-10-19T12:44:24.125150+0000","last_peered":"2023-10-19T12:44:24.125150+0000","last_clean":"2023-10-19T12:44:24.125150+0000","last_became_active":"2023-10-19T12:44:23.125365+0000","last_became_peered":"2023-10-19T12:44:23.125365+0000","last_unstale":"2023-10-19T12:44:24.125150+0000","last_undegraded":"2023-10-19T12:44:24.125150+0000","last_fullsized":"2023-10-19T12:44:24.125150+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:40:13.396617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.114883+0000","last_change":"2023-10-19T12:44:19.091229+0000","last_active":"2023-10-19T12:44:23.114883+0000","last_peered":"2023-10-19T12:44:23.114883+0000","last_clean":"2023-10-19T12:44:23.114883+0000","last_became_active":"2023-10-19T12:44:19.090912+0000","last_became_peered":"2023-10-19T12:44:19.090912+0000","last_unstale":"2023-10-19T12:44:23.114883+0000","last_undegraded":"2023-10-19T12:44:23.114883+0000","last_fullsized":"2023-10-19T12:44:23.114883+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:46:39.634619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.123572+0000","last_change":"2023-10-19T12:44:18.072463+0000","last_active":"2023-10-19T12:44:23.123572+0000","last_peered":"2023-10-19T12:44:23.123572+0000","last_clean":"2023-10-19T12:44:23.123572+0000","last_became_active":"2023-10-19T12:44:18.072156+0000","last_became_peered":"2023-10-19T12:44:18.072156+0000","last_unstale":"2023-10-19T12:44:23.123572+0000","last_undegraded":"2023-10-19T12:44:23.123572+0000","last_fullsized":"2023-10-19T12:44:23.123572+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:44:58.807931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.124138+0000","last_change":"2023-10-19T12:44:23.130655+0000","last_active":"2023-10-19T12:44:24.124138+0000","last_peered":"2023-10-19T12:44:24.124138+0000","last_clean":"2023-10-19T12:44:24.124138+0000","last_became_active":"2023-10-19T12:44:23.130142+0000","last_became_peered":"2023-10-19T12:44:23.130142+0000","last_unstale":"2023-10-19T12:44:24.124138+0000","last_undegraded":"2023-10-19T12:44:24.124138+0000","last_fullsized":"2023-10-19T12:44:24.124138+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:30:09.977362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.113707+0000","last_change":"2023-10-19T12:44:19.084143+0000","last_active":"2023-10-19T12:44:23.113707+0000","last_peered":"2023-10-19T12:44:23.113707+0000","last_clean":"2023-10-19T12:44:23.113707+0000","last_became_active":"2023-10-19T12:44:19.083824+0000","last_became_peered":"2023-10-19T12:44:19.083824+0000","last_unstale":"2023-10-19T12:44:23.113707+0000","last_undegraded":"2023-10-19T12:44:23.113707+0000","last_fullsized":"2023-10-19T12:44:23.113707+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:01:32.286452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.106975+0000","last_change":"2023-10-19T12:44:18.070592+0000","last_active":"2023-10-19T12:44:23.106975+0000","last_peered":"2023-10-19T12:44:23.106975+0000","last_clean":"2023-10-19T12:44:23.106975+0000","last_became_active":"2023-10-19T12:44:18.070297+0000","last_became_peered":"2023-10-19T12:44:18.070297+0000","last_unstale":"2023-10-19T12:44:23.106975+0000","last_undegraded":"2023-10-19T12:44:23.106975+0000","last_fullsized":"2023-10-19T12:44:23.106975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:24:31.294273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.258600+0000","last_change":"2023-10-19T12:44:23.131766+0000","last_active":"2023-10-19T12:44:24.258600+0000","last_peered":"2023-10-19T12:44:24.258600+0000","last_clean":"2023-10-19T12:44:24.258600+0000","last_became_active":"2023-10-19T12:44:23.131349+0000","last_became_peered":"2023-10-19T12:44:23.131349+0000","last_unstale":"2023-10-19T12:44:24.258600+0000","last_undegraded":"2023-10-19T12:44:24.258600+0000","last_fullsized":"2023-10-19T12:44:24.258600+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:55:55.981359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.106934+0000","last_change":"2023-10-19T12:44:19.089065+0000","last_active":"2023-10-19T12:44:23.106934+0000","last_peered":"2023-10-19T12:44:23.106934+0000","last_clean":"2023-10-19T12:44:23.106934+0000","last_became_active":"2023-10-19T12:44:19.088295+0000","last_became_peered":"2023-10-19T12:44:19.088295+0000","last_unstale":"2023-10-19T12:44:23.106934+0000","last_undegraded":"2023-10-19T12:44:23.106934+0000","last_fullsized":"2023-10-19T12:44:23.106934+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:30:05.930806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.116850+0000","last_change":"2023-10-19T12:44:18.071630+0000","last_active":"2023-10-19T12:44:23.116850+0000","last_peered":"2023-10-19T12:44:23.116850+0000","last_clean":"2023-10-19T12:44:23.116850+0000","last_became_active":"2023-10-19T12:44:18.071374+0000","last_became_peered":"2023-10-19T12:44:18.071374+0000","last_unstale":"2023-10-19T12:44:23.116850+0000","last_undegraded":"2023-10-19T12:44:23.116850+0000","last_fullsized":"2023-10-19T12:44:23.116850+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:11:21.842323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.259022+0000","last_change":"2023-10-19T12:44:23.261083+0000","last_active":"2023-10-19T12:44:24.259022+0000","last_peered":"2023-10-19T12:44:24.259022+0000","last_clean":"2023-10-19T12:44:24.259022+0000","last_became_active":"2023-10-19T12:44:23.260643+0000","last_became_peered":"2023-10-19T12:44:23.260643+0000","last_unstale":"2023-10-19T12:44:24.259022+0000","last_undegraded":"2023-10-19T12:44:24.259022+0000","last_fullsized":"2023-10-19T12:44:24.259022+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:25:26.010997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.259319+0000","last_change":"2023-10-19T12:44:23.263298+0000","last_active":"2023-10-19T12:44:24.259319+0000","last_peered":"2023-10-19T12:44:24.259319+0000","last_clean":"2023-10-19T12:44:24.259319+0000","last_became_active":"2023-10-19T12:44:23.262936+0000","last_became_peered":"2023-10-19T12:44:23.262936+0000","last_unstale":"2023-10-19T12:44:24.259319+0000","last_undegraded":"2023-10-19T12:44:24.259319+0000","last_fullsized":"2023-10-19T12:44:24.259319+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:40:13.396617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.122817+0000","last_change":"2023-10-19T12:44:19.089929+0000","last_active":"2023-10-19T12:44:23.122817+0000","last_peered":"2023-10-19T12:44:23.122817+0000","last_clean":"2023-10-19T12:44:23.122817+0000","last_became_active":"2023-10-19T12:44:19.089601+0000","last_became_peered":"2023-10-19T12:44:19.089601+0000","last_unstale":"2023-10-19T12:44:23.122817+0000","last_undegraded":"2023-10-19T12:44:23.122817+0000","last_fullsized":"2023-10-19T12:44:23.122817+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:22:48.368572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.107187+0000","last_change":"2023-10-19T12:44:18.069653+0000","last_active":"2023-10-19T12:44:23.107187+0000","last_peered":"2023-10-19T12:44:23.107187+0000","last_clean":"2023-10-19T12:44:23.107187+0000","last_became_active":"2023-10-19T12:44:18.068221+0000","last_became_peered":"2023-10-19T12:44:18.068221+0000","last_unstale":"2023-10-19T12:44:23.107187+0000","last_undegraded":"2023-10-19T12:44:23.107187+0000","last_fullsized":"2023-10-19T12:44:23.107187+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:54:37.033219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.258743+0000","last_change":"2023-10-19T12:44:23.131662+0000","last_active":"2023-10-19T12:44:24.258743+0000","last_peered":"2023-10-19T12:44:24.258743+0000","last_clean":"2023-10-19T12:44:24.258743+0000","last_became_active":"2023-10-19T12:44:23.131182+0000","last_became_peered":"2023-10-19T12:44:23.131182+0000","last_unstale":"2023-10-19T12:44:24.258743+0000","last_undegraded":"2023-10-19T12:44:24.258743+0000","last_fullsized":"2023-10-19T12:44:24.258743+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:37:44.936303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.109433+0000","last_change":"2023-10-19T12:44:19.081926+0000","last_active":"2023-10-19T12:44:23.109433+0000","last_peered":"2023-10-19T12:44:23.109433+0000","last_clean":"2023-10-19T12:44:23.109433+0000","last_became_active":"2023-10-19T12:44:19.081619+0000","last_became_peered":"2023-10-19T12:44:19.081619+0000","last_unstale":"2023-10-19T12:44:23.109433+0000","last_undegraded":"2023-10-19T12:44:23.109433+0000","last_fullsized":"2023-10-19T12:44:23.109433+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:22:48.368572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.108872+0000","last_change":"2023-10-19T12:44:18.071670+0000","last_active":"2023-10-19T12:44:23.108872+0000","last_peered":"2023-10-19T12:44:23.108872+0000","last_clean":"2023-10-19T12:44:23.108872+0000","last_became_active":"2023-10-19T12:44:18.071370+0000","last_became_peered":"2023-10-19T12:44:18.071370+0000","last_unstale":"2023-10-19T12:44:23.108872+0000","last_undegraded":"2023-10-19T12:44:23.108872+0000","last_fullsized":"2023-10-19T12:44:23.108872+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:15:18.575278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.116496+0000","last_change":"2023-10-19T12:44:18.068200+0000","last_active":"2023-10-19T12:44:23.116496+0000","last_peered":"2023-10-19T12:44:23.116496+0000","last_clean":"2023-10-19T12:44:23.116496+0000","last_became_active":"2023-10-19T12:44:18.067903+0000","last_became_peered":"2023-10-19T12:44:18.067903+0000","last_unstale":"2023-10-19T12:44:23.116496+0000","last_undegraded":"2023-10-19T12:44:23.116496+0000","last_fullsized":"2023-10-19T12:44:23.116496+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:27:49.242393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.113730+0000","last_change":"2023-10-19T12:44:19.073697+0000","last_active":"2023-10-19T12:44:23.113730+0000","last_peered":"2023-10-19T12:44:23.113730+0000","last_clean":"2023-10-19T12:44:23.113730+0000","last_became_active":"2023-10-19T12:44:19.073366+0000","last_became_peered":"2023-10-19T12:44:19.073366+0000","last_unstale":"2023-10-19T12:44:23.113730+0000","last_undegraded":"2023-10-19T12:44:23.113730+0000","last_fullsized":"2023-10-19T12:44:23.113730+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:46:55.355893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.258799+0000","last_change":"2023-10-19T12:44:23.131575+0000","last_active":"2023-10-19T12:44:24.258799+0000","last_peered":"2023-10-19T12:44:24.258799+0000","last_clean":"2023-10-19T12:44:24.258799+0000","last_became_active":"2023-10-19T12:44:23.131125+0000","last_became_peered":"2023-10-19T12:44:23.131125+0000","last_unstale":"2023-10-19T12:44:24.258799+0000","last_undegraded":"2023-10-19T12:44:24.258799+0000","last_fullsized":"2023-10-19T12:44:24.258799+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:15:23.625806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.108702+0000","last_change":"2023-10-19T12:44:18.071298+0000","last_active":"2023-10-19T12:44:23.108702+0000","last_peered":"2023-10-19T12:44:23.108702+0000","last_clean":"2023-10-19T12:44:23.108702+0000","last_became_active":"2023-10-19T12:44:18.070913+0000","last_became_peered":"2023-10-19T12:44:18.070913+0000","last_unstale":"2023-10-19T12:44:23.108702+0000","last_undegraded":"2023-10-19T12:44:23.108702+0000","last_fullsized":"2023-10-19T12:44:23.108702+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:17.042941+0000","last_clean_scrub_stamp":"2023-10-19T12:44:17.042941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:27:49.242393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.115487+0000","last_change":"2023-10-19T12:44:19.085944+0000","last_active":"2023-10-19T12:44:23.115487+0000","last_peered":"2023-10-19T12:44:23.115487+0000","last_clean":"2023-10-19T12:44:23.115487+0000","last_became_active":"2023-10-19T12:44:19.085621+0000","last_became_peered":"2023-10-19T12:44:19.085621+0000","last_unstale":"2023-10-19T12:44:23.115487+0000","last_undegraded":"2023-10-19T12:44:23.115487+0000","last_fullsized":"2023-10-19T12:44:23.115487+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:12:59.097861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-19T12:44:24.260011+0000","last_change":"2023-10-19T12:44:23.126882+0000","last_active":"2023-10-19T12:44:24.260011+0000","last_peered":"2023-10-19T12:44:24.260011+0000","last_clean":"2023-10-19T12:44:24.260011+0000","last_became_active":"2023-10-19T12:44:23.126421+0000","last_became_peered":"2023-10-19T12:44:23.126421+0000","last_unstale":"2023-10-19T12:44:24.260011+0000","last_undegraded":"2023-10-19T12:44:24.260011+0000","last_fullsized":"2023-10-19T12:44:24.260011+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:22.093469+0000","last_clean_scrub_stamp":"2023-10-19T12:44:22.093469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:32:54.650601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.114291+0000","last_change":"2023-10-19T12:44:19.080194+0000","last_active":"2023-10-19T12:44:23.114291+0000","last_peered":"2023-10-19T12:44:23.114291+0000","last_clean":"2023-10-19T12:44:23.114291+0000","last_became_active":"2023-10-19T12:44:19.079941+0000","last_became_peered":"2023-10-19T12:44:19.079941+0000","last_unstale":"2023-10-19T12:44:23.114291+0000","last_undegraded":"2023-10-19T12:44:23.114291+0000","last_fullsized":"2023-10-19T12:44:23.114291+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:50:42.817437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.107859+0000","last_change":"2023-10-19T12:44:19.098110+0000","last_active":"2023-10-19T12:44:23.107859+0000","last_peered":"2023-10-19T12:44:23.107859+0000","last_clean":"2023-10-19T12:44:23.107859+0000","last_became_active":"2023-10-19T12:44:19.097774+0000","last_became_peered":"2023-10-19T12:44:19.097774+0000","last_unstale":"2023-10-19T12:44:23.107859+0000","last_undegraded":"2023-10-19T12:44:23.107859+0000","last_fullsized":"2023-10-19T12:44:23.107859+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:24:32.298244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.106333+0000","last_change":"2023-10-19T12:44:19.083631+0000","last_active":"2023-10-19T12:44:23.106333+0000","last_peered":"2023-10-19T12:44:23.106333+0000","last_clean":"2023-10-19T12:44:23.106333+0000","last_became_active":"2023-10-19T12:44:19.082464+0000","last_became_peered":"2023-10-19T12:44:19.082464+0000","last_unstale":"2023-10-19T12:44:23.106333+0000","last_undegraded":"2023-10-19T12:44:23.106333+0000","last_fullsized":"2023-10-19T12:44:23.106333+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:40:09.350060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.113640+0000","last_change":"2023-10-19T12:44:19.083390+0000","last_active":"2023-10-19T12:44:23.113640+0000","last_peered":"2023-10-19T12:44:23.113640+0000","last_clean":"2023-10-19T12:44:23.113640+0000","last_became_active":"2023-10-19T12:44:19.082423+0000","last_became_peered":"2023-10-19T12:44:19.082423+0000","last_unstale":"2023-10-19T12:44:23.113640+0000","last_undegraded":"2023-10-19T12:44:23.113640+0000","last_fullsized":"2023-10-19T12:44:23.113640+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:54:38.037190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.110543+0000","last_change":"2023-10-19T12:44:19.090250+0000","last_active":"2023-10-19T12:44:23.110543+0000","last_peered":"2023-10-19T12:44:23.110543+0000","last_clean":"2023-10-19T12:44:23.110543+0000","last_became_active":"2023-10-19T12:44:19.089587+0000","last_became_peered":"2023-10-19T12:44:19.089587+0000","last_unstale":"2023-10-19T12:44:23.110543+0000","last_undegraded":"2023-10-19T12:44:23.110543+0000","last_fullsized":"2023-10-19T12:44:23.110543+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:46:55.355893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.122832+0000","last_change":"2023-10-19T12:44:19.088635+0000","last_active":"2023-10-19T12:44:23.122832+0000","last_peered":"2023-10-19T12:44:23.122832+0000","last_clean":"2023-10-19T12:44:23.122832+0000","last_became_active":"2023-10-19T12:44:19.088378+0000","last_became_peered":"2023-10-19T12:44:19.088378+0000","last_unstale":"2023-10-19T12:44:23.122832+0000","last_undegraded":"2023-10-19T12:44:23.122832+0000","last_fullsized":"2023-10-19T12:44:23.122832+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:00:15.982705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.115903+0000","last_change":"2023-10-19T12:44:19.089688+0000","last_active":"2023-10-19T12:44:23.115903+0000","last_peered":"2023-10-19T12:44:23.115903+0000","last_clean":"2023-10-19T12:44:23.115903+0000","last_became_active":"2023-10-19T12:44:19.089438+0000","last_became_peered":"2023-10-19T12:44:19.089438+0000","last_unstale":"2023-10-19T12:44:23.115903+0000","last_undegraded":"2023-10-19T12:44:23.115903+0000","last_fullsized":"2023-10-19T12:44:23.115903+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:46:55.355893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.107774+0000","last_change":"2023-10-19T12:44:19.087749+0000","last_active":"2023-10-19T12:44:23.107774+0000","last_peered":"2023-10-19T12:44:23.107774+0000","last_clean":"2023-10-19T12:44:23.107774+0000","last_became_active":"2023-10-19T12:44:19.087020+0000","last_became_peered":"2023-10-19T12:44:19.087020+0000","last_unstale":"2023-10-19T12:44:23.107774+0000","last_undegraded":"2023-10-19T12:44:23.107774+0000","last_fullsized":"2023-10-19T12:44:23.107774+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:05:15.835724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.106909+0000","last_change":"2023-10-19T12:44:19.085297+0000","last_active":"2023-10-19T12:44:23.106909+0000","last_peered":"2023-10-19T12:44:23.106909+0000","last_clean":"2023-10-19T12:44:23.106909+0000","last_became_active":"2023-10-19T12:44:19.084652+0000","last_became_peered":"2023-10-19T12:44:19.084652+0000","last_unstale":"2023-10-19T12:44:23.106909+0000","last_undegraded":"2023-10-19T12:44:23.106909+0000","last_fullsized":"2023-10-19T12:44:23.106909+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:05:15.835724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.108790+0000","last_change":"2023-10-19T12:44:19.084748+0000","last_active":"2023-10-19T12:44:23.108790+0000","last_peered":"2023-10-19T12:44:23.108790+0000","last_clean":"2023-10-19T12:44:23.108790+0000","last_became_active":"2023-10-19T12:44:19.084460+0000","last_became_peered":"2023-10-19T12:44:19.084460+0000","last_unstale":"2023-10-19T12:44:23.108790+0000","last_undegraded":"2023-10-19T12:44:23.108790+0000","last_fullsized":"2023-10-19T12:44:23.108790+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:54:38.037190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.106299+0000","last_change":"2023-10-19T12:44:19.087125+0000","last_active":"2023-10-19T12:44:23.106299+0000","last_peered":"2023-10-19T12:44:23.106299+0000","last_clean":"2023-10-19T12:44:23.106299+0000","last_became_active":"2023-10-19T12:44:19.086845+0000","last_became_peered":"2023-10-19T12:44:19.086845+0000","last_unstale":"2023-10-19T12:44:23.106299+0000","last_undegraded":"2023-10-19T12:44:23.106299+0000","last_fullsized":"2023-10-19T12:44:23.106299+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:37:40.889746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.113534+0000","last_change":"2023-10-19T12:44:19.089064+0000","last_active":"2023-10-19T12:44:23.113534+0000","last_peered":"2023-10-19T12:44:23.113534+0000","last_clean":"2023-10-19T12:44:23.113534+0000","last_became_active":"2023-10-19T12:44:19.088780+0000","last_became_peered":"2023-10-19T12:44:19.088780+0000","last_unstale":"2023-10-19T12:44:23.113534+0000","last_undegraded":"2023-10-19T12:44:23.113534+0000","last_fullsized":"2023-10-19T12:44:23.113534+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:10:08.050707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.115561+0000","last_change":"2023-10-19T12:44:19.087124+0000","last_active":"2023-10-19T12:44:23.115561+0000","last_peered":"2023-10-19T12:44:23.115561+0000","last_clean":"2023-10-19T12:44:23.115561+0000","last_became_active":"2023-10-19T12:44:19.086851+0000","last_became_peered":"2023-10-19T12:44:19.086851+0000","last_unstale":"2023-10-19T12:44:23.115561+0000","last_undegraded":"2023-10-19T12:44:23.115561+0000","last_fullsized":"2023-10-19T12:44:23.115561+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:10:08.050707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.260204+0000","last_change":"2023-10-19T12:44:19.091368+0000","last_active":"2023-10-19T12:44:23.260204+0000","last_peered":"2023-10-19T12:44:23.260204+0000","last_clean":"2023-10-19T12:44:23.260204+0000","last_became_active":"2023-10-19T12:44:19.091052+0000","last_became_peered":"2023-10-19T12:44:19.091052+0000","last_unstale":"2023-10-19T12:44:23.260204+0000","last_undegraded":"2023-10-19T12:44:23.260204+0000","last_fullsized":"2023-10-19T12:44:23.260204+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:40:09.350060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.108300+0000","last_change":"2023-10-19T12:44:19.080430+0000","last_active":"2023-10-19T12:44:23.108300+0000","last_peered":"2023-10-19T12:44:23.108300+0000","last_clean":"2023-10-19T12:44:23.108300+0000","last_became_active":"2023-10-19T12:44:19.079493+0000","last_became_peered":"2023-10-19T12:44:19.079493+0000","last_unstale":"2023-10-19T12:44:23.108300+0000","last_undegraded":"2023-10-19T12:44:23.108300+0000","last_fullsized":"2023-10-19T12:44:23.108300+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:46:39.634619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.107179+0000","last_change":"2023-10-19T12:44:19.088751+0000","last_active":"2023-10-19T12:44:23.107179+0000","last_peered":"2023-10-19T12:44:23.107179+0000","last_clean":"2023-10-19T12:44:23.107179+0000","last_became_active":"2023-10-19T12:44:19.088430+0000","last_became_peered":"2023-10-19T12:44:19.088430+0000","last_unstale":"2023-10-19T12:44:23.107179+0000","last_undegraded":"2023-10-19T12:44:23.107179+0000","last_fullsized":"2023-10-19T12:44:23.107179+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:24:48.423014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.106786+0000","last_change":"2023-10-19T12:44:19.082935+0000","last_active":"2023-10-19T12:44:23.106786+0000","last_peered":"2023-10-19T12:44:23.106786+0000","last_clean":"2023-10-19T12:44:23.106786+0000","last_became_active":"2023-10-19T12:44:19.082692+0000","last_became_peered":"2023-10-19T12:44:19.082692+0000","last_unstale":"2023-10-19T12:44:23.106786+0000","last_undegraded":"2023-10-19T12:44:23.106786+0000","last_fullsized":"2023-10-19T12:44:23.106786+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:10:08.050707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.122913+0000","last_change":"2023-10-19T12:44:19.087441+0000","last_active":"2023-10-19T12:44:23.122913+0000","last_peered":"2023-10-19T12:44:23.122913+0000","last_clean":"2023-10-19T12:44:23.122913+0000","last_became_active":"2023-10-19T12:44:19.087147+0000","last_became_peered":"2023-10-19T12:44:19.087147+0000","last_unstale":"2023-10-19T12:44:23.122913+0000","last_undegraded":"2023-10-19T12:44:23.122913+0000","last_fullsized":"2023-10-19T12:44:23.122913+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:50:42.817437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.113370+0000","last_change":"2023-10-19T12:44:19.085136+0000","last_active":"2023-10-19T12:44:23.113370+0000","last_peered":"2023-10-19T12:44:23.113370+0000","last_clean":"2023-10-19T12:44:23.113370+0000","last_became_active":"2023-10-19T12:44:19.084856+0000","last_became_peered":"2023-10-19T12:44:19.084856+0000","last_unstale":"2023-10-19T12:44:23.113370+0000","last_undegraded":"2023-10-19T12:44:23.113370+0000","last_fullsized":"2023-10-19T12:44:23.113370+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:46:30.889830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.109109+0000","last_change":"2023-10-19T12:44:19.080419+0000","last_active":"2023-10-19T12:44:23.109109+0000","last_peered":"2023-10-19T12:44:23.109109+0000","last_clean":"2023-10-19T12:44:23.109109+0000","last_became_active":"2023-10-19T12:44:19.080123+0000","last_became_peered":"2023-10-19T12:44:19.080123+0000","last_unstale":"2023-10-19T12:44:23.109109+0000","last_undegraded":"2023-10-19T12:44:23.109109+0000","last_fullsized":"2023-10-19T12:44:23.109109+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:10:08.050707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.114806+0000","last_change":"2023-10-19T12:44:19.080702+0000","last_active":"2023-10-19T12:44:23.114806+0000","last_peered":"2023-10-19T12:44:23.114806+0000","last_clean":"2023-10-19T12:44:23.114806+0000","last_became_active":"2023-10-19T12:44:19.079347+0000","last_became_peered":"2023-10-19T12:44:19.079347+0000","last_unstale":"2023-10-19T12:44:23.114806+0000","last_undegraded":"2023-10-19T12:44:23.114806+0000","last_fullsized":"2023-10-19T12:44:23.114806+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:43:48.414841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.115372+0000","last_change":"2023-10-19T12:44:19.081326+0000","last_active":"2023-10-19T12:44:23.115372+0000","last_peered":"2023-10-19T12:44:23.115372+0000","last_clean":"2023-10-19T12:44:23.115372+0000","last_became_active":"2023-10-19T12:44:19.081063+0000","last_became_peered":"2023-10-19T12:44:19.081063+0000","last_unstale":"2023-10-19T12:44:23.115372+0000","last_undegraded":"2023-10-19T12:44:23.115372+0000","last_fullsized":"2023-10-19T12:44:23.115372+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:00:15.982705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.115814+0000","last_change":"2023-10-19T12:44:19.094478+0000","last_active":"2023-10-19T12:44:23.115814+0000","last_peered":"2023-10-19T12:44:23.115814+0000","last_clean":"2023-10-19T12:44:23.115814+0000","last_became_active":"2023-10-19T12:44:19.094217+0000","last_became_peered":"2023-10-19T12:44:19.094217+0000","last_unstale":"2023-10-19T12:44:23.115814+0000","last_undegraded":"2023-10-19T12:44:23.115814+0000","last_fullsized":"2023-10-19T12:44:23.115814+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:01:32.286452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.260360+0000","last_change":"2023-10-19T12:44:19.093197+0000","last_active":"2023-10-19T12:44:23.260360+0000","last_peered":"2023-10-19T12:44:23.260360+0000","last_clean":"2023-10-19T12:44:23.260360+0000","last_became_active":"2023-10-19T12:44:19.092926+0000","last_became_peered":"2023-10-19T12:44:19.092926+0000","last_unstale":"2023-10-19T12:44:23.260360+0000","last_undegraded":"2023-10-19T12:44:23.260360+0000","last_fullsized":"2023-10-19T12:44:23.260360+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:30:05.930806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.109339+0000","last_change":"2023-10-19T12:44:19.095093+0000","last_active":"2023-10-19T12:44:23.109339+0000","last_peered":"2023-10-19T12:44:23.109339+0000","last_clean":"2023-10-19T12:44:23.109339+0000","last_became_active":"2023-10-19T12:44:19.094816+0000","last_became_peered":"2023-10-19T12:44:19.094816+0000","last_unstale":"2023-10-19T12:44:23.109339+0000","last_undegraded":"2023-10-19T12:44:23.109339+0000","last_fullsized":"2023-10-19T12:44:23.109339+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:07:41.515848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.259584+0000","last_change":"2023-10-19T12:44:19.086048+0000","last_active":"2023-10-19T12:44:23.259584+0000","last_peered":"2023-10-19T12:44:23.259584+0000","last_clean":"2023-10-19T12:44:23.259584+0000","last_became_active":"2023-10-19T12:44:19.085086+0000","last_became_peered":"2023-10-19T12:44:19.085086+0000","last_unstale":"2023-10-19T12:44:23.259584+0000","last_undegraded":"2023-10-19T12:44:23.259584+0000","last_fullsized":"2023-10-19T12:44:23.259584+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:05:15.835724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.106214+0000","last_change":"2023-10-19T12:44:19.086341+0000","last_active":"2023-10-19T12:44:23.106214+0000","last_peered":"2023-10-19T12:44:23.106214+0000","last_clean":"2023-10-19T12:44:23.106214+0000","last_became_active":"2023-10-19T12:44:19.083365+0000","last_became_peered":"2023-10-19T12:44:19.083365+0000","last_unstale":"2023-10-19T12:44:23.106214+0000","last_undegraded":"2023-10-19T12:44:23.106214+0000","last_fullsized":"2023-10-19T12:44:23.106214+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:46:55.355893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.259588+0000","last_change":"2023-10-19T12:44:19.091228+0000","last_active":"2023-10-19T12:44:23.259588+0000","last_peered":"2023-10-19T12:44:23.259588+0000","last_clean":"2023-10-19T12:44:23.259588+0000","last_became_active":"2023-10-19T12:44:19.087203+0000","last_became_peered":"2023-10-19T12:44:19.087203+0000","last_unstale":"2023-10-19T12:44:23.259588+0000","last_undegraded":"2023-10-19T12:44:23.259588+0000","last_fullsized":"2023-10-19T12:44:23.259588+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:29:52.686703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.106856+0000","last_change":"2023-10-19T12:44:19.086112+0000","last_active":"2023-10-19T12:44:23.106856+0000","last_peered":"2023-10-19T12:44:23.106856+0000","last_clean":"2023-10-19T12:44:23.106856+0000","last_became_active":"2023-10-19T12:44:19.085822+0000","last_became_peered":"2023-10-19T12:44:19.085822+0000","last_unstale":"2023-10-19T12:44:23.106856+0000","last_undegraded":"2023-10-19T12:44:23.106856+0000","last_fullsized":"2023-10-19T12:44:23.106856+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:01:32.286452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.113429+0000","last_change":"2023-10-19T12:44:19.078205+0000","last_active":"2023-10-19T12:44:23.113429+0000","last_peered":"2023-10-19T12:44:23.113429+0000","last_clean":"2023-10-19T12:44:23.113429+0000","last_became_active":"2023-10-19T12:44:19.077824+0000","last_became_peered":"2023-10-19T12:44:19.077824+0000","last_unstale":"2023-10-19T12:44:23.113429+0000","last_undegraded":"2023-10-19T12:44:23.113429+0000","last_fullsized":"2023-10-19T12:44:23.113429+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:50:42.817437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.123225+0000","last_change":"2023-10-19T12:44:19.084212+0000","last_active":"2023-10-19T12:44:23.123225+0000","last_peered":"2023-10-19T12:44:23.123225+0000","last_clean":"2023-10-19T12:44:23.123225+0000","last_became_active":"2023-10-19T12:44:19.083928+0000","last_became_peered":"2023-10-19T12:44:19.083928+0000","last_unstale":"2023-10-19T12:44:23.123225+0000","last_undegraded":"2023-10-19T12:44:23.123225+0000","last_fullsized":"2023-10-19T12:44:23.123225+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:10:08.050707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-10-19T12:44:23.260753+0000","last_change":"2023-10-19T12:44:19.087938+0000","last_active":"2023-10-19T12:44:23.260753+0000","last_peered":"2023-10-19T12:44:23.260753+0000","last_clean":"2023-10-19T12:44:23.260753+0000","last_became_active":"2023-10-19T12:44:19.087552+0000","last_became_peered":"2023-10-19T12:44:19.087552+0000","last_unstale":"2023-10-19T12:44:23.260753+0000","last_undegraded":"2023-10-19T12:44:23.260753+0000","last_fullsized":"2023-10-19T12:44:23.260753+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:18.046913+0000","last_clean_scrub_stamp":"2023-10-19T12:44:18.046913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:24:48.423014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4639,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574859,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27868,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343972,"statfs":{"total":96636764160,"available":96608227328,"internally_reserved":0,"allocated":864256,"data_stored":695401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78500000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71299999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69899999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76600000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39500000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42999999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53800000000000003}]}]},{"osd":6,"up_from":13,"seq":55834574859,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":108292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54600000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71999999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57399999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67600000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.628}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.495}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73799999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574859,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":270336,"data_stored":104541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39200000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47099999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65900000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60299999999999998}]}]},{"osd":4,"up_from":13,"seq":55834574859,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":278528,"data_stored":104577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56000000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60999999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57999999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45300000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41399999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.43099999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574858,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31448,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340392,"statfs":{"total":96636764160,"available":96604561408,"internally_reserved":0,"allocated":278528,"data_stored":108292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27400000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33500000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39900000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59699999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61299999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68799999999999994}]}]},{"osd":2,"up_from":13,"seq":55834574860,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31448,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340392,"statfs":{"total":96636764160,"available":96604561408,"internally_reserved":0,"allocated":266240,"data_stored":104451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39100000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36899999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.435}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.504}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55700000000000005}]}]},{"osd":1,"up_from":13,"seq":55834574859,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31460,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340380,"statfs":{"total":96636764160,"available":96604549120,"internally_reserved":0,"allocated":274432,"data_stored":104611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.219}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.255}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35199999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70199999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52600000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64200000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71599999999999997}]}]},{"osd":0,"up_from":13,"seq":55834574860,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32032,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339808,"statfs":{"total":96636764160,"available":96603963392,"internally_reserved":0,"allocated":868352,"data_stored":694838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.219}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.318}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30099999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57899999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54300000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.502}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-19T12:44:47.321 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-19T12:44:47.322 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-10-19T12:44:47.322 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-10-19T12:44:47.322 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-19T12:44:47.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.459+0000 7f498ea44640 1 -- 172.21.15.57:0/3493385180 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49880ceb40 msgr2=0x7f49880b0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:47.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.459+0000 7f498ea44640 1 --2- 172.21.15.57:0/3493385180 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49880ceb40 0x7f49880b0650 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f498808a120 tx=0x7f498002c080 comp rx=0 tx=0).stop 2023-10-19T12:44:47.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.459+0000 7f498ea44640 1 -- 172.21.15.57:0/3493385180 shutdown_connections 2023-10-19T12:44:47.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.459+0000 7f498ea44640 1 --2- 172.21.15.57:0/3493385180 >> v2:172.21.15.179:3300/0 conn(0x7f498815b2a0 0x7f49880d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:47.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.459+0000 7f498ea44640 1 --2- 172.21.15.57:0/3493385180 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49880ceb40 0x7f49880b0650 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:47.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.459+0000 7f498ea44640 1 -- 172.21.15.57:0/3493385180 >> 172.21.15.57:0/3493385180 conn(0x7f49880d0d60 msgr2=0x7f49880d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:47.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.459+0000 7f498ea44640 1 -- 172.21.15.57:0/3493385180 shutdown_connections 2023-10-19T12:44:47.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.459+0000 7f498ea44640 1 -- 172.21.15.57:0/3493385180 wait complete. 2023-10-19T12:44:47.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.461+0000 7f498ea44640 1 Processor -- start 2023-10-19T12:44:47.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.461+0000 7f498ea44640 1 -- start start 2023-10-19T12:44:47.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.461+0000 7f498ea44640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49880ceb40 0x7f49880c7e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:47.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.461+0000 7f498ea44640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f49880cf340 0x7f49880c95f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:47.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.461+0000 7f498ea44640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f498815b2a0 0x7f4988157210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:47.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.461+0000 7f498ea44640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4988153ad0 con 0x7f49880ceb40 2023-10-19T12:44:47.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.461+0000 7f498ea44640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4988158a10 con 0x7f498815b2a0 2023-10-19T12:44:47.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.461+0000 7f498ea44640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49880c8dd0 con 0x7f49880cf340 2023-10-19T12:44:47.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.462+0000 7f4987fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49880ceb40 0x7f49880c7e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:47.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.462+0000 7f4987fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49880ceb40 0x7f49880c7e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59168/0 (socket says 172.21.15.57:59168) 2023-10-19T12:44:47.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.462+0000 7f4987fff640 1 -- 172.21.15.57:0/3379505643 learned_addr learned my addr 172.21.15.57:0/3379505643 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:47.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.462+0000 7f498cfe1640 1 --2- 172.21.15.57:0/3379505643 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f498815b2a0 0x7f4988157210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:47.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.462+0000 7f49877fe640 1 --2- 172.21.15.57:0/3379505643 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f49880cf340 0x7f49880c95f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:47.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.462+0000 7f4987fff640 1 -- 172.21.15.57:0/3379505643 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f49880cf340 msgr2=0x7f49880c95f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:47.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.462+0000 7f4987fff640 1 --2- 172.21.15.57:0/3379505643 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f49880cf340 0x7f49880c95f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:47.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.462+0000 7f4987fff640 1 -- 172.21.15.57:0/3379505643 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f498815b2a0 msgr2=0x7f4988157210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:47.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.462+0000 7f4987fff640 1 --2- 172.21.15.57:0/3379505643 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f498815b2a0 0x7f4988157210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:47.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.462+0000 7f4987fff640 1 -- 172.21.15.57:0/3379505643 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4980009840 con 0x7f49880ceb40 2023-10-19T12:44:47.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.462+0000 7f498cfe1640 1 --2- 172.21.15.57:0/3379505643 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f498815b2a0 0x7f4988157210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:44:47.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.462+0000 7f4987fff640 1 --2- 172.21.15.57:0/3379505643 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49880ceb40 0x7f49880c7e30 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f497c00c400 tx=0x7f497c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:47.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.462+0000 7f4984ff9640 1 -- 172.21.15.57:0/3379505643 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f497c004230 con 0x7f49880ceb40 2023-10-19T12:44:47.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.462+0000 7f4984ff9640 1 -- 172.21.15.57:0/3379505643 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f497c0044a0 con 0x7f49880ceb40 2023-10-19T12:44:47.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.462+0000 7f498ea44640 1 -- 172.21.15.57:0/3379505643 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49880c9b30 con 0x7f49880ceb40 2023-10-19T12:44:47.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.462+0000 7f4984ff9640 1 -- 172.21.15.57:0/3379505643 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f497c003850 con 0x7f49880ceb40 2023-10-19T12:44:47.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.462+0000 7f498ea44640 1 -- 172.21.15.57:0/3379505643 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49880c8810 con 0x7f49880ceb40 2023-10-19T12:44:47.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.463+0000 7f498ea44640 1 -- 172.21.15.57:0/3379505643 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49880690e0 con 0x7f49880ceb40 2023-10-19T12:44:47.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.464+0000 7f4984ff9640 1 -- 172.21.15.57:0/3379505643 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f497c0039b0 con 0x7f49880ceb40 2023-10-19T12:44:47.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.464+0000 7f4984ff9640 1 --2- 172.21.15.57:0/3379505643 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f497406ae70 0x7f497406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:47.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.464+0000 7f4984ff9640 1 -- 172.21.15.57:0/3379505643 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f497c088fb0 con 0x7f49880ceb40 2023-10-19T12:44:47.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.464+0000 7f49877fe640 1 --2- 172.21.15.57:0/3379505643 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f497406ae70 0x7f497406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:47.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.465+0000 7f49877fe640 1 --2- 172.21.15.57:0/3379505643 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f497406ae70 0x7f497406d330 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f49880b6700 tx=0x7f497800c440 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:47.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.468+0000 7f4984ff9640 1 -- 172.21.15.57:0/3379505643 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f497c0bbca0 con 0x7f49880ceb40 2023-10-19T12:44:47.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.682+0000 7f498ea44640 1 -- 172.21.15.57:0/3379505643 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f49880b0d00 con 0x7f49880ceb40 2023-10-19T12:44:47.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.682+0000 7f4984ff9640 1 -- 172.21.15.57:0/3379505643 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f497c017020 con 0x7f49880ceb40 2023-10-19T12:44:47.686 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:44:47.686 INFO:teuthology.orchestra.run.smithi057.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-10-19T12:44:47.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.684+0000 7f498ea44640 1 -- 172.21.15.57:0/3379505643 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f497406ae70 msgr2=0x7f497406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:47.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.684+0000 7f498ea44640 1 --2- 172.21.15.57:0/3379505643 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f497406ae70 0x7f497406d330 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f49880b6700 tx=0x7f497800c440 comp rx=0 tx=0).stop 2023-10-19T12:44:47.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.684+0000 7f498ea44640 1 -- 172.21.15.57:0/3379505643 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49880ceb40 msgr2=0x7f49880c7e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:47.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.684+0000 7f498ea44640 1 --2- 172.21.15.57:0/3379505643 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49880ceb40 0x7f49880c7e30 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f497c00c400 tx=0x7f497c00c8c0 comp rx=0 tx=0).stop 2023-10-19T12:44:47.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.684+0000 7f498ea44640 1 -- 172.21.15.57:0/3379505643 shutdown_connections 2023-10-19T12:44:47.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.684+0000 7f498ea44640 1 --2- 172.21.15.57:0/3379505643 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f497406ae70 0x7f497406d330 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:47.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.684+0000 7f498ea44640 1 --2- 172.21.15.57:0/3379505643 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f498815b2a0 0x7f4988157210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:47.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.684+0000 7f498ea44640 1 --2- 172.21.15.57:0/3379505643 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f49880cf340 0x7f49880c95f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:47.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.684+0000 7f498ea44640 1 --2- 172.21.15.57:0/3379505643 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49880ceb40 0x7f49880c7e30 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:47.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.685+0000 7f498ea44640 1 -- 172.21.15.57:0/3379505643 >> 172.21.15.57:0/3379505643 conn(0x7f49880d0d60 msgr2=0x7f498806b650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:47.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.685+0000 7f498ea44640 1 -- 172.21.15.57:0/3379505643 shutdown_connections 2023-10-19T12:44:47.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.685+0000 7f498ea44640 1 -- 172.21.15.57:0/3379505643 wait complete. 2023-10-19T12:44:47.698 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-10-19T12:44:47.699 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T12:44:47.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.805+0000 7ff7d487c640 1 -- 172.21.15.57:0/199992395 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff7d0158910 msgr2=0x7ff7d0158cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:47.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.805+0000 7ff7d487c640 1 --2- 172.21.15.57:0/199992395 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff7d0158910 0x7ff7d0158cf0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ff7d0072c60 tx=0x7ff7c802be40 comp rx=0 tx=0).stop 2023-10-19T12:44:47.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.805+0000 7ff7d487c640 1 -- 172.21.15.57:0/199992395 shutdown_connections 2023-10-19T12:44:47.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.805+0000 7ff7d487c640 1 --2- 172.21.15.57:0/199992395 >> v2:172.21.15.179:3300/0 conn(0x7ff7d00cd9e0 0x7ff7d00cde50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:47.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.805+0000 7ff7d487c640 1 --2- 172.21.15.57:0/199992395 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff7d0158910 0x7ff7d0158cf0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:47.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.805+0000 7ff7d487c640 1 -- 172.21.15.57:0/199992395 >> 172.21.15.57:0/199992395 conn(0x7ff7d00af7a0 msgr2=0x7ff7d00afb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:47.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.805+0000 7ff7d487c640 1 -- 172.21.15.57:0/199992395 shutdown_connections 2023-10-19T12:44:47.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.805+0000 7ff7d487c640 1 -- 172.21.15.57:0/199992395 wait complete. 2023-10-19T12:44:47.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.806+0000 7ff7d487c640 1 Processor -- start 2023-10-19T12:44:47.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.806+0000 7ff7d487c640 1 -- start start 2023-10-19T12:44:47.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.807+0000 7ff7d487c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff7d00cd1c0 0x7ff7d014b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:47.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.807+0000 7ff7d487c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff7d00cd9e0 0x7ff7d0159a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:47.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.807+0000 7ff7d487c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff7d0158910 0x7ff7d0159f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:47.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.807+0000 7ff7d487c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d00c77a0 con 0x7ff7d0158910 2023-10-19T12:44:47.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.807+0000 7ff7d487c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d00b2410 con 0x7ff7d00cd1c0 2023-10-19T12:44:47.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.807+0000 7ff7d487c640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d00b26f0 con 0x7ff7d00cd9e0 2023-10-19T12:44:47.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.807+0000 7ff7ced9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff7d0158910 0x7ff7d0159f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:47.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.807+0000 7ff7ced9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff7d0158910 0x7ff7d0159f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59190/0 (socket says 172.21.15.57:59190) 2023-10-19T12:44:47.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.807+0000 7ff7ced9d640 1 -- 172.21.15.57:0/3327140896 learned_addr learned my addr 172.21.15.57:0/3327140896 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:47.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.807+0000 7ff7ce59c640 1 --2- 172.21.15.57:0/3327140896 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff7d00cd1c0 0x7ff7d014b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:47.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.807+0000 7ff7cdd9b640 1 --2- 172.21.15.57:0/3327140896 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff7d00cd9e0 0x7ff7d0159a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:47.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.807+0000 7ff7ced9d640 1 -- 172.21.15.57:0/3327140896 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff7d00cd9e0 msgr2=0x7ff7d0159a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:47.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.807+0000 7ff7ced9d640 1 --2- 172.21.15.57:0/3327140896 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff7d00cd9e0 0x7ff7d0159a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:47.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.807+0000 7ff7ced9d640 1 -- 172.21.15.57:0/3327140896 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff7d00cd1c0 msgr2=0x7ff7d014b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:47.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.807+0000 7ff7ced9d640 1 --2- 172.21.15.57:0/3327140896 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff7d00cd1c0 0x7ff7d014b4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:47.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.807+0000 7ff7ced9d640 1 -- 172.21.15.57:0/3327140896 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7c8009560 con 0x7ff7d0158910 2023-10-19T12:44:47.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.808+0000 7ff7ced9d640 1 --2- 172.21.15.57:0/3327140896 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff7d0158910 0x7ff7d0159f40 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ff7c000eaa0 tx=0x7ff7c000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:47.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.808+0000 7ff7beffd640 1 -- 172.21.15.57:0/3327140896 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7c000cdb0 con 0x7ff7d0158910 2023-10-19T12:44:47.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.808+0000 7ff7beffd640 1 -- 172.21.15.57:0/3327140896 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7ff7c0004b20 con 0x7ff7d0158910 2023-10-19T12:44:47.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.808+0000 7ff7d487c640 1 -- 172.21.15.57:0/3327140896 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7d00c79b0 con 0x7ff7d0158910 2023-10-19T12:44:47.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.808+0000 7ff7beffd640 1 -- 172.21.15.57:0/3327140896 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7c0005020 con 0x7ff7d0158910 2023-10-19T12:44:47.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.808+0000 7ff7d487c640 1 -- 172.21.15.57:0/3327140896 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7d014ba20 con 0x7ff7d0158910 2023-10-19T12:44:47.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.809+0000 7ff7d487c640 1 -- 172.21.15.57:0/3327140896 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7d0158cf0 con 0x7ff7d0158910 2023-10-19T12:44:47.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.809+0000 7ff7beffd640 1 -- 172.21.15.57:0/3327140896 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7ff7c0005180 con 0x7ff7d0158910 2023-10-19T12:44:47.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.810+0000 7ff7beffd640 1 --2- 172.21.15.57:0/3327140896 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7ff7b006ae40 0x7ff7b006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:47.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.810+0000 7ff7beffd640 1 -- 172.21.15.57:0/3327140896 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7ff7c008a310 con 0x7ff7d0158910 2023-10-19T12:44:47.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.810+0000 7ff7ce59c640 1 --2- 172.21.15.57:0/3327140896 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7ff7b006ae40 0x7ff7b006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:47.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.811+0000 7ff7ce59c640 1 --2- 172.21.15.57:0/3327140896 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7ff7b006ae40 0x7ff7b006d300 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ff7c8035000 tx=0x7ff7c8004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:47.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.814+0000 7ff7beffd640 1 -- 172.21.15.57:0/3327140896 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff7c0055160 con 0x7ff7d0158910 2023-10-19T12:44:47.992 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.989+0000 7ff7d487c640 1 -- 172.21.15.57:0/3327140896 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff7d00690e0 con 0x7ff7d0158910 2023-10-19T12:44:47.993 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.990+0000 7ff7beffd640 1 -- 172.21.15.57:0/3327140896 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18358 (secure 0 0 0) 0x7ff7c0058780 con 0x7ff7d0158910 2023-10-19T12:44:47.994 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:44:47.994 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":25,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T12:44:23.472399+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-19T12:44:16.517510+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-19T12:44:17.398297+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-19T12:44:21.129403+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T12:44:47.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.992+0000 7ff7d487c640 1 -- 172.21.15.57:0/3327140896 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7ff7b006ae40 msgr2=0x7ff7b006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:47.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.992+0000 7ff7d487c640 1 --2- 172.21.15.57:0/3327140896 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7ff7b006ae40 0x7ff7b006d300 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ff7c8035000 tx=0x7ff7c8004730 comp rx=0 tx=0).stop 2023-10-19T12:44:47.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.992+0000 7ff7d487c640 1 -- 172.21.15.57:0/3327140896 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff7d0158910 msgr2=0x7ff7d0159f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:47.997 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.992+0000 7ff7d487c640 1 --2- 172.21.15.57:0/3327140896 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff7d0158910 0x7ff7d0159f40 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ff7c000eaa0 tx=0x7ff7c000ef60 comp rx=0 tx=0).stop 2023-10-19T12:44:47.997 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.992+0000 7ff7d487c640 1 -- 172.21.15.57:0/3327140896 shutdown_connections 2023-10-19T12:44:47.997 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.992+0000 7ff7d487c640 1 --2- 172.21.15.57:0/3327140896 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7ff7b006ae40 0x7ff7b006d300 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:47.997 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.992+0000 7ff7d487c640 1 --2- 172.21.15.57:0/3327140896 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff7d0158910 0x7ff7d0159f40 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:47.997 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.992+0000 7ff7d487c640 1 --2- 172.21.15.57:0/3327140896 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff7d00cd9e0 0x7ff7d0159a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:47.997 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.992+0000 7ff7d487c640 1 --2- 172.21.15.57:0/3327140896 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff7d00cd1c0 0x7ff7d014b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:47.998 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.992+0000 7ff7d487c640 1 -- 172.21.15.57:0/3327140896 >> 172.21.15.57:0/3327140896 conn(0x7ff7d00af7a0 msgr2=0x7ff7d0070a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:47.998 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.993+0000 7ff7d487c640 1 -- 172.21.15.57:0/3327140896 shutdown_connections 2023-10-19T12:44:47.998 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:47.993+0000 7ff7d487c640 1 -- 172.21.15.57:0/3327140896 wait complete. 2023-10-19T12:44:48.006 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-19T12:44:07.371186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-19T12:44:16.517510+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-19T12:44:17.398297+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-19T12:44:21.129403+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-19T12:44:48.007 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-19T12:44:48.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.117+0000 7f7b2576c640 1 -- 172.21.15.57:0/1642491497 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7b200cbd20 msgr2=0x7f7b20158300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:48.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.117+0000 7f7b2576c640 1 --2- 172.21.15.57:0/1642491497 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7b200cbd20 0x7f7b20158300 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f7b10009830 tx=0x7f7b1002bd60 comp rx=0 tx=0).stop 2023-10-19T12:44:48.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.117+0000 7f7b2576c640 1 -- 172.21.15.57:0/1642491497 shutdown_connections 2023-10-19T12:44:48.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.117+0000 7f7b2576c640 1 --2- 172.21.15.57:0/1642491497 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7b200b0530 0x7f7b200b09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:48.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.117+0000 7f7b2576c640 1 --2- 172.21.15.57:0/1642491497 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7b200cbd20 0x7f7b20158300 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:48.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.117+0000 7f7b2576c640 1 -- 172.21.15.57:0/1642491497 >> 172.21.15.57:0/1642491497 conn(0x7f7b20069930 msgr2=0x7f7b20069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:48.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.117+0000 7f7b2576c640 1 -- 172.21.15.57:0/1642491497 shutdown_connections 2023-10-19T12:44:48.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.117+0000 7f7b2576c640 1 -- 172.21.15.57:0/1642491497 wait complete. 2023-10-19T12:44:48.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.118+0000 7f7b2576c640 1 Processor -- start 2023-10-19T12:44:48.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.118+0000 7f7b2576c640 1 -- start start 2023-10-19T12:44:48.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.119+0000 7f7b2576c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7b200afa20 0x7f7b20145e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:48.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.119+0000 7f7b2576c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7b200b0530 0x7f7b20149880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:48.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.119+0000 7f7b2576c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7b200cbd20 0x7f7b2014cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:48.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.119+0000 7f7b2576c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b2014c910 con 0x7f7b200b0530 2023-10-19T12:44:48.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.119+0000 7f7b2576c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b201463c0 con 0x7f7b200cbd20 2023-10-19T12:44:48.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.119+0000 7f7b2576c640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b201468e0 con 0x7f7b200afa20 2023-10-19T12:44:48.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.119+0000 7f7b17fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7b200b0530 0x7f7b20149880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:48.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.119+0000 7f7b17fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7b200b0530 0x7f7b20149880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59214/0 (socket says 172.21.15.57:59214) 2023-10-19T12:44:48.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.119+0000 7f7b17fff640 1 -- 172.21.15.57:0/509456637 learned_addr learned my addr 172.21.15.57:0/509456637 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:48.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.119+0000 7f7b1effd640 1 --2- 172.21.15.57:0/509456637 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7b200afa20 0x7f7b20145e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:48.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.119+0000 7f7b1f7fe640 1 --2- 172.21.15.57:0/509456637 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7b200cbd20 0x7f7b2014cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:48.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.119+0000 7f7b17fff640 1 -- 172.21.15.57:0/509456637 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7b200afa20 msgr2=0x7f7b20145e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:48.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.119+0000 7f7b17fff640 1 --2- 172.21.15.57:0/509456637 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7b200afa20 0x7f7b20145e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:48.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.119+0000 7f7b17fff640 1 -- 172.21.15.57:0/509456637 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7b200cbd20 msgr2=0x7f7b2014cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:48.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.119+0000 7f7b17fff640 1 --2- 172.21.15.57:0/509456637 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7b200cbd20 0x7f7b2014cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:48.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.119+0000 7f7b17fff640 1 -- 172.21.15.57:0/509456637 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b10009490 con 0x7f7b200b0530 2023-10-19T12:44:48.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.119+0000 7f7b1f7fe640 1 --2- 172.21.15.57:0/509456637 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7b200cbd20 0x7f7b2014cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:44:48.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.119+0000 7f7b1effd640 1 --2- 172.21.15.57:0/509456637 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7b200afa20 0x7f7b20145e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:44:48.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.119+0000 7f7b17fff640 1 --2- 172.21.15.57:0/509456637 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7b200b0530 0x7f7b20149880 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f7b0800c530 tx=0x7f7b0800c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:48.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.119+0000 7f7b177fe640 1 -- 172.21.15.57:0/509456637 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b08012830 con 0x7f7b200b0530 2023-10-19T12:44:48.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.119+0000 7f7b177fe640 1 -- 172.21.15.57:0/509456637 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f7b08012a80 con 0x7f7b200b0530 2023-10-19T12:44:48.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.119+0000 7f7b177fe640 1 -- 172.21.15.57:0/509456637 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b08012e50 con 0x7f7b200b0530 2023-10-19T12:44:48.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.119+0000 7f7b2576c640 1 -- 172.21.15.57:0/509456637 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b2014bc90 con 0x7f7b200b0530 2023-10-19T12:44:48.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.120+0000 7f7b2576c640 1 -- 172.21.15.57:0/509456637 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b2014c250 con 0x7f7b200b0530 2023-10-19T12:44:48.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.121+0000 7f7b2576c640 1 -- 172.21.15.57:0/509456637 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b200690e0 con 0x7f7b200b0530 2023-10-19T12:44:48.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.121+0000 7f7b177fe640 1 -- 172.21.15.57:0/509456637 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f7b0801b400 con 0x7f7b200b0530 2023-10-19T12:44:48.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.121+0000 7f7b177fe640 1 --2- 172.21.15.57:0/509456637 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f7b0006ae40 0x7f7b0006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:48.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.121+0000 7f7b177fe640 1 -- 172.21.15.57:0/509456637 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f7b08089cb0 con 0x7f7b200b0530 2023-10-19T12:44:48.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.122+0000 7f7b1effd640 1 --2- 172.21.15.57:0/509456637 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f7b0006ae40 0x7f7b0006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:48.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.122+0000 7f7b1effd640 1 --2- 172.21.15.57:0/509456637 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f7b0006ae40 0x7f7b0006d300 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7b10036000 tx=0x7f7b10004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:48.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.126+0000 7f7b177fe640 1 -- 172.21.15.57:0/509456637 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7b08058500 con 0x7f7b200b0530 2023-10-19T12:44:48.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.305+0000 7f7b2576c640 1 -- 172.21.15.57:0/509456637 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7b2006f700 con 0x7f7b200b0530 2023-10-19T12:44:48.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.305+0000 7f7b177fe640 1 -- 172.21.15.57:0/509456637 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7b08016020 con 0x7f7b200b0530 2023-10-19T12:44:48.309 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-19T12:44:48.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.308+0000 7f7b2576c640 1 -- 172.21.15.57:0/509456637 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f7b0006ae40 msgr2=0x7f7b0006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:48.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.308+0000 7f7b2576c640 1 --2- 172.21.15.57:0/509456637 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f7b0006ae40 0x7f7b0006d300 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7b10036000 tx=0x7f7b10004730 comp rx=0 tx=0).stop 2023-10-19T12:44:48.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.308+0000 7f7b2576c640 1 -- 172.21.15.57:0/509456637 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7b200b0530 msgr2=0x7f7b20149880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:48.312 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.308+0000 7f7b2576c640 1 --2- 172.21.15.57:0/509456637 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7b200b0530 0x7f7b20149880 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f7b0800c530 tx=0x7f7b0800c9f0 comp rx=0 tx=0).stop 2023-10-19T12:44:48.312 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.308+0000 7f7b2576c640 1 -- 172.21.15.57:0/509456637 shutdown_connections 2023-10-19T12:44:48.312 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.308+0000 7f7b2576c640 1 --2- 172.21.15.57:0/509456637 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f7b0006ae40 0x7f7b0006d300 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:48.312 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.308+0000 7f7b2576c640 1 --2- 172.21.15.57:0/509456637 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7b200cbd20 0x7f7b2014cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:48.312 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.308+0000 7f7b2576c640 1 --2- 172.21.15.57:0/509456637 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7b200b0530 0x7f7b20149880 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:48.313 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.308+0000 7f7b2576c640 1 --2- 172.21.15.57:0/509456637 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7b200afa20 0x7f7b20145e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:48.313 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.308+0000 7f7b2576c640 1 -- 172.21.15.57:0/509456637 >> 172.21.15.57:0/509456637 conn(0x7f7b20069930 msgr2=0x7f7b200ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:48.313 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.308+0000 7f7b2576c640 1 -- 172.21.15.57:0/509456637 shutdown_connections 2023-10-19T12:44:48.313 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.308+0000 7f7b2576c640 1 -- 172.21.15.57:0/509456637 wait complete. 2023-10-19T12:44:48.321 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-19T12:44:48.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.430+0000 7f64ed064640 1 -- 172.21.15.57:0/3928037507 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f64d4003970 msgr2=0x7f64d4003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:48.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.430+0000 7f64ed064640 1 --2- 172.21.15.57:0/3928037507 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f64d4003970 0x7f64d4003d90 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f64e80c9790 tx=0x7f64d40057f0 comp rx=0 tx=0).stop 2023-10-19T12:44:48.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.430+0000 7f64e759e640 1 -- 172.21.15.57:0/3928037507 reap_dead start 2023-10-19T12:44:48.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.430+0000 7f64ed064640 1 -- 172.21.15.57:0/3928037507 shutdown_connections 2023-10-19T12:44:48.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.430+0000 7f64ed064640 1 -- 172.21.15.57:0/3928037507 >> 172.21.15.57:0/3928037507 conn(0x7f64e8069930 msgr2=0x7f64e8069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:48.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.430+0000 7f64ed064640 1 -- 172.21.15.57:0/3928037507 shutdown_connections 2023-10-19T12:44:48.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.430+0000 7f64ed064640 1 -- 172.21.15.57:0/3928037507 wait complete. 2023-10-19T12:44:48.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.432+0000 7f64ed064640 1 Processor -- start 2023-10-19T12:44:48.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.432+0000 7f64ed064640 1 -- start start 2023-10-19T12:44:48.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.432+0000 7f64ed064640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f64d4003970 0x7f64e80a8dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:48.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.432+0000 7f64ed064640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f64e813ae10 0x7f64e813b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:48.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.432+0000 7f64ed064640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f64e81e1ff0 0x7f64e81e24a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:48.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.432+0000 7f64ed064640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64e813b7d0 con 0x7f64e813ae10 2023-10-19T12:44:48.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.432+0000 7f64ed064640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64e81e2d20 con 0x7f64d4003970 2023-10-19T12:44:48.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.432+0000 7f64ed064640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64e81e3000 con 0x7f64e81e1ff0 2023-10-19T12:44:48.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.433+0000 7f64e659c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f64e813ae10 0x7f64e813b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:48.439 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.433+0000 7f64e659c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f64e813ae10 0x7f64e813b290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59230/0 (socket says 172.21.15.57:59230) 2023-10-19T12:44:48.439 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.433+0000 7f64e659c640 1 -- 172.21.15.57:0/419478088 learned_addr learned my addr 172.21.15.57:0/419478088 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:48.439 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.433+0000 7f64e759e640 1 --2- 172.21.15.57:0/419478088 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f64e81e1ff0 0x7f64e81e24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:48.439 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.433+0000 7f64e6d9d640 1 --2- 172.21.15.57:0/419478088 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f64d4003970 0x7f64e80a8dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:48.439 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.433+0000 7f64e659c640 1 -- 172.21.15.57:0/419478088 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f64e81e1ff0 msgr2=0x7f64e81e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:48.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.433+0000 7f64e659c640 1 --2- 172.21.15.57:0/419478088 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f64e81e1ff0 0x7f64e81e24a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:48.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.433+0000 7f64e659c640 1 -- 172.21.15.57:0/419478088 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f64d4003970 msgr2=0x7f64e80a8dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:48.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.433+0000 7f64e659c640 1 --2- 172.21.15.57:0/419478088 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f64d4003970 0x7f64e80a8dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:48.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.433+0000 7f64e659c640 1 -- 172.21.15.57:0/419478088 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64e80a6710 con 0x7f64e813ae10 2023-10-19T12:44:48.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.433+0000 7f64e759e640 1 --2- 172.21.15.57:0/419478088 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f64e81e1ff0 0x7f64e81e24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:44:48.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.433+0000 7f64e659c640 1 --2- 172.21.15.57:0/419478088 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f64e813ae10 0x7f64e813b290 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f64d0011ab0 tx=0x7f64d0011f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:48.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.433+0000 7f64df7fe640 1 -- 172.21.15.57:0/419478088 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64d000fc70 con 0x7f64e813ae10 2023-10-19T12:44:48.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.433+0000 7f64df7fe640 1 -- 172.21.15.57:0/419478088 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f64d000fdd0 con 0x7f64e813ae10 2023-10-19T12:44:48.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.433+0000 7f64ed064640 1 -- 172.21.15.57:0/419478088 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64e80a69f0 con 0x7f64e813ae10 2023-10-19T12:44:48.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.433+0000 7f64df7fe640 1 -- 172.21.15.57:0/419478088 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64d0019670 con 0x7f64e813ae10 2023-10-19T12:44:48.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.433+0000 7f64ed064640 1 -- 172.21.15.57:0/419478088 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64e81e29e0 con 0x7f64e813ae10 2023-10-19T12:44:48.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.434+0000 7f64ed064640 1 -- 172.21.15.57:0/419478088 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64e80c9dd0 con 0x7f64e813ae10 2023-10-19T12:44:48.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.435+0000 7f64df7fe640 1 -- 172.21.15.57:0/419478088 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f64d00197d0 con 0x7f64e813ae10 2023-10-19T12:44:48.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.435+0000 7f64df7fe640 1 --2- 172.21.15.57:0/419478088 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f64c806ae40 0x7f64c806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:48.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.435+0000 7f64df7fe640 1 -- 172.21.15.57:0/419478088 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f64d008a450 con 0x7f64e813ae10 2023-10-19T12:44:48.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.435+0000 7f64e6d9d640 1 --2- 172.21.15.57:0/419478088 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f64c806ae40 0x7f64c806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:48.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.436+0000 7f64e6d9d640 1 --2- 172.21.15.57:0/419478088 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f64c806ae40 0x7f64c806d300 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f64e0037000 tx=0x7f64e002e040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:48.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.439+0000 7f64df7fe640 1 -- 172.21.15.57:0/419478088 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f64d0055220 con 0x7f64e813ae10 2023-10-19T12:44:48.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.614+0000 7f64ed064640 1 -- 172.21.15.57:0/419478088 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f64e80690e0 con 0x7f64e813ae10 2023-10-19T12:44:48.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.614+0000 7f64df7fe640 1 -- 172.21.15.57:0/419478088 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) v1 ==== 104+0+11 (secure 0 0 0) 0x7f64d0058840 con 0x7f64e813ae10 2023-10-19T12:44:48.618 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2023-10-19T12:44:48.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.616+0000 7f64ed064640 1 -- 172.21.15.57:0/419478088 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f64c806ae40 msgr2=0x7f64c806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:48.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.616+0000 7f64ed064640 1 --2- 172.21.15.57:0/419478088 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f64c806ae40 0x7f64c806d300 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f64e0037000 tx=0x7f64e002e040 comp rx=0 tx=0).stop 2023-10-19T12:44:48.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.616+0000 7f64ed064640 1 -- 172.21.15.57:0/419478088 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f64e813ae10 msgr2=0x7f64e813b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:48.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.616+0000 7f64ed064640 1 --2- 172.21.15.57:0/419478088 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f64e813ae10 0x7f64e813b290 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f64d0011ab0 tx=0x7f64d0011f70 comp rx=0 tx=0).stop 2023-10-19T12:44:48.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.616+0000 7f64ed064640 1 -- 172.21.15.57:0/419478088 shutdown_connections 2023-10-19T12:44:48.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.616+0000 7f64ed064640 1 --2- 172.21.15.57:0/419478088 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f64c806ae40 0x7f64c806d300 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:48.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.616+0000 7f64ed064640 1 --2- 172.21.15.57:0/419478088 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f64e81e1ff0 0x7f64e81e24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:48.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.616+0000 7f64ed064640 1 --2- 172.21.15.57:0/419478088 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f64e813ae10 0x7f64e813b290 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:48.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.616+0000 7f64ed064640 1 --2- 172.21.15.57:0/419478088 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f64d4003970 0x7f64e80a8dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:48.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.616+0000 7f64ed064640 1 -- 172.21.15.57:0/419478088 >> 172.21.15.57:0/419478088 conn(0x7f64e8069930 msgr2=0x7f64e8069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:48.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.616+0000 7f64ed064640 1 -- 172.21.15.57:0/419478088 shutdown_connections 2023-10-19T12:44:48.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.616+0000 7f64ed064640 1 -- 172.21.15.57:0/419478088 wait complete. 2023-10-19T12:44:48.630 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-19T12:44:48.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.739+0000 7f6cdf64b640 1 -- 172.21.15.57:0/2933485056 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6cd8070920 msgr2=0x7f6cd80affe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:48.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.739+0000 7f6cdf64b640 1 --2- 172.21.15.57:0/2933485056 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6cd8070920 0x7f6cd80affe0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f6cd80cbdc0 tx=0x7f6cd002c080 comp rx=0 tx=0).stop 2023-10-19T12:44:48.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.739+0000 7f6cdf64b640 1 -- 172.21.15.57:0/2933485056 shutdown_connections 2023-10-19T12:44:48.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.739+0000 7f6cdf64b640 1 --2- 172.21.15.57:0/2933485056 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6cd8070920 0x7f6cd80affe0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:48.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.739+0000 7f6cdf64b640 1 --2- 172.21.15.57:0/2933485056 >> v2:172.21.15.179:3300/0 conn(0x7f6cd8141690 0x7f6cd80703e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:48.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.739+0000 7f6cdf64b640 1 -- 172.21.15.57:0/2933485056 >> 172.21.15.57:0/2933485056 conn(0x7f6cd8069930 msgr2=0x7f6cd8069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:48.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.739+0000 7f6cdf64b640 1 -- 172.21.15.57:0/2933485056 shutdown_connections 2023-10-19T12:44:48.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.739+0000 7f6cdf64b640 1 -- 172.21.15.57:0/2933485056 wait complete. 2023-10-19T12:44:48.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.740+0000 7f6cdf64b640 1 Processor -- start 2023-10-19T12:44:48.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.740+0000 7f6cdf64b640 1 -- start start 2023-10-19T12:44:48.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.741+0000 7f6cdf64b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6cd8070920 0x7f6cd81e3e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:48.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.741+0000 7f6cdf64b640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6cd813e520 0x7f6cd813b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:48.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.741+0000 7f6cdf64b640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6cd8141690 0x7f6cd81e2240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:48.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.741+0000 7f6cdf64b640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cd81e3980 con 0x7f6cd8070920 2023-10-19T12:44:48.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.741+0000 7f6cdf64b640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cd81e2d70 con 0x7f6cd813e520 2023-10-19T12:44:48.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.741+0000 7f6cdf64b640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cd81e3050 con 0x7f6cd8141690 2023-10-19T12:44:48.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.741+0000 7f6cdd3e7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6cd8070920 0x7f6cd81e3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:48.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.741+0000 7f6cdd3e7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6cd8070920 0x7f6cd81e3e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59260/0 (socket says 172.21.15.57:59260) 2023-10-19T12:44:48.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.741+0000 7f6cdd3e7640 1 -- 172.21.15.57:0/772156012 learned_addr learned my addr 172.21.15.57:0/772156012 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:48.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.741+0000 7f6cddbe8640 1 --2- 172.21.15.57:0/772156012 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6cd8141690 0x7f6cd81e2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:48.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.741+0000 7f6cdcbe6640 1 --2- 172.21.15.57:0/772156012 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6cd813e520 0x7f6cd813b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:48.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.741+0000 7f6cdd3e7640 1 -- 172.21.15.57:0/772156012 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6cd8141690 msgr2=0x7f6cd81e2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:48.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.741+0000 7f6cdd3e7640 1 --2- 172.21.15.57:0/772156012 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6cd8141690 0x7f6cd81e2240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:48.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.741+0000 7f6cdd3e7640 1 -- 172.21.15.57:0/772156012 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6cd813e520 msgr2=0x7f6cd813b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:48.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.741+0000 7f6cdd3e7640 1 --2- 172.21.15.57:0/772156012 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6cd813e520 0x7f6cd813b1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:48.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.741+0000 7f6cdd3e7640 1 -- 172.21.15.57:0/772156012 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cc8009560 con 0x7f6cd8070920 2023-10-19T12:44:48.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.741+0000 7f6cdd3e7640 1 --2- 172.21.15.57:0/772156012 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6cd8070920 0x7f6cd81e3e10 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f6cd400c490 tx=0x7f6cd400c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:48.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.742+0000 7f6ccdffb640 1 -- 172.21.15.57:0/772156012 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd40052d0 con 0x7f6cd8070920 2023-10-19T12:44:48.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.742+0000 7f6ccdffb640 1 -- 172.21.15.57:0/772156012 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f6cd4005430 con 0x7f6cd8070920 2023-10-19T12:44:48.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.742+0000 7f6cdf64b640 1 -- 172.21.15.57:0/772156012 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cd0009840 con 0x7f6cd8070920 2023-10-19T12:44:48.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.742+0000 7f6ccdffb640 1 -- 172.21.15.57:0/772156012 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd4012890 con 0x7f6cd8070920 2023-10-19T12:44:48.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.742+0000 7f6cdf64b640 1 -- 172.21.15.57:0/772156012 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cd81e3610 con 0x7f6cd8070920 2023-10-19T12:44:48.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.743+0000 7f6cdf64b640 1 -- 172.21.15.57:0/772156012 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cd80690e0 con 0x7f6cd8070920 2023-10-19T12:44:48.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.743+0000 7f6ccdffb640 1 -- 172.21.15.57:0/772156012 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f6cd40035f0 con 0x7f6cd8070920 2023-10-19T12:44:48.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.744+0000 7f6ccdffb640 1 --2- 172.21.15.57:0/772156012 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f6cbc06ae70 0x7f6cbc06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:48.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.744+0000 7f6ccdffb640 1 -- 172.21.15.57:0/772156012 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f6cd40895b0 con 0x7f6cd8070920 2023-10-19T12:44:48.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.744+0000 7f6cdcbe6640 1 --2- 172.21.15.57:0/772156012 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f6cbc06ae70 0x7f6cbc06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:48.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.744+0000 7f6cdcbe6640 1 --2- 172.21.15.57:0/772156012 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f6cbc06ae70 0x7f6cbc06d330 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f6cd80cb710 tx=0x7f6cc8009260 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:48.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.748+0000 7f6ccdffb640 1 -- 172.21.15.57:0/772156012 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6cd4054400 con 0x7f6cd8070920 2023-10-19T12:44:48.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.924+0000 7f6cdf64b640 1 -- 172.21.15.57:0/772156012 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f6cd80c9bf0 con 0x7f6cd8070920 2023-10-19T12:44:48.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.924+0000 7f6ccdffb640 1 -- 172.21.15.57:0/772156012 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) v1 ==== 100+0+11 (secure 0 0 0) 0x7f6cd4057a20 con 0x7f6cd8070920 2023-10-19T12:44:48.928 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 64 2023-10-19T12:44:48.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.926+0000 7f6cdf64b640 1 -- 172.21.15.57:0/772156012 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f6cbc06ae70 msgr2=0x7f6cbc06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:48.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.926+0000 7f6cdf64b640 1 --2- 172.21.15.57:0/772156012 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f6cbc06ae70 0x7f6cbc06d330 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f6cd80cb710 tx=0x7f6cc8009260 comp rx=0 tx=0).stop 2023-10-19T12:44:48.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.926+0000 7f6cdf64b640 1 -- 172.21.15.57:0/772156012 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6cd8070920 msgr2=0x7f6cd81e3e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:48.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.926+0000 7f6cdf64b640 1 --2- 172.21.15.57:0/772156012 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6cd8070920 0x7f6cd81e3e10 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f6cd400c490 tx=0x7f6cd400c950 comp rx=0 tx=0).stop 2023-10-19T12:44:48.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.926+0000 7f6cdf64b640 1 -- 172.21.15.57:0/772156012 shutdown_connections 2023-10-19T12:44:48.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.926+0000 7f6cdf64b640 1 --2- 172.21.15.57:0/772156012 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f6cbc06ae70 0x7f6cbc06d330 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:48.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.926+0000 7f6cdf64b640 1 --2- 172.21.15.57:0/772156012 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6cd8141690 0x7f6cd81e2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:48.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.926+0000 7f6cdf64b640 1 --2- 172.21.15.57:0/772156012 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6cd813e520 0x7f6cd813b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:48.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.926+0000 7f6cdf64b640 1 --2- 172.21.15.57:0/772156012 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6cd8070920 0x7f6cd81e3e10 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:48.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.926+0000 7f6cdf64b640 1 -- 172.21.15.57:0/772156012 >> 172.21.15.57:0/772156012 conn(0x7f6cd8069930 msgr2=0x7f6cd80b0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:48.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.926+0000 7f6cdf64b640 1 -- 172.21.15.57:0/772156012 shutdown_connections 2023-10-19T12:44:48.932 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:48.926+0000 7f6cdf64b640 1 -- 172.21.15.57:0/772156012 wait complete. 2023-10-19T12:44:48.940 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-19T12:44:49.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.048+0000 7fcdb2f4c640 1 -- 172.21.15.57:0/35624686 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcdac069e30 msgr2=0x7fcdac0cdb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:49.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.048+0000 7fcdb2f4c640 1 --2- 172.21.15.57:0/35624686 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcdac069e30 0x7fcdac0cdb30 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fcdac072420 tx=0x7fcda402c260 comp rx=0 tx=0).stop 2023-10-19T12:44:49.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.048+0000 7fcdb2f4c640 1 -- 172.21.15.57:0/35624686 shutdown_connections 2023-10-19T12:44:49.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.049+0000 7fcdb2f4c640 1 --2- 172.21.15.57:0/35624686 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcdac069e30 0x7fcdac0cdb30 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:49.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.049+0000 7fcdb2f4c640 1 --2- 172.21.15.57:0/35624686 >> v2:172.21.15.179:3300/0 conn(0x7fcdac142850 0x7fcdac159440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:49.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.049+0000 7fcdb2f4c640 1 -- 172.21.15.57:0/35624686 >> 172.21.15.57:0/35624686 conn(0x7fcdac0af7a0 msgr2=0x7fcdac0afb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:49.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.049+0000 7fcdb2f4c640 1 -- 172.21.15.57:0/35624686 shutdown_connections 2023-10-19T12:44:49.054 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.049+0000 7fcdb2f4c640 1 -- 172.21.15.57:0/35624686 wait complete. 2023-10-19T12:44:49.057 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.050+0000 7fcdb2f4c640 1 Processor -- start 2023-10-19T12:44:49.057 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.050+0000 7fcdb2f4c640 1 -- start start 2023-10-19T12:44:49.057 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.051+0000 7fcdb2f4c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcdac069e30 0x7fcdac158820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:49.057 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.051+0000 7fcdb2f4c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcdac0cd6d0 0x7fcdac0ab910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:49.058 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.051+0000 7fcdb2f4c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcdac142850 0x7fcdac0c3a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:49.058 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.051+0000 7fcdb2f4c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdac0ace90 con 0x7fcdac142850 2023-10-19T12:44:49.058 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.051+0000 7fcdb2f4c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdac158d60 con 0x7fcdac0cd6d0 2023-10-19T12:44:49.058 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.051+0000 7fcdb2f4c640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdac0ad2b0 con 0x7fcdac069e30 2023-10-19T12:44:49.058 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.051+0000 7fcdb14e9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcdac142850 0x7fcdac0c3a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:49.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.051+0000 7fcdb14e9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcdac142850 0x7fcdac0c3a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59288/0 (socket says 172.21.15.57:59288) 2023-10-19T12:44:49.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.051+0000 7fcdb14e9640 1 -- 172.21.15.57:0/3755929251 learned_addr learned my addr 172.21.15.57:0/3755929251 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:49.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.051+0000 7fcdabfff640 1 --2- 172.21.15.57:0/3755929251 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcdac0cd6d0 0x7fcdac0ab910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:49.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.051+0000 7fcdb0ce8640 1 --2- 172.21.15.57:0/3755929251 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcdac069e30 0x7fcdac158820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:49.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.051+0000 7fcdb14e9640 1 -- 172.21.15.57:0/3755929251 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcdac069e30 msgr2=0x7fcdac158820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:49.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.051+0000 7fcdb14e9640 1 --2- 172.21.15.57:0/3755929251 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcdac069e30 0x7fcdac158820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:49.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.051+0000 7fcdb14e9640 1 -- 172.21.15.57:0/3755929251 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcdac0cd6d0 msgr2=0x7fcdac0ab910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:49.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.051+0000 7fcdb14e9640 1 --2- 172.21.15.57:0/3755929251 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcdac0cd6d0 0x7fcdac0ab910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:49.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.051+0000 7fcdb14e9640 1 -- 172.21.15.57:0/3755929251 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcda4009900 con 0x7fcdac142850 2023-10-19T12:44:49.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.051+0000 7fcdb14e9640 1 --2- 172.21.15.57:0/3755929251 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcdac142850 0x7fcdac0c3a90 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fcda4037040 tx=0x7fcda4003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:49.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.051+0000 7fcda97fa640 1 -- 172.21.15.57:0/3755929251 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcda4004240 con 0x7fcdac142850 2023-10-19T12:44:49.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.051+0000 7fcda97fa640 1 -- 172.21.15.57:0/3755929251 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fcda4004490 con 0x7fcdac142850 2023-10-19T12:44:49.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.051+0000 7fcdb2f4c640 1 -- 172.21.15.57:0/3755929251 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdac0ac210 con 0x7fcdac142850 2023-10-19T12:44:49.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.051+0000 7fcda97fa640 1 -- 172.21.15.57:0/3755929251 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcda4004880 con 0x7fcdac142850 2023-10-19T12:44:49.062 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.052+0000 7fcdb2f4c640 1 -- 172.21.15.57:0/3755929251 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdac0ac7d0 con 0x7fcdac142850 2023-10-19T12:44:49.062 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.052+0000 7fcdb2f4c640 1 -- 172.21.15.57:0/3755929251 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdac0acfd0 con 0x7fcdac142850 2023-10-19T12:44:49.062 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.053+0000 7fcda97fa640 1 -- 172.21.15.57:0/3755929251 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fcda403b040 con 0x7fcdac142850 2023-10-19T12:44:49.062 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.053+0000 7fcda97fa640 1 --2- 172.21.15.57:0/3755929251 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fcd9006ae40 0x7fcd9006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:49.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.053+0000 7fcda97fa640 1 -- 172.21.15.57:0/3755929251 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fcda40ac4f0 con 0x7fcdac142850 2023-10-19T12:44:49.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.054+0000 7fcdb0ce8640 1 --2- 172.21.15.57:0/3755929251 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fcd9006ae40 0x7fcd9006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:49.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.054+0000 7fcdb0ce8640 1 --2- 172.21.15.57:0/3755929251 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fcd9006ae40 0x7fcd9006d300 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fcda0009b30 tx=0x7fcda0005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:49.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.057+0000 7fcda97fa640 1 -- 172.21.15.57:0/3755929251 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fcdac0acfd0 con 0x7fcdac142850 2023-10-19T12:44:49.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.232+0000 7fcdb2f4c640 1 -- 172.21.15.57:0/3755929251 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fcdac0acfd0 con 0x7fcdac142850 2023-10-19T12:44:49.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.232+0000 7fcda97fa640 1 -- 172.21.15.57:0/3755929251 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) v1 ==== 103+0+11 (secure 0 0 0) 0x7fcdac0acfd0 con 0x7fcdac142850 2023-10-19T12:44:49.236 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2023-10-19T12:44:49.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.234+0000 7fcdb2f4c640 1 -- 172.21.15.57:0/3755929251 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fcd9006ae40 msgr2=0x7fcd9006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:49.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.234+0000 7fcdb2f4c640 1 --2- 172.21.15.57:0/3755929251 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fcd9006ae40 0x7fcd9006d300 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fcda0009b30 tx=0x7fcda0005f90 comp rx=0 tx=0).stop 2023-10-19T12:44:49.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.234+0000 7fcdb2f4c640 1 -- 172.21.15.57:0/3755929251 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcdac142850 msgr2=0x7fcdac0c3a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:49.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.234+0000 7fcdb2f4c640 1 --2- 172.21.15.57:0/3755929251 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcdac142850 0x7fcdac0c3a90 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fcda4037040 tx=0x7fcda4003ba0 comp rx=0 tx=0).stop 2023-10-19T12:44:49.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.234+0000 7fcdb2f4c640 1 -- 172.21.15.57:0/3755929251 shutdown_connections 2023-10-19T12:44:49.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.234+0000 7fcdb2f4c640 1 --2- 172.21.15.57:0/3755929251 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fcd9006ae40 0x7fcd9006d300 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:49.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.234+0000 7fcdb2f4c640 1 --2- 172.21.15.57:0/3755929251 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcdac142850 0x7fcdac0c3a90 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:49.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.234+0000 7fcdb2f4c640 1 --2- 172.21.15.57:0/3755929251 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcdac0cd6d0 0x7fcdac0ab910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:49.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.234+0000 7fcdb2f4c640 1 --2- 172.21.15.57:0/3755929251 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcdac069e30 0x7fcdac158820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:49.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.234+0000 7fcdb2f4c640 1 -- 172.21.15.57:0/3755929251 >> 172.21.15.57:0/3755929251 conn(0x7fcdac0af7a0 msgr2=0x7fcdac15bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:49.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.234+0000 7fcdb2f4c640 1 -- 172.21.15.57:0/3755929251 shutdown_connections 2023-10-19T12:44:49.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.234+0000 7fcdb2f4c640 1 -- 172.21.15.57:0/3755929251 wait complete. 2023-10-19T12:44:49.248 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T12:44:49.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.355+0000 7f0b82056640 1 -- 172.21.15.57:0/2013397647 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0b64004d00 msgr2=0x7f0b64005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:49.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.355+0000 7f0b82056640 1 --2- 172.21.15.57:0/2013397647 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0b64004d00 0x7f0b64005180 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f0b74009f00 tx=0x7f0b640057f0 comp rx=0 tx=0).stop 2023-10-19T12:44:49.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.355+0000 7f0b7bfff640 1 -- 172.21.15.57:0/2013397647 reap_dead start 2023-10-19T12:44:49.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.355+0000 7f0b82056640 1 -- 172.21.15.57:0/2013397647 shutdown_connections 2023-10-19T12:44:49.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.355+0000 7f0b82056640 1 -- 172.21.15.57:0/2013397647 >> 172.21.15.57:0/2013397647 conn(0x7f0b7c069930 msgr2=0x7f0b7c069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:49.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.355+0000 7f0b82056640 1 -- 172.21.15.57:0/2013397647 shutdown_connections 2023-10-19T12:44:49.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.355+0000 7f0b82056640 1 -- 172.21.15.57:0/2013397647 wait complete. 2023-10-19T12:44:49.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.356+0000 7f0b82056640 1 Processor -- start 2023-10-19T12:44:49.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.356+0000 7f0b82056640 1 -- start start 2023-10-19T12:44:49.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.357+0000 7f0b82056640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0b64004d00 0x7f0b7c157e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:49.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.357+0000 7f0b82056640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0b7c14e960 0x7f0b7c14ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:49.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.357+0000 7f0b82056640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0b7c13e5b0 0x7f0b7c13ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:49.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.357+0000 7f0b82056640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b7c152cd0 con 0x7f0b7c14e960 2023-10-19T12:44:49.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.357+0000 7f0b82056640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b7c14f320 con 0x7f0b64004d00 2023-10-19T12:44:49.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.357+0000 7f0b82056640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b7c13ef70 con 0x7f0b7c13e5b0 2023-10-19T12:44:49.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.357+0000 7f0b7affd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0b7c14e960 0x7f0b7c14ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:49.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.357+0000 7f0b7affd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0b7c14e960 0x7f0b7c14ede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59304/0 (socket says 172.21.15.57:59304) 2023-10-19T12:44:49.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.357+0000 7f0b7affd640 1 -- 172.21.15.57:0/1076621201 learned_addr learned my addr 172.21.15.57:0/1076621201 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:49.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.357+0000 7f0b7b7fe640 1 --2- 172.21.15.57:0/1076621201 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0b64004d00 0x7f0b7c157e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:49.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.357+0000 7f0b7bfff640 1 --2- 172.21.15.57:0/1076621201 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0b7c13e5b0 0x7f0b7c13ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:49.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.357+0000 7f0b7affd640 1 -- 172.21.15.57:0/1076621201 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0b7c13e5b0 msgr2=0x7f0b7c13ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:49.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.357+0000 7f0b7affd640 1 --2- 172.21.15.57:0/1076621201 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0b7c13e5b0 0x7f0b7c13ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:49.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.357+0000 7f0b7affd640 1 -- 172.21.15.57:0/1076621201 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0b64004d00 msgr2=0x7f0b7c157e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:49.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.357+0000 7f0b7affd640 1 --2- 172.21.15.57:0/1076621201 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0b64004d00 0x7f0b7c157e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:49.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.357+0000 7f0b7affd640 1 -- 172.21.15.57:0/1076621201 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b7c152450 con 0x7f0b7c14e960 2023-10-19T12:44:49.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.358+0000 7f0b7affd640 1 --2- 172.21.15.57:0/1076621201 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0b7c14e960 0x7f0b7c14ede0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f0b70006fa0 tx=0x7f0b700039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:49.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.358+0000 7f0b63fff640 1 -- 172.21.15.57:0/1076621201 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b70011700 con 0x7f0b7c14e960 2023-10-19T12:44:49.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.358+0000 7f0b63fff640 1 -- 172.21.15.57:0/1076621201 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f0b70011860 con 0x7f0b7c14e960 2023-10-19T12:44:49.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.358+0000 7f0b63fff640 1 -- 172.21.15.57:0/1076621201 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b70011c50 con 0x7f0b7c14e960 2023-10-19T12:44:49.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.358+0000 7f0b82056640 1 -- 172.21.15.57:0/1076621201 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b7c152710 con 0x7f0b7c14e960 2023-10-19T12:44:49.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.358+0000 7f0b82056640 1 -- 172.21.15.57:0/1076621201 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b7c13c270 con 0x7f0b7c14e960 2023-10-19T12:44:49.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.359+0000 7f0b82056640 1 -- 172.21.15.57:0/1076621201 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b7c13c0d0 con 0x7f0b7c14e960 2023-10-19T12:44:49.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.359+0000 7f0b63fff640 1 -- 172.21.15.57:0/1076621201 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f0b70010420 con 0x7f0b7c14e960 2023-10-19T12:44:49.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.360+0000 7f0b63fff640 1 --2- 172.21.15.57:0/1076621201 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f0b6806ae70 0x7f0b6806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:49.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.360+0000 7f0b63fff640 1 -- 172.21.15.57:0/1076621201 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f0b7008a3d0 con 0x7f0b7c14e960 2023-10-19T12:44:49.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.360+0000 7f0b7b7fe640 1 --2- 172.21.15.57:0/1076621201 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f0b6806ae70 0x7f0b6806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:49.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.360+0000 7f0b7b7fe640 1 --2- 172.21.15.57:0/1076621201 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f0b6806ae70 0x7f0b6806d330 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f0b6c009c50 tx=0x7f0b6c005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:49.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.364+0000 7f0b63fff640 1 -- 172.21.15.57:0/1076621201 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0b7c13c0d0 con 0x7f0b7c14e960 2023-10-19T12:44:49.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.577+0000 7f0b82056640 1 -- 172.21.15.57:0/1076621201 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0b7c13c0d0 con 0x7f0b7c14e960 2023-10-19T12:44:49.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.578+0000 7f0b63fff640 1 -- 172.21.15.57:0/1076621201 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4906 (secure 0 0 0) 0x7f0b7c13c0d0 con 0x7f0b7c14e960 2023-10-19T12:44:49.581 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:44:49.582 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4357,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/4228712742","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":4228712742},{"type":"v1","addr":"172.21.15.57:6835","nonce":4228712742}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4360,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/971892630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":971892630},{"type":"v1","addr":"172.21.15.57:6834","nonce":971892630}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4372,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/104495983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":104495983},{"type":"v1","addr":"172.21.15.179:6836","nonce":104495983}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T12:44:18.419669+0000","modified":"2023-10-19T12:44:40.545928+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4374},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4374":{"gid":4374,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.179:6837/622550365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":622550365},{"type":"v1","addr":"172.21.15.179:6837","nonce":622550365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-19T12:44:49.582 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 9 2023-10-19T12:44:49.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.580+0000 7f0b82056640 1 -- 172.21.15.57:0/1076621201 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f0b6806ae70 msgr2=0x7f0b6806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:49.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.580+0000 7f0b82056640 1 --2- 172.21.15.57:0/1076621201 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f0b6806ae70 0x7f0b6806d330 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f0b6c009c50 tx=0x7f0b6c005f90 comp rx=0 tx=0).stop 2023-10-19T12:44:49.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.580+0000 7f0b82056640 1 -- 172.21.15.57:0/1076621201 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0b7c14e960 msgr2=0x7f0b7c14ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:49.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.580+0000 7f0b82056640 1 --2- 172.21.15.57:0/1076621201 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0b7c14e960 0x7f0b7c14ede0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f0b70006fa0 tx=0x7f0b700039c0 comp rx=0 tx=0).stop 2023-10-19T12:44:49.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.580+0000 7f0b82056640 1 -- 172.21.15.57:0/1076621201 shutdown_connections 2023-10-19T12:44:49.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.580+0000 7f0b82056640 1 --2- 172.21.15.57:0/1076621201 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f0b6806ae70 0x7f0b6806d330 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:49.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.580+0000 7f0b82056640 1 --2- 172.21.15.57:0/1076621201 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0b7c13e5b0 0x7f0b7c13ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:49.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.580+0000 7f0b82056640 1 --2- 172.21.15.57:0/1076621201 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0b7c14e960 0x7f0b7c14ede0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:49.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.580+0000 7f0b82056640 1 --2- 172.21.15.57:0/1076621201 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0b64004d00 0x7f0b7c157e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:49.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.580+0000 7f0b82056640 1 -- 172.21.15.57:0/1076621201 >> 172.21.15.57:0/1076621201 conn(0x7f0b7c069930 msgr2=0x7f0b7c0ca870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:49.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.580+0000 7f0b82056640 1 -- 172.21.15.57:0/1076621201 shutdown_connections 2023-10-19T12:44:49.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.580+0000 7f0b82056640 1 -- 172.21.15.57:0/1076621201 wait complete. 2023-10-19T12:44:49.594 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T12:44:49.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.701+0000 7f3da60e5640 1 -- 172.21.15.57:0/4124994571 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3d88004a80 msgr2=0x7f3d88004ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:49.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.701+0000 7f3da60e5640 1 --2- 172.21.15.57:0/4124994571 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3d88004a80 0x7f3d88004ea0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f3d90001680 tx=0x7f3d88006900 comp rx=0 tx=0).stop 2023-10-19T12:44:49.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.701+0000 7f3da60e5640 1 -- 172.21.15.57:0/4124994571 shutdown_connections 2023-10-19T12:44:49.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.701+0000 7f3da60e5640 1 --2- 172.21.15.57:0/4124994571 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3d88004a80 0x7f3d88004ea0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:49.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.701+0000 7f3da60e5640 1 -- 172.21.15.57:0/4124994571 >> 172.21.15.57:0/4124994571 conn(0x7f3da006fae0 msgr2=0x7f3da006fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:49.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.701+0000 7f3da60e5640 1 -- 172.21.15.57:0/4124994571 shutdown_connections 2023-10-19T12:44:49.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.701+0000 7f3da60e5640 1 -- 172.21.15.57:0/4124994571 wait complete. 2023-10-19T12:44:49.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.702+0000 7f3da60e5640 1 Processor -- start 2023-10-19T12:44:49.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.703+0000 7f3da60e5640 1 -- start start 2023-10-19T12:44:49.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.703+0000 7f3da60e5640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3d88004a80 0x7f3da0159150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:49.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.703+0000 7f3da60e5640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3da014c370 0x7f3da014a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:49.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.703+0000 7f3da60e5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3da014adc0 0x7f3da013c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:49.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.703+0000 7f3da60e5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3da014b2d0 con 0x7f3da014adc0 2023-10-19T12:44:49.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.703+0000 7f3da60e5640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3da014b5b0 con 0x7f3d88004a80 2023-10-19T12:44:49.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.703+0000 7f3da60e5640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3da014b890 con 0x7f3da014c370 2023-10-19T12:44:49.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.703+0000 7f3d9ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3da014adc0 0x7f3da013c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:49.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.703+0000 7f3d9ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3da014adc0 0x7f3da013c610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59330/0 (socket says 172.21.15.57:59330) 2023-10-19T12:44:49.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.703+0000 7f3d9ffff640 1 -- 172.21.15.57:0/3830107332 learned_addr learned my addr 172.21.15.57:0/3830107332 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:49.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.704+0000 7f3d9f7fe640 1 --2- 172.21.15.57:0/3830107332 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3d88004a80 0x7f3da0159150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:49.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.704+0000 7f3d9effd640 1 --2- 172.21.15.57:0/3830107332 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3da014c370 0x7f3da014a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:49.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.704+0000 7f3d9ffff640 1 -- 172.21.15.57:0/3830107332 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3da014c370 msgr2=0x7f3da014a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:49.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.704+0000 7f3d9ffff640 1 --2- 172.21.15.57:0/3830107332 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3da014c370 0x7f3da014a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:49.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.704+0000 7f3d9ffff640 1 -- 172.21.15.57:0/3830107332 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3d88004a80 msgr2=0x7f3da0159150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:49.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.704+0000 7f3d9ffff640 1 --2- 172.21.15.57:0/3830107332 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3d88004a80 0x7f3da0159150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:49.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.704+0000 7f3d9ffff640 1 -- 172.21.15.57:0/3830107332 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3da014bb70 con 0x7f3da014adc0 2023-10-19T12:44:49.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.704+0000 7f3d9ffff640 1 --2- 172.21.15.57:0/3830107332 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3da014adc0 0x7f3da013c610 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f3da006db70 tx=0x7f3d98011f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:49.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.704+0000 7f3d87fff640 1 -- 172.21.15.57:0/3830107332 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d98004b20 con 0x7f3da014adc0 2023-10-19T12:44:49.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.704+0000 7f3d87fff640 1 -- 172.21.15.57:0/3830107332 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f3d98004c80 con 0x7f3da014adc0 2023-10-19T12:44:49.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.704+0000 7f3da60e5640 1 -- 172.21.15.57:0/3830107332 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3da014be50 con 0x7f3da014adc0 2023-10-19T12:44:49.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.704+0000 7f3d87fff640 1 -- 172.21.15.57:0/3830107332 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d98003bd0 con 0x7f3da014adc0 2023-10-19T12:44:49.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.704+0000 7f3da60e5640 1 -- 172.21.15.57:0/3830107332 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3da0159690 con 0x7f3da014adc0 2023-10-19T12:44:49.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.705+0000 7f3da60e5640 1 -- 172.21.15.57:0/3830107332 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3da014c090 con 0x7f3da014adc0 2023-10-19T12:44:49.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.705+0000 7f3d87fff640 1 -- 172.21.15.57:0/3830107332 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f3d98003d30 con 0x7f3da014adc0 2023-10-19T12:44:49.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.706+0000 7f3d87fff640 1 --2- 172.21.15.57:0/3830107332 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f3d8c06ae40 0x7f3d8c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:49.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.706+0000 7f3d87fff640 1 -- 172.21.15.57:0/3830107332 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f3d98089db0 con 0x7f3da014adc0 2023-10-19T12:44:49.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.706+0000 7f3d9f7fe640 1 --2- 172.21.15.57:0/3830107332 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f3d8c06ae40 0x7f3d8c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:49.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.707+0000 7f3d9f7fe640 1 --2- 172.21.15.57:0/3830107332 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f3d8c06ae40 0x7f3d8c06d300 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f3d90034000 tx=0x7f3d90001190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:49.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.710+0000 7f3d87fff640 1 -- 172.21.15.57:0/3830107332 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3da014c090 con 0x7f3da014adc0 2023-10-19T12:44:49.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.886+0000 7f3da60e5640 1 -- 172.21.15.57:0/3830107332 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3da014c090 con 0x7f3da014adc0 2023-10-19T12:44:49.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.888+0000 7f3d87fff640 1 -- 172.21.15.57:0/3830107332 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18358 (secure 0 0 0) 0x7f3da014c090 con 0x7f3da014adc0 2023-10-19T12:44:49.891 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:44:49.891 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":25,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T12:44:23.472399+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-19T12:44:16.517510+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-19T12:44:17.398297+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-19T12:44:21.129403+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T12:44:49.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.889+0000 7f3da60e5640 1 -- 172.21.15.57:0/3830107332 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f3d8c06ae40 msgr2=0x7f3d8c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:49.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.889+0000 7f3da60e5640 1 --2- 172.21.15.57:0/3830107332 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f3d8c06ae40 0x7f3d8c06d300 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f3d90034000 tx=0x7f3d90001190 comp rx=0 tx=0).stop 2023-10-19T12:44:49.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.889+0000 7f3da60e5640 1 -- 172.21.15.57:0/3830107332 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3da014adc0 msgr2=0x7f3da013c610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:49.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.889+0000 7f3da60e5640 1 --2- 172.21.15.57:0/3830107332 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3da014adc0 0x7f3da013c610 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f3da006db70 tx=0x7f3d98011f40 comp rx=0 tx=0).stop 2023-10-19T12:44:49.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.889+0000 7f3da60e5640 1 -- 172.21.15.57:0/3830107332 shutdown_connections 2023-10-19T12:44:49.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.889+0000 7f3da60e5640 1 --2- 172.21.15.57:0/3830107332 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f3d8c06ae40 0x7f3d8c06d300 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:49.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.889+0000 7f3da60e5640 1 --2- 172.21.15.57:0/3830107332 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3da014adc0 0x7f3da013c610 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:49.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.889+0000 7f3da60e5640 1 --2- 172.21.15.57:0/3830107332 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3da014c370 0x7f3da014a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:49.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.890+0000 7f3da60e5640 1 --2- 172.21.15.57:0/3830107332 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3d88004a80 0x7f3da0159150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:49.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.890+0000 7f3da60e5640 1 -- 172.21.15.57:0/3830107332 >> 172.21.15.57:0/3830107332 conn(0x7f3da006fae0 msgr2=0x7f3da00b1850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:49.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.890+0000 7f3da60e5640 1 -- 172.21.15.57:0/3830107332 shutdown_connections 2023-10-19T12:44:49.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:49.890+0000 7f3da60e5640 1 -- 172.21.15.57:0/3830107332 wait complete. 2023-10-19T12:44:49.904 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-19T12:44:07.371186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-19T12:44:16.517510+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-19T12:44:17.398297+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-19T12:44:21.129403+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-19T12:44:49.905 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-19T12:44:50.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.014+0000 7fbc50064640 1 -- 172.21.15.57:0/3289424084 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbc4806bf20 msgr2=0x7fbc4806ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:50.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.014+0000 7fbc50064640 1 --2- 172.21.15.57:0/3289424084 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbc4806bf20 0x7fbc4806ae60 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fbc38009830 tx=0x7fbc3802bde0 comp rx=0 tx=0).stop 2023-10-19T12:44:50.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.014+0000 7fbc50064640 1 -- 172.21.15.57:0/3289424084 shutdown_connections 2023-10-19T12:44:50.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.014+0000 7fbc50064640 1 --2- 172.21.15.57:0/3289424084 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbc4806bf20 0x7fbc4806ae60 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:50.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.014+0000 7fbc50064640 1 --2- 172.21.15.57:0/3289424084 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbc480cd0c0 0x7fbc480cd4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:50.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.014+0000 7fbc50064640 1 -- 172.21.15.57:0/3289424084 >> 172.21.15.57:0/3289424084 conn(0x7fbc480b0670 msgr2=0x7fbc480b0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:50.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.014+0000 7fbc50064640 1 -- 172.21.15.57:0/3289424084 shutdown_connections 2023-10-19T12:44:50.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.014+0000 7fbc50064640 1 -- 172.21.15.57:0/3289424084 wait complete. 2023-10-19T12:44:50.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.015+0000 7fbc50064640 1 Processor -- start 2023-10-19T12:44:50.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.016+0000 7fbc50064640 1 -- start start 2023-10-19T12:44:50.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.016+0000 7fbc50064640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbc4806bf20 0x7fbc481523d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:50.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.016+0000 7fbc50064640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbc4806dce0 0x7fbc4806ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:50.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.016+0000 7fbc50064640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc480cd0c0 0x7fbc4806b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:50.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.016+0000 7fbc50064640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc4809a0b0 con 0x7fbc480cd0c0 2023-10-19T12:44:50.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.016+0000 7fbc50064640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc48152910 con 0x7fbc4806bf20 2023-10-19T12:44:50.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.016+0000 7fbc50064640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc48156480 con 0x7fbc4806dce0 2023-10-19T12:44:50.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.016+0000 7fbc4e601640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc480cd0c0 0x7fbc4806b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:50.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.016+0000 7fbc4e601640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc480cd0c0 0x7fbc4806b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59334/0 (socket says 172.21.15.57:59334) 2023-10-19T12:44:50.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.016+0000 7fbc4e601640 1 -- 172.21.15.57:0/3205854243 learned_addr learned my addr 172.21.15.57:0/3205854243 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:50.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.017+0000 7fbc4de00640 1 --2- 172.21.15.57:0/3205854243 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbc4806bf20 0x7fbc481523d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:50.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.017+0000 7fbc4d5ff640 1 --2- 172.21.15.57:0/3205854243 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbc4806dce0 0x7fbc4806ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:50.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.017+0000 7fbc4e601640 1 -- 172.21.15.57:0/3205854243 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbc4806dce0 msgr2=0x7fbc4806ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:50.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.017+0000 7fbc4e601640 1 --2- 172.21.15.57:0/3205854243 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbc4806dce0 0x7fbc4806ada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:50.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.017+0000 7fbc4e601640 1 -- 172.21.15.57:0/3205854243 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbc4806bf20 msgr2=0x7fbc481523d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:50.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.017+0000 7fbc4e601640 1 --2- 172.21.15.57:0/3205854243 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbc4806bf20 0x7fbc481523d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:50.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.017+0000 7fbc4e601640 1 -- 172.21.15.57:0/3205854243 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc38009490 con 0x7fbc480cd0c0 2023-10-19T12:44:50.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.017+0000 7fbc4de00640 1 --2- 172.21.15.57:0/3205854243 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbc4806bf20 0x7fbc481523d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:44:50.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.017+0000 7fbc4e601640 1 --2- 172.21.15.57:0/3205854243 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc480cd0c0 0x7fbc4806b2e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fbc4000c430 tx=0x7fbc4000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:50.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.017+0000 7fbc3e7fc640 1 -- 172.21.15.57:0/3205854243 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc40004230 con 0x7fbc480cd0c0 2023-10-19T12:44:50.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.017+0000 7fbc3e7fc640 1 -- 172.21.15.57:0/3205854243 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fbc400044a0 con 0x7fbc480cd0c0 2023-10-19T12:44:50.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.017+0000 7fbc50064640 1 -- 172.21.15.57:0/3205854243 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc4806bd00 con 0x7fbc480cd0c0 2023-10-19T12:44:50.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.017+0000 7fbc3e7fc640 1 -- 172.21.15.57:0/3205854243 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc40003850 con 0x7fbc480cd0c0 2023-10-19T12:44:50.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.017+0000 7fbc50064640 1 -- 172.21.15.57:0/3205854243 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc48155ec0 con 0x7fbc480cd0c0 2023-10-19T12:44:50.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.018+0000 7fbc50064640 1 -- 172.21.15.57:0/3205854243 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc480690e0 con 0x7fbc480cd0c0 2023-10-19T12:44:50.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.019+0000 7fbc3e7fc640 1 -- 172.21.15.57:0/3205854243 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fbc400039b0 con 0x7fbc480cd0c0 2023-10-19T12:44:50.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.019+0000 7fbc3e7fc640 1 --2- 172.21.15.57:0/3205854243 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fbc2c06ae40 0x7fbc2c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:50.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.019+0000 7fbc3e7fc640 1 -- 172.21.15.57:0/3205854243 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fbc40089160 con 0x7fbc480cd0c0 2023-10-19T12:44:50.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.019+0000 7fbc4de00640 1 --2- 172.21.15.57:0/3205854243 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fbc2c06ae40 0x7fbc2c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:50.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.020+0000 7fbc4de00640 1 --2- 172.21.15.57:0/3205854243 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fbc2c06ae40 0x7fbc2c06d300 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fbc480b4400 tx=0x7fbc44009500 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:50.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.023+0000 7fbc3e7fc640 1 -- 172.21.15.57:0/3205854243 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fbc40053f20 con 0x7fbc480cd0c0 2023-10-19T12:44:50.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.200+0000 7fbc50064640 1 -- 172.21.15.57:0/3205854243 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fbc480b0ca0 con 0x7fbc480cd0c0 2023-10-19T12:44:50.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.201+0000 7fbc3e7fc640 1 -- 172.21.15.57:0/3205854243 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) v1 ==== 93+0+10 (secure 0 0 0) 0x7fbc40057540 con 0x7fbc480cd0c0 2023-10-19T12:44:50.205 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-19T12:44:50.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.203+0000 7fbc50064640 1 -- 172.21.15.57:0/3205854243 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fbc2c06ae40 msgr2=0x7fbc2c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:50.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.203+0000 7fbc50064640 1 --2- 172.21.15.57:0/3205854243 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fbc2c06ae40 0x7fbc2c06d300 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fbc480b4400 tx=0x7fbc44009500 comp rx=0 tx=0).stop 2023-10-19T12:44:50.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.203+0000 7fbc50064640 1 -- 172.21.15.57:0/3205854243 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc480cd0c0 msgr2=0x7fbc4806b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:50.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.203+0000 7fbc50064640 1 --2- 172.21.15.57:0/3205854243 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc480cd0c0 0x7fbc4806b2e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fbc4000c430 tx=0x7fbc4000c8f0 comp rx=0 tx=0).stop 2023-10-19T12:44:50.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.203+0000 7fbc50064640 1 -- 172.21.15.57:0/3205854243 shutdown_connections 2023-10-19T12:44:50.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.203+0000 7fbc50064640 1 --2- 172.21.15.57:0/3205854243 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fbc2c06ae40 0x7fbc2c06d300 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:50.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.203+0000 7fbc50064640 1 --2- 172.21.15.57:0/3205854243 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc480cd0c0 0x7fbc4806b2e0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:50.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.203+0000 7fbc50064640 1 --2- 172.21.15.57:0/3205854243 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbc4806dce0 0x7fbc4806ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:50.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.203+0000 7fbc50064640 1 --2- 172.21.15.57:0/3205854243 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbc4806bf20 0x7fbc481523d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:50.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.203+0000 7fbc50064640 1 -- 172.21.15.57:0/3205854243 >> 172.21.15.57:0/3205854243 conn(0x7fbc480b0670 msgr2=0x7fbc480aeee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:50.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.203+0000 7fbc50064640 1 -- 172.21.15.57:0/3205854243 shutdown_connections 2023-10-19T12:44:50.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.203+0000 7fbc50064640 1 -- 172.21.15.57:0/3205854243 wait complete. 2023-10-19T12:44:50.216 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-19T12:44:50.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.326+0000 7fdd6e955640 1 -- 172.21.15.57:0/1863002721 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd50004d00 msgr2=0x7fdd50005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:50.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.326+0000 7fdd6e955640 1 --2- 172.21.15.57:0/1863002721 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd50004d00 0x7fdd50005180 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fdd6000c610 tx=0x7fdd500057f0 comp rx=0 tx=0).stop 2023-10-19T12:44:50.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.326+0000 7fdd6cef2640 1 -- 172.21.15.57:0/1863002721 reap_dead start 2023-10-19T12:44:50.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.326+0000 7fdd6e955640 1 -- 172.21.15.57:0/1863002721 shutdown_connections 2023-10-19T12:44:50.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.326+0000 7fdd6e955640 1 -- 172.21.15.57:0/1863002721 >> 172.21.15.57:0/1863002721 conn(0x7fdd680af7a0 msgr2=0x7fdd680afb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:50.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.326+0000 7fdd6e955640 1 -- 172.21.15.57:0/1863002721 shutdown_connections 2023-10-19T12:44:50.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.326+0000 7fdd6e955640 1 -- 172.21.15.57:0/1863002721 wait complete. 2023-10-19T12:44:50.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.327+0000 7fdd6e955640 1 Processor -- start 2023-10-19T12:44:50.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.328+0000 7fdd6e955640 1 -- start start 2023-10-19T12:44:50.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.328+0000 7fdd6e955640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdd681edf50 0x7fdd6814ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:50.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.328+0000 7fdd6e955640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd6814cef0 0x7fdd681ee330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:50.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.328+0000 7fdd6e955640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdd681ee870 0x7fdd68150120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:50.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.328+0000 7fdd6e955640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd6814dfe0 con 0x7fdd6814cef0 2023-10-19T12:44:50.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.328+0000 7fdd6e955640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd6814d740 con 0x7fdd681edf50 2023-10-19T12:44:50.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.328+0000 7fdd6e955640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd6814da20 con 0x7fdd681ee870 2023-10-19T12:44:50.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.328+0000 7fdd677fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd6814cef0 0x7fdd681ee330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:50.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.328+0000 7fdd677fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd6814cef0 0x7fdd681ee330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59352/0 (socket says 172.21.15.57:59352) 2023-10-19T12:44:50.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.329+0000 7fdd677fe640 1 -- 172.21.15.57:0/1969991472 learned_addr learned my addr 172.21.15.57:0/1969991472 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:50.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.329+0000 7fdd6cef2640 1 --2- 172.21.15.57:0/1969991472 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdd681ee870 0x7fdd68150120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:50.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.329+0000 7fdd67fff640 1 --2- 172.21.15.57:0/1969991472 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdd681edf50 0x7fdd6814ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:50.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.329+0000 7fdd677fe640 1 -- 172.21.15.57:0/1969991472 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdd681ee870 msgr2=0x7fdd68150120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:50.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.329+0000 7fdd677fe640 1 --2- 172.21.15.57:0/1969991472 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdd681ee870 0x7fdd68150120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:50.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.329+0000 7fdd677fe640 1 -- 172.21.15.57:0/1969991472 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdd681edf50 msgr2=0x7fdd6814ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:50.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.329+0000 7fdd677fe640 1 --2- 172.21.15.57:0/1969991472 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdd681edf50 0x7fdd6814ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:50.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.329+0000 7fdd677fe640 1 -- 172.21.15.57:0/1969991472 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd6814dd00 con 0x7fdd6814cef0 2023-10-19T12:44:50.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.329+0000 7fdd67fff640 1 --2- 172.21.15.57:0/1969991472 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdd681edf50 0x7fdd6814ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:44:50.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.329+0000 7fdd677fe640 1 --2- 172.21.15.57:0/1969991472 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd6814cef0 0x7fdd681ee330 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fdd5c011ae0 tx=0x7fdd5c009e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:50.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.329+0000 7fdd64ff9640 1 -- 172.21.15.57:0/1969991472 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd5c00fe70 con 0x7fdd6814cef0 2023-10-19T12:44:50.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.329+0000 7fdd64ff9640 1 -- 172.21.15.57:0/1969991472 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fdd5c004050 con 0x7fdd6814cef0 2023-10-19T12:44:50.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.329+0000 7fdd6e955640 1 -- 172.21.15.57:0/1969991472 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd6814fc00 con 0x7fdd6814cef0 2023-10-19T12:44:50.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.329+0000 7fdd64ff9640 1 -- 172.21.15.57:0/1969991472 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd5c004460 con 0x7fdd6814cef0 2023-10-19T12:44:50.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.329+0000 7fdd6e955640 1 -- 172.21.15.57:0/1969991472 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd6814d330 con 0x7fdd6814cef0 2023-10-19T12:44:50.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.330+0000 7fdd6e955640 1 -- 172.21.15.57:0/1969991472 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd6814fa60 con 0x7fdd6814cef0 2023-10-19T12:44:50.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.331+0000 7fdd64ff9640 1 -- 172.21.15.57:0/1969991472 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fdd5c0035f0 con 0x7fdd6814cef0 2023-10-19T12:44:50.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.331+0000 7fdd64ff9640 1 --2- 172.21.15.57:0/1969991472 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fdd5406ae70 0x7fdd5406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:50.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.331+0000 7fdd64ff9640 1 -- 172.21.15.57:0/1969991472 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fdd5c08a7f0 con 0x7fdd6814cef0 2023-10-19T12:44:50.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.331+0000 7fdd67fff640 1 --2- 172.21.15.57:0/1969991472 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fdd5406ae70 0x7fdd5406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:50.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.332+0000 7fdd67fff640 1 --2- 172.21.15.57:0/1969991472 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fdd5406ae70 0x7fdd5406d330 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fdd58009cd0 tx=0x7fdd58005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:50.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.335+0000 7fdd64ff9640 1 -- 172.21.15.57:0/1969991472 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fdd6814fa60 con 0x7fdd6814cef0 2023-10-19T12:44:50.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.511+0000 7fdd6e955640 1 -- 172.21.15.57:0/1969991472 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fdd6814fa60 con 0x7fdd6814cef0 2023-10-19T12:44:50.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.511+0000 7fdd64ff9640 1 -- 172.21.15.57:0/1969991472 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) v1 ==== 104+0+11 (secure 0 0 0) 0x7fdd5c01e020 con 0x7fdd6814cef0 2023-10-19T12:44:50.515 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2023-10-19T12:44:50.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.513+0000 7fdd6e955640 1 -- 172.21.15.57:0/1969991472 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fdd5406ae70 msgr2=0x7fdd5406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:50.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.513+0000 7fdd6e955640 1 --2- 172.21.15.57:0/1969991472 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fdd5406ae70 0x7fdd5406d330 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fdd58009cd0 tx=0x7fdd58005f90 comp rx=0 tx=0).stop 2023-10-19T12:44:50.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.513+0000 7fdd6e955640 1 -- 172.21.15.57:0/1969991472 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd6814cef0 msgr2=0x7fdd681ee330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:50.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.513+0000 7fdd6e955640 1 --2- 172.21.15.57:0/1969991472 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd6814cef0 0x7fdd681ee330 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fdd5c011ae0 tx=0x7fdd5c009e90 comp rx=0 tx=0).stop 2023-10-19T12:44:50.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.513+0000 7fdd6e955640 1 -- 172.21.15.57:0/1969991472 shutdown_connections 2023-10-19T12:44:50.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.513+0000 7fdd6e955640 1 --2- 172.21.15.57:0/1969991472 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fdd5406ae70 0x7fdd5406d330 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:50.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.513+0000 7fdd6e955640 1 --2- 172.21.15.57:0/1969991472 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdd681ee870 0x7fdd68150120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:50.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.513+0000 7fdd6e955640 1 --2- 172.21.15.57:0/1969991472 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd6814cef0 0x7fdd681ee330 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:50.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.513+0000 7fdd6e955640 1 --2- 172.21.15.57:0/1969991472 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdd681edf50 0x7fdd6814ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:50.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.513+0000 7fdd6e955640 1 -- 172.21.15.57:0/1969991472 >> 172.21.15.57:0/1969991472 conn(0x7fdd680af7a0 msgr2=0x7fdd68159870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:50.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.513+0000 7fdd6e955640 1 -- 172.21.15.57:0/1969991472 shutdown_connections 2023-10-19T12:44:50.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.513+0000 7fdd6e955640 1 -- 172.21.15.57:0/1969991472 wait complete. 2023-10-19T12:44:50.527 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-19T12:44:50.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.635+0000 7fd9e4bc0640 1 -- 172.21.15.57:0/254564985 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd9e00cf340 msgr2=0x7fd9e00d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:50.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.635+0000 7fd9e4bc0640 1 --2- 172.21.15.57:0/254564985 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd9e00cf340 0x7fd9e00d1910 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fd9e006c850 tx=0x7fd9d802bd10 comp rx=0 tx=0).stop 2023-10-19T12:44:50.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.635+0000 7fd9e4bc0640 1 -- 172.21.15.57:0/254564985 shutdown_connections 2023-10-19T12:44:50.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.635+0000 7fd9e4bc0640 1 --2- 172.21.15.57:0/254564985 >> v2:172.21.15.179:3300/0 conn(0x7fd9e00ceb40 0x7fd9e00b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:50.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.635+0000 7fd9e4bc0640 1 --2- 172.21.15.57:0/254564985 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd9e015b2a0 0x7fd9e00cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:50.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.635+0000 7fd9e4bc0640 1 --2- 172.21.15.57:0/254564985 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd9e00cf340 0x7fd9e00d1910 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:50.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.635+0000 7fd9e4bc0640 1 -- 172.21.15.57:0/254564985 >> 172.21.15.57:0/254564985 conn(0x7fd9e00d0d60 msgr2=0x7fd9e00d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:50.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.635+0000 7fd9e4bc0640 1 -- 172.21.15.57:0/254564985 shutdown_connections 2023-10-19T12:44:50.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.635+0000 7fd9e4bc0640 1 -- 172.21.15.57:0/254564985 wait complete. 2023-10-19T12:44:50.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.636+0000 7fd9e4bc0640 1 Processor -- start 2023-10-19T12:44:50.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.637+0000 7fd9e4bc0640 1 -- start start 2023-10-19T12:44:50.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.637+0000 7fd9e4bc0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd9e00ceb40 0x7fd9e006b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:50.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.637+0000 7fd9e4bc0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd9e00cf340 0x7fd9e0158fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:50.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.637+0000 7fd9e4bc0640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd9e015b2a0 0x7fd9e01594e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:50.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.637+0000 7fd9e4bc0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e00ca420 con 0x7fd9e00ceb40 2023-10-19T12:44:50.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.637+0000 7fd9e4bc0640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e00caf60 con 0x7fd9e00cf340 2023-10-19T12:44:50.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.637+0000 7fd9e4bc0640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e00ca6c0 con 0x7fd9e015b2a0 2023-10-19T12:44:50.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.637+0000 7fd9de59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd9e00ceb40 0x7fd9e006b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:50.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.637+0000 7fd9de59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd9e00ceb40 0x7fd9e006b480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59364/0 (socket says 172.21.15.57:59364) 2023-10-19T12:44:50.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.637+0000 7fd9de59c640 1 -- 172.21.15.57:0/2797446312 learned_addr learned my addr 172.21.15.57:0/2797446312 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:50.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.638+0000 7fd9ddd9b640 1 --2- 172.21.15.57:0/2797446312 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd9e00cf340 0x7fd9e0158fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:50.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.638+0000 7fd9de59c640 1 -- 172.21.15.57:0/2797446312 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd9e015b2a0 msgr2=0x7fd9e01594e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:50.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.638+0000 7fd9ded9d640 1 --2- 172.21.15.57:0/2797446312 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd9e015b2a0 0x7fd9e01594e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:50.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.638+0000 7fd9de59c640 1 --2- 172.21.15.57:0/2797446312 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd9e015b2a0 0x7fd9e01594e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:50.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.638+0000 7fd9de59c640 1 -- 172.21.15.57:0/2797446312 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd9e00cf340 msgr2=0x7fd9e0158fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:50.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.638+0000 7fd9de59c640 1 --2- 172.21.15.57:0/2797446312 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd9e00cf340 0x7fd9e0158fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:50.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.638+0000 7fd9de59c640 1 -- 172.21.15.57:0/2797446312 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9d8009560 con 0x7fd9e00ceb40 2023-10-19T12:44:50.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.638+0000 7fd9ddd9b640 1 --2- 172.21.15.57:0/2797446312 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd9e00cf340 0x7fd9e0158fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:44:50.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.638+0000 7fd9de59c640 1 --2- 172.21.15.57:0/2797446312 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd9e00ceb40 0x7fd9e006b480 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fd9d8009850 tx=0x7fd9d802da40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:50.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.638+0000 7fd9d6ffd640 1 -- 172.21.15.57:0/2797446312 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9d8003800 con 0x7fd9e00ceb40 2023-10-19T12:44:50.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.638+0000 7fd9d6ffd640 1 -- 172.21.15.57:0/2797446312 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fd9d8003a50 con 0x7fd9e00ceb40 2023-10-19T12:44:50.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.638+0000 7fd9e4bc0640 1 -- 172.21.15.57:0/2797446312 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9e00ca9a0 con 0x7fd9e00ceb40 2023-10-19T12:44:50.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.638+0000 7fd9d6ffd640 1 -- 172.21.15.57:0/2797446312 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9d8003e20 con 0x7fd9e00ceb40 2023-10-19T12:44:50.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.638+0000 7fd9e4bc0640 1 -- 172.21.15.57:0/2797446312 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9e006b9c0 con 0x7fd9e00ceb40 2023-10-19T12:44:50.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.639+0000 7fd9e4bc0640 1 -- 172.21.15.57:0/2797446312 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9e00690e0 con 0x7fd9e00ceb40 2023-10-19T12:44:50.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.639+0000 7fd9d6ffd640 1 -- 172.21.15.57:0/2797446312 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fd9d803e450 con 0x7fd9e00ceb40 2023-10-19T12:44:50.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.640+0000 7fd9d6ffd640 1 --2- 172.21.15.57:0/2797446312 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fd9c006ae70 0x7fd9c006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:50.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.640+0000 7fd9d6ffd640 1 -- 172.21.15.57:0/2797446312 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fd9d80ac430 con 0x7fd9e00ceb40 2023-10-19T12:44:50.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.640+0000 7fd9ddd9b640 1 --2- 172.21.15.57:0/2797446312 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fd9c006ae70 0x7fd9c006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:50.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.641+0000 7fd9ddd9b640 1 --2- 172.21.15.57:0/2797446312 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fd9c006ae70 0x7fd9c006d330 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fd9e00ca040 tx=0x7fd9c8005db0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:50.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.644+0000 7fd9d6ffd640 1 -- 172.21.15.57:0/2797446312 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd9d80771f0 con 0x7fd9e00ceb40 2023-10-19T12:44:50.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.819+0000 7fd9e4bc0640 1 -- 172.21.15.57:0/2797446312 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd9e00d0b40 con 0x7fd9e00ceb40 2023-10-19T12:44:50.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.819+0000 7fd9d6ffd640 1 -- 172.21.15.57:0/2797446312 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd9d807a810 con 0x7fd9e00ceb40 2023-10-19T12:44:50.823 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 64 2023-10-19T12:44:50.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.821+0000 7fd9e4bc0640 1 -- 172.21.15.57:0/2797446312 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fd9c006ae70 msgr2=0x7fd9c006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:50.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.821+0000 7fd9e4bc0640 1 --2- 172.21.15.57:0/2797446312 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fd9c006ae70 0x7fd9c006d330 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fd9e00ca040 tx=0x7fd9c8005db0 comp rx=0 tx=0).stop 2023-10-19T12:44:50.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.821+0000 7fd9e4bc0640 1 -- 172.21.15.57:0/2797446312 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd9e00ceb40 msgr2=0x7fd9e006b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:50.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.821+0000 7fd9e4bc0640 1 --2- 172.21.15.57:0/2797446312 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd9e00ceb40 0x7fd9e006b480 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fd9d8009850 tx=0x7fd9d802da40 comp rx=0 tx=0).stop 2023-10-19T12:44:50.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.821+0000 7fd9e4bc0640 1 -- 172.21.15.57:0/2797446312 shutdown_connections 2023-10-19T12:44:50.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.821+0000 7fd9e4bc0640 1 --2- 172.21.15.57:0/2797446312 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fd9c006ae70 0x7fd9c006d330 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:50.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.821+0000 7fd9e4bc0640 1 --2- 172.21.15.57:0/2797446312 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd9e015b2a0 0x7fd9e01594e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:50.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.821+0000 7fd9e4bc0640 1 --2- 172.21.15.57:0/2797446312 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd9e00cf340 0x7fd9e0158fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:50.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.821+0000 7fd9e4bc0640 1 --2- 172.21.15.57:0/2797446312 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd9e00ceb40 0x7fd9e006b480 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:50.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.821+0000 7fd9e4bc0640 1 -- 172.21.15.57:0/2797446312 >> 172.21.15.57:0/2797446312 conn(0x7fd9e00d0d60 msgr2=0x7fd9e01e5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:50.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.821+0000 7fd9e4bc0640 1 -- 172.21.15.57:0/2797446312 shutdown_connections 2023-10-19T12:44:50.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.821+0000 7fd9e4bc0640 1 -- 172.21.15.57:0/2797446312 wait complete. 2023-10-19T12:44:50.835 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-19T12:44:50.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.943+0000 7fd2b1b6e640 1 -- 172.21.15.57:0/4161971368 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd2ac0cf340 msgr2=0x7fd2ac0d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:50.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.943+0000 7fd2b1b6e640 1 --2- 172.21.15.57:0/4161971368 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd2ac0cf340 0x7fd2ac0d1910 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fd29c009830 tx=0x7fd29c02bcd0 comp rx=0 tx=0).stop 2023-10-19T12:44:50.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.943+0000 7fd2b1b6e640 1 -- 172.21.15.57:0/4161971368 shutdown_connections 2023-10-19T12:44:50.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.943+0000 7fd2b1b6e640 1 --2- 172.21.15.57:0/4161971368 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd2ac0cf340 0x7fd2ac0d1910 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:50.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.943+0000 7fd2b1b6e640 1 -- 172.21.15.57:0/4161971368 >> 172.21.15.57:0/4161971368 conn(0x7fd2ac0d0d60 msgr2=0x7fd2ac0d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:50.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.943+0000 7fd2b1b6e640 1 -- 172.21.15.57:0/4161971368 shutdown_connections 2023-10-19T12:44:50.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.943+0000 7fd2b1b6e640 1 -- 172.21.15.57:0/4161971368 wait complete. 2023-10-19T12:44:50.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.945+0000 7fd2b1b6e640 1 Processor -- start 2023-10-19T12:44:50.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.945+0000 7fd2b1b6e640 1 -- start start 2023-10-19T12:44:50.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.945+0000 7fd2b1b6e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd2ac0ceb40 0x7fd2ac0b43a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:50.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.945+0000 7fd2b1b6e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd2ac0cf340 0x7fd2ac153330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:50.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.945+0000 7fd2b1b6e640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd2ac15b2a0 0x7fd2ac1f1750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:50.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.945+0000 7fd2b1b6e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2ac0b38e0 con 0x7fd2ac0ceb40 2023-10-19T12:44:50.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.945+0000 7fd2b1b6e640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2ac153870 con 0x7fd2ac0cf340 2023-10-19T12:44:50.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.945+0000 7fd2b1b6e640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2ac1f2280 con 0x7fd2ac15b2a0 2023-10-19T12:44:50.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.946+0000 7fd2ab7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd2ac0ceb40 0x7fd2ac0b43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:50.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.946+0000 7fd2ab7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd2ac0ceb40 0x7fd2ac0b43a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59376/0 (socket says 172.21.15.57:59376) 2023-10-19T12:44:50.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.946+0000 7fd2ab7fe640 1 -- 172.21.15.57:0/322258645 learned_addr learned my addr 172.21.15.57:0/322258645 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:50.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.946+0000 7fd2aaffd640 1 --2- 172.21.15.57:0/322258645 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd2ac0cf340 0x7fd2ac153330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:50.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.946+0000 7fd2abfff640 1 --2- 172.21.15.57:0/322258645 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd2ac15b2a0 0x7fd2ac1f1750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:50.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.946+0000 7fd2ab7fe640 1 -- 172.21.15.57:0/322258645 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd2ac15b2a0 msgr2=0x7fd2ac1f1750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:50.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.946+0000 7fd2ab7fe640 1 --2- 172.21.15.57:0/322258645 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd2ac15b2a0 0x7fd2ac1f1750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:50.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.946+0000 7fd2ab7fe640 1 -- 172.21.15.57:0/322258645 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd2ac0cf340 msgr2=0x7fd2ac153330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:50.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.946+0000 7fd2ab7fe640 1 --2- 172.21.15.57:0/322258645 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd2ac0cf340 0x7fd2ac153330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:50.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.946+0000 7fd2ab7fe640 1 -- 172.21.15.57:0/322258645 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd29c009490 con 0x7fd2ac0ceb40 2023-10-19T12:44:50.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.946+0000 7fd2aaffd640 1 --2- 172.21.15.57:0/322258645 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd2ac0cf340 0x7fd2ac153330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:44:50.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.946+0000 7fd2ab7fe640 1 --2- 172.21.15.57:0/322258645 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd2ac0ceb40 0x7fd2ac0b43a0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fd29c009780 tx=0x7fd29c0038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:50.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.946+0000 7fd293fff640 1 -- 172.21.15.57:0/322258645 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd29c003ea0 con 0x7fd2ac0ceb40 2023-10-19T12:44:50.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.946+0000 7fd293fff640 1 -- 172.21.15.57:0/322258645 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fd29c034550 con 0x7fd2ac0ceb40 2023-10-19T12:44:50.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.946+0000 7fd293fff640 1 -- 172.21.15.57:0/322258645 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd29c034960 con 0x7fd2ac0ceb40 2023-10-19T12:44:50.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.946+0000 7fd2b1b6e640 1 -- 172.21.15.57:0/322258645 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2ac1f2560 con 0x7fd2ac0ceb40 2023-10-19T12:44:50.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.946+0000 7fd2b1b6e640 1 -- 172.21.15.57:0/322258645 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2ac0b3e80 con 0x7fd2ac0ceb40 2023-10-19T12:44:50.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.947+0000 7fd2b1b6e640 1 -- 172.21.15.57:0/322258645 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2ac1f26f0 con 0x7fd2ac0ceb40 2023-10-19T12:44:50.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.948+0000 7fd293fff640 1 -- 172.21.15.57:0/322258645 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fd29c03b040 con 0x7fd2ac0ceb40 2023-10-19T12:44:50.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.948+0000 7fd293fff640 1 --2- 172.21.15.57:0/322258645 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fd29806ae40 0x7fd29806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:50.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.948+0000 7fd293fff640 1 -- 172.21.15.57:0/322258645 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fd29c0abf80 con 0x7fd2ac0ceb40 2023-10-19T12:44:50.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.948+0000 7fd2aaffd640 1 --2- 172.21.15.57:0/322258645 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fd29806ae40 0x7fd29806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:50.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.949+0000 7fd2aaffd640 1 --2- 172.21.15.57:0/322258645 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fd29806ae40 0x7fd29806d300 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fd2ac15a300 tx=0x7fd2a0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:50.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:50.952+0000 7fd293fff640 1 -- 172.21.15.57:0/322258645 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd29c0deca0 con 0x7fd2ac0ceb40 2023-10-19T12:44:51.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.129+0000 7fd2b1b6e640 1 -- 172.21.15.57:0/322258645 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fd2ac0b4070 con 0x7fd2ac0ceb40 2023-10-19T12:44:51.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.129+0000 7fd293fff640 1 -- 172.21.15.57:0/322258645 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) v1 ==== 103+0+11 (secure 0 0 0) 0x7fd29c0390a0 con 0x7fd2ac0ceb40 2023-10-19T12:44:51.134 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2023-10-19T12:44:51.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.131+0000 7fd2b1b6e640 1 -- 172.21.15.57:0/322258645 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fd29806ae40 msgr2=0x7fd29806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:51.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.131+0000 7fd2b1b6e640 1 --2- 172.21.15.57:0/322258645 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fd29806ae40 0x7fd29806d300 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fd2ac15a300 tx=0x7fd2a0008040 comp rx=0 tx=0).stop 2023-10-19T12:44:51.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.131+0000 7fd2b1b6e640 1 -- 172.21.15.57:0/322258645 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd2ac0ceb40 msgr2=0x7fd2ac0b43a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:51.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.131+0000 7fd2b1b6e640 1 --2- 172.21.15.57:0/322258645 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd2ac0ceb40 0x7fd2ac0b43a0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fd29c009780 tx=0x7fd29c0038a0 comp rx=0 tx=0).stop 2023-10-19T12:44:51.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.131+0000 7fd2b1b6e640 1 -- 172.21.15.57:0/322258645 shutdown_connections 2023-10-19T12:44:51.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.131+0000 7fd2b1b6e640 1 --2- 172.21.15.57:0/322258645 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fd29806ae40 0x7fd29806d300 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:51.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.131+0000 7fd2b1b6e640 1 --2- 172.21.15.57:0/322258645 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd2ac15b2a0 0x7fd2ac1f1750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:51.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.131+0000 7fd2b1b6e640 1 --2- 172.21.15.57:0/322258645 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd2ac0cf340 0x7fd2ac153330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:51.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.131+0000 7fd2b1b6e640 1 --2- 172.21.15.57:0/322258645 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd2ac0ceb40 0x7fd2ac0b43a0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:51.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.131+0000 7fd2b1b6e640 1 -- 172.21.15.57:0/322258645 >> 172.21.15.57:0/322258645 conn(0x7fd2ac0d0d60 msgr2=0x7fd2ac0ad8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:51.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.131+0000 7fd2b1b6e640 1 -- 172.21.15.57:0/322258645 shutdown_connections 2023-10-19T12:44:51.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.131+0000 7fd2b1b6e640 1 -- 172.21.15.57:0/322258645 wait complete. 2023-10-19T12:44:51.146 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T12:44:51.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.254+0000 7f488bd6c640 1 -- 172.21.15.57:0/3837830218 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f48840d1910 msgr2=0x7f48840b0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:51.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.254+0000 7f488bd6c640 1 --2- 172.21.15.57:0/3837830218 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f48840d1910 0x7f48840b0650 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f487c005fd0 tx=0x7f487c02c0e0 comp rx=0 tx=0).stop 2023-10-19T12:44:51.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.254+0000 7f488bd6c640 1 -- 172.21.15.57:0/3837830218 shutdown_connections 2023-10-19T12:44:51.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.254+0000 7f488bd6c640 1 --2- 172.21.15.57:0/3837830218 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f48840d1910 0x7f48840b0650 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:51.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.254+0000 7f488bd6c640 1 -- 172.21.15.57:0/3837830218 >> 172.21.15.57:0/3837830218 conn(0x7f48840d0d60 msgr2=0x7f48840d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:51.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.255+0000 7f488bd6c640 1 -- 172.21.15.57:0/3837830218 shutdown_connections 2023-10-19T12:44:51.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.255+0000 7f488bd6c640 1 -- 172.21.15.57:0/3837830218 wait complete. 2023-10-19T12:44:51.262 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.256+0000 7f488bd6c640 1 Processor -- start 2023-10-19T12:44:51.262 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.256+0000 7f488bd6c640 1 -- start start 2023-10-19T12:44:51.262 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.256+0000 7f488bd6c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f48840cf340 0x7f48841f1000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:51.262 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.256+0000 7f488bd6c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f48840d1910 0x7f48840b2830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:51.262 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.256+0000 7f488bd6c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f488415b2a0 0x7f48840b2d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:51.263 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.256+0000 7f488bd6c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f488415c870 con 0x7f48840cf340 2023-10-19T12:44:51.263 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.256+0000 7f488bd6c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f488415cb50 con 0x7f488415b2a0 2023-10-19T12:44:51.263 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.256+0000 7f488bd6c640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f488415ce30 con 0x7f48840d1910 2023-10-19T12:44:51.263 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.257+0000 7f4889b08640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f48840cf340 0x7f48841f1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:51.263 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.257+0000 7f4889b08640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f48840cf340 0x7f48841f1000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59388/0 (socket says 172.21.15.57:59388) 2023-10-19T12:44:51.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.257+0000 7f4889b08640 1 -- 172.21.15.57:0/2292749707 learned_addr learned my addr 172.21.15.57:0/2292749707 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:51.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.257+0000 7f488a309640 1 --2- 172.21.15.57:0/2292749707 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f488415b2a0 0x7f48840b2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:51.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.257+0000 7f4889307640 1 --2- 172.21.15.57:0/2292749707 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f48840d1910 0x7f48840b2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:51.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.257+0000 7f4889b08640 1 -- 172.21.15.57:0/2292749707 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f48840d1910 msgr2=0x7f48840b2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:51.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.257+0000 7f4889b08640 1 --2- 172.21.15.57:0/2292749707 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f48840d1910 0x7f48840b2830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:51.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.257+0000 7f4889b08640 1 -- 172.21.15.57:0/2292749707 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f488415b2a0 msgr2=0x7f48840b2d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:51.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.257+0000 7f4889b08640 1 --2- 172.21.15.57:0/2292749707 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f488415b2a0 0x7f48840b2d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:51.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.257+0000 7f4889b08640 1 -- 172.21.15.57:0/2292749707 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f487c009840 con 0x7f48840cf340 2023-10-19T12:44:51.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.257+0000 7f488a309640 1 --2- 172.21.15.57:0/2292749707 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f488415b2a0 0x7f48840b2d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:44:51.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.257+0000 7f4889b08640 1 --2- 172.21.15.57:0/2292749707 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f48840cf340 0x7f48841f1000 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f488000c450 tx=0x7f488000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:51.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.257+0000 7f487a7fc640 1 -- 172.21.15.57:0/2292749707 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4880012830 con 0x7f48840cf340 2023-10-19T12:44:51.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.257+0000 7f487a7fc640 1 -- 172.21.15.57:0/2292749707 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f4880012aa0 con 0x7f48840cf340 2023-10-19T12:44:51.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.257+0000 7f488bd6c640 1 -- 172.21.15.57:0/2292749707 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f488415d110 con 0x7f48840cf340 2023-10-19T12:44:51.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.257+0000 7f487a7fc640 1 -- 172.21.15.57:0/2292749707 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4880011380 con 0x7f48840cf340 2023-10-19T12:44:51.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.257+0000 7f488bd6c640 1 -- 172.21.15.57:0/2292749707 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48841f1720 con 0x7f48840cf340 2023-10-19T12:44:51.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.258+0000 7f488bd6c640 1 -- 172.21.15.57:0/2292749707 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48840d07d0 con 0x7f48840cf340 2023-10-19T12:44:51.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.259+0000 7f487a7fc640 1 -- 172.21.15.57:0/2292749707 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f48800114e0 con 0x7f48840cf340 2023-10-19T12:44:51.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.259+0000 7f487a7fc640 1 --2- 172.21.15.57:0/2292749707 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f486806ae40 0x7f486806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:51.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.259+0000 7f487a7fc640 1 -- 172.21.15.57:0/2292749707 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f4880089960 con 0x7f48840cf340 2023-10-19T12:44:51.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.259+0000 7f4889307640 1 --2- 172.21.15.57:0/2292749707 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f486806ae40 0x7f486806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:51.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.260+0000 7f4889307640 1 --2- 172.21.15.57:0/2292749707 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f486806ae40 0x7f486806d300 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f488409ec30 tx=0x7f4874009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:51.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.263+0000 7f487a7fc640 1 -- 172.21.15.57:0/2292749707 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f48800547b0 con 0x7f48840cf340 2023-10-19T12:44:51.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.476+0000 7f488bd6c640 1 -- 172.21.15.57:0/2292749707 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f48840690e0 con 0x7f48840cf340 2023-10-19T12:44:51.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.477+0000 7f487a7fc640 1 -- 172.21.15.57:0/2292749707 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4906 (secure 0 0 0) 0x7f4880016090 con 0x7f48840cf340 2023-10-19T12:44:51.480 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:44:51.481 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4357,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/4228712742","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":4228712742},{"type":"v1","addr":"172.21.15.57:6835","nonce":4228712742}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4360,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/971892630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":971892630},{"type":"v1","addr":"172.21.15.57:6834","nonce":971892630}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4372,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/104495983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":104495983},{"type":"v1","addr":"172.21.15.179:6836","nonce":104495983}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T12:44:18.419669+0000","modified":"2023-10-19T12:44:40.545928+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4374},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4374":{"gid":4374,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.179:6837/622550365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":622550365},{"type":"v1","addr":"172.21.15.179:6837","nonce":622550365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-19T12:44:51.481 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 9 2023-10-19T12:44:51.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.479+0000 7f488bd6c640 1 -- 172.21.15.57:0/2292749707 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f486806ae40 msgr2=0x7f486806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:51.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.479+0000 7f488bd6c640 1 --2- 172.21.15.57:0/2292749707 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f486806ae40 0x7f486806d300 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f488409ec30 tx=0x7f4874009040 comp rx=0 tx=0).stop 2023-10-19T12:44:51.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.479+0000 7f488bd6c640 1 -- 172.21.15.57:0/2292749707 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f48840cf340 msgr2=0x7f48841f1000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:51.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.479+0000 7f488bd6c640 1 --2- 172.21.15.57:0/2292749707 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f48840cf340 0x7f48841f1000 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f488000c450 tx=0x7f488000c910 comp rx=0 tx=0).stop 2023-10-19T12:44:51.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.479+0000 7f488bd6c640 1 -- 172.21.15.57:0/2292749707 shutdown_connections 2023-10-19T12:44:51.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.479+0000 7f488bd6c640 1 --2- 172.21.15.57:0/2292749707 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f486806ae40 0x7f486806d300 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:51.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.479+0000 7f488bd6c640 1 --2- 172.21.15.57:0/2292749707 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f488415b2a0 0x7f48840b2d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:51.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.479+0000 7f488bd6c640 1 --2- 172.21.15.57:0/2292749707 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f48840d1910 0x7f48840b2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:51.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.479+0000 7f488bd6c640 1 --2- 172.21.15.57:0/2292749707 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f48840cf340 0x7f48841f1000 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:51.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.479+0000 7f488bd6c640 1 -- 172.21.15.57:0/2292749707 >> 172.21.15.57:0/2292749707 conn(0x7f48840d0d60 msgr2=0x7f488406b810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:51.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.479+0000 7f488bd6c640 1 -- 172.21.15.57:0/2292749707 shutdown_connections 2023-10-19T12:44:51.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.479+0000 7f488bd6c640 1 -- 172.21.15.57:0/2292749707 wait complete. 2023-10-19T12:44:51.493 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T12:44:51.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.599+0000 7f73abfff640 1 -- 172.21.15.57:0/253073967 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f73ac156ff0 msgr2=0x7f73ac0b0690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:51.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.599+0000 7f73abfff640 1 --2- 172.21.15.57:0/253073967 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f73ac156ff0 0x7f73ac0b0690 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f73ac06d950 tx=0x7f73a402bdb0 comp rx=0 tx=0).stop 2023-10-19T12:44:51.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.599+0000 7f73abfff640 1 -- 172.21.15.57:0/253073967 shutdown_connections 2023-10-19T12:44:51.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.599+0000 7f73abfff640 1 --2- 172.21.15.57:0/253073967 >> v2:172.21.15.179:3300/0 conn(0x7f73ac070870 0x7f73ac06ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:51.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.599+0000 7f73abfff640 1 --2- 172.21.15.57:0/253073967 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f73ac156ff0 0x7f73ac0b0690 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:51.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.599+0000 7f73abfff640 1 -- 172.21.15.57:0/253073967 >> 172.21.15.57:0/253073967 conn(0x7f73ac0ae5d0 msgr2=0x7f73ac0ae9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:51.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.599+0000 7f73abfff640 1 -- 172.21.15.57:0/253073967 shutdown_connections 2023-10-19T12:44:51.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.599+0000 7f73abfff640 1 -- 172.21.15.57:0/253073967 wait complete. 2023-10-19T12:44:51.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.600+0000 7f73abfff640 1 Processor -- start 2023-10-19T12:44:51.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.600+0000 7f73abfff640 1 -- start start 2023-10-19T12:44:51.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.601+0000 7f73abfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f73ac06c840 0x7f73ac0ad6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:51.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.601+0000 7f73abfff640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f73ac070870 0x7f73ac13bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:51.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.601+0000 7f73abfff640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f73ac156ff0 0x7f73ac13c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:51.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.601+0000 7f73abfff640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73ac152330 con 0x7f73ac06c840 2023-10-19T12:44:51.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.601+0000 7f73abfff640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73ac1efe70 con 0x7f73ac156ff0 2023-10-19T12:44:51.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.601+0000 7f73abfff640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73ac1ef2a0 con 0x7f73ac070870 2023-10-19T12:44:51.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.601+0000 7f73aaffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f73ac06c840 0x7f73ac0ad6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:51.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.601+0000 7f73aaffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f73ac06c840 0x7f73ac0ad6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59402/0 (socket says 172.21.15.57:59402) 2023-10-19T12:44:51.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.601+0000 7f73aaffd640 1 -- 172.21.15.57:0/3904462404 learned_addr learned my addr 172.21.15.57:0/3904462404 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:51.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.601+0000 7f73aaffd640 1 -- 172.21.15.57:0/3904462404 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f73ac070870 msgr2=0x7f73ac13bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:51.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.601+0000 7f73aaffd640 1 --2- 172.21.15.57:0/3904462404 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f73ac070870 0x7f73ac13bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:51.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.601+0000 7f73aaffd640 1 -- 172.21.15.57:0/3904462404 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f73ac156ff0 msgr2=0x7f73ac13c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:51.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.601+0000 7f73ab7fe640 1 --2- 172.21.15.57:0/3904462404 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f73ac156ff0 0x7f73ac13c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:51.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.601+0000 7f73aaffd640 1 --2- 172.21.15.57:0/3904462404 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f73ac156ff0 0x7f73ac13c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:51.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.601+0000 7f73aaffd640 1 -- 172.21.15.57:0/3904462404 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73a4009560 con 0x7f73ac06c840 2023-10-19T12:44:51.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.601+0000 7f73aaffd640 1 --2- 172.21.15.57:0/3904462404 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f73ac06c840 0x7f73ac0ad6e0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f73a4000c00 tx=0x7f73a402dd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:51.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.602+0000 7f73a37fe640 1 -- 172.21.15.57:0/3904462404 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a4005670 con 0x7f73ac06c840 2023-10-19T12:44:51.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.602+0000 7f73a37fe640 1 -- 172.21.15.57:0/3904462404 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f73a40038f0 con 0x7f73ac06c840 2023-10-19T12:44:51.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.602+0000 7f73abfff640 1 -- 172.21.15.57:0/3904462404 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73ac1ef540 con 0x7f73ac06c840 2023-10-19T12:44:51.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.602+0000 7f73a37fe640 1 -- 172.21.15.57:0/3904462404 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a4003ce0 con 0x7f73ac06c840 2023-10-19T12:44:51.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.602+0000 7f73abfff640 1 -- 172.21.15.57:0/3904462404 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73ac1efb00 con 0x7f73ac06c840 2023-10-19T12:44:51.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.603+0000 7f73abfff640 1 -- 172.21.15.57:0/3904462404 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73ac04da70 con 0x7f73ac06c840 2023-10-19T12:44:51.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.603+0000 7f73a37fe640 1 -- 172.21.15.57:0/3904462404 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f73a403e450 con 0x7f73ac06c840 2023-10-19T12:44:51.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.604+0000 7f73a37fe640 1 --2- 172.21.15.57:0/3904462404 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f738c06ae40 0x7f738c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:51.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.604+0000 7f73a37fe640 1 -- 172.21.15.57:0/3904462404 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f73a40ac280 con 0x7f73ac06c840 2023-10-19T12:44:51.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.604+0000 7f73aa7fc640 1 --2- 172.21.15.57:0/3904462404 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f738c06ae40 0x7f738c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:51.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.604+0000 7f73aa7fc640 1 --2- 172.21.15.57:0/3904462404 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f738c06ae40 0x7f738c06d300 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f73ac151640 tx=0x7f73940079b0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:51.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.608+0000 7f73a37fe640 1 -- 172.21.15.57:0/3904462404 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f73a40771f0 con 0x7f73ac06c840 2023-10-19T12:44:51.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.786+0000 7f73abfff640 1 -- 172.21.15.57:0/3904462404 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f73ac1567f0 con 0x7f73ac06c840 2023-10-19T12:44:51.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.787+0000 7f73a37fe640 1 -- 172.21.15.57:0/3904462404 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18358 (secure 0 0 0) 0x7f73a407a810 con 0x7f73ac06c840 2023-10-19T12:44:51.791 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:44:51.791 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":25,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T12:44:23.472399+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-19T12:44:16.517510+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-19T12:44:17.398297+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-19T12:44:21.129403+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T12:44:51.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.789+0000 7f73abfff640 1 -- 172.21.15.57:0/3904462404 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f738c06ae40 msgr2=0x7f738c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:51.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.789+0000 7f73abfff640 1 --2- 172.21.15.57:0/3904462404 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f738c06ae40 0x7f738c06d300 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f73ac151640 tx=0x7f73940079b0 comp rx=0 tx=0).stop 2023-10-19T12:44:51.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.789+0000 7f73abfff640 1 -- 172.21.15.57:0/3904462404 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f73ac06c840 msgr2=0x7f73ac0ad6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:51.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.789+0000 7f73abfff640 1 --2- 172.21.15.57:0/3904462404 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f73ac06c840 0x7f73ac0ad6e0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f73a4000c00 tx=0x7f73a402dd80 comp rx=0 tx=0).stop 2023-10-19T12:44:51.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.789+0000 7f73abfff640 1 -- 172.21.15.57:0/3904462404 shutdown_connections 2023-10-19T12:44:51.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.789+0000 7f73abfff640 1 --2- 172.21.15.57:0/3904462404 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f738c06ae40 0x7f738c06d300 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:51.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.789+0000 7f73abfff640 1 --2- 172.21.15.57:0/3904462404 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f73ac156ff0 0x7f73ac13c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:51.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.789+0000 7f73abfff640 1 --2- 172.21.15.57:0/3904462404 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f73ac070870 0x7f73ac13bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:51.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.789+0000 7f73abfff640 1 --2- 172.21.15.57:0/3904462404 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f73ac06c840 0x7f73ac0ad6e0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:51.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.789+0000 7f73abfff640 1 -- 172.21.15.57:0/3904462404 >> 172.21.15.57:0/3904462404 conn(0x7f73ac0ae5d0 msgr2=0x7f73ac06afa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:51.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.789+0000 7f73abfff640 1 -- 172.21.15.57:0/3904462404 shutdown_connections 2023-10-19T12:44:51.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.789+0000 7f73abfff640 1 -- 172.21.15.57:0/3904462404 wait complete. 2023-10-19T12:44:51.803 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T12:44:51.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.909+0000 7fd57fd4c640 1 -- 172.21.15.57:0/2974638773 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd57806f0a0 msgr2=0x7fd57806ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:51.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.909+0000 7fd57fd4c640 1 --2- 172.21.15.57:0/2974638773 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd57806f0a0 0x7fd57806ddc0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fd5780c9200 tx=0x7fd56802be40 comp rx=0 tx=0).stop 2023-10-19T12:44:51.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.909+0000 7fd57fd4c640 1 -- 172.21.15.57:0/2974638773 shutdown_connections 2023-10-19T12:44:51.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.909+0000 7fd57fd4c640 1 --2- 172.21.15.57:0/2974638773 >> v2:172.21.15.179:3300/0 conn(0x7fd5780ad370 0x7fd5780ad800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:51.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.909+0000 7fd57fd4c640 1 --2- 172.21.15.57:0/2974638773 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd57806f0a0 0x7fd57806ddc0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:51.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.909+0000 7fd57fd4c640 1 -- 172.21.15.57:0/2974638773 >> 172.21.15.57:0/2974638773 conn(0x7fd5780b0670 msgr2=0x7fd5780b0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:51.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.910+0000 7fd57fd4c640 1 -- 172.21.15.57:0/2974638773 shutdown_connections 2023-10-19T12:44:51.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.910+0000 7fd57fd4c640 1 -- 172.21.15.57:0/2974638773 wait complete. 2023-10-19T12:44:51.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.911+0000 7fd57fd4c640 1 Processor -- start 2023-10-19T12:44:51.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.911+0000 7fd57fd4c640 1 -- start start 2023-10-19T12:44:51.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.911+0000 7fd57fd4c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd57806d2d0 0x7fd5780c81e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:51.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.911+0000 7fd57fd4c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd57806f0a0 0x7fd5781f07c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:51.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.911+0000 7fd57fd4c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd5780ad370 0x7fd5781f0d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:51.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.911+0000 7fd57fd4c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5780c7140 con 0x7fd57806d2d0 2023-10-19T12:44:51.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.911+0000 7fd57fd4c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5780c7420 con 0x7fd5780ad370 2023-10-19T12:44:51.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.911+0000 7fd57fd4c640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5780c7700 con 0x7fd57806f0a0 2023-10-19T12:44:51.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.912+0000 7fd57dae8640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd57806d2d0 0x7fd5780c81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:51.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.912+0000 7fd57dae8640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd57806d2d0 0x7fd5780c81e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59422/0 (socket says 172.21.15.57:59422) 2023-10-19T12:44:51.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.912+0000 7fd57dae8640 1 -- 172.21.15.57:0/471774970 learned_addr learned my addr 172.21.15.57:0/471774970 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:51.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.912+0000 7fd57d2e7640 1 --2- 172.21.15.57:0/471774970 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd57806f0a0 0x7fd5781f07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:51.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.912+0000 7fd57e2e9640 1 --2- 172.21.15.57:0/471774970 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd5780ad370 0x7fd5781f0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:51.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.912+0000 7fd57dae8640 1 -- 172.21.15.57:0/471774970 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd57806f0a0 msgr2=0x7fd5781f07c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:51.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.912+0000 7fd57dae8640 1 --2- 172.21.15.57:0/471774970 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd57806f0a0 0x7fd5781f07c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:51.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.912+0000 7fd57dae8640 1 -- 172.21.15.57:0/471774970 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd5780ad370 msgr2=0x7fd5781f0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:51.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.912+0000 7fd57dae8640 1 --2- 172.21.15.57:0/471774970 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd5780ad370 0x7fd5781f0d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:51.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.912+0000 7fd57dae8640 1 -- 172.21.15.57:0/471774970 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd568009560 con 0x7fd57806d2d0 2023-10-19T12:44:51.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.912+0000 7fd57dae8640 1 --2- 172.21.15.57:0/471774970 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd57806d2d0 0x7fd5780c81e0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fd57400c870 tx=0x7fd57400cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:51.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.912+0000 7fd56e7fc640 1 -- 172.21.15.57:0/471774970 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd574003b70 con 0x7fd57806d2d0 2023-10-19T12:44:51.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.912+0000 7fd56e7fc640 1 -- 172.21.15.57:0/471774970 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fd574003cf0 con 0x7fd57806d2d0 2023-10-19T12:44:51.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.912+0000 7fd56e7fc640 1 -- 172.21.15.57:0/471774970 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd574003ea0 con 0x7fd57806d2d0 2023-10-19T12:44:51.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.912+0000 7fd57fd4c640 1 -- 172.21.15.57:0/471774970 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5780c79e0 con 0x7fd57806d2d0 2023-10-19T12:44:51.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.912+0000 7fd57fd4c640 1 -- 172.21.15.57:0/471774970 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5780c8720 con 0x7fd57806d2d0 2023-10-19T12:44:51.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.913+0000 7fd57fd4c640 1 -- 172.21.15.57:0/471774970 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5780ce1d0 con 0x7fd57806d2d0 2023-10-19T12:44:51.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.914+0000 7fd56e7fc640 1 -- 172.21.15.57:0/471774970 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fd57400aab0 con 0x7fd57806d2d0 2023-10-19T12:44:51.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.914+0000 7fd56e7fc640 1 --2- 172.21.15.57:0/471774970 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fd55c06ae40 0x7fd55c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:51.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.914+0000 7fd56e7fc640 1 -- 172.21.15.57:0/471774970 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fd5740899f0 con 0x7fd57806d2d0 2023-10-19T12:44:51.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.914+0000 7fd57d2e7640 1 --2- 172.21.15.57:0/471774970 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fd55c06ae40 0x7fd55c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:51.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.915+0000 7fd57d2e7640 1 --2- 172.21.15.57:0/471774970 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fd55c06ae40 0x7fd55c06d300 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fd568035000 tx=0x7fd568004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:51.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:51.918+0000 7fd56e7fc640 1 -- 172.21.15.57:0/471774970 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd574054730 con 0x7fd57806d2d0 2023-10-19T12:44:52.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.131+0000 7fd57fd4c640 1 -- 172.21.15.57:0/471774970 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd5780cc490 con 0x7fd57806d2d0 2023-10-19T12:44:52.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.131+0000 7fd56e7fc640 1 -- 172.21.15.57:0/471774970 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4906 (secure 0 0 0) 0x7fd574057d50 con 0x7fd57806d2d0 2023-10-19T12:44:52.135 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:44:52.136 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4357,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/4228712742","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":4228712742},{"type":"v1","addr":"172.21.15.57:6835","nonce":4228712742}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4360,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/971892630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":971892630},{"type":"v1","addr":"172.21.15.57:6834","nonce":971892630}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4372,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/104495983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":104495983},{"type":"v1","addr":"172.21.15.179:6836","nonce":104495983}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T12:44:18.419669+0000","modified":"2023-10-19T12:44:40.545928+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4374},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4374":{"gid":4374,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.179:6837/622550365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":622550365},{"type":"v1","addr":"172.21.15.179:6837","nonce":622550365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-19T12:44:52.137 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 9 2023-10-19T12:44:52.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.133+0000 7fd57fd4c640 1 -- 172.21.15.57:0/471774970 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fd55c06ae40 msgr2=0x7fd55c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:52.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.133+0000 7fd57fd4c640 1 --2- 172.21.15.57:0/471774970 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fd55c06ae40 0x7fd55c06d300 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fd568035000 tx=0x7fd568004730 comp rx=0 tx=0).stop 2023-10-19T12:44:52.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.133+0000 7fd57fd4c640 1 -- 172.21.15.57:0/471774970 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd57806d2d0 msgr2=0x7fd5780c81e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:52.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.133+0000 7fd57fd4c640 1 --2- 172.21.15.57:0/471774970 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd57806d2d0 0x7fd5780c81e0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fd57400c870 tx=0x7fd57400cd30 comp rx=0 tx=0).stop 2023-10-19T12:44:52.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.133+0000 7fd57fd4c640 1 -- 172.21.15.57:0/471774970 shutdown_connections 2023-10-19T12:44:52.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.133+0000 7fd57fd4c640 1 --2- 172.21.15.57:0/471774970 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fd55c06ae40 0x7fd55c06d300 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:52.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.133+0000 7fd57fd4c640 1 --2- 172.21.15.57:0/471774970 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd5780ad370 0x7fd5781f0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:52.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.133+0000 7fd57fd4c640 1 --2- 172.21.15.57:0/471774970 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd57806f0a0 0x7fd5781f07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:52.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.133+0000 7fd57fd4c640 1 --2- 172.21.15.57:0/471774970 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd57806d2d0 0x7fd5780c81e0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:52.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.133+0000 7fd57fd4c640 1 -- 172.21.15.57:0/471774970 >> 172.21.15.57:0/471774970 conn(0x7fd5780b0670 msgr2=0x7fd5781587c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:52.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.133+0000 7fd57fd4c640 1 -- 172.21.15.57:0/471774970 shutdown_connections 2023-10-19T12:44:52.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.133+0000 7fd57fd4c640 1 -- 172.21.15.57:0/471774970 wait complete. 2023-10-19T12:44:52.549 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 9, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-19T12:44:18.419669+0000', 'modified': '2023-10-19T12:44:40.545928+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4374}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4374': {'gid': 4374, 'name': 'd', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.179:6837/622550365', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6835', 'nonce': 622550365}, {'type': 'v1', 'addr': '172.21.15.179:6837', 'nonce': 622550365}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-19T12:44:52.550 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-19T12:44:52.550 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2023-10-19T12:44:52.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.664+0000 7f2335e10640 1 -- 172.21.15.57:0/1593634358 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f23300cbd20 msgr2=0x7f2330158300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:52.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.664+0000 7f2335e10640 1 --2- 172.21.15.57:0/1593634358 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f23300cbd20 0x7f2330158300 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f2320009830 tx=0x7f232002bd60 comp rx=0 tx=0).stop 2023-10-19T12:44:52.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.664+0000 7f2335e10640 1 -- 172.21.15.57:0/1593634358 shutdown_connections 2023-10-19T12:44:52.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.664+0000 7f2335e10640 1 --2- 172.21.15.57:0/1593634358 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f23300afa20 0x7f23300705a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:52.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.664+0000 7f2335e10640 1 --2- 172.21.15.57:0/1593634358 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f23300cbd20 0x7f2330158300 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:52.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.664+0000 7f2335e10640 1 -- 172.21.15.57:0/1593634358 >> 172.21.15.57:0/1593634358 conn(0x7f2330069930 msgr2=0x7f2330069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:52.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.664+0000 7f2335e10640 1 -- 172.21.15.57:0/1593634358 shutdown_connections 2023-10-19T12:44:52.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.664+0000 7f2335e10640 1 -- 172.21.15.57:0/1593634358 wait complete. 2023-10-19T12:44:52.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.665+0000 7f2335e10640 1 Processor -- start 2023-10-19T12:44:52.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.666+0000 7f2335e10640 1 -- start start 2023-10-19T12:44:52.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.666+0000 7f2335e10640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f23300afa20 0x7f2330148510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:52.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.666+0000 7f2335e10640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f23300b0530 0x7f2330159d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:52.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.666+0000 7f2335e10640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f23300cbd20 0x7f233013d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:52.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.666+0000 7f2335e10640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2330139d90 con 0x7f23300cbd20 2023-10-19T12:44:52.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.666+0000 7f2335e10640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f233015a2d0 con 0x7f23300afa20 2023-10-19T12:44:52.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.666+0000 7f2335e10640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2330158cf0 con 0x7f23300b0530 2023-10-19T12:44:52.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.667+0000 7f232ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f23300cbd20 0x7f233013d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:52.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.667+0000 7f232ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f23300cbd20 0x7f233013d150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59424/0 (socket says 172.21.15.57:59424) 2023-10-19T12:44:52.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.667+0000 7f232ffff640 1 -- 172.21.15.57:0/618223662 learned_addr learned my addr 172.21.15.57:0/618223662 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:52.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.667+0000 7f232f7fe640 1 --2- 172.21.15.57:0/618223662 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f23300afa20 0x7f2330148510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:52.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.667+0000 7f232effd640 1 --2- 172.21.15.57:0/618223662 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f23300b0530 0x7f2330159d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:52.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.667+0000 7f232ffff640 1 -- 172.21.15.57:0/618223662 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f23300b0530 msgr2=0x7f2330159d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:52.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.667+0000 7f232ffff640 1 --2- 172.21.15.57:0/618223662 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f23300b0530 0x7f2330159d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:52.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.667+0000 7f232ffff640 1 -- 172.21.15.57:0/618223662 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f23300afa20 msgr2=0x7f2330148510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:52.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.667+0000 7f232ffff640 1 --2- 172.21.15.57:0/618223662 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f23300afa20 0x7f2330148510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:52.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.667+0000 7f232ffff640 1 -- 172.21.15.57:0/618223662 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2324009490 con 0x7f23300cbd20 2023-10-19T12:44:52.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.667+0000 7f232ffff640 1 --2- 172.21.15.57:0/618223662 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f23300cbd20 0x7f233013d150 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f232800cbc0 tx=0x7f232800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:52.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.667+0000 7f2317fff640 1 -- 172.21.15.57:0/618223662 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f232800aca0 con 0x7f23300cbd20 2023-10-19T12:44:52.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.667+0000 7f2317fff640 1 -- 172.21.15.57:0/618223662 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f2328003970 con 0x7f23300cbd20 2023-10-19T12:44:52.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.667+0000 7f2317fff640 1 -- 172.21.15.57:0/618223662 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2328003d80 con 0x7f23300cbd20 2023-10-19T12:44:52.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.667+0000 7f2335e10640 1 -- 172.21.15.57:0/618223662 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2320009490 con 0x7f23300cbd20 2023-10-19T12:44:52.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.667+0000 7f2335e10640 1 -- 172.21.15.57:0/618223662 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23301592b0 con 0x7f23300cbd20 2023-10-19T12:44:52.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.669+0000 7f2317fff640 1 -- 172.21.15.57:0/618223662 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f2328004030 con 0x7f23300cbd20 2023-10-19T12:44:52.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.669+0000 7f2317fff640 1 --2- 172.21.15.57:0/618223662 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f231c06ae70 0x7f231c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:52.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.669+0000 7f2317fff640 1 -- 172.21.15.57:0/618223662 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f232808a230 con 0x7f23300cbd20 2023-10-19T12:44:52.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.669+0000 7f232f7fe640 1 --2- 172.21.15.57:0/618223662 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f231c06ae70 0x7f231c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:52.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.670+0000 7f232f7fe640 1 --2- 172.21.15.57:0/618223662 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f231c06ae70 0x7f231c06d330 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f2320036000 tx=0x7f2320004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:52.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.670+0000 7f233899e740 1 Processor -- start 2023-10-19T12:44:52.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.670+0000 7f233899e740 1 -- start start 2023-10-19T12:44:52.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.671+0000 7f233899e740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557a65bb7b40 0x557a65b8aae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:52.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.671+0000 7f233899e740 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x557a65bb7f20 0x557a65bba8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:52.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.671+0000 7f233899e740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557a65b8b020 0x557a65bbae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:52.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.671+0000 7f233899e740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a65a9c730 con 0x557a65bb7b40 2023-10-19T12:44:52.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.671+0000 7f233899e740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a65afd780 con 0x557a65b8b020 2023-10-19T12:44:52.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.671+0000 7f233899e740 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x557a65afae70 con 0x557a65bb7f20 2023-10-19T12:44:52.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.671+0000 7f232f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557a65bb7b40 0x557a65b8aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:52.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.671+0000 7f232f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557a65bb7b40 0x557a65b8aae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59434/0 (socket says 172.21.15.57:59434) 2023-10-19T12:44:52.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.671+0000 7f232f7fe640 1 -- 172.21.15.57:0/1741710950 learned_addr learned my addr 172.21.15.57:0/1741710950 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:52.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.671+0000 7f232ffff640 1 --2- 172.21.15.57:0/1741710950 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557a65b8b020 0x557a65bbae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:52.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.671+0000 7f232effd640 1 --2- 172.21.15.57:0/1741710950 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x557a65bb7f20 0x557a65bba8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:52.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.671+0000 7f232f7fe640 1 -- 172.21.15.57:0/1741710950 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x557a65bb7f20 msgr2=0x557a65bba8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T12:44:52.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.671+0000 7f232f7fe640 1 --2- 172.21.15.57:0/1741710950 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x557a65bb7f20 0x557a65bba8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:52.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.671+0000 7f232f7fe640 1 -- 172.21.15.57:0/1741710950 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557a65b8b020 msgr2=0x557a65bbae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T12:44:52.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.671+0000 7f232f7fe640 1 --2- 172.21.15.57:0/1741710950 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557a65b8b020 0x557a65bbae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:52.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.671+0000 7f232f7fe640 1 -- 172.21.15.57:0/1741710950 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557a65afb3d0 con 0x557a65bb7b40 2023-10-19T12:44:52.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.671+0000 7f232f7fe640 1 --2- 172.21.15.57:0/1741710950 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557a65bb7b40 0x557a65b8aae0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f232002c370 tx=0x7f2320040000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8c72107ef7b8cb server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:52.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.672+0000 7f23157fa640 1 -- 172.21.15.57:0/1741710950 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2320003730 con 0x557a65bb7b40 2023-10-19T12:44:52.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.672+0000 7f23157fa640 1 -- 172.21.15.57:0/1741710950 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f2320003980 con 0x557a65bb7b40 2023-10-19T12:44:52.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.672+0000 7f23157fa640 1 -- 172.21.15.57:0/1741710950 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2320003d70 con 0x557a65bb7b40 2023-10-19T12:44:52.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.672+0000 7f233899e740 1 -- 172.21.15.57:0/1741710950 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557a65bb7b40 msgr2=0x557a65b8aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:52.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.672+0000 7f233899e740 1 --2- 172.21.15.57:0/1741710950 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557a65bb7b40 0x557a65b8aae0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f232002c370 tx=0x7f2320040000 comp rx=0 tx=0).stop 2023-10-19T12:44:52.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.672+0000 7f233899e740 1 -- 172.21.15.57:0/1741710950 shutdown_connections 2023-10-19T12:44:52.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.672+0000 7f233899e740 1 --2- 172.21.15.57:0/1741710950 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557a65b8b020 0x557a65bbae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:52.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.672+0000 7f233899e740 1 --2- 172.21.15.57:0/1741710950 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x557a65bb7f20 0x557a65bba8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:52.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.672+0000 7f233899e740 1 --2- 172.21.15.57:0/1741710950 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557a65bb7b40 0x557a65b8aae0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:52.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.672+0000 7f233899e740 1 -- 172.21.15.57:0/1741710950 >> 172.21.15.57:0/1741710950 conn(0x557a65b8cd50 msgr2=0x557a65bd58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:52.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.672+0000 7f233899e740 1 -- 172.21.15.57:0/1741710950 shutdown_connections 2023-10-19T12:44:52.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.672+0000 7f233899e740 1 -- 172.21.15.57:0/1741710950 wait complete. 2023-10-19T12:44:52.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.673+0000 7f233899e740 1 -- start start 2023-10-19T12:44:52.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.673+0000 7f233899e740 1 Processor -- start 2023-10-19T12:44:52.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.674+0000 7f233899e740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557a65bb7b40 0x557a65bbdcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:52.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.674+0000 7f233899e740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557a65bb7f20 0x557a65bcc8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:52.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.674+0000 7f233899e740 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x557a65bb5200 0x557a65bcce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:52.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.674+0000 7f233899e740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a65af6760 con 0x557a65bb7f20 2023-10-19T12:44:52.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.674+0000 7f233899e740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a65ab18f0 con 0x557a65bb7b40 2023-10-19T12:44:52.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.674+0000 7f233899e740 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x557a65ab1590 con 0x557a65bb5200 2023-10-19T12:44:52.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.674+0000 7f233899e740 10 client.0 _ll_register_callbacks cb 0x557a65aedee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-19T12:44:52.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.674+0000 7f232effd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557a65bb7f20 0x557a65bcc8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:52.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.674+0000 7f232effd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557a65bb7f20 0x557a65bcc8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59448/0 (socket says 172.21.15.57:59448) 2023-10-19T12:44:52.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.674+0000 7f232effd640 1 -- 172.21.15.57:0/598184669 learned_addr learned my addr 172.21.15.57:0/598184669 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:52.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.674+0000 7f232f7fe640 1 --2- 172.21.15.57:0/598184669 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557a65bb7b40 0x557a65bbdcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:52.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.674+0000 7f232ffff640 1 --2- 172.21.15.57:0/598184669 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x557a65bb5200 0x557a65bcce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:52.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.674+0000 7f232effd640 1 -- 172.21.15.57:0/598184669 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x557a65bb5200 msgr2=0x557a65bcce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T12:44:52.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.674+0000 7f232effd640 1 --2- 172.21.15.57:0/598184669 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x557a65bb5200 0x557a65bcce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:52.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.674+0000 7f232effd640 1 -- 172.21.15.57:0/598184669 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557a65bb7b40 msgr2=0x557a65bbdcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T12:44:52.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.674+0000 7f232effd640 1 --2- 172.21.15.57:0/598184669 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557a65bb7b40 0x557a65bbdcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:52.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.674+0000 7f232effd640 1 -- 172.21.15.57:0/598184669 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f233006d270 con 0x557a65bb7f20 2023-10-19T12:44:52.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.674+0000 7f232effd640 1 --2- 172.21.15.57:0/598184669 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557a65bb7f20 0x557a65bcc8c0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f2324012840 tx=0x7f2324012d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0ef633e44fec050 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:52.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.674+0000 7f230e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2023-10-19T12:44:52.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.675+0000 7f230e7fc640 1 -- 172.21.15.57:0/598184669 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2324017070 con 0x557a65bb7f20 2023-10-19T12:44:52.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.675+0000 7f233899e740 1 -- 172.21.15.57:0/598184669 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557a65aef740 con 0x557a65bb7f20 2023-10-19T12:44:52.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.675+0000 7f230e7fc640 1 -- 172.21.15.57:0/598184669 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f232401ddc0 con 0x557a65bb7f20 2023-10-19T12:44:52.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.675+0000 7f230e7fc640 1 -- 172.21.15.57:0/598184669 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=9) v2 ==== 24+0+0 (secure 0 0 0) 0x7f232400f920 con 0x557a65bb7f20 2023-10-19T12:44:52.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.675+0000 7f233899e740 10 client.4962 fetch_fsmap learned FSMap version 9 2023-10-19T12:44:52.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.675+0000 7f233899e740 1 -- 172.21.15.57:0/598184669 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=9,monmap=2+}) v3 -- 0x557a65af4de0 con 0x557a65bb7f20 2023-10-19T12:44:52.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.675+0000 7f230e7fc640 1 -- 172.21.15.57:0/598184669 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x557a65aef740 con 0x557a65bb7f20 2023-10-19T12:44:52.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.675+0000 7f230e7fc640 1 -- 172.21.15.57:0/598184669 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 9) v1 ==== 3302+0+0 (secure 0 0 0) 0x7f232400fa80 con 0x557a65bb7f20 2023-10-19T12:44:52.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.675+0000 7f233899e740 10 client.4962 fetch_fsmap finished waiting for FSMap version 9 2023-10-19T12:44:52.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.675+0000 7f233899e740 10 client.4962 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-19T12:44:52.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.675+0000 7f233899e740 20 client.4962 populate_metadata read hostname 'smithi057' 2023-10-19T12:44:52.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.675+0000 7f233899e740 1 --2- 172.21.15.57:0/598184669 >> [v2:172.21.15.179:6835/622550365,v1:172.21.15.179:6837/622550365] conn(0x557a65be56d0 0x557a65bce2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:52.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.675+0000 7f233899e740 4 client.4962 mds_command: new command op to 4374 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-19T12:44:52.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.675+0000 7f233899e740 1 -- 172.21.15.57:0/598184669 --> [v2:172.21.15.179:6835/622550365,v1:172.21.15.179:6837/622550365] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557a65aef460 con 0x557a65be56d0 2023-10-19T12:44:52.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.675+0000 7f232f7fe640 1 --2- 172.21.15.57:0/598184669 >> [v2:172.21.15.179:6835/622550365,v1:172.21.15.179:6837/622550365] conn(0x557a65be56d0 0x557a65bce2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:52.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.676+0000 7f232f7fe640 1 --2- 172.21.15.57:0/598184669 >> [v2:172.21.15.179:6835/622550365,v1:172.21.15.179:6837/622550365] conn(0x557a65be56d0 0x557a65bce2b0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b200fc3e755fa958 server_cookie=7a2b7b13108e120d in_seq=0 out_seq=0 2023-10-19T12:44:52.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.676+0000 7f230e7fc640 10 client.4962 ms_handle_connect on v2:172.21.15.179:6835/622550365 2023-10-19T12:44:52.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.676+0000 7f230e7fc640 1 -- 172.21.15.57:0/598184669 <== mds.0 v2:172.21.15.179:6835/622550365 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x557a65aef460 con 0x557a65be56d0 2023-10-19T12:44:52.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.676+0000 7f230e7fc640 10 client.4962 handle_command_reply: tid=0 2023-10-19T12:44:52.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.676+0000 7f233899e740 1 client.4962 shutdown 2023-10-19T12:44:52.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.676+0000 7f233899e740 1 -- 172.21.15.57:0/598184669 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557a65bb7f20 msgr2=0x557a65bcc8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:52.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.676+0000 7f233899e740 1 --2- 172.21.15.57:0/598184669 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557a65bb7f20 0x557a65bcc8c0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f2324012840 tx=0x7f2324012d00 comp rx=0 tx=0).stop 2023-10-19T12:44:52.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.676+0000 7f233899e740 1 -- 172.21.15.57:0/598184669 shutdown_connections 2023-10-19T12:44:52.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.676+0000 7f233899e740 1 --2- 172.21.15.57:0/598184669 >> [v2:172.21.15.179:6835/622550365,v1:172.21.15.179:6837/622550365] conn(0x557a65be56d0 0x557a65bce2b0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:52.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.676+0000 7f233899e740 1 --2- 172.21.15.57:0/598184669 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x557a65bb5200 0x557a65bcce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:52.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.676+0000 7f233899e740 1 --2- 172.21.15.57:0/598184669 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557a65bb7f20 0x557a65bcc8c0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:52.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.676+0000 7f230e7fc640 0 client.4962 ms_handle_reset on v2:172.21.15.179:6835/622550365 2023-10-19T12:44:52.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.676+0000 7f233899e740 1 --2- 172.21.15.57:0/598184669 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557a65bb7b40 0x557a65bbdcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:52.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.677+0000 7f233899e740 1 -- 172.21.15.57:0/598184669 >> 172.21.15.57:0/598184669 conn(0x557a65b8b020 msgr2=0x557a65bd8de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:52.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.677+0000 7f233899e740 1 -- 172.21.15.57:0/598184669 shutdown_connections 2023-10-19T12:44:52.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.677+0000 7f233899e740 1 -- 172.21.15.57:0/598184669 wait complete. 2023-10-19T12:44:52.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.677+0000 7f233899e740 20 client.4962 trim_cache size 0 max 16384 2023-10-19T12:44:52.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.692+0000 7f233899e740 1 Processor -- start 2023-10-19T12:44:52.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.692+0000 7f233899e740 1 -- start start 2023-10-19T12:44:52.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.693+0000 7f233899e740 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x557a65bb5200 0x557a65bbdba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:52.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.693+0000 7f233899e740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557a65bb7b40 0x557a65bd71f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:52.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.693+0000 7f233899e740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557a65bb7f20 0x557a65bb41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:52.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.693+0000 7f233899e740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a65a77270 con 0x557a65bb7f20 2023-10-19T12:44:52.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.693+0000 7f233899e740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a65bbced0 con 0x557a65bb7b40 2023-10-19T12:44:52.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.693+0000 7f233899e740 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x557a65aacee0 con 0x557a65bb5200 2023-10-19T12:44:52.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.693+0000 7f232ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557a65bb7f20 0x557a65bb41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:52.707 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.693+0000 7f232ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557a65bb7f20 0x557a65bb41b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59456/0 (socket says 172.21.15.57:59456) 2023-10-19T12:44:52.707 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.693+0000 7f232ffff640 1 -- 172.21.15.57:0/1661825280 learned_addr learned my addr 172.21.15.57:0/1661825280 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:52.707 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.693+0000 7f232f7fe640 1 --2- 172.21.15.57:0/1661825280 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x557a65bb5200 0x557a65bbdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:52.707 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.693+0000 7f232effd640 1 --2- 172.21.15.57:0/1661825280 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557a65bb7b40 0x557a65bd71f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:52.707 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.693+0000 7f232ffff640 1 -- 172.21.15.57:0/1661825280 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x557a65bb5200 msgr2=0x557a65bbdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T12:44:52.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.693+0000 7f232ffff640 1 --2- 172.21.15.57:0/1661825280 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x557a65bb5200 0x557a65bbdba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:52.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.693+0000 7f232ffff640 1 -- 172.21.15.57:0/1661825280 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557a65bb7b40 msgr2=0x557a65bd71f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T12:44:52.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.693+0000 7f232ffff640 1 --2- 172.21.15.57:0/1661825280 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557a65bb7b40 0x557a65bd71f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:52.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.693+0000 7f232ffff640 1 -- 172.21.15.57:0/1661825280 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557a65a93d00 con 0x557a65bb7f20 2023-10-19T12:44:52.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.693+0000 7f232ffff640 1 --2- 172.21.15.57:0/1661825280 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557a65bb7f20 0x557a65bb41b0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f2328058cf0 tx=0x7f232801d040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=216b8c1eb1cac1e5 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:52.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.693+0000 7f2315ffb640 1 -- 172.21.15.57:0/1661825280 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2328060fb0 con 0x557a65bb7f20 2023-10-19T12:44:52.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.694+0000 7f2315ffb640 1 -- 172.21.15.57:0/1661825280 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f2328057100 con 0x557a65bb7f20 2023-10-19T12:44:52.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.694+0000 7f2315ffb640 1 -- 172.21.15.57:0/1661825280 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2328058250 con 0x557a65bb7f20 2023-10-19T12:44:52.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.694+0000 7f233899e740 1 -- 172.21.15.57:0/1661825280 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557a65bb7f20 msgr2=0x557a65bb41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:52.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.694+0000 7f233899e740 1 --2- 172.21.15.57:0/1661825280 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557a65bb7f20 0x557a65bb41b0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f2328058cf0 tx=0x7f232801d040 comp rx=0 tx=0).stop 2023-10-19T12:44:52.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.694+0000 7f233899e740 1 -- 172.21.15.57:0/1661825280 shutdown_connections 2023-10-19T12:44:52.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.694+0000 7f233899e740 1 --2- 172.21.15.57:0/1661825280 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557a65bb7f20 0x557a65bb41b0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:52.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.694+0000 7f233899e740 1 --2- 172.21.15.57:0/1661825280 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557a65bb7b40 0x557a65bd71f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:52.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.694+0000 7f233899e740 1 --2- 172.21.15.57:0/1661825280 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x557a65bb5200 0x557a65bbdba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:52.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.694+0000 7f233899e740 1 -- 172.21.15.57:0/1661825280 >> 172.21.15.57:0/1661825280 conn(0x557a65b8b020 msgr2=0x557a65b86440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:52.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.694+0000 7f233899e740 1 -- 172.21.15.57:0/1661825280 shutdown_connections 2023-10-19T12:44:52.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.694+0000 7f233899e740 1 -- 172.21.15.57:0/1661825280 wait complete. 2023-10-19T12:44:52.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.695+0000 7f233899e740 1 -- start start 2023-10-19T12:44:52.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.695+0000 7f233899e740 1 Processor -- start 2023-10-19T12:44:52.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.695+0000 7f233899e740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557a65bb5200 0x557a65c42210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:52.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.696+0000 7f233899e740 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x557a65bb7b40 0x557a65c27a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:52.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.696+0000 7f233899e740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557a65bb7f20 0x557a65c27f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:52.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.696+0000 7f233899e740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a65c42aa0 con 0x557a65bb5200 2023-10-19T12:44:52.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.696+0000 7f233899e740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a65bcd0b0 con 0x557a65bb7f20 2023-10-19T12:44:52.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.696+0000 7f233899e740 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x557a65bcd5e0 con 0x557a65bb7b40 2023-10-19T12:44:52.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.696+0000 7f233899e740 10 client.0 _ll_register_callbacks cb 0x7f2320043070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-19T12:44:52.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.696+0000 7f232f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557a65bb5200 0x557a65c42210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:52.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.696+0000 7f232f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557a65bb5200 0x557a65c42210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59472/0 (socket says 172.21.15.57:59472) 2023-10-19T12:44:52.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.696+0000 7f232f7fe640 1 -- 172.21.15.57:0/2798991790 learned_addr learned my addr 172.21.15.57:0/2798991790 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:52.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.696+0000 7f232ffff640 1 --2- 172.21.15.57:0/2798991790 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557a65bb7f20 0x557a65c27f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:52.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.696+0000 7f232effd640 1 --2- 172.21.15.57:0/2798991790 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x557a65bb7b40 0x557a65c27a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:52.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.696+0000 7f232f7fe640 1 -- 172.21.15.57:0/2798991790 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x557a65bb7b40 msgr2=0x557a65c27a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T12:44:52.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.696+0000 7f232f7fe640 1 --2- 172.21.15.57:0/2798991790 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x557a65bb7b40 0x557a65c27a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:52.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.696+0000 7f232f7fe640 1 -- 172.21.15.57:0/2798991790 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557a65bb7f20 msgr2=0x557a65c27f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T12:44:52.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.696+0000 7f232f7fe640 1 --2- 172.21.15.57:0/2798991790 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557a65bb7f20 0x557a65c27f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:52.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.696+0000 7f232f7fe640 1 -- 172.21.15.57:0/2798991790 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23301592b0 con 0x557a65bb5200 2023-10-19T12:44:52.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.696+0000 7f232f7fe640 1 --2- 172.21.15.57:0/2798991790 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557a65bb5200 0x557a65c42210 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f232004c000 tx=0x7f2320005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe1e74668a623193 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:52.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.696+0000 7f230e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2023-10-19T12:44:52.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.696+0000 7f230e7fc640 1 -- 172.21.15.57:0/2798991790 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f232005b040 con 0x557a65bb5200 2023-10-19T12:44:52.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.696+0000 7f230e7fc640 1 -- 172.21.15.57:0/2798991790 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f2320061970 con 0x557a65bb5200 2023-10-19T12:44:52.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.696+0000 7f233899e740 1 -- 172.21.15.57:0/2798991790 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2320003980 con 0x557a65bb5200 2023-10-19T12:44:52.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.696+0000 7f230e7fc640 1 -- 172.21.15.57:0/2798991790 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=9) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2320061b30 con 0x557a65bb5200 2023-10-19T12:44:52.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.696+0000 7f233899e740 10 client.4968 fetch_fsmap learned FSMap version 9 2023-10-19T12:44:52.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.696+0000 7f233899e740 1 -- 172.21.15.57:0/2798991790 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=9,monmap=2+}) v3 -- 0x557a65ab94f0 con 0x557a65bb5200 2023-10-19T12:44:52.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.697+0000 7f230e7fc640 1 -- 172.21.15.57:0/2798991790 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f2320003980 con 0x557a65bb5200 2023-10-19T12:44:52.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.697+0000 7f230e7fc640 1 -- 172.21.15.57:0/2798991790 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 9) v1 ==== 3302+0+0 (secure 0 0 0) 0x7f2320061c90 con 0x557a65bb5200 2023-10-19T12:44:52.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.697+0000 7f233899e740 10 client.4968 fetch_fsmap finished waiting for FSMap version 9 2023-10-19T12:44:52.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.697+0000 7f233899e740 10 client.4968 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-19T12:44:52.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.697+0000 7f233899e740 20 client.4968 populate_metadata read hostname 'smithi057' 2023-10-19T12:44:52.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.697+0000 7f233899e740 1 --2- 172.21.15.57:0/2798991790 >> [v2:172.21.15.179:6835/622550365,v1:172.21.15.179:6837/622550365] conn(0x557a65be56d0 0x557a65bd8fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:52.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.697+0000 7f233899e740 4 client.4968 mds_command: new command op to 4374 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-19T12:44:52.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.697+0000 7f233899e740 1 -- 172.21.15.57:0/2798991790 --> [v2:172.21.15.179:6835/622550365,v1:172.21.15.179:6837/622550365] -- command(tid 0: {"prefix": "status"}) v1 -- 0x557a657f86c0 con 0x557a65be56d0 2023-10-19T12:44:52.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.697+0000 7f232effd640 1 --2- 172.21.15.57:0/2798991790 >> [v2:172.21.15.179:6835/622550365,v1:172.21.15.179:6837/622550365] conn(0x557a65be56d0 0x557a65bd8fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:52.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.697+0000 7f230e7fc640 10 client.4968 ms_handle_connect on v2:172.21.15.179:6835/622550365 2023-10-19T12:44:52.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.697+0000 7f232effd640 1 --2- 172.21.15.57:0/2798991790 >> [v2:172.21.15.179:6835/622550365,v1:172.21.15.179:6837/622550365] conn(0x557a65be56d0 0x557a65bd8fd0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=275d4d0b9f65d380 server_cookie=ee3c32a6ab559e22 in_seq=0 out_seq=0 2023-10-19T12:44:52.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.698+0000 7f230e7fc640 1 -- 172.21.15.57:0/2798991790 <== mds.0 v2:172.21.15.179:6835/622550365 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+330 (crc 0 0 0) 0x557a657f86c0 con 0x557a65be56d0 2023-10-19T12:44:52.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.698+0000 7f230e7fc640 10 client.4968 handle_command_reply: tid=0 2023-10-19T12:44:52.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.698+0000 7f233899e740 1 client.4968 shutdown 2023-10-19T12:44:52.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.698+0000 7f233899e740 1 -- 172.21.15.57:0/2798991790 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557a65bb5200 msgr2=0x557a65c42210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:52.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.698+0000 7f233899e740 1 --2- 172.21.15.57:0/2798991790 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557a65bb5200 0x557a65c42210 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f232004c000 tx=0x7f2320005850 comp rx=0 tx=0).stop 2023-10-19T12:44:52.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.698+0000 7f233899e740 1 -- 172.21.15.57:0/2798991790 shutdown_connections 2023-10-19T12:44:52.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.698+0000 7f233899e740 1 --2- 172.21.15.57:0/2798991790 >> [v2:172.21.15.179:6835/622550365,v1:172.21.15.179:6837/622550365] conn(0x557a65be56d0 0x557a65bd8fd0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:52.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.698+0000 7f233899e740 1 --2- 172.21.15.57:0/2798991790 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557a65bb7f20 0x557a65c27f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:52.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.698+0000 7f230e7fc640 0 client.4968 ms_handle_reset on v2:172.21.15.179:6835/622550365 2023-10-19T12:44:52.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.698+0000 7f233899e740 1 --2- 172.21.15.57:0/2798991790 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x557a65bb7b40 0x557a65c27a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:52.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.698+0000 7f233899e740 1 --2- 172.21.15.57:0/2798991790 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557a65bb5200 0x557a65c42210 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:52.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.698+0000 7f233899e740 1 -- 172.21.15.57:0/2798991790 >> 172.21.15.57:0/2798991790 conn(0x557a65b8b020 msgr2=0x557a65bbdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:52.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.698+0000 7f233899e740 1 -- 172.21.15.57:0/2798991790 shutdown_connections 2023-10-19T12:44:52.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.698+0000 7f233899e740 1 -- 172.21.15.57:0/2798991790 wait complete. 2023-10-19T12:44:52.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.698+0000 7f233899e740 20 client.4968 trim_cache size 0 max 16384 2023-10-19T12:44:52.722 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2023-10-19T12:44:52.722 INFO:teuthology.orchestra.run.smithi057.stdout: "cluster_fsid": "1e9beb19-3260-4dd7-85ef-5b399ce774ac", 2023-10-19T12:44:52.723 INFO:teuthology.orchestra.run.smithi057.stdout: "whoami": 0, 2023-10-19T12:44:52.723 INFO:teuthology.orchestra.run.smithi057.stdout: "id": 4374, 2023-10-19T12:44:52.723 INFO:teuthology.orchestra.run.smithi057.stdout: "want_state": "up:active", 2023-10-19T12:44:52.723 INFO:teuthology.orchestra.run.smithi057.stdout: "state": "up:active", 2023-10-19T12:44:52.724 INFO:teuthology.orchestra.run.smithi057.stdout: "fs_name": "cephfs", 2023-10-19T12:44:52.724 INFO:teuthology.orchestra.run.smithi057.stdout: "rank_uptime": 33.619154727000002, 2023-10-19T12:44:52.724 INFO:teuthology.orchestra.run.smithi057.stdout: "mdsmap_epoch": 9, 2023-10-19T12:44:52.724 INFO:teuthology.orchestra.run.smithi057.stdout: "osdmap_epoch": 25, 2023-10-19T12:44:52.724 INFO:teuthology.orchestra.run.smithi057.stdout: "osdmap_epoch_barrier": 25, 2023-10-19T12:44:52.725 INFO:teuthology.orchestra.run.smithi057.stdout: "uptime": 40.675179653000001 2023-10-19T12:44:52.725 INFO:teuthology.orchestra.run.smithi057.stdout:} 2023-10-19T12:44:52.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.698+0000 7f2335e10640 1 -- 172.21.15.57:0/618223662 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f231c06ae70 msgr2=0x7f231c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:52.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.698+0000 7f2335e10640 1 --2- 172.21.15.57:0/618223662 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f231c06ae70 0x7f231c06d330 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f2320036000 tx=0x7f2320004730 comp rx=0 tx=0).stop 2023-10-19T12:44:52.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.698+0000 7f2335e10640 1 -- 172.21.15.57:0/618223662 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f23300cbd20 msgr2=0x7f233013d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:52.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.698+0000 7f2335e10640 1 --2- 172.21.15.57:0/618223662 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f23300cbd20 0x7f233013d150 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f232800cbc0 tx=0x7f232800a060 comp rx=0 tx=0).stop 2023-10-19T12:44:52.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.698+0000 7f2335e10640 1 -- 172.21.15.57:0/618223662 shutdown_connections 2023-10-19T12:44:52.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.698+0000 7f2335e10640 1 --2- 172.21.15.57:0/618223662 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f231c06ae70 0x7f231c06d330 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:52.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.698+0000 7f2335e10640 1 --2- 172.21.15.57:0/618223662 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f23300cbd20 0x7f233013d150 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:52.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.698+0000 7f2335e10640 1 --2- 172.21.15.57:0/618223662 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f23300b0530 0x7f2330159d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:52.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.698+0000 7f2335e10640 1 --2- 172.21.15.57:0/618223662 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f23300afa20 0x7f2330148510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:52.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.698+0000 7f2335e10640 1 -- 172.21.15.57:0/618223662 >> 172.21.15.57:0/618223662 conn(0x7f2330069930 msgr2=0x7f23300c9730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:52.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.699+0000 7f2335e10640 1 -- 172.21.15.57:0/618223662 shutdown_connections 2023-10-19T12:44:52.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:52.699+0000 7f2335e10640 1 -- 172.21.15.57:0/618223662 wait complete. 2023-10-19T12:44:52.729 INFO:teuthology.run_tasks:Running task ceph-fuse... 2023-10-19T12:44:52.740 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2023-10-19T12:44:52.741 INFO:tasks.ceph_fuse:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2023-10-19T12:44:52.741 INFO:tasks.ceph_fuse:client.0 config is {} 2023-10-19T12:44:52.741 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-10-19T12:44:52.742 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-19T12:44:52.742 INFO:tasks.ceph_fuse:client.2 config is {} 2023-10-19T12:44:52.742 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-10-19T12:44:52.742 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2023-10-19T12:44:52.743 INFO:tasks.ceph_fuse:client.1 config is {} 2023-10-19T12:44:52.743 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-10-19T12:44:52.743 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-10-19T12:44:52.744 INFO:tasks.ceph_fuse:client.3 config is {} 2023-10-19T12:44:52.744 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-10-19T12:44:52.744 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2023-10-19T12:44:52.744 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:44:52.745 DEBUG:teuthology.orchestra.run.smithi179:> ip netns list 2023-10-19T12:44:52.763 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:44:52.763 DEBUG:teuthology.orchestra.run.smithi179:> sudo ip link delete ceph-brx 2023-10-19T12:44:52.834 INFO:teuthology.orchestra.run.smithi179.stderr:Cannot find device "ceph-brx" 2023-10-19T12:44:52.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-19T12:44:52.836 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:44:52.837 DEBUG:teuthology.orchestra.run.smithi057:> ip netns list 2023-10-19T12:44:52.852 INFO:teuthology.orchestra.run.smithi057.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2023-10-19T12:44:52.853 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:44:52.853 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2023-10-19T12:44:52.923 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:44:52.924 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip link delete ceph-brx 2023-10-19T12:44:53.024 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2023-10-19T12:44:53.024 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-19T12:44:53.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:53.131+0000 7f7bad38a640 1 -- 172.21.15.57:0/1106472449 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ba80cbd20 msgr2=0x7f7ba8158300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:53.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:53.131+0000 7f7bad38a640 1 --2- 172.21.15.57:0/1106472449 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ba80cbd20 0x7f7ba8158300 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f7ba80b10f0 tx=0x7f7b9802bd70 comp rx=0 tx=0).stop 2023-10-19T12:44:53.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:53.131+0000 7f7bad38a640 1 -- 172.21.15.57:0/1106472449 shutdown_connections 2023-10-19T12:44:53.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:53.131+0000 7f7bad38a640 1 --2- 172.21.15.57:0/1106472449 >> v2:172.21.15.179:3300/0 conn(0x7f7ba80b0530 0x7f7ba80b09a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:53.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:53.131+0000 7f7bad38a640 1 --2- 172.21.15.57:0/1106472449 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ba80cbd20 0x7f7ba8158300 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:53.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:53.131+0000 7f7bad38a640 1 -- 172.21.15.57:0/1106472449 >> 172.21.15.57:0/1106472449 conn(0x7f7ba8069930 msgr2=0x7f7ba8069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:53.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:53.131+0000 7f7bad38a640 1 -- 172.21.15.57:0/1106472449 shutdown_connections 2023-10-19T12:44:53.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:53.131+0000 7f7bad38a640 1 -- 172.21.15.57:0/1106472449 wait complete. 2023-10-19T12:44:53.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:53.132+0000 7f7bad38a640 1 Processor -- start 2023-10-19T12:44:53.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:53.132+0000 7f7bad38a640 1 -- start start 2023-10-19T12:44:53.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:53.133+0000 7f7bad38a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ba80afa20 0x7f7ba8156750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:53.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:53.133+0000 7f7bad38a640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7ba80b0530 0x7f7ba8158980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:53.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:53.133+0000 7f7bad38a640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7ba80cbd20 0x7f7ba814f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:53.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:53.133+0000 7f7bad38a640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ba81f12d0 con 0x7f7ba80afa20 2023-10-19T12:44:53.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:53.133+0000 7f7bad38a640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ba8158ec0 con 0x7f7ba80b0530 2023-10-19T12:44:53.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:53.133+0000 7f7bad38a640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ba8159190 con 0x7f7ba80cbd20 2023-10-19T12:44:53.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:53.133+0000 7f7ba6ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ba80afa20 0x7f7ba8156750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:53.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:53.133+0000 7f7ba6ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ba80afa20 0x7f7ba8156750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59482/0 (socket says 172.21.15.57:59482) 2023-10-19T12:44:53.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:53.133+0000 7f7ba6ffd640 1 -- 172.21.15.57:0/1750976198 learned_addr learned my addr 172.21.15.57:0/1750976198 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:53.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:53.133+0000 7f7ba77fe640 1 --2- 172.21.15.57:0/1750976198 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7ba80cbd20 0x7f7ba814f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:53.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:53.133+0000 7f7b9ffff640 1 --2- 172.21.15.57:0/1750976198 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7ba80b0530 0x7f7ba8158980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:53.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:53.133+0000 7f7ba6ffd640 1 -- 172.21.15.57:0/1750976198 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7ba80cbd20 msgr2=0x7f7ba814f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:53.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:53.133+0000 7f7ba6ffd640 1 --2- 172.21.15.57:0/1750976198 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7ba80cbd20 0x7f7ba814f380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:53.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:53.133+0000 7f7ba6ffd640 1 -- 172.21.15.57:0/1750976198 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7ba80b0530 msgr2=0x7f7ba8158980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:53.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:53.133+0000 7f7ba6ffd640 1 --2- 172.21.15.57:0/1750976198 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7ba80b0530 0x7f7ba8158980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:53.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:53.133+0000 7f7ba6ffd640 1 -- 172.21.15.57:0/1750976198 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b98009490 con 0x7f7ba80afa20 2023-10-19T12:44:53.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:53.134+0000 7f7ba6ffd640 1 --2- 172.21.15.57:0/1750976198 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ba80afa20 0x7f7ba8156750 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f7b98009780 tx=0x7f7b980038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:53.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:53.134+0000 7f7b9f7fe640 1 -- 172.21.15.57:0/1750976198 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b98003ea0 con 0x7f7ba80afa20 2023-10-19T12:44:53.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:53.134+0000 7f7b9f7fe640 1 -- 172.21.15.57:0/1750976198 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f7b98005110 con 0x7f7ba80afa20 2023-10-19T12:44:53.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:53.134+0000 7f7bad38a640 1 -- 172.21.15.57:0/1750976198 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ba8159470 con 0x7f7ba80afa20 2023-10-19T12:44:53.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:53.134+0000 7f7b9f7fe640 1 -- 172.21.15.57:0/1750976198 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b98005500 con 0x7f7ba80afa20 2023-10-19T12:44:53.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:53.134+0000 7f7bad38a640 1 -- 172.21.15.57:0/1750976198 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ba8159a30 con 0x7f7ba80afa20 2023-10-19T12:44:53.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:53.135+0000 7f7bad38a640 1 -- 172.21.15.57:0/1750976198 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ba81f1410 con 0x7f7ba80afa20 2023-10-19T12:44:53.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:53.135+0000 7f7b9f7fe640 1 -- 172.21.15.57:0/1750976198 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f7b980047a0 con 0x7f7ba80afa20 2023-10-19T12:44:53.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:53.136+0000 7f7b9f7fe640 1 --2- 172.21.15.57:0/1750976198 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f7b8806ae40 0x7f7b8806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:53.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:53.136+0000 7f7b9f7fe640 1 -- 172.21.15.57:0/1750976198 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f7b980abcf0 con 0x7f7ba80afa20 2023-10-19T12:44:53.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:53.136+0000 7f7b9ffff640 1 --2- 172.21.15.57:0/1750976198 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f7b8806ae40 0x7f7b8806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:53.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:53.137+0000 7f7b9ffff640 1 --2- 172.21.15.57:0/1750976198 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f7b8806ae40 0x7f7b8806d300 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f7ba8069ef0 tx=0x7f7b90007e50 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:53.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:53.140+0000 7f7b9f7fe640 1 -- 172.21.15.57:0/1750976198 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7ba81f1410 con 0x7f7ba80afa20 2023-10-19T12:44:53.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:53.354+0000 7f7bad38a640 1 -- 172.21.15.57:0/1750976198 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f7ba81f1410 con 0x7f7ba80afa20 2023-10-19T12:44:53.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:53.354+0000 7f7b9f7fe640 1 -- 172.21.15.57:0/1750976198 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) v1 ==== 53+0+88 (secure 0 0 0) 0x7f7ba81f1410 con 0x7f7ba80afa20 2023-10-19T12:44:53.358 INFO:teuthology.orchestra.run.smithi057.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-10-19T12:44:53.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:53.356+0000 7f7bad38a640 1 -- 172.21.15.57:0/1750976198 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f7b8806ae40 msgr2=0x7f7b8806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:53.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:53.356+0000 7f7bad38a640 1 --2- 172.21.15.57:0/1750976198 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f7b8806ae40 0x7f7b8806d300 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f7ba8069ef0 tx=0x7f7b90007e50 comp rx=0 tx=0).stop 2023-10-19T12:44:53.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:53.356+0000 7f7bad38a640 1 -- 172.21.15.57:0/1750976198 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ba80afa20 msgr2=0x7f7ba8156750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:53.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:53.356+0000 7f7bad38a640 1 --2- 172.21.15.57:0/1750976198 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ba80afa20 0x7f7ba8156750 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f7b98009780 tx=0x7f7b980038a0 comp rx=0 tx=0).stop 2023-10-19T12:44:53.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:53.356+0000 7f7bad38a640 1 -- 172.21.15.57:0/1750976198 shutdown_connections 2023-10-19T12:44:53.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:53.356+0000 7f7bad38a640 1 --2- 172.21.15.57:0/1750976198 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f7b8806ae40 0x7f7b8806d300 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:53.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:53.356+0000 7f7bad38a640 1 --2- 172.21.15.57:0/1750976198 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7ba80cbd20 0x7f7ba814f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:53.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:53.356+0000 7f7bad38a640 1 --2- 172.21.15.57:0/1750976198 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7ba80b0530 0x7f7ba8158980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:53.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:53.356+0000 7f7bad38a640 1 --2- 172.21.15.57:0/1750976198 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ba80afa20 0x7f7ba8156750 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:53.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:53.356+0000 7f7bad38a640 1 -- 172.21.15.57:0/1750976198 >> 172.21.15.57:0/1750976198 conn(0x7f7ba8069930 msgr2=0x7f7ba80ca870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:53.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:53.356+0000 7f7bad38a640 1 -- 172.21.15.57:0/1750976198 shutdown_connections 2023-10-19T12:44:53.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:53.356+0000 7f7bad38a640 1 -- 172.21.15.57:0/1750976198 wait complete. 2023-10-19T12:44:53.372 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-10-19T12:44:53.373 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-19T12:44:53.373 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi057.front.sepia.ceph.com 2023-10-19T12:44:53.373 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-19T12:44:53.374 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-19T12:44:53.374 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-10-19T12:44:53.374 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-10-19T12:44:53.374 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-19T12:44:53.375 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-19T12:44:53.375 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:44:53.375 DEBUG:teuthology.orchestra.run.smithi057:> ip addr 2023-10-19T12:44:53.391 INFO:teuthology.orchestra.run.smithi057.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-19T12:44:53.391 INFO:teuthology.orchestra.run.smithi057.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-19T12:44:53.391 INFO:teuthology.orchestra.run.smithi057.stdout: inet 127.0.0.1/8 scope host lo 2023-10-19T12:44:53.392 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T12:44:53.392 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 ::1/128 scope host 2023-10-19T12:44:53.392 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T12:44:53.392 INFO:teuthology.orchestra.run.smithi057.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-19T12:44:53.392 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:6b:ff:86 brd ff:ff:ff:ff:ff:ff 2023-10-19T12:44:53.393 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp7s0f0 2023-10-19T12:44:53.393 INFO:teuthology.orchestra.run.smithi057.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-19T12:44:53.393 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:6b:ff:87 brd ff:ff:ff:ff:ff:ff 2023-10-19T12:44:53.393 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp7s0f1 2023-10-19T12:44:53.393 INFO:teuthology.orchestra.run.smithi057.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-19T12:44:53.394 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:bd:15:62 brd ff:ff:ff:ff:ff:ff 2023-10-19T12:44:53.394 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp1s0f0 2023-10-19T12:44:53.394 INFO:teuthology.orchestra.run.smithi057.stdout: inet 172.21.15.57/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-19T12:44:53.394 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft 42617sec preferred_lft 42617sec 2023-10-19T12:44:53.394 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 fe80::ec4:7aff:febd:1562/64 scope link noprefixroute 2023-10-19T12:44:53.395 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T12:44:53.395 INFO:teuthology.orchestra.run.smithi057.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-19T12:44:53.395 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:bd:15:63 brd ff:ff:ff:ff:ff:ff 2023-10-19T12:44:53.395 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp1s0f1 2023-10-19T12:44:53.396 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-10-19T12:44:53.396 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:44:53.396 DEBUG:teuthology.orchestra.run.smithi057:> (cd / && exec bash -c ' 2023-10-19T12:44:53.396 DEBUG:teuthology.orchestra.run.smithi057:> set -e 2023-10-19T12:44:53.397 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip link add name ceph-brx type bridge 2023-10-19T12:44:53.397 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip addr flush dev ceph-brx 2023-10-19T12:44:53.397 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip link set ceph-brx up 2023-10-19T12:44:53.398 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-10-19T12:44:53.398 DEBUG:teuthology.orchestra.run.smithi057:> ') 2023-10-19T12:44:53.512 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:44:53.513 DEBUG:teuthology.orchestra.run.smithi057:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-19T12:44:53.542 INFO:teuthology.orchestra.run.smithi057.stdout:1 2023-10-19T12:44:53.543 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:44:53.544 DEBUG:teuthology.orchestra.run.smithi057:> route 2023-10-19T12:44:53.599 INFO:teuthology.orchestra.run.smithi057.stdout:Kernel IP routing table 2023-10-19T12:44:53.600 INFO:teuthology.orchestra.run.smithi057.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-19T12:44:53.600 INFO:teuthology.orchestra.run.smithi057.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-19T12:44:53.600 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-19T12:44:53.600 INFO:teuthology.orchestra.run.smithi057.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-19T12:44:53.601 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:44:53.601 DEBUG:teuthology.orchestra.run.smithi057:> (cd / && exec bash -c ' 2023-10-19T12:44:53.601 DEBUG:teuthology.orchestra.run.smithi057:> set -e 2023-10-19T12:44:53.602 DEBUG:teuthology.orchestra.run.smithi057:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-19T12:44:53.602 DEBUG:teuthology.orchestra.run.smithi057:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-19T12:44:53.602 DEBUG:teuthology.orchestra.run.smithi057:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-19T12:44:53.603 DEBUG:teuthology.orchestra.run.smithi057:> ') 2023-10-19T12:44:53.706 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:44:53.706 DEBUG:teuthology.orchestra.run.smithi057:> ip netns list 2023-10-19T12:44:53.722 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:44:53.722 DEBUG:teuthology.orchestra.run.smithi057:> ip netns list-id 2023-10-19T12:44:53.777 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:44:53.777 DEBUG:teuthology.orchestra.run.smithi057:> (cd / && exec bash -c ' 2023-10-19T12:44:53.777 DEBUG:teuthology.orchestra.run.smithi057:> set -e 2023-10-19T12:44:53.778 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T12:44:53.778 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2023-10-19T12:44:53.778 DEBUG:teuthology.orchestra.run.smithi057:> ') 2023-10-19T12:44:53.867 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2023-10-19T12:44:53.867 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:44:53.868 DEBUG:teuthology.orchestra.run.smithi057:> (cd / && exec bash -c ' 2023-10-19T12:44:53.868 DEBUG:teuthology.orchestra.run.smithi057:> set -e 2023-10-19T12:44:53.868 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2023-10-19T12:44:53.868 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-10-19T12:44:53.869 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2023-10-19T12:44:53.869 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2023-10-19T12:44:53.869 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2023-10-19T12:44:53.870 DEBUG:teuthology.orchestra.run.smithi057:> ') 2023-10-19T12:44:53.972 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:44:53.972 DEBUG:teuthology.orchestra.run.smithi057:> (cd / && exec bash -c ' 2023-10-19T12:44:53.973 DEBUG:teuthology.orchestra.run.smithi057:> set -e 2023-10-19T12:44:53.973 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip link set brx.0 up 2023-10-19T12:44:53.973 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip link set dev brx.0 master ceph-brx 2023-10-19T12:44:53.974 DEBUG:teuthology.orchestra.run.smithi057:> ') 2023-10-19T12:44:54.025 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-10-19T12:44:54.025 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-19T12:44:54.026 DEBUG:teuthology.orchestra.run.smithi057:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-19T12:44:54.040 INFO:teuthology.orchestra.run.smithi057.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-19T12:44:54.041 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-19T12:44:54.041 DEBUG:teuthology.orchestra.run.smithi057:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-19T12:44:54.096 DEBUG:teuthology.orchestra.run.smithi057:> sudo modprobe fuse 2023-10-19T12:44:54.167 DEBUG:teuthology.orchestra.run.smithi057:> cat /proc/self/mounts | awk '{print $2}' 2023-10-19T12:44:54.224 INFO:teuthology.orchestra.run.smithi057.stdout:/proc 2023-10-19T12:44:54.224 INFO:teuthology.orchestra.run.smithi057.stdout:/sys 2023-10-19T12:44:54.224 INFO:teuthology.orchestra.run.smithi057.stdout:/dev 2023-10-19T12:44:54.224 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/security 2023-10-19T12:44:54.225 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/shm 2023-10-19T12:44:54.225 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/pts 2023-10-19T12:44:54.225 INFO:teuthology.orchestra.run.smithi057.stdout:/run 2023-10-19T12:44:54.225 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/cgroup 2023-10-19T12:44:54.225 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/pstore 2023-10-19T12:44:54.226 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/bpf 2023-10-19T12:44:54.226 INFO:teuthology.orchestra.run.smithi057.stdout:/ 2023-10-19T12:44:54.226 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/selinux 2023-10-19T12:44:54.226 INFO:teuthology.orchestra.run.smithi057.stdout:/proc/sys/fs/binfmt_misc 2023-10-19T12:44:54.226 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/hugepages 2023-10-19T12:44:54.227 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/mqueue 2023-10-19T12:44:54.227 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/debug 2023-10-19T12:44:54.227 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/tracing 2023-10-19T12:44:54.227 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/fuse/connections 2023-10-19T12:44:54.227 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/config 2023-10-19T12:44:54.228 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-sysctl.service 2023-10-19T12:44:54.228 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-10-19T12:44:54.228 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-10-19T12:44:54.228 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/nfs/rpc_pipefs 2023-10-19T12:44:54.228 INFO:teuthology.orchestra.run.smithi057.stdout:/run/user/1000 2023-10-19T12:44:54.229 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph 2023-10-19T12:44:54.229 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-0 2023-10-19T12:44:54.229 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-1 2023-10-19T12:44:54.229 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-2 2023-10-19T12:44:54.229 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-3 2023-10-19T12:44:54.229 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns 2023-10-19T12:44:54.230 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T12:44:54.230 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T12:44:54.230 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:44:54.231 DEBUG:teuthology.orchestra.run.smithi057:> ls /sys/fs/fuse/connections 2023-10-19T12:44:54.279 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-10-19T12:44:54.279 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-10-19T12:44:54.321 DEBUG:teuthology.orchestra.run.smithi057:> sudo modprobe fuse 2023-10-19T12:44:54.352 DEBUG:teuthology.orchestra.run.smithi057:> cat /proc/self/mounts | awk '{print $2}' 2023-10-19T12:44:54.408 INFO:teuthology.orchestra.run.smithi057.stdout:/proc 2023-10-19T12:44:54.408 INFO:teuthology.orchestra.run.smithi057.stdout:/sys 2023-10-19T12:44:54.408 INFO:teuthology.orchestra.run.smithi057.stdout:/dev 2023-10-19T12:44:54.409 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/security 2023-10-19T12:44:54.409 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/shm 2023-10-19T12:44:54.409 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/pts 2023-10-19T12:44:54.409 INFO:teuthology.orchestra.run.smithi057.stdout:/run 2023-10-19T12:44:54.409 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/cgroup 2023-10-19T12:44:54.410 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/pstore 2023-10-19T12:44:54.410 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/bpf 2023-10-19T12:44:54.410 INFO:teuthology.orchestra.run.smithi057.stdout:/ 2023-10-19T12:44:54.410 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/selinux 2023-10-19T12:44:54.410 INFO:teuthology.orchestra.run.smithi057.stdout:/proc/sys/fs/binfmt_misc 2023-10-19T12:44:54.411 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/hugepages 2023-10-19T12:44:54.411 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/mqueue 2023-10-19T12:44:54.411 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/debug 2023-10-19T12:44:54.411 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/tracing 2023-10-19T12:44:54.411 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/fuse/connections 2023-10-19T12:44:54.412 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/config 2023-10-19T12:44:54.412 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-sysctl.service 2023-10-19T12:44:54.412 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-10-19T12:44:54.412 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-10-19T12:44:54.412 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/nfs/rpc_pipefs 2023-10-19T12:44:54.412 INFO:teuthology.orchestra.run.smithi057.stdout:/run/user/1000 2023-10-19T12:44:54.413 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph 2023-10-19T12:44:54.413 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-0 2023-10-19T12:44:54.413 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-1 2023-10-19T12:44:54.413 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-2 2023-10-19T12:44:54.414 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-3 2023-10-19T12:44:54.414 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns 2023-10-19T12:44:54.414 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T12:44:54.414 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T12:44:54.415 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:44:54.415 DEBUG:teuthology.orchestra.run.smithi057:> ls /sys/fs/fuse/connections 2023-10-19T12:44:54.416 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi057.stderr:2023-10-19T12:44:54.411+0000 7fba60a44480 -1 init, newargv = 0x557d6baf7c40 newargc=15 2023-10-19T12:44:54.416 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi057.stderr:ceph-fuse[99099]: starting ceph client 2023-10-19T12:44:54.419 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi057.stderr:ceph-fuse[99099]: starting fuse 2023-10-19T12:44:54.431 INFO:teuthology.orchestra.run.smithi057.stdout:43 2023-10-19T12:44:54.431 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2023-10-19T12:44:54.432 DEBUG:teuthology.orchestra.run.smithi057:> sudo stdin-killer -- python3 -c ' 2023-10-19T12:44:54.432 DEBUG:teuthology.orchestra.run.smithi057:> import glob 2023-10-19T12:44:54.432 DEBUG:teuthology.orchestra.run.smithi057:> import re 2023-10-19T12:44:54.433 DEBUG:teuthology.orchestra.run.smithi057:> import os 2023-10-19T12:44:54.433 DEBUG:teuthology.orchestra.run.smithi057:> import subprocess 2023-10-19T12:44:54.433 DEBUG:teuthology.orchestra.run.smithi057:> 2023-10-19T12:44:54.434 DEBUG:teuthology.orchestra.run.smithi057:> def _find_admin_socket(client_name): 2023-10-19T12:44:54.434 DEBUG:teuthology.orchestra.run.smithi057:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-10-19T12:44:54.434 DEBUG:teuthology.orchestra.run.smithi057:> files = glob.glob(asok_path) 2023-10-19T12:44:54.434 DEBUG:teuthology.orchestra.run.smithi057:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-10-19T12:44:54.435 DEBUG:teuthology.orchestra.run.smithi057:> 2023-10-19T12:44:54.435 DEBUG:teuthology.orchestra.run.smithi057:> # Given a non-glob path, it better be there 2023-10-19T12:44:54.435 DEBUG:teuthology.orchestra.run.smithi057:> if "*" not in asok_path: 2023-10-19T12:44:54.436 DEBUG:teuthology.orchestra.run.smithi057:> assert(len(files) == 1) 2023-10-19T12:44:54.436 DEBUG:teuthology.orchestra.run.smithi057:> return files[0] 2023-10-19T12:44:54.436 DEBUG:teuthology.orchestra.run.smithi057:> 2023-10-19T12:44:54.436 DEBUG:teuthology.orchestra.run.smithi057:> for f in files: 2023-10-19T12:44:54.437 DEBUG:teuthology.orchestra.run.smithi057:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-10-19T12:44:54.437 DEBUG:teuthology.orchestra.run.smithi057:> if os.path.exists("/proc/{0}".format(pid)): 2023-10-19T12:44:54.437 DEBUG:teuthology.orchestra.run.smithi057:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-10-19T12:44:54.438 DEBUG:teuthology.orchestra.run.smithi057:> contents = proc_f.read() 2023-10-19T12:44:54.438 DEBUG:teuthology.orchestra.run.smithi057:> if mountpoint in contents: 2023-10-19T12:44:54.438 DEBUG:teuthology.orchestra.run.smithi057:> return f 2023-10-19T12:44:54.438 DEBUG:teuthology.orchestra.run.smithi057:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-10-19T12:44:54.439 DEBUG:teuthology.orchestra.run.smithi057:> 2023-10-19T12:44:54.439 DEBUG:teuthology.orchestra.run.smithi057:> print(_find_admin_socket("client.0")) 2023-10-19T12:44:54.439 DEBUG:teuthology.orchestra.run.smithi057:> ' 2023-10-19T12:44:54.555 INFO:teuthology.orchestra.run.smithi057.stdout:/var/run/ceph/ceph-client.0.99099.asok 2023-10-19T12:44:54.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2023-10-19T12:44:54.566 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.99099.asok 2023-10-19T12:44:54.566 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:44:54.566 DEBUG:teuthology.orchestra.run.smithi057:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.99099.asok status 2023-10-19T12:44:54.707 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2023-10-19T12:44:54.707 INFO:teuthology.orchestra.run.smithi057.stdout: "metadata": { 2023-10-19T12:44:54.707 INFO:teuthology.orchestra.run.smithi057.stdout: "ceph_sha1": "f7420d7f8e55ceea786dcbf536c338ce920c21eb", 2023-10-19T12:44:54.708 INFO:teuthology.orchestra.run.smithi057.stdout: "ceph_version": "ceph version 18.0.0-6786-gf7420d7f (f7420d7f8e55ceea786dcbf536c338ce920c21eb) reef (dev)", 2023-10-19T12:44:54.708 INFO:teuthology.orchestra.run.smithi057.stdout: "entity_id": "0", 2023-10-19T12:44:54.708 INFO:teuthology.orchestra.run.smithi057.stdout: "hostname": "smithi057", 2023-10-19T12:44:54.708 INFO:teuthology.orchestra.run.smithi057.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-10-19T12:44:54.708 INFO:teuthology.orchestra.run.smithi057.stdout: "pid": "99099", 2023-10-19T12:44:54.709 INFO:teuthology.orchestra.run.smithi057.stdout: "root": "/" 2023-10-19T12:44:54.709 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T12:44:54.709 INFO:teuthology.orchestra.run.smithi057.stdout: "dentry_count": 0, 2023-10-19T12:44:54.709 INFO:teuthology.orchestra.run.smithi057.stdout: "dentry_pinned_count": 0, 2023-10-19T12:44:54.709 INFO:teuthology.orchestra.run.smithi057.stdout: "id": 4983, 2023-10-19T12:44:54.709 INFO:teuthology.orchestra.run.smithi057.stdout: "inst": { 2023-10-19T12:44:54.710 INFO:teuthology.orchestra.run.smithi057.stdout: "name": { 2023-10-19T12:44:54.710 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "client", 2023-10-19T12:44:54.710 INFO:teuthology.orchestra.run.smithi057.stdout: "num": 4983 2023-10-19T12:44:54.710 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T12:44:54.710 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": { 2023-10-19T12:44:54.711 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "v1", 2023-10-19T12:44:54.711 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": "192.168.0.1:0", 2023-10-19T12:44:54.711 INFO:teuthology.orchestra.run.smithi057.stdout: "nonce": 129484081 2023-10-19T12:44:54.711 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T12:44:54.711 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T12:44:54.712 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": { 2023-10-19T12:44:54.712 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "v1", 2023-10-19T12:44:54.712 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": "192.168.0.1:0", 2023-10-19T12:44:54.712 INFO:teuthology.orchestra.run.smithi057.stdout: "nonce": 129484081 2023-10-19T12:44:54.712 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T12:44:54.712 INFO:teuthology.orchestra.run.smithi057.stdout: "inst_str": "client.4983 192.168.0.1:0/129484081", 2023-10-19T12:44:54.713 INFO:teuthology.orchestra.run.smithi057.stdout: "addr_str": "192.168.0.1:0/129484081", 2023-10-19T12:44:54.713 INFO:teuthology.orchestra.run.smithi057.stdout: "inode_count": 1, 2023-10-19T12:44:54.713 INFO:teuthology.orchestra.run.smithi057.stdout: "mds_epoch": 9, 2023-10-19T12:44:54.713 INFO:teuthology.orchestra.run.smithi057.stdout: "osd_epoch": 25, 2023-10-19T12:44:54.713 INFO:teuthology.orchestra.run.smithi057.stdout: "osd_epoch_barrier": 0, 2023-10-19T12:44:54.714 INFO:teuthology.orchestra.run.smithi057.stdout: "blocklisted": false, 2023-10-19T12:44:54.714 INFO:teuthology.orchestra.run.smithi057.stdout: "fs_name": "cephfs" 2023-10-19T12:44:54.714 INFO:teuthology.orchestra.run.smithi057.stdout:} 2023-10-19T12:44:54.716 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-19T12:44:54.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:54.822+0000 7f6c25f6c640 1 -- 172.21.15.57:0/1996030736 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6c2006bec0 msgr2=0x7f6c2006a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:54.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:54.822+0000 7f6c25f6c640 1 --2- 172.21.15.57:0/1996030736 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6c2006bec0 0x7f6c2006a730 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f6c18009be0 tx=0x7f6c1802c0c0 comp rx=0 tx=0).stop 2023-10-19T12:44:54.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:54.822+0000 7f6c25f6c640 1 -- 172.21.15.57:0/1996030736 shutdown_connections 2023-10-19T12:44:54.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:54.822+0000 7f6c25f6c640 1 --2- 172.21.15.57:0/1996030736 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6c2006bec0 0x7f6c2006a730 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:54.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:54.822+0000 7f6c25f6c640 1 --2- 172.21.15.57:0/1996030736 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6c200ac870 0x7f6c2006b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:54.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:54.822+0000 7f6c25f6c640 1 -- 172.21.15.57:0/1996030736 >> 172.21.15.57:0/1996030736 conn(0x7f6c2006fae0 msgr2=0x7f6c2006fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:54.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:54.822+0000 7f6c25f6c640 1 -- 172.21.15.57:0/1996030736 shutdown_connections 2023-10-19T12:44:54.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:54.822+0000 7f6c25f6c640 1 -- 172.21.15.57:0/1996030736 wait complete. 2023-10-19T12:44:54.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:54.823+0000 7f6c25f6c640 1 Processor -- start 2023-10-19T12:44:54.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:54.824+0000 7f6c25f6c640 1 -- start start 2023-10-19T12:44:54.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:54.824+0000 7f6c25f6c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6c2006bec0 0x7f6c200b0580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:54.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:54.824+0000 7f6c25f6c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6c200ac870 0x7f6c200aa8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:54.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:54.824+0000 7f6c25f6c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6c20156790 0x7f6c200aae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:54.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:54.824+0000 7f6c25f6c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c200b0b60 con 0x7f6c2006bec0 2023-10-19T12:44:54.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:54.824+0000 7f6c25f6c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c200b0e40 con 0x7f6c200ac870 2023-10-19T12:44:54.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:54.824+0000 7f6c25f6c640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c200b1120 con 0x7f6c20156790 2023-10-19T12:44:54.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:54.824+0000 7f6c1f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6c2006bec0 0x7f6c200b0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:54.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:54.824+0000 7f6c1f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6c2006bec0 0x7f6c200b0580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59494/0 (socket says 172.21.15.57:59494) 2023-10-19T12:44:54.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:54.824+0000 7f6c1f7fe640 1 -- 172.21.15.57:0/3978937979 learned_addr learned my addr 172.21.15.57:0/3978937979 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:54.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:54.825+0000 7f6c1effd640 1 --2- 172.21.15.57:0/3978937979 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6c200ac870 0x7f6c200aa8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:54.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:54.825+0000 7f6c1ffff640 1 --2- 172.21.15.57:0/3978937979 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6c20156790 0x7f6c200aae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:54.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:54.825+0000 7f6c1f7fe640 1 -- 172.21.15.57:0/3978937979 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6c20156790 msgr2=0x7f6c200aae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:54.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:54.825+0000 7f6c1f7fe640 1 --2- 172.21.15.57:0/3978937979 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6c20156790 0x7f6c200aae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:54.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:54.825+0000 7f6c1f7fe640 1 -- 172.21.15.57:0/3978937979 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6c200ac870 msgr2=0x7f6c200aa8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:54.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:54.825+0000 7f6c1f7fe640 1 --2- 172.21.15.57:0/3978937979 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6c200ac870 0x7f6c200aa8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:54.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:54.825+0000 7f6c1f7fe640 1 -- 172.21.15.57:0/3978937979 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c18009840 con 0x7f6c2006bec0 2023-10-19T12:44:54.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:54.825+0000 7f6c1f7fe640 1 --2- 172.21.15.57:0/3978937979 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6c2006bec0 0x7f6c200b0580 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f6c1000c430 tx=0x7f6c1000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:54.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:54.825+0000 7f6c07fff640 1 -- 172.21.15.57:0/3978937979 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c10004230 con 0x7f6c2006bec0 2023-10-19T12:44:54.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:54.825+0000 7f6c07fff640 1 -- 172.21.15.57:0/3978937979 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f6c100044a0 con 0x7f6c2006bec0 2023-10-19T12:44:54.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:54.825+0000 7f6c07fff640 1 -- 172.21.15.57:0/3978937979 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c10003850 con 0x7f6c2006bec0 2023-10-19T12:44:54.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:54.825+0000 7f6c25f6c640 1 -- 172.21.15.57:0/3978937979 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c200b1400 con 0x7f6c2006bec0 2023-10-19T12:44:54.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:54.825+0000 7f6c25f6c640 1 -- 172.21.15.57:0/3978937979 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c200c9b00 con 0x7f6c2006bec0 2023-10-19T12:44:54.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:54.826+0000 7f6c25f6c640 1 -- 172.21.15.57:0/3978937979 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c2006d9c0 con 0x7f6c2006bec0 2023-10-19T12:44:54.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:54.827+0000 7f6c07fff640 1 -- 172.21.15.57:0/3978937979 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f6c10003a30 con 0x7f6c2006bec0 2023-10-19T12:44:54.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:54.827+0000 7f6c07fff640 1 --2- 172.21.15.57:0/3978937979 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f6c0c06ae40 0x7f6c0c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:54.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:54.827+0000 7f6c07fff640 1 -- 172.21.15.57:0/3978937979 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f6c100890d0 con 0x7f6c2006bec0 2023-10-19T12:44:54.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:54.827+0000 7f6c1effd640 1 --2- 172.21.15.57:0/3978937979 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f6c0c06ae40 0x7f6c0c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:54.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:54.828+0000 7f6c1effd640 1 --2- 172.21.15.57:0/3978937979 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f6c0c06ae40 0x7f6c0c06d300 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f6c200ad010 tx=0x7f6c14005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:54.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:54.831+0000 7f6c07fff640 1 -- 172.21.15.57:0/3978937979 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6c10054000 con 0x7f6c2006bec0 2023-10-19T12:44:55.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:55.044+0000 7f6c25f6c640 1 -- 172.21.15.57:0/3978937979 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f6c200690e0 con 0x7f6c2006bec0 2023-10-19T12:44:55.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:55.044+0000 7f6c07fff640 1 -- 172.21.15.57:0/3978937979 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) v1 ==== 53+0+88 (secure 0 0 0) 0x7f6c10057620 con 0x7f6c2006bec0 2023-10-19T12:44:55.048 INFO:teuthology.orchestra.run.smithi057.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-10-19T12:44:55.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:55.046+0000 7f6c25f6c640 1 -- 172.21.15.57:0/3978937979 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f6c0c06ae40 msgr2=0x7f6c0c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:55.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:55.046+0000 7f6c25f6c640 1 --2- 172.21.15.57:0/3978937979 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f6c0c06ae40 0x7f6c0c06d300 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f6c200ad010 tx=0x7f6c14005f90 comp rx=0 tx=0).stop 2023-10-19T12:44:55.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:55.046+0000 7f6c25f6c640 1 -- 172.21.15.57:0/3978937979 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6c2006bec0 msgr2=0x7f6c200b0580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:55.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:55.046+0000 7f6c25f6c640 1 --2- 172.21.15.57:0/3978937979 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6c2006bec0 0x7f6c200b0580 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f6c1000c430 tx=0x7f6c1000c8f0 comp rx=0 tx=0).stop 2023-10-19T12:44:55.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:55.046+0000 7f6c25f6c640 1 -- 172.21.15.57:0/3978937979 shutdown_connections 2023-10-19T12:44:55.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:55.046+0000 7f6c25f6c640 1 --2- 172.21.15.57:0/3978937979 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f6c0c06ae40 0x7f6c0c06d300 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:55.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:55.046+0000 7f6c25f6c640 1 --2- 172.21.15.57:0/3978937979 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6c20156790 0x7f6c200aae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:55.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:55.046+0000 7f6c25f6c640 1 --2- 172.21.15.57:0/3978937979 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6c200ac870 0x7f6c200aa8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:55.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:55.046+0000 7f6c25f6c640 1 --2- 172.21.15.57:0/3978937979 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6c2006bec0 0x7f6c200b0580 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:55.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:55.046+0000 7f6c25f6c640 1 -- 172.21.15.57:0/3978937979 >> 172.21.15.57:0/3978937979 conn(0x7f6c2006fae0 msgr2=0x7f6c201574f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:55.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:55.047+0000 7f6c25f6c640 1 -- 172.21.15.57:0/3978937979 shutdown_connections 2023-10-19T12:44:55.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:55.047+0000 7f6c25f6c640 1 -- 172.21.15.57:0/3978937979 wait complete. 2023-10-19T12:44:55.060 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-10-19T12:44:55.060 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-19T12:44:55.060 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi057.front.sepia.ceph.com 2023-10-19T12:44:55.060 INFO:tasks.cephfs.mount:self.client.name = client.2 2023-10-19T12:44:55.061 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2023-10-19T12:44:55.061 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-10-19T12:44:55.061 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-10-19T12:44:55.061 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-19T12:44:55.062 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2023-10-19T12:44:55.062 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:44:55.062 DEBUG:teuthology.orchestra.run.smithi057:> ip addr 2023-10-19T12:44:55.078 INFO:teuthology.orchestra.run.smithi057.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-19T12:44:55.078 INFO:teuthology.orchestra.run.smithi057.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-19T12:44:55.079 INFO:teuthology.orchestra.run.smithi057.stdout: inet 127.0.0.1/8 scope host lo 2023-10-19T12:44:55.079 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T12:44:55.079 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 ::1/128 scope host 2023-10-19T12:44:55.079 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T12:44:55.080 INFO:teuthology.orchestra.run.smithi057.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-19T12:44:55.080 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:6b:ff:86 brd ff:ff:ff:ff:ff:ff 2023-10-19T12:44:55.080 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp7s0f0 2023-10-19T12:44:55.080 INFO:teuthology.orchestra.run.smithi057.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-19T12:44:55.080 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:6b:ff:87 brd ff:ff:ff:ff:ff:ff 2023-10-19T12:44:55.081 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp7s0f1 2023-10-19T12:44:55.081 INFO:teuthology.orchestra.run.smithi057.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-19T12:44:55.081 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:bd:15:62 brd ff:ff:ff:ff:ff:ff 2023-10-19T12:44:55.081 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp1s0f0 2023-10-19T12:44:55.081 INFO:teuthology.orchestra.run.smithi057.stdout: inet 172.21.15.57/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-19T12:44:55.082 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft 42615sec preferred_lft 42615sec 2023-10-19T12:44:55.082 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 fe80::ec4:7aff:febd:1562/64 scope link noprefixroute 2023-10-19T12:44:55.082 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T12:44:55.082 INFO:teuthology.orchestra.run.smithi057.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-19T12:44:55.082 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:bd:15:63 brd ff:ff:ff:ff:ff:ff 2023-10-19T12:44:55.083 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp1s0f1 2023-10-19T12:44:55.083 INFO:teuthology.orchestra.run.smithi057.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-19T12:44:55.083 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0e:fe:f1:b5:41:59 brd ff:ff:ff:ff:ff:ff 2023-10-19T12:44:55.083 INFO:teuthology.orchestra.run.smithi057.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-19T12:44:55.083 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T12:44:55.083 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 fe80::64b1:a2ff:feb3:b08/64 scope link 2023-10-19T12:44:55.084 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T12:44:55.084 INFO:teuthology.orchestra.run.smithi057.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-19T12:44:55.084 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0e:fe:f1:b5:41:59 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T12:44:55.084 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 fe80::cfe:f1ff:feb5:4159/64 scope link tentative 2023-10-19T12:44:55.084 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T12:44:55.085 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:44:55.085 DEBUG:teuthology.orchestra.run.smithi057:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-19T12:44:55.151 INFO:teuthology.orchestra.run.smithi057.stdout:1 2023-10-19T12:44:55.151 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:44:55.152 DEBUG:teuthology.orchestra.run.smithi057:> route 2023-10-19T12:44:55.207 INFO:teuthology.orchestra.run.smithi057.stdout:Kernel IP routing table 2023-10-19T12:44:55.207 INFO:teuthology.orchestra.run.smithi057.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-19T12:44:55.207 INFO:teuthology.orchestra.run.smithi057.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-19T12:44:55.207 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-19T12:44:55.208 INFO:teuthology.orchestra.run.smithi057.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-19T12:44:55.208 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:44:55.208 DEBUG:teuthology.orchestra.run.smithi057:> (cd / && exec bash -c ' 2023-10-19T12:44:55.209 DEBUG:teuthology.orchestra.run.smithi057:> set -e 2023-10-19T12:44:55.209 DEBUG:teuthology.orchestra.run.smithi057:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-19T12:44:55.209 DEBUG:teuthology.orchestra.run.smithi057:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-19T12:44:55.209 DEBUG:teuthology.orchestra.run.smithi057:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-19T12:44:55.210 DEBUG:teuthology.orchestra.run.smithi057:> ') 2023-10-19T12:44:55.313 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:44:55.313 DEBUG:teuthology.orchestra.run.smithi057:> ip netns list 2023-10-19T12:44:55.328 INFO:teuthology.orchestra.run.smithi057.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-19T12:44:55.329 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:44:55.329 DEBUG:teuthology.orchestra.run.smithi057:> ip netns list-id 2023-10-19T12:44:55.383 INFO:teuthology.orchestra.run.smithi057.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2023-10-19T12:44:55.384 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:44:55.384 DEBUG:teuthology.orchestra.run.smithi057:> (cd / && exec bash -c ' 2023-10-19T12:44:55.384 DEBUG:teuthology.orchestra.run.smithi057:> set -e 2023-10-19T12:44:55.385 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-19T12:44:55.385 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2023-10-19T12:44:55.385 DEBUG:teuthology.orchestra.run.smithi057:> ') 2023-10-19T12:44:55.474 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:44:55.474 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2023-10-19T12:44:55.504 INFO:teuthology.orchestra.run.smithi057.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-19T12:44:55.504 INFO:teuthology.orchestra.run.smithi057.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-19T12:44:55.505 INFO:teuthology.orchestra.run.smithi057.stdout: inet 127.0.0.1/8 scope host lo 2023-10-19T12:44:55.505 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T12:44:55.505 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 ::1/128 scope host 2023-10-19T12:44:55.505 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T12:44:55.505 INFO:teuthology.orchestra.run.smithi057.stdout:2: veth0@if9: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-19T12:44:55.506 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether b6:a7:ce:46:78:b5 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-10-19T12:44:55.506 INFO:teuthology.orchestra.run.smithi057.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-10-19T12:44:55.506 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T12:44:55.506 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 fe80::b4a7:ceff:fe46:78b5/64 scope link tentative 2023-10-19T12:44:55.506 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T12:44:55.507 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:44:55.508 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2023-10-19T12:44:55.578 INFO:teuthology.orchestra.run.smithi057.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-19T12:44:55.578 INFO:teuthology.orchestra.run.smithi057.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-19T12:44:55.578 INFO:teuthology.orchestra.run.smithi057.stdout: inet 127.0.0.1/8 scope host lo 2023-10-19T12:44:55.579 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T12:44:55.579 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 ::1/128 scope host 2023-10-19T12:44:55.579 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T12:44:55.579 INFO:teuthology.orchestra.run.smithi057.stdout:2: veth0@if9: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-19T12:44:55.579 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether b6:a7:ce:46:78:b5 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-10-19T12:44:55.580 INFO:teuthology.orchestra.run.smithi057.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-10-19T12:44:55.580 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T12:44:55.580 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 fe80::b4a7:ceff:fe46:78b5/64 scope link 2023-10-19T12:44:55.581 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T12:44:55.582 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2023-10-19T12:44:55.583 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:44:55.583 DEBUG:teuthology.orchestra.run.smithi057:> (cd / && exec bash -c ' 2023-10-19T12:44:55.584 DEBUG:teuthology.orchestra.run.smithi057:> set -e 2023-10-19T12:44:55.585 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2023-10-19T12:44:55.585 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2023-10-19T12:44:55.586 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2023-10-19T12:44:55.586 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2023-10-19T12:44:55.587 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2023-10-19T12:44:55.587 DEBUG:teuthology.orchestra.run.smithi057:> ') 2023-10-19T12:44:55.724 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:44:55.725 DEBUG:teuthology.orchestra.run.smithi057:> (cd / && exec bash -c ' 2023-10-19T12:44:55.725 DEBUG:teuthology.orchestra.run.smithi057:> set -e 2023-10-19T12:44:55.725 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip link set brx.1 up 2023-10-19T12:44:55.726 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip link set dev brx.1 master ceph-brx 2023-10-19T12:44:55.726 DEBUG:teuthology.orchestra.run.smithi057:> ') 2023-10-19T12:44:55.774 INFO:tasks.cephfs.fuse_mount:Client client.2 config is {} 2023-10-19T12:44:55.775 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-19T12:44:55.775 DEBUG:teuthology.orchestra.run.smithi057:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2023-10-19T12:44:55.790 INFO:teuthology.orchestra.run.smithi057.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2023-10-19T12:44:55.790 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-19T12:44:55.790 DEBUG:teuthology.orchestra.run.smithi057:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2023-10-19T12:44:55.845 DEBUG:teuthology.orchestra.run.smithi057:> sudo modprobe fuse 2023-10-19T12:44:55.915 DEBUG:teuthology.orchestra.run.smithi057:> cat /proc/self/mounts | awk '{print $2}' 2023-10-19T12:44:55.972 INFO:teuthology.orchestra.run.smithi057.stdout:/proc 2023-10-19T12:44:55.972 INFO:teuthology.orchestra.run.smithi057.stdout:/sys 2023-10-19T12:44:55.972 INFO:teuthology.orchestra.run.smithi057.stdout:/dev 2023-10-19T12:44:55.972 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/security 2023-10-19T12:44:55.973 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/shm 2023-10-19T12:44:55.973 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/pts 2023-10-19T12:44:55.973 INFO:teuthology.orchestra.run.smithi057.stdout:/run 2023-10-19T12:44:55.973 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/cgroup 2023-10-19T12:44:55.973 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/pstore 2023-10-19T12:44:55.973 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/bpf 2023-10-19T12:44:55.974 INFO:teuthology.orchestra.run.smithi057.stdout:/ 2023-10-19T12:44:55.974 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/selinux 2023-10-19T12:44:55.974 INFO:teuthology.orchestra.run.smithi057.stdout:/proc/sys/fs/binfmt_misc 2023-10-19T12:44:55.974 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/hugepages 2023-10-19T12:44:55.974 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/mqueue 2023-10-19T12:44:55.975 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/debug 2023-10-19T12:44:55.975 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/tracing 2023-10-19T12:44:55.975 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/fuse/connections 2023-10-19T12:44:55.975 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/config 2023-10-19T12:44:55.975 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-sysctl.service 2023-10-19T12:44:55.976 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-10-19T12:44:55.976 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-10-19T12:44:55.976 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/nfs/rpc_pipefs 2023-10-19T12:44:55.976 INFO:teuthology.orchestra.run.smithi057.stdout:/run/user/1000 2023-10-19T12:44:55.976 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph 2023-10-19T12:44:55.976 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-0 2023-10-19T12:44:55.977 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-1 2023-10-19T12:44:55.977 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-2 2023-10-19T12:44:55.977 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-3 2023-10-19T12:44:55.977 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns 2023-10-19T12:44:55.977 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T12:44:55.977 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T12:44:55.978 INFO:teuthology.orchestra.run.smithi057.stdout:/home/ubuntu/cephtest/mnt.0 2023-10-19T12:44:55.978 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-19T12:44:55.978 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-19T12:44:55.978 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:44:55.979 DEBUG:teuthology.orchestra.run.smithi057:> ls /sys/fs/fuse/connections 2023-10-19T12:44:56.027 INFO:teuthology.orchestra.run.smithi057.stdout:43 2023-10-19T12:44:56.027 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [43] 2023-10-19T12:44:56.028 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.2 --id 2) 2023-10-19T12:44:56.071 DEBUG:teuthology.orchestra.run.smithi057:> sudo modprobe fuse 2023-10-19T12:44:56.101 DEBUG:teuthology.orchestra.run.smithi057:> cat /proc/self/mounts | awk '{print $2}' 2023-10-19T12:44:56.157 INFO:teuthology.orchestra.run.smithi057.stdout:/proc 2023-10-19T12:44:56.157 INFO:teuthology.orchestra.run.smithi057.stdout:/sys 2023-10-19T12:44:56.157 INFO:teuthology.orchestra.run.smithi057.stdout:/dev 2023-10-19T12:44:56.157 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/security 2023-10-19T12:44:56.157 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/shm 2023-10-19T12:44:56.158 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/pts 2023-10-19T12:44:56.158 INFO:teuthology.orchestra.run.smithi057.stdout:/run 2023-10-19T12:44:56.158 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/cgroup 2023-10-19T12:44:56.158 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/pstore 2023-10-19T12:44:56.158 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/bpf 2023-10-19T12:44:56.159 INFO:teuthology.orchestra.run.smithi057.stdout:/ 2023-10-19T12:44:56.159 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/selinux 2023-10-19T12:44:56.159 INFO:teuthology.orchestra.run.smithi057.stdout:/proc/sys/fs/binfmt_misc 2023-10-19T12:44:56.159 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/hugepages 2023-10-19T12:44:56.159 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/mqueue 2023-10-19T12:44:56.160 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/debug 2023-10-19T12:44:56.160 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/tracing 2023-10-19T12:44:56.160 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/fuse/connections 2023-10-19T12:44:56.160 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/config 2023-10-19T12:44:56.161 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-sysctl.service 2023-10-19T12:44:56.161 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-10-19T12:44:56.161 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-10-19T12:44:56.161 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/nfs/rpc_pipefs 2023-10-19T12:44:56.161 INFO:teuthology.orchestra.run.smithi057.stdout:/run/user/1000 2023-10-19T12:44:56.161 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph 2023-10-19T12:44:56.162 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-0 2023-10-19T12:44:56.162 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-1 2023-10-19T12:44:56.162 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-2 2023-10-19T12:44:56.162 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-3 2023-10-19T12:44:56.162 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns 2023-10-19T12:44:56.163 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T12:44:56.163 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T12:44:56.163 INFO:teuthology.orchestra.run.smithi057.stdout:/home/ubuntu/cephtest/mnt.0 2023-10-19T12:44:56.163 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-19T12:44:56.163 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-19T12:44:56.164 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:44:56.164 DEBUG:teuthology.orchestra.run.smithi057:> ls /sys/fs/fuse/connections 2023-10-19T12:44:56.165 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi057.stderr:2023-10-19T12:44:56.159+0000 7f9b51640480 -1 init, newargv = 0x55fed7996c20 newargc=15 2023-10-19T12:44:56.165 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi057.stderr:ceph-fuse[99593]: starting ceph client 2023-10-19T12:44:56.167 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi057.stderr:ceph-fuse[99593]: starting fuse 2023-10-19T12:44:56.180 INFO:teuthology.orchestra.run.smithi057.stdout:43 2023-10-19T12:44:56.180 INFO:teuthology.orchestra.run.smithi057.stdout:44 2023-10-19T12:44:56.181 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43, 44] 2023-10-19T12:44:56.181 DEBUG:teuthology.orchestra.run.smithi057:> sudo stdin-killer -- python3 -c ' 2023-10-19T12:44:56.181 DEBUG:teuthology.orchestra.run.smithi057:> import glob 2023-10-19T12:44:56.182 DEBUG:teuthology.orchestra.run.smithi057:> import re 2023-10-19T12:44:56.182 DEBUG:teuthology.orchestra.run.smithi057:> import os 2023-10-19T12:44:56.182 DEBUG:teuthology.orchestra.run.smithi057:> import subprocess 2023-10-19T12:44:56.182 DEBUG:teuthology.orchestra.run.smithi057:> 2023-10-19T12:44:56.183 DEBUG:teuthology.orchestra.run.smithi057:> def _find_admin_socket(client_name): 2023-10-19T12:44:56.183 DEBUG:teuthology.orchestra.run.smithi057:> asok_path = "/var/run/ceph/ceph-client.2.*.asok" 2023-10-19T12:44:56.183 DEBUG:teuthology.orchestra.run.smithi057:> files = glob.glob(asok_path) 2023-10-19T12:44:56.184 DEBUG:teuthology.orchestra.run.smithi057:> mountpoint = "/home/ubuntu/cephtest/mnt.2" 2023-10-19T12:44:56.184 DEBUG:teuthology.orchestra.run.smithi057:> 2023-10-19T12:44:56.184 DEBUG:teuthology.orchestra.run.smithi057:> # Given a non-glob path, it better be there 2023-10-19T12:44:56.184 DEBUG:teuthology.orchestra.run.smithi057:> if "*" not in asok_path: 2023-10-19T12:44:56.185 DEBUG:teuthology.orchestra.run.smithi057:> assert(len(files) == 1) 2023-10-19T12:44:56.185 DEBUG:teuthology.orchestra.run.smithi057:> return files[0] 2023-10-19T12:44:56.185 DEBUG:teuthology.orchestra.run.smithi057:> 2023-10-19T12:44:56.185 DEBUG:teuthology.orchestra.run.smithi057:> for f in files: 2023-10-19T12:44:56.186 DEBUG:teuthology.orchestra.run.smithi057:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-10-19T12:44:56.186 DEBUG:teuthology.orchestra.run.smithi057:> if os.path.exists("/proc/{0}".format(pid)): 2023-10-19T12:44:56.186 DEBUG:teuthology.orchestra.run.smithi057:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-10-19T12:44:56.186 DEBUG:teuthology.orchestra.run.smithi057:> contents = proc_f.read() 2023-10-19T12:44:56.187 DEBUG:teuthology.orchestra.run.smithi057:> if mountpoint in contents: 2023-10-19T12:44:56.187 DEBUG:teuthology.orchestra.run.smithi057:> return f 2023-10-19T12:44:56.187 DEBUG:teuthology.orchestra.run.smithi057:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-10-19T12:44:56.188 DEBUG:teuthology.orchestra.run.smithi057:> 2023-10-19T12:44:56.188 DEBUG:teuthology.orchestra.run.smithi057:> print(_find_admin_socket("client.2")) 2023-10-19T12:44:56.188 DEBUG:teuthology.orchestra.run.smithi057:> ' 2023-10-19T12:44:56.305 INFO:teuthology.orchestra.run.smithi057.stdout:/var/run/ceph/ceph-client.2.99593.asok 2023-10-19T12:44:56.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2023-10-19T12:44:56.316 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.2.99593.asok 2023-10-19T12:44:56.316 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:44:56.316 DEBUG:teuthology.orchestra.run.smithi057:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.2.99593.asok status 2023-10-19T12:44:56.458 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2023-10-19T12:44:56.458 INFO:teuthology.orchestra.run.smithi057.stdout: "metadata": { 2023-10-19T12:44:56.458 INFO:teuthology.orchestra.run.smithi057.stdout: "ceph_sha1": "f7420d7f8e55ceea786dcbf536c338ce920c21eb", 2023-10-19T12:44:56.459 INFO:teuthology.orchestra.run.smithi057.stdout: "ceph_version": "ceph version 18.0.0-6786-gf7420d7f (f7420d7f8e55ceea786dcbf536c338ce920c21eb) reef (dev)", 2023-10-19T12:44:56.459 INFO:teuthology.orchestra.run.smithi057.stdout: "entity_id": "2", 2023-10-19T12:44:56.459 INFO:teuthology.orchestra.run.smithi057.stdout: "hostname": "smithi057", 2023-10-19T12:44:56.459 INFO:teuthology.orchestra.run.smithi057.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.2", 2023-10-19T12:44:56.459 INFO:teuthology.orchestra.run.smithi057.stdout: "pid": "99593", 2023-10-19T12:44:56.460 INFO:teuthology.orchestra.run.smithi057.stdout: "root": "/" 2023-10-19T12:44:56.460 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T12:44:56.460 INFO:teuthology.orchestra.run.smithi057.stdout: "dentry_count": 0, 2023-10-19T12:44:56.460 INFO:teuthology.orchestra.run.smithi057.stdout: "dentry_pinned_count": 0, 2023-10-19T12:44:56.460 INFO:teuthology.orchestra.run.smithi057.stdout: "id": 4995, 2023-10-19T12:44:56.461 INFO:teuthology.orchestra.run.smithi057.stdout: "inst": { 2023-10-19T12:44:56.461 INFO:teuthology.orchestra.run.smithi057.stdout: "name": { 2023-10-19T12:44:56.461 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "client", 2023-10-19T12:44:56.461 INFO:teuthology.orchestra.run.smithi057.stdout: "num": 4995 2023-10-19T12:44:56.461 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T12:44:56.462 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": { 2023-10-19T12:44:56.462 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "v1", 2023-10-19T12:44:56.462 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": "192.168.0.2:0", 2023-10-19T12:44:56.462 INFO:teuthology.orchestra.run.smithi057.stdout: "nonce": 126653162 2023-10-19T12:44:56.462 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T12:44:56.462 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T12:44:56.463 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": { 2023-10-19T12:44:56.463 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "v1", 2023-10-19T12:44:56.463 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": "192.168.0.2:0", 2023-10-19T12:44:56.463 INFO:teuthology.orchestra.run.smithi057.stdout: "nonce": 126653162 2023-10-19T12:44:56.463 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T12:44:56.464 INFO:teuthology.orchestra.run.smithi057.stdout: "inst_str": "client.4995 192.168.0.2:0/126653162", 2023-10-19T12:44:56.464 INFO:teuthology.orchestra.run.smithi057.stdout: "addr_str": "192.168.0.2:0/126653162", 2023-10-19T12:44:56.464 INFO:teuthology.orchestra.run.smithi057.stdout: "inode_count": 1, 2023-10-19T12:44:56.464 INFO:teuthology.orchestra.run.smithi057.stdout: "mds_epoch": 9, 2023-10-19T12:44:56.464 INFO:teuthology.orchestra.run.smithi057.stdout: "osd_epoch": 25, 2023-10-19T12:44:56.464 INFO:teuthology.orchestra.run.smithi057.stdout: "osd_epoch_barrier": 0, 2023-10-19T12:44:56.465 INFO:teuthology.orchestra.run.smithi057.stdout: "blocklisted": false, 2023-10-19T12:44:56.465 INFO:teuthology.orchestra.run.smithi057.stdout: "fs_name": "cephfs" 2023-10-19T12:44:56.465 INFO:teuthology.orchestra.run.smithi057.stdout:} 2023-10-19T12:44:56.466 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-19T12:44:56.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:56.572+0000 7f1945e9b640 1 -- 172.21.15.57:0/1781656584 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f194013e520 msgr2=0x7f194006bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:56.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:56.572+0000 7f1945e9b640 1 --2- 172.21.15.57:0/1781656584 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f194013e520 0x7f194006bb60 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f1930005fd0 tx=0x7f193002bcf0 comp rx=0 tx=0).stop 2023-10-19T12:44:56.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:56.572+0000 7f1945e9b640 1 -- 172.21.15.57:0/1781656584 shutdown_connections 2023-10-19T12:44:56.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:56.572+0000 7f1945e9b640 1 --2- 172.21.15.57:0/1781656584 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1940070af0 0x7f19400affe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:56.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:56.572+0000 7f1945e9b640 1 --2- 172.21.15.57:0/1781656584 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f194013e520 0x7f194006bb60 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:56.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:56.572+0000 7f1945e9b640 1 -- 172.21.15.57:0/1781656584 >> 172.21.15.57:0/1781656584 conn(0x7f1940069930 msgr2=0x7f1940069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:56.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:56.572+0000 7f1945e9b640 1 -- 172.21.15.57:0/1781656584 shutdown_connections 2023-10-19T12:44:56.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:56.572+0000 7f1945e9b640 1 -- 172.21.15.57:0/1781656584 wait complete. 2023-10-19T12:44:56.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:56.573+0000 7f1945e9b640 1 Processor -- start 2023-10-19T12:44:56.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:56.573+0000 7f1945e9b640 1 -- start start 2023-10-19T12:44:56.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:56.574+0000 7f1945e9b640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1940070af0 0x7f194013a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:56.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:56.574+0000 7f1945e9b640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f194013e520 0x7f19400add80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:56.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:56.574+0000 7f1945e9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1940141690 0x7f19400ca870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:56.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:56.574+0000 7f1945e9b640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f194013c0a0 con 0x7f1940141690 2023-10-19T12:44:56.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:56.574+0000 7f1945e9b640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f194013c660 con 0x7f1940070af0 2023-10-19T12:44:56.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:56.574+0000 7f1945e9b640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19400abd40 con 0x7f194013e520 2023-10-19T12:44:56.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:56.574+0000 7f193ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1940141690 0x7f19400ca870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:56.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:56.574+0000 7f193ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1940141690 0x7f19400ca870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59506/0 (socket says 172.21.15.57:59506) 2023-10-19T12:44:56.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:56.574+0000 7f193ffff640 1 -- 172.21.15.57:0/1418606980 learned_addr learned my addr 172.21.15.57:0/1418606980 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:56.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:56.574+0000 7f193f7fe640 1 --2- 172.21.15.57:0/1418606980 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1940070af0 0x7f194013a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:56.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:56.574+0000 7f193effd640 1 --2- 172.21.15.57:0/1418606980 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f194013e520 0x7f19400add80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:56.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:56.574+0000 7f193ffff640 1 -- 172.21.15.57:0/1418606980 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f194013e520 msgr2=0x7f19400add80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:56.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:56.574+0000 7f193ffff640 1 --2- 172.21.15.57:0/1418606980 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f194013e520 0x7f19400add80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:56.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:56.574+0000 7f193ffff640 1 -- 172.21.15.57:0/1418606980 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1940070af0 msgr2=0x7f194013a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:56.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:56.574+0000 7f193ffff640 1 --2- 172.21.15.57:0/1418606980 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1940070af0 0x7f194013a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:56.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:56.574+0000 7f193ffff640 1 -- 172.21.15.57:0/1418606980 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1930009560 con 0x7f1940141690 2023-10-19T12:44:56.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:56.574+0000 7f193f7fe640 1 --2- 172.21.15.57:0/1418606980 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1940070af0 0x7f194013a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:44:56.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:56.574+0000 7f193ffff640 1 --2- 172.21.15.57:0/1418606980 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1940141690 0x7f19400ca870 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f193800cb00 tx=0x7f193800a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:56.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:56.575+0000 7f1927fff640 1 -- 172.21.15.57:0/1418606980 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f193800ae90 con 0x7f1940141690 2023-10-19T12:44:56.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:56.575+0000 7f1927fff640 1 -- 172.21.15.57:0/1418606980 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f1938003aa0 con 0x7f1940141690 2023-10-19T12:44:56.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:56.575+0000 7f1927fff640 1 -- 172.21.15.57:0/1418606980 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1938004030 con 0x7f1940141690 2023-10-19T12:44:56.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:56.575+0000 7f1945e9b640 1 -- 172.21.15.57:0/1418606980 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19400ac020 con 0x7f1940141690 2023-10-19T12:44:56.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:56.575+0000 7f1945e9b640 1 -- 172.21.15.57:0/1418606980 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f194013ac30 con 0x7f1940141690 2023-10-19T12:44:56.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:56.576+0000 7f1945e9b640 1 -- 172.21.15.57:0/1418606980 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19400ac240 con 0x7f1940141690 2023-10-19T12:44:56.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:56.576+0000 7f1927fff640 1 -- 172.21.15.57:0/1418606980 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f1938004190 con 0x7f1940141690 2023-10-19T12:44:56.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:56.576+0000 7f1927fff640 1 --2- 172.21.15.57:0/1418606980 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f192c06ae40 0x7f192c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:56.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:56.577+0000 7f1927fff640 1 -- 172.21.15.57:0/1418606980 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f193808a1b0 con 0x7f1940141690 2023-10-19T12:44:56.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:56.577+0000 7f193f7fe640 1 --2- 172.21.15.57:0/1418606980 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f192c06ae40 0x7f192c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:56.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:56.577+0000 7f193f7fe640 1 --2- 172.21.15.57:0/1418606980 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f192c06ae40 0x7f192c06d300 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f1930035000 tx=0x7f1930004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:56.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:56.581+0000 7f1927fff640 1 -- 172.21.15.57:0/1418606980 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f1938054f70 con 0x7f1940141690 2023-10-19T12:44:56.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:56.791+0000 7f1945e9b640 1 -- 172.21.15.57:0/1418606980 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f19400690e0 con 0x7f1940141690 2023-10-19T12:44:56.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:56.791+0000 7f1927fff640 1 -- 172.21.15.57:0/1418606980 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) v1 ==== 53+0+88 (secure 0 0 0) 0x7f1938058590 con 0x7f1940141690 2023-10-19T12:44:56.795 INFO:teuthology.orchestra.run.smithi057.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-10-19T12:44:56.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:56.793+0000 7f1945e9b640 1 -- 172.21.15.57:0/1418606980 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f192c06ae40 msgr2=0x7f192c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:56.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:56.793+0000 7f1945e9b640 1 --2- 172.21.15.57:0/1418606980 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f192c06ae40 0x7f192c06d300 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f1930035000 tx=0x7f1930004730 comp rx=0 tx=0).stop 2023-10-19T12:44:56.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:56.793+0000 7f1945e9b640 1 -- 172.21.15.57:0/1418606980 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1940141690 msgr2=0x7f19400ca870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:56.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:56.793+0000 7f1945e9b640 1 --2- 172.21.15.57:0/1418606980 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1940141690 0x7f19400ca870 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f193800cb00 tx=0x7f193800a250 comp rx=0 tx=0).stop 2023-10-19T12:44:56.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:56.793+0000 7f1945e9b640 1 -- 172.21.15.57:0/1418606980 shutdown_connections 2023-10-19T12:44:56.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:56.793+0000 7f1945e9b640 1 --2- 172.21.15.57:0/1418606980 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f192c06ae40 0x7f192c06d300 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:56.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:56.793+0000 7f1945e9b640 1 --2- 172.21.15.57:0/1418606980 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1940141690 0x7f19400ca870 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:56.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:56.793+0000 7f1945e9b640 1 --2- 172.21.15.57:0/1418606980 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f194013e520 0x7f19400add80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:56.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:56.793+0000 7f1945e9b640 1 --2- 172.21.15.57:0/1418606980 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1940070af0 0x7f194013a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:56.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:56.793+0000 7f1945e9b640 1 -- 172.21.15.57:0/1418606980 >> 172.21.15.57:0/1418606980 conn(0x7f1940069930 msgr2=0x7f1940140380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:56.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:56.793+0000 7f1945e9b640 1 -- 172.21.15.57:0/1418606980 shutdown_connections 2023-10-19T12:44:56.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:56.793+0000 7f1945e9b640 1 -- 172.21.15.57:0/1418606980 wait complete. 2023-10-19T12:44:56.806 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-10-19T12:44:56.807 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-19T12:44:56.807 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi179.front.sepia.ceph.com 2023-10-19T12:44:56.807 INFO:tasks.cephfs.mount:self.client.name = client.1 2023-10-19T12:44:56.808 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-10-19T12:44:56.808 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-10-19T12:44:56.808 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-10-19T12:44:56.809 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-19T12:44:56.809 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2023-10-19T12:44:56.809 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:44:56.809 DEBUG:teuthology.orchestra.run.smithi179:> ip addr 2023-10-19T12:44:56.828 INFO:teuthology.orchestra.run.smithi179.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-19T12:44:56.828 INFO:teuthology.orchestra.run.smithi179.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-19T12:44:56.828 INFO:teuthology.orchestra.run.smithi179.stdout: inet 127.0.0.1/8 scope host lo 2023-10-19T12:44:56.828 INFO:teuthology.orchestra.run.smithi179.stdout: valid_lft forever preferred_lft forever 2023-10-19T12:44:56.829 INFO:teuthology.orchestra.run.smithi179.stdout: inet6 ::1/128 scope host 2023-10-19T12:44:56.829 INFO:teuthology.orchestra.run.smithi179.stdout: valid_lft forever preferred_lft forever 2023-10-19T12:44:56.829 INFO:teuthology.orchestra.run.smithi179.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-19T12:44:56.829 INFO:teuthology.orchestra.run.smithi179.stdout: link/ether 0c:c4:7a:d9:3b:34 brd ff:ff:ff:ff:ff:ff 2023-10-19T12:44:56.829 INFO:teuthology.orchestra.run.smithi179.stdout: altname enp7s0f0 2023-10-19T12:44:56.830 INFO:teuthology.orchestra.run.smithi179.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-19T12:44:56.830 INFO:teuthology.orchestra.run.smithi179.stdout: link/ether 0c:c4:7a:d9:3b:35 brd ff:ff:ff:ff:ff:ff 2023-10-19T12:44:56.830 INFO:teuthology.orchestra.run.smithi179.stdout: altname enp7s0f1 2023-10-19T12:44:56.830 INFO:teuthology.orchestra.run.smithi179.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-19T12:44:56.830 INFO:teuthology.orchestra.run.smithi179.stdout: link/ether 0c:c4:7a:88:72:5a brd ff:ff:ff:ff:ff:ff 2023-10-19T12:44:56.831 INFO:teuthology.orchestra.run.smithi179.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-19T12:44:56.831 INFO:teuthology.orchestra.run.smithi179.stdout: link/ether 0c:c4:7a:88:72:5b brd ff:ff:ff:ff:ff:ff 2023-10-19T12:44:56.831 INFO:teuthology.orchestra.run.smithi179.stdout: inet 172.21.15.179/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-10-19T12:44:56.831 INFO:teuthology.orchestra.run.smithi179.stdout: valid_lft 42599sec preferred_lft 42599sec 2023-10-19T12:44:56.831 INFO:teuthology.orchestra.run.smithi179.stdout: inet6 fe80::ec4:7aff:fe88:725b/64 scope link 2023-10-19T12:44:56.831 INFO:teuthology.orchestra.run.smithi179.stdout: valid_lft forever preferred_lft forever 2023-10-19T12:44:56.832 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-10-19T12:44:56.832 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:44:56.832 DEBUG:teuthology.orchestra.run.smithi179:> (cd / && exec bash -c ' 2023-10-19T12:44:56.833 DEBUG:teuthology.orchestra.run.smithi179:> set -e 2023-10-19T12:44:56.833 DEBUG:teuthology.orchestra.run.smithi179:> sudo ip link add name ceph-brx type bridge 2023-10-19T12:44:56.833 DEBUG:teuthology.orchestra.run.smithi179:> sudo ip addr flush dev ceph-brx 2023-10-19T12:44:56.834 DEBUG:teuthology.orchestra.run.smithi179:> sudo ip link set ceph-brx up 2023-10-19T12:44:56.834 DEBUG:teuthology.orchestra.run.smithi179:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-10-19T12:44:56.834 DEBUG:teuthology.orchestra.run.smithi179:> ') 2023-10-19T12:44:57.081 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:44:57.082 DEBUG:teuthology.orchestra.run.smithi179:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-19T12:44:57.110 INFO:teuthology.orchestra.run.smithi179.stdout:1 2023-10-19T12:44:57.111 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:44:57.112 DEBUG:teuthology.orchestra.run.smithi179:> route 2023-10-19T12:44:57.179 INFO:teuthology.orchestra.run.smithi179.stdout:Kernel IP routing table 2023-10-19T12:44:57.179 INFO:teuthology.orchestra.run.smithi179.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-19T12:44:57.180 INFO:teuthology.orchestra.run.smithi179.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-10-19T12:44:57.180 INFO:teuthology.orchestra.run.smithi179.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-10-19T12:44:57.180 INFO:teuthology.orchestra.run.smithi179.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-19T12:44:57.181 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:44:57.181 DEBUG:teuthology.orchestra.run.smithi179:> (cd / && exec bash -c ' 2023-10-19T12:44:57.182 DEBUG:teuthology.orchestra.run.smithi179:> set -e 2023-10-19T12:44:57.182 DEBUG:teuthology.orchestra.run.smithi179:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-10-19T12:44:57.183 DEBUG:teuthology.orchestra.run.smithi179:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-10-19T12:44:57.183 DEBUG:teuthology.orchestra.run.smithi179:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-10-19T12:44:57.183 DEBUG:teuthology.orchestra.run.smithi179:> ') 2023-10-19T12:44:57.454 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:44:57.455 DEBUG:teuthology.orchestra.run.smithi179:> ip netns list 2023-10-19T12:44:57.470 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:44:57.470 DEBUG:teuthology.orchestra.run.smithi179:> ip netns list-id 2023-10-19T12:44:57.524 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:44:57.525 DEBUG:teuthology.orchestra.run.smithi179:> (cd / && exec bash -c ' 2023-10-19T12:44:57.525 DEBUG:teuthology.orchestra.run.smithi179:> set -e 2023-10-19T12:44:57.525 DEBUG:teuthology.orchestra.run.smithi179:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2023-10-19T12:44:57.526 DEBUG:teuthology.orchestra.run.smithi179:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2023-10-19T12:44:57.526 DEBUG:teuthology.orchestra.run.smithi179:> ') 2023-10-19T12:44:57.609 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2023-10-19T12:44:57.609 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:44:57.609 DEBUG:teuthology.orchestra.run.smithi179:> (cd / && exec bash -c ' 2023-10-19T12:44:57.610 DEBUG:teuthology.orchestra.run.smithi179:> set -e 2023-10-19T12:44:57.610 DEBUG:teuthology.orchestra.run.smithi179:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2023-10-19T12:44:57.610 DEBUG:teuthology.orchestra.run.smithi179:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-10-19T12:44:57.611 DEBUG:teuthology.orchestra.run.smithi179:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2023-10-19T12:44:57.611 DEBUG:teuthology.orchestra.run.smithi179:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2023-10-19T12:44:57.611 DEBUG:teuthology.orchestra.run.smithi179:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2023-10-19T12:44:57.611 DEBUG:teuthology.orchestra.run.smithi179:> ') 2023-10-19T12:44:57.748 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:44:57.749 DEBUG:teuthology.orchestra.run.smithi179:> (cd / && exec bash -c ' 2023-10-19T12:44:57.749 DEBUG:teuthology.orchestra.run.smithi179:> set -e 2023-10-19T12:44:57.749 DEBUG:teuthology.orchestra.run.smithi179:> sudo ip link set brx.0 up 2023-10-19T12:44:57.749 DEBUG:teuthology.orchestra.run.smithi179:> sudo ip link set dev brx.0 master ceph-brx 2023-10-19T12:44:57.750 DEBUG:teuthology.orchestra.run.smithi179:> ') 2023-10-19T12:44:57.790 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2023-10-19T12:44:57.791 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-19T12:44:57.791 DEBUG:teuthology.orchestra.run.smithi179:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2023-10-19T12:44:57.849 INFO:teuthology.orchestra.run.smithi179.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2023-10-19T12:44:57.850 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-19T12:44:57.850 DEBUG:teuthology.orchestra.run.smithi179:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2023-10-19T12:44:57.904 DEBUG:teuthology.orchestra.run.smithi179:> sudo modprobe fuse 2023-10-19T12:44:57.971 DEBUG:teuthology.orchestra.run.smithi179:> cat /proc/self/mounts | awk '{print $2}' 2023-10-19T12:44:58.027 INFO:teuthology.orchestra.run.smithi179.stdout:/proc 2023-10-19T12:44:58.027 INFO:teuthology.orchestra.run.smithi179.stdout:/sys 2023-10-19T12:44:58.027 INFO:teuthology.orchestra.run.smithi179.stdout:/dev 2023-10-19T12:44:58.028 INFO:teuthology.orchestra.run.smithi179.stdout:/sys/kernel/security 2023-10-19T12:44:58.028 INFO:teuthology.orchestra.run.smithi179.stdout:/dev/shm 2023-10-19T12:44:58.028 INFO:teuthology.orchestra.run.smithi179.stdout:/dev/pts 2023-10-19T12:44:58.028 INFO:teuthology.orchestra.run.smithi179.stdout:/run 2023-10-19T12:44:58.028 INFO:teuthology.orchestra.run.smithi179.stdout:/sys/fs/cgroup 2023-10-19T12:44:58.028 INFO:teuthology.orchestra.run.smithi179.stdout:/sys/fs/pstore 2023-10-19T12:44:58.029 INFO:teuthology.orchestra.run.smithi179.stdout:/sys/fs/bpf 2023-10-19T12:44:58.029 INFO:teuthology.orchestra.run.smithi179.stdout:/ 2023-10-19T12:44:58.029 INFO:teuthology.orchestra.run.smithi179.stdout:/sys/fs/selinux 2023-10-19T12:44:58.029 INFO:teuthology.orchestra.run.smithi179.stdout:/proc/sys/fs/binfmt_misc 2023-10-19T12:44:58.029 INFO:teuthology.orchestra.run.smithi179.stdout:/dev/hugepages 2023-10-19T12:44:58.030 INFO:teuthology.orchestra.run.smithi179.stdout:/dev/mqueue 2023-10-19T12:44:58.030 INFO:teuthology.orchestra.run.smithi179.stdout:/sys/kernel/debug 2023-10-19T12:44:58.030 INFO:teuthology.orchestra.run.smithi179.stdout:/sys/kernel/tracing 2023-10-19T12:44:58.030 INFO:teuthology.orchestra.run.smithi179.stdout:/sys/fs/fuse/connections 2023-10-19T12:44:58.031 INFO:teuthology.orchestra.run.smithi179.stdout:/sys/kernel/config 2023-10-19T12:44:58.031 INFO:teuthology.orchestra.run.smithi179.stdout:/run/credentials/systemd-sysctl.service 2023-10-19T12:44:58.031 INFO:teuthology.orchestra.run.smithi179.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-10-19T12:44:58.031 INFO:teuthology.orchestra.run.smithi179.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-10-19T12:44:58.031 INFO:teuthology.orchestra.run.smithi179.stdout:/var/lib/nfs/rpc_pipefs 2023-10-19T12:44:58.031 INFO:teuthology.orchestra.run.smithi179.stdout:/run/user/1000 2023-10-19T12:44:58.032 INFO:teuthology.orchestra.run.smithi179.stdout:/var/lib/ceph 2023-10-19T12:44:58.032 INFO:teuthology.orchestra.run.smithi179.stdout:/var/lib/ceph/osd/ceph-4 2023-10-19T12:44:58.032 INFO:teuthology.orchestra.run.smithi179.stdout:/var/lib/ceph/osd/ceph-5 2023-10-19T12:44:58.032 INFO:teuthology.orchestra.run.smithi179.stdout:/var/lib/ceph/osd/ceph-6 2023-10-19T12:44:58.032 INFO:teuthology.orchestra.run.smithi179.stdout:/var/lib/ceph/osd/ceph-7 2023-10-19T12:44:58.032 INFO:teuthology.orchestra.run.smithi179.stdout:/run/netns 2023-10-19T12:44:58.033 INFO:teuthology.orchestra.run.smithi179.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2023-10-19T12:44:58.033 INFO:teuthology.orchestra.run.smithi179.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2023-10-19T12:44:58.033 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:44:58.033 DEBUG:teuthology.orchestra.run.smithi179:> ls /sys/fs/fuse/connections 2023-10-19T12:44:58.082 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-10-19T12:44:58.082 DEBUG:teuthology.orchestra.run.smithi179:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2023-10-19T12:44:58.124 DEBUG:teuthology.orchestra.run.smithi179:> sudo modprobe fuse 2023-10-19T12:44:58.149 DEBUG:teuthology.orchestra.run.smithi179:> cat /proc/self/mounts | awk '{print $2}' 2023-10-19T12:44:58.204 INFO:teuthology.orchestra.run.smithi179.stdout:/proc 2023-10-19T12:44:58.205 INFO:teuthology.orchestra.run.smithi179.stdout:/sys 2023-10-19T12:44:58.205 INFO:teuthology.orchestra.run.smithi179.stdout:/dev 2023-10-19T12:44:58.205 INFO:teuthology.orchestra.run.smithi179.stdout:/sys/kernel/security 2023-10-19T12:44:58.205 INFO:teuthology.orchestra.run.smithi179.stdout:/dev/shm 2023-10-19T12:44:58.205 INFO:teuthology.orchestra.run.smithi179.stdout:/dev/pts 2023-10-19T12:44:58.205 INFO:teuthology.orchestra.run.smithi179.stdout:/run 2023-10-19T12:44:58.206 INFO:teuthology.orchestra.run.smithi179.stdout:/sys/fs/cgroup 2023-10-19T12:44:58.206 INFO:teuthology.orchestra.run.smithi179.stdout:/sys/fs/pstore 2023-10-19T12:44:58.206 INFO:teuthology.orchestra.run.smithi179.stdout:/sys/fs/bpf 2023-10-19T12:44:58.206 INFO:teuthology.orchestra.run.smithi179.stdout:/ 2023-10-19T12:44:58.206 INFO:teuthology.orchestra.run.smithi179.stdout:/sys/fs/selinux 2023-10-19T12:44:58.207 INFO:teuthology.orchestra.run.smithi179.stdout:/proc/sys/fs/binfmt_misc 2023-10-19T12:44:58.207 INFO:teuthology.orchestra.run.smithi179.stdout:/dev/hugepages 2023-10-19T12:44:58.207 INFO:teuthology.orchestra.run.smithi179.stdout:/dev/mqueue 2023-10-19T12:44:58.207 INFO:teuthology.orchestra.run.smithi179.stdout:/sys/kernel/debug 2023-10-19T12:44:58.207 INFO:teuthology.orchestra.run.smithi179.stdout:/sys/kernel/tracing 2023-10-19T12:44:58.208 INFO:teuthology.orchestra.run.smithi179.stdout:/sys/fs/fuse/connections 2023-10-19T12:44:58.208 INFO:teuthology.orchestra.run.smithi179.stdout:/sys/kernel/config 2023-10-19T12:44:58.208 INFO:teuthology.orchestra.run.smithi179.stdout:/run/credentials/systemd-sysctl.service 2023-10-19T12:44:58.208 INFO:teuthology.orchestra.run.smithi179.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-10-19T12:44:58.208 INFO:teuthology.orchestra.run.smithi179.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-10-19T12:44:58.209 INFO:teuthology.orchestra.run.smithi179.stdout:/var/lib/nfs/rpc_pipefs 2023-10-19T12:44:58.209 INFO:teuthology.orchestra.run.smithi179.stdout:/run/user/1000 2023-10-19T12:44:58.209 INFO:teuthology.orchestra.run.smithi179.stdout:/var/lib/ceph 2023-10-19T12:44:58.209 INFO:teuthology.orchestra.run.smithi179.stdout:/var/lib/ceph/osd/ceph-4 2023-10-19T12:44:58.209 INFO:teuthology.orchestra.run.smithi179.stdout:/var/lib/ceph/osd/ceph-5 2023-10-19T12:44:58.209 INFO:teuthology.orchestra.run.smithi179.stdout:/var/lib/ceph/osd/ceph-6 2023-10-19T12:44:58.210 INFO:teuthology.orchestra.run.smithi179.stdout:/var/lib/ceph/osd/ceph-7 2023-10-19T12:44:58.210 INFO:teuthology.orchestra.run.smithi179.stdout:/run/netns 2023-10-19T12:44:58.210 INFO:teuthology.orchestra.run.smithi179.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2023-10-19T12:44:58.210 INFO:teuthology.orchestra.run.smithi179.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2023-10-19T12:44:58.210 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:44:58.211 DEBUG:teuthology.orchestra.run.smithi179:> ls /sys/fs/fuse/connections 2023-10-19T12:44:58.223 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi179.stderr:2023-10-19T12:44:58.220+0000 7fef9a00c480 -1 init, newargv = 0x55b5c75e6cb0 newargc=15 2023-10-19T12:44:58.223 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi179.stderr:ceph-fuse[94341]: starting ceph client 2023-10-19T12:44:58.228 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi179.stderr:ceph-fuse[94341]: starting fuse 2023-10-19T12:44:58.237 INFO:teuthology.orchestra.run.smithi179.stdout:43 2023-10-19T12:44:58.238 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2023-10-19T12:44:58.238 DEBUG:teuthology.orchestra.run.smithi179:> sudo stdin-killer -- python3 -c ' 2023-10-19T12:44:58.239 DEBUG:teuthology.orchestra.run.smithi179:> import glob 2023-10-19T12:44:58.239 DEBUG:teuthology.orchestra.run.smithi179:> import re 2023-10-19T12:44:58.240 DEBUG:teuthology.orchestra.run.smithi179:> import os 2023-10-19T12:44:58.240 DEBUG:teuthology.orchestra.run.smithi179:> import subprocess 2023-10-19T12:44:58.240 DEBUG:teuthology.orchestra.run.smithi179:> 2023-10-19T12:44:58.241 DEBUG:teuthology.orchestra.run.smithi179:> def _find_admin_socket(client_name): 2023-10-19T12:44:58.241 DEBUG:teuthology.orchestra.run.smithi179:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2023-10-19T12:44:58.241 DEBUG:teuthology.orchestra.run.smithi179:> files = glob.glob(asok_path) 2023-10-19T12:44:58.241 DEBUG:teuthology.orchestra.run.smithi179:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2023-10-19T12:44:58.242 DEBUG:teuthology.orchestra.run.smithi179:> 2023-10-19T12:44:58.242 DEBUG:teuthology.orchestra.run.smithi179:> # Given a non-glob path, it better be there 2023-10-19T12:44:58.242 DEBUG:teuthology.orchestra.run.smithi179:> if "*" not in asok_path: 2023-10-19T12:44:58.243 DEBUG:teuthology.orchestra.run.smithi179:> assert(len(files) == 1) 2023-10-19T12:44:58.243 DEBUG:teuthology.orchestra.run.smithi179:> return files[0] 2023-10-19T12:44:58.243 DEBUG:teuthology.orchestra.run.smithi179:> 2023-10-19T12:44:58.244 DEBUG:teuthology.orchestra.run.smithi179:> for f in files: 2023-10-19T12:44:58.244 DEBUG:teuthology.orchestra.run.smithi179:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-10-19T12:44:58.244 DEBUG:teuthology.orchestra.run.smithi179:> if os.path.exists("/proc/{0}".format(pid)): 2023-10-19T12:44:58.244 DEBUG:teuthology.orchestra.run.smithi179:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-10-19T12:44:58.245 DEBUG:teuthology.orchestra.run.smithi179:> contents = proc_f.read() 2023-10-19T12:44:58.245 DEBUG:teuthology.orchestra.run.smithi179:> if mountpoint in contents: 2023-10-19T12:44:58.245 DEBUG:teuthology.orchestra.run.smithi179:> return f 2023-10-19T12:44:58.246 DEBUG:teuthology.orchestra.run.smithi179:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-10-19T12:44:58.246 DEBUG:teuthology.orchestra.run.smithi179:> 2023-10-19T12:44:58.246 DEBUG:teuthology.orchestra.run.smithi179:> print(_find_admin_socket("client.1")) 2023-10-19T12:44:58.246 DEBUG:teuthology.orchestra.run.smithi179:> ' 2023-10-19T12:44:58.357 INFO:teuthology.orchestra.run.smithi179.stdout:/var/run/ceph/ceph-client.1.94341.asok 2023-10-19T12:44:58.361 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:44:58 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2023-10-19T12:44:58.368 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.94341.asok 2023-10-19T12:44:58.368 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:44:58.368 DEBUG:teuthology.orchestra.run.smithi179:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.94341.asok status 2023-10-19T12:44:58.504 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2023-10-19T12:44:58.505 INFO:teuthology.orchestra.run.smithi179.stdout: "metadata": { 2023-10-19T12:44:58.505 INFO:teuthology.orchestra.run.smithi179.stdout: "ceph_sha1": "f7420d7f8e55ceea786dcbf536c338ce920c21eb", 2023-10-19T12:44:58.505 INFO:teuthology.orchestra.run.smithi179.stdout: "ceph_version": "ceph version 18.0.0-6786-gf7420d7f (f7420d7f8e55ceea786dcbf536c338ce920c21eb) reef (dev)", 2023-10-19T12:44:58.505 INFO:teuthology.orchestra.run.smithi179.stdout: "entity_id": "1", 2023-10-19T12:44:58.505 INFO:teuthology.orchestra.run.smithi179.stdout: "hostname": "smithi179", 2023-10-19T12:44:58.506 INFO:teuthology.orchestra.run.smithi179.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2023-10-19T12:44:58.506 INFO:teuthology.orchestra.run.smithi179.stdout: "pid": "94341", 2023-10-19T12:44:58.506 INFO:teuthology.orchestra.run.smithi179.stdout: "root": "/" 2023-10-19T12:44:58.507 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2023-10-19T12:44:58.507 INFO:teuthology.orchestra.run.smithi179.stdout: "dentry_count": 0, 2023-10-19T12:44:58.507 INFO:teuthology.orchestra.run.smithi179.stdout: "dentry_pinned_count": 0, 2023-10-19T12:44:58.507 INFO:teuthology.orchestra.run.smithi179.stdout: "id": 4667, 2023-10-19T12:44:58.507 INFO:teuthology.orchestra.run.smithi179.stdout: "inst": { 2023-10-19T12:44:58.508 INFO:teuthology.orchestra.run.smithi179.stdout: "name": { 2023-10-19T12:44:58.508 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "client", 2023-10-19T12:44:58.508 INFO:teuthology.orchestra.run.smithi179.stdout: "num": 4667 2023-10-19T12:44:58.508 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2023-10-19T12:44:58.508 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": { 2023-10-19T12:44:58.509 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v1", 2023-10-19T12:44:58.509 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "192.168.0.1:0", 2023-10-19T12:44:58.509 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 109532630 2023-10-19T12:44:58.509 INFO:teuthology.orchestra.run.smithi179.stdout: } 2023-10-19T12:44:58.509 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2023-10-19T12:44:58.510 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": { 2023-10-19T12:44:58.510 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v1", 2023-10-19T12:44:58.510 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "192.168.0.1:0", 2023-10-19T12:44:58.511 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 109532630 2023-10-19T12:44:58.511 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2023-10-19T12:44:58.511 INFO:teuthology.orchestra.run.smithi179.stdout: "inst_str": "client.4667 192.168.0.1:0/109532630", 2023-10-19T12:44:58.511 INFO:teuthology.orchestra.run.smithi179.stdout: "addr_str": "192.168.0.1:0/109532630", 2023-10-19T12:44:58.511 INFO:teuthology.orchestra.run.smithi179.stdout: "inode_count": 1, 2023-10-19T12:44:58.511 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_epoch": 9, 2023-10-19T12:44:58.512 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_epoch": 26, 2023-10-19T12:44:58.512 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_epoch_barrier": 0, 2023-10-19T12:44:58.512 INFO:teuthology.orchestra.run.smithi179.stdout: "blocklisted": false, 2023-10-19T12:44:58.512 INFO:teuthology.orchestra.run.smithi179.stdout: "fs_name": "cephfs" 2023-10-19T12:44:58.513 INFO:teuthology.orchestra.run.smithi179.stdout:} 2023-10-19T12:44:58.514 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-19T12:44:58.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:58.632+0000 7f4d67256640 1 -- 172.21.15.57:0/1251630746 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4d600ceb40 msgr2=0x7f4d600b0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:58.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:58.632+0000 7f4d67256640 1 --2- 172.21.15.57:0/1251630746 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4d600ceb40 0x7f4d600b0650 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f4d58005fd0 tx=0x7f4d5802c240 comp rx=0 tx=0).stop 2023-10-19T12:44:58.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:58.632+0000 7f4d67256640 1 -- 172.21.15.57:0/1251630746 shutdown_connections 2023-10-19T12:44:58.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:58.632+0000 7f4d67256640 1 --2- 172.21.15.57:0/1251630746 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4d600ceb40 0x7f4d600b0650 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:58.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:58.632+0000 7f4d67256640 1 --2- 172.21.15.57:0/1251630746 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4d6015b2a0 0x7f4d600d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:58.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:58.632+0000 7f4d67256640 1 -- 172.21.15.57:0/1251630746 >> 172.21.15.57:0/1251630746 conn(0x7f4d600d0d60 msgr2=0x7f4d600d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:58.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:58.632+0000 7f4d67256640 1 -- 172.21.15.57:0/1251630746 shutdown_connections 2023-10-19T12:44:58.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:58.632+0000 7f4d67256640 1 -- 172.21.15.57:0/1251630746 wait complete. 2023-10-19T12:44:58.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:58.634+0000 7f4d67256640 1 Processor -- start 2023-10-19T12:44:58.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:58.634+0000 7f4d67256640 1 -- start start 2023-10-19T12:44:58.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:58.634+0000 7f4d67256640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4d600ceb40 0x7f4d6014e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:58.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:58.634+0000 7f4d67256640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4d600cf340 0x7f4d6015b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:58.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:58.634+0000 7f4d67256640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4d6015b2a0 0x7f4d6015bc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:58.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:58.634+0000 7f4d67256640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d600ca6f0 con 0x7f4d600cf340 2023-10-19T12:44:58.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:58.634+0000 7f4d67256640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d6014ecd0 con 0x7f4d6015b2a0 2023-10-19T12:44:58.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:58.634+0000 7f4d67256640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d600ca410 con 0x7f4d600ceb40 2023-10-19T12:44:58.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:58.635+0000 7f4d5ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4d600cf340 0x7f4d6015b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:58.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:58.635+0000 7f4d5ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4d600cf340 0x7f4d6015b6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50266/0 (socket says 172.21.15.57:50266) 2023-10-19T12:44:58.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:58.635+0000 7f4d5ffff640 1 -- 172.21.15.57:0/1645960247 learned_addr learned my addr 172.21.15.57:0/1645960247 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:44:58.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:58.635+0000 7f4d657f3640 1 --2- 172.21.15.57:0/1645960247 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4d6015b2a0 0x7f4d6015bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:58.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:58.635+0000 7f4d64ff2640 1 --2- 172.21.15.57:0/1645960247 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4d600ceb40 0x7f4d6014e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:58.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:58.635+0000 7f4d5ffff640 1 -- 172.21.15.57:0/1645960247 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4d600ceb40 msgr2=0x7f4d6014e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:58.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:58.635+0000 7f4d5ffff640 1 --2- 172.21.15.57:0/1645960247 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4d600ceb40 0x7f4d6014e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:58.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:58.635+0000 7f4d5ffff640 1 -- 172.21.15.57:0/1645960247 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4d6015b2a0 msgr2=0x7f4d6015bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:58.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:58.635+0000 7f4d5ffff640 1 --2- 172.21.15.57:0/1645960247 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4d6015b2a0 0x7f4d6015bc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:58.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:58.635+0000 7f4d5ffff640 1 -- 172.21.15.57:0/1645960247 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d58009900 con 0x7f4d600cf340 2023-10-19T12:44:58.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:58.635+0000 7f4d5ffff640 1 --2- 172.21.15.57:0/1645960247 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4d600cf340 0x7f4d6015b6d0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f4d4c00d6d0 tx=0x7f4d4c00db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:58.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:58.635+0000 7f4d5d7fa640 1 -- 172.21.15.57:0/1645960247 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d4c005220 con 0x7f4d600cf340 2023-10-19T12:44:58.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:58.635+0000 7f4d5d7fa640 1 -- 172.21.15.57:0/1645960247 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f4d4c005490 con 0x7f4d600cf340 2023-10-19T12:44:58.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:58.635+0000 7f4d67256640 1 -- 172.21.15.57:0/1645960247 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d60151ce0 con 0x7f4d600cf340 2023-10-19T12:44:58.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:58.635+0000 7f4d5d7fa640 1 -- 172.21.15.57:0/1645960247 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d4c009a80 con 0x7f4d600cf340 2023-10-19T12:44:58.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:58.635+0000 7f4d67256640 1 -- 172.21.15.57:0/1645960247 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d601522a0 con 0x7f4d600cf340 2023-10-19T12:44:58.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:58.636+0000 7f4d67256640 1 -- 172.21.15.57:0/1645960247 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d600690e0 con 0x7f4d600cf340 2023-10-19T12:44:58.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:58.637+0000 7f4d5d7fa640 1 -- 172.21.15.57:0/1645960247 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f4d4c0047a0 con 0x7f4d600cf340 2023-10-19T12:44:58.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:58.637+0000 7f4d5d7fa640 1 --2- 172.21.15.57:0/1645960247 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f4d4406ae70 0x7f4d4406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:44:58.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:58.637+0000 7f4d5d7fa640 1 -- 172.21.15.57:0/1645960247 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f4d4c089cc0 con 0x7f4d600cf340 2023-10-19T12:44:58.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:58.637+0000 7f4d64ff2640 1 --2- 172.21.15.57:0/1645960247 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f4d4406ae70 0x7f4d4406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:44:58.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:58.638+0000 7f4d64ff2640 1 --2- 172.21.15.57:0/1645960247 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f4d4406ae70 0x7f4d4406d330 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f4d54000f60 tx=0x7f4d54009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:44:58.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:58.642+0000 7f4d5d7fa640 1 -- 172.21.15.57:0/1645960247 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4d4c090080 con 0x7f4d600cf340 2023-10-19T12:44:58.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:58.853+0000 7f4d67256640 1 -- 172.21.15.57:0/1645960247 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f4d600cb980 con 0x7f4d600cf340 2023-10-19T12:44:58.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:58.854+0000 7f4d5d7fa640 1 -- 172.21.15.57:0/1645960247 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) v1 ==== 53+0+88 (secure 0 0 0) 0x7f4d4c017020 con 0x7f4d600cf340 2023-10-19T12:44:58.857 INFO:teuthology.orchestra.run.smithi057.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-10-19T12:44:58.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:58.855+0000 7f4d67256640 1 -- 172.21.15.57:0/1645960247 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f4d4406ae70 msgr2=0x7f4d4406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:58.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:58.855+0000 7f4d67256640 1 --2- 172.21.15.57:0/1645960247 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f4d4406ae70 0x7f4d4406d330 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f4d54000f60 tx=0x7f4d54009040 comp rx=0 tx=0).stop 2023-10-19T12:44:58.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:58.855+0000 7f4d67256640 1 -- 172.21.15.57:0/1645960247 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4d600cf340 msgr2=0x7f4d6015b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:44:58.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:58.855+0000 7f4d67256640 1 --2- 172.21.15.57:0/1645960247 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4d600cf340 0x7f4d6015b6d0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f4d4c00d6d0 tx=0x7f4d4c00db90 comp rx=0 tx=0).stop 2023-10-19T12:44:58.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:58.855+0000 7f4d67256640 1 -- 172.21.15.57:0/1645960247 shutdown_connections 2023-10-19T12:44:58.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:58.855+0000 7f4d67256640 1 --2- 172.21.15.57:0/1645960247 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f4d4406ae70 0x7f4d4406d330 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:58.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:58.855+0000 7f4d67256640 1 --2- 172.21.15.57:0/1645960247 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4d6015b2a0 0x7f4d6015bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:58.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:58.855+0000 7f4d67256640 1 --2- 172.21.15.57:0/1645960247 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4d600cf340 0x7f4d6015b6d0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:58.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:58.855+0000 7f4d67256640 1 --2- 172.21.15.57:0/1645960247 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4d600ceb40 0x7f4d6014e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:44:58.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:58.856+0000 7f4d67256640 1 -- 172.21.15.57:0/1645960247 >> 172.21.15.57:0/1645960247 conn(0x7f4d600d0d60 msgr2=0x7f4d60151240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:44:58.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:58.856+0000 7f4d67256640 1 -- 172.21.15.57:0/1645960247 shutdown_connections 2023-10-19T12:44:58.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:44:58.856+0000 7f4d67256640 1 -- 172.21.15.57:0/1645960247 wait complete. 2023-10-19T12:44:58.869 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-10-19T12:44:58.869 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-19T12:44:58.869 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi179.front.sepia.ceph.com 2023-10-19T12:44:58.869 INFO:tasks.cephfs.mount:self.client.name = client.3 2023-10-19T12:44:58.870 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2023-10-19T12:44:58.870 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-10-19T12:44:58.870 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-10-19T12:44:58.870 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-19T12:44:58.871 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2023-10-19T12:44:58.871 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:44:58.871 DEBUG:teuthology.orchestra.run.smithi179:> ip addr 2023-10-19T12:44:58.887 INFO:teuthology.orchestra.run.smithi179.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-19T12:44:58.887 INFO:teuthology.orchestra.run.smithi179.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-19T12:44:58.887 INFO:teuthology.orchestra.run.smithi179.stdout: inet 127.0.0.1/8 scope host lo 2023-10-19T12:44:58.887 INFO:teuthology.orchestra.run.smithi179.stdout: valid_lft forever preferred_lft forever 2023-10-19T12:44:58.888 INFO:teuthology.orchestra.run.smithi179.stdout: inet6 ::1/128 scope host 2023-10-19T12:44:58.888 INFO:teuthology.orchestra.run.smithi179.stdout: valid_lft forever preferred_lft forever 2023-10-19T12:44:58.888 INFO:teuthology.orchestra.run.smithi179.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-19T12:44:58.888 INFO:teuthology.orchestra.run.smithi179.stdout: link/ether 0c:c4:7a:d9:3b:34 brd ff:ff:ff:ff:ff:ff 2023-10-19T12:44:58.888 INFO:teuthology.orchestra.run.smithi179.stdout: altname enp7s0f0 2023-10-19T12:44:58.889 INFO:teuthology.orchestra.run.smithi179.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-19T12:44:58.889 INFO:teuthology.orchestra.run.smithi179.stdout: link/ether 0c:c4:7a:d9:3b:35 brd ff:ff:ff:ff:ff:ff 2023-10-19T12:44:58.889 INFO:teuthology.orchestra.run.smithi179.stdout: altname enp7s0f1 2023-10-19T12:44:58.889 INFO:teuthology.orchestra.run.smithi179.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-19T12:44:58.889 INFO:teuthology.orchestra.run.smithi179.stdout: link/ether 0c:c4:7a:88:72:5a brd ff:ff:ff:ff:ff:ff 2023-10-19T12:44:58.889 INFO:teuthology.orchestra.run.smithi179.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-19T12:44:58.890 INFO:teuthology.orchestra.run.smithi179.stdout: link/ether 0c:c4:7a:88:72:5b brd ff:ff:ff:ff:ff:ff 2023-10-19T12:44:58.890 INFO:teuthology.orchestra.run.smithi179.stdout: inet 172.21.15.179/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-10-19T12:44:58.890 INFO:teuthology.orchestra.run.smithi179.stdout: valid_lft 42597sec preferred_lft 42597sec 2023-10-19T12:44:58.890 INFO:teuthology.orchestra.run.smithi179.stdout: inet6 fe80::ec4:7aff:fe88:725b/64 scope link 2023-10-19T12:44:58.890 INFO:teuthology.orchestra.run.smithi179.stdout: valid_lft forever preferred_lft forever 2023-10-19T12:44:58.891 INFO:teuthology.orchestra.run.smithi179.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-19T12:44:58.891 INFO:teuthology.orchestra.run.smithi179.stdout: link/ether 6e:ea:0f:b8:58:54 brd ff:ff:ff:ff:ff:ff 2023-10-19T12:44:58.891 INFO:teuthology.orchestra.run.smithi179.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-19T12:44:58.891 INFO:teuthology.orchestra.run.smithi179.stdout: valid_lft forever preferred_lft forever 2023-10-19T12:44:58.891 INFO:teuthology.orchestra.run.smithi179.stdout: inet6 fe80::f08a:c0ff:fe62:7839/64 scope link 2023-10-19T12:44:58.892 INFO:teuthology.orchestra.run.smithi179.stdout: valid_lft forever preferred_lft forever 2023-10-19T12:44:58.892 INFO:teuthology.orchestra.run.smithi179.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-19T12:44:58.892 INFO:teuthology.orchestra.run.smithi179.stdout: link/ether 6e:ea:0f:b8:58:54 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2023-10-19T12:44:58.892 INFO:teuthology.orchestra.run.smithi179.stdout: inet6 fe80::6cea:fff:feb8:5854/64 scope link tentative 2023-10-19T12:44:58.892 INFO:teuthology.orchestra.run.smithi179.stdout: valid_lft forever preferred_lft forever 2023-10-19T12:44:58.893 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:44:58.893 DEBUG:teuthology.orchestra.run.smithi179:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-19T12:44:58.953 INFO:teuthology.orchestra.run.smithi179.stdout:1 2023-10-19T12:44:58.954 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:44:58.955 DEBUG:teuthology.orchestra.run.smithi179:> route 2023-10-19T12:44:59.012 INFO:teuthology.orchestra.run.smithi179.stdout:Kernel IP routing table 2023-10-19T12:44:59.012 INFO:teuthology.orchestra.run.smithi179.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-19T12:44:59.012 INFO:teuthology.orchestra.run.smithi179.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-10-19T12:44:59.012 INFO:teuthology.orchestra.run.smithi179.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-10-19T12:44:59.013 INFO:teuthology.orchestra.run.smithi179.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-19T12:44:59.013 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:44:59.013 DEBUG:teuthology.orchestra.run.smithi179:> (cd / && exec bash -c ' 2023-10-19T12:44:59.014 DEBUG:teuthology.orchestra.run.smithi179:> set -e 2023-10-19T12:44:59.014 DEBUG:teuthology.orchestra.run.smithi179:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-10-19T12:44:59.014 DEBUG:teuthology.orchestra.run.smithi179:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-10-19T12:44:59.015 DEBUG:teuthology.orchestra.run.smithi179:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-10-19T12:44:59.015 DEBUG:teuthology.orchestra.run.smithi179:> ') 2023-10-19T12:44:59.107 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:44:59.108 DEBUG:teuthology.orchestra.run.smithi179:> ip netns list 2023-10-19T12:44:59.123 INFO:teuthology.orchestra.run.smithi179.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-10-19T12:44:59.123 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:44:59.123 DEBUG:teuthology.orchestra.run.smithi179:> ip netns list-id 2023-10-19T12:44:59.179 INFO:teuthology.orchestra.run.smithi179.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2023-10-19T12:44:59.179 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:44:59.180 DEBUG:teuthology.orchestra.run.smithi179:> (cd / && exec bash -c ' 2023-10-19T12:44:59.180 DEBUG:teuthology.orchestra.run.smithi179:> set -e 2023-10-19T12:44:59.180 DEBUG:teuthology.orchestra.run.smithi179:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2023-10-19T12:44:59.181 DEBUG:teuthology.orchestra.run.smithi179:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2023-10-19T12:44:59.181 DEBUG:teuthology.orchestra.run.smithi179:> ') 2023-10-19T12:44:59.260 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:44:59.261 DEBUG:teuthology.orchestra.run.smithi179:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2023-10-19T12:44:59.327 INFO:teuthology.orchestra.run.smithi179.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-19T12:44:59.327 INFO:teuthology.orchestra.run.smithi179.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-19T12:44:59.327 INFO:teuthology.orchestra.run.smithi179.stdout: inet 127.0.0.1/8 scope host lo 2023-10-19T12:44:59.328 INFO:teuthology.orchestra.run.smithi179.stdout: valid_lft forever preferred_lft forever 2023-10-19T12:44:59.328 INFO:teuthology.orchestra.run.smithi179.stdout: inet6 ::1/128 scope host 2023-10-19T12:44:59.328 INFO:teuthology.orchestra.run.smithi179.stdout: valid_lft forever preferred_lft forever 2023-10-19T12:44:59.328 INFO:teuthology.orchestra.run.smithi179.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-19T12:44:59.328 INFO:teuthology.orchestra.run.smithi179.stdout: link/ether 96:19:11:0b:93:e1 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-10-19T12:44:59.328 INFO:teuthology.orchestra.run.smithi179.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-10-19T12:44:59.329 INFO:teuthology.orchestra.run.smithi179.stdout: valid_lft forever preferred_lft forever 2023-10-19T12:44:59.329 INFO:teuthology.orchestra.run.smithi179.stdout: inet6 fe80::9419:11ff:fe0b:93e1/64 scope link 2023-10-19T12:44:59.329 INFO:teuthology.orchestra.run.smithi179.stdout: valid_lft forever preferred_lft forever 2023-10-19T12:44:59.330 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:44:59.331 DEBUG:teuthology.orchestra.run.smithi179:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2023-10-19T12:44:59.397 INFO:teuthology.orchestra.run.smithi179.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-19T12:44:59.397 INFO:teuthology.orchestra.run.smithi179.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-19T12:44:59.397 INFO:teuthology.orchestra.run.smithi179.stdout: inet 127.0.0.1/8 scope host lo 2023-10-19T12:44:59.397 INFO:teuthology.orchestra.run.smithi179.stdout: valid_lft forever preferred_lft forever 2023-10-19T12:44:59.398 INFO:teuthology.orchestra.run.smithi179.stdout: inet6 ::1/128 scope host 2023-10-19T12:44:59.398 INFO:teuthology.orchestra.run.smithi179.stdout: valid_lft forever preferred_lft forever 2023-10-19T12:44:59.398 INFO:teuthology.orchestra.run.smithi179.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-19T12:44:59.398 INFO:teuthology.orchestra.run.smithi179.stdout: link/ether 96:19:11:0b:93:e1 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-10-19T12:44:59.398 INFO:teuthology.orchestra.run.smithi179.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-10-19T12:44:59.399 INFO:teuthology.orchestra.run.smithi179.stdout: valid_lft forever preferred_lft forever 2023-10-19T12:44:59.399 INFO:teuthology.orchestra.run.smithi179.stdout: inet6 fe80::9419:11ff:fe0b:93e1/64 scope link 2023-10-19T12:44:59.399 INFO:teuthology.orchestra.run.smithi179.stdout: valid_lft forever preferred_lft forever 2023-10-19T12:44:59.400 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2023-10-19T12:44:59.400 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:44:59.401 DEBUG:teuthology.orchestra.run.smithi179:> (cd / && exec bash -c ' 2023-10-19T12:44:59.401 DEBUG:teuthology.orchestra.run.smithi179:> set -e 2023-10-19T12:44:59.401 DEBUG:teuthology.orchestra.run.smithi179:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2023-10-19T12:44:59.402 DEBUG:teuthology.orchestra.run.smithi179:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2023-10-19T12:44:59.402 DEBUG:teuthology.orchestra.run.smithi179:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2023-10-19T12:44:59.402 DEBUG:teuthology.orchestra.run.smithi179:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2023-10-19T12:44:59.402 DEBUG:teuthology.orchestra.run.smithi179:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2023-10-19T12:44:59.403 DEBUG:teuthology.orchestra.run.smithi179:> ') 2023-10-19T12:44:59.524 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:44:59.525 DEBUG:teuthology.orchestra.run.smithi179:> (cd / && exec bash -c ' 2023-10-19T12:44:59.525 DEBUG:teuthology.orchestra.run.smithi179:> set -e 2023-10-19T12:44:59.525 DEBUG:teuthology.orchestra.run.smithi179:> sudo ip link set brx.1 up 2023-10-19T12:44:59.526 DEBUG:teuthology.orchestra.run.smithi179:> sudo ip link set dev brx.1 master ceph-brx 2023-10-19T12:44:59.526 DEBUG:teuthology.orchestra.run.smithi179:> ') 2023-10-19T12:44:59.567 INFO:tasks.cephfs.fuse_mount:Client client.3 config is {} 2023-10-19T12:44:59.567 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-19T12:44:59.567 DEBUG:teuthology.orchestra.run.smithi179:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2023-10-19T12:44:59.625 INFO:teuthology.orchestra.run.smithi179.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2023-10-19T12:44:59.625 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-19T12:44:59.626 DEBUG:teuthology.orchestra.run.smithi179:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2023-10-19T12:44:59.679 DEBUG:teuthology.orchestra.run.smithi179:> sudo modprobe fuse 2023-10-19T12:44:59.746 DEBUG:teuthology.orchestra.run.smithi179:> cat /proc/self/mounts | awk '{print $2}' 2023-10-19T12:44:59.801 INFO:teuthology.orchestra.run.smithi179.stdout:/proc 2023-10-19T12:44:59.802 INFO:teuthology.orchestra.run.smithi179.stdout:/sys 2023-10-19T12:44:59.802 INFO:teuthology.orchestra.run.smithi179.stdout:/dev 2023-10-19T12:44:59.802 INFO:teuthology.orchestra.run.smithi179.stdout:/sys/kernel/security 2023-10-19T12:44:59.802 INFO:teuthology.orchestra.run.smithi179.stdout:/dev/shm 2023-10-19T12:44:59.803 INFO:teuthology.orchestra.run.smithi179.stdout:/dev/pts 2023-10-19T12:44:59.803 INFO:teuthology.orchestra.run.smithi179.stdout:/run 2023-10-19T12:44:59.803 INFO:teuthology.orchestra.run.smithi179.stdout:/sys/fs/cgroup 2023-10-19T12:44:59.803 INFO:teuthology.orchestra.run.smithi179.stdout:/sys/fs/pstore 2023-10-19T12:44:59.803 INFO:teuthology.orchestra.run.smithi179.stdout:/sys/fs/bpf 2023-10-19T12:44:59.804 INFO:teuthology.orchestra.run.smithi179.stdout:/ 2023-10-19T12:44:59.804 INFO:teuthology.orchestra.run.smithi179.stdout:/sys/fs/selinux 2023-10-19T12:44:59.804 INFO:teuthology.orchestra.run.smithi179.stdout:/proc/sys/fs/binfmt_misc 2023-10-19T12:44:59.804 INFO:teuthology.orchestra.run.smithi179.stdout:/dev/hugepages 2023-10-19T12:44:59.804 INFO:teuthology.orchestra.run.smithi179.stdout:/dev/mqueue 2023-10-19T12:44:59.805 INFO:teuthology.orchestra.run.smithi179.stdout:/sys/kernel/debug 2023-10-19T12:44:59.805 INFO:teuthology.orchestra.run.smithi179.stdout:/sys/kernel/tracing 2023-10-19T12:44:59.805 INFO:teuthology.orchestra.run.smithi179.stdout:/sys/fs/fuse/connections 2023-10-19T12:44:59.805 INFO:teuthology.orchestra.run.smithi179.stdout:/sys/kernel/config 2023-10-19T12:44:59.805 INFO:teuthology.orchestra.run.smithi179.stdout:/run/credentials/systemd-sysctl.service 2023-10-19T12:44:59.806 INFO:teuthology.orchestra.run.smithi179.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-10-19T12:44:59.806 INFO:teuthology.orchestra.run.smithi179.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-10-19T12:44:59.806 INFO:teuthology.orchestra.run.smithi179.stdout:/var/lib/nfs/rpc_pipefs 2023-10-19T12:44:59.806 INFO:teuthology.orchestra.run.smithi179.stdout:/run/user/1000 2023-10-19T12:44:59.806 INFO:teuthology.orchestra.run.smithi179.stdout:/var/lib/ceph 2023-10-19T12:44:59.807 INFO:teuthology.orchestra.run.smithi179.stdout:/var/lib/ceph/osd/ceph-4 2023-10-19T12:44:59.807 INFO:teuthology.orchestra.run.smithi179.stdout:/var/lib/ceph/osd/ceph-5 2023-10-19T12:44:59.807 INFO:teuthology.orchestra.run.smithi179.stdout:/var/lib/ceph/osd/ceph-6 2023-10-19T12:44:59.807 INFO:teuthology.orchestra.run.smithi179.stdout:/var/lib/ceph/osd/ceph-7 2023-10-19T12:44:59.807 INFO:teuthology.orchestra.run.smithi179.stdout:/run/netns 2023-10-19T12:44:59.808 INFO:teuthology.orchestra.run.smithi179.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2023-10-19T12:44:59.808 INFO:teuthology.orchestra.run.smithi179.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2023-10-19T12:44:59.808 INFO:teuthology.orchestra.run.smithi179.stdout:/home/ubuntu/cephtest/mnt.1 2023-10-19T12:44:59.808 INFO:teuthology.orchestra.run.smithi179.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2023-10-19T12:44:59.808 INFO:teuthology.orchestra.run.smithi179.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2023-10-19T12:44:59.809 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:44:59.809 DEBUG:teuthology.orchestra.run.smithi179:> ls /sys/fs/fuse/connections 2023-10-19T12:44:59.856 INFO:teuthology.orchestra.run.smithi179.stdout:43 2023-10-19T12:44:59.857 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [43] 2023-10-19T12:44:59.857 DEBUG:teuthology.orchestra.run.smithi179:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.3 --id 3) 2023-10-19T12:44:59.899 DEBUG:teuthology.orchestra.run.smithi179:> sudo modprobe fuse 2023-10-19T12:44:59.926 DEBUG:teuthology.orchestra.run.smithi179:> cat /proc/self/mounts | awk '{print $2}' 2023-10-19T12:44:59.981 INFO:teuthology.orchestra.run.smithi179.stdout:/proc 2023-10-19T12:44:59.981 INFO:teuthology.orchestra.run.smithi179.stdout:/sys 2023-10-19T12:44:59.981 INFO:teuthology.orchestra.run.smithi179.stdout:/dev 2023-10-19T12:44:59.982 INFO:teuthology.orchestra.run.smithi179.stdout:/sys/kernel/security 2023-10-19T12:44:59.982 INFO:teuthology.orchestra.run.smithi179.stdout:/dev/shm 2023-10-19T12:44:59.982 INFO:teuthology.orchestra.run.smithi179.stdout:/dev/pts 2023-10-19T12:44:59.982 INFO:teuthology.orchestra.run.smithi179.stdout:/run 2023-10-19T12:44:59.983 INFO:teuthology.orchestra.run.smithi179.stdout:/sys/fs/cgroup 2023-10-19T12:44:59.983 INFO:teuthology.orchestra.run.smithi179.stdout:/sys/fs/pstore 2023-10-19T12:44:59.983 INFO:teuthology.orchestra.run.smithi179.stdout:/sys/fs/bpf 2023-10-19T12:44:59.983 INFO:teuthology.orchestra.run.smithi179.stdout:/ 2023-10-19T12:44:59.983 INFO:teuthology.orchestra.run.smithi179.stdout:/sys/fs/selinux 2023-10-19T12:44:59.983 INFO:teuthology.orchestra.run.smithi179.stdout:/proc/sys/fs/binfmt_misc 2023-10-19T12:44:59.984 INFO:teuthology.orchestra.run.smithi179.stdout:/dev/hugepages 2023-10-19T12:44:59.984 INFO:teuthology.orchestra.run.smithi179.stdout:/dev/mqueue 2023-10-19T12:44:59.984 INFO:teuthology.orchestra.run.smithi179.stdout:/sys/kernel/debug 2023-10-19T12:44:59.984 INFO:teuthology.orchestra.run.smithi179.stdout:/sys/kernel/tracing 2023-10-19T12:44:59.984 INFO:teuthology.orchestra.run.smithi179.stdout:/sys/fs/fuse/connections 2023-10-19T12:44:59.985 INFO:teuthology.orchestra.run.smithi179.stdout:/sys/kernel/config 2023-10-19T12:44:59.985 INFO:teuthology.orchestra.run.smithi179.stdout:/run/credentials/systemd-sysctl.service 2023-10-19T12:44:59.985 INFO:teuthology.orchestra.run.smithi179.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-10-19T12:44:59.985 INFO:teuthology.orchestra.run.smithi179.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-10-19T12:44:59.985 INFO:teuthology.orchestra.run.smithi179.stdout:/var/lib/nfs/rpc_pipefs 2023-10-19T12:44:59.985 INFO:teuthology.orchestra.run.smithi179.stdout:/run/user/1000 2023-10-19T12:44:59.986 INFO:teuthology.orchestra.run.smithi179.stdout:/var/lib/ceph 2023-10-19T12:44:59.986 INFO:teuthology.orchestra.run.smithi179.stdout:/var/lib/ceph/osd/ceph-4 2023-10-19T12:44:59.986 INFO:teuthology.orchestra.run.smithi179.stdout:/var/lib/ceph/osd/ceph-5 2023-10-19T12:44:59.986 INFO:teuthology.orchestra.run.smithi179.stdout:/var/lib/ceph/osd/ceph-6 2023-10-19T12:44:59.986 INFO:teuthology.orchestra.run.smithi179.stdout:/var/lib/ceph/osd/ceph-7 2023-10-19T12:44:59.987 INFO:teuthology.orchestra.run.smithi179.stdout:/run/netns 2023-10-19T12:44:59.987 INFO:teuthology.orchestra.run.smithi179.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2023-10-19T12:44:59.987 INFO:teuthology.orchestra.run.smithi179.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2023-10-19T12:44:59.987 INFO:teuthology.orchestra.run.smithi179.stdout:/home/ubuntu/cephtest/mnt.1 2023-10-19T12:44:59.987 INFO:teuthology.orchestra.run.smithi179.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2023-10-19T12:44:59.988 INFO:teuthology.orchestra.run.smithi179.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2023-10-19T12:44:59.988 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:44:59.988 DEBUG:teuthology.orchestra.run.smithi179:> ls /sys/fs/fuse/connections 2023-10-19T12:44:59.990 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi179.stderr:2023-10-19T12:44:59.979+0000 7fe87fa6f480 -1 init, newargv = 0x55f565041a70 newargc=15 2023-10-19T12:44:59.990 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi179.stderr:ceph-fuse[94778]: starting ceph client 2023-10-19T12:44:59.990 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi179.stderr:ceph-fuse[94778]: starting fuse 2023-10-19T12:45:00.004 INFO:teuthology.orchestra.run.smithi179.stdout:43 2023-10-19T12:45:00.004 INFO:teuthology.orchestra.run.smithi179.stdout:44 2023-10-19T12:45:00.004 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43, 44] 2023-10-19T12:45:00.005 DEBUG:teuthology.orchestra.run.smithi179:> sudo stdin-killer -- python3 -c ' 2023-10-19T12:45:00.005 DEBUG:teuthology.orchestra.run.smithi179:> import glob 2023-10-19T12:45:00.005 DEBUG:teuthology.orchestra.run.smithi179:> import re 2023-10-19T12:45:00.006 DEBUG:teuthology.orchestra.run.smithi179:> import os 2023-10-19T12:45:00.006 DEBUG:teuthology.orchestra.run.smithi179:> import subprocess 2023-10-19T12:45:00.006 DEBUG:teuthology.orchestra.run.smithi179:> 2023-10-19T12:45:00.006 DEBUG:teuthology.orchestra.run.smithi179:> def _find_admin_socket(client_name): 2023-10-19T12:45:00.007 DEBUG:teuthology.orchestra.run.smithi179:> asok_path = "/var/run/ceph/ceph-client.3.*.asok" 2023-10-19T12:45:00.007 DEBUG:teuthology.orchestra.run.smithi179:> files = glob.glob(asok_path) 2023-10-19T12:45:00.007 DEBUG:teuthology.orchestra.run.smithi179:> mountpoint = "/home/ubuntu/cephtest/mnt.3" 2023-10-19T12:45:00.008 DEBUG:teuthology.orchestra.run.smithi179:> 2023-10-19T12:45:00.008 DEBUG:teuthology.orchestra.run.smithi179:> # Given a non-glob path, it better be there 2023-10-19T12:45:00.008 DEBUG:teuthology.orchestra.run.smithi179:> if "*" not in asok_path: 2023-10-19T12:45:00.008 DEBUG:teuthology.orchestra.run.smithi179:> assert(len(files) == 1) 2023-10-19T12:45:00.009 DEBUG:teuthology.orchestra.run.smithi179:> return files[0] 2023-10-19T12:45:00.009 DEBUG:teuthology.orchestra.run.smithi179:> 2023-10-19T12:45:00.009 DEBUG:teuthology.orchestra.run.smithi179:> for f in files: 2023-10-19T12:45:00.010 DEBUG:teuthology.orchestra.run.smithi179:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-10-19T12:45:00.010 DEBUG:teuthology.orchestra.run.smithi179:> if os.path.exists("/proc/{0}".format(pid)): 2023-10-19T12:45:00.010 DEBUG:teuthology.orchestra.run.smithi179:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-10-19T12:45:00.011 DEBUG:teuthology.orchestra.run.smithi179:> contents = proc_f.read() 2023-10-19T12:45:00.011 DEBUG:teuthology.orchestra.run.smithi179:> if mountpoint in contents: 2023-10-19T12:45:00.011 DEBUG:teuthology.orchestra.run.smithi179:> return f 2023-10-19T12:45:00.011 DEBUG:teuthology.orchestra.run.smithi179:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-10-19T12:45:00.012 DEBUG:teuthology.orchestra.run.smithi179:> 2023-10-19T12:45:00.012 DEBUG:teuthology.orchestra.run.smithi179:> print(_find_admin_socket("client.3")) 2023-10-19T12:45:00.012 DEBUG:teuthology.orchestra.run.smithi179:> ' 2023-10-19T12:45:00.123 INFO:teuthology.orchestra.run.smithi179.stdout:/var/run/ceph/ceph-client.3.94778.asok 2023-10-19T12:45:00.126 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-19T12:45:00 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2023-10-19T12:45:00.134 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.3.94778.asok 2023-10-19T12:45:00.134 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:45:00.135 DEBUG:teuthology.orchestra.run.smithi179:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.3.94778.asok status 2023-10-19T12:45:00.269 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2023-10-19T12:45:00.269 INFO:teuthology.orchestra.run.smithi179.stdout: "metadata": { 2023-10-19T12:45:00.269 INFO:teuthology.orchestra.run.smithi179.stdout: "ceph_sha1": "f7420d7f8e55ceea786dcbf536c338ce920c21eb", 2023-10-19T12:45:00.269 INFO:teuthology.orchestra.run.smithi179.stdout: "ceph_version": "ceph version 18.0.0-6786-gf7420d7f (f7420d7f8e55ceea786dcbf536c338ce920c21eb) reef (dev)", 2023-10-19T12:45:00.270 INFO:teuthology.orchestra.run.smithi179.stdout: "entity_id": "3", 2023-10-19T12:45:00.270 INFO:teuthology.orchestra.run.smithi179.stdout: "hostname": "smithi179", 2023-10-19T12:45:00.270 INFO:teuthology.orchestra.run.smithi179.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.3", 2023-10-19T12:45:00.270 INFO:teuthology.orchestra.run.smithi179.stdout: "pid": "94778", 2023-10-19T12:45:00.270 INFO:teuthology.orchestra.run.smithi179.stdout: "root": "/" 2023-10-19T12:45:00.271 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2023-10-19T12:45:00.271 INFO:teuthology.orchestra.run.smithi179.stdout: "dentry_count": 0, 2023-10-19T12:45:00.271 INFO:teuthology.orchestra.run.smithi179.stdout: "dentry_pinned_count": 0, 2023-10-19T12:45:00.271 INFO:teuthology.orchestra.run.smithi179.stdout: "id": 4822, 2023-10-19T12:45:00.271 INFO:teuthology.orchestra.run.smithi179.stdout: "inst": { 2023-10-19T12:45:00.272 INFO:teuthology.orchestra.run.smithi179.stdout: "name": { 2023-10-19T12:45:00.272 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "client", 2023-10-19T12:45:00.272 INFO:teuthology.orchestra.run.smithi179.stdout: "num": 4822 2023-10-19T12:45:00.272 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2023-10-19T12:45:00.272 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": { 2023-10-19T12:45:00.272 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v1", 2023-10-19T12:45:00.273 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "192.168.0.2:0", 2023-10-19T12:45:00.273 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 145451260 2023-10-19T12:45:00.273 INFO:teuthology.orchestra.run.smithi179.stdout: } 2023-10-19T12:45:00.273 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2023-10-19T12:45:00.273 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": { 2023-10-19T12:45:00.273 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v1", 2023-10-19T12:45:00.274 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "192.168.0.2:0", 2023-10-19T12:45:00.274 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 145451260 2023-10-19T12:45:00.274 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2023-10-19T12:45:00.274 INFO:teuthology.orchestra.run.smithi179.stdout: "inst_str": "client.4822 192.168.0.2:0/145451260", 2023-10-19T12:45:00.274 INFO:teuthology.orchestra.run.smithi179.stdout: "addr_str": "192.168.0.2:0/145451260", 2023-10-19T12:45:00.275 INFO:teuthology.orchestra.run.smithi179.stdout: "inode_count": 1, 2023-10-19T12:45:00.275 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_epoch": 9, 2023-10-19T12:45:00.275 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_epoch": 27, 2023-10-19T12:45:00.275 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_epoch_barrier": 0, 2023-10-19T12:45:00.275 INFO:teuthology.orchestra.run.smithi179.stdout: "blocklisted": false, 2023-10-19T12:45:00.276 INFO:teuthology.orchestra.run.smithi179.stdout: "fs_name": "cephfs" 2023-10-19T12:45:00.276 INFO:teuthology.orchestra.run.smithi179.stdout:} 2023-10-19T12:45:00.277 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:45:00.277 DEBUG:teuthology.orchestra.run.smithi057:> stat --file-system '--printf=%T 2023-10-19T12:45:00.277 DEBUG:teuthology.orchestra.run.smithi057:> ' -- /home/ubuntu/cephtest/mnt.0 2023-10-19T12:45:00.297 INFO:teuthology.orchestra.run.smithi057.stdout:fuseblk 2023-10-19T12:45:00.297 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-10-19T12:45:00.297 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:45:00.298 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-19T12:45:00.373 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:45:00.373 DEBUG:teuthology.orchestra.run.smithi057:> stat --file-system '--printf=%T 2023-10-19T12:45:00.374 DEBUG:teuthology.orchestra.run.smithi057:> ' -- /home/ubuntu/cephtest/mnt.2 2023-10-19T12:45:00.432 INFO:teuthology.orchestra.run.smithi057.stdout:fuseblk 2023-10-19T12:45:00.432 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.2 2023-10-19T12:45:00.433 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:45:00.433 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2023-10-19T12:45:00.507 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:45:00.507 DEBUG:teuthology.orchestra.run.smithi179:> stat --file-system '--printf=%T 2023-10-19T12:45:00.507 DEBUG:teuthology.orchestra.run.smithi179:> ' -- /home/ubuntu/cephtest/mnt.1 2023-10-19T12:45:00.523 INFO:teuthology.orchestra.run.smithi179.stdout:fuseblk 2023-10-19T12:45:00.523 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2023-10-19T12:45:00.523 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:45:00.524 DEBUG:teuthology.orchestra.run.smithi179:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2023-10-19T12:45:00.590 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:45:00.590 DEBUG:teuthology.orchestra.run.smithi179:> stat --file-system '--printf=%T 2023-10-19T12:45:00.591 DEBUG:teuthology.orchestra.run.smithi179:> ' -- /home/ubuntu/cephtest/mnt.3 2023-10-19T12:45:00.644 INFO:teuthology.orchestra.run.smithi179.stdout:fuseblk 2023-10-19T12:45:00.645 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.3 2023-10-19T12:45:00.645 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:45:00.646 DEBUG:teuthology.orchestra.run.smithi179:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2023-10-19T12:45:00.711 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2023-10-19T12:45:00.727 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T12:45:00.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:00.846+0000 7f04cb597640 1 -- 172.21.15.57:0/2753776133 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f04c40cf9e0 msgr2=0x7f04c40d1810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:00.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:00.846+0000 7f04cb597640 1 --2- 172.21.15.57:0/2753776133 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f04c40cf9e0 0x7f04c40d1810 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f04b4005fd0 tx=0x7f04b402bd90 comp rx=0 tx=0).stop 2023-10-19T12:45:00.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:00.846+0000 7f04cb597640 1 -- 172.21.15.57:0/2753776133 shutdown_connections 2023-10-19T12:45:00.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:00.846+0000 7f04cb597640 1 --2- 172.21.15.57:0/2753776133 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f04c415b3e0 0x7f04c40b0c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:00.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:00.846+0000 7f04cb597640 1 --2- 172.21.15.57:0/2753776133 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f04c40cf9e0 0x7f04c40d1810 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:00.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:00.846+0000 7f04cb597640 1 -- 172.21.15.57:0/2753776133 >> 172.21.15.57:0/2753776133 conn(0x7f04c40d0cf0 msgr2=0x7f04c40d1100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:00.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:00.846+0000 7f04cb597640 1 -- 172.21.15.57:0/2753776133 shutdown_connections 2023-10-19T12:45:00.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:00.846+0000 7f04cb597640 1 -- 172.21.15.57:0/2753776133 wait complete. 2023-10-19T12:45:00.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:00.847+0000 7f04cb597640 1 Processor -- start 2023-10-19T12:45:00.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:00.847+0000 7f04cb597640 1 -- start start 2023-10-19T12:45:00.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:00.848+0000 7f04cb597640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f04c40cf9e0 0x7f04c41f2240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:00.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:00.848+0000 7f04cb597640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f04c415b000 0x7f04c41eec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:00.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:00.848+0000 7f04cb597640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f04c415b3e0 0x7f04c40c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:00.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:00.848+0000 7f04cb597640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04c41f2820 con 0x7f04c415b000 2023-10-19T12:45:00.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:00.848+0000 7f04cb597640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04c40b61b0 con 0x7f04c415b3e0 2023-10-19T12:45:00.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:00.848+0000 7f04cb597640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04c40b6490 con 0x7f04c40cf9e0 2023-10-19T12:45:00.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:00.848+0000 7f04c8b32640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f04c415b000 0x7f04c41eec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:00.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:00.848+0000 7f04c8b32640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f04c415b000 0x7f04c41eec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50284/0 (socket says 172.21.15.57:50284) 2023-10-19T12:45:00.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:00.848+0000 7f04c8b32640 1 -- 172.21.15.57:0/1540276278 learned_addr learned my addr 172.21.15.57:0/1540276278 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:00.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:00.848+0000 7f04c9b34640 1 --2- 172.21.15.57:0/1540276278 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f04c415b3e0 0x7f04c40c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:00.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:00.848+0000 7f04c9333640 1 --2- 172.21.15.57:0/1540276278 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f04c40cf9e0 0x7f04c41f2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:00.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:00.848+0000 7f04c8b32640 1 -- 172.21.15.57:0/1540276278 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f04c40cf9e0 msgr2=0x7f04c41f2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:00.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:00.848+0000 7f04c8b32640 1 --2- 172.21.15.57:0/1540276278 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f04c40cf9e0 0x7f04c41f2240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:00.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:00.848+0000 7f04c8b32640 1 -- 172.21.15.57:0/1540276278 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f04c415b3e0 msgr2=0x7f04c40c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:00.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:00.848+0000 7f04c8b32640 1 --2- 172.21.15.57:0/1540276278 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f04c415b3e0 0x7f04c40c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:00.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:00.849+0000 7f04c8b32640 1 -- 172.21.15.57:0/1540276278 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04b4009560 con 0x7f04c415b000 2023-10-19T12:45:00.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:00.849+0000 7f04c8b32640 1 --2- 172.21.15.57:0/1540276278 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f04c415b000 0x7f04c41eec40 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f04b4036040 tx=0x7f04b402dd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:00.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:00.849+0000 7f04b9ffb640 1 -- 172.21.15.57:0/1540276278 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04b4005650 con 0x7f04c415b000 2023-10-19T12:45:00.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:00.849+0000 7f04b9ffb640 1 -- 172.21.15.57:0/1540276278 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f04b40038f0 con 0x7f04c415b000 2023-10-19T12:45:00.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:00.849+0000 7f04cb597640 1 -- 172.21.15.57:0/1540276278 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04c41f2b20 con 0x7f04c415b000 2023-10-19T12:45:00.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:00.849+0000 7f04b9ffb640 1 -- 172.21.15.57:0/1540276278 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04b4003ca0 con 0x7f04c415b000 2023-10-19T12:45:00.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:00.849+0000 7f04cb597640 1 -- 172.21.15.57:0/1540276278 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04c409ab20 con 0x7f04c415b000 2023-10-19T12:45:00.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:00.850+0000 7f04cb597640 1 -- 172.21.15.57:0/1540276278 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04c41f2980 con 0x7f04c415b000 2023-10-19T12:45:00.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:00.851+0000 7f04b9ffb640 1 -- 172.21.15.57:0/1540276278 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f04b403e450 con 0x7f04c415b000 2023-10-19T12:45:00.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:00.851+0000 7f04b9ffb640 1 --2- 172.21.15.57:0/1540276278 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f04a806ae40 0x7f04a806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:00.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:00.851+0000 7f04b9ffb640 1 -- 172.21.15.57:0/1540276278 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f04b40ac4d0 con 0x7f04c415b000 2023-10-19T12:45:00.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:00.851+0000 7f04c9333640 1 --2- 172.21.15.57:0/1540276278 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f04a806ae40 0x7f04a806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:00.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:00.852+0000 7f04c9333640 1 --2- 172.21.15.57:0/1540276278 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f04a806ae40 0x7f04a806d300 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f04c0004f60 tx=0x7f04c0007e30 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:00.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:00.855+0000 7f04b9ffb640 1 -- 172.21.15.57:0/1540276278 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f04c41f2980 con 0x7f04c415b000 2023-10-19T12:45:01.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.032+0000 7f04cb597640 1 -- 172.21.15.57:0/1540276278 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f04c41f2980 con 0x7f04c415b000 2023-10-19T12:45:01.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.033+0000 7f04b9ffb640 1 -- 172.21.15.57:0/1540276278 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18748 (secure 0 0 0) 0x7f04c41f2980 con 0x7f04c415b000 2023-10-19T12:45:01.036 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:45:01.036 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":27,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T12:44:58.978403+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-19T12:44:16.517510+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-19T12:44:17.398297+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-19T12:44:21.129403+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.23","mappings":[{"from":3,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"3.39","mappings":[{"from":7,"to":1}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.1d","mappings":[{"from":1,"to":5}]},{"pgid":"4.1e","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T12:45:01.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.035+0000 7f04cb597640 1 -- 172.21.15.57:0/1540276278 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f04a806ae40 msgr2=0x7f04a806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:01.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.035+0000 7f04cb597640 1 --2- 172.21.15.57:0/1540276278 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f04a806ae40 0x7f04a806d300 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f04c0004f60 tx=0x7f04c0007e30 comp rx=0 tx=0).stop 2023-10-19T12:45:01.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.035+0000 7f04cb597640 1 -- 172.21.15.57:0/1540276278 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f04c415b000 msgr2=0x7f04c41eec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:01.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.035+0000 7f04cb597640 1 --2- 172.21.15.57:0/1540276278 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f04c415b000 0x7f04c41eec40 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f04b4036040 tx=0x7f04b402dd60 comp rx=0 tx=0).stop 2023-10-19T12:45:01.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.035+0000 7f04cb597640 1 -- 172.21.15.57:0/1540276278 shutdown_connections 2023-10-19T12:45:01.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.035+0000 7f04cb597640 1 --2- 172.21.15.57:0/1540276278 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f04a806ae40 0x7f04a806d300 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:01.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.035+0000 7f04cb597640 1 --2- 172.21.15.57:0/1540276278 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f04c415b3e0 0x7f04c40c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:01.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.035+0000 7f04cb597640 1 --2- 172.21.15.57:0/1540276278 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f04c415b000 0x7f04c41eec40 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:01.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.035+0000 7f04cb597640 1 --2- 172.21.15.57:0/1540276278 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f04c40cf9e0 0x7f04c41f2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:01.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.035+0000 7f04cb597640 1 -- 172.21.15.57:0/1540276278 >> 172.21.15.57:0/1540276278 conn(0x7f04c40d0cf0 msgr2=0x7f04c40d1470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:01.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.035+0000 7f04cb597640 1 -- 172.21.15.57:0/1540276278 shutdown_connections 2023-10-19T12:45:01.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.035+0000 7f04cb597640 1 -- 172.21.15.57:0/1540276278 wait complete. 2023-10-19T12:45:01.048 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-19T12:44:07.371186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-19T12:44:16.517510+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-19T12:44:17.398297+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-19T12:44:21.129403+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-19T12:45:01.048 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-19T12:45:01.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.158+0000 7f02f79c0640 1 -- 172.21.15.57:0/497109388 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f02f0069e30 msgr2=0x7f02f00cdb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:01.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.158+0000 7f02f79c0640 1 --2- 172.21.15.57:0/497109388 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f02f0069e30 0x7f02f00cdb30 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f02e0005fd0 tx=0x7f02e002c0e0 comp rx=0 tx=0).stop 2023-10-19T12:45:01.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.158+0000 7f02f79c0640 1 -- 172.21.15.57:0/497109388 shutdown_connections 2023-10-19T12:45:01.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.158+0000 7f02f79c0640 1 --2- 172.21.15.57:0/497109388 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f02f0069e30 0x7f02f00cdb30 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:01.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.158+0000 7f02f79c0640 1 -- 172.21.15.57:0/497109388 >> 172.21.15.57:0/497109388 conn(0x7f02f00af7a0 msgr2=0x7f02f00afb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:01.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.158+0000 7f02f79c0640 1 -- 172.21.15.57:0/497109388 shutdown_connections 2023-10-19T12:45:01.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.158+0000 7f02f79c0640 1 -- 172.21.15.57:0/497109388 wait complete. 2023-10-19T12:45:01.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.159+0000 7f02f79c0640 1 Processor -- start 2023-10-19T12:45:01.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.159+0000 7f02f79c0640 1 -- start start 2023-10-19T12:45:01.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.160+0000 7f02f79c0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f02f0069e30 0x7f02f014c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:01.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.160+0000 7f02f79c0640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f02f00cd6d0 0x7f02f01eec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:01.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.160+0000 7f02f79c0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f02f0142850 0x7f02f0070110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:01.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.160+0000 7f02f79c0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02f014cea0 con 0x7f02f0142850 2023-10-19T12:45:01.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.160+0000 7f02f79c0640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02f01f2c90 con 0x7f02f0069e30 2023-10-19T12:45:01.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.160+0000 7f02f79c0640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02f01f2f70 con 0x7f02f00cd6d0 2023-10-19T12:45:01.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.160+0000 7f02f5f5d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f02f0142850 0x7f02f0070110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:01.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.160+0000 7f02f5f5d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f02f0142850 0x7f02f0070110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50298/0 (socket says 172.21.15.57:50298) 2023-10-19T12:45:01.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.160+0000 7f02f5f5d640 1 -- 172.21.15.57:0/3763000919 learned_addr learned my addr 172.21.15.57:0/3763000919 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:01.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.160+0000 7f02f575c640 1 --2- 172.21.15.57:0/3763000919 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f02f0069e30 0x7f02f014c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:01.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.160+0000 7f02f4f5b640 1 --2- 172.21.15.57:0/3763000919 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f02f00cd6d0 0x7f02f01eec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:01.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.160+0000 7f02f5f5d640 1 -- 172.21.15.57:0/3763000919 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f02f00cd6d0 msgr2=0x7f02f01eec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:01.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.160+0000 7f02f5f5d640 1 --2- 172.21.15.57:0/3763000919 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f02f00cd6d0 0x7f02f01eec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:01.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.160+0000 7f02f5f5d640 1 -- 172.21.15.57:0/3763000919 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f02f0069e30 msgr2=0x7f02f014c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:01.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.160+0000 7f02f5f5d640 1 --2- 172.21.15.57:0/3763000919 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f02f0069e30 0x7f02f014c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:01.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.160+0000 7f02f5f5d640 1 -- 172.21.15.57:0/3763000919 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02e0009840 con 0x7f02f0142850 2023-10-19T12:45:01.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.160+0000 7f02f575c640 1 --2- 172.21.15.57:0/3763000919 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f02f0069e30 0x7f02f014c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:45:01.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.160+0000 7f02f4f5b640 1 --2- 172.21.15.57:0/3763000919 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f02f00cd6d0 0x7f02f01eec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:45:01.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.161+0000 7f02f5f5d640 1 --2- 172.21.15.57:0/3763000919 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f02f0142850 0x7f02f0070110 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f02e0005fd0 tx=0x7f02e0004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:01.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.161+0000 7f02edffb640 1 -- 172.21.15.57:0/3763000919 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02e0004820 con 0x7f02f0142850 2023-10-19T12:45:01.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.161+0000 7f02edffb640 1 -- 172.21.15.57:0/3763000919 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f02e00346b0 con 0x7f02f0142850 2023-10-19T12:45:01.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.161+0000 7f02f79c0640 1 -- 172.21.15.57:0/3763000919 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02f01f3250 con 0x7f02f0142850 2023-10-19T12:45:01.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.161+0000 7f02edffb640 1 -- 172.21.15.57:0/3763000919 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02e0034ac0 con 0x7f02f0142850 2023-10-19T12:45:01.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.161+0000 7f02f79c0640 1 -- 172.21.15.57:0/3763000919 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02f01f3810 con 0x7f02f0142850 2023-10-19T12:45:01.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.162+0000 7f02f79c0640 1 -- 172.21.15.57:0/3763000919 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02f00690e0 con 0x7f02f0142850 2023-10-19T12:45:01.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.162+0000 7f02edffb640 1 -- 172.21.15.57:0/3763000919 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f02e0034c20 con 0x7f02f0142850 2023-10-19T12:45:01.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.163+0000 7f02edffb640 1 --2- 172.21.15.57:0/3763000919 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f02dc06ae40 0x7f02dc06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:01.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.163+0000 7f02edffb640 1 -- 172.21.15.57:0/3763000919 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f02e00accc0 con 0x7f02f0142850 2023-10-19T12:45:01.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.163+0000 7f02f575c640 1 --2- 172.21.15.57:0/3763000919 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f02dc06ae40 0x7f02dc06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:01.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.164+0000 7f02f575c640 1 --2- 172.21.15.57:0/3763000919 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f02dc06ae40 0x7f02dc06d300 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f02e8000f60 tx=0x7f02e8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:01.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.167+0000 7f02edffb640 1 -- 172.21.15.57:0/3763000919 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f02e00b3080 con 0x7f02f0142850 2023-10-19T12:45:01.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.344+0000 7f02f79c0640 1 -- 172.21.15.57:0/3763000919 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f02f01417f0 con 0x7f02f0142850 2023-10-19T12:45:01.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.344+0000 7f02edffb640 1 -- 172.21.15.57:0/3763000919 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7f02e00430a0 con 0x7f02f0142850 2023-10-19T12:45:01.346 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-19T12:45:01.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.346+0000 7f02f79c0640 1 -- 172.21.15.57:0/3763000919 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f02dc06ae40 msgr2=0x7f02dc06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:01.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.346+0000 7f02f79c0640 1 --2- 172.21.15.57:0/3763000919 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f02dc06ae40 0x7f02dc06d300 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f02e8000f60 tx=0x7f02e8009040 comp rx=0 tx=0).stop 2023-10-19T12:45:01.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.346+0000 7f02f79c0640 1 -- 172.21.15.57:0/3763000919 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f02f0142850 msgr2=0x7f02f0070110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:01.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.346+0000 7f02f79c0640 1 --2- 172.21.15.57:0/3763000919 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f02f0142850 0x7f02f0070110 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f02e0005fd0 tx=0x7f02e0004360 comp rx=0 tx=0).stop 2023-10-19T12:45:01.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.346+0000 7f02f79c0640 1 -- 172.21.15.57:0/3763000919 shutdown_connections 2023-10-19T12:45:01.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.346+0000 7f02f79c0640 1 --2- 172.21.15.57:0/3763000919 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f02dc06ae40 0x7f02dc06d300 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:01.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.346+0000 7f02f79c0640 1 --2- 172.21.15.57:0/3763000919 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f02f0142850 0x7f02f0070110 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:01.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.346+0000 7f02f79c0640 1 --2- 172.21.15.57:0/3763000919 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f02f00cd6d0 0x7f02f01eec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:01.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.346+0000 7f02f79c0640 1 --2- 172.21.15.57:0/3763000919 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f02f0069e30 0x7f02f014c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:01.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.346+0000 7f02f79c0640 1 -- 172.21.15.57:0/3763000919 >> 172.21.15.57:0/3763000919 conn(0x7f02f00af7a0 msgr2=0x7f02f01405b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:01.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.346+0000 7f02f79c0640 1 -- 172.21.15.57:0/3763000919 shutdown_connections 2023-10-19T12:45:01.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.346+0000 7f02f79c0640 1 -- 172.21.15.57:0/3763000919 wait complete. 2023-10-19T12:45:01.359 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-19T12:45:01.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.468+0000 7fcaedd25640 1 -- 172.21.15.57:0/3125612819 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcae80abc10 msgr2=0x7fcae80c9b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:01.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.468+0000 7fcaedd25640 1 --2- 172.21.15.57:0/3125612819 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcae80abc10 0x7fcae80c9b30 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fcae0009ca0 tx=0x7fcae002c2e0 comp rx=0 tx=0).stop 2023-10-19T12:45:01.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.469+0000 7fcaedd25640 1 -- 172.21.15.57:0/3125612819 shutdown_connections 2023-10-19T12:45:01.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.469+0000 7fcaedd25640 1 --2- 172.21.15.57:0/3125612819 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcae80abc10 0x7fcae80c9b30 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:01.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.469+0000 7fcaedd25640 1 --2- 172.21.15.57:0/3125612819 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcae806af00 0x7fcae806b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:01.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.469+0000 7fcaedd25640 1 -- 172.21.15.57:0/3125612819 >> 172.21.15.57:0/3125612819 conn(0x7fcae8069930 msgr2=0x7fcae8069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:01.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.469+0000 7fcaedd25640 1 -- 172.21.15.57:0/3125612819 shutdown_connections 2023-10-19T12:45:01.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.469+0000 7fcaedd25640 1 -- 172.21.15.57:0/3125612819 wait complete. 2023-10-19T12:45:01.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.470+0000 7fcaedd25640 1 Processor -- start 2023-10-19T12:45:01.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.470+0000 7fcaedd25640 1 -- start start 2023-10-19T12:45:01.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.471+0000 7fcaedd25640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcae806af00 0x7fcae8158bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:01.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.471+0000 7fcaedd25640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcae80abc10 0x7fcae81f4a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:01.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.471+0000 7fcaedd25640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcae80c96b0 0x7fcae813d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:01.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.471+0000 7fcaedd25640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcae81f1770 con 0x7fcae80c96b0 2023-10-19T12:45:01.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.471+0000 7fcaedd25640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcae81f4f90 con 0x7fcae80abc10 2023-10-19T12:45:01.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.471+0000 7fcaedd25640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcae81f1410 con 0x7fcae806af00 2023-10-19T12:45:01.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.471+0000 7fcae7fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcae80c96b0 0x7fcae813d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:01.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.471+0000 7fcae7fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcae80c96b0 0x7fcae813d7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50316/0 (socket says 172.21.15.57:50316) 2023-10-19T12:45:01.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.471+0000 7fcae7fff640 1 -- 172.21.15.57:0/3438911280 learned_addr learned my addr 172.21.15.57:0/3438911280 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:01.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.471+0000 7fcae77fe640 1 --2- 172.21.15.57:0/3438911280 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcae806af00 0x7fcae8158bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:01.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.471+0000 7fcae6ffd640 1 --2- 172.21.15.57:0/3438911280 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcae80abc10 0x7fcae81f4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:01.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.471+0000 7fcae7fff640 1 -- 172.21.15.57:0/3438911280 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcae806af00 msgr2=0x7fcae8158bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:01.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.471+0000 7fcae7fff640 1 --2- 172.21.15.57:0/3438911280 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcae806af00 0x7fcae8158bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:01.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.471+0000 7fcae7fff640 1 -- 172.21.15.57:0/3438911280 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcae80abc10 msgr2=0x7fcae81f4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:01.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.471+0000 7fcae7fff640 1 --2- 172.21.15.57:0/3438911280 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcae80abc10 0x7fcae81f4a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:01.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.471+0000 7fcae7fff640 1 -- 172.21.15.57:0/3438911280 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcae0009900 con 0x7fcae80c96b0 2023-10-19T12:45:01.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.471+0000 7fcae7fff640 1 --2- 172.21.15.57:0/3438911280 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcae80c96b0 0x7fcae813d7a0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fcae0009bf0 tx=0x7fcae0035ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:01.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.471+0000 7fcacffff640 1 -- 172.21.15.57:0/3438911280 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcae0004240 con 0x7fcae80c96b0 2023-10-19T12:45:01.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.471+0000 7fcacffff640 1 -- 172.21.15.57:0/3438911280 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fcae0004490 con 0x7fcae80c96b0 2023-10-19T12:45:01.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.471+0000 7fcacffff640 1 -- 172.21.15.57:0/3438911280 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcae00048a0 con 0x7fcae80c96b0 2023-10-19T12:45:01.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.471+0000 7fcaedd25640 1 -- 172.21.15.57:0/3438911280 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcae81f51d0 con 0x7fcae80c96b0 2023-10-19T12:45:01.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.472+0000 7fcaedd25640 1 -- 172.21.15.57:0/3438911280 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcae81f5790 con 0x7fcae80c96b0 2023-10-19T12:45:01.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.472+0000 7fcaedd25640 1 -- 172.21.15.57:0/3438911280 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcae81f18b0 con 0x7fcae80c96b0 2023-10-19T12:45:01.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.473+0000 7fcacffff640 1 -- 172.21.15.57:0/3438911280 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fcae003b050 con 0x7fcae80c96b0 2023-10-19T12:45:01.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.473+0000 7fcacffff640 1 --2- 172.21.15.57:0/3438911280 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fcad406ae40 0x7fcad406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:01.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.473+0000 7fcacffff640 1 -- 172.21.15.57:0/3438911280 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fcae00ac8c0 con 0x7fcae80c96b0 2023-10-19T12:45:01.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.473+0000 7fcae77fe640 1 --2- 172.21.15.57:0/3438911280 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fcad406ae40 0x7fcad406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:01.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.474+0000 7fcae77fe640 1 --2- 172.21.15.57:0/3438911280 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fcad406ae40 0x7fcad406d300 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fcadc005fd0 tx=0x7fcadc005850 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:01.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.478+0000 7fcacffff640 1 -- 172.21.15.57:0/3438911280 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fcae00776e0 con 0x7fcae80c96b0 2023-10-19T12:45:01.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.654+0000 7fcaedd25640 1 -- 172.21.15.57:0/3438911280 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fcae806c7e0 con 0x7fcae80c96b0 2023-10-19T12:45:01.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.655+0000 7fcacffff640 1 -- 172.21.15.57:0/3438911280 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7fcae007ad00 con 0x7fcae80c96b0 2023-10-19T12:45:01.657 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2023-10-19T12:45:01.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.657+0000 7fcaedd25640 1 -- 172.21.15.57:0/3438911280 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fcad406ae40 msgr2=0x7fcad406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:01.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.657+0000 7fcaedd25640 1 --2- 172.21.15.57:0/3438911280 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fcad406ae40 0x7fcad406d300 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fcadc005fd0 tx=0x7fcadc005850 comp rx=0 tx=0).stop 2023-10-19T12:45:01.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.657+0000 7fcaedd25640 1 -- 172.21.15.57:0/3438911280 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcae80c96b0 msgr2=0x7fcae813d7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:01.660 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.657+0000 7fcaedd25640 1 --2- 172.21.15.57:0/3438911280 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcae80c96b0 0x7fcae813d7a0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fcae0009bf0 tx=0x7fcae0035ad0 comp rx=0 tx=0).stop 2023-10-19T12:45:01.660 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.657+0000 7fcaedd25640 1 -- 172.21.15.57:0/3438911280 shutdown_connections 2023-10-19T12:45:01.660 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.657+0000 7fcaedd25640 1 --2- 172.21.15.57:0/3438911280 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fcad406ae40 0x7fcad406d300 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:01.660 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.657+0000 7fcaedd25640 1 --2- 172.21.15.57:0/3438911280 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcae80c96b0 0x7fcae813d7a0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:01.660 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.657+0000 7fcaedd25640 1 --2- 172.21.15.57:0/3438911280 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcae80abc10 0x7fcae81f4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:01.660 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.657+0000 7fcaedd25640 1 --2- 172.21.15.57:0/3438911280 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcae806af00 0x7fcae8158bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:01.661 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.657+0000 7fcaedd25640 1 -- 172.21.15.57:0/3438911280 >> 172.21.15.57:0/3438911280 conn(0x7fcae8069930 msgr2=0x7fcae8069d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:01.661 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.657+0000 7fcaedd25640 1 -- 172.21.15.57:0/3438911280 shutdown_connections 2023-10-19T12:45:01.661 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.657+0000 7fcaedd25640 1 -- 172.21.15.57:0/3438911280 wait complete. 2023-10-19T12:45:01.670 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-19T12:45:01.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.779+0000 7fc282bf0640 1 -- 172.21.15.57:0/3505096762 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc27c0ad370 msgr2=0x7fc27c0ad800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:01.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.779+0000 7fc282bf0640 1 --2- 172.21.15.57:0/3505096762 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc27c0ad370 0x7fc27c0ad800 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fc27c066970 tx=0x7fc26c02bdb0 comp rx=0 tx=0).stop 2023-10-19T12:45:01.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.779+0000 7fc282bf0640 1 -- 172.21.15.57:0/3505096762 shutdown_connections 2023-10-19T12:45:01.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.779+0000 7fc282bf0640 1 --2- 172.21.15.57:0/3505096762 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc27c0ad370 0x7fc27c0ad800 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:01.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.779+0000 7fc282bf0640 1 --2- 172.21.15.57:0/3505096762 >> v2:172.21.15.179:3300/0 conn(0x7fc27c06d2d0 0x7fc27c06d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:01.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.779+0000 7fc282bf0640 1 -- 172.21.15.57:0/3505096762 >> 172.21.15.57:0/3505096762 conn(0x7fc27c0b0670 msgr2=0x7fc27c0b0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:01.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.779+0000 7fc282bf0640 1 -- 172.21.15.57:0/3505096762 shutdown_connections 2023-10-19T12:45:01.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.779+0000 7fc282bf0640 1 -- 172.21.15.57:0/3505096762 wait complete. 2023-10-19T12:45:01.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.780+0000 7fc282bf0640 1 Processor -- start 2023-10-19T12:45:01.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.780+0000 7fc282bf0640 1 -- start start 2023-10-19T12:45:01.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.781+0000 7fc282bf0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc27c06d2d0 0x7fc27c149ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:01.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.781+0000 7fc282bf0640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc27c06f0a0 0x7fc27c0c97b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:01.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.781+0000 7fc282bf0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc27c0ad370 0x7fc27c0ab950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:01.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.781+0000 7fc282bf0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc27c0cc4c0 con 0x7fc27c0ad370 2023-10-19T12:45:01.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.781+0000 7fc282bf0640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc27c0cb940 con 0x7fc27c06d2d0 2023-10-19T12:45:01.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.781+0000 7fc282bf0640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc27c0cbc20 con 0x7fc27c06f0a0 2023-10-19T12:45:01.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.781+0000 7fc28118d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc27c0ad370 0x7fc27c0ab950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:01.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.781+0000 7fc28118d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc27c0ad370 0x7fc27c0ab950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50340/0 (socket says 172.21.15.57:50340) 2023-10-19T12:45:01.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.781+0000 7fc28118d640 1 -- 172.21.15.57:0/2598915130 learned_addr learned my addr 172.21.15.57:0/2598915130 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:01.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.781+0000 7fc28098c640 1 --2- 172.21.15.57:0/2598915130 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc27c06d2d0 0x7fc27c149ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:01.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.781+0000 7fc27bfff640 1 --2- 172.21.15.57:0/2598915130 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc27c06f0a0 0x7fc27c0c97b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:01.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.781+0000 7fc28118d640 1 -- 172.21.15.57:0/2598915130 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc27c06f0a0 msgr2=0x7fc27c0c97b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:01.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.781+0000 7fc28118d640 1 --2- 172.21.15.57:0/2598915130 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc27c06f0a0 0x7fc27c0c97b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:01.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.781+0000 7fc28118d640 1 -- 172.21.15.57:0/2598915130 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc27c06d2d0 msgr2=0x7fc27c149ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:01.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.781+0000 7fc28118d640 1 --2- 172.21.15.57:0/2598915130 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc27c06d2d0 0x7fc27c149ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:01.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.781+0000 7fc28118d640 1 -- 172.21.15.57:0/2598915130 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc26c009560 con 0x7fc27c0ad370 2023-10-19T12:45:01.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.781+0000 7fc28098c640 1 --2- 172.21.15.57:0/2598915130 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc27c06d2d0 0x7fc27c149ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:45:01.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.782+0000 7fc27bfff640 1 --2- 172.21.15.57:0/2598915130 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc27c06f0a0 0x7fc27c0c97b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:45:01.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.782+0000 7fc28118d640 1 --2- 172.21.15.57:0/2598915130 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc27c0ad370 0x7fc27c0ab950 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fc26c036040 tx=0x7fc26c02db70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:01.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.782+0000 7fc2797fa640 1 -- 172.21.15.57:0/2598915130 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc26c003800 con 0x7fc27c0ad370 2023-10-19T12:45:01.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.782+0000 7fc2797fa640 1 -- 172.21.15.57:0/2598915130 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fc26c003a50 con 0x7fc27c0ad370 2023-10-19T12:45:01.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.782+0000 7fc2797fa640 1 -- 172.21.15.57:0/2598915130 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc26c003e60 con 0x7fc27c0ad370 2023-10-19T12:45:01.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.782+0000 7fc282bf0640 1 -- 172.21.15.57:0/2598915130 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc27c0cbf00 con 0x7fc27c0ad370 2023-10-19T12:45:01.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.782+0000 7fc282bf0640 1 -- 172.21.15.57:0/2598915130 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc27c14a3e0 con 0x7fc27c0ad370 2023-10-19T12:45:01.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.783+0000 7fc282bf0640 1 -- 172.21.15.57:0/2598915130 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc27c0ce1d0 con 0x7fc27c0ad370 2023-10-19T12:45:01.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.783+0000 7fc2797fa640 1 -- 172.21.15.57:0/2598915130 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fc26c03e450 con 0x7fc27c0ad370 2023-10-19T12:45:01.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.784+0000 7fc2797fa640 1 --2- 172.21.15.57:0/2598915130 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fc26806ae40 0x7fc26806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:01.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.784+0000 7fc2797fa640 1 -- 172.21.15.57:0/2598915130 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fc26c0ac4c0 con 0x7fc27c0ad370 2023-10-19T12:45:01.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.784+0000 7fc28098c640 1 --2- 172.21.15.57:0/2598915130 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fc26806ae40 0x7fc26806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:01.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.785+0000 7fc28098c640 1 --2- 172.21.15.57:0/2598915130 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fc26806ae40 0x7fc26806d300 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc274005fd0 tx=0x7fc274008310 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:01.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.788+0000 7fc2797fa640 1 -- 172.21.15.57:0/2598915130 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc26c077360 con 0x7fc27c0ad370 2023-10-19T12:45:01.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.961+0000 7fc282bf0640 1 -- 172.21.15.57:0/2598915130 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc27c0690e0 con 0x7fc27c0ad370 2023-10-19T12:45:01.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.962+0000 7fc2797fa640 1 -- 172.21.15.57:0/2598915130 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc26c07a980 con 0x7fc27c0ad370 2023-10-19T12:45:01.964 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 64 2023-10-19T12:45:01.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.964+0000 7fc282bf0640 1 -- 172.21.15.57:0/2598915130 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fc26806ae40 msgr2=0x7fc26806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:01.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.964+0000 7fc282bf0640 1 --2- 172.21.15.57:0/2598915130 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fc26806ae40 0x7fc26806d300 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc274005fd0 tx=0x7fc274008310 comp rx=0 tx=0).stop 2023-10-19T12:45:01.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.964+0000 7fc282bf0640 1 -- 172.21.15.57:0/2598915130 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc27c0ad370 msgr2=0x7fc27c0ab950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:01.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.964+0000 7fc282bf0640 1 --2- 172.21.15.57:0/2598915130 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc27c0ad370 0x7fc27c0ab950 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fc26c036040 tx=0x7fc26c02db70 comp rx=0 tx=0).stop 2023-10-19T12:45:01.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.964+0000 7fc282bf0640 1 -- 172.21.15.57:0/2598915130 shutdown_connections 2023-10-19T12:45:01.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.964+0000 7fc282bf0640 1 --2- 172.21.15.57:0/2598915130 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fc26806ae40 0x7fc26806d300 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:01.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.964+0000 7fc282bf0640 1 --2- 172.21.15.57:0/2598915130 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc27c0ad370 0x7fc27c0ab950 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:01.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.964+0000 7fc282bf0640 1 --2- 172.21.15.57:0/2598915130 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc27c06f0a0 0x7fc27c0c97b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:01.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.964+0000 7fc282bf0640 1 --2- 172.21.15.57:0/2598915130 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc27c06d2d0 0x7fc27c149ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:01.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.964+0000 7fc282bf0640 1 -- 172.21.15.57:0/2598915130 >> 172.21.15.57:0/2598915130 conn(0x7fc27c0b0670 msgr2=0x7fc27c0c1e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:01.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.964+0000 7fc282bf0640 1 -- 172.21.15.57:0/2598915130 shutdown_connections 2023-10-19T12:45:01.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:01.964+0000 7fc282bf0640 1 -- 172.21.15.57:0/2598915130 wait complete. 2023-10-19T12:45:01.977 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-19T12:45:02.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.088+0000 7fb0e45d0640 1 -- 172.21.15.57:0/1690471876 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb0c8005180 msgr2=0x7fb0c800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:02.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.088+0000 7fb0e45d0640 1 --2- 172.21.15.57:0/1690471876 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb0c8005180 0x7fb0c800b980 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fb0d4009ed0 tx=0x7fb0c80057f0 comp rx=0 tx=0).stop 2023-10-19T12:45:02.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.088+0000 7fb0e45d0640 1 -- 172.21.15.57:0/1690471876 shutdown_connections 2023-10-19T12:45:02.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.088+0000 7fb0e45d0640 1 --2- 172.21.15.57:0/1690471876 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb0c8005180 0x7fb0c800b980 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:02.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.088+0000 7fb0e45d0640 1 -- 172.21.15.57:0/1690471876 >> 172.21.15.57:0/1690471876 conn(0x7fb0dc069930 msgr2=0x7fb0dc069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:02.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.089+0000 7fb0e45d0640 1 -- 172.21.15.57:0/1690471876 shutdown_connections 2023-10-19T12:45:02.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.089+0000 7fb0e45d0640 1 -- 172.21.15.57:0/1690471876 wait complete. 2023-10-19T12:45:02.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.090+0000 7fb0e45d0640 1 Processor -- start 2023-10-19T12:45:02.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.090+0000 7fb0e45d0640 1 -- start start 2023-10-19T12:45:02.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.090+0000 7fb0e45d0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb0c8005180 0x7fb0dc0af110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:02.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.090+0000 7fb0e45d0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb0dc154690 0x7fb0dc151b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:02.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.090+0000 7fb0e45d0640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb0dc156130 0x7fb0dc13a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:02.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.090+0000 7fb0e45d0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0dc154b10 con 0x7fb0dc154690 2023-10-19T12:45:02.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.090+0000 7fb0e45d0640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0dc0afb50 con 0x7fb0c8005180 2023-10-19T12:45:02.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.090+0000 7fb0e45d0640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0dc0afdf0 con 0x7fb0dc156130 2023-10-19T12:45:02.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.091+0000 7fb0e1b6b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb0dc154690 0x7fb0dc151b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:02.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.091+0000 7fb0e1b6b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb0dc154690 0x7fb0dc151b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50360/0 (socket says 172.21.15.57:50360) 2023-10-19T12:45:02.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.091+0000 7fb0e236c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb0c8005180 0x7fb0dc0af110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:02.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.091+0000 7fb0e1b6b640 1 -- 172.21.15.57:0/3192596361 learned_addr learned my addr 172.21.15.57:0/3192596361 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:02.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.091+0000 7fb0e2b6d640 1 --2- 172.21.15.57:0/3192596361 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb0dc156130 0x7fb0dc13a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:02.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.091+0000 7fb0e1b6b640 1 -- 172.21.15.57:0/3192596361 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb0dc156130 msgr2=0x7fb0dc13a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:02.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.091+0000 7fb0e1b6b640 1 --2- 172.21.15.57:0/3192596361 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb0dc156130 0x7fb0dc13a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:02.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.091+0000 7fb0e1b6b640 1 -- 172.21.15.57:0/3192596361 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb0c8005180 msgr2=0x7fb0dc0af110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:02.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.091+0000 7fb0e1b6b640 1 --2- 172.21.15.57:0/3192596361 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb0c8005180 0x7fb0dc0af110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:02.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.091+0000 7fb0e1b6b640 1 -- 172.21.15.57:0/3192596361 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0dc0b00d0 con 0x7fb0dc154690 2023-10-19T12:45:02.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.091+0000 7fb0e236c640 1 --2- 172.21.15.57:0/3192596361 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb0c8005180 0x7fb0dc0af110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T12:45:02.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.091+0000 7fb0e1b6b640 1 --2- 172.21.15.57:0/3192596361 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb0dc154690 0x7fb0dc151b00 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc0017a0 tx=0x7fb0cc00ea00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:02.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.091+0000 7fb0d2ffd640 1 -- 172.21.15.57:0/3192596361 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0cc00c8f0 con 0x7fb0dc154690 2023-10-19T12:45:02.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.091+0000 7fb0d2ffd640 1 -- 172.21.15.57:0/3192596361 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fb0cc00cb40 con 0x7fb0dc154690 2023-10-19T12:45:02.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.091+0000 7fb0e45d0640 1 -- 172.21.15.57:0/3192596361 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0dc0b03b0 con 0x7fb0dc154690 2023-10-19T12:45:02.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.091+0000 7fb0d2ffd640 1 -- 172.21.15.57:0/3192596361 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0cc011460 con 0x7fb0dc154690 2023-10-19T12:45:02.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.091+0000 7fb0e45d0640 1 -- 172.21.15.57:0/3192596361 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0dc152040 con 0x7fb0dc154690 2023-10-19T12:45:02.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.092+0000 7fb0e45d0640 1 -- 172.21.15.57:0/3192596361 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0dc0b05f0 con 0x7fb0dc154690 2023-10-19T12:45:02.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.093+0000 7fb0d2ffd640 1 -- 172.21.15.57:0/3192596361 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fb0cc0115c0 con 0x7fb0dc154690 2023-10-19T12:45:02.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.093+0000 7fb0d2ffd640 1 --2- 172.21.15.57:0/3192596361 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fb0c006ae40 0x7fb0c006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:02.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.093+0000 7fb0d2ffd640 1 -- 172.21.15.57:0/3192596361 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fb0cc089ff0 con 0x7fb0dc154690 2023-10-19T12:45:02.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.093+0000 7fb0e236c640 1 --2- 172.21.15.57:0/3192596361 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fb0c006ae40 0x7fb0c006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:02.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.094+0000 7fb0e236c640 1 --2- 172.21.15.57:0/3192596361 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fb0c006ae40 0x7fb0c006d300 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fb0dc1f2540 tx=0x7fb0d8006c60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:02.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.097+0000 7fb0d2ffd640 1 -- 172.21.15.57:0/3192596361 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb0dc0b05f0 con 0x7fb0dc154690 2023-10-19T12:45:02.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.273+0000 7fb0e45d0640 1 -- 172.21.15.57:0/3192596361 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fb0dc0b05f0 con 0x7fb0dc154690 2023-10-19T12:45:02.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.273+0000 7fb0d2ffd640 1 -- 172.21.15.57:0/3192596361 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7fb0dc0b05f0 con 0x7fb0dc154690 2023-10-19T12:45:02.276 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2023-10-19T12:45:02.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.275+0000 7fb0e45d0640 1 -- 172.21.15.57:0/3192596361 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fb0c006ae40 msgr2=0x7fb0c006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:02.278 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.275+0000 7fb0e45d0640 1 --2- 172.21.15.57:0/3192596361 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fb0c006ae40 0x7fb0c006d300 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fb0dc1f2540 tx=0x7fb0d8006c60 comp rx=0 tx=0).stop 2023-10-19T12:45:02.278 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.275+0000 7fb0e45d0640 1 -- 172.21.15.57:0/3192596361 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb0dc154690 msgr2=0x7fb0dc151b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:02.278 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.275+0000 7fb0e45d0640 1 --2- 172.21.15.57:0/3192596361 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb0dc154690 0x7fb0dc151b00 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc0017a0 tx=0x7fb0cc00ea00 comp rx=0 tx=0).stop 2023-10-19T12:45:02.278 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.275+0000 7fb0e45d0640 1 -- 172.21.15.57:0/3192596361 shutdown_connections 2023-10-19T12:45:02.278 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.275+0000 7fb0e45d0640 1 --2- 172.21.15.57:0/3192596361 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fb0c006ae40 0x7fb0c006d300 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:02.279 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.275+0000 7fb0e45d0640 1 --2- 172.21.15.57:0/3192596361 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb0dc156130 0x7fb0dc13a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:02.279 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.275+0000 7fb0e45d0640 1 --2- 172.21.15.57:0/3192596361 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb0dc154690 0x7fb0dc151b00 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:02.279 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.275+0000 7fb0e45d0640 1 --2- 172.21.15.57:0/3192596361 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb0c8005180 0x7fb0dc0af110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:02.279 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.275+0000 7fb0e45d0640 1 -- 172.21.15.57:0/3192596361 >> 172.21.15.57:0/3192596361 conn(0x7fb0dc069930 msgr2=0x7fb0dc15b590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:02.280 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.275+0000 7fb0e45d0640 1 -- 172.21.15.57:0/3192596361 shutdown_connections 2023-10-19T12:45:02.280 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.275+0000 7fb0e45d0640 1 -- 172.21.15.57:0/3192596361 wait complete. 2023-10-19T12:45:02.288 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T12:45:02.397 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.395+0000 7f2725ae5640 1 -- 172.21.15.57:0/3315870625 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f272015b2a0 msgr2=0x7f27200d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:02.398 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.395+0000 7f2725ae5640 1 --2- 172.21.15.57:0/3315870625 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f272015b2a0 0x7f27200d1910 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f2714005fd0 tx=0x7f271402bd90 comp rx=0 tx=0).stop 2023-10-19T12:45:02.398 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.395+0000 7f2725ae5640 1 -- 172.21.15.57:0/3315870625 shutdown_connections 2023-10-19T12:45:02.398 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.395+0000 7f2725ae5640 1 --2- 172.21.15.57:0/3315870625 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f27200ceb40 0x7f27200b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:02.398 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.395+0000 7f2725ae5640 1 --2- 172.21.15.57:0/3315870625 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f272015b2a0 0x7f27200d1910 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:02.398 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.395+0000 7f2725ae5640 1 -- 172.21.15.57:0/3315870625 >> 172.21.15.57:0/3315870625 conn(0x7f27200d0d60 msgr2=0x7f27200d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:02.399 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.395+0000 7f2725ae5640 1 -- 172.21.15.57:0/3315870625 shutdown_connections 2023-10-19T12:45:02.399 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.395+0000 7f2725ae5640 1 -- 172.21.15.57:0/3315870625 wait complete. 2023-10-19T12:45:02.401 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.396+0000 7f2725ae5640 1 Processor -- start 2023-10-19T12:45:02.401 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.396+0000 7f2725ae5640 1 -- start start 2023-10-19T12:45:02.401 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.397+0000 7f2725ae5640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f27200ceb40 0x7f272015e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:02.401 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.397+0000 7f2725ae5640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f27200cf340 0x7f27201f1000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:02.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.397+0000 7f2725ae5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f272015b2a0 0x7f27200b50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:02.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.397+0000 7f2725ae5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27201f1540 con 0x7f272015b2a0 2023-10-19T12:45:02.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.397+0000 7f2725ae5640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27200b3e30 con 0x7f27200ceb40 2023-10-19T12:45:02.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.397+0000 7f2725ae5640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27200b4110 con 0x7f27200cf340 2023-10-19T12:45:02.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.397+0000 7f271ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f272015b2a0 0x7f27200b50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:02.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.397+0000 7f271ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f272015b2a0 0x7f27200b50e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50372/0 (socket says 172.21.15.57:50372) 2023-10-19T12:45:02.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.397+0000 7f271ffff640 1 -- 172.21.15.57:0/3796841879 learned_addr learned my addr 172.21.15.57:0/3796841879 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:02.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.397+0000 7f271f7fe640 1 --2- 172.21.15.57:0/3796841879 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f27200ceb40 0x7f272015e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:02.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.397+0000 7f271effd640 1 --2- 172.21.15.57:0/3796841879 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f27200cf340 0x7f27201f1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:02.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.398+0000 7f271ffff640 1 -- 172.21.15.57:0/3796841879 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f27200cf340 msgr2=0x7f27201f1000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:02.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.398+0000 7f271ffff640 1 --2- 172.21.15.57:0/3796841879 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f27200cf340 0x7f27201f1000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:02.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.398+0000 7f271ffff640 1 -- 172.21.15.57:0/3796841879 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f27200ceb40 msgr2=0x7f272015e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:02.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.398+0000 7f271ffff640 1 --2- 172.21.15.57:0/3796841879 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f27200ceb40 0x7f272015e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:02.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.398+0000 7f271ffff640 1 -- 172.21.15.57:0/3796841879 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2714009560 con 0x7f272015b2a0 2023-10-19T12:45:02.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.398+0000 7f271ffff640 1 --2- 172.21.15.57:0/3796841879 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f272015b2a0 0x7f27200b50e0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f271800eb10 tx=0x7f271800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:02.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.398+0000 7f2707fff640 1 -- 172.21.15.57:0/3796841879 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2718004b20 con 0x7f272015b2a0 2023-10-19T12:45:02.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.398+0000 7f2707fff640 1 -- 172.21.15.57:0/3796841879 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f2718004ca0 con 0x7f272015b2a0 2023-10-19T12:45:02.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.398+0000 7f2725ae5640 1 -- 172.21.15.57:0/3796841879 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27200b0d50 con 0x7f272015b2a0 2023-10-19T12:45:02.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.398+0000 7f2707fff640 1 -- 172.21.15.57:0/3796841879 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2718005280 con 0x7f272015b2a0 2023-10-19T12:45:02.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.398+0000 7f2725ae5640 1 -- 172.21.15.57:0/3796841879 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27200b46d0 con 0x7f272015b2a0 2023-10-19T12:45:02.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.399+0000 7f2725ae5640 1 -- 172.21.15.57:0/3796841879 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27200d04f0 con 0x7f272015b2a0 2023-10-19T12:45:02.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.400+0000 7f2707fff640 1 -- 172.21.15.57:0/3796841879 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f27180053e0 con 0x7f272015b2a0 2023-10-19T12:45:02.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.400+0000 7f2707fff640 1 --2- 172.21.15.57:0/3796841879 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f270c06ae70 0x7f270c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:02.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.400+0000 7f2707fff640 1 -- 172.21.15.57:0/3796841879 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f2718014070 con 0x7f272015b2a0 2023-10-19T12:45:02.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.400+0000 7f271f7fe640 1 --2- 172.21.15.57:0/3796841879 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f270c06ae70 0x7f270c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:02.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.401+0000 7f271f7fe640 1 --2- 172.21.15.57:0/3796841879 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f270c06ae70 0x7f270c06d330 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f2710000f60 tx=0x7f2710009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:02.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.404+0000 7f2707fff640 1 -- 172.21.15.57:0/3796841879 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2718055380 con 0x7f272015b2a0 2023-10-19T12:45:02.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.579+0000 7f2725ae5640 1 -- 172.21.15.57:0/3796841879 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f27200690e0 con 0x7f272015b2a0 2023-10-19T12:45:02.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.581+0000 7f2707fff640 1 -- 172.21.15.57:0/3796841879 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18748 (secure 0 0 0) 0x7f27180589a0 con 0x7f272015b2a0 2023-10-19T12:45:02.584 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:45:02.584 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":27,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T12:44:58.978403+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-19T12:44:16.517510+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-19T12:44:17.398297+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-19T12:44:21.129403+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.23","mappings":[{"from":3,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"3.39","mappings":[{"from":7,"to":1}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.1d","mappings":[{"from":1,"to":5}]},{"pgid":"4.1e","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T12:45:02.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.583+0000 7f2725ae5640 1 -- 172.21.15.57:0/3796841879 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f270c06ae70 msgr2=0x7f270c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:02.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.583+0000 7f2725ae5640 1 --2- 172.21.15.57:0/3796841879 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f270c06ae70 0x7f270c06d330 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f2710000f60 tx=0x7f2710009040 comp rx=0 tx=0).stop 2023-10-19T12:45:02.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.583+0000 7f2725ae5640 1 -- 172.21.15.57:0/3796841879 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f272015b2a0 msgr2=0x7f27200b50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:02.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.583+0000 7f2725ae5640 1 --2- 172.21.15.57:0/3796841879 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f272015b2a0 0x7f27200b50e0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f271800eb10 tx=0x7f271800c2d0 comp rx=0 tx=0).stop 2023-10-19T12:45:02.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.583+0000 7f2725ae5640 1 -- 172.21.15.57:0/3796841879 shutdown_connections 2023-10-19T12:45:02.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.583+0000 7f2725ae5640 1 --2- 172.21.15.57:0/3796841879 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f270c06ae70 0x7f270c06d330 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:02.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.583+0000 7f2725ae5640 1 --2- 172.21.15.57:0/3796841879 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f272015b2a0 0x7f27200b50e0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:02.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.583+0000 7f2725ae5640 1 --2- 172.21.15.57:0/3796841879 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f27200cf340 0x7f27201f1000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:02.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.583+0000 7f2725ae5640 1 --2- 172.21.15.57:0/3796841879 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f27200ceb40 0x7f272015e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:02.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.583+0000 7f2725ae5640 1 -- 172.21.15.57:0/3796841879 >> 172.21.15.57:0/3796841879 conn(0x7f27200d0d60 msgr2=0x7f27200d1140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:02.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.583+0000 7f2725ae5640 1 -- 172.21.15.57:0/3796841879 shutdown_connections 2023-10-19T12:45:02.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.583+0000 7f2725ae5640 1 -- 172.21.15.57:0/3796841879 wait complete. 2023-10-19T12:45:02.597 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-19T12:44:07.371186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-19T12:44:16.517510+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-19T12:44:17.398297+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-19T12:44:21.129403+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-19T12:45:02.598 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-19T12:45:02.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.706+0000 7fa92898e640 1 -- 172.21.15.57:0/2070208567 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa9240cf340 msgr2=0x7fa9240d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:02.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.706+0000 7fa92898e640 1 --2- 172.21.15.57:0/2070208567 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa9240cf340 0x7fa9240d1910 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fa914005fd0 tx=0x7fa91402bd90 comp rx=0 tx=0).stop 2023-10-19T12:45:02.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.706+0000 7fa92898e640 1 -- 172.21.15.57:0/2070208567 shutdown_connections 2023-10-19T12:45:02.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.706+0000 7fa92898e640 1 --2- 172.21.15.57:0/2070208567 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa92415b2a0 0x7fa9240cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:02.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.706+0000 7fa92898e640 1 --2- 172.21.15.57:0/2070208567 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa9240cf340 0x7fa9240d1910 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:02.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.706+0000 7fa92898e640 1 -- 172.21.15.57:0/2070208567 >> 172.21.15.57:0/2070208567 conn(0x7fa9240d0d60 msgr2=0x7fa9240d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:02.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.706+0000 7fa92898e640 1 -- 172.21.15.57:0/2070208567 shutdown_connections 2023-10-19T12:45:02.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.706+0000 7fa92898e640 1 -- 172.21.15.57:0/2070208567 wait complete. 2023-10-19T12:45:02.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.708+0000 7fa92898e640 1 Processor -- start 2023-10-19T12:45:02.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.708+0000 7fa92898e640 1 -- start start 2023-10-19T12:45:02.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.708+0000 7fa92898e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa9240ceb40 0x7fa9241e4310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:02.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.708+0000 7fa92898e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa9240cf340 0x7fa92406af20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:02.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.708+0000 7fa92898e640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa92415b2a0 0x7fa92406b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:02.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.708+0000 7fa92898e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa924158490 con 0x7fa9240ceb40 2023-10-19T12:45:02.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.708+0000 7fa92898e640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9241f1610 con 0x7fa9240cf340 2023-10-19T12:45:02.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.708+0000 7fa92898e640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa924157810 con 0x7fa92415b2a0 2023-10-19T12:45:02.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.709+0000 7fa92259c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa9240ceb40 0x7fa9241e4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:02.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.709+0000 7fa92259c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa9240ceb40 0x7fa9241e4310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50382/0 (socket says 172.21.15.57:50382) 2023-10-19T12:45:02.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.709+0000 7fa92259c640 1 -- 172.21.15.57:0/1552415523 learned_addr learned my addr 172.21.15.57:0/1552415523 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:02.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.709+0000 7fa919d9b640 1 --2- 172.21.15.57:0/1552415523 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa9240cf340 0x7fa92406af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:02.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.709+0000 7fa922d9d640 1 --2- 172.21.15.57:0/1552415523 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa92415b2a0 0x7fa92406b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:02.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.709+0000 7fa92259c640 1 -- 172.21.15.57:0/1552415523 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa92415b2a0 msgr2=0x7fa92406b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:02.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.709+0000 7fa92259c640 1 --2- 172.21.15.57:0/1552415523 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa92415b2a0 0x7fa92406b460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:02.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.709+0000 7fa92259c640 1 -- 172.21.15.57:0/1552415523 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa9240cf340 msgr2=0x7fa92406af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:02.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.709+0000 7fa92259c640 1 --2- 172.21.15.57:0/1552415523 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa9240cf340 0x7fa92406af20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:02.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.709+0000 7fa92259c640 1 -- 172.21.15.57:0/1552415523 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa914009560 con 0x7fa9240ceb40 2023-10-19T12:45:02.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.709+0000 7fa92259c640 1 --2- 172.21.15.57:0/1552415523 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa9240ceb40 0x7fa9241e4310 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fa914009850 tx=0x7fa91402d9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:02.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.709+0000 7fa91b7fe640 1 -- 172.21.15.57:0/1552415523 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa91402ddd0 con 0x7fa9240ceb40 2023-10-19T12:45:02.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.709+0000 7fa91b7fe640 1 -- 172.21.15.57:0/1552415523 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fa9140036e0 con 0x7fa9240ceb40 2023-10-19T12:45:02.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.709+0000 7fa92898e640 1 -- 172.21.15.57:0/1552415523 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa924157af0 con 0x7fa9240ceb40 2023-10-19T12:45:02.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.709+0000 7fa91b7fe640 1 -- 172.21.15.57:0/1552415523 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa914003a90 con 0x7fa9240ceb40 2023-10-19T12:45:02.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.709+0000 7fa92898e640 1 -- 172.21.15.57:0/1552415523 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9241580b0 con 0x7fa9240ceb40 2023-10-19T12:45:02.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.710+0000 7fa92898e640 1 -- 172.21.15.57:0/1552415523 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa924158240 con 0x7fa9240ceb40 2023-10-19T12:45:02.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.711+0000 7fa91b7fe640 1 -- 172.21.15.57:0/1552415523 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fa914003bf0 con 0x7fa9240ceb40 2023-10-19T12:45:02.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.711+0000 7fa91b7fe640 1 --2- 172.21.15.57:0/1552415523 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fa90406ae40 0x7fa90406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:02.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.711+0000 7fa91b7fe640 1 -- 172.21.15.57:0/1552415523 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fa9140ac350 con 0x7fa9240ceb40 2023-10-19T12:45:02.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.711+0000 7fa919d9b640 1 --2- 172.21.15.57:0/1552415523 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fa90406ae40 0x7fa90406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:02.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.712+0000 7fa919d9b640 1 --2- 172.21.15.57:0/1552415523 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fa90406ae40 0x7fa90406d300 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fa9241e5de0 tx=0x7fa90c005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:02.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.715+0000 7fa91b7fe640 1 -- 172.21.15.57:0/1552415523 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa914077220 con 0x7fa9240ceb40 2023-10-19T12:45:02.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.889+0000 7fa92898e640 1 -- 172.21.15.57:0/1552415523 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa9240690e0 con 0x7fa9240ceb40 2023-10-19T12:45:02.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.889+0000 7fa91b7fe640 1 -- 172.21.15.57:0/1552415523 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa91407a840 con 0x7fa9240ceb40 2023-10-19T12:45:02.893 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-19T12:45:02.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.891+0000 7fa92898e640 1 -- 172.21.15.57:0/1552415523 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fa90406ae40 msgr2=0x7fa90406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:02.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.891+0000 7fa92898e640 1 --2- 172.21.15.57:0/1552415523 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fa90406ae40 0x7fa90406d300 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fa9241e5de0 tx=0x7fa90c005f90 comp rx=0 tx=0).stop 2023-10-19T12:45:02.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.891+0000 7fa92898e640 1 -- 172.21.15.57:0/1552415523 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa9240ceb40 msgr2=0x7fa9241e4310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:02.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.891+0000 7fa92898e640 1 --2- 172.21.15.57:0/1552415523 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa9240ceb40 0x7fa9241e4310 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fa914009850 tx=0x7fa91402d9d0 comp rx=0 tx=0).stop 2023-10-19T12:45:02.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.891+0000 7fa92898e640 1 -- 172.21.15.57:0/1552415523 shutdown_connections 2023-10-19T12:45:02.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.891+0000 7fa92898e640 1 --2- 172.21.15.57:0/1552415523 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fa90406ae40 0x7fa90406d300 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:02.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.891+0000 7fa92898e640 1 --2- 172.21.15.57:0/1552415523 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa92415b2a0 0x7fa92406b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:02.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.891+0000 7fa92898e640 1 --2- 172.21.15.57:0/1552415523 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa9240cf340 0x7fa92406af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:02.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.891+0000 7fa92898e640 1 --2- 172.21.15.57:0/1552415523 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa9240ceb40 0x7fa9241e4310 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:02.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.891+0000 7fa92898e640 1 -- 172.21.15.57:0/1552415523 >> 172.21.15.57:0/1552415523 conn(0x7fa9240d0d60 msgr2=0x7fa92415a1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:02.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.891+0000 7fa92898e640 1 -- 172.21.15.57:0/1552415523 shutdown_connections 2023-10-19T12:45:02.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:02.891+0000 7fa92898e640 1 -- 172.21.15.57:0/1552415523 wait complete. 2023-10-19T12:45:02.905 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-19T12:45:03.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.015+0000 7f45f7f83640 1 -- 172.21.15.57:0/2084531981 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f45f015d400 msgr2=0x7f45f00cdfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:03.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.015+0000 7f45f7f83640 1 --2- 172.21.15.57:0/2084531981 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f45f015d400 0x7f45f00cdfe0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f45e8005fd0 tx=0x7f45e802c250 comp rx=0 tx=0).stop 2023-10-19T12:45:03.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.016+0000 7f45f7f83640 1 -- 172.21.15.57:0/2084531981 shutdown_connections 2023-10-19T12:45:03.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.016+0000 7f45f7f83640 1 --2- 172.21.15.57:0/2084531981 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f45f015d400 0x7f45f00cdfe0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:03.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.016+0000 7f45f7f83640 1 --2- 172.21.15.57:0/2084531981 >> v2:172.21.15.179:3300/0 conn(0x7f45f0159060 0x7f45f006d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:03.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.016+0000 7f45f7f83640 1 -- 172.21.15.57:0/2084531981 >> 172.21.15.57:0/2084531981 conn(0x7f45f00b0670 msgr2=0x7f45f00b0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:03.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.016+0000 7f45f7f83640 1 -- 172.21.15.57:0/2084531981 shutdown_connections 2023-10-19T12:45:03.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.016+0000 7f45f7f83640 1 -- 172.21.15.57:0/2084531981 wait complete. 2023-10-19T12:45:03.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.017+0000 7f45f7f83640 1 Processor -- start 2023-10-19T12:45:03.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.017+0000 7f45f7f83640 1 -- start start 2023-10-19T12:45:03.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.017+0000 7f45f7f83640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f45f00cdb30 0x7f45f00c6600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:03.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.017+0000 7f45f7f83640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f45f0159060 0x7f45f0150960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:03.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.017+0000 7f45f7f83640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f45f015d400 0x7f45f00b3590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:03.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.017+0000 7f45f7f83640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45f00c5600 con 0x7f45f015d400 2023-10-19T12:45:03.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.017+0000 7f45f7f83640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45f00b3100 con 0x7f45f0159060 2023-10-19T12:45:03.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.017+0000 7f45f7f83640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45f0150ea0 con 0x7f45f00cdb30 2023-10-19T12:45:03.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.018+0000 7f45f6520640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f45f015d400 0x7f45f00b3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:03.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.018+0000 7f45f6520640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f45f015d400 0x7f45f00b3590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50398/0 (socket says 172.21.15.57:50398) 2023-10-19T12:45:03.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.018+0000 7f45f6520640 1 -- 172.21.15.57:0/3903475318 learned_addr learned my addr 172.21.15.57:0/3903475318 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:03.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.018+0000 7f45f551e640 1 --2- 172.21.15.57:0/3903475318 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f45f0159060 0x7f45f0150960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:03.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.018+0000 7f45f6520640 1 -- 172.21.15.57:0/3903475318 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f45f00cdb30 msgr2=0x7f45f00c6600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:03.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.018+0000 7f45f6520640 1 --2- 172.21.15.57:0/3903475318 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f45f00cdb30 0x7f45f00c6600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:03.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.018+0000 7f45f6520640 1 -- 172.21.15.57:0/3903475318 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f45f0159060 msgr2=0x7f45f0150960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:03.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.018+0000 7f45f6520640 1 --2- 172.21.15.57:0/3903475318 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f45f0159060 0x7f45f0150960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:03.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.018+0000 7f45f6520640 1 -- 172.21.15.57:0/3903475318 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45e8009900 con 0x7f45f015d400 2023-10-19T12:45:03.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.018+0000 7f45f551e640 1 --2- 172.21.15.57:0/3903475318 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f45f0159060 0x7f45f0150960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:45:03.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.018+0000 7f45f6520640 1 --2- 172.21.15.57:0/3903475318 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f45f015d400 0x7f45f00b3590 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f45e8037040 tx=0x7f45e8035db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:03.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.018+0000 7f45e67fc640 1 -- 172.21.15.57:0/3903475318 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45e8034440 con 0x7f45f015d400 2023-10-19T12:45:03.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.018+0000 7f45e67fc640 1 -- 172.21.15.57:0/3903475318 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f45e8004250 con 0x7f45f015d400 2023-10-19T12:45:03.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.018+0000 7f45e67fc640 1 -- 172.21.15.57:0/3903475318 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45e8004600 con 0x7f45f015d400 2023-10-19T12:45:03.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.018+0000 7f45f7f83640 1 -- 172.21.15.57:0/3903475318 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45f00c83b0 con 0x7f45f015d400 2023-10-19T12:45:03.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.018+0000 7f45f7f83640 1 -- 172.21.15.57:0/3903475318 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45f00b2ab0 con 0x7f45f015d400 2023-10-19T12:45:03.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.019+0000 7f45f7f83640 1 -- 172.21.15.57:0/3903475318 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45f00675f0 con 0x7f45f015d400 2023-10-19T12:45:03.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.020+0000 7f45e67fc640 1 -- 172.21.15.57:0/3903475318 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f45e803b040 con 0x7f45f015d400 2023-10-19T12:45:03.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.020+0000 7f45e67fc640 1 --2- 172.21.15.57:0/3903475318 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f45d406ae40 0x7f45d406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:03.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.020+0000 7f45e67fc640 1 -- 172.21.15.57:0/3903475318 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f45e80ac650 con 0x7f45f015d400 2023-10-19T12:45:03.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.020+0000 7f45f5d1f640 1 --2- 172.21.15.57:0/3903475318 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f45d406ae40 0x7f45d406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:03.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.021+0000 7f45f5d1f640 1 --2- 172.21.15.57:0/3903475318 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f45d406ae40 0x7f45d406d300 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f45ec005fd0 tx=0x7f45ec005850 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:03.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.024+0000 7f45e67fc640 1 -- 172.21.15.57:0/3903475318 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f45e80773d0 con 0x7f45f015d400 2023-10-19T12:45:03.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.202+0000 7f45f7f83640 1 -- 172.21.15.57:0/3903475318 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f45f00b4aa0 con 0x7f45f015d400 2023-10-19T12:45:03.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.202+0000 7f45e67fc640 1 -- 172.21.15.57:0/3903475318 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7f45e807a9f0 con 0x7f45f015d400 2023-10-19T12:45:03.206 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2023-10-19T12:45:03.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.204+0000 7f45f7f83640 1 -- 172.21.15.57:0/3903475318 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f45d406ae40 msgr2=0x7f45d406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:03.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.204+0000 7f45f7f83640 1 --2- 172.21.15.57:0/3903475318 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f45d406ae40 0x7f45d406d300 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f45ec005fd0 tx=0x7f45ec005850 comp rx=0 tx=0).stop 2023-10-19T12:45:03.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.204+0000 7f45f7f83640 1 -- 172.21.15.57:0/3903475318 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f45f015d400 msgr2=0x7f45f00b3590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:03.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.204+0000 7f45f7f83640 1 --2- 172.21.15.57:0/3903475318 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f45f015d400 0x7f45f00b3590 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f45e8037040 tx=0x7f45e8035db0 comp rx=0 tx=0).stop 2023-10-19T12:45:03.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.204+0000 7f45f7f83640 1 -- 172.21.15.57:0/3903475318 shutdown_connections 2023-10-19T12:45:03.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.204+0000 7f45f7f83640 1 --2- 172.21.15.57:0/3903475318 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f45d406ae40 0x7f45d406d300 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:03.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.204+0000 7f45f7f83640 1 --2- 172.21.15.57:0/3903475318 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f45f015d400 0x7f45f00b3590 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:03.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.204+0000 7f45f7f83640 1 --2- 172.21.15.57:0/3903475318 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f45f0159060 0x7f45f0150960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:03.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.204+0000 7f45f7f83640 1 --2- 172.21.15.57:0/3903475318 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f45f00cdb30 0x7f45f00c6600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:03.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.204+0000 7f45f7f83640 1 -- 172.21.15.57:0/3903475318 >> 172.21.15.57:0/3903475318 conn(0x7f45f00b0670 msgr2=0x7f45f00cf390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:03.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.204+0000 7f45f7f83640 1 -- 172.21.15.57:0/3903475318 shutdown_connections 2023-10-19T12:45:03.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.204+0000 7f45f7f83640 1 -- 172.21.15.57:0/3903475318 wait complete. 2023-10-19T12:45:03.218 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-19T12:45:03.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.328+0000 7fe3d864b640 1 -- 172.21.15.57:0/62927444 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3d013e520 msgr2=0x7fe3d006bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:03.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.328+0000 7fe3d864b640 1 --2- 172.21.15.57:0/62927444 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3d013e520 0x7fe3d006bb60 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fe3c8009830 tx=0x7fe3c802bcd0 comp rx=0 tx=0).stop 2023-10-19T12:45:03.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.328+0000 7fe3d864b640 1 -- 172.21.15.57:0/62927444 shutdown_connections 2023-10-19T12:45:03.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.328+0000 7fe3d864b640 1 --2- 172.21.15.57:0/62927444 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe3d0070af0 0x7fe3d00affe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:03.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.328+0000 7fe3d864b640 1 --2- 172.21.15.57:0/62927444 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3d013e520 0x7fe3d006bb60 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:03.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.328+0000 7fe3d864b640 1 -- 172.21.15.57:0/62927444 >> 172.21.15.57:0/62927444 conn(0x7fe3d0069930 msgr2=0x7fe3d0069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:03.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.328+0000 7fe3d864b640 1 -- 172.21.15.57:0/62927444 shutdown_connections 2023-10-19T12:45:03.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.328+0000 7fe3d864b640 1 -- 172.21.15.57:0/62927444 wait complete. 2023-10-19T12:45:03.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.329+0000 7fe3d864b640 1 Processor -- start 2023-10-19T12:45:03.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.329+0000 7fe3d864b640 1 -- start start 2023-10-19T12:45:03.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.330+0000 7fe3d864b640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe3d0070af0 0x7fe3d01578e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:03.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.330+0000 7fe3d864b640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe3d013e520 0x7fe3d013b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:03.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.330+0000 7fe3d864b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3d0141690 0x7fe3d015a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:03.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.330+0000 7fe3d864b640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3d00c9300 con 0x7fe3d0141690 2023-10-19T12:45:03.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.330+0000 7fe3d864b640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3d013ffa0 con 0x7fe3d013e520 2023-10-19T12:45:03.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.330+0000 7fe3d864b640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3d00c8d40 con 0x7fe3d0070af0 2023-10-19T12:45:03.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.330+0000 7fe3d6be8640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3d0141690 0x7fe3d015a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:03.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.330+0000 7fe3d6be8640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3d0141690 0x7fe3d015a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50416/0 (socket says 172.21.15.57:50416) 2023-10-19T12:45:03.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.330+0000 7fe3d6be8640 1 -- 172.21.15.57:0/2190552304 learned_addr learned my addr 172.21.15.57:0/2190552304 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:03.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.330+0000 7fe3d63e7640 1 --2- 172.21.15.57:0/2190552304 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe3d0070af0 0x7fe3d01578e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:03.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.330+0000 7fe3d5be6640 1 --2- 172.21.15.57:0/2190552304 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe3d013e520 0x7fe3d013b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:03.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.330+0000 7fe3d6be8640 1 -- 172.21.15.57:0/2190552304 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe3d0070af0 msgr2=0x7fe3d01578e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:03.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.330+0000 7fe3d6be8640 1 --2- 172.21.15.57:0/2190552304 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe3d0070af0 0x7fe3d01578e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:03.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.330+0000 7fe3d6be8640 1 -- 172.21.15.57:0/2190552304 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe3d013e520 msgr2=0x7fe3d013b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:03.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.330+0000 7fe3d6be8640 1 --2- 172.21.15.57:0/2190552304 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe3d013e520 0x7fe3d013b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:03.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.330+0000 7fe3d6be8640 1 -- 172.21.15.57:0/2190552304 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3c8009490 con 0x7fe3d0141690 2023-10-19T12:45:03.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.331+0000 7fe3d6be8640 1 --2- 172.21.15.57:0/2190552304 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3d0141690 0x7fe3d015a0b0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fe3cc00ccb0 tx=0x7fe3cc00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:03.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.331+0000 7fe3c6ffd640 1 -- 172.21.15.57:0/2190552304 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3cc003970 con 0x7fe3d0141690 2023-10-19T12:45:03.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.331+0000 7fe3c6ffd640 1 -- 172.21.15.57:0/2190552304 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fe3cc003bc0 con 0x7fe3d0141690 2023-10-19T12:45:03.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.331+0000 7fe3d864b640 1 -- 172.21.15.57:0/2190552304 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3d00c9020 con 0x7fe3d0141690 2023-10-19T12:45:03.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.331+0000 7fe3c6ffd640 1 -- 172.21.15.57:0/2190552304 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3cc004030 con 0x7fe3d0141690 2023-10-19T12:45:03.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.331+0000 7fe3d864b640 1 -- 172.21.15.57:0/2190552304 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3d00c8a10 con 0x7fe3d0141690 2023-10-19T12:45:03.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.332+0000 7fe3d864b640 1 -- 172.21.15.57:0/2190552304 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3d00690e0 con 0x7fe3d0141690 2023-10-19T12:45:03.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.332+0000 7fe3c6ffd640 1 -- 172.21.15.57:0/2190552304 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fe3cc004190 con 0x7fe3d0141690 2023-10-19T12:45:03.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.333+0000 7fe3c6ffd640 1 --2- 172.21.15.57:0/2190552304 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fe3b406ae40 0x7fe3b406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:03.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.333+0000 7fe3c6ffd640 1 -- 172.21.15.57:0/2190552304 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fe3cc08a310 con 0x7fe3d0141690 2023-10-19T12:45:03.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.333+0000 7fe3d63e7640 1 --2- 172.21.15.57:0/2190552304 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fe3b406ae40 0x7fe3b406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:03.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.334+0000 7fe3d63e7640 1 --2- 172.21.15.57:0/2190552304 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fe3b406ae40 0x7fe3b406d300 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fe3c8036000 tx=0x7fe3c8004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:03.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.337+0000 7fe3c6ffd640 1 -- 172.21.15.57:0/2190552304 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe3cc055000 con 0x7fe3d0141690 2023-10-19T12:45:03.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.511+0000 7fe3d864b640 1 -- 172.21.15.57:0/2190552304 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe3d01404c0 con 0x7fe3d0141690 2023-10-19T12:45:03.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.512+0000 7fe3c6ffd640 1 -- 172.21.15.57:0/2190552304 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe3cc058620 con 0x7fe3d0141690 2023-10-19T12:45:03.516 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 64 2023-10-19T12:45:03.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.514+0000 7fe3d864b640 1 -- 172.21.15.57:0/2190552304 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fe3b406ae40 msgr2=0x7fe3b406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:03.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.514+0000 7fe3d864b640 1 --2- 172.21.15.57:0/2190552304 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fe3b406ae40 0x7fe3b406d300 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fe3c8036000 tx=0x7fe3c8004730 comp rx=0 tx=0).stop 2023-10-19T12:45:03.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.514+0000 7fe3d864b640 1 -- 172.21.15.57:0/2190552304 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3d0141690 msgr2=0x7fe3d015a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:03.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.514+0000 7fe3d864b640 1 --2- 172.21.15.57:0/2190552304 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3d0141690 0x7fe3d015a0b0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fe3cc00ccb0 tx=0x7fe3cc00a620 comp rx=0 tx=0).stop 2023-10-19T12:45:03.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.514+0000 7fe3d864b640 1 -- 172.21.15.57:0/2190552304 shutdown_connections 2023-10-19T12:45:03.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.514+0000 7fe3d864b640 1 --2- 172.21.15.57:0/2190552304 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fe3b406ae40 0x7fe3b406d300 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:03.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.514+0000 7fe3d864b640 1 --2- 172.21.15.57:0/2190552304 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3d0141690 0x7fe3d015a0b0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:03.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.514+0000 7fe3d864b640 1 --2- 172.21.15.57:0/2190552304 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe3d013e520 0x7fe3d013b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:03.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.514+0000 7fe3d864b640 1 --2- 172.21.15.57:0/2190552304 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe3d0070af0 0x7fe3d01578e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:03.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.514+0000 7fe3d864b640 1 -- 172.21.15.57:0/2190552304 >> 172.21.15.57:0/2190552304 conn(0x7fe3d0069930 msgr2=0x7fe3d00703e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:03.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.514+0000 7fe3d864b640 1 -- 172.21.15.57:0/2190552304 shutdown_connections 2023-10-19T12:45:03.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.514+0000 7fe3d864b640 1 -- 172.21.15.57:0/2190552304 wait complete. 2023-10-19T12:45:03.527 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-19T12:45:03.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.636+0000 7f8bc9fa4640 1 -- 172.21.15.57:0/957388758 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8bc4158910 msgr2=0x7f8bc4158cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:03.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.636+0000 7f8bc9fa4640 1 --2- 172.21.15.57:0/957388758 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8bc4158910 0x7f8bc4158cf0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8bb4009830 tx=0x7f8bb402bd60 comp rx=0 tx=0).stop 2023-10-19T12:45:03.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.636+0000 7f8bc9fa4640 1 -- 172.21.15.57:0/957388758 shutdown_connections 2023-10-19T12:45:03.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.636+0000 7f8bc9fa4640 1 --2- 172.21.15.57:0/957388758 >> v2:172.21.15.179:3300/0 conn(0x7f8bc40cd9e0 0x7f8bc40cde70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:03.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.636+0000 7f8bc9fa4640 1 --2- 172.21.15.57:0/957388758 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8bc4158910 0x7f8bc4158cf0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:03.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.636+0000 7f8bc9fa4640 1 -- 172.21.15.57:0/957388758 >> 172.21.15.57:0/957388758 conn(0x7f8bc40af7a0 msgr2=0x7f8bc40afb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:03.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.636+0000 7f8bc9fa4640 1 -- 172.21.15.57:0/957388758 shutdown_connections 2023-10-19T12:45:03.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.636+0000 7f8bc9fa4640 1 -- 172.21.15.57:0/957388758 wait complete. 2023-10-19T12:45:03.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.637+0000 7f8bc9fa4640 1 Processor -- start 2023-10-19T12:45:03.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.637+0000 7f8bc9fa4640 1 -- start start 2023-10-19T12:45:03.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.638+0000 7f8bc9fa4640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8bc40cd1c0 0x7f8bc413f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:03.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.638+0000 7f8bc9fa4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8bc40cd9e0 0x7f8bc414fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:03.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.638+0000 7f8bc9fa4640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8bc4158910 0x7f8bc4156ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:03.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.638+0000 7f8bc9fa4640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc414d1f0 con 0x7f8bc40cd9e0 2023-10-19T12:45:03.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.638+0000 7f8bc9fa4640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc414d4d0 con 0x7f8bc40cd1c0 2023-10-19T12:45:03.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.638+0000 7f8bc9fa4640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc414d7b0 con 0x7f8bc4158910 2023-10-19T12:45:03.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.638+0000 7f8bc2ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8bc40cd9e0 0x7f8bc414fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:03.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.638+0000 7f8bc2ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8bc40cd9e0 0x7f8bc414fa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50418/0 (socket says 172.21.15.57:50418) 2023-10-19T12:45:03.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.638+0000 7f8bc2ffd640 1 -- 172.21.15.57:0/444352196 learned_addr learned my addr 172.21.15.57:0/444352196 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:03.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.638+0000 7f8bc37fe640 1 --2- 172.21.15.57:0/444352196 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8bc40cd1c0 0x7f8bc413f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:03.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.638+0000 7f8bc3fff640 1 --2- 172.21.15.57:0/444352196 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8bc4158910 0x7f8bc4156ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:03.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.639+0000 7f8bc2ffd640 1 -- 172.21.15.57:0/444352196 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8bc4158910 msgr2=0x7f8bc4156ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:03.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.639+0000 7f8bc2ffd640 1 --2- 172.21.15.57:0/444352196 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8bc4158910 0x7f8bc4156ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:03.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.639+0000 7f8bc2ffd640 1 -- 172.21.15.57:0/444352196 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8bc40cd1c0 msgr2=0x7f8bc413f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:03.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.639+0000 7f8bc2ffd640 1 --2- 172.21.15.57:0/444352196 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8bc40cd1c0 0x7f8bc413f340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:03.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.639+0000 7f8bc2ffd640 1 -- 172.21.15.57:0/444352196 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bbc009840 con 0x7f8bc40cd9e0 2023-10-19T12:45:03.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.639+0000 7f8bc37fe640 1 --2- 172.21.15.57:0/444352196 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8bc40cd1c0 0x7f8bc413f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T12:45:03.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.639+0000 7f8bc2ffd640 1 --2- 172.21.15.57:0/444352196 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8bc40cd9e0 0x7f8bc414fa80 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f8bb800c400 tx=0x7f8bb800c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:03.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.639+0000 7f8babfff640 1 -- 172.21.15.57:0/444352196 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bb8004230 con 0x7f8bc40cd9e0 2023-10-19T12:45:03.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.639+0000 7f8babfff640 1 -- 172.21.15.57:0/444352196 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f8bb80044a0 con 0x7f8bc40cd9e0 2023-10-19T12:45:03.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.639+0000 7f8bc9fa4640 1 -- 172.21.15.57:0/444352196 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bb4009490 con 0x7f8bc40cd9e0 2023-10-19T12:45:03.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.639+0000 7f8babfff640 1 -- 172.21.15.57:0/444352196 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bb8003850 con 0x7f8bc40cd9e0 2023-10-19T12:45:03.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.639+0000 7f8bc9fa4640 1 -- 172.21.15.57:0/444352196 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bc414da90 con 0x7f8bc40cd9e0 2023-10-19T12:45:03.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.640+0000 7f8bc9fa4640 1 -- 172.21.15.57:0/444352196 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bc40690e0 con 0x7f8bc40cd9e0 2023-10-19T12:45:03.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.641+0000 7f8babfff640 1 -- 172.21.15.57:0/444352196 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f8bb80039b0 con 0x7f8bc40cd9e0 2023-10-19T12:45:03.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.641+0000 7f8babfff640 1 --2- 172.21.15.57:0/444352196 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f8bb006ae40 0x7f8bb006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:03.650 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.641+0000 7f8babfff640 1 -- 172.21.15.57:0/444352196 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f8bb8088e90 con 0x7f8bc40cd9e0 2023-10-19T12:45:03.650 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.641+0000 7f8bc37fe640 1 --2- 172.21.15.57:0/444352196 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f8bb006ae40 0x7f8bb006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:03.650 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.642+0000 7f8bc37fe640 1 --2- 172.21.15.57:0/444352196 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f8bb006ae40 0x7f8bb006d300 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8bb4036000 tx=0x7f8bb4004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:03.650 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.645+0000 7f8babfff640 1 -- 172.21.15.57:0/444352196 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8bb8053cc0 con 0x7f8bc40cd9e0 2023-10-19T12:45:03.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.818+0000 7f8bc9fa4640 1 -- 172.21.15.57:0/444352196 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f8bc415c7f0 con 0x7f8bc40cd9e0 2023-10-19T12:45:03.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.819+0000 7f8babfff640 1 -- 172.21.15.57:0/444352196 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7f8bc415c7f0 con 0x7f8bc40cd9e0 2023-10-19T12:45:03.822 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2023-10-19T12:45:03.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.820+0000 7f8bc9fa4640 1 -- 172.21.15.57:0/444352196 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f8bb006ae40 msgr2=0x7f8bb006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:03.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.820+0000 7f8bc9fa4640 1 --2- 172.21.15.57:0/444352196 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f8bb006ae40 0x7f8bb006d300 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8bb4036000 tx=0x7f8bb4004730 comp rx=0 tx=0).stop 2023-10-19T12:45:03.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.820+0000 7f8bc9fa4640 1 -- 172.21.15.57:0/444352196 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8bc40cd9e0 msgr2=0x7f8bc414fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:03.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.820+0000 7f8bc9fa4640 1 --2- 172.21.15.57:0/444352196 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8bc40cd9e0 0x7f8bc414fa80 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f8bb800c400 tx=0x7f8bb800c800 comp rx=0 tx=0).stop 2023-10-19T12:45:03.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.821+0000 7f8bc9fa4640 1 -- 172.21.15.57:0/444352196 shutdown_connections 2023-10-19T12:45:03.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.821+0000 7f8bc9fa4640 1 --2- 172.21.15.57:0/444352196 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f8bb006ae40 0x7f8bb006d300 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:03.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.821+0000 7f8bc9fa4640 1 --2- 172.21.15.57:0/444352196 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8bc4158910 0x7f8bc4156ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:03.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.821+0000 7f8bc9fa4640 1 --2- 172.21.15.57:0/444352196 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8bc40cd9e0 0x7f8bc414fa80 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:03.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.821+0000 7f8bc9fa4640 1 --2- 172.21.15.57:0/444352196 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8bc40cd1c0 0x7f8bc413f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:03.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.821+0000 7f8bc9fa4640 1 -- 172.21.15.57:0/444352196 >> 172.21.15.57:0/444352196 conn(0x7f8bc40af7a0 msgr2=0x7f8bc4069720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:03.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.821+0000 7f8bc9fa4640 1 -- 172.21.15.57:0/444352196 shutdown_connections 2023-10-19T12:45:03.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.821+0000 7f8bc9fa4640 1 -- 172.21.15.57:0/444352196 wait complete. 2023-10-19T12:45:03.834 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T12:45:03.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.941+0000 7f7995f89640 1 -- 172.21.15.57:0/4210899677 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f79780039a0 msgr2=0x7f7978003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:03.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.941+0000 7f7995f89640 1 --2- 172.21.15.57:0/4210899677 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f79780039a0 0x7f7978003dc0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f7980006fd0 tx=0x7f79780057f0 comp rx=0 tx=0).stop 2023-10-19T12:45:03.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.941+0000 7f798ffff640 1 -- 172.21.15.57:0/4210899677 reap_dead start 2023-10-19T12:45:03.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.941+0000 7f7995f89640 1 -- 172.21.15.57:0/4210899677 shutdown_connections 2023-10-19T12:45:03.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.941+0000 7f7995f89640 1 -- 172.21.15.57:0/4210899677 >> 172.21.15.57:0/4210899677 conn(0x7f79900d0d60 msgr2=0x7f79900d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:03.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.941+0000 7f7995f89640 1 -- 172.21.15.57:0/4210899677 shutdown_connections 2023-10-19T12:45:03.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.941+0000 7f7995f89640 1 -- 172.21.15.57:0/4210899677 wait complete. 2023-10-19T12:45:03.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.942+0000 7f7995f89640 1 Processor -- start 2023-10-19T12:45:03.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.942+0000 7f7995f89640 1 -- start start 2023-10-19T12:45:03.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.943+0000 7f7995f89640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f79780039a0 0x7f7990158a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:03.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.943+0000 7f7995f89640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f799009c3b0 0x7f79900c7e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:03.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.943+0000 7f7995f89640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f799009c830 0x7f79900c8370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:03.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.943+0000 7f7995f89640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f799009dea0 con 0x7f799009c3b0 2023-10-19T12:45:03.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.943+0000 7f7995f89640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f799009ce00 con 0x7f799009c830 2023-10-19T12:45:03.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.943+0000 7f7995f89640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f799009d0e0 con 0x7f79780039a0 2023-10-19T12:45:03.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.943+0000 7f798effd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f799009c3b0 0x7f79900c7e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:03.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.943+0000 7f798effd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f799009c3b0 0x7f79900c7e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50438/0 (socket says 172.21.15.57:50438) 2023-10-19T12:45:03.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.943+0000 7f798effd640 1 -- 172.21.15.57:0/160301240 learned_addr learned my addr 172.21.15.57:0/160301240 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:03.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.943+0000 7f798f7fe640 1 --2- 172.21.15.57:0/160301240 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f79780039a0 0x7f7990158a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:03.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.943+0000 7f798ffff640 1 --2- 172.21.15.57:0/160301240 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f799009c830 0x7f79900c8370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:03.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.943+0000 7f798effd640 1 -- 172.21.15.57:0/160301240 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f79780039a0 msgr2=0x7f7990158a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:03.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.943+0000 7f798effd640 1 --2- 172.21.15.57:0/160301240 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f79780039a0 0x7f7990158a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:03.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.943+0000 7f798effd640 1 -- 172.21.15.57:0/160301240 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f799009c830 msgr2=0x7f79900c8370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:03.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.943+0000 7f798effd640 1 --2- 172.21.15.57:0/160301240 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f799009c830 0x7f79900c8370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:03.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.943+0000 7f798effd640 1 -- 172.21.15.57:0/160301240 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f799009d3c0 con 0x7f799009c3b0 2023-10-19T12:45:03.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.944+0000 7f798effd640 1 --2- 172.21.15.57:0/160301240 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f799009c3b0 0x7f79900c7e30 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f798400fa00 tx=0x7f798400fec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:03.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.944+0000 7f7977fff640 1 -- 172.21.15.57:0/160301240 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f798400bcb0 con 0x7f799009c3b0 2023-10-19T12:45:03.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.944+0000 7f7977fff640 1 -- 172.21.15.57:0/160301240 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f7984011440 con 0x7f799009c3b0 2023-10-19T12:45:03.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.944+0000 7f7995f89640 1 -- 172.21.15.57:0/160301240 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f799009d6a0 con 0x7f799009c3b0 2023-10-19T12:45:03.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.944+0000 7f7977fff640 1 -- 172.21.15.57:0/160301240 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7984011830 con 0x7f799009c3b0 2023-10-19T12:45:03.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.944+0000 7f7995f89640 1 -- 172.21.15.57:0/160301240 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79901594c0 con 0x7f799009c3b0 2023-10-19T12:45:03.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.945+0000 7f7995f89640 1 -- 172.21.15.57:0/160301240 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79900690e0 con 0x7f799009c3b0 2023-10-19T12:45:03.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.945+0000 7f7977fff640 1 -- 172.21.15.57:0/160301240 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f7984011990 con 0x7f799009c3b0 2023-10-19T12:45:03.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.946+0000 7f7977fff640 1 --2- 172.21.15.57:0/160301240 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f797c06ae40 0x7f797c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:03.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.946+0000 7f7977fff640 1 -- 172.21.15.57:0/160301240 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f798408a630 con 0x7f799009c3b0 2023-10-19T12:45:03.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.946+0000 7f798f7fe640 1 --2- 172.21.15.57:0/160301240 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f797c06ae40 0x7f797c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:03.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.946+0000 7f798f7fe640 1 --2- 172.21.15.57:0/160301240 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f797c06ae40 0x7f797c06d300 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f7980034000 tx=0x7f798002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:03.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:03.950+0000 7f7977fff640 1 -- 172.21.15.57:0/160301240 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7984055320 con 0x7f799009c3b0 2023-10-19T12:45:04.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.132+0000 7f7995f89640 1 -- 172.21.15.57:0/160301240 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f79900c76c0 con 0x7f799009c3b0 2023-10-19T12:45:04.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.133+0000 7f7977fff640 1 -- 172.21.15.57:0/160301240 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18748 (secure 0 0 0) 0x7f7984058940 con 0x7f799009c3b0 2023-10-19T12:45:04.136 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:45:04.136 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":27,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T12:44:58.978403+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-19T12:44:16.517510+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-19T12:44:17.398297+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-19T12:44:21.129403+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.23","mappings":[{"from":3,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"3.39","mappings":[{"from":7,"to":1}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.1d","mappings":[{"from":1,"to":5}]},{"pgid":"4.1e","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T12:45:04.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.135+0000 7f7995f89640 1 -- 172.21.15.57:0/160301240 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f797c06ae40 msgr2=0x7f797c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:04.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.135+0000 7f7995f89640 1 --2- 172.21.15.57:0/160301240 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f797c06ae40 0x7f797c06d300 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f7980034000 tx=0x7f798002e000 comp rx=0 tx=0).stop 2023-10-19T12:45:04.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.135+0000 7f7995f89640 1 -- 172.21.15.57:0/160301240 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f799009c3b0 msgr2=0x7f79900c7e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:04.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.135+0000 7f7995f89640 1 --2- 172.21.15.57:0/160301240 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f799009c3b0 0x7f79900c7e30 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f798400fa00 tx=0x7f798400fec0 comp rx=0 tx=0).stop 2023-10-19T12:45:04.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.135+0000 7f7995f89640 1 -- 172.21.15.57:0/160301240 shutdown_connections 2023-10-19T12:45:04.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.135+0000 7f7995f89640 1 --2- 172.21.15.57:0/160301240 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f797c06ae40 0x7f797c06d300 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:04.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.135+0000 7f7995f89640 1 --2- 172.21.15.57:0/160301240 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f799009c830 0x7f79900c8370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:04.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.135+0000 7f7995f89640 1 --2- 172.21.15.57:0/160301240 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f799009c3b0 0x7f79900c7e30 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:04.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.135+0000 7f7995f89640 1 --2- 172.21.15.57:0/160301240 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f79780039a0 0x7f7990158a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:04.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.135+0000 7f7995f89640 1 -- 172.21.15.57:0/160301240 >> 172.21.15.57:0/160301240 conn(0x7f79900d0d60 msgr2=0x7f79900ada60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:04.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.135+0000 7f7995f89640 1 -- 172.21.15.57:0/160301240 shutdown_connections 2023-10-19T12:45:04.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.135+0000 7f7995f89640 1 -- 172.21.15.57:0/160301240 wait complete. 2023-10-19T12:45:04.149 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-19T12:44:07.371186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-19T12:44:16.517510+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-19T12:44:17.398297+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-19T12:44:21.129403+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-19T12:45:04.149 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-19T12:45:04.262 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.259+0000 7f645dd6c640 1 -- 172.21.15.57:0/3513231975 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6440004d00 msgr2=0x7f6440005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:04.263 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.259+0000 7f645dd6c640 1 --2- 172.21.15.57:0/3513231975 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6440004d00 0x7f6440005180 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f6450009f00 tx=0x7f64400057f0 comp rx=0 tx=0).stop 2023-10-19T12:45:04.263 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.259+0000 7f6457fff640 1 -- 172.21.15.57:0/3513231975 reap_dead start 2023-10-19T12:45:04.263 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.259+0000 7f645dd6c640 1 -- 172.21.15.57:0/3513231975 shutdown_connections 2023-10-19T12:45:04.263 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.259+0000 7f645dd6c640 1 -- 172.21.15.57:0/3513231975 >> 172.21.15.57:0/3513231975 conn(0x7f645809a030 msgr2=0x7f64580d0d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:04.263 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.259+0000 7f645dd6c640 1 -- 172.21.15.57:0/3513231975 shutdown_connections 2023-10-19T12:45:04.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.259+0000 7f645dd6c640 1 -- 172.21.15.57:0/3513231975 wait complete. 2023-10-19T12:45:04.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.260+0000 7f645dd6c640 1 Processor -- start 2023-10-19T12:45:04.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.260+0000 7f645dd6c640 1 -- start start 2023-10-19T12:45:04.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.261+0000 7f645dd6c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6440004d00 0x7f64580ca070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:04.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.261+0000 7f645dd6c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f64580b38e0 0x7f6458154e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:04.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.261+0000 7f645dd6c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f64580b3d60 0x7f6458155380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:04.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.261+0000 7f645dd6c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f645815ba20 con 0x7f64580b38e0 2023-10-19T12:45:04.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.261+0000 7f645dd6c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f645815e460 con 0x7f6440004d00 2023-10-19T12:45:04.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.261+0000 7f645dd6c640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f645815e740 con 0x7f64580b3d60 2023-10-19T12:45:04.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.261+0000 7f6456ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f64580b38e0 0x7f6458154e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:04.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.261+0000 7f6456ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f64580b38e0 0x7f6458154e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50458/0 (socket says 172.21.15.57:50458) 2023-10-19T12:45:04.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.261+0000 7f6456ffd640 1 -- 172.21.15.57:0/1894372616 learned_addr learned my addr 172.21.15.57:0/1894372616 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:04.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.261+0000 7f64577fe640 1 --2- 172.21.15.57:0/1894372616 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6440004d00 0x7f64580ca070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:04.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.261+0000 7f6456ffd640 1 -- 172.21.15.57:0/1894372616 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f64580b3d60 msgr2=0x7f6458155380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:04.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.261+0000 7f6456ffd640 1 --2- 172.21.15.57:0/1894372616 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f64580b3d60 0x7f6458155380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:04.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.261+0000 7f6456ffd640 1 -- 172.21.15.57:0/1894372616 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6440004d00 msgr2=0x7f64580ca070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:04.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.261+0000 7f6456ffd640 1 --2- 172.21.15.57:0/1894372616 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6440004d00 0x7f64580ca070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:04.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.261+0000 7f6456ffd640 1 -- 172.21.15.57:0/1894372616 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f645815bc70 con 0x7f64580b38e0 2023-10-19T12:45:04.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.261+0000 7f64577fe640 1 --2- 172.21.15.57:0/1894372616 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6440004d00 0x7f64580ca070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:45:04.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.262+0000 7f6456ffd640 1 --2- 172.21.15.57:0/1894372616 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f64580b38e0 0x7f6458154e40 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f644c011a60 tx=0x7f644c011f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:04.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.262+0000 7f643ffff640 1 -- 172.21.15.57:0/1894372616 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f644c00fc40 con 0x7f64580b38e0 2023-10-19T12:45:04.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.262+0000 7f643ffff640 1 -- 172.21.15.57:0/1894372616 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f644c00fda0 con 0x7f64580b38e0 2023-10-19T12:45:04.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.262+0000 7f645dd6c640 1 -- 172.21.15.57:0/1894372616 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f645815bf50 con 0x7f64580b38e0 2023-10-19T12:45:04.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.262+0000 7f643ffff640 1 -- 172.21.15.57:0/1894372616 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f644c004280 con 0x7f64580b38e0 2023-10-19T12:45:04.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.262+0000 7f645dd6c640 1 -- 172.21.15.57:0/1894372616 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64580ca790 con 0x7f64580b38e0 2023-10-19T12:45:04.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.263+0000 7f645dd6c640 1 -- 172.21.15.57:0/1894372616 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f645808a450 con 0x7f64580b38e0 2023-10-19T12:45:04.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.263+0000 7f643ffff640 1 -- 172.21.15.57:0/1894372616 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f644c0035f0 con 0x7f64580b38e0 2023-10-19T12:45:04.272 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.264+0000 7f643ffff640 1 --2- 172.21.15.57:0/1894372616 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f644406ae70 0x7f644406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:04.272 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.264+0000 7f643ffff640 1 -- 172.21.15.57:0/1894372616 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f644c08a2a0 con 0x7f64580b38e0 2023-10-19T12:45:04.272 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.264+0000 7f64577fe640 1 --2- 172.21.15.57:0/1894372616 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f644406ae70 0x7f644406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:04.272 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.265+0000 7f64577fe640 1 --2- 172.21.15.57:0/1894372616 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f644406ae70 0x7f644406d330 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f6448000fb0 tx=0x7f64480067a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:04.272 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.268+0000 7f643ffff640 1 -- 172.21.15.57:0/1894372616 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f644c054f90 con 0x7f64580b38e0 2023-10-19T12:45:04.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.442+0000 7f645dd6c640 1 -- 172.21.15.57:0/1894372616 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f64580ca790 con 0x7f64580b38e0 2023-10-19T12:45:04.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.442+0000 7f643ffff640 1 -- 172.21.15.57:0/1894372616 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7f64580ca790 con 0x7f64580b38e0 2023-10-19T12:45:04.446 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-19T12:45:04.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.444+0000 7f645dd6c640 1 -- 172.21.15.57:0/1894372616 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f644406ae70 msgr2=0x7f644406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:04.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.444+0000 7f645dd6c640 1 --2- 172.21.15.57:0/1894372616 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f644406ae70 0x7f644406d330 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f6448000fb0 tx=0x7f64480067a0 comp rx=0 tx=0).stop 2023-10-19T12:45:04.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.444+0000 7f645dd6c640 1 -- 172.21.15.57:0/1894372616 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f64580b38e0 msgr2=0x7f6458154e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:04.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.444+0000 7f645dd6c640 1 --2- 172.21.15.57:0/1894372616 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f64580b38e0 0x7f6458154e40 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f644c011a60 tx=0x7f644c011f20 comp rx=0 tx=0).stop 2023-10-19T12:45:04.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.444+0000 7f645dd6c640 1 -- 172.21.15.57:0/1894372616 shutdown_connections 2023-10-19T12:45:04.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.444+0000 7f645dd6c640 1 --2- 172.21.15.57:0/1894372616 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f644406ae70 0x7f644406d330 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:04.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.444+0000 7f645dd6c640 1 --2- 172.21.15.57:0/1894372616 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f64580b3d60 0x7f6458155380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:04.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.444+0000 7f645dd6c640 1 --2- 172.21.15.57:0/1894372616 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f64580b38e0 0x7f6458154e40 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:04.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.444+0000 7f645dd6c640 1 --2- 172.21.15.57:0/1894372616 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6440004d00 0x7f64580ca070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:04.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.444+0000 7f645dd6c640 1 -- 172.21.15.57:0/1894372616 >> 172.21.15.57:0/1894372616 conn(0x7f645809a030 msgr2=0x7f64580af010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:04.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.444+0000 7f645dd6c640 1 -- 172.21.15.57:0/1894372616 shutdown_connections 2023-10-19T12:45:04.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.444+0000 7f645dd6c640 1 -- 172.21.15.57:0/1894372616 wait complete. 2023-10-19T12:45:04.458 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-19T12:45:04.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.567+0000 7f5934116640 1 -- 172.21.15.57:0/1133021422 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f592c0acd40 msgr2=0x7f592c1597c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:04.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.567+0000 7f5934116640 1 --2- 172.21.15.57:0/1133021422 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f592c0acd40 0x7f592c1597c0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f5924005fd0 tx=0x7f592402bd90 comp rx=0 tx=0).stop 2023-10-19T12:45:04.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.567+0000 7f5934116640 1 -- 172.21.15.57:0/1133021422 shutdown_connections 2023-10-19T12:45:04.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.567+0000 7f5934116640 1 --2- 172.21.15.57:0/1133021422 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f592c0acd40 0x7f592c1597c0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:04.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.567+0000 7f5934116640 1 --2- 172.21.15.57:0/1133021422 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f592c0b24d0 0x7f592c0b28b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:04.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.567+0000 7f5934116640 1 -- 172.21.15.57:0/1133021422 >> 172.21.15.57:0/1133021422 conn(0x7f592c0b3f00 msgr2=0x7f592c0b4310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:04.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.567+0000 7f5934116640 1 -- 172.21.15.57:0/1133021422 shutdown_connections 2023-10-19T12:45:04.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.567+0000 7f5934116640 1 -- 172.21.15.57:0/1133021422 wait complete. 2023-10-19T12:45:04.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.568+0000 7f5934116640 1 Processor -- start 2023-10-19T12:45:04.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.568+0000 7f5934116640 1 -- start start 2023-10-19T12:45:04.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.569+0000 7f5934116640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f592c0acd40 0x7f592c06a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:04.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.569+0000 7f5934116640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f592c0b24d0 0x7f592c06c390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:04.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.569+0000 7f5934116640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f592c0b3070 0x7f592c148870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:04.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.569+0000 7f5934116640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f592c145ea0 con 0x7f592c0acd40 2023-10-19T12:45:04.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.569+0000 7f5934116640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f592c06c8d0 con 0x7f592c0b24d0 2023-10-19T12:45:04.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.569+0000 7f5934116640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f592c0ae810 con 0x7f592c0b3070 2023-10-19T12:45:04.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.569+0000 7f5931eb2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f592c0acd40 0x7f592c06a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:04.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.569+0000 7f5931eb2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f592c0acd40 0x7f592c06a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50470/0 (socket says 172.21.15.57:50470) 2023-10-19T12:45:04.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.569+0000 7f5931eb2640 1 -- 172.21.15.57:0/1865260042 learned_addr learned my addr 172.21.15.57:0/1865260042 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:04.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.569+0000 7f59316b1640 1 --2- 172.21.15.57:0/1865260042 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f592c0b24d0 0x7f592c06c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:04.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.569+0000 7f59326b3640 1 --2- 172.21.15.57:0/1865260042 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f592c0b3070 0x7f592c148870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:04.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.569+0000 7f5931eb2640 1 -- 172.21.15.57:0/1865260042 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f592c0b3070 msgr2=0x7f592c148870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:04.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.569+0000 7f5931eb2640 1 --2- 172.21.15.57:0/1865260042 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f592c0b3070 0x7f592c148870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:04.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.569+0000 7f5931eb2640 1 -- 172.21.15.57:0/1865260042 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f592c0b24d0 msgr2=0x7f592c06c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:04.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.569+0000 7f5931eb2640 1 --2- 172.21.15.57:0/1865260042 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f592c0b24d0 0x7f592c06c390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:04.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.569+0000 7f5931eb2640 1 -- 172.21.15.57:0/1865260042 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5924009560 con 0x7f592c0acd40 2023-10-19T12:45:04.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.569+0000 7f59326b3640 1 --2- 172.21.15.57:0/1865260042 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f592c0b3070 0x7f592c148870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:45:04.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.570+0000 7f5931eb2640 1 --2- 172.21.15.57:0/1865260042 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f592c0acd40 0x7f592c06a830 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f591c00eab0 tx=0x7f591c00ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:04.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.570+0000 7f592a7fc640 1 -- 172.21.15.57:0/1865260042 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f591c00ce40 con 0x7f592c0acd40 2023-10-19T12:45:04.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.570+0000 7f592a7fc640 1 -- 172.21.15.57:0/1865260042 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f591c004c10 con 0x7f592c0acd40 2023-10-19T12:45:04.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.570+0000 7f592a7fc640 1 -- 172.21.15.57:0/1865260042 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f591c005020 con 0x7f592c0acd40 2023-10-19T12:45:04.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.570+0000 7f5934116640 1 -- 172.21.15.57:0/1865260042 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f592c06b7d0 con 0x7f592c0acd40 2023-10-19T12:45:04.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.570+0000 7f5934116640 1 -- 172.21.15.57:0/1865260042 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f592c148db0 con 0x7f592c0acd40 2023-10-19T12:45:04.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.571+0000 7f5934116640 1 -- 172.21.15.57:0/1865260042 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f592c0690e0 con 0x7f592c0acd40 2023-10-19T12:45:04.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.571+0000 7f592a7fc640 1 -- 172.21.15.57:0/1865260042 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f591c005180 con 0x7f592c0acd40 2023-10-19T12:45:04.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.572+0000 7f592a7fc640 1 --2- 172.21.15.57:0/1865260042 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f591806ae70 0x7f591806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:04.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.572+0000 7f592a7fc640 1 -- 172.21.15.57:0/1865260042 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f591c08a380 con 0x7f592c0acd40 2023-10-19T12:45:04.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.572+0000 7f59316b1640 1 --2- 172.21.15.57:0/1865260042 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f591806ae70 0x7f591806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:04.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.572+0000 7f59316b1640 1 --2- 172.21.15.57:0/1865260042 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f591806ae70 0x7f591806d330 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f592c066320 tx=0x7f5920009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:04.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.576+0000 7f592a7fc640 1 -- 172.21.15.57:0/1865260042 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f591c055100 con 0x7f592c0acd40 2023-10-19T12:45:04.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.751+0000 7f5934116640 1 -- 172.21.15.57:0/1865260042 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f592c0727b0 con 0x7f592c0acd40 2023-10-19T12:45:04.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.752+0000 7f592a7fc640 1 -- 172.21.15.57:0/1865260042 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7f591c058720 con 0x7f592c0acd40 2023-10-19T12:45:04.755 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2023-10-19T12:45:04.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.754+0000 7f5934116640 1 -- 172.21.15.57:0/1865260042 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f591806ae70 msgr2=0x7f591806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:04.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.754+0000 7f5934116640 1 --2- 172.21.15.57:0/1865260042 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f591806ae70 0x7f591806d330 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f592c066320 tx=0x7f5920009040 comp rx=0 tx=0).stop 2023-10-19T12:45:04.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.754+0000 7f5934116640 1 -- 172.21.15.57:0/1865260042 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f592c0acd40 msgr2=0x7f592c06a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:04.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.754+0000 7f5934116640 1 --2- 172.21.15.57:0/1865260042 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f592c0acd40 0x7f592c06a830 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f591c00eab0 tx=0x7f591c00ef70 comp rx=0 tx=0).stop 2023-10-19T12:45:04.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.754+0000 7f5934116640 1 -- 172.21.15.57:0/1865260042 shutdown_connections 2023-10-19T12:45:04.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.754+0000 7f5934116640 1 --2- 172.21.15.57:0/1865260042 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f591806ae70 0x7f591806d330 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:04.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.754+0000 7f5934116640 1 --2- 172.21.15.57:0/1865260042 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f592c0b3070 0x7f592c148870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:04.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.754+0000 7f5934116640 1 --2- 172.21.15.57:0/1865260042 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f592c0b24d0 0x7f592c06c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:04.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.754+0000 7f5934116640 1 --2- 172.21.15.57:0/1865260042 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f592c0acd40 0x7f592c06a830 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:04.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.754+0000 7f5934116640 1 -- 172.21.15.57:0/1865260042 >> 172.21.15.57:0/1865260042 conn(0x7f592c0b3f00 msgr2=0x7f592c153470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:04.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.754+0000 7f5934116640 1 -- 172.21.15.57:0/1865260042 shutdown_connections 2023-10-19T12:45:04.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.754+0000 7f5934116640 1 -- 172.21.15.57:0/1865260042 wait complete. 2023-10-19T12:45:04.767 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-19T12:45:04.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.877+0000 7f34776d7640 1 -- 172.21.15.57:0/2007830731 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3458004d00 msgr2=0x7f3458005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:04.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.877+0000 7f34776d7640 1 --2- 172.21.15.57:0/2007830731 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3458004d00 0x7f3458005180 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f3468009b50 tx=0x7f34580057f0 comp rx=0 tx=0).stop 2023-10-19T12:45:04.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.877+0000 7f3475c74640 1 -- 172.21.15.57:0/2007830731 reap_dead start 2023-10-19T12:45:04.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.877+0000 7f34776d7640 1 -- 172.21.15.57:0/2007830731 shutdown_connections 2023-10-19T12:45:04.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.877+0000 7f34776d7640 1 -- 172.21.15.57:0/2007830731 >> 172.21.15.57:0/2007830731 conn(0x7f34700d0d60 msgr2=0x7f34700d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:04.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.877+0000 7f34776d7640 1 -- 172.21.15.57:0/2007830731 shutdown_connections 2023-10-19T12:45:04.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.877+0000 7f34776d7640 1 -- 172.21.15.57:0/2007830731 wait complete. 2023-10-19T12:45:04.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.878+0000 7f34776d7640 1 Processor -- start 2023-10-19T12:45:04.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.878+0000 7f34776d7640 1 -- start start 2023-10-19T12:45:04.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.879+0000 7f34776d7640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3458004d00 0x7f347015ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:04.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.879+0000 7f34776d7640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f34701e3510 0x7f34701490f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:04.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.879+0000 7f34776d7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f34700c9e40 0x7f34700ca2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:04.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.879+0000 7f34776d7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34700ca800 con 0x7f34700c9e40 2023-10-19T12:45:04.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.879+0000 7f34776d7640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34701e3d20 con 0x7f3458004d00 2023-10-19T12:45:04.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.879+0000 7f34776d7640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34701e4000 con 0x7f34701e3510 2023-10-19T12:45:04.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.879+0000 7f3475c74640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f34700c9e40 0x7f34700ca2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:04.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.879+0000 7f3475c74640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f34700c9e40 0x7f34700ca2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50488/0 (socket says 172.21.15.57:50488) 2023-10-19T12:45:04.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.879+0000 7f3475c74640 1 -- 172.21.15.57:0/260074950 learned_addr learned my addr 172.21.15.57:0/260074950 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:04.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.879+0000 7f3475473640 1 --2- 172.21.15.57:0/260074950 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3458004d00 0x7f347015ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:04.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.879+0000 7f3474c72640 1 --2- 172.21.15.57:0/260074950 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f34701e3510 0x7f34701490f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:04.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.879+0000 7f3475c74640 1 -- 172.21.15.57:0/260074950 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f34701e3510 msgr2=0x7f34701490f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:04.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.879+0000 7f3475c74640 1 --2- 172.21.15.57:0/260074950 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f34701e3510 0x7f34701490f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:04.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.879+0000 7f3475c74640 1 -- 172.21.15.57:0/260074950 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3458004d00 msgr2=0x7f347015ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:04.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.879+0000 7f3475c74640 1 --2- 172.21.15.57:0/260074950 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3458004d00 0x7f347015ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:04.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.879+0000 7f3475c74640 1 -- 172.21.15.57:0/260074950 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34701e42e0 con 0x7f34700c9e40 2023-10-19T12:45:04.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.879+0000 7f3475c74640 1 --2- 172.21.15.57:0/260074950 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f34700c9e40 0x7f34700ca2c0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f3468009c70 tx=0x7f3468033b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:04.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.879+0000 7f346dffb640 1 -- 172.21.15.57:0/260074950 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3468033ea0 con 0x7f34700c9e40 2023-10-19T12:45:04.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.879+0000 7f346dffb640 1 -- 172.21.15.57:0/260074950 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f3468004230 con 0x7f34700c9e40 2023-10-19T12:45:04.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.879+0000 7f34776d7640 1 -- 172.21.15.57:0/260074950 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34701e45c0 con 0x7f34700c9e40 2023-10-19T12:45:04.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.879+0000 7f346dffb640 1 -- 172.21.15.57:0/260074950 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34680035f0 con 0x7f34700c9e40 2023-10-19T12:45:04.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.880+0000 7f34776d7640 1 -- 172.21.15.57:0/260074950 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3470149630 con 0x7f34700c9e40 2023-10-19T12:45:04.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.881+0000 7f34776d7640 1 -- 172.21.15.57:0/260074950 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34701e4b80 con 0x7f34700c9e40 2023-10-19T12:45:04.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.881+0000 7f346dffb640 1 -- 172.21.15.57:0/260074950 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f3468003750 con 0x7f34700c9e40 2023-10-19T12:45:04.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.881+0000 7f346dffb640 1 --2- 172.21.15.57:0/260074950 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f345c06ae70 0x7f345c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:04.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.882+0000 7f346dffb640 1 -- 172.21.15.57:0/260074950 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f34680abf80 con 0x7f34700c9e40 2023-10-19T12:45:04.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.882+0000 7f3475473640 1 --2- 172.21.15.57:0/260074950 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f345c06ae70 0x7f345c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:04.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.882+0000 7f3475473640 1 --2- 172.21.15.57:0/260074950 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f345c06ae70 0x7f345c06d330 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f3460001a30 tx=0x7f3460003970 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:04.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:04.886+0000 7f346dffb640 1 -- 172.21.15.57:0/260074950 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f34701e4b80 con 0x7f34700c9e40 2023-10-19T12:45:05.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.062+0000 7f34776d7640 1 -- 172.21.15.57:0/260074950 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f34701e4b80 con 0x7f34700c9e40 2023-10-19T12:45:05.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.062+0000 7f346dffb640 1 -- 172.21.15.57:0/260074950 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7f34701e4b80 con 0x7f34700c9e40 2023-10-19T12:45:05.065 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 64 2023-10-19T12:45:05.067 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.064+0000 7f34776d7640 1 -- 172.21.15.57:0/260074950 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f345c06ae70 msgr2=0x7f345c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:05.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.064+0000 7f34776d7640 1 --2- 172.21.15.57:0/260074950 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f345c06ae70 0x7f345c06d330 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f3460001a30 tx=0x7f3460003970 comp rx=0 tx=0).stop 2023-10-19T12:45:05.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.064+0000 7f34776d7640 1 -- 172.21.15.57:0/260074950 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f34700c9e40 msgr2=0x7f34700ca2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:05.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.064+0000 7f34776d7640 1 --2- 172.21.15.57:0/260074950 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f34700c9e40 0x7f34700ca2c0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f3468009c70 tx=0x7f3468033b60 comp rx=0 tx=0).stop 2023-10-19T12:45:05.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.064+0000 7f34776d7640 1 -- 172.21.15.57:0/260074950 shutdown_connections 2023-10-19T12:45:05.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.064+0000 7f34776d7640 1 --2- 172.21.15.57:0/260074950 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f345c06ae70 0x7f345c06d330 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:05.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.064+0000 7f34776d7640 1 --2- 172.21.15.57:0/260074950 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f34700c9e40 0x7f34700ca2c0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:05.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.064+0000 7f34776d7640 1 --2- 172.21.15.57:0/260074950 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f34701e3510 0x7f34701490f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:05.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.064+0000 7f34776d7640 1 --2- 172.21.15.57:0/260074950 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3458004d00 0x7f347015ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:05.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.064+0000 7f34776d7640 1 -- 172.21.15.57:0/260074950 >> 172.21.15.57:0/260074950 conn(0x7f34700d0d60 msgr2=0x7f347006b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:05.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.064+0000 7f34776d7640 1 -- 172.21.15.57:0/260074950 shutdown_connections 2023-10-19T12:45:05.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.064+0000 7f34776d7640 1 -- 172.21.15.57:0/260074950 wait complete. 2023-10-19T12:45:05.077 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-19T12:45:05.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.186+0000 7f788822b640 1 -- 172.21.15.57:0/956235938 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7880159060 msgr2=0x7f788006d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:05.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.186+0000 7f788822b640 1 --2- 172.21.15.57:0/956235938 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7880159060 0x7f788006d260 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f7878009900 tx=0x7f787802bd70 comp rx=0 tx=0).stop 2023-10-19T12:45:05.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.186+0000 7f788822b640 1 -- 172.21.15.57:0/956235938 shutdown_connections 2023-10-19T12:45:05.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.186+0000 7f788822b640 1 --2- 172.21.15.57:0/956235938 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f788015d400 0x7f78800cdfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:05.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.186+0000 7f788822b640 1 --2- 172.21.15.57:0/956235938 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7880159060 0x7f788006d260 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:05.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.186+0000 7f788822b640 1 -- 172.21.15.57:0/956235938 >> 172.21.15.57:0/956235938 conn(0x7f78800b0670 msgr2=0x7f78800b0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:05.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.186+0000 7f788822b640 1 -- 172.21.15.57:0/956235938 shutdown_connections 2023-10-19T12:45:05.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.186+0000 7f788822b640 1 -- 172.21.15.57:0/956235938 wait complete. 2023-10-19T12:45:05.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.187+0000 7f788822b640 1 Processor -- start 2023-10-19T12:45:05.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.187+0000 7f788822b640 1 -- start start 2023-10-19T12:45:05.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.188+0000 7f788822b640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f78800cdb30 0x7f78801f1e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:05.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.188+0000 7f788822b640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7880159060 0x7f7880159740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:05.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.188+0000 7f788822b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f788015d400 0x7f78800ac080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:05.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.188+0000 7f788822b640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78800b2250 con 0x7f788015d400 2023-10-19T12:45:05.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.188+0000 7f788822b640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78801f2390 con 0x7f78800cdb30 2023-10-19T12:45:05.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.188+0000 7f788822b640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7880157050 con 0x7f7880159060 2023-10-19T12:45:05.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.188+0000 7f78867c8640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f788015d400 0x7f78800ac080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:05.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.188+0000 7f78867c8640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f788015d400 0x7f78800ac080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50500/0 (socket says 172.21.15.57:50500) 2023-10-19T12:45:05.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.188+0000 7f78867c8640 1 -- 172.21.15.57:0/3779880142 learned_addr learned my addr 172.21.15.57:0/3779880142 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:05.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.188+0000 7f7885fc7640 1 --2- 172.21.15.57:0/3779880142 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f78800cdb30 0x7f78801f1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:05.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.188+0000 7f78857c6640 1 --2- 172.21.15.57:0/3779880142 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7880159060 0x7f7880159740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:05.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.189+0000 7f78867c8640 1 -- 172.21.15.57:0/3779880142 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7880159060 msgr2=0x7f7880159740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:05.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.189+0000 7f78867c8640 1 --2- 172.21.15.57:0/3779880142 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7880159060 0x7f7880159740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:05.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.189+0000 7f78867c8640 1 -- 172.21.15.57:0/3779880142 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f78800cdb30 msgr2=0x7f78801f1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:05.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.189+0000 7f78867c8640 1 --2- 172.21.15.57:0/3779880142 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f78800cdb30 0x7f78801f1e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:05.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.189+0000 7f78867c8640 1 -- 172.21.15.57:0/3779880142 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7878009560 con 0x7f788015d400 2023-10-19T12:45:05.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.189+0000 7f78867c8640 1 --2- 172.21.15.57:0/3779880142 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f788015d400 0x7f78800ac080 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f787000e960 tx=0x7f787000ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:05.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.189+0000 7f787e7fc640 1 -- 172.21.15.57:0/3779880142 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7870004d20 con 0x7f788015d400 2023-10-19T12:45:05.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.189+0000 7f787e7fc640 1 -- 172.21.15.57:0/3779880142 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f787000cb20 con 0x7f788015d400 2023-10-19T12:45:05.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.189+0000 7f788822b640 1 -- 172.21.15.57:0/3779880142 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78800b2430 con 0x7f788015d400 2023-10-19T12:45:05.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.189+0000 7f787e7fc640 1 -- 172.21.15.57:0/3779880142 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7870005020 con 0x7f788015d400 2023-10-19T12:45:05.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.189+0000 7f788822b640 1 -- 172.21.15.57:0/3779880142 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78800b29f0 con 0x7f788015d400 2023-10-19T12:45:05.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.190+0000 7f788822b640 1 -- 172.21.15.57:0/3779880142 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78800675f0 con 0x7f788015d400 2023-10-19T12:45:05.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.190+0000 7f787e7fc640 1 -- 172.21.15.57:0/3779880142 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f7870005180 con 0x7f788015d400 2023-10-19T12:45:05.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.191+0000 7f787e7fc640 1 --2- 172.21.15.57:0/3779880142 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f786c06ae70 0x7f786c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:05.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.191+0000 7f787e7fc640 1 -- 172.21.15.57:0/3779880142 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f787008a2c0 con 0x7f788015d400 2023-10-19T12:45:05.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.191+0000 7f7885fc7640 1 --2- 172.21.15.57:0/3779880142 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f786c06ae70 0x7f786c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:05.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.191+0000 7f7885fc7640 1 --2- 172.21.15.57:0/3779880142 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f786c06ae70 0x7f786c06d330 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f7878035000 tx=0x7f7878004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:05.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.195+0000 7f787e7fc640 1 -- 172.21.15.57:0/3779880142 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7870055070 con 0x7f788015d400 2023-10-19T12:45:05.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.369+0000 7f788822b640 1 -- 172.21.15.57:0/3779880142 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f78800b4aa0 con 0x7f788015d400 2023-10-19T12:45:05.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.369+0000 7f787e7fc640 1 -- 172.21.15.57:0/3779880142 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7f7870058690 con 0x7f788015d400 2023-10-19T12:45:05.373 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2023-10-19T12:45:05.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.371+0000 7f788822b640 1 -- 172.21.15.57:0/3779880142 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f786c06ae70 msgr2=0x7f786c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:05.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.371+0000 7f788822b640 1 --2- 172.21.15.57:0/3779880142 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f786c06ae70 0x7f786c06d330 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f7878035000 tx=0x7f7878004730 comp rx=0 tx=0).stop 2023-10-19T12:45:05.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.371+0000 7f788822b640 1 -- 172.21.15.57:0/3779880142 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f788015d400 msgr2=0x7f78800ac080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:05.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.371+0000 7f788822b640 1 --2- 172.21.15.57:0/3779880142 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f788015d400 0x7f78800ac080 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f787000e960 tx=0x7f787000ee20 comp rx=0 tx=0).stop 2023-10-19T12:45:05.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.371+0000 7f788822b640 1 -- 172.21.15.57:0/3779880142 shutdown_connections 2023-10-19T12:45:05.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.371+0000 7f788822b640 1 --2- 172.21.15.57:0/3779880142 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f786c06ae70 0x7f786c06d330 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:05.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.371+0000 7f788822b640 1 --2- 172.21.15.57:0/3779880142 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f788015d400 0x7f78800ac080 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:05.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.371+0000 7f788822b640 1 --2- 172.21.15.57:0/3779880142 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7880159060 0x7f7880159740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:05.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.371+0000 7f788822b640 1 --2- 172.21.15.57:0/3779880142 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f78800cdb30 0x7f78801f1e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:05.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.371+0000 7f788822b640 1 -- 172.21.15.57:0/3779880142 >> 172.21.15.57:0/3779880142 conn(0x7f78800b0670 msgr2=0x7f78800ada00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:05.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.371+0000 7f788822b640 1 -- 172.21.15.57:0/3779880142 shutdown_connections 2023-10-19T12:45:05.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.371+0000 7f788822b640 1 -- 172.21.15.57:0/3779880142 wait complete. 2023-10-19T12:45:05.385 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T12:45:05.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.490+0000 7f26de2da640 1 -- 172.21.15.57:0/1758378933 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f26d80cbd20 msgr2=0x7f26d8158300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:05.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.490+0000 7f26de2da640 1 --2- 172.21.15.57:0/1758378933 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f26d80cbd20 0x7f26d8158300 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f26d80cc490 tx=0x7f26c002be40 comp rx=0 tx=0).stop 2023-10-19T12:45:05.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.491+0000 7f26de2da640 1 -- 172.21.15.57:0/1758378933 shutdown_connections 2023-10-19T12:45:05.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.491+0000 7f26de2da640 1 --2- 172.21.15.57:0/1758378933 >> v2:172.21.15.179:3300/0 conn(0x7f26d80afa20 0x7f26d80705a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:05.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.491+0000 7f26de2da640 1 --2- 172.21.15.57:0/1758378933 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f26d80cbd20 0x7f26d8158300 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:05.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.491+0000 7f26de2da640 1 -- 172.21.15.57:0/1758378933 >> 172.21.15.57:0/1758378933 conn(0x7f26d8069930 msgr2=0x7f26d8069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:05.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.491+0000 7f26de2da640 1 -- 172.21.15.57:0/1758378933 shutdown_connections 2023-10-19T12:45:05.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.491+0000 7f26de2da640 1 -- 172.21.15.57:0/1758378933 wait complete. 2023-10-19T12:45:05.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.492+0000 7f26de2da640 1 Processor -- start 2023-10-19T12:45:05.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.492+0000 7f26de2da640 1 -- start start 2023-10-19T12:45:05.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.493+0000 7f26de2da640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f26d80afa20 0x7f26d8141480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:05.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.493+0000 7f26de2da640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f26d80b0530 0x7f26d8145340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:05.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.493+0000 7f26de2da640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f26d80cbd20 0x7f26d8142470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:05.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.493+0000 7f26de2da640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26d8146750 con 0x7f26d80b0530 2023-10-19T12:45:05.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.493+0000 7f26de2da640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26d8145880 con 0x7f26d80afa20 2023-10-19T12:45:05.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.493+0000 7f26de2da640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26d8145ad0 con 0x7f26d80cbd20 2023-10-19T12:45:05.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.493+0000 7f26d77fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f26d80b0530 0x7f26d8145340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:05.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.493+0000 7f26d77fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f26d80b0530 0x7f26d8145340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50516/0 (socket says 172.21.15.57:50516) 2023-10-19T12:45:05.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.493+0000 7f26d77fe640 1 -- 172.21.15.57:0/3600883009 learned_addr learned my addr 172.21.15.57:0/3600883009 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:05.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.493+0000 7f26d7fff640 1 --2- 172.21.15.57:0/3600883009 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f26d80afa20 0x7f26d8141480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:05.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.493+0000 7f26dc877640 1 --2- 172.21.15.57:0/3600883009 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f26d80cbd20 0x7f26d8142470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:05.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.493+0000 7f26d77fe640 1 -- 172.21.15.57:0/3600883009 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f26d80cbd20 msgr2=0x7f26d8142470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:05.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.493+0000 7f26d77fe640 1 --2- 172.21.15.57:0/3600883009 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f26d80cbd20 0x7f26d8142470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:05.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.493+0000 7f26d77fe640 1 -- 172.21.15.57:0/3600883009 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f26d80afa20 msgr2=0x7f26d8141480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:05.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.493+0000 7f26d77fe640 1 --2- 172.21.15.57:0/3600883009 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f26d80afa20 0x7f26d8141480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:05.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.493+0000 7f26d77fe640 1 -- 172.21.15.57:0/3600883009 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26c0009560 con 0x7f26d80b0530 2023-10-19T12:45:05.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.493+0000 7f26d7fff640 1 --2- 172.21.15.57:0/3600883009 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f26d80afa20 0x7f26d8141480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:45:05.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.493+0000 7f26d77fe640 1 --2- 172.21.15.57:0/3600883009 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f26d80b0530 0x7f26d8145340 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f26c800c920 tx=0x7f26c800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:05.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.494+0000 7f26d4ff9640 1 -- 172.21.15.57:0/3600883009 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26c8007c70 con 0x7f26d80b0530 2023-10-19T12:45:05.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.494+0000 7f26d4ff9640 1 -- 172.21.15.57:0/3600883009 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f26c800f450 con 0x7f26d80b0530 2023-10-19T12:45:05.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.494+0000 7f26de2da640 1 -- 172.21.15.57:0/3600883009 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26d8070cf0 con 0x7f26d80b0530 2023-10-19T12:45:05.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.494+0000 7f26d4ff9640 1 -- 172.21.15.57:0/3600883009 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26c800f860 con 0x7f26d80b0530 2023-10-19T12:45:05.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.494+0000 7f26de2da640 1 -- 172.21.15.57:0/3600883009 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26d8146090 con 0x7f26d80b0530 2023-10-19T12:45:05.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.495+0000 7f26de2da640 1 -- 172.21.15.57:0/3600883009 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26d80690e0 con 0x7f26d80b0530 2023-10-19T12:45:05.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.495+0000 7f26d4ff9640 1 -- 172.21.15.57:0/3600883009 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f26c800f9c0 con 0x7f26d80b0530 2023-10-19T12:45:05.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.495+0000 7f26d4ff9640 1 --2- 172.21.15.57:0/3600883009 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f26b806ae40 0x7f26b806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:05.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.495+0000 7f26d4ff9640 1 -- 172.21.15.57:0/3600883009 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f26c808a220 con 0x7f26d80b0530 2023-10-19T12:45:05.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.496+0000 7f26d7fff640 1 --2- 172.21.15.57:0/3600883009 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f26b806ae40 0x7f26b806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:05.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.496+0000 7f26d7fff640 1 --2- 172.21.15.57:0/3600883009 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f26b806ae40 0x7f26b806d300 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f26c0035000 tx=0x7f26c0004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:05.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.500+0000 7f26d4ff9640 1 -- 172.21.15.57:0/3600883009 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f26c8054fa0 con 0x7f26d80b0530 2023-10-19T12:45:05.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.675+0000 7f26de2da640 1 -- 172.21.15.57:0/3600883009 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f26d80cc1e0 con 0x7f26d80b0530 2023-10-19T12:45:05.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.676+0000 7f26d4ff9640 1 -- 172.21.15.57:0/3600883009 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18748 (secure 0 0 0) 0x7f26c80585c0 con 0x7f26d80b0530 2023-10-19T12:45:05.679 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:45:05.680 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":27,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T12:44:58.978403+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-19T12:44:16.517510+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-19T12:44:17.398297+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-19T12:44:21.129403+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.23","mappings":[{"from":3,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"3.39","mappings":[{"from":7,"to":1}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.1d","mappings":[{"from":1,"to":5}]},{"pgid":"4.1e","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T12:45:05.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.678+0000 7f26de2da640 1 -- 172.21.15.57:0/3600883009 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f26b806ae40 msgr2=0x7f26b806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:05.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.678+0000 7f26de2da640 1 --2- 172.21.15.57:0/3600883009 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f26b806ae40 0x7f26b806d300 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f26c0035000 tx=0x7f26c0004730 comp rx=0 tx=0).stop 2023-10-19T12:45:05.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.678+0000 7f26de2da640 1 -- 172.21.15.57:0/3600883009 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f26d80b0530 msgr2=0x7f26d8145340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:05.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.678+0000 7f26de2da640 1 --2- 172.21.15.57:0/3600883009 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f26d80b0530 0x7f26d8145340 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f26c800c920 tx=0x7f26c800cde0 comp rx=0 tx=0).stop 2023-10-19T12:45:05.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.678+0000 7f26de2da640 1 -- 172.21.15.57:0/3600883009 shutdown_connections 2023-10-19T12:45:05.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.678+0000 7f26de2da640 1 --2- 172.21.15.57:0/3600883009 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f26b806ae40 0x7f26b806d300 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:05.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.678+0000 7f26de2da640 1 --2- 172.21.15.57:0/3600883009 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f26d80cbd20 0x7f26d8142470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:05.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.678+0000 7f26de2da640 1 --2- 172.21.15.57:0/3600883009 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f26d80b0530 0x7f26d8145340 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:05.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.678+0000 7f26de2da640 1 --2- 172.21.15.57:0/3600883009 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f26d80afa20 0x7f26d8141480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:05.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.678+0000 7f26de2da640 1 -- 172.21.15.57:0/3600883009 >> 172.21.15.57:0/3600883009 conn(0x7f26d8069930 msgr2=0x7f26d8069d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:05.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.678+0000 7f26de2da640 1 -- 172.21.15.57:0/3600883009 shutdown_connections 2023-10-19T12:45:05.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.678+0000 7f26de2da640 1 -- 172.21.15.57:0/3600883009 wait complete. 2023-10-19T12:45:05.692 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-19T12:44:07.371186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-19T12:44:16.517510+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-19T12:44:17.398297+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-19T12:44:21.129403+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-19T12:45:05.692 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-19T12:45:05.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.801+0000 7f41a4232640 1 -- 172.21.15.57:0/90560672 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f419c06af00 msgr2=0x7f419c06b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:05.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.801+0000 7f41a4232640 1 --2- 172.21.15.57:0/90560672 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f419c06af00 0x7f419c06b2e0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f4194005fd0 tx=0x7f419402c240 comp rx=0 tx=0).stop 2023-10-19T12:45:05.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.801+0000 7f41a4232640 1 -- 172.21.15.57:0/90560672 shutdown_connections 2023-10-19T12:45:05.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.801+0000 7f41a4232640 1 --2- 172.21.15.57:0/90560672 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f419c0ab500 0x7f419c0ab990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:05.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.801+0000 7f41a4232640 1 --2- 172.21.15.57:0/90560672 >> v2:172.21.15.179:3300/0 conn(0x7f419c0c96b0 0x7f419c0c9b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:05.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.801+0000 7f41a4232640 1 --2- 172.21.15.57:0/90560672 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f419c06af00 0x7f419c06b2e0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:05.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.802+0000 7f41a4232640 1 -- 172.21.15.57:0/90560672 >> 172.21.15.57:0/90560672 conn(0x7f419c069930 msgr2=0x7f419c069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:05.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.802+0000 7f41a4232640 1 -- 172.21.15.57:0/90560672 shutdown_connections 2023-10-19T12:45:05.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.802+0000 7f41a4232640 1 -- 172.21.15.57:0/90560672 wait complete. 2023-10-19T12:45:05.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.803+0000 7f41a4232640 1 Processor -- start 2023-10-19T12:45:05.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.803+0000 7f41a4232640 1 -- start start 2023-10-19T12:45:05.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.804+0000 7f41a4232640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f419c06af00 0x7f419c13a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:05.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.804+0000 7f41a4232640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f419c0ab500 0x7f419c0aac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:05.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.804+0000 7f41a4232640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f419c0c96b0 0x7f419c1ef400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:05.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.804+0000 7f41a4232640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f419c0a90c0 con 0x7f419c0c96b0 2023-10-19T12:45:05.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.804+0000 7f41a4232640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f419c0a93a0 con 0x7f419c06af00 2023-10-19T12:45:05.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.804+0000 7f41a4232640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f419c0a9680 con 0x7f419c0ab500 2023-10-19T12:45:05.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.804+0000 7f41a27cf640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f419c0c96b0 0x7f419c1ef400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:05.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.804+0000 7f41a27cf640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f419c0c96b0 0x7f419c1ef400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50538/0 (socket says 172.21.15.57:50538) 2023-10-19T12:45:05.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.804+0000 7f41a27cf640 1 -- 172.21.15.57:0/4127226361 learned_addr learned my addr 172.21.15.57:0/4127226361 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:05.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.804+0000 7f41a1fce640 1 --2- 172.21.15.57:0/4127226361 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f419c06af00 0x7f419c13a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:05.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.804+0000 7f41a17cd640 1 --2- 172.21.15.57:0/4127226361 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f419c0ab500 0x7f419c0aac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:05.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.804+0000 7f41a27cf640 1 -- 172.21.15.57:0/4127226361 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f419c0ab500 msgr2=0x7f419c0aac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:05.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.804+0000 7f41a27cf640 1 --2- 172.21.15.57:0/4127226361 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f419c0ab500 0x7f419c0aac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:05.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.804+0000 7f41a27cf640 1 -- 172.21.15.57:0/4127226361 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f419c06af00 msgr2=0x7f419c13a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:05.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.804+0000 7f41a27cf640 1 --2- 172.21.15.57:0/4127226361 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f419c06af00 0x7f419c13a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:05.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.804+0000 7f41a27cf640 1 -- 172.21.15.57:0/4127226361 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4194009900 con 0x7f419c0c96b0 2023-10-19T12:45:05.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.804+0000 7f41a27cf640 1 --2- 172.21.15.57:0/4127226361 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f419c0c96b0 0x7f419c1ef400 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f418c00e970 tx=0x7f418c00ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:05.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.804+0000 7f419a7fc640 1 -- 172.21.15.57:0/4127226361 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f418c009720 con 0x7f419c0c96b0 2023-10-19T12:45:05.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.804+0000 7f419a7fc640 1 -- 172.21.15.57:0/4127226361 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f418c00ccc0 con 0x7f419c0c96b0 2023-10-19T12:45:05.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.804+0000 7f41a4232640 1 -- 172.21.15.57:0/4127226361 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f419c0a9960 con 0x7f419c0c96b0 2023-10-19T12:45:05.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.804+0000 7f419a7fc640 1 -- 172.21.15.57:0/4127226361 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f418c005280 con 0x7f419c0c96b0 2023-10-19T12:45:05.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.805+0000 7f41a4232640 1 -- 172.21.15.57:0/4127226361 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f419c0ab1c0 con 0x7f419c0c96b0 2023-10-19T12:45:05.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.806+0000 7f41a4232640 1 -- 172.21.15.57:0/4127226361 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f419c0690e0 con 0x7f419c0c96b0 2023-10-19T12:45:05.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.806+0000 7f419a7fc640 1 -- 172.21.15.57:0/4127226361 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f418c0053e0 con 0x7f419c0c96b0 2023-10-19T12:45:05.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.806+0000 7f419a7fc640 1 --2- 172.21.15.57:0/4127226361 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f418806ae40 0x7f418806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:05.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.807+0000 7f419a7fc640 1 -- 172.21.15.57:0/4127226361 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f418c014070 con 0x7f419c0c96b0 2023-10-19T12:45:05.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.807+0000 7f41a1fce640 1 --2- 172.21.15.57:0/4127226361 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f418806ae40 0x7f418806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:05.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.807+0000 7f41a1fce640 1 --2- 172.21.15.57:0/4127226361 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f418806ae40 0x7f418806d300 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f4194036000 tx=0x7f4194004ab0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:05.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.811+0000 7f419a7fc640 1 -- 172.21.15.57:0/4127226361 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f418c058ba0 con 0x7f419c0c96b0 2023-10-19T12:45:05.989 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.986+0000 7f41a4232640 1 -- 172.21.15.57:0/4127226361 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f419c0cc420 con 0x7f419c0c96b0 2023-10-19T12:45:05.989 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.986+0000 7f419a7fc640 1 -- 172.21.15.57:0/4127226361 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7f418c058730 con 0x7f419c0c96b0 2023-10-19T12:45:05.990 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-19T12:45:05.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.988+0000 7f41a4232640 1 -- 172.21.15.57:0/4127226361 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f418806ae40 msgr2=0x7f418806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:05.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.988+0000 7f41a4232640 1 --2- 172.21.15.57:0/4127226361 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f418806ae40 0x7f418806d300 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f4194036000 tx=0x7f4194004ab0 comp rx=0 tx=0).stop 2023-10-19T12:45:05.992 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.988+0000 7f41a4232640 1 -- 172.21.15.57:0/4127226361 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f419c0c96b0 msgr2=0x7f419c1ef400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:05.992 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.988+0000 7f41a4232640 1 --2- 172.21.15.57:0/4127226361 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f419c0c96b0 0x7f419c1ef400 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f418c00e970 tx=0x7f418c00ed70 comp rx=0 tx=0).stop 2023-10-19T12:45:05.992 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.988+0000 7f41a4232640 1 -- 172.21.15.57:0/4127226361 shutdown_connections 2023-10-19T12:45:05.992 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.988+0000 7f41a4232640 1 --2- 172.21.15.57:0/4127226361 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f418806ae40 0x7f418806d300 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:05.992 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.988+0000 7f41a4232640 1 --2- 172.21.15.57:0/4127226361 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f419c0c96b0 0x7f419c1ef400 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:05.993 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.988+0000 7f41a4232640 1 --2- 172.21.15.57:0/4127226361 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f419c0ab500 0x7f419c0aac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:05.993 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.988+0000 7f41a4232640 1 --2- 172.21.15.57:0/4127226361 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f419c06af00 0x7f419c13a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:05.993 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.988+0000 7f41a4232640 1 -- 172.21.15.57:0/4127226361 >> 172.21.15.57:0/4127226361 conn(0x7f419c069930 msgr2=0x7f419c0cafc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:05.993 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.988+0000 7f41a4232640 1 -- 172.21.15.57:0/4127226361 shutdown_connections 2023-10-19T12:45:05.993 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:05.988+0000 7f41a4232640 1 -- 172.21.15.57:0/4127226361 wait complete. 2023-10-19T12:45:06.002 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-19T12:45:06.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.114+0000 7efd41fe2640 1 -- 172.21.15.57:0/2105895431 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7efd3c15eba0 msgr2=0x7efd3c0b00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:06.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.114+0000 7efd41fe2640 1 --2- 172.21.15.57:0/2105895431 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7efd3c15eba0 0x7efd3c0b00f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7efd30009830 tx=0x7efd3002bde0 comp rx=0 tx=0).stop 2023-10-19T12:45:06.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.114+0000 7efd41fe2640 1 -- 172.21.15.57:0/2105895431 shutdown_connections 2023-10-19T12:45:06.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.114+0000 7efd41fe2640 1 --2- 172.21.15.57:0/2105895431 >> v2:172.21.15.179:3300/0 conn(0x7efd3c0cf2a0 0x7efd3c0cf730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:06.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.114+0000 7efd41fe2640 1 --2- 172.21.15.57:0/2105895431 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7efd3c15eba0 0x7efd3c0b00f0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:06.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.114+0000 7efd41fe2640 1 -- 172.21.15.57:0/2105895431 >> 172.21.15.57:0/2105895431 conn(0x7efd3c08a650 msgr2=0x7efd3c099f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:06.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.114+0000 7efd41fe2640 1 -- 172.21.15.57:0/2105895431 shutdown_connections 2023-10-19T12:45:06.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.114+0000 7efd41fe2640 1 -- 172.21.15.57:0/2105895431 wait complete. 2023-10-19T12:45:06.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.115+0000 7efd41fe2640 1 Processor -- start 2023-10-19T12:45:06.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.115+0000 7efd41fe2640 1 -- start start 2023-10-19T12:45:06.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.116+0000 7efd41fe2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efd3c0cf2a0 0x7efd3c0af210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:06.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.116+0000 7efd41fe2640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efd3c15b200 0x7efd3c1f4b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:06.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.116+0000 7efd41fe2640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7efd3c15eba0 0x7efd3c14ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:06.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.116+0000 7efd41fe2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd3c1f5a70 con 0x7efd3c0cf2a0 2023-10-19T12:45:06.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.116+0000 7efd41fe2640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd3c1f54b0 con 0x7efd3c15b200 2023-10-19T12:45:06.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.116+0000 7efd41fe2640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd3c1f5790 con 0x7efd3c15eba0 2023-10-19T12:45:06.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.116+0000 7efd3b7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efd3c0cf2a0 0x7efd3c0af210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:06.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.116+0000 7efd3b7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efd3c0cf2a0 0x7efd3c0af210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50546/0 (socket says 172.21.15.57:50546) 2023-10-19T12:45:06.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.116+0000 7efd3b7fe640 1 -- 172.21.15.57:0/413046614 learned_addr learned my addr 172.21.15.57:0/413046614 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:06.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.116+0000 7efd3affd640 1 --2- 172.21.15.57:0/413046614 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efd3c15b200 0x7efd3c1f4b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:06.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.116+0000 7efd3bfff640 1 --2- 172.21.15.57:0/413046614 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7efd3c15eba0 0x7efd3c14ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:06.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.116+0000 7efd3b7fe640 1 -- 172.21.15.57:0/413046614 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7efd3c15eba0 msgr2=0x7efd3c14ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:06.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.116+0000 7efd3b7fe640 1 --2- 172.21.15.57:0/413046614 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7efd3c15eba0 0x7efd3c14ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:06.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.116+0000 7efd3b7fe640 1 -- 172.21.15.57:0/413046614 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efd3c15b200 msgr2=0x7efd3c1f4b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:06.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.116+0000 7efd3b7fe640 1 --2- 172.21.15.57:0/413046614 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efd3c15b200 0x7efd3c1f4b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:06.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.116+0000 7efd3b7fe640 1 -- 172.21.15.57:0/413046614 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd34009840 con 0x7efd3c0cf2a0 2023-10-19T12:45:06.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.117+0000 7efd3b7fe640 1 --2- 172.21.15.57:0/413046614 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efd3c0cf2a0 0x7efd3c0af210 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7efd2c00c500 tx=0x7efd2c00c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:06.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.117+0000 7efd23fff640 1 -- 172.21.15.57:0/413046614 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd2c011790 con 0x7efd3c0cf2a0 2023-10-19T12:45:06.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.117+0000 7efd23fff640 1 -- 172.21.15.57:0/413046614 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7efd2c0119e0 con 0x7efd3c0cf2a0 2023-10-19T12:45:06.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.117+0000 7efd41fe2640 1 -- 172.21.15.57:0/413046614 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd30009490 con 0x7efd3c0cf2a0 2023-10-19T12:45:06.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.117+0000 7efd23fff640 1 -- 172.21.15.57:0/413046614 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd2c011dd0 con 0x7efd3c0cf2a0 2023-10-19T12:45:06.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.117+0000 7efd41fe2640 1 -- 172.21.15.57:0/413046614 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd3c0b4df0 con 0x7efd3c0cf2a0 2023-10-19T12:45:06.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.118+0000 7efd41fe2640 1 -- 172.21.15.57:0/413046614 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd3c0b4970 con 0x7efd3c0cf2a0 2023-10-19T12:45:06.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.118+0000 7efd23fff640 1 -- 172.21.15.57:0/413046614 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7efd2c01c460 con 0x7efd3c0cf2a0 2023-10-19T12:45:06.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.119+0000 7efd23fff640 1 --2- 172.21.15.57:0/413046614 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7efd2806ae70 0x7efd2806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:06.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.119+0000 7efd23fff640 1 -- 172.21.15.57:0/413046614 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7efd2c089a20 con 0x7efd3c0cf2a0 2023-10-19T12:45:06.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.119+0000 7efd3affd640 1 --2- 172.21.15.57:0/413046614 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7efd2806ae70 0x7efd2806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:06.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.119+0000 7efd3affd640 1 --2- 172.21.15.57:0/413046614 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7efd2806ae70 0x7efd2806d330 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7efd30036000 tx=0x7efd30004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:06.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.123+0000 7efd23fff640 1 -- 172.21.15.57:0/413046614 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7efd3c0b4970 con 0x7efd3c0cf2a0 2023-10-19T12:45:06.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.299+0000 7efd41fe2640 1 -- 172.21.15.57:0/413046614 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7efd3c0b4970 con 0x7efd3c0cf2a0 2023-10-19T12:45:06.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.299+0000 7efd23fff640 1 -- 172.21.15.57:0/413046614 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7efd3c0b4970 con 0x7efd3c0cf2a0 2023-10-19T12:45:06.303 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2023-10-19T12:45:06.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.301+0000 7efd41fe2640 1 -- 172.21.15.57:0/413046614 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7efd2806ae70 msgr2=0x7efd2806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:06.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.302+0000 7efd41fe2640 1 --2- 172.21.15.57:0/413046614 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7efd2806ae70 0x7efd2806d330 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7efd30036000 tx=0x7efd30004730 comp rx=0 tx=0).stop 2023-10-19T12:45:06.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.302+0000 7efd41fe2640 1 -- 172.21.15.57:0/413046614 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efd3c0cf2a0 msgr2=0x7efd3c0af210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:06.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.302+0000 7efd41fe2640 1 --2- 172.21.15.57:0/413046614 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efd3c0cf2a0 0x7efd3c0af210 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7efd2c00c500 tx=0x7efd2c00c900 comp rx=0 tx=0).stop 2023-10-19T12:45:06.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.302+0000 7efd41fe2640 1 -- 172.21.15.57:0/413046614 shutdown_connections 2023-10-19T12:45:06.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.302+0000 7efd41fe2640 1 --2- 172.21.15.57:0/413046614 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7efd2806ae70 0x7efd2806d330 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:06.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.302+0000 7efd41fe2640 1 --2- 172.21.15.57:0/413046614 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7efd3c15eba0 0x7efd3c14ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:06.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.302+0000 7efd41fe2640 1 --2- 172.21.15.57:0/413046614 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efd3c15b200 0x7efd3c1f4b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:06.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.302+0000 7efd41fe2640 1 --2- 172.21.15.57:0/413046614 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efd3c0cf2a0 0x7efd3c0af210 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:06.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.302+0000 7efd41fe2640 1 -- 172.21.15.57:0/413046614 >> 172.21.15.57:0/413046614 conn(0x7efd3c08a650 msgr2=0x7efd3c0cd9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:06.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.302+0000 7efd41fe2640 1 -- 172.21.15.57:0/413046614 shutdown_connections 2023-10-19T12:45:06.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.302+0000 7efd41fe2640 1 -- 172.21.15.57:0/413046614 wait complete. 2023-10-19T12:45:06.315 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-19T12:45:06.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.425+0000 7f879bebb640 1 -- 172.21.15.57:0/1490123345 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f877c003970 msgr2=0x7f877c003d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:06.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.425+0000 7f879bebb640 1 --2- 172.21.15.57:0/1490123345 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f877c003970 0x7f877c003d70 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f878c006fc0 tx=0x7f877c0057f0 comp rx=0 tx=0).stop 2023-10-19T12:45:06.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.425+0000 7f879bebb640 1 -- 172.21.15.57:0/1490123345 shutdown_connections 2023-10-19T12:45:06.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.425+0000 7f879bebb640 1 --2- 172.21.15.57:0/1490123345 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f877c003970 0x7f877c003d70 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:06.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.425+0000 7f879bebb640 1 -- 172.21.15.57:0/1490123345 >> 172.21.15.57:0/1490123345 conn(0x7f87940d0d60 msgr2=0x7f87940d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:06.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.425+0000 7f879bebb640 1 -- 172.21.15.57:0/1490123345 shutdown_connections 2023-10-19T12:45:06.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.425+0000 7f879bebb640 1 -- 172.21.15.57:0/1490123345 wait complete. 2023-10-19T12:45:06.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.426+0000 7f879bebb640 1 Processor -- start 2023-10-19T12:45:06.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.426+0000 7f879bebb640 1 -- start start 2023-10-19T12:45:06.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.427+0000 7f879bebb640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f877c003970 0x7f8794158500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:06.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.427+0000 7f879bebb640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f879415e330 0x7f879406ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:06.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.427+0000 7f879bebb640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f879415d300 0x7f879406b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:06.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.427+0000 7f879bebb640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f879415c5f0 con 0x7f879415d300 2023-10-19T12:45:06.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.427+0000 7f879bebb640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f879415b9e0 con 0x7f877c003970 2023-10-19T12:45:06.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.427+0000 7f879bebb640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f879415bcc0 con 0x7f879415e330 2023-10-19T12:45:06.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.427+0000 7f879a458640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f879415d300 0x7f879406b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:06.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.427+0000 7f879a458640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f879415d300 0x7f879406b260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50562/0 (socket says 172.21.15.57:50562) 2023-10-19T12:45:06.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.427+0000 7f879a458640 1 -- 172.21.15.57:0/4026501398 learned_addr learned my addr 172.21.15.57:0/4026501398 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:06.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.427+0000 7f8799c57640 1 --2- 172.21.15.57:0/4026501398 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f877c003970 0x7f8794158500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:06.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.427+0000 7f8799456640 1 --2- 172.21.15.57:0/4026501398 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f879415e330 0x7f879406ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:06.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.427+0000 7f879a458640 1 -- 172.21.15.57:0/4026501398 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f879415e330 msgr2=0x7f879406ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:06.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.427+0000 7f879a458640 1 --2- 172.21.15.57:0/4026501398 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f879415e330 0x7f879406ad20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:06.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.427+0000 7f879a458640 1 -- 172.21.15.57:0/4026501398 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f877c003970 msgr2=0x7f8794158500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:06.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.427+0000 7f879a458640 1 --2- 172.21.15.57:0/4026501398 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f877c003970 0x7f8794158500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:06.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.427+0000 7f879a458640 1 -- 172.21.15.57:0/4026501398 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f879415bfa0 con 0x7f879415d300 2023-10-19T12:45:06.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.427+0000 7f8799c57640 1 --2- 172.21.15.57:0/4026501398 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f877c003970 0x7f8794158500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:45:06.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.427+0000 7f879a458640 1 --2- 172.21.15.57:0/4026501398 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f879415d300 0x7f879406b260 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f879406c850 tx=0x7f878400d970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:06.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.427+0000 7f87927fc640 1 -- 172.21.15.57:0/4026501398 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f878400f880 con 0x7f879415d300 2023-10-19T12:45:06.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.427+0000 7f87927fc640 1 -- 172.21.15.57:0/4026501398 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f878400fad0 con 0x7f879415d300 2023-10-19T12:45:06.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.427+0000 7f879bebb640 1 -- 172.21.15.57:0/4026501398 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f879415c280 con 0x7f879415d300 2023-10-19T12:45:06.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.427+0000 7f87927fc640 1 -- 172.21.15.57:0/4026501398 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8784010450 con 0x7f879415d300 2023-10-19T12:45:06.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.428+0000 7f879bebb640 1 -- 172.21.15.57:0/4026501398 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87940c7e30 con 0x7f879415d300 2023-10-19T12:45:06.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.428+0000 7f879bebb640 1 -- 172.21.15.57:0/4026501398 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f879415c410 con 0x7f879415d300 2023-10-19T12:45:06.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.429+0000 7f87927fc640 1 -- 172.21.15.57:0/4026501398 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f87840105b0 con 0x7f879415d300 2023-10-19T12:45:06.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.429+0000 7f87927fc640 1 --2- 172.21.15.57:0/4026501398 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f878006ae70 0x7f878006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:06.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.429+0000 7f87927fc640 1 -- 172.21.15.57:0/4026501398 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f8784089e90 con 0x7f879415d300 2023-10-19T12:45:06.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.430+0000 7f8799c57640 1 --2- 172.21.15.57:0/4026501398 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f878006ae70 0x7f878006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:06.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.430+0000 7f8799c57640 1 --2- 172.21.15.57:0/4026501398 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f878006ae70 0x7f878006d330 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f878c034000 tx=0x7f878c036040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:06.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.433+0000 7f87927fc640 1 -- 172.21.15.57:0/4026501398 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8784054af0 con 0x7f879415d300 2023-10-19T12:45:06.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.608+0000 7f879bebb640 1 -- 172.21.15.57:0/4026501398 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f87940d07d0 con 0x7f879415d300 2023-10-19T12:45:06.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.609+0000 7f87927fc640 1 -- 172.21.15.57:0/4026501398 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7f878408b020 con 0x7f879415d300 2023-10-19T12:45:06.612 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 64 2023-10-19T12:45:06.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.611+0000 7f879bebb640 1 -- 172.21.15.57:0/4026501398 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f878006ae70 msgr2=0x7f878006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:06.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.611+0000 7f879bebb640 1 --2- 172.21.15.57:0/4026501398 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f878006ae70 0x7f878006d330 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f878c034000 tx=0x7f878c036040 comp rx=0 tx=0).stop 2023-10-19T12:45:06.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.611+0000 7f879bebb640 1 -- 172.21.15.57:0/4026501398 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f879415d300 msgr2=0x7f879406b260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:06.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.611+0000 7f879bebb640 1 --2- 172.21.15.57:0/4026501398 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f879415d300 0x7f879406b260 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f879406c850 tx=0x7f878400d970 comp rx=0 tx=0).stop 2023-10-19T12:45:06.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.611+0000 7f879bebb640 1 -- 172.21.15.57:0/4026501398 shutdown_connections 2023-10-19T12:45:06.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.611+0000 7f879bebb640 1 --2- 172.21.15.57:0/4026501398 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f878006ae70 0x7f878006d330 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:06.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.611+0000 7f879bebb640 1 --2- 172.21.15.57:0/4026501398 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f879415d300 0x7f879406b260 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:06.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.611+0000 7f879bebb640 1 --2- 172.21.15.57:0/4026501398 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f879415e330 0x7f879406ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:06.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.611+0000 7f879bebb640 1 --2- 172.21.15.57:0/4026501398 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f877c003970 0x7f8794158500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:06.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.611+0000 7f879bebb640 1 -- 172.21.15.57:0/4026501398 >> 172.21.15.57:0/4026501398 conn(0x7f87940d0d60 msgr2=0x7f879415a180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:06.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.611+0000 7f879bebb640 1 -- 172.21.15.57:0/4026501398 shutdown_connections 2023-10-19T12:45:06.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.611+0000 7f879bebb640 1 -- 172.21.15.57:0/4026501398 wait complete. 2023-10-19T12:45:06.625 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-19T12:45:06.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.733+0000 7fb9ce5d0640 1 -- 172.21.15.57:0/2496998376 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb9c8070af0 msgr2=0x7fb9c80affe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:06.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.733+0000 7fb9ce5d0640 1 --2- 172.21.15.57:0/2496998376 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb9c8070af0 0x7fb9c80affe0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fb9c80cbd00 tx=0x7fb9c002c260 comp rx=0 tx=0).stop 2023-10-19T12:45:06.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.733+0000 7fb9ce5d0640 1 -- 172.21.15.57:0/2496998376 shutdown_connections 2023-10-19T12:45:06.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.733+0000 7fb9ce5d0640 1 --2- 172.21.15.57:0/2496998376 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb9c8070af0 0x7fb9c80affe0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:06.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.733+0000 7fb9ce5d0640 1 --2- 172.21.15.57:0/2496998376 >> v2:172.21.15.179:3300/0 conn(0x7fb9c813e520 0x7fb9c806bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:06.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.733+0000 7fb9ce5d0640 1 -- 172.21.15.57:0/2496998376 >> 172.21.15.57:0/2496998376 conn(0x7fb9c8069930 msgr2=0x7fb9c8069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:06.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.733+0000 7fb9ce5d0640 1 -- 172.21.15.57:0/2496998376 shutdown_connections 2023-10-19T12:45:06.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.733+0000 7fb9ce5d0640 1 -- 172.21.15.57:0/2496998376 wait complete. 2023-10-19T12:45:06.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.734+0000 7fb9ce5d0640 1 Processor -- start 2023-10-19T12:45:06.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.735+0000 7fb9ce5d0640 1 -- start start 2023-10-19T12:45:06.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.735+0000 7fb9ce5d0640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb9c8070af0 0x7fb9c8146f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:06.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.735+0000 7fb9ce5d0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb9c813e520 0x7fb9c813b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:06.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.735+0000 7fb9ce5d0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb9c8141690 0x7fb9c80aaef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:06.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.735+0000 7fb9ce5d0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c81e2340 con 0x7fb9c813e520 2023-10-19T12:45:06.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.735+0000 7fb9ce5d0640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c80a9040 con 0x7fb9c8141690 2023-10-19T12:45:06.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.735+0000 7fb9ce5d0640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c80a86c0 con 0x7fb9c8070af0 2023-10-19T12:45:06.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.735+0000 7fb9c77fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb9c813e520 0x7fb9c813b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:06.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.735+0000 7fb9c77fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb9c813e520 0x7fb9c813b1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59726/0 (socket says 172.21.15.57:59726) 2023-10-19T12:45:06.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.735+0000 7fb9c77fe640 1 -- 172.21.15.57:0/228269308 learned_addr learned my addr 172.21.15.57:0/228269308 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:06.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.736+0000 7fb9ccb6d640 1 --2- 172.21.15.57:0/228269308 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb9c8141690 0x7fb9c80aaef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:06.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.736+0000 7fb9c7fff640 1 --2- 172.21.15.57:0/228269308 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb9c8070af0 0x7fb9c8146f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:06.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.736+0000 7fb9c77fe640 1 -- 172.21.15.57:0/228269308 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb9c8070af0 msgr2=0x7fb9c8146f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:06.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.736+0000 7fb9c77fe640 1 --2- 172.21.15.57:0/228269308 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb9c8070af0 0x7fb9c8146f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:06.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.736+0000 7fb9c77fe640 1 -- 172.21.15.57:0/228269308 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb9c8141690 msgr2=0x7fb9c80aaef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:06.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.736+0000 7fb9c77fe640 1 --2- 172.21.15.57:0/228269308 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb9c8141690 0x7fb9c80aaef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:06.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.736+0000 7fb9c77fe640 1 -- 172.21.15.57:0/228269308 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9c0009900 con 0x7fb9c813e520 2023-10-19T12:45:06.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.736+0000 7fb9c77fe640 1 --2- 172.21.15.57:0/228269308 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb9c813e520 0x7fb9c813b1b0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fb9b000c440 tx=0x7fb9b000c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:06.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.736+0000 7fb9c4ff9640 1 -- 172.21.15.57:0/228269308 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9b0005480 con 0x7fb9c813e520 2023-10-19T12:45:06.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.736+0000 7fb9c4ff9640 1 -- 172.21.15.57:0/228269308 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fb9b00035f0 con 0x7fb9c813e520 2023-10-19T12:45:06.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.736+0000 7fb9ce5d0640 1 -- 172.21.15.57:0/228269308 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9c80ab970 con 0x7fb9c813e520 2023-10-19T12:45:06.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.736+0000 7fb9c4ff9640 1 -- 172.21.15.57:0/228269308 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9b0003a00 con 0x7fb9c813e520 2023-10-19T12:45:06.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.736+0000 7fb9ce5d0640 1 -- 172.21.15.57:0/228269308 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9c80a8c60 con 0x7fb9c813e520 2023-10-19T12:45:06.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.737+0000 7fb9ce5d0640 1 -- 172.21.15.57:0/228269308 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9c8067930 con 0x7fb9c813e520 2023-10-19T12:45:06.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.738+0000 7fb9c4ff9640 1 -- 172.21.15.57:0/228269308 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fb9b0003b60 con 0x7fb9c813e520 2023-10-19T12:45:06.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.738+0000 7fb9c4ff9640 1 --2- 172.21.15.57:0/228269308 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fb9a806ae40 0x7fb9a806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:06.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.738+0000 7fb9c4ff9640 1 -- 172.21.15.57:0/228269308 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fb9b0088ff0 con 0x7fb9c813e520 2023-10-19T12:45:06.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.738+0000 7fb9c7fff640 1 --2- 172.21.15.57:0/228269308 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fb9a806ae40 0x7fb9a806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:06.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.739+0000 7fb9c7fff640 1 --2- 172.21.15.57:0/228269308 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fb9a806ae40 0x7fb9a806d300 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fb9b80058c0 tx=0x7fb9b800a2d0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:06.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.742+0000 7fb9c4ff9640 1 -- 172.21.15.57:0/228269308 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb9b0053e20 con 0x7fb9c813e520 2023-10-19T12:45:06.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.916+0000 7fb9ce5d0640 1 -- 172.21.15.57:0/228269308 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fb9c80690e0 con 0x7fb9c813e520 2023-10-19T12:45:06.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.917+0000 7fb9c4ff9640 1 -- 172.21.15.57:0/228269308 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7fb9b0017020 con 0x7fb9c813e520 2023-10-19T12:45:06.920 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2023-10-19T12:45:06.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.918+0000 7fb9ce5d0640 1 -- 172.21.15.57:0/228269308 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fb9a806ae40 msgr2=0x7fb9a806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:06.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.918+0000 7fb9ce5d0640 1 --2- 172.21.15.57:0/228269308 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fb9a806ae40 0x7fb9a806d300 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fb9b80058c0 tx=0x7fb9b800a2d0 comp rx=0 tx=0).stop 2023-10-19T12:45:06.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.918+0000 7fb9ce5d0640 1 -- 172.21.15.57:0/228269308 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb9c813e520 msgr2=0x7fb9c813b1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:06.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.918+0000 7fb9ce5d0640 1 --2- 172.21.15.57:0/228269308 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb9c813e520 0x7fb9c813b1b0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fb9b000c440 tx=0x7fb9b000c900 comp rx=0 tx=0).stop 2023-10-19T12:45:06.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.918+0000 7fb9ce5d0640 1 -- 172.21.15.57:0/228269308 shutdown_connections 2023-10-19T12:45:06.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.918+0000 7fb9ce5d0640 1 --2- 172.21.15.57:0/228269308 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fb9a806ae40 0x7fb9a806d300 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:06.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.918+0000 7fb9ce5d0640 1 --2- 172.21.15.57:0/228269308 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb9c8141690 0x7fb9c80aaef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:06.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.918+0000 7fb9ce5d0640 1 --2- 172.21.15.57:0/228269308 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb9c813e520 0x7fb9c813b1b0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:06.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.918+0000 7fb9ce5d0640 1 --2- 172.21.15.57:0/228269308 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb9c8070af0 0x7fb9c8146f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:06.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.918+0000 7fb9ce5d0640 1 -- 172.21.15.57:0/228269308 >> 172.21.15.57:0/228269308 conn(0x7fb9c8069930 msgr2=0x7fb9c806f080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:06.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.918+0000 7fb9ce5d0640 1 -- 172.21.15.57:0/228269308 shutdown_connections 2023-10-19T12:45:06.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:06.918+0000 7fb9ce5d0640 1 -- 172.21.15.57:0/228269308 wait complete. 2023-10-19T12:45:06.942 INFO:tasks.cephfs_test_runner:Starting test: test_concurrent_snap_creates (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test concurrent snap creates in same file-system without db issues 2023-10-19T12:45:06.943 INFO:tasks.cephfs_test_runner:test_concurrent_snap_creates (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-10-19T12:45:06.943 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T12:45:07.054 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.050+0000 7f95e692c640 1 -- 172.21.15.57:0/561795939 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f95c80047c0 msgr2=0x7f95c8004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:07.054 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.050+0000 7f95e692c640 1 --2- 172.21.15.57:0/561795939 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f95c80047c0 0x7f95c8004c40 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f95d0009b50 tx=0x7f95c80057f0 comp rx=0 tx=0).stop 2023-10-19T12:45:07.054 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.050+0000 7f95e692c640 1 -- 172.21.15.57:0/561795939 shutdown_connections 2023-10-19T12:45:07.055 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.050+0000 7f95e692c640 1 --2- 172.21.15.57:0/561795939 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f95c80047c0 0x7f95c8004c40 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:07.055 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.050+0000 7f95e692c640 1 -- 172.21.15.57:0/561795939 >> 172.21.15.57:0/561795939 conn(0x7f95e00d0cf0 msgr2=0x7f95e00d1100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:07.055 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.051+0000 7f95e692c640 1 -- 172.21.15.57:0/561795939 shutdown_connections 2023-10-19T12:45:07.055 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.051+0000 7f95e692c640 1 -- 172.21.15.57:0/561795939 wait complete. 2023-10-19T12:45:07.057 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.052+0000 7f95e692c640 1 Processor -- start 2023-10-19T12:45:07.057 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.052+0000 7f95e692c640 1 -- start start 2023-10-19T12:45:07.058 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.053+0000 7f95e692c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f95c80047c0 0x7f95e009e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:07.058 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.053+0000 7f95e692c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f95e00ca350 0x7f95e00ca7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:07.058 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.053+0000 7f95e692c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f95e00c82f0 0x7f95e00c8770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:07.058 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.053+0000 7f95e692c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95e00c8180 con 0x7f95c80047c0 2023-10-19T12:45:07.058 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.053+0000 7f95e692c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95e00cad10 con 0x7f95e00c82f0 2023-10-19T12:45:07.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.053+0000 7f95e692c640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95e00c9b80 con 0x7f95e00ca350 2023-10-19T12:45:07.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.053+0000 7f95dffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f95c80047c0 0x7f95e009e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:07.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.053+0000 7f95dffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f95c80047c0 0x7f95e009e4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59744/0 (socket says 172.21.15.57:59744) 2023-10-19T12:45:07.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.053+0000 7f95dffff640 1 -- 172.21.15.57:0/2018426360 learned_addr learned my addr 172.21.15.57:0/2018426360 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:07.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.053+0000 7f95e4ec9640 1 --2- 172.21.15.57:0/2018426360 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f95e00c82f0 0x7f95e00c8770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:07.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.053+0000 7f95df7fe640 1 --2- 172.21.15.57:0/2018426360 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f95e00ca350 0x7f95e00ca7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:07.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.053+0000 7f95dffff640 1 -- 172.21.15.57:0/2018426360 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f95e00ca350 msgr2=0x7f95e00ca7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:07.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.053+0000 7f95dffff640 1 --2- 172.21.15.57:0/2018426360 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f95e00ca350 0x7f95e00ca7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:07.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.053+0000 7f95dffff640 1 -- 172.21.15.57:0/2018426360 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f95e00c82f0 msgr2=0x7f95e00c8770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:07.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.053+0000 7f95dffff640 1 --2- 172.21.15.57:0/2018426360 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f95e00c82f0 0x7f95e00c8770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:07.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.053+0000 7f95dffff640 1 -- 172.21.15.57:0/2018426360 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95e009e9f0 con 0x7f95c80047c0 2023-10-19T12:45:07.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.053+0000 7f95dffff640 1 --2- 172.21.15.57:0/2018426360 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f95c80047c0 0x7f95e009e4b0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f95d4004ce0 tx=0x7f95d4001630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:07.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.053+0000 7f95dcff9640 1 -- 172.21.15.57:0/2018426360 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95d4011660 con 0x7f95c80047c0 2023-10-19T12:45:07.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.053+0000 7f95dcff9640 1 -- 172.21.15.57:0/2018426360 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f95d40118b0 con 0x7f95c80047c0 2023-10-19T12:45:07.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.053+0000 7f95e692c640 1 -- 172.21.15.57:0/2018426360 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95e00c8cb0 con 0x7f95c80047c0 2023-10-19T12:45:07.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.053+0000 7f95dcff9640 1 -- 172.21.15.57:0/2018426360 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95d4011ca0 con 0x7f95c80047c0 2023-10-19T12:45:07.062 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.053+0000 7f95e692c640 1 -- 172.21.15.57:0/2018426360 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95e00c9250 con 0x7f95c80047c0 2023-10-19T12:45:07.062 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.054+0000 7f95e692c640 1 -- 172.21.15.57:0/2018426360 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95e00690e0 con 0x7f95c80047c0 2023-10-19T12:45:07.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.055+0000 7f95dcff9640 1 -- 172.21.15.57:0/2018426360 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f95d401a400 con 0x7f95c80047c0 2023-10-19T12:45:07.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.055+0000 7f95dcff9640 1 --2- 172.21.15.57:0/2018426360 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f95cc06ae70 0x7f95cc06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:07.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.055+0000 7f95dcff9640 1 -- 172.21.15.57:0/2018426360 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f95d408a690 con 0x7f95c80047c0 2023-10-19T12:45:07.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.055+0000 7f95df7fe640 1 --2- 172.21.15.57:0/2018426360 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f95cc06ae70 0x7f95cc06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:07.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.056+0000 7f95df7fe640 1 --2- 172.21.15.57:0/2018426360 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f95cc06ae70 0x7f95cc06d330 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f95d0036000 tx=0x7f95d00046c0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:07.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.059+0000 7f95dcff9640 1 -- 172.21.15.57:0/2018426360 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f95d4055300 con 0x7f95c80047c0 2023-10-19T12:45:07.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.241+0000 7f95e692c640 1 -- 172.21.15.57:0/2018426360 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f95e015a7b0 con 0x7f95c80047c0 2023-10-19T12:45:07.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.243+0000 7f95dcff9640 1 -- 172.21.15.57:0/2018426360 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18748 (secure 0 0 0) 0x7f95d4058920 con 0x7f95c80047c0 2023-10-19T12:45:07.246 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:45:07.246 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":27,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T12:44:58.978403+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-19T12:44:16.517510+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-19T12:44:17.398297+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-19T12:44:21.129403+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.23","mappings":[{"from":3,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"3.39","mappings":[{"from":7,"to":1}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.1d","mappings":[{"from":1,"to":5}]},{"pgid":"4.1e","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T12:45:07.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.245+0000 7f95e692c640 1 -- 172.21.15.57:0/2018426360 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f95cc06ae70 msgr2=0x7f95cc06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:07.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.245+0000 7f95e692c640 1 --2- 172.21.15.57:0/2018426360 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f95cc06ae70 0x7f95cc06d330 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f95d0036000 tx=0x7f95d00046c0 comp rx=0 tx=0).stop 2023-10-19T12:45:07.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.245+0000 7f95e692c640 1 -- 172.21.15.57:0/2018426360 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f95c80047c0 msgr2=0x7f95e009e4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:07.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.245+0000 7f95e692c640 1 --2- 172.21.15.57:0/2018426360 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f95c80047c0 0x7f95e009e4b0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f95d4004ce0 tx=0x7f95d4001630 comp rx=0 tx=0).stop 2023-10-19T12:45:07.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.245+0000 7f95e692c640 1 -- 172.21.15.57:0/2018426360 shutdown_connections 2023-10-19T12:45:07.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.245+0000 7f95e692c640 1 --2- 172.21.15.57:0/2018426360 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f95cc06ae70 0x7f95cc06d330 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:07.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.245+0000 7f95e692c640 1 --2- 172.21.15.57:0/2018426360 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f95e00c82f0 0x7f95e00c8770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:07.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.245+0000 7f95e692c640 1 --2- 172.21.15.57:0/2018426360 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f95e00ca350 0x7f95e00ca7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:07.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.245+0000 7f95e692c640 1 --2- 172.21.15.57:0/2018426360 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f95c80047c0 0x7f95e009e4b0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:07.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.245+0000 7f95e692c640 1 -- 172.21.15.57:0/2018426360 >> 172.21.15.57:0/2018426360 conn(0x7f95e00d0cf0 msgr2=0x7f95e00af620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:07.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.245+0000 7f95e692c640 1 -- 172.21.15.57:0/2018426360 shutdown_connections 2023-10-19T12:45:07.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.245+0000 7f95e692c640 1 -- 172.21.15.57:0/2018426360 wait complete. 2023-10-19T12:45:07.259 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-19T12:44:07.371186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-19T12:44:16.517510+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-19T12:44:17.398297+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-19T12:44:21.129403+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-19T12:45:07.260 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-19T12:45:07.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.368+0000 7fda9caf2640 1 -- 172.21.15.57:0/2326323298 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fda981579f0 msgr2=0x7fda980b0e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:07.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.368+0000 7fda9caf2640 1 --2- 172.21.15.57:0/2326323298 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fda981579f0 0x7fda980b0e90 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fda88009830 tx=0x7fda8802bd60 comp rx=0 tx=0).stop 2023-10-19T12:45:07.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.368+0000 7fda9caf2640 1 -- 172.21.15.57:0/2326323298 shutdown_connections 2023-10-19T12:45:07.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.368+0000 7fda9caf2640 1 --2- 172.21.15.57:0/2326323298 >> v2:172.21.15.179:3300/0 conn(0x7fda9815c1f0 0x7fda9806a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:07.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.368+0000 7fda9caf2640 1 --2- 172.21.15.57:0/2326323298 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fda981579f0 0x7fda980b0e90 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:07.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.368+0000 7fda9caf2640 1 -- 172.21.15.57:0/2326323298 >> 172.21.15.57:0/2326323298 conn(0x7fda9806fae0 msgr2=0x7fda9806fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:07.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.368+0000 7fda9caf2640 1 -- 172.21.15.57:0/2326323298 shutdown_connections 2023-10-19T12:45:07.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.368+0000 7fda9caf2640 1 -- 172.21.15.57:0/2326323298 wait complete. 2023-10-19T12:45:07.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.370+0000 7fda9caf2640 1 Processor -- start 2023-10-19T12:45:07.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.370+0000 7fda9caf2640 1 -- start start 2023-10-19T12:45:07.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.370+0000 7fda9caf2640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fda9806ac70 0x7fda981466f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:07.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.370+0000 7fda9caf2640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fda981579f0 0x7fda9814b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:07.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.370+0000 7fda9caf2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fda9815c1f0 0x7fda9814baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:07.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.370+0000 7fda9caf2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda98145c10 con 0x7fda9815c1f0 2023-10-19T12:45:07.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.370+0000 7fda9caf2640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda9814c400 con 0x7fda981579f0 2023-10-19T12:45:07.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.370+0000 7fda9caf2640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda9814f5c0 con 0x7fda9806ac70 2023-10-19T12:45:07.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.371+0000 7fda96d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fda9815c1f0 0x7fda9814baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:07.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.371+0000 7fda96d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fda9815c1f0 0x7fda9814baf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59758/0 (socket says 172.21.15.57:59758) 2023-10-19T12:45:07.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.371+0000 7fda96d9d640 1 -- 172.21.15.57:0/3348032404 learned_addr learned my addr 172.21.15.57:0/3348032404 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:07.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.371+0000 7fda8dd9b640 1 --2- 172.21.15.57:0/3348032404 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fda981579f0 0x7fda9814b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:07.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.371+0000 7fda9659c640 1 --2- 172.21.15.57:0/3348032404 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fda9806ac70 0x7fda981466f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:07.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.371+0000 7fda96d9d640 1 -- 172.21.15.57:0/3348032404 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fda9806ac70 msgr2=0x7fda981466f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:07.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.371+0000 7fda96d9d640 1 --2- 172.21.15.57:0/3348032404 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fda9806ac70 0x7fda981466f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:07.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.371+0000 7fda96d9d640 1 -- 172.21.15.57:0/3348032404 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fda981579f0 msgr2=0x7fda9814b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:07.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.371+0000 7fda96d9d640 1 --2- 172.21.15.57:0/3348032404 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fda981579f0 0x7fda9814b5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:07.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.371+0000 7fda96d9d640 1 -- 172.21.15.57:0/3348032404 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda88009490 con 0x7fda9815c1f0 2023-10-19T12:45:07.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.371+0000 7fda96d9d640 1 --2- 172.21.15.57:0/3348032404 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fda9815c1f0 0x7fda9814baf0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fda9000cbf0 tx=0x7fda9000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:07.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.371+0000 7fda8f7fe640 1 -- 172.21.15.57:0/3348032404 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda9000aca0 con 0x7fda9815c1f0 2023-10-19T12:45:07.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.371+0000 7fda8f7fe640 1 -- 172.21.15.57:0/3348032404 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fda90003970 con 0x7fda9815c1f0 2023-10-19T12:45:07.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.371+0000 7fda9caf2640 1 -- 172.21.15.57:0/3348032404 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda9814f860 con 0x7fda9815c1f0 2023-10-19T12:45:07.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.371+0000 7fda8f7fe640 1 -- 172.21.15.57:0/3348032404 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda90003d80 con 0x7fda9815c1f0 2023-10-19T12:45:07.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.371+0000 7fda9caf2640 1 -- 172.21.15.57:0/3348032404 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda9814fb00 con 0x7fda9815c1f0 2023-10-19T12:45:07.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.372+0000 7fda9caf2640 1 -- 172.21.15.57:0/3348032404 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda980690e0 con 0x7fda9815c1f0 2023-10-19T12:45:07.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.373+0000 7fda8f7fe640 1 -- 172.21.15.57:0/3348032404 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fda90004030 con 0x7fda9815c1f0 2023-10-19T12:45:07.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.373+0000 7fda8f7fe640 1 --2- 172.21.15.57:0/3348032404 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fda7806ae40 0x7fda7806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:07.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.373+0000 7fda8f7fe640 1 -- 172.21.15.57:0/3348032404 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fda9008a2b0 con 0x7fda9815c1f0 2023-10-19T12:45:07.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.373+0000 7fda9659c640 1 --2- 172.21.15.57:0/3348032404 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fda7806ae40 0x7fda7806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:07.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.374+0000 7fda9659c640 1 --2- 172.21.15.57:0/3348032404 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fda7806ae40 0x7fda7806d300 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fda88036000 tx=0x7fda88004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:07.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.377+0000 7fda8f7fe640 1 -- 172.21.15.57:0/3348032404 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fda90055050 con 0x7fda9815c1f0 2023-10-19T12:45:07.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.556+0000 7fda9caf2640 1 -- 172.21.15.57:0/3348032404 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fda9806cc40 con 0x7fda9815c1f0 2023-10-19T12:45:07.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.556+0000 7fda8f7fe640 1 -- 172.21.15.57:0/3348032404 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7fda90058670 con 0x7fda9815c1f0 2023-10-19T12:45:07.560 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-19T12:45:07.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.558+0000 7fda9caf2640 1 -- 172.21.15.57:0/3348032404 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fda7806ae40 msgr2=0x7fda7806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:07.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.558+0000 7fda9caf2640 1 --2- 172.21.15.57:0/3348032404 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fda7806ae40 0x7fda7806d300 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fda88036000 tx=0x7fda88004730 comp rx=0 tx=0).stop 2023-10-19T12:45:07.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.558+0000 7fda9caf2640 1 -- 172.21.15.57:0/3348032404 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fda9815c1f0 msgr2=0x7fda9814baf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:07.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.558+0000 7fda9caf2640 1 --2- 172.21.15.57:0/3348032404 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fda9815c1f0 0x7fda9814baf0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fda9000cbf0 tx=0x7fda9000a060 comp rx=0 tx=0).stop 2023-10-19T12:45:07.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.558+0000 7fda9caf2640 1 -- 172.21.15.57:0/3348032404 shutdown_connections 2023-10-19T12:45:07.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.558+0000 7fda9caf2640 1 --2- 172.21.15.57:0/3348032404 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fda7806ae40 0x7fda7806d300 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:07.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.558+0000 7fda9caf2640 1 --2- 172.21.15.57:0/3348032404 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fda9815c1f0 0x7fda9814baf0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:07.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.558+0000 7fda9caf2640 1 --2- 172.21.15.57:0/3348032404 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fda981579f0 0x7fda9814b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:07.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.558+0000 7fda9caf2640 1 --2- 172.21.15.57:0/3348032404 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fda9806ac70 0x7fda981466f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:07.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.558+0000 7fda9caf2640 1 -- 172.21.15.57:0/3348032404 >> 172.21.15.57:0/3348032404 conn(0x7fda9806fae0 msgr2=0x7fda980cb8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:07.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.558+0000 7fda9caf2640 1 -- 172.21.15.57:0/3348032404 shutdown_connections 2023-10-19T12:45:07.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.558+0000 7fda9caf2640 1 -- 172.21.15.57:0/3348032404 wait complete. 2023-10-19T12:45:07.571 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-19T12:45:07.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.681+0000 7f68affb6640 1 -- 172.21.15.57:0/3353074001 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6890005180 msgr2=0x7f689000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:07.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.681+0000 7f68affb6640 1 --2- 172.21.15.57:0/3353074001 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6890005180 0x7f689000b980 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f6898001d80 tx=0x7f68900057f0 comp rx=0 tx=0).stop 2023-10-19T12:45:07.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.681+0000 7f68affb6640 1 -- 172.21.15.57:0/3353074001 shutdown_connections 2023-10-19T12:45:07.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.681+0000 7f68affb6640 1 --2- 172.21.15.57:0/3353074001 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6890005180 0x7f689000b980 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:07.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.681+0000 7f68affb6640 1 -- 172.21.15.57:0/3353074001 >> 172.21.15.57:0/3353074001 conn(0x7f68a80af7a0 msgr2=0x7f68a80afb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:07.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.681+0000 7f68affb6640 1 -- 172.21.15.57:0/3353074001 shutdown_connections 2023-10-19T12:45:07.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.681+0000 7f68affb6640 1 -- 172.21.15.57:0/3353074001 wait complete. 2023-10-19T12:45:07.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.682+0000 7f68affb6640 1 Processor -- start 2023-10-19T12:45:07.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.682+0000 7f68affb6640 1 -- start start 2023-10-19T12:45:07.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.683+0000 7f68affb6640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6890005180 0x7f68a814f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:07.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.683+0000 7f68affb6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f68a80c7c20 0x7f68a814d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:07.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.683+0000 7f68affb6640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f68a815a550 0x7f68a815a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:07.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.683+0000 7f68affb6640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68a815af10 con 0x7f68a80c7c20 2023-10-19T12:45:07.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.683+0000 7f68affb6640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68a80c84a0 con 0x7f6890005180 2023-10-19T12:45:07.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.683+0000 7f68affb6640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68a80c8780 con 0x7f68a815a550 2023-10-19T12:45:07.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.683+0000 7f68ad551640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f68a80c7c20 0x7f68a814d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:07.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.683+0000 7f68ad551640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f68a80c7c20 0x7f68a814d600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59776/0 (socket says 172.21.15.57:59776) 2023-10-19T12:45:07.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.683+0000 7f68ad551640 1 -- 172.21.15.57:0/3821896004 learned_addr learned my addr 172.21.15.57:0/3821896004 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:07.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.683+0000 7f68add52640 1 --2- 172.21.15.57:0/3821896004 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6890005180 0x7f68a814f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:07.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.683+0000 7f68ae553640 1 --2- 172.21.15.57:0/3821896004 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f68a815a550 0x7f68a815a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:07.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.683+0000 7f68ad551640 1 -- 172.21.15.57:0/3821896004 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f68a815a550 msgr2=0x7f68a815a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:07.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.683+0000 7f68ad551640 1 --2- 172.21.15.57:0/3821896004 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f68a815a550 0x7f68a815a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:07.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.683+0000 7f68ad551640 1 -- 172.21.15.57:0/3821896004 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6890005180 msgr2=0x7f68a814f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:07.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.683+0000 7f68ad551640 1 --2- 172.21.15.57:0/3821896004 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6890005180 0x7f68a814f640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:07.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.683+0000 7f68ad551640 1 -- 172.21.15.57:0/3821896004 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68a80c8a60 con 0x7f68a80c7c20 2023-10-19T12:45:07.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.683+0000 7f68add52640 1 --2- 172.21.15.57:0/3821896004 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6890005180 0x7f68a814f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:45:07.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.683+0000 7f68ad551640 1 --2- 172.21.15.57:0/3821896004 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f68a80c7c20 0x7f68a814d600 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f689c011ab0 tx=0x7f689c011f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:07.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.684+0000 7f68a67fc640 1 -- 172.21.15.57:0/3821896004 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f689c00fe70 con 0x7f68a80c7c20 2023-10-19T12:45:07.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.684+0000 7f68a67fc640 1 -- 172.21.15.57:0/3821896004 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f689c004050 con 0x7f68a80c7c20 2023-10-19T12:45:07.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.684+0000 7f68affb6640 1 -- 172.21.15.57:0/3821896004 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68a80c8d40 con 0x7f68a80c7c20 2023-10-19T12:45:07.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.684+0000 7f68a67fc640 1 -- 172.21.15.57:0/3821896004 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f689c004440 con 0x7f68a80c7c20 2023-10-19T12:45:07.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.684+0000 7f68affb6640 1 -- 172.21.15.57:0/3821896004 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68a814db40 con 0x7f68a80c7c20 2023-10-19T12:45:07.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.685+0000 7f68affb6640 1 -- 172.21.15.57:0/3821896004 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68a80690e0 con 0x7f68a80c7c20 2023-10-19T12:45:07.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.685+0000 7f68a67fc640 1 -- 172.21.15.57:0/3821896004 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f689c0035f0 con 0x7f68a80c7c20 2023-10-19T12:45:07.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.686+0000 7f68a67fc640 1 --2- 172.21.15.57:0/3821896004 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f689406ae40 0x7f689406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:07.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.686+0000 7f68a67fc640 1 -- 172.21.15.57:0/3821896004 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f689c08a710 con 0x7f68a80c7c20 2023-10-19T12:45:07.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.686+0000 7f68add52640 1 --2- 172.21.15.57:0/3821896004 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f689406ae40 0x7f689406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:07.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.686+0000 7f68add52640 1 --2- 172.21.15.57:0/3821896004 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f689406ae40 0x7f689406d300 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f68a0001050 tx=0x7f68a0001c70 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:07.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.690+0000 7f68a67fc640 1 -- 172.21.15.57:0/3821896004 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f689c0554b0 con 0x7f68a80c7c20 2023-10-19T12:45:07.868 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.865+0000 7f68affb6640 1 -- 172.21.15.57:0/3821896004 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f68a80ce5e0 con 0x7f68a80c7c20 2023-10-19T12:45:07.869 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.865+0000 7f68a67fc640 1 -- 172.21.15.57:0/3821896004 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7f689c058ad0 con 0x7f68a80c7c20 2023-10-19T12:45:07.869 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2023-10-19T12:45:07.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.867+0000 7f68affb6640 1 -- 172.21.15.57:0/3821896004 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f689406ae40 msgr2=0x7f689406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:07.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.867+0000 7f68affb6640 1 --2- 172.21.15.57:0/3821896004 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f689406ae40 0x7f689406d300 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f68a0001050 tx=0x7f68a0001c70 comp rx=0 tx=0).stop 2023-10-19T12:45:07.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.867+0000 7f68affb6640 1 -- 172.21.15.57:0/3821896004 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f68a80c7c20 msgr2=0x7f68a814d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:07.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.867+0000 7f68affb6640 1 --2- 172.21.15.57:0/3821896004 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f68a80c7c20 0x7f68a814d600 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f689c011ab0 tx=0x7f689c011f70 comp rx=0 tx=0).stop 2023-10-19T12:45:07.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.867+0000 7f68affb6640 1 -- 172.21.15.57:0/3821896004 shutdown_connections 2023-10-19T12:45:07.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.867+0000 7f68affb6640 1 --2- 172.21.15.57:0/3821896004 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f689406ae40 0x7f689406d300 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:07.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.867+0000 7f68affb6640 1 --2- 172.21.15.57:0/3821896004 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f68a815a550 0x7f68a815a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:07.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.867+0000 7f68affb6640 1 --2- 172.21.15.57:0/3821896004 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f68a80c7c20 0x7f68a814d600 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:07.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.867+0000 7f68affb6640 1 --2- 172.21.15.57:0/3821896004 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6890005180 0x7f68a814f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:07.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.867+0000 7f68affb6640 1 -- 172.21.15.57:0/3821896004 >> 172.21.15.57:0/3821896004 conn(0x7f68a80af7a0 msgr2=0x7f68a80ad710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:07.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.867+0000 7f68affb6640 1 -- 172.21.15.57:0/3821896004 shutdown_connections 2023-10-19T12:45:07.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.867+0000 7f68affb6640 1 -- 172.21.15.57:0/3821896004 wait complete. 2023-10-19T12:45:07.881 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-19T12:45:07.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.991+0000 7fa0c29b6640 1 -- 172.21.15.57:0/760379060 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0a4004370 msgr2=0x7fa0a4004840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:07.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.991+0000 7fa0c29b6640 1 --2- 172.21.15.57:0/760379060 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0a4004370 0x7fa0a4004840 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fa0b0009b50 tx=0x7fa0a40057f0 comp rx=0 tx=0).stop 2023-10-19T12:45:07.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.991+0000 7fa0c29b6640 1 -- 172.21.15.57:0/760379060 shutdown_connections 2023-10-19T12:45:07.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.991+0000 7fa0c29b6640 1 --2- 172.21.15.57:0/760379060 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0a4004370 0x7fa0a4004840 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:07.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.991+0000 7fa0c29b6640 1 -- 172.21.15.57:0/760379060 >> 172.21.15.57:0/760379060 conn(0x7fa0bc06fae0 msgr2=0x7fa0bc06fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:07.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.991+0000 7fa0c29b6640 1 -- 172.21.15.57:0/760379060 shutdown_connections 2023-10-19T12:45:07.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.991+0000 7fa0c29b6640 1 -- 172.21.15.57:0/760379060 wait complete. 2023-10-19T12:45:07.999 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.992+0000 7fa0c29b6640 1 Processor -- start 2023-10-19T12:45:08.000 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.992+0000 7fa0c29b6640 1 -- start start 2023-10-19T12:45:08.000 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.993+0000 7fa0c29b6640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa0a4004370 0x7fa0bc14b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:08.000 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.993+0000 7fa0c29b6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0bc1f4c10 0x7fa0bc149cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:08.000 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.993+0000 7fa0c29b6640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa0bc14c520 0x7fa0bc06b980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:08.000 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.993+0000 7fa0c29b6640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0bc1f51b0 con 0x7fa0bc1f4c10 2023-10-19T12:45:08.001 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.993+0000 7fa0c29b6640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0bc1f5490 con 0x7fa0a4004370 2023-10-19T12:45:08.001 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.993+0000 7fa0c29b6640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0bc1f5770 con 0x7fa0bc14c520 2023-10-19T12:45:08.001 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.993+0000 7fa0bb7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0bc1f4c10 0x7fa0bc149cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:08.001 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.993+0000 7fa0bb7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0bc1f4c10 0x7fa0bc149cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59800/0 (socket says 172.21.15.57:59800) 2023-10-19T12:45:08.001 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.993+0000 7fa0bb7fe640 1 -- 172.21.15.57:0/2979326330 learned_addr learned my addr 172.21.15.57:0/2979326330 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:08.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.993+0000 7fa0bbfff640 1 --2- 172.21.15.57:0/2979326330 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa0a4004370 0x7fa0bc14b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:08.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.993+0000 7fa0c0f53640 1 --2- 172.21.15.57:0/2979326330 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa0bc14c520 0x7fa0bc06b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:08.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.993+0000 7fa0bb7fe640 1 -- 172.21.15.57:0/2979326330 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa0bc14c520 msgr2=0x7fa0bc06b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:08.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.993+0000 7fa0bb7fe640 1 --2- 172.21.15.57:0/2979326330 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa0bc14c520 0x7fa0bc06b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:08.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.993+0000 7fa0bb7fe640 1 -- 172.21.15.57:0/2979326330 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa0a4004370 msgr2=0x7fa0bc14b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:08.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.993+0000 7fa0bb7fe640 1 --2- 172.21.15.57:0/2979326330 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa0a4004370 0x7fa0bc14b770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:08.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.993+0000 7fa0bb7fe640 1 -- 172.21.15.57:0/2979326330 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0bc0b2330 con 0x7fa0bc1f4c10 2023-10-19T12:45:08.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.993+0000 7fa0bbfff640 1 --2- 172.21.15.57:0/2979326330 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa0a4004370 0x7fa0bc14b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:45:08.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.994+0000 7fa0c0f53640 1 --2- 172.21.15.57:0/2979326330 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa0bc14c520 0x7fa0bc06b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:45:08.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.994+0000 7fa0bb7fe640 1 --2- 172.21.15.57:0/2979326330 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0bc1f4c10 0x7fa0bc149cc0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fa0b0009a70 tx=0x7fa0b0033c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:08.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.994+0000 7fa0b8ff9640 1 -- 172.21.15.57:0/2979326330 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0b0004220 con 0x7fa0bc1f4c10 2023-10-19T12:45:08.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.994+0000 7fa0b8ff9640 1 -- 172.21.15.57:0/2979326330 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fa0b0004380 con 0x7fa0bc1f4c10 2023-10-19T12:45:08.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.994+0000 7fa0c29b6640 1 -- 172.21.15.57:0/2979326330 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0bc1f5a50 con 0x7fa0bc1f4c10 2023-10-19T12:45:08.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.994+0000 7fa0b8ff9640 1 -- 172.21.15.57:0/2979326330 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0b003d940 con 0x7fa0bc1f4c10 2023-10-19T12:45:08.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.994+0000 7fa0c29b6640 1 -- 172.21.15.57:0/2979326330 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0bc14bcb0 con 0x7fa0bc1f4c10 2023-10-19T12:45:08.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.995+0000 7fa0c29b6640 1 -- 172.21.15.57:0/2979326330 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0bc0b37a0 con 0x7fa0bc1f4c10 2023-10-19T12:45:08.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.995+0000 7fa0b8ff9640 1 -- 172.21.15.57:0/2979326330 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fa0b003daa0 con 0x7fa0bc1f4c10 2023-10-19T12:45:08.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.996+0000 7fa0b8ff9640 1 --2- 172.21.15.57:0/2979326330 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fa0a806ae70 0x7fa0a806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:08.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.996+0000 7fa0b8ff9640 1 -- 172.21.15.57:0/2979326330 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fa0b00ac720 con 0x7fa0bc1f4c10 2023-10-19T12:45:08.006 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.996+0000 7fa0bbfff640 1 --2- 172.21.15.57:0/2979326330 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fa0a806ae70 0x7fa0a806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:08.006 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:07.996+0000 7fa0bbfff640 1 --2- 172.21.15.57:0/2979326330 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fa0a806ae70 0x7fa0a806d330 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fa0ac0017a0 tx=0x7fa0ac001be0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:08.006 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.000+0000 7fa0b8ff9640 1 -- 172.21.15.57:0/2979326330 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa0b00774a0 con 0x7fa0bc1f4c10 2023-10-19T12:45:08.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.177+0000 7fa0c29b6640 1 -- 172.21.15.57:0/2979326330 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa0bc0690e0 con 0x7fa0bc1f4c10 2023-10-19T12:45:08.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.177+0000 7fa0b8ff9640 1 -- 172.21.15.57:0/2979326330 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa0b004a0a0 con 0x7fa0bc1f4c10 2023-10-19T12:45:08.181 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 64 2023-10-19T12:45:08.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.179+0000 7fa0c29b6640 1 -- 172.21.15.57:0/2979326330 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fa0a806ae70 msgr2=0x7fa0a806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:08.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.179+0000 7fa0c29b6640 1 --2- 172.21.15.57:0/2979326330 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fa0a806ae70 0x7fa0a806d330 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fa0ac0017a0 tx=0x7fa0ac001be0 comp rx=0 tx=0).stop 2023-10-19T12:45:08.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.179+0000 7fa0c29b6640 1 -- 172.21.15.57:0/2979326330 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0bc1f4c10 msgr2=0x7fa0bc149cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:08.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.179+0000 7fa0c29b6640 1 --2- 172.21.15.57:0/2979326330 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0bc1f4c10 0x7fa0bc149cc0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fa0b0009a70 tx=0x7fa0b0033c90 comp rx=0 tx=0).stop 2023-10-19T12:45:08.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.179+0000 7fa0c29b6640 1 -- 172.21.15.57:0/2979326330 shutdown_connections 2023-10-19T12:45:08.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.179+0000 7fa0c29b6640 1 --2- 172.21.15.57:0/2979326330 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fa0a806ae70 0x7fa0a806d330 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:08.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.179+0000 7fa0c29b6640 1 --2- 172.21.15.57:0/2979326330 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa0bc14c520 0x7fa0bc06b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:08.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.179+0000 7fa0c29b6640 1 --2- 172.21.15.57:0/2979326330 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0bc1f4c10 0x7fa0bc149cc0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:08.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.179+0000 7fa0c29b6640 1 --2- 172.21.15.57:0/2979326330 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa0a4004370 0x7fa0bc14b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:08.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.179+0000 7fa0c29b6640 1 -- 172.21.15.57:0/2979326330 >> 172.21.15.57:0/2979326330 conn(0x7fa0bc06fae0 msgr2=0x7fa0bc15a9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:08.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.179+0000 7fa0c29b6640 1 -- 172.21.15.57:0/2979326330 shutdown_connections 2023-10-19T12:45:08.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.179+0000 7fa0c29b6640 1 -- 172.21.15.57:0/2979326330 wait complete. 2023-10-19T12:45:08.193 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-19T12:45:08.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.301+0000 7fab10ac2640 1 -- 172.21.15.57:0/3514426579 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab0c0b0530 msgr2=0x7fab0c0b09a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:08.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.301+0000 7fab10ac2640 1 --2- 172.21.15.57:0/3514426579 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab0c0b0530 0x7fab0c0b09a0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fab0c0cc490 tx=0x7faafc02c110 comp rx=0 tx=0).stop 2023-10-19T12:45:08.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.301+0000 7fab10ac2640 1 -- 172.21.15.57:0/3514426579 shutdown_connections 2023-10-19T12:45:08.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.301+0000 7fab10ac2640 1 --2- 172.21.15.57:0/3514426579 >> v2:172.21.15.179:3300/0 conn(0x7fab0c0afa20 0x7fab0c158300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:08.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.301+0000 7fab10ac2640 1 --2- 172.21.15.57:0/3514426579 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab0c0b0530 0x7fab0c0b09a0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:08.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.301+0000 7fab10ac2640 1 -- 172.21.15.57:0/3514426579 >> 172.21.15.57:0/3514426579 conn(0x7fab0c069930 msgr2=0x7fab0c069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:08.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.301+0000 7fab10ac2640 1 -- 172.21.15.57:0/3514426579 shutdown_connections 2023-10-19T12:45:08.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.301+0000 7fab10ac2640 1 -- 172.21.15.57:0/3514426579 wait complete. 2023-10-19T12:45:08.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.302+0000 7fab10ac2640 1 Processor -- start 2023-10-19T12:45:08.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.302+0000 7fab10ac2640 1 -- start start 2023-10-19T12:45:08.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.303+0000 7fab10ac2640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fab0c0afa20 0x7fab0c1399a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:08.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.303+0000 7fab10ac2640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fab0c0b0530 0x7fab0c1376e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:08.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.303+0000 7fab10ac2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab0c0cbd20 0x7fab0c138000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:08.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.303+0000 7fab10ac2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab0c138900 con 0x7fab0c0cbd20 2023-10-19T12:45:08.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.303+0000 7fab10ac2640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab0c138be0 con 0x7fab0c0afa20 2023-10-19T12:45:08.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.303+0000 7fab10ac2640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab0c138ec0 con 0x7fab0c0b0530 2023-10-19T12:45:08.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.303+0000 7fab0ad9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab0c0cbd20 0x7fab0c138000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:08.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.303+0000 7fab0ad9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab0c0cbd20 0x7fab0c138000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59828/0 (socket says 172.21.15.57:59828) 2023-10-19T12:45:08.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.303+0000 7fab0ad9d640 1 -- 172.21.15.57:0/3245364957 learned_addr learned my addr 172.21.15.57:0/3245364957 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:08.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.304+0000 7fab0ad9d640 1 -- 172.21.15.57:0/3245364957 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fab0c0b0530 msgr2=0x7fab0c1376e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:08.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.304+0000 7fab0ad9d640 1 --2- 172.21.15.57:0/3245364957 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fab0c0b0530 0x7fab0c1376e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:08.313 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.304+0000 7fab0ad9d640 1 -- 172.21.15.57:0/3245364957 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fab0c0afa20 msgr2=0x7fab0c1399a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:08.313 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.304+0000 7fab0a59c640 1 --2- 172.21.15.57:0/3245364957 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fab0c0afa20 0x7fab0c1399a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:08.314 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.304+0000 7fab0ad9d640 1 --2- 172.21.15.57:0/3245364957 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fab0c0afa20 0x7fab0c1399a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:08.314 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.304+0000 7fab0ad9d640 1 -- 172.21.15.57:0/3245364957 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faafc009840 con 0x7fab0c0cbd20 2023-10-19T12:45:08.314 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.304+0000 7fab0ad9d640 1 --2- 172.21.15.57:0/3245364957 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab0c0cbd20 0x7fab0c138000 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7faafc009ca0 tx=0x7faafc0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:08.314 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.304+0000 7fab02ffd640 1 -- 172.21.15.57:0/3245364957 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faafc0046f0 con 0x7fab0c0cbd20 2023-10-19T12:45:08.314 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.304+0000 7fab02ffd640 1 -- 172.21.15.57:0/3245364957 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7faafc004850 con 0x7fab0c0cbd20 2023-10-19T12:45:08.315 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.304+0000 7fab10ac2640 1 -- 172.21.15.57:0/3245364957 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab0c1391a0 con 0x7fab0c0cbd20 2023-10-19T12:45:08.315 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.304+0000 7fab02ffd640 1 -- 172.21.15.57:0/3245364957 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faafc034800 con 0x7fab0c0cbd20 2023-10-19T12:45:08.315 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.304+0000 7fab10ac2640 1 -- 172.21.15.57:0/3245364957 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab0c137c20 con 0x7fab0c0cbd20 2023-10-19T12:45:08.315 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.305+0000 7fab10ac2640 1 -- 172.21.15.57:0/3245364957 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab0c1396c0 con 0x7fab0c0cbd20 2023-10-19T12:45:08.315 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.306+0000 7fab02ffd640 1 -- 172.21.15.57:0/3245364957 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7faafc034960 con 0x7fab0c0cbd20 2023-10-19T12:45:08.316 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.306+0000 7fab02ffd640 1 --2- 172.21.15.57:0/3245364957 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7faaec06ae40 0x7faaec06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:08.317 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.306+0000 7fab02ffd640 1 -- 172.21.15.57:0/3245364957 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7faafc0acc70 con 0x7fab0c0cbd20 2023-10-19T12:45:08.317 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.306+0000 7fab0a59c640 1 --2- 172.21.15.57:0/3245364957 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7faaec06ae40 0x7faaec06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:08.317 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.307+0000 7fab0a59c640 1 --2- 172.21.15.57:0/3245364957 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7faaec06ae40 0x7faaec06d300 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fab04000f60 tx=0x7fab04009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:08.317 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.310+0000 7fab02ffd640 1 -- 172.21.15.57:0/3245364957 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fab0c1396c0 con 0x7fab0c0cbd20 2023-10-19T12:45:08.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.484+0000 7fab10ac2640 1 -- 172.21.15.57:0/3245364957 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fab0c1396c0 con 0x7fab0c0cbd20 2023-10-19T12:45:08.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.484+0000 7fab02ffd640 1 -- 172.21.15.57:0/3245364957 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7fab0c1396c0 con 0x7fab0c0cbd20 2023-10-19T12:45:08.488 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2023-10-19T12:45:08.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.486+0000 7fab10ac2640 1 -- 172.21.15.57:0/3245364957 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7faaec06ae40 msgr2=0x7faaec06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:08.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.486+0000 7fab10ac2640 1 --2- 172.21.15.57:0/3245364957 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7faaec06ae40 0x7faaec06d300 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fab04000f60 tx=0x7fab04009040 comp rx=0 tx=0).stop 2023-10-19T12:45:08.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.486+0000 7fab10ac2640 1 -- 172.21.15.57:0/3245364957 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab0c0cbd20 msgr2=0x7fab0c138000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:08.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.486+0000 7fab10ac2640 1 --2- 172.21.15.57:0/3245364957 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab0c0cbd20 0x7fab0c138000 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7faafc009ca0 tx=0x7faafc0042f0 comp rx=0 tx=0).stop 2023-10-19T12:45:08.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.486+0000 7fab10ac2640 1 -- 172.21.15.57:0/3245364957 shutdown_connections 2023-10-19T12:45:08.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.486+0000 7fab10ac2640 1 --2- 172.21.15.57:0/3245364957 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7faaec06ae40 0x7faaec06d300 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:08.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.486+0000 7fab10ac2640 1 --2- 172.21.15.57:0/3245364957 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab0c0cbd20 0x7fab0c138000 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:08.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.486+0000 7fab10ac2640 1 --2- 172.21.15.57:0/3245364957 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fab0c0b0530 0x7fab0c1376e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:08.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.486+0000 7fab10ac2640 1 --2- 172.21.15.57:0/3245364957 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fab0c0afa20 0x7fab0c1399a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:08.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.486+0000 7fab10ac2640 1 -- 172.21.15.57:0/3245364957 >> 172.21.15.57:0/3245364957 conn(0x7fab0c069930 msgr2=0x7fab0c069d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:08.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.486+0000 7fab10ac2640 1 -- 172.21.15.57:0/3245364957 shutdown_connections 2023-10-19T12:45:08.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.486+0000 7fab10ac2640 1 -- 172.21.15.57:0/3245364957 wait complete. 2023-10-19T12:45:08.500 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates' 2023-10-19T12:45:08.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.606+0000 7fec6fdd0640 1 -- 172.21.15.57:0/3925181915 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec6815b2a0 msgr2=0x7fec680cf720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:08.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.606+0000 7fec6fdd0640 1 --2- 172.21.15.57:0/3925181915 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec6815b2a0 0x7fec680cf720 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fec58005fd0 tx=0x7fec5802bd90 comp rx=0 tx=0).stop 2023-10-19T12:45:08.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.606+0000 7fec6fdd0640 1 -- 172.21.15.57:0/3925181915 shutdown_connections 2023-10-19T12:45:08.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.606+0000 7fec6fdd0640 1 --2- 172.21.15.57:0/3925181915 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec6815b2a0 0x7fec680cf720 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:08.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.606+0000 7fec6fdd0640 1 --2- 172.21.15.57:0/3925181915 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fec680cf340 0x7fec680d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:08.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.606+0000 7fec6fdd0640 1 -- 172.21.15.57:0/3925181915 >> 172.21.15.57:0/3925181915 conn(0x7fec680d0d60 msgr2=0x7fec680d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:08.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.606+0000 7fec6fdd0640 1 -- 172.21.15.57:0/3925181915 shutdown_connections 2023-10-19T12:45:08.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.606+0000 7fec6fdd0640 1 -- 172.21.15.57:0/3925181915 wait complete. 2023-10-19T12:45:08.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.607+0000 7fec6fdd0640 1 Processor -- start 2023-10-19T12:45:08.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.607+0000 7fec6fdd0640 1 -- start start 2023-10-19T12:45:08.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.608+0000 7fec6fdd0640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fec680ceb40 0x7fec6815e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:08.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.608+0000 7fec6fdd0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec680cf340 0x7fec6815cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:08.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.608+0000 7fec6fdd0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fec6815b2a0 0x7fec68158040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:08.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.608+0000 7fec6fdd0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec6815c130 con 0x7fec680cf340 2023-10-19T12:45:08.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.608+0000 7fec6fdd0640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec6815d7d0 con 0x7fec6815b2a0 2023-10-19T12:45:08.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.608+0000 7fec6fdd0640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec6815b7d0 con 0x7fec680ceb40 2023-10-19T12:45:08.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.608+0000 7fec6d36b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec680cf340 0x7fec6815cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:08.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.608+0000 7fec6d36b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec680cf340 0x7fec6815cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59842/0 (socket says 172.21.15.57:59842) 2023-10-19T12:45:08.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.608+0000 7fec6d36b640 1 -- 172.21.15.57:0/624652660 learned_addr learned my addr 172.21.15.57:0/624652660 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:08.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.608+0000 7fec6db6c640 1 --2- 172.21.15.57:0/624652660 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fec680ceb40 0x7fec6815e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:08.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.608+0000 7fec6e36d640 1 --2- 172.21.15.57:0/624652660 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fec6815b2a0 0x7fec68158040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:08.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.608+0000 7fec6d36b640 1 -- 172.21.15.57:0/624652660 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fec680ceb40 msgr2=0x7fec6815e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:08.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.608+0000 7fec6d36b640 1 --2- 172.21.15.57:0/624652660 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fec680ceb40 0x7fec6815e260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:08.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.608+0000 7fec6d36b640 1 -- 172.21.15.57:0/624652660 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fec6815b2a0 msgr2=0x7fec68158040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:08.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.609+0000 7fec6d36b640 1 --2- 172.21.15.57:0/624652660 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fec6815b2a0 0x7fec68158040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:08.618 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.609+0000 7fec6d36b640 1 -- 172.21.15.57:0/624652660 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec58009560 con 0x7fec680cf340 2023-10-19T12:45:08.618 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.609+0000 7fec6db6c640 1 --2- 172.21.15.57:0/624652660 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fec680ceb40 0x7fec6815e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:45:08.618 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.609+0000 7fec6e36d640 1 --2- 172.21.15.57:0/624652660 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fec6815b2a0 0x7fec68158040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:45:08.618 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.609+0000 7fec6d36b640 1 --2- 172.21.15.57:0/624652660 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec680cf340 0x7fec6815cff0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fec58009850 tx=0x7fec5802da60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:08.618 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.609+0000 7fec5e7fc640 1 -- 172.21.15.57:0/624652660 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec5802de60 con 0x7fec680cf340 2023-10-19T12:45:08.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.609+0000 7fec5e7fc640 1 -- 172.21.15.57:0/624652660 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fec580036e0 con 0x7fec680cf340 2023-10-19T12:45:08.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.609+0000 7fec6fdd0640 1 -- 172.21.15.57:0/624652660 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec6815d530 con 0x7fec680cf340 2023-10-19T12:45:08.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.609+0000 7fec5e7fc640 1 -- 172.21.15.57:0/624652660 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec58003a90 con 0x7fec680cf340 2023-10-19T12:45:08.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.609+0000 7fec6fdd0640 1 -- 172.21.15.57:0/624652660 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec6815bd50 con 0x7fec680cf340 2023-10-19T12:45:08.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.610+0000 7fec6fdd0640 1 -- 172.21.15.57:0/624652660 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec6815c270 con 0x7fec680cf340 2023-10-19T12:45:08.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.610+0000 7fec5e7fc640 1 -- 172.21.15.57:0/624652660 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fec58003bf0 con 0x7fec680cf340 2023-10-19T12:45:08.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.611+0000 7fec5e7fc640 1 --2- 172.21.15.57:0/624652660 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fec4c06ae40 0x7fec4c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:08.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.611+0000 7fec5e7fc640 1 -- 172.21.15.57:0/624652660 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fec580ac5b0 con 0x7fec680cf340 2023-10-19T12:45:08.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.611+0000 7fec6db6c640 1 --2- 172.21.15.57:0/624652660 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fec4c06ae40 0x7fec4c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:08.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.612+0000 7fec6db6c640 1 --2- 172.21.15.57:0/624652660 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fec4c06ae40 0x7fec4c06d300 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fec640058c0 tx=0x7fec64005850 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:08.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.615+0000 7fec5e7fc640 1 -- 172.21.15.57:0/624652660 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fec6815c270 con 0x7fec680cf340 2023-10-19T12:45:08.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:08.858+0000 7fec6fdd0640 1 -- 172.21.15.57:0/624652660 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates"]} v 0) v1 -- 0x7fec6815c270 con 0x7fec680cf340 2023-10-19T12:45:09.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.036+0000 7fec5e7fc640 1 -- 172.21.15.57:0/624652660 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates"]}]=0 v69) v1 ==== 161+0+0 (secure 0 0 0) 0x7fec6815c270 con 0x7fec680cf340 2023-10-19T12:45:09.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.038+0000 7fec6fdd0640 1 -- 172.21.15.57:0/624652660 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fec4c06ae40 msgr2=0x7fec4c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:09.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.038+0000 7fec6fdd0640 1 --2- 172.21.15.57:0/624652660 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fec4c06ae40 0x7fec4c06d300 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fec640058c0 tx=0x7fec64005850 comp rx=0 tx=0).stop 2023-10-19T12:45:09.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.038+0000 7fec6fdd0640 1 -- 172.21.15.57:0/624652660 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec680cf340 msgr2=0x7fec6815cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:09.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.038+0000 7fec6fdd0640 1 --2- 172.21.15.57:0/624652660 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec680cf340 0x7fec6815cff0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fec58009850 tx=0x7fec5802da60 comp rx=0 tx=0).stop 2023-10-19T12:45:09.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.038+0000 7fec6fdd0640 1 -- 172.21.15.57:0/624652660 shutdown_connections 2023-10-19T12:45:09.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.038+0000 7fec6fdd0640 1 --2- 172.21.15.57:0/624652660 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fec4c06ae40 0x7fec4c06d300 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:09.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.038+0000 7fec6fdd0640 1 --2- 172.21.15.57:0/624652660 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fec6815b2a0 0x7fec68158040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:09.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.038+0000 7fec6fdd0640 1 --2- 172.21.15.57:0/624652660 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec680cf340 0x7fec6815cff0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:09.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.038+0000 7fec6fdd0640 1 --2- 172.21.15.57:0/624652660 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fec680ceb40 0x7fec6815e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:09.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.038+0000 7fec6fdd0640 1 -- 172.21.15.57:0/624652660 >> 172.21.15.57:0/624652660 conn(0x7fec680d0d60 msgr2=0x7fec680ae180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:09.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.038+0000 7fec6fdd0640 1 -- 172.21.15.57:0/624652660 shutdown_connections 2023-10-19T12:45:09.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.038+0000 7fec6fdd0640 1 -- 172.21.15.57:0/624652660 wait complete. 2023-10-19T12:45:09.052 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2023-10-19T12:45:09.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.158+0000 7fbc08c51640 1 -- 172.21.15.57:0/1239759542 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc04157c00 msgr2=0x7fbc040c98b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:09.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.158+0000 7fbc08c51640 1 --2- 172.21.15.57:0/1239759542 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc04157c00 0x7fbc040c98b0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fbc040cc8e0 tx=0x7fbbf402bc80 comp rx=0 tx=0).stop 2023-10-19T12:45:09.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.158+0000 7fbc08c51640 1 -- 172.21.15.57:0/1239759542 shutdown_connections 2023-10-19T12:45:09.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.158+0000 7fbc08c51640 1 --2- 172.21.15.57:0/1239759542 >> v2:172.21.15.179:3300/0 conn(0x7fbc04070c30 0x7fbc0406bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:09.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.158+0000 7fbc08c51640 1 --2- 172.21.15.57:0/1239759542 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc04157c00 0x7fbc040c98b0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:09.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.158+0000 7fbc08c51640 1 -- 172.21.15.57:0/1239759542 >> 172.21.15.57:0/1239759542 conn(0x7fbc040aec40 msgr2=0x7fbc040af050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:09.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.158+0000 7fbc08c51640 1 -- 172.21.15.57:0/1239759542 shutdown_connections 2023-10-19T12:45:09.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.158+0000 7fbc08c51640 1 -- 172.21.15.57:0/1239759542 wait complete. 2023-10-19T12:45:09.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.159+0000 7fbc08c51640 1 Processor -- start 2023-10-19T12:45:09.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.159+0000 7fbc08c51640 1 -- start start 2023-10-19T12:45:09.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.160+0000 7fbc08c51640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc04070c30 0x7fbc04139e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:09.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.160+0000 7fbc08c51640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbc040cb2f0 0x7fbc0414ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:09.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.160+0000 7fbc08c51640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbc04157c00 0x7fbc040ad3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:09.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.160+0000 7fbc08c51640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc0413b640 con 0x7fbc04070c30 2023-10-19T12:45:09.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.160+0000 7fbc08c51640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc0413b920 con 0x7fbc04157c00 2023-10-19T12:45:09.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.160+0000 7fbc08c51640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc0413bc00 con 0x7fbc040cb2f0 2023-10-19T12:45:09.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.160+0000 7fbc0259c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc04070c30 0x7fbc04139e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:09.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.160+0000 7fbc0259c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc04070c30 0x7fbc04139e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59854/0 (socket says 172.21.15.57:59854) 2023-10-19T12:45:09.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.160+0000 7fbc0259c640 1 -- 172.21.15.57:0/1285918194 learned_addr learned my addr 172.21.15.57:0/1285918194 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:09.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.160+0000 7fbc02d9d640 1 --2- 172.21.15.57:0/1285918194 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbc04157c00 0x7fbc040ad3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:09.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.160+0000 7fbbfbfff640 1 --2- 172.21.15.57:0/1285918194 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbc040cb2f0 0x7fbc0414ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:09.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.160+0000 7fbc0259c640 1 -- 172.21.15.57:0/1285918194 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbc040cb2f0 msgr2=0x7fbc0414ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:09.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.160+0000 7fbc0259c640 1 --2- 172.21.15.57:0/1285918194 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbc040cb2f0 0x7fbc0414ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:09.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.160+0000 7fbc0259c640 1 -- 172.21.15.57:0/1285918194 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbc04157c00 msgr2=0x7fbc040ad3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:09.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.160+0000 7fbc0259c640 1 --2- 172.21.15.57:0/1285918194 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbc04157c00 0x7fbc040ad3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:09.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.160+0000 7fbc0259c640 1 -- 172.21.15.57:0/1285918194 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbf4009560 con 0x7fbc04070c30 2023-10-19T12:45:09.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.160+0000 7fbc02d9d640 1 --2- 172.21.15.57:0/1285918194 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbc04157c00 0x7fbc040ad3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:45:09.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.160+0000 7fbc0259c640 1 --2- 172.21.15.57:0/1285918194 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc04070c30 0x7fbc04139e60 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fbbf4009850 tx=0x7fbbf402dc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:09.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.161+0000 7fbbfaffd640 1 -- 172.21.15.57:0/1285918194 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbf4005650 con 0x7fbc04070c30 2023-10-19T12:45:09.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.161+0000 7fbbfaffd640 1 -- 172.21.15.57:0/1285918194 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fbbf40038f0 con 0x7fbc04070c30 2023-10-19T12:45:09.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.161+0000 7fbc08c51640 1 -- 172.21.15.57:0/1285918194 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc040c9ee0 con 0x7fbc04070c30 2023-10-19T12:45:09.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.161+0000 7fbbfaffd640 1 -- 172.21.15.57:0/1285918194 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbf4003ce0 con 0x7fbc04070c30 2023-10-19T12:45:09.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.161+0000 7fbc08c51640 1 -- 172.21.15.57:0/1285918194 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc0413c1c0 con 0x7fbc04070c30 2023-10-19T12:45:09.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.162+0000 7fbc08c51640 1 -- 172.21.15.57:0/1285918194 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc040690e0 con 0x7fbc04070c30 2023-10-19T12:45:09.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.162+0000 7fbbfaffd640 1 -- 172.21.15.57:0/1285918194 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fbbf403e450 con 0x7fbc04070c30 2023-10-19T12:45:09.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.162+0000 7fbbfaffd640 1 --2- 172.21.15.57:0/1285918194 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fbbe406ae40 0x7fbbe406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:09.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.162+0000 7fbbfaffd640 1 -- 172.21.15.57:0/1285918194 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fbbf402de40 con 0x7fbc04070c30 2023-10-19T12:45:09.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.163+0000 7fbbfbfff640 1 --2- 172.21.15.57:0/1285918194 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fbbe406ae40 0x7fbbe406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:09.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.163+0000 7fbbfbfff640 1 --2- 172.21.15.57:0/1285918194 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fbbe406ae40 0x7fbbe406d300 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fbbec007fd0 tx=0x7fbbec009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:09.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.167+0000 7fbbfaffd640 1 -- 172.21.15.57:0/1285918194 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fbbf4077010 con 0x7fbc04070c30 2023-10-19T12:45:09.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.342+0000 7fbc08c51640 1 -- 172.21.15.57:0/1285918194 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "config dump"} v 0) v1 -- 0x7fbc0413c3b0 con 0x7fbc04070c30 2023-10-19T12:45:09.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.343+0000 7fbbfaffd640 1 -- 172.21.15.57:0/1285918194 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v30) v1 ==== 59+0+2220 (secure 0 0 0) 0x7fbbf40428c0 con 0x7fbc04070c30 2023-10-19T12:45:09.346 INFO:teuthology.orchestra.run.smithi057.stdout:WHO MASK LEVEL OPTION VALUE RO 2023-10-19T12:45:09.346 INFO:teuthology.orchestra.run.smithi057.stdout:mon advanced mon_op_complaint_time 120 2023-10-19T12:45:09.347 INFO:teuthology.orchestra.run.smithi057.stdout:osd advanced osd_op_complaint_time 180.000000 2023-10-19T12:45:09.347 INFO:teuthology.orchestra.run.smithi057.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 4775.995373 2023-10-19T12:45:09.347 INFO:teuthology.orchestra.run.smithi057.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 4208.118410 2023-10-19T12:45:09.347 INFO:teuthology.orchestra.run.smithi057.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 4541.123030 2023-10-19T12:45:09.347 INFO:teuthology.orchestra.run.smithi057.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 4877.592206 2023-10-19T12:45:09.348 INFO:teuthology.orchestra.run.smithi057.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 5019.748032 2023-10-19T12:45:09.348 INFO:teuthology.orchestra.run.smithi057.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 5000.074376 2023-10-19T12:45:09.348 INFO:teuthology.orchestra.run.smithi057.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 5153.479439 2023-10-19T12:45:09.348 INFO:teuthology.orchestra.run.smithi057.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 4977.363845 2023-10-19T12:45:09.348 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced debug_mds 20/20 2023-10-19T12:45:09.349 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced debug_mds_balancer 20/20 2023-10-19T12:45:09.349 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced debug_ms 1/1 2023-10-19T12:45:09.349 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced mds_bal_fragment_size_max 10000 2023-10-19T12:45:09.349 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced mds_bal_merge_size 5 2023-10-19T12:45:09.349 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced mds_bal_split_bits 3 2023-10-19T12:45:09.349 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced mds_bal_split_size 100 2023-10-19T12:45:09.350 INFO:teuthology.orchestra.run.smithi057.stdout:mds dev mds_debug_frag true 2023-10-19T12:45:09.350 INFO:teuthology.orchestra.run.smithi057.stdout:mds dev mds_debug_scatterstat true 2023-10-19T12:45:09.350 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced mds_op_complaint_time 180.000000 2023-10-19T12:45:09.350 INFO:teuthology.orchestra.run.smithi057.stdout:mds dev mds_verify_scatter true 2023-10-19T12:45:09.350 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced osd_op_complaint_time 180.000000 2023-10-19T12:45:09.351 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced rados_mon_op_timeout 900 2023-10-19T12:45:09.351 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced rados_osd_op_timeout 900 2023-10-19T12:45:09.351 INFO:teuthology.orchestra.run.smithi057.stdout:client advanced client_mount_timeout 600 2023-10-19T12:45:09.351 INFO:teuthology.orchestra.run.smithi057.stdout:client advanced debug_client 20/20 2023-10-19T12:45:09.351 INFO:teuthology.orchestra.run.smithi057.stdout:client advanced debug_ms 1/1 2023-10-19T12:45:09.351 INFO:teuthology.orchestra.run.smithi057.stdout:client advanced rados_mon_op_timeout 900 2023-10-19T12:45:09.352 INFO:teuthology.orchestra.run.smithi057.stdout:client advanced rados_osd_op_timeout 900 2023-10-19T12:45:09.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.344+0000 7fbc08c51640 1 -- 172.21.15.57:0/1285918194 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fbbe406ae40 msgr2=0x7fbbe406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:09.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.344+0000 7fbc08c51640 1 --2- 172.21.15.57:0/1285918194 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fbbe406ae40 0x7fbbe406d300 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fbbec007fd0 tx=0x7fbbec009040 comp rx=0 tx=0).stop 2023-10-19T12:45:09.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.344+0000 7fbc08c51640 1 -- 172.21.15.57:0/1285918194 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc04070c30 msgr2=0x7fbc04139e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:09.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.344+0000 7fbc08c51640 1 --2- 172.21.15.57:0/1285918194 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc04070c30 0x7fbc04139e60 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fbbf4009850 tx=0x7fbbf402dc50 comp rx=0 tx=0).stop 2023-10-19T12:45:09.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.345+0000 7fbc08c51640 1 -- 172.21.15.57:0/1285918194 shutdown_connections 2023-10-19T12:45:09.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.345+0000 7fbc08c51640 1 --2- 172.21.15.57:0/1285918194 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fbbe406ae40 0x7fbbe406d300 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:09.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.345+0000 7fbc08c51640 1 --2- 172.21.15.57:0/1285918194 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbc04157c00 0x7fbc040ad3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:09.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.345+0000 7fbc08c51640 1 --2- 172.21.15.57:0/1285918194 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbc040cb2f0 0x7fbc0414ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:09.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.345+0000 7fbc08c51640 1 --2- 172.21.15.57:0/1285918194 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc04070c30 0x7fbc04139e60 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:09.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.345+0000 7fbc08c51640 1 -- 172.21.15.57:0/1285918194 >> 172.21.15.57:0/1285918194 conn(0x7fbc040aec40 msgr2=0x7fbc0415a2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:09.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.345+0000 7fbc08c51640 1 -- 172.21.15.57:0/1285918194 shutdown_connections 2023-10-19T12:45:09.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.345+0000 7fbc08c51640 1 -- 172.21.15.57:0/1285918194 wait complete. 2023-10-19T12:45:09.358 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-19T12:45:09.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.468+0000 7ff0c298a640 1 -- 172.21.15.57:0/3882999033 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff0a4003970 msgr2=0x7ff0a4003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:09.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.468+0000 7ff0c298a640 1 --2- 172.21.15.57:0/3882999033 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff0a4003970 0x7ff0a4003d90 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7ff0ac006fd0 tx=0x7ff0a40057f0 comp rx=0 tx=0).stop 2023-10-19T12:45:09.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.468+0000 7ff0c298a640 1 -- 172.21.15.57:0/3882999033 shutdown_connections 2023-10-19T12:45:09.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.468+0000 7ff0c298a640 1 --2- 172.21.15.57:0/3882999033 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff0a4003970 0x7ff0a4003d90 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:09.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.468+0000 7ff0c298a640 1 -- 172.21.15.57:0/3882999033 >> 172.21.15.57:0/3882999033 conn(0x7ff0bc0d0d60 msgr2=0x7ff0bc0d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:09.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.468+0000 7ff0c298a640 1 -- 172.21.15.57:0/3882999033 shutdown_connections 2023-10-19T12:45:09.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.468+0000 7ff0c298a640 1 -- 172.21.15.57:0/3882999033 wait complete. 2023-10-19T12:45:09.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.469+0000 7ff0c298a640 1 Processor -- start 2023-10-19T12:45:09.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.469+0000 7ff0c298a640 1 -- start start 2023-10-19T12:45:09.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.470+0000 7ff0c298a640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff0a4003970 0x7ff0bc14ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:09.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.470+0000 7ff0c298a640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff0bc09aba0 0x7ff0bc09c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:09.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.470+0000 7ff0c298a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff0bc09b000 0x7ff0bc0b0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:09.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.470+0000 7ff0c298a640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0bc14b250 con 0x7ff0bc09b000 2023-10-19T12:45:09.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.470+0000 7ff0c298a640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0bc14b4a0 con 0x7ff0a4003970 2023-10-19T12:45:09.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.470+0000 7ff0c298a640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0bc14b780 con 0x7ff0bc09aba0 2023-10-19T12:45:09.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.470+0000 7ff0c0f27640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff0bc09b000 0x7ff0bc0b0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:09.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.470+0000 7ff0c0f27640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff0bc09b000 0x7ff0bc0b0ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59860/0 (socket says 172.21.15.57:59860) 2023-10-19T12:45:09.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.470+0000 7ff0c0f27640 1 -- 172.21.15.57:0/313857945 learned_addr learned my addr 172.21.15.57:0/313857945 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:09.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.470+0000 7ff0bbfff640 1 --2- 172.21.15.57:0/313857945 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff0a4003970 0x7ff0bc14ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:09.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.470+0000 7ff0bb7fe640 1 --2- 172.21.15.57:0/313857945 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff0bc09aba0 0x7ff0bc09c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:09.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.470+0000 7ff0c0f27640 1 -- 172.21.15.57:0/313857945 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff0bc09aba0 msgr2=0x7ff0bc09c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:09.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.470+0000 7ff0c0f27640 1 --2- 172.21.15.57:0/313857945 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff0bc09aba0 0x7ff0bc09c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:09.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.470+0000 7ff0c0f27640 1 -- 172.21.15.57:0/313857945 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff0a4003970 msgr2=0x7ff0bc14ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:09.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.470+0000 7ff0c0f27640 1 --2- 172.21.15.57:0/313857945 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff0a4003970 0x7ff0bc14ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:09.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.470+0000 7ff0c0f27640 1 -- 172.21.15.57:0/313857945 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0bc14ba60 con 0x7ff0bc09b000 2023-10-19T12:45:09.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.470+0000 7ff0bb7fe640 1 --2- 172.21.15.57:0/313857945 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff0bc09aba0 0x7ff0bc09c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:45:09.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.470+0000 7ff0c0f27640 1 --2- 172.21.15.57:0/313857945 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff0bc09b000 0x7ff0bc0b0ed0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7ff0bc0b6e10 tx=0x7ff0b40019b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:09.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.471+0000 7ff0b8ff9640 1 -- 172.21.15.57:0/313857945 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0b4011870 con 0x7ff0bc09b000 2023-10-19T12:45:09.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.471+0000 7ff0b8ff9640 1 -- 172.21.15.57:0/313857945 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7ff0b4011ac0 con 0x7ff0bc09b000 2023-10-19T12:45:09.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.471+0000 7ff0c298a640 1 -- 172.21.15.57:0/313857945 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0bc14bd40 con 0x7ff0bc09b000 2023-10-19T12:45:09.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.471+0000 7ff0b8ff9640 1 -- 172.21.15.57:0/313857945 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0b4002090 con 0x7ff0bc09b000 2023-10-19T12:45:09.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.471+0000 7ff0c298a640 1 -- 172.21.15.57:0/313857945 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0bc09cba0 con 0x7ff0bc09b000 2023-10-19T12:45:09.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.472+0000 7ff0c298a640 1 -- 172.21.15.57:0/313857945 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0bc14c260 con 0x7ff0bc09b000 2023-10-19T12:45:09.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.472+0000 7ff0b8ff9640 1 -- 172.21.15.57:0/313857945 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7ff0b401c460 con 0x7ff0bc09b000 2023-10-19T12:45:09.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.472+0000 7ff0b8ff9640 1 --2- 172.21.15.57:0/313857945 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7ff0a806ae40 0x7ff0a806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:09.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.473+0000 7ff0b8ff9640 1 -- 172.21.15.57:0/313857945 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7ff0b408a340 con 0x7ff0bc09b000 2023-10-19T12:45:09.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.473+0000 7ff0bbfff640 1 --2- 172.21.15.57:0/313857945 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7ff0a806ae40 0x7ff0a806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:09.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.473+0000 7ff0bbfff640 1 --2- 172.21.15.57:0/313857945 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7ff0a806ae40 0x7ff0a806d300 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ff0ac034000 tx=0x7ff0ac02e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:09.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.477+0000 7ff0b8ff9640 1 -- 172.21.15.57:0/313857945 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff0bc14c260 con 0x7ff0bc09b000 2023-10-19T12:45:09.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.652+0000 7ff0c298a640 1 -- 172.21.15.57:0/313857945 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7ff0bc14c260 con 0x7ff0bc09b000 2023-10-19T12:45:09.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.656+0000 7ff0b8ff9640 1 -- 172.21.15.57:0/313857945 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v31)=0 v31) v1 ==== 122+0+0 (secure 0 0 0) 0x7ff0bc14c260 con 0x7ff0bc09b000 2023-10-19T12:45:09.662 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.658+0000 7ff0c298a640 1 -- 172.21.15.57:0/313857945 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7ff0a806ae40 msgr2=0x7ff0a806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:09.662 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.658+0000 7ff0c298a640 1 --2- 172.21.15.57:0/313857945 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7ff0a806ae40 0x7ff0a806d300 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ff0ac034000 tx=0x7ff0ac02e000 comp rx=0 tx=0).stop 2023-10-19T12:45:09.662 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.658+0000 7ff0c298a640 1 -- 172.21.15.57:0/313857945 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff0bc09b000 msgr2=0x7ff0bc0b0ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:09.662 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.658+0000 7ff0c298a640 1 --2- 172.21.15.57:0/313857945 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff0bc09b000 0x7ff0bc0b0ed0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7ff0bc0b6e10 tx=0x7ff0b40019b0 comp rx=0 tx=0).stop 2023-10-19T12:45:09.663 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.658+0000 7ff0c298a640 1 -- 172.21.15.57:0/313857945 shutdown_connections 2023-10-19T12:45:09.663 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.658+0000 7ff0c298a640 1 --2- 172.21.15.57:0/313857945 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7ff0a806ae40 0x7ff0a806d300 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:09.663 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.658+0000 7ff0c298a640 1 --2- 172.21.15.57:0/313857945 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff0bc09b000 0x7ff0bc0b0ed0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:09.663 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.658+0000 7ff0c298a640 1 --2- 172.21.15.57:0/313857945 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff0bc09aba0 0x7ff0bc09c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:09.663 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.658+0000 7ff0c298a640 1 --2- 172.21.15.57:0/313857945 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff0a4003970 0x7ff0bc14ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:09.663 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.658+0000 7ff0c298a640 1 -- 172.21.15.57:0/313857945 >> 172.21.15.57:0/313857945 conn(0x7ff0bc0d0d60 msgr2=0x7ff0bc15d690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:09.664 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.659+0000 7ff0c298a640 1 -- 172.21.15.57:0/313857945 shutdown_connections 2023-10-19T12:45:09.664 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.659+0000 7ff0c298a640 1 -- 172.21.15.57:0/313857945 wait complete. 2023-10-19T12:45:09.672 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-19T12:45:09.673 DEBUG:teuthology.orchestra.run.smithi057:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-19T12:45:09.717 DEBUG:teuthology.orchestra.run.smithi179:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-19T12:45:09.767 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-19T12:45:09.768 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T12:45:09.768 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T12:45:09.784 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T12:45:09.785 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T12:45:09.840 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T12:45:09.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.985+0000 7f5262e75640 1 -- 172.21.15.57:0/1734576276 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f525c0cfa20 msgr2=0x7f525c0cb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:09.989 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.985+0000 7f5262e75640 1 --2- 172.21.15.57:0/1734576276 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f525c0cfa20 0x7f525c0cb9e0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f5244005fd0 tx=0x7f524402be20 comp rx=0 tx=0).stop 2023-10-19T12:45:09.989 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.985+0000 7f5262e75640 1 -- 172.21.15.57:0/1734576276 shutdown_connections 2023-10-19T12:45:09.989 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.985+0000 7f5262e75640 1 --2- 172.21.15.57:0/1734576276 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f525c0cfa20 0x7f525c0cb9e0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:09.989 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.985+0000 7f5262e75640 1 -- 172.21.15.57:0/1734576276 >> 172.21.15.57:0/1734576276 conn(0x7f525c09a030 msgr2=0x7f525c0d0d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:09.990 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.985+0000 7f5262e75640 1 -- 172.21.15.57:0/1734576276 shutdown_connections 2023-10-19T12:45:09.990 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.985+0000 7f5262e75640 1 -- 172.21.15.57:0/1734576276 wait complete. 2023-10-19T12:45:09.992 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.986+0000 7f5262e75640 1 Processor -- start 2023-10-19T12:45:09.992 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.986+0000 7f5262e75640 1 -- start start 2023-10-19T12:45:09.992 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.987+0000 7f5262e75640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f525c0cfa20 0x7f525c1e6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:09.993 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.987+0000 7f5262e75640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f525c15b260 0x7f525c1e5120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:09.993 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.987+0000 7f5262e75640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f525c15b640 0x7f525c150e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:09.993 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.987+0000 7f5262e75640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f525c1509e0 con 0x7f525c0cfa20 2023-10-19T12:45:09.993 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.987+0000 7f5262e75640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f525c0b5360 con 0x7f525c15b260 2023-10-19T12:45:09.993 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.987+0000 7f5262e75640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f525c1e6950 con 0x7f525c15b640 2023-10-19T12:45:09.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.987+0000 7f5260c11640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f525c0cfa20 0x7f525c1e6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:09.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.987+0000 7f5260c11640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f525c0cfa20 0x7f525c1e6410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59876/0 (socket says 172.21.15.57:59876) 2023-10-19T12:45:09.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.987+0000 7f5260c11640 1 -- 172.21.15.57:0/687981555 learned_addr learned my addr 172.21.15.57:0/687981555 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:09.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.988+0000 7f525bfff640 1 --2- 172.21.15.57:0/687981555 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f525c15b260 0x7f525c1e5120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:09.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.988+0000 7f5260c11640 1 -- 172.21.15.57:0/687981555 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f525c15b640 msgr2=0x7f525c150e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:09.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.988+0000 7f5260c11640 1 --2- 172.21.15.57:0/687981555 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f525c15b640 0x7f525c150e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:09.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.988+0000 7f5260c11640 1 -- 172.21.15.57:0/687981555 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f525c15b260 msgr2=0x7f525c1e5120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:09.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.988+0000 7f5260c11640 1 --2- 172.21.15.57:0/687981555 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f525c15b260 0x7f525c1e5120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:09.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.988+0000 7f5260c11640 1 -- 172.21.15.57:0/687981555 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5244009560 con 0x7f525c0cfa20 2023-10-19T12:45:09.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.988+0000 7f5260c11640 1 --2- 172.21.15.57:0/687981555 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f525c0cfa20 0x7f525c1e6410 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f525400c8a0 tx=0x7f525400cd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:09.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.988+0000 7f52597fa640 1 -- 172.21.15.57:0/687981555 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5254003b70 con 0x7f525c0cfa20 2023-10-19T12:45:09.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.988+0000 7f52597fa640 1 -- 172.21.15.57:0/687981555 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f5254003cf0 con 0x7f525c0cfa20 2023-10-19T12:45:09.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.988+0000 7f5262e75640 1 -- 172.21.15.57:0/687981555 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f525c0b4da0 con 0x7f525c0cfa20 2023-10-19T12:45:09.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.988+0000 7f52597fa640 1 -- 172.21.15.57:0/687981555 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5254003ea0 con 0x7f525c0cfa20 2023-10-19T12:45:09.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.988+0000 7f5262e75640 1 -- 172.21.15.57:0/687981555 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f525c150670 con 0x7f525c0cfa20 2023-10-19T12:45:09.997 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.989+0000 7f5262e75640 1 -- 172.21.15.57:0/687981555 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f525c0d0610 con 0x7f525c0cfa20 2023-10-19T12:45:09.998 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.990+0000 7f52597fa640 1 -- 172.21.15.57:0/687981555 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f525400aab0 con 0x7f525c0cfa20 2023-10-19T12:45:09.998 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.990+0000 7f52597fa640 1 --2- 172.21.15.57:0/687981555 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f523c06ae70 0x7f523c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:09.998 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.990+0000 7f52597fa640 1 -- 172.21.15.57:0/687981555 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f5254089a90 con 0x7f525c0cfa20 2023-10-19T12:45:09.998 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.990+0000 7f525bfff640 1 --2- 172.21.15.57:0/687981555 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f523c06ae70 0x7f523c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:09.998 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.991+0000 7f525bfff640 1 --2- 172.21.15.57:0/687981555 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f523c06ae70 0x7f523c06d330 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f5244035000 tx=0x7f5244004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:09.999 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:09.994+0000 7f52597fa640 1 -- 172.21.15.57:0/687981555 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f5254054790 con 0x7f525c0cfa20 2023-10-19T12:45:10.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.173+0000 7f5262e75640 1 -- 172.21.15.57:0/687981555 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f525c0675f0 con 0x7f525c0cfa20 2023-10-19T12:45:10.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.174+0000 7f52597fa640 1 -- 172.21.15.57:0/687981555 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18748 (secure 0 0 0) 0x7f5254057db0 con 0x7f525c0cfa20 2023-10-19T12:45:10.178 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:45:10.178 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":27,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T12:44:58.978403+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-19T12:44:16.517510+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-19T12:44:17.398297+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-19T12:44:21.129403+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.23","mappings":[{"from":3,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"3.39","mappings":[{"from":7,"to":1}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.1d","mappings":[{"from":1,"to":5}]},{"pgid":"4.1e","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T12:45:10.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.176+0000 7f5262e75640 1 -- 172.21.15.57:0/687981555 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f523c06ae70 msgr2=0x7f523c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:10.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.176+0000 7f5262e75640 1 --2- 172.21.15.57:0/687981555 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f523c06ae70 0x7f523c06d330 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f5244035000 tx=0x7f5244004730 comp rx=0 tx=0).stop 2023-10-19T12:45:10.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.176+0000 7f5262e75640 1 -- 172.21.15.57:0/687981555 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f525c0cfa20 msgr2=0x7f525c1e6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:10.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.176+0000 7f5262e75640 1 --2- 172.21.15.57:0/687981555 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f525c0cfa20 0x7f525c1e6410 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f525400c8a0 tx=0x7f525400cd60 comp rx=0 tx=0).stop 2023-10-19T12:45:10.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.176+0000 7f5262e75640 1 -- 172.21.15.57:0/687981555 shutdown_connections 2023-10-19T12:45:10.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.176+0000 7f5262e75640 1 --2- 172.21.15.57:0/687981555 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f523c06ae70 0x7f523c06d330 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:10.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.176+0000 7f5262e75640 1 --2- 172.21.15.57:0/687981555 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f525c15b640 0x7f525c150e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:10.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.176+0000 7f5262e75640 1 --2- 172.21.15.57:0/687981555 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f525c15b260 0x7f525c1e5120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:10.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.176+0000 7f5262e75640 1 --2- 172.21.15.57:0/687981555 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f525c0cfa20 0x7f525c1e6410 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:10.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.176+0000 7f5262e75640 1 -- 172.21.15.57:0/687981555 >> 172.21.15.57:0/687981555 conn(0x7f525c09a030 msgr2=0x7f525c0ccd40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:10.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.177+0000 7f5262e75640 1 -- 172.21.15.57:0/687981555 shutdown_connections 2023-10-19T12:45:10.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.177+0000 7f5262e75640 1 -- 172.21.15.57:0/687981555 wait complete. 2023-10-19T12:45:10.190 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-19T12:44:07.371186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-19T12:44:16.517510+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-19T12:44:17.398297+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-19T12:44:21.129403+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-19T12:45:10.191 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-19T12:45:10.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.300+0000 7fb496b8e640 1 -- 172.21.15.57:0/2440764247 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4900b1ca0 msgr2=0x7fb4900ce9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:10.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.300+0000 7fb496b8e640 1 --2- 172.21.15.57:0/2440764247 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4900b1ca0 0x7fb4900ce9a0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fb480005fd0 tx=0x7fb48002bd90 comp rx=0 tx=0).stop 2023-10-19T12:45:10.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.300+0000 7fb496b8e640 1 -- 172.21.15.57:0/2440764247 shutdown_connections 2023-10-19T12:45:10.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.300+0000 7fb496b8e640 1 --2- 172.21.15.57:0/2440764247 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4900b1ca0 0x7fb4900ce9a0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:10.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.300+0000 7fb496b8e640 1 -- 172.21.15.57:0/2440764247 >> 172.21.15.57:0/2440764247 conn(0x7fb4900d0c50 msgr2=0x7fb4900d1060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:10.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.300+0000 7fb496b8e640 1 -- 172.21.15.57:0/2440764247 shutdown_connections 2023-10-19T12:45:10.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.300+0000 7fb496b8e640 1 -- 172.21.15.57:0/2440764247 wait complete. 2023-10-19T12:45:10.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.301+0000 7fb496b8e640 1 Processor -- start 2023-10-19T12:45:10.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.301+0000 7fb496b8e640 1 -- start start 2023-10-19T12:45:10.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.302+0000 7fb496b8e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4900b1ca0 0x7fb4900b1080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:10.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.302+0000 7fb496b8e640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb4900cf8b0 0x7fb49006b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:10.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.302+0000 7fb496b8e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb49015e5e0 0x7fb49006b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:10.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.302+0000 7fb496b8e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4900c5560 con 0x7fb4900b1ca0 2023-10-19T12:45:10.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.302+0000 7fb496b8e640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4900c46d0 con 0x7fb49015e5e0 2023-10-19T12:45:10.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.302+0000 7fb496b8e640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4900c48e0 con 0x7fb4900cf8b0 2023-10-19T12:45:10.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.302+0000 7fb49492a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4900b1ca0 0x7fb4900b1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:10.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.302+0000 7fb49492a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4900b1ca0 0x7fb4900b1080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59896/0 (socket says 172.21.15.57:59896) 2023-10-19T12:45:10.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.302+0000 7fb49492a640 1 -- 172.21.15.57:0/1456161754 learned_addr learned my addr 172.21.15.57:0/1456161754 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:10.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.303+0000 7fb49512b640 1 --2- 172.21.15.57:0/1456161754 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb49015e5e0 0x7fb49006b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:10.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.303+0000 7fb48ffff640 1 --2- 172.21.15.57:0/1456161754 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb4900cf8b0 0x7fb49006b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:10.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.303+0000 7fb49492a640 1 -- 172.21.15.57:0/1456161754 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb4900cf8b0 msgr2=0x7fb49006b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:10.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.303+0000 7fb49492a640 1 --2- 172.21.15.57:0/1456161754 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb4900cf8b0 0x7fb49006b090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:10.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.303+0000 7fb49492a640 1 -- 172.21.15.57:0/1456161754 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb49015e5e0 msgr2=0x7fb49006b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:10.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.303+0000 7fb49492a640 1 --2- 172.21.15.57:0/1456161754 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb49015e5e0 0x7fb49006b5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:10.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.303+0000 7fb49492a640 1 -- 172.21.15.57:0/1456161754 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb480009560 con 0x7fb4900b1ca0 2023-10-19T12:45:10.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.303+0000 7fb49512b640 1 --2- 172.21.15.57:0/1456161754 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb49015e5e0 0x7fb49006b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:45:10.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.303+0000 7fb49492a640 1 --2- 172.21.15.57:0/1456161754 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4900b1ca0 0x7fb4900b1080 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fb48800ba50 tx=0x7fb48800bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:10.312 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.303+0000 7fb48d7fa640 1 -- 172.21.15.57:0/1456161754 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb48800cbe0 con 0x7fb4900b1ca0 2023-10-19T12:45:10.312 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.303+0000 7fb48d7fa640 1 -- 172.21.15.57:0/1456161754 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fb48800ce70 con 0x7fb4900b1ca0 2023-10-19T12:45:10.312 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.303+0000 7fb496b8e640 1 -- 172.21.15.57:0/1456161754 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4900c4bc0 con 0x7fb4900b1ca0 2023-10-19T12:45:10.312 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.303+0000 7fb48d7fa640 1 -- 172.21.15.57:0/1456161754 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb488014680 con 0x7fb4900b1ca0 2023-10-19T12:45:10.312 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.303+0000 7fb496b8e640 1 -- 172.21.15.57:0/1456161754 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4900c5180 con 0x7fb4900b1ca0 2023-10-19T12:45:10.312 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.304+0000 7fb496b8e640 1 -- 172.21.15.57:0/1456161754 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4900c5880 con 0x7fb4900b1ca0 2023-10-19T12:45:10.313 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.304+0000 7fb48d7fa640 1 -- 172.21.15.57:0/1456161754 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fb4880147e0 con 0x7fb4900b1ca0 2023-10-19T12:45:10.314 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.305+0000 7fb48d7fa640 1 --2- 172.21.15.57:0/1456161754 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fb47006ae40 0x7fb47006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:10.314 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.305+0000 7fb48d7fa640 1 -- 172.21.15.57:0/1456161754 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fb488089ba0 con 0x7fb4900b1ca0 2023-10-19T12:45:10.314 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.305+0000 7fb48ffff640 1 --2- 172.21.15.57:0/1456161754 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fb47006ae40 0x7fb47006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:10.314 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.306+0000 7fb48ffff640 1 --2- 172.21.15.57:0/1456161754 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fb47006ae40 0x7fb47006d300 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb49015b1d0 tx=0x7fb478009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:10.314 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.309+0000 7fb48d7fa640 1 -- 172.21.15.57:0/1456161754 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb4880548a0 con 0x7fb4900b1ca0 2023-10-19T12:45:10.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.483+0000 7fb496b8e640 1 -- 172.21.15.57:0/1456161754 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb4900690e0 con 0x7fb4900b1ca0 2023-10-19T12:45:10.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.484+0000 7fb48d7fa640 1 -- 172.21.15.57:0/1456161754 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb488019020 con 0x7fb4900b1ca0 2023-10-19T12:45:10.487 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-19T12:45:10.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.486+0000 7fb496b8e640 1 -- 172.21.15.57:0/1456161754 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fb47006ae40 msgr2=0x7fb47006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:10.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.486+0000 7fb496b8e640 1 --2- 172.21.15.57:0/1456161754 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fb47006ae40 0x7fb47006d300 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb49015b1d0 tx=0x7fb478009040 comp rx=0 tx=0).stop 2023-10-19T12:45:10.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.486+0000 7fb496b8e640 1 -- 172.21.15.57:0/1456161754 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4900b1ca0 msgr2=0x7fb4900b1080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:10.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.486+0000 7fb496b8e640 1 --2- 172.21.15.57:0/1456161754 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4900b1ca0 0x7fb4900b1080 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fb48800ba50 tx=0x7fb48800bf10 comp rx=0 tx=0).stop 2023-10-19T12:45:10.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.486+0000 7fb496b8e640 1 -- 172.21.15.57:0/1456161754 shutdown_connections 2023-10-19T12:45:10.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.486+0000 7fb496b8e640 1 --2- 172.21.15.57:0/1456161754 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fb47006ae40 0x7fb47006d300 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:10.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.486+0000 7fb496b8e640 1 --2- 172.21.15.57:0/1456161754 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb49015e5e0 0x7fb49006b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:10.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.486+0000 7fb496b8e640 1 --2- 172.21.15.57:0/1456161754 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb4900cf8b0 0x7fb49006b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:10.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.486+0000 7fb496b8e640 1 --2- 172.21.15.57:0/1456161754 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4900b1ca0 0x7fb4900b1080 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:10.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.486+0000 7fb496b8e640 1 -- 172.21.15.57:0/1456161754 >> 172.21.15.57:0/1456161754 conn(0x7fb4900d0c50 msgr2=0x7fb4900d1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:10.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.486+0000 7fb496b8e640 1 -- 172.21.15.57:0/1456161754 shutdown_connections 2023-10-19T12:45:10.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.486+0000 7fb496b8e640 1 -- 172.21.15.57:0/1456161754 wait complete. 2023-10-19T12:45:10.500 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-19T12:45:10.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.610+0000 7f8b76ee5640 1 -- 172.21.15.57:0/269821263 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8b58003ef0 msgr2=0x7f8b580047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:10.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.610+0000 7f8b76ee5640 1 --2- 172.21.15.57:0/269821263 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8b58003ef0 0x7f8b580047c0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f8b64006fd0 tx=0x7f8b580057f0 comp rx=0 tx=0).stop 2023-10-19T12:45:10.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.610+0000 7f8b75482640 1 -- 172.21.15.57:0/269821263 reap_dead start 2023-10-19T12:45:10.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.610+0000 7f8b76ee5640 1 -- 172.21.15.57:0/269821263 shutdown_connections 2023-10-19T12:45:10.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.610+0000 7f8b76ee5640 1 -- 172.21.15.57:0/269821263 >> 172.21.15.57:0/269821263 conn(0x7f8b700af7a0 msgr2=0x7f8b700afb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:10.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.610+0000 7f8b76ee5640 1 -- 172.21.15.57:0/269821263 shutdown_connections 2023-10-19T12:45:10.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.610+0000 7f8b76ee5640 1 -- 172.21.15.57:0/269821263 wait complete. 2023-10-19T12:45:10.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.611+0000 7f8b76ee5640 1 Processor -- start 2023-10-19T12:45:10.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.611+0000 7f8b76ee5640 1 -- start start 2023-10-19T12:45:10.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.612+0000 7f8b76ee5640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8b58003ef0 0x7f8b70151ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:10.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.612+0000 7f8b76ee5640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8b7014e300 0x7f8b70140370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:10.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.612+0000 7f8b76ee5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8b70153850 0x7f8b70153cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:10.618 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.612+0000 7f8b76ee5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b7014f0d0 con 0x7f8b70153850 2023-10-19T12:45:10.618 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.612+0000 7f8b76ee5640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b7014fa60 con 0x7f8b58003ef0 2023-10-19T12:45:10.618 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.612+0000 7f8b76ee5640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b70154210 con 0x7f8b7014e300 2023-10-19T12:45:10.618 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.612+0000 7f8b75482640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8b70153850 0x7f8b70153cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:10.618 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.612+0000 7f8b75482640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8b70153850 0x7f8b70153cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59912/0 (socket says 172.21.15.57:59912) 2023-10-19T12:45:10.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.612+0000 7f8b75482640 1 -- 172.21.15.57:0/3309952224 learned_addr learned my addr 172.21.15.57:0/3309952224 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:10.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.612+0000 7f8b74c81640 1 --2- 172.21.15.57:0/3309952224 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8b58003ef0 0x7f8b70151ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:10.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.612+0000 7f8b6ffff640 1 --2- 172.21.15.57:0/3309952224 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8b7014e300 0x7f8b70140370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:10.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.612+0000 7f8b75482640 1 -- 172.21.15.57:0/3309952224 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8b7014e300 msgr2=0x7f8b70140370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:10.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.612+0000 7f8b75482640 1 --2- 172.21.15.57:0/3309952224 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8b7014e300 0x7f8b70140370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:10.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.612+0000 7f8b75482640 1 -- 172.21.15.57:0/3309952224 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8b58003ef0 msgr2=0x7f8b70151ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:10.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.612+0000 7f8b75482640 1 --2- 172.21.15.57:0/3309952224 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8b58003ef0 0x7f8b70151ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:10.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.612+0000 7f8b75482640 1 -- 172.21.15.57:0/3309952224 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b7014e830 con 0x7f8b70153850 2023-10-19T12:45:10.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.612+0000 7f8b6ffff640 1 --2- 172.21.15.57:0/3309952224 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8b7014e300 0x7f8b70140370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:45:10.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.612+0000 7f8b75482640 1 --2- 172.21.15.57:0/3309952224 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8b70153850 0x7f8b70153cd0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f8b70068f80 tx=0x7f8b60011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:10.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.613+0000 7f8b6d7fa640 1 -- 172.21.15.57:0/3309952224 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b60004440 con 0x7f8b70153850 2023-10-19T12:45:10.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.613+0000 7f8b6d7fa640 1 -- 172.21.15.57:0/3309952224 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f8b6000f8f0 con 0x7f8b70153850 2023-10-19T12:45:10.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.613+0000 7f8b6d7fa640 1 -- 172.21.15.57:0/3309952224 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b6000fd00 con 0x7f8b70153850 2023-10-19T12:45:10.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.613+0000 7f8b76ee5640 1 -- 172.21.15.57:0/3309952224 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b7014eb10 con 0x7f8b70153850 2023-10-19T12:45:10.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.613+0000 7f8b76ee5640 1 -- 172.21.15.57:0/3309952224 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b70152420 con 0x7f8b70153850 2023-10-19T12:45:10.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.614+0000 7f8b76ee5640 1 -- 172.21.15.57:0/3309952224 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b7014f830 con 0x7f8b70153850 2023-10-19T12:45:10.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.614+0000 7f8b6d7fa640 1 -- 172.21.15.57:0/3309952224 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f8b600035f0 con 0x7f8b70153850 2023-10-19T12:45:10.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.615+0000 7f8b6d7fa640 1 --2- 172.21.15.57:0/3309952224 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f8b5c06ae40 0x7f8b5c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:10.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.615+0000 7f8b6d7fa640 1 -- 172.21.15.57:0/3309952224 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f8b6008a600 con 0x7f8b70153850 2023-10-19T12:45:10.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.615+0000 7f8b74c81640 1 --2- 172.21.15.57:0/3309952224 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f8b5c06ae40 0x7f8b5c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:10.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.615+0000 7f8b74c81640 1 --2- 172.21.15.57:0/3309952224 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f8b5c06ae40 0x7f8b5c06d300 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f8b68005fd0 tx=0x7f8b68009500 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:10.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.619+0000 7f8b6d7fa640 1 -- 172.21.15.57:0/3309952224 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8b7014f830 con 0x7f8b70153850 2023-10-19T12:45:10.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.796+0000 7f8b76ee5640 1 -- 172.21.15.57:0/3309952224 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8b7014f830 con 0x7f8b70153850 2023-10-19T12:45:10.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.797+0000 7f8b6d7fa640 1 -- 172.21.15.57:0/3309952224 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8b7014f830 con 0x7f8b70153850 2023-10-19T12:45:10.801 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2023-10-19T12:45:10.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.799+0000 7f8b76ee5640 1 -- 172.21.15.57:0/3309952224 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f8b5c06ae40 msgr2=0x7f8b5c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:10.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.799+0000 7f8b76ee5640 1 --2- 172.21.15.57:0/3309952224 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f8b5c06ae40 0x7f8b5c06d300 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f8b68005fd0 tx=0x7f8b68009500 comp rx=0 tx=0).stop 2023-10-19T12:45:10.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.799+0000 7f8b76ee5640 1 -- 172.21.15.57:0/3309952224 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8b70153850 msgr2=0x7f8b70153cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:10.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.799+0000 7f8b76ee5640 1 --2- 172.21.15.57:0/3309952224 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8b70153850 0x7f8b70153cd0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f8b70068f80 tx=0x7f8b60011b50 comp rx=0 tx=0).stop 2023-10-19T12:45:10.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.799+0000 7f8b76ee5640 1 -- 172.21.15.57:0/3309952224 shutdown_connections 2023-10-19T12:45:10.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.799+0000 7f8b76ee5640 1 --2- 172.21.15.57:0/3309952224 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f8b5c06ae40 0x7f8b5c06d300 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:10.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.799+0000 7f8b76ee5640 1 --2- 172.21.15.57:0/3309952224 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8b70153850 0x7f8b70153cd0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:10.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.799+0000 7f8b76ee5640 1 --2- 172.21.15.57:0/3309952224 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8b7014e300 0x7f8b70140370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:10.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.799+0000 7f8b76ee5640 1 --2- 172.21.15.57:0/3309952224 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8b58003ef0 0x7f8b70151ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:10.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.799+0000 7f8b76ee5640 1 -- 172.21.15.57:0/3309952224 >> 172.21.15.57:0/3309952224 conn(0x7f8b700af7a0 msgr2=0x7f8b7014b120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:10.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.799+0000 7f8b76ee5640 1 -- 172.21.15.57:0/3309952224 shutdown_connections 2023-10-19T12:45:10.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.799+0000 7f8b76ee5640 1 -- 172.21.15.57:0/3309952224 wait complete. 2023-10-19T12:45:10.813 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-19T12:45:10.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.922+0000 7f7cd5c87640 1 -- 172.21.15.57:0/4147271297 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7cd0070af0 msgr2=0x7f7cd01628b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:10.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.922+0000 7f7cd5c87640 1 --2- 172.21.15.57:0/4147271297 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7cd0070af0 0x7f7cd01628b0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f7cc8009830 tx=0x7f7cc802bc50 comp rx=0 tx=0).stop 2023-10-19T12:45:10.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.922+0000 7f7cd5c87640 1 -- 172.21.15.57:0/4147271297 shutdown_connections 2023-10-19T12:45:10.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.922+0000 7f7cd5c87640 1 --2- 172.21.15.57:0/4147271297 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7cd0070af0 0x7f7cd01628b0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:10.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.922+0000 7f7cd5c87640 1 -- 172.21.15.57:0/4147271297 >> 172.21.15.57:0/4147271297 conn(0x7f7cd0069930 msgr2=0x7f7cd0069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:10.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.922+0000 7f7cd5c87640 1 -- 172.21.15.57:0/4147271297 shutdown_connections 2023-10-19T12:45:10.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.922+0000 7f7cd5c87640 1 -- 172.21.15.57:0/4147271297 wait complete. 2023-10-19T12:45:10.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.923+0000 7f7cd5c87640 1 Processor -- start 2023-10-19T12:45:10.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.923+0000 7f7cd5c87640 1 -- start start 2023-10-19T12:45:10.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.924+0000 7f7cd5c87640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7cd0070af0 0x7f7cd013a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:10.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.924+0000 7f7cd5c87640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7cd013e520 0x7f7cd01539c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:10.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.924+0000 7f7cd5c87640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7cd0141690 0x7f7cd0153f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:10.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.924+0000 7f7cd5c87640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cd00aaa60 con 0x7f7cd0141690 2023-10-19T12:45:10.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.924+0000 7f7cd5c87640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cd013ac30 con 0x7f7cd013e520 2023-10-19T12:45:10.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.924+0000 7f7cd5c87640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cd01409b0 con 0x7f7cd0070af0 2023-10-19T12:45:10.932 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.924+0000 7f7ccffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7cd0141690 0x7f7cd0153f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:10.932 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.924+0000 7f7ccffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7cd0141690 0x7f7cd0153f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59922/0 (socket says 172.21.15.57:59922) 2023-10-19T12:45:10.932 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.924+0000 7f7ccffff640 1 -- 172.21.15.57:0/221560739 learned_addr learned my addr 172.21.15.57:0/221560739 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:10.932 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.924+0000 7f7cceffd640 1 --2- 172.21.15.57:0/221560739 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7cd013e520 0x7f7cd01539c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:10.932 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.924+0000 7f7ccffff640 1 -- 172.21.15.57:0/221560739 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7cd0070af0 msgr2=0x7f7cd013a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:10.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.924+0000 7f7ccf7fe640 1 --2- 172.21.15.57:0/221560739 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7cd0070af0 0x7f7cd013a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:10.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.924+0000 7f7ccffff640 1 --2- 172.21.15.57:0/221560739 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7cd0070af0 0x7f7cd013a6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:10.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.924+0000 7f7ccffff640 1 -- 172.21.15.57:0/221560739 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7cd013e520 msgr2=0x7f7cd01539c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:10.934 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.924+0000 7f7ccffff640 1 --2- 172.21.15.57:0/221560739 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7cd013e520 0x7f7cd01539c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:10.934 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.924+0000 7f7ccffff640 1 -- 172.21.15.57:0/221560739 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cc8009490 con 0x7f7cd0141690 2023-10-19T12:45:10.934 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.924+0000 7f7cceffd640 1 --2- 172.21.15.57:0/221560739 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7cd013e520 0x7f7cd01539c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:45:10.934 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.924+0000 7f7ccffff640 1 --2- 172.21.15.57:0/221560739 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7cd0141690 0x7f7cd0153f00 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f7cc8009950 tx=0x7f7cc802de00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:10.935 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.924+0000 7f7cb7fff640 1 -- 172.21.15.57:0/221560739 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cc8005220 con 0x7f7cd0141690 2023-10-19T12:45:10.935 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.924+0000 7f7cd5c87640 1 -- 172.21.15.57:0/221560739 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cd013fb60 con 0x7f7cd0141690 2023-10-19T12:45:10.935 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.924+0000 7f7cb7fff640 1 -- 172.21.15.57:0/221560739 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f7cc8005470 con 0x7f7cd0141690 2023-10-19T12:45:10.935 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.924+0000 7f7cb7fff640 1 -- 172.21.15.57:0/221560739 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cc8003a60 con 0x7f7cd0141690 2023-10-19T12:45:10.935 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.925+0000 7f7cd5c87640 1 -- 172.21.15.57:0/221560739 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cd00ae3c0 con 0x7f7cd0141690 2023-10-19T12:45:10.935 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.926+0000 7f7cd5c87640 1 -- 172.21.15.57:0/221560739 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cd00aaba0 con 0x7f7cd0141690 2023-10-19T12:45:10.936 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.926+0000 7f7cb7fff640 1 -- 172.21.15.57:0/221560739 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f7cc8003bc0 con 0x7f7cd0141690 2023-10-19T12:45:10.936 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.926+0000 7f7cb7fff640 1 --2- 172.21.15.57:0/221560739 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f7cbc06ae40 0x7f7cbc06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:10.936 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.926+0000 7f7cb7fff640 1 -- 172.21.15.57:0/221560739 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f7cc80abaf0 con 0x7f7cd0141690 2023-10-19T12:45:10.936 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.927+0000 7f7ccf7fe640 1 --2- 172.21.15.57:0/221560739 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f7cbc06ae40 0x7f7cbc06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:10.936 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.927+0000 7f7ccf7fe640 1 --2- 172.21.15.57:0/221560739 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f7cbc06ae40 0x7f7cbc06d300 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f7cc0007fd0 tx=0x7f7cc0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:10.937 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:10.931+0000 7f7cb7fff640 1 -- 172.21.15.57:0/221560739 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7cd00aaba0 con 0x7f7cd0141690 2023-10-19T12:45:11.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.107+0000 7f7cd5c87640 1 -- 172.21.15.57:0/221560739 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7cd00aaba0 con 0x7f7cd0141690 2023-10-19T12:45:11.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.108+0000 7f7cb7fff640 1 -- 172.21.15.57:0/221560739 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7cd00aaba0 con 0x7f7cd0141690 2023-10-19T12:45:11.111 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 64 2023-10-19T12:45:11.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.110+0000 7f7cd5c87640 1 -- 172.21.15.57:0/221560739 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f7cbc06ae40 msgr2=0x7f7cbc06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:11.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.110+0000 7f7cd5c87640 1 --2- 172.21.15.57:0/221560739 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f7cbc06ae40 0x7f7cbc06d300 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f7cc0007fd0 tx=0x7f7cc0009040 comp rx=0 tx=0).stop 2023-10-19T12:45:11.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.110+0000 7f7cd5c87640 1 -- 172.21.15.57:0/221560739 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7cd0141690 msgr2=0x7f7cd0153f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:11.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.110+0000 7f7cd5c87640 1 --2- 172.21.15.57:0/221560739 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7cd0141690 0x7f7cd0153f00 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f7cc8009950 tx=0x7f7cc802de00 comp rx=0 tx=0).stop 2023-10-19T12:45:11.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.110+0000 7f7cd5c87640 1 -- 172.21.15.57:0/221560739 shutdown_connections 2023-10-19T12:45:11.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.110+0000 7f7cd5c87640 1 --2- 172.21.15.57:0/221560739 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f7cbc06ae40 0x7f7cbc06d300 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:11.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.110+0000 7f7cd5c87640 1 --2- 172.21.15.57:0/221560739 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7cd0141690 0x7f7cd0153f00 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:11.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.110+0000 7f7cd5c87640 1 --2- 172.21.15.57:0/221560739 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7cd013e520 0x7f7cd01539c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:11.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.110+0000 7f7cd5c87640 1 --2- 172.21.15.57:0/221560739 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7cd0070af0 0x7f7cd013a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:11.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.110+0000 7f7cd5c87640 1 -- 172.21.15.57:0/221560739 >> 172.21.15.57:0/221560739 conn(0x7f7cd0069930 msgr2=0x7f7cd0069d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:11.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.110+0000 7f7cd5c87640 1 -- 172.21.15.57:0/221560739 shutdown_connections 2023-10-19T12:45:11.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.110+0000 7f7cd5c87640 1 -- 172.21.15.57:0/221560739 wait complete. 2023-10-19T12:45:11.124 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-19T12:45:11.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.234+0000 7f1239133640 1 -- 172.21.15.57:0/916252435 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f12340ad370 msgr2=0x7f12340ad800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:11.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.234+0000 7f1239133640 1 --2- 172.21.15.57:0/916252435 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f12340ad370 0x7f12340ad800 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f122c005fd0 tx=0x7f122c02c0e0 comp rx=0 tx=0).stop 2023-10-19T12:45:11.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.234+0000 7f1239133640 1 -- 172.21.15.57:0/916252435 shutdown_connections 2023-10-19T12:45:11.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.234+0000 7f1239133640 1 --2- 172.21.15.57:0/916252435 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f12340ad370 0x7f12340ad800 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:11.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.234+0000 7f1239133640 1 --2- 172.21.15.57:0/916252435 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f123406d2d0 0x7f123406d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:11.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.234+0000 7f1239133640 1 -- 172.21.15.57:0/916252435 >> 172.21.15.57:0/916252435 conn(0x7f12340b0670 msgr2=0x7f12340b0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:11.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.234+0000 7f1239133640 1 -- 172.21.15.57:0/916252435 shutdown_connections 2023-10-19T12:45:11.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.234+0000 7f1239133640 1 -- 172.21.15.57:0/916252435 wait complete. 2023-10-19T12:45:11.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.235+0000 7f1239133640 1 Processor -- start 2023-10-19T12:45:11.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.236+0000 7f1239133640 1 -- start start 2023-10-19T12:45:11.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.236+0000 7f1239133640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f123406d2d0 0x7f123414e200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:11.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.236+0000 7f1239133640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f123406f0a0 0x7f12340b3b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:11.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.236+0000 7f1239133640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f12340ad370 0x7f12340b40c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:11.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.236+0000 7f1239133640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12340cac00 con 0x7f123406f0a0 2023-10-19T12:45:11.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.236+0000 7f1239133640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12340b4600 con 0x7f123406d2d0 2023-10-19T12:45:11.243 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.236+0000 7f1239133640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12340ca260 con 0x7f12340ad370 2023-10-19T12:45:11.243 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.236+0000 7f122a59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f123406f0a0 0x7f12340b3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:11.243 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.236+0000 7f122a59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f123406f0a0 0x7f12340b3b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59926/0 (socket says 172.21.15.57:59926) 2023-10-19T12:45:11.243 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.236+0000 7f122a59c640 1 -- 172.21.15.57:0/152302737 learned_addr learned my addr 172.21.15.57:0/152302737 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:11.243 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.237+0000 7f1232d9d640 1 --2- 172.21.15.57:0/152302737 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f123406d2d0 0x7f123414e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:11.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.237+0000 7f123359e640 1 --2- 172.21.15.57:0/152302737 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f12340ad370 0x7f12340b40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:11.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.237+0000 7f122a59c640 1 -- 172.21.15.57:0/152302737 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f12340ad370 msgr2=0x7f12340b40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:11.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.237+0000 7f122a59c640 1 --2- 172.21.15.57:0/152302737 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f12340ad370 0x7f12340b40c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:11.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.237+0000 7f122a59c640 1 -- 172.21.15.57:0/152302737 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f123406d2d0 msgr2=0x7f123414e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:11.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.237+0000 7f122a59c640 1 --2- 172.21.15.57:0/152302737 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f123406d2d0 0x7f123414e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:11.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.237+0000 7f122a59c640 1 -- 172.21.15.57:0/152302737 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f122c009840 con 0x7f123406f0a0 2023-10-19T12:45:11.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.237+0000 7f123359e640 1 --2- 172.21.15.57:0/152302737 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f12340ad370 0x7f12340b40c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:45:11.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.237+0000 7f122a59c640 1 --2- 172.21.15.57:0/152302737 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f123406f0a0 0x7f12340b3b80 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f121c00c500 tx=0x7f121c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:11.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.237+0000 7f122bfff640 1 -- 172.21.15.57:0/152302737 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f121c012830 con 0x7f123406f0a0 2023-10-19T12:45:11.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.237+0000 7f1239133640 1 -- 172.21.15.57:0/152302737 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12340ca540 con 0x7f123406f0a0 2023-10-19T12:45:11.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.237+0000 7f122bfff640 1 -- 172.21.15.57:0/152302737 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f121c012a80 con 0x7f123406f0a0 2023-10-19T12:45:11.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.237+0000 7f122bfff640 1 -- 172.21.15.57:0/152302737 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f121c012e90 con 0x7f123406f0a0 2023-10-19T12:45:11.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.237+0000 7f1239133640 1 -- 172.21.15.57:0/152302737 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12340c8980 con 0x7f123406f0a0 2023-10-19T12:45:11.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.238+0000 7f1239133640 1 -- 172.21.15.57:0/152302737 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12340ce1d0 con 0x7f123406f0a0 2023-10-19T12:45:11.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.239+0000 7f122bfff640 1 -- 172.21.15.57:0/152302737 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f121c01a3b0 con 0x7f123406f0a0 2023-10-19T12:45:11.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.239+0000 7f122bfff640 1 --2- 172.21.15.57:0/152302737 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f121406ae70 0x7f121406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:11.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.239+0000 7f122bfff640 1 -- 172.21.15.57:0/152302737 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f121c089820 con 0x7f123406f0a0 2023-10-19T12:45:11.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.240+0000 7f1232d9d640 1 --2- 172.21.15.57:0/152302737 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f121406ae70 0x7f121406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:11.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.240+0000 7f1232d9d640 1 --2- 172.21.15.57:0/152302737 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f121406ae70 0x7f121406d330 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f1224009bd0 tx=0x7f1224005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:11.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.243+0000 7f122bfff640 1 -- 172.21.15.57:0/152302737 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f121c054520 con 0x7f123406f0a0 2023-10-19T12:45:11.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.421+0000 7f1239133640 1 -- 172.21.15.57:0/152302737 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f12340cad40 con 0x7f123406f0a0 2023-10-19T12:45:11.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.421+0000 7f122bfff640 1 -- 172.21.15.57:0/152302737 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7f121c016020 con 0x7f123406f0a0 2023-10-19T12:45:11.425 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2023-10-19T12:45:11.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.423+0000 7f1239133640 1 -- 172.21.15.57:0/152302737 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f121406ae70 msgr2=0x7f121406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:11.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.423+0000 7f1239133640 1 --2- 172.21.15.57:0/152302737 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f121406ae70 0x7f121406d330 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f1224009bd0 tx=0x7f1224005f90 comp rx=0 tx=0).stop 2023-10-19T12:45:11.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.423+0000 7f1239133640 1 -- 172.21.15.57:0/152302737 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f123406f0a0 msgr2=0x7f12340b3b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:11.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.423+0000 7f1239133640 1 --2- 172.21.15.57:0/152302737 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f123406f0a0 0x7f12340b3b80 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f121c00c500 tx=0x7f121c00c9c0 comp rx=0 tx=0).stop 2023-10-19T12:45:11.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.423+0000 7f1239133640 1 -- 172.21.15.57:0/152302737 shutdown_connections 2023-10-19T12:45:11.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.423+0000 7f1239133640 1 --2- 172.21.15.57:0/152302737 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f121406ae70 0x7f121406d330 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:11.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.423+0000 7f1239133640 1 --2- 172.21.15.57:0/152302737 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f12340ad370 0x7f12340b40c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:11.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.423+0000 7f1239133640 1 --2- 172.21.15.57:0/152302737 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f123406f0a0 0x7f12340b3b80 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:11.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.423+0000 7f1239133640 1 --2- 172.21.15.57:0/152302737 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f123406d2d0 0x7f123414e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:11.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.423+0000 7f1239133640 1 -- 172.21.15.57:0/152302737 >> 172.21.15.57:0/152302737 conn(0x7f12340b0670 msgr2=0x7f1234158ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:11.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.423+0000 7f1239133640 1 -- 172.21.15.57:0/152302737 shutdown_connections 2023-10-19T12:45:11.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.423+0000 7f1239133640 1 -- 172.21.15.57:0/152302737 wait complete. 2023-10-19T12:45:11.438 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-19T12:45:11.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.544+0000 7f7ed43e2640 1 -- 172.21.15.57:0/3608148630 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ecc0afa20 msgr2=0x7f7ecc158300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:11.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.544+0000 7f7ed43e2640 1 --2- 172.21.15.57:0/3608148630 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ecc0afa20 0x7f7ecc158300 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc005fd0 tx=0x7f7ebc02be10 comp rx=0 tx=0).stop 2023-10-19T12:45:11.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.544+0000 7f7ed43e2640 1 -- 172.21.15.57:0/3608148630 shutdown_connections 2023-10-19T12:45:11.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.544+0000 7f7ed43e2640 1 --2- 172.21.15.57:0/3608148630 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7ecc0b0530 0x7f7ecc0b09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:11.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.544+0000 7f7ed43e2640 1 --2- 172.21.15.57:0/3608148630 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ecc0afa20 0x7f7ecc158300 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:11.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.544+0000 7f7ed43e2640 1 -- 172.21.15.57:0/3608148630 >> 172.21.15.57:0/3608148630 conn(0x7f7ecc069930 msgr2=0x7f7ecc069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:11.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.544+0000 7f7ed43e2640 1 -- 172.21.15.57:0/3608148630 shutdown_connections 2023-10-19T12:45:11.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.544+0000 7f7ed43e2640 1 -- 172.21.15.57:0/3608148630 wait complete. 2023-10-19T12:45:11.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.546+0000 7f7ed43e2640 1 Processor -- start 2023-10-19T12:45:11.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.546+0000 7f7ed43e2640 1 -- start start 2023-10-19T12:45:11.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.546+0000 7f7ed43e2640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7ecc0afa20 0x7f7ecc143760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:11.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.546+0000 7f7ed43e2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ecc0b0530 0x7f7ecc13e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:11.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.546+0000 7f7ed43e2640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7ecc0cbd20 0x7f7ecc1ef290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:11.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.546+0000 7f7ed43e2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ecc1479f0 con 0x7f7ecc0b0530 2023-10-19T12:45:11.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.546+0000 7f7ed43e2640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ecc1426f0 con 0x7f7ecc0afa20 2023-10-19T12:45:11.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.546+0000 7f7ed43e2640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ecc146d70 con 0x7f7ecc0cbd20 2023-10-19T12:45:11.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.546+0000 7f7ed197d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ecc0b0530 0x7f7ecc13e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:11.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.546+0000 7f7ed197d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ecc0b0530 0x7f7ecc13e490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59944/0 (socket says 172.21.15.57:59944) 2023-10-19T12:45:11.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.546+0000 7f7ed197d640 1 -- 172.21.15.57:0/1642047393 learned_addr learned my addr 172.21.15.57:0/1642047393 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:11.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.547+0000 7f7ed197d640 1 -- 172.21.15.57:0/1642047393 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7ecc0cbd20 msgr2=0x7f7ecc1ef290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:11.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.547+0000 7f7ed197d640 1 --2- 172.21.15.57:0/1642047393 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7ecc0cbd20 0x7f7ecc1ef290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:11.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.547+0000 7f7ed197d640 1 -- 172.21.15.57:0/1642047393 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7ecc0afa20 msgr2=0x7f7ecc143760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:11.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.547+0000 7f7ed197d640 1 --2- 172.21.15.57:0/1642047393 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7ecc0afa20 0x7f7ecc143760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:11.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.547+0000 7f7ed197d640 1 -- 172.21.15.57:0/1642047393 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ebc009560 con 0x7f7ecc0b0530 2023-10-19T12:45:11.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.547+0000 7f7ed197d640 1 --2- 172.21.15.57:0/1642047393 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ecc0b0530 0x7f7ecc13e490 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc009850 tx=0x7f7ebc02dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:11.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.547+0000 7f7ec2ffd640 1 -- 172.21.15.57:0/1642047393 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ebc0035f0 con 0x7f7ecc0b0530 2023-10-19T12:45:11.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.547+0000 7f7ec2ffd640 1 -- 172.21.15.57:0/1642047393 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f7ebc003840 con 0x7f7ecc0b0530 2023-10-19T12:45:11.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.547+0000 7f7ed43e2640 1 -- 172.21.15.57:0/1642047393 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ecc147050 con 0x7f7ecc0b0530 2023-10-19T12:45:11.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.547+0000 7f7ec2ffd640 1 -- 172.21.15.57:0/1642047393 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ebc003bf0 con 0x7f7ecc0b0530 2023-10-19T12:45:11.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.547+0000 7f7ed43e2640 1 -- 172.21.15.57:0/1642047393 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ecc147610 con 0x7f7ecc0b0530 2023-10-19T12:45:11.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.548+0000 7f7ed43e2640 1 -- 172.21.15.57:0/1642047393 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ecc147b30 con 0x7f7ecc0b0530 2023-10-19T12:45:11.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.549+0000 7f7ec2ffd640 1 -- 172.21.15.57:0/1642047393 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f7ebc003d50 con 0x7f7ecc0b0530 2023-10-19T12:45:11.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.549+0000 7f7ec2ffd640 1 --2- 172.21.15.57:0/1642047393 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f7eb006ae70 0x7f7eb006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:11.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.549+0000 7f7ec2ffd640 1 -- 172.21.15.57:0/1642047393 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f7ebc0ac610 con 0x7f7ecc0b0530 2023-10-19T12:45:11.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.549+0000 7f7ed217e640 1 --2- 172.21.15.57:0/1642047393 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f7eb006ae70 0x7f7eb006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:11.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.550+0000 7f7ed217e640 1 --2- 172.21.15.57:0/1642047393 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f7eb006ae70 0x7f7eb006d330 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f7ec8000f60 tx=0x7f7ec80079b0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:11.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.553+0000 7f7ec2ffd640 1 -- 172.21.15.57:0/1642047393 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7ecc147b30 con 0x7f7ecc0b0530 2023-10-19T12:45:11.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.746+0000 7f7ed43e2640 1 -- 172.21.15.57:0/1642047393 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f7ecc147b30 con 0x7f7ecc0b0530 2023-10-19T12:45:11.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.746+0000 7f7ec2ffd640 1 -- 172.21.15.57:0/1642047393 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) v1 ==== 80+0+0 (secure 0 0 0) 0x7f7ecc147b30 con 0x7f7ecc0b0530 2023-10-19T12:45:11.750 INFO:teuthology.orchestra.run.smithi057.stderr:listed 0 entries 2023-10-19T12:45:11.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.749+0000 7f7ed43e2640 1 -- 172.21.15.57:0/1642047393 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f7eb006ae70 msgr2=0x7f7eb006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:11.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.749+0000 7f7ed43e2640 1 --2- 172.21.15.57:0/1642047393 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f7eb006ae70 0x7f7eb006d330 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f7ec8000f60 tx=0x7f7ec80079b0 comp rx=0 tx=0).stop 2023-10-19T12:45:11.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.749+0000 7f7ed43e2640 1 -- 172.21.15.57:0/1642047393 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ecc0b0530 msgr2=0x7f7ecc13e490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:11.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.749+0000 7f7ed43e2640 1 --2- 172.21.15.57:0/1642047393 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ecc0b0530 0x7f7ecc13e490 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc009850 tx=0x7f7ebc02dae0 comp rx=0 tx=0).stop 2023-10-19T12:45:11.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.749+0000 7f7ed43e2640 1 -- 172.21.15.57:0/1642047393 shutdown_connections 2023-10-19T12:45:11.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.749+0000 7f7ed43e2640 1 --2- 172.21.15.57:0/1642047393 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f7eb006ae70 0x7f7eb006d330 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:11.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.749+0000 7f7ed43e2640 1 --2- 172.21.15.57:0/1642047393 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7ecc0cbd20 0x7f7ecc1ef290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:11.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.749+0000 7f7ed43e2640 1 --2- 172.21.15.57:0/1642047393 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ecc0b0530 0x7f7ecc13e490 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:11.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.749+0000 7f7ed43e2640 1 --2- 172.21.15.57:0/1642047393 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7ecc0afa20 0x7f7ecc143760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:11.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.749+0000 7f7ed43e2640 1 -- 172.21.15.57:0/1642047393 >> 172.21.15.57:0/1642047393 conn(0x7f7ecc069930 msgr2=0x7f7ecc0aed70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:11.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.749+0000 7f7ed43e2640 1 -- 172.21.15.57:0/1642047393 shutdown_connections 2023-10-19T12:45:11.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.749+0000 7f7ed43e2640 1 -- 172.21.15.57:0/1642047393 wait complete. 2023-10-19T12:45:11.763 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T12:45:11.763 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T12:45:11.780 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T12:45:11.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.926+0000 7fc424c4b640 1 -- 172.21.15.57:0/2260621142 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc4200cf340 msgr2=0x7fc4200d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:11.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.926+0000 7fc424c4b640 1 --2- 172.21.15.57:0/2260621142 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc4200cf340 0x7fc4200d1910 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fc410005fd0 tx=0x7fc41002bd90 comp rx=0 tx=0).stop 2023-10-19T12:45:11.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.926+0000 7fc424c4b640 1 -- 172.21.15.57:0/2260621142 shutdown_connections 2023-10-19T12:45:11.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.926+0000 7fc424c4b640 1 --2- 172.21.15.57:0/2260621142 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc42015b2a0 0x7fc4200cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:11.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.926+0000 7fc424c4b640 1 --2- 172.21.15.57:0/2260621142 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc4200cf340 0x7fc4200d1910 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:11.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.926+0000 7fc424c4b640 1 -- 172.21.15.57:0/2260621142 >> 172.21.15.57:0/2260621142 conn(0x7fc4200d0d60 msgr2=0x7fc4200d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:11.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.926+0000 7fc424c4b640 1 -- 172.21.15.57:0/2260621142 shutdown_connections 2023-10-19T12:45:11.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.926+0000 7fc424c4b640 1 -- 172.21.15.57:0/2260621142 wait complete. 2023-10-19T12:45:11.932 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.927+0000 7fc424c4b640 1 Processor -- start 2023-10-19T12:45:11.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.927+0000 7fc424c4b640 1 -- start start 2023-10-19T12:45:11.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.928+0000 7fc424c4b640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc4200ceb40 0x7fc4200b5680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:11.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.928+0000 7fc424c4b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4200cf340 0x7fc42015b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:11.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.928+0000 7fc424c4b640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc42015b2a0 0x7fc420157e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:11.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.928+0000 7fc424c4b640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc42015bc10 con 0x7fc4200cf340 2023-10-19T12:45:11.934 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.928+0000 7fc424c4b640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4200b67b0 con 0x7fc42015b2a0 2023-10-19T12:45:11.934 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.928+0000 7fc424c4b640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc420158360 con 0x7fc4200ceb40 2023-10-19T12:45:11.934 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.928+0000 7fc417fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4200cf340 0x7fc42015b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:11.934 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.928+0000 7fc417fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4200cf340 0x7fc42015b6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59946/0 (socket says 172.21.15.57:59946) 2023-10-19T12:45:11.935 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.928+0000 7fc417fff640 1 -- 172.21.15.57:0/3877841669 learned_addr learned my addr 172.21.15.57:0/3877841669 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:11.935 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.928+0000 7fc41e59c640 1 --2- 172.21.15.57:0/3877841669 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc4200ceb40 0x7fc4200b5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:11.935 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.928+0000 7fc41ed9d640 1 --2- 172.21.15.57:0/3877841669 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc42015b2a0 0x7fc420157e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:11.935 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.928+0000 7fc417fff640 1 -- 172.21.15.57:0/3877841669 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc4200ceb40 msgr2=0x7fc4200b5680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:11.935 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.928+0000 7fc417fff640 1 --2- 172.21.15.57:0/3877841669 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc4200ceb40 0x7fc4200b5680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:11.936 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.928+0000 7fc417fff640 1 -- 172.21.15.57:0/3877841669 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc42015b2a0 msgr2=0x7fc420157e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:11.936 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.928+0000 7fc417fff640 1 --2- 172.21.15.57:0/3877841669 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc42015b2a0 0x7fc420157e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:11.936 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.928+0000 7fc417fff640 1 -- 172.21.15.57:0/3877841669 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc410009560 con 0x7fc4200cf340 2023-10-19T12:45:11.936 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.928+0000 7fc417fff640 1 --2- 172.21.15.57:0/3877841669 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4200cf340 0x7fc42015b6d0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fc40800e7c0 tx=0x7fc40800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:11.936 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.929+0000 7fc416ffd640 1 -- 172.21.15.57:0/3877841669 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc408009580 con 0x7fc4200cf340 2023-10-19T12:45:11.936 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.929+0000 7fc424c4b640 1 -- 172.21.15.57:0/3877841669 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc42015be40 con 0x7fc4200cf340 2023-10-19T12:45:11.937 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.929+0000 7fc416ffd640 1 -- 172.21.15.57:0/3877841669 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fc40800cd50 con 0x7fc4200cf340 2023-10-19T12:45:11.937 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.929+0000 7fc416ffd640 1 -- 172.21.15.57:0/3877841669 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc40800cee0 con 0x7fc4200cf340 2023-10-19T12:45:11.937 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.929+0000 7fc424c4b640 1 -- 172.21.15.57:0/3877841669 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc42015c400 con 0x7fc4200cf340 2023-10-19T12:45:11.937 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.930+0000 7fc424c4b640 1 -- 172.21.15.57:0/3877841669 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4200690e0 con 0x7fc4200cf340 2023-10-19T12:45:11.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.930+0000 7fc416ffd640 1 -- 172.21.15.57:0/3877841669 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fc408010690 con 0x7fc4200cf340 2023-10-19T12:45:11.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.931+0000 7fc416ffd640 1 --2- 172.21.15.57:0/3877841669 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fc40006ae40 0x7fc40006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:11.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.931+0000 7fc416ffd640 1 -- 172.21.15.57:0/3877841669 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fc408014070 con 0x7fc4200cf340 2023-10-19T12:45:11.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.931+0000 7fc41e59c640 1 --2- 172.21.15.57:0/3877841669 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fc40006ae40 0x7fc40006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:11.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.931+0000 7fc41e59c640 1 --2- 172.21.15.57:0/3877841669 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fc40006ae40 0x7fc40006d300 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fc410035000 tx=0x7fc410004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:11.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:11.935+0000 7fc416ffd640 1 -- 172.21.15.57:0/3877841669 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc408051ca0 con 0x7fc4200cf340 2023-10-19T12:45:12.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.116+0000 7fc424c4b640 1 -- 172.21.15.57:0/3877841669 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc4200d0b40 con 0x7fc4200cf340 2023-10-19T12:45:12.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.117+0000 7fc416ffd640 1 -- 172.21.15.57:0/3877841669 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18748 (secure 0 0 0) 0x7fc408058b40 con 0x7fc4200cf340 2023-10-19T12:45:12.121 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:45:12.121 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":27,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T12:44:58.978403+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-19T12:44:16.517510+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-19T12:44:17.398297+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-19T12:44:21.129403+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.23","mappings":[{"from":3,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"3.39","mappings":[{"from":7,"to":1}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.1d","mappings":[{"from":1,"to":5}]},{"pgid":"4.1e","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T12:45:12.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.119+0000 7fc424c4b640 1 -- 172.21.15.57:0/3877841669 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fc40006ae40 msgr2=0x7fc40006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:12.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.119+0000 7fc424c4b640 1 --2- 172.21.15.57:0/3877841669 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fc40006ae40 0x7fc40006d300 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fc410035000 tx=0x7fc410004730 comp rx=0 tx=0).stop 2023-10-19T12:45:12.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.119+0000 7fc424c4b640 1 -- 172.21.15.57:0/3877841669 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4200cf340 msgr2=0x7fc42015b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:12.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.119+0000 7fc424c4b640 1 --2- 172.21.15.57:0/3877841669 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4200cf340 0x7fc42015b6d0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fc40800e7c0 tx=0x7fc40800ec80 comp rx=0 tx=0).stop 2023-10-19T12:45:12.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.119+0000 7fc424c4b640 1 -- 172.21.15.57:0/3877841669 shutdown_connections 2023-10-19T12:45:12.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.119+0000 7fc424c4b640 1 --2- 172.21.15.57:0/3877841669 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fc40006ae40 0x7fc40006d300 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:12.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.119+0000 7fc424c4b640 1 --2- 172.21.15.57:0/3877841669 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc42015b2a0 0x7fc420157e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:12.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.119+0000 7fc424c4b640 1 --2- 172.21.15.57:0/3877841669 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4200cf340 0x7fc42015b6d0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:12.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.119+0000 7fc424c4b640 1 --2- 172.21.15.57:0/3877841669 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc4200ceb40 0x7fc4200b5680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:12.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.119+0000 7fc424c4b640 1 -- 172.21.15.57:0/3877841669 >> 172.21.15.57:0/3877841669 conn(0x7fc4200d0d60 msgr2=0x7fc42015a1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:12.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.120+0000 7fc424c4b640 1 -- 172.21.15.57:0/3877841669 shutdown_connections 2023-10-19T12:45:12.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.120+0000 7fc424c4b640 1 -- 172.21.15.57:0/3877841669 wait complete. 2023-10-19T12:45:12.134 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-19T12:44:07.371186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-19T12:44:16.517510+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-19T12:44:17.398297+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-19T12:44:21.129403+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-19T12:45:12.134 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-19T12:45:12.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.244+0000 7fbfc007c640 1 -- 172.21.15.57:0/1393332220 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbfb806f0a0 msgr2=0x7fbfb806dbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:12.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.244+0000 7fbfc007c640 1 --2- 172.21.15.57:0/1393332220 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbfb806f0a0 0x7fbfb806dbf0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fbfac009830 tx=0x7fbfac02bd60 comp rx=0 tx=0).stop 2023-10-19T12:45:12.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.244+0000 7fbfc007c640 1 -- 172.21.15.57:0/1393332220 shutdown_connections 2023-10-19T12:45:12.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.244+0000 7fbfc007c640 1 --2- 172.21.15.57:0/1393332220 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbfb806f0a0 0x7fbfb806dbf0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:12.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.244+0000 7fbfc007c640 1 --2- 172.21.15.57:0/1393332220 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbfb806d2d0 0x7fbfb806d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:12.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.244+0000 7fbfc007c640 1 -- 172.21.15.57:0/1393332220 >> 172.21.15.57:0/1393332220 conn(0x7fbfb80b0670 msgr2=0x7fbfb80b0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:12.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.245+0000 7fbfc007c640 1 -- 172.21.15.57:0/1393332220 shutdown_connections 2023-10-19T12:45:12.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.245+0000 7fbfc007c640 1 -- 172.21.15.57:0/1393332220 wait complete. 2023-10-19T12:45:12.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.246+0000 7fbfc007c640 1 Processor -- start 2023-10-19T12:45:12.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.246+0000 7fbfc007c640 1 -- start start 2023-10-19T12:45:12.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.246+0000 7fbfc007c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbfb806d2d0 0x7fbfb806a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:12.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.246+0000 7fbfc007c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbfb806f0a0 0x7fbfb81f5e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:12.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.246+0000 7fbfc007c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbfb80ab950 0x7fbfb80c9580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:12.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.246+0000 7fbfc007c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfb806ae10 con 0x7fbfb806d2d0 2023-10-19T12:45:12.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.247+0000 7fbfc007c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfb806b0f0 con 0x7fbfb80ab950 2023-10-19T12:45:12.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.247+0000 7fbfc007c640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfb806b3d0 con 0x7fbfb806f0a0 2023-10-19T12:45:12.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.247+0000 7fbfbde18640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbfb806d2d0 0x7fbfb806a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:12.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.247+0000 7fbfbde18640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbfb806d2d0 0x7fbfb806a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59952/0 (socket says 172.21.15.57:59952) 2023-10-19T12:45:12.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.247+0000 7fbfbde18640 1 -- 172.21.15.57:0/1039787943 learned_addr learned my addr 172.21.15.57:0/1039787943 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:12.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.247+0000 7fbfbe619640 1 --2- 172.21.15.57:0/1039787943 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbfb80ab950 0x7fbfb80c9580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:12.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.247+0000 7fbfbd617640 1 --2- 172.21.15.57:0/1039787943 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbfb806f0a0 0x7fbfb81f5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:12.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.247+0000 7fbfbde18640 1 -- 172.21.15.57:0/1039787943 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbfb806f0a0 msgr2=0x7fbfb81f5e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:12.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.247+0000 7fbfbde18640 1 --2- 172.21.15.57:0/1039787943 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbfb806f0a0 0x7fbfb81f5e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:12.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.247+0000 7fbfbde18640 1 -- 172.21.15.57:0/1039787943 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbfb80ab950 msgr2=0x7fbfb80c9580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:12.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.247+0000 7fbfbde18640 1 --2- 172.21.15.57:0/1039787943 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbfb80ab950 0x7fbfb80c9580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:12.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.247+0000 7fbfbde18640 1 -- 172.21.15.57:0/1039787943 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfac009490 con 0x7fbfb806d2d0 2023-10-19T12:45:12.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.247+0000 7fbfbde18640 1 --2- 172.21.15.57:0/1039787943 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbfb806d2d0 0x7fbfb806a830 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fbfa800efb0 tx=0x7fbfa800c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:12.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.247+0000 7fbfb67fc640 1 -- 172.21.15.57:0/1039787943 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfa800b570 con 0x7fbfb806d2d0 2023-10-19T12:45:12.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.247+0000 7fbfb67fc640 1 -- 172.21.15.57:0/1039787943 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fbfa800b7c0 con 0x7fbfb806d2d0 2023-10-19T12:45:12.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.247+0000 7fbfc007c640 1 -- 172.21.15.57:0/1039787943 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfb806b6b0 con 0x7fbfb806d2d0 2023-10-19T12:45:12.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.247+0000 7fbfb67fc640 1 -- 172.21.15.57:0/1039787943 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfa800bbd0 con 0x7fbfb806d2d0 2023-10-19T12:45:12.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.248+0000 7fbfc007c640 1 -- 172.21.15.57:0/1039787943 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfb81f63a0 con 0x7fbfb806d2d0 2023-10-19T12:45:12.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.248+0000 7fbfc007c640 1 -- 172.21.15.57:0/1039787943 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfb80ce1d0 con 0x7fbfb806d2d0 2023-10-19T12:45:12.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.249+0000 7fbfb67fc640 1 -- 172.21.15.57:0/1039787943 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fbfa800a040 con 0x7fbfb806d2d0 2023-10-19T12:45:12.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.249+0000 7fbfb67fc640 1 --2- 172.21.15.57:0/1039787943 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fbfa406ae40 0x7fbfa406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:12.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.249+0000 7fbfb67fc640 1 -- 172.21.15.57:0/1039787943 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fbfa808a170 con 0x7fbfb806d2d0 2023-10-19T12:45:12.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.250+0000 7fbfbd617640 1 --2- 172.21.15.57:0/1039787943 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fbfa406ae40 0x7fbfa406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:12.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.250+0000 7fbfbd617640 1 --2- 172.21.15.57:0/1039787943 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fbfa406ae40 0x7fbfa406d300 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fbfac036000 tx=0x7fbfac004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:12.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.254+0000 7fbfb67fc640 1 -- 172.21.15.57:0/1039787943 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fbfa8054e90 con 0x7fbfb806d2d0 2023-10-19T12:45:12.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.428+0000 7fbfc007c640 1 -- 172.21.15.57:0/1039787943 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fbfb80690e0 con 0x7fbfb806d2d0 2023-10-19T12:45:12.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.429+0000 7fbfb67fc640 1 -- 172.21.15.57:0/1039787943 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7fbfa80584b0 con 0x7fbfb806d2d0 2023-10-19T12:45:12.432 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-19T12:45:12.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.430+0000 7fbfc007c640 1 -- 172.21.15.57:0/1039787943 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fbfa406ae40 msgr2=0x7fbfa406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:12.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.431+0000 7fbfc007c640 1 --2- 172.21.15.57:0/1039787943 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fbfa406ae40 0x7fbfa406d300 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fbfac036000 tx=0x7fbfac004730 comp rx=0 tx=0).stop 2023-10-19T12:45:12.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.431+0000 7fbfc007c640 1 -- 172.21.15.57:0/1039787943 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbfb806d2d0 msgr2=0x7fbfb806a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:12.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.431+0000 7fbfc007c640 1 --2- 172.21.15.57:0/1039787943 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbfb806d2d0 0x7fbfb806a830 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fbfa800efb0 tx=0x7fbfa800c800 comp rx=0 tx=0).stop 2023-10-19T12:45:12.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.431+0000 7fbfc007c640 1 -- 172.21.15.57:0/1039787943 shutdown_connections 2023-10-19T12:45:12.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.431+0000 7fbfc007c640 1 --2- 172.21.15.57:0/1039787943 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fbfa406ae40 0x7fbfa406d300 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:12.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.431+0000 7fbfc007c640 1 --2- 172.21.15.57:0/1039787943 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbfb80ab950 0x7fbfb80c9580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:12.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.431+0000 7fbfc007c640 1 --2- 172.21.15.57:0/1039787943 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbfb806f0a0 0x7fbfb81f5e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:12.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.431+0000 7fbfc007c640 1 --2- 172.21.15.57:0/1039787943 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbfb806d2d0 0x7fbfb806a830 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:12.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.431+0000 7fbfc007c640 1 -- 172.21.15.57:0/1039787943 >> 172.21.15.57:0/1039787943 conn(0x7fbfb80b0670 msgr2=0x7fbfb8159c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:12.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.431+0000 7fbfc007c640 1 -- 172.21.15.57:0/1039787943 shutdown_connections 2023-10-19T12:45:12.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.431+0000 7fbfc007c640 1 -- 172.21.15.57:0/1039787943 wait complete. 2023-10-19T12:45:12.444 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-19T12:45:12.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.601+0000 7f94365d0640 1 -- 172.21.15.57:0/503872963 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f943006ae40 msgr2=0x7f94300b0e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:12.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.601+0000 7f94365d0640 1 --2- 172.21.15.57:0/503872963 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f943006ae40 0x7f94300b0e90 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f9420005fd0 tx=0x7f942002be20 comp rx=0 tx=0).stop 2023-10-19T12:45:12.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.601+0000 7f94365d0640 1 -- 172.21.15.57:0/503872963 shutdown_connections 2023-10-19T12:45:12.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.601+0000 7f94365d0640 1 --2- 172.21.15.57:0/503872963 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f943006ae40 0x7f94300b0e90 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:12.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.601+0000 7f94365d0640 1 -- 172.21.15.57:0/503872963 >> 172.21.15.57:0/503872963 conn(0x7f943006fae0 msgr2=0x7f943006fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:12.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.601+0000 7f94365d0640 1 -- 172.21.15.57:0/503872963 shutdown_connections 2023-10-19T12:45:12.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.601+0000 7f94365d0640 1 -- 172.21.15.57:0/503872963 wait complete. 2023-10-19T12:45:12.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.602+0000 7f94365d0640 1 Processor -- start 2023-10-19T12:45:12.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.602+0000 7f94365d0640 1 -- start start 2023-10-19T12:45:12.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.603+0000 7f94365d0640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f943006ae40 0x7f94301e3560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:12.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.603+0000 7f94365d0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f94301579f0 0x7f9430152c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:12.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.603+0000 7f94365d0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f943015c1f0 0x7f9430154850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:12.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.603+0000 7f94365d0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94301e20e0 con 0x7f94301579f0 2023-10-19T12:45:12.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.603+0000 7f94365d0640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94301e3e40 con 0x7f943015c1f0 2023-10-19T12:45:12.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.603+0000 7f94365d0640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94301e4120 con 0x7f943006ae40 2023-10-19T12:45:12.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.603+0000 7f942f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f94301579f0 0x7f9430152c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:12.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.603+0000 7f942f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f94301579f0 0x7f9430152c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59962/0 (socket says 172.21.15.57:59962) 2023-10-19T12:45:12.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.603+0000 7f942f7fe640 1 -- 172.21.15.57:0/2140886228 learned_addr learned my addr 172.21.15.57:0/2140886228 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:12.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.603+0000 7f9434b6d640 1 --2- 172.21.15.57:0/2140886228 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f943015c1f0 0x7f9430154850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:12.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.603+0000 7f942ffff640 1 --2- 172.21.15.57:0/2140886228 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f943006ae40 0x7f94301e3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:12.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.603+0000 7f942f7fe640 1 -- 172.21.15.57:0/2140886228 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f943006ae40 msgr2=0x7f94301e3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:12.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.603+0000 7f942f7fe640 1 --2- 172.21.15.57:0/2140886228 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f943006ae40 0x7f94301e3560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:12.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.603+0000 7f942f7fe640 1 -- 172.21.15.57:0/2140886228 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f943015c1f0 msgr2=0x7f9430154850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:12.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.603+0000 7f942f7fe640 1 --2- 172.21.15.57:0/2140886228 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f943015c1f0 0x7f9430154850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:12.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.603+0000 7f942f7fe640 1 -- 172.21.15.57:0/2140886228 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9420009560 con 0x7f94301579f0 2023-10-19T12:45:12.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.604+0000 7f942f7fe640 1 --2- 172.21.15.57:0/2140886228 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f94301579f0 0x7f9430152c20 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f9420006010 tx=0x7f942002ddf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:12.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.604+0000 7f942cff9640 1 -- 172.21.15.57:0/2140886228 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9420005650 con 0x7f94301579f0 2023-10-19T12:45:12.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.604+0000 7f942cff9640 1 -- 172.21.15.57:0/2140886228 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f94200038f0 con 0x7f94301579f0 2023-10-19T12:45:12.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.604+0000 7f94365d0640 1 -- 172.21.15.57:0/2140886228 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9430154050 con 0x7f94301579f0 2023-10-19T12:45:12.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.604+0000 7f942cff9640 1 -- 172.21.15.57:0/2140886228 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9420003ca0 con 0x7f94301579f0 2023-10-19T12:45:12.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.604+0000 7f94365d0640 1 -- 172.21.15.57:0/2140886228 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94301e3aa0 con 0x7f94301579f0 2023-10-19T12:45:12.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.605+0000 7f94365d0640 1 -- 172.21.15.57:0/2140886228 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9430153eb0 con 0x7f94301579f0 2023-10-19T12:45:12.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.605+0000 7f942cff9640 1 -- 172.21.15.57:0/2140886228 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f942003e450 con 0x7f94301579f0 2023-10-19T12:45:12.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.606+0000 7f942cff9640 1 --2- 172.21.15.57:0/2140886228 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f941c06ae40 0x7f941c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:12.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.606+0000 7f942cff9640 1 -- 172.21.15.57:0/2140886228 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f94200ac6b0 con 0x7f94301579f0 2023-10-19T12:45:12.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.606+0000 7f942ffff640 1 --2- 172.21.15.57:0/2140886228 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f941c06ae40 0x7f941c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:12.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.607+0000 7f942ffff640 1 --2- 172.21.15.57:0/2140886228 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f941c06ae40 0x7f941c06d300 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f9424007fd0 tx=0x7f9424009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:12.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.610+0000 7f942cff9640 1 -- 172.21.15.57:0/2140886228 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9430153eb0 con 0x7f94301579f0 2023-10-19T12:45:12.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.784+0000 7f94365d0640 1 -- 172.21.15.57:0/2140886228 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9430153eb0 con 0x7f94301579f0 2023-10-19T12:45:12.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.784+0000 7f942cff9640 1 -- 172.21.15.57:0/2140886228 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7f9430153eb0 con 0x7f94301579f0 2023-10-19T12:45:12.788 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2023-10-19T12:45:12.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.786+0000 7f94365d0640 1 -- 172.21.15.57:0/2140886228 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f941c06ae40 msgr2=0x7f941c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:12.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.786+0000 7f94365d0640 1 --2- 172.21.15.57:0/2140886228 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f941c06ae40 0x7f941c06d300 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f9424007fd0 tx=0x7f9424009040 comp rx=0 tx=0).stop 2023-10-19T12:45:12.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.786+0000 7f94365d0640 1 -- 172.21.15.57:0/2140886228 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f94301579f0 msgr2=0x7f9430152c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:12.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.786+0000 7f94365d0640 1 --2- 172.21.15.57:0/2140886228 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f94301579f0 0x7f9430152c20 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f9420006010 tx=0x7f942002ddf0 comp rx=0 tx=0).stop 2023-10-19T12:45:12.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.786+0000 7f94365d0640 1 -- 172.21.15.57:0/2140886228 shutdown_connections 2023-10-19T12:45:12.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.786+0000 7f94365d0640 1 --2- 172.21.15.57:0/2140886228 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f941c06ae40 0x7f941c06d300 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:12.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.786+0000 7f94365d0640 1 --2- 172.21.15.57:0/2140886228 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f943015c1f0 0x7f9430154850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:12.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.786+0000 7f94365d0640 1 --2- 172.21.15.57:0/2140886228 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f94301579f0 0x7f9430152c20 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:12.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.786+0000 7f94365d0640 1 --2- 172.21.15.57:0/2140886228 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f943006ae40 0x7f94301e3560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:12.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.786+0000 7f94365d0640 1 -- 172.21.15.57:0/2140886228 >> 172.21.15.57:0/2140886228 conn(0x7f943006fae0 msgr2=0x7f94300cb8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:12.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.786+0000 7f94365d0640 1 -- 172.21.15.57:0/2140886228 shutdown_connections 2023-10-19T12:45:12.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.786+0000 7f94365d0640 1 -- 172.21.15.57:0/2140886228 wait complete. 2023-10-19T12:45:12.800 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-19T12:45:12.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.909+0000 7f2ef1a5d640 1 -- 172.21.15.57:0/2987280787 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2ed8004d00 msgr2=0x7f2ed8005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:12.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.909+0000 7f2ef1a5d640 1 --2- 172.21.15.57:0/2987280787 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2ed8004d00 0x7f2ed8005180 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f2edc005fd0 tx=0x7f2ed80057f0 comp rx=0 tx=0).stop 2023-10-19T12:45:12.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.910+0000 7f2eeb7fe640 1 -- 172.21.15.57:0/2987280787 reap_dead start 2023-10-19T12:45:12.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.910+0000 7f2ef1a5d640 1 -- 172.21.15.57:0/2987280787 shutdown_connections 2023-10-19T12:45:12.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.910+0000 7f2ef1a5d640 1 -- 172.21.15.57:0/2987280787 >> 172.21.15.57:0/2987280787 conn(0x7f2eec0b0670 msgr2=0x7f2eec0b0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:12.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.910+0000 7f2ef1a5d640 1 -- 172.21.15.57:0/2987280787 shutdown_connections 2023-10-19T12:45:12.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.910+0000 7f2ef1a5d640 1 -- 172.21.15.57:0/2987280787 wait complete. 2023-10-19T12:45:12.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.911+0000 7f2ef1a5d640 1 Processor -- start 2023-10-19T12:45:12.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.911+0000 7f2ef1a5d640 1 -- start start 2023-10-19T12:45:12.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.912+0000 7f2ef1a5d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2eec09b9a0 0x7f2eec1485a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:12.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.912+0000 7f2ef1a5d640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2eec09bd80 0x7f2eec149c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:12.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.912+0000 7f2ef1a5d640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2eec14a160 0x7f2eec098d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:12.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.912+0000 7f2ef1a5d640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eec148b80 con 0x7f2eec09b9a0 2023-10-19T12:45:12.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.912+0000 7f2ef1a5d640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eec148e60 con 0x7f2eec14a160 2023-10-19T12:45:12.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.912+0000 7f2ef1a5d640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2eec149140 con 0x7f2eec09bd80 2023-10-19T12:45:12.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.912+0000 7f2eeaffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2eec09b9a0 0x7f2eec1485a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:12.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.912+0000 7f2eeaffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2eec09b9a0 0x7f2eec1485a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59976/0 (socket says 172.21.15.57:59976) 2023-10-19T12:45:12.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.912+0000 7f2eeaffd640 1 -- 172.21.15.57:0/507898960 learned_addr learned my addr 172.21.15.57:0/507898960 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:12.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.912+0000 7f2eeb7fe640 1 --2- 172.21.15.57:0/507898960 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2eec14a160 0x7f2eec098d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:12.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.912+0000 7f2eea7fc640 1 --2- 172.21.15.57:0/507898960 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2eec09bd80 0x7f2eec149c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:12.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.912+0000 7f2eeaffd640 1 -- 172.21.15.57:0/507898960 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2eec09bd80 msgr2=0x7f2eec149c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:12.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.912+0000 7f2eeaffd640 1 --2- 172.21.15.57:0/507898960 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2eec09bd80 0x7f2eec149c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:12.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.912+0000 7f2eeaffd640 1 -- 172.21.15.57:0/507898960 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2eec14a160 msgr2=0x7f2eec098d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:12.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.912+0000 7f2eeaffd640 1 --2- 172.21.15.57:0/507898960 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2eec14a160 0x7f2eec098d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:12.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.912+0000 7f2eeaffd640 1 -- 172.21.15.57:0/507898960 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2eec149420 con 0x7f2eec09b9a0 2023-10-19T12:45:12.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.912+0000 7f2eeb7fe640 1 --2- 172.21.15.57:0/507898960 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2eec14a160 0x7f2eec098d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:45:12.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.912+0000 7f2eeaffd640 1 --2- 172.21.15.57:0/507898960 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2eec09b9a0 0x7f2eec1485a0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f2ee40011e0 tx=0x7f2ee4003aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:12.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.912+0000 7f2ee3fff640 1 -- 172.21.15.57:0/507898960 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ee400f920 con 0x7f2eec09b9a0 2023-10-19T12:45:12.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.912+0000 7f2ee3fff640 1 -- 172.21.15.57:0/507898960 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f2ee400fa80 con 0x7f2eec09b9a0 2023-10-19T12:45:12.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.912+0000 7f2ef1a5d640 1 -- 172.21.15.57:0/507898960 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2eec149700 con 0x7f2eec09b9a0 2023-10-19T12:45:12.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.912+0000 7f2ee3fff640 1 -- 172.21.15.57:0/507898960 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ee400fe90 con 0x7f2eec09b9a0 2023-10-19T12:45:12.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.912+0000 7f2ef1a5d640 1 -- 172.21.15.57:0/507898960 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2eec099430 con 0x7f2eec09b9a0 2023-10-19T12:45:12.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.913+0000 7f2ef1a5d640 1 -- 172.21.15.57:0/507898960 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2eec0ce1d0 con 0x7f2eec09b9a0 2023-10-19T12:45:12.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.914+0000 7f2ee3fff640 1 -- 172.21.15.57:0/507898960 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f2ee4011430 con 0x7f2eec09b9a0 2023-10-19T12:45:12.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.914+0000 7f2ee3fff640 1 --2- 172.21.15.57:0/507898960 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f2ecc06ae70 0x7f2ecc06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:12.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.914+0000 7f2ee3fff640 1 -- 172.21.15.57:0/507898960 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f2ee408a880 con 0x7f2eec09b9a0 2023-10-19T12:45:12.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.915+0000 7f2eea7fc640 1 --2- 172.21.15.57:0/507898960 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f2ecc06ae70 0x7f2ecc06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:12.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.915+0000 7f2eea7fc640 1 --2- 172.21.15.57:0/507898960 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f2ecc06ae70 0x7f2ecc06d330 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f2eec09b700 tx=0x7f2ed4006b60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:12.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:12.919+0000 7f2ee3fff640 1 -- 172.21.15.57:0/507898960 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2ee40554e0 con 0x7f2eec09b9a0 2023-10-19T12:45:13.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.095+0000 7f2ef1a5d640 1 -- 172.21.15.57:0/507898960 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f2eec099650 con 0x7f2eec09b9a0 2023-10-19T12:45:13.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.095+0000 7f2ee3fff640 1 -- 172.21.15.57:0/507898960 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7f2ee4015020 con 0x7f2eec09b9a0 2023-10-19T12:45:13.099 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 64 2023-10-19T12:45:13.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.097+0000 7f2ef1a5d640 1 -- 172.21.15.57:0/507898960 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f2ecc06ae70 msgr2=0x7f2ecc06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:13.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.097+0000 7f2ef1a5d640 1 --2- 172.21.15.57:0/507898960 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f2ecc06ae70 0x7f2ecc06d330 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f2eec09b700 tx=0x7f2ed4006b60 comp rx=0 tx=0).stop 2023-10-19T12:45:13.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.097+0000 7f2ef1a5d640 1 -- 172.21.15.57:0/507898960 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2eec09b9a0 msgr2=0x7f2eec1485a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:13.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.097+0000 7f2ef1a5d640 1 --2- 172.21.15.57:0/507898960 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2eec09b9a0 0x7f2eec1485a0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f2ee40011e0 tx=0x7f2ee4003aa0 comp rx=0 tx=0).stop 2023-10-19T12:45:13.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.097+0000 7f2ef1a5d640 1 -- 172.21.15.57:0/507898960 shutdown_connections 2023-10-19T12:45:13.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.097+0000 7f2ef1a5d640 1 --2- 172.21.15.57:0/507898960 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f2ecc06ae70 0x7f2ecc06d330 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:13.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.097+0000 7f2ef1a5d640 1 --2- 172.21.15.57:0/507898960 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2eec14a160 0x7f2eec098d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:13.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.097+0000 7f2ef1a5d640 1 --2- 172.21.15.57:0/507898960 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2eec09bd80 0x7f2eec149c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:13.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.097+0000 7f2ef1a5d640 1 --2- 172.21.15.57:0/507898960 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2eec09b9a0 0x7f2eec1485a0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:13.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.097+0000 7f2ef1a5d640 1 -- 172.21.15.57:0/507898960 >> 172.21.15.57:0/507898960 conn(0x7f2eec0b0670 msgr2=0x7f2eec0c95a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:13.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.098+0000 7f2ef1a5d640 1 -- 172.21.15.57:0/507898960 shutdown_connections 2023-10-19T12:45:13.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.098+0000 7f2ef1a5d640 1 -- 172.21.15.57:0/507898960 wait complete. 2023-10-19T12:45:13.111 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-19T12:45:13.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.220+0000 7fd73596c640 1 -- 172.21.15.57:0/3273393718 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd7300cf850 msgr2=0x7fd7300cc9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:13.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.220+0000 7fd73596c640 1 --2- 172.21.15.57:0/3273393718 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd7300cf850 0x7fd7300cc9f0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fd7300894b0 tx=0x7fd72802c260 comp rx=0 tx=0).stop 2023-10-19T12:45:13.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.220+0000 7fd73596c640 1 -- 172.21.15.57:0/3273393718 shutdown_connections 2023-10-19T12:45:13.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.220+0000 7fd73596c640 1 --2- 172.21.15.57:0/3273393718 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd7300cf850 0x7fd7300cc9f0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:13.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.220+0000 7fd73596c640 1 --2- 172.21.15.57:0/3273393718 >> v2:172.21.15.179:3300/0 conn(0x7fd73015b260 0x7fd7300cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:13.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.220+0000 7fd73596c640 1 -- 172.21.15.57:0/3273393718 >> 172.21.15.57:0/3273393718 conn(0x7fd73009a030 msgr2=0x7fd7300d0d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:13.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.220+0000 7fd73596c640 1 -- 172.21.15.57:0/3273393718 shutdown_connections 2023-10-19T12:45:13.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.220+0000 7fd73596c640 1 -- 172.21.15.57:0/3273393718 wait complete. 2023-10-19T12:45:13.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.221+0000 7fd73596c640 1 Processor -- start 2023-10-19T12:45:13.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.221+0000 7fd73596c640 1 -- start start 2023-10-19T12:45:13.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.222+0000 7fd73596c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd7300cf850 0x7fd730152280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:13.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.222+0000 7fd73596c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd73015b260 0x7fd730150a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:13.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.222+0000 7fd73596c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd73015b640 0x7fd7300aec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:13.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.222+0000 7fd73596c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd730153d50 con 0x7fd73015b260 2023-10-19T12:45:13.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.222+0000 7fd73596c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd730153f40 con 0x7fd7300cf850 2023-10-19T12:45:13.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.222+0000 7fd73596c640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd730154220 con 0x7fd73015b640 2023-10-19T12:45:13.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.222+0000 7fd7267fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd73015b260 0x7fd730150a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:13.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.222+0000 7fd7267fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd73015b260 0x7fd730150a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59998/0 (socket says 172.21.15.57:59998) 2023-10-19T12:45:13.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.222+0000 7fd72effd640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd7300cf850 0x7fd730152280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:13.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.222+0000 7fd7267fc640 1 -- 172.21.15.57:0/1345415480 learned_addr learned my addr 172.21.15.57:0/1345415480 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:13.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.222+0000 7fd72f7fe640 1 --2- 172.21.15.57:0/1345415480 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd73015b640 0x7fd7300aec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:13.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.223+0000 7fd7267fc640 1 -- 172.21.15.57:0/1345415480 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd73015b640 msgr2=0x7fd7300aec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:13.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.223+0000 7fd7267fc640 1 --2- 172.21.15.57:0/1345415480 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd73015b640 0x7fd7300aec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:13.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.223+0000 7fd7267fc640 1 -- 172.21.15.57:0/1345415480 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd7300cf850 msgr2=0x7fd730152280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:13.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.223+0000 7fd7267fc640 1 --2- 172.21.15.57:0/1345415480 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd7300cf850 0x7fd730152280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:13.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.223+0000 7fd7267fc640 1 -- 172.21.15.57:0/1345415480 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd728009900 con 0x7fd73015b260 2023-10-19T12:45:13.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.223+0000 7fd7267fc640 1 --2- 172.21.15.57:0/1345415480 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd73015b260 0x7fd730150a90 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fd71800b3f0 tx=0x7fd71800b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:13.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.223+0000 7fd72cff9640 1 -- 172.21.15.57:0/1345415480 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd718011830 con 0x7fd73015b260 2023-10-19T12:45:13.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.223+0000 7fd72cff9640 1 -- 172.21.15.57:0/1345415480 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fd718011a80 con 0x7fd73015b260 2023-10-19T12:45:13.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.223+0000 7fd72cff9640 1 -- 172.21.15.57:0/1345415480 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd718010380 con 0x7fd73015b260 2023-10-19T12:45:13.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.223+0000 7fd73596c640 1 -- 172.21.15.57:0/1345415480 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd730154500 con 0x7fd73015b260 2023-10-19T12:45:13.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.223+0000 7fd73596c640 1 -- 172.21.15.57:0/1345415480 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd730152860 con 0x7fd73015b260 2023-10-19T12:45:13.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.224+0000 7fd73596c640 1 -- 172.21.15.57:0/1345415480 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd730154a20 con 0x7fd73015b260 2023-10-19T12:45:13.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.225+0000 7fd72cff9640 1 -- 172.21.15.57:0/1345415480 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fd7180104e0 con 0x7fd73015b260 2023-10-19T12:45:13.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.225+0000 7fd72cff9640 1 --2- 172.21.15.57:0/1345415480 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fd71006ae40 0x7fd71006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:13.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.225+0000 7fd72cff9640 1 -- 172.21.15.57:0/1345415480 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fd718089930 con 0x7fd73015b260 2023-10-19T12:45:13.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.225+0000 7fd72effd640 1 --2- 172.21.15.57:0/1345415480 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fd71006ae40 0x7fd71006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:13.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.226+0000 7fd72effd640 1 --2- 172.21.15.57:0/1345415480 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fd71006ae40 0x7fd71006d300 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fd72000abe0 tx=0x7fd72000a2d0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:13.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.229+0000 7fd72cff9640 1 -- 172.21.15.57:0/1345415480 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd730154a20 con 0x7fd73015b260 2023-10-19T12:45:13.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.405+0000 7fd73596c640 1 -- 172.21.15.57:0/1345415480 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fd730154a20 con 0x7fd73015b260 2023-10-19T12:45:13.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.405+0000 7fd72cff9640 1 -- 172.21.15.57:0/1345415480 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7fd730154a20 con 0x7fd73015b260 2023-10-19T12:45:13.409 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2023-10-19T12:45:13.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.407+0000 7fd73596c640 1 -- 172.21.15.57:0/1345415480 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fd71006ae40 msgr2=0x7fd71006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:13.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.407+0000 7fd73596c640 1 --2- 172.21.15.57:0/1345415480 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fd71006ae40 0x7fd71006d300 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fd72000abe0 tx=0x7fd72000a2d0 comp rx=0 tx=0).stop 2023-10-19T12:45:13.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.407+0000 7fd73596c640 1 -- 172.21.15.57:0/1345415480 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd73015b260 msgr2=0x7fd730150a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:13.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.407+0000 7fd73596c640 1 --2- 172.21.15.57:0/1345415480 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd73015b260 0x7fd730150a90 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fd71800b3f0 tx=0x7fd71800b7f0 comp rx=0 tx=0).stop 2023-10-19T12:45:13.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.407+0000 7fd73596c640 1 -- 172.21.15.57:0/1345415480 shutdown_connections 2023-10-19T12:45:13.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.407+0000 7fd73596c640 1 --2- 172.21.15.57:0/1345415480 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fd71006ae40 0x7fd71006d300 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:13.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.407+0000 7fd73596c640 1 --2- 172.21.15.57:0/1345415480 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd73015b640 0x7fd7300aec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:13.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.407+0000 7fd73596c640 1 --2- 172.21.15.57:0/1345415480 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd73015b260 0x7fd730150a90 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:13.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.407+0000 7fd73596c640 1 --2- 172.21.15.57:0/1345415480 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd7300cf850 0x7fd730152280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:13.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.407+0000 7fd73596c640 1 -- 172.21.15.57:0/1345415480 >> 172.21.15.57:0/1345415480 conn(0x7fd73009a030 msgr2=0x7fd73006a150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:13.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.407+0000 7fd73596c640 1 -- 172.21.15.57:0/1345415480 shutdown_connections 2023-10-19T12:45:13.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.407+0000 7fd73596c640 1 -- 172.21.15.57:0/1345415480 wait complete. 2023-10-19T12:45:13.421 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-19T12:45:13.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.528+0000 7f9681856640 1 -- 172.21.15.57:0/535074184 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f967c15b260 msgr2=0x7f967c0cf310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:13.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.528+0000 7f9681856640 1 --2- 172.21.15.57:0/535074184 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f967c15b260 0x7f967c0cf310 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f966c005fd0 tx=0x7f966c02be10 comp rx=0 tx=0).stop 2023-10-19T12:45:13.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.528+0000 7f9681856640 1 -- 172.21.15.57:0/535074184 shutdown_connections 2023-10-19T12:45:13.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.528+0000 7f9681856640 1 --2- 172.21.15.57:0/535074184 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f967c0cf850 0x7f967c0cc9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:13.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.528+0000 7f9681856640 1 --2- 172.21.15.57:0/535074184 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f967c15b260 0x7f967c0cf310 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:13.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.528+0000 7f9681856640 1 -- 172.21.15.57:0/535074184 >> 172.21.15.57:0/535074184 conn(0x7f967c09a030 msgr2=0x7f967c0d0d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:13.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.528+0000 7f9681856640 1 -- 172.21.15.57:0/535074184 shutdown_connections 2023-10-19T12:45:13.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.528+0000 7f9681856640 1 -- 172.21.15.57:0/535074184 wait complete. 2023-10-19T12:45:13.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.529+0000 7f9681856640 1 Processor -- start 2023-10-19T12:45:13.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.529+0000 7f9681856640 1 -- start start 2023-10-19T12:45:13.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.530+0000 7f9681856640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f967c0cf850 0x7f967c0c97d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:13.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.530+0000 7f9681856640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f967c15b260 0x7f967c15d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:13.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.530+0000 7f9681856640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f967c15b640 0x7f967c1522d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:13.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.530+0000 7f9681856640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f967c0c8800 con 0x7f967c0cf850 2023-10-19T12:45:13.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.530+0000 7f9681856640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f967c0c8a10 con 0x7f967c15b640 2023-10-19T12:45:13.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.530+0000 7f9681856640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f967c0c8cf0 con 0x7f967c15b260 2023-10-19T12:45:13.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.530+0000 7f967affd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f967c0cf850 0x7f967c0c97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:13.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.530+0000 7f967affd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f967c0cf850 0x7f967c0c97d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:60026/0 (socket says 172.21.15.57:60026) 2023-10-19T12:45:13.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.530+0000 7f967affd640 1 -- 172.21.15.57:0/3052729369 learned_addr learned my addr 172.21.15.57:0/3052729369 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:13.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.530+0000 7f967b7fe640 1 --2- 172.21.15.57:0/3052729369 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f967c15b640 0x7f967c1522d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:13.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.530+0000 7f967a7fc640 1 --2- 172.21.15.57:0/3052729369 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f967c15b260 0x7f967c15d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:13.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.530+0000 7f967affd640 1 -- 172.21.15.57:0/3052729369 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f967c15b260 msgr2=0x7f967c15d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:13.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.530+0000 7f967affd640 1 --2- 172.21.15.57:0/3052729369 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f967c15b260 0x7f967c15d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:13.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.530+0000 7f967affd640 1 -- 172.21.15.57:0/3052729369 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f967c15b640 msgr2=0x7f967c1522d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:13.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.530+0000 7f967affd640 1 --2- 172.21.15.57:0/3052729369 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f967c15b640 0x7f967c1522d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:13.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.530+0000 7f967affd640 1 -- 172.21.15.57:0/3052729369 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f966c009560 con 0x7f967c0cf850 2023-10-19T12:45:13.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.530+0000 7f967b7fe640 1 --2- 172.21.15.57:0/3052729369 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f967c15b640 0x7f967c1522d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:45:13.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.531+0000 7f967affd640 1 --2- 172.21.15.57:0/3052729369 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f967c0cf850 0x7f967c0c97d0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f966c006010 tx=0x7f966c02dbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:13.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.531+0000 7f9663fff640 1 -- 172.21.15.57:0/3052729369 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f966c0035f0 con 0x7f967c0cf850 2023-10-19T12:45:13.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.531+0000 7f9663fff640 1 -- 172.21.15.57:0/3052729369 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f966c003840 con 0x7f967c0cf850 2023-10-19T12:45:13.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.531+0000 7f9681856640 1 -- 172.21.15.57:0/3052729369 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f967c0c8fd0 con 0x7f967c0cf850 2023-10-19T12:45:13.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.531+0000 7f9663fff640 1 -- 172.21.15.57:0/3052729369 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f966c003c50 con 0x7f967c0cf850 2023-10-19T12:45:13.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.531+0000 7f9681856640 1 -- 172.21.15.57:0/3052729369 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f967c0c9d10 con 0x7f967c0cf850 2023-10-19T12:45:13.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.532+0000 7f9681856640 1 -- 172.21.15.57:0/3052729369 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f967c0675f0 con 0x7f967c0cf850 2023-10-19T12:45:13.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.532+0000 7f9663fff640 1 -- 172.21.15.57:0/3052729369 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f966c03e450 con 0x7f967c0cf850 2023-10-19T12:45:13.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.533+0000 7f9663fff640 1 --2- 172.21.15.57:0/3052729369 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f966806ae40 0x7f966806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:13.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.533+0000 7f9663fff640 1 -- 172.21.15.57:0/3052729369 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f966c0ac6d0 con 0x7f967c0cf850 2023-10-19T12:45:13.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.533+0000 7f967a7fc640 1 --2- 172.21.15.57:0/3052729369 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f966806ae40 0x7f966806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:13.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.534+0000 7f967a7fc640 1 --2- 172.21.15.57:0/3052729369 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f966806ae40 0x7f966806d300 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f967c1586c0 tx=0x7f9670009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:13.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.537+0000 7f9663fff640 1 -- 172.21.15.57:0/3052729369 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f966c0774f0 con 0x7f967c0cf850 2023-10-19T12:45:13.581 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T12:45:13.583 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T12:45:13.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.727+0000 7f9681856640 1 -- 172.21.15.57:0/3052729369 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f967c0cd8c0 con 0x7f967c0cf850 2023-10-19T12:45:13.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.728+0000 7f9663fff640 1 -- 172.21.15.57:0/3052729369 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) v1 ==== 80+0+0 (secure 0 0 0) 0x7f966c003db0 con 0x7f967c0cf850 2023-10-19T12:45:13.731 INFO:teuthology.orchestra.run.smithi057.stderr:listed 0 entries 2023-10-19T12:45:13.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.730+0000 7f9681856640 1 -- 172.21.15.57:0/3052729369 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f966806ae40 msgr2=0x7f966806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:13.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.730+0000 7f9681856640 1 --2- 172.21.15.57:0/3052729369 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f966806ae40 0x7f966806d300 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f967c1586c0 tx=0x7f9670009040 comp rx=0 tx=0).stop 2023-10-19T12:45:13.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.730+0000 7f9681856640 1 -- 172.21.15.57:0/3052729369 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f967c0cf850 msgr2=0x7f967c0c97d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:13.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.730+0000 7f9681856640 1 --2- 172.21.15.57:0/3052729369 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f967c0cf850 0x7f967c0c97d0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f966c006010 tx=0x7f966c02dbb0 comp rx=0 tx=0).stop 2023-10-19T12:45:13.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.730+0000 7f9681856640 1 -- 172.21.15.57:0/3052729369 shutdown_connections 2023-10-19T12:45:13.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.730+0000 7f9681856640 1 --2- 172.21.15.57:0/3052729369 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f966806ae40 0x7f966806d300 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:13.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.730+0000 7f9681856640 1 --2- 172.21.15.57:0/3052729369 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f967c15b640 0x7f967c1522d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:13.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.730+0000 7f9681856640 1 --2- 172.21.15.57:0/3052729369 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f967c15b260 0x7f967c15d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:13.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.730+0000 7f9681856640 1 --2- 172.21.15.57:0/3052729369 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f967c0cf850 0x7f967c0c97d0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:13.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.730+0000 7f9681856640 1 -- 172.21.15.57:0/3052729369 >> 172.21.15.57:0/3052729369 conn(0x7f967c09a030 msgr2=0x7f967c0ad990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:13.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.730+0000 7f9681856640 1 -- 172.21.15.57:0/3052729369 shutdown_connections 2023-10-19T12:45:13.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:13.730+0000 7f9681856640 1 -- 172.21.15.57:0/3052729369 wait complete. 2023-10-19T12:45:13.743 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi057.front.sepia.ceph.com... 2023-10-19T12:45:13.744 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:45:13.744 DEBUG:teuthology.orchestra.run.smithi057:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-10-19T12:45:13.775 INFO:teuthology.orchestra.run:waiting for 300 2023-10-19T12:45:13.775 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi057.front.sepia.ceph.com 2023-10-19T12:45:13.775 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:45:13.776 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-19T12:45:13.830 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-19T12:45:13.831 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T12:45:13.849 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-19T12:45:13.850 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T12:45:13.906 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T12:45:14.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.012+0000 7f475d4e1640 1 -- 172.21.15.57:0/1734222320 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f475815b2a0 msgr2=0x7f47580d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:14.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.012+0000 7f475d4e1640 1 --2- 172.21.15.57:0/1734222320 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f475815b2a0 0x7f47580d1910 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f4740005fd0 tx=0x7f474002be20 comp rx=0 tx=0).stop 2023-10-19T12:45:14.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.012+0000 7f475d4e1640 1 -- 172.21.15.57:0/1734222320 shutdown_connections 2023-10-19T12:45:14.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.012+0000 7f475d4e1640 1 --2- 172.21.15.57:0/1734222320 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f475815b2a0 0x7f47580d1910 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:14.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.012+0000 7f475d4e1640 1 -- 172.21.15.57:0/1734222320 >> 172.21.15.57:0/1734222320 conn(0x7f47580d0d60 msgr2=0x7f47580d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:14.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.012+0000 7f475d4e1640 1 -- 172.21.15.57:0/1734222320 shutdown_connections 2023-10-19T12:45:14.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.012+0000 7f475d4e1640 1 -- 172.21.15.57:0/1734222320 wait complete. 2023-10-19T12:45:14.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.013+0000 7f475d4e1640 1 Processor -- start 2023-10-19T12:45:14.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.013+0000 7f475d4e1640 1 -- start start 2023-10-19T12:45:14.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.014+0000 7f475d4e1640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f47580ceb40 0x7f47580ca880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:14.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.014+0000 7f475d4e1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f47580cf340 0x7f47580b4330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:14.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.014+0000 7f475d4e1640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f475815b2a0 0x7f47581f4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:14.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.014+0000 7f475d4e1640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f475815bbe0 con 0x7f47580cf340 2023-10-19T12:45:14.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.014+0000 7f475d4e1640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47580b5300 con 0x7f47580ceb40 2023-10-19T12:45:14.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.014+0000 7f475d4e1640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47580b4d40 con 0x7f475815b2a0 2023-10-19T12:45:14.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.014+0000 7f47567fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f47580cf340 0x7f47580b4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:14.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.014+0000 7f47567fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f47580cf340 0x7f47580b4330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:60038/0 (socket says 172.21.15.57:60038) 2023-10-19T12:45:14.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.014+0000 7f47567fc640 1 -- 172.21.15.57:0/752064616 learned_addr learned my addr 172.21.15.57:0/752064616 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:14.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.014+0000 7f4756ffd640 1 --2- 172.21.15.57:0/752064616 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f47580ceb40 0x7f47580ca880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:14.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.014+0000 7f47577fe640 1 --2- 172.21.15.57:0/752064616 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f475815b2a0 0x7f47581f4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:14.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.015+0000 7f47567fc640 1 -- 172.21.15.57:0/752064616 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f475815b2a0 msgr2=0x7f47581f4ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:14.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.015+0000 7f47567fc640 1 --2- 172.21.15.57:0/752064616 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f475815b2a0 0x7f47581f4ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:14.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.015+0000 7f47567fc640 1 -- 172.21.15.57:0/752064616 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f47580ceb40 msgr2=0x7f47580ca880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:14.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.015+0000 7f47567fc640 1 --2- 172.21.15.57:0/752064616 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f47580ceb40 0x7f47580ca880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:14.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.015+0000 7f47567fc640 1 -- 172.21.15.57:0/752064616 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4740009560 con 0x7f47580cf340 2023-10-19T12:45:14.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.015+0000 7f47567fc640 1 --2- 172.21.15.57:0/752064616 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f47580cf340 0x7f47580b4330 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f47400099f0 tx=0x7f474002ddf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:14.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.015+0000 7f474f7fe640 1 -- 172.21.15.57:0/752064616 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4740005650 con 0x7f47580cf340 2023-10-19T12:45:14.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.015+0000 7f474f7fe640 1 -- 172.21.15.57:0/752064616 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f47400038f0 con 0x7f47580cf340 2023-10-19T12:45:14.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.015+0000 7f474f7fe640 1 -- 172.21.15.57:0/752064616 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4740003d00 con 0x7f47580cf340 2023-10-19T12:45:14.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.015+0000 7f475d4e1640 1 -- 172.21.15.57:0/752064616 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47580b5020 con 0x7f47580cf340 2023-10-19T12:45:14.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.015+0000 7f475d4e1640 1 -- 172.21.15.57:0/752064616 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47581586f0 con 0x7f47580cf340 2023-10-19T12:45:14.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.016+0000 7f475d4e1640 1 -- 172.21.15.57:0/752064616 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f475815bd20 con 0x7f47580cf340 2023-10-19T12:45:14.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.017+0000 7f474f7fe640 1 -- 172.21.15.57:0/752064616 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f474003e450 con 0x7f47580cf340 2023-10-19T12:45:14.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.017+0000 7f474f7fe640 1 --2- 172.21.15.57:0/752064616 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f473806ae40 0x7f473806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:14.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.017+0000 7f474f7fe640 1 -- 172.21.15.57:0/752064616 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f47400ac2f0 con 0x7f47580cf340 2023-10-19T12:45:14.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.017+0000 7f4756ffd640 1 --2- 172.21.15.57:0/752064616 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f473806ae40 0x7f473806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:14.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.018+0000 7f4756ffd640 1 --2- 172.21.15.57:0/752064616 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f473806ae40 0x7f473806d300 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f4750001780 tx=0x7f4750009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:14.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.021+0000 7f474f7fe640 1 -- 172.21.15.57:0/752064616 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f475815bd20 con 0x7f47580cf340 2023-10-19T12:45:14.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.202+0000 7f475d4e1640 1 -- 172.21.15.57:0/752064616 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f475815bd20 con 0x7f47580cf340 2023-10-19T12:45:14.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.203+0000 7f474f7fe640 1 -- 172.21.15.57:0/752064616 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18748 (secure 0 0 0) 0x7f475815bd20 con 0x7f47580cf340 2023-10-19T12:45:14.207 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:45:14.207 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":27,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T12:44:58.978403+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-19T12:44:16.517510+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-19T12:44:17.398297+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-19T12:44:21.129403+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.23","mappings":[{"from":3,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"3.39","mappings":[{"from":7,"to":1}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.1d","mappings":[{"from":1,"to":5}]},{"pgid":"4.1e","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T12:45:14.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.205+0000 7f475d4e1640 1 -- 172.21.15.57:0/752064616 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f473806ae40 msgr2=0x7f473806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:14.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.205+0000 7f475d4e1640 1 --2- 172.21.15.57:0/752064616 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f473806ae40 0x7f473806d300 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f4750001780 tx=0x7f4750009040 comp rx=0 tx=0).stop 2023-10-19T12:45:14.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.205+0000 7f475d4e1640 1 -- 172.21.15.57:0/752064616 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f47580cf340 msgr2=0x7f47580b4330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:14.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.205+0000 7f475d4e1640 1 --2- 172.21.15.57:0/752064616 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f47580cf340 0x7f47580b4330 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f47400099f0 tx=0x7f474002ddf0 comp rx=0 tx=0).stop 2023-10-19T12:45:14.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.205+0000 7f475d4e1640 1 -- 172.21.15.57:0/752064616 shutdown_connections 2023-10-19T12:45:14.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.205+0000 7f475d4e1640 1 --2- 172.21.15.57:0/752064616 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f473806ae40 0x7f473806d300 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:14.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.205+0000 7f475d4e1640 1 --2- 172.21.15.57:0/752064616 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f475815b2a0 0x7f47581f4ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:14.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.205+0000 7f475d4e1640 1 --2- 172.21.15.57:0/752064616 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f47580cf340 0x7f47580b4330 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:14.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.205+0000 7f475d4e1640 1 --2- 172.21.15.57:0/752064616 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f47580ceb40 0x7f47580ca880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:14.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.205+0000 7f475d4e1640 1 -- 172.21.15.57:0/752064616 >> 172.21.15.57:0/752064616 conn(0x7f47580d0d60 msgr2=0x7f475815a210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:14.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.205+0000 7f475d4e1640 1 -- 172.21.15.57:0/752064616 shutdown_connections 2023-10-19T12:45:14.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.205+0000 7f475d4e1640 1 -- 172.21.15.57:0/752064616 wait complete. 2023-10-19T12:45:14.219 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-19T12:44:07.371186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-19T12:44:16.517510+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-19T12:44:17.398297+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-19T12:44:21.129403+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-19T12:45:14.220 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-19T12:45:14.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.328+0000 7fab7f626640 1 -- 172.21.15.57:0/2227157776 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab78141690 msgr2=0x7fab780703e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:14.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.328+0000 7fab7f626640 1 --2- 172.21.15.57:0/2227157776 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab78141690 0x7fab780703e0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fab68005fd0 tx=0x7fab6802bd90 comp rx=0 tx=0).stop 2023-10-19T12:45:14.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.328+0000 7fab7f626640 1 -- 172.21.15.57:0/2227157776 shutdown_connections 2023-10-19T12:45:14.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.328+0000 7fab7f626640 1 --2- 172.21.15.57:0/2227157776 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fab78070920 0x7fab780affe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:14.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.328+0000 7fab7f626640 1 --2- 172.21.15.57:0/2227157776 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab78141690 0x7fab780703e0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:14.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.328+0000 7fab7f626640 1 -- 172.21.15.57:0/2227157776 >> 172.21.15.57:0/2227157776 conn(0x7fab78069930 msgr2=0x7fab78069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:14.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.328+0000 7fab7f626640 1 -- 172.21.15.57:0/2227157776 shutdown_connections 2023-10-19T12:45:14.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.328+0000 7fab7f626640 1 -- 172.21.15.57:0/2227157776 wait complete. 2023-10-19T12:45:14.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.330+0000 7fab7f626640 1 Processor -- start 2023-10-19T12:45:14.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.330+0000 7fab7f626640 1 -- start start 2023-10-19T12:45:14.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.330+0000 7fab7f626640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fab78070920 0x7fab781e3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:14.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.330+0000 7fab7f626640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab7813e520 0x7fab780ca870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:14.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.330+0000 7fab7f626640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fab78141690 0x7fab780cadb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:14.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.330+0000 7fab7f626640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab781cf700 con 0x7fab7813e520 2023-10-19T12:45:14.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.330+0000 7fab7f626640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab781e44b0 con 0x7fab78141690 2023-10-19T12:45:14.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.330+0000 7fab7f626640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab780aa5d0 con 0x7fab78070920 2023-10-19T12:45:14.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.330+0000 7fab7cbc1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab7813e520 0x7fab780ca870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:14.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.330+0000 7fab7cbc1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab7813e520 0x7fab780ca870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:60062/0 (socket says 172.21.15.57:60062) 2023-10-19T12:45:14.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.330+0000 7fab7cbc1640 1 -- 172.21.15.57:0/3736860318 learned_addr learned my addr 172.21.15.57:0/3736860318 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:14.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.331+0000 7fab7dbc3640 1 --2- 172.21.15.57:0/3736860318 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fab78141690 0x7fab780cadb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:14.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.331+0000 7fab7d3c2640 1 --2- 172.21.15.57:0/3736860318 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fab78070920 0x7fab781e3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:14.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.331+0000 7fab7cbc1640 1 -- 172.21.15.57:0/3736860318 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fab78070920 msgr2=0x7fab781e3f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:14.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.331+0000 7fab7cbc1640 1 --2- 172.21.15.57:0/3736860318 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fab78070920 0x7fab781e3f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:14.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.331+0000 7fab7cbc1640 1 -- 172.21.15.57:0/3736860318 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fab78141690 msgr2=0x7fab780cadb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:14.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.331+0000 7fab7cbc1640 1 --2- 172.21.15.57:0/3736860318 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fab78141690 0x7fab780cadb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:14.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.331+0000 7fab7cbc1640 1 -- 172.21.15.57:0/3736860318 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab68009560 con 0x7fab7813e520 2023-10-19T12:45:14.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.331+0000 7fab7dbc3640 1 --2- 172.21.15.57:0/3736860318 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fab78141690 0x7fab780cadb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:45:14.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.331+0000 7fab7cbc1640 1 --2- 172.21.15.57:0/3736860318 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab7813e520 0x7fab780ca870 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fab68006010 tx=0x7fab6802d9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:14.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.331+0000 7fab6dffb640 1 -- 172.21.15.57:0/3736860318 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab6802ddf0 con 0x7fab7813e520 2023-10-19T12:45:14.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.331+0000 7fab6dffb640 1 -- 172.21.15.57:0/3736860318 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fab680036e0 con 0x7fab7813e520 2023-10-19T12:45:14.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.331+0000 7fab7f626640 1 -- 172.21.15.57:0/3736860318 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab781ced60 con 0x7fab7813e520 2023-10-19T12:45:14.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.331+0000 7fab6dffb640 1 -- 172.21.15.57:0/3736860318 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab68003af0 con 0x7fab7813e520 2023-10-19T12:45:14.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.331+0000 7fab7f626640 1 -- 172.21.15.57:0/3736860318 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab781cf320 con 0x7fab7813e520 2023-10-19T12:45:14.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.332+0000 7fab7f626640 1 -- 172.21.15.57:0/3736860318 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab781cf840 con 0x7fab7813e520 2023-10-19T12:45:14.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.333+0000 7fab6dffb640 1 -- 172.21.15.57:0/3736860318 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fab68003c50 con 0x7fab7813e520 2023-10-19T12:45:14.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.333+0000 7fab6dffb640 1 --2- 172.21.15.57:0/3736860318 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fab5c06ae40 0x7fab5c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:14.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.333+0000 7fab6dffb640 1 -- 172.21.15.57:0/3736860318 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fab680ac3b0 con 0x7fab7813e520 2023-10-19T12:45:14.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.333+0000 7fab7d3c2640 1 --2- 172.21.15.57:0/3736860318 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fab5c06ae40 0x7fab5c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:14.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.334+0000 7fab7d3c2640 1 --2- 172.21.15.57:0/3736860318 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fab5c06ae40 0x7fab5c06d300 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fab74001780 tx=0x7fab74009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:14.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.337+0000 7fab6dffb640 1 -- 172.21.15.57:0/3736860318 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fab68077130 con 0x7fab7813e520 2023-10-19T12:45:14.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.512+0000 7fab7f626640 1 -- 172.21.15.57:0/3736860318 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fab780b1bd0 con 0x7fab7813e520 2023-10-19T12:45:14.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.512+0000 7fab6dffb640 1 -- 172.21.15.57:0/3736860318 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7fab6807a750 con 0x7fab7813e520 2023-10-19T12:45:14.516 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-19T12:45:14.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.514+0000 7fab7f626640 1 -- 172.21.15.57:0/3736860318 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fab5c06ae40 msgr2=0x7fab5c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:14.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.514+0000 7fab7f626640 1 --2- 172.21.15.57:0/3736860318 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fab5c06ae40 0x7fab5c06d300 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fab74001780 tx=0x7fab74009040 comp rx=0 tx=0).stop 2023-10-19T12:45:14.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.514+0000 7fab7f626640 1 -- 172.21.15.57:0/3736860318 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab7813e520 msgr2=0x7fab780ca870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:14.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.514+0000 7fab7f626640 1 --2- 172.21.15.57:0/3736860318 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab7813e520 0x7fab780ca870 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fab68006010 tx=0x7fab6802d9f0 comp rx=0 tx=0).stop 2023-10-19T12:45:14.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.514+0000 7fab7f626640 1 -- 172.21.15.57:0/3736860318 shutdown_connections 2023-10-19T12:45:14.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.514+0000 7fab7f626640 1 --2- 172.21.15.57:0/3736860318 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fab5c06ae40 0x7fab5c06d300 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:14.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.514+0000 7fab7f626640 1 --2- 172.21.15.57:0/3736860318 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fab78141690 0x7fab780cadb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:14.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.514+0000 7fab7f626640 1 --2- 172.21.15.57:0/3736860318 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab7813e520 0x7fab780ca870 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:14.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.514+0000 7fab7f626640 1 --2- 172.21.15.57:0/3736860318 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fab78070920 0x7fab781e3f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:14.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.514+0000 7fab7f626640 1 -- 172.21.15.57:0/3736860318 >> 172.21.15.57:0/3736860318 conn(0x7fab78069930 msgr2=0x7fab78069d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:14.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.514+0000 7fab7f626640 1 -- 172.21.15.57:0/3736860318 shutdown_connections 2023-10-19T12:45:14.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.514+0000 7fab7f626640 1 -- 172.21.15.57:0/3736860318 wait complete. 2023-10-19T12:45:14.527 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-19T12:45:14.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.635+0000 7f922dac9640 1 -- 172.21.15.57:0/576047041 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9228144a80 msgr2=0x7f9228070520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:14.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.635+0000 7f922dac9640 1 --2- 172.21.15.57:0/576047041 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9228144a80 0x7f9228070520 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f9220005fd0 tx=0x7f922002c2e0 comp rx=0 tx=0).stop 2023-10-19T12:45:14.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.635+0000 7f922dac9640 1 -- 172.21.15.57:0/576047041 shutdown_connections 2023-10-19T12:45:14.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.635+0000 7f922dac9640 1 --2- 172.21.15.57:0/576047041 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f922815c090 0x7f922806d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:14.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.635+0000 7f922dac9640 1 --2- 172.21.15.57:0/576047041 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9228144a80 0x7f9228070520 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:14.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.635+0000 7f922dac9640 1 -- 172.21.15.57:0/576047041 >> 172.21.15.57:0/576047041 conn(0x7f92280aec40 msgr2=0x7f92280af050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:14.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.636+0000 7f922dac9640 1 -- 172.21.15.57:0/576047041 shutdown_connections 2023-10-19T12:45:14.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.636+0000 7f922dac9640 1 -- 172.21.15.57:0/576047041 wait complete. 2023-10-19T12:45:14.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.637+0000 7f922dac9640 1 Processor -- start 2023-10-19T12:45:14.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.637+0000 7f922dac9640 1 -- start start 2023-10-19T12:45:14.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.637+0000 7f922dac9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f92281446a0 0x7f9228133910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:14.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.637+0000 7f922dac9640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9228144a80 0x7f92281317b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:14.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.638+0000 7f922dac9640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f922815c090 0x7f9228136d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:14.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.638+0000 7f922dac9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9228131cf0 con 0x7f92281446a0 2023-10-19T12:45:14.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.638+0000 7f922dac9640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9228132b10 con 0x7f922815c090 2023-10-19T12:45:14.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.638+0000 7f922dac9640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9228131f00 con 0x7f9228144a80 2023-10-19T12:45:14.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.638+0000 7f92277fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f92281446a0 0x7f9228133910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:14.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.638+0000 7f92277fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f92281446a0 0x7f9228133910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:60092/0 (socket says 172.21.15.57:60092) 2023-10-19T12:45:14.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.638+0000 7f92277fe640 1 -- 172.21.15.57:0/1605210859 learned_addr learned my addr 172.21.15.57:0/1605210859 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:14.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.638+0000 7f9226ffd640 1 --2- 172.21.15.57:0/1605210859 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9228144a80 0x7f92281317b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:14.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.638+0000 7f9227fff640 1 --2- 172.21.15.57:0/1605210859 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f922815c090 0x7f9228136d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:14.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.638+0000 7f92277fe640 1 -- 172.21.15.57:0/1605210859 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9228144a80 msgr2=0x7f92281317b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:14.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.638+0000 7f92277fe640 1 --2- 172.21.15.57:0/1605210859 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9228144a80 0x7f92281317b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:14.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.638+0000 7f92277fe640 1 -- 172.21.15.57:0/1605210859 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f922815c090 msgr2=0x7f9228136d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:14.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.638+0000 7f92277fe640 1 --2- 172.21.15.57:0/1605210859 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f922815c090 0x7f9228136d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:14.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.638+0000 7f92277fe640 1 -- 172.21.15.57:0/1605210859 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9220009900 con 0x7f92281446a0 2023-10-19T12:45:14.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.638+0000 7f9226ffd640 1 --2- 172.21.15.57:0/1605210859 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9228144a80 0x7f92281317b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T12:45:14.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.638+0000 7f92277fe640 1 --2- 172.21.15.57:0/1605210859 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f92281446a0 0x7f9228133910 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f921c00c480 tx=0x7f921c00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:14.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.638+0000 7f920ffff640 1 -- 172.21.15.57:0/1605210859 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f921c0052d0 con 0x7f92281446a0 2023-10-19T12:45:14.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.638+0000 7f920ffff640 1 -- 172.21.15.57:0/1605210859 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f921c005430 con 0x7f92281446a0 2023-10-19T12:45:14.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.638+0000 7f922dac9640 1 -- 172.21.15.57:0/1605210859 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92281321e0 con 0x7f92281446a0 2023-10-19T12:45:14.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.638+0000 7f920ffff640 1 -- 172.21.15.57:0/1605210859 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f921c012890 con 0x7f92281446a0 2023-10-19T12:45:14.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.638+0000 7f922dac9640 1 -- 172.21.15.57:0/1605210859 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92281327a0 con 0x7f92281446a0 2023-10-19T12:45:14.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.639+0000 7f922dac9640 1 -- 172.21.15.57:0/1605210859 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92280690e0 con 0x7f92281446a0 2023-10-19T12:45:14.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.640+0000 7f920ffff640 1 -- 172.21.15.57:0/1605210859 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f921c0035f0 con 0x7f92281446a0 2023-10-19T12:45:14.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.640+0000 7f920ffff640 1 --2- 172.21.15.57:0/1605210859 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f921406ae40 0x7f921406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:14.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.640+0000 7f920ffff640 1 -- 172.21.15.57:0/1605210859 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f921c089540 con 0x7f92281446a0 2023-10-19T12:45:14.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.641+0000 7f9226ffd640 1 --2- 172.21.15.57:0/1605210859 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f921406ae40 0x7f921406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:14.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.641+0000 7f9226ffd640 1 --2- 172.21.15.57:0/1605210859 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f921406ae40 0x7f921406d300 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f92280b1300 tx=0x7f9218005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:14.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.644+0000 7f920ffff640 1 -- 172.21.15.57:0/1605210859 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f921c054370 con 0x7f92281446a0 2023-10-19T12:45:14.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.821+0000 7f922dac9640 1 -- 172.21.15.57:0/1605210859 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9228132c50 con 0x7f92281446a0 2023-10-19T12:45:14.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.821+0000 7f920ffff640 1 -- 172.21.15.57:0/1605210859 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7f921c057990 con 0x7f92281446a0 2023-10-19T12:45:14.825 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2023-10-19T12:45:14.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.823+0000 7f922dac9640 1 -- 172.21.15.57:0/1605210859 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f921406ae40 msgr2=0x7f921406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:14.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.823+0000 7f922dac9640 1 --2- 172.21.15.57:0/1605210859 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f921406ae40 0x7f921406d300 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f92280b1300 tx=0x7f9218005f90 comp rx=0 tx=0).stop 2023-10-19T12:45:14.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.823+0000 7f922dac9640 1 -- 172.21.15.57:0/1605210859 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f92281446a0 msgr2=0x7f9228133910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:14.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.823+0000 7f922dac9640 1 --2- 172.21.15.57:0/1605210859 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f92281446a0 0x7f9228133910 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f921c00c480 tx=0x7f921c00c940 comp rx=0 tx=0).stop 2023-10-19T12:45:14.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.823+0000 7f922dac9640 1 -- 172.21.15.57:0/1605210859 shutdown_connections 2023-10-19T12:45:14.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.823+0000 7f922dac9640 1 --2- 172.21.15.57:0/1605210859 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f921406ae40 0x7f921406d300 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:14.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.823+0000 7f922dac9640 1 --2- 172.21.15.57:0/1605210859 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f922815c090 0x7f9228136d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:14.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.823+0000 7f922dac9640 1 --2- 172.21.15.57:0/1605210859 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9228144a80 0x7f92281317b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:14.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.823+0000 7f922dac9640 1 --2- 172.21.15.57:0/1605210859 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f92281446a0 0x7f9228133910 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:14.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.823+0000 7f922dac9640 1 -- 172.21.15.57:0/1605210859 >> 172.21.15.57:0/1605210859 conn(0x7f92280aec40 msgr2=0x7f92280aa8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:14.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.823+0000 7f922dac9640 1 -- 172.21.15.57:0/1605210859 shutdown_connections 2023-10-19T12:45:14.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.823+0000 7f922dac9640 1 -- 172.21.15.57:0/1605210859 wait complete. 2023-10-19T12:45:14.837 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-19T12:45:14.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.946+0000 7fce57487640 1 -- 172.21.15.57:0/1827087643 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fce50157220 msgr2=0x7fce500ca870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:14.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.946+0000 7fce57487640 1 --2- 172.21.15.57:0/1827087643 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fce50157220 0x7fce500ca870 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fce4c009830 tx=0x7fce4c02bcd0 comp rx=0 tx=0).stop 2023-10-19T12:45:14.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.946+0000 7fce57487640 1 -- 172.21.15.57:0/1827087643 shutdown_connections 2023-10-19T12:45:14.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.946+0000 7fce57487640 1 --2- 172.21.15.57:0/1827087643 >> v2:172.21.15.179:3300/0 conn(0x7fce500cadb0 0x7fce5006bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:14.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.946+0000 7fce57487640 1 --2- 172.21.15.57:0/1827087643 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fce50157220 0x7fce500ca870 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:14.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.946+0000 7fce57487640 1 -- 172.21.15.57:0/1827087643 >> 172.21.15.57:0/1827087643 conn(0x7fce50069930 msgr2=0x7fce50069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:14.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.946+0000 7fce57487640 1 -- 172.21.15.57:0/1827087643 shutdown_connections 2023-10-19T12:45:14.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.946+0000 7fce57487640 1 -- 172.21.15.57:0/1827087643 wait complete. 2023-10-19T12:45:14.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.947+0000 7fce57487640 1 Processor -- start 2023-10-19T12:45:14.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.947+0000 7fce57487640 1 -- start start 2023-10-19T12:45:14.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.948+0000 7fce57487640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fce5006cf30 0x7fce50141e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:14.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.948+0000 7fce57487640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fce500cadb0 0x7fce500ae7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:14.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.948+0000 7fce57487640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fce50157220 0x7fce500aed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:14.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.948+0000 7fce57487640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce5012dc50 con 0x7fce5006cf30 2023-10-19T12:45:14.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.948+0000 7fce57487640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce50143a40 con 0x7fce50157220 2023-10-19T12:45:14.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.948+0000 7fce57487640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce50143ce0 con 0x7fce500cadb0 2023-10-19T12:45:14.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.948+0000 7fce55223640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fce5006cf30 0x7fce50141e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:14.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.948+0000 7fce55223640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fce5006cf30 0x7fce50141e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:60108/0 (socket says 172.21.15.57:60108) 2023-10-19T12:45:14.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.948+0000 7fce55223640 1 -- 172.21.15.57:0/1265091930 learned_addr learned my addr 172.21.15.57:0/1265091930 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:14.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.948+0000 7fce55a24640 1 --2- 172.21.15.57:0/1265091930 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fce50157220 0x7fce500aed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:14.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.948+0000 7fce54a22640 1 --2- 172.21.15.57:0/1265091930 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fce500cadb0 0x7fce500ae7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:14.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.948+0000 7fce55223640 1 -- 172.21.15.57:0/1265091930 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fce500cadb0 msgr2=0x7fce500ae7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:14.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.948+0000 7fce55223640 1 --2- 172.21.15.57:0/1265091930 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fce500cadb0 0x7fce500ae7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:14.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.948+0000 7fce55223640 1 -- 172.21.15.57:0/1265091930 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fce50157220 msgr2=0x7fce500aed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:14.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.948+0000 7fce55223640 1 --2- 172.21.15.57:0/1265091930 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fce50157220 0x7fce500aed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:14.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.948+0000 7fce55223640 1 -- 172.21.15.57:0/1265091930 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce40009490 con 0x7fce5006cf30 2023-10-19T12:45:14.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.948+0000 7fce55a24640 1 --2- 172.21.15.57:0/1265091930 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fce50157220 0x7fce500aed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:45:14.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.949+0000 7fce55223640 1 --2- 172.21.15.57:0/1265091930 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fce5006cf30 0x7fce50141e10 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fce4c000c00 tx=0x7fce4c003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:14.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.949+0000 7fce45ffb640 1 -- 172.21.15.57:0/1265091930 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce4c003e20 con 0x7fce5006cf30 2023-10-19T12:45:14.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.949+0000 7fce45ffb640 1 -- 172.21.15.57:0/1265091930 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fce4c034440 con 0x7fce5006cf30 2023-10-19T12:45:14.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.949+0000 7fce57487640 1 -- 172.21.15.57:0/1265091930 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce4c009490 con 0x7fce5006cf30 2023-10-19T12:45:14.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.949+0000 7fce45ffb640 1 -- 172.21.15.57:0/1265091930 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce4c034850 con 0x7fce5006cf30 2023-10-19T12:45:14.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.949+0000 7fce57487640 1 -- 172.21.15.57:0/1265091930 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce50144580 con 0x7fce5006cf30 2023-10-19T12:45:14.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.950+0000 7fce57487640 1 -- 172.21.15.57:0/1265091930 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce50143ea0 con 0x7fce5006cf30 2023-10-19T12:45:14.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.950+0000 7fce45ffb640 1 -- 172.21.15.57:0/1265091930 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fce4c0349b0 con 0x7fce5006cf30 2023-10-19T12:45:14.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.951+0000 7fce45ffb640 1 --2- 172.21.15.57:0/1265091930 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fce3406ae70 0x7fce3406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:14.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.951+0000 7fce45ffb640 1 -- 172.21.15.57:0/1265091930 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fce4c0ac110 con 0x7fce5006cf30 2023-10-19T12:45:14.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.951+0000 7fce54a22640 1 --2- 172.21.15.57:0/1265091930 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fce3406ae70 0x7fce3406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:14.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.952+0000 7fce54a22640 1 --2- 172.21.15.57:0/1265091930 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fce3406ae70 0x7fce3406d330 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fce50156c30 tx=0x7fce40005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:14.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:14.955+0000 7fce45ffb640 1 -- 172.21.15.57:0/1265091930 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fce4c076e00 con 0x7fce5006cf30 2023-10-19T12:45:15.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.133+0000 7fce57487640 1 -- 172.21.15.57:0/1265091930 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fce500690e0 con 0x7fce5006cf30 2023-10-19T12:45:15.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.133+0000 7fce45ffb640 1 -- 172.21.15.57:0/1265091930 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7fce4c07a420 con 0x7fce5006cf30 2023-10-19T12:45:15.137 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 64 2023-10-19T12:45:15.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.135+0000 7fce57487640 1 -- 172.21.15.57:0/1265091930 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fce3406ae70 msgr2=0x7fce3406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:15.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.135+0000 7fce57487640 1 --2- 172.21.15.57:0/1265091930 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fce3406ae70 0x7fce3406d330 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fce50156c30 tx=0x7fce40005f90 comp rx=0 tx=0).stop 2023-10-19T12:45:15.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.135+0000 7fce57487640 1 -- 172.21.15.57:0/1265091930 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fce5006cf30 msgr2=0x7fce50141e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:15.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.135+0000 7fce57487640 1 --2- 172.21.15.57:0/1265091930 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fce5006cf30 0x7fce50141e10 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fce4c000c00 tx=0x7fce4c003820 comp rx=0 tx=0).stop 2023-10-19T12:45:15.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.135+0000 7fce57487640 1 -- 172.21.15.57:0/1265091930 shutdown_connections 2023-10-19T12:45:15.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.135+0000 7fce57487640 1 --2- 172.21.15.57:0/1265091930 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fce3406ae70 0x7fce3406d330 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:15.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.135+0000 7fce57487640 1 --2- 172.21.15.57:0/1265091930 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fce50157220 0x7fce500aed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:15.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.135+0000 7fce57487640 1 --2- 172.21.15.57:0/1265091930 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fce500cadb0 0x7fce500ae7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:15.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.135+0000 7fce57487640 1 --2- 172.21.15.57:0/1265091930 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fce5006cf30 0x7fce50141e10 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:15.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.135+0000 7fce57487640 1 -- 172.21.15.57:0/1265091930 >> 172.21.15.57:0/1265091930 conn(0x7fce50069930 msgr2=0x7fce50159a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:15.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.135+0000 7fce57487640 1 -- 172.21.15.57:0/1265091930 shutdown_connections 2023-10-19T12:45:15.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.136+0000 7fce57487640 1 -- 172.21.15.57:0/1265091930 wait complete. 2023-10-19T12:45:15.149 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-19T12:45:15.263 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.259+0000 7f49a78ec640 1 -- 172.21.15.57:0/1507692187 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f498c004370 msgr2=0x7f498c004840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:15.263 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.259+0000 7f49a78ec640 1 --2- 172.21.15.57:0/1507692187 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f498c004370 0x7f498c004840 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f4990009d20 tx=0x7f498c0057f0 comp rx=0 tx=0).stop 2023-10-19T12:45:15.263 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.259+0000 7f49a78ec640 1 -- 172.21.15.57:0/1507692187 shutdown_connections 2023-10-19T12:45:15.263 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.259+0000 7f49a78ec640 1 --2- 172.21.15.57:0/1507692187 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f498c004370 0x7f498c004840 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:15.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.259+0000 7f49a78ec640 1 -- 172.21.15.57:0/1507692187 >> 172.21.15.57:0/1507692187 conn(0x7f49a006fae0 msgr2=0x7f49a006fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:15.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.259+0000 7f49a78ec640 1 -- 172.21.15.57:0/1507692187 shutdown_connections 2023-10-19T12:45:15.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.259+0000 7f49a78ec640 1 -- 172.21.15.57:0/1507692187 wait complete. 2023-10-19T12:45:15.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.260+0000 7f49a78ec640 1 Processor -- start 2023-10-19T12:45:15.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.260+0000 7f49a78ec640 1 -- start start 2023-10-19T12:45:15.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.261+0000 7f49a78ec640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f498c004370 0x7f49a0138be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:15.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.261+0000 7f49a78ec640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49a014f780 0x7f49a0149c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:15.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.261+0000 7f49a78ec640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f49a014a1d0 0x7f49a014e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:15.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.261+0000 7f49a78ec640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49a014a680 con 0x7f49a014f780 2023-10-19T12:45:15.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.261+0000 7f49a78ec640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49a014d5c0 con 0x7f498c004370 2023-10-19T12:45:15.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.261+0000 7f49a78ec640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49a01e2ed0 con 0x7f49a014a1d0 2023-10-19T12:45:15.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.261+0000 7f49a4e87640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49a014f780 0x7f49a0149c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:15.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.261+0000 7f49a4e87640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49a014f780 0x7f49a0149c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:60118/0 (socket says 172.21.15.57:60118) 2023-10-19T12:45:15.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.261+0000 7f49a4e87640 1 -- 172.21.15.57:0/1994199258 learned_addr learned my addr 172.21.15.57:0/1994199258 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:15.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.261+0000 7f49a5688640 1 --2- 172.21.15.57:0/1994199258 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f498c004370 0x7f49a0138be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:15.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.261+0000 7f49a5e89640 1 --2- 172.21.15.57:0/1994199258 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f49a014a1d0 0x7f49a014e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:15.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.261+0000 7f49a4e87640 1 -- 172.21.15.57:0/1994199258 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f49a014a1d0 msgr2=0x7f49a014e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:15.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.261+0000 7f49a4e87640 1 --2- 172.21.15.57:0/1994199258 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f49a014a1d0 0x7f49a014e0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:15.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.261+0000 7f49a4e87640 1 -- 172.21.15.57:0/1994199258 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f498c004370 msgr2=0x7f49a0138be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:15.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.261+0000 7f49a4e87640 1 --2- 172.21.15.57:0/1994199258 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f498c004370 0x7f49a0138be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:15.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.262+0000 7f49a4e87640 1 -- 172.21.15.57:0/1994199258 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49a01e3120 con 0x7f49a014f780 2023-10-19T12:45:15.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.262+0000 7f49a5e89640 1 --2- 172.21.15.57:0/1994199258 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f49a014a1d0 0x7f49a014e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:45:15.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.262+0000 7f49a4e87640 1 --2- 172.21.15.57:0/1994199258 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49a014f780 0x7f49a0149c90 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f4990036040 tx=0x7f49900043f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:15.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.262+0000 7f4995ffb640 1 -- 172.21.15.57:0/1994199258 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4990004a90 con 0x7f49a014f780 2023-10-19T12:45:15.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.262+0000 7f4995ffb640 1 -- 172.21.15.57:0/1994199258 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f4990004ce0 con 0x7f49a014f780 2023-10-19T12:45:15.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.262+0000 7f49a78ec640 1 -- 172.21.15.57:0/1994199258 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49a01e3400 con 0x7f49a014f780 2023-10-19T12:45:15.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.262+0000 7f4995ffb640 1 -- 172.21.15.57:0/1994199258 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4990003850 con 0x7f49a014f780 2023-10-19T12:45:15.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.262+0000 7f49a78ec640 1 -- 172.21.15.57:0/1994199258 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49a01e39c0 con 0x7f49a014f780 2023-10-19T12:45:15.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.263+0000 7f49a78ec640 1 -- 172.21.15.57:0/1994199258 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49a014a5e0 con 0x7f49a014f780 2023-10-19T12:45:15.272 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.264+0000 7f4995ffb640 1 -- 172.21.15.57:0/1994199258 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f49900039b0 con 0x7f49a014f780 2023-10-19T12:45:15.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.264+0000 7f4995ffb640 1 --2- 172.21.15.57:0/1994199258 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f498406ae70 0x7f498406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:15.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.264+0000 7f4995ffb640 1 -- 172.21.15.57:0/1994199258 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f49900ac630 con 0x7f49a014f780 2023-10-19T12:45:15.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.264+0000 7f49a5688640 1 --2- 172.21.15.57:0/1994199258 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f498406ae70 0x7f498406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:15.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.265+0000 7f49a5688640 1 --2- 172.21.15.57:0/1994199258 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f498406ae70 0x7f498406d330 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f49a00682a0 tx=0x7f499c006170 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:15.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.268+0000 7f4995ffb640 1 -- 172.21.15.57:0/1994199258 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f49900773b0 con 0x7f49a014f780 2023-10-19T12:45:15.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.443+0000 7f49a78ec640 1 -- 172.21.15.57:0/1994199258 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f49a00690e0 con 0x7f49a014f780 2023-10-19T12:45:15.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.443+0000 7f4995ffb640 1 -- 172.21.15.57:0/1994199258 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7f4990042020 con 0x7f49a014f780 2023-10-19T12:45:15.447 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2023-10-19T12:45:15.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.445+0000 7f49a78ec640 1 -- 172.21.15.57:0/1994199258 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f498406ae70 msgr2=0x7f498406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:15.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.445+0000 7f49a78ec640 1 --2- 172.21.15.57:0/1994199258 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f498406ae70 0x7f498406d330 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f49a00682a0 tx=0x7f499c006170 comp rx=0 tx=0).stop 2023-10-19T12:45:15.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.445+0000 7f49a78ec640 1 -- 172.21.15.57:0/1994199258 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49a014f780 msgr2=0x7f49a0149c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:15.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.445+0000 7f49a78ec640 1 --2- 172.21.15.57:0/1994199258 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49a014f780 0x7f49a0149c90 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f4990036040 tx=0x7f49900043f0 comp rx=0 tx=0).stop 2023-10-19T12:45:15.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.445+0000 7f49a78ec640 1 -- 172.21.15.57:0/1994199258 shutdown_connections 2023-10-19T12:45:15.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.445+0000 7f49a78ec640 1 --2- 172.21.15.57:0/1994199258 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f498406ae70 0x7f498406d330 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:15.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.445+0000 7f49a78ec640 1 --2- 172.21.15.57:0/1994199258 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f49a014a1d0 0x7f49a014e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:15.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.445+0000 7f49a78ec640 1 --2- 172.21.15.57:0/1994199258 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49a014f780 0x7f49a0149c90 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:15.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.445+0000 7f49a78ec640 1 --2- 172.21.15.57:0/1994199258 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f498c004370 0x7f49a0138be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:15.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.445+0000 7f49a78ec640 1 -- 172.21.15.57:0/1994199258 >> 172.21.15.57:0/1994199258 conn(0x7f49a006fae0 msgr2=0x7f49a0155150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:15.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.445+0000 7f49a78ec640 1 -- 172.21.15.57:0/1994199258 shutdown_connections 2023-10-19T12:45:15.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.445+0000 7f49a78ec640 1 -- 172.21.15.57:0/1994199258 wait complete. 2023-10-19T12:45:15.459 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-19T12:45:15.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.566+0000 7f2771239640 1 -- 172.21.15.57:0/1610030764 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f276c0cbd20 msgr2=0x7f276c158300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:15.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.566+0000 7f2771239640 1 --2- 172.21.15.57:0/1610030764 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f276c0cbd20 0x7f276c158300 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f2764005fd0 tx=0x7f276402be10 comp rx=0 tx=0).stop 2023-10-19T12:45:15.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.566+0000 7f2771239640 1 -- 172.21.15.57:0/1610030764 shutdown_connections 2023-10-19T12:45:15.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.566+0000 7f2771239640 1 --2- 172.21.15.57:0/1610030764 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f276c0afa20 0x7f276c0705a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:15.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.566+0000 7f2771239640 1 --2- 172.21.15.57:0/1610030764 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f276c0cbd20 0x7f276c158300 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:15.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.566+0000 7f2771239640 1 -- 172.21.15.57:0/1610030764 >> 172.21.15.57:0/1610030764 conn(0x7f276c069930 msgr2=0x7f276c069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:15.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.566+0000 7f2771239640 1 -- 172.21.15.57:0/1610030764 shutdown_connections 2023-10-19T12:45:15.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.566+0000 7f2771239640 1 -- 172.21.15.57:0/1610030764 wait complete. 2023-10-19T12:45:15.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.567+0000 7f2771239640 1 Processor -- start 2023-10-19T12:45:15.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.568+0000 7f2771239640 1 -- start start 2023-10-19T12:45:15.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.568+0000 7f2771239640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f276c0afa20 0x7f276c130e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:15.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.568+0000 7f2771239640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f276c0b0530 0x7f276c1454c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:15.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.568+0000 7f2771239640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f276c0cbd20 0x7f276c145a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:15.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.568+0000 7f2771239640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f276c143e60 con 0x7f276c0cbd20 2023-10-19T12:45:15.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.568+0000 7f2771239640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f276c144140 con 0x7f276c0b0530 2023-10-19T12:45:15.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.568+0000 7f2771239640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f276c144420 con 0x7f276c0afa20 2023-10-19T12:45:15.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.568+0000 7f276b59e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f276c0cbd20 0x7f276c145a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:15.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.568+0000 7f276b59e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f276c0cbd20 0x7f276c145a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:60140/0 (socket says 172.21.15.57:60140) 2023-10-19T12:45:15.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.568+0000 7f276b59e640 1 -- 172.21.15.57:0/1192956231 learned_addr learned my addr 172.21.15.57:0/1192956231 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:15.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.569+0000 7f276a59c640 1 --2- 172.21.15.57:0/1192956231 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f276c0b0530 0x7f276c1454c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:15.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.569+0000 7f276ad9d640 1 --2- 172.21.15.57:0/1192956231 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f276c0afa20 0x7f276c130e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:15.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.569+0000 7f276b59e640 1 -- 172.21.15.57:0/1192956231 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f276c0afa20 msgr2=0x7f276c130e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:15.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.569+0000 7f276b59e640 1 --2- 172.21.15.57:0/1192956231 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f276c0afa20 0x7f276c130e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:15.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.569+0000 7f276b59e640 1 -- 172.21.15.57:0/1192956231 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f276c0b0530 msgr2=0x7f276c1454c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:15.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.569+0000 7f276b59e640 1 --2- 172.21.15.57:0/1192956231 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f276c0b0530 0x7f276c1454c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:15.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.569+0000 7f276b59e640 1 -- 172.21.15.57:0/1192956231 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2764009560 con 0x7f276c0cbd20 2023-10-19T12:45:15.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.569+0000 7f276a59c640 1 --2- 172.21.15.57:0/1192956231 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f276c0b0530 0x7f276c1454c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:45:15.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.569+0000 7f276b59e640 1 --2- 172.21.15.57:0/1192956231 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f276c0cbd20 0x7f276c145a00 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f275c00c7a0 tx=0x7f275c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:15.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.569+0000 7f27537fe640 1 -- 172.21.15.57:0/1192956231 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f275c003b70 con 0x7f276c0cbd20 2023-10-19T12:45:15.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.569+0000 7f27537fe640 1 -- 172.21.15.57:0/1192956231 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f275c003cf0 con 0x7f276c0cbd20 2023-10-19T12:45:15.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.569+0000 7f27537fe640 1 -- 172.21.15.57:0/1192956231 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f275c00aab0 con 0x7f276c0cbd20 2023-10-19T12:45:15.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.569+0000 7f2771239640 1 -- 172.21.15.57:0/1192956231 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f276c070cf0 con 0x7f276c0cbd20 2023-10-19T12:45:15.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.569+0000 7f2771239640 1 -- 172.21.15.57:0/1192956231 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f276c1449e0 con 0x7f276c0cbd20 2023-10-19T12:45:15.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.570+0000 7f2771239640 1 -- 172.21.15.57:0/1192956231 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f276c0690e0 con 0x7f276c0cbd20 2023-10-19T12:45:15.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.570+0000 7f27537fe640 1 -- 172.21.15.57:0/1192956231 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f275c00ac10 con 0x7f276c0cbd20 2023-10-19T12:45:15.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.571+0000 7f27537fe640 1 --2- 172.21.15.57:0/1192956231 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f275806ae40 0x7f275806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:15.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.571+0000 7f27537fe640 1 -- 172.21.15.57:0/1192956231 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f275c0899d0 con 0x7f276c0cbd20 2023-10-19T12:45:15.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.571+0000 7f276ad9d640 1 --2- 172.21.15.57:0/1192956231 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f275806ae40 0x7f275806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:15.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.572+0000 7f276ad9d640 1 --2- 172.21.15.57:0/1192956231 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f275806ae40 0x7f275806d300 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f2764035000 tx=0x7f2764004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:15.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.575+0000 7f27537fe640 1 -- 172.21.15.57:0/1192956231 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f275c0546c0 con 0x7f276c0cbd20 2023-10-19T12:45:15.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.763+0000 7f2771239640 1 -- 172.21.15.57:0/1192956231 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f276c0cc1e0 con 0x7f276c0cbd20 2023-10-19T12:45:15.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.764+0000 7f27537fe640 1 -- 172.21.15.57:0/1192956231 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) v1 ==== 80+0+0 (secure 0 0 0) 0x7f275c057ce0 con 0x7f276c0cbd20 2023-10-19T12:45:15.767 INFO:teuthology.orchestra.run.smithi057.stderr:listed 0 entries 2023-10-19T12:45:15.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.765+0000 7f2771239640 1 -- 172.21.15.57:0/1192956231 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f275806ae40 msgr2=0x7f275806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:15.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.765+0000 7f2771239640 1 --2- 172.21.15.57:0/1192956231 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f275806ae40 0x7f275806d300 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f2764035000 tx=0x7f2764004730 comp rx=0 tx=0).stop 2023-10-19T12:45:15.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.765+0000 7f2771239640 1 -- 172.21.15.57:0/1192956231 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f276c0cbd20 msgr2=0x7f276c145a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:15.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.765+0000 7f2771239640 1 --2- 172.21.15.57:0/1192956231 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f276c0cbd20 0x7f276c145a00 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f275c00c7a0 tx=0x7f275c00cc60 comp rx=0 tx=0).stop 2023-10-19T12:45:15.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.766+0000 7f2771239640 1 -- 172.21.15.57:0/1192956231 shutdown_connections 2023-10-19T12:45:15.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.766+0000 7f2771239640 1 --2- 172.21.15.57:0/1192956231 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f275806ae40 0x7f275806d300 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:15.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.766+0000 7f2771239640 1 --2- 172.21.15.57:0/1192956231 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f276c0cbd20 0x7f276c145a00 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:15.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.766+0000 7f2771239640 1 --2- 172.21.15.57:0/1192956231 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f276c0b0530 0x7f276c1454c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:15.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.766+0000 7f2771239640 1 --2- 172.21.15.57:0/1192956231 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f276c0afa20 0x7f276c130e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:15.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.766+0000 7f2771239640 1 -- 172.21.15.57:0/1192956231 >> 172.21.15.57:0/1192956231 conn(0x7f276c069930 msgr2=0x7f276c157d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:15.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.766+0000 7f2771239640 1 -- 172.21.15.57:0/1192956231 shutdown_connections 2023-10-19T12:45:15.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.766+0000 7f2771239640 1 -- 172.21.15.57:0/1192956231 wait complete. 2023-10-19T12:45:15.780 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-19T12:45:15.780 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T12:45:15.797 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T12:45:15.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.903+0000 7f4015bd0640 1 -- 172.21.15.57:0/2312741146 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4010158aa0 msgr2=0x7f40100cc370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:15.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.903+0000 7f4015bd0640 1 --2- 172.21.15.57:0/2312741146 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4010158aa0 0x7f40100cc370 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f4000005fd0 tx=0x7f400002bd90 comp rx=0 tx=0).stop 2023-10-19T12:45:15.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.903+0000 7f4015bd0640 1 -- 172.21.15.57:0/2312741146 shutdown_connections 2023-10-19T12:45:15.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.903+0000 7f4015bd0640 1 --2- 172.21.15.57:0/2312741146 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f40100c9250 0x7f4010070520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:15.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.903+0000 7f4015bd0640 1 --2- 172.21.15.57:0/2312741146 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4010158aa0 0x7f40100cc370 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:15.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.903+0000 7f4015bd0640 1 -- 172.21.15.57:0/2312741146 >> 172.21.15.57:0/2312741146 conn(0x7f40100aec40 msgr2=0x7f40100af050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:15.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.903+0000 7f4015bd0640 1 -- 172.21.15.57:0/2312741146 shutdown_connections 2023-10-19T12:45:15.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.903+0000 7f4015bd0640 1 -- 172.21.15.57:0/2312741146 wait complete. 2023-10-19T12:45:15.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.904+0000 7f4015bd0640 1 Processor -- start 2023-10-19T12:45:15.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.904+0000 7f4015bd0640 1 -- start start 2023-10-19T12:45:15.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.905+0000 7f4015bd0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4010070a60 0x7f401014d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:15.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.905+0000 7f4015bd0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f40100c9250 0x7f401006cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:15.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.905+0000 7f4015bd0640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4010158aa0 0x7f40101591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:15.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.905+0000 7f4015bd0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f401014ddc0 con 0x7f40100c9250 2023-10-19T12:45:15.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.905+0000 7f4015bd0640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40100aa8c0 con 0x7f4010070a60 2023-10-19T12:45:15.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.905+0000 7f4015bd0640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40100aaba0 con 0x7f4010158aa0 2023-10-19T12:45:15.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.905+0000 7f400effd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f40100c9250 0x7f401006cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:15.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.905+0000 7f400effd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f40100c9250 0x7f401006cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:60144/0 (socket says 172.21.15.57:60144) 2023-10-19T12:45:15.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.905+0000 7f400effd640 1 -- 172.21.15.57:0/1047549250 learned_addr learned my addr 172.21.15.57:0/1047549250 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:15.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.905+0000 7f400f7fe640 1 --2- 172.21.15.57:0/1047549250 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4010070a60 0x7f401014d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:15.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.905+0000 7f400ffff640 1 --2- 172.21.15.57:0/1047549250 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4010158aa0 0x7f40101591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:15.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.905+0000 7f400effd640 1 -- 172.21.15.57:0/1047549250 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4010158aa0 msgr2=0x7f40101591e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:15.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.905+0000 7f400effd640 1 --2- 172.21.15.57:0/1047549250 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4010158aa0 0x7f40101591e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:15.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.905+0000 7f400effd640 1 -- 172.21.15.57:0/1047549250 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4010070a60 msgr2=0x7f401014d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:15.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.905+0000 7f400effd640 1 --2- 172.21.15.57:0/1047549250 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4010070a60 0x7f401014d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:15.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.905+0000 7f400effd640 1 -- 172.21.15.57:0/1047549250 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4000009560 con 0x7f40100c9250 2023-10-19T12:45:15.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.905+0000 7f400f7fe640 1 --2- 172.21.15.57:0/1047549250 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4010070a60 0x7f401014d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:45:15.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.906+0000 7f400effd640 1 --2- 172.21.15.57:0/1047549250 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f40100c9250 0x7f401006cbb0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f400400c970 tx=0x7f400400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:15.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.906+0000 7f3ff7fff640 1 -- 172.21.15.57:0/1047549250 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4004007bb0 con 0x7f40100c9250 2023-10-19T12:45:15.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.906+0000 7f3ff7fff640 1 -- 172.21.15.57:0/1047549250 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f4004007e20 con 0x7f40100c9250 2023-10-19T12:45:15.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.906+0000 7f4015bd0640 1 -- 172.21.15.57:0/1047549250 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40100aae80 con 0x7f40100c9250 2023-10-19T12:45:15.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.906+0000 7f3ff7fff640 1 -- 172.21.15.57:0/1047549250 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4004005280 con 0x7f40100c9250 2023-10-19T12:45:15.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.906+0000 7f4015bd0640 1 -- 172.21.15.57:0/1047549250 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40100ab440 con 0x7f40100c9250 2023-10-19T12:45:15.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.907+0000 7f4015bd0640 1 -- 172.21.15.57:0/1047549250 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40100690e0 con 0x7f40100c9250 2023-10-19T12:45:15.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.907+0000 7f3ff7fff640 1 -- 172.21.15.57:0/1047549250 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f40040053e0 con 0x7f40100c9250 2023-10-19T12:45:15.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.908+0000 7f3ff7fff640 1 --2- 172.21.15.57:0/1047549250 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f3ffc06ae40 0x7f3ffc06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:15.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.908+0000 7f3ff7fff640 1 -- 172.21.15.57:0/1047549250 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f400408a000 con 0x7f40100c9250 2023-10-19T12:45:15.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.908+0000 7f400f7fe640 1 --2- 172.21.15.57:0/1047549250 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f3ffc06ae40 0x7f3ffc06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:15.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.908+0000 7f400f7fe640 1 --2- 172.21.15.57:0/1047549250 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f3ffc06ae40 0x7f3ffc06d300 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f4000035000 tx=0x7f4000004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:15.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:15.912+0000 7f3ff7fff640 1 -- 172.21.15.57:0/1047549250 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4004054c60 con 0x7f40100c9250 2023-10-19T12:45:16.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.089+0000 7f4015bd0640 1 -- 172.21.15.57:0/1047549250 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4010071120 con 0x7f40100c9250 2023-10-19T12:45:16.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.090+0000 7f3ff7fff640 1 -- 172.21.15.57:0/1047549250 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18748 (secure 0 0 0) 0x7f400401d070 con 0x7f40100c9250 2023-10-19T12:45:16.094 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:45:16.094 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":27,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T12:44:58.978403+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-19T12:44:16.517510+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-19T12:44:17.398297+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-19T12:44:21.129403+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.23","mappings":[{"from":3,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"3.39","mappings":[{"from":7,"to":1}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.1d","mappings":[{"from":1,"to":5}]},{"pgid":"4.1e","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T12:45:16.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.093+0000 7f4015bd0640 1 -- 172.21.15.57:0/1047549250 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f3ffc06ae40 msgr2=0x7f3ffc06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:16.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.093+0000 7f4015bd0640 1 --2- 172.21.15.57:0/1047549250 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f3ffc06ae40 0x7f3ffc06d300 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f4000035000 tx=0x7f4000004730 comp rx=0 tx=0).stop 2023-10-19T12:45:16.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.093+0000 7f4015bd0640 1 -- 172.21.15.57:0/1047549250 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f40100c9250 msgr2=0x7f401006cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:16.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.093+0000 7f4015bd0640 1 --2- 172.21.15.57:0/1047549250 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f40100c9250 0x7f401006cbb0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f400400c970 tx=0x7f400400ce30 comp rx=0 tx=0).stop 2023-10-19T12:45:16.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.093+0000 7f4015bd0640 1 -- 172.21.15.57:0/1047549250 shutdown_connections 2023-10-19T12:45:16.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.093+0000 7f4015bd0640 1 --2- 172.21.15.57:0/1047549250 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f3ffc06ae40 0x7f3ffc06d300 secure :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f4000035000 tx=0x7f4000004730 comp rx=0 tx=0).stop 2023-10-19T12:45:16.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.093+0000 7f4015bd0640 1 --2- 172.21.15.57:0/1047549250 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4010158aa0 0x7f40101591e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:16.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.093+0000 7f4015bd0640 1 --2- 172.21.15.57:0/1047549250 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f40100c9250 0x7f401006cbb0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:16.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.093+0000 7f4015bd0640 1 --2- 172.21.15.57:0/1047549250 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4010070a60 0x7f401014d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:16.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.093+0000 7f4015bd0640 1 -- 172.21.15.57:0/1047549250 >> 172.21.15.57:0/1047549250 conn(0x7f40100aec40 msgr2=0x7f4010154ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:16.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.093+0000 7f4015bd0640 1 -- 172.21.15.57:0/1047549250 shutdown_connections 2023-10-19T12:45:16.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.093+0000 7f4015bd0640 1 -- 172.21.15.57:0/1047549250 wait complete. 2023-10-19T12:45:16.107 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-19T12:44:07.371186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-19T12:44:16.517510+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-19T12:44:17.398297+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-19T12:44:21.129403+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-19T12:45:16.108 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-19T12:45:16.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.217+0000 7f25a76e5640 1 -- 172.21.15.57:0/107986374 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25a00ceb40 msgr2=0x7f25a00b0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:16.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.217+0000 7f25a76e5640 1 --2- 172.21.15.57:0/107986374 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25a00ceb40 0x7f25a00b0650 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f25a008a810 tx=0x7f259802bcf0 comp rx=0 tx=0).stop 2023-10-19T12:45:16.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.217+0000 7f25a76e5640 1 -- 172.21.15.57:0/107986374 shutdown_connections 2023-10-19T12:45:16.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.217+0000 7f25a76e5640 1 --2- 172.21.15.57:0/107986374 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25a00ceb40 0x7f25a00b0650 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:16.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.217+0000 7f25a76e5640 1 --2- 172.21.15.57:0/107986374 >> v2:172.21.15.179:3300/0 conn(0x7f25a00cf340 0x7f25a00d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:16.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.217+0000 7f25a76e5640 1 -- 172.21.15.57:0/107986374 >> 172.21.15.57:0/107986374 conn(0x7f25a00d0d60 msgr2=0x7f25a00d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:16.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.217+0000 7f25a76e5640 1 -- 172.21.15.57:0/107986374 shutdown_connections 2023-10-19T12:45:16.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.217+0000 7f25a76e5640 1 -- 172.21.15.57:0/107986374 wait complete. 2023-10-19T12:45:16.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.218+0000 7f25a76e5640 1 Processor -- start 2023-10-19T12:45:16.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.218+0000 7f25a76e5640 1 -- start start 2023-10-19T12:45:16.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.219+0000 7f25a76e5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25a00ceb40 0x7f25a009e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:16.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.219+0000 7f25a76e5640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f25a00cf340 0x7f25a0159430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:16.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.219+0000 7f25a76e5640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f25a015b2a0 0x7f25a01f4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:16.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.219+0000 7f25a76e5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25a0159010 con 0x7f25a00ceb40 2023-10-19T12:45:16.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.219+0000 7f25a76e5640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25a009e950 con 0x7f25a00cf340 2023-10-19T12:45:16.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.219+0000 7f25a76e5640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25a01f5ad0 con 0x7f25a015b2a0 2023-10-19T12:45:16.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.219+0000 7f25a5481640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25a00ceb40 0x7f25a009e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:16.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.219+0000 7f25a5481640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25a00ceb40 0x7f25a009e410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:60156/0 (socket says 172.21.15.57:60156) 2023-10-19T12:45:16.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.219+0000 7f25a5481640 1 -- 172.21.15.57:0/2974933645 learned_addr learned my addr 172.21.15.57:0/2974933645 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:16.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.219+0000 7f25a5c82640 1 --2- 172.21.15.57:0/2974933645 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f25a015b2a0 0x7f25a01f4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:16.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.219+0000 7f25a4c80640 1 --2- 172.21.15.57:0/2974933645 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f25a00cf340 0x7f25a0159430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:16.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.219+0000 7f25a5481640 1 -- 172.21.15.57:0/2974933645 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f25a015b2a0 msgr2=0x7f25a01f4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:16.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.219+0000 7f25a5481640 1 --2- 172.21.15.57:0/2974933645 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f25a015b2a0 0x7f25a01f4bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:16.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.219+0000 7f25a5481640 1 -- 172.21.15.57:0/2974933645 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f25a00cf340 msgr2=0x7f25a0159430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:16.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.219+0000 7f25a5481640 1 --2- 172.21.15.57:0/2974933645 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f25a00cf340 0x7f25a0159430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:16.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.219+0000 7f25a5481640 1 -- 172.21.15.57:0/2974933645 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2598009490 con 0x7f25a00ceb40 2023-10-19T12:45:16.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.219+0000 7f25a4c80640 1 --2- 172.21.15.57:0/2974933645 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f25a00cf340 0x7f25a0159430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:45:16.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.220+0000 7f25a5481640 1 --2- 172.21.15.57:0/2974933645 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25a00ceb40 0x7f25a009e410 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f259c00eb10 tx=0x7f259c00c270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:16.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.220+0000 7f2595ffb640 1 -- 172.21.15.57:0/2974933645 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f259c00ce80 con 0x7f25a00ceb40 2023-10-19T12:45:16.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.220+0000 7f2595ffb640 1 -- 172.21.15.57:0/2974933645 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f259c011560 con 0x7f25a00ceb40 2023-10-19T12:45:16.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.220+0000 7f25a76e5640 1 -- 172.21.15.57:0/2974933645 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25a0159970 con 0x7f25a00ceb40 2023-10-19T12:45:16.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.220+0000 7f2595ffb640 1 -- 172.21.15.57:0/2974933645 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f259c011950 con 0x7f25a00ceb40 2023-10-19T12:45:16.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.220+0000 7f25a76e5640 1 -- 172.21.15.57:0/2974933645 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25a01f57f0 con 0x7f25a00ceb40 2023-10-19T12:45:16.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.221+0000 7f25a76e5640 1 -- 172.21.15.57:0/2974933645 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25a00690e0 con 0x7f25a00ceb40 2023-10-19T12:45:16.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.221+0000 7f2595ffb640 1 -- 172.21.15.57:0/2974933645 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f259c011ab0 con 0x7f25a00ceb40 2023-10-19T12:45:16.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.222+0000 7f2595ffb640 1 --2- 172.21.15.57:0/2974933645 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f258406ae40 0x7f258406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:16.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.222+0000 7f2595ffb640 1 -- 172.21.15.57:0/2974933645 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f259c08a690 con 0x7f25a00ceb40 2023-10-19T12:45:16.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.222+0000 7f25a4c80640 1 --2- 172.21.15.57:0/2974933645 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f258406ae40 0x7f258406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:16.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.222+0000 7f25a4c80640 1 --2- 172.21.15.57:0/2974933645 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f258406ae40 0x7f258406d300 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f25a008a550 tx=0x7f2590009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:16.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.226+0000 7f2595ffb640 1 -- 172.21.15.57:0/2974933645 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f259c055380 con 0x7f25a00ceb40 2023-10-19T12:45:16.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.401+0000 7f25a76e5640 1 -- 172.21.15.57:0/2974933645 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f25a00b0d00 con 0x7f25a00ceb40 2023-10-19T12:45:16.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.402+0000 7f2595ffb640 1 -- 172.21.15.57:0/2974933645 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7f259c01f020 con 0x7f25a00ceb40 2023-10-19T12:45:16.405 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-19T12:45:16.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.404+0000 7f25a76e5640 1 -- 172.21.15.57:0/2974933645 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f258406ae40 msgr2=0x7f258406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:16.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.404+0000 7f25a76e5640 1 --2- 172.21.15.57:0/2974933645 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f258406ae40 0x7f258406d300 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f25a008a550 tx=0x7f2590009040 comp rx=0 tx=0).stop 2023-10-19T12:45:16.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.404+0000 7f25a76e5640 1 -- 172.21.15.57:0/2974933645 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25a00ceb40 msgr2=0x7f25a009e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:16.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.404+0000 7f25a76e5640 1 --2- 172.21.15.57:0/2974933645 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25a00ceb40 0x7f25a009e410 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f259c00eb10 tx=0x7f259c00c270 comp rx=0 tx=0).stop 2023-10-19T12:45:16.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.404+0000 7f25a76e5640 1 -- 172.21.15.57:0/2974933645 shutdown_connections 2023-10-19T12:45:16.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.404+0000 7f25a76e5640 1 --2- 172.21.15.57:0/2974933645 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f258406ae40 0x7f258406d300 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:16.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.404+0000 7f25a76e5640 1 --2- 172.21.15.57:0/2974933645 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f25a015b2a0 0x7f25a01f4bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:16.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.404+0000 7f25a76e5640 1 --2- 172.21.15.57:0/2974933645 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f25a00cf340 0x7f25a0159430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:16.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.404+0000 7f25a76e5640 1 --2- 172.21.15.57:0/2974933645 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25a00ceb40 0x7f25a009e410 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:16.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.404+0000 7f25a76e5640 1 -- 172.21.15.57:0/2974933645 >> 172.21.15.57:0/2974933645 conn(0x7f25a00d0d60 msgr2=0x7f25a015a210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:16.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.404+0000 7f25a76e5640 1 -- 172.21.15.57:0/2974933645 shutdown_connections 2023-10-19T12:45:16.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.404+0000 7f25a76e5640 1 -- 172.21.15.57:0/2974933645 wait complete. 2023-10-19T12:45:16.417 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-19T12:45:16.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.526+0000 7efbfe826640 1 -- 172.21.15.57:0/863711730 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efbf8158ec0 msgr2=0x7efbf80cd090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:16.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.526+0000 7efbfe826640 1 --2- 172.21.15.57:0/863711730 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efbf8158ec0 0x7efbf80cd090 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7efbe8005fd0 tx=0x7efbe802bcf0 comp rx=0 tx=0).stop 2023-10-19T12:45:16.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.526+0000 7efbfe826640 1 -- 172.21.15.57:0/863711730 shutdown_connections 2023-10-19T12:45:16.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.526+0000 7efbfe826640 1 --2- 172.21.15.57:0/863711730 >> v2:172.21.15.179:3300/0 conn(0x7efbf80cdb10 0x7efbf815c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:16.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.526+0000 7efbfe826640 1 --2- 172.21.15.57:0/863711730 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7efbf806fe50 0x7efbf80cd5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:16.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.526+0000 7efbfe826640 1 --2- 172.21.15.57:0/863711730 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efbf8158ec0 0x7efbf80cd090 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:16.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.526+0000 7efbfe826640 1 -- 172.21.15.57:0/863711730 >> 172.21.15.57:0/863711730 conn(0x7efbf80b0cd0 msgr2=0x7efbf80b1f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:16.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.526+0000 7efbfe826640 1 -- 172.21.15.57:0/863711730 shutdown_connections 2023-10-19T12:45:16.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.526+0000 7efbfe826640 1 -- 172.21.15.57:0/863711730 wait complete. 2023-10-19T12:45:16.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.527+0000 7efbfe826640 1 Processor -- start 2023-10-19T12:45:16.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.527+0000 7efbfe826640 1 -- start start 2023-10-19T12:45:16.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.528+0000 7efbfe826640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efbf806fe50 0x7efbf80aaac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:16.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.528+0000 7efbfe826640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7efbf80cdb10 0x7efbf806cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:16.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.528+0000 7efbfe826640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efbf80ab000 0x7efbf806d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:16.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.528+0000 7efbfe826640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbf80c8b50 con 0x7efbf80ab000 2023-10-19T12:45:16.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.528+0000 7efbfe826640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbf80c7c30 con 0x7efbf806fe50 2023-10-19T12:45:16.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.528+0000 7efbfe826640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efbf80c7ed0 con 0x7efbf80cdb10 2023-10-19T12:45:16.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.528+0000 7efbfcdc3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efbf80ab000 0x7efbf806d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:16.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.528+0000 7efbfcdc3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efbf80ab000 0x7efbf806d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:60178/0 (socket says 172.21.15.57:60178) 2023-10-19T12:45:16.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.528+0000 7efbfcdc3640 1 -- 172.21.15.57:0/3262175636 learned_addr learned my addr 172.21.15.57:0/3262175636 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:16.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.528+0000 7efbf7fff640 1 --2- 172.21.15.57:0/3262175636 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efbf806fe50 0x7efbf80aaac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:16.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.528+0000 7efbf77fe640 1 --2- 172.21.15.57:0/3262175636 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7efbf80cdb10 0x7efbf806cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:16.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.528+0000 7efbfcdc3640 1 -- 172.21.15.57:0/3262175636 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7efbf80cdb10 msgr2=0x7efbf806cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:16.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.528+0000 7efbfcdc3640 1 --2- 172.21.15.57:0/3262175636 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7efbf80cdb10 0x7efbf806cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:16.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.528+0000 7efbfcdc3640 1 -- 172.21.15.57:0/3262175636 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efbf806fe50 msgr2=0x7efbf80aaac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:16.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.528+0000 7efbfcdc3640 1 --2- 172.21.15.57:0/3262175636 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efbf806fe50 0x7efbf80aaac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:16.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.528+0000 7efbfcdc3640 1 -- 172.21.15.57:0/3262175636 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbe8009560 con 0x7efbf80ab000 2023-10-19T12:45:16.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.528+0000 7efbfcdc3640 1 --2- 172.21.15.57:0/3262175636 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efbf80ab000 0x7efbf806d090 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7efbf000b9e0 tx=0x7efbf000bde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:16.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.529+0000 7efbf4ff9640 1 -- 172.21.15.57:0/3262175636 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf000ca60 con 0x7efbf80ab000 2023-10-19T12:45:16.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.529+0000 7efbf4ff9640 1 -- 172.21.15.57:0/3262175636 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7efbf000ccd0 con 0x7efbf80ab000 2023-10-19T12:45:16.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.529+0000 7efbfe826640 1 -- 172.21.15.57:0/3262175636 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efbf80c81b0 con 0x7efbf80ab000 2023-10-19T12:45:16.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.529+0000 7efbf4ff9640 1 -- 172.21.15.57:0/3262175636 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf00146a0 con 0x7efbf80ab000 2023-10-19T12:45:16.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.529+0000 7efbfe826640 1 -- 172.21.15.57:0/3262175636 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efbf80c8770 con 0x7efbf80ab000 2023-10-19T12:45:16.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.530+0000 7efbfe826640 1 -- 172.21.15.57:0/3262175636 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efbf80c8c90 con 0x7efbf80ab000 2023-10-19T12:45:16.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.530+0000 7efbf4ff9640 1 -- 172.21.15.57:0/3262175636 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7efbf001d3b0 con 0x7efbf80ab000 2023-10-19T12:45:16.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.531+0000 7efbf4ff9640 1 --2- 172.21.15.57:0/3262175636 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7efbe406ae40 0x7efbe406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:16.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.531+0000 7efbf4ff9640 1 -- 172.21.15.57:0/3262175636 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7efbf0089980 con 0x7efbf80ab000 2023-10-19T12:45:16.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.531+0000 7efbf7fff640 1 --2- 172.21.15.57:0/3262175636 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7efbe406ae40 0x7efbe406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:16.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.531+0000 7efbf7fff640 1 --2- 172.21.15.57:0/3262175636 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7efbe406ae40 0x7efbe406d300 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7efbe8035000 tx=0x7efbe8004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:16.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.535+0000 7efbf4ff9640 1 -- 172.21.15.57:0/3262175636 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7efbf80c8c90 con 0x7efbf80ab000 2023-10-19T12:45:16.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.712+0000 7efbfe826640 1 -- 172.21.15.57:0/3262175636 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7efbf80c8c90 con 0x7efbf80ab000 2023-10-19T12:45:16.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.713+0000 7efbf4ff9640 1 -- 172.21.15.57:0/3262175636 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7efbf0019020 con 0x7efbf80ab000 2023-10-19T12:45:16.716 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2023-10-19T12:45:16.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.715+0000 7efbfe826640 1 -- 172.21.15.57:0/3262175636 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7efbe406ae40 msgr2=0x7efbe406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:16.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.715+0000 7efbfe826640 1 --2- 172.21.15.57:0/3262175636 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7efbe406ae40 0x7efbe406d300 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7efbe8035000 tx=0x7efbe8004730 comp rx=0 tx=0).stop 2023-10-19T12:45:16.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.715+0000 7efbfe826640 1 -- 172.21.15.57:0/3262175636 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efbf80ab000 msgr2=0x7efbf806d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:16.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.715+0000 7efbfe826640 1 --2- 172.21.15.57:0/3262175636 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efbf80ab000 0x7efbf806d090 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7efbf000b9e0 tx=0x7efbf000bde0 comp rx=0 tx=0).stop 2023-10-19T12:45:16.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.715+0000 7efbfe826640 1 -- 172.21.15.57:0/3262175636 shutdown_connections 2023-10-19T12:45:16.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.715+0000 7efbfe826640 1 --2- 172.21.15.57:0/3262175636 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7efbe406ae40 0x7efbe406d300 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:16.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.715+0000 7efbfe826640 1 --2- 172.21.15.57:0/3262175636 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efbf80ab000 0x7efbf806d090 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:16.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.715+0000 7efbfe826640 1 --2- 172.21.15.57:0/3262175636 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7efbf80cdb10 0x7efbf806cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:16.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.715+0000 7efbfe826640 1 --2- 172.21.15.57:0/3262175636 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efbf806fe50 0x7efbf80aaac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:16.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.715+0000 7efbfe826640 1 -- 172.21.15.57:0/3262175636 >> 172.21.15.57:0/3262175636 conn(0x7efbf80b0cd0 msgr2=0x7efbf806bf90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:16.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.715+0000 7efbfe826640 1 -- 172.21.15.57:0/3262175636 shutdown_connections 2023-10-19T12:45:16.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.715+0000 7efbfe826640 1 -- 172.21.15.57:0/3262175636 wait complete. 2023-10-19T12:45:16.729 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-19T12:45:16.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.837+0000 7f0654cd7640 1 -- 172.21.15.57:0/93265975 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f065015b2a0 msgr2=0x7f06500cf720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:16.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.837+0000 7f0654cd7640 1 --2- 172.21.15.57:0/93265975 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f065015b2a0 0x7f06500cf720 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0638005fd0 tx=0x7f063802bd90 comp rx=0 tx=0).stop 2023-10-19T12:45:16.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.838+0000 7f0654cd7640 1 -- 172.21.15.57:0/93265975 shutdown_connections 2023-10-19T12:45:16.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.838+0000 7f0654cd7640 1 --2- 172.21.15.57:0/93265975 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f065015b2a0 0x7f06500cf720 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:16.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.838+0000 7f0654cd7640 1 --2- 172.21.15.57:0/93265975 >> v2:172.21.15.179:3300/0 conn(0x7f06500cf340 0x7f06500d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:16.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.838+0000 7f0654cd7640 1 -- 172.21.15.57:0/93265975 >> 172.21.15.57:0/93265975 conn(0x7f06500d0d60 msgr2=0x7f06500d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:16.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.838+0000 7f0654cd7640 1 -- 172.21.15.57:0/93265975 shutdown_connections 2023-10-19T12:45:16.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.838+0000 7f0654cd7640 1 -- 172.21.15.57:0/93265975 wait complete. 2023-10-19T12:45:16.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.839+0000 7f0654cd7640 1 Processor -- start 2023-10-19T12:45:16.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.839+0000 7f0654cd7640 1 -- start start 2023-10-19T12:45:16.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.840+0000 7f0654cd7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06500ceb40 0x7f06500c7e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:16.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.840+0000 7f0654cd7640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f06500cf340 0x7f06500b5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:16.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.840+0000 7f0654cd7640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f065015b2a0 0x7f065006b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:16.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.840+0000 7f0654cd7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06500b6590 con 0x7f06500ceb40 2023-10-19T12:45:16.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.840+0000 7f0654cd7640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06500b5a80 con 0x7f065015b2a0 2023-10-19T12:45:16.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.840+0000 7f0654cd7640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06501f4ef0 con 0x7f06500cf340 2023-10-19T12:45:16.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.840+0000 7f064e59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06500ceb40 0x7f06500c7e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:16.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.840+0000 7f064e59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06500ceb40 0x7f06500c7e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50188/0 (socket says 172.21.15.57:50188) 2023-10-19T12:45:16.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.840+0000 7f064e59c640 1 -- 172.21.15.57:0/1146016885 learned_addr learned my addr 172.21.15.57:0/1146016885 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:16.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.840+0000 7f064dd9b640 1 --2- 172.21.15.57:0/1146016885 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f06500cf340 0x7f06500b5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:16.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.840+0000 7f064ed9d640 1 --2- 172.21.15.57:0/1146016885 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f065015b2a0 0x7f065006b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:16.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.840+0000 7f064e59c640 1 -- 172.21.15.57:0/1146016885 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f06500cf340 msgr2=0x7f06500b5540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:16.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.840+0000 7f064e59c640 1 --2- 172.21.15.57:0/1146016885 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f06500cf340 0x7f06500b5540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:16.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.840+0000 7f064e59c640 1 -- 172.21.15.57:0/1146016885 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f065015b2a0 msgr2=0x7f065006b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:16.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.840+0000 7f064e59c640 1 --2- 172.21.15.57:0/1146016885 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f065015b2a0 0x7f065006b000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:16.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.840+0000 7f064e59c640 1 -- 172.21.15.57:0/1146016885 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0648009490 con 0x7f06500ceb40 2023-10-19T12:45:16.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.840+0000 7f064ed9d640 1 --2- 172.21.15.57:0/1146016885 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f065015b2a0 0x7f065006b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T12:45:16.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.840+0000 7f064e59c640 1 --2- 172.21.15.57:0/1146016885 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06500ceb40 0x7f06500c7e30 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f064800ea60 tx=0x7f064800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:16.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.841+0000 7f0646ffd640 1 -- 172.21.15.57:0/1146016885 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f064800cde0 con 0x7f06500ceb40 2023-10-19T12:45:16.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.841+0000 7f0646ffd640 1 -- 172.21.15.57:0/1146016885 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f0648004020 con 0x7f06500ceb40 2023-10-19T12:45:16.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.841+0000 7f0654cd7640 1 -- 172.21.15.57:0/1146016885 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0638009560 con 0x7f06500ceb40 2023-10-19T12:45:16.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.841+0000 7f0646ffd640 1 -- 172.21.15.57:0/1146016885 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0648004410 con 0x7f06500ceb40 2023-10-19T12:45:16.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.841+0000 7f0654cd7640 1 -- 172.21.15.57:0/1146016885 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06501f54b0 con 0x7f06500ceb40 2023-10-19T12:45:16.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.842+0000 7f0654cd7640 1 -- 172.21.15.57:0/1146016885 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06500690e0 con 0x7f06500ceb40 2023-10-19T12:45:16.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.842+0000 7f0646ffd640 1 -- 172.21.15.57:0/1146016885 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f06480035f0 con 0x7f06500ceb40 2023-10-19T12:45:16.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.842+0000 7f0646ffd640 1 --2- 172.21.15.57:0/1146016885 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f063006ae70 0x7f063006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:16.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.842+0000 7f0646ffd640 1 -- 172.21.15.57:0/1146016885 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f0648014070 con 0x7f06500ceb40 2023-10-19T12:45:16.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.843+0000 7f064dd9b640 1 --2- 172.21.15.57:0/1146016885 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f063006ae70 0x7f063006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:16.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.843+0000 7f064dd9b640 1 --2- 172.21.15.57:0/1146016885 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f063006ae70 0x7f063006d330 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f0638035000 tx=0x7f0638004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:16.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:16.847+0000 7f0646ffd640 1 -- 172.21.15.57:0/1146016885 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0648055190 con 0x7f06500ceb40 2023-10-19T12:45:17.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.024+0000 7f0654cd7640 1 -- 172.21.15.57:0/1146016885 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f06500b3980 con 0x7f06500ceb40 2023-10-19T12:45:17.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.024+0000 7f0646ffd640 1 -- 172.21.15.57:0/1146016885 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7f06480587b0 con 0x7f06500ceb40 2023-10-19T12:45:17.028 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 64 2023-10-19T12:45:17.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.026+0000 7f0654cd7640 1 -- 172.21.15.57:0/1146016885 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f063006ae70 msgr2=0x7f063006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:17.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.026+0000 7f0654cd7640 1 --2- 172.21.15.57:0/1146016885 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f063006ae70 0x7f063006d330 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f0638035000 tx=0x7f0638004730 comp rx=0 tx=0).stop 2023-10-19T12:45:17.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.026+0000 7f0654cd7640 1 -- 172.21.15.57:0/1146016885 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06500ceb40 msgr2=0x7f06500c7e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:17.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.026+0000 7f0654cd7640 1 --2- 172.21.15.57:0/1146016885 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06500ceb40 0x7f06500c7e30 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f064800ea60 tx=0x7f064800ef20 comp rx=0 tx=0).stop 2023-10-19T12:45:17.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.026+0000 7f0654cd7640 1 -- 172.21.15.57:0/1146016885 shutdown_connections 2023-10-19T12:45:17.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.026+0000 7f0654cd7640 1 --2- 172.21.15.57:0/1146016885 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f063006ae70 0x7f063006d330 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:17.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.026+0000 7f0654cd7640 1 --2- 172.21.15.57:0/1146016885 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f065015b2a0 0x7f065006b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:17.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.026+0000 7f0654cd7640 1 --2- 172.21.15.57:0/1146016885 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f06500cf340 0x7f06500b5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:17.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.026+0000 7f0654cd7640 1 --2- 172.21.15.57:0/1146016885 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06500ceb40 0x7f06500c7e30 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:17.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.026+0000 7f0654cd7640 1 -- 172.21.15.57:0/1146016885 >> 172.21.15.57:0/1146016885 conn(0x7f06500d0d60 msgr2=0x7f06500b0ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:17.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.026+0000 7f0654cd7640 1 -- 172.21.15.57:0/1146016885 shutdown_connections 2023-10-19T12:45:17.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.026+0000 7f0654cd7640 1 -- 172.21.15.57:0/1146016885 wait complete. 2023-10-19T12:45:17.040 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-19T12:45:17.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.149+0000 7ff85cc61640 1 -- 172.21.15.57:0/1265485303 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff85806f0a0 msgr2=0x7ff85806ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:17.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.149+0000 7ff85cc61640 1 --2- 172.21.15.57:0/1265485303 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff85806f0a0 0x7ff85806ddc0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff84c009830 tx=0x7ff84c02bdf0 comp rx=0 tx=0).stop 2023-10-19T12:45:17.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.149+0000 7ff85cc61640 1 -- 172.21.15.57:0/1265485303 shutdown_connections 2023-10-19T12:45:17.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.149+0000 7ff85cc61640 1 --2- 172.21.15.57:0/1265485303 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff85806f0a0 0x7ff85806ddc0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:17.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.149+0000 7ff85cc61640 1 -- 172.21.15.57:0/1265485303 >> 172.21.15.57:0/1265485303 conn(0x7ff8580b0670 msgr2=0x7ff8580b0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:17.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.149+0000 7ff85cc61640 1 -- 172.21.15.57:0/1265485303 shutdown_connections 2023-10-19T12:45:17.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.149+0000 7ff85cc61640 1 -- 172.21.15.57:0/1265485303 wait complete. 2023-10-19T12:45:17.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.151+0000 7ff85cc61640 1 Processor -- start 2023-10-19T12:45:17.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.151+0000 7ff85cc61640 1 -- start start 2023-10-19T12:45:17.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.151+0000 7ff85cc61640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff85806d2d0 0x7ff858145630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:17.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.151+0000 7ff85cc61640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff85806f0a0 0x7ff8580c5d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:17.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.151+0000 7ff85cc61640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff8580ad370 0x7ff85813fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:17.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.151+0000 7ff85cc61640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8581f2c80 con 0x7ff8580ad370 2023-10-19T12:45:17.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.151+0000 7ff85cc61640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff858155ff0 con 0x7ff85806d2d0 2023-10-19T12:45:17.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.151+0000 7ff85cc61640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8580c6460 con 0x7ff85806f0a0 2023-10-19T12:45:17.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.151+0000 7ff856d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff8580ad370 0x7ff85813fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:17.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.152+0000 7ff856d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff8580ad370 0x7ff85813fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50204/0 (socket says 172.21.15.57:50204) 2023-10-19T12:45:17.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.152+0000 7ff856d9d640 1 -- 172.21.15.57:0/705164133 learned_addr learned my addr 172.21.15.57:0/705164133 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:17.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.152+0000 7ff856d9d640 1 -- 172.21.15.57:0/705164133 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff85806f0a0 msgr2=0x7ff8580c5d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:17.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.152+0000 7ff856d9d640 1 --2- 172.21.15.57:0/705164133 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff85806f0a0 0x7ff8580c5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:17.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.152+0000 7ff85659c640 1 --2- 172.21.15.57:0/705164133 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff85806d2d0 0x7ff858145630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:17.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.152+0000 7ff856d9d640 1 -- 172.21.15.57:0/705164133 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff85806d2d0 msgr2=0x7ff858145630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:17.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.152+0000 7ff856d9d640 1 --2- 172.21.15.57:0/705164133 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff85806d2d0 0x7ff858145630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:17.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.152+0000 7ff856d9d640 1 -- 172.21.15.57:0/705164133 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff84c009490 con 0x7ff8580ad370 2023-10-19T12:45:17.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.152+0000 7ff85659c640 1 --2- 172.21.15.57:0/705164133 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff85806d2d0 0x7ff858145630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:45:17.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.152+0000 7ff856d9d640 1 --2- 172.21.15.57:0/705164133 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff8580ad370 0x7ff85813fb80 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7ff84800cb10 tx=0x7ff84800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:17.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.152+0000 7ff846ffd640 1 -- 172.21.15.57:0/705164133 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff84800ac70 con 0x7ff8580ad370 2023-10-19T12:45:17.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.152+0000 7ff846ffd640 1 -- 172.21.15.57:0/705164133 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7ff848012420 con 0x7ff8580ad370 2023-10-19T12:45:17.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.152+0000 7ff85cc61640 1 -- 172.21.15.57:0/705164133 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8580c8a30 con 0x7ff8580ad370 2023-10-19T12:45:17.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.152+0000 7ff846ffd640 1 -- 172.21.15.57:0/705164133 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff848012830 con 0x7ff8580ad370 2023-10-19T12:45:17.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.152+0000 7ff85cc61640 1 -- 172.21.15.57:0/705164133 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8580c9720 con 0x7ff8580ad370 2023-10-19T12:45:17.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.153+0000 7ff85cc61640 1 -- 172.21.15.57:0/705164133 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8580ce1d0 con 0x7ff8580ad370 2023-10-19T12:45:17.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.154+0000 7ff846ffd640 1 -- 172.21.15.57:0/705164133 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7ff84801b400 con 0x7ff8580ad370 2023-10-19T12:45:17.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.154+0000 7ff846ffd640 1 --2- 172.21.15.57:0/705164133 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7ff83806ae40 0x7ff83806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:17.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.154+0000 7ff846ffd640 1 -- 172.21.15.57:0/705164133 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7ff84808a480 con 0x7ff8580ad370 2023-10-19T12:45:17.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.155+0000 7ff85659c640 1 --2- 172.21.15.57:0/705164133 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7ff83806ae40 0x7ff83806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:17.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.155+0000 7ff85659c640 1 --2- 172.21.15.57:0/705164133 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7ff83806ae40 0x7ff83806d300 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7ff85809cca0 tx=0x7ff850009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:17.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.158+0000 7ff846ffd640 1 -- 172.21.15.57:0/705164133 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff848055180 con 0x7ff8580ad370 2023-10-19T12:45:17.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.333+0000 7ff85cc61640 1 -- 172.21.15.57:0/705164133 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7ff858155dc0 con 0x7ff8580ad370 2023-10-19T12:45:17.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.334+0000 7ff846ffd640 1 -- 172.21.15.57:0/705164133 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7ff858155dc0 con 0x7ff8580ad370 2023-10-19T12:45:17.337 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2023-10-19T12:45:17.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.335+0000 7ff85cc61640 1 -- 172.21.15.57:0/705164133 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7ff83806ae40 msgr2=0x7ff83806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:17.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.335+0000 7ff85cc61640 1 --2- 172.21.15.57:0/705164133 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7ff83806ae40 0x7ff83806d300 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7ff85809cca0 tx=0x7ff850009040 comp rx=0 tx=0).stop 2023-10-19T12:45:17.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.335+0000 7ff85cc61640 1 -- 172.21.15.57:0/705164133 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff8580ad370 msgr2=0x7ff85813fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:17.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.335+0000 7ff85cc61640 1 --2- 172.21.15.57:0/705164133 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff8580ad370 0x7ff85813fb80 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7ff84800cb10 tx=0x7ff84800a060 comp rx=0 tx=0).stop 2023-10-19T12:45:17.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.335+0000 7ff85cc61640 1 -- 172.21.15.57:0/705164133 shutdown_connections 2023-10-19T12:45:17.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.335+0000 7ff85cc61640 1 --2- 172.21.15.57:0/705164133 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7ff83806ae40 0x7ff83806d300 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:17.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.335+0000 7ff85cc61640 1 --2- 172.21.15.57:0/705164133 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff8580ad370 0x7ff85813fb80 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:17.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.335+0000 7ff85cc61640 1 --2- 172.21.15.57:0/705164133 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff85806f0a0 0x7ff8580c5d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:17.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.335+0000 7ff85cc61640 1 --2- 172.21.15.57:0/705164133 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff85806d2d0 0x7ff858145630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:17.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.335+0000 7ff85cc61640 1 -- 172.21.15.57:0/705164133 >> 172.21.15.57:0/705164133 conn(0x7ff8580b0670 msgr2=0x7ff8580ad7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:17.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.335+0000 7ff85cc61640 1 -- 172.21.15.57:0/705164133 shutdown_connections 2023-10-19T12:45:17.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.336+0000 7ff85cc61640 1 -- 172.21.15.57:0/705164133 wait complete. 2023-10-19T12:45:17.349 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-19T12:45:17.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.456+0000 7f16825f0640 1 -- 172.21.15.57:0/2950914546 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f167c06f0a0 msgr2=0x7f167c06dbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:17.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.456+0000 7f16825f0640 1 --2- 172.21.15.57:0/2950914546 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f167c06f0a0 0x7f167c06dbf0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f166c005fd0 tx=0x7f166c02be20 comp rx=0 tx=0).stop 2023-10-19T12:45:17.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.457+0000 7f16825f0640 1 -- 172.21.15.57:0/2950914546 shutdown_connections 2023-10-19T12:45:17.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.457+0000 7f16825f0640 1 --2- 172.21.15.57:0/2950914546 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f167c06f0a0 0x7f167c06dbf0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:17.461 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.457+0000 7f16825f0640 1 --2- 172.21.15.57:0/2950914546 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f167c06d2d0 0x7f167c06d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:17.461 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.457+0000 7f16825f0640 1 -- 172.21.15.57:0/2950914546 >> 172.21.15.57:0/2950914546 conn(0x7f167c0b0670 msgr2=0x7f167c0b0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:17.461 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.457+0000 7f16825f0640 1 -- 172.21.15.57:0/2950914546 shutdown_connections 2023-10-19T12:45:17.461 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.457+0000 7f16825f0640 1 -- 172.21.15.57:0/2950914546 wait complete. 2023-10-19T12:45:17.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.458+0000 7f16825f0640 1 Processor -- start 2023-10-19T12:45:17.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.458+0000 7f16825f0640 1 -- start start 2023-10-19T12:45:17.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.459+0000 7f16825f0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f167c06d2d0 0x7f167c145910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:17.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.459+0000 7f16825f0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f167c06f0a0 0x7f167c0c8270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:17.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.459+0000 7f16825f0640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f167c0ab950 0x7f167c1f2040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:17.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.459+0000 7f16825f0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f167c0c87b0 con 0x7f167c06f0a0 2023-10-19T12:45:17.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.459+0000 7f16825f0640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f167c0c8a50 con 0x7f167c06d2d0 2023-10-19T12:45:17.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.459+0000 7f16825f0640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f167c1f2970 con 0x7f167c0ab950 2023-10-19T12:45:17.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.459+0000 7f167b7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f167c06f0a0 0x7f167c0c8270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:17.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.459+0000 7f167b7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f167c06f0a0 0x7f167c0c8270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50214/0 (socket says 172.21.15.57:50214) 2023-10-19T12:45:17.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.459+0000 7f167b7fe640 1 -- 172.21.15.57:0/2075028345 learned_addr learned my addr 172.21.15.57:0/2075028345 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:17.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.459+0000 7f167bfff640 1 --2- 172.21.15.57:0/2075028345 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f167c06d2d0 0x7f167c145910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:17.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.459+0000 7f1680b8d640 1 --2- 172.21.15.57:0/2075028345 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f167c0ab950 0x7f167c1f2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:17.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.459+0000 7f167b7fe640 1 -- 172.21.15.57:0/2075028345 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f167c0ab950 msgr2=0x7f167c1f2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:17.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.459+0000 7f167b7fe640 1 --2- 172.21.15.57:0/2075028345 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f167c0ab950 0x7f167c1f2040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:17.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.459+0000 7f167b7fe640 1 -- 172.21.15.57:0/2075028345 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f167c06d2d0 msgr2=0x7f167c145910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:17.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.459+0000 7f167b7fe640 1 --2- 172.21.15.57:0/2075028345 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f167c06d2d0 0x7f167c145910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:17.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.459+0000 7f167b7fe640 1 -- 172.21.15.57:0/2075028345 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f166c009560 con 0x7f167c06f0a0 2023-10-19T12:45:17.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.459+0000 7f167bfff640 1 --2- 172.21.15.57:0/2075028345 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f167c06d2d0 0x7f167c145910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:45:17.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.459+0000 7f167b7fe640 1 --2- 172.21.15.57:0/2075028345 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f167c06f0a0 0x7f167c0c8270 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f166c009850 tx=0x7f166c02dba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:17.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.459+0000 7f1678ff9640 1 -- 172.21.15.57:0/2075028345 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f166c0035f0 con 0x7f167c06f0a0 2023-10-19T12:45:17.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.459+0000 7f1678ff9640 1 -- 172.21.15.57:0/2075028345 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f166c003840 con 0x7f167c06f0a0 2023-10-19T12:45:17.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.459+0000 7f16825f0640 1 -- 172.21.15.57:0/2075028345 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f167c1f2c50 con 0x7f167c06f0a0 2023-10-19T12:45:17.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.459+0000 7f1678ff9640 1 -- 172.21.15.57:0/2075028345 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f166c003bf0 con 0x7f167c06f0a0 2023-10-19T12:45:17.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.460+0000 7f16825f0640 1 -- 172.21.15.57:0/2075028345 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f167c1f3210 con 0x7f167c06f0a0 2023-10-19T12:45:17.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.461+0000 7f16825f0640 1 -- 172.21.15.57:0/2075028345 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f167c0ce1d0 con 0x7f167c06f0a0 2023-10-19T12:45:17.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.461+0000 7f1678ff9640 1 -- 172.21.15.57:0/2075028345 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f166c003d50 con 0x7f167c06f0a0 2023-10-19T12:45:17.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.461+0000 7f1678ff9640 1 --2- 172.21.15.57:0/2075028345 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f166806ae40 0x7f166806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:17.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.461+0000 7f1678ff9640 1 -- 172.21.15.57:0/2075028345 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f166c0ac660 con 0x7f167c06f0a0 2023-10-19T12:45:17.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.462+0000 7f167bfff640 1 --2- 172.21.15.57:0/2075028345 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f166806ae40 0x7f166806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:17.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.462+0000 7f167bfff640 1 --2- 172.21.15.57:0/2075028345 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f166806ae40 0x7f166806d300 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f167000abe0 tx=0x7f167000a5c0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:17.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.466+0000 7f1678ff9640 1 -- 172.21.15.57:0/2075028345 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f166c077500 con 0x7f167c06f0a0 2023-10-19T12:45:17.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.654+0000 7f16825f0640 1 -- 172.21.15.57:0/2075028345 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f167c0690e0 con 0x7f167c06f0a0 2023-10-19T12:45:17.658 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.654+0000 7f1678ff9640 1 -- 172.21.15.57:0/2075028345 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) v1 ==== 80+0+0 (secure 0 0 0) 0x7f166c07ab20 con 0x7f167c06f0a0 2023-10-19T12:45:17.658 INFO:teuthology.orchestra.run.smithi057.stderr:listed 0 entries 2023-10-19T12:45:17.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.656+0000 7f16825f0640 1 -- 172.21.15.57:0/2075028345 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f166806ae40 msgr2=0x7f166806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:17.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.656+0000 7f16825f0640 1 --2- 172.21.15.57:0/2075028345 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f166806ae40 0x7f166806d300 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f167000abe0 tx=0x7f167000a5c0 comp rx=0 tx=0).stop 2023-10-19T12:45:17.660 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.656+0000 7f16825f0640 1 -- 172.21.15.57:0/2075028345 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f167c06f0a0 msgr2=0x7f167c0c8270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:17.660 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.656+0000 7f16825f0640 1 --2- 172.21.15.57:0/2075028345 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f167c06f0a0 0x7f167c0c8270 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f166c009850 tx=0x7f166c02dba0 comp rx=0 tx=0).stop 2023-10-19T12:45:17.660 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.656+0000 7f16825f0640 1 -- 172.21.15.57:0/2075028345 shutdown_connections 2023-10-19T12:45:17.660 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.656+0000 7f16825f0640 1 --2- 172.21.15.57:0/2075028345 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f166806ae40 0x7f166806d300 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:17.660 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.656+0000 7f16825f0640 1 --2- 172.21.15.57:0/2075028345 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f167c0ab950 0x7f167c1f2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:17.661 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.656+0000 7f16825f0640 1 --2- 172.21.15.57:0/2075028345 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f167c06f0a0 0x7f167c0c8270 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:17.661 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.656+0000 7f16825f0640 1 --2- 172.21.15.57:0/2075028345 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f167c06d2d0 0x7f167c145910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:17.661 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.656+0000 7f16825f0640 1 -- 172.21.15.57:0/2075028345 >> 172.21.15.57:0/2075028345 conn(0x7f167c0b0670 msgr2=0x7f167c0c3290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:17.661 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.656+0000 7f16825f0640 1 -- 172.21.15.57:0/2075028345 shutdown_connections 2023-10-19T12:45:17.661 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:17.656+0000 7f16825f0640 1 -- 172.21.15.57:0/2075028345 wait complete. 2023-10-19T12:45:17.670 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi179.front.sepia.ceph.com... 2023-10-19T12:45:17.670 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:45:17.671 DEBUG:teuthology.orchestra.run.smithi179:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2023-10-19T12:45:17.703 INFO:teuthology.orchestra.run:waiting for 300 2023-10-19T12:45:17.704 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi179.front.sepia.ceph.com 2023-10-19T12:45:17.704 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:45:17.704 DEBUG:teuthology.orchestra.run.smithi179:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-10-19T12:45:17.824 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T12:45:17.825 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T12:45:17.841 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T12:45:17.841 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T12:45:17.896 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T12:45:18.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.043+0000 7fa153875640 1 -- 172.21.15.57:0/1486639175 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa138003970 msgr2=0x7fa138003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:18.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.043+0000 7fa153875640 1 --2- 172.21.15.57:0/1486639175 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa138003970 0x7fa138003d90 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fa14c0b3920 tx=0x7fa1380057f0 comp rx=0 tx=0).stop 2023-10-19T12:45:18.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.043+0000 7fa151e12640 1 -- 172.21.15.57:0/1486639175 reap_dead start 2023-10-19T12:45:18.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.043+0000 7fa153875640 1 -- 172.21.15.57:0/1486639175 shutdown_connections 2023-10-19T12:45:18.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.043+0000 7fa153875640 1 -- 172.21.15.57:0/1486639175 >> 172.21.15.57:0/1486639175 conn(0x7fa14c0d0d60 msgr2=0x7fa14c0d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:18.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.043+0000 7fa153875640 1 -- 172.21.15.57:0/1486639175 shutdown_connections 2023-10-19T12:45:18.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.043+0000 7fa153875640 1 -- 172.21.15.57:0/1486639175 wait complete. 2023-10-19T12:45:18.049 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.044+0000 7fa153875640 1 Processor -- start 2023-10-19T12:45:18.049 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.044+0000 7fa153875640 1 -- start start 2023-10-19T12:45:18.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.045+0000 7fa153875640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa138003970 0x7fa14c14a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:18.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.045+0000 7fa153875640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa14c0b4db0 0x7fa14c0b6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:18.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.045+0000 7fa153875640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa14c15d8e0 0x7fa14c0ca790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:18.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.045+0000 7fa153875640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa14c0b5f10 con 0x7fa138003970 2023-10-19T12:45:18.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.045+0000 7fa153875640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa14c14a960 con 0x7fa14c15d8e0 2023-10-19T12:45:18.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.045+0000 7fa153875640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa14c0b5290 con 0x7fa14c0b4db0 2023-10-19T12:45:18.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.045+0000 7fa151611640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa138003970 0x7fa14c14a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:18.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.045+0000 7fa151611640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa138003970 0x7fa14c14a420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50242/0 (socket says 172.21.15.57:50242) 2023-10-19T12:45:18.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.045+0000 7fa151611640 1 -- 172.21.15.57:0/3177317067 learned_addr learned my addr 172.21.15.57:0/3177317067 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:18.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.045+0000 7fa151611640 1 -- 172.21.15.57:0/3177317067 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa14c0b4db0 msgr2=0x7fa14c0b6330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:18.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.045+0000 7fa151611640 1 --2- 172.21.15.57:0/3177317067 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa14c0b4db0 0x7fa14c0b6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:18.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.045+0000 7fa151611640 1 -- 172.21.15.57:0/3177317067 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa14c15d8e0 msgr2=0x7fa14c0ca790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:18.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.045+0000 7fa151611640 1 --2- 172.21.15.57:0/3177317067 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa14c15d8e0 0x7fa14c0ca790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:18.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.045+0000 7fa151611640 1 -- 172.21.15.57:0/3177317067 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa14c0b5570 con 0x7fa138003970 2023-10-19T12:45:18.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.045+0000 7fa151611640 1 --2- 172.21.15.57:0/3177317067 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa138003970 0x7fa14c14a420 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fa148038040 tx=0x7fa1480366b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:18.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.046+0000 7fa141ffb640 1 -- 172.21.15.57:0/3177317067 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa148005d30 con 0x7fa138003970 2023-10-19T12:45:18.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.046+0000 7fa141ffb640 1 -- 172.21.15.57:0/3177317067 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fa148036b20 con 0x7fa138003970 2023-10-19T12:45:18.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.046+0000 7fa153875640 1 -- 172.21.15.57:0/3177317067 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa14c0b5850 con 0x7fa138003970 2023-10-19T12:45:18.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.046+0000 7fa141ffb640 1 -- 172.21.15.57:0/3177317067 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa148005020 con 0x7fa138003970 2023-10-19T12:45:18.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.046+0000 7fa153875640 1 -- 172.21.15.57:0/3177317067 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa14c0b6870 con 0x7fa138003970 2023-10-19T12:45:18.054 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.047+0000 7fa153875640 1 -- 172.21.15.57:0/3177317067 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa14c0690e0 con 0x7fa138003970 2023-10-19T12:45:18.055 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.047+0000 7fa141ffb640 1 -- 172.21.15.57:0/3177317067 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fa148005180 con 0x7fa138003970 2023-10-19T12:45:18.055 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.048+0000 7fa141ffb640 1 --2- 172.21.15.57:0/3177317067 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fa13006ae40 0x7fa13006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:18.055 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.048+0000 7fa141ffb640 1 -- 172.21.15.57:0/3177317067 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fa1480ac290 con 0x7fa138003970 2023-10-19T12:45:18.055 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.048+0000 7fa150e10640 1 --2- 172.21.15.57:0/3177317067 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fa13006ae40 0x7fa13006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:18.055 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.049+0000 7fa150e10640 1 --2- 172.21.15.57:0/3177317067 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fa13006ae40 0x7fa13006d300 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa13c000c00 tx=0x7fa13c001bc0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:18.056 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.052+0000 7fa141ffb640 1 -- 172.21.15.57:0/3177317067 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa148076f80 con 0x7fa138003970 2023-10-19T12:45:18.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.229+0000 7fa153875640 1 -- 172.21.15.57:0/3177317067 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa14c0d0b40 con 0x7fa138003970 2023-10-19T12:45:18.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.230+0000 7fa141ffb640 1 -- 172.21.15.57:0/3177317067 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18748 (secure 0 0 0) 0x7fa14807a5a0 con 0x7fa138003970 2023-10-19T12:45:18.234 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:45:18.234 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":27,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T12:44:58.978403+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-19T12:44:16.517510+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-19T12:44:17.398297+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-19T12:44:21.129403+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.23","mappings":[{"from":3,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"3.39","mappings":[{"from":7,"to":1}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.1d","mappings":[{"from":1,"to":5}]},{"pgid":"4.1e","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T12:45:18.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.232+0000 7fa153875640 1 -- 172.21.15.57:0/3177317067 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fa13006ae40 msgr2=0x7fa13006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:18.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.232+0000 7fa153875640 1 --2- 172.21.15.57:0/3177317067 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fa13006ae40 0x7fa13006d300 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa13c000c00 tx=0x7fa13c001bc0 comp rx=0 tx=0).stop 2023-10-19T12:45:18.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.232+0000 7fa153875640 1 -- 172.21.15.57:0/3177317067 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa138003970 msgr2=0x7fa14c14a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:18.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.232+0000 7fa153875640 1 --2- 172.21.15.57:0/3177317067 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa138003970 0x7fa14c14a420 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fa148038040 tx=0x7fa1480366b0 comp rx=0 tx=0).stop 2023-10-19T12:45:18.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.232+0000 7fa153875640 1 -- 172.21.15.57:0/3177317067 shutdown_connections 2023-10-19T12:45:18.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.232+0000 7fa153875640 1 --2- 172.21.15.57:0/3177317067 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fa13006ae40 0x7fa13006d300 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:18.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.232+0000 7fa153875640 1 --2- 172.21.15.57:0/3177317067 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa14c15d8e0 0x7fa14c0ca790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:18.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.232+0000 7fa153875640 1 --2- 172.21.15.57:0/3177317067 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa14c0b4db0 0x7fa14c0b6330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:18.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.232+0000 7fa153875640 1 --2- 172.21.15.57:0/3177317067 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa138003970 0x7fa14c14a420 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:18.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.232+0000 7fa153875640 1 -- 172.21.15.57:0/3177317067 >> 172.21.15.57:0/3177317067 conn(0x7fa14c0d0d60 msgr2=0x7fa14c15a290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:18.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.232+0000 7fa153875640 1 -- 172.21.15.57:0/3177317067 shutdown_connections 2023-10-19T12:45:18.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.232+0000 7fa153875640 1 -- 172.21.15.57:0/3177317067 wait complete. 2023-10-19T12:45:18.246 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-19T12:44:07.371186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-19T12:44:16.517510+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-19T12:44:17.398297+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-19T12:44:21.129403+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-19T12:45:18.247 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-19T12:45:18.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.356+0000 7fc8e2444640 1 -- 172.21.15.57:0/1086730398 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc8dc13e520 msgr2=0x7fc8dc06bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:18.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.356+0000 7fc8e2444640 1 --2- 172.21.15.57:0/1086730398 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc8dc13e520 0x7fc8dc06bb60 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc8d0009830 tx=0x7fc8d002bb90 comp rx=0 tx=0).stop 2023-10-19T12:45:18.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.356+0000 7fc8e2444640 1 -- 172.21.15.57:0/1086730398 shutdown_connections 2023-10-19T12:45:18.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.356+0000 7fc8e2444640 1 --2- 172.21.15.57:0/1086730398 >> v2:172.21.15.179:3300/0 conn(0x7fc8dc070af0 0x7fc8dc0affe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:18.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.356+0000 7fc8e2444640 1 --2- 172.21.15.57:0/1086730398 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc8dc13e520 0x7fc8dc06bb60 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:18.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.356+0000 7fc8e2444640 1 -- 172.21.15.57:0/1086730398 >> 172.21.15.57:0/1086730398 conn(0x7fc8dc069930 msgr2=0x7fc8dc069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:18.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.356+0000 7fc8e2444640 1 -- 172.21.15.57:0/1086730398 shutdown_connections 2023-10-19T12:45:18.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.356+0000 7fc8e2444640 1 -- 172.21.15.57:0/1086730398 wait complete. 2023-10-19T12:45:18.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.357+0000 7fc8e2444640 1 Processor -- start 2023-10-19T12:45:18.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.357+0000 7fc8e2444640 1 -- start start 2023-10-19T12:45:18.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.358+0000 7fc8e2444640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc8dc070af0 0x7fc8dc1e1280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:18.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.358+0000 7fc8e2444640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc8dc13e520 0x7fc8dc13b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:18.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.358+0000 7fc8e2444640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc8dc141690 0x7fc8dc1e3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:18.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.358+0000 7fc8e2444640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8dc1f0580 con 0x7fc8dc141690 2023-10-19T12:45:18.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.358+0000 7fc8e2444640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8dc1e17c0 con 0x7fc8dc070af0 2023-10-19T12:45:18.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.358+0000 7fc8e2444640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8dc13b760 con 0x7fc8dc13e520 2023-10-19T12:45:18.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.358+0000 7fc8e09e1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc8dc141690 0x7fc8dc1e3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:18.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.358+0000 7fc8e09e1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc8dc141690 0x7fc8dc1e3060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50250/0 (socket says 172.21.15.57:50250) 2023-10-19T12:45:18.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.358+0000 7fc8e09e1640 1 -- 172.21.15.57:0/1641367657 learned_addr learned my addr 172.21.15.57:0/1641367657 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:18.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.359+0000 7fc8dbfff640 1 --2- 172.21.15.57:0/1641367657 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc8dc070af0 0x7fc8dc1e1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:18.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.359+0000 7fc8db7fe640 1 --2- 172.21.15.57:0/1641367657 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc8dc13e520 0x7fc8dc13b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:18.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.359+0000 7fc8e09e1640 1 -- 172.21.15.57:0/1641367657 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc8dc13e520 msgr2=0x7fc8dc13b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:18.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.359+0000 7fc8e09e1640 1 --2- 172.21.15.57:0/1641367657 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc8dc13e520 0x7fc8dc13b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:18.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.359+0000 7fc8e09e1640 1 -- 172.21.15.57:0/1641367657 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc8dc070af0 msgr2=0x7fc8dc1e1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:18.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.359+0000 7fc8e09e1640 1 --2- 172.21.15.57:0/1641367657 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc8dc070af0 0x7fc8dc1e1280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:18.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.359+0000 7fc8e09e1640 1 -- 172.21.15.57:0/1641367657 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8d0009490 con 0x7fc8dc141690 2023-10-19T12:45:18.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.359+0000 7fc8dbfff640 1 --2- 172.21.15.57:0/1641367657 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc8dc070af0 0x7fc8dc1e1280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:45:18.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.359+0000 7fc8e09e1640 1 --2- 172.21.15.57:0/1641367657 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc8dc141690 0x7fc8dc1e3060 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fc8d400eb10 tx=0x7fc8d400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:18.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.359+0000 7fc8d8ff9640 1 -- 172.21.15.57:0/1641367657 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8d400ce80 con 0x7fc8dc141690 2023-10-19T12:45:18.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.359+0000 7fc8d8ff9640 1 -- 172.21.15.57:0/1641367657 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fc8d4004c10 con 0x7fc8dc141690 2023-10-19T12:45:18.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.359+0000 7fc8d8ff9640 1 -- 172.21.15.57:0/1641367657 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8d4005020 con 0x7fc8dc141690 2023-10-19T12:45:18.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.359+0000 7fc8e2444640 1 -- 172.21.15.57:0/1641367657 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8dc1efa00 con 0x7fc8dc141690 2023-10-19T12:45:18.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.359+0000 7fc8e2444640 1 -- 172.21.15.57:0/1641367657 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8dc1effc0 con 0x7fc8dc141690 2023-10-19T12:45:18.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.360+0000 7fc8e2444640 1 -- 172.21.15.57:0/1641367657 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8dc0b0fc0 con 0x7fc8dc141690 2023-10-19T12:45:18.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.360+0000 7fc8d8ff9640 1 -- 172.21.15.57:0/1641367657 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fc8d4005180 con 0x7fc8dc141690 2023-10-19T12:45:18.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.361+0000 7fc8d8ff9640 1 --2- 172.21.15.57:0/1641367657 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fc8c806ae40 0x7fc8c806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:18.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.361+0000 7fc8d8ff9640 1 -- 172.21.15.57:0/1641367657 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fc8d4014070 con 0x7fc8dc141690 2023-10-19T12:45:18.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.361+0000 7fc8dbfff640 1 --2- 172.21.15.57:0/1641367657 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fc8c806ae40 0x7fc8c806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:18.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.362+0000 7fc8dbfff640 1 --2- 172.21.15.57:0/1641367657 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fc8c806ae40 0x7fc8c806d300 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fc8d0036000 tx=0x7fc8d0004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:18.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.365+0000 7fc8d8ff9640 1 -- 172.21.15.57:0/1641367657 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc8d4055590 con 0x7fc8dc141690 2023-10-19T12:45:18.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.540+0000 7fc8e2444640 1 -- 172.21.15.57:0/1641367657 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc8dc0b2d10 con 0x7fc8dc141690 2023-10-19T12:45:18.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.540+0000 7fc8d8ff9640 1 -- 172.21.15.57:0/1641367657 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc8d401d020 con 0x7fc8dc141690 2023-10-19T12:45:18.544 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-19T12:45:18.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.543+0000 7fc8e2444640 1 -- 172.21.15.57:0/1641367657 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fc8c806ae40 msgr2=0x7fc8c806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:18.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.543+0000 7fc8e2444640 1 --2- 172.21.15.57:0/1641367657 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fc8c806ae40 0x7fc8c806d300 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fc8d0036000 tx=0x7fc8d0004730 comp rx=0 tx=0).stop 2023-10-19T12:45:18.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.543+0000 7fc8e2444640 1 -- 172.21.15.57:0/1641367657 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc8dc141690 msgr2=0x7fc8dc1e3060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:18.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.543+0000 7fc8e2444640 1 --2- 172.21.15.57:0/1641367657 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc8dc141690 0x7fc8dc1e3060 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fc8d400eb10 tx=0x7fc8d400c2d0 comp rx=0 tx=0).stop 2023-10-19T12:45:18.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.543+0000 7fc8e2444640 1 -- 172.21.15.57:0/1641367657 shutdown_connections 2023-10-19T12:45:18.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.543+0000 7fc8e2444640 1 --2- 172.21.15.57:0/1641367657 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fc8c806ae40 0x7fc8c806d300 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:18.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.543+0000 7fc8e2444640 1 --2- 172.21.15.57:0/1641367657 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc8dc141690 0x7fc8dc1e3060 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:18.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.543+0000 7fc8e2444640 1 --2- 172.21.15.57:0/1641367657 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc8dc13e520 0x7fc8dc13b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:18.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.543+0000 7fc8e2444640 1 --2- 172.21.15.57:0/1641367657 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc8dc070af0 0x7fc8dc1e1280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:18.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.543+0000 7fc8e2444640 1 -- 172.21.15.57:0/1641367657 >> 172.21.15.57:0/1641367657 conn(0x7fc8dc069930 msgr2=0x7fc8dc0acf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:18.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.543+0000 7fc8e2444640 1 -- 172.21.15.57:0/1641367657 shutdown_connections 2023-10-19T12:45:18.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.543+0000 7fc8e2444640 1 -- 172.21.15.57:0/1641367657 wait complete. 2023-10-19T12:45:18.557 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-19T12:45:18.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.665+0000 7f27d58bb640 1 -- 172.21.15.57:0/4174206449 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f27d006f0a0 msgr2=0x7f27d006dbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:18.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.665+0000 7f27d58bb640 1 --2- 172.21.15.57:0/4174206449 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f27d006f0a0 0x7f27d006dbf0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f27d0066970 tx=0x7f27b802bda0 comp rx=0 tx=0).stop 2023-10-19T12:45:18.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.665+0000 7f27d58bb640 1 -- 172.21.15.57:0/4174206449 shutdown_connections 2023-10-19T12:45:18.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.665+0000 7f27d58bb640 1 --2- 172.21.15.57:0/4174206449 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f27d006f0a0 0x7f27d006dbf0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:18.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.665+0000 7f27d58bb640 1 --2- 172.21.15.57:0/4174206449 >> v2:172.21.15.179:3300/0 conn(0x7f27d006d2d0 0x7f27d006d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:18.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.665+0000 7f27d58bb640 1 -- 172.21.15.57:0/4174206449 >> 172.21.15.57:0/4174206449 conn(0x7f27d00b0670 msgr2=0x7f27d00b0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:18.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.665+0000 7f27d58bb640 1 -- 172.21.15.57:0/4174206449 shutdown_connections 2023-10-19T12:45:18.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.665+0000 7f27d58bb640 1 -- 172.21.15.57:0/4174206449 wait complete. 2023-10-19T12:45:18.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.666+0000 7f27d58bb640 1 Processor -- start 2023-10-19T12:45:18.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.666+0000 7f27d58bb640 1 -- start start 2023-10-19T12:45:18.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.667+0000 7f27d58bb640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f27d006d2d0 0x7f27d00b3190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:18.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.667+0000 7f27d58bb640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f27d006f0a0 0x7f27d00cbc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:18.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.667+0000 7f27d58bb640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f27d00ab950 0x7f27d0144590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:18.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.667+0000 7f27d58bb640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27d00c4550 con 0x7f27d006f0a0 2023-10-19T12:45:18.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.667+0000 7f27d58bb640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27d00b4670 con 0x7f27d00ab950 2023-10-19T12:45:18.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.667+0000 7f27d58bb640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27d00b3af0 con 0x7f27d006d2d0 2023-10-19T12:45:18.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.667+0000 7f27ce7fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f27d006f0a0 0x7f27d00cbc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:18.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.667+0000 7f27ce7fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f27d006f0a0 0x7f27d00cbc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50274/0 (socket says 172.21.15.57:50274) 2023-10-19T12:45:18.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.667+0000 7f27ce7fc640 1 -- 172.21.15.57:0/1584771362 learned_addr learned my addr 172.21.15.57:0/1584771362 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:18.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.667+0000 7f27cf7fe640 1 --2- 172.21.15.57:0/1584771362 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f27d00ab950 0x7f27d0144590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:18.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.667+0000 7f27ceffd640 1 --2- 172.21.15.57:0/1584771362 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f27d006d2d0 0x7f27d00b3190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:18.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.667+0000 7f27ce7fc640 1 -- 172.21.15.57:0/1584771362 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f27d006d2d0 msgr2=0x7f27d00b3190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:18.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.667+0000 7f27ce7fc640 1 --2- 172.21.15.57:0/1584771362 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f27d006d2d0 0x7f27d00b3190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:18.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.667+0000 7f27ce7fc640 1 -- 172.21.15.57:0/1584771362 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f27d00ab950 msgr2=0x7f27d0144590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:18.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.667+0000 7f27ce7fc640 1 --2- 172.21.15.57:0/1584771362 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f27d00ab950 0x7f27d0144590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:18.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.667+0000 7f27ce7fc640 1 -- 172.21.15.57:0/1584771362 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27b8009560 con 0x7f27d006f0a0 2023-10-19T12:45:18.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.667+0000 7f27ceffd640 1 --2- 172.21.15.57:0/1584771362 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f27d006d2d0 0x7f27d00b3190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:45:18.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.667+0000 7f27ce7fc640 1 --2- 172.21.15.57:0/1584771362 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f27d006f0a0 0x7f27d00cbc80 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f27b8000c00 tx=0x7f27b802dd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:18.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.668+0000 7f27c7fff640 1 -- 172.21.15.57:0/1584771362 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27b8005670 con 0x7f27d006f0a0 2023-10-19T12:45:18.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.668+0000 7f27d58bb640 1 -- 172.21.15.57:0/1584771362 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27d00b3dd0 con 0x7f27d006f0a0 2023-10-19T12:45:18.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.668+0000 7f27c7fff640 1 -- 172.21.15.57:0/1584771362 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f27b80038f0 con 0x7f27d006f0a0 2023-10-19T12:45:18.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.668+0000 7f27c7fff640 1 -- 172.21.15.57:0/1584771362 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27b8003ce0 con 0x7f27d006f0a0 2023-10-19T12:45:18.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.668+0000 7f27d58bb640 1 -- 172.21.15.57:0/1584771362 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27d00b4390 con 0x7f27d006f0a0 2023-10-19T12:45:18.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.668+0000 7f27d58bb640 1 -- 172.21.15.57:0/1584771362 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27d00ce1d0 con 0x7f27d006f0a0 2023-10-19T12:45:18.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.669+0000 7f27c7fff640 1 -- 172.21.15.57:0/1584771362 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f27b803e450 con 0x7f27d006f0a0 2023-10-19T12:45:18.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.669+0000 7f27c7fff640 1 --2- 172.21.15.57:0/1584771362 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f27b006ae40 0x7f27b006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:18.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.669+0000 7f27c7fff640 1 -- 172.21.15.57:0/1584771362 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f27b80ac460 con 0x7f27d006f0a0 2023-10-19T12:45:18.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.670+0000 7f27ceffd640 1 --2- 172.21.15.57:0/1584771362 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f27b006ae40 0x7f27b006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:18.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.670+0000 7f27ceffd640 1 --2- 172.21.15.57:0/1584771362 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f27b006ae40 0x7f27b006d300 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f27c0007e70 tx=0x7f27c0007850 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:18.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.673+0000 7f27c7fff640 1 -- 172.21.15.57:0/1584771362 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f27b80771f0 con 0x7f27d006f0a0 2023-10-19T12:45:18.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.846+0000 7f27d58bb640 1 -- 172.21.15.57:0/1584771362 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f27d0142bb0 con 0x7f27d006f0a0 2023-10-19T12:45:18.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.847+0000 7f27c7fff640 1 -- 172.21.15.57:0/1584771362 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7f27d0142bb0 con 0x7f27d006f0a0 2023-10-19T12:45:18.850 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2023-10-19T12:45:18.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.848+0000 7f27d58bb640 1 -- 172.21.15.57:0/1584771362 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f27b006ae40 msgr2=0x7f27b006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:18.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.848+0000 7f27d58bb640 1 --2- 172.21.15.57:0/1584771362 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f27b006ae40 0x7f27b006d300 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f27c0007e70 tx=0x7f27c0007850 comp rx=0 tx=0).stop 2023-10-19T12:45:18.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.848+0000 7f27d58bb640 1 -- 172.21.15.57:0/1584771362 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f27d006f0a0 msgr2=0x7f27d00cbc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:18.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.848+0000 7f27d58bb640 1 --2- 172.21.15.57:0/1584771362 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f27d006f0a0 0x7f27d00cbc80 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f27b8000c00 tx=0x7f27b802dd70 comp rx=0 tx=0).stop 2023-10-19T12:45:18.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.849+0000 7f27d58bb640 1 -- 172.21.15.57:0/1584771362 shutdown_connections 2023-10-19T12:45:18.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.849+0000 7f27d58bb640 1 --2- 172.21.15.57:0/1584771362 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f27b006ae40 0x7f27b006d300 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:18.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.849+0000 7f27d58bb640 1 --2- 172.21.15.57:0/1584771362 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f27d00ab950 0x7f27d0144590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:18.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.849+0000 7f27d58bb640 1 --2- 172.21.15.57:0/1584771362 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f27d006f0a0 0x7f27d00cbc80 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:18.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.849+0000 7f27d58bb640 1 --2- 172.21.15.57:0/1584771362 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f27d006d2d0 0x7f27d00b3190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:18.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.849+0000 7f27d58bb640 1 -- 172.21.15.57:0/1584771362 >> 172.21.15.57:0/1584771362 conn(0x7f27d00b0670 msgr2=0x7f27d00ce9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:18.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.849+0000 7f27d58bb640 1 -- 172.21.15.57:0/1584771362 shutdown_connections 2023-10-19T12:45:18.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.849+0000 7f27d58bb640 1 -- 172.21.15.57:0/1584771362 wait complete. 2023-10-19T12:45:18.863 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-19T12:45:18.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.971+0000 7fe94e7a0640 1 -- 172.21.15.57:0/523461235 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe9480d1740 msgr2=0x7fe9480cd890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:18.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.971+0000 7fe94e7a0640 1 --2- 172.21.15.57:0/523461235 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe9480d1740 0x7fe9480cd890 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fe944005fd0 tx=0x7fe94402c0e0 comp rx=0 tx=0).stop 2023-10-19T12:45:18.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.971+0000 7fe94e7a0640 1 -- 172.21.15.57:0/523461235 shutdown_connections 2023-10-19T12:45:18.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.971+0000 7fe94e7a0640 1 --2- 172.21.15.57:0/523461235 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe9480d1740 0x7fe9480cd890 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:18.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.971+0000 7fe94e7a0640 1 --2- 172.21.15.57:0/523461235 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe94815b2f0 0x7fe9480cd350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:18.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.971+0000 7fe94e7a0640 1 -- 172.21.15.57:0/523461235 >> 172.21.15.57:0/523461235 conn(0x7fe9480d0c20 msgr2=0x7fe9480d1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:18.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.972+0000 7fe94e7a0640 1 -- 172.21.15.57:0/523461235 shutdown_connections 2023-10-19T12:45:18.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.972+0000 7fe94e7a0640 1 -- 172.21.15.57:0/523461235 wait complete. 2023-10-19T12:45:18.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.973+0000 7fe94e7a0640 1 Processor -- start 2023-10-19T12:45:18.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.973+0000 7fe94e7a0640 1 -- start start 2023-10-19T12:45:18.979 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.973+0000 7fe94e7a0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe9480ce2f0 0x7fe948150a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:18.979 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.973+0000 7fe94e7a0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe9480d1740 0x7fe9480b6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:18.979 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.973+0000 7fe94e7a0640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe94815b2f0 0x7fe9480b26f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:18.979 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.973+0000 7fe94e7a0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe948150f80 con 0x7fe9480ce2f0 2023-10-19T12:45:18.979 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.973+0000 7fe94e7a0640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9481516c0 con 0x7fe9480d1740 2023-10-19T12:45:18.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.973+0000 7fe94e7a0640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9481519a0 con 0x7fe94815b2f0 2023-10-19T12:45:18.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.973+0000 7fe94d79e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe9480ce2f0 0x7fe948150a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:18.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.974+0000 7fe94d79e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe9480ce2f0 0x7fe948150a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50296/0 (socket says 172.21.15.57:50296) 2023-10-19T12:45:18.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.974+0000 7fe94d79e640 1 -- 172.21.15.57:0/1326230844 learned_addr learned my addr 172.21.15.57:0/1326230844 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:18.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.974+0000 7fe94df9f640 1 --2- 172.21.15.57:0/1326230844 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe94815b2f0 0x7fe9480b26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:18.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.974+0000 7fe94cf9d640 1 --2- 172.21.15.57:0/1326230844 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe9480d1740 0x7fe9480b6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:18.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.974+0000 7fe94d79e640 1 -- 172.21.15.57:0/1326230844 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe94815b2f0 msgr2=0x7fe9480b26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:18.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.974+0000 7fe94d79e640 1 --2- 172.21.15.57:0/1326230844 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe94815b2f0 0x7fe9480b26f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:18.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.974+0000 7fe94d79e640 1 -- 172.21.15.57:0/1326230844 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe9480d1740 msgr2=0x7fe9480b6320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:18.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.974+0000 7fe94d79e640 1 --2- 172.21.15.57:0/1326230844 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe9480d1740 0x7fe9480b6320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:18.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.974+0000 7fe94d79e640 1 -- 172.21.15.57:0/1326230844 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe944009840 con 0x7fe9480ce2f0 2023-10-19T12:45:18.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.974+0000 7fe94df9f640 1 --2- 172.21.15.57:0/1326230844 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe94815b2f0 0x7fe9480b26f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:45:18.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.974+0000 7fe94cf9d640 1 --2- 172.21.15.57:0/1326230844 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe9480d1740 0x7fe9480b6320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:45:18.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.974+0000 7fe94d79e640 1 --2- 172.21.15.57:0/1326230844 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe9480ce2f0 0x7fe948150a40 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fe94000d950 tx=0x7fe94000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:18.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.974+0000 7fe93dffb640 1 -- 172.21.15.57:0/1326230844 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe940009bd0 con 0x7fe9480ce2f0 2023-10-19T12:45:18.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.974+0000 7fe93dffb640 1 -- 172.21.15.57:0/1326230844 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fe940009e20 con 0x7fe9480ce2f0 2023-10-19T12:45:18.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.974+0000 7fe94e7a0640 1 -- 172.21.15.57:0/1326230844 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe948151c80 con 0x7fe9480ce2f0 2023-10-19T12:45:18.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.974+0000 7fe93dffb640 1 -- 172.21.15.57:0/1326230844 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9400106c0 con 0x7fe9480ce2f0 2023-10-19T12:45:18.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.974+0000 7fe94e7a0640 1 -- 172.21.15.57:0/1326230844 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe948152240 con 0x7fe9480ce2f0 2023-10-19T12:45:18.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.975+0000 7fe94e7a0640 1 -- 172.21.15.57:0/1326230844 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe948066960 con 0x7fe9480ce2f0 2023-10-19T12:45:18.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.976+0000 7fe93dffb640 1 -- 172.21.15.57:0/1326230844 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fe940010820 con 0x7fe9480ce2f0 2023-10-19T12:45:18.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.976+0000 7fe93dffb640 1 --2- 172.21.15.57:0/1326230844 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fe92c06ae70 0x7fe92c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:18.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.976+0000 7fe93dffb640 1 -- 172.21.15.57:0/1326230844 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fe940089d90 con 0x7fe9480ce2f0 2023-10-19T12:45:18.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.976+0000 7fe94cf9d640 1 --2- 172.21.15.57:0/1326230844 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fe92c06ae70 0x7fe92c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:18.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.977+0000 7fe94cf9d640 1 --2- 172.21.15.57:0/1326230844 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fe92c06ae70 0x7fe92c06d330 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fe9480c6ca0 tx=0x7fe938009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:18.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:18.980+0000 7fe93dffb640 1 -- 172.21.15.57:0/1326230844 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe940054b10 con 0x7fe9480ce2f0 2023-10-19T12:45:19.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.156+0000 7fe94e7a0640 1 -- 172.21.15.57:0/1326230844 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe9481523d0 con 0x7fe9480ce2f0 2023-10-19T12:45:19.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.157+0000 7fe93dffb640 1 -- 172.21.15.57:0/1326230844 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe9400170a0 con 0x7fe9480ce2f0 2023-10-19T12:45:19.160 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 64 2023-10-19T12:45:19.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.159+0000 7fe94e7a0640 1 -- 172.21.15.57:0/1326230844 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fe92c06ae70 msgr2=0x7fe92c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:19.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.159+0000 7fe94e7a0640 1 --2- 172.21.15.57:0/1326230844 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fe92c06ae70 0x7fe92c06d330 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fe9480c6ca0 tx=0x7fe938009040 comp rx=0 tx=0).stop 2023-10-19T12:45:19.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.159+0000 7fe94e7a0640 1 -- 172.21.15.57:0/1326230844 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe9480ce2f0 msgr2=0x7fe948150a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:19.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.159+0000 7fe94e7a0640 1 --2- 172.21.15.57:0/1326230844 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe9480ce2f0 0x7fe948150a40 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fe94000d950 tx=0x7fe94000de10 comp rx=0 tx=0).stop 2023-10-19T12:45:19.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.159+0000 7fe94e7a0640 1 -- 172.21.15.57:0/1326230844 shutdown_connections 2023-10-19T12:45:19.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.159+0000 7fe94e7a0640 1 --2- 172.21.15.57:0/1326230844 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fe92c06ae70 0x7fe92c06d330 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:19.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.159+0000 7fe94e7a0640 1 --2- 172.21.15.57:0/1326230844 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe94815b2f0 0x7fe9480b26f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:19.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.159+0000 7fe94e7a0640 1 --2- 172.21.15.57:0/1326230844 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe9480d1740 0x7fe9480b6320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:19.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.159+0000 7fe94e7a0640 1 --2- 172.21.15.57:0/1326230844 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe9480ce2f0 0x7fe948150a40 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:19.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.159+0000 7fe94e7a0640 1 -- 172.21.15.57:0/1326230844 >> 172.21.15.57:0/1326230844 conn(0x7fe9480d0c20 msgr2=0x7fe9480c6440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:19.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.159+0000 7fe94e7a0640 1 -- 172.21.15.57:0/1326230844 shutdown_connections 2023-10-19T12:45:19.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.159+0000 7fe94e7a0640 1 -- 172.21.15.57:0/1326230844 wait complete. 2023-10-19T12:45:19.173 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-19T12:45:19.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.283+0000 7f33fa6a9640 1 -- 172.21.15.57:0/1920458489 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f33f406a730 msgr2=0x7f33f406abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:19.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.283+0000 7f33fa6a9640 1 --2- 172.21.15.57:0/1920458489 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f33f406a730 0x7f33f406abc0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f33ec009ca0 tx=0x7f33ec02c230 comp rx=0 tx=0).stop 2023-10-19T12:45:19.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.283+0000 7f33fa6a9640 1 -- 172.21.15.57:0/1920458489 shutdown_connections 2023-10-19T12:45:19.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.283+0000 7f33fa6a9640 1 --2- 172.21.15.57:0/1920458489 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f33f406a730 0x7f33f406abc0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:19.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.283+0000 7f33fa6a9640 1 --2- 172.21.15.57:0/1920458489 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f33f4156790 0x7f33f40cc990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:19.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.283+0000 7f33fa6a9640 1 -- 172.21.15.57:0/1920458489 >> 172.21.15.57:0/1920458489 conn(0x7f33f406fae0 msgr2=0x7f33f406fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:19.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.283+0000 7f33fa6a9640 1 -- 172.21.15.57:0/1920458489 shutdown_connections 2023-10-19T12:45:19.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.283+0000 7f33fa6a9640 1 -- 172.21.15.57:0/1920458489 wait complete. 2023-10-19T12:45:19.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.284+0000 7f33fa6a9640 1 Processor -- start 2023-10-19T12:45:19.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.284+0000 7f33fa6a9640 1 -- start start 2023-10-19T12:45:19.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.285+0000 7f33fa6a9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f33f406a730 0x7f33f41f1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:19.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.285+0000 7f33fa6a9640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f33f40ac870 0x7f33f41f2f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:19.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.285+0000 7f33fa6a9640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f33f4156790 0x7f33f4157cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:19.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.285+0000 7f33fa6a9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33f414ee40 con 0x7f33f406a730 2023-10-19T12:45:19.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.285+0000 7f33fa6a9640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33f41f2300 con 0x7f33f40ac870 2023-10-19T12:45:19.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.285+0000 7f33fa6a9640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33f40ab320 con 0x7f33f4156790 2023-10-19T12:45:19.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.285+0000 7f33f3fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f33f406a730 0x7f33f41f1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:19.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.285+0000 7f33f3fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f33f406a730 0x7f33f41f1dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50312/0 (socket says 172.21.15.57:50312) 2023-10-19T12:45:19.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.285+0000 7f33f3fff640 1 -- 172.21.15.57:0/2263006294 learned_addr learned my addr 172.21.15.57:0/2263006294 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:19.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.285+0000 7f33f37fe640 1 --2- 172.21.15.57:0/2263006294 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f33f40ac870 0x7f33f41f2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:19.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.285+0000 7f33f8c46640 1 --2- 172.21.15.57:0/2263006294 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f33f4156790 0x7f33f4157cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:19.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.285+0000 7f33f3fff640 1 -- 172.21.15.57:0/2263006294 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f33f4156790 msgr2=0x7f33f4157cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:19.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.285+0000 7f33f3fff640 1 --2- 172.21.15.57:0/2263006294 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f33f4156790 0x7f33f4157cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:19.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.285+0000 7f33f3fff640 1 -- 172.21.15.57:0/2263006294 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f33f40ac870 msgr2=0x7f33f41f2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:19.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.285+0000 7f33f3fff640 1 --2- 172.21.15.57:0/2263006294 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f33f40ac870 0x7f33f41f2f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:19.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.285+0000 7f33f3fff640 1 -- 172.21.15.57:0/2263006294 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33ec009900 con 0x7f33f406a730 2023-10-19T12:45:19.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.285+0000 7f33f3fff640 1 --2- 172.21.15.57:0/2263006294 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f33f406a730 0x7f33f41f1dc0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f33e800c9a0 tx=0x7f33e800ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:19.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.286+0000 7f33f0ff9640 1 -- 172.21.15.57:0/2263006294 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33e8007b50 con 0x7f33f406a730 2023-10-19T12:45:19.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.286+0000 7f33f0ff9640 1 -- 172.21.15.57:0/2263006294 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f33e8007dc0 con 0x7f33f406a730 2023-10-19T12:45:19.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.286+0000 7f33f0ff9640 1 -- 172.21.15.57:0/2263006294 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33e8005280 con 0x7f33f406a730 2023-10-19T12:45:19.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.286+0000 7f33fa6a9640 1 -- 172.21.15.57:0/2263006294 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33f41415a0 con 0x7f33f406a730 2023-10-19T12:45:19.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.286+0000 7f33fa6a9640 1 -- 172.21.15.57:0/2263006294 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33f40a8390 con 0x7f33f406a730 2023-10-19T12:45:19.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.287+0000 7f33fa6a9640 1 -- 172.21.15.57:0/2263006294 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33f40690e0 con 0x7f33f406a730 2023-10-19T12:45:19.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.287+0000 7f33f0ff9640 1 -- 172.21.15.57:0/2263006294 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f33e80053e0 con 0x7f33f406a730 2023-10-19T12:45:19.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.287+0000 7f33f0ff9640 1 --2- 172.21.15.57:0/2263006294 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f33e006ae40 0x7f33e006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:19.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.288+0000 7f33f0ff9640 1 -- 172.21.15.57:0/2263006294 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f33e8089ee0 con 0x7f33f406a730 2023-10-19T12:45:19.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.288+0000 7f33f37fe640 1 --2- 172.21.15.57:0/2263006294 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f33e006ae40 0x7f33e006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:19.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.288+0000 7f33f37fe640 1 --2- 172.21.15.57:0/2263006294 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f33e006ae40 0x7f33e006d300 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f33f40a7db0 tx=0x7f33e4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:19.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.292+0000 7f33f0ff9640 1 -- 172.21.15.57:0/2263006294 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f33e8054c60 con 0x7f33f406a730 2023-10-19T12:45:19.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.467+0000 7f33fa6a9640 1 -- 172.21.15.57:0/2263006294 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f33f415d1c0 con 0x7f33f406a730 2023-10-19T12:45:19.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.467+0000 7f33f0ff9640 1 -- 172.21.15.57:0/2263006294 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7f33e801d070 con 0x7f33f406a730 2023-10-19T12:45:19.471 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2023-10-19T12:45:19.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.469+0000 7f33fa6a9640 1 -- 172.21.15.57:0/2263006294 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f33e006ae40 msgr2=0x7f33e006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:19.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.469+0000 7f33fa6a9640 1 --2- 172.21.15.57:0/2263006294 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f33e006ae40 0x7f33e006d300 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f33f40a7db0 tx=0x7f33e4009040 comp rx=0 tx=0).stop 2023-10-19T12:45:19.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.469+0000 7f33fa6a9640 1 -- 172.21.15.57:0/2263006294 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f33f406a730 msgr2=0x7f33f41f1dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:19.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.469+0000 7f33fa6a9640 1 --2- 172.21.15.57:0/2263006294 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f33f406a730 0x7f33f41f1dc0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f33e800c9a0 tx=0x7f33e800ce60 comp rx=0 tx=0).stop 2023-10-19T12:45:19.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.469+0000 7f33fa6a9640 1 -- 172.21.15.57:0/2263006294 shutdown_connections 2023-10-19T12:45:19.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.469+0000 7f33fa6a9640 1 --2- 172.21.15.57:0/2263006294 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f33e006ae40 0x7f33e006d300 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:19.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.469+0000 7f33fa6a9640 1 --2- 172.21.15.57:0/2263006294 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f33f4156790 0x7f33f4157cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:19.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.469+0000 7f33fa6a9640 1 --2- 172.21.15.57:0/2263006294 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f33f40ac870 0x7f33f41f2f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:19.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.469+0000 7f33fa6a9640 1 --2- 172.21.15.57:0/2263006294 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f33f406a730 0x7f33f41f1dc0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:19.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.469+0000 7f33fa6a9640 1 -- 172.21.15.57:0/2263006294 >> 172.21.15.57:0/2263006294 conn(0x7f33f406fae0 msgr2=0x7f33f4155ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:19.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.469+0000 7f33fa6a9640 1 -- 172.21.15.57:0/2263006294 shutdown_connections 2023-10-19T12:45:19.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.469+0000 7f33fa6a9640 1 -- 172.21.15.57:0/2263006294 wait complete. 2023-10-19T12:45:19.483 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-19T12:45:19.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.590+0000 7f52041b6640 1 -- 172.21.15.57:0/1871785766 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51fc06af00 msgr2=0x7f51fc06b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:19.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.590+0000 7f52041b6640 1 --2- 172.21.15.57:0/1871785766 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51fc06af00 0x7f51fc06b2e0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f51fc06d590 tx=0x7f51f802bda0 comp rx=0 tx=0).stop 2023-10-19T12:45:19.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.590+0000 7f52041b6640 1 -- 172.21.15.57:0/1871785766 shutdown_connections 2023-10-19T12:45:19.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.590+0000 7f52041b6640 1 --2- 172.21.15.57:0/1871785766 >> v2:172.21.15.179:3300/0 conn(0x7f51fc0ab500 0x7f51fc0ab990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:19.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.590+0000 7f52041b6640 1 --2- 172.21.15.57:0/1871785766 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f51fc0c96b0 0x7f51fc0c9b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:19.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.590+0000 7f52041b6640 1 --2- 172.21.15.57:0/1871785766 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51fc06af00 0x7f51fc06b2e0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:19.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.590+0000 7f52041b6640 1 -- 172.21.15.57:0/1871785766 >> 172.21.15.57:0/1871785766 conn(0x7f51fc069930 msgr2=0x7f51fc069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:19.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.591+0000 7f52041b6640 1 -- 172.21.15.57:0/1871785766 shutdown_connections 2023-10-19T12:45:19.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.591+0000 7f52041b6640 1 -- 172.21.15.57:0/1871785766 wait complete. 2023-10-19T12:45:19.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.592+0000 7f52041b6640 1 Processor -- start 2023-10-19T12:45:19.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.592+0000 7f52041b6640 1 -- start start 2023-10-19T12:45:19.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.592+0000 7f52041b6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51fc06af00 0x7f51fc139290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:19.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.593+0000 7f52041b6640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f51fc0ab500 0x7f51fc0b0710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:19.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.593+0000 7f52041b6640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f51fc0c96b0 0x7f51fc0aa8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:19.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.593+0000 7f52041b6640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51fc06f1b0 con 0x7f51fc06af00 2023-10-19T12:45:19.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.593+0000 7f52041b6640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51fc0b11d0 con 0x7f51fc0c96b0 2023-10-19T12:45:19.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.593+0000 7f52041b6640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51fc0b14b0 con 0x7f51fc0ab500 2023-10-19T12:45:19.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.593+0000 7f5201f52640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51fc06af00 0x7f51fc139290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:19.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.593+0000 7f5201f52640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51fc06af00 0x7f51fc139290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50338/0 (socket says 172.21.15.57:50338) 2023-10-19T12:45:19.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.593+0000 7f5201f52640 1 -- 172.21.15.57:0/3144833553 learned_addr learned my addr 172.21.15.57:0/3144833553 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:19.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.593+0000 7f5202753640 1 --2- 172.21.15.57:0/3144833553 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f51fc0c96b0 0x7f51fc0aa8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:19.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.593+0000 7f5201751640 1 --2- 172.21.15.57:0/3144833553 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f51fc0ab500 0x7f51fc0b0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:19.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.593+0000 7f5201f52640 1 -- 172.21.15.57:0/3144833553 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f51fc0ab500 msgr2=0x7f51fc0b0710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:19.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.593+0000 7f5201f52640 1 --2- 172.21.15.57:0/3144833553 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f51fc0ab500 0x7f51fc0b0710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:19.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.593+0000 7f5201f52640 1 -- 172.21.15.57:0/3144833553 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f51fc0c96b0 msgr2=0x7f51fc0aa8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:19.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.593+0000 7f5201f52640 1 --2- 172.21.15.57:0/3144833553 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f51fc0c96b0 0x7f51fc0aa8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:19.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.593+0000 7f5201f52640 1 -- 172.21.15.57:0/3144833553 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51f8009560 con 0x7f51fc06af00 2023-10-19T12:45:19.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.593+0000 7f5201751640 1 --2- 172.21.15.57:0/3144833553 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f51fc0ab500 0x7f51fc0b0710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:45:19.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.593+0000 7f5201f52640 1 --2- 172.21.15.57:0/3144833553 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51fc06af00 0x7f51fc139290 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f51f8036040 tx=0x7f51f802dd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:19.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.593+0000 7f51f27fc640 1 -- 172.21.15.57:0/3144833553 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51f8005d50 con 0x7f51fc06af00 2023-10-19T12:45:19.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.593+0000 7f51f27fc640 1 -- 172.21.15.57:0/3144833553 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f51f8003910 con 0x7f51fc06af00 2023-10-19T12:45:19.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.593+0000 7f52041b6640 1 -- 172.21.15.57:0/3144833553 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51fc0b1790 con 0x7f51fc06af00 2023-10-19T12:45:19.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.593+0000 7f51f27fc640 1 -- 172.21.15.57:0/3144833553 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51f8003d20 con 0x7f51fc06af00 2023-10-19T12:45:19.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.593+0000 7f52041b6640 1 -- 172.21.15.57:0/3144833553 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51fc158c70 con 0x7f51fc06af00 2023-10-19T12:45:19.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.594+0000 7f52041b6640 1 -- 172.21.15.57:0/3144833553 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51fc0690e0 con 0x7f51fc06af00 2023-10-19T12:45:19.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.595+0000 7f51f27fc640 1 -- 172.21.15.57:0/3144833553 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f51f803e450 con 0x7f51fc06af00 2023-10-19T12:45:19.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.595+0000 7f51f27fc640 1 --2- 172.21.15.57:0/3144833553 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f51e006ae40 0x7f51e006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:19.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.595+0000 7f51f27fc640 1 -- 172.21.15.57:0/3144833553 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f51f80ac2d0 con 0x7f51fc06af00 2023-10-19T12:45:19.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.595+0000 7f5201751640 1 --2- 172.21.15.57:0/3144833553 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f51e006ae40 0x7f51e006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:19.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.596+0000 7f5201751640 1 --2- 172.21.15.57:0/3144833553 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f51e006ae40 0x7f51e006d300 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f51fc06a0c0 tx=0x7f51ec005d30 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:19.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.599+0000 7f51f27fc640 1 -- 172.21.15.57:0/3144833553 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f51f80770e0 con 0x7f51fc06af00 2023-10-19T12:45:19.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.788+0000 7f52041b6640 1 -- 172.21.15.57:0/3144833553 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f51fc158f90 con 0x7f51fc06af00 2023-10-19T12:45:19.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.788+0000 7f51f27fc640 1 -- 172.21.15.57:0/3144833553 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) v1 ==== 80+0+0 (secure 0 0 0) 0x7f51f807a700 con 0x7f51fc06af00 2023-10-19T12:45:19.792 INFO:teuthology.orchestra.run.smithi057.stderr:listed 0 entries 2023-10-19T12:45:19.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.790+0000 7f52041b6640 1 -- 172.21.15.57:0/3144833553 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f51e006ae40 msgr2=0x7f51e006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:19.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.790+0000 7f52041b6640 1 --2- 172.21.15.57:0/3144833553 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f51e006ae40 0x7f51e006d300 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f51fc06a0c0 tx=0x7f51ec005d30 comp rx=0 tx=0).stop 2023-10-19T12:45:19.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.790+0000 7f52041b6640 1 -- 172.21.15.57:0/3144833553 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51fc06af00 msgr2=0x7f51fc139290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:19.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.790+0000 7f52041b6640 1 --2- 172.21.15.57:0/3144833553 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51fc06af00 0x7f51fc139290 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f51f8036040 tx=0x7f51f802dd70 comp rx=0 tx=0).stop 2023-10-19T12:45:19.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.790+0000 7f52041b6640 1 -- 172.21.15.57:0/3144833553 shutdown_connections 2023-10-19T12:45:19.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.790+0000 7f52041b6640 1 --2- 172.21.15.57:0/3144833553 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f51e006ae40 0x7f51e006d300 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:19.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.790+0000 7f52041b6640 1 --2- 172.21.15.57:0/3144833553 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f51fc0c96b0 0x7f51fc0aa8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:19.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.790+0000 7f52041b6640 1 --2- 172.21.15.57:0/3144833553 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f51fc0ab500 0x7f51fc0b0710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:19.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.790+0000 7f52041b6640 1 --2- 172.21.15.57:0/3144833553 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51fc06af00 0x7f51fc139290 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:19.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.790+0000 7f52041b6640 1 -- 172.21.15.57:0/3144833553 >> 172.21.15.57:0/3144833553 conn(0x7f51fc069930 msgr2=0x7f51fc13b260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:19.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.790+0000 7f52041b6640 1 -- 172.21.15.57:0/3144833553 shutdown_connections 2023-10-19T12:45:19.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.790+0000 7f52041b6640 1 -- 172.21.15.57:0/3144833553 wait complete. 2023-10-19T12:45:19.804 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T12:45:19.804 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T12:45:19.820 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T12:45:19.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.966+0000 7fc14056e640 1 -- 172.21.15.57:0/1821599828 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc138070af0 msgr2=0x7fc1381628b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:19.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.966+0000 7fc14056e640 1 --2- 172.21.15.57:0/1821599828 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc138070af0 0x7fc1381628b0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fc130005fd0 tx=0x7fc13002c2e0 comp rx=0 tx=0).stop 2023-10-19T12:45:19.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.966+0000 7fc14056e640 1 -- 172.21.15.57:0/1821599828 shutdown_connections 2023-10-19T12:45:19.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.966+0000 7fc14056e640 1 --2- 172.21.15.57:0/1821599828 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc138070af0 0x7fc1381628b0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:19.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.966+0000 7fc14056e640 1 -- 172.21.15.57:0/1821599828 >> 172.21.15.57:0/1821599828 conn(0x7fc138069930 msgr2=0x7fc138069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:19.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.966+0000 7fc14056e640 1 -- 172.21.15.57:0/1821599828 shutdown_connections 2023-10-19T12:45:19.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.967+0000 7fc14056e640 1 -- 172.21.15.57:0/1821599828 wait complete. 2023-10-19T12:45:19.973 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.968+0000 7fc14056e640 1 Processor -- start 2023-10-19T12:45:19.974 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.968+0000 7fc14056e640 1 -- start start 2023-10-19T12:45:19.974 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.968+0000 7fc14056e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc138070af0 0x7fc13815ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:19.974 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.968+0000 7fc14056e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc13813e520 0x7fc138159e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:19.974 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.968+0000 7fc14056e640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc138141690 0x7fc13813a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:19.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.968+0000 7fc14056e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc138158df0 con 0x7fc13813e520 2023-10-19T12:45:19.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.968+0000 7fc14056e640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1381590d0 con 0x7fc138070af0 2023-10-19T12:45:19.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.968+0000 7fc14056e640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1381593b0 con 0x7fc138141690 2023-10-19T12:45:19.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.968+0000 7fc13db09640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc13813e520 0x7fc138159e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:19.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.968+0000 7fc13db09640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc13813e520 0x7fc138159e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50358/0 (socket says 172.21.15.57:50358) 2023-10-19T12:45:19.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.968+0000 7fc13db09640 1 -- 172.21.15.57:0/838171229 learned_addr learned my addr 172.21.15.57:0/838171229 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:19.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.969+0000 7fc13e30a640 1 --2- 172.21.15.57:0/838171229 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc138070af0 0x7fc13815ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:19.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.969+0000 7fc13eb0b640 1 --2- 172.21.15.57:0/838171229 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc138141690 0x7fc13813a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:19.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.969+0000 7fc13db09640 1 -- 172.21.15.57:0/838171229 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc138141690 msgr2=0x7fc13813a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:19.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.969+0000 7fc13db09640 1 --2- 172.21.15.57:0/838171229 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc138141690 0x7fc13813a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:19.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.969+0000 7fc13db09640 1 -- 172.21.15.57:0/838171229 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc138070af0 msgr2=0x7fc13815ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:19.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.969+0000 7fc13db09640 1 --2- 172.21.15.57:0/838171229 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc138070af0 0x7fc13815ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:19.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.969+0000 7fc13db09640 1 -- 172.21.15.57:0/838171229 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc130009900 con 0x7fc13813e520 2023-10-19T12:45:19.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.969+0000 7fc13e30a640 1 --2- 172.21.15.57:0/838171229 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc138070af0 0x7fc13815ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:45:19.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.969+0000 7fc13eb0b640 1 --2- 172.21.15.57:0/838171229 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc138141690 0x7fc13813a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:45:19.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.969+0000 7fc13db09640 1 --2- 172.21.15.57:0/838171229 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc13813e520 0x7fc138159e90 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fc12800c400 tx=0x7fc12800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:19.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.969+0000 7fc12effd640 1 -- 172.21.15.57:0/838171229 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1280052d0 con 0x7fc13813e520 2023-10-19T12:45:19.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.969+0000 7fc12effd640 1 -- 172.21.15.57:0/838171229 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fc1280035f0 con 0x7fc13813e520 2023-10-19T12:45:19.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.969+0000 7fc14056e640 1 -- 172.21.15.57:0/838171229 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc138159690 con 0x7fc13813e520 2023-10-19T12:45:19.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.969+0000 7fc12effd640 1 -- 172.21.15.57:0/838171229 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc128003a00 con 0x7fc13813e520 2023-10-19T12:45:19.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.969+0000 7fc14056e640 1 -- 172.21.15.57:0/838171229 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc13815a3d0 con 0x7fc13813e520 2023-10-19T12:45:19.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.970+0000 7fc14056e640 1 -- 172.21.15.57:0/838171229 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc138159bb0 con 0x7fc13813e520 2023-10-19T12:45:19.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.971+0000 7fc12effd640 1 -- 172.21.15.57:0/838171229 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fc128003b60 con 0x7fc13813e520 2023-10-19T12:45:19.979 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.971+0000 7fc12effd640 1 --2- 172.21.15.57:0/838171229 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fc11c06ae70 0x7fc11c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:19.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.971+0000 7fc12effd640 1 -- 172.21.15.57:0/838171229 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fc1280890d0 con 0x7fc13813e520 2023-10-19T12:45:19.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.971+0000 7fc13e30a640 1 --2- 172.21.15.57:0/838171229 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fc11c06ae70 0x7fc11c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:19.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.972+0000 7fc13e30a640 1 --2- 172.21.15.57:0/838171229 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fc11c06ae70 0x7fc11c06d330 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fc134000f60 tx=0x7fc134009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:19.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:19.975+0000 7fc12effd640 1 -- 172.21.15.57:0/838171229 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc138159bb0 con 0x7fc13813e520 2023-10-19T12:45:20.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.153+0000 7fc14056e640 1 -- 172.21.15.57:0/838171229 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc138159bb0 con 0x7fc13813e520 2023-10-19T12:45:20.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.154+0000 7fc12effd640 1 -- 172.21.15.57:0/838171229 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18748 (secure 0 0 0) 0x7fc138159bb0 con 0x7fc13813e520 2023-10-19T12:45:20.158 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:45:20.158 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":27,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T12:44:58.978403+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-19T12:44:16.517510+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-19T12:44:17.398297+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-19T12:44:21.129403+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.23","mappings":[{"from":3,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"3.39","mappings":[{"from":7,"to":1}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.1d","mappings":[{"from":1,"to":5}]},{"pgid":"4.1e","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T12:45:20.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.156+0000 7fc14056e640 1 -- 172.21.15.57:0/838171229 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fc11c06ae70 msgr2=0x7fc11c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:20.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.156+0000 7fc14056e640 1 --2- 172.21.15.57:0/838171229 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fc11c06ae70 0x7fc11c06d330 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fc134000f60 tx=0x7fc134009040 comp rx=0 tx=0).stop 2023-10-19T12:45:20.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.156+0000 7fc14056e640 1 -- 172.21.15.57:0/838171229 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc13813e520 msgr2=0x7fc138159e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:20.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.156+0000 7fc14056e640 1 --2- 172.21.15.57:0/838171229 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc13813e520 0x7fc138159e90 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fc12800c400 tx=0x7fc12800c8c0 comp rx=0 tx=0).stop 2023-10-19T12:45:20.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.156+0000 7fc14056e640 1 -- 172.21.15.57:0/838171229 shutdown_connections 2023-10-19T12:45:20.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.156+0000 7fc14056e640 1 --2- 172.21.15.57:0/838171229 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fc11c06ae70 0x7fc11c06d330 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:20.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.156+0000 7fc14056e640 1 --2- 172.21.15.57:0/838171229 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc138141690 0x7fc13813a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:20.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.156+0000 7fc14056e640 1 --2- 172.21.15.57:0/838171229 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc13813e520 0x7fc138159e90 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:20.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.156+0000 7fc14056e640 1 --2- 172.21.15.57:0/838171229 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc138070af0 0x7fc13815ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:20.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.156+0000 7fc14056e640 1 -- 172.21.15.57:0/838171229 >> 172.21.15.57:0/838171229 conn(0x7fc138069930 msgr2=0x7fc13806bb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:20.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.156+0000 7fc14056e640 1 -- 172.21.15.57:0/838171229 shutdown_connections 2023-10-19T12:45:20.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.156+0000 7fc14056e640 1 -- 172.21.15.57:0/838171229 wait complete. 2023-10-19T12:45:20.170 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-19T12:44:07.371186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-19T12:44:16.517510+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-19T12:44:17.398297+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-19T12:44:21.129403+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-19T12:45:20.171 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-19T12:45:20.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.282+0000 7f9c374a9640 1 -- 172.21.15.57:0/734290374 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9c3006f0a0 msgr2=0x7f9c3006dbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:20.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.282+0000 7f9c374a9640 1 --2- 172.21.15.57:0/734290374 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9c3006f0a0 0x7f9c3006dbf0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f9c300724a0 tx=0x7f9c2002bda0 comp rx=0 tx=0).stop 2023-10-19T12:45:20.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.282+0000 7f9c374a9640 1 -- 172.21.15.57:0/734290374 shutdown_connections 2023-10-19T12:45:20.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.282+0000 7f9c374a9640 1 --2- 172.21.15.57:0/734290374 >> v2:172.21.15.179:3300/0 conn(0x7f9c300ab950 0x7f9c300abde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:20.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.282+0000 7f9c374a9640 1 --2- 172.21.15.57:0/734290374 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9c3006f0a0 0x7f9c3006dbf0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:20.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.282+0000 7f9c374a9640 1 --2- 172.21.15.57:0/734290374 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9c3006d2d0 0x7f9c3006d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:20.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.282+0000 7f9c374a9640 1 -- 172.21.15.57:0/734290374 >> 172.21.15.57:0/734290374 conn(0x7f9c300b0670 msgr2=0x7f9c300b0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:20.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.282+0000 7f9c374a9640 1 -- 172.21.15.57:0/734290374 shutdown_connections 2023-10-19T12:45:20.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.282+0000 7f9c374a9640 1 -- 172.21.15.57:0/734290374 wait complete. 2023-10-19T12:45:20.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.283+0000 7f9c374a9640 1 Processor -- start 2023-10-19T12:45:20.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.283+0000 7f9c374a9640 1 -- start start 2023-10-19T12:45:20.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.284+0000 7f9c374a9640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9c3006d2d0 0x7f9c30143710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:20.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.284+0000 7f9c374a9640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9c3006f0a0 0x7f9c301eee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:20.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.284+0000 7f9c374a9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9c300ab950 0x7f9c301f3400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:20.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.284+0000 7f9c374a9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c301f3940 con 0x7f9c300ab950 2023-10-19T12:45:20.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.284+0000 7f9c374a9640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c30141780 con 0x7f9c3006d2d0 2023-10-19T12:45:20.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.284+0000 7f9c374a9640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c30143d30 con 0x7f9c3006f0a0 2023-10-19T12:45:20.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.284+0000 7f9c35a46640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9c300ab950 0x7f9c301f3400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:20.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.284+0000 7f9c35a46640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9c300ab950 0x7f9c301f3400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50380/0 (socket says 172.21.15.57:50380) 2023-10-19T12:45:20.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.284+0000 7f9c35a46640 1 -- 172.21.15.57:0/2260241442 learned_addr learned my addr 172.21.15.57:0/2260241442 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:20.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.284+0000 7f9c35245640 1 --2- 172.21.15.57:0/2260241442 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9c3006d2d0 0x7f9c30143710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:20.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.284+0000 7f9c34a44640 1 --2- 172.21.15.57:0/2260241442 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9c3006f0a0 0x7f9c301eee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:20.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.284+0000 7f9c35a46640 1 -- 172.21.15.57:0/2260241442 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9c3006f0a0 msgr2=0x7f9c301eee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:20.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.284+0000 7f9c35a46640 1 --2- 172.21.15.57:0/2260241442 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9c3006f0a0 0x7f9c301eee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:20.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.284+0000 7f9c35a46640 1 -- 172.21.15.57:0/2260241442 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9c3006d2d0 msgr2=0x7f9c30143710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:20.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.284+0000 7f9c35a46640 1 --2- 172.21.15.57:0/2260241442 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9c3006d2d0 0x7f9c30143710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:20.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.284+0000 7f9c35a46640 1 -- 172.21.15.57:0/2260241442 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c20009560 con 0x7f9c300ab950 2023-10-19T12:45:20.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.284+0000 7f9c35245640 1 --2- 172.21.15.57:0/2260241442 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9c3006d2d0 0x7f9c30143710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:45:20.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.285+0000 7f9c35a46640 1 --2- 172.21.15.57:0/2260241442 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9c300ab950 0x7f9c301f3400 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f9c2800d6a0 tx=0x7f9c2800db60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:20.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.285+0000 7f9c25ffb640 1 -- 172.21.15.57:0/2260241442 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c28004220 con 0x7f9c300ab950 2023-10-19T12:45:20.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.285+0000 7f9c25ffb640 1 -- 172.21.15.57:0/2260241442 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f9c28004470 con 0x7f9c300ab950 2023-10-19T12:45:20.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.285+0000 7f9c374a9640 1 -- 172.21.15.57:0/2260241442 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c30144010 con 0x7f9c300ab950 2023-10-19T12:45:20.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.285+0000 7f9c25ffb640 1 -- 172.21.15.57:0/2260241442 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c28005260 con 0x7f9c300ab950 2023-10-19T12:45:20.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.285+0000 7f9c374a9640 1 -- 172.21.15.57:0/2260241442 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c301445d0 con 0x7f9c300ab950 2023-10-19T12:45:20.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.286+0000 7f9c374a9640 1 -- 172.21.15.57:0/2260241442 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c300ce1d0 con 0x7f9c300ab950 2023-10-19T12:45:20.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.286+0000 7f9c25ffb640 1 -- 172.21.15.57:0/2260241442 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f9c280053c0 con 0x7f9c300ab950 2023-10-19T12:45:20.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.287+0000 7f9c25ffb640 1 --2- 172.21.15.57:0/2260241442 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f9c1406ae70 0x7f9c1406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:20.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.287+0000 7f9c25ffb640 1 -- 172.21.15.57:0/2260241442 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f9c28089ec0 con 0x7f9c300ab950 2023-10-19T12:45:20.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.287+0000 7f9c35245640 1 --2- 172.21.15.57:0/2260241442 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f9c1406ae70 0x7f9c1406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:20.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.288+0000 7f9c35245640 1 --2- 172.21.15.57:0/2260241442 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f9c1406ae70 0x7f9c1406d330 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f9c2c005fd0 tx=0x7f9c2c009480 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:20.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.291+0000 7f9c25ffb640 1 -- 172.21.15.57:0/2260241442 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9c28054b10 con 0x7f9c300ab950 2023-10-19T12:45:20.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.467+0000 7f9c374a9640 1 -- 172.21.15.57:0/2260241442 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9c300690e0 con 0x7f9c300ab950 2023-10-19T12:45:20.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.467+0000 7f9c25ffb640 1 -- 172.21.15.57:0/2260241442 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9c28017070 con 0x7f9c300ab950 2023-10-19T12:45:20.471 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-19T12:45:20.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.470+0000 7f9c374a9640 1 -- 172.21.15.57:0/2260241442 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f9c1406ae70 msgr2=0x7f9c1406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:20.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.470+0000 7f9c374a9640 1 --2- 172.21.15.57:0/2260241442 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f9c1406ae70 0x7f9c1406d330 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f9c2c005fd0 tx=0x7f9c2c009480 comp rx=0 tx=0).stop 2023-10-19T12:45:20.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.470+0000 7f9c374a9640 1 -- 172.21.15.57:0/2260241442 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9c300ab950 msgr2=0x7f9c301f3400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:20.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.470+0000 7f9c374a9640 1 --2- 172.21.15.57:0/2260241442 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9c300ab950 0x7f9c301f3400 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f9c2800d6a0 tx=0x7f9c2800db60 comp rx=0 tx=0).stop 2023-10-19T12:45:20.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.470+0000 7f9c374a9640 1 -- 172.21.15.57:0/2260241442 shutdown_connections 2023-10-19T12:45:20.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.470+0000 7f9c374a9640 1 --2- 172.21.15.57:0/2260241442 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f9c1406ae70 0x7f9c1406d330 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:20.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.470+0000 7f9c374a9640 1 --2- 172.21.15.57:0/2260241442 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9c300ab950 0x7f9c301f3400 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:20.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.470+0000 7f9c374a9640 1 --2- 172.21.15.57:0/2260241442 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9c3006f0a0 0x7f9c301eee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:20.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.470+0000 7f9c374a9640 1 --2- 172.21.15.57:0/2260241442 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9c3006d2d0 0x7f9c30143710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:20.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.470+0000 7f9c374a9640 1 -- 172.21.15.57:0/2260241442 >> 172.21.15.57:0/2260241442 conn(0x7f9c300b0670 msgr2=0x7f9c300cbd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:20.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.470+0000 7f9c374a9640 1 -- 172.21.15.57:0/2260241442 shutdown_connections 2023-10-19T12:45:20.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.470+0000 7f9c374a9640 1 -- 172.21.15.57:0/2260241442 wait complete. 2023-10-19T12:45:20.483 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-19T12:45:20.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.593+0000 7f3381ae7640 1 -- 172.21.15.57:0/3654211596 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f337c06f0a0 msgr2=0x7f337c06ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:20.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.593+0000 7f3381ae7640 1 --2- 172.21.15.57:0/3654211596 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f337c06f0a0 0x7f337c06ddc0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f3370009830 tx=0x7f337002bd60 comp rx=0 tx=0).stop 2023-10-19T12:45:20.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.593+0000 7f3381ae7640 1 -- 172.21.15.57:0/3654211596 shutdown_connections 2023-10-19T12:45:20.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.593+0000 7f3381ae7640 1 --2- 172.21.15.57:0/3654211596 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f337c0ad370 0x7f337c0ad800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:20.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.593+0000 7f3381ae7640 1 --2- 172.21.15.57:0/3654211596 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f337c06f0a0 0x7f337c06ddc0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:20.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.593+0000 7f3381ae7640 1 -- 172.21.15.57:0/3654211596 >> 172.21.15.57:0/3654211596 conn(0x7f337c0b0670 msgr2=0x7f337c0b0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:20.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.593+0000 7f3381ae7640 1 -- 172.21.15.57:0/3654211596 shutdown_connections 2023-10-19T12:45:20.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.593+0000 7f3381ae7640 1 -- 172.21.15.57:0/3654211596 wait complete. 2023-10-19T12:45:20.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.594+0000 7f3381ae7640 1 Processor -- start 2023-10-19T12:45:20.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.594+0000 7f3381ae7640 1 -- start start 2023-10-19T12:45:20.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.595+0000 7f3381ae7640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f337c06d2d0 0x7f337c06a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:20.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.595+0000 7f3381ae7640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f337c06f0a0 0x7f337c154490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:20.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.595+0000 7f3381ae7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f337c0ad370 0x7f337c1e3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:20.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.595+0000 7f3381ae7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f337c06ae10 con 0x7f337c0ad370 2023-10-19T12:45:20.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.595+0000 7f3381ae7640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f337c06b0f0 con 0x7f337c06f0a0 2023-10-19T12:45:20.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.595+0000 7f3381ae7640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f337c06b3d0 con 0x7f337c06d2d0 2023-10-19T12:45:20.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.595+0000 7f337bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f337c0ad370 0x7f337c1e3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:20.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.595+0000 7f337bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f337c0ad370 0x7f337c1e3fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50392/0 (socket says 172.21.15.57:50392) 2023-10-19T12:45:20.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.595+0000 7f337bfff640 1 -- 172.21.15.57:0/2308908580 learned_addr learned my addr 172.21.15.57:0/2308908580 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:20.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.595+0000 7f337b7fe640 1 --2- 172.21.15.57:0/2308908580 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f337c06d2d0 0x7f337c06a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:20.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.595+0000 7f337affd640 1 --2- 172.21.15.57:0/2308908580 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f337c06f0a0 0x7f337c154490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:20.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.595+0000 7f337bfff640 1 -- 172.21.15.57:0/2308908580 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f337c06d2d0 msgr2=0x7f337c06a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:20.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.595+0000 7f337bfff640 1 --2- 172.21.15.57:0/2308908580 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f337c06d2d0 0x7f337c06a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:20.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.595+0000 7f337bfff640 1 -- 172.21.15.57:0/2308908580 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f337c06f0a0 msgr2=0x7f337c154490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:20.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.595+0000 7f337bfff640 1 --2- 172.21.15.57:0/2308908580 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f337c06f0a0 0x7f337c154490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:20.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.595+0000 7f337bfff640 1 -- 172.21.15.57:0/2308908580 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3370009490 con 0x7f337c0ad370 2023-10-19T12:45:20.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.595+0000 7f337b7fe640 1 --2- 172.21.15.57:0/2308908580 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f337c06d2d0 0x7f337c06a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:45:20.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.595+0000 7f337affd640 1 --2- 172.21.15.57:0/2308908580 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f337c06f0a0 0x7f337c154490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:45:20.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.595+0000 7f337bfff640 1 --2- 172.21.15.57:0/2308908580 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f337c0ad370 0x7f337c1e3fa0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f337400cad0 tx=0x7f337400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:20.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.596+0000 7f3363fff640 1 -- 172.21.15.57:0/2308908580 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f337400ae20 con 0x7f337c0ad370 2023-10-19T12:45:20.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.596+0000 7f3363fff640 1 -- 172.21.15.57:0/2308908580 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f337400f450 con 0x7f337c0ad370 2023-10-19T12:45:20.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.596+0000 7f3363fff640 1 -- 172.21.15.57:0/2308908580 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f337400f8c0 con 0x7f337c0ad370 2023-10-19T12:45:20.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.596+0000 7f3381ae7640 1 -- 172.21.15.57:0/2308908580 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f337c0cd310 con 0x7f337c0ad370 2023-10-19T12:45:20.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.596+0000 7f3381ae7640 1 -- 172.21.15.57:0/2308908580 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f337c06b990 con 0x7f337c0ad370 2023-10-19T12:45:20.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.597+0000 7f3381ae7640 1 -- 172.21.15.57:0/2308908580 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f337c0ce1d0 con 0x7f337c0ad370 2023-10-19T12:45:20.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.597+0000 7f3363fff640 1 -- 172.21.15.57:0/2308908580 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f337400fa20 con 0x7f337c0ad370 2023-10-19T12:45:20.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.598+0000 7f3363fff640 1 --2- 172.21.15.57:0/2308908580 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f336806ae40 0x7f336806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:20.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.598+0000 7f3363fff640 1 -- 172.21.15.57:0/2308908580 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f337408a530 con 0x7f337c0ad370 2023-10-19T12:45:20.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.598+0000 7f337b7fe640 1 --2- 172.21.15.57:0/2308908580 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f336806ae40 0x7f336806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:20.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.598+0000 7f337b7fe640 1 --2- 172.21.15.57:0/2308908580 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f336806ae40 0x7f336806d300 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f336c000f60 tx=0x7f336c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:20.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.602+0000 7f3363fff640 1 -- 172.21.15.57:0/2308908580 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3374058d40 con 0x7f337c0ad370 2023-10-19T12:45:20.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.776+0000 7f3381ae7640 1 -- 172.21.15.57:0/2308908580 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f337c0690e0 con 0x7f337c0ad370 2023-10-19T12:45:20.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.776+0000 7f3363fff640 1 -- 172.21.15.57:0/2308908580 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7f33740588d0 con 0x7f337c0ad370 2023-10-19T12:45:20.780 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2023-10-19T12:45:20.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.778+0000 7f3381ae7640 1 -- 172.21.15.57:0/2308908580 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f336806ae40 msgr2=0x7f336806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:20.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.778+0000 7f3381ae7640 1 --2- 172.21.15.57:0/2308908580 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f336806ae40 0x7f336806d300 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f336c000f60 tx=0x7f336c009040 comp rx=0 tx=0).stop 2023-10-19T12:45:20.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.778+0000 7f3381ae7640 1 -- 172.21.15.57:0/2308908580 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f337c0ad370 msgr2=0x7f337c1e3fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:20.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.778+0000 7f3381ae7640 1 --2- 172.21.15.57:0/2308908580 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f337c0ad370 0x7f337c1e3fa0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f337400cad0 tx=0x7f337400cf90 comp rx=0 tx=0).stop 2023-10-19T12:45:20.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.778+0000 7f3381ae7640 1 -- 172.21.15.57:0/2308908580 shutdown_connections 2023-10-19T12:45:20.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.778+0000 7f3381ae7640 1 --2- 172.21.15.57:0/2308908580 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f336806ae40 0x7f336806d300 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:20.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.778+0000 7f3381ae7640 1 --2- 172.21.15.57:0/2308908580 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f337c0ad370 0x7f337c1e3fa0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:20.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.778+0000 7f3381ae7640 1 --2- 172.21.15.57:0/2308908580 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f337c06f0a0 0x7f337c154490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:20.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.778+0000 7f3381ae7640 1 --2- 172.21.15.57:0/2308908580 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f337c06d2d0 0x7f337c06a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:20.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.778+0000 7f3381ae7640 1 -- 172.21.15.57:0/2308908580 >> 172.21.15.57:0/2308908580 conn(0x7f337c0b0670 msgr2=0x7f337c06d6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:20.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.778+0000 7f3381ae7640 1 -- 172.21.15.57:0/2308908580 shutdown_connections 2023-10-19T12:45:20.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.778+0000 7f3381ae7640 1 -- 172.21.15.57:0/2308908580 wait complete. 2023-10-19T12:45:20.792 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-19T12:45:20.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.901+0000 7fdd4aa10640 1 -- 172.21.15.57:0/1686150019 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd2c004d00 msgr2=0x7fdd2c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:20.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.901+0000 7fdd4aa10640 1 --2- 172.21.15.57:0/1686150019 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd2c004d00 0x7fdd2c005180 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c00c610 tx=0x7fdd2c0057f0 comp rx=0 tx=0).stop 2023-10-19T12:45:20.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.901+0000 7fdd48fad640 1 -- 172.21.15.57:0/1686150019 reap_dead start 2023-10-19T12:45:20.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.901+0000 7fdd4aa10640 1 -- 172.21.15.57:0/1686150019 shutdown_connections 2023-10-19T12:45:20.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.901+0000 7fdd4aa10640 1 -- 172.21.15.57:0/1686150019 >> 172.21.15.57:0/1686150019 conn(0x7fdd4406fae0 msgr2=0x7fdd4406fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:20.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.901+0000 7fdd4aa10640 1 -- 172.21.15.57:0/1686150019 shutdown_connections 2023-10-19T12:45:20.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.901+0000 7fdd4aa10640 1 -- 172.21.15.57:0/1686150019 wait complete. 2023-10-19T12:45:20.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.903+0000 7fdd4aa10640 1 Processor -- start 2023-10-19T12:45:20.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.903+0000 7fdd4aa10640 1 -- start start 2023-10-19T12:45:20.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.903+0000 7fdd4aa10640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdd4413fde0 0x7fdd440ad760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:20.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.903+0000 7fdd4aa10640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdd441401c0 0x7fdd44159180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:20.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.903+0000 7fdd4aa10640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd440adca0 0x7fdd441596c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:20.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.903+0000 7fdd4aa10640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd440aee60 con 0x7fdd440adca0 2023-10-19T12:45:20.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.903+0000 7fdd4aa10640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd440ae540 con 0x7fdd4413fde0 2023-10-19T12:45:20.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.903+0000 7fdd4aa10640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd440ac250 con 0x7fdd441401c0 2023-10-19T12:45:20.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.904+0000 7fdd48fad640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd440adca0 0x7fdd441596c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:20.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.904+0000 7fdd48fad640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd440adca0 0x7fdd441596c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50404/0 (socket says 172.21.15.57:50404) 2023-10-19T12:45:20.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.904+0000 7fdd48fad640 1 -- 172.21.15.57:0/1856383130 learned_addr learned my addr 172.21.15.57:0/1856383130 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:20.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.904+0000 7fdd43fff640 1 --2- 172.21.15.57:0/1856383130 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdd4413fde0 0x7fdd440ad760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:20.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.904+0000 7fdd437fe640 1 --2- 172.21.15.57:0/1856383130 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdd441401c0 0x7fdd44159180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:20.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.904+0000 7fdd48fad640 1 -- 172.21.15.57:0/1856383130 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdd441401c0 msgr2=0x7fdd44159180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:20.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.904+0000 7fdd48fad640 1 --2- 172.21.15.57:0/1856383130 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdd441401c0 0x7fdd44159180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:20.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.904+0000 7fdd48fad640 1 -- 172.21.15.57:0/1856383130 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdd4413fde0 msgr2=0x7fdd440ad760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:20.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.904+0000 7fdd48fad640 1 --2- 172.21.15.57:0/1856383130 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdd4413fde0 0x7fdd440ad760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:20.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.904+0000 7fdd48fad640 1 -- 172.21.15.57:0/1856383130 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd440ab660 con 0x7fdd440adca0 2023-10-19T12:45:20.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.904+0000 7fdd48fad640 1 --2- 172.21.15.57:0/1856383130 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd440adca0 0x7fdd441596c0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c02cc60 tx=0x7fdd3c001840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:20.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.904+0000 7fdd40ff9640 1 -- 172.21.15.57:0/1856383130 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd3c003b70 con 0x7fdd440adca0 2023-10-19T12:45:20.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.904+0000 7fdd40ff9640 1 -- 172.21.15.57:0/1856383130 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fdd3c003cd0 con 0x7fdd440adca0 2023-10-19T12:45:20.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.904+0000 7fdd4aa10640 1 -- 172.21.15.57:0/1856383130 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd440ab920 con 0x7fdd440adca0 2023-10-19T12:45:20.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.904+0000 7fdd40ff9640 1 -- 172.21.15.57:0/1856383130 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd3c006330 con 0x7fdd440adca0 2023-10-19T12:45:20.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.904+0000 7fdd4aa10640 1 -- 172.21.15.57:0/1856383130 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd440abee0 con 0x7fdd440adca0 2023-10-19T12:45:20.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.905+0000 7fdd4aa10640 1 -- 172.21.15.57:0/1856383130 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd440cbc10 con 0x7fdd440adca0 2023-10-19T12:45:20.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.906+0000 7fdd40ff9640 1 -- 172.21.15.57:0/1856383130 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fdd3c031030 con 0x7fdd440adca0 2023-10-19T12:45:20.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.906+0000 7fdd40ff9640 1 --2- 172.21.15.57:0/1856383130 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fdd3006ae70 0x7fdd3006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:20.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.906+0000 7fdd40ff9640 1 -- 172.21.15.57:0/1856383130 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fdd3c0aaa80 con 0x7fdd440adca0 2023-10-19T12:45:20.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.906+0000 7fdd43fff640 1 --2- 172.21.15.57:0/1856383130 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fdd3006ae70 0x7fdd3006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:20.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.907+0000 7fdd43fff640 1 --2- 172.21.15.57:0/1856383130 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fdd3006ae70 0x7fdd3006d330 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fdd38009cd0 tx=0x7fdd38005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:20.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:20.910+0000 7fdd40ff9640 1 -- 172.21.15.57:0/1856383130 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fdd3c079360 con 0x7fdd440adca0 2023-10-19T12:45:21.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.084+0000 7fdd4aa10640 1 -- 172.21.15.57:0/1856383130 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fdd440665b0 con 0x7fdd440adca0 2023-10-19T12:45:21.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.085+0000 7fdd40ff9640 1 -- 172.21.15.57:0/1856383130 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7fdd3c0019b0 con 0x7fdd440adca0 2023-10-19T12:45:21.088 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 64 2023-10-19T12:45:21.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.087+0000 7fdd4aa10640 1 -- 172.21.15.57:0/1856383130 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fdd3006ae70 msgr2=0x7fdd3006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:21.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.087+0000 7fdd4aa10640 1 --2- 172.21.15.57:0/1856383130 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fdd3006ae70 0x7fdd3006d330 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fdd38009cd0 tx=0x7fdd38005f90 comp rx=0 tx=0).stop 2023-10-19T12:45:21.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.087+0000 7fdd4aa10640 1 -- 172.21.15.57:0/1856383130 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd440adca0 msgr2=0x7fdd441596c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:21.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.087+0000 7fdd4aa10640 1 --2- 172.21.15.57:0/1856383130 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd440adca0 0x7fdd441596c0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c02cc60 tx=0x7fdd3c001840 comp rx=0 tx=0).stop 2023-10-19T12:45:21.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.087+0000 7fdd4aa10640 1 -- 172.21.15.57:0/1856383130 shutdown_connections 2023-10-19T12:45:21.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.087+0000 7fdd4aa10640 1 --2- 172.21.15.57:0/1856383130 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fdd3006ae70 0x7fdd3006d330 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:21.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.087+0000 7fdd4aa10640 1 --2- 172.21.15.57:0/1856383130 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd440adca0 0x7fdd441596c0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:21.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.087+0000 7fdd4aa10640 1 --2- 172.21.15.57:0/1856383130 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdd441401c0 0x7fdd44159180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:21.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.087+0000 7fdd4aa10640 1 --2- 172.21.15.57:0/1856383130 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdd4413fde0 0x7fdd440ad760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:21.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.087+0000 7fdd4aa10640 1 -- 172.21.15.57:0/1856383130 >> 172.21.15.57:0/1856383130 conn(0x7fdd4406fae0 msgr2=0x7fdd4414fb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:21.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.087+0000 7fdd4aa10640 1 -- 172.21.15.57:0/1856383130 shutdown_connections 2023-10-19T12:45:21.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.087+0000 7fdd4aa10640 1 -- 172.21.15.57:0/1856383130 wait complete. 2023-10-19T12:45:21.101 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-19T12:45:21.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.211+0000 7f40b011d640 1 -- 172.21.15.57:0/3143028368 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f40a80b0530 msgr2=0x7f40a80b09c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:21.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.211+0000 7f40b011d640 1 --2- 172.21.15.57:0/3143028368 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f40a80b0530 0x7f40a80b09c0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f40a0005fd0 tx=0x7f40a002c0e0 comp rx=0 tx=0).stop 2023-10-19T12:45:21.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.211+0000 7f40b011d640 1 -- 172.21.15.57:0/3143028368 shutdown_connections 2023-10-19T12:45:21.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.211+0000 7f40b011d640 1 --2- 172.21.15.57:0/3143028368 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f40a80b0530 0x7f40a80b09c0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:21.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.211+0000 7f40b011d640 1 --2- 172.21.15.57:0/3143028368 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f40a80cbd20 0x7f40a8158300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:21.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.211+0000 7f40b011d640 1 -- 172.21.15.57:0/3143028368 >> 172.21.15.57:0/3143028368 conn(0x7f40a8069930 msgr2=0x7f40a8069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:21.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.211+0000 7f40b011d640 1 -- 172.21.15.57:0/3143028368 shutdown_connections 2023-10-19T12:45:21.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.211+0000 7f40b011d640 1 -- 172.21.15.57:0/3143028368 wait complete. 2023-10-19T12:45:21.217 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.212+0000 7f40b011d640 1 Processor -- start 2023-10-19T12:45:21.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.212+0000 7f40b011d640 1 -- start start 2023-10-19T12:45:21.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.213+0000 7f40b011d640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f40a80afa20 0x7f40a813d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:21.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.213+0000 7f40b011d640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f40a80b0530 0x7f40a81e0520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:21.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.213+0000 7f40b011d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f40a80cbd20 0x7f40a8149d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:21.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.213+0000 7f40b011d640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40a8148ce0 con 0x7f40a80cbd20 2023-10-19T12:45:21.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.213+0000 7f40b011d640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40a8148fc0 con 0x7f40a80b0530 2023-10-19T12:45:21.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.213+0000 7f40b011d640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40a81492a0 con 0x7f40a80afa20 2023-10-19T12:45:21.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.213+0000 7f40ae6ba640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f40a80cbd20 0x7f40a8149d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:21.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.213+0000 7f40ae6ba640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f40a80cbd20 0x7f40a8149d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50424/0 (socket says 172.21.15.57:50424) 2023-10-19T12:45:21.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.213+0000 7f40ae6ba640 1 -- 172.21.15.57:0/2870519130 learned_addr learned my addr 172.21.15.57:0/2870519130 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:21.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.213+0000 7f40adeb9640 1 --2- 172.21.15.57:0/2870519130 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f40a80afa20 0x7f40a813d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:21.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.213+0000 7f40ad6b8640 1 --2- 172.21.15.57:0/2870519130 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f40a80b0530 0x7f40a81e0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:21.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.213+0000 7f40ae6ba640 1 -- 172.21.15.57:0/2870519130 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f40a80afa20 msgr2=0x7f40a813d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:21.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.213+0000 7f40ae6ba640 1 --2- 172.21.15.57:0/2870519130 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f40a80afa20 0x7f40a813d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:21.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.213+0000 7f40ae6ba640 1 -- 172.21.15.57:0/2870519130 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f40a80b0530 msgr2=0x7f40a81e0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:21.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.213+0000 7f40ae6ba640 1 --2- 172.21.15.57:0/2870519130 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f40a80b0530 0x7f40a81e0520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:21.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.213+0000 7f40ae6ba640 1 -- 172.21.15.57:0/2870519130 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40a0009840 con 0x7f40a80cbd20 2023-10-19T12:45:21.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.213+0000 7f40ad6b8640 1 --2- 172.21.15.57:0/2870519130 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f40a80b0530 0x7f40a81e0520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:45:21.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.213+0000 7f40ae6ba640 1 --2- 172.21.15.57:0/2870519130 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f40a80cbd20 0x7f40a8149d80 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f40a0036040 tx=0x7f40a0005c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:21.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.213+0000 7f409e7fc640 1 -- 172.21.15.57:0/2870519130 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40a0003e60 con 0x7f40a80cbd20 2023-10-19T12:45:21.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.213+0000 7f409e7fc640 1 -- 172.21.15.57:0/2870519130 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f40a0004790 con 0x7f40a80cbd20 2023-10-19T12:45:21.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.213+0000 7f40b011d640 1 -- 172.21.15.57:0/2870519130 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40a8149580 con 0x7f40a80cbd20 2023-10-19T12:45:21.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.214+0000 7f409e7fc640 1 -- 172.21.15.57:0/2870519130 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40a0034800 con 0x7f40a80cbd20 2023-10-19T12:45:21.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.214+0000 7f40b011d640 1 -- 172.21.15.57:0/2870519130 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40a8148690 con 0x7f40a80cbd20 2023-10-19T12:45:21.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.214+0000 7f40b011d640 1 -- 172.21.15.57:0/2870519130 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40a8149a40 con 0x7f40a80cbd20 2023-10-19T12:45:21.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.215+0000 7f409e7fc640 1 -- 172.21.15.57:0/2870519130 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f40a0034960 con 0x7f40a80cbd20 2023-10-19T12:45:21.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.215+0000 7f409e7fc640 1 --2- 172.21.15.57:0/2870519130 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f408c06ae40 0x7f408c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:21.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.215+0000 7f409e7fc640 1 -- 172.21.15.57:0/2870519130 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f40a00aca20 con 0x7f40a80cbd20 2023-10-19T12:45:21.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.216+0000 7f40adeb9640 1 --2- 172.21.15.57:0/2870519130 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f408c06ae40 0x7f408c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:21.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.216+0000 7f40adeb9640 1 --2- 172.21.15.57:0/2870519130 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f408c06ae40 0x7f408c06d300 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f40a400aba0 tx=0x7f40a4010040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:21.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.219+0000 7f409e7fc640 1 -- 172.21.15.57:0/2870519130 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f40a0077710 con 0x7f40a80cbd20 2023-10-19T12:45:21.397 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.394+0000 7f40b011d640 1 -- 172.21.15.57:0/2870519130 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f40a80690e0 con 0x7f40a80cbd20 2023-10-19T12:45:21.397 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.394+0000 7f409e7fc640 1 -- 172.21.15.57:0/2870519130 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7f40a007ad30 con 0x7f40a80cbd20 2023-10-19T12:45:21.397 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2023-10-19T12:45:21.399 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.396+0000 7f40b011d640 1 -- 172.21.15.57:0/2870519130 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f408c06ae40 msgr2=0x7f408c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:21.400 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.396+0000 7f40b011d640 1 --2- 172.21.15.57:0/2870519130 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f408c06ae40 0x7f408c06d300 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f40a400aba0 tx=0x7f40a4010040 comp rx=0 tx=0).stop 2023-10-19T12:45:21.400 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.396+0000 7f40b011d640 1 -- 172.21.15.57:0/2870519130 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f40a80cbd20 msgr2=0x7f40a8149d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:21.400 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.396+0000 7f40b011d640 1 --2- 172.21.15.57:0/2870519130 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f40a80cbd20 0x7f40a8149d80 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f40a0036040 tx=0x7f40a0005c80 comp rx=0 tx=0).stop 2023-10-19T12:45:21.400 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.396+0000 7f40b011d640 1 -- 172.21.15.57:0/2870519130 shutdown_connections 2023-10-19T12:45:21.400 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.396+0000 7f40b011d640 1 --2- 172.21.15.57:0/2870519130 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f408c06ae40 0x7f408c06d300 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:21.401 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.396+0000 7f40b011d640 1 --2- 172.21.15.57:0/2870519130 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f40a80cbd20 0x7f40a8149d80 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:21.401 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.396+0000 7f40b011d640 1 --2- 172.21.15.57:0/2870519130 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f40a80b0530 0x7f40a81e0520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:21.401 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.396+0000 7f40b011d640 1 --2- 172.21.15.57:0/2870519130 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f40a80afa20 0x7f40a813d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:21.401 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.396+0000 7f40b011d640 1 -- 172.21.15.57:0/2870519130 >> 172.21.15.57:0/2870519130 conn(0x7f40a8069930 msgr2=0x7f40a806b1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:21.401 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.396+0000 7f40b011d640 1 -- 172.21.15.57:0/2870519130 shutdown_connections 2023-10-19T12:45:21.401 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.396+0000 7f40b011d640 1 -- 172.21.15.57:0/2870519130 wait complete. 2023-10-19T12:45:21.410 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-19T12:45:21.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.516+0000 7f3f545f0640 1 -- 172.21.15.57:0/73271323 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3f4c0ceb40 msgr2=0x7f3f4c0b0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:21.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.516+0000 7f3f545f0640 1 --2- 172.21.15.57:0/73271323 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3f4c0ceb40 0x7f3f4c0b0650 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f3f4c08a810 tx=0x7f3f3c02c260 comp rx=0 tx=0).stop 2023-10-19T12:45:21.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.516+0000 7f3f545f0640 1 -- 172.21.15.57:0/73271323 shutdown_connections 2023-10-19T12:45:21.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.516+0000 7f3f545f0640 1 --2- 172.21.15.57:0/73271323 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3f4c0ceb40 0x7f3f4c0b0650 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:21.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.516+0000 7f3f545f0640 1 --2- 172.21.15.57:0/73271323 >> v2:172.21.15.179:3300/0 conn(0x7f3f4c0cf340 0x7f3f4c0d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:21.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.516+0000 7f3f545f0640 1 -- 172.21.15.57:0/73271323 >> 172.21.15.57:0/73271323 conn(0x7f3f4c0d0d60 msgr2=0x7f3f4c0d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:21.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.516+0000 7f3f545f0640 1 -- 172.21.15.57:0/73271323 shutdown_connections 2023-10-19T12:45:21.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.516+0000 7f3f545f0640 1 -- 172.21.15.57:0/73271323 wait complete. 2023-10-19T12:45:21.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.517+0000 7f3f545f0640 1 Processor -- start 2023-10-19T12:45:21.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.517+0000 7f3f545f0640 1 -- start start 2023-10-19T12:45:21.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.518+0000 7f3f545f0640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3f4c0ceb40 0x7f3f4c157e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:21.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.518+0000 7f3f545f0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3f4c0cf340 0x7f3f4c09e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:21.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.518+0000 7f3f545f0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3f4c15b2a0 0x7f3f4c158950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:21.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.518+0000 7f3f545f0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f4c15d280 con 0x7f3f4c15b2a0 2023-10-19T12:45:21.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.518+0000 7f3f545f0640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f4c158360 con 0x7f3f4c0cf340 2023-10-19T12:45:21.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.518+0000 7f3f545f0640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f4c15d4d0 con 0x7f3f4c0ceb40 2023-10-19T12:45:21.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.518+0000 7f3f52b8d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3f4c15b2a0 0x7f3f4c158950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:21.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.518+0000 7f3f52b8d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3f4c15b2a0 0x7f3f4c158950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50440/0 (socket says 172.21.15.57:50440) 2023-10-19T12:45:21.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.518+0000 7f3f52b8d640 1 -- 172.21.15.57:0/3914054065 learned_addr learned my addr 172.21.15.57:0/3914054065 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:21.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.518+0000 7f3f51b8b640 1 --2- 172.21.15.57:0/3914054065 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3f4c0cf340 0x7f3f4c09e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:21.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.518+0000 7f3f5238c640 1 --2- 172.21.15.57:0/3914054065 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3f4c0ceb40 0x7f3f4c157e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:21.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.519+0000 7f3f52b8d640 1 -- 172.21.15.57:0/3914054065 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3f4c0ceb40 msgr2=0x7f3f4c157e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:21.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.519+0000 7f3f52b8d640 1 --2- 172.21.15.57:0/3914054065 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3f4c0ceb40 0x7f3f4c157e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:21.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.519+0000 7f3f52b8d640 1 -- 172.21.15.57:0/3914054065 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3f4c0cf340 msgr2=0x7f3f4c09e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:21.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.519+0000 7f3f52b8d640 1 --2- 172.21.15.57:0/3914054065 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3f4c0cf340 0x7f3f4c09e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:21.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.519+0000 7f3f52b8d640 1 -- 172.21.15.57:0/3914054065 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f3c009900 con 0x7f3f4c15b2a0 2023-10-19T12:45:21.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.519+0000 7f3f52b8d640 1 --2- 172.21.15.57:0/3914054065 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3f4c15b2a0 0x7f3f4c158950 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c037040 tx=0x7f3f3c003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:21.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.519+0000 7f3f4affd640 1 -- 172.21.15.57:0/3914054065 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f3c004240 con 0x7f3f4c15b2a0 2023-10-19T12:45:21.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.519+0000 7f3f545f0640 1 -- 172.21.15.57:0/3914054065 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f4c15d7b0 con 0x7f3f4c15b2a0 2023-10-19T12:45:21.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.519+0000 7f3f4affd640 1 -- 172.21.15.57:0/3914054065 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f3f3c004490 con 0x7f3f4c15b2a0 2023-10-19T12:45:21.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.519+0000 7f3f4affd640 1 -- 172.21.15.57:0/3914054065 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f3c004880 con 0x7f3f4c15b2a0 2023-10-19T12:45:21.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.519+0000 7f3f545f0640 1 -- 172.21.15.57:0/3914054065 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f4c15dd70 con 0x7f3f4c15b2a0 2023-10-19T12:45:21.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.520+0000 7f3f545f0640 1 -- 172.21.15.57:0/3914054065 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f4c15df90 con 0x7f3f4c15b2a0 2023-10-19T12:45:21.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.520+0000 7f3f4affd640 1 -- 172.21.15.57:0/3914054065 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f3f3c03b040 con 0x7f3f4c15b2a0 2023-10-19T12:45:21.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.521+0000 7f3f4affd640 1 --2- 172.21.15.57:0/3914054065 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f3f3806ae40 0x7f3f3806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:21.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.521+0000 7f3f4affd640 1 -- 172.21.15.57:0/3914054065 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f3f3c0ac740 con 0x7f3f4c15b2a0 2023-10-19T12:45:21.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.521+0000 7f3f5238c640 1 --2- 172.21.15.57:0/3914054065 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f3f3806ae40 0x7f3f3806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:21.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.522+0000 7f3f5238c640 1 --2- 172.21.15.57:0/3914054065 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f3f3806ae40 0x7f3f3806d300 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f3f44005fd0 tx=0x7f3f44005850 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:21.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.525+0000 7f3f4affd640 1 -- 172.21.15.57:0/3914054065 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3f4c15df90 con 0x7f3f4c15b2a0 2023-10-19T12:45:21.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.713+0000 7f3f545f0640 1 -- 172.21.15.57:0/3914054065 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f3f4c15df90 con 0x7f3f4c15b2a0 2023-10-19T12:45:21.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.713+0000 7f3f4affd640 1 -- 172.21.15.57:0/3914054065 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) v1 ==== 80+0+0 (secure 0 0 0) 0x7f3f4c15df90 con 0x7f3f4c15b2a0 2023-10-19T12:45:21.716 INFO:teuthology.orchestra.run.smithi057.stderr:listed 0 entries 2023-10-19T12:45:21.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.715+0000 7f3f545f0640 1 -- 172.21.15.57:0/3914054065 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f3f3806ae40 msgr2=0x7f3f3806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:21.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.715+0000 7f3f545f0640 1 --2- 172.21.15.57:0/3914054065 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f3f3806ae40 0x7f3f3806d300 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f3f44005fd0 tx=0x7f3f44005850 comp rx=0 tx=0).stop 2023-10-19T12:45:21.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.715+0000 7f3f545f0640 1 -- 172.21.15.57:0/3914054065 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3f4c15b2a0 msgr2=0x7f3f4c158950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:21.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.715+0000 7f3f545f0640 1 --2- 172.21.15.57:0/3914054065 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3f4c15b2a0 0x7f3f4c158950 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c037040 tx=0x7f3f3c003ba0 comp rx=0 tx=0).stop 2023-10-19T12:45:21.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.715+0000 7f3f545f0640 1 -- 172.21.15.57:0/3914054065 shutdown_connections 2023-10-19T12:45:21.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.715+0000 7f3f545f0640 1 --2- 172.21.15.57:0/3914054065 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f3f3806ae40 0x7f3f3806d300 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:21.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.715+0000 7f3f545f0640 1 --2- 172.21.15.57:0/3914054065 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3f4c15b2a0 0x7f3f4c158950 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:21.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.715+0000 7f3f545f0640 1 --2- 172.21.15.57:0/3914054065 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3f4c0cf340 0x7f3f4c09e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:21.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.715+0000 7f3f545f0640 1 --2- 172.21.15.57:0/3914054065 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3f4c0ceb40 0x7f3f4c157e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:21.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.715+0000 7f3f545f0640 1 -- 172.21.15.57:0/3914054065 >> 172.21.15.57:0/3914054065 conn(0x7f3f4c0d0d60 msgr2=0x7f3f4c15a0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:21.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.715+0000 7f3f545f0640 1 -- 172.21.15.57:0/3914054065 shutdown_connections 2023-10-19T12:45:21.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.715+0000 7f3f545f0640 1 -- 172.21.15.57:0/3914054065 wait complete. 2023-10-19T12:45:21.729 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi057.front.sepia.ceph.com... 2023-10-19T12:45:21.729 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:45:21.730 DEBUG:teuthology.orchestra.run.smithi057:> sudo fusermount -u /home/ubuntu/cephtest/mnt.2 2023-10-19T12:45:21.761 INFO:teuthology.orchestra.run:waiting for 300 2023-10-19T12:45:21.761 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi057.front.sepia.ceph.com 2023-10-19T12:45:21.761 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:45:21.762 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-10-19T12:45:21.815 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-19T12:45:21.816 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T12:45:21.834 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-19T12:45:21.835 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T12:45:21.892 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T12:45:22.000 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.997+0000 7faff0d7d640 1 -- 172.21.15.57:0/1123418105 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fafec0cf340 msgr2=0x7fafec0d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:22.001 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.997+0000 7faff0d7d640 1 --2- 172.21.15.57:0/1123418105 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fafec0cf340 0x7fafec0d1910 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fafdc009830 tx=0x7fafdc02bcd0 comp rx=0 tx=0).stop 2023-10-19T12:45:22.001 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.997+0000 7faff0d7d640 1 -- 172.21.15.57:0/1123418105 shutdown_connections 2023-10-19T12:45:22.001 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.997+0000 7faff0d7d640 1 --2- 172.21.15.57:0/1123418105 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fafec0cf340 0x7fafec0d1910 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:22.001 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.997+0000 7faff0d7d640 1 -- 172.21.15.57:0/1123418105 >> 172.21.15.57:0/1123418105 conn(0x7fafec0d0d60 msgr2=0x7fafec0d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:22.001 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.997+0000 7faff0d7d640 1 -- 172.21.15.57:0/1123418105 shutdown_connections 2023-10-19T12:45:22.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.997+0000 7faff0d7d640 1 -- 172.21.15.57:0/1123418105 wait complete. 2023-10-19T12:45:22.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.999+0000 7faff0d7d640 1 Processor -- start 2023-10-19T12:45:22.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.999+0000 7faff0d7d640 1 -- start start 2023-10-19T12:45:22.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.999+0000 7faff0d7d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fafec0ceb40 0x7fafec1eedc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:22.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.999+0000 7faff0d7d640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fafec0cf340 0x7fafec0b4800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:22.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.999+0000 7faff0d7d640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fafec15b2a0 0x7fafec1f4e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:22.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.999+0000 7faff0d7d640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafec0b3c90 con 0x7fafec0ceb40 2023-10-19T12:45:22.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.999+0000 7faff0d7d640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafec0b5650 con 0x7fafec15b2a0 2023-10-19T12:45:22.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.999+0000 7faff0d7d640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafec0b5930 con 0x7fafec0cf340 2023-10-19T12:45:22.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.999+0000 7fafea59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fafec0ceb40 0x7fafec1eedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:22.006 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.999+0000 7fafea59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fafec0ceb40 0x7fafec1eedc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50454/0 (socket says 172.21.15.57:50454) 2023-10-19T12:45:22.006 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:21.999+0000 7fafea59c640 1 -- 172.21.15.57:0/2932151587 learned_addr learned my addr 172.21.15.57:0/2932151587 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:22.006 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.000+0000 7fafea59c640 1 -- 172.21.15.57:0/2932151587 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fafec0cf340 msgr2=0x7fafec0b4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:22.006 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.000+0000 7fafea59c640 1 --2- 172.21.15.57:0/2932151587 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fafec0cf340 0x7fafec0b4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:22.006 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.000+0000 7fafea59c640 1 -- 172.21.15.57:0/2932151587 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fafec15b2a0 msgr2=0x7fafec1f4e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:22.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.000+0000 7fafea59c640 1 --2- 172.21.15.57:0/2932151587 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fafec15b2a0 0x7fafec1f4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:22.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.000+0000 7fafea59c640 1 -- 172.21.15.57:0/2932151587 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafdc009490 con 0x7fafec0ceb40 2023-10-19T12:45:22.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.000+0000 7fafea59c640 1 --2- 172.21.15.57:0/2932151587 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fafec0ceb40 0x7fafec1eedc0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fafdc009780 tx=0x7fafdc0038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:22.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.000+0000 7fafe2ffd640 1 -- 172.21.15.57:0/2932151587 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafdc034440 con 0x7fafec0ceb40 2023-10-19T12:45:22.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.000+0000 7fafe2ffd640 1 -- 172.21.15.57:0/2932151587 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fafdc034690 con 0x7fafec0ceb40 2023-10-19T12:45:22.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.000+0000 7faff0d7d640 1 -- 172.21.15.57:0/2932151587 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafec0b5c10 con 0x7fafec0ceb40 2023-10-19T12:45:22.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.000+0000 7fafe2ffd640 1 -- 172.21.15.57:0/2932151587 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafdc034aa0 con 0x7fafec0ceb40 2023-10-19T12:45:22.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.000+0000 7faff0d7d640 1 -- 172.21.15.57:0/2932151587 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafec0b61d0 con 0x7fafec0ceb40 2023-10-19T12:45:22.010 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.001+0000 7faff0d7d640 1 -- 172.21.15.57:0/2932151587 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafec0b63c0 con 0x7fafec0ceb40 2023-10-19T12:45:22.011 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.001+0000 7fafe2ffd640 1 -- 172.21.15.57:0/2932151587 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fafdc03b040 con 0x7fafec0ceb40 2023-10-19T12:45:22.011 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.002+0000 7fafe2ffd640 1 --2- 172.21.15.57:0/2932151587 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fafcc06ae40 0x7fafcc06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:22.011 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.002+0000 7fafe2ffd640 1 -- 172.21.15.57:0/2932151587 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fafdc0abff0 con 0x7fafec0ceb40 2023-10-19T12:45:22.011 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.002+0000 7fafe3fff640 1 --2- 172.21.15.57:0/2932151587 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fafcc06ae40 0x7fafcc06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:22.012 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.003+0000 7fafe3fff640 1 --2- 172.21.15.57:0/2932151587 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fafcc06ae40 0x7fafcc06d300 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fafec15e9f0 tx=0x7fafd4006ba0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:22.012 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.006+0000 7fafe2ffd640 1 -- 172.21.15.57:0/2932151587 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fafdc076d70 con 0x7fafec0ceb40 2023-10-19T12:45:22.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.186+0000 7faff0d7d640 1 -- 172.21.15.57:0/2932151587 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fafec0690e0 con 0x7fafec0ceb40 2023-10-19T12:45:22.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.187+0000 7fafe2ffd640 1 -- 172.21.15.57:0/2932151587 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18748 (secure 0 0 0) 0x7fafdc041090 con 0x7fafec0ceb40 2023-10-19T12:45:22.191 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:45:22.191 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":27,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T12:44:58.978403+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-19T12:44:16.517510+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-19T12:44:17.398297+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-19T12:44:21.129403+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.23","mappings":[{"from":3,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"3.39","mappings":[{"from":7,"to":1}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.1d","mappings":[{"from":1,"to":5}]},{"pgid":"4.1e","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T12:45:22.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.189+0000 7faff0d7d640 1 -- 172.21.15.57:0/2932151587 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fafcc06ae40 msgr2=0x7fafcc06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:22.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.189+0000 7faff0d7d640 1 --2- 172.21.15.57:0/2932151587 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fafcc06ae40 0x7fafcc06d300 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fafec15e9f0 tx=0x7fafd4006ba0 comp rx=0 tx=0).stop 2023-10-19T12:45:22.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.189+0000 7faff0d7d640 1 -- 172.21.15.57:0/2932151587 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fafec0ceb40 msgr2=0x7fafec1eedc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:22.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.189+0000 7faff0d7d640 1 --2- 172.21.15.57:0/2932151587 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fafec0ceb40 0x7fafec1eedc0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fafdc009780 tx=0x7fafdc0038a0 comp rx=0 tx=0).stop 2023-10-19T12:45:22.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.189+0000 7faff0d7d640 1 -- 172.21.15.57:0/2932151587 shutdown_connections 2023-10-19T12:45:22.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.189+0000 7faff0d7d640 1 --2- 172.21.15.57:0/2932151587 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fafcc06ae40 0x7fafcc06d300 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:22.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.189+0000 7faff0d7d640 1 --2- 172.21.15.57:0/2932151587 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fafec15b2a0 0x7fafec1f4e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:22.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.189+0000 7faff0d7d640 1 --2- 172.21.15.57:0/2932151587 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fafec0cf340 0x7fafec0b4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:22.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.189+0000 7faff0d7d640 1 --2- 172.21.15.57:0/2932151587 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fafec0ceb40 0x7fafec1eedc0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:22.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.189+0000 7faff0d7d640 1 -- 172.21.15.57:0/2932151587 >> 172.21.15.57:0/2932151587 conn(0x7fafec0d0d60 msgr2=0x7fafec0c9e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:22.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.189+0000 7faff0d7d640 1 -- 172.21.15.57:0/2932151587 shutdown_connections 2023-10-19T12:45:22.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.189+0000 7faff0d7d640 1 -- 172.21.15.57:0/2932151587 wait complete. 2023-10-19T12:45:22.203 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-19T12:44:07.371186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-19T12:44:16.517510+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-19T12:44:17.398297+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-19T12:44:21.129403+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-19T12:45:22.204 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-19T12:45:22.316 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.313+0000 7f49be226640 1 -- 172.21.15.57:0/259928723 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49b80abc10 msgr2=0x7f49b80c9b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:22.316 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.313+0000 7f49be226640 1 --2- 172.21.15.57:0/259928723 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49b80abc10 0x7f49b80c9b60 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f49b0005fd0 tx=0x7f49b002c050 comp rx=0 tx=0).stop 2023-10-19T12:45:22.317 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.313+0000 7f49be226640 1 -- 172.21.15.57:0/259928723 shutdown_connections 2023-10-19T12:45:22.317 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.313+0000 7f49be226640 1 --2- 172.21.15.57:0/259928723 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49b80abc10 0x7f49b80c9b60 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:22.317 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.313+0000 7f49be226640 1 --2- 172.21.15.57:0/259928723 >> v2:172.21.15.179:3300/0 conn(0x7f49b806af00 0x7f49b806b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:22.317 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.313+0000 7f49be226640 1 -- 172.21.15.57:0/259928723 >> 172.21.15.57:0/259928723 conn(0x7f49b8069930 msgr2=0x7f49b8069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:22.318 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.313+0000 7f49be226640 1 -- 172.21.15.57:0/259928723 shutdown_connections 2023-10-19T12:45:22.318 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.313+0000 7f49be226640 1 -- 172.21.15.57:0/259928723 wait complete. 2023-10-19T12:45:22.320 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.314+0000 7f49be226640 1 Processor -- start 2023-10-19T12:45:22.320 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.314+0000 7f49be226640 1 -- start start 2023-10-19T12:45:22.320 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.315+0000 7f49be226640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f49b806af00 0x7f49b81f1d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:22.321 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.315+0000 7f49be226640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49b80abc10 0x7f49b80b08d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:22.321 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.315+0000 7f49be226640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f49b80c96b0 0x7f49b80b0e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:22.321 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.315+0000 7f49be226640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49b806f500 con 0x7f49b80abc10 2023-10-19T12:45:22.321 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.315+0000 7f49be226640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49b81f22d0 con 0x7f49b80c96b0 2023-10-19T12:45:22.321 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.315+0000 7f49be226640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49b80a6f70 con 0x7f49b806af00 2023-10-19T12:45:22.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.315+0000 7f49b6ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49b80abc10 0x7f49b80b08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:22.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.315+0000 7f49b6ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49b80abc10 0x7f49b80b08d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50482/0 (socket says 172.21.15.57:50482) 2023-10-19T12:45:22.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.315+0000 7f49b6ffd640 1 -- 172.21.15.57:0/3724924694 learned_addr learned my addr 172.21.15.57:0/3724924694 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:22.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.315+0000 7f49b77fe640 1 --2- 172.21.15.57:0/3724924694 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f49b806af00 0x7f49b81f1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:22.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.315+0000 7f49b7fff640 1 --2- 172.21.15.57:0/3724924694 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f49b80c96b0 0x7f49b80b0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:22.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.315+0000 7f49b6ffd640 1 -- 172.21.15.57:0/3724924694 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f49b806af00 msgr2=0x7f49b81f1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:22.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.315+0000 7f49b6ffd640 1 --2- 172.21.15.57:0/3724924694 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f49b806af00 0x7f49b81f1d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:22.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.315+0000 7f49b6ffd640 1 -- 172.21.15.57:0/3724924694 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f49b80c96b0 msgr2=0x7f49b80b0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:22.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.315+0000 7f49b6ffd640 1 --2- 172.21.15.57:0/3724924694 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f49b80c96b0 0x7f49b80b0e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:22.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.315+0000 7f49b6ffd640 1 -- 172.21.15.57:0/3724924694 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49b0009840 con 0x7f49b80abc10 2023-10-19T12:45:22.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.315+0000 7f49b7fff640 1 --2- 172.21.15.57:0/3724924694 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f49b80c96b0 0x7f49b80b0e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:45:22.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.315+0000 7f49b6ffd640 1 --2- 172.21.15.57:0/3724924694 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49b80abc10 0x7f49b80b08d0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f49ac00c3c0 tx=0x7f49ac00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:22.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.316+0000 7f499ffff640 1 -- 172.21.15.57:0/3724924694 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49ac004230 con 0x7f49b80abc10 2023-10-19T12:45:22.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.316+0000 7f499ffff640 1 -- 172.21.15.57:0/3724924694 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f49ac0044a0 con 0x7f49b80abc10 2023-10-19T12:45:22.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.316+0000 7f49be226640 1 -- 172.21.15.57:0/3724924694 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49b80a7250 con 0x7f49b80abc10 2023-10-19T12:45:22.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.316+0000 7f499ffff640 1 -- 172.21.15.57:0/3724924694 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49ac003850 con 0x7f49b80abc10 2023-10-19T12:45:22.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.316+0000 7f49be226640 1 -- 172.21.15.57:0/3724924694 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49b806f120 con 0x7f49b80abc10 2023-10-19T12:45:22.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.317+0000 7f49be226640 1 -- 172.21.15.57:0/3724924694 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49b80690e0 con 0x7f49b80abc10 2023-10-19T12:45:22.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.317+0000 7f499ffff640 1 -- 172.21.15.57:0/3724924694 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f49ac0039b0 con 0x7f49b80abc10 2023-10-19T12:45:22.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.317+0000 7f499ffff640 1 --2- 172.21.15.57:0/3724924694 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f49a406ae40 0x7f49a406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:22.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.318+0000 7f499ffff640 1 -- 172.21.15.57:0/3724924694 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f49ac089100 con 0x7f49b80abc10 2023-10-19T12:45:22.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.318+0000 7f49b77fe640 1 --2- 172.21.15.57:0/3724924694 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f49a406ae40 0x7f49a406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:22.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.318+0000 7f49b77fe640 1 --2- 172.21.15.57:0/3724924694 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f49a406ae40 0x7f49a406d300 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f49a8009890 tx=0x7f49a8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:22.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.322+0000 7f499ffff640 1 -- 172.21.15.57:0/3724924694 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f49ac053de0 con 0x7f49b80abc10 2023-10-19T12:45:22.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.496+0000 7f49be226640 1 -- 172.21.15.57:0/3724924694 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f49b8158f90 con 0x7f49b80abc10 2023-10-19T12:45:22.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.496+0000 7f499ffff640 1 -- 172.21.15.57:0/3724924694 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7f49ac017020 con 0x7f49b80abc10 2023-10-19T12:45:22.500 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-19T12:45:22.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.498+0000 7f49be226640 1 -- 172.21.15.57:0/3724924694 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f49a406ae40 msgr2=0x7f49a406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:22.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.498+0000 7f49be226640 1 --2- 172.21.15.57:0/3724924694 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f49a406ae40 0x7f49a406d300 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f49a8009890 tx=0x7f49a8008040 comp rx=0 tx=0).stop 2023-10-19T12:45:22.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.498+0000 7f49be226640 1 -- 172.21.15.57:0/3724924694 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49b80abc10 msgr2=0x7f49b80b08d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:22.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.498+0000 7f49be226640 1 --2- 172.21.15.57:0/3724924694 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49b80abc10 0x7f49b80b08d0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f49ac00c3c0 tx=0x7f49ac00c880 comp rx=0 tx=0).stop 2023-10-19T12:45:22.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.498+0000 7f49be226640 1 -- 172.21.15.57:0/3724924694 shutdown_connections 2023-10-19T12:45:22.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.498+0000 7f49be226640 1 --2- 172.21.15.57:0/3724924694 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f49a406ae40 0x7f49a406d300 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:22.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.498+0000 7f49be226640 1 --2- 172.21.15.57:0/3724924694 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f49b80c96b0 0x7f49b80b0e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:22.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.498+0000 7f49be226640 1 --2- 172.21.15.57:0/3724924694 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49b80abc10 0x7f49b80b08d0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:22.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.498+0000 7f49be226640 1 --2- 172.21.15.57:0/3724924694 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f49b806af00 0x7f49b81f1d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:22.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.498+0000 7f49be226640 1 -- 172.21.15.57:0/3724924694 >> 172.21.15.57:0/3724924694 conn(0x7f49b8069930 msgr2=0x7f49b8069d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:22.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.498+0000 7f49be226640 1 -- 172.21.15.57:0/3724924694 shutdown_connections 2023-10-19T12:45:22.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.498+0000 7f49be226640 1 -- 172.21.15.57:0/3724924694 wait complete. 2023-10-19T12:45:22.512 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-19T12:45:22.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.622+0000 7fcfd359e640 1 -- 172.21.15.57:0/1752492498 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcfb40047c0 msgr2=0x7fcfb4004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:22.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.622+0000 7fcfd359e640 1 --2- 172.21.15.57:0/1752492498 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcfb40047c0 0x7fcfb4004c40 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fcfc0009b50 tx=0x7fcfb40057f0 comp rx=0 tx=0).stop 2023-10-19T12:45:22.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.622+0000 7fcfd359e640 1 -- 172.21.15.57:0/1752492498 shutdown_connections 2023-10-19T12:45:22.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.622+0000 7fcfd359e640 1 --2- 172.21.15.57:0/1752492498 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcfb40047c0 0x7fcfb4004c40 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:22.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.622+0000 7fcfd359e640 1 -- 172.21.15.57:0/1752492498 >> 172.21.15.57:0/1752492498 conn(0x7fcfcc0d0d60 msgr2=0x7fcfcc0d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:22.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.622+0000 7fcfd359e640 1 -- 172.21.15.57:0/1752492498 shutdown_connections 2023-10-19T12:45:22.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.622+0000 7fcfd359e640 1 -- 172.21.15.57:0/1752492498 wait complete. 2023-10-19T12:45:22.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.623+0000 7fcfd359e640 1 Processor -- start 2023-10-19T12:45:22.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.623+0000 7fcfd359e640 1 -- start start 2023-10-19T12:45:22.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.624+0000 7fcfd359e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcfb40047c0 0x7fcfcc0cad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:22.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.624+0000 7fcfd359e640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcfcc14ac50 0x7fcfcc1f0b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:22.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.624+0000 7fcfd359e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcfcc0b0ea0 0x7fcfcc0b1320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:22.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.624+0000 7fcfd359e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfcc148ed0 con 0x7fcfcc0b0ea0 2023-10-19T12:45:22.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.624+0000 7fcfd359e640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfcc148bc0 con 0x7fcfb40047c0 2023-10-19T12:45:22.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.624+0000 7fcfd359e640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfcc0b1860 con 0x7fcfcc14ac50 2023-10-19T12:45:22.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.624+0000 7fcfd1b3b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcfcc0b0ea0 0x7fcfcc0b1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:22.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.624+0000 7fcfd1b3b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcfcc0b0ea0 0x7fcfcc0b1320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50494/0 (socket says 172.21.15.57:50494) 2023-10-19T12:45:22.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.624+0000 7fcfd1b3b640 1 -- 172.21.15.57:0/181372131 learned_addr learned my addr 172.21.15.57:0/181372131 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:22.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.624+0000 7fcfd0b39640 1 --2- 172.21.15.57:0/181372131 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcfcc14ac50 0x7fcfcc1f0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:22.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.624+0000 7fcfd133a640 1 --2- 172.21.15.57:0/181372131 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcfb40047c0 0x7fcfcc0cad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:22.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.624+0000 7fcfd1b3b640 1 -- 172.21.15.57:0/181372131 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcfcc14ac50 msgr2=0x7fcfcc1f0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:22.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.624+0000 7fcfd1b3b640 1 --2- 172.21.15.57:0/181372131 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcfcc14ac50 0x7fcfcc1f0b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:22.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.624+0000 7fcfd1b3b640 1 -- 172.21.15.57:0/181372131 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcfb40047c0 msgr2=0x7fcfcc0cad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:22.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.624+0000 7fcfd1b3b640 1 --2- 172.21.15.57:0/181372131 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcfb40047c0 0x7fcfcc0cad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:22.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.624+0000 7fcfd1b3b640 1 -- 172.21.15.57:0/181372131 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfcc14b410 con 0x7fcfcc0b0ea0 2023-10-19T12:45:22.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.624+0000 7fcfd133a640 1 --2- 172.21.15.57:0/181372131 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcfb40047c0 0x7fcfcc0cad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:45:22.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.625+0000 7fcfd1b3b640 1 --2- 172.21.15.57:0/181372131 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcfcc0b0ea0 0x7fcfcc0b1320 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc0b6e10 tx=0x7fcfbc006720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:22.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.625+0000 7fcfc9ffb640 1 -- 172.21.15.57:0/181372131 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfbc00fa20 con 0x7fcfcc0b0ea0 2023-10-19T12:45:22.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.625+0000 7fcfc9ffb640 1 -- 172.21.15.57:0/181372131 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fcfbc00fba0 con 0x7fcfcc0b0ea0 2023-10-19T12:45:22.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.625+0000 7fcfd359e640 1 -- 172.21.15.57:0/181372131 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfcc14b6f0 con 0x7fcfcc0b0ea0 2023-10-19T12:45:22.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.625+0000 7fcfc9ffb640 1 -- 172.21.15.57:0/181372131 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfbc011430 con 0x7fcfcc0b0ea0 2023-10-19T12:45:22.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.625+0000 7fcfd359e640 1 -- 172.21.15.57:0/181372131 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfcc1f10c0 con 0x7fcfcc0b0ea0 2023-10-19T12:45:22.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.626+0000 7fcfd359e640 1 -- 172.21.15.57:0/181372131 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfcc149010 con 0x7fcfcc0b0ea0 2023-10-19T12:45:22.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.627+0000 7fcfc9ffb640 1 -- 172.21.15.57:0/181372131 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fcfbc011590 con 0x7fcfcc0b0ea0 2023-10-19T12:45:22.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.627+0000 7fcfc9ffb640 1 --2- 172.21.15.57:0/181372131 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fcfb806ae70 0x7fcfb806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:22.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.627+0000 7fcfc9ffb640 1 -- 172.21.15.57:0/181372131 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fcfbc08a7e0 con 0x7fcfcc0b0ea0 2023-10-19T12:45:22.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.628+0000 7fcfd133a640 1 --2- 172.21.15.57:0/181372131 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fcfb806ae70 0x7fcfb806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:22.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.628+0000 7fcfd133a640 1 --2- 172.21.15.57:0/181372131 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fcfb806ae70 0x7fcfb806d330 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc0682a0 tx=0x7fcfc4006b30 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:22.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.631+0000 7fcfc9ffb640 1 -- 172.21.15.57:0/181372131 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fcfcc149010 con 0x7fcfcc0b0ea0 2023-10-19T12:45:22.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.805+0000 7fcfd359e640 1 -- 172.21.15.57:0/181372131 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fcfcc149010 con 0x7fcfcc0b0ea0 2023-10-19T12:45:22.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.806+0000 7fcfc9ffb640 1 -- 172.21.15.57:0/181372131 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7fcfcc149010 con 0x7fcfcc0b0ea0 2023-10-19T12:45:22.809 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2023-10-19T12:45:22.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.807+0000 7fcfd359e640 1 -- 172.21.15.57:0/181372131 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fcfb806ae70 msgr2=0x7fcfb806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:22.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.807+0000 7fcfd359e640 1 --2- 172.21.15.57:0/181372131 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fcfb806ae70 0x7fcfb806d330 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc0682a0 tx=0x7fcfc4006b30 comp rx=0 tx=0).stop 2023-10-19T12:45:22.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.807+0000 7fcfd359e640 1 -- 172.21.15.57:0/181372131 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcfcc0b0ea0 msgr2=0x7fcfcc0b1320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:22.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.807+0000 7fcfd359e640 1 --2- 172.21.15.57:0/181372131 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcfcc0b0ea0 0x7fcfcc0b1320 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc0b6e10 tx=0x7fcfbc006720 comp rx=0 tx=0).stop 2023-10-19T12:45:22.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.807+0000 7fcfd359e640 1 -- 172.21.15.57:0/181372131 shutdown_connections 2023-10-19T12:45:22.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.807+0000 7fcfd359e640 1 --2- 172.21.15.57:0/181372131 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fcfb806ae70 0x7fcfb806d330 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:22.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.807+0000 7fcfd359e640 1 --2- 172.21.15.57:0/181372131 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcfcc0b0ea0 0x7fcfcc0b1320 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:22.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.807+0000 7fcfd359e640 1 --2- 172.21.15.57:0/181372131 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcfcc14ac50 0x7fcfcc1f0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:22.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.807+0000 7fcfd359e640 1 --2- 172.21.15.57:0/181372131 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcfb40047c0 0x7fcfcc0cad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:22.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.807+0000 7fcfd359e640 1 -- 172.21.15.57:0/181372131 >> 172.21.15.57:0/181372131 conn(0x7fcfcc0d0d60 msgr2=0x7fcfcc0ad8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:22.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.808+0000 7fcfd359e640 1 -- 172.21.15.57:0/181372131 shutdown_connections 2023-10-19T12:45:22.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.808+0000 7fcfd359e640 1 -- 172.21.15.57:0/181372131 wait complete. 2023-10-19T12:45:22.821 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-19T12:45:22.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.930+0000 7f13ed1a0640 1 -- 172.21.15.57:0/3308625477 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f13e80ce2f0 msgr2=0x7f13e80c5550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:22.934 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.930+0000 7f13ed1a0640 1 --2- 172.21.15.57:0/3308625477 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f13e80ce2f0 0x7f13e80c5550 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f13e0005fd0 tx=0x7f13e002c0e0 comp rx=0 tx=0).stop 2023-10-19T12:45:22.934 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.930+0000 7f13ed1a0640 1 -- 172.21.15.57:0/3308625477 shutdown_connections 2023-10-19T12:45:22.934 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.930+0000 7f13ed1a0640 1 --2- 172.21.15.57:0/3308625477 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f13e80cda60 0x7f13e80c5010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:22.934 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.930+0000 7f13ed1a0640 1 --2- 172.21.15.57:0/3308625477 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f13e80ce2f0 0x7f13e80c5550 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:22.935 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.930+0000 7f13ed1a0640 1 -- 172.21.15.57:0/3308625477 >> 172.21.15.57:0/3308625477 conn(0x7f13e80d0c20 msgr2=0x7f13e80d1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:22.935 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.930+0000 7f13ed1a0640 1 -- 172.21.15.57:0/3308625477 shutdown_connections 2023-10-19T12:45:22.935 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.930+0000 7f13ed1a0640 1 -- 172.21.15.57:0/3308625477 wait complete. 2023-10-19T12:45:22.937 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.931+0000 7f13ed1a0640 1 Processor -- start 2023-10-19T12:45:22.937 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.931+0000 7f13ed1a0640 1 -- start start 2023-10-19T12:45:22.937 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.932+0000 7f13ed1a0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f13e80cda60 0x7f13e8153fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:22.937 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.932+0000 7f13ed1a0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f13e80ce2f0 0x7f13e80ca370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:22.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.932+0000 7f13ed1a0640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f13e815b2f0 0x7f13e81e6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:22.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.932+0000 7f13ed1a0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13e80b3ed0 con 0x7f13e80ce2f0 2023-10-19T12:45:22.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.932+0000 7f13ed1a0640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13e80b41b0 con 0x7f13e80cda60 2023-10-19T12:45:22.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.932+0000 7f13ed1a0640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13e80b4490 con 0x7f13e815b2f0 2023-10-19T12:45:22.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.932+0000 7f13e77fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f13e80ce2f0 0x7f13e80ca370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:22.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.932+0000 7f13e77fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f13e80ce2f0 0x7f13e80ca370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50506/0 (socket says 172.21.15.57:50506) 2023-10-19T12:45:22.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.932+0000 7f13e77fe640 1 -- 172.21.15.57:0/2656868207 learned_addr learned my addr 172.21.15.57:0/2656868207 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:22.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.933+0000 7f13ec99f640 1 --2- 172.21.15.57:0/2656868207 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f13e815b2f0 0x7f13e81e6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:22.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.933+0000 7f13e7fff640 1 --2- 172.21.15.57:0/2656868207 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f13e80cda60 0x7f13e8153fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:22.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.933+0000 7f13e77fe640 1 -- 172.21.15.57:0/2656868207 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f13e815b2f0 msgr2=0x7f13e81e6280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:22.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.933+0000 7f13e77fe640 1 --2- 172.21.15.57:0/2656868207 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f13e815b2f0 0x7f13e81e6280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:22.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.933+0000 7f13e77fe640 1 -- 172.21.15.57:0/2656868207 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f13e80cda60 msgr2=0x7f13e8153fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:22.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.933+0000 7f13e77fe640 1 --2- 172.21.15.57:0/2656868207 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f13e80cda60 0x7f13e8153fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:22.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.933+0000 7f13e77fe640 1 -- 172.21.15.57:0/2656868207 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13e0009840 con 0x7f13e80ce2f0 2023-10-19T12:45:22.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.933+0000 7f13e77fe640 1 --2- 172.21.15.57:0/2656868207 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f13e80ce2f0 0x7f13e80ca370 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f13d800c970 tx=0x7f13d800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:22.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.933+0000 7f13e4ff9640 1 -- 172.21.15.57:0/2656868207 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13d8007bb0 con 0x7f13e80ce2f0 2023-10-19T12:45:22.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.933+0000 7f13e4ff9640 1 -- 172.21.15.57:0/2656868207 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f13d8007e00 con 0x7f13e80ce2f0 2023-10-19T12:45:22.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.933+0000 7f13e4ff9640 1 -- 172.21.15.57:0/2656868207 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13d800f6b0 con 0x7f13e80ce2f0 2023-10-19T12:45:22.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.933+0000 7f13ed1a0640 1 -- 172.21.15.57:0/2656868207 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13e80b4770 con 0x7f13e80ce2f0 2023-10-19T12:45:22.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.933+0000 7f13ed1a0640 1 -- 172.21.15.57:0/2656868207 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13e81544e0 con 0x7f13e80ce2f0 2023-10-19T12:45:22.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.934+0000 7f13ed1a0640 1 -- 172.21.15.57:0/2656868207 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13e8066960 con 0x7f13e80ce2f0 2023-10-19T12:45:22.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.934+0000 7f13e4ff9640 1 -- 172.21.15.57:0/2656868207 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f13d800f810 con 0x7f13e80ce2f0 2023-10-19T12:45:22.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.935+0000 7f13e4ff9640 1 --2- 172.21.15.57:0/2656868207 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f13d406ae40 0x7f13d406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:22.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.935+0000 7f13e4ff9640 1 -- 172.21.15.57:0/2656868207 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f13d808a270 con 0x7f13e80ce2f0 2023-10-19T12:45:22.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.935+0000 7f13e7fff640 1 --2- 172.21.15.57:0/2656868207 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f13d406ae40 0x7f13d406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:22.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.936+0000 7f13e7fff640 1 --2- 172.21.15.57:0/2656868207 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f13d406ae40 0x7f13d406d300 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f13dc0046b0 tx=0x7f13dc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:22.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:22.939+0000 7f13e4ff9640 1 -- 172.21.15.57:0/2656868207 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f13d8054ed0 con 0x7f13e80ce2f0 2023-10-19T12:45:23.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.116+0000 7f13ed1a0640 1 -- 172.21.15.57:0/2656868207 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f13e804da70 con 0x7f13e80ce2f0 2023-10-19T12:45:23.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.117+0000 7f13e4ff9640 1 -- 172.21.15.57:0/2656868207 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7f13d80584f0 con 0x7f13e80ce2f0 2023-10-19T12:45:23.120 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 64 2023-10-19T12:45:23.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.119+0000 7f13ed1a0640 1 -- 172.21.15.57:0/2656868207 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f13d406ae40 msgr2=0x7f13d406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:23.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.119+0000 7f13ed1a0640 1 --2- 172.21.15.57:0/2656868207 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f13d406ae40 0x7f13d406d300 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f13dc0046b0 tx=0x7f13dc009040 comp rx=0 tx=0).stop 2023-10-19T12:45:23.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.119+0000 7f13ed1a0640 1 -- 172.21.15.57:0/2656868207 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f13e80ce2f0 msgr2=0x7f13e80ca370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:23.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.119+0000 7f13ed1a0640 1 --2- 172.21.15.57:0/2656868207 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f13e80ce2f0 0x7f13e80ca370 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f13d800c970 tx=0x7f13d800ce30 comp rx=0 tx=0).stop 2023-10-19T12:45:23.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.119+0000 7f13ed1a0640 1 -- 172.21.15.57:0/2656868207 shutdown_connections 2023-10-19T12:45:23.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.119+0000 7f13ed1a0640 1 --2- 172.21.15.57:0/2656868207 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f13d406ae40 0x7f13d406d300 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:23.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.119+0000 7f13ed1a0640 1 --2- 172.21.15.57:0/2656868207 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f13e815b2f0 0x7f13e81e6280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:23.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.119+0000 7f13ed1a0640 1 --2- 172.21.15.57:0/2656868207 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f13e80ce2f0 0x7f13e80ca370 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:23.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.119+0000 7f13ed1a0640 1 --2- 172.21.15.57:0/2656868207 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f13e80cda60 0x7f13e8153fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:23.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.119+0000 7f13ed1a0640 1 -- 172.21.15.57:0/2656868207 >> 172.21.15.57:0/2656868207 conn(0x7f13e80d0c20 msgr2=0x7f13e80c63c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:23.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.119+0000 7f13ed1a0640 1 -- 172.21.15.57:0/2656868207 shutdown_connections 2023-10-19T12:45:23.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.119+0000 7f13ed1a0640 1 -- 172.21.15.57:0/2656868207 wait complete. 2023-10-19T12:45:23.132 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-19T12:45:23.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.241+0000 7f4a989b6640 1 -- 172.21.15.57:0/1852169508 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a940cdb30 msgr2=0x7f4a940cdfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:23.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.241+0000 7f4a989b6640 1 --2- 172.21.15.57:0/1852169508 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a940cdb30 0x7f4a940cdfb0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f4a940ceab0 tx=0x7f4a7c02bd10 comp rx=0 tx=0).stop 2023-10-19T12:45:23.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.241+0000 7f4a989b6640 1 -- 172.21.15.57:0/1852169508 shutdown_connections 2023-10-19T12:45:23.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.241+0000 7f4a989b6640 1 --2- 172.21.15.57:0/1852169508 >> v2:172.21.15.179:3300/0 conn(0x7f4a9415ccf0 0x7f4a9415d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:23.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.241+0000 7f4a989b6640 1 --2- 172.21.15.57:0/1852169508 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a940cdb30 0x7f4a940cdfb0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:23.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.241+0000 7f4a989b6640 1 --2- 172.21.15.57:0/1852169508 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4a94159060 0x7f4a9406d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:23.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.241+0000 7f4a989b6640 1 -- 172.21.15.57:0/1852169508 >> 172.21.15.57:0/1852169508 conn(0x7f4a940b0670 msgr2=0x7f4a940b0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:23.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.241+0000 7f4a989b6640 1 -- 172.21.15.57:0/1852169508 shutdown_connections 2023-10-19T12:45:23.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.241+0000 7f4a989b6640 1 -- 172.21.15.57:0/1852169508 wait complete. 2023-10-19T12:45:23.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.242+0000 7f4a989b6640 1 Processor -- start 2023-10-19T12:45:23.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.242+0000 7f4a989b6640 1 -- start start 2023-10-19T12:45:23.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.243+0000 7f4a989b6640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4a940cdb30 0x7f4a941ef840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:23.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.243+0000 7f4a989b6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a94159060 0x7f4a941579d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:23.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.243+0000 7f4a989b6640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4a9415ccf0 0x7f4a941f23b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:23.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.243+0000 7f4a989b6640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a941eedc0 con 0x7f4a94159060 2023-10-19T12:45:23.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.243+0000 7f4a989b6640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a941efd80 con 0x7f4a9415ccf0 2023-10-19T12:45:23.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.243+0000 7f4a989b6640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a941f2e50 con 0x7f4a940cdb30 2023-10-19T12:45:23.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.243+0000 7f4a91d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a94159060 0x7f4a941579d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:23.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.243+0000 7f4a91d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a94159060 0x7f4a941579d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50530/0 (socket says 172.21.15.57:50530) 2023-10-19T12:45:23.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.243+0000 7f4a91d9b640 1 -- 172.21.15.57:0/684025608 learned_addr learned my addr 172.21.15.57:0/684025608 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:23.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.244+0000 7f4a9259c640 1 --2- 172.21.15.57:0/684025608 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4a940cdb30 0x7f4a941ef840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:23.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.244+0000 7f4a91d9b640 1 -- 172.21.15.57:0/684025608 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4a940cdb30 msgr2=0x7f4a941ef840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:23.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.244+0000 7f4a91d9b640 1 --2- 172.21.15.57:0/684025608 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4a940cdb30 0x7f4a941ef840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:23.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.244+0000 7f4a91d9b640 1 -- 172.21.15.57:0/684025608 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4a9415ccf0 msgr2=0x7f4a941f23b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:23.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.244+0000 7f4a91d9b640 1 --2- 172.21.15.57:0/684025608 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4a9415ccf0 0x7f4a941f23b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:23.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.244+0000 7f4a91d9b640 1 -- 172.21.15.57:0/684025608 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a7c009560 con 0x7f4a94159060 2023-10-19T12:45:23.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.244+0000 7f4a91d9b640 1 --2- 172.21.15.57:0/684025608 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a94159060 0x7f4a941579d0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f4a7c009850 tx=0x7f4a7c02da40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:23.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.244+0000 7f4a8affd640 1 -- 172.21.15.57:0/684025608 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a7c003800 con 0x7f4a94159060 2023-10-19T12:45:23.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.244+0000 7f4a8affd640 1 -- 172.21.15.57:0/684025608 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f4a7c003a50 con 0x7f4a94159060 2023-10-19T12:45:23.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.244+0000 7f4a989b6640 1 -- 172.21.15.57:0/684025608 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a941f3130 con 0x7f4a94159060 2023-10-19T12:45:23.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.244+0000 7f4a8affd640 1 -- 172.21.15.57:0/684025608 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a7c003e20 con 0x7f4a94159060 2023-10-19T12:45:23.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.244+0000 7f4a989b6640 1 -- 172.21.15.57:0/684025608 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a941ef320 con 0x7f4a94159060 2023-10-19T12:45:23.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.245+0000 7f4a989b6640 1 -- 172.21.15.57:0/684025608 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a941ef510 con 0x7f4a94159060 2023-10-19T12:45:23.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.245+0000 7f4a8affd640 1 -- 172.21.15.57:0/684025608 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f4a7c03e450 con 0x7f4a94159060 2023-10-19T12:45:23.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.246+0000 7f4a8affd640 1 --2- 172.21.15.57:0/684025608 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f4a7406ae70 0x7f4a7406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:23.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.246+0000 7f4a8affd640 1 -- 172.21.15.57:0/684025608 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f4a7c0ac460 con 0x7f4a94159060 2023-10-19T12:45:23.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.246+0000 7f4a9259c640 1 --2- 172.21.15.57:0/684025608 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f4a7406ae70 0x7f4a7406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:23.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.247+0000 7f4a9259c640 1 --2- 172.21.15.57:0/684025608 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f4a7406ae70 0x7f4a7406d330 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c005fd0 tx=0x7f4a8c005850 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:23.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.250+0000 7f4a8affd640 1 -- 172.21.15.57:0/684025608 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4a7c0771e0 con 0x7f4a94159060 2023-10-19T12:45:23.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.424+0000 7f4a989b6640 1 -- 172.21.15.57:0/684025608 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f4a940675f0 con 0x7f4a94159060 2023-10-19T12:45:23.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.425+0000 7f4a8affd640 1 -- 172.21.15.57:0/684025608 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7f4a7c07a800 con 0x7f4a94159060 2023-10-19T12:45:23.429 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2023-10-19T12:45:23.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.427+0000 7f4a989b6640 1 -- 172.21.15.57:0/684025608 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f4a7406ae70 msgr2=0x7f4a7406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:23.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.427+0000 7f4a989b6640 1 --2- 172.21.15.57:0/684025608 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f4a7406ae70 0x7f4a7406d330 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c005fd0 tx=0x7f4a8c005850 comp rx=0 tx=0).stop 2023-10-19T12:45:23.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.427+0000 7f4a989b6640 1 -- 172.21.15.57:0/684025608 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a94159060 msgr2=0x7f4a941579d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:23.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.427+0000 7f4a989b6640 1 --2- 172.21.15.57:0/684025608 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a94159060 0x7f4a941579d0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f4a7c009850 tx=0x7f4a7c02da40 comp rx=0 tx=0).stop 2023-10-19T12:45:23.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.427+0000 7f4a989b6640 1 -- 172.21.15.57:0/684025608 shutdown_connections 2023-10-19T12:45:23.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.427+0000 7f4a989b6640 1 --2- 172.21.15.57:0/684025608 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f4a7406ae70 0x7f4a7406d330 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:23.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.427+0000 7f4a989b6640 1 --2- 172.21.15.57:0/684025608 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4a9415ccf0 0x7f4a941f23b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:23.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.427+0000 7f4a989b6640 1 --2- 172.21.15.57:0/684025608 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a94159060 0x7f4a941579d0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:23.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.427+0000 7f4a989b6640 1 --2- 172.21.15.57:0/684025608 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4a940cdb30 0x7f4a941ef840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:23.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.427+0000 7f4a989b6640 1 -- 172.21.15.57:0/684025608 >> 172.21.15.57:0/684025608 conn(0x7f4a940b0670 msgr2=0x7f4a940b0a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:23.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.427+0000 7f4a989b6640 1 -- 172.21.15.57:0/684025608 shutdown_connections 2023-10-19T12:45:23.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.427+0000 7f4a989b6640 1 -- 172.21.15.57:0/684025608 wait complete. 2023-10-19T12:45:23.441 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-19T12:45:23.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.547+0000 7fac59eb4640 1 -- 172.21.15.57:0/266810494 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac3c003ee0 msgr2=0x7fac3c0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:23.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.547+0000 7fac59eb4640 1 --2- 172.21.15.57:0/266810494 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac3c003ee0 0x7fac3c0047c0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fac4800ac60 tx=0x7fac3c0057f0 comp rx=0 tx=0).stop 2023-10-19T12:45:23.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.547+0000 7fac53fff640 1 -- 172.21.15.57:0/266810494 reap_dead start 2023-10-19T12:45:23.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.547+0000 7fac59eb4640 1 -- 172.21.15.57:0/266810494 shutdown_connections 2023-10-19T12:45:23.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.547+0000 7fac59eb4640 1 -- 172.21.15.57:0/266810494 >> 172.21.15.57:0/266810494 conn(0x7fac54069930 msgr2=0x7fac54069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:23.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.547+0000 7fac59eb4640 1 -- 172.21.15.57:0/266810494 shutdown_connections 2023-10-19T12:45:23.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.548+0000 7fac59eb4640 1 -- 172.21.15.57:0/266810494 wait complete. 2023-10-19T12:45:23.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.549+0000 7fac59eb4640 1 Processor -- start 2023-10-19T12:45:23.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.549+0000 7fac59eb4640 1 -- start start 2023-10-19T12:45:23.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.549+0000 7fac59eb4640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fac3c003ee0 0x7fac541494c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:23.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.549+0000 7fac59eb4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac5415a5e0 0x7fac5413e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:23.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.550+0000 7fac59eb4640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fac541562c0 0x7fac54156740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:23.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.550+0000 7fac59eb4640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac54149dc0 con 0x7fac5415a5e0 2023-10-19T12:45:23.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.550+0000 7fac59eb4640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac54156c80 con 0x7fac3c003ee0 2023-10-19T12:45:23.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.550+0000 7fac59eb4640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac5414f6e0 con 0x7fac541562c0 2023-10-19T12:45:23.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.550+0000 7fac52ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac5415a5e0 0x7fac5413e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:23.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.550+0000 7fac52ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac5415a5e0 0x7fac5413e400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50536/0 (socket says 172.21.15.57:50536) 2023-10-19T12:45:23.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.550+0000 7fac52ffd640 1 -- 172.21.15.57:0/1134474087 learned_addr learned my addr 172.21.15.57:0/1134474087 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:23.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.550+0000 7fac537fe640 1 --2- 172.21.15.57:0/1134474087 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fac3c003ee0 0x7fac541494c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:23.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.550+0000 7fac53fff640 1 --2- 172.21.15.57:0/1134474087 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fac541562c0 0x7fac54156740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:23.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.550+0000 7fac52ffd640 1 -- 172.21.15.57:0/1134474087 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fac541562c0 msgr2=0x7fac54156740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:23.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.550+0000 7fac52ffd640 1 --2- 172.21.15.57:0/1134474087 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fac541562c0 0x7fac54156740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:23.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.550+0000 7fac52ffd640 1 -- 172.21.15.57:0/1134474087 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fac3c003ee0 msgr2=0x7fac541494c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:23.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.550+0000 7fac52ffd640 1 --2- 172.21.15.57:0/1134474087 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fac3c003ee0 0x7fac541494c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:23.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.550+0000 7fac52ffd640 1 -- 172.21.15.57:0/1134474087 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac54150520 con 0x7fac5415a5e0 2023-10-19T12:45:23.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.550+0000 7fac52ffd640 1 --2- 172.21.15.57:0/1134474087 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac5415a5e0 0x7fac5413e400 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fac4800ac60 tx=0x7fac48034ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:23.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.550+0000 7fac3bfff640 1 -- 172.21.15.57:0/1134474087 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac48004500 con 0x7fac5415a5e0 2023-10-19T12:45:23.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.550+0000 7fac3bfff640 1 -- 172.21.15.57:0/1134474087 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fac480035f0 con 0x7fac5415a5e0 2023-10-19T12:45:23.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.550+0000 7fac3bfff640 1 -- 172.21.15.57:0/1134474087 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac480039c0 con 0x7fac5415a5e0 2023-10-19T12:45:23.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.550+0000 7fac59eb4640 1 -- 172.21.15.57:0/1134474087 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac5413e940 con 0x7fac5415a5e0 2023-10-19T12:45:23.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.551+0000 7fac59eb4640 1 -- 172.21.15.57:0/1134474087 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac5414fbf0 con 0x7fac5415a5e0 2023-10-19T12:45:23.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.551+0000 7fac59eb4640 1 -- 172.21.15.57:0/1134474087 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac541506b0 con 0x7fac5415a5e0 2023-10-19T12:45:23.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.552+0000 7fac3bfff640 1 -- 172.21.15.57:0/1134474087 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fac48003b20 con 0x7fac5415a5e0 2023-10-19T12:45:23.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.552+0000 7fac3bfff640 1 --2- 172.21.15.57:0/1134474087 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fac4006ae40 0x7fac4006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:23.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.552+0000 7fac3bfff640 1 -- 172.21.15.57:0/1134474087 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fac480acbb0 con 0x7fac5415a5e0 2023-10-19T12:45:23.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.553+0000 7fac537fe640 1 --2- 172.21.15.57:0/1134474087 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fac4006ae40 0x7fac4006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:23.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.553+0000 7fac537fe640 1 --2- 172.21.15.57:0/1134474087 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fac4006ae40 0x7fac4006d300 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fac44001200 tx=0x7fac44001150 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:23.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.557+0000 7fac3bfff640 1 -- 172.21.15.57:0/1134474087 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fac4807b3c0 con 0x7fac5415a5e0 2023-10-19T12:45:23.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.746+0000 7fac59eb4640 1 -- 172.21.15.57:0/1134474087 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fac54151780 con 0x7fac5415a5e0 2023-10-19T12:45:23.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.746+0000 7fac3bfff640 1 -- 172.21.15.57:0/1134474087 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) v1 ==== 80+0+0 (secure 0 0 0) 0x7fac4804c020 con 0x7fac5415a5e0 2023-10-19T12:45:23.750 INFO:teuthology.orchestra.run.smithi057.stderr:listed 0 entries 2023-10-19T12:45:23.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.748+0000 7fac59eb4640 1 -- 172.21.15.57:0/1134474087 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fac4006ae40 msgr2=0x7fac4006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:23.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.748+0000 7fac59eb4640 1 --2- 172.21.15.57:0/1134474087 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fac4006ae40 0x7fac4006d300 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fac44001200 tx=0x7fac44001150 comp rx=0 tx=0).stop 2023-10-19T12:45:23.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.748+0000 7fac59eb4640 1 -- 172.21.15.57:0/1134474087 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac5415a5e0 msgr2=0x7fac5413e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:23.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.748+0000 7fac59eb4640 1 --2- 172.21.15.57:0/1134474087 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac5415a5e0 0x7fac5413e400 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fac4800ac60 tx=0x7fac48034ed0 comp rx=0 tx=0).stop 2023-10-19T12:45:23.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.748+0000 7fac59eb4640 1 -- 172.21.15.57:0/1134474087 shutdown_connections 2023-10-19T12:45:23.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.748+0000 7fac59eb4640 1 --2- 172.21.15.57:0/1134474087 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fac4006ae40 0x7fac4006d300 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:23.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.748+0000 7fac59eb4640 1 --2- 172.21.15.57:0/1134474087 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fac541562c0 0x7fac54156740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:23.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.748+0000 7fac59eb4640 1 --2- 172.21.15.57:0/1134474087 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac5415a5e0 0x7fac5413e400 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:23.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.748+0000 7fac59eb4640 1 --2- 172.21.15.57:0/1134474087 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fac3c003ee0 0x7fac541494c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:23.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.748+0000 7fac59eb4640 1 -- 172.21.15.57:0/1134474087 >> 172.21.15.57:0/1134474087 conn(0x7fac54069930 msgr2=0x7fac5415ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:23.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.748+0000 7fac59eb4640 1 -- 172.21.15.57:0/1134474087 shutdown_connections 2023-10-19T12:45:23.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.748+0000 7fac59eb4640 1 -- 172.21.15.57:0/1134474087 wait complete. 2023-10-19T12:45:23.762 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-19T12:45:23.762 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T12:45:23.781 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T12:45:23.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.887+0000 7f5a7324f640 1 -- 172.21.15.57:0/1273638581 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a6c06d2d0 msgr2=0x7f5a6c06d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:23.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.887+0000 7f5a7324f640 1 --2- 172.21.15.57:0/1273638581 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a6c06d2d0 0x7f5a6c06d6b0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f5a6c06fc50 tx=0x7f5a6402bd90 comp rx=0 tx=0).stop 2023-10-19T12:45:23.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.887+0000 7f5a7324f640 1 -- 172.21.15.57:0/1273638581 shutdown_connections 2023-10-19T12:45:23.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.887+0000 7f5a7324f640 1 --2- 172.21.15.57:0/1273638581 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5a6c0ab950 0x7f5a6c0abde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:23.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.887+0000 7f5a7324f640 1 --2- 172.21.15.57:0/1273638581 >> v2:172.21.15.179:3300/0 conn(0x7f5a6c06f0a0 0x7f5a6c06dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:23.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.887+0000 7f5a7324f640 1 --2- 172.21.15.57:0/1273638581 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a6c06d2d0 0x7f5a6c06d6b0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:23.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.887+0000 7f5a7324f640 1 -- 172.21.15.57:0/1273638581 >> 172.21.15.57:0/1273638581 conn(0x7f5a6c0b0670 msgr2=0x7f5a6c0b0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:23.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.887+0000 7f5a7324f640 1 -- 172.21.15.57:0/1273638581 shutdown_connections 2023-10-19T12:45:23.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.887+0000 7f5a7324f640 1 -- 172.21.15.57:0/1273638581 wait complete. 2023-10-19T12:45:23.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.888+0000 7f5a7324f640 1 Processor -- start 2023-10-19T12:45:23.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.888+0000 7f5a7324f640 1 -- start start 2023-10-19T12:45:23.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.889+0000 7f5a7324f640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5a6c06d2d0 0x7f5a6c14fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:23.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.889+0000 7f5a7324f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a6c06f0a0 0x7f5a6c153dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:23.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.889+0000 7f5a7324f640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5a6c0ab950 0x7f5a6c152610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:23.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.889+0000 7f5a7324f640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a6c151890 con 0x7f5a6c06f0a0 2023-10-19T12:45:23.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.889+0000 7f5a7324f640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a6c1f04e0 con 0x7f5a6c06d2d0 2023-10-19T12:45:23.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.889+0000 7f5a7324f640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a6c151b30 con 0x7f5a6c0ab950 2023-10-19T12:45:23.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.889+0000 7f5a6bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a6c06f0a0 0x7f5a6c153dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:23.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.889+0000 7f5a6bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a6c06f0a0 0x7f5a6c153dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50552/0 (socket says 172.21.15.57:50552) 2023-10-19T12:45:23.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.889+0000 7f5a6bfff640 1 -- 172.21.15.57:0/3962792989 learned_addr learned my addr 172.21.15.57:0/3962792989 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:23.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.889+0000 7f5a6bfff640 1 -- 172.21.15.57:0/3962792989 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5a6c0ab950 msgr2=0x7f5a6c152610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:23.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.889+0000 7f5a6bfff640 1 --2- 172.21.15.57:0/3962792989 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5a6c0ab950 0x7f5a6c152610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:23.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.889+0000 7f5a6bfff640 1 -- 172.21.15.57:0/3962792989 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5a6c06d2d0 msgr2=0x7f5a6c14fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:23.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.889+0000 7f5a70feb640 1 --2- 172.21.15.57:0/3962792989 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5a6c06d2d0 0x7f5a6c14fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:23.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.889+0000 7f5a6bfff640 1 --2- 172.21.15.57:0/3962792989 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5a6c06d2d0 0x7f5a6c14fdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:23.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.889+0000 7f5a6bfff640 1 -- 172.21.15.57:0/3962792989 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a64009560 con 0x7f5a6c06f0a0 2023-10-19T12:45:23.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.890+0000 7f5a6bfff640 1 --2- 172.21.15.57:0/3962792989 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a6c06f0a0 0x7f5a6c153dc0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f5a5c00ead0 tx=0x7f5a5c00ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:23.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.890+0000 7f5a697fa640 1 -- 172.21.15.57:0/3962792989 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a5c00cde0 con 0x7f5a6c06f0a0 2023-10-19T12:45:23.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.890+0000 7f5a697fa640 1 -- 172.21.15.57:0/3962792989 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f5a5c011470 con 0x7f5a6c06f0a0 2023-10-19T12:45:23.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.890+0000 7f5a7324f640 1 -- 172.21.15.57:0/3962792989 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a6c151e10 con 0x7f5a6c06f0a0 2023-10-19T12:45:23.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.890+0000 7f5a697fa640 1 -- 172.21.15.57:0/3962792989 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a5c011860 con 0x7f5a6c06f0a0 2023-10-19T12:45:23.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.890+0000 7f5a7324f640 1 -- 172.21.15.57:0/3962792989 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a6c150330 con 0x7f5a6c06f0a0 2023-10-19T12:45:23.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.891+0000 7f5a7324f640 1 -- 172.21.15.57:0/3962792989 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a6c0ce1d0 con 0x7f5a6c06f0a0 2023-10-19T12:45:23.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.891+0000 7f5a697fa640 1 -- 172.21.15.57:0/3962792989 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f5a5c0119c0 con 0x7f5a6c06f0a0 2023-10-19T12:45:23.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.892+0000 7f5a697fa640 1 --2- 172.21.15.57:0/3962792989 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f5a4c06ae40 0x7f5a4c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:23.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.892+0000 7f5a697fa640 1 -- 172.21.15.57:0/3962792989 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f5a5c08a660 con 0x7f5a6c06f0a0 2023-10-19T12:45:23.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.892+0000 7f5a70feb640 1 --2- 172.21.15.57:0/3962792989 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f5a4c06ae40 0x7f5a4c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:23.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.892+0000 7f5a70feb640 1 --2- 172.21.15.57:0/3962792989 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f5a4c06ae40 0x7f5a4c06d300 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f5a64035000 tx=0x7f5a64004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:23.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:23.896+0000 7f5a697fa640 1 -- 172.21.15.57:0/3962792989 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f5a5c0553e0 con 0x7f5a6c06f0a0 2023-10-19T12:45:24.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.073+0000 7f5a7324f640 1 -- 172.21.15.57:0/3962792989 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5a6c15c2e0 con 0x7f5a6c06f0a0 2023-10-19T12:45:24.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.074+0000 7f5a697fa640 1 -- 172.21.15.57:0/3962792989 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18748 (secure 0 0 0) 0x7f5a5c058a00 con 0x7f5a6c06f0a0 2023-10-19T12:45:24.077 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:45:24.077 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":27,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T12:44:58.978403+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-19T12:44:16.517510+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-19T12:44:17.398297+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-19T12:44:21.129403+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.23","mappings":[{"from":3,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"3.39","mappings":[{"from":7,"to":1}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.1d","mappings":[{"from":1,"to":5}]},{"pgid":"4.1e","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T12:45:24.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.076+0000 7f5a7324f640 1 -- 172.21.15.57:0/3962792989 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f5a4c06ae40 msgr2=0x7f5a4c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:24.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.076+0000 7f5a7324f640 1 --2- 172.21.15.57:0/3962792989 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f5a4c06ae40 0x7f5a4c06d300 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f5a64035000 tx=0x7f5a64004730 comp rx=0 tx=0).stop 2023-10-19T12:45:24.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.076+0000 7f5a7324f640 1 -- 172.21.15.57:0/3962792989 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a6c06f0a0 msgr2=0x7f5a6c153dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:24.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.076+0000 7f5a7324f640 1 --2- 172.21.15.57:0/3962792989 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a6c06f0a0 0x7f5a6c153dc0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f5a5c00ead0 tx=0x7f5a5c00ef90 comp rx=0 tx=0).stop 2023-10-19T12:45:24.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.076+0000 7f5a7324f640 1 -- 172.21.15.57:0/3962792989 shutdown_connections 2023-10-19T12:45:24.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.076+0000 7f5a7324f640 1 --2- 172.21.15.57:0/3962792989 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f5a4c06ae40 0x7f5a4c06d300 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:24.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.076+0000 7f5a7324f640 1 --2- 172.21.15.57:0/3962792989 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5a6c0ab950 0x7f5a6c152610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:24.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.076+0000 7f5a7324f640 1 --2- 172.21.15.57:0/3962792989 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a6c06f0a0 0x7f5a6c153dc0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:24.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.076+0000 7f5a7324f640 1 --2- 172.21.15.57:0/3962792989 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5a6c06d2d0 0x7f5a6c14fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:24.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.076+0000 7f5a7324f640 1 -- 172.21.15.57:0/3962792989 >> 172.21.15.57:0/3962792989 conn(0x7f5a6c0b0670 msgr2=0x7f5a6c0cc690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:24.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.076+0000 7f5a7324f640 1 -- 172.21.15.57:0/3962792989 shutdown_connections 2023-10-19T12:45:24.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.076+0000 7f5a7324f640 1 -- 172.21.15.57:0/3962792989 wait complete. 2023-10-19T12:45:24.090 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-19T12:44:07.371186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-19T12:44:16.517510+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-19T12:44:17.398297+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-19T12:44:21.129403+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-19T12:45:24.091 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-19T12:45:24.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.200+0000 7f7ffa373640 1 -- 172.21.15.57:0/428402875 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ff4156790 msgr2=0x7f7ff40cc990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:24.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.200+0000 7f7ffa373640 1 --2- 172.21.15.57:0/428402875 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ff4156790 0x7f7ff40cc990 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f7fdc005fd0 tx=0x7f7fdc02bd90 comp rx=0 tx=0).stop 2023-10-19T12:45:24.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.200+0000 7f7ffa373640 1 -- 172.21.15.57:0/428402875 shutdown_connections 2023-10-19T12:45:24.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.200+0000 7f7ffa373640 1 --2- 172.21.15.57:0/428402875 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7ff406a730 0x7f7ff406abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:24.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.200+0000 7f7ffa373640 1 --2- 172.21.15.57:0/428402875 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ff4156790 0x7f7ff40cc990 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:24.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.200+0000 7f7ffa373640 1 -- 172.21.15.57:0/428402875 >> 172.21.15.57:0/428402875 conn(0x7f7ff406fae0 msgr2=0x7f7ff406fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:24.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.200+0000 7f7ffa373640 1 -- 172.21.15.57:0/428402875 shutdown_connections 2023-10-19T12:45:24.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.200+0000 7f7ffa373640 1 -- 172.21.15.57:0/428402875 wait complete. 2023-10-19T12:45:24.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.201+0000 7f7ffa373640 1 Processor -- start 2023-10-19T12:45:24.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.201+0000 7f7ffa373640 1 -- start start 2023-10-19T12:45:24.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.202+0000 7f7ffa373640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7ff406a730 0x7f7ff415b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:24.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.202+0000 7f7ffa373640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ff40ac870 0x7f7ff41529c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:24.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.202+0000 7f7ffa373640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7ff4156790 0x7f7ff415cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:24.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.202+0000 7f7ffa373640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff415a320 con 0x7f7ff40ac870 2023-10-19T12:45:24.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.202+0000 7f7ffa373640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff415a600 con 0x7f7ff406a730 2023-10-19T12:45:24.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.202+0000 7f7ffa373640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff415a8e0 con 0x7f7ff4156790 2023-10-19T12:45:24.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.202+0000 7f7ff37fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ff40ac870 0x7f7ff41529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:24.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.202+0000 7f7ff37fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ff40ac870 0x7f7ff41529c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50570/0 (socket says 172.21.15.57:50570) 2023-10-19T12:45:24.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.202+0000 7f7ff37fe640 1 -- 172.21.15.57:0/3994366002 learned_addr learned my addr 172.21.15.57:0/3994366002 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:24.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.202+0000 7f7ff3fff640 1 --2- 172.21.15.57:0/3994366002 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7ff406a730 0x7f7ff415b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:24.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.202+0000 7f7ff8910640 1 --2- 172.21.15.57:0/3994366002 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7ff4156790 0x7f7ff415cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:24.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.202+0000 7f7ff37fe640 1 -- 172.21.15.57:0/3994366002 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7ff4156790 msgr2=0x7f7ff415cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:24.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.202+0000 7f7ff37fe640 1 --2- 172.21.15.57:0/3994366002 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7ff4156790 0x7f7ff415cc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:24.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.202+0000 7f7ff37fe640 1 -- 172.21.15.57:0/3994366002 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7ff406a730 msgr2=0x7f7ff415b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:24.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.202+0000 7f7ff37fe640 1 --2- 172.21.15.57:0/3994366002 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7ff406a730 0x7f7ff415b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:24.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.202+0000 7f7ff37fe640 1 -- 172.21.15.57:0/3994366002 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fdc009560 con 0x7f7ff40ac870 2023-10-19T12:45:24.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.202+0000 7f7ff37fe640 1 --2- 172.21.15.57:0/3994366002 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ff40ac870 0x7f7ff41529c0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f7fe400c430 tx=0x7f7fe400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:24.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.202+0000 7f7ff0ff9640 1 -- 172.21.15.57:0/3994366002 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fe4011790 con 0x7f7ff40ac870 2023-10-19T12:45:24.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.202+0000 7f7ff0ff9640 1 -- 172.21.15.57:0/3994366002 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f7fe40119e0 con 0x7f7ff40ac870 2023-10-19T12:45:24.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.202+0000 7f7ffa373640 1 -- 172.21.15.57:0/3994366002 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ff415abc0 con 0x7f7ff40ac870 2023-10-19T12:45:24.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.202+0000 7f7ff0ff9640 1 -- 172.21.15.57:0/3994366002 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fe4011e30 con 0x7f7ff40ac870 2023-10-19T12:45:24.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.203+0000 7f7ffa373640 1 -- 172.21.15.57:0/3994366002 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ff415d160 con 0x7f7ff40ac870 2023-10-19T12:45:24.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.204+0000 7f7ffa373640 1 -- 172.21.15.57:0/3994366002 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ff415b0e0 con 0x7f7ff40ac870 2023-10-19T12:45:24.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.204+0000 7f7ff0ff9640 1 -- 172.21.15.57:0/3994366002 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f7fe401c460 con 0x7f7ff40ac870 2023-10-19T12:45:24.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.204+0000 7f7ff0ff9640 1 --2- 172.21.15.57:0/3994366002 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f7fd406ae40 0x7f7fd406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:24.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.204+0000 7f7ff0ff9640 1 -- 172.21.15.57:0/3994366002 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f7fe4089b50 con 0x7f7ff40ac870 2023-10-19T12:45:24.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.204+0000 7f7ff3fff640 1 --2- 172.21.15.57:0/3994366002 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f7fd406ae40 0x7f7fd406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:24.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.205+0000 7f7ff3fff640 1 --2- 172.21.15.57:0/3994366002 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f7fd406ae40 0x7f7fd406d300 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f7fdc035000 tx=0x7f7fdc004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:24.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.209+0000 7f7ff0ff9640 1 -- 172.21.15.57:0/3994366002 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7ff415b0e0 con 0x7f7ff40ac870 2023-10-19T12:45:24.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.382+0000 7f7ffa373640 1 -- 172.21.15.57:0/3994366002 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7ff415b0e0 con 0x7f7ff40ac870 2023-10-19T12:45:24.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.383+0000 7f7ff0ff9640 1 -- 172.21.15.57:0/3994366002 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7ff415b0e0 con 0x7f7ff40ac870 2023-10-19T12:45:24.386 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-19T12:45:24.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.384+0000 7f7ffa373640 1 -- 172.21.15.57:0/3994366002 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f7fd406ae40 msgr2=0x7f7fd406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:24.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.384+0000 7f7ffa373640 1 --2- 172.21.15.57:0/3994366002 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f7fd406ae40 0x7f7fd406d300 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f7fdc035000 tx=0x7f7fdc004730 comp rx=0 tx=0).stop 2023-10-19T12:45:24.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.384+0000 7f7ffa373640 1 -- 172.21.15.57:0/3994366002 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ff40ac870 msgr2=0x7f7ff41529c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:24.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.385+0000 7f7ffa373640 1 --2- 172.21.15.57:0/3994366002 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ff40ac870 0x7f7ff41529c0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f7fe400c430 tx=0x7f7fe400c8f0 comp rx=0 tx=0).stop 2023-10-19T12:45:24.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.385+0000 7f7ffa373640 1 -- 172.21.15.57:0/3994366002 shutdown_connections 2023-10-19T12:45:24.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.385+0000 7f7ffa373640 1 --2- 172.21.15.57:0/3994366002 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f7fd406ae40 0x7f7fd406d300 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:24.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.385+0000 7f7ffa373640 1 --2- 172.21.15.57:0/3994366002 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7ff4156790 0x7f7ff415cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:24.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.385+0000 7f7ffa373640 1 --2- 172.21.15.57:0/3994366002 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ff40ac870 0x7f7ff41529c0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:24.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.385+0000 7f7ffa373640 1 --2- 172.21.15.57:0/3994366002 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7ff406a730 0x7f7ff415b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:24.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.385+0000 7f7ffa373640 1 -- 172.21.15.57:0/3994366002 >> 172.21.15.57:0/3994366002 conn(0x7f7ff406fae0 msgr2=0x7f7ff4157510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:24.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.385+0000 7f7ffa373640 1 -- 172.21.15.57:0/3994366002 shutdown_connections 2023-10-19T12:45:24.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.385+0000 7f7ffa373640 1 -- 172.21.15.57:0/3994366002 wait complete. 2023-10-19T12:45:24.398 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-19T12:45:24.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.508+0000 7f8f7a3c0640 1 -- 172.21.15.57:0/3358412753 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8f5c004d00 msgr2=0x7f8f5c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:24.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.508+0000 7f8f7a3c0640 1 --2- 172.21.15.57:0/3358412753 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8f5c004d00 0x7f8f5c005180 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f8f6c00c640 tx=0x7f8f5c0057f0 comp rx=0 tx=0).stop 2023-10-19T12:45:24.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.508+0000 7f8f7895d640 1 -- 172.21.15.57:0/3358412753 reap_dead start 2023-10-19T12:45:24.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.508+0000 7f8f7a3c0640 1 -- 172.21.15.57:0/3358412753 shutdown_connections 2023-10-19T12:45:24.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.508+0000 7f8f7a3c0640 1 -- 172.21.15.57:0/3358412753 >> 172.21.15.57:0/3358412753 conn(0x7f8f740b3f00 msgr2=0x7f8f740b4310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:24.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.509+0000 7f8f7a3c0640 1 -- 172.21.15.57:0/3358412753 shutdown_connections 2023-10-19T12:45:24.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.509+0000 7f8f7a3c0640 1 -- 172.21.15.57:0/3358412753 wait complete. 2023-10-19T12:45:24.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.510+0000 7f8f7a3c0640 1 Processor -- start 2023-10-19T12:45:24.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.510+0000 7f8f7a3c0640 1 -- start start 2023-10-19T12:45:24.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.511+0000 7f8f7a3c0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8f7414c200 0x7f8f741509f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:24.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.511+0000 7f8f7a3c0640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8f7414cc00 0x7f8f74152e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:24.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.511+0000 7f8f7a3c0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8f7414c5e0 0x7f8f741533c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:24.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.511+0000 7f8f7a3c0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f74153900 con 0x7f8f7414c200 2023-10-19T12:45:24.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.511+0000 7f8f7a3c0640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f74151550 con 0x7f8f7414c5e0 2023-10-19T12:45:24.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.511+0000 7f8f7a3c0640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f74151830 con 0x7f8f7414cc00 2023-10-19T12:45:24.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.511+0000 7f8f73fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8f7414c200 0x7f8f741509f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:24.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.511+0000 7f8f73fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8f7414c200 0x7f8f741509f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50580/0 (socket says 172.21.15.57:50580) 2023-10-19T12:45:24.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.511+0000 7f8f73fff640 1 -- 172.21.15.57:0/487929081 learned_addr learned my addr 172.21.15.57:0/487929081 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:24.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.511+0000 7f8f7895d640 1 --2- 172.21.15.57:0/487929081 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8f7414c5e0 0x7f8f741533c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:24.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.511+0000 7f8f737fe640 1 --2- 172.21.15.57:0/487929081 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8f7414cc00 0x7f8f74152e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:24.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.511+0000 7f8f73fff640 1 -- 172.21.15.57:0/487929081 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8f7414cc00 msgr2=0x7f8f74152e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:24.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.511+0000 7f8f73fff640 1 --2- 172.21.15.57:0/487929081 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8f7414cc00 0x7f8f74152e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:24.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.511+0000 7f8f73fff640 1 -- 172.21.15.57:0/487929081 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8f7414c5e0 msgr2=0x7f8f741533c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:24.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.511+0000 7f8f73fff640 1 --2- 172.21.15.57:0/487929081 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8f7414c5e0 0x7f8f741533c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:24.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.511+0000 7f8f73fff640 1 -- 172.21.15.57:0/487929081 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f7414e3f0 con 0x7f8f7414c200 2023-10-19T12:45:24.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.511+0000 7f8f737fe640 1 --2- 172.21.15.57:0/487929081 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8f7414cc00 0x7f8f74152e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:45:24.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.511+0000 7f8f73fff640 1 --2- 172.21.15.57:0/487929081 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8f7414c200 0x7f8f741509f0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f8f68011ae0 tx=0x7f8f68009e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:24.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.511+0000 7f8f70ff9640 1 -- 172.21.15.57:0/487929081 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f6800fe70 con 0x7f8f7414c200 2023-10-19T12:45:24.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.511+0000 7f8f70ff9640 1 -- 172.21.15.57:0/487929081 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f8f68004050 con 0x7f8f7414c200 2023-10-19T12:45:24.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.511+0000 7f8f7a3c0640 1 -- 172.21.15.57:0/487929081 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f7414e6d0 con 0x7f8f7414c200 2023-10-19T12:45:24.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.511+0000 7f8f70ff9640 1 -- 172.21.15.57:0/487929081 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f68004460 con 0x7f8f7414c200 2023-10-19T12:45:24.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.511+0000 7f8f7a3c0640 1 -- 172.21.15.57:0/487929081 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f74151200 con 0x7f8f7414c200 2023-10-19T12:45:24.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.512+0000 7f8f7a3c0640 1 -- 172.21.15.57:0/487929081 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f740690e0 con 0x7f8f7414c200 2023-10-19T12:45:24.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.513+0000 7f8f70ff9640 1 -- 172.21.15.57:0/487929081 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f8f680035f0 con 0x7f8f7414c200 2023-10-19T12:45:24.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.513+0000 7f8f70ff9640 1 --2- 172.21.15.57:0/487929081 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f8f6006ae40 0x7f8f6006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:24.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.513+0000 7f8f70ff9640 1 -- 172.21.15.57:0/487929081 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f8f6808a790 con 0x7f8f7414c200 2023-10-19T12:45:24.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.513+0000 7f8f737fe640 1 --2- 172.21.15.57:0/487929081 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f8f6006ae40 0x7f8f6006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:24.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.514+0000 7f8f737fe640 1 --2- 172.21.15.57:0/487929081 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f8f6006ae40 0x7f8f6006d300 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f8f7406c420 tx=0x7f8f64006260 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:24.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.517+0000 7f8f70ff9640 1 -- 172.21.15.57:0/487929081 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8f680555f0 con 0x7f8f7414c200 2023-10-19T12:45:24.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.695+0000 7f8f7a3c0640 1 -- 172.21.15.57:0/487929081 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8f7414d230 con 0x7f8f7414c200 2023-10-19T12:45:24.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.695+0000 7f8f70ff9640 1 -- 172.21.15.57:0/487929081 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8f68058c10 con 0x7f8f7414c200 2023-10-19T12:45:24.699 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2023-10-19T12:45:24.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.697+0000 7f8f7a3c0640 1 -- 172.21.15.57:0/487929081 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f8f6006ae40 msgr2=0x7f8f6006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:24.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.697+0000 7f8f7a3c0640 1 --2- 172.21.15.57:0/487929081 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f8f6006ae40 0x7f8f6006d300 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f8f7406c420 tx=0x7f8f64006260 comp rx=0 tx=0).stop 2023-10-19T12:45:24.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.697+0000 7f8f7a3c0640 1 -- 172.21.15.57:0/487929081 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8f7414c200 msgr2=0x7f8f741509f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:24.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.697+0000 7f8f7a3c0640 1 --2- 172.21.15.57:0/487929081 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8f7414c200 0x7f8f741509f0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f8f68011ae0 tx=0x7f8f68009e90 comp rx=0 tx=0).stop 2023-10-19T12:45:24.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.697+0000 7f8f7a3c0640 1 -- 172.21.15.57:0/487929081 shutdown_connections 2023-10-19T12:45:24.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.697+0000 7f8f7a3c0640 1 --2- 172.21.15.57:0/487929081 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f8f6006ae40 0x7f8f6006d300 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:24.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.697+0000 7f8f7a3c0640 1 --2- 172.21.15.57:0/487929081 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8f7414c5e0 0x7f8f741533c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:24.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.697+0000 7f8f7a3c0640 1 --2- 172.21.15.57:0/487929081 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8f7414cc00 0x7f8f74152e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:24.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.697+0000 7f8f7a3c0640 1 --2- 172.21.15.57:0/487929081 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8f7414c200 0x7f8f741509f0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:24.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.697+0000 7f8f7a3c0640 1 -- 172.21.15.57:0/487929081 >> 172.21.15.57:0/487929081 conn(0x7f8f740b3f00 msgr2=0x7f8f741577d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:24.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.697+0000 7f8f7a3c0640 1 -- 172.21.15.57:0/487929081 shutdown_connections 2023-10-19T12:45:24.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.697+0000 7f8f7a3c0640 1 -- 172.21.15.57:0/487929081 wait complete. 2023-10-19T12:45:24.711 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-19T12:45:24.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.820+0000 7f0e74cc9640 1 -- 172.21.15.57:0/4140531271 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0e700b0530 msgr2=0x7f0e700b09a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:24.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.820+0000 7f0e74cc9640 1 --2- 172.21.15.57:0/4140531271 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0e700b0530 0x7f0e700b09a0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f0e700cc490 tx=0x7f0e6002c2f0 comp rx=0 tx=0).stop 2023-10-19T12:45:24.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.820+0000 7f0e74cc9640 1 -- 172.21.15.57:0/4140531271 shutdown_connections 2023-10-19T12:45:24.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.820+0000 7f0e74cc9640 1 --2- 172.21.15.57:0/4140531271 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0e700b0530 0x7f0e700b09a0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:24.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.820+0000 7f0e74cc9640 1 --2- 172.21.15.57:0/4140531271 >> v2:172.21.15.179:3300/0 conn(0x7f0e700afa20 0x7f0e70158300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:24.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.820+0000 7f0e74cc9640 1 -- 172.21.15.57:0/4140531271 >> 172.21.15.57:0/4140531271 conn(0x7f0e70069930 msgr2=0x7f0e70069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:24.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.820+0000 7f0e74cc9640 1 -- 172.21.15.57:0/4140531271 shutdown_connections 2023-10-19T12:45:24.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.820+0000 7f0e74cc9640 1 -- 172.21.15.57:0/4140531271 wait complete. 2023-10-19T12:45:24.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.821+0000 7f0e74cc9640 1 Processor -- start 2023-10-19T12:45:24.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.821+0000 7f0e74cc9640 1 -- start start 2023-10-19T12:45:24.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.822+0000 7f0e74cc9640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0e700afa20 0x7f0e701e5d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:24.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.822+0000 7f0e74cc9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0e700b0530 0x7f0e7013b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:24.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.822+0000 7f0e74cc9640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0e700cbd20 0x7f0e7014c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:24.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.822+0000 7f0e74cc9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e70149c70 con 0x7f0e700b0530 2023-10-19T12:45:24.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.822+0000 7f0e74cc9640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e7014d7d0 con 0x7f0e700afa20 2023-10-19T12:45:24.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.822+0000 7f0e74cc9640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e7014cc60 con 0x7f0e700cbd20 2023-10-19T12:45:24.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.822+0000 7f0e6dd9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0e700b0530 0x7f0e7013b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:24.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.822+0000 7f0e6dd9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0e700b0530 0x7f0e7013b2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50592/0 (socket says 172.21.15.57:50592) 2023-10-19T12:45:24.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.822+0000 7f0e6dd9b640 1 -- 172.21.15.57:0/3899292304 learned_addr learned my addr 172.21.15.57:0/3899292304 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:24.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.822+0000 7f0e6ed9d640 1 --2- 172.21.15.57:0/3899292304 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0e700cbd20 0x7f0e7014c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:24.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.822+0000 7f0e6e59c640 1 --2- 172.21.15.57:0/3899292304 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0e700afa20 0x7f0e701e5d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:24.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.822+0000 7f0e6dd9b640 1 -- 172.21.15.57:0/3899292304 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0e700cbd20 msgr2=0x7f0e7014c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:24.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.822+0000 7f0e6dd9b640 1 --2- 172.21.15.57:0/3899292304 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0e700cbd20 0x7f0e7014c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:24.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.822+0000 7f0e6dd9b640 1 -- 172.21.15.57:0/3899292304 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0e700afa20 msgr2=0x7f0e701e5d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:24.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.822+0000 7f0e6dd9b640 1 --2- 172.21.15.57:0/3899292304 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0e700afa20 0x7f0e701e5d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:24.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.823+0000 7f0e6dd9b640 1 -- 172.21.15.57:0/3899292304 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e60009900 con 0x7f0e700b0530 2023-10-19T12:45:24.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.823+0000 7f0e6dd9b640 1 --2- 172.21.15.57:0/3899292304 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0e700b0530 0x7f0e7013b2b0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f0e6400dbc0 tx=0x7f0e6400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:24.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.823+0000 7f0e5effd640 1 -- 172.21.15.57:0/3899292304 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e64005220 con 0x7f0e700b0530 2023-10-19T12:45:24.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.823+0000 7f0e5effd640 1 -- 172.21.15.57:0/3899292304 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f0e64005470 con 0x7f0e700b0530 2023-10-19T12:45:24.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.823+0000 7f0e5effd640 1 -- 172.21.15.57:0/3899292304 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e64004a00 con 0x7f0e700b0530 2023-10-19T12:45:24.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.823+0000 7f0e74cc9640 1 -- 172.21.15.57:0/3899292304 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e7014cf30 con 0x7f0e700b0530 2023-10-19T12:45:24.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.823+0000 7f0e74cc9640 1 -- 172.21.15.57:0/3899292304 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e7014d4f0 con 0x7f0e700b0530 2023-10-19T12:45:24.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.824+0000 7f0e74cc9640 1 -- 172.21.15.57:0/3899292304 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e701499d0 con 0x7f0e700b0530 2023-10-19T12:45:24.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.824+0000 7f0e5effd640 1 -- 172.21.15.57:0/3899292304 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f0e64004b60 con 0x7f0e700b0530 2023-10-19T12:45:24.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.825+0000 7f0e5effd640 1 --2- 172.21.15.57:0/3899292304 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f0e5006ae70 0x7f0e5006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:24.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.825+0000 7f0e5effd640 1 -- 172.21.15.57:0/3899292304 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f0e6408a150 con 0x7f0e700b0530 2023-10-19T12:45:24.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.825+0000 7f0e6e59c640 1 --2- 172.21.15.57:0/3899292304 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f0e5006ae70 0x7f0e5006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:24.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.825+0000 7f0e6e59c640 1 --2- 172.21.15.57:0/3899292304 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f0e5006ae70 0x7f0e5006d330 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f0e700ae090 tx=0x7f0e68009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:24.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:24.829+0000 7f0e5effd640 1 -- 172.21.15.57:0/3899292304 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0e701499d0 con 0x7f0e700b0530 2023-10-19T12:45:25.010 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.007+0000 7f0e74cc9640 1 -- 172.21.15.57:0/3899292304 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0e70149db0 con 0x7f0e700b0530 2023-10-19T12:45:25.010 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.007+0000 7f0e5effd640 1 -- 172.21.15.57:0/3899292304 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0e70149db0 con 0x7f0e700b0530 2023-10-19T12:45:25.011 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 64 2023-10-19T12:45:25.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.009+0000 7f0e74cc9640 1 -- 172.21.15.57:0/3899292304 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f0e5006ae70 msgr2=0x7f0e5006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:25.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.009+0000 7f0e74cc9640 1 --2- 172.21.15.57:0/3899292304 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f0e5006ae70 0x7f0e5006d330 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f0e700ae090 tx=0x7f0e68009040 comp rx=0 tx=0).stop 2023-10-19T12:45:25.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.009+0000 7f0e74cc9640 1 -- 172.21.15.57:0/3899292304 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0e700b0530 msgr2=0x7f0e7013b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:25.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.009+0000 7f0e74cc9640 1 --2- 172.21.15.57:0/3899292304 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0e700b0530 0x7f0e7013b2b0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f0e6400dbc0 tx=0x7f0e6400a620 comp rx=0 tx=0).stop 2023-10-19T12:45:25.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.009+0000 7f0e74cc9640 1 -- 172.21.15.57:0/3899292304 shutdown_connections 2023-10-19T12:45:25.014 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.009+0000 7f0e74cc9640 1 --2- 172.21.15.57:0/3899292304 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f0e5006ae70 0x7f0e5006d330 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:25.014 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.009+0000 7f0e74cc9640 1 --2- 172.21.15.57:0/3899292304 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0e700cbd20 0x7f0e7014c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:25.014 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.009+0000 7f0e74cc9640 1 --2- 172.21.15.57:0/3899292304 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0e700b0530 0x7f0e7013b2b0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:25.014 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.009+0000 7f0e74cc9640 1 --2- 172.21.15.57:0/3899292304 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0e700afa20 0x7f0e701e5d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:25.014 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.009+0000 7f0e74cc9640 1 -- 172.21.15.57:0/3899292304 >> 172.21.15.57:0/3899292304 conn(0x7f0e70069930 msgr2=0x7f0e7006bcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:25.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.009+0000 7f0e74cc9640 1 -- 172.21.15.57:0/3899292304 shutdown_connections 2023-10-19T12:45:25.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.009+0000 7f0e74cc9640 1 -- 172.21.15.57:0/3899292304 wait complete. 2023-10-19T12:45:25.023 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-19T12:45:25.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.132+0000 7f310efd7640 1 -- 172.21.15.57:0/3122835782 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3108157c00 msgr2=0x7f31080c98b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:25.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.132+0000 7f310efd7640 1 --2- 172.21.15.57:0/3122835782 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3108157c00 0x7f31080c98b0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f310806b7a0 tx=0x7f30fc02bc80 comp rx=0 tx=0).stop 2023-10-19T12:45:25.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.132+0000 7f310efd7640 1 -- 172.21.15.57:0/3122835782 shutdown_connections 2023-10-19T12:45:25.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.132+0000 7f310efd7640 1 --2- 172.21.15.57:0/3122835782 >> v2:172.21.15.179:3300/0 conn(0x7f3108070a60 0x7f310815b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:25.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.132+0000 7f310efd7640 1 --2- 172.21.15.57:0/3122835782 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f31080cb2f0 0x7f3108070520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:25.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.132+0000 7f310efd7640 1 --2- 172.21.15.57:0/3122835782 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3108157c00 0x7f31080c98b0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:25.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.132+0000 7f310efd7640 1 -- 172.21.15.57:0/3122835782 >> 172.21.15.57:0/3122835782 conn(0x7f31080aec40 msgr2=0x7f31080af050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:25.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.132+0000 7f310efd7640 1 -- 172.21.15.57:0/3122835782 shutdown_connections 2023-10-19T12:45:25.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.132+0000 7f310efd7640 1 -- 172.21.15.57:0/3122835782 wait complete. 2023-10-19T12:45:25.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.133+0000 7f310efd7640 1 Processor -- start 2023-10-19T12:45:25.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.133+0000 7f310efd7640 1 -- start start 2023-10-19T12:45:25.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.134+0000 7f310efd7640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3108070a60 0x7f3108159c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:25.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.134+0000 7f310efd7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f31080cb2f0 0x7f3108158b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:25.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.134+0000 7f310efd7640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3108157c00 0x7f31081f1270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:25.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.134+0000 7f310efd7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3108158090 con 0x7f31080cb2f0 2023-10-19T12:45:25.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.134+0000 7f310efd7640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31081f1d80 con 0x7f3108070a60 2023-10-19T12:45:25.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.134+0000 7f310efd7640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31081f2060 con 0x7f3108157c00 2023-10-19T12:45:25.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.134+0000 7f3107fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f31080cb2f0 0x7f3108158b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:25.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.134+0000 7f3107fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f31080cb2f0 0x7f3108158b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50614/0 (socket says 172.21.15.57:50614) 2023-10-19T12:45:25.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.134+0000 7f3107fff640 1 -- 172.21.15.57:0/3525768801 learned_addr learned my addr 172.21.15.57:0/3525768801 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:25.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.134+0000 7f310d574640 1 --2- 172.21.15.57:0/3525768801 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3108157c00 0x7f31081f1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:25.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.134+0000 7f310cd73640 1 --2- 172.21.15.57:0/3525768801 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3108070a60 0x7f3108159c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:25.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.134+0000 7f3107fff640 1 -- 172.21.15.57:0/3525768801 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3108157c00 msgr2=0x7f31081f1270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:25.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.134+0000 7f3107fff640 1 --2- 172.21.15.57:0/3525768801 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3108157c00 0x7f31081f1270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:25.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.134+0000 7f3107fff640 1 -- 172.21.15.57:0/3525768801 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3108070a60 msgr2=0x7f3108159c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:25.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.134+0000 7f3107fff640 1 --2- 172.21.15.57:0/3525768801 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3108070a60 0x7f3108159c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:25.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.134+0000 7f3107fff640 1 -- 172.21.15.57:0/3525768801 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30fc009560 con 0x7f31080cb2f0 2023-10-19T12:45:25.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.134+0000 7f310d574640 1 --2- 172.21.15.57:0/3525768801 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3108157c00 0x7f31081f1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:45:25.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.134+0000 7f310cd73640 1 --2- 172.21.15.57:0/3525768801 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3108070a60 0x7f3108159c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:45:25.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.134+0000 7f3107fff640 1 --2- 172.21.15.57:0/3525768801 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f31080cb2f0 0x7f3108158b30 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f30f400e9f0 tx=0x7f30f400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:25.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.141+0000 7f31057fa640 1 -- 172.21.15.57:0/3525768801 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30f4009700 con 0x7f31080cb2f0 2023-10-19T12:45:25.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.141+0000 7f310efd7640 1 -- 172.21.15.57:0/3525768801 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3108158330 con 0x7f31080cb2f0 2023-10-19T12:45:25.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.141+0000 7f31057fa640 1 -- 172.21.15.57:0/3525768801 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f30f400ce70 con 0x7f31080cb2f0 2023-10-19T12:45:25.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.141+0000 7f31057fa640 1 -- 172.21.15.57:0/3525768801 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30f4010670 con 0x7f31080cb2f0 2023-10-19T12:45:25.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.141+0000 7f310efd7640 1 -- 172.21.15.57:0/3525768801 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f310815a190 con 0x7f31080cb2f0 2023-10-19T12:45:25.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.143+0000 7f310efd7640 1 -- 172.21.15.57:0/3525768801 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31080690e0 con 0x7f31080cb2f0 2023-10-19T12:45:25.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.143+0000 7f31057fa640 1 -- 172.21.15.57:0/3525768801 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f30f40107d0 con 0x7f31080cb2f0 2023-10-19T12:45:25.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.143+0000 7f31057fa640 1 --2- 172.21.15.57:0/3525768801 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f30ec06ae40 0x7f30ec06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:25.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.143+0000 7f31057fa640 1 -- 172.21.15.57:0/3525768801 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f30f4014070 con 0x7f31080cb2f0 2023-10-19T12:45:25.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.144+0000 7f310cd73640 1 --2- 172.21.15.57:0/3525768801 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f30ec06ae40 0x7f30ec06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:25.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.144+0000 7f310cd73640 1 --2- 172.21.15.57:0/3525768801 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f30ec06ae40 0x7f30ec06d300 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f30fc035000 tx=0x7f30fc004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:25.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.147+0000 7f31057fa640 1 -- 172.21.15.57:0/3525768801 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f30f4051ca0 con 0x7f31080cb2f0 2023-10-19T12:45:25.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.326+0000 7f310efd7640 1 -- 172.21.15.57:0/3525768801 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f3108071440 con 0x7f31080cb2f0 2023-10-19T12:45:25.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.327+0000 7f31057fa640 1 -- 172.21.15.57:0/3525768801 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7f30f4058e40 con 0x7f31080cb2f0 2023-10-19T12:45:25.330 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2023-10-19T12:45:25.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.329+0000 7f310efd7640 1 -- 172.21.15.57:0/3525768801 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f30ec06ae40 msgr2=0x7f30ec06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:25.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.329+0000 7f310efd7640 1 --2- 172.21.15.57:0/3525768801 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f30ec06ae40 0x7f30ec06d300 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f30fc035000 tx=0x7f30fc004730 comp rx=0 tx=0).stop 2023-10-19T12:45:25.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.329+0000 7f310efd7640 1 -- 172.21.15.57:0/3525768801 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f31080cb2f0 msgr2=0x7f3108158b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:25.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.329+0000 7f310efd7640 1 --2- 172.21.15.57:0/3525768801 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f31080cb2f0 0x7f3108158b30 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f30f400e9f0 tx=0x7f30f400eeb0 comp rx=0 tx=0).stop 2023-10-19T12:45:25.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.329+0000 7f310efd7640 1 -- 172.21.15.57:0/3525768801 shutdown_connections 2023-10-19T12:45:25.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.329+0000 7f310efd7640 1 --2- 172.21.15.57:0/3525768801 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f30ec06ae40 0x7f30ec06d300 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:25.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.329+0000 7f310efd7640 1 --2- 172.21.15.57:0/3525768801 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3108157c00 0x7f31081f1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:25.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.329+0000 7f310efd7640 1 --2- 172.21.15.57:0/3525768801 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f31080cb2f0 0x7f3108158b30 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:25.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.329+0000 7f310efd7640 1 --2- 172.21.15.57:0/3525768801 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3108070a60 0x7f3108159c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:25.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.329+0000 7f310efd7640 1 -- 172.21.15.57:0/3525768801 >> 172.21.15.57:0/3525768801 conn(0x7f31080aec40 msgr2=0x7f31080b0d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:25.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.329+0000 7f310efd7640 1 -- 172.21.15.57:0/3525768801 shutdown_connections 2023-10-19T12:45:25.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.329+0000 7f310efd7640 1 -- 172.21.15.57:0/3525768801 wait complete. 2023-10-19T12:45:25.343 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-19T12:45:25.453 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.450+0000 7fc5b7b2c640 1 -- 172.21.15.57:0/3519446809 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5b00cbd20 msgr2=0x7fc5b0158300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:25.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.450+0000 7fc5b7b2c640 1 --2- 172.21.15.57:0/3519446809 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5b00cbd20 0x7fc5b0158300 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fc5b00b10f0 tx=0x7fc5a802be40 comp rx=0 tx=0).stop 2023-10-19T12:45:25.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.450+0000 7fc5b7b2c640 1 -- 172.21.15.57:0/3519446809 shutdown_connections 2023-10-19T12:45:25.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.450+0000 7fc5b7b2c640 1 --2- 172.21.15.57:0/3519446809 >> v2:172.21.15.179:3300/0 conn(0x7fc5b00b0530 0x7fc5b00b09a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:25.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.450+0000 7fc5b7b2c640 1 --2- 172.21.15.57:0/3519446809 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5b00cbd20 0x7fc5b0158300 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:25.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.450+0000 7fc5b7b2c640 1 -- 172.21.15.57:0/3519446809 >> 172.21.15.57:0/3519446809 conn(0x7fc5b0069930 msgr2=0x7fc5b0069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:25.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.450+0000 7fc5b7b2c640 1 -- 172.21.15.57:0/3519446809 shutdown_connections 2023-10-19T12:45:25.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.450+0000 7fc5b7b2c640 1 -- 172.21.15.57:0/3519446809 wait complete. 2023-10-19T12:45:25.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.451+0000 7fc5b7b2c640 1 Processor -- start 2023-10-19T12:45:25.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.451+0000 7fc5b7b2c640 1 -- start start 2023-10-19T12:45:25.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.452+0000 7fc5b7b2c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc5b00afa20 0x7fc5b0133210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:25.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.452+0000 7fc5b7b2c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5b00b0530 0x7fc5b01562c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:25.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.452+0000 7fc5b7b2c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc5b00cbd20 0x7fc5b0143150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:25.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.452+0000 7fc5b7b2c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b01461f0 con 0x7fc5b00b0530 2023-10-19T12:45:25.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.452+0000 7fc5b7b2c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b0133750 con 0x7fc5b00afa20 2023-10-19T12:45:25.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.452+0000 7fc5b7b2c640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b0156af0 con 0x7fc5b00cbd20 2023-10-19T12:45:25.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.452+0000 7fc5b50c7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5b00b0530 0x7fc5b01562c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:25.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.452+0000 7fc5b50c7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5b00b0530 0x7fc5b01562c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50628/0 (socket says 172.21.15.57:50628) 2023-10-19T12:45:25.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.452+0000 7fc5b50c7640 1 -- 172.21.15.57:0/1661055654 learned_addr learned my addr 172.21.15.57:0/1661055654 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:25.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.452+0000 7fc5b58c8640 1 --2- 172.21.15.57:0/1661055654 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc5b00afa20 0x7fc5b0133210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:25.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.452+0000 7fc5b60c9640 1 --2- 172.21.15.57:0/1661055654 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc5b00cbd20 0x7fc5b0143150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:25.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.452+0000 7fc5b50c7640 1 -- 172.21.15.57:0/1661055654 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc5b00cbd20 msgr2=0x7fc5b0143150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:25.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.452+0000 7fc5b50c7640 1 --2- 172.21.15.57:0/1661055654 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc5b00cbd20 0x7fc5b0143150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:25.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.452+0000 7fc5b50c7640 1 -- 172.21.15.57:0/1661055654 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc5b00afa20 msgr2=0x7fc5b0133210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:25.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.452+0000 7fc5b50c7640 1 --2- 172.21.15.57:0/1661055654 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc5b00afa20 0x7fc5b0133210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:25.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.452+0000 7fc5b50c7640 1 -- 172.21.15.57:0/1661055654 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5a8009560 con 0x7fc5b00b0530 2023-10-19T12:45:25.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.452+0000 7fc5b58c8640 1 --2- 172.21.15.57:0/1661055654 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc5b00afa20 0x7fc5b0133210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:45:25.461 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.452+0000 7fc5b50c7640 1 --2- 172.21.15.57:0/1661055654 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5b00b0530 0x7fc5b01562c0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fc5a400c450 tx=0x7fc5a400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:25.461 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.453+0000 7fc5ae7fc640 1 -- 172.21.15.57:0/1661055654 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5a40052d0 con 0x7fc5b00b0530 2023-10-19T12:45:25.461 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.453+0000 7fc5ae7fc640 1 -- 172.21.15.57:0/1661055654 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fc5a4005430 con 0x7fc5b00b0530 2023-10-19T12:45:25.461 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.453+0000 7fc5b7b2c640 1 -- 172.21.15.57:0/1661055654 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5b0156dd0 con 0x7fc5b00b0530 2023-10-19T12:45:25.461 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.453+0000 7fc5ae7fc640 1 -- 172.21.15.57:0/1661055654 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5a4012890 con 0x7fc5b00b0530 2023-10-19T12:45:25.462 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.453+0000 7fc5b7b2c640 1 -- 172.21.15.57:0/1661055654 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5b0158d00 con 0x7fc5b00b0530 2023-10-19T12:45:25.462 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.454+0000 7fc5b7b2c640 1 -- 172.21.15.57:0/1661055654 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5b00690e0 con 0x7fc5b00b0530 2023-10-19T12:45:25.462 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.454+0000 7fc5ae7fc640 1 -- 172.21.15.57:0/1661055654 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fc5a40035f0 con 0x7fc5b00b0530 2023-10-19T12:45:25.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.454+0000 7fc5ae7fc640 1 --2- 172.21.15.57:0/1661055654 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fc59c06ae70 0x7fc59c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:25.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.455+0000 7fc5ae7fc640 1 -- 172.21.15.57:0/1661055654 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fc5a40896e0 con 0x7fc5b00b0530 2023-10-19T12:45:25.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.455+0000 7fc5b58c8640 1 --2- 172.21.15.57:0/1661055654 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fc59c06ae70 0x7fc59c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:25.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.455+0000 7fc5b58c8640 1 --2- 172.21.15.57:0/1661055654 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fc59c06ae70 0x7fc59c06d330 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fc5a8035000 tx=0x7fc5a8004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:25.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.459+0000 7fc5ae7fc640 1 -- 172.21.15.57:0/1661055654 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc5a4054340 con 0x7fc5b00b0530 2023-10-19T12:45:25.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.646+0000 7fc5b7b2c640 1 -- 172.21.15.57:0/1661055654 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fc5b006f700 con 0x7fc5b00b0530 2023-10-19T12:45:25.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.646+0000 7fc5ae7fc640 1 -- 172.21.15.57:0/1661055654 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) v1 ==== 80+0+0 (secure 0 0 0) 0x7fc5a4057960 con 0x7fc5b00b0530 2023-10-19T12:45:25.650 INFO:teuthology.orchestra.run.smithi057.stderr:listed 0 entries 2023-10-19T12:45:25.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.648+0000 7fc5b7b2c640 1 -- 172.21.15.57:0/1661055654 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fc59c06ae70 msgr2=0x7fc59c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:25.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.648+0000 7fc5b7b2c640 1 --2- 172.21.15.57:0/1661055654 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fc59c06ae70 0x7fc59c06d330 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fc5a8035000 tx=0x7fc5a8004730 comp rx=0 tx=0).stop 2023-10-19T12:45:25.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.648+0000 7fc5b7b2c640 1 -- 172.21.15.57:0/1661055654 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5b00b0530 msgr2=0x7fc5b01562c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:25.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.648+0000 7fc5b7b2c640 1 --2- 172.21.15.57:0/1661055654 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5b00b0530 0x7fc5b01562c0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fc5a400c450 tx=0x7fc5a400c910 comp rx=0 tx=0).stop 2023-10-19T12:45:25.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.648+0000 7fc5b7b2c640 1 -- 172.21.15.57:0/1661055654 shutdown_connections 2023-10-19T12:45:25.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.648+0000 7fc5b7b2c640 1 --2- 172.21.15.57:0/1661055654 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fc59c06ae70 0x7fc59c06d330 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:25.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.648+0000 7fc5b7b2c640 1 --2- 172.21.15.57:0/1661055654 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc5b00cbd20 0x7fc5b0143150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:25.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.648+0000 7fc5b7b2c640 1 --2- 172.21.15.57:0/1661055654 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5b00b0530 0x7fc5b01562c0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:25.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.648+0000 7fc5b7b2c640 1 --2- 172.21.15.57:0/1661055654 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc5b00afa20 0x7fc5b0133210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:25.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.648+0000 7fc5b7b2c640 1 -- 172.21.15.57:0/1661055654 >> 172.21.15.57:0/1661055654 conn(0x7fc5b0069930 msgr2=0x7fc5b0148e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:25.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.648+0000 7fc5b7b2c640 1 -- 172.21.15.57:0/1661055654 shutdown_connections 2023-10-19T12:45:25.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.648+0000 7fc5b7b2c640 1 -- 172.21.15.57:0/1661055654 wait complete. 2023-10-19T12:45:25.661 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi179.front.sepia.ceph.com... 2023-10-19T12:45:25.662 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:45:25.662 DEBUG:teuthology.orchestra.run.smithi179:> sudo fusermount -u /home/ubuntu/cephtest/mnt.3 2023-10-19T12:45:25.695 INFO:teuthology.orchestra.run:waiting for 300 2023-10-19T12:45:25.695 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi179.front.sepia.ceph.com 2023-10-19T12:45:25.696 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:45:25.696 DEBUG:teuthology.orchestra.run.smithi179:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-10-19T12:45:25.752 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-19T12:45:25.752 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T12:45:25.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.859+0000 7f0023fff640 1 -- 172.21.15.57:0/768162640 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f000c005180 msgr2=0x7f000c00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:25.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.859+0000 7f0023fff640 1 --2- 172.21.15.57:0/768162640 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f000c005180 0x7f000c00b980 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f001c00c640 tx=0x7f000c0057f0 comp rx=0 tx=0).stop 2023-10-19T12:45:25.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.859+0000 7f00237fe640 1 -- 172.21.15.57:0/768162640 reap_dead start 2023-10-19T12:45:25.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.859+0000 7f0023fff640 1 -- 172.21.15.57:0/768162640 shutdown_connections 2023-10-19T12:45:25.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.859+0000 7f0023fff640 1 -- 172.21.15.57:0/768162640 >> 172.21.15.57:0/768162640 conn(0x7f00240ae5d0 msgr2=0x7f00240ae9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:25.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.859+0000 7f0023fff640 1 -- 172.21.15.57:0/768162640 shutdown_connections 2023-10-19T12:45:25.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.859+0000 7f0023fff640 1 -- 172.21.15.57:0/768162640 wait complete. 2023-10-19T12:45:25.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.861+0000 7f0023fff640 1 Processor -- start 2023-10-19T12:45:25.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.861+0000 7f0023fff640 1 -- start start 2023-10-19T12:45:25.868 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.861+0000 7f0023fff640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f000c005180 0x7f00241f2250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:25.868 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.861+0000 7f0023fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f002413fcc0 0x7f0024137ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:25.868 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.861+0000 7f0023fff640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0024140140 0x7f0024138400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:25.868 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.861+0000 7f0023fff640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00241f1e30 con 0x7f002413fcc0 2023-10-19T12:45:25.868 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.861+0000 7f0023fff640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00241ef800 con 0x7f000c005180 2023-10-19T12:45:25.869 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.861+0000 7f0023fff640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00241f2790 con 0x7f0024140140 2023-10-19T12:45:25.869 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.862+0000 7f00227fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f002413fcc0 0x7f0024137ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:25.869 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.862+0000 7f00227fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f002413fcc0 0x7f0024137ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50658/0 (socket says 172.21.15.57:50658) 2023-10-19T12:45:25.869 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.862+0000 7f00227fc640 1 -- 172.21.15.57:0/3257545456 learned_addr learned my addr 172.21.15.57:0/3257545456 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:25.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.862+0000 7f00237fe640 1 --2- 172.21.15.57:0/3257545456 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0024140140 0x7f0024138400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:25.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.862+0000 7f0022ffd640 1 --2- 172.21.15.57:0/3257545456 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f000c005180 0x7f00241f2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:25.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.862+0000 7f00227fc640 1 -- 172.21.15.57:0/3257545456 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0024140140 msgr2=0x7f0024138400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:25.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.862+0000 7f00227fc640 1 --2- 172.21.15.57:0/3257545456 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0024140140 0x7f0024138400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:25.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.862+0000 7f00227fc640 1 -- 172.21.15.57:0/3257545456 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f000c005180 msgr2=0x7f00241f2250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:25.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.862+0000 7f00227fc640 1 --2- 172.21.15.57:0/3257545456 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f000c005180 0x7f00241f2250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:25.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.862+0000 7f00227fc640 1 -- 172.21.15.57:0/3257545456 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00241f11b0 con 0x7f002413fcc0 2023-10-19T12:45:25.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.862+0000 7f00227fc640 1 --2- 172.21.15.57:0/3257545456 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f002413fcc0 0x7f0024137ec0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f0018004ef0 tx=0x7f0018001630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:25.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.862+0000 7f000b7fe640 1 -- 172.21.15.57:0/3257545456 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0018011870 con 0x7f002413fcc0 2023-10-19T12:45:25.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.862+0000 7f000b7fe640 1 -- 172.21.15.57:0/3257545456 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f0018011ac0 con 0x7f002413fcc0 2023-10-19T12:45:25.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.862+0000 7f0023fff640 1 -- 172.21.15.57:0/3257545456 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00241f1490 con 0x7f002413fcc0 2023-10-19T12:45:25.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.862+0000 7f000b7fe640 1 -- 172.21.15.57:0/3257545456 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0018011e90 con 0x7f002413fcc0 2023-10-19T12:45:25.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.862+0000 7f0023fff640 1 -- 172.21.15.57:0/3257545456 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00241f1a50 con 0x7f002413fcc0 2023-10-19T12:45:25.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.863+0000 7f0023fff640 1 -- 172.21.15.57:0/3257545456 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00241f20a0 con 0x7f002413fcc0 2023-10-19T12:45:25.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.864+0000 7f000b7fe640 1 -- 172.21.15.57:0/3257545456 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f001801a400 con 0x7f002413fcc0 2023-10-19T12:45:25.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.864+0000 7f000b7fe640 1 --2- 172.21.15.57:0/3257545456 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f001006ae40 0x7f001006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:25.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.864+0000 7f000b7fe640 1 -- 172.21.15.57:0/3257545456 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f001808a790 con 0x7f002413fcc0 2023-10-19T12:45:25.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.865+0000 7f0022ffd640 1 --2- 172.21.15.57:0/3257545456 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f001006ae40 0x7f001006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:25.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.865+0000 7f0022ffd640 1 --2- 172.21.15.57:0/3257545456 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f001006ae40 0x7f001006d300 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f0014005fd0 tx=0x7f0014005850 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:25.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:25.868+0000 7f000b7fe640 1 -- 172.21.15.57:0/3257545456 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0018055510 con 0x7f002413fcc0 2023-10-19T12:45:26.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.079+0000 7f0023fff640 1 -- 172.21.15.57:0/3257545456 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f002404da70 con 0x7f002413fcc0 2023-10-19T12:45:26.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.080+0000 7f000b7fe640 1 -- 172.21.15.57:0/3257545456 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4906 (secure 0 0 0) 0x7f0018058b30 con 0x7f002413fcc0 2023-10-19T12:45:26.083 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:45:26.083 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4357,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/4228712742","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":4228712742},{"type":"v1","addr":"172.21.15.57:6835","nonce":4228712742}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4360,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/971892630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":971892630},{"type":"v1","addr":"172.21.15.57:6834","nonce":971892630}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4372,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/104495983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":104495983},{"type":"v1","addr":"172.21.15.179:6836","nonce":104495983}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T12:44:18.419669+0000","modified":"2023-10-19T12:44:40.545928+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4374},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4374":{"gid":4374,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.179:6837/622550365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":622550365},{"type":"v1","addr":"172.21.15.179:6837","nonce":622550365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-19T12:45:26.084 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 9 2023-10-19T12:45:26.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.081+0000 7f0023fff640 1 -- 172.21.15.57:0/3257545456 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f001006ae40 msgr2=0x7f001006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:26.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.081+0000 7f0023fff640 1 --2- 172.21.15.57:0/3257545456 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f001006ae40 0x7f001006d300 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f0014005fd0 tx=0x7f0014005850 comp rx=0 tx=0).stop 2023-10-19T12:45:26.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.081+0000 7f0023fff640 1 -- 172.21.15.57:0/3257545456 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f002413fcc0 msgr2=0x7f0024137ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:26.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.081+0000 7f0023fff640 1 --2- 172.21.15.57:0/3257545456 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f002413fcc0 0x7f0024137ec0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f0018004ef0 tx=0x7f0018001630 comp rx=0 tx=0).stop 2023-10-19T12:45:26.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.081+0000 7f0023fff640 1 -- 172.21.15.57:0/3257545456 shutdown_connections 2023-10-19T12:45:26.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.081+0000 7f0023fff640 1 --2- 172.21.15.57:0/3257545456 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f001006ae40 0x7f001006d300 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:26.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.081+0000 7f0023fff640 1 --2- 172.21.15.57:0/3257545456 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0024140140 0x7f0024138400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:26.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.081+0000 7f0023fff640 1 --2- 172.21.15.57:0/3257545456 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f002413fcc0 0x7f0024137ec0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:26.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.081+0000 7f0023fff640 1 --2- 172.21.15.57:0/3257545456 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f000c005180 0x7f00241f2250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:26.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.081+0000 7f0023fff640 1 -- 172.21.15.57:0/3257545456 >> 172.21.15.57:0/3257545456 conn(0x7f00240ae5d0 msgr2=0x7f002413d710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:26.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.081+0000 7f0023fff640 1 -- 172.21.15.57:0/3257545456 shutdown_connections 2023-10-19T12:45:26.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.082+0000 7f0023fff640 1 -- 172.21.15.57:0/3257545456 wait complete. 2023-10-19T12:45:26.098 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T12:45:26.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.206+0000 7f059dba9640 1 -- 172.21.15.57:0/2510854670 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0580003fd0 msgr2=0x7f05800047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:26.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.207+0000 7f059dba9640 1 --2- 172.21.15.57:0/2510854670 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0580003fd0 0x7f05800047c0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f058c001680 tx=0x7f05800057f0 comp rx=0 tx=0).stop 2023-10-19T12:45:26.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.207+0000 7f0597fff640 1 -- 172.21.15.57:0/2510854670 reap_dead start 2023-10-19T12:45:26.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.207+0000 7f059dba9640 1 -- 172.21.15.57:0/2510854670 shutdown_connections 2023-10-19T12:45:26.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.207+0000 7f059dba9640 1 -- 172.21.15.57:0/2510854670 >> 172.21.15.57:0/2510854670 conn(0x7f0598069930 msgr2=0x7f0598069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:26.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.207+0000 7f059dba9640 1 -- 172.21.15.57:0/2510854670 shutdown_connections 2023-10-19T12:45:26.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.207+0000 7f059dba9640 1 -- 172.21.15.57:0/2510854670 wait complete. 2023-10-19T12:45:26.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.208+0000 7f059dba9640 1 Processor -- start 2023-10-19T12:45:26.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.208+0000 7f059dba9640 1 -- start start 2023-10-19T12:45:26.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.209+0000 7f059dba9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0580003fd0 0x7f059812d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:26.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.209+0000 7f059dba9640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f059812caa0 0x7f059812fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:26.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.209+0000 7f059dba9640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f05980aacc0 0x7f059812be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:26.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.209+0000 7f059dba9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0598129e30 con 0x7f0580003fd0 2023-10-19T12:45:26.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.209+0000 7f059dba9640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f059812a110 con 0x7f059812caa0 2023-10-19T12:45:26.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.209+0000 7f059dba9640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f059812a3f0 con 0x7f05980aacc0 2023-10-19T12:45:26.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.209+0000 7f05977fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0580003fd0 0x7f059812d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:26.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.209+0000 7f05977fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0580003fd0 0x7f059812d280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50684/0 (socket says 172.21.15.57:50684) 2023-10-19T12:45:26.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.209+0000 7f05977fe640 1 -- 172.21.15.57:0/2859974314 learned_addr learned my addr 172.21.15.57:0/2859974314 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:26.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.209+0000 7f0597fff640 1 --2- 172.21.15.57:0/2859974314 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f05980aacc0 0x7f059812be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:26.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.209+0000 7f0596ffd640 1 --2- 172.21.15.57:0/2859974314 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f059812caa0 0x7f059812fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:26.216 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.209+0000 7f05977fe640 1 -- 172.21.15.57:0/2859974314 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f05980aacc0 msgr2=0x7f059812be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:26.216 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.209+0000 7f05977fe640 1 --2- 172.21.15.57:0/2859974314 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f05980aacc0 0x7f059812be20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:26.216 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.209+0000 7f05977fe640 1 -- 172.21.15.57:0/2859974314 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f059812caa0 msgr2=0x7f059812fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:26.216 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.209+0000 7f05977fe640 1 --2- 172.21.15.57:0/2859974314 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f059812caa0 0x7f059812fd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:26.216 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.209+0000 7f05977fe640 1 -- 172.21.15.57:0/2859974314 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f059812a6d0 con 0x7f0580003fd0 2023-10-19T12:45:26.216 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.209+0000 7f0596ffd640 1 --2- 172.21.15.57:0/2859974314 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f059812caa0 0x7f059812fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:45:26.217 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.209+0000 7f05977fe640 1 --2- 172.21.15.57:0/2859974314 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0580003fd0 0x7f059812d280 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f0588004650 tx=0x7f058800ba40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:26.217 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.209+0000 7f057ffff640 1 -- 172.21.15.57:0/2859974314 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05880041d0 con 0x7f0580003fd0 2023-10-19T12:45:26.217 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.209+0000 7f057ffff640 1 -- 172.21.15.57:0/2859974314 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f0588015070 con 0x7f0580003fd0 2023-10-19T12:45:26.217 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.209+0000 7f057ffff640 1 -- 172.21.15.57:0/2859974314 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0588010a50 con 0x7f0580003fd0 2023-10-19T12:45:26.217 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.209+0000 7f059dba9640 1 -- 172.21.15.57:0/2859974314 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f059812a9b0 con 0x7f0580003fd0 2023-10-19T12:45:26.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.209+0000 7f059dba9640 1 -- 172.21.15.57:0/2859974314 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05980ab140 con 0x7f0580003fd0 2023-10-19T12:45:26.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.210+0000 7f059dba9640 1 -- 172.21.15.57:0/2859974314 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05980ab3c0 con 0x7f0580003fd0 2023-10-19T12:45:26.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.211+0000 7f057ffff640 1 -- 172.21.15.57:0/2859974314 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f0588017030 con 0x7f0580003fd0 2023-10-19T12:45:26.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.211+0000 7f057ffff640 1 --2- 172.21.15.57:0/2859974314 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f058406ae40 0x7f058406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:26.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.212+0000 7f057ffff640 1 -- 172.21.15.57:0/2859974314 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f058808a350 con 0x7f0580003fd0 2023-10-19T12:45:26.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.212+0000 7f0596ffd640 1 --2- 172.21.15.57:0/2859974314 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f058406ae40 0x7f058406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:26.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.212+0000 7f0596ffd640 1 --2- 172.21.15.57:0/2859974314 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f058406ae40 0x7f058406d300 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f058c034000 tx=0x7f058c001190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:26.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.216+0000 7f057ffff640 1 -- 172.21.15.57:0/2859974314 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0588058ae0 con 0x7f0580003fd0 2023-10-19T12:45:26.398 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.395+0000 7f059dba9640 1 -- 172.21.15.57:0/2859974314 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f059815d630 con 0x7f0580003fd0 2023-10-19T12:45:26.399 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.396+0000 7f057ffff640 1 -- 172.21.15.57:0/2859974314 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18748 (secure 0 0 0) 0x7f0588058670 con 0x7f0580003fd0 2023-10-19T12:45:26.399 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:45:26.400 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":27,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T12:44:58.978403+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-19T12:44:16.517510+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-19T12:44:17.398297+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-19T12:44:21.129403+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.23","mappings":[{"from":3,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"3.39","mappings":[{"from":7,"to":1}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.1d","mappings":[{"from":1,"to":5}]},{"pgid":"4.1e","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T12:45:26.401 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.398+0000 7f059dba9640 1 -- 172.21.15.57:0/2859974314 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f058406ae40 msgr2=0x7f058406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:26.401 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.398+0000 7f059dba9640 1 --2- 172.21.15.57:0/2859974314 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f058406ae40 0x7f058406d300 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f058c034000 tx=0x7f058c001190 comp rx=0 tx=0).stop 2023-10-19T12:45:26.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.398+0000 7f059dba9640 1 -- 172.21.15.57:0/2859974314 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0580003fd0 msgr2=0x7f059812d280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:26.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.398+0000 7f059dba9640 1 --2- 172.21.15.57:0/2859974314 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0580003fd0 0x7f059812d280 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f0588004650 tx=0x7f058800ba40 comp rx=0 tx=0).stop 2023-10-19T12:45:26.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.398+0000 7f059dba9640 1 -- 172.21.15.57:0/2859974314 shutdown_connections 2023-10-19T12:45:26.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.398+0000 7f059dba9640 1 --2- 172.21.15.57:0/2859974314 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f058406ae40 0x7f058406d300 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:26.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.398+0000 7f059dba9640 1 --2- 172.21.15.57:0/2859974314 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f05980aacc0 0x7f059812be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:26.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.398+0000 7f059dba9640 1 --2- 172.21.15.57:0/2859974314 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f059812caa0 0x7f059812fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:26.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.398+0000 7f059dba9640 1 --2- 172.21.15.57:0/2859974314 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0580003fd0 0x7f059812d280 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:26.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.398+0000 7f059dba9640 1 -- 172.21.15.57:0/2859974314 >> 172.21.15.57:0/2859974314 conn(0x7f0598069930 msgr2=0x7f0598069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:26.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.398+0000 7f059dba9640 1 -- 172.21.15.57:0/2859974314 shutdown_connections 2023-10-19T12:45:26.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.398+0000 7f059dba9640 1 -- 172.21.15.57:0/2859974314 wait complete. 2023-10-19T12:45:26.412 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-19T12:44:07.371186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-19T12:44:16.517510+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-19T12:44:17.398297+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-19T12:44:21.129403+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-19T12:45:26.413 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-19T12:45:26.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.521+0000 7fe6dcdf0640 1 -- 172.21.15.57:0/2044709979 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe6d806f0a0 msgr2=0x7fe6d806ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:26.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.521+0000 7fe6dcdf0640 1 --2- 172.21.15.57:0/2044709979 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe6d806f0a0 0x7fe6d806ddc0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fe6d80c9200 tx=0x7fe6c002bdb0 comp rx=0 tx=0).stop 2023-10-19T12:45:26.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.521+0000 7fe6dcdf0640 1 -- 172.21.15.57:0/2044709979 shutdown_connections 2023-10-19T12:45:26.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.521+0000 7fe6dcdf0640 1 --2- 172.21.15.57:0/2044709979 >> v2:172.21.15.179:3300/0 conn(0x7fe6d80ad370 0x7fe6d80ad800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:26.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.521+0000 7fe6dcdf0640 1 --2- 172.21.15.57:0/2044709979 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe6d806f0a0 0x7fe6d806ddc0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:26.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.521+0000 7fe6dcdf0640 1 -- 172.21.15.57:0/2044709979 >> 172.21.15.57:0/2044709979 conn(0x7fe6d80b0670 msgr2=0x7fe6d80b0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:26.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.521+0000 7fe6dcdf0640 1 -- 172.21.15.57:0/2044709979 shutdown_connections 2023-10-19T12:45:26.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.521+0000 7fe6dcdf0640 1 -- 172.21.15.57:0/2044709979 wait complete. 2023-10-19T12:45:26.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.523+0000 7fe6dcdf0640 1 Processor -- start 2023-10-19T12:45:26.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.523+0000 7fe6dcdf0640 1 -- start start 2023-10-19T12:45:26.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.523+0000 7fe6dcdf0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe6d806d2d0 0x7fe6d80c3e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:26.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.523+0000 7fe6dcdf0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe6d806f0a0 0x7fe6d80b2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:26.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.523+0000 7fe6dcdf0640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe6d80ad370 0x7fe6d8154a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:26.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.523+0000 7fe6dcdf0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d80ca0a0 con 0x7fe6d806f0a0 2023-10-19T12:45:26.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.523+0000 7fe6dcdf0640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d80c9700 con 0x7fe6d806d2d0 2023-10-19T12:45:26.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.523+0000 7fe6dcdf0640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d80c99e0 con 0x7fe6d80ad370 2023-10-19T12:45:26.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.524+0000 7fe6d5d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe6d806f0a0 0x7fe6d80b2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:26.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.524+0000 7fe6d5d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe6d806f0a0 0x7fe6d80b2070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50690/0 (socket says 172.21.15.57:50690) 2023-10-19T12:45:26.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.524+0000 7fe6d5d9b640 1 -- 172.21.15.57:0/1781157265 learned_addr learned my addr 172.21.15.57:0/1781157265 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:26.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.524+0000 7fe6d6d9d640 1 --2- 172.21.15.57:0/1781157265 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe6d80ad370 0x7fe6d8154a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:26.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.524+0000 7fe6d659c640 1 --2- 172.21.15.57:0/1781157265 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe6d806d2d0 0x7fe6d80c3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:26.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.524+0000 7fe6d5d9b640 1 -- 172.21.15.57:0/1781157265 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe6d80ad370 msgr2=0x7fe6d8154a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:26.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.524+0000 7fe6d5d9b640 1 --2- 172.21.15.57:0/1781157265 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe6d80ad370 0x7fe6d8154a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:26.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.524+0000 7fe6d5d9b640 1 -- 172.21.15.57:0/1781157265 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe6d806d2d0 msgr2=0x7fe6d80c3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:26.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.524+0000 7fe6d5d9b640 1 --2- 172.21.15.57:0/1781157265 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe6d806d2d0 0x7fe6d80c3e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:26.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.524+0000 7fe6d5d9b640 1 -- 172.21.15.57:0/1781157265 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6c0009560 con 0x7fe6d806f0a0 2023-10-19T12:45:26.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.524+0000 7fe6d5d9b640 1 --2- 172.21.15.57:0/1781157265 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe6d806f0a0 0x7fe6d80b2070 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fe6c0009850 tx=0x7fe6c002dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:26.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.524+0000 7fe6ceffd640 1 -- 172.21.15.57:0/1781157265 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6c0003800 con 0x7fe6d806f0a0 2023-10-19T12:45:26.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.524+0000 7fe6ceffd640 1 -- 172.21.15.57:0/1781157265 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fe6c0003a50 con 0x7fe6d806f0a0 2023-10-19T12:45:26.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.524+0000 7fe6dcdf0640 1 -- 172.21.15.57:0/1781157265 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6d80c9cc0 con 0x7fe6d806f0a0 2023-10-19T12:45:26.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.524+0000 7fe6ceffd640 1 -- 172.21.15.57:0/1781157265 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6c0003e20 con 0x7fe6d806f0a0 2023-10-19T12:45:26.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.524+0000 7fe6dcdf0640 1 -- 172.21.15.57:0/1781157265 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6d80c43c0 con 0x7fe6d806f0a0 2023-10-19T12:45:26.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.525+0000 7fe6dcdf0640 1 -- 172.21.15.57:0/1781157265 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6d80ce1d0 con 0x7fe6d806f0a0 2023-10-19T12:45:26.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.526+0000 7fe6ceffd640 1 -- 172.21.15.57:0/1781157265 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fe6c003e450 con 0x7fe6d806f0a0 2023-10-19T12:45:26.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.526+0000 7fe6ceffd640 1 --2- 172.21.15.57:0/1781157265 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fe6b806ae40 0x7fe6b806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:26.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.526+0000 7fe6ceffd640 1 -- 172.21.15.57:0/1781157265 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fe6c00ac460 con 0x7fe6d806f0a0 2023-10-19T12:45:26.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.526+0000 7fe6d659c640 1 --2- 172.21.15.57:0/1781157265 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fe6b806ae40 0x7fe6b806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:26.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.527+0000 7fe6d659c640 1 --2- 172.21.15.57:0/1781157265 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fe6b806ae40 0x7fe6b806d300 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fe6c8007fd0 tx=0x7fe6c8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:26.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.530+0000 7fe6ceffd640 1 -- 172.21.15.57:0/1781157265 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe6c00771e0 con 0x7fe6d806f0a0 2023-10-19T12:45:26.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.705+0000 7fe6dcdf0640 1 -- 172.21.15.57:0/1781157265 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe6d80ca1e0 con 0x7fe6d806f0a0 2023-10-19T12:45:26.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.706+0000 7fe6ceffd640 1 -- 172.21.15.57:0/1781157265 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe6d80ca1e0 con 0x7fe6d806f0a0 2023-10-19T12:45:26.709 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-19T12:45:26.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.708+0000 7fe6dcdf0640 1 -- 172.21.15.57:0/1781157265 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fe6b806ae40 msgr2=0x7fe6b806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:26.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.708+0000 7fe6dcdf0640 1 --2- 172.21.15.57:0/1781157265 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fe6b806ae40 0x7fe6b806d300 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fe6c8007fd0 tx=0x7fe6c8009040 comp rx=0 tx=0).stop 2023-10-19T12:45:26.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.708+0000 7fe6dcdf0640 1 -- 172.21.15.57:0/1781157265 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe6d806f0a0 msgr2=0x7fe6d80b2070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:26.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.708+0000 7fe6dcdf0640 1 --2- 172.21.15.57:0/1781157265 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe6d806f0a0 0x7fe6d80b2070 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fe6c0009850 tx=0x7fe6c002dae0 comp rx=0 tx=0).stop 2023-10-19T12:45:26.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.708+0000 7fe6dcdf0640 1 -- 172.21.15.57:0/1781157265 shutdown_connections 2023-10-19T12:45:26.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.708+0000 7fe6dcdf0640 1 --2- 172.21.15.57:0/1781157265 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fe6b806ae40 0x7fe6b806d300 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:26.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.708+0000 7fe6dcdf0640 1 --2- 172.21.15.57:0/1781157265 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe6d80ad370 0x7fe6d8154a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:26.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.708+0000 7fe6dcdf0640 1 --2- 172.21.15.57:0/1781157265 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe6d806f0a0 0x7fe6d80b2070 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:26.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.708+0000 7fe6dcdf0640 1 --2- 172.21.15.57:0/1781157265 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe6d806d2d0 0x7fe6d80c3e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:26.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.708+0000 7fe6dcdf0640 1 -- 172.21.15.57:0/1781157265 >> 172.21.15.57:0/1781157265 conn(0x7fe6d80b0670 msgr2=0x7fe6d80b0a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:26.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.708+0000 7fe6dcdf0640 1 -- 172.21.15.57:0/1781157265 shutdown_connections 2023-10-19T12:45:26.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.708+0000 7fe6dcdf0640 1 -- 172.21.15.57:0/1781157265 wait complete. 2023-10-19T12:45:26.721 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-19T12:45:26.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.831+0000 7f0185ca9640 1 -- 172.21.15.57:0/89965224 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0168003970 msgr2=0x7f0168003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:26.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.831+0000 7f0185ca9640 1 --2- 172.21.15.57:0/89965224 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0168003970 0x7f0168003d90 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f0170009b80 tx=0x7f01680057f0 comp rx=0 tx=0).stop 2023-10-19T12:45:26.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.831+0000 7f017ffff640 1 -- 172.21.15.57:0/89965224 reap_dead start 2023-10-19T12:45:26.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.831+0000 7f0185ca9640 1 -- 172.21.15.57:0/89965224 shutdown_connections 2023-10-19T12:45:26.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.831+0000 7f0185ca9640 1 -- 172.21.15.57:0/89965224 >> 172.21.15.57:0/89965224 conn(0x7f01800d0d60 msgr2=0x7f01800d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:26.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.831+0000 7f0185ca9640 1 -- 172.21.15.57:0/89965224 shutdown_connections 2023-10-19T12:45:26.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.831+0000 7f0185ca9640 1 -- 172.21.15.57:0/89965224 wait complete. 2023-10-19T12:45:26.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.832+0000 7f0185ca9640 1 Processor -- start 2023-10-19T12:45:26.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.833+0000 7f0185ca9640 1 -- start start 2023-10-19T12:45:26.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.833+0000 7f0185ca9640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0168003970 0x7f018015e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:26.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.833+0000 7f0185ca9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f018015ccc0 0x7f01801f1f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:26.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.833+0000 7f0185ca9640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f01800b60c0 0x7f01800c85b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:26.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.833+0000 7f0185ca9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01801f2450 con 0x7f018015ccc0 2023-10-19T12:45:26.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.833+0000 7f0185ca9640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f018015ddb0 con 0x7f01800b60c0 2023-10-19T12:45:26.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.833+0000 7f0185ca9640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f018015d1a0 con 0x7f0168003970 2023-10-19T12:45:26.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.833+0000 7f017effd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f018015ccc0 0x7f01801f1f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:26.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.833+0000 7f017effd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f018015ccc0 0x7f01801f1f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33282/0 (socket says 172.21.15.57:33282) 2023-10-19T12:45:26.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.833+0000 7f017effd640 1 -- 172.21.15.57:0/2815915882 learned_addr learned my addr 172.21.15.57:0/2815915882 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:26.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.834+0000 7f017f7fe640 1 --2- 172.21.15.57:0/2815915882 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0168003970 0x7f018015e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:26.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.834+0000 7f017ffff640 1 --2- 172.21.15.57:0/2815915882 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f01800b60c0 0x7f01800c85b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:26.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.834+0000 7f017effd640 1 -- 172.21.15.57:0/2815915882 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0168003970 msgr2=0x7f018015e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:26.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.834+0000 7f017effd640 1 --2- 172.21.15.57:0/2815915882 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0168003970 0x7f018015e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:26.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.834+0000 7f017effd640 1 -- 172.21.15.57:0/2815915882 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f01800b60c0 msgr2=0x7f01800c85b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:26.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.834+0000 7f017effd640 1 --2- 172.21.15.57:0/2815915882 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f01800b60c0 0x7f01800c85b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:26.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.834+0000 7f017effd640 1 -- 172.21.15.57:0/2815915882 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f018015d480 con 0x7f018015ccc0 2023-10-19T12:45:26.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.834+0000 7f017ffff640 1 --2- 172.21.15.57:0/2815915882 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f01800b60c0 0x7f01800c85b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:45:26.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.834+0000 7f017f7fe640 1 --2- 172.21.15.57:0/2815915882 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0168003970 0x7f018015e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:45:26.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.834+0000 7f017effd640 1 --2- 172.21.15.57:0/2815915882 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f018015ccc0 0x7f01801f1f10 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f017400e600 tx=0x7f017400eac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:26.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.834+0000 7f0167fff640 1 -- 172.21.15.57:0/2815915882 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f017400c940 con 0x7f018015ccc0 2023-10-19T12:45:26.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.834+0000 7f0167fff640 1 -- 172.21.15.57:0/2815915882 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f017400cb90 con 0x7f018015ccc0 2023-10-19T12:45:26.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.834+0000 7f0185ca9640 1 -- 172.21.15.57:0/2815915882 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f018015d760 con 0x7f018015ccc0 2023-10-19T12:45:26.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.834+0000 7f0167fff640 1 -- 172.21.15.57:0/2815915882 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0174011460 con 0x7f018015ccc0 2023-10-19T12:45:26.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.834+0000 7f0185ca9640 1 -- 172.21.15.57:0/2815915882 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01800cab70 con 0x7f018015ccc0 2023-10-19T12:45:26.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.835+0000 7f0185ca9640 1 -- 172.21.15.57:0/2815915882 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f018015df20 con 0x7f018015ccc0 2023-10-19T12:45:26.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.836+0000 7f0167fff640 1 -- 172.21.15.57:0/2815915882 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f01740115c0 con 0x7f018015ccc0 2023-10-19T12:45:26.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.836+0000 7f0167fff640 1 --2- 172.21.15.57:0/2815915882 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f016c06ae70 0x7f016c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:26.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.836+0000 7f0167fff640 1 -- 172.21.15.57:0/2815915882 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f017408a130 con 0x7f018015ccc0 2023-10-19T12:45:26.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.836+0000 7f017f7fe640 1 --2- 172.21.15.57:0/2815915882 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f016c06ae70 0x7f016c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:26.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.837+0000 7f017f7fe640 1 --2- 172.21.15.57:0/2815915882 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f016c06ae70 0x7f016c06d330 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f0170036000 tx=0x7f0170035040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:26.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:26.840+0000 7f0167fff640 1 -- 172.21.15.57:0/2815915882 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0174054eb0 con 0x7f018015ccc0 2023-10-19T12:45:27.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.013+0000 7f0185ca9640 1 -- 172.21.15.57:0/2815915882 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f01800690e0 con 0x7f018015ccc0 2023-10-19T12:45:27.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.014+0000 7f0167fff640 1 -- 172.21.15.57:0/2815915882 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7f0174015020 con 0x7f018015ccc0 2023-10-19T12:45:27.017 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2023-10-19T12:45:27.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.015+0000 7f0185ca9640 1 -- 172.21.15.57:0/2815915882 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f016c06ae70 msgr2=0x7f016c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:27.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.015+0000 7f0185ca9640 1 --2- 172.21.15.57:0/2815915882 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f016c06ae70 0x7f016c06d330 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f0170036000 tx=0x7f0170035040 comp rx=0 tx=0).stop 2023-10-19T12:45:27.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.015+0000 7f0185ca9640 1 -- 172.21.15.57:0/2815915882 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f018015ccc0 msgr2=0x7f01801f1f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:27.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.015+0000 7f0185ca9640 1 --2- 172.21.15.57:0/2815915882 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f018015ccc0 0x7f01801f1f10 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f017400e600 tx=0x7f017400eac0 comp rx=0 tx=0).stop 2023-10-19T12:45:27.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.015+0000 7f0185ca9640 1 -- 172.21.15.57:0/2815915882 shutdown_connections 2023-10-19T12:45:27.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.015+0000 7f0185ca9640 1 --2- 172.21.15.57:0/2815915882 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f016c06ae70 0x7f016c06d330 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:27.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.015+0000 7f0185ca9640 1 --2- 172.21.15.57:0/2815915882 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f01800b60c0 0x7f01800c85b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:27.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.015+0000 7f0185ca9640 1 --2- 172.21.15.57:0/2815915882 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f018015ccc0 0x7f01801f1f10 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:27.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.015+0000 7f0185ca9640 1 --2- 172.21.15.57:0/2815915882 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0168003970 0x7f018015e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:27.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.016+0000 7f0185ca9640 1 -- 172.21.15.57:0/2815915882 >> 172.21.15.57:0/2815915882 conn(0x7f01800d0d60 msgr2=0x7f0180158a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:27.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.016+0000 7f0185ca9640 1 -- 172.21.15.57:0/2815915882 shutdown_connections 2023-10-19T12:45:27.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.016+0000 7f0185ca9640 1 -- 172.21.15.57:0/2815915882 wait complete. 2023-10-19T12:45:27.029 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-19T12:45:27.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.139+0000 7fc51bfd0640 1 -- 172.21.15.57:0/1737903412 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc51415b2a0 msgr2=0x7fc5140cf720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:27.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.139+0000 7fc51bfd0640 1 --2- 172.21.15.57:0/1737903412 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc51415b2a0 0x7fc5140cf720 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fc51408a810 tx=0x7fc50402bdb0 comp rx=0 tx=0).stop 2023-10-19T12:45:27.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.139+0000 7fc51bfd0640 1 -- 172.21.15.57:0/1737903412 shutdown_connections 2023-10-19T12:45:27.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.139+0000 7fc51bfd0640 1 --2- 172.21.15.57:0/1737903412 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc51415b2a0 0x7fc5140cf720 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:27.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.139+0000 7fc51bfd0640 1 --2- 172.21.15.57:0/1737903412 >> v2:172.21.15.179:3300/0 conn(0x7fc5140cf340 0x7fc5140d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:27.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.139+0000 7fc51bfd0640 1 -- 172.21.15.57:0/1737903412 >> 172.21.15.57:0/1737903412 conn(0x7fc5140d0d60 msgr2=0x7fc5140d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:27.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.139+0000 7fc51bfd0640 1 -- 172.21.15.57:0/1737903412 shutdown_connections 2023-10-19T12:45:27.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.139+0000 7fc51bfd0640 1 -- 172.21.15.57:0/1737903412 wait complete. 2023-10-19T12:45:27.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.140+0000 7fc51bfd0640 1 Processor -- start 2023-10-19T12:45:27.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.141+0000 7fc51bfd0640 1 -- start start 2023-10-19T12:45:27.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.141+0000 7fc51bfd0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5140ceb40 0x7fc51414f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:27.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.141+0000 7fc51bfd0640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc5140cf340 0x7fc51415b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:27.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.141+0000 7fc51bfd0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc51415b2a0 0x7fc5141e3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:27.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.141+0000 7fc51bfd0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5140c9f50 con 0x7fc5140ceb40 2023-10-19T12:45:27.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.141+0000 7fc51bfd0640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc51414f9b0 con 0x7fc51415b2a0 2023-10-19T12:45:27.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.141+0000 7fc51bfd0640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5140c9340 con 0x7fc5140cf340 2023-10-19T12:45:27.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.141+0000 7fc519d6c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5140ceb40 0x7fc51414f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:27.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.141+0000 7fc519d6c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5140ceb40 0x7fc51414f470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33310/0 (socket says 172.21.15.57:33310) 2023-10-19T12:45:27.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.141+0000 7fc519d6c640 1 -- 172.21.15.57:0/598869462 learned_addr learned my addr 172.21.15.57:0/598869462 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:27.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.142+0000 7fc51956b640 1 --2- 172.21.15.57:0/598869462 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc5140cf340 0x7fc51415b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:27.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.142+0000 7fc51a56d640 1 --2- 172.21.15.57:0/598869462 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc51415b2a0 0x7fc5141e3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:27.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.142+0000 7fc519d6c640 1 -- 172.21.15.57:0/598869462 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc5140cf340 msgr2=0x7fc51415b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:27.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.142+0000 7fc519d6c640 1 --2- 172.21.15.57:0/598869462 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc5140cf340 0x7fc51415b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:27.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.142+0000 7fc519d6c640 1 -- 172.21.15.57:0/598869462 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc51415b2a0 msgr2=0x7fc5141e3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:27.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.142+0000 7fc519d6c640 1 --2- 172.21.15.57:0/598869462 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc51415b2a0 0x7fc5141e3870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:27.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.142+0000 7fc519d6c640 1 -- 172.21.15.57:0/598869462 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc504009560 con 0x7fc5140ceb40 2023-10-19T12:45:27.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.142+0000 7fc51956b640 1 --2- 172.21.15.57:0/598869462 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc5140cf340 0x7fc51415b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:45:27.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.142+0000 7fc51a56d640 1 --2- 172.21.15.57:0/598869462 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc51415b2a0 0x7fc5141e3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:45:27.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.142+0000 7fc519d6c640 1 --2- 172.21.15.57:0/598869462 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5140ceb40 0x7fc51414f470 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fc51000d980 tx=0x7fc51000de40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:27.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.142+0000 7fc50a7fc640 1 -- 172.21.15.57:0/598869462 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc510009c30 con 0x7fc5140ceb40 2023-10-19T12:45:27.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.142+0000 7fc50a7fc640 1 -- 172.21.15.57:0/598869462 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fc510009e80 con 0x7fc5140ceb40 2023-10-19T12:45:27.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.142+0000 7fc51bfd0640 1 -- 172.21.15.57:0/598869462 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5140c9620 con 0x7fc5140ceb40 2023-10-19T12:45:27.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.142+0000 7fc50a7fc640 1 -- 172.21.15.57:0/598869462 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5100106a0 con 0x7fc5140ceb40 2023-10-19T12:45:27.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.142+0000 7fc51bfd0640 1 -- 172.21.15.57:0/598869462 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5140c9be0 con 0x7fc5140ceb40 2023-10-19T12:45:27.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.143+0000 7fc51bfd0640 1 -- 172.21.15.57:0/598869462 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5140690e0 con 0x7fc5140ceb40 2023-10-19T12:45:27.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.144+0000 7fc50a7fc640 1 -- 172.21.15.57:0/598869462 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fc510010800 con 0x7fc5140ceb40 2023-10-19T12:45:27.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.144+0000 7fc50a7fc640 1 --2- 172.21.15.57:0/598869462 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fc4f806ae40 0x7fc4f806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:27.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.144+0000 7fc50a7fc640 1 -- 172.21.15.57:0/598869462 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fc510089ec0 con 0x7fc5140ceb40 2023-10-19T12:45:27.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.144+0000 7fc51956b640 1 --2- 172.21.15.57:0/598869462 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fc4f806ae40 0x7fc4f806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:27.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.145+0000 7fc51956b640 1 --2- 172.21.15.57:0/598869462 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fc4f806ae40 0x7fc4f806d300 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fc504035000 tx=0x7fc504004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:27.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.148+0000 7fc50a7fc640 1 -- 172.21.15.57:0/598869462 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc510054bb0 con 0x7fc5140ceb40 2023-10-19T12:45:27.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.324+0000 7fc51bfd0640 1 -- 172.21.15.57:0/598869462 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc5140cb9d0 con 0x7fc5140ceb40 2023-10-19T12:45:27.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.324+0000 7fc50a7fc640 1 -- 172.21.15.57:0/598869462 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc5100170a0 con 0x7fc5140ceb40 2023-10-19T12:45:27.328 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 64 2023-10-19T12:45:27.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.327+0000 7fc51bfd0640 1 -- 172.21.15.57:0/598869462 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fc4f806ae40 msgr2=0x7fc4f806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:27.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.327+0000 7fc51bfd0640 1 --2- 172.21.15.57:0/598869462 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fc4f806ae40 0x7fc4f806d300 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fc504035000 tx=0x7fc504004730 comp rx=0 tx=0).stop 2023-10-19T12:45:27.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.327+0000 7fc51bfd0640 1 -- 172.21.15.57:0/598869462 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5140ceb40 msgr2=0x7fc51414f470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:27.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.327+0000 7fc51bfd0640 1 --2- 172.21.15.57:0/598869462 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5140ceb40 0x7fc51414f470 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fc51000d980 tx=0x7fc51000de40 comp rx=0 tx=0).stop 2023-10-19T12:45:27.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.327+0000 7fc51bfd0640 1 -- 172.21.15.57:0/598869462 shutdown_connections 2023-10-19T12:45:27.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.327+0000 7fc51bfd0640 1 --2- 172.21.15.57:0/598869462 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fc4f806ae40 0x7fc4f806d300 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:27.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.327+0000 7fc51bfd0640 1 --2- 172.21.15.57:0/598869462 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc51415b2a0 0x7fc5141e3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:27.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.327+0000 7fc51bfd0640 1 --2- 172.21.15.57:0/598869462 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc5140cf340 0x7fc51415b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:27.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.327+0000 7fc51bfd0640 1 --2- 172.21.15.57:0/598869462 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5140ceb40 0x7fc51414f470 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:27.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.327+0000 7fc51bfd0640 1 -- 172.21.15.57:0/598869462 >> 172.21.15.57:0/598869462 conn(0x7fc5140d0d60 msgr2=0x7fc51415a180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:27.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.327+0000 7fc51bfd0640 1 -- 172.21.15.57:0/598869462 shutdown_connections 2023-10-19T12:45:27.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.327+0000 7fc51bfd0640 1 -- 172.21.15.57:0/598869462 wait complete. 2023-10-19T12:45:27.340 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-19T12:45:27.453 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.450+0000 7fb4c1b53640 1 -- 172.21.15.57:0/2637998515 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4a40047c0 msgr2=0x7fb4a4004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:27.453 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.450+0000 7fb4c1b53640 1 --2- 172.21.15.57:0/2637998515 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4a40047c0 0x7fb4a4004c40 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fb4b0009b20 tx=0x7fb4a40057f0 comp rx=0 tx=0).stop 2023-10-19T12:45:27.453 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.450+0000 7fb4bbfff640 1 -- 172.21.15.57:0/2637998515 reap_dead start 2023-10-19T12:45:27.453 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.450+0000 7fb4c1b53640 1 -- 172.21.15.57:0/2637998515 shutdown_connections 2023-10-19T12:45:27.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.450+0000 7fb4c1b53640 1 -- 172.21.15.57:0/2637998515 >> 172.21.15.57:0/2637998515 conn(0x7fb4bc0af7a0 msgr2=0x7fb4bc0afb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:27.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.450+0000 7fb4c1b53640 1 -- 172.21.15.57:0/2637998515 shutdown_connections 2023-10-19T12:45:27.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.450+0000 7fb4c1b53640 1 -- 172.21.15.57:0/2637998515 wait complete. 2023-10-19T12:45:27.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.451+0000 7fb4c1b53640 1 Processor -- start 2023-10-19T12:45:27.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.451+0000 7fb4c1b53640 1 -- start start 2023-10-19T12:45:27.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.452+0000 7fb4c1b53640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4a40047c0 0x7fb4bc148ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:27.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.452+0000 7fb4c1b53640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb4bc136470 0x7fb4bc14c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:27.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.452+0000 7fb4c1b53640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb4bc1e16e0 0x7fb4bc1e1b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:27.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.452+0000 7fb4c1b53640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4bc1e2060 con 0x7fb4a40047c0 2023-10-19T12:45:27.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.452+0000 7fb4c1b53640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4bc14c840 con 0x7fb4bc1e16e0 2023-10-19T12:45:27.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.452+0000 7fb4c1b53640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4bc1368f0 con 0x7fb4bc136470 2023-10-19T12:45:27.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.452+0000 7fb4bb7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4a40047c0 0x7fb4bc148ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:27.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.452+0000 7fb4bb7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4a40047c0 0x7fb4bc148ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33324/0 (socket says 172.21.15.57:33324) 2023-10-19T12:45:27.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.452+0000 7fb4bb7fe640 1 -- 172.21.15.57:0/2835840148 learned_addr learned my addr 172.21.15.57:0/2835840148 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:27.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.452+0000 7fb4baffd640 1 --2- 172.21.15.57:0/2835840148 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb4bc136470 0x7fb4bc14c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:27.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.452+0000 7fb4bbfff640 1 --2- 172.21.15.57:0/2835840148 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb4bc1e16e0 0x7fb4bc1e1b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:27.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.452+0000 7fb4bb7fe640 1 -- 172.21.15.57:0/2835840148 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb4bc136470 msgr2=0x7fb4bc14c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:27.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.452+0000 7fb4bb7fe640 1 --2- 172.21.15.57:0/2835840148 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb4bc136470 0x7fb4bc14c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:27.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.452+0000 7fb4bb7fe640 1 -- 172.21.15.57:0/2835840148 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb4bc1e16e0 msgr2=0x7fb4bc1e1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:27.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.452+0000 7fb4bb7fe640 1 --2- 172.21.15.57:0/2835840148 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb4bc1e16e0 0x7fb4bc1e1b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:27.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.452+0000 7fb4bb7fe640 1 -- 172.21.15.57:0/2835840148 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4bc14cae0 con 0x7fb4a40047c0 2023-10-19T12:45:27.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.452+0000 7fb4bb7fe640 1 --2- 172.21.15.57:0/2835840148 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4a40047c0 0x7fb4bc148ef0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac001200 tx=0x7fb4ac00eb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:27.461 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.452+0000 7fb4a3fff640 1 -- 172.21.15.57:0/2835840148 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4ac004250 con 0x7fb4a40047c0 2023-10-19T12:45:27.461 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.452+0000 7fb4a3fff640 1 -- 172.21.15.57:0/2835840148 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fb4ac0044a0 con 0x7fb4a40047c0 2023-10-19T12:45:27.461 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.452+0000 7fb4a3fff640 1 -- 172.21.15.57:0/2835840148 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4ac00ca70 con 0x7fb4a40047c0 2023-10-19T12:45:27.461 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.452+0000 7fb4c1b53640 1 -- 172.21.15.57:0/2835840148 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4bc14cdc0 con 0x7fb4a40047c0 2023-10-19T12:45:27.461 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.453+0000 7fb4c1b53640 1 -- 172.21.15.57:0/2835840148 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4bc14d380 con 0x7fb4a40047c0 2023-10-19T12:45:27.461 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.453+0000 7fb4c1b53640 1 -- 172.21.15.57:0/2835840148 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4bc0690e0 con 0x7fb4a40047c0 2023-10-19T12:45:27.462 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.454+0000 7fb4a3fff640 1 -- 172.21.15.57:0/2835840148 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fb4ac00cbd0 con 0x7fb4a40047c0 2023-10-19T12:45:27.462 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.454+0000 7fb4a3fff640 1 --2- 172.21.15.57:0/2835840148 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fb4a806ae40 0x7fb4a806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:27.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.454+0000 7fb4a3fff640 1 -- 172.21.15.57:0/2835840148 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fb4ac089f40 con 0x7fb4a40047c0 2023-10-19T12:45:27.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.455+0000 7fb4baffd640 1 --2- 172.21.15.57:0/2835840148 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fb4a806ae40 0x7fb4a806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:27.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.455+0000 7fb4baffd640 1 --2- 172.21.15.57:0/2835840148 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fb4a806ae40 0x7fb4a806d300 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fb4b0036000 tx=0x7fb4b00046c0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:27.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.458+0000 7fb4a3fff640 1 -- 172.21.15.57:0/2835840148 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb4ac054d70 con 0x7fb4a40047c0 2023-10-19T12:45:27.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.632+0000 7fb4c1b53640 1 -- 172.21.15.57:0/2835840148 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fb4bc0cbe10 con 0x7fb4a40047c0 2023-10-19T12:45:27.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.632+0000 7fb4a3fff640 1 -- 172.21.15.57:0/2835840148 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7fb4ac058390 con 0x7fb4a40047c0 2023-10-19T12:45:27.636 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2023-10-19T12:45:27.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.634+0000 7fb4c1b53640 1 -- 172.21.15.57:0/2835840148 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fb4a806ae40 msgr2=0x7fb4a806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:27.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.634+0000 7fb4c1b53640 1 --2- 172.21.15.57:0/2835840148 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fb4a806ae40 0x7fb4a806d300 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fb4b0036000 tx=0x7fb4b00046c0 comp rx=0 tx=0).stop 2023-10-19T12:45:27.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.634+0000 7fb4c1b53640 1 -- 172.21.15.57:0/2835840148 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4a40047c0 msgr2=0x7fb4bc148ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:27.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.634+0000 7fb4c1b53640 1 --2- 172.21.15.57:0/2835840148 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4a40047c0 0x7fb4bc148ef0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac001200 tx=0x7fb4ac00eb60 comp rx=0 tx=0).stop 2023-10-19T12:45:27.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.634+0000 7fb4c1b53640 1 -- 172.21.15.57:0/2835840148 shutdown_connections 2023-10-19T12:45:27.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.634+0000 7fb4c1b53640 1 --2- 172.21.15.57:0/2835840148 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fb4a806ae40 0x7fb4a806d300 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:27.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.634+0000 7fb4c1b53640 1 --2- 172.21.15.57:0/2835840148 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb4bc1e16e0 0x7fb4bc1e1b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:27.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.634+0000 7fb4c1b53640 1 --2- 172.21.15.57:0/2835840148 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb4bc136470 0x7fb4bc14c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:27.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.634+0000 7fb4c1b53640 1 --2- 172.21.15.57:0/2835840148 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4a40047c0 0x7fb4bc148ef0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:27.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.634+0000 7fb4c1b53640 1 -- 172.21.15.57:0/2835840148 >> 172.21.15.57:0/2835840148 conn(0x7fb4bc0af7a0 msgr2=0x7fb4bc0cc590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:27.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.634+0000 7fb4c1b53640 1 -- 172.21.15.57:0/2835840148 shutdown_connections 2023-10-19T12:45:27.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.634+0000 7fb4c1b53640 1 -- 172.21.15.57:0/2835840148 wait complete. 2023-10-19T12:45:27.648 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T12:45:27.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.754+0000 7f5a83c15640 1 -- 172.21.15.57:0/2871312432 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a7c0cd0c0 msgr2=0x7f5a7c0cd4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:27.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.754+0000 7f5a83c15640 1 --2- 172.21.15.57:0/2871312432 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a7c0cd0c0 0x7f5a7c0cd4a0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f5a7c0667c0 tx=0x7f5a7802c2e0 comp rx=0 tx=0).stop 2023-10-19T12:45:27.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.754+0000 7f5a83c15640 1 -- 172.21.15.57:0/2871312432 shutdown_connections 2023-10-19T12:45:27.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.754+0000 7f5a83c15640 1 --2- 172.21.15.57:0/2871312432 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5a7c06dce0 0x7f5a7c06b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:27.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.754+0000 7f5a83c15640 1 --2- 172.21.15.57:0/2871312432 >> v2:172.21.15.179:3300/0 conn(0x7f5a7c06bf20 0x7f5a7c06ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:27.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.754+0000 7f5a83c15640 1 --2- 172.21.15.57:0/2871312432 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a7c0cd0c0 0x7f5a7c0cd4a0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:27.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.754+0000 7f5a83c15640 1 -- 172.21.15.57:0/2871312432 >> 172.21.15.57:0/2871312432 conn(0x7f5a7c0b0670 msgr2=0x7f5a7c0b0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:27.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.754+0000 7f5a83c15640 1 -- 172.21.15.57:0/2871312432 shutdown_connections 2023-10-19T12:45:27.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.754+0000 7f5a83c15640 1 -- 172.21.15.57:0/2871312432 wait complete. 2023-10-19T12:45:27.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.755+0000 7f5a83c15640 1 Processor -- start 2023-10-19T12:45:27.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.755+0000 7f5a83c15640 1 -- start start 2023-10-19T12:45:27.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.756+0000 7f5a83c15640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a7c06bf20 0x7f5a7c154160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:27.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.756+0000 7f5a83c15640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5a7c06dce0 0x7f5a7c1f1460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:27.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.756+0000 7f5a83c15640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5a7c0cd0c0 0x7f5a7c1e28c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:27.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.756+0000 7f5a83c15640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a7c1e0390 con 0x7f5a7c06bf20 2023-10-19T12:45:27.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.756+0000 7f5a83c15640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a7c157240 con 0x7f5a7c0cd0c0 2023-10-19T12:45:27.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.756+0000 7f5a83c15640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a7c1e05e0 con 0x7f5a7c06dce0 2023-10-19T12:45:27.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.756+0000 7f5a819b1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a7c06bf20 0x7f5a7c154160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:27.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.756+0000 7f5a819b1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a7c06bf20 0x7f5a7c154160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33346/0 (socket says 172.21.15.57:33346) 2023-10-19T12:45:27.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.756+0000 7f5a819b1640 1 -- 172.21.15.57:0/2034211554 learned_addr learned my addr 172.21.15.57:0/2034211554 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:27.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.756+0000 7f5a821b2640 1 --2- 172.21.15.57:0/2034211554 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5a7c0cd0c0 0x7f5a7c1e28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:27.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.756+0000 7f5a819b1640 1 -- 172.21.15.57:0/2034211554 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5a7c06dce0 msgr2=0x7f5a7c1f1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:27.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.756+0000 7f5a811b0640 1 --2- 172.21.15.57:0/2034211554 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5a7c06dce0 0x7f5a7c1f1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:27.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.756+0000 7f5a819b1640 1 --2- 172.21.15.57:0/2034211554 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5a7c06dce0 0x7f5a7c1f1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:27.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.756+0000 7f5a819b1640 1 -- 172.21.15.57:0/2034211554 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5a7c0cd0c0 msgr2=0x7f5a7c1e28c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:27.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.756+0000 7f5a819b1640 1 --2- 172.21.15.57:0/2034211554 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5a7c0cd0c0 0x7f5a7c1e28c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:27.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.756+0000 7f5a819b1640 1 -- 172.21.15.57:0/2034211554 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a78009900 con 0x7f5a7c06bf20 2023-10-19T12:45:27.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.756+0000 7f5a821b2640 1 --2- 172.21.15.57:0/2034211554 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5a7c0cd0c0 0x7f5a7c1e28c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:45:27.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.757+0000 7f5a819b1640 1 --2- 172.21.15.57:0/2034211554 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a7c06bf20 0x7f5a7c154160 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f5a78000f80 tx=0x7f5a78035ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:27.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.757+0000 7f5a727fc640 1 -- 172.21.15.57:0/2034211554 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a78004240 con 0x7f5a7c06bf20 2023-10-19T12:45:27.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.757+0000 7f5a727fc640 1 -- 172.21.15.57:0/2034211554 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f5a78004490 con 0x7f5a7c06bf20 2023-10-19T12:45:27.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.757+0000 7f5a83c15640 1 -- 172.21.15.57:0/2034211554 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a7c1e08c0 con 0x7f5a7c06bf20 2023-10-19T12:45:27.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.757+0000 7f5a727fc640 1 -- 172.21.15.57:0/2034211554 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a78004880 con 0x7f5a7c06bf20 2023-10-19T12:45:27.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.757+0000 7f5a83c15640 1 -- 172.21.15.57:0/2034211554 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a7c1e2e00 con 0x7f5a7c06bf20 2023-10-19T12:45:27.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.758+0000 7f5a83c15640 1 -- 172.21.15.57:0/2034211554 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a7c0690e0 con 0x7f5a7c06bf20 2023-10-19T12:45:27.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.758+0000 7f5a727fc640 1 -- 172.21.15.57:0/2034211554 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f5a7803b040 con 0x7f5a7c06bf20 2023-10-19T12:45:27.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.759+0000 7f5a727fc640 1 --2- 172.21.15.57:0/2034211554 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f5a6006ae40 0x7f5a6006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:27.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.759+0000 7f5a727fc640 1 -- 172.21.15.57:0/2034211554 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f5a780ac6c0 con 0x7f5a7c06bf20 2023-10-19T12:45:27.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.759+0000 7f5a811b0640 1 --2- 172.21.15.57:0/2034211554 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f5a6006ae40 0x7f5a6006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:27.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.759+0000 7f5a811b0640 1 --2- 172.21.15.57:0/2034211554 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f5a6006ae40 0x7f5a6006d300 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f5a6c009960 tx=0x7f5a6c005c90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:27.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.763+0000 7f5a727fc640 1 -- 172.21.15.57:0/2034211554 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f5a780773a0 con 0x7f5a7c06bf20 2023-10-19T12:45:27.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.974+0000 7f5a83c15640 1 -- 172.21.15.57:0/2034211554 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5a7c0675f0 con 0x7f5a7c06bf20 2023-10-19T12:45:27.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.974+0000 7f5a727fc640 1 -- 172.21.15.57:0/2034211554 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4906 (secure 0 0 0) 0x7f5a7807a9c0 con 0x7f5a7c06bf20 2023-10-19T12:45:27.978 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:45:27.978 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4357,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/4228712742","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":4228712742},{"type":"v1","addr":"172.21.15.57:6835","nonce":4228712742}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4360,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/971892630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":971892630},{"type":"v1","addr":"172.21.15.57:6834","nonce":971892630}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4372,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/104495983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":104495983},{"type":"v1","addr":"172.21.15.179:6836","nonce":104495983}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T12:44:18.419669+0000","modified":"2023-10-19T12:44:40.545928+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4374},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4374":{"gid":4374,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.179:6837/622550365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":622550365},{"type":"v1","addr":"172.21.15.179:6837","nonce":622550365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-19T12:45:27.980 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 9 2023-10-19T12:45:27.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.976+0000 7f5a83c15640 1 -- 172.21.15.57:0/2034211554 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f5a6006ae40 msgr2=0x7f5a6006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:27.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.976+0000 7f5a83c15640 1 --2- 172.21.15.57:0/2034211554 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f5a6006ae40 0x7f5a6006d300 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f5a6c009960 tx=0x7f5a6c005c90 comp rx=0 tx=0).stop 2023-10-19T12:45:27.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.976+0000 7f5a83c15640 1 -- 172.21.15.57:0/2034211554 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a7c06bf20 msgr2=0x7f5a7c154160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:27.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.976+0000 7f5a83c15640 1 --2- 172.21.15.57:0/2034211554 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a7c06bf20 0x7f5a7c154160 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f5a78000f80 tx=0x7f5a78035ad0 comp rx=0 tx=0).stop 2023-10-19T12:45:27.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.976+0000 7f5a83c15640 1 -- 172.21.15.57:0/2034211554 shutdown_connections 2023-10-19T12:45:27.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.976+0000 7f5a83c15640 1 --2- 172.21.15.57:0/2034211554 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f5a6006ae40 0x7f5a6006d300 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:27.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.976+0000 7f5a83c15640 1 --2- 172.21.15.57:0/2034211554 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5a7c0cd0c0 0x7f5a7c1e28c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:27.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.976+0000 7f5a83c15640 1 --2- 172.21.15.57:0/2034211554 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5a7c06dce0 0x7f5a7c1f1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:27.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.976+0000 7f5a83c15640 1 --2- 172.21.15.57:0/2034211554 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a7c06bf20 0x7f5a7c154160 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:27.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.976+0000 7f5a83c15640 1 -- 172.21.15.57:0/2034211554 >> 172.21.15.57:0/2034211554 conn(0x7f5a7c0b0670 msgr2=0x7f5a7c159eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:27.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.976+0000 7f5a83c15640 1 -- 172.21.15.57:0/2034211554 shutdown_connections 2023-10-19T12:45:27.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:27.976+0000 7f5a83c15640 1 -- 172.21.15.57:0/2034211554 wait complete. 2023-10-19T12:45:27.990 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T12:45:28.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.096+0000 7fe87b064640 1 -- 172.21.15.57:0/2124666515 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe8740ad370 msgr2=0x7fe8740ad800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:28.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.096+0000 7fe87b064640 1 --2- 172.21.15.57:0/2124666515 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe8740ad370 0x7fe8740ad800 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fe86c005fd0 tx=0x7fe86c02c0e0 comp rx=0 tx=0).stop 2023-10-19T12:45:28.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.096+0000 7fe87b064640 1 -- 172.21.15.57:0/2124666515 shutdown_connections 2023-10-19T12:45:28.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.096+0000 7fe87b064640 1 --2- 172.21.15.57:0/2124666515 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe8740ad370 0x7fe8740ad800 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:28.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.096+0000 7fe87b064640 1 --2- 172.21.15.57:0/2124666515 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe87406d2d0 0x7fe87406d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:28.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.096+0000 7fe87b064640 1 -- 172.21.15.57:0/2124666515 >> 172.21.15.57:0/2124666515 conn(0x7fe8740b0670 msgr2=0x7fe8740b0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:28.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.096+0000 7fe87b064640 1 -- 172.21.15.57:0/2124666515 shutdown_connections 2023-10-19T12:45:28.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.096+0000 7fe87b064640 1 -- 172.21.15.57:0/2124666515 wait complete. 2023-10-19T12:45:28.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.097+0000 7fe87b064640 1 Processor -- start 2023-10-19T12:45:28.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.097+0000 7fe87b064640 1 -- start start 2023-10-19T12:45:28.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.098+0000 7fe87b064640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe87406d2d0 0x7fe874155fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:28.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.098+0000 7fe87b064640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe87406f0a0 0x7fe8740c5c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:28.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.098+0000 7fe87b064640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe8740ad370 0x7fe8741eed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:28.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.098+0000 7fe87b064640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8741566b0 con 0x7fe87406f0a0 2023-10-19T12:45:28.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.098+0000 7fe87b064640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe874156990 con 0x7fe87406d2d0 2023-10-19T12:45:28.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.098+0000 7fe87b064640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe874156c70 con 0x7fe8740ad370 2023-10-19T12:45:28.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.098+0000 7fe873fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe87406f0a0 0x7fe8740c5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:28.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.098+0000 7fe873fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe87406f0a0 0x7fe8740c5c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33358/0 (socket says 172.21.15.57:33358) 2023-10-19T12:45:28.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.098+0000 7fe873fff640 1 -- 172.21.15.57:0/621682701 learned_addr learned my addr 172.21.15.57:0/621682701 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:28.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.098+0000 7fe879601640 1 --2- 172.21.15.57:0/621682701 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe8740ad370 0x7fe8741eed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:28.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.098+0000 7fe878e00640 1 --2- 172.21.15.57:0/621682701 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe87406d2d0 0x7fe874155fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:28.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.098+0000 7fe873fff640 1 -- 172.21.15.57:0/621682701 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe8740ad370 msgr2=0x7fe8741eed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:28.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.098+0000 7fe873fff640 1 --2- 172.21.15.57:0/621682701 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe8740ad370 0x7fe8741eed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:28.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.098+0000 7fe873fff640 1 -- 172.21.15.57:0/621682701 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe87406d2d0 msgr2=0x7fe874155fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:28.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.098+0000 7fe873fff640 1 --2- 172.21.15.57:0/621682701 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe87406d2d0 0x7fe874155fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:28.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.098+0000 7fe873fff640 1 -- 172.21.15.57:0/621682701 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe86c009840 con 0x7fe87406f0a0 2023-10-19T12:45:28.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.098+0000 7fe878e00640 1 --2- 172.21.15.57:0/621682701 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe87406d2d0 0x7fe874155fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:45:28.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.098+0000 7fe873fff640 1 --2- 172.21.15.57:0/621682701 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe87406f0a0 0x7fe8740c5c20 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fe86800c430 tx=0x7fe86800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:28.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.099+0000 7fe8717fa640 1 -- 172.21.15.57:0/621682701 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe868004230 con 0x7fe87406f0a0 2023-10-19T12:45:28.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.099+0000 7fe8717fa640 1 -- 172.21.15.57:0/621682701 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fe8680044a0 con 0x7fe87406f0a0 2023-10-19T12:45:28.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.099+0000 7fe87b064640 1 -- 172.21.15.57:0/621682701 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe874156f50 con 0x7fe87406f0a0 2023-10-19T12:45:28.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.099+0000 7fe8717fa640 1 -- 172.21.15.57:0/621682701 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe868003850 con 0x7fe87406f0a0 2023-10-19T12:45:28.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.099+0000 7fe87b064640 1 -- 172.21.15.57:0/621682701 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8741ef240 con 0x7fe87406f0a0 2023-10-19T12:45:28.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.100+0000 7fe87b064640 1 -- 172.21.15.57:0/621682701 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8740ce1d0 con 0x7fe87406f0a0 2023-10-19T12:45:28.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.100+0000 7fe8717fa640 1 -- 172.21.15.57:0/621682701 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fe868003a30 con 0x7fe87406f0a0 2023-10-19T12:45:28.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.101+0000 7fe8717fa640 1 --2- 172.21.15.57:0/621682701 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fe86006ae40 0x7fe86006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:28.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.101+0000 7fe8717fa640 1 -- 172.21.15.57:0/621682701 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fe8680891d0 con 0x7fe87406f0a0 2023-10-19T12:45:28.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.101+0000 7fe878e00640 1 --2- 172.21.15.57:0/621682701 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fe86006ae40 0x7fe86006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:28.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.101+0000 7fe878e00640 1 --2- 172.21.15.57:0/621682701 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fe86006ae40 0x7fe86006d300 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fe8640058c0 tx=0x7fe864005850 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:28.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.105+0000 7fe8717fa640 1 -- 172.21.15.57:0/621682701 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe868057ac0 con 0x7fe87406f0a0 2023-10-19T12:45:28.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.284+0000 7fe87b064640 1 -- 172.21.15.57:0/621682701 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe8740690e0 con 0x7fe87406f0a0 2023-10-19T12:45:28.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.285+0000 7fe8717fa640 1 -- 172.21.15.57:0/621682701 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18748 (secure 0 0 0) 0x7fe868057650 con 0x7fe87406f0a0 2023-10-19T12:45:28.289 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:45:28.289 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":27,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T12:44:58.978403+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-19T12:44:16.517510+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-19T12:44:17.398297+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-19T12:44:21.129403+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.23","mappings":[{"from":3,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"3.39","mappings":[{"from":7,"to":1}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.1d","mappings":[{"from":1,"to":5}]},{"pgid":"4.1e","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T12:45:28.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.287+0000 7fe87b064640 1 -- 172.21.15.57:0/621682701 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fe86006ae40 msgr2=0x7fe86006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:28.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.287+0000 7fe87b064640 1 --2- 172.21.15.57:0/621682701 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fe86006ae40 0x7fe86006d300 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fe8640058c0 tx=0x7fe864005850 comp rx=0 tx=0).stop 2023-10-19T12:45:28.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.287+0000 7fe87b064640 1 -- 172.21.15.57:0/621682701 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe87406f0a0 msgr2=0x7fe8740c5c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:28.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.287+0000 7fe87b064640 1 --2- 172.21.15.57:0/621682701 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe87406f0a0 0x7fe8740c5c20 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fe86800c430 tx=0x7fe86800c8f0 comp rx=0 tx=0).stop 2023-10-19T12:45:28.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.287+0000 7fe87b064640 1 -- 172.21.15.57:0/621682701 shutdown_connections 2023-10-19T12:45:28.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.287+0000 7fe87b064640 1 --2- 172.21.15.57:0/621682701 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fe86006ae40 0x7fe86006d300 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:28.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.287+0000 7fe87b064640 1 --2- 172.21.15.57:0/621682701 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe8740ad370 0x7fe8741eed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:28.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.287+0000 7fe87b064640 1 --2- 172.21.15.57:0/621682701 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe87406f0a0 0x7fe8740c5c20 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:28.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.288+0000 7fe87b064640 1 --2- 172.21.15.57:0/621682701 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe87406d2d0 0x7fe874155fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:28.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.288+0000 7fe87b064640 1 -- 172.21.15.57:0/621682701 >> 172.21.15.57:0/621682701 conn(0x7fe8740b0670 msgr2=0x7fe8740b0a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:28.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.288+0000 7fe87b064640 1 -- 172.21.15.57:0/621682701 shutdown_connections 2023-10-19T12:45:28.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.288+0000 7fe87b064640 1 -- 172.21.15.57:0/621682701 wait complete. 2023-10-19T12:45:28.302 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-19T12:45:28.302 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T12:45:28.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.407+0000 7ff6a2b8e640 1 -- 172.21.15.57:0/3313963091 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff69c15b2a0 msgr2=0x7ff69c0d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:28.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.407+0000 7ff6a2b8e640 1 --2- 172.21.15.57:0/3313963091 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff69c15b2a0 0x7ff69c0d1910 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7ff69c15b680 tx=0x7ff68802bdb0 comp rx=0 tx=0).stop 2023-10-19T12:45:28.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.407+0000 7ff6a2b8e640 1 -- 172.21.15.57:0/3313963091 shutdown_connections 2023-10-19T12:45:28.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.407+0000 7ff6a2b8e640 1 --2- 172.21.15.57:0/3313963091 >> v2:172.21.15.179:3300/0 conn(0x7ff69c0ceb40 0x7ff69c0b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:28.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.407+0000 7ff6a2b8e640 1 --2- 172.21.15.57:0/3313963091 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff69c15b2a0 0x7ff69c0d1910 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:28.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.407+0000 7ff6a2b8e640 1 -- 172.21.15.57:0/3313963091 >> 172.21.15.57:0/3313963091 conn(0x7ff69c0d0d60 msgr2=0x7ff69c0d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:28.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.407+0000 7ff6a2b8e640 1 -- 172.21.15.57:0/3313963091 shutdown_connections 2023-10-19T12:45:28.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.407+0000 7ff6a2b8e640 1 -- 172.21.15.57:0/3313963091 wait complete. 2023-10-19T12:45:28.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.408+0000 7ff6a2b8e640 1 Processor -- start 2023-10-19T12:45:28.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.409+0000 7ff6a2b8e640 1 -- start start 2023-10-19T12:45:28.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.409+0000 7ff6a2b8e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff69c0ceb40 0x7ff69c0ae840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:28.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.409+0000 7ff6a2b8e640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff69c0cf340 0x7ff69c1f3e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:28.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.409+0000 7ff6a2b8e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff69c15b2a0 0x7ff69c1f43a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:28.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.409+0000 7ff6a2b8e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff69c0c8b10 con 0x7ff69c0ceb40 2023-10-19T12:45:28.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.409+0000 7ff6a2b8e640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff69c0c7ed0 con 0x7ff69c15b2a0 2023-10-19T12:45:28.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.409+0000 7ff6a2b8e640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff69c0c8170 con 0x7ff69c0cf340 2023-10-19T12:45:28.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.409+0000 7ff6a092a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff69c0ceb40 0x7ff69c0ae840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:28.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.410+0000 7ff6a092a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff69c0ceb40 0x7ff69c0ae840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33370/0 (socket says 172.21.15.57:33370) 2023-10-19T12:45:28.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.410+0000 7ff6a092a640 1 -- 172.21.15.57:0/3926171270 learned_addr learned my addr 172.21.15.57:0/3926171270 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:28.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.410+0000 7ff6a092a640 1 -- 172.21.15.57:0/3926171270 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff69c0cf340 msgr2=0x7ff69c1f3e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:28.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.410+0000 7ff6a092a640 1 --2- 172.21.15.57:0/3926171270 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff69c0cf340 0x7ff69c1f3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:28.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.410+0000 7ff6a092a640 1 -- 172.21.15.57:0/3926171270 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff69c15b2a0 msgr2=0x7ff69c1f43a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:28.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.410+0000 7ff6a092a640 1 --2- 172.21.15.57:0/3926171270 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff69c15b2a0 0x7ff69c1f43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:28.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.410+0000 7ff6a092a640 1 -- 172.21.15.57:0/3926171270 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff688009560 con 0x7ff69c0ceb40 2023-10-19T12:45:28.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.410+0000 7ff6a092a640 1 --2- 172.21.15.57:0/3926171270 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff69c0ceb40 0x7ff69c0ae840 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7ff69000c4b0 tx=0x7ff69000c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:28.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.410+0000 7ff6997fa640 1 -- 172.21.15.57:0/3926171270 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff690005480 con 0x7ff69c0ceb40 2023-10-19T12:45:28.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.410+0000 7ff6997fa640 1 -- 172.21.15.57:0/3926171270 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7ff6900036e0 con 0x7ff69c0ceb40 2023-10-19T12:45:28.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.410+0000 7ff6997fa640 1 -- 172.21.15.57:0/3926171270 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff690003b30 con 0x7ff69c0ceb40 2023-10-19T12:45:28.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.410+0000 7ff6a2b8e640 1 -- 172.21.15.57:0/3926171270 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff69c0c8450 con 0x7ff69c0ceb40 2023-10-19T12:45:28.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.410+0000 7ff6a2b8e640 1 -- 172.21.15.57:0/3926171270 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff69c0aed80 con 0x7ff69c0ceb40 2023-10-19T12:45:28.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.411+0000 7ff6a2b8e640 1 -- 172.21.15.57:0/3926171270 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff69c0aef70 con 0x7ff69c0ceb40 2023-10-19T12:45:28.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.412+0000 7ff6997fa640 1 -- 172.21.15.57:0/3926171270 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7ff690003c90 con 0x7ff69c0ceb40 2023-10-19T12:45:28.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.412+0000 7ff6997fa640 1 --2- 172.21.15.57:0/3926171270 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7ff68006ae70 0x7ff68006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:28.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.412+0000 7ff6997fa640 1 -- 172.21.15.57:0/3926171270 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7ff690089120 con 0x7ff69c0ceb40 2023-10-19T12:45:28.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.412+0000 7ff69bfff640 1 --2- 172.21.15.57:0/3926171270 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7ff68006ae70 0x7ff68006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:28.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.413+0000 7ff69bfff640 1 --2- 172.21.15.57:0/3926171270 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7ff68006ae70 0x7ff68006d330 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7ff688035000 tx=0x7ff6880046c0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:28.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.416+0000 7ff6997fa640 1 -- 172.21.15.57:0/3926171270 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff690053ea0 con 0x7ff69c0ceb40 2023-10-19T12:45:28.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.628+0000 7ff6a2b8e640 1 -- 172.21.15.57:0/3926171270 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff69c0c8c50 con 0x7ff69c0ceb40 2023-10-19T12:45:28.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.628+0000 7ff6997fa640 1 -- 172.21.15.57:0/3926171270 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4906 (secure 0 0 0) 0x7ff69c0c8c50 con 0x7ff69c0ceb40 2023-10-19T12:45:28.632 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:45:28.632 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4357,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/4228712742","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":4228712742},{"type":"v1","addr":"172.21.15.57:6835","nonce":4228712742}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4360,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/971892630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":971892630},{"type":"v1","addr":"172.21.15.57:6834","nonce":971892630}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4372,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/104495983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":104495983},{"type":"v1","addr":"172.21.15.179:6836","nonce":104495983}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T12:44:18.419669+0000","modified":"2023-10-19T12:44:40.545928+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4374},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4374":{"gid":4374,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.179:6837/622550365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":622550365},{"type":"v1","addr":"172.21.15.179:6837","nonce":622550365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-19T12:45:28.633 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 9 2023-10-19T12:45:28.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.631+0000 7ff6a2b8e640 1 -- 172.21.15.57:0/3926171270 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7ff68006ae70 msgr2=0x7ff68006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:28.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.631+0000 7ff6a2b8e640 1 --2- 172.21.15.57:0/3926171270 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7ff68006ae70 0x7ff68006d330 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7ff688035000 tx=0x7ff6880046c0 comp rx=0 tx=0).stop 2023-10-19T12:45:28.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.631+0000 7ff6a2b8e640 1 -- 172.21.15.57:0/3926171270 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff69c0ceb40 msgr2=0x7ff69c0ae840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:28.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.631+0000 7ff6a2b8e640 1 --2- 172.21.15.57:0/3926171270 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff69c0ceb40 0x7ff69c0ae840 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7ff69000c4b0 tx=0x7ff69000c970 comp rx=0 tx=0).stop 2023-10-19T12:45:28.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.631+0000 7ff6a2b8e640 1 -- 172.21.15.57:0/3926171270 shutdown_connections 2023-10-19T12:45:28.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.631+0000 7ff6a2b8e640 1 --2- 172.21.15.57:0/3926171270 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7ff68006ae70 0x7ff68006d330 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:28.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.631+0000 7ff6a2b8e640 1 --2- 172.21.15.57:0/3926171270 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff69c15b2a0 0x7ff69c1f43a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:28.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.631+0000 7ff6a2b8e640 1 --2- 172.21.15.57:0/3926171270 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff69c0cf340 0x7ff69c1f3e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:28.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.631+0000 7ff6a2b8e640 1 --2- 172.21.15.57:0/3926171270 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff69c0ceb40 0x7ff69c0ae840 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:28.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.631+0000 7ff6a2b8e640 1 -- 172.21.15.57:0/3926171270 >> 172.21.15.57:0/3926171270 conn(0x7ff69c0d0d60 msgr2=0x7ff69c0ad940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:28.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.631+0000 7ff6a2b8e640 1 -- 172.21.15.57:0/3926171270 shutdown_connections 2023-10-19T12:45:28.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.631+0000 7ff6a2b8e640 1 -- 172.21.15.57:0/3926171270 wait complete. 2023-10-19T12:45:28.644 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T12:45:28.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.750+0000 7fae4798a640 1 -- 172.21.15.57:0/2018991496 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fae400b12b0 msgr2=0x7fae4006a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:28.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.750+0000 7fae4798a640 1 --2- 172.21.15.57:0/2018991496 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fae400b12b0 0x7fae4006a730 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fae400712e0 tx=0x7fae3002be40 comp rx=0 tx=0).stop 2023-10-19T12:45:28.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.750+0000 7fae4798a640 1 -- 172.21.15.57:0/2018991496 shutdown_connections 2023-10-19T12:45:28.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.750+0000 7fae4798a640 1 --2- 172.21.15.57:0/2018991496 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fae400b12b0 0x7fae4006a730 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:28.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.750+0000 7fae4798a640 1 --2- 172.21.15.57:0/2018991496 >> v2:172.21.15.179:3300/0 conn(0x7fae40155580 0x7fae400b0d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:28.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.750+0000 7fae4798a640 1 -- 172.21.15.57:0/2018991496 >> 172.21.15.57:0/2018991496 conn(0x7fae4006fae0 msgr2=0x7fae4006fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:28.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.750+0000 7fae4798a640 1 -- 172.21.15.57:0/2018991496 shutdown_connections 2023-10-19T12:45:28.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.750+0000 7fae4798a640 1 -- 172.21.15.57:0/2018991496 wait complete. 2023-10-19T12:45:28.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.751+0000 7fae4798a640 1 Processor -- start 2023-10-19T12:45:28.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.751+0000 7fae4798a640 1 -- start start 2023-10-19T12:45:28.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.752+0000 7fae4798a640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fae4006ac70 0x7fae4015d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:28.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.752+0000 7fae4798a640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fae400b12b0 0x7fae4015dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:28.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.752+0000 7fae4798a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fae40155580 0x7fae40152b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:28.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.752+0000 7fae4798a640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae40148c90 con 0x7fae40155580 2023-10-19T12:45:28.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.752+0000 7fae4798a640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae401483e0 con 0x7fae4006ac70 2023-10-19T12:45:28.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.752+0000 7fae4798a640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae4014e720 con 0x7fae400b12b0 2023-10-19T12:45:28.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.752+0000 7fae45f27640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fae40155580 0x7fae40152b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:28.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.752+0000 7fae45f27640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fae40155580 0x7fae40152b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33382/0 (socket says 172.21.15.57:33382) 2023-10-19T12:45:28.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.752+0000 7fae45f27640 1 -- 172.21.15.57:0/2862384173 learned_addr learned my addr 172.21.15.57:0/2862384173 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:28.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.752+0000 7fae44f25640 1 --2- 172.21.15.57:0/2862384173 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fae400b12b0 0x7fae4015dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:28.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.752+0000 7fae45726640 1 --2- 172.21.15.57:0/2862384173 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fae4006ac70 0x7fae4015d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:28.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.752+0000 7fae45f27640 1 -- 172.21.15.57:0/2862384173 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fae400b12b0 msgr2=0x7fae4015dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:28.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.752+0000 7fae45f27640 1 --2- 172.21.15.57:0/2862384173 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fae400b12b0 0x7fae4015dc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:28.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.752+0000 7fae45f27640 1 -- 172.21.15.57:0/2862384173 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fae4006ac70 msgr2=0x7fae4015d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:28.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.752+0000 7fae45f27640 1 --2- 172.21.15.57:0/2862384173 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fae4006ac70 0x7fae4015d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:28.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.752+0000 7fae45f27640 1 -- 172.21.15.57:0/2862384173 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae30009560 con 0x7fae40155580 2023-10-19T12:45:28.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.752+0000 7fae45f27640 1 --2- 172.21.15.57:0/2862384173 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fae40155580 0x7fae40152b70 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fae3c00c760 tx=0x7fae3c00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:28.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.753+0000 7fae35ffb640 1 -- 172.21.15.57:0/2862384173 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae3c003da0 con 0x7fae40155580 2023-10-19T12:45:28.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.753+0000 7fae35ffb640 1 -- 172.21.15.57:0/2862384173 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fae3c00ce70 con 0x7fae40155580 2023-10-19T12:45:28.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.753+0000 7fae4798a640 1 -- 172.21.15.57:0/2862384173 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae4014ea00 con 0x7fae40155580 2023-10-19T12:45:28.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.753+0000 7fae35ffb640 1 -- 172.21.15.57:0/2862384173 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae3c0052b0 con 0x7fae40155580 2023-10-19T12:45:28.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.753+0000 7fae4798a640 1 -- 172.21.15.57:0/2862384173 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae4014efc0 con 0x7fae40155580 2023-10-19T12:45:28.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.754+0000 7fae4798a640 1 -- 172.21.15.57:0/2862384173 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae40148dd0 con 0x7fae40155580 2023-10-19T12:45:28.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.754+0000 7fae35ffb640 1 -- 172.21.15.57:0/2862384173 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fae3c005410 con 0x7fae40155580 2023-10-19T12:45:28.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.755+0000 7fae35ffb640 1 --2- 172.21.15.57:0/2862384173 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fae2406ae40 0x7fae2406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:28.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.755+0000 7fae35ffb640 1 -- 172.21.15.57:0/2862384173 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fae3c0899f0 con 0x7fae40155580 2023-10-19T12:45:28.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.755+0000 7fae45726640 1 --2- 172.21.15.57:0/2862384173 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fae2406ae40 0x7fae2406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:28.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.755+0000 7fae45726640 1 --2- 172.21.15.57:0/2862384173 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fae2406ae40 0x7fae2406d300 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fae3800b810 tx=0x7fae38008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:28.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.759+0000 7fae35ffb640 1 -- 172.21.15.57:0/2862384173 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fae40148dd0 con 0x7fae40155580 2023-10-19T12:45:28.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.969+0000 7fae4798a640 1 -- 172.21.15.57:0/2862384173 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fae40148dd0 con 0x7fae40155580 2023-10-19T12:45:28.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.969+0000 7fae35ffb640 1 -- 172.21.15.57:0/2862384173 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4906 (secure 0 0 0) 0x7fae40148dd0 con 0x7fae40155580 2023-10-19T12:45:28.973 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:45:28.973 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4357,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/4228712742","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":4228712742},{"type":"v1","addr":"172.21.15.57:6835","nonce":4228712742}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4360,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/971892630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":971892630},{"type":"v1","addr":"172.21.15.57:6834","nonce":971892630}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4372,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/104495983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":104495983},{"type":"v1","addr":"172.21.15.179:6836","nonce":104495983}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T12:44:18.419669+0000","modified":"2023-10-19T12:44:40.545928+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4374},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4374":{"gid":4374,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.179:6837/622550365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":622550365},{"type":"v1","addr":"172.21.15.179:6837","nonce":622550365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-19T12:45:28.973 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 9 2023-10-19T12:45:28.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.971+0000 7fae4798a640 1 -- 172.21.15.57:0/2862384173 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fae2406ae40 msgr2=0x7fae2406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:28.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.971+0000 7fae4798a640 1 --2- 172.21.15.57:0/2862384173 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fae2406ae40 0x7fae2406d300 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fae3800b810 tx=0x7fae38008040 comp rx=0 tx=0).stop 2023-10-19T12:45:28.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.971+0000 7fae4798a640 1 -- 172.21.15.57:0/2862384173 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fae40155580 msgr2=0x7fae40152b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:28.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.971+0000 7fae4798a640 1 --2- 172.21.15.57:0/2862384173 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fae40155580 0x7fae40152b70 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fae3c00c760 tx=0x7fae3c00cc20 comp rx=0 tx=0).stop 2023-10-19T12:45:28.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.971+0000 7fae4798a640 1 -- 172.21.15.57:0/2862384173 shutdown_connections 2023-10-19T12:45:28.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.971+0000 7fae4798a640 1 --2- 172.21.15.57:0/2862384173 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fae2406ae40 0x7fae2406d300 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:28.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.971+0000 7fae4798a640 1 --2- 172.21.15.57:0/2862384173 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fae40155580 0x7fae40152b70 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:28.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.971+0000 7fae4798a640 1 --2- 172.21.15.57:0/2862384173 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fae400b12b0 0x7fae4015dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:28.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.971+0000 7fae4798a640 1 --2- 172.21.15.57:0/2862384173 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fae4006ac70 0x7fae4015d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:28.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.971+0000 7fae4798a640 1 -- 172.21.15.57:0/2862384173 >> 172.21.15.57:0/2862384173 conn(0x7fae4006fae0 msgr2=0x7fae4006fec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:28.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.972+0000 7fae4798a640 1 -- 172.21.15.57:0/2862384173 shutdown_connections 2023-10-19T12:45:28.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:28.972+0000 7fae4798a640 1 -- 172.21.15.57:0/2862384173 wait complete. 2023-10-19T12:45:28.985 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T12:45:29.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.091+0000 7f5275c61640 1 -- 172.21.15.57:0/33791962 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5258004d00 msgr2=0x7f5258005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:29.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.091+0000 7f5275c61640 1 --2- 172.21.15.57:0/33791962 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5258004d00 0x7f5258005180 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f526800c610 tx=0x7f52580057f0 comp rx=0 tx=0).stop 2023-10-19T12:45:29.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.092+0000 7f526ffff640 1 -- 172.21.15.57:0/33791962 reap_dead start 2023-10-19T12:45:29.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.092+0000 7f5275c61640 1 -- 172.21.15.57:0/33791962 shutdown_connections 2023-10-19T12:45:29.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.092+0000 7f5275c61640 1 -- 172.21.15.57:0/33791962 >> 172.21.15.57:0/33791962 conn(0x7f5270069930 msgr2=0x7f5270069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:29.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.092+0000 7f5275c61640 1 -- 172.21.15.57:0/33791962 shutdown_connections 2023-10-19T12:45:29.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.092+0000 7f5275c61640 1 -- 172.21.15.57:0/33791962 wait complete. 2023-10-19T12:45:29.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.093+0000 7f5275c61640 1 Processor -- start 2023-10-19T12:45:29.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.093+0000 7f5275c61640 1 -- start start 2023-10-19T12:45:29.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.094+0000 7f5275c61640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5258004d00 0x7f527013c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:29.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.094+0000 7f5275c61640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f527014ddc0 0x7f527014e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:29.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.094+0000 7f5275c61640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f527013cfc0 0x7f527013d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:29.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.094+0000 7f5275c61640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f527014e780 con 0x7f527013cfc0 2023-10-19T12:45:29.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.094+0000 7f5275c61640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f527013cbb0 con 0x7f527014ddc0 2023-10-19T12:45:29.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.094+0000 7f5275c61640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f527013dd60 con 0x7f5258004d00 2023-10-19T12:45:29.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.094+0000 7f526ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f527013cfc0 0x7f527013d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:29.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.094+0000 7f526ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f527013cfc0 0x7f527013d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33398/0 (socket says 172.21.15.57:33398) 2023-10-19T12:45:29.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.094+0000 7f526ffff640 1 -- 172.21.15.57:0/1671942941 learned_addr learned my addr 172.21.15.57:0/1671942941 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:29.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.094+0000 7f526effd640 1 --2- 172.21.15.57:0/1671942941 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f527014ddc0 0x7f527014e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:29.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.094+0000 7f526f7fe640 1 --2- 172.21.15.57:0/1671942941 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5258004d00 0x7f527013c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:29.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.094+0000 7f526ffff640 1 -- 172.21.15.57:0/1671942941 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5258004d00 msgr2=0x7f527013c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:29.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.094+0000 7f526ffff640 1 --2- 172.21.15.57:0/1671942941 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5258004d00 0x7f527013c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:29.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.094+0000 7f526ffff640 1 -- 172.21.15.57:0/1671942941 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f527014ddc0 msgr2=0x7f527014e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:29.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.094+0000 7f526ffff640 1 --2- 172.21.15.57:0/1671942941 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f527014ddc0 0x7f527014e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:29.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.094+0000 7f526ffff640 1 -- 172.21.15.57:0/1671942941 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f527013e040 con 0x7f527013cfc0 2023-10-19T12:45:29.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.094+0000 7f526ffff640 1 --2- 172.21.15.57:0/1671942941 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f527013cfc0 0x7f527013d440 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f526802cbe0 tx=0x7f52680017b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:29.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.094+0000 7f5257fff640 1 -- 172.21.15.57:0/1671942941 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52680019e0 con 0x7f527013cfc0 2023-10-19T12:45:29.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.094+0000 7f5257fff640 1 -- 172.21.15.57:0/1671942941 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f5268003970 con 0x7f527013cfc0 2023-10-19T12:45:29.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.094+0000 7f5275c61640 1 -- 172.21.15.57:0/1671942941 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f527013e320 con 0x7f527013cfc0 2023-10-19T12:45:29.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.094+0000 7f5257fff640 1 -- 172.21.15.57:0/1671942941 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5268003d40 con 0x7f527013cfc0 2023-10-19T12:45:29.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.095+0000 7f5275c61640 1 -- 172.21.15.57:0/1671942941 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f527013e8e0 con 0x7f527013cfc0 2023-10-19T12:45:29.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.095+0000 7f5275c61640 1 -- 172.21.15.57:0/1671942941 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52700690e0 con 0x7f527013cfc0 2023-10-19T12:45:29.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.096+0000 7f5257fff640 1 -- 172.21.15.57:0/1671942941 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f5268031030 con 0x7f527013cfc0 2023-10-19T12:45:29.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.096+0000 7f5257fff640 1 --2- 172.21.15.57:0/1671942941 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f525c06ae40 0x7f525c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:29.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.096+0000 7f5257fff640 1 -- 172.21.15.57:0/1671942941 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f52680aab20 con 0x7f527013cfc0 2023-10-19T12:45:29.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.097+0000 7f526f7fe640 1 --2- 172.21.15.57:0/1671942941 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f525c06ae40 0x7f525c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:29.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.097+0000 7f526f7fe640 1 --2- 172.21.15.57:0/1671942941 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f525c06ae40 0x7f525c06d300 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f5260003700 tx=0x7f526000b260 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:29.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.100+0000 7f5257fff640 1 -- 172.21.15.57:0/1671942941 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f52680758c0 con 0x7f527013cfc0 2023-10-19T12:45:29.280 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.277+0000 7f5275c61640 1 -- 172.21.15.57:0/1671942941 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f527006f700 con 0x7f527013cfc0 2023-10-19T12:45:29.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.279+0000 7f5257fff640 1 -- 172.21.15.57:0/1671942941 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18748 (secure 0 0 0) 0x7f5268043070 con 0x7f527013cfc0 2023-10-19T12:45:29.282 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:45:29.282 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":27,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T12:44:58.978403+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-19T12:44:16.517510+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-19T12:44:17.398297+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-19T12:44:21.129403+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.23","mappings":[{"from":3,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"3.39","mappings":[{"from":7,"to":1}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.1d","mappings":[{"from":1,"to":5}]},{"pgid":"4.1e","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T12:45:29.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.280+0000 7f5275c61640 1 -- 172.21.15.57:0/1671942941 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f525c06ae40 msgr2=0x7f525c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:29.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.280+0000 7f5275c61640 1 --2- 172.21.15.57:0/1671942941 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f525c06ae40 0x7f525c06d300 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f5260003700 tx=0x7f526000b260 comp rx=0 tx=0).stop 2023-10-19T12:45:29.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.280+0000 7f5275c61640 1 -- 172.21.15.57:0/1671942941 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f527013cfc0 msgr2=0x7f527013d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:29.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.280+0000 7f5275c61640 1 --2- 172.21.15.57:0/1671942941 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f527013cfc0 0x7f527013d440 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f526802cbe0 tx=0x7f52680017b0 comp rx=0 tx=0).stop 2023-10-19T12:45:29.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.280+0000 7f5275c61640 1 -- 172.21.15.57:0/1671942941 shutdown_connections 2023-10-19T12:45:29.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.280+0000 7f5275c61640 1 --2- 172.21.15.57:0/1671942941 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f525c06ae40 0x7f525c06d300 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:29.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.280+0000 7f5275c61640 1 --2- 172.21.15.57:0/1671942941 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f527013cfc0 0x7f527013d440 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:29.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.280+0000 7f5275c61640 1 --2- 172.21.15.57:0/1671942941 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f527014ddc0 0x7f527014e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:29.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.280+0000 7f5275c61640 1 --2- 172.21.15.57:0/1671942941 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5258004d00 0x7f527013c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:29.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.281+0000 7f5275c61640 1 -- 172.21.15.57:0/1671942941 >> 172.21.15.57:0/1671942941 conn(0x7f5270069930 msgr2=0x7f527014fbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:29.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.281+0000 7f5275c61640 1 -- 172.21.15.57:0/1671942941 shutdown_connections 2023-10-19T12:45:29.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.281+0000 7f5275c61640 1 -- 172.21.15.57:0/1671942941 wait complete. 2023-10-19T12:45:29.295 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-19T12:45:29.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.402+0000 7f3653f25640 1 -- 172.21.15.57:0/2310473771 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f36380047c0 msgr2=0x7f3638004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:29.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.402+0000 7f3653f25640 1 --2- 172.21.15.57:0/2310473771 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f36380047c0 0x7f3638004c40 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f3640001680 tx=0x7f36380057f0 comp rx=0 tx=0).stop 2023-10-19T12:45:29.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.402+0000 7f3653f25640 1 -- 172.21.15.57:0/2310473771 shutdown_connections 2023-10-19T12:45:29.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.402+0000 7f3653f25640 1 --2- 172.21.15.57:0/2310473771 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f36380047c0 0x7f3638004c40 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:29.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.402+0000 7f3653f25640 1 -- 172.21.15.57:0/2310473771 >> 172.21.15.57:0/2310473771 conn(0x7f364c06fae0 msgr2=0x7f364c06fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:29.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.403+0000 7f3653f25640 1 -- 172.21.15.57:0/2310473771 shutdown_connections 2023-10-19T12:45:29.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.403+0000 7f3653f25640 1 -- 172.21.15.57:0/2310473771 wait complete. 2023-10-19T12:45:29.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.404+0000 7f3653f25640 1 Processor -- start 2023-10-19T12:45:29.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.404+0000 7f3653f25640 1 -- start start 2023-10-19T12:45:29.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.404+0000 7f3653f25640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f36380047c0 0x7f364c13fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:29.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.404+0000 7f3653f25640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f364c13e730 0x7f364c13cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:29.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.404+0000 7f3653f25640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f364c06b980 0x7f364c06bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:29.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.404+0000 7f3653f25640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f364c06c320 con 0x7f364c13e730 2023-10-19T12:45:29.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.404+0000 7f3653f25640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f364c13d3e0 con 0x7f36380047c0 2023-10-19T12:45:29.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.404+0000 7f3653f25640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f364c13d690 con 0x7f364c06b980 2023-10-19T12:45:29.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.404+0000 7f36514c0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f364c13e730 0x7f364c13cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:29.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.405+0000 7f36514c0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f364c13e730 0x7f364c13cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33420/0 (socket says 172.21.15.57:33420) 2023-10-19T12:45:29.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.405+0000 7f36514c0640 1 -- 172.21.15.57:0/2919667753 learned_addr learned my addr 172.21.15.57:0/2919667753 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:29.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.405+0000 7f3651cc1640 1 --2- 172.21.15.57:0/2919667753 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f36380047c0 0x7f364c13fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:29.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.405+0000 7f36524c2640 1 --2- 172.21.15.57:0/2919667753 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f364c06b980 0x7f364c06bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:29.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.405+0000 7f36514c0640 1 -- 172.21.15.57:0/2919667753 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f364c06b980 msgr2=0x7f364c06bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:29.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.405+0000 7f36514c0640 1 --2- 172.21.15.57:0/2919667753 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f364c06b980 0x7f364c06bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:29.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.405+0000 7f36514c0640 1 -- 172.21.15.57:0/2919667753 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f36380047c0 msgr2=0x7f364c13fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:29.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.405+0000 7f36514c0640 1 --2- 172.21.15.57:0/2919667753 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f36380047c0 0x7f364c13fb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:29.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.405+0000 7f36514c0640 1 -- 172.21.15.57:0/2919667753 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f364c0b2330 con 0x7f364c13e730 2023-10-19T12:45:29.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.405+0000 7f3651cc1640 1 --2- 172.21.15.57:0/2919667753 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f36380047c0 0x7f364c13fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:45:29.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.405+0000 7f36514c0640 1 --2- 172.21.15.57:0/2919667753 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f364c13e730 0x7f364c13cea0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f364002c0d0 tx=0x7f3640006360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:29.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.405+0000 7f363e7fc640 1 -- 172.21.15.57:0/2919667753 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3640006540 con 0x7f364c13e730 2023-10-19T12:45:29.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.405+0000 7f363e7fc640 1 -- 172.21.15.57:0/2919667753 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f3640003c70 con 0x7f364c13e730 2023-10-19T12:45:29.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.405+0000 7f3653f25640 1 -- 172.21.15.57:0/2919667753 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f364c13d970 con 0x7f364c13e730 2023-10-19T12:45:29.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.405+0000 7f363e7fc640 1 -- 172.21.15.57:0/2919667753 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f364002c790 con 0x7f364c13e730 2023-10-19T12:45:29.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.405+0000 7f3653f25640 1 -- 172.21.15.57:0/2919667753 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f364c13df30 con 0x7f364c13e730 2023-10-19T12:45:29.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.406+0000 7f3653f25640 1 -- 172.21.15.57:0/2919667753 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f364c0acaf0 con 0x7f364c13e730 2023-10-19T12:45:29.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.407+0000 7f363e7fc640 1 -- 172.21.15.57:0/2919667753 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f364002c8f0 con 0x7f364c13e730 2023-10-19T12:45:29.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.407+0000 7f363e7fc640 1 --2- 172.21.15.57:0/2919667753 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f363006ae40 0x7f363006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:29.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.407+0000 7f363e7fc640 1 -- 172.21.15.57:0/2919667753 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f36400aac50 con 0x7f364c13e730 2023-10-19T12:45:29.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.407+0000 7f3651cc1640 1 --2- 172.21.15.57:0/2919667753 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f363006ae40 0x7f363006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:29.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.408+0000 7f3651cc1640 1 --2- 172.21.15.57:0/2919667753 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f363006ae40 0x7f363006d300 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f3648000fb0 tx=0x7f3648001e70 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:29.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.411+0000 7f363e7fc640 1 -- 172.21.15.57:0/2919667753 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f36400b0020 con 0x7f364c13e730 2023-10-19T12:45:29.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.627+0000 7f3653f25640 1 -- 172.21.15.57:0/2919667753 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f364c0665b0 con 0x7f364c13e730 2023-10-19T12:45:29.631 INFO:tasks.ceph.mon.a.smithi057.stderr:2023-10-19T12:45:29.628+0000 7f1253ff3640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-19T12:45:29.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.629+0000 7f363e7fc640 1 -- 172.21.15.57:0/2919667753 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v10) v1 ==== 161+0+0 (secure 0 0 0) 0x7f3640044030 con 0x7f364c13e730 2023-10-19T12:45:29.632 INFO:teuthology.orchestra.run.smithi057.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-19T12:45:29.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.631+0000 7f3653f25640 1 -- 172.21.15.57:0/2919667753 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f363006ae40 msgr2=0x7f363006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:29.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.631+0000 7f3653f25640 1 --2- 172.21.15.57:0/2919667753 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f363006ae40 0x7f363006d300 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f3648000fb0 tx=0x7f3648001e70 comp rx=0 tx=0).stop 2023-10-19T12:45:29.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.631+0000 7f3653f25640 1 -- 172.21.15.57:0/2919667753 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f364c13e730 msgr2=0x7f364c13cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:29.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.631+0000 7f3653f25640 1 --2- 172.21.15.57:0/2919667753 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f364c13e730 0x7f364c13cea0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f364002c0d0 tx=0x7f3640006360 comp rx=0 tx=0).stop 2023-10-19T12:45:29.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.632+0000 7f3653f25640 1 -- 172.21.15.57:0/2919667753 shutdown_connections 2023-10-19T12:45:29.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.632+0000 7f3653f25640 1 --2- 172.21.15.57:0/2919667753 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f363006ae40 0x7f363006d300 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:29.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.632+0000 7f3653f25640 1 --2- 172.21.15.57:0/2919667753 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f364c06b980 0x7f364c06bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:29.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.632+0000 7f3653f25640 1 --2- 172.21.15.57:0/2919667753 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f364c13e730 0x7f364c13cea0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:29.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.632+0000 7f3653f25640 1 --2- 172.21.15.57:0/2919667753 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f36380047c0 0x7f364c13fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:29.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.632+0000 7f3653f25640 1 -- 172.21.15.57:0/2919667753 >> 172.21.15.57:0/2919667753 conn(0x7f364c06fae0 msgr2=0x7f364c0adee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:29.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.632+0000 7f3653f25640 1 -- 172.21.15.57:0/2919667753 shutdown_connections 2023-10-19T12:45:29.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.632+0000 7f3653f25640 1 -- 172.21.15.57:0/2919667753 wait complete. 2023-10-19T12:45:29.638 INFO:tasks.ceph.mds.d.smithi179.stderr: -4144> 2023-10-19T12:44:23.029+0000 7f7aa057d640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-19T12:45:29.647 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-19T12:45:29.670 INFO:tasks.ceph.mds.d.smithi179.stdout:starting mds.d at 2023-10-19T12:45:29.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.755+0000 7f1d4d51d640 1 -- 172.21.15.57:0/2086438413 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1d34003b30 msgr2=0x7f1d34003f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:29.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.755+0000 7f1d4d51d640 1 --2- 172.21.15.57:0/2086438413 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1d34003b30 0x7f1d34003f30 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f1d38006f80 tx=0x7f1d340057f0 comp rx=0 tx=0).stop 2023-10-19T12:45:29.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.755+0000 7f1d477fe640 1 -- 172.21.15.57:0/2086438413 reap_dead start 2023-10-19T12:45:29.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.755+0000 7f1d4d51d640 1 -- 172.21.15.57:0/2086438413 shutdown_connections 2023-10-19T12:45:29.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.755+0000 7f1d4d51d640 1 -- 172.21.15.57:0/2086438413 >> 172.21.15.57:0/2086438413 conn(0x7f1d48069930 msgr2=0x7f1d48069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:29.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.755+0000 7f1d4d51d640 1 -- 172.21.15.57:0/2086438413 shutdown_connections 2023-10-19T12:45:29.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.756+0000 7f1d4d51d640 1 -- 172.21.15.57:0/2086438413 wait complete. 2023-10-19T12:45:29.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.757+0000 7f1d4d51d640 1 Processor -- start 2023-10-19T12:45:29.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.757+0000 7f1d4d51d640 1 -- start start 2023-10-19T12:45:29.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.757+0000 7f1d4d51d640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1d34003b30 0x7f1d4813fe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:29.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.757+0000 7f1d4d51d640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1d4814aed0 0x7f1d4814bd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:29.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.757+0000 7f1d4d51d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1d4814b350 0x7f1d4814c240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:29.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.758+0000 7f1d4d51d640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d4814aab0 con 0x7f1d4814b350 2023-10-19T12:45:29.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.758+0000 7f1d4d51d640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d48140380 con 0x7f1d4814aed0 2023-10-19T12:45:29.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.758+0000 7f1d4d51d640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d48140840 con 0x7f1d34003b30 2023-10-19T12:45:29.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.758+0000 7f1d477fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1d4814b350 0x7f1d4814c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:29.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.758+0000 7f1d477fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1d4814b350 0x7f1d4814c240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33430/0 (socket says 172.21.15.57:33430) 2023-10-19T12:45:29.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.758+0000 7f1d477fe640 1 -- 172.21.15.57:0/2207919931 learned_addr learned my addr 172.21.15.57:0/2207919931 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:29.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.758+0000 7f1d46ffd640 1 --2- 172.21.15.57:0/2207919931 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1d34003b30 0x7f1d4813fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:29.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.758+0000 7f1d3ffff640 1 --2- 172.21.15.57:0/2207919931 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1d4814aed0 0x7f1d4814bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:29.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.758+0000 7f1d477fe640 1 -- 172.21.15.57:0/2207919931 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1d34003b30 msgr2=0x7f1d4813fe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:29.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.758+0000 7f1d477fe640 1 --2- 172.21.15.57:0/2207919931 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1d34003b30 0x7f1d4813fe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:29.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.758+0000 7f1d477fe640 1 -- 172.21.15.57:0/2207919931 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1d4814aed0 msgr2=0x7f1d4814bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:29.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.758+0000 7f1d477fe640 1 --2- 172.21.15.57:0/2207919931 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1d4814aed0 0x7f1d4814bd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:29.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.758+0000 7f1d477fe640 1 -- 172.21.15.57:0/2207919931 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d4814a150 con 0x7f1d4814b350 2023-10-19T12:45:29.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.758+0000 7f1d477fe640 1 --2- 172.21.15.57:0/2207919931 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1d4814b350 0x7f1d4814c240 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f1d4806f410 tx=0x7f1d400064f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:29.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.758+0000 7f1d3f7fe640 1 -- 172.21.15.57:0/2207919931 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d40017030 con 0x7f1d4814b350 2023-10-19T12:45:29.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.758+0000 7f1d3f7fe640 1 -- 172.21.15.57:0/2207919931 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f1d4000f940 con 0x7f1d4814b350 2023-10-19T12:45:29.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.758+0000 7f1d4d51d640 1 -- 172.21.15.57:0/2207919931 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d4814a3f0 con 0x7f1d4814b350 2023-10-19T12:45:29.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.758+0000 7f1d3f7fe640 1 -- 172.21.15.57:0/2207919931 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d4000fd50 con 0x7f1d4814b350 2023-10-19T12:45:29.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.758+0000 7f1d4d51d640 1 -- 172.21.15.57:0/2207919931 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d4814c870 con 0x7f1d4814b350 2023-10-19T12:45:29.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.759+0000 7f1d4d51d640 1 -- 172.21.15.57:0/2207919931 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d4814abf0 con 0x7f1d4814b350 2023-10-19T12:45:29.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.760+0000 7f1d3f7fe640 1 -- 172.21.15.57:0/2207919931 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f1d4001a070 con 0x7f1d4814b350 2023-10-19T12:45:29.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.760+0000 7f1d3f7fe640 1 --2- 172.21.15.57:0/2207919931 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f1d2806ae40 0x7f1d2806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:29.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.760+0000 7f1d3f7fe640 1 -- 172.21.15.57:0/2207919931 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6749+0+0 (secure 0 0 0) 0x7f1d40018070 con 0x7f1d4814b350 2023-10-19T12:45:29.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.760+0000 7f1d46ffd640 1 --2- 172.21.15.57:0/2207919931 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f1d2806ae40 0x7f1d2806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:29.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.761+0000 7f1d46ffd640 1 --2- 172.21.15.57:0/2207919931 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f1d2806ae40 0x7f1d2806d300 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f1d38034000 tx=0x7f1d3802e040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:29.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.764+0000 7f1d3f7fe640 1 -- 172.21.15.57:0/2207919931 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f1d4814abf0 con 0x7f1d4814b350 2023-10-19T12:45:29.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:29.981+0000 7f1d4d51d640 1 -- 172.21.15.57:0/2207919931 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f1d4814abf0 con 0x7f1d4814b350 2023-10-19T12:45:30.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:30.635+0000 7f1d3f7fe640 1 -- 172.21.15.57:0/2207919931 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v11) v1 ==== 104+0+0 (secure 0 0 0) 0x7f1d40054c50 con 0x7f1d4814b350 2023-10-19T12:45:30.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:30.638+0000 7f1d4d51d640 1 -- 172.21.15.57:0/2207919931 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f1d2806ae40 msgr2=0x7f1d2806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:30.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:30.638+0000 7f1d4d51d640 1 --2- 172.21.15.57:0/2207919931 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f1d2806ae40 0x7f1d2806d300 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f1d38034000 tx=0x7f1d3802e040 comp rx=0 tx=0).stop 2023-10-19T12:45:30.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:30.638+0000 7f1d4d51d640 1 -- 172.21.15.57:0/2207919931 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1d4814b350 msgr2=0x7f1d4814c240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:30.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:30.638+0000 7f1d4d51d640 1 --2- 172.21.15.57:0/2207919931 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1d4814b350 0x7f1d4814c240 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f1d4806f410 tx=0x7f1d400064f0 comp rx=0 tx=0).stop 2023-10-19T12:45:30.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:30.638+0000 7f1d4d51d640 1 -- 172.21.15.57:0/2207919931 shutdown_connections 2023-10-19T12:45:30.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:30.638+0000 7f1d4d51d640 1 --2- 172.21.15.57:0/2207919931 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f1d2806ae40 0x7f1d2806d300 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:30.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:30.638+0000 7f1d4d51d640 1 --2- 172.21.15.57:0/2207919931 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1d4814b350 0x7f1d4814c240 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:30.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:30.638+0000 7f1d4d51d640 1 --2- 172.21.15.57:0/2207919931 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1d4814aed0 0x7f1d4814bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:30.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:30.638+0000 7f1d4d51d640 1 --2- 172.21.15.57:0/2207919931 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1d34003b30 0x7f1d4813fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:30.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:30.638+0000 7f1d4d51d640 1 -- 172.21.15.57:0/2207919931 >> 172.21.15.57:0/2207919931 conn(0x7f1d48069930 msgr2=0x7f1d48070510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:30.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:30.638+0000 7f1d4d51d640 1 -- 172.21.15.57:0/2207919931 shutdown_connections 2023-10-19T12:45:30.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:30.638+0000 7f1d4d51d640 1 -- 172.21.15.57:0/2207919931 wait complete. 2023-10-19T12:45:30.654 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-19T12:45:30.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:30.771+0000 7f900bc87640 1 -- 172.21.15.57:0/2562376254 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f90040b0290 msgr2=0x7f9004069720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:30.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:30.771+0000 7f900bc87640 1 --2- 172.21.15.57:0/2562376254 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f90040b0290 0x7f9004069720 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f8ff8005fd0 tx=0x7f8ff802bd90 comp rx=0 tx=0).stop 2023-10-19T12:45:30.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:30.771+0000 7f900bc87640 1 -- 172.21.15.57:0/2562376254 shutdown_connections 2023-10-19T12:45:30.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:30.771+0000 7f900bc87640 1 --2- 172.21.15.57:0/2562376254 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f90040b0290 0x7f9004069720 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:30.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:30.771+0000 7f900bc87640 1 --2- 172.21.15.57:0/2562376254 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f900415b270 0x7f90040cc590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:30.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:30.771+0000 7f900bc87640 1 -- 172.21.15.57:0/2562376254 >> 172.21.15.57:0/2562376254 conn(0x7f90040af7a0 msgr2=0x7f90040afb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:30.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:30.771+0000 7f900bc87640 1 -- 172.21.15.57:0/2562376254 shutdown_connections 2023-10-19T12:45:30.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:30.771+0000 7f900bc87640 1 -- 172.21.15.57:0/2562376254 wait complete. 2023-10-19T12:45:30.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:30.772+0000 7f900bc87640 1 Processor -- start 2023-10-19T12:45:30.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:30.772+0000 7f900bc87640 1 -- start start 2023-10-19T12:45:30.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:30.773+0000 7f900bc87640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9004069c60 0x7f90041f1310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:30.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:30.773+0000 7f900bc87640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f90040b0290 0x7f90040abcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:30.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:30.773+0000 7f900bc87640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f900415b270 0x7f90041f3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:30.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:30.773+0000 7f900bc87640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9004153590 con 0x7f9004069c60 2023-10-19T12:45:30.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:30.773+0000 7f900bc87640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9004152ef0 con 0x7f90040b0290 2023-10-19T12:45:30.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:30.773+0000 7f900bc87640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90041f5520 con 0x7f900415b270 2023-10-19T12:45:30.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:30.773+0000 7f9009a23640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9004069c60 0x7f90041f1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:30.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:30.773+0000 7f9009a23640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9004069c60 0x7f90041f1310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33452/0 (socket says 172.21.15.57:33452) 2023-10-19T12:45:30.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:30.773+0000 7f9009a23640 1 -- 172.21.15.57:0/3076311306 learned_addr learned my addr 172.21.15.57:0/3076311306 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:30.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:30.773+0000 7f9009222640 1 --2- 172.21.15.57:0/3076311306 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f90040b0290 0x7f90040abcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:30.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:30.773+0000 7f900a224640 1 --2- 172.21.15.57:0/3076311306 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f900415b270 0x7f90041f3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:30.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:30.773+0000 7f9009a23640 1 -- 172.21.15.57:0/3076311306 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f900415b270 msgr2=0x7f90041f3cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:30.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:30.773+0000 7f9009a23640 1 --2- 172.21.15.57:0/3076311306 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f900415b270 0x7f90041f3cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:30.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:30.773+0000 7f9009a23640 1 -- 172.21.15.57:0/3076311306 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f90040b0290 msgr2=0x7f90040abcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:30.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:30.773+0000 7f9009a23640 1 --2- 172.21.15.57:0/3076311306 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f90040b0290 0x7f90040abcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:30.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:30.773+0000 7f9009a23640 1 -- 172.21.15.57:0/3076311306 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ff8009560 con 0x7f9004069c60 2023-10-19T12:45:30.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:30.773+0000 7f900a224640 1 --2- 172.21.15.57:0/3076311306 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f900415b270 0x7f90041f3cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:45:30.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:30.774+0000 7f9009a23640 1 --2- 172.21.15.57:0/3076311306 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9004069c60 0x7f90041f1310 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc00b410 tx=0x7f8ffc00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:30.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:30.774+0000 7f90027fc640 1 -- 172.21.15.57:0/3076311306 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ffc011830 con 0x7f9004069c60 2023-10-19T12:45:30.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:30.774+0000 7f90027fc640 1 -- 172.21.15.57:0/3076311306 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f8ffc011aa0 con 0x7f9004069c60 2023-10-19T12:45:30.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:30.774+0000 7f900bc87640 1 -- 172.21.15.57:0/3076311306 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90041531b0 con 0x7f9004069c60 2023-10-19T12:45:30.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:30.774+0000 7f90027fc640 1 -- 172.21.15.57:0/3076311306 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ffc010380 con 0x7f9004069c60 2023-10-19T12:45:30.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:30.774+0000 7f900bc87640 1 -- 172.21.15.57:0/3076311306 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90040ac3d0 con 0x7f9004069c60 2023-10-19T12:45:30.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:30.775+0000 7f900bc87640 1 -- 172.21.15.57:0/3076311306 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9004152d50 con 0x7f9004069c60 2023-10-19T12:45:30.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:30.775+0000 7f90027fc640 1 -- 172.21.15.57:0/3076311306 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f8ffc0104e0 con 0x7f9004069c60 2023-10-19T12:45:30.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:30.776+0000 7f90027fc640 1 --2- 172.21.15.57:0/3076311306 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f8ff006ae40 0x7f8ff006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:30.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:30.776+0000 7f90027fc640 1 -- 172.21.15.57:0/3076311306 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6749+0+0 (secure 0 0 0) 0x7f8ffc089c70 con 0x7f9004069c60 2023-10-19T12:45:30.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:30.776+0000 7f9009222640 1 --2- 172.21.15.57:0/3076311306 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f8ff006ae40 0x7f8ff006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:30.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:30.776+0000 7f9009222640 1 --2- 172.21.15.57:0/3076311306 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f8ff006ae40 0x7f8ff006d300 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f8ff8035000 tx=0x7f8ff8004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:30.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:30.780+0000 7f90027fc640 1 -- 172.21.15.57:0/3076311306 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9004152d50 con 0x7f9004069c60 2023-10-19T12:45:30.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:30.960+0000 7f900bc87640 1 -- 172.21.15.57:0/3076311306 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f9004152d50 con 0x7f9004069c60 2023-10-19T12:45:31.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:31.636+0000 7f90027fc640 1 -- 172.21.15.57:0/3076311306 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v29) v1 ==== 181+0+0 (secure 0 0 0) 0x7f9004152d50 con 0x7f9004069c60 2023-10-19T12:45:31.640 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'cephfs_metadata' removed 2023-10-19T12:45:31.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:31.639+0000 7f900bc87640 1 -- 172.21.15.57:0/3076311306 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f8ff006ae40 msgr2=0x7f8ff006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:31.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:31.639+0000 7f900bc87640 1 --2- 172.21.15.57:0/3076311306 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f8ff006ae40 0x7f8ff006d300 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f8ff8035000 tx=0x7f8ff8004730 comp rx=0 tx=0).stop 2023-10-19T12:45:31.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:31.639+0000 7f900bc87640 1 -- 172.21.15.57:0/3076311306 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9004069c60 msgr2=0x7f90041f1310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:31.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:31.639+0000 7f900bc87640 1 --2- 172.21.15.57:0/3076311306 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9004069c60 0x7f90041f1310 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc00b410 tx=0x7f8ffc00b8d0 comp rx=0 tx=0).stop 2023-10-19T12:45:31.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:31.639+0000 7f900bc87640 1 -- 172.21.15.57:0/3076311306 shutdown_connections 2023-10-19T12:45:31.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:31.639+0000 7f900bc87640 1 --2- 172.21.15.57:0/3076311306 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f8ff006ae40 0x7f8ff006d300 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:31.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:31.639+0000 7f900bc87640 1 --2- 172.21.15.57:0/3076311306 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f900415b270 0x7f90041f3cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:31.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:31.639+0000 7f900bc87640 1 --2- 172.21.15.57:0/3076311306 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f90040b0290 0x7f90040abcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:31.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:31.639+0000 7f900bc87640 1 --2- 172.21.15.57:0/3076311306 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9004069c60 0x7f90041f1310 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:31.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:31.639+0000 7f900bc87640 1 -- 172.21.15.57:0/3076311306 >> 172.21.15.57:0/3076311306 conn(0x7f90040af7a0 msgr2=0x7f90040afb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:31.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:31.639+0000 7f900bc87640 1 -- 172.21.15.57:0/3076311306 shutdown_connections 2023-10-19T12:45:31.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:31.639+0000 7f900bc87640 1 -- 172.21.15.57:0/3076311306 wait complete. 2023-10-19T12:45:31.660 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-19T12:45:31.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:31.776+0000 7f66b83d0640 1 -- 172.21.15.57:0/115563349 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f66b00d1910 msgr2=0x7f66b00b0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:31.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:31.776+0000 7f66b83d0640 1 --2- 172.21.15.57:0/115563349 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f66b00d1910 0x7f66b00b0650 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f66ac005fd0 tx=0x7f66ac02c050 comp rx=0 tx=0).stop 2023-10-19T12:45:31.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:31.776+0000 7f66b83d0640 1 -- 172.21.15.57:0/115563349 shutdown_connections 2023-10-19T12:45:31.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:31.776+0000 7f66b83d0640 1 --2- 172.21.15.57:0/115563349 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f66b00d1910 0x7f66b00b0650 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:31.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:31.776+0000 7f66b83d0640 1 -- 172.21.15.57:0/115563349 >> 172.21.15.57:0/115563349 conn(0x7f66b00d0d60 msgr2=0x7f66b00d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:31.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:31.776+0000 7f66b83d0640 1 -- 172.21.15.57:0/115563349 shutdown_connections 2023-10-19T12:45:31.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:31.776+0000 7f66b83d0640 1 -- 172.21.15.57:0/115563349 wait complete. 2023-10-19T12:45:31.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:31.777+0000 7f66b83d0640 1 Processor -- start 2023-10-19T12:45:31.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:31.777+0000 7f66b83d0640 1 -- start start 2023-10-19T12:45:31.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:31.778+0000 7f66b83d0640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f66b00cf340 0x7f66b00b42f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:31.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:31.778+0000 7f66b83d0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f66b00d1910 0x7f66b01553b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:31.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:31.778+0000 7f66b83d0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f66b015b2a0 0x7f66b00c9ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:31.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:31.778+0000 7f66b83d0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66b00b5550 con 0x7f66b00d1910 2023-10-19T12:45:31.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:31.778+0000 7f66b83d0640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66b01558f0 con 0x7f66b015b2a0 2023-10-19T12:45:31.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:31.778+0000 7f66b83d0640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66b00b48d0 con 0x7f66b00cf340 2023-10-19T12:45:31.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:31.778+0000 7f66b596b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f66b00d1910 0x7f66b01553b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:31.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:31.778+0000 7f66b596b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f66b00d1910 0x7f66b01553b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33468/0 (socket says 172.21.15.57:33468) 2023-10-19T12:45:31.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:31.778+0000 7f66b596b640 1 -- 172.21.15.57:0/2720548860 learned_addr learned my addr 172.21.15.57:0/2720548860 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:31.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:31.778+0000 7f66b696d640 1 --2- 172.21.15.57:0/2720548860 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f66b015b2a0 0x7f66b00c9ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:31.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:31.778+0000 7f66b616c640 1 --2- 172.21.15.57:0/2720548860 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f66b00cf340 0x7f66b00b42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:31.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:31.778+0000 7f66b596b640 1 -- 172.21.15.57:0/2720548860 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f66b00cf340 msgr2=0x7f66b00b42f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:31.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:31.778+0000 7f66b596b640 1 --2- 172.21.15.57:0/2720548860 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f66b00cf340 0x7f66b00b42f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:31.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:31.778+0000 7f66b596b640 1 -- 172.21.15.57:0/2720548860 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f66b015b2a0 msgr2=0x7f66b00c9ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:31.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:31.778+0000 7f66b596b640 1 --2- 172.21.15.57:0/2720548860 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f66b015b2a0 0x7f66b00c9ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:31.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:31.778+0000 7f66b596b640 1 -- 172.21.15.57:0/2720548860 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66ac009840 con 0x7f66b00d1910 2023-10-19T12:45:31.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:31.778+0000 7f66b596b640 1 --2- 172.21.15.57:0/2720548860 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f66b00d1910 0x7f66b01553b0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f66a000b3f0 tx=0x7f66a000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:31.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:31.779+0000 7f66a6ffd640 1 -- 172.21.15.57:0/2720548860 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66a0011830 con 0x7f66b00d1910 2023-10-19T12:45:31.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:31.779+0000 7f66a6ffd640 1 -- 172.21.15.57:0/2720548860 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f66a0011a80 con 0x7f66b00d1910 2023-10-19T12:45:31.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:31.779+0000 7f66b83d0640 1 -- 172.21.15.57:0/2720548860 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66b00b4bb0 con 0x7f66b00d1910 2023-10-19T12:45:31.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:31.779+0000 7f66a6ffd640 1 -- 172.21.15.57:0/2720548860 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66a0011e90 con 0x7f66b00d1910 2023-10-19T12:45:31.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:31.779+0000 7f66b83d0640 1 -- 172.21.15.57:0/2720548860 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66b00b5170 con 0x7f66b00d1910 2023-10-19T12:45:31.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:31.779+0000 7f66b83d0640 1 -- 172.21.15.57:0/2720548860 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66b00690e0 con 0x7f66b00d1910 2023-10-19T12:45:31.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:31.780+0000 7f66a6ffd640 1 -- 172.21.15.57:0/2720548860 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f66a001a400 con 0x7f66b00d1910 2023-10-19T12:45:31.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:31.781+0000 7f66a6ffd640 1 --2- 172.21.15.57:0/2720548860 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f669406ae40 0x7f669406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:31.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:31.781+0000 7f66a6ffd640 1 -- 172.21.15.57:0/2720548860 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6311+0+0 (secure 0 0 0) 0x7f66a0089a00 con 0x7f66b00d1910 2023-10-19T12:45:31.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:31.781+0000 7f66b616c640 1 --2- 172.21.15.57:0/2720548860 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f669406ae40 0x7f669406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:31.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:31.781+0000 7f66b616c640 1 --2- 172.21.15.57:0/2720548860 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f669406ae40 0x7f669406d300 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f66a80046b0 tx=0x7f66a8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:31.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:31.785+0000 7f66a6ffd640 1 -- 172.21.15.57:0/2720548860 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f66a00547b0 con 0x7f66b00d1910 2023-10-19T12:45:31.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:31.961+0000 7f66b83d0640 1 -- 172.21.15.57:0/2720548860 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f66b00d0b40 con 0x7f66b00d1910 2023-10-19T12:45:32.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:32.644+0000 7f66a6ffd640 1 -- 172.21.15.57:0/2720548860 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v30) v1 ==== 169+0+0 (secure 0 0 0) 0x7f66a00150d0 con 0x7f66b00d1910 2023-10-19T12:45:32.647 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'cephfs_data' removed 2023-10-19T12:45:32.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:32.650+0000 7f66b83d0640 1 -- 172.21.15.57:0/2720548860 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f669406ae40 msgr2=0x7f669406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:32.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:32.650+0000 7f66b83d0640 1 --2- 172.21.15.57:0/2720548860 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f669406ae40 0x7f669406d300 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f66a80046b0 tx=0x7f66a8009040 comp rx=0 tx=0).stop 2023-10-19T12:45:32.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:32.650+0000 7f66b83d0640 1 -- 172.21.15.57:0/2720548860 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f66b00d1910 msgr2=0x7f66b01553b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:32.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:32.650+0000 7f66b83d0640 1 --2- 172.21.15.57:0/2720548860 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f66b00d1910 0x7f66b01553b0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f66a000b3f0 tx=0x7f66a000b8b0 comp rx=0 tx=0).stop 2023-10-19T12:45:32.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:32.651+0000 7f66b83d0640 1 -- 172.21.15.57:0/2720548860 shutdown_connections 2023-10-19T12:45:32.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:32.651+0000 7f66b83d0640 1 --2- 172.21.15.57:0/2720548860 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f669406ae40 0x7f669406d300 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:32.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:32.651+0000 7f66b83d0640 1 --2- 172.21.15.57:0/2720548860 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f66b015b2a0 0x7f66b00c9ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:32.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:32.651+0000 7f66b83d0640 1 --2- 172.21.15.57:0/2720548860 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f66b00d1910 0x7f66b01553b0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:32.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:32.651+0000 7f66b83d0640 1 --2- 172.21.15.57:0/2720548860 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f66b00cf340 0x7f66b00b42f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:32.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:32.651+0000 7f66b83d0640 1 -- 172.21.15.57:0/2720548860 >> 172.21.15.57:0/2720548860 conn(0x7f66b00d0d60 msgr2=0x7f66b00ad950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:32.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:32.651+0000 7f66b83d0640 1 -- 172.21.15.57:0/2720548860 shutdown_connections 2023-10-19T12:45:32.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:32.651+0000 7f66b83d0640 1 -- 172.21.15.57:0/2720548860 wait complete. 2023-10-19T12:45:32.672 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2023-10-19T12:45:32.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:32.786+0000 7fded0db6640 1 -- 172.21.15.57:0/1248862125 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdecc15b2a0 msgr2=0x7fdecc0d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:32.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:32.786+0000 7fded0db6640 1 --2- 172.21.15.57:0/1248862125 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdecc15b2a0 0x7fdecc0d1910 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fdecc15b680 tx=0x7fdeb402bcf0 comp rx=0 tx=0).stop 2023-10-19T12:45:32.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:32.786+0000 7fded0db6640 1 -- 172.21.15.57:0/1248862125 shutdown_connections 2023-10-19T12:45:32.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:32.786+0000 7fded0db6640 1 --2- 172.21.15.57:0/1248862125 >> v2:172.21.15.179:3300/0 conn(0x7fdecc0ceb40 0x7fdecc0b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:32.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:32.786+0000 7fded0db6640 1 --2- 172.21.15.57:0/1248862125 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdecc15b2a0 0x7fdecc0d1910 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:32.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:32.786+0000 7fded0db6640 1 -- 172.21.15.57:0/1248862125 >> 172.21.15.57:0/1248862125 conn(0x7fdecc0d0d60 msgr2=0x7fdecc0d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:32.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:32.786+0000 7fded0db6640 1 -- 172.21.15.57:0/1248862125 shutdown_connections 2023-10-19T12:45:32.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:32.786+0000 7fded0db6640 1 -- 172.21.15.57:0/1248862125 wait complete. 2023-10-19T12:45:32.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:32.788+0000 7fded0db6640 1 Processor -- start 2023-10-19T12:45:32.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:32.788+0000 7fded0db6640 1 -- start start 2023-10-19T12:45:32.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:32.788+0000 7fded0db6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdecc0ceb40 0x7fdecc1e6290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:32.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:32.788+0000 7fded0db6640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdecc0cf340 0x7fdecc158140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:32.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:32.788+0000 7fded0db6640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdecc15b2a0 0x7fdecc0b0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:32.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:32.789+0000 7fded0db6640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdecc0c9be0 con 0x7fdecc0ceb40 2023-10-19T12:45:32.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:32.789+0000 7fded0db6640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdecc0ca3f0 con 0x7fdecc0cf340 2023-10-19T12:45:32.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:32.789+0000 7fded0db6640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdecc1e67d0 con 0x7fdecc15b2a0 2023-10-19T12:45:32.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:32.789+0000 7fdeca59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdecc0ceb40 0x7fdecc1e6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:32.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:32.789+0000 7fdeca59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdecc0ceb40 0x7fdecc1e6290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33494/0 (socket says 172.21.15.57:33494) 2023-10-19T12:45:32.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:32.789+0000 7fdeca59c640 1 -- 172.21.15.57:0/1250251109 learned_addr learned my addr 172.21.15.57:0/1250251109 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:32.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:32.789+0000 7fdec9d9b640 1 --2- 172.21.15.57:0/1250251109 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdecc0cf340 0x7fdecc158140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:32.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:32.789+0000 7fdecad9d640 1 --2- 172.21.15.57:0/1250251109 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdecc15b2a0 0x7fdecc0b0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:32.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:32.789+0000 7fdeca59c640 1 -- 172.21.15.57:0/1250251109 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdecc15b2a0 msgr2=0x7fdecc0b0ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:32.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:32.789+0000 7fdeca59c640 1 --2- 172.21.15.57:0/1250251109 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdecc15b2a0 0x7fdecc0b0ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:32.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:32.789+0000 7fdeca59c640 1 -- 172.21.15.57:0/1250251109 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdecc0cf340 msgr2=0x7fdecc158140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:32.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:32.789+0000 7fdeca59c640 1 --2- 172.21.15.57:0/1250251109 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdecc0cf340 0x7fdecc158140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:32.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:32.789+0000 7fdeca59c640 1 -- 172.21.15.57:0/1250251109 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdeb4009490 con 0x7fdecc0ceb40 2023-10-19T12:45:32.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:32.789+0000 7fdecad9d640 1 --2- 172.21.15.57:0/1250251109 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdecc15b2a0 0x7fdecc0b0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:45:32.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:32.789+0000 7fdeca59c640 1 --2- 172.21.15.57:0/1250251109 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdecc0ceb40 0x7fdecc1e6290 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fdec400c530 tx=0x7fdec400c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:32.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:32.789+0000 7fdec2ffd640 1 -- 172.21.15.57:0/1250251109 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdec4012830 con 0x7fdecc0ceb40 2023-10-19T12:45:32.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:32.789+0000 7fdec2ffd640 1 -- 172.21.15.57:0/1250251109 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fdec4012a80 con 0x7fdecc0ceb40 2023-10-19T12:45:32.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:32.789+0000 7fdec2ffd640 1 -- 172.21.15.57:0/1250251109 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdec4012e50 con 0x7fdecc0ceb40 2023-10-19T12:45:32.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:32.789+0000 7fded0db6640 1 -- 172.21.15.57:0/1250251109 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdecc0c9e30 con 0x7fdecc0ceb40 2023-10-19T12:45:32.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:32.789+0000 7fded0db6640 1 -- 172.21.15.57:0/1250251109 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdecc15b800 con 0x7fdecc0ceb40 2023-10-19T12:45:32.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:32.790+0000 7fded0db6640 1 -- 172.21.15.57:0/1250251109 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdecc0690e0 con 0x7fdecc0ceb40 2023-10-19T12:45:32.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:32.791+0000 7fdec2ffd640 1 -- 172.21.15.57:0/1250251109 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fdec401a3b0 con 0x7fdecc0ceb40 2023-10-19T12:45:32.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:32.791+0000 7fdec2ffd640 1 --2- 172.21.15.57:0/1250251109 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fdeac06ae40 0x7fdeac06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:32.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:32.791+0000 7fdec2ffd640 1 -- 172.21.15.57:0/1250251109 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5781+0+0 (secure 0 0 0) 0x7fdec40895c0 con 0x7fdecc0ceb40 2023-10-19T12:45:32.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:32.791+0000 7fdec9d9b640 1 --2- 172.21.15.57:0/1250251109 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fdeac06ae40 0x7fdeac06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:32.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:32.792+0000 7fdec9d9b640 1 --2- 172.21.15.57:0/1250251109 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fdeac06ae40 0x7fdeac06d300 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fdeb4036000 tx=0x7fdeb4004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:32.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:32.795+0000 7fdec2ffd640 1 -- 172.21.15.57:0/1250251109 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fdec40546b0 con 0x7fdecc0ceb40 2023-10-19T12:45:32.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:32.979+0000 7fded0db6640 1 -- 172.21.15.57:0/1250251109 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fdecc0b0d00 con 0x7fdecc0ceb40 2023-10-19T12:45:33.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:33.965+0000 7fdec2ffd640 1 -- 172.21.15.57:0/1250251109 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v31) v1 ==== 178+0+0 (secure 0 0 0) 0x7fdec4057cd0 con 0x7fdecc0ceb40 2023-10-19T12:45:33.968 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'cephfs_data_ec' removed 2023-10-19T12:45:33.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:33.968+0000 7fded0db6640 1 -- 172.21.15.57:0/1250251109 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fdeac06ae40 msgr2=0x7fdeac06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:33.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:33.968+0000 7fded0db6640 1 --2- 172.21.15.57:0/1250251109 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fdeac06ae40 0x7fdeac06d300 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fdeb4036000 tx=0x7fdeb4004730 comp rx=0 tx=0).stop 2023-10-19T12:45:33.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:33.968+0000 7fded0db6640 1 -- 172.21.15.57:0/1250251109 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdecc0ceb40 msgr2=0x7fdecc1e6290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:33.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:33.968+0000 7fded0db6640 1 --2- 172.21.15.57:0/1250251109 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdecc0ceb40 0x7fdecc1e6290 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fdec400c530 tx=0x7fdec400c9f0 comp rx=0 tx=0).stop 2023-10-19T12:45:33.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:33.968+0000 7fded0db6640 1 -- 172.21.15.57:0/1250251109 shutdown_connections 2023-10-19T12:45:33.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:33.968+0000 7fded0db6640 1 --2- 172.21.15.57:0/1250251109 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fdeac06ae40 0x7fdeac06d300 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:33.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:33.968+0000 7fded0db6640 1 --2- 172.21.15.57:0/1250251109 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdecc15b2a0 0x7fdecc0b0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:33.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:33.968+0000 7fded0db6640 1 --2- 172.21.15.57:0/1250251109 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdecc0cf340 0x7fdecc158140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:33.973 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:33.968+0000 7fded0db6640 1 --2- 172.21.15.57:0/1250251109 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdecc0ceb40 0x7fdecc1e6290 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:33.973 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:33.968+0000 7fded0db6640 1 -- 172.21.15.57:0/1250251109 >> 172.21.15.57:0/1250251109 conn(0x7fdecc0d0d60 msgr2=0x7fdecc15a3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:33.973 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:33.968+0000 7fded0db6640 1 -- 172.21.15.57:0/1250251109 shutdown_connections 2023-10-19T12:45:33.973 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:33.968+0000 7fded0db6640 1 -- 172.21.15.57:0/1250251109 wait complete. 2023-10-19T12:45:33.990 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-19T12:45:33.990 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-19T12:45:33.991 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-19T12:45:33.991 INFO:teuthology.orchestra.run:waiting for 300 2023-10-19T12:45:33.991 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-19T12:45:33.992 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-19T12:45:33.992 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-19T12:45:33.992 INFO:teuthology.orchestra.run:waiting for 300 2023-10-19T12:45:33.993 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-19T12:45:33.993 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-19T12:45:33.993 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-19T12:45:33.994 INFO:teuthology.orchestra.run:waiting for 300 2023-10-19T12:45:33.994 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-19T12:45:33.994 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-19T12:45:33.995 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-19T12:45:33.995 INFO:teuthology.orchestra.run:waiting for 300 2023-10-19T12:45:34.002 INFO:tasks.ceph.mds.a:Stopped 2023-10-19T12:45:34.002 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-19T12:45:34.003 INFO:tasks.ceph.mds.c:Stopped 2023-10-19T12:45:34.003 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-19T12:45:34.005 INFO:tasks.ceph.mds.b:Stopped 2023-10-19T12:45:34.005 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-19T12:45:34.006 INFO:tasks.ceph.mds.d:Stopped 2023-10-19T12:45:34.007 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-19T12:45:34.044 INFO:tasks.ceph.mds.a:Started 2023-10-19T12:45:34.045 DEBUG:teuthology.parallel:result is None 2023-10-19T12:45:34.045 INFO:tasks.ceph.mds.c:Started 2023-10-19T12:45:34.046 DEBUG:teuthology.parallel:result is None 2023-10-19T12:45:34.049 INFO:tasks.ceph.mds.b:Started 2023-10-19T12:45:34.049 DEBUG:teuthology.parallel:result is None 2023-10-19T12:45:34.050 INFO:tasks.ceph.mds.d:Started 2023-10-19T12:45:34.051 DEBUG:teuthology.parallel:result is None 2023-10-19T12:45:34.051 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-19T12:45:34.134 INFO:tasks.ceph.mds.c.smithi057.stdout:starting mds.c at 2023-10-19T12:45:34.134 INFO:tasks.ceph.mds.a.smithi057.stdout:starting mds.a at 2023-10-19T12:45:34.138 INFO:tasks.ceph.mds.b.smithi179.stdout:starting mds.b at 2023-10-19T12:45:34.138 INFO:tasks.ceph.mds.d.smithi179.stdout:starting mds.d at 2023-10-19T12:45:34.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:34.172+0000 7faa16b7d640 1 -- 172.21.15.57:0/538259804 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7faa100cd9e0 msgr2=0x7faa100cde70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:34.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:34.172+0000 7faa16b7d640 1 --2- 172.21.15.57:0/538259804 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7faa100cd9e0 0x7faa100cde70 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7faa00009830 tx=0x7faa0002bd60 comp rx=0 tx=0).stop 2023-10-19T12:45:34.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:34.172+0000 7faa16b7d640 1 -- 172.21.15.57:0/538259804 shutdown_connections 2023-10-19T12:45:34.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:34.172+0000 7faa16b7d640 1 --2- 172.21.15.57:0/538259804 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7faa100cd9e0 0x7faa100cde70 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:34.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:34.172+0000 7faa16b7d640 1 --2- 172.21.15.57:0/538259804 >> v2:172.21.15.179:3300/0 conn(0x7faa10158910 0x7faa10158cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:34.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:34.173+0000 7faa16b7d640 1 -- 172.21.15.57:0/538259804 >> 172.21.15.57:0/538259804 conn(0x7faa100af7a0 msgr2=0x7faa100afb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:34.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:34.173+0000 7faa16b7d640 1 -- 172.21.15.57:0/538259804 shutdown_connections 2023-10-19T12:45:34.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:34.173+0000 7faa16b7d640 1 -- 172.21.15.57:0/538259804 wait complete. 2023-10-19T12:45:34.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:34.174+0000 7faa16b7d640 1 Processor -- start 2023-10-19T12:45:34.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:34.174+0000 7faa16b7d640 1 -- start start 2023-10-19T12:45:34.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:34.175+0000 7faa16b7d640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7faa100cd1c0 0x7faa1014b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:34.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:34.175+0000 7faa16b7d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa100cd9e0 0x7faa1014ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:34.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:34.175+0000 7faa16b7d640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faa10158910 0x7faa1014d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:34.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:34.175+0000 7faa16b7d640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa101ef5a0 con 0x7faa100cd9e0 2023-10-19T12:45:34.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:34.175+0000 7faa16b7d640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa1014d980 con 0x7faa10158910 2023-10-19T12:45:34.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:34.175+0000 7faa16b7d640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa1014dc60 con 0x7faa100cd1c0 2023-10-19T12:45:34.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:34.175+0000 7faa0ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa100cd9e0 0x7faa1014ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:34.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:34.175+0000 7faa1511a640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faa10158910 0x7faa1014d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:34.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:34.175+0000 7faa0ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa100cd9e0 0x7faa1014ce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33524/0 (socket says 172.21.15.57:33524) 2023-10-19T12:45:34.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:34.175+0000 7faa0ffff640 1 -- 172.21.15.57:0/1772978682 learned_addr learned my addr 172.21.15.57:0/1772978682 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:34.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:34.175+0000 7faa14919640 1 --2- 172.21.15.57:0/1772978682 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7faa100cd1c0 0x7faa1014b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:34.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:34.175+0000 7faa0ffff640 1 -- 172.21.15.57:0/1772978682 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7faa100cd1c0 msgr2=0x7faa1014b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:34.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:34.175+0000 7faa0ffff640 1 --2- 172.21.15.57:0/1772978682 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7faa100cd1c0 0x7faa1014b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:34.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:34.175+0000 7faa0ffff640 1 -- 172.21.15.57:0/1772978682 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faa10158910 msgr2=0x7faa1014d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:34.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:34.175+0000 7faa0ffff640 1 --2- 172.21.15.57:0/1772978682 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faa10158910 0x7faa1014d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:34.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:34.175+0000 7faa0ffff640 1 -- 172.21.15.57:0/1772978682 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa00009490 con 0x7faa100cd9e0 2023-10-19T12:45:34.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:34.175+0000 7faa1511a640 1 --2- 172.21.15.57:0/1772978682 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faa10158910 0x7faa1014d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T12:45:34.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:34.175+0000 7faa14919640 1 --2- 172.21.15.57:0/1772978682 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7faa100cd1c0 0x7faa1014b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:45:34.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:34.175+0000 7faa0ffff640 1 --2- 172.21.15.57:0/1772978682 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa100cd9e0 0x7faa1014ce30 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7faa0400c700 tx=0x7faa0400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:34.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:34.175+0000 7faa0d7fa640 1 -- 172.21.15.57:0/1772978682 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa0400a840 con 0x7faa100cd9e0 2023-10-19T12:45:34.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:34.175+0000 7faa16b7d640 1 -- 172.21.15.57:0/1772978682 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa1014df40 con 0x7faa100cd9e0 2023-10-19T12:45:34.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:34.175+0000 7faa0d7fa640 1 -- 172.21.15.57:0/1772978682 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7faa0400aa90 con 0x7faa100cd9e0 2023-10-19T12:45:34.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:34.175+0000 7faa0d7fa640 1 -- 172.21.15.57:0/1772978682 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa04012420 con 0x7faa100cd9e0 2023-10-19T12:45:34.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:34.175+0000 7faa16b7d640 1 -- 172.21.15.57:0/1772978682 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa101559e0 con 0x7faa100cd9e0 2023-10-19T12:45:34.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:34.176+0000 7faa16b7d640 1 -- 172.21.15.57:0/1772978682 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa1014e460 con 0x7faa100cd9e0 2023-10-19T12:45:34.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:34.177+0000 7faa0d7fa640 1 -- 172.21.15.57:0/1772978682 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7faa0401a3b0 con 0x7faa100cd9e0 2023-10-19T12:45:34.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:34.177+0000 7faa0d7fa640 1 --2- 172.21.15.57:0/1772978682 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fa9fc06ae40 0x7fa9fc06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:34.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:34.177+0000 7faa0d7fa640 1 -- 172.21.15.57:0/1772978682 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5263+0+0 (secure 0 0 0) 0x7faa040893f0 con 0x7faa100cd9e0 2023-10-19T12:45:34.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:34.177+0000 7faa14919640 1 --2- 172.21.15.57:0/1772978682 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fa9fc06ae40 0x7fa9fc06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:34.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:34.178+0000 7faa14919640 1 --2- 172.21.15.57:0/1772978682 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fa9fc06ae40 0x7fa9fc06d300 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7faa10150360 tx=0x7faa08009500 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:34.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:34.182+0000 7faa0d7fa640 1 -- 172.21.15.57:0/1772978682 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7faa040545d0 con 0x7faa100cd9e0 2023-10-19T12:45:34.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:34.370+0000 7faa16b7d640 1 -- 172.21.15.57:0/1772978682 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7faa1015c7f0 con 0x7faa100cd9e0 2023-10-19T12:45:34.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:34.991+0000 7faa0d7fa640 1 -- 172.21.15.57:0/1772978682 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v33) v1 ==== 97+0+0 (secure 0 0 0) 0x7faa1015c7f0 con 0x7faa100cd9e0 2023-10-19T12:45:34.995 INFO:teuthology.orchestra.run.smithi057.stderr: removed all blocklist entries 2023-10-19T12:45:34.998 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:34.995+0000 7faa16b7d640 1 -- 172.21.15.57:0/1772978682 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fa9fc06ae40 msgr2=0x7fa9fc06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:34.998 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:34.995+0000 7faa16b7d640 1 --2- 172.21.15.57:0/1772978682 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fa9fc06ae40 0x7fa9fc06d300 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7faa10150360 tx=0x7faa08009500 comp rx=0 tx=0).stop 2023-10-19T12:45:34.999 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:34.995+0000 7faa16b7d640 1 -- 172.21.15.57:0/1772978682 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa100cd9e0 msgr2=0x7faa1014ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:34.999 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:34.995+0000 7faa16b7d640 1 --2- 172.21.15.57:0/1772978682 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa100cd9e0 0x7faa1014ce30 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7faa0400c700 tx=0x7faa0400cbc0 comp rx=0 tx=0).stop 2023-10-19T12:45:34.999 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:34.995+0000 7faa16b7d640 1 -- 172.21.15.57:0/1772978682 shutdown_connections 2023-10-19T12:45:34.999 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:34.995+0000 7faa16b7d640 1 --2- 172.21.15.57:0/1772978682 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fa9fc06ae40 0x7fa9fc06d300 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:34.999 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:34.995+0000 7faa16b7d640 1 --2- 172.21.15.57:0/1772978682 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faa10158910 0x7faa1014d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:35.000 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:34.995+0000 7faa16b7d640 1 --2- 172.21.15.57:0/1772978682 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa100cd9e0 0x7faa1014ce30 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:35.000 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:34.995+0000 7faa16b7d640 1 --2- 172.21.15.57:0/1772978682 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7faa100cd1c0 0x7faa1014b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:35.000 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:34.995+0000 7faa16b7d640 1 -- 172.21.15.57:0/1772978682 >> 172.21.15.57:0/1772978682 conn(0x7faa100af7a0 msgr2=0x7faa100cc270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:35.000 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:34.995+0000 7faa16b7d640 1 -- 172.21.15.57:0/1772978682 shutdown_connections 2023-10-19T12:45:35.000 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:34.995+0000 7faa16b7d640 1 -- 172.21.15.57:0/1772978682 wait complete. 2023-10-19T12:45:35.013 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-19T12:45:35.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.124+0000 7f4d49ec9640 1 -- 172.21.15.57:0/2356037544 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4d4406b840 msgr2=0x7f4d440b34c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:35.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.124+0000 7f4d49ec9640 1 --2- 172.21.15.57:0/2356037544 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4d4406b840 0x7f4d440b34c0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f4d38005fd0 tx=0x7f4d3802be10 comp rx=0 tx=0).stop 2023-10-19T12:45:35.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.124+0000 7f4d49ec9640 1 -- 172.21.15.57:0/2356037544 shutdown_connections 2023-10-19T12:45:35.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.124+0000 7f4d49ec9640 1 --2- 172.21.15.57:0/2356037544 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4d4415bb90 0x7f4d4415bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:35.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.124+0000 7f4d49ec9640 1 --2- 172.21.15.57:0/2356037544 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4d4406b840 0x7f4d440b34c0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:35.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.124+0000 7f4d49ec9640 1 -- 172.21.15.57:0/2356037544 >> 172.21.15.57:0/2356037544 conn(0x7f4d440b3f00 msgr2=0x7f4d440b4310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:35.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.125+0000 7f4d49ec9640 1 -- 172.21.15.57:0/2356037544 shutdown_connections 2023-10-19T12:45:35.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.125+0000 7f4d49ec9640 1 -- 172.21.15.57:0/2356037544 wait complete. 2023-10-19T12:45:35.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.126+0000 7f4d49ec9640 1 Processor -- start 2023-10-19T12:45:35.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.126+0000 7f4d49ec9640 1 -- start start 2023-10-19T12:45:35.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.127+0000 7f4d49ec9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4d4406b840 0x7f4d44151700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:35.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.127+0000 7f4d49ec9640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4d4415b7b0 0x7f4d44157dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:35.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.127+0000 7f4d49ec9640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4d4415bb90 0x7f4d440adc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:35.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.127+0000 7f4d49ec9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d44157930 con 0x7f4d4406b840 2023-10-19T12:45:35.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.127+0000 7f4d49ec9640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d4415bfd0 con 0x7f4d4415b7b0 2023-10-19T12:45:35.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.127+0000 7f4d49ec9640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d4414ff80 con 0x7f4d4415bb90 2023-10-19T12:45:35.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.127+0000 7f4d437fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4d4406b840 0x7f4d44151700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:35.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.127+0000 7f4d437fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4d4406b840 0x7f4d44151700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33536/0 (socket says 172.21.15.57:33536) 2023-10-19T12:45:35.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.127+0000 7f4d437fe640 1 -- 172.21.15.57:0/3641686614 learned_addr learned my addr 172.21.15.57:0/3641686614 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:35.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.127+0000 7f4d43fff640 1 --2- 172.21.15.57:0/3641686614 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4d4415bb90 0x7f4d440adc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:35.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.127+0000 7f4d42ffd640 1 --2- 172.21.15.57:0/3641686614 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4d4415b7b0 0x7f4d44157dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:35.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.127+0000 7f4d437fe640 1 -- 172.21.15.57:0/3641686614 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4d4415bb90 msgr2=0x7f4d440adc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:35.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.127+0000 7f4d437fe640 1 --2- 172.21.15.57:0/3641686614 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4d4415bb90 0x7f4d440adc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:35.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.127+0000 7f4d437fe640 1 -- 172.21.15.57:0/3641686614 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4d4415b7b0 msgr2=0x7f4d44157dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:35.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.127+0000 7f4d437fe640 1 --2- 172.21.15.57:0/3641686614 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4d4415b7b0 0x7f4d44157dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:35.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.127+0000 7f4d437fe640 1 -- 172.21.15.57:0/3641686614 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d38009560 con 0x7f4d4406b840 2023-10-19T12:45:35.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.127+0000 7f4d437fe640 1 --2- 172.21.15.57:0/3641686614 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4d4406b840 0x7f4d44151700 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f4d3400c450 tx=0x7f4d3400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:35.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.127+0000 7f4d2bfff640 1 -- 172.21.15.57:0/3641686614 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d340052d0 con 0x7f4d4406b840 2023-10-19T12:45:35.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.127+0000 7f4d2bfff640 1 -- 172.21.15.57:0/3641686614 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f4d34005430 con 0x7f4d4406b840 2023-10-19T12:45:35.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.127+0000 7f4d49ec9640 1 -- 172.21.15.57:0/3641686614 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d441572e0 con 0x7f4d4406b840 2023-10-19T12:45:35.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.127+0000 7f4d2bfff640 1 -- 172.21.15.57:0/3641686614 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d34012890 con 0x7f4d4406b840 2023-10-19T12:45:35.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.128+0000 7f4d49ec9640 1 -- 172.21.15.57:0/3641686614 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d4415c270 con 0x7f4d4406b840 2023-10-19T12:45:35.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.129+0000 7f4d49ec9640 1 -- 172.21.15.57:0/3641686614 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d440675f0 con 0x7f4d4406b840 2023-10-19T12:45:35.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.129+0000 7f4d2bfff640 1 -- 172.21.15.57:0/3641686614 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f4d340035f0 con 0x7f4d4406b840 2023-10-19T12:45:35.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.130+0000 7f4d2bfff640 1 --2- 172.21.15.57:0/3641686614 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f4d3006ae40 0x7f4d3006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:35.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.130+0000 7f4d2bfff640 1 -- 172.21.15.57:0/3641686614 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f4d34088f70 con 0x7f4d4406b840 2023-10-19T12:45:35.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.130+0000 7f4d42ffd640 1 --2- 172.21.15.57:0/3641686614 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f4d3006ae40 0x7f4d3006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:35.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.130+0000 7f4d42ffd640 1 --2- 172.21.15.57:0/3641686614 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f4d3006ae40 0x7f4d3006d300 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f4d38035000 tx=0x7f4d38004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:35.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.134+0000 7f4d2bfff640 1 -- 172.21.15.57:0/3641686614 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4d340bbca0 con 0x7f4d4406b840 2023-10-19T12:45:35.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.376+0000 7f4d49ec9640 1 -- 172.21.15.57:0/3641686614 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f4d440690e0 con 0x7f4d4406b840 2023-10-19T12:45:35.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.377+0000 7f4d2bfff640 1 -- 172.21.15.57:0/3641686614 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v12) v1 ==== 80+0+6645 (secure 0 0 0) 0x7f4d340542e0 con 0x7f4d4406b840 2023-10-19T12:45:35.380 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:45:35.381 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2023-10-19T12:45:35.381 INFO:teuthology.orchestra.run.smithi057.stdout: "auth_dump": [ 2023-10-19T12:45:35.381 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T12:45:35.381 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "mds.a", 2023-10-19T12:45:35.381 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBjJDFlmbVANhAAP/ynHlU/OxaRq5lQoAa8Ng==", 2023-10-19T12:45:35.382 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T12:45:35.382 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T12:45:35.382 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow *", 2023-10-19T12:45:35.382 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow *", 2023-10-19T12:45:35.382 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T12:45:35.383 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T12:45:35.383 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T12:45:35.383 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T12:45:35.383 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "mds.b", 2023-10-19T12:45:35.383 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFl9l7RBxAAC0geI2tJ1SIZZr1++kS/BA==", 2023-10-19T12:45:35.383 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T12:45:35.384 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T12:45:35.384 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow *", 2023-10-19T12:45:35.384 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow *", 2023-10-19T12:45:35.384 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T12:45:35.384 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T12:45:35.384 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T12:45:35.385 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T12:45:35.385 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "mds.c", 2023-10-19T12:45:35.385 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFlFkaoAhAA/PBizkJ4bm958oP4CmPuIQ==", 2023-10-19T12:45:35.385 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T12:45:35.385 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T12:45:35.386 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow *", 2023-10-19T12:45:35.386 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow *", 2023-10-19T12:45:35.386 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T12:45:35.386 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T12:45:35.386 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T12:45:35.386 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T12:45:35.387 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "mds.d", 2023-10-19T12:45:35.387 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFlx8skDxAAAfCHTYtYJgJik0WHSWeQfQ==", 2023-10-19T12:45:35.387 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T12:45:35.387 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T12:45:35.387 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow *", 2023-10-19T12:45:35.388 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow *", 2023-10-19T12:45:35.388 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T12:45:35.388 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T12:45:35.388 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T12:45:35.388 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T12:45:35.388 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.0", 2023-10-19T12:45:35.389 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBrJDFlCo9sBxAAQN2cID4nCQgRwGaGFEV8LQ==", 2023-10-19T12:45:35.389 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T12:45:35.389 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T12:45:35.389 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T12:45:35.389 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T12:45:35.390 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T12:45:35.390 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T12:45:35.390 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T12:45:35.390 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.1", 2023-10-19T12:45:35.390 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBuJDFl4fsLCBAAp3Pi80yZtc5sju0GisEuVA==", 2023-10-19T12:45:35.391 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T12:45:35.391 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T12:45:35.391 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T12:45:35.391 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T12:45:35.391 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T12:45:35.392 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T12:45:35.392 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T12:45:35.392 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.2", 2023-10-19T12:45:35.392 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBxJDFl+KAKCBAAd4N9cluARxPVfBLU+7Hd7A==", 2023-10-19T12:45:35.392 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T12:45:35.392 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T12:45:35.393 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T12:45:35.393 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T12:45:35.393 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T12:45:35.393 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T12:45:35.393 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T12:45:35.393 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.3", 2023-10-19T12:45:35.394 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQB0JDFlU/toCBAAP0Ou1zcHKNx5SfbybJkh4Q==", 2023-10-19T12:45:35.394 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T12:45:35.394 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T12:45:35.394 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T12:45:35.394 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T12:45:35.395 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T12:45:35.395 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T12:45:35.395 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T12:45:35.395 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.4", 2023-10-19T12:45:35.395 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQB6JDFlc6stEBAALkgNyxoiCTzZnKpBSPxI6w==", 2023-10-19T12:45:35.395 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T12:45:35.396 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T12:45:35.396 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T12:45:35.396 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T12:45:35.396 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T12:45:35.396 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T12:45:35.396 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T12:45:35.397 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.5", 2023-10-19T12:45:35.397 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQB9JDFlI3cNDhAAuG3tHJ6RvPwAzPE76wTIVQ==", 2023-10-19T12:45:35.397 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T12:45:35.397 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T12:45:35.397 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T12:45:35.397 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T12:45:35.398 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T12:45:35.398 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T12:45:35.398 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T12:45:35.398 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.6", 2023-10-19T12:45:35.399 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQCAJDFlLPumCxAA/iy08t/2N9rXfn71KtbSQw==", 2023-10-19T12:45:35.399 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T12:45:35.399 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T12:45:35.399 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T12:45:35.399 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T12:45:35.400 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T12:45:35.400 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T12:45:35.400 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T12:45:35.400 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.7", 2023-10-19T12:45:35.400 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQCDJDFlHmrbCRAA+hpWygGIJydfU6cHWlfx/Q==", 2023-10-19T12:45:35.401 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T12:45:35.401 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T12:45:35.401 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T12:45:35.401 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T12:45:35.401 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T12:45:35.402 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T12:45:35.402 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T12:45:35.402 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "client.0", 2023-10-19T12:45:35.402 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFlg9yuExAATxiWDfS+p07G1F0pORbJEg==", 2023-10-19T12:45:35.402 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T12:45:35.403 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T12:45:35.403 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow r", 2023-10-19T12:45:35.403 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow rw", 2023-10-19T12:45:35.403 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow rwx" 2023-10-19T12:45:35.403 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T12:45:35.404 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T12:45:35.404 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T12:45:35.404 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "client.1", 2023-10-19T12:45:35.404 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFl9IwYGxAAHPituiJSPkHDuD0QyG4Hww==", 2023-10-19T12:45:35.404 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T12:45:35.405 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T12:45:35.405 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow r", 2023-10-19T12:45:35.405 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow rw", 2023-10-19T12:45:35.405 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow rwx" 2023-10-19T12:45:35.405 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T12:45:35.405 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T12:45:35.406 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T12:45:35.406 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "client.2", 2023-10-19T12:45:35.406 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFlMd+AFxAAS/UTESDt76GmrhhN2o/Fdw==", 2023-10-19T12:45:35.406 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T12:45:35.406 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T12:45:35.407 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow r", 2023-10-19T12:45:35.407 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow rw", 2023-10-19T12:45:35.407 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow rwx" 2023-10-19T12:45:35.407 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T12:45:35.407 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T12:45:35.408 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T12:45:35.408 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "client.3", 2023-10-19T12:45:35.408 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFlHZ3fIBAA42B7n8opLS5smXIwJyVYzw==", 2023-10-19T12:45:35.408 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T12:45:35.408 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T12:45:35.409 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow r", 2023-10-19T12:45:35.409 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow rw", 2023-10-19T12:45:35.409 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow rwx" 2023-10-19T12:45:35.409 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T12:45:35.409 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T12:45:35.410 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T12:45:35.410 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "client.admin", 2023-10-19T12:45:35.410 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBjJDFlBgasFxAAMA24j8KszcXMEpYRQDXj4g==", 2023-10-19T12:45:35.410 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T12:45:35.410 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow *", 2023-10-19T12:45:35.411 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow *", 2023-10-19T12:45:35.411 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow *", 2023-10-19T12:45:35.411 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T12:45:35.411 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T12:45:35.411 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T12:45:35.411 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T12:45:35.412 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "client.bootstrap-mds", 2023-10-19T12:45:35.412 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQCJJDFlaLRwChAAcgEGweApFyoWpu71tSGhRg==", 2023-10-19T12:45:35.412 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T12:45:35.412 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile bootstrap-mds" 2023-10-19T12:45:35.412 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T12:45:35.413 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T12:45:35.413 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T12:45:35.413 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "client.bootstrap-mgr", 2023-10-19T12:45:35.413 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQCJJDFlJr9wChAA9gzNuJJ+nQdNVMiKP87Uhg==", 2023-10-19T12:45:35.413 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T12:45:35.414 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile bootstrap-mgr" 2023-10-19T12:45:35.414 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T12:45:35.414 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T12:45:35.414 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T12:45:35.414 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "client.bootstrap-osd", 2023-10-19T12:45:35.415 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQCJJDFlM8hwChAAjgQt3yeIyKdf7RBZHNBghg==", 2023-10-19T12:45:35.415 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T12:45:35.415 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile bootstrap-osd" 2023-10-19T12:45:35.415 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T12:45:35.415 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T12:45:35.416 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T12:45:35.416 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "client.bootstrap-rbd", 2023-10-19T12:45:35.416 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQCJJDFlrM9wChAA+px3FR2Xk9pqeP2I782lag==", 2023-10-19T12:45:35.416 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T12:45:35.416 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile bootstrap-rbd" 2023-10-19T12:45:35.417 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T12:45:35.417 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T12:45:35.417 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T12:45:35.417 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "client.bootstrap-rbd-mirror", 2023-10-19T12:45:35.417 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQCJJDFla9dwChAAOAkf4P9JiTrt3bqbGTUC2A==", 2023-10-19T12:45:35.418 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T12:45:35.418 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2023-10-19T12:45:35.418 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T12:45:35.418 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T12:45:35.418 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T12:45:35.419 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "client.bootstrap-rgw", 2023-10-19T12:45:35.419 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQCJJDFlDd9wChAAq8CVjRYRJeuTMgBuNI/vDg==", 2023-10-19T12:45:35.419 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T12:45:35.419 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile bootstrap-rgw" 2023-10-19T12:45:35.419 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T12:45:35.420 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T12:45:35.420 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T12:45:35.420 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "mgr.x", 2023-10-19T12:45:35.420 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBjJDFllFBfMhAA5+KPXWmYZqcESmLObaUbBA==", 2023-10-19T12:45:35.420 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T12:45:35.420 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow *", 2023-10-19T12:45:35.421 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile mgr", 2023-10-19T12:45:35.421 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T12:45:35.421 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T12:45:35.421 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T12:45:35.421 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T12:45:35.421 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "mgr.y", 2023-10-19T12:45:35.422 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBjJDFlxRzJLRAAGjG58o7qXAHvIEIRYwl6QA==", 2023-10-19T12:45:35.422 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T12:45:35.422 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow *", 2023-10-19T12:45:35.422 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile mgr", 2023-10-19T12:45:35.422 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T12:45:35.423 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T12:45:35.423 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T12:45:35.423 INFO:teuthology.orchestra.run.smithi057.stdout: ] 2023-10-19T12:45:35.423 INFO:teuthology.orchestra.run.smithi057.stdout:} 2023-10-19T12:45:35.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.379+0000 7f4d49ec9640 1 -- 172.21.15.57:0/3641686614 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f4d3006ae40 msgr2=0x7f4d3006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:35.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.379+0000 7f4d49ec9640 1 --2- 172.21.15.57:0/3641686614 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f4d3006ae40 0x7f4d3006d300 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f4d38035000 tx=0x7f4d38004730 comp rx=0 tx=0).stop 2023-10-19T12:45:35.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.379+0000 7f4d49ec9640 1 -- 172.21.15.57:0/3641686614 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4d4406b840 msgr2=0x7f4d44151700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:35.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.379+0000 7f4d49ec9640 1 --2- 172.21.15.57:0/3641686614 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4d4406b840 0x7f4d44151700 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f4d3400c450 tx=0x7f4d3400c910 comp rx=0 tx=0).stop 2023-10-19T12:45:35.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.379+0000 7f4d49ec9640 1 -- 172.21.15.57:0/3641686614 shutdown_connections 2023-10-19T12:45:35.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.379+0000 7f4d49ec9640 1 --2- 172.21.15.57:0/3641686614 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f4d3006ae40 0x7f4d3006d300 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:35.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.379+0000 7f4d49ec9640 1 --2- 172.21.15.57:0/3641686614 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4d4415bb90 0x7f4d440adc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:35.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.379+0000 7f4d49ec9640 1 --2- 172.21.15.57:0/3641686614 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4d4415b7b0 0x7f4d44157dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:35.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.379+0000 7f4d49ec9640 1 --2- 172.21.15.57:0/3641686614 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4d4406b840 0x7f4d44151700 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:35.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.379+0000 7f4d49ec9640 1 -- 172.21.15.57:0/3641686614 >> 172.21.15.57:0/3641686614 conn(0x7f4d440b3f00 msgr2=0x7f4d4415c8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:35.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.379+0000 7f4d49ec9640 1 -- 172.21.15.57:0/3641686614 shutdown_connections 2023-10-19T12:45:35.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.379+0000 7f4d49ec9640 1 -- 172.21.15.57:0/3641686614 wait complete. 2023-10-19T12:45:35.427 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2023-10-19T12:45:35.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.534+0000 7f974ff5e640 1 -- 172.21.15.57:0/2507096763 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f974815b2a0 msgr2=0x7f97480d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:35.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.534+0000 7f974ff5e640 1 --2- 172.21.15.57:0/2507096763 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f974815b2a0 0x7f97480d1910 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f9738009830 tx=0x7f973802bd60 comp rx=0 tx=0).stop 2023-10-19T12:45:35.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.534+0000 7f974ff5e640 1 -- 172.21.15.57:0/2507096763 shutdown_connections 2023-10-19T12:45:35.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.534+0000 7f974ff5e640 1 --2- 172.21.15.57:0/2507096763 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f974815b2a0 0x7f97480d1910 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:35.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.534+0000 7f974ff5e640 1 -- 172.21.15.57:0/2507096763 >> 172.21.15.57:0/2507096763 conn(0x7f97480d0d60 msgr2=0x7f97480d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:35.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.534+0000 7f974ff5e640 1 -- 172.21.15.57:0/2507096763 shutdown_connections 2023-10-19T12:45:35.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.534+0000 7f974ff5e640 1 -- 172.21.15.57:0/2507096763 wait complete. 2023-10-19T12:45:35.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.535+0000 7f974ff5e640 1 Processor -- start 2023-10-19T12:45:35.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.535+0000 7f974ff5e640 1 -- start start 2023-10-19T12:45:35.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.536+0000 7f974ff5e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f97480ceb40 0x7f9748151200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:35.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.536+0000 7f974ff5e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f97480cf340 0x7f97480b38e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:35.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.536+0000 7f974ff5e640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f974815b2a0 0x7f97480b0ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:35.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.536+0000 7f974ff5e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9748151740 con 0x7f97480cf340 2023-10-19T12:45:35.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.536+0000 7f974ff5e640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f974815d370 con 0x7f97480ceb40 2023-10-19T12:45:35.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.536+0000 7f974ff5e640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f974815c760 con 0x7f974815b2a0 2023-10-19T12:45:35.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.536+0000 7f974d4f9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f97480cf340 0x7f97480b38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:35.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.536+0000 7f974d4f9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f97480cf340 0x7f97480b38e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33552/0 (socket says 172.21.15.57:33552) 2023-10-19T12:45:35.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.536+0000 7f974d4f9640 1 -- 172.21.15.57:0/196500350 learned_addr learned my addr 172.21.15.57:0/196500350 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:35.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.536+0000 7f974e4fb640 1 --2- 172.21.15.57:0/196500350 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f974815b2a0 0x7f97480b0ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:35.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.536+0000 7f974dcfa640 1 --2- 172.21.15.57:0/196500350 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f97480ceb40 0x7f9748151200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:35.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.537+0000 7f974d4f9640 1 -- 172.21.15.57:0/196500350 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f974815b2a0 msgr2=0x7f97480b0ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:35.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.537+0000 7f974d4f9640 1 --2- 172.21.15.57:0/196500350 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f974815b2a0 0x7f97480b0ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:35.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.537+0000 7f974d4f9640 1 -- 172.21.15.57:0/196500350 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f97480ceb40 msgr2=0x7f9748151200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:35.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.537+0000 7f974d4f9640 1 --2- 172.21.15.57:0/196500350 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f97480ceb40 0x7f9748151200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:35.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.537+0000 7f974d4f9640 1 -- 172.21.15.57:0/196500350 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9738009490 con 0x7f97480cf340 2023-10-19T12:45:35.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.537+0000 7f974e4fb640 1 --2- 172.21.15.57:0/196500350 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f974815b2a0 0x7f97480b0ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:45:35.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.537+0000 7f974d4f9640 1 --2- 172.21.15.57:0/196500350 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f97480cf340 0x7f97480b38e0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f9738000c00 tx=0x7f97380038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:35.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.537+0000 7f973e7fc640 1 -- 172.21.15.57:0/196500350 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9738003ea0 con 0x7f97480cf340 2023-10-19T12:45:35.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.537+0000 7f973e7fc640 1 -- 172.21.15.57:0/196500350 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f9738034530 con 0x7f97480cf340 2023-10-19T12:45:35.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.537+0000 7f973e7fc640 1 -- 172.21.15.57:0/196500350 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9738034940 con 0x7f97480cf340 2023-10-19T12:45:35.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.537+0000 7f974ff5e640 1 -- 172.21.15.57:0/196500350 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f974815ca40 con 0x7f97480cf340 2023-10-19T12:45:35.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.537+0000 7f974ff5e640 1 -- 172.21.15.57:0/196500350 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f974815d000 con 0x7f97480cf340 2023-10-19T12:45:35.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.538+0000 7f974ff5e640 1 -- 172.21.15.57:0/196500350 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f974815d500 con 0x7f97480cf340 2023-10-19T12:45:35.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.538+0000 7f973e7fc640 1 -- 172.21.15.57:0/196500350 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f9738034aa0 con 0x7f97480cf340 2023-10-19T12:45:35.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.539+0000 7f973e7fc640 1 --2- 172.21.15.57:0/196500350 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f972c06ae40 0x7f972c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:35.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.539+0000 7f973e7fc640 1 -- 172.21.15.57:0/196500350 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f97380abd30 con 0x7f97480cf340 2023-10-19T12:45:35.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.539+0000 7f974dcfa640 1 --2- 172.21.15.57:0/196500350 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f972c06ae40 0x7f972c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:35.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.539+0000 7f974dcfa640 1 --2- 172.21.15.57:0/196500350 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f972c06ae40 0x7f972c06d300 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f97440046b0 tx=0x7f9744009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:35.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.543+0000 7f973e7fc640 1 -- 172.21.15.57:0/196500350 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9738077080 con 0x7f97480cf340 2023-10-19T12:45:35.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.784+0000 7f974ff5e640 1 -- 172.21.15.57:0/196500350 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) v1 -- 0x7f97480cca00 con 0x7f97480cf340 2023-10-19T12:45:35.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.787+0000 7f973e7fc640 1 -- 172.21.15.57:0/196500350 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v13) v1 ==== 90+0+0 (secure 0 0 0) 0x7f97380390a0 con 0x7f97480cf340 2023-10-19T12:45:35.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.789+0000 7f974ff5e640 1 -- 172.21.15.57:0/196500350 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f972c06ae40 msgr2=0x7f972c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:35.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.789+0000 7f974ff5e640 1 --2- 172.21.15.57:0/196500350 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f972c06ae40 0x7f972c06d300 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f97440046b0 tx=0x7f9744009040 comp rx=0 tx=0).stop 2023-10-19T12:45:35.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.789+0000 7f974ff5e640 1 -- 172.21.15.57:0/196500350 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f97480cf340 msgr2=0x7f97480b38e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:35.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.789+0000 7f974ff5e640 1 --2- 172.21.15.57:0/196500350 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f97480cf340 0x7f97480b38e0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f9738000c00 tx=0x7f97380038a0 comp rx=0 tx=0).stop 2023-10-19T12:45:35.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.789+0000 7f974ff5e640 1 -- 172.21.15.57:0/196500350 shutdown_connections 2023-10-19T12:45:35.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.789+0000 7f974ff5e640 1 --2- 172.21.15.57:0/196500350 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f972c06ae40 0x7f972c06d300 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:35.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.789+0000 7f974ff5e640 1 --2- 172.21.15.57:0/196500350 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f974815b2a0 0x7f97480b0ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:35.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.789+0000 7f974ff5e640 1 --2- 172.21.15.57:0/196500350 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f97480cf340 0x7f97480b38e0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:35.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.789+0000 7f974ff5e640 1 --2- 172.21.15.57:0/196500350 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f97480ceb40 0x7f9748151200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:35.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.789+0000 7f974ff5e640 1 -- 172.21.15.57:0/196500350 >> 172.21.15.57:0/196500350 conn(0x7f97480d0d60 msgr2=0x7f97480b0630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:35.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.789+0000 7f974ff5e640 1 -- 172.21.15.57:0/196500350 shutdown_connections 2023-10-19T12:45:35.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.789+0000 7f974ff5e640 1 -- 172.21.15.57:0/196500350 wait complete. 2023-10-19T12:45:35.803 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2023-10-19T12:45:35.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.913+0000 7feab79ce640 1 -- 172.21.15.57:0/1514911984 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fea9c0058d0 msgr2=0x7fea9c005d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:35.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.913+0000 7feab79ce640 1 --2- 172.21.15.57:0/1514911984 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fea9c0058d0 0x7fea9c005d50 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7feaa000c260 tx=0x7fea9c006900 comp rx=0 tx=0).stop 2023-10-19T12:45:35.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.913+0000 7feab5f6b640 1 -- 172.21.15.57:0/1514911984 reap_dead start 2023-10-19T12:45:35.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.913+0000 7feab79ce640 1 -- 172.21.15.57:0/1514911984 shutdown_connections 2023-10-19T12:45:35.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.913+0000 7feab79ce640 1 -- 172.21.15.57:0/1514911984 >> 172.21.15.57:0/1514911984 conn(0x7feab0069930 msgr2=0x7feab0069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:35.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.913+0000 7feab79ce640 1 -- 172.21.15.57:0/1514911984 shutdown_connections 2023-10-19T12:45:35.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.913+0000 7feab79ce640 1 -- 172.21.15.57:0/1514911984 wait complete. 2023-10-19T12:45:35.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.914+0000 7feab79ce640 1 Processor -- start 2023-10-19T12:45:35.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.914+0000 7feab79ce640 1 -- start start 2023-10-19T12:45:35.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.915+0000 7feab79ce640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fea9c0058d0 0x7feab01ecfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:35.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.915+0000 7feab79ce640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7feab006b4b0 0x7feab00c1990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:35.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.915+0000 7feab79ce640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feab00c1ed0 0x7feab0141e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:35.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.915+0000 7feab79ce640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feab0154370 con 0x7feab00c1ed0 2023-10-19T12:45:35.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.915+0000 7feab79ce640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feab0155190 con 0x7fea9c0058d0 2023-10-19T12:45:35.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.915+0000 7feab79ce640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feab0154580 con 0x7feab006b4b0 2023-10-19T12:45:35.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.915+0000 7feab5f6b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feab00c1ed0 0x7feab0141e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:35.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.915+0000 7feab5f6b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feab00c1ed0 0x7feab0141e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33570/0 (socket says 172.21.15.57:33570) 2023-10-19T12:45:35.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.915+0000 7feab5f6b640 1 -- 172.21.15.57:0/2028443975 learned_addr learned my addr 172.21.15.57:0/2028443975 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:35.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.915+0000 7feab576a640 1 --2- 172.21.15.57:0/2028443975 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fea9c0058d0 0x7feab01ecfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:35.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.915+0000 7feab5f6b640 1 -- 172.21.15.57:0/2028443975 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7feab006b4b0 msgr2=0x7feab00c1990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:35.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.915+0000 7feab5f6b640 1 --2- 172.21.15.57:0/2028443975 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7feab006b4b0 0x7feab00c1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:35.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.915+0000 7feab5f6b640 1 -- 172.21.15.57:0/2028443975 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fea9c0058d0 msgr2=0x7feab01ecfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:35.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.915+0000 7feab5f6b640 1 --2- 172.21.15.57:0/2028443975 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fea9c0058d0 0x7feab01ecfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:35.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.915+0000 7feab5f6b640 1 -- 172.21.15.57:0/2028443975 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feab0154860 con 0x7feab00c1ed0 2023-10-19T12:45:35.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.915+0000 7feab576a640 1 --2- 172.21.15.57:0/2028443975 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fea9c0058d0 0x7feab01ecfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:45:35.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.915+0000 7feab5f6b640 1 --2- 172.21.15.57:0/2028443975 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feab00c1ed0 0x7feab0141e80 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7feab006f750 tx=0x7feaa8010f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:35.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.915+0000 7feaa5ffb640 1 -- 172.21.15.57:0/2028443975 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaa8003970 con 0x7feab00c1ed0 2023-10-19T12:45:35.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.915+0000 7feaa5ffb640 1 -- 172.21.15.57:0/2028443975 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7feaa8003bc0 con 0x7feab00c1ed0 2023-10-19T12:45:35.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.916+0000 7feaa5ffb640 1 -- 172.21.15.57:0/2028443975 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaa8004030 con 0x7feab00c1ed0 2023-10-19T12:45:35.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.916+0000 7feab79ce640 1 -- 172.21.15.57:0/2028443975 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feab0154b40 con 0x7feab00c1ed0 2023-10-19T12:45:35.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.916+0000 7feab79ce640 1 -- 172.21.15.57:0/2028443975 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feab014f9d0 con 0x7feab00c1ed0 2023-10-19T12:45:35.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.917+0000 7feab79ce640 1 -- 172.21.15.57:0/2028443975 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feab0155300 con 0x7feab00c1ed0 2023-10-19T12:45:35.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.917+0000 7feaa5ffb640 1 -- 172.21.15.57:0/2028443975 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7feaa8019030 con 0x7feab00c1ed0 2023-10-19T12:45:35.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.918+0000 7feaa5ffb640 1 --2- 172.21.15.57:0/2028443975 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fea9406ae40 0x7fea9406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:35.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.918+0000 7feaa5ffb640 1 -- 172.21.15.57:0/2028443975 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5177+0+0 (secure 0 0 0) 0x7feaa8017070 con 0x7feab00c1ed0 2023-10-19T12:45:35.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.918+0000 7feab576a640 1 --2- 172.21.15.57:0/2028443975 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fea9406ae40 0x7fea9406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:35.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.918+0000 7feab576a640 1 --2- 172.21.15.57:0/2028443975 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fea9406ae40 0x7fea9406d300 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7feaac005bc0 tx=0x7feaac005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:35.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:35.922+0000 7feaa5ffb640 1 -- 172.21.15.57:0/2028443975 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7feaa8055630 con 0x7feab00c1ed0 2023-10-19T12:45:36.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.168+0000 7feab79ce640 1 -- 172.21.15.57:0/2028443975 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) v1 -- 0x7feab00690e0 con 0x7feab00c1ed0 2023-10-19T12:45:36.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.175+0000 7feaa5ffb640 1 -- 172.21.15.57:0/2028443975 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v14) v1 ==== 90+0+0 (secure 0 0 0) 0x7feaa8058c50 con 0x7feab00c1ed0 2023-10-19T12:45:36.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.177+0000 7feab79ce640 1 -- 172.21.15.57:0/2028443975 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fea9406ae40 msgr2=0x7fea9406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:36.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.177+0000 7feab79ce640 1 --2- 172.21.15.57:0/2028443975 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fea9406ae40 0x7fea9406d300 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7feaac005bc0 tx=0x7feaac005f90 comp rx=0 tx=0).stop 2023-10-19T12:45:36.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.177+0000 7feab79ce640 1 -- 172.21.15.57:0/2028443975 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feab00c1ed0 msgr2=0x7feab0141e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:36.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.177+0000 7feab79ce640 1 --2- 172.21.15.57:0/2028443975 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feab00c1ed0 0x7feab0141e80 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7feab006f750 tx=0x7feaa8010f40 comp rx=0 tx=0).stop 2023-10-19T12:45:36.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.177+0000 7feab79ce640 1 -- 172.21.15.57:0/2028443975 shutdown_connections 2023-10-19T12:45:36.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.177+0000 7feab79ce640 1 --2- 172.21.15.57:0/2028443975 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fea9406ae40 0x7fea9406d300 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:36.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.177+0000 7feab79ce640 1 --2- 172.21.15.57:0/2028443975 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feab00c1ed0 0x7feab0141e80 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:36.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.177+0000 7feab79ce640 1 --2- 172.21.15.57:0/2028443975 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7feab006b4b0 0x7feab00c1990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:36.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.177+0000 7feab79ce640 1 --2- 172.21.15.57:0/2028443975 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fea9c0058d0 0x7feab01ecfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:36.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.177+0000 7feab79ce640 1 -- 172.21.15.57:0/2028443975 >> 172.21.15.57:0/2028443975 conn(0x7feab0069930 msgr2=0x7feab006c020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:36.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.177+0000 7feab79ce640 1 -- 172.21.15.57:0/2028443975 shutdown_connections 2023-10-19T12:45:36.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.177+0000 7feab79ce640 1 -- 172.21.15.57:0/2028443975 wait complete. 2023-10-19T12:45:36.190 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2023-10-19T12:45:36.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.298+0000 7f2880b8a640 1 -- 172.21.15.57:0/857079066 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f287c0c96b0 msgr2=0x7f287c0c9b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:36.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.298+0000 7f2880b8a640 1 --2- 172.21.15.57:0/857079066 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f287c0c96b0 0x7f287c0c9b30 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f2870009860 tx=0x7f287002bd60 comp rx=0 tx=0).stop 2023-10-19T12:45:36.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.298+0000 7f2880b8a640 1 -- 172.21.15.57:0/857079066 shutdown_connections 2023-10-19T12:45:36.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.298+0000 7f2880b8a640 1 --2- 172.21.15.57:0/857079066 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f287c0c96b0 0x7f287c0c9b30 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:36.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.298+0000 7f2880b8a640 1 --2- 172.21.15.57:0/857079066 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f287c06af00 0x7f287c06b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:36.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.298+0000 7f2880b8a640 1 -- 172.21.15.57:0/857079066 >> 172.21.15.57:0/857079066 conn(0x7f287c069930 msgr2=0x7f287c069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:36.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.298+0000 7f2880b8a640 1 -- 172.21.15.57:0/857079066 shutdown_connections 2023-10-19T12:45:36.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.298+0000 7f2880b8a640 1 -- 172.21.15.57:0/857079066 wait complete. 2023-10-19T12:45:36.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.299+0000 7f2880b8a640 1 Processor -- start 2023-10-19T12:45:36.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.299+0000 7f2880b8a640 1 -- start start 2023-10-19T12:45:36.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.300+0000 7f2880b8a640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f287c06af00 0x7f287c1f5570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:36.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.300+0000 7f2880b8a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f287c0ab500 0x7f287c158b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:36.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.300+0000 7f2880b8a640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f287c0c96b0 0x7f287c13b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:36.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.300+0000 7f2880b8a640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f287c06f290 con 0x7f287c0ab500 2023-10-19T12:45:36.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.300+0000 7f2880b8a640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f287c0ab310 con 0x7f287c06af00 2023-10-19T12:45:36.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.300+0000 7f2880b8a640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f287c151350 con 0x7f287c0c96b0 2023-10-19T12:45:36.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.300+0000 7f2879d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f287c0ab500 0x7f287c158b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:36.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.300+0000 7f2879d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f287c0ab500 0x7f287c158b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33594/0 (socket says 172.21.15.57:33594) 2023-10-19T12:45:36.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.300+0000 7f2879d9b640 1 -- 172.21.15.57:0/23837604 learned_addr learned my addr 172.21.15.57:0/23837604 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:36.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.300+0000 7f287ad9d640 1 --2- 172.21.15.57:0/23837604 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f287c0c96b0 0x7f287c13b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:36.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.300+0000 7f287a59c640 1 --2- 172.21.15.57:0/23837604 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f287c06af00 0x7f287c1f5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:36.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.300+0000 7f2879d9b640 1 -- 172.21.15.57:0/23837604 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f287c0c96b0 msgr2=0x7f287c13b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:36.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.300+0000 7f2879d9b640 1 --2- 172.21.15.57:0/23837604 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f287c0c96b0 0x7f287c13b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:36.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.300+0000 7f2879d9b640 1 -- 172.21.15.57:0/23837604 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f287c06af00 msgr2=0x7f287c1f5570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:36.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.300+0000 7f2879d9b640 1 --2- 172.21.15.57:0/23837604 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f287c06af00 0x7f287c1f5570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:36.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.300+0000 7f2879d9b640 1 -- 172.21.15.57:0/23837604 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2870009490 con 0x7f287c0ab500 2023-10-19T12:45:36.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.300+0000 7f2879d9b640 1 --2- 172.21.15.57:0/23837604 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f287c0ab500 0x7f287c158b60 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f2870009950 tx=0x7f28700038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:36.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.300+0000 7f286affd640 1 -- 172.21.15.57:0/23837604 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2870003ea0 con 0x7f287c0ab500 2023-10-19T12:45:36.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.301+0000 7f286affd640 1 -- 172.21.15.57:0/23837604 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f2870005110 con 0x7f287c0ab500 2023-10-19T12:45:36.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.301+0000 7f2880b8a640 1 -- 172.21.15.57:0/23837604 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f287c0aad50 con 0x7f287c0ab500 2023-10-19T12:45:36.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.301+0000 7f286affd640 1 -- 172.21.15.57:0/23837604 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2870005520 con 0x7f287c0ab500 2023-10-19T12:45:36.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.301+0000 7f2880b8a640 1 -- 172.21.15.57:0/23837604 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f287c13b710 con 0x7f287c0ab500 2023-10-19T12:45:36.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.301+0000 7f2880b8a640 1 -- 172.21.15.57:0/23837604 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f287c06f3d0 con 0x7f287c0ab500 2023-10-19T12:45:36.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.302+0000 7f286affd640 1 -- 172.21.15.57:0/23837604 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f28700047a0 con 0x7f287c0ab500 2023-10-19T12:45:36.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.302+0000 7f286affd640 1 --2- 172.21.15.57:0/23837604 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f285c06ae40 0x7f285c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:36.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.302+0000 7f286affd640 1 -- 172.21.15.57:0/23837604 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f28700abd60 con 0x7f287c0ab500 2023-10-19T12:45:36.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.303+0000 7f287a59c640 1 --2- 172.21.15.57:0/23837604 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f285c06ae40 0x7f285c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:36.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.303+0000 7f287a59c640 1 --2- 172.21.15.57:0/23837604 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f285c06ae40 0x7f285c06d300 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f286c009f20 tx=0x7f286c0094f0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:36.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.306+0000 7f286affd640 1 -- 172.21.15.57:0/23837604 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2870076fa0 con 0x7f287c0ab500 2023-10-19T12:45:36.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.550+0000 7f2880b8a640 1 -- 172.21.15.57:0/23837604 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) v1 -- 0x7f287c0690e0 con 0x7f287c0ab500 2023-10-19T12:45:36.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.552+0000 7f286affd640 1 -- 172.21.15.57:0/23837604 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v15) v1 ==== 90+0+0 (secure 0 0 0) 0x7f28700390a0 con 0x7f287c0ab500 2023-10-19T12:45:36.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.554+0000 7f2880b8a640 1 -- 172.21.15.57:0/23837604 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f285c06ae40 msgr2=0x7f285c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:36.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.554+0000 7f2880b8a640 1 --2- 172.21.15.57:0/23837604 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f285c06ae40 0x7f285c06d300 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f286c009f20 tx=0x7f286c0094f0 comp rx=0 tx=0).stop 2023-10-19T12:45:36.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.554+0000 7f2880b8a640 1 -- 172.21.15.57:0/23837604 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f287c0ab500 msgr2=0x7f287c158b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:36.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.554+0000 7f2880b8a640 1 --2- 172.21.15.57:0/23837604 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f287c0ab500 0x7f287c158b60 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f2870009950 tx=0x7f28700038a0 comp rx=0 tx=0).stop 2023-10-19T12:45:36.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.554+0000 7f2880b8a640 1 -- 172.21.15.57:0/23837604 shutdown_connections 2023-10-19T12:45:36.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.554+0000 7f2880b8a640 1 --2- 172.21.15.57:0/23837604 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f285c06ae40 0x7f285c06d300 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:36.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.554+0000 7f2880b8a640 1 --2- 172.21.15.57:0/23837604 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f287c0c96b0 0x7f287c13b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:36.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.554+0000 7f2880b8a640 1 --2- 172.21.15.57:0/23837604 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f287c0ab500 0x7f287c158b60 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:36.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.554+0000 7f2880b8a640 1 --2- 172.21.15.57:0/23837604 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f287c06af00 0x7f287c1f5570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:36.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.554+0000 7f2880b8a640 1 -- 172.21.15.57:0/23837604 >> 172.21.15.57:0/23837604 conn(0x7f287c069930 msgr2=0x7f287c0ab990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:36.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.554+0000 7f2880b8a640 1 -- 172.21.15.57:0/23837604 shutdown_connections 2023-10-19T12:45:36.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.554+0000 7f2880b8a640 1 -- 172.21.15.57:0/23837604 wait complete. 2023-10-19T12:45:36.567 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2023-10-19T12:45:36.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.674+0000 7f739dc4b640 1 -- 172.21.15.57:0/3083967441 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f73980c99e0 msgr2=0x7f7398152c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:36.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.674+0000 7f739dc4b640 1 --2- 172.21.15.57:0/3083967441 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f73980c99e0 0x7f7398152c90 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f7388009830 tx=0x7f738802bcd0 comp rx=0 tx=0).stop 2023-10-19T12:45:36.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.674+0000 7f739dc4b640 1 -- 172.21.15.57:0/3083967441 shutdown_connections 2023-10-19T12:45:36.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.674+0000 7f739dc4b640 1 --2- 172.21.15.57:0/3083967441 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f73980c99e0 0x7f7398152c90 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:36.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.674+0000 7f739dc4b640 1 -- 172.21.15.57:0/3083967441 >> 172.21.15.57:0/3083967441 conn(0x7f7398069930 msgr2=0x7f7398069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:36.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.674+0000 7f739dc4b640 1 -- 172.21.15.57:0/3083967441 shutdown_connections 2023-10-19T12:45:36.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.674+0000 7f739dc4b640 1 -- 172.21.15.57:0/3083967441 wait complete. 2023-10-19T12:45:36.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.675+0000 7f739dc4b640 1 Processor -- start 2023-10-19T12:45:36.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.676+0000 7f739dc4b640 1 -- start start 2023-10-19T12:45:36.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.676+0000 7f739dc4b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f73980abc10 0x7f739813b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:36.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.676+0000 7f739dc4b640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f73980c99e0 0x7f7398070a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:36.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.676+0000 7f739dc4b640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f73980c9dc0 0x7f73981554f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:36.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.676+0000 7f739dc4b640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73981f1830 con 0x7f73980abc10 2023-10-19T12:45:36.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.676+0000 7f739dc4b640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73980ab6d0 con 0x7f73980c9dc0 2023-10-19T12:45:36.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.676+0000 7f739dc4b640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73980aae30 con 0x7f73980c99e0 2023-10-19T12:45:36.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.676+0000 7f73977fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f73980abc10 0x7f739813b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:36.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.676+0000 7f73977fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f73980abc10 0x7f739813b1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59066/0 (socket says 172.21.15.57:59066) 2023-10-19T12:45:36.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.677+0000 7f73977fe640 1 -- 172.21.15.57:0/1254428051 learned_addr learned my addr 172.21.15.57:0/1254428051 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:36.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.677+0000 7f7397fff640 1 --2- 172.21.15.57:0/1254428051 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f73980c9dc0 0x7f73981554f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:36.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.677+0000 7f7396ffd640 1 --2- 172.21.15.57:0/1254428051 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f73980c99e0 0x7f7398070a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:36.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.677+0000 7f73977fe640 1 -- 172.21.15.57:0/1254428051 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f73980c99e0 msgr2=0x7f7398070a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:36.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.677+0000 7f73977fe640 1 --2- 172.21.15.57:0/1254428051 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f73980c99e0 0x7f7398070a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:36.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.677+0000 7f73977fe640 1 -- 172.21.15.57:0/1254428051 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f73980c9dc0 msgr2=0x7f73981554f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:36.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.677+0000 7f73977fe640 1 --2- 172.21.15.57:0/1254428051 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f73980c9dc0 0x7f73981554f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:36.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.677+0000 7f73977fe640 1 -- 172.21.15.57:0/1254428051 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7388009490 con 0x7f73980abc10 2023-10-19T12:45:36.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.677+0000 7f7397fff640 1 --2- 172.21.15.57:0/1254428051 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f73980c9dc0 0x7f73981554f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:45:36.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.677+0000 7f7396ffd640 1 --2- 172.21.15.57:0/1254428051 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f73980c99e0 0x7f7398070a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:45:36.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.677+0000 7f73977fe640 1 --2- 172.21.15.57:0/1254428051 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f73980abc10 0x7f739813b1d0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f7388000c00 tx=0x7f738802de00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:36.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.677+0000 7f737ffff640 1 -- 172.21.15.57:0/1254428051 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7388003a00 con 0x7f73980abc10 2023-10-19T12:45:36.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.677+0000 7f739dc4b640 1 -- 172.21.15.57:0/1254428051 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f739815d040 con 0x7f73980abc10 2023-10-19T12:45:36.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.677+0000 7f737ffff640 1 -- 172.21.15.57:0/1254428051 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f7388003c50 con 0x7f73980abc10 2023-10-19T12:45:36.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.677+0000 7f737ffff640 1 -- 172.21.15.57:0/1254428051 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7388034440 con 0x7f73980abc10 2023-10-19T12:45:36.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.677+0000 7f739dc4b640 1 -- 172.21.15.57:0/1254428051 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73980ab3f0 con 0x7f73980abc10 2023-10-19T12:45:36.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.678+0000 7f739dc4b640 1 -- 172.21.15.57:0/1254428051 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f739815ab30 con 0x7f73980abc10 2023-10-19T12:45:36.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.679+0000 7f737ffff640 1 -- 172.21.15.57:0/1254428051 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f73880345a0 con 0x7f73980abc10 2023-10-19T12:45:36.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.679+0000 7f737ffff640 1 --2- 172.21.15.57:0/1254428051 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f738406ae40 0x7f738406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:36.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.679+0000 7f737ffff640 1 -- 172.21.15.57:0/1254428051 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f73880ab9b0 con 0x7f73980abc10 2023-10-19T12:45:36.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.679+0000 7f7396ffd640 1 --2- 172.21.15.57:0/1254428051 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f738406ae40 0x7f738406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:36.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.680+0000 7f7396ffd640 1 --2- 172.21.15.57:0/1254428051 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f738406ae40 0x7f738406d300 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f738c0011e0 tx=0x7f738c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:36.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.683+0000 7f737ffff640 1 -- 172.21.15.57:0/1254428051 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7388076bf0 con 0x7f73980abc10 2023-10-19T12:45:36.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.926+0000 7f739dc4b640 1 -- 172.21.15.57:0/1254428051 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) v1 -- 0x7f73980690e0 con 0x7f73980abc10 2023-10-19T12:45:36.935 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.932+0000 7f737ffff640 1 -- 172.21.15.57:0/1254428051 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v16) v1 ==== 90+0+0 (secure 0 0 0) 0x7f738807a210 con 0x7f73980abc10 2023-10-19T12:45:36.937 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.934+0000 7f739dc4b640 1 -- 172.21.15.57:0/1254428051 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f738406ae40 msgr2=0x7f738406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:36.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.934+0000 7f739dc4b640 1 --2- 172.21.15.57:0/1254428051 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f738406ae40 0x7f738406d300 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f738c0011e0 tx=0x7f738c009040 comp rx=0 tx=0).stop 2023-10-19T12:45:36.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.934+0000 7f739dc4b640 1 -- 172.21.15.57:0/1254428051 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f73980abc10 msgr2=0x7f739813b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:36.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.934+0000 7f739dc4b640 1 --2- 172.21.15.57:0/1254428051 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f73980abc10 0x7f739813b1d0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f7388000c00 tx=0x7f738802de00 comp rx=0 tx=0).stop 2023-10-19T12:45:36.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.934+0000 7f739dc4b640 1 -- 172.21.15.57:0/1254428051 shutdown_connections 2023-10-19T12:45:36.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.934+0000 7f739dc4b640 1 --2- 172.21.15.57:0/1254428051 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f738406ae40 0x7f738406d300 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:36.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.934+0000 7f739dc4b640 1 --2- 172.21.15.57:0/1254428051 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f73980c9dc0 0x7f73981554f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:36.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.934+0000 7f739dc4b640 1 --2- 172.21.15.57:0/1254428051 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f73980c99e0 0x7f7398070a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:36.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.934+0000 7f739dc4b640 1 --2- 172.21.15.57:0/1254428051 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f73980abc10 0x7f739813b1d0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:36.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.934+0000 7f739dc4b640 1 -- 172.21.15.57:0/1254428051 >> 172.21.15.57:0/1254428051 conn(0x7f7398069930 msgr2=0x7f739815bba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:36.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.934+0000 7f739dc4b640 1 -- 172.21.15.57:0/1254428051 shutdown_connections 2023-10-19T12:45:36.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:36.934+0000 7f739dc4b640 1 -- 172.21.15.57:0/1254428051 wait complete. 2023-10-19T12:45:36.948 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2023-10-19T12:45:37.058 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.055+0000 7f553f155640 1 -- 172.21.15.57:0/286697166 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f55380ceb40 msgr2=0x7f55380b0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:37.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.055+0000 7f553f155640 1 --2- 172.21.15.57:0/286697166 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f55380ceb40 0x7f55380b0650 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f553806c510 tx=0x7f552002bd10 comp rx=0 tx=0).stop 2023-10-19T12:45:37.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.055+0000 7f553f155640 1 -- 172.21.15.57:0/286697166 shutdown_connections 2023-10-19T12:45:37.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.055+0000 7f553f155640 1 --2- 172.21.15.57:0/286697166 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f55380ceb40 0x7f55380b0650 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:37.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.055+0000 7f553f155640 1 --2- 172.21.15.57:0/286697166 >> v2:172.21.15.179:3300/0 conn(0x7f553815b2a0 0x7f55380cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:37.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.055+0000 7f553f155640 1 --2- 172.21.15.57:0/286697166 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f55380cf340 0x7f55380d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:37.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.055+0000 7f553f155640 1 -- 172.21.15.57:0/286697166 >> 172.21.15.57:0/286697166 conn(0x7f55380d0d60 msgr2=0x7f55380d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:37.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.055+0000 7f553f155640 1 -- 172.21.15.57:0/286697166 shutdown_connections 2023-10-19T12:45:37.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.055+0000 7f553f155640 1 -- 172.21.15.57:0/286697166 wait complete. 2023-10-19T12:45:37.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.056+0000 7f553f155640 1 Processor -- start 2023-10-19T12:45:37.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.056+0000 7f553f155640 1 -- start start 2023-10-19T12:45:37.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.057+0000 7f553f155640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f55380ceb40 0x7f55381597d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:37.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.057+0000 7f553f155640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f55380cf340 0x7f55380ca6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:37.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.057+0000 7f553f155640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f553815b2a0 0x7f553809bd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:37.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.057+0000 7f553f155640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55380cb160 con 0x7f55380ceb40 2023-10-19T12:45:37.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.057+0000 7f553f155640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55380c8890 con 0x7f55380cf340 2023-10-19T12:45:37.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.057+0000 7f553f155640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55380c7f60 con 0x7f553815b2a0 2023-10-19T12:45:37.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.057+0000 7f553cef1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f55380ceb40 0x7f55381597d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:37.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.057+0000 7f553cef1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f55380ceb40 0x7f55381597d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59086/0 (socket says 172.21.15.57:59086) 2023-10-19T12:45:37.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.057+0000 7f553cef1640 1 -- 172.21.15.57:0/3575482413 learned_addr learned my addr 172.21.15.57:0/3575482413 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:37.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.058+0000 7f553cef1640 1 -- 172.21.15.57:0/3575482413 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f553815b2a0 msgr2=0x7f553809bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:37.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.058+0000 7f553cef1640 1 --2- 172.21.15.57:0/3575482413 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f553815b2a0 0x7f553809bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:37.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.058+0000 7f553cef1640 1 -- 172.21.15.57:0/3575482413 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f55380cf340 msgr2=0x7f55380ca6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:37.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.058+0000 7f553cef1640 1 --2- 172.21.15.57:0/3575482413 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f55380cf340 0x7f55380ca6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:37.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.058+0000 7f553cef1640 1 -- 172.21.15.57:0/3575482413 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5520009560 con 0x7f55380ceb40 2023-10-19T12:45:37.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.058+0000 7f553cef1640 1 --2- 172.21.15.57:0/3575482413 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f55380ceb40 0x7f55381597d0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f553000d780 tx=0x7f553000dc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:37.067 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.058+0000 7f55357fa640 1 -- 172.21.15.57:0/3575482413 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5530009bd0 con 0x7f55380ceb40 2023-10-19T12:45:37.067 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.058+0000 7f553f155640 1 -- 172.21.15.57:0/3575482413 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55380c8240 con 0x7f55380ceb40 2023-10-19T12:45:37.067 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.058+0000 7f55357fa640 1 -- 172.21.15.57:0/3575482413 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f5530009e20 con 0x7f55380ceb40 2023-10-19T12:45:37.067 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.058+0000 7f55357fa640 1 -- 172.21.15.57:0/3575482413 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5530005280 con 0x7f55380ceb40 2023-10-19T12:45:37.067 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.058+0000 7f553f155640 1 -- 172.21.15.57:0/3575482413 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f553815e530 con 0x7f55380ceb40 2023-10-19T12:45:37.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.059+0000 7f553f155640 1 -- 172.21.15.57:0/3575482413 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55380690e0 con 0x7f55380ceb40 2023-10-19T12:45:37.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.059+0000 7f55357fa640 1 -- 172.21.15.57:0/3575482413 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f55300053e0 con 0x7f55380ceb40 2023-10-19T12:45:37.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.060+0000 7f55357fa640 1 --2- 172.21.15.57:0/3575482413 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f551806ae40 0x7f551806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:37.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.060+0000 7f55357fa640 1 -- 172.21.15.57:0/3575482413 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f5530089610 con 0x7f55380ceb40 2023-10-19T12:45:37.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.060+0000 7f5537fff640 1 --2- 172.21.15.57:0/3575482413 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f551806ae40 0x7f551806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:37.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.060+0000 7f5537fff640 1 --2- 172.21.15.57:0/3575482413 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f551806ae40 0x7f551806d300 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f55280092f0 tx=0x7f5528009240 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:37.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.064+0000 7f55357fa640 1 -- 172.21.15.57:0/3575482413 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f55300548e0 con 0x7f55380ceb40 2023-10-19T12:45:37.315 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.312+0000 7f553f155640 1 -- 172.21.15.57:0/3575482413 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) v1 -- 0x7f55380d0b40 con 0x7f55380ceb40 2023-10-19T12:45:37.317 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.315+0000 7f55357fa640 1 -- 172.21.15.57:0/3575482413 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v17) v1 ==== 97+0+0 (secure 0 0 0) 0x7f5530089030 con 0x7f55380ceb40 2023-10-19T12:45:37.320 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.317+0000 7f553f155640 1 -- 172.21.15.57:0/3575482413 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f551806ae40 msgr2=0x7f551806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:37.320 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.317+0000 7f553f155640 1 --2- 172.21.15.57:0/3575482413 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f551806ae40 0x7f551806d300 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f55280092f0 tx=0x7f5528009240 comp rx=0 tx=0).stop 2023-10-19T12:45:37.320 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.317+0000 7f553f155640 1 -- 172.21.15.57:0/3575482413 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f55380ceb40 msgr2=0x7f55381597d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:37.321 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.317+0000 7f553f155640 1 --2- 172.21.15.57:0/3575482413 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f55380ceb40 0x7f55381597d0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f553000d780 tx=0x7f553000dc40 comp rx=0 tx=0).stop 2023-10-19T12:45:37.321 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.317+0000 7f553f155640 1 -- 172.21.15.57:0/3575482413 shutdown_connections 2023-10-19T12:45:37.321 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.317+0000 7f553f155640 1 --2- 172.21.15.57:0/3575482413 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f551806ae40 0x7f551806d300 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:37.321 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.317+0000 7f553f155640 1 --2- 172.21.15.57:0/3575482413 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f553815b2a0 0x7f553809bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:37.321 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.317+0000 7f553f155640 1 --2- 172.21.15.57:0/3575482413 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f55380cf340 0x7f55380ca6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:37.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.317+0000 7f553f155640 1 --2- 172.21.15.57:0/3575482413 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f55380ceb40 0x7f55381597d0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:37.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.317+0000 7f553f155640 1 -- 172.21.15.57:0/3575482413 >> 172.21.15.57:0/3575482413 conn(0x7f55380d0d60 msgr2=0x7f55380af120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:37.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.317+0000 7f553f155640 1 -- 172.21.15.57:0/3575482413 shutdown_connections 2023-10-19T12:45:37.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.317+0000 7f553f155640 1 -- 172.21.15.57:0/3575482413 wait complete. 2023-10-19T12:45:37.331 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2023-10-19T12:45:37.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.438+0000 7f215cb33640 1 -- 172.21.15.57:0/644448762 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f21580ceb40 msgr2=0x7f21580b0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:37.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.438+0000 7f215cb33640 1 --2- 172.21.15.57:0/644448762 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f21580ceb40 0x7f21580b0650 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2148005fd0 tx=0x7f214802c050 comp rx=0 tx=0).stop 2023-10-19T12:45:37.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.438+0000 7f215cb33640 1 -- 172.21.15.57:0/644448762 shutdown_connections 2023-10-19T12:45:37.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.438+0000 7f215cb33640 1 --2- 172.21.15.57:0/644448762 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f21580ceb40 0x7f21580b0650 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:37.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.438+0000 7f215cb33640 1 --2- 172.21.15.57:0/644448762 >> v2:172.21.15.179:3300/0 conn(0x7f21580cf340 0x7f21580d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:37.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.438+0000 7f215cb33640 1 -- 172.21.15.57:0/644448762 >> 172.21.15.57:0/644448762 conn(0x7f21580d0d60 msgr2=0x7f21580d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:37.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.438+0000 7f215cb33640 1 -- 172.21.15.57:0/644448762 shutdown_connections 2023-10-19T12:45:37.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.438+0000 7f215cb33640 1 -- 172.21.15.57:0/644448762 wait complete. 2023-10-19T12:45:37.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.439+0000 7f215cb33640 1 Processor -- start 2023-10-19T12:45:37.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.439+0000 7f215cb33640 1 -- start start 2023-10-19T12:45:37.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.440+0000 7f215cb33640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f21580ceb40 0x7f2158158500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:37.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.440+0000 7f215cb33640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f21580cf340 0x7f21580b5e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:37.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.440+0000 7f215cb33640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f215815b2a0 0x7f215806a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:37.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.440+0000 7f215cb33640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f215815bf00 con 0x7f215815b2a0 2023-10-19T12:45:37.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.440+0000 7f215cb33640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21580b3c50 con 0x7f21580ceb40 2023-10-19T12:45:37.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.440+0000 7f215cb33640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21580b3f30 con 0x7f21580cf340 2023-10-19T12:45:37.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.440+0000 7f2156d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f215815b2a0 0x7f215806a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:37.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.440+0000 7f2156d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f215815b2a0 0x7f215806a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59090/0 (socket says 172.21.15.57:59090) 2023-10-19T12:45:37.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.440+0000 7f2156d9d640 1 -- 172.21.15.57:0/3492519444 learned_addr learned my addr 172.21.15.57:0/3492519444 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:37.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.440+0000 7f2155d9b640 1 --2- 172.21.15.57:0/3492519444 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f21580cf340 0x7f21580b5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:37.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.440+0000 7f215659c640 1 --2- 172.21.15.57:0/3492519444 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f21580ceb40 0x7f2158158500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:37.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.440+0000 7f2156d9d640 1 -- 172.21.15.57:0/3492519444 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f21580cf340 msgr2=0x7f21580b5e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:37.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.440+0000 7f2156d9d640 1 --2- 172.21.15.57:0/3492519444 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f21580cf340 0x7f21580b5e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:37.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.440+0000 7f2156d9d640 1 -- 172.21.15.57:0/3492519444 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f21580ceb40 msgr2=0x7f2158158500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:37.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.440+0000 7f2156d9d640 1 --2- 172.21.15.57:0/3492519444 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f21580ceb40 0x7f2158158500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:37.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.440+0000 7f2156d9d640 1 -- 172.21.15.57:0/3492519444 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2150009560 con 0x7f215815b2a0 2023-10-19T12:45:37.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.440+0000 7f2156d9d640 1 --2- 172.21.15.57:0/3492519444 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f215815b2a0 0x7f215806a200 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f2148009cd0 tx=0x7f21480042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:37.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.441+0000 7f214effd640 1 -- 172.21.15.57:0/3492519444 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f214803e460 con 0x7f215815b2a0 2023-10-19T12:45:37.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.441+0000 7f215cb33640 1 -- 172.21.15.57:0/3492519444 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2148009840 con 0x7f215815b2a0 2023-10-19T12:45:37.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.441+0000 7f214effd640 1 -- 172.21.15.57:0/3492519444 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f214803e6b0 con 0x7f215815b2a0 2023-10-19T12:45:37.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.441+0000 7f214effd640 1 -- 172.21.15.57:0/3492519444 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f214803ea80 con 0x7f215815b2a0 2023-10-19T12:45:37.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.441+0000 7f215cb33640 1 -- 172.21.15.57:0/3492519444 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21580b0d50 con 0x7f215815b2a0 2023-10-19T12:45:37.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.442+0000 7f215cb33640 1 -- 172.21.15.57:0/3492519444 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f215815c100 con 0x7f215815b2a0 2023-10-19T12:45:37.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.442+0000 7f214effd640 1 -- 172.21.15.57:0/3492519444 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f214803ebe0 con 0x7f215815b2a0 2023-10-19T12:45:37.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.442+0000 7f214effd640 1 --2- 172.21.15.57:0/3492519444 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f213806ae70 0x7f213806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:37.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.443+0000 7f214effd640 1 -- 172.21.15.57:0/3492519444 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f21480ac330 con 0x7f215815b2a0 2023-10-19T12:45:37.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.443+0000 7f215659c640 1 --2- 172.21.15.57:0/3492519444 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f213806ae70 0x7f213806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:37.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.443+0000 7f215659c640 1 --2- 172.21.15.57:0/3492519444 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f213806ae70 0x7f213806d330 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f2150009530 tx=0x7f2150009260 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:37.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.447+0000 7f214effd640 1 -- 172.21.15.57:0/3492519444 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f215815c100 con 0x7f215815b2a0 2023-10-19T12:45:37.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.690+0000 7f215cb33640 1 -- 172.21.15.57:0/3492519444 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) v1 -- 0x7f215815c0c0 con 0x7f215815b2a0 2023-10-19T12:45:37.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.693+0000 7f214effd640 1 -- 172.21.15.57:0/3492519444 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v18) v1 ==== 90+0+0 (secure 0 0 0) 0x7f2148077620 con 0x7f215815b2a0 2023-10-19T12:45:37.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.695+0000 7f215cb33640 1 -- 172.21.15.57:0/3492519444 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f213806ae70 msgr2=0x7f213806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:37.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.695+0000 7f215cb33640 1 --2- 172.21.15.57:0/3492519444 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f213806ae70 0x7f213806d330 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f2150009530 tx=0x7f2150009260 comp rx=0 tx=0).stop 2023-10-19T12:45:37.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.695+0000 7f215cb33640 1 -- 172.21.15.57:0/3492519444 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f215815b2a0 msgr2=0x7f215806a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:37.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.695+0000 7f215cb33640 1 --2- 172.21.15.57:0/3492519444 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f215815b2a0 0x7f215806a200 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f2148009cd0 tx=0x7f21480042f0 comp rx=0 tx=0).stop 2023-10-19T12:45:37.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.695+0000 7f215cb33640 1 -- 172.21.15.57:0/3492519444 shutdown_connections 2023-10-19T12:45:37.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.695+0000 7f215cb33640 1 --2- 172.21.15.57:0/3492519444 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f213806ae70 0x7f213806d330 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:37.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.695+0000 7f215cb33640 1 --2- 172.21.15.57:0/3492519444 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f215815b2a0 0x7f215806a200 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:37.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.695+0000 7f215cb33640 1 --2- 172.21.15.57:0/3492519444 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f21580cf340 0x7f21580b5e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:37.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.695+0000 7f215cb33640 1 --2- 172.21.15.57:0/3492519444 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f21580ceb40 0x7f2158158500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:37.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.695+0000 7f215cb33640 1 -- 172.21.15.57:0/3492519444 >> 172.21.15.57:0/3492519444 conn(0x7f21580d0d60 msgr2=0x7f21580cf720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:37.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.695+0000 7f215cb33640 1 -- 172.21.15.57:0/3492519444 shutdown_connections 2023-10-19T12:45:37.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.695+0000 7f215cb33640 1 -- 172.21.15.57:0/3492519444 wait complete. 2023-10-19T12:45:37.709 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T12:45:37.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.815+0000 7fb56f451640 1 -- 172.21.15.57:0/4027111636 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb56806ae40 msgr2=0x7fb5680b0e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:37.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.815+0000 7fb56f451640 1 --2- 172.21.15.57:0/4027111636 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb56806ae40 0x7fb5680b0e90 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fb558009830 tx=0x7fb55802bd60 comp rx=0 tx=0).stop 2023-10-19T12:45:37.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.815+0000 7fb56f451640 1 -- 172.21.15.57:0/4027111636 shutdown_connections 2023-10-19T12:45:37.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.815+0000 7fb56f451640 1 --2- 172.21.15.57:0/4027111636 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb56806ae40 0x7fb5680b0e90 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:37.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.815+0000 7fb56f451640 1 -- 172.21.15.57:0/4027111636 >> 172.21.15.57:0/4027111636 conn(0x7fb56806fae0 msgr2=0x7fb56806fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:37.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.815+0000 7fb56f451640 1 -- 172.21.15.57:0/4027111636 shutdown_connections 2023-10-19T12:45:37.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.815+0000 7fb56f451640 1 -- 172.21.15.57:0/4027111636 wait complete. 2023-10-19T12:45:37.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.816+0000 7fb56f451640 1 Processor -- start 2023-10-19T12:45:37.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.816+0000 7fb56f451640 1 -- start start 2023-10-19T12:45:37.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.817+0000 7fb56f451640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb56806ae40 0x7fb568146620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:37.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.817+0000 7fb56f451640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb5681579f0 0x7fb5680ac1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:37.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.817+0000 7fb56f451640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb56815c1f0 0x7fb5680ac710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:37.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.817+0000 7fb56f451640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb56814fac0 con 0x7fb56815c1f0 2023-10-19T12:45:37.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.817+0000 7fb56f451640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5680acd60 con 0x7fb56806ae40 2023-10-19T12:45:37.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.817+0000 7fb56f451640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5680adbc0 con 0x7fb5681579f0 2023-10-19T12:45:37.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.817+0000 7fb56d9ee640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb56815c1f0 0x7fb5680ac710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:37.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.817+0000 7fb56d9ee640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb56815c1f0 0x7fb5680ac710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59108/0 (socket says 172.21.15.57:59108) 2023-10-19T12:45:37.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.817+0000 7fb56d9ee640 1 -- 172.21.15.57:0/4226959517 learned_addr learned my addr 172.21.15.57:0/4226959517 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:37.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.817+0000 7fb56d1ed640 1 --2- 172.21.15.57:0/4226959517 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb56806ae40 0x7fb568146620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:37.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.817+0000 7fb56c9ec640 1 --2- 172.21.15.57:0/4226959517 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb5681579f0 0x7fb5680ac1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:37.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.817+0000 7fb56d9ee640 1 -- 172.21.15.57:0/4226959517 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb5681579f0 msgr2=0x7fb5680ac1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:37.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.817+0000 7fb56d9ee640 1 --2- 172.21.15.57:0/4226959517 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb5681579f0 0x7fb5680ac1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:37.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.817+0000 7fb56d9ee640 1 -- 172.21.15.57:0/4226959517 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb56806ae40 msgr2=0x7fb568146620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:37.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.817+0000 7fb56d9ee640 1 --2- 172.21.15.57:0/4226959517 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb56806ae40 0x7fb568146620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:37.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.817+0000 7fb56d9ee640 1 -- 172.21.15.57:0/4226959517 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb558009490 con 0x7fb56815c1f0 2023-10-19T12:45:37.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.817+0000 7fb56d9ee640 1 --2- 172.21.15.57:0/4226959517 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb56815c1f0 0x7fb5680ac710 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fb56000c740 tx=0x7fb56000cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:37.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.817+0000 7fb55dffb640 1 -- 172.21.15.57:0/4226959517 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb560003da0 con 0x7fb56815c1f0 2023-10-19T12:45:37.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.817+0000 7fb55dffb640 1 -- 172.21.15.57:0/4226959517 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fb56000ce50 con 0x7fb56815c1f0 2023-10-19T12:45:37.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.818+0000 7fb56f451640 1 -- 172.21.15.57:0/4226959517 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5680acfb0 con 0x7fb56815c1f0 2023-10-19T12:45:37.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.818+0000 7fb55dffb640 1 -- 172.21.15.57:0/4226959517 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5600052b0 con 0x7fb56815c1f0 2023-10-19T12:45:37.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.818+0000 7fb56f451640 1 -- 172.21.15.57:0/4226959517 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5680ad570 con 0x7fb56815c1f0 2023-10-19T12:45:37.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.819+0000 7fb56f451640 1 -- 172.21.15.57:0/4226959517 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5680690e0 con 0x7fb56815c1f0 2023-10-19T12:45:37.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.819+0000 7fb55dffb640 1 -- 172.21.15.57:0/4226959517 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fb560005410 con 0x7fb56815c1f0 2023-10-19T12:45:37.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.820+0000 7fb55dffb640 1 --2- 172.21.15.57:0/4226959517 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fb54c06ae40 0x7fb54c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:37.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.820+0000 7fb55dffb640 1 -- 172.21.15.57:0/4226959517 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fb560088dd0 con 0x7fb56815c1f0 2023-10-19T12:45:37.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.820+0000 7fb56d1ed640 1 --2- 172.21.15.57:0/4226959517 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fb54c06ae40 0x7fb54c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:37.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.820+0000 7fb56d1ed640 1 --2- 172.21.15.57:0/4226959517 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fb54c06ae40 0x7fb54c06d300 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fb568153bd0 tx=0x7fb564009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:37.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:37.824+0000 7fb55dffb640 1 -- 172.21.15.57:0/4226959517 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb560054140 con 0x7fb56815c1f0 2023-10-19T12:45:38.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.000+0000 7fb56f451640 1 -- 172.21.15.57:0/4226959517 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb568158df0 con 0x7fb56815c1f0 2023-10-19T12:45:38.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.001+0000 7fb55dffb640 1 -- 172.21.15.57:0/4226959517 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v35) v1 ==== 74+0+13074 (secure 0 0 0) 0x7fb56008a090 con 0x7fb56815c1f0 2023-10-19T12:45:38.004 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:45:38.005 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":35,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T12:45:34.999979+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T12:45:38.006 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.003+0000 7fb56f451640 1 -- 172.21.15.57:0/4226959517 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fb54c06ae40 msgr2=0x7fb54c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:38.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.003+0000 7fb56f451640 1 --2- 172.21.15.57:0/4226959517 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fb54c06ae40 0x7fb54c06d300 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fb568153bd0 tx=0x7fb564009040 comp rx=0 tx=0).stop 2023-10-19T12:45:38.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.003+0000 7fb56f451640 1 -- 172.21.15.57:0/4226959517 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb56815c1f0 msgr2=0x7fb5680ac710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:38.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.003+0000 7fb56f451640 1 --2- 172.21.15.57:0/4226959517 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb56815c1f0 0x7fb5680ac710 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fb56000c740 tx=0x7fb56000cc00 comp rx=0 tx=0).stop 2023-10-19T12:45:38.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.003+0000 7fb56f451640 1 -- 172.21.15.57:0/4226959517 shutdown_connections 2023-10-19T12:45:38.008 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.003+0000 7fb56f451640 1 --2- 172.21.15.57:0/4226959517 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fb54c06ae40 0x7fb54c06d300 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:38.008 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.003+0000 7fb56f451640 1 --2- 172.21.15.57:0/4226959517 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb56815c1f0 0x7fb5680ac710 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:38.008 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.003+0000 7fb56f451640 1 --2- 172.21.15.57:0/4226959517 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb5681579f0 0x7fb5680ac1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:38.008 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.003+0000 7fb56f451640 1 --2- 172.21.15.57:0/4226959517 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb56806ae40 0x7fb568146620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:38.008 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.003+0000 7fb56f451640 1 -- 172.21.15.57:0/4226959517 >> 172.21.15.57:0/4226959517 conn(0x7fb56806fae0 msgr2=0x7fb568069720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:38.008 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.003+0000 7fb56f451640 1 -- 172.21.15.57:0/4226959517 shutdown_connections 2023-10-19T12:45:38.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.003+0000 7fb56f451640 1 -- 172.21.15.57:0/4226959517 wait complete. 2023-10-19T12:45:38.017 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-19T12:44:07.371186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-19T12:45:38.018 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-19T12:45:38.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.127+0000 7f7bf73fc640 1 -- 172.21.15.57:0/1823176995 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7bf00ceb40 msgr2=0x7f7bf00b0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:38.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.127+0000 7f7bf73fc640 1 --2- 172.21.15.57:0/1823176995 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7bf00ceb40 0x7f7bf00b0650 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f7be0005fd0 tx=0x7f7be002c1b0 comp rx=0 tx=0).stop 2023-10-19T12:45:38.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.127+0000 7f7bf73fc640 1 -- 172.21.15.57:0/1823176995 shutdown_connections 2023-10-19T12:45:38.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.127+0000 7f7bf73fc640 1 --2- 172.21.15.57:0/1823176995 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7bf00ceb40 0x7f7bf00b0650 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:38.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.127+0000 7f7bf73fc640 1 --2- 172.21.15.57:0/1823176995 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7bf015b2a0 0x7f7bf00d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:38.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.127+0000 7f7bf73fc640 1 -- 172.21.15.57:0/1823176995 >> 172.21.15.57:0/1823176995 conn(0x7f7bf00d0d60 msgr2=0x7f7bf00d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:38.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.127+0000 7f7bf73fc640 1 -- 172.21.15.57:0/1823176995 shutdown_connections 2023-10-19T12:45:38.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.127+0000 7f7bf73fc640 1 -- 172.21.15.57:0/1823176995 wait complete. 2023-10-19T12:45:38.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.128+0000 7f7bf73fc640 1 Processor -- start 2023-10-19T12:45:38.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.129+0000 7f7bf73fc640 1 -- start start 2023-10-19T12:45:38.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.129+0000 7f7bf73fc640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7bf00ceb40 0x7f7bf0154950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:38.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.129+0000 7f7bf73fc640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7bf00cf340 0x7f7bf01588f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:38.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.129+0000 7f7bf73fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7bf015b2a0 0x7f7bf0158e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:38.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.129+0000 7f7bf73fc640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf00c80b0 con 0x7f7bf015b2a0 2023-10-19T12:45:38.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.129+0000 7f7bf73fc640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf00b4a50 con 0x7f7bf00ceb40 2023-10-19T12:45:38.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.129+0000 7f7bf73fc640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf00b4d30 con 0x7f7bf00cf340 2023-10-19T12:45:38.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.129+0000 7f7bf5999640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7bf015b2a0 0x7f7bf0158e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:38.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.129+0000 7f7bf5999640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7bf015b2a0 0x7f7bf0158e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59124/0 (socket says 172.21.15.57:59124) 2023-10-19T12:45:38.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.129+0000 7f7bf5999640 1 -- 172.21.15.57:0/3084452737 learned_addr learned my addr 172.21.15.57:0/3084452737 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:38.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.130+0000 7f7bf5198640 1 --2- 172.21.15.57:0/3084452737 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7bf00ceb40 0x7f7bf0154950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:38.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.130+0000 7f7bf4997640 1 --2- 172.21.15.57:0/3084452737 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7bf00cf340 0x7f7bf01588f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:38.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.130+0000 7f7bf5999640 1 -- 172.21.15.57:0/3084452737 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7bf00cf340 msgr2=0x7f7bf01588f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:38.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.130+0000 7f7bf5999640 1 --2- 172.21.15.57:0/3084452737 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7bf00cf340 0x7f7bf01588f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:38.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.130+0000 7f7bf5999640 1 -- 172.21.15.57:0/3084452737 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7bf00ceb40 msgr2=0x7f7bf0154950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:38.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.130+0000 7f7bf5999640 1 --2- 172.21.15.57:0/3084452737 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7bf00ceb40 0x7f7bf0154950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:38.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.130+0000 7f7bf5999640 1 -- 172.21.15.57:0/3084452737 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7be0009900 con 0x7f7bf015b2a0 2023-10-19T12:45:38.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.130+0000 7f7bf5198640 1 --2- 172.21.15.57:0/3084452737 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7bf00ceb40 0x7f7bf0154950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:45:38.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.130+0000 7f7bf4997640 1 --2- 172.21.15.57:0/3084452737 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7bf00cf340 0x7f7bf01588f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:45:38.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.130+0000 7f7bf5999640 1 --2- 172.21.15.57:0/3084452737 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7bf015b2a0 0x7f7bf0158e30 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f7be0009bf0 tx=0x7f7be0035e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:38.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.130+0000 7f7bedffb640 1 -- 172.21.15.57:0/3084452737 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7be0034440 con 0x7f7bf015b2a0 2023-10-19T12:45:38.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.130+0000 7f7bedffb640 1 -- 172.21.15.57:0/3084452737 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f7be0004540 con 0x7f7bf015b2a0 2023-10-19T12:45:38.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.130+0000 7f7bf73fc640 1 -- 172.21.15.57:0/3084452737 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bf00b5010 con 0x7f7bf015b2a0 2023-10-19T12:45:38.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.130+0000 7f7bedffb640 1 -- 172.21.15.57:0/3084452737 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7be0040450 con 0x7f7bf015b2a0 2023-10-19T12:45:38.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.130+0000 7f7bf73fc640 1 -- 172.21.15.57:0/3084452737 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bf0154e90 con 0x7f7bf015b2a0 2023-10-19T12:45:38.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.131+0000 7f7bf73fc640 1 -- 172.21.15.57:0/3084452737 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bf00c81f0 con 0x7f7bf015b2a0 2023-10-19T12:45:38.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.132+0000 7f7bedffb640 1 -- 172.21.15.57:0/3084452737 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f7be000a040 con 0x7f7bf015b2a0 2023-10-19T12:45:38.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.132+0000 7f7bedffb640 1 --2- 172.21.15.57:0/3084452737 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f7bdc06ae40 0x7f7bdc06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:38.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.132+0000 7f7bedffb640 1 -- 172.21.15.57:0/3084452737 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f7be00ac530 con 0x7f7bf015b2a0 2023-10-19T12:45:38.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.132+0000 7f7bf5198640 1 --2- 172.21.15.57:0/3084452737 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f7bdc06ae40 0x7f7bdc06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:38.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.133+0000 7f7bf5198640 1 --2- 172.21.15.57:0/3084452737 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f7bdc06ae40 0x7f7bdc06d300 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f7be8007fd0 tx=0x7f7be8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:38.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.137+0000 7f7bedffb640 1 -- 172.21.15.57:0/3084452737 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7bf00c81f0 con 0x7f7bf015b2a0 2023-10-19T12:45:38.315 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.312+0000 7f7bf73fc640 1 -- 172.21.15.57:0/3084452737 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7bf00c81f0 con 0x7f7bf015b2a0 2023-10-19T12:45:38.315 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.312+0000 7f7bedffb640 1 -- 172.21.15.57:0/3084452737 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v35) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7bf00c81f0 con 0x7f7bf015b2a0 2023-10-19T12:45:38.316 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-19T12:45:38.317 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.314+0000 7f7bf73fc640 1 -- 172.21.15.57:0/3084452737 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f7bdc06ae40 msgr2=0x7f7bdc06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:38.317 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.314+0000 7f7bf73fc640 1 --2- 172.21.15.57:0/3084452737 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f7bdc06ae40 0x7f7bdc06d300 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f7be8007fd0 tx=0x7f7be8009040 comp rx=0 tx=0).stop 2023-10-19T12:45:38.317 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.314+0000 7f7bf73fc640 1 -- 172.21.15.57:0/3084452737 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7bf015b2a0 msgr2=0x7f7bf0158e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:38.318 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.314+0000 7f7bf73fc640 1 --2- 172.21.15.57:0/3084452737 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7bf015b2a0 0x7f7bf0158e30 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f7be0009bf0 tx=0x7f7be0035e30 comp rx=0 tx=0).stop 2023-10-19T12:45:38.318 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.314+0000 7f7bf73fc640 1 -- 172.21.15.57:0/3084452737 shutdown_connections 2023-10-19T12:45:38.318 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.314+0000 7f7bf73fc640 1 --2- 172.21.15.57:0/3084452737 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f7bdc06ae40 0x7f7bdc06d300 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:38.318 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.314+0000 7f7bf73fc640 1 --2- 172.21.15.57:0/3084452737 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7bf015b2a0 0x7f7bf0158e30 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:38.318 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.314+0000 7f7bf73fc640 1 --2- 172.21.15.57:0/3084452737 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7bf00cf340 0x7f7bf01588f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:38.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.314+0000 7f7bf73fc640 1 --2- 172.21.15.57:0/3084452737 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7bf00ceb40 0x7f7bf0154950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:38.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.314+0000 7f7bf73fc640 1 -- 172.21.15.57:0/3084452737 >> 172.21.15.57:0/3084452737 conn(0x7f7bf00d0d60 msgr2=0x7f7bf006b3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:38.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.314+0000 7f7bf73fc640 1 -- 172.21.15.57:0/3084452737 shutdown_connections 2023-10-19T12:45:38.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.314+0000 7f7bf73fc640 1 -- 172.21.15.57:0/3084452737 wait complete. 2023-10-19T12:45:38.328 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-19T12:45:38.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.433+0000 7fa746b1d640 1 -- 172.21.15.57:0/4057572173 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa7400cf340 msgr2=0x7fa7400d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:38.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.433+0000 7fa746b1d640 1 --2- 172.21.15.57:0/4057572173 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa7400cf340 0x7fa7400d1910 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fa734009830 tx=0x7fa73402bcd0 comp rx=0 tx=0).stop 2023-10-19T12:45:38.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.433+0000 7fa746b1d640 1 -- 172.21.15.57:0/4057572173 shutdown_connections 2023-10-19T12:45:38.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.433+0000 7fa746b1d640 1 --2- 172.21.15.57:0/4057572173 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa7400cf340 0x7fa7400d1910 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:38.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.433+0000 7fa746b1d640 1 -- 172.21.15.57:0/4057572173 >> 172.21.15.57:0/4057572173 conn(0x7fa7400d0d60 msgr2=0x7fa7400d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:38.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.433+0000 7fa746b1d640 1 -- 172.21.15.57:0/4057572173 shutdown_connections 2023-10-19T12:45:38.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.433+0000 7fa746b1d640 1 -- 172.21.15.57:0/4057572173 wait complete. 2023-10-19T12:45:38.439 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.434+0000 7fa746b1d640 1 Processor -- start 2023-10-19T12:45:38.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.435+0000 7fa746b1d640 1 -- start start 2023-10-19T12:45:38.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.435+0000 7fa746b1d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa7400ceb40 0x7fa740156df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:38.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.435+0000 7fa746b1d640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa7400cf340 0x7fa7401575a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:38.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.435+0000 7fa746b1d640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa74015b2a0 0x7fa7400b58b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:38.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.435+0000 7fa746b1d640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7400b3a10 con 0x7fa7400ceb40 2023-10-19T12:45:38.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.435+0000 7fa746b1d640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa740157f80 con 0x7fa7400cf340 2023-10-19T12:45:38.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.435+0000 7fa746b1d640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa740158260 con 0x7fa74015b2a0 2023-10-19T12:45:38.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.436+0000 7fa7448b9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa7400ceb40 0x7fa740156df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:38.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.436+0000 7fa7448b9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa7400ceb40 0x7fa740156df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59150/0 (socket says 172.21.15.57:59150) 2023-10-19T12:45:38.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.436+0000 7fa7448b9640 1 -- 172.21.15.57:0/1894144979 learned_addr learned my addr 172.21.15.57:0/1894144979 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:38.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.436+0000 7fa7448b9640 1 -- 172.21.15.57:0/1894144979 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa74015b2a0 msgr2=0x7fa7400b58b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:38.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.436+0000 7fa7448b9640 1 --2- 172.21.15.57:0/1894144979 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa74015b2a0 0x7fa7400b58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:38.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.436+0000 7fa7448b9640 1 -- 172.21.15.57:0/1894144979 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa7400cf340 msgr2=0x7fa7401575a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:38.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.436+0000 7fa7448b9640 1 --2- 172.21.15.57:0/1894144979 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa7400cf340 0x7fa7401575a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:38.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.436+0000 7fa7448b9640 1 -- 172.21.15.57:0/1894144979 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa734009490 con 0x7fa7400ceb40 2023-10-19T12:45:38.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.436+0000 7fa7448b9640 1 --2- 172.21.15.57:0/1894144979 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa7400ceb40 0x7fa740156df0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fa734000c00 tx=0x7fa7340038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:38.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.436+0000 7fa73d7fa640 1 -- 172.21.15.57:0/1894144979 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa734003ea0 con 0x7fa7400ceb40 2023-10-19T12:45:38.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.436+0000 7fa73d7fa640 1 -- 172.21.15.57:0/1894144979 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fa734034530 con 0x7fa7400ceb40 2023-10-19T12:45:38.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.436+0000 7fa746b1d640 1 -- 172.21.15.57:0/1894144979 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa740158540 con 0x7fa7400ceb40 2023-10-19T12:45:38.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.436+0000 7fa73d7fa640 1 -- 172.21.15.57:0/1894144979 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa734034940 con 0x7fa7400ceb40 2023-10-19T12:45:38.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.436+0000 7fa746b1d640 1 -- 172.21.15.57:0/1894144979 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa740157330 con 0x7fa7400ceb40 2023-10-19T12:45:38.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.437+0000 7fa746b1d640 1 -- 172.21.15.57:0/1894144979 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7400690e0 con 0x7fa7400ceb40 2023-10-19T12:45:38.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.438+0000 7fa73d7fa640 1 -- 172.21.15.57:0/1894144979 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fa734034aa0 con 0x7fa7400ceb40 2023-10-19T12:45:38.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.438+0000 7fa73d7fa640 1 --2- 172.21.15.57:0/1894144979 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fa72406ae70 0x7fa72406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:38.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.438+0000 7fa73d7fa640 1 -- 172.21.15.57:0/1894144979 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fa7340abce0 con 0x7fa7400ceb40 2023-10-19T12:45:38.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.438+0000 7fa73ffff640 1 --2- 172.21.15.57:0/1894144979 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fa72406ae70 0x7fa72406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:38.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.439+0000 7fa73ffff640 1 --2- 172.21.15.57:0/1894144979 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fa72406ae70 0x7fa72406d330 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fa72c000f60 tx=0x7fa72c007a30 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:38.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.442+0000 7fa73d7fa640 1 -- 172.21.15.57:0/1894144979 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa734077030 con 0x7fa7400ceb40 2023-10-19T12:45:38.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.621+0000 7fa746b1d640 1 -- 172.21.15.57:0/1894144979 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fa7400690e0 con 0x7fa7400ceb40 2023-10-19T12:45:38.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.622+0000 7fa73d7fa640 1 -- 172.21.15.57:0/1894144979 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v35) v1 ==== 84+0+65 (secure 0 0 0) 0x7fa734042060 con 0x7fa7400ceb40 2023-10-19T12:45:38.625 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:45:38.626 INFO:teuthology.orchestra.run.smithi057.stdout:[ 2023-10-19T12:45:38.626 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T12:45:38.626 INFO:teuthology.orchestra.run.smithi057.stdout: "poolnum": 1, 2023-10-19T12:45:38.626 INFO:teuthology.orchestra.run.smithi057.stdout: "poolname": ".mgr" 2023-10-19T12:45:38.626 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T12:45:38.627 INFO:teuthology.orchestra.run.smithi057.stdout:] 2023-10-19T12:45:38.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.624+0000 7fa746b1d640 1 -- 172.21.15.57:0/1894144979 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fa72406ae70 msgr2=0x7fa72406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:38.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.624+0000 7fa746b1d640 1 --2- 172.21.15.57:0/1894144979 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fa72406ae70 0x7fa72406d330 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fa72c000f60 tx=0x7fa72c007a30 comp rx=0 tx=0).stop 2023-10-19T12:45:38.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.624+0000 7fa746b1d640 1 -- 172.21.15.57:0/1894144979 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa7400ceb40 msgr2=0x7fa740156df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:38.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.624+0000 7fa746b1d640 1 --2- 172.21.15.57:0/1894144979 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa7400ceb40 0x7fa740156df0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fa734000c00 tx=0x7fa7340038a0 comp rx=0 tx=0).stop 2023-10-19T12:45:38.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.624+0000 7fa746b1d640 1 -- 172.21.15.57:0/1894144979 shutdown_connections 2023-10-19T12:45:38.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.624+0000 7fa746b1d640 1 --2- 172.21.15.57:0/1894144979 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fa72406ae70 0x7fa72406d330 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:38.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.624+0000 7fa746b1d640 1 --2- 172.21.15.57:0/1894144979 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa74015b2a0 0x7fa7400b58b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:38.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.624+0000 7fa746b1d640 1 --2- 172.21.15.57:0/1894144979 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa7400cf340 0x7fa7401575a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:38.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.624+0000 7fa746b1d640 1 --2- 172.21.15.57:0/1894144979 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa7400ceb40 0x7fa740156df0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:38.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.624+0000 7fa746b1d640 1 -- 172.21.15.57:0/1894144979 >> 172.21.15.57:0/1894144979 conn(0x7fa7400d0d60 msgr2=0x7fa74015a3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:38.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.624+0000 7fa746b1d640 1 -- 172.21.15.57:0/1894144979 shutdown_connections 2023-10-19T12:45:38.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.624+0000 7fa746b1d640 1 -- 172.21.15.57:0/1894144979 wait complete. 2023-10-19T12:45:38.638 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-19T12:45:38.638 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-19T12:45:38.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.748+0000 7f783fec9640 1 -- 172.21.15.57:0/4077481143 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f78380cd6d0 msgr2=0x7f78380cdb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:38.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.748+0000 7f783fec9640 1 --2- 172.21.15.57:0/4077481143 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f78380cd6d0 0x7f78380cdb30 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f7838072420 tx=0x7f782c02bd10 comp rx=0 tx=0).stop 2023-10-19T12:45:38.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.748+0000 7f783fec9640 1 -- 172.21.15.57:0/4077481143 shutdown_connections 2023-10-19T12:45:38.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.748+0000 7f783fec9640 1 --2- 172.21.15.57:0/4077481143 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7838069720 0x7f7838069bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:38.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.748+0000 7f783fec9640 1 --2- 172.21.15.57:0/4077481143 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f78380cd6d0 0x7f78380cdb30 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:38.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.748+0000 7f783fec9640 1 --2- 172.21.15.57:0/4077481143 >> v2:172.21.15.179:3300/0 conn(0x7f7838142850 0x7f7838159440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:38.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.748+0000 7f783fec9640 1 -- 172.21.15.57:0/4077481143 >> 172.21.15.57:0/4077481143 conn(0x7f78380af7a0 msgr2=0x7f78380afb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:38.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.748+0000 7f783fec9640 1 -- 172.21.15.57:0/4077481143 shutdown_connections 2023-10-19T12:45:38.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.748+0000 7f783fec9640 1 -- 172.21.15.57:0/4077481143 wait complete. 2023-10-19T12:45:38.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.749+0000 7f783fec9640 1 Processor -- start 2023-10-19T12:45:38.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.749+0000 7f783fec9640 1 -- start start 2023-10-19T12:45:38.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.750+0000 7f783fec9640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7838069720 0x7f783814a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:38.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.750+0000 7f783fec9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f78380cd6d0 0x7f783813e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:38.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.750+0000 7f783fec9640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7838142850 0x7f783813eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:38.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.750+0000 7f783fec9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f783813e570 con 0x7f78380cd6d0 2023-10-19T12:45:38.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.750+0000 7f783fec9640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f783813bd60 con 0x7f7838069720 2023-10-19T12:45:38.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.750+0000 7f783fec9640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f783813c040 con 0x7f7838142850 2023-10-19T12:45:38.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.750+0000 7f783d464640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f78380cd6d0 0x7f783813e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:38.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.750+0000 7f783d464640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f78380cd6d0 0x7f783813e030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59170/0 (socket says 172.21.15.57:59170) 2023-10-19T12:45:38.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.750+0000 7f783d464640 1 -- 172.21.15.57:0/3245814368 learned_addr learned my addr 172.21.15.57:0/3245814368 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:38.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.750+0000 7f783dc65640 1 --2- 172.21.15.57:0/3245814368 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7838069720 0x7f783814a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:38.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.750+0000 7f783d464640 1 -- 172.21.15.57:0/3245814368 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7838142850 msgr2=0x7f783813eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:38.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.750+0000 7f783d464640 1 --2- 172.21.15.57:0/3245814368 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7838142850 0x7f783813eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:38.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.750+0000 7f783d464640 1 -- 172.21.15.57:0/3245814368 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7838069720 msgr2=0x7f783814a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:38.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.750+0000 7f783d464640 1 --2- 172.21.15.57:0/3245814368 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7838069720 0x7f783814a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:38.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.750+0000 7f783d464640 1 -- 172.21.15.57:0/3245814368 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f782c009560 con 0x7f78380cd6d0 2023-10-19T12:45:38.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.751+0000 7f783d464640 1 --2- 172.21.15.57:0/3245814368 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f78380cd6d0 0x7f783813e030 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f782c000c00 tx=0x7f782c02dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:38.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.751+0000 7f78367fc640 1 -- 172.21.15.57:0/3245814368 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f782c003800 con 0x7f78380cd6d0 2023-10-19T12:45:38.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.751+0000 7f78367fc640 1 -- 172.21.15.57:0/3245814368 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f782c003a50 con 0x7f78380cd6d0 2023-10-19T12:45:38.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.751+0000 7f783fec9640 1 -- 172.21.15.57:0/3245814368 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f783813c790 con 0x7f78380cd6d0 2023-10-19T12:45:38.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.751+0000 7f78367fc640 1 -- 172.21.15.57:0/3245814368 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f782c003e60 con 0x7f78380cd6d0 2023-10-19T12:45:38.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.751+0000 7f783fec9640 1 -- 172.21.15.57:0/3245814368 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f783813c320 con 0x7f78380cd6d0 2023-10-19T12:45:38.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.752+0000 7f783fec9640 1 -- 172.21.15.57:0/3245814368 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78380690e0 con 0x7f78380cd6d0 2023-10-19T12:45:38.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.752+0000 7f78367fc640 1 -- 172.21.15.57:0/3245814368 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f782c03e450 con 0x7f78380cd6d0 2023-10-19T12:45:38.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.753+0000 7f78367fc640 1 --2- 172.21.15.57:0/3245814368 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f782406ae40 0x7f782406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:38.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.753+0000 7f78367fc640 1 -- 172.21.15.57:0/3245814368 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f782c0ac300 con 0x7f78380cd6d0 2023-10-19T12:45:38.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.753+0000 7f783dc65640 1 --2- 172.21.15.57:0/3245814368 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f782406ae40 0x7f782406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:38.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.754+0000 7f783dc65640 1 --2- 172.21.15.57:0/3245814368 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f782406ae40 0x7f782406d300 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f7830009960 tx=0x7f7830005c90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:38.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.756+0000 7f78367fc640 1 -- 172.21.15.57:0/3245814368 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f782c077750 con 0x7f78380cd6d0 2023-10-19T12:45:38.936 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:38.933+0000 7f783fec9640 1 -- 172.21.15.57:0/3245814368 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f783813c570 con 0x7f78380cd6d0 2023-10-19T12:45:39.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:39.326+0000 7f78367fc640 1 -- 172.21.15.57:0/3245814368 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v36) v1 ==== 138+0+0 (secure 0 0 0) 0x7f782c042020 con 0x7f78380cd6d0 2023-10-19T12:45:39.329 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'cephfs_metadata' created 2023-10-19T12:45:39.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:39.328+0000 7f783fec9640 1 -- 172.21.15.57:0/3245814368 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f782406ae40 msgr2=0x7f782406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:39.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:39.328+0000 7f783fec9640 1 --2- 172.21.15.57:0/3245814368 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f782406ae40 0x7f782406d300 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f7830009960 tx=0x7f7830005c90 comp rx=0 tx=0).stop 2023-10-19T12:45:39.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:39.328+0000 7f783fec9640 1 -- 172.21.15.57:0/3245814368 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f78380cd6d0 msgr2=0x7f783813e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:39.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:39.328+0000 7f783fec9640 1 --2- 172.21.15.57:0/3245814368 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f78380cd6d0 0x7f783813e030 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f782c000c00 tx=0x7f782c02dab0 comp rx=0 tx=0).stop 2023-10-19T12:45:39.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:39.328+0000 7f783fec9640 1 -- 172.21.15.57:0/3245814368 shutdown_connections 2023-10-19T12:45:39.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:39.328+0000 7f783fec9640 1 --2- 172.21.15.57:0/3245814368 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f782406ae40 0x7f782406d300 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:39.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:39.328+0000 7f783fec9640 1 --2- 172.21.15.57:0/3245814368 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7838142850 0x7f783813eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:39.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:39.328+0000 7f783fec9640 1 --2- 172.21.15.57:0/3245814368 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f78380cd6d0 0x7f783813e030 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:39.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:39.328+0000 7f783fec9640 1 --2- 172.21.15.57:0/3245814368 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7838069720 0x7f783814a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:39.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:39.328+0000 7f783fec9640 1 -- 172.21.15.57:0/3245814368 >> 172.21.15.57:0/3245814368 conn(0x7f78380af7a0 msgr2=0x7f78380cc2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:39.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:39.329+0000 7f783fec9640 1 -- 172.21.15.57:0/3245814368 shutdown_connections 2023-10-19T12:45:39.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:39.329+0000 7f783fec9640 1 -- 172.21.15.57:0/3245814368 wait complete. 2023-10-19T12:45:39.348 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-19T12:45:39.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:39.465+0000 7f5bf2e4f640 1 -- 172.21.15.57:0/3587418266 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5bd8003970 msgr2=0x7f5bd8003d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:39.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:39.465+0000 7f5bf2e4f640 1 --2- 172.21.15.57:0/3587418266 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5bd8003970 0x7f5bd8003d70 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f5be400c610 tx=0x7f5bd80057f0 comp rx=0 tx=0).stop 2023-10-19T12:45:39.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:39.465+0000 7f5bf2e4f640 1 -- 172.21.15.57:0/3587418266 shutdown_connections 2023-10-19T12:45:39.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:39.465+0000 7f5bf2e4f640 1 --2- 172.21.15.57:0/3587418266 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5bd8003970 0x7f5bd8003d70 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:39.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:39.465+0000 7f5bf2e4f640 1 -- 172.21.15.57:0/3587418266 >> 172.21.15.57:0/3587418266 conn(0x7f5bec0b0670 msgr2=0x7f5bec0b0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:39.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:39.466+0000 7f5bf2e4f640 1 -- 172.21.15.57:0/3587418266 shutdown_connections 2023-10-19T12:45:39.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:39.466+0000 7f5bf2e4f640 1 -- 172.21.15.57:0/3587418266 wait complete. 2023-10-19T12:45:39.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:39.467+0000 7f5bf2e4f640 1 Processor -- start 2023-10-19T12:45:39.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:39.467+0000 7f5bf2e4f640 1 -- start start 2023-10-19T12:45:39.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:39.468+0000 7f5bf2e4f640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5bd8003970 0x7f5bec0c5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:39.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:39.468+0000 7f5bf2e4f640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5bec14ff90 0x7f5bec1546d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:39.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:39.468+0000 7f5bf2e4f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5bec154c10 0x7f5bec0b0a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:39.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:39.468+0000 7f5bf2e4f640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bec06b720 con 0x7f5bec154c10 2023-10-19T12:45:39.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:39.468+0000 7f5bf2e4f640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bec0c60f0 con 0x7f5bec14ff90 2023-10-19T12:45:39.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:39.468+0000 7f5bf2e4f640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bec156760 con 0x7f5bd8003970 2023-10-19T12:45:39.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:39.468+0000 7f5bf13ec640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5bec154c10 0x7f5bec0b0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:39.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:39.468+0000 7f5bf13ec640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5bec154c10 0x7f5bec0b0a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59182/0 (socket says 172.21.15.57:59182) 2023-10-19T12:45:39.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:39.468+0000 7f5bf13ec640 1 -- 172.21.15.57:0/3350145594 learned_addr learned my addr 172.21.15.57:0/3350145594 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:39.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:39.468+0000 7f5bf0beb640 1 --2- 172.21.15.57:0/3350145594 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5bd8003970 0x7f5bec0c5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:39.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:39.468+0000 7f5bebfff640 1 --2- 172.21.15.57:0/3350145594 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5bec14ff90 0x7f5bec1546d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:39.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:39.468+0000 7f5bf13ec640 1 -- 172.21.15.57:0/3350145594 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5bd8003970 msgr2=0x7f5bec0c5bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:39.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:39.468+0000 7f5bf13ec640 1 --2- 172.21.15.57:0/3350145594 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5bd8003970 0x7f5bec0c5bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:39.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:39.468+0000 7f5bf13ec640 1 -- 172.21.15.57:0/3350145594 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5bec14ff90 msgr2=0x7f5bec1546d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:39.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:39.468+0000 7f5bf13ec640 1 --2- 172.21.15.57:0/3350145594 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5bec14ff90 0x7f5bec1546d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:39.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:39.468+0000 7f5bf13ec640 1 -- 172.21.15.57:0/3350145594 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bec06a830 con 0x7f5bec154c10 2023-10-19T12:45:39.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:39.468+0000 7f5bf13ec640 1 --2- 172.21.15.57:0/3350145594 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5bec154c10 0x7f5bec0b0a50 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f5bec072c40 tx=0x7f5bd400faa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:39.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:39.469+0000 7f5be97fa640 1 -- 172.21.15.57:0/3350145594 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bd4004220 con 0x7f5bec154c10 2023-10-19T12:45:39.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:39.469+0000 7f5be97fa640 1 -- 172.21.15.57:0/3350145594 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f5bd4004470 con 0x7f5bec154c10 2023-10-19T12:45:39.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:39.469+0000 7f5be97fa640 1 -- 172.21.15.57:0/3350145594 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bd4005260 con 0x7f5bec154c10 2023-10-19T12:45:39.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:39.469+0000 7f5bf2e4f640 1 -- 172.21.15.57:0/3350145594 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bec06ab10 con 0x7f5bec154c10 2023-10-19T12:45:39.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:39.469+0000 7f5bf2e4f640 1 -- 172.21.15.57:0/3350145594 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bec06b0d0 con 0x7f5bec154c10 2023-10-19T12:45:39.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:39.469+0000 7f5bf2e4f640 1 -- 172.21.15.57:0/3350145594 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bec0ce1d0 con 0x7f5bec154c10 2023-10-19T12:45:39.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:39.470+0000 7f5be97fa640 1 -- 172.21.15.57:0/3350145594 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f5bd40053c0 con 0x7f5bec154c10 2023-10-19T12:45:39.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:39.471+0000 7f5be97fa640 1 --2- 172.21.15.57:0/3350145594 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f5bcc06ae40 0x7f5bcc06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:39.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:39.471+0000 7f5be97fa640 1 -- 172.21.15.57:0/3350145594 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5547+0+0 (secure 0 0 0) 0x7f5bd408a060 con 0x7f5bec154c10 2023-10-19T12:45:39.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:39.471+0000 7f5bf0beb640 1 --2- 172.21.15.57:0/3350145594 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f5bcc06ae40 0x7f5bcc06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:39.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:39.471+0000 7f5bf0beb640 1 --2- 172.21.15.57:0/3350145594 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f5bcc06ae40 0x7f5bcc06d300 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f5be4034000 tx=0x7f5be402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:39.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:39.475+0000 7f5be97fa640 1 -- 172.21.15.57:0/3350145594 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f5bd40552f0 con 0x7f5bec154c10 2023-10-19T12:45:39.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:39.652+0000 7f5bf2e4f640 1 -- 172.21.15.57:0/3350145594 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f5bec0690e0 con 0x7f5bec154c10 2023-10-19T12:45:40.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:40.337+0000 7f5be97fa640 1 -- 172.21.15.57:0/3350145594 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v37) v1 ==== 170+0+0 (secure 0 0 0) 0x7f5bd40200d0 con 0x7f5bec154c10 2023-10-19T12:45:40.341 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'cephfs_data' created 2023-10-19T12:45:40.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:40.340+0000 7f5bf2e4f640 1 -- 172.21.15.57:0/3350145594 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f5bcc06ae40 msgr2=0x7f5bcc06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:40.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:40.340+0000 7f5bf2e4f640 1 --2- 172.21.15.57:0/3350145594 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f5bcc06ae40 0x7f5bcc06d300 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f5be4034000 tx=0x7f5be402e000 comp rx=0 tx=0).stop 2023-10-19T12:45:40.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:40.340+0000 7f5bf2e4f640 1 -- 172.21.15.57:0/3350145594 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5bec154c10 msgr2=0x7f5bec0b0a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:40.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:40.340+0000 7f5bf2e4f640 1 --2- 172.21.15.57:0/3350145594 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5bec154c10 0x7f5bec0b0a50 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f5bec072c40 tx=0x7f5bd400faa0 comp rx=0 tx=0).stop 2023-10-19T12:45:40.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:40.341+0000 7f5bf2e4f640 1 -- 172.21.15.57:0/3350145594 shutdown_connections 2023-10-19T12:45:40.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:40.341+0000 7f5bf2e4f640 1 --2- 172.21.15.57:0/3350145594 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f5bcc06ae40 0x7f5bcc06d300 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:40.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:40.341+0000 7f5bf2e4f640 1 --2- 172.21.15.57:0/3350145594 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5bec154c10 0x7f5bec0b0a50 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:40.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:40.341+0000 7f5bf2e4f640 1 --2- 172.21.15.57:0/3350145594 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5bec14ff90 0x7f5bec1546d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:40.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:40.341+0000 7f5bf2e4f640 1 --2- 172.21.15.57:0/3350145594 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5bd8003970 0x7f5bec0c5bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:40.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:40.341+0000 7f5bf2e4f640 1 -- 172.21.15.57:0/3350145594 >> 172.21.15.57:0/3350145594 conn(0x7f5bec0b0670 msgr2=0x7f5bec0c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:40.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:40.342+0000 7f5bf2e4f640 1 -- 172.21.15.57:0/3350145594 shutdown_connections 2023-10-19T12:45:40.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:40.342+0000 7f5bf2e4f640 1 -- 172.21.15.57:0/3350145594 wait complete. 2023-10-19T12:45:40.367 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-19T12:45:40.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:40.480+0000 7f8fd7275640 1 -- 172.21.15.57:0/3474867567 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8fd015b150 msgr2=0x7f8fd00acb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:40.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:40.480+0000 7f8fd7275640 1 --2- 172.21.15.57:0/3474867567 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8fd015b150 0x7f8fd00acb80 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc009830 tx=0x7f8fcc02bcd0 comp rx=0 tx=0).stop 2023-10-19T12:45:40.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:40.481+0000 7f8fd7275640 1 -- 172.21.15.57:0/3474867567 shutdown_connections 2023-10-19T12:45:40.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:40.481+0000 7f8fd7275640 1 --2- 172.21.15.57:0/3474867567 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8fd00cb9e0 0x7f8fd00cc600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:40.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:40.481+0000 7f8fd7275640 1 --2- 172.21.15.57:0/3474867567 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8fd015b150 0x7f8fd00acb80 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:40.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:40.481+0000 7f8fd7275640 1 -- 172.21.15.57:0/3474867567 >> 172.21.15.57:0/3474867567 conn(0x7f8fd00af7a0 msgr2=0x7f8fd00afb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:40.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:40.481+0000 7f8fd7275640 1 -- 172.21.15.57:0/3474867567 shutdown_connections 2023-10-19T12:45:40.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:40.481+0000 7f8fd7275640 1 -- 172.21.15.57:0/3474867567 wait complete. 2023-10-19T12:45:40.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:40.482+0000 7f8fd7275640 1 Processor -- start 2023-10-19T12:45:40.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:40.482+0000 7f8fd7275640 1 -- start start 2023-10-19T12:45:40.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:40.483+0000 7f8fd7275640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8fd00cb9e0 0x7f8fd01e0db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:40.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:40.483+0000 7f8fd7275640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8fd00ccb40 0x7f8fd01e21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:40.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:40.483+0000 7f8fd7275640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8fd015b150 0x7f8fd0141380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:40.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:40.483+0000 7f8fd7275640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd01e26e0 con 0x7f8fd00cb9e0 2023-10-19T12:45:40.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:40.483+0000 7f8fd7275640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd01e2910 con 0x7f8fd015b150 2023-10-19T12:45:40.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:40.483+0000 7f8fd7275640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd01e2bf0 con 0x7f8fd00ccb40 2023-10-19T12:45:40.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:40.483+0000 7f8fd5011640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8fd00cb9e0 0x7f8fd01e0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:40.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:40.483+0000 7f8fd5011640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8fd00cb9e0 0x7f8fd01e0db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59208/0 (socket says 172.21.15.57:59208) 2023-10-19T12:45:40.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:40.483+0000 7f8fd5011640 1 -- 172.21.15.57:0/3180084732 learned_addr learned my addr 172.21.15.57:0/3180084732 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:40.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:40.483+0000 7f8fd4810640 1 --2- 172.21.15.57:0/3180084732 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8fd00ccb40 0x7f8fd01e21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:40.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:40.483+0000 7f8fd5812640 1 --2- 172.21.15.57:0/3180084732 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8fd015b150 0x7f8fd0141380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:40.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:40.483+0000 7f8fd5011640 1 -- 172.21.15.57:0/3180084732 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8fd00ccb40 msgr2=0x7f8fd01e21a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:40.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:40.483+0000 7f8fd5011640 1 --2- 172.21.15.57:0/3180084732 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8fd00ccb40 0x7f8fd01e21a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:40.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:40.483+0000 7f8fd5011640 1 -- 172.21.15.57:0/3180084732 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8fd015b150 msgr2=0x7f8fd0141380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:40.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:40.483+0000 7f8fd5011640 1 --2- 172.21.15.57:0/3180084732 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8fd015b150 0x7f8fd0141380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:40.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:40.483+0000 7f8fd5011640 1 -- 172.21.15.57:0/3180084732 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fcc009490 con 0x7f8fd00cb9e0 2023-10-19T12:45:40.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:40.483+0000 7f8fd5011640 1 --2- 172.21.15.57:0/3180084732 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8fd00cb9e0 0x7f8fd01e0db0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc009780 tx=0x7f8fcc0038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:40.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:40.483+0000 7f8fc5ffb640 1 -- 172.21.15.57:0/3180084732 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fcc003ea0 con 0x7f8fd00cb9e0 2023-10-19T12:45:40.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:40.483+0000 7f8fc5ffb640 1 -- 172.21.15.57:0/3180084732 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f8fcc034550 con 0x7f8fd00cb9e0 2023-10-19T12:45:40.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:40.483+0000 7f8fd7275640 1 -- 172.21.15.57:0/3180084732 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fd01e2ed0 con 0x7f8fd00cb9e0 2023-10-19T12:45:40.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:40.483+0000 7f8fc5ffb640 1 -- 172.21.15.57:0/3180084732 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fcc034960 con 0x7f8fd00cb9e0 2023-10-19T12:45:40.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:40.484+0000 7f8fd7275640 1 -- 172.21.15.57:0/3180084732 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fd01418c0 con 0x7f8fd00cb9e0 2023-10-19T12:45:40.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:40.485+0000 7f8fd7275640 1 -- 172.21.15.57:0/3180084732 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fd0151f30 con 0x7f8fd00cb9e0 2023-10-19T12:45:40.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:40.485+0000 7f8fc5ffb640 1 -- 172.21.15.57:0/3180084732 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f8fcc03b040 con 0x7f8fd00cb9e0 2023-10-19T12:45:40.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:40.486+0000 7f8fc5ffb640 1 --2- 172.21.15.57:0/3180084732 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f8fb406ae40 0x7f8fb406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:40.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:40.486+0000 7f8fc5ffb640 1 -- 172.21.15.57:0/3180084732 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5929+0+0 (secure 0 0 0) 0x7f8fcc0abb70 con 0x7f8fd00cb9e0 2023-10-19T12:45:40.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:40.486+0000 7f8fd4810640 1 --2- 172.21.15.57:0/3180084732 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f8fb406ae40 0x7f8fb406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:40.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:40.486+0000 7f8fd4810640 1 --2- 172.21.15.57:0/3180084732 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f8fb406ae40 0x7f8fb406d300 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f8fd00acf60 tx=0x7f8fc000b390 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:40.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:40.490+0000 7f8fc5ffb640 1 -- 172.21.15.57:0/3180084732 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8fd0151f30 con 0x7f8fd00cb9e0 2023-10-19T12:45:40.707 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:40.704+0000 7f8fd7275640 1 -- 172.21.15.57:0/3180084732 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f8fd0151f30 con 0x7f8fd00cb9e0 2023-10-19T12:45:41.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.339+0000 7f8fc5ffb640 1 -- 172.21.15.57:0/3180084732 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 5 and data pool 6 v14) v1 ==== 172+0+0 (secure 0 0 0) 0x7f8fd0151f30 con 0x7f8fd00cb9e0 2023-10-19T12:45:41.342 INFO:teuthology.orchestra.run.smithi057.stderr:new fs with metadata pool 5 and data pool 6 2023-10-19T12:45:41.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.343+0000 7f8fd7275640 1 -- 172.21.15.57:0/3180084732 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f8fb406ae40 msgr2=0x7f8fb406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:41.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.343+0000 7f8fd7275640 1 --2- 172.21.15.57:0/3180084732 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f8fb406ae40 0x7f8fb406d300 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f8fd00acf60 tx=0x7f8fc000b390 comp rx=0 tx=0).stop 2023-10-19T12:45:41.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.343+0000 7f8fd7275640 1 -- 172.21.15.57:0/3180084732 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8fd00cb9e0 msgr2=0x7f8fd01e0db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:41.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.343+0000 7f8fd7275640 1 --2- 172.21.15.57:0/3180084732 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8fd00cb9e0 0x7f8fd01e0db0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc009780 tx=0x7f8fcc0038a0 comp rx=0 tx=0).stop 2023-10-19T12:45:41.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.343+0000 7f8fd7275640 1 -- 172.21.15.57:0/3180084732 shutdown_connections 2023-10-19T12:45:41.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.343+0000 7f8fd7275640 1 --2- 172.21.15.57:0/3180084732 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f8fb406ae40 0x7f8fb406d300 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:41.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.343+0000 7f8fd7275640 1 --2- 172.21.15.57:0/3180084732 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8fd015b150 0x7f8fd0141380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:41.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.343+0000 7f8fd7275640 1 --2- 172.21.15.57:0/3180084732 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8fd00ccb40 0x7f8fd01e21a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:41.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.343+0000 7f8fd7275640 1 --2- 172.21.15.57:0/3180084732 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8fd00cb9e0 0x7f8fd01e0db0 secure :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc009780 tx=0x7f8fcc0038a0 comp rx=0 tx=0).stop 2023-10-19T12:45:41.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.343+0000 7f8fd7275640 1 -- 172.21.15.57:0/3180084732 >> 172.21.15.57:0/3180084732 conn(0x7f8fd00af7a0 msgr2=0x7f8fd00c9d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:41.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.343+0000 7f8fd7275640 1 -- 172.21.15.57:0/3180084732 shutdown_connections 2023-10-19T12:45:41.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.343+0000 7f8fd7275640 1 -- 172.21.15.57:0/3180084732 wait complete. 2023-10-19T12:45:41.373 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T12:45:41.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.485+0000 7f59d0683640 1 -- 172.21.15.57:0/2034047725 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f59c80c96b0 msgr2=0x7f59c80c9b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:41.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.485+0000 7f59d0683640 1 --2- 172.21.15.57:0/2034047725 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f59c80c96b0 0x7f59c80c9b60 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f59b8005fd0 tx=0x7f59b802bd10 comp rx=0 tx=0).stop 2023-10-19T12:45:41.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.485+0000 7f59d0683640 1 -- 172.21.15.57:0/2034047725 shutdown_connections 2023-10-19T12:45:41.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.485+0000 7f59d0683640 1 --2- 172.21.15.57:0/2034047725 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f59c80c96b0 0x7f59c80c9b60 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:41.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.485+0000 7f59d0683640 1 --2- 172.21.15.57:0/2034047725 >> v2:172.21.15.179:3300/0 conn(0x7f59c806af00 0x7f59c806b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:41.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.485+0000 7f59d0683640 1 -- 172.21.15.57:0/2034047725 >> 172.21.15.57:0/2034047725 conn(0x7f59c8069930 msgr2=0x7f59c8069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:41.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.485+0000 7f59d0683640 1 -- 172.21.15.57:0/2034047725 shutdown_connections 2023-10-19T12:45:41.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.485+0000 7f59d0683640 1 -- 172.21.15.57:0/2034047725 wait complete. 2023-10-19T12:45:41.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.486+0000 7f59d0683640 1 Processor -- start 2023-10-19T12:45:41.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.486+0000 7f59d0683640 1 -- start start 2023-10-19T12:45:41.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.487+0000 7f59d0683640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f59c806af00 0x7f59c8158240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:41.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.487+0000 7f59d0683640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f59c80ab500 0x7f59c81324f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:41.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.487+0000 7f59d0683640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f59c80c96b0 0x7f59c80aa8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:41.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.487+0000 7f59d0683640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59c81f1c30 con 0x7f59c80ab500 2023-10-19T12:45:41.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.487+0000 7f59d0683640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59c81331b0 con 0x7f59c806af00 2023-10-19T12:45:41.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.487+0000 7f59d0683640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59c8158780 con 0x7f59c80c96b0 2023-10-19T12:45:41.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.487+0000 7f59cdc1e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f59c80ab500 0x7f59c81324f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:41.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.487+0000 7f59cdc1e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f59c80ab500 0x7f59c81324f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59232/0 (socket says 172.21.15.57:59232) 2023-10-19T12:45:41.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.487+0000 7f59cdc1e640 1 -- 172.21.15.57:0/1754191963 learned_addr learned my addr 172.21.15.57:0/1754191963 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:41.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.487+0000 7f59ce41f640 1 --2- 172.21.15.57:0/1754191963 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f59c806af00 0x7f59c8158240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:41.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.487+0000 7f59cec20640 1 --2- 172.21.15.57:0/1754191963 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f59c80c96b0 0x7f59c80aa8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:41.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.487+0000 7f59cdc1e640 1 -- 172.21.15.57:0/1754191963 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f59c80c96b0 msgr2=0x7f59c80aa8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:41.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.487+0000 7f59cdc1e640 1 --2- 172.21.15.57:0/1754191963 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f59c80c96b0 0x7f59c80aa8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:41.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.487+0000 7f59cdc1e640 1 -- 172.21.15.57:0/1754191963 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f59c806af00 msgr2=0x7f59c8158240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:41.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.487+0000 7f59cdc1e640 1 --2- 172.21.15.57:0/1754191963 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f59c806af00 0x7f59c8158240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:41.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.487+0000 7f59cdc1e640 1 -- 172.21.15.57:0/1754191963 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59b8009560 con 0x7f59c80ab500 2023-10-19T12:45:41.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.487+0000 7f59cdc1e640 1 --2- 172.21.15.57:0/1754191963 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f59c80ab500 0x7f59c81324f0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f59b8009850 tx=0x7f59b802d9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:41.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.487+0000 7f59beffd640 1 -- 172.21.15.57:0/1754191963 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59b802dde0 con 0x7f59c80ab500 2023-10-19T12:45:41.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.487+0000 7f59beffd640 1 -- 172.21.15.57:0/1754191963 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f59b80036e0 con 0x7f59c80ab500 2023-10-19T12:45:41.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.487+0000 7f59d0683640 1 -- 172.21.15.57:0/1754191963 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59c81f1310 con 0x7f59c80ab500 2023-10-19T12:45:41.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.487+0000 7f59beffd640 1 -- 172.21.15.57:0/1754191963 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59b8003a90 con 0x7f59c80ab500 2023-10-19T12:45:41.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.487+0000 7f59d0683640 1 -- 172.21.15.57:0/1754191963 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59c81f1570 con 0x7f59c80ab500 2023-10-19T12:45:41.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.488+0000 7f59d0683640 1 -- 172.21.15.57:0/1754191963 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59c81f1d70 con 0x7f59c80ab500 2023-10-19T12:45:41.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.489+0000 7f59beffd640 1 -- 172.21.15.57:0/1754191963 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f59b8003bf0 con 0x7f59c80ab500 2023-10-19T12:45:41.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.489+0000 7f59beffd640 1 --2- 172.21.15.57:0/1754191963 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f59ac06ae40 0x7f59ac06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:41.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.489+0000 7f59beffd640 1 -- 172.21.15.57:0/1754191963 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f59b80ac140 con 0x7f59c80ab500 2023-10-19T12:45:41.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.489+0000 7f59ce41f640 1 --2- 172.21.15.57:0/1754191963 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f59ac06ae40 0x7f59ac06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:41.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.490+0000 7f59ce41f640 1 --2- 172.21.15.57:0/1754191963 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f59ac06ae40 0x7f59ac06d300 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f59c80add90 tx=0x7f59c400b390 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:41.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.493+0000 7f59beffd640 1 -- 172.21.15.57:0/1754191963 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f59c81f1d70 con 0x7f59c80ab500 2023-10-19T12:45:41.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.668+0000 7f59d0683640 1 -- 172.21.15.57:0/1754191963 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f59c81f1d70 con 0x7f59c80ab500 2023-10-19T12:45:41.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.669+0000 7f59beffd640 1 -- 172.21.15.57:0/1754191963 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) v1 ==== 74+0+16799 (secure 0 0 0) 0x7f59c81f1d70 con 0x7f59c80ab500 2023-10-19T12:45:41.673 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:45:41.673 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":38,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T12:45:40.706199+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2023-10-19T12:45:38.964640+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2023-10-19T12:45:39.683197+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T12:45:41.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.671+0000 7f59d0683640 1 -- 172.21.15.57:0/1754191963 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f59ac06ae40 msgr2=0x7f59ac06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:41.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.671+0000 7f59d0683640 1 --2- 172.21.15.57:0/1754191963 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f59ac06ae40 0x7f59ac06d300 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f59c80add90 tx=0x7f59c400b390 comp rx=0 tx=0).stop 2023-10-19T12:45:41.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.671+0000 7f59d0683640 1 -- 172.21.15.57:0/1754191963 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f59c80ab500 msgr2=0x7f59c81324f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:41.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.671+0000 7f59d0683640 1 --2- 172.21.15.57:0/1754191963 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f59c80ab500 0x7f59c81324f0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f59b8009850 tx=0x7f59b802d9e0 comp rx=0 tx=0).stop 2023-10-19T12:45:41.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.671+0000 7f59d0683640 1 -- 172.21.15.57:0/1754191963 shutdown_connections 2023-10-19T12:45:41.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.671+0000 7f59d0683640 1 --2- 172.21.15.57:0/1754191963 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f59ac06ae40 0x7f59ac06d300 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:41.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.671+0000 7f59d0683640 1 --2- 172.21.15.57:0/1754191963 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f59c80c96b0 0x7f59c80aa8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:41.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.671+0000 7f59d0683640 1 --2- 172.21.15.57:0/1754191963 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f59c80ab500 0x7f59c81324f0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:41.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.671+0000 7f59d0683640 1 --2- 172.21.15.57:0/1754191963 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f59c806af00 0x7f59c8158240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:41.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.671+0000 7f59d0683640 1 -- 172.21.15.57:0/1754191963 >> 172.21.15.57:0/1754191963 conn(0x7f59c8069930 msgr2=0x7f59c815a440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:41.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.671+0000 7f59d0683640 1 -- 172.21.15.57:0/1754191963 shutdown_connections 2023-10-19T12:45:41.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.671+0000 7f59d0683640 1 -- 172.21.15.57:0/1754191963 wait complete. 2023-10-19T12:45:41.685 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T12:45:41.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.790+0000 7ff9d4075640 1 -- 172.21.15.57:0/1325535218 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9cc06abc0 msgr2=0x7ff9cc0ac1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:41.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.790+0000 7ff9d4075640 1 --2- 172.21.15.57:0/1325535218 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9cc06abc0 0x7ff9cc0ac1f0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7ff9c4009830 tx=0x7ff9c402bd60 comp rx=0 tx=0).stop 2023-10-19T12:45:41.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.790+0000 7ff9d4075640 1 -- 172.21.15.57:0/1325535218 shutdown_connections 2023-10-19T12:45:41.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.790+0000 7ff9d4075640 1 --2- 172.21.15.57:0/1325535218 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9cc06abc0 0x7ff9cc0ac1f0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:41.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.790+0000 7ff9d4075640 1 -- 172.21.15.57:0/1325535218 >> 172.21.15.57:0/1325535218 conn(0x7ff9cc06fae0 msgr2=0x7ff9cc06fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:41.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.790+0000 7ff9d4075640 1 -- 172.21.15.57:0/1325535218 shutdown_connections 2023-10-19T12:45:41.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.790+0000 7ff9d4075640 1 -- 172.21.15.57:0/1325535218 wait complete. 2023-10-19T12:45:41.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.792+0000 7ff9d4075640 1 Processor -- start 2023-10-19T12:45:41.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.792+0000 7ff9d4075640 1 -- start start 2023-10-19T12:45:41.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.792+0000 7ff9d4075640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9cc06a730 0x7ff9cc147d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:41.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.792+0000 7ff9d4075640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff9cc06abc0 0x7ff9cc148af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:41.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.792+0000 7ff9d4075640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff9cc155580 0x7ff9cc0cbf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:41.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.792+0000 7ff9d4075640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9cc14e280 con 0x7ff9cc06a730 2023-10-19T12:45:41.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.792+0000 7ff9d4075640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9cc152d10 con 0x7ff9cc06abc0 2023-10-19T12:45:41.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.792+0000 7ff9d4075640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9cc152ff0 con 0x7ff9cc155580 2023-10-19T12:45:41.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.792+0000 7ff9d1e11640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9cc06a730 0x7ff9cc147d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:41.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.793+0000 7ff9d1e11640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9cc06a730 0x7ff9cc147d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59244/0 (socket says 172.21.15.57:59244) 2023-10-19T12:45:41.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.793+0000 7ff9d1e11640 1 -- 172.21.15.57:0/1406353583 learned_addr learned my addr 172.21.15.57:0/1406353583 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:41.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.793+0000 7ff9d1610640 1 --2- 172.21.15.57:0/1406353583 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff9cc06abc0 0x7ff9cc148af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:41.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.793+0000 7ff9d2612640 1 --2- 172.21.15.57:0/1406353583 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff9cc155580 0x7ff9cc0cbf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:41.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.793+0000 7ff9d1e11640 1 -- 172.21.15.57:0/1406353583 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff9cc155580 msgr2=0x7ff9cc0cbf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:41.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.793+0000 7ff9d1e11640 1 --2- 172.21.15.57:0/1406353583 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff9cc155580 0x7ff9cc0cbf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:41.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.793+0000 7ff9d1e11640 1 -- 172.21.15.57:0/1406353583 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff9cc06abc0 msgr2=0x7ff9cc148af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:41.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.793+0000 7ff9d1e11640 1 --2- 172.21.15.57:0/1406353583 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff9cc06abc0 0x7ff9cc148af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:41.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.793+0000 7ff9d1e11640 1 -- 172.21.15.57:0/1406353583 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9c4009490 con 0x7ff9cc06a730 2023-10-19T12:45:41.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.793+0000 7ff9d1610640 1 --2- 172.21.15.57:0/1406353583 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff9cc06abc0 0x7ff9cc148af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:45:41.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.793+0000 7ff9d1e11640 1 --2- 172.21.15.57:0/1406353583 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9cc06a730 0x7ff9cc147d60 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7ff9bc00c7f0 tx=0x7ff9bc00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:41.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.793+0000 7ff9ca7fc640 1 -- 172.21.15.57:0/1406353583 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9bc003da0 con 0x7ff9cc06a730 2023-10-19T12:45:41.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.793+0000 7ff9ca7fc640 1 -- 172.21.15.57:0/1406353583 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7ff9bc005070 con 0x7ff9cc06a730 2023-10-19T12:45:41.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.793+0000 7ff9ca7fc640 1 -- 172.21.15.57:0/1406353583 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9bc005460 con 0x7ff9cc06a730 2023-10-19T12:45:41.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.793+0000 7ff9d4075640 1 -- 172.21.15.57:0/1406353583 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9cc1532d0 con 0x7ff9cc06a730 2023-10-19T12:45:41.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.793+0000 7ff9d4075640 1 -- 172.21.15.57:0/1406353583 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9cc1482a0 con 0x7ff9cc06a730 2023-10-19T12:45:41.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.794+0000 7ff9d4075640 1 -- 172.21.15.57:0/1406353583 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9cc1537f0 con 0x7ff9cc06a730 2023-10-19T12:45:41.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.795+0000 7ff9ca7fc640 1 -- 172.21.15.57:0/1406353583 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7ff9bc0055c0 con 0x7ff9cc06a730 2023-10-19T12:45:41.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.795+0000 7ff9ca7fc640 1 --2- 172.21.15.57:0/1406353583 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7ff9b806ae40 0x7ff9b806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:41.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.795+0000 7ff9ca7fc640 1 -- 172.21.15.57:0/1406353583 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6029+0+0 (secure 0 0 0) 0x7ff9bc0891c0 con 0x7ff9cc06a730 2023-10-19T12:45:41.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.795+0000 7ff9d1610640 1 --2- 172.21.15.57:0/1406353583 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7ff9b806ae40 0x7ff9b806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:41.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.796+0000 7ff9d1610640 1 --2- 172.21.15.57:0/1406353583 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7ff9b806ae40 0x7ff9b806d300 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc15b270 tx=0x7ff9c0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:41.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.799+0000 7ff9ca7fc640 1 -- 172.21.15.57:0/1406353583 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff9cc1537f0 con 0x7ff9cc06a730 2023-10-19T12:45:41.979 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.976+0000 7ff9d4075640 1 -- 172.21.15.57:0/1406353583 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9cc1537f0 con 0x7ff9cc06a730 2023-10-19T12:45:41.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.977+0000 7ff9ca7fc640 1 -- 172.21.15.57:0/1406353583 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) v1 ==== 74+0+16799 (secure 0 0 0) 0x7ff9cc1537f0 con 0x7ff9cc06a730 2023-10-19T12:45:41.981 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:45:41.981 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":38,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T12:45:40.706199+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2023-10-19T12:45:38.964640+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2023-10-19T12:45:39.683197+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T12:45:41.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.979+0000 7ff9d4075640 1 -- 172.21.15.57:0/1406353583 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7ff9b806ae40 msgr2=0x7ff9b806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:41.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.979+0000 7ff9d4075640 1 --2- 172.21.15.57:0/1406353583 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7ff9b806ae40 0x7ff9b806d300 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc15b270 tx=0x7ff9c0009040 comp rx=0 tx=0).stop 2023-10-19T12:45:41.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.979+0000 7ff9d4075640 1 -- 172.21.15.57:0/1406353583 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9cc06a730 msgr2=0x7ff9cc147d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:41.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.979+0000 7ff9d4075640 1 --2- 172.21.15.57:0/1406353583 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9cc06a730 0x7ff9cc147d60 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7ff9bc00c7f0 tx=0x7ff9bc00ccb0 comp rx=0 tx=0).stop 2023-10-19T12:45:41.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.979+0000 7ff9d4075640 1 -- 172.21.15.57:0/1406353583 shutdown_connections 2023-10-19T12:45:41.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.979+0000 7ff9d4075640 1 --2- 172.21.15.57:0/1406353583 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7ff9b806ae40 0x7ff9b806d300 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:41.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.979+0000 7ff9d4075640 1 --2- 172.21.15.57:0/1406353583 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff9cc155580 0x7ff9cc0cbf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:41.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.979+0000 7ff9d4075640 1 --2- 172.21.15.57:0/1406353583 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff9cc06abc0 0x7ff9cc148af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:41.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.979+0000 7ff9d4075640 1 --2- 172.21.15.57:0/1406353583 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9cc06a730 0x7ff9cc147d60 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:41.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.979+0000 7ff9d4075640 1 -- 172.21.15.57:0/1406353583 >> 172.21.15.57:0/1406353583 conn(0x7ff9cc06fae0 msgr2=0x7ff9cc0ae330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:41.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.979+0000 7ff9d4075640 1 -- 172.21.15.57:0/1406353583 shutdown_connections 2023-10-19T12:45:41.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:41.979+0000 7ff9d4075640 1 -- 172.21.15.57:0/1406353583 wait complete. 2023-10-19T12:45:41.993 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-19T12:45:42.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.102+0000 7fca5cad7640 1 -- 172.21.15.57:0/68019246 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fca580ab500 msgr2=0x7fca580ab9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:42.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.102+0000 7fca5cad7640 1 --2- 172.21.15.57:0/68019246 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fca580ab500 0x7fca580ab9b0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fca50005fd0 tx=0x7fca5002c0e0 comp rx=0 tx=0).stop 2023-10-19T12:45:42.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.102+0000 7fca5cad7640 1 -- 172.21.15.57:0/68019246 shutdown_connections 2023-10-19T12:45:42.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.102+0000 7fca5cad7640 1 --2- 172.21.15.57:0/68019246 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fca580ab500 0x7fca580ab9b0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:42.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.102+0000 7fca5cad7640 1 --2- 172.21.15.57:0/68019246 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fca580c9dc0 0x7fca5806b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:42.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.102+0000 7fca5cad7640 1 -- 172.21.15.57:0/68019246 >> 172.21.15.57:0/68019246 conn(0x7fca58069930 msgr2=0x7fca58069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:42.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.102+0000 7fca5cad7640 1 -- 172.21.15.57:0/68019246 shutdown_connections 2023-10-19T12:45:42.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.102+0000 7fca5cad7640 1 -- 172.21.15.57:0/68019246 wait complete. 2023-10-19T12:45:42.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.103+0000 7fca5cad7640 1 Processor -- start 2023-10-19T12:45:42.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.104+0000 7fca5cad7640 1 -- start start 2023-10-19T12:45:42.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.104+0000 7fca5cad7640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fca5806af00 0x7fca5813ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:42.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.104+0000 7fca5cad7640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fca580ab500 0x7fca58152bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:42.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.104+0000 7fca5cad7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fca580c9dc0 0x7fca580aa8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:42.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.104+0000 7fca5cad7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca580afe50 con 0x7fca580c9dc0 2023-10-19T12:45:42.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.104+0000 7fca5cad7640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca5813b070 con 0x7fca5806af00 2023-10-19T12:45:42.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.104+0000 7fca5cad7640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca580b02e0 con 0x7fca580ab500 2023-10-19T12:45:42.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.104+0000 7fca56d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fca580c9dc0 0x7fca580aa8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:42.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.104+0000 7fca56d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fca580c9dc0 0x7fca580aa8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59258/0 (socket says 172.21.15.57:59258) 2023-10-19T12:45:42.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.104+0000 7fca56d9d640 1 -- 172.21.15.57:0/1097936696 learned_addr learned my addr 172.21.15.57:0/1097936696 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:42.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.105+0000 7fca5659c640 1 --2- 172.21.15.57:0/1097936696 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fca5806af00 0x7fca5813ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:42.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.105+0000 7fca4ffff640 1 --2- 172.21.15.57:0/1097936696 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fca580ab500 0x7fca58152bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:42.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.105+0000 7fca56d9d640 1 -- 172.21.15.57:0/1097936696 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fca580ab500 msgr2=0x7fca58152bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:42.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.105+0000 7fca56d9d640 1 --2- 172.21.15.57:0/1097936696 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fca580ab500 0x7fca58152bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:42.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.105+0000 7fca56d9d640 1 -- 172.21.15.57:0/1097936696 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fca5806af00 msgr2=0x7fca5813ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:42.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.105+0000 7fca56d9d640 1 --2- 172.21.15.57:0/1097936696 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fca5806af00 0x7fca5813ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:42.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.105+0000 7fca56d9d640 1 -- 172.21.15.57:0/1097936696 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca50009840 con 0x7fca580c9dc0 2023-10-19T12:45:42.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.105+0000 7fca56d9d640 1 --2- 172.21.15.57:0/1097936696 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fca580c9dc0 0x7fca580aa8c0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fca50006010 tx=0x7fca50004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:42.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.105+0000 7fca4effd640 1 -- 172.21.15.57:0/1097936696 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca50004670 con 0x7fca580c9dc0 2023-10-19T12:45:42.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.105+0000 7fca4effd640 1 -- 172.21.15.57:0/1097936696 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fca500345a0 con 0x7fca580c9dc0 2023-10-19T12:45:42.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.105+0000 7fca4effd640 1 -- 172.21.15.57:0/1097936696 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca50034970 con 0x7fca580c9dc0 2023-10-19T12:45:42.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.105+0000 7fca5cad7640 1 -- 172.21.15.57:0/1097936696 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca580af240 con 0x7fca580c9dc0 2023-10-19T12:45:42.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.105+0000 7fca5cad7640 1 -- 172.21.15.57:0/1097936696 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca580af800 con 0x7fca580c9dc0 2023-10-19T12:45:42.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.106+0000 7fca5cad7640 1 -- 172.21.15.57:0/1097936696 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca580690e0 con 0x7fca580c9dc0 2023-10-19T12:45:42.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.107+0000 7fca4effd640 1 -- 172.21.15.57:0/1097936696 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fca50034ad0 con 0x7fca580c9dc0 2023-10-19T12:45:42.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.107+0000 7fca4effd640 1 --2- 172.21.15.57:0/1097936696 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fca3806ae40 0x7fca3806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:42.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.107+0000 7fca4effd640 1 -- 172.21.15.57:0/1097936696 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fca500ac7f0 con 0x7fca580c9dc0 2023-10-19T12:45:42.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.108+0000 7fca5659c640 1 --2- 172.21.15.57:0/1097936696 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fca3806ae40 0x7fca3806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:42.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.108+0000 7fca5659c640 1 --2- 172.21.15.57:0/1097936696 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fca3806ae40 0x7fca3806d300 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fca480011e0 tx=0x7fca48009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:42.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.111+0000 7fca4effd640 1 -- 172.21.15.57:0/1097936696 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fca50077750 con 0x7fca580c9dc0 2023-10-19T12:45:42.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.327+0000 7fca5cad7640 1 -- 172.21.15.57:0/1097936696 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fca580cc420 con 0x7fca580c9dc0 2023-10-19T12:45:42.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.352+0000 7fca4effd640 1 -- 172.21.15.57:0/1097936696 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v15) v1 ==== 118+0+0 (secure 0 0 0) 0x7fca5007ad70 con 0x7fca580c9dc0 2023-10-19T12:45:42.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.355+0000 7fca5cad7640 1 -- 172.21.15.57:0/1097936696 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fca3806ae40 msgr2=0x7fca3806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:42.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.355+0000 7fca5cad7640 1 --2- 172.21.15.57:0/1097936696 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fca3806ae40 0x7fca3806d300 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fca480011e0 tx=0x7fca48009040 comp rx=0 tx=0).stop 2023-10-19T12:45:42.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.355+0000 7fca5cad7640 1 -- 172.21.15.57:0/1097936696 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fca580c9dc0 msgr2=0x7fca580aa8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:42.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.355+0000 7fca5cad7640 1 --2- 172.21.15.57:0/1097936696 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fca580c9dc0 0x7fca580aa8c0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fca50006010 tx=0x7fca50004270 comp rx=0 tx=0).stop 2023-10-19T12:45:42.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.355+0000 7fca5cad7640 1 -- 172.21.15.57:0/1097936696 shutdown_connections 2023-10-19T12:45:42.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.355+0000 7fca5cad7640 1 --2- 172.21.15.57:0/1097936696 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fca3806ae40 0x7fca3806d300 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:42.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.355+0000 7fca5cad7640 1 --2- 172.21.15.57:0/1097936696 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fca580c9dc0 0x7fca580aa8c0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:42.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.355+0000 7fca5cad7640 1 --2- 172.21.15.57:0/1097936696 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fca580ab500 0x7fca58152bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:42.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.355+0000 7fca5cad7640 1 --2- 172.21.15.57:0/1097936696 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fca5806af00 0x7fca5813ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:42.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.355+0000 7fca5cad7640 1 -- 172.21.15.57:0/1097936696 >> 172.21.15.57:0/1097936696 conn(0x7fca58069930 msgr2=0x7fca580c9680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:42.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.355+0000 7fca5cad7640 1 -- 172.21.15.57:0/1097936696 shutdown_connections 2023-10-19T12:45:42.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.355+0000 7fca5cad7640 1 -- 172.21.15.57:0/1097936696 wait complete. 2023-10-19T12:45:42.370 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-19T12:45:42.371 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-19T12:45:42.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.479+0000 7ff9d5c75640 1 -- 172.21.15.57:0/2555085881 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9d00afa20 msgr2=0x7ff9d0158300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:42.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.479+0000 7ff9d5c75640 1 --2- 172.21.15.57:0/2555085881 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9d00afa20 0x7ff9d0158300 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7ff9d00b10f0 tx=0x7ff9c402bd70 comp rx=0 tx=0).stop 2023-10-19T12:45:42.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.479+0000 7ff9d5c75640 1 -- 172.21.15.57:0/2555085881 shutdown_connections 2023-10-19T12:45:42.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.479+0000 7ff9d5c75640 1 --2- 172.21.15.57:0/2555085881 >> v2:172.21.15.179:3300/0 conn(0x7ff9d00b0530 0x7ff9d00b09a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:42.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.479+0000 7ff9d5c75640 1 --2- 172.21.15.57:0/2555085881 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9d00afa20 0x7ff9d0158300 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:42.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.479+0000 7ff9d5c75640 1 -- 172.21.15.57:0/2555085881 >> 172.21.15.57:0/2555085881 conn(0x7ff9d0069930 msgr2=0x7ff9d0069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:42.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.479+0000 7ff9d5c75640 1 -- 172.21.15.57:0/2555085881 shutdown_connections 2023-10-19T12:45:42.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.479+0000 7ff9d5c75640 1 -- 172.21.15.57:0/2555085881 wait complete. 2023-10-19T12:45:42.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.480+0000 7ff9d5c75640 1 Processor -- start 2023-10-19T12:45:42.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.480+0000 7ff9d5c75640 1 -- start start 2023-10-19T12:45:42.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.481+0000 7ff9d5c75640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff9d00afa20 0x7ff9d0128c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:42.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.481+0000 7ff9d5c75640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff9d00b0530 0x7ff9d006b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:42.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.481+0000 7ff9d5c75640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9d00cbd20 0x7ff9d0141fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:42.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.481+0000 7ff9d5c75640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d0144a10 con 0x7ff9d00cbd20 2023-10-19T12:45:42.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.481+0000 7ff9d5c75640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d0143d90 con 0x7ff9d00b0530 2023-10-19T12:45:42.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.481+0000 7ff9d5c75640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d0144070 con 0x7ff9d00afa20 2023-10-19T12:45:42.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.481+0000 7ff9cffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9d00cbd20 0x7ff9d0141fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:42.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.481+0000 7ff9cffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9d00cbd20 0x7ff9d0141fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59276/0 (socket says 172.21.15.57:59276) 2023-10-19T12:45:42.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.481+0000 7ff9cffff640 1 -- 172.21.15.57:0/176457359 learned_addr learned my addr 172.21.15.57:0/176457359 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:42.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.482+0000 7ff9cf7fe640 1 --2- 172.21.15.57:0/176457359 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff9d00afa20 0x7ff9d0128c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:42.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.482+0000 7ff9ceffd640 1 --2- 172.21.15.57:0/176457359 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff9d00b0530 0x7ff9d006b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:42.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.482+0000 7ff9cffff640 1 -- 172.21.15.57:0/176457359 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff9d00afa20 msgr2=0x7ff9d0128c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:42.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.482+0000 7ff9cffff640 1 --2- 172.21.15.57:0/176457359 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff9d00afa20 0x7ff9d0128c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:42.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.482+0000 7ff9cffff640 1 -- 172.21.15.57:0/176457359 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff9d00b0530 msgr2=0x7ff9d006b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:42.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.482+0000 7ff9cffff640 1 --2- 172.21.15.57:0/176457359 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff9d00b0530 0x7ff9d006b740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:42.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.482+0000 7ff9cffff640 1 -- 172.21.15.57:0/176457359 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9c4009490 con 0x7ff9d00cbd20 2023-10-19T12:45:42.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.482+0000 7ff9ceffd640 1 --2- 172.21.15.57:0/176457359 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff9d00b0530 0x7ff9d006b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:45:42.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.482+0000 7ff9cf7fe640 1 --2- 172.21.15.57:0/176457359 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff9d00afa20 0x7ff9d0128c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:45:42.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.482+0000 7ff9cffff640 1 --2- 172.21.15.57:0/176457359 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9d00cbd20 0x7ff9d0141fa0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7ff9c800eaa0 tx=0x7ff9c800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:42.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.482+0000 7ff9b7fff640 1 -- 172.21.15.57:0/176457359 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9c800cdb0 con 0x7ff9d00cbd20 2023-10-19T12:45:42.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.482+0000 7ff9b7fff640 1 -- 172.21.15.57:0/176457359 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7ff9c8004b20 con 0x7ff9d00cbd20 2023-10-19T12:45:42.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.482+0000 7ff9b7fff640 1 -- 172.21.15.57:0/176457359 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9c8005020 con 0x7ff9d00cbd20 2023-10-19T12:45:42.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.482+0000 7ff9d5c75640 1 -- 172.21.15.57:0/176457359 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9d0144350 con 0x7ff9d00cbd20 2023-10-19T12:45:42.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.482+0000 7ff9d5c75640 1 -- 172.21.15.57:0/176457359 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9d01291c0 con 0x7ff9d00cbd20 2023-10-19T12:45:42.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.483+0000 7ff9d5c75640 1 -- 172.21.15.57:0/176457359 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9d0144b50 con 0x7ff9d00cbd20 2023-10-19T12:45:42.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.483+0000 7ff9b7fff640 1 -- 172.21.15.57:0/176457359 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7ff9c8005180 con 0x7ff9d00cbd20 2023-10-19T12:45:42.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.484+0000 7ff9b7fff640 1 --2- 172.21.15.57:0/176457359 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7ff9bc06ae40 0x7ff9bc06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:42.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.484+0000 7ff9b7fff640 1 -- 172.21.15.57:0/176457359 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7ff9c808a110 con 0x7ff9d00cbd20 2023-10-19T12:45:42.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.484+0000 7ff9cf7fe640 1 --2- 172.21.15.57:0/176457359 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7ff9bc06ae40 0x7ff9bc06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:42.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.485+0000 7ff9cf7fe640 1 --2- 172.21.15.57:0/176457359 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7ff9bc06ae40 0x7ff9bc06d300 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7ff9c0000f60 tx=0x7ff9c0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:42.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.488+0000 7ff9b7fff640 1 -- 172.21.15.57:0/176457359 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff9d0144b50 con 0x7ff9d00cbd20 2023-10-19T12:45:42.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:42.701+0000 7ff9d5c75640 1 -- 172.21.15.57:0/176457359 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7ff9d0144b50 con 0x7ff9d00cbd20 2023-10-19T12:45:43.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.363+0000 7ff9b7fff640 1 -- 172.21.15.57:0/176457359 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v16) v1 ==== 122+0+0 (secure 0 0 0) 0x7ff9d0144b50 con 0x7ff9d00cbd20 2023-10-19T12:45:43.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.366+0000 7ff9d5c75640 1 -- 172.21.15.57:0/176457359 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7ff9bc06ae40 msgr2=0x7ff9bc06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:43.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.366+0000 7ff9d5c75640 1 --2- 172.21.15.57:0/176457359 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7ff9bc06ae40 0x7ff9bc06d300 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7ff9c0000f60 tx=0x7ff9c0009040 comp rx=0 tx=0).stop 2023-10-19T12:45:43.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.366+0000 7ff9d5c75640 1 -- 172.21.15.57:0/176457359 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9d00cbd20 msgr2=0x7ff9d0141fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:43.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.366+0000 7ff9d5c75640 1 --2- 172.21.15.57:0/176457359 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9d00cbd20 0x7ff9d0141fa0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7ff9c800eaa0 tx=0x7ff9c800ef60 comp rx=0 tx=0).stop 2023-10-19T12:45:43.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.366+0000 7ff9d5c75640 1 -- 172.21.15.57:0/176457359 shutdown_connections 2023-10-19T12:45:43.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.366+0000 7ff9d5c75640 1 --2- 172.21.15.57:0/176457359 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7ff9bc06ae40 0x7ff9bc06d300 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:43.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.366+0000 7ff9d5c75640 1 --2- 172.21.15.57:0/176457359 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9d00cbd20 0x7ff9d0141fa0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:43.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.366+0000 7ff9d5c75640 1 --2- 172.21.15.57:0/176457359 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff9d00b0530 0x7ff9d006b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:43.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.366+0000 7ff9d5c75640 1 --2- 172.21.15.57:0/176457359 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff9d00afa20 0x7ff9d0128c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:43.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.366+0000 7ff9d5c75640 1 -- 172.21.15.57:0/176457359 >> 172.21.15.57:0/176457359 conn(0x7ff9d0069930 msgr2=0x7ff9d0069d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:43.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.366+0000 7ff9d5c75640 1 -- 172.21.15.57:0/176457359 shutdown_connections 2023-10-19T12:45:43.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.366+0000 7ff9d5c75640 1 -- 172.21.15.57:0/176457359 wait complete. 2023-10-19T12:45:43.382 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-19T12:45:43.382 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T12:45:43.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.495+0000 7f39f7ae5640 1 -- 172.21.15.57:0/929672872 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f39dc004d00 msgr2=0x7f39dc005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:43.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.495+0000 7f39f7ae5640 1 --2- 172.21.15.57:0/929672872 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f39dc004d00 0x7f39dc005180 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f39e800c640 tx=0x7f39dc0057f0 comp rx=0 tx=0).stop 2023-10-19T12:45:43.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.495+0000 7f39f6082640 1 -- 172.21.15.57:0/929672872 reap_dead start 2023-10-19T12:45:43.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.495+0000 7f39f7ae5640 1 -- 172.21.15.57:0/929672872 shutdown_connections 2023-10-19T12:45:43.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.495+0000 7f39f7ae5640 1 -- 172.21.15.57:0/929672872 >> 172.21.15.57:0/929672872 conn(0x7f39f00b0670 msgr2=0x7f39f00b0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:43.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.495+0000 7f39f7ae5640 1 -- 172.21.15.57:0/929672872 shutdown_connections 2023-10-19T12:45:43.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.495+0000 7f39f7ae5640 1 -- 172.21.15.57:0/929672872 wait complete. 2023-10-19T12:45:43.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.496+0000 7f39f7ae5640 1 Processor -- start 2023-10-19T12:45:43.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.496+0000 7f39f7ae5640 1 -- start start 2023-10-19T12:45:43.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.497+0000 7f39f7ae5640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f39dc004d00 0x7f39f01f0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:43.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.497+0000 7f39f7ae5640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f39f014c420 0x7f39f00b4100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:43.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.497+0000 7f39f7ae5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f39f00b4640 0x7f39f0149dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:43.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.497+0000 7f39f7ae5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39f00b3060 con 0x7f39f00b4640 2023-10-19T12:45:43.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.497+0000 7f39f7ae5640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39f00b3340 con 0x7f39f014c420 2023-10-19T12:45:43.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.497+0000 7f39f7ae5640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39f00b3620 con 0x7f39dc004d00 2023-10-19T12:45:43.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.497+0000 7f39f6082640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f39f00b4640 0x7f39f0149dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:43.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.497+0000 7f39f6082640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f39f00b4640 0x7f39f0149dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59306/0 (socket says 172.21.15.57:59306) 2023-10-19T12:45:43.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.497+0000 7f39f6082640 1 -- 172.21.15.57:0/1014323628 learned_addr learned my addr 172.21.15.57:0/1014323628 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:43.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.497+0000 7f39f5080640 1 --2- 172.21.15.57:0/1014323628 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f39f014c420 0x7f39f00b4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:43.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.497+0000 7f39f5881640 1 --2- 172.21.15.57:0/1014323628 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f39dc004d00 0x7f39f01f0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:43.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.497+0000 7f39f6082640 1 -- 172.21.15.57:0/1014323628 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f39dc004d00 msgr2=0x7f39f01f0cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:43.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.497+0000 7f39f6082640 1 --2- 172.21.15.57:0/1014323628 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f39dc004d00 0x7f39f01f0cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:43.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.497+0000 7f39f6082640 1 -- 172.21.15.57:0/1014323628 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f39f014c420 msgr2=0x7f39f00b4100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:43.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.497+0000 7f39f6082640 1 --2- 172.21.15.57:0/1014323628 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f39f014c420 0x7f39f00b4100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:43.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.497+0000 7f39f6082640 1 -- 172.21.15.57:0/1014323628 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39f00b3900 con 0x7f39f00b4640 2023-10-19T12:45:43.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.498+0000 7f39f6082640 1 --2- 172.21.15.57:0/1014323628 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f39f00b4640 0x7f39f0149dc0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f39e800c760 tx=0x7f39e80019e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:43.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.498+0000 7f39e27fc640 1 -- 172.21.15.57:0/1014323628 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39e8006480 con 0x7f39f00b4640 2023-10-19T12:45:43.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.498+0000 7f39e27fc640 1 -- 172.21.15.57:0/1014323628 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f39e8006600 con 0x7f39f00b4640 2023-10-19T12:45:43.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.498+0000 7f39e27fc640 1 -- 172.21.15.57:0/1014323628 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39e8002020 con 0x7f39f00b4640 2023-10-19T12:45:43.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.498+0000 7f39f7ae5640 1 -- 172.21.15.57:0/1014323628 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39f00b3be0 con 0x7f39f00b4640 2023-10-19T12:45:43.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.498+0000 7f39f7ae5640 1 -- 172.21.15.57:0/1014323628 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39f01f11f0 con 0x7f39f00b4640 2023-10-19T12:45:43.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.499+0000 7f39f7ae5640 1 -- 172.21.15.57:0/1014323628 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39f00b3dd0 con 0x7f39f00b4640 2023-10-19T12:45:43.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.499+0000 7f39e27fc640 1 -- 172.21.15.57:0/1014323628 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f39e80353f0 con 0x7f39f00b4640 2023-10-19T12:45:43.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.500+0000 7f39e27fc640 1 --2- 172.21.15.57:0/1014323628 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f39d406ae70 0x7f39d406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:43.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.500+0000 7f39e27fc640 1 -- 172.21.15.57:0/1014323628 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f39e80ab560 con 0x7f39f00b4640 2023-10-19T12:45:43.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.500+0000 7f39f5881640 1 --2- 172.21.15.57:0/1014323628 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f39d406ae70 0x7f39d406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:43.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.500+0000 7f39f5881640 1 --2- 172.21.15.57:0/1014323628 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f39d406ae70 0x7f39d406d330 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f39ec004610 tx=0x7f39ec00b2e0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:43.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.504+0000 7f39e27fc640 1 -- 172.21.15.57:0/1014323628 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f39e80764e0 con 0x7f39f00b4640 2023-10-19T12:45:43.682 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T12:45:43.685 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T12:45:43.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.716+0000 7f39f7ae5640 1 -- 172.21.15.57:0/1014323628 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f39f00690e0 con 0x7f39f00b4640 2023-10-19T12:45:43.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.716+0000 7f39e27fc640 1 -- 172.21.15.57:0/1014323628 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4916 (secure 0 0 0) 0x7f39e8079b00 con 0x7f39f00b4640 2023-10-19T12:45:43.719 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:45:43.720 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5024,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/808517451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":808517451},{"type":"v1","addr":"172.21.15.179:6836","nonce":808517451}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":5625,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/3207366629","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":3207366629},{"type":"v1","addr":"172.21.15.57:6834","nonce":3207366629}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":5628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/2353910009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2353910009},{"type":"v1","addr":"172.21.15.57:6835","nonce":2353910009}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T12:45:40.706506+0000","modified":"2023-10-19T12:45:42.702983+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5634},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5634":{"gid":5634,"name":"d","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.179:6837/1237463333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":1237463333},{"type":"v1","addr":"172.21.15.179:6837","nonce":1237463333}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-19T12:45:43.720 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 16 2023-10-19T12:45:43.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.718+0000 7f39f7ae5640 1 -- 172.21.15.57:0/1014323628 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f39d406ae70 msgr2=0x7f39d406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:43.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.718+0000 7f39f7ae5640 1 --2- 172.21.15.57:0/1014323628 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f39d406ae70 0x7f39d406d330 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f39ec004610 tx=0x7f39ec00b2e0 comp rx=0 tx=0).stop 2023-10-19T12:45:43.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.718+0000 7f39f7ae5640 1 -- 172.21.15.57:0/1014323628 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f39f00b4640 msgr2=0x7f39f0149dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:43.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.718+0000 7f39f7ae5640 1 --2- 172.21.15.57:0/1014323628 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f39f00b4640 0x7f39f0149dc0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f39e800c760 tx=0x7f39e80019e0 comp rx=0 tx=0).stop 2023-10-19T12:45:43.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.718+0000 7f39f7ae5640 1 -- 172.21.15.57:0/1014323628 shutdown_connections 2023-10-19T12:45:43.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.718+0000 7f39f7ae5640 1 --2- 172.21.15.57:0/1014323628 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f39d406ae70 0x7f39d406d330 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:43.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.718+0000 7f39f7ae5640 1 --2- 172.21.15.57:0/1014323628 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f39f00b4640 0x7f39f0149dc0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:43.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.718+0000 7f39f7ae5640 1 --2- 172.21.15.57:0/1014323628 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f39f014c420 0x7f39f00b4100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:43.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.718+0000 7f39f7ae5640 1 --2- 172.21.15.57:0/1014323628 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f39dc004d00 0x7f39f01f0cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:43.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.718+0000 7f39f7ae5640 1 -- 172.21.15.57:0/1014323628 >> 172.21.15.57:0/1014323628 conn(0x7f39f00b0670 msgr2=0x7f39f01e2b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:43.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.718+0000 7f39f7ae5640 1 -- 172.21.15.57:0/1014323628 shutdown_connections 2023-10-19T12:45:43.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.718+0000 7f39f7ae5640 1 -- 172.21.15.57:0/1014323628 wait complete. 2023-10-19T12:45:43.731 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T12:45:43.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.877+0000 7f23cf1f5640 1 -- 172.21.15.57:0/3871012343 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f23c80abc10 msgr2=0x7f23c80c9b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:43.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.877+0000 7f23cf1f5640 1 --2- 172.21.15.57:0/3871012343 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f23c80abc10 0x7f23c80c9b60 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f23b0005fd0 tx=0x7f23b002bd10 comp rx=0 tx=0).stop 2023-10-19T12:45:43.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.877+0000 7f23cf1f5640 1 -- 172.21.15.57:0/3871012343 shutdown_connections 2023-10-19T12:45:43.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.877+0000 7f23cf1f5640 1 --2- 172.21.15.57:0/3871012343 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f23c80abc10 0x7f23c80c9b60 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:43.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.877+0000 7f23cf1f5640 1 --2- 172.21.15.57:0/3871012343 >> v2:172.21.15.179:3300/0 conn(0x7f23c806af00 0x7f23c806b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:43.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.877+0000 7f23cf1f5640 1 -- 172.21.15.57:0/3871012343 >> 172.21.15.57:0/3871012343 conn(0x7f23c8069930 msgr2=0x7f23c8069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:43.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.877+0000 7f23cf1f5640 1 -- 172.21.15.57:0/3871012343 shutdown_connections 2023-10-19T12:45:43.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.877+0000 7f23cf1f5640 1 -- 172.21.15.57:0/3871012343 wait complete. 2023-10-19T12:45:43.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.878+0000 7f23cf1f5640 1 Processor -- start 2023-10-19T12:45:43.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.878+0000 7f23cf1f5640 1 -- start start 2023-10-19T12:45:43.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.879+0000 7f23cf1f5640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f23c806af00 0x7f23c8138ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:43.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.879+0000 7f23cf1f5640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f23c80abc10 0x7f23c806f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:43.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.879+0000 7f23cf1f5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f23c80c96b0 0x7f23c80ad9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:43.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.879+0000 7f23cf1f5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23c80ac690 con 0x7f23c80c96b0 2023-10-19T12:45:43.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.879+0000 7f23cf1f5640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23c8139430 con 0x7f23c80abc10 2023-10-19T12:45:43.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.879+0000 7f23cf1f5640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23c80aaa40 con 0x7f23c806af00 2023-10-19T12:45:43.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.879+0000 7f23cd792640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f23c80c96b0 0x7f23c80ad9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:43.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.879+0000 7f23cd792640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f23c80c96b0 0x7f23c80ad9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59320/0 (socket says 172.21.15.57:59320) 2023-10-19T12:45:43.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.879+0000 7f23cd792640 1 -- 172.21.15.57:0/1258858371 learned_addr learned my addr 172.21.15.57:0/1258858371 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:43.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.879+0000 7f23c7fff640 1 --2- 172.21.15.57:0/1258858371 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f23c80abc10 0x7f23c806f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:43.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.879+0000 7f23ccf91640 1 --2- 172.21.15.57:0/1258858371 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f23c806af00 0x7f23c8138ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:43.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.879+0000 7f23cd792640 1 -- 172.21.15.57:0/1258858371 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f23c806af00 msgr2=0x7f23c8138ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:43.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.879+0000 7f23cd792640 1 --2- 172.21.15.57:0/1258858371 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f23c806af00 0x7f23c8138ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:43.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.879+0000 7f23cd792640 1 -- 172.21.15.57:0/1258858371 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f23c80abc10 msgr2=0x7f23c806f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:43.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.879+0000 7f23cd792640 1 --2- 172.21.15.57:0/1258858371 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f23c80abc10 0x7f23c806f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:43.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.879+0000 7f23cd792640 1 -- 172.21.15.57:0/1258858371 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23b0009560 con 0x7f23c80c96b0 2023-10-19T12:45:43.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.879+0000 7f23ccf91640 1 --2- 172.21.15.57:0/1258858371 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f23c806af00 0x7f23c8138ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:45:43.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.880+0000 7f23cd792640 1 --2- 172.21.15.57:0/1258858371 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f23c80c96b0 0x7f23c80ad9e0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f23b0006010 tx=0x7f23b002da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:43.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.880+0000 7f23c57fa640 1 -- 172.21.15.57:0/1258858371 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23b002de90 con 0x7f23c80c96b0 2023-10-19T12:45:43.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.880+0000 7f23c57fa640 1 -- 172.21.15.57:0/1258858371 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f23b0003700 con 0x7f23c80c96b0 2023-10-19T12:45:43.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.880+0000 7f23cf1f5640 1 -- 172.21.15.57:0/1258858371 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23c80aad20 con 0x7f23c80c96b0 2023-10-19T12:45:43.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.880+0000 7f23c57fa640 1 -- 172.21.15.57:0/1258858371 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23b0003b10 con 0x7f23c80c96b0 2023-10-19T12:45:43.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.880+0000 7f23cf1f5640 1 -- 172.21.15.57:0/1258858371 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23c80ab2e0 con 0x7f23c80c96b0 2023-10-19T12:45:43.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.881+0000 7f23cf1f5640 1 -- 172.21.15.57:0/1258858371 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23c80ac800 con 0x7f23c80c96b0 2023-10-19T12:45:43.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.881+0000 7f23c57fa640 1 -- 172.21.15.57:0/1258858371 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f23b0003c70 con 0x7f23c80c96b0 2023-10-19T12:45:43.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.882+0000 7f23c57fa640 1 --2- 172.21.15.57:0/1258858371 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f23a806ae40 0x7f23a806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:43.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.882+0000 7f23c57fa640 1 -- 172.21.15.57:0/1258858371 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f23b00ac500 con 0x7f23c80c96b0 2023-10-19T12:45:43.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.882+0000 7f23ccf91640 1 --2- 172.21.15.57:0/1258858371 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f23a806ae40 0x7f23a806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:43.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.882+0000 7f23ccf91640 1 --2- 172.21.15.57:0/1258858371 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f23a806ae40 0x7f23a806d300 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f23c0006fd0 tx=0x7f23c0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:43.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:43.886+0000 7f23c57fa640 1 -- 172.21.15.57:0/1258858371 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f23c80ac800 con 0x7f23c80c96b0 2023-10-19T12:45:44.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.063+0000 7f23cf1f5640 1 -- 172.21.15.57:0/1258858371 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f23c80ac7d0 con 0x7f23c80c96b0 2023-10-19T12:45:44.067 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.064+0000 7f23c57fa640 1 -- 172.21.15.57:0/1258858371 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) v1 ==== 74+0+16786 (secure 0 0 0) 0x7f23b00775c0 con 0x7f23c80c96b0 2023-10-19T12:45:44.067 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:45:44.068 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":39,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T12:45:42.352881+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2023-10-19T12:45:38.964640+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2023-10-19T12:45:39.683197+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T12:45:44.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.066+0000 7f23cf1f5640 1 -- 172.21.15.57:0/1258858371 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f23a806ae40 msgr2=0x7f23a806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:44.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.066+0000 7f23cf1f5640 1 --2- 172.21.15.57:0/1258858371 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f23a806ae40 0x7f23a806d300 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f23c0006fd0 tx=0x7f23c0008040 comp rx=0 tx=0).stop 2023-10-19T12:45:44.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.066+0000 7f23cf1f5640 1 -- 172.21.15.57:0/1258858371 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f23c80c96b0 msgr2=0x7f23c80ad9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:44.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.066+0000 7f23cf1f5640 1 --2- 172.21.15.57:0/1258858371 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f23c80c96b0 0x7f23c80ad9e0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f23b0006010 tx=0x7f23b002da90 comp rx=0 tx=0).stop 2023-10-19T12:45:44.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.066+0000 7f23cf1f5640 1 -- 172.21.15.57:0/1258858371 shutdown_connections 2023-10-19T12:45:44.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.066+0000 7f23cf1f5640 1 --2- 172.21.15.57:0/1258858371 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f23a806ae40 0x7f23a806d300 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:44.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.066+0000 7f23cf1f5640 1 --2- 172.21.15.57:0/1258858371 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f23c80c96b0 0x7f23c80ad9e0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:44.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.066+0000 7f23cf1f5640 1 --2- 172.21.15.57:0/1258858371 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f23c80abc10 0x7f23c806f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:44.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.066+0000 7f23cf1f5640 1 --2- 172.21.15.57:0/1258858371 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f23c806af00 0x7f23c8138ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:44.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.066+0000 7f23cf1f5640 1 -- 172.21.15.57:0/1258858371 >> 172.21.15.57:0/1258858371 conn(0x7f23c8069930 msgr2=0x7f23c8069d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:44.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.066+0000 7f23cf1f5640 1 -- 172.21.15.57:0/1258858371 shutdown_connections 2023-10-19T12:45:44.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.066+0000 7f23cf1f5640 1 -- 172.21.15.57:0/1258858371 wait complete. 2023-10-19T12:45:44.080 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-19T12:45:44.080 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-19T12:45:44.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.187+0000 7f7b6aaa9640 1 -- 172.21.15.57:0/1328638140 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7b500047c0 msgr2=0x7f7b50004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:44.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.187+0000 7f7b6aaa9640 1 --2- 172.21.15.57:0/1328638140 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7b500047c0 0x7f7b50004c40 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f7b54009b50 tx=0x7f7b500057f0 comp rx=0 tx=0).stop 2023-10-19T12:45:44.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.187+0000 7f7b69046640 1 -- 172.21.15.57:0/1328638140 reap_dead start 2023-10-19T12:45:44.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.187+0000 7f7b6aaa9640 1 -- 172.21.15.57:0/1328638140 shutdown_connections 2023-10-19T12:45:44.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.187+0000 7f7b6aaa9640 1 -- 172.21.15.57:0/1328638140 >> 172.21.15.57:0/1328638140 conn(0x7f7b640aec40 msgr2=0x7f7b640af050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:44.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.187+0000 7f7b6aaa9640 1 -- 172.21.15.57:0/1328638140 shutdown_connections 2023-10-19T12:45:44.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.187+0000 7f7b6aaa9640 1 -- 172.21.15.57:0/1328638140 wait complete. 2023-10-19T12:45:44.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.188+0000 7f7b6aaa9640 1 Processor -- start 2023-10-19T12:45:44.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.188+0000 7f7b6aaa9640 1 -- start start 2023-10-19T12:45:44.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.189+0000 7f7b6aaa9640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7b500047c0 0x7f7b64148da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:44.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.189+0000 7f7b6aaa9640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7b641f02c0 0x7f7b6414abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:44.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.189+0000 7f7b6aaa9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7b641efb80 0x7f7b6406c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:44.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.189+0000 7f7b6aaa9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b64156290 con 0x7f7b641efb80 2023-10-19T12:45:44.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.189+0000 7f7b6aaa9640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b6414b110 con 0x7f7b641f02c0 2023-10-19T12:45:44.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.189+0000 7f7b6aaa9640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b641f0780 con 0x7f7b500047c0 2023-10-19T12:45:44.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.189+0000 7f7b69046640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7b641efb80 0x7f7b6406c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:44.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.189+0000 7f7b69046640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7b641efb80 0x7f7b6406c530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59338/0 (socket says 172.21.15.57:59338) 2023-10-19T12:45:44.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.189+0000 7f7b69046640 1 -- 172.21.15.57:0/2838281317 learned_addr learned my addr 172.21.15.57:0/2838281317 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:44.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.189+0000 7f7b63fff640 1 --2- 172.21.15.57:0/2838281317 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7b641f02c0 0x7f7b6414abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:44.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.189+0000 7f7b68845640 1 --2- 172.21.15.57:0/2838281317 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7b500047c0 0x7f7b64148da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:44.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.189+0000 7f7b69046640 1 -- 172.21.15.57:0/2838281317 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7b500047c0 msgr2=0x7f7b64148da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:44.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.189+0000 7f7b69046640 1 --2- 172.21.15.57:0/2838281317 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7b500047c0 0x7f7b64148da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:44.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.189+0000 7f7b69046640 1 -- 172.21.15.57:0/2838281317 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7b641f02c0 msgr2=0x7f7b6414abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:44.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.189+0000 7f7b69046640 1 --2- 172.21.15.57:0/2838281317 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7b641f02c0 0x7f7b6414abd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:44.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.189+0000 7f7b69046640 1 -- 172.21.15.57:0/2838281317 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b641f0a60 con 0x7f7b641efb80 2023-10-19T12:45:44.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.189+0000 7f7b68845640 1 --2- 172.21.15.57:0/2838281317 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7b500047c0 0x7f7b64148da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:45:44.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.189+0000 7f7b69046640 1 --2- 172.21.15.57:0/2838281317 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7b641efb80 0x7f7b6406c530 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f7b6406ad10 tx=0x7f7b4c001630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:44.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.189+0000 7f7b617fa640 1 -- 172.21.15.57:0/2838281317 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b4c017030 con 0x7f7b641efb80 2023-10-19T12:45:44.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.190+0000 7f7b617fa640 1 -- 172.21.15.57:0/2838281317 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f7b4c011430 con 0x7f7b641efb80 2023-10-19T12:45:44.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.190+0000 7f7b6aaa9640 1 -- 172.21.15.57:0/2838281317 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b641f0d40 con 0x7f7b641efb80 2023-10-19T12:45:44.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.190+0000 7f7b617fa640 1 -- 172.21.15.57:0/2838281317 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b4c011840 con 0x7f7b641efb80 2023-10-19T12:45:44.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.190+0000 7f7b6aaa9640 1 -- 172.21.15.57:0/2838281317 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b641f1300 con 0x7f7b641efb80 2023-10-19T12:45:44.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.191+0000 7f7b6aaa9640 1 -- 172.21.15.57:0/2838281317 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b640690e0 con 0x7f7b641efb80 2023-10-19T12:45:44.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.191+0000 7f7b617fa640 1 -- 172.21.15.57:0/2838281317 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f7b4c01a070 con 0x7f7b641efb80 2023-10-19T12:45:44.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.191+0000 7f7b617fa640 1 --2- 172.21.15.57:0/2838281317 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f7b4406ae40 0x7f7b4406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:44.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.192+0000 7f7b617fa640 1 -- 172.21.15.57:0/2838281317 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f7b4c018070 con 0x7f7b641efb80 2023-10-19T12:45:44.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.192+0000 7f7b68845640 1 --2- 172.21.15.57:0/2838281317 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f7b4406ae40 0x7f7b4406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:44.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.192+0000 7f7b68845640 1 --2- 172.21.15.57:0/2838281317 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f7b4406ae40 0x7f7b4406d300 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c001560 tx=0x7f7b5c001ff0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:44.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.196+0000 7f7b617fa640 1 -- 172.21.15.57:0/2838281317 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7b4c054b20 con 0x7f7b641efb80 2023-10-19T12:45:44.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.369+0000 7f7b6aaa9640 1 -- 172.21.15.57:0/2838281317 --> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7b640b1360 con 0x7f7b4406ae40 2023-10-19T12:45:44.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.378+0000 7f7b617fa640 1 -- 172.21.15.57:0/2838281317 <== mgr.4100 v2:172.21.15.179:6832/2609646285 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286395 (secure 0 0 0) 0x7f7b5c003e00 con 0x7f7b4406ae40 2023-10-19T12:45:44.383 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:45:44.385 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2023-10-19T12:45:44.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.381+0000 7f7b6aaa9640 1 -- 172.21.15.57:0/2838281317 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f7b4406ae40 msgr2=0x7f7b4406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:44.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.381+0000 7f7b6aaa9640 1 --2- 172.21.15.57:0/2838281317 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f7b4406ae40 0x7f7b4406d300 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c001560 tx=0x7f7b5c001ff0 comp rx=0 tx=0).stop 2023-10-19T12:45:44.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.381+0000 7f7b6aaa9640 1 -- 172.21.15.57:0/2838281317 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7b641efb80 msgr2=0x7f7b6406c530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:44.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.381+0000 7f7b6aaa9640 1 --2- 172.21.15.57:0/2838281317 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7b641efb80 0x7f7b6406c530 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f7b6406ad10 tx=0x7f7b4c001630 comp rx=0 tx=0).stop 2023-10-19T12:45:44.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.381+0000 7f7b6aaa9640 1 -- 172.21.15.57:0/2838281317 shutdown_connections 2023-10-19T12:45:44.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.381+0000 7f7b6aaa9640 1 --2- 172.21.15.57:0/2838281317 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f7b4406ae40 0x7f7b4406d300 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:44.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.381+0000 7f7b6aaa9640 1 --2- 172.21.15.57:0/2838281317 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7b641efb80 0x7f7b6406c530 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:44.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.381+0000 7f7b6aaa9640 1 --2- 172.21.15.57:0/2838281317 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7b641f02c0 0x7f7b6414abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:44.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.381+0000 7f7b6aaa9640 1 --2- 172.21.15.57:0/2838281317 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7b500047c0 0x7f7b64148da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:44.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.381+0000 7f7b6aaa9640 1 -- 172.21.15.57:0/2838281317 >> 172.21.15.57:0/2838281317 conn(0x7f7b640aec40 msgr2=0x7f7b6414ca50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:44.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.381+0000 7f7b6aaa9640 1 -- 172.21.15.57:0/2838281317 shutdown_connections 2023-10-19T12:45:44.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.381+0000 7f7b6aaa9640 1 -- 172.21.15.57:0/2838281317 wait complete. 2023-10-19T12:45:44.394 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":81,"stamp":"2023-10-19T12:45:43.336334+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590438,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":168,"num_write_kb":2417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":126,"ondisk_log_size":126,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":56,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":237300,"kb_used_data":4496,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754737420,"statfs":{"total":773094113280,"available":772851118080,"internally_reserved":0,"allocated":4603904,"data_stored":2628204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":238276153},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":70,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.008337"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.347634+0000","last_change":"2023-10-19T12:45:40.347634+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.347634+0000","last_undegraded":"2023-10-19T12:45:40.347634+0000","last_fullsized":"2023-10-19T12:45:40.347634+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.349030+0000","last_change":"2023-10-19T12:45:40.349030+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.349030+0000","last_undegraded":"2023-10-19T12:45:40.349030+0000","last_fullsized":"2023-10-19T12:45:40.349030+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.376066+0000","last_change":"2023-10-19T12:45:41.371589+0000","last_active":"2023-10-19T12:45:41.376066+0000","last_peered":"2023-10-19T12:45:41.376066+0000","last_clean":"2023-10-19T12:45:41.376066+0000","last_became_active":"2023-10-19T12:45:41.371287+0000","last_became_peered":"2023-10-19T12:45:41.371287+0000","last_unstale":"2023-10-19T12:45:41.376066+0000","last_undegraded":"2023-10-19T12:45:41.376066+0000","last_fullsized":"2023-10-19T12:45:41.376066+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:16:37.419534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.370328+0000","last_change":"2023-10-19T12:45:41.363930+0000","last_active":"2023-10-19T12:45:41.370328+0000","last_peered":"2023-10-19T12:45:41.370328+0000","last_clean":"2023-10-19T12:45:41.370328+0000","last_became_active":"2023-10-19T12:45:41.363668+0000","last_became_peered":"2023-10-19T12:45:41.363668+0000","last_unstale":"2023-10-19T12:45:41.370328+0000","last_undegraded":"2023-10-19T12:45:41.370328+0000","last_fullsized":"2023-10-19T12:45:41.370328+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:51:05.063828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.370788+0000","last_change":"2023-10-19T12:45:41.363093+0000","last_active":"2023-10-19T12:45:41.370788+0000","last_peered":"2023-10-19T12:45:41.370788+0000","last_clean":"2023-10-19T12:45:41.370788+0000","last_became_active":"2023-10-19T12:45:41.362804+0000","last_became_peered":"2023-10-19T12:45:41.362804+0000","last_unstale":"2023-10-19T12:45:41.370788+0000","last_undegraded":"2023-10-19T12:45:41.370788+0000","last_fullsized":"2023-10-19T12:45:41.370788+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:42:33.722992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.374791+0000","last_change":"2023-10-19T12:45:41.359415+0000","last_active":"2023-10-19T12:45:41.374791+0000","last_peered":"2023-10-19T12:45:41.374791+0000","last_clean":"2023-10-19T12:45:41.374791+0000","last_became_active":"2023-10-19T12:45:41.359120+0000","last_became_peered":"2023-10-19T12:45:41.359120+0000","last_unstale":"2023-10-19T12:45:41.374791+0000","last_undegraded":"2023-10-19T12:45:41.374791+0000","last_fullsized":"2023-10-19T12:45:41.374791+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:48:25.218615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":3,"reported_epoch":38,"state":"creating+activating","last_fresh":"2023-10-19T12:45:41.350760+0000","last_change":"2023-10-19T12:45:41.350760+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:41.350760+0000","last_undegraded":"2023-10-19T12:45:41.350760+0000","last_fullsized":"2023-10-19T12:45:41.350760+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:38:37.793736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.349543+0000","last_change":"2023-10-19T12:45:40.349543+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.349543+0000","last_undegraded":"2023-10-19T12:45:40.349543+0000","last_fullsized":"2023-10-19T12:45:40.349543+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.375561+0000","last_change":"2023-10-19T12:45:41.368731+0000","last_active":"2023-10-19T12:45:41.375561+0000","last_peered":"2023-10-19T12:45:41.375561+0000","last_clean":"2023-10-19T12:45:41.375561+0000","last_became_active":"2023-10-19T12:45:41.368006+0000","last_became_peered":"2023-10-19T12:45:41.368006+0000","last_unstale":"2023-10-19T12:45:41.375561+0000","last_undegraded":"2023-10-19T12:45:41.375561+0000","last_fullsized":"2023-10-19T12:45:41.375561+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:48:25.218615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.345142+0000","last_change":"2023-10-19T12:45:40.345142+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.345142+0000","last_undegraded":"2023-10-19T12:45:40.345142+0000","last_fullsized":"2023-10-19T12:45:40.345142+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.346201+0000","last_change":"2023-10-19T12:45:40.346201+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.346201+0000","last_undegraded":"2023-10-19T12:45:40.346201+0000","last_fullsized":"2023-10-19T12:45:40.346201+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.370682+0000","last_change":"2023-10-19T12:45:41.357975+0000","last_active":"2023-10-19T12:45:41.370682+0000","last_peered":"2023-10-19T12:45:41.370682+0000","last_clean":"2023-10-19T12:45:41.370682+0000","last_became_active":"2023-10-19T12:45:41.357717+0000","last_became_peered":"2023-10-19T12:45:41.357717+0000","last_unstale":"2023-10-19T12:45:41.370682+0000","last_undegraded":"2023-10-19T12:45:41.370682+0000","last_fullsized":"2023-10-19T12:45:41.370682+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:49:21.932303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.345686+0000","last_change":"2023-10-19T12:45:40.345686+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.345686+0000","last_undegraded":"2023-10-19T12:45:40.345686+0000","last_fullsized":"2023-10-19T12:45:40.345686+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.351066+0000","last_change":"2023-10-19T12:45:40.351066+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.351066+0000","last_undegraded":"2023-10-19T12:45:40.351066+0000","last_fullsized":"2023-10-19T12:45:40.351066+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":3,"reported_epoch":38,"state":"creating+activating","last_fresh":"2023-10-19T12:45:41.350355+0000","last_change":"2023-10-19T12:45:41.350355+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:41.350355+0000","last_undegraded":"2023-10-19T12:45:41.350355+0000","last_fullsized":"2023-10-19T12:45:41.350355+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:10:04.203620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":3,"reported_epoch":38,"state":"creating+activating","last_fresh":"2023-10-19T12:45:41.352539+0000","last_change":"2023-10-19T12:45:41.352539+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:41.352539+0000","last_undegraded":"2023-10-19T12:45:41.352539+0000","last_fullsized":"2023-10-19T12:45:41.352539+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:21:51.336716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":3,"reported_epoch":38,"state":"creating+activating","last_fresh":"2023-10-19T12:45:41.351475+0000","last_change":"2023-10-19T12:45:41.351475+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:41.351475+0000","last_undegraded":"2023-10-19T12:45:41.351475+0000","last_fullsized":"2023-10-19T12:45:41.351475+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:32:37.014614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":3,"reported_epoch":38,"state":"creating+activating","last_fresh":"2023-10-19T12:45:41.348402+0000","last_change":"2023-10-19T12:45:41.348402+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:41.348402+0000","last_undegraded":"2023-10-19T12:45:41.348402+0000","last_fullsized":"2023-10-19T12:45:41.348402+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:53:30.629136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.364662+0000","last_change":"2023-10-19T12:45:41.356683+0000","last_active":"2023-10-19T12:45:41.364662+0000","last_peered":"2023-10-19T12:45:41.364662+0000","last_clean":"2023-10-19T12:45:41.364662+0000","last_became_active":"2023-10-19T12:45:41.356404+0000","last_became_peered":"2023-10-19T12:45:41.356404+0000","last_unstale":"2023-10-19T12:45:41.364662+0000","last_undegraded":"2023-10-19T12:45:41.364662+0000","last_fullsized":"2023-10-19T12:45:41.364662+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:30:20.055674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":3,"reported_epoch":38,"state":"creating+activating","last_fresh":"2023-10-19T12:45:41.351396+0000","last_change":"2023-10-19T12:45:41.351396+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:41.351396+0000","last_undegraded":"2023-10-19T12:45:41.351396+0000","last_fullsized":"2023-10-19T12:45:41.351396+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:48:25.218615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.344680+0000","last_change":"2023-10-19T12:45:40.344680+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.344680+0000","last_undegraded":"2023-10-19T12:45:40.344680+0000","last_fullsized":"2023-10-19T12:45:40.344680+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.348310+0000","last_change":"2023-10-19T12:45:40.348310+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.348310+0000","last_undegraded":"2023-10-19T12:45:40.348310+0000","last_fullsized":"2023-10-19T12:45:40.348310+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":3,"reported_epoch":38,"state":"creating+activating","last_fresh":"2023-10-19T12:45:41.350788+0000","last_change":"2023-10-19T12:45:41.350788+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:41.350788+0000","last_undegraded":"2023-10-19T12:45:41.350788+0000","last_fullsized":"2023-10-19T12:45:41.350788+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:44:04.457483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.371142+0000","last_change":"2023-10-19T12:45:41.361356+0000","last_active":"2023-10-19T12:45:41.371142+0000","last_peered":"2023-10-19T12:45:41.371142+0000","last_clean":"2023-10-19T12:45:41.371142+0000","last_became_active":"2023-10-19T12:45:41.361082+0000","last_became_peered":"2023-10-19T12:45:41.361082+0000","last_unstale":"2023-10-19T12:45:41.371142+0000","last_undegraded":"2023-10-19T12:45:41.371142+0000","last_fullsized":"2023-10-19T12:45:41.371142+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:45:11.031973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":3,"reported_epoch":38,"state":"creating+activating","last_fresh":"2023-10-19T12:45:41.350918+0000","last_change":"2023-10-19T12:45:41.350918+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:41.350918+0000","last_undegraded":"2023-10-19T12:45:41.350918+0000","last_fullsized":"2023-10-19T12:45:41.350918+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:49:21.932303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.375784+0000","last_change":"2023-10-19T12:45:41.372032+0000","last_active":"2023-10-19T12:45:41.375784+0000","last_peered":"2023-10-19T12:45:41.375784+0000","last_clean":"2023-10-19T12:45:41.375784+0000","last_became_active":"2023-10-19T12:45:41.371582+0000","last_became_peered":"2023-10-19T12:45:41.371582+0000","last_unstale":"2023-10-19T12:45:41.375784+0000","last_undegraded":"2023-10-19T12:45:41.375784+0000","last_fullsized":"2023-10-19T12:45:41.375784+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:55:02.268260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":3,"reported_epoch":38,"state":"creating+activating","last_fresh":"2023-10-19T12:45:41.348601+0000","last_change":"2023-10-19T12:45:41.348601+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:41.348601+0000","last_undegraded":"2023-10-19T12:45:41.348601+0000","last_fullsized":"2023-10-19T12:45:41.348601+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:36:07.043432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.345780+0000","last_change":"2023-10-19T12:45:40.345780+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.345780+0000","last_undegraded":"2023-10-19T12:45:40.345780+0000","last_fullsized":"2023-10-19T12:45:40.345780+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":3,"reported_epoch":38,"state":"creating+activating","last_fresh":"2023-10-19T12:45:41.351166+0000","last_change":"2023-10-19T12:45:41.351166+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:41.351166+0000","last_undegraded":"2023-10-19T12:45:41.351166+0000","last_fullsized":"2023-10-19T12:45:41.351166+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:16:37.419534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":3,"reported_epoch":38,"state":"creating+activating","last_fresh":"2023-10-19T12:45:41.348399+0000","last_change":"2023-10-19T12:45:41.348399+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:41.348399+0000","last_undegraded":"2023-10-19T12:45:41.348399+0000","last_fullsized":"2023-10-19T12:45:41.348399+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T12:45:47.935169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":["6","3"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":3,"reported_epoch":38,"state":"creating+activating","last_fresh":"2023-10-19T12:45:41.351136+0000","last_change":"2023-10-19T12:45:41.351136+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:41.351136+0000","last_undegraded":"2023-10-19T12:45:41.351136+0000","last_fullsized":"2023-10-19T12:45:41.351136+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:10:04.203620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.369704+0000","last_change":"2023-10-19T12:45:41.356009+0000","last_active":"2023-10-19T12:45:41.369704+0000","last_peered":"2023-10-19T12:45:41.369704+0000","last_clean":"2023-10-19T12:45:41.369704+0000","last_became_active":"2023-10-19T12:45:41.355714+0000","last_became_peered":"2023-10-19T12:45:41.355714+0000","last_unstale":"2023-10-19T12:45:41.369704+0000","last_undegraded":"2023-10-19T12:45:41.369704+0000","last_fullsized":"2023-10-19T12:45:41.369704+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:16:37.419534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":3,"reported_epoch":38,"state":"creating+activating","last_fresh":"2023-10-19T12:45:41.351973+0000","last_change":"2023-10-19T12:45:41.351973+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:41.351973+0000","last_undegraded":"2023-10-19T12:45:41.351973+0000","last_fullsized":"2023-10-19T12:45:41.351973+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:53:29.708835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-19T12:45:40.364691+0000","last_change":"2023-10-19T12:45:40.359697+0000","last_active":"2023-10-19T12:45:40.364691+0000","last_peered":"2023-10-19T12:45:40.364691+0000","last_clean":"2023-10-19T12:45:40.364691+0000","last_became_active":"2023-10-19T12:45:40.359413+0000","last_became_peered":"2023-10-19T12:45:40.359413+0000","last_unstale":"2023-10-19T12:45:40.364691+0000","last_undegraded":"2023-10-19T12:45:40.364691+0000","last_fullsized":"2023-10-19T12:45:40.364691+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:36:38.697134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.375801+0000","last_change":"2023-10-19T12:45:41.372556+0000","last_active":"2023-10-19T12:45:41.375801+0000","last_peered":"2023-10-19T12:45:41.375801+0000","last_clean":"2023-10-19T12:45:41.375801+0000","last_became_active":"2023-10-19T12:45:41.372184+0000","last_became_peered":"2023-10-19T12:45:41.372184+0000","last_unstale":"2023-10-19T12:45:41.375801+0000","last_undegraded":"2023-10-19T12:45:41.375801+0000","last_fullsized":"2023-10-19T12:45:41.375801+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:10:04.203620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"38'1","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.370313+0000","last_change":"2023-10-19T12:45:40.352581+0000","last_active":"2023-10-19T12:45:41.370313+0000","last_peered":"2023-10-19T12:45:41.370313+0000","last_clean":"2023-10-19T12:45:41.370313+0000","last_became_active":"2023-10-19T12:45:40.352300+0000","last_became_peered":"2023-10-19T12:45:40.352300+0000","last_unstale":"2023-10-19T12:45:41.370313+0000","last_undegraded":"2023-10-19T12:45:41.370313+0000","last_fullsized":"2023-10-19T12:45:41.370313+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:59:58.100040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.348927+0000","last_change":"2023-10-19T12:45:40.348927+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.348927+0000","last_undegraded":"2023-10-19T12:45:40.348927+0000","last_fullsized":"2023-10-19T12:45:40.348927+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"38'2","reported_seq":13,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.374990+0000","last_change":"2023-10-19T12:45:40.355990+0000","last_active":"2023-10-19T12:45:41.374990+0000","last_peered":"2023-10-19T12:45:41.374990+0000","last_clean":"2023-10-19T12:45:41.374990+0000","last_became_active":"2023-10-19T12:45:40.355666+0000","last_became_peered":"2023-10-19T12:45:40.355666+0000","last_unstale":"2023-10-19T12:45:41.374990+0000","last_undegraded":"2023-10-19T12:45:41.374990+0000","last_fullsized":"2023-10-19T12:45:41.374990+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:53:28.702370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.349506+0000","last_change":"2023-10-19T12:45:40.349506+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.349506+0000","last_undegraded":"2023-10-19T12:45:40.349506+0000","last_fullsized":"2023-10-19T12:45:40.349506+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-19T12:45:40.367475+0000","last_change":"2023-10-19T12:45:40.364018+0000","last_active":"2023-10-19T12:45:40.367475+0000","last_peered":"2023-10-19T12:45:40.367475+0000","last_clean":"2023-10-19T12:45:40.367475+0000","last_became_active":"2023-10-19T12:45:40.363763+0000","last_became_peered":"2023-10-19T12:45:40.363763+0000","last_unstale":"2023-10-19T12:45:40.367475+0000","last_undegraded":"2023-10-19T12:45:40.367475+0000","last_fullsized":"2023-10-19T12:45:40.367475+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:36:38.697134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.363923+0000","last_change":"2023-10-19T12:45:41.355176+0000","last_active":"2023-10-19T12:45:41.363923+0000","last_peered":"2023-10-19T12:45:41.363923+0000","last_clean":"2023-10-19T12:45:41.363923+0000","last_became_active":"2023-10-19T12:45:41.354687+0000","last_became_peered":"2023-10-19T12:45:41.354687+0000","last_unstale":"2023-10-19T12:45:41.363923+0000","last_undegraded":"2023-10-19T12:45:41.363923+0000","last_fullsized":"2023-10-19T12:45:41.363923+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:53:29.708835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-19T12:45:40.362034+0000","last_change":"2023-10-19T12:45:40.357988+0000","last_active":"2023-10-19T12:45:40.362034+0000","last_peered":"2023-10-19T12:45:40.362034+0000","last_clean":"2023-10-19T12:45:40.362034+0000","last_became_active":"2023-10-19T12:45:40.357478+0000","last_became_peered":"2023-10-19T12:45:40.357478+0000","last_unstale":"2023-10-19T12:45:40.362034+0000","last_undegraded":"2023-10-19T12:45:40.362034+0000","last_fullsized":"2023-10-19T12:45:40.362034+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:49:20.925838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":3,"reported_epoch":38,"state":"creating+activating","last_fresh":"2023-10-19T12:45:41.349960+0000","last_change":"2023-10-19T12:45:41.349960+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:41.349960+0000","last_undegraded":"2023-10-19T12:45:41.349960+0000","last_fullsized":"2023-10-19T12:45:41.349960+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:44:04.457483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":11,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.349382+0000","last_change":"2023-10-19T12:45:40.355247+0000","last_active":"2023-10-19T12:45:41.349382+0000","last_peered":"2023-10-19T12:45:41.349382+0000","last_clean":"2023-10-19T12:45:41.349382+0000","last_became_active":"2023-10-19T12:45:40.354967+0000","last_became_peered":"2023-10-19T12:45:40.354967+0000","last_unstale":"2023-10-19T12:45:41.349382+0000","last_undegraded":"2023-10-19T12:45:41.349382+0000","last_fullsized":"2023-10-19T12:45:41.349382+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:48:24.212150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.350186+0000","last_change":"2023-10-19T12:45:40.350186+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.350186+0000","last_undegraded":"2023-10-19T12:45:40.350186+0000","last_fullsized":"2023-10-19T12:45:40.350186+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":11,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.348919+0000","last_change":"2023-10-19T12:45:40.359949+0000","last_active":"2023-10-19T12:45:41.348919+0000","last_peered":"2023-10-19T12:45:41.348919+0000","last_clean":"2023-10-19T12:45:41.348919+0000","last_became_active":"2023-10-19T12:45:40.359690+0000","last_became_peered":"2023-10-19T12:45:40.359690+0000","last_unstale":"2023-10-19T12:45:41.348919+0000","last_undegraded":"2023-10-19T12:45:41.348919+0000","last_fullsized":"2023-10-19T12:45:41.348919+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:44:03.451018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.351154+0000","last_change":"2023-10-19T12:45:40.351154+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.351154+0000","last_undegraded":"2023-10-19T12:45:40.351154+0000","last_fullsized":"2023-10-19T12:45:40.351154+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-19T12:45:40.364109+0000","last_change":"2023-10-19T12:45:40.360289+0000","last_active":"2023-10-19T12:45:40.364109+0000","last_peered":"2023-10-19T12:45:40.364109+0000","last_clean":"2023-10-19T12:45:40.364109+0000","last_became_active":"2023-10-19T12:45:40.359850+0000","last_became_peered":"2023-10-19T12:45:40.359850+0000","last_unstale":"2023-10-19T12:45:40.364109+0000","last_undegraded":"2023-10-19T12:45:40.364109+0000","last_fullsized":"2023-10-19T12:45:40.364109+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:30:19.049209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.370304+0000","last_change":"2023-10-19T12:45:41.363350+0000","last_active":"2023-10-19T12:45:41.370304+0000","last_peered":"2023-10-19T12:45:41.370304+0000","last_clean":"2023-10-19T12:45:41.370304+0000","last_became_active":"2023-10-19T12:45:41.363067+0000","last_became_peered":"2023-10-19T12:45:41.363067+0000","last_unstale":"2023-10-19T12:45:41.370304+0000","last_undegraded":"2023-10-19T12:45:41.370304+0000","last_fullsized":"2023-10-19T12:45:41.370304+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:09:22.266843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-19T12:45:40.363651+0000","last_change":"2023-10-19T12:45:40.356475+0000","last_active":"2023-10-19T12:45:40.363651+0000","last_peered":"2023-10-19T12:45:40.363651+0000","last_clean":"2023-10-19T12:45:40.363651+0000","last_became_active":"2023-10-19T12:45:40.356178+0000","last_became_peered":"2023-10-19T12:45:40.356178+0000","last_unstale":"2023-10-19T12:45:40.363651+0000","last_undegraded":"2023-10-19T12:45:40.363651+0000","last_fullsized":"2023-10-19T12:45:40.363651+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:59:58.100040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":3,"reported_epoch":38,"state":"creating+activating","last_fresh":"2023-10-19T12:45:41.350085+0000","last_change":"2023-10-19T12:45:41.350085+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:41.350085+0000","last_undegraded":"2023-10-19T12:45:41.350085+0000","last_fullsized":"2023-10-19T12:45:41.350085+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:29:00.286113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-19T12:45:40.364238+0000","last_change":"2023-10-19T12:45:40.360586+0000","last_active":"2023-10-19T12:45:40.364238+0000","last_peered":"2023-10-19T12:45:40.364238+0000","last_clean":"2023-10-19T12:45:40.364238+0000","last_became_active":"2023-10-19T12:45:40.360327+0000","last_became_peered":"2023-10-19T12:45:40.360327+0000","last_unstale":"2023-10-19T12:45:40.364238+0000","last_undegraded":"2023-10-19T12:45:40.364238+0000","last_fullsized":"2023-10-19T12:45:40.364238+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:55:01.261795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.375472+0000","last_change":"2023-10-19T12:45:41.366187+0000","last_active":"2023-10-19T12:45:41.375472+0000","last_peered":"2023-10-19T12:45:41.375472+0000","last_clean":"2023-10-19T12:45:41.375472+0000","last_became_active":"2023-10-19T12:45:41.365896+0000","last_became_peered":"2023-10-19T12:45:41.365896+0000","last_unstale":"2023-10-19T12:45:41.375472+0000","last_undegraded":"2023-10-19T12:45:41.375472+0000","last_fullsized":"2023-10-19T12:45:41.375472+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:55:02.268260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-19T12:45:40.361876+0000","last_change":"2023-10-19T12:45:40.358267+0000","last_active":"2023-10-19T12:45:40.361876+0000","last_peered":"2023-10-19T12:45:40.361876+0000","last_clean":"2023-10-19T12:45:40.361876+0000","last_became_active":"2023-10-19T12:45:40.358000+0000","last_became_peered":"2023-10-19T12:45:40.358000+0000","last_unstale":"2023-10-19T12:45:40.361876+0000","last_undegraded":"2023-10-19T12:45:40.361876+0000","last_fullsized":"2023-10-19T12:45:40.361876+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:48:24.212150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.348513+0000","last_change":"2023-10-19T12:45:40.348513+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.348513+0000","last_undegraded":"2023-10-19T12:45:40.348513+0000","last_fullsized":"2023-10-19T12:45:40.348513+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-19T12:45:40.364295+0000","last_change":"2023-10-19T12:45:40.361715+0000","last_active":"2023-10-19T12:45:40.364295+0000","last_peered":"2023-10-19T12:45:40.364295+0000","last_clean":"2023-10-19T12:45:40.364295+0000","last_became_active":"2023-10-19T12:45:40.361442+0000","last_became_peered":"2023-10-19T12:45:40.361442+0000","last_unstale":"2023-10-19T12:45:40.364295+0000","last_undegraded":"2023-10-19T12:45:40.364295+0000","last_fullsized":"2023-10-19T12:45:40.364295+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:44:03.451018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.375150+0000","last_change":"2023-10-19T12:45:41.364032+0000","last_active":"2023-10-19T12:45:41.375150+0000","last_peered":"2023-10-19T12:45:41.375150+0000","last_clean":"2023-10-19T12:45:41.375150+0000","last_became_active":"2023-10-19T12:45:41.363734+0000","last_became_peered":"2023-10-19T12:45:41.363734+0000","last_unstale":"2023-10-19T12:45:41.375150+0000","last_undegraded":"2023-10-19T12:45:41.375150+0000","last_fullsized":"2023-10-19T12:45:41.375150+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:32:37.014614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-19T12:45:40.363994+0000","last_change":"2023-10-19T12:45:40.360672+0000","last_active":"2023-10-19T12:45:40.363994+0000","last_peered":"2023-10-19T12:45:40.363994+0000","last_clean":"2023-10-19T12:45:40.363994+0000","last_became_active":"2023-10-19T12:45:40.360396+0000","last_became_peered":"2023-10-19T12:45:40.360396+0000","last_unstale":"2023-10-19T12:45:40.363994+0000","last_undegraded":"2023-10-19T12:45:40.363994+0000","last_fullsized":"2023-10-19T12:45:40.363994+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:28:59.279648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.347231+0000","last_change":"2023-10-19T12:45:40.347231+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.347231+0000","last_undegraded":"2023-10-19T12:45:40.347231+0000","last_fullsized":"2023-10-19T12:45:40.347231+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":11,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.348645+0000","last_change":"2023-10-19T12:45:40.359407+0000","last_active":"2023-10-19T12:45:41.348645+0000","last_peered":"2023-10-19T12:45:41.348645+0000","last_clean":"2023-10-19T12:45:41.348645+0000","last_became_active":"2023-10-19T12:45:40.359134+0000","last_became_peered":"2023-10-19T12:45:40.359134+0000","last_unstale":"2023-10-19T12:45:41.348645+0000","last_undegraded":"2023-10-19T12:45:41.348645+0000","last_fullsized":"2023-10-19T12:45:41.348645+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:59:58.100040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":3,"reported_epoch":38,"state":"creating+activating","last_fresh":"2023-10-19T12:45:41.349119+0000","last_change":"2023-10-19T12:45:41.349119+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:41.349119+0000","last_undegraded":"2023-10-19T12:45:41.349119+0000","last_fullsized":"2023-10-19T12:45:41.349119+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:16:37.419534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":11,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.350155+0000","last_change":"2023-10-19T12:45:40.359548+0000","last_active":"2023-10-19T12:45:41.350155+0000","last_peered":"2023-10-19T12:45:41.350155+0000","last_clean":"2023-10-19T12:45:41.350155+0000","last_became_active":"2023-10-19T12:45:40.359282+0000","last_became_peered":"2023-10-19T12:45:40.359282+0000","last_unstale":"2023-10-19T12:45:41.350155+0000","last_undegraded":"2023-10-19T12:45:41.350155+0000","last_fullsized":"2023-10-19T12:45:41.350155+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:39:09.771537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.350379+0000","last_change":"2023-10-19T12:45:40.350379+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.350379+0000","last_undegraded":"2023-10-19T12:45:40.350379+0000","last_fullsized":"2023-10-19T12:45:40.350379+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"38'3","reported_seq":11,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.352796+0000","last_change":"2023-10-19T12:45:40.358054+0000","last_active":"2023-10-19T12:45:41.352796+0000","last_peered":"2023-10-19T12:45:41.352796+0000","last_clean":"2023-10-19T12:45:41.352796+0000","last_became_active":"2023-10-19T12:45:40.357803+0000","last_became_peered":"2023-10-19T12:45:40.357803+0000","last_unstale":"2023-10-19T12:45:41.352796+0000","last_undegraded":"2023-10-19T12:45:41.352796+0000","last_fullsized":"2023-10-19T12:45:41.352796+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:07:55.739920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.350100+0000","last_change":"2023-10-19T12:45:40.350100+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.350100+0000","last_undegraded":"2023-10-19T12:45:40.350100+0000","last_fullsized":"2023-10-19T12:45:40.350100+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-19T12:45:40.362007+0000","last_change":"2023-10-19T12:45:40.358450+0000","last_active":"2023-10-19T12:45:40.362007+0000","last_peered":"2023-10-19T12:45:40.362007+0000","last_clean":"2023-10-19T12:45:40.362007+0000","last_became_active":"2023-10-19T12:45:40.356778+0000","last_became_peered":"2023-10-19T12:45:40.356778+0000","last_unstale":"2023-10-19T12:45:40.362007+0000","last_undegraded":"2023-10-19T12:45:40.362007+0000","last_fullsized":"2023-10-19T12:45:40.362007+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:16:36.413069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.345759+0000","last_change":"2023-10-19T12:45:40.345759+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.345759+0000","last_undegraded":"2023-10-19T12:45:40.345759+0000","last_fullsized":"2023-10-19T12:45:40.345759+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-19T12:45:40.361384+0000","last_change":"2023-10-19T12:45:40.357947+0000","last_active":"2023-10-19T12:45:40.361384+0000","last_peered":"2023-10-19T12:45:40.361384+0000","last_clean":"2023-10-19T12:45:40.361384+0000","last_became_active":"2023-10-19T12:45:40.357669+0000","last_became_peered":"2023-10-19T12:45:40.357669+0000","last_unstale":"2023-10-19T12:45:40.361384+0000","last_undegraded":"2023-10-19T12:45:40.361384+0000","last_fullsized":"2023-10-19T12:45:40.361384+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T12:45:46.928704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.350198+0000","last_change":"2023-10-19T12:45:40.350198+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.350198+0000","last_undegraded":"2023-10-19T12:45:40.350198+0000","last_fullsized":"2023-10-19T12:45:40.350198+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"38'1","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.370536+0000","last_change":"2023-10-19T12:45:40.354628+0000","last_active":"2023-10-19T12:45:41.370536+0000","last_peered":"2023-10-19T12:45:41.370536+0000","last_clean":"2023-10-19T12:45:41.370536+0000","last_became_active":"2023-10-19T12:45:40.354367+0000","last_became_peered":"2023-10-19T12:45:40.354367+0000","last_unstale":"2023-10-19T12:45:41.370536+0000","last_undegraded":"2023-10-19T12:45:41.370536+0000","last_fullsized":"2023-10-19T12:45:41.370536+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:33:49.817824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.375358+0000","last_change":"2023-10-19T12:45:41.367225+0000","last_active":"2023-10-19T12:45:41.375358+0000","last_peered":"2023-10-19T12:45:41.375358+0000","last_clean":"2023-10-19T12:45:41.375358+0000","last_became_active":"2023-10-19T12:45:41.366782+0000","last_became_peered":"2023-10-19T12:45:41.366782+0000","last_unstale":"2023-10-19T12:45:41.375358+0000","last_undegraded":"2023-10-19T12:45:41.375358+0000","last_fullsized":"2023-10-19T12:45:41.375358+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:50:19.159560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":11,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.349510+0000","last_change":"2023-10-19T12:45:40.357501+0000","last_active":"2023-10-19T12:45:41.349510+0000","last_peered":"2023-10-19T12:45:41.349510+0000","last_clean":"2023-10-19T12:45:41.349510+0000","last_became_active":"2023-10-19T12:45:40.355411+0000","last_became_peered":"2023-10-19T12:45:40.355411+0000","last_unstale":"2023-10-19T12:45:41.349510+0000","last_undegraded":"2023-10-19T12:45:41.349510+0000","last_fullsized":"2023-10-19T12:45:41.349510+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:32:36.008149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":3,"reported_epoch":38,"state":"creating+activating","last_fresh":"2023-10-19T12:45:41.353064+0000","last_change":"2023-10-19T12:45:41.353064+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:41.353064+0000","last_undegraded":"2023-10-19T12:45:41.353064+0000","last_fullsized":"2023-10-19T12:45:41.353064+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:59:55.870354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"38'2","reported_seq":13,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.376098+0000","last_change":"2023-10-19T12:45:40.357725+0000","last_active":"2023-10-19T12:45:41.376098+0000","last_peered":"2023-10-19T12:45:41.376098+0000","last_clean":"2023-10-19T12:45:41.376098+0000","last_became_active":"2023-10-19T12:45:40.357417+0000","last_became_peered":"2023-10-19T12:45:40.357417+0000","last_unstale":"2023-10-19T12:45:41.376098+0000","last_undegraded":"2023-10-19T12:45:41.376098+0000","last_fullsized":"2023-10-19T12:45:41.376098+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:26:34.780735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.375456+0000","last_change":"2023-10-19T12:45:41.366181+0000","last_active":"2023-10-19T12:45:41.375456+0000","last_peered":"2023-10-19T12:45:41.375456+0000","last_clean":"2023-10-19T12:45:41.375456+0000","last_became_active":"2023-10-19T12:45:41.365882+0000","last_became_peered":"2023-10-19T12:45:41.365882+0000","last_unstale":"2023-10-19T12:45:41.375456+0000","last_undegraded":"2023-10-19T12:45:41.375456+0000","last_fullsized":"2023-10-19T12:45:41.375456+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:29:00.286113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":11,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.343006+0000","last_change":"2023-10-19T12:45:40.355357+0000","last_active":"2023-10-19T12:45:41.343006+0000","last_peered":"2023-10-19T12:45:41.343006+0000","last_clean":"2023-10-19T12:45:41.343006+0000","last_became_active":"2023-10-19T12:45:40.355097+0000","last_became_peered":"2023-10-19T12:45:40.355097+0000","last_unstale":"2023-10-19T12:45:41.343006+0000","last_undegraded":"2023-10-19T12:45:41.343006+0000","last_fullsized":"2023-10-19T12:45:41.343006+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:23:29.827547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":3,"reported_epoch":38,"state":"creating+activating","last_fresh":"2023-10-19T12:45:41.349483+0000","last_change":"2023-10-19T12:45:41.349483+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:41.349483+0000","last_undegraded":"2023-10-19T12:45:41.349483+0000","last_fullsized":"2023-10-19T12:45:41.349483+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:59:59.106505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":11,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.351381+0000","last_change":"2023-10-19T12:45:40.360123+0000","last_active":"2023-10-19T12:45:41.351381+0000","last_peered":"2023-10-19T12:45:41.351381+0000","last_clean":"2023-10-19T12:45:41.351381+0000","last_became_active":"2023-10-19T12:45:40.359767+0000","last_became_peered":"2023-10-19T12:45:40.359767+0000","last_unstale":"2023-10-19T12:45:41.351381+0000","last_undegraded":"2023-10-19T12:45:41.351381+0000","last_fullsized":"2023-10-19T12:45:41.351381+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:38:36.787271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.346878+0000","last_change":"2023-10-19T12:45:40.346878+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.346878+0000","last_undegraded":"2023-10-19T12:45:40.346878+0000","last_fullsized":"2023-10-19T12:45:40.346878+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":11,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.343223+0000","last_change":"2023-10-19T12:45:40.357695+0000","last_active":"2023-10-19T12:45:41.343223+0000","last_peered":"2023-10-19T12:45:41.343223+0000","last_clean":"2023-10-19T12:45:41.343223+0000","last_became_active":"2023-10-19T12:45:40.357393+0000","last_became_peered":"2023-10-19T12:45:40.357393+0000","last_unstale":"2023-10-19T12:45:41.343223+0000","last_undegraded":"2023-10-19T12:45:41.343223+0000","last_fullsized":"2023-10-19T12:45:41.343223+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:43:31.256604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.375615+0000","last_change":"2023-10-19T12:45:41.371258+0000","last_active":"2023-10-19T12:45:41.375615+0000","last_peered":"2023-10-19T12:45:41.375615+0000","last_clean":"2023-10-19T12:45:41.375615+0000","last_became_active":"2023-10-19T12:45:41.370963+0000","last_became_peered":"2023-10-19T12:45:41.370963+0000","last_unstale":"2023-10-19T12:45:41.375615+0000","last_undegraded":"2023-10-19T12:45:41.375615+0000","last_fullsized":"2023-10-19T12:45:41.375615+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:38:37.793736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-19T12:45:40.361905+0000","last_change":"2023-10-19T12:45:40.358263+0000","last_active":"2023-10-19T12:45:40.361905+0000","last_peered":"2023-10-19T12:45:40.361905+0000","last_clean":"2023-10-19T12:45:40.361905+0000","last_became_active":"2023-10-19T12:45:40.357402+0000","last_became_peered":"2023-10-19T12:45:40.357402+0000","last_unstale":"2023-10-19T12:45:40.361905+0000","last_undegraded":"2023-10-19T12:45:40.361905+0000","last_fullsized":"2023-10-19T12:45:40.361905+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:28:59.279648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":3,"reported_epoch":38,"state":"creating+activating","last_fresh":"2023-10-19T12:45:41.350527+0000","last_change":"2023-10-19T12:45:41.350527+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:41.350527+0000","last_undegraded":"2023-10-19T12:45:41.350527+0000","last_fullsized":"2023-10-19T12:45:41.350527+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:59:59.106505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":11,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.343650+0000","last_change":"2023-10-19T12:45:40.352686+0000","last_active":"2023-10-19T12:45:41.343650+0000","last_peered":"2023-10-19T12:45:41.343650+0000","last_clean":"2023-10-19T12:45:41.343650+0000","last_became_active":"2023-10-19T12:45:40.352420+0000","last_became_peered":"2023-10-19T12:45:40.352420+0000","last_unstale":"2023-10-19T12:45:41.343650+0000","last_undegraded":"2023-10-19T12:45:41.343650+0000","last_fullsized":"2023-10-19T12:45:41.343650+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:36:38.697134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.346413+0000","last_change":"2023-10-19T12:45:40.346413+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.346413+0000","last_undegraded":"2023-10-19T12:45:40.346413+0000","last_fullsized":"2023-10-19T12:45:40.346413+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":11,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.348286+0000","last_change":"2023-10-19T12:45:40.358578+0000","last_active":"2023-10-19T12:45:41.348286+0000","last_peered":"2023-10-19T12:45:41.348286+0000","last_clean":"2023-10-19T12:45:41.348286+0000","last_became_active":"2023-10-19T12:45:40.358057+0000","last_became_peered":"2023-10-19T12:45:40.358057+0000","last_unstale":"2023-10-19T12:45:41.348286+0000","last_undegraded":"2023-10-19T12:45:41.348286+0000","last_fullsized":"2023-10-19T12:45:41.348286+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:38:36.787271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.375444+0000","last_change":"2023-10-19T12:45:41.367997+0000","last_active":"2023-10-19T12:45:41.375444+0000","last_peered":"2023-10-19T12:45:41.375444+0000","last_clean":"2023-10-19T12:45:41.375444+0000","last_became_active":"2023-10-19T12:45:41.367687+0000","last_became_peered":"2023-10-19T12:45:41.367687+0000","last_unstale":"2023-10-19T12:45:41.375444+0000","last_undegraded":"2023-10-19T12:45:41.375444+0000","last_fullsized":"2023-10-19T12:45:41.375444+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:10:04.203620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":11,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.348125+0000","last_change":"2023-10-19T12:45:40.360184+0000","last_active":"2023-10-19T12:45:41.348125+0000","last_peered":"2023-10-19T12:45:41.348125+0000","last_clean":"2023-10-19T12:45:41.348125+0000","last_became_active":"2023-10-19T12:45:40.359917+0000","last_became_peered":"2023-10-19T12:45:40.359917+0000","last_unstale":"2023-10-19T12:45:41.348125+0000","last_undegraded":"2023-10-19T12:45:41.348125+0000","last_fullsized":"2023-10-19T12:45:41.348125+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:44:03.451018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.374901+0000","last_change":"2023-10-19T12:45:41.364206+0000","last_active":"2023-10-19T12:45:41.374901+0000","last_peered":"2023-10-19T12:45:41.374901+0000","last_clean":"2023-10-19T12:45:41.374901+0000","last_became_active":"2023-10-19T12:45:41.363926+0000","last_became_peered":"2023-10-19T12:45:41.363926+0000","last_unstale":"2023-10-19T12:45:41.374901+0000","last_undegraded":"2023-10-19T12:45:41.374901+0000","last_fullsized":"2023-10-19T12:45:41.374901+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:29:00.286113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-19T12:45:40.364717+0000","last_change":"2023-10-19T12:45:40.360715+0000","last_active":"2023-10-19T12:45:40.364717+0000","last_peered":"2023-10-19T12:45:40.364717+0000","last_clean":"2023-10-19T12:45:40.364717+0000","last_became_active":"2023-10-19T12:45:40.359801+0000","last_became_peered":"2023-10-19T12:45:40.359801+0000","last_unstale":"2023-10-19T12:45:40.364717+0000","last_undegraded":"2023-10-19T12:45:40.364717+0000","last_fullsized":"2023-10-19T12:45:40.364717+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:45:10.025508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.350603+0000","last_change":"2023-10-19T12:45:40.350603+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.350603+0000","last_undegraded":"2023-10-19T12:45:40.350603+0000","last_fullsized":"2023-10-19T12:45:40.350603+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-19T12:45:40.367503+0000","last_change":"2023-10-19T12:45:40.364278+0000","last_active":"2023-10-19T12:45:40.367503+0000","last_peered":"2023-10-19T12:45:40.367503+0000","last_clean":"2023-10-19T12:45:40.367503+0000","last_became_active":"2023-10-19T12:45:40.363981+0000","last_became_peered":"2023-10-19T12:45:40.363981+0000","last_unstale":"2023-10-19T12:45:40.367503+0000","last_undegraded":"2023-10-19T12:45:40.367503+0000","last_fullsized":"2023-10-19T12:45:40.367503+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:50:18.153095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":3,"reported_epoch":38,"state":"creating+activating","last_fresh":"2023-10-19T12:45:41.349666+0000","last_change":"2023-10-19T12:45:41.349666+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:41.349666+0000","last_undegraded":"2023-10-19T12:45:41.349666+0000","last_fullsized":"2023-10-19T12:45:41.349666+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:55:02.268260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-19T12:45:40.364011+0000","last_change":"2023-10-19T12:45:40.359438+0000","last_active":"2023-10-19T12:45:40.364011+0000","last_peered":"2023-10-19T12:45:40.364011+0000","last_clean":"2023-10-19T12:45:40.364011+0000","last_became_active":"2023-10-19T12:45:40.359156+0000","last_became_peered":"2023-10-19T12:45:40.359156+0000","last_unstale":"2023-10-19T12:45:40.364011+0000","last_undegraded":"2023-10-19T12:45:40.364011+0000","last_fullsized":"2023-10-19T12:45:40.364011+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:10:03.197155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.344906+0000","last_change":"2023-10-19T12:45:40.344906+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.344906+0000","last_undegraded":"2023-10-19T12:45:40.344906+0000","last_fullsized":"2023-10-19T12:45:40.344906+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":11,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.351976+0000","last_change":"2023-10-19T12:45:40.359310+0000","last_active":"2023-10-19T12:45:41.351976+0000","last_peered":"2023-10-19T12:45:41.351976+0000","last_clean":"2023-10-19T12:45:41.351976+0000","last_became_active":"2023-10-19T12:45:40.359010+0000","last_became_peered":"2023-10-19T12:45:40.359010+0000","last_unstale":"2023-10-19T12:45:41.351976+0000","last_undegraded":"2023-10-19T12:45:41.351976+0000","last_fullsized":"2023-10-19T12:45:41.351976+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:57:29.725854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":179,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-19T12:45:40.349832+0000","last_change":"2023-10-19T12:44:08.636613+0000","last_active":"2023-10-19T12:45:40.349832+0000","last_peered":"2023-10-19T12:45:40.349832+0000","last_clean":"2023-10-19T12:45:40.349832+0000","last_became_active":"2023-10-19T12:44:08.635697+0000","last_became_peered":"2023-10-19T12:44:08.635697+0000","last_unstale":"2023-10-19T12:45:40.349832+0000","last_undegraded":"2023-10-19T12:45:40.349832+0000","last_fullsized":"2023-10-19T12:45:40.349832+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_clean_scrub_stamp":"2023-10-19T12:44:07.399738+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:15:23.685485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":70,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574871,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28028,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343812,"statfs":{"total":96636764160,"available":96608063488,"internally_reserved":0,"allocated":1032192,"data_stored":777587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:45:08 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.53400000000000003}]},{"osd":1,"last update":"Thu Oct 19 12:45:08 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.495}]},{"osd":2,"last update":"Thu Oct 19 12:45:08 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.64300000000000002}]},{"osd":3,"last update":"Thu Oct 19 12:45:08 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.68200000000000005}]},{"osd":4,"last update":"Thu Oct 19 12:45:08 2023","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.51200000000000001}]},{"osd":5,"last update":"Thu Oct 19 12:45:08 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.441}]},{"osd":6,"last update":"Thu Oct 19 12:45:08 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.61199999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574871,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27436,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344404,"statfs":{"total":96636764160,"available":96608669696,"internally_reserved":0,"allocated":425984,"data_stored":180124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:45:06 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.56599999999999995}]},{"osd":1,"last update":"Thu Oct 19 12:45:06 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.626},{"interface":"front","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.075,"5min":1.075,"15min":1.075},"last":0.50600000000000001}]},{"osd":2,"last update":"Thu Oct 19 12:45:06 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.59199999999999997}]},{"osd":3,"last update":"Thu Oct 19 12:45:06 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.57999999999999996}]},{"osd":4,"last update":"Thu Oct 19 12:45:06 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.371},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.32900000000000001}]},{"osd":5,"last update":"Thu Oct 19 12:45:06 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.44700000000000001}]},{"osd":7,"last update":"Thu Oct 19 12:45:06 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.48499999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574871,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27436,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344404,"statfs":{"total":96636764160,"available":96608669696,"internally_reserved":0,"allocated":425984,"data_stored":180124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.55000000000000004}]},{"osd":1,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.57999999999999996}]},{"osd":2,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.625},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.60899999999999999}]},{"osd":3,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.65400000000000003}]},{"osd":4,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.42599999999999999}]},{"osd":6,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.59199999999999997}]},{"osd":7,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.52900000000000003}]}]},{"osd":4,"up_from":13,"seq":55834574871,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27436,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344404,"statfs":{"total":96636764160,"available":96608669696,"internally_reserved":0,"allocated":425984,"data_stored":180124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:45:08 2023","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.46000000000000002}]},{"osd":1,"last update":"Thu Oct 19 12:45:08 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.57699999999999996}]},{"osd":2,"last update":"Thu Oct 19 12:45:08 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.44600000000000001}]},{"osd":3,"last update":"Thu Oct 19 12:45:08 2023","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.629}]},{"osd":5,"last update":"Thu Oct 19 12:45:08 2023","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.29099999999999998}]},{"osd":6,"last update":"Thu Oct 19 12:45:08 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.32600000000000001}]},{"osd":7,"last update":"Thu Oct 19 12:45:08 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.373},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.40999999999999998}]}]},{"osd":3,"up_from":13,"seq":55834574870,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31596,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340244,"statfs":{"total":96636764160,"available":96604409856,"internally_reserved":0,"allocated":425984,"data_stored":180124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":1.04,"5min":1.04,"15min":1.04},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.375,"5min":0.375,"15min":0.375},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"last":0.32500000000000001}]},{"osd":1,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.371},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.39700000000000002}]},{"osd":2,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.42699999999999999}]},{"osd":4,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.65600000000000003}]},{"osd":5,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.53400000000000003}]},{"osd":6,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.51700000000000002}]},{"osd":7,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.61899999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574872,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31596,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340244,"statfs":{"total":96636764160,"available":96604409856,"internally_reserved":0,"allocated":425984,"data_stored":180124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.314},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"last":0.251}]},{"osd":1,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.35399999999999998}]},{"osd":3,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.46800000000000003}]},{"osd":4,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.503},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.53800000000000003}]},{"osd":5,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.52500000000000002}]},{"osd":6,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.58899999999999997}]},{"osd":7,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.111,"5min":1.111,"15min":1.111},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.018,"5min":1.018,"15min":1.018},"last":0.55500000000000005}]}]},{"osd":1,"up_from":13,"seq":55834574871,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31596,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340244,"statfs":{"total":96636764160,"available":96604409856,"internally_reserved":0,"allocated":425984,"data_stored":180124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:45:05 2023","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.29699999999999999}]},{"osd":2,"last update":"Thu Oct 19 12:45:05 2023","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.441},{"interface":"front","average":{"1min":0.433,"5min":0.433,"15min":0.433},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.38}]},{"osd":3,"last update":"Thu Oct 19 12:45:05 2023","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.36399999999999999}]},{"osd":4,"last update":"Thu Oct 19 12:45:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.1910000000000001,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.314}]},{"osd":5,"last update":"Thu Oct 19 12:45:05 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.117,"5min":1.117,"15min":1.117},"last":0.41599999999999998}]},{"osd":6,"last update":"Thu Oct 19 12:45:05 2023","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.077,"5min":1.077,"15min":1.077},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.51400000000000001}]},{"osd":7,"last update":"Thu Oct 19 12:45:05 2023","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.45700000000000002}]}]},{"osd":0,"up_from":13,"seq":55834574872,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32176,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339664,"statfs":{"total":96636764160,"available":96603815936,"internally_reserved":0,"allocated":1015808,"data_stored":769873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Oct 19 12:45:09 2023","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"last":0.246},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"last":0.32800000000000001}]},{"osd":2,"last update":"Thu Oct 19 12:45:09 2023","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.29999999999999999}]},{"osd":3,"last update":"Thu Oct 19 12:45:09 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"last":0.311},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.40999999999999998}]},{"osd":4,"last update":"Thu Oct 19 12:45:09 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.39800000000000002}]},{"osd":5,"last update":"Thu Oct 19 12:45:09 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.056,"5min":1.056,"15min":1.056},"last":0.49399999999999999}]},{"osd":6,"last update":"Thu Oct 19 12:45:09 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.109,"5min":1.109,"15min":1.109},"last":0.42099999999999999}]},{"osd":7,"last update":"Thu Oct 19 12:45:09 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.44500000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-19T12:45:44.402 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-19T12:45:44.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.541+0000 7f40c0c97640 1 -- 172.21.15.57:0/3900977956 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f40bc06ac70 msgr2=0x7f40bc06ca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:44.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.541+0000 7f40c0c97640 1 --2- 172.21.15.57:0/3900977956 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f40bc06ac70 0x7f40bc06ca90 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f40bc071f00 tx=0x7f40b402c250 comp rx=0 tx=0).stop 2023-10-19T12:45:44.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.541+0000 7f40c0c97640 1 -- 172.21.15.57:0/3900977956 shutdown_connections 2023-10-19T12:45:44.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.541+0000 7f40c0c97640 1 --2- 172.21.15.57:0/3900977956 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f40bc06ac70 0x7f40bc06ca90 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:44.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.541+0000 7f40c0c97640 1 --2- 172.21.15.57:0/3900977956 >> v2:172.21.15.179:3300/0 conn(0x7f40bc15c1f0 0x7f40bc06a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:44.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.541+0000 7f40c0c97640 1 --2- 172.21.15.57:0/3900977956 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f40bc1579f0 0x7f40bc0b0e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:44.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.541+0000 7f40c0c97640 1 -- 172.21.15.57:0/3900977956 >> 172.21.15.57:0/3900977956 conn(0x7f40bc06fae0 msgr2=0x7f40bc06fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:44.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.541+0000 7f40c0c97640 1 -- 172.21.15.57:0/3900977956 shutdown_connections 2023-10-19T12:45:44.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.541+0000 7f40c0c97640 1 -- 172.21.15.57:0/3900977956 wait complete. 2023-10-19T12:45:44.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.542+0000 7f40c0c97640 1 Processor -- start 2023-10-19T12:45:44.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.542+0000 7f40c0c97640 1 -- start start 2023-10-19T12:45:44.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.543+0000 7f40c0c97640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f40bc06ac70 0x7f40bc13d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:44.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.543+0000 7f40c0c97640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f40bc1579f0 0x7f40bc13de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:44.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.543+0000 7f40c0c97640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f40bc15c1f0 0x7f40bc0cc0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:44.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.543+0000 7f40c0c97640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40bc13e8c0 con 0x7f40bc15c1f0 2023-10-19T12:45:44.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.543+0000 7f40c0c97640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40bc13eba0 con 0x7f40bc06ac70 2023-10-19T12:45:44.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.543+0000 7f40c0c97640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40bc13ee80 con 0x7f40bc1579f0 2023-10-19T12:45:44.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.543+0000 7f40bad9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f40bc15c1f0 0x7f40bc0cc0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:44.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.543+0000 7f40bad9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f40bc15c1f0 0x7f40bc0cc0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59356/0 (socket says 172.21.15.57:59356) 2023-10-19T12:45:44.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.543+0000 7f40bad9d640 1 -- 172.21.15.57:0/2163468679 learned_addr learned my addr 172.21.15.57:0/2163468679 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:44.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.543+0000 7f40ba59c640 1 --2- 172.21.15.57:0/2163468679 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f40bc06ac70 0x7f40bc13d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:44.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.543+0000 7f40b9d9b640 1 --2- 172.21.15.57:0/2163468679 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f40bc1579f0 0x7f40bc13de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:44.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.543+0000 7f40bad9d640 1 -- 172.21.15.57:0/2163468679 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f40bc1579f0 msgr2=0x7f40bc13de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:44.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.543+0000 7f40bad9d640 1 --2- 172.21.15.57:0/2163468679 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f40bc1579f0 0x7f40bc13de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:44.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.543+0000 7f40bad9d640 1 -- 172.21.15.57:0/2163468679 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f40bc06ac70 msgr2=0x7f40bc13d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:44.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.543+0000 7f40bad9d640 1 --2- 172.21.15.57:0/2163468679 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f40bc06ac70 0x7f40bc13d6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:44.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.543+0000 7f40bad9d640 1 -- 172.21.15.57:0/2163468679 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40b4009900 con 0x7f40bc15c1f0 2023-10-19T12:45:44.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.544+0000 7f40bad9d640 1 --2- 172.21.15.57:0/2163468679 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f40bc15c1f0 0x7f40bc0cc0a0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f40b4009d60 tx=0x7f40b4035e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:44.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.544+0000 7f40aaffd640 1 -- 172.21.15.57:0/2163468679 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40b4034440 con 0x7f40bc15c1f0 2023-10-19T12:45:44.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.544+0000 7f40aaffd640 1 -- 172.21.15.57:0/2163468679 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f40b4004450 con 0x7f40bc15c1f0 2023-10-19T12:45:44.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.544+0000 7f40aaffd640 1 -- 172.21.15.57:0/2163468679 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40b4004860 con 0x7f40bc15c1f0 2023-10-19T12:45:44.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.544+0000 7f40c0c97640 1 -- 172.21.15.57:0/2163468679 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40bc13f160 con 0x7f40bc15c1f0 2023-10-19T12:45:44.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.544+0000 7f40c0c97640 1 -- 172.21.15.57:0/2163468679 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40bc14e170 con 0x7f40bc15c1f0 2023-10-19T12:45:44.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.545+0000 7f40c0c97640 1 -- 172.21.15.57:0/2163468679 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40bc13f680 con 0x7f40bc15c1f0 2023-10-19T12:45:44.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.545+0000 7f40aaffd640 1 -- 172.21.15.57:0/2163468679 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f40b400a040 con 0x7f40bc15c1f0 2023-10-19T12:45:44.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.546+0000 7f40aaffd640 1 --2- 172.21.15.57:0/2163468679 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f409c06ae40 0x7f409c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:44.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.546+0000 7f40aaffd640 1 -- 172.21.15.57:0/2163468679 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f40b40ac390 con 0x7f40bc15c1f0 2023-10-19T12:45:44.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.546+0000 7f40ba59c640 1 --2- 172.21.15.57:0/2163468679 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f409c06ae40 0x7f409c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:44.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.546+0000 7f40ba59c640 1 --2- 172.21.15.57:0/2163468679 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f409c06ae40 0x7f409c06d300 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f40ac005fd0 tx=0x7f40ac00b390 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:44.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.550+0000 7f40aaffd640 1 -- 172.21.15.57:0/2163468679 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f40bc13f680 con 0x7f40bc15c1f0 2023-10-19T12:45:44.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.724+0000 7f40c0c97640 1 -- 172.21.15.57:0/2163468679 --> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f40bc13ed10 con 0x7f409c06ae40 2023-10-19T12:45:44.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.732+0000 7f40aaffd640 1 -- 172.21.15.57:0/2163468679 <== mgr.4100 v2:172.21.15.179:6832/2609646285 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286395 (secure 0 0 0) 0x7f40bc13ed10 con 0x7f409c06ae40 2023-10-19T12:45:44.738 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:45:44.740 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2023-10-19T12:45:44.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.736+0000 7f40c0c97640 1 -- 172.21.15.57:0/2163468679 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f409c06ae40 msgr2=0x7f409c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:44.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.736+0000 7f40c0c97640 1 --2- 172.21.15.57:0/2163468679 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f409c06ae40 0x7f409c06d300 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f40ac005fd0 tx=0x7f40ac00b390 comp rx=0 tx=0).stop 2023-10-19T12:45:44.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.736+0000 7f40c0c97640 1 -- 172.21.15.57:0/2163468679 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f40bc15c1f0 msgr2=0x7f40bc0cc0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:44.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.736+0000 7f40c0c97640 1 --2- 172.21.15.57:0/2163468679 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f40bc15c1f0 0x7f40bc0cc0a0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f40b4009d60 tx=0x7f40b4035e30 comp rx=0 tx=0).stop 2023-10-19T12:45:44.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.736+0000 7f40c0c97640 1 -- 172.21.15.57:0/2163468679 shutdown_connections 2023-10-19T12:45:44.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.736+0000 7f40c0c97640 1 --2- 172.21.15.57:0/2163468679 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f409c06ae40 0x7f409c06d300 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:44.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.736+0000 7f40c0c97640 1 --2- 172.21.15.57:0/2163468679 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f40bc15c1f0 0x7f40bc0cc0a0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:44.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.736+0000 7f40c0c97640 1 --2- 172.21.15.57:0/2163468679 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f40bc1579f0 0x7f40bc13de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:44.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.736+0000 7f40c0c97640 1 --2- 172.21.15.57:0/2163468679 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f40bc06ac70 0x7f40bc13d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:44.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.736+0000 7f40c0c97640 1 -- 172.21.15.57:0/2163468679 >> 172.21.15.57:0/2163468679 conn(0x7f40bc06fae0 msgr2=0x7f40bc14fd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:44.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.736+0000 7f40c0c97640 1 -- 172.21.15.57:0/2163468679 shutdown_connections 2023-10-19T12:45:44.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.736+0000 7f40c0c97640 1 -- 172.21.15.57:0/2163468679 wait complete. 2023-10-19T12:45:44.749 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":81,"stamp":"2023-10-19T12:45:43.336334+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590438,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":168,"num_write_kb":2417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":126,"ondisk_log_size":126,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":56,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":237300,"kb_used_data":4496,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754737420,"statfs":{"total":773094113280,"available":772851118080,"internally_reserved":0,"allocated":4603904,"data_stored":2628204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":238276153},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":70,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.008337"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.347634+0000","last_change":"2023-10-19T12:45:40.347634+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.347634+0000","last_undegraded":"2023-10-19T12:45:40.347634+0000","last_fullsized":"2023-10-19T12:45:40.347634+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.349030+0000","last_change":"2023-10-19T12:45:40.349030+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.349030+0000","last_undegraded":"2023-10-19T12:45:40.349030+0000","last_fullsized":"2023-10-19T12:45:40.349030+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.376066+0000","last_change":"2023-10-19T12:45:41.371589+0000","last_active":"2023-10-19T12:45:41.376066+0000","last_peered":"2023-10-19T12:45:41.376066+0000","last_clean":"2023-10-19T12:45:41.376066+0000","last_became_active":"2023-10-19T12:45:41.371287+0000","last_became_peered":"2023-10-19T12:45:41.371287+0000","last_unstale":"2023-10-19T12:45:41.376066+0000","last_undegraded":"2023-10-19T12:45:41.376066+0000","last_fullsized":"2023-10-19T12:45:41.376066+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:16:37.419534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.370328+0000","last_change":"2023-10-19T12:45:41.363930+0000","last_active":"2023-10-19T12:45:41.370328+0000","last_peered":"2023-10-19T12:45:41.370328+0000","last_clean":"2023-10-19T12:45:41.370328+0000","last_became_active":"2023-10-19T12:45:41.363668+0000","last_became_peered":"2023-10-19T12:45:41.363668+0000","last_unstale":"2023-10-19T12:45:41.370328+0000","last_undegraded":"2023-10-19T12:45:41.370328+0000","last_fullsized":"2023-10-19T12:45:41.370328+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:51:05.063828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.370788+0000","last_change":"2023-10-19T12:45:41.363093+0000","last_active":"2023-10-19T12:45:41.370788+0000","last_peered":"2023-10-19T12:45:41.370788+0000","last_clean":"2023-10-19T12:45:41.370788+0000","last_became_active":"2023-10-19T12:45:41.362804+0000","last_became_peered":"2023-10-19T12:45:41.362804+0000","last_unstale":"2023-10-19T12:45:41.370788+0000","last_undegraded":"2023-10-19T12:45:41.370788+0000","last_fullsized":"2023-10-19T12:45:41.370788+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:42:33.722992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.374791+0000","last_change":"2023-10-19T12:45:41.359415+0000","last_active":"2023-10-19T12:45:41.374791+0000","last_peered":"2023-10-19T12:45:41.374791+0000","last_clean":"2023-10-19T12:45:41.374791+0000","last_became_active":"2023-10-19T12:45:41.359120+0000","last_became_peered":"2023-10-19T12:45:41.359120+0000","last_unstale":"2023-10-19T12:45:41.374791+0000","last_undegraded":"2023-10-19T12:45:41.374791+0000","last_fullsized":"2023-10-19T12:45:41.374791+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:48:25.218615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":3,"reported_epoch":38,"state":"creating+activating","last_fresh":"2023-10-19T12:45:41.350760+0000","last_change":"2023-10-19T12:45:41.350760+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:41.350760+0000","last_undegraded":"2023-10-19T12:45:41.350760+0000","last_fullsized":"2023-10-19T12:45:41.350760+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:38:37.793736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.349543+0000","last_change":"2023-10-19T12:45:40.349543+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.349543+0000","last_undegraded":"2023-10-19T12:45:40.349543+0000","last_fullsized":"2023-10-19T12:45:40.349543+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.375561+0000","last_change":"2023-10-19T12:45:41.368731+0000","last_active":"2023-10-19T12:45:41.375561+0000","last_peered":"2023-10-19T12:45:41.375561+0000","last_clean":"2023-10-19T12:45:41.375561+0000","last_became_active":"2023-10-19T12:45:41.368006+0000","last_became_peered":"2023-10-19T12:45:41.368006+0000","last_unstale":"2023-10-19T12:45:41.375561+0000","last_undegraded":"2023-10-19T12:45:41.375561+0000","last_fullsized":"2023-10-19T12:45:41.375561+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:48:25.218615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.345142+0000","last_change":"2023-10-19T12:45:40.345142+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.345142+0000","last_undegraded":"2023-10-19T12:45:40.345142+0000","last_fullsized":"2023-10-19T12:45:40.345142+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.346201+0000","last_change":"2023-10-19T12:45:40.346201+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.346201+0000","last_undegraded":"2023-10-19T12:45:40.346201+0000","last_fullsized":"2023-10-19T12:45:40.346201+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.370682+0000","last_change":"2023-10-19T12:45:41.357975+0000","last_active":"2023-10-19T12:45:41.370682+0000","last_peered":"2023-10-19T12:45:41.370682+0000","last_clean":"2023-10-19T12:45:41.370682+0000","last_became_active":"2023-10-19T12:45:41.357717+0000","last_became_peered":"2023-10-19T12:45:41.357717+0000","last_unstale":"2023-10-19T12:45:41.370682+0000","last_undegraded":"2023-10-19T12:45:41.370682+0000","last_fullsized":"2023-10-19T12:45:41.370682+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:49:21.932303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.345686+0000","last_change":"2023-10-19T12:45:40.345686+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.345686+0000","last_undegraded":"2023-10-19T12:45:40.345686+0000","last_fullsized":"2023-10-19T12:45:40.345686+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.351066+0000","last_change":"2023-10-19T12:45:40.351066+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.351066+0000","last_undegraded":"2023-10-19T12:45:40.351066+0000","last_fullsized":"2023-10-19T12:45:40.351066+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":3,"reported_epoch":38,"state":"creating+activating","last_fresh":"2023-10-19T12:45:41.350355+0000","last_change":"2023-10-19T12:45:41.350355+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:41.350355+0000","last_undegraded":"2023-10-19T12:45:41.350355+0000","last_fullsized":"2023-10-19T12:45:41.350355+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:10:04.203620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":3,"reported_epoch":38,"state":"creating+activating","last_fresh":"2023-10-19T12:45:41.352539+0000","last_change":"2023-10-19T12:45:41.352539+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:41.352539+0000","last_undegraded":"2023-10-19T12:45:41.352539+0000","last_fullsized":"2023-10-19T12:45:41.352539+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:21:51.336716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":3,"reported_epoch":38,"state":"creating+activating","last_fresh":"2023-10-19T12:45:41.351475+0000","last_change":"2023-10-19T12:45:41.351475+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:41.351475+0000","last_undegraded":"2023-10-19T12:45:41.351475+0000","last_fullsized":"2023-10-19T12:45:41.351475+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:32:37.014614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":3,"reported_epoch":38,"state":"creating+activating","last_fresh":"2023-10-19T12:45:41.348402+0000","last_change":"2023-10-19T12:45:41.348402+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:41.348402+0000","last_undegraded":"2023-10-19T12:45:41.348402+0000","last_fullsized":"2023-10-19T12:45:41.348402+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:53:30.629136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.364662+0000","last_change":"2023-10-19T12:45:41.356683+0000","last_active":"2023-10-19T12:45:41.364662+0000","last_peered":"2023-10-19T12:45:41.364662+0000","last_clean":"2023-10-19T12:45:41.364662+0000","last_became_active":"2023-10-19T12:45:41.356404+0000","last_became_peered":"2023-10-19T12:45:41.356404+0000","last_unstale":"2023-10-19T12:45:41.364662+0000","last_undegraded":"2023-10-19T12:45:41.364662+0000","last_fullsized":"2023-10-19T12:45:41.364662+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:30:20.055674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":3,"reported_epoch":38,"state":"creating+activating","last_fresh":"2023-10-19T12:45:41.351396+0000","last_change":"2023-10-19T12:45:41.351396+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:41.351396+0000","last_undegraded":"2023-10-19T12:45:41.351396+0000","last_fullsized":"2023-10-19T12:45:41.351396+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:48:25.218615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.344680+0000","last_change":"2023-10-19T12:45:40.344680+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.344680+0000","last_undegraded":"2023-10-19T12:45:40.344680+0000","last_fullsized":"2023-10-19T12:45:40.344680+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.348310+0000","last_change":"2023-10-19T12:45:40.348310+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.348310+0000","last_undegraded":"2023-10-19T12:45:40.348310+0000","last_fullsized":"2023-10-19T12:45:40.348310+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":3,"reported_epoch":38,"state":"creating+activating","last_fresh":"2023-10-19T12:45:41.350788+0000","last_change":"2023-10-19T12:45:41.350788+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:41.350788+0000","last_undegraded":"2023-10-19T12:45:41.350788+0000","last_fullsized":"2023-10-19T12:45:41.350788+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:44:04.457483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.371142+0000","last_change":"2023-10-19T12:45:41.361356+0000","last_active":"2023-10-19T12:45:41.371142+0000","last_peered":"2023-10-19T12:45:41.371142+0000","last_clean":"2023-10-19T12:45:41.371142+0000","last_became_active":"2023-10-19T12:45:41.361082+0000","last_became_peered":"2023-10-19T12:45:41.361082+0000","last_unstale":"2023-10-19T12:45:41.371142+0000","last_undegraded":"2023-10-19T12:45:41.371142+0000","last_fullsized":"2023-10-19T12:45:41.371142+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:45:11.031973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":3,"reported_epoch":38,"state":"creating+activating","last_fresh":"2023-10-19T12:45:41.350918+0000","last_change":"2023-10-19T12:45:41.350918+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:41.350918+0000","last_undegraded":"2023-10-19T12:45:41.350918+0000","last_fullsized":"2023-10-19T12:45:41.350918+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:49:21.932303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.375784+0000","last_change":"2023-10-19T12:45:41.372032+0000","last_active":"2023-10-19T12:45:41.375784+0000","last_peered":"2023-10-19T12:45:41.375784+0000","last_clean":"2023-10-19T12:45:41.375784+0000","last_became_active":"2023-10-19T12:45:41.371582+0000","last_became_peered":"2023-10-19T12:45:41.371582+0000","last_unstale":"2023-10-19T12:45:41.375784+0000","last_undegraded":"2023-10-19T12:45:41.375784+0000","last_fullsized":"2023-10-19T12:45:41.375784+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:55:02.268260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":3,"reported_epoch":38,"state":"creating+activating","last_fresh":"2023-10-19T12:45:41.348601+0000","last_change":"2023-10-19T12:45:41.348601+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:41.348601+0000","last_undegraded":"2023-10-19T12:45:41.348601+0000","last_fullsized":"2023-10-19T12:45:41.348601+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:36:07.043432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.345780+0000","last_change":"2023-10-19T12:45:40.345780+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.345780+0000","last_undegraded":"2023-10-19T12:45:40.345780+0000","last_fullsized":"2023-10-19T12:45:40.345780+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":3,"reported_epoch":38,"state":"creating+activating","last_fresh":"2023-10-19T12:45:41.351166+0000","last_change":"2023-10-19T12:45:41.351166+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:41.351166+0000","last_undegraded":"2023-10-19T12:45:41.351166+0000","last_fullsized":"2023-10-19T12:45:41.351166+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:16:37.419534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":3,"reported_epoch":38,"state":"creating+activating","last_fresh":"2023-10-19T12:45:41.348399+0000","last_change":"2023-10-19T12:45:41.348399+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:41.348399+0000","last_undegraded":"2023-10-19T12:45:41.348399+0000","last_fullsized":"2023-10-19T12:45:41.348399+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T12:45:47.935169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":["6","3"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":3,"reported_epoch":38,"state":"creating+activating","last_fresh":"2023-10-19T12:45:41.351136+0000","last_change":"2023-10-19T12:45:41.351136+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:41.351136+0000","last_undegraded":"2023-10-19T12:45:41.351136+0000","last_fullsized":"2023-10-19T12:45:41.351136+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:10:04.203620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.369704+0000","last_change":"2023-10-19T12:45:41.356009+0000","last_active":"2023-10-19T12:45:41.369704+0000","last_peered":"2023-10-19T12:45:41.369704+0000","last_clean":"2023-10-19T12:45:41.369704+0000","last_became_active":"2023-10-19T12:45:41.355714+0000","last_became_peered":"2023-10-19T12:45:41.355714+0000","last_unstale":"2023-10-19T12:45:41.369704+0000","last_undegraded":"2023-10-19T12:45:41.369704+0000","last_fullsized":"2023-10-19T12:45:41.369704+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:16:37.419534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":3,"reported_epoch":38,"state":"creating+activating","last_fresh":"2023-10-19T12:45:41.351973+0000","last_change":"2023-10-19T12:45:41.351973+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:41.351973+0000","last_undegraded":"2023-10-19T12:45:41.351973+0000","last_fullsized":"2023-10-19T12:45:41.351973+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:53:29.708835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-19T12:45:40.364691+0000","last_change":"2023-10-19T12:45:40.359697+0000","last_active":"2023-10-19T12:45:40.364691+0000","last_peered":"2023-10-19T12:45:40.364691+0000","last_clean":"2023-10-19T12:45:40.364691+0000","last_became_active":"2023-10-19T12:45:40.359413+0000","last_became_peered":"2023-10-19T12:45:40.359413+0000","last_unstale":"2023-10-19T12:45:40.364691+0000","last_undegraded":"2023-10-19T12:45:40.364691+0000","last_fullsized":"2023-10-19T12:45:40.364691+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:36:38.697134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.375801+0000","last_change":"2023-10-19T12:45:41.372556+0000","last_active":"2023-10-19T12:45:41.375801+0000","last_peered":"2023-10-19T12:45:41.375801+0000","last_clean":"2023-10-19T12:45:41.375801+0000","last_became_active":"2023-10-19T12:45:41.372184+0000","last_became_peered":"2023-10-19T12:45:41.372184+0000","last_unstale":"2023-10-19T12:45:41.375801+0000","last_undegraded":"2023-10-19T12:45:41.375801+0000","last_fullsized":"2023-10-19T12:45:41.375801+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:10:04.203620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"38'1","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.370313+0000","last_change":"2023-10-19T12:45:40.352581+0000","last_active":"2023-10-19T12:45:41.370313+0000","last_peered":"2023-10-19T12:45:41.370313+0000","last_clean":"2023-10-19T12:45:41.370313+0000","last_became_active":"2023-10-19T12:45:40.352300+0000","last_became_peered":"2023-10-19T12:45:40.352300+0000","last_unstale":"2023-10-19T12:45:41.370313+0000","last_undegraded":"2023-10-19T12:45:41.370313+0000","last_fullsized":"2023-10-19T12:45:41.370313+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:59:58.100040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.348927+0000","last_change":"2023-10-19T12:45:40.348927+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.348927+0000","last_undegraded":"2023-10-19T12:45:40.348927+0000","last_fullsized":"2023-10-19T12:45:40.348927+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"38'2","reported_seq":13,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.374990+0000","last_change":"2023-10-19T12:45:40.355990+0000","last_active":"2023-10-19T12:45:41.374990+0000","last_peered":"2023-10-19T12:45:41.374990+0000","last_clean":"2023-10-19T12:45:41.374990+0000","last_became_active":"2023-10-19T12:45:40.355666+0000","last_became_peered":"2023-10-19T12:45:40.355666+0000","last_unstale":"2023-10-19T12:45:41.374990+0000","last_undegraded":"2023-10-19T12:45:41.374990+0000","last_fullsized":"2023-10-19T12:45:41.374990+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:53:28.702370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.349506+0000","last_change":"2023-10-19T12:45:40.349506+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.349506+0000","last_undegraded":"2023-10-19T12:45:40.349506+0000","last_fullsized":"2023-10-19T12:45:40.349506+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-19T12:45:40.367475+0000","last_change":"2023-10-19T12:45:40.364018+0000","last_active":"2023-10-19T12:45:40.367475+0000","last_peered":"2023-10-19T12:45:40.367475+0000","last_clean":"2023-10-19T12:45:40.367475+0000","last_became_active":"2023-10-19T12:45:40.363763+0000","last_became_peered":"2023-10-19T12:45:40.363763+0000","last_unstale":"2023-10-19T12:45:40.367475+0000","last_undegraded":"2023-10-19T12:45:40.367475+0000","last_fullsized":"2023-10-19T12:45:40.367475+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:36:38.697134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.363923+0000","last_change":"2023-10-19T12:45:41.355176+0000","last_active":"2023-10-19T12:45:41.363923+0000","last_peered":"2023-10-19T12:45:41.363923+0000","last_clean":"2023-10-19T12:45:41.363923+0000","last_became_active":"2023-10-19T12:45:41.354687+0000","last_became_peered":"2023-10-19T12:45:41.354687+0000","last_unstale":"2023-10-19T12:45:41.363923+0000","last_undegraded":"2023-10-19T12:45:41.363923+0000","last_fullsized":"2023-10-19T12:45:41.363923+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:53:29.708835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-19T12:45:40.362034+0000","last_change":"2023-10-19T12:45:40.357988+0000","last_active":"2023-10-19T12:45:40.362034+0000","last_peered":"2023-10-19T12:45:40.362034+0000","last_clean":"2023-10-19T12:45:40.362034+0000","last_became_active":"2023-10-19T12:45:40.357478+0000","last_became_peered":"2023-10-19T12:45:40.357478+0000","last_unstale":"2023-10-19T12:45:40.362034+0000","last_undegraded":"2023-10-19T12:45:40.362034+0000","last_fullsized":"2023-10-19T12:45:40.362034+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:49:20.925838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":3,"reported_epoch":38,"state":"creating+activating","last_fresh":"2023-10-19T12:45:41.349960+0000","last_change":"2023-10-19T12:45:41.349960+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:41.349960+0000","last_undegraded":"2023-10-19T12:45:41.349960+0000","last_fullsized":"2023-10-19T12:45:41.349960+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:44:04.457483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":11,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.349382+0000","last_change":"2023-10-19T12:45:40.355247+0000","last_active":"2023-10-19T12:45:41.349382+0000","last_peered":"2023-10-19T12:45:41.349382+0000","last_clean":"2023-10-19T12:45:41.349382+0000","last_became_active":"2023-10-19T12:45:40.354967+0000","last_became_peered":"2023-10-19T12:45:40.354967+0000","last_unstale":"2023-10-19T12:45:41.349382+0000","last_undegraded":"2023-10-19T12:45:41.349382+0000","last_fullsized":"2023-10-19T12:45:41.349382+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:48:24.212150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.350186+0000","last_change":"2023-10-19T12:45:40.350186+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.350186+0000","last_undegraded":"2023-10-19T12:45:40.350186+0000","last_fullsized":"2023-10-19T12:45:40.350186+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":11,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.348919+0000","last_change":"2023-10-19T12:45:40.359949+0000","last_active":"2023-10-19T12:45:41.348919+0000","last_peered":"2023-10-19T12:45:41.348919+0000","last_clean":"2023-10-19T12:45:41.348919+0000","last_became_active":"2023-10-19T12:45:40.359690+0000","last_became_peered":"2023-10-19T12:45:40.359690+0000","last_unstale":"2023-10-19T12:45:41.348919+0000","last_undegraded":"2023-10-19T12:45:41.348919+0000","last_fullsized":"2023-10-19T12:45:41.348919+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:44:03.451018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.351154+0000","last_change":"2023-10-19T12:45:40.351154+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.351154+0000","last_undegraded":"2023-10-19T12:45:40.351154+0000","last_fullsized":"2023-10-19T12:45:40.351154+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-19T12:45:40.364109+0000","last_change":"2023-10-19T12:45:40.360289+0000","last_active":"2023-10-19T12:45:40.364109+0000","last_peered":"2023-10-19T12:45:40.364109+0000","last_clean":"2023-10-19T12:45:40.364109+0000","last_became_active":"2023-10-19T12:45:40.359850+0000","last_became_peered":"2023-10-19T12:45:40.359850+0000","last_unstale":"2023-10-19T12:45:40.364109+0000","last_undegraded":"2023-10-19T12:45:40.364109+0000","last_fullsized":"2023-10-19T12:45:40.364109+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:30:19.049209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.370304+0000","last_change":"2023-10-19T12:45:41.363350+0000","last_active":"2023-10-19T12:45:41.370304+0000","last_peered":"2023-10-19T12:45:41.370304+0000","last_clean":"2023-10-19T12:45:41.370304+0000","last_became_active":"2023-10-19T12:45:41.363067+0000","last_became_peered":"2023-10-19T12:45:41.363067+0000","last_unstale":"2023-10-19T12:45:41.370304+0000","last_undegraded":"2023-10-19T12:45:41.370304+0000","last_fullsized":"2023-10-19T12:45:41.370304+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:09:22.266843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-19T12:45:40.363651+0000","last_change":"2023-10-19T12:45:40.356475+0000","last_active":"2023-10-19T12:45:40.363651+0000","last_peered":"2023-10-19T12:45:40.363651+0000","last_clean":"2023-10-19T12:45:40.363651+0000","last_became_active":"2023-10-19T12:45:40.356178+0000","last_became_peered":"2023-10-19T12:45:40.356178+0000","last_unstale":"2023-10-19T12:45:40.363651+0000","last_undegraded":"2023-10-19T12:45:40.363651+0000","last_fullsized":"2023-10-19T12:45:40.363651+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:59:58.100040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":3,"reported_epoch":38,"state":"creating+activating","last_fresh":"2023-10-19T12:45:41.350085+0000","last_change":"2023-10-19T12:45:41.350085+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:41.350085+0000","last_undegraded":"2023-10-19T12:45:41.350085+0000","last_fullsized":"2023-10-19T12:45:41.350085+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:29:00.286113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-19T12:45:40.364238+0000","last_change":"2023-10-19T12:45:40.360586+0000","last_active":"2023-10-19T12:45:40.364238+0000","last_peered":"2023-10-19T12:45:40.364238+0000","last_clean":"2023-10-19T12:45:40.364238+0000","last_became_active":"2023-10-19T12:45:40.360327+0000","last_became_peered":"2023-10-19T12:45:40.360327+0000","last_unstale":"2023-10-19T12:45:40.364238+0000","last_undegraded":"2023-10-19T12:45:40.364238+0000","last_fullsized":"2023-10-19T12:45:40.364238+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:55:01.261795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.375472+0000","last_change":"2023-10-19T12:45:41.366187+0000","last_active":"2023-10-19T12:45:41.375472+0000","last_peered":"2023-10-19T12:45:41.375472+0000","last_clean":"2023-10-19T12:45:41.375472+0000","last_became_active":"2023-10-19T12:45:41.365896+0000","last_became_peered":"2023-10-19T12:45:41.365896+0000","last_unstale":"2023-10-19T12:45:41.375472+0000","last_undegraded":"2023-10-19T12:45:41.375472+0000","last_fullsized":"2023-10-19T12:45:41.375472+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:55:02.268260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-19T12:45:40.361876+0000","last_change":"2023-10-19T12:45:40.358267+0000","last_active":"2023-10-19T12:45:40.361876+0000","last_peered":"2023-10-19T12:45:40.361876+0000","last_clean":"2023-10-19T12:45:40.361876+0000","last_became_active":"2023-10-19T12:45:40.358000+0000","last_became_peered":"2023-10-19T12:45:40.358000+0000","last_unstale":"2023-10-19T12:45:40.361876+0000","last_undegraded":"2023-10-19T12:45:40.361876+0000","last_fullsized":"2023-10-19T12:45:40.361876+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:48:24.212150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.348513+0000","last_change":"2023-10-19T12:45:40.348513+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.348513+0000","last_undegraded":"2023-10-19T12:45:40.348513+0000","last_fullsized":"2023-10-19T12:45:40.348513+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-19T12:45:40.364295+0000","last_change":"2023-10-19T12:45:40.361715+0000","last_active":"2023-10-19T12:45:40.364295+0000","last_peered":"2023-10-19T12:45:40.364295+0000","last_clean":"2023-10-19T12:45:40.364295+0000","last_became_active":"2023-10-19T12:45:40.361442+0000","last_became_peered":"2023-10-19T12:45:40.361442+0000","last_unstale":"2023-10-19T12:45:40.364295+0000","last_undegraded":"2023-10-19T12:45:40.364295+0000","last_fullsized":"2023-10-19T12:45:40.364295+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:44:03.451018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.375150+0000","last_change":"2023-10-19T12:45:41.364032+0000","last_active":"2023-10-19T12:45:41.375150+0000","last_peered":"2023-10-19T12:45:41.375150+0000","last_clean":"2023-10-19T12:45:41.375150+0000","last_became_active":"2023-10-19T12:45:41.363734+0000","last_became_peered":"2023-10-19T12:45:41.363734+0000","last_unstale":"2023-10-19T12:45:41.375150+0000","last_undegraded":"2023-10-19T12:45:41.375150+0000","last_fullsized":"2023-10-19T12:45:41.375150+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:32:37.014614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-19T12:45:40.363994+0000","last_change":"2023-10-19T12:45:40.360672+0000","last_active":"2023-10-19T12:45:40.363994+0000","last_peered":"2023-10-19T12:45:40.363994+0000","last_clean":"2023-10-19T12:45:40.363994+0000","last_became_active":"2023-10-19T12:45:40.360396+0000","last_became_peered":"2023-10-19T12:45:40.360396+0000","last_unstale":"2023-10-19T12:45:40.363994+0000","last_undegraded":"2023-10-19T12:45:40.363994+0000","last_fullsized":"2023-10-19T12:45:40.363994+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:28:59.279648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.347231+0000","last_change":"2023-10-19T12:45:40.347231+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.347231+0000","last_undegraded":"2023-10-19T12:45:40.347231+0000","last_fullsized":"2023-10-19T12:45:40.347231+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":11,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.348645+0000","last_change":"2023-10-19T12:45:40.359407+0000","last_active":"2023-10-19T12:45:41.348645+0000","last_peered":"2023-10-19T12:45:41.348645+0000","last_clean":"2023-10-19T12:45:41.348645+0000","last_became_active":"2023-10-19T12:45:40.359134+0000","last_became_peered":"2023-10-19T12:45:40.359134+0000","last_unstale":"2023-10-19T12:45:41.348645+0000","last_undegraded":"2023-10-19T12:45:41.348645+0000","last_fullsized":"2023-10-19T12:45:41.348645+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:59:58.100040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":3,"reported_epoch":38,"state":"creating+activating","last_fresh":"2023-10-19T12:45:41.349119+0000","last_change":"2023-10-19T12:45:41.349119+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:41.349119+0000","last_undegraded":"2023-10-19T12:45:41.349119+0000","last_fullsized":"2023-10-19T12:45:41.349119+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:16:37.419534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":11,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.350155+0000","last_change":"2023-10-19T12:45:40.359548+0000","last_active":"2023-10-19T12:45:41.350155+0000","last_peered":"2023-10-19T12:45:41.350155+0000","last_clean":"2023-10-19T12:45:41.350155+0000","last_became_active":"2023-10-19T12:45:40.359282+0000","last_became_peered":"2023-10-19T12:45:40.359282+0000","last_unstale":"2023-10-19T12:45:41.350155+0000","last_undegraded":"2023-10-19T12:45:41.350155+0000","last_fullsized":"2023-10-19T12:45:41.350155+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:39:09.771537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.350379+0000","last_change":"2023-10-19T12:45:40.350379+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.350379+0000","last_undegraded":"2023-10-19T12:45:40.350379+0000","last_fullsized":"2023-10-19T12:45:40.350379+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"38'3","reported_seq":11,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.352796+0000","last_change":"2023-10-19T12:45:40.358054+0000","last_active":"2023-10-19T12:45:41.352796+0000","last_peered":"2023-10-19T12:45:41.352796+0000","last_clean":"2023-10-19T12:45:41.352796+0000","last_became_active":"2023-10-19T12:45:40.357803+0000","last_became_peered":"2023-10-19T12:45:40.357803+0000","last_unstale":"2023-10-19T12:45:41.352796+0000","last_undegraded":"2023-10-19T12:45:41.352796+0000","last_fullsized":"2023-10-19T12:45:41.352796+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:07:55.739920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.350100+0000","last_change":"2023-10-19T12:45:40.350100+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.350100+0000","last_undegraded":"2023-10-19T12:45:40.350100+0000","last_fullsized":"2023-10-19T12:45:40.350100+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-19T12:45:40.362007+0000","last_change":"2023-10-19T12:45:40.358450+0000","last_active":"2023-10-19T12:45:40.362007+0000","last_peered":"2023-10-19T12:45:40.362007+0000","last_clean":"2023-10-19T12:45:40.362007+0000","last_became_active":"2023-10-19T12:45:40.356778+0000","last_became_peered":"2023-10-19T12:45:40.356778+0000","last_unstale":"2023-10-19T12:45:40.362007+0000","last_undegraded":"2023-10-19T12:45:40.362007+0000","last_fullsized":"2023-10-19T12:45:40.362007+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:16:36.413069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.345759+0000","last_change":"2023-10-19T12:45:40.345759+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.345759+0000","last_undegraded":"2023-10-19T12:45:40.345759+0000","last_fullsized":"2023-10-19T12:45:40.345759+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-19T12:45:40.361384+0000","last_change":"2023-10-19T12:45:40.357947+0000","last_active":"2023-10-19T12:45:40.361384+0000","last_peered":"2023-10-19T12:45:40.361384+0000","last_clean":"2023-10-19T12:45:40.361384+0000","last_became_active":"2023-10-19T12:45:40.357669+0000","last_became_peered":"2023-10-19T12:45:40.357669+0000","last_unstale":"2023-10-19T12:45:40.361384+0000","last_undegraded":"2023-10-19T12:45:40.361384+0000","last_fullsized":"2023-10-19T12:45:40.361384+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T12:45:46.928704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.350198+0000","last_change":"2023-10-19T12:45:40.350198+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.350198+0000","last_undegraded":"2023-10-19T12:45:40.350198+0000","last_fullsized":"2023-10-19T12:45:40.350198+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"38'1","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.370536+0000","last_change":"2023-10-19T12:45:40.354628+0000","last_active":"2023-10-19T12:45:41.370536+0000","last_peered":"2023-10-19T12:45:41.370536+0000","last_clean":"2023-10-19T12:45:41.370536+0000","last_became_active":"2023-10-19T12:45:40.354367+0000","last_became_peered":"2023-10-19T12:45:40.354367+0000","last_unstale":"2023-10-19T12:45:41.370536+0000","last_undegraded":"2023-10-19T12:45:41.370536+0000","last_fullsized":"2023-10-19T12:45:41.370536+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:33:49.817824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.375358+0000","last_change":"2023-10-19T12:45:41.367225+0000","last_active":"2023-10-19T12:45:41.375358+0000","last_peered":"2023-10-19T12:45:41.375358+0000","last_clean":"2023-10-19T12:45:41.375358+0000","last_became_active":"2023-10-19T12:45:41.366782+0000","last_became_peered":"2023-10-19T12:45:41.366782+0000","last_unstale":"2023-10-19T12:45:41.375358+0000","last_undegraded":"2023-10-19T12:45:41.375358+0000","last_fullsized":"2023-10-19T12:45:41.375358+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:50:19.159560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":11,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.349510+0000","last_change":"2023-10-19T12:45:40.357501+0000","last_active":"2023-10-19T12:45:41.349510+0000","last_peered":"2023-10-19T12:45:41.349510+0000","last_clean":"2023-10-19T12:45:41.349510+0000","last_became_active":"2023-10-19T12:45:40.355411+0000","last_became_peered":"2023-10-19T12:45:40.355411+0000","last_unstale":"2023-10-19T12:45:41.349510+0000","last_undegraded":"2023-10-19T12:45:41.349510+0000","last_fullsized":"2023-10-19T12:45:41.349510+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:32:36.008149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":3,"reported_epoch":38,"state":"creating+activating","last_fresh":"2023-10-19T12:45:41.353064+0000","last_change":"2023-10-19T12:45:41.353064+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:41.353064+0000","last_undegraded":"2023-10-19T12:45:41.353064+0000","last_fullsized":"2023-10-19T12:45:41.353064+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:59:55.870354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"38'2","reported_seq":13,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.376098+0000","last_change":"2023-10-19T12:45:40.357725+0000","last_active":"2023-10-19T12:45:41.376098+0000","last_peered":"2023-10-19T12:45:41.376098+0000","last_clean":"2023-10-19T12:45:41.376098+0000","last_became_active":"2023-10-19T12:45:40.357417+0000","last_became_peered":"2023-10-19T12:45:40.357417+0000","last_unstale":"2023-10-19T12:45:41.376098+0000","last_undegraded":"2023-10-19T12:45:41.376098+0000","last_fullsized":"2023-10-19T12:45:41.376098+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:26:34.780735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.375456+0000","last_change":"2023-10-19T12:45:41.366181+0000","last_active":"2023-10-19T12:45:41.375456+0000","last_peered":"2023-10-19T12:45:41.375456+0000","last_clean":"2023-10-19T12:45:41.375456+0000","last_became_active":"2023-10-19T12:45:41.365882+0000","last_became_peered":"2023-10-19T12:45:41.365882+0000","last_unstale":"2023-10-19T12:45:41.375456+0000","last_undegraded":"2023-10-19T12:45:41.375456+0000","last_fullsized":"2023-10-19T12:45:41.375456+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:29:00.286113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":11,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.343006+0000","last_change":"2023-10-19T12:45:40.355357+0000","last_active":"2023-10-19T12:45:41.343006+0000","last_peered":"2023-10-19T12:45:41.343006+0000","last_clean":"2023-10-19T12:45:41.343006+0000","last_became_active":"2023-10-19T12:45:40.355097+0000","last_became_peered":"2023-10-19T12:45:40.355097+0000","last_unstale":"2023-10-19T12:45:41.343006+0000","last_undegraded":"2023-10-19T12:45:41.343006+0000","last_fullsized":"2023-10-19T12:45:41.343006+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:23:29.827547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":3,"reported_epoch":38,"state":"creating+activating","last_fresh":"2023-10-19T12:45:41.349483+0000","last_change":"2023-10-19T12:45:41.349483+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:41.349483+0000","last_undegraded":"2023-10-19T12:45:41.349483+0000","last_fullsized":"2023-10-19T12:45:41.349483+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:59:59.106505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":11,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.351381+0000","last_change":"2023-10-19T12:45:40.360123+0000","last_active":"2023-10-19T12:45:41.351381+0000","last_peered":"2023-10-19T12:45:41.351381+0000","last_clean":"2023-10-19T12:45:41.351381+0000","last_became_active":"2023-10-19T12:45:40.359767+0000","last_became_peered":"2023-10-19T12:45:40.359767+0000","last_unstale":"2023-10-19T12:45:41.351381+0000","last_undegraded":"2023-10-19T12:45:41.351381+0000","last_fullsized":"2023-10-19T12:45:41.351381+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:38:36.787271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.346878+0000","last_change":"2023-10-19T12:45:40.346878+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.346878+0000","last_undegraded":"2023-10-19T12:45:40.346878+0000","last_fullsized":"2023-10-19T12:45:40.346878+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":11,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.343223+0000","last_change":"2023-10-19T12:45:40.357695+0000","last_active":"2023-10-19T12:45:41.343223+0000","last_peered":"2023-10-19T12:45:41.343223+0000","last_clean":"2023-10-19T12:45:41.343223+0000","last_became_active":"2023-10-19T12:45:40.357393+0000","last_became_peered":"2023-10-19T12:45:40.357393+0000","last_unstale":"2023-10-19T12:45:41.343223+0000","last_undegraded":"2023-10-19T12:45:41.343223+0000","last_fullsized":"2023-10-19T12:45:41.343223+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:43:31.256604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.375615+0000","last_change":"2023-10-19T12:45:41.371258+0000","last_active":"2023-10-19T12:45:41.375615+0000","last_peered":"2023-10-19T12:45:41.375615+0000","last_clean":"2023-10-19T12:45:41.375615+0000","last_became_active":"2023-10-19T12:45:41.370963+0000","last_became_peered":"2023-10-19T12:45:41.370963+0000","last_unstale":"2023-10-19T12:45:41.375615+0000","last_undegraded":"2023-10-19T12:45:41.375615+0000","last_fullsized":"2023-10-19T12:45:41.375615+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:38:37.793736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-19T12:45:40.361905+0000","last_change":"2023-10-19T12:45:40.358263+0000","last_active":"2023-10-19T12:45:40.361905+0000","last_peered":"2023-10-19T12:45:40.361905+0000","last_clean":"2023-10-19T12:45:40.361905+0000","last_became_active":"2023-10-19T12:45:40.357402+0000","last_became_peered":"2023-10-19T12:45:40.357402+0000","last_unstale":"2023-10-19T12:45:40.361905+0000","last_undegraded":"2023-10-19T12:45:40.361905+0000","last_fullsized":"2023-10-19T12:45:40.361905+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:28:59.279648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":3,"reported_epoch":38,"state":"creating+activating","last_fresh":"2023-10-19T12:45:41.350527+0000","last_change":"2023-10-19T12:45:41.350527+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:41.350527+0000","last_undegraded":"2023-10-19T12:45:41.350527+0000","last_fullsized":"2023-10-19T12:45:41.350527+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:59:59.106505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":11,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.343650+0000","last_change":"2023-10-19T12:45:40.352686+0000","last_active":"2023-10-19T12:45:41.343650+0000","last_peered":"2023-10-19T12:45:41.343650+0000","last_clean":"2023-10-19T12:45:41.343650+0000","last_became_active":"2023-10-19T12:45:40.352420+0000","last_became_peered":"2023-10-19T12:45:40.352420+0000","last_unstale":"2023-10-19T12:45:41.343650+0000","last_undegraded":"2023-10-19T12:45:41.343650+0000","last_fullsized":"2023-10-19T12:45:41.343650+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:36:38.697134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.346413+0000","last_change":"2023-10-19T12:45:40.346413+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.346413+0000","last_undegraded":"2023-10-19T12:45:40.346413+0000","last_fullsized":"2023-10-19T12:45:40.346413+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":11,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.348286+0000","last_change":"2023-10-19T12:45:40.358578+0000","last_active":"2023-10-19T12:45:41.348286+0000","last_peered":"2023-10-19T12:45:41.348286+0000","last_clean":"2023-10-19T12:45:41.348286+0000","last_became_active":"2023-10-19T12:45:40.358057+0000","last_became_peered":"2023-10-19T12:45:40.358057+0000","last_unstale":"2023-10-19T12:45:41.348286+0000","last_undegraded":"2023-10-19T12:45:41.348286+0000","last_fullsized":"2023-10-19T12:45:41.348286+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:38:36.787271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.375444+0000","last_change":"2023-10-19T12:45:41.367997+0000","last_active":"2023-10-19T12:45:41.375444+0000","last_peered":"2023-10-19T12:45:41.375444+0000","last_clean":"2023-10-19T12:45:41.375444+0000","last_became_active":"2023-10-19T12:45:41.367687+0000","last_became_peered":"2023-10-19T12:45:41.367687+0000","last_unstale":"2023-10-19T12:45:41.375444+0000","last_undegraded":"2023-10-19T12:45:41.375444+0000","last_fullsized":"2023-10-19T12:45:41.375444+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:10:04.203620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":11,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.348125+0000","last_change":"2023-10-19T12:45:40.360184+0000","last_active":"2023-10-19T12:45:41.348125+0000","last_peered":"2023-10-19T12:45:41.348125+0000","last_clean":"2023-10-19T12:45:41.348125+0000","last_became_active":"2023-10-19T12:45:40.359917+0000","last_became_peered":"2023-10-19T12:45:40.359917+0000","last_unstale":"2023-10-19T12:45:41.348125+0000","last_undegraded":"2023-10-19T12:45:41.348125+0000","last_fullsized":"2023-10-19T12:45:41.348125+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:44:03.451018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.374901+0000","last_change":"2023-10-19T12:45:41.364206+0000","last_active":"2023-10-19T12:45:41.374901+0000","last_peered":"2023-10-19T12:45:41.374901+0000","last_clean":"2023-10-19T12:45:41.374901+0000","last_became_active":"2023-10-19T12:45:41.363926+0000","last_became_peered":"2023-10-19T12:45:41.363926+0000","last_unstale":"2023-10-19T12:45:41.374901+0000","last_undegraded":"2023-10-19T12:45:41.374901+0000","last_fullsized":"2023-10-19T12:45:41.374901+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:29:00.286113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-19T12:45:40.364717+0000","last_change":"2023-10-19T12:45:40.360715+0000","last_active":"2023-10-19T12:45:40.364717+0000","last_peered":"2023-10-19T12:45:40.364717+0000","last_clean":"2023-10-19T12:45:40.364717+0000","last_became_active":"2023-10-19T12:45:40.359801+0000","last_became_peered":"2023-10-19T12:45:40.359801+0000","last_unstale":"2023-10-19T12:45:40.364717+0000","last_undegraded":"2023-10-19T12:45:40.364717+0000","last_fullsized":"2023-10-19T12:45:40.364717+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:45:10.025508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.350603+0000","last_change":"2023-10-19T12:45:40.350603+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.350603+0000","last_undegraded":"2023-10-19T12:45:40.350603+0000","last_fullsized":"2023-10-19T12:45:40.350603+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-19T12:45:40.367503+0000","last_change":"2023-10-19T12:45:40.364278+0000","last_active":"2023-10-19T12:45:40.367503+0000","last_peered":"2023-10-19T12:45:40.367503+0000","last_clean":"2023-10-19T12:45:40.367503+0000","last_became_active":"2023-10-19T12:45:40.363981+0000","last_became_peered":"2023-10-19T12:45:40.363981+0000","last_unstale":"2023-10-19T12:45:40.367503+0000","last_undegraded":"2023-10-19T12:45:40.367503+0000","last_fullsized":"2023-10-19T12:45:40.367503+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:50:18.153095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":3,"reported_epoch":38,"state":"creating+activating","last_fresh":"2023-10-19T12:45:41.349666+0000","last_change":"2023-10-19T12:45:41.349666+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:41.349666+0000","last_undegraded":"2023-10-19T12:45:41.349666+0000","last_fullsized":"2023-10-19T12:45:41.349666+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:55:02.268260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-19T12:45:40.364011+0000","last_change":"2023-10-19T12:45:40.359438+0000","last_active":"2023-10-19T12:45:40.364011+0000","last_peered":"2023-10-19T12:45:40.364011+0000","last_clean":"2023-10-19T12:45:40.364011+0000","last_became_active":"2023-10-19T12:45:40.359156+0000","last_became_peered":"2023-10-19T12:45:40.359156+0000","last_unstale":"2023-10-19T12:45:40.364011+0000","last_undegraded":"2023-10-19T12:45:40.364011+0000","last_fullsized":"2023-10-19T12:45:40.364011+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:10:03.197155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.344906+0000","last_change":"2023-10-19T12:45:40.344906+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.344906+0000","last_undegraded":"2023-10-19T12:45:40.344906+0000","last_fullsized":"2023-10-19T12:45:40.344906+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":11,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.351976+0000","last_change":"2023-10-19T12:45:40.359310+0000","last_active":"2023-10-19T12:45:41.351976+0000","last_peered":"2023-10-19T12:45:41.351976+0000","last_clean":"2023-10-19T12:45:41.351976+0000","last_became_active":"2023-10-19T12:45:40.359010+0000","last_became_peered":"2023-10-19T12:45:40.359010+0000","last_unstale":"2023-10-19T12:45:41.351976+0000","last_undegraded":"2023-10-19T12:45:41.351976+0000","last_fullsized":"2023-10-19T12:45:41.351976+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:57:29.725854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":179,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-19T12:45:40.349832+0000","last_change":"2023-10-19T12:44:08.636613+0000","last_active":"2023-10-19T12:45:40.349832+0000","last_peered":"2023-10-19T12:45:40.349832+0000","last_clean":"2023-10-19T12:45:40.349832+0000","last_became_active":"2023-10-19T12:44:08.635697+0000","last_became_peered":"2023-10-19T12:44:08.635697+0000","last_unstale":"2023-10-19T12:45:40.349832+0000","last_undegraded":"2023-10-19T12:45:40.349832+0000","last_fullsized":"2023-10-19T12:45:40.349832+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_clean_scrub_stamp":"2023-10-19T12:44:07.399738+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:15:23.685485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":70,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574871,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28028,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343812,"statfs":{"total":96636764160,"available":96608063488,"internally_reserved":0,"allocated":1032192,"data_stored":777587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:45:08 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.53400000000000003}]},{"osd":1,"last update":"Thu Oct 19 12:45:08 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.495}]},{"osd":2,"last update":"Thu Oct 19 12:45:08 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.64300000000000002}]},{"osd":3,"last update":"Thu Oct 19 12:45:08 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.68200000000000005}]},{"osd":4,"last update":"Thu Oct 19 12:45:08 2023","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.51200000000000001}]},{"osd":5,"last update":"Thu Oct 19 12:45:08 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.441}]},{"osd":6,"last update":"Thu Oct 19 12:45:08 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.61199999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574871,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27436,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344404,"statfs":{"total":96636764160,"available":96608669696,"internally_reserved":0,"allocated":425984,"data_stored":180124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:45:06 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.56599999999999995}]},{"osd":1,"last update":"Thu Oct 19 12:45:06 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.626},{"interface":"front","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.075,"5min":1.075,"15min":1.075},"last":0.50600000000000001}]},{"osd":2,"last update":"Thu Oct 19 12:45:06 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.59199999999999997}]},{"osd":3,"last update":"Thu Oct 19 12:45:06 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.57999999999999996}]},{"osd":4,"last update":"Thu Oct 19 12:45:06 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.371},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.32900000000000001}]},{"osd":5,"last update":"Thu Oct 19 12:45:06 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.44700000000000001}]},{"osd":7,"last update":"Thu Oct 19 12:45:06 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.48499999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574871,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27436,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344404,"statfs":{"total":96636764160,"available":96608669696,"internally_reserved":0,"allocated":425984,"data_stored":180124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.55000000000000004}]},{"osd":1,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.57999999999999996}]},{"osd":2,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.625},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.60899999999999999}]},{"osd":3,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.65400000000000003}]},{"osd":4,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.42599999999999999}]},{"osd":6,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.59199999999999997}]},{"osd":7,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.52900000000000003}]}]},{"osd":4,"up_from":13,"seq":55834574871,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27436,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344404,"statfs":{"total":96636764160,"available":96608669696,"internally_reserved":0,"allocated":425984,"data_stored":180124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:45:08 2023","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.46000000000000002}]},{"osd":1,"last update":"Thu Oct 19 12:45:08 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.57699999999999996}]},{"osd":2,"last update":"Thu Oct 19 12:45:08 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.44600000000000001}]},{"osd":3,"last update":"Thu Oct 19 12:45:08 2023","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.629}]},{"osd":5,"last update":"Thu Oct 19 12:45:08 2023","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.29099999999999998}]},{"osd":6,"last update":"Thu Oct 19 12:45:08 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.32600000000000001}]},{"osd":7,"last update":"Thu Oct 19 12:45:08 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.373},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.40999999999999998}]}]},{"osd":3,"up_from":13,"seq":55834574870,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31596,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340244,"statfs":{"total":96636764160,"available":96604409856,"internally_reserved":0,"allocated":425984,"data_stored":180124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":1.04,"5min":1.04,"15min":1.04},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.375,"5min":0.375,"15min":0.375},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"last":0.32500000000000001}]},{"osd":1,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.371},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.39700000000000002}]},{"osd":2,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.42699999999999999}]},{"osd":4,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.65600000000000003}]},{"osd":5,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.53400000000000003}]},{"osd":6,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.51700000000000002}]},{"osd":7,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.61899999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574872,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31596,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340244,"statfs":{"total":96636764160,"available":96604409856,"internally_reserved":0,"allocated":425984,"data_stored":180124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.314},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"last":0.251}]},{"osd":1,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.35399999999999998}]},{"osd":3,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.46800000000000003}]},{"osd":4,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.503},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.53800000000000003}]},{"osd":5,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.52500000000000002}]},{"osd":6,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.58899999999999997}]},{"osd":7,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.111,"5min":1.111,"15min":1.111},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.018,"5min":1.018,"15min":1.018},"last":0.55500000000000005}]}]},{"osd":1,"up_from":13,"seq":55834574871,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31596,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340244,"statfs":{"total":96636764160,"available":96604409856,"internally_reserved":0,"allocated":425984,"data_stored":180124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:45:05 2023","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.29699999999999999}]},{"osd":2,"last update":"Thu Oct 19 12:45:05 2023","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.441},{"interface":"front","average":{"1min":0.433,"5min":0.433,"15min":0.433},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.38}]},{"osd":3,"last update":"Thu Oct 19 12:45:05 2023","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.36399999999999999}]},{"osd":4,"last update":"Thu Oct 19 12:45:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.1910000000000001,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.314}]},{"osd":5,"last update":"Thu Oct 19 12:45:05 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.117,"5min":1.117,"15min":1.117},"last":0.41599999999999998}]},{"osd":6,"last update":"Thu Oct 19 12:45:05 2023","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.077,"5min":1.077,"15min":1.077},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.51400000000000001}]},{"osd":7,"last update":"Thu Oct 19 12:45:05 2023","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.45700000000000002}]}]},{"osd":0,"up_from":13,"seq":55834574872,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32176,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339664,"statfs":{"total":96636764160,"available":96603815936,"internally_reserved":0,"allocated":1015808,"data_stored":769873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Oct 19 12:45:09 2023","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"last":0.246},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"last":0.32800000000000001}]},{"osd":2,"last update":"Thu Oct 19 12:45:09 2023","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.29999999999999999}]},{"osd":3,"last update":"Thu Oct 19 12:45:09 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"last":0.311},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.40999999999999998}]},{"osd":4,"last update":"Thu Oct 19 12:45:09 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.39800000000000002}]},{"osd":5,"last update":"Thu Oct 19 12:45:09 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.056,"5min":1.056,"15min":1.056},"last":0.49399999999999999}]},{"osd":6,"last update":"Thu Oct 19 12:45:09 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.109,"5min":1.109,"15min":1.109},"last":0.42099999999999999}]},{"osd":7,"last update":"Thu Oct 19 12:45:09 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.44500000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-19T12:45:44.755 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-19T12:45:44.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.896+0000 7fb8c307c640 1 -- 172.21.15.57:0/4136418538 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8bc158910 msgr2=0x7fb8bc158cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:44.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.896+0000 7fb8c307c640 1 --2- 172.21.15.57:0/4136418538 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8bc158910 0x7fb8bc158cf0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fb8b0009830 tx=0x7fb8b002bd60 comp rx=0 tx=0).stop 2023-10-19T12:45:44.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.896+0000 7fb8c307c640 1 -- 172.21.15.57:0/4136418538 shutdown_connections 2023-10-19T12:45:44.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.896+0000 7fb8c307c640 1 --2- 172.21.15.57:0/4136418538 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8bc158910 0x7fb8bc158cf0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:44.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.896+0000 7fb8c307c640 1 -- 172.21.15.57:0/4136418538 >> 172.21.15.57:0/4136418538 conn(0x7fb8bc0af7a0 msgr2=0x7fb8bc0afb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:44.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.896+0000 7fb8c307c640 1 -- 172.21.15.57:0/4136418538 shutdown_connections 2023-10-19T12:45:44.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.896+0000 7fb8c307c640 1 -- 172.21.15.57:0/4136418538 wait complete. 2023-10-19T12:45:44.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.897+0000 7fb8c307c640 1 Processor -- start 2023-10-19T12:45:44.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.897+0000 7fb8c307c640 1 -- start start 2023-10-19T12:45:44.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.898+0000 7fb8c307c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb8bc0cd1c0 0x7fb8bc146690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:44.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.898+0000 7fb8c307c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb8bc0cd9e0 0x7fb8bc0c7830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:44.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.898+0000 7fb8c307c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8bc158910 0x7fb8bc0c7d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:44.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.898+0000 7fb8c307c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8bc0b1bf0 con 0x7fb8bc158910 2023-10-19T12:45:44.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.898+0000 7fb8c307c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8bc0c83f0 con 0x7fb8bc0cd1c0 2023-10-19T12:45:44.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.898+0000 7fb8c307c640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8bc0c86d0 con 0x7fb8bc0cd9e0 2023-10-19T12:45:44.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.898+0000 7fb8c1619640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8bc158910 0x7fb8bc0c7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:44.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.898+0000 7fb8c1619640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8bc158910 0x7fb8bc0c7d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59372/0 (socket says 172.21.15.57:59372) 2023-10-19T12:45:44.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.898+0000 7fb8c1619640 1 -- 172.21.15.57:0/2253382813 learned_addr learned my addr 172.21.15.57:0/2253382813 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:44.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.898+0000 7fb8c0e18640 1 --2- 172.21.15.57:0/2253382813 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb8bc0cd1c0 0x7fb8bc146690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:44.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.898+0000 7fb8bbfff640 1 --2- 172.21.15.57:0/2253382813 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb8bc0cd9e0 0x7fb8bc0c7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:44.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.898+0000 7fb8c1619640 1 -- 172.21.15.57:0/2253382813 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb8bc0cd9e0 msgr2=0x7fb8bc0c7830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:44.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.898+0000 7fb8c1619640 1 --2- 172.21.15.57:0/2253382813 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb8bc0cd9e0 0x7fb8bc0c7830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:44.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.898+0000 7fb8c1619640 1 -- 172.21.15.57:0/2253382813 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb8bc0cd1c0 msgr2=0x7fb8bc146690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:44.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.898+0000 7fb8c1619640 1 --2- 172.21.15.57:0/2253382813 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb8bc0cd1c0 0x7fb8bc146690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:44.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.898+0000 7fb8c1619640 1 -- 172.21.15.57:0/2253382813 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8b0009490 con 0x7fb8bc158910 2023-10-19T12:45:44.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.898+0000 7fb8c0e18640 1 --2- 172.21.15.57:0/2253382813 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb8bc0cd1c0 0x7fb8bc146690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:45:44.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.898+0000 7fb8c1619640 1 --2- 172.21.15.57:0/2253382813 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8bc158910 0x7fb8bc0c7d70 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fb8b400cae0 tx=0x7fb8b400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:44.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.898+0000 7fb8b97fa640 1 -- 172.21.15.57:0/2253382813 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8b400ac40 con 0x7fb8bc158910 2023-10-19T12:45:44.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.898+0000 7fb8b97fa640 1 -- 172.21.15.57:0/2253382813 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fb8b400ae90 con 0x7fb8bc158910 2023-10-19T12:45:44.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.898+0000 7fb8c307c640 1 -- 172.21.15.57:0/2253382813 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8bc0c89b0 con 0x7fb8bc158910 2023-10-19T12:45:44.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.898+0000 7fb8b97fa640 1 -- 172.21.15.57:0/2253382813 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8b4003bd0 con 0x7fb8bc158910 2023-10-19T12:45:44.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.899+0000 7fb8c307c640 1 -- 172.21.15.57:0/2253382813 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8bc0c8f70 con 0x7fb8bc158910 2023-10-19T12:45:44.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.900+0000 7fb8c307c640 1 -- 172.21.15.57:0/2253382813 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8bc0690e0 con 0x7fb8bc158910 2023-10-19T12:45:44.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.900+0000 7fb8b97fa640 1 -- 172.21.15.57:0/2253382813 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fb8b4003d30 con 0x7fb8bc158910 2023-10-19T12:45:44.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.900+0000 7fb8b97fa640 1 --2- 172.21.15.57:0/2253382813 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fb8a006ae40 0x7fb8a006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:44.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.900+0000 7fb8b97fa640 1 -- 172.21.15.57:0/2253382813 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fb8b4089fc0 con 0x7fb8bc158910 2023-10-19T12:45:44.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.901+0000 7fb8c0e18640 1 --2- 172.21.15.57:0/2253382813 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fb8a006ae40 0x7fb8a006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:44.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.901+0000 7fb8c0e18640 1 --2- 172.21.15.57:0/2253382813 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fb8a006ae40 0x7fb8a006d300 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fb8b0036000 tx=0x7fb8b0004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:44.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:44.905+0000 7fb8b97fa640 1 -- 172.21.15.57:0/2253382813 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb8b40bcca0 con 0x7fb8bc158910 2023-10-19T12:45:45.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.076+0000 7fb8c307c640 1 -- 172.21.15.57:0/2253382813 --> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb8bc15c7f0 con 0x7fb8a006ae40 2023-10-19T12:45:45.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.084+0000 7fb8b97fa640 1 -- 172.21.15.57:0/2253382813 <== mgr.4100 v2:172.21.15.179:6832/2609646285 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286395 (secure 0 0 0) 0x7fb8bc15c7f0 con 0x7fb8a006ae40 2023-10-19T12:45:45.089 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:45:45.091 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2023-10-19T12:45:45.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.086+0000 7fb8c307c640 1 -- 172.21.15.57:0/2253382813 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fb8a006ae40 msgr2=0x7fb8a006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:45.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.086+0000 7fb8c307c640 1 --2- 172.21.15.57:0/2253382813 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fb8a006ae40 0x7fb8a006d300 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fb8b0036000 tx=0x7fb8b0004730 comp rx=0 tx=0).stop 2023-10-19T12:45:45.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.086+0000 7fb8c307c640 1 -- 172.21.15.57:0/2253382813 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8bc158910 msgr2=0x7fb8bc0c7d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:45.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.086+0000 7fb8c307c640 1 --2- 172.21.15.57:0/2253382813 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8bc158910 0x7fb8bc0c7d70 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fb8b400cae0 tx=0x7fb8b400a060 comp rx=0 tx=0).stop 2023-10-19T12:45:45.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.086+0000 7fb8c307c640 1 -- 172.21.15.57:0/2253382813 shutdown_connections 2023-10-19T12:45:45.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.087+0000 7fb8c307c640 1 --2- 172.21.15.57:0/2253382813 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fb8a006ae40 0x7fb8a006d300 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:45.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.087+0000 7fb8c307c640 1 --2- 172.21.15.57:0/2253382813 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8bc158910 0x7fb8bc0c7d70 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:45.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.087+0000 7fb8c307c640 1 --2- 172.21.15.57:0/2253382813 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb8bc0cd9e0 0x7fb8bc0c7830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:45.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.087+0000 7fb8c307c640 1 --2- 172.21.15.57:0/2253382813 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb8bc0cd1c0 0x7fb8bc146690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:45.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.087+0000 7fb8c307c640 1 -- 172.21.15.57:0/2253382813 >> 172.21.15.57:0/2253382813 conn(0x7fb8bc0af7a0 msgr2=0x7fb8bc06db10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:45.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.087+0000 7fb8c307c640 1 -- 172.21.15.57:0/2253382813 shutdown_connections 2023-10-19T12:45:45.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.087+0000 7fb8c307c640 1 -- 172.21.15.57:0/2253382813 wait complete. 2023-10-19T12:45:45.100 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":81,"stamp":"2023-10-19T12:45:43.336334+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590438,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":168,"num_write_kb":2417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":126,"ondisk_log_size":126,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":56,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":237300,"kb_used_data":4496,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754737420,"statfs":{"total":773094113280,"available":772851118080,"internally_reserved":0,"allocated":4603904,"data_stored":2628204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":238276153},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":70,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.008337"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.347634+0000","last_change":"2023-10-19T12:45:40.347634+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.347634+0000","last_undegraded":"2023-10-19T12:45:40.347634+0000","last_fullsized":"2023-10-19T12:45:40.347634+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.349030+0000","last_change":"2023-10-19T12:45:40.349030+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.349030+0000","last_undegraded":"2023-10-19T12:45:40.349030+0000","last_fullsized":"2023-10-19T12:45:40.349030+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.376066+0000","last_change":"2023-10-19T12:45:41.371589+0000","last_active":"2023-10-19T12:45:41.376066+0000","last_peered":"2023-10-19T12:45:41.376066+0000","last_clean":"2023-10-19T12:45:41.376066+0000","last_became_active":"2023-10-19T12:45:41.371287+0000","last_became_peered":"2023-10-19T12:45:41.371287+0000","last_unstale":"2023-10-19T12:45:41.376066+0000","last_undegraded":"2023-10-19T12:45:41.376066+0000","last_fullsized":"2023-10-19T12:45:41.376066+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:16:37.419534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.370328+0000","last_change":"2023-10-19T12:45:41.363930+0000","last_active":"2023-10-19T12:45:41.370328+0000","last_peered":"2023-10-19T12:45:41.370328+0000","last_clean":"2023-10-19T12:45:41.370328+0000","last_became_active":"2023-10-19T12:45:41.363668+0000","last_became_peered":"2023-10-19T12:45:41.363668+0000","last_unstale":"2023-10-19T12:45:41.370328+0000","last_undegraded":"2023-10-19T12:45:41.370328+0000","last_fullsized":"2023-10-19T12:45:41.370328+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:51:05.063828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.370788+0000","last_change":"2023-10-19T12:45:41.363093+0000","last_active":"2023-10-19T12:45:41.370788+0000","last_peered":"2023-10-19T12:45:41.370788+0000","last_clean":"2023-10-19T12:45:41.370788+0000","last_became_active":"2023-10-19T12:45:41.362804+0000","last_became_peered":"2023-10-19T12:45:41.362804+0000","last_unstale":"2023-10-19T12:45:41.370788+0000","last_undegraded":"2023-10-19T12:45:41.370788+0000","last_fullsized":"2023-10-19T12:45:41.370788+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:42:33.722992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.374791+0000","last_change":"2023-10-19T12:45:41.359415+0000","last_active":"2023-10-19T12:45:41.374791+0000","last_peered":"2023-10-19T12:45:41.374791+0000","last_clean":"2023-10-19T12:45:41.374791+0000","last_became_active":"2023-10-19T12:45:41.359120+0000","last_became_peered":"2023-10-19T12:45:41.359120+0000","last_unstale":"2023-10-19T12:45:41.374791+0000","last_undegraded":"2023-10-19T12:45:41.374791+0000","last_fullsized":"2023-10-19T12:45:41.374791+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:48:25.218615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":3,"reported_epoch":38,"state":"creating+activating","last_fresh":"2023-10-19T12:45:41.350760+0000","last_change":"2023-10-19T12:45:41.350760+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:41.350760+0000","last_undegraded":"2023-10-19T12:45:41.350760+0000","last_fullsized":"2023-10-19T12:45:41.350760+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:38:37.793736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.349543+0000","last_change":"2023-10-19T12:45:40.349543+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.349543+0000","last_undegraded":"2023-10-19T12:45:40.349543+0000","last_fullsized":"2023-10-19T12:45:40.349543+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.375561+0000","last_change":"2023-10-19T12:45:41.368731+0000","last_active":"2023-10-19T12:45:41.375561+0000","last_peered":"2023-10-19T12:45:41.375561+0000","last_clean":"2023-10-19T12:45:41.375561+0000","last_became_active":"2023-10-19T12:45:41.368006+0000","last_became_peered":"2023-10-19T12:45:41.368006+0000","last_unstale":"2023-10-19T12:45:41.375561+0000","last_undegraded":"2023-10-19T12:45:41.375561+0000","last_fullsized":"2023-10-19T12:45:41.375561+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:48:25.218615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.345142+0000","last_change":"2023-10-19T12:45:40.345142+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.345142+0000","last_undegraded":"2023-10-19T12:45:40.345142+0000","last_fullsized":"2023-10-19T12:45:40.345142+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.346201+0000","last_change":"2023-10-19T12:45:40.346201+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.346201+0000","last_undegraded":"2023-10-19T12:45:40.346201+0000","last_fullsized":"2023-10-19T12:45:40.346201+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.370682+0000","last_change":"2023-10-19T12:45:41.357975+0000","last_active":"2023-10-19T12:45:41.370682+0000","last_peered":"2023-10-19T12:45:41.370682+0000","last_clean":"2023-10-19T12:45:41.370682+0000","last_became_active":"2023-10-19T12:45:41.357717+0000","last_became_peered":"2023-10-19T12:45:41.357717+0000","last_unstale":"2023-10-19T12:45:41.370682+0000","last_undegraded":"2023-10-19T12:45:41.370682+0000","last_fullsized":"2023-10-19T12:45:41.370682+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:49:21.932303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.345686+0000","last_change":"2023-10-19T12:45:40.345686+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.345686+0000","last_undegraded":"2023-10-19T12:45:40.345686+0000","last_fullsized":"2023-10-19T12:45:40.345686+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.351066+0000","last_change":"2023-10-19T12:45:40.351066+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.351066+0000","last_undegraded":"2023-10-19T12:45:40.351066+0000","last_fullsized":"2023-10-19T12:45:40.351066+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":3,"reported_epoch":38,"state":"creating+activating","last_fresh":"2023-10-19T12:45:41.350355+0000","last_change":"2023-10-19T12:45:41.350355+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:41.350355+0000","last_undegraded":"2023-10-19T12:45:41.350355+0000","last_fullsized":"2023-10-19T12:45:41.350355+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:10:04.203620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":3,"reported_epoch":38,"state":"creating+activating","last_fresh":"2023-10-19T12:45:41.352539+0000","last_change":"2023-10-19T12:45:41.352539+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:41.352539+0000","last_undegraded":"2023-10-19T12:45:41.352539+0000","last_fullsized":"2023-10-19T12:45:41.352539+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:21:51.336716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":3,"reported_epoch":38,"state":"creating+activating","last_fresh":"2023-10-19T12:45:41.351475+0000","last_change":"2023-10-19T12:45:41.351475+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:41.351475+0000","last_undegraded":"2023-10-19T12:45:41.351475+0000","last_fullsized":"2023-10-19T12:45:41.351475+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:32:37.014614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":3,"reported_epoch":38,"state":"creating+activating","last_fresh":"2023-10-19T12:45:41.348402+0000","last_change":"2023-10-19T12:45:41.348402+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:41.348402+0000","last_undegraded":"2023-10-19T12:45:41.348402+0000","last_fullsized":"2023-10-19T12:45:41.348402+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:53:30.629136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.364662+0000","last_change":"2023-10-19T12:45:41.356683+0000","last_active":"2023-10-19T12:45:41.364662+0000","last_peered":"2023-10-19T12:45:41.364662+0000","last_clean":"2023-10-19T12:45:41.364662+0000","last_became_active":"2023-10-19T12:45:41.356404+0000","last_became_peered":"2023-10-19T12:45:41.356404+0000","last_unstale":"2023-10-19T12:45:41.364662+0000","last_undegraded":"2023-10-19T12:45:41.364662+0000","last_fullsized":"2023-10-19T12:45:41.364662+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:30:20.055674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":3,"reported_epoch":38,"state":"creating+activating","last_fresh":"2023-10-19T12:45:41.351396+0000","last_change":"2023-10-19T12:45:41.351396+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:41.351396+0000","last_undegraded":"2023-10-19T12:45:41.351396+0000","last_fullsized":"2023-10-19T12:45:41.351396+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:48:25.218615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.344680+0000","last_change":"2023-10-19T12:45:40.344680+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.344680+0000","last_undegraded":"2023-10-19T12:45:40.344680+0000","last_fullsized":"2023-10-19T12:45:40.344680+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.348310+0000","last_change":"2023-10-19T12:45:40.348310+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.348310+0000","last_undegraded":"2023-10-19T12:45:40.348310+0000","last_fullsized":"2023-10-19T12:45:40.348310+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":3,"reported_epoch":38,"state":"creating+activating","last_fresh":"2023-10-19T12:45:41.350788+0000","last_change":"2023-10-19T12:45:41.350788+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:41.350788+0000","last_undegraded":"2023-10-19T12:45:41.350788+0000","last_fullsized":"2023-10-19T12:45:41.350788+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:44:04.457483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.371142+0000","last_change":"2023-10-19T12:45:41.361356+0000","last_active":"2023-10-19T12:45:41.371142+0000","last_peered":"2023-10-19T12:45:41.371142+0000","last_clean":"2023-10-19T12:45:41.371142+0000","last_became_active":"2023-10-19T12:45:41.361082+0000","last_became_peered":"2023-10-19T12:45:41.361082+0000","last_unstale":"2023-10-19T12:45:41.371142+0000","last_undegraded":"2023-10-19T12:45:41.371142+0000","last_fullsized":"2023-10-19T12:45:41.371142+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:45:11.031973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":3,"reported_epoch":38,"state":"creating+activating","last_fresh":"2023-10-19T12:45:41.350918+0000","last_change":"2023-10-19T12:45:41.350918+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:41.350918+0000","last_undegraded":"2023-10-19T12:45:41.350918+0000","last_fullsized":"2023-10-19T12:45:41.350918+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:49:21.932303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.375784+0000","last_change":"2023-10-19T12:45:41.372032+0000","last_active":"2023-10-19T12:45:41.375784+0000","last_peered":"2023-10-19T12:45:41.375784+0000","last_clean":"2023-10-19T12:45:41.375784+0000","last_became_active":"2023-10-19T12:45:41.371582+0000","last_became_peered":"2023-10-19T12:45:41.371582+0000","last_unstale":"2023-10-19T12:45:41.375784+0000","last_undegraded":"2023-10-19T12:45:41.375784+0000","last_fullsized":"2023-10-19T12:45:41.375784+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:55:02.268260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":3,"reported_epoch":38,"state":"creating+activating","last_fresh":"2023-10-19T12:45:41.348601+0000","last_change":"2023-10-19T12:45:41.348601+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:41.348601+0000","last_undegraded":"2023-10-19T12:45:41.348601+0000","last_fullsized":"2023-10-19T12:45:41.348601+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:36:07.043432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.345780+0000","last_change":"2023-10-19T12:45:40.345780+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.345780+0000","last_undegraded":"2023-10-19T12:45:40.345780+0000","last_fullsized":"2023-10-19T12:45:40.345780+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":3,"reported_epoch":38,"state":"creating+activating","last_fresh":"2023-10-19T12:45:41.351166+0000","last_change":"2023-10-19T12:45:41.351166+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:41.351166+0000","last_undegraded":"2023-10-19T12:45:41.351166+0000","last_fullsized":"2023-10-19T12:45:41.351166+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:16:37.419534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":3,"reported_epoch":38,"state":"creating+activating","last_fresh":"2023-10-19T12:45:41.348399+0000","last_change":"2023-10-19T12:45:41.348399+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:41.348399+0000","last_undegraded":"2023-10-19T12:45:41.348399+0000","last_fullsized":"2023-10-19T12:45:41.348399+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T12:45:47.935169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":["6","3"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":3,"reported_epoch":38,"state":"creating+activating","last_fresh":"2023-10-19T12:45:41.351136+0000","last_change":"2023-10-19T12:45:41.351136+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:41.351136+0000","last_undegraded":"2023-10-19T12:45:41.351136+0000","last_fullsized":"2023-10-19T12:45:41.351136+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:10:04.203620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.369704+0000","last_change":"2023-10-19T12:45:41.356009+0000","last_active":"2023-10-19T12:45:41.369704+0000","last_peered":"2023-10-19T12:45:41.369704+0000","last_clean":"2023-10-19T12:45:41.369704+0000","last_became_active":"2023-10-19T12:45:41.355714+0000","last_became_peered":"2023-10-19T12:45:41.355714+0000","last_unstale":"2023-10-19T12:45:41.369704+0000","last_undegraded":"2023-10-19T12:45:41.369704+0000","last_fullsized":"2023-10-19T12:45:41.369704+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:16:37.419534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":3,"reported_epoch":38,"state":"creating+activating","last_fresh":"2023-10-19T12:45:41.351973+0000","last_change":"2023-10-19T12:45:41.351973+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:41.351973+0000","last_undegraded":"2023-10-19T12:45:41.351973+0000","last_fullsized":"2023-10-19T12:45:41.351973+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:53:29.708835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-19T12:45:40.364691+0000","last_change":"2023-10-19T12:45:40.359697+0000","last_active":"2023-10-19T12:45:40.364691+0000","last_peered":"2023-10-19T12:45:40.364691+0000","last_clean":"2023-10-19T12:45:40.364691+0000","last_became_active":"2023-10-19T12:45:40.359413+0000","last_became_peered":"2023-10-19T12:45:40.359413+0000","last_unstale":"2023-10-19T12:45:40.364691+0000","last_undegraded":"2023-10-19T12:45:40.364691+0000","last_fullsized":"2023-10-19T12:45:40.364691+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:36:38.697134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.375801+0000","last_change":"2023-10-19T12:45:41.372556+0000","last_active":"2023-10-19T12:45:41.375801+0000","last_peered":"2023-10-19T12:45:41.375801+0000","last_clean":"2023-10-19T12:45:41.375801+0000","last_became_active":"2023-10-19T12:45:41.372184+0000","last_became_peered":"2023-10-19T12:45:41.372184+0000","last_unstale":"2023-10-19T12:45:41.375801+0000","last_undegraded":"2023-10-19T12:45:41.375801+0000","last_fullsized":"2023-10-19T12:45:41.375801+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:10:04.203620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"38'1","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.370313+0000","last_change":"2023-10-19T12:45:40.352581+0000","last_active":"2023-10-19T12:45:41.370313+0000","last_peered":"2023-10-19T12:45:41.370313+0000","last_clean":"2023-10-19T12:45:41.370313+0000","last_became_active":"2023-10-19T12:45:40.352300+0000","last_became_peered":"2023-10-19T12:45:40.352300+0000","last_unstale":"2023-10-19T12:45:41.370313+0000","last_undegraded":"2023-10-19T12:45:41.370313+0000","last_fullsized":"2023-10-19T12:45:41.370313+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:59:58.100040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.348927+0000","last_change":"2023-10-19T12:45:40.348927+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.348927+0000","last_undegraded":"2023-10-19T12:45:40.348927+0000","last_fullsized":"2023-10-19T12:45:40.348927+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"38'2","reported_seq":13,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.374990+0000","last_change":"2023-10-19T12:45:40.355990+0000","last_active":"2023-10-19T12:45:41.374990+0000","last_peered":"2023-10-19T12:45:41.374990+0000","last_clean":"2023-10-19T12:45:41.374990+0000","last_became_active":"2023-10-19T12:45:40.355666+0000","last_became_peered":"2023-10-19T12:45:40.355666+0000","last_unstale":"2023-10-19T12:45:41.374990+0000","last_undegraded":"2023-10-19T12:45:41.374990+0000","last_fullsized":"2023-10-19T12:45:41.374990+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:53:28.702370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.349506+0000","last_change":"2023-10-19T12:45:40.349506+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.349506+0000","last_undegraded":"2023-10-19T12:45:40.349506+0000","last_fullsized":"2023-10-19T12:45:40.349506+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-19T12:45:40.367475+0000","last_change":"2023-10-19T12:45:40.364018+0000","last_active":"2023-10-19T12:45:40.367475+0000","last_peered":"2023-10-19T12:45:40.367475+0000","last_clean":"2023-10-19T12:45:40.367475+0000","last_became_active":"2023-10-19T12:45:40.363763+0000","last_became_peered":"2023-10-19T12:45:40.363763+0000","last_unstale":"2023-10-19T12:45:40.367475+0000","last_undegraded":"2023-10-19T12:45:40.367475+0000","last_fullsized":"2023-10-19T12:45:40.367475+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:36:38.697134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.363923+0000","last_change":"2023-10-19T12:45:41.355176+0000","last_active":"2023-10-19T12:45:41.363923+0000","last_peered":"2023-10-19T12:45:41.363923+0000","last_clean":"2023-10-19T12:45:41.363923+0000","last_became_active":"2023-10-19T12:45:41.354687+0000","last_became_peered":"2023-10-19T12:45:41.354687+0000","last_unstale":"2023-10-19T12:45:41.363923+0000","last_undegraded":"2023-10-19T12:45:41.363923+0000","last_fullsized":"2023-10-19T12:45:41.363923+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:53:29.708835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-19T12:45:40.362034+0000","last_change":"2023-10-19T12:45:40.357988+0000","last_active":"2023-10-19T12:45:40.362034+0000","last_peered":"2023-10-19T12:45:40.362034+0000","last_clean":"2023-10-19T12:45:40.362034+0000","last_became_active":"2023-10-19T12:45:40.357478+0000","last_became_peered":"2023-10-19T12:45:40.357478+0000","last_unstale":"2023-10-19T12:45:40.362034+0000","last_undegraded":"2023-10-19T12:45:40.362034+0000","last_fullsized":"2023-10-19T12:45:40.362034+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:49:20.925838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":3,"reported_epoch":38,"state":"creating+activating","last_fresh":"2023-10-19T12:45:41.349960+0000","last_change":"2023-10-19T12:45:41.349960+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:41.349960+0000","last_undegraded":"2023-10-19T12:45:41.349960+0000","last_fullsized":"2023-10-19T12:45:41.349960+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:44:04.457483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":11,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.349382+0000","last_change":"2023-10-19T12:45:40.355247+0000","last_active":"2023-10-19T12:45:41.349382+0000","last_peered":"2023-10-19T12:45:41.349382+0000","last_clean":"2023-10-19T12:45:41.349382+0000","last_became_active":"2023-10-19T12:45:40.354967+0000","last_became_peered":"2023-10-19T12:45:40.354967+0000","last_unstale":"2023-10-19T12:45:41.349382+0000","last_undegraded":"2023-10-19T12:45:41.349382+0000","last_fullsized":"2023-10-19T12:45:41.349382+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:48:24.212150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.350186+0000","last_change":"2023-10-19T12:45:40.350186+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.350186+0000","last_undegraded":"2023-10-19T12:45:40.350186+0000","last_fullsized":"2023-10-19T12:45:40.350186+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":11,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.348919+0000","last_change":"2023-10-19T12:45:40.359949+0000","last_active":"2023-10-19T12:45:41.348919+0000","last_peered":"2023-10-19T12:45:41.348919+0000","last_clean":"2023-10-19T12:45:41.348919+0000","last_became_active":"2023-10-19T12:45:40.359690+0000","last_became_peered":"2023-10-19T12:45:40.359690+0000","last_unstale":"2023-10-19T12:45:41.348919+0000","last_undegraded":"2023-10-19T12:45:41.348919+0000","last_fullsized":"2023-10-19T12:45:41.348919+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:44:03.451018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.351154+0000","last_change":"2023-10-19T12:45:40.351154+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.351154+0000","last_undegraded":"2023-10-19T12:45:40.351154+0000","last_fullsized":"2023-10-19T12:45:40.351154+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-19T12:45:40.364109+0000","last_change":"2023-10-19T12:45:40.360289+0000","last_active":"2023-10-19T12:45:40.364109+0000","last_peered":"2023-10-19T12:45:40.364109+0000","last_clean":"2023-10-19T12:45:40.364109+0000","last_became_active":"2023-10-19T12:45:40.359850+0000","last_became_peered":"2023-10-19T12:45:40.359850+0000","last_unstale":"2023-10-19T12:45:40.364109+0000","last_undegraded":"2023-10-19T12:45:40.364109+0000","last_fullsized":"2023-10-19T12:45:40.364109+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:30:19.049209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.370304+0000","last_change":"2023-10-19T12:45:41.363350+0000","last_active":"2023-10-19T12:45:41.370304+0000","last_peered":"2023-10-19T12:45:41.370304+0000","last_clean":"2023-10-19T12:45:41.370304+0000","last_became_active":"2023-10-19T12:45:41.363067+0000","last_became_peered":"2023-10-19T12:45:41.363067+0000","last_unstale":"2023-10-19T12:45:41.370304+0000","last_undegraded":"2023-10-19T12:45:41.370304+0000","last_fullsized":"2023-10-19T12:45:41.370304+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:09:22.266843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-19T12:45:40.363651+0000","last_change":"2023-10-19T12:45:40.356475+0000","last_active":"2023-10-19T12:45:40.363651+0000","last_peered":"2023-10-19T12:45:40.363651+0000","last_clean":"2023-10-19T12:45:40.363651+0000","last_became_active":"2023-10-19T12:45:40.356178+0000","last_became_peered":"2023-10-19T12:45:40.356178+0000","last_unstale":"2023-10-19T12:45:40.363651+0000","last_undegraded":"2023-10-19T12:45:40.363651+0000","last_fullsized":"2023-10-19T12:45:40.363651+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:59:58.100040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":3,"reported_epoch":38,"state":"creating+activating","last_fresh":"2023-10-19T12:45:41.350085+0000","last_change":"2023-10-19T12:45:41.350085+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:41.350085+0000","last_undegraded":"2023-10-19T12:45:41.350085+0000","last_fullsized":"2023-10-19T12:45:41.350085+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:29:00.286113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-19T12:45:40.364238+0000","last_change":"2023-10-19T12:45:40.360586+0000","last_active":"2023-10-19T12:45:40.364238+0000","last_peered":"2023-10-19T12:45:40.364238+0000","last_clean":"2023-10-19T12:45:40.364238+0000","last_became_active":"2023-10-19T12:45:40.360327+0000","last_became_peered":"2023-10-19T12:45:40.360327+0000","last_unstale":"2023-10-19T12:45:40.364238+0000","last_undegraded":"2023-10-19T12:45:40.364238+0000","last_fullsized":"2023-10-19T12:45:40.364238+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:55:01.261795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.375472+0000","last_change":"2023-10-19T12:45:41.366187+0000","last_active":"2023-10-19T12:45:41.375472+0000","last_peered":"2023-10-19T12:45:41.375472+0000","last_clean":"2023-10-19T12:45:41.375472+0000","last_became_active":"2023-10-19T12:45:41.365896+0000","last_became_peered":"2023-10-19T12:45:41.365896+0000","last_unstale":"2023-10-19T12:45:41.375472+0000","last_undegraded":"2023-10-19T12:45:41.375472+0000","last_fullsized":"2023-10-19T12:45:41.375472+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:55:02.268260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-19T12:45:40.361876+0000","last_change":"2023-10-19T12:45:40.358267+0000","last_active":"2023-10-19T12:45:40.361876+0000","last_peered":"2023-10-19T12:45:40.361876+0000","last_clean":"2023-10-19T12:45:40.361876+0000","last_became_active":"2023-10-19T12:45:40.358000+0000","last_became_peered":"2023-10-19T12:45:40.358000+0000","last_unstale":"2023-10-19T12:45:40.361876+0000","last_undegraded":"2023-10-19T12:45:40.361876+0000","last_fullsized":"2023-10-19T12:45:40.361876+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:48:24.212150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.348513+0000","last_change":"2023-10-19T12:45:40.348513+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.348513+0000","last_undegraded":"2023-10-19T12:45:40.348513+0000","last_fullsized":"2023-10-19T12:45:40.348513+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-19T12:45:40.364295+0000","last_change":"2023-10-19T12:45:40.361715+0000","last_active":"2023-10-19T12:45:40.364295+0000","last_peered":"2023-10-19T12:45:40.364295+0000","last_clean":"2023-10-19T12:45:40.364295+0000","last_became_active":"2023-10-19T12:45:40.361442+0000","last_became_peered":"2023-10-19T12:45:40.361442+0000","last_unstale":"2023-10-19T12:45:40.364295+0000","last_undegraded":"2023-10-19T12:45:40.364295+0000","last_fullsized":"2023-10-19T12:45:40.364295+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:44:03.451018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.375150+0000","last_change":"2023-10-19T12:45:41.364032+0000","last_active":"2023-10-19T12:45:41.375150+0000","last_peered":"2023-10-19T12:45:41.375150+0000","last_clean":"2023-10-19T12:45:41.375150+0000","last_became_active":"2023-10-19T12:45:41.363734+0000","last_became_peered":"2023-10-19T12:45:41.363734+0000","last_unstale":"2023-10-19T12:45:41.375150+0000","last_undegraded":"2023-10-19T12:45:41.375150+0000","last_fullsized":"2023-10-19T12:45:41.375150+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:32:37.014614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-19T12:45:40.363994+0000","last_change":"2023-10-19T12:45:40.360672+0000","last_active":"2023-10-19T12:45:40.363994+0000","last_peered":"2023-10-19T12:45:40.363994+0000","last_clean":"2023-10-19T12:45:40.363994+0000","last_became_active":"2023-10-19T12:45:40.360396+0000","last_became_peered":"2023-10-19T12:45:40.360396+0000","last_unstale":"2023-10-19T12:45:40.363994+0000","last_undegraded":"2023-10-19T12:45:40.363994+0000","last_fullsized":"2023-10-19T12:45:40.363994+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:28:59.279648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.347231+0000","last_change":"2023-10-19T12:45:40.347231+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.347231+0000","last_undegraded":"2023-10-19T12:45:40.347231+0000","last_fullsized":"2023-10-19T12:45:40.347231+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":11,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.348645+0000","last_change":"2023-10-19T12:45:40.359407+0000","last_active":"2023-10-19T12:45:41.348645+0000","last_peered":"2023-10-19T12:45:41.348645+0000","last_clean":"2023-10-19T12:45:41.348645+0000","last_became_active":"2023-10-19T12:45:40.359134+0000","last_became_peered":"2023-10-19T12:45:40.359134+0000","last_unstale":"2023-10-19T12:45:41.348645+0000","last_undegraded":"2023-10-19T12:45:41.348645+0000","last_fullsized":"2023-10-19T12:45:41.348645+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:59:58.100040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":3,"reported_epoch":38,"state":"creating+activating","last_fresh":"2023-10-19T12:45:41.349119+0000","last_change":"2023-10-19T12:45:41.349119+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:41.349119+0000","last_undegraded":"2023-10-19T12:45:41.349119+0000","last_fullsized":"2023-10-19T12:45:41.349119+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:16:37.419534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":11,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.350155+0000","last_change":"2023-10-19T12:45:40.359548+0000","last_active":"2023-10-19T12:45:41.350155+0000","last_peered":"2023-10-19T12:45:41.350155+0000","last_clean":"2023-10-19T12:45:41.350155+0000","last_became_active":"2023-10-19T12:45:40.359282+0000","last_became_peered":"2023-10-19T12:45:40.359282+0000","last_unstale":"2023-10-19T12:45:41.350155+0000","last_undegraded":"2023-10-19T12:45:41.350155+0000","last_fullsized":"2023-10-19T12:45:41.350155+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:39:09.771537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.350379+0000","last_change":"2023-10-19T12:45:40.350379+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.350379+0000","last_undegraded":"2023-10-19T12:45:40.350379+0000","last_fullsized":"2023-10-19T12:45:40.350379+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"38'3","reported_seq":11,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.352796+0000","last_change":"2023-10-19T12:45:40.358054+0000","last_active":"2023-10-19T12:45:41.352796+0000","last_peered":"2023-10-19T12:45:41.352796+0000","last_clean":"2023-10-19T12:45:41.352796+0000","last_became_active":"2023-10-19T12:45:40.357803+0000","last_became_peered":"2023-10-19T12:45:40.357803+0000","last_unstale":"2023-10-19T12:45:41.352796+0000","last_undegraded":"2023-10-19T12:45:41.352796+0000","last_fullsized":"2023-10-19T12:45:41.352796+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:07:55.739920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.350100+0000","last_change":"2023-10-19T12:45:40.350100+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.350100+0000","last_undegraded":"2023-10-19T12:45:40.350100+0000","last_fullsized":"2023-10-19T12:45:40.350100+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-19T12:45:40.362007+0000","last_change":"2023-10-19T12:45:40.358450+0000","last_active":"2023-10-19T12:45:40.362007+0000","last_peered":"2023-10-19T12:45:40.362007+0000","last_clean":"2023-10-19T12:45:40.362007+0000","last_became_active":"2023-10-19T12:45:40.356778+0000","last_became_peered":"2023-10-19T12:45:40.356778+0000","last_unstale":"2023-10-19T12:45:40.362007+0000","last_undegraded":"2023-10-19T12:45:40.362007+0000","last_fullsized":"2023-10-19T12:45:40.362007+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:16:36.413069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.345759+0000","last_change":"2023-10-19T12:45:40.345759+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.345759+0000","last_undegraded":"2023-10-19T12:45:40.345759+0000","last_fullsized":"2023-10-19T12:45:40.345759+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-19T12:45:40.361384+0000","last_change":"2023-10-19T12:45:40.357947+0000","last_active":"2023-10-19T12:45:40.361384+0000","last_peered":"2023-10-19T12:45:40.361384+0000","last_clean":"2023-10-19T12:45:40.361384+0000","last_became_active":"2023-10-19T12:45:40.357669+0000","last_became_peered":"2023-10-19T12:45:40.357669+0000","last_unstale":"2023-10-19T12:45:40.361384+0000","last_undegraded":"2023-10-19T12:45:40.361384+0000","last_fullsized":"2023-10-19T12:45:40.361384+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T12:45:46.928704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.350198+0000","last_change":"2023-10-19T12:45:40.350198+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.350198+0000","last_undegraded":"2023-10-19T12:45:40.350198+0000","last_fullsized":"2023-10-19T12:45:40.350198+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"38'1","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.370536+0000","last_change":"2023-10-19T12:45:40.354628+0000","last_active":"2023-10-19T12:45:41.370536+0000","last_peered":"2023-10-19T12:45:41.370536+0000","last_clean":"2023-10-19T12:45:41.370536+0000","last_became_active":"2023-10-19T12:45:40.354367+0000","last_became_peered":"2023-10-19T12:45:40.354367+0000","last_unstale":"2023-10-19T12:45:41.370536+0000","last_undegraded":"2023-10-19T12:45:41.370536+0000","last_fullsized":"2023-10-19T12:45:41.370536+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:33:49.817824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.375358+0000","last_change":"2023-10-19T12:45:41.367225+0000","last_active":"2023-10-19T12:45:41.375358+0000","last_peered":"2023-10-19T12:45:41.375358+0000","last_clean":"2023-10-19T12:45:41.375358+0000","last_became_active":"2023-10-19T12:45:41.366782+0000","last_became_peered":"2023-10-19T12:45:41.366782+0000","last_unstale":"2023-10-19T12:45:41.375358+0000","last_undegraded":"2023-10-19T12:45:41.375358+0000","last_fullsized":"2023-10-19T12:45:41.375358+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:50:19.159560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":11,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.349510+0000","last_change":"2023-10-19T12:45:40.357501+0000","last_active":"2023-10-19T12:45:41.349510+0000","last_peered":"2023-10-19T12:45:41.349510+0000","last_clean":"2023-10-19T12:45:41.349510+0000","last_became_active":"2023-10-19T12:45:40.355411+0000","last_became_peered":"2023-10-19T12:45:40.355411+0000","last_unstale":"2023-10-19T12:45:41.349510+0000","last_undegraded":"2023-10-19T12:45:41.349510+0000","last_fullsized":"2023-10-19T12:45:41.349510+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:32:36.008149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":3,"reported_epoch":38,"state":"creating+activating","last_fresh":"2023-10-19T12:45:41.353064+0000","last_change":"2023-10-19T12:45:41.353064+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:41.353064+0000","last_undegraded":"2023-10-19T12:45:41.353064+0000","last_fullsized":"2023-10-19T12:45:41.353064+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:59:55.870354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"38'2","reported_seq":13,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.376098+0000","last_change":"2023-10-19T12:45:40.357725+0000","last_active":"2023-10-19T12:45:41.376098+0000","last_peered":"2023-10-19T12:45:41.376098+0000","last_clean":"2023-10-19T12:45:41.376098+0000","last_became_active":"2023-10-19T12:45:40.357417+0000","last_became_peered":"2023-10-19T12:45:40.357417+0000","last_unstale":"2023-10-19T12:45:41.376098+0000","last_undegraded":"2023-10-19T12:45:41.376098+0000","last_fullsized":"2023-10-19T12:45:41.376098+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:26:34.780735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.375456+0000","last_change":"2023-10-19T12:45:41.366181+0000","last_active":"2023-10-19T12:45:41.375456+0000","last_peered":"2023-10-19T12:45:41.375456+0000","last_clean":"2023-10-19T12:45:41.375456+0000","last_became_active":"2023-10-19T12:45:41.365882+0000","last_became_peered":"2023-10-19T12:45:41.365882+0000","last_unstale":"2023-10-19T12:45:41.375456+0000","last_undegraded":"2023-10-19T12:45:41.375456+0000","last_fullsized":"2023-10-19T12:45:41.375456+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:29:00.286113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":11,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.343006+0000","last_change":"2023-10-19T12:45:40.355357+0000","last_active":"2023-10-19T12:45:41.343006+0000","last_peered":"2023-10-19T12:45:41.343006+0000","last_clean":"2023-10-19T12:45:41.343006+0000","last_became_active":"2023-10-19T12:45:40.355097+0000","last_became_peered":"2023-10-19T12:45:40.355097+0000","last_unstale":"2023-10-19T12:45:41.343006+0000","last_undegraded":"2023-10-19T12:45:41.343006+0000","last_fullsized":"2023-10-19T12:45:41.343006+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:23:29.827547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":3,"reported_epoch":38,"state":"creating+activating","last_fresh":"2023-10-19T12:45:41.349483+0000","last_change":"2023-10-19T12:45:41.349483+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:41.349483+0000","last_undegraded":"2023-10-19T12:45:41.349483+0000","last_fullsized":"2023-10-19T12:45:41.349483+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:59:59.106505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":11,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.351381+0000","last_change":"2023-10-19T12:45:40.360123+0000","last_active":"2023-10-19T12:45:41.351381+0000","last_peered":"2023-10-19T12:45:41.351381+0000","last_clean":"2023-10-19T12:45:41.351381+0000","last_became_active":"2023-10-19T12:45:40.359767+0000","last_became_peered":"2023-10-19T12:45:40.359767+0000","last_unstale":"2023-10-19T12:45:41.351381+0000","last_undegraded":"2023-10-19T12:45:41.351381+0000","last_fullsized":"2023-10-19T12:45:41.351381+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:38:36.787271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.346878+0000","last_change":"2023-10-19T12:45:40.346878+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.346878+0000","last_undegraded":"2023-10-19T12:45:40.346878+0000","last_fullsized":"2023-10-19T12:45:40.346878+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":11,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.343223+0000","last_change":"2023-10-19T12:45:40.357695+0000","last_active":"2023-10-19T12:45:41.343223+0000","last_peered":"2023-10-19T12:45:41.343223+0000","last_clean":"2023-10-19T12:45:41.343223+0000","last_became_active":"2023-10-19T12:45:40.357393+0000","last_became_peered":"2023-10-19T12:45:40.357393+0000","last_unstale":"2023-10-19T12:45:41.343223+0000","last_undegraded":"2023-10-19T12:45:41.343223+0000","last_fullsized":"2023-10-19T12:45:41.343223+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:43:31.256604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.375615+0000","last_change":"2023-10-19T12:45:41.371258+0000","last_active":"2023-10-19T12:45:41.375615+0000","last_peered":"2023-10-19T12:45:41.375615+0000","last_clean":"2023-10-19T12:45:41.375615+0000","last_became_active":"2023-10-19T12:45:41.370963+0000","last_became_peered":"2023-10-19T12:45:41.370963+0000","last_unstale":"2023-10-19T12:45:41.375615+0000","last_undegraded":"2023-10-19T12:45:41.375615+0000","last_fullsized":"2023-10-19T12:45:41.375615+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:38:37.793736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-19T12:45:40.361905+0000","last_change":"2023-10-19T12:45:40.358263+0000","last_active":"2023-10-19T12:45:40.361905+0000","last_peered":"2023-10-19T12:45:40.361905+0000","last_clean":"2023-10-19T12:45:40.361905+0000","last_became_active":"2023-10-19T12:45:40.357402+0000","last_became_peered":"2023-10-19T12:45:40.357402+0000","last_unstale":"2023-10-19T12:45:40.361905+0000","last_undegraded":"2023-10-19T12:45:40.361905+0000","last_fullsized":"2023-10-19T12:45:40.361905+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:28:59.279648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":3,"reported_epoch":38,"state":"creating+activating","last_fresh":"2023-10-19T12:45:41.350527+0000","last_change":"2023-10-19T12:45:41.350527+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:41.350527+0000","last_undegraded":"2023-10-19T12:45:41.350527+0000","last_fullsized":"2023-10-19T12:45:41.350527+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:59:59.106505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":11,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.343650+0000","last_change":"2023-10-19T12:45:40.352686+0000","last_active":"2023-10-19T12:45:41.343650+0000","last_peered":"2023-10-19T12:45:41.343650+0000","last_clean":"2023-10-19T12:45:41.343650+0000","last_became_active":"2023-10-19T12:45:40.352420+0000","last_became_peered":"2023-10-19T12:45:40.352420+0000","last_unstale":"2023-10-19T12:45:41.343650+0000","last_undegraded":"2023-10-19T12:45:41.343650+0000","last_fullsized":"2023-10-19T12:45:41.343650+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:36:38.697134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.346413+0000","last_change":"2023-10-19T12:45:40.346413+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.346413+0000","last_undegraded":"2023-10-19T12:45:40.346413+0000","last_fullsized":"2023-10-19T12:45:40.346413+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":11,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.348286+0000","last_change":"2023-10-19T12:45:40.358578+0000","last_active":"2023-10-19T12:45:41.348286+0000","last_peered":"2023-10-19T12:45:41.348286+0000","last_clean":"2023-10-19T12:45:41.348286+0000","last_became_active":"2023-10-19T12:45:40.358057+0000","last_became_peered":"2023-10-19T12:45:40.358057+0000","last_unstale":"2023-10-19T12:45:41.348286+0000","last_undegraded":"2023-10-19T12:45:41.348286+0000","last_fullsized":"2023-10-19T12:45:41.348286+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:38:36.787271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.375444+0000","last_change":"2023-10-19T12:45:41.367997+0000","last_active":"2023-10-19T12:45:41.375444+0000","last_peered":"2023-10-19T12:45:41.375444+0000","last_clean":"2023-10-19T12:45:41.375444+0000","last_became_active":"2023-10-19T12:45:41.367687+0000","last_became_peered":"2023-10-19T12:45:41.367687+0000","last_unstale":"2023-10-19T12:45:41.375444+0000","last_undegraded":"2023-10-19T12:45:41.375444+0000","last_fullsized":"2023-10-19T12:45:41.375444+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:10:04.203620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":11,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.348125+0000","last_change":"2023-10-19T12:45:40.360184+0000","last_active":"2023-10-19T12:45:41.348125+0000","last_peered":"2023-10-19T12:45:41.348125+0000","last_clean":"2023-10-19T12:45:41.348125+0000","last_became_active":"2023-10-19T12:45:40.359917+0000","last_became_peered":"2023-10-19T12:45:40.359917+0000","last_unstale":"2023-10-19T12:45:41.348125+0000","last_undegraded":"2023-10-19T12:45:41.348125+0000","last_fullsized":"2023-10-19T12:45:41.348125+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:44:03.451018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.374901+0000","last_change":"2023-10-19T12:45:41.364206+0000","last_active":"2023-10-19T12:45:41.374901+0000","last_peered":"2023-10-19T12:45:41.374901+0000","last_clean":"2023-10-19T12:45:41.374901+0000","last_became_active":"2023-10-19T12:45:41.363926+0000","last_became_peered":"2023-10-19T12:45:41.363926+0000","last_unstale":"2023-10-19T12:45:41.374901+0000","last_undegraded":"2023-10-19T12:45:41.374901+0000","last_fullsized":"2023-10-19T12:45:41.374901+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:29:00.286113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-19T12:45:40.364717+0000","last_change":"2023-10-19T12:45:40.360715+0000","last_active":"2023-10-19T12:45:40.364717+0000","last_peered":"2023-10-19T12:45:40.364717+0000","last_clean":"2023-10-19T12:45:40.364717+0000","last_became_active":"2023-10-19T12:45:40.359801+0000","last_became_peered":"2023-10-19T12:45:40.359801+0000","last_unstale":"2023-10-19T12:45:40.364717+0000","last_undegraded":"2023-10-19T12:45:40.364717+0000","last_fullsized":"2023-10-19T12:45:40.364717+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:45:10.025508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.350603+0000","last_change":"2023-10-19T12:45:40.350603+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.350603+0000","last_undegraded":"2023-10-19T12:45:40.350603+0000","last_fullsized":"2023-10-19T12:45:40.350603+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-19T12:45:40.367503+0000","last_change":"2023-10-19T12:45:40.364278+0000","last_active":"2023-10-19T12:45:40.367503+0000","last_peered":"2023-10-19T12:45:40.367503+0000","last_clean":"2023-10-19T12:45:40.367503+0000","last_became_active":"2023-10-19T12:45:40.363981+0000","last_became_peered":"2023-10-19T12:45:40.363981+0000","last_unstale":"2023-10-19T12:45:40.367503+0000","last_undegraded":"2023-10-19T12:45:40.367503+0000","last_fullsized":"2023-10-19T12:45:40.367503+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:50:18.153095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":3,"reported_epoch":38,"state":"creating+activating","last_fresh":"2023-10-19T12:45:41.349666+0000","last_change":"2023-10-19T12:45:41.349666+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:41.349666+0000","last_undegraded":"2023-10-19T12:45:41.349666+0000","last_fullsized":"2023-10-19T12:45:41.349666+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:55:02.268260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-19T12:45:40.364011+0000","last_change":"2023-10-19T12:45:40.359438+0000","last_active":"2023-10-19T12:45:40.364011+0000","last_peered":"2023-10-19T12:45:40.364011+0000","last_clean":"2023-10-19T12:45:40.364011+0000","last_became_active":"2023-10-19T12:45:40.359156+0000","last_became_peered":"2023-10-19T12:45:40.359156+0000","last_unstale":"2023-10-19T12:45:40.364011+0000","last_undegraded":"2023-10-19T12:45:40.364011+0000","last_fullsized":"2023-10-19T12:45:40.364011+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:10:03.197155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.344906+0000","last_change":"2023-10-19T12:45:40.344906+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.344906+0000","last_undegraded":"2023-10-19T12:45:40.344906+0000","last_fullsized":"2023-10-19T12:45:40.344906+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":11,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.351976+0000","last_change":"2023-10-19T12:45:40.359310+0000","last_active":"2023-10-19T12:45:41.351976+0000","last_peered":"2023-10-19T12:45:41.351976+0000","last_clean":"2023-10-19T12:45:41.351976+0000","last_became_active":"2023-10-19T12:45:40.359010+0000","last_became_peered":"2023-10-19T12:45:40.359010+0000","last_unstale":"2023-10-19T12:45:41.351976+0000","last_undegraded":"2023-10-19T12:45:41.351976+0000","last_fullsized":"2023-10-19T12:45:41.351976+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:57:29.725854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":179,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-19T12:45:40.349832+0000","last_change":"2023-10-19T12:44:08.636613+0000","last_active":"2023-10-19T12:45:40.349832+0000","last_peered":"2023-10-19T12:45:40.349832+0000","last_clean":"2023-10-19T12:45:40.349832+0000","last_became_active":"2023-10-19T12:44:08.635697+0000","last_became_peered":"2023-10-19T12:44:08.635697+0000","last_unstale":"2023-10-19T12:45:40.349832+0000","last_undegraded":"2023-10-19T12:45:40.349832+0000","last_fullsized":"2023-10-19T12:45:40.349832+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_clean_scrub_stamp":"2023-10-19T12:44:07.399738+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:15:23.685485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":70,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574871,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28028,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343812,"statfs":{"total":96636764160,"available":96608063488,"internally_reserved":0,"allocated":1032192,"data_stored":777587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:45:08 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.53400000000000003}]},{"osd":1,"last update":"Thu Oct 19 12:45:08 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.495}]},{"osd":2,"last update":"Thu Oct 19 12:45:08 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.64300000000000002}]},{"osd":3,"last update":"Thu Oct 19 12:45:08 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.68200000000000005}]},{"osd":4,"last update":"Thu Oct 19 12:45:08 2023","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.51200000000000001}]},{"osd":5,"last update":"Thu Oct 19 12:45:08 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.441}]},{"osd":6,"last update":"Thu Oct 19 12:45:08 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.61199999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574871,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27436,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344404,"statfs":{"total":96636764160,"available":96608669696,"internally_reserved":0,"allocated":425984,"data_stored":180124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:45:06 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.56599999999999995}]},{"osd":1,"last update":"Thu Oct 19 12:45:06 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.626},{"interface":"front","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.075,"5min":1.075,"15min":1.075},"last":0.50600000000000001}]},{"osd":2,"last update":"Thu Oct 19 12:45:06 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.59199999999999997}]},{"osd":3,"last update":"Thu Oct 19 12:45:06 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.57999999999999996}]},{"osd":4,"last update":"Thu Oct 19 12:45:06 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.371},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.32900000000000001}]},{"osd":5,"last update":"Thu Oct 19 12:45:06 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.44700000000000001}]},{"osd":7,"last update":"Thu Oct 19 12:45:06 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.48499999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574871,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27436,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344404,"statfs":{"total":96636764160,"available":96608669696,"internally_reserved":0,"allocated":425984,"data_stored":180124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.55000000000000004}]},{"osd":1,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.57999999999999996}]},{"osd":2,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.625},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.60899999999999999}]},{"osd":3,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.65400000000000003}]},{"osd":4,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.42599999999999999}]},{"osd":6,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.59199999999999997}]},{"osd":7,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.52900000000000003}]}]},{"osd":4,"up_from":13,"seq":55834574871,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27436,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344404,"statfs":{"total":96636764160,"available":96608669696,"internally_reserved":0,"allocated":425984,"data_stored":180124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:45:08 2023","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.46000000000000002}]},{"osd":1,"last update":"Thu Oct 19 12:45:08 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.57699999999999996}]},{"osd":2,"last update":"Thu Oct 19 12:45:08 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.44600000000000001}]},{"osd":3,"last update":"Thu Oct 19 12:45:08 2023","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.629}]},{"osd":5,"last update":"Thu Oct 19 12:45:08 2023","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.29099999999999998}]},{"osd":6,"last update":"Thu Oct 19 12:45:08 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.32600000000000001}]},{"osd":7,"last update":"Thu Oct 19 12:45:08 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.373},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.40999999999999998}]}]},{"osd":3,"up_from":13,"seq":55834574870,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31596,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340244,"statfs":{"total":96636764160,"available":96604409856,"internally_reserved":0,"allocated":425984,"data_stored":180124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":1.04,"5min":1.04,"15min":1.04},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.375,"5min":0.375,"15min":0.375},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"last":0.32500000000000001}]},{"osd":1,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.371},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.39700000000000002}]},{"osd":2,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.42699999999999999}]},{"osd":4,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.65600000000000003}]},{"osd":5,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.53400000000000003}]},{"osd":6,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.51700000000000002}]},{"osd":7,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.61899999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574872,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31596,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340244,"statfs":{"total":96636764160,"available":96604409856,"internally_reserved":0,"allocated":425984,"data_stored":180124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.314},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"last":0.251}]},{"osd":1,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.35399999999999998}]},{"osd":3,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.46800000000000003}]},{"osd":4,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.503},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.53800000000000003}]},{"osd":5,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.52500000000000002}]},{"osd":6,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.58899999999999997}]},{"osd":7,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.111,"5min":1.111,"15min":1.111},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.018,"5min":1.018,"15min":1.018},"last":0.55500000000000005}]}]},{"osd":1,"up_from":13,"seq":55834574871,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31596,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340244,"statfs":{"total":96636764160,"available":96604409856,"internally_reserved":0,"allocated":425984,"data_stored":180124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:45:05 2023","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.29699999999999999}]},{"osd":2,"last update":"Thu Oct 19 12:45:05 2023","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.441},{"interface":"front","average":{"1min":0.433,"5min":0.433,"15min":0.433},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.38}]},{"osd":3,"last update":"Thu Oct 19 12:45:05 2023","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.36399999999999999}]},{"osd":4,"last update":"Thu Oct 19 12:45:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.1910000000000001,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.314}]},{"osd":5,"last update":"Thu Oct 19 12:45:05 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.117,"5min":1.117,"15min":1.117},"last":0.41599999999999998}]},{"osd":6,"last update":"Thu Oct 19 12:45:05 2023","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.077,"5min":1.077,"15min":1.077},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.51400000000000001}]},{"osd":7,"last update":"Thu Oct 19 12:45:05 2023","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.45700000000000002}]}]},{"osd":0,"up_from":13,"seq":55834574872,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32176,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339664,"statfs":{"total":96636764160,"available":96603815936,"internally_reserved":0,"allocated":1015808,"data_stored":769873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Oct 19 12:45:09 2023","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"last":0.246},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"last":0.32800000000000001}]},{"osd":2,"last update":"Thu Oct 19 12:45:09 2023","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.29999999999999999}]},{"osd":3,"last update":"Thu Oct 19 12:45:09 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"last":0.311},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.40999999999999998}]},{"osd":4,"last update":"Thu Oct 19 12:45:09 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.39800000000000002}]},{"osd":5,"last update":"Thu Oct 19 12:45:09 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.056,"5min":1.056,"15min":1.056},"last":0.49399999999999999}]},{"osd":6,"last update":"Thu Oct 19 12:45:09 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.109,"5min":1.109,"15min":1.109},"last":0.42099999999999999}]},{"osd":7,"last update":"Thu Oct 19 12:45:09 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.44500000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-19T12:45:45.106 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3f is not active+clean 2023-10-19T12:45:45.106 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-10-19T12:45:40.347634+0000', 'last_change': '2023-10-19T12:45:40.347634+0000', 'last_active': '2023-10-19T12:45:40.327526+0000', 'last_peered': '2023-10-19T12:45:40.327526+0000', 'last_clean': '2023-10-19T12:45:40.327526+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T12:45:40.347634+0000', 'last_undegraded': '2023-10-19T12:45:40.347634+0000', 'last_fullsized': '2023-10-19T12:45:40.347634+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_clean_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-10-19T12:45:45.106 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3e is not active+clean 2023-10-19T12:45:45.107 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-10-19T12:45:40.349030+0000', 'last_change': '2023-10-19T12:45:40.349030+0000', 'last_active': '2023-10-19T12:45:40.327526+0000', 'last_peered': '2023-10-19T12:45:40.327526+0000', 'last_clean': '2023-10-19T12:45:40.327526+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T12:45:40.349030+0000', 'last_undegraded': '2023-10-19T12:45:40.349030+0000', 'last_fullsized': '2023-10-19T12:45:40.349030+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_clean_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-10-19T12:45:45.107 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.39 is not active+clean 2023-10-19T12:45:45.108 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.39', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 38, 'state': 'creating+activating', 'last_fresh': '2023-10-19T12:45:41.350760+0000', 'last_change': '2023-10-19T12:45:41.350760+0000', 'last_active': '2023-10-19T12:45:40.327526+0000', 'last_peered': '2023-10-19T12:45:40.327526+0000', 'last_clean': '2023-10-19T12:45:40.327526+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T12:45:41.350760+0000', 'last_undegraded': '2023-10-19T12:45:41.350760+0000', 'last_fullsized': '2023-10-19T12:45:41.350760+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_clean_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-20T13:38:37.793736+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': ['5', '3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-10-19T12:45:45.108 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.38 is not active+clean 2023-10-19T12:45:45.108 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.38', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-10-19T12:45:40.349543+0000', 'last_change': '2023-10-19T12:45:40.349543+0000', 'last_active': '2023-10-19T12:45:40.327526+0000', 'last_peered': '2023-10-19T12:45:40.327526+0000', 'last_clean': '2023-10-19T12:45:40.327526+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T12:45:40.349543+0000', 'last_undegraded': '2023-10-19T12:45:40.349543+0000', 'last_fullsized': '2023-10-19T12:45:40.349543+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_clean_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-10-19T12:45:45.109 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.36 is not active+clean 2023-10-19T12:45:45.109 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.36', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-10-19T12:45:40.345142+0000', 'last_change': '2023-10-19T12:45:40.345142+0000', 'last_active': '2023-10-19T12:45:40.327526+0000', 'last_peered': '2023-10-19T12:45:40.327526+0000', 'last_clean': '2023-10-19T12:45:40.327526+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T12:45:40.345142+0000', 'last_undegraded': '2023-10-19T12:45:40.345142+0000', 'last_fullsized': '2023-10-19T12:45:40.345142+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_clean_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-10-19T12:45:45.110 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.35 is not active+clean 2023-10-19T12:45:45.110 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.35', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-10-19T12:45:40.346201+0000', 'last_change': '2023-10-19T12:45:40.346201+0000', 'last_active': '2023-10-19T12:45:40.327526+0000', 'last_peered': '2023-10-19T12:45:40.327526+0000', 'last_clean': '2023-10-19T12:45:40.327526+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T12:45:40.346201+0000', 'last_undegraded': '2023-10-19T12:45:40.346201+0000', 'last_fullsized': '2023-10-19T12:45:40.346201+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_clean_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-10-19T12:45:45.110 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.33 is not active+clean 2023-10-19T12:45:45.111 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.33', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-10-19T12:45:40.345686+0000', 'last_change': '2023-10-19T12:45:40.345686+0000', 'last_active': '2023-10-19T12:45:40.327526+0000', 'last_peered': '2023-10-19T12:45:40.327526+0000', 'last_clean': '2023-10-19T12:45:40.327526+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T12:45:40.345686+0000', 'last_undegraded': '2023-10-19T12:45:40.345686+0000', 'last_fullsized': '2023-10-19T12:45:40.345686+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_clean_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-10-19T12:45:45.111 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.32 is not active+clean 2023-10-19T12:45:45.112 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.32', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-10-19T12:45:40.351066+0000', 'last_change': '2023-10-19T12:45:40.351066+0000', 'last_active': '2023-10-19T12:45:40.327526+0000', 'last_peered': '2023-10-19T12:45:40.327526+0000', 'last_clean': '2023-10-19T12:45:40.327526+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T12:45:40.351066+0000', 'last_undegraded': '2023-10-19T12:45:40.351066+0000', 'last_fullsized': '2023-10-19T12:45:40.351066+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_clean_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-10-19T12:45:45.112 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.31 is not active+clean 2023-10-19T12:45:45.112 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.31', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 38, 'state': 'creating+activating', 'last_fresh': '2023-10-19T12:45:41.350355+0000', 'last_change': '2023-10-19T12:45:41.350355+0000', 'last_active': '2023-10-19T12:45:40.327526+0000', 'last_peered': '2023-10-19T12:45:40.327526+0000', 'last_clean': '2023-10-19T12:45:40.327526+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T12:45:41.350355+0000', 'last_undegraded': '2023-10-19T12:45:41.350355+0000', 'last_fullsized': '2023-10-19T12:45:41.350355+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_clean_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-20T21:10:04.203620+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': ['5', '2'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-10-19T12:45:45.113 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.30 is not active+clean 2023-10-19T12:45:45.113 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.30', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 38, 'state': 'creating+activating', 'last_fresh': '2023-10-19T12:45:41.352539+0000', 'last_change': '2023-10-19T12:45:41.352539+0000', 'last_active': '2023-10-19T12:45:40.327526+0000', 'last_peered': '2023-10-19T12:45:40.327526+0000', 'last_clean': '2023-10-19T12:45:40.327526+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T12:45:41.352539+0000', 'last_undegraded': '2023-10-19T12:45:41.352539+0000', 'last_fullsized': '2023-10-19T12:45:41.352539+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_clean_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-20T23:21:51.336716+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': ['7', '5'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-10-19T12:45:45.113 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2f is not active+clean 2023-10-19T12:45:45.114 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2f', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 38, 'state': 'creating+activating', 'last_fresh': '2023-10-19T12:45:41.351475+0000', 'last_change': '2023-10-19T12:45:41.351475+0000', 'last_active': '2023-10-19T12:45:40.327526+0000', 'last_peered': '2023-10-19T12:45:40.327526+0000', 'last_clean': '2023-10-19T12:45:40.327526+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T12:45:41.351475+0000', 'last_undegraded': '2023-10-19T12:45:41.351475+0000', 'last_fullsized': '2023-10-19T12:45:41.351475+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_clean_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-21T00:32:37.014614+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': ['7', '5'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-10-19T12:45:45.114 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2e is not active+clean 2023-10-19T12:45:45.115 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2e', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 38, 'state': 'creating+activating', 'last_fresh': '2023-10-19T12:45:41.348402+0000', 'last_change': '2023-10-19T12:45:41.348402+0000', 'last_active': '2023-10-19T12:45:40.327526+0000', 'last_peered': '2023-10-19T12:45:40.327526+0000', 'last_clean': '2023-10-19T12:45:40.327526+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T12:45:41.348402+0000', 'last_undegraded': '2023-10-19T12:45:41.348402+0000', 'last_fullsized': '2023-10-19T12:45:41.348402+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_clean_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-20T14:53:30.629136+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': ['6', '4'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-10-19T12:45:45.115 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2c is not active+clean 2023-10-19T12:45:45.115 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2c', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 38, 'state': 'creating+activating', 'last_fresh': '2023-10-19T12:45:41.351396+0000', 'last_change': '2023-10-19T12:45:41.351396+0000', 'last_active': '2023-10-19T12:45:40.327526+0000', 'last_peered': '2023-10-19T12:45:40.327526+0000', 'last_clean': '2023-10-19T12:45:40.327526+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T12:45:41.351396+0000', 'last_undegraded': '2023-10-19T12:45:41.351396+0000', 'last_fullsized': '2023-10-19T12:45:41.351396+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_clean_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-20T18:48:25.218615+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': ['6', '4'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-10-19T12:45:45.116 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2b is not active+clean 2023-10-19T12:45:45.116 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-10-19T12:45:40.344680+0000', 'last_change': '2023-10-19T12:45:40.344680+0000', 'last_active': '2023-10-19T12:45:40.327526+0000', 'last_peered': '2023-10-19T12:45:40.327526+0000', 'last_clean': '2023-10-19T12:45:40.327526+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T12:45:40.344680+0000', 'last_undegraded': '2023-10-19T12:45:40.344680+0000', 'last_fullsized': '2023-10-19T12:45:40.344680+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_clean_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-10-19T12:45:45.117 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2a is not active+clean 2023-10-19T12:45:45.117 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-10-19T12:45:40.348310+0000', 'last_change': '2023-10-19T12:45:40.348310+0000', 'last_active': '2023-10-19T12:45:40.327526+0000', 'last_peered': '2023-10-19T12:45:40.327526+0000', 'last_clean': '2023-10-19T12:45:40.327526+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T12:45:40.348310+0000', 'last_undegraded': '2023-10-19T12:45:40.348310+0000', 'last_fullsized': '2023-10-19T12:45:40.348310+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_clean_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-10-19T12:45:45.117 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.29 is not active+clean 2023-10-19T12:45:45.118 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.29', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 38, 'state': 'creating+activating', 'last_fresh': '2023-10-19T12:45:41.350788+0000', 'last_change': '2023-10-19T12:45:41.350788+0000', 'last_active': '2023-10-19T12:45:40.327526+0000', 'last_peered': '2023-10-19T12:45:40.327526+0000', 'last_clean': '2023-10-19T12:45:40.327526+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T12:45:41.350788+0000', 'last_undegraded': '2023-10-19T12:45:41.350788+0000', 'last_fullsized': '2023-10-19T12:45:41.350788+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_clean_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-20T20:44:04.457483+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': ['7', '2'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-10-19T12:45:45.118 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.27 is not active+clean 2023-10-19T12:45:45.119 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.27', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 38, 'state': 'creating+activating', 'last_fresh': '2023-10-19T12:45:41.350918+0000', 'last_change': '2023-10-19T12:45:41.350918+0000', 'last_active': '2023-10-19T12:45:40.327526+0000', 'last_peered': '2023-10-19T12:45:40.327526+0000', 'last_clean': '2023-10-19T12:45:40.327526+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T12:45:41.350918+0000', 'last_undegraded': '2023-10-19T12:45:41.350918+0000', 'last_fullsized': '2023-10-19T12:45:41.350918+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_clean_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-20T15:49:21.932303+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': ['6', '0'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-10-19T12:45:45.119 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.25 is not active+clean 2023-10-19T12:45:45.119 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.25', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 38, 'state': 'creating+activating', 'last_fresh': '2023-10-19T12:45:41.348601+0000', 'last_change': '2023-10-19T12:45:41.348601+0000', 'last_active': '2023-10-19T12:45:40.327526+0000', 'last_peered': '2023-10-19T12:45:40.327526+0000', 'last_clean': '2023-10-19T12:45:40.327526+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T12:45:41.348601+0000', 'last_undegraded': '2023-10-19T12:45:41.348601+0000', 'last_fullsized': '2023-10-19T12:45:41.348601+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_clean_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-20T22:36:07.043432+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': ['6', '2'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-10-19T12:45:45.120 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.24 is not active+clean 2023-10-19T12:45:45.120 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.24', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-10-19T12:45:40.345780+0000', 'last_change': '2023-10-19T12:45:40.345780+0000', 'last_active': '2023-10-19T12:45:40.327526+0000', 'last_peered': '2023-10-19T12:45:40.327526+0000', 'last_clean': '2023-10-19T12:45:40.327526+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T12:45:40.345780+0000', 'last_undegraded': '2023-10-19T12:45:40.345780+0000', 'last_fullsized': '2023-10-19T12:45:40.345780+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_clean_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-10-19T12:45:45.121 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.23 is not active+clean 2023-10-19T12:45:45.121 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.23', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 38, 'state': 'creating+activating', 'last_fresh': '2023-10-19T12:45:41.351166+0000', 'last_change': '2023-10-19T12:45:41.351166+0000', 'last_active': '2023-10-19T12:45:40.327526+0000', 'last_peered': '2023-10-19T12:45:40.327526+0000', 'last_clean': '2023-10-19T12:45:40.327526+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T12:45:41.351166+0000', 'last_undegraded': '2023-10-19T12:45:41.351166+0000', 'last_fullsized': '2023-10-19T12:45:41.351166+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_clean_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-20T14:16:37.419534+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': ['6', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-10-19T12:45:45.122 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.22 is not active+clean 2023-10-19T12:45:45.122 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.22', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 38, 'state': 'creating+activating', 'last_fresh': '2023-10-19T12:45:41.348399+0000', 'last_change': '2023-10-19T12:45:41.348399+0000', 'last_active': '2023-10-19T12:45:40.327526+0000', 'last_peered': '2023-10-19T12:45:40.327526+0000', 'last_clean': '2023-10-19T12:45:40.327526+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T12:45:41.348399+0000', 'last_undegraded': '2023-10-19T12:45:41.348399+0000', 'last_fullsized': '2023-10-19T12:45:41.348399+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_clean_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-20T12:45:47.935169+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': ['6', '3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-10-19T12:45:45.123 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.21 is not active+clean 2023-10-19T12:45:45.123 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.21', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 38, 'state': 'creating+activating', 'last_fresh': '2023-10-19T12:45:41.351136+0000', 'last_change': '2023-10-19T12:45:41.351136+0000', 'last_active': '2023-10-19T12:45:40.327526+0000', 'last_peered': '2023-10-19T12:45:40.327526+0000', 'last_clean': '2023-10-19T12:45:40.327526+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T12:45:41.351136+0000', 'last_undegraded': '2023-10-19T12:45:41.351136+0000', 'last_fullsized': '2023-10-19T12:45:41.351136+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_clean_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-20T21:10:04.203620+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': ['7', '2'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-10-19T12:45:45.123 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1c is not active+clean 2023-10-19T12:45:45.124 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1c', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 38, 'state': 'creating+activating', 'last_fresh': '2023-10-19T12:45:41.351973+0000', 'last_change': '2023-10-19T12:45:41.351973+0000', 'last_active': '2023-10-19T12:45:40.327526+0000', 'last_peered': '2023-10-19T12:45:40.327526+0000', 'last_clean': '2023-10-19T12:45:40.327526+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T12:45:41.351973+0000', 'last_undegraded': '2023-10-19T12:45:41.351973+0000', 'last_fullsized': '2023-10-19T12:45:41.351973+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_clean_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-20T13:53:29.708835+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': ['7', '5'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-10-19T12:45:45.124 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1e is not active+clean 2023-10-19T12:45:45.124 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-10-19T12:45:40.348927+0000', 'last_change': '2023-10-19T12:45:40.348927+0000', 'last_active': '2023-10-19T12:45:40.327526+0000', 'last_peered': '2023-10-19T12:45:40.327526+0000', 'last_clean': '2023-10-19T12:45:40.327526+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T12:45:40.348927+0000', 'last_undegraded': '2023-10-19T12:45:40.348927+0000', 'last_fullsized': '2023-10-19T12:45:40.348927+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_clean_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-10-19T12:45:45.125 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1f is not active+clean 2023-10-19T12:45:45.125 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-10-19T12:45:40.349506+0000', 'last_change': '2023-10-19T12:45:40.349506+0000', 'last_active': '2023-10-19T12:45:40.327526+0000', 'last_peered': '2023-10-19T12:45:40.327526+0000', 'last_clean': '2023-10-19T12:45:40.327526+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T12:45:40.349506+0000', 'last_undegraded': '2023-10-19T12:45:40.349506+0000', 'last_fullsized': '2023-10-19T12:45:40.349506+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_clean_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-10-19T12:45:45.126 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.19 is not active+clean 2023-10-19T12:45:45.126 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.19', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 38, 'state': 'creating+activating', 'last_fresh': '2023-10-19T12:45:41.349960+0000', 'last_change': '2023-10-19T12:45:41.349960+0000', 'last_active': '2023-10-19T12:45:40.327526+0000', 'last_peered': '2023-10-19T12:45:40.327526+0000', 'last_clean': '2023-10-19T12:45:40.327526+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T12:45:41.349960+0000', 'last_undegraded': '2023-10-19T12:45:41.349960+0000', 'last_fullsized': '2023-10-19T12:45:41.349960+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_clean_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-20T20:44:04.457483+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': ['5', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-10-19T12:45:45.126 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1a is not active+clean 2023-10-19T12:45:45.127 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-10-19T12:45:40.350186+0000', 'last_change': '2023-10-19T12:45:40.350186+0000', 'last_active': '2023-10-19T12:45:40.327526+0000', 'last_peered': '2023-10-19T12:45:40.327526+0000', 'last_clean': '2023-10-19T12:45:40.327526+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T12:45:40.350186+0000', 'last_undegraded': '2023-10-19T12:45:40.350186+0000', 'last_fullsized': '2023-10-19T12:45:40.350186+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_clean_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-10-19T12:45:45.127 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1b is not active+clean 2023-10-19T12:45:45.127 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-10-19T12:45:40.351154+0000', 'last_change': '2023-10-19T12:45:40.351154+0000', 'last_active': '2023-10-19T12:45:40.327526+0000', 'last_peered': '2023-10-19T12:45:40.327526+0000', 'last_clean': '2023-10-19T12:45:40.327526+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T12:45:40.351154+0000', 'last_undegraded': '2023-10-19T12:45:40.351154+0000', 'last_fullsized': '2023-10-19T12:45:40.351154+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_clean_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-10-19T12:45:45.128 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.15 is not active+clean 2023-10-19T12:45:45.128 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.15', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 38, 'state': 'creating+activating', 'last_fresh': '2023-10-19T12:45:41.350085+0000', 'last_change': '2023-10-19T12:45:41.350085+0000', 'last_active': '2023-10-19T12:45:40.327526+0000', 'last_peered': '2023-10-19T12:45:40.327526+0000', 'last_clean': '2023-10-19T12:45:40.327526+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T12:45:41.350085+0000', 'last_undegraded': '2023-10-19T12:45:41.350085+0000', 'last_fullsized': '2023-10-19T12:45:41.350085+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_clean_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-20T21:29:00.286113+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': ['7', '6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-10-19T12:45:45.128 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.17 is not active+clean 2023-10-19T12:45:45.129 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-10-19T12:45:40.348513+0000', 'last_change': '2023-10-19T12:45:40.348513+0000', 'last_active': '2023-10-19T12:45:40.327526+0000', 'last_peered': '2023-10-19T12:45:40.327526+0000', 'last_clean': '2023-10-19T12:45:40.327526+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T12:45:40.348513+0000', 'last_undegraded': '2023-10-19T12:45:40.348513+0000', 'last_fullsized': '2023-10-19T12:45:40.348513+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_clean_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-10-19T12:45:45.129 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.11 is not active+clean 2023-10-19T12:45:45.129 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-10-19T12:45:40.347231+0000', 'last_change': '2023-10-19T12:45:40.347231+0000', 'last_active': '2023-10-19T12:45:40.327526+0000', 'last_peered': '2023-10-19T12:45:40.327526+0000', 'last_clean': '2023-10-19T12:45:40.327526+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T12:45:40.347231+0000', 'last_undegraded': '2023-10-19T12:45:40.347231+0000', 'last_fullsized': '2023-10-19T12:45:40.347231+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_clean_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-10-19T12:45:45.130 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.12 is not active+clean 2023-10-19T12:45:45.130 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.12', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 38, 'state': 'creating+activating', 'last_fresh': '2023-10-19T12:45:41.349119+0000', 'last_change': '2023-10-19T12:45:41.349119+0000', 'last_active': '2023-10-19T12:45:40.327526+0000', 'last_peered': '2023-10-19T12:45:40.327526+0000', 'last_clean': '2023-10-19T12:45:40.327526+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T12:45:41.349119+0000', 'last_undegraded': '2023-10-19T12:45:41.349119+0000', 'last_fullsized': '2023-10-19T12:45:41.349119+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_clean_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-20T14:16:37.419534+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': ['7', '2'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-10-19T12:45:45.131 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.13 is not active+clean 2023-10-19T12:45:45.131 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-10-19T12:45:40.350379+0000', 'last_change': '2023-10-19T12:45:40.350379+0000', 'last_active': '2023-10-19T12:45:40.327526+0000', 'last_peered': '2023-10-19T12:45:40.327526+0000', 'last_clean': '2023-10-19T12:45:40.327526+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T12:45:40.350379+0000', 'last_undegraded': '2023-10-19T12:45:40.350379+0000', 'last_fullsized': '2023-10-19T12:45:40.350379+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_clean_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-10-19T12:45:45.131 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.c is not active+clean 2023-10-19T12:45:45.132 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-10-19T12:45:40.350100+0000', 'last_change': '2023-10-19T12:45:40.350100+0000', 'last_active': '2023-10-19T12:45:40.327526+0000', 'last_peered': '2023-10-19T12:45:40.327526+0000', 'last_clean': '2023-10-19T12:45:40.327526+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T12:45:40.350100+0000', 'last_undegraded': '2023-10-19T12:45:40.350100+0000', 'last_fullsized': '2023-10-19T12:45:40.350100+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_clean_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-10-19T12:45:45.132 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.d is not active+clean 2023-10-19T12:45:45.132 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-10-19T12:45:40.345759+0000', 'last_change': '2023-10-19T12:45:40.345759+0000', 'last_active': '2023-10-19T12:45:40.327526+0000', 'last_peered': '2023-10-19T12:45:40.327526+0000', 'last_clean': '2023-10-19T12:45:40.327526+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T12:45:40.345759+0000', 'last_undegraded': '2023-10-19T12:45:40.345759+0000', 'last_fullsized': '2023-10-19T12:45:40.345759+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_clean_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-10-19T12:45:45.133 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.e is not active+clean 2023-10-19T12:45:45.133 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-10-19T12:45:40.350198+0000', 'last_change': '2023-10-19T12:45:40.350198+0000', 'last_active': '2023-10-19T12:45:40.327526+0000', 'last_peered': '2023-10-19T12:45:40.327526+0000', 'last_clean': '2023-10-19T12:45:40.327526+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T12:45:40.350198+0000', 'last_undegraded': '2023-10-19T12:45:40.350198+0000', 'last_fullsized': '2023-10-19T12:45:40.350198+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_clean_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-10-19T12:45:45.133 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.8 is not active+clean 2023-10-19T12:45:45.134 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.8', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 38, 'state': 'creating+activating', 'last_fresh': '2023-10-19T12:45:41.353064+0000', 'last_change': '2023-10-19T12:45:41.353064+0000', 'last_active': '2023-10-19T12:45:40.327526+0000', 'last_peered': '2023-10-19T12:45:40.327526+0000', 'last_clean': '2023-10-19T12:45:40.327526+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T12:45:41.353064+0000', 'last_undegraded': '2023-10-19T12:45:41.353064+0000', 'last_fullsized': '2023-10-19T12:45:41.353064+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_clean_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-20T16:59:55.870354+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': ['7', '2'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-10-19T12:45:45.134 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.a is not active+clean 2023-10-19T12:45:45.135 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.a', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 38, 'state': 'creating+activating', 'last_fresh': '2023-10-19T12:45:41.349483+0000', 'last_change': '2023-10-19T12:45:41.349483+0000', 'last_active': '2023-10-19T12:45:40.327526+0000', 'last_peered': '2023-10-19T12:45:40.327526+0000', 'last_clean': '2023-10-19T12:45:40.327526+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T12:45:41.349483+0000', 'last_undegraded': '2023-10-19T12:45:41.349483+0000', 'last_fullsized': '2023-10-19T12:45:41.349483+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_clean_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-20T23:59:59.106505+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': ['5', '6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-10-19T12:45:45.135 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.b is not active+clean 2023-10-19T12:45:45.135 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-10-19T12:45:40.346878+0000', 'last_change': '2023-10-19T12:45:40.346878+0000', 'last_active': '2023-10-19T12:45:40.327526+0000', 'last_peered': '2023-10-19T12:45:40.327526+0000', 'last_clean': '2023-10-19T12:45:40.327526+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T12:45:40.346878+0000', 'last_undegraded': '2023-10-19T12:45:40.346878+0000', 'last_fullsized': '2023-10-19T12:45:40.346878+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_clean_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-10-19T12:45:45.136 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.5 is not active+clean 2023-10-19T12:45:45.136 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.5', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 38, 'state': 'creating+activating', 'last_fresh': '2023-10-19T12:45:41.350527+0000', 'last_change': '2023-10-19T12:45:41.350527+0000', 'last_active': '2023-10-19T12:45:40.327526+0000', 'last_peered': '2023-10-19T12:45:40.327526+0000', 'last_clean': '2023-10-19T12:45:40.327526+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T12:45:41.350527+0000', 'last_undegraded': '2023-10-19T12:45:41.350527+0000', 'last_fullsized': '2023-10-19T12:45:41.350527+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_clean_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-20T23:59:59.106505+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': ['7', '6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-10-19T12:45:45.136 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.6 is not active+clean 2023-10-19T12:45:45.137 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-10-19T12:45:40.346413+0000', 'last_change': '2023-10-19T12:45:40.346413+0000', 'last_active': '2023-10-19T12:45:40.327526+0000', 'last_peered': '2023-10-19T12:45:40.327526+0000', 'last_clean': '2023-10-19T12:45:40.327526+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T12:45:40.346413+0000', 'last_undegraded': '2023-10-19T12:45:40.346413+0000', 'last_fullsized': '2023-10-19T12:45:40.346413+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_clean_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-10-19T12:45:45.137 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2 is not active+clean 2023-10-19T12:45:45.137 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-10-19T12:45:40.350603+0000', 'last_change': '2023-10-19T12:45:40.350603+0000', 'last_active': '2023-10-19T12:45:40.327526+0000', 'last_peered': '2023-10-19T12:45:40.327526+0000', 'last_clean': '2023-10-19T12:45:40.327526+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T12:45:40.350603+0000', 'last_undegraded': '2023-10-19T12:45:40.350603+0000', 'last_fullsized': '2023-10-19T12:45:40.350603+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_clean_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-10-19T12:45:45.138 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3 is not active+clean 2023-10-19T12:45:45.138 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 38, 'state': 'creating+activating', 'last_fresh': '2023-10-19T12:45:41.349666+0000', 'last_change': '2023-10-19T12:45:41.349666+0000', 'last_active': '2023-10-19T12:45:40.327526+0000', 'last_peered': '2023-10-19T12:45:40.327526+0000', 'last_clean': '2023-10-19T12:45:40.327526+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T12:45:41.349666+0000', 'last_undegraded': '2023-10-19T12:45:41.349666+0000', 'last_fullsized': '2023-10-19T12:45:41.349666+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_clean_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-20T13:55:02.268260+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': ['7', '6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-10-19T12:45:45.139 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.7 is not active+clean 2023-10-19T12:45:45.139 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2023-10-19T12:45:40.344906+0000', 'last_change': '2023-10-19T12:45:40.344906+0000', 'last_active': '2023-10-19T12:45:40.327526+0000', 'last_peered': '2023-10-19T12:45:40.327526+0000', 'last_clean': '2023-10-19T12:45:40.327526+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T12:45:40.344906+0000', 'last_undegraded': '2023-10-19T12:45:40.344906+0000', 'last_fullsized': '2023-10-19T12:45:40.344906+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'last_clean_scrub_stamp': '2023-10-19T12:45:40.327526+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-10-19T12:45:45.140 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2023-10-19T12:45:45.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.245+0000 7fec47ebb640 1 -- 172.21.15.57:0/210468219 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fec4006dce0 msgr2=0x7fec400abe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:45.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.245+0000 7fec47ebb640 1 --2- 172.21.15.57:0/210468219 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fec4006dce0 0x7fec400abe90 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fec3c005fd0 tx=0x7fec3c02c170 comp rx=0 tx=0).stop 2023-10-19T12:45:45.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.245+0000 7fec47ebb640 1 -- 172.21.15.57:0/210468219 shutdown_connections 2023-10-19T12:45:45.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.245+0000 7fec47ebb640 1 --2- 172.21.15.57:0/210468219 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fec4006dce0 0x7fec400abe90 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:45.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.245+0000 7fec47ebb640 1 --2- 172.21.15.57:0/210468219 >> v2:172.21.15.179:3300/0 conn(0x7fec4006bf20 0x7fec400ab950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:45.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.245+0000 7fec47ebb640 1 -- 172.21.15.57:0/210468219 >> 172.21.15.57:0/210468219 conn(0x7fec400b0670 msgr2=0x7fec400b0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:45.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.246+0000 7fec47ebb640 1 -- 172.21.15.57:0/210468219 shutdown_connections 2023-10-19T12:45:45.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.246+0000 7fec47ebb640 1 -- 172.21.15.57:0/210468219 wait complete. 2023-10-19T12:45:45.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.247+0000 7fec47ebb640 1 Processor -- start 2023-10-19T12:45:45.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.247+0000 7fec47ebb640 1 -- start start 2023-10-19T12:45:45.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.247+0000 7fec47ebb640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec4006bf20 0x7fec400c51d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:45.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.247+0000 7fec47ebb640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fec4006dce0 0x7fec40156060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:45.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.248+0000 7fec47ebb640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fec400cd0c0 0x7fec4009c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:45.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.248+0000 7fec47ebb640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec400c5710 con 0x7fec4006bf20 2023-10-19T12:45:45.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.248+0000 7fec47ebb640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec400c4d40 con 0x7fec400cd0c0 2023-10-19T12:45:45.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.248+0000 7fec47ebb640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec400c4130 con 0x7fec4006dce0 2023-10-19T12:45:45.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.248+0000 7fec45c57640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec4006bf20 0x7fec400c51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:45.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.248+0000 7fec45c57640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec4006bf20 0x7fec400c51d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59384/0 (socket says 172.21.15.57:59384) 2023-10-19T12:45:45.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.248+0000 7fec45c57640 1 -- 172.21.15.57:0/3294572670 learned_addr learned my addr 172.21.15.57:0/3294572670 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:45.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.248+0000 7fec45456640 1 --2- 172.21.15.57:0/3294572670 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fec4006dce0 0x7fec40156060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:45.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.248+0000 7fec46458640 1 --2- 172.21.15.57:0/3294572670 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fec400cd0c0 0x7fec4009c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:45.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.248+0000 7fec45c57640 1 -- 172.21.15.57:0/3294572670 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fec4006dce0 msgr2=0x7fec40156060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:45.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.248+0000 7fec45c57640 1 --2- 172.21.15.57:0/3294572670 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fec4006dce0 0x7fec40156060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:45.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.248+0000 7fec45c57640 1 -- 172.21.15.57:0/3294572670 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fec400cd0c0 msgr2=0x7fec4009c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:45.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.248+0000 7fec45c57640 1 --2- 172.21.15.57:0/3294572670 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fec400cd0c0 0x7fec4009c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:45.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.248+0000 7fec45c57640 1 -- 172.21.15.57:0/3294572670 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec30009490 con 0x7fec4006bf20 2023-10-19T12:45:45.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.248+0000 7fec45c57640 1 --2- 172.21.15.57:0/3294572670 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec4006bf20 0x7fec400c51d0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fec3800c460 tx=0x7fec3800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:45.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.248+0000 7fec367fc640 1 -- 172.21.15.57:0/3294572670 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec38004230 con 0x7fec4006bf20 2023-10-19T12:45:45.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.248+0000 7fec367fc640 1 -- 172.21.15.57:0/3294572670 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fec380044a0 con 0x7fec4006bf20 2023-10-19T12:45:45.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.248+0000 7fec47ebb640 1 -- 172.21.15.57:0/3294572670 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec3c009840 con 0x7fec4006bf20 2023-10-19T12:45:45.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.248+0000 7fec367fc640 1 -- 172.21.15.57:0/3294572670 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec38012890 con 0x7fec4006bf20 2023-10-19T12:45:45.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.248+0000 7fec47ebb640 1 -- 172.21.15.57:0/3294572670 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec400c46f0 con 0x7fec4006bf20 2023-10-19T12:45:45.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.249+0000 7fec47ebb640 1 -- 172.21.15.57:0/3294572670 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec400690e0 con 0x7fec4006bf20 2023-10-19T12:45:45.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.250+0000 7fec367fc640 1 -- 172.21.15.57:0/3294572670 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fec380035f0 con 0x7fec4006bf20 2023-10-19T12:45:45.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.250+0000 7fec367fc640 1 --2- 172.21.15.57:0/3294572670 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fec2406ae40 0x7fec2406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:45.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.250+0000 7fec367fc640 1 -- 172.21.15.57:0/3294572670 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fec38089b70 con 0x7fec4006bf20 2023-10-19T12:45:45.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.251+0000 7fec45456640 1 --2- 172.21.15.57:0/3294572670 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fec2406ae40 0x7fec2406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:45.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.251+0000 7fec45456640 1 --2- 172.21.15.57:0/3294572670 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fec2406ae40 0x7fec2406d300 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fec400b22a0 tx=0x7fec30005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:45.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.255+0000 7fec367fc640 1 -- 172.21.15.57:0/3294572670 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fec38058630 con 0x7fec4006bf20 2023-10-19T12:45:45.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.498+0000 7fec47ebb640 1 -- 172.21.15.57:0/3294572670 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fec400cc2f0 con 0x7fec4006bf20 2023-10-19T12:45:45.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.499+0000 7fec367fc640 1 -- 172.21.15.57:0/3294572670 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1144 (secure 0 0 0) 0x7fec380581c0 con 0x7fec4006bf20 2023-10-19T12:45:45.502 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:45:45.503 INFO:teuthology.orchestra.run.smithi057.stdout:{"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":112,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":39,"num_osds":8,"num_up_osds":8,"osd_up_since":1697719445,"num_in_osds":8,"osd_in_since":1697719436,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":53},{"state_name":"creating+peering","count":24},{"state_name":"creating+activating","count":20}],"num_pgs":97,"num_pools":3,"num_objects":7,"data_bytes":590438,"bytes_used":242995200,"bytes_avail":772851118080,"bytes_total":773094113280,"inactive_pgs_ratio":0.45360824465751648,"write_bytes_sec":170,"read_op_per_sec":0,"write_op_per_sec":1},"fsmap":{"epoch":16,"id":2,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":2,"rank":0,"name":"d","status":"up:active","gid":5634}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2023-10-19T12:45:23.331894+0000","services":{}},"progress_events":{}} 2023-10-19T12:45:45.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.501+0000 7fec47ebb640 1 -- 172.21.15.57:0/3294572670 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fec2406ae40 msgr2=0x7fec2406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:45.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.501+0000 7fec47ebb640 1 --2- 172.21.15.57:0/3294572670 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fec2406ae40 0x7fec2406d300 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fec400b22a0 tx=0x7fec30005f90 comp rx=0 tx=0).stop 2023-10-19T12:45:45.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.501+0000 7fec47ebb640 1 -- 172.21.15.57:0/3294572670 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec4006bf20 msgr2=0x7fec400c51d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:45.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.501+0000 7fec47ebb640 1 --2- 172.21.15.57:0/3294572670 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec4006bf20 0x7fec400c51d0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fec3800c460 tx=0x7fec3800c920 comp rx=0 tx=0).stop 2023-10-19T12:45:45.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.501+0000 7fec47ebb640 1 -- 172.21.15.57:0/3294572670 shutdown_connections 2023-10-19T12:45:45.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.501+0000 7fec47ebb640 1 --2- 172.21.15.57:0/3294572670 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fec2406ae40 0x7fec2406d300 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:45.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.501+0000 7fec47ebb640 1 --2- 172.21.15.57:0/3294572670 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fec400cd0c0 0x7fec4009c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:45.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.501+0000 7fec47ebb640 1 --2- 172.21.15.57:0/3294572670 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fec4006dce0 0x7fec40156060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:45.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.501+0000 7fec47ebb640 1 --2- 172.21.15.57:0/3294572670 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec4006bf20 0x7fec400c51d0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:45.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.501+0000 7fec47ebb640 1 -- 172.21.15.57:0/3294572670 >> 172.21.15.57:0/3294572670 conn(0x7fec400b0670 msgr2=0x7fec4015a300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:45.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.501+0000 7fec47ebb640 1 -- 172.21.15.57:0/3294572670 shutdown_connections 2023-10-19T12:45:45.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.501+0000 7fec47ebb640 1 -- 172.21.15.57:0/3294572670 wait complete. 2023-10-19T12:45:45.514 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2023-10-19T12:45:45.515 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-19T12:45:45.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.621+0000 7fd363f33640 1 -- 172.21.15.57:0/3642921777 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd3440047c0 msgr2=0x7fd344004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:45.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.621+0000 7fd363f33640 1 --2- 172.21.15.57:0/3642921777 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd3440047c0 0x7fd344004c40 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fd350006f80 tx=0x7fd3440057f0 comp rx=0 tx=0).stop 2023-10-19T12:45:45.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.621+0000 7fd3624d0640 1 -- 172.21.15.57:0/3642921777 reap_dead start 2023-10-19T12:45:45.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.621+0000 7fd363f33640 1 -- 172.21.15.57:0/3642921777 shutdown_connections 2023-10-19T12:45:45.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.621+0000 7fd363f33640 1 -- 172.21.15.57:0/3642921777 >> 172.21.15.57:0/3642921777 conn(0x7fd35c069930 msgr2=0x7fd35c069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:45.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.621+0000 7fd363f33640 1 -- 172.21.15.57:0/3642921777 shutdown_connections 2023-10-19T12:45:45.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.621+0000 7fd363f33640 1 -- 172.21.15.57:0/3642921777 wait complete. 2023-10-19T12:45:45.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.622+0000 7fd363f33640 1 Processor -- start 2023-10-19T12:45:45.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.622+0000 7fd363f33640 1 -- start start 2023-10-19T12:45:45.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.623+0000 7fd363f33640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd3440047c0 0x7fd35c13fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:45.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.623+0000 7fd363f33640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd35c13ee10 0x7fd35c15e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:45.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.623+0000 7fd363f33640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd35c15ea80 0x7fd35c15ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:45.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.623+0000 7fd363f33640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd35c140860 con 0x7fd35c13ee10 2023-10-19T12:45:45.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.623+0000 7fd363f33640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd35c140580 con 0x7fd3440047c0 2023-10-19T12:45:45.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.623+0000 7fd363f33640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd35c153070 con 0x7fd35c15ea80 2023-10-19T12:45:45.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.623+0000 7fd3614ce640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd35c13ee10 0x7fd35c15e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:45.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.623+0000 7fd3614ce640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd35c13ee10 0x7fd35c15e540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59396/0 (socket says 172.21.15.57:59396) 2023-10-19T12:45:45.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.623+0000 7fd3614ce640 1 -- 172.21.15.57:0/3714565690 learned_addr learned my addr 172.21.15.57:0/3714565690 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:45.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.623+0000 7fd3624d0640 1 --2- 172.21.15.57:0/3714565690 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd35c15ea80 0x7fd35c15ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:45.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.623+0000 7fd3614ce640 1 -- 172.21.15.57:0/3714565690 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd35c15ea80 msgr2=0x7fd35c15ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:45.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.623+0000 7fd3614ce640 1 --2- 172.21.15.57:0/3714565690 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd35c15ea80 0x7fd35c15ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:45.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.623+0000 7fd3614ce640 1 -- 172.21.15.57:0/3714565690 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd3440047c0 msgr2=0x7fd35c13fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:45.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.623+0000 7fd361ccf640 1 --2- 172.21.15.57:0/3714565690 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd3440047c0 0x7fd35c13fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:45.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.623+0000 7fd3614ce640 1 --2- 172.21.15.57:0/3714565690 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd3440047c0 0x7fd35c13fd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:45.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.623+0000 7fd3614ce640 1 -- 172.21.15.57:0/3714565690 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd35c153350 con 0x7fd35c13ee10 2023-10-19T12:45:45.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.624+0000 7fd3614ce640 1 --2- 172.21.15.57:0/3714565690 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd35c13ee10 0x7fd35c15e540 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fd35002c880 tx=0x7fd350001720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:45.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.624+0000 7fd35a7fc640 1 -- 172.21.15.57:0/3714565690 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd350003a00 con 0x7fd35c13ee10 2023-10-19T12:45:45.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.624+0000 7fd35a7fc640 1 -- 172.21.15.57:0/3714565690 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fd350003c50 con 0x7fd35c13ee10 2023-10-19T12:45:45.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.624+0000 7fd363f33640 1 -- 172.21.15.57:0/3714565690 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd35c153630 con 0x7fd35c13ee10 2023-10-19T12:45:45.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.624+0000 7fd35a7fc640 1 -- 172.21.15.57:0/3714565690 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd350035470 con 0x7fd35c13ee10 2023-10-19T12:45:45.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.624+0000 7fd363f33640 1 -- 172.21.15.57:0/3714565690 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd35c1524d0 con 0x7fd35c13ee10 2023-10-19T12:45:45.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.625+0000 7fd363f33640 1 -- 172.21.15.57:0/3714565690 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd35c1537c0 con 0x7fd35c13ee10 2023-10-19T12:45:45.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.625+0000 7fd35a7fc640 1 -- 172.21.15.57:0/3714565690 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fd350032070 con 0x7fd35c13ee10 2023-10-19T12:45:45.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.626+0000 7fd35a7fc640 1 --2- 172.21.15.57:0/3714565690 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fd34806ae70 0x7fd34806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:45.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.626+0000 7fd35a7fc640 1 -- 172.21.15.57:0/3714565690 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fd3500aac00 con 0x7fd35c13ee10 2023-10-19T12:45:45.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.626+0000 7fd361ccf640 1 --2- 172.21.15.57:0/3714565690 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fd34806ae70 0x7fd34806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:45.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.626+0000 7fd361ccf640 1 --2- 172.21.15.57:0/3714565690 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fd34806ae70 0x7fd34806d330 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fd354006630 tx=0x7fd354006050 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:45.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.630+0000 7fd35a7fc640 1 -- 172.21.15.57:0/3714565690 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd3500b0020 con 0x7fd35c13ee10 2023-10-19T12:45:45.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.801+0000 7fd363f33640 1 -- 172.21.15.57:0/3714565690 --> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd35c0b12b0 con 0x7fd34806ae70 2023-10-19T12:45:45.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.809+0000 7fd35a7fc640 1 -- 172.21.15.57:0/3714565690 <== mgr.4100 v2:172.21.15.179:6832/2609646285 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286395 (secure 0 0 0) 0x7fd354006e80 con 0x7fd34806ae70 2023-10-19T12:45:45.815 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:45:45.816 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2023-10-19T12:45:45.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.812+0000 7fd363f33640 1 -- 172.21.15.57:0/3714565690 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fd34806ae70 msgr2=0x7fd34806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:45.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.812+0000 7fd363f33640 1 --2- 172.21.15.57:0/3714565690 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fd34806ae70 0x7fd34806d330 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fd354006630 tx=0x7fd354006050 comp rx=0 tx=0).stop 2023-10-19T12:45:45.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.812+0000 7fd363f33640 1 -- 172.21.15.57:0/3714565690 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd35c13ee10 msgr2=0x7fd35c15e540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:45.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.812+0000 7fd363f33640 1 --2- 172.21.15.57:0/3714565690 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd35c13ee10 0x7fd35c15e540 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fd35002c880 tx=0x7fd350001720 comp rx=0 tx=0).stop 2023-10-19T12:45:45.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.812+0000 7fd363f33640 1 -- 172.21.15.57:0/3714565690 shutdown_connections 2023-10-19T12:45:45.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.812+0000 7fd363f33640 1 --2- 172.21.15.57:0/3714565690 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fd34806ae70 0x7fd34806d330 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:45.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.812+0000 7fd363f33640 1 --2- 172.21.15.57:0/3714565690 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd35c15ea80 0x7fd35c15ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:45.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.812+0000 7fd363f33640 1 --2- 172.21.15.57:0/3714565690 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd35c13ee10 0x7fd35c15e540 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:45.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.812+0000 7fd363f33640 1 --2- 172.21.15.57:0/3714565690 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd3440047c0 0x7fd35c13fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:45.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.812+0000 7fd363f33640 1 -- 172.21.15.57:0/3714565690 >> 172.21.15.57:0/3714565690 conn(0x7fd35c069930 msgr2=0x7fd35c0703e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:45.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.812+0000 7fd363f33640 1 -- 172.21.15.57:0/3714565690 shutdown_connections 2023-10-19T12:45:45.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:45.812+0000 7fd363f33640 1 -- 172.21.15.57:0/3714565690 wait complete. 2023-10-19T12:45:45.826 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":82,"stamp":"2023-10-19T12:45:45.336903+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590438,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":168,"num_write_kb":2417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":126,"ondisk_log_size":126,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":56,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":237300,"kb_used_data":4496,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754737420,"statfs":{"total":773094113280,"available":772851118080,"internally_reserved":0,"allocated":4603904,"data_stored":2628204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":238276153},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":70,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001380"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.347634+0000","last_change":"2023-10-19T12:45:40.347634+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.347634+0000","last_undegraded":"2023-10-19T12:45:40.347634+0000","last_fullsized":"2023-10-19T12:45:40.347634+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.349030+0000","last_change":"2023-10-19T12:45:40.349030+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.349030+0000","last_undegraded":"2023-10-19T12:45:40.349030+0000","last_fullsized":"2023-10-19T12:45:40.349030+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.376066+0000","last_change":"2023-10-19T12:45:41.371589+0000","last_active":"2023-10-19T12:45:41.376066+0000","last_peered":"2023-10-19T12:45:41.376066+0000","last_clean":"2023-10-19T12:45:41.376066+0000","last_became_active":"2023-10-19T12:45:41.371287+0000","last_became_peered":"2023-10-19T12:45:41.371287+0000","last_unstale":"2023-10-19T12:45:41.376066+0000","last_undegraded":"2023-10-19T12:45:41.376066+0000","last_fullsized":"2023-10-19T12:45:41.376066+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:16:37.419534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.370328+0000","last_change":"2023-10-19T12:45:41.363930+0000","last_active":"2023-10-19T12:45:41.370328+0000","last_peered":"2023-10-19T12:45:41.370328+0000","last_clean":"2023-10-19T12:45:41.370328+0000","last_became_active":"2023-10-19T12:45:41.363668+0000","last_became_peered":"2023-10-19T12:45:41.363668+0000","last_unstale":"2023-10-19T12:45:41.370328+0000","last_undegraded":"2023-10-19T12:45:41.370328+0000","last_fullsized":"2023-10-19T12:45:41.370328+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:51:05.063828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.370788+0000","last_change":"2023-10-19T12:45:41.363093+0000","last_active":"2023-10-19T12:45:41.370788+0000","last_peered":"2023-10-19T12:45:41.370788+0000","last_clean":"2023-10-19T12:45:41.370788+0000","last_became_active":"2023-10-19T12:45:41.362804+0000","last_became_peered":"2023-10-19T12:45:41.362804+0000","last_unstale":"2023-10-19T12:45:41.370788+0000","last_undegraded":"2023-10-19T12:45:41.370788+0000","last_fullsized":"2023-10-19T12:45:41.370788+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:42:33.722992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.374791+0000","last_change":"2023-10-19T12:45:41.359415+0000","last_active":"2023-10-19T12:45:41.374791+0000","last_peered":"2023-10-19T12:45:41.374791+0000","last_clean":"2023-10-19T12:45:41.374791+0000","last_became_active":"2023-10-19T12:45:41.359120+0000","last_became_peered":"2023-10-19T12:45:41.359120+0000","last_unstale":"2023-10-19T12:45:41.374791+0000","last_undegraded":"2023-10-19T12:45:41.374791+0000","last_fullsized":"2023-10-19T12:45:41.374791+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:48:25.218615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":3,"reported_epoch":38,"state":"creating+activating","last_fresh":"2023-10-19T12:45:41.350760+0000","last_change":"2023-10-19T12:45:41.350760+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:41.350760+0000","last_undegraded":"2023-10-19T12:45:41.350760+0000","last_fullsized":"2023-10-19T12:45:41.350760+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:38:37.793736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.349543+0000","last_change":"2023-10-19T12:45:40.349543+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.349543+0000","last_undegraded":"2023-10-19T12:45:40.349543+0000","last_fullsized":"2023-10-19T12:45:40.349543+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.375561+0000","last_change":"2023-10-19T12:45:41.368731+0000","last_active":"2023-10-19T12:45:41.375561+0000","last_peered":"2023-10-19T12:45:41.375561+0000","last_clean":"2023-10-19T12:45:41.375561+0000","last_became_active":"2023-10-19T12:45:41.368006+0000","last_became_peered":"2023-10-19T12:45:41.368006+0000","last_unstale":"2023-10-19T12:45:41.375561+0000","last_undegraded":"2023-10-19T12:45:41.375561+0000","last_fullsized":"2023-10-19T12:45:41.375561+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:48:25.218615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.345142+0000","last_change":"2023-10-19T12:45:40.345142+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.345142+0000","last_undegraded":"2023-10-19T12:45:40.345142+0000","last_fullsized":"2023-10-19T12:45:40.345142+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.346201+0000","last_change":"2023-10-19T12:45:40.346201+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.346201+0000","last_undegraded":"2023-10-19T12:45:40.346201+0000","last_fullsized":"2023-10-19T12:45:40.346201+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.370682+0000","last_change":"2023-10-19T12:45:41.357975+0000","last_active":"2023-10-19T12:45:41.370682+0000","last_peered":"2023-10-19T12:45:41.370682+0000","last_clean":"2023-10-19T12:45:41.370682+0000","last_became_active":"2023-10-19T12:45:41.357717+0000","last_became_peered":"2023-10-19T12:45:41.357717+0000","last_unstale":"2023-10-19T12:45:41.370682+0000","last_undegraded":"2023-10-19T12:45:41.370682+0000","last_fullsized":"2023-10-19T12:45:41.370682+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:49:21.932303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.345686+0000","last_change":"2023-10-19T12:45:40.345686+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.345686+0000","last_undegraded":"2023-10-19T12:45:40.345686+0000","last_fullsized":"2023-10-19T12:45:40.345686+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.351066+0000","last_change":"2023-10-19T12:45:40.351066+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.351066+0000","last_undegraded":"2023-10-19T12:45:40.351066+0000","last_fullsized":"2023-10-19T12:45:40.351066+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":3,"reported_epoch":38,"state":"creating+activating","last_fresh":"2023-10-19T12:45:41.350355+0000","last_change":"2023-10-19T12:45:41.350355+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:41.350355+0000","last_undegraded":"2023-10-19T12:45:41.350355+0000","last_fullsized":"2023-10-19T12:45:41.350355+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:10:04.203620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":3,"reported_epoch":38,"state":"creating+activating","last_fresh":"2023-10-19T12:45:41.352539+0000","last_change":"2023-10-19T12:45:41.352539+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:41.352539+0000","last_undegraded":"2023-10-19T12:45:41.352539+0000","last_fullsized":"2023-10-19T12:45:41.352539+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:21:51.336716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":3,"reported_epoch":38,"state":"creating+activating","last_fresh":"2023-10-19T12:45:41.351475+0000","last_change":"2023-10-19T12:45:41.351475+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:41.351475+0000","last_undegraded":"2023-10-19T12:45:41.351475+0000","last_fullsized":"2023-10-19T12:45:41.351475+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:32:37.014614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":3,"reported_epoch":38,"state":"creating+activating","last_fresh":"2023-10-19T12:45:41.348402+0000","last_change":"2023-10-19T12:45:41.348402+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:41.348402+0000","last_undegraded":"2023-10-19T12:45:41.348402+0000","last_fullsized":"2023-10-19T12:45:41.348402+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:53:30.629136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.364662+0000","last_change":"2023-10-19T12:45:41.356683+0000","last_active":"2023-10-19T12:45:41.364662+0000","last_peered":"2023-10-19T12:45:41.364662+0000","last_clean":"2023-10-19T12:45:41.364662+0000","last_became_active":"2023-10-19T12:45:41.356404+0000","last_became_peered":"2023-10-19T12:45:41.356404+0000","last_unstale":"2023-10-19T12:45:41.364662+0000","last_undegraded":"2023-10-19T12:45:41.364662+0000","last_fullsized":"2023-10-19T12:45:41.364662+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:30:20.055674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":3,"reported_epoch":38,"state":"creating+activating","last_fresh":"2023-10-19T12:45:41.351396+0000","last_change":"2023-10-19T12:45:41.351396+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:41.351396+0000","last_undegraded":"2023-10-19T12:45:41.351396+0000","last_fullsized":"2023-10-19T12:45:41.351396+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:48:25.218615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.344680+0000","last_change":"2023-10-19T12:45:40.344680+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.344680+0000","last_undegraded":"2023-10-19T12:45:40.344680+0000","last_fullsized":"2023-10-19T12:45:40.344680+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.348310+0000","last_change":"2023-10-19T12:45:40.348310+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.348310+0000","last_undegraded":"2023-10-19T12:45:40.348310+0000","last_fullsized":"2023-10-19T12:45:40.348310+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":3,"reported_epoch":38,"state":"creating+activating","last_fresh":"2023-10-19T12:45:41.350788+0000","last_change":"2023-10-19T12:45:41.350788+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:41.350788+0000","last_undegraded":"2023-10-19T12:45:41.350788+0000","last_fullsized":"2023-10-19T12:45:41.350788+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:44:04.457483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.371142+0000","last_change":"2023-10-19T12:45:41.361356+0000","last_active":"2023-10-19T12:45:41.371142+0000","last_peered":"2023-10-19T12:45:41.371142+0000","last_clean":"2023-10-19T12:45:41.371142+0000","last_became_active":"2023-10-19T12:45:41.361082+0000","last_became_peered":"2023-10-19T12:45:41.361082+0000","last_unstale":"2023-10-19T12:45:41.371142+0000","last_undegraded":"2023-10-19T12:45:41.371142+0000","last_fullsized":"2023-10-19T12:45:41.371142+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:45:11.031973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":3,"reported_epoch":38,"state":"creating+activating","last_fresh":"2023-10-19T12:45:41.350918+0000","last_change":"2023-10-19T12:45:41.350918+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:41.350918+0000","last_undegraded":"2023-10-19T12:45:41.350918+0000","last_fullsized":"2023-10-19T12:45:41.350918+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:49:21.932303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.375784+0000","last_change":"2023-10-19T12:45:41.372032+0000","last_active":"2023-10-19T12:45:41.375784+0000","last_peered":"2023-10-19T12:45:41.375784+0000","last_clean":"2023-10-19T12:45:41.375784+0000","last_became_active":"2023-10-19T12:45:41.371582+0000","last_became_peered":"2023-10-19T12:45:41.371582+0000","last_unstale":"2023-10-19T12:45:41.375784+0000","last_undegraded":"2023-10-19T12:45:41.375784+0000","last_fullsized":"2023-10-19T12:45:41.375784+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:55:02.268260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":3,"reported_epoch":38,"state":"creating+activating","last_fresh":"2023-10-19T12:45:41.348601+0000","last_change":"2023-10-19T12:45:41.348601+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:41.348601+0000","last_undegraded":"2023-10-19T12:45:41.348601+0000","last_fullsized":"2023-10-19T12:45:41.348601+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:36:07.043432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.345780+0000","last_change":"2023-10-19T12:45:40.345780+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.345780+0000","last_undegraded":"2023-10-19T12:45:40.345780+0000","last_fullsized":"2023-10-19T12:45:40.345780+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":3,"reported_epoch":38,"state":"creating+activating","last_fresh":"2023-10-19T12:45:41.351166+0000","last_change":"2023-10-19T12:45:41.351166+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:41.351166+0000","last_undegraded":"2023-10-19T12:45:41.351166+0000","last_fullsized":"2023-10-19T12:45:41.351166+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:16:37.419534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":3,"reported_epoch":38,"state":"creating+activating","last_fresh":"2023-10-19T12:45:41.348399+0000","last_change":"2023-10-19T12:45:41.348399+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:41.348399+0000","last_undegraded":"2023-10-19T12:45:41.348399+0000","last_fullsized":"2023-10-19T12:45:41.348399+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T12:45:47.935169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":["6","3"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":3,"reported_epoch":38,"state":"creating+activating","last_fresh":"2023-10-19T12:45:41.351136+0000","last_change":"2023-10-19T12:45:41.351136+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:41.351136+0000","last_undegraded":"2023-10-19T12:45:41.351136+0000","last_fullsized":"2023-10-19T12:45:41.351136+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:10:04.203620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.369704+0000","last_change":"2023-10-19T12:45:41.356009+0000","last_active":"2023-10-19T12:45:41.369704+0000","last_peered":"2023-10-19T12:45:41.369704+0000","last_clean":"2023-10-19T12:45:41.369704+0000","last_became_active":"2023-10-19T12:45:41.355714+0000","last_became_peered":"2023-10-19T12:45:41.355714+0000","last_unstale":"2023-10-19T12:45:41.369704+0000","last_undegraded":"2023-10-19T12:45:41.369704+0000","last_fullsized":"2023-10-19T12:45:41.369704+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:16:37.419534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":3,"reported_epoch":38,"state":"creating+activating","last_fresh":"2023-10-19T12:45:41.351973+0000","last_change":"2023-10-19T12:45:41.351973+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:41.351973+0000","last_undegraded":"2023-10-19T12:45:41.351973+0000","last_fullsized":"2023-10-19T12:45:41.351973+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:53:29.708835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-19T12:45:40.364691+0000","last_change":"2023-10-19T12:45:40.359697+0000","last_active":"2023-10-19T12:45:40.364691+0000","last_peered":"2023-10-19T12:45:40.364691+0000","last_clean":"2023-10-19T12:45:40.364691+0000","last_became_active":"2023-10-19T12:45:40.359413+0000","last_became_peered":"2023-10-19T12:45:40.359413+0000","last_unstale":"2023-10-19T12:45:40.364691+0000","last_undegraded":"2023-10-19T12:45:40.364691+0000","last_fullsized":"2023-10-19T12:45:40.364691+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:36:38.697134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.375801+0000","last_change":"2023-10-19T12:45:41.372556+0000","last_active":"2023-10-19T12:45:41.375801+0000","last_peered":"2023-10-19T12:45:41.375801+0000","last_clean":"2023-10-19T12:45:41.375801+0000","last_became_active":"2023-10-19T12:45:41.372184+0000","last_became_peered":"2023-10-19T12:45:41.372184+0000","last_unstale":"2023-10-19T12:45:41.375801+0000","last_undegraded":"2023-10-19T12:45:41.375801+0000","last_fullsized":"2023-10-19T12:45:41.375801+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:10:04.203620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"38'1","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.370313+0000","last_change":"2023-10-19T12:45:40.352581+0000","last_active":"2023-10-19T12:45:41.370313+0000","last_peered":"2023-10-19T12:45:41.370313+0000","last_clean":"2023-10-19T12:45:41.370313+0000","last_became_active":"2023-10-19T12:45:40.352300+0000","last_became_peered":"2023-10-19T12:45:40.352300+0000","last_unstale":"2023-10-19T12:45:41.370313+0000","last_undegraded":"2023-10-19T12:45:41.370313+0000","last_fullsized":"2023-10-19T12:45:41.370313+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:59:58.100040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.348927+0000","last_change":"2023-10-19T12:45:40.348927+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.348927+0000","last_undegraded":"2023-10-19T12:45:40.348927+0000","last_fullsized":"2023-10-19T12:45:40.348927+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"38'2","reported_seq":13,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.374990+0000","last_change":"2023-10-19T12:45:40.355990+0000","last_active":"2023-10-19T12:45:41.374990+0000","last_peered":"2023-10-19T12:45:41.374990+0000","last_clean":"2023-10-19T12:45:41.374990+0000","last_became_active":"2023-10-19T12:45:40.355666+0000","last_became_peered":"2023-10-19T12:45:40.355666+0000","last_unstale":"2023-10-19T12:45:41.374990+0000","last_undegraded":"2023-10-19T12:45:41.374990+0000","last_fullsized":"2023-10-19T12:45:41.374990+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:53:28.702370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.349506+0000","last_change":"2023-10-19T12:45:40.349506+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.349506+0000","last_undegraded":"2023-10-19T12:45:40.349506+0000","last_fullsized":"2023-10-19T12:45:40.349506+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-19T12:45:40.367475+0000","last_change":"2023-10-19T12:45:40.364018+0000","last_active":"2023-10-19T12:45:40.367475+0000","last_peered":"2023-10-19T12:45:40.367475+0000","last_clean":"2023-10-19T12:45:40.367475+0000","last_became_active":"2023-10-19T12:45:40.363763+0000","last_became_peered":"2023-10-19T12:45:40.363763+0000","last_unstale":"2023-10-19T12:45:40.367475+0000","last_undegraded":"2023-10-19T12:45:40.367475+0000","last_fullsized":"2023-10-19T12:45:40.367475+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:36:38.697134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.363923+0000","last_change":"2023-10-19T12:45:41.355176+0000","last_active":"2023-10-19T12:45:41.363923+0000","last_peered":"2023-10-19T12:45:41.363923+0000","last_clean":"2023-10-19T12:45:41.363923+0000","last_became_active":"2023-10-19T12:45:41.354687+0000","last_became_peered":"2023-10-19T12:45:41.354687+0000","last_unstale":"2023-10-19T12:45:41.363923+0000","last_undegraded":"2023-10-19T12:45:41.363923+0000","last_fullsized":"2023-10-19T12:45:41.363923+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:53:29.708835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-19T12:45:40.362034+0000","last_change":"2023-10-19T12:45:40.357988+0000","last_active":"2023-10-19T12:45:40.362034+0000","last_peered":"2023-10-19T12:45:40.362034+0000","last_clean":"2023-10-19T12:45:40.362034+0000","last_became_active":"2023-10-19T12:45:40.357478+0000","last_became_peered":"2023-10-19T12:45:40.357478+0000","last_unstale":"2023-10-19T12:45:40.362034+0000","last_undegraded":"2023-10-19T12:45:40.362034+0000","last_fullsized":"2023-10-19T12:45:40.362034+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:49:20.925838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":3,"reported_epoch":38,"state":"creating+activating","last_fresh":"2023-10-19T12:45:41.349960+0000","last_change":"2023-10-19T12:45:41.349960+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:41.349960+0000","last_undegraded":"2023-10-19T12:45:41.349960+0000","last_fullsized":"2023-10-19T12:45:41.349960+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:44:04.457483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":11,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.349382+0000","last_change":"2023-10-19T12:45:40.355247+0000","last_active":"2023-10-19T12:45:41.349382+0000","last_peered":"2023-10-19T12:45:41.349382+0000","last_clean":"2023-10-19T12:45:41.349382+0000","last_became_active":"2023-10-19T12:45:40.354967+0000","last_became_peered":"2023-10-19T12:45:40.354967+0000","last_unstale":"2023-10-19T12:45:41.349382+0000","last_undegraded":"2023-10-19T12:45:41.349382+0000","last_fullsized":"2023-10-19T12:45:41.349382+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:48:24.212150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.350186+0000","last_change":"2023-10-19T12:45:40.350186+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.350186+0000","last_undegraded":"2023-10-19T12:45:40.350186+0000","last_fullsized":"2023-10-19T12:45:40.350186+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":11,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.348919+0000","last_change":"2023-10-19T12:45:40.359949+0000","last_active":"2023-10-19T12:45:41.348919+0000","last_peered":"2023-10-19T12:45:41.348919+0000","last_clean":"2023-10-19T12:45:41.348919+0000","last_became_active":"2023-10-19T12:45:40.359690+0000","last_became_peered":"2023-10-19T12:45:40.359690+0000","last_unstale":"2023-10-19T12:45:41.348919+0000","last_undegraded":"2023-10-19T12:45:41.348919+0000","last_fullsized":"2023-10-19T12:45:41.348919+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:44:03.451018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.351154+0000","last_change":"2023-10-19T12:45:40.351154+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.351154+0000","last_undegraded":"2023-10-19T12:45:40.351154+0000","last_fullsized":"2023-10-19T12:45:40.351154+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-19T12:45:40.364109+0000","last_change":"2023-10-19T12:45:40.360289+0000","last_active":"2023-10-19T12:45:40.364109+0000","last_peered":"2023-10-19T12:45:40.364109+0000","last_clean":"2023-10-19T12:45:40.364109+0000","last_became_active":"2023-10-19T12:45:40.359850+0000","last_became_peered":"2023-10-19T12:45:40.359850+0000","last_unstale":"2023-10-19T12:45:40.364109+0000","last_undegraded":"2023-10-19T12:45:40.364109+0000","last_fullsized":"2023-10-19T12:45:40.364109+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:30:19.049209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.370304+0000","last_change":"2023-10-19T12:45:41.363350+0000","last_active":"2023-10-19T12:45:41.370304+0000","last_peered":"2023-10-19T12:45:41.370304+0000","last_clean":"2023-10-19T12:45:41.370304+0000","last_became_active":"2023-10-19T12:45:41.363067+0000","last_became_peered":"2023-10-19T12:45:41.363067+0000","last_unstale":"2023-10-19T12:45:41.370304+0000","last_undegraded":"2023-10-19T12:45:41.370304+0000","last_fullsized":"2023-10-19T12:45:41.370304+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:09:22.266843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-19T12:45:40.363651+0000","last_change":"2023-10-19T12:45:40.356475+0000","last_active":"2023-10-19T12:45:40.363651+0000","last_peered":"2023-10-19T12:45:40.363651+0000","last_clean":"2023-10-19T12:45:40.363651+0000","last_became_active":"2023-10-19T12:45:40.356178+0000","last_became_peered":"2023-10-19T12:45:40.356178+0000","last_unstale":"2023-10-19T12:45:40.363651+0000","last_undegraded":"2023-10-19T12:45:40.363651+0000","last_fullsized":"2023-10-19T12:45:40.363651+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:59:58.100040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":3,"reported_epoch":38,"state":"creating+activating","last_fresh":"2023-10-19T12:45:41.350085+0000","last_change":"2023-10-19T12:45:41.350085+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:41.350085+0000","last_undegraded":"2023-10-19T12:45:41.350085+0000","last_fullsized":"2023-10-19T12:45:41.350085+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:29:00.286113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-19T12:45:40.364238+0000","last_change":"2023-10-19T12:45:40.360586+0000","last_active":"2023-10-19T12:45:40.364238+0000","last_peered":"2023-10-19T12:45:40.364238+0000","last_clean":"2023-10-19T12:45:40.364238+0000","last_became_active":"2023-10-19T12:45:40.360327+0000","last_became_peered":"2023-10-19T12:45:40.360327+0000","last_unstale":"2023-10-19T12:45:40.364238+0000","last_undegraded":"2023-10-19T12:45:40.364238+0000","last_fullsized":"2023-10-19T12:45:40.364238+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:55:01.261795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.375472+0000","last_change":"2023-10-19T12:45:41.366187+0000","last_active":"2023-10-19T12:45:41.375472+0000","last_peered":"2023-10-19T12:45:41.375472+0000","last_clean":"2023-10-19T12:45:41.375472+0000","last_became_active":"2023-10-19T12:45:41.365896+0000","last_became_peered":"2023-10-19T12:45:41.365896+0000","last_unstale":"2023-10-19T12:45:41.375472+0000","last_undegraded":"2023-10-19T12:45:41.375472+0000","last_fullsized":"2023-10-19T12:45:41.375472+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:55:02.268260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-19T12:45:40.361876+0000","last_change":"2023-10-19T12:45:40.358267+0000","last_active":"2023-10-19T12:45:40.361876+0000","last_peered":"2023-10-19T12:45:40.361876+0000","last_clean":"2023-10-19T12:45:40.361876+0000","last_became_active":"2023-10-19T12:45:40.358000+0000","last_became_peered":"2023-10-19T12:45:40.358000+0000","last_unstale":"2023-10-19T12:45:40.361876+0000","last_undegraded":"2023-10-19T12:45:40.361876+0000","last_fullsized":"2023-10-19T12:45:40.361876+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:48:24.212150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.348513+0000","last_change":"2023-10-19T12:45:40.348513+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.348513+0000","last_undegraded":"2023-10-19T12:45:40.348513+0000","last_fullsized":"2023-10-19T12:45:40.348513+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-19T12:45:40.364295+0000","last_change":"2023-10-19T12:45:40.361715+0000","last_active":"2023-10-19T12:45:40.364295+0000","last_peered":"2023-10-19T12:45:40.364295+0000","last_clean":"2023-10-19T12:45:40.364295+0000","last_became_active":"2023-10-19T12:45:40.361442+0000","last_became_peered":"2023-10-19T12:45:40.361442+0000","last_unstale":"2023-10-19T12:45:40.364295+0000","last_undegraded":"2023-10-19T12:45:40.364295+0000","last_fullsized":"2023-10-19T12:45:40.364295+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:44:03.451018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.375150+0000","last_change":"2023-10-19T12:45:41.364032+0000","last_active":"2023-10-19T12:45:41.375150+0000","last_peered":"2023-10-19T12:45:41.375150+0000","last_clean":"2023-10-19T12:45:41.375150+0000","last_became_active":"2023-10-19T12:45:41.363734+0000","last_became_peered":"2023-10-19T12:45:41.363734+0000","last_unstale":"2023-10-19T12:45:41.375150+0000","last_undegraded":"2023-10-19T12:45:41.375150+0000","last_fullsized":"2023-10-19T12:45:41.375150+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:32:37.014614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-19T12:45:40.363994+0000","last_change":"2023-10-19T12:45:40.360672+0000","last_active":"2023-10-19T12:45:40.363994+0000","last_peered":"2023-10-19T12:45:40.363994+0000","last_clean":"2023-10-19T12:45:40.363994+0000","last_became_active":"2023-10-19T12:45:40.360396+0000","last_became_peered":"2023-10-19T12:45:40.360396+0000","last_unstale":"2023-10-19T12:45:40.363994+0000","last_undegraded":"2023-10-19T12:45:40.363994+0000","last_fullsized":"2023-10-19T12:45:40.363994+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:28:59.279648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.347231+0000","last_change":"2023-10-19T12:45:40.347231+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.347231+0000","last_undegraded":"2023-10-19T12:45:40.347231+0000","last_fullsized":"2023-10-19T12:45:40.347231+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":11,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.348645+0000","last_change":"2023-10-19T12:45:40.359407+0000","last_active":"2023-10-19T12:45:41.348645+0000","last_peered":"2023-10-19T12:45:41.348645+0000","last_clean":"2023-10-19T12:45:41.348645+0000","last_became_active":"2023-10-19T12:45:40.359134+0000","last_became_peered":"2023-10-19T12:45:40.359134+0000","last_unstale":"2023-10-19T12:45:41.348645+0000","last_undegraded":"2023-10-19T12:45:41.348645+0000","last_fullsized":"2023-10-19T12:45:41.348645+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:59:58.100040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":3,"reported_epoch":38,"state":"creating+activating","last_fresh":"2023-10-19T12:45:41.349119+0000","last_change":"2023-10-19T12:45:41.349119+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:41.349119+0000","last_undegraded":"2023-10-19T12:45:41.349119+0000","last_fullsized":"2023-10-19T12:45:41.349119+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:16:37.419534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":11,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.350155+0000","last_change":"2023-10-19T12:45:40.359548+0000","last_active":"2023-10-19T12:45:41.350155+0000","last_peered":"2023-10-19T12:45:41.350155+0000","last_clean":"2023-10-19T12:45:41.350155+0000","last_became_active":"2023-10-19T12:45:40.359282+0000","last_became_peered":"2023-10-19T12:45:40.359282+0000","last_unstale":"2023-10-19T12:45:41.350155+0000","last_undegraded":"2023-10-19T12:45:41.350155+0000","last_fullsized":"2023-10-19T12:45:41.350155+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:39:09.771537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.350379+0000","last_change":"2023-10-19T12:45:40.350379+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.350379+0000","last_undegraded":"2023-10-19T12:45:40.350379+0000","last_fullsized":"2023-10-19T12:45:40.350379+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"38'3","reported_seq":11,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.352796+0000","last_change":"2023-10-19T12:45:40.358054+0000","last_active":"2023-10-19T12:45:41.352796+0000","last_peered":"2023-10-19T12:45:41.352796+0000","last_clean":"2023-10-19T12:45:41.352796+0000","last_became_active":"2023-10-19T12:45:40.357803+0000","last_became_peered":"2023-10-19T12:45:40.357803+0000","last_unstale":"2023-10-19T12:45:41.352796+0000","last_undegraded":"2023-10-19T12:45:41.352796+0000","last_fullsized":"2023-10-19T12:45:41.352796+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:07:55.739920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.350100+0000","last_change":"2023-10-19T12:45:40.350100+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.350100+0000","last_undegraded":"2023-10-19T12:45:40.350100+0000","last_fullsized":"2023-10-19T12:45:40.350100+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-19T12:45:40.362007+0000","last_change":"2023-10-19T12:45:40.358450+0000","last_active":"2023-10-19T12:45:40.362007+0000","last_peered":"2023-10-19T12:45:40.362007+0000","last_clean":"2023-10-19T12:45:40.362007+0000","last_became_active":"2023-10-19T12:45:40.356778+0000","last_became_peered":"2023-10-19T12:45:40.356778+0000","last_unstale":"2023-10-19T12:45:40.362007+0000","last_undegraded":"2023-10-19T12:45:40.362007+0000","last_fullsized":"2023-10-19T12:45:40.362007+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:16:36.413069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.345759+0000","last_change":"2023-10-19T12:45:40.345759+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.345759+0000","last_undegraded":"2023-10-19T12:45:40.345759+0000","last_fullsized":"2023-10-19T12:45:40.345759+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-19T12:45:40.361384+0000","last_change":"2023-10-19T12:45:40.357947+0000","last_active":"2023-10-19T12:45:40.361384+0000","last_peered":"2023-10-19T12:45:40.361384+0000","last_clean":"2023-10-19T12:45:40.361384+0000","last_became_active":"2023-10-19T12:45:40.357669+0000","last_became_peered":"2023-10-19T12:45:40.357669+0000","last_unstale":"2023-10-19T12:45:40.361384+0000","last_undegraded":"2023-10-19T12:45:40.361384+0000","last_fullsized":"2023-10-19T12:45:40.361384+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T12:45:46.928704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.350198+0000","last_change":"2023-10-19T12:45:40.350198+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.350198+0000","last_undegraded":"2023-10-19T12:45:40.350198+0000","last_fullsized":"2023-10-19T12:45:40.350198+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"38'1","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.370536+0000","last_change":"2023-10-19T12:45:40.354628+0000","last_active":"2023-10-19T12:45:41.370536+0000","last_peered":"2023-10-19T12:45:41.370536+0000","last_clean":"2023-10-19T12:45:41.370536+0000","last_became_active":"2023-10-19T12:45:40.354367+0000","last_became_peered":"2023-10-19T12:45:40.354367+0000","last_unstale":"2023-10-19T12:45:41.370536+0000","last_undegraded":"2023-10-19T12:45:41.370536+0000","last_fullsized":"2023-10-19T12:45:41.370536+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:33:49.817824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.375358+0000","last_change":"2023-10-19T12:45:41.367225+0000","last_active":"2023-10-19T12:45:41.375358+0000","last_peered":"2023-10-19T12:45:41.375358+0000","last_clean":"2023-10-19T12:45:41.375358+0000","last_became_active":"2023-10-19T12:45:41.366782+0000","last_became_peered":"2023-10-19T12:45:41.366782+0000","last_unstale":"2023-10-19T12:45:41.375358+0000","last_undegraded":"2023-10-19T12:45:41.375358+0000","last_fullsized":"2023-10-19T12:45:41.375358+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:50:19.159560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":11,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.349510+0000","last_change":"2023-10-19T12:45:40.357501+0000","last_active":"2023-10-19T12:45:41.349510+0000","last_peered":"2023-10-19T12:45:41.349510+0000","last_clean":"2023-10-19T12:45:41.349510+0000","last_became_active":"2023-10-19T12:45:40.355411+0000","last_became_peered":"2023-10-19T12:45:40.355411+0000","last_unstale":"2023-10-19T12:45:41.349510+0000","last_undegraded":"2023-10-19T12:45:41.349510+0000","last_fullsized":"2023-10-19T12:45:41.349510+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:32:36.008149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":3,"reported_epoch":38,"state":"creating+activating","last_fresh":"2023-10-19T12:45:41.353064+0000","last_change":"2023-10-19T12:45:41.353064+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:41.353064+0000","last_undegraded":"2023-10-19T12:45:41.353064+0000","last_fullsized":"2023-10-19T12:45:41.353064+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:59:55.870354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"38'2","reported_seq":13,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.376098+0000","last_change":"2023-10-19T12:45:40.357725+0000","last_active":"2023-10-19T12:45:41.376098+0000","last_peered":"2023-10-19T12:45:41.376098+0000","last_clean":"2023-10-19T12:45:41.376098+0000","last_became_active":"2023-10-19T12:45:40.357417+0000","last_became_peered":"2023-10-19T12:45:40.357417+0000","last_unstale":"2023-10-19T12:45:41.376098+0000","last_undegraded":"2023-10-19T12:45:41.376098+0000","last_fullsized":"2023-10-19T12:45:41.376098+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:26:34.780735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.375456+0000","last_change":"2023-10-19T12:45:41.366181+0000","last_active":"2023-10-19T12:45:41.375456+0000","last_peered":"2023-10-19T12:45:41.375456+0000","last_clean":"2023-10-19T12:45:41.375456+0000","last_became_active":"2023-10-19T12:45:41.365882+0000","last_became_peered":"2023-10-19T12:45:41.365882+0000","last_unstale":"2023-10-19T12:45:41.375456+0000","last_undegraded":"2023-10-19T12:45:41.375456+0000","last_fullsized":"2023-10-19T12:45:41.375456+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:29:00.286113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":11,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.343006+0000","last_change":"2023-10-19T12:45:40.355357+0000","last_active":"2023-10-19T12:45:41.343006+0000","last_peered":"2023-10-19T12:45:41.343006+0000","last_clean":"2023-10-19T12:45:41.343006+0000","last_became_active":"2023-10-19T12:45:40.355097+0000","last_became_peered":"2023-10-19T12:45:40.355097+0000","last_unstale":"2023-10-19T12:45:41.343006+0000","last_undegraded":"2023-10-19T12:45:41.343006+0000","last_fullsized":"2023-10-19T12:45:41.343006+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:23:29.827547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":3,"reported_epoch":38,"state":"creating+activating","last_fresh":"2023-10-19T12:45:41.349483+0000","last_change":"2023-10-19T12:45:41.349483+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:41.349483+0000","last_undegraded":"2023-10-19T12:45:41.349483+0000","last_fullsized":"2023-10-19T12:45:41.349483+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:59:59.106505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":11,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.351381+0000","last_change":"2023-10-19T12:45:40.360123+0000","last_active":"2023-10-19T12:45:41.351381+0000","last_peered":"2023-10-19T12:45:41.351381+0000","last_clean":"2023-10-19T12:45:41.351381+0000","last_became_active":"2023-10-19T12:45:40.359767+0000","last_became_peered":"2023-10-19T12:45:40.359767+0000","last_unstale":"2023-10-19T12:45:41.351381+0000","last_undegraded":"2023-10-19T12:45:41.351381+0000","last_fullsized":"2023-10-19T12:45:41.351381+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:38:36.787271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.346878+0000","last_change":"2023-10-19T12:45:40.346878+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.346878+0000","last_undegraded":"2023-10-19T12:45:40.346878+0000","last_fullsized":"2023-10-19T12:45:40.346878+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":11,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.343223+0000","last_change":"2023-10-19T12:45:40.357695+0000","last_active":"2023-10-19T12:45:41.343223+0000","last_peered":"2023-10-19T12:45:41.343223+0000","last_clean":"2023-10-19T12:45:41.343223+0000","last_became_active":"2023-10-19T12:45:40.357393+0000","last_became_peered":"2023-10-19T12:45:40.357393+0000","last_unstale":"2023-10-19T12:45:41.343223+0000","last_undegraded":"2023-10-19T12:45:41.343223+0000","last_fullsized":"2023-10-19T12:45:41.343223+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:43:31.256604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.375615+0000","last_change":"2023-10-19T12:45:41.371258+0000","last_active":"2023-10-19T12:45:41.375615+0000","last_peered":"2023-10-19T12:45:41.375615+0000","last_clean":"2023-10-19T12:45:41.375615+0000","last_became_active":"2023-10-19T12:45:41.370963+0000","last_became_peered":"2023-10-19T12:45:41.370963+0000","last_unstale":"2023-10-19T12:45:41.375615+0000","last_undegraded":"2023-10-19T12:45:41.375615+0000","last_fullsized":"2023-10-19T12:45:41.375615+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:38:37.793736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-19T12:45:40.361905+0000","last_change":"2023-10-19T12:45:40.358263+0000","last_active":"2023-10-19T12:45:40.361905+0000","last_peered":"2023-10-19T12:45:40.361905+0000","last_clean":"2023-10-19T12:45:40.361905+0000","last_became_active":"2023-10-19T12:45:40.357402+0000","last_became_peered":"2023-10-19T12:45:40.357402+0000","last_unstale":"2023-10-19T12:45:40.361905+0000","last_undegraded":"2023-10-19T12:45:40.361905+0000","last_fullsized":"2023-10-19T12:45:40.361905+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:28:59.279648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":3,"reported_epoch":38,"state":"creating+activating","last_fresh":"2023-10-19T12:45:41.350527+0000","last_change":"2023-10-19T12:45:41.350527+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:41.350527+0000","last_undegraded":"2023-10-19T12:45:41.350527+0000","last_fullsized":"2023-10-19T12:45:41.350527+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:59:59.106505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":11,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.343650+0000","last_change":"2023-10-19T12:45:40.352686+0000","last_active":"2023-10-19T12:45:41.343650+0000","last_peered":"2023-10-19T12:45:41.343650+0000","last_clean":"2023-10-19T12:45:41.343650+0000","last_became_active":"2023-10-19T12:45:40.352420+0000","last_became_peered":"2023-10-19T12:45:40.352420+0000","last_unstale":"2023-10-19T12:45:41.343650+0000","last_undegraded":"2023-10-19T12:45:41.343650+0000","last_fullsized":"2023-10-19T12:45:41.343650+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:36:38.697134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.346413+0000","last_change":"2023-10-19T12:45:40.346413+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.346413+0000","last_undegraded":"2023-10-19T12:45:40.346413+0000","last_fullsized":"2023-10-19T12:45:40.346413+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":11,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.348286+0000","last_change":"2023-10-19T12:45:40.358578+0000","last_active":"2023-10-19T12:45:41.348286+0000","last_peered":"2023-10-19T12:45:41.348286+0000","last_clean":"2023-10-19T12:45:41.348286+0000","last_became_active":"2023-10-19T12:45:40.358057+0000","last_became_peered":"2023-10-19T12:45:40.358057+0000","last_unstale":"2023-10-19T12:45:41.348286+0000","last_undegraded":"2023-10-19T12:45:41.348286+0000","last_fullsized":"2023-10-19T12:45:41.348286+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:38:36.787271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.375444+0000","last_change":"2023-10-19T12:45:41.367997+0000","last_active":"2023-10-19T12:45:41.375444+0000","last_peered":"2023-10-19T12:45:41.375444+0000","last_clean":"2023-10-19T12:45:41.375444+0000","last_became_active":"2023-10-19T12:45:41.367687+0000","last_became_peered":"2023-10-19T12:45:41.367687+0000","last_unstale":"2023-10-19T12:45:41.375444+0000","last_undegraded":"2023-10-19T12:45:41.375444+0000","last_fullsized":"2023-10-19T12:45:41.375444+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:10:04.203620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":11,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.348125+0000","last_change":"2023-10-19T12:45:40.360184+0000","last_active":"2023-10-19T12:45:41.348125+0000","last_peered":"2023-10-19T12:45:41.348125+0000","last_clean":"2023-10-19T12:45:41.348125+0000","last_became_active":"2023-10-19T12:45:40.359917+0000","last_became_peered":"2023-10-19T12:45:40.359917+0000","last_unstale":"2023-10-19T12:45:41.348125+0000","last_undegraded":"2023-10-19T12:45:41.348125+0000","last_fullsized":"2023-10-19T12:45:41.348125+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:44:03.451018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.374901+0000","last_change":"2023-10-19T12:45:41.364206+0000","last_active":"2023-10-19T12:45:41.374901+0000","last_peered":"2023-10-19T12:45:41.374901+0000","last_clean":"2023-10-19T12:45:41.374901+0000","last_became_active":"2023-10-19T12:45:41.363926+0000","last_became_peered":"2023-10-19T12:45:41.363926+0000","last_unstale":"2023-10-19T12:45:41.374901+0000","last_undegraded":"2023-10-19T12:45:41.374901+0000","last_fullsized":"2023-10-19T12:45:41.374901+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:29:00.286113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-19T12:45:40.364717+0000","last_change":"2023-10-19T12:45:40.360715+0000","last_active":"2023-10-19T12:45:40.364717+0000","last_peered":"2023-10-19T12:45:40.364717+0000","last_clean":"2023-10-19T12:45:40.364717+0000","last_became_active":"2023-10-19T12:45:40.359801+0000","last_became_peered":"2023-10-19T12:45:40.359801+0000","last_unstale":"2023-10-19T12:45:40.364717+0000","last_undegraded":"2023-10-19T12:45:40.364717+0000","last_fullsized":"2023-10-19T12:45:40.364717+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:45:10.025508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.350603+0000","last_change":"2023-10-19T12:45:40.350603+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.350603+0000","last_undegraded":"2023-10-19T12:45:40.350603+0000","last_fullsized":"2023-10-19T12:45:40.350603+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-19T12:45:40.367503+0000","last_change":"2023-10-19T12:45:40.364278+0000","last_active":"2023-10-19T12:45:40.367503+0000","last_peered":"2023-10-19T12:45:40.367503+0000","last_clean":"2023-10-19T12:45:40.367503+0000","last_became_active":"2023-10-19T12:45:40.363981+0000","last_became_peered":"2023-10-19T12:45:40.363981+0000","last_unstale":"2023-10-19T12:45:40.367503+0000","last_undegraded":"2023-10-19T12:45:40.367503+0000","last_fullsized":"2023-10-19T12:45:40.367503+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:50:18.153095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":3,"reported_epoch":38,"state":"creating+activating","last_fresh":"2023-10-19T12:45:41.349666+0000","last_change":"2023-10-19T12:45:41.349666+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:41.349666+0000","last_undegraded":"2023-10-19T12:45:41.349666+0000","last_fullsized":"2023-10-19T12:45:41.349666+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:55:02.268260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-19T12:45:40.364011+0000","last_change":"2023-10-19T12:45:40.359438+0000","last_active":"2023-10-19T12:45:40.364011+0000","last_peered":"2023-10-19T12:45:40.364011+0000","last_clean":"2023-10-19T12:45:40.364011+0000","last_became_active":"2023-10-19T12:45:40.359156+0000","last_became_peered":"2023-10-19T12:45:40.359156+0000","last_unstale":"2023-10-19T12:45:40.364011+0000","last_undegraded":"2023-10-19T12:45:40.364011+0000","last_fullsized":"2023-10-19T12:45:40.364011+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:10:03.197155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2023-10-19T12:45:40.344906+0000","last_change":"2023-10-19T12:45:40.344906+0000","last_active":"2023-10-19T12:45:40.327526+0000","last_peered":"2023-10-19T12:45:40.327526+0000","last_clean":"2023-10-19T12:45:40.327526+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:45:40.344906+0000","last_undegraded":"2023-10-19T12:45:40.344906+0000","last_fullsized":"2023-10-19T12:45:40.344906+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":11,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.351976+0000","last_change":"2023-10-19T12:45:40.359310+0000","last_active":"2023-10-19T12:45:41.351976+0000","last_peered":"2023-10-19T12:45:41.351976+0000","last_clean":"2023-10-19T12:45:41.351976+0000","last_became_active":"2023-10-19T12:45:40.359010+0000","last_became_peered":"2023-10-19T12:45:40.359010+0000","last_unstale":"2023-10-19T12:45:41.351976+0000","last_undegraded":"2023-10-19T12:45:41.351976+0000","last_fullsized":"2023-10-19T12:45:41.351976+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:57:29.725854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":179,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-19T12:45:40.349832+0000","last_change":"2023-10-19T12:44:08.636613+0000","last_active":"2023-10-19T12:45:40.349832+0000","last_peered":"2023-10-19T12:45:40.349832+0000","last_clean":"2023-10-19T12:45:40.349832+0000","last_became_active":"2023-10-19T12:44:08.635697+0000","last_became_peered":"2023-10-19T12:44:08.635697+0000","last_unstale":"2023-10-19T12:45:40.349832+0000","last_undegraded":"2023-10-19T12:45:40.349832+0000","last_fullsized":"2023-10-19T12:45:40.349832+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_clean_scrub_stamp":"2023-10-19T12:44:07.399738+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:15:23.685485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":70,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574871,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28028,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343812,"statfs":{"total":96636764160,"available":96608063488,"internally_reserved":0,"allocated":1032192,"data_stored":777587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:45:08 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.53400000000000003}]},{"osd":1,"last update":"Thu Oct 19 12:45:08 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.495}]},{"osd":2,"last update":"Thu Oct 19 12:45:08 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.64300000000000002}]},{"osd":3,"last update":"Thu Oct 19 12:45:08 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.68200000000000005}]},{"osd":4,"last update":"Thu Oct 19 12:45:08 2023","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.51200000000000001}]},{"osd":5,"last update":"Thu Oct 19 12:45:08 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.441}]},{"osd":6,"last update":"Thu Oct 19 12:45:08 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.61199999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574871,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27436,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344404,"statfs":{"total":96636764160,"available":96608669696,"internally_reserved":0,"allocated":425984,"data_stored":180124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:45:06 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.56599999999999995}]},{"osd":1,"last update":"Thu Oct 19 12:45:06 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.626},{"interface":"front","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.075,"5min":1.075,"15min":1.075},"last":0.50600000000000001}]},{"osd":2,"last update":"Thu Oct 19 12:45:06 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.59199999999999997}]},{"osd":3,"last update":"Thu Oct 19 12:45:06 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.57999999999999996}]},{"osd":4,"last update":"Thu Oct 19 12:45:06 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.371},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.32900000000000001}]},{"osd":5,"last update":"Thu Oct 19 12:45:06 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.44700000000000001}]},{"osd":7,"last update":"Thu Oct 19 12:45:06 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.48499999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574871,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27436,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344404,"statfs":{"total":96636764160,"available":96608669696,"internally_reserved":0,"allocated":425984,"data_stored":180124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.55000000000000004}]},{"osd":1,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.57999999999999996}]},{"osd":2,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.625},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.60899999999999999}]},{"osd":3,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.65400000000000003}]},{"osd":4,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.42599999999999999}]},{"osd":6,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.59199999999999997}]},{"osd":7,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.52900000000000003}]}]},{"osd":4,"up_from":13,"seq":55834574871,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27436,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344404,"statfs":{"total":96636764160,"available":96608669696,"internally_reserved":0,"allocated":425984,"data_stored":180124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:45:08 2023","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.46000000000000002}]},{"osd":1,"last update":"Thu Oct 19 12:45:08 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.57699999999999996}]},{"osd":2,"last update":"Thu Oct 19 12:45:08 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.44600000000000001}]},{"osd":3,"last update":"Thu Oct 19 12:45:08 2023","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.629}]},{"osd":5,"last update":"Thu Oct 19 12:45:08 2023","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.29099999999999998}]},{"osd":6,"last update":"Thu Oct 19 12:45:08 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.32600000000000001}]},{"osd":7,"last update":"Thu Oct 19 12:45:08 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.373},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.40999999999999998}]}]},{"osd":3,"up_from":13,"seq":55834574870,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31596,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340244,"statfs":{"total":96636764160,"available":96604409856,"internally_reserved":0,"allocated":425984,"data_stored":180124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":1.04,"5min":1.04,"15min":1.04},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.375,"5min":0.375,"15min":0.375},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"last":0.32500000000000001}]},{"osd":1,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.371},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.39700000000000002}]},{"osd":2,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.42699999999999999}]},{"osd":4,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.65600000000000003}]},{"osd":5,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.53400000000000003}]},{"osd":6,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.51700000000000002}]},{"osd":7,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.61899999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574872,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31596,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340244,"statfs":{"total":96636764160,"available":96604409856,"internally_reserved":0,"allocated":425984,"data_stored":180124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.314},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"last":0.251}]},{"osd":1,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.35399999999999998}]},{"osd":3,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.46800000000000003}]},{"osd":4,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.503},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.53800000000000003}]},{"osd":5,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.52500000000000002}]},{"osd":6,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.58899999999999997}]},{"osd":7,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.111,"5min":1.111,"15min":1.111},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.018,"5min":1.018,"15min":1.018},"last":0.55500000000000005}]}]},{"osd":1,"up_from":13,"seq":55834574871,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31596,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340244,"statfs":{"total":96636764160,"available":96604409856,"internally_reserved":0,"allocated":425984,"data_stored":180124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:45:05 2023","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.29699999999999999}]},{"osd":2,"last update":"Thu Oct 19 12:45:05 2023","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.441},{"interface":"front","average":{"1min":0.433,"5min":0.433,"15min":0.433},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.38}]},{"osd":3,"last update":"Thu Oct 19 12:45:05 2023","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.36399999999999999}]},{"osd":4,"last update":"Thu Oct 19 12:45:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.1910000000000001,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.314}]},{"osd":5,"last update":"Thu Oct 19 12:45:05 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.117,"5min":1.117,"15min":1.117},"last":0.41599999999999998}]},{"osd":6,"last update":"Thu Oct 19 12:45:05 2023","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.077,"5min":1.077,"15min":1.077},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.51400000000000001}]},{"osd":7,"last update":"Thu Oct 19 12:45:05 2023","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.45700000000000002}]}]},{"osd":0,"up_from":13,"seq":55834574872,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32176,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339664,"statfs":{"total":96636764160,"available":96603815936,"internally_reserved":0,"allocated":1015808,"data_stored":769873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Oct 19 12:45:09 2023","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"last":0.246},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"last":0.32800000000000001}]},{"osd":2,"last update":"Thu Oct 19 12:45:09 2023","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.29999999999999999}]},{"osd":3,"last update":"Thu Oct 19 12:45:09 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"last":0.311},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.40999999999999998}]},{"osd":4,"last update":"Thu Oct 19 12:45:09 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.39800000000000002}]},{"osd":5,"last update":"Thu Oct 19 12:45:09 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.056,"5min":1.056,"15min":1.056},"last":0.49399999999999999}]},{"osd":6,"last update":"Thu Oct 19 12:45:09 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.109,"5min":1.109,"15min":1.109},"last":0.42099999999999999}]},{"osd":7,"last update":"Thu Oct 19 12:45:09 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.44500000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-19T12:45:46.012 INFO:tasks.ceph.mds.d.smithi179.stderr:2023-10-19T12:45:46.009+0000 7f74ceadc640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-19T12:45:48.833 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-19T12:45:48.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:48.952+0000 7f260e9fc640 1 -- 172.21.15.57:0/1167297155 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25f40039a0 msgr2=0x7f25f4003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:48.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:48.952+0000 7f260e9fc640 1 --2- 172.21.15.57:0/1167297155 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25f40039a0 0x7f25f4003dc0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f25f0001680 tx=0x7f25f40057f0 comp rx=0 tx=0).stop 2023-10-19T12:45:48.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:48.952+0000 7f260cf99640 1 -- 172.21.15.57:0/1167297155 reap_dead start 2023-10-19T12:45:48.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:48.952+0000 7f260e9fc640 1 -- 172.21.15.57:0/1167297155 shutdown_connections 2023-10-19T12:45:48.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:48.952+0000 7f260e9fc640 1 -- 172.21.15.57:0/1167297155 >> 172.21.15.57:0/1167297155 conn(0x7f26080d0d60 msgr2=0x7f26080d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:48.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:48.952+0000 7f260e9fc640 1 -- 172.21.15.57:0/1167297155 shutdown_connections 2023-10-19T12:45:48.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:48.952+0000 7f260e9fc640 1 -- 172.21.15.57:0/1167297155 wait complete. 2023-10-19T12:45:48.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:48.953+0000 7f260e9fc640 1 Processor -- start 2023-10-19T12:45:48.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:48.953+0000 7f260e9fc640 1 -- start start 2023-10-19T12:45:48.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:48.954+0000 7f260e9fc640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f25f40039a0 0x7f26080c8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:48.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:48.954+0000 7f260e9fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f26080b10c0 0x7f260815d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:48.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:48.954+0000 7f260e9fc640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f26080b1540 0x7f260815dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:48.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:48.954+0000 7f260e9fc640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26080c8690 con 0x7f26080b10c0 2023-10-19T12:45:48.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:48.954+0000 7f260e9fc640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26080c8ff0 con 0x7f25f40039a0 2023-10-19T12:45:48.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:48.954+0000 7f260e9fc640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f260815e160 con 0x7f26080b1540 2023-10-19T12:45:48.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:48.954+0000 7f26077fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f26080b10c0 0x7f260815d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:48.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:48.954+0000 7f26077fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f26080b10c0 0x7f260815d6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37468/0 (socket says 172.21.15.57:37468) 2023-10-19T12:45:48.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:48.954+0000 7f26077fe640 1 -- 172.21.15.57:0/2936659653 learned_addr learned my addr 172.21.15.57:0/2936659653 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:48.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:48.954+0000 7f2607fff640 1 --2- 172.21.15.57:0/2936659653 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f25f40039a0 0x7f26080c8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:48.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:48.954+0000 7f260cf99640 1 --2- 172.21.15.57:0/2936659653 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f26080b1540 0x7f260815dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:48.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:48.955+0000 7f26077fe640 1 -- 172.21.15.57:0/2936659653 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f26080b1540 msgr2=0x7f260815dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:48.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:48.955+0000 7f26077fe640 1 --2- 172.21.15.57:0/2936659653 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f26080b1540 0x7f260815dc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:48.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:48.955+0000 7f26077fe640 1 -- 172.21.15.57:0/2936659653 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f25f40039a0 msgr2=0x7f26080c8ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:48.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:48.955+0000 7f26077fe640 1 --2- 172.21.15.57:0/2936659653 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f25f40039a0 0x7f26080c8ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:48.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:48.955+0000 7f26077fe640 1 -- 172.21.15.57:0/2936659653 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f260815e410 con 0x7f26080b10c0 2023-10-19T12:45:48.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:48.955+0000 7f2607fff640 1 --2- 172.21.15.57:0/2936659653 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f25f40039a0 0x7f26080c8ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:45:48.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:48.955+0000 7f260cf99640 1 --2- 172.21.15.57:0/2936659653 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f26080b1540 0x7f260815dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:45:48.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:48.955+0000 7f26077fe640 1 --2- 172.21.15.57:0/2936659653 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f26080b10c0 0x7f260815d6e0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f25f800e630 tx=0x7f25f800eaf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:48.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:48.955+0000 7f2604ff9640 1 -- 172.21.15.57:0/2936659653 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25f800c940 con 0x7f26080b10c0 2023-10-19T12:45:48.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:48.955+0000 7f2604ff9640 1 -- 172.21.15.57:0/2936659653 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f25f800cbb0 con 0x7f26080b10c0 2023-10-19T12:45:48.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:48.955+0000 7f2604ff9640 1 -- 172.21.15.57:0/2936659653 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25f8011460 con 0x7f26080b10c0 2023-10-19T12:45:48.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:48.955+0000 7f260e9fc640 1 -- 172.21.15.57:0/2936659653 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26080c7fd0 con 0x7f26080b10c0 2023-10-19T12:45:48.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:48.955+0000 7f260e9fc640 1 -- 172.21.15.57:0/2936659653 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f260814d9e0 con 0x7f26080b10c0 2023-10-19T12:45:48.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:48.956+0000 7f260e9fc640 1 -- 172.21.15.57:0/2936659653 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26080c7e10 con 0x7f26080b10c0 2023-10-19T12:45:48.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:48.956+0000 7f2604ff9640 1 -- 172.21.15.57:0/2936659653 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f25f80115c0 con 0x7f26080b10c0 2023-10-19T12:45:48.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:48.957+0000 7f2604ff9640 1 --2- 172.21.15.57:0/2936659653 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f25e806ae40 0x7f25e806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:48.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:48.957+0000 7f2604ff9640 1 -- 172.21.15.57:0/2936659653 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f25f808a150 con 0x7f26080b10c0 2023-10-19T12:45:48.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:48.957+0000 7f2607fff640 1 --2- 172.21.15.57:0/2936659653 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f25e806ae40 0x7f25e806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:48.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:48.958+0000 7f2607fff640 1 --2- 172.21.15.57:0/2936659653 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f25e806ae40 0x7f25e806d300 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f25f0034000 tx=0x7f25f0039040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:48.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:48.961+0000 7f2604ff9640 1 -- 172.21.15.57:0/2936659653 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f25f80550c0 con 0x7f26080b10c0 2023-10-19T12:45:49.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.138+0000 7f260e9fc640 1 -- 172.21.15.57:0/2936659653 --> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f26080690e0 con 0x7f25e806ae40 2023-10-19T12:45:49.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.146+0000 7f2604ff9640 1 -- 172.21.15.57:0/2936659653 <== mgr.4100 v2:172.21.15.179:6832/2609646285 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289122 (secure 0 0 0) 0x7f26080690e0 con 0x7f25e806ae40 2023-10-19T12:45:49.151 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:45:49.153 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2023-10-19T12:45:49.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.148+0000 7f260e9fc640 1 -- 172.21.15.57:0/2936659653 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f25e806ae40 msgr2=0x7f25e806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:49.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.148+0000 7f260e9fc640 1 --2- 172.21.15.57:0/2936659653 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f25e806ae40 0x7f25e806d300 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f25f0034000 tx=0x7f25f0039040 comp rx=0 tx=0).stop 2023-10-19T12:45:49.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.148+0000 7f260e9fc640 1 -- 172.21.15.57:0/2936659653 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f26080b10c0 msgr2=0x7f260815d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:49.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.148+0000 7f260e9fc640 1 --2- 172.21.15.57:0/2936659653 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f26080b10c0 0x7f260815d6e0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f25f800e630 tx=0x7f25f800eaf0 comp rx=0 tx=0).stop 2023-10-19T12:45:49.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.148+0000 7f260e9fc640 1 -- 172.21.15.57:0/2936659653 shutdown_connections 2023-10-19T12:45:49.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.148+0000 7f260e9fc640 1 --2- 172.21.15.57:0/2936659653 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f25e806ae40 0x7f25e806d300 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:49.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.148+0000 7f260e9fc640 1 --2- 172.21.15.57:0/2936659653 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f26080b1540 0x7f260815dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:49.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.148+0000 7f260e9fc640 1 --2- 172.21.15.57:0/2936659653 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f26080b10c0 0x7f260815d6e0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:49.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.149+0000 7f260e9fc640 1 --2- 172.21.15.57:0/2936659653 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f25f40039a0 0x7f26080c8ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:49.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.149+0000 7f260e9fc640 1 -- 172.21.15.57:0/2936659653 >> 172.21.15.57:0/2936659653 conn(0x7f26080d0d60 msgr2=0x7f26080b2060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:49.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.149+0000 7f260e9fc640 1 -- 172.21.15.57:0/2936659653 shutdown_connections 2023-10-19T12:45:49.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.149+0000 7f260e9fc640 1 -- 172.21.15.57:0/2936659653 wait complete. 2023-10-19T12:45:49.162 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":83,"stamp":"2023-10-19T12:45:47.337335+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":237580,"kb_used_data":4740,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754737140,"statfs":{"total":773094113280,"available":772850831360,"internally_reserved":0,"allocated":4853760,"data_stored":2735649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":238276153},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.995109"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-19T12:45:42.360519+0000","last_change":"2023-10-19T12:45:41.367170+0000","last_active":"2023-10-19T12:45:42.360519+0000","last_peered":"2023-10-19T12:45:42.360519+0000","last_clean":"2023-10-19T12:45:42.360519+0000","last_became_active":"2023-10-19T12:45:41.366893+0000","last_became_peered":"2023-10-19T12:45:41.366893+0000","last_unstale":"2023-10-19T12:45:42.360519+0000","last_undegraded":"2023-10-19T12:45:42.360519+0000","last_fullsized":"2023-10-19T12:45:42.360519+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:41:53.768379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-19T12:45:42.360171+0000","last_change":"2023-10-19T12:45:41.368553+0000","last_active":"2023-10-19T12:45:42.360171+0000","last_peered":"2023-10-19T12:45:42.360171+0000","last_clean":"2023-10-19T12:45:42.360171+0000","last_became_active":"2023-10-19T12:45:41.368245+0000","last_became_peered":"2023-10-19T12:45:41.368245+0000","last_unstale":"2023-10-19T12:45:42.360171+0000","last_undegraded":"2023-10-19T12:45:42.360171+0000","last_fullsized":"2023-10-19T12:45:42.360171+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:19:39.712368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-19T12:45:42.866255+0000","last_change":"2023-10-19T12:45:41.371589+0000","last_active":"2023-10-19T12:45:42.866255+0000","last_peered":"2023-10-19T12:45:42.866255+0000","last_clean":"2023-10-19T12:45:42.866255+0000","last_became_active":"2023-10-19T12:45:41.371287+0000","last_became_peered":"2023-10-19T12:45:41.371287+0000","last_unstale":"2023-10-19T12:45:42.866255+0000","last_undegraded":"2023-10-19T12:45:42.866255+0000","last_fullsized":"2023-10-19T12:45:42.866255+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:40:51.330048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-19T12:45:42.865891+0000","last_change":"2023-10-19T12:45:41.363930+0000","last_active":"2023-10-19T12:45:42.865891+0000","last_peered":"2023-10-19T12:45:42.865891+0000","last_clean":"2023-10-19T12:45:42.865891+0000","last_became_active":"2023-10-19T12:45:41.363668+0000","last_became_peered":"2023-10-19T12:45:41.363668+0000","last_unstale":"2023-10-19T12:45:42.865891+0000","last_undegraded":"2023-10-19T12:45:42.865891+0000","last_fullsized":"2023-10-19T12:45:42.865891+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:39:10.778002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-19T12:45:42.865715+0000","last_change":"2023-10-19T12:45:41.363093+0000","last_active":"2023-10-19T12:45:42.865715+0000","last_peered":"2023-10-19T12:45:42.865715+0000","last_clean":"2023-10-19T12:45:42.865715+0000","last_became_active":"2023-10-19T12:45:41.362804+0000","last_became_peered":"2023-10-19T12:45:41.362804+0000","last_unstale":"2023-10-19T12:45:42.865715+0000","last_undegraded":"2023-10-19T12:45:42.865715+0000","last_fullsized":"2023-10-19T12:45:42.865715+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:53:30.629136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-19T12:45:42.359512+0000","last_change":"2023-10-19T12:45:41.359415+0000","last_active":"2023-10-19T12:45:42.359512+0000","last_peered":"2023-10-19T12:45:42.359512+0000","last_clean":"2023-10-19T12:45:42.359512+0000","last_became_active":"2023-10-19T12:45:41.359120+0000","last_became_peered":"2023-10-19T12:45:41.359120+0000","last_unstale":"2023-10-19T12:45:42.359512+0000","last_undegraded":"2023-10-19T12:45:42.359512+0000","last_fullsized":"2023-10-19T12:45:42.359512+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:13:18.501715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-19T12:45:42.360572+0000","last_change":"2023-10-19T12:45:41.376272+0000","last_active":"2023-10-19T12:45:42.360572+0000","last_peered":"2023-10-19T12:45:42.360572+0000","last_clean":"2023-10-19T12:45:42.360572+0000","last_became_active":"2023-10-19T12:45:41.376016+0000","last_became_peered":"2023-10-19T12:45:41.376016+0000","last_unstale":"2023-10-19T12:45:42.360572+0000","last_undegraded":"2023-10-19T12:45:42.360572+0000","last_fullsized":"2023-10-19T12:45:42.360572+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:13:10.162844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-19T12:45:42.359890+0000","last_change":"2023-10-19T12:45:41.369506+0000","last_active":"2023-10-19T12:45:42.359890+0000","last_peered":"2023-10-19T12:45:42.359890+0000","last_clean":"2023-10-19T12:45:42.359890+0000","last_became_active":"2023-10-19T12:45:41.369167+0000","last_became_peered":"2023-10-19T12:45:41.369167+0000","last_unstale":"2023-10-19T12:45:42.359890+0000","last_undegraded":"2023-10-19T12:45:42.359890+0000","last_fullsized":"2023-10-19T12:45:42.359890+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:37:15.429962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-19T12:45:42.866179+0000","last_change":"2023-10-19T12:45:41.368731+0000","last_active":"2023-10-19T12:45:42.866179+0000","last_peered":"2023-10-19T12:45:42.866179+0000","last_clean":"2023-10-19T12:45:42.866179+0000","last_became_active":"2023-10-19T12:45:41.368006+0000","last_became_peered":"2023-10-19T12:45:41.368006+0000","last_unstale":"2023-10-19T12:45:42.866179+0000","last_undegraded":"2023-10-19T12:45:42.866179+0000","last_fullsized":"2023-10-19T12:45:42.866179+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:11:10.437258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-19T12:45:42.360248+0000","last_change":"2023-10-19T12:45:41.378317+0000","last_active":"2023-10-19T12:45:42.360248+0000","last_peered":"2023-10-19T12:45:42.360248+0000","last_clean":"2023-10-19T12:45:42.360248+0000","last_became_active":"2023-10-19T12:45:41.378045+0000","last_became_peered":"2023-10-19T12:45:41.378045+0000","last_unstale":"2023-10-19T12:45:42.360248+0000","last_undegraded":"2023-10-19T12:45:42.360248+0000","last_fullsized":"2023-10-19T12:45:42.360248+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:37:15.429962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-19T12:45:42.360378+0000","last_change":"2023-10-19T12:45:41.369546+0000","last_active":"2023-10-19T12:45:42.360378+0000","last_peered":"2023-10-19T12:45:42.360378+0000","last_clean":"2023-10-19T12:45:42.360378+0000","last_became_active":"2023-10-19T12:45:41.369297+0000","last_became_peered":"2023-10-19T12:45:41.369297+0000","last_unstale":"2023-10-19T12:45:42.360378+0000","last_undegraded":"2023-10-19T12:45:42.360378+0000","last_fullsized":"2023-10-19T12:45:42.360378+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:41:54.261976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-19T12:45:42.866236+0000","last_change":"2023-10-19T12:45:41.357975+0000","last_active":"2023-10-19T12:45:42.866236+0000","last_peered":"2023-10-19T12:45:42.866236+0000","last_clean":"2023-10-19T12:45:42.866236+0000","last_became_active":"2023-10-19T12:45:41.357717+0000","last_became_peered":"2023-10-19T12:45:41.357717+0000","last_unstale":"2023-10-19T12:45:42.866236+0000","last_undegraded":"2023-10-19T12:45:42.866236+0000","last_fullsized":"2023-10-19T12:45:42.866236+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:45:47.252852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-19T12:45:42.360301+0000","last_change":"2023-10-19T12:45:41.378467+0000","last_active":"2023-10-19T12:45:42.360301+0000","last_peered":"2023-10-19T12:45:42.360301+0000","last_clean":"2023-10-19T12:45:42.360301+0000","last_became_active":"2023-10-19T12:45:41.378061+0000","last_became_peered":"2023-10-19T12:45:41.378061+0000","last_unstale":"2023-10-19T12:45:42.360301+0000","last_undegraded":"2023-10-19T12:45:42.360301+0000","last_fullsized":"2023-10-19T12:45:42.360301+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:19:32.104724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-19T12:45:42.865688+0000","last_change":"2023-10-19T12:45:41.367824+0000","last_active":"2023-10-19T12:45:42.865688+0000","last_peered":"2023-10-19T12:45:42.865688+0000","last_clean":"2023-10-19T12:45:42.865688+0000","last_became_active":"2023-10-19T12:45:41.367471+0000","last_became_peered":"2023-10-19T12:45:41.367471+0000","last_unstale":"2023-10-19T12:45:42.865688+0000","last_undegraded":"2023-10-19T12:45:42.865688+0000","last_fullsized":"2023-10-19T12:45:42.865688+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:38:37.793736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-19T12:45:42.359957+0000","last_change":"2023-10-19T12:45:41.363070+0000","last_active":"2023-10-19T12:45:42.359957+0000","last_peered":"2023-10-19T12:45:42.359957+0000","last_clean":"2023-10-19T12:45:42.359957+0000","last_became_active":"2023-10-19T12:45:41.362799+0000","last_became_peered":"2023-10-19T12:45:41.362799+0000","last_unstale":"2023-10-19T12:45:42.359957+0000","last_undegraded":"2023-10-19T12:45:42.359957+0000","last_fullsized":"2023-10-19T12:45:42.359957+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:34:52.376596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-19T12:45:42.865839+0000","last_change":"2023-10-19T12:45:41.376538+0000","last_active":"2023-10-19T12:45:42.865839+0000","last_peered":"2023-10-19T12:45:42.865839+0000","last_clean":"2023-10-19T12:45:42.865839+0000","last_became_active":"2023-10-19T12:45:41.376269+0000","last_became_peered":"2023-10-19T12:45:41.376269+0000","last_unstale":"2023-10-19T12:45:42.865839+0000","last_undegraded":"2023-10-19T12:45:42.865839+0000","last_fullsized":"2023-10-19T12:45:42.865839+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:40:51.330048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-19T12:45:42.866667+0000","last_change":"2023-10-19T12:45:41.372627+0000","last_active":"2023-10-19T12:45:42.866667+0000","last_peered":"2023-10-19T12:45:42.866667+0000","last_clean":"2023-10-19T12:45:42.866667+0000","last_became_active":"2023-10-19T12:45:41.371596+0000","last_became_peered":"2023-10-19T12:45:41.371596+0000","last_unstale":"2023-10-19T12:45:42.866667+0000","last_undegraded":"2023-10-19T12:45:42.866667+0000","last_fullsized":"2023-10-19T12:45:42.866667+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:19:32.104724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-19T12:45:42.804721+0000","last_change":"2023-10-19T12:45:41.368406+0000","last_active":"2023-10-19T12:45:42.804721+0000","last_peered":"2023-10-19T12:45:42.804721+0000","last_clean":"2023-10-19T12:45:42.804721+0000","last_became_active":"2023-10-19T12:45:41.367955+0000","last_became_peered":"2023-10-19T12:45:41.367955+0000","last_unstale":"2023-10-19T12:45:42.804721+0000","last_undegraded":"2023-10-19T12:45:42.804721+0000","last_fullsized":"2023-10-19T12:45:42.804721+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:57:30.732319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-19T12:45:42.865826+0000","last_change":"2023-10-19T12:45:41.356683+0000","last_active":"2023-10-19T12:45:42.865826+0000","last_peered":"2023-10-19T12:45:42.865826+0000","last_clean":"2023-10-19T12:45:42.865826+0000","last_became_active":"2023-10-19T12:45:41.356404+0000","last_became_peered":"2023-10-19T12:45:41.356404+0000","last_unstale":"2023-10-19T12:45:42.865826+0000","last_undegraded":"2023-10-19T12:45:42.865826+0000","last_fullsized":"2023-10-19T12:45:42.865826+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:36:07.043432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-19T12:45:42.804466+0000","last_change":"2023-10-19T12:45:41.366830+0000","last_active":"2023-10-19T12:45:42.804466+0000","last_peered":"2023-10-19T12:45:42.804466+0000","last_clean":"2023-10-19T12:45:42.804466+0000","last_became_active":"2023-10-19T12:45:41.366563+0000","last_became_peered":"2023-10-19T12:45:41.366563+0000","last_unstale":"2023-10-19T12:45:42.804466+0000","last_undegraded":"2023-10-19T12:45:42.804466+0000","last_fullsized":"2023-10-19T12:45:42.804466+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:44:04.457483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-19T12:45:42.360026+0000","last_change":"2023-10-19T12:45:41.367405+0000","last_active":"2023-10-19T12:45:42.360026+0000","last_peered":"2023-10-19T12:45:42.360026+0000","last_clean":"2023-10-19T12:45:42.360026+0000","last_became_active":"2023-10-19T12:45:41.367134+0000","last_became_peered":"2023-10-19T12:45:41.367134+0000","last_unstale":"2023-10-19T12:45:42.360026+0000","last_undegraded":"2023-10-19T12:45:42.360026+0000","last_fullsized":"2023-10-19T12:45:42.360026+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:39:22.485898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-19T12:45:42.866070+0000","last_change":"2023-10-19T12:45:41.366616+0000","last_active":"2023-10-19T12:45:42.866070+0000","last_peered":"2023-10-19T12:45:42.866070+0000","last_clean":"2023-10-19T12:45:42.866070+0000","last_became_active":"2023-10-19T12:45:41.366342+0000","last_became_peered":"2023-10-19T12:45:41.366342+0000","last_unstale":"2023-10-19T12:45:42.866070+0000","last_undegraded":"2023-10-19T12:45:42.866070+0000","last_fullsized":"2023-10-19T12:45:42.866070+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:21:51.336716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-19T12:45:42.865942+0000","last_change":"2023-10-19T12:45:41.373500+0000","last_active":"2023-10-19T12:45:42.865942+0000","last_peered":"2023-10-19T12:45:42.865942+0000","last_clean":"2023-10-19T12:45:42.865942+0000","last_became_active":"2023-10-19T12:45:41.372751+0000","last_became_peered":"2023-10-19T12:45:41.372751+0000","last_unstale":"2023-10-19T12:45:42.865942+0000","last_undegraded":"2023-10-19T12:45:42.865942+0000","last_fullsized":"2023-10-19T12:45:42.865942+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:45:47.252852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-19T12:45:42.865614+0000","last_change":"2023-10-19T12:45:41.361356+0000","last_active":"2023-10-19T12:45:42.865614+0000","last_peered":"2023-10-19T12:45:42.865614+0000","last_clean":"2023-10-19T12:45:42.865614+0000","last_became_active":"2023-10-19T12:45:41.361082+0000","last_became_peered":"2023-10-19T12:45:41.361082+0000","last_unstale":"2023-10-19T12:45:42.865614+0000","last_undegraded":"2023-10-19T12:45:42.865614+0000","last_fullsized":"2023-10-19T12:45:42.865614+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T12:45:47.935169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-19T12:45:42.804703+0000","last_change":"2023-10-19T12:45:41.367622+0000","last_active":"2023-10-19T12:45:42.804703+0000","last_peered":"2023-10-19T12:45:42.804703+0000","last_clean":"2023-10-19T12:45:42.804703+0000","last_became_active":"2023-10-19T12:45:41.367300+0000","last_became_peered":"2023-10-19T12:45:41.367300+0000","last_unstale":"2023-10-19T12:45:42.804703+0000","last_undegraded":"2023-10-19T12:45:42.804703+0000","last_fullsized":"2023-10-19T12:45:42.804703+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:53:29.708835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-19T12:45:42.359506+0000","last_change":"2023-10-19T12:45:41.372032+0000","last_active":"2023-10-19T12:45:42.359506+0000","last_peered":"2023-10-19T12:45:42.359506+0000","last_clean":"2023-10-19T12:45:42.359506+0000","last_became_active":"2023-10-19T12:45:41.371582+0000","last_became_peered":"2023-10-19T12:45:41.371582+0000","last_unstale":"2023-10-19T12:45:42.359506+0000","last_undegraded":"2023-10-19T12:45:42.359506+0000","last_fullsized":"2023-10-19T12:45:42.359506+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:59:55.870354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-19T12:45:42.804659+0000","last_change":"2023-10-19T12:45:41.362389+0000","last_active":"2023-10-19T12:45:42.804659+0000","last_peered":"2023-10-19T12:45:42.804659+0000","last_clean":"2023-10-19T12:45:42.804659+0000","last_became_active":"2023-10-19T12:45:41.360676+0000","last_became_peered":"2023-10-19T12:45:41.360676+0000","last_unstale":"2023-10-19T12:45:42.804659+0000","last_undegraded":"2023-10-19T12:45:42.804659+0000","last_fullsized":"2023-10-19T12:45:42.804659+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:32:37.014614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-19T12:45:42.360405+0000","last_change":"2023-10-19T12:45:41.377661+0000","last_active":"2023-10-19T12:45:42.360405+0000","last_peered":"2023-10-19T12:45:42.360405+0000","last_clean":"2023-10-19T12:45:42.360405+0000","last_became_active":"2023-10-19T12:45:41.377389+0000","last_became_peered":"2023-10-19T12:45:41.377389+0000","last_unstale":"2023-10-19T12:45:42.360405+0000","last_undegraded":"2023-10-19T12:45:42.360405+0000","last_fullsized":"2023-10-19T12:45:42.360405+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:19:39.712368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-19T12:45:42.804478+0000","last_change":"2023-10-19T12:45:41.363236+0000","last_active":"2023-10-19T12:45:42.804478+0000","last_peered":"2023-10-19T12:45:42.804478+0000","last_clean":"2023-10-19T12:45:42.804478+0000","last_became_active":"2023-10-19T12:45:41.362830+0000","last_became_peered":"2023-10-19T12:45:41.362830+0000","last_unstale":"2023-10-19T12:45:42.804478+0000","last_undegraded":"2023-10-19T12:45:42.804478+0000","last_fullsized":"2023-10-19T12:45:42.804478+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:10:04.203620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-19T12:45:42.804503+0000","last_change":"2023-10-19T12:45:41.367639+0000","last_active":"2023-10-19T12:45:42.804503+0000","last_peered":"2023-10-19T12:45:42.804503+0000","last_clean":"2023-10-19T12:45:42.804503+0000","last_became_active":"2023-10-19T12:45:41.367369+0000","last_became_peered":"2023-10-19T12:45:41.367369+0000","last_unstale":"2023-10-19T12:45:42.804503+0000","last_undegraded":"2023-10-19T12:45:42.804503+0000","last_fullsized":"2023-10-19T12:45:42.804503+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:38:37.793736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-19T12:45:42.866097+0000","last_change":"2023-10-19T12:45:41.374298+0000","last_active":"2023-10-19T12:45:42.866097+0000","last_peered":"2023-10-19T12:45:42.866097+0000","last_clean":"2023-10-19T12:45:42.866097+0000","last_became_active":"2023-10-19T12:45:41.374019+0000","last_became_peered":"2023-10-19T12:45:41.374019+0000","last_unstale":"2023-10-19T12:45:42.866097+0000","last_undegraded":"2023-10-19T12:45:42.866097+0000","last_fullsized":"2023-10-19T12:45:42.866097+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:11:10.437258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-19T12:45:42.359318+0000","last_change":"2023-10-19T12:45:41.356009+0000","last_active":"2023-10-19T12:45:42.359318+0000","last_peered":"2023-10-19T12:45:42.359318+0000","last_clean":"2023-10-19T12:45:42.359318+0000","last_became_active":"2023-10-19T12:45:41.355714+0000","last_became_peered":"2023-10-19T12:45:41.355714+0000","last_unstale":"2023-10-19T12:45:42.359318+0000","last_undegraded":"2023-10-19T12:45:42.359318+0000","last_fullsized":"2023-10-19T12:45:42.359318+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:57:30.732319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-19T12:45:42.866662+0000","last_change":"2023-10-19T12:45:41.377029+0000","last_active":"2023-10-19T12:45:42.866662+0000","last_peered":"2023-10-19T12:45:42.866662+0000","last_clean":"2023-10-19T12:45:42.866662+0000","last_became_active":"2023-10-19T12:45:41.376028+0000","last_became_peered":"2023-10-19T12:45:41.376028+0000","last_unstale":"2023-10-19T12:45:42.866662+0000","last_undegraded":"2023-10-19T12:45:42.866662+0000","last_fullsized":"2023-10-19T12:45:42.866662+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:37:15.429962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"38'4","reported_seq":16,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.375457+0000","last_change":"2023-10-19T12:45:40.359697+0000","last_active":"2023-10-19T12:45:41.375457+0000","last_peered":"2023-10-19T12:45:41.375457+0000","last_clean":"2023-10-19T12:45:41.375457+0000","last_became_active":"2023-10-19T12:45:40.359413+0000","last_became_peered":"2023-10-19T12:45:40.359413+0000","last_unstale":"2023-10-19T12:45:41.375457+0000","last_undegraded":"2023-10-19T12:45:41.375457+0000","last_fullsized":"2023-10-19T12:45:41.375457+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:28:59.279648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-19T12:45:42.359644+0000","last_change":"2023-10-19T12:45:41.372556+0000","last_active":"2023-10-19T12:45:42.359644+0000","last_peered":"2023-10-19T12:45:42.359644+0000","last_clean":"2023-10-19T12:45:42.359644+0000","last_became_active":"2023-10-19T12:45:41.372184+0000","last_became_peered":"2023-10-19T12:45:41.372184+0000","last_unstale":"2023-10-19T12:45:42.359644+0000","last_undegraded":"2023-10-19T12:45:42.359644+0000","last_fullsized":"2023-10-19T12:45:42.359644+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:45:47.252852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"38'1","reported_seq":13,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.384119+0000","last_change":"2023-10-19T12:45:40.352581+0000","last_active":"2023-10-19T12:45:41.384119+0000","last_peered":"2023-10-19T12:45:41.384119+0000","last_clean":"2023-10-19T12:45:41.384119+0000","last_became_active":"2023-10-19T12:45:40.352300+0000","last_became_peered":"2023-10-19T12:45:40.352300+0000","last_unstale":"2023-10-19T12:45:41.384119+0000","last_undegraded":"2023-10-19T12:45:41.384119+0000","last_fullsized":"2023-10-19T12:45:41.384119+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:59:58.100040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-19T12:45:42.866389+0000","last_change":"2023-10-19T12:45:41.371870+0000","last_active":"2023-10-19T12:45:42.866389+0000","last_peered":"2023-10-19T12:45:42.866389+0000","last_clean":"2023-10-19T12:45:42.866389+0000","last_became_active":"2023-10-19T12:45:41.371586+0000","last_became_peered":"2023-10-19T12:45:41.371586+0000","last_unstale":"2023-10-19T12:45:42.866389+0000","last_undegraded":"2023-10-19T12:45:42.866389+0000","last_fullsized":"2023-10-19T12:45:42.866389+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:59:55.870354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"38'2","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.384498+0000","last_change":"2023-10-19T12:45:40.355990+0000","last_active":"2023-10-19T12:45:41.384498+0000","last_peered":"2023-10-19T12:45:41.384498+0000","last_clean":"2023-10-19T12:45:41.384498+0000","last_became_active":"2023-10-19T12:45:40.355666+0000","last_became_peered":"2023-10-19T12:45:40.355666+0000","last_unstale":"2023-10-19T12:45:41.384498+0000","last_undegraded":"2023-10-19T12:45:41.384498+0000","last_fullsized":"2023-10-19T12:45:41.384498+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:53:28.702370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-19T12:45:42.360519+0000","last_change":"2023-10-19T12:45:41.362839+0000","last_active":"2023-10-19T12:45:42.360519+0000","last_peered":"2023-10-19T12:45:42.360519+0000","last_clean":"2023-10-19T12:45:42.360519+0000","last_became_active":"2023-10-19T12:45:41.362552+0000","last_became_peered":"2023-10-19T12:45:41.362552+0000","last_unstale":"2023-10-19T12:45:42.360519+0000","last_undegraded":"2023-10-19T12:45:42.360519+0000","last_fullsized":"2023-10-19T12:45:42.360519+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:03:18.069891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"38'2","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.376779+0000","last_change":"2023-10-19T12:45:40.364018+0000","last_active":"2023-10-19T12:45:41.376779+0000","last_peered":"2023-10-19T12:45:41.376779+0000","last_clean":"2023-10-19T12:45:41.376779+0000","last_became_active":"2023-10-19T12:45:40.363763+0000","last_became_peered":"2023-10-19T12:45:40.363763+0000","last_unstale":"2023-10-19T12:45:41.376779+0000","last_undegraded":"2023-10-19T12:45:41.376779+0000","last_fullsized":"2023-10-19T12:45:41.376779+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:28:59.279648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-19T12:45:42.866033+0000","last_change":"2023-10-19T12:45:41.355176+0000","last_active":"2023-10-19T12:45:42.866033+0000","last_peered":"2023-10-19T12:45:42.866033+0000","last_clean":"2023-10-19T12:45:42.866033+0000","last_became_active":"2023-10-19T12:45:41.354687+0000","last_became_peered":"2023-10-19T12:45:41.354687+0000","last_unstale":"2023-10-19T12:45:42.866033+0000","last_undegraded":"2023-10-19T12:45:42.866033+0000","last_fullsized":"2023-10-19T12:45:42.866033+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:57:30.732319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"38'1","reported_seq":13,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.372572+0000","last_change":"2023-10-19T12:45:40.357988+0000","last_active":"2023-10-19T12:45:41.372572+0000","last_peered":"2023-10-19T12:45:41.372572+0000","last_clean":"2023-10-19T12:45:41.372572+0000","last_became_active":"2023-10-19T12:45:40.357478+0000","last_became_peered":"2023-10-19T12:45:40.357478+0000","last_unstale":"2023-10-19T12:45:41.372572+0000","last_undegraded":"2023-10-19T12:45:41.372572+0000","last_fullsized":"2023-10-19T12:45:41.372572+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:45:46.246387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-19T12:45:42.360444+0000","last_change":"2023-10-19T12:45:41.363659+0000","last_active":"2023-10-19T12:45:42.360444+0000","last_peered":"2023-10-19T12:45:42.360444+0000","last_clean":"2023-10-19T12:45:42.360444+0000","last_became_active":"2023-10-19T12:45:41.363342+0000","last_became_peered":"2023-10-19T12:45:41.363342+0000","last_unstale":"2023-10-19T12:45:42.360444+0000","last_undegraded":"2023-10-19T12:45:42.360444+0000","last_fullsized":"2023-10-19T12:45:42.360444+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:41:53.768379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"38'1","reported_seq":13,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.365026+0000","last_change":"2023-10-19T12:45:40.355247+0000","last_active":"2023-10-19T12:45:41.365026+0000","last_peered":"2023-10-19T12:45:41.365026+0000","last_clean":"2023-10-19T12:45:41.365026+0000","last_became_active":"2023-10-19T12:45:40.354967+0000","last_became_peered":"2023-10-19T12:45:40.354967+0000","last_unstale":"2023-10-19T12:45:41.365026+0000","last_undegraded":"2023-10-19T12:45:41.365026+0000","last_fullsized":"2023-10-19T12:45:41.365026+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:48:24.212150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-19T12:45:42.865841+0000","last_change":"2023-10-19T12:45:41.371867+0000","last_active":"2023-10-19T12:45:42.865841+0000","last_peered":"2023-10-19T12:45:42.865841+0000","last_clean":"2023-10-19T12:45:42.865841+0000","last_became_active":"2023-10-19T12:45:41.371581+0000","last_became_peered":"2023-10-19T12:45:41.371581+0000","last_unstale":"2023-10-19T12:45:42.865841+0000","last_undegraded":"2023-10-19T12:45:42.865841+0000","last_fullsized":"2023-10-19T12:45:42.865841+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:32:37.014614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.384436+0000","last_change":"2023-10-19T12:45:40.359949+0000","last_active":"2023-10-19T12:45:41.384436+0000","last_peered":"2023-10-19T12:45:41.384436+0000","last_clean":"2023-10-19T12:45:41.384436+0000","last_became_active":"2023-10-19T12:45:40.359690+0000","last_became_peered":"2023-10-19T12:45:40.359690+0000","last_unstale":"2023-10-19T12:45:41.384436+0000","last_undegraded":"2023-10-19T12:45:41.384436+0000","last_fullsized":"2023-10-19T12:45:41.384436+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:44:03.451018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-19T12:45:42.360029+0000","last_change":"2023-10-19T12:45:41.363405+0000","last_active":"2023-10-19T12:45:42.360029+0000","last_peered":"2023-10-19T12:45:42.360029+0000","last_clean":"2023-10-19T12:45:42.360029+0000","last_became_active":"2023-10-19T12:45:41.361479+0000","last_became_peered":"2023-10-19T12:45:41.361479+0000","last_unstale":"2023-10-19T12:45:42.360029+0000","last_undegraded":"2023-10-19T12:45:42.360029+0000","last_fullsized":"2023-10-19T12:45:42.360029+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:19:32.104724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.352636+0000","last_change":"2023-10-19T12:45:40.360289+0000","last_active":"2023-10-19T12:45:41.352636+0000","last_peered":"2023-10-19T12:45:41.352636+0000","last_clean":"2023-10-19T12:45:41.352636+0000","last_became_active":"2023-10-19T12:45:40.359850+0000","last_became_peered":"2023-10-19T12:45:40.359850+0000","last_unstale":"2023-10-19T12:45:41.352636+0000","last_undegraded":"2023-10-19T12:45:41.352636+0000","last_fullsized":"2023-10-19T12:45:41.352636+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:16:36.413069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-19T12:45:42.865920+0000","last_change":"2023-10-19T12:45:41.363350+0000","last_active":"2023-10-19T12:45:42.865920+0000","last_peered":"2023-10-19T12:45:42.865920+0000","last_clean":"2023-10-19T12:45:42.865920+0000","last_became_active":"2023-10-19T12:45:41.363067+0000","last_became_peered":"2023-10-19T12:45:41.363067+0000","last_unstale":"2023-10-19T12:45:42.865920+0000","last_undegraded":"2023-10-19T12:45:42.865920+0000","last_fullsized":"2023-10-19T12:45:42.865920+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:49:21.932303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.355519+0000","last_change":"2023-10-19T12:45:40.356475+0000","last_active":"2023-10-19T12:45:41.355519+0000","last_peered":"2023-10-19T12:45:41.355519+0000","last_clean":"2023-10-19T12:45:41.355519+0000","last_became_active":"2023-10-19T12:45:40.356178+0000","last_became_peered":"2023-10-19T12:45:40.356178+0000","last_unstale":"2023-10-19T12:45:41.355519+0000","last_undegraded":"2023-10-19T12:45:41.355519+0000","last_fullsized":"2023-10-19T12:45:41.355519+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:37:43.719030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-19T12:45:42.866419+0000","last_change":"2023-10-19T12:45:41.371807+0000","last_active":"2023-10-19T12:45:42.866419+0000","last_peered":"2023-10-19T12:45:42.866419+0000","last_clean":"2023-10-19T12:45:42.866419+0000","last_became_active":"2023-10-19T12:45:41.371111+0000","last_became_peered":"2023-10-19T12:45:41.371111+0000","last_unstale":"2023-10-19T12:45:42.866419+0000","last_undegraded":"2023-10-19T12:45:42.866419+0000","last_fullsized":"2023-10-19T12:45:42.866419+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:37:44.725495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.358681+0000","last_change":"2023-10-19T12:45:40.360586+0000","last_active":"2023-10-19T12:45:41.358681+0000","last_peered":"2023-10-19T12:45:41.358681+0000","last_clean":"2023-10-19T12:45:41.358681+0000","last_became_active":"2023-10-19T12:45:40.360327+0000","last_became_peered":"2023-10-19T12:45:40.360327+0000","last_unstale":"2023-10-19T12:45:41.358681+0000","last_undegraded":"2023-10-19T12:45:41.358681+0000","last_fullsized":"2023-10-19T12:45:41.358681+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:11:09.430793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-19T12:45:42.866073+0000","last_change":"2023-10-19T12:45:41.366187+0000","last_active":"2023-10-19T12:45:42.866073+0000","last_peered":"2023-10-19T12:45:42.866073+0000","last_clean":"2023-10-19T12:45:42.866073+0000","last_became_active":"2023-10-19T12:45:41.365896+0000","last_became_peered":"2023-10-19T12:45:41.365896+0000","last_unstale":"2023-10-19T12:45:42.866073+0000","last_undegraded":"2023-10-19T12:45:42.866073+0000","last_fullsized":"2023-10-19T12:45:42.866073+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:21:51.336716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"38'3","reported_seq":15,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.378155+0000","last_change":"2023-10-19T12:45:40.358267+0000","last_active":"2023-10-19T12:45:41.378155+0000","last_peered":"2023-10-19T12:45:41.378155+0000","last_clean":"2023-10-19T12:45:41.378155+0000","last_became_active":"2023-10-19T12:45:40.358000+0000","last_became_peered":"2023-10-19T12:45:40.358000+0000","last_unstale":"2023-10-19T12:45:41.378155+0000","last_undegraded":"2023-10-19T12:45:41.378155+0000","last_fullsized":"2023-10-19T12:45:41.378155+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:37:43.719030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-19T12:45:42.866004+0000","last_change":"2023-10-19T12:45:41.366527+0000","last_active":"2023-10-19T12:45:42.866004+0000","last_peered":"2023-10-19T12:45:42.866004+0000","last_clean":"2023-10-19T12:45:42.866004+0000","last_became_active":"2023-10-19T12:45:41.366095+0000","last_became_peered":"2023-10-19T12:45:41.366095+0000","last_unstale":"2023-10-19T12:45:42.866004+0000","last_undegraded":"2023-10-19T12:45:42.866004+0000","last_fullsized":"2023-10-19T12:45:42.866004+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:57:30.732319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"38'6","reported_seq":18,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.385297+0000","last_change":"2023-10-19T12:45:40.361715+0000","last_active":"2023-10-19T12:45:41.385297+0000","last_peered":"2023-10-19T12:45:41.385297+0000","last_clean":"2023-10-19T12:45:41.385297+0000","last_became_active":"2023-10-19T12:45:40.361442+0000","last_became_peered":"2023-10-19T12:45:40.361442+0000","last_unstale":"2023-10-19T12:45:41.385297+0000","last_undegraded":"2023-10-19T12:45:41.385297+0000","last_fullsized":"2023-10-19T12:45:41.385297+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:39:21.479433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-19T12:45:42.866169+0000","last_change":"2023-10-19T12:45:41.364032+0000","last_active":"2023-10-19T12:45:42.866169+0000","last_peered":"2023-10-19T12:45:42.866169+0000","last_clean":"2023-10-19T12:45:42.866169+0000","last_became_active":"2023-10-19T12:45:41.363734+0000","last_became_peered":"2023-10-19T12:45:41.363734+0000","last_unstale":"2023-10-19T12:45:42.866169+0000","last_undegraded":"2023-10-19T12:45:42.866169+0000","last_fullsized":"2023-10-19T12:45:42.866169+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:59:55.870354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"38'2","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.376374+0000","last_change":"2023-10-19T12:45:40.360672+0000","last_active":"2023-10-19T12:45:41.376374+0000","last_peered":"2023-10-19T12:45:41.376374+0000","last_clean":"2023-10-19T12:45:41.376374+0000","last_became_active":"2023-10-19T12:45:40.360396+0000","last_became_peered":"2023-10-19T12:45:40.360396+0000","last_unstale":"2023-10-19T12:45:41.376374+0000","last_undegraded":"2023-10-19T12:45:41.376374+0000","last_fullsized":"2023-10-19T12:45:41.376374+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:33:56.743150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-19T12:45:42.360095+0000","last_change":"2023-10-19T12:45:41.364993+0000","last_active":"2023-10-19T12:45:42.360095+0000","last_peered":"2023-10-19T12:45:42.360095+0000","last_clean":"2023-10-19T12:45:42.360095+0000","last_became_active":"2023-10-19T12:45:41.364718+0000","last_became_peered":"2023-10-19T12:45:41.364718+0000","last_unstale":"2023-10-19T12:45:42.360095+0000","last_undegraded":"2023-10-19T12:45:42.360095+0000","last_fullsized":"2023-10-19T12:45:42.360095+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:30:21.861991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.384458+0000","last_change":"2023-10-19T12:45:40.359407+0000","last_active":"2023-10-19T12:45:41.384458+0000","last_peered":"2023-10-19T12:45:41.384458+0000","last_clean":"2023-10-19T12:45:41.384458+0000","last_became_active":"2023-10-19T12:45:40.359134+0000","last_became_peered":"2023-10-19T12:45:40.359134+0000","last_unstale":"2023-10-19T12:45:41.384458+0000","last_undegraded":"2023-10-19T12:45:41.384458+0000","last_fullsized":"2023-10-19T12:45:41.384458+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:59:58.100040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-19T12:45:42.866280+0000","last_change":"2023-10-19T12:45:41.372352+0000","last_active":"2023-10-19T12:45:42.866280+0000","last_peered":"2023-10-19T12:45:42.866280+0000","last_clean":"2023-10-19T12:45:42.866280+0000","last_became_active":"2023-10-19T12:45:41.372095+0000","last_became_peered":"2023-10-19T12:45:41.372095+0000","last_unstale":"2023-10-19T12:45:42.866280+0000","last_undegraded":"2023-10-19T12:45:42.866280+0000","last_fullsized":"2023-10-19T12:45:42.866280+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:33:57.749615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.352080+0000","last_change":"2023-10-19T12:45:40.359548+0000","last_active":"2023-10-19T12:45:41.352080+0000","last_peered":"2023-10-19T12:45:41.352080+0000","last_clean":"2023-10-19T12:45:41.352080+0000","last_became_active":"2023-10-19T12:45:40.359282+0000","last_became_peered":"2023-10-19T12:45:40.359282+0000","last_unstale":"2023-10-19T12:45:41.352080+0000","last_undegraded":"2023-10-19T12:45:41.352080+0000","last_fullsized":"2023-10-19T12:45:41.352080+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:39:09.771537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-19T12:45:42.360164+0000","last_change":"2023-10-19T12:45:41.367825+0000","last_active":"2023-10-19T12:45:42.360164+0000","last_peered":"2023-10-19T12:45:42.360164+0000","last_clean":"2023-10-19T12:45:42.360164+0000","last_became_active":"2023-10-19T12:45:41.367180+0000","last_became_peered":"2023-10-19T12:45:41.367180+0000","last_unstale":"2023-10-19T12:45:42.360164+0000","last_undegraded":"2023-10-19T12:45:42.360164+0000","last_fullsized":"2023-10-19T12:45:42.360164+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:41:54.261976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"38'8","reported_seq":20,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.368495+0000","last_change":"2023-10-19T12:45:40.358054+0000","last_active":"2023-10-19T12:45:41.368495+0000","last_peered":"2023-10-19T12:45:41.368495+0000","last_clean":"2023-10-19T12:45:41.368495+0000","last_became_active":"2023-10-19T12:45:40.357803+0000","last_became_peered":"2023-10-19T12:45:40.357803+0000","last_unstale":"2023-10-19T12:45:41.368495+0000","last_undegraded":"2023-10-19T12:45:41.368495+0000","last_fullsized":"2023-10-19T12:45:41.368495+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:07:55.739920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-19T12:45:42.360197+0000","last_change":"2023-10-19T12:45:41.370023+0000","last_active":"2023-10-19T12:45:42.360197+0000","last_peered":"2023-10-19T12:45:42.360197+0000","last_clean":"2023-10-19T12:45:42.360197+0000","last_became_active":"2023-10-19T12:45:41.369662+0000","last_became_peered":"2023-10-19T12:45:41.369662+0000","last_unstale":"2023-10-19T12:45:42.360197+0000","last_undegraded":"2023-10-19T12:45:42.360197+0000","last_fullsized":"2023-10-19T12:45:42.360197+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:32:20.977883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"38'1","reported_seq":13,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.369686+0000","last_change":"2023-10-19T12:45:40.358450+0000","last_active":"2023-10-19T12:45:41.369686+0000","last_peered":"2023-10-19T12:45:41.369686+0000","last_clean":"2023-10-19T12:45:41.369686+0000","last_became_active":"2023-10-19T12:45:40.356778+0000","last_became_peered":"2023-10-19T12:45:40.356778+0000","last_unstale":"2023-10-19T12:45:41.369686+0000","last_undegraded":"2023-10-19T12:45:41.369686+0000","last_fullsized":"2023-10-19T12:45:41.369686+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:07:55.739920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-19T12:45:42.360347+0000","last_change":"2023-10-19T12:45:41.371988+0000","last_active":"2023-10-19T12:45:42.360347+0000","last_peered":"2023-10-19T12:45:42.360347+0000","last_clean":"2023-10-19T12:45:42.360347+0000","last_became_active":"2023-10-19T12:45:41.371711+0000","last_became_peered":"2023-10-19T12:45:41.371711+0000","last_unstale":"2023-10-19T12:45:42.360347+0000","last_undegraded":"2023-10-19T12:45:42.360347+0000","last_fullsized":"2023-10-19T12:45:42.360347+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:30:21.861991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"38'2","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.370245+0000","last_change":"2023-10-19T12:45:40.357947+0000","last_active":"2023-10-19T12:45:41.370245+0000","last_peered":"2023-10-19T12:45:41.370245+0000","last_clean":"2023-10-19T12:45:41.370245+0000","last_became_active":"2023-10-19T12:45:40.357669+0000","last_became_peered":"2023-10-19T12:45:40.357669+0000","last_unstale":"2023-10-19T12:45:41.370245+0000","last_undegraded":"2023-10-19T12:45:41.370245+0000","last_fullsized":"2023-10-19T12:45:41.370245+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:10:03.197155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-19T12:45:42.866366+0000","last_change":"2023-10-19T12:45:41.371048+0000","last_active":"2023-10-19T12:45:42.866366+0000","last_peered":"2023-10-19T12:45:42.866366+0000","last_clean":"2023-10-19T12:45:42.866366+0000","last_became_active":"2023-10-19T12:45:41.370644+0000","last_became_peered":"2023-10-19T12:45:41.370644+0000","last_unstale":"2023-10-19T12:45:42.866366+0000","last_undegraded":"2023-10-19T12:45:42.866366+0000","last_fullsized":"2023-10-19T12:45:42.866366+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:07:56.746385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"38'1","reported_seq":13,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.383567+0000","last_change":"2023-10-19T12:45:40.354628+0000","last_active":"2023-10-19T12:45:41.383567+0000","last_peered":"2023-10-19T12:45:41.383567+0000","last_clean":"2023-10-19T12:45:41.383567+0000","last_became_active":"2023-10-19T12:45:40.354367+0000","last_became_peered":"2023-10-19T12:45:40.354367+0000","last_unstale":"2023-10-19T12:45:41.383567+0000","last_undegraded":"2023-10-19T12:45:41.383567+0000","last_fullsized":"2023-10-19T12:45:41.383567+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:33:49.817824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-19T12:45:42.865959+0000","last_change":"2023-10-19T12:45:41.367225+0000","last_active":"2023-10-19T12:45:42.865959+0000","last_peered":"2023-10-19T12:45:42.865959+0000","last_clean":"2023-10-19T12:45:42.865959+0000","last_became_active":"2023-10-19T12:45:41.366782+0000","last_became_peered":"2023-10-19T12:45:41.366782+0000","last_unstale":"2023-10-19T12:45:42.865959+0000","last_undegraded":"2023-10-19T12:45:42.865959+0000","last_fullsized":"2023-10-19T12:45:42.865959+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:16:37.419534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.384478+0000","last_change":"2023-10-19T12:45:40.357501+0000","last_active":"2023-10-19T12:45:41.384478+0000","last_peered":"2023-10-19T12:45:41.384478+0000","last_clean":"2023-10-19T12:45:41.384478+0000","last_became_active":"2023-10-19T12:45:40.355411+0000","last_became_peered":"2023-10-19T12:45:40.355411+0000","last_unstale":"2023-10-19T12:45:41.384478+0000","last_undegraded":"2023-10-19T12:45:41.384478+0000","last_fullsized":"2023-10-19T12:45:41.384478+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:32:36.008149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-19T12:45:42.865934+0000","last_change":"2023-10-19T12:45:41.373695+0000","last_active":"2023-10-19T12:45:42.865934+0000","last_peered":"2023-10-19T12:45:42.865934+0000","last_clean":"2023-10-19T12:45:42.865934+0000","last_became_active":"2023-10-19T12:45:41.373270+0000","last_became_peered":"2023-10-19T12:45:41.373270+0000","last_unstale":"2023-10-19T12:45:42.865934+0000","last_undegraded":"2023-10-19T12:45:42.865934+0000","last_fullsized":"2023-10-19T12:45:42.865934+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:13:18.501715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"38'2","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.383532+0000","last_change":"2023-10-19T12:45:40.357725+0000","last_active":"2023-10-19T12:45:41.383532+0000","last_peered":"2023-10-19T12:45:41.383532+0000","last_clean":"2023-10-19T12:45:41.383532+0000","last_became_active":"2023-10-19T12:45:40.357417+0000","last_became_peered":"2023-10-19T12:45:40.357417+0000","last_unstale":"2023-10-19T12:45:41.383532+0000","last_undegraded":"2023-10-19T12:45:41.383532+0000","last_fullsized":"2023-10-19T12:45:41.383532+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:26:34.780735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-19T12:45:42.866093+0000","last_change":"2023-10-19T12:45:41.366181+0000","last_active":"2023-10-19T12:45:42.866093+0000","last_peered":"2023-10-19T12:45:42.866093+0000","last_clean":"2023-10-19T12:45:42.866093+0000","last_became_active":"2023-10-19T12:45:41.365882+0000","last_became_peered":"2023-10-19T12:45:41.365882+0000","last_unstale":"2023-10-19T12:45:42.866093+0000","last_undegraded":"2023-10-19T12:45:42.866093+0000","last_fullsized":"2023-10-19T12:45:42.866093+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:07:56.746385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.383477+0000","last_change":"2023-10-19T12:45:40.355357+0000","last_active":"2023-10-19T12:45:41.383477+0000","last_peered":"2023-10-19T12:45:41.383477+0000","last_clean":"2023-10-19T12:45:41.383477+0000","last_became_active":"2023-10-19T12:45:40.355097+0000","last_became_peered":"2023-10-19T12:45:40.355097+0000","last_unstale":"2023-10-19T12:45:41.383477+0000","last_undegraded":"2023-10-19T12:45:41.383477+0000","last_fullsized":"2023-10-19T12:45:41.383477+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:23:29.827547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-19T12:45:42.360533+0000","last_change":"2023-10-19T12:45:41.370902+0000","last_active":"2023-10-19T12:45:42.360533+0000","last_peered":"2023-10-19T12:45:42.360533+0000","last_clean":"2023-10-19T12:45:42.360533+0000","last_became_active":"2023-10-19T12:45:41.370600+0000","last_became_peered":"2023-10-19T12:45:41.370600+0000","last_unstale":"2023-10-19T12:45:42.360533+0000","last_undegraded":"2023-10-19T12:45:42.360533+0000","last_fullsized":"2023-10-19T12:45:42.360533+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:32:20.977883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.355521+0000","last_change":"2023-10-19T12:45:40.360123+0000","last_active":"2023-10-19T12:45:41.355521+0000","last_peered":"2023-10-19T12:45:41.355521+0000","last_clean":"2023-10-19T12:45:41.355521+0000","last_became_active":"2023-10-19T12:45:40.359767+0000","last_became_peered":"2023-10-19T12:45:40.359767+0000","last_unstale":"2023-10-19T12:45:41.355521+0000","last_undegraded":"2023-10-19T12:45:41.355521+0000","last_fullsized":"2023-10-19T12:45:41.355521+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:38:36.787271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-19T12:45:42.360191+0000","last_change":"2023-10-19T12:45:41.359775+0000","last_active":"2023-10-19T12:45:42.360191+0000","last_peered":"2023-10-19T12:45:42.360191+0000","last_clean":"2023-10-19T12:45:42.360191+0000","last_became_active":"2023-10-19T12:45:41.359511+0000","last_became_peered":"2023-10-19T12:45:41.359511+0000","last_unstale":"2023-10-19T12:45:42.360191+0000","last_undegraded":"2023-10-19T12:45:42.360191+0000","last_fullsized":"2023-10-19T12:45:42.360191+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:38:12.163601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.383466+0000","last_change":"2023-10-19T12:45:40.357695+0000","last_active":"2023-10-19T12:45:41.383466+0000","last_peered":"2023-10-19T12:45:41.383466+0000","last_clean":"2023-10-19T12:45:41.383466+0000","last_became_active":"2023-10-19T12:45:40.357393+0000","last_became_peered":"2023-10-19T12:45:40.357393+0000","last_unstale":"2023-10-19T12:45:41.383466+0000","last_undegraded":"2023-10-19T12:45:41.383466+0000","last_fullsized":"2023-10-19T12:45:41.383466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:43:31.256604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-19T12:45:42.359352+0000","last_change":"2023-10-19T12:45:41.371258+0000","last_active":"2023-10-19T12:45:42.359352+0000","last_peered":"2023-10-19T12:45:42.359352+0000","last_clean":"2023-10-19T12:45:42.359352+0000","last_became_active":"2023-10-19T12:45:41.370963+0000","last_became_peered":"2023-10-19T12:45:41.370963+0000","last_unstale":"2023-10-19T12:45:42.359352+0000","last_undegraded":"2023-10-19T12:45:42.359352+0000","last_fullsized":"2023-10-19T12:45:42.359352+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:21:51.336716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"38'2","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.376962+0000","last_change":"2023-10-19T12:45:40.358263+0000","last_active":"2023-10-19T12:45:41.376962+0000","last_peered":"2023-10-19T12:45:41.376962+0000","last_clean":"2023-10-19T12:45:41.376962+0000","last_became_active":"2023-10-19T12:45:40.357402+0000","last_became_peered":"2023-10-19T12:45:40.357402+0000","last_unstale":"2023-10-19T12:45:41.376962+0000","last_undegraded":"2023-10-19T12:45:41.376962+0000","last_fullsized":"2023-10-19T12:45:41.376962+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:59:54.863889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-19T12:45:42.866529+0000","last_change":"2023-10-19T12:45:41.373152+0000","last_active":"2023-10-19T12:45:42.866529+0000","last_peered":"2023-10-19T12:45:42.866529+0000","last_clean":"2023-10-19T12:45:42.866529+0000","last_became_active":"2023-10-19T12:45:41.372895+0000","last_became_peered":"2023-10-19T12:45:41.372895+0000","last_unstale":"2023-10-19T12:45:42.866529+0000","last_undegraded":"2023-10-19T12:45:42.866529+0000","last_fullsized":"2023-10-19T12:45:42.866529+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:34:52.376596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"39'1","reported_seq":13,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-19T12:45:42.865758+0000","last_change":"2023-10-19T12:45:40.352686+0000","last_active":"2023-10-19T12:45:42.865758+0000","last_peered":"2023-10-19T12:45:42.865758+0000","last_clean":"2023-10-19T12:45:42.865758+0000","last_became_active":"2023-10-19T12:45:40.352420+0000","last_became_peered":"2023-10-19T12:45:40.352420+0000","last_unstale":"2023-10-19T12:45:42.865758+0000","last_undegraded":"2023-10-19T12:45:42.865758+0000","last_fullsized":"2023-10-19T12:45:42.865758+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:36:38.697134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-19T12:45:42.360355+0000","last_change":"2023-10-19T12:45:41.366288+0000","last_active":"2023-10-19T12:45:42.360355+0000","last_peered":"2023-10-19T12:45:42.360355+0000","last_clean":"2023-10-19T12:45:42.360355+0000","last_became_active":"2023-10-19T12:45:41.366038+0000","last_became_peered":"2023-10-19T12:45:41.366038+0000","last_unstale":"2023-10-19T12:45:42.360355+0000","last_undegraded":"2023-10-19T12:45:42.360355+0000","last_fullsized":"2023-10-19T12:45:42.360355+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:13:10.162844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.384100+0000","last_change":"2023-10-19T12:45:40.358578+0000","last_active":"2023-10-19T12:45:41.384100+0000","last_peered":"2023-10-19T12:45:41.384100+0000","last_clean":"2023-10-19T12:45:41.384100+0000","last_became_active":"2023-10-19T12:45:40.358057+0000","last_became_peered":"2023-10-19T12:45:40.358057+0000","last_unstale":"2023-10-19T12:45:41.384100+0000","last_undegraded":"2023-10-19T12:45:41.384100+0000","last_fullsized":"2023-10-19T12:45:41.384100+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:38:36.787271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-19T12:45:42.866194+0000","last_change":"2023-10-19T12:45:41.367997+0000","last_active":"2023-10-19T12:45:42.866194+0000","last_peered":"2023-10-19T12:45:42.866194+0000","last_clean":"2023-10-19T12:45:42.866194+0000","last_became_active":"2023-10-19T12:45:41.367687+0000","last_became_peered":"2023-10-19T12:45:41.367687+0000","last_unstale":"2023-10-19T12:45:42.866194+0000","last_undegraded":"2023-10-19T12:45:42.866194+0000","last_fullsized":"2023-10-19T12:45:42.866194+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:13:18.501715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.384069+0000","last_change":"2023-10-19T12:45:40.360184+0000","last_active":"2023-10-19T12:45:41.384069+0000","last_peered":"2023-10-19T12:45:41.384069+0000","last_clean":"2023-10-19T12:45:41.384069+0000","last_became_active":"2023-10-19T12:45:40.359917+0000","last_became_peered":"2023-10-19T12:45:40.359917+0000","last_unstale":"2023-10-19T12:45:41.384069+0000","last_undegraded":"2023-10-19T12:45:41.384069+0000","last_fullsized":"2023-10-19T12:45:41.384069+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:44:03.451018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-19T12:45:42.359389+0000","last_change":"2023-10-19T12:45:41.364206+0000","last_active":"2023-10-19T12:45:42.359389+0000","last_peered":"2023-10-19T12:45:42.359389+0000","last_clean":"2023-10-19T12:45:42.359389+0000","last_became_active":"2023-10-19T12:45:41.363926+0000","last_became_peered":"2023-10-19T12:45:41.363926+0000","last_unstale":"2023-10-19T12:45:42.359389+0000","last_undegraded":"2023-10-19T12:45:42.359389+0000","last_fullsized":"2023-10-19T12:45:42.359389+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:07:56.746385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.353616+0000","last_change":"2023-10-19T12:45:40.360715+0000","last_active":"2023-10-19T12:45:41.353616+0000","last_peered":"2023-10-19T12:45:41.353616+0000","last_clean":"2023-10-19T12:45:41.353616+0000","last_became_active":"2023-10-19T12:45:40.359801+0000","last_became_peered":"2023-10-19T12:45:40.359801+0000","last_unstale":"2023-10-19T12:45:41.353616+0000","last_undegraded":"2023-10-19T12:45:41.353616+0000","last_fullsized":"2023-10-19T12:45:41.353616+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:55:01.261795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-19T12:45:42.865869+0000","last_change":"2023-10-19T12:45:41.369258+0000","last_active":"2023-10-19T12:45:42.865869+0000","last_peered":"2023-10-19T12:45:42.865869+0000","last_clean":"2023-10-19T12:45:42.865869+0000","last_became_active":"2023-10-19T12:45:41.369005+0000","last_became_peered":"2023-10-19T12:45:41.369005+0000","last_unstale":"2023-10-19T12:45:42.865869+0000","last_undegraded":"2023-10-19T12:45:42.865869+0000","last_fullsized":"2023-10-19T12:45:42.865869+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:53:29.708835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.352492+0000","last_change":"2023-10-19T12:45:40.364278+0000","last_active":"2023-10-19T12:45:41.352492+0000","last_peered":"2023-10-19T12:45:41.352492+0000","last_clean":"2023-10-19T12:45:41.352492+0000","last_became_active":"2023-10-19T12:45:40.363981+0000","last_became_peered":"2023-10-19T12:45:40.363981+0000","last_unstale":"2023-10-19T12:45:41.352492+0000","last_undegraded":"2023-10-19T12:45:41.352492+0000","last_fullsized":"2023-10-19T12:45:41.352492+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:49:20.925838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-19T12:45:42.866596+0000","last_change":"2023-10-19T12:45:41.364179+0000","last_active":"2023-10-19T12:45:42.866596+0000","last_peered":"2023-10-19T12:45:42.866596+0000","last_clean":"2023-10-19T12:45:42.866596+0000","last_became_active":"2023-10-19T12:45:41.363871+0000","last_became_peered":"2023-10-19T12:45:41.363871+0000","last_unstale":"2023-10-19T12:45:42.866596+0000","last_undegraded":"2023-10-19T12:45:42.866596+0000","last_fullsized":"2023-10-19T12:45:42.866596+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:39:22.485898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.355676+0000","last_change":"2023-10-19T12:45:40.359438+0000","last_active":"2023-10-19T12:45:41.355676+0000","last_peered":"2023-10-19T12:45:41.355676+0000","last_clean":"2023-10-19T12:45:41.355676+0000","last_became_active":"2023-10-19T12:45:40.359156+0000","last_became_peered":"2023-10-19T12:45:40.359156+0000","last_unstale":"2023-10-19T12:45:41.355676+0000","last_undegraded":"2023-10-19T12:45:41.355676+0000","last_fullsized":"2023-10-19T12:45:41.355676+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:34:51.370131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-10-19T12:45:42.360209+0000","last_change":"2023-10-19T12:45:41.377271+0000","last_active":"2023-10-19T12:45:42.360209+0000","last_peered":"2023-10-19T12:45:42.360209+0000","last_clean":"2023-10-19T12:45:42.360209+0000","last_became_active":"2023-10-19T12:45:41.376969+0000","last_became_peered":"2023-10-19T12:45:41.376969+0000","last_unstale":"2023-10-19T12:45:42.360209+0000","last_undegraded":"2023-10-19T12:45:42.360209+0000","last_fullsized":"2023-10-19T12:45:42.360209+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:40.327526+0000","last_clean_scrub_stamp":"2023-10-19T12:45:40.327526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:38:12.163601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.355690+0000","last_change":"2023-10-19T12:45:40.359310+0000","last_active":"2023-10-19T12:45:41.355690+0000","last_peered":"2023-10-19T12:45:41.355690+0000","last_clean":"2023-10-19T12:45:41.355690+0000","last_became_active":"2023-10-19T12:45:40.359010+0000","last_became_peered":"2023-10-19T12:45:40.359010+0000","last_unstale":"2023-10-19T12:45:41.355690+0000","last_undegraded":"2023-10-19T12:45:41.355690+0000","last_fullsized":"2023-10-19T12:45:41.355690+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:45:39.321061+0000","last_clean_scrub_stamp":"2023-10-19T12:45:39.321061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:57:29.725854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":180,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-19T12:45:41.353264+0000","last_change":"2023-10-19T12:44:08.636613+0000","last_active":"2023-10-19T12:45:41.353264+0000","last_peered":"2023-10-19T12:45:41.353264+0000","last_clean":"2023-10-19T12:45:41.353264+0000","last_became_active":"2023-10-19T12:44:08.635697+0000","last_became_peered":"2023-10-19T12:44:08.635697+0000","last_unstale":"2023-10-19T12:45:41.353264+0000","last_undegraded":"2023-10-19T12:45:41.353264+0000","last_fullsized":"2023-10-19T12:45:41.353264+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_clean_scrub_stamp":"2023-10-19T12:44:07.399738+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:15:23.685485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574872,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28060,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343780,"statfs":{"total":96636764160,"available":96608030720,"internally_reserved":0,"allocated":1060864,"data_stored":785889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:45:08 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.40000000000000002}]},{"osd":1,"last update":"Thu Oct 19 12:45:08 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.374}]},{"osd":2,"last update":"Thu Oct 19 12:45:08 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.45200000000000001}]},{"osd":3,"last update":"Thu Oct 19 12:45:08 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.56100000000000005}]},{"osd":4,"last update":"Thu Oct 19 12:45:08 2023","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.51100000000000001}]},{"osd":5,"last update":"Thu Oct 19 12:45:08 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.497}]},{"osd":6,"last update":"Thu Oct 19 12:45:08 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.35399999999999998}]}]},{"osd":6,"up_from":13,"seq":55834574872,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27472,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344368,"statfs":{"total":96636764160,"available":96608632832,"internally_reserved":0,"allocated":454656,"data_stored":194421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:45:06 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.58599999999999997}]},{"osd":1,"last update":"Thu Oct 19 12:45:06 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.075,"5min":1.075,"15min":1.075},"last":0.52500000000000002}]},{"osd":2,"last update":"Thu Oct 19 12:45:06 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.441},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.60599999999999998}]},{"osd":3,"last update":"Thu Oct 19 12:45:06 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.498}]},{"osd":4,"last update":"Thu Oct 19 12:45:06 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.65400000000000003}]},{"osd":5,"last update":"Thu Oct 19 12:45:06 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.64100000000000001}]},{"osd":7,"last update":"Thu Oct 19 12:45:06 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.62},{"interface":"front","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.54800000000000004}]}]},{"osd":5,"up_from":13,"seq":55834574872,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27476,"kb_used_data":448,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344364,"statfs":{"total":96636764160,"available":96608628736,"internally_reserved":0,"allocated":458752,"data_stored":193951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.67800000000000005}]},{"osd":1,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.63900000000000001}]},{"osd":2,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.56699999999999995}]},{"osd":3,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.58599999999999997}]},{"osd":4,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.49299999999999999}]},{"osd":6,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.623}]},{"osd":7,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.65900000000000003}]}]},{"osd":4,"up_from":13,"seq":55834574872,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27468,"kb_used_data":448,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344372,"statfs":{"total":96636764160,"available":96608636928,"internally_reserved":0,"allocated":458752,"data_stored":194443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:45:08 2023","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.53900000000000003}]},{"osd":1,"last update":"Thu Oct 19 12:45:08 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.63300000000000001}]},{"osd":2,"last update":"Thu Oct 19 12:45:08 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.60199999999999998}]},{"osd":3,"last update":"Thu Oct 19 12:45:08 2023","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.64700000000000002}]},{"osd":5,"last update":"Thu Oct 19 12:45:08 2023","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.39000000000000001}]},{"osd":6,"last update":"Thu Oct 19 12:45:08 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.498}]},{"osd":7,"last update":"Thu Oct 19 12:45:08 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.68500000000000005}]}]},{"osd":3,"up_from":13,"seq":55834574871,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31644,"kb_used_data":456,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340196,"statfs":{"total":96636764160,"available":96604360704,"internally_reserved":0,"allocated":466944,"data_stored":195485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":1.04,"5min":1.04,"15min":1.04},"last":0.438},{"interface":"front","average":{"1min":0.375,"5min":0.375,"15min":0.375},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"last":0.49199999999999999}]},{"osd":1,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.47799999999999998}]},{"osd":2,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.54900000000000004}]},{"osd":4,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.79500000000000004}]},{"osd":5,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.68899999999999995}]},{"osd":6,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.747}]},{"osd":7,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.72599999999999998}]}]},{"osd":2,"up_from":13,"seq":55834574873,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31624,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340216,"statfs":{"total":96636764160,"available":96604381184,"internally_reserved":0,"allocated":454656,"data_stored":193909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.253},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"last":0.38600000000000001}]},{"osd":1,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.312},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.29299999999999998}]},{"osd":3,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.35699999999999998}]},{"osd":4,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.65800000000000003}]},{"osd":5,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.55400000000000005}]},{"osd":6,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.73599999999999999}]},{"osd":7,"last update":"Thu Oct 19 12:45:07 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.111,"5min":1.111,"15min":1.111},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.018,"5min":1.018,"15min":1.018},"last":0.76600000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574872,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31632,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340208,"statfs":{"total":96636764160,"available":96604372992,"internally_reserved":0,"allocated":454656,"data_stored":193929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:45:05 2023","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"last":0.22600000000000001},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.29199999999999998}]},{"osd":2,"last update":"Thu Oct 19 12:45:05 2023","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.27800000000000002},{"interface":"front","average":{"1min":0.433,"5min":0.433,"15min":0.433},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.32800000000000001}]},{"osd":3,"last update":"Thu Oct 19 12:45:05 2023","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.309}]},{"osd":4,"last update":"Thu Oct 19 12:45:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.1910000000000001,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.64700000000000002}]},{"osd":5,"last update":"Thu Oct 19 12:45:05 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.879},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.117,"5min":1.117,"15min":1.117},"last":0.68700000000000006}]},{"osd":6,"last update":"Thu Oct 19 12:45:05 2023","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.077,"5min":1.077,"15min":1.077},"last":0.626},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.77500000000000002}]},{"osd":7,"last update":"Thu Oct 19 12:45:05 2023","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.73499999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574873,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32204,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339636,"statfs":{"total":96636764160,"available":96603787264,"internally_reserved":0,"allocated":1044480,"data_stored":783622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Oct 19 12:45:09 2023","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"last":0.33900000000000002}]},{"osd":2,"last update":"Thu Oct 19 12:45:09 2023","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.36799999999999999}]},{"osd":3,"last update":"Thu Oct 19 12:45:09 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.42499999999999999}]},{"osd":4,"last update":"Thu Oct 19 12:45:09 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.56200000000000006}]},{"osd":5,"last update":"Thu Oct 19 12:45:09 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.056,"5min":1.056,"15min":1.056},"last":0.69199999999999995}]},{"osd":6,"last update":"Thu Oct 19 12:45:09 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.109,"5min":1.109,"15min":1.109},"last":0.65300000000000002}]},{"osd":7,"last update":"Thu Oct 19 12:45:09 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.61699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-19T12:45:49.168 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-19T12:45:49.169 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-19T12:45:49.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.315+0000 7f679cfa0640 1 -- 172.21.15.57:0/997058459 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f67980cd890 msgr2=0x7f67980cc730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:49.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.315+0000 7f679cfa0640 1 --2- 172.21.15.57:0/997058459 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f67980cd890 0x7f67980cc730 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f6788005fd0 tx=0x7f678802be20 comp rx=0 tx=0).stop 2023-10-19T12:45:49.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.315+0000 7f679cfa0640 1 -- 172.21.15.57:0/997058459 shutdown_connections 2023-10-19T12:45:49.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.315+0000 7f679cfa0640 1 --2- 172.21.15.57:0/997058459 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f67980cd890 0x7f67980cc730 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:49.320 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.315+0000 7f679cfa0640 1 -- 172.21.15.57:0/997058459 >> 172.21.15.57:0/997058459 conn(0x7f67980d0c30 msgr2=0x7f67980d1040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:49.320 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.315+0000 7f679cfa0640 1 -- 172.21.15.57:0/997058459 shutdown_connections 2023-10-19T12:45:49.320 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.315+0000 7f679cfa0640 1 -- 172.21.15.57:0/997058459 wait complete. 2023-10-19T12:45:49.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.316+0000 7f679cfa0640 1 Processor -- start 2023-10-19T12:45:49.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.317+0000 7f679cfa0640 1 -- start start 2023-10-19T12:45:49.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.317+0000 7f679cfa0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f67980cd890 0x7f67980b01a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:49.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.317+0000 7f679cfa0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f67980cdc70 0x7f6798158850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:49.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.317+0000 7f679cfa0640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f679815e840 0x7f6798148bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:49.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.317+0000 7f679cfa0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67980c8df0 con 0x7f67980cd890 2023-10-19T12:45:49.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.317+0000 7f679cfa0640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67980c90d0 con 0x7f67980cdc70 2023-10-19T12:45:49.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.317+0000 7f679cfa0640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67980c93b0 con 0x7f679815e840 2023-10-19T12:45:49.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.317+0000 7f67977fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f67980cd890 0x7f67980b01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:49.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.317+0000 7f67977fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f67980cd890 0x7f67980b01a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37482/0 (socket says 172.21.15.57:37482) 2023-10-19T12:45:49.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.317+0000 7f67977fe640 1 -- 172.21.15.57:0/3396253599 learned_addr learned my addr 172.21.15.57:0/3396253599 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:49.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.318+0000 7f6796ffd640 1 --2- 172.21.15.57:0/3396253599 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f67980cdc70 0x7f6798158850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:49.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.318+0000 7f6797fff640 1 --2- 172.21.15.57:0/3396253599 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f679815e840 0x7f6798148bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:49.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.318+0000 7f67977fe640 1 -- 172.21.15.57:0/3396253599 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f679815e840 msgr2=0x7f6798148bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:49.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.318+0000 7f67977fe640 1 --2- 172.21.15.57:0/3396253599 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f679815e840 0x7f6798148bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:49.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.318+0000 7f67977fe640 1 -- 172.21.15.57:0/3396253599 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f67980cdc70 msgr2=0x7f6798158850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:49.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.318+0000 7f67977fe640 1 --2- 172.21.15.57:0/3396253599 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f67980cdc70 0x7f6798158850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:49.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.318+0000 7f67977fe640 1 -- 172.21.15.57:0/3396253599 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6788009560 con 0x7f67980cd890 2023-10-19T12:45:49.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.318+0000 7f6796ffd640 1 --2- 172.21.15.57:0/3396253599 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f67980cdc70 0x7f6798158850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:45:49.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.318+0000 7f6797fff640 1 --2- 172.21.15.57:0/3396253599 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f679815e840 0x7f6798148bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:45:49.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.318+0000 7f67977fe640 1 --2- 172.21.15.57:0/3396253599 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f67980cd890 0x7f67980b01a0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f6788006010 tx=0x7f678802ddf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:49.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.318+0000 7f677ffff640 1 -- 172.21.15.57:0/3396253599 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6788005650 con 0x7f67980cd890 2023-10-19T12:45:49.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.318+0000 7f677ffff640 1 -- 172.21.15.57:0/3396253599 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f67880038f0 con 0x7f67980cd890 2023-10-19T12:45:49.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.318+0000 7f677ffff640 1 -- 172.21.15.57:0/3396253599 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6788003ca0 con 0x7f67980cd890 2023-10-19T12:45:49.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.318+0000 7f679cfa0640 1 -- 172.21.15.57:0/3396253599 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67980c9690 con 0x7f67980cd890 2023-10-19T12:45:49.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.318+0000 7f679cfa0640 1 -- 172.21.15.57:0/3396253599 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6798155850 con 0x7f67980cd890 2023-10-19T12:45:49.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.319+0000 7f679cfa0640 1 -- 172.21.15.57:0/3396253599 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6798066960 con 0x7f67980cd890 2023-10-19T12:45:49.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.320+0000 7f677ffff640 1 -- 172.21.15.57:0/3396253599 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f678803e450 con 0x7f67980cd890 2023-10-19T12:45:49.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.320+0000 7f677ffff640 1 --2- 172.21.15.57:0/3396253599 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f678406ae40 0x7f678406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:49.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.320+0000 7f677ffff640 1 -- 172.21.15.57:0/3396253599 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f67880ac450 con 0x7f67980cd890 2023-10-19T12:45:49.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.321+0000 7f6796ffd640 1 --2- 172.21.15.57:0/3396253599 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f678406ae40 0x7f678406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:49.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.321+0000 7f6796ffd640 1 --2- 172.21.15.57:0/3396253599 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f678406ae40 0x7f678406d300 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f6798158660 tx=0x7f678c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:49.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.325+0000 7f677ffff640 1 -- 172.21.15.57:0/3396253599 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f67880775a0 con 0x7f67980cd890 2023-10-19T12:45:49.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.575+0000 7f679cfa0640 1 -- 172.21.15.57:0/3396253599 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f679804da70 con 0x7f67980cd890 2023-10-19T12:45:49.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.577+0000 7f677ffff640 1 -- 172.21.15.57:0/3396253599 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v19) v1 ==== 190+0+147 (secure 0 0 0) 0x7f67880430b0 con 0x7f67980cd890 2023-10-19T12:45:49.580 INFO:teuthology.orchestra.run.smithi057.stdout:[client.0] 2023-10-19T12:45:49.580 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQBkJDFlg9yuExAATxiWDfS+p07G1F0pORbJEg== 2023-10-19T12:45:49.581 INFO:teuthology.orchestra.run.smithi057.stdout: caps mds = "allow" 2023-10-19T12:45:49.581 INFO:teuthology.orchestra.run.smithi057.stdout: caps mon = "allow r" 2023-10-19T12:45:49.581 INFO:teuthology.orchestra.run.smithi057.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-19T12:45:49.581 INFO:teuthology.orchestra.run.smithi057.stderr:updated caps for client.0 2023-10-19T12:45:49.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.579+0000 7f679cfa0640 1 -- 172.21.15.57:0/3396253599 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f678406ae40 msgr2=0x7f678406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:49.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.579+0000 7f679cfa0640 1 --2- 172.21.15.57:0/3396253599 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f678406ae40 0x7f678406d300 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f6798158660 tx=0x7f678c009040 comp rx=0 tx=0).stop 2023-10-19T12:45:49.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.579+0000 7f679cfa0640 1 -- 172.21.15.57:0/3396253599 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f67980cd890 msgr2=0x7f67980b01a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:49.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.579+0000 7f679cfa0640 1 --2- 172.21.15.57:0/3396253599 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f67980cd890 0x7f67980b01a0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f6788006010 tx=0x7f678802ddf0 comp rx=0 tx=0).stop 2023-10-19T12:45:49.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.579+0000 7f679cfa0640 1 -- 172.21.15.57:0/3396253599 shutdown_connections 2023-10-19T12:45:49.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.579+0000 7f679cfa0640 1 --2- 172.21.15.57:0/3396253599 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f678406ae40 0x7f678406d300 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:49.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.579+0000 7f679cfa0640 1 --2- 172.21.15.57:0/3396253599 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f679815e840 0x7f6798148bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:49.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.579+0000 7f679cfa0640 1 --2- 172.21.15.57:0/3396253599 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f67980cdc70 0x7f6798158850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:49.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.579+0000 7f679cfa0640 1 --2- 172.21.15.57:0/3396253599 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f67980cd890 0x7f67980b01a0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:49.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.579+0000 7f679cfa0640 1 -- 172.21.15.57:0/3396253599 >> 172.21.15.57:0/3396253599 conn(0x7f67980d0c30 msgr2=0x7f67980ae770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:49.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.579+0000 7f679cfa0640 1 -- 172.21.15.57:0/3396253599 shutdown_connections 2023-10-19T12:45:49.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.579+0000 7f679cfa0640 1 -- 172.21.15.57:0/3396253599 wait complete. 2023-10-19T12:45:49.592 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T12:45:49.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.698+0000 7fd8310f2640 1 -- 172.21.15.57:0/1380201827 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd82c0cd6d0 msgr2=0x7fd82c0cdb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:49.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.698+0000 7fd8310f2640 1 --2- 172.21.15.57:0/1380201827 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd82c0cd6d0 0x7fd82c0cdb30 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fd82c071e10 tx=0x7fd82002bcf0 comp rx=0 tx=0).stop 2023-10-19T12:45:49.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.698+0000 7fd8310f2640 1 -- 172.21.15.57:0/1380201827 shutdown_connections 2023-10-19T12:45:49.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.698+0000 7fd8310f2640 1 --2- 172.21.15.57:0/1380201827 >> v2:172.21.15.179:3300/0 conn(0x7fd82c069720 0x7fd82c069bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:49.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.698+0000 7fd8310f2640 1 --2- 172.21.15.57:0/1380201827 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd82c0cd6d0 0x7fd82c0cdb30 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:49.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.698+0000 7fd8310f2640 1 -- 172.21.15.57:0/1380201827 >> 172.21.15.57:0/1380201827 conn(0x7fd82c0af7a0 msgr2=0x7fd82c0afb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:49.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.698+0000 7fd8310f2640 1 -- 172.21.15.57:0/1380201827 shutdown_connections 2023-10-19T12:45:49.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.698+0000 7fd8310f2640 1 -- 172.21.15.57:0/1380201827 wait complete. 2023-10-19T12:45:49.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.699+0000 7fd8310f2640 1 Processor -- start 2023-10-19T12:45:49.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.699+0000 7fd8310f2640 1 -- start start 2023-10-19T12:45:49.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.700+0000 7fd8310f2640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd82c069720 0x7fd82c1ef0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:49.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.700+0000 7fd8310f2640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd82c0cd6d0 0x7fd82c136bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:49.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.700+0000 7fd8310f2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd82c142850 0x7fd82c14d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:49.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.700+0000 7fd8310f2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd82c138050 con 0x7fd82c142850 2023-10-19T12:45:49.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.700+0000 7fd8310f2640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd82c137440 con 0x7fd82c069720 2023-10-19T12:45:49.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.700+0000 7fd8310f2640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd82c137720 con 0x7fd82c0cd6d0 2023-10-19T12:45:49.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.700+0000 7fd82b59e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd82c142850 0x7fd82c14d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:49.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.700+0000 7fd82b59e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd82c142850 0x7fd82c14d7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37496/0 (socket says 172.21.15.57:37496) 2023-10-19T12:45:49.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.700+0000 7fd82b59e640 1 -- 172.21.15.57:0/1673537054 learned_addr learned my addr 172.21.15.57:0/1673537054 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:49.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.700+0000 7fd82a59c640 1 --2- 172.21.15.57:0/1673537054 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd82c0cd6d0 0x7fd82c136bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:49.707 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.700+0000 7fd82ad9d640 1 --2- 172.21.15.57:0/1673537054 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd82c069720 0x7fd82c1ef0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:49.707 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.700+0000 7fd82b59e640 1 -- 172.21.15.57:0/1673537054 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd82c0cd6d0 msgr2=0x7fd82c136bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:49.707 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.700+0000 7fd82b59e640 1 --2- 172.21.15.57:0/1673537054 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd82c0cd6d0 0x7fd82c136bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:49.707 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.700+0000 7fd82b59e640 1 -- 172.21.15.57:0/1673537054 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd82c069720 msgr2=0x7fd82c1ef0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:49.707 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.700+0000 7fd82b59e640 1 --2- 172.21.15.57:0/1673537054 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd82c069720 0x7fd82c1ef0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:49.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.700+0000 7fd82b59e640 1 -- 172.21.15.57:0/1673537054 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd820009490 con 0x7fd82c142850 2023-10-19T12:45:49.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.700+0000 7fd82ad9d640 1 --2- 172.21.15.57:0/1673537054 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd82c069720 0x7fd82c1ef0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:45:49.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.700+0000 7fd82a59c640 1 --2- 172.21.15.57:0/1673537054 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd82c0cd6d0 0x7fd82c136bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:45:49.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.701+0000 7fd82b59e640 1 --2- 172.21.15.57:0/1673537054 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd82c142850 0x7fd82c14d7d0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fd81c00eaa0 tx=0x7fd81c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:49.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.701+0000 7fd8137fe640 1 -- 172.21.15.57:0/1673537054 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd81c00cdb0 con 0x7fd82c142850 2023-10-19T12:45:49.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.701+0000 7fd8137fe640 1 -- 172.21.15.57:0/1673537054 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fd81c004b20 con 0x7fd82c142850 2023-10-19T12:45:49.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.701+0000 7fd8310f2640 1 -- 172.21.15.57:0/1673537054 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd82c137a00 con 0x7fd82c142850 2023-10-19T12:45:49.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.701+0000 7fd8137fe640 1 -- 172.21.15.57:0/1673537054 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd81c005020 con 0x7fd82c142850 2023-10-19T12:45:49.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.701+0000 7fd8310f2640 1 -- 172.21.15.57:0/1673537054 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd82c1370f0 con 0x7fd82c142850 2023-10-19T12:45:49.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.702+0000 7fd8310f2640 1 -- 172.21.15.57:0/1673537054 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd82c06a440 con 0x7fd82c142850 2023-10-19T12:45:49.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.702+0000 7fd8137fe640 1 -- 172.21.15.57:0/1673537054 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fd81c005180 con 0x7fd82c142850 2023-10-19T12:45:49.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.703+0000 7fd8137fe640 1 --2- 172.21.15.57:0/1673537054 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fd81806ae40 0x7fd81806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:49.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.703+0000 7fd8137fe640 1 -- 172.21.15.57:0/1673537054 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fd81c08a1e0 con 0x7fd82c142850 2023-10-19T12:45:49.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.703+0000 7fd82ad9d640 1 --2- 172.21.15.57:0/1673537054 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fd81806ae40 0x7fd81806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:49.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.704+0000 7fd82ad9d640 1 --2- 172.21.15.57:0/1673537054 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fd81806ae40 0x7fd81806d300 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fd824000f60 tx=0x7fd824009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:49.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.707+0000 7fd8137fe640 1 -- 172.21.15.57:0/1673537054 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd81c055150 con 0x7fd82c142850 2023-10-19T12:45:49.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.918+0000 7fd8310f2640 1 -- 172.21.15.57:0/1673537054 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd82c0690e0 con 0x7fd82c142850 2023-10-19T12:45:49.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.919+0000 7fd8137fe640 1 -- 172.21.15.57:0/1673537054 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4916 (secure 0 0 0) 0x7fd81c058770 con 0x7fd82c142850 2023-10-19T12:45:49.922 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:45:49.923 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5024,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/808517451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":808517451},{"type":"v1","addr":"172.21.15.179:6836","nonce":808517451}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":5625,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/3207366629","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":3207366629},{"type":"v1","addr":"172.21.15.57:6834","nonce":3207366629}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":5628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/2353910009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2353910009},{"type":"v1","addr":"172.21.15.57:6835","nonce":2353910009}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T12:45:40.706506+0000","modified":"2023-10-19T12:45:42.702983+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5634},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5634":{"gid":5634,"name":"d","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.179:6837/1237463333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":1237463333},{"type":"v1","addr":"172.21.15.179:6837","nonce":1237463333}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-19T12:45:49.923 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 16 2023-10-19T12:45:49.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.921+0000 7fd8310f2640 1 -- 172.21.15.57:0/1673537054 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fd81806ae40 msgr2=0x7fd81806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:49.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.921+0000 7fd8310f2640 1 --2- 172.21.15.57:0/1673537054 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fd81806ae40 0x7fd81806d300 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fd824000f60 tx=0x7fd824009040 comp rx=0 tx=0).stop 2023-10-19T12:45:49.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.921+0000 7fd8310f2640 1 -- 172.21.15.57:0/1673537054 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd82c142850 msgr2=0x7fd82c14d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:49.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.921+0000 7fd8310f2640 1 --2- 172.21.15.57:0/1673537054 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd82c142850 0x7fd82c14d7d0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fd81c00eaa0 tx=0x7fd81c00ef60 comp rx=0 tx=0).stop 2023-10-19T12:45:49.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.921+0000 7fd8310f2640 1 -- 172.21.15.57:0/1673537054 shutdown_connections 2023-10-19T12:45:49.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.921+0000 7fd8310f2640 1 --2- 172.21.15.57:0/1673537054 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fd81806ae40 0x7fd81806d300 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:49.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.921+0000 7fd8310f2640 1 --2- 172.21.15.57:0/1673537054 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd82c142850 0x7fd82c14d7d0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:49.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.921+0000 7fd8310f2640 1 --2- 172.21.15.57:0/1673537054 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd82c0cd6d0 0x7fd82c136bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:49.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.921+0000 7fd8310f2640 1 --2- 172.21.15.57:0/1673537054 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd82c069720 0x7fd82c1ef0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:49.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.921+0000 7fd8310f2640 1 -- 172.21.15.57:0/1673537054 >> 172.21.15.57:0/1673537054 conn(0x7fd82c0af7a0 msgr2=0x7fd82c13f3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:49.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.921+0000 7fd8310f2640 1 -- 172.21.15.57:0/1673537054 shutdown_connections 2023-10-19T12:45:49.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:49.921+0000 7fd8310f2640 1 -- 172.21.15.57:0/1673537054 wait complete. 2023-10-19T12:45:50.337 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 16, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-19T12:45:40.706506+0000', 'modified': '2023-10-19T12:45:42.702983+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5634}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5634': {'gid': 5634, 'name': 'd', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.179:6837/1237463333', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6835', 'nonce': 1237463333}, {'type': 'v1', 'addr': '172.21.15.179:6837', 'nonce': 1237463333}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-19T12:45:50.338 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-19T12:45:50.338 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2023-10-19T12:45:50.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.451+0000 7f732b1e9640 1 -- 172.21.15.57:0/2454021918 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f73241579f0 msgr2=0x7f73240b0e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:50.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.451+0000 7f732b1e9640 1 --2- 172.21.15.57:0/2454021918 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f73241579f0 0x7f73240b0e90 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f7318005fd0 tx=0x7f731802bd90 comp rx=0 tx=0).stop 2023-10-19T12:45:50.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.451+0000 7f732b1e9640 1 -- 172.21.15.57:0/2454021918 shutdown_connections 2023-10-19T12:45:50.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.451+0000 7f732b1e9640 1 --2- 172.21.15.57:0/2454021918 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f732415c1f0 0x7f732406a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:50.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.451+0000 7f732b1e9640 1 --2- 172.21.15.57:0/2454021918 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f73241579f0 0x7f73240b0e90 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:50.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.451+0000 7f732b1e9640 1 -- 172.21.15.57:0/2454021918 >> 172.21.15.57:0/2454021918 conn(0x7f732406fae0 msgr2=0x7f732406fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:50.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.451+0000 7f732b1e9640 1 -- 172.21.15.57:0/2454021918 shutdown_connections 2023-10-19T12:45:50.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.451+0000 7f732b1e9640 1 -- 172.21.15.57:0/2454021918 wait complete. 2023-10-19T12:45:50.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.452+0000 7f732b1e9640 1 Processor -- start 2023-10-19T12:45:50.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.452+0000 7f732b1e9640 1 -- start start 2023-10-19T12:45:50.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.453+0000 7f732b1e9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f732406ac70 0x7f7324156760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:50.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.453+0000 7f732b1e9640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f73241579f0 0x7f732414fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:50.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.453+0000 7f732b1e9640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f732415c1f0 0x7f73240a64b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:50.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.453+0000 7f732b1e9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73240ae7e0 con 0x7f732406ac70 2023-10-19T12:45:50.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.453+0000 7f732b1e9640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73240aeac0 con 0x7f732415c1f0 2023-10-19T12:45:50.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.453+0000 7f732b1e9640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73240aeda0 con 0x7f73241579f0 2023-10-19T12:45:50.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.453+0000 7f7328f85640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f732406ac70 0x7f7324156760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:50.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.453+0000 7f7328f85640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f732406ac70 0x7f7324156760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37524/0 (socket says 172.21.15.57:37524) 2023-10-19T12:45:50.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.453+0000 7f7328f85640 1 -- 172.21.15.57:0/1740029357 learned_addr learned my addr 172.21.15.57:0/1740029357 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:50.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.453+0000 7f7323fff640 1 --2- 172.21.15.57:0/1740029357 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f73241579f0 0x7f732414fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:50.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.453+0000 7f7329786640 1 --2- 172.21.15.57:0/1740029357 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f732415c1f0 0x7f73240a64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:50.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.453+0000 7f7328f85640 1 -- 172.21.15.57:0/1740029357 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f73241579f0 msgr2=0x7f732414fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:50.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.453+0000 7f7328f85640 1 --2- 172.21.15.57:0/1740029357 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f73241579f0 0x7f732414fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:50.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.453+0000 7f7328f85640 1 -- 172.21.15.57:0/1740029357 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f732415c1f0 msgr2=0x7f73240a64b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:50.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.453+0000 7f7328f85640 1 --2- 172.21.15.57:0/1740029357 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f732415c1f0 0x7f73240a64b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:50.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.453+0000 7f7328f85640 1 -- 172.21.15.57:0/1740029357 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7318009560 con 0x7f732406ac70 2023-10-19T12:45:50.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.453+0000 7f7328f85640 1 --2- 172.21.15.57:0/1740029357 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f732406ac70 0x7f7324156760 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f7318036040 tx=0x7f731802dd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:50.461 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.454+0000 7f73217fa640 1 -- 172.21.15.57:0/1740029357 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7318005650 con 0x7f732406ac70 2023-10-19T12:45:50.461 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.454+0000 7f73217fa640 1 -- 172.21.15.57:0/1740029357 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f73180038f0 con 0x7f732406ac70 2023-10-19T12:45:50.461 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.454+0000 7f732b1e9640 1 -- 172.21.15.57:0/1740029357 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7324156240 con 0x7f732406ac70 2023-10-19T12:45:50.461 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.454+0000 7f73217fa640 1 -- 172.21.15.57:0/1740029357 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7318003ca0 con 0x7f732406ac70 2023-10-19T12:45:50.461 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.454+0000 7f732b1e9640 1 -- 172.21.15.57:0/1740029357 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7324150260 con 0x7f732406ac70 2023-10-19T12:45:50.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.455+0000 7f73217fa640 1 -- 172.21.15.57:0/1740029357 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f731803e450 con 0x7f732406ac70 2023-10-19T12:45:50.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.456+0000 7f73217fa640 1 --2- 172.21.15.57:0/1740029357 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f730806ae40 0x7f730806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:50.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.456+0000 7f73217fa640 1 -- 172.21.15.57:0/1740029357 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f73180ac2e0 con 0x7f732406ac70 2023-10-19T12:45:50.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.456+0000 7f7323fff640 1 --2- 172.21.15.57:0/1740029357 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f730806ae40 0x7f730806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:50.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.457+0000 7f732dd7a740 1 Processor -- start 2023-10-19T12:45:50.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.457+0000 7f732dd7a740 1 -- start start 2023-10-19T12:45:50.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.457+0000 7f7323fff640 1 --2- 172.21.15.57:0/1740029357 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f730806ae40 0x7f730806d300 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f73241f2330 tx=0x7f731000a420 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:50.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.457+0000 7f732dd7a740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x555ff7cddf50 0x555ff7cb0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:50.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.457+0000 7f732dd7a740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555ff7cde330 0x555ff7ce0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:50.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.457+0000 7f732dd7a740 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x555ff7cb1430 0x555ff7ce1220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:50.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.457+0000 7f732dd7a740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ff7bbe730 con 0x555ff7cddf50 2023-10-19T12:45:50.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.457+0000 7f732dd7a740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ff7c1f780 con 0x555ff7cde330 2023-10-19T12:45:50.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.457+0000 7f732dd7a740 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x555ff7c1ce70 con 0x555ff7cb1430 2023-10-19T12:45:50.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.458+0000 7f7328f85640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x555ff7cddf50 0x555ff7cb0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:50.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.458+0000 7f7328f85640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x555ff7cddf50 0x555ff7cb0ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37532/0 (socket says 172.21.15.57:37532) 2023-10-19T12:45:50.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.458+0000 7f7328f85640 1 -- 172.21.15.57:0/1885310654 learned_addr learned my addr 172.21.15.57:0/1885310654 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:50.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.458+0000 7f7328f85640 1 -- 172.21.15.57:0/1885310654 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x555ff7cb1430 msgr2=0x555ff7ce1220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T12:45:50.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.458+0000 7f7328f85640 1 --2- 172.21.15.57:0/1885310654 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x555ff7cb1430 0x555ff7ce1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:50.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.458+0000 7f7328f85640 1 -- 172.21.15.57:0/1885310654 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555ff7cde330 msgr2=0x555ff7ce0ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T12:45:50.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.458+0000 7f7323fff640 1 --2- 172.21.15.57:0/1885310654 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555ff7cde330 0x555ff7ce0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:50.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.458+0000 7f7328f85640 1 --2- 172.21.15.57:0/1885310654 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555ff7cde330 0x555ff7ce0ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:50.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.458+0000 7f7328f85640 1 -- 172.21.15.57:0/1885310654 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555ff7c1d3d0 con 0x555ff7cddf50 2023-10-19T12:45:50.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.458+0000 7f7323fff640 1 --2- 172.21.15.57:0/1885310654 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555ff7cde330 0x555ff7ce0ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:45:50.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.458+0000 7f7328f85640 1 --2- 172.21.15.57:0/1885310654 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x555ff7cddf50 0x555ff7cb0ef0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f73180aa640 tx=0x7f731802bd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56dc725fea418f72 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:50.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.458+0000 7f730e7fc640 1 -- 172.21.15.57:0/1885310654 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73180ac0b0 con 0x555ff7cddf50 2023-10-19T12:45:50.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.458+0000 7f730e7fc640 1 -- 172.21.15.57:0/1885310654 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f7318079100 con 0x555ff7cddf50 2023-10-19T12:45:50.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.458+0000 7f730e7fc640 1 -- 172.21.15.57:0/1885310654 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7318079360 con 0x555ff7cddf50 2023-10-19T12:45:50.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.459+0000 7f732dd7a740 1 -- 172.21.15.57:0/1885310654 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x555ff7cddf50 msgr2=0x555ff7cb0ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:50.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.459+0000 7f732dd7a740 1 --2- 172.21.15.57:0/1885310654 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x555ff7cddf50 0x555ff7cb0ef0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f73180aa640 tx=0x7f731802bd90 comp rx=0 tx=0).stop 2023-10-19T12:45:50.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.459+0000 7f732dd7a740 1 -- 172.21.15.57:0/1885310654 shutdown_connections 2023-10-19T12:45:50.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.459+0000 7f732dd7a740 1 --2- 172.21.15.57:0/1885310654 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x555ff7cb1430 0x555ff7ce1220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:50.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.459+0000 7f732dd7a740 1 --2- 172.21.15.57:0/1885310654 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555ff7cde330 0x555ff7ce0ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:50.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.459+0000 7f732dd7a740 1 --2- 172.21.15.57:0/1885310654 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x555ff7cddf50 0x555ff7cb0ef0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:50.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.459+0000 7f732dd7a740 1 -- 172.21.15.57:0/1885310654 >> 172.21.15.57:0/1885310654 conn(0x555ff7cb3160 msgr2=0x555ff7cfbce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:50.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.459+0000 7f732dd7a740 1 -- 172.21.15.57:0/1885310654 shutdown_connections 2023-10-19T12:45:50.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.459+0000 7f732dd7a740 1 -- 172.21.15.57:0/1885310654 wait complete. 2023-10-19T12:45:50.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.459+0000 7f732dd7a740 1 -- start start 2023-10-19T12:45:50.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.460+0000 7f732dd7a740 1 Processor -- start 2023-10-19T12:45:50.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.460+0000 7f732dd7a740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555ff7cddf50 0x555ff7cfbd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:50.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.460+0000 7f732dd7a740 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x555ff7cde330 0x555ff7ce3730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:50.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.460+0000 7f732dd7a740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x555ff7ce3c70 0x555ff7cf55e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:50.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.460+0000 7f732dd7a740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ff7c18760 con 0x555ff7ce3c70 2023-10-19T12:45:50.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.460+0000 7f732dd7a740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ff7bd38f0 con 0x555ff7cddf50 2023-10-19T12:45:50.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.460+0000 7f732dd7a740 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x555ff7bd3590 con 0x555ff7cde330 2023-10-19T12:45:50.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.460+0000 7f732dd7a740 10 client.0 _ll_register_callbacks cb 0x555ff7c0fee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-19T12:45:50.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.460+0000 7f7329786640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x555ff7ce3c70 0x555ff7cf55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:50.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.460+0000 7f7329786640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x555ff7ce3c70 0x555ff7cf55e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37548/0 (socket says 172.21.15.57:37548) 2023-10-19T12:45:50.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.460+0000 7f7329786640 1 -- 172.21.15.57:0/4051405805 learned_addr learned my addr 172.21.15.57:0/4051405805 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:50.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.461+0000 7f7328f85640 1 --2- 172.21.15.57:0/4051405805 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555ff7cddf50 0x555ff7cfbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:50.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.461+0000 7f7323fff640 1 --2- 172.21.15.57:0/4051405805 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x555ff7cde330 0x555ff7ce3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:50.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.461+0000 7f7329786640 1 -- 172.21.15.57:0/4051405805 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x555ff7cde330 msgr2=0x555ff7ce3730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T12:45:50.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.461+0000 7f7329786640 1 --2- 172.21.15.57:0/4051405805 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x555ff7cde330 0x555ff7ce3730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:50.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.461+0000 7f7329786640 1 -- 172.21.15.57:0/4051405805 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555ff7cddf50 msgr2=0x555ff7cfbd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T12:45:50.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.461+0000 7f7329786640 1 --2- 172.21.15.57:0/4051405805 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555ff7cddf50 0x555ff7cfbd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:50.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.461+0000 7f7329786640 1 -- 172.21.15.57:0/4051405805 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7324150260 con 0x555ff7ce3c70 2023-10-19T12:45:50.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.461+0000 7f7329786640 1 --2- 172.21.15.57:0/4051405805 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x555ff7ce3c70 0x555ff7cf55e0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f731c00f590 tx=0x7f731c00fa50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd35dd07e1b05f4c server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:50.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.461+0000 7f73037fe640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2023-10-19T12:45:50.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.461+0000 7f73037fe640 1 -- 172.21.15.57:0/4051405805 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f731c003b80 con 0x555ff7ce3c70 2023-10-19T12:45:50.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.461+0000 7f73037fe640 1 -- 172.21.15.57:0/4051405805 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f731c003ce0 con 0x555ff7ce3c70 2023-10-19T12:45:50.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.461+0000 7f732dd7a740 1 -- 172.21.15.57:0/4051405805 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555ff7c11740 con 0x555ff7ce3c70 2023-10-19T12:45:50.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.461+0000 7f73037fe640 1 -- 172.21.15.57:0/4051405805 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7f731c0050d0 con 0x555ff7ce3c70 2023-10-19T12:45:50.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.461+0000 7f732dd7a740 10 client.5832 fetch_fsmap learned FSMap version 16 2023-10-19T12:45:50.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.461+0000 7f732dd7a740 1 -- 172.21.15.57:0/4051405805 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) v3 -- 0x555ff7c16de0 con 0x555ff7ce3c70 2023-10-19T12:45:50.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.461+0000 7f73037fe640 1 -- 172.21.15.57:0/4051405805 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x555ff7c11740 con 0x555ff7ce3c70 2023-10-19T12:45:50.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.462+0000 7f73037fe640 1 -- 172.21.15.57:0/4051405805 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 16) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f731c012070 con 0x555ff7ce3c70 2023-10-19T12:45:50.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.462+0000 7f732dd7a740 10 client.5832 fetch_fsmap finished waiting for FSMap version 16 2023-10-19T12:45:50.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.462+0000 7f732dd7a740 10 client.5832 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-19T12:45:50.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.462+0000 7f732dd7a740 20 client.5832 populate_metadata read hostname 'smithi057' 2023-10-19T12:45:50.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.462+0000 7f732dd7a740 1 --2- 172.21.15.57:0/4051405805 >> [v2:172.21.15.179:6835/1237463333,v1:172.21.15.179:6837/1237463333] conn(0x555ff7cccd10 0x555ff7ccd0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:50.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.462+0000 7f732dd7a740 4 client.5832 mds_command: new command op to 5634 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-19T12:45:50.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.462+0000 7f732dd7a740 1 -- 172.21.15.57:0/4051405805 --> [v2:172.21.15.179:6835/1237463333,v1:172.21.15.179:6837/1237463333] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555ff7c11460 con 0x555ff7cccd10 2023-10-19T12:45:50.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.462+0000 7f7328f85640 1 --2- 172.21.15.57:0/4051405805 >> [v2:172.21.15.179:6835/1237463333,v1:172.21.15.179:6837/1237463333] conn(0x555ff7cccd10 0x555ff7ccd0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:50.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.462+0000 7f7328f85640 1 --2- 172.21.15.57:0/4051405805 >> [v2:172.21.15.179:6835/1237463333,v1:172.21.15.179:6837/1237463333] conn(0x555ff7cccd10 0x555ff7ccd0f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ef1b0617f1f31aa server_cookie=e4d71d27ba1d60c9 in_seq=0 out_seq=0 2023-10-19T12:45:50.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.462+0000 7f73037fe640 10 client.5832 ms_handle_connect on v2:172.21.15.179:6835/1237463333 2023-10-19T12:45:50.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.463+0000 7f73037fe640 1 -- 172.21.15.57:0/4051405805 <== mds.0 v2:172.21.15.179:6835/1237463333 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x555ff7c11460 con 0x555ff7cccd10 2023-10-19T12:45:50.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.463+0000 7f73037fe640 10 client.5832 handle_command_reply: tid=0 2023-10-19T12:45:50.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.463+0000 7f732dd7a740 1 client.5832 shutdown 2023-10-19T12:45:50.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.463+0000 7f732dd7a740 1 -- 172.21.15.57:0/4051405805 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x555ff7ce3c70 msgr2=0x555ff7cf55e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:50.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.463+0000 7f732dd7a740 1 --2- 172.21.15.57:0/4051405805 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x555ff7ce3c70 0x555ff7cf55e0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f731c00f590 tx=0x7f731c00fa50 comp rx=0 tx=0).stop 2023-10-19T12:45:50.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.463+0000 7f732dd7a740 1 -- 172.21.15.57:0/4051405805 shutdown_connections 2023-10-19T12:45:50.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.463+0000 7f732dd7a740 1 --2- 172.21.15.57:0/4051405805 >> [v2:172.21.15.179:6835/1237463333,v1:172.21.15.179:6837/1237463333] conn(0x555ff7cccd10 0x555ff7ccd0f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:50.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.463+0000 7f732dd7a740 1 --2- 172.21.15.57:0/4051405805 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x555ff7ce3c70 0x555ff7cf55e0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:50.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.463+0000 7f732dd7a740 1 --2- 172.21.15.57:0/4051405805 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x555ff7cde330 0x555ff7ce3730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:50.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.463+0000 7f73037fe640 0 client.5832 ms_handle_reset on v2:172.21.15.179:6835/1237463333 2023-10-19T12:45:50.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.463+0000 7f732dd7a740 1 --2- 172.21.15.57:0/4051405805 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555ff7cddf50 0x555ff7cfbd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:50.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.463+0000 7f732dd7a740 1 -- 172.21.15.57:0/4051405805 >> 172.21.15.57:0/4051405805 conn(0x555ff7cb1430 msgr2=0x555ff7cff1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:50.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.463+0000 7f732dd7a740 1 -- 172.21.15.57:0/4051405805 shutdown_connections 2023-10-19T12:45:50.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.463+0000 7f732dd7a740 1 -- 172.21.15.57:0/4051405805 wait complete. 2023-10-19T12:45:50.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.463+0000 7f732dd7a740 20 client.5832 trim_cache size 0 max 16384 2023-10-19T12:45:50.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.479+0000 7f732dd7a740 1 Processor -- start 2023-10-19T12:45:50.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.479+0000 7f732dd7a740 1 -- start start 2023-10-19T12:45:50.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.479+0000 7f732dd7a740 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x555ff7cddf50 0x555ff7ce10f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:50.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.479+0000 7f732dd7a740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x555ff7cde330 0x555ff7cdc890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:50.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.479+0000 7f732dd7a740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555ff7ce3c70 0x555ff7cdcdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:50.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.479+0000 7f732dd7a740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ff7b99270 con 0x555ff7cde330 2023-10-19T12:45:50.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.479+0000 7f732dd7a740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ff7cced00 con 0x555ff7ce3c70 2023-10-19T12:45:50.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.479+0000 7f732dd7a740 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x555ff7ce33d0 con 0x555ff7cddf50 2023-10-19T12:45:50.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.480+0000 7f7323fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x555ff7cde330 0x555ff7cdc890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:50.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.480+0000 7f7323fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x555ff7cde330 0x555ff7cdc890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37558/0 (socket says 172.21.15.57:37558) 2023-10-19T12:45:50.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.480+0000 7f7323fff640 1 -- 172.21.15.57:0/254782444 learned_addr learned my addr 172.21.15.57:0/254782444 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:50.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.480+0000 7f7328f85640 1 --2- 172.21.15.57:0/254782444 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x555ff7cddf50 0x555ff7ce10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:50.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.480+0000 7f7329786640 1 --2- 172.21.15.57:0/254782444 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555ff7ce3c70 0x555ff7cdcdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:50.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.480+0000 7f7323fff640 1 -- 172.21.15.57:0/254782444 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x555ff7cddf50 msgr2=0x555ff7ce10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T12:45:50.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.480+0000 7f7323fff640 1 --2- 172.21.15.57:0/254782444 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x555ff7cddf50 0x555ff7ce10f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:50.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.480+0000 7f7323fff640 1 -- 172.21.15.57:0/254782444 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555ff7ce3c70 msgr2=0x555ff7cdcdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T12:45:50.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.480+0000 7f7323fff640 1 --2- 172.21.15.57:0/254782444 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555ff7ce3c70 0x555ff7cdcdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:50.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.480+0000 7f7323fff640 1 -- 172.21.15.57:0/254782444 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555ff7bb5d00 con 0x555ff7cde330 2023-10-19T12:45:50.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.480+0000 7f7329786640 1 --2- 172.21.15.57:0/254782444 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555ff7ce3c70 0x555ff7cdcdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:45:50.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.480+0000 7f7323fff640 1 --2- 172.21.15.57:0/254782444 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x555ff7cde330 0x555ff7cdc890 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f73100058c0 tx=0x7f7310016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41e6a83a94c3da9b server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:50.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.480+0000 7f730effd640 1 -- 172.21.15.57:0/254782444 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7310021040 con 0x555ff7cde330 2023-10-19T12:45:50.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.480+0000 7f730effd640 1 -- 172.21.15.57:0/254782444 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f7310014070 con 0x555ff7cde330 2023-10-19T12:45:50.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.480+0000 7f730effd640 1 -- 172.21.15.57:0/254782444 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7310005260 con 0x555ff7cde330 2023-10-19T12:45:50.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.481+0000 7f732dd7a740 1 -- 172.21.15.57:0/254782444 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x555ff7cde330 msgr2=0x555ff7cdc890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:50.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.481+0000 7f732dd7a740 1 --2- 172.21.15.57:0/254782444 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x555ff7cde330 0x555ff7cdc890 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f73100058c0 tx=0x7f7310016000 comp rx=0 tx=0).stop 2023-10-19T12:45:50.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.481+0000 7f732dd7a740 1 -- 172.21.15.57:0/254782444 shutdown_connections 2023-10-19T12:45:50.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.481+0000 7f732dd7a740 1 --2- 172.21.15.57:0/254782444 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555ff7ce3c70 0x555ff7cdcdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:50.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.481+0000 7f732dd7a740 1 --2- 172.21.15.57:0/254782444 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x555ff7cde330 0x555ff7cdc890 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:50.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.481+0000 7f732dd7a740 1 --2- 172.21.15.57:0/254782444 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x555ff7cddf50 0x555ff7ce10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:50.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.481+0000 7f732dd7a740 1 -- 172.21.15.57:0/254782444 >> 172.21.15.57:0/254782444 conn(0x555ff7cb1430 msgr2=0x555ff7d1e310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:50.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.481+0000 7f732dd7a740 1 -- 172.21.15.57:0/254782444 shutdown_connections 2023-10-19T12:45:50.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.481+0000 7f732dd7a740 1 -- 172.21.15.57:0/254782444 wait complete. 2023-10-19T12:45:50.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.481+0000 7f732dd7a740 1 -- start start 2023-10-19T12:45:50.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.482+0000 7f732dd7a740 1 Processor -- start 2023-10-19T12:45:50.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.482+0000 7f732dd7a740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x555ff7cddf50 0x555ff7d638c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:50.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.482+0000 7f732dd7a740 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x555ff7cde330 0x555ff7cd1160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:50.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.482+0000 7f732dd7a740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555ff7ce3c70 0x555ff7cf1670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:50.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.482+0000 7f732dd7a740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ff7d0b3d0 con 0x555ff7cddf50 2023-10-19T12:45:50.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.482+0000 7f732dd7a740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ff7cd1b70 con 0x555ff7ce3c70 2023-10-19T12:45:50.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.482+0000 7f732dd7a740 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x555ff7cd16a0 con 0x555ff7cde330 2023-10-19T12:45:50.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.482+0000 7f732dd7a740 10 client.0 _ll_register_callbacks cb 0x7f7318077300 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-19T12:45:50.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.482+0000 7f7328f85640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x555ff7cddf50 0x555ff7d638c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:50.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.482+0000 7f7328f85640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x555ff7cddf50 0x555ff7d638c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37562/0 (socket says 172.21.15.57:37562) 2023-10-19T12:45:50.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.482+0000 7f7328f85640 1 -- 172.21.15.57:0/2383241924 learned_addr learned my addr 172.21.15.57:0/2383241924 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:50.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.483+0000 7f7329786640 1 --2- 172.21.15.57:0/2383241924 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555ff7ce3c70 0x555ff7cf1670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:50.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.483+0000 7f7323fff640 1 --2- 172.21.15.57:0/2383241924 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x555ff7cde330 0x555ff7cd1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:50.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.483+0000 7f7328f85640 1 -- 172.21.15.57:0/2383241924 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x555ff7cde330 msgr2=0x555ff7cd1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T12:45:50.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.483+0000 7f7328f85640 1 --2- 172.21.15.57:0/2383241924 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x555ff7cde330 0x555ff7cd1160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:50.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.483+0000 7f7328f85640 1 -- 172.21.15.57:0/2383241924 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555ff7ce3c70 msgr2=0x555ff7cf1670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T12:45:50.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.483+0000 7f7328f85640 1 --2- 172.21.15.57:0/2383241924 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555ff7ce3c70 0x555ff7cf1670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:50.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.483+0000 7f7328f85640 1 -- 172.21.15.57:0/2383241924 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f731000fc40 con 0x555ff7cddf50 2023-10-19T12:45:50.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.483+0000 7f7329786640 1 --2- 172.21.15.57:0/2383241924 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555ff7ce3c70 0x555ff7cf1670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:45:50.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.483+0000 7f7328f85640 1 --2- 172.21.15.57:0/2383241924 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x555ff7cddf50 0x555ff7d638c0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f73180774c0 tx=0x7f73180ac040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bcfee7e11272141 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:50.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.483+0000 7f73037fe640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2023-10-19T12:45:50.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.483+0000 7f73037fe640 1 -- 172.21.15.57:0/2383241924 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73180aa040 con 0x555ff7cddf50 2023-10-19T12:45:50.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.483+0000 7f73037fe640 1 -- 172.21.15.57:0/2383241924 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f73180a9070 con 0x555ff7cddf50 2023-10-19T12:45:50.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.483+0000 7f732dd7a740 1 -- 172.21.15.57:0/2383241924 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7318079100 con 0x555ff7cddf50 2023-10-19T12:45:50.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.483+0000 7f73037fe640 1 -- 172.21.15.57:0/2383241924 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7318081170 con 0x555ff7cddf50 2023-10-19T12:45:50.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.483+0000 7f732dd7a740 10 client.5838 fetch_fsmap learned FSMap version 16 2023-10-19T12:45:50.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.483+0000 7f732dd7a740 1 -- 172.21.15.57:0/2383241924 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) v3 -- 0x555ff7bdb4f0 con 0x555ff7cddf50 2023-10-19T12:45:50.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.483+0000 7f73037fe640 1 -- 172.21.15.57:0/2383241924 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f7318079100 con 0x555ff7cddf50 2023-10-19T12:45:50.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.483+0000 7f73037fe640 1 -- 172.21.15.57:0/2383241924 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 16) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f731804f020 con 0x555ff7cddf50 2023-10-19T12:45:50.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.483+0000 7f732dd7a740 10 client.5838 fetch_fsmap finished waiting for FSMap version 16 2023-10-19T12:45:50.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.483+0000 7f732dd7a740 10 client.5838 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-19T12:45:50.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.483+0000 7f732dd7a740 20 client.5838 populate_metadata read hostname 'smithi057' 2023-10-19T12:45:50.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.483+0000 7f732dd7a740 1 --2- 172.21.15.57:0/2383241924 >> [v2:172.21.15.179:6835/1237463333,v1:172.21.15.179:6837/1237463333] conn(0x555ff7cf1bb0 0x555ff7d0da40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:50.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.483+0000 7f732dd7a740 4 client.5838 mds_command: new command op to 5634 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-19T12:45:50.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.483+0000 7f732dd7a740 1 -- 172.21.15.57:0/2383241924 --> [v2:172.21.15.179:6835/1237463333,v1:172.21.15.179:6837/1237463333] -- command(tid 0: {"prefix": "status"}) v1 -- 0x555ff791a6c0 con 0x555ff7cf1bb0 2023-10-19T12:45:50.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.484+0000 7f7323fff640 1 --2- 172.21.15.57:0/2383241924 >> [v2:172.21.15.179:6835/1237463333,v1:172.21.15.179:6837/1237463333] conn(0x555ff7cf1bb0 0x555ff7d0da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:50.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.484+0000 7f73037fe640 10 client.5838 ms_handle_connect on v2:172.21.15.179:6835/1237463333 2023-10-19T12:45:50.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.484+0000 7f7323fff640 1 --2- 172.21.15.57:0/2383241924 >> [v2:172.21.15.179:6835/1237463333,v1:172.21.15.179:6837/1237463333] conn(0x555ff7cf1bb0 0x555ff7d0da40 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8215505c024a9f92 server_cookie=bea025c20da04273 in_seq=0 out_seq=0 2023-10-19T12:45:50.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.484+0000 7f73037fe640 1 -- 172.21.15.57:0/2383241924 <== mds.0 v2:172.21.15.179:6835/1237463333 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x555ff791a6c0 con 0x555ff7cf1bb0 2023-10-19T12:45:50.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.484+0000 7f73037fe640 10 client.5838 handle_command_reply: tid=0 2023-10-19T12:45:50.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.485+0000 7f732dd7a740 1 client.5838 shutdown 2023-10-19T12:45:50.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.485+0000 7f732dd7a740 1 -- 172.21.15.57:0/2383241924 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x555ff7cddf50 msgr2=0x555ff7d638c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:50.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.485+0000 7f732dd7a740 1 --2- 172.21.15.57:0/2383241924 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x555ff7cddf50 0x555ff7d638c0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f73180774c0 tx=0x7f73180ac040 comp rx=0 tx=0).stop 2023-10-19T12:45:50.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.485+0000 7f732dd7a740 1 -- 172.21.15.57:0/2383241924 shutdown_connections 2023-10-19T12:45:50.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.485+0000 7f732dd7a740 1 --2- 172.21.15.57:0/2383241924 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555ff7ce3c70 0x555ff7cf1670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:50.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.485+0000 7f732dd7a740 1 --2- 172.21.15.57:0/2383241924 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x555ff7cde330 0x555ff7cd1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:50.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.485+0000 7f732dd7a740 1 --2- 172.21.15.57:0/2383241924 >> [v2:172.21.15.179:6835/1237463333,v1:172.21.15.179:6837/1237463333] conn(0x555ff7cf1bb0 0x555ff7d0da40 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:50.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.485+0000 7f732dd7a740 1 --2- 172.21.15.57:0/2383241924 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x555ff7cddf50 0x555ff7d638c0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:50.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.485+0000 7f73037fe640 0 client.5838 ms_handle_reset on v2:172.21.15.179:6835/1237463333 2023-10-19T12:45:50.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.485+0000 7f732dd7a740 1 -- 172.21.15.57:0/2383241924 >> 172.21.15.57:0/2383241924 conn(0x555ff7cb1430 msgr2=0x555ff7cff1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:50.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.485+0000 7f732dd7a740 1 -- 172.21.15.57:0/2383241924 shutdown_connections 2023-10-19T12:45:50.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.485+0000 7f732dd7a740 1 -- 172.21.15.57:0/2383241924 wait complete. 2023-10-19T12:45:50.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.485+0000 7f732dd7a740 20 client.5838 trim_cache size 0 max 16384 2023-10-19T12:45:50.504 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2023-10-19T12:45:50.504 INFO:teuthology.orchestra.run.smithi057.stdout: "cluster_fsid": "1e9beb19-3260-4dd7-85ef-5b399ce774ac", 2023-10-19T12:45:50.505 INFO:teuthology.orchestra.run.smithi057.stdout: "whoami": 0, 2023-10-19T12:45:50.505 INFO:teuthology.orchestra.run.smithi057.stdout: "id": 5634, 2023-10-19T12:45:50.505 INFO:teuthology.orchestra.run.smithi057.stdout: "want_state": "up:active", 2023-10-19T12:45:50.505 INFO:teuthology.orchestra.run.smithi057.stdout: "state": "up:active", 2023-10-19T12:45:50.505 INFO:teuthology.orchestra.run.smithi057.stdout: "fs_name": "cephfs", 2023-10-19T12:45:50.506 INFO:teuthology.orchestra.run.smithi057.stdout: "rank_uptime": 9.1449260589999994, 2023-10-19T12:45:50.506 INFO:teuthology.orchestra.run.smithi057.stdout: "mdsmap_epoch": 16, 2023-10-19T12:45:50.506 INFO:teuthology.orchestra.run.smithi057.stdout: "osdmap_epoch": 39, 2023-10-19T12:45:50.506 INFO:teuthology.orchestra.run.smithi057.stdout: "osdmap_epoch_barrier": 39, 2023-10-19T12:45:50.506 INFO:teuthology.orchestra.run.smithi057.stdout: "uptime": 16.348255996999999 2023-10-19T12:45:50.507 INFO:teuthology.orchestra.run.smithi057.stdout:} 2023-10-19T12:45:50.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.485+0000 7f732b1e9640 1 -- 172.21.15.57:0/1740029357 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f730806ae40 msgr2=0x7f730806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:50.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.485+0000 7f732b1e9640 1 --2- 172.21.15.57:0/1740029357 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f730806ae40 0x7f730806d300 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f73241f2330 tx=0x7f731000a420 comp rx=0 tx=0).stop 2023-10-19T12:45:50.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.485+0000 7f732b1e9640 1 -- 172.21.15.57:0/1740029357 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f732406ac70 msgr2=0x7f7324156760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:50.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.485+0000 7f732b1e9640 1 --2- 172.21.15.57:0/1740029357 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f732406ac70 0x7f7324156760 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f7318036040 tx=0x7f731802dd60 comp rx=0 tx=0).stop 2023-10-19T12:45:50.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.485+0000 7f732b1e9640 1 -- 172.21.15.57:0/1740029357 shutdown_connections 2023-10-19T12:45:50.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.485+0000 7f732b1e9640 1 --2- 172.21.15.57:0/1740029357 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f730806ae40 0x7f730806d300 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:50.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.485+0000 7f732b1e9640 1 --2- 172.21.15.57:0/1740029357 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f732415c1f0 0x7f73240a64b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:50.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.485+0000 7f732b1e9640 1 --2- 172.21.15.57:0/1740029357 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f73241579f0 0x7f732414fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:50.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.485+0000 7f732b1e9640 1 --2- 172.21.15.57:0/1740029357 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f732406ac70 0x7f7324156760 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:50.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.485+0000 7f732b1e9640 1 -- 172.21.15.57:0/1740029357 >> 172.21.15.57:0/1740029357 conn(0x7f732406fae0 msgr2=0x7f7324159120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:50.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.486+0000 7f732b1e9640 1 -- 172.21.15.57:0/1740029357 shutdown_connections 2023-10-19T12:45:50.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.486+0000 7f732b1e9640 1 -- 172.21.15.57:0/1740029357 wait complete. 2023-10-19T12:45:50.510 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-19T12:45:50.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.616+0000 7ffa877b7640 1 -- 172.21.15.57:0/1766616646 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffa80140660 msgr2=0x7ffa800c97c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:50.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.616+0000 7ffa877b7640 1 --2- 172.21.15.57:0/1766616646 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffa80140660 0x7ffa800c97c0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7ffa70009830 tx=0x7ffa7002bd60 comp rx=0 tx=0).stop 2023-10-19T12:45:50.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.616+0000 7ffa877b7640 1 -- 172.21.15.57:0/1766616646 shutdown_connections 2023-10-19T12:45:50.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.616+0000 7ffa877b7640 1 --2- 172.21.15.57:0/1766616646 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ffa800c9d00 0x7ffa8006b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:50.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.616+0000 7ffa877b7640 1 --2- 172.21.15.57:0/1766616646 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffa80140660 0x7ffa800c97c0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:50.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.616+0000 7ffa877b7640 1 -- 172.21.15.57:0/1766616646 >> 172.21.15.57:0/1766616646 conn(0x7ffa80069930 msgr2=0x7ffa80069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:50.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.616+0000 7ffa877b7640 1 -- 172.21.15.57:0/1766616646 shutdown_connections 2023-10-19T12:45:50.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.616+0000 7ffa877b7640 1 -- 172.21.15.57:0/1766616646 wait complete. 2023-10-19T12:45:50.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.617+0000 7ffa877b7640 1 Processor -- start 2023-10-19T12:45:50.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.617+0000 7ffa877b7640 1 -- start start 2023-10-19T12:45:50.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.618+0000 7ffa877b7640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffa800c9d00 0x7ffa80157d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:50.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.618+0000 7ffa877b7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffa8013d360 0x7ffa80158b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:50.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.618+0000 7ffa877b7640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ffa80140660 0x7ffa80159090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:50.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.618+0000 7ffa877b7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa80134a50 con 0x7ffa8013d360 2023-10-19T12:45:50.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.618+0000 7ffa877b7640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa80158270 con 0x7ffa800c9d00 2023-10-19T12:45:50.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.618+0000 7ffa877b7640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa80135d40 con 0x7ffa80140660 2023-10-19T12:45:50.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.618+0000 7ffa84d52640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffa8013d360 0x7ffa80158b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:50.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.618+0000 7ffa84d52640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffa8013d360 0x7ffa80158b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37576/0 (socket says 172.21.15.57:37576) 2023-10-19T12:45:50.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.618+0000 7ffa84d52640 1 -- 172.21.15.57:0/3616964657 learned_addr learned my addr 172.21.15.57:0/3616964657 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:50.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.618+0000 7ffa85d54640 1 --2- 172.21.15.57:0/3616964657 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ffa80140660 0x7ffa80159090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:50.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.618+0000 7ffa85553640 1 --2- 172.21.15.57:0/3616964657 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffa800c9d00 0x7ffa80157d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:50.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.618+0000 7ffa84d52640 1 -- 172.21.15.57:0/3616964657 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ffa80140660 msgr2=0x7ffa80159090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:50.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.618+0000 7ffa84d52640 1 --2- 172.21.15.57:0/3616964657 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ffa80140660 0x7ffa80159090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:50.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.618+0000 7ffa84d52640 1 -- 172.21.15.57:0/3616964657 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffa800c9d00 msgr2=0x7ffa80157d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:50.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.618+0000 7ffa84d52640 1 --2- 172.21.15.57:0/3616964657 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffa800c9d00 0x7ffa80157d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:50.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.618+0000 7ffa84d52640 1 -- 172.21.15.57:0/3616964657 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa70009490 con 0x7ffa8013d360 2023-10-19T12:45:50.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.618+0000 7ffa85553640 1 --2- 172.21.15.57:0/3616964657 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffa800c9d00 0x7ffa80157d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:45:50.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.618+0000 7ffa84d52640 1 --2- 172.21.15.57:0/3616964657 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffa8013d360 0x7ffa80158b50 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7ffa70000c00 tx=0x7ffa700038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:50.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.619+0000 7ffa75ffb640 1 -- 172.21.15.57:0/3616964657 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa70003ea0 con 0x7ffa8013d360 2023-10-19T12:45:50.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.619+0000 7ffa75ffb640 1 -- 172.21.15.57:0/3616964657 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7ffa70034440 con 0x7ffa8013d360 2023-10-19T12:45:50.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.619+0000 7ffa877b7640 1 -- 172.21.15.57:0/3616964657 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa80134ca0 con 0x7ffa8013d360 2023-10-19T12:45:50.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.619+0000 7ffa75ffb640 1 -- 172.21.15.57:0/3616964657 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa70034850 con 0x7ffa8013d360 2023-10-19T12:45:50.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.619+0000 7ffa877b7640 1 -- 172.21.15.57:0/3616964657 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa80135260 con 0x7ffa8013d360 2023-10-19T12:45:50.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.620+0000 7ffa877b7640 1 -- 172.21.15.57:0/3616964657 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa800690e0 con 0x7ffa8013d360 2023-10-19T12:45:50.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.620+0000 7ffa75ffb640 1 -- 172.21.15.57:0/3616964657 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7ffa700349b0 con 0x7ffa8013d360 2023-10-19T12:45:50.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.621+0000 7ffa75ffb640 1 --2- 172.21.15.57:0/3616964657 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7ffa6406ae40 0x7ffa6406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:50.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.621+0000 7ffa75ffb640 1 -- 172.21.15.57:0/3616964657 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7ffa700abf00 con 0x7ffa8013d360 2023-10-19T12:45:50.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.621+0000 7ffa85553640 1 --2- 172.21.15.57:0/3616964657 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7ffa6406ae40 0x7ffa6406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:50.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.621+0000 7ffa85553640 1 --2- 172.21.15.57:0/3616964657 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7ffa6406ae40 0x7ffa6406d300 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7ffa780021f0 tx=0x7ffa78009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:50.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.625+0000 7ffa75ffb640 1 -- 172.21.15.57:0/3616964657 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ffa70076e80 con 0x7ffa8013d360 2023-10-19T12:45:50.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.836+0000 7ffa877b7640 1 -- 172.21.15.57:0/3616964657 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7ffa8015a170 con 0x7ffa8013d360 2023-10-19T12:45:50.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.837+0000 7ffa75ffb640 1 -- 172.21.15.57:0/3616964657 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) v1 ==== 53+0+73 (secure 0 0 0) 0x7ffa70039060 con 0x7ffa8013d360 2023-10-19T12:45:50.840 INFO:teuthology.orchestra.run.smithi057.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-19T12:45:50.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.839+0000 7ffa877b7640 1 -- 172.21.15.57:0/3616964657 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7ffa6406ae40 msgr2=0x7ffa6406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:50.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.839+0000 7ffa877b7640 1 --2- 172.21.15.57:0/3616964657 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7ffa6406ae40 0x7ffa6406d300 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7ffa780021f0 tx=0x7ffa78009040 comp rx=0 tx=0).stop 2023-10-19T12:45:50.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.839+0000 7ffa877b7640 1 -- 172.21.15.57:0/3616964657 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffa8013d360 msgr2=0x7ffa80158b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:50.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.839+0000 7ffa877b7640 1 --2- 172.21.15.57:0/3616964657 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffa8013d360 0x7ffa80158b50 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7ffa70000c00 tx=0x7ffa700038a0 comp rx=0 tx=0).stop 2023-10-19T12:45:50.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.839+0000 7ffa877b7640 1 -- 172.21.15.57:0/3616964657 shutdown_connections 2023-10-19T12:45:50.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.839+0000 7ffa877b7640 1 --2- 172.21.15.57:0/3616964657 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7ffa6406ae40 0x7ffa6406d300 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:50.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.839+0000 7ffa877b7640 1 --2- 172.21.15.57:0/3616964657 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ffa80140660 0x7ffa80159090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:50.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.839+0000 7ffa877b7640 1 --2- 172.21.15.57:0/3616964657 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffa8013d360 0x7ffa80158b50 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:50.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.839+0000 7ffa877b7640 1 --2- 172.21.15.57:0/3616964657 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffa800c9d00 0x7ffa80157d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:50.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.839+0000 7ffa877b7640 1 -- 172.21.15.57:0/3616964657 >> 172.21.15.57:0/3616964657 conn(0x7ffa80069930 msgr2=0x7ffa8013e450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:50.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.839+0000 7ffa877b7640 1 -- 172.21.15.57:0/3616964657 shutdown_connections 2023-10-19T12:45:50.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:50.839+0000 7ffa877b7640 1 -- 172.21.15.57:0/3616964657 wait complete. 2023-10-19T12:45:50.852 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-10-19T12:45:50.853 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-19T12:45:50.853 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi057.front.sepia.ceph.com 2023-10-19T12:45:50.853 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-19T12:45:50.854 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-19T12:45:50.854 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-10-19T12:45:50.854 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-10-19T12:45:50.854 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-19T12:45:50.855 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-19T12:45:50.855 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:45:50.856 DEBUG:teuthology.orchestra.run.smithi057:> ip addr 2023-10-19T12:45:50.871 INFO:teuthology.orchestra.run.smithi057.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-19T12:45:50.871 INFO:teuthology.orchestra.run.smithi057.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-19T12:45:50.871 INFO:teuthology.orchestra.run.smithi057.stdout: inet 127.0.0.1/8 scope host lo 2023-10-19T12:45:50.872 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T12:45:50.872 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 ::1/128 scope host 2023-10-19T12:45:50.872 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T12:45:50.872 INFO:teuthology.orchestra.run.smithi057.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-19T12:45:50.872 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:6b:ff:86 brd ff:ff:ff:ff:ff:ff 2023-10-19T12:45:50.872 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp7s0f0 2023-10-19T12:45:50.873 INFO:teuthology.orchestra.run.smithi057.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-19T12:45:50.873 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:6b:ff:87 brd ff:ff:ff:ff:ff:ff 2023-10-19T12:45:50.873 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp7s0f1 2023-10-19T12:45:50.873 INFO:teuthology.orchestra.run.smithi057.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-19T12:45:50.873 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:bd:15:62 brd ff:ff:ff:ff:ff:ff 2023-10-19T12:45:50.873 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp1s0f0 2023-10-19T12:45:50.874 INFO:teuthology.orchestra.run.smithi057.stdout: inet 172.21.15.57/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-19T12:45:50.874 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft 42560sec preferred_lft 42560sec 2023-10-19T12:45:50.874 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 fe80::ec4:7aff:febd:1562/64 scope link noprefixroute 2023-10-19T12:45:50.874 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T12:45:50.874 INFO:teuthology.orchestra.run.smithi057.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-19T12:45:50.875 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:bd:15:63 brd ff:ff:ff:ff:ff:ff 2023-10-19T12:45:50.875 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp1s0f1 2023-10-19T12:45:50.875 INFO:teuthology.orchestra.run.smithi057.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-19T12:45:50.875 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0e:fe:f1:b5:41:59 brd ff:ff:ff:ff:ff:ff 2023-10-19T12:45:50.875 INFO:teuthology.orchestra.run.smithi057.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-19T12:45:50.875 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T12:45:50.876 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 fe80::64b1:a2ff:feb3:b08/64 scope link 2023-10-19T12:45:50.876 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T12:45:50.876 INFO:teuthology.orchestra.run.smithi057.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-19T12:45:50.876 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0e:fe:f1:b5:41:59 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T12:45:50.876 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 fe80::cfe:f1ff:feb5:4159/64 scope link 2023-10-19T12:45:50.876 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T12:45:50.877 INFO:teuthology.orchestra.run.smithi057.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-19T12:45:50.877 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 62:53:72:9d:52:71 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-19T12:45:50.877 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 fe80::6053:72ff:fe9d:5271/64 scope link 2023-10-19T12:45:50.877 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T12:45:50.877 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:45:50.878 DEBUG:teuthology.orchestra.run.smithi057:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-19T12:45:50.941 INFO:teuthology.orchestra.run.smithi057.stdout:1 2023-10-19T12:45:50.942 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:45:50.943 DEBUG:teuthology.orchestra.run.smithi057:> route 2023-10-19T12:45:50.998 INFO:teuthology.orchestra.run.smithi057.stdout:Kernel IP routing table 2023-10-19T12:45:50.999 INFO:teuthology.orchestra.run.smithi057.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-19T12:45:50.999 INFO:teuthology.orchestra.run.smithi057.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-19T12:45:50.999 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-19T12:45:50.999 INFO:teuthology.orchestra.run.smithi057.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-19T12:45:51.000 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:45:51.000 DEBUG:teuthology.orchestra.run.smithi057:> (cd / && exec bash -c ' 2023-10-19T12:45:51.001 DEBUG:teuthology.orchestra.run.smithi057:> set -e 2023-10-19T12:45:51.001 DEBUG:teuthology.orchestra.run.smithi057:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-19T12:45:51.001 DEBUG:teuthology.orchestra.run.smithi057:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-19T12:45:51.002 DEBUG:teuthology.orchestra.run.smithi057:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-19T12:45:51.002 DEBUG:teuthology.orchestra.run.smithi057:> ') 2023-10-19T12:45:51.105 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:45:51.105 DEBUG:teuthology.orchestra.run.smithi057:> ip netns list 2023-10-19T12:45:51.121 INFO:teuthology.orchestra.run.smithi057.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-19T12:45:51.121 INFO:teuthology.orchestra.run.smithi057.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-19T12:45:51.121 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-19T12:45:51.122 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:45:51.122 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip link set brx.0 up 2023-10-19T12:45:51.191 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-10-19T12:45:51.191 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-19T12:45:51.192 DEBUG:teuthology.orchestra.run.smithi057:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-19T12:45:51.246 INFO:teuthology.orchestra.run.smithi057.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-19T12:45:51.246 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-19T12:45:51.246 DEBUG:teuthology.orchestra.run.smithi057:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-19T12:45:51.300 DEBUG:teuthology.orchestra.run.smithi057:> sudo modprobe fuse 2023-10-19T12:45:51.371 DEBUG:teuthology.orchestra.run.smithi057:> cat /proc/self/mounts | awk '{print $2}' 2023-10-19T12:45:51.427 INFO:teuthology.orchestra.run.smithi057.stdout:/proc 2023-10-19T12:45:51.428 INFO:teuthology.orchestra.run.smithi057.stdout:/sys 2023-10-19T12:45:51.428 INFO:teuthology.orchestra.run.smithi057.stdout:/dev 2023-10-19T12:45:51.428 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/security 2023-10-19T12:45:51.428 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/shm 2023-10-19T12:45:51.428 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/pts 2023-10-19T12:45:51.429 INFO:teuthology.orchestra.run.smithi057.stdout:/run 2023-10-19T12:45:51.429 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/cgroup 2023-10-19T12:45:51.429 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/pstore 2023-10-19T12:45:51.429 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/bpf 2023-10-19T12:45:51.429 INFO:teuthology.orchestra.run.smithi057.stdout:/ 2023-10-19T12:45:51.430 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/selinux 2023-10-19T12:45:51.430 INFO:teuthology.orchestra.run.smithi057.stdout:/proc/sys/fs/binfmt_misc 2023-10-19T12:45:51.430 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/hugepages 2023-10-19T12:45:51.430 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/mqueue 2023-10-19T12:45:51.430 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/debug 2023-10-19T12:45:51.431 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/tracing 2023-10-19T12:45:51.431 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/fuse/connections 2023-10-19T12:45:51.431 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/config 2023-10-19T12:45:51.431 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-sysctl.service 2023-10-19T12:45:51.432 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-10-19T12:45:51.432 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-10-19T12:45:51.432 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/nfs/rpc_pipefs 2023-10-19T12:45:51.432 INFO:teuthology.orchestra.run.smithi057.stdout:/run/user/1000 2023-10-19T12:45:51.432 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph 2023-10-19T12:45:51.432 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-0 2023-10-19T12:45:51.433 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-1 2023-10-19T12:45:51.433 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-2 2023-10-19T12:45:51.433 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-3 2023-10-19T12:45:51.433 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns 2023-10-19T12:45:51.433 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T12:45:51.434 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T12:45:51.434 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-19T12:45:51.434 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-19T12:45:51.434 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:45:51.435 DEBUG:teuthology.orchestra.run.smithi057:> ls /sys/fs/fuse/connections 2023-10-19T12:45:51.482 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-10-19T12:45:51.483 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-10-19T12:45:51.525 DEBUG:teuthology.orchestra.run.smithi057:> sudo modprobe fuse 2023-10-19T12:45:51.555 DEBUG:teuthology.orchestra.run.smithi057:> cat /proc/self/mounts | awk '{print $2}' 2023-10-19T12:45:51.611 INFO:teuthology.orchestra.run.smithi057.stdout:/proc 2023-10-19T12:45:51.612 INFO:teuthology.orchestra.run.smithi057.stdout:/sys 2023-10-19T12:45:51.612 INFO:teuthology.orchestra.run.smithi057.stdout:/dev 2023-10-19T12:45:51.612 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/security 2023-10-19T12:45:51.612 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/shm 2023-10-19T12:45:51.612 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/pts 2023-10-19T12:45:51.613 INFO:teuthology.orchestra.run.smithi057.stdout:/run 2023-10-19T12:45:51.613 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/cgroup 2023-10-19T12:45:51.613 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/pstore 2023-10-19T12:45:51.613 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/bpf 2023-10-19T12:45:51.613 INFO:teuthology.orchestra.run.smithi057.stdout:/ 2023-10-19T12:45:51.614 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/selinux 2023-10-19T12:45:51.614 INFO:teuthology.orchestra.run.smithi057.stdout:/proc/sys/fs/binfmt_misc 2023-10-19T12:45:51.614 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/hugepages 2023-10-19T12:45:51.614 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/mqueue 2023-10-19T12:45:51.614 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/debug 2023-10-19T12:45:51.614 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/tracing 2023-10-19T12:45:51.615 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/fuse/connections 2023-10-19T12:45:51.615 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/config 2023-10-19T12:45:51.615 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-sysctl.service 2023-10-19T12:45:51.615 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-10-19T12:45:51.615 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-10-19T12:45:51.616 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/nfs/rpc_pipefs 2023-10-19T12:45:51.616 INFO:teuthology.orchestra.run.smithi057.stdout:/run/user/1000 2023-10-19T12:45:51.616 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph 2023-10-19T12:45:51.616 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-0 2023-10-19T12:45:51.616 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-1 2023-10-19T12:45:51.616 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-2 2023-10-19T12:45:51.617 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-3 2023-10-19T12:45:51.617 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns 2023-10-19T12:45:51.617 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T12:45:51.617 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T12:45:51.617 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-19T12:45:51.618 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-19T12:45:51.618 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:45:51.618 DEBUG:teuthology.orchestra.run.smithi057:> ls /sys/fs/fuse/connections 2023-10-19T12:45:51.619 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi057.stderr:2023-10-19T12:45:51.614+0000 7fa1a3a3d480 -1 init, newargv = 0x55e5f8c31c30 newargc=15 2023-10-19T12:45:51.620 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi057.stderr:ceph-fuse[105539]: starting ceph client 2023-10-19T12:45:51.622 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi057.stderr:ceph-fuse[105539]: starting fuse 2023-10-19T12:45:51.635 INFO:teuthology.orchestra.run.smithi057.stdout:43 2023-10-19T12:45:51.635 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2023-10-19T12:45:51.636 DEBUG:teuthology.orchestra.run.smithi057:> sudo stdin-killer -- python3 -c ' 2023-10-19T12:45:51.636 DEBUG:teuthology.orchestra.run.smithi057:> import glob 2023-10-19T12:45:51.636 DEBUG:teuthology.orchestra.run.smithi057:> import re 2023-10-19T12:45:51.637 DEBUG:teuthology.orchestra.run.smithi057:> import os 2023-10-19T12:45:51.637 DEBUG:teuthology.orchestra.run.smithi057:> import subprocess 2023-10-19T12:45:51.637 DEBUG:teuthology.orchestra.run.smithi057:> 2023-10-19T12:45:51.638 DEBUG:teuthology.orchestra.run.smithi057:> def _find_admin_socket(client_name): 2023-10-19T12:45:51.638 DEBUG:teuthology.orchestra.run.smithi057:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-10-19T12:45:51.639 DEBUG:teuthology.orchestra.run.smithi057:> files = glob.glob(asok_path) 2023-10-19T12:45:51.639 DEBUG:teuthology.orchestra.run.smithi057:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-10-19T12:45:51.639 DEBUG:teuthology.orchestra.run.smithi057:> 2023-10-19T12:45:51.640 DEBUG:teuthology.orchestra.run.smithi057:> # Given a non-glob path, it better be there 2023-10-19T12:45:51.640 DEBUG:teuthology.orchestra.run.smithi057:> if "*" not in asok_path: 2023-10-19T12:45:51.641 DEBUG:teuthology.orchestra.run.smithi057:> assert(len(files) == 1) 2023-10-19T12:45:51.641 DEBUG:teuthology.orchestra.run.smithi057:> return files[0] 2023-10-19T12:45:51.641 DEBUG:teuthology.orchestra.run.smithi057:> 2023-10-19T12:45:51.642 DEBUG:teuthology.orchestra.run.smithi057:> for f in files: 2023-10-19T12:45:51.642 DEBUG:teuthology.orchestra.run.smithi057:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-10-19T12:45:51.642 DEBUG:teuthology.orchestra.run.smithi057:> if os.path.exists("/proc/{0}".format(pid)): 2023-10-19T12:45:51.643 DEBUG:teuthology.orchestra.run.smithi057:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-10-19T12:45:51.643 DEBUG:teuthology.orchestra.run.smithi057:> contents = proc_f.read() 2023-10-19T12:45:51.643 DEBUG:teuthology.orchestra.run.smithi057:> if mountpoint in contents: 2023-10-19T12:45:51.644 DEBUG:teuthology.orchestra.run.smithi057:> return f 2023-10-19T12:45:51.644 DEBUG:teuthology.orchestra.run.smithi057:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-10-19T12:45:51.644 DEBUG:teuthology.orchestra.run.smithi057:> 2023-10-19T12:45:51.645 DEBUG:teuthology.orchestra.run.smithi057:> print(_find_admin_socket("client.0")) 2023-10-19T12:45:51.645 DEBUG:teuthology.orchestra.run.smithi057:> ' 2023-10-19T12:45:51.759 INFO:teuthology.orchestra.run.smithi057.stdout:/var/run/ceph/ceph-client.0.105539.asok 2023-10-19T12:45:51.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2023-10-19T12:45:51.769 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.105539.asok 2023-10-19T12:45:51.769 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:45:51.770 DEBUG:teuthology.orchestra.run.smithi057:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.105539.asok status 2023-10-19T12:45:51.910 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2023-10-19T12:45:51.911 INFO:teuthology.orchestra.run.smithi057.stdout: "metadata": { 2023-10-19T12:45:51.911 INFO:teuthology.orchestra.run.smithi057.stdout: "ceph_sha1": "f7420d7f8e55ceea786dcbf536c338ce920c21eb", 2023-10-19T12:45:51.911 INFO:teuthology.orchestra.run.smithi057.stdout: "ceph_version": "ceph version 18.0.0-6786-gf7420d7f (f7420d7f8e55ceea786dcbf536c338ce920c21eb) reef (dev)", 2023-10-19T12:45:51.911 INFO:teuthology.orchestra.run.smithi057.stdout: "entity_id": "0", 2023-10-19T12:45:51.911 INFO:teuthology.orchestra.run.smithi057.stdout: "hostname": "smithi057", 2023-10-19T12:45:51.912 INFO:teuthology.orchestra.run.smithi057.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-10-19T12:45:51.912 INFO:teuthology.orchestra.run.smithi057.stdout: "pid": "105539", 2023-10-19T12:45:51.912 INFO:teuthology.orchestra.run.smithi057.stdout: "root": "/" 2023-10-19T12:45:51.912 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T12:45:51.912 INFO:teuthology.orchestra.run.smithi057.stdout: "dentry_count": 0, 2023-10-19T12:45:51.913 INFO:teuthology.orchestra.run.smithi057.stdout: "dentry_pinned_count": 0, 2023-10-19T12:45:51.913 INFO:teuthology.orchestra.run.smithi057.stdout: "id": 5853, 2023-10-19T12:45:51.913 INFO:teuthology.orchestra.run.smithi057.stdout: "inst": { 2023-10-19T12:45:51.913 INFO:teuthology.orchestra.run.smithi057.stdout: "name": { 2023-10-19T12:45:51.913 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "client", 2023-10-19T12:45:51.913 INFO:teuthology.orchestra.run.smithi057.stdout: "num": 5853 2023-10-19T12:45:51.914 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T12:45:51.914 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": { 2023-10-19T12:45:51.914 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "v1", 2023-10-19T12:45:51.914 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": "192.168.0.1:0", 2023-10-19T12:45:51.914 INFO:teuthology.orchestra.run.smithi057.stdout: "nonce": 552210579 2023-10-19T12:45:51.915 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T12:45:51.915 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T12:45:51.915 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": { 2023-10-19T12:45:51.915 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "v1", 2023-10-19T12:45:51.915 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": "192.168.0.1:0", 2023-10-19T12:45:51.916 INFO:teuthology.orchestra.run.smithi057.stdout: "nonce": 552210579 2023-10-19T12:45:51.916 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T12:45:51.916 INFO:teuthology.orchestra.run.smithi057.stdout: "inst_str": "client.5853 192.168.0.1:0/552210579", 2023-10-19T12:45:51.916 INFO:teuthology.orchestra.run.smithi057.stdout: "addr_str": "192.168.0.1:0/552210579", 2023-10-19T12:45:51.916 INFO:teuthology.orchestra.run.smithi057.stdout: "inode_count": 1, 2023-10-19T12:45:51.917 INFO:teuthology.orchestra.run.smithi057.stdout: "mds_epoch": 16, 2023-10-19T12:45:51.917 INFO:teuthology.orchestra.run.smithi057.stdout: "osd_epoch": 39, 2023-10-19T12:45:51.917 INFO:teuthology.orchestra.run.smithi057.stdout: "osd_epoch_barrier": 0, 2023-10-19T12:45:51.917 INFO:teuthology.orchestra.run.smithi057.stdout: "blocklisted": false, 2023-10-19T12:45:51.917 INFO:teuthology.orchestra.run.smithi057.stdout: "fs_name": "cephfs" 2023-10-19T12:45:51.918 INFO:teuthology.orchestra.run.smithi057.stdout:} 2023-10-19T12:45:51.919 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:45:51.919 DEBUG:teuthology.orchestra.run.smithi057:> stat --file-system '--printf=%T 2023-10-19T12:45:51.920 DEBUG:teuthology.orchestra.run.smithi057:> ' -- /home/ubuntu/cephtest/mnt.0 2023-10-19T12:45:51.935 INFO:teuthology.orchestra.run.smithi057.stdout:fuseblk 2023-10-19T12:45:51.935 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-10-19T12:45:51.936 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:45:51.936 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-19T12:45:52.008 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-19T12:45:52.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.155+0000 7fc65359e640 1 -- 172.21.15.57:0/1980390058 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc63c0039a0 msgr2=0x7fc63c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:52.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.155+0000 7fc65359e640 1 --2- 172.21.15.57:0/1980390058 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc63c0039a0 0x7fc63c003dc0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fc64400c260 tx=0x7fc63c0057f0 comp rx=0 tx=0).stop 2023-10-19T12:45:52.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.155+0000 7fc652d9d640 1 -- 172.21.15.57:0/1980390058 reap_dead start 2023-10-19T12:45:52.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.155+0000 7fc65359e640 1 -- 172.21.15.57:0/1980390058 shutdown_connections 2023-10-19T12:45:52.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.155+0000 7fc65359e640 1 -- 172.21.15.57:0/1980390058 >> 172.21.15.57:0/1980390058 conn(0x7fc654099eb0 msgr2=0x7fc6540d0c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:52.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.155+0000 7fc65359e640 1 -- 172.21.15.57:0/1980390058 shutdown_connections 2023-10-19T12:45:52.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.155+0000 7fc65359e640 1 -- 172.21.15.57:0/1980390058 wait complete. 2023-10-19T12:45:52.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.156+0000 7fc65359e640 1 Processor -- start 2023-10-19T12:45:52.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.156+0000 7fc65359e640 1 -- start start 2023-10-19T12:45:52.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.157+0000 7fc65359e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc63c0039a0 0x7fc654152370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:52.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.157+0000 7fc65359e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc6541f5570 0x7fc65415b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:52.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.157+0000 7fc65359e640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc65415cf40 0x7fc6541f4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:52.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.157+0000 7fc65359e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc65415bea0 con 0x7fc63c0039a0 2023-10-19T12:45:52.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.157+0000 7fc65359e640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc65415c180 con 0x7fc6541f5570 2023-10-19T12:45:52.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.157+0000 7fc65359e640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc65415c460 con 0x7fc65415cf40 2023-10-19T12:45:52.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.157+0000 7fc65259c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc63c0039a0 0x7fc654152370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:52.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.157+0000 7fc65259c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc63c0039a0 0x7fc654152370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37602/0 (socket says 172.21.15.57:37602) 2023-10-19T12:45:52.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.157+0000 7fc65259c640 1 -- 172.21.15.57:0/624654289 learned_addr learned my addr 172.21.15.57:0/624654289 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:52.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.157+0000 7fc65259c640 1 -- 172.21.15.57:0/624654289 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc65415cf40 msgr2=0x7fc6541f4960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:52.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.157+0000 7fc65259c640 1 --2- 172.21.15.57:0/624654289 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc65415cf40 0x7fc6541f4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:52.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.157+0000 7fc65259c640 1 -- 172.21.15.57:0/624654289 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc6541f5570 msgr2=0x7fc65415b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:52.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.157+0000 7fc65259c640 1 --2- 172.21.15.57:0/624654289 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc6541f5570 0x7fc65415b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:52.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.157+0000 7fc65259c640 1 -- 172.21.15.57:0/624654289 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc65415c740 con 0x7fc63c0039a0 2023-10-19T12:45:52.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.158+0000 7fc65259c640 1 --2- 172.21.15.57:0/624654289 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc63c0039a0 0x7fc654152370 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fc64400c380 tx=0x7fc6440015f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:52.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.158+0000 7fc642ffd640 1 -- 172.21.15.57:0/624654289 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc644006440 con 0x7fc63c0039a0 2023-10-19T12:45:52.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.158+0000 7fc642ffd640 1 -- 172.21.15.57:0/624654289 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fc6440065a0 con 0x7fc63c0039a0 2023-10-19T12:45:52.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.158+0000 7fc65359e640 1 -- 172.21.15.57:0/624654289 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc65415ca20 con 0x7fc63c0039a0 2023-10-19T12:45:52.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.158+0000 7fc642ffd640 1 -- 172.21.15.57:0/624654289 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc644003850 con 0x7fc63c0039a0 2023-10-19T12:45:52.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.158+0000 7fc65359e640 1 -- 172.21.15.57:0/624654289 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc65415bae0 con 0x7fc63c0039a0 2023-10-19T12:45:52.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.159+0000 7fc65359e640 1 -- 172.21.15.57:0/624654289 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc654066960 con 0x7fc63c0039a0 2023-10-19T12:45:52.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.159+0000 7fc642ffd640 1 -- 172.21.15.57:0/624654289 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fc644031030 con 0x7fc63c0039a0 2023-10-19T12:45:52.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.160+0000 7fc642ffd640 1 --2- 172.21.15.57:0/624654289 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fc63406ae40 0x7fc63406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:52.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.160+0000 7fc642ffd640 1 -- 172.21.15.57:0/624654289 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fc6440aac80 con 0x7fc63c0039a0 2023-10-19T12:45:52.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.160+0000 7fc651d9b640 1 --2- 172.21.15.57:0/624654289 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fc63406ae40 0x7fc63406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:52.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.160+0000 7fc651d9b640 1 --2- 172.21.15.57:0/624654289 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fc63406ae40 0x7fc63406d300 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fc654158130 tx=0x7fc648009660 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:52.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.164+0000 7fc642ffd640 1 -- 172.21.15.57:0/624654289 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc6440795f0 con 0x7fc63c0039a0 2023-10-19T12:45:52.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.361+0000 7fc65359e640 1 -- 172.21.15.57:0/624654289 --> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fc65404da70 con 0x7fc63406ae40 2023-10-19T12:45:52.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.363+0000 7fc642ffd640 1 -- 172.21.15.57:0/624654289 <== mgr.4100 v2:172.21.15.179:6832/2609646285 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fc65404da70 con 0x7fc63406ae40 2023-10-19T12:45:52.366 INFO:teuthology.orchestra.run.smithi057.stdout:[ 2023-10-19T12:45:52.366 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T12:45:52.367 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "cephfs" 2023-10-19T12:45:52.367 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T12:45:52.367 INFO:teuthology.orchestra.run.smithi057.stdout:] 2023-10-19T12:45:52.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.365+0000 7fc65359e640 1 -- 172.21.15.57:0/624654289 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fc63406ae40 msgr2=0x7fc63406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:52.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.365+0000 7fc65359e640 1 --2- 172.21.15.57:0/624654289 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fc63406ae40 0x7fc63406d300 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fc654158130 tx=0x7fc648009660 comp rx=0 tx=0).stop 2023-10-19T12:45:52.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.365+0000 7fc65359e640 1 -- 172.21.15.57:0/624654289 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc63c0039a0 msgr2=0x7fc654152370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:52.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.365+0000 7fc65359e640 1 --2- 172.21.15.57:0/624654289 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc63c0039a0 0x7fc654152370 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fc64400c380 tx=0x7fc6440015f0 comp rx=0 tx=0).stop 2023-10-19T12:45:52.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.365+0000 7fc65359e640 1 -- 172.21.15.57:0/624654289 shutdown_connections 2023-10-19T12:45:52.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.365+0000 7fc65359e640 1 --2- 172.21.15.57:0/624654289 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fc63406ae40 0x7fc63406d300 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:52.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.365+0000 7fc65359e640 1 --2- 172.21.15.57:0/624654289 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc65415cf40 0x7fc6541f4960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:52.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.365+0000 7fc65359e640 1 --2- 172.21.15.57:0/624654289 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc6541f5570 0x7fc65415b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:52.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.365+0000 7fc65359e640 1 --2- 172.21.15.57:0/624654289 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc63c0039a0 0x7fc654152370 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:52.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.365+0000 7fc65359e640 1 -- 172.21.15.57:0/624654289 >> 172.21.15.57:0/624654289 conn(0x7fc654099eb0 msgr2=0x7fc65406b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:52.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.365+0000 7fc65359e640 1 -- 172.21.15.57:0/624654289 shutdown_connections 2023-10-19T12:45:52.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.365+0000 7fc65359e640 1 -- 172.21.15.57:0/624654289 wait complete. 2023-10-19T12:45:52.378 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-19T12:45:52.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.486+0000 7f106d2a2640 1 -- 172.21.15.57:0/4014470216 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1068159bd0 msgr2=0x7f106815c770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:52.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.486+0000 7f106d2a2640 1 --2- 172.21.15.57:0/4014470216 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1068159bd0 0x7f106815c770 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f10680cc8a0 tx=0x7f105002bd10 comp rx=0 tx=0).stop 2023-10-19T12:45:52.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.486+0000 7f106d2a2640 1 -- 172.21.15.57:0/4014470216 shutdown_connections 2023-10-19T12:45:52.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.486+0000 7f106d2a2640 1 --2- 172.21.15.57:0/4014470216 >> v2:172.21.15.179:3300/0 conn(0x7f106815ccb0 0x7f10680c9750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:52.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.486+0000 7f106d2a2640 1 --2- 172.21.15.57:0/4014470216 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1068159bd0 0x7f106815c770 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:52.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.486+0000 7f106d2a2640 1 --2- 172.21.15.57:0/4014470216 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f106806b030 0x7f106806b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:52.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.486+0000 7f106d2a2640 1 -- 172.21.15.57:0/4014470216 >> 172.21.15.57:0/4014470216 conn(0x7f1068069930 msgr2=0x7f1068069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:52.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.486+0000 7f106d2a2640 1 -- 172.21.15.57:0/4014470216 shutdown_connections 2023-10-19T12:45:52.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.486+0000 7f106d2a2640 1 -- 172.21.15.57:0/4014470216 wait complete. 2023-10-19T12:45:52.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.487+0000 7f106d2a2640 1 Processor -- start 2023-10-19T12:45:52.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.487+0000 7f106d2a2640 1 -- start start 2023-10-19T12:45:52.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.488+0000 7f106d2a2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f106806b030 0x7f106814fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:52.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.488+0000 7f106d2a2640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1068159bd0 0x7f10681f2640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:52.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.488+0000 7f106d2a2640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f106815ccb0 0x7f10680ab620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:52.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.488+0000 7f106d2a2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10681f3570 con 0x7f106806b030 2023-10-19T12:45:52.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.488+0000 7f106d2a2640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1068137180 con 0x7f1068159bd0 2023-10-19T12:45:52.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.488+0000 7f106d2a2640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10681f2fb0 con 0x7f106815ccb0 2023-10-19T12:45:52.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.488+0000 7f1066ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f106806b030 0x7f106814fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:52.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.488+0000 7f1066ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f106806b030 0x7f106814fa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37618/0 (socket says 172.21.15.57:37618) 2023-10-19T12:45:52.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.488+0000 7f1066ffd640 1 -- 172.21.15.57:0/3680373333 learned_addr learned my addr 172.21.15.57:0/3680373333 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:52.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.488+0000 7f10677fe640 1 --2- 172.21.15.57:0/3680373333 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f106815ccb0 0x7f10680ab620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:52.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.488+0000 7f10667fc640 1 --2- 172.21.15.57:0/3680373333 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1068159bd0 0x7f10681f2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:52.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.488+0000 7f1066ffd640 1 -- 172.21.15.57:0/3680373333 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f106815ccb0 msgr2=0x7f10680ab620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:52.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.488+0000 7f1066ffd640 1 --2- 172.21.15.57:0/3680373333 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f106815ccb0 0x7f10680ab620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:52.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.488+0000 7f1066ffd640 1 -- 172.21.15.57:0/3680373333 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1068159bd0 msgr2=0x7f10681f2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:52.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.488+0000 7f1066ffd640 1 --2- 172.21.15.57:0/3680373333 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1068159bd0 0x7f10681f2640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:52.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.488+0000 7f1066ffd640 1 -- 172.21.15.57:0/3680373333 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1050009560 con 0x7f106806b030 2023-10-19T12:45:52.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.488+0000 7f10667fc640 1 --2- 172.21.15.57:0/3680373333 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1068159bd0 0x7f10681f2640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:45:52.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.489+0000 7f1066ffd640 1 --2- 172.21.15.57:0/3680373333 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f106806b030 0x7f106814fa80 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f106000d950 tx=0x7f106000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:52.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.489+0000 7f105f7fe640 1 -- 172.21.15.57:0/3680373333 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1060009c30 con 0x7f106806b030 2023-10-19T12:45:52.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.489+0000 7f105f7fe640 1 -- 172.21.15.57:0/3680373333 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f1060009e80 con 0x7f106806b030 2023-10-19T12:45:52.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.489+0000 7f106d2a2640 1 -- 172.21.15.57:0/3680373333 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10681f3290 con 0x7f106806b030 2023-10-19T12:45:52.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.489+0000 7f105f7fe640 1 -- 172.21.15.57:0/3680373333 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10600106c0 con 0x7f106806b030 2023-10-19T12:45:52.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.489+0000 7f106d2a2640 1 -- 172.21.15.57:0/3680373333 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10680a73e0 con 0x7f106806b030 2023-10-19T12:45:52.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.490+0000 7f106d2a2640 1 -- 172.21.15.57:0/3680373333 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f106806c290 con 0x7f106806b030 2023-10-19T12:45:52.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.490+0000 7f105f7fe640 1 -- 172.21.15.57:0/3680373333 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f1060010820 con 0x7f106806b030 2023-10-19T12:45:52.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.491+0000 7f105f7fe640 1 --2- 172.21.15.57:0/3680373333 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f104806ae40 0x7f104806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:52.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.491+0000 7f105f7fe640 1 -- 172.21.15.57:0/3680373333 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f1060089bc0 con 0x7f106806b030 2023-10-19T12:45:52.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.491+0000 7f10667fc640 1 --2- 172.21.15.57:0/3680373333 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f104806ae40 0x7f104806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:52.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.491+0000 7f10667fc640 1 --2- 172.21.15.57:0/3680373333 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f104806ae40 0x7f104806d300 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f1050035000 tx=0x7f1050004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:52.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.495+0000 7f105f7fe640 1 -- 172.21.15.57:0/3680373333 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f1060054bc0 con 0x7f106806b030 2023-10-19T12:45:52.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.720+0000 7f106d2a2640 1 -- 172.21.15.57:0/3680373333 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f10680690e0 con 0x7f106806b030 2023-10-19T12:45:52.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.974+0000 7f105f7fe640 1 -- 172.21.15.57:0/3680373333 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v6) v1 ==== 92+0+0 (secure 0 0 0) 0x7f10600170a0 con 0x7f106806b030 2023-10-19T12:45:52.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.976+0000 7f106d2a2640 1 -- 172.21.15.57:0/3680373333 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f104806ae40 msgr2=0x7f104806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:52.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.976+0000 7f106d2a2640 1 --2- 172.21.15.57:0/3680373333 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f104806ae40 0x7f104806d300 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f1050035000 tx=0x7f1050004730 comp rx=0 tx=0).stop 2023-10-19T12:45:52.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.976+0000 7f106d2a2640 1 -- 172.21.15.57:0/3680373333 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f106806b030 msgr2=0x7f106814fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:52.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.976+0000 7f106d2a2640 1 --2- 172.21.15.57:0/3680373333 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f106806b030 0x7f106814fa80 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f106000d950 tx=0x7f106000de10 comp rx=0 tx=0).stop 2023-10-19T12:45:52.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.977+0000 7f106d2a2640 1 -- 172.21.15.57:0/3680373333 shutdown_connections 2023-10-19T12:45:52.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.977+0000 7f106d2a2640 1 --2- 172.21.15.57:0/3680373333 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f104806ae40 0x7f104806d300 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:52.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.977+0000 7f106d2a2640 1 --2- 172.21.15.57:0/3680373333 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f106815ccb0 0x7f10680ab620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:52.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.977+0000 7f106d2a2640 1 --2- 172.21.15.57:0/3680373333 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1068159bd0 0x7f10681f2640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:52.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.977+0000 7f106d2a2640 1 --2- 172.21.15.57:0/3680373333 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f106806b030 0x7f106814fa80 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:52.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.977+0000 7f106d2a2640 1 -- 172.21.15.57:0/3680373333 >> 172.21.15.57:0/3680373333 conn(0x7f1068069930 msgr2=0x7f106813b0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:52.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.977+0000 7f106d2a2640 1 -- 172.21.15.57:0/3680373333 shutdown_connections 2023-10-19T12:45:52.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:52.977+0000 7f106d2a2640 1 -- 172.21.15.57:0/3680373333 wait complete. 2023-10-19T12:45:52.996 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-19T12:45:53.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.109+0000 7fe325e39640 1 -- 172.21.15.57:0/671070238 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3200cf340 msgr2=0x7fe3200d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:53.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.109+0000 7fe325e39640 1 --2- 172.21.15.57:0/671070238 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3200cf340 0x7fe3200d1910 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fe32015b680 tx=0x7fe31002bdb0 comp rx=0 tx=0).stop 2023-10-19T12:45:53.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.109+0000 7fe325e39640 1 -- 172.21.15.57:0/671070238 shutdown_connections 2023-10-19T12:45:53.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.109+0000 7fe325e39640 1 --2- 172.21.15.57:0/671070238 >> v2:172.21.15.179:3300/0 conn(0x7fe3200ceb40 0x7fe3200b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:53.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.109+0000 7fe325e39640 1 --2- 172.21.15.57:0/671070238 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3200cf340 0x7fe3200d1910 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:53.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.109+0000 7fe325e39640 1 -- 172.21.15.57:0/671070238 >> 172.21.15.57:0/671070238 conn(0x7fe3200d0d60 msgr2=0x7fe3200d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:53.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.109+0000 7fe325e39640 1 -- 172.21.15.57:0/671070238 shutdown_connections 2023-10-19T12:45:53.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.109+0000 7fe325e39640 1 -- 172.21.15.57:0/671070238 wait complete. 2023-10-19T12:45:53.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.110+0000 7fe325e39640 1 Processor -- start 2023-10-19T12:45:53.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.110+0000 7fe325e39640 1 -- start start 2023-10-19T12:45:53.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.111+0000 7fe325e39640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe3200ceb40 0x7fe3200cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:53.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.111+0000 7fe325e39640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3200cf340 0x7fe320152f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:53.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.111+0000 7fe325e39640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe32015b2a0 0x7fe320153480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:53.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.111+0000 7fe325e39640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3200b58b0 con 0x7fe3200cf340 2023-10-19T12:45:53.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.111+0000 7fe325e39640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3200b4f10 con 0x7fe32015b2a0 2023-10-19T12:45:53.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.111+0000 7fe325e39640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3200b51f0 con 0x7fe3200ceb40 2023-10-19T12:45:53.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.111+0000 7fe31effd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3200cf340 0x7fe320152f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:53.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.111+0000 7fe31effd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3200cf340 0x7fe320152f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37658/0 (socket says 172.21.15.57:37658) 2023-10-19T12:45:53.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.111+0000 7fe31effd640 1 -- 172.21.15.57:0/1314654024 learned_addr learned my addr 172.21.15.57:0/1314654024 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:53.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.111+0000 7fe31ffff640 1 --2- 172.21.15.57:0/1314654024 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe32015b2a0 0x7fe320153480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:53.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.111+0000 7fe31f7fe640 1 --2- 172.21.15.57:0/1314654024 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe3200ceb40 0x7fe3200cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:53.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.112+0000 7fe31effd640 1 -- 172.21.15.57:0/1314654024 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe3200ceb40 msgr2=0x7fe3200cd1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:53.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.112+0000 7fe31effd640 1 --2- 172.21.15.57:0/1314654024 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe3200ceb40 0x7fe3200cd1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:53.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.112+0000 7fe31effd640 1 -- 172.21.15.57:0/1314654024 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe32015b2a0 msgr2=0x7fe320153480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:53.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.112+0000 7fe31effd640 1 --2- 172.21.15.57:0/1314654024 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe32015b2a0 0x7fe320153480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:53.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.112+0000 7fe31effd640 1 -- 172.21.15.57:0/1314654024 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe310009560 con 0x7fe3200cf340 2023-10-19T12:45:53.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.112+0000 7fe31effd640 1 --2- 172.21.15.57:0/1314654024 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3200cf340 0x7fe320152f40 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fe31400c3c0 tx=0x7fe31400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:53.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.112+0000 7fe307fff640 1 -- 172.21.15.57:0/1314654024 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3140052d0 con 0x7fe3200cf340 2023-10-19T12:45:53.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.112+0000 7fe307fff640 1 -- 172.21.15.57:0/1314654024 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fe314007020 con 0x7fe3200cf340 2023-10-19T12:45:53.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.112+0000 7fe325e39640 1 -- 172.21.15.57:0/1314654024 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3200b54d0 con 0x7fe3200cf340 2023-10-19T12:45:53.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.112+0000 7fe307fff640 1 -- 172.21.15.57:0/1314654024 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe314007470 con 0x7fe3200cf340 2023-10-19T12:45:53.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.112+0000 7fe325e39640 1 -- 172.21.15.57:0/1314654024 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3200b5eb0 con 0x7fe3200cf340 2023-10-19T12:45:53.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.113+0000 7fe325e39640 1 -- 172.21.15.57:0/1314654024 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3200b59f0 con 0x7fe3200cf340 2023-10-19T12:45:53.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.114+0000 7fe307fff640 1 -- 172.21.15.57:0/1314654024 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 6) v1 ==== 89246+0+0 (secure 0 0 0) 0x7fe3140067d0 con 0x7fe3200cf340 2023-10-19T12:45:53.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.114+0000 7fe307fff640 1 --2- 172.21.15.57:0/1314654024 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fe30c06ae90 0x7fe30c06d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:53.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.114+0000 7fe31f7fe640 1 -- 172.21.15.57:0/1314654024 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fe30c06ae90 msgr2=0x7fe30c06d350 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/2609646285 2023-10-19T12:45:53.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.114+0000 7fe307fff640 1 -- 172.21.15.57:0/1314654024 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fe314088b00 con 0x7fe3200cf340 2023-10-19T12:45:53.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.114+0000 7fe31f7fe640 1 --2- 172.21.15.57:0/1314654024 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fe30c06ae90 0x7fe30c06d350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T12:45:53.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.118+0000 7fe307fff640 1 -- 172.21.15.57:0/1314654024 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe3200b59f0 con 0x7fe3200cf340 2023-10-19T12:45:53.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.296+0000 7fe325e39640 1 -- 172.21.15.57:0/1314654024 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7fe3200b5eb0 con 0x7fe3200cf340 2023-10-19T12:45:53.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.304+0000 7fe307fff640 1 -- 172.21.15.57:0/1314654024 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v32)=0 v32) v1 ==== 138+0+0 (secure 0 0 0) 0x7fe3200b5eb0 con 0x7fe3200cf340 2023-10-19T12:45:53.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.306+0000 7fe325e39640 1 -- 172.21.15.57:0/1314654024 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fe30c06ae90 msgr2=0x7fe30c06d350 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T12:45:53.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.306+0000 7fe325e39640 1 --2- 172.21.15.57:0/1314654024 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fe30c06ae90 0x7fe30c06d350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:53.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.306+0000 7fe325e39640 1 -- 172.21.15.57:0/1314654024 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3200cf340 msgr2=0x7fe320152f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:53.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.306+0000 7fe325e39640 1 --2- 172.21.15.57:0/1314654024 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3200cf340 0x7fe320152f40 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fe31400c3c0 tx=0x7fe31400c880 comp rx=0 tx=0).stop 2023-10-19T12:45:53.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.306+0000 7fe325e39640 1 -- 172.21.15.57:0/1314654024 shutdown_connections 2023-10-19T12:45:53.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.306+0000 7fe325e39640 1 --2- 172.21.15.57:0/1314654024 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fe30c06ae90 0x7fe30c06d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:53.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.306+0000 7fe325e39640 1 --2- 172.21.15.57:0/1314654024 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe32015b2a0 0x7fe320153480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:53.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.306+0000 7fe325e39640 1 --2- 172.21.15.57:0/1314654024 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3200cf340 0x7fe320152f40 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:53.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.306+0000 7fe325e39640 1 --2- 172.21.15.57:0/1314654024 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe3200ceb40 0x7fe3200cd1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:53.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.306+0000 7fe325e39640 1 -- 172.21.15.57:0/1314654024 >> 172.21.15.57:0/1314654024 conn(0x7fe3200d0d60 msgr2=0x7fe320158cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:53.312 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.306+0000 7fe325e39640 1 -- 172.21.15.57:0/1314654024 shutdown_connections 2023-10-19T12:45:53.312 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.306+0000 7fe325e39640 1 -- 172.21.15.57:0/1314654024 wait complete. 2023-10-19T12:45:53.320 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-19T12:45:53.354 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:45:53.351+0000 7fe521a24040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-19T12:45:53.383 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:45:53.380+0000 7f338a62c040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-19T12:45:53.399 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:45:53.396+0000 7fe521a24040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-19T12:45:53.431 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:45:53.429+0000 7f338a62c040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-19T12:45:53.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.431+0000 7fe30831d640 1 -- 172.21.15.57:0/768098389 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe2ec003970 msgr2=0x7fe2ec003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:53.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.431+0000 7fe30831d640 1 --2- 172.21.15.57:0/768098389 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe2ec003970 0x7fe2ec003d90 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fe2f0009d00 tx=0x7fe2ec0057f0 comp rx=0 tx=0).stop 2023-10-19T12:45:53.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.431+0000 7fe3068ba640 1 -- 172.21.15.57:0/768098389 reap_dead start 2023-10-19T12:45:53.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.431+0000 7fe30831d640 1 -- 172.21.15.57:0/768098389 shutdown_connections 2023-10-19T12:45:53.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.431+0000 7fe30831d640 1 -- 172.21.15.57:0/768098389 >> 172.21.15.57:0/768098389 conn(0x7fe3000d0d60 msgr2=0x7fe3000d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:53.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.431+0000 7fe30831d640 1 -- 172.21.15.57:0/768098389 shutdown_connections 2023-10-19T12:45:53.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.431+0000 7fe30831d640 1 -- 172.21.15.57:0/768098389 wait complete. 2023-10-19T12:45:53.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.432+0000 7fe30831d640 1 Processor -- start 2023-10-19T12:45:53.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.432+0000 7fe30831d640 1 -- start start 2023-10-19T12:45:53.439 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.433+0000 7fe30831d640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe2ec003970 0x7fe3000c8800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:53.439 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.433+0000 7fe30831d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3000cd1a0 0x7fe3001f2690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:53.439 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.433+0000 7fe30831d640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe30015b6d0 0x7fe30015bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:53.439 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.433+0000 7fe30831d640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3001f3010 con 0x7fe3000cd1a0 2023-10-19T12:45:53.439 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.433+0000 7fe30831d640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3001f32f0 con 0x7fe30015b6d0 2023-10-19T12:45:53.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.433+0000 7fe30831d640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3001f35d0 con 0x7fe2ec003970 2023-10-19T12:45:53.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.433+0000 7fe3058b8640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3000cd1a0 0x7fe3001f2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:53.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.433+0000 7fe3058b8640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3000cd1a0 0x7fe3001f2690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37682/0 (socket says 172.21.15.57:37682) 2023-10-19T12:45:53.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.433+0000 7fe3058b8640 1 -- 172.21.15.57:0/1886404303 learned_addr learned my addr 172.21.15.57:0/1886404303 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:53.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.433+0000 7fe3060b9640 1 --2- 172.21.15.57:0/1886404303 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe2ec003970 0x7fe3000c8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:53.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.433+0000 7fe3068ba640 1 --2- 172.21.15.57:0/1886404303 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe30015b6d0 0x7fe30015bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:53.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.434+0000 7fe3058b8640 1 -- 172.21.15.57:0/1886404303 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe2ec003970 msgr2=0x7fe3000c8800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:53.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.434+0000 7fe3058b8640 1 --2- 172.21.15.57:0/1886404303 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe2ec003970 0x7fe3000c8800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:53.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.434+0000 7fe3058b8640 1 -- 172.21.15.57:0/1886404303 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe30015b6d0 msgr2=0x7fe30015bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:53.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.434+0000 7fe3058b8640 1 --2- 172.21.15.57:0/1886404303 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe30015b6d0 0x7fe30015bb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:53.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.434+0000 7fe3058b8640 1 -- 172.21.15.57:0/1886404303 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3001f38b0 con 0x7fe3000cd1a0 2023-10-19T12:45:53.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.434+0000 7fe3058b8640 1 --2- 172.21.15.57:0/1886404303 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3000cd1a0 0x7fe3001f2690 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fe2fc001050 tx=0x7fe2fc0039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:53.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.434+0000 7fe2f6ffd640 1 -- 172.21.15.57:0/1886404303 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2fc011900 con 0x7fe3000cd1a0 2023-10-19T12:45:53.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.434+0000 7fe2f6ffd640 1 -- 172.21.15.57:0/1886404303 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fe2fc011a60 con 0x7fe3000cd1a0 2023-10-19T12:45:53.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.434+0000 7fe30831d640 1 -- 172.21.15.57:0/1886404303 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3001f3b90 con 0x7fe3000cd1a0 2023-10-19T12:45:53.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.434+0000 7fe2f6ffd640 1 -- 172.21.15.57:0/1886404303 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2fc011e70 con 0x7fe3000cd1a0 2023-10-19T12:45:53.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.434+0000 7fe30831d640 1 -- 172.21.15.57:0/1886404303 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3000c8f20 con 0x7fe3000cd1a0 2023-10-19T12:45:53.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.435+0000 7fe30831d640 1 -- 172.21.15.57:0/1886404303 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3001f3dd0 con 0x7fe3000cd1a0 2023-10-19T12:45:53.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.435+0000 7fe2f6ffd640 1 -- 172.21.15.57:0/1886404303 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 6) v1 ==== 89246+0+0 (secure 0 0 0) 0x7fe2fc010420 con 0x7fe3000cd1a0 2023-10-19T12:45:53.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.436+0000 7fe2f6ffd640 1 --2- 172.21.15.57:0/1886404303 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fe2e406aec0 0x7fe2e406d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:53.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.436+0000 7fe2f6ffd640 1 -- 172.21.15.57:0/1886404303 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fe2fc08a6d0 con 0x7fe3000cd1a0 2023-10-19T12:45:53.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.436+0000 7fe3060b9640 1 -- 172.21.15.57:0/1886404303 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fe2e406aec0 msgr2=0x7fe2e406d380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/2609646285 2023-10-19T12:45:53.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.436+0000 7fe3060b9640 1 --2- 172.21.15.57:0/1886404303 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fe2e406aec0 0x7fe2e406d380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T12:45:53.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.440+0000 7fe2f6ffd640 1 -- 172.21.15.57:0/1886404303 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe3001f3dd0 con 0x7fe3000cd1a0 2023-10-19T12:45:53.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.617+0000 7fe30831d640 1 -- 172.21.15.57:0/1886404303 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7fe3000c8f20 con 0x7fe3000cd1a0 2023-10-19T12:45:53.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.620+0000 7fe2f6ffd640 1 -- 172.21.15.57:0/1886404303 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v33)=0 v33) v1 ==== 133+0+0 (secure 0 0 0) 0x7fe3000c8f20 con 0x7fe3000cd1a0 2023-10-19T12:45:53.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.622+0000 7fe30831d640 1 -- 172.21.15.57:0/1886404303 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fe2e406aec0 msgr2=0x7fe2e406d380 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T12:45:53.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.622+0000 7fe30831d640 1 --2- 172.21.15.57:0/1886404303 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fe2e406aec0 0x7fe2e406d380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:53.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.622+0000 7fe30831d640 1 -- 172.21.15.57:0/1886404303 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3000cd1a0 msgr2=0x7fe3001f2690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:53.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.622+0000 7fe30831d640 1 --2- 172.21.15.57:0/1886404303 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3000cd1a0 0x7fe3001f2690 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fe2fc001050 tx=0x7fe2fc0039c0 comp rx=0 tx=0).stop 2023-10-19T12:45:53.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.622+0000 7fe30831d640 1 -- 172.21.15.57:0/1886404303 shutdown_connections 2023-10-19T12:45:53.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.622+0000 7fe30831d640 1 --2- 172.21.15.57:0/1886404303 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7fe2e406aec0 0x7fe2e406d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:53.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.622+0000 7fe30831d640 1 --2- 172.21.15.57:0/1886404303 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe30015b6d0 0x7fe30015bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:53.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.622+0000 7fe30831d640 1 --2- 172.21.15.57:0/1886404303 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3000cd1a0 0x7fe3001f2690 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:53.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.622+0000 7fe30831d640 1 --2- 172.21.15.57:0/1886404303 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe2ec003970 0x7fe3000c8800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:53.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.622+0000 7fe30831d640 1 -- 172.21.15.57:0/1886404303 >> 172.21.15.57:0/1886404303 conn(0x7fe3000d0d60 msgr2=0x7fe3000b0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:53.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.622+0000 7fe30831d640 1 -- 172.21.15.57:0/1886404303 shutdown_connections 2023-10-19T12:45:53.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.622+0000 7fe30831d640 1 -- 172.21.15.57:0/1886404303 wait complete. 2023-10-19T12:45:53.636 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:45:53.637 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-10-19T12:45:53.657 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:45:53.657 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir0) 2023-10-19T12:45:53.715 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir0 1m 2023-10-19T12:45:53.792 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:45:53.789+0000 7fe521a24040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-19T12:45:53.836 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:45:53.833+0000 7fe521a24040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-19T12:45:53.850 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:45:53.847+0000 7f338a62c040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-19T12:45:53.868 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.864+0000 7f9d80de2640 1 -- 172.21.15.57:0/995582650 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9d7c15b2a0 msgr2=0x7f9d7c0cf720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:53.868 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.864+0000 7f9d80de2640 1 --2- 172.21.15.57:0/995582650 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9d7c15b2a0 0x7f9d7c0cf720 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f9d7c08a810 tx=0x7f9d7002bdb0 comp rx=0 tx=0).stop 2023-10-19T12:45:53.868 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.865+0000 7f9d80de2640 1 -- 172.21.15.57:0/995582650 shutdown_connections 2023-10-19T12:45:53.869 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.865+0000 7f9d80de2640 1 --2- 172.21.15.57:0/995582650 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9d7c15b2a0 0x7f9d7c0cf720 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:53.869 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.865+0000 7f9d80de2640 1 --2- 172.21.15.57:0/995582650 >> v2:172.21.15.179:3300/0 conn(0x7f9d7c0cf340 0x7f9d7c0d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:53.869 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.865+0000 7f9d80de2640 1 -- 172.21.15.57:0/995582650 >> 172.21.15.57:0/995582650 conn(0x7f9d7c0d0d60 msgr2=0x7f9d7c0d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:53.869 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.865+0000 7f9d80de2640 1 -- 172.21.15.57:0/995582650 shutdown_connections 2023-10-19T12:45:53.869 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.865+0000 7f9d80de2640 1 -- 172.21.15.57:0/995582650 wait complete. 2023-10-19T12:45:53.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.866+0000 7f9d80de2640 1 Processor -- start 2023-10-19T12:45:53.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.866+0000 7f9d80de2640 1 -- start start 2023-10-19T12:45:53.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.867+0000 7f9d80de2640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9d7c0ceb40 0x7f9d7c14d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:53.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.867+0000 7f9d80de2640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9d7c0cf340 0x7f9d7c0ca000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:53.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.867+0000 7f9d80de2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9d7c15b2a0 0x7f9d7c0ca540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:53.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.867+0000 7f9d80de2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d7c14d940 con 0x7f9d7c15b2a0 2023-10-19T12:45:53.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.867+0000 7f9d80de2640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d7c09e990 con 0x7f9d7c0ceb40 2023-10-19T12:45:53.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.867+0000 7f9d80de2640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d7c09d8f0 con 0x7f9d7c0cf340 2023-10-19T12:45:53.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.867+0000 7f9d7ad9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9d7c15b2a0 0x7f9d7c0ca540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:53.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.867+0000 7f9d7ad9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9d7c15b2a0 0x7f9d7c0ca540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37700/0 (socket says 172.21.15.57:37700) 2023-10-19T12:45:53.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.867+0000 7f9d7ad9d640 1 -- 172.21.15.57:0/958981348 learned_addr learned my addr 172.21.15.57:0/958981348 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:53.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.867+0000 7f9d7a59c640 1 --2- 172.21.15.57:0/958981348 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9d7c0ceb40 0x7f9d7c14d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:53.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.867+0000 7f9d79d9b640 1 --2- 172.21.15.57:0/958981348 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9d7c0cf340 0x7f9d7c0ca000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:53.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.867+0000 7f9d7ad9d640 1 -- 172.21.15.57:0/958981348 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9d7c0cf340 msgr2=0x7f9d7c0ca000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:53.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.867+0000 7f9d7ad9d640 1 --2- 172.21.15.57:0/958981348 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9d7c0cf340 0x7f9d7c0ca000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:53.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.867+0000 7f9d7ad9d640 1 -- 172.21.15.57:0/958981348 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9d7c0ceb40 msgr2=0x7f9d7c14d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:53.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.867+0000 7f9d7ad9d640 1 --2- 172.21.15.57:0/958981348 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9d7c0ceb40 0x7f9d7c14d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:53.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.867+0000 7f9d7ad9d640 1 -- 172.21.15.57:0/958981348 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d70009560 con 0x7f9d7c15b2a0 2023-10-19T12:45:53.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.867+0000 7f9d7a59c640 1 --2- 172.21.15.57:0/958981348 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9d7c0ceb40 0x7f9d7c14d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T12:45:53.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.868+0000 7f9d7ad9d640 1 --2- 172.21.15.57:0/958981348 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9d7c15b2a0 0x7f9d7c0ca540 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f9d7400c730 tx=0x7f9d7400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:53.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.868+0000 7f9d6affd640 1 -- 172.21.15.57:0/958981348 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d74003b90 con 0x7f9d7c15b2a0 2023-10-19T12:45:53.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.868+0000 7f9d6affd640 1 -- 172.21.15.57:0/958981348 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f9d74003cf0 con 0x7f9d7c15b2a0 2023-10-19T12:45:53.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.868+0000 7f9d6affd640 1 -- 172.21.15.57:0/958981348 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d7400aab0 con 0x7f9d7c15b2a0 2023-10-19T12:45:53.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.868+0000 7f9d80de2640 1 -- 172.21.15.57:0/958981348 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d7c09dbd0 con 0x7f9d7c15b2a0 2023-10-19T12:45:53.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.868+0000 7f9d80de2640 1 -- 172.21.15.57:0/958981348 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d7c09e190 con 0x7f9d7c15b2a0 2023-10-19T12:45:53.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.869+0000 7f9d80de2640 1 -- 172.21.15.57:0/958981348 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d7c0690e0 con 0x7f9d7c15b2a0 2023-10-19T12:45:53.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.870+0000 7f9d6affd640 1 -- 172.21.15.57:0/958981348 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 6) v1 ==== 89246+0+0 (secure 0 0 0) 0x7f9d7400ac10 con 0x7f9d7c15b2a0 2023-10-19T12:45:53.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.870+0000 7f9d6affd640 1 --2- 172.21.15.57:0/958981348 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f9d5c06ae90 0x7f9d5c06d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:53.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.870+0000 7f9d7a59c640 1 -- 172.21.15.57:0/958981348 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f9d5c06ae90 msgr2=0x7f9d5c06d350 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/2609646285 2023-10-19T12:45:53.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.870+0000 7f9d6affd640 1 -- 172.21.15.57:0/958981348 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f9d74089690 con 0x7f9d7c15b2a0 2023-10-19T12:45:53.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.870+0000 7f9d7a59c640 1 --2- 172.21.15.57:0/958981348 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f9d5c06ae90 0x7f9d5c06d350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T12:45:53.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:53.874+0000 7f9d6affd640 1 -- 172.21.15.57:0/958981348 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9d74054610 con 0x7f9d7c15b2a0 2023-10-19T12:45:53.879 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:45:53.876+0000 7fe521a24040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-19T12:45:53.899 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:45:53.896+0000 7f338a62c040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-19T12:45:53.931 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:45:53.927+0000 7fe521a24040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-19T12:45:53.944 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:45:53.941+0000 7f338a62c040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-19T12:45:53.971 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:45:53.968+0000 7fe521a24040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-19T12:45:53.999 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:45:53.996+0000 7f338a62c040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-19T12:45:54.043 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:45:54.040+0000 7f338a62c040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-19T12:45:54.069 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:45:54.066+0000 7fe521a24040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-19T12:45:54.074 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:54.071+0000 7f9d7a59c640 1 -- 172.21.15.57:0/958981348 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f9d5c06ae90 msgr2=0x7f9d5c06d350 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/2609646285 2023-10-19T12:45:54.074 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:54.071+0000 7f9d7a59c640 1 --2- 172.21.15.57:0/958981348 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f9d5c06ae90 0x7f9d5c06d350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-19T12:45:54.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:54.072+0000 7f9d80de2640 1 -- 172.21.15.57:0/958981348 --> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir0", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f9d7c0cb9d0 con 0x7f9d5c06ae90 2023-10-19T12:45:54.122 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:45:54.119+0000 7fe521a24040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-19T12:45:54.147 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:45:54.144+0000 7f338a62c040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-19T12:45:54.163 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:45:54.160+0000 7fe521a24040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-19T12:45:54.203 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:45:54.200+0000 7f338a62c040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-19T12:45:54.212 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:45:54.209+0000 7fe521a24040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-19T12:45:54.247 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:45:54.244+0000 7f338a62c040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-19T12:45:54.253 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:45:54.250+0000 7fe521a24040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-19T12:45:54.297 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:45:54.294+0000 7f338a62c040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-19T12:45:54.342 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:45:54.339+0000 7f338a62c040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-19T12:45:54.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:54.472+0000 7f9d7a59c640 1 -- 172.21.15.57:0/958981348 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f9d5c06ae90 msgr2=0x7f9d5c06d350 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/2609646285 2023-10-19T12:45:54.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:54.472+0000 7f9d7a59c640 1 --2- 172.21.15.57:0/958981348 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f9d5c06ae90 0x7f9d5c06d350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-19T12:45:54.570 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:45:54.567+0000 7fe521a24040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-19T12:45:54.675 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:45:54.672+0000 7f338a62c040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-19T12:45:54.700 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:45:54.697+0000 7fe521a24040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-19T12:45:54.790 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:45:54.787+0000 7fe521a24040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-19T12:45:54.814 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:45:54.811+0000 7f338a62c040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-19T12:45:54.832 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:45:54.829+0000 7fe521a24040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-19T12:45:54.910 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:45:54.908+0000 7f338a62c040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-19T12:45:54.926 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:45:54.923+0000 7fe521a24040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-19T12:45:54.955 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:45:54.952+0000 7f338a62c040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-19T12:45:55.016 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:45:55.013+0000 7fe521a24040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-19T12:45:55.054 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:45:55.052+0000 7f338a62c040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-19T12:45:55.150 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:45:55.147+0000 7f338a62c040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-19T12:45:55.152 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:45:55.149+0000 7fe521a24040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-19T12:45:55.194 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:45:55.191+0000 7fe521a24040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-19T12:45:55.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:55.273+0000 7f9d7a59c640 1 -- 172.21.15.57:0/958981348 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f9d5c06ae90 msgr2=0x7f9d5c06d350 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/2609646285 2023-10-19T12:45:55.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:55.273+0000 7f9d7a59c640 1 --2- 172.21.15.57:0/958981348 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f9d5c06ae90 0x7f9d5c06d350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-19T12:45:55.294 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:45:55.291+0000 7f338a62c040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-19T12:45:55.299 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:45:55.296+0000 7fe521a24040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-19T12:45:55.338 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:45:55.335+0000 7f338a62c040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-19T12:45:55.448 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:45:55.444+0000 7f338a62c040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-19T12:45:55.907 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:45:55.904+0000 7fe521a24040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-19T12:45:56.086 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:45:56.083+0000 7f338a62c040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-19T12:45:56.168 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:45:56.164+0000 7fe521a24040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-19T12:45:56.217 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:45:56.214+0000 7fe521a24040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-19T12:45:56.316 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:45:56.313+0000 7fe521a24040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-19T12:45:56.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:56.333+0000 7f9d6affd640 1 -- 172.21.15.57:0/958981348 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mgrmap(e 7) v1 ==== 88957+0+0 (secure 0 0 0) 0x7f9d740549e0 con 0x7f9d7c15b2a0 2023-10-19T12:45:56.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:56.333+0000 7f9d6affd640 1 -- 172.21.15.57:0/958981348 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f9d5c06ae90 msgr2=0x7f9d5c06d350 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T12:45:56.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:56.333+0000 7f9d6affd640 1 --2- 172.21.15.57:0/958981348 >> [v2:172.21.15.179:6832/2609646285,v1:172.21.15.179:6833/2609646285] conn(0x7f9d5c06ae90 0x7f9d5c06d350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:56.366 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:45:56.363+0000 7f338a62c040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-19T12:45:56.422 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:45:56.419+0000 7f338a62c040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-19T12:45:56.445 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:45:56.442+0000 7fe4a111d640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T12:45:56.445 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:45:56.442+0000 7fe4a111d640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T12:45:56.445 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:45:56.442+0000 7fe4a111d640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T12:45:56.446 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:45:56.442+0000 7fe4a111d640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T12:45:56.446 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:45:56.442+0000 7fe4a111d640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T12:45:56.446 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:45:56.442+0000 7fe49e0d7640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T12:45:56.446 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:45:56.442+0000 7fe49e0d7640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T12:45:56.447 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:45:56.442+0000 7fe49e0d7640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T12:45:56.447 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:45:56.442+0000 7fe49e0d7640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T12:45:56.447 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:45:56.442+0000 7fe49e0d7640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T12:45:56.528 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:45:56.525+0000 7f338a62c040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-19T12:45:57.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:57.337+0000 7f9d6affd640 1 -- 172.21.15.57:0/958981348 <== mon.0 v2:172.21.15.57:3300/0 8 ==== mgrmap(e 8) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f9d740577c0 con 0x7f9d7c15b2a0 2023-10-19T12:45:57.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:57.338+0000 7f9d6affd640 1 --2- 172.21.15.57:0/958981348 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f9d5c035a70 0x7f9d5c075260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:57.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:57.338+0000 7f9d6affd640 1 -- 172.21.15.57:0/958981348 --> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir0", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f9d7c0cb9d0 con 0x7f9d5c035a70 2023-10-19T12:45:57.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:57.339+0000 7f9d7a59c640 1 --2- 172.21.15.57:0/958981348 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f9d5c035a70 0x7f9d5c075260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:57.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:57.342+0000 7f9d7a59c640 1 --2- 172.21.15.57:0/958981348 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f9d5c035a70 0x7f9d5c075260 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f9d7c0b6de0 tx=0x7f9d6c00b390 comp rx=0 tx=0).ready entity=mgr.5428 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:57.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:57.510+0000 7f9d6affd640 1 -- 172.21.15.57:0/958981348 <== mgr.5428 v2:172.21.15.179:6832/922398470 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f9d7c0cb9d0 con 0x7f9d5c035a70 2023-10-19T12:45:57.514 INFO:teuthology.orchestra.run.smithi057.stdout:Schedule set for path /snap_test_dir1/dir0 2023-10-19T12:45:57.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:57.512+0000 7f9d80de2640 1 -- 172.21.15.57:0/958981348 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f9d5c035a70 msgr2=0x7f9d5c075260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:57.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:57.513+0000 7f9d80de2640 1 --2- 172.21.15.57:0/958981348 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f9d5c035a70 0x7f9d5c075260 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f9d7c0b6de0 tx=0x7f9d6c00b390 comp rx=0 tx=0).stop 2023-10-19T12:45:57.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:57.513+0000 7f9d80de2640 1 -- 172.21.15.57:0/958981348 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9d7c15b2a0 msgr2=0x7f9d7c0ca540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:57.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:57.513+0000 7f9d80de2640 1 --2- 172.21.15.57:0/958981348 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9d7c15b2a0 0x7f9d7c0ca540 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f9d7400c730 tx=0x7f9d7400cbf0 comp rx=0 tx=0).stop 2023-10-19T12:45:57.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:57.513+0000 7f9d7ad9d640 1 -- 172.21.15.57:0/958981348 reap_dead start 2023-10-19T12:45:57.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:57.513+0000 7f9d80de2640 1 -- 172.21.15.57:0/958981348 shutdown_connections 2023-10-19T12:45:57.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:57.513+0000 7f9d80de2640 1 -- 172.21.15.57:0/958981348 >> 172.21.15.57:0/958981348 conn(0x7f9d7c0d0d60 msgr2=0x7f9d7c15a2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:57.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:57.513+0000 7f9d80de2640 1 -- 172.21.15.57:0/958981348 shutdown_connections 2023-10-19T12:45:57.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:57.513+0000 7f9d80de2640 1 -- 172.21.15.57:0/958981348 wait complete. 2023-10-19T12:45:57.528 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir0 2023-10-19T12:45:57.528 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:45:57.529 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir1) 2023-10-19T12:45:57.548 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir1 1m 2023-10-19T12:45:57.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:57.701+0000 7fb24c9ce640 1 -- 172.21.15.57:0/2811602065 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb2480c9dc0 msgr2=0x7fb2480ab500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:57.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:57.701+0000 7fb24c9ce640 1 --2- 172.21.15.57:0/2811602065 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb2480c9dc0 0x7fb2480ab500 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fb230005fd0 tx=0x7fb23002be10 comp rx=0 tx=0).stop 2023-10-19T12:45:57.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:57.701+0000 7fb24c9ce640 1 -- 172.21.15.57:0/2811602065 shutdown_connections 2023-10-19T12:45:57.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:57.701+0000 7fb24c9ce640 1 --2- 172.21.15.57:0/2811602065 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb2481533a0 0x7fb2480aba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:57.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:57.701+0000 7fb24c9ce640 1 --2- 172.21.15.57:0/2811602065 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb2480c9dc0 0x7fb2480ab500 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:57.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:57.701+0000 7fb24c9ce640 1 -- 172.21.15.57:0/2811602065 >> 172.21.15.57:0/2811602065 conn(0x7fb248069930 msgr2=0x7fb248069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:57.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:57.701+0000 7fb24c9ce640 1 -- 172.21.15.57:0/2811602065 shutdown_connections 2023-10-19T12:45:57.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:57.701+0000 7fb24c9ce640 1 -- 172.21.15.57:0/2811602065 wait complete. 2023-10-19T12:45:57.707 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:57.702+0000 7fb24c9ce640 1 Processor -- start 2023-10-19T12:45:57.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:57.702+0000 7fb24c9ce640 1 -- start start 2023-10-19T12:45:57.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:57.703+0000 7fb24c9ce640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb2480c99e0 0x7fb248156320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:57.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:57.703+0000 7fb24c9ce640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb2480c9dc0 0x7fb2480b08e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:57.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:57.703+0000 7fb24c9ce640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb2481533a0 0x7fb248136170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:57.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:57.703+0000 7fb24c9ce640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb248148d30 con 0x7fb2481533a0 2023-10-19T12:45:57.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:57.703+0000 7fb24c9ce640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2480af880 con 0x7fb2480c99e0 2023-10-19T12:45:57.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:57.703+0000 7fb24c9ce640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2480afb20 con 0x7fb2480c9dc0 2023-10-19T12:45:57.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:57.703+0000 7fb246d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb2481533a0 0x7fb248136170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:57.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:57.703+0000 7fb246d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb2481533a0 0x7fb248136170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:36084/0 (socket says 172.21.15.57:36084) 2023-10-19T12:45:57.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:57.703+0000 7fb246d9d640 1 -- 172.21.15.57:0/351549897 learned_addr learned my addr 172.21.15.57:0/351549897 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:57.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:57.703+0000 7fb23dd9b640 1 --2- 172.21.15.57:0/351549897 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb2480c9dc0 0x7fb2480b08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:57.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:57.703+0000 7fb24659c640 1 --2- 172.21.15.57:0/351549897 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb2480c99e0 0x7fb248156320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:57.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:57.703+0000 7fb246d9d640 1 -- 172.21.15.57:0/351549897 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb2480c9dc0 msgr2=0x7fb2480b08e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:57.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:57.703+0000 7fb246d9d640 1 --2- 172.21.15.57:0/351549897 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb2480c9dc0 0x7fb2480b08e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:57.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:57.703+0000 7fb246d9d640 1 -- 172.21.15.57:0/351549897 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb2480c99e0 msgr2=0x7fb248156320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:57.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:57.703+0000 7fb246d9d640 1 --2- 172.21.15.57:0/351549897 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb2480c99e0 0x7fb248156320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:57.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:57.703+0000 7fb246d9d640 1 -- 172.21.15.57:0/351549897 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb230009560 con 0x7fb2481533a0 2023-10-19T12:45:57.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:57.704+0000 7fb246d9d640 1 --2- 172.21.15.57:0/351549897 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb2481533a0 0x7fb248136170 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fb24000e960 tx=0x7fb24000ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:57.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:57.704+0000 7fb23f7fe640 1 -- 172.21.15.57:0/351549897 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb240004d20 con 0x7fb2481533a0 2023-10-19T12:45:57.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:57.704+0000 7fb23f7fe640 1 -- 172.21.15.57:0/351549897 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fb24000cb20 con 0x7fb2481533a0 2023-10-19T12:45:57.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:57.704+0000 7fb23f7fe640 1 -- 172.21.15.57:0/351549897 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb240005020 con 0x7fb2481533a0 2023-10-19T12:45:57.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:57.704+0000 7fb24c9ce640 1 -- 172.21.15.57:0/351549897 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2480afe00 con 0x7fb2481533a0 2023-10-19T12:45:57.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:57.704+0000 7fb24c9ce640 1 -- 172.21.15.57:0/351549897 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2480b03c0 con 0x7fb2481533a0 2023-10-19T12:45:57.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:57.705+0000 7fb24c9ce640 1 -- 172.21.15.57:0/351549897 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2480b05b0 con 0x7fb2481533a0 2023-10-19T12:45:57.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:57.705+0000 7fb23f7fe640 1 -- 172.21.15.57:0/351549897 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 8) v1 ==== 89084+0+0 (secure 0 0 0) 0x7fb240005180 con 0x7fb2481533a0 2023-10-19T12:45:57.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:57.706+0000 7fb23f7fe640 1 --2- 172.21.15.57:0/351549897 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fb22806ace0 0x7fb22806d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:57.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:57.706+0000 7fb23f7fe640 1 -- 172.21.15.57:0/351549897 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7fb240089e20 con 0x7fb2481533a0 2023-10-19T12:45:57.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:57.706+0000 7fb24659c640 1 --2- 172.21.15.57:0/351549897 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fb22806ace0 0x7fb22806d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:57.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:57.706+0000 7fb24659c640 1 --2- 172.21.15.57:0/351549897 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fb22806ace0 0x7fb22806d1a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fb238000f60 tx=0x7fb238009040 comp rx=0 tx=0).ready entity=mgr.5428 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:57.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:57.710+0000 7fb23f7fe640 1 -- 172.21.15.57:0/351549897 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb240055000 con 0x7fb2481533a0 2023-10-19T12:45:57.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:57.909+0000 7fb24c9ce640 1 -- 172.21.15.57:0/351549897 --> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fb24815d270 con 0x7fb22806ace0 2023-10-19T12:45:57.974 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:57.971+0000 7fb23f7fe640 1 -- 172.21.15.57:0/351549897 <== mgr.5428 v2:172.21.15.179:6832/922398470 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fb2380014a0 con 0x7fb22806ace0 2023-10-19T12:45:57.974 INFO:teuthology.orchestra.run.smithi057.stdout:Schedule set for path /snap_test_dir1/dir1 2023-10-19T12:45:57.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:57.973+0000 7fb24c9ce640 1 -- 172.21.15.57:0/351549897 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fb22806ace0 msgr2=0x7fb22806d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:57.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:57.973+0000 7fb24c9ce640 1 --2- 172.21.15.57:0/351549897 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fb22806ace0 0x7fb22806d1a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fb238000f60 tx=0x7fb238009040 comp rx=0 tx=0).stop 2023-10-19T12:45:57.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:57.973+0000 7fb24c9ce640 1 -- 172.21.15.57:0/351549897 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb2481533a0 msgr2=0x7fb248136170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:57.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:57.973+0000 7fb24c9ce640 1 --2- 172.21.15.57:0/351549897 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb2481533a0 0x7fb248136170 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fb24000e960 tx=0x7fb24000ee20 comp rx=0 tx=0).stop 2023-10-19T12:45:57.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:57.973+0000 7fb24c9ce640 1 -- 172.21.15.57:0/351549897 shutdown_connections 2023-10-19T12:45:57.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:57.973+0000 7fb24c9ce640 1 --2- 172.21.15.57:0/351549897 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fb22806ace0 0x7fb22806d1a0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:57.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:57.973+0000 7fb24c9ce640 1 --2- 172.21.15.57:0/351549897 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb2481533a0 0x7fb248136170 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:57.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:57.973+0000 7fb24c9ce640 1 --2- 172.21.15.57:0/351549897 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb2480c9dc0 0x7fb2480b08e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:57.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:57.973+0000 7fb24c9ce640 1 --2- 172.21.15.57:0/351549897 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb2480c99e0 0x7fb248156320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:57.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:57.973+0000 7fb24c9ce640 1 -- 172.21.15.57:0/351549897 >> 172.21.15.57:0/351549897 conn(0x7fb248069930 msgr2=0x7fb24813b1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:57.979 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:57.973+0000 7fb24c9ce640 1 -- 172.21.15.57:0/351549897 shutdown_connections 2023-10-19T12:45:57.979 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:57.973+0000 7fb24c9ce640 1 -- 172.21.15.57:0/351549897 wait complete. 2023-10-19T12:45:57.987 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir1 2023-10-19T12:45:57.987 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:45:57.988 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir2) 2023-10-19T12:45:58.005 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir2 1m 2023-10-19T12:45:58.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.153+0000 7f0357e7c640 1 -- 172.21.15.57:0/2618003128 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f035015b2a0 msgr2=0x7f03500d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:58.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.153+0000 7f0357e7c640 1 --2- 172.21.15.57:0/2618003128 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f035015b2a0 0x7f03500d1910 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f0344009830 tx=0x7f034402bcd0 comp rx=0 tx=0).stop 2023-10-19T12:45:58.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.153+0000 7f0357e7c640 1 -- 172.21.15.57:0/2618003128 shutdown_connections 2023-10-19T12:45:58.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.153+0000 7f0357e7c640 1 --2- 172.21.15.57:0/2618003128 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f03500ceb40 0x7f03500b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:58.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.153+0000 7f0357e7c640 1 --2- 172.21.15.57:0/2618003128 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f035015b2a0 0x7f03500d1910 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:58.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.153+0000 7f0357e7c640 1 -- 172.21.15.57:0/2618003128 >> 172.21.15.57:0/2618003128 conn(0x7f03500d0d60 msgr2=0x7f03500d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:58.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.154+0000 7f0357e7c640 1 -- 172.21.15.57:0/2618003128 shutdown_connections 2023-10-19T12:45:58.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.154+0000 7f0357e7c640 1 -- 172.21.15.57:0/2618003128 wait complete. 2023-10-19T12:45:58.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.155+0000 7f0357e7c640 1 Processor -- start 2023-10-19T12:45:58.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.155+0000 7f0357e7c640 1 -- start start 2023-10-19T12:45:58.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.155+0000 7f0357e7c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f03500ceb40 0x7f03500c99b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:58.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.155+0000 7f0357e7c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f03500cf340 0x7f03501f4390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:58.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.155+0000 7f0357e7c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f035015b2a0 0x7f03500aea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:58.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.155+0000 7f0357e7c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03501e5450 con 0x7f03500cf340 2023-10-19T12:45:58.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.155+0000 7f0357e7c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03501e5730 con 0x7f035015b2a0 2023-10-19T12:45:58.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.155+0000 7f0357e7c640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03501e47d0 con 0x7f03500ceb40 2023-10-19T12:45:58.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.156+0000 7f0355417640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f03500cf340 0x7f03501f4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:58.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.156+0000 7f0355417640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f03500cf340 0x7f03501f4390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:36086/0 (socket says 172.21.15.57:36086) 2023-10-19T12:45:58.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.156+0000 7f0355417640 1 -- 172.21.15.57:0/1612346967 learned_addr learned my addr 172.21.15.57:0/1612346967 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:58.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.156+0000 7f0355c18640 1 --2- 172.21.15.57:0/1612346967 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f03500ceb40 0x7f03500c99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:58.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.156+0000 7f0356419640 1 --2- 172.21.15.57:0/1612346967 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f035015b2a0 0x7f03500aea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:58.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.156+0000 7f0355417640 1 -- 172.21.15.57:0/1612346967 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f03500ceb40 msgr2=0x7f03500c99b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:58.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.156+0000 7f0355417640 1 --2- 172.21.15.57:0/1612346967 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f03500ceb40 0x7f03500c99b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:58.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.156+0000 7f0355417640 1 -- 172.21.15.57:0/1612346967 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f035015b2a0 msgr2=0x7f03500aea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:58.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.156+0000 7f0355417640 1 --2- 172.21.15.57:0/1612346967 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f035015b2a0 0x7f03500aea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:58.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.156+0000 7f0355417640 1 -- 172.21.15.57:0/1612346967 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0340009900 con 0x7f03500cf340 2023-10-19T12:45:58.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.156+0000 7f0355417640 1 --2- 172.21.15.57:0/1612346967 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f03500cf340 0x7f03501f4390 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f0344009950 tx=0x7f03440038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:58.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.156+0000 7f034e7fc640 1 -- 172.21.15.57:0/1612346967 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0344003ea0 con 0x7f03500cf340 2023-10-19T12:45:58.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.156+0000 7f034e7fc640 1 -- 172.21.15.57:0/1612346967 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f0344034440 con 0x7f03500cf340 2023-10-19T12:45:58.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.156+0000 7f0357e7c640 1 -- 172.21.15.57:0/1612346967 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0344009490 con 0x7f03500cf340 2023-10-19T12:45:58.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.156+0000 7f034e7fc640 1 -- 172.21.15.57:0/1612346967 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0344034850 con 0x7f03500cf340 2023-10-19T12:45:58.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.156+0000 7f0357e7c640 1 -- 172.21.15.57:0/1612346967 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03501e4d90 con 0x7f03500cf340 2023-10-19T12:45:58.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.157+0000 7f0357e7c640 1 -- 172.21.15.57:0/1612346967 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03501e5590 con 0x7f03500cf340 2023-10-19T12:45:58.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.158+0000 7f034e7fc640 1 -- 172.21.15.57:0/1612346967 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 8) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f03440349b0 con 0x7f03500cf340 2023-10-19T12:45:58.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.158+0000 7f034e7fc640 1 --2- 172.21.15.57:0/1612346967 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f033c06ace0 0x7f033c06d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:58.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.158+0000 7f034e7fc640 1 -- 172.21.15.57:0/1612346967 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7f03440abbc0 con 0x7f03500cf340 2023-10-19T12:45:58.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.158+0000 7f0355c18640 1 --2- 172.21.15.57:0/1612346967 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f033c06ace0 0x7f033c06d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:58.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.159+0000 7f0355c18640 1 --2- 172.21.15.57:0/1612346967 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f033c06ace0 0x7f033c06d1a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f0348001780 tx=0x7f0348009040 comp rx=0 tx=0).ready entity=mgr.5428 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:58.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.162+0000 7f034e7fc640 1 -- 172.21.15.57:0/1612346967 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f03501e5590 con 0x7f03500cf340 2023-10-19T12:45:58.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.359+0000 7f0357e7c640 1 -- 172.21.15.57:0/1612346967 --> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir2", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f03501e5590 con 0x7f033c06ace0 2023-10-19T12:45:58.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.418+0000 7f034e7fc640 1 -- 172.21.15.57:0/1612346967 <== mgr.5428 v2:172.21.15.179:6832/922398470 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f03501e5590 con 0x7f033c06ace0 2023-10-19T12:45:58.422 INFO:teuthology.orchestra.run.smithi057.stdout:Schedule set for path /snap_test_dir1/dir2 2023-10-19T12:45:58.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.421+0000 7f0357e7c640 1 -- 172.21.15.57:0/1612346967 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f033c06ace0 msgr2=0x7f033c06d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:58.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.421+0000 7f0357e7c640 1 --2- 172.21.15.57:0/1612346967 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f033c06ace0 0x7f033c06d1a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f0348001780 tx=0x7f0348009040 comp rx=0 tx=0).stop 2023-10-19T12:45:58.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.421+0000 7f0357e7c640 1 -- 172.21.15.57:0/1612346967 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f03500cf340 msgr2=0x7f03501f4390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:58.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.421+0000 7f0357e7c640 1 --2- 172.21.15.57:0/1612346967 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f03500cf340 0x7f03501f4390 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f0344009950 tx=0x7f03440038a0 comp rx=0 tx=0).stop 2023-10-19T12:45:58.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.421+0000 7f0357e7c640 1 -- 172.21.15.57:0/1612346967 shutdown_connections 2023-10-19T12:45:58.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.421+0000 7f0357e7c640 1 --2- 172.21.15.57:0/1612346967 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f033c06ace0 0x7f033c06d1a0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:58.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.421+0000 7f0357e7c640 1 --2- 172.21.15.57:0/1612346967 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f035015b2a0 0x7f03500aea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:58.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.421+0000 7f0357e7c640 1 --2- 172.21.15.57:0/1612346967 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f03500cf340 0x7f03501f4390 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:58.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.421+0000 7f0357e7c640 1 --2- 172.21.15.57:0/1612346967 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f03500ceb40 0x7f03500c99b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:58.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.421+0000 7f0357e7c640 1 -- 172.21.15.57:0/1612346967 >> 172.21.15.57:0/1612346967 conn(0x7f03500d0d60 msgr2=0x7f0350069f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:58.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.421+0000 7f0357e7c640 1 -- 172.21.15.57:0/1612346967 shutdown_connections 2023-10-19T12:45:58.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.421+0000 7f0357e7c640 1 -- 172.21.15.57:0/1612346967 wait complete. 2023-10-19T12:45:58.434 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir2 2023-10-19T12:45:58.435 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:45:58.435 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir3) 2023-10-19T12:45:58.452 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir3 1m 2023-10-19T12:45:58.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.600+0000 7fe22f2f2640 1 -- 172.21.15.57:0/3674988205 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe22813e520 msgr2=0x7fe22806bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:58.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.600+0000 7fe22f2f2640 1 --2- 172.21.15.57:0/3674988205 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe22813e520 0x7fe22806bb60 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe220009830 tx=0x7fe22002bcd0 comp rx=0 tx=0).stop 2023-10-19T12:45:58.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.600+0000 7fe22f2f2640 1 -- 172.21.15.57:0/3674988205 shutdown_connections 2023-10-19T12:45:58.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.600+0000 7fe22f2f2640 1 --2- 172.21.15.57:0/3674988205 >> v2:172.21.15.179:3300/0 conn(0x7fe228070af0 0x7fe2280affe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:58.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.600+0000 7fe22f2f2640 1 --2- 172.21.15.57:0/3674988205 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe22813e520 0x7fe22806bb60 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:58.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.600+0000 7fe22f2f2640 1 -- 172.21.15.57:0/3674988205 >> 172.21.15.57:0/3674988205 conn(0x7fe228069930 msgr2=0x7fe228069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:58.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.600+0000 7fe22f2f2640 1 -- 172.21.15.57:0/3674988205 shutdown_connections 2023-10-19T12:45:58.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.600+0000 7fe22f2f2640 1 -- 172.21.15.57:0/3674988205 wait complete. 2023-10-19T12:45:58.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.602+0000 7fe22f2f2640 1 Processor -- start 2023-10-19T12:45:58.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.602+0000 7fe22f2f2640 1 -- start start 2023-10-19T12:45:58.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.602+0000 7fe22f2f2640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe228070af0 0x7fe2281e1b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:58.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.602+0000 7fe22f2f2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe22813e520 0x7fe2281e3700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:58.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.602+0000 7fe22f2f2640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe228141690 0x7fe2280ae950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:58.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.602+0000 7fe22f2f2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2281e3270 con 0x7fe22813e520 2023-10-19T12:45:58.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.602+0000 7fe22f2f2640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2281e2940 con 0x7fe228070af0 2023-10-19T12:45:58.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.602+0000 7fe22f2f2640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2281e2c20 con 0x7fe228141690 2023-10-19T12:45:58.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.602+0000 7fe22c88d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe22813e520 0x7fe2281e3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:58.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.603+0000 7fe22c88d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe22813e520 0x7fe2281e3700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:36092/0 (socket says 172.21.15.57:36092) 2023-10-19T12:45:58.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.603+0000 7fe22c88d640 1 -- 172.21.15.57:0/934490091 learned_addr learned my addr 172.21.15.57:0/934490091 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:58.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.603+0000 7fe22d08e640 1 --2- 172.21.15.57:0/934490091 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe228070af0 0x7fe2281e1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:58.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.603+0000 7fe22d88f640 1 --2- 172.21.15.57:0/934490091 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe228141690 0x7fe2280ae950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:58.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.603+0000 7fe22c88d640 1 -- 172.21.15.57:0/934490091 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe228141690 msgr2=0x7fe2280ae950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:58.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.603+0000 7fe22c88d640 1 --2- 172.21.15.57:0/934490091 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe228141690 0x7fe2280ae950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:58.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.603+0000 7fe22c88d640 1 -- 172.21.15.57:0/934490091 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe228070af0 msgr2=0x7fe2281e1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:58.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.603+0000 7fe22c88d640 1 --2- 172.21.15.57:0/934490091 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe228070af0 0x7fe2281e1b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:58.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.603+0000 7fe22c88d640 1 -- 172.21.15.57:0/934490091 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe218009840 con 0x7fe22813e520 2023-10-19T12:45:58.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.603+0000 7fe22c88d640 1 --2- 172.21.15.57:0/934490091 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe22813e520 0x7fe2281e3700 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fe21c00c390 tx=0x7fe21c00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:58.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.603+0000 7fe225ffb640 1 -- 172.21.15.57:0/934490091 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe21c004230 con 0x7fe22813e520 2023-10-19T12:45:58.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.603+0000 7fe225ffb640 1 -- 172.21.15.57:0/934490091 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fe21c0044a0 con 0x7fe22813e520 2023-10-19T12:45:58.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.603+0000 7fe22f2f2640 1 -- 172.21.15.57:0/934490091 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe220009490 con 0x7fe22813e520 2023-10-19T12:45:58.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.603+0000 7fe225ffb640 1 -- 172.21.15.57:0/934490091 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe21c003850 con 0x7fe22813e520 2023-10-19T12:45:58.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.603+0000 7fe22f2f2640 1 -- 172.21.15.57:0/934490091 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2281e20c0 con 0x7fe22813e520 2023-10-19T12:45:58.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.604+0000 7fe22f2f2640 1 -- 172.21.15.57:0/934490091 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2281e3420 con 0x7fe22813e520 2023-10-19T12:45:58.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.605+0000 7fe225ffb640 1 -- 172.21.15.57:0/934490091 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 8) v1 ==== 89084+0+0 (secure 0 0 0) 0x7fe21c0039f0 con 0x7fe22813e520 2023-10-19T12:45:58.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.605+0000 7fe225ffb640 1 --2- 172.21.15.57:0/934490091 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fe21406ace0 0x7fe21406d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:58.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.605+0000 7fe225ffb640 1 -- 172.21.15.57:0/934490091 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7fe21c088e40 con 0x7fe22813e520 2023-10-19T12:45:58.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.606+0000 7fe22d08e640 1 --2- 172.21.15.57:0/934490091 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fe21406ace0 0x7fe21406d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:58.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.606+0000 7fe22d08e640 1 --2- 172.21.15.57:0/934490091 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fe21406ace0 0x7fe21406d1a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe220036000 tx=0x7fe220004730 comp rx=0 tx=0).ready entity=mgr.5428 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:58.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.609+0000 7fe225ffb640 1 -- 172.21.15.57:0/934490091 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe2281e3420 con 0x7fe22813e520 2023-10-19T12:45:58.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.804+0000 7fe22f2f2640 1 -- 172.21.15.57:0/934490091 --> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir3", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fe2280b1bd0 con 0x7fe21406ace0 2023-10-19T12:45:58.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.892+0000 7fe225ffb640 1 -- 172.21.15.57:0/934490091 <== mgr.5428 v2:172.21.15.179:6832/922398470 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fe2280b1bd0 con 0x7fe21406ace0 2023-10-19T12:45:58.895 INFO:teuthology.orchestra.run.smithi057.stdout:Schedule set for path /snap_test_dir1/dir3 2023-10-19T12:45:58.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.894+0000 7fe22f2f2640 1 -- 172.21.15.57:0/934490091 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fe21406ace0 msgr2=0x7fe21406d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:58.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.894+0000 7fe22f2f2640 1 --2- 172.21.15.57:0/934490091 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fe21406ace0 0x7fe21406d1a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe220036000 tx=0x7fe220004730 comp rx=0 tx=0).stop 2023-10-19T12:45:58.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.894+0000 7fe22f2f2640 1 -- 172.21.15.57:0/934490091 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe22813e520 msgr2=0x7fe2281e3700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:58.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.894+0000 7fe22f2f2640 1 --2- 172.21.15.57:0/934490091 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe22813e520 0x7fe2281e3700 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fe21c00c390 tx=0x7fe21c00c850 comp rx=0 tx=0).stop 2023-10-19T12:45:58.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.894+0000 7fe22f2f2640 1 -- 172.21.15.57:0/934490091 shutdown_connections 2023-10-19T12:45:58.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.894+0000 7fe22f2f2640 1 --2- 172.21.15.57:0/934490091 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fe21406ace0 0x7fe21406d1a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:58.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.894+0000 7fe22f2f2640 1 --2- 172.21.15.57:0/934490091 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe228141690 0x7fe2280ae950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:58.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.894+0000 7fe22f2f2640 1 --2- 172.21.15.57:0/934490091 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe22813e520 0x7fe2281e3700 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:58.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.894+0000 7fe22f2f2640 1 --2- 172.21.15.57:0/934490091 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe228070af0 0x7fe2281e1b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:58.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.894+0000 7fe22f2f2640 1 -- 172.21.15.57:0/934490091 >> 172.21.15.57:0/934490091 conn(0x7fe228069930 msgr2=0x7fe2281e0770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:58.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.894+0000 7fe22f2f2640 1 -- 172.21.15.57:0/934490091 shutdown_connections 2023-10-19T12:45:58.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:58.894+0000 7fe22f2f2640 1 -- 172.21.15.57:0/934490091 wait complete. 2023-10-19T12:45:58.908 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir3 2023-10-19T12:45:58.908 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:45:58.908 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir4) 2023-10-19T12:45:58.926 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir4 1m 2023-10-19T12:45:59.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.075+0000 7fb690c44640 1 -- 172.21.15.57:0/3447151026 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb68c0ab500 msgr2=0x7fb68c0ab9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:59.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.075+0000 7fb690c44640 1 --2- 172.21.15.57:0/3447151026 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb68c0ab500 0x7fb68c0ab9b0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fb67c005fd0 tx=0x7fb67c02c2e0 comp rx=0 tx=0).stop 2023-10-19T12:45:59.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.075+0000 7fb690c44640 1 -- 172.21.15.57:0/3447151026 shutdown_connections 2023-10-19T12:45:59.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.075+0000 7fb690c44640 1 --2- 172.21.15.57:0/3447151026 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb68c0c9dc0 0x7fb68c06b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:59.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.075+0000 7fb690c44640 1 --2- 172.21.15.57:0/3447151026 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb68c0ab500 0x7fb68c0ab9b0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:59.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.075+0000 7fb690c44640 1 -- 172.21.15.57:0/3447151026 >> 172.21.15.57:0/3447151026 conn(0x7fb68c069930 msgr2=0x7fb68c069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:59.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.075+0000 7fb690c44640 1 -- 172.21.15.57:0/3447151026 shutdown_connections 2023-10-19T12:45:59.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.075+0000 7fb690c44640 1 -- 172.21.15.57:0/3447151026 wait complete. 2023-10-19T12:45:59.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.076+0000 7fb690c44640 1 Processor -- start 2023-10-19T12:45:59.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.076+0000 7fb690c44640 1 -- start start 2023-10-19T12:45:59.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.077+0000 7fb690c44640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb68c06af00 0x7fb68c13b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:59.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.077+0000 7fb690c44640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb68c0ab500 0x7fb68c06f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:59.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.077+0000 7fb690c44640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb68c0c9dc0 0x7fb68c0aa8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:59.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.077+0000 7fb690c44640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb68c13b7f0 con 0x7fb68c06af00 2023-10-19T12:45:59.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.077+0000 7fb690c44640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb68c06f5c0 con 0x7fb68c0ab500 2023-10-19T12:45:59.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.077+0000 7fb690c44640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb68c06f890 con 0x7fb68c0c9dc0 2023-10-19T12:45:59.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.077+0000 7fb68a59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb68c06af00 0x7fb68c13b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:59.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.077+0000 7fb68a59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb68c06af00 0x7fb68c13b2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:36112/0 (socket says 172.21.15.57:36112) 2023-10-19T12:45:59.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.077+0000 7fb68a59c640 1 -- 172.21.15.57:0/690608262 learned_addr learned my addr 172.21.15.57:0/690608262 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:59.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.077+0000 7fb689d9b640 1 --2- 172.21.15.57:0/690608262 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb68c0ab500 0x7fb68c06f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:59.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.077+0000 7fb68ad9d640 1 --2- 172.21.15.57:0/690608262 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb68c0c9dc0 0x7fb68c0aa8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:59.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.077+0000 7fb68a59c640 1 -- 172.21.15.57:0/690608262 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb68c0c9dc0 msgr2=0x7fb68c0aa8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:59.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.077+0000 7fb68a59c640 1 --2- 172.21.15.57:0/690608262 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb68c0c9dc0 0x7fb68c0aa8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:59.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.077+0000 7fb68a59c640 1 -- 172.21.15.57:0/690608262 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb68c0ab500 msgr2=0x7fb68c06f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:59.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.078+0000 7fb68a59c640 1 --2- 172.21.15.57:0/690608262 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb68c0ab500 0x7fb68c06f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:59.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.078+0000 7fb68a59c640 1 -- 172.21.15.57:0/690608262 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb67c009900 con 0x7fb68c06af00 2023-10-19T12:45:59.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.078+0000 7fb68ad9d640 1 --2- 172.21.15.57:0/690608262 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb68c0c9dc0 0x7fb68c0aa8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:45:59.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.078+0000 7fb689d9b640 1 --2- 172.21.15.57:0/690608262 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb68c0ab500 0x7fb68c06f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:45:59.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.078+0000 7fb68a59c640 1 --2- 172.21.15.57:0/690608262 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb68c06af00 0x7fb68c13b2b0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fb68400c450 tx=0x7fb68400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:59.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.078+0000 7fb682ffd640 1 -- 172.21.15.57:0/690608262 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6840052d0 con 0x7fb68c06af00 2023-10-19T12:45:59.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.078+0000 7fb682ffd640 1 -- 172.21.15.57:0/690608262 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fb684005430 con 0x7fb68c06af00 2023-10-19T12:45:59.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.078+0000 7fb690c44640 1 -- 172.21.15.57:0/690608262 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb68c06fb70 con 0x7fb68c06af00 2023-10-19T12:45:59.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.078+0000 7fb682ffd640 1 -- 172.21.15.57:0/690608262 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb684012890 con 0x7fb68c06af00 2023-10-19T12:45:59.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.078+0000 7fb690c44640 1 -- 172.21.15.57:0/690608262 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb68c070130 con 0x7fb68c06af00 2023-10-19T12:45:59.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.079+0000 7fb690c44640 1 -- 172.21.15.57:0/690608262 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb68c0690e0 con 0x7fb68c06af00 2023-10-19T12:45:59.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.079+0000 7fb682ffd640 1 -- 172.21.15.57:0/690608262 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 8) v1 ==== 89084+0+0 (secure 0 0 0) 0x7fb6840035f0 con 0x7fb68c06af00 2023-10-19T12:45:59.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.080+0000 7fb682ffd640 1 --2- 172.21.15.57:0/690608262 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fb66c06ace0 0x7fb66c06d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:59.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.080+0000 7fb682ffd640 1 -- 172.21.15.57:0/690608262 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7fb684089170 con 0x7fb68c06af00 2023-10-19T12:45:59.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.080+0000 7fb689d9b640 1 --2- 172.21.15.57:0/690608262 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fb66c06ace0 0x7fb66c06d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:59.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.081+0000 7fb689d9b640 1 --2- 172.21.15.57:0/690608262 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fb66c06ace0 0x7fb66c06d1a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb674005fd0 tx=0x7fb67400f360 comp rx=0 tx=0).ready entity=mgr.5428 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:59.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.084+0000 7fb682ffd640 1 -- 172.21.15.57:0/690608262 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb684054320 con 0x7fb68c06af00 2023-10-19T12:45:59.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.281+0000 7fb690c44640 1 -- 172.21.15.57:0/690608262 --> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir4", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fb68c159c50 con 0x7fb66c06ace0 2023-10-19T12:45:59.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.346+0000 7fb682ffd640 1 -- 172.21.15.57:0/690608262 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mgrmap(e 9) v1 ==== 89349+0+0 (secure 0 0 0) 0x7fb6840546f0 con 0x7fb68c06af00 2023-10-19T12:45:59.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.373+0000 7fb682ffd640 1 -- 172.21.15.57:0/690608262 <== mgr.5428 v2:172.21.15.179:6832/922398470 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fb67400b890 con 0x7fb66c06ace0 2023-10-19T12:45:59.377 INFO:teuthology.orchestra.run.smithi057.stdout:Schedule set for path /snap_test_dir1/dir4 2023-10-19T12:45:59.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.375+0000 7fb690c44640 1 -- 172.21.15.57:0/690608262 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fb66c06ace0 msgr2=0x7fb66c06d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:59.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.375+0000 7fb690c44640 1 --2- 172.21.15.57:0/690608262 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fb66c06ace0 0x7fb66c06d1a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb674005fd0 tx=0x7fb67400f360 comp rx=0 tx=0).stop 2023-10-19T12:45:59.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.375+0000 7fb690c44640 1 -- 172.21.15.57:0/690608262 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb68c06af00 msgr2=0x7fb68c13b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:59.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.375+0000 7fb690c44640 1 --2- 172.21.15.57:0/690608262 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb68c06af00 0x7fb68c13b2b0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fb68400c450 tx=0x7fb68400c910 comp rx=0 tx=0).stop 2023-10-19T12:45:59.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.375+0000 7fb690c44640 1 -- 172.21.15.57:0/690608262 shutdown_connections 2023-10-19T12:45:59.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.375+0000 7fb690c44640 1 --2- 172.21.15.57:0/690608262 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fb66c06ace0 0x7fb66c06d1a0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:59.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.375+0000 7fb690c44640 1 --2- 172.21.15.57:0/690608262 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb68c0c9dc0 0x7fb68c0aa8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:59.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.375+0000 7fb690c44640 1 --2- 172.21.15.57:0/690608262 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb68c0ab500 0x7fb68c06f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:59.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.375+0000 7fb690c44640 1 --2- 172.21.15.57:0/690608262 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb68c06af00 0x7fb68c13b2b0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:59.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.375+0000 7fb690c44640 1 -- 172.21.15.57:0/690608262 >> 172.21.15.57:0/690608262 conn(0x7fb68c069930 msgr2=0x7fb68c0c9680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:59.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.376+0000 7fb690c44640 1 -- 172.21.15.57:0/690608262 shutdown_connections 2023-10-19T12:45:59.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.376+0000 7fb690c44640 1 -- 172.21.15.57:0/690608262 wait complete. 2023-10-19T12:45:59.389 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir4 2023-10-19T12:45:59.389 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:45:59.390 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir5) 2023-10-19T12:45:59.407 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir5 1m 2023-10-19T12:45:59.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.556+0000 7fd767897640 1 -- 172.21.15.57:0/1035832044 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd74c003970 msgr2=0x7fd74c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:59.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.556+0000 7fd767897640 1 --2- 172.21.15.57:0/1035832044 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd74c003970 0x7fd74c003d90 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fd75c009b50 tx=0x7fd74c0057f0 comp rx=0 tx=0).stop 2023-10-19T12:45:59.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.556+0000 7fd765e34640 1 -- 172.21.15.57:0/1035832044 reap_dead start 2023-10-19T12:45:59.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.556+0000 7fd767897640 1 -- 172.21.15.57:0/1035832044 shutdown_connections 2023-10-19T12:45:59.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.556+0000 7fd767897640 1 -- 172.21.15.57:0/1035832044 >> 172.21.15.57:0/1035832044 conn(0x7fd7600d0d60 msgr2=0x7fd7600d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:59.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.556+0000 7fd767897640 1 -- 172.21.15.57:0/1035832044 shutdown_connections 2023-10-19T12:45:59.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.556+0000 7fd767897640 1 -- 172.21.15.57:0/1035832044 wait complete. 2023-10-19T12:45:59.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.557+0000 7fd767897640 1 Processor -- start 2023-10-19T12:45:59.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.557+0000 7fd767897640 1 -- start start 2023-10-19T12:45:59.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.558+0000 7fd767897640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd74c003970 0x7fd76015e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:59.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.558+0000 7fd767897640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd76015ccc0 0x7fd7601f1f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:59.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.558+0000 7fd767897640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd7600b60c0 0x7fd7600c85b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:59.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.558+0000 7fd767897640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7601f2450 con 0x7fd74c003970 2023-10-19T12:45:59.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.558+0000 7fd767897640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd76015ddb0 con 0x7fd76015ccc0 2023-10-19T12:45:59.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.558+0000 7fd767897640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd76015d1a0 con 0x7fd7600b60c0 2023-10-19T12:45:59.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.558+0000 7fd765633640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd74c003970 0x7fd76015e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:59.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.558+0000 7fd765633640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd74c003970 0x7fd76015e240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:36140/0 (socket says 172.21.15.57:36140) 2023-10-19T12:45:59.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.558+0000 7fd765633640 1 -- 172.21.15.57:0/1793898731 learned_addr learned my addr 172.21.15.57:0/1793898731 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:45:59.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.558+0000 7fd764e32640 1 --2- 172.21.15.57:0/1793898731 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd76015ccc0 0x7fd7601f1f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:59.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.558+0000 7fd765e34640 1 --2- 172.21.15.57:0/1793898731 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd7600b60c0 0x7fd7600c85b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:59.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.558+0000 7fd765633640 1 -- 172.21.15.57:0/1793898731 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd7600b60c0 msgr2=0x7fd7600c85b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:59.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.558+0000 7fd765633640 1 --2- 172.21.15.57:0/1793898731 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd7600b60c0 0x7fd7600c85b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:59.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.558+0000 7fd765633640 1 -- 172.21.15.57:0/1793898731 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd76015ccc0 msgr2=0x7fd7601f1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:59.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.558+0000 7fd765633640 1 --2- 172.21.15.57:0/1793898731 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd76015ccc0 0x7fd7601f1f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:59.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.558+0000 7fd765633640 1 -- 172.21.15.57:0/1793898731 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd76015d480 con 0x7fd74c003970 2023-10-19T12:45:59.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.558+0000 7fd765633640 1 --2- 172.21.15.57:0/1793898731 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd74c003970 0x7fd76015e240 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fd75c009b50 tx=0x7fd75c033c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:59.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.559+0000 7fd755ffb640 1 -- 172.21.15.57:0/1793898731 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd75c033ea0 con 0x7fd74c003970 2023-10-19T12:45:59.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.559+0000 7fd755ffb640 1 -- 172.21.15.57:0/1793898731 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fd75c004460 con 0x7fd74c003970 2023-10-19T12:45:59.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.559+0000 7fd767897640 1 -- 172.21.15.57:0/1793898731 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd76015d760 con 0x7fd74c003970 2023-10-19T12:45:59.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.559+0000 7fd755ffb640 1 -- 172.21.15.57:0/1793898731 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd75c003850 con 0x7fd74c003970 2023-10-19T12:45:59.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.559+0000 7fd767897640 1 -- 172.21.15.57:0/1793898731 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7600cab70 con 0x7fd74c003970 2023-10-19T12:45:59.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.560+0000 7fd767897640 1 -- 172.21.15.57:0/1793898731 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7600690e0 con 0x7fd74c003970 2023-10-19T12:45:59.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.560+0000 7fd755ffb640 1 -- 172.21.15.57:0/1793898731 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 9) v1 ==== 89349+0+0 (secure 0 0 0) 0x7fd75c0039b0 con 0x7fd74c003970 2023-10-19T12:45:59.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.561+0000 7fd755ffb640 1 --2- 172.21.15.57:0/1793898731 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fd74406afe0 0x7fd74406d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:45:59.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.561+0000 7fd755ffb640 1 -- 172.21.15.57:0/1793898731 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7fd75c0ac710 con 0x7fd74c003970 2023-10-19T12:45:59.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.561+0000 7fd764e32640 1 --2- 172.21.15.57:0/1793898731 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fd74406afe0 0x7fd74406d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:45:59.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.561+0000 7fd764e32640 1 --2- 172.21.15.57:0/1793898731 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fd74406afe0 0x7fd74406d4a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd76009ec30 tx=0x7fd750006040 comp rx=0 tx=0).ready entity=mgr.5428 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:45:59.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.565+0000 7fd755ffb640 1 -- 172.21.15.57:0/1793898731 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd75c077400 con 0x7fd74c003970 2023-10-19T12:45:59.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.764+0000 7fd767897640 1 -- 172.21.15.57:0/1793898731 --> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir5", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fd7601f2450 con 0x7fd74406afe0 2023-10-19T12:45:59.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.842+0000 7fd755ffb640 1 -- 172.21.15.57:0/1793898731 <== mgr.5428 v2:172.21.15.179:6832/922398470 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fd7601f2450 con 0x7fd74406afe0 2023-10-19T12:45:59.846 INFO:teuthology.orchestra.run.smithi057.stdout:Schedule set for path /snap_test_dir1/dir5 2023-10-19T12:45:59.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.844+0000 7fd767897640 1 -- 172.21.15.57:0/1793898731 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fd74406afe0 msgr2=0x7fd74406d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:59.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.844+0000 7fd767897640 1 --2- 172.21.15.57:0/1793898731 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fd74406afe0 0x7fd74406d4a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd76009ec30 tx=0x7fd750006040 comp rx=0 tx=0).stop 2023-10-19T12:45:59.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.844+0000 7fd767897640 1 -- 172.21.15.57:0/1793898731 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd74c003970 msgr2=0x7fd76015e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:45:59.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.844+0000 7fd767897640 1 --2- 172.21.15.57:0/1793898731 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd74c003970 0x7fd76015e240 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fd75c009b50 tx=0x7fd75c033c70 comp rx=0 tx=0).stop 2023-10-19T12:45:59.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.845+0000 7fd767897640 1 -- 172.21.15.57:0/1793898731 shutdown_connections 2023-10-19T12:45:59.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.845+0000 7fd767897640 1 --2- 172.21.15.57:0/1793898731 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fd74406afe0 0x7fd74406d4a0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:59.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.845+0000 7fd767897640 1 --2- 172.21.15.57:0/1793898731 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd7600b60c0 0x7fd7600c85b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:59.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.845+0000 7fd767897640 1 --2- 172.21.15.57:0/1793898731 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd76015ccc0 0x7fd7601f1f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:59.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.845+0000 7fd767897640 1 --2- 172.21.15.57:0/1793898731 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd74c003970 0x7fd76015e240 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:45:59.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.845+0000 7fd767897640 1 -- 172.21.15.57:0/1793898731 >> 172.21.15.57:0/1793898731 conn(0x7fd7600d0d60 msgr2=0x7fd760158a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:45:59.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.845+0000 7fd767897640 1 -- 172.21.15.57:0/1793898731 shutdown_connections 2023-10-19T12:45:59.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:45:59.845+0000 7fd767897640 1 -- 172.21.15.57:0/1793898731 wait complete. 2023-10-19T12:45:59.858 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir5 2023-10-19T12:45:59.859 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:45:59.859 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir6) 2023-10-19T12:45:59.876 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir6 1m 2023-10-19T12:46:00.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.026+0000 7f8e5e26b640 1 -- 172.21.15.57:0/4167571966 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e580ceb40 msgr2=0x7f8e580b0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:46:00.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.026+0000 7f8e5e26b640 1 --2- 172.21.15.57:0/4167571966 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e580ceb40 0x7f8e580b0650 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f8e5806c510 tx=0x7f8e5002c1d0 comp rx=0 tx=0).stop 2023-10-19T12:46:00.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.026+0000 7f8e5e26b640 1 -- 172.21.15.57:0/4167571966 shutdown_connections 2023-10-19T12:46:00.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.026+0000 7f8e5e26b640 1 --2- 172.21.15.57:0/4167571966 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e580ceb40 0x7f8e580b0650 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:00.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.026+0000 7f8e5e26b640 1 --2- 172.21.15.57:0/4167571966 >> v2:172.21.15.179:3300/0 conn(0x7f8e5815b2a0 0x7f8e580cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:00.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.026+0000 7f8e5e26b640 1 --2- 172.21.15.57:0/4167571966 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8e580cf340 0x7f8e580d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:00.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.026+0000 7f8e5e26b640 1 -- 172.21.15.57:0/4167571966 >> 172.21.15.57:0/4167571966 conn(0x7f8e580d0d60 msgr2=0x7f8e580d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:46:00.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.026+0000 7f8e5e26b640 1 -- 172.21.15.57:0/4167571966 shutdown_connections 2023-10-19T12:46:00.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.026+0000 7f8e5e26b640 1 -- 172.21.15.57:0/4167571966 wait complete. 2023-10-19T12:46:00.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.027+0000 7f8e5e26b640 1 Processor -- start 2023-10-19T12:46:00.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.028+0000 7f8e5e26b640 1 -- start start 2023-10-19T12:46:00.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.028+0000 7f8e5e26b640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8e580ceb40 0x7f8e58149080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:46:00.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.028+0000 7f8e5e26b640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8e580cf340 0x7f8e581499a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:46:00.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.028+0000 7f8e5e26b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e5815b2a0 0x7f8e580c9170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:46:00.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.028+0000 7f8e5e26b640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e580c7e30 con 0x7f8e5815b2a0 2023-10-19T12:46:00.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.028+0000 7f8e5e26b640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e580c80d0 con 0x7f8e580ceb40 2023-10-19T12:46:00.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.028+0000 7f8e5e26b640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e580c83b0 con 0x7f8e580cf340 2023-10-19T12:46:00.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.028+0000 7f8e5c808640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e5815b2a0 0x7f8e580c9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:46:00.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.028+0000 7f8e5c808640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e5815b2a0 0x7f8e580c9170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:36164/0 (socket says 172.21.15.57:36164) 2023-10-19T12:46:00.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.028+0000 7f8e5c808640 1 -- 172.21.15.57:0/1827896878 learned_addr learned my addr 172.21.15.57:0/1827896878 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:46:00.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.029+0000 7f8e57fff640 1 --2- 172.21.15.57:0/1827896878 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8e580ceb40 0x7f8e58149080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:46:00.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.029+0000 7f8e577fe640 1 --2- 172.21.15.57:0/1827896878 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8e580cf340 0x7f8e581499a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:46:00.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.029+0000 7f8e5c808640 1 -- 172.21.15.57:0/1827896878 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8e580cf340 msgr2=0x7f8e581499a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:46:00.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.029+0000 7f8e5c808640 1 --2- 172.21.15.57:0/1827896878 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8e580cf340 0x7f8e581499a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:00.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.029+0000 7f8e5c808640 1 -- 172.21.15.57:0/1827896878 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8e580ceb40 msgr2=0x7f8e58149080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:46:00.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.029+0000 7f8e5c808640 1 --2- 172.21.15.57:0/1827896878 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8e580ceb40 0x7f8e58149080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:00.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.029+0000 7f8e5c808640 1 -- 172.21.15.57:0/1827896878 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e50009900 con 0x7f8e5815b2a0 2023-10-19T12:46:00.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.029+0000 7f8e5c808640 1 --2- 172.21.15.57:0/1827896878 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e5815b2a0 0x7f8e580c9170 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f8e50037040 tx=0x7f8e50035d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:46:00.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.029+0000 7f8e54ff9640 1 -- 172.21.15.57:0/1827896878 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e50034440 con 0x7f8e5815b2a0 2023-10-19T12:46:00.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.029+0000 7f8e54ff9640 1 -- 172.21.15.57:0/1827896878 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f8e50004540 con 0x7f8e5815b2a0 2023-10-19T12:46:00.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.029+0000 7f8e5e26b640 1 -- 172.21.15.57:0/1827896878 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e580c8690 con 0x7f8e5815b2a0 2023-10-19T12:46:00.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.029+0000 7f8e54ff9640 1 -- 172.21.15.57:0/1827896878 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e50003e60 con 0x7f8e5815b2a0 2023-10-19T12:46:00.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.029+0000 7f8e5e26b640 1 -- 172.21.15.57:0/1827896878 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e580c8c50 con 0x7f8e5815b2a0 2023-10-19T12:46:00.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.030+0000 7f8e5e26b640 1 -- 172.21.15.57:0/1827896878 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e580c8e40 con 0x7f8e5815b2a0 2023-10-19T12:46:00.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.031+0000 7f8e54ff9640 1 -- 172.21.15.57:0/1827896878 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 9) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f8e5003b040 con 0x7f8e5815b2a0 2023-10-19T12:46:00.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.031+0000 7f8e54ff9640 1 --2- 172.21.15.57:0/1827896878 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f8e3806afe0 0x7f8e3806d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:46:00.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.031+0000 7f8e54ff9640 1 -- 172.21.15.57:0/1827896878 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7f8e500ac5c0 con 0x7f8e5815b2a0 2023-10-19T12:46:00.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.031+0000 7f8e57fff640 1 --2- 172.21.15.57:0/1827896878 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f8e3806afe0 0x7f8e3806d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:46:00.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.032+0000 7f8e57fff640 1 --2- 172.21.15.57:0/1827896878 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f8e3806afe0 0x7f8e3806d4a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8e48009b50 tx=0x7f8e48005f90 comp rx=0 tx=0).ready entity=mgr.5428 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:46:00.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.035+0000 7f8e54ff9640 1 -- 172.21.15.57:0/1827896878 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8e50077340 con 0x7f8e5815b2a0 2023-10-19T12:46:00.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.233+0000 7f8e5e26b640 1 -- 172.21.15.57:0/1827896878 --> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir6", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f8e580cc170 con 0x7f8e3806afe0 2023-10-19T12:46:00.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.302+0000 7f8e54ff9640 1 -- 172.21.15.57:0/1827896878 <== mgr.5428 v2:172.21.15.179:6832/922398470 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f8e48009880 con 0x7f8e3806afe0 2023-10-19T12:46:00.306 INFO:teuthology.orchestra.run.smithi057.stdout:Schedule set for path /snap_test_dir1/dir6 2023-10-19T12:46:00.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.305+0000 7f8e5e26b640 1 -- 172.21.15.57:0/1827896878 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f8e3806afe0 msgr2=0x7f8e3806d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:46:00.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.305+0000 7f8e5e26b640 1 --2- 172.21.15.57:0/1827896878 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f8e3806afe0 0x7f8e3806d4a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8e48009b50 tx=0x7f8e48005f90 comp rx=0 tx=0).stop 2023-10-19T12:46:00.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.305+0000 7f8e5e26b640 1 -- 172.21.15.57:0/1827896878 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e5815b2a0 msgr2=0x7f8e580c9170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:46:00.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.305+0000 7f8e5e26b640 1 --2- 172.21.15.57:0/1827896878 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e5815b2a0 0x7f8e580c9170 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f8e50037040 tx=0x7f8e50035d00 comp rx=0 tx=0).stop 2023-10-19T12:46:00.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.305+0000 7f8e5e26b640 1 -- 172.21.15.57:0/1827896878 shutdown_connections 2023-10-19T12:46:00.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.305+0000 7f8e5e26b640 1 --2- 172.21.15.57:0/1827896878 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f8e3806afe0 0x7f8e3806d4a0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:00.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.305+0000 7f8e5e26b640 1 --2- 172.21.15.57:0/1827896878 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e5815b2a0 0x7f8e580c9170 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:00.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.305+0000 7f8e5e26b640 1 --2- 172.21.15.57:0/1827896878 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8e580cf340 0x7f8e581499a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:00.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.305+0000 7f8e5e26b640 1 --2- 172.21.15.57:0/1827896878 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8e580ceb40 0x7f8e58149080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:00.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.305+0000 7f8e5e26b640 1 -- 172.21.15.57:0/1827896878 >> 172.21.15.57:0/1827896878 conn(0x7f8e580d0d60 msgr2=0x7f8e5806b520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:46:00.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.305+0000 7f8e5e26b640 1 -- 172.21.15.57:0/1827896878 shutdown_connections 2023-10-19T12:46:00.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.305+0000 7f8e5e26b640 1 -- 172.21.15.57:0/1827896878 wait complete. 2023-10-19T12:46:00.318 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir6 2023-10-19T12:46:00.319 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:46:00.319 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir7) 2023-10-19T12:46:00.336 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir7 1m 2023-10-19T12:46:00.385 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:46:00.383+0000 7fe48a830640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T12:46:00.386 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:46:00.383+0000 7fe48a830640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T12:46:00.386 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:46:00.383+0000 7fe48a830640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T12:46:00.386 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:46:00.383+0000 7fe48a830640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T12:46:00.386 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:46:00.383+0000 7fe48a830640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T12:46:00.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.492+0000 7fe96071d640 1 -- 172.21.15.57:0/3650682724 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe95813e520 msgr2=0x7fe95806bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:46:00.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.492+0000 7fe96071d640 1 --2- 172.21.15.57:0/3650682724 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe95813e520 0x7fe95806bb60 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe954009860 tx=0x7fe95402bd60 comp rx=0 tx=0).stop 2023-10-19T12:46:00.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.492+0000 7fe96071d640 1 -- 172.21.15.57:0/3650682724 shutdown_connections 2023-10-19T12:46:00.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.492+0000 7fe96071d640 1 --2- 172.21.15.57:0/3650682724 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe95813e520 0x7fe95806bb60 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:00.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.492+0000 7fe96071d640 1 -- 172.21.15.57:0/3650682724 >> 172.21.15.57:0/3650682724 conn(0x7fe958069930 msgr2=0x7fe958069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:46:00.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.492+0000 7fe96071d640 1 -- 172.21.15.57:0/3650682724 shutdown_connections 2023-10-19T12:46:00.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.492+0000 7fe96071d640 1 -- 172.21.15.57:0/3650682724 wait complete. 2023-10-19T12:46:00.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.494+0000 7fe96071d640 1 Processor -- start 2023-10-19T12:46:00.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.494+0000 7fe96071d640 1 -- start start 2023-10-19T12:46:00.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.495+0000 7fe96071d640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe958070af0 0x7fe9581e25a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:46:00.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.495+0000 7fe96071d640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe95813e520 0x7fe9581e1140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:46:00.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.495+0000 7fe96071d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe958141690 0x7fe95813b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:46:00.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.495+0000 7fe96071d640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9581da320 con 0x7fe958141690 2023-10-19T12:46:00.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.495+0000 7fe96071d640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9581e1680 con 0x7fe95813e520 2023-10-19T12:46:00.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.495+0000 7fe96071d640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe95813b7f0 con 0x7fe958070af0 2023-10-19T12:46:00.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.495+0000 7fe95ecba640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe958141690 0x7fe95813b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:46:00.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.495+0000 7fe95ecba640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe958141690 0x7fe95813b2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:36176/0 (socket says 172.21.15.57:36176) 2023-10-19T12:46:00.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.495+0000 7fe95ecba640 1 -- 172.21.15.57:0/1359455069 learned_addr learned my addr 172.21.15.57:0/1359455069 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:46:00.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.495+0000 7fe95dcb8640 1 --2- 172.21.15.57:0/1359455069 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe95813e520 0x7fe9581e1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:46:00.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.495+0000 7fe95e4b9640 1 --2- 172.21.15.57:0/1359455069 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe958070af0 0x7fe9581e25a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:46:00.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.495+0000 7fe95ecba640 1 -- 172.21.15.57:0/1359455069 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe958070af0 msgr2=0x7fe9581e25a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:46:00.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.495+0000 7fe95ecba640 1 --2- 172.21.15.57:0/1359455069 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe958070af0 0x7fe9581e25a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:00.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.495+0000 7fe95ecba640 1 -- 172.21.15.57:0/1359455069 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe95813e520 msgr2=0x7fe9581e1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:46:00.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.495+0000 7fe95ecba640 1 --2- 172.21.15.57:0/1359455069 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe95813e520 0x7fe9581e1140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:00.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.495+0000 7fe95ecba640 1 -- 172.21.15.57:0/1359455069 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe954009490 con 0x7fe958141690 2023-10-19T12:46:00.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.495+0000 7fe95e4b9640 1 --2- 172.21.15.57:0/1359455069 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe958070af0 0x7fe9581e25a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T12:46:00.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.495+0000 7fe95ecba640 1 --2- 172.21.15.57:0/1359455069 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe958141690 0x7fe95813b2b0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fe95000cbc0 tx=0x7fe95000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:46:00.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.495+0000 7fe94effd640 1 -- 172.21.15.57:0/1359455069 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe95000ad40 con 0x7fe958141690 2023-10-19T12:46:00.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.496+0000 7fe94effd640 1 -- 172.21.15.57:0/1359455069 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fe950012420 con 0x7fe958141690 2023-10-19T12:46:00.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.496+0000 7fe96071d640 1 -- 172.21.15.57:0/1359455069 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9581d9a30 con 0x7fe958141690 2023-10-19T12:46:00.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.496+0000 7fe94effd640 1 -- 172.21.15.57:0/1359455069 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe950012870 con 0x7fe958141690 2023-10-19T12:46:00.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.496+0000 7fe96071d640 1 -- 172.21.15.57:0/1359455069 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9581e2f80 con 0x7fe958141690 2023-10-19T12:46:00.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.497+0000 7fe96071d640 1 -- 172.21.15.57:0/1359455069 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9581da460 con 0x7fe958141690 2023-10-19T12:46:00.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.497+0000 7fe94effd640 1 -- 172.21.15.57:0/1359455069 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 9) v1 ==== 89349+0+0 (secure 0 0 0) 0x7fe95001a3b0 con 0x7fe958141690 2023-10-19T12:46:00.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.498+0000 7fe94effd640 1 --2- 172.21.15.57:0/1359455069 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fe93c06afe0 0x7fe93c06d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:46:00.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.498+0000 7fe94effd640 1 -- 172.21.15.57:0/1359455069 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7fe95008a0f0 con 0x7fe958141690 2023-10-19T12:46:00.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.498+0000 7fe95e4b9640 1 --2- 172.21.15.57:0/1359455069 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fe93c06afe0 0x7fe93c06d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:46:00.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.498+0000 7fe95e4b9640 1 --2- 172.21.15.57:0/1359455069 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fe93c06afe0 0x7fe93c06d4a0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe954036000 tx=0x7fe954004730 comp rx=0 tx=0).ready entity=mgr.5428 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:46:00.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.502+0000 7fe94effd640 1 -- 172.21.15.57:0/1359455069 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe950054e80 con 0x7fe958141690 2023-10-19T12:46:00.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.711+0000 7fe96071d640 1 -- 172.21.15.57:0/1359455069 --> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir7", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fe9580690e0 con 0x7fe93c06afe0 2023-10-19T12:46:00.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.798+0000 7fe94effd640 1 -- 172.21.15.57:0/1359455069 <== mgr.5428 v2:172.21.15.179:6832/922398470 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fe9580690e0 con 0x7fe93c06afe0 2023-10-19T12:46:00.802 INFO:teuthology.orchestra.run.smithi057.stdout:Schedule set for path /snap_test_dir1/dir7 2023-10-19T12:46:00.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.800+0000 7fe96071d640 1 -- 172.21.15.57:0/1359455069 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fe93c06afe0 msgr2=0x7fe93c06d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:46:00.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.800+0000 7fe96071d640 1 --2- 172.21.15.57:0/1359455069 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fe93c06afe0 0x7fe93c06d4a0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe954036000 tx=0x7fe954004730 comp rx=0 tx=0).stop 2023-10-19T12:46:00.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.800+0000 7fe96071d640 1 -- 172.21.15.57:0/1359455069 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe958141690 msgr2=0x7fe95813b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:46:00.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.800+0000 7fe96071d640 1 --2- 172.21.15.57:0/1359455069 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe958141690 0x7fe95813b2b0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fe95000cbc0 tx=0x7fe95000a430 comp rx=0 tx=0).stop 2023-10-19T12:46:00.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.800+0000 7fe96071d640 1 -- 172.21.15.57:0/1359455069 shutdown_connections 2023-10-19T12:46:00.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.801+0000 7fe96071d640 1 --2- 172.21.15.57:0/1359455069 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fe93c06afe0 0x7fe93c06d4a0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:00.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.801+0000 7fe96071d640 1 --2- 172.21.15.57:0/1359455069 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe958141690 0x7fe95813b2b0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:00.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.801+0000 7fe96071d640 1 --2- 172.21.15.57:0/1359455069 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe95813e520 0x7fe9581e1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:00.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.801+0000 7fe96071d640 1 --2- 172.21.15.57:0/1359455069 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe958070af0 0x7fe9581e25a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:00.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.801+0000 7fe96071d640 1 -- 172.21.15.57:0/1359455069 >> 172.21.15.57:0/1359455069 conn(0x7fe958069930 msgr2=0x7fe9580c9ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:46:00.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.801+0000 7fe96071d640 1 -- 172.21.15.57:0/1359455069 shutdown_connections 2023-10-19T12:46:00.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.801+0000 7fe96071d640 1 -- 172.21.15.57:0/1359455069 wait complete. 2023-10-19T12:46:00.814 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir7 2023-10-19T12:46:00.814 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:46:00.815 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir8) 2023-10-19T12:46:00.832 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir8 1m 2023-10-19T12:46:00.993 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.990+0000 7f72b74d7640 1 -- 172.21.15.57:0/3012848302 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f72b00afa20 msgr2=0x7f72b0158300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:46:00.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.990+0000 7f72b74d7640 1 --2- 172.21.15.57:0/3012848302 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f72b00afa20 0x7f72b0158300 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f72b00b10f0 tx=0x7f72a002be40 comp rx=0 tx=0).stop 2023-10-19T12:46:00.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.990+0000 7f72b74d7640 1 -- 172.21.15.57:0/3012848302 shutdown_connections 2023-10-19T12:46:00.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.990+0000 7f72b74d7640 1 --2- 172.21.15.57:0/3012848302 >> v2:172.21.15.179:3300/0 conn(0x7f72b00b0530 0x7f72b00b09a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:00.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.990+0000 7f72b74d7640 1 --2- 172.21.15.57:0/3012848302 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f72b00afa20 0x7f72b0158300 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:00.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.990+0000 7f72b74d7640 1 -- 172.21.15.57:0/3012848302 >> 172.21.15.57:0/3012848302 conn(0x7f72b0069930 msgr2=0x7f72b0069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:46:00.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.990+0000 7f72b74d7640 1 -- 172.21.15.57:0/3012848302 shutdown_connections 2023-10-19T12:46:00.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.990+0000 7f72b74d7640 1 -- 172.21.15.57:0/3012848302 wait complete. 2023-10-19T12:46:00.998 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.992+0000 7f72b74d7640 1 Processor -- start 2023-10-19T12:46:00.998 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.992+0000 7f72b74d7640 1 -- start start 2023-10-19T12:46:00.998 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.993+0000 7f72b74d7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f72b00afa20 0x7f72b015a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:46:00.998 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.993+0000 7f72b74d7640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f72b00b0530 0x7f72b013d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:46:00.998 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.993+0000 7f72b74d7640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f72b00cbd20 0x7f72b0153ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:46:00.999 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.993+0000 7f72b74d7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72b013e120 con 0x7f72b00afa20 2023-10-19T12:46:00.999 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.993+0000 7f72b74d7640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72b013e400 con 0x7f72b00cbd20 2023-10-19T12:46:00.999 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.993+0000 7f72b74d7640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72b013e6e0 con 0x7f72b00b0530 2023-10-19T12:46:00.999 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.993+0000 7f72b5273640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f72b00afa20 0x7f72b015a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:46:00.999 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.993+0000 7f72b5273640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f72b00afa20 0x7f72b015a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:36182/0 (socket says 172.21.15.57:36182) 2023-10-19T12:46:01.000 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.993+0000 7f72b5a74640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f72b00cbd20 0x7f72b0153ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:46:01.000 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.993+0000 7f72b5273640 1 -- 172.21.15.57:0/3105815680 learned_addr learned my addr 172.21.15.57:0/3105815680 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:46:01.000 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.993+0000 7f72b4a72640 1 --2- 172.21.15.57:0/3105815680 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f72b00b0530 0x7f72b013d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:46:01.000 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.993+0000 7f72b5273640 1 -- 172.21.15.57:0/3105815680 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f72b00b0530 msgr2=0x7f72b013d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:46:01.000 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.993+0000 7f72b5273640 1 --2- 172.21.15.57:0/3105815680 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f72b00b0530 0x7f72b013d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:01.001 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.993+0000 7f72b5273640 1 -- 172.21.15.57:0/3105815680 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f72b00cbd20 msgr2=0x7f72b0153ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:46:01.001 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.993+0000 7f72b5273640 1 --2- 172.21.15.57:0/3105815680 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f72b00cbd20 0x7f72b0153ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:01.001 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.993+0000 7f72b5273640 1 -- 172.21.15.57:0/3105815680 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72a0009560 con 0x7f72b00afa20 2023-10-19T12:46:01.001 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.993+0000 7f72b5a74640 1 --2- 172.21.15.57:0/3105815680 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f72b00cbd20 0x7f72b0153ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T12:46:01.001 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.993+0000 7f72b5273640 1 --2- 172.21.15.57:0/3105815680 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f72b00afa20 0x7f72b015a410 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f72ac00c400 tx=0x7f72ac00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:46:01.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.994+0000 7f72a5ffb640 1 -- 172.21.15.57:0/3105815680 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72ac0052d0 con 0x7f72b00afa20 2023-10-19T12:46:01.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.994+0000 7f72a5ffb640 1 -- 172.21.15.57:0/3105815680 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f72ac0035f0 con 0x7f72b00afa20 2023-10-19T12:46:01.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.994+0000 7f72b74d7640 1 -- 172.21.15.57:0/3105815680 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72b013e9c0 con 0x7f72b00afa20 2023-10-19T12:46:01.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.994+0000 7f72a5ffb640 1 -- 172.21.15.57:0/3105815680 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72ac003a00 con 0x7f72b00afa20 2023-10-19T12:46:01.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.994+0000 7f72b74d7640 1 -- 172.21.15.57:0/3105815680 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72b013ef80 con 0x7f72b00afa20 2023-10-19T12:46:01.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.995+0000 7f72b74d7640 1 -- 172.21.15.57:0/3105815680 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72b0067930 con 0x7f72b00afa20 2023-10-19T12:46:01.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.995+0000 7f72a5ffb640 1 -- 172.21.15.57:0/3105815680 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 10) v1 ==== 89405+0+0 (secure 0 0 0) 0x7f72ac003b60 con 0x7f72b00afa20 2023-10-19T12:46:01.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.996+0000 7f72a5ffb640 1 --2- 172.21.15.57:0/3105815680 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f729406b040 0x7f729406d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:46:01.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.996+0000 7f72a5ffb640 1 -- 172.21.15.57:0/3105815680 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7f72ac089290 con 0x7f72b00afa20 2023-10-19T12:46:01.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.996+0000 7f72b4a72640 1 --2- 172.21.15.57:0/3105815680 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f729406b040 0x7f729406d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:46:01.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:00.997+0000 7f72b4a72640 1 --2- 172.21.15.57:0/3105815680 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f729406b040 0x7f729406d500 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f72a0035000 tx=0x7f72a0004730 comp rx=0 tx=0).ready entity=mgr.5428 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:46:01.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.001+0000 7f72a5ffb640 1 -- 172.21.15.57:0/3105815680 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f72ac053fe0 con 0x7f72b00afa20 2023-10-19T12:46:01.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.202+0000 7f72b74d7640 1 -- 172.21.15.57:0/3105815680 --> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir8", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f72b00cc100 con 0x7f729406b040 2023-10-19T12:46:01.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.270+0000 7f72a5ffb640 1 -- 172.21.15.57:0/3105815680 <== mgr.5428 v2:172.21.15.179:6832/922398470 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f72a00055e0 con 0x7f729406b040 2023-10-19T12:46:01.273 INFO:teuthology.orchestra.run.smithi057.stdout:Schedule set for path /snap_test_dir1/dir8 2023-10-19T12:46:01.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.272+0000 7f72b74d7640 1 -- 172.21.15.57:0/3105815680 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f729406b040 msgr2=0x7f729406d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:46:01.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.272+0000 7f72b74d7640 1 --2- 172.21.15.57:0/3105815680 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f729406b040 0x7f729406d500 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f72a0035000 tx=0x7f72a0004730 comp rx=0 tx=0).stop 2023-10-19T12:46:01.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.272+0000 7f72b74d7640 1 -- 172.21.15.57:0/3105815680 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f72b00afa20 msgr2=0x7f72b015a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:46:01.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.272+0000 7f72b74d7640 1 --2- 172.21.15.57:0/3105815680 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f72b00afa20 0x7f72b015a410 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f72ac00c400 tx=0x7f72ac00c8c0 comp rx=0 tx=0).stop 2023-10-19T12:46:01.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.272+0000 7f72b74d7640 1 -- 172.21.15.57:0/3105815680 shutdown_connections 2023-10-19T12:46:01.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.272+0000 7f72b74d7640 1 --2- 172.21.15.57:0/3105815680 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f729406b040 0x7f729406d500 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:01.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.272+0000 7f72b74d7640 1 --2- 172.21.15.57:0/3105815680 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f72b00cbd20 0x7f72b0153ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:01.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.272+0000 7f72b74d7640 1 --2- 172.21.15.57:0/3105815680 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f72b00b0530 0x7f72b013d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:01.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.272+0000 7f72b74d7640 1 --2- 172.21.15.57:0/3105815680 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f72b00afa20 0x7f72b015a410 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:01.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.272+0000 7f72b74d7640 1 -- 172.21.15.57:0/3105815680 >> 172.21.15.57:0/3105815680 conn(0x7f72b0069930 msgr2=0x7f72b006b1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:46:01.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.272+0000 7f72b74d7640 1 -- 172.21.15.57:0/3105815680 shutdown_connections 2023-10-19T12:46:01.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.272+0000 7f72b74d7640 1 -- 172.21.15.57:0/3105815680 wait complete. 2023-10-19T12:46:01.285 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir8 2023-10-19T12:46:01.285 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:46:01.286 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir9) 2023-10-19T12:46:01.304 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir9 1m 2023-10-19T12:46:01.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.452+0000 7f28aa67c640 1 -- 172.21.15.57:0/3691301409 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f28a40ceb40 msgr2=0x7f28a40b0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:46:01.456 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.452+0000 7f28aa67c640 1 --2- 172.21.15.57:0/3691301409 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f28a40ceb40 0x7f28a40b0650 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f289c005fd0 tx=0x7f289c02c240 comp rx=0 tx=0).stop 2023-10-19T12:46:01.456 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.452+0000 7f28aa67c640 1 -- 172.21.15.57:0/3691301409 shutdown_connections 2023-10-19T12:46:01.456 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.452+0000 7f28aa67c640 1 --2- 172.21.15.57:0/3691301409 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f28a40ceb40 0x7f28a40b0650 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:01.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.452+0000 7f28aa67c640 1 --2- 172.21.15.57:0/3691301409 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f28a415b2a0 0x7f28a40d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:01.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.452+0000 7f28aa67c640 1 -- 172.21.15.57:0/3691301409 >> 172.21.15.57:0/3691301409 conn(0x7f28a40d0d60 msgr2=0x7f28a40d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:46:01.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.452+0000 7f28aa67c640 1 -- 172.21.15.57:0/3691301409 shutdown_connections 2023-10-19T12:46:01.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.452+0000 7f28aa67c640 1 -- 172.21.15.57:0/3691301409 wait complete. 2023-10-19T12:46:01.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.453+0000 7f28aa67c640 1 Processor -- start 2023-10-19T12:46:01.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.454+0000 7f28aa67c640 1 -- start start 2023-10-19T12:46:01.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.454+0000 7f28aa67c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f28a40ceb40 0x7f28a4158690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:46:01.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.454+0000 7f28aa67c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f28a40cf340 0x7f28a415d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:46:01.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.454+0000 7f28aa67c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f28a415b2a0 0x7f28a415dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:46:01.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.454+0000 7f28aa67c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28a415b800 con 0x7f28a40ceb40 2023-10-19T12:46:01.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.454+0000 7f28aa67c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28a41f2560 con 0x7f28a40cf340 2023-10-19T12:46:01.461 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.454+0000 7f28aa67c640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28a40c9d40 con 0x7f28a415b2a0 2023-10-19T12:46:01.461 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.455+0000 7f28a3fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f28a40ceb40 0x7f28a4158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:46:01.461 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.455+0000 7f28a3fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f28a40ceb40 0x7f28a4158690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:36196/0 (socket says 172.21.15.57:36196) 2023-10-19T12:46:01.461 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.455+0000 7f28a3fff640 1 -- 172.21.15.57:0/2527991701 learned_addr learned my addr 172.21.15.57:0/2527991701 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:46:01.461 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.455+0000 7f28a8c19640 1 --2- 172.21.15.57:0/2527991701 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f28a415b2a0 0x7f28a415dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:46:01.462 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.455+0000 7f28a37fe640 1 --2- 172.21.15.57:0/2527991701 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f28a40cf340 0x7f28a415d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:46:01.462 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.455+0000 7f28a3fff640 1 -- 172.21.15.57:0/2527991701 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f28a415b2a0 msgr2=0x7f28a415dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:46:01.462 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.455+0000 7f28a3fff640 1 --2- 172.21.15.57:0/2527991701 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f28a415b2a0 0x7f28a415dc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:01.462 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.455+0000 7f28a3fff640 1 -- 172.21.15.57:0/2527991701 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f28a40cf340 msgr2=0x7f28a415d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:46:01.462 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.455+0000 7f28a3fff640 1 --2- 172.21.15.57:0/2527991701 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f28a40cf340 0x7f28a415d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:01.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.455+0000 7f28a3fff640 1 -- 172.21.15.57:0/2527991701 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f289c009900 con 0x7f28a40ceb40 2023-10-19T12:46:01.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.455+0000 7f28a3fff640 1 --2- 172.21.15.57:0/2527991701 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f28a40ceb40 0x7f28a4158690 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f289400c450 tx=0x7f289400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:46:01.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.455+0000 7f28a0ff9640 1 -- 172.21.15.57:0/2527991701 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28940052d0 con 0x7f28a40ceb40 2023-10-19T12:46:01.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.455+0000 7f28a0ff9640 1 -- 172.21.15.57:0/2527991701 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f2894005430 con 0x7f28a40ceb40 2023-10-19T12:46:01.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.455+0000 7f28a0ff9640 1 -- 172.21.15.57:0/2527991701 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2894012890 con 0x7f28a40ceb40 2023-10-19T12:46:01.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.455+0000 7f28aa67c640 1 -- 172.21.15.57:0/2527991701 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28a40ca020 con 0x7f28a40ceb40 2023-10-19T12:46:01.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.455+0000 7f28aa67c640 1 -- 172.21.15.57:0/2527991701 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28a40ca300 con 0x7f28a40ceb40 2023-10-19T12:46:01.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.456+0000 7f28aa67c640 1 -- 172.21.15.57:0/2527991701 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28a40690e0 con 0x7f28a40ceb40 2023-10-19T12:46:01.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.457+0000 7f28a0ff9640 1 -- 172.21.15.57:0/2527991701 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 10) v1 ==== 89405+0+0 (secure 0 0 0) 0x7f28940035f0 con 0x7f28a40ceb40 2023-10-19T12:46:01.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.457+0000 7f28a0ff9640 1 --2- 172.21.15.57:0/2527991701 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f289006b040 0x7f289006d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:46:01.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.457+0000 7f28a0ff9640 1 -- 172.21.15.57:0/2527991701 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7f28940896a0 con 0x7f28a40ceb40 2023-10-19T12:46:01.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.457+0000 7f28a37fe640 1 --2- 172.21.15.57:0/2527991701 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f289006b040 0x7f289006d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:46:01.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.458+0000 7f28a37fe640 1 --2- 172.21.15.57:0/2527991701 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f289006b040 0x7f289006d500 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f28a40c76d0 tx=0x7f2898005f90 comp rx=0 tx=0).ready entity=mgr.5428 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:46:01.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.461+0000 7f28a0ff9640 1 -- 172.21.15.57:0/2527991701 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f28940543d0 con 0x7f28a40ceb40 2023-10-19T12:46:01.662 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.659+0000 7f28aa67c640 1 -- 172.21.15.57:0/2527991701 --> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir9", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f28a40cb980 con 0x7f289006b040 2023-10-19T12:46:01.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.756+0000 7f28a0ff9640 1 -- 172.21.15.57:0/2527991701 <== mgr.5428 v2:172.21.15.179:6832/922398470 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f28a40cb980 con 0x7f289006b040 2023-10-19T12:46:01.759 INFO:teuthology.orchestra.run.smithi057.stdout:Schedule set for path /snap_test_dir1/dir9 2023-10-19T12:46:01.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.758+0000 7f28aa67c640 1 -- 172.21.15.57:0/2527991701 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f289006b040 msgr2=0x7f289006d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:46:01.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.758+0000 7f28aa67c640 1 --2- 172.21.15.57:0/2527991701 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f289006b040 0x7f289006d500 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f28a40c76d0 tx=0x7f2898005f90 comp rx=0 tx=0).stop 2023-10-19T12:46:01.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.758+0000 7f28aa67c640 1 -- 172.21.15.57:0/2527991701 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f28a40ceb40 msgr2=0x7f28a4158690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:46:01.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.758+0000 7f28aa67c640 1 --2- 172.21.15.57:0/2527991701 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f28a40ceb40 0x7f28a4158690 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f289400c450 tx=0x7f289400c910 comp rx=0 tx=0).stop 2023-10-19T12:46:01.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.758+0000 7f28aa67c640 1 -- 172.21.15.57:0/2527991701 shutdown_connections 2023-10-19T12:46:01.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.758+0000 7f28aa67c640 1 --2- 172.21.15.57:0/2527991701 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f289006b040 0x7f289006d500 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:01.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.758+0000 7f28aa67c640 1 --2- 172.21.15.57:0/2527991701 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f28a415b2a0 0x7f28a415dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:01.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.758+0000 7f28aa67c640 1 --2- 172.21.15.57:0/2527991701 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f28a40cf340 0x7f28a415d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:01.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.758+0000 7f28aa67c640 1 --2- 172.21.15.57:0/2527991701 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f28a40ceb40 0x7f28a4158690 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:01.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.758+0000 7f28aa67c640 1 -- 172.21.15.57:0/2527991701 >> 172.21.15.57:0/2527991701 conn(0x7f28a40d0d60 msgr2=0x7f28a415a340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:46:01.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.758+0000 7f28aa67c640 1 -- 172.21.15.57:0/2527991701 shutdown_connections 2023-10-19T12:46:01.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.758+0000 7f28aa67c640 1 -- 172.21.15.57:0/2527991701 wait complete. 2023-10-19T12:46:01.771 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir9 2023-10-19T12:46:01.772 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir0 1m 2023-10-19T12:46:01.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.881+0000 7fc3ca133640 1 -- 172.21.15.57:0/2258343342 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc3ac003970 msgr2=0x7fc3ac003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:46:01.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.881+0000 7fc3ca133640 1 --2- 172.21.15.57:0/2258343342 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc3ac003970 0x7fc3ac003d90 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fc3b4001680 tx=0x7fc3ac0057f0 comp rx=0 tx=0).stop 2023-10-19T12:46:01.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.881+0000 7fc3ca133640 1 -- 172.21.15.57:0/2258343342 shutdown_connections 2023-10-19T12:46:01.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.881+0000 7fc3ca133640 1 --2- 172.21.15.57:0/2258343342 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc3ac003970 0x7fc3ac003d90 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:01.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.881+0000 7fc3ca133640 1 -- 172.21.15.57:0/2258343342 >> 172.21.15.57:0/2258343342 conn(0x7fc3c4069930 msgr2=0x7fc3c4069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:46:01.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.881+0000 7fc3ca133640 1 -- 172.21.15.57:0/2258343342 shutdown_connections 2023-10-19T12:46:01.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.881+0000 7fc3ca133640 1 -- 172.21.15.57:0/2258343342 wait complete. 2023-10-19T12:46:01.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.883+0000 7fc3ca133640 1 Processor -- start 2023-10-19T12:46:01.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.883+0000 7fc3ca133640 1 -- start start 2023-10-19T12:46:01.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.883+0000 7fc3ca133640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc3ac003970 0x7fc3c41e2610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:46:01.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.883+0000 7fc3ca133640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc3c414be80 0x7fc3c41e0c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:46:01.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.884+0000 7fc3ca133640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc3c4146750 0x7fc3c4146bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:46:01.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.884+0000 7fc3ca133640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3c41e1570 con 0x7fc3c4146750 2023-10-19T12:46:01.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.884+0000 7fc3ca133640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3c41e1850 con 0x7fc3c414be80 2023-10-19T12:46:01.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.884+0000 7fc3ca133640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3c41e1b30 con 0x7fc3ac003970 2023-10-19T12:46:01.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.884+0000 7fc3c3fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc3c4146750 0x7fc3c4146bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:46:01.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.884+0000 7fc3c3fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc3c4146750 0x7fc3c4146bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:36214/0 (socket says 172.21.15.57:36214) 2023-10-19T12:46:01.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.884+0000 7fc3c3fff640 1 -- 172.21.15.57:0/2153635967 learned_addr learned my addr 172.21.15.57:0/2153635967 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:46:01.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.884+0000 7fc3c37fe640 1 --2- 172.21.15.57:0/2153635967 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc3ac003970 0x7fc3c41e2610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:46:01.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.884+0000 7fc3c2ffd640 1 --2- 172.21.15.57:0/2153635967 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc3c414be80 0x7fc3c41e0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:46:01.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.884+0000 7fc3c3fff640 1 -- 172.21.15.57:0/2153635967 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc3ac003970 msgr2=0x7fc3c41e2610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:46:01.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.884+0000 7fc3c3fff640 1 --2- 172.21.15.57:0/2153635967 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc3ac003970 0x7fc3c41e2610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:01.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.884+0000 7fc3c3fff640 1 -- 172.21.15.57:0/2153635967 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc3c414be80 msgr2=0x7fc3c41e0c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:46:01.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.884+0000 7fc3c3fff640 1 --2- 172.21.15.57:0/2153635967 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc3c414be80 0x7fc3c41e0c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:01.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.884+0000 7fc3c3fff640 1 -- 172.21.15.57:0/2153635967 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3c41e1e10 con 0x7fc3c4146750 2023-10-19T12:46:01.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.884+0000 7fc3c3fff640 1 --2- 172.21.15.57:0/2153635967 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc3c4146750 0x7fc3c4146bd0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fc3c40ca740 tx=0x7fc3bc00d7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:46:01.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.884+0000 7fc3abfff640 1 -- 172.21.15.57:0/2153635967 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3bc013070 con 0x7fc3c4146750 2023-10-19T12:46:01.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.884+0000 7fc3abfff640 1 -- 172.21.15.57:0/2153635967 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fc3bc003b80 con 0x7fc3c4146750 2023-10-19T12:46:01.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.884+0000 7fc3ca133640 1 -- 172.21.15.57:0/2153635967 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3c41e20f0 con 0x7fc3c4146750 2023-10-19T12:46:01.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.884+0000 7fc3abfff640 1 -- 172.21.15.57:0/2153635967 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3bc004030 con 0x7fc3c4146750 2023-10-19T12:46:01.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.884+0000 7fc3ca133640 1 -- 172.21.15.57:0/2153635967 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3c41e11d0 con 0x7fc3c4146750 2023-10-19T12:46:01.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.886+0000 7fc3ca133640 1 -- 172.21.15.57:0/2153635967 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3c40690e0 con 0x7fc3c4146750 2023-10-19T12:46:01.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.886+0000 7fc3abfff640 1 -- 172.21.15.57:0/2153635967 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 10) v1 ==== 89405+0+0 (secure 0 0 0) 0x7fc3bc004190 con 0x7fc3c4146750 2023-10-19T12:46:01.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.886+0000 7fc3abfff640 1 --2- 172.21.15.57:0/2153635967 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fc3b006b070 0x7fc3b006d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:46:01.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.886+0000 7fc3c37fe640 1 --2- 172.21.15.57:0/2153635967 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fc3b006b070 0x7fc3b006d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:46:01.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.886+0000 7fc3abfff640 1 -- 172.21.15.57:0/2153635967 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7fc3bc08a5f0 con 0x7fc3c4146750 2023-10-19T12:46:01.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.887+0000 7fc3c37fe640 1 --2- 172.21.15.57:0/2153635967 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fc3b006b070 0x7fc3b006d530 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fc3b4034000 tx=0x7fc3b4001190 comp rx=0 tx=0).ready entity=mgr.5428 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:46:01.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:01.891+0000 7fc3abfff640 1 -- 172.21.15.57:0/2153635967 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc3bc0550d0 con 0x7fc3c4146750 2023-10-19T12:46:02.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.091+0000 7fc3ca133640 1 -- 172.21.15.57:0/2153635967 --> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir0", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fc3c406f700 con 0x7fc3b006b070 2023-10-19T12:46:02.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.154+0000 7fc3abfff640 1 -- 172.21.15.57:0/2153635967 <== mgr.5428 v2:172.21.15.179:6832/922398470 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fc3c406f700 con 0x7fc3b006b070 2023-10-19T12:46:02.158 INFO:teuthology.orchestra.run.smithi057.stdout:Schedule activated for path /snap_test_dir1/dir0 2023-10-19T12:46:02.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.157+0000 7fc3ca133640 1 -- 172.21.15.57:0/2153635967 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fc3b006b070 msgr2=0x7fc3b006d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:46:02.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.157+0000 7fc3ca133640 1 --2- 172.21.15.57:0/2153635967 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fc3b006b070 0x7fc3b006d530 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fc3b4034000 tx=0x7fc3b4001190 comp rx=0 tx=0).stop 2023-10-19T12:46:02.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.157+0000 7fc3ca133640 1 -- 172.21.15.57:0/2153635967 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc3c4146750 msgr2=0x7fc3c4146bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:46:02.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.157+0000 7fc3ca133640 1 --2- 172.21.15.57:0/2153635967 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc3c4146750 0x7fc3c4146bd0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fc3c40ca740 tx=0x7fc3bc00d7b0 comp rx=0 tx=0).stop 2023-10-19T12:46:02.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.157+0000 7fc3ca133640 1 -- 172.21.15.57:0/2153635967 shutdown_connections 2023-10-19T12:46:02.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.157+0000 7fc3ca133640 1 --2- 172.21.15.57:0/2153635967 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fc3b006b070 0x7fc3b006d530 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:02.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.157+0000 7fc3ca133640 1 --2- 172.21.15.57:0/2153635967 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc3c4146750 0x7fc3c4146bd0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:02.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.157+0000 7fc3ca133640 1 --2- 172.21.15.57:0/2153635967 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc3c414be80 0x7fc3c41e0c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:02.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.157+0000 7fc3ca133640 1 --2- 172.21.15.57:0/2153635967 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc3ac003970 0x7fc3c41e2610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:02.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.157+0000 7fc3ca133640 1 -- 172.21.15.57:0/2153635967 >> 172.21.15.57:0/2153635967 conn(0x7fc3c4069930 msgr2=0x7fc3c406c050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:46:02.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.157+0000 7fc3ca133640 1 -- 172.21.15.57:0/2153635967 shutdown_connections 2023-10-19T12:46:02.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.157+0000 7fc3ca133640 1 -- 172.21.15.57:0/2153635967 wait complete. 2023-10-19T12:46:02.170 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir0 2023-10-19T12:46:02.171 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir1 1m 2023-10-19T12:46:02.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.280+0000 7fd14a305640 1 -- 172.21.15.57:0/2149310516 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd12c003970 msgr2=0x7fd12c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:46:02.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.280+0000 7fd14a305640 1 --2- 172.21.15.57:0/2149310516 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd12c003970 0x7fd12c003d90 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fd13400c290 tx=0x7fd12c0057f0 comp rx=0 tx=0).stop 2023-10-19T12:46:02.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.280+0000 7fd1488a2640 1 -- 172.21.15.57:0/2149310516 reap_dead start 2023-10-19T12:46:02.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.280+0000 7fd14a305640 1 -- 172.21.15.57:0/2149310516 shutdown_connections 2023-10-19T12:46:02.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.280+0000 7fd14a305640 1 -- 172.21.15.57:0/2149310516 >> 172.21.15.57:0/2149310516 conn(0x7fd1440d0d60 msgr2=0x7fd1440d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:46:02.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.280+0000 7fd14a305640 1 -- 172.21.15.57:0/2149310516 shutdown_connections 2023-10-19T12:46:02.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.280+0000 7fd14a305640 1 -- 172.21.15.57:0/2149310516 wait complete. 2023-10-19T12:46:02.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.281+0000 7fd14a305640 1 Processor -- start 2023-10-19T12:46:02.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.281+0000 7fd14a305640 1 -- start start 2023-10-19T12:46:02.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.282+0000 7fd14a305640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd12c003970 0x7fd144145d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:46:02.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.282+0000 7fd14a305640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd144148700 0x7fd144148b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:46:02.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.282+0000 7fd14a305640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd144146830 0x7fd144146cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:46:02.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.282+0000 7fd14a305640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1441490c0 con 0x7fd144146830 2023-10-19T12:46:02.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.282+0000 7fd14a305640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd144148190 con 0x7fd12c003970 2023-10-19T12:46:02.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.282+0000 7fd14a305640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd144147580 con 0x7fd144148700 2023-10-19T12:46:02.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.282+0000 7fd1488a2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd144146830 0x7fd144146cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:46:02.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.282+0000 7fd1488a2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd144146830 0x7fd144146cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:36242/0 (socket says 172.21.15.57:36242) 2023-10-19T12:46:02.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.282+0000 7fd1488a2640 1 -- 172.21.15.57:0/3495447490 learned_addr learned my addr 172.21.15.57:0/3495447490 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:46:02.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.282+0000 7fd143fff640 1 --2- 172.21.15.57:0/3495447490 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd12c003970 0x7fd144145d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:46:02.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.282+0000 7fd1437fe640 1 --2- 172.21.15.57:0/3495447490 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd144148700 0x7fd144148b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:46:02.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.282+0000 7fd1488a2640 1 -- 172.21.15.57:0/3495447490 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd144148700 msgr2=0x7fd144148b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:46:02.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.282+0000 7fd1488a2640 1 --2- 172.21.15.57:0/3495447490 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd144148700 0x7fd144148b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:02.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.282+0000 7fd1488a2640 1 -- 172.21.15.57:0/3495447490 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd12c003970 msgr2=0x7fd144145d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:46:02.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.282+0000 7fd1488a2640 1 --2- 172.21.15.57:0/3495447490 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd12c003970 0x7fd144145d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:02.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.282+0000 7fd1488a2640 1 -- 172.21.15.57:0/3495447490 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd144147860 con 0x7fd144146830 2023-10-19T12:46:02.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.282+0000 7fd143fff640 1 --2- 172.21.15.57:0/3495447490 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd12c003970 0x7fd144145d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:46:02.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.283+0000 7fd1488a2640 1 --2- 172.21.15.57:0/3495447490 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd144146830 0x7fd144146cb0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fd13c005fd0 tx=0x7fd13c00f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:46:02.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.283+0000 7fd140ff9640 1 -- 172.21.15.57:0/3495447490 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd13c003d80 con 0x7fd144146830 2023-10-19T12:46:02.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.283+0000 7fd140ff9640 1 -- 172.21.15.57:0/3495447490 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fd13c004b20 con 0x7fd144146830 2023-10-19T12:46:02.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.283+0000 7fd14a305640 1 -- 172.21.15.57:0/3495447490 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd144147b40 con 0x7fd144146830 2023-10-19T12:46:02.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.283+0000 7fd140ff9640 1 -- 172.21.15.57:0/3495447490 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd13c004030 con 0x7fd144146830 2023-10-19T12:46:02.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.283+0000 7fd14a305640 1 -- 172.21.15.57:0/3495447490 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1441471f0 con 0x7fd144146830 2023-10-19T12:46:02.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.284+0000 7fd14a305640 1 -- 172.21.15.57:0/3495447490 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1440690e0 con 0x7fd144146830 2023-10-19T12:46:02.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.284+0000 7fd140ff9640 1 -- 172.21.15.57:0/3495447490 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 10) v1 ==== 89405+0+0 (secure 0 0 0) 0x7fd13c004190 con 0x7fd144146830 2023-10-19T12:46:02.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.284+0000 7fd140ff9640 1 --2- 172.21.15.57:0/3495447490 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fd13006b070 0x7fd13006d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:46:02.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.285+0000 7fd140ff9640 1 -- 172.21.15.57:0/3495447490 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7fd13c08aa50 con 0x7fd144146830 2023-10-19T12:46:02.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.285+0000 7fd143fff640 1 --2- 172.21.15.57:0/3495447490 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fd13006b070 0x7fd13006d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:46:02.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.285+0000 7fd143fff640 1 --2- 172.21.15.57:0/3495447490 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fd13006b070 0x7fd13006d530 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fd134034000 tx=0x7fd134001a70 comp rx=0 tx=0).ready entity=mgr.5428 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:46:02.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.289+0000 7fd140ff9640 1 -- 172.21.15.57:0/3495447490 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd13c0555c0 con 0x7fd144146830 2023-10-19T12:46:02.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.487+0000 7fd14a305640 1 -- 172.21.15.57:0/3495447490 --> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir1", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fd1440d1910 con 0x7fd13006b070 2023-10-19T12:46:02.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.533+0000 7fd140ff9640 1 -- 172.21.15.57:0/3495447490 <== mgr.5428 v2:172.21.15.179:6832/922398470 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fd134001550 con 0x7fd13006b070 2023-10-19T12:46:02.537 INFO:teuthology.orchestra.run.smithi057.stdout:Schedule activated for path /snap_test_dir1/dir1 2023-10-19T12:46:02.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.535+0000 7fd14a305640 1 -- 172.21.15.57:0/3495447490 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fd13006b070 msgr2=0x7fd13006d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:46:02.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.535+0000 7fd14a305640 1 --2- 172.21.15.57:0/3495447490 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fd13006b070 0x7fd13006d530 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fd134034000 tx=0x7fd134001a70 comp rx=0 tx=0).stop 2023-10-19T12:46:02.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.535+0000 7fd14a305640 1 -- 172.21.15.57:0/3495447490 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd144146830 msgr2=0x7fd144146cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:46:02.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.535+0000 7fd14a305640 1 --2- 172.21.15.57:0/3495447490 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd144146830 0x7fd144146cb0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fd13c005fd0 tx=0x7fd13c00f790 comp rx=0 tx=0).stop 2023-10-19T12:46:02.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.535+0000 7fd14a305640 1 -- 172.21.15.57:0/3495447490 shutdown_connections 2023-10-19T12:46:02.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.535+0000 7fd14a305640 1 --2- 172.21.15.57:0/3495447490 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fd13006b070 0x7fd13006d530 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:02.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.535+0000 7fd14a305640 1 --2- 172.21.15.57:0/3495447490 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd144146830 0x7fd144146cb0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:02.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.535+0000 7fd14a305640 1 --2- 172.21.15.57:0/3495447490 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd144148700 0x7fd144148b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:02.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.535+0000 7fd14a305640 1 --2- 172.21.15.57:0/3495447490 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd12c003970 0x7fd144145d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:02.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.535+0000 7fd14a305640 1 -- 172.21.15.57:0/3495447490 >> 172.21.15.57:0/3495447490 conn(0x7fd1440d0d60 msgr2=0x7fd1440aef70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:46:02.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.535+0000 7fd14a305640 1 -- 172.21.15.57:0/3495447490 shutdown_connections 2023-10-19T12:46:02.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.535+0000 7fd14a305640 1 -- 172.21.15.57:0/3495447490 wait complete. 2023-10-19T12:46:02.549 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir1 2023-10-19T12:46:02.550 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir2 1m 2023-10-19T12:46:02.664 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.659+0000 7f9c4d639640 1 -- 172.21.15.57:0/1510390927 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9c34004d00 msgr2=0x7f9c34005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:46:02.664 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.659+0000 7f9c4d639640 1 --2- 172.21.15.57:0/1510390927 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9c34004d00 0x7f9c34005180 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f9c4000c550 tx=0x7f9c340057f0 comp rx=0 tx=0).stop 2023-10-19T12:46:02.664 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.659+0000 7f9c477fe640 1 -- 172.21.15.57:0/1510390927 reap_dead start 2023-10-19T12:46:02.665 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.659+0000 7f9c4d639640 1 -- 172.21.15.57:0/1510390927 shutdown_connections 2023-10-19T12:46:02.665 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.659+0000 7f9c4d639640 1 -- 172.21.15.57:0/1510390927 >> 172.21.15.57:0/1510390927 conn(0x7f9c4806fae0 msgr2=0x7f9c4806fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:46:02.665 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.659+0000 7f9c4d639640 1 -- 172.21.15.57:0/1510390927 shutdown_connections 2023-10-19T12:46:02.665 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.659+0000 7f9c4d639640 1 -- 172.21.15.57:0/1510390927 wait complete. 2023-10-19T12:46:02.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.661+0000 7f9c4d639640 1 Processor -- start 2023-10-19T12:46:02.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.661+0000 7f9c4d639640 1 -- start start 2023-10-19T12:46:02.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.661+0000 7f9c4d639640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9c34004d00 0x7f9c480c3630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:46:02.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.661+0000 7f9c4d639640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9c48157670 0x7f9c480aab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:46:02.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.661+0000 7f9c4d639640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9c48157af0 0x7f9c48153530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:46:02.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.661+0000 7f9c4d639640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c480c2590 con 0x7f9c48157af0 2023-10-19T12:46:02.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.661+0000 7f9c4d639640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c480c2870 con 0x7f9c48157670 2023-10-19T12:46:02.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.661+0000 7f9c4d639640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c480c2b50 con 0x7f9c34004d00 2023-10-19T12:46:02.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.662+0000 7f9c477fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9c48157af0 0x7f9c48153530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:46:02.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.662+0000 7f9c477fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9c48157af0 0x7f9c48153530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:36266/0 (socket says 172.21.15.57:36266) 2023-10-19T12:46:02.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.662+0000 7f9c477fe640 1 -- 172.21.15.57:0/1031300446 learned_addr learned my addr 172.21.15.57:0/1031300446 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:46:02.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.662+0000 7f9c46ffd640 1 --2- 172.21.15.57:0/1031300446 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9c34004d00 0x7f9c480c3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:46:02.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.662+0000 7f9c3ffff640 1 --2- 172.21.15.57:0/1031300446 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9c48157670 0x7f9c480aab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:46:02.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.662+0000 7f9c477fe640 1 -- 172.21.15.57:0/1031300446 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9c34004d00 msgr2=0x7f9c480c3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:46:02.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.662+0000 7f9c477fe640 1 --2- 172.21.15.57:0/1031300446 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9c34004d00 0x7f9c480c3630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:02.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.662+0000 7f9c477fe640 1 -- 172.21.15.57:0/1031300446 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9c48157670 msgr2=0x7f9c480aab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:46:02.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.662+0000 7f9c477fe640 1 --2- 172.21.15.57:0/1031300446 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9c48157670 0x7f9c480aab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:02.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.662+0000 7f9c477fe640 1 -- 172.21.15.57:0/1031300446 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c480c2e30 con 0x7f9c48157af0 2023-10-19T12:46:02.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.662+0000 7f9c477fe640 1 --2- 172.21.15.57:0/1031300446 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9c48157af0 0x7f9c48153530 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f9c40001580 tx=0x7f9c40001230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:46:02.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.662+0000 7f9c3f7fe640 1 -- 172.21.15.57:0/1031300446 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c40003970 con 0x7f9c48157af0 2023-10-19T12:46:02.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.662+0000 7f9c3f7fe640 1 -- 172.21.15.57:0/1031300446 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f9c40003ad0 con 0x7f9c48157af0 2023-10-19T12:46:02.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.662+0000 7f9c4d639640 1 -- 172.21.15.57:0/1031300446 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c480c3110 con 0x7f9c48157af0 2023-10-19T12:46:02.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.662+0000 7f9c3f7fe640 1 -- 172.21.15.57:0/1031300446 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c400060e0 con 0x7f9c48157af0 2023-10-19T12:46:02.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.662+0000 7f9c4d639640 1 -- 172.21.15.57:0/1031300446 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c480ab070 con 0x7f9c48157af0 2023-10-19T12:46:02.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.663+0000 7f9c4d639640 1 -- 172.21.15.57:0/1031300446 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c480ab260 con 0x7f9c48157af0 2023-10-19T12:46:02.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.664+0000 7f9c3f7fe640 1 -- 172.21.15.57:0/1031300446 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 10) v1 ==== 89405+0+0 (secure 0 0 0) 0x7f9c40031030 con 0x7f9c48157af0 2023-10-19T12:46:02.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.664+0000 7f9c3f7fe640 1 --2- 172.21.15.57:0/1031300446 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f9c2806b040 0x7f9c2806d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:46:02.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.664+0000 7f9c3f7fe640 1 -- 172.21.15.57:0/1031300446 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7f9c400aa660 con 0x7f9c48157af0 2023-10-19T12:46:02.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.664+0000 7f9c46ffd640 1 --2- 172.21.15.57:0/1031300446 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f9c2806b040 0x7f9c2806d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:46:02.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.665+0000 7f9c46ffd640 1 --2- 172.21.15.57:0/1031300446 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f9c2806b040 0x7f9c2806d500 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f9c38003640 tx=0x7f9c38006b70 comp rx=0 tx=0).ready entity=mgr.5428 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:46:02.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.668+0000 7f9c3f7fe640 1 -- 172.21.15.57:0/1031300446 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9c40075200 con 0x7f9c48157af0 2023-10-19T12:46:02.868 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.865+0000 7f9c4d639640 1 -- 172.21.15.57:0/1031300446 --> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir2", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f9c4806b0a0 con 0x7f9c2806b040 2023-10-19T12:46:02.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.903+0000 7f9c3f7fe640 1 -- 172.21.15.57:0/1031300446 <== mgr.5428 v2:172.21.15.179:6832/922398470 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f9c38001630 con 0x7f9c2806b040 2023-10-19T12:46:02.907 INFO:teuthology.orchestra.run.smithi057.stdout:Schedule activated for path /snap_test_dir1/dir2 2023-10-19T12:46:02.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.905+0000 7f9c4d639640 1 -- 172.21.15.57:0/1031300446 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f9c2806b040 msgr2=0x7f9c2806d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:46:02.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.905+0000 7f9c4d639640 1 --2- 172.21.15.57:0/1031300446 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f9c2806b040 0x7f9c2806d500 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f9c38003640 tx=0x7f9c38006b70 comp rx=0 tx=0).stop 2023-10-19T12:46:02.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.905+0000 7f9c4d639640 1 -- 172.21.15.57:0/1031300446 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9c48157af0 msgr2=0x7f9c48153530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:46:02.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.905+0000 7f9c4d639640 1 --2- 172.21.15.57:0/1031300446 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9c48157af0 0x7f9c48153530 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f9c40001580 tx=0x7f9c40001230 comp rx=0 tx=0).stop 2023-10-19T12:46:02.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.905+0000 7f9c4d639640 1 -- 172.21.15.57:0/1031300446 shutdown_connections 2023-10-19T12:46:02.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.905+0000 7f9c4d639640 1 --2- 172.21.15.57:0/1031300446 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f9c2806b040 0x7f9c2806d500 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:02.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.905+0000 7f9c4d639640 1 --2- 172.21.15.57:0/1031300446 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9c48157af0 0x7f9c48153530 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:02.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.905+0000 7f9c4d639640 1 --2- 172.21.15.57:0/1031300446 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9c48157670 0x7f9c480aab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:02.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.905+0000 7f9c4d639640 1 --2- 172.21.15.57:0/1031300446 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9c34004d00 0x7f9c480c3630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:02.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.905+0000 7f9c4d639640 1 -- 172.21.15.57:0/1031300446 >> 172.21.15.57:0/1031300446 conn(0x7f9c4806fae0 msgr2=0x7f9c48156080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:46:02.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.905+0000 7f9c4d639640 1 -- 172.21.15.57:0/1031300446 shutdown_connections 2023-10-19T12:46:02.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:02.906+0000 7f9c4d639640 1 -- 172.21.15.57:0/1031300446 wait complete. 2023-10-19T12:46:02.919 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir2 2023-10-19T12:46:02.920 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir3 1m 2023-10-19T12:46:03.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.031+0000 7fb6f776e640 1 -- 172.21.15.57:0/2360567360 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb6f00cf340 msgr2=0x7fb6f00d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:46:03.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.031+0000 7fb6f776e640 1 --2- 172.21.15.57:0/2360567360 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb6f00cf340 0x7fb6f00d1910 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fb6e8009830 tx=0x7fb6e802bcd0 comp rx=0 tx=0).stop 2023-10-19T12:46:03.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.031+0000 7fb6f776e640 1 -- 172.21.15.57:0/2360567360 shutdown_connections 2023-10-19T12:46:03.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.031+0000 7fb6f776e640 1 --2- 172.21.15.57:0/2360567360 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb6f015b2a0 0x7fb6f00cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:03.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.031+0000 7fb6f776e640 1 --2- 172.21.15.57:0/2360567360 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb6f00cf340 0x7fb6f00d1910 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:03.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.031+0000 7fb6f776e640 1 -- 172.21.15.57:0/2360567360 >> 172.21.15.57:0/2360567360 conn(0x7fb6f00d0d60 msgr2=0x7fb6f00d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:46:03.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.032+0000 7fb6f776e640 1 -- 172.21.15.57:0/2360567360 shutdown_connections 2023-10-19T12:46:03.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.032+0000 7fb6f776e640 1 -- 172.21.15.57:0/2360567360 wait complete. 2023-10-19T12:46:03.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.033+0000 7fb6f776e640 1 Processor -- start 2023-10-19T12:46:03.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.033+0000 7fb6f776e640 1 -- start start 2023-10-19T12:46:03.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.033+0000 7fb6f776e640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb6f00ceb40 0x7fb6f0158560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:46:03.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.033+0000 7fb6f776e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb6f00cf340 0x7fb6f00b42f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:46:03.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.033+0000 7fb6f776e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb6f015b2a0 0x7fb6f0156e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:46:03.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.033+0000 7fb6f776e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6f0158140 con 0x7fb6f015b2a0 2023-10-19T12:46:03.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.034+0000 7fb6f776e640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6f00c7fd0 con 0x7fb6f00cf340 2023-10-19T12:46:03.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.034+0000 7fb6f776e640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6f01577a0 con 0x7fb6f00ceb40 2023-10-19T12:46:03.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.034+0000 7fb6f5d0b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb6f015b2a0 0x7fb6f0156e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:46:03.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.034+0000 7fb6f4d09640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb6f00cf340 0x7fb6f00b42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:46:03.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.034+0000 7fb6f550a640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb6f00ceb40 0x7fb6f0158560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:46:03.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.034+0000 7fb6f4d09640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb6f00cf340 0x7fb6f00b42f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.57:49596/0 (socket says 172.21.15.57:49596) 2023-10-19T12:46:03.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.034+0000 7fb6f550a640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb6f00ceb40 0x7fb6f0158560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3301/0 says I am v2:172.21.15.57:46164/0 (socket says 172.21.15.57:46164) 2023-10-19T12:46:03.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.034+0000 7fb6f4d09640 1 -- 172.21.15.57:0/2848941434 learned_addr learned my addr 172.21.15.57:0/2848941434 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:46:03.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.034+0000 7fb6f5d0b640 1 -- 172.21.15.57:0/2848941434 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb6f00ceb40 msgr2=0x7fb6f0158560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:46:03.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.034+0000 7fb6f5d0b640 1 --2- 172.21.15.57:0/2848941434 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb6f00ceb40 0x7fb6f0158560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:03.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.034+0000 7fb6f5d0b640 1 -- 172.21.15.57:0/2848941434 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb6f00cf340 msgr2=0x7fb6f00b42f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:46:03.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.034+0000 7fb6f5d0b640 1 --2- 172.21.15.57:0/2848941434 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb6f00cf340 0x7fb6f00b42f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:03.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.034+0000 7fb6f5d0b640 1 -- 172.21.15.57:0/2848941434 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6e4009560 con 0x7fb6f015b2a0 2023-10-19T12:46:03.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.034+0000 7fb6f5d0b640 1 --2- 172.21.15.57:0/2848941434 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb6f015b2a0 0x7fb6f0156e10 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fb6e000c760 tx=0x7fb6e000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:46:03.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.034+0000 7fb6edffb640 1 -- 172.21.15.57:0/2848941434 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6e0006d50 con 0x7fb6f015b2a0 2023-10-19T12:46:03.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.034+0000 7fb6f776e640 1 -- 172.21.15.57:0/2848941434 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6e8009490 con 0x7fb6f015b2a0 2023-10-19T12:46:03.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.034+0000 7fb6edffb640 1 -- 172.21.15.57:0/2848941434 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fb6e000ce10 con 0x7fb6f015b2a0 2023-10-19T12:46:03.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.034+0000 7fb6edffb640 1 -- 172.21.15.57:0/2848941434 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6e0003bd0 con 0x7fb6f015b2a0 2023-10-19T12:46:03.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.035+0000 7fb6f776e640 1 -- 172.21.15.57:0/2848941434 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6f0157d60 con 0x7fb6f015b2a0 2023-10-19T12:46:03.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.036+0000 7fb6f776e640 1 -- 172.21.15.57:0/2848941434 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6f015a960 con 0x7fb6f015b2a0 2023-10-19T12:46:03.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.036+0000 7fb6edffb640 1 -- 172.21.15.57:0/2848941434 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 11) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fb6e0003d30 con 0x7fb6f015b2a0 2023-10-19T12:46:03.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.036+0000 7fb6edffb640 1 --2- 172.21.15.57:0/2848941434 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fb6dc06b0d0 0x7fb6dc06d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:46:03.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.037+0000 7fb6edffb640 1 -- 172.21.15.57:0/2848941434 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7fb6e0089970 con 0x7fb6f015b2a0 2023-10-19T12:46:03.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.037+0000 7fb6f550a640 1 --2- 172.21.15.57:0/2848941434 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fb6dc06b0d0 0x7fb6dc06d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:46:03.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.037+0000 7fb6f550a640 1 --2- 172.21.15.57:0/2848941434 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fb6dc06b0d0 0x7fb6dc06d590 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb6e8036000 tx=0x7fb6e8004730 comp rx=0 tx=0).ready entity=mgr.5428 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:46:03.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.041+0000 7fb6edffb640 1 -- 172.21.15.57:0/2848941434 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb6e0054540 con 0x7fb6f015b2a0 2023-10-19T12:46:03.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.239+0000 7fb6f776e640 1 -- 172.21.15.57:0/2848941434 --> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir3", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fb6f00cc170 con 0x7fb6dc06b0d0 2023-10-19T12:46:03.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.288+0000 7fb6edffb640 1 -- 172.21.15.57:0/2848941434 <== mgr.5428 v2:172.21.15.179:6832/922398470 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fb6e8004140 con 0x7fb6dc06b0d0 2023-10-19T12:46:03.292 INFO:teuthology.orchestra.run.smithi057.stdout:Schedule activated for path /snap_test_dir1/dir3 2023-10-19T12:46:03.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.290+0000 7fb6f776e640 1 -- 172.21.15.57:0/2848941434 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fb6dc06b0d0 msgr2=0x7fb6dc06d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:46:03.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.290+0000 7fb6f776e640 1 --2- 172.21.15.57:0/2848941434 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fb6dc06b0d0 0x7fb6dc06d590 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb6e8036000 tx=0x7fb6e8004730 comp rx=0 tx=0).stop 2023-10-19T12:46:03.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.290+0000 7fb6f776e640 1 -- 172.21.15.57:0/2848941434 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb6f015b2a0 msgr2=0x7fb6f0156e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:46:03.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.290+0000 7fb6f776e640 1 --2- 172.21.15.57:0/2848941434 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb6f015b2a0 0x7fb6f0156e10 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fb6e000c760 tx=0x7fb6e000cc20 comp rx=0 tx=0).stop 2023-10-19T12:46:03.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.290+0000 7fb6f776e640 1 -- 172.21.15.57:0/2848941434 shutdown_connections 2023-10-19T12:46:03.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.290+0000 7fb6f776e640 1 --2- 172.21.15.57:0/2848941434 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fb6dc06b0d0 0x7fb6dc06d590 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:03.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.290+0000 7fb6f776e640 1 --2- 172.21.15.57:0/2848941434 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb6f015b2a0 0x7fb6f0156e10 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:03.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.290+0000 7fb6f776e640 1 --2- 172.21.15.57:0/2848941434 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb6f00cf340 0x7fb6f00b42f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:03.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.290+0000 7fb6f776e640 1 --2- 172.21.15.57:0/2848941434 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb6f00ceb40 0x7fb6f0158560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:03.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.290+0000 7fb6f776e640 1 -- 172.21.15.57:0/2848941434 >> 172.21.15.57:0/2848941434 conn(0x7fb6f00d0d60 msgr2=0x7fb6f006b610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:46:03.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.291+0000 7fb6f776e640 1 -- 172.21.15.57:0/2848941434 shutdown_connections 2023-10-19T12:46:03.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.291+0000 7fb6f776e640 1 -- 172.21.15.57:0/2848941434 wait complete. 2023-10-19T12:46:03.304 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir3 2023-10-19T12:46:03.304 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir4 1m 2023-10-19T12:46:03.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.414+0000 7f7558325640 1 -- 172.21.15.57:0/4048685696 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7550142850 msgr2=0x7f7550159440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:46:03.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.414+0000 7f7558325640 1 --2- 172.21.15.57:0/4048685696 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7550142850 0x7f7550159440 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f7550072c60 tx=0x7f754c02bd10 comp rx=0 tx=0).stop 2023-10-19T12:46:03.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.414+0000 7f7558325640 1 -- 172.21.15.57:0/4048685696 shutdown_connections 2023-10-19T12:46:03.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.414+0000 7f7558325640 1 --2- 172.21.15.57:0/4048685696 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7550069720 0x7f7550069bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:03.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.414+0000 7f7558325640 1 --2- 172.21.15.57:0/4048685696 >> v2:172.21.15.179:3300/0 conn(0x7f75500cd6d0 0x7f75500cdb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:03.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.414+0000 7f7558325640 1 --2- 172.21.15.57:0/4048685696 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7550142850 0x7f7550159440 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:03.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.414+0000 7f7558325640 1 -- 172.21.15.57:0/4048685696 >> 172.21.15.57:0/4048685696 conn(0x7f75500af7a0 msgr2=0x7f75500afb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:46:03.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.414+0000 7f7558325640 1 -- 172.21.15.57:0/4048685696 shutdown_connections 2023-10-19T12:46:03.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.414+0000 7f7558325640 1 -- 172.21.15.57:0/4048685696 wait complete. 2023-10-19T12:46:03.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.415+0000 7f7558325640 1 Processor -- start 2023-10-19T12:46:03.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.415+0000 7f7558325640 1 -- start start 2023-10-19T12:46:03.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.416+0000 7f7558325640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7550069720 0x7f75501589b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:46:03.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.416+0000 7f7558325640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f75500cd6d0 0x7f755015be50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:46:03.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.416+0000 7f7558325640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7550142850 0x7f75501e3780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:46:03.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.416+0000 7f7558325640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7550158ef0 con 0x7f7550069720 2023-10-19T12:46:03.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.416+0000 7f7558325640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7550139f80 con 0x7f75500cd6d0 2023-10-19T12:46:03.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.416+0000 7f7558325640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f755013a260 con 0x7f7550142850 2023-10-19T12:46:03.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.416+0000 7f75560c1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7550069720 0x7f75501589b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:46:03.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.416+0000 7f75560c1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7550069720 0x7f75501589b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:36282/0 (socket says 172.21.15.57:36282) 2023-10-19T12:46:03.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.416+0000 7f75560c1640 1 -- 172.21.15.57:0/984454866 learned_addr learned my addr 172.21.15.57:0/984454866 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:46:03.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.416+0000 7f75558c0640 1 --2- 172.21.15.57:0/984454866 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f75500cd6d0 0x7f755015be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:46:03.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.416+0000 7f75568c2640 1 --2- 172.21.15.57:0/984454866 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7550142850 0x7f75501e3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:46:03.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.416+0000 7f75560c1640 1 -- 172.21.15.57:0/984454866 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7550142850 msgr2=0x7f75501e3780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:46:03.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.416+0000 7f75560c1640 1 --2- 172.21.15.57:0/984454866 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7550142850 0x7f75501e3780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:03.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.416+0000 7f75560c1640 1 -- 172.21.15.57:0/984454866 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f75500cd6d0 msgr2=0x7f755015be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:46:03.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.416+0000 7f75560c1640 1 --2- 172.21.15.57:0/984454866 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f75500cd6d0 0x7f755015be50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:03.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.416+0000 7f75560c1640 1 -- 172.21.15.57:0/984454866 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f754c009560 con 0x7f7550069720 2023-10-19T12:46:03.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.416+0000 7f75560c1640 1 --2- 172.21.15.57:0/984454866 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7550069720 0x7f75501589b0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f754c009850 tx=0x7f754c02d9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:46:03.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.416+0000 7f7546ffd640 1 -- 172.21.15.57:0/984454866 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f754c02dde0 con 0x7f7550069720 2023-10-19T12:46:03.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.417+0000 7f7546ffd640 1 -- 172.21.15.57:0/984454866 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f754c0036e0 con 0x7f7550069720 2023-10-19T12:46:03.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.417+0000 7f7558325640 1 -- 172.21.15.57:0/984454866 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7550159c30 con 0x7f7550069720 2023-10-19T12:46:03.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.417+0000 7f7546ffd640 1 -- 172.21.15.57:0/984454866 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f754c003ab0 con 0x7f7550069720 2023-10-19T12:46:03.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.417+0000 7f7558325640 1 -- 172.21.15.57:0/984454866 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f755015a1f0 con 0x7f7550069720 2023-10-19T12:46:03.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.418+0000 7f7558325640 1 -- 172.21.15.57:0/984454866 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f755006a5f0 con 0x7f7550069720 2023-10-19T12:46:03.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.418+0000 7f7546ffd640 1 -- 172.21.15.57:0/984454866 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 11) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f754c003c10 con 0x7f7550069720 2023-10-19T12:46:03.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.419+0000 7f7546ffd640 1 --2- 172.21.15.57:0/984454866 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f753406b0d0 0x7f753406d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:46:03.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.419+0000 7f7546ffd640 1 -- 172.21.15.57:0/984454866 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7f754c0ac260 con 0x7f7550069720 2023-10-19T12:46:03.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.419+0000 7f75558c0640 1 --2- 172.21.15.57:0/984454866 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f753406b0d0 0x7f753406d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:46:03.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.419+0000 7f75558c0640 1 --2- 172.21.15.57:0/984454866 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f753406b0d0 0x7f753406d590 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f755013f060 tx=0x7f7540005f90 comp rx=0 tx=0).ready entity=mgr.5428 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:46:03.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.423+0000 7f7546ffd640 1 -- 172.21.15.57:0/984454866 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f754c076e20 con 0x7f7550069720 2023-10-19T12:46:03.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.620+0000 7f7558325640 1 -- 172.21.15.57:0/984454866 --> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir4", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f75500b1e70 con 0x7f753406b0d0 2023-10-19T12:46:03.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.673+0000 7f7546ffd640 1 -- 172.21.15.57:0/984454866 <== mgr.5428 v2:172.21.15.179:6832/922398470 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f75400098c0 con 0x7f753406b0d0 2023-10-19T12:46:03.678 INFO:teuthology.orchestra.run.smithi057.stdout:Schedule activated for path /snap_test_dir1/dir4 2023-10-19T12:46:03.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.675+0000 7f7558325640 1 -- 172.21.15.57:0/984454866 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f753406b0d0 msgr2=0x7f753406d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:46:03.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.675+0000 7f7558325640 1 --2- 172.21.15.57:0/984454866 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f753406b0d0 0x7f753406d590 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f755013f060 tx=0x7f7540005f90 comp rx=0 tx=0).stop 2023-10-19T12:46:03.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.675+0000 7f7558325640 1 -- 172.21.15.57:0/984454866 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7550069720 msgr2=0x7f75501589b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:46:03.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.675+0000 7f7558325640 1 --2- 172.21.15.57:0/984454866 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7550069720 0x7f75501589b0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f754c009850 tx=0x7f754c02d9e0 comp rx=0 tx=0).stop 2023-10-19T12:46:03.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.675+0000 7f7558325640 1 -- 172.21.15.57:0/984454866 shutdown_connections 2023-10-19T12:46:03.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.675+0000 7f7558325640 1 --2- 172.21.15.57:0/984454866 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f753406b0d0 0x7f753406d590 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:03.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.675+0000 7f7558325640 1 --2- 172.21.15.57:0/984454866 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7550142850 0x7f75501e3780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:03.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.675+0000 7f7558325640 1 --2- 172.21.15.57:0/984454866 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f75500cd6d0 0x7f755015be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:03.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.675+0000 7f7558325640 1 --2- 172.21.15.57:0/984454866 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7550069720 0x7f75501589b0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:03.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.675+0000 7f7558325640 1 -- 172.21.15.57:0/984454866 >> 172.21.15.57:0/984454866 conn(0x7f75500af7a0 msgr2=0x7f75500af010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:46:03.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.675+0000 7f7558325640 1 -- 172.21.15.57:0/984454866 shutdown_connections 2023-10-19T12:46:03.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.675+0000 7f7558325640 1 -- 172.21.15.57:0/984454866 wait complete. 2023-10-19T12:46:03.689 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir4 2023-10-19T12:46:03.690 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir5 1m 2023-10-19T12:46:03.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.798+0000 7f39a33f5640 1 -- 172.21.15.57:0/3833200505 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f399c0ceb40 msgr2=0x7f399c0b0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:46:03.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.798+0000 7f39a33f5640 1 --2- 172.21.15.57:0/3833200505 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f399c0ceb40 0x7f399c0b0650 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f399c08a810 tx=0x7f399402c1d0 comp rx=0 tx=0).stop 2023-10-19T12:46:03.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.798+0000 7f39a33f5640 1 -- 172.21.15.57:0/3833200505 shutdown_connections 2023-10-19T12:46:03.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.798+0000 7f39a33f5640 1 --2- 172.21.15.57:0/3833200505 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f399c0ceb40 0x7f399c0b0650 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:03.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.798+0000 7f39a33f5640 1 --2- 172.21.15.57:0/3833200505 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f399c15b2a0 0x7f399c0cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:03.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.798+0000 7f39a33f5640 1 --2- 172.21.15.57:0/3833200505 >> v2:172.21.15.179:3300/0 conn(0x7f399c0cf340 0x7f399c0d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:03.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.798+0000 7f39a33f5640 1 -- 172.21.15.57:0/3833200505 >> 172.21.15.57:0/3833200505 conn(0x7f399c0d0d60 msgr2=0x7f399c0d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:46:03.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.798+0000 7f39a33f5640 1 -- 172.21.15.57:0/3833200505 shutdown_connections 2023-10-19T12:46:03.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.798+0000 7f39a33f5640 1 -- 172.21.15.57:0/3833200505 wait complete. 2023-10-19T12:46:03.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.800+0000 7f39a33f5640 1 Processor -- start 2023-10-19T12:46:03.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.800+0000 7f39a33f5640 1 -- start start 2023-10-19T12:46:03.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.800+0000 7f39a33f5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f399c0ceb40 0x7f399c0c7e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:46:03.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.800+0000 7f39a33f5640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f399c0cf340 0x7f399c15def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:46:03.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.800+0000 7f39a33f5640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f399c15b2a0 0x7f399c14ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:46:03.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.800+0000 7f39a33f5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f399c0c92e0 con 0x7f399c0ceb40 2023-10-19T12:46:03.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.800+0000 7f39a33f5640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f399c0c8660 con 0x7f399c0cf340 2023-10-19T12:46:03.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.800+0000 7f39a33f5640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f399c0c8940 con 0x7f399c15b2a0 2023-10-19T12:46:03.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.800+0000 7f39a1191640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f399c0ceb40 0x7f399c0c7e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:46:03.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.800+0000 7f39a1191640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f399c0ceb40 0x7f399c0c7e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:36300/0 (socket says 172.21.15.57:36300) 2023-10-19T12:46:03.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.800+0000 7f39a1191640 1 -- 172.21.15.57:0/3331660281 learned_addr learned my addr 172.21.15.57:0/3331660281 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:46:03.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.801+0000 7f39a0990640 1 --2- 172.21.15.57:0/3331660281 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f399c0cf340 0x7f399c15def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:46:03.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.801+0000 7f39a1992640 1 --2- 172.21.15.57:0/3331660281 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f399c15b2a0 0x7f399c14ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:46:03.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.801+0000 7f39a1191640 1 -- 172.21.15.57:0/3331660281 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f399c15b2a0 msgr2=0x7f399c14ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:46:03.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.801+0000 7f39a1191640 1 --2- 172.21.15.57:0/3331660281 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f399c15b2a0 0x7f399c14ad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:03.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.801+0000 7f39a1191640 1 -- 172.21.15.57:0/3331660281 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f399c0cf340 msgr2=0x7f399c15def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:46:03.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.801+0000 7f39a1191640 1 --2- 172.21.15.57:0/3331660281 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f399c0cf340 0x7f399c15def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:03.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.801+0000 7f39a1191640 1 -- 172.21.15.57:0/3331660281 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3994009900 con 0x7f399c0ceb40 2023-10-19T12:46:03.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.801+0000 7f39a1992640 1 --2- 172.21.15.57:0/3331660281 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f399c15b2a0 0x7f399c14ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:46:03.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.801+0000 7f39a1191640 1 --2- 172.21.15.57:0/3331660281 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f399c0ceb40 0x7f399c0c7e30 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f399800e9f0 tx=0x7f399800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:46:03.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.801+0000 7f3991ffb640 1 -- 172.21.15.57:0/3331660281 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f399800ccc0 con 0x7f399c0ceb40 2023-10-19T12:46:03.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.801+0000 7f3991ffb640 1 -- 172.21.15.57:0/3331660281 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f3998010430 con 0x7f399c0ceb40 2023-10-19T12:46:03.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.801+0000 7f39a33f5640 1 -- 172.21.15.57:0/3331660281 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f399c0c8c20 con 0x7f399c0ceb40 2023-10-19T12:46:03.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.801+0000 7f3991ffb640 1 -- 172.21.15.57:0/3331660281 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3998010800 con 0x7f399c0ceb40 2023-10-19T12:46:03.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.801+0000 7f39a33f5640 1 -- 172.21.15.57:0/3331660281 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f399c0c8370 con 0x7f399c0ceb40 2023-10-19T12:46:03.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.802+0000 7f39a33f5640 1 -- 172.21.15.57:0/3331660281 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f399c0c9550 con 0x7f399c0ceb40 2023-10-19T12:46:03.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.803+0000 7f3991ffb640 1 -- 172.21.15.57:0/3331660281 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 11) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f3998010960 con 0x7f399c0ceb40 2023-10-19T12:46:03.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.803+0000 7f3991ffb640 1 --2- 172.21.15.57:0/3331660281 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f398006b0d0 0x7f398006d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:46:03.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.803+0000 7f3991ffb640 1 -- 172.21.15.57:0/3331660281 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7f3998014070 con 0x7f399c0ceb40 2023-10-19T12:46:03.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.803+0000 7f39a0990640 1 --2- 172.21.15.57:0/3331660281 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f398006b0d0 0x7f398006d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:46:03.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.804+0000 7f39a0990640 1 --2- 172.21.15.57:0/3331660281 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f398006b0d0 0x7f398006d590 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f399c0b58f0 tx=0x7f398c005f90 comp rx=0 tx=0).ready entity=mgr.5428 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:46:03.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:03.807+0000 7f3991ffb640 1 -- 172.21.15.57:0/3331660281 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3998051ca0 con 0x7f399c0ceb40 2023-10-19T12:46:04.010 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.007+0000 7f39a33f5640 1 -- 172.21.15.57:0/3331660281 --> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir5", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f399c0cc170 con 0x7f398006b0d0 2023-10-19T12:46:04.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.047+0000 7f3991ffb640 1 -- 172.21.15.57:0/3331660281 <== mgr.5428 v2:172.21.15.179:6832/922398470 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f398c016070 con 0x7f398006b0d0 2023-10-19T12:46:04.050 INFO:teuthology.orchestra.run.smithi057.stdout:Schedule activated for path /snap_test_dir1/dir5 2023-10-19T12:46:04.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.049+0000 7f39a33f5640 1 -- 172.21.15.57:0/3331660281 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f398006b0d0 msgr2=0x7f398006d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:46:04.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.049+0000 7f39a33f5640 1 --2- 172.21.15.57:0/3331660281 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f398006b0d0 0x7f398006d590 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f399c0b58f0 tx=0x7f398c005f90 comp rx=0 tx=0).stop 2023-10-19T12:46:04.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.049+0000 7f39a33f5640 1 -- 172.21.15.57:0/3331660281 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f399c0ceb40 msgr2=0x7f399c0c7e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:46:04.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.049+0000 7f39a33f5640 1 --2- 172.21.15.57:0/3331660281 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f399c0ceb40 0x7f399c0c7e30 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f399800e9f0 tx=0x7f399800eeb0 comp rx=0 tx=0).stop 2023-10-19T12:46:04.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.049+0000 7f39a33f5640 1 -- 172.21.15.57:0/3331660281 shutdown_connections 2023-10-19T12:46:04.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.049+0000 7f39a33f5640 1 --2- 172.21.15.57:0/3331660281 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f398006b0d0 0x7f398006d590 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:04.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.049+0000 7f39a33f5640 1 --2- 172.21.15.57:0/3331660281 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f399c15b2a0 0x7f399c14ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:04.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.049+0000 7f39a33f5640 1 --2- 172.21.15.57:0/3331660281 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f399c0cf340 0x7f399c15def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:04.054 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.049+0000 7f39a33f5640 1 --2- 172.21.15.57:0/3331660281 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f399c0ceb40 0x7f399c0c7e30 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:04.054 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.049+0000 7f39a33f5640 1 -- 172.21.15.57:0/3331660281 >> 172.21.15.57:0/3331660281 conn(0x7f399c0d0d60 msgr2=0x7f399c15a2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:46:04.054 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.049+0000 7f39a33f5640 1 -- 172.21.15.57:0/3331660281 shutdown_connections 2023-10-19T12:46:04.054 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.049+0000 7f39a33f5640 1 -- 172.21.15.57:0/3331660281 wait complete. 2023-10-19T12:46:04.062 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir5 2023-10-19T12:46:04.063 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir6 1m 2023-10-19T12:46:04.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.173+0000 7f9a6b1a9640 1 -- 172.21.15.57:0/3438051360 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a640cbd20 msgr2=0x7f9a64158300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:46:04.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.173+0000 7f9a6b1a9640 1 --2- 172.21.15.57:0/3438051360 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a640cbd20 0x7f9a64158300 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f9a58005fd0 tx=0x7f9a5802d0d0 comp rx=0 tx=0).stop 2023-10-19T12:46:04.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.173+0000 7f9a6b1a9640 1 -- 172.21.15.57:0/3438051360 shutdown_connections 2023-10-19T12:46:04.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.173+0000 7f9a6b1a9640 1 --2- 172.21.15.57:0/3438051360 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9a640afa20 0x7f9a640705a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:04.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.173+0000 7f9a6b1a9640 1 --2- 172.21.15.57:0/3438051360 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a640cbd20 0x7f9a64158300 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:04.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.173+0000 7f9a6b1a9640 1 -- 172.21.15.57:0/3438051360 >> 172.21.15.57:0/3438051360 conn(0x7f9a64069930 msgr2=0x7f9a64069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:46:04.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.173+0000 7f9a6b1a9640 1 -- 172.21.15.57:0/3438051360 shutdown_connections 2023-10-19T12:46:04.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.173+0000 7f9a6b1a9640 1 -- 172.21.15.57:0/3438051360 wait complete. 2023-10-19T12:46:04.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.174+0000 7f9a6b1a9640 1 Processor -- start 2023-10-19T12:46:04.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.174+0000 7f9a6b1a9640 1 -- start start 2023-10-19T12:46:04.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.175+0000 7f9a6b1a9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a640afa20 0x7f9a6414a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:46:04.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.175+0000 7f9a6b1a9640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9a640b0530 0x7f9a641e16a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:46:04.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.175+0000 7f9a6b1a9640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9a640cbd20 0x7f9a641e0520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:46:04.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.175+0000 7f9a6b1a9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a6414a010 con 0x7f9a640afa20 2023-10-19T12:46:04.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.175+0000 7f9a6b1a9640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a6414a970 con 0x7f9a640cbd20 2023-10-19T12:46:04.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.175+0000 7f9a6b1a9640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a64149390 con 0x7f9a640b0530 2023-10-19T12:46:04.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.175+0000 7f9a68f45640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a640afa20 0x7f9a6414a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:46:04.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.175+0000 7f9a68f45640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a640afa20 0x7f9a6414a430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:36314/0 (socket says 172.21.15.57:36314) 2023-10-19T12:46:04.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.175+0000 7f9a68f45640 1 -- 172.21.15.57:0/2854126469 learned_addr learned my addr 172.21.15.57:0/2854126469 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:46:04.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.175+0000 7f9a63fff640 1 --2- 172.21.15.57:0/2854126469 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9a640b0530 0x7f9a641e16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:46:04.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.175+0000 7f9a69746640 1 --2- 172.21.15.57:0/2854126469 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9a640cbd20 0x7f9a641e0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:46:04.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.175+0000 7f9a68f45640 1 -- 172.21.15.57:0/2854126469 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9a640b0530 msgr2=0x7f9a641e16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:46:04.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.175+0000 7f9a68f45640 1 --2- 172.21.15.57:0/2854126469 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9a640b0530 0x7f9a641e16a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:04.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.175+0000 7f9a68f45640 1 -- 172.21.15.57:0/2854126469 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9a640cbd20 msgr2=0x7f9a641e0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:46:04.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.175+0000 7f9a68f45640 1 --2- 172.21.15.57:0/2854126469 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9a640cbd20 0x7f9a641e0520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:04.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.175+0000 7f9a68f45640 1 -- 172.21.15.57:0/2854126469 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a5800a670 con 0x7f9a640afa20 2023-10-19T12:46:04.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.175+0000 7f9a68f45640 1 --2- 172.21.15.57:0/2854126469 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a640afa20 0x7f9a6414a430 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f9a5800a960 tx=0x7f9a580047a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:46:04.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.175+0000 7f9a617fa640 1 -- 172.21.15.57:0/2854126469 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a58004da0 con 0x7f9a640afa20 2023-10-19T12:46:04.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.175+0000 7f9a617fa640 1 -- 172.21.15.57:0/2854126469 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f9a580035f0 con 0x7f9a640afa20 2023-10-19T12:46:04.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.175+0000 7f9a6b1a9640 1 -- 172.21.15.57:0/2854126469 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a64070cf0 con 0x7f9a640afa20 2023-10-19T12:46:04.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.175+0000 7f9a617fa640 1 -- 172.21.15.57:0/2854126469 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a580039a0 con 0x7f9a640afa20 2023-10-19T12:46:04.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.176+0000 7f9a6b1a9640 1 -- 172.21.15.57:0/2854126469 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a64149950 con 0x7f9a640afa20 2023-10-19T12:46:04.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.177+0000 7f9a6b1a9640 1 -- 172.21.15.57:0/2854126469 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a6414a150 con 0x7f9a640afa20 2023-10-19T12:46:04.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.177+0000 7f9a617fa640 1 -- 172.21.15.57:0/2854126469 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 11) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f9a58003b00 con 0x7f9a640afa20 2023-10-19T12:46:04.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.177+0000 7f9a617fa640 1 --2- 172.21.15.57:0/2854126469 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f9a4806b0d0 0x7f9a4806d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:46:04.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.177+0000 7f9a617fa640 1 -- 172.21.15.57:0/2854126469 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7f9a580acdc0 con 0x7f9a640afa20 2023-10-19T12:46:04.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.178+0000 7f9a63fff640 1 --2- 172.21.15.57:0/2854126469 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f9a4806b0d0 0x7f9a4806d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:46:04.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.178+0000 7f9a63fff640 1 --2- 172.21.15.57:0/2854126469 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f9a4806b0d0 0x7f9a4806d590 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9a6413db70 tx=0x7f9a5000b390 comp rx=0 tx=0).ready entity=mgr.5428 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:46:04.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.182+0000 7f9a617fa640 1 -- 172.21.15.57:0/2854126469 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9a58077a10 con 0x7f9a640afa20 2023-10-19T12:46:04.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.379+0000 7f9a6b1a9640 1 -- 172.21.15.57:0/2854126469 --> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir6", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f9a640cc1e0 con 0x7f9a4806b0d0 2023-10-19T12:46:04.439 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.436+0000 7f9a617fa640 1 -- 172.21.15.57:0/2854126469 <== mgr.5428 v2:172.21.15.179:6832/922398470 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f9a640cc1e0 con 0x7f9a4806b0d0 2023-10-19T12:46:04.439 INFO:teuthology.orchestra.run.smithi057.stdout:Schedule activated for path /snap_test_dir1/dir6 2023-10-19T12:46:04.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.438+0000 7f9a6b1a9640 1 -- 172.21.15.57:0/2854126469 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f9a4806b0d0 msgr2=0x7f9a4806d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:46:04.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.438+0000 7f9a6b1a9640 1 --2- 172.21.15.57:0/2854126469 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f9a4806b0d0 0x7f9a4806d590 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9a6413db70 tx=0x7f9a5000b390 comp rx=0 tx=0).stop 2023-10-19T12:46:04.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.438+0000 7f9a6b1a9640 1 -- 172.21.15.57:0/2854126469 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a640afa20 msgr2=0x7f9a6414a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:46:04.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.438+0000 7f9a6b1a9640 1 --2- 172.21.15.57:0/2854126469 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a640afa20 0x7f9a6414a430 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f9a5800a960 tx=0x7f9a580047a0 comp rx=0 tx=0).stop 2023-10-19T12:46:04.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.438+0000 7f9a6b1a9640 1 -- 172.21.15.57:0/2854126469 shutdown_connections 2023-10-19T12:46:04.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.438+0000 7f9a6b1a9640 1 --2- 172.21.15.57:0/2854126469 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f9a4806b0d0 0x7f9a4806d590 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:04.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.438+0000 7f9a6b1a9640 1 --2- 172.21.15.57:0/2854126469 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9a640cbd20 0x7f9a641e0520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:04.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.438+0000 7f9a6b1a9640 1 --2- 172.21.15.57:0/2854126469 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9a640b0530 0x7f9a641e16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:04.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.438+0000 7f9a6b1a9640 1 --2- 172.21.15.57:0/2854126469 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a640afa20 0x7f9a6414a430 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:04.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.438+0000 7f9a6b1a9640 1 -- 172.21.15.57:0/2854126469 >> 172.21.15.57:0/2854126469 conn(0x7f9a64069930 msgr2=0x7f9a640c98e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:46:04.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.438+0000 7f9a6b1a9640 1 -- 172.21.15.57:0/2854126469 shutdown_connections 2023-10-19T12:46:04.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.438+0000 7f9a6b1a9640 1 -- 172.21.15.57:0/2854126469 wait complete. 2023-10-19T12:46:04.451 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir6 2023-10-19T12:46:04.452 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir7 1m 2023-10-19T12:46:04.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.560+0000 7f9b67305640 1 -- 172.21.15.57:0/1129336612 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b600cf340 msgr2=0x7f9b600d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:46:04.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.560+0000 7f9b67305640 1 --2- 172.21.15.57:0/1129336612 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b600cf340 0x7f9b600d1910 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f9b58005fd0 tx=0x7f9b5802bd90 comp rx=0 tx=0).stop 2023-10-19T12:46:04.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.560+0000 7f9b67305640 1 -- 172.21.15.57:0/1129336612 shutdown_connections 2023-10-19T12:46:04.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.560+0000 7f9b67305640 1 --2- 172.21.15.57:0/1129336612 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9b600ceb40 0x7f9b600b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:04.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.560+0000 7f9b67305640 1 --2- 172.21.15.57:0/1129336612 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b600cf340 0x7f9b600d1910 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:04.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.560+0000 7f9b67305640 1 -- 172.21.15.57:0/1129336612 >> 172.21.15.57:0/1129336612 conn(0x7f9b600d0d60 msgr2=0x7f9b600d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:46:04.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.560+0000 7f9b67305640 1 -- 172.21.15.57:0/1129336612 shutdown_connections 2023-10-19T12:46:04.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.560+0000 7f9b67305640 1 -- 172.21.15.57:0/1129336612 wait complete. 2023-10-19T12:46:04.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.561+0000 7f9b67305640 1 Processor -- start 2023-10-19T12:46:04.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.561+0000 7f9b67305640 1 -- start start 2023-10-19T12:46:04.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.562+0000 7f9b67305640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9b600ceb40 0x7f9b600b39c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:46:04.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.562+0000 7f9b67305640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9b600cf340 0x7f9b600b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:46:04.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.562+0000 7f9b67305640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b6015b2a0 0x7f9b6009d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:46:04.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.562+0000 7f9b67305640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b6015d170 con 0x7f9b6015b2a0 2023-10-19T12:46:04.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.562+0000 7f9b67305640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b600b4a10 con 0x7f9b600cf340 2023-10-19T12:46:04.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.562+0000 7f9b67305640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b600b4cb0 con 0x7f9b600ceb40 2023-10-19T12:46:04.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.562+0000 7f9b658a2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b6015b2a0 0x7f9b6009d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:46:04.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.562+0000 7f9b658a2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b6015b2a0 0x7f9b6009d470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:36336/0 (socket says 172.21.15.57:36336) 2023-10-19T12:46:04.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.562+0000 7f9b658a2640 1 -- 172.21.15.57:0/2192267298 learned_addr learned my addr 172.21.15.57:0/2192267298 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:46:04.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.562+0000 7f9b648a0640 1 --2- 172.21.15.57:0/2192267298 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9b600cf340 0x7f9b600b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:46:04.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.562+0000 7f9b650a1640 1 --2- 172.21.15.57:0/2192267298 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9b600ceb40 0x7f9b600b39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:46:04.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.563+0000 7f9b658a2640 1 -- 172.21.15.57:0/2192267298 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9b600ceb40 msgr2=0x7f9b600b39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:46:04.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.563+0000 7f9b658a2640 1 --2- 172.21.15.57:0/2192267298 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9b600ceb40 0x7f9b600b39c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:04.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.563+0000 7f9b658a2640 1 -- 172.21.15.57:0/2192267298 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9b600cf340 msgr2=0x7f9b600b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:46:04.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.563+0000 7f9b658a2640 1 --2- 172.21.15.57:0/2192267298 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9b600cf340 0x7f9b600b44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:04.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.563+0000 7f9b658a2640 1 -- 172.21.15.57:0/2192267298 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b58009560 con 0x7f9b6015b2a0 2023-10-19T12:46:04.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.563+0000 7f9b658a2640 1 --2- 172.21.15.57:0/2192267298 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b6015b2a0 0x7f9b6009d470 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f9b5000eaa0 tx=0x7f9b5000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:46:04.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.563+0000 7f9b5dffb640 1 -- 172.21.15.57:0/2192267298 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b5000cdb0 con 0x7f9b6015b2a0 2023-10-19T12:46:04.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.563+0000 7f9b5dffb640 1 -- 172.21.15.57:0/2192267298 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f9b50004b20 con 0x7f9b6015b2a0 2023-10-19T12:46:04.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.563+0000 7f9b67305640 1 -- 172.21.15.57:0/2192267298 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b600b0d50 con 0x7f9b6015b2a0 2023-10-19T12:46:04.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.563+0000 7f9b5dffb640 1 -- 172.21.15.57:0/2192267298 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b50005020 con 0x7f9b6015b2a0 2023-10-19T12:46:04.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.563+0000 7f9b67305640 1 -- 172.21.15.57:0/2192267298 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b6015cd90 con 0x7f9b6015b2a0 2023-10-19T12:46:04.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.564+0000 7f9b67305640 1 -- 172.21.15.57:0/2192267298 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b6015d2b0 con 0x7f9b6015b2a0 2023-10-19T12:46:04.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.565+0000 7f9b5dffb640 1 -- 172.21.15.57:0/2192267298 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 11) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f9b50005180 con 0x7f9b6015b2a0 2023-10-19T12:46:04.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.565+0000 7f9b5dffb640 1 --2- 172.21.15.57:0/2192267298 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f9b4c06b0d0 0x7f9b4c06d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:46:04.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.565+0000 7f9b5dffb640 1 -- 172.21.15.57:0/2192267298 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7f9b5008a6a0 con 0x7f9b6015b2a0 2023-10-19T12:46:04.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.565+0000 7f9b650a1640 1 --2- 172.21.15.57:0/2192267298 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f9b4c06b0d0 0x7f9b4c06d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:46:04.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.566+0000 7f9b650a1640 1 --2- 172.21.15.57:0/2192267298 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f9b4c06b0d0 0x7f9b4c06d590 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9b58035000 tx=0x7f9b58004730 comp rx=0 tx=0).ready entity=mgr.5428 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:46:04.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.569+0000 7f9b5dffb640 1 -- 172.21.15.57:0/2192267298 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9b6015d2b0 con 0x7f9b6015b2a0 2023-10-19T12:46:04.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.767+0000 7f9b67305640 1 -- 172.21.15.57:0/2192267298 --> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir7", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f9b6015d2b0 con 0x7f9b4c06b0d0 2023-10-19T12:46:04.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.831+0000 7f9b5dffb640 1 -- 172.21.15.57:0/2192267298 <== mgr.5428 v2:172.21.15.179:6832/922398470 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f9b6015d2b0 con 0x7f9b4c06b0d0 2023-10-19T12:46:04.835 INFO:teuthology.orchestra.run.smithi057.stdout:Schedule activated for path /snap_test_dir1/dir7 2023-10-19T12:46:04.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.833+0000 7f9b67305640 1 -- 172.21.15.57:0/2192267298 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f9b4c06b0d0 msgr2=0x7f9b4c06d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:46:04.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.833+0000 7f9b67305640 1 --2- 172.21.15.57:0/2192267298 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f9b4c06b0d0 0x7f9b4c06d590 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9b58035000 tx=0x7f9b58004730 comp rx=0 tx=0).stop 2023-10-19T12:46:04.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.833+0000 7f9b67305640 1 -- 172.21.15.57:0/2192267298 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b6015b2a0 msgr2=0x7f9b6009d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:46:04.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.833+0000 7f9b67305640 1 --2- 172.21.15.57:0/2192267298 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b6015b2a0 0x7f9b6009d470 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f9b5000eaa0 tx=0x7f9b5000ef60 comp rx=0 tx=0).stop 2023-10-19T12:46:04.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.834+0000 7f9b67305640 1 -- 172.21.15.57:0/2192267298 shutdown_connections 2023-10-19T12:46:04.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.834+0000 7f9b67305640 1 --2- 172.21.15.57:0/2192267298 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f9b4c06b0d0 0x7f9b4c06d590 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:04.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.834+0000 7f9b67305640 1 --2- 172.21.15.57:0/2192267298 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b6015b2a0 0x7f9b6009d470 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:04.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.834+0000 7f9b67305640 1 --2- 172.21.15.57:0/2192267298 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9b600cf340 0x7f9b600b44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:04.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.834+0000 7f9b67305640 1 --2- 172.21.15.57:0/2192267298 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9b600ceb40 0x7f9b600b39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:04.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.834+0000 7f9b67305640 1 -- 172.21.15.57:0/2192267298 >> 172.21.15.57:0/2192267298 conn(0x7f9b600d0d60 msgr2=0x7f9b6015a560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:46:04.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.834+0000 7f9b67305640 1 -- 172.21.15.57:0/2192267298 shutdown_connections 2023-10-19T12:46:04.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.834+0000 7f9b67305640 1 -- 172.21.15.57:0/2192267298 wait complete. 2023-10-19T12:46:04.847 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir7 2023-10-19T12:46:04.847 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir8 1m 2023-10-19T12:46:04.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.956+0000 7f74fb426640 1 -- 172.21.15.57:0/2919092906 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f74f415b2a0 msgr2=0x7f74f40cf720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:46:04.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.956+0000 7f74fb426640 1 --2- 172.21.15.57:0/2919092906 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f74f415b2a0 0x7f74f40cf720 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f74e8005fd0 tx=0x7f74e802bd90 comp rx=0 tx=0).stop 2023-10-19T12:46:04.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.957+0000 7f74fb426640 1 -- 172.21.15.57:0/2919092906 shutdown_connections 2023-10-19T12:46:04.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.957+0000 7f74fb426640 1 --2- 172.21.15.57:0/2919092906 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f74f415b2a0 0x7f74f40cf720 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:04.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.957+0000 7f74fb426640 1 --2- 172.21.15.57:0/2919092906 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f74f40cf340 0x7f74f40d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:04.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.957+0000 7f74fb426640 1 -- 172.21.15.57:0/2919092906 >> 172.21.15.57:0/2919092906 conn(0x7f74f40d0d60 msgr2=0x7f74f40d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:46:04.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.957+0000 7f74fb426640 1 -- 172.21.15.57:0/2919092906 shutdown_connections 2023-10-19T12:46:04.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.957+0000 7f74fb426640 1 -- 172.21.15.57:0/2919092906 wait complete. 2023-10-19T12:46:04.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.958+0000 7f74fb426640 1 Processor -- start 2023-10-19T12:46:04.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.958+0000 7f74fb426640 1 -- start start 2023-10-19T12:46:04.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.958+0000 7f74fb426640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f74f40ceb40 0x7f74f40b4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:46:04.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.958+0000 7f74fb426640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f74f40cf340 0x7f74f40b5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:46:04.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.958+0000 7f74fb426640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f74f415b2a0 0x7f74f40b3980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:46:04.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.958+0000 7f74fb426640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74f40c7e30 con 0x7f74f415b2a0 2023-10-19T12:46:04.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.958+0000 7f74fb426640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74f40c8040 con 0x7f74f40ceb40 2023-10-19T12:46:04.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.958+0000 7f74fb426640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74f40c8320 con 0x7f74f40cf340 2023-10-19T12:46:04.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.959+0000 7f74f99c3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f74f415b2a0 0x7f74f40b3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:46:04.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.959+0000 7f74f99c3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f74f415b2a0 0x7f74f40b3980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:36338/0 (socket says 172.21.15.57:36338) 2023-10-19T12:46:04.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.959+0000 7f74f99c3640 1 -- 172.21.15.57:0/2523145814 learned_addr learned my addr 172.21.15.57:0/2523145814 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:46:04.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.959+0000 7f74f89c1640 1 --2- 172.21.15.57:0/2523145814 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f74f40cf340 0x7f74f40b5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:46:04.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.959+0000 7f74f91c2640 1 --2- 172.21.15.57:0/2523145814 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f74f40ceb40 0x7f74f40b4db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:46:04.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.959+0000 7f74f99c3640 1 -- 172.21.15.57:0/2523145814 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f74f40cf340 msgr2=0x7f74f40b5790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:46:04.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.959+0000 7f74f99c3640 1 --2- 172.21.15.57:0/2523145814 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f74f40cf340 0x7f74f40b5790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:04.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.959+0000 7f74f99c3640 1 -- 172.21.15.57:0/2523145814 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f74f40ceb40 msgr2=0x7f74f40b4db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:46:04.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.959+0000 7f74f99c3640 1 --2- 172.21.15.57:0/2523145814 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f74f40ceb40 0x7f74f40b4db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:04.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.959+0000 7f74f99c3640 1 -- 172.21.15.57:0/2523145814 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74ec009840 con 0x7f74f415b2a0 2023-10-19T12:46:04.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.959+0000 7f74f91c2640 1 --2- 172.21.15.57:0/2523145814 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f74f40ceb40 0x7f74f40b4db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:46:04.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.959+0000 7f74f99c3640 1 --2- 172.21.15.57:0/2523145814 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f74f415b2a0 0x7f74f40b3980 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f74e400c3c0 tx=0x7f74e400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:46:04.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.959+0000 7f74f1ffb640 1 -- 172.21.15.57:0/2523145814 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74e40052d0 con 0x7f74f415b2a0 2023-10-19T12:46:04.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.959+0000 7f74fb426640 1 -- 172.21.15.57:0/2523145814 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74e8009560 con 0x7f74f415b2a0 2023-10-19T12:46:04.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.959+0000 7f74f1ffb640 1 -- 172.21.15.57:0/2523145814 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f74e4007020 con 0x7f74f415b2a0 2023-10-19T12:46:04.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.959+0000 7f74f1ffb640 1 -- 172.21.15.57:0/2523145814 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74e4007430 con 0x7f74f415b2a0 2023-10-19T12:46:04.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.960+0000 7f74fb426640 1 -- 172.21.15.57:0/2523145814 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74f40c88e0 con 0x7f74f415b2a0 2023-10-19T12:46:04.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.961+0000 7f74fb426640 1 -- 172.21.15.57:0/2523145814 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74f40690e0 con 0x7f74f415b2a0 2023-10-19T12:46:04.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.961+0000 7f74f1ffb640 1 -- 172.21.15.57:0/2523145814 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 11) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f74e4007590 con 0x7f74f415b2a0 2023-10-19T12:46:04.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.961+0000 7f74f1ffb640 1 --2- 172.21.15.57:0/2523145814 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f74e006b100 0x7f74e006d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:46:04.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.962+0000 7f74f1ffb640 1 -- 172.21.15.57:0/2523145814 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7f74e40890d0 con 0x7f74f415b2a0 2023-10-19T12:46:04.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.962+0000 7f74f91c2640 1 --2- 172.21.15.57:0/2523145814 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f74e006b100 0x7f74e006d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:46:04.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.962+0000 7f74f91c2640 1 --2- 172.21.15.57:0/2523145814 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f74e006b100 0x7f74e006d5c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f74f409e770 tx=0x7f74ec009500 comp rx=0 tx=0).ready entity=mgr.5428 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:46:04.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:04.966+0000 7f74f1ffb640 1 -- 172.21.15.57:0/2523145814 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f74e4053d70 con 0x7f74f415b2a0 2023-10-19T12:46:05.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:05.165+0000 7f74fb426640 1 -- 172.21.15.57:0/2523145814 --> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir8", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f74f40cc170 con 0x7f74e006b100 2023-10-19T12:46:05.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:05.204+0000 7f74f1ffb640 1 -- 172.21.15.57:0/2523145814 <== mgr.5428 v2:172.21.15.179:6832/922398470 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f74ec015070 con 0x7f74e006b100 2023-10-19T12:46:05.207 INFO:teuthology.orchestra.run.smithi057.stdout:Schedule activated for path /snap_test_dir1/dir8 2023-10-19T12:46:05.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:05.206+0000 7f74fb426640 1 -- 172.21.15.57:0/2523145814 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f74e006b100 msgr2=0x7f74e006d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:46:05.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:05.206+0000 7f74fb426640 1 --2- 172.21.15.57:0/2523145814 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f74e006b100 0x7f74e006d5c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f74f409e770 tx=0x7f74ec009500 comp rx=0 tx=0).stop 2023-10-19T12:46:05.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:05.206+0000 7f74fb426640 1 -- 172.21.15.57:0/2523145814 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f74f415b2a0 msgr2=0x7f74f40b3980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:46:05.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:05.206+0000 7f74fb426640 1 --2- 172.21.15.57:0/2523145814 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f74f415b2a0 0x7f74f40b3980 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f74e400c3c0 tx=0x7f74e400c880 comp rx=0 tx=0).stop 2023-10-19T12:46:05.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:05.206+0000 7f74fb426640 1 -- 172.21.15.57:0/2523145814 shutdown_connections 2023-10-19T12:46:05.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:05.206+0000 7f74fb426640 1 --2- 172.21.15.57:0/2523145814 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f74e006b100 0x7f74e006d5c0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:05.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:05.206+0000 7f74fb426640 1 --2- 172.21.15.57:0/2523145814 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f74f415b2a0 0x7f74f40b3980 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:05.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:05.206+0000 7f74fb426640 1 --2- 172.21.15.57:0/2523145814 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f74f40cf340 0x7f74f40b5790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:05.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:05.206+0000 7f74fb426640 1 --2- 172.21.15.57:0/2523145814 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f74f40ceb40 0x7f74f40b4db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:05.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:05.206+0000 7f74fb426640 1 -- 172.21.15.57:0/2523145814 >> 172.21.15.57:0/2523145814 conn(0x7f74f40d0d60 msgr2=0x7f74f40b0ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:46:05.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:05.206+0000 7f74fb426640 1 -- 172.21.15.57:0/2523145814 shutdown_connections 2023-10-19T12:46:05.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:05.206+0000 7f74fb426640 1 -- 172.21.15.57:0/2523145814 wait complete. 2023-10-19T12:46:05.219 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir8 2023-10-19T12:46:05.220 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir9 1m 2023-10-19T12:46:05.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:05.329+0000 7f8224e40640 1 -- 172.21.15.57:0/1949147649 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f82200afa20 msgr2=0x7f8220158300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:46:05.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:05.329+0000 7f8224e40640 1 --2- 172.21.15.57:0/1949147649 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f82200afa20 0x7f8220158300 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f821400a940 tx=0x7f821402d000 comp rx=0 tx=0).stop 2023-10-19T12:46:05.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:05.329+0000 7f8224e40640 1 -- 172.21.15.57:0/1949147649 shutdown_connections 2023-10-19T12:46:05.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:05.329+0000 7f8224e40640 1 --2- 172.21.15.57:0/1949147649 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f82200afa20 0x7f8220158300 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:05.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:05.329+0000 7f8224e40640 1 -- 172.21.15.57:0/1949147649 >> 172.21.15.57:0/1949147649 conn(0x7f8220069930 msgr2=0x7f8220069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:46:05.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:05.329+0000 7f8224e40640 1 -- 172.21.15.57:0/1949147649 shutdown_connections 2023-10-19T12:46:05.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:05.329+0000 7f8224e40640 1 -- 172.21.15.57:0/1949147649 wait complete. 2023-10-19T12:46:05.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:05.330+0000 7f8224e40640 1 Processor -- start 2023-10-19T12:46:05.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:05.330+0000 7f8224e40640 1 -- start start 2023-10-19T12:46:05.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:05.331+0000 7f8224e40640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f82200afa20 0x7f8220156330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:46:05.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:05.331+0000 7f8224e40640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f82200b0530 0x7f822013b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:46:05.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:05.331+0000 7f8224e40640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f82200cbd20 0x7f8220152870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:46:05.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:05.331+0000 7f8224e40640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8220156f00 con 0x7f82200b0530 2023-10-19T12:46:05.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:05.331+0000 7f8224e40640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8220156940 con 0x7f82200cbd20 2023-10-19T12:46:05.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:05.331+0000 7f8224e40640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8220156c20 con 0x7f82200afa20 2023-10-19T12:46:05.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:05.331+0000 7f821dd9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f82200b0530 0x7f822013b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:46:05.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:05.331+0000 7f821dd9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f82200b0530 0x7f822013b1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:36354/0 (socket says 172.21.15.57:36354) 2023-10-19T12:46:05.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:05.331+0000 7f821dd9b640 1 -- 172.21.15.57:0/2866469907 learned_addr learned my addr 172.21.15.57:0/2866469907 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:46:05.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:05.331+0000 7f821ed9d640 1 --2- 172.21.15.57:0/2866469907 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f82200cbd20 0x7f8220152870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:46:05.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:05.331+0000 7f821e59c640 1 --2- 172.21.15.57:0/2866469907 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f82200afa20 0x7f8220156330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:46:05.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:05.331+0000 7f821dd9b640 1 -- 172.21.15.57:0/2866469907 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f82200afa20 msgr2=0x7f8220156330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:46:05.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:05.331+0000 7f821dd9b640 1 --2- 172.21.15.57:0/2866469907 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f82200afa20 0x7f8220156330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:05.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:05.331+0000 7f821dd9b640 1 -- 172.21.15.57:0/2866469907 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f82200cbd20 msgr2=0x7f8220152870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:46:05.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:05.331+0000 7f821dd9b640 1 --2- 172.21.15.57:0/2866469907 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f82200cbd20 0x7f8220152870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:05.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:05.331+0000 7f821dd9b640 1 -- 172.21.15.57:0/2866469907 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f821400a5a0 con 0x7f82200b0530 2023-10-19T12:46:05.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:05.331+0000 7f821ed9d640 1 --2- 172.21.15.57:0/2866469907 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f82200cbd20 0x7f8220152870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T12:46:05.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:05.332+0000 7f821dd9b640 1 --2- 172.21.15.57:0/2866469907 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f82200b0530 0x7f822013b1b0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f8214000c00 tx=0x7f8214003ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:46:05.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:05.332+0000 7f820effd640 1 -- 172.21.15.57:0/2866469907 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8214035640 con 0x7f82200b0530 2023-10-19T12:46:05.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:05.332+0000 7f820effd640 1 -- 172.21.15.57:0/2866469907 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f8214035890 con 0x7f82200b0530 2023-10-19T12:46:05.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:05.332+0000 7f8224e40640 1 -- 172.21.15.57:0/2866469907 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82201ef920 con 0x7f82200b0530 2023-10-19T12:46:05.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:05.332+0000 7f820effd640 1 -- 172.21.15.57:0/2866469907 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8214035ca0 con 0x7f82200b0530 2023-10-19T12:46:05.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:05.332+0000 7f8224e40640 1 -- 172.21.15.57:0/2866469907 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82200ad900 con 0x7f82200b0530 2023-10-19T12:46:05.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:05.333+0000 7f8224e40640 1 -- 172.21.15.57:0/2866469907 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82201efe40 con 0x7f82200b0530 2023-10-19T12:46:05.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:05.333+0000 7f820effd640 1 -- 172.21.15.57:0/2866469907 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 11) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f821403c040 con 0x7f82200b0530 2023-10-19T12:46:05.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:05.334+0000 7f820effd640 1 --2- 172.21.15.57:0/2866469907 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f820006b0d0 0x7f820006d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:46:05.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:05.334+0000 7f820effd640 1 -- 172.21.15.57:0/2866469907 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6287+0+0 (secure 0 0 0) 0x7f82140ac560 con 0x7f82200b0530 2023-10-19T12:46:05.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:05.334+0000 7f821e59c640 1 --2- 172.21.15.57:0/2866469907 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f820006b0d0 0x7f820006d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:46:05.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:05.334+0000 7f821e59c640 1 --2- 172.21.15.57:0/2866469907 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f820006b0d0 0x7f820006d590 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f82180011e0 tx=0x7f8218004600 comp rx=0 tx=0).ready entity=mgr.5428 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:46:05.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:05.338+0000 7f820effd640 1 -- 172.21.15.57:0/2866469907 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f82201efe40 con 0x7f82200b0530 2023-10-19T12:46:05.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:05.538+0000 7f8224e40640 1 -- 172.21.15.57:0/2866469907 --> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir9", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f82201efe40 con 0x7f820006b0d0 2023-10-19T12:46:05.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:05.585+0000 7f820effd640 1 -- 172.21.15.57:0/2866469907 <== mgr.5428 v2:172.21.15.179:6832/922398470 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f82201efe40 con 0x7f820006b0d0 2023-10-19T12:46:05.589 INFO:teuthology.orchestra.run.smithi057.stdout:Schedule activated for path /snap_test_dir1/dir9 2023-10-19T12:46:05.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:05.588+0000 7f8224e40640 1 -- 172.21.15.57:0/2866469907 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f820006b0d0 msgr2=0x7f820006d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:46:05.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:05.588+0000 7f8224e40640 1 --2- 172.21.15.57:0/2866469907 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f820006b0d0 0x7f820006d590 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f82180011e0 tx=0x7f8218004600 comp rx=0 tx=0).stop 2023-10-19T12:46:05.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:05.588+0000 7f8224e40640 1 -- 172.21.15.57:0/2866469907 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f82200b0530 msgr2=0x7f822013b1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:46:05.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:05.588+0000 7f8224e40640 1 --2- 172.21.15.57:0/2866469907 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f82200b0530 0x7f822013b1b0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f8214000c00 tx=0x7f8214003ae0 comp rx=0 tx=0).stop 2023-10-19T12:46:05.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:05.588+0000 7f8224e40640 1 -- 172.21.15.57:0/2866469907 shutdown_connections 2023-10-19T12:46:05.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:05.588+0000 7f8224e40640 1 --2- 172.21.15.57:0/2866469907 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f820006b0d0 0x7f820006d590 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:05.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:05.588+0000 7f8224e40640 1 --2- 172.21.15.57:0/2866469907 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f82200cbd20 0x7f8220152870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:05.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:05.588+0000 7f8224e40640 1 --2- 172.21.15.57:0/2866469907 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f82200b0530 0x7f822013b1b0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:05.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:05.588+0000 7f8224e40640 1 --2- 172.21.15.57:0/2866469907 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f82200afa20 0x7f8220156330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:46:05.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:05.588+0000 7f8224e40640 1 -- 172.21.15.57:0/2866469907 >> 172.21.15.57:0/2866469907 conn(0x7f8220069930 msgr2=0x7f82200ae700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:46:05.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:05.588+0000 7f8224e40640 1 -- 172.21.15.57:0/2866469907 shutdown_connections 2023-10-19T12:46:05.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:46:05.588+0000 7f8224e40640 1 -- 172.21.15.57:0/2866469907 wait complete. 2023-10-19T12:46:05.601 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir9 2023-10-19T12:46:13.814 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T12:46:13.818 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T12:46:43.880 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T12:46:43.883 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T12:47:14.035 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T12:47:14.039 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T12:47:44.116 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T12:47:44.119 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T12:48:14.263 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T12:48:14.267 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T12:48:44.335 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T12:48:44.339 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T12:49:14.483 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T12:49:14.486 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T12:49:44.697 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T12:49:44.701 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T12:50:14.836 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T12:50:14.840 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T12:50:44.950 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T12:50:44.954 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T12:51:00.837 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:51:00.834+0000 7fe4908bc640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T12:51:00.837 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:51:00.834+0000 7fe4908bc640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T12:51:00.838 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:51:00.834+0000 7fe4908bc640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T12:51:00.838 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:51:00.834+0000 7fe4908bc640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T12:51:00.838 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:51:00.834+0000 7fe4908bc640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T12:51:15.105 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T12:51:15.109 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T12:51:45.254 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T12:51:45.257 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T12:52:15.375 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T12:52:15.378 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T12:52:45.514 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T12:52:45.519 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T12:53:15.594 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T12:53:15.597 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T12:53:45.684 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T12:53:45.688 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T12:54:15.771 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T12:54:15.776 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T12:54:45.912 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T12:54:45.916 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T12:55:00.147 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:55:00.144+0000 7fe483021640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T12:55:00.147 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:55:00.144+0000 7fe483021640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T12:55:00.147 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:55:00.144+0000 7fe483021640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T12:55:00.148 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:55:00.144+0000 7fe483021640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T12:55:00.148 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:55:00.144+0000 7fe483021640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T12:55:15.975 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T12:55:15.979 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T12:55:46.125 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T12:55:46.129 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T12:56:16.346 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T12:56:16.350 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T12:56:46.468 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T12:56:46.472 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T12:57:16.558 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T12:57:16.561 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T12:57:19.603 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir0 1m 2023-10-19T12:57:19.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:19.725+0000 7fa907e4b640 1 -- 172.21.15.57:0/2530538203 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa90013d360 msgr2=0x7fa90015a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:19.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:19.725+0000 7fa907e4b640 1 --2- 172.21.15.57:0/2530538203 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa90013d360 0x7fa90015a9f0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fa90006f010 tx=0x7fa8fc02be40 comp rx=0 tx=0).stop 2023-10-19T12:57:19.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:19.725+0000 7fa907e4b640 1 -- 172.21.15.57:0/2530538203 shutdown_connections 2023-10-19T12:57:19.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:19.725+0000 7fa907e4b640 1 --2- 172.21.15.57:0/2530538203 >> v2:172.21.15.179:3300/0 conn(0x7fa9000c9ed0 0x7fa90006b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:19.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:19.725+0000 7fa907e4b640 1 --2- 172.21.15.57:0/2530538203 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa90013d360 0x7fa90015a9f0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:19.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:19.725+0000 7fa907e4b640 1 -- 172.21.15.57:0/2530538203 >> 172.21.15.57:0/2530538203 conn(0x7fa900069930 msgr2=0x7fa900069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:19.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:19.725+0000 7fa907e4b640 1 -- 172.21.15.57:0/2530538203 shutdown_connections 2023-10-19T12:57:19.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:19.725+0000 7fa907e4b640 1 -- 172.21.15.57:0/2530538203 wait complete. 2023-10-19T12:57:19.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:19.727+0000 7fa907e4b640 1 Processor -- start 2023-10-19T12:57:19.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:19.727+0000 7fa907e4b640 1 -- start start 2023-10-19T12:57:19.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:19.727+0000 7fa907e4b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa9000c9ed0 0x7fa9001f4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:19.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:19.727+0000 7fa907e4b640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa90013d360 0x7fa900158cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:19.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:19.727+0000 7fa907e4b640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa900140660 0x7fa900136a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:19.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:19.727+0000 7fa907e4b640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa900158460 con 0x7fa9000c9ed0 2023-10-19T12:57:19.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:19.727+0000 7fa907e4b640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa900157850 con 0x7fa90013d360 2023-10-19T12:57:19.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:19.727+0000 7fa907e4b640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa900157b30 con 0x7fa900140660 2023-10-19T12:57:19.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:19.728+0000 7fa905be7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa9000c9ed0 0x7fa9001f4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:19.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:19.728+0000 7fa905be7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa9000c9ed0 0x7fa9001f4ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33926/0 (socket says 172.21.15.57:33926) 2023-10-19T12:57:19.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:19.728+0000 7fa905be7640 1 -- 172.21.15.57:0/819612148 learned_addr learned my addr 172.21.15.57:0/819612148 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:57:19.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:19.728+0000 7fa9053e6640 1 --2- 172.21.15.57:0/819612148 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa90013d360 0x7fa900158cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:19.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:19.728+0000 7fa9063e8640 1 --2- 172.21.15.57:0/819612148 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa900140660 0x7fa900136a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:19.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:19.728+0000 7fa905be7640 1 -- 172.21.15.57:0/819612148 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa900140660 msgr2=0x7fa900136a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:19.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:19.728+0000 7fa905be7640 1 --2- 172.21.15.57:0/819612148 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa900140660 0x7fa900136a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:19.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:19.728+0000 7fa905be7640 1 -- 172.21.15.57:0/819612148 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa90013d360 msgr2=0x7fa900158cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:19.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:19.728+0000 7fa905be7640 1 --2- 172.21.15.57:0/819612148 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa90013d360 0x7fa900158cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:19.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:19.728+0000 7fa905be7640 1 -- 172.21.15.57:0/819612148 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8fc009560 con 0x7fa9000c9ed0 2023-10-19T12:57:19.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:19.728+0000 7fa9053e6640 1 --2- 172.21.15.57:0/819612148 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa90013d360 0x7fa900158cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:57:19.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:19.728+0000 7fa905be7640 1 --2- 172.21.15.57:0/819612148 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa9000c9ed0 0x7fa9001f4ba0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fa8fc036040 tx=0x7fa8fc02de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:19.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:19.728+0000 7fa8f67fc640 1 -- 172.21.15.57:0/819612148 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8fc005d50 con 0x7fa9000c9ed0 2023-10-19T12:57:19.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:19.728+0000 7fa8f67fc640 1 -- 172.21.15.57:0/819612148 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fa8fc003910 con 0x7fa9000c9ed0 2023-10-19T12:57:19.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:19.728+0000 7fa907e4b640 1 -- 172.21.15.57:0/819612148 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa900157e10 con 0x7fa9000c9ed0 2023-10-19T12:57:19.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:19.728+0000 7fa8f67fc640 1 -- 172.21.15.57:0/819612148 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8fc003d20 con 0x7fa9000c9ed0 2023-10-19T12:57:19.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:19.728+0000 7fa907e4b640 1 -- 172.21.15.57:0/819612148 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9001f50e0 con 0x7fa9000c9ed0 2023-10-19T12:57:19.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:19.730+0000 7fa907e4b640 1 -- 172.21.15.57:0/819612148 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa90013feb0 con 0x7fa9000c9ed0 2023-10-19T12:57:19.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:19.730+0000 7fa8f67fc640 1 -- 172.21.15.57:0/819612148 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 15) v1 ==== 89417+0+0 (secure 0 0 0) 0x7fa8fc03e450 con 0x7fa9000c9ed0 2023-10-19T12:57:19.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:19.730+0000 7fa8f67fc640 1 --2- 172.21.15.57:0/819612148 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fa8e406b040 0x7fa8e406d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:19.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:19.730+0000 7fa8f67fc640 1 -- 172.21.15.57:0/819612148 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7fa8fc0ac620 con 0x7fa9000c9ed0 2023-10-19T12:57:19.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:19.731+0000 7fa9053e6640 1 --2- 172.21.15.57:0/819612148 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fa8e406b040 0x7fa8e406d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:19.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:19.731+0000 7fa9053e6640 1 --2- 172.21.15.57:0/819612148 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fa8e406b040 0x7fa8e406d500 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa900069fa0 tx=0x7fa8f0009040 comp rx=0 tx=0).ready entity=mgr.5428 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:19.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:19.735+0000 7fa8f67fc640 1 -- 172.21.15.57:0/819612148 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa8fc0771f0 con 0x7fa9000c9ed0 2023-10-19T12:57:19.934 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:19.931+0000 7fa907e4b640 1 -- 172.21.15.57:0/819612148 --> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir0", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fa90006be50 con 0x7fa8e406b040 2023-10-19T12:57:20.014 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.012+0000 7fa8f67fc640 1 -- 172.21.15.57:0/819612148 <== mgr.5428 v2:172.21.15.179:6832/922398470 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fa8f00012a0 con 0x7fa8e406b040 2023-10-19T12:57:20.032 INFO:teuthology.orchestra.run.smithi057.stdout:Schedule deactivated for path /snap_test_dir1/dir0 2023-10-19T12:57:20.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.014+0000 7fa907e4b640 1 -- 172.21.15.57:0/819612148 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fa8e406b040 msgr2=0x7fa8e406d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:20.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.014+0000 7fa907e4b640 1 --2- 172.21.15.57:0/819612148 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fa8e406b040 0x7fa8e406d500 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa900069fa0 tx=0x7fa8f0009040 comp rx=0 tx=0).stop 2023-10-19T12:57:20.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.014+0000 7fa907e4b640 1 -- 172.21.15.57:0/819612148 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa9000c9ed0 msgr2=0x7fa9001f4ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:20.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.014+0000 7fa907e4b640 1 --2- 172.21.15.57:0/819612148 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa9000c9ed0 0x7fa9001f4ba0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fa8fc036040 tx=0x7fa8fc02de10 comp rx=0 tx=0).stop 2023-10-19T12:57:20.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.014+0000 7fa907e4b640 1 -- 172.21.15.57:0/819612148 shutdown_connections 2023-10-19T12:57:20.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.014+0000 7fa907e4b640 1 --2- 172.21.15.57:0/819612148 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fa8e406b040 0x7fa8e406d500 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:20.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.014+0000 7fa907e4b640 1 --2- 172.21.15.57:0/819612148 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa900140660 0x7fa900136a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:20.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.014+0000 7fa907e4b640 1 --2- 172.21.15.57:0/819612148 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa90013d360 0x7fa900158cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:20.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.014+0000 7fa907e4b640 1 --2- 172.21.15.57:0/819612148 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa9000c9ed0 0x7fa9001f4ba0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:20.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.014+0000 7fa907e4b640 1 -- 172.21.15.57:0/819612148 >> 172.21.15.57:0/819612148 conn(0x7fa900069930 msgr2=0x7fa90013c090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:20.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.014+0000 7fa907e4b640 1 -- 172.21.15.57:0/819612148 shutdown_connections 2023-10-19T12:57:20.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.014+0000 7fa907e4b640 1 -- 172.21.15.57:0/819612148 wait complete. 2023-10-19T12:57:20.037 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir0 2023-10-19T12:57:20.037 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir1 1m 2023-10-19T12:57:20.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.146+0000 7fd24d7f5640 1 -- 172.21.15.57:0/46536473 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd24815b2a0 msgr2=0x7fd2480cf720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:20.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.146+0000 7fd24d7f5640 1 --2- 172.21.15.57:0/46536473 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd24815b2a0 0x7fd2480cf720 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fd24808a810 tx=0x7fd23002bd10 comp rx=0 tx=0).stop 2023-10-19T12:57:20.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.146+0000 7fd24d7f5640 1 -- 172.21.15.57:0/46536473 shutdown_connections 2023-10-19T12:57:20.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.146+0000 7fd24d7f5640 1 --2- 172.21.15.57:0/46536473 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd2480ceb40 0x7fd2480b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:20.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.146+0000 7fd24d7f5640 1 --2- 172.21.15.57:0/46536473 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd24815b2a0 0x7fd2480cf720 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:20.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.146+0000 7fd24d7f5640 1 --2- 172.21.15.57:0/46536473 >> v2:172.21.15.179:3300/0 conn(0x7fd2480cf340 0x7fd2480d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:20.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.146+0000 7fd24d7f5640 1 -- 172.21.15.57:0/46536473 >> 172.21.15.57:0/46536473 conn(0x7fd2480d0d60 msgr2=0x7fd2480d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:20.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.146+0000 7fd24d7f5640 1 -- 172.21.15.57:0/46536473 shutdown_connections 2023-10-19T12:57:20.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.146+0000 7fd24d7f5640 1 -- 172.21.15.57:0/46536473 wait complete. 2023-10-19T12:57:20.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.147+0000 7fd24d7f5640 1 Processor -- start 2023-10-19T12:57:20.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.147+0000 7fd24d7f5640 1 -- start start 2023-10-19T12:57:20.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.148+0000 7fd24d7f5640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd2480ceb40 0x7fd2481520e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:20.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.148+0000 7fd24d7f5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd2480cf340 0x7fd24815da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:20.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.148+0000 7fd24d7f5640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd24815b2a0 0x7fd2480b5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:20.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.148+0000 7fd24d7f5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd248156560 con 0x7fd2480cf340 2023-10-19T12:57:20.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.148+0000 7fd24d7f5640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd24815c570 con 0x7fd24815b2a0 2023-10-19T12:57:20.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.148+0000 7fd24d7f5640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd24815c850 con 0x7fd2480ceb40 2023-10-19T12:57:20.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.148+0000 7fd2467fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd2480cf340 0x7fd24815da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:20.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.148+0000 7fd2467fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd2480cf340 0x7fd24815da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33940/0 (socket says 172.21.15.57:33940) 2023-10-19T12:57:20.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.148+0000 7fd2467fc640 1 -- 172.21.15.57:0/2742417619 learned_addr learned my addr 172.21.15.57:0/2742417619 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:57:20.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.148+0000 7fd246ffd640 1 --2- 172.21.15.57:0/2742417619 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd2480ceb40 0x7fd2481520e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:20.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.148+0000 7fd2467fc640 1 -- 172.21.15.57:0/2742417619 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd2480ceb40 msgr2=0x7fd2481520e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:20.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.148+0000 7fd2467fc640 1 --2- 172.21.15.57:0/2742417619 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd2480ceb40 0x7fd2481520e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:20.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.148+0000 7fd2477fe640 1 --2- 172.21.15.57:0/2742417619 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd24815b2a0 0x7fd2480b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:20.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.148+0000 7fd2467fc640 1 -- 172.21.15.57:0/2742417619 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd24815b2a0 msgr2=0x7fd2480b5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:20.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.148+0000 7fd2467fc640 1 --2- 172.21.15.57:0/2742417619 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd24815b2a0 0x7fd2480b5050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:20.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.148+0000 7fd2467fc640 1 -- 172.21.15.57:0/2742417619 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd230009560 con 0x7fd2480cf340 2023-10-19T12:57:20.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.149+0000 7fd2467fc640 1 --2- 172.21.15.57:0/2742417619 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd2480cf340 0x7fd24815da00 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fd230009850 tx=0x7fd23002da40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:20.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.149+0000 7fd23f7fe640 1 -- 172.21.15.57:0/2742417619 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd230003800 con 0x7fd2480cf340 2023-10-19T12:57:20.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.149+0000 7fd23f7fe640 1 -- 172.21.15.57:0/2742417619 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fd230003a50 con 0x7fd2480cf340 2023-10-19T12:57:20.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.149+0000 7fd24d7f5640 1 -- 172.21.15.57:0/2742417619 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd24815cb30 con 0x7fd2480cf340 2023-10-19T12:57:20.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.149+0000 7fd23f7fe640 1 -- 172.21.15.57:0/2742417619 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd230003e20 con 0x7fd2480cf340 2023-10-19T12:57:20.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.149+0000 7fd24d7f5640 1 -- 172.21.15.57:0/2742417619 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd248152620 con 0x7fd2480cf340 2023-10-19T12:57:20.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.150+0000 7fd24d7f5640 1 -- 172.21.15.57:0/2742417619 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2481566d0 con 0x7fd2480cf340 2023-10-19T12:57:20.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.150+0000 7fd23f7fe640 1 -- 172.21.15.57:0/2742417619 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 15) v1 ==== 89417+0+0 (secure 0 0 0) 0x7fd23003e450 con 0x7fd2480cf340 2023-10-19T12:57:20.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.151+0000 7fd23f7fe640 1 --2- 172.21.15.57:0/2742417619 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fd22806b040 0x7fd22806d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:20.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.151+0000 7fd23f7fe640 1 -- 172.21.15.57:0/2742417619 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7fd2300ac740 con 0x7fd2480cf340 2023-10-19T12:57:20.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.151+0000 7fd246ffd640 1 --2- 172.21.15.57:0/2742417619 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fd22806b040 0x7fd22806d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:20.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.152+0000 7fd246ffd640 1 --2- 172.21.15.57:0/2742417619 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fd22806b040 0x7fd22806d500 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd240005fd0 tx=0x7fd240005850 comp rx=0 tx=0).ready entity=mgr.5428 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:20.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.155+0000 7fd23f7fe640 1 -- 172.21.15.57:0/2742417619 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd230077300 con 0x7fd2480cf340 2023-10-19T12:57:20.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.352+0000 7fd24d7f5640 1 -- 172.21.15.57:0/2742417619 --> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir1", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fd2480cc170 con 0x7fd22806b040 2023-10-19T12:57:20.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.435+0000 7fd23f7fe640 1 -- 172.21.15.57:0/2742417619 <== mgr.5428 v2:172.21.15.179:6832/922398470 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fd240009770 con 0x7fd22806b040 2023-10-19T12:57:20.438 INFO:teuthology.orchestra.run.smithi057.stdout:Schedule deactivated for path /snap_test_dir1/dir1 2023-10-19T12:57:20.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.437+0000 7fd24d7f5640 1 -- 172.21.15.57:0/2742417619 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fd22806b040 msgr2=0x7fd22806d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:20.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.437+0000 7fd24d7f5640 1 --2- 172.21.15.57:0/2742417619 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fd22806b040 0x7fd22806d500 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd240005fd0 tx=0x7fd240005850 comp rx=0 tx=0).stop 2023-10-19T12:57:20.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.437+0000 7fd24d7f5640 1 -- 172.21.15.57:0/2742417619 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd2480cf340 msgr2=0x7fd24815da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:20.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.437+0000 7fd24d7f5640 1 --2- 172.21.15.57:0/2742417619 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd2480cf340 0x7fd24815da00 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fd230009850 tx=0x7fd23002da40 comp rx=0 tx=0).stop 2023-10-19T12:57:20.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.437+0000 7fd24d7f5640 1 -- 172.21.15.57:0/2742417619 shutdown_connections 2023-10-19T12:57:20.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.437+0000 7fd24d7f5640 1 --2- 172.21.15.57:0/2742417619 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fd22806b040 0x7fd22806d500 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:20.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.437+0000 7fd24d7f5640 1 --2- 172.21.15.57:0/2742417619 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd24815b2a0 0x7fd2480b5050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:20.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.437+0000 7fd24d7f5640 1 --2- 172.21.15.57:0/2742417619 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd2480cf340 0x7fd24815da00 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:20.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.437+0000 7fd24d7f5640 1 --2- 172.21.15.57:0/2742417619 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd2480ceb40 0x7fd2481520e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:20.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.437+0000 7fd24d7f5640 1 -- 172.21.15.57:0/2742417619 >> 172.21.15.57:0/2742417619 conn(0x7fd2480d0d60 msgr2=0x7fd2480b0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:20.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.437+0000 7fd24d7f5640 1 -- 172.21.15.57:0/2742417619 shutdown_connections 2023-10-19T12:57:20.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.437+0000 7fd24d7f5640 1 -- 172.21.15.57:0/2742417619 wait complete. 2023-10-19T12:57:20.450 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir1 2023-10-19T12:57:20.450 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir2 1m 2023-10-19T12:57:20.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.559+0000 7f5578c61640 1 -- 172.21.15.57:0/3171447325 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f55740cb980 msgr2=0x7f557413db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:20.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.559+0000 7f5578c61640 1 --2- 172.21.15.57:0/3171447325 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f55740cb980 0x7f557413db80 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f557406d4f0 tx=0x7f556402bd70 comp rx=0 tx=0).stop 2023-10-19T12:57:20.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.559+0000 7f5578c61640 1 -- 172.21.15.57:0/3171447325 shutdown_connections 2023-10-19T12:57:20.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.559+0000 7f5578c61640 1 --2- 172.21.15.57:0/3171447325 >> v2:172.21.15.179:3300/0 conn(0x7f5574070350 0x7f5574070780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:20.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.559+0000 7f5578c61640 1 --2- 172.21.15.57:0/3171447325 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f55740cb980 0x7f557413db80 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:20.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.559+0000 7f5578c61640 1 -- 172.21.15.57:0/3171447325 >> 172.21.15.57:0/3171447325 conn(0x7f5574069930 msgr2=0x7f5574069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:20.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.559+0000 7f5578c61640 1 -- 172.21.15.57:0/3171447325 shutdown_connections 2023-10-19T12:57:20.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.559+0000 7f5578c61640 1 -- 172.21.15.57:0/3171447325 wait complete. 2023-10-19T12:57:20.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.561+0000 7f5578c61640 1 Processor -- start 2023-10-19T12:57:20.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.561+0000 7f5578c61640 1 -- start start 2023-10-19T12:57:20.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.561+0000 7f5578c61640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5574070350 0x7f55741f17f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:20.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.562+0000 7f5578c61640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f55740cb980 0x7f55741f57f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:20.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.562+0000 7f5578c61640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f55740afbd0 0x7f557413c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:20.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.562+0000 7f5578c61640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55741544f0 con 0x7f5574070350 2023-10-19T12:57:20.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.562+0000 7f5578c61640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55741f5d30 con 0x7f55740afbd0 2023-10-19T12:57:20.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.562+0000 7f5578c61640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55741f1e40 con 0x7f55740cb980 2023-10-19T12:57:20.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.562+0000 7f557259c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5574070350 0x7f55741f17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:20.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.562+0000 7f557259c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5574070350 0x7f55741f17f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33962/0 (socket says 172.21.15.57:33962) 2023-10-19T12:57:20.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.562+0000 7f557259c640 1 -- 172.21.15.57:0/2786508907 learned_addr learned my addr 172.21.15.57:0/2786508907 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:57:20.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.562+0000 7f557259c640 1 -- 172.21.15.57:0/2786508907 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f55740cb980 msgr2=0x7f55741f57f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:20.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.562+0000 7f557259c640 1 --2- 172.21.15.57:0/2786508907 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f55740cb980 0x7f55741f57f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:20.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.562+0000 7f557259c640 1 -- 172.21.15.57:0/2786508907 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f55740afbd0 msgr2=0x7f557413c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:20.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.562+0000 7f5572d9d640 1 --2- 172.21.15.57:0/2786508907 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f55740afbd0 0x7f557413c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:20.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.562+0000 7f557259c640 1 --2- 172.21.15.57:0/2786508907 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f55740afbd0 0x7f557413c540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:20.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.562+0000 7f557259c640 1 -- 172.21.15.57:0/2786508907 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5564009490 con 0x7f5574070350 2023-10-19T12:57:20.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.562+0000 7f557259c640 1 --2- 172.21.15.57:0/2786508907 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5574070350 0x7f55741f17f0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f5564009920 tx=0x7f55640038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:20.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.562+0000 7f556b7fe640 1 -- 172.21.15.57:0/2786508907 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5564003ea0 con 0x7f5574070350 2023-10-19T12:57:20.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.562+0000 7f556b7fe640 1 -- 172.21.15.57:0/2786508907 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f5564034550 con 0x7f5574070350 2023-10-19T12:57:20.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.562+0000 7f5578c61640 1 -- 172.21.15.57:0/2786508907 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55741f2120 con 0x7f5574070350 2023-10-19T12:57:20.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.562+0000 7f556b7fe640 1 -- 172.21.15.57:0/2786508907 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5564034960 con 0x7f5574070350 2023-10-19T12:57:20.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.562+0000 7f5578c61640 1 -- 172.21.15.57:0/2786508907 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55740b0080 con 0x7f5574070350 2023-10-19T12:57:20.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.563+0000 7f5578c61640 1 -- 172.21.15.57:0/2786508907 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55740690e0 con 0x7f5574070350 2023-10-19T12:57:20.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.564+0000 7f556b7fe640 1 -- 172.21.15.57:0/2786508907 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 15) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f556403b040 con 0x7f5574070350 2023-10-19T12:57:20.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.564+0000 7f556b7fe640 1 --2- 172.21.15.57:0/2786508907 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f555406b040 0x7f555406d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:20.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.564+0000 7f556b7fe640 1 -- 172.21.15.57:0/2786508907 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f55640ac1b0 con 0x7f5574070350 2023-10-19T12:57:20.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.564+0000 7f5569d9b640 1 --2- 172.21.15.57:0/2786508907 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f555406b040 0x7f555406d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:20.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.565+0000 7f5569d9b640 1 --2- 172.21.15.57:0/2786508907 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f555406b040 0x7f555406d500 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f555c000f60 tx=0x7f555c007a30 comp rx=0 tx=0).ready entity=mgr.5428 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:20.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.569+0000 7f556b7fe640 1 -- 172.21.15.57:0/2786508907 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f5564076d80 con 0x7f5574070350 2023-10-19T12:57:20.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.768+0000 7f5578c61640 1 -- 172.21.15.57:0/2786508907 --> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir2", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f5574070cc0 con 0x7f555406b040 2023-10-19T12:57:20.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.828+0000 7f556b7fe640 1 -- 172.21.15.57:0/2786508907 <== mgr.5428 v2:172.21.15.179:6832/922398470 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f555c014400 con 0x7f555406b040 2023-10-19T12:57:20.832 INFO:teuthology.orchestra.run.smithi057.stdout:Schedule deactivated for path /snap_test_dir1/dir2 2023-10-19T12:57:20.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.830+0000 7f5578c61640 1 -- 172.21.15.57:0/2786508907 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f555406b040 msgr2=0x7f555406d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:20.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.830+0000 7f5578c61640 1 --2- 172.21.15.57:0/2786508907 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f555406b040 0x7f555406d500 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f555c000f60 tx=0x7f555c007a30 comp rx=0 tx=0).stop 2023-10-19T12:57:20.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.830+0000 7f5578c61640 1 -- 172.21.15.57:0/2786508907 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5574070350 msgr2=0x7f55741f17f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:20.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.830+0000 7f5578c61640 1 --2- 172.21.15.57:0/2786508907 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5574070350 0x7f55741f17f0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f5564009920 tx=0x7f55640038a0 comp rx=0 tx=0).stop 2023-10-19T12:57:20.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.831+0000 7f5578c61640 1 -- 172.21.15.57:0/2786508907 shutdown_connections 2023-10-19T12:57:20.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.831+0000 7f5578c61640 1 --2- 172.21.15.57:0/2786508907 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f555406b040 0x7f555406d500 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:20.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.831+0000 7f5578c61640 1 --2- 172.21.15.57:0/2786508907 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f55740afbd0 0x7f557413c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:20.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.831+0000 7f5578c61640 1 --2- 172.21.15.57:0/2786508907 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f55740cb980 0x7f55741f57f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:20.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.831+0000 7f5578c61640 1 --2- 172.21.15.57:0/2786508907 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5574070350 0x7f55741f17f0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:20.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.831+0000 7f5578c61640 1 -- 172.21.15.57:0/2786508907 >> 172.21.15.57:0/2786508907 conn(0x7f5574069930 msgr2=0x7f55740ca870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:20.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.831+0000 7f5578c61640 1 -- 172.21.15.57:0/2786508907 shutdown_connections 2023-10-19T12:57:20.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.831+0000 7f5578c61640 1 -- 172.21.15.57:0/2786508907 wait complete. 2023-10-19T12:57:20.844 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir2 2023-10-19T12:57:20.844 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir3 1m 2023-10-19T12:57:20.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.953+0000 7fc2e2f00640 1 -- 172.21.15.57:0/3429376060 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc2dc0ab500 msgr2=0x7fc2dc0ab990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:20.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.953+0000 7fc2e2f00640 1 --2- 172.21.15.57:0/3429376060 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc2dc0ab500 0x7fc2dc0ab990 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc0ca260 tx=0x7fc2cc02bd70 comp rx=0 tx=0).stop 2023-10-19T12:57:20.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.953+0000 7fc2e2f00640 1 -- 172.21.15.57:0/3429376060 shutdown_connections 2023-10-19T12:57:20.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.953+0000 7fc2e2f00640 1 --2- 172.21.15.57:0/3429376060 >> v2:172.21.15.179:3300/0 conn(0x7fc2dc0c9dc0 0x7fc2dc06b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:20.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.953+0000 7fc2e2f00640 1 --2- 172.21.15.57:0/3429376060 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc2dc0ab500 0x7fc2dc0ab990 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:20.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.953+0000 7fc2e2f00640 1 -- 172.21.15.57:0/3429376060 >> 172.21.15.57:0/3429376060 conn(0x7fc2dc069930 msgr2=0x7fc2dc069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:20.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.953+0000 7fc2e2f00640 1 -- 172.21.15.57:0/3429376060 shutdown_connections 2023-10-19T12:57:20.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.953+0000 7fc2e2f00640 1 -- 172.21.15.57:0/3429376060 wait complete. 2023-10-19T12:57:20.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.954+0000 7fc2e2f00640 1 Processor -- start 2023-10-19T12:57:20.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.954+0000 7fc2e2f00640 1 -- start start 2023-10-19T12:57:20.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.955+0000 7fc2e2f00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc2dc06af00 0x7fc2dc14ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:20.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.955+0000 7fc2e2f00640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc2dc0ab500 0x7fc2dc0b03d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:20.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.955+0000 7fc2e2f00640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc2dc0c9dc0 0x7fc2dc13d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:20.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.955+0000 7fc2e2f00640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2dc151da0 con 0x7fc2dc0ab500 2023-10-19T12:57:20.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.955+0000 7fc2e2f00640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2dc14f480 con 0x7fc2dc06af00 2023-10-19T12:57:20.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.955+0000 7fc2e2f00640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2dc151120 con 0x7fc2dc0c9dc0 2023-10-19T12:57:20.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.955+0000 7fc2dbfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc2dc0ab500 0x7fc2dc0b03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:20.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.955+0000 7fc2dbfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc2dc0ab500 0x7fc2dc0b03d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33994/0 (socket says 172.21.15.57:33994) 2023-10-19T12:57:20.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.955+0000 7fc2dbfff640 1 -- 172.21.15.57:0/355964944 learned_addr learned my addr 172.21.15.57:0/355964944 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:57:20.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.955+0000 7fc2e0c9c640 1 --2- 172.21.15.57:0/355964944 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc2dc06af00 0x7fc2dc14ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:20.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.955+0000 7fc2e149d640 1 --2- 172.21.15.57:0/355964944 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc2dc0c9dc0 0x7fc2dc13d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:20.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.955+0000 7fc2dbfff640 1 -- 172.21.15.57:0/355964944 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc2dc0c9dc0 msgr2=0x7fc2dc13d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:20.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.955+0000 7fc2dbfff640 1 --2- 172.21.15.57:0/355964944 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc2dc0c9dc0 0x7fc2dc13d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:20.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.956+0000 7fc2dbfff640 1 -- 172.21.15.57:0/355964944 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc2dc06af00 msgr2=0x7fc2dc14ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:20.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.956+0000 7fc2dbfff640 1 --2- 172.21.15.57:0/355964944 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc2dc06af00 0x7fc2dc14ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:20.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.956+0000 7fc2dbfff640 1 -- 172.21.15.57:0/355964944 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2cc009490 con 0x7fc2dc0ab500 2023-10-19T12:57:20.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.956+0000 7fc2e0c9c640 1 --2- 172.21.15.57:0/355964944 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc2dc06af00 0x7fc2dc14ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:57:20.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.956+0000 7fc2dbfff640 1 --2- 172.21.15.57:0/355964944 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc2dc0ab500 0x7fc2dc0b03d0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fc2c400d740 tx=0x7fc2c400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:20.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.956+0000 7fc2d97fa640 1 -- 172.21.15.57:0/355964944 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2c4009c30 con 0x7fc2dc0ab500 2023-10-19T12:57:20.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.956+0000 7fc2d97fa640 1 -- 172.21.15.57:0/355964944 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fc2c4009e80 con 0x7fc2dc0ab500 2023-10-19T12:57:20.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.956+0000 7fc2e2f00640 1 -- 172.21.15.57:0/355964944 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2dc151400 con 0x7fc2dc0ab500 2023-10-19T12:57:20.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.956+0000 7fc2d97fa640 1 -- 172.21.15.57:0/355964944 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2c40106c0 con 0x7fc2dc0ab500 2023-10-19T12:57:20.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.956+0000 7fc2e2f00640 1 -- 172.21.15.57:0/355964944 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2dc1519c0 con 0x7fc2dc0ab500 2023-10-19T12:57:20.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.957+0000 7fc2e2f00640 1 -- 172.21.15.57:0/355964944 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2dc152010 con 0x7fc2dc0ab500 2023-10-19T12:57:20.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.957+0000 7fc2d97fa640 1 -- 172.21.15.57:0/355964944 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 15) v1 ==== 89417+0+0 (secure 0 0 0) 0x7fc2c4010820 con 0x7fc2dc0ab500 2023-10-19T12:57:20.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.958+0000 7fc2d97fa640 1 --2- 172.21.15.57:0/355964944 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fc2bc06b040 0x7fc2bc06d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:20.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.958+0000 7fc2d97fa640 1 -- 172.21.15.57:0/355964944 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7fc2c408a180 con 0x7fc2dc0ab500 2023-10-19T12:57:20.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.958+0000 7fc2e0c9c640 1 --2- 172.21.15.57:0/355964944 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fc2bc06b040 0x7fc2bc06d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:20.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.958+0000 7fc2e0c9c640 1 --2- 172.21.15.57:0/355964944 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fc2bc06b040 0x7fc2bc06d500 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc2d4001560 tx=0x7fc2d4009040 comp rx=0 tx=0).ready entity=mgr.5428 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:20.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:20.962+0000 7fc2d97fa640 1 -- 172.21.15.57:0/355964944 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc2c4054cd0 con 0x7fc2dc0ab500 2023-10-19T12:57:21.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.161+0000 7fc2e2f00640 1 -- 172.21.15.57:0/355964944 --> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir3", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fc2dc0690e0 con 0x7fc2bc06b040 2023-10-19T12:57:21.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.224+0000 7fc2d97fa640 1 -- 172.21.15.57:0/355964944 <== mgr.5428 v2:172.21.15.179:6832/922398470 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fc2dc0690e0 con 0x7fc2bc06b040 2023-10-19T12:57:21.227 INFO:teuthology.orchestra.run.smithi057.stdout:Schedule deactivated for path /snap_test_dir1/dir3 2023-10-19T12:57:21.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.226+0000 7fc2e2f00640 1 -- 172.21.15.57:0/355964944 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fc2bc06b040 msgr2=0x7fc2bc06d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:21.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.226+0000 7fc2e2f00640 1 --2- 172.21.15.57:0/355964944 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fc2bc06b040 0x7fc2bc06d500 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc2d4001560 tx=0x7fc2d4009040 comp rx=0 tx=0).stop 2023-10-19T12:57:21.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.226+0000 7fc2e2f00640 1 -- 172.21.15.57:0/355964944 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc2dc0ab500 msgr2=0x7fc2dc0b03d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:21.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.226+0000 7fc2e2f00640 1 --2- 172.21.15.57:0/355964944 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc2dc0ab500 0x7fc2dc0b03d0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fc2c400d740 tx=0x7fc2c400dc00 comp rx=0 tx=0).stop 2023-10-19T12:57:21.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.226+0000 7fc2e2f00640 1 -- 172.21.15.57:0/355964944 shutdown_connections 2023-10-19T12:57:21.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.226+0000 7fc2e2f00640 1 --2- 172.21.15.57:0/355964944 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fc2bc06b040 0x7fc2bc06d500 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:21.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.226+0000 7fc2e2f00640 1 --2- 172.21.15.57:0/355964944 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc2dc0c9dc0 0x7fc2dc13d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:21.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.226+0000 7fc2e2f00640 1 --2- 172.21.15.57:0/355964944 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc2dc0ab500 0x7fc2dc0b03d0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:21.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.226+0000 7fc2e2f00640 1 --2- 172.21.15.57:0/355964944 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc2dc06af00 0x7fc2dc14ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:21.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.226+0000 7fc2e2f00640 1 -- 172.21.15.57:0/355964944 >> 172.21.15.57:0/355964944 conn(0x7fc2dc069930 msgr2=0x7fc2dc13b260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:21.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.226+0000 7fc2e2f00640 1 -- 172.21.15.57:0/355964944 shutdown_connections 2023-10-19T12:57:21.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.226+0000 7fc2e2f00640 1 -- 172.21.15.57:0/355964944 wait complete. 2023-10-19T12:57:21.239 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir3 2023-10-19T12:57:21.239 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir4 1m 2023-10-19T12:57:21.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.348+0000 7f5947039640 1 -- 172.21.15.57:0/782851921 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f594006bec0 msgr2=0x7f594006a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:21.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.348+0000 7f5947039640 1 --2- 172.21.15.57:0/782851921 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f594006bec0 0x7f594006a730 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f59400713b0 tx=0x7f593802c250 comp rx=0 tx=0).stop 2023-10-19T12:57:21.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.348+0000 7f5947039640 1 -- 172.21.15.57:0/782851921 shutdown_connections 2023-10-19T12:57:21.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.348+0000 7f5947039640 1 --2- 172.21.15.57:0/782851921 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f594006bec0 0x7f594006a730 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:21.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.348+0000 7f5947039640 1 --2- 172.21.15.57:0/782851921 >> v2:172.21.15.179:3300/0 conn(0x7f59400ac870 0x7f594006b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:21.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.348+0000 7f5947039640 1 --2- 172.21.15.57:0/782851921 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5940156790 0x7f59400cc990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:21.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.348+0000 7f5947039640 1 -- 172.21.15.57:0/782851921 >> 172.21.15.57:0/782851921 conn(0x7f594006fae0 msgr2=0x7f594006fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:21.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.348+0000 7f5947039640 1 -- 172.21.15.57:0/782851921 shutdown_connections 2023-10-19T12:57:21.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.348+0000 7f5947039640 1 -- 172.21.15.57:0/782851921 wait complete. 2023-10-19T12:57:21.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.349+0000 7f5947039640 1 Processor -- start 2023-10-19T12:57:21.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.349+0000 7f5947039640 1 -- start start 2023-10-19T12:57:21.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.350+0000 7f5947039640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f594006bec0 0x7f594015c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:21.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.350+0000 7f5947039640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f59400ac870 0x7f5940140f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:21.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.350+0000 7f5947039640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5940156790 0x7f594006cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:21.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.350+0000 7f5947039640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5940144420 con 0x7f594006bec0 2023-10-19T12:57:21.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.350+0000 7f5947039640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f594015ceb0 con 0x7f59400ac870 2023-10-19T12:57:21.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.350+0000 7f5947039640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59401414c0 con 0x7f5940156790 2023-10-19T12:57:21.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.350+0000 7f5944dd5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f594006bec0 0x7f594015c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:21.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.350+0000 7f5944dd5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f594006bec0 0x7f594015c970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34018/0 (socket says 172.21.15.57:34018) 2023-10-19T12:57:21.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.350+0000 7f5944dd5640 1 -- 172.21.15.57:0/2620314653 learned_addr learned my addr 172.21.15.57:0/2620314653 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:57:21.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.350+0000 7f59455d6640 1 --2- 172.21.15.57:0/2620314653 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5940156790 0x7f594006cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:21.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.350+0000 7f593ffff640 1 --2- 172.21.15.57:0/2620314653 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f59400ac870 0x7f5940140f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:21.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.350+0000 7f5944dd5640 1 -- 172.21.15.57:0/2620314653 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5940156790 msgr2=0x7f594006cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:21.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.350+0000 7f5944dd5640 1 --2- 172.21.15.57:0/2620314653 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5940156790 0x7f594006cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:21.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.350+0000 7f5944dd5640 1 -- 172.21.15.57:0/2620314653 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f59400ac870 msgr2=0x7f5940140f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:21.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.350+0000 7f5944dd5640 1 --2- 172.21.15.57:0/2620314653 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f59400ac870 0x7f5940140f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:21.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.350+0000 7f5944dd5640 1 -- 172.21.15.57:0/2620314653 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5938009900 con 0x7f594006bec0 2023-10-19T12:57:21.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.350+0000 7f593ffff640 1 --2- 172.21.15.57:0/2620314653 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f59400ac870 0x7f5940140f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:57:21.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.350+0000 7f59455d6640 1 --2- 172.21.15.57:0/2620314653 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5940156790 0x7f594006cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:57:21.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.351+0000 7f5944dd5640 1 --2- 172.21.15.57:0/2620314653 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f594006bec0 0x7f594015c970 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f593000d950 tx=0x7f593000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:21.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.351+0000 7f593d7fa640 1 -- 172.21.15.57:0/2620314653 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5930009bd0 con 0x7f594006bec0 2023-10-19T12:57:21.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.351+0000 7f593d7fa640 1 -- 172.21.15.57:0/2620314653 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f5930009e20 con 0x7f594006bec0 2023-10-19T12:57:21.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.351+0000 7f5947039640 1 -- 172.21.15.57:0/2620314653 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f594015a1c0 con 0x7f594006bec0 2023-10-19T12:57:21.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.351+0000 7f593d7fa640 1 -- 172.21.15.57:0/2620314653 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59300106c0 con 0x7f594006bec0 2023-10-19T12:57:21.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.351+0000 7f5947039640 1 -- 172.21.15.57:0/2620314653 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f594015a780 con 0x7f594006bec0 2023-10-19T12:57:21.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.352+0000 7f5947039640 1 -- 172.21.15.57:0/2620314653 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59400690e0 con 0x7f594006bec0 2023-10-19T12:57:21.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.352+0000 7f593d7fa640 1 -- 172.21.15.57:0/2620314653 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 15) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f5930010820 con 0x7f594006bec0 2023-10-19T12:57:21.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.353+0000 7f593d7fa640 1 --2- 172.21.15.57:0/2620314653 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f592c06b040 0x7f592c06d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:21.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.353+0000 7f593d7fa640 1 -- 172.21.15.57:0/2620314653 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f593008a1f0 con 0x7f594006bec0 2023-10-19T12:57:21.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.353+0000 7f593ffff640 1 --2- 172.21.15.57:0/2620314653 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f592c06b040 0x7f592c06d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:21.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.353+0000 7f593ffff640 1 --2- 172.21.15.57:0/2620314653 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f592c06b040 0x7f592c06d500 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5934005fd0 tx=0x7f59340094d0 comp rx=0 tx=0).ready entity=mgr.5428 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:21.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.357+0000 7f593d7fa640 1 -- 172.21.15.57:0/2620314653 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f5930054c20 con 0x7f594006bec0 2023-10-19T12:57:21.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.553+0000 7f5947039640 1 -- 172.21.15.57:0/2620314653 --> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir4", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f59401596e0 con 0x7f592c06b040 2023-10-19T12:57:21.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.616+0000 7f593d7fa640 1 -- 172.21.15.57:0/2620314653 <== mgr.5428 v2:172.21.15.179:6832/922398470 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f59340096e0 con 0x7f592c06b040 2023-10-19T12:57:21.619 INFO:teuthology.orchestra.run.smithi057.stdout:Schedule deactivated for path /snap_test_dir1/dir4 2023-10-19T12:57:21.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.618+0000 7f5947039640 1 -- 172.21.15.57:0/2620314653 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f592c06b040 msgr2=0x7f592c06d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:21.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.618+0000 7f5947039640 1 --2- 172.21.15.57:0/2620314653 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f592c06b040 0x7f592c06d500 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5934005fd0 tx=0x7f59340094d0 comp rx=0 tx=0).stop 2023-10-19T12:57:21.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.618+0000 7f5947039640 1 -- 172.21.15.57:0/2620314653 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f594006bec0 msgr2=0x7f594015c970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:21.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.618+0000 7f5947039640 1 --2- 172.21.15.57:0/2620314653 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f594006bec0 0x7f594015c970 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f593000d950 tx=0x7f593000de10 comp rx=0 tx=0).stop 2023-10-19T12:57:21.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.618+0000 7f5947039640 1 -- 172.21.15.57:0/2620314653 shutdown_connections 2023-10-19T12:57:21.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.618+0000 7f5947039640 1 --2- 172.21.15.57:0/2620314653 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f592c06b040 0x7f592c06d500 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:21.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.618+0000 7f5947039640 1 --2- 172.21.15.57:0/2620314653 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5940156790 0x7f594006cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:21.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.618+0000 7f5947039640 1 --2- 172.21.15.57:0/2620314653 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f59400ac870 0x7f5940140f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:21.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.618+0000 7f5947039640 1 --2- 172.21.15.57:0/2620314653 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f594006bec0 0x7f594015c970 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:21.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.618+0000 7f5947039640 1 -- 172.21.15.57:0/2620314653 >> 172.21.15.57:0/2620314653 conn(0x7f594006fae0 msgr2=0x7f59400c9670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:21.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.618+0000 7f5947039640 1 -- 172.21.15.57:0/2620314653 shutdown_connections 2023-10-19T12:57:21.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.618+0000 7f5947039640 1 -- 172.21.15.57:0/2620314653 wait complete. 2023-10-19T12:57:21.631 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir4 2023-10-19T12:57:21.631 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir5 1m 2023-10-19T12:57:21.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.740+0000 7f9c13797640 1 -- 172.21.15.57:0/1492321942 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9c0c15b3e0 msgr2=0x7f9c0c0b0c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:21.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.740+0000 7f9c13797640 1 --2- 172.21.15.57:0/1492321942 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9c0c15b3e0 0x7f9c0c0b0c10 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f9c0c0b37f0 tx=0x7f9bfc02c2f0 comp rx=0 tx=0).stop 2023-10-19T12:57:21.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.740+0000 7f9c13797640 1 -- 172.21.15.57:0/1492321942 shutdown_connections 2023-10-19T12:57:21.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.740+0000 7f9c13797640 1 --2- 172.21.15.57:0/1492321942 >> v2:172.21.15.179:3300/0 conn(0x7f9c0c0cf9e0 0x7f9c0c0d1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:21.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.740+0000 7f9c13797640 1 --2- 172.21.15.57:0/1492321942 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9c0c15b3e0 0x7f9c0c0b0c10 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:21.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.740+0000 7f9c13797640 1 -- 172.21.15.57:0/1492321942 >> 172.21.15.57:0/1492321942 conn(0x7f9c0c0d0cf0 msgr2=0x7f9c0c0d1100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:21.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.740+0000 7f9c13797640 1 -- 172.21.15.57:0/1492321942 shutdown_connections 2023-10-19T12:57:21.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.740+0000 7f9c13797640 1 -- 172.21.15.57:0/1492321942 wait complete. 2023-10-19T12:57:21.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.741+0000 7f9c13797640 1 Processor -- start 2023-10-19T12:57:21.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.741+0000 7f9c13797640 1 -- start start 2023-10-19T12:57:21.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.742+0000 7f9c13797640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9c0c0cf9e0 0x7f9c0c0c50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:21.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.742+0000 7f9c13797640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9c0c15b000 0x7f9c0c0c5630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:21.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.742+0000 7f9c13797640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9c0c15b3e0 0x7f9c0c1f2f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:21.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.742+0000 7f9c13797640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c0c0c69a0 con 0x7f9c0c15b3e0 2023-10-19T12:57:21.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.742+0000 7f9c13797640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c0c0c5c60 con 0x7f9c0c0cf9e0 2023-10-19T12:57:21.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.742+0000 7f9c13797640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c0c0c5f40 con 0x7f9c0c15b000 2023-10-19T12:57:21.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.742+0000 7f9c11d34640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9c0c15b3e0 0x7f9c0c1f2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:21.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.742+0000 7f9c11d34640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9c0c15b3e0 0x7f9c0c1f2f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34044/0 (socket says 172.21.15.57:34044) 2023-10-19T12:57:21.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.742+0000 7f9c11d34640 1 -- 172.21.15.57:0/2679145780 learned_addr learned my addr 172.21.15.57:0/2679145780 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:57:21.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.742+0000 7f9c11533640 1 --2- 172.21.15.57:0/2679145780 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9c0c0cf9e0 0x7f9c0c0c50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:21.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.742+0000 7f9c10d32640 1 --2- 172.21.15.57:0/2679145780 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9c0c15b000 0x7f9c0c0c5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:21.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.742+0000 7f9c11d34640 1 -- 172.21.15.57:0/2679145780 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9c0c15b000 msgr2=0x7f9c0c0c5630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:21.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.743+0000 7f9c11d34640 1 --2- 172.21.15.57:0/2679145780 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9c0c15b000 0x7f9c0c0c5630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:21.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.743+0000 7f9c11d34640 1 -- 172.21.15.57:0/2679145780 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9c0c0cf9e0 msgr2=0x7f9c0c0c50f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:21.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.743+0000 7f9c11d34640 1 --2- 172.21.15.57:0/2679145780 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9c0c0cf9e0 0x7f9c0c0c50f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:21.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.743+0000 7f9c11d34640 1 -- 172.21.15.57:0/2679145780 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bfc009900 con 0x7f9c0c15b3e0 2023-10-19T12:57:21.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.743+0000 7f9c11533640 1 --2- 172.21.15.57:0/2679145780 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9c0c0cf9e0 0x7f9c0c0c50f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:57:21.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.743+0000 7f9c10d32640 1 --2- 172.21.15.57:0/2679145780 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9c0c15b000 0x7f9c0c0c5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:57:21.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.743+0000 7f9c11d34640 1 --2- 172.21.15.57:0/2679145780 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9c0c15b3e0 0x7f9c0c1f2f60 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f9bfc009d60 tx=0x7f9bfc035ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:21.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.743+0000 7f9c09ffb640 1 -- 172.21.15.57:0/2679145780 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bfc004250 con 0x7f9c0c15b3e0 2023-10-19T12:57:21.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.743+0000 7f9c09ffb640 1 -- 172.21.15.57:0/2679145780 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f9bfc0044a0 con 0x7f9c0c15b3e0 2023-10-19T12:57:21.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.743+0000 7f9c13797640 1 -- 172.21.15.57:0/2679145780 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c0c1f24a0 con 0x7f9c0c15b3e0 2023-10-19T12:57:21.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.743+0000 7f9c09ffb640 1 -- 172.21.15.57:0/2679145780 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bfc040450 con 0x7f9c0c15b3e0 2023-10-19T12:57:21.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.743+0000 7f9c13797640 1 -- 172.21.15.57:0/2679145780 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c0c1f2a60 con 0x7f9c0c15b3e0 2023-10-19T12:57:21.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.744+0000 7f9c13797640 1 -- 172.21.15.57:0/2679145780 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c0c0690e0 con 0x7f9c0c15b3e0 2023-10-19T12:57:21.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.745+0000 7f9c09ffb640 1 -- 172.21.15.57:0/2679145780 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 15) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f9bfc00a040 con 0x7f9c0c15b3e0 2023-10-19T12:57:21.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.745+0000 7f9c09ffb640 1 --2- 172.21.15.57:0/2679145780 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f9bf806b040 0x7f9bf806d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:21.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.745+0000 7f9c09ffb640 1 -- 172.21.15.57:0/2679145780 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f9bfc0acf70 con 0x7f9c0c15b3e0 2023-10-19T12:57:21.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.745+0000 7f9c11533640 1 --2- 172.21.15.57:0/2679145780 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f9bf806b040 0x7f9bf806d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:21.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.746+0000 7f9c11533640 1 --2- 172.21.15.57:0/2679145780 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f9bf806b040 0x7f9bf806d500 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9c04000f60 tx=0x7f9c04009040 comp rx=0 tx=0).ready entity=mgr.5428 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:21.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.749+0000 7f9c09ffb640 1 -- 172.21.15.57:0/2679145780 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9bfc0dfca0 con 0x7f9c0c15b3e0 2023-10-19T12:57:21.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:21.946+0000 7f9c13797640 1 -- 172.21.15.57:0/2679145780 --> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir5", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f9c0c0b0140 con 0x7f9bf806b040 2023-10-19T12:57:22.008 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.005+0000 7f9c09ffb640 1 -- 172.21.15.57:0/2679145780 <== mgr.5428 v2:172.21.15.179:6832/922398470 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f9c04001d50 con 0x7f9bf806b040 2023-10-19T12:57:22.009 INFO:teuthology.orchestra.run.smithi057.stdout:Schedule deactivated for path /snap_test_dir1/dir5 2023-10-19T12:57:22.011 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.007+0000 7f9c13797640 1 -- 172.21.15.57:0/2679145780 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f9bf806b040 msgr2=0x7f9bf806d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:22.011 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.007+0000 7f9c13797640 1 --2- 172.21.15.57:0/2679145780 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f9bf806b040 0x7f9bf806d500 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9c04000f60 tx=0x7f9c04009040 comp rx=0 tx=0).stop 2023-10-19T12:57:22.011 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.007+0000 7f9c13797640 1 -- 172.21.15.57:0/2679145780 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9c0c15b3e0 msgr2=0x7f9c0c1f2f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:22.011 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.007+0000 7f9c13797640 1 --2- 172.21.15.57:0/2679145780 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9c0c15b3e0 0x7f9c0c1f2f60 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f9bfc009d60 tx=0x7f9bfc035ed0 comp rx=0 tx=0).stop 2023-10-19T12:57:22.011 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.007+0000 7f9c13797640 1 -- 172.21.15.57:0/2679145780 shutdown_connections 2023-10-19T12:57:22.012 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.007+0000 7f9c13797640 1 --2- 172.21.15.57:0/2679145780 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f9bf806b040 0x7f9bf806d500 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:22.012 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.007+0000 7f9c13797640 1 --2- 172.21.15.57:0/2679145780 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9c0c15b3e0 0x7f9c0c1f2f60 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:22.012 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.007+0000 7f9c13797640 1 --2- 172.21.15.57:0/2679145780 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9c0c15b000 0x7f9c0c0c5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:22.012 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.007+0000 7f9c13797640 1 --2- 172.21.15.57:0/2679145780 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9c0c0cf9e0 0x7f9c0c0c50f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:22.012 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.007+0000 7f9c13797640 1 -- 172.21.15.57:0/2679145780 >> 172.21.15.57:0/2679145780 conn(0x7f9c0c0d0cf0 msgr2=0x7f9c0c15a240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:22.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.008+0000 7f9c13797640 1 -- 172.21.15.57:0/2679145780 shutdown_connections 2023-10-19T12:57:22.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.008+0000 7f9c13797640 1 -- 172.21.15.57:0/2679145780 wait complete. 2023-10-19T12:57:22.021 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir5 2023-10-19T12:57:22.021 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir6 1m 2023-10-19T12:57:22.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.130+0000 7f466a6c2640 1 -- 172.21.15.57:0/2584163883 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f46640b0430 msgr2=0x7f46640b0840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:22.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.130+0000 7f466a6c2640 1 --2- 172.21.15.57:0/2584163883 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f46640b0430 0x7f46640b0840 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f466406f1a0 tx=0x7f465c02c080 comp rx=0 tx=0).stop 2023-10-19T12:57:22.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.130+0000 7f466a6c2640 1 -- 172.21.15.57:0/2584163883 shutdown_connections 2023-10-19T12:57:22.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.130+0000 7f466a6c2640 1 --2- 172.21.15.57:0/2584163883 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f46640b0430 0x7f46640b0840 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:22.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.130+0000 7f466a6c2640 1 --2- 172.21.15.57:0/2584163883 >> v2:172.21.15.179:3300/0 conn(0x7f46640cbc80 0x7f4664158310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:22.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.130+0000 7f466a6c2640 1 -- 172.21.15.57:0/2584163883 >> 172.21.15.57:0/2584163883 conn(0x7f4664069930 msgr2=0x7f4664069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:22.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.131+0000 7f466a6c2640 1 -- 172.21.15.57:0/2584163883 shutdown_connections 2023-10-19T12:57:22.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.131+0000 7f466a6c2640 1 -- 172.21.15.57:0/2584163883 wait complete. 2023-10-19T12:57:22.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.132+0000 7f466a6c2640 1 Processor -- start 2023-10-19T12:57:22.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.132+0000 7f466a6c2640 1 -- start start 2023-10-19T12:57:22.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.133+0000 7f466a6c2640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f466406d020 0x7f466414bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:22.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.133+0000 7f466a6c2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f46640b0430 0x7f466413d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:22.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.133+0000 7f466a6c2640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f46640cbc80 0x7f466415a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:22.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.133+0000 7f466a6c2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4664158e40 con 0x7f46640b0430 2023-10-19T12:57:22.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.133+0000 7f466a6c2640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4664159120 con 0x7f46640cbc80 2023-10-19T12:57:22.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.133+0000 7f466a6c2640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4664159400 con 0x7f466406d020 2023-10-19T12:57:22.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.133+0000 7f46637fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f46640b0430 0x7f466413d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:22.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.133+0000 7f46637fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f46640b0430 0x7f466413d1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34064/0 (socket says 172.21.15.57:34064) 2023-10-19T12:57:22.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.133+0000 7f46637fe640 1 -- 172.21.15.57:0/1901277258 learned_addr learned my addr 172.21.15.57:0/1901277258 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:57:22.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.133+0000 7f4663fff640 1 --2- 172.21.15.57:0/1901277258 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f466406d020 0x7f466414bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:22.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.133+0000 7f4668c5f640 1 --2- 172.21.15.57:0/1901277258 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f46640cbc80 0x7f466415a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:22.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.133+0000 7f46637fe640 1 -- 172.21.15.57:0/1901277258 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f466406d020 msgr2=0x7f466414bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:22.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.133+0000 7f46637fe640 1 --2- 172.21.15.57:0/1901277258 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f466406d020 0x7f466414bb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:22.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.133+0000 7f46637fe640 1 -- 172.21.15.57:0/1901277258 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f46640cbc80 msgr2=0x7f466415a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:22.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.133+0000 7f46637fe640 1 --2- 172.21.15.57:0/1901277258 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f46640cbc80 0x7f466415a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:22.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.133+0000 7f46637fe640 1 -- 172.21.15.57:0/1901277258 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f465c009840 con 0x7f46640b0430 2023-10-19T12:57:22.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.133+0000 7f46637fe640 1 --2- 172.21.15.57:0/1901277258 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f46640b0430 0x7f466413d1d0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f465800c400 tx=0x7f465800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:22.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.133+0000 7f4660ff9640 1 -- 172.21.15.57:0/1901277258 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4658004230 con 0x7f46640b0430 2023-10-19T12:57:22.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.133+0000 7f4660ff9640 1 -- 172.21.15.57:0/1901277258 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f46580044a0 con 0x7f46640b0430 2023-10-19T12:57:22.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.133+0000 7f466a6c2640 1 -- 172.21.15.57:0/1901277258 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46641596e0 con 0x7f46640b0430 2023-10-19T12:57:22.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.133+0000 7f4660ff9640 1 -- 172.21.15.57:0/1901277258 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4658003850 con 0x7f46640b0430 2023-10-19T12:57:22.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.133+0000 7f466a6c2640 1 -- 172.21.15.57:0/1901277258 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4664159ca0 con 0x7f46640b0430 2023-10-19T12:57:22.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.134+0000 7f466a6c2640 1 -- 172.21.15.57:0/1901277258 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4664159e90 con 0x7f46640b0430 2023-10-19T12:57:22.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.135+0000 7f4660ff9640 1 -- 172.21.15.57:0/1901277258 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 15) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f46580039b0 con 0x7f46640b0430 2023-10-19T12:57:22.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.135+0000 7f4660ff9640 1 --2- 172.21.15.57:0/1901277258 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f465006b040 0x7f465006d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:22.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.135+0000 7f4660ff9640 1 -- 172.21.15.57:0/1901277258 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f46580892b0 con 0x7f46640b0430 2023-10-19T12:57:22.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.135+0000 7f4663fff640 1 --2- 172.21.15.57:0/1901277258 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f465006b040 0x7f465006d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:22.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.136+0000 7f4663fff640 1 --2- 172.21.15.57:0/1901277258 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f465006b040 0x7f465006d500 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f465400bc50 tx=0x7f465400b3e0 comp rx=0 tx=0).ready entity=mgr.5428 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:22.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.139+0000 7f4660ff9640 1 -- 172.21.15.57:0/1901277258 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4658053ff0 con 0x7f46640b0430 2023-10-19T12:57:22.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.339+0000 7f466a6c2640 1 -- 172.21.15.57:0/1901277258 --> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir6", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f46640c96f0 con 0x7f465006b040 2023-10-19T12:57:22.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.410+0000 7f4660ff9640 1 -- 172.21.15.57:0/1901277258 <== mgr.5428 v2:172.21.15.179:6832/922398470 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f465400bc80 con 0x7f465006b040 2023-10-19T12:57:22.413 INFO:teuthology.orchestra.run.smithi057.stdout:Schedule deactivated for path /snap_test_dir1/dir6 2023-10-19T12:57:22.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.412+0000 7f466a6c2640 1 -- 172.21.15.57:0/1901277258 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f465006b040 msgr2=0x7f465006d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:22.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.412+0000 7f466a6c2640 1 --2- 172.21.15.57:0/1901277258 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f465006b040 0x7f465006d500 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f465400bc50 tx=0x7f465400b3e0 comp rx=0 tx=0).stop 2023-10-19T12:57:22.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.412+0000 7f466a6c2640 1 -- 172.21.15.57:0/1901277258 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f46640b0430 msgr2=0x7f466413d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:22.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.412+0000 7f466a6c2640 1 --2- 172.21.15.57:0/1901277258 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f46640b0430 0x7f466413d1d0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f465800c400 tx=0x7f465800c8c0 comp rx=0 tx=0).stop 2023-10-19T12:57:22.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.412+0000 7f466a6c2640 1 -- 172.21.15.57:0/1901277258 shutdown_connections 2023-10-19T12:57:22.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.412+0000 7f466a6c2640 1 --2- 172.21.15.57:0/1901277258 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f465006b040 0x7f465006d500 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:22.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.412+0000 7f466a6c2640 1 --2- 172.21.15.57:0/1901277258 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f46640cbc80 0x7f466415a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:22.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.412+0000 7f466a6c2640 1 --2- 172.21.15.57:0/1901277258 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f46640b0430 0x7f466413d1d0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:22.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.412+0000 7f466a6c2640 1 --2- 172.21.15.57:0/1901277258 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f466406d020 0x7f466414bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:22.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.412+0000 7f466a6c2640 1 -- 172.21.15.57:0/1901277258 >> 172.21.15.57:0/1901277258 conn(0x7f4664069930 msgr2=0x7f46640c8bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:22.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.412+0000 7f466a6c2640 1 -- 172.21.15.57:0/1901277258 shutdown_connections 2023-10-19T12:57:22.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.412+0000 7f466a6c2640 1 -- 172.21.15.57:0/1901277258 wait complete. 2023-10-19T12:57:22.425 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir6 2023-10-19T12:57:22.426 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir7 1m 2023-10-19T12:57:22.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.535+0000 7f0c3555e640 1 -- 172.21.15.57:0/1239831411 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0c3006a730 msgr2=0x7f0c3006abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:22.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.535+0000 7f0c3555e640 1 --2- 172.21.15.57:0/1239831411 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0c3006a730 0x7f0c3006abc0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f0c28005fd0 tx=0x7f0c2802bed0 comp rx=0 tx=0).stop 2023-10-19T12:57:22.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.535+0000 7f0c3555e640 1 -- 172.21.15.57:0/1239831411 shutdown_connections 2023-10-19T12:57:22.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.535+0000 7f0c3555e640 1 --2- 172.21.15.57:0/1239831411 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0c3006a730 0x7f0c3006abc0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:22.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.535+0000 7f0c3555e640 1 -- 172.21.15.57:0/1239831411 >> 172.21.15.57:0/1239831411 conn(0x7f0c3006fae0 msgr2=0x7f0c3006fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:22.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.535+0000 7f0c3555e640 1 -- 172.21.15.57:0/1239831411 shutdown_connections 2023-10-19T12:57:22.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.535+0000 7f0c3555e640 1 -- 172.21.15.57:0/1239831411 wait complete. 2023-10-19T12:57:22.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.536+0000 7f0c3555e640 1 Processor -- start 2023-10-19T12:57:22.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.536+0000 7f0c3555e640 1 -- start start 2023-10-19T12:57:22.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.537+0000 7f0c3555e640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0c3006a730 0x7f0c30148d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:22.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.537+0000 7f0c3555e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0c300ac870 0x7f0c3006cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:22.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.537+0000 7f0c3555e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0c30156790 0x7f0c301f45a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:22.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.537+0000 7f0c3555e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c3006c810 con 0x7f0c300ac870 2023-10-19T12:57:22.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.537+0000 7f0c3555e640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c301f4ff0 con 0x7f0c30156790 2023-10-19T12:57:22.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.537+0000 7f0c3555e640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c301f52d0 con 0x7f0c3006a730 2023-10-19T12:57:22.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.537+0000 7f0c2e7fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0c300ac870 0x7f0c3006cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:22.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.537+0000 7f0c2e7fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0c300ac870 0x7f0c3006cc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34078/0 (socket says 172.21.15.57:34078) 2023-10-19T12:57:22.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.537+0000 7f0c2e7fc640 1 -- 172.21.15.57:0/997547221 learned_addr learned my addr 172.21.15.57:0/997547221 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:57:22.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.537+0000 7f0c2effd640 1 --2- 172.21.15.57:0/997547221 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0c3006a730 0x7f0c30148d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:22.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.537+0000 7f0c2f7fe640 1 --2- 172.21.15.57:0/997547221 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0c30156790 0x7f0c301f45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:22.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.537+0000 7f0c2e7fc640 1 -- 172.21.15.57:0/997547221 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0c3006a730 msgr2=0x7f0c30148d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:22.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.537+0000 7f0c2e7fc640 1 --2- 172.21.15.57:0/997547221 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0c3006a730 0x7f0c30148d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:22.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.537+0000 7f0c2e7fc640 1 -- 172.21.15.57:0/997547221 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0c30156790 msgr2=0x7f0c301f45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:22.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.537+0000 7f0c2e7fc640 1 --2- 172.21.15.57:0/997547221 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0c30156790 0x7f0c301f45a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:22.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.537+0000 7f0c2e7fc640 1 -- 172.21.15.57:0/997547221 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c28009560 con 0x7f0c300ac870 2023-10-19T12:57:22.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.537+0000 7f0c2effd640 1 --2- 172.21.15.57:0/997547221 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0c3006a730 0x7f0c30148d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:57:22.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.537+0000 7f0c2e7fc640 1 --2- 172.21.15.57:0/997547221 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0c300ac870 0x7f0c3006cc30 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f0c1800c450 tx=0x7f0c1800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:22.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.538+0000 7f0c277fe640 1 -- 172.21.15.57:0/997547221 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c180052d0 con 0x7f0c300ac870 2023-10-19T12:57:22.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.538+0000 7f0c277fe640 1 -- 172.21.15.57:0/997547221 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f0c18005430 con 0x7f0c300ac870 2023-10-19T12:57:22.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.538+0000 7f0c3555e640 1 -- 172.21.15.57:0/997547221 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c3006c430 con 0x7f0c300ac870 2023-10-19T12:57:22.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.538+0000 7f0c277fe640 1 -- 172.21.15.57:0/997547221 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c18012890 con 0x7f0c300ac870 2023-10-19T12:57:22.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.538+0000 7f0c3555e640 1 -- 172.21.15.57:0/997547221 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c3006d170 con 0x7f0c300ac870 2023-10-19T12:57:22.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.539+0000 7f0c3555e640 1 -- 172.21.15.57:0/997547221 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c301f5510 con 0x7f0c300ac870 2023-10-19T12:57:22.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.539+0000 7f0c277fe640 1 -- 172.21.15.57:0/997547221 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 15) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f0c180035f0 con 0x7f0c300ac870 2023-10-19T12:57:22.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.540+0000 7f0c277fe640 1 --2- 172.21.15.57:0/997547221 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f0c1006b040 0x7f0c1006d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:22.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.540+0000 7f0c277fe640 1 -- 172.21.15.57:0/997547221 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f0c180898b0 con 0x7f0c300ac870 2023-10-19T12:57:22.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.540+0000 7f0c2effd640 1 --2- 172.21.15.57:0/997547221 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f0c1006b040 0x7f0c1006d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:22.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.540+0000 7f0c2effd640 1 --2- 172.21.15.57:0/997547221 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f0c1006b040 0x7f0c1006d500 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0c301547b0 tx=0x7f0c20009040 comp rx=0 tx=0).ready entity=mgr.5428 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:22.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.544+0000 7f0c277fe640 1 -- 172.21.15.57:0/997547221 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0c18054480 con 0x7f0c300ac870 2023-10-19T12:57:22.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.742+0000 7f0c3555e640 1 -- 172.21.15.57:0/997547221 --> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir7", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f0c3006c280 con 0x7f0c1006b040 2023-10-19T12:57:22.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.826+0000 7f0c277fe640 1 -- 172.21.15.57:0/997547221 <== mgr.5428 v2:172.21.15.179:6832/922398470 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f0c3006c280 con 0x7f0c1006b040 2023-10-19T12:57:22.829 INFO:teuthology.orchestra.run.smithi057.stdout:Schedule deactivated for path /snap_test_dir1/dir7 2023-10-19T12:57:22.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.828+0000 7f0c3555e640 1 -- 172.21.15.57:0/997547221 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f0c1006b040 msgr2=0x7f0c1006d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:22.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.828+0000 7f0c3555e640 1 --2- 172.21.15.57:0/997547221 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f0c1006b040 0x7f0c1006d500 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0c301547b0 tx=0x7f0c20009040 comp rx=0 tx=0).stop 2023-10-19T12:57:22.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.828+0000 7f0c3555e640 1 -- 172.21.15.57:0/997547221 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0c300ac870 msgr2=0x7f0c3006cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:22.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.828+0000 7f0c3555e640 1 --2- 172.21.15.57:0/997547221 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0c300ac870 0x7f0c3006cc30 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f0c1800c450 tx=0x7f0c1800c910 comp rx=0 tx=0).stop 2023-10-19T12:57:22.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.828+0000 7f0c3555e640 1 -- 172.21.15.57:0/997547221 shutdown_connections 2023-10-19T12:57:22.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.828+0000 7f0c3555e640 1 --2- 172.21.15.57:0/997547221 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f0c1006b040 0x7f0c1006d500 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:22.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.828+0000 7f0c3555e640 1 --2- 172.21.15.57:0/997547221 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0c30156790 0x7f0c301f45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:22.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.828+0000 7f0c3555e640 1 --2- 172.21.15.57:0/997547221 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0c300ac870 0x7f0c3006cc30 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:22.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.828+0000 7f0c3555e640 1 --2- 172.21.15.57:0/997547221 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0c3006a730 0x7f0c30148d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:22.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.828+0000 7f0c3555e640 1 -- 172.21.15.57:0/997547221 >> 172.21.15.57:0/997547221 conn(0x7f0c3006fae0 msgr2=0x7f0c300c9670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:22.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.828+0000 7f0c3555e640 1 -- 172.21.15.57:0/997547221 shutdown_connections 2023-10-19T12:57:22.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.828+0000 7f0c3555e640 1 -- 172.21.15.57:0/997547221 wait complete. 2023-10-19T12:57:22.841 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir7 2023-10-19T12:57:22.842 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir8 1m 2023-10-19T12:57:22.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.953+0000 7f1dd8cc9640 1 -- 172.21.15.57:0/702172216 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1dc0005180 msgr2=0x7f1dc000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:22.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.953+0000 7f1dd8cc9640 1 --2- 172.21.15.57:0/702172216 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1dc0005180 0x7f1dc000b980 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f1dc400c640 tx=0x7f1dc00057f0 comp rx=0 tx=0).stop 2023-10-19T12:57:22.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.953+0000 7f1dd8cc9640 1 -- 172.21.15.57:0/702172216 shutdown_connections 2023-10-19T12:57:22.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.953+0000 7f1dd8cc9640 1 --2- 172.21.15.57:0/702172216 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1dc0005180 0x7f1dc000b980 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:22.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.953+0000 7f1dd8cc9640 1 -- 172.21.15.57:0/702172216 >> 172.21.15.57:0/702172216 conn(0x7f1dd40b0670 msgr2=0x7f1dd40b0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:22.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.953+0000 7f1dd8cc9640 1 -- 172.21.15.57:0/702172216 shutdown_connections 2023-10-19T12:57:22.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.953+0000 7f1dd8cc9640 1 -- 172.21.15.57:0/702172216 wait complete. 2023-10-19T12:57:22.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.954+0000 7f1dd8cc9640 1 Processor -- start 2023-10-19T12:57:22.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.954+0000 7f1dd8cc9640 1 -- start start 2023-10-19T12:57:22.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.955+0000 7f1dd8cc9640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1dc0005180 0x7f1dd40b1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:22.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.955+0000 7f1dd8cc9640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1dd40b1ca0 0x7f1dd40aef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:22.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.955+0000 7f1dd8cc9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1dd40af4b0 0x7f1dd406b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:22.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.955+0000 7f1dd8cc9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd406b550 con 0x7f1dd40af4b0 2023-10-19T12:57:22.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.955+0000 7f1dd8cc9640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd40c1840 con 0x7f1dd40b1ca0 2023-10-19T12:57:22.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.955+0000 7f1dd8cc9640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd406a940 con 0x7f1dc0005180 2023-10-19T12:57:22.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.955+0000 7f1dd2d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1dd40af4b0 0x7f1dd406b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:22.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.955+0000 7f1dd2d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1dd40af4b0 0x7f1dd406b9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34096/0 (socket says 172.21.15.57:34096) 2023-10-19T12:57:22.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.955+0000 7f1dd2d9d640 1 -- 172.21.15.57:0/2090464811 learned_addr learned my addr 172.21.15.57:0/2090464811 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:57:22.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.955+0000 7f1dd1d9b640 1 --2- 172.21.15.57:0/2090464811 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1dd40b1ca0 0x7f1dd40aef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:22.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.955+0000 7f1dd259c640 1 --2- 172.21.15.57:0/2090464811 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1dc0005180 0x7f1dd40b1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:22.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.955+0000 7f1dd2d9d640 1 -- 172.21.15.57:0/2090464811 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1dc0005180 msgr2=0x7f1dd40b1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:22.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.955+0000 7f1dd2d9d640 1 --2- 172.21.15.57:0/2090464811 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1dc0005180 0x7f1dd40b1760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:22.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.955+0000 7f1dd2d9d640 1 -- 172.21.15.57:0/2090464811 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1dd40b1ca0 msgr2=0x7f1dd40aef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:22.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.955+0000 7f1dd2d9d640 1 --2- 172.21.15.57:0/2090464811 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1dd40b1ca0 0x7f1dd40aef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:22.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.955+0000 7f1dd2d9d640 1 -- 172.21.15.57:0/2090464811 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dd40cd310 con 0x7f1dd40af4b0 2023-10-19T12:57:22.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.955+0000 7f1dd1d9b640 1 --2- 172.21.15.57:0/2090464811 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1dd40b1ca0 0x7f1dd40aef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:57:22.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.955+0000 7f1dd2d9d640 1 --2- 172.21.15.57:0/2090464811 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1dd40af4b0 0x7f1dd406b9e0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f1dc400c760 tx=0x7f1dc4004a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:22.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.955+0000 7f1dcaffd640 1 -- 172.21.15.57:0/2090464811 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dc40017b0 con 0x7f1dd40af4b0 2023-10-19T12:57:22.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.955+0000 7f1dd8cc9640 1 -- 172.21.15.57:0/2090464811 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dd406ac20 con 0x7f1dd40af4b0 2023-10-19T12:57:22.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.955+0000 7f1dcaffd640 1 -- 172.21.15.57:0/2090464811 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f1dc4001930 con 0x7f1dd40af4b0 2023-10-19T12:57:22.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.955+0000 7f1dcaffd640 1 -- 172.21.15.57:0/2090464811 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dc4006030 con 0x7f1dd40af4b0 2023-10-19T12:57:22.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.956+0000 7f1dd8cc9640 1 -- 172.21.15.57:0/2090464811 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dd406b1e0 con 0x7f1dd40af4b0 2023-10-19T12:57:22.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.956+0000 7f1dd8cc9640 1 -- 172.21.15.57:0/2090464811 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1dd40ce1d0 con 0x7f1dd40af4b0 2023-10-19T12:57:22.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.957+0000 7f1dcaffd640 1 -- 172.21.15.57:0/2090464811 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 15) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f1dc4031030 con 0x7f1dd40af4b0 2023-10-19T12:57:22.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.957+0000 7f1dcaffd640 1 --2- 172.21.15.57:0/2090464811 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f1db406b040 0x7f1db406d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:22.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.958+0000 7f1dcaffd640 1 -- 172.21.15.57:0/2090464811 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f1dc40aab80 con 0x7f1dd40af4b0 2023-10-19T12:57:22.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.958+0000 7f1dd259c640 1 --2- 172.21.15.57:0/2090464811 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f1db406b040 0x7f1db406d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:22.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.958+0000 7f1dd259c640 1 --2- 172.21.15.57:0/2090464811 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f1db406b040 0x7f1db406d500 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1dcc009a10 tx=0x7f1dcc005f90 comp rx=0 tx=0).ready entity=mgr.5428 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:22.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:22.962+0000 7f1dcaffd640 1 -- 172.21.15.57:0/2090464811 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f1dc4075670 con 0x7f1dd40af4b0 2023-10-19T12:57:23.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.159+0000 7f1dd8cc9640 1 -- 172.21.15.57:0/2090464811 --> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir8", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f1dd40c1da0 con 0x7f1db406b040 2023-10-19T12:57:23.217 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.214+0000 7f1dcaffd640 1 -- 172.21.15.57:0/2090464811 <== mgr.5428 v2:172.21.15.179:6832/922398470 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f1dd40c1da0 con 0x7f1db406b040 2023-10-19T12:57:23.218 INFO:teuthology.orchestra.run.smithi057.stdout:Schedule deactivated for path /snap_test_dir1/dir8 2023-10-19T12:57:23.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.216+0000 7f1dd8cc9640 1 -- 172.21.15.57:0/2090464811 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f1db406b040 msgr2=0x7f1db406d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:23.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.216+0000 7f1dd8cc9640 1 --2- 172.21.15.57:0/2090464811 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f1db406b040 0x7f1db406d500 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1dcc009a10 tx=0x7f1dcc005f90 comp rx=0 tx=0).stop 2023-10-19T12:57:23.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.216+0000 7f1dd8cc9640 1 -- 172.21.15.57:0/2090464811 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1dd40af4b0 msgr2=0x7f1dd406b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:23.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.216+0000 7f1dd8cc9640 1 --2- 172.21.15.57:0/2090464811 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1dd40af4b0 0x7f1dd406b9e0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f1dc400c760 tx=0x7f1dc4004a00 comp rx=0 tx=0).stop 2023-10-19T12:57:23.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.216+0000 7f1dd8cc9640 1 -- 172.21.15.57:0/2090464811 shutdown_connections 2023-10-19T12:57:23.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.216+0000 7f1dd8cc9640 1 --2- 172.21.15.57:0/2090464811 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f1db406b040 0x7f1db406d500 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:23.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.216+0000 7f1dd8cc9640 1 --2- 172.21.15.57:0/2090464811 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1dd40af4b0 0x7f1dd406b9e0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:23.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.216+0000 7f1dd8cc9640 1 --2- 172.21.15.57:0/2090464811 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1dd40b1ca0 0x7f1dd40aef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:23.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.216+0000 7f1dd8cc9640 1 --2- 172.21.15.57:0/2090464811 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1dc0005180 0x7f1dd40b1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:23.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.216+0000 7f1dd8cc9640 1 -- 172.21.15.57:0/2090464811 >> 172.21.15.57:0/2090464811 conn(0x7f1dd40b0670 msgr2=0x7f1dd414fe80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:23.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.216+0000 7f1dd8cc9640 1 -- 172.21.15.57:0/2090464811 shutdown_connections 2023-10-19T12:57:23.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.216+0000 7f1dd8cc9640 1 -- 172.21.15.57:0/2090464811 wait complete. 2023-10-19T12:57:23.230 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir8 2023-10-19T12:57:23.231 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir9 1m 2023-10-19T12:57:23.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.341+0000 7f93be773640 1 -- 172.21.15.57:0/2903663111 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f93a0003ef0 msgr2=0x7f93a00047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:23.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.341+0000 7f93be773640 1 --2- 172.21.15.57:0/2903663111 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f93a0003ef0 0x7f93a00047c0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f93a8009b50 tx=0x7f93a00057f0 comp rx=0 tx=0).stop 2023-10-19T12:57:23.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.341+0000 7f93bcd10640 1 -- 172.21.15.57:0/2903663111 reap_dead start 2023-10-19T12:57:23.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.341+0000 7f93be773640 1 -- 172.21.15.57:0/2903663111 shutdown_connections 2023-10-19T12:57:23.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.341+0000 7f93be773640 1 -- 172.21.15.57:0/2903663111 >> 172.21.15.57:0/2903663111 conn(0x7f93b80b0670 msgr2=0x7f93b80b0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:23.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.341+0000 7f93be773640 1 -- 172.21.15.57:0/2903663111 shutdown_connections 2023-10-19T12:57:23.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.341+0000 7f93be773640 1 -- 172.21.15.57:0/2903663111 wait complete. 2023-10-19T12:57:23.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.343+0000 7f93be773640 1 Processor -- start 2023-10-19T12:57:23.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.343+0000 7f93be773640 1 -- start start 2023-10-19T12:57:23.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.343+0000 7f93be773640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f93a0003ef0 0x7f93b8145f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:23.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.343+0000 7f93be773640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f93b806b210 0x7f93b80c5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:23.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.343+0000 7f93be773640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f93b80c60f0 0x7f93b80cbae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:23.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.343+0000 7f93be773640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93b806b6d0 con 0x7f93b80c60f0 2023-10-19T12:57:23.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.343+0000 7f93be773640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93b806b9b0 con 0x7f93b806b210 2023-10-19T12:57:23.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.343+0000 7f93be773640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93b806bc90 con 0x7f93a0003ef0 2023-10-19T12:57:23.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.344+0000 7f93bcd10640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f93b80c60f0 0x7f93b80cbae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:23.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.344+0000 7f93bcd10640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f93b80c60f0 0x7f93b80cbae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34118/0 (socket says 172.21.15.57:34118) 2023-10-19T12:57:23.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.344+0000 7f93bcd10640 1 -- 172.21.15.57:0/4197284753 learned_addr learned my addr 172.21.15.57:0/4197284753 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:57:23.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.344+0000 7f93b7fff640 1 --2- 172.21.15.57:0/4197284753 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f93a0003ef0 0x7f93b8145f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:23.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.344+0000 7f93b77fe640 1 --2- 172.21.15.57:0/4197284753 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f93b806b210 0x7f93b80c5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:23.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.344+0000 7f93bcd10640 1 -- 172.21.15.57:0/4197284753 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f93a0003ef0 msgr2=0x7f93b8145f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:23.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.344+0000 7f93bcd10640 1 --2- 172.21.15.57:0/4197284753 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f93a0003ef0 0x7f93b8145f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:23.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.344+0000 7f93bcd10640 1 -- 172.21.15.57:0/4197284753 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f93b806b210 msgr2=0x7f93b80c5bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:23.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.344+0000 7f93bcd10640 1 --2- 172.21.15.57:0/4197284753 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f93b806b210 0x7f93b80c5bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:23.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.344+0000 7f93bcd10640 1 -- 172.21.15.57:0/4197284753 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93b806bf70 con 0x7f93b80c60f0 2023-10-19T12:57:23.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.344+0000 7f93bcd10640 1 --2- 172.21.15.57:0/4197284753 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f93b80c60f0 0x7f93b80cbae0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f93b8070890 tx=0x7f93b0006660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:23.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.344+0000 7f93b4ff9640 1 -- 172.21.15.57:0/4197284753 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93b000f9a0 con 0x7f93b80c60f0 2023-10-19T12:57:23.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.344+0000 7f93b4ff9640 1 -- 172.21.15.57:0/4197284753 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f93b000fb40 con 0x7f93b80c60f0 2023-10-19T12:57:23.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.344+0000 7f93be773640 1 -- 172.21.15.57:0/4197284753 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93b81464d0 con 0x7f93b80c60f0 2023-10-19T12:57:23.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.344+0000 7f93b4ff9640 1 -- 172.21.15.57:0/4197284753 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93b0011430 con 0x7f93b80c60f0 2023-10-19T12:57:23.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.344+0000 7f93be773640 1 -- 172.21.15.57:0/4197284753 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93b80cc200 con 0x7f93b80c60f0 2023-10-19T12:57:23.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.345+0000 7f93be773640 1 -- 172.21.15.57:0/4197284753 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93b80ce1d0 con 0x7f93b80c60f0 2023-10-19T12:57:23.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.346+0000 7f93b4ff9640 1 -- 172.21.15.57:0/4197284753 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 15) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f93b0011590 con 0x7f93b80c60f0 2023-10-19T12:57:23.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.346+0000 7f93b4ff9640 1 --2- 172.21.15.57:0/4197284753 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f93a406b070 0x7f93a406d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:23.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.346+0000 7f93b4ff9640 1 -- 172.21.15.57:0/4197284753 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f93b008a620 con 0x7f93b80c60f0 2023-10-19T12:57:23.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.346+0000 7f93b7fff640 1 --2- 172.21.15.57:0/4197284753 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f93a406b070 0x7f93a406d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:23.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.347+0000 7f93b7fff640 1 --2- 172.21.15.57:0/4197284753 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f93a406b070 0x7f93a406d530 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f93ac009cd0 tx=0x7f93ac005f90 comp rx=0 tx=0).ready entity=mgr.5428 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:23.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.350+0000 7f93b4ff9640 1 -- 172.21.15.57:0/4197284753 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f93b0055050 con 0x7f93b80c60f0 2023-10-19T12:57:23.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.548+0000 7f93be773640 1 -- 172.21.15.57:0/4197284753 --> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir9", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f93b80ae380 con 0x7f93a406b070 2023-10-19T12:57:23.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.622+0000 7f93b4ff9640 1 -- 172.21.15.57:0/4197284753 <== mgr.5428 v2:172.21.15.179:6832/922398470 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f93ac009920 con 0x7f93a406b070 2023-10-19T12:57:23.625 INFO:teuthology.orchestra.run.smithi057.stdout:Schedule deactivated for path /snap_test_dir1/dir9 2023-10-19T12:57:23.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.624+0000 7f93be773640 1 -- 172.21.15.57:0/4197284753 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f93a406b070 msgr2=0x7f93a406d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:23.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.624+0000 7f93be773640 1 --2- 172.21.15.57:0/4197284753 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f93a406b070 0x7f93a406d530 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f93ac009cd0 tx=0x7f93ac005f90 comp rx=0 tx=0).stop 2023-10-19T12:57:23.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.624+0000 7f93be773640 1 -- 172.21.15.57:0/4197284753 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f93b80c60f0 msgr2=0x7f93b80cbae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:23.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.624+0000 7f93be773640 1 --2- 172.21.15.57:0/4197284753 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f93b80c60f0 0x7f93b80cbae0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f93b8070890 tx=0x7f93b0006660 comp rx=0 tx=0).stop 2023-10-19T12:57:23.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.624+0000 7f93be773640 1 -- 172.21.15.57:0/4197284753 shutdown_connections 2023-10-19T12:57:23.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.624+0000 7f93be773640 1 --2- 172.21.15.57:0/4197284753 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f93a406b070 0x7f93a406d530 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:23.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.624+0000 7f93be773640 1 --2- 172.21.15.57:0/4197284753 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f93b80c60f0 0x7f93b80cbae0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:23.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.624+0000 7f93be773640 1 --2- 172.21.15.57:0/4197284753 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f93b806b210 0x7f93b80c5bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:23.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.624+0000 7f93be773640 1 --2- 172.21.15.57:0/4197284753 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f93a0003ef0 0x7f93b8145f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:23.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.624+0000 7f93be773640 1 -- 172.21.15.57:0/4197284753 >> 172.21.15.57:0/4197284753 conn(0x7f93b80b0670 msgr2=0x7f93b80c4e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:23.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.624+0000 7f93be773640 1 -- 172.21.15.57:0/4197284753 shutdown_connections 2023-10-19T12:57:23.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.624+0000 7f93be773640 1 -- 172.21.15.57:0/4197284753 wait complete. 2023-10-19T12:57:23.637 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir9 2023-10-19T12:57:23.638 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:23.638 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir0/.snap) 2023-10-19T12:57:23.659 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_46_00_UTC 2023-10-19T12:57:23.659 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_47_00_UTC 2023-10-19T12:57:23.659 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_48_00_UTC 2023-10-19T12:57:23.659 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_49_00_UTC 2023-10-19T12:57:23.660 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_50_00_UTC 2023-10-19T12:57:23.660 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_51_00_UTC 2023-10-19T12:57:23.660 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_52_00_UTC 2023-10-19T12:57:23.660 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_53_00_UTC 2023-10-19T12:57:23.660 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_54_00_UTC 2023-10-19T12:57:23.661 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_55_00_UTC 2023-10-19T12:57:23.661 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_56_00_UTC 2023-10-19T12:57:23.661 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_57_00_UTC 2023-10-19T12:57:23.661 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-19-12_46_00_UTC', 'scheduled-2023-10-19-12_47_00_UTC', 'scheduled-2023-10-19-12_48_00_UTC', 'scheduled-2023-10-19-12_49_00_UTC', 'scheduled-2023-10-19-12_50_00_UTC', 'scheduled-2023-10-19-12_51_00_UTC', 'scheduled-2023-10-19-12_52_00_UTC', 'scheduled-2023-10-19-12_53_00_UTC', 'scheduled-2023-10-19-12_54_00_UTC', 'scheduled-2023-10-19-12_55_00_UTC', 'scheduled-2023-10-19-12_56_00_UTC', 'scheduled-2023-10-19-12_57_00_UTC'] 2023-10-19T12:57:23.662 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir0 2023-10-19T12:57:23.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.809+0000 7fbe21a87640 1 -- 172.21.15.57:0/1193951888 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbe040039a0 msgr2=0x7fbe04003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:23.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.809+0000 7fbe21a87640 1 --2- 172.21.15.57:0/1193951888 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbe040039a0 0x7fbe04003dc0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fbe0c00bb20 tx=0x7fbe040057f0 comp rx=0 tx=0).stop 2023-10-19T12:57:23.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.809+0000 7fbe1bfff640 1 -- 172.21.15.57:0/1193951888 reap_dead start 2023-10-19T12:57:23.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.809+0000 7fbe21a87640 1 -- 172.21.15.57:0/1193951888 shutdown_connections 2023-10-19T12:57:23.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.809+0000 7fbe21a87640 1 -- 172.21.15.57:0/1193951888 >> 172.21.15.57:0/1193951888 conn(0x7fbe1c0d0d60 msgr2=0x7fbe1c0d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:23.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.809+0000 7fbe21a87640 1 -- 172.21.15.57:0/1193951888 shutdown_connections 2023-10-19T12:57:23.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.809+0000 7fbe21a87640 1 -- 172.21.15.57:0/1193951888 wait complete. 2023-10-19T12:57:23.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.810+0000 7fbe21a87640 1 Processor -- start 2023-10-19T12:57:23.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.810+0000 7fbe21a87640 1 -- start start 2023-10-19T12:57:23.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.811+0000 7fbe21a87640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbe1c159a20 0x7fbe1c15df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:23.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.811+0000 7fbe21a87640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbe1c156b90 0x7fbe1c1f5950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:23.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.811+0000 7fbe21a87640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbe1c0b1980 0x7fbe1c0b1e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:23.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.811+0000 7fbe21a87640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe1c15e5c0 con 0x7fbe1c159a20 2023-10-19T12:57:23.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.811+0000 7fbe21a87640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe1c0b2340 con 0x7fbe1c156b90 2023-10-19T12:57:23.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.811+0000 7fbe21a87640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe1c0c7190 con 0x7fbe1c0b1980 2023-10-19T12:57:23.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.811+0000 7fbe1b7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbe1c159a20 0x7fbe1c15df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:23.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.811+0000 7fbe1b7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbe1c159a20 0x7fbe1c15df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34138/0 (socket says 172.21.15.57:34138) 2023-10-19T12:57:23.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.811+0000 7fbe1b7fe640 1 -- 172.21.15.57:0/1612314113 learned_addr learned my addr 172.21.15.57:0/1612314113 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:57:23.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.811+0000 7fbe1affd640 1 --2- 172.21.15.57:0/1612314113 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbe1c156b90 0x7fbe1c1f5950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:23.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.811+0000 7fbe1bfff640 1 --2- 172.21.15.57:0/1612314113 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbe1c0b1980 0x7fbe1c0b1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:23.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.811+0000 7fbe1b7fe640 1 -- 172.21.15.57:0/1612314113 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbe1c0b1980 msgr2=0x7fbe1c0b1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:23.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.811+0000 7fbe1b7fe640 1 --2- 172.21.15.57:0/1612314113 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbe1c0b1980 0x7fbe1c0b1e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:23.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.811+0000 7fbe1b7fe640 1 -- 172.21.15.57:0/1612314113 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbe1c156b90 msgr2=0x7fbe1c1f5950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:23.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.811+0000 7fbe1b7fe640 1 --2- 172.21.15.57:0/1612314113 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbe1c156b90 0x7fbe1c1f5950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:23.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.811+0000 7fbe1b7fe640 1 -- 172.21.15.57:0/1612314113 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe1c14e350 con 0x7fbe1c159a20 2023-10-19T12:57:23.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.811+0000 7fbe1affd640 1 --2- 172.21.15.57:0/1612314113 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbe1c156b90 0x7fbe1c1f5950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:57:23.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.811+0000 7fbe1b7fe640 1 --2- 172.21.15.57:0/1612314113 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbe1c159a20 0x7fbe1c15df70 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fbe0c00bb20 tx=0x7fbe0c0367d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:23.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.812+0000 7fbe03fff640 1 -- 172.21.15.57:0/1612314113 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe0c036dd0 con 0x7fbe1c159a20 2023-10-19T12:57:23.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.812+0000 7fbe03fff640 1 -- 172.21.15.57:0/1612314113 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fbe0c004120 con 0x7fbe1c159a20 2023-10-19T12:57:23.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.812+0000 7fbe03fff640 1 -- 172.21.15.57:0/1612314113 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe0c0035f0 con 0x7fbe1c159a20 2023-10-19T12:57:23.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.812+0000 7fbe21a87640 1 -- 172.21.15.57:0/1612314113 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe1c1f54c0 con 0x7fbe1c159a20 2023-10-19T12:57:23.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.812+0000 7fbe21a87640 1 -- 172.21.15.57:0/1612314113 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe1c14aba0 con 0x7fbe1c159a20 2023-10-19T12:57:23.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.813+0000 7fbe21a87640 1 -- 172.21.15.57:0/1612314113 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe1c0690e0 con 0x7fbe1c159a20 2023-10-19T12:57:23.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.813+0000 7fbe03fff640 1 -- 172.21.15.57:0/1612314113 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 15) v1 ==== 89417+0+0 (secure 0 0 0) 0x7fbe0c003750 con 0x7fbe1c159a20 2023-10-19T12:57:23.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.814+0000 7fbe03fff640 1 --2- 172.21.15.57:0/1612314113 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fbe0806b070 0x7fbe0806d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:23.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.814+0000 7fbe03fff640 1 -- 172.21.15.57:0/1612314113 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7fbe0c0abd10 con 0x7fbe1c159a20 2023-10-19T12:57:23.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.814+0000 7fbe1affd640 1 --2- 172.21.15.57:0/1612314113 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fbe0806b070 0x7fbe0806d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:23.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.814+0000 7fbe1affd640 1 --2- 172.21.15.57:0/1612314113 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fbe0806b070 0x7fbe0806d530 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fbe10001200 tx=0x7fbe10001110 comp rx=0 tx=0).ready entity=mgr.5428 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:23.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:23.818+0000 7fbe03fff640 1 -- 172.21.15.57:0/1612314113 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fbe0c076860 con 0x7fbe1c159a20 2023-10-19T12:57:24.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.013+0000 7fbe21a87640 1 -- 172.21.15.57:0/1612314113 --> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir0", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbe1c0b0390 con 0x7fbe0806b070 2023-10-19T12:57:24.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.026+0000 7fbe03fff640 1 -- 172.21.15.57:0/1612314113 <== mgr.5428 v2:172.21.15.179:6832/922398470 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7fbe1c0b0390 con 0x7fbe0806b070 2023-10-19T12:57:24.029 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:57:24.029 INFO:teuthology.orchestra.run.smithi057.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir0", "rel_path": "/snap_test_dir1/dir0", "schedule": "1m", "retention": {}, "start": "2023-10-19T00:00:00", "created": "2023-10-19T12:45:57", "first": "2023-10-19T12:46:00", "last": "2023-10-19T12:57:00", "last_pruned": null, "created_count": 12, "pruned_count": 0, "active": false}] 2023-10-19T12:57:24.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.027+0000 7fbe21a87640 1 -- 172.21.15.57:0/1612314113 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fbe0806b070 msgr2=0x7fbe0806d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:24.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.027+0000 7fbe21a87640 1 --2- 172.21.15.57:0/1612314113 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fbe0806b070 0x7fbe0806d530 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fbe10001200 tx=0x7fbe10001110 comp rx=0 tx=0).stop 2023-10-19T12:57:24.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.027+0000 7fbe21a87640 1 -- 172.21.15.57:0/1612314113 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbe1c159a20 msgr2=0x7fbe1c15df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:24.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.027+0000 7fbe21a87640 1 --2- 172.21.15.57:0/1612314113 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbe1c159a20 0x7fbe1c15df70 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fbe0c00bb20 tx=0x7fbe0c0367d0 comp rx=0 tx=0).stop 2023-10-19T12:57:24.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.027+0000 7fbe21a87640 1 -- 172.21.15.57:0/1612314113 shutdown_connections 2023-10-19T12:57:24.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.027+0000 7fbe21a87640 1 --2- 172.21.15.57:0/1612314113 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fbe0806b070 0x7fbe0806d530 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:24.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.027+0000 7fbe21a87640 1 --2- 172.21.15.57:0/1612314113 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbe1c0b1980 0x7fbe1c0b1e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:24.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.027+0000 7fbe21a87640 1 --2- 172.21.15.57:0/1612314113 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbe1c156b90 0x7fbe1c1f5950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:24.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.028+0000 7fbe21a87640 1 --2- 172.21.15.57:0/1612314113 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbe1c159a20 0x7fbe1c15df70 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:24.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.028+0000 7fbe21a87640 1 -- 172.21.15.57:0/1612314113 >> 172.21.15.57:0/1612314113 conn(0x7fbe1c0d0d60 msgr2=0x7fbe1c0b5960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:24.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.028+0000 7fbe21a87640 1 -- 172.21.15.57:0/1612314113 shutdown_connections 2023-10-19T12:57:24.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.028+0000 7fbe21a87640 1 -- 172.21.15.57:0/1612314113 wait complete. 2023-10-19T12:57:24.041 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir0", "rel_path": "/snap_test_dir1/dir0", "schedule": "1m", "retention": {}, "start": "2023-10-19T00:00:00", "created": "2023-10-19T12:45:57", "first": "2023-10-19T12:46:00", "last": "2023-10-19T12:57:00", "last_pruned": null, "created_count": 12, "pruned_count": 0, "active": false}] 2023-10-19T12:57:24.041 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir0', 'rel_path': '/snap_test_dir1/dir0', 'schedule': '1m', 'retention': {}, 'start': '2023-10-19T00:00:00', 'created': '2023-10-19T12:45:57', 'first': '2023-10-19T12:46:00', 'last': '2023-10-19T12:57:00', 'last_pruned': None, 'created_count': 12, 'pruned_count': 0, 'active': False} 2023-10-19T12:57:24.042 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 12 2023-10-19T12:57:24.042 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 12 2023-10-19T12:57:24.043 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:24.043 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir1/.snap) 2023-10-19T12:57:24.063 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_46_00_UTC 2023-10-19T12:57:24.063 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_47_00_UTC 2023-10-19T12:57:24.064 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_48_00_UTC 2023-10-19T12:57:24.064 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_49_00_UTC 2023-10-19T12:57:24.064 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_50_00_UTC 2023-10-19T12:57:24.064 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_51_00_UTC 2023-10-19T12:57:24.064 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_52_00_UTC 2023-10-19T12:57:24.065 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_53_00_UTC 2023-10-19T12:57:24.065 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_54_00_UTC 2023-10-19T12:57:24.065 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_55_01_UTC 2023-10-19T12:57:24.065 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_56_00_UTC 2023-10-19T12:57:24.065 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_57_00_UTC 2023-10-19T12:57:24.066 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-19-12_46_00_UTC', 'scheduled-2023-10-19-12_47_00_UTC', 'scheduled-2023-10-19-12_48_00_UTC', 'scheduled-2023-10-19-12_49_00_UTC', 'scheduled-2023-10-19-12_50_00_UTC', 'scheduled-2023-10-19-12_51_00_UTC', 'scheduled-2023-10-19-12_52_00_UTC', 'scheduled-2023-10-19-12_53_00_UTC', 'scheduled-2023-10-19-12_54_00_UTC', 'scheduled-2023-10-19-12_55_01_UTC', 'scheduled-2023-10-19-12_56_00_UTC', 'scheduled-2023-10-19-12_57_00_UTC'] 2023-10-19T12:57:24.066 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir1 2023-10-19T12:57:24.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.212+0000 7fbdb5083640 1 -- 172.21.15.57:0/1077786698 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd98005e10 msgr2=0x7fbd98006290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:24.216 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.212+0000 7fbdb5083640 1 --2- 172.21.15.57:0/1077786698 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd98005e10 0x7fbd98006290 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fbda800c610 tx=0x7fbd98006900 comp rx=0 tx=0).stop 2023-10-19T12:57:24.216 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.212+0000 7fbdaf59e640 1 -- 172.21.15.57:0/1077786698 reap_dead start 2023-10-19T12:57:24.216 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.212+0000 7fbdb5083640 1 -- 172.21.15.57:0/1077786698 shutdown_connections 2023-10-19T12:57:24.217 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.212+0000 7fbdb5083640 1 -- 172.21.15.57:0/1077786698 >> 172.21.15.57:0/1077786698 conn(0x7fbdb0069930 msgr2=0x7fbdb0069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:24.217 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.212+0000 7fbdb5083640 1 -- 172.21.15.57:0/1077786698 shutdown_connections 2023-10-19T12:57:24.217 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.212+0000 7fbdb5083640 1 -- 172.21.15.57:0/1077786698 wait complete. 2023-10-19T12:57:24.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.213+0000 7fbdb5083640 1 Processor -- start 2023-10-19T12:57:24.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.214+0000 7fbdb5083640 1 -- start start 2023-10-19T12:57:24.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.214+0000 7fbdb5083640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbd98005e10 0x7fbdb01dfb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:24.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.214+0000 7fbdb5083640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbdb013c2c0 0x7fbdb0150520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:24.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.214+0000 7fbdb5083640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbdb014c3f0 0x7fbdb014c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:24.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.214+0000 7fbdb5083640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb014cdb0 con 0x7fbdb014c3f0 2023-10-19T12:57:24.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.214+0000 7fbdb5083640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb0150090 con 0x7fbd98005e10 2023-10-19T12:57:24.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.214+0000 7fbdb5083640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb01e0050 con 0x7fbdb013c2c0 2023-10-19T12:57:24.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.214+0000 7fbdaf59e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbdb014c3f0 0x7fbdb014c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:24.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.214+0000 7fbdaf59e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbdb014c3f0 0x7fbdb014c870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34152/0 (socket says 172.21.15.57:34152) 2023-10-19T12:57:24.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.214+0000 7fbdaf59e640 1 -- 172.21.15.57:0/4185891754 learned_addr learned my addr 172.21.15.57:0/4185891754 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:57:24.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.215+0000 7fbdaed9d640 1 --2- 172.21.15.57:0/4185891754 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbd98005e10 0x7fbdb01dfb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:24.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.215+0000 7fbdae59c640 1 --2- 172.21.15.57:0/4185891754 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbdb013c2c0 0x7fbdb0150520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:24.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.215+0000 7fbdaf59e640 1 -- 172.21.15.57:0/4185891754 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbdb013c2c0 msgr2=0x7fbdb0150520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:24.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.215+0000 7fbdaf59e640 1 --2- 172.21.15.57:0/4185891754 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbdb013c2c0 0x7fbdb0150520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:24.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.215+0000 7fbdaf59e640 1 -- 172.21.15.57:0/4185891754 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbd98005e10 msgr2=0x7fbdb01dfb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:24.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.215+0000 7fbdaf59e640 1 --2- 172.21.15.57:0/4185891754 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbd98005e10 0x7fbdb01dfb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:24.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.215+0000 7fbdaf59e640 1 -- 172.21.15.57:0/4185891754 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdb014f660 con 0x7fbdb014c3f0 2023-10-19T12:57:24.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.215+0000 7fbdaf59e640 1 --2- 172.21.15.57:0/4185891754 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbdb014c3f0 0x7fbdb014c870 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fbda802cc80 tx=0x7fbda8002100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:24.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.215+0000 7fbd977fe640 1 -- 172.21.15.57:0/4185891754 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbda8001830 con 0x7fbdb014c3f0 2023-10-19T12:57:24.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.215+0000 7fbd977fe640 1 -- 172.21.15.57:0/4185891754 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fbda8001990 con 0x7fbdb014c3f0 2023-10-19T12:57:24.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.215+0000 7fbdb5083640 1 -- 172.21.15.57:0/4185891754 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdb014f380 con 0x7fbdb014c3f0 2023-10-19T12:57:24.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.215+0000 7fbd977fe640 1 -- 172.21.15.57:0/4185891754 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbda8003bb0 con 0x7fbdb014c3f0 2023-10-19T12:57:24.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.215+0000 7fbdb5083640 1 -- 172.21.15.57:0/4185891754 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdb014fd20 con 0x7fbdb014c3f0 2023-10-19T12:57:24.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.216+0000 7fbdb5083640 1 -- 172.21.15.57:0/4185891754 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdb014fee0 con 0x7fbdb014c3f0 2023-10-19T12:57:24.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.217+0000 7fbd977fe640 1 -- 172.21.15.57:0/4185891754 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 15) v1 ==== 89417+0+0 (secure 0 0 0) 0x7fbda8031030 con 0x7fbdb014c3f0 2023-10-19T12:57:24.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.217+0000 7fbd977fe640 1 --2- 172.21.15.57:0/4185891754 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fbd9c06b040 0x7fbd9c06d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:24.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.217+0000 7fbd977fe640 1 -- 172.21.15.57:0/4185891754 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7fbda80aa8e0 con 0x7fbdb014c3f0 2023-10-19T12:57:24.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.217+0000 7fbdaed9d640 1 --2- 172.21.15.57:0/4185891754 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fbd9c06b040 0x7fbd9c06d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:24.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.218+0000 7fbdaed9d640 1 --2- 172.21.15.57:0/4185891754 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fbd9c06b040 0x7fbd9c06d500 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fbda0009cd0 tx=0x7fbda0005f90 comp rx=0 tx=0).ready entity=mgr.5428 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:24.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.221+0000 7fbd977fe640 1 -- 172.21.15.57:0/4185891754 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fbda8075580 con 0x7fbdb014c3f0 2023-10-19T12:57:24.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.417+0000 7fbdb5083640 1 -- 172.21.15.57:0/4185891754 --> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbdb00cc1b0 con 0x7fbd9c06b040 2023-10-19T12:57:24.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.426+0000 7fbd977fe640 1 -- 172.21.15.57:0/4185891754 <== mgr.5428 v2:172.21.15.179:6832/922398470 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7fbdb00cc1b0 con 0x7fbd9c06b040 2023-10-19T12:57:24.430 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:57:24.430 INFO:teuthology.orchestra.run.smithi057.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir1", "rel_path": "/snap_test_dir1/dir1", "schedule": "1m", "retention": {}, "start": "2023-10-19T00:00:00", "created": "2023-10-19T12:45:57", "first": "2023-10-19T12:46:00", "last": "2023-10-19T12:57:00", "last_pruned": null, "created_count": 12, "pruned_count": 0, "active": false}] 2023-10-19T12:57:24.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.429+0000 7fbdb5083640 1 -- 172.21.15.57:0/4185891754 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fbd9c06b040 msgr2=0x7fbd9c06d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:24.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.429+0000 7fbdb5083640 1 --2- 172.21.15.57:0/4185891754 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fbd9c06b040 0x7fbd9c06d500 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fbda0009cd0 tx=0x7fbda0005f90 comp rx=0 tx=0).stop 2023-10-19T12:57:24.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.429+0000 7fbdb5083640 1 -- 172.21.15.57:0/4185891754 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbdb014c3f0 msgr2=0x7fbdb014c870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:24.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.429+0000 7fbdb5083640 1 --2- 172.21.15.57:0/4185891754 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbdb014c3f0 0x7fbdb014c870 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fbda802cc80 tx=0x7fbda8002100 comp rx=0 tx=0).stop 2023-10-19T12:57:24.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.429+0000 7fbdb5083640 1 -- 172.21.15.57:0/4185891754 shutdown_connections 2023-10-19T12:57:24.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.429+0000 7fbdb5083640 1 --2- 172.21.15.57:0/4185891754 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fbd9c06b040 0x7fbd9c06d500 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:24.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.429+0000 7fbdb5083640 1 --2- 172.21.15.57:0/4185891754 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbdb014c3f0 0x7fbdb014c870 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:24.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.429+0000 7fbdb5083640 1 --2- 172.21.15.57:0/4185891754 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbdb013c2c0 0x7fbdb0150520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:24.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.429+0000 7fbdb5083640 1 --2- 172.21.15.57:0/4185891754 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbd98005e10 0x7fbdb01dfb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:24.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.429+0000 7fbdb5083640 1 -- 172.21.15.57:0/4185891754 >> 172.21.15.57:0/4185891754 conn(0x7fbdb0069930 msgr2=0x7fbdb00ad980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:24.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.429+0000 7fbdb5083640 1 -- 172.21.15.57:0/4185891754 shutdown_connections 2023-10-19T12:57:24.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.429+0000 7fbdb5083640 1 -- 172.21.15.57:0/4185891754 wait complete. 2023-10-19T12:57:24.442 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir1", "rel_path": "/snap_test_dir1/dir1", "schedule": "1m", "retention": {}, "start": "2023-10-19T00:00:00", "created": "2023-10-19T12:45:57", "first": "2023-10-19T12:46:00", "last": "2023-10-19T12:57:00", "last_pruned": null, "created_count": 12, "pruned_count": 0, "active": false}] 2023-10-19T12:57:24.442 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir1', 'rel_path': '/snap_test_dir1/dir1', 'schedule': '1m', 'retention': {}, 'start': '2023-10-19T00:00:00', 'created': '2023-10-19T12:45:57', 'first': '2023-10-19T12:46:00', 'last': '2023-10-19T12:57:00', 'last_pruned': None, 'created_count': 12, 'pruned_count': 0, 'active': False} 2023-10-19T12:57:24.443 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 12 2023-10-19T12:57:24.443 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 12 2023-10-19T12:57:24.443 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:24.444 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir2/.snap) 2023-10-19T12:57:24.464 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_46_00_UTC 2023-10-19T12:57:24.465 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_47_00_UTC 2023-10-19T12:57:24.465 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_48_00_UTC 2023-10-19T12:57:24.465 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_49_00_UTC 2023-10-19T12:57:24.465 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_50_00_UTC 2023-10-19T12:57:24.465 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_51_00_UTC 2023-10-19T12:57:24.466 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_52_00_UTC 2023-10-19T12:57:24.466 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_53_00_UTC 2023-10-19T12:57:24.466 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_54_00_UTC 2023-10-19T12:57:24.466 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_55_00_UTC 2023-10-19T12:57:24.466 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_56_00_UTC 2023-10-19T12:57:24.467 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_57_00_UTC 2023-10-19T12:57:24.467 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-19-12_46_00_UTC', 'scheduled-2023-10-19-12_47_00_UTC', 'scheduled-2023-10-19-12_48_00_UTC', 'scheduled-2023-10-19-12_49_00_UTC', 'scheduled-2023-10-19-12_50_00_UTC', 'scheduled-2023-10-19-12_51_00_UTC', 'scheduled-2023-10-19-12_52_00_UTC', 'scheduled-2023-10-19-12_53_00_UTC', 'scheduled-2023-10-19-12_54_00_UTC', 'scheduled-2023-10-19-12_55_00_UTC', 'scheduled-2023-10-19-12_56_00_UTC', 'scheduled-2023-10-19-12_57_00_UTC'] 2023-10-19T12:57:24.467 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir2 2023-10-19T12:57:24.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.612+0000 7f933c933640 1 -- 172.21.15.57:0/2410012609 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f933806af00 msgr2=0x7f933806b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:24.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.612+0000 7f933c933640 1 --2- 172.21.15.57:0/2410012609 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f933806af00 0x7f933806b2e0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f9328009900 tx=0x7f932802be00 comp rx=0 tx=0).stop 2023-10-19T12:57:24.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.613+0000 7f933c933640 1 -- 172.21.15.57:0/2410012609 shutdown_connections 2023-10-19T12:57:24.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.613+0000 7f933c933640 1 --2- 172.21.15.57:0/2410012609 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f93380abc10 0x7f93380c9b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:24.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.613+0000 7f933c933640 1 --2- 172.21.15.57:0/2410012609 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f933806af00 0x7f933806b2e0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:24.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.613+0000 7f933c933640 1 -- 172.21.15.57:0/2410012609 >> 172.21.15.57:0/2410012609 conn(0x7f9338069930 msgr2=0x7f9338069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:24.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.613+0000 7f933c933640 1 -- 172.21.15.57:0/2410012609 shutdown_connections 2023-10-19T12:57:24.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.613+0000 7f933c933640 1 -- 172.21.15.57:0/2410012609 wait complete. 2023-10-19T12:57:24.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.614+0000 7f933c933640 1 Processor -- start 2023-10-19T12:57:24.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.614+0000 7f933c933640 1 -- start start 2023-10-19T12:57:24.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.615+0000 7f933c933640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f933806af00 0x7f93381f2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:24.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.615+0000 7f933c933640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f93380abc10 0x7f933815b710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:24.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.615+0000 7f933c933640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f93380c96b0 0x7f933812e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:24.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.615+0000 7f933c933640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f933812ec20 con 0x7f93380abc10 2023-10-19T12:57:24.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.615+0000 7f933c933640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f933812ee50 con 0x7f933806af00 2023-10-19T12:57:24.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.615+0000 7f933c933640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f933812f130 con 0x7f93380c96b0 2023-10-19T12:57:24.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.615+0000 7f9335d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f93380abc10 0x7f933815b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:24.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.615+0000 7f933659c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f933806af00 0x7f93381f2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:24.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.615+0000 7f9335d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f93380abc10 0x7f933815b710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34170/0 (socket says 172.21.15.57:34170) 2023-10-19T12:57:24.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.615+0000 7f9335d9b640 1 -- 172.21.15.57:0/1252966097 learned_addr learned my addr 172.21.15.57:0/1252966097 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:57:24.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.615+0000 7f9336d9d640 1 --2- 172.21.15.57:0/1252966097 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f93380c96b0 0x7f933812e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:24.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.615+0000 7f9335d9b640 1 -- 172.21.15.57:0/1252966097 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f93380c96b0 msgr2=0x7f933812e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:24.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.615+0000 7f9335d9b640 1 --2- 172.21.15.57:0/1252966097 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f93380c96b0 0x7f933812e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:24.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.615+0000 7f9335d9b640 1 -- 172.21.15.57:0/1252966097 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f933806af00 msgr2=0x7f93381f2120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:24.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.615+0000 7f9335d9b640 1 --2- 172.21.15.57:0/1252966097 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f933806af00 0x7f93381f2120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:24.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.615+0000 7f9335d9b640 1 -- 172.21.15.57:0/1252966097 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9328009560 con 0x7f93380abc10 2023-10-19T12:57:24.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.615+0000 7f9335d9b640 1 --2- 172.21.15.57:0/1252966097 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f93380abc10 0x7f933815b710 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f932c00c410 tx=0x7f932c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:24.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.616+0000 7f9326ffd640 1 -- 172.21.15.57:0/1252966097 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f932c0035f0 con 0x7f93380abc10 2023-10-19T12:57:24.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.616+0000 7f9326ffd640 1 -- 172.21.15.57:0/1252966097 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f932c003770 con 0x7f93380abc10 2023-10-19T12:57:24.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.616+0000 7f9326ffd640 1 -- 172.21.15.57:0/1252966097 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f932c003b80 con 0x7f93380abc10 2023-10-19T12:57:24.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.616+0000 7f933c933640 1 -- 172.21.15.57:0/1252966097 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f933812f410 con 0x7f93380abc10 2023-10-19T12:57:24.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.616+0000 7f933c933640 1 -- 172.21.15.57:0/1252966097 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f933812f9d0 con 0x7f93380abc10 2023-10-19T12:57:24.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.617+0000 7f933c933640 1 -- 172.21.15.57:0/1252966097 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93380690e0 con 0x7f93380abc10 2023-10-19T12:57:24.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.617+0000 7f9326ffd640 1 -- 172.21.15.57:0/1252966097 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 15) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f932c003ce0 con 0x7f93380abc10 2023-10-19T12:57:24.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.618+0000 7f9326ffd640 1 --2- 172.21.15.57:0/1252966097 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f931806b040 0x7f931806d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:24.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.618+0000 7f9326ffd640 1 -- 172.21.15.57:0/1252966097 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f932c0891a0 con 0x7f93380abc10 2023-10-19T12:57:24.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.618+0000 7f933659c640 1 --2- 172.21.15.57:0/1252966097 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f931806b040 0x7f931806d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:24.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.618+0000 7f933659c640 1 --2- 172.21.15.57:0/1252966097 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f931806b040 0x7f931806d500 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9328035000 tx=0x7f9328004730 comp rx=0 tx=0).ready entity=mgr.5428 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:24.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.622+0000 7f9326ffd640 1 -- 172.21.15.57:0/1252966097 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f932c053f60 con 0x7f93380abc10 2023-10-19T12:57:24.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.820+0000 7f933c933640 1 -- 172.21.15.57:0/1252966097 --> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir2", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9338159c50 con 0x7f931806b040 2023-10-19T12:57:24.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.828+0000 7f9326ffd640 1 -- 172.21.15.57:0/1252966097 <== mgr.5428 v2:172.21.15.179:6832/922398470 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7f9328004370 con 0x7f931806b040 2023-10-19T12:57:24.831 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:57:24.831 INFO:teuthology.orchestra.run.smithi057.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir2", "rel_path": "/snap_test_dir1/dir2", "schedule": "1m", "retention": {}, "start": "2023-10-19T00:00:00", "created": "2023-10-19T12:45:58", "first": "2023-10-19T12:46:00", "last": "2023-10-19T12:57:00", "last_pruned": null, "created_count": 12, "pruned_count": 0, "active": false}] 2023-10-19T12:57:24.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.830+0000 7f933c933640 1 -- 172.21.15.57:0/1252966097 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f931806b040 msgr2=0x7f931806d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:24.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.830+0000 7f933c933640 1 --2- 172.21.15.57:0/1252966097 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f931806b040 0x7f931806d500 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9328035000 tx=0x7f9328004730 comp rx=0 tx=0).stop 2023-10-19T12:57:24.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.830+0000 7f933c933640 1 -- 172.21.15.57:0/1252966097 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f93380abc10 msgr2=0x7f933815b710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:24.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.830+0000 7f933c933640 1 --2- 172.21.15.57:0/1252966097 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f93380abc10 0x7f933815b710 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f932c00c410 tx=0x7f932c00c8d0 comp rx=0 tx=0).stop 2023-10-19T12:57:24.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.830+0000 7f933c933640 1 -- 172.21.15.57:0/1252966097 shutdown_connections 2023-10-19T12:57:24.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.830+0000 7f933c933640 1 --2- 172.21.15.57:0/1252966097 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f931806b040 0x7f931806d500 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:24.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.830+0000 7f933c933640 1 --2- 172.21.15.57:0/1252966097 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f93380c96b0 0x7f933812e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:24.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.830+0000 7f933c933640 1 --2- 172.21.15.57:0/1252966097 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f93380abc10 0x7f933815b710 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:24.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.830+0000 7f933c933640 1 --2- 172.21.15.57:0/1252966097 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f933806af00 0x7f93381f2120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:24.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.830+0000 7f933c933640 1 -- 172.21.15.57:0/1252966097 >> 172.21.15.57:0/1252966097 conn(0x7f9338069930 msgr2=0x7f9338069d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:24.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.830+0000 7f933c933640 1 -- 172.21.15.57:0/1252966097 shutdown_connections 2023-10-19T12:57:24.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:24.830+0000 7f933c933640 1 -- 172.21.15.57:0/1252966097 wait complete. 2023-10-19T12:57:24.843 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir2", "rel_path": "/snap_test_dir1/dir2", "schedule": "1m", "retention": {}, "start": "2023-10-19T00:00:00", "created": "2023-10-19T12:45:58", "first": "2023-10-19T12:46:00", "last": "2023-10-19T12:57:00", "last_pruned": null, "created_count": 12, "pruned_count": 0, "active": false}] 2023-10-19T12:57:24.843 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir2', 'rel_path': '/snap_test_dir1/dir2', 'schedule': '1m', 'retention': {}, 'start': '2023-10-19T00:00:00', 'created': '2023-10-19T12:45:58', 'first': '2023-10-19T12:46:00', 'last': '2023-10-19T12:57:00', 'last_pruned': None, 'created_count': 12, 'pruned_count': 0, 'active': False} 2023-10-19T12:57:24.844 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 12 2023-10-19T12:57:24.844 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 12 2023-10-19T12:57:24.844 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:24.845 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir3/.snap) 2023-10-19T12:57:24.867 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_46_00_UTC 2023-10-19T12:57:24.868 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_47_00_UTC 2023-10-19T12:57:24.868 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_48_00_UTC 2023-10-19T12:57:24.868 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_49_00_UTC 2023-10-19T12:57:24.869 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_50_00_UTC 2023-10-19T12:57:24.869 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_51_00_UTC 2023-10-19T12:57:24.869 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_52_00_UTC 2023-10-19T12:57:24.870 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_53_00_UTC 2023-10-19T12:57:24.870 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_54_00_UTC 2023-10-19T12:57:24.871 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_55_00_UTC 2023-10-19T12:57:24.871 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_56_00_UTC 2023-10-19T12:57:24.871 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_57_00_UTC 2023-10-19T12:57:24.872 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-19-12_46_00_UTC', 'scheduled-2023-10-19-12_47_00_UTC', 'scheduled-2023-10-19-12_48_00_UTC', 'scheduled-2023-10-19-12_49_00_UTC', 'scheduled-2023-10-19-12_50_00_UTC', 'scheduled-2023-10-19-12_51_00_UTC', 'scheduled-2023-10-19-12_52_00_UTC', 'scheduled-2023-10-19-12_53_00_UTC', 'scheduled-2023-10-19-12_54_00_UTC', 'scheduled-2023-10-19-12_55_00_UTC', 'scheduled-2023-10-19-12_56_00_UTC', 'scheduled-2023-10-19-12_57_00_UTC'] 2023-10-19T12:57:24.872 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir3 2023-10-19T12:57:25.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.016+0000 7f0d3e4c9640 1 -- 172.21.15.57:0/310214007 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0d20005180 msgr2=0x7f0d2000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:25.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.016+0000 7f0d3e4c9640 1 --2- 172.21.15.57:0/310214007 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0d20005180 0x7f0d2000b980 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f0d3000c610 tx=0x7f0d200057f0 comp rx=0 tx=0).stop 2023-10-19T12:57:25.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.016+0000 7f0d3e4c9640 1 -- 172.21.15.57:0/310214007 shutdown_connections 2023-10-19T12:57:25.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.016+0000 7f0d3e4c9640 1 --2- 172.21.15.57:0/310214007 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0d20005180 0x7f0d2000b980 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:25.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.016+0000 7f0d3e4c9640 1 -- 172.21.15.57:0/310214007 >> 172.21.15.57:0/310214007 conn(0x7f0d380af7a0 msgr2=0x7f0d380afb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:25.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.016+0000 7f0d3e4c9640 1 -- 172.21.15.57:0/310214007 shutdown_connections 2023-10-19T12:57:25.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.016+0000 7f0d3e4c9640 1 -- 172.21.15.57:0/310214007 wait complete. 2023-10-19T12:57:25.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.017+0000 7f0d3e4c9640 1 Processor -- start 2023-10-19T12:57:25.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.017+0000 7f0d3e4c9640 1 -- start start 2023-10-19T12:57:25.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.018+0000 7f0d3e4c9640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0d20005180 0x7f0d380c44c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:25.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.018+0000 7f0d3e4c9640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0d380b2520 0x7f0d38140370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:25.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.018+0000 7f0d3e4c9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0d38156ec0 0x7f0d381ee530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:25.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.018+0000 7f0d3e4c9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d380c3420 con 0x7f0d38156ec0 2023-10-19T12:57:25.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.018+0000 7f0d3e4c9640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d380c3700 con 0x7f0d20005180 2023-10-19T12:57:25.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.018+0000 7f0d3e4c9640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d380c39e0 con 0x7f0d380b2520 2023-10-19T12:57:25.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.018+0000 7f0d3ca66640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0d38156ec0 0x7f0d381ee530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:25.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.018+0000 7f0d3ca66640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0d38156ec0 0x7f0d381ee530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34192/0 (socket says 172.21.15.57:34192) 2023-10-19T12:57:25.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.018+0000 7f0d3ca66640 1 -- 172.21.15.57:0/1998492702 learned_addr learned my addr 172.21.15.57:0/1998492702 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:57:25.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.018+0000 7f0d37fff640 1 --2- 172.21.15.57:0/1998492702 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0d20005180 0x7f0d380c44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:25.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.018+0000 7f0d3ca66640 1 -- 172.21.15.57:0/1998492702 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0d380b2520 msgr2=0x7f0d38140370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:25.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.018+0000 7f0d3ca66640 1 --2- 172.21.15.57:0/1998492702 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0d380b2520 0x7f0d38140370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:25.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.018+0000 7f0d3ca66640 1 -- 172.21.15.57:0/1998492702 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0d20005180 msgr2=0x7f0d380c44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:25.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.018+0000 7f0d3ca66640 1 --2- 172.21.15.57:0/1998492702 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0d20005180 0x7f0d380c44c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:25.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.019+0000 7f0d3ca66640 1 -- 172.21.15.57:0/1998492702 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d380c3cc0 con 0x7f0d38156ec0 2023-10-19T12:57:25.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.019+0000 7f0d3ca66640 1 --2- 172.21.15.57:0/1998492702 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0d38156ec0 0x7f0d381ee530 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f0d3002cd20 tx=0x7f0d300017b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:25.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.019+0000 7f0d34ff9640 1 -- 172.21.15.57:0/1998492702 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d30003970 con 0x7f0d38156ec0 2023-10-19T12:57:25.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.019+0000 7f0d34ff9640 1 -- 172.21.15.57:0/1998492702 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f0d30003ad0 con 0x7f0d38156ec0 2023-10-19T12:57:25.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.019+0000 7f0d3e4c9640 1 -- 172.21.15.57:0/1998492702 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d380c3fa0 con 0x7f0d38156ec0 2023-10-19T12:57:25.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.019+0000 7f0d34ff9640 1 -- 172.21.15.57:0/1998492702 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d30006070 con 0x7f0d38156ec0 2023-10-19T12:57:25.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.019+0000 7f0d3e4c9640 1 -- 172.21.15.57:0/1998492702 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d381408b0 con 0x7f0d38156ec0 2023-10-19T12:57:25.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.020+0000 7f0d3e4c9640 1 -- 172.21.15.57:0/1998492702 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d380c41c0 con 0x7f0d38156ec0 2023-10-19T12:57:25.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.020+0000 7f0d34ff9640 1 -- 172.21.15.57:0/1998492702 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 15) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f0d30031030 con 0x7f0d38156ec0 2023-10-19T12:57:25.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.021+0000 7f0d34ff9640 1 --2- 172.21.15.57:0/1998492702 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f0d2406b040 0x7f0d2406d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:25.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.021+0000 7f0d34ff9640 1 -- 172.21.15.57:0/1998492702 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f0d300aab40 con 0x7f0d38156ec0 2023-10-19T12:57:25.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.021+0000 7f0d37fff640 1 --2- 172.21.15.57:0/1998492702 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f0d2406b040 0x7f0d2406d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:25.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.022+0000 7f0d37fff640 1 --2- 172.21.15.57:0/1998492702 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f0d2406b040 0x7f0d2406d500 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0d28006610 tx=0x7f0d28006b70 comp rx=0 tx=0).ready entity=mgr.5428 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:25.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.025+0000 7f0d34ff9640 1 -- 172.21.15.57:0/1998492702 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0d30075560 con 0x7f0d38156ec0 2023-10-19T12:57:25.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.221+0000 7f0d3e4c9640 1 -- 172.21.15.57:0/1998492702 --> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir3", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0d380690e0 con 0x7f0d2406b040 2023-10-19T12:57:25.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.230+0000 7f0d34ff9640 1 -- 172.21.15.57:0/1998492702 <== mgr.5428 v2:172.21.15.179:6832/922398470 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7f0d380690e0 con 0x7f0d2406b040 2023-10-19T12:57:25.234 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:57:25.234 INFO:teuthology.orchestra.run.smithi057.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir3", "rel_path": "/snap_test_dir1/dir3", "schedule": "1m", "retention": {}, "start": "2023-10-19T00:00:00", "created": "2023-10-19T12:45:58", "first": "2023-10-19T12:46:00", "last": "2023-10-19T12:57:00", "last_pruned": null, "created_count": 12, "pruned_count": 0, "active": false}] 2023-10-19T12:57:25.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.232+0000 7f0d3e4c9640 1 -- 172.21.15.57:0/1998492702 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f0d2406b040 msgr2=0x7f0d2406d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:25.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.232+0000 7f0d3e4c9640 1 --2- 172.21.15.57:0/1998492702 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f0d2406b040 0x7f0d2406d500 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0d28006610 tx=0x7f0d28006b70 comp rx=0 tx=0).stop 2023-10-19T12:57:25.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.232+0000 7f0d3e4c9640 1 -- 172.21.15.57:0/1998492702 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0d38156ec0 msgr2=0x7f0d381ee530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:25.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.232+0000 7f0d3e4c9640 1 --2- 172.21.15.57:0/1998492702 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0d38156ec0 0x7f0d381ee530 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f0d3002cd20 tx=0x7f0d300017b0 comp rx=0 tx=0).stop 2023-10-19T12:57:25.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.232+0000 7f0d3e4c9640 1 -- 172.21.15.57:0/1998492702 shutdown_connections 2023-10-19T12:57:25.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.232+0000 7f0d3e4c9640 1 --2- 172.21.15.57:0/1998492702 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f0d2406b040 0x7f0d2406d500 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:25.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.232+0000 7f0d3e4c9640 1 --2- 172.21.15.57:0/1998492702 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0d38156ec0 0x7f0d381ee530 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:25.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.232+0000 7f0d3e4c9640 1 --2- 172.21.15.57:0/1998492702 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0d380b2520 0x7f0d38140370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:25.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.232+0000 7f0d3e4c9640 1 --2- 172.21.15.57:0/1998492702 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0d20005180 0x7f0d380c44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:25.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.232+0000 7f0d3e4c9640 1 -- 172.21.15.57:0/1998492702 >> 172.21.15.57:0/1998492702 conn(0x7f0d380af7a0 msgr2=0x7f0d380cb950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:25.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.233+0000 7f0d3e4c9640 1 -- 172.21.15.57:0/1998492702 shutdown_connections 2023-10-19T12:57:25.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.233+0000 7f0d3e4c9640 1 -- 172.21.15.57:0/1998492702 wait complete. 2023-10-19T12:57:25.246 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir3", "rel_path": "/snap_test_dir1/dir3", "schedule": "1m", "retention": {}, "start": "2023-10-19T00:00:00", "created": "2023-10-19T12:45:58", "first": "2023-10-19T12:46:00", "last": "2023-10-19T12:57:00", "last_pruned": null, "created_count": 12, "pruned_count": 0, "active": false}] 2023-10-19T12:57:25.246 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir3', 'rel_path': '/snap_test_dir1/dir3', 'schedule': '1m', 'retention': {}, 'start': '2023-10-19T00:00:00', 'created': '2023-10-19T12:45:58', 'first': '2023-10-19T12:46:00', 'last': '2023-10-19T12:57:00', 'last_pruned': None, 'created_count': 12, 'pruned_count': 0, 'active': False} 2023-10-19T12:57:25.247 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 12 2023-10-19T12:57:25.247 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 12 2023-10-19T12:57:25.247 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:25.248 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir4/.snap) 2023-10-19T12:57:25.269 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_46_00_UTC 2023-10-19T12:57:25.270 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_47_00_UTC 2023-10-19T12:57:25.270 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_48_00_UTC 2023-10-19T12:57:25.270 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_49_00_UTC 2023-10-19T12:57:25.271 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_50_00_UTC 2023-10-19T12:57:25.271 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_51_00_UTC 2023-10-19T12:57:25.271 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_52_00_UTC 2023-10-19T12:57:25.272 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_53_01_UTC 2023-10-19T12:57:25.272 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_54_00_UTC 2023-10-19T12:57:25.272 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_55_00_UTC 2023-10-19T12:57:25.272 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_56_00_UTC 2023-10-19T12:57:25.273 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_57_00_UTC 2023-10-19T12:57:25.273 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-19-12_46_00_UTC', 'scheduled-2023-10-19-12_47_00_UTC', 'scheduled-2023-10-19-12_48_00_UTC', 'scheduled-2023-10-19-12_49_00_UTC', 'scheduled-2023-10-19-12_50_00_UTC', 'scheduled-2023-10-19-12_51_00_UTC', 'scheduled-2023-10-19-12_52_00_UTC', 'scheduled-2023-10-19-12_53_01_UTC', 'scheduled-2023-10-19-12_54_00_UTC', 'scheduled-2023-10-19-12_55_00_UTC', 'scheduled-2023-10-19-12_56_00_UTC', 'scheduled-2023-10-19-12_57_00_UTC'] 2023-10-19T12:57:25.274 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir4 2023-10-19T12:57:25.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.419+0000 7f5964801640 1 -- 172.21.15.57:0/759191120 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5960158da0 msgr2=0x7f59600c9340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:25.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.419+0000 7f5964801640 1 --2- 172.21.15.57:0/759191120 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5960158da0 0x7f59600c9340 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f59600715e0 tx=0x7f595002c250 comp rx=0 tx=0).stop 2023-10-19T12:57:25.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.419+0000 7f5964801640 1 -- 172.21.15.57:0/759191120 shutdown_connections 2023-10-19T12:57:25.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.419+0000 7f5964801640 1 --2- 172.21.15.57:0/759191120 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f596006d570 0x7f596006da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:25.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.419+0000 7f5964801640 1 --2- 172.21.15.57:0/759191120 >> v2:172.21.15.179:3300/0 conn(0x7f596006efd0 0x7f59600c9880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:25.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.419+0000 7f5964801640 1 --2- 172.21.15.57:0/759191120 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5960158da0 0x7f59600c9340 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:25.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.419+0000 7f5964801640 1 -- 172.21.15.57:0/759191120 >> 172.21.15.57:0/759191120 conn(0x7f59600b06e0 msgr2=0x7f59600b0af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:25.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.419+0000 7f5964801640 1 -- 172.21.15.57:0/759191120 shutdown_connections 2023-10-19T12:57:25.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.419+0000 7f5964801640 1 -- 172.21.15.57:0/759191120 wait complete. 2023-10-19T12:57:25.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.420+0000 7f5964801640 1 Processor -- start 2023-10-19T12:57:25.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.420+0000 7f5964801640 1 -- start start 2023-10-19T12:57:25.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.421+0000 7f5964801640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f596006d570 0x7f596015b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:25.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.421+0000 7f5964801640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f596006efd0 0x7f596015bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:25.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.421+0000 7f5964801640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5960158da0 0x7f59600b3970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:25.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.421+0000 7f5964801640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f596015a920 con 0x7f596006efd0 2023-10-19T12:57:25.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.421+0000 7f5964801640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f596015c400 con 0x7f596006d570 2023-10-19T12:57:25.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.421+0000 7f5964801640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f596015ad40 con 0x7f5960158da0 2023-10-19T12:57:25.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.421+0000 7f595e59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f596006efd0 0x7f596015bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:25.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.421+0000 7f595e59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f596006efd0 0x7f596015bc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34204/0 (socket says 172.21.15.57:34204) 2023-10-19T12:57:25.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.421+0000 7f595e59c640 1 -- 172.21.15.57:0/551331259 learned_addr learned my addr 172.21.15.57:0/551331259 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:57:25.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.421+0000 7f595e59c640 1 -- 172.21.15.57:0/551331259 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5960158da0 msgr2=0x7f59600b3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:25.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.421+0000 7f595e59c640 1 --2- 172.21.15.57:0/551331259 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5960158da0 0x7f59600b3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:25.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.421+0000 7f595e59c640 1 -- 172.21.15.57:0/551331259 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f596006d570 msgr2=0x7f596015b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:25.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.421+0000 7f595ed9d640 1 --2- 172.21.15.57:0/551331259 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f596006d570 0x7f596015b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:25.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.421+0000 7f595e59c640 1 --2- 172.21.15.57:0/551331259 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f596006d570 0x7f596015b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:25.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.421+0000 7f595e59c640 1 -- 172.21.15.57:0/551331259 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5950009900 con 0x7f596006efd0 2023-10-19T12:57:25.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.421+0000 7f595ed9d640 1 --2- 172.21.15.57:0/551331259 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f596006d570 0x7f596015b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:57:25.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.422+0000 7f595e59c640 1 --2- 172.21.15.57:0/551331259 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f596006efd0 0x7f596015bc20 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f595400b480 tx=0x7f595400b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:25.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.422+0000 7f59477fe640 1 -- 172.21.15.57:0/551331259 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5954010870 con 0x7f596006efd0 2023-10-19T12:57:25.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.422+0000 7f59477fe640 1 -- 172.21.15.57:0/551331259 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f5954010ac0 con 0x7f596006efd0 2023-10-19T12:57:25.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.422+0000 7f59477fe640 1 -- 172.21.15.57:0/551331259 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f595400f3c0 con 0x7f596006efd0 2023-10-19T12:57:25.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.422+0000 7f5964801640 1 -- 172.21.15.57:0/551331259 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f596015c160 con 0x7f596006efd0 2023-10-19T12:57:25.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.422+0000 7f5964801640 1 -- 172.21.15.57:0/551331259 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f596015a260 con 0x7f596006efd0 2023-10-19T12:57:25.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.423+0000 7f5964801640 1 -- 172.21.15.57:0/551331259 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f596015aa60 con 0x7f596006efd0 2023-10-19T12:57:25.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.423+0000 7f59477fe640 1 -- 172.21.15.57:0/551331259 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 15) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f595400f520 con 0x7f596006efd0 2023-10-19T12:57:25.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.424+0000 7f59477fe640 1 --2- 172.21.15.57:0/551331259 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f594c06b040 0x7f594c06d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:25.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.424+0000 7f59477fe640 1 -- 172.21.15.57:0/551331259 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f5954089db0 con 0x7f596006efd0 2023-10-19T12:57:25.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.424+0000 7f595ed9d640 1 --2- 172.21.15.57:0/551331259 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f594c06b040 0x7f594c06d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:25.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.424+0000 7f595ed9d640 1 --2- 172.21.15.57:0/551331259 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f594c06b040 0x7f594c06d500 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5950036000 tx=0x7f5950004ab0 comp rx=0 tx=0).ready entity=mgr.5428 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:25.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.428+0000 7f59477fe640 1 -- 172.21.15.57:0/551331259 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f596015aa60 con 0x7f596006efd0 2023-10-19T12:57:25.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.624+0000 7f5964801640 1 -- 172.21.15.57:0/551331259 --> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir4", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f596006df40 con 0x7f594c06b040 2023-10-19T12:57:25.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.633+0000 7f59477fe640 1 -- 172.21.15.57:0/551331259 <== mgr.5428 v2:172.21.15.179:6832/922398470 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7f5950003a70 con 0x7f594c06b040 2023-10-19T12:57:25.636 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:57:25.637 INFO:teuthology.orchestra.run.smithi057.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir4", "rel_path": "/snap_test_dir1/dir4", "schedule": "1m", "retention": {}, "start": "2023-10-19T00:00:00", "created": "2023-10-19T12:45:59", "first": "2023-10-19T12:46:00", "last": "2023-10-19T12:57:00", "last_pruned": null, "created_count": 12, "pruned_count": 0, "active": false}] 2023-10-19T12:57:25.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.636+0000 7f5964801640 1 -- 172.21.15.57:0/551331259 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f594c06b040 msgr2=0x7f594c06d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:25.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.636+0000 7f5964801640 1 --2- 172.21.15.57:0/551331259 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f594c06b040 0x7f594c06d500 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5950036000 tx=0x7f5950004ab0 comp rx=0 tx=0).stop 2023-10-19T12:57:25.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.636+0000 7f5964801640 1 -- 172.21.15.57:0/551331259 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f596006efd0 msgr2=0x7f596015bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:25.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.636+0000 7f5964801640 1 --2- 172.21.15.57:0/551331259 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f596006efd0 0x7f596015bc20 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f595400b480 tx=0x7f595400b940 comp rx=0 tx=0).stop 2023-10-19T12:57:25.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.636+0000 7f5964801640 1 -- 172.21.15.57:0/551331259 shutdown_connections 2023-10-19T12:57:25.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.636+0000 7f5964801640 1 --2- 172.21.15.57:0/551331259 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f594c06b040 0x7f594c06d500 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:25.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.636+0000 7f5964801640 1 --2- 172.21.15.57:0/551331259 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5960158da0 0x7f59600b3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:25.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.636+0000 7f5964801640 1 --2- 172.21.15.57:0/551331259 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f596006efd0 0x7f596015bc20 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:25.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.636+0000 7f5964801640 1 --2- 172.21.15.57:0/551331259 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f596006d570 0x7f596015b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:25.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.636+0000 7f5964801640 1 -- 172.21.15.57:0/551331259 >> 172.21.15.57:0/551331259 conn(0x7f59600b06e0 msgr2=0x7f59600b0ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:25.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.636+0000 7f5964801640 1 -- 172.21.15.57:0/551331259 shutdown_connections 2023-10-19T12:57:25.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.636+0000 7f5964801640 1 -- 172.21.15.57:0/551331259 wait complete. 2023-10-19T12:57:25.649 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir4", "rel_path": "/snap_test_dir1/dir4", "schedule": "1m", "retention": {}, "start": "2023-10-19T00:00:00", "created": "2023-10-19T12:45:59", "first": "2023-10-19T12:46:00", "last": "2023-10-19T12:57:00", "last_pruned": null, "created_count": 12, "pruned_count": 0, "active": false}] 2023-10-19T12:57:25.649 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir4', 'rel_path': '/snap_test_dir1/dir4', 'schedule': '1m', 'retention': {}, 'start': '2023-10-19T00:00:00', 'created': '2023-10-19T12:45:59', 'first': '2023-10-19T12:46:00', 'last': '2023-10-19T12:57:00', 'last_pruned': None, 'created_count': 12, 'pruned_count': 0, 'active': False} 2023-10-19T12:57:25.650 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 12 2023-10-19T12:57:25.650 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 12 2023-10-19T12:57:25.650 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:25.651 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir5/.snap) 2023-10-19T12:57:25.672 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_46_00_UTC 2023-10-19T12:57:25.672 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_47_00_UTC 2023-10-19T12:57:25.672 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_48_00_UTC 2023-10-19T12:57:25.673 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_49_00_UTC 2023-10-19T12:57:25.673 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_50_00_UTC 2023-10-19T12:57:25.673 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_51_00_UTC 2023-10-19T12:57:25.673 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_52_00_UTC 2023-10-19T12:57:25.673 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_53_00_UTC 2023-10-19T12:57:25.673 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_54_00_UTC 2023-10-19T12:57:25.674 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_55_00_UTC 2023-10-19T12:57:25.674 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_56_00_UTC 2023-10-19T12:57:25.674 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_57_00_UTC 2023-10-19T12:57:25.675 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-19-12_46_00_UTC', 'scheduled-2023-10-19-12_47_00_UTC', 'scheduled-2023-10-19-12_48_00_UTC', 'scheduled-2023-10-19-12_49_00_UTC', 'scheduled-2023-10-19-12_50_00_UTC', 'scheduled-2023-10-19-12_51_00_UTC', 'scheduled-2023-10-19-12_52_00_UTC', 'scheduled-2023-10-19-12_53_00_UTC', 'scheduled-2023-10-19-12_54_00_UTC', 'scheduled-2023-10-19-12_55_00_UTC', 'scheduled-2023-10-19-12_56_00_UTC', 'scheduled-2023-10-19-12_57_00_UTC'] 2023-10-19T12:57:25.675 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir5 2023-10-19T12:57:25.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.820+0000 7fc39e1d0640 1 -- 172.21.15.57:0/1299194778 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc3980ad370 msgr2=0x7fc3980ad800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:25.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.820+0000 7fc39e1d0640 1 --2- 172.21.15.57:0/1299194778 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc3980ad370 0x7fc3980ad800 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fc39806fc50 tx=0x7fc39002c110 comp rx=0 tx=0).stop 2023-10-19T12:57:25.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.821+0000 7fc39e1d0640 1 -- 172.21.15.57:0/1299194778 shutdown_connections 2023-10-19T12:57:25.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.821+0000 7fc39e1d0640 1 --2- 172.21.15.57:0/1299194778 >> v2:172.21.15.179:3300/0 conn(0x7fc39806f0a0 0x7fc39806ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:25.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.821+0000 7fc39e1d0640 1 --2- 172.21.15.57:0/1299194778 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc3980ad370 0x7fc3980ad800 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:25.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.821+0000 7fc39e1d0640 1 -- 172.21.15.57:0/1299194778 >> 172.21.15.57:0/1299194778 conn(0x7fc3980b0670 msgr2=0x7fc3980b0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:25.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.821+0000 7fc39e1d0640 1 -- 172.21.15.57:0/1299194778 shutdown_connections 2023-10-19T12:57:25.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.821+0000 7fc39e1d0640 1 -- 172.21.15.57:0/1299194778 wait complete. 2023-10-19T12:57:25.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.822+0000 7fc39e1d0640 1 Processor -- start 2023-10-19T12:57:25.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.822+0000 7fc39e1d0640 1 -- start start 2023-10-19T12:57:25.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.823+0000 7fc39e1d0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc39806d2d0 0x7fc3980ca810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:25.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.823+0000 7fc39e1d0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc39806f0a0 0x7fc3981e41c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:25.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.823+0000 7fc39e1d0640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc3980ad370 0x7fc398158e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:25.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.823+0000 7fc39e1d0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3980afdd0 con 0x7fc39806d2d0 2023-10-19T12:57:25.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.823+0000 7fc39e1d0640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3981e4700 con 0x7fc39806f0a0 2023-10-19T12:57:25.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.823+0000 7fc39e1d0640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3980af150 con 0x7fc3980ad370 2023-10-19T12:57:25.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.823+0000 7fc3977fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc39806d2d0 0x7fc3980ca810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:25.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.823+0000 7fc3977fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc39806d2d0 0x7fc3980ca810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34226/0 (socket says 172.21.15.57:34226) 2023-10-19T12:57:25.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.823+0000 7fc3977fe640 1 -- 172.21.15.57:0/1101951432 learned_addr learned my addr 172.21.15.57:0/1101951432 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:57:25.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.823+0000 7fc396ffd640 1 --2- 172.21.15.57:0/1101951432 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc39806f0a0 0x7fc3981e41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:25.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.823+0000 7fc397fff640 1 --2- 172.21.15.57:0/1101951432 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc3980ad370 0x7fc398158e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:25.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.823+0000 7fc3977fe640 1 -- 172.21.15.57:0/1101951432 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc3980ad370 msgr2=0x7fc398158e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:25.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.823+0000 7fc3977fe640 1 --2- 172.21.15.57:0/1101951432 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc3980ad370 0x7fc398158e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:25.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.823+0000 7fc3977fe640 1 -- 172.21.15.57:0/1101951432 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc39806f0a0 msgr2=0x7fc3981e41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:25.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.823+0000 7fc3977fe640 1 --2- 172.21.15.57:0/1101951432 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc39806f0a0 0x7fc3981e41c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:25.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.823+0000 7fc3977fe640 1 -- 172.21.15.57:0/1101951432 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc390009840 con 0x7fc39806d2d0 2023-10-19T12:57:25.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.823+0000 7fc397fff640 1 --2- 172.21.15.57:0/1101951432 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc3980ad370 0x7fc398158e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:57:25.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.823+0000 7fc3977fe640 1 --2- 172.21.15.57:0/1101951432 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc39806d2d0 0x7fc3980ca810 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fc38000c4b0 tx=0x7fc38000c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:25.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.823+0000 7fc38ffff640 1 -- 172.21.15.57:0/1101951432 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc380005480 con 0x7fc39806d2d0 2023-10-19T12:57:25.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.823+0000 7fc38ffff640 1 -- 172.21.15.57:0/1101951432 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fc3800036e0 con 0x7fc39806d2d0 2023-10-19T12:57:25.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.824+0000 7fc39e1d0640 1 -- 172.21.15.57:0/1101951432 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3980af430 con 0x7fc39806d2d0 2023-10-19T12:57:25.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.824+0000 7fc38ffff640 1 -- 172.21.15.57:0/1101951432 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc380003b30 con 0x7fc39806d2d0 2023-10-19T12:57:25.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.824+0000 7fc39e1d0640 1 -- 172.21.15.57:0/1101951432 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3980af9f0 con 0x7fc39806d2d0 2023-10-19T12:57:25.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.825+0000 7fc39e1d0640 1 -- 172.21.15.57:0/1101951432 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3980ce1d0 con 0x7fc39806d2d0 2023-10-19T12:57:25.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.825+0000 7fc38ffff640 1 -- 172.21.15.57:0/1101951432 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 15) v1 ==== 89417+0+0 (secure 0 0 0) 0x7fc380003c90 con 0x7fc39806d2d0 2023-10-19T12:57:25.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.825+0000 7fc38ffff640 1 --2- 172.21.15.57:0/1101951432 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fc37806b040 0x7fc37806d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:25.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.825+0000 7fc38ffff640 1 -- 172.21.15.57:0/1101951432 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7fc3800892d0 con 0x7fc39806d2d0 2023-10-19T12:57:25.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.826+0000 7fc396ffd640 1 --2- 172.21.15.57:0/1101951432 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fc37806b040 0x7fc37806d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:25.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.826+0000 7fc396ffd640 1 --2- 172.21.15.57:0/1101951432 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fc37806b040 0x7fc37806d500 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc3980cad50 tx=0x7fc38800b390 comp rx=0 tx=0).ready entity=mgr.5428 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:25.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:25.829+0000 7fc38ffff640 1 -- 172.21.15.57:0/1101951432 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc380053fe0 con 0x7fc39806d2d0 2023-10-19T12:57:26.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.025+0000 7fc39e1d0640 1 -- 172.21.15.57:0/1101951432 --> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir5", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc3980c2630 con 0x7fc37806b040 2023-10-19T12:57:26.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.036+0000 7fc38ffff640 1 -- 172.21.15.57:0/1101951432 <== mgr.5428 v2:172.21.15.179:6832/922398470 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7fc388015070 con 0x7fc37806b040 2023-10-19T12:57:26.039 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:57:26.039 INFO:teuthology.orchestra.run.smithi057.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir5", "rel_path": "/snap_test_dir1/dir5", "schedule": "1m", "retention": {}, "start": "2023-10-19T00:00:00", "created": "2023-10-19T12:45:59", "first": "2023-10-19T12:46:00", "last": "2023-10-19T12:57:00", "last_pruned": null, "created_count": 12, "pruned_count": 0, "active": false}] 2023-10-19T12:57:26.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.038+0000 7fc39e1d0640 1 -- 172.21.15.57:0/1101951432 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fc37806b040 msgr2=0x7fc37806d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:26.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.038+0000 7fc39e1d0640 1 --2- 172.21.15.57:0/1101951432 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fc37806b040 0x7fc37806d500 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc3980cad50 tx=0x7fc38800b390 comp rx=0 tx=0).stop 2023-10-19T12:57:26.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.038+0000 7fc39e1d0640 1 -- 172.21.15.57:0/1101951432 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc39806d2d0 msgr2=0x7fc3980ca810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:26.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.038+0000 7fc39e1d0640 1 --2- 172.21.15.57:0/1101951432 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc39806d2d0 0x7fc3980ca810 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fc38000c4b0 tx=0x7fc38000c970 comp rx=0 tx=0).stop 2023-10-19T12:57:26.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.038+0000 7fc39e1d0640 1 -- 172.21.15.57:0/1101951432 shutdown_connections 2023-10-19T12:57:26.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.038+0000 7fc39e1d0640 1 --2- 172.21.15.57:0/1101951432 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fc37806b040 0x7fc37806d500 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:26.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.038+0000 7fc39e1d0640 1 --2- 172.21.15.57:0/1101951432 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc3980ad370 0x7fc398158e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:26.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.038+0000 7fc39e1d0640 1 --2- 172.21.15.57:0/1101951432 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc39806f0a0 0x7fc3981e41c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:26.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.038+0000 7fc39e1d0640 1 --2- 172.21.15.57:0/1101951432 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc39806d2d0 0x7fc3980ca810 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:26.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.038+0000 7fc39e1d0640 1 -- 172.21.15.57:0/1101951432 >> 172.21.15.57:0/1101951432 conn(0x7fc3980b0670 msgr2=0x7fc398158570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:26.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.038+0000 7fc39e1d0640 1 -- 172.21.15.57:0/1101951432 shutdown_connections 2023-10-19T12:57:26.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.038+0000 7fc39e1d0640 1 -- 172.21.15.57:0/1101951432 wait complete. 2023-10-19T12:57:26.051 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir5", "rel_path": "/snap_test_dir1/dir5", "schedule": "1m", "retention": {}, "start": "2023-10-19T00:00:00", "created": "2023-10-19T12:45:59", "first": "2023-10-19T12:46:00", "last": "2023-10-19T12:57:00", "last_pruned": null, "created_count": 12, "pruned_count": 0, "active": false}] 2023-10-19T12:57:26.051 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir5', 'rel_path': '/snap_test_dir1/dir5', 'schedule': '1m', 'retention': {}, 'start': '2023-10-19T00:00:00', 'created': '2023-10-19T12:45:59', 'first': '2023-10-19T12:46:00', 'last': '2023-10-19T12:57:00', 'last_pruned': None, 'created_count': 12, 'pruned_count': 0, 'active': False} 2023-10-19T12:57:26.051 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 12 2023-10-19T12:57:26.052 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 12 2023-10-19T12:57:26.052 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:26.053 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir6/.snap) 2023-10-19T12:57:26.073 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_47_00_UTC 2023-10-19T12:57:26.074 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_48_00_UTC 2023-10-19T12:57:26.074 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_49_00_UTC 2023-10-19T12:57:26.074 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_50_00_UTC 2023-10-19T12:57:26.074 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_51_00_UTC 2023-10-19T12:57:26.075 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_52_00_UTC 2023-10-19T12:57:26.075 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_53_00_UTC 2023-10-19T12:57:26.075 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_54_00_UTC 2023-10-19T12:57:26.075 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_55_00_UTC 2023-10-19T12:57:26.076 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_56_01_UTC 2023-10-19T12:57:26.076 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_57_00_UTC 2023-10-19T12:57:26.076 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-19-12_47_00_UTC', 'scheduled-2023-10-19-12_48_00_UTC', 'scheduled-2023-10-19-12_49_00_UTC', 'scheduled-2023-10-19-12_50_00_UTC', 'scheduled-2023-10-19-12_51_00_UTC', 'scheduled-2023-10-19-12_52_00_UTC', 'scheduled-2023-10-19-12_53_00_UTC', 'scheduled-2023-10-19-12_54_00_UTC', 'scheduled-2023-10-19-12_55_00_UTC', 'scheduled-2023-10-19-12_56_01_UTC', 'scheduled-2023-10-19-12_57_00_UTC'] 2023-10-19T12:57:26.077 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir6 2023-10-19T12:57:26.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.223+0000 7f83f056e640 1 -- 172.21.15.57:0/2288068247 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f83d4004a80 msgr2=0x7f83d4004ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:26.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.223+0000 7f83f056e640 1 --2- 172.21.15.57:0/2288068247 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f83d4004a80 0x7f83d4004ea0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f83e0009b50 tx=0x7f83d4006900 comp rx=0 tx=0).stop 2023-10-19T12:57:26.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.223+0000 7f83eeb0b640 1 -- 172.21.15.57:0/2288068247 reap_dead start 2023-10-19T12:57:26.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.223+0000 7f83f056e640 1 -- 172.21.15.57:0/2288068247 shutdown_connections 2023-10-19T12:57:26.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.223+0000 7f83f056e640 1 -- 172.21.15.57:0/2288068247 >> 172.21.15.57:0/2288068247 conn(0x7f83e8069930 msgr2=0x7f83e8069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:26.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.223+0000 7f83f056e640 1 -- 172.21.15.57:0/2288068247 shutdown_connections 2023-10-19T12:57:26.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.223+0000 7f83f056e640 1 -- 172.21.15.57:0/2288068247 wait complete. 2023-10-19T12:57:26.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.224+0000 7f83f056e640 1 Processor -- start 2023-10-19T12:57:26.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.224+0000 7f83f056e640 1 -- start start 2023-10-19T12:57:26.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.225+0000 7f83f056e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f83d4004a80 0x7f83e815a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:26.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.225+0000 7f83f056e640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f83e814bfe0 0x7f83e813a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:26.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.225+0000 7f83f056e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f83e8158980 0x7f83e81e50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:26.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.225+0000 7f83f056e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83e81592f0 con 0x7f83e8158980 2023-10-19T12:57:26.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.225+0000 7f83f056e640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83e81595d0 con 0x7f83d4004a80 2023-10-19T12:57:26.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.225+0000 7f83f056e640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83e81598b0 con 0x7f83e814bfe0 2023-10-19T12:57:26.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.225+0000 7f83eeb0b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f83e8158980 0x7f83e81e50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:26.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.225+0000 7f83eeb0b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f83e8158980 0x7f83e81e50a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34234/0 (socket says 172.21.15.57:34234) 2023-10-19T12:57:26.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.225+0000 7f83eeb0b640 1 -- 172.21.15.57:0/1003839575 learned_addr learned my addr 172.21.15.57:0/1003839575 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:57:26.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.225+0000 7f83edb09640 1 --2- 172.21.15.57:0/1003839575 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f83e814bfe0 0x7f83e813a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:26.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.225+0000 7f83ee30a640 1 --2- 172.21.15.57:0/1003839575 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f83d4004a80 0x7f83e815a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:26.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.225+0000 7f83eeb0b640 1 -- 172.21.15.57:0/1003839575 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f83e814bfe0 msgr2=0x7f83e813a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:26.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.225+0000 7f83eeb0b640 1 --2- 172.21.15.57:0/1003839575 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f83e814bfe0 0x7f83e813a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:26.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.225+0000 7f83eeb0b640 1 -- 172.21.15.57:0/1003839575 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f83d4004a80 msgr2=0x7f83e815a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:26.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.225+0000 7f83eeb0b640 1 --2- 172.21.15.57:0/1003839575 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f83d4004a80 0x7f83e815a390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:26.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.225+0000 7f83eeb0b640 1 -- 172.21.15.57:0/1003839575 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83e8159b90 con 0x7f83e8158980 2023-10-19T12:57:26.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.225+0000 7f83eeb0b640 1 --2- 172.21.15.57:0/1003839575 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f83e8158980 0x7f83e81e50a0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f83e4001580 tx=0x7f83e400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:26.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.226+0000 7f83deffd640 1 -- 172.21.15.57:0/1003839575 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83e400cba0 con 0x7f83e8158980 2023-10-19T12:57:26.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.226+0000 7f83deffd640 1 -- 172.21.15.57:0/1003839575 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f83e400ce30 con 0x7f83e8158980 2023-10-19T12:57:26.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.226+0000 7f83f056e640 1 -- 172.21.15.57:0/1003839575 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83e8159e70 con 0x7f83e8158980 2023-10-19T12:57:26.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.226+0000 7f83deffd640 1 -- 172.21.15.57:0/1003839575 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83e40116c0 con 0x7f83e8158980 2023-10-19T12:57:26.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.226+0000 7f83f056e640 1 -- 172.21.15.57:0/1003839575 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83e8158de0 con 0x7f83e8158980 2023-10-19T12:57:26.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.227+0000 7f83f056e640 1 -- 172.21.15.57:0/1003839575 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83e8159060 con 0x7f83e8158980 2023-10-19T12:57:26.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.227+0000 7f83deffd640 1 -- 172.21.15.57:0/1003839575 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 15) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f83e4011820 con 0x7f83e8158980 2023-10-19T12:57:26.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.227+0000 7f83deffd640 1 --2- 172.21.15.57:0/1003839575 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f83cc06b070 0x7f83cc06d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:26.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.228+0000 7f83deffd640 1 -- 172.21.15.57:0/1003839575 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f83e408a260 con 0x7f83e8158980 2023-10-19T12:57:26.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.228+0000 7f83ee30a640 1 --2- 172.21.15.57:0/1003839575 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f83cc06b070 0x7f83cc06d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:26.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.228+0000 7f83ee30a640 1 --2- 172.21.15.57:0/1003839575 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f83cc06b070 0x7f83cc06d530 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f83e0036000 tx=0x7f83e0035040 comp rx=0 tx=0).ready entity=mgr.5428 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:26.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.232+0000 7f83deffd640 1 -- 172.21.15.57:0/1003839575 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f83e4054f10 con 0x7f83e8158980 2023-10-19T12:57:26.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.431+0000 7f83f056e640 1 -- 172.21.15.57:0/1003839575 --> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir6", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f83e815a060 con 0x7f83cc06b070 2023-10-19T12:57:26.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.440+0000 7f83deffd640 1 -- 172.21.15.57:0/1003839575 <== mgr.5428 v2:172.21.15.179:6832/922398470 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7f83e815a060 con 0x7f83cc06b070 2023-10-19T12:57:26.443 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:57:26.444 INFO:teuthology.orchestra.run.smithi057.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir6", "rel_path": "/snap_test_dir1/dir6", "schedule": "1m", "retention": {}, "start": "2023-10-19T00:00:00", "created": "2023-10-19T12:46:00", "first": "2023-10-19T12:47:00", "last": "2023-10-19T12:57:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-19T12:57:26.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.442+0000 7f83f056e640 1 -- 172.21.15.57:0/1003839575 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f83cc06b070 msgr2=0x7f83cc06d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:26.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.442+0000 7f83f056e640 1 --2- 172.21.15.57:0/1003839575 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f83cc06b070 0x7f83cc06d530 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f83e0036000 tx=0x7f83e0035040 comp rx=0 tx=0).stop 2023-10-19T12:57:26.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.442+0000 7f83f056e640 1 -- 172.21.15.57:0/1003839575 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f83e8158980 msgr2=0x7f83e81e50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:26.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.442+0000 7f83f056e640 1 --2- 172.21.15.57:0/1003839575 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f83e8158980 0x7f83e81e50a0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f83e4001580 tx=0x7f83e400ed30 comp rx=0 tx=0).stop 2023-10-19T12:57:26.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.442+0000 7f83f056e640 1 -- 172.21.15.57:0/1003839575 shutdown_connections 2023-10-19T12:57:26.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.442+0000 7f83f056e640 1 --2- 172.21.15.57:0/1003839575 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f83cc06b070 0x7f83cc06d530 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:26.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.442+0000 7f83f056e640 1 --2- 172.21.15.57:0/1003839575 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f83e8158980 0x7f83e81e50a0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:26.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.442+0000 7f83f056e640 1 --2- 172.21.15.57:0/1003839575 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f83e814bfe0 0x7f83e813a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:26.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.442+0000 7f83f056e640 1 --2- 172.21.15.57:0/1003839575 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f83d4004a80 0x7f83e815a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:26.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.442+0000 7f83f056e640 1 -- 172.21.15.57:0/1003839575 >> 172.21.15.57:0/1003839575 conn(0x7f83e8069930 msgr2=0x7f83e80caba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:26.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.443+0000 7f83f056e640 1 -- 172.21.15.57:0/1003839575 shutdown_connections 2023-10-19T12:57:26.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.443+0000 7f83f056e640 1 -- 172.21.15.57:0/1003839575 wait complete. 2023-10-19T12:57:26.456 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir6", "rel_path": "/snap_test_dir1/dir6", "schedule": "1m", "retention": {}, "start": "2023-10-19T00:00:00", "created": "2023-10-19T12:46:00", "first": "2023-10-19T12:47:00", "last": "2023-10-19T12:57:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-19T12:57:26.456 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir6', 'rel_path': '/snap_test_dir1/dir6', 'schedule': '1m', 'retention': {}, 'start': '2023-10-19T00:00:00', 'created': '2023-10-19T12:46:00', 'first': '2023-10-19T12:47:00', 'last': '2023-10-19T12:57:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-10-19T12:57:26.457 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-10-19T12:57:26.457 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-10-19T12:57:26.458 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:26.458 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir7/.snap) 2023-10-19T12:57:26.479 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_47_00_UTC 2023-10-19T12:57:26.480 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_48_00_UTC 2023-10-19T12:57:26.480 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_49_00_UTC 2023-10-19T12:57:26.480 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_50_00_UTC 2023-10-19T12:57:26.480 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_51_00_UTC 2023-10-19T12:57:26.480 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_52_00_UTC 2023-10-19T12:57:26.481 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_53_00_UTC 2023-10-19T12:57:26.481 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_54_00_UTC 2023-10-19T12:57:26.481 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_55_00_UTC 2023-10-19T12:57:26.481 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_56_00_UTC 2023-10-19T12:57:26.481 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_57_00_UTC 2023-10-19T12:57:26.482 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-19-12_47_00_UTC', 'scheduled-2023-10-19-12_48_00_UTC', 'scheduled-2023-10-19-12_49_00_UTC', 'scheduled-2023-10-19-12_50_00_UTC', 'scheduled-2023-10-19-12_51_00_UTC', 'scheduled-2023-10-19-12_52_00_UTC', 'scheduled-2023-10-19-12_53_00_UTC', 'scheduled-2023-10-19-12_54_00_UTC', 'scheduled-2023-10-19-12_55_00_UTC', 'scheduled-2023-10-19-12_56_00_UTC', 'scheduled-2023-10-19-12_57_00_UTC'] 2023-10-19T12:57:26.482 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir7 2023-10-19T12:57:26.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.627+0000 7f091dc75640 1 -- 172.21.15.57:0/4154512150 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f09000047c0 msgr2=0x7f0900004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:26.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.627+0000 7f091dc75640 1 --2- 172.21.15.57:0/4154512150 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f09000047c0 0x7f0900004c40 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f090c009b50 tx=0x7f09000057f0 comp rx=0 tx=0).stop 2023-10-19T12:57:26.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.628+0000 7f091dc75640 1 -- 172.21.15.57:0/4154512150 shutdown_connections 2023-10-19T12:57:26.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.628+0000 7f091dc75640 1 --2- 172.21.15.57:0/4154512150 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f09000047c0 0x7f0900004c40 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:26.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.628+0000 7f091dc75640 1 -- 172.21.15.57:0/4154512150 >> 172.21.15.57:0/4154512150 conn(0x7f09180b0670 msgr2=0x7f09180b0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:26.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.628+0000 7f091dc75640 1 -- 172.21.15.57:0/4154512150 shutdown_connections 2023-10-19T12:57:26.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.628+0000 7f091dc75640 1 -- 172.21.15.57:0/4154512150 wait complete. 2023-10-19T12:57:26.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.629+0000 7f091dc75640 1 Processor -- start 2023-10-19T12:57:26.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.629+0000 7f091dc75640 1 -- start start 2023-10-19T12:57:26.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.630+0000 7f091dc75640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f09000047c0 0x7f09181531b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:26.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.630+0000 7f091dc75640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f091814f910 0x7f0918150940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:26.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.630+0000 7f091dc75640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0918150e80 0x7f09180c9580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:26.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.630+0000 7f091dc75640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0918152a80 con 0x7f0918150e80 2023-10-19T12:57:26.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.630+0000 7f091dc75640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09181512e0 con 0x7f091814f910 2023-10-19T12:57:26.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.630+0000 7f091dc75640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0918151510 con 0x7f09000047c0 2023-10-19T12:57:26.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.630+0000 7f0917fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0918150e80 0x7f09180c9580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:26.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.630+0000 7f0917fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0918150e80 0x7f09180c9580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:54174/0 (socket says 172.21.15.57:54174) 2023-10-19T12:57:26.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.630+0000 7f0917fff640 1 -- 172.21.15.57:0/756966246 learned_addr learned my addr 172.21.15.57:0/756966246 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:57:26.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.630+0000 7f09177fe640 1 --2- 172.21.15.57:0/756966246 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f09000047c0 0x7f09181531b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:26.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.630+0000 7f0916ffd640 1 --2- 172.21.15.57:0/756966246 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f091814f910 0x7f0918150940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:26.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.630+0000 7f0917fff640 1 -- 172.21.15.57:0/756966246 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f09000047c0 msgr2=0x7f09181531b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:26.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.630+0000 7f0917fff640 1 --2- 172.21.15.57:0/756966246 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f09000047c0 0x7f09181531b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:26.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.630+0000 7f0917fff640 1 -- 172.21.15.57:0/756966246 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f091814f910 msgr2=0x7f0918150940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:26.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.630+0000 7f0917fff640 1 --2- 172.21.15.57:0/756966246 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f091814f910 0x7f0918150940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:26.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.630+0000 7f0917fff640 1 -- 172.21.15.57:0/756966246 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09181517f0 con 0x7f0918150e80 2023-10-19T12:57:26.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.630+0000 7f0917fff640 1 --2- 172.21.15.57:0/756966246 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0918150e80 0x7f09180c9580 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f09180670f0 tx=0x7f0910006430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:26.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.630+0000 7f08fffff640 1 -- 172.21.15.57:0/756966246 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0910011900 con 0x7f0918150e80 2023-10-19T12:57:26.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.630+0000 7f08fffff640 1 -- 172.21.15.57:0/756966246 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f0910011a60 con 0x7f0918150e80 2023-10-19T12:57:26.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.630+0000 7f08fffff640 1 -- 172.21.15.57:0/756966246 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0910011df0 con 0x7f0918150e80 2023-10-19T12:57:26.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.631+0000 7f091dc75640 1 -- 172.21.15.57:0/756966246 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0918151ad0 con 0x7f0918150e80 2023-10-19T12:57:26.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.631+0000 7f091dc75640 1 -- 172.21.15.57:0/756966246 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09181536f0 con 0x7f0918150e80 2023-10-19T12:57:26.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.631+0000 7f091dc75640 1 -- 172.21.15.57:0/756966246 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09180ce1d0 con 0x7f0918150e80 2023-10-19T12:57:26.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.632+0000 7f08fffff640 1 -- 172.21.15.57:0/756966246 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 15) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f091001c460 con 0x7f0918150e80 2023-10-19T12:57:26.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.632+0000 7f08fffff640 1 --2- 172.21.15.57:0/756966246 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f090406b040 0x7f090406d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:26.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.633+0000 7f08fffff640 1 -- 172.21.15.57:0/756966246 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f091008a880 con 0x7f0918150e80 2023-10-19T12:57:26.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.633+0000 7f09177fe640 1 --2- 172.21.15.57:0/756966246 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f090406b040 0x7f090406d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:26.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.633+0000 7f09177fe640 1 --2- 172.21.15.57:0/756966246 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f090406b040 0x7f090406d500 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0908006fd0 tx=0x7f0908006b30 comp rx=0 tx=0).ready entity=mgr.5428 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:26.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.637+0000 7f08fffff640 1 -- 172.21.15.57:0/756966246 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f09100552a0 con 0x7f0918150e80 2023-10-19T12:57:26.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.836+0000 7f091dc75640 1 -- 172.21.15.57:0/756966246 --> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir7", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f09180c2630 con 0x7f090406b040 2023-10-19T12:57:26.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.846+0000 7f08fffff640 1 -- 172.21.15.57:0/756966246 <== mgr.5428 v2:172.21.15.179:6832/922398470 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7f0908012640 con 0x7f090406b040 2023-10-19T12:57:26.849 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:57:26.849 INFO:teuthology.orchestra.run.smithi057.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir7", "rel_path": "/snap_test_dir1/dir7", "schedule": "1m", "retention": {}, "start": "2023-10-19T00:00:00", "created": "2023-10-19T12:46:00", "first": "2023-10-19T12:47:00", "last": "2023-10-19T12:57:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-19T12:57:26.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.848+0000 7f091dc75640 1 -- 172.21.15.57:0/756966246 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f090406b040 msgr2=0x7f090406d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:26.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.848+0000 7f091dc75640 1 --2- 172.21.15.57:0/756966246 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f090406b040 0x7f090406d500 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0908006fd0 tx=0x7f0908006b30 comp rx=0 tx=0).stop 2023-10-19T12:57:26.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.848+0000 7f091dc75640 1 -- 172.21.15.57:0/756966246 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0918150e80 msgr2=0x7f09180c9580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:26.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.848+0000 7f091dc75640 1 --2- 172.21.15.57:0/756966246 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0918150e80 0x7f09180c9580 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f09180670f0 tx=0x7f0910006430 comp rx=0 tx=0).stop 2023-10-19T12:57:26.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.848+0000 7f091dc75640 1 -- 172.21.15.57:0/756966246 shutdown_connections 2023-10-19T12:57:26.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.848+0000 7f091dc75640 1 --2- 172.21.15.57:0/756966246 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f090406b040 0x7f090406d500 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:26.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.848+0000 7f091dc75640 1 --2- 172.21.15.57:0/756966246 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0918150e80 0x7f09180c9580 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:26.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.848+0000 7f091dc75640 1 --2- 172.21.15.57:0/756966246 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f091814f910 0x7f0918150940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:26.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.848+0000 7f091dc75640 1 --2- 172.21.15.57:0/756966246 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f09000047c0 0x7f09181531b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:26.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.848+0000 7f091dc75640 1 -- 172.21.15.57:0/756966246 >> 172.21.15.57:0/756966246 conn(0x7f09180b0670 msgr2=0x7f09180c48e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:26.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.848+0000 7f091dc75640 1 -- 172.21.15.57:0/756966246 shutdown_connections 2023-10-19T12:57:26.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:26.848+0000 7f091dc75640 1 -- 172.21.15.57:0/756966246 wait complete. 2023-10-19T12:57:26.861 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir7", "rel_path": "/snap_test_dir1/dir7", "schedule": "1m", "retention": {}, "start": "2023-10-19T00:00:00", "created": "2023-10-19T12:46:00", "first": "2023-10-19T12:47:00", "last": "2023-10-19T12:57:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-19T12:57:26.862 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir7', 'rel_path': '/snap_test_dir1/dir7', 'schedule': '1m', 'retention': {}, 'start': '2023-10-19T00:00:00', 'created': '2023-10-19T12:46:00', 'first': '2023-10-19T12:47:00', 'last': '2023-10-19T12:57:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-10-19T12:57:26.862 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-10-19T12:57:26.862 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-10-19T12:57:26.863 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:26.863 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir8/.snap) 2023-10-19T12:57:26.884 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_47_00_UTC 2023-10-19T12:57:26.884 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_48_00_UTC 2023-10-19T12:57:26.884 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_49_00_UTC 2023-10-19T12:57:26.884 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_50_00_UTC 2023-10-19T12:57:26.885 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_51_00_UTC 2023-10-19T12:57:26.885 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_52_00_UTC 2023-10-19T12:57:26.885 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_53_00_UTC 2023-10-19T12:57:26.885 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_54_00_UTC 2023-10-19T12:57:26.886 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_55_00_UTC 2023-10-19T12:57:26.886 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_56_00_UTC 2023-10-19T12:57:26.886 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_57_00_UTC 2023-10-19T12:57:26.886 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-19-12_47_00_UTC', 'scheduled-2023-10-19-12_48_00_UTC', 'scheduled-2023-10-19-12_49_00_UTC', 'scheduled-2023-10-19-12_50_00_UTC', 'scheduled-2023-10-19-12_51_00_UTC', 'scheduled-2023-10-19-12_52_00_UTC', 'scheduled-2023-10-19-12_53_00_UTC', 'scheduled-2023-10-19-12_54_00_UTC', 'scheduled-2023-10-19-12_55_00_UTC', 'scheduled-2023-10-19-12_56_00_UTC', 'scheduled-2023-10-19-12_57_00_UTC'] 2023-10-19T12:57:26.887 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir8 2023-10-19T12:57:27.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.033+0000 7fd22f797640 1 -- 172.21.15.57:0/1911125752 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd214003970 msgr2=0x7fd214003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:27.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.033+0000 7fd22f797640 1 --2- 172.21.15.57:0/1911125752 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd214003970 0x7fd214003d90 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fd22400c260 tx=0x7fd2140057f0 comp rx=0 tx=0).stop 2023-10-19T12:57:27.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.033+0000 7fd22dd34640 1 -- 172.21.15.57:0/1911125752 reap_dead start 2023-10-19T12:57:27.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.033+0000 7fd22f797640 1 -- 172.21.15.57:0/1911125752 shutdown_connections 2023-10-19T12:57:27.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.033+0000 7fd22f797640 1 -- 172.21.15.57:0/1911125752 >> 172.21.15.57:0/1911125752 conn(0x7fd228069930 msgr2=0x7fd228069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:27.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.033+0000 7fd22f797640 1 -- 172.21.15.57:0/1911125752 shutdown_connections 2023-10-19T12:57:27.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.033+0000 7fd22f797640 1 -- 172.21.15.57:0/1911125752 wait complete. 2023-10-19T12:57:27.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.034+0000 7fd22f797640 1 Processor -- start 2023-10-19T12:57:27.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.034+0000 7fd22f797640 1 -- start start 2023-10-19T12:57:27.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.035+0000 7fd22f797640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd214003970 0x7fd2281463f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:27.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.035+0000 7fd22f797640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd2281559d0 0x7fd228155e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:27.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.035+0000 7fd22f797640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd2280aab60 0x7fd2280aafe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:27.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.035+0000 7fd22f797640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd228146930 con 0x7fd2281559d0 2023-10-19T12:57:27.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.035+0000 7fd22f797640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd228156390 con 0x7fd2280aab60 2023-10-19T12:57:27.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.035+0000 7fd22f797640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2281f1fa0 con 0x7fd214003970 2023-10-19T12:57:27.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.035+0000 7fd22cd32640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd2281559d0 0x7fd228155e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:27.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.035+0000 7fd22cd32640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd2281559d0 0x7fd228155e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:54196/0 (socket says 172.21.15.57:54196) 2023-10-19T12:57:27.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.035+0000 7fd22cd32640 1 -- 172.21.15.57:0/2756352445 learned_addr learned my addr 172.21.15.57:0/2756352445 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:57:27.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.035+0000 7fd22dd34640 1 --2- 172.21.15.57:0/2756352445 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd2280aab60 0x7fd2280aafe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:27.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.035+0000 7fd22d533640 1 --2- 172.21.15.57:0/2756352445 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd214003970 0x7fd2281463f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:27.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.035+0000 7fd22cd32640 1 -- 172.21.15.57:0/2756352445 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd214003970 msgr2=0x7fd2281463f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:27.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.035+0000 7fd22cd32640 1 --2- 172.21.15.57:0/2756352445 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd214003970 0x7fd2281463f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:27.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.035+0000 7fd22cd32640 1 -- 172.21.15.57:0/2756352445 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd2280aab60 msgr2=0x7fd2280aafe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:27.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.035+0000 7fd22cd32640 1 --2- 172.21.15.57:0/2756352445 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd2280aab60 0x7fd2280aafe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:27.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.035+0000 7fd22cd32640 1 -- 172.21.15.57:0/2756352445 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2280ab8d0 con 0x7fd2281559d0 2023-10-19T12:57:27.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.036+0000 7fd22cd32640 1 --2- 172.21.15.57:0/2756352445 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd2281559d0 0x7fd228155e50 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fd21800eb90 tx=0x7fd21800d7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:27.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.036+0000 7fd21dffb640 1 -- 172.21.15.57:0/2756352445 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd218013070 con 0x7fd2281559d0 2023-10-19T12:57:27.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.036+0000 7fd22f797640 1 -- 172.21.15.57:0/2756352445 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2280abbb0 con 0x7fd2281559d0 2023-10-19T12:57:27.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.036+0000 7fd21dffb640 1 -- 172.21.15.57:0/2756352445 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fd21800c5e0 con 0x7fd2281559d0 2023-10-19T12:57:27.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.036+0000 7fd21dffb640 1 -- 172.21.15.57:0/2756352445 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd21800c9f0 con 0x7fd2281559d0 2023-10-19T12:57:27.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.036+0000 7fd22f797640 1 -- 172.21.15.57:0/2756352445 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2280ac170 con 0x7fd2281559d0 2023-10-19T12:57:27.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.037+0000 7fd22f797640 1 -- 172.21.15.57:0/2756352445 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2280690e0 con 0x7fd2281559d0 2023-10-19T12:57:27.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.037+0000 7fd21dffb640 1 -- 172.21.15.57:0/2756352445 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 15) v1 ==== 89417+0+0 (secure 0 0 0) 0x7fd21800cb50 con 0x7fd2281559d0 2023-10-19T12:57:27.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.038+0000 7fd21dffb640 1 --2- 172.21.15.57:0/2756352445 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fd20c06b070 0x7fd20c06d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:27.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.038+0000 7fd21dffb640 1 -- 172.21.15.57:0/2756352445 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7fd21808a740 con 0x7fd2281559d0 2023-10-19T12:57:27.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.038+0000 7fd22d533640 1 --2- 172.21.15.57:0/2756352445 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fd20c06b070 0x7fd20c06d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:27.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.039+0000 7fd22d533640 1 --2- 172.21.15.57:0/2756352445 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fd20c06b070 0x7fd20c06d530 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd224034000 tx=0x7fd224001a70 comp rx=0 tx=0).ready entity=mgr.5428 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:27.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.042+0000 7fd21dffb640 1 -- 172.21.15.57:0/2756352445 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd218055230 con 0x7fd2281559d0 2023-10-19T12:57:27.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.239+0000 7fd22f797640 1 -- 172.21.15.57:0/2756352445 --> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir8", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd2280cc100 con 0x7fd20c06b070 2023-10-19T12:57:27.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.250+0000 7fd21dffb640 1 -- 172.21.15.57:0/2756352445 <== mgr.5428 v2:172.21.15.179:6832/922398470 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7fd224006050 con 0x7fd20c06b070 2023-10-19T12:57:27.253 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:57:27.253 INFO:teuthology.orchestra.run.smithi057.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir8", "rel_path": "/snap_test_dir1/dir8", "schedule": "1m", "retention": {}, "start": "2023-10-19T00:00:00", "created": "2023-10-19T12:46:01", "first": "2023-10-19T12:47:00", "last": "2023-10-19T12:57:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-19T12:57:27.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.252+0000 7fd22f797640 1 -- 172.21.15.57:0/2756352445 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fd20c06b070 msgr2=0x7fd20c06d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:27.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.252+0000 7fd22f797640 1 --2- 172.21.15.57:0/2756352445 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fd20c06b070 0x7fd20c06d530 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd224034000 tx=0x7fd224001a70 comp rx=0 tx=0).stop 2023-10-19T12:57:27.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.252+0000 7fd22f797640 1 -- 172.21.15.57:0/2756352445 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd2281559d0 msgr2=0x7fd228155e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:27.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.252+0000 7fd22f797640 1 --2- 172.21.15.57:0/2756352445 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd2281559d0 0x7fd228155e50 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fd21800eb90 tx=0x7fd21800d7b0 comp rx=0 tx=0).stop 2023-10-19T12:57:27.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.252+0000 7fd22f797640 1 -- 172.21.15.57:0/2756352445 shutdown_connections 2023-10-19T12:57:27.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.252+0000 7fd22f797640 1 --2- 172.21.15.57:0/2756352445 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fd20c06b070 0x7fd20c06d530 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:27.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.252+0000 7fd22f797640 1 --2- 172.21.15.57:0/2756352445 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd2280aab60 0x7fd2280aafe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:27.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.252+0000 7fd22f797640 1 --2- 172.21.15.57:0/2756352445 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd2281559d0 0x7fd228155e50 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:27.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.252+0000 7fd22f797640 1 --2- 172.21.15.57:0/2756352445 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd214003970 0x7fd2281463f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:27.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.252+0000 7fd22f797640 1 -- 172.21.15.57:0/2756352445 >> 172.21.15.57:0/2756352445 conn(0x7fd228069930 msgr2=0x7fd2281589e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:27.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.252+0000 7fd22f797640 1 -- 172.21.15.57:0/2756352445 shutdown_connections 2023-10-19T12:57:27.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.252+0000 7fd22f797640 1 -- 172.21.15.57:0/2756352445 wait complete. 2023-10-19T12:57:27.265 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir8", "rel_path": "/snap_test_dir1/dir8", "schedule": "1m", "retention": {}, "start": "2023-10-19T00:00:00", "created": "2023-10-19T12:46:01", "first": "2023-10-19T12:47:00", "last": "2023-10-19T12:57:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-19T12:57:27.265 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir8', 'rel_path': '/snap_test_dir1/dir8', 'schedule': '1m', 'retention': {}, 'start': '2023-10-19T00:00:00', 'created': '2023-10-19T12:46:01', 'first': '2023-10-19T12:47:00', 'last': '2023-10-19T12:57:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-10-19T12:57:27.265 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-10-19T12:57:27.266 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-10-19T12:57:27.266 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:27.266 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir9/.snap) 2023-10-19T12:57:27.290 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_47_00_UTC 2023-10-19T12:57:27.290 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_48_00_UTC 2023-10-19T12:57:27.291 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_49_00_UTC 2023-10-19T12:57:27.291 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_50_00_UTC 2023-10-19T12:57:27.292 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_51_00_UTC 2023-10-19T12:57:27.292 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_52_00_UTC 2023-10-19T12:57:27.292 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_53_00_UTC 2023-10-19T12:57:27.293 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_54_01_UTC 2023-10-19T12:57:27.293 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_55_00_UTC 2023-10-19T12:57:27.293 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_56_00_UTC 2023-10-19T12:57:27.294 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_57_00_UTC 2023-10-19T12:57:27.294 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-19-12_47_00_UTC', 'scheduled-2023-10-19-12_48_00_UTC', 'scheduled-2023-10-19-12_49_00_UTC', 'scheduled-2023-10-19-12_50_00_UTC', 'scheduled-2023-10-19-12_51_00_UTC', 'scheduled-2023-10-19-12_52_00_UTC', 'scheduled-2023-10-19-12_53_00_UTC', 'scheduled-2023-10-19-12_54_01_UTC', 'scheduled-2023-10-19-12_55_00_UTC', 'scheduled-2023-10-19-12_56_00_UTC', 'scheduled-2023-10-19-12_57_00_UTC'] 2023-10-19T12:57:27.295 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir9 2023-10-19T12:57:27.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.439+0000 7fedddc1a640 1 -- 172.21.15.57:0/3641014536 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fedc0003970 msgr2=0x7fedc0003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:27.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.439+0000 7fedddc1a640 1 --2- 172.21.15.57:0/3641014536 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fedc0003970 0x7fedc0003d90 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fedc8009b50 tx=0x7fedc00057f0 comp rx=0 tx=0).stop 2023-10-19T12:57:27.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.439+0000 7fedddc1a640 1 -- 172.21.15.57:0/3641014536 shutdown_connections 2023-10-19T12:57:27.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.439+0000 7fedddc1a640 1 --2- 172.21.15.57:0/3641014536 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fedc0003970 0x7fedc0003d90 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:27.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.439+0000 7fedddc1a640 1 -- 172.21.15.57:0/3641014536 >> 172.21.15.57:0/3641014536 conn(0x7fedd8069930 msgr2=0x7fedd8069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:27.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.439+0000 7fedddc1a640 1 -- 172.21.15.57:0/3641014536 shutdown_connections 2023-10-19T12:57:27.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.439+0000 7fedddc1a640 1 -- 172.21.15.57:0/3641014536 wait complete. 2023-10-19T12:57:27.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.440+0000 7fedddc1a640 1 Processor -- start 2023-10-19T12:57:27.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.440+0000 7fedddc1a640 1 -- start start 2023-10-19T12:57:27.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.441+0000 7fedddc1a640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fedc0003970 0x7fedd813d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:27.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.441+0000 7fedddc1a640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fedd81e19d0 0x7fedd81e1e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:27.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.441+0000 7fedddc1a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fedd80b0fa0 0x7fedd80b13e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:27.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.441+0000 7fedddc1a640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedd81d6460 con 0x7fedd80b0fa0 2023-10-19T12:57:27.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.441+0000 7fedddc1a640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedd81e1020 con 0x7fedd81e19d0 2023-10-19T12:57:27.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.441+0000 7fedddc1a640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedd80b1920 con 0x7fedc0003970 2023-10-19T12:57:27.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.441+0000 7fedd7fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fedd80b0fa0 0x7fedd80b13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:27.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.441+0000 7fedd7fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fedd80b0fa0 0x7fedd80b13e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:54208/0 (socket says 172.21.15.57:54208) 2023-10-19T12:57:27.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.441+0000 7fedd7fff640 1 -- 172.21.15.57:0/2699386033 learned_addr learned my addr 172.21.15.57:0/2699386033 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:57:27.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.441+0000 7fedd6ffd640 1 --2- 172.21.15.57:0/2699386033 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fedd81e19d0 0x7fedd81e1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:27.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.441+0000 7fedd77fe640 1 --2- 172.21.15.57:0/2699386033 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fedc0003970 0x7fedd813d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:27.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.441+0000 7fedd7fff640 1 -- 172.21.15.57:0/2699386033 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fedc0003970 msgr2=0x7fedd813d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:27.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.441+0000 7fedd7fff640 1 --2- 172.21.15.57:0/2699386033 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fedc0003970 0x7fedd813d890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:27.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.441+0000 7fedd7fff640 1 -- 172.21.15.57:0/2699386033 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fedd81e19d0 msgr2=0x7fedd81e1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:27.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.441+0000 7fedd7fff640 1 --2- 172.21.15.57:0/2699386033 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fedd81e19d0 0x7fedd81e1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:27.452 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.441+0000 7fedd7fff640 1 -- 172.21.15.57:0/2699386033 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedd81e0d40 con 0x7fedd80b0fa0 2023-10-19T12:57:27.452 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.441+0000 7fedd7fff640 1 --2- 172.21.15.57:0/2699386033 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fedd80b0fa0 0x7fedd80b13e0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fedd80ca400 tx=0x7fedd00019b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:27.452 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.442+0000 7fedbffff640 1 -- 172.21.15.57:0/2699386033 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd0011870 con 0x7fedd80b0fa0 2023-10-19T12:57:27.453 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.442+0000 7fedbffff640 1 -- 172.21.15.57:0/2699386033 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fedd0011ac0 con 0x7fedd80b0fa0 2023-10-19T12:57:27.453 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.442+0000 7fedddc1a640 1 -- 172.21.15.57:0/2699386033 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedd813cdb0 con 0x7fedd80b0fa0 2023-10-19T12:57:27.453 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.442+0000 7fedbffff640 1 -- 172.21.15.57:0/2699386033 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd00020c0 con 0x7fedd80b0fa0 2023-10-19T12:57:27.453 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.442+0000 7fedddc1a640 1 -- 172.21.15.57:0/2699386033 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedd813d370 con 0x7fedd80b0fa0 2023-10-19T12:57:27.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.443+0000 7fedddc1a640 1 -- 172.21.15.57:0/2699386033 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedd813ddd0 con 0x7fedd80b0fa0 2023-10-19T12:57:27.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.443+0000 7fedbffff640 1 -- 172.21.15.57:0/2699386033 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 15) v1 ==== 89417+0+0 (secure 0 0 0) 0x7fedd001a400 con 0x7fedd80b0fa0 2023-10-19T12:57:27.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.443+0000 7fedbffff640 1 --2- 172.21.15.57:0/2699386033 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fedc406b040 0x7fedc406d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:27.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.443+0000 7fedbffff640 1 -- 172.21.15.57:0/2699386033 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7fedd008a890 con 0x7fedd80b0fa0 2023-10-19T12:57:27.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.444+0000 7fedd77fe640 1 --2- 172.21.15.57:0/2699386033 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fedc406b040 0x7fedc406d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:27.456 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.444+0000 7fedd77fe640 1 --2- 172.21.15.57:0/2699386033 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fedc406b040 0x7fedc406d500 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fedc8036000 tx=0x7fedc80046c0 comp rx=0 tx=0).ready entity=mgr.5428 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:27.456 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.448+0000 7fedbffff640 1 -- 172.21.15.57:0/2699386033 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fedd00553e0 con 0x7fedd80b0fa0 2023-10-19T12:57:27.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.642+0000 7fedddc1a640 1 -- 172.21.15.57:0/2699386033 --> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir9", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fedd80708d0 con 0x7fedc406b040 2023-10-19T12:57:27.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.651+0000 7fedbffff640 1 -- 172.21.15.57:0/2699386033 <== mgr.5428 v2:172.21.15.179:6832/922398470 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+345 (secure 0 0 0) 0x7fedc8041070 con 0x7fedc406b040 2023-10-19T12:57:27.654 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:57:27.654 INFO:teuthology.orchestra.run.smithi057.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir9", "rel_path": "/snap_test_dir1/dir9", "schedule": "1m", "retention": {}, "start": "2023-10-19T00:00:00", "created": "2023-10-19T12:46:01", "first": "2023-10-19T12:47:00", "last": "2023-10-19T12:57:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-19T12:57:27.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.653+0000 7fedddc1a640 1 -- 172.21.15.57:0/2699386033 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fedc406b040 msgr2=0x7fedc406d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:27.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.653+0000 7fedddc1a640 1 --2- 172.21.15.57:0/2699386033 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fedc406b040 0x7fedc406d500 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fedc8036000 tx=0x7fedc80046c0 comp rx=0 tx=0).stop 2023-10-19T12:57:27.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.653+0000 7fedddc1a640 1 -- 172.21.15.57:0/2699386033 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fedd80b0fa0 msgr2=0x7fedd80b13e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:27.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.653+0000 7fedddc1a640 1 --2- 172.21.15.57:0/2699386033 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fedd80b0fa0 0x7fedd80b13e0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fedd80ca400 tx=0x7fedd00019b0 comp rx=0 tx=0).stop 2023-10-19T12:57:27.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.653+0000 7fedddc1a640 1 -- 172.21.15.57:0/2699386033 shutdown_connections 2023-10-19T12:57:27.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.653+0000 7fedddc1a640 1 --2- 172.21.15.57:0/2699386033 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fedc406b040 0x7fedc406d500 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:27.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.653+0000 7fedddc1a640 1 --2- 172.21.15.57:0/2699386033 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fedd80b0fa0 0x7fedd80b13e0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:27.658 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.653+0000 7fedddc1a640 1 --2- 172.21.15.57:0/2699386033 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fedd81e19d0 0x7fedd81e1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:27.658 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.653+0000 7fedddc1a640 1 --2- 172.21.15.57:0/2699386033 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fedc0003970 0x7fedd813d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:27.658 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.653+0000 7fedddc1a640 1 -- 172.21.15.57:0/2699386033 >> 172.21.15.57:0/2699386033 conn(0x7fedd8069930 msgr2=0x7fedd80af280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:27.658 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.653+0000 7fedddc1a640 1 -- 172.21.15.57:0/2699386033 shutdown_connections 2023-10-19T12:57:27.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.653+0000 7fedddc1a640 1 -- 172.21.15.57:0/2699386033 wait complete. 2023-10-19T12:57:27.667 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/dir9", "rel_path": "/snap_test_dir1/dir9", "schedule": "1m", "retention": {}, "start": "2023-10-19T00:00:00", "created": "2023-10-19T12:46:01", "first": "2023-10-19T12:47:00", "last": "2023-10-19T12:57:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-19T12:57:27.667 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/dir9', 'rel_path': '/snap_test_dir1/dir9', 'schedule': '1m', 'retention': {}, 'start': '2023-10-19T00:00:00', 'created': '2023-10-19T12:46:01', 'first': '2023-10-19T12:47:00', 'last': '2023-10-19T12:57:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-10-19T12:57:27.668 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-10-19T12:57:27.669 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-10-19T12:57:27.669 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir0 1m 2023-10-19T12:57:27.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.779+0000 7f30ee075640 1 -- 172.21.15.57:0/3135945336 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f30e80cf340 msgr2=0x7f30e80d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:27.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.779+0000 7f30ee075640 1 --2- 172.21.15.57:0/3135945336 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f30e80cf340 0x7f30e80d1910 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f30d8009830 tx=0x7f30d802bcd0 comp rx=0 tx=0).stop 2023-10-19T12:57:27.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.779+0000 7f30ee075640 1 -- 172.21.15.57:0/3135945336 shutdown_connections 2023-10-19T12:57:27.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.779+0000 7f30ee075640 1 --2- 172.21.15.57:0/3135945336 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f30e80cf340 0x7f30e80d1910 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:27.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.779+0000 7f30ee075640 1 -- 172.21.15.57:0/3135945336 >> 172.21.15.57:0/3135945336 conn(0x7f30e80d0d60 msgr2=0x7f30e80d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:27.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.779+0000 7f30ee075640 1 -- 172.21.15.57:0/3135945336 shutdown_connections 2023-10-19T12:57:27.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.779+0000 7f30ee075640 1 -- 172.21.15.57:0/3135945336 wait complete. 2023-10-19T12:57:27.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.780+0000 7f30ee075640 1 Processor -- start 2023-10-19T12:57:27.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.780+0000 7f30ee075640 1 -- start start 2023-10-19T12:57:27.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.781+0000 7f30ee075640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f30e80ceb40 0x7f30e8150180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:27.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.781+0000 7f30ee075640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f30e80cf340 0x7f30e815dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:27.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.781+0000 7f30ee075640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f30e815b2a0 0x7f30e81f1a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:27.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.781+0000 7f30ee075640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30e80b6cd0 con 0x7f30e815b2a0 2023-10-19T12:57:27.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.781+0000 7f30ee075640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30e80c8fb0 con 0x7f30e80cf340 2023-10-19T12:57:27.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.781+0000 7f30ee075640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30e815e6f0 con 0x7f30e80ceb40 2023-10-19T12:57:27.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.781+0000 7f30e7fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f30e815b2a0 0x7f30e81f1a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:27.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.781+0000 7f30e7fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f30e815b2a0 0x7f30e81f1a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:54226/0 (socket says 172.21.15.57:54226) 2023-10-19T12:57:27.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.781+0000 7f30e7fff640 1 -- 172.21.15.57:0/3088593691 learned_addr learned my addr 172.21.15.57:0/3088593691 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:57:27.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.782+0000 7f30e77fe640 1 --2- 172.21.15.57:0/3088593691 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f30e80ceb40 0x7f30e8150180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:27.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.782+0000 7f30e6ffd640 1 --2- 172.21.15.57:0/3088593691 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f30e80cf340 0x7f30e815dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:27.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.782+0000 7f30e7fff640 1 -- 172.21.15.57:0/3088593691 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f30e80ceb40 msgr2=0x7f30e8150180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:27.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.782+0000 7f30e7fff640 1 --2- 172.21.15.57:0/3088593691 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f30e80ceb40 0x7f30e8150180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:27.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.782+0000 7f30e7fff640 1 -- 172.21.15.57:0/3088593691 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f30e80cf340 msgr2=0x7f30e815dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:27.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.782+0000 7f30e7fff640 1 --2- 172.21.15.57:0/3088593691 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f30e80cf340 0x7f30e815dea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:27.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.782+0000 7f30e7fff640 1 -- 172.21.15.57:0/3088593691 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30d8009490 con 0x7f30e815b2a0 2023-10-19T12:57:27.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.782+0000 7f30e7fff640 1 --2- 172.21.15.57:0/3088593691 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f30e815b2a0 0x7f30e81f1a30 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f30e000cae0 tx=0x7f30e000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:27.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.782+0000 7f30cffff640 1 -- 172.21.15.57:0/3088593691 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30e000ac40 con 0x7f30e815b2a0 2023-10-19T12:57:27.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.782+0000 7f30cffff640 1 -- 172.21.15.57:0/3088593691 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f30e000ae90 con 0x7f30e815b2a0 2023-10-19T12:57:27.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.782+0000 7f30cffff640 1 -- 172.21.15.57:0/3088593691 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30e0003bd0 con 0x7f30e815b2a0 2023-10-19T12:57:27.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.782+0000 7f30ee075640 1 -- 172.21.15.57:0/3088593691 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30e80c9250 con 0x7f30e815b2a0 2023-10-19T12:57:27.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.782+0000 7f30ee075640 1 -- 172.21.15.57:0/3088593691 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30e80c9810 con 0x7f30e815b2a0 2023-10-19T12:57:27.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.783+0000 7f30ee075640 1 -- 172.21.15.57:0/3088593691 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30e80690e0 con 0x7f30e815b2a0 2023-10-19T12:57:27.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.783+0000 7f30cffff640 1 -- 172.21.15.57:0/3088593691 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 15) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f30e0003d30 con 0x7f30e815b2a0 2023-10-19T12:57:27.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.784+0000 7f30cffff640 1 --2- 172.21.15.57:0/3088593691 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f30d406b070 0x7f30d406d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:27.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.784+0000 7f30cffff640 1 -- 172.21.15.57:0/3088593691 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f30e008a510 con 0x7f30e815b2a0 2023-10-19T12:57:27.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.784+0000 7f30e77fe640 1 --2- 172.21.15.57:0/3088593691 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f30d406b070 0x7f30d406d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:27.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.785+0000 7f30e77fe640 1 --2- 172.21.15.57:0/3088593691 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f30d406b070 0x7f30d406d530 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f30d8036000 tx=0x7f30d8004730 comp rx=0 tx=0).ready entity=mgr.5428 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:27.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.788+0000 7f30cffff640 1 -- 172.21.15.57:0/3088593691 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f30e0055170 con 0x7f30e815b2a0 2023-10-19T12:57:27.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:27.988+0000 7f30ee075640 1 -- 172.21.15.57:0/3088593691 --> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir0", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f30e806bab0 con 0x7f30d406b070 2023-10-19T12:57:28.073 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:28.071+0000 7f30cffff640 1 -- 172.21.15.57:0/3088593691 <== mgr.5428 v2:172.21.15.179:6832/922398470 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f30e806bab0 con 0x7f30d406b070 2023-10-19T12:57:28.074 INFO:teuthology.orchestra.run.smithi057.stdout:Schedule removed for path /snap_test_dir1/dir0 2023-10-19T12:57:28.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:28.073+0000 7f30ee075640 1 -- 172.21.15.57:0/3088593691 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f30d406b070 msgr2=0x7f30d406d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:28.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:28.073+0000 7f30ee075640 1 --2- 172.21.15.57:0/3088593691 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f30d406b070 0x7f30d406d530 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f30d8036000 tx=0x7f30d8004730 comp rx=0 tx=0).stop 2023-10-19T12:57:28.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:28.073+0000 7f30ee075640 1 -- 172.21.15.57:0/3088593691 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f30e815b2a0 msgr2=0x7f30e81f1a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:28.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:28.073+0000 7f30ee075640 1 --2- 172.21.15.57:0/3088593691 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f30e815b2a0 0x7f30e81f1a30 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f30e000cae0 tx=0x7f30e000a060 comp rx=0 tx=0).stop 2023-10-19T12:57:28.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:28.073+0000 7f30ee075640 1 -- 172.21.15.57:0/3088593691 shutdown_connections 2023-10-19T12:57:28.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:28.073+0000 7f30ee075640 1 --2- 172.21.15.57:0/3088593691 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f30d406b070 0x7f30d406d530 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:28.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:28.073+0000 7f30ee075640 1 --2- 172.21.15.57:0/3088593691 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f30e815b2a0 0x7f30e81f1a30 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:28.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:28.073+0000 7f30ee075640 1 --2- 172.21.15.57:0/3088593691 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f30e80cf340 0x7f30e815dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:28.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:28.073+0000 7f30ee075640 1 --2- 172.21.15.57:0/3088593691 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f30e80ceb40 0x7f30e8150180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:28.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:28.073+0000 7f30ee075640 1 -- 172.21.15.57:0/3088593691 >> 172.21.15.57:0/3088593691 conn(0x7f30e80d0d60 msgr2=0x7f30e806a8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:28.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:28.073+0000 7f30ee075640 1 -- 172.21.15.57:0/3088593691 shutdown_connections 2023-10-19T12:57:28.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:28.073+0000 7f30ee075640 1 -- 172.21.15.57:0/3088593691 wait complete. 2023-10-19T12:57:28.086 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir0 2023-10-19T12:57:28.087 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:28.087 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir0/.snap) 2023-10-19T12:57:28.108 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_46_00_UTC 2023-10-19T12:57:28.109 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_47_00_UTC 2023-10-19T12:57:28.109 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_48_00_UTC 2023-10-19T12:57:28.109 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_49_00_UTC 2023-10-19T12:57:28.110 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_50_00_UTC 2023-10-19T12:57:28.110 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_51_00_UTC 2023-10-19T12:57:28.110 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_52_00_UTC 2023-10-19T12:57:28.110 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_53_00_UTC 2023-10-19T12:57:28.110 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_54_00_UTC 2023-10-19T12:57:28.111 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_55_00_UTC 2023-10-19T12:57:28.111 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_56_00_UTC 2023-10-19T12:57:28.111 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_57_00_UTC 2023-10-19T12:57:28.111 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-10-19-12_46_00_UTC 2023-10-19T12:57:28.112 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:28.112 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-10-19-12_46_00_UTC) 2023-10-19T12:57:28.172 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-10-19-12_47_00_UTC 2023-10-19T12:57:28.173 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:28.173 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-10-19-12_47_00_UTC) 2023-10-19T12:57:28.235 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-10-19-12_48_00_UTC 2023-10-19T12:57:28.236 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:28.236 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-10-19-12_48_00_UTC) 2023-10-19T12:57:28.298 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-10-19-12_49_00_UTC 2023-10-19T12:57:28.299 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:28.299 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-10-19-12_49_00_UTC) 2023-10-19T12:57:28.361 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-10-19-12_50_00_UTC 2023-10-19T12:57:28.362 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:28.362 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-10-19-12_50_00_UTC) 2023-10-19T12:57:28.424 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-10-19-12_51_00_UTC 2023-10-19T12:57:28.425 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:28.425 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-10-19-12_51_00_UTC) 2023-10-19T12:57:28.503 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-10-19-12_52_00_UTC 2023-10-19T12:57:28.504 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:28.504 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-10-19-12_52_00_UTC) 2023-10-19T12:57:28.566 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-10-19-12_53_00_UTC 2023-10-19T12:57:28.567 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:28.567 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-10-19-12_53_00_UTC) 2023-10-19T12:57:28.629 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-10-19-12_54_00_UTC 2023-10-19T12:57:28.630 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:28.630 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-10-19-12_54_00_UTC) 2023-10-19T12:57:28.692 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-10-19-12_55_00_UTC 2023-10-19T12:57:28.692 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:28.693 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-10-19-12_55_00_UTC) 2023-10-19T12:57:28.766 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-10-19-12_56_00_UTC 2023-10-19T12:57:28.766 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:28.767 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-10-19-12_56_00_UTC) 2023-10-19T12:57:28.828 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-10-19-12_57_00_UTC 2023-10-19T12:57:28.829 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:28.829 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-10-19-12_57_00_UTC) 2023-10-19T12:57:28.891 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:28.892 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0) 2023-10-19T12:57:28.951 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir1 1m 2023-10-19T12:57:29.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:29.099+0000 7fb57cb73640 1 -- 172.21.15.57:0/3818697368 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb578158cf0 msgr2=0x7fb57815c0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:29.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:29.099+0000 7fb57cb73640 1 --2- 172.21.15.57:0/3818697368 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb578158cf0 0x7fb57815c0e0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fb578072c60 tx=0x7fb56002bdb0 comp rx=0 tx=0).stop 2023-10-19T12:57:29.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:29.099+0000 7fb57cb73640 1 -- 172.21.15.57:0/3818697368 shutdown_connections 2023-10-19T12:57:29.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:29.099+0000 7fb57cb73640 1 --2- 172.21.15.57:0/3818697368 >> v2:172.21.15.179:3300/0 conn(0x7fb5780cd9e0 0x7fb5780cde50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:29.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:29.099+0000 7fb57cb73640 1 --2- 172.21.15.57:0/3818697368 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb578158cf0 0x7fb57815c0e0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:29.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:29.099+0000 7fb57cb73640 1 -- 172.21.15.57:0/3818697368 >> 172.21.15.57:0/3818697368 conn(0x7fb5780af7a0 msgr2=0x7fb5780afb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:29.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:29.099+0000 7fb57cb73640 1 -- 172.21.15.57:0/3818697368 shutdown_connections 2023-10-19T12:57:29.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:29.099+0000 7fb57cb73640 1 -- 172.21.15.57:0/3818697368 wait complete. 2023-10-19T12:57:29.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:29.100+0000 7fb57cb73640 1 Processor -- start 2023-10-19T12:57:29.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:29.100+0000 7fb57cb73640 1 -- start start 2023-10-19T12:57:29.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:29.101+0000 7fb57cb73640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb5780cd9e0 0x7fb578159f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:29.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:29.101+0000 7fb57cb73640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb578158910 0x7fb5780ad5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:29.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:29.101+0000 7fb57cb73640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb578158cf0 0x7fb5780adaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:29.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:29.101+0000 7fb57cb73640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb578155260 con 0x7fb578158cf0 2023-10-19T12:57:29.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:29.101+0000 7fb57cb73640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb578155540 con 0x7fb5780cd9e0 2023-10-19T12:57:29.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:29.101+0000 7fb57cb73640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb578155820 con 0x7fb578158910 2023-10-19T12:57:29.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:29.101+0000 7fb576d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb578158cf0 0x7fb5780adaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:29.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:29.101+0000 7fb576d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb578158cf0 0x7fb5780adaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:54232/0 (socket says 172.21.15.57:54232) 2023-10-19T12:57:29.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:29.101+0000 7fb576d9d640 1 -- 172.21.15.57:0/1672643963 learned_addr learned my addr 172.21.15.57:0/1672643963 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:57:29.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:29.101+0000 7fb57659c640 1 --2- 172.21.15.57:0/1672643963 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb5780cd9e0 0x7fb578159f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:29.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:29.101+0000 7fb575d9b640 1 --2- 172.21.15.57:0/1672643963 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb578158910 0x7fb5780ad5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:29.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:29.101+0000 7fb576d9d640 1 -- 172.21.15.57:0/1672643963 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb578158910 msgr2=0x7fb5780ad5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:29.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:29.102+0000 7fb576d9d640 1 --2- 172.21.15.57:0/1672643963 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb578158910 0x7fb5780ad5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:29.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:29.102+0000 7fb576d9d640 1 -- 172.21.15.57:0/1672643963 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb5780cd9e0 msgr2=0x7fb578159f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:29.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:29.102+0000 7fb576d9d640 1 --2- 172.21.15.57:0/1672643963 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb5780cd9e0 0x7fb578159f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:29.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:29.102+0000 7fb576d9d640 1 -- 172.21.15.57:0/1672643963 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb560009560 con 0x7fb578158cf0 2023-10-19T12:57:29.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:29.102+0000 7fb57659c640 1 --2- 172.21.15.57:0/1672643963 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb5780cd9e0 0x7fb578159f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:57:29.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:29.102+0000 7fb576d9d640 1 --2- 172.21.15.57:0/1672643963 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb578158cf0 0x7fb5780adaf0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fb56800eaa0 tx=0x7fb56800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:29.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:29.102+0000 7fb56effd640 1 -- 172.21.15.57:0/1672643963 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb56800cdb0 con 0x7fb578158cf0 2023-10-19T12:57:29.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:29.102+0000 7fb56effd640 1 -- 172.21.15.57:0/1672643963 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fb568004b20 con 0x7fb578158cf0 2023-10-19T12:57:29.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:29.102+0000 7fb57cb73640 1 -- 172.21.15.57:0/1672643963 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5780ae970 con 0x7fb578158cf0 2023-10-19T12:57:29.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:29.102+0000 7fb56effd640 1 -- 172.21.15.57:0/1672643963 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb568005020 con 0x7fb578158cf0 2023-10-19T12:57:29.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:29.102+0000 7fb57cb73640 1 -- 172.21.15.57:0/1672643963 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5780b0890 con 0x7fb578158cf0 2023-10-19T12:57:29.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:29.103+0000 7fb57cb73640 1 -- 172.21.15.57:0/1672643963 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5780aee90 con 0x7fb578158cf0 2023-10-19T12:57:29.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:29.103+0000 7fb56effd640 1 -- 172.21.15.57:0/1672643963 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 15) v1 ==== 89417+0+0 (secure 0 0 0) 0x7fb568005180 con 0x7fb578158cf0 2023-10-19T12:57:29.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:29.104+0000 7fb56effd640 1 --2- 172.21.15.57:0/1672643963 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fb55806b040 0x7fb55806d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:29.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:29.104+0000 7fb56effd640 1 -- 172.21.15.57:0/1672643963 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6316+0+0 (secure 0 0 0) 0x7fb56808a6a0 con 0x7fb578158cf0 2023-10-19T12:57:29.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:29.104+0000 7fb57659c640 1 --2- 172.21.15.57:0/1672643963 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fb55806b040 0x7fb55806d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:29.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:29.105+0000 7fb57659c640 1 --2- 172.21.15.57:0/1672643963 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fb55806b040 0x7fb55806d500 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb578067660 tx=0x7fb570009040 comp rx=0 tx=0).ready entity=mgr.5428 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:29.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:29.108+0000 7fb56effd640 1 -- 172.21.15.57:0/1672643963 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb5680551f0 con 0x7fb578158cf0 2023-10-19T12:57:29.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:29.307+0000 7fb57cb73640 1 -- 172.21.15.57:0/1672643963 --> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir1", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fb5781556b0 con 0x7fb55806b040 2023-10-19T12:57:29.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:29.408+0000 7fb56effd640 1 -- 172.21.15.57:0/1672643963 <== mgr.5428 v2:172.21.15.179:6832/922398470 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7fb5781556b0 con 0x7fb55806b040 2023-10-19T12:57:29.412 INFO:teuthology.orchestra.run.smithi057.stdout:Schedule removed for path /snap_test_dir1/dir1 2023-10-19T12:57:29.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:29.410+0000 7fb57cb73640 1 -- 172.21.15.57:0/1672643963 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fb55806b040 msgr2=0x7fb55806d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:29.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:29.410+0000 7fb57cb73640 1 --2- 172.21.15.57:0/1672643963 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fb55806b040 0x7fb55806d500 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb578067660 tx=0x7fb570009040 comp rx=0 tx=0).stop 2023-10-19T12:57:29.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:29.410+0000 7fb57cb73640 1 -- 172.21.15.57:0/1672643963 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb578158cf0 msgr2=0x7fb5780adaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:29.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:29.410+0000 7fb57cb73640 1 --2- 172.21.15.57:0/1672643963 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb578158cf0 0x7fb5780adaf0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fb56800eaa0 tx=0x7fb56800ef60 comp rx=0 tx=0).stop 2023-10-19T12:57:29.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:29.411+0000 7fb57cb73640 1 -- 172.21.15.57:0/1672643963 shutdown_connections 2023-10-19T12:57:29.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:29.411+0000 7fb57cb73640 1 --2- 172.21.15.57:0/1672643963 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fb55806b040 0x7fb55806d500 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:29.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:29.411+0000 7fb57cb73640 1 --2- 172.21.15.57:0/1672643963 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb578158cf0 0x7fb5780adaf0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:29.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:29.411+0000 7fb57cb73640 1 --2- 172.21.15.57:0/1672643963 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb578158910 0x7fb5780ad5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:29.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:29.411+0000 7fb57cb73640 1 --2- 172.21.15.57:0/1672643963 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb5780cd9e0 0x7fb578159f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:29.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:29.411+0000 7fb57cb73640 1 -- 172.21.15.57:0/1672643963 >> 172.21.15.57:0/1672643963 conn(0x7fb5780af7a0 msgr2=0x7fb578152090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:29.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:29.411+0000 7fb57cb73640 1 -- 172.21.15.57:0/1672643963 shutdown_connections 2023-10-19T12:57:29.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:29.411+0000 7fb57cb73640 1 -- 172.21.15.57:0/1672643963 wait complete. 2023-10-19T12:57:29.424 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir1 2023-10-19T12:57:29.424 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:29.424 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir1/.snap) 2023-10-19T12:57:29.445 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_46_00_UTC 2023-10-19T12:57:29.445 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_47_00_UTC 2023-10-19T12:57:29.446 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_48_00_UTC 2023-10-19T12:57:29.446 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_49_00_UTC 2023-10-19T12:57:29.446 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_50_00_UTC 2023-10-19T12:57:29.446 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_51_00_UTC 2023-10-19T12:57:29.446 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_52_00_UTC 2023-10-19T12:57:29.447 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_53_00_UTC 2023-10-19T12:57:29.447 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_54_00_UTC 2023-10-19T12:57:29.447 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_55_01_UTC 2023-10-19T12:57:29.447 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_56_00_UTC 2023-10-19T12:57:29.447 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_57_00_UTC 2023-10-19T12:57:29.448 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-10-19-12_46_00_UTC 2023-10-19T12:57:29.448 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:29.448 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-10-19-12_46_00_UTC) 2023-10-19T12:57:29.511 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-10-19-12_47_00_UTC 2023-10-19T12:57:29.511 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:29.512 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-10-19-12_47_00_UTC) 2023-10-19T12:57:29.573 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-10-19-12_48_00_UTC 2023-10-19T12:57:29.574 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:29.574 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-10-19-12_48_00_UTC) 2023-10-19T12:57:29.635 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-10-19-12_49_00_UTC 2023-10-19T12:57:29.636 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:29.636 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-10-19-12_49_00_UTC) 2023-10-19T12:57:29.698 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-10-19-12_50_00_UTC 2023-10-19T12:57:29.699 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:29.699 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-10-19-12_50_00_UTC) 2023-10-19T12:57:29.760 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-10-19-12_51_00_UTC 2023-10-19T12:57:29.761 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:29.761 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-10-19-12_51_00_UTC) 2023-10-19T12:57:29.823 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-10-19-12_52_00_UTC 2023-10-19T12:57:29.823 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:29.823 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-10-19-12_52_00_UTC) 2023-10-19T12:57:29.885 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-10-19-12_53_00_UTC 2023-10-19T12:57:29.885 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:29.885 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-10-19-12_53_00_UTC) 2023-10-19T12:57:29.947 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-10-19-12_54_00_UTC 2023-10-19T12:57:29.947 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:29.948 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-10-19-12_54_00_UTC) 2023-10-19T12:57:30.013 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-10-19-12_55_01_UTC 2023-10-19T12:57:30.013 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:30.014 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-10-19-12_55_01_UTC) 2023-10-19T12:57:30.076 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-10-19-12_56_00_UTC 2023-10-19T12:57:30.076 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:30.077 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-10-19-12_56_00_UTC) 2023-10-19T12:57:30.141 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-10-19-12_57_00_UTC 2023-10-19T12:57:30.141 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:30.142 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-10-19-12_57_00_UTC) 2023-10-19T12:57:30.208 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:30.209 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1) 2023-10-19T12:57:30.268 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir2 1m 2023-10-19T12:57:30.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:30.417+0000 7f65abdd0640 1 -- 172.21.15.57:0/3464774002 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f65a40c99e0 msgr2=0x7f65a4152c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:30.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:30.417+0000 7f65abdd0640 1 --2- 172.21.15.57:0/3464774002 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f65a40c99e0 0x7f65a4152c90 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f659c005fd0 tx=0x7f659c02c2e0 comp rx=0 tx=0).stop 2023-10-19T12:57:30.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:30.417+0000 7f65abdd0640 1 -- 172.21.15.57:0/3464774002 shutdown_connections 2023-10-19T12:57:30.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:30.417+0000 7f65abdd0640 1 --2- 172.21.15.57:0/3464774002 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f65a40c9dc0 0x7f65a41531d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:30.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:30.417+0000 7f65abdd0640 1 --2- 172.21.15.57:0/3464774002 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f65a40c99e0 0x7f65a4152c90 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:30.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:30.417+0000 7f65abdd0640 1 -- 172.21.15.57:0/3464774002 >> 172.21.15.57:0/3464774002 conn(0x7f65a4069930 msgr2=0x7f65a4069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:30.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:30.417+0000 7f65abdd0640 1 -- 172.21.15.57:0/3464774002 shutdown_connections 2023-10-19T12:57:30.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:30.417+0000 7f65abdd0640 1 -- 172.21.15.57:0/3464774002 wait complete. 2023-10-19T12:57:30.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:30.418+0000 7f65abdd0640 1 Processor -- start 2023-10-19T12:57:30.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:30.418+0000 7f65abdd0640 1 -- start start 2023-10-19T12:57:30.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:30.419+0000 7f65abdd0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f65a40ab500 0x7f65a415b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:30.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:30.419+0000 7f65abdd0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f65a40c99e0 0x7f65a40b0c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:30.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:30.419+0000 7f65abdd0640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f65a40c9dc0 0x7f65a40b11c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:30.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:30.419+0000 7f65abdd0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65a40af8d0 con 0x7f65a40c99e0 2023-10-19T12:57:30.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:30.419+0000 7f65abdd0640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65a40afbb0 con 0x7f65a40ab500 2023-10-19T12:57:30.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:30.419+0000 7f65abdd0640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65a40afe90 con 0x7f65a40c9dc0 2023-10-19T12:57:30.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:30.419+0000 7f65a936b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f65a40c99e0 0x7f65a40b0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:30.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:30.419+0000 7f65a936b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f65a40c99e0 0x7f65a40b0c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:54256/0 (socket says 172.21.15.57:54256) 2023-10-19T12:57:30.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:30.419+0000 7f65a936b640 1 -- 172.21.15.57:0/1819710510 learned_addr learned my addr 172.21.15.57:0/1819710510 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:57:30.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:30.419+0000 7f65a9b6c640 1 --2- 172.21.15.57:0/1819710510 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f65a40ab500 0x7f65a415b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:30.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:30.419+0000 7f65aa36d640 1 --2- 172.21.15.57:0/1819710510 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f65a40c9dc0 0x7f65a40b11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:30.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:30.419+0000 7f65a936b640 1 -- 172.21.15.57:0/1819710510 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f65a40c9dc0 msgr2=0x7f65a40b11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:30.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:30.419+0000 7f65a936b640 1 --2- 172.21.15.57:0/1819710510 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f65a40c9dc0 0x7f65a40b11c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:30.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:30.419+0000 7f65a936b640 1 -- 172.21.15.57:0/1819710510 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f65a40ab500 msgr2=0x7f65a415b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:30.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:30.419+0000 7f65a936b640 1 --2- 172.21.15.57:0/1819710510 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f65a40ab500 0x7f65a415b630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:30.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:30.419+0000 7f65a936b640 1 -- 172.21.15.57:0/1819710510 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f659c009900 con 0x7f65a40c99e0 2023-10-19T12:57:30.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:30.419+0000 7f65a9b6c640 1 --2- 172.21.15.57:0/1819710510 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f65a40ab500 0x7f65a415b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:57:30.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:30.419+0000 7f65a936b640 1 --2- 172.21.15.57:0/1819710510 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f65a40c99e0 0x7f65a40b0c80 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f659800d740 tx=0x7f659800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:30.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:30.420+0000 7f65a27fc640 1 -- 172.21.15.57:0/1819710510 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6598009c30 con 0x7f65a40c99e0 2023-10-19T12:57:30.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:30.420+0000 7f65a27fc640 1 -- 172.21.15.57:0/1819710510 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f6598009e80 con 0x7f65a40c99e0 2023-10-19T12:57:30.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:30.420+0000 7f65abdd0640 1 -- 172.21.15.57:0/1819710510 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65a40b0170 con 0x7f65a40c99e0 2023-10-19T12:57:30.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:30.420+0000 7f65a27fc640 1 -- 172.21.15.57:0/1819710510 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65980106c0 con 0x7f65a40c99e0 2023-10-19T12:57:30.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:30.420+0000 7f65abdd0640 1 -- 172.21.15.57:0/1819710510 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65a415bb70 con 0x7f65a40c99e0 2023-10-19T12:57:30.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:30.421+0000 7f65abdd0640 1 -- 172.21.15.57:0/1819710510 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65a40690e0 con 0x7f65a40c99e0 2023-10-19T12:57:30.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:30.421+0000 7f65a27fc640 1 -- 172.21.15.57:0/1819710510 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 15) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f6598010820 con 0x7f65a40c99e0 2023-10-19T12:57:30.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:30.421+0000 7f65a27fc640 1 --2- 172.21.15.57:0/1819710510 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f659006b040 0x7f659006d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:30.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:30.422+0000 7f65a27fc640 1 -- 172.21.15.57:0/1819710510 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6724+0+0 (secure 0 0 0) 0x7f659808a3e0 con 0x7f65a40c99e0 2023-10-19T12:57:30.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:30.422+0000 7f65a9b6c640 1 --2- 172.21.15.57:0/1819710510 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f659006b040 0x7f659006d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:30.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:30.422+0000 7f65a9b6c640 1 --2- 172.21.15.57:0/1819710510 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f659006b040 0x7f659006d500 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f659c036000 tx=0x7f659c004ab0 comp rx=0 tx=0).ready entity=mgr.5428 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:30.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:30.426+0000 7f65a27fc640 1 -- 172.21.15.57:0/1819710510 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6598054d00 con 0x7f65a40c99e0 2023-10-19T12:57:30.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:30.649+0000 7f65abdd0640 1 -- 172.21.15.57:0/1819710510 --> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir2", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f65a415d270 con 0x7f659006b040 2023-10-19T12:57:30.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:30.733+0000 7f65a27fc640 1 -- 172.21.15.57:0/1819710510 <== mgr.5428 v2:172.21.15.179:6832/922398470 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f659c003d70 con 0x7f659006b040 2023-10-19T12:57:30.737 INFO:teuthology.orchestra.run.smithi057.stdout:Schedule removed for path /snap_test_dir1/dir2 2023-10-19T12:57:30.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:30.736+0000 7f65abdd0640 1 -- 172.21.15.57:0/1819710510 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f659006b040 msgr2=0x7f659006d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:30.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:30.736+0000 7f65abdd0640 1 --2- 172.21.15.57:0/1819710510 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f659006b040 0x7f659006d500 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f659c036000 tx=0x7f659c004ab0 comp rx=0 tx=0).stop 2023-10-19T12:57:30.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:30.736+0000 7f65abdd0640 1 -- 172.21.15.57:0/1819710510 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f65a40c99e0 msgr2=0x7f65a40b0c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:30.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:30.736+0000 7f65abdd0640 1 --2- 172.21.15.57:0/1819710510 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f65a40c99e0 0x7f65a40b0c80 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f659800d740 tx=0x7f659800dc00 comp rx=0 tx=0).stop 2023-10-19T12:57:30.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:30.736+0000 7f65abdd0640 1 -- 172.21.15.57:0/1819710510 shutdown_connections 2023-10-19T12:57:30.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:30.736+0000 7f65abdd0640 1 --2- 172.21.15.57:0/1819710510 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f659006b040 0x7f659006d500 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:30.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:30.736+0000 7f65abdd0640 1 --2- 172.21.15.57:0/1819710510 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f65a40c9dc0 0x7f65a40b11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:30.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:30.736+0000 7f65abdd0640 1 --2- 172.21.15.57:0/1819710510 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f65a40c99e0 0x7f65a40b0c80 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:30.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:30.736+0000 7f65abdd0640 1 --2- 172.21.15.57:0/1819710510 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f65a40ab500 0x7f65a415b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:30.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:30.736+0000 7f65abdd0640 1 -- 172.21.15.57:0/1819710510 >> 172.21.15.57:0/1819710510 conn(0x7f65a4069930 msgr2=0x7f65a40addb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:30.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:30.736+0000 7f65abdd0640 1 -- 172.21.15.57:0/1819710510 shutdown_connections 2023-10-19T12:57:30.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:30.736+0000 7f65abdd0640 1 -- 172.21.15.57:0/1819710510 wait complete. 2023-10-19T12:57:30.749 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir2 2023-10-19T12:57:30.749 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:30.749 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir2/.snap) 2023-10-19T12:57:30.773 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_46_00_UTC 2023-10-19T12:57:30.774 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_47_00_UTC 2023-10-19T12:57:30.774 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_48_00_UTC 2023-10-19T12:57:30.774 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_49_00_UTC 2023-10-19T12:57:30.774 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_50_00_UTC 2023-10-19T12:57:30.775 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_51_00_UTC 2023-10-19T12:57:30.775 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_52_00_UTC 2023-10-19T12:57:30.775 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_53_00_UTC 2023-10-19T12:57:30.775 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_54_00_UTC 2023-10-19T12:57:30.775 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_55_00_UTC 2023-10-19T12:57:30.775 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_56_00_UTC 2023-10-19T12:57:30.776 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_57_00_UTC 2023-10-19T12:57:30.776 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-10-19-12_46_00_UTC 2023-10-19T12:57:30.776 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:30.777 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-10-19-12_46_00_UTC) 2023-10-19T12:57:30.836 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-10-19-12_47_00_UTC 2023-10-19T12:57:30.836 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:30.837 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-10-19-12_47_00_UTC) 2023-10-19T12:57:30.899 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-10-19-12_48_00_UTC 2023-10-19T12:57:30.899 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:30.900 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-10-19-12_48_00_UTC) 2023-10-19T12:57:30.961 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-10-19-12_49_00_UTC 2023-10-19T12:57:30.962 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:30.963 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-10-19-12_49_00_UTC) 2023-10-19T12:57:31.024 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-10-19-12_50_00_UTC 2023-10-19T12:57:31.024 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:31.025 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-10-19-12_50_00_UTC) 2023-10-19T12:57:31.087 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-10-19-12_51_00_UTC 2023-10-19T12:57:31.087 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:31.088 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-10-19-12_51_00_UTC) 2023-10-19T12:57:31.152 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-10-19-12_52_00_UTC 2023-10-19T12:57:31.153 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:31.153 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-10-19-12_52_00_UTC) 2023-10-19T12:57:31.222 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-10-19-12_53_00_UTC 2023-10-19T12:57:31.222 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:31.223 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-10-19-12_53_00_UTC) 2023-10-19T12:57:31.283 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-10-19-12_54_00_UTC 2023-10-19T12:57:31.284 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:31.284 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-10-19-12_54_00_UTC) 2023-10-19T12:57:31.345 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-10-19-12_55_00_UTC 2023-10-19T12:57:31.345 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:31.346 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-10-19-12_55_00_UTC) 2023-10-19T12:57:31.418 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-10-19-12_56_00_UTC 2023-10-19T12:57:31.418 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:31.419 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-10-19-12_56_00_UTC) 2023-10-19T12:57:31.479 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-10-19-12_57_00_UTC 2023-10-19T12:57:31.480 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:31.480 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-10-19-12_57_00_UTC) 2023-10-19T12:57:31.562 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:31.562 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2) 2023-10-19T12:57:31.621 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir3 1m 2023-10-19T12:57:31.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:31.769+0000 7ffb9d173640 1 -- 172.21.15.57:0/493957745 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffb84003f20 msgr2=0x7ffb840047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:31.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:31.769+0000 7ffb9d173640 1 --2- 172.21.15.57:0/493957745 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffb84003f20 0x7ffb840047c0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7ffb80006fd0 tx=0x7ffb840057f0 comp rx=0 tx=0).stop 2023-10-19T12:57:31.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:31.769+0000 7ffb9759e640 1 -- 172.21.15.57:0/493957745 reap_dead start 2023-10-19T12:57:31.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:31.769+0000 7ffb9d173640 1 -- 172.21.15.57:0/493957745 shutdown_connections 2023-10-19T12:57:31.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:31.769+0000 7ffb9d173640 1 -- 172.21.15.57:0/493957745 >> 172.21.15.57:0/493957745 conn(0x7ffb980af7a0 msgr2=0x7ffb980afb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:31.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:31.769+0000 7ffb9d173640 1 -- 172.21.15.57:0/493957745 shutdown_connections 2023-10-19T12:57:31.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:31.769+0000 7ffb9d173640 1 -- 172.21.15.57:0/493957745 wait complete. 2023-10-19T12:57:31.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:31.771+0000 7ffb9d173640 1 Processor -- start 2023-10-19T12:57:31.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:31.771+0000 7ffb9d173640 1 -- start start 2023-10-19T12:57:31.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:31.771+0000 7ffb9d173640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ffb84003f20 0x7ffb98159340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:31.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:31.771+0000 7ffb9d173640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffb981f33e0 0x7ffb9815c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:31.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:31.771+0000 7ffb9d173640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffb98158850 0x7ffb980ae570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:31.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:31.771+0000 7ffb9d173640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb981f3860 con 0x7ffb981f33e0 2023-10-19T12:57:31.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:31.771+0000 7ffb9d173640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb981f2990 con 0x7ffb98158850 2023-10-19T12:57:31.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:31.771+0000 7ffb9d173640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb9815c720 con 0x7ffb84003f20 2023-10-19T12:57:31.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:31.771+0000 7ffb8ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffb981f33e0 0x7ffb9815c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:31.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:31.771+0000 7ffb8ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffb981f33e0 0x7ffb9815c1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:54278/0 (socket says 172.21.15.57:54278) 2023-10-19T12:57:31.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:31.771+0000 7ffb8ffff640 1 -- 172.21.15.57:0/2804295273 learned_addr learned my addr 172.21.15.57:0/2804295273 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:57:31.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:31.772+0000 7ffb96d9d640 1 --2- 172.21.15.57:0/2804295273 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ffb84003f20 0x7ffb98159340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:31.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:31.772+0000 7ffb9759e640 1 --2- 172.21.15.57:0/2804295273 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffb98158850 0x7ffb980ae570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:31.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:31.772+0000 7ffb8ffff640 1 -- 172.21.15.57:0/2804295273 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ffb84003f20 msgr2=0x7ffb98159340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:31.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:31.772+0000 7ffb8ffff640 1 --2- 172.21.15.57:0/2804295273 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ffb84003f20 0x7ffb98159340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:31.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:31.772+0000 7ffb8ffff640 1 -- 172.21.15.57:0/2804295273 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffb98158850 msgr2=0x7ffb980ae570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:31.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:31.772+0000 7ffb8ffff640 1 --2- 172.21.15.57:0/2804295273 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffb98158850 0x7ffb980ae570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:31.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:31.772+0000 7ffb8ffff640 1 -- 172.21.15.57:0/2804295273 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb98159a00 con 0x7ffb981f33e0 2023-10-19T12:57:31.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:31.772+0000 7ffb96d9d640 1 --2- 172.21.15.57:0/2804295273 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ffb84003f20 0x7ffb98159340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:57:31.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:31.772+0000 7ffb9759e640 1 --2- 172.21.15.57:0/2804295273 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffb98158850 0x7ffb980ae570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:57:31.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:31.772+0000 7ffb8ffff640 1 --2- 172.21.15.57:0/2804295273 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffb981f33e0 0x7ffb9815c1e0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7ffb80001200 tx=0x7ffb800014f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:31.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:31.772+0000 7ffb8f7fe640 1 -- 172.21.15.57:0/2804295273 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb80006250 con 0x7ffb981f33e0 2023-10-19T12:57:31.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:31.772+0000 7ffb8f7fe640 1 -- 172.21.15.57:0/2804295273 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7ffb800063b0 con 0x7ffb981f33e0 2023-10-19T12:57:31.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:31.772+0000 7ffb9d173640 1 -- 172.21.15.57:0/2804295273 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb98159ce0 con 0x7ffb981f33e0 2023-10-19T12:57:31.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:31.772+0000 7ffb8f7fe640 1 -- 172.21.15.57:0/2804295273 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb80003850 con 0x7ffb981f33e0 2023-10-19T12:57:31.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:31.772+0000 7ffb9d173640 1 -- 172.21.15.57:0/2804295273 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb981f2ec0 con 0x7ffb981f33e0 2023-10-19T12:57:31.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:31.773+0000 7ffb9d173640 1 -- 172.21.15.57:0/2804295273 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb981f3050 con 0x7ffb981f33e0 2023-10-19T12:57:31.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:31.774+0000 7ffb8f7fe640 1 -- 172.21.15.57:0/2804295273 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 15) v1 ==== 89417+0+0 (secure 0 0 0) 0x7ffb80031030 con 0x7ffb981f33e0 2023-10-19T12:57:31.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:31.774+0000 7ffb8f7fe640 1 --2- 172.21.15.57:0/2804295273 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7ffb7806b040 0x7ffb7806d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:31.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:31.774+0000 7ffb8f7fe640 1 -- 172.21.15.57:0/2804295273 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6900+0+0 (secure 0 0 0) 0x7ffb800ab6e0 con 0x7ffb981f33e0 2023-10-19T12:57:31.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:31.774+0000 7ffb96d9d640 1 --2- 172.21.15.57:0/2804295273 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7ffb7806b040 0x7ffb7806d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:31.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:31.775+0000 7ffb96d9d640 1 --2- 172.21.15.57:0/2804295273 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7ffb7806b040 0x7ffb7806d500 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ffb88009df0 tx=0x7ffb88005f90 comp rx=0 tx=0).ready entity=mgr.5428 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:31.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:31.778+0000 7ffb8f7fe640 1 -- 172.21.15.57:0/2804295273 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ffb80075fe0 con 0x7ffb981f33e0 2023-10-19T12:57:31.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:31.973+0000 7ffb9d173640 1 -- 172.21.15.57:0/2804295273 --> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir3", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7ffb981f3050 con 0x7ffb7806b040 2023-10-19T12:57:32.057 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:32.054+0000 7ffb8f7fe640 1 -- 172.21.15.57:0/2804295273 <== mgr.5428 v2:172.21.15.179:6832/922398470 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7ffb981f3050 con 0x7ffb7806b040 2023-10-19T12:57:32.058 INFO:teuthology.orchestra.run.smithi057.stdout:Schedule removed for path /snap_test_dir1/dir3 2023-10-19T12:57:32.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:32.056+0000 7ffb9d173640 1 -- 172.21.15.57:0/2804295273 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7ffb7806b040 msgr2=0x7ffb7806d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:32.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:32.056+0000 7ffb9d173640 1 --2- 172.21.15.57:0/2804295273 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7ffb7806b040 0x7ffb7806d500 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ffb88009df0 tx=0x7ffb88005f90 comp rx=0 tx=0).stop 2023-10-19T12:57:32.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:32.056+0000 7ffb9d173640 1 -- 172.21.15.57:0/2804295273 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffb981f33e0 msgr2=0x7ffb9815c1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:32.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:32.056+0000 7ffb9d173640 1 --2- 172.21.15.57:0/2804295273 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffb981f33e0 0x7ffb9815c1e0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7ffb80001200 tx=0x7ffb800014f0 comp rx=0 tx=0).stop 2023-10-19T12:57:32.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:32.056+0000 7ffb9d173640 1 -- 172.21.15.57:0/2804295273 shutdown_connections 2023-10-19T12:57:32.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:32.056+0000 7ffb9d173640 1 --2- 172.21.15.57:0/2804295273 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7ffb7806b040 0x7ffb7806d500 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:32.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:32.056+0000 7ffb9d173640 1 --2- 172.21.15.57:0/2804295273 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffb98158850 0x7ffb980ae570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:32.062 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:32.056+0000 7ffb9d173640 1 --2- 172.21.15.57:0/2804295273 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffb981f33e0 0x7ffb9815c1e0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:32.062 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:32.056+0000 7ffb9d173640 1 --2- 172.21.15.57:0/2804295273 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ffb84003f20 0x7ffb98159340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:32.062 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:32.056+0000 7ffb9d173640 1 -- 172.21.15.57:0/2804295273 >> 172.21.15.57:0/2804295273 conn(0x7ffb980af7a0 msgr2=0x7ffb98069720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:32.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:32.056+0000 7ffb9d173640 1 -- 172.21.15.57:0/2804295273 shutdown_connections 2023-10-19T12:57:32.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:32.056+0000 7ffb9d173640 1 -- 172.21.15.57:0/2804295273 wait complete. 2023-10-19T12:57:32.069 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir3 2023-10-19T12:57:32.070 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:32.071 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir3/.snap) 2023-10-19T12:57:32.094 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_46_00_UTC 2023-10-19T12:57:32.095 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_47_00_UTC 2023-10-19T12:57:32.095 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_48_00_UTC 2023-10-19T12:57:32.095 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_49_00_UTC 2023-10-19T12:57:32.095 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_50_00_UTC 2023-10-19T12:57:32.096 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_51_00_UTC 2023-10-19T12:57:32.096 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_52_00_UTC 2023-10-19T12:57:32.096 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_53_00_UTC 2023-10-19T12:57:32.096 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_54_00_UTC 2023-10-19T12:57:32.096 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_55_00_UTC 2023-10-19T12:57:32.097 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_56_00_UTC 2023-10-19T12:57:32.097 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_57_00_UTC 2023-10-19T12:57:32.097 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-10-19-12_46_00_UTC 2023-10-19T12:57:32.097 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:32.098 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-10-19-12_46_00_UTC) 2023-10-19T12:57:32.161 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-10-19-12_47_00_UTC 2023-10-19T12:57:32.162 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:32.162 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-10-19-12_47_00_UTC) 2023-10-19T12:57:32.224 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-10-19-12_48_00_UTC 2023-10-19T12:57:32.224 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:32.225 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-10-19-12_48_00_UTC) 2023-10-19T12:57:32.288 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-10-19-12_49_00_UTC 2023-10-19T12:57:32.288 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:32.288 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-10-19-12_49_00_UTC) 2023-10-19T12:57:32.350 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-10-19-12_50_00_UTC 2023-10-19T12:57:32.350 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:32.351 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-10-19-12_50_00_UTC) 2023-10-19T12:57:32.413 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-10-19-12_51_00_UTC 2023-10-19T12:57:32.414 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:32.415 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-10-19-12_51_00_UTC) 2023-10-19T12:57:32.485 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-10-19-12_52_00_UTC 2023-10-19T12:57:32.486 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:32.486 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-10-19-12_52_00_UTC) 2023-10-19T12:57:32.548 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-10-19-12_53_00_UTC 2023-10-19T12:57:32.549 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:32.549 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-10-19-12_53_00_UTC) 2023-10-19T12:57:32.611 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-10-19-12_54_00_UTC 2023-10-19T12:57:32.611 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:32.612 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-10-19-12_54_00_UTC) 2023-10-19T12:57:32.674 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-10-19-12_55_00_UTC 2023-10-19T12:57:32.674 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:32.674 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-10-19-12_55_00_UTC) 2023-10-19T12:57:32.741 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-10-19-12_56_00_UTC 2023-10-19T12:57:32.741 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:32.742 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-10-19-12_56_00_UTC) 2023-10-19T12:57:32.803 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-10-19-12_57_00_UTC 2023-10-19T12:57:32.803 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:32.804 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-10-19-12_57_00_UTC) 2023-10-19T12:57:32.866 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:32.867 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3) 2023-10-19T12:57:32.925 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir4 1m 2023-10-19T12:57:33.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:33.075+0000 7f83ffd8e640 1 -- 172.21.15.57:0/1880008603 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f83f80c9ed0 msgr2=0x7f83f806b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:33.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:33.075+0000 7f83ffd8e640 1 --2- 172.21.15.57:0/1880008603 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f83f80c9ed0 0x7f83f806b910 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f83f4005fd0 tx=0x7f83f402c2e0 comp rx=0 tx=0).stop 2023-10-19T12:57:33.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:33.075+0000 7f83ffd8e640 1 -- 172.21.15.57:0/1880008603 shutdown_connections 2023-10-19T12:57:33.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:33.075+0000 7f83ffd8e640 1 --2- 172.21.15.57:0/1880008603 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f83f80c9ed0 0x7f83f806b910 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:33.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:33.075+0000 7f83ffd8e640 1 --2- 172.21.15.57:0/1880008603 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f83f813d360 0x7f83f815a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:33.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:33.075+0000 7f83ffd8e640 1 -- 172.21.15.57:0/1880008603 >> 172.21.15.57:0/1880008603 conn(0x7f83f8069930 msgr2=0x7f83f8069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:33.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:33.075+0000 7f83ffd8e640 1 -- 172.21.15.57:0/1880008603 shutdown_connections 2023-10-19T12:57:33.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:33.075+0000 7f83ffd8e640 1 -- 172.21.15.57:0/1880008603 wait complete. 2023-10-19T12:57:33.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:33.076+0000 7f83ffd8e640 1 Processor -- start 2023-10-19T12:57:33.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:33.076+0000 7f83ffd8e640 1 -- start start 2023-10-19T12:57:33.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:33.077+0000 7f83ffd8e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f83f80c9ed0 0x7f83f8137210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:33.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:33.077+0000 7f83ffd8e640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f83f813d360 0x7f83f8155ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:33.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:33.077+0000 7f83ffd8e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f83f8140660 0x7f83f81565a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:33.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:33.077+0000 7f83ffd8e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83f8155a80 con 0x7f83f80c9ed0 2023-10-19T12:57:33.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:33.077+0000 7f83ffd8e640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83f8156ae0 con 0x7f83f8140660 2023-10-19T12:57:33.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:33.077+0000 7f83ffd8e640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83f81550e0 con 0x7f83f813d360 2023-10-19T12:57:33.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:33.077+0000 7f83fdb2a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f83f80c9ed0 0x7f83f8137210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:33.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:33.077+0000 7f83fdb2a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f83f80c9ed0 0x7f83f8137210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:54296/0 (socket says 172.21.15.57:54296) 2023-10-19T12:57:33.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:33.077+0000 7f83fdb2a640 1 -- 172.21.15.57:0/1954236023 learned_addr learned my addr 172.21.15.57:0/1954236023 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:57:33.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:33.077+0000 7f83fe32b640 1 --2- 172.21.15.57:0/1954236023 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f83f8140660 0x7f83f81565a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:33.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:33.077+0000 7f83fd329640 1 --2- 172.21.15.57:0/1954236023 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f83f813d360 0x7f83f8155ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:33.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:33.078+0000 7f83fdb2a640 1 -- 172.21.15.57:0/1954236023 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f83f813d360 msgr2=0x7f83f8155ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:33.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:33.078+0000 7f83fdb2a640 1 --2- 172.21.15.57:0/1954236023 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f83f813d360 0x7f83f8155ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:33.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:33.078+0000 7f83fdb2a640 1 -- 172.21.15.57:0/1954236023 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f83f8140660 msgr2=0x7f83f81565a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:33.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:33.078+0000 7f83fdb2a640 1 --2- 172.21.15.57:0/1954236023 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f83f8140660 0x7f83f81565a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:33.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:33.078+0000 7f83fdb2a640 1 -- 172.21.15.57:0/1954236023 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83f4009900 con 0x7f83f80c9ed0 2023-10-19T12:57:33.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:33.078+0000 7f83fd329640 1 --2- 172.21.15.57:0/1954236023 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f83f813d360 0x7f83f8155ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:57:33.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:33.078+0000 7f83fe32b640 1 --2- 172.21.15.57:0/1954236023 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f83f8140660 0x7f83f81565a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:57:33.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:33.078+0000 7f83fdb2a640 1 --2- 172.21.15.57:0/1954236023 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f83f80c9ed0 0x7f83f8137210 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f83f000d950 tx=0x7f83f000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:33.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:33.078+0000 7f83ee7fc640 1 -- 172.21.15.57:0/1954236023 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83f0009a30 con 0x7f83f80c9ed0 2023-10-19T12:57:33.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:33.078+0000 7f83ee7fc640 1 -- 172.21.15.57:0/1954236023 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f83f0009c80 con 0x7f83f80c9ed0 2023-10-19T12:57:33.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:33.078+0000 7f83ffd8e640 1 -- 172.21.15.57:0/1954236023 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83f81553c0 con 0x7f83f80c9ed0 2023-10-19T12:57:33.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:33.078+0000 7f83ee7fc640 1 -- 172.21.15.57:0/1954236023 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83f0004280 con 0x7f83f80c9ed0 2023-10-19T12:57:33.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:33.078+0000 7f83ffd8e640 1 -- 172.21.15.57:0/1954236023 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83f8156cf0 con 0x7f83f80c9ed0 2023-10-19T12:57:33.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:33.079+0000 7f83ffd8e640 1 -- 172.21.15.57:0/1954236023 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83f8155cf0 con 0x7f83f80c9ed0 2023-10-19T12:57:33.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:33.079+0000 7f83ee7fc640 1 -- 172.21.15.57:0/1954236023 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 15) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f83f0005020 con 0x7f83f80c9ed0 2023-10-19T12:57:33.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:33.080+0000 7f83ee7fc640 1 --2- 172.21.15.57:0/1954236023 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f83dc06b040 0x7f83dc06d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:33.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:33.080+0000 7f83ee7fc640 1 -- 172.21.15.57:0/1954236023 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6884+0+0 (secure 0 0 0) 0x7f83f008a250 con 0x7f83f80c9ed0 2023-10-19T12:57:33.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:33.080+0000 7f83fd329640 1 --2- 172.21.15.57:0/1954236023 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f83dc06b040 0x7f83dc06d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:33.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:33.081+0000 7f83fd329640 1 --2- 172.21.15.57:0/1954236023 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f83dc06b040 0x7f83dc06d500 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f83f81371c0 tx=0x7f83e8009040 comp rx=0 tx=0).ready entity=mgr.5428 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:33.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:33.084+0000 7f83ee7fc640 1 -- 172.21.15.57:0/1954236023 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f83f0054a30 con 0x7f83f80c9ed0 2023-10-19T12:57:33.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:33.309+0000 7f83ffd8e640 1 -- 172.21.15.57:0/1954236023 --> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir4", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f83f8155250 con 0x7f83dc06b040 2023-10-19T12:57:33.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:33.410+0000 7f83ee7fc640 1 -- 172.21.15.57:0/1954236023 <== mgr.5428 v2:172.21.15.179:6832/922398470 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f83f8155250 con 0x7f83dc06b040 2023-10-19T12:57:33.413 INFO:teuthology.orchestra.run.smithi057.stdout:Schedule removed for path /snap_test_dir1/dir4 2023-10-19T12:57:33.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:33.412+0000 7f83ffd8e640 1 -- 172.21.15.57:0/1954236023 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f83dc06b040 msgr2=0x7f83dc06d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:33.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:33.412+0000 7f83ffd8e640 1 --2- 172.21.15.57:0/1954236023 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f83dc06b040 0x7f83dc06d500 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f83f81371c0 tx=0x7f83e8009040 comp rx=0 tx=0).stop 2023-10-19T12:57:33.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:33.412+0000 7f83ffd8e640 1 -- 172.21.15.57:0/1954236023 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f83f80c9ed0 msgr2=0x7f83f8137210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:33.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:33.412+0000 7f83ffd8e640 1 --2- 172.21.15.57:0/1954236023 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f83f80c9ed0 0x7f83f8137210 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f83f000d950 tx=0x7f83f000de10 comp rx=0 tx=0).stop 2023-10-19T12:57:33.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:33.412+0000 7f83ffd8e640 1 -- 172.21.15.57:0/1954236023 shutdown_connections 2023-10-19T12:57:33.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:33.412+0000 7f83ffd8e640 1 --2- 172.21.15.57:0/1954236023 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f83dc06b040 0x7f83dc06d500 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:33.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:33.412+0000 7f83ffd8e640 1 --2- 172.21.15.57:0/1954236023 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f83f8140660 0x7f83f81565a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:33.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:33.412+0000 7f83ffd8e640 1 --2- 172.21.15.57:0/1954236023 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f83f813d360 0x7f83f8155ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:33.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:33.412+0000 7f83ffd8e640 1 --2- 172.21.15.57:0/1954236023 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f83f80c9ed0 0x7f83f8137210 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:33.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:33.412+0000 7f83ffd8e640 1 -- 172.21.15.57:0/1954236023 >> 172.21.15.57:0/1954236023 conn(0x7f83f8069930 msgr2=0x7f83f8069d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:33.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:33.412+0000 7f83ffd8e640 1 -- 172.21.15.57:0/1954236023 shutdown_connections 2023-10-19T12:57:33.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:33.412+0000 7f83ffd8e640 1 -- 172.21.15.57:0/1954236023 wait complete. 2023-10-19T12:57:33.425 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir4 2023-10-19T12:57:33.426 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:33.426 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir4/.snap) 2023-10-19T12:57:33.453 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_46_00_UTC 2023-10-19T12:57:33.453 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_47_00_UTC 2023-10-19T12:57:33.453 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_48_00_UTC 2023-10-19T12:57:33.453 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_49_00_UTC 2023-10-19T12:57:33.454 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_50_00_UTC 2023-10-19T12:57:33.454 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_51_00_UTC 2023-10-19T12:57:33.454 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_52_00_UTC 2023-10-19T12:57:33.454 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_53_01_UTC 2023-10-19T12:57:33.455 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_54_00_UTC 2023-10-19T12:57:33.455 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_55_00_UTC 2023-10-19T12:57:33.455 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_56_00_UTC 2023-10-19T12:57:33.455 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_57_00_UTC 2023-10-19T12:57:33.456 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-10-19-12_46_00_UTC 2023-10-19T12:57:33.456 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:33.456 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-10-19-12_46_00_UTC) 2023-10-19T12:57:33.518 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-10-19-12_47_00_UTC 2023-10-19T12:57:33.519 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:33.519 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-10-19-12_47_00_UTC) 2023-10-19T12:57:33.581 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-10-19-12_48_00_UTC 2023-10-19T12:57:33.581 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:33.582 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-10-19-12_48_00_UTC) 2023-10-19T12:57:33.642 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-10-19-12_49_00_UTC 2023-10-19T12:57:33.643 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:33.643 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-10-19-12_49_00_UTC) 2023-10-19T12:57:33.712 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-10-19-12_50_00_UTC 2023-10-19T12:57:33.712 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:33.713 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-10-19-12_50_00_UTC) 2023-10-19T12:57:33.785 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-10-19-12_51_00_UTC 2023-10-19T12:57:33.785 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:33.786 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-10-19-12_51_00_UTC) 2023-10-19T12:57:33.848 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-10-19-12_52_00_UTC 2023-10-19T12:57:33.849 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:33.849 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-10-19-12_52_00_UTC) 2023-10-19T12:57:33.911 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-10-19-12_53_01_UTC 2023-10-19T12:57:33.911 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:33.912 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-10-19-12_53_01_UTC) 2023-10-19T12:57:33.985 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-10-19-12_54_00_UTC 2023-10-19T12:57:33.986 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:33.986 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-10-19-12_54_00_UTC) 2023-10-19T12:57:34.048 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-10-19-12_55_00_UTC 2023-10-19T12:57:34.049 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:34.049 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-10-19-12_55_00_UTC) 2023-10-19T12:57:34.113 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-10-19-12_56_00_UTC 2023-10-19T12:57:34.114 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:34.114 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-10-19-12_56_00_UTC) 2023-10-19T12:57:34.209 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-10-19-12_57_00_UTC 2023-10-19T12:57:34.210 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:34.210 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-10-19-12_57_00_UTC) 2023-10-19T12:57:34.240 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:34.240 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4) 2023-10-19T12:57:34.298 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir5 1m 2023-10-19T12:57:34.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:34.447+0000 7fe65edee640 1 -- 172.21.15.57:0/627961535 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe65813e520 msgr2=0x7fe65806bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:34.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:34.447+0000 7fe65edee640 1 --2- 172.21.15.57:0/627961535 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe65813e520 0x7fe65806bb60 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe648009830 tx=0x7fe64802bb90 comp rx=0 tx=0).stop 2023-10-19T12:57:34.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:34.447+0000 7fe65edee640 1 -- 172.21.15.57:0/627961535 shutdown_connections 2023-10-19T12:57:34.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:34.447+0000 7fe65edee640 1 --2- 172.21.15.57:0/627961535 >> v2:172.21.15.179:3300/0 conn(0x7fe658070af0 0x7fe6580affe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:34.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:34.447+0000 7fe65edee640 1 --2- 172.21.15.57:0/627961535 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe65813e520 0x7fe65806bb60 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:34.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:34.447+0000 7fe65edee640 1 -- 172.21.15.57:0/627961535 >> 172.21.15.57:0/627961535 conn(0x7fe658069930 msgr2=0x7fe658069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:34.452 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:34.447+0000 7fe65edee640 1 -- 172.21.15.57:0/627961535 shutdown_connections 2023-10-19T12:57:34.452 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:34.447+0000 7fe65edee640 1 -- 172.21.15.57:0/627961535 wait complete. 2023-10-19T12:57:34.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:34.448+0000 7fe65edee640 1 Processor -- start 2023-10-19T12:57:34.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:34.448+0000 7fe65edee640 1 -- start start 2023-10-19T12:57:34.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:34.449+0000 7fe65edee640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe658070af0 0x7fe6581e1280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:34.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:34.449+0000 7fe65edee640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe65813e520 0x7fe65813b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:34.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:34.449+0000 7fe65edee640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe658141690 0x7fe6581e3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:34.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:34.449+0000 7fe65edee640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6581f0580 con 0x7fe658070af0 2023-10-19T12:57:34.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:34.449+0000 7fe65edee640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6581e17c0 con 0x7fe658141690 2023-10-19T12:57:34.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:34.449+0000 7fe65edee640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe65813b760 con 0x7fe65813e520 2023-10-19T12:57:34.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:34.449+0000 7fe65cb8a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe658070af0 0x7fe6581e1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:34.456 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:34.449+0000 7fe65cb8a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe658070af0 0x7fe6581e1280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:54306/0 (socket says 172.21.15.57:54306) 2023-10-19T12:57:34.456 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:34.449+0000 7fe65cb8a640 1 -- 172.21.15.57:0/2161338756 learned_addr learned my addr 172.21.15.57:0/2161338756 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:57:34.456 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:34.449+0000 7fe65d38b640 1 --2- 172.21.15.57:0/2161338756 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe658141690 0x7fe6581e3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:34.456 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:34.449+0000 7fe657fff640 1 --2- 172.21.15.57:0/2161338756 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe65813e520 0x7fe65813b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:34.456 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:34.450+0000 7fe65cb8a640 1 -- 172.21.15.57:0/2161338756 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe65813e520 msgr2=0x7fe65813b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:34.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:34.450+0000 7fe65cb8a640 1 --2- 172.21.15.57:0/2161338756 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe65813e520 0x7fe65813b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:34.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:34.450+0000 7fe65cb8a640 1 -- 172.21.15.57:0/2161338756 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe658141690 msgr2=0x7fe6581e3060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:34.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:34.450+0000 7fe65cb8a640 1 --2- 172.21.15.57:0/2161338756 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe658141690 0x7fe6581e3060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:34.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:34.450+0000 7fe65cb8a640 1 -- 172.21.15.57:0/2161338756 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe648009490 con 0x7fe658070af0 2023-10-19T12:57:34.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:34.450+0000 7fe65cb8a640 1 --2- 172.21.15.57:0/2161338756 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe658070af0 0x7fe6581e1280 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fe648009950 tx=0x7fe64802ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:34.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:34.450+0000 7fe6557fa640 1 -- 172.21.15.57:0/2161338756 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe648003a00 con 0x7fe658070af0 2023-10-19T12:57:34.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:34.450+0000 7fe6557fa640 1 -- 172.21.15.57:0/2161338756 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fe648003c70 con 0x7fe658070af0 2023-10-19T12:57:34.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:34.450+0000 7fe65edee640 1 -- 172.21.15.57:0/2161338756 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6581efa00 con 0x7fe658070af0 2023-10-19T12:57:34.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:34.450+0000 7fe6557fa640 1 -- 172.21.15.57:0/2161338756 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6480416d0 con 0x7fe658070af0 2023-10-19T12:57:34.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:34.450+0000 7fe65edee640 1 -- 172.21.15.57:0/2161338756 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6581effc0 con 0x7fe658070af0 2023-10-19T12:57:34.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:34.451+0000 7fe65edee640 1 -- 172.21.15.57:0/2161338756 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6580b0fc0 con 0x7fe658070af0 2023-10-19T12:57:34.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:34.451+0000 7fe6557fa640 1 -- 172.21.15.57:0/2161338756 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 15) v1 ==== 89417+0+0 (secure 0 0 0) 0x7fe648041830 con 0x7fe658070af0 2023-10-19T12:57:34.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:34.452+0000 7fe6557fa640 1 --2- 172.21.15.57:0/2161338756 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fe64406b040 0x7fe64406d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:34.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:34.452+0000 7fe6557fa640 1 -- 172.21.15.57:0/2161338756 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 7236+0+0 (secure 0 0 0) 0x7fe648031070 con 0x7fe658070af0 2023-10-19T12:57:34.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:34.452+0000 7fe657fff640 1 --2- 172.21.15.57:0/2161338756 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fe64406b040 0x7fe64406d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:34.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:34.452+0000 7fe657fff640 1 --2- 172.21.15.57:0/2161338756 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fe64406b040 0x7fe64406d500 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe6580af8a0 tx=0x7fe64c001d60 comp rx=0 tx=0).ready entity=mgr.5428 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:34.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:34.456+0000 7fe6557fa640 1 -- 172.21.15.57:0/2161338756 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe648077660 con 0x7fe658070af0 2023-10-19T12:57:34.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:34.671+0000 7fe65edee640 1 -- 172.21.15.57:0/2161338756 --> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir5", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fe6580b2d10 con 0x7fe64406b040 2023-10-19T12:57:34.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:34.737+0000 7fe6557fa640 1 -- 172.21.15.57:0/2161338756 <== mgr.5428 v2:172.21.15.179:6832/922398470 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7fe6580b2d10 con 0x7fe64406b040 2023-10-19T12:57:34.740 INFO:teuthology.orchestra.run.smithi057.stdout:Schedule removed for path /snap_test_dir1/dir5 2023-10-19T12:57:34.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:34.739+0000 7fe65edee640 1 -- 172.21.15.57:0/2161338756 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fe64406b040 msgr2=0x7fe64406d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:34.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:34.739+0000 7fe65edee640 1 --2- 172.21.15.57:0/2161338756 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fe64406b040 0x7fe64406d500 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe6580af8a0 tx=0x7fe64c001d60 comp rx=0 tx=0).stop 2023-10-19T12:57:34.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:34.739+0000 7fe65edee640 1 -- 172.21.15.57:0/2161338756 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe658070af0 msgr2=0x7fe6581e1280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:34.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:34.739+0000 7fe65edee640 1 --2- 172.21.15.57:0/2161338756 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe658070af0 0x7fe6581e1280 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fe648009950 tx=0x7fe64802ed50 comp rx=0 tx=0).stop 2023-10-19T12:57:34.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:34.739+0000 7fe65edee640 1 -- 172.21.15.57:0/2161338756 shutdown_connections 2023-10-19T12:57:34.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:34.739+0000 7fe65edee640 1 --2- 172.21.15.57:0/2161338756 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fe64406b040 0x7fe64406d500 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:34.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:34.739+0000 7fe65edee640 1 --2- 172.21.15.57:0/2161338756 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe658141690 0x7fe6581e3060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:34.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:34.739+0000 7fe65edee640 1 --2- 172.21.15.57:0/2161338756 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe65813e520 0x7fe65813b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:34.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:34.739+0000 7fe65edee640 1 --2- 172.21.15.57:0/2161338756 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe658070af0 0x7fe6581e1280 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:34.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:34.739+0000 7fe65edee640 1 -- 172.21.15.57:0/2161338756 >> 172.21.15.57:0/2161338756 conn(0x7fe658069930 msgr2=0x7fe6580acf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:34.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:34.740+0000 7fe65edee640 1 -- 172.21.15.57:0/2161338756 shutdown_connections 2023-10-19T12:57:34.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:34.740+0000 7fe65edee640 1 -- 172.21.15.57:0/2161338756 wait complete. 2023-10-19T12:57:34.753 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir5 2023-10-19T12:57:34.753 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:34.754 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir5/.snap) 2023-10-19T12:57:34.777 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_46_00_UTC 2023-10-19T12:57:34.777 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_47_00_UTC 2023-10-19T12:57:34.777 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_48_00_UTC 2023-10-19T12:57:34.777 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_49_00_UTC 2023-10-19T12:57:34.778 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_50_00_UTC 2023-10-19T12:57:34.778 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_51_00_UTC 2023-10-19T12:57:34.778 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_52_00_UTC 2023-10-19T12:57:34.778 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_53_00_UTC 2023-10-19T12:57:34.779 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_54_00_UTC 2023-10-19T12:57:34.779 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_55_00_UTC 2023-10-19T12:57:34.779 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_56_00_UTC 2023-10-19T12:57:34.779 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_57_00_UTC 2023-10-19T12:57:34.780 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-10-19-12_46_00_UTC 2023-10-19T12:57:34.780 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:34.781 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-10-19-12_46_00_UTC) 2023-10-19T12:57:34.844 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-10-19-12_47_00_UTC 2023-10-19T12:57:34.845 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:34.845 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-10-19-12_47_00_UTC) 2023-10-19T12:57:34.907 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-10-19-12_48_00_UTC 2023-10-19T12:57:34.907 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:34.907 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-10-19-12_48_00_UTC) 2023-10-19T12:57:34.968 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-10-19-12_49_00_UTC 2023-10-19T12:57:34.969 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:34.969 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-10-19-12_49_00_UTC) 2023-10-19T12:57:35.031 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-10-19-12_50_00_UTC 2023-10-19T12:57:35.032 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:35.032 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-10-19-12_50_00_UTC) 2023-10-19T12:57:35.101 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-10-19-12_51_00_UTC 2023-10-19T12:57:35.102 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:35.102 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-10-19-12_51_00_UTC) 2023-10-19T12:57:35.164 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-10-19-12_52_00_UTC 2023-10-19T12:57:35.165 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:35.166 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-10-19-12_52_00_UTC) 2023-10-19T12:57:35.244 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-10-19-12_53_00_UTC 2023-10-19T12:57:35.245 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:35.245 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-10-19-12_53_00_UTC) 2023-10-19T12:57:35.309 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-10-19-12_54_00_UTC 2023-10-19T12:57:35.310 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:35.310 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-10-19-12_54_00_UTC) 2023-10-19T12:57:35.372 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-10-19-12_55_00_UTC 2023-10-19T12:57:35.372 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:35.373 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-10-19-12_55_00_UTC) 2023-10-19T12:57:35.436 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-10-19-12_56_00_UTC 2023-10-19T12:57:35.436 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:35.437 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-10-19-12_56_00_UTC) 2023-10-19T12:57:35.499 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-10-19-12_57_00_UTC 2023-10-19T12:57:35.499 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:35.500 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-10-19-12_57_00_UTC) 2023-10-19T12:57:35.561 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:35.561 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5) 2023-10-19T12:57:35.619 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir6 1m 2023-10-19T12:57:35.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:35.768+0000 7f5a72583640 1 -- 172.21.15.57:0/2672445778 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a580039a0 msgr2=0x7f5a58003d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:35.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:35.768+0000 7f5a72583640 1 --2- 172.21.15.57:0/2672445778 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a580039a0 0x7f5a58003d80 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f5a5400ac60 tx=0x7f5a580057f0 comp rx=0 tx=0).stop 2023-10-19T12:57:35.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:35.768+0000 7f5a70b20640 1 -- 172.21.15.57:0/2672445778 reap_dead start 2023-10-19T12:57:35.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:35.768+0000 7f5a72583640 1 -- 172.21.15.57:0/2672445778 shutdown_connections 2023-10-19T12:57:35.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:35.768+0000 7f5a72583640 1 -- 172.21.15.57:0/2672445778 >> 172.21.15.57:0/2672445778 conn(0x7f5a6c069930 msgr2=0x7f5a6c069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:35.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:35.768+0000 7f5a72583640 1 -- 172.21.15.57:0/2672445778 shutdown_connections 2023-10-19T12:57:35.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:35.768+0000 7f5a72583640 1 -- 172.21.15.57:0/2672445778 wait complete. 2023-10-19T12:57:35.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:35.770+0000 7f5a72583640 1 Processor -- start 2023-10-19T12:57:35.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:35.770+0000 7f5a72583640 1 -- start start 2023-10-19T12:57:35.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:35.770+0000 7f5a72583640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a580039a0 0x7f5a6c0ac590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:35.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:35.770+0000 7f5a72583640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5a6c0acad0 0x7f5a6c14ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:35.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:35.770+0000 7f5a72583640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5a6c14b150 0x7f5a6c1eee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:35.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:35.770+0000 7f5a72583640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a6c0acf50 con 0x7f5a580039a0 2023-10-19T12:57:35.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:35.770+0000 7f5a72583640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a6c1e2330 con 0x7f5a6c14b150 2023-10-19T12:57:35.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:35.770+0000 7f5a72583640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a6c1e2610 con 0x7f5a6c0acad0 2023-10-19T12:57:35.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:35.771+0000 7f5a6bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a580039a0 0x7f5a6c0ac590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:35.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:35.771+0000 7f5a6bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a580039a0 0x7f5a6c0ac590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:54320/0 (socket says 172.21.15.57:54320) 2023-10-19T12:57:35.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:35.771+0000 7f5a6bfff640 1 -- 172.21.15.57:0/1193940209 learned_addr learned my addr 172.21.15.57:0/1193940209 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:57:35.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:35.771+0000 7f5a6b7fe640 1 --2- 172.21.15.57:0/1193940209 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5a6c0acad0 0x7f5a6c14ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:35.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:35.771+0000 7f5a70b20640 1 --2- 172.21.15.57:0/1193940209 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5a6c14b150 0x7f5a6c1eee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:35.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:35.771+0000 7f5a6bfff640 1 -- 172.21.15.57:0/1193940209 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5a6c0acad0 msgr2=0x7f5a6c14ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:35.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:35.771+0000 7f5a6bfff640 1 --2- 172.21.15.57:0/1193940209 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5a6c0acad0 0x7f5a6c14ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:35.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:35.771+0000 7f5a6bfff640 1 -- 172.21.15.57:0/1193940209 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5a6c14b150 msgr2=0x7f5a6c1eee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:35.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:35.771+0000 7f5a6bfff640 1 --2- 172.21.15.57:0/1193940209 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5a6c14b150 0x7f5a6c1eee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:35.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:35.771+0000 7f5a6bfff640 1 -- 172.21.15.57:0/1193940209 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a6c1e28f0 con 0x7f5a580039a0 2023-10-19T12:57:35.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:35.771+0000 7f5a6bfff640 1 --2- 172.21.15.57:0/1193940209 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a580039a0 0x7f5a6c0ac590 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f5a5400ac60 tx=0x7f5a540346a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:35.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:35.771+0000 7f5a68ff9640 1 -- 172.21.15.57:0/1193940209 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a54004230 con 0x7f5a580039a0 2023-10-19T12:57:35.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:35.771+0000 7f5a68ff9640 1 -- 172.21.15.57:0/1193940209 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f5a54042070 con 0x7f5a580039a0 2023-10-19T12:57:35.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:35.771+0000 7f5a72583640 1 -- 172.21.15.57:0/1193940209 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a6c1e2bd0 con 0x7f5a580039a0 2023-10-19T12:57:35.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:35.771+0000 7f5a68ff9640 1 -- 172.21.15.57:0/1193940209 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a5403d720 con 0x7f5a580039a0 2023-10-19T12:57:35.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:35.771+0000 7f5a72583640 1 -- 172.21.15.57:0/1193940209 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a6c069fd0 con 0x7f5a580039a0 2023-10-19T12:57:35.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:35.772+0000 7f5a72583640 1 -- 172.21.15.57:0/1193940209 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a6c1e2750 con 0x7f5a580039a0 2023-10-19T12:57:35.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:35.773+0000 7f5a68ff9640 1 -- 172.21.15.57:0/1193940209 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 15) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f5a5403d880 con 0x7f5a580039a0 2023-10-19T12:57:35.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:35.773+0000 7f5a68ff9640 1 --2- 172.21.15.57:0/1193940209 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f5a4c06b040 0x7f5a4c06d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:35.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:35.773+0000 7f5a68ff9640 1 -- 172.21.15.57:0/1193940209 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 7316+0+0 (secure 0 0 0) 0x7f5a540acd00 con 0x7f5a580039a0 2023-10-19T12:57:35.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:35.773+0000 7f5a6b7fe640 1 --2- 172.21.15.57:0/1193940209 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f5a4c06b040 0x7f5a4c06d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:35.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:35.774+0000 7f5a6b7fe640 1 --2- 172.21.15.57:0/1193940209 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f5a4c06b040 0x7f5a4c06d500 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5a6c14bcc0 tx=0x7f5a5c006110 comp rx=0 tx=0).ready entity=mgr.5428 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:35.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:35.777+0000 7f5a68ff9640 1 -- 172.21.15.57:0/1193940209 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f5a6c1e2750 con 0x7f5a580039a0 2023-10-19T12:57:35.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:35.975+0000 7f5a72583640 1 -- 172.21.15.57:0/1193940209 --> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir6", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f5a6c1e2750 con 0x7f5a4c06b040 2023-10-19T12:57:36.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:36.045+0000 7f5a68ff9640 1 -- 172.21.15.57:0/1193940209 <== mgr.5428 v2:172.21.15.179:6832/922398470 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f5a6c1e2750 con 0x7f5a4c06b040 2023-10-19T12:57:36.049 INFO:teuthology.orchestra.run.smithi057.stdout:Schedule removed for path /snap_test_dir1/dir6 2023-10-19T12:57:36.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:36.048+0000 7f5a72583640 1 -- 172.21.15.57:0/1193940209 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f5a4c06b040 msgr2=0x7f5a4c06d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:36.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:36.048+0000 7f5a72583640 1 --2- 172.21.15.57:0/1193940209 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f5a4c06b040 0x7f5a4c06d500 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5a6c14bcc0 tx=0x7f5a5c006110 comp rx=0 tx=0).stop 2023-10-19T12:57:36.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:36.048+0000 7f5a72583640 1 -- 172.21.15.57:0/1193940209 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a580039a0 msgr2=0x7f5a6c0ac590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:36.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:36.048+0000 7f5a72583640 1 --2- 172.21.15.57:0/1193940209 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a580039a0 0x7f5a6c0ac590 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f5a5400ac60 tx=0x7f5a540346a0 comp rx=0 tx=0).stop 2023-10-19T12:57:36.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:36.048+0000 7f5a72583640 1 -- 172.21.15.57:0/1193940209 shutdown_connections 2023-10-19T12:57:36.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:36.048+0000 7f5a72583640 1 --2- 172.21.15.57:0/1193940209 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f5a4c06b040 0x7f5a4c06d500 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:36.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:36.048+0000 7f5a72583640 1 --2- 172.21.15.57:0/1193940209 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5a6c14b150 0x7f5a6c1eee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:36.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:36.048+0000 7f5a72583640 1 --2- 172.21.15.57:0/1193940209 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5a6c0acad0 0x7f5a6c14ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:36.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:36.048+0000 7f5a72583640 1 --2- 172.21.15.57:0/1193940209 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a580039a0 0x7f5a6c0ac590 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:36.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:36.048+0000 7f5a72583640 1 -- 172.21.15.57:0/1193940209 >> 172.21.15.57:0/1193940209 conn(0x7f5a6c069930 msgr2=0x7f5a6c1578d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:36.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:36.048+0000 7f5a72583640 1 -- 172.21.15.57:0/1193940209 shutdown_connections 2023-10-19T12:57:36.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:36.048+0000 7f5a72583640 1 -- 172.21.15.57:0/1193940209 wait complete. 2023-10-19T12:57:36.061 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir6 2023-10-19T12:57:36.061 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:36.061 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir6/.snap) 2023-10-19T12:57:36.084 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_47_00_UTC 2023-10-19T12:57:36.085 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_48_00_UTC 2023-10-19T12:57:36.085 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_49_00_UTC 2023-10-19T12:57:36.085 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_50_00_UTC 2023-10-19T12:57:36.085 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_51_00_UTC 2023-10-19T12:57:36.086 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_52_00_UTC 2023-10-19T12:57:36.086 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_53_00_UTC 2023-10-19T12:57:36.086 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_54_00_UTC 2023-10-19T12:57:36.086 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_55_00_UTC 2023-10-19T12:57:36.086 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_56_01_UTC 2023-10-19T12:57:36.087 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_57_00_UTC 2023-10-19T12:57:36.087 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-10-19-12_47_00_UTC 2023-10-19T12:57:36.087 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:36.088 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-10-19-12_47_00_UTC) 2023-10-19T12:57:36.147 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-10-19-12_48_00_UTC 2023-10-19T12:57:36.147 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:36.148 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-10-19-12_48_00_UTC) 2023-10-19T12:57:36.228 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-10-19-12_49_00_UTC 2023-10-19T12:57:36.229 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:36.229 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-10-19-12_49_00_UTC) 2023-10-19T12:57:36.291 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-10-19-12_50_00_UTC 2023-10-19T12:57:36.291 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:36.292 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-10-19-12_50_00_UTC) 2023-10-19T12:57:36.353 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-10-19-12_51_00_UTC 2023-10-19T12:57:36.354 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:36.354 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-10-19-12_51_00_UTC) 2023-10-19T12:57:36.416 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-10-19-12_52_00_UTC 2023-10-19T12:57:36.416 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:36.417 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-10-19-12_52_00_UTC) 2023-10-19T12:57:36.489 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-10-19-12_53_00_UTC 2023-10-19T12:57:36.490 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:36.490 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-10-19-12_53_00_UTC) 2023-10-19T12:57:36.552 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-10-19-12_54_00_UTC 2023-10-19T12:57:36.553 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:36.553 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-10-19-12_54_00_UTC) 2023-10-19T12:57:36.625 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-10-19-12_55_00_UTC 2023-10-19T12:57:36.625 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:36.626 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-10-19-12_55_00_UTC) 2023-10-19T12:57:36.688 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-10-19-12_56_01_UTC 2023-10-19T12:57:36.688 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:36.688 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-10-19-12_56_01_UTC) 2023-10-19T12:57:36.750 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-10-19-12_57_00_UTC 2023-10-19T12:57:36.750 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:36.751 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-10-19-12_57_00_UTC) 2023-10-19T12:57:36.812 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:36.812 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6) 2023-10-19T12:57:36.870 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir7 1m 2023-10-19T12:57:37.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:37.020+0000 7ff58294c640 1 -- 172.21.15.57:0/610431597 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff564003970 msgr2=0x7ff564003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:37.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:37.020+0000 7ff58294c640 1 --2- 172.21.15.57:0/610431597 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff564003970 0x7ff564003d90 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7ff56c006fc0 tx=0x7ff5640057f0 comp rx=0 tx=0).stop 2023-10-19T12:57:37.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:37.020+0000 7ff58294c640 1 -- 172.21.15.57:0/610431597 shutdown_connections 2023-10-19T12:57:37.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:37.020+0000 7ff58294c640 1 --2- 172.21.15.57:0/610431597 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff564003970 0x7ff564003d90 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:37.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:37.020+0000 7ff58294c640 1 -- 172.21.15.57:0/610431597 >> 172.21.15.57:0/610431597 conn(0x7ff57c0d0d60 msgr2=0x7ff57c0d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:37.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:37.020+0000 7ff58294c640 1 -- 172.21.15.57:0/610431597 shutdown_connections 2023-10-19T12:57:37.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:37.020+0000 7ff58294c640 1 -- 172.21.15.57:0/610431597 wait complete. 2023-10-19T12:57:37.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:37.021+0000 7ff58294c640 1 Processor -- start 2023-10-19T12:57:37.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:37.021+0000 7ff58294c640 1 -- start start 2023-10-19T12:57:37.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:37.022+0000 7ff58294c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff564003970 0x7ff57c1f13b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:37.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:37.022+0000 7ff58294c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff57c151520 0x7ff57c0b5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:37.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:37.022+0000 7ff58294c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff57c0b5ac0 0x7ff57c15dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:37.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:37.022+0000 7ff58294c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff57c1f18f0 con 0x7ff564003970 2023-10-19T12:57:37.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:37.022+0000 7ff58294c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff57c0b4960 con 0x7ff57c151520 2023-10-19T12:57:37.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:37.022+0000 7ff58294c640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff57c152080 con 0x7ff57c0b5ac0 2023-10-19T12:57:37.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:37.022+0000 7ff57bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff564003970 0x7ff57c1f13b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:37.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:37.022+0000 7ff57bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff564003970 0x7ff57c1f13b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:53454/0 (socket says 172.21.15.57:53454) 2023-10-19T12:57:37.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:37.022+0000 7ff57bfff640 1 -- 172.21.15.57:0/3808794949 learned_addr learned my addr 172.21.15.57:0/3808794949 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:57:37.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:37.022+0000 7ff57b7fe640 1 --2- 172.21.15.57:0/3808794949 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff57c151520 0x7ff57c0b5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:37.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:37.022+0000 7ff580ee9640 1 --2- 172.21.15.57:0/3808794949 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff57c0b5ac0 0x7ff57c15dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:37.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:37.022+0000 7ff57bfff640 1 -- 172.21.15.57:0/3808794949 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff57c0b5ac0 msgr2=0x7ff57c15dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:37.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:37.022+0000 7ff57bfff640 1 --2- 172.21.15.57:0/3808794949 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff57c0b5ac0 0x7ff57c15dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:37.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:37.022+0000 7ff57bfff640 1 -- 172.21.15.57:0/3808794949 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff57c151520 msgr2=0x7ff57c0b5580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:37.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:37.022+0000 7ff57bfff640 1 --2- 172.21.15.57:0/3808794949 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff57c151520 0x7ff57c0b5580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:37.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:37.022+0000 7ff57bfff640 1 -- 172.21.15.57:0/3808794949 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff57c152360 con 0x7ff564003970 2023-10-19T12:57:37.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:37.022+0000 7ff580ee9640 1 --2- 172.21.15.57:0/3808794949 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff57c0b5ac0 0x7ff57c15dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:57:37.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:37.022+0000 7ff57bfff640 1 --2- 172.21.15.57:0/3808794949 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff564003970 0x7ff57c1f13b0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7ff56c000f60 tx=0x7ff56c0016c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:37.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:37.023+0000 7ff578ff9640 1 -- 172.21.15.57:0/3808794949 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff56c03c070 con 0x7ff564003970 2023-10-19T12:57:37.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:37.023+0000 7ff578ff9640 1 -- 172.21.15.57:0/3808794949 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7ff56c006550 con 0x7ff564003970 2023-10-19T12:57:37.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:37.023+0000 7ff58294c640 1 -- 172.21.15.57:0/3808794949 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff57c152640 con 0x7ff564003970 2023-10-19T12:57:37.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:37.023+0000 7ff578ff9640 1 -- 172.21.15.57:0/3808794949 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff56c003830 con 0x7ff564003970 2023-10-19T12:57:37.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:37.023+0000 7ff58294c640 1 -- 172.21.15.57:0/3808794949 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff57c1519a0 con 0x7ff564003970 2023-10-19T12:57:37.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:37.024+0000 7ff58294c640 1 -- 172.21.15.57:0/3808794949 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff57c0b5ea0 con 0x7ff564003970 2023-10-19T12:57:37.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:37.024+0000 7ff578ff9640 1 -- 172.21.15.57:0/3808794949 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 15) v1 ==== 89417+0+0 (secure 0 0 0) 0x7ff56c032070 con 0x7ff564003970 2023-10-19T12:57:37.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:37.025+0000 7ff578ff9640 1 --2- 172.21.15.57:0/3808794949 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7ff56806b040 0x7ff56806d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:37.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:37.025+0000 7ff578ff9640 1 -- 172.21.15.57:0/3808794949 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 7268+0+0 (secure 0 0 0) 0x7ff56c0aae60 con 0x7ff564003970 2023-10-19T12:57:37.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:37.025+0000 7ff57b7fe640 1 --2- 172.21.15.57:0/3808794949 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7ff56806b040 0x7ff56806d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:37.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:37.025+0000 7ff57b7fe640 1 --2- 172.21.15.57:0/3808794949 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7ff56806b040 0x7ff56806d500 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff57c15a260 tx=0x7ff570006ae0 comp rx=0 tx=0).ready entity=mgr.5428 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:37.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:37.029+0000 7ff578ff9640 1 -- 172.21.15.57:0/3808794949 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff56c075700 con 0x7ff564003970 2023-10-19T12:57:37.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:37.273+0000 7ff58294c640 1 -- 172.21.15.57:0/3808794949 --> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir7", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7ff57c0adb00 con 0x7ff56806b040 2023-10-19T12:57:37.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:37.334+0000 7ff578ff9640 1 -- 172.21.15.57:0/3808794949 <== mgr.5428 v2:172.21.15.179:6832/922398470 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7ff57c0adb00 con 0x7ff56806b040 2023-10-19T12:57:37.337 INFO:teuthology.orchestra.run.smithi057.stdout:Schedule removed for path /snap_test_dir1/dir7 2023-10-19T12:57:37.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:37.336+0000 7ff58294c640 1 -- 172.21.15.57:0/3808794949 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7ff56806b040 msgr2=0x7ff56806d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:37.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:37.336+0000 7ff58294c640 1 --2- 172.21.15.57:0/3808794949 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7ff56806b040 0x7ff56806d500 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff57c15a260 tx=0x7ff570006ae0 comp rx=0 tx=0).stop 2023-10-19T12:57:37.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:37.336+0000 7ff58294c640 1 -- 172.21.15.57:0/3808794949 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff564003970 msgr2=0x7ff57c1f13b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:37.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:37.336+0000 7ff58294c640 1 --2- 172.21.15.57:0/3808794949 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff564003970 0x7ff57c1f13b0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7ff56c000f60 tx=0x7ff56c0016c0 comp rx=0 tx=0).stop 2023-10-19T12:57:37.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:37.336+0000 7ff58294c640 1 -- 172.21.15.57:0/3808794949 shutdown_connections 2023-10-19T12:57:37.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:37.336+0000 7ff58294c640 1 --2- 172.21.15.57:0/3808794949 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7ff56806b040 0x7ff56806d500 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:37.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:37.336+0000 7ff58294c640 1 --2- 172.21.15.57:0/3808794949 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff57c0b5ac0 0x7ff57c15dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:37.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:37.336+0000 7ff58294c640 1 --2- 172.21.15.57:0/3808794949 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff57c151520 0x7ff57c0b5580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:37.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:37.336+0000 7ff58294c640 1 --2- 172.21.15.57:0/3808794949 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff564003970 0x7ff57c1f13b0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:37.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:37.336+0000 7ff58294c640 1 -- 172.21.15.57:0/3808794949 >> 172.21.15.57:0/3808794949 conn(0x7ff57c0d0d60 msgr2=0x7ff57c06a310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:37.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:37.336+0000 7ff58294c640 1 -- 172.21.15.57:0/3808794949 shutdown_connections 2023-10-19T12:57:37.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:37.336+0000 7ff58294c640 1 -- 172.21.15.57:0/3808794949 wait complete. 2023-10-19T12:57:37.349 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir7 2023-10-19T12:57:37.350 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:37.350 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir7/.snap) 2023-10-19T12:57:37.373 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_47_00_UTC 2023-10-19T12:57:37.374 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_48_00_UTC 2023-10-19T12:57:37.374 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_49_00_UTC 2023-10-19T12:57:37.374 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_50_00_UTC 2023-10-19T12:57:37.374 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_51_00_UTC 2023-10-19T12:57:37.374 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_52_00_UTC 2023-10-19T12:57:37.375 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_53_00_UTC 2023-10-19T12:57:37.375 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_54_00_UTC 2023-10-19T12:57:37.375 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_55_00_UTC 2023-10-19T12:57:37.375 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_56_00_UTC 2023-10-19T12:57:37.375 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_57_00_UTC 2023-10-19T12:57:37.376 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-10-19-12_47_00_UTC 2023-10-19T12:57:37.376 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:37.376 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-10-19-12_47_00_UTC) 2023-10-19T12:57:37.437 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-10-19-12_48_00_UTC 2023-10-19T12:57:37.438 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:37.438 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-10-19-12_48_00_UTC) 2023-10-19T12:57:37.502 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-10-19-12_49_00_UTC 2023-10-19T12:57:37.504 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:37.504 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-10-19-12_49_00_UTC) 2023-10-19T12:57:37.565 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-10-19-12_50_00_UTC 2023-10-19T12:57:37.565 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:37.566 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-10-19-12_50_00_UTC) 2023-10-19T12:57:37.627 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-10-19-12_51_00_UTC 2023-10-19T12:57:37.628 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:37.628 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-10-19-12_51_00_UTC) 2023-10-19T12:57:37.690 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-10-19-12_52_00_UTC 2023-10-19T12:57:37.690 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:37.691 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-10-19-12_52_00_UTC) 2023-10-19T12:57:37.754 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-10-19-12_53_00_UTC 2023-10-19T12:57:37.755 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:37.755 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-10-19-12_53_00_UTC) 2023-10-19T12:57:37.818 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-10-19-12_54_00_UTC 2023-10-19T12:57:37.819 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:37.819 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-10-19-12_54_00_UTC) 2023-10-19T12:57:37.882 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-10-19-12_55_00_UTC 2023-10-19T12:57:37.883 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:37.883 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-10-19-12_55_00_UTC) 2023-10-19T12:57:37.947 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-10-19-12_56_00_UTC 2023-10-19T12:57:37.948 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:37.948 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-10-19-12_56_00_UTC) 2023-10-19T12:57:38.010 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-10-19-12_57_00_UTC 2023-10-19T12:57:38.011 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:38.011 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-10-19-12_57_00_UTC) 2023-10-19T12:57:38.074 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:38.075 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7) 2023-10-19T12:57:38.139 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir8 1m 2023-10-19T12:57:38.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:38.294+0000 7f34fc733640 1 -- 172.21.15.57:0/4062993519 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f34e0003ef0 msgr2=0x7f34e00047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:38.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:38.294+0000 7f34fc733640 1 --2- 172.21.15.57:0/4062993519 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f34e0003ef0 0x7f34e00047c0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f34e400c260 tx=0x7f34e00057f0 comp rx=0 tx=0).stop 2023-10-19T12:57:38.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:38.294+0000 7f34facd0640 1 -- 172.21.15.57:0/4062993519 reap_dead start 2023-10-19T12:57:38.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:38.294+0000 7f34fc733640 1 -- 172.21.15.57:0/4062993519 shutdown_connections 2023-10-19T12:57:38.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:38.294+0000 7f34fc733640 1 -- 172.21.15.57:0/4062993519 >> 172.21.15.57:0/4062993519 conn(0x7f34f40b0cd0 msgr2=0x7f34f40b1f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:38.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:38.294+0000 7f34fc733640 1 -- 172.21.15.57:0/4062993519 shutdown_connections 2023-10-19T12:57:38.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:38.294+0000 7f34fc733640 1 -- 172.21.15.57:0/4062993519 wait complete. 2023-10-19T12:57:38.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:38.295+0000 7f34fc733640 1 Processor -- start 2023-10-19T12:57:38.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:38.295+0000 7f34fc733640 1 -- start start 2023-10-19T12:57:38.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:38.296+0000 7f34fc733640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f34e0003ef0 0x7f34f41510e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:38.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:38.296+0000 7f34fc733640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f34f4155420 0x7f34f415af60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:38.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:38.296+0000 7f34fc733640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f34f41558a0 0x7f34f415b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:38.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:38.296+0000 7f34fc733640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34f4155000 con 0x7f34f41558a0 2023-10-19T12:57:38.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:38.296+0000 7f34fc733640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34f4154380 con 0x7f34f4155420 2023-10-19T12:57:38.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:38.296+0000 7f34fc733640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34f4154660 con 0x7f34e0003ef0 2023-10-19T12:57:38.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:38.296+0000 7f34facd0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f34f41558a0 0x7f34f415b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:38.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:38.296+0000 7f34facd0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f34f41558a0 0x7f34f415b4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:53470/0 (socket says 172.21.15.57:53470) 2023-10-19T12:57:38.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:38.296+0000 7f34facd0640 1 -- 172.21.15.57:0/4151221170 learned_addr learned my addr 172.21.15.57:0/4151221170 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:57:38.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:38.296+0000 7f34f9cce640 1 --2- 172.21.15.57:0/4151221170 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f34f4155420 0x7f34f415af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:38.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:38.296+0000 7f34fa4cf640 1 --2- 172.21.15.57:0/4151221170 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f34e0003ef0 0x7f34f41510e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:38.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:38.296+0000 7f34facd0640 1 -- 172.21.15.57:0/4151221170 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f34e0003ef0 msgr2=0x7f34f41510e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:38.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:38.296+0000 7f34facd0640 1 --2- 172.21.15.57:0/4151221170 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f34e0003ef0 0x7f34f41510e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:38.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:38.296+0000 7f34facd0640 1 -- 172.21.15.57:0/4151221170 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f34f4155420 msgr2=0x7f34f415af60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:38.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:38.296+0000 7f34facd0640 1 --2- 172.21.15.57:0/4151221170 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f34f4155420 0x7f34f415af60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:38.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:38.296+0000 7f34facd0640 1 -- 172.21.15.57:0/4151221170 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34f4154940 con 0x7f34f41558a0 2023-10-19T12:57:38.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:38.297+0000 7f34facd0640 1 --2- 172.21.15.57:0/4151221170 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f34f41558a0 0x7f34f415b4a0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f34f4066170 tx=0x7f34ec011ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:38.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:38.297+0000 7f34eaffd640 1 -- 172.21.15.57:0/4151221170 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34ec004b20 con 0x7f34f41558a0 2023-10-19T12:57:38.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:38.297+0000 7f34eaffd640 1 -- 172.21.15.57:0/4151221170 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f34ec004c80 con 0x7f34f41558a0 2023-10-19T12:57:38.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:38.297+0000 7f34eaffd640 1 -- 172.21.15.57:0/4151221170 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34ec003bd0 con 0x7f34f41558a0 2023-10-19T12:57:38.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:38.297+0000 7f34fc733640 1 -- 172.21.15.57:0/4151221170 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34f4154c20 con 0x7f34f41558a0 2023-10-19T12:57:38.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:38.297+0000 7f34fc733640 1 -- 172.21.15.57:0/4151221170 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34f41ee210 con 0x7f34f41558a0 2023-10-19T12:57:38.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:38.298+0000 7f34fc733640 1 -- 172.21.15.57:0/4151221170 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34f4155140 con 0x7f34f41558a0 2023-10-19T12:57:38.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:38.298+0000 7f34eaffd640 1 -- 172.21.15.57:0/4151221170 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 15) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f34ec003d30 con 0x7f34f41558a0 2023-10-19T12:57:38.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:38.299+0000 7f34eaffd640 1 --2- 172.21.15.57:0/4151221170 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f34d806b070 0x7f34d806d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:38.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:38.299+0000 7f34eaffd640 1 -- 172.21.15.57:0/4151221170 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 7488+0+0 (secure 0 0 0) 0x7f34ec08ad30 con 0x7f34f41558a0 2023-10-19T12:57:38.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:38.299+0000 7f34fa4cf640 1 --2- 172.21.15.57:0/4151221170 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f34d806b070 0x7f34d806d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:38.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:38.300+0000 7f34fa4cf640 1 --2- 172.21.15.57:0/4151221170 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f34d806b070 0x7f34d806d530 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f34f0009b70 tx=0x7f34f0005f90 comp rx=0 tx=0).ready entity=mgr.5428 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:38.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:38.303+0000 7f34eaffd640 1 -- 172.21.15.57:0/4151221170 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f34f4155140 con 0x7f34f41558a0 2023-10-19T12:57:38.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:38.504+0000 7f34fc733640 1 -- 172.21.15.57:0/4151221170 --> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir8", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f34f4069d60 con 0x7f34d806b070 2023-10-19T12:57:38.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:38.587+0000 7f34eaffd640 1 -- 172.21.15.57:0/4151221170 <== mgr.5428 v2:172.21.15.179:6832/922398470 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f34f00097e0 con 0x7f34d806b070 2023-10-19T12:57:38.590 INFO:teuthology.orchestra.run.smithi057.stdout:Schedule removed for path /snap_test_dir1/dir8 2023-10-19T12:57:38.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:38.589+0000 7f34fc733640 1 -- 172.21.15.57:0/4151221170 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f34d806b070 msgr2=0x7f34d806d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:38.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:38.589+0000 7f34fc733640 1 --2- 172.21.15.57:0/4151221170 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f34d806b070 0x7f34d806d530 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f34f0009b70 tx=0x7f34f0005f90 comp rx=0 tx=0).stop 2023-10-19T12:57:38.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:38.589+0000 7f34fc733640 1 -- 172.21.15.57:0/4151221170 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f34f41558a0 msgr2=0x7f34f415b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:38.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:38.589+0000 7f34fc733640 1 --2- 172.21.15.57:0/4151221170 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f34f41558a0 0x7f34f415b4a0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f34f4066170 tx=0x7f34ec011ef0 comp rx=0 tx=0).stop 2023-10-19T12:57:38.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:38.589+0000 7f34fc733640 1 -- 172.21.15.57:0/4151221170 shutdown_connections 2023-10-19T12:57:38.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:38.589+0000 7f34fc733640 1 --2- 172.21.15.57:0/4151221170 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f34d806b070 0x7f34d806d530 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:38.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:38.589+0000 7f34fc733640 1 --2- 172.21.15.57:0/4151221170 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f34f41558a0 0x7f34f415b4a0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:38.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:38.589+0000 7f34fc733640 1 --2- 172.21.15.57:0/4151221170 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f34f4155420 0x7f34f415af60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:38.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:38.589+0000 7f34fc733640 1 --2- 172.21.15.57:0/4151221170 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f34e0003ef0 0x7f34f41510e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:38.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:38.589+0000 7f34fc733640 1 -- 172.21.15.57:0/4151221170 >> 172.21.15.57:0/4151221170 conn(0x7f34f40b0cd0 msgr2=0x7f34f40aa7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:38.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:38.589+0000 7f34fc733640 1 -- 172.21.15.57:0/4151221170 shutdown_connections 2023-10-19T12:57:38.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:38.589+0000 7f34fc733640 1 -- 172.21.15.57:0/4151221170 wait complete. 2023-10-19T12:57:38.602 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir8 2023-10-19T12:57:38.603 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:38.603 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir8/.snap) 2023-10-19T12:57:38.626 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_47_00_UTC 2023-10-19T12:57:38.626 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_48_00_UTC 2023-10-19T12:57:38.627 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_49_00_UTC 2023-10-19T12:57:38.627 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_50_00_UTC 2023-10-19T12:57:38.627 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_51_00_UTC 2023-10-19T12:57:38.627 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_52_00_UTC 2023-10-19T12:57:38.627 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_53_00_UTC 2023-10-19T12:57:38.628 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_54_00_UTC 2023-10-19T12:57:38.628 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_55_00_UTC 2023-10-19T12:57:38.628 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_56_00_UTC 2023-10-19T12:57:38.628 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_57_00_UTC 2023-10-19T12:57:38.629 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-10-19-12_47_00_UTC 2023-10-19T12:57:38.629 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:38.629 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-10-19-12_47_00_UTC) 2023-10-19T12:57:38.692 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-10-19-12_48_00_UTC 2023-10-19T12:57:38.692 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:38.693 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-10-19-12_48_00_UTC) 2023-10-19T12:57:38.756 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-10-19-12_49_00_UTC 2023-10-19T12:57:38.756 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:38.757 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-10-19-12_49_00_UTC) 2023-10-19T12:57:38.819 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-10-19-12_50_00_UTC 2023-10-19T12:57:38.820 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:38.820 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-10-19-12_50_00_UTC) 2023-10-19T12:57:38.882 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-10-19-12_51_00_UTC 2023-10-19T12:57:38.882 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:38.883 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-10-19-12_51_00_UTC) 2023-10-19T12:57:38.946 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-10-19-12_52_00_UTC 2023-10-19T12:57:38.946 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:38.947 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-10-19-12_52_00_UTC) 2023-10-19T12:57:39.012 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-10-19-12_53_00_UTC 2023-10-19T12:57:39.013 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:39.013 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-10-19-12_53_00_UTC) 2023-10-19T12:57:39.076 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-10-19-12_54_00_UTC 2023-10-19T12:57:39.077 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:39.077 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-10-19-12_54_00_UTC) 2023-10-19T12:57:39.140 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-10-19-12_55_00_UTC 2023-10-19T12:57:39.140 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:39.141 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-10-19-12_55_00_UTC) 2023-10-19T12:57:39.225 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-10-19-12_56_00_UTC 2023-10-19T12:57:39.226 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:39.226 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-10-19-12_56_00_UTC) 2023-10-19T12:57:39.262 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-10-19-12_57_00_UTC 2023-10-19T12:57:39.262 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:39.263 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-10-19-12_57_00_UTC) 2023-10-19T12:57:39.325 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:39.325 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8) 2023-10-19T12:57:39.384 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir9 1m 2023-10-19T12:57:39.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:39.533+0000 7f5782d97640 1 -- 172.21.15.57:0/4096100157 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f577c0cf340 msgr2=0x7f577c0d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:39.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:39.533+0000 7f5782d97640 1 --2- 172.21.15.57:0/4096100157 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f577c0cf340 0x7f577c0d1910 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f577c08a120 tx=0x7f576c02bd10 comp rx=0 tx=0).stop 2023-10-19T12:57:39.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:39.533+0000 7f5782d97640 1 -- 172.21.15.57:0/4096100157 shutdown_connections 2023-10-19T12:57:39.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:39.533+0000 7f5782d97640 1 --2- 172.21.15.57:0/4096100157 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f577c0ceb40 0x7f577c0b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:39.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:39.533+0000 7f5782d97640 1 --2- 172.21.15.57:0/4096100157 >> v2:172.21.15.179:3300/0 conn(0x7f577c15b2a0 0x7f577c0cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:39.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:39.533+0000 7f5782d97640 1 --2- 172.21.15.57:0/4096100157 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f577c0cf340 0x7f577c0d1910 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:39.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:39.533+0000 7f5782d97640 1 -- 172.21.15.57:0/4096100157 >> 172.21.15.57:0/4096100157 conn(0x7f577c0d0d60 msgr2=0x7f577c0d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:39.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:39.533+0000 7f5782d97640 1 -- 172.21.15.57:0/4096100157 shutdown_connections 2023-10-19T12:57:39.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:39.533+0000 7f5782d97640 1 -- 172.21.15.57:0/4096100157 wait complete. 2023-10-19T12:57:39.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:39.534+0000 7f5782d97640 1 Processor -- start 2023-10-19T12:57:39.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:39.534+0000 7f5782d97640 1 -- start start 2023-10-19T12:57:39.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:39.535+0000 7f5782d97640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f577c0ceb40 0x7f577c158400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:39.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:39.535+0000 7f5782d97640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f577c0cf340 0x7f577c0b5a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:39.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:39.535+0000 7f5782d97640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f577c15b2a0 0x7f577c0c8910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:39.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:39.535+0000 7f5782d97640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f577c158940 con 0x7f577c0ceb40 2023-10-19T12:57:39.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:39.535+0000 7f5782d97640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f577c0b63f0 con 0x7f577c0cf340 2023-10-19T12:57:39.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:39.535+0000 7f5782d97640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f577c0b66d0 con 0x7f577c15b2a0 2023-10-19T12:57:39.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:39.535+0000 7f5780b33640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f577c0ceb40 0x7f577c158400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:39.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:39.535+0000 7f5780b33640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f577c0ceb40 0x7f577c158400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:53486/0 (socket says 172.21.15.57:53486) 2023-10-19T12:57:39.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:39.535+0000 7f5780b33640 1 -- 172.21.15.57:0/891712034 learned_addr learned my addr 172.21.15.57:0/891712034 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:57:39.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:39.535+0000 7f577bfff640 1 --2- 172.21.15.57:0/891712034 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f577c0cf340 0x7f577c0b5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:39.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:39.535+0000 7f5781334640 1 --2- 172.21.15.57:0/891712034 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f577c15b2a0 0x7f577c0c8910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:39.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:39.535+0000 7f5780b33640 1 -- 172.21.15.57:0/891712034 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f577c15b2a0 msgr2=0x7f577c0c8910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:39.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:39.535+0000 7f5780b33640 1 --2- 172.21.15.57:0/891712034 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f577c15b2a0 0x7f577c0c8910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:39.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:39.535+0000 7f5780b33640 1 -- 172.21.15.57:0/891712034 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f577c0cf340 msgr2=0x7f577c0b5a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:39.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:39.535+0000 7f5780b33640 1 --2- 172.21.15.57:0/891712034 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f577c0cf340 0x7f577c0b5a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:39.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:39.535+0000 7f5780b33640 1 -- 172.21.15.57:0/891712034 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f576c009560 con 0x7f577c0ceb40 2023-10-19T12:57:39.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:39.536+0000 7f5780b33640 1 --2- 172.21.15.57:0/891712034 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f577c0ceb40 0x7f577c158400 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f576c009850 tx=0x7f576c02da40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:39.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:39.536+0000 7f57797fa640 1 -- 172.21.15.57:0/891712034 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f576c003800 con 0x7f577c0ceb40 2023-10-19T12:57:39.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:39.536+0000 7f57797fa640 1 -- 172.21.15.57:0/891712034 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f576c003a50 con 0x7f577c0ceb40 2023-10-19T12:57:39.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:39.536+0000 7f5782d97640 1 -- 172.21.15.57:0/891712034 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f577c0c7e30 con 0x7f577c0ceb40 2023-10-19T12:57:39.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:39.536+0000 7f57797fa640 1 -- 172.21.15.57:0/891712034 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f576c003e20 con 0x7f577c0ceb40 2023-10-19T12:57:39.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:39.536+0000 7f5782d97640 1 -- 172.21.15.57:0/891712034 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f577c0c83f0 con 0x7f577c0ceb40 2023-10-19T12:57:39.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:39.537+0000 7f5782d97640 1 -- 172.21.15.57:0/891712034 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f577c0677b0 con 0x7f577c0ceb40 2023-10-19T12:57:39.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:39.537+0000 7f57797fa640 1 -- 172.21.15.57:0/891712034 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 15) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f576c03e450 con 0x7f577c0ceb40 2023-10-19T12:57:39.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:39.538+0000 7f57797fa640 1 --2- 172.21.15.57:0/891712034 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f576806b040 0x7f576806d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:39.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:39.538+0000 7f57797fa640 1 -- 172.21.15.57:0/891712034 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6948+0+0 (secure 0 0 0) 0x7f576c0ac9b0 con 0x7f577c0ceb40 2023-10-19T12:57:39.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:39.538+0000 7f577bfff640 1 --2- 172.21.15.57:0/891712034 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f576806b040 0x7f576806d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:39.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:39.538+0000 7f577bfff640 1 --2- 172.21.15.57:0/891712034 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f576806b040 0x7f576806d500 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5770005fd0 tx=0x7f5770005850 comp rx=0 tx=0).ready entity=mgr.5428 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:39.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:39.542+0000 7f57797fa640 1 -- 172.21.15.57:0/891712034 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f576c077300 con 0x7f577c0ceb40 2023-10-19T12:57:39.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:39.740+0000 7f5782d97640 1 -- 172.21.15.57:0/891712034 --> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir9", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f577c0cc170 con 0x7f576806b040 2023-10-19T12:57:39.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:39.802+0000 7f57797fa640 1 -- 172.21.15.57:0/891712034 <== mgr.5428 v2:172.21.15.179:6832/922398470 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f577000b7f0 con 0x7f576806b040 2023-10-19T12:57:39.805 INFO:teuthology.orchestra.run.smithi057.stdout:Schedule removed for path /snap_test_dir1/dir9 2023-10-19T12:57:39.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:39.804+0000 7f5782d97640 1 -- 172.21.15.57:0/891712034 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f576806b040 msgr2=0x7f576806d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:39.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:39.804+0000 7f5782d97640 1 --2- 172.21.15.57:0/891712034 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f576806b040 0x7f576806d500 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5770005fd0 tx=0x7f5770005850 comp rx=0 tx=0).stop 2023-10-19T12:57:39.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:39.804+0000 7f5782d97640 1 -- 172.21.15.57:0/891712034 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f577c0ceb40 msgr2=0x7f577c158400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:39.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:39.804+0000 7f5782d97640 1 --2- 172.21.15.57:0/891712034 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f577c0ceb40 0x7f577c158400 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f576c009850 tx=0x7f576c02da40 comp rx=0 tx=0).stop 2023-10-19T12:57:39.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:39.804+0000 7f5782d97640 1 -- 172.21.15.57:0/891712034 shutdown_connections 2023-10-19T12:57:39.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:39.804+0000 7f5782d97640 1 --2- 172.21.15.57:0/891712034 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f576806b040 0x7f576806d500 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:39.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:39.804+0000 7f5782d97640 1 --2- 172.21.15.57:0/891712034 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f577c15b2a0 0x7f577c0c8910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:39.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:39.804+0000 7f5782d97640 1 --2- 172.21.15.57:0/891712034 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f577c0cf340 0x7f577c0b5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:39.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:39.804+0000 7f5782d97640 1 --2- 172.21.15.57:0/891712034 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f577c0ceb40 0x7f577c158400 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:39.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:39.804+0000 7f5782d97640 1 -- 172.21.15.57:0/891712034 >> 172.21.15.57:0/891712034 conn(0x7f577c0d0d60 msgr2=0x7f577c0b1e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:39.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:39.804+0000 7f5782d97640 1 -- 172.21.15.57:0/891712034 shutdown_connections 2023-10-19T12:57:39.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:39.804+0000 7f5782d97640 1 -- 172.21.15.57:0/891712034 wait complete. 2023-10-19T12:57:39.817 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir9 2023-10-19T12:57:39.818 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:39.818 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir9/.snap) 2023-10-19T12:57:39.847 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_47_00_UTC 2023-10-19T12:57:39.848 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_48_00_UTC 2023-10-19T12:57:39.848 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_49_00_UTC 2023-10-19T12:57:39.848 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_50_00_UTC 2023-10-19T12:57:39.848 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_51_00_UTC 2023-10-19T12:57:39.849 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_52_00_UTC 2023-10-19T12:57:39.849 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_53_00_UTC 2023-10-19T12:57:39.849 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_54_01_UTC 2023-10-19T12:57:39.849 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_55_00_UTC 2023-10-19T12:57:39.849 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_56_00_UTC 2023-10-19T12:57:39.850 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_57_00_UTC 2023-10-19T12:57:39.850 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-10-19-12_47_00_UTC 2023-10-19T12:57:39.850 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:39.851 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-10-19-12_47_00_UTC) 2023-10-19T12:57:39.910 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-10-19-12_48_00_UTC 2023-10-19T12:57:39.910 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:39.911 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-10-19-12_48_00_UTC) 2023-10-19T12:57:39.975 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-10-19-12_49_00_UTC 2023-10-19T12:57:39.976 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:39.976 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-10-19-12_49_00_UTC) 2023-10-19T12:57:40.038 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-10-19-12_50_00_UTC 2023-10-19T12:57:40.038 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:40.039 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-10-19-12_50_00_UTC) 2023-10-19T12:57:40.103 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-10-19-12_51_00_UTC 2023-10-19T12:57:40.103 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:40.104 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-10-19-12_51_00_UTC) 2023-10-19T12:57:40.204 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-10-19-12_52_00_UTC 2023-10-19T12:57:40.204 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:40.204 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-10-19-12_52_00_UTC) 2023-10-19T12:57:40.231 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-10-19-12_53_00_UTC 2023-10-19T12:57:40.231 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:40.232 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-10-19-12_53_00_UTC) 2023-10-19T12:57:40.296 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-10-19-12_54_01_UTC 2023-10-19T12:57:40.297 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:40.297 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-10-19-12_54_01_UTC) 2023-10-19T12:57:40.358 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-10-19-12_55_00_UTC 2023-10-19T12:57:40.359 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:40.359 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-10-19-12_55_00_UTC) 2023-10-19T12:57:40.423 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-10-19-12_56_00_UTC 2023-10-19T12:57:40.423 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:40.424 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-10-19-12_56_00_UTC) 2023-10-19T12:57:40.486 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-10-19-12_57_00_UTC 2023-10-19T12:57:40.486 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:40.487 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-10-19-12_57_00_UTC) 2023-10-19T12:57:40.551 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:40.552 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9) 2023-10-19T12:57:40.611 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-19T12:57:40.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:40.759+0000 7f7a469d0640 1 -- 172.21.15.57:0/3543260126 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7a2c0047c0 msgr2=0x7f7a2c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:40.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:40.759+0000 7f7a469d0640 1 --2- 172.21.15.57:0/3543260126 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7a2c0047c0 0x7f7a2c004c40 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f7a28009b50 tx=0x7f7a2c0057f0 comp rx=0 tx=0).stop 2023-10-19T12:57:40.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:40.759+0000 7f7a44f6d640 1 -- 172.21.15.57:0/3543260126 reap_dead start 2023-10-19T12:57:40.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:40.760+0000 7f7a469d0640 1 -- 172.21.15.57:0/3543260126 shutdown_connections 2023-10-19T12:57:40.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:40.760+0000 7f7a469d0640 1 -- 172.21.15.57:0/3543260126 >> 172.21.15.57:0/3543260126 conn(0x7f7a4006fae0 msgr2=0x7f7a4006fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:40.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:40.760+0000 7f7a469d0640 1 -- 172.21.15.57:0/3543260126 shutdown_connections 2023-10-19T12:57:40.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:40.760+0000 7f7a469d0640 1 -- 172.21.15.57:0/3543260126 wait complete. 2023-10-19T12:57:40.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:40.761+0000 7f7a469d0640 1 Processor -- start 2023-10-19T12:57:40.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:40.761+0000 7f7a469d0640 1 -- start start 2023-10-19T12:57:40.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:40.762+0000 7f7a469d0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7a2c0047c0 0x7f7a40155910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:40.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:40.762+0000 7f7a469d0640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7a400cc450 0x7f7a400cc8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:40.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:40.762+0000 7f7a469d0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7a4014e0f0 0x7f7a4014e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:40.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:40.762+0000 7f7a469d0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a400cce10 con 0x7f7a4014e0f0 2023-10-19T12:57:40.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:40.762+0000 7f7a469d0640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a4014dc60 con 0x7f7a2c0047c0 2023-10-19T12:57:40.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:40.762+0000 7f7a469d0640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a4014d050 con 0x7f7a400cc450 2023-10-19T12:57:40.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:40.762+0000 7f7a44f6d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7a4014e0f0 0x7f7a4014e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:40.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:40.762+0000 7f7a44f6d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7a4014e0f0 0x7f7a4014e570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:53496/0 (socket says 172.21.15.57:53496) 2023-10-19T12:57:40.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:40.762+0000 7f7a44f6d640 1 -- 172.21.15.57:0/2498410212 learned_addr learned my addr 172.21.15.57:0/2498410212 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:57:40.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:40.762+0000 7f7a3ffff640 1 --2- 172.21.15.57:0/2498410212 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7a2c0047c0 0x7f7a40155910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:40.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:40.762+0000 7f7a3f7fe640 1 --2- 172.21.15.57:0/2498410212 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7a400cc450 0x7f7a400cc8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:40.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:40.762+0000 7f7a44f6d640 1 -- 172.21.15.57:0/2498410212 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7a400cc450 msgr2=0x7f7a400cc8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:40.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:40.762+0000 7f7a44f6d640 1 --2- 172.21.15.57:0/2498410212 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7a400cc450 0x7f7a400cc8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:40.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:40.762+0000 7f7a44f6d640 1 -- 172.21.15.57:0/2498410212 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7a2c0047c0 msgr2=0x7f7a40155910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:40.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:40.762+0000 7f7a44f6d640 1 --2- 172.21.15.57:0/2498410212 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7a2c0047c0 0x7f7a40155910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:40.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:40.762+0000 7f7a44f6d640 1 -- 172.21.15.57:0/2498410212 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a400b2330 con 0x7f7a4014e0f0 2023-10-19T12:57:40.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:40.762+0000 7f7a3ffff640 1 --2- 172.21.15.57:0/2498410212 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7a2c0047c0 0x7f7a40155910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:57:40.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:40.762+0000 7f7a44f6d640 1 --2- 172.21.15.57:0/2498410212 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7a4014e0f0 0x7f7a4014e570 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f7a4006b8c0 tx=0x7f7a3800e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:40.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:40.762+0000 7f7a3cff9640 1 -- 172.21.15.57:0/2498410212 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a38003b70 con 0x7f7a4014e0f0 2023-10-19T12:57:40.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:40.762+0000 7f7a3cff9640 1 -- 172.21.15.57:0/2498410212 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f7a38003cd0 con 0x7f7a4014e0f0 2023-10-19T12:57:40.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:40.762+0000 7f7a469d0640 1 -- 172.21.15.57:0/2498410212 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a4014d330 con 0x7f7a4014e0f0 2023-10-19T12:57:40.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:40.762+0000 7f7a3cff9640 1 -- 172.21.15.57:0/2498410212 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a38004290 con 0x7f7a4014e0f0 2023-10-19T12:57:40.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:40.763+0000 7f7a469d0640 1 -- 172.21.15.57:0/2498410212 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a4014d8f0 con 0x7f7a4014e0f0 2023-10-19T12:57:40.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:40.764+0000 7f7a469d0640 1 -- 172.21.15.57:0/2498410212 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a400690e0 con 0x7f7a4014e0f0 2023-10-19T12:57:40.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:40.764+0000 7f7a3cff9640 1 -- 172.21.15.57:0/2498410212 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 15) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f7a38019030 con 0x7f7a4014e0f0 2023-10-19T12:57:40.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:40.764+0000 7f7a3cff9640 1 --2- 172.21.15.57:0/2498410212 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f7a2006b040 0x7f7a2006d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:40.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:40.765+0000 7f7a3cff9640 1 -- 172.21.15.57:0/2498410212 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6900+0+0 (secure 0 0 0) 0x7f7a38017070 con 0x7f7a4014e0f0 2023-10-19T12:57:40.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:40.765+0000 7f7a3ffff640 1 --2- 172.21.15.57:0/2498410212 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f7a2006b040 0x7f7a2006d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:40.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:40.765+0000 7f7a3ffff640 1 --2- 172.21.15.57:0/2498410212 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f7a2006b040 0x7f7a2006d500 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f7a30001eb0 tx=0x7f7a30001d60 comp rx=0 tx=0).ready entity=mgr.5428 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:40.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:40.768+0000 7f7a3cff9640 1 -- 172.21.15.57:0/2498410212 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7a38055550 con 0x7f7a4014e0f0 2023-10-19T12:57:40.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:40.992+0000 7f7a469d0640 1 -- 172.21.15.57:0/2498410212 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f7a400b0b80 con 0x7f7a4014e0f0 2023-10-19T12:57:41.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.156+0000 7f7a3cff9640 1 -- 172.21.15.57:0/2498410212 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v16) v1 ==== 93+0+0 (secure 0 0 0) 0x7f7a3800eea0 con 0x7f7a4014e0f0 2023-10-19T12:57:41.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.162+0000 7f7a469d0640 1 -- 172.21.15.57:0/2498410212 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f7a2006b040 msgr2=0x7f7a2006d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:41.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.162+0000 7f7a469d0640 1 --2- 172.21.15.57:0/2498410212 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f7a2006b040 0x7f7a2006d500 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f7a30001eb0 tx=0x7f7a30001d60 comp rx=0 tx=0).stop 2023-10-19T12:57:41.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.162+0000 7f7a469d0640 1 -- 172.21.15.57:0/2498410212 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7a4014e0f0 msgr2=0x7f7a4014e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:41.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.162+0000 7f7a469d0640 1 --2- 172.21.15.57:0/2498410212 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7a4014e0f0 0x7f7a4014e570 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f7a4006b8c0 tx=0x7f7a3800e720 comp rx=0 tx=0).stop 2023-10-19T12:57:41.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.163+0000 7f7a469d0640 1 -- 172.21.15.57:0/2498410212 shutdown_connections 2023-10-19T12:57:41.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.163+0000 7f7a469d0640 1 --2- 172.21.15.57:0/2498410212 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f7a2006b040 0x7f7a2006d500 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:41.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.163+0000 7f7a469d0640 1 --2- 172.21.15.57:0/2498410212 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7a4014e0f0 0x7f7a4014e570 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:41.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.163+0000 7f7a469d0640 1 --2- 172.21.15.57:0/2498410212 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7a400cc450 0x7f7a400cc8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:41.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.163+0000 7f7a469d0640 1 --2- 172.21.15.57:0/2498410212 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7a2c0047c0 0x7f7a40155910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:41.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.163+0000 7f7a469d0640 1 -- 172.21.15.57:0/2498410212 >> 172.21.15.57:0/2498410212 conn(0x7f7a4006fae0 msgr2=0x7f7a400cb8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:41.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.163+0000 7f7a469d0640 1 -- 172.21.15.57:0/2498410212 shutdown_connections 2023-10-19T12:57:41.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.165+0000 7f7a469d0640 1 -- 172.21.15.57:0/2498410212 wait complete. 2023-10-19T12:57:41.190 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-19T12:57:41.190 DEBUG:teuthology.orchestra.run.smithi057:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-19T12:57:41.255 DEBUG:teuthology.orchestra.run.smithi179:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-19T12:57:41.297 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-19T12:57:41.297 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T12:57:41.298 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T12:57:41.314 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T12:57:41.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.466+0000 7f8a2d7f0640 1 -- 172.21.15.57:0/1623577737 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8a2815b000 msgr2=0x7f8a280d1810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:41.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.466+0000 7f8a2d7f0640 1 --2- 172.21.15.57:0/1623577737 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8a2815b000 0x7f8a280d1810 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f8a1c009860 tx=0x7f8a1c02bde0 comp rx=0 tx=0).stop 2023-10-19T12:57:41.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.466+0000 7f8a2d7f0640 1 -- 172.21.15.57:0/1623577737 shutdown_connections 2023-10-19T12:57:41.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.466+0000 7f8a2d7f0640 1 --2- 172.21.15.57:0/1623577737 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8a2815b000 0x7f8a280d1810 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:41.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.466+0000 7f8a2d7f0640 1 -- 172.21.15.57:0/1623577737 >> 172.21.15.57:0/1623577737 conn(0x7f8a280d0cf0 msgr2=0x7f8a280d1100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:41.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.466+0000 7f8a2d7f0640 1 -- 172.21.15.57:0/1623577737 shutdown_connections 2023-10-19T12:57:41.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.466+0000 7f8a2d7f0640 1 -- 172.21.15.57:0/1623577737 wait complete. 2023-10-19T12:57:41.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.468+0000 7f8a2d7f0640 1 Processor -- start 2023-10-19T12:57:41.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.468+0000 7f8a2d7f0640 1 -- start start 2023-10-19T12:57:41.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.468+0000 7f8a2d7f0640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8a280cf9e0 0x7f8a2814f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:41.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.468+0000 7f8a2d7f0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8a2815b000 0x7f8a281e23d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:41.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.468+0000 7f8a2d7f0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8a2815b3e0 0x7f8a280c9220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:41.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.468+0000 7f8a2d7f0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a281e0fb0 con 0x7f8a2815b3e0 2023-10-19T12:57:41.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.468+0000 7f8a2d7f0640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a280ca150 con 0x7f8a2815b000 2023-10-19T12:57:41.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.468+0000 7f8a2d7f0640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a281e1200 con 0x7f8a280cf9e0 2023-10-19T12:57:41.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.469+0000 7f8a277fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8a2815b3e0 0x7f8a280c9220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:41.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.469+0000 7f8a277fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8a2815b3e0 0x7f8a280c9220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:53512/0 (socket says 172.21.15.57:53512) 2023-10-19T12:57:41.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.469+0000 7f8a277fe640 1 -- 172.21.15.57:0/1161651200 learned_addr learned my addr 172.21.15.57:0/1161651200 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:57:41.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.469+0000 7f8a26ffd640 1 --2- 172.21.15.57:0/1161651200 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8a280cf9e0 0x7f8a2814f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:41.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.469+0000 7f8a267fc640 1 --2- 172.21.15.57:0/1161651200 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8a2815b000 0x7f8a281e23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:41.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.469+0000 7f8a277fe640 1 -- 172.21.15.57:0/1161651200 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8a280cf9e0 msgr2=0x7f8a2814f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:41.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.469+0000 7f8a277fe640 1 --2- 172.21.15.57:0/1161651200 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8a280cf9e0 0x7f8a2814f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:41.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.469+0000 7f8a277fe640 1 -- 172.21.15.57:0/1161651200 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8a2815b000 msgr2=0x7f8a281e23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:41.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.469+0000 7f8a277fe640 1 --2- 172.21.15.57:0/1161651200 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8a2815b000 0x7f8a281e23d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:41.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.469+0000 7f8a277fe640 1 -- 172.21.15.57:0/1161651200 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a1c009490 con 0x7f8a2815b3e0 2023-10-19T12:57:41.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.469+0000 7f8a277fe640 1 --2- 172.21.15.57:0/1161651200 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8a2815b3e0 0x7f8a280c9220 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f8a2000cbb0 tx=0x7f8a2000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:41.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.469+0000 7f8a0f7fe640 1 -- 172.21.15.57:0/1161651200 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a2000ad40 con 0x7f8a2815b3e0 2023-10-19T12:57:41.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.469+0000 7f8a2d7f0640 1 -- 172.21.15.57:0/1161651200 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a281e14e0 con 0x7f8a2815b3e0 2023-10-19T12:57:41.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.469+0000 7f8a0f7fe640 1 -- 172.21.15.57:0/1161651200 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f8a20012420 con 0x7f8a2815b3e0 2023-10-19T12:57:41.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.469+0000 7f8a0f7fe640 1 -- 172.21.15.57:0/1161651200 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a20012830 con 0x7f8a2815b3e0 2023-10-19T12:57:41.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.469+0000 7f8a2d7f0640 1 -- 172.21.15.57:0/1161651200 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a281e1aa0 con 0x7f8a2815b3e0 2023-10-19T12:57:41.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.470+0000 7f8a2d7f0640 1 -- 172.21.15.57:0/1161651200 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a280690e0 con 0x7f8a2815b3e0 2023-10-19T12:57:41.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.471+0000 7f8a0f7fe640 1 -- 172.21.15.57:0/1161651200 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) v1 ==== 89400+0+0 (secure 0 0 0) 0x7f8a2001a3b0 con 0x7f8a2815b3e0 2023-10-19T12:57:41.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.471+0000 7f8a0f7fe640 1 --2- 172.21.15.57:0/1161651200 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f8a1406aff0 0x7f8a1406d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:41.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.471+0000 7f8a26ffd640 1 -- 172.21.15.57:0/1161651200 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f8a1406aff0 msgr2=0x7f8a1406d4b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/922398470 2023-10-19T12:57:41.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.471+0000 7f8a0f7fe640 1 -- 172.21.15.57:0/1161651200 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6884+0+0 (secure 0 0 0) 0x7f8a2008a520 con 0x7f8a2815b3e0 2023-10-19T12:57:41.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.471+0000 7f8a26ffd640 1 --2- 172.21.15.57:0/1161651200 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f8a1406aff0 0x7f8a1406d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T12:57:41.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.475+0000 7f8a0f7fe640 1 -- 172.21.15.57:0/1161651200 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8a20054ee0 con 0x7f8a2815b3e0 2023-10-19T12:57:41.578 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:57:41.576+0000 7fa555645040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-19T12:57:41.594 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:57:41.592+0000 7fb99e00d040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-19T12:57:41.623 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:57:41.620+0000 7fa555645040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-19T12:57:41.642 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:57:41.639+0000 7fb99e00d040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-19T12:57:41.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.665+0000 7f8a2d7f0640 1 -- 172.21.15.57:0/1161651200 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8a280d1450 con 0x7f8a2815b3e0 2023-10-19T12:57:41.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.666+0000 7f8a0f7fe640 1 -- 172.21.15.57:0/1161651200 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v54) v1 ==== 74+0+18047 (secure 0 0 0) 0x7f8a20058500 con 0x7f8a2815b3e0 2023-10-19T12:57:41.670 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:57:41.670 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":54,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T12:57:41.146471+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2023-10-19T12:45:38.964640+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2023-10-19T12:45:39.683197+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":225,"snap_epoch":54,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.31","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.179:6833/2609646285":"2023-10-20T12:45:56.327729+0000","172.21.15.179:6832/2609646285":"2023-10-20T12:45:56.327729+0000","172.21.15.179:0/1377585784":"2023-10-20T12:45:56.327729+0000","172.21.15.179:0/515590766":"2023-10-20T12:45:56.327729+0000","172.21.15.179:0/2223195228":"2023-10-20T12:45:56.327729+0000","172.21.15.179:0/2313691653":"2023-10-20T12:45:56.327729+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":2,"length":2},{"begin":5,"length":2},{"begin":8,"length":1},{"begin":10,"length":3},{"begin":14,"length":5},{"begin":20,"length":3},{"begin":24,"length":6},{"begin":31,"length":3},{"begin":35,"length":6},{"begin":42,"length":3},{"begin":47,"length":4},{"begin":52,"length":3},{"begin":57,"length":4},{"begin":62,"length":3},{"begin":67,"length":4},{"begin":72,"length":3},{"begin":77,"length":5},{"begin":83,"length":3},{"begin":89,"length":4},{"begin":94,"length":3},{"begin":98,"length":1},{"begin":100,"length":4},{"begin":105,"length":5},{"begin":111,"length":4},{"begin":116,"length":2},{"begin":140,"length":86}]}],"new_removed_snaps":[{"pool":6,"snaps":[{"begin":14,"length":1},{"begin":24,"length":1},{"begin":35,"length":1},{"begin":83,"length":1},{"begin":94,"length":1},{"begin":105,"length":1},{"begin":116,"length":1},{"begin":219,"length":7}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T12:57:41.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.669+0000 7f8a2d7f0640 1 -- 172.21.15.57:0/1161651200 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f8a1406aff0 msgr2=0x7f8a1406d4b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T12:57:41.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.669+0000 7f8a2d7f0640 1 --2- 172.21.15.57:0/1161651200 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f8a1406aff0 0x7f8a1406d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:41.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.669+0000 7f8a2d7f0640 1 -- 172.21.15.57:0/1161651200 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8a2815b3e0 msgr2=0x7f8a280c9220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:41.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.669+0000 7f8a2d7f0640 1 --2- 172.21.15.57:0/1161651200 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8a2815b3e0 0x7f8a280c9220 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f8a2000cbb0 tx=0x7f8a2000a430 comp rx=0 tx=0).stop 2023-10-19T12:57:41.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.669+0000 7f8a2d7f0640 1 -- 172.21.15.57:0/1161651200 shutdown_connections 2023-10-19T12:57:41.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.669+0000 7f8a2d7f0640 1 --2- 172.21.15.57:0/1161651200 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f8a1406aff0 0x7f8a1406d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:41.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.669+0000 7f8a2d7f0640 1 --2- 172.21.15.57:0/1161651200 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8a2815b3e0 0x7f8a280c9220 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:41.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.669+0000 7f8a2d7f0640 1 --2- 172.21.15.57:0/1161651200 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8a2815b000 0x7f8a281e23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:41.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.669+0000 7f8a2d7f0640 1 --2- 172.21.15.57:0/1161651200 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8a280cf9e0 0x7f8a2814f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:41.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.669+0000 7f8a2d7f0640 1 -- 172.21.15.57:0/1161651200 >> 172.21.15.57:0/1161651200 conn(0x7f8a280d0cf0 msgr2=0x7f8a2815a180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:41.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.669+0000 7f8a2d7f0640 1 -- 172.21.15.57:0/1161651200 shutdown_connections 2023-10-19T12:57:41.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.669+0000 7f8a2d7f0640 1 -- 172.21.15.57:0/1161651200 wait complete. 2023-10-19T12:57:41.683 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-19T12:44:07.371186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-19T12:45:38.964640+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2023-10-19T12:45:39.683197+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '54', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 225, 'snap_epoch': 54, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-19T12:57:41.684 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-19T12:57:41.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.795+0000 7efc8fd16640 1 -- 172.21.15.57:0/2298127239 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efc88159060 msgr2=0x7efc8806d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:41.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.795+0000 7efc8fd16640 1 --2- 172.21.15.57:0/2298127239 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efc88159060 0x7efc8806d260 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7efc880ceab0 tx=0x7efc8402bd10 comp rx=0 tx=0).stop 2023-10-19T12:57:41.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.795+0000 7efc8fd16640 1 -- 172.21.15.57:0/2298127239 shutdown_connections 2023-10-19T12:57:41.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.795+0000 7efc8fd16640 1 --2- 172.21.15.57:0/2298127239 >> v2:172.21.15.179:3300/0 conn(0x7efc8815ccf0 0x7efc8815d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:41.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.795+0000 7efc8fd16640 1 --2- 172.21.15.57:0/2298127239 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7efc880cdb30 0x7efc880cdfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:41.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.795+0000 7efc8fd16640 1 --2- 172.21.15.57:0/2298127239 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efc88159060 0x7efc8806d260 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:41.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.795+0000 7efc8fd16640 1 -- 172.21.15.57:0/2298127239 >> 172.21.15.57:0/2298127239 conn(0x7efc880b0670 msgr2=0x7efc880b0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:41.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.795+0000 7efc8fd16640 1 -- 172.21.15.57:0/2298127239 shutdown_connections 2023-10-19T12:57:41.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.795+0000 7efc8fd16640 1 -- 172.21.15.57:0/2298127239 wait complete. 2023-10-19T12:57:41.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.796+0000 7efc8fd16640 1 Processor -- start 2023-10-19T12:57:41.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.796+0000 7efc8fd16640 1 -- start start 2023-10-19T12:57:41.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.797+0000 7efc8fd16640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efc880cdb30 0x7efc88155fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:41.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.797+0000 7efc8fd16640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7efc88159060 0x7efc8815a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:41.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.797+0000 7efc8fd16640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efc8815ccf0 0x7efc880b2d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:41.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.797+0000 7efc8fd16640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc8815ab70 con 0x7efc880cdb30 2023-10-19T12:57:41.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.797+0000 7efc8fd16640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc8815b9d0 con 0x7efc8815ccf0 2023-10-19T12:57:41.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.797+0000 7efc8fd16640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc88156a60 con 0x7efc88159060 2023-10-19T12:57:41.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.797+0000 7efc8dab2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efc880cdb30 0x7efc88155fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:41.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.797+0000 7efc8dab2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efc880cdb30 0x7efc88155fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:53532/0 (socket says 172.21.15.57:53532) 2023-10-19T12:57:41.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.797+0000 7efc8dab2640 1 -- 172.21.15.57:0/673716975 learned_addr learned my addr 172.21.15.57:0/673716975 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:57:41.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.797+0000 7efc8e2b3640 1 --2- 172.21.15.57:0/673716975 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efc8815ccf0 0x7efc880b2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:41.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.797+0000 7efc8d2b1640 1 --2- 172.21.15.57:0/673716975 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7efc88159060 0x7efc8815a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:41.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.797+0000 7efc8dab2640 1 -- 172.21.15.57:0/673716975 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7efc88159060 msgr2=0x7efc8815a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:41.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.797+0000 7efc8dab2640 1 --2- 172.21.15.57:0/673716975 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7efc88159060 0x7efc8815a150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:41.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.797+0000 7efc8dab2640 1 -- 172.21.15.57:0/673716975 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efc8815ccf0 msgr2=0x7efc880b2d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:41.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.797+0000 7efc8dab2640 1 --2- 172.21.15.57:0/673716975 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efc8815ccf0 0x7efc880b2d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:41.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.797+0000 7efc8dab2640 1 -- 172.21.15.57:0/673716975 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc84009560 con 0x7efc880cdb30 2023-10-19T12:57:41.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.797+0000 7efc8dab2640 1 --2- 172.21.15.57:0/673716975 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efc880cdb30 0x7efc88155fa0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7efc84036040 tx=0x7efc8402d970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:41.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.798+0000 7efc7e7fc640 1 -- 172.21.15.57:0/673716975 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc8402dd70 con 0x7efc880cdb30 2023-10-19T12:57:41.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.798+0000 7efc7e7fc640 1 -- 172.21.15.57:0/673716975 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7efc840035f0 con 0x7efc880cdb30 2023-10-19T12:57:41.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.798+0000 7efc7e7fc640 1 -- 172.21.15.57:0/673716975 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc840039c0 con 0x7efc880cdb30 2023-10-19T12:57:41.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.798+0000 7efc8fd16640 1 -- 172.21.15.57:0/673716975 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc881564e0 con 0x7efc880cdb30 2023-10-19T12:57:41.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.798+0000 7efc8fd16640 1 -- 172.21.15.57:0/673716975 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc8815b0a0 con 0x7efc880cdb30 2023-10-19T12:57:41.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.799+0000 7efc8fd16640 1 -- 172.21.15.57:0/673716975 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc880675f0 con 0x7efc880cdb30 2023-10-19T12:57:41.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.799+0000 7efc7e7fc640 1 -- 172.21.15.57:0/673716975 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) v1 ==== 89400+0+0 (secure 0 0 0) 0x7efc84003b20 con 0x7efc880cdb30 2023-10-19T12:57:41.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.799+0000 7efc7e7fc640 1 --2- 172.21.15.57:0/673716975 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7efc6c06b020 0x7efc6c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:41.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.800+0000 7efc8d2b1640 1 -- 172.21.15.57:0/673716975 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7efc6c06b020 msgr2=0x7efc6c06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/922398470 2023-10-19T12:57:41.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.800+0000 7efc7e7fc640 1 -- 172.21.15.57:0/673716975 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6884+0+0 (secure 0 0 0) 0x7efc840ac3b0 con 0x7efc880cdb30 2023-10-19T12:57:41.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.800+0000 7efc8d2b1640 1 --2- 172.21.15.57:0/673716975 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7efc6c06b020 0x7efc6c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T12:57:41.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.804+0000 7efc7e7fc640 1 -- 172.21.15.57:0/673716975 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7efc84076f10 con 0x7efc880cdb30 2023-10-19T12:57:41.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.978+0000 7efc8fd16640 1 -- 172.21.15.57:0/673716975 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7efc88157160 con 0x7efc880cdb30 2023-10-19T12:57:41.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.979+0000 7efc7e7fc640 1 -- 172.21.15.57:0/673716975 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v54) v1 ==== 93+0+10 (secure 0 0 0) 0x7efc8407a530 con 0x7efc880cdb30 2023-10-19T12:57:41.982 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-19T12:57:41.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.980+0000 7efc8fd16640 1 -- 172.21.15.57:0/673716975 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7efc6c06b020 msgr2=0x7efc6c06d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T12:57:41.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.980+0000 7efc8fd16640 1 --2- 172.21.15.57:0/673716975 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7efc6c06b020 0x7efc6c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:41.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.980+0000 7efc8fd16640 1 -- 172.21.15.57:0/673716975 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efc880cdb30 msgr2=0x7efc88155fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:41.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.980+0000 7efc8fd16640 1 --2- 172.21.15.57:0/673716975 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efc880cdb30 0x7efc88155fa0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7efc84036040 tx=0x7efc8402d970 comp rx=0 tx=0).stop 2023-10-19T12:57:41.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.981+0000 7efc8fd16640 1 -- 172.21.15.57:0/673716975 shutdown_connections 2023-10-19T12:57:41.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.981+0000 7efc8fd16640 1 --2- 172.21.15.57:0/673716975 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7efc6c06b020 0x7efc6c06d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:41.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.981+0000 7efc8fd16640 1 --2- 172.21.15.57:0/673716975 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efc8815ccf0 0x7efc880b2d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:41.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.981+0000 7efc8fd16640 1 --2- 172.21.15.57:0/673716975 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7efc88159060 0x7efc8815a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:41.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.981+0000 7efc8fd16640 1 --2- 172.21.15.57:0/673716975 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efc880cdb30 0x7efc88155fa0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:41.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.981+0000 7efc8fd16640 1 -- 172.21.15.57:0/673716975 >> 172.21.15.57:0/673716975 conn(0x7efc880b0670 msgr2=0x7efc880cca20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:41.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.981+0000 7efc8fd16640 1 -- 172.21.15.57:0/673716975 shutdown_connections 2023-10-19T12:57:41.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:41.981+0000 7efc8fd16640 1 -- 172.21.15.57:0/673716975 wait complete. 2023-10-19T12:57:41.994 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-19T12:57:42.009 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:57:42.007+0000 7fa555645040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-19T12:57:42.052 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:57:42.050+0000 7fa555645040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-19T12:57:42.059 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:57:42.056+0000 7fb99e00d040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-19T12:57:42.094 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:57:42.092+0000 7fa555645040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-19T12:57:42.106 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:57:42.104+0000 7fb99e00d040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-19T12:57:42.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.105+0000 7fbf5b097640 1 -- 172.21.15.57:0/1590800305 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbf5415b2a0 msgr2=0x7fbf540d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:42.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.105+0000 7fbf5b097640 1 --2- 172.21.15.57:0/1590800305 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbf5415b2a0 0x7fbf540d1910 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fbf40009830 tx=0x7fbf4002bcd0 comp rx=0 tx=0).stop 2023-10-19T12:57:42.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.105+0000 7fbf5b097640 1 -- 172.21.15.57:0/1590800305 shutdown_connections 2023-10-19T12:57:42.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.105+0000 7fbf5b097640 1 --2- 172.21.15.57:0/1590800305 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbf540ceb40 0x7fbf540b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:42.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.105+0000 7fbf5b097640 1 --2- 172.21.15.57:0/1590800305 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbf5415b2a0 0x7fbf540d1910 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:42.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.105+0000 7fbf5b097640 1 -- 172.21.15.57:0/1590800305 >> 172.21.15.57:0/1590800305 conn(0x7fbf540d0d60 msgr2=0x7fbf540d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:42.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.105+0000 7fbf5b097640 1 -- 172.21.15.57:0/1590800305 shutdown_connections 2023-10-19T12:57:42.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.105+0000 7fbf5b097640 1 -- 172.21.15.57:0/1590800305 wait complete. 2023-10-19T12:57:42.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.106+0000 7fbf5b097640 1 Processor -- start 2023-10-19T12:57:42.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.106+0000 7fbf5b097640 1 -- start start 2023-10-19T12:57:42.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.107+0000 7fbf5b097640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbf540ceb40 0x7fbf540c99b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:42.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.107+0000 7fbf5b097640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbf540cf340 0x7fbf541f4390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:42.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.107+0000 7fbf5b097640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbf5415b2a0 0x7fbf540aea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:42.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.107+0000 7fbf5b097640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf541e5450 con 0x7fbf5415b2a0 2023-10-19T12:57:42.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.107+0000 7fbf5b097640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf541e5730 con 0x7fbf540cf340 2023-10-19T12:57:42.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.107+0000 7fbf5b097640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf541e47d0 con 0x7fbf540ceb40 2023-10-19T12:57:42.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.107+0000 7fbf59634640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbf5415b2a0 0x7fbf540aea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:42.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.107+0000 7fbf59634640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbf5415b2a0 0x7fbf540aea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:53540/0 (socket says 172.21.15.57:53540) 2023-10-19T12:57:42.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.107+0000 7fbf59634640 1 -- 172.21.15.57:0/2814373708 learned_addr learned my addr 172.21.15.57:0/2814373708 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:57:42.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.107+0000 7fbf58e33640 1 --2- 172.21.15.57:0/2814373708 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbf540ceb40 0x7fbf540c99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:42.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.107+0000 7fbf53fff640 1 --2- 172.21.15.57:0/2814373708 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbf540cf340 0x7fbf541f4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:42.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.108+0000 7fbf58e33640 1 -- 172.21.15.57:0/2814373708 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbf540cf340 msgr2=0x7fbf541f4390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:42.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.108+0000 7fbf58e33640 1 --2- 172.21.15.57:0/2814373708 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbf540cf340 0x7fbf541f4390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:42.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.108+0000 7fbf58e33640 1 -- 172.21.15.57:0/2814373708 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbf5415b2a0 msgr2=0x7fbf540aea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:42.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.108+0000 7fbf58e33640 1 --2- 172.21.15.57:0/2814373708 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbf5415b2a0 0x7fbf540aea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:42.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.108+0000 7fbf58e33640 1 -- 172.21.15.57:0/2814373708 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf4c009840 con 0x7fbf540ceb40 2023-10-19T12:57:42.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.108+0000 7fbf59634640 1 --2- 172.21.15.57:0/2814373708 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbf5415b2a0 0x7fbf540aea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T12:57:42.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.108+0000 7fbf58e33640 1 --2- 172.21.15.57:0/2814373708 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbf540ceb40 0x7fbf540c99b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fbf4800c370 tx=0x7fbf4800c830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:42.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.108+0000 7fbf517fa640 1 -- 172.21.15.57:0/2814373708 <== mon.2 v2:172.21.15.179:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf48004230 con 0x7fbf540ceb40 2023-10-19T12:57:42.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.108+0000 7fbf517fa640 1 -- 172.21.15.57:0/2814373708 <== mon.2 v2:172.21.15.179:3301/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fbf480044a0 con 0x7fbf540ceb40 2023-10-19T12:57:42.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.108+0000 7fbf5b097640 1 -- 172.21.15.57:0/2814373708 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf40009490 con 0x7fbf540ceb40 2023-10-19T12:57:42.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.108+0000 7fbf517fa640 1 -- 172.21.15.57:0/2814373708 <== mon.2 v2:172.21.15.179:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf48003850 con 0x7fbf540ceb40 2023-10-19T12:57:42.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.108+0000 7fbf5b097640 1 -- 172.21.15.57:0/2814373708 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf541e4d90 con 0x7fbf540ceb40 2023-10-19T12:57:42.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.109+0000 7fbf5b097640 1 -- 172.21.15.57:0/2814373708 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf541e5590 con 0x7fbf540ceb40 2023-10-19T12:57:42.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.110+0000 7fbf517fa640 1 -- 172.21.15.57:0/2814373708 <== mon.2 v2:172.21.15.179:3301/0 4 ==== mgrmap(e 16) v1 ==== 89400+0+0 (secure 0 0 0) 0x7fbf480039b0 con 0x7fbf540ceb40 2023-10-19T12:57:42.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.110+0000 7fbf517fa640 1 --2- 172.21.15.57:0/2814373708 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fbf3806aff0 0x7fbf3806d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:42.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.110+0000 7fbf53fff640 1 -- 172.21.15.57:0/2814373708 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fbf3806aff0 msgr2=0x7fbf3806d4b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/922398470 2023-10-19T12:57:42.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.110+0000 7fbf517fa640 1 -- 172.21.15.57:0/2814373708 <== mon.2 v2:172.21.15.179:3301/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6884+0+0 (secure 0 0 0) 0x7fbf48089330 con 0x7fbf540ceb40 2023-10-19T12:57:42.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.110+0000 7fbf53fff640 1 --2- 172.21.15.57:0/2814373708 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fbf3806aff0 0x7fbf3806d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T12:57:42.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.115+0000 7fbf517fa640 1 -- 172.21.15.57:0/2814373708 <== mon.2 v2:172.21.15.179:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fbf541e5590 con 0x7fbf540ceb40 2023-10-19T12:57:42.147 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:57:42.144+0000 7fa555645040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-19T12:57:42.152 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:57:42.149+0000 7fb99e00d040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-19T12:57:42.203 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:57:42.201+0000 7fa555645040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-19T12:57:42.227 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:57:42.224+0000 7fb99e00d040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-19T12:57:42.278 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:57:42.275+0000 7fb99e00d040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-19T12:57:42.309 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:57:42.306+0000 7fa555645040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-19T12:57:42.313 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.311+0000 7fbf53fff640 1 -- 172.21.15.57:0/2814373708 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fbf3806aff0 msgr2=0x7fbf3806d4b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/922398470 2023-10-19T12:57:42.313 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.311+0000 7fbf53fff640 1 --2- 172.21.15.57:0/2814373708 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fbf3806aff0 0x7fbf3806d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-19T12:57:42.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.326+0000 7fbf5b097640 1 -- 172.21.15.57:0/2814373708 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fbf541e5590 con 0x7fbf540ceb40 2023-10-19T12:57:42.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.327+0000 7fbf517fa640 1 -- 172.21.15.57:0/2814373708 <== mon.2 v2:172.21.15.179:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v55) v1 ==== 104+0+11 (secure 0 0 0) 0x7fbf48017020 con 0x7fbf540ceb40 2023-10-19T12:57:42.330 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2023-10-19T12:57:42.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.329+0000 7fbf5b097640 1 -- 172.21.15.57:0/2814373708 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fbf3806aff0 msgr2=0x7fbf3806d4b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T12:57:42.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.329+0000 7fbf5b097640 1 --2- 172.21.15.57:0/2814373708 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fbf3806aff0 0x7fbf3806d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:42.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.329+0000 7fbf5b097640 1 -- 172.21.15.57:0/2814373708 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbf540ceb40 msgr2=0x7fbf540c99b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:42.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.329+0000 7fbf5b097640 1 --2- 172.21.15.57:0/2814373708 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbf540ceb40 0x7fbf540c99b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fbf4800c370 tx=0x7fbf4800c830 comp rx=0 tx=0).stop 2023-10-19T12:57:42.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.329+0000 7fbf5b097640 1 -- 172.21.15.57:0/2814373708 shutdown_connections 2023-10-19T12:57:42.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.329+0000 7fbf5b097640 1 --2- 172.21.15.57:0/2814373708 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fbf3806aff0 0x7fbf3806d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:42.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.329+0000 7fbf5b097640 1 --2- 172.21.15.57:0/2814373708 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbf5415b2a0 0x7fbf540aea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:42.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.329+0000 7fbf5b097640 1 --2- 172.21.15.57:0/2814373708 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbf540cf340 0x7fbf541f4390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:42.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.329+0000 7fbf5b097640 1 --2- 172.21.15.57:0/2814373708 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbf540ceb40 0x7fbf540c99b0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:42.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.329+0000 7fbf5b097640 1 -- 172.21.15.57:0/2814373708 >> 172.21.15.57:0/2814373708 conn(0x7fbf540d0d60 msgr2=0x7fbf54069f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:42.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.329+0000 7fbf5b097640 1 -- 172.21.15.57:0/2814373708 shutdown_connections 2023-10-19T12:57:42.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.329+0000 7fbf5b097640 1 -- 172.21.15.57:0/2814373708 wait complete. 2023-10-19T12:57:42.342 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-19T12:57:42.361 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:57:42.358+0000 7fa555645040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-19T12:57:42.383 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:57:42.381+0000 7fb99e00d040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-19T12:57:42.409 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:57:42.406+0000 7fa555645040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-19T12:57:42.442 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:57:42.440+0000 7fb99e00d040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-19T12:57:42.456 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:57:42.454+0000 7fa555645040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-19T12:57:42.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.455+0000 7fe1bec44640 1 -- 172.21.15.57:0/285065463 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1b80cf340 msgr2=0x7fe1b80d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:42.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.455+0000 7fe1bec44640 1 --2- 172.21.15.57:0/285065463 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1b80cf340 0x7fe1b80d1910 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fe1b815b680 tx=0x7fe1ac02bdb0 comp rx=0 tx=0).stop 2023-10-19T12:57:42.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.455+0000 7fe1bec44640 1 -- 172.21.15.57:0/285065463 shutdown_connections 2023-10-19T12:57:42.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.455+0000 7fe1bec44640 1 --2- 172.21.15.57:0/285065463 >> v2:172.21.15.179:3300/0 conn(0x7fe1b80ceb40 0x7fe1b80b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:42.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.455+0000 7fe1bec44640 1 --2- 172.21.15.57:0/285065463 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1b80cf340 0x7fe1b80d1910 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:42.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.455+0000 7fe1bec44640 1 -- 172.21.15.57:0/285065463 >> 172.21.15.57:0/285065463 conn(0x7fe1b80d0d60 msgr2=0x7fe1b80d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:42.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.456+0000 7fe1bec44640 1 -- 172.21.15.57:0/285065463 shutdown_connections 2023-10-19T12:57:42.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.456+0000 7fe1bec44640 1 -- 172.21.15.57:0/285065463 wait complete. 2023-10-19T12:57:42.462 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.457+0000 7fe1bec44640 1 Processor -- start 2023-10-19T12:57:42.462 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.457+0000 7fe1bec44640 1 -- start start 2023-10-19T12:57:42.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.458+0000 7fe1bec44640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe1b80ceb40 0x7fe1b80b58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:42.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.458+0000 7fe1bec44640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe1b80cf340 0x7fe1b815def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:42.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.458+0000 7fe1bec44640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1b815b2a0 0x7fe1b81f17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:42.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.458+0000 7fe1bec44640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1b80b54c0 con 0x7fe1b815b2a0 2023-10-19T12:57:42.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.458+0000 7fe1bec44640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1b81f51e0 con 0x7fe1b80ceb40 2023-10-19T12:57:42.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.458+0000 7fe1bec44640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1b81f54c0 con 0x7fe1b80cf340 2023-10-19T12:57:42.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.458+0000 7fe1bd1e1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1b815b2a0 0x7fe1b81f17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:42.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.458+0000 7fe1bd1e1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1b815b2a0 0x7fe1b81f17a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:53556/0 (socket says 172.21.15.57:53556) 2023-10-19T12:57:42.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.458+0000 7fe1bc9e0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe1b80ceb40 0x7fe1b80b58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:42.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.458+0000 7fe1bd1e1640 1 -- 172.21.15.57:0/1037576891 learned_addr learned my addr 172.21.15.57:0/1037576891 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:57:42.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.458+0000 7fe1b7fff640 1 --2- 172.21.15.57:0/1037576891 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe1b80cf340 0x7fe1b815def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:42.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.458+0000 7fe1bd1e1640 1 -- 172.21.15.57:0/1037576891 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe1b80cf340 msgr2=0x7fe1b815def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:42.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.458+0000 7fe1bd1e1640 1 --2- 172.21.15.57:0/1037576891 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe1b80cf340 0x7fe1b815def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:42.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.458+0000 7fe1bd1e1640 1 -- 172.21.15.57:0/1037576891 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe1b80ceb40 msgr2=0x7fe1b80b58f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:42.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.458+0000 7fe1bd1e1640 1 --2- 172.21.15.57:0/1037576891 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe1b80ceb40 0x7fe1b80b58f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:42.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.458+0000 7fe1bd1e1640 1 -- 172.21.15.57:0/1037576891 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1ac009560 con 0x7fe1b815b2a0 2023-10-19T12:57:42.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.458+0000 7fe1bc9e0640 1 --2- 172.21.15.57:0/1037576891 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe1b80ceb40 0x7fe1b80b58f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T12:57:42.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.458+0000 7fe1bd1e1640 1 --2- 172.21.15.57:0/1037576891 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1b815b2a0 0x7fe1b81f17a0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fe1b000eb10 tx=0x7fe1b000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:42.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.458+0000 7fe1b57fa640 1 -- 172.21.15.57:0/1037576891 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1b0004b20 con 0x7fe1b815b2a0 2023-10-19T12:57:42.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.459+0000 7fe1b57fa640 1 -- 172.21.15.57:0/1037576891 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fe1b0004ca0 con 0x7fe1b815b2a0 2023-10-19T12:57:42.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.459+0000 7fe1b57fa640 1 -- 172.21.15.57:0/1037576891 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1b0005280 con 0x7fe1b815b2a0 2023-10-19T12:57:42.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.459+0000 7fe1bec44640 1 -- 172.21.15.57:0/1037576891 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1b81f57a0 con 0x7fe1b815b2a0 2023-10-19T12:57:42.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.459+0000 7fe1bec44640 1 -- 172.21.15.57:0/1037576891 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1b80b5e30 con 0x7fe1b815b2a0 2023-10-19T12:57:42.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.460+0000 7fe1bec44640 1 -- 172.21.15.57:0/1037576891 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1b80cfc50 con 0x7fe1b815b2a0 2023-10-19T12:57:42.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.460+0000 7fe1b57fa640 1 -- 172.21.15.57:0/1037576891 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) v1 ==== 89400+0+0 (secure 0 0 0) 0x7fe1b00053e0 con 0x7fe1b815b2a0 2023-10-19T12:57:42.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.460+0000 7fe1b57fa640 1 --2- 172.21.15.57:0/1037576891 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fe19c06b020 0x7fe19c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:42.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.461+0000 7fe1bc9e0640 1 -- 172.21.15.57:0/1037576891 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fe19c06b020 msgr2=0x7fe19c06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/922398470 2023-10-19T12:57:42.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.461+0000 7fe1b57fa640 1 -- 172.21.15.57:0/1037576891 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6852+0+0 (secure 0 0 0) 0x7fe1b0014070 con 0x7fe1b815b2a0 2023-10-19T12:57:42.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.461+0000 7fe1bc9e0640 1 --2- 172.21.15.57:0/1037576891 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fe19c06b020 0x7fe19c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T12:57:42.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.465+0000 7fe1b57fa640 1 -- 172.21.15.57:0/1037576891 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe1b0055440 con 0x7fe1b815b2a0 2023-10-19T12:57:42.488 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:57:42.486+0000 7fb99e00d040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-19T12:57:42.500 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:57:42.497+0000 7fa555645040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-19T12:57:42.539 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:57:42.536+0000 7fb99e00d040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-19T12:57:42.584 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:57:42.581+0000 7fb99e00d040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-19T12:57:42.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.641+0000 7fe1bec44640 1 -- 172.21.15.57:0/1037576891 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe1b80b0d00 con 0x7fe1b815b2a0 2023-10-19T12:57:42.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.641+0000 7fe1b57fa640 1 -- 172.21.15.57:0/1037576891 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v55) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe1b0058a60 con 0x7fe1b815b2a0 2023-10-19T12:57:42.644 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 64 2023-10-19T12:57:42.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.643+0000 7fe1bec44640 1 -- 172.21.15.57:0/1037576891 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fe19c06b020 msgr2=0x7fe19c06d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T12:57:42.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.643+0000 7fe1bec44640 1 --2- 172.21.15.57:0/1037576891 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fe19c06b020 0x7fe19c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:42.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.643+0000 7fe1bec44640 1 -- 172.21.15.57:0/1037576891 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1b815b2a0 msgr2=0x7fe1b81f17a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:42.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.643+0000 7fe1bec44640 1 --2- 172.21.15.57:0/1037576891 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1b815b2a0 0x7fe1b81f17a0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fe1b000eb10 tx=0x7fe1b000c2d0 comp rx=0 tx=0).stop 2023-10-19T12:57:42.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.643+0000 7fe1bec44640 1 -- 172.21.15.57:0/1037576891 shutdown_connections 2023-10-19T12:57:42.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.643+0000 7fe1bec44640 1 --2- 172.21.15.57:0/1037576891 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fe19c06b020 0x7fe19c06d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:42.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.643+0000 7fe1bec44640 1 --2- 172.21.15.57:0/1037576891 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1b815b2a0 0x7fe1b81f17a0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:42.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.643+0000 7fe1bec44640 1 --2- 172.21.15.57:0/1037576891 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe1b80cf340 0x7fe1b815def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:42.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.643+0000 7fe1bec44640 1 --2- 172.21.15.57:0/1037576891 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe1b80ceb40 0x7fe1b80b58f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:42.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.643+0000 7fe1bec44640 1 -- 172.21.15.57:0/1037576891 >> 172.21.15.57:0/1037576891 conn(0x7fe1b80d0d60 msgr2=0x7fe1b815a440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:42.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.643+0000 7fe1bec44640 1 -- 172.21.15.57:0/1037576891 shutdown_connections 2023-10-19T12:57:42.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.643+0000 7fe1bec44640 1 -- 172.21.15.57:0/1037576891 wait complete. 2023-10-19T12:57:42.657 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-19T12:57:42.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.811+0000 7f731a09b640 1 -- 172.21.15.57:0/3858155173 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7314140660 msgr2=0x7f731415a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:42.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.811+0000 7f731a09b640 1 --2- 172.21.15.57:0/3858155173 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7314140660 0x7f731415a9f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f73140cbd90 tx=0x7f730802df00 comp rx=0 tx=0).stop 2023-10-19T12:57:42.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.812+0000 7f731a09b640 1 -- 172.21.15.57:0/3858155173 shutdown_connections 2023-10-19T12:57:42.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.812+0000 7f731a09b640 1 --2- 172.21.15.57:0/3858155173 >> v2:172.21.15.179:3300/0 conn(0x7f73140ca870 0x7f73140cad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:42.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.812+0000 7f731a09b640 1 --2- 172.21.15.57:0/3858155173 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7314140660 0x7f731415a9f0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:42.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.812+0000 7f731a09b640 1 --2- 172.21.15.57:0/3858155173 >> v2:172.21.15.57:3300/0 conn(0x7f731406d170 0x7f731413d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:42.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.812+0000 7f731a09b640 1 -- 172.21.15.57:0/3858155173 >> 172.21.15.57:0/3858155173 conn(0x7f7314069930 msgr2=0x7f7314069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:42.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.812+0000 7f731a09b640 1 -- 172.21.15.57:0/3858155173 shutdown_connections 2023-10-19T12:57:42.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.812+0000 7f731a09b640 1 -- 172.21.15.57:0/3858155173 wait complete. 2023-10-19T12:57:42.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.814+0000 7f731a09b640 1 Processor -- start 2023-10-19T12:57:42.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.814+0000 7f731a09b640 1 -- start start 2023-10-19T12:57:42.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.815+0000 7f731a09b640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f731406d170 0x7f7314156740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:42.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.815+0000 7f731a09b640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f73140ca870 0x7f73140ac590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:42.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.815+0000 7f731a09b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7314140660 0x7f73141f53d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:42.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.815+0000 7f731a09b640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73141f6430 con 0x7f7314140660 2023-10-19T12:57:42.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.815+0000 7f731a09b640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73140afbf0 con 0x7f731406d170 2023-10-19T12:57:42.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.815+0000 7f731a09b640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73141f5e70 con 0x7f73140ca870 2023-10-19T12:57:42.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.816+0000 7f7313fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7314140660 0x7f73141f53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:42.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.816+0000 7f7313fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7314140660 0x7f73141f53d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:53568/0 (socket says 172.21.15.57:53568) 2023-10-19T12:57:42.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.816+0000 7f7313fff640 1 -- 172.21.15.57:0/3185607329 learned_addr learned my addr 172.21.15.57:0/3185607329 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:57:42.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.816+0000 7f73137fe640 1 --2- 172.21.15.57:0/3185607329 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f731406d170 0x7f7314156740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:42.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.816+0000 7f7312ffd640 1 --2- 172.21.15.57:0/3185607329 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f73140ca870 0x7f73140ac590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:42.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.816+0000 7f7313fff640 1 -- 172.21.15.57:0/3185607329 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f73140ca870 msgr2=0x7f73140ac590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:42.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.816+0000 7f7313fff640 1 --2- 172.21.15.57:0/3185607329 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f73140ca870 0x7f73140ac590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:42.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.816+0000 7f7313fff640 1 -- 172.21.15.57:0/3185607329 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f731406d170 msgr2=0x7f7314156740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:42.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.816+0000 7f7313fff640 1 --2- 172.21.15.57:0/3185607329 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f731406d170 0x7f7314156740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:42.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.816+0000 7f7313fff640 1 -- 172.21.15.57:0/3185607329 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7308009490 con 0x7f7314140660 2023-10-19T12:57:42.823 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:57:42.819+0000 7fa555645040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-19T12:57:42.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.816+0000 7f7313fff640 1 --2- 172.21.15.57:0/3185607329 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7314140660 0x7f73141f53d0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f730c00e9c0 tx=0x7f730c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:42.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.817+0000 7f72fbfff640 1 -- 172.21.15.57:0/3185607329 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f730c00cce0 con 0x7f7314140660 2023-10-19T12:57:42.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.817+0000 7f72fbfff640 1 -- 172.21.15.57:0/3185607329 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f730c005020 con 0x7f7314140660 2023-10-19T12:57:42.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.817+0000 7f731a09b640 1 -- 172.21.15.57:0/3185607329 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73141f6150 con 0x7f7314140660 2023-10-19T12:57:42.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.817+0000 7f731a09b640 1 -- 172.21.15.57:0/3185607329 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73140b0170 con 0x7f7314140660 2023-10-19T12:57:42.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.817+0000 7f72fbfff640 1 -- 172.21.15.57:0/3185607329 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f730c005430 con 0x7f7314140660 2023-10-19T12:57:42.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.818+0000 7f731a09b640 1 -- 172.21.15.57:0/3185607329 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73140b0390 con 0x7f7314140660 2023-10-19T12:57:42.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.819+0000 7f72fbfff640 1 -- 172.21.15.57:0/3185607329 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) v1 ==== 89400+0+0 (secure 0 0 0) 0x7f730c005590 con 0x7f7314140660 2023-10-19T12:57:42.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.820+0000 7f72fbfff640 1 --2- 172.21.15.57:0/3185607329 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f730006aff0 0x7f730006d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:42.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.820+0000 7f72fbfff640 1 -- 172.21.15.57:0/3185607329 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6852+0+0 (secure 0 0 0) 0x7f730c08a400 con 0x7f7314140660 2023-10-19T12:57:42.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.820+0000 7f73137fe640 1 -- 172.21.15.57:0/3185607329 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f730006aff0 msgr2=0x7f730006d4b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/922398470 2023-10-19T12:57:42.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.820+0000 7f73137fe640 1 --2- 172.21.15.57:0/3185607329 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f730006aff0 0x7f730006d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T12:57:42.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:42.824+0000 7f72fbfff640 1 -- 172.21.15.57:0/3185607329 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f730c054cb0 con 0x7f7314140660 2023-10-19T12:57:42.929 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:57:42.927+0000 7fb99e00d040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-19T12:57:42.954 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:57:42.952+0000 7fa555645040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-19T12:57:43.014 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.012+0000 7f731a09b640 1 -- 172.21.15.57:0/3185607329 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f731415b040 con 0x7f7314140660 2023-10-19T12:57:43.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.012+0000 7f72fbfff640 1 -- 172.21.15.57:0/3185607329 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 6 entries v55) v1 ==== 80+0+359 (secure 0 0 0) 0x7f730c01a080 con 0x7f7314140660 2023-10-19T12:57:43.016 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:6833/2609646285 2023-10-20T12:45:56.327729+0000 2023-10-19T12:57:43.016 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:6832/2609646285 2023-10-20T12:45:56.327729+0000 2023-10-19T12:57:43.016 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/1377585784 2023-10-20T12:45:56.327729+0000 2023-10-19T12:57:43.016 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/515590766 2023-10-20T12:45:56.327729+0000 2023-10-19T12:57:43.016 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/2223195228 2023-10-20T12:45:56.327729+0000 2023-10-19T12:57:43.017 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/2313691653 2023-10-20T12:45:56.327729+0000 2023-10-19T12:57:43.018 INFO:teuthology.orchestra.run.smithi057.stderr:listed 6 entries 2023-10-19T12:57:43.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.014+0000 7f731a09b640 1 -- 172.21.15.57:0/3185607329 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f730006aff0 msgr2=0x7f730006d4b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T12:57:43.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.014+0000 7f731a09b640 1 --2- 172.21.15.57:0/3185607329 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f730006aff0 0x7f730006d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:43.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.014+0000 7f731a09b640 1 -- 172.21.15.57:0/3185607329 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7314140660 msgr2=0x7f73141f53d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:43.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.014+0000 7f731a09b640 1 --2- 172.21.15.57:0/3185607329 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7314140660 0x7f73141f53d0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f730c00e9c0 tx=0x7f730c00ee80 comp rx=0 tx=0).stop 2023-10-19T12:57:43.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.014+0000 7f731a09b640 1 -- 172.21.15.57:0/3185607329 shutdown_connections 2023-10-19T12:57:43.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.014+0000 7f731a09b640 1 --2- 172.21.15.57:0/3185607329 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f730006aff0 0x7f730006d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:43.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.014+0000 7f731a09b640 1 --2- 172.21.15.57:0/3185607329 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7314140660 0x7f73141f53d0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:43.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.014+0000 7f731a09b640 1 --2- 172.21.15.57:0/3185607329 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f73140ca870 0x7f73140ac590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:43.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.014+0000 7f731a09b640 1 --2- 172.21.15.57:0/3185607329 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f731406d170 0x7f7314156740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:43.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.014+0000 7f731a09b640 1 -- 172.21.15.57:0/3185607329 >> 172.21.15.57:0/3185607329 conn(0x7f7314069930 msgr2=0x7f7314069d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:43.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.014+0000 7f731a09b640 1 -- 172.21.15.57:0/3185607329 shutdown_connections 2023-10-19T12:57:43.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.014+0000 7f731a09b640 1 -- 172.21.15.57:0/3185607329 wait complete. 2023-10-19T12:57:43.027 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi057.front.sepia.ceph.com... 2023-10-19T12:57:43.028 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:43.028 DEBUG:teuthology.orchestra.run.smithi057:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-10-19T12:57:43.046 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:57:43.043+0000 7fa555645040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-19T12:57:43.058 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi057.stderr:ceph-fuse[105539]: fuse finished with error 0 and tester_r 0 2023-10-19T12:57:43.060 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi057.front.sepia.ceph.com 2023-10-19T12:57:43.060 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:43.061 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-19T12:57:43.073 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:57:43.070+0000 7fb99e00d040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-19T12:57:43.089 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:57:43.087+0000 7fa555645040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-19T12:57:43.172 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:57:43.169+0000 7fb99e00d040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-19T12:57:43.192 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:57:43.189+0000 7fa555645040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-19T12:57:43.199 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-19T12:57:43.200 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T12:57:43.217 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054833,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=284 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3247632k,nr_inodes=811908,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-19T12:57:43.217 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi179.front.sepia.ceph.com 2023-10-19T12:57:43.218 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:43.218 DEBUG:teuthology.orchestra.run.smithi179:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-10-19T12:57:43.219 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:57:43.216+0000 7fb99e00d040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-19T12:57:43.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-19T12:57:43.272 INFO:teuthology.orchestra.run.smithi179.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2023-10-19T12:57:43.272 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T12:57:43.272 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T12:57:43.288 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055614,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=358 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248260k,nr_inodes=812065,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-19T12:57:43.289 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi057.front.sepia.ceph.com 2023-10-19T12:57:43.289 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:43.290 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-10-19T12:57:43.290 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:57:43.286+0000 7fa555645040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-19T12:57:43.320 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:57:43.317+0000 7fb99e00d040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-19T12:57:43.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-19T12:57:43.335 INFO:teuthology.orchestra.run.smithi057.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2023-10-19T12:57:43.336 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-19T12:57:43.336 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T12:57:43.351 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054833,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=284 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3247632k,nr_inodes=811908,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-19T12:57:43.352 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi179.front.sepia.ceph.com 2023-10-19T12:57:43.352 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:57:43.353 DEBUG:teuthology.orchestra.run.smithi179:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-10-19T12:57:43.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-19T12:57:43.406 INFO:teuthology.orchestra.run.smithi179.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2023-10-19T12:57:43.407 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T12:57:43.415 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:57:43.413+0000 7fb99e00d040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-19T12:57:43.430 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:57:43.427+0000 7fa555645040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-19T12:57:43.471 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:57:43.469+0000 7fa555645040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-19T12:57:43.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.515+0000 7fae9f256640 1 -- 172.21.15.57:0/4071842546 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fae980cf340 msgr2=0x7fae980d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:43.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.515+0000 7fae9f256640 1 --2- 172.21.15.57:0/4071842546 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fae980cf340 0x7fae980d1910 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fae9806c850 tx=0x7fae8c02bd10 comp rx=0 tx=0).stop 2023-10-19T12:57:43.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.515+0000 7fae9f256640 1 -- 172.21.15.57:0/4071842546 shutdown_connections 2023-10-19T12:57:43.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.515+0000 7fae9f256640 1 --2- 172.21.15.57:0/4071842546 >> v2:172.21.15.179:3300/0 conn(0x7fae980ceb40 0x7fae980b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:43.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.515+0000 7fae9f256640 1 --2- 172.21.15.57:0/4071842546 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fae9815b2a0 0x7fae980cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:43.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.515+0000 7fae9f256640 1 --2- 172.21.15.57:0/4071842546 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fae980cf340 0x7fae980d1910 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:43.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.515+0000 7fae9f256640 1 -- 172.21.15.57:0/4071842546 >> 172.21.15.57:0/4071842546 conn(0x7fae980d0d60 msgr2=0x7fae980d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:43.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.515+0000 7fae9f256640 1 -- 172.21.15.57:0/4071842546 shutdown_connections 2023-10-19T12:57:43.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.515+0000 7fae9f256640 1 -- 172.21.15.57:0/4071842546 wait complete. 2023-10-19T12:57:43.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.516+0000 7fae9f256640 1 Processor -- start 2023-10-19T12:57:43.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.516+0000 7fae9f256640 1 -- start start 2023-10-19T12:57:43.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.517+0000 7fae9f256640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fae980ceb40 0x7fae98157e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:43.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.517+0000 7fae9f256640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fae980cf340 0x7fae9806b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:43.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.518+0000 7fae9f256640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fae9815b2a0 0x7fae9806b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:43.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.518+0000 7fae9f256640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae98158360 con 0x7fae980ceb40 2023-10-19T12:57:43.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.518+0000 7fae9f256640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae981e5620 con 0x7fae9815b2a0 2023-10-19T12:57:43.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.518+0000 7fae9f256640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae981e5900 con 0x7fae980cf340 2023-10-19T12:57:43.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.518+0000 7fae9cff2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fae980ceb40 0x7fae98157e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:43.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.518+0000 7fae9cff2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fae980ceb40 0x7fae98157e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:53588/0 (socket says 172.21.15.57:53588) 2023-10-19T12:57:43.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.518+0000 7fae9d7f3640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fae9815b2a0 0x7fae9806b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:43.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.518+0000 7fae9cff2640 1 -- 172.21.15.57:0/830139362 learned_addr learned my addr 172.21.15.57:0/830139362 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:57:43.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.518+0000 7fae97fff640 1 --2- 172.21.15.57:0/830139362 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fae980cf340 0x7fae9806b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:43.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.518+0000 7fae9d7f3640 1 -- 172.21.15.57:0/830139362 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fae980cf340 msgr2=0x7fae9806b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:43.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.518+0000 7fae9d7f3640 1 --2- 172.21.15.57:0/830139362 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fae980cf340 0x7fae9806b160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:43.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.518+0000 7fae9d7f3640 1 -- 172.21.15.57:0/830139362 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fae980ceb40 msgr2=0x7fae98157e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:43.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.518+0000 7fae9d7f3640 1 --2- 172.21.15.57:0/830139362 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fae980ceb40 0x7fae98157e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:43.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.518+0000 7fae9d7f3640 1 -- 172.21.15.57:0/830139362 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae8c009560 con 0x7fae9815b2a0 2023-10-19T12:57:43.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.518+0000 7fae9d7f3640 1 --2- 172.21.15.57:0/830139362 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fae9815b2a0 0x7fae9806b6a0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fae9000ea80 tx=0x7fae9000ef40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:43.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.518+0000 7fae957fa640 1 -- 172.21.15.57:0/830139362 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae9000ce40 con 0x7fae9815b2a0 2023-10-19T12:57:43.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.519+0000 7fae957fa640 1 -- 172.21.15.57:0/830139362 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fae90004b20 con 0x7fae9815b2a0 2023-10-19T12:57:43.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.519+0000 7fae9f256640 1 -- 172.21.15.57:0/830139362 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae981e5be0 con 0x7fae9815b2a0 2023-10-19T12:57:43.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.519+0000 7fae957fa640 1 -- 172.21.15.57:0/830139362 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae90005020 con 0x7fae9815b2a0 2023-10-19T12:57:43.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.519+0000 7fae9f256640 1 -- 172.21.15.57:0/830139362 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae9815b770 con 0x7fae9815b2a0 2023-10-19T12:57:43.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.520+0000 7fae9f256640 1 -- 172.21.15.57:0/830139362 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae9815e800 con 0x7fae9815b2a0 2023-10-19T12:57:43.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.520+0000 7fae957fa640 1 -- 172.21.15.57:0/830139362 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 16) v1 ==== 89400+0+0 (secure 0 0 0) 0x7fae90005180 con 0x7fae9815b2a0 2023-10-19T12:57:43.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.520+0000 7fae957fa640 1 --2- 172.21.15.57:0/830139362 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fae7c06aff0 0x7fae7c06d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:43.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.520+0000 7fae957fa640 1 -- 172.21.15.57:0/830139362 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6772+0+0 (secure 0 0 0) 0x7fae90014070 con 0x7fae9815b2a0 2023-10-19T12:57:43.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.520+0000 7fae9cff2640 1 -- 172.21.15.57:0/830139362 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fae7c06aff0 msgr2=0x7fae7c06d4b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/922398470 2023-10-19T12:57:43.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.520+0000 7fae9cff2640 1 --2- 172.21.15.57:0/830139362 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fae7c06aff0 0x7fae7c06d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T12:57:43.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.525+0000 7fae957fa640 1 -- 172.21.15.57:0/830139362 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fae90055370 con 0x7fae9815b2a0 2023-10-19T12:57:43.564 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:57:43.561+0000 7fb99e00d040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-19T12:57:43.576 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:57:43.573+0000 7fa555645040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-19T12:57:43.608 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:57:43.605+0000 7fb99e00d040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-19T12:57:43.719 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:57:43.716+0000 7fb99e00d040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-19T12:57:43.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.721+0000 7fae9cff2640 1 -- 172.21.15.57:0/830139362 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fae7c06aff0 msgr2=0x7fae7c06d4b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/922398470 2023-10-19T12:57:43.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.721+0000 7fae9cff2640 1 --2- 172.21.15.57:0/830139362 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fae7c06aff0 0x7fae7c06d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-19T12:57:43.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.741+0000 7fae9f256640 1 -- 172.21.15.57:0/830139362 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fae9815b990 con 0x7fae9815b2a0 2023-10-19T12:57:43.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.742+0000 7fae957fa640 1 -- 172.21.15.57:0/830139362 <== mon.1 v2:172.21.15.179:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4916 (secure 0 0 0) 0x7fae90058990 con 0x7fae9815b2a0 2023-10-19T12:57:43.745 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:57:43.746 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5024,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/808517451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":808517451},{"type":"v1","addr":"172.21.15.179:6836","nonce":808517451}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":5625,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/3207366629","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":3207366629},{"type":"v1","addr":"172.21.15.57:6834","nonce":3207366629}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":5628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/2353910009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2353910009},{"type":"v1","addr":"172.21.15.57:6835","nonce":2353910009}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T12:45:40.706506+0000","modified":"2023-10-19T12:45:42.702983+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5634},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5634":{"gid":5634,"name":"d","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.179:6837/1237463333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":1237463333},{"type":"v1","addr":"172.21.15.179:6837","nonce":1237463333}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-19T12:57:43.746 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 16 2023-10-19T12:57:43.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.744+0000 7fae9f256640 1 -- 172.21.15.57:0/830139362 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fae7c06aff0 msgr2=0x7fae7c06d4b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T12:57:43.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.744+0000 7fae9f256640 1 --2- 172.21.15.57:0/830139362 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fae7c06aff0 0x7fae7c06d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:43.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.744+0000 7fae9f256640 1 -- 172.21.15.57:0/830139362 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fae9815b2a0 msgr2=0x7fae9806b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:43.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.744+0000 7fae9f256640 1 --2- 172.21.15.57:0/830139362 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fae9815b2a0 0x7fae9806b6a0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fae9000ea80 tx=0x7fae9000ef40 comp rx=0 tx=0).stop 2023-10-19T12:57:43.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.744+0000 7fae9f256640 1 -- 172.21.15.57:0/830139362 shutdown_connections 2023-10-19T12:57:43.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.744+0000 7fae9f256640 1 --2- 172.21.15.57:0/830139362 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fae7c06aff0 0x7fae7c06d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:43.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.744+0000 7fae9f256640 1 --2- 172.21.15.57:0/830139362 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fae9815b2a0 0x7fae9806b6a0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:43.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.744+0000 7fae9f256640 1 --2- 172.21.15.57:0/830139362 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fae980cf340 0x7fae9806b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:43.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.744+0000 7fae9f256640 1 --2- 172.21.15.57:0/830139362 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fae980ceb40 0x7fae98157e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:43.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.744+0000 7fae9f256640 1 -- 172.21.15.57:0/830139362 >> 172.21.15.57:0/830139362 conn(0x7fae980d0d60 msgr2=0x7fae9815a1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:43.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.744+0000 7fae9f256640 1 -- 172.21.15.57:0/830139362 shutdown_connections 2023-10-19T12:57:43.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.744+0000 7fae9f256640 1 -- 172.21.15.57:0/830139362 wait complete. 2023-10-19T12:57:43.758 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T12:57:43.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.904+0000 7fbae816c640 1 -- 172.21.15.57:0/2079966069 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbae00ceb40 msgr2=0x7fbae00b0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:43.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.904+0000 7fbae816c640 1 --2- 172.21.15.57:0/2079966069 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbae00ceb40 0x7fbae00b0650 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fbae008a810 tx=0x7fbad802c1d0 comp rx=0 tx=0).stop 2023-10-19T12:57:43.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.904+0000 7fbae816c640 1 -- 172.21.15.57:0/2079966069 shutdown_connections 2023-10-19T12:57:43.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.904+0000 7fbae816c640 1 --2- 172.21.15.57:0/2079966069 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbae00ceb40 0x7fbae00b0650 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:43.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.904+0000 7fbae816c640 1 --2- 172.21.15.57:0/2079966069 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbae015b2a0 0x7fbae00cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:43.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.904+0000 7fbae816c640 1 --2- 172.21.15.57:0/2079966069 >> v2:172.21.15.179:3300/0 conn(0x7fbae00cf340 0x7fbae00d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:43.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.904+0000 7fbae816c640 1 -- 172.21.15.57:0/2079966069 >> 172.21.15.57:0/2079966069 conn(0x7fbae00d0d60 msgr2=0x7fbae00d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:43.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.904+0000 7fbae816c640 1 -- 172.21.15.57:0/2079966069 shutdown_connections 2023-10-19T12:57:43.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.904+0000 7fbae816c640 1 -- 172.21.15.57:0/2079966069 wait complete. 2023-10-19T12:57:43.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.906+0000 7fbae816c640 1 Processor -- start 2023-10-19T12:57:43.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.906+0000 7fbae816c640 1 -- start start 2023-10-19T12:57:43.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.906+0000 7fbae816c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbae00ceb40 0x7fbae0159610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:43.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.906+0000 7fbae816c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbae00cf340 0x7fbae00b4c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:43.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.906+0000 7fbae816c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbae015b2a0 0x7fbae00aeeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:43.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.906+0000 7fbae816c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbae0159b50 con 0x7fbae00ceb40 2023-10-19T12:57:43.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.906+0000 7fbae816c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbae015e970 con 0x7fbae00cf340 2023-10-19T12:57:43.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.906+0000 7fbae816c640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbae015bb70 con 0x7fbae015b2a0 2023-10-19T12:57:43.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.906+0000 7fbae5f08640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbae00ceb40 0x7fbae0159610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:43.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.906+0000 7fbae5f08640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbae00ceb40 0x7fbae0159610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:53600/0 (socket says 172.21.15.57:53600) 2023-10-19T12:57:43.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.907+0000 7fbae5f08640 1 -- 172.21.15.57:0/928237918 learned_addr learned my addr 172.21.15.57:0/928237918 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:57:43.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.907+0000 7fbae6709640 1 --2- 172.21.15.57:0/928237918 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbae015b2a0 0x7fbae00aeeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:43.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.907+0000 7fbae5707640 1 --2- 172.21.15.57:0/928237918 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbae00cf340 0x7fbae00b4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:43.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.907+0000 7fbae5f08640 1 -- 172.21.15.57:0/928237918 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbae015b2a0 msgr2=0x7fbae00aeeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:43.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.907+0000 7fbae5f08640 1 --2- 172.21.15.57:0/928237918 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbae015b2a0 0x7fbae00aeeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:43.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.907+0000 7fbae5f08640 1 -- 172.21.15.57:0/928237918 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbae00cf340 msgr2=0x7fbae00b4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:43.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.907+0000 7fbae5f08640 1 --2- 172.21.15.57:0/928237918 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbae00cf340 0x7fbae00b4c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:43.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.907+0000 7fbae5f08640 1 -- 172.21.15.57:0/928237918 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbad8009900 con 0x7fbae00ceb40 2023-10-19T12:57:43.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.907+0000 7fbae5f08640 1 --2- 172.21.15.57:0/928237918 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbae00ceb40 0x7fbae0159610 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fbadc00d770 tx=0x7fbadc00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:43.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.907+0000 7fbad67fc640 1 -- 172.21.15.57:0/928237918 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbadc009c30 con 0x7fbae00ceb40 2023-10-19T12:57:43.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.907+0000 7fbad67fc640 1 -- 172.21.15.57:0/928237918 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fbadc009e80 con 0x7fbae00ceb40 2023-10-19T12:57:43.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.907+0000 7fbad67fc640 1 -- 172.21.15.57:0/928237918 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbadc0106a0 con 0x7fbae00ceb40 2023-10-19T12:57:43.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.907+0000 7fbae816c640 1 -- 172.21.15.57:0/928237918 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbae00b51d0 con 0x7fbae00ceb40 2023-10-19T12:57:43.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.907+0000 7fbae816c640 1 -- 172.21.15.57:0/928237918 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbae015e690 con 0x7fbae00ceb40 2023-10-19T12:57:43.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.908+0000 7fbae816c640 1 -- 172.21.15.57:0/928237918 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbae00d04f0 con 0x7fbae00ceb40 2023-10-19T12:57:43.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.909+0000 7fbad67fc640 1 -- 172.21.15.57:0/928237918 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) v1 ==== 89400+0+0 (secure 0 0 0) 0x7fbadc010800 con 0x7fbae00ceb40 2023-10-19T12:57:43.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.909+0000 7fbad67fc640 1 --2- 172.21.15.57:0/928237918 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fbac406b020 0x7fbac406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:43.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.910+0000 7fbae5707640 1 -- 172.21.15.57:0/928237918 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fbac406b020 msgr2=0x7fbac406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/922398470 2023-10-19T12:57:43.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.910+0000 7fbad67fc640 1 -- 172.21.15.57:0/928237918 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6772+0+0 (secure 0 0 0) 0x7fbadc08a220 con 0x7fbae00ceb40 2023-10-19T12:57:43.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.910+0000 7fbae5707640 1 --2- 172.21.15.57:0/928237918 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fbac406b020 0x7fbac406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T12:57:43.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:43.914+0000 7fbad67fc640 1 -- 172.21.15.57:0/928237918 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fbadc054c60 con 0x7fbae00ceb40 2023-10-19T12:57:44.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.093+0000 7fbae816c640 1 -- 172.21.15.57:0/928237918 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbae00690e0 con 0x7fbae00ceb40 2023-10-19T12:57:44.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.094+0000 7fbad67fc640 1 -- 172.21.15.57:0/928237918 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) v1 ==== 74+0+17875 (secure 0 0 0) 0x7fbadc058280 con 0x7fbae00ceb40 2023-10-19T12:57:44.097 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:57:44.098 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":56,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T12:57:43.126763+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2023-10-19T12:45:38.964640+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2023-10-19T12:45:39.683197+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":232,"snap_epoch":55,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.31","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.179:6833/2609646285":"2023-10-20T12:45:56.327729+0000","172.21.15.179:6832/2609646285":"2023-10-20T12:45:56.327729+0000","172.21.15.179:0/1377585784":"2023-10-20T12:45:56.327729+0000","172.21.15.179:0/515590766":"2023-10-20T12:45:56.327729+0000","172.21.15.179:0/2223195228":"2023-10-20T12:45:56.327729+0000","172.21.15.179:0/2313691653":"2023-10-20T12:45:56.327729+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":8,"length":1},{"begin":10,"length":1},{"begin":12,"length":1},{"begin":14,"length":5},{"begin":20,"length":3},{"begin":24,"length":6},{"begin":31,"length":3},{"begin":35,"length":6},{"begin":42,"length":3},{"begin":46,"length":5},{"begin":52,"length":3},{"begin":56,"length":5},{"begin":62,"length":3},{"begin":66,"length":5},{"begin":72,"length":3},{"begin":76,"length":6},{"begin":83,"length":3},{"begin":87,"length":6},{"begin":94,"length":3},{"begin":98,"length":6},{"begin":105,"length":5},{"begin":111,"length":4},{"begin":116,"length":2},{"begin":140,"length":93}]}],"new_removed_snaps":[],"new_purged_snaps":[{"pool":6,"snaps":[{"begin":2,"length":2},{"begin":11,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T12:57:44.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.096+0000 7fbae816c640 1 -- 172.21.15.57:0/928237918 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fbac406b020 msgr2=0x7fbac406d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T12:57:44.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.096+0000 7fbae816c640 1 --2- 172.21.15.57:0/928237918 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fbac406b020 0x7fbac406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:44.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.096+0000 7fbae816c640 1 -- 172.21.15.57:0/928237918 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbae00ceb40 msgr2=0x7fbae0159610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:44.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.096+0000 7fbae816c640 1 --2- 172.21.15.57:0/928237918 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbae00ceb40 0x7fbae0159610 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fbadc00d770 tx=0x7fbadc00dc30 comp rx=0 tx=0).stop 2023-10-19T12:57:44.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.096+0000 7fbae816c640 1 -- 172.21.15.57:0/928237918 shutdown_connections 2023-10-19T12:57:44.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.096+0000 7fbae816c640 1 --2- 172.21.15.57:0/928237918 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fbac406b020 0x7fbac406d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:44.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.096+0000 7fbae816c640 1 --2- 172.21.15.57:0/928237918 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbae015b2a0 0x7fbae00aeeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:44.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.096+0000 7fbae816c640 1 --2- 172.21.15.57:0/928237918 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbae00cf340 0x7fbae00b4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:44.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.096+0000 7fbae816c640 1 --2- 172.21.15.57:0/928237918 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbae00ceb40 0x7fbae0159610 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:44.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.096+0000 7fbae816c640 1 -- 172.21.15.57:0/928237918 >> 172.21.15.57:0/928237918 conn(0x7fbae00d0d60 msgr2=0x7fbae00ae050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:44.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.097+0000 7fbae816c640 1 -- 172.21.15.57:0/928237918 shutdown_connections 2023-10-19T12:57:44.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.097+0000 7fbae816c640 1 -- 172.21.15.57:0/928237918 wait complete. 2023-10-19T12:57:44.110 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-19T12:44:07.371186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-19T12:45:38.964640+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2023-10-19T12:45:39.683197+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '55', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 232, 'snap_epoch': 55, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-19T12:57:44.111 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-19T12:57:44.187 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:57:44.185+0000 7fa555645040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-19T12:57:44.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.229+0000 7fe85ce51640 1 -- 172.21.15.57:0/4127565277 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe8580cfa20 msgr2=0x7fe8580cb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:44.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.229+0000 7fe85ce51640 1 --2- 172.21.15.57:0/4127565277 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe8580cfa20 0x7fe8580cb9e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fe84c009830 tx=0x7fe84c02bcd0 comp rx=0 tx=0).stop 2023-10-19T12:57:44.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.229+0000 7fe85ce51640 1 -- 172.21.15.57:0/4127565277 shutdown_connections 2023-10-19T12:57:44.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.229+0000 7fe85ce51640 1 --2- 172.21.15.57:0/4127565277 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe8580cfa20 0x7fe8580cb9e0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:44.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.229+0000 7fe85ce51640 1 -- 172.21.15.57:0/4127565277 >> 172.21.15.57:0/4127565277 conn(0x7fe85809a030 msgr2=0x7fe8580d0d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:44.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.229+0000 7fe85ce51640 1 -- 172.21.15.57:0/4127565277 shutdown_connections 2023-10-19T12:57:44.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.229+0000 7fe85ce51640 1 -- 172.21.15.57:0/4127565277 wait complete. 2023-10-19T12:57:44.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.231+0000 7fe85ce51640 1 Processor -- start 2023-10-19T12:57:44.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.231+0000 7fe85ce51640 1 -- start start 2023-10-19T12:57:44.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.233+0000 7fe85ce51640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe8580cfa20 0x7fe8580c9970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:44.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.233+0000 7fe85ce51640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe85815b260 0x7fe8580ca0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:44.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.233+0000 7fe85ce51640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe85815b640 0x7fe8580ca630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:44.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.233+0000 7fe85ce51640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe85815e6e0 con 0x7fe85815b640 2023-10-19T12:57:44.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.233+0000 7fe85ce51640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8580c9eb0 con 0x7fe8580cfa20 2023-10-19T12:57:44.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.233+0000 7fe85ce51640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8580b58a0 con 0x7fe85815b260 2023-10-19T12:57:44.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.233+0000 7fe856d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe85815b640 0x7fe8580ca630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:44.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.233+0000 7fe855d9b640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe85815b260 0x7fe8580ca0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:44.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.233+0000 7fe856d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe85815b640 0x7fe8580ca630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:53604/0 (socket says 172.21.15.57:53604) 2023-10-19T12:57:44.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.233+0000 7fe85659c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe8580cfa20 0x7fe8580c9970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:44.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.233+0000 7fe856d9d640 1 -- 172.21.15.57:0/3740488324 learned_addr learned my addr 172.21.15.57:0/3740488324 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:57:44.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.233+0000 7fe856d9d640 1 -- 172.21.15.57:0/3740488324 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe85815b260 msgr2=0x7fe8580ca0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:44.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.233+0000 7fe856d9d640 1 --2- 172.21.15.57:0/3740488324 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe85815b260 0x7fe8580ca0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:44.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.233+0000 7fe856d9d640 1 -- 172.21.15.57:0/3740488324 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe8580cfa20 msgr2=0x7fe8580c9970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:44.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.233+0000 7fe856d9d640 1 --2- 172.21.15.57:0/3740488324 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe8580cfa20 0x7fe8580c9970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:44.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.233+0000 7fe856d9d640 1 -- 172.21.15.57:0/3740488324 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe84c009490 con 0x7fe85815b640 2023-10-19T12:57:44.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.233+0000 7fe856d9d640 1 --2- 172.21.15.57:0/3740488324 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe85815b640 0x7fe8580ca630 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fe85000c4d0 tx=0x7fe85000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:44.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.233+0000 7fe846ffd640 1 -- 172.21.15.57:0/3740488324 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe850011790 con 0x7fe85815b640 2023-10-19T12:57:44.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.233+0000 7fe846ffd640 1 -- 172.21.15.57:0/3740488324 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fe850011a00 con 0x7fe85815b640 2023-10-19T12:57:44.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.234+0000 7fe85ce51640 1 -- 172.21.15.57:0/3740488324 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8580b5b80 con 0x7fe85815b640 2023-10-19T12:57:44.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.234+0000 7fe846ffd640 1 -- 172.21.15.57:0/3740488324 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe850011e10 con 0x7fe85815b640 2023-10-19T12:57:44.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.234+0000 7fe85ce51640 1 -- 172.21.15.57:0/3740488324 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8580b6140 con 0x7fe85815b640 2023-10-19T12:57:44.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.235+0000 7fe85ce51640 1 -- 172.21.15.57:0/3740488324 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8580675f0 con 0x7fe85815b640 2023-10-19T12:57:44.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.235+0000 7fe846ffd640 1 -- 172.21.15.57:0/3740488324 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) v1 ==== 89400+0+0 (secure 0 0 0) 0x7fe85001c460 con 0x7fe85815b640 2023-10-19T12:57:44.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.235+0000 7fe846ffd640 1 --2- 172.21.15.57:0/3740488324 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fe83806aff0 0x7fe83806d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:44.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.235+0000 7fe85659c640 1 -- 172.21.15.57:0/3740488324 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fe83806aff0 msgr2=0x7fe83806d4b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/922398470 2023-10-19T12:57:44.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.235+0000 7fe846ffd640 1 -- 172.21.15.57:0/3740488324 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6756+0+0 (secure 0 0 0) 0x7fe850089ed0 con 0x7fe85815b640 2023-10-19T12:57:44.243 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.235+0000 7fe85659c640 1 --2- 172.21.15.57:0/3740488324 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fe83806aff0 0x7fe83806d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T12:57:44.243 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.239+0000 7fe846ffd640 1 -- 172.21.15.57:0/3740488324 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe850054860 con 0x7fe85815b640 2023-10-19T12:57:44.391 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:57:44.389+0000 7fb99e00d040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-19T12:57:44.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.426+0000 7fe85ce51640 1 -- 172.21.15.57:0/3740488324 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe8580afbc0 con 0x7fe85815b640 2023-10-19T12:57:44.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.426+0000 7fe846ffd640 1 -- 172.21.15.57:0/3740488324 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v57) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe850057e80 con 0x7fe85815b640 2023-10-19T12:57:44.429 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-19T12:57:44.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.428+0000 7fe85ce51640 1 -- 172.21.15.57:0/3740488324 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fe83806aff0 msgr2=0x7fe83806d4b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T12:57:44.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.428+0000 7fe85ce51640 1 --2- 172.21.15.57:0/3740488324 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fe83806aff0 0x7fe83806d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:44.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.428+0000 7fe85ce51640 1 -- 172.21.15.57:0/3740488324 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe85815b640 msgr2=0x7fe8580ca630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:44.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.428+0000 7fe85ce51640 1 --2- 172.21.15.57:0/3740488324 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe85815b640 0x7fe8580ca630 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fe85000c4d0 tx=0x7fe85000c8d0 comp rx=0 tx=0).stop 2023-10-19T12:57:44.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.428+0000 7fe85ce51640 1 -- 172.21.15.57:0/3740488324 shutdown_connections 2023-10-19T12:57:44.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.428+0000 7fe85ce51640 1 --2- 172.21.15.57:0/3740488324 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7fe83806aff0 0x7fe83806d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:44.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.428+0000 7fe85ce51640 1 --2- 172.21.15.57:0/3740488324 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe85815b640 0x7fe8580ca630 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:44.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.428+0000 7fe85ce51640 1 --2- 172.21.15.57:0/3740488324 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe85815b260 0x7fe8580ca0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:44.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.428+0000 7fe85ce51640 1 --2- 172.21.15.57:0/3740488324 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe8580cfa20 0x7fe8580c9970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:44.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.428+0000 7fe85ce51640 1 -- 172.21.15.57:0/3740488324 >> 172.21.15.57:0/3740488324 conn(0x7fe85809a030 msgr2=0x7fe8580b25b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:44.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.428+0000 7fe85ce51640 1 -- 172.21.15.57:0/3740488324 shutdown_connections 2023-10-19T12:57:44.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.428+0000 7fe85ce51640 1 -- 172.21.15.57:0/3740488324 wait complete. 2023-10-19T12:57:44.441 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-19T12:57:44.449 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:57:44.447+0000 7fa555645040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-19T12:57:44.498 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:57:44.494+0000 7fa555645040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-19T12:57:44.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.551+0000 7f9323dfc640 1 -- 172.21.15.57:0/2538055706 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f931c0cf340 msgr2=0x7f931c0d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:44.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.551+0000 7f9323dfc640 1 --2- 172.21.15.57:0/2538055706 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f931c0cf340 0x7f931c0d1910 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f9318005fd0 tx=0x7f931802be20 comp rx=0 tx=0).stop 2023-10-19T12:57:44.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.551+0000 7f9323dfc640 1 -- 172.21.15.57:0/2538055706 shutdown_connections 2023-10-19T12:57:44.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.551+0000 7f9323dfc640 1 --2- 172.21.15.57:0/2538055706 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f931c0cf340 0x7f931c0d1910 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:44.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.551+0000 7f9323dfc640 1 -- 172.21.15.57:0/2538055706 >> 172.21.15.57:0/2538055706 conn(0x7f931c0d0d60 msgr2=0x7f931c0d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:44.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.552+0000 7f9323dfc640 1 -- 172.21.15.57:0/2538055706 shutdown_connections 2023-10-19T12:57:44.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.552+0000 7f9323dfc640 1 -- 172.21.15.57:0/2538055706 wait complete. 2023-10-19T12:57:44.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.553+0000 7f9323dfc640 1 Processor -- start 2023-10-19T12:57:44.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.553+0000 7f9323dfc640 1 -- start start 2023-10-19T12:57:44.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.555+0000 7f9323dfc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f931c0ceb40 0x7f931c0b5a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:44.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.555+0000 7f9323dfc640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f931c0cf340 0x7f931c1f4d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:44.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.555+0000 7f9323dfc640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f931c15b2a0 0x7f931c09b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:44.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.555+0000 7f9323dfc640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f931c1f5550 con 0x7f931c0ceb40 2023-10-19T12:57:44.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.555+0000 7f9323dfc640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f931c1f5830 con 0x7f931c15b2a0 2023-10-19T12:57:44.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.555+0000 7f9323dfc640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f931c1f5b10 con 0x7f931c0cf340 2023-10-19T12:57:44.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.555+0000 7f9321b98640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f931c0ceb40 0x7f931c0b5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:44.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.555+0000 7f9321b98640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f931c0ceb40 0x7f931c0b5a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:53620/0 (socket says 172.21.15.57:53620) 2023-10-19T12:57:44.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.555+0000 7f9321b98640 1 -- 172.21.15.57:0/2463148068 learned_addr learned my addr 172.21.15.57:0/2463148068 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:57:44.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.555+0000 7f9322399640 1 --2- 172.21.15.57:0/2463148068 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f931c15b2a0 0x7f931c09b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:44.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.555+0000 7f9321397640 1 --2- 172.21.15.57:0/2463148068 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f931c0cf340 0x7f931c1f4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:44.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.555+0000 7f9321b98640 1 -- 172.21.15.57:0/2463148068 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f931c0cf340 msgr2=0x7f931c1f4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:44.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.555+0000 7f9321b98640 1 --2- 172.21.15.57:0/2463148068 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f931c0cf340 0x7f931c1f4d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:44.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.555+0000 7f9321b98640 1 -- 172.21.15.57:0/2463148068 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f931c15b2a0 msgr2=0x7f931c09b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:44.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.555+0000 7f9321b98640 1 --2- 172.21.15.57:0/2463148068 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f931c15b2a0 0x7f931c09b000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:44.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.555+0000 7f9321b98640 1 -- 172.21.15.57:0/2463148068 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9318009560 con 0x7f931c0ceb40 2023-10-19T12:57:44.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.555+0000 7f9321b98640 1 --2- 172.21.15.57:0/2463148068 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f931c0ceb40 0x7f931c0b5a20 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f9318006010 tx=0x7f931802ddf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:44.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.555+0000 7f93127fc640 1 -- 172.21.15.57:0/2463148068 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9318005650 con 0x7f931c0ceb40 2023-10-19T12:57:44.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.556+0000 7f93127fc640 1 -- 172.21.15.57:0/2463148068 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f93180038f0 con 0x7f931c0ceb40 2023-10-19T12:57:44.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.556+0000 7f9323dfc640 1 -- 172.21.15.57:0/2463148068 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f931c0c8e80 con 0x7f931c0ceb40 2023-10-19T12:57:44.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.556+0000 7f93127fc640 1 -- 172.21.15.57:0/2463148068 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9318003ca0 con 0x7f931c0ceb40 2023-10-19T12:57:44.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.556+0000 7f9323dfc640 1 -- 172.21.15.57:0/2463148068 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f931c1f5260 con 0x7f931c0ceb40 2023-10-19T12:57:44.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.557+0000 7f9323dfc640 1 -- 172.21.15.57:0/2463148068 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f931c0690e0 con 0x7f931c0ceb40 2023-10-19T12:57:44.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.557+0000 7f93127fc640 1 -- 172.21.15.57:0/2463148068 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) v1 ==== 89400+0+0 (secure 0 0 0) 0x7f931803e450 con 0x7f931c0ceb40 2023-10-19T12:57:44.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.558+0000 7f93127fc640 1 --2- 172.21.15.57:0/2463148068 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f930006aff0 0x7f930006d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:44.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.558+0000 7f9321397640 1 -- 172.21.15.57:0/2463148068 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f930006aff0 msgr2=0x7f930006d4b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/922398470 2023-10-19T12:57:44.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.558+0000 7f93127fc640 1 -- 172.21.15.57:0/2463148068 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6756+0+0 (secure 0 0 0) 0x7f93180ac940 con 0x7f931c0ceb40 2023-10-19T12:57:44.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.558+0000 7f9321397640 1 --2- 172.21.15.57:0/2463148068 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f930006aff0 0x7f930006d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T12:57:44.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.562+0000 7f93127fc640 1 -- 172.21.15.57:0/2463148068 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f93180775d0 con 0x7f931c0ceb40 2023-10-19T12:57:44.596 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:57:44.594+0000 7fa555645040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-19T12:57:44.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.618+0000 7f93127fc640 1 -- 172.21.15.57:0/2463148068 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mgrmap(e 17) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f93180779a0 con 0x7f931c0ceb40 2023-10-19T12:57:44.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.620+0000 7f93127fc640 1 -- 172.21.15.57:0/2463148068 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f930006aff0 msgr2=0x7f930006d4b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T12:57:44.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.620+0000 7f93127fc640 1 --2- 172.21.15.57:0/2463148068 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f930006aff0 0x7f930006d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:44.675 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:57:44.671+0000 7fa4da940640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T12:57:44.675 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:57:44.671+0000 7fa4da940640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T12:57:44.675 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:57:44.671+0000 7fa4da940640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T12:57:44.675 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:57:44.671+0000 7fa4da940640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T12:57:44.676 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:57:44.671+0000 7fa4da940640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T12:57:44.676 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:57:44.672+0000 7fa4d793a640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T12:57:44.676 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:57:44.672+0000 7fa4d793a640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T12:57:44.676 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:57:44.672+0000 7fa4d793a640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T12:57:44.676 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:57:44.672+0000 7fa4d793a640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T12:57:44.677 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:57:44.672+0000 7fa4d793a640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T12:57:44.679 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:57:44.676+0000 7fb99e00d040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-19T12:57:44.730 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:57:44.727+0000 7fb99e00d040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-19T12:57:44.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.745+0000 7f9323dfc640 1 -- 172.21.15.57:0/2463148068 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f931c0d0b40 con 0x7f931c0ceb40 2023-10-19T12:57:44.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.745+0000 7f93127fc640 1 -- 172.21.15.57:0/2463148068 <== mon.0 v2:172.21.15.57:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v58) v1 ==== 104+0+11 (secure 0 0 0) 0x7f93180a4e10 con 0x7f931c0ceb40 2023-10-19T12:57:44.748 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2023-10-19T12:57:44.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.747+0000 7f9323dfc640 1 -- 172.21.15.57:0/2463148068 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f931c0ceb40 msgr2=0x7f931c0b5a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:44.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.747+0000 7f9323dfc640 1 --2- 172.21.15.57:0/2463148068 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f931c0ceb40 0x7f931c0b5a20 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f9318006010 tx=0x7f931802ddf0 comp rx=0 tx=0).stop 2023-10-19T12:57:44.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.747+0000 7f9323dfc640 1 -- 172.21.15.57:0/2463148068 shutdown_connections 2023-10-19T12:57:44.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.747+0000 7f9323dfc640 1 --2- 172.21.15.57:0/2463148068 >> [v2:172.21.15.179:6832/922398470,v1:172.21.15.179:6833/922398470] conn(0x7f930006aff0 0x7f930006d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:44.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.747+0000 7f9323dfc640 1 --2- 172.21.15.57:0/2463148068 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f931c15b2a0 0x7f931c09b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:44.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.747+0000 7f9323dfc640 1 --2- 172.21.15.57:0/2463148068 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f931c0cf340 0x7f931c1f4d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:44.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.747+0000 7f9323dfc640 1 --2- 172.21.15.57:0/2463148068 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f931c0ceb40 0x7f931c0b5a20 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:44.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.747+0000 7f9323dfc640 1 -- 172.21.15.57:0/2463148068 >> 172.21.15.57:0/2463148068 conn(0x7f931c0d0d60 msgr2=0x7f931c15a340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:44.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.747+0000 7f9323dfc640 1 -- 172.21.15.57:0/2463148068 shutdown_connections 2023-10-19T12:57:44.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.748+0000 7f9323dfc640 1 -- 172.21.15.57:0/2463148068 wait complete. 2023-10-19T12:57:44.761 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-19T12:57:44.838 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:57:44.835+0000 7fb99e00d040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-19T12:57:44.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.871+0000 7f48766e5640 1 -- 172.21.15.57:0/1277728381 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f48700c9ed0 msgr2=0x7f4870161b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:44.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.871+0000 7f48766e5640 1 --2- 172.21.15.57:0/1277728381 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f48700c9ed0 0x7f4870161b70 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f4868005fd0 tx=0x7f486802d270 comp rx=0 tx=0).stop 2023-10-19T12:57:44.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.871+0000 7f48766e5640 1 -- 172.21.15.57:0/1277728381 shutdown_connections 2023-10-19T12:57:44.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.871+0000 7f48766e5640 1 --2- 172.21.15.57:0/1277728381 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f48700c9ed0 0x7f4870161b70 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:44.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.871+0000 7f48766e5640 1 -- 172.21.15.57:0/1277728381 >> 172.21.15.57:0/1277728381 conn(0x7f4870069930 msgr2=0x7f4870069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:44.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.871+0000 7f48766e5640 1 -- 172.21.15.57:0/1277728381 shutdown_connections 2023-10-19T12:57:44.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.871+0000 7f48766e5640 1 -- 172.21.15.57:0/1277728381 wait complete. 2023-10-19T12:57:44.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.872+0000 7f48766e5640 1 Processor -- start 2023-10-19T12:57:44.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.872+0000 7f48766e5640 1 -- start start 2023-10-19T12:57:44.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.873+0000 7f48766e5640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f48700c9ed0 0x7f4870153e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:44.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.873+0000 7f48766e5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f487013d360 0x7f4870153830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:44.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.873+0000 7f48766e5640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4870140660 0x7f487013aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:44.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.873+0000 7f48766e5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4870133e10 con 0x7f487013d360 2023-10-19T12:57:44.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.873+0000 7f48766e5640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48701340b0 con 0x7f4870140660 2023-10-19T12:57:44.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.873+0000 7f48766e5640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4870134390 con 0x7f48700c9ed0 2023-10-19T12:57:44.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.873+0000 7f486f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f487013d360 0x7f4870153830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:44.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.873+0000 7f486f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f487013d360 0x7f4870153830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:53636/0 (socket says 172.21.15.57:53636) 2023-10-19T12:57:44.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.873+0000 7f486f7fe640 1 -- 172.21.15.57:0/1008482469 learned_addr learned my addr 172.21.15.57:0/1008482469 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:57:44.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.873+0000 7f4874c82640 1 --2- 172.21.15.57:0/1008482469 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4870140660 0x7f487013aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:44.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.873+0000 7f486ffff640 1 --2- 172.21.15.57:0/1008482469 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f48700c9ed0 0x7f4870153e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:44.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.873+0000 7f486f7fe640 1 -- 172.21.15.57:0/1008482469 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f48700c9ed0 msgr2=0x7f4870153e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:44.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.873+0000 7f486f7fe640 1 --2- 172.21.15.57:0/1008482469 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f48700c9ed0 0x7f4870153e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:44.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.873+0000 7f486f7fe640 1 -- 172.21.15.57:0/1008482469 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4870140660 msgr2=0x7f487013aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:44.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.873+0000 7f486f7fe640 1 --2- 172.21.15.57:0/1008482469 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4870140660 0x7f487013aef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:44.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.873+0000 7f486f7fe640 1 -- 172.21.15.57:0/1008482469 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f486800aa10 con 0x7f487013d360 2023-10-19T12:57:44.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.873+0000 7f486f7fe640 1 --2- 172.21.15.57:0/1008482469 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f487013d360 0x7f4870153830 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f486000c400 tx=0x7f486000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:44.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.874+0000 7f486cff9640 1 -- 172.21.15.57:0/1008482469 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48600052d0 con 0x7f487013d360 2023-10-19T12:57:44.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.874+0000 7f486cff9640 1 -- 172.21.15.57:0/1008482469 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f48600035f0 con 0x7f487013d360 2023-10-19T12:57:44.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.874+0000 7f486cff9640 1 -- 172.21.15.57:0/1008482469 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4860003a00 con 0x7f487013d360 2023-10-19T12:57:44.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.874+0000 7f48766e5640 1 -- 172.21.15.57:0/1008482469 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4870134670 con 0x7f487013d360 2023-10-19T12:57:44.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.874+0000 7f48766e5640 1 -- 172.21.15.57:0/1008482469 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4870134950 con 0x7f487013d360 2023-10-19T12:57:44.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.875+0000 7f48766e5640 1 -- 172.21.15.57:0/1008482469 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48700690e0 con 0x7f487013d360 2023-10-19T12:57:44.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.875+0000 7f486cff9640 1 -- 172.21.15.57:0/1008482469 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 17) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f4860003b60 con 0x7f487013d360 2023-10-19T12:57:44.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.875+0000 7f486cff9640 1 -- 172.21.15.57:0/1008482469 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7099+0+0 (secure 0 0 0) 0x7f4860088d70 con 0x7f487013d360 2023-10-19T12:57:44.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:44.880+0000 7f486cff9640 1 -- 172.21.15.57:0/1008482469 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4860053db0 con 0x7f487013d360 2023-10-19T12:57:45.057 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.054+0000 7f48766e5640 1 -- 172.21.15.57:0/1008482469 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f487006a050 con 0x7f487013d360 2023-10-19T12:57:45.057 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.054+0000 7f486cff9640 1 -- 172.21.15.57:0/1008482469 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v58) v1 ==== 100+0+11 (secure 0 0 0) 0x7f48600170a0 con 0x7f487013d360 2023-10-19T12:57:45.058 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 64 2023-10-19T12:57:45.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.056+0000 7f48766e5640 1 -- 172.21.15.57:0/1008482469 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f487013d360 msgr2=0x7f4870153830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:45.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.056+0000 7f48766e5640 1 --2- 172.21.15.57:0/1008482469 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f487013d360 0x7f4870153830 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f486000c400 tx=0x7f486000c8c0 comp rx=0 tx=0).stop 2023-10-19T12:57:45.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.056+0000 7f48766e5640 1 -- 172.21.15.57:0/1008482469 shutdown_connections 2023-10-19T12:57:45.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.056+0000 7f48766e5640 1 --2- 172.21.15.57:0/1008482469 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4870140660 0x7f487013aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:45.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.056+0000 7f48766e5640 1 --2- 172.21.15.57:0/1008482469 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f487013d360 0x7f4870153830 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:45.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.056+0000 7f48766e5640 1 --2- 172.21.15.57:0/1008482469 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f48700c9ed0 0x7f4870153e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:45.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.056+0000 7f48766e5640 1 -- 172.21.15.57:0/1008482469 >> 172.21.15.57:0/1008482469 conn(0x7f4870069930 msgr2=0x7f487006c110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:45.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.056+0000 7f48766e5640 1 -- 172.21.15.57:0/1008482469 shutdown_connections 2023-10-19T12:57:45.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.056+0000 7f48766e5640 1 -- 172.21.15.57:0/1008482469 wait complete. 2023-10-19T12:57:45.069 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T12:57:45.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.176+0000 7fd4aec2b640 1 -- 172.21.15.57:0/2736879943 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd4a815b2a0 msgr2=0x7fd4a80cf720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:45.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.176+0000 7fd4aec2b640 1 --2- 172.21.15.57:0/2736879943 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd4a815b2a0 0x7fd4a80cf720 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fd4a808a810 tx=0x7fd49402bdb0 comp rx=0 tx=0).stop 2023-10-19T12:57:45.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.176+0000 7fd4aec2b640 1 -- 172.21.15.57:0/2736879943 shutdown_connections 2023-10-19T12:57:45.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.176+0000 7fd4aec2b640 1 --2- 172.21.15.57:0/2736879943 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd4a815b2a0 0x7fd4a80cf720 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:45.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.176+0000 7fd4aec2b640 1 --2- 172.21.15.57:0/2736879943 >> v2:172.21.15.179:3300/0 conn(0x7fd4a80cf340 0x7fd4a80d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:45.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.176+0000 7fd4aec2b640 1 -- 172.21.15.57:0/2736879943 >> 172.21.15.57:0/2736879943 conn(0x7fd4a80d0d60 msgr2=0x7fd4a80d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:45.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.176+0000 7fd4aec2b640 1 -- 172.21.15.57:0/2736879943 shutdown_connections 2023-10-19T12:57:45.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.176+0000 7fd4aec2b640 1 -- 172.21.15.57:0/2736879943 wait complete. 2023-10-19T12:57:45.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.177+0000 7fd4aec2b640 1 Processor -- start 2023-10-19T12:57:45.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.177+0000 7fd4aec2b640 1 -- start start 2023-10-19T12:57:45.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.178+0000 7fd4aec2b640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd4a80ceb40 0x7fd4a815e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:45.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.178+0000 7fd4aec2b640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd4a80cf340 0x7fd4a81e4760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:45.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.178+0000 7fd4aec2b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd4a815b2a0 0x7fd4a80b4a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:45.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.178+0000 7fd4aec2b640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4a81e5d80 con 0x7fd4a815b2a0 2023-10-19T12:57:45.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.178+0000 7fd4aec2b640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4a815e910 con 0x7fd4a80ceb40 2023-10-19T12:57:45.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.178+0000 7fd4aec2b640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4a81e5170 con 0x7fd4a80cf340 2023-10-19T12:57:45.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.178+0000 7fd4ad1c8640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd4a815b2a0 0x7fd4a80b4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:45.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.178+0000 7fd4ad1c8640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd4a815b2a0 0x7fd4a80b4a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:53642/0 (socket says 172.21.15.57:53642) 2023-10-19T12:57:45.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.178+0000 7fd4ad1c8640 1 -- 172.21.15.57:0/1874373567 learned_addr learned my addr 172.21.15.57:0/1874373567 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:57:45.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.178+0000 7fd4ac9c7640 1 --2- 172.21.15.57:0/1874373567 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd4a80ceb40 0x7fd4a815e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:45.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.178+0000 7fd4a7fff640 1 --2- 172.21.15.57:0/1874373567 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd4a80cf340 0x7fd4a81e4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:45.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.178+0000 7fd4ad1c8640 1 -- 172.21.15.57:0/1874373567 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd4a80cf340 msgr2=0x7fd4a81e4760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:45.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.178+0000 7fd4ad1c8640 1 --2- 172.21.15.57:0/1874373567 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd4a80cf340 0x7fd4a81e4760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:45.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.178+0000 7fd4ad1c8640 1 -- 172.21.15.57:0/1874373567 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd4a80ceb40 msgr2=0x7fd4a815e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:45.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.178+0000 7fd4ad1c8640 1 --2- 172.21.15.57:0/1874373567 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd4a80ceb40 0x7fd4a815e3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:45.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.178+0000 7fd4ad1c8640 1 -- 172.21.15.57:0/1874373567 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd494009560 con 0x7fd4a815b2a0 2023-10-19T12:57:45.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.179+0000 7fd4ad1c8640 1 --2- 172.21.15.57:0/1874373567 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd4a815b2a0 0x7fd4a80b4a30 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fd4a000c730 tx=0x7fd4a000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:45.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.179+0000 7fd4a57fa640 1 -- 172.21.15.57:0/1874373567 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4a0003b90 con 0x7fd4a815b2a0 2023-10-19T12:57:45.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.179+0000 7fd4a57fa640 1 -- 172.21.15.57:0/1874373567 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fd4a0003d10 con 0x7fd4a815b2a0 2023-10-19T12:57:45.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.179+0000 7fd4aec2b640 1 -- 172.21.15.57:0/1874373567 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4a81e5450 con 0x7fd4a815b2a0 2023-10-19T12:57:45.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.179+0000 7fd4a57fa640 1 -- 172.21.15.57:0/1874373567 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4a000aab0 con 0x7fd4a815b2a0 2023-10-19T12:57:45.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.179+0000 7fd4aec2b640 1 -- 172.21.15.57:0/1874373567 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4a81e5a10 con 0x7fd4a815b2a0 2023-10-19T12:57:45.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.180+0000 7fd4aec2b640 1 -- 172.21.15.57:0/1874373567 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4a81e5ef0 con 0x7fd4a815b2a0 2023-10-19T12:57:45.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.180+0000 7fd4a57fa640 1 -- 172.21.15.57:0/1874373567 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 17) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fd4a000ac10 con 0x7fd4a815b2a0 2023-10-19T12:57:45.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.181+0000 7fd4a57fa640 1 -- 172.21.15.57:0/1874373567 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7099+0+0 (secure 0 0 0) 0x7fd4a0089570 con 0x7fd4a815b2a0 2023-10-19T12:57:45.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.185+0000 7fd4a57fa640 1 -- 172.21.15.57:0/1874373567 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd4a00545b0 con 0x7fd4a815b2a0 2023-10-19T12:57:45.398 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.396+0000 7fd4aec2b640 1 -- 172.21.15.57:0/1874373567 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd4a80cb9d0 con 0x7fd4a815b2a0 2023-10-19T12:57:45.399 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.396+0000 7fd4a57fa640 1 -- 172.21.15.57:0/1874373567 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4916 (secure 0 0 0) 0x7fd4a0057bd0 con 0x7fd4a815b2a0 2023-10-19T12:57:45.400 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:57:45.400 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5024,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/808517451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":808517451},{"type":"v1","addr":"172.21.15.179:6836","nonce":808517451}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":5625,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/3207366629","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":3207366629},{"type":"v1","addr":"172.21.15.57:6834","nonce":3207366629}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":5628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/2353910009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2353910009},{"type":"v1","addr":"172.21.15.57:6835","nonce":2353910009}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T12:45:40.706506+0000","modified":"2023-10-19T12:45:42.702983+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5634},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5634":{"gid":5634,"name":"d","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.179:6837/1237463333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":1237463333},{"type":"v1","addr":"172.21.15.179:6837","nonce":1237463333}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-19T12:57:45.400 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 16 2023-10-19T12:57:45.401 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.398+0000 7fd4aec2b640 1 -- 172.21.15.57:0/1874373567 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd4a815b2a0 msgr2=0x7fd4a80b4a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:45.401 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.398+0000 7fd4aec2b640 1 --2- 172.21.15.57:0/1874373567 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd4a815b2a0 0x7fd4a80b4a30 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fd4a000c730 tx=0x7fd4a000cbf0 comp rx=0 tx=0).stop 2023-10-19T12:57:45.401 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.398+0000 7fd4aec2b640 1 -- 172.21.15.57:0/1874373567 shutdown_connections 2023-10-19T12:57:45.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.398+0000 7fd4aec2b640 1 --2- 172.21.15.57:0/1874373567 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd4a815b2a0 0x7fd4a80b4a30 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:45.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.398+0000 7fd4aec2b640 1 --2- 172.21.15.57:0/1874373567 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd4a80cf340 0x7fd4a81e4760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:45.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.398+0000 7fd4aec2b640 1 --2- 172.21.15.57:0/1874373567 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd4a80ceb40 0x7fd4a815e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:45.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.398+0000 7fd4aec2b640 1 -- 172.21.15.57:0/1874373567 >> 172.21.15.57:0/1874373567 conn(0x7fd4a80d0d60 msgr2=0x7fd4a815a210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:45.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.398+0000 7fd4aec2b640 1 -- 172.21.15.57:0/1874373567 shutdown_connections 2023-10-19T12:57:45.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.398+0000 7fd4aec2b640 1 -- 172.21.15.57:0/1874373567 wait complete. 2023-10-19T12:57:45.412 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T12:57:45.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.519+0000 7fea23b7d640 1 -- 172.21.15.57:0/4165377772 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fea1c0cf340 msgr2=0x7fea1c0d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:45.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.519+0000 7fea23b7d640 1 --2- 172.21.15.57:0/4165377772 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fea1c0cf340 0x7fea1c0d1910 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fea18005fd0 tx=0x7fea1802bd90 comp rx=0 tx=0).stop 2023-10-19T12:57:45.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.519+0000 7fea23b7d640 1 -- 172.21.15.57:0/4165377772 shutdown_connections 2023-10-19T12:57:45.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.519+0000 7fea23b7d640 1 --2- 172.21.15.57:0/4165377772 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fea1c15b2a0 0x7fea1c0cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:45.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.519+0000 7fea23b7d640 1 --2- 172.21.15.57:0/4165377772 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fea1c0cf340 0x7fea1c0d1910 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:45.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.519+0000 7fea23b7d640 1 -- 172.21.15.57:0/4165377772 >> 172.21.15.57:0/4165377772 conn(0x7fea1c0d0d60 msgr2=0x7fea1c0d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:45.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.519+0000 7fea23b7d640 1 -- 172.21.15.57:0/4165377772 shutdown_connections 2023-10-19T12:57:45.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.519+0000 7fea23b7d640 1 -- 172.21.15.57:0/4165377772 wait complete. 2023-10-19T12:57:45.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.520+0000 7fea23b7d640 1 Processor -- start 2023-10-19T12:57:45.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.520+0000 7fea23b7d640 1 -- start start 2023-10-19T12:57:45.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.521+0000 7fea23b7d640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fea1c0ceb40 0x7fea1c0b6820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:45.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.521+0000 7fea23b7d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fea1c0cf340 0x7fea1c15b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:45.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.521+0000 7fea23b7d640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fea1c15b2a0 0x7fea1c157fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:45.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.521+0000 7fea23b7d640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea1c15bc10 con 0x7fea1c0cf340 2023-10-19T12:57:45.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.521+0000 7fea23b7d640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea1c15be90 con 0x7fea1c0ceb40 2023-10-19T12:57:45.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.521+0000 7fea23b7d640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea1c15c170 con 0x7fea1c15b2a0 2023-10-19T12:57:45.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.521+0000 7fea21118640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fea1c0cf340 0x7fea1c15b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:45.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.521+0000 7fea21118640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fea1c0cf340 0x7fea1c15b6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:53658/0 (socket says 172.21.15.57:53658) 2023-10-19T12:57:45.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.521+0000 7fea21118640 1 -- 172.21.15.57:0/1808655141 learned_addr learned my addr 172.21.15.57:0/1808655141 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:57:45.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.521+0000 7fea21919640 1 --2- 172.21.15.57:0/1808655141 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fea1c0ceb40 0x7fea1c0b6820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:45.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.521+0000 7fea2211a640 1 --2- 172.21.15.57:0/1808655141 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fea1c15b2a0 0x7fea1c157fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:45.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.521+0000 7fea21118640 1 -- 172.21.15.57:0/1808655141 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fea1c15b2a0 msgr2=0x7fea1c157fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:45.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.521+0000 7fea21118640 1 --2- 172.21.15.57:0/1808655141 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fea1c15b2a0 0x7fea1c157fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:45.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.521+0000 7fea21118640 1 -- 172.21.15.57:0/1808655141 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fea1c0ceb40 msgr2=0x7fea1c0b6820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:45.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.521+0000 7fea21118640 1 --2- 172.21.15.57:0/1808655141 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fea1c0ceb40 0x7fea1c0b6820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:45.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.521+0000 7fea21118640 1 -- 172.21.15.57:0/1808655141 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea18009560 con 0x7fea1c0cf340 2023-10-19T12:57:45.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.521+0000 7fea21118640 1 --2- 172.21.15.57:0/1808655141 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fea1c0cf340 0x7fea1c15b6d0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fea0c00e790 tx=0x7fea0c00ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:45.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.521+0000 7fea127fc640 1 -- 172.21.15.57:0/1808655141 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea0c009580 con 0x7fea1c0cf340 2023-10-19T12:57:45.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.521+0000 7fea127fc640 1 -- 172.21.15.57:0/1808655141 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fea0c00cdb0 con 0x7fea1c0cf340 2023-10-19T12:57:45.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.521+0000 7fea23b7d640 1 -- 172.21.15.57:0/1808655141 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea1c15c450 con 0x7fea1c0cf340 2023-10-19T12:57:45.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.521+0000 7fea127fc640 1 -- 172.21.15.57:0/1808655141 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea0c010690 con 0x7fea1c0cf340 2023-10-19T12:57:45.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.522+0000 7fea23b7d640 1 -- 172.21.15.57:0/1808655141 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea1c15ca10 con 0x7fea1c0cf340 2023-10-19T12:57:45.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.522+0000 7fea23b7d640 1 -- 172.21.15.57:0/1808655141 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea1c0690e0 con 0x7fea1c0cf340 2023-10-19T12:57:45.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.523+0000 7fea127fc640 1 -- 172.21.15.57:0/1808655141 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 17) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fea0c0107f0 con 0x7fea1c0cf340 2023-10-19T12:57:45.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.523+0000 7fea127fc640 1 -- 172.21.15.57:0/1808655141 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7099+0+0 (secure 0 0 0) 0x7fea0c014070 con 0x7fea1c0cf340 2023-10-19T12:57:45.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.527+0000 7fea127fc640 1 -- 172.21.15.57:0/1808655141 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fea0c051ca0 con 0x7fea1c0cf340 2023-10-19T12:57:45.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.618+0000 7fea127fc640 1 -- 172.21.15.57:0/1808655141 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mgrmap(e 18) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fea0c054b40 con 0x7fea1c0cf340 2023-10-19T12:57:45.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.618+0000 7fea127fc640 1 --2- 172.21.15.57:0/1808655141 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7fea000710a0 0x7fea00073490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:45.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.619+0000 7fea21919640 1 --2- 172.21.15.57:0/1808655141 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7fea000710a0 0x7fea00073490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:45.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.624+0000 7fea21919640 1 --2- 172.21.15.57:0/1808655141 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7fea000710a0 0x7fea00073490 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fea18035000 tx=0x7fea18004730 comp rx=0 tx=0).ready entity=mgr.5809 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:45.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.710+0000 7fea23b7d640 1 -- 172.21.15.57:0/1808655141 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fea1c0d0b40 con 0x7fea1c0cf340 2023-10-19T12:57:45.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.711+0000 7fea127fc640 1 -- 172.21.15.57:0/1808655141 <== mon.0 v2:172.21.15.57:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) v1 ==== 74+0+18354 (secure 0 0 0) 0x7fea0c087bc0 con 0x7fea1c0cf340 2023-10-19T12:57:45.715 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:57:45.715 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":58,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T12:57:44.608979+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2023-10-19T12:45:38.964640+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2023-10-19T12:45:39.683197+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":233,"snap_epoch":57,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.31","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.179:0/2997987784":"2023-10-20T12:57:44.608940+0000","172.21.15.179:0/2313691653":"2023-10-20T12:45:56.327729+0000","172.21.15.179:0/2223195228":"2023-10-20T12:45:56.327729+0000","172.21.15.179:0/515590766":"2023-10-20T12:45:56.327729+0000","172.21.15.179:0/4235058089":"2023-10-20T12:57:44.608940+0000","172.21.15.179:0/1377585784":"2023-10-20T12:45:56.327729+0000","172.21.15.179:0/3471628125":"2023-10-20T12:57:44.608940+0000","172.21.15.179:6832/2609646285":"2023-10-20T12:45:56.327729+0000","172.21.15.179:6833/2609646285":"2023-10-20T12:45:56.327729+0000","172.21.15.179:6832/922398470":"2023-10-20T12:57:44.608940+0000","172.21.15.179:6833/922398470":"2023-10-20T12:57:44.608940+0000","172.21.15.179:0/2754375794":"2023-10-20T12:57:44.608940+0000","172.21.15.179:0/161411072":"2023-10-20T12:57:44.608940+0000","172.21.15.179:0/4022258517":"2023-10-20T12:57:44.608940+0000","172.21.15.179:0/3663257206":"2023-10-20T12:57:44.608940+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":8,"length":1},{"begin":10,"length":1},{"begin":12,"length":1},{"begin":14,"length":5},{"begin":20,"length":3},{"begin":24,"length":6},{"begin":31,"length":3},{"begin":35,"length":6},{"begin":42,"length":3},{"begin":46,"length":5},{"begin":52,"length":3},{"begin":56,"length":5},{"begin":62,"length":3},{"begin":66,"length":5},{"begin":72,"length":3},{"begin":76,"length":6},{"begin":83,"length":3},{"begin":87,"length":6},{"begin":94,"length":3},{"begin":98,"length":6},{"begin":105,"length":10},{"begin":116,"length":2},{"begin":140,"length":94}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T12:57:45.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.713+0000 7fea23b7d640 1 -- 172.21.15.57:0/1808655141 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7fea000710a0 msgr2=0x7fea00073490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:45.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.713+0000 7fea23b7d640 1 --2- 172.21.15.57:0/1808655141 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7fea000710a0 0x7fea00073490 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fea18035000 tx=0x7fea18004730 comp rx=0 tx=0).stop 2023-10-19T12:57:45.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.713+0000 7fea23b7d640 1 -- 172.21.15.57:0/1808655141 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fea1c0cf340 msgr2=0x7fea1c15b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:45.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.713+0000 7fea23b7d640 1 --2- 172.21.15.57:0/1808655141 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fea1c0cf340 0x7fea1c15b6d0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fea0c00e790 tx=0x7fea0c00ec50 comp rx=0 tx=0).stop 2023-10-19T12:57:45.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.713+0000 7fea23b7d640 1 -- 172.21.15.57:0/1808655141 shutdown_connections 2023-10-19T12:57:45.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.713+0000 7fea23b7d640 1 --2- 172.21.15.57:0/1808655141 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7fea000710a0 0x7fea00073490 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:45.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.713+0000 7fea23b7d640 1 --2- 172.21.15.57:0/1808655141 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fea1c15b2a0 0x7fea1c157fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:45.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.713+0000 7fea23b7d640 1 --2- 172.21.15.57:0/1808655141 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fea1c0cf340 0x7fea1c15b6d0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:45.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.713+0000 7fea23b7d640 1 --2- 172.21.15.57:0/1808655141 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fea1c0ceb40 0x7fea1c0b6820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:45.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.713+0000 7fea23b7d640 1 -- 172.21.15.57:0/1808655141 >> 172.21.15.57:0/1808655141 conn(0x7fea1c0d0d60 msgr2=0x7fea1c15a0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:45.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.713+0000 7fea23b7d640 1 -- 172.21.15.57:0/1808655141 shutdown_connections 2023-10-19T12:57:45.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.713+0000 7fea23b7d640 1 -- 172.21.15.57:0/1808655141 wait complete. 2023-10-19T12:57:45.727 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-19T12:57:45.727 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T12:57:45.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.833+0000 7f49910da640 1 -- 172.21.15.57:0/1323233722 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f498c0ceb40 msgr2=0x7f498c0b0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:45.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.833+0000 7f49910da640 1 --2- 172.21.15.57:0/1323233722 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f498c0ceb40 0x7f498c0b0650 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f497c005fd0 tx=0x7f497c02c050 comp rx=0 tx=0).stop 2023-10-19T12:57:45.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.833+0000 7f49910da640 1 -- 172.21.15.57:0/1323233722 shutdown_connections 2023-10-19T12:57:45.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.833+0000 7f49910da640 1 --2- 172.21.15.57:0/1323233722 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f498c0ceb40 0x7f498c0b0650 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:45.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.833+0000 7f49910da640 1 --2- 172.21.15.57:0/1323233722 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f498c15b2a0 0x7f498c0d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:45.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.833+0000 7f49910da640 1 -- 172.21.15.57:0/1323233722 >> 172.21.15.57:0/1323233722 conn(0x7f498c0d0d60 msgr2=0x7f498c0d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:45.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.833+0000 7f49910da640 1 -- 172.21.15.57:0/1323233722 shutdown_connections 2023-10-19T12:57:45.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.833+0000 7f49910da640 1 -- 172.21.15.57:0/1323233722 wait complete. 2023-10-19T12:57:45.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.835+0000 7f49910da640 1 Processor -- start 2023-10-19T12:57:45.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.835+0000 7f49910da640 1 -- start start 2023-10-19T12:57:45.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.836+0000 7f49910da640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f498c0ceb40 0x7f498c14c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:45.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.836+0000 7f49910da640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f498c0cf340 0x7f498c14a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:45.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.836+0000 7f49910da640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f498c15b2a0 0x7f498c14ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:45.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.836+0000 7f49910da640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f498c14b650 con 0x7f498c0cf340 2023-10-19T12:57:45.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.836+0000 7f49910da640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f498c14b930 con 0x7f498c0ceb40 2023-10-19T12:57:45.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.836+0000 7f49910da640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f498c14bc10 con 0x7f498c15b2a0 2023-10-19T12:57:45.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.836+0000 7f498a59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f498c0cf340 0x7f498c14a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:45.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.836+0000 7f498a59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f498c0cf340 0x7f498c14a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:53666/0 (socket says 172.21.15.57:53666) 2023-10-19T12:57:45.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.836+0000 7f498a59c640 1 -- 172.21.15.57:0/633700534 learned_addr learned my addr 172.21.15.57:0/633700534 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:57:45.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.836+0000 7f498ad9d640 1 --2- 172.21.15.57:0/633700534 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f498c0ceb40 0x7f498c14c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:45.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.836+0000 7f498b59e640 1 --2- 172.21.15.57:0/633700534 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f498c15b2a0 0x7f498c14ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:45.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.836+0000 7f498a59c640 1 -- 172.21.15.57:0/633700534 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f498c15b2a0 msgr2=0x7f498c14ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:45.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.836+0000 7f498a59c640 1 --2- 172.21.15.57:0/633700534 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f498c15b2a0 0x7f498c14ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:45.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.836+0000 7f498a59c640 1 -- 172.21.15.57:0/633700534 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f498c0ceb40 msgr2=0x7f498c14c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:45.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.836+0000 7f498a59c640 1 --2- 172.21.15.57:0/633700534 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f498c0ceb40 0x7f498c14c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:45.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.836+0000 7f498a59c640 1 -- 172.21.15.57:0/633700534 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4974009560 con 0x7f498c0cf340 2023-10-19T12:57:45.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.836+0000 7f498a59c640 1 --2- 172.21.15.57:0/633700534 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f498c0cf340 0x7f498c14a540 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f497400e9c0 tx=0x7f497400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:45.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.836+0000 7f49837fe640 1 -- 172.21.15.57:0/633700534 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f497400cd80 con 0x7f498c0cf340 2023-10-19T12:57:45.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.836+0000 7f49837fe640 1 -- 172.21.15.57:0/633700534 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f4974004020 con 0x7f498c0cf340 2023-10-19T12:57:45.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.837+0000 7f49910da640 1 -- 172.21.15.57:0/633700534 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f497c009840 con 0x7f498c0cf340 2023-10-19T12:57:45.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.837+0000 7f49837fe640 1 -- 172.21.15.57:0/633700534 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4974004410 con 0x7f498c0cf340 2023-10-19T12:57:45.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.837+0000 7f49910da640 1 -- 172.21.15.57:0/633700534 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f498c0b0d50 con 0x7f498c0cf340 2023-10-19T12:57:45.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.838+0000 7f49910da640 1 -- 172.21.15.57:0/633700534 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f498c0690e0 con 0x7f498c0cf340 2023-10-19T12:57:45.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.838+0000 7f49837fe640 1 -- 172.21.15.57:0/633700534 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 18) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f49740035f0 con 0x7f498c0cf340 2023-10-19T12:57:45.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.838+0000 7f49837fe640 1 --2- 172.21.15.57:0/633700534 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f496c06ac90 0x7f496c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:45.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.839+0000 7f49837fe640 1 -- 172.21.15.57:0/633700534 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7099+0+0 (secure 0 0 0) 0x7f4974014070 con 0x7f498c0cf340 2023-10-19T12:57:45.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.839+0000 7f498ad9d640 1 --2- 172.21.15.57:0/633700534 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f496c06ac90 0x7f496c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:45.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.839+0000 7f498ad9d640 1 --2- 172.21.15.57:0/633700534 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f496c06ac90 0x7f496c06d150 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f4984001780 tx=0x7f4984009040 comp rx=0 tx=0).ready entity=mgr.5809 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:45.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:45.843+0000 7f49837fe640 1 -- 172.21.15.57:0/633700534 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4974055080 con 0x7f498c0cf340 2023-10-19T12:57:46.056 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.053+0000 7f49910da640 1 -- 172.21.15.57:0/633700534 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f498c09ea10 con 0x7f498c0cf340 2023-10-19T12:57:46.056 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.054+0000 7f49837fe640 1 -- 172.21.15.57:0/633700534 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4916 (secure 0 0 0) 0x7f49740586a0 con 0x7f498c0cf340 2023-10-19T12:57:46.057 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:57:46.057 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5024,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/808517451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":808517451},{"type":"v1","addr":"172.21.15.179:6836","nonce":808517451}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":5625,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/3207366629","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":3207366629},{"type":"v1","addr":"172.21.15.57:6834","nonce":3207366629}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":5628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/2353910009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2353910009},{"type":"v1","addr":"172.21.15.57:6835","nonce":2353910009}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T12:45:40.706506+0000","modified":"2023-10-19T12:45:42.702983+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5634},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5634":{"gid":5634,"name":"d","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.179:6837/1237463333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":1237463333},{"type":"v1","addr":"172.21.15.179:6837","nonce":1237463333}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-19T12:57:46.058 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 16 2023-10-19T12:57:46.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.055+0000 7f49910da640 1 -- 172.21.15.57:0/633700534 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f496c06ac90 msgr2=0x7f496c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:46.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.055+0000 7f49910da640 1 --2- 172.21.15.57:0/633700534 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f496c06ac90 0x7f496c06d150 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f4984001780 tx=0x7f4984009040 comp rx=0 tx=0).stop 2023-10-19T12:57:46.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.055+0000 7f49910da640 1 -- 172.21.15.57:0/633700534 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f498c0cf340 msgr2=0x7f498c14a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:46.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.055+0000 7f49910da640 1 --2- 172.21.15.57:0/633700534 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f498c0cf340 0x7f498c14a540 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f497400e9c0 tx=0x7f497400ee80 comp rx=0 tx=0).stop 2023-10-19T12:57:46.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.055+0000 7f49910da640 1 -- 172.21.15.57:0/633700534 shutdown_connections 2023-10-19T12:57:46.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.055+0000 7f49910da640 1 --2- 172.21.15.57:0/633700534 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f496c06ac90 0x7f496c06d150 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:46.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.055+0000 7f49910da640 1 --2- 172.21.15.57:0/633700534 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f498c15b2a0 0x7f498c14ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:46.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.055+0000 7f49910da640 1 --2- 172.21.15.57:0/633700534 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f498c0cf340 0x7f498c14a540 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:46.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.056+0000 7f49910da640 1 --2- 172.21.15.57:0/633700534 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f498c0ceb40 0x7f498c14c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:46.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.056+0000 7f49910da640 1 -- 172.21.15.57:0/633700534 >> 172.21.15.57:0/633700534 conn(0x7f498c0d0d60 msgr2=0x7f498c06b240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:46.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.056+0000 7f49910da640 1 -- 172.21.15.57:0/633700534 shutdown_connections 2023-10-19T12:57:46.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.056+0000 7f49910da640 1 -- 172.21.15.57:0/633700534 wait complete. 2023-10-19T12:57:46.069 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T12:57:46.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.177+0000 7fa70f1d7640 1 -- 172.21.15.57:0/2032599359 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa70815c1f0 msgr2=0x7fa708157dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:46.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.177+0000 7fa70f1d7640 1 --2- 172.21.15.57:0/2032599359 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa70815c1f0 0x7fa708157dd0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fa700005fd0 tx=0x7fa70002c2e0 comp rx=0 tx=0).stop 2023-10-19T12:57:46.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.177+0000 7fa70f1d7640 1 -- 172.21.15.57:0/2032599359 shutdown_connections 2023-10-19T12:57:46.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.177+0000 7fa70f1d7640 1 --2- 172.21.15.57:0/2032599359 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa70815c1f0 0x7fa708157dd0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:46.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.177+0000 7fa70f1d7640 1 --2- 172.21.15.57:0/2032599359 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa70806ae40 0x7fa7080b0e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:46.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.177+0000 7fa70f1d7640 1 -- 172.21.15.57:0/2032599359 >> 172.21.15.57:0/2032599359 conn(0x7fa70806fae0 msgr2=0x7fa70806fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:46.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.177+0000 7fa70f1d7640 1 -- 172.21.15.57:0/2032599359 shutdown_connections 2023-10-19T12:57:46.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.177+0000 7fa70f1d7640 1 -- 172.21.15.57:0/2032599359 wait complete. 2023-10-19T12:57:46.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.179+0000 7fa70f1d7640 1 Processor -- start 2023-10-19T12:57:46.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.179+0000 7fa70f1d7640 1 -- start start 2023-10-19T12:57:46.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.179+0000 7fa70f1d7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa70806ae40 0x7fa7080c9ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:46.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.179+0000 7fa70f1d7640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa7081579f0 0x7fa7081f0f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:46.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.179+0000 7fa70f1d7640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa70815c1f0 0x7fa7080ada30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:46.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.179+0000 7fa70f1d7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7080c2520 con 0x7fa70806ae40 2023-10-19T12:57:46.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.179+0000 7fa70f1d7640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7080c27c0 con 0x7fa70815c1f0 2023-10-19T12:57:46.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.179+0000 7fa70f1d7640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7080c2aa0 con 0x7fa7081579f0 2023-10-19T12:57:46.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.180+0000 7fa70cf73640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa70806ae40 0x7fa7080c9ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:46.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.180+0000 7fa70cf73640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa70806ae40 0x7fa7080c9ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:53674/0 (socket says 172.21.15.57:53674) 2023-10-19T12:57:46.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.180+0000 7fa70cf73640 1 -- 172.21.15.57:0/465001471 learned_addr learned my addr 172.21.15.57:0/465001471 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:57:46.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.180+0000 7fa70d774640 1 --2- 172.21.15.57:0/465001471 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa70815c1f0 0x7fa7080ada30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:46.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.180+0000 7fa707fff640 1 --2- 172.21.15.57:0/465001471 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa7081579f0 0x7fa7081f0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:46.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.180+0000 7fa70cf73640 1 -- 172.21.15.57:0/465001471 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa7081579f0 msgr2=0x7fa7081f0f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:46.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.180+0000 7fa70cf73640 1 --2- 172.21.15.57:0/465001471 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa7081579f0 0x7fa7081f0f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:46.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.180+0000 7fa70cf73640 1 -- 172.21.15.57:0/465001471 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa70815c1f0 msgr2=0x7fa7080ada30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:46.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.180+0000 7fa70cf73640 1 --2- 172.21.15.57:0/465001471 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa70815c1f0 0x7fa7080ada30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:46.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.180+0000 7fa70cf73640 1 -- 172.21.15.57:0/465001471 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa700009900 con 0x7fa70806ae40 2023-10-19T12:57:46.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.180+0000 7fa70d774640 1 --2- 172.21.15.57:0/465001471 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa70815c1f0 0x7fa7080ada30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:57:46.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.180+0000 7fa707fff640 1 --2- 172.21.15.57:0/465001471 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa7081579f0 0x7fa7081f0f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:57:46.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.180+0000 7fa70cf73640 1 --2- 172.21.15.57:0/465001471 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa70806ae40 0x7fa7080c9ce0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fa6fc00c450 tx=0x7fa6fc00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:46.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.180+0000 7fa7057fa640 1 -- 172.21.15.57:0/465001471 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6fc0052d0 con 0x7fa70806ae40 2023-10-19T12:57:46.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.180+0000 7fa7057fa640 1 -- 172.21.15.57:0/465001471 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fa6fc005430 con 0x7fa70806ae40 2023-10-19T12:57:46.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.180+0000 7fa70f1d7640 1 -- 172.21.15.57:0/465001471 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7080b2330 con 0x7fa70806ae40 2023-10-19T12:57:46.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.180+0000 7fa7057fa640 1 -- 172.21.15.57:0/465001471 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6fc012890 con 0x7fa70806ae40 2023-10-19T12:57:46.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.180+0000 7fa70f1d7640 1 -- 172.21.15.57:0/465001471 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7080c3060 con 0x7fa70806ae40 2023-10-19T12:57:46.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.181+0000 7fa70f1d7640 1 -- 172.21.15.57:0/465001471 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7080690e0 con 0x7fa70806ae40 2023-10-19T12:57:46.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.182+0000 7fa7057fa640 1 -- 172.21.15.57:0/465001471 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 18) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fa6fc0035f0 con 0x7fa70806ae40 2023-10-19T12:57:46.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.182+0000 7fa7057fa640 1 --2- 172.21.15.57:0/465001471 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7fa6ec06ac90 0x7fa6ec06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:46.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.182+0000 7fa7057fa640 1 -- 172.21.15.57:0/465001471 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7099+0+0 (secure 0 0 0) 0x7fa6fc089450 con 0x7fa70806ae40 2023-10-19T12:57:46.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.183+0000 7fa707fff640 1 --2- 172.21.15.57:0/465001471 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7fa6ec06ac90 0x7fa6ec06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:46.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.183+0000 7fa707fff640 1 --2- 172.21.15.57:0/465001471 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7fa6ec06ac90 0x7fa6ec06d150 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa7081f4510 tx=0x7fa6f4005e50 comp rx=0 tx=0).ready entity=mgr.5809 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:46.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.187+0000 7fa7057fa640 1 -- 172.21.15.57:0/465001471 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa6fc0542d0 con 0x7fa70806ae40 2023-10-19T12:57:46.401 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.399+0000 7fa70f1d7640 1 -- 172.21.15.57:0/465001471 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa7081539c0 con 0x7fa70806ae40 2023-10-19T12:57:46.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.399+0000 7fa7057fa640 1 -- 172.21.15.57:0/465001471 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4916 (secure 0 0 0) 0x7fa6fc0578f0 con 0x7fa70806ae40 2023-10-19T12:57:46.402 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:57:46.403 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5024,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/808517451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":808517451},{"type":"v1","addr":"172.21.15.179:6836","nonce":808517451}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":5625,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/3207366629","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":3207366629},{"type":"v1","addr":"172.21.15.57:6834","nonce":3207366629}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":5628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/2353910009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2353910009},{"type":"v1","addr":"172.21.15.57:6835","nonce":2353910009}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T12:45:40.706506+0000","modified":"2023-10-19T12:45:42.702983+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5634},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5634":{"gid":5634,"name":"d","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.179:6837/1237463333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":1237463333},{"type":"v1","addr":"172.21.15.179:6837","nonce":1237463333}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-19T12:57:46.403 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 16 2023-10-19T12:57:46.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.401+0000 7fa70f1d7640 1 -- 172.21.15.57:0/465001471 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7fa6ec06ac90 msgr2=0x7fa6ec06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:46.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.401+0000 7fa70f1d7640 1 --2- 172.21.15.57:0/465001471 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7fa6ec06ac90 0x7fa6ec06d150 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa7081f4510 tx=0x7fa6f4005e50 comp rx=0 tx=0).stop 2023-10-19T12:57:46.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.401+0000 7fa70f1d7640 1 -- 172.21.15.57:0/465001471 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa70806ae40 msgr2=0x7fa7080c9ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:46.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.401+0000 7fa70f1d7640 1 --2- 172.21.15.57:0/465001471 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa70806ae40 0x7fa7080c9ce0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fa6fc00c450 tx=0x7fa6fc00c910 comp rx=0 tx=0).stop 2023-10-19T12:57:46.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.401+0000 7fa70f1d7640 1 -- 172.21.15.57:0/465001471 shutdown_connections 2023-10-19T12:57:46.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.401+0000 7fa70f1d7640 1 --2- 172.21.15.57:0/465001471 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7fa6ec06ac90 0x7fa6ec06d150 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:46.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.401+0000 7fa70f1d7640 1 --2- 172.21.15.57:0/465001471 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa70815c1f0 0x7fa7080ada30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:46.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.401+0000 7fa70f1d7640 1 --2- 172.21.15.57:0/465001471 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa7081579f0 0x7fa7081f0f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:46.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.401+0000 7fa70f1d7640 1 --2- 172.21.15.57:0/465001471 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa70806ae40 0x7fa7080c9ce0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:46.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.401+0000 7fa70f1d7640 1 -- 172.21.15.57:0/465001471 >> 172.21.15.57:0/465001471 conn(0x7fa70806fae0 msgr2=0x7fa7080cc780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:46.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.401+0000 7fa70f1d7640 1 -- 172.21.15.57:0/465001471 shutdown_connections 2023-10-19T12:57:46.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.401+0000 7fa70f1d7640 1 -- 172.21.15.57:0/465001471 wait complete. 2023-10-19T12:57:46.415 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T12:57:46.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.521+0000 7f7e23d95640 1 -- 172.21.15.57:0/904473680 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e1c13d350 msgr2=0x7f7e1c0cbd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:46.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.521+0000 7f7e23d95640 1 --2- 172.21.15.57:0/904473680 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e1c13d350 0x7f7e1c0cbd60 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c06a7e0 tx=0x7f7e1002bda0 comp rx=0 tx=0).stop 2023-10-19T12:57:46.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.521+0000 7f7e23d95640 1 -- 172.21.15.57:0/904473680 shutdown_connections 2023-10-19T12:57:46.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.521+0000 7f7e23d95640 1 --2- 172.21.15.57:0/904473680 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7e1c0c9750 0x7f7e1c0c9be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:46.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.521+0000 7f7e23d95640 1 --2- 172.21.15.57:0/904473680 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e1c13d350 0x7f7e1c0cbd60 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:46.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.521+0000 7f7e23d95640 1 --2- 172.21.15.57:0/904473680 >> v2:172.21.15.179:3300/0 conn(0x7f7e1c0cb980 0x7f7e1c13db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:46.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.521+0000 7f7e23d95640 1 -- 172.21.15.57:0/904473680 >> 172.21.15.57:0/904473680 conn(0x7f7e1c069930 msgr2=0x7f7e1c069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:46.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.521+0000 7f7e23d95640 1 -- 172.21.15.57:0/904473680 shutdown_connections 2023-10-19T12:57:46.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.521+0000 7f7e23d95640 1 -- 172.21.15.57:0/904473680 wait complete. 2023-10-19T12:57:46.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.522+0000 7f7e23d95640 1 Processor -- start 2023-10-19T12:57:46.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.522+0000 7f7e23d95640 1 -- start start 2023-10-19T12:57:46.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.523+0000 7f7e23d95640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7e1c0c9750 0x7f7e1c15aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:46.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.523+0000 7f7e23d95640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e1c0cb980 0x7f7e1c0b09a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:46.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.523+0000 7f7e23d95640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7e1c0b0ee0 0x7f7e1c1f0d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:46.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.523+0000 7f7e23d95640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e1c0b1390 con 0x7f7e1c0cb980 2023-10-19T12:57:46.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.523+0000 7f7e23d95640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e1c1f1bb0 con 0x7f7e1c0b0ee0 2023-10-19T12:57:46.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.523+0000 7f7e23d95640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e1c1f1e90 con 0x7f7e1c0c9750 2023-10-19T12:57:46.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.523+0000 7f7e21330640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e1c0cb980 0x7f7e1c0b09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:46.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.523+0000 7f7e21330640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e1c0cb980 0x7f7e1c0b09a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:53684/0 (socket says 172.21.15.57:53684) 2023-10-19T12:57:46.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.523+0000 7f7e21330640 1 -- 172.21.15.57:0/4036505369 learned_addr learned my addr 172.21.15.57:0/4036505369 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:57:46.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.523+0000 7f7e22332640 1 --2- 172.21.15.57:0/4036505369 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7e1c0b0ee0 0x7f7e1c1f0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:46.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.523+0000 7f7e21b31640 1 --2- 172.21.15.57:0/4036505369 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7e1c0c9750 0x7f7e1c15aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:46.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.523+0000 7f7e21330640 1 -- 172.21.15.57:0/4036505369 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7e1c0c9750 msgr2=0x7f7e1c15aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:46.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.523+0000 7f7e21330640 1 --2- 172.21.15.57:0/4036505369 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7e1c0c9750 0x7f7e1c15aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:46.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.523+0000 7f7e21330640 1 -- 172.21.15.57:0/4036505369 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7e1c0b0ee0 msgr2=0x7f7e1c1f0d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:46.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.523+0000 7f7e21330640 1 --2- 172.21.15.57:0/4036505369 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7e1c0b0ee0 0x7f7e1c1f0d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:46.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.523+0000 7f7e21330640 1 -- 172.21.15.57:0/4036505369 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e10009560 con 0x7f7e1c0cb980 2023-10-19T12:57:46.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.524+0000 7f7e21330640 1 --2- 172.21.15.57:0/4036505369 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e1c0cb980 0x7f7e1c0b09a0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f7e10036040 tx=0x7f7e1002dd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:46.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.524+0000 7f7e1a7fc640 1 -- 172.21.15.57:0/4036505369 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e10003800 con 0x7f7e1c0cb980 2023-10-19T12:57:46.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.524+0000 7f7e1a7fc640 1 -- 172.21.15.57:0/4036505369 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f7e10003a50 con 0x7f7e1c0cb980 2023-10-19T12:57:46.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.524+0000 7f7e23d95640 1 -- 172.21.15.57:0/4036505369 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e1c1f2170 con 0x7f7e1c0cb980 2023-10-19T12:57:46.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.524+0000 7f7e1a7fc640 1 -- 172.21.15.57:0/4036505369 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e10003de0 con 0x7f7e1c0cb980 2023-10-19T12:57:46.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.524+0000 7f7e23d95640 1 -- 172.21.15.57:0/4036505369 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e1c1502c0 con 0x7f7e1c0cb980 2023-10-19T12:57:46.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.525+0000 7f7e23d95640 1 -- 172.21.15.57:0/4036505369 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e1c1f1fd0 con 0x7f7e1c0cb980 2023-10-19T12:57:46.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.525+0000 7f7e1a7fc640 1 -- 172.21.15.57:0/4036505369 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 18) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f7e1003e450 con 0x7f7e1c0cb980 2023-10-19T12:57:46.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.526+0000 7f7e1a7fc640 1 --2- 172.21.15.57:0/4036505369 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f7e0806ac90 0x7f7e0806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:46.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.526+0000 7f7e1a7fc640 1 -- 172.21.15.57:0/4036505369 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7099+0+0 (secure 0 0 0) 0x7f7e100ac060 con 0x7f7e1c0cb980 2023-10-19T12:57:46.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.526+0000 7f7e21b31640 1 --2- 172.21.15.57:0/4036505369 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f7e0806ac90 0x7f7e0806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:46.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.526+0000 7f7e21b31640 1 --2- 172.21.15.57:0/4036505369 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f7e0806ac90 0x7f7e0806d150 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f7e140058c0 tx=0x7f7e1400a2d0 comp rx=0 tx=0).ready entity=mgr.5809 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:46.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.530+0000 7f7e1a7fc640 1 -- 172.21.15.57:0/4036505369 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7e1c1f1fd0 con 0x7f7e1c0cb980 2023-10-19T12:57:46.705 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T12:57:46.708 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T12:57:46.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.708+0000 7f7e23d95640 1 -- 172.21.15.57:0/4036505369 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7e1c1f1fd0 con 0x7f7e1c0cb980 2023-10-19T12:57:46.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.709+0000 7f7e1a7fc640 1 -- 172.21.15.57:0/4036505369 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) v1 ==== 74+0+18354 (secure 0 0 0) 0x7f7e1c1f1fd0 con 0x7f7e1c0cb980 2023-10-19T12:57:46.712 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:57:46.712 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":58,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T12:57:44.608979+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2023-10-19T12:45:38.964640+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2023-10-19T12:45:39.683197+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":233,"snap_epoch":57,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.31","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.179:0/2997987784":"2023-10-20T12:57:44.608940+0000","172.21.15.179:0/2313691653":"2023-10-20T12:45:56.327729+0000","172.21.15.179:0/2223195228":"2023-10-20T12:45:56.327729+0000","172.21.15.179:0/515590766":"2023-10-20T12:45:56.327729+0000","172.21.15.179:0/4235058089":"2023-10-20T12:57:44.608940+0000","172.21.15.179:0/1377585784":"2023-10-20T12:45:56.327729+0000","172.21.15.179:0/3471628125":"2023-10-20T12:57:44.608940+0000","172.21.15.179:6832/2609646285":"2023-10-20T12:45:56.327729+0000","172.21.15.179:6833/2609646285":"2023-10-20T12:45:56.327729+0000","172.21.15.179:6832/922398470":"2023-10-20T12:57:44.608940+0000","172.21.15.179:6833/922398470":"2023-10-20T12:57:44.608940+0000","172.21.15.179:0/2754375794":"2023-10-20T12:57:44.608940+0000","172.21.15.179:0/161411072":"2023-10-20T12:57:44.608940+0000","172.21.15.179:0/4022258517":"2023-10-20T12:57:44.608940+0000","172.21.15.179:0/3663257206":"2023-10-20T12:57:44.608940+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":5,"length":2},{"begin":8,"length":1},{"begin":10,"length":1},{"begin":12,"length":1},{"begin":14,"length":5},{"begin":20,"length":3},{"begin":24,"length":6},{"begin":31,"length":3},{"begin":35,"length":6},{"begin":42,"length":3},{"begin":46,"length":5},{"begin":52,"length":3},{"begin":56,"length":5},{"begin":62,"length":3},{"begin":66,"length":5},{"begin":72,"length":3},{"begin":76,"length":6},{"begin":83,"length":3},{"begin":87,"length":6},{"begin":94,"length":3},{"begin":98,"length":6},{"begin":105,"length":10},{"begin":116,"length":2},{"begin":140,"length":94}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T12:57:46.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.711+0000 7f7e23d95640 1 -- 172.21.15.57:0/4036505369 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f7e0806ac90 msgr2=0x7f7e0806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:46.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.711+0000 7f7e23d95640 1 --2- 172.21.15.57:0/4036505369 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f7e0806ac90 0x7f7e0806d150 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f7e140058c0 tx=0x7f7e1400a2d0 comp rx=0 tx=0).stop 2023-10-19T12:57:46.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.711+0000 7f7e23d95640 1 -- 172.21.15.57:0/4036505369 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e1c0cb980 msgr2=0x7f7e1c0b09a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:46.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.711+0000 7f7e23d95640 1 --2- 172.21.15.57:0/4036505369 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e1c0cb980 0x7f7e1c0b09a0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f7e10036040 tx=0x7f7e1002dd70 comp rx=0 tx=0).stop 2023-10-19T12:57:46.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.711+0000 7f7e23d95640 1 -- 172.21.15.57:0/4036505369 shutdown_connections 2023-10-19T12:57:46.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.711+0000 7f7e23d95640 1 --2- 172.21.15.57:0/4036505369 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f7e0806ac90 0x7f7e0806d150 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:46.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.711+0000 7f7e23d95640 1 --2- 172.21.15.57:0/4036505369 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7e1c0b0ee0 0x7f7e1c1f0d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:46.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.711+0000 7f7e23d95640 1 --2- 172.21.15.57:0/4036505369 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e1c0cb980 0x7f7e1c0b09a0 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:46.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.711+0000 7f7e23d95640 1 --2- 172.21.15.57:0/4036505369 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7e1c0c9750 0x7f7e1c15aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:46.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.711+0000 7f7e23d95640 1 -- 172.21.15.57:0/4036505369 >> 172.21.15.57:0/4036505369 conn(0x7f7e1c069930 msgr2=0x7f7e1c069d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:46.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.711+0000 7f7e23d95640 1 -- 172.21.15.57:0/4036505369 shutdown_connections 2023-10-19T12:57:46.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.711+0000 7f7e23d95640 1 -- 172.21.15.57:0/4036505369 wait complete. 2023-10-19T12:57:46.726 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-19T12:57:46.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.873+0000 7f2b3cc94640 1 -- 172.21.15.57:0/2450079681 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2b240047c0 msgr2=0x7f2b24004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:46.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.873+0000 7f2b3cc94640 1 --2- 172.21.15.57:0/2450079681 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2b240047c0 0x7f2b24004c40 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f2b2000c260 tx=0x7f2b240057f0 comp rx=0 tx=0).stop 2023-10-19T12:57:46.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.873+0000 7f2b36d9d640 1 -- 172.21.15.57:0/2450079681 reap_dead start 2023-10-19T12:57:46.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.873+0000 7f2b3cc94640 1 -- 172.21.15.57:0/2450079681 shutdown_connections 2023-10-19T12:57:46.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.873+0000 7f2b3cc94640 1 -- 172.21.15.57:0/2450079681 >> 172.21.15.57:0/2450079681 conn(0x7f2b3806fae0 msgr2=0x7f2b3806fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:46.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.873+0000 7f2b3cc94640 1 -- 172.21.15.57:0/2450079681 shutdown_connections 2023-10-19T12:57:46.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.873+0000 7f2b3cc94640 1 -- 172.21.15.57:0/2450079681 wait complete. 2023-10-19T12:57:46.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.874+0000 7f2b3cc94640 1 Processor -- start 2023-10-19T12:57:46.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.875+0000 7f2b3cc94640 1 -- start start 2023-10-19T12:57:46.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.875+0000 7f2b3cc94640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2b240047c0 0x7f2b38153eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:46.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.875+0000 7f2b3cc94640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2b381e2b90 0x7f2b381517b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:46.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.875+0000 7f2b3cc94640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2b38151cf0 0x7f2b381521a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:46.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.875+0000 7f2b3cc94640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b380abcf0 con 0x7f2b381e2b90 2023-10-19T12:57:46.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.875+0000 7f2b3cc94640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b381e3310 con 0x7f2b38151cf0 2023-10-19T12:57:46.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.875+0000 7f2b3cc94640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b3814f4b0 con 0x7f2b240047c0 2023-10-19T12:57:46.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.875+0000 7f2b2ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2b381e2b90 0x7f2b381517b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:46.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.875+0000 7f2b2ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2b381e2b90 0x7f2b381517b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43710/0 (socket says 172.21.15.57:43710) 2023-10-19T12:57:46.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.875+0000 7f2b2ffff640 1 -- 172.21.15.57:0/2461921960 learned_addr learned my addr 172.21.15.57:0/2461921960 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:57:46.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.876+0000 7f2b3659c640 1 --2- 172.21.15.57:0/2461921960 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2b240047c0 0x7f2b38153eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:46.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.876+0000 7f2b36d9d640 1 --2- 172.21.15.57:0/2461921960 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2b38151cf0 0x7f2b381521a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:46.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.876+0000 7f2b2ffff640 1 -- 172.21.15.57:0/2461921960 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2b240047c0 msgr2=0x7f2b38153eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:46.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.876+0000 7f2b2ffff640 1 --2- 172.21.15.57:0/2461921960 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2b240047c0 0x7f2b38153eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:46.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.876+0000 7f2b2ffff640 1 -- 172.21.15.57:0/2461921960 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2b38151cf0 msgr2=0x7f2b381521a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:46.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.876+0000 7f2b2ffff640 1 --2- 172.21.15.57:0/2461921960 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2b38151cf0 0x7f2b381521a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:46.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.876+0000 7f2b2ffff640 1 -- 172.21.15.57:0/2461921960 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b381e3030 con 0x7f2b381e2b90 2023-10-19T12:57:46.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.876+0000 7f2b2ffff640 1 --2- 172.21.15.57:0/2461921960 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2b381e2b90 0x7f2b381517b0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f2b2000c380 tx=0x7f2b20001700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:46.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.876+0000 7f2b2effd640 1 -- 172.21.15.57:0/2461921960 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b20006230 con 0x7f2b381e2b90 2023-10-19T12:57:46.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.876+0000 7f2b2effd640 1 -- 172.21.15.57:0/2461921960 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f2b20006390 con 0x7f2b381e2b90 2023-10-19T12:57:46.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.876+0000 7f2b3cc94640 1 -- 172.21.15.57:0/2461921960 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b38150cd0 con 0x7f2b381e2b90 2023-10-19T12:57:46.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.876+0000 7f2b2effd640 1 -- 172.21.15.57:0/2461921960 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b200035f0 con 0x7f2b381e2b90 2023-10-19T12:57:46.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.876+0000 7f2b3cc94640 1 -- 172.21.15.57:0/2461921960 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b38151290 con 0x7f2b381e2b90 2023-10-19T12:57:46.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.877+0000 7f2b3cc94640 1 -- 172.21.15.57:0/2461921960 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b381e3840 con 0x7f2b381e2b90 2023-10-19T12:57:46.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.878+0000 7f2b2effd640 1 -- 172.21.15.57:0/2461921960 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 18) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f2b20031030 con 0x7f2b381e2b90 2023-10-19T12:57:46.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.878+0000 7f2b2effd640 1 --2- 172.21.15.57:0/2461921960 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f2b1806ac90 0x7f2b1806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:46.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.878+0000 7f2b2effd640 1 -- 172.21.15.57:0/2461921960 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7099+0+0 (secure 0 0 0) 0x7f2b200aa5b0 con 0x7f2b381e2b90 2023-10-19T12:57:46.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.878+0000 7f2b3659c640 1 --2- 172.21.15.57:0/2461921960 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f2b1806ac90 0x7f2b1806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:46.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.879+0000 7f2b3659c640 1 --2- 172.21.15.57:0/2461921960 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f2b1806ac90 0x7f2b1806d150 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2b28009aa0 tx=0x7f2b28005f90 comp rx=0 tx=0).ready entity=mgr.5809 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:46.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:46.882+0000 7f2b2effd640 1 -- 172.21.15.57:0/2461921960 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2b20078ec0 con 0x7f2b381e2b90 2023-10-19T12:57:47.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:47.095+0000 7f2b3cc94640 1 -- 172.21.15.57:0/2461921960 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f2b38151420 con 0x7f2b381e2b90 2023-10-19T12:57:47.098 INFO:tasks.ceph.mon.a.smithi057.stderr:2023-10-19T12:57:47.095+0000 7f1253ff3640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-19T12:57:47.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:47.097+0000 7f2b2effd640 1 -- 172.21.15.57:0/2461921960 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v17) v1 ==== 161+0+0 (secure 0 0 0) 0x7f2b20078a50 con 0x7f2b381e2b90 2023-10-19T12:57:47.100 INFO:teuthology.orchestra.run.smithi057.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-19T12:57:47.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:47.099+0000 7f2b3cc94640 1 -- 172.21.15.57:0/2461921960 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f2b1806ac90 msgr2=0x7f2b1806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:47.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:47.099+0000 7f2b3cc94640 1 --2- 172.21.15.57:0/2461921960 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f2b1806ac90 0x7f2b1806d150 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2b28009aa0 tx=0x7f2b28005f90 comp rx=0 tx=0).stop 2023-10-19T12:57:47.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:47.099+0000 7f2b3cc94640 1 -- 172.21.15.57:0/2461921960 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2b381e2b90 msgr2=0x7f2b381517b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:47.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:47.099+0000 7f2b3cc94640 1 --2- 172.21.15.57:0/2461921960 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2b381e2b90 0x7f2b381517b0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f2b2000c380 tx=0x7f2b20001700 comp rx=0 tx=0).stop 2023-10-19T12:57:47.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:47.099+0000 7f2b3cc94640 1 -- 172.21.15.57:0/2461921960 shutdown_connections 2023-10-19T12:57:47.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:47.099+0000 7f2b3cc94640 1 --2- 172.21.15.57:0/2461921960 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f2b1806ac90 0x7f2b1806d150 secure :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2b28009aa0 tx=0x7f2b28005f90 comp rx=0 tx=0).stop 2023-10-19T12:57:47.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:47.099+0000 7f2b3cc94640 1 --2- 172.21.15.57:0/2461921960 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2b38151cf0 0x7f2b381521a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:47.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:47.099+0000 7f2b3cc94640 1 --2- 172.21.15.57:0/2461921960 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2b381e2b90 0x7f2b381517b0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:47.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:47.099+0000 7f2b3cc94640 1 --2- 172.21.15.57:0/2461921960 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2b240047c0 0x7f2b38153eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:47.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:47.099+0000 7f2b3cc94640 1 -- 172.21.15.57:0/2461921960 >> 172.21.15.57:0/2461921960 conn(0x7f2b3806fae0 msgr2=0x7f2b3806c140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:47.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:47.099+0000 7f2b3cc94640 1 -- 172.21.15.57:0/2461921960 shutdown_connections 2023-10-19T12:57:47.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:47.099+0000 7f2b3cc94640 1 -- 172.21.15.57:0/2461921960 wait complete. 2023-10-19T12:57:47.112 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-19T12:57:47.140 INFO:tasks.ceph.mds.d.smithi179.stdout:starting mds.d at 2023-10-19T12:57:47.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:47.222+0000 7f955b59e640 1 -- 172.21.15.57:0/1960269298 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f955c15b040 msgr2=0x7f955c0cd620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:47.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:47.222+0000 7f955b59e640 1 --2- 172.21.15.57:0/1960269298 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f955c15b040 0x7f955c0cd620 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f954c009830 tx=0x7f954c02bd60 comp rx=0 tx=0).stop 2023-10-19T12:57:47.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:47.222+0000 7f955b59e640 1 -- 172.21.15.57:0/1960269298 shutdown_connections 2023-10-19T12:57:47.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:47.222+0000 7f955b59e640 1 --2- 172.21.15.57:0/1960269298 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f955c15b040 0x7f955c0cd620 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:47.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:47.222+0000 7f955b59e640 1 -- 172.21.15.57:0/1960269298 >> 172.21.15.57:0/1960269298 conn(0x7f955c099eb0 msgr2=0x7f955c0d0c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:47.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:47.222+0000 7f955b59e640 1 -- 172.21.15.57:0/1960269298 shutdown_connections 2023-10-19T12:57:47.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:47.222+0000 7f955b59e640 1 -- 172.21.15.57:0/1960269298 wait complete. 2023-10-19T12:57:47.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:47.223+0000 7f955b59e640 1 Processor -- start 2023-10-19T12:57:47.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:47.223+0000 7f955b59e640 1 -- start start 2023-10-19T12:57:47.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:47.224+0000 7f955b59e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f955c0cbe10 0x7f955c1f0af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:47.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:47.224+0000 7f955b59e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f955c0cdb60 0x7f955c0c4590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:47.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:47.224+0000 7f955b59e640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f955c15b040 0x7f955c0c4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:47.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:47.224+0000 7f955b59e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f955c0b3fd0 con 0x7f955c0cbe10 2023-10-19T12:57:47.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:47.224+0000 7f955b59e640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f955c15d8a0 con 0x7f955c0cdb60 2023-10-19T12:57:47.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:47.224+0000 7f955b59e640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f955c15d5c0 con 0x7f955c15b040 2023-10-19T12:57:47.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:47.224+0000 7f955a59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f955c0cbe10 0x7f955c1f0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:47.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:47.224+0000 7f955a59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f955c0cbe10 0x7f955c1f0af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43724/0 (socket says 172.21.15.57:43724) 2023-10-19T12:57:47.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:47.224+0000 7f955a59c640 1 -- 172.21.15.57:0/2561224426 learned_addr learned my addr 172.21.15.57:0/2561224426 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:57:47.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:47.224+0000 7f9553fff640 1 --2- 172.21.15.57:0/2561224426 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f955c0cdb60 0x7f955c0c4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:47.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:47.224+0000 7f955ad9d640 1 --2- 172.21.15.57:0/2561224426 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f955c15b040 0x7f955c0c4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:47.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:47.224+0000 7f955a59c640 1 -- 172.21.15.57:0/2561224426 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f955c15b040 msgr2=0x7f955c0c4ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:47.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:47.224+0000 7f955a59c640 1 --2- 172.21.15.57:0/2561224426 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f955c15b040 0x7f955c0c4ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:47.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:47.224+0000 7f955a59c640 1 -- 172.21.15.57:0/2561224426 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f955c0cdb60 msgr2=0x7f955c0c4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:47.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:47.224+0000 7f955a59c640 1 --2- 172.21.15.57:0/2561224426 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f955c0cdb60 0x7f955c0c4590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:47.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:47.224+0000 7f955a59c640 1 -- 172.21.15.57:0/2561224426 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f954c009490 con 0x7f955c0cbe10 2023-10-19T12:57:47.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:47.224+0000 7f955ad9d640 1 --2- 172.21.15.57:0/2561224426 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f955c15b040 0x7f955c0c4ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:57:47.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:47.224+0000 7f955a59c640 1 --2- 172.21.15.57:0/2561224426 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f955c0cbe10 0x7f955c1f0af0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f954c000c00 tx=0x7f954c0038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:47.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:47.224+0000 7f9552ffd640 1 -- 172.21.15.57:0/2561224426 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f954c003ea0 con 0x7f955c0cbe10 2023-10-19T12:57:47.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:47.225+0000 7f9552ffd640 1 -- 172.21.15.57:0/2561224426 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f954c034530 con 0x7f955c0cbe10 2023-10-19T12:57:47.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:47.225+0000 7f955b59e640 1 -- 172.21.15.57:0/2561224426 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f955c0b3980 con 0x7f955c0cbe10 2023-10-19T12:57:47.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:47.225+0000 7f9552ffd640 1 -- 172.21.15.57:0/2561224426 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f954c034940 con 0x7f955c0cbe10 2023-10-19T12:57:47.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:47.225+0000 7f955b59e640 1 -- 172.21.15.57:0/2561224426 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f955c1f1030 con 0x7f955c0cbe10 2023-10-19T12:57:47.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:47.225+0000 7f955b59e640 1 -- 172.21.15.57:0/2561224426 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f955c066960 con 0x7f955c0cbe10 2023-10-19T12:57:47.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:47.226+0000 7f9552ffd640 1 -- 172.21.15.57:0/2561224426 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 18) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f954c034aa0 con 0x7f955c0cbe10 2023-10-19T12:57:47.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:47.226+0000 7f9552ffd640 1 --2- 172.21.15.57:0/2561224426 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f953c06ac90 0x7f953c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:47.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:47.226+0000 7f9552ffd640 1 -- 172.21.15.57:0/2561224426 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 7185+0+0 (secure 0 0 0) 0x7f954c0ac1f0 con 0x7f955c0cbe10 2023-10-19T12:57:47.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:47.227+0000 7f9553fff640 1 --2- 172.21.15.57:0/2561224426 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f953c06ac90 0x7f953c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:47.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:47.227+0000 7f9553fff640 1 --2- 172.21.15.57:0/2561224426 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f953c06ac90 0x7f953c06d150 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f955c150c50 tx=0x7f9544007e50 comp rx=0 tx=0).ready entity=mgr.5809 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:47.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:47.230+0000 7f9552ffd640 1 -- 172.21.15.57:0/2561224426 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f954c076f40 con 0x7f955c0cbe10 2023-10-19T12:57:47.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:47.443+0000 7f955b59e640 1 -- 172.21.15.57:0/2561224426 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f955c04da70 con 0x7f955c0cbe10 2023-10-19T12:57:47.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:47.626+0000 7f9552ffd640 1 -- 172.21.15.57:0/2561224426 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mgrmap(e 19) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f954c034d50 con 0x7f955c0cbe10 2023-10-19T12:57:48.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:48.100+0000 7f9552ffd640 1 -- 172.21.15.57:0/2561224426 <== mon.0 v2:172.21.15.57:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v18) v1 ==== 104+0+0 (secure 0 0 0) 0x7f954c090b40 con 0x7f955c0cbe10 2023-10-19T12:57:48.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:48.102+0000 7f955b59e640 1 -- 172.21.15.57:0/2561224426 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f953c06ac90 msgr2=0x7f953c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:48.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:48.102+0000 7f955b59e640 1 --2- 172.21.15.57:0/2561224426 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f953c06ac90 0x7f953c06d150 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f955c150c50 tx=0x7f9544007e50 comp rx=0 tx=0).stop 2023-10-19T12:57:48.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:48.102+0000 7f955b59e640 1 -- 172.21.15.57:0/2561224426 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f955c0cbe10 msgr2=0x7f955c1f0af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:48.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:48.102+0000 7f955b59e640 1 --2- 172.21.15.57:0/2561224426 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f955c0cbe10 0x7f955c1f0af0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f954c000c00 tx=0x7f954c0038a0 comp rx=0 tx=0).stop 2023-10-19T12:57:48.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:48.102+0000 7f955b59e640 1 -- 172.21.15.57:0/2561224426 shutdown_connections 2023-10-19T12:57:48.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:48.102+0000 7f955b59e640 1 --2- 172.21.15.57:0/2561224426 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f953c06ac90 0x7f953c06d150 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:48.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:48.102+0000 7f955b59e640 1 --2- 172.21.15.57:0/2561224426 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f955c15b040 0x7f955c0c4ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:48.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:48.102+0000 7f955b59e640 1 --2- 172.21.15.57:0/2561224426 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f955c0cdb60 0x7f955c0c4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:48.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:48.102+0000 7f955b59e640 1 --2- 172.21.15.57:0/2561224426 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f955c0cbe10 0x7f955c1f0af0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:48.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:48.102+0000 7f955b59e640 1 -- 172.21.15.57:0/2561224426 >> 172.21.15.57:0/2561224426 conn(0x7f955c099eb0 msgr2=0x7f955c150da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:48.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:48.102+0000 7f955b59e640 1 -- 172.21.15.57:0/2561224426 shutdown_connections 2023-10-19T12:57:48.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:48.102+0000 7f955b59e640 1 -- 172.21.15.57:0/2561224426 wait complete. 2023-10-19T12:57:48.117 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-19T12:57:48.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:48.235+0000 7f7931639640 1 -- 172.21.15.57:0/1382258576 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f79180039a0 msgr2=0x7f7918003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:48.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:48.235+0000 7f7931639640 1 --2- 172.21.15.57:0/1382258576 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f79180039a0 0x7f7918003da0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f792400c710 tx=0x7f79180057f0 comp rx=0 tx=0).stop 2023-10-19T12:57:48.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:48.235+0000 7f792b7fe640 1 -- 172.21.15.57:0/1382258576 reap_dead start 2023-10-19T12:57:48.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:48.235+0000 7f7931639640 1 -- 172.21.15.57:0/1382258576 shutdown_connections 2023-10-19T12:57:48.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:48.235+0000 7f7931639640 1 -- 172.21.15.57:0/1382258576 >> 172.21.15.57:0/1382258576 conn(0x7f792c0b0670 msgr2=0x7f792c0b0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:48.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:48.235+0000 7f7931639640 1 -- 172.21.15.57:0/1382258576 shutdown_connections 2023-10-19T12:57:48.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:48.235+0000 7f7931639640 1 -- 172.21.15.57:0/1382258576 wait complete. 2023-10-19T12:57:48.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:48.236+0000 7f7931639640 1 Processor -- start 2023-10-19T12:57:48.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:48.236+0000 7f7931639640 1 -- start start 2023-10-19T12:57:48.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:48.237+0000 7f7931639640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f79180039a0 0x7f792c155ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:48.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:48.237+0000 7f7931639640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f792c0997e0 0x7f792c0c80e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:48.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:48.237+0000 7f7931639640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f792c099c60 0x7f792c0c8620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:48.243 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:48.237+0000 7f7931639640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f792c0c66d0 con 0x7f792c099c60 2023-10-19T12:57:48.243 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:48.237+0000 7f7931639640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f792c0b40e0 con 0x7f79180039a0 2023-10-19T12:57:48.243 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:48.237+0000 7f7931639640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f792c1560e0 con 0x7f792c0997e0 2023-10-19T12:57:48.243 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:48.237+0000 7f792b7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f792c099c60 0x7f792c0c8620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:48.243 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:48.237+0000 7f792b7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f792c099c60 0x7f792c0c8620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43744/0 (socket says 172.21.15.57:43744) 2023-10-19T12:57:48.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:48.237+0000 7f792b7fe640 1 -- 172.21.15.57:0/2880308136 learned_addr learned my addr 172.21.15.57:0/2880308136 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:57:48.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:48.237+0000 7f792a7fc640 1 --2- 172.21.15.57:0/2880308136 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f792c0997e0 0x7f792c0c80e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:48.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:48.237+0000 7f792affd640 1 --2- 172.21.15.57:0/2880308136 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f79180039a0 0x7f792c155ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:48.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:48.237+0000 7f792b7fe640 1 -- 172.21.15.57:0/2880308136 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f792c0997e0 msgr2=0x7f792c0c80e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:48.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:48.237+0000 7f792b7fe640 1 --2- 172.21.15.57:0/2880308136 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f792c0997e0 0x7f792c0c80e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:48.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:48.237+0000 7f792b7fe640 1 -- 172.21.15.57:0/2880308136 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f79180039a0 msgr2=0x7f792c155ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:48.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:48.237+0000 7f792b7fe640 1 --2- 172.21.15.57:0/2880308136 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f79180039a0 0x7f792c155ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:48.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:48.237+0000 7f792b7fe640 1 -- 172.21.15.57:0/2880308136 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f792c1565f0 con 0x7f792c099c60 2023-10-19T12:57:48.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:48.238+0000 7f792b7fe640 1 --2- 172.21.15.57:0/2880308136 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f792c099c60 0x7f792c0c8620 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f792c072c40 tx=0x7f791c00fd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:48.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:48.238+0000 7f79237fe640 1 -- 172.21.15.57:0/2880308136 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f791c019070 con 0x7f792c099c60 2023-10-19T12:57:48.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:48.238+0000 7f79237fe640 1 -- 172.21.15.57:0/2880308136 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f791c004c10 con 0x7f792c099c60 2023-10-19T12:57:48.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:48.238+0000 7f7931639640 1 -- 172.21.15.57:0/2880308136 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f792c0c5d30 con 0x7f792c099c60 2023-10-19T12:57:48.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:48.238+0000 7f79237fe640 1 -- 172.21.15.57:0/2880308136 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f791c00daa0 con 0x7f792c099c60 2023-10-19T12:57:48.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:48.238+0000 7f7931639640 1 -- 172.21.15.57:0/2880308136 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f792c0c62f0 con 0x7f792c099c60 2023-10-19T12:57:48.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:48.239+0000 7f7931639640 1 -- 172.21.15.57:0/2880308136 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f792c0ce1d0 con 0x7f792c099c60 2023-10-19T12:57:48.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:48.239+0000 7f79237fe640 1 -- 172.21.15.57:0/2880308136 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 19) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f791c00dc00 con 0x7f792c099c60 2023-10-19T12:57:48.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:48.240+0000 7f79237fe640 1 --2- 172.21.15.57:0/2880308136 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f790c06aed0 0x7f790c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:48.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:48.240+0000 7f79237fe640 1 -- 172.21.15.57:0/2880308136 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 7185+0+0 (secure 0 0 0) 0x7f791c089cf0 con 0x7f792c099c60 2023-10-19T12:57:48.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:48.240+0000 7f792affd640 1 --2- 172.21.15.57:0/2880308136 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f790c06aed0 0x7f790c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:48.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:48.240+0000 7f792affd640 1 --2- 172.21.15.57:0/2880308136 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f790c06aed0 0x7f790c06d390 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f7924034000 tx=0x7f7924003760 comp rx=0 tx=0).ready entity=mgr.5809 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:48.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:48.244+0000 7f79237fe640 1 -- 172.21.15.57:0/2880308136 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f791c054700 con 0x7f792c099c60 2023-10-19T12:57:48.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:48.423+0000 7f7931639640 1 -- 172.21.15.57:0/2880308136 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f792c0c6810 con 0x7f792c099c60 2023-10-19T12:57:49.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:49.180+0000 7f79237fe640 1 -- 172.21.15.57:0/2880308136 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v61) v1 ==== 181+0+0 (secure 0 0 0) 0x7f792c0c6810 con 0x7f792c099c60 2023-10-19T12:57:49.184 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'cephfs_metadata' removed 2023-10-19T12:57:49.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:49.187+0000 7f7931639640 1 -- 172.21.15.57:0/2880308136 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f790c06aed0 msgr2=0x7f790c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:49.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:49.187+0000 7f7931639640 1 --2- 172.21.15.57:0/2880308136 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f790c06aed0 0x7f790c06d390 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f7924034000 tx=0x7f7924003760 comp rx=0 tx=0).stop 2023-10-19T12:57:49.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:49.187+0000 7f7931639640 1 -- 172.21.15.57:0/2880308136 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f792c099c60 msgr2=0x7f792c0c8620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:49.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:49.188+0000 7f7931639640 1 --2- 172.21.15.57:0/2880308136 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f792c099c60 0x7f792c0c8620 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f792c072c40 tx=0x7f791c00fd90 comp rx=0 tx=0).stop 2023-10-19T12:57:49.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:49.188+0000 7f79237fe640 1 -- 172.21.15.57:0/2880308136 <== mon.0 v2:172.21.15.57:3300/0 8 ==== mgrmap(e 20) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f791c054ad0 con 0x7f792c099c60 2023-10-19T12:57:49.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:49.189+0000 7f7931639640 1 -- 172.21.15.57:0/2880308136 shutdown_connections 2023-10-19T12:57:49.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:49.190+0000 7f7931639640 1 --2- 172.21.15.57:0/2880308136 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f790c06aed0 0x7f790c06d390 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:49.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:49.190+0000 7f7931639640 1 --2- 172.21.15.57:0/2880308136 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f792c099c60 0x7f792c0c8620 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:49.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:49.190+0000 7f7931639640 1 --2- 172.21.15.57:0/2880308136 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f792c0997e0 0x7f792c0c80e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:49.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:49.190+0000 7f7931639640 1 --2- 172.21.15.57:0/2880308136 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f79180039a0 0x7f792c155ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:49.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:49.190+0000 7f7931639640 1 -- 172.21.15.57:0/2880308136 >> 172.21.15.57:0/2880308136 conn(0x7f792c0b0670 msgr2=0x7f792c06d6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:49.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:49.190+0000 7f7931639640 1 -- 172.21.15.57:0/2880308136 shutdown_connections 2023-10-19T12:57:49.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:49.190+0000 7f7931639640 1 -- 172.21.15.57:0/2880308136 wait complete. 2023-10-19T12:57:49.208 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-19T12:57:49.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:49.328+0000 7fdff891d640 1 -- 172.21.15.57:0/202450108 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdff4141690 msgr2=0x7fdff40703e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:49.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:49.328+0000 7fdff891d640 1 --2- 172.21.15.57:0/202450108 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdff4141690 0x7fdff40703e0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fdff40cbd00 tx=0x7fdfdc02bcf0 comp rx=0 tx=0).stop 2023-10-19T12:57:49.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:49.328+0000 7fdff891d640 1 -- 172.21.15.57:0/202450108 shutdown_connections 2023-10-19T12:57:49.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:49.328+0000 7fdff891d640 1 --2- 172.21.15.57:0/202450108 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdff4141690 0x7fdff40703e0 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:49.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:49.328+0000 7fdff891d640 1 --2- 172.21.15.57:0/202450108 >> v2:172.21.15.179:3300/0 conn(0x7fdff413e520 0x7fdff406bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:49.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:49.328+0000 7fdff891d640 1 -- 172.21.15.57:0/202450108 >> 172.21.15.57:0/202450108 conn(0x7fdff4069930 msgr2=0x7fdff4069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:49.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:49.328+0000 7fdff891d640 1 -- 172.21.15.57:0/202450108 shutdown_connections 2023-10-19T12:57:49.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:49.328+0000 7fdff891d640 1 -- 172.21.15.57:0/202450108 wait complete. 2023-10-19T12:57:49.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:49.329+0000 7fdff891d640 1 Processor -- start 2023-10-19T12:57:49.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:49.329+0000 7fdff891d640 1 -- start start 2023-10-19T12:57:49.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:49.330+0000 7fdff891d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdff406c0a0 0x7fdff41e1d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:49.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:49.330+0000 7fdff891d640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdff413e520 0x7fdff41e3840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:49.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:49.330+0000 7fdff891d640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdff4141690 0x7fdff41e3d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:49.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:49.330+0000 7fdff891d640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdff41f00f0 con 0x7fdff406c0a0 2023-10-19T12:57:49.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:49.330+0000 7fdff891d640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdff41ef4e0 con 0x7fdff413e520 2023-10-19T12:57:49.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:49.330+0000 7fdff891d640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdff41ef7c0 con 0x7fdff4141690 2023-10-19T12:57:49.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:49.330+0000 7fdff259c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdff406c0a0 0x7fdff41e1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:49.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:49.330+0000 7fdff259c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdff406c0a0 0x7fdff41e1d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43762/0 (socket says 172.21.15.57:43762) 2023-10-19T12:57:49.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:49.330+0000 7fdff259c640 1 -- 172.21.15.57:0/3805557348 learned_addr learned my addr 172.21.15.57:0/3805557348 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:57:49.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:49.330+0000 7fdfebfff640 1 --2- 172.21.15.57:0/3805557348 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdff413e520 0x7fdff41e3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:49.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:49.330+0000 7fdff259c640 1 -- 172.21.15.57:0/3805557348 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdff4141690 msgr2=0x7fdff41e3d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:49.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:49.330+0000 7fdff2d9d640 1 --2- 172.21.15.57:0/3805557348 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdff4141690 0x7fdff41e3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:49.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:49.330+0000 7fdff259c640 1 --2- 172.21.15.57:0/3805557348 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdff4141690 0x7fdff41e3d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:49.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:49.330+0000 7fdff259c640 1 -- 172.21.15.57:0/3805557348 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdff413e520 msgr2=0x7fdff41e3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:49.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:49.330+0000 7fdff259c640 1 --2- 172.21.15.57:0/3805557348 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdff413e520 0x7fdff41e3840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:49.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:49.330+0000 7fdff259c640 1 -- 172.21.15.57:0/3805557348 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfdc009490 con 0x7fdff406c0a0 2023-10-19T12:57:49.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:49.330+0000 7fdfebfff640 1 --2- 172.21.15.57:0/3805557348 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdff413e520 0x7fdff41e3840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:57:49.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:49.330+0000 7fdff2d9d640 1 --2- 172.21.15.57:0/3805557348 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdff4141690 0x7fdff41e3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:57:49.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:49.330+0000 7fdff259c640 1 --2- 172.21.15.57:0/3805557348 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdff406c0a0 0x7fdff41e1d20 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fdfe400d950 tx=0x7fdfe400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:49.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:49.331+0000 7fdfeaffd640 1 -- 172.21.15.57:0/3805557348 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfe4009c30 con 0x7fdff406c0a0 2023-10-19T12:57:49.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:49.331+0000 7fdfeaffd640 1 -- 172.21.15.57:0/3805557348 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fdfe4009e80 con 0x7fdff406c0a0 2023-10-19T12:57:49.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:49.331+0000 7fdff891d640 1 -- 172.21.15.57:0/3805557348 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdff41efaa0 con 0x7fdff406c0a0 2023-10-19T12:57:49.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:49.331+0000 7fdfeaffd640 1 -- 172.21.15.57:0/3805557348 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfe40106c0 con 0x7fdff406c0a0 2023-10-19T12:57:49.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:49.331+0000 7fdff891d640 1 -- 172.21.15.57:0/3805557348 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdff41e42c0 con 0x7fdff406c0a0 2023-10-19T12:57:49.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:49.331+0000 7fdff891d640 1 -- 172.21.15.57:0/3805557348 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdff40b1bd0 con 0x7fdff406c0a0 2023-10-19T12:57:49.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:49.332+0000 7fdfeaffd640 1 -- 172.21.15.57:0/3805557348 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 20) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fdfe4010820 con 0x7fdff406c0a0 2023-10-19T12:57:49.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:49.332+0000 7fdfeaffd640 1 --2- 172.21.15.57:0/3805557348 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7fdfd406af90 0x7fdfd406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:49.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:49.333+0000 7fdfeaffd640 1 -- 172.21.15.57:0/3805557348 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6351+0+0 (secure 0 0 0) 0x7fdfe408a060 con 0x7fdff406c0a0 2023-10-19T12:57:49.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:49.333+0000 7fdfebfff640 1 --2- 172.21.15.57:0/3805557348 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7fdfd406af90 0x7fdfd406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:49.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:49.333+0000 7fdfebfff640 1 --2- 172.21.15.57:0/3805557348 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7fdfd406af90 0x7fdfd406d450 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fdfdc036000 tx=0x7fdfdc004730 comp rx=0 tx=0).ready entity=mgr.5809 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:49.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:49.337+0000 7fdfeaffd640 1 -- 172.21.15.57:0/3805557348 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fdfe40bcca0 con 0x7fdff406c0a0 2023-10-19T12:57:49.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:49.515+0000 7fdff891d640 1 -- 172.21.15.57:0/3805557348 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fdff40aef70 con 0x7fdff406c0a0 2023-10-19T12:57:50.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.190+0000 7fdfeaffd640 1 -- 172.21.15.57:0/3805557348 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v62) v1 ==== 169+0+0 (secure 0 0 0) 0x7fdfe40170d0 con 0x7fdff406c0a0 2023-10-19T12:57:50.193 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'cephfs_data' removed 2023-10-19T12:57:50.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.193+0000 7fdff891d640 1 -- 172.21.15.57:0/3805557348 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7fdfd406af90 msgr2=0x7fdfd406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:50.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.193+0000 7fdff891d640 1 --2- 172.21.15.57:0/3805557348 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7fdfd406af90 0x7fdfd406d450 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fdfdc036000 tx=0x7fdfdc004730 comp rx=0 tx=0).stop 2023-10-19T12:57:50.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.193+0000 7fdff891d640 1 -- 172.21.15.57:0/3805557348 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdff406c0a0 msgr2=0x7fdff41e1d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:50.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.193+0000 7fdff891d640 1 --2- 172.21.15.57:0/3805557348 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdff406c0a0 0x7fdff41e1d20 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fdfe400d950 tx=0x7fdfe400de10 comp rx=0 tx=0).stop 2023-10-19T12:57:50.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.193+0000 7fdff891d640 1 -- 172.21.15.57:0/3805557348 shutdown_connections 2023-10-19T12:57:50.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.193+0000 7fdff891d640 1 --2- 172.21.15.57:0/3805557348 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7fdfd406af90 0x7fdfd406d450 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:50.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.193+0000 7fdff891d640 1 --2- 172.21.15.57:0/3805557348 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdff4141690 0x7fdff41e3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:50.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.193+0000 7fdff891d640 1 --2- 172.21.15.57:0/3805557348 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdff413e520 0x7fdff41e3840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:50.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.193+0000 7fdff891d640 1 --2- 172.21.15.57:0/3805557348 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdff406c0a0 0x7fdff41e1d20 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:50.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.193+0000 7fdff891d640 1 -- 172.21.15.57:0/3805557348 >> 172.21.15.57:0/3805557348 conn(0x7fdff4069930 msgr2=0x7fdff4069d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:50.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.193+0000 7fdff891d640 1 -- 172.21.15.57:0/3805557348 shutdown_connections 2023-10-19T12:57:50.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.193+0000 7fdff891d640 1 -- 172.21.15.57:0/3805557348 wait complete. 2023-10-19T12:57:50.213 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 30 2023-10-19T12:57:50.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.327+0000 7f323df25640 1 -- 172.21.15.57:0/3691030636 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f32381579f0 msgr2=0x7f32380b0e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:50.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.327+0000 7f323df25640 1 --2- 172.21.15.57:0/3691030636 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f32381579f0 0x7f32380b0e90 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f32380b1560 tx=0x7f322802bcb0 comp rx=0 tx=0).stop 2023-10-19T12:57:50.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.327+0000 7f323df25640 1 -- 172.21.15.57:0/3691030636 shutdown_connections 2023-10-19T12:57:50.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.327+0000 7f323df25640 1 --2- 172.21.15.57:0/3691030636 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f32381579f0 0x7f32380b0e90 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:50.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.327+0000 7f323df25640 1 -- 172.21.15.57:0/3691030636 >> 172.21.15.57:0/3691030636 conn(0x7f323806fae0 msgr2=0x7f323806fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:50.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.327+0000 7f323df25640 1 -- 172.21.15.57:0/3691030636 shutdown_connections 2023-10-19T12:57:50.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.327+0000 7f323df25640 1 -- 172.21.15.57:0/3691030636 wait complete. 2023-10-19T12:57:50.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.328+0000 7f323df25640 1 Processor -- start 2023-10-19T12:57:50.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.328+0000 7f323df25640 1 -- start start 2023-10-19T12:57:50.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.329+0000 7f323df25640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f323806ae40 0x7f3238152040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:50.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.329+0000 7f323df25640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f32381579f0 0x7f3238152580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:50.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.329+0000 7f323df25640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f323815c1f0 0x7f3238159500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:50.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.329+0000 7f323df25640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3238150ea0 con 0x7f32381579f0 2023-10-19T12:57:50.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.329+0000 7f323df25640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3238151180 con 0x7f323815c1f0 2023-10-19T12:57:50.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.329+0000 7f323df25640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32380c7300 con 0x7f323806ae40 2023-10-19T12:57:50.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.329+0000 7f3236ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f32381579f0 0x7f3238152580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:50.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.329+0000 7f3236ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f32381579f0 0x7f3238152580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43764/0 (socket says 172.21.15.57:43764) 2023-10-19T12:57:50.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.329+0000 7f3236ffd640 1 -- 172.21.15.57:0/4083817753 learned_addr learned my addr 172.21.15.57:0/4083817753 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:57:50.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.329+0000 7f32377fe640 1 --2- 172.21.15.57:0/4083817753 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f323806ae40 0x7f3238152040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:50.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.329+0000 7f3237fff640 1 --2- 172.21.15.57:0/4083817753 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f323815c1f0 0x7f3238159500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:50.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.330+0000 7f3236ffd640 1 -- 172.21.15.57:0/4083817753 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f323806ae40 msgr2=0x7f3238152040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:50.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.330+0000 7f3236ffd640 1 --2- 172.21.15.57:0/4083817753 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f323806ae40 0x7f3238152040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:50.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.330+0000 7f3236ffd640 1 -- 172.21.15.57:0/4083817753 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f323815c1f0 msgr2=0x7f3238159500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:50.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.330+0000 7f3236ffd640 1 --2- 172.21.15.57:0/4083817753 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f323815c1f0 0x7f3238159500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:50.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.330+0000 7f3236ffd640 1 -- 172.21.15.57:0/4083817753 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3228009490 con 0x7f32381579f0 2023-10-19T12:57:50.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.330+0000 7f3237fff640 1 --2- 172.21.15.57:0/4083817753 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f323815c1f0 0x7f3238159500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:57:50.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.330+0000 7f3236ffd640 1 --2- 172.21.15.57:0/4083817753 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f32381579f0 0x7f3238152580 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f322000c520 tx=0x7f322000c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:50.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.330+0000 7f322ffff640 1 -- 172.21.15.57:0/4083817753 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3220012830 con 0x7f32381579f0 2023-10-19T12:57:50.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.330+0000 7f322ffff640 1 -- 172.21.15.57:0/4083817753 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f3220012aa0 con 0x7f32381579f0 2023-10-19T12:57:50.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.330+0000 7f323df25640 1 -- 172.21.15.57:0/4083817753 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32380c75e0 con 0x7f32381579f0 2023-10-19T12:57:50.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.330+0000 7f322ffff640 1 -- 172.21.15.57:0/4083817753 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3220011380 con 0x7f32381579f0 2023-10-19T12:57:50.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.330+0000 7f323df25640 1 -- 172.21.15.57:0/4083817753 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32381e3640 con 0x7f32381579f0 2023-10-19T12:57:50.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.331+0000 7f323df25640 1 -- 172.21.15.57:0/4083817753 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32381e3860 con 0x7f32381579f0 2023-10-19T12:57:50.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.331+0000 7f322ffff640 1 -- 172.21.15.57:0/4083817753 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 20) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f32200114e0 con 0x7f32381579f0 2023-10-19T12:57:50.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.332+0000 7f322ffff640 1 --2- 172.21.15.57:0/4083817753 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f321806af60 0x7f321806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:50.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.332+0000 7f322ffff640 1 -- 172.21.15.57:0/4083817753 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5908+0+0 (secure 0 0 0) 0x7f3220089a70 con 0x7f32381579f0 2023-10-19T12:57:50.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.332+0000 7f32377fe640 1 --2- 172.21.15.57:0/4083817753 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f321806af60 0x7f321806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:50.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.333+0000 7f32377fe640 1 --2- 172.21.15.57:0/4083817753 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f321806af60 0x7f321806d420 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f3228036000 tx=0x7f3228004730 comp rx=0 tx=0).ready entity=mgr.5809 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:50.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.336+0000 7f322ffff640 1 -- 172.21.15.57:0/4083817753 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f32381e3860 con 0x7f32381579f0 2023-10-19T12:57:50.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.513+0000 7f323df25640 1 -- 172.21.15.57:0/4083817753 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "config reset", "num": 30} v 0) v1 -- 0x7f32381e3860 con 0x7f32381579f0 2023-10-19T12:57:50.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.516+0000 7f322ffff640 1 -- 172.21.15.57:0/4083817753 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 30}]=0 v34) v1 ==== 71+0+0 (secure 0 0 0) 0x7f32381e3860 con 0x7f32381579f0 2023-10-19T12:57:50.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.518+0000 7f323df25640 1 -- 172.21.15.57:0/4083817753 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f321806af60 msgr2=0x7f321806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:50.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.518+0000 7f323df25640 1 --2- 172.21.15.57:0/4083817753 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f321806af60 0x7f321806d420 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f3228036000 tx=0x7f3228004730 comp rx=0 tx=0).stop 2023-10-19T12:57:50.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.518+0000 7f323df25640 1 -- 172.21.15.57:0/4083817753 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f32381579f0 msgr2=0x7f3238152580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:50.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.518+0000 7f323df25640 1 --2- 172.21.15.57:0/4083817753 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f32381579f0 0x7f3238152580 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f322000c520 tx=0x7f322000c9e0 comp rx=0 tx=0).stop 2023-10-19T12:57:50.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.518+0000 7f323df25640 1 -- 172.21.15.57:0/4083817753 shutdown_connections 2023-10-19T12:57:50.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.518+0000 7f323df25640 1 --2- 172.21.15.57:0/4083817753 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f321806af60 0x7f321806d420 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:50.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.518+0000 7f323df25640 1 --2- 172.21.15.57:0/4083817753 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f323815c1f0 0x7f3238159500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:50.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.518+0000 7f323df25640 1 --2- 172.21.15.57:0/4083817753 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f32381579f0 0x7f3238152580 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:50.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.518+0000 7f323df25640 1 --2- 172.21.15.57:0/4083817753 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f323806ae40 0x7f3238152040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:50.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.518+0000 7f323df25640 1 -- 172.21.15.57:0/4083817753 >> 172.21.15.57:0/4083817753 conn(0x7f323806fae0 msgr2=0x7f3238152fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:50.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.518+0000 7f323df25640 1 -- 172.21.15.57:0/4083817753 shutdown_connections 2023-10-19T12:57:50.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.518+0000 7f323df25640 1 -- 172.21.15.57:0/4083817753 wait complete. 2023-10-19T12:57:50.532 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates' 2023-10-19T12:57:50.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.639+0000 7ff15d94c640 1 -- 172.21.15.57:0/2985873712 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff144003f20 msgr2=0x7ff1440047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:50.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.639+0000 7ff15d94c640 1 --2- 172.21.15.57:0/2985873712 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff144003f20 0x7ff1440047c0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7ff140006fd0 tx=0x7ff1440057f0 comp rx=0 tx=0).stop 2023-10-19T12:57:50.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.639+0000 7ff1577fe640 1 -- 172.21.15.57:0/2985873712 reap_dead start 2023-10-19T12:57:50.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.639+0000 7ff15d94c640 1 -- 172.21.15.57:0/2985873712 shutdown_connections 2023-10-19T12:57:50.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.639+0000 7ff15d94c640 1 -- 172.21.15.57:0/2985873712 >> 172.21.15.57:0/2985873712 conn(0x7ff158069930 msgr2=0x7ff158069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:50.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.639+0000 7ff15d94c640 1 -- 172.21.15.57:0/2985873712 shutdown_connections 2023-10-19T12:57:50.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.639+0000 7ff15d94c640 1 -- 172.21.15.57:0/2985873712 wait complete. 2023-10-19T12:57:50.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.640+0000 7ff15d94c640 1 Processor -- start 2023-10-19T12:57:50.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.640+0000 7ff15d94c640 1 -- start start 2023-10-19T12:57:50.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.641+0000 7ff15d94c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff144003f20 0x7ff158155170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:50.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.641+0000 7ff15d94c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff1581ef790 0x7ff15813ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:50.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.641+0000 7ff15d94c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff158157a70 0x7ff158157ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:50.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.641+0000 7ff15d94c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1580aaf00 con 0x7ff1581ef790 2023-10-19T12:57:50.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.641+0000 7ff15d94c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff158155830 con 0x7ff158157a70 2023-10-19T12:57:50.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.641+0000 7ff15d94c640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff158155b10 con 0x7ff144003f20 2023-10-19T12:57:50.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.641+0000 7ff14ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff1581ef790 0x7ff15813ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:50.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.641+0000 7ff14ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff1581ef790 0x7ff15813ae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43788/0 (socket says 172.21.15.57:43788) 2023-10-19T12:57:50.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.641+0000 7ff14ffff640 1 -- 172.21.15.57:0/1456486138 learned_addr learned my addr 172.21.15.57:0/1456486138 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:57:50.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.641+0000 7ff1577fe640 1 --2- 172.21.15.57:0/1456486138 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff158157a70 0x7ff158157ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:50.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.641+0000 7ff156ffd640 1 --2- 172.21.15.57:0/1456486138 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff144003f20 0x7ff158155170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:50.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.641+0000 7ff14ffff640 1 -- 172.21.15.57:0/1456486138 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff144003f20 msgr2=0x7ff158155170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:50.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.641+0000 7ff14ffff640 1 --2- 172.21.15.57:0/1456486138 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff144003f20 0x7ff158155170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:50.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.641+0000 7ff14ffff640 1 -- 172.21.15.57:0/1456486138 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff158157a70 msgr2=0x7ff158157ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:50.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.641+0000 7ff14ffff640 1 --2- 172.21.15.57:0/1456486138 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff158157a70 0x7ff158157ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:50.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.641+0000 7ff14ffff640 1 -- 172.21.15.57:0/1456486138 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff158159a20 con 0x7ff1581ef790 2023-10-19T12:57:50.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.641+0000 7ff1577fe640 1 --2- 172.21.15.57:0/1456486138 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff158157a70 0x7ff158157ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:57:50.650 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.642+0000 7ff14ffff640 1 --2- 172.21.15.57:0/1456486138 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff1581ef790 0x7ff15813ae50 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7ff14002c900 tx=0x7ff1400016e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:50.650 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.642+0000 7ff154ff9640 1 -- 172.21.15.57:0/1456486138 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff140006230 con 0x7ff1581ef790 2023-10-19T12:57:50.650 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.642+0000 7ff154ff9640 1 -- 172.21.15.57:0/1456486138 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7ff140006390 con 0x7ff1581ef790 2023-10-19T12:57:50.650 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.642+0000 7ff15d94c640 1 -- 172.21.15.57:0/1456486138 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff158155df0 con 0x7ff1581ef790 2023-10-19T12:57:50.650 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.642+0000 7ff154ff9640 1 -- 172.21.15.57:0/1456486138 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1400035f0 con 0x7ff1581ef790 2023-10-19T12:57:50.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.642+0000 7ff15d94c640 1 -- 172.21.15.57:0/1456486138 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1581563b0 con 0x7ff1581ef790 2023-10-19T12:57:50.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.643+0000 7ff15d94c640 1 -- 172.21.15.57:0/1456486138 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff158156540 con 0x7ff1581ef790 2023-10-19T12:57:50.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.643+0000 7ff154ff9640 1 -- 172.21.15.57:0/1456486138 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 20) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff140031030 con 0x7ff1581ef790 2023-10-19T12:57:50.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.644+0000 7ff154ff9640 1 --2- 172.21.15.57:0/1456486138 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7ff13806af90 0x7ff13806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:50.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.644+0000 7ff154ff9640 1 -- 172.21.15.57:0/1456486138 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5908+0+0 (secure 0 0 0) 0x7ff1400ab3a0 con 0x7ff1581ef790 2023-10-19T12:57:50.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.644+0000 7ff156ffd640 1 --2- 172.21.15.57:0/1456486138 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7ff13806af90 0x7ff13806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:50.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.645+0000 7ff156ffd640 1 --2- 172.21.15.57:0/1456486138 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7ff13806af90 0x7ff13806d450 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7ff148009ab0 tx=0x7ff148005f90 comp rx=0 tx=0).ready entity=mgr.5809 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:50.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.648+0000 7ff154ff9640 1 -- 172.21.15.57:0/1456486138 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff140076160 con 0x7ff1581ef790 2023-10-19T12:57:50.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:50.896+0000 7ff15d94c640 1 -- 172.21.15.57:0/1456486138 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates"]} v 0) v1 -- 0x7ff158156540 con 0x7ff1581ef790 2023-10-19T12:57:51.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.192+0000 7ff154ff9640 1 -- 172.21.15.57:0/1456486138 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates"]}]=0 v493) v1 ==== 158+0+0 (secure 0 0 0) 0x7ff140079780 con 0x7ff1581ef790 2023-10-19T12:57:51.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.194+0000 7ff15d94c640 1 -- 172.21.15.57:0/1456486138 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7ff13806af90 msgr2=0x7ff13806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:51.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.194+0000 7ff15d94c640 1 --2- 172.21.15.57:0/1456486138 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7ff13806af90 0x7ff13806d450 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7ff148009ab0 tx=0x7ff148005f90 comp rx=0 tx=0).stop 2023-10-19T12:57:51.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.194+0000 7ff15d94c640 1 -- 172.21.15.57:0/1456486138 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff1581ef790 msgr2=0x7ff15813ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:51.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.194+0000 7ff15d94c640 1 --2- 172.21.15.57:0/1456486138 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff1581ef790 0x7ff15813ae50 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7ff14002c900 tx=0x7ff1400016e0 comp rx=0 tx=0).stop 2023-10-19T12:57:51.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.194+0000 7ff15d94c640 1 -- 172.21.15.57:0/1456486138 shutdown_connections 2023-10-19T12:57:51.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.194+0000 7ff15d94c640 1 --2- 172.21.15.57:0/1456486138 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7ff13806af90 0x7ff13806d450 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:51.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.194+0000 7ff15d94c640 1 --2- 172.21.15.57:0/1456486138 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff158157a70 0x7ff158157ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:51.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.194+0000 7ff15d94c640 1 --2- 172.21.15.57:0/1456486138 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff1581ef790 0x7ff15813ae50 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:51.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.194+0000 7ff15d94c640 1 --2- 172.21.15.57:0/1456486138 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff144003f20 0x7ff158155170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:51.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.194+0000 7ff15d94c640 1 -- 172.21.15.57:0/1456486138 >> 172.21.15.57:0/1456486138 conn(0x7ff158069930 msgr2=0x7ff15815a3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:51.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.195+0000 7ff15d94c640 1 -- 172.21.15.57:0/1456486138 shutdown_connections 2023-10-19T12:57:51.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.195+0000 7ff15d94c640 1 -- 172.21.15.57:0/1456486138 wait complete. 2023-10-19T12:57:51.209 INFO:tasks.cephfs_test_runner:Test concurrent snap creates in same file-system without db issues ... ok 2023-10-19T12:57:51.209 INFO:tasks.cephfs_test_runner:Starting test: test_multi_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test exisitence of multiple scheduled snapshots 2023-10-19T12:57:51.210 INFO:tasks.cephfs_test_runner:test_multi_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-10-19T12:57:51.210 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T12:57:51.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.319+0000 7f75ec61f640 1 -- 172.21.15.57:0/527962048 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f75cc0047c0 msgr2=0x7f75cc004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:51.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.319+0000 7f75ec61f640 1 --2- 172.21.15.57:0/527962048 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f75cc0047c0 0x7f75cc004c40 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f75d800c260 tx=0x7f75cc0057f0 comp rx=0 tx=0).stop 2023-10-19T12:57:51.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.319+0000 7f75eabbc640 1 -- 172.21.15.57:0/527962048 reap_dead start 2023-10-19T12:57:51.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.319+0000 7f75ec61f640 1 -- 172.21.15.57:0/527962048 shutdown_connections 2023-10-19T12:57:51.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.319+0000 7f75ec61f640 1 -- 172.21.15.57:0/527962048 >> 172.21.15.57:0/527962048 conn(0x7f75e40b0670 msgr2=0x7f75e40b0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:51.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.319+0000 7f75ec61f640 1 -- 172.21.15.57:0/527962048 shutdown_connections 2023-10-19T12:57:51.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.319+0000 7f75ec61f640 1 -- 172.21.15.57:0/527962048 wait complete. 2023-10-19T12:57:51.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.321+0000 7f75ec61f640 1 Processor -- start 2023-10-19T12:57:51.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.321+0000 7f75ec61f640 1 -- start start 2023-10-19T12:57:51.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.321+0000 7f75ec61f640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f75cc0047c0 0x7f75e4151b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:51.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.321+0000 7f75ec61f640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f75e414fb40 0x7f75e40cb690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:51.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.321+0000 7f75ec61f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f75e41510e0 0x7f75e40c4970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:51.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.321+0000 7f75ec61f640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75e4150cc0 con 0x7f75e41510e0 2023-10-19T12:57:51.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.321+0000 7f75ec61f640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75e41537d0 con 0x7f75e414fb40 2023-10-19T12:57:51.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.321+0000 7f75ec61f640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75e40cbbd0 con 0x7f75cc0047c0 2023-10-19T12:57:51.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.321+0000 7f75eabbc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f75e41510e0 0x7f75e40c4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:51.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.321+0000 7f75eabbc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f75e41510e0 0x7f75e40c4970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43810/0 (socket says 172.21.15.57:43810) 2023-10-19T12:57:51.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.321+0000 7f75eabbc640 1 -- 172.21.15.57:0/512526705 learned_addr learned my addr 172.21.15.57:0/512526705 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:57:51.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.322+0000 7f75ea3bb640 1 --2- 172.21.15.57:0/512526705 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f75cc0047c0 0x7f75e4151b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:51.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.322+0000 7f75e9bba640 1 --2- 172.21.15.57:0/512526705 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f75e414fb40 0x7f75e40cb690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:51.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.322+0000 7f75eabbc640 1 -- 172.21.15.57:0/512526705 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f75cc0047c0 msgr2=0x7f75e4151b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:51.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.322+0000 7f75eabbc640 1 --2- 172.21.15.57:0/512526705 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f75cc0047c0 0x7f75e4151b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:51.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.322+0000 7f75eabbc640 1 -- 172.21.15.57:0/512526705 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f75e414fb40 msgr2=0x7f75e40cb690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:51.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.322+0000 7f75eabbc640 1 --2- 172.21.15.57:0/512526705 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f75e414fb40 0x7f75e40cb690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:51.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.322+0000 7f75eabbc640 1 -- 172.21.15.57:0/512526705 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75e4152e80 con 0x7f75e41510e0 2023-10-19T12:57:51.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.322+0000 7f75e9bba640 1 --2- 172.21.15.57:0/512526705 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f75e414fb40 0x7f75e40cb690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:57:51.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.322+0000 7f75eabbc640 1 --2- 172.21.15.57:0/512526705 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f75e41510e0 0x7f75e40c4970 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f75e4070890 tx=0x7f75dc011b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:51.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.322+0000 7f75e2ffd640 1 -- 172.21.15.57:0/512526705 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75dc004240 con 0x7f75e41510e0 2023-10-19T12:57:51.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.322+0000 7f75e2ffd640 1 -- 172.21.15.57:0/512526705 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f75dc004490 con 0x7f75e41510e0 2023-10-19T12:57:51.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.322+0000 7f75e2ffd640 1 -- 172.21.15.57:0/512526705 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75dc00fa90 con 0x7f75e41510e0 2023-10-19T12:57:51.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.322+0000 7f75ec61f640 1 -- 172.21.15.57:0/512526705 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75e40cbde0 con 0x7f75e41510e0 2023-10-19T12:57:51.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.322+0000 7f75ec61f640 1 -- 172.21.15.57:0/512526705 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75e41508e0 con 0x7f75e41510e0 2023-10-19T12:57:51.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.323+0000 7f75ec61f640 1 -- 172.21.15.57:0/512526705 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75e40ce1d0 con 0x7f75e41510e0 2023-10-19T12:57:51.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.323+0000 7f75e2ffd640 1 -- 172.21.15.57:0/512526705 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 20) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f75dc018070 con 0x7f75e41510e0 2023-10-19T12:57:51.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.324+0000 7f75e2ffd640 1 --2- 172.21.15.57:0/512526705 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f75d006af90 0x7f75d006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:51.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.324+0000 7f75e2ffd640 1 -- 172.21.15.57:0/512526705 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5908+0+0 (secure 0 0 0) 0x7f75dc08a160 con 0x7f75e41510e0 2023-10-19T12:57:51.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.324+0000 7f75ea3bb640 1 --2- 172.21.15.57:0/512526705 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f75d006af90 0x7f75d006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:51.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.325+0000 7f75ea3bb640 1 --2- 172.21.15.57:0/512526705 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f75d006af90 0x7f75d006d450 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f75d4005fd0 tx=0x7f75d4009500 comp rx=0 tx=0).ready entity=mgr.5809 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:51.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.328+0000 7f75e2ffd640 1 -- 172.21.15.57:0/512526705 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f75dc0550c0 con 0x7f75e41510e0 2023-10-19T12:57:51.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.506+0000 7f75ec61f640 1 -- 172.21.15.57:0/512526705 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f75e4150e00 con 0x7f75e41510e0 2023-10-19T12:57:51.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.507+0000 7f75e2ffd640 1 -- 172.21.15.57:0/512526705 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v62) v1 ==== 74+0+14158 (secure 0 0 0) 0x7f75e4150e00 con 0x7f75e41510e0 2023-10-19T12:57:51.510 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:57:51.510 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":62,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T12:57:50.182182+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.179:6835/1237463333":"2023-10-20T12:57:47.096461+0000","172.21.15.179:0/2997987784":"2023-10-20T12:57:44.608940+0000","172.21.15.179:6837/1237463333":"2023-10-20T12:57:47.096461+0000","172.21.15.179:0/2313691653":"2023-10-20T12:45:56.327729+0000","172.21.15.179:0/2223195228":"2023-10-20T12:45:56.327729+0000","172.21.15.179:0/515590766":"2023-10-20T12:45:56.327729+0000","172.21.15.179:0/4235058089":"2023-10-20T12:57:44.608940+0000","172.21.15.179:0/1377585784":"2023-10-20T12:45:56.327729+0000","172.21.15.179:0/3471628125":"2023-10-20T12:57:44.608940+0000","172.21.15.179:6832/2609646285":"2023-10-20T12:45:56.327729+0000","172.21.15.179:6833/2609646285":"2023-10-20T12:45:56.327729+0000","172.21.15.179:6832/922398470":"2023-10-20T12:57:44.608940+0000","172.21.15.179:6833/922398470":"2023-10-20T12:57:44.608940+0000","172.21.15.179:0/2754375794":"2023-10-20T12:57:44.608940+0000","172.21.15.179:0/161411072":"2023-10-20T12:57:44.608940+0000","172.21.15.179:0/4022258517":"2023-10-20T12:57:44.608940+0000","172.21.15.179:0/3663257206":"2023-10-20T12:57:44.608940+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T12:57:51.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.509+0000 7f75ec61f640 1 -- 172.21.15.57:0/512526705 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f75d006af90 msgr2=0x7f75d006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:51.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.509+0000 7f75ec61f640 1 --2- 172.21.15.57:0/512526705 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f75d006af90 0x7f75d006d450 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f75d4005fd0 tx=0x7f75d4009500 comp rx=0 tx=0).stop 2023-10-19T12:57:51.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.509+0000 7f75ec61f640 1 -- 172.21.15.57:0/512526705 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f75e41510e0 msgr2=0x7f75e40c4970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:51.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.509+0000 7f75ec61f640 1 --2- 172.21.15.57:0/512526705 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f75e41510e0 0x7f75e40c4970 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f75e4070890 tx=0x7f75dc011b30 comp rx=0 tx=0).stop 2023-10-19T12:57:51.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.509+0000 7f75ec61f640 1 -- 172.21.15.57:0/512526705 shutdown_connections 2023-10-19T12:57:51.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.509+0000 7f75ec61f640 1 --2- 172.21.15.57:0/512526705 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f75d006af90 0x7f75d006d450 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:51.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.509+0000 7f75ec61f640 1 --2- 172.21.15.57:0/512526705 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f75e41510e0 0x7f75e40c4970 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:51.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.509+0000 7f75ec61f640 1 --2- 172.21.15.57:0/512526705 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f75e414fb40 0x7f75e40cb690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:51.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.509+0000 7f75ec61f640 1 --2- 172.21.15.57:0/512526705 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f75cc0047c0 0x7f75e4151b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:51.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.509+0000 7f75ec61f640 1 -- 172.21.15.57:0/512526705 >> 172.21.15.57:0/512526705 conn(0x7f75e40b0670 msgr2=0x7f75e40cc690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:51.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.509+0000 7f75ec61f640 1 -- 172.21.15.57:0/512526705 shutdown_connections 2023-10-19T12:57:51.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.509+0000 7f75ec61f640 1 -- 172.21.15.57:0/512526705 wait complete. 2023-10-19T12:57:51.523 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-19T12:44:07.371186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-19T12:57:51.524 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-19T12:57:51.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.633+0000 7f273bef7640 1 -- 172.21.15.57:0/1777354953 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f27340cd6d0 msgr2=0x7f27340cdb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:51.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.633+0000 7f273bef7640 1 --2- 172.21.15.57:0/1777354953 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f27340cd6d0 0x7f27340cdb30 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f2734071e10 tx=0x7f272402bd30 comp rx=0 tx=0).stop 2023-10-19T12:57:51.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.633+0000 7f273bef7640 1 -- 172.21.15.57:0/1777354953 shutdown_connections 2023-10-19T12:57:51.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.633+0000 7f273bef7640 1 --2- 172.21.15.57:0/1777354953 >> v2:172.21.15.179:3300/0 conn(0x7f2734069720 0x7f2734069bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:51.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.633+0000 7f273bef7640 1 --2- 172.21.15.57:0/1777354953 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f27340cd6d0 0x7f27340cdb30 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:51.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.633+0000 7f273bef7640 1 -- 172.21.15.57:0/1777354953 >> 172.21.15.57:0/1777354953 conn(0x7f27340af7a0 msgr2=0x7f27340afb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:51.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.633+0000 7f273bef7640 1 -- 172.21.15.57:0/1777354953 shutdown_connections 2023-10-19T12:57:51.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.633+0000 7f273bef7640 1 -- 172.21.15.57:0/1777354953 wait complete. 2023-10-19T12:57:51.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.634+0000 7f273bef7640 1 Processor -- start 2023-10-19T12:57:51.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.634+0000 7f273bef7640 1 -- start start 2023-10-19T12:57:51.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.635+0000 7f273bef7640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2734069720 0x7f27340a6640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:51.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.635+0000 7f273bef7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f27340cd6d0 0x7f27341efbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:51.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.635+0000 7f273bef7640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2734142850 0x7f2734070110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:51.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.635+0000 7f273bef7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27340a9c90 con 0x7f27340cd6d0 2023-10-19T12:57:51.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.635+0000 7f273bef7640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27340acc80 con 0x7f2734069720 2023-10-19T12:57:51.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.635+0000 7f273bef7640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27341ef110 con 0x7f2734142850 2023-10-19T12:57:51.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.635+0000 7f2739492640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f27340cd6d0 0x7f27341efbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:51.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.635+0000 7f2739492640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f27340cd6d0 0x7f27341efbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43834/0 (socket says 172.21.15.57:43834) 2023-10-19T12:57:51.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.635+0000 7f2739492640 1 -- 172.21.15.57:0/1347043228 learned_addr learned my addr 172.21.15.57:0/1347043228 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:57:51.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.635+0000 7f2739c93640 1 --2- 172.21.15.57:0/1347043228 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2734069720 0x7f27340a6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:51.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.635+0000 7f273a494640 1 --2- 172.21.15.57:0/1347043228 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2734142850 0x7f2734070110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:51.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.635+0000 7f2739492640 1 -- 172.21.15.57:0/1347043228 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2734142850 msgr2=0x7f2734070110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:51.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.635+0000 7f2739492640 1 --2- 172.21.15.57:0/1347043228 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2734142850 0x7f2734070110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:51.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.635+0000 7f2739492640 1 -- 172.21.15.57:0/1347043228 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2734069720 msgr2=0x7f27340a6640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:51.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.635+0000 7f2739492640 1 --2- 172.21.15.57:0/1347043228 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2734069720 0x7f27340a6640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:51.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.635+0000 7f2739492640 1 -- 172.21.15.57:0/1347043228 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2724009560 con 0x7f27340cd6d0 2023-10-19T12:57:51.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.635+0000 7f273a494640 1 --2- 172.21.15.57:0/1347043228 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2734142850 0x7f2734070110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:57:51.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.635+0000 7f2739492640 1 --2- 172.21.15.57:0/1347043228 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f27340cd6d0 0x7f27341efbd0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f2724009850 tx=0x7f272402dd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:51.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.635+0000 7f272a7fc640 1 -- 172.21.15.57:0/1347043228 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2724005650 con 0x7f27340cd6d0 2023-10-19T12:57:51.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.635+0000 7f272a7fc640 1 -- 172.21.15.57:0/1347043228 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f27240038f0 con 0x7f27340cd6d0 2023-10-19T12:57:51.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.635+0000 7f273bef7640 1 -- 172.21.15.57:0/1347043228 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27340a6b80 con 0x7f27340cd6d0 2023-10-19T12:57:51.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.635+0000 7f272a7fc640 1 -- 172.21.15.57:0/1347043228 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2724003ce0 con 0x7f27340cd6d0 2023-10-19T12:57:51.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.635+0000 7f273bef7640 1 -- 172.21.15.57:0/1347043228 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27340ad220 con 0x7f27340cd6d0 2023-10-19T12:57:51.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.636+0000 7f273bef7640 1 -- 172.21.15.57:0/1347043228 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27340690e0 con 0x7f27340cd6d0 2023-10-19T12:57:51.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.637+0000 7f272a7fc640 1 -- 172.21.15.57:0/1347043228 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 20) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f272403e450 con 0x7f27340cd6d0 2023-10-19T12:57:51.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.637+0000 7f272a7fc640 1 --2- 172.21.15.57:0/1347043228 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f271806af60 0x7f271806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:51.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.637+0000 7f272a7fc640 1 -- 172.21.15.57:0/1347043228 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5908+0+0 (secure 0 0 0) 0x7f27240ac150 con 0x7f27340cd6d0 2023-10-19T12:57:51.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.638+0000 7f2739c93640 1 --2- 172.21.15.57:0/1347043228 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f271806af60 0x7f271806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:51.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.638+0000 7f2739c93640 1 --2- 172.21.15.57:0/1347043228 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f271806af60 0x7f271806d420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f2730006fd0 tx=0x7f2730009060 comp rx=0 tx=0).ready entity=mgr.5809 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:51.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.641+0000 7f272a7fc640 1 -- 172.21.15.57:0/1347043228 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2724077130 con 0x7f27340cd6d0 2023-10-19T12:57:51.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.817+0000 7f273bef7640 1 -- 172.21.15.57:0/1347043228 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f273415c5e0 con 0x7f27340cd6d0 2023-10-19T12:57:51.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.817+0000 7f272a7fc640 1 -- 172.21.15.57:0/1347043228 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v62) v1 ==== 93+0+10 (secure 0 0 0) 0x7f272407a750 con 0x7f27340cd6d0 2023-10-19T12:57:51.820 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-19T12:57:51.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.819+0000 7f273bef7640 1 -- 172.21.15.57:0/1347043228 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f271806af60 msgr2=0x7f271806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:51.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.819+0000 7f273bef7640 1 --2- 172.21.15.57:0/1347043228 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f271806af60 0x7f271806d420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f2730006fd0 tx=0x7f2730009060 comp rx=0 tx=0).stop 2023-10-19T12:57:51.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.819+0000 7f273bef7640 1 -- 172.21.15.57:0/1347043228 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f27340cd6d0 msgr2=0x7f27341efbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:51.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.819+0000 7f273bef7640 1 --2- 172.21.15.57:0/1347043228 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f27340cd6d0 0x7f27341efbd0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f2724009850 tx=0x7f272402dd00 comp rx=0 tx=0).stop 2023-10-19T12:57:51.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.819+0000 7f273bef7640 1 -- 172.21.15.57:0/1347043228 shutdown_connections 2023-10-19T12:57:51.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.819+0000 7f273bef7640 1 --2- 172.21.15.57:0/1347043228 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f271806af60 0x7f271806d420 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:51.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.819+0000 7f273bef7640 1 --2- 172.21.15.57:0/1347043228 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2734142850 0x7f2734070110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:51.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.819+0000 7f273bef7640 1 --2- 172.21.15.57:0/1347043228 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f27340cd6d0 0x7f27341efbd0 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:51.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.819+0000 7f273bef7640 1 --2- 172.21.15.57:0/1347043228 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2734069720 0x7f27340a6640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:51.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.819+0000 7f273bef7640 1 -- 172.21.15.57:0/1347043228 >> 172.21.15.57:0/1347043228 conn(0x7f27340af7a0 msgr2=0x7f27340adf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:51.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.819+0000 7f273bef7640 1 -- 172.21.15.57:0/1347043228 shutdown_connections 2023-10-19T12:57:51.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.819+0000 7f273bef7640 1 -- 172.21.15.57:0/1347043228 wait complete. 2023-10-19T12:57:51.832 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule' 2023-10-19T12:57:51.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.939+0000 7f7845324640 1 -- 172.21.15.57:0/289531686 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f78400b1920 msgr2=0x7f784006fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:51.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.939+0000 7f7845324640 1 --2- 172.21.15.57:0/289531686 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f78400b1920 0x7f784006fa70 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7838005fd0 tx=0x7f783802be10 comp rx=0 tx=0).stop 2023-10-19T12:57:51.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.939+0000 7f7845324640 1 -- 172.21.15.57:0/289531686 shutdown_connections 2023-10-19T12:57:51.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.939+0000 7f7845324640 1 --2- 172.21.15.57:0/289531686 >> v2:172.21.15.179:3300/0 conn(0x7f784006ffb0 0x7f78400c96b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:51.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.939+0000 7f7845324640 1 --2- 172.21.15.57:0/289531686 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f78400b1920 0x7f784006fa70 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:51.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.939+0000 7f7845324640 1 -- 172.21.15.57:0/289531686 >> 172.21.15.57:0/289531686 conn(0x7f7840069930 msgr2=0x7f7840069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:51.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.939+0000 7f7845324640 1 -- 172.21.15.57:0/289531686 shutdown_connections 2023-10-19T12:57:51.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.939+0000 7f7845324640 1 -- 172.21.15.57:0/289531686 wait complete. 2023-10-19T12:57:51.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.941+0000 7f7845324640 1 Processor -- start 2023-10-19T12:57:51.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.941+0000 7f7845324640 1 -- start start 2023-10-19T12:57:51.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.941+0000 7f7845324640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f784006ffb0 0x7f7840149c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:51.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.941+0000 7f7845324640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f78400b1920 0x7f78401f4af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:51.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.941+0000 7f7845324640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f78400c9bf0 0x7f78401433c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:51.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.941+0000 7f7845324640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78401f5030 con 0x7f78400b1920 2023-10-19T12:57:51.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.941+0000 7f7845324640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78401f5220 con 0x7f784006ffb0 2023-10-19T12:57:51.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.941+0000 7f7845324640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78401f5500 con 0x7f78400c9bf0 2023-10-19T12:57:51.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.942+0000 7f783e7fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f78400b1920 0x7f78401f4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:51.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.942+0000 7f783e7fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f78400b1920 0x7f78401f4af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43848/0 (socket says 172.21.15.57:43848) 2023-10-19T12:57:51.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.942+0000 7f783e7fc640 1 -- 172.21.15.57:0/2807247925 learned_addr learned my addr 172.21.15.57:0/2807247925 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:57:51.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.942+0000 7f783f7fe640 1 --2- 172.21.15.57:0/2807247925 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f78400c9bf0 0x7f78401433c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:51.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.942+0000 7f783effd640 1 --2- 172.21.15.57:0/2807247925 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f784006ffb0 0x7f7840149c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:51.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.942+0000 7f783e7fc640 1 -- 172.21.15.57:0/2807247925 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f78400c9bf0 msgr2=0x7f78401433c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:51.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.942+0000 7f783e7fc640 1 --2- 172.21.15.57:0/2807247925 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f78400c9bf0 0x7f78401433c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:51.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.942+0000 7f783e7fc640 1 -- 172.21.15.57:0/2807247925 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f784006ffb0 msgr2=0x7f7840149c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:51.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.942+0000 7f783e7fc640 1 --2- 172.21.15.57:0/2807247925 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f784006ffb0 0x7f7840149c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:51.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.942+0000 7f783e7fc640 1 -- 172.21.15.57:0/2807247925 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7838009560 con 0x7f78400b1920 2023-10-19T12:57:51.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.942+0000 7f783e7fc640 1 --2- 172.21.15.57:0/2807247925 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f78400b1920 0x7f78401f4af0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f782800e790 tx=0x7f782800ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:51.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.942+0000 7f78377fe640 1 -- 172.21.15.57:0/2807247925 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7828009580 con 0x7f78400b1920 2023-10-19T12:57:51.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.942+0000 7f78377fe640 1 -- 172.21.15.57:0/2807247925 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f782800cdb0 con 0x7f78400b1920 2023-10-19T12:57:51.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.942+0000 7f7845324640 1 -- 172.21.15.57:0/2807247925 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78401f57e0 con 0x7f78400b1920 2023-10-19T12:57:51.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.942+0000 7f78377fe640 1 -- 172.21.15.57:0/2807247925 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7828010690 con 0x7f78400b1920 2023-10-19T12:57:51.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.942+0000 7f7845324640 1 -- 172.21.15.57:0/2807247925 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7840143900 con 0x7f78400b1920 2023-10-19T12:57:51.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.943+0000 7f7845324640 1 -- 172.21.15.57:0/2807247925 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78401f5a00 con 0x7f78400b1920 2023-10-19T12:57:51.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.944+0000 7f78377fe640 1 -- 172.21.15.57:0/2807247925 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 20) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f78280107f0 con 0x7f78400b1920 2023-10-19T12:57:51.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.944+0000 7f78377fe640 1 --2- 172.21.15.57:0/2807247925 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f782006af90 0x7f782006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:51.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.944+0000 7f78377fe640 1 -- 172.21.15.57:0/2807247925 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5908+0+0 (secure 0 0 0) 0x7f7828014070 con 0x7f78400b1920 2023-10-19T12:57:51.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.945+0000 7f783effd640 1 --2- 172.21.15.57:0/2807247925 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f782006af90 0x7f782006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:51.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.945+0000 7f783effd640 1 --2- 172.21.15.57:0/2807247925 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f782006af90 0x7f782006d450 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f7838035000 tx=0x7f7838004730 comp rx=0 tx=0).ready entity=mgr.5809 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:51.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:51.948+0000 7f78377fe640 1 -- 172.21.15.57:0/2807247925 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7828051ca0 con 0x7f78400b1920 2023-10-19T12:57:52.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:52.196+0000 7f7845324640 1 -- 172.21.15.57:0/2807247925 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule"]} v 0) v1 -- 0x7f784015c360 con 0x7f78400b1920 2023-10-19T12:57:53.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.200+0000 7f78377fe640 1 -- 172.21.15.57:0/2807247925 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule"]}]=0 v495) v1 ==== 157+0+0 (secure 0 0 0) 0x7f78280551d0 con 0x7f78400b1920 2023-10-19T12:57:53.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.203+0000 7f7845324640 1 -- 172.21.15.57:0/2807247925 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f782006af90 msgr2=0x7f782006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:53.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.203+0000 7f7845324640 1 --2- 172.21.15.57:0/2807247925 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f782006af90 0x7f782006d450 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f7838035000 tx=0x7f7838004730 comp rx=0 tx=0).stop 2023-10-19T12:57:53.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.203+0000 7f7845324640 1 -- 172.21.15.57:0/2807247925 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f78400b1920 msgr2=0x7f78401f4af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:53.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.203+0000 7f7845324640 1 --2- 172.21.15.57:0/2807247925 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f78400b1920 0x7f78401f4af0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f782800e790 tx=0x7f782800ec50 comp rx=0 tx=0).stop 2023-10-19T12:57:53.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.203+0000 7f7845324640 1 -- 172.21.15.57:0/2807247925 shutdown_connections 2023-10-19T12:57:53.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.203+0000 7f7845324640 1 --2- 172.21.15.57:0/2807247925 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f782006af90 0x7f782006d450 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:53.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.203+0000 7f7845324640 1 --2- 172.21.15.57:0/2807247925 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f78400c9bf0 0x7f78401433c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:53.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.203+0000 7f7845324640 1 --2- 172.21.15.57:0/2807247925 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f78400b1920 0x7f78401f4af0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:53.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.203+0000 7f7845324640 1 --2- 172.21.15.57:0/2807247925 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f784006ffb0 0x7f7840149c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:53.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.203+0000 7f7845324640 1 -- 172.21.15.57:0/2807247925 >> 172.21.15.57:0/2807247925 conn(0x7f7840069930 msgr2=0x7f7840069d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:53.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.203+0000 7f7845324640 1 -- 172.21.15.57:0/2807247925 shutdown_connections 2023-10-19T12:57:53.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.203+0000 7f7845324640 1 -- 172.21.15.57:0/2807247925 wait complete. 2023-10-19T12:57:53.219 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2023-10-19T12:57:53.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.332+0000 7fe20bb9e640 1 -- 172.21.15.57:0/1020369366 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1ec003970 msgr2=0x7fe1ec003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:53.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.332+0000 7fe20bb9e640 1 --2- 172.21.15.57:0/1020369366 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1ec003970 0x7fe1ec003d90 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fe1fc001680 tx=0x7fe1ec0057f0 comp rx=0 tx=0).stop 2023-10-19T12:57:53.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.332+0000 7fe20bb9e640 1 -- 172.21.15.57:0/1020369366 shutdown_connections 2023-10-19T12:57:53.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.332+0000 7fe20bb9e640 1 --2- 172.21.15.57:0/1020369366 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1ec003970 0x7fe1ec003d90 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:53.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.332+0000 7fe20bb9e640 1 -- 172.21.15.57:0/1020369366 >> 172.21.15.57:0/1020369366 conn(0x7fe2040d0d60 msgr2=0x7fe2040d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:53.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.332+0000 7fe20bb9e640 1 -- 172.21.15.57:0/1020369366 shutdown_connections 2023-10-19T12:57:53.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.332+0000 7fe20bb9e640 1 -- 172.21.15.57:0/1020369366 wait complete. 2023-10-19T12:57:53.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.333+0000 7fe20bb9e640 1 Processor -- start 2023-10-19T12:57:53.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.333+0000 7fe20bb9e640 1 -- start start 2023-10-19T12:57:53.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.334+0000 7fe20bb9e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1ec003970 0x7fe2041588f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:53.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.334+0000 7fe20bb9e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe204150030 0x7fe20415d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:53.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.334+0000 7fe20bb9e640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe20415dce0 0x7fe2040b2830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:53.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.334+0000 7fe20bb9e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2041586f0 con 0x7fe1ec003970 2023-10-19T12:57:53.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.334+0000 7fe20bb9e640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe20415c700 con 0x7fe204150030 2023-10-19T12:57:53.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.334+0000 7fe20bb9e640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe20415c9e0 con 0x7fe20415dce0 2023-10-19T12:57:53.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.334+0000 7fe20993a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1ec003970 0x7fe2041588f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:53.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.334+0000 7fe20993a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1ec003970 0x7fe2041588f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43872/0 (socket says 172.21.15.57:43872) 2023-10-19T12:57:53.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.334+0000 7fe20993a640 1 -- 172.21.15.57:0/2192022145 learned_addr learned my addr 172.21.15.57:0/2192022145 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:57:53.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.334+0000 7fe20993a640 1 -- 172.21.15.57:0/2192022145 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe20415dce0 msgr2=0x7fe2040b2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:53.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.334+0000 7fe20993a640 1 --2- 172.21.15.57:0/2192022145 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe20415dce0 0x7fe2040b2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:53.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.334+0000 7fe20993a640 1 -- 172.21.15.57:0/2192022145 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe204150030 msgr2=0x7fe20415d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:53.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.334+0000 7fe20993a640 1 --2- 172.21.15.57:0/2192022145 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe204150030 0x7fe20415d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:53.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.334+0000 7fe20993a640 1 -- 172.21.15.57:0/2192022145 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe20415ccc0 con 0x7fe1ec003970 2023-10-19T12:57:53.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.335+0000 7fe20993a640 1 --2- 172.21.15.57:0/2192022145 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1ec003970 0x7fe2041588f0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fe1fc02c0d0 tx=0x7fe1fc0062e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:53.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.335+0000 7fe2027fc640 1 -- 172.21.15.57:0/2192022145 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1fc006540 con 0x7fe1ec003970 2023-10-19T12:57:53.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.335+0000 7fe2027fc640 1 -- 172.21.15.57:0/2192022145 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fe1fc003bf0 con 0x7fe1ec003970 2023-10-19T12:57:53.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.335+0000 7fe2027fc640 1 -- 172.21.15.57:0/2192022145 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1fc02c530 con 0x7fe1ec003970 2023-10-19T12:57:53.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.335+0000 7fe20bb9e640 1 -- 172.21.15.57:0/2192022145 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe20415cfa0 con 0x7fe1ec003970 2023-10-19T12:57:53.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.335+0000 7fe20bb9e640 1 -- 172.21.15.57:0/2192022145 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2041504b0 con 0x7fe1ec003970 2023-10-19T12:57:53.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.336+0000 7fe20bb9e640 1 -- 172.21.15.57:0/2192022145 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2041586f0 con 0x7fe1ec003970 2023-10-19T12:57:53.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.336+0000 7fe2027fc640 1 -- 172.21.15.57:0/2192022145 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 20) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe1fc02c690 con 0x7fe1ec003970 2023-10-19T12:57:53.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.336+0000 7fe2027fc640 1 --2- 172.21.15.57:0/2192022145 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7fe1f006af60 0x7fe1f006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:53.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.337+0000 7fe2027fc640 1 -- 172.21.15.57:0/2192022145 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5908+0+0 (secure 0 0 0) 0x7fe1fc0aaac0 con 0x7fe1ec003970 2023-10-19T12:57:53.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.337+0000 7fe209139640 1 --2- 172.21.15.57:0/2192022145 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7fe1f006af60 0x7fe1f006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:53.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.337+0000 7fe209139640 1 --2- 172.21.15.57:0/2192022145 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7fe1f006af60 0x7fe1f006d420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe1f8001200 tx=0x7fe1f8001110 comp rx=0 tx=0).ready entity=mgr.5809 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:53.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.341+0000 7fe2027fc640 1 -- 172.21.15.57:0/2192022145 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe1fc075990 con 0x7fe1ec003970 2023-10-19T12:57:53.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.515+0000 7fe20bb9e640 1 -- 172.21.15.57:0/2192022145 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "config dump"} v 0) v1 -- 0x7fe2040cd8b0 con 0x7fe1ec003970 2023-10-19T12:57:53.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.515+0000 7fe2027fc640 1 -- 172.21.15.57:0/2192022145 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v34) v1 ==== 59+0+2220 (secure 0 0 0) 0x7fe1fc0aae60 con 0x7fe1ec003970 2023-10-19T12:57:53.519 INFO:teuthology.orchestra.run.smithi057.stdout:WHO MASK LEVEL OPTION VALUE RO 2023-10-19T12:57:53.519 INFO:teuthology.orchestra.run.smithi057.stdout:mon advanced mon_op_complaint_time 120 2023-10-19T12:57:53.519 INFO:teuthology.orchestra.run.smithi057.stdout:osd advanced osd_op_complaint_time 180.000000 2023-10-19T12:57:53.519 INFO:teuthology.orchestra.run.smithi057.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 4775.995373 2023-10-19T12:57:53.520 INFO:teuthology.orchestra.run.smithi057.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 4208.118410 2023-10-19T12:57:53.520 INFO:teuthology.orchestra.run.smithi057.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 4541.123030 2023-10-19T12:57:53.520 INFO:teuthology.orchestra.run.smithi057.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 4877.592206 2023-10-19T12:57:53.520 INFO:teuthology.orchestra.run.smithi057.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 5019.748032 2023-10-19T12:57:53.520 INFO:teuthology.orchestra.run.smithi057.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 5000.074376 2023-10-19T12:57:53.521 INFO:teuthology.orchestra.run.smithi057.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 5153.479439 2023-10-19T12:57:53.521 INFO:teuthology.orchestra.run.smithi057.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 4977.363845 2023-10-19T12:57:53.521 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced debug_mds 20/20 2023-10-19T12:57:53.521 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced debug_mds_balancer 20/20 2023-10-19T12:57:53.521 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced debug_ms 1/1 2023-10-19T12:57:53.521 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced mds_bal_fragment_size_max 10000 2023-10-19T12:57:53.522 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced mds_bal_merge_size 5 2023-10-19T12:57:53.522 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced mds_bal_split_bits 3 2023-10-19T12:57:53.522 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced mds_bal_split_size 100 2023-10-19T12:57:53.522 INFO:teuthology.orchestra.run.smithi057.stdout:mds dev mds_debug_frag true 2023-10-19T12:57:53.522 INFO:teuthology.orchestra.run.smithi057.stdout:mds dev mds_debug_scatterstat true 2023-10-19T12:57:53.523 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced mds_op_complaint_time 180.000000 2023-10-19T12:57:53.523 INFO:teuthology.orchestra.run.smithi057.stdout:mds dev mds_verify_scatter true 2023-10-19T12:57:53.523 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced osd_op_complaint_time 180.000000 2023-10-19T12:57:53.523 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced rados_mon_op_timeout 900 2023-10-19T12:57:53.523 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced rados_osd_op_timeout 900 2023-10-19T12:57:53.524 INFO:teuthology.orchestra.run.smithi057.stdout:client advanced client_mount_timeout 600 2023-10-19T12:57:53.524 INFO:teuthology.orchestra.run.smithi057.stdout:client advanced debug_client 20/20 2023-10-19T12:57:53.524 INFO:teuthology.orchestra.run.smithi057.stdout:client advanced debug_ms 1/1 2023-10-19T12:57:53.524 INFO:teuthology.orchestra.run.smithi057.stdout:client advanced rados_mon_op_timeout 900 2023-10-19T12:57:53.524 INFO:teuthology.orchestra.run.smithi057.stdout:client advanced rados_osd_op_timeout 900 2023-10-19T12:57:53.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.517+0000 7fe20bb9e640 1 -- 172.21.15.57:0/2192022145 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7fe1f006af60 msgr2=0x7fe1f006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:53.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.517+0000 7fe20bb9e640 1 --2- 172.21.15.57:0/2192022145 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7fe1f006af60 0x7fe1f006d420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe1f8001200 tx=0x7fe1f8001110 comp rx=0 tx=0).stop 2023-10-19T12:57:53.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.517+0000 7fe20bb9e640 1 -- 172.21.15.57:0/2192022145 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1ec003970 msgr2=0x7fe2041588f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:53.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.517+0000 7fe20bb9e640 1 --2- 172.21.15.57:0/2192022145 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1ec003970 0x7fe2041588f0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fe1fc02c0d0 tx=0x7fe1fc0062e0 comp rx=0 tx=0).stop 2023-10-19T12:57:53.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.517+0000 7fe20bb9e640 1 -- 172.21.15.57:0/2192022145 shutdown_connections 2023-10-19T12:57:53.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.517+0000 7fe20bb9e640 1 --2- 172.21.15.57:0/2192022145 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7fe1f006af60 0x7fe1f006d420 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:53.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.517+0000 7fe20bb9e640 1 --2- 172.21.15.57:0/2192022145 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe20415dce0 0x7fe2040b2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:53.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.517+0000 7fe20bb9e640 1 --2- 172.21.15.57:0/2192022145 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe204150030 0x7fe20415d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:53.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.517+0000 7fe20bb9e640 1 --2- 172.21.15.57:0/2192022145 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1ec003970 0x7fe2041588f0 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:53.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.517+0000 7fe20bb9e640 1 -- 172.21.15.57:0/2192022145 >> 172.21.15.57:0/2192022145 conn(0x7fe2040d0d60 msgr2=0x7fe2040ae2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:53.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.517+0000 7fe20bb9e640 1 -- 172.21.15.57:0/2192022145 shutdown_connections 2023-10-19T12:57:53.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.517+0000 7fe20bb9e640 1 -- 172.21.15.57:0/2192022145 wait complete. 2023-10-19T12:57:53.530 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-19T12:57:53.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.640+0000 7f7af57e2640 1 -- 172.21.15.57:0/1941285156 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7af00ab950 msgr2=0x7f7af00abde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:53.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.640+0000 7f7af57e2640 1 --2- 172.21.15.57:0/1941285156 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7af00ab950 0x7f7af00abde0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f7af0066ef0 tx=0x7f7ae002c250 comp rx=0 tx=0).stop 2023-10-19T12:57:53.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.640+0000 7f7af57e2640 1 -- 172.21.15.57:0/1941285156 shutdown_connections 2023-10-19T12:57:53.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.640+0000 7f7af57e2640 1 --2- 172.21.15.57:0/1941285156 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7af00ab950 0x7f7af00abde0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:53.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.640+0000 7f7af57e2640 1 --2- 172.21.15.57:0/1941285156 >> v2:172.21.15.179:3300/0 conn(0x7f7af006f0a0 0x7f7af006dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:53.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.640+0000 7f7af57e2640 1 --2- 172.21.15.57:0/1941285156 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7af006d2d0 0x7f7af006d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:53.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.640+0000 7f7af57e2640 1 -- 172.21.15.57:0/1941285156 >> 172.21.15.57:0/1941285156 conn(0x7f7af00b0670 msgr2=0x7f7af00b0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:53.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.640+0000 7f7af57e2640 1 -- 172.21.15.57:0/1941285156 shutdown_connections 2023-10-19T12:57:53.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.640+0000 7f7af57e2640 1 -- 172.21.15.57:0/1941285156 wait complete. 2023-10-19T12:57:53.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.641+0000 7f7af57e2640 1 Processor -- start 2023-10-19T12:57:53.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.641+0000 7f7af57e2640 1 -- start start 2023-10-19T12:57:53.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.642+0000 7f7af57e2640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7af006d2d0 0x7f7af01569f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:53.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.642+0000 7f7af57e2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7af006f0a0 0x7f7af01ef7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:53.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.642+0000 7f7af57e2640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7af00ab950 0x7f7af014f960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:53.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.642+0000 7f7af57e2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7af0154310 con 0x7f7af006f0a0 2023-10-19T12:57:53.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.642+0000 7f7af57e2640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7af01545f0 con 0x7f7af006d2d0 2023-10-19T12:57:53.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.642+0000 7f7af57e2640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7af01548d0 con 0x7f7af00ab950 2023-10-19T12:57:53.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.642+0000 7f7aee7fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7af006f0a0 0x7f7af01ef7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:53.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.642+0000 7f7aee7fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7af006f0a0 0x7f7af01ef7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43886/0 (socket says 172.21.15.57:43886) 2023-10-19T12:57:53.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.642+0000 7f7aee7fc640 1 -- 172.21.15.57:0/2610829277 learned_addr learned my addr 172.21.15.57:0/2610829277 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:57:53.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.642+0000 7f7aeeffd640 1 --2- 172.21.15.57:0/2610829277 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7af006d2d0 0x7f7af01569f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:53.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.642+0000 7f7aef7fe640 1 --2- 172.21.15.57:0/2610829277 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7af00ab950 0x7f7af014f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:53.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.642+0000 7f7aee7fc640 1 -- 172.21.15.57:0/2610829277 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7af00ab950 msgr2=0x7f7af014f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:53.650 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.642+0000 7f7aee7fc640 1 --2- 172.21.15.57:0/2610829277 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7af00ab950 0x7f7af014f960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:53.650 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.642+0000 7f7aee7fc640 1 -- 172.21.15.57:0/2610829277 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7af006d2d0 msgr2=0x7f7af01569f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:53.650 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.642+0000 7f7aee7fc640 1 --2- 172.21.15.57:0/2610829277 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7af006d2d0 0x7f7af01569f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:53.650 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.642+0000 7f7aee7fc640 1 -- 172.21.15.57:0/2610829277 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ae0009900 con 0x7f7af006f0a0 2023-10-19T12:57:53.650 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.642+0000 7f7aeeffd640 1 --2- 172.21.15.57:0/2610829277 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7af006d2d0 0x7f7af01569f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:57:53.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.642+0000 7f7aef7fe640 1 --2- 172.21.15.57:0/2610829277 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7af00ab950 0x7f7af014f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:57:53.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.642+0000 7f7aee7fc640 1 --2- 172.21.15.57:0/2610829277 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7af006f0a0 0x7f7af01ef7f0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f7ae400d950 tx=0x7f7ae400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:53.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.643+0000 7f7ad77fe640 1 -- 172.21.15.57:0/2610829277 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ae4009c30 con 0x7f7af006f0a0 2023-10-19T12:57:53.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.643+0000 7f7ad77fe640 1 -- 172.21.15.57:0/2610829277 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f7ae4009e80 con 0x7f7af006f0a0 2023-10-19T12:57:53.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.643+0000 7f7ad77fe640 1 -- 172.21.15.57:0/2610829277 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ae40106c0 con 0x7f7af006f0a0 2023-10-19T12:57:53.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.643+0000 7f7af57e2640 1 -- 172.21.15.57:0/2610829277 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7af0154bb0 con 0x7f7af006f0a0 2023-10-19T12:57:53.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.643+0000 7f7af57e2640 1 -- 172.21.15.57:0/2610829277 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7af0156f30 con 0x7f7af006f0a0 2023-10-19T12:57:53.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.644+0000 7f7af57e2640 1 -- 172.21.15.57:0/2610829277 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7af00ce1d0 con 0x7f7af006f0a0 2023-10-19T12:57:53.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.644+0000 7f7ad77fe640 1 -- 172.21.15.57:0/2610829277 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 20) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f7ae4010820 con 0x7f7af006f0a0 2023-10-19T12:57:53.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.644+0000 7f7ad77fe640 1 --2- 172.21.15.57:0/2610829277 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f7adc06af60 0x7f7adc06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:53.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.645+0000 7f7ad77fe640 1 -- 172.21.15.57:0/2610829277 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5908+0+0 (secure 0 0 0) 0x7f7ae4089e60 con 0x7f7af006f0a0 2023-10-19T12:57:53.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.645+0000 7f7aeeffd640 1 --2- 172.21.15.57:0/2610829277 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f7adc06af60 0x7f7adc06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:53.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.645+0000 7f7aeeffd640 1 --2- 172.21.15.57:0/2610829277 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f7adc06af60 0x7f7adc06d420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7ae8005fd0 tx=0x7f7ae80094d0 comp rx=0 tx=0).ready entity=mgr.5809 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:53.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.649+0000 7f7ad77fe640 1 -- 172.21.15.57:0/2610829277 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7ae4054c20 con 0x7f7af006f0a0 2023-10-19T12:57:53.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.825+0000 7f7af57e2640 1 -- 172.21.15.57:0/2610829277 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f7af00690e0 con 0x7f7af006f0a0 2023-10-19T12:57:53.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.829+0000 7f7ad77fe640 1 -- 172.21.15.57:0/2610829277 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v35)=0 v35) v1 ==== 122+0+0 (secure 0 0 0) 0x7f7ae40170c0 con 0x7f7af006f0a0 2023-10-19T12:57:53.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.831+0000 7f7af57e2640 1 -- 172.21.15.57:0/2610829277 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f7adc06af60 msgr2=0x7f7adc06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:53.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.831+0000 7f7af57e2640 1 --2- 172.21.15.57:0/2610829277 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f7adc06af60 0x7f7adc06d420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7ae8005fd0 tx=0x7f7ae80094d0 comp rx=0 tx=0).stop 2023-10-19T12:57:53.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.831+0000 7f7af57e2640 1 -- 172.21.15.57:0/2610829277 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7af006f0a0 msgr2=0x7f7af01ef7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:53.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.831+0000 7f7af57e2640 1 --2- 172.21.15.57:0/2610829277 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7af006f0a0 0x7f7af01ef7f0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f7ae400d950 tx=0x7f7ae400de10 comp rx=0 tx=0).stop 2023-10-19T12:57:53.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.831+0000 7f7af57e2640 1 -- 172.21.15.57:0/2610829277 shutdown_connections 2023-10-19T12:57:53.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.831+0000 7f7af57e2640 1 --2- 172.21.15.57:0/2610829277 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f7adc06af60 0x7f7adc06d420 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:53.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.831+0000 7f7af57e2640 1 --2- 172.21.15.57:0/2610829277 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7af00ab950 0x7f7af014f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:53.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.831+0000 7f7af57e2640 1 --2- 172.21.15.57:0/2610829277 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7af006f0a0 0x7f7af01ef7f0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:53.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.831+0000 7f7af57e2640 1 --2- 172.21.15.57:0/2610829277 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7af006d2d0 0x7f7af01569f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:53.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.831+0000 7f7af57e2640 1 -- 172.21.15.57:0/2610829277 >> 172.21.15.57:0/2610829277 conn(0x7f7af00b0670 msgr2=0x7f7af00c4650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:53.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.831+0000 7f7af57e2640 1 -- 172.21.15.57:0/2610829277 shutdown_connections 2023-10-19T12:57:53.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:53.831+0000 7f7af57e2640 1 -- 172.21.15.57:0/2610829277 wait complete. 2023-10-19T12:57:53.844 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-19T12:57:53.845 DEBUG:teuthology.orchestra.run.smithi057:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-19T12:57:53.890 DEBUG:teuthology.orchestra.run.smithi179:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-19T12:57:53.943 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-19T12:57:53.944 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T12:57:53.945 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T12:57:53.960 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055614,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=358 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248260k,nr_inodes=812065,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-19T12:57:53.961 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-19T12:57:53.961 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T12:57:53.979 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054833,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=284 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3247632k,nr_inodes=811908,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-19T12:57:53.979 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T12:57:53.979 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T12:57:54.016 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055614,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=358 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248260k,nr_inodes=812065,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-19T12:57:54.016 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-19T12:57:54.016 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T12:57:54.036 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054833,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=284 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3247632k,nr_inodes=811908,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-19T12:57:54.037 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-19T12:57:54.037 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T12:57:54.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.162+0000 7ff0f0ee5640 1 -- 172.21.15.57:0/1107077551 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff0ec0afa20 msgr2=0x7ff0ec0cbd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:54.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.162+0000 7ff0f0ee5640 1 --2- 172.21.15.57:0/1107077551 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff0ec0afa20 0x7ff0ec0cbd20 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7ff0e4005fd0 tx=0x7ff0e402bd90 comp rx=0 tx=0).stop 2023-10-19T12:57:54.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.162+0000 7ff0f0ee5640 1 -- 172.21.15.57:0/1107077551 shutdown_connections 2023-10-19T12:57:54.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.162+0000 7ff0f0ee5640 1 --2- 172.21.15.57:0/1107077551 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff0ec158300 0x7ff0ec0c9730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:54.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.162+0000 7ff0f0ee5640 1 --2- 172.21.15.57:0/1107077551 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff0ec0afa20 0x7ff0ec0cbd20 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:54.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.162+0000 7ff0f0ee5640 1 -- 172.21.15.57:0/1107077551 >> 172.21.15.57:0/1107077551 conn(0x7ff0ec069930 msgr2=0x7ff0ec069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:54.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.162+0000 7ff0f0ee5640 1 -- 172.21.15.57:0/1107077551 shutdown_connections 2023-10-19T12:57:54.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.162+0000 7ff0f0ee5640 1 -- 172.21.15.57:0/1107077551 wait complete. 2023-10-19T12:57:54.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.163+0000 7ff0f0ee5640 1 Processor -- start 2023-10-19T12:57:54.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.163+0000 7ff0f0ee5640 1 -- start start 2023-10-19T12:57:54.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.164+0000 7ff0f0ee5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff0ec0705a0 0x7ff0ec1473a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:54.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.164+0000 7ff0f0ee5640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff0ec0afa20 0x7ff0ec13a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:54.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.164+0000 7ff0f0ee5640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff0ec158300 0x7ff0ec1ef650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:54.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.164+0000 7ff0f0ee5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0ec13a870 con 0x7ff0ec0705a0 2023-10-19T12:57:54.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.164+0000 7ff0f0ee5640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0ec1efb90 con 0x7ff0ec0afa20 2023-10-19T12:57:54.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.164+0000 7ff0f0ee5640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0ec15c700 con 0x7ff0ec158300 2023-10-19T12:57:54.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.164+0000 7ff0ea59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff0ec0705a0 0x7ff0ec1473a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:54.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.164+0000 7ff0ea59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff0ec0705a0 0x7ff0ec1473a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43910/0 (socket says 172.21.15.57:43910) 2023-10-19T12:57:54.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.164+0000 7ff0ea59c640 1 -- 172.21.15.57:0/658385827 learned_addr learned my addr 172.21.15.57:0/658385827 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:57:54.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.164+0000 7ff0ea59c640 1 -- 172.21.15.57:0/658385827 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff0ec158300 msgr2=0x7ff0ec1ef650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:54.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.164+0000 7ff0e9d9b640 1 --2- 172.21.15.57:0/658385827 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff0ec0afa20 0x7ff0ec13a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:54.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.164+0000 7ff0ead9d640 1 --2- 172.21.15.57:0/658385827 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff0ec158300 0x7ff0ec1ef650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:54.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.165+0000 7ff0ea59c640 1 --2- 172.21.15.57:0/658385827 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff0ec158300 0x7ff0ec1ef650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:54.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.165+0000 7ff0ea59c640 1 -- 172.21.15.57:0/658385827 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff0ec0afa20 msgr2=0x7ff0ec13a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:54.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.165+0000 7ff0ea59c640 1 --2- 172.21.15.57:0/658385827 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff0ec0afa20 0x7ff0ec13a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:54.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.165+0000 7ff0ea59c640 1 -- 172.21.15.57:0/658385827 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0e4009560 con 0x7ff0ec0705a0 2023-10-19T12:57:54.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.165+0000 7ff0e9d9b640 1 --2- 172.21.15.57:0/658385827 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff0ec0afa20 0x7ff0ec13a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:57:54.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.165+0000 7ff0ead9d640 1 --2- 172.21.15.57:0/658385827 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff0ec158300 0x7ff0ec1ef650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:57:54.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.165+0000 7ff0ea59c640 1 --2- 172.21.15.57:0/658385827 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff0ec0705a0 0x7ff0ec1473a0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7ff0e4006010 tx=0x7ff0e402dd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:54.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.165+0000 7ff0daffd640 1 -- 172.21.15.57:0/658385827 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0e4005650 con 0x7ff0ec0705a0 2023-10-19T12:57:54.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.165+0000 7ff0daffd640 1 -- 172.21.15.57:0/658385827 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7ff0e40038f0 con 0x7ff0ec0705a0 2023-10-19T12:57:54.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.165+0000 7ff0f0ee5640 1 -- 172.21.15.57:0/658385827 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0ec150cf0 con 0x7ff0ec0705a0 2023-10-19T12:57:54.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.165+0000 7ff0daffd640 1 -- 172.21.15.57:0/658385827 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0e4003d00 con 0x7ff0ec0705a0 2023-10-19T12:57:54.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.165+0000 7ff0f0ee5640 1 -- 172.21.15.57:0/658385827 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0ec15cc30 con 0x7ff0ec0705a0 2023-10-19T12:57:54.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.166+0000 7ff0f0ee5640 1 -- 172.21.15.57:0/658385827 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0ec0690e0 con 0x7ff0ec0705a0 2023-10-19T12:57:54.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.167+0000 7ff0daffd640 1 -- 172.21.15.57:0/658385827 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 20) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff0e403e450 con 0x7ff0ec0705a0 2023-10-19T12:57:54.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.167+0000 7ff0daffd640 1 --2- 172.21.15.57:0/658385827 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7ff0cc06af60 0x7ff0cc06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:54.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.167+0000 7ff0daffd640 1 -- 172.21.15.57:0/658385827 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5908+0+0 (secure 0 0 0) 0x7ff0e40ac480 con 0x7ff0ec0705a0 2023-10-19T12:57:54.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.167+0000 7ff0e9d9b640 1 --2- 172.21.15.57:0/658385827 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7ff0cc06af60 0x7ff0cc06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:54.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.168+0000 7ff0e9d9b640 1 --2- 172.21.15.57:0/658385827 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7ff0cc06af60 0x7ff0cc06d420 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff0ec0b0830 tx=0x7ff0e0005c90 comp rx=0 tx=0).ready entity=mgr.5809 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:54.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.171+0000 7ff0daffd640 1 -- 172.21.15.57:0/658385827 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff0e40775e0 con 0x7ff0ec0705a0 2023-10-19T12:57:54.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.383+0000 7ff0f0ee5640 1 -- 172.21.15.57:0/658385827 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff0ec0700d0 con 0x7ff0ec0705a0 2023-10-19T12:57:54.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.383+0000 7ff0daffd640 1 -- 172.21.15.57:0/658385827 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+3702 (secure 0 0 0) 0x7ff0e4043e90 con 0x7ff0ec0705a0 2023-10-19T12:57:54.386 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:57:54.387 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":18,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5024,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/808517451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":808517451},{"type":"v1","addr":"172.21.15.179:6836","nonce":808517451}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":5625,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/3207366629","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":3207366629},{"type":"v1","addr":"172.21.15.57:6834","nonce":3207366629}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":5628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/2353910009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2353910009},{"type":"v1","addr":"172.21.15.57:6835","nonce":2353910009}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5950,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/3829752423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":3829752423},{"type":"v1","addr":"172.21.15.179:6837","nonce":3829752423}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18}],"filesystems":[]} 2023-10-19T12:57:54.387 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 18 2023-10-19T12:57:54.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.385+0000 7ff0f0ee5640 1 -- 172.21.15.57:0/658385827 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7ff0cc06af60 msgr2=0x7ff0cc06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:54.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.385+0000 7ff0f0ee5640 1 --2- 172.21.15.57:0/658385827 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7ff0cc06af60 0x7ff0cc06d420 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff0ec0b0830 tx=0x7ff0e0005c90 comp rx=0 tx=0).stop 2023-10-19T12:57:54.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.385+0000 7ff0f0ee5640 1 -- 172.21.15.57:0/658385827 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff0ec0705a0 msgr2=0x7ff0ec1473a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:54.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.385+0000 7ff0f0ee5640 1 --2- 172.21.15.57:0/658385827 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff0ec0705a0 0x7ff0ec1473a0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7ff0e4006010 tx=0x7ff0e402dd60 comp rx=0 tx=0).stop 2023-10-19T12:57:54.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.385+0000 7ff0f0ee5640 1 -- 172.21.15.57:0/658385827 shutdown_connections 2023-10-19T12:57:54.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.385+0000 7ff0f0ee5640 1 --2- 172.21.15.57:0/658385827 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7ff0cc06af60 0x7ff0cc06d420 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:54.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.385+0000 7ff0f0ee5640 1 --2- 172.21.15.57:0/658385827 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff0ec158300 0x7ff0ec1ef650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:54.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.385+0000 7ff0f0ee5640 1 --2- 172.21.15.57:0/658385827 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff0ec0afa20 0x7ff0ec13a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:54.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.385+0000 7ff0f0ee5640 1 --2- 172.21.15.57:0/658385827 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff0ec0705a0 0x7ff0ec1473a0 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:54.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.385+0000 7ff0f0ee5640 1 -- 172.21.15.57:0/658385827 >> 172.21.15.57:0/658385827 conn(0x7ff0ec069930 msgr2=0x7ff0ec069d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:54.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.385+0000 7ff0f0ee5640 1 -- 172.21.15.57:0/658385827 shutdown_connections 2023-10-19T12:57:54.391 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.385+0000 7ff0f0ee5640 1 -- 172.21.15.57:0/658385827 wait complete. 2023-10-19T12:57:54.399 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-19T12:57:54.399 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-19T12:57:54.399 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-19T12:57:54.400 INFO:teuthology.orchestra.run:waiting for 300 2023-10-19T12:57:54.400 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-19T12:57:54.400 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-19T12:57:54.401 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-19T12:57:54.401 INFO:teuthology.orchestra.run:waiting for 300 2023-10-19T12:57:54.401 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-19T12:57:54.401 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-19T12:57:54.402 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-19T12:57:54.402 INFO:teuthology.orchestra.run:waiting for 300 2023-10-19T12:57:54.402 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-19T12:57:54.403 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-19T12:57:54.403 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-19T12:57:54.403 INFO:teuthology.orchestra.run:waiting for 300 2023-10-19T12:57:54.409 INFO:tasks.ceph.mds.a:Stopped 2023-10-19T12:57:54.410 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-19T12:57:54.411 INFO:tasks.ceph.mds.c:Stopped 2023-10-19T12:57:54.411 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-19T12:57:54.414 INFO:tasks.ceph.mds.b:Stopped 2023-10-19T12:57:54.414 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-19T12:57:54.415 INFO:tasks.ceph.mds.d:Stopped 2023-10-19T12:57:54.415 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-19T12:57:54.453 INFO:tasks.ceph.mds.a:Started 2023-10-19T12:57:54.453 DEBUG:teuthology.parallel:result is None 2023-10-19T12:57:54.455 INFO:tasks.ceph.mds.c:Started 2023-10-19T12:57:54.455 DEBUG:teuthology.parallel:result is None 2023-10-19T12:57:54.456 INFO:tasks.ceph.mds.b:Started 2023-10-19T12:57:54.456 DEBUG:teuthology.parallel:result is None 2023-10-19T12:57:54.457 INFO:tasks.ceph.mds.d:Started 2023-10-19T12:57:54.457 DEBUG:teuthology.parallel:result is None 2023-10-19T12:57:54.458 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-19T12:57:54.534 INFO:tasks.ceph.mds.a.smithi057.stdout:starting mds.a at 2023-10-19T12:57:54.535 INFO:tasks.ceph.mds.c.smithi057.stdout:starting mds.c at 2023-10-19T12:57:54.538 INFO:tasks.ceph.mds.b.smithi179.stdout:starting mds.b at 2023-10-19T12:57:54.538 INFO:tasks.ceph.mds.d.smithi179.stdout:starting mds.d at 2023-10-19T12:57:54.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.569+0000 7f672a097640 1 -- 172.21.15.57:0/3781001466 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f67240cf710 msgr2=0x7f67240ce940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:54.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.569+0000 7f672a097640 1 --2- 172.21.15.57:0/3781001466 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f67240cf710 0x7f67240ce940 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f67240ccf90 tx=0x7f671802be40 comp rx=0 tx=0).stop 2023-10-19T12:57:54.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.569+0000 7f672a097640 1 -- 172.21.15.57:0/3781001466 shutdown_connections 2023-10-19T12:57:54.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.569+0000 7f672a097640 1 --2- 172.21.15.57:0/3781001466 >> v2:172.21.15.179:3300/0 conn(0x7f67240d1970 0x7f67240ae1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:54.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.569+0000 7f672a097640 1 --2- 172.21.15.57:0/3781001466 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f67240cf710 0x7f67240ce940 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:54.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.569+0000 7f672a097640 1 -- 172.21.15.57:0/3781001466 >> 172.21.15.57:0/3781001466 conn(0x7f67240d0d70 msgr2=0x7f67240d1180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:54.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.569+0000 7f672a097640 1 -- 172.21.15.57:0/3781001466 shutdown_connections 2023-10-19T12:57:54.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.569+0000 7f672a097640 1 -- 172.21.15.57:0/3781001466 wait complete. 2023-10-19T12:57:54.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.570+0000 7f672a097640 1 Processor -- start 2023-10-19T12:57:54.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.571+0000 7f672a097640 1 -- start start 2023-10-19T12:57:54.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.571+0000 7f672a097640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f67240cf710 0x7f67240b6510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:54.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.572+0000 7f672a097640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f67240d1970 0x7f67240c7e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:54.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.572+0000 7f672a097640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f672415b0f0 0x7f67241580e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:54.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.572+0000 7f672a097640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67240c8650 con 0x7f67240d1970 2023-10-19T12:57:54.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.572+0000 7f672a097640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67240c5300 con 0x7f672415b0f0 2023-10-19T12:57:54.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.572+0000 7f672a097640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67240c8380 con 0x7f67240cf710 2023-10-19T12:57:54.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.572+0000 7f6722ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f67240d1970 0x7f67240c7e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:54.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.572+0000 7f6722ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f67240d1970 0x7f67240c7e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43962/0 (socket says 172.21.15.57:43962) 2023-10-19T12:57:54.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.572+0000 7f6722ffd640 1 -- 172.21.15.57:0/1504684486 learned_addr learned my addr 172.21.15.57:0/1504684486 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:57:54.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.572+0000 7f67237fe640 1 --2- 172.21.15.57:0/1504684486 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f67240cf710 0x7f67240b6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:54.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.572+0000 7f6723fff640 1 --2- 172.21.15.57:0/1504684486 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f672415b0f0 0x7f67241580e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:54.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.572+0000 7f6722ffd640 1 -- 172.21.15.57:0/1504684486 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f67240cf710 msgr2=0x7f67240b6510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:54.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.572+0000 7f6722ffd640 1 --2- 172.21.15.57:0/1504684486 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f67240cf710 0x7f67240b6510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:54.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.572+0000 7f6722ffd640 1 -- 172.21.15.57:0/1504684486 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f672415b0f0 msgr2=0x7f67241580e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:54.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.572+0000 7f6722ffd640 1 --2- 172.21.15.57:0/1504684486 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f672415b0f0 0x7f67241580e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:54.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.572+0000 7f6722ffd640 1 -- 172.21.15.57:0/1504684486 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6718009560 con 0x7f67240d1970 2023-10-19T12:57:54.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.572+0000 7f6723fff640 1 --2- 172.21.15.57:0/1504684486 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f672415b0f0 0x7f67241580e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:57:54.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.572+0000 7f6722ffd640 1 --2- 172.21.15.57:0/1504684486 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f67240d1970 0x7f67240c7e40 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f6718036040 tx=0x7f671802db70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:54.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.572+0000 7f670bfff640 1 -- 172.21.15.57:0/1504684486 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6718003800 con 0x7f67240d1970 2023-10-19T12:57:54.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.572+0000 7f670bfff640 1 -- 172.21.15.57:0/1504684486 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f6718003a50 con 0x7f67240d1970 2023-10-19T12:57:54.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.572+0000 7f672a097640 1 -- 172.21.15.57:0/1504684486 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67240c49d0 con 0x7f67240d1970 2023-10-19T12:57:54.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.572+0000 7f670bfff640 1 -- 172.21.15.57:0/1504684486 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6718003e20 con 0x7f67240d1970 2023-10-19T12:57:54.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.572+0000 7f672a097640 1 -- 172.21.15.57:0/1504684486 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67240c4f90 con 0x7f67240d1970 2023-10-19T12:57:54.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.574+0000 7f672a097640 1 -- 172.21.15.57:0/1504684486 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67240c5490 con 0x7f67240d1970 2023-10-19T12:57:54.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.574+0000 7f670bfff640 1 -- 172.21.15.57:0/1504684486 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 20) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f671803e450 con 0x7f67240d1970 2023-10-19T12:57:54.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.574+0000 7f670bfff640 1 --2- 172.21.15.57:0/1504684486 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f671006af60 0x7f671006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:54.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.574+0000 7f670bfff640 1 -- 172.21.15.57:0/1504684486 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5908+0+0 (secure 0 0 0) 0x7f67180ac300 con 0x7f67240d1970 2023-10-19T12:57:54.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.575+0000 7f67237fe640 1 --2- 172.21.15.57:0/1504684486 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f671006af60 0x7f671006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:54.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.575+0000 7f67237fe640 1 --2- 172.21.15.57:0/1504684486 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f671006af60 0x7f671006d420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f6714000f60 tx=0x7f6714009040 comp rx=0 tx=0).ready entity=mgr.5809 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:54.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.579+0000 7f670bfff640 1 -- 172.21.15.57:0/1504684486 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6718077380 con 0x7f67240d1970 2023-10-19T12:57:54.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:54.768+0000 7f672a097640 1 -- 172.21.15.57:0/1504684486 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f67240690e0 con 0x7f67240d1970 2023-10-19T12:57:55.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.216+0000 7f670bfff640 1 -- 172.21.15.57:0/1504684486 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v64) v1 ==== 97+0+0 (secure 0 0 0) 0x7f671807a9a0 con 0x7f67240d1970 2023-10-19T12:57:55.219 INFO:teuthology.orchestra.run.smithi057.stderr: removed all blocklist entries 2023-10-19T12:57:55.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.218+0000 7f672a097640 1 -- 172.21.15.57:0/1504684486 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f671006af60 msgr2=0x7f671006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:55.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.218+0000 7f672a097640 1 --2- 172.21.15.57:0/1504684486 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f671006af60 0x7f671006d420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f6714000f60 tx=0x7f6714009040 comp rx=0 tx=0).stop 2023-10-19T12:57:55.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.218+0000 7f672a097640 1 -- 172.21.15.57:0/1504684486 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f67240d1970 msgr2=0x7f67240c7e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:55.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.218+0000 7f672a097640 1 --2- 172.21.15.57:0/1504684486 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f67240d1970 0x7f67240c7e40 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f6718036040 tx=0x7f671802db70 comp rx=0 tx=0).stop 2023-10-19T12:57:55.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.218+0000 7f672a097640 1 -- 172.21.15.57:0/1504684486 shutdown_connections 2023-10-19T12:57:55.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.218+0000 7f672a097640 1 --2- 172.21.15.57:0/1504684486 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f671006af60 0x7f671006d420 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:55.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.218+0000 7f672a097640 1 --2- 172.21.15.57:0/1504684486 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f672415b0f0 0x7f67241580e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:55.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.218+0000 7f672a097640 1 --2- 172.21.15.57:0/1504684486 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f67240d1970 0x7f67240c7e40 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:55.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.218+0000 7f672a097640 1 --2- 172.21.15.57:0/1504684486 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f67240cf710 0x7f67240b6510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:55.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.218+0000 7f672a097640 1 -- 172.21.15.57:0/1504684486 >> 172.21.15.57:0/1504684486 conn(0x7f67240d0d70 msgr2=0x7f672406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:55.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.218+0000 7f672a097640 1 -- 172.21.15.57:0/1504684486 shutdown_connections 2023-10-19T12:57:55.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.218+0000 7f672a097640 1 -- 172.21.15.57:0/1504684486 wait complete. 2023-10-19T12:57:55.235 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-19T12:57:55.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.342+0000 7f588b0c2640 1 -- 172.21.15.57:0/2590630958 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5884141690 msgr2=0x7f58840703e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:55.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.342+0000 7f588b0c2640 1 --2- 172.21.15.57:0/2590630958 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5884141690 0x7f58840703e0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f58840cbd00 tx=0x7f587402bd30 comp rx=0 tx=0).stop 2023-10-19T12:57:55.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.343+0000 7f588b0c2640 1 -- 172.21.15.57:0/2590630958 shutdown_connections 2023-10-19T12:57:55.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.343+0000 7f588b0c2640 1 --2- 172.21.15.57:0/2590630958 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5884141690 0x7f58840703e0 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:55.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.343+0000 7f588b0c2640 1 --2- 172.21.15.57:0/2590630958 >> v2:172.21.15.179:3300/0 conn(0x7f588413e520 0x7f588406bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:55.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.343+0000 7f588b0c2640 1 -- 172.21.15.57:0/2590630958 >> 172.21.15.57:0/2590630958 conn(0x7f5884069930 msgr2=0x7f5884069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:55.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.343+0000 7f588b0c2640 1 -- 172.21.15.57:0/2590630958 shutdown_connections 2023-10-19T12:57:55.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.343+0000 7f588b0c2640 1 -- 172.21.15.57:0/2590630958 wait complete. 2023-10-19T12:57:55.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.344+0000 7f588b0c2640 1 Processor -- start 2023-10-19T12:57:55.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.344+0000 7f588b0c2640 1 -- start start 2023-10-19T12:57:55.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.345+0000 7f588b0c2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f588406c0a0 0x7f588413d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:55.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.345+0000 7f588b0c2640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f588413e520 0x7f58840c6010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:55.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.345+0000 7f588b0c2640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5884141690 0x7f588413ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:55.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.345+0000 7f588b0c2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58840a6540 con 0x7f588406c0a0 2023-10-19T12:57:55.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.345+0000 7f588b0c2640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58840aa690 con 0x7f5884141690 2023-10-19T12:57:55.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.345+0000 7f588b0c2640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58840aa970 con 0x7f588413e520 2023-10-19T12:57:55.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.345+0000 7f5888e5e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f588406c0a0 0x7f588413d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:55.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.345+0000 7f5888e5e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f588406c0a0 0x7f588413d9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43980/0 (socket says 172.21.15.57:43980) 2023-10-19T12:57:55.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.345+0000 7f5888e5e640 1 -- 172.21.15.57:0/2473824661 learned_addr learned my addr 172.21.15.57:0/2473824661 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:57:55.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.345+0000 7f588965f640 1 --2- 172.21.15.57:0/2473824661 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5884141690 0x7f588413ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:55.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.345+0000 7f587bfff640 1 --2- 172.21.15.57:0/2473824661 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f588413e520 0x7f58840c6010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:55.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.345+0000 7f5888e5e640 1 -- 172.21.15.57:0/2473824661 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f588413e520 msgr2=0x7f58840c6010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:55.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.345+0000 7f5888e5e640 1 --2- 172.21.15.57:0/2473824661 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f588413e520 0x7f58840c6010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:55.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.345+0000 7f5888e5e640 1 -- 172.21.15.57:0/2473824661 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5884141690 msgr2=0x7f588413ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:55.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.345+0000 7f5888e5e640 1 --2- 172.21.15.57:0/2473824661 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5884141690 0x7f588413ae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:55.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.345+0000 7f5888e5e640 1 -- 172.21.15.57:0/2473824661 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5874009560 con 0x7f588406c0a0 2023-10-19T12:57:55.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.345+0000 7f5888e5e640 1 --2- 172.21.15.57:0/2473824661 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f588406c0a0 0x7f588413d9b0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f588000c890 tx=0x7f588000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:55.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.345+0000 7f58797fa640 1 -- 172.21.15.57:0/2473824661 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5880004220 con 0x7f588406c0a0 2023-10-19T12:57:55.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.345+0000 7f58797fa640 1 -- 172.21.15.57:0/2473824661 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f5880004470 con 0x7f588406c0a0 2023-10-19T12:57:55.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.345+0000 7f588b0c2640 1 -- 172.21.15.57:0/2473824661 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58840aac50 con 0x7f588406c0a0 2023-10-19T12:57:55.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.345+0000 7f58797fa640 1 -- 172.21.15.57:0/2473824661 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5880007c10 con 0x7f588406c0a0 2023-10-19T12:57:55.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.345+0000 7f588b0c2640 1 -- 172.21.15.57:0/2473824661 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58841e24b0 con 0x7f588406c0a0 2023-10-19T12:57:55.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.346+0000 7f588b0c2640 1 -- 172.21.15.57:0/2473824661 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58840690e0 con 0x7f588406c0a0 2023-10-19T12:57:55.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.347+0000 7f58797fa640 1 -- 172.21.15.57:0/2473824661 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 20) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f5880005020 con 0x7f588406c0a0 2023-10-19T12:57:55.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.347+0000 7f58797fa640 1 --2- 172.21.15.57:0/2473824661 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f587006af90 0x7f587006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:55.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.347+0000 7f58797fa640 1 -- 172.21.15.57:0/2473824661 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f5880089b30 con 0x7f588406c0a0 2023-10-19T12:57:55.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.348+0000 7f587bfff640 1 --2- 172.21.15.57:0/2473824661 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f587006af90 0x7f587006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:55.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.348+0000 7f587bfff640 1 --2- 172.21.15.57:0/2473824661 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f587006af90 0x7f587006d450 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5874035000 tx=0x7f5874004730 comp rx=0 tx=0).ready entity=mgr.5809 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:55.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.352+0000 7f58797fa640 1 -- 172.21.15.57:0/2473824661 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f5880054c60 con 0x7f588406c0a0 2023-10-19T12:57:55.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.598+0000 7f588b0c2640 1 -- 172.21.15.57:0/2473824661 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f58840ae490 con 0x7f588406c0a0 2023-10-19T12:57:55.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.599+0000 7f58797fa640 1 -- 172.21.15.57:0/2473824661 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v21) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f588001d070 con 0x7f588406c0a0 2023-10-19T12:57:55.602 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:57:55.602 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2023-10-19T12:57:55.603 INFO:teuthology.orchestra.run.smithi057.stdout: "auth_dump": [ 2023-10-19T12:57:55.603 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T12:57:55.603 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "mds.a", 2023-10-19T12:57:55.603 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBjJDFlmbVANhAAP/ynHlU/OxaRq5lQoAa8Ng==", 2023-10-19T12:57:55.603 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T12:57:55.604 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T12:57:55.604 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow *", 2023-10-19T12:57:55.604 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow *", 2023-10-19T12:57:55.604 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T12:57:55.604 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T12:57:55.605 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T12:57:55.605 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T12:57:55.605 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "mds.b", 2023-10-19T12:57:55.605 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFl9l7RBxAAC0geI2tJ1SIZZr1++kS/BA==", 2023-10-19T12:57:55.605 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T12:57:55.606 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T12:57:55.606 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow *", 2023-10-19T12:57:55.606 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow *", 2023-10-19T12:57:55.606 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T12:57:55.606 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T12:57:55.607 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T12:57:55.607 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T12:57:55.607 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "mds.c", 2023-10-19T12:57:55.607 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFlFkaoAhAA/PBizkJ4bm958oP4CmPuIQ==", 2023-10-19T12:57:55.607 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T12:57:55.608 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T12:57:55.608 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow *", 2023-10-19T12:57:55.608 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow *", 2023-10-19T12:57:55.608 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T12:57:55.608 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T12:57:55.609 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T12:57:55.609 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T12:57:55.609 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "mds.d", 2023-10-19T12:57:55.609 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFlx8skDxAAAfCHTYtYJgJik0WHSWeQfQ==", 2023-10-19T12:57:55.609 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T12:57:55.610 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T12:57:55.610 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow *", 2023-10-19T12:57:55.610 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow *", 2023-10-19T12:57:55.610 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T12:57:55.610 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T12:57:55.611 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T12:57:55.611 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T12:57:55.611 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.0", 2023-10-19T12:57:55.611 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBrJDFlCo9sBxAAQN2cID4nCQgRwGaGFEV8LQ==", 2023-10-19T12:57:55.611 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T12:57:55.612 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T12:57:55.612 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T12:57:55.612 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T12:57:55.612 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T12:57:55.612 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T12:57:55.612 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T12:57:55.613 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.1", 2023-10-19T12:57:55.613 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBuJDFl4fsLCBAAp3Pi80yZtc5sju0GisEuVA==", 2023-10-19T12:57:55.613 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T12:57:55.613 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T12:57:55.613 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T12:57:55.614 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T12:57:55.614 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T12:57:55.614 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T12:57:55.614 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T12:57:55.614 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.2", 2023-10-19T12:57:55.615 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBxJDFl+KAKCBAAd4N9cluARxPVfBLU+7Hd7A==", 2023-10-19T12:57:55.615 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T12:57:55.615 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T12:57:55.615 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T12:57:55.615 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T12:57:55.616 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T12:57:55.616 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T12:57:55.616 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T12:57:55.616 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.3", 2023-10-19T12:57:55.616 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQB0JDFlU/toCBAAP0Ou1zcHKNx5SfbybJkh4Q==", 2023-10-19T12:57:55.616 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T12:57:55.617 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T12:57:55.617 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T12:57:55.617 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T12:57:55.617 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T12:57:55.617 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T12:57:55.618 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T12:57:55.618 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.4", 2023-10-19T12:57:55.618 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQB6JDFlc6stEBAALkgNyxoiCTzZnKpBSPxI6w==", 2023-10-19T12:57:55.618 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T12:57:55.618 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T12:57:55.619 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T12:57:55.619 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T12:57:55.619 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T12:57:55.619 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T12:57:55.619 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T12:57:55.620 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.5", 2023-10-19T12:57:55.620 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQB9JDFlI3cNDhAAuG3tHJ6RvPwAzPE76wTIVQ==", 2023-10-19T12:57:55.620 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T12:57:55.620 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T12:57:55.620 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T12:57:55.621 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T12:57:55.621 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T12:57:55.621 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T12:57:55.621 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T12:57:55.621 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.6", 2023-10-19T12:57:55.622 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQCAJDFlLPumCxAA/iy08t/2N9rXfn71KtbSQw==", 2023-10-19T12:57:55.622 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T12:57:55.622 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T12:57:55.622 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T12:57:55.622 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T12:57:55.623 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T12:57:55.623 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T12:57:55.623 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T12:57:55.623 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.7", 2023-10-19T12:57:55.623 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQCDJDFlHmrbCRAA+hpWygGIJydfU6cHWlfx/Q==", 2023-10-19T12:57:55.624 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T12:57:55.624 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T12:57:55.624 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T12:57:55.624 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T12:57:55.624 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T12:57:55.624 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T12:57:55.625 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T12:57:55.625 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "client.0", 2023-10-19T12:57:55.625 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFlg9yuExAATxiWDfS+p07G1F0pORbJEg==", 2023-10-19T12:57:55.625 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T12:57:55.625 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T12:57:55.626 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow r", 2023-10-19T12:57:55.626 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-19T12:57:55.626 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T12:57:55.626 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T12:57:55.626 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T12:57:55.627 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "client.1", 2023-10-19T12:57:55.627 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFl9IwYGxAAHPituiJSPkHDuD0QyG4Hww==", 2023-10-19T12:57:55.627 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T12:57:55.627 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T12:57:55.627 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow r", 2023-10-19T12:57:55.628 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow rw", 2023-10-19T12:57:55.628 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow rwx" 2023-10-19T12:57:55.628 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T12:57:55.628 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T12:57:55.628 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T12:57:55.628 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "client.2", 2023-10-19T12:57:55.629 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFlMd+AFxAAS/UTESDt76GmrhhN2o/Fdw==", 2023-10-19T12:57:55.629 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T12:57:55.629 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T12:57:55.629 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow r", 2023-10-19T12:57:55.629 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow rw", 2023-10-19T12:57:55.630 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow rwx" 2023-10-19T12:57:55.630 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T12:57:55.630 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T12:57:55.630 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T12:57:55.630 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "client.3", 2023-10-19T12:57:55.631 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFlHZ3fIBAA42B7n8opLS5smXIwJyVYzw==", 2023-10-19T12:57:55.631 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T12:57:55.631 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T12:57:55.631 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow r", 2023-10-19T12:57:55.631 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow rw", 2023-10-19T12:57:55.632 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow rwx" 2023-10-19T12:57:55.632 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T12:57:55.632 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T12:57:55.632 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T12:57:55.632 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "client.admin", 2023-10-19T12:57:55.633 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBjJDFlBgasFxAAMA24j8KszcXMEpYRQDXj4g==", 2023-10-19T12:57:55.633 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T12:57:55.633 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow *", 2023-10-19T12:57:55.633 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow *", 2023-10-19T12:57:55.633 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow *", 2023-10-19T12:57:55.634 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T12:57:55.634 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T12:57:55.634 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T12:57:55.634 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T12:57:55.634 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "mgr.x", 2023-10-19T12:57:55.635 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBjJDFllFBfMhAA5+KPXWmYZqcESmLObaUbBA==", 2023-10-19T12:57:55.635 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T12:57:55.635 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow *", 2023-10-19T12:57:55.635 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile mgr", 2023-10-19T12:57:55.635 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T12:57:55.636 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T12:57:55.636 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T12:57:55.636 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T12:57:55.636 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "mgr.y", 2023-10-19T12:57:55.636 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBjJDFlxRzJLRAAGjG58o7qXAHvIEIRYwl6QA==", 2023-10-19T12:57:55.637 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T12:57:55.637 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow *", 2023-10-19T12:57:55.637 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile mgr", 2023-10-19T12:57:55.637 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T12:57:55.638 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T12:57:55.638 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T12:57:55.638 INFO:teuthology.orchestra.run.smithi057.stdout: ] 2023-10-19T12:57:55.638 INFO:teuthology.orchestra.run.smithi057.stdout:} 2023-10-19T12:57:55.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.601+0000 7f588b0c2640 1 -- 172.21.15.57:0/2473824661 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f587006af90 msgr2=0x7f587006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:55.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.601+0000 7f588b0c2640 1 --2- 172.21.15.57:0/2473824661 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f587006af90 0x7f587006d450 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5874035000 tx=0x7f5874004730 comp rx=0 tx=0).stop 2023-10-19T12:57:55.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.601+0000 7f588b0c2640 1 -- 172.21.15.57:0/2473824661 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f588406c0a0 msgr2=0x7f588413d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:55.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.601+0000 7f588b0c2640 1 --2- 172.21.15.57:0/2473824661 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f588406c0a0 0x7f588413d9b0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f588000c890 tx=0x7f588000cd50 comp rx=0 tx=0).stop 2023-10-19T12:57:55.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.601+0000 7f588b0c2640 1 -- 172.21.15.57:0/2473824661 shutdown_connections 2023-10-19T12:57:55.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.601+0000 7f588b0c2640 1 --2- 172.21.15.57:0/2473824661 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f587006af90 0x7f587006d450 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:55.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.601+0000 7f588b0c2640 1 --2- 172.21.15.57:0/2473824661 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5884141690 0x7f588413ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:55.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.601+0000 7f588b0c2640 1 --2- 172.21.15.57:0/2473824661 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f588413e520 0x7f58840c6010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:55.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.601+0000 7f588b0c2640 1 --2- 172.21.15.57:0/2473824661 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f588406c0a0 0x7f588413d9b0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:55.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.601+0000 7f588b0c2640 1 -- 172.21.15.57:0/2473824661 >> 172.21.15.57:0/2473824661 conn(0x7f5884069930 msgr2=0x7f588406f080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:55.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.601+0000 7f588b0c2640 1 -- 172.21.15.57:0/2473824661 shutdown_connections 2023-10-19T12:57:55.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.601+0000 7f588b0c2640 1 -- 172.21.15.57:0/2473824661 wait complete. 2023-10-19T12:57:55.643 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T12:57:55.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.751+0000 7ff9163f5640 1 -- 172.21.15.57:0/2564564804 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff9100cbd20 msgr2=0x7ff910158300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:55.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.751+0000 7ff9163f5640 1 --2- 172.21.15.57:0/2564564804 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff9100cbd20 0x7ff910158300 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7ff904009830 tx=0x7ff90402bd60 comp rx=0 tx=0).stop 2023-10-19T12:57:55.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.751+0000 7ff9163f5640 1 -- 172.21.15.57:0/2564564804 shutdown_connections 2023-10-19T12:57:55.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.751+0000 7ff9163f5640 1 --2- 172.21.15.57:0/2564564804 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff9100afa20 0x7ff9100705a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:55.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.751+0000 7ff9163f5640 1 --2- 172.21.15.57:0/2564564804 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff9100cbd20 0x7ff910158300 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:55.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.751+0000 7ff9163f5640 1 -- 172.21.15.57:0/2564564804 >> 172.21.15.57:0/2564564804 conn(0x7ff910069930 msgr2=0x7ff910069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:55.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.751+0000 7ff9163f5640 1 -- 172.21.15.57:0/2564564804 shutdown_connections 2023-10-19T12:57:55.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.751+0000 7ff9163f5640 1 -- 172.21.15.57:0/2564564804 wait complete. 2023-10-19T12:57:55.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.752+0000 7ff9163f5640 1 Processor -- start 2023-10-19T12:57:55.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.752+0000 7ff9163f5640 1 -- start start 2023-10-19T12:57:55.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.753+0000 7ff9163f5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9100afa20 0x7ff910146490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:55.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.753+0000 7ff9163f5640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff9100b0530 0x7ff910133650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:55.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.753+0000 7ff9163f5640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff9100cbd20 0x7ff910146e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:55.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.753+0000 7ff9163f5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff910131ab0 con 0x7ff9100afa20 2023-10-19T12:57:55.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.753+0000 7ff9163f5640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9101340a0 con 0x7ff9100b0530 2023-10-19T12:57:55.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.753+0000 7ff9163f5640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff910134380 con 0x7ff9100cbd20 2023-10-19T12:57:55.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.753+0000 7ff90ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9100afa20 0x7ff910146490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:55.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.753+0000 7ff90ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9100afa20 0x7ff910146490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43992/0 (socket says 172.21.15.57:43992) 2023-10-19T12:57:55.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.753+0000 7ff90ffff640 1 -- 172.21.15.57:0/1561703767 learned_addr learned my addr 172.21.15.57:0/1561703767 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:57:55.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.754+0000 7ff90f7fe640 1 --2- 172.21.15.57:0/1561703767 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff9100b0530 0x7ff910133650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:55.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.754+0000 7ff914992640 1 --2- 172.21.15.57:0/1561703767 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff9100cbd20 0x7ff910146e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:55.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.754+0000 7ff90ffff640 1 -- 172.21.15.57:0/1561703767 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff9100cbd20 msgr2=0x7ff910146e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:55.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.754+0000 7ff90ffff640 1 --2- 172.21.15.57:0/1561703767 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff9100cbd20 0x7ff910146e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:55.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.754+0000 7ff90ffff640 1 -- 172.21.15.57:0/1561703767 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff9100b0530 msgr2=0x7ff910133650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:55.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.754+0000 7ff90ffff640 1 --2- 172.21.15.57:0/1561703767 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff9100b0530 0x7ff910133650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:55.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.754+0000 7ff90ffff640 1 -- 172.21.15.57:0/1561703767 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff900009560 con 0x7ff9100afa20 2023-10-19T12:57:55.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.754+0000 7ff90f7fe640 1 --2- 172.21.15.57:0/1561703767 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff9100b0530 0x7ff910133650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:57:55.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.754+0000 7ff90ffff640 1 --2- 172.21.15.57:0/1561703767 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9100afa20 0x7ff910146490 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7ff904009780 tx=0x7ff9040038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:55.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.754+0000 7ff90cff9640 1 -- 172.21.15.57:0/1561703767 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff904003ea0 con 0x7ff9100afa20 2023-10-19T12:57:55.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.754+0000 7ff90cff9640 1 -- 172.21.15.57:0/1561703767 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7ff904034440 con 0x7ff9100afa20 2023-10-19T12:57:55.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.754+0000 7ff9163f5640 1 -- 172.21.15.57:0/1561703767 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff904009490 con 0x7ff9100afa20 2023-10-19T12:57:55.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.754+0000 7ff90cff9640 1 -- 172.21.15.57:0/1561703767 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff904034850 con 0x7ff9100afa20 2023-10-19T12:57:55.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.754+0000 7ff9163f5640 1 -- 172.21.15.57:0/1561703767 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff910070cf0 con 0x7ff9100afa20 2023-10-19T12:57:55.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.755+0000 7ff9163f5640 1 -- 172.21.15.57:0/1561703767 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9100690e0 con 0x7ff9100afa20 2023-10-19T12:57:55.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.756+0000 7ff90cff9640 1 -- 172.21.15.57:0/1561703767 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 20) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff90403b040 con 0x7ff9100afa20 2023-10-19T12:57:55.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.756+0000 7ff90cff9640 1 --2- 172.21.15.57:0/1561703767 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7ff8fc06af60 0x7ff8fc06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:55.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.756+0000 7ff90cff9640 1 -- 172.21.15.57:0/1561703767 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5177+0+0 (secure 0 0 0) 0x7ff9040aba90 con 0x7ff9100afa20 2023-10-19T12:57:55.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.756+0000 7ff90f7fe640 1 --2- 172.21.15.57:0/1561703767 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7ff8fc06af60 0x7ff8fc06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:55.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.757+0000 7ff90f7fe640 1 --2- 172.21.15.57:0/1561703767 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7ff8fc06af60 0x7ff8fc06d420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff9100ac4e0 tx=0x7ff900009260 comp rx=0 tx=0).ready entity=mgr.5809 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:55.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.760+0000 7ff90cff9640 1 -- 172.21.15.57:0/1561703767 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff904076cf0 con 0x7ff9100afa20 2023-10-19T12:57:55.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.937+0000 7ff9163f5640 1 -- 172.21.15.57:0/1561703767 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff91006f700 con 0x7ff9100afa20 2023-10-19T12:57:55.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.938+0000 7ff90cff9640 1 -- 172.21.15.57:0/1561703767 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v66) v1 ==== 74+0+13074 (secure 0 0 0) 0x7ff904041090 con 0x7ff9100afa20 2023-10-19T12:57:55.941 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:57:55.941 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":66,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T12:57:55.230355+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T12:57:55.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.940+0000 7ff9163f5640 1 -- 172.21.15.57:0/1561703767 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7ff8fc06af60 msgr2=0x7ff8fc06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:55.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.940+0000 7ff9163f5640 1 --2- 172.21.15.57:0/1561703767 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7ff8fc06af60 0x7ff8fc06d420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff9100ac4e0 tx=0x7ff900009260 comp rx=0 tx=0).stop 2023-10-19T12:57:55.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.940+0000 7ff9163f5640 1 -- 172.21.15.57:0/1561703767 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9100afa20 msgr2=0x7ff910146490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:55.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.940+0000 7ff9163f5640 1 --2- 172.21.15.57:0/1561703767 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9100afa20 0x7ff910146490 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7ff904009780 tx=0x7ff9040038a0 comp rx=0 tx=0).stop 2023-10-19T12:57:55.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.940+0000 7ff9163f5640 1 -- 172.21.15.57:0/1561703767 shutdown_connections 2023-10-19T12:57:55.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.940+0000 7ff9163f5640 1 --2- 172.21.15.57:0/1561703767 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7ff8fc06af60 0x7ff8fc06d420 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:55.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.940+0000 7ff9163f5640 1 --2- 172.21.15.57:0/1561703767 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff9100cbd20 0x7ff910146e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:55.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.940+0000 7ff9163f5640 1 --2- 172.21.15.57:0/1561703767 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff9100b0530 0x7ff910133650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:55.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.940+0000 7ff9163f5640 1 --2- 172.21.15.57:0/1561703767 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9100afa20 0x7ff910146490 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:55.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.940+0000 7ff9163f5640 1 -- 172.21.15.57:0/1561703767 >> 172.21.15.57:0/1561703767 conn(0x7ff910069930 msgr2=0x7ff910151000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:55.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.940+0000 7ff9163f5640 1 -- 172.21.15.57:0/1561703767 shutdown_connections 2023-10-19T12:57:55.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:55.940+0000 7ff9163f5640 1 -- 172.21.15.57:0/1561703767 wait complete. 2023-10-19T12:57:55.954 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-19T12:44:07.371186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-19T12:57:55.955 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-19T12:57:56.067 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.064+0000 7fc8f4a75640 1 -- 172.21.15.57:0/4233500257 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc8f00cf340 msgr2=0x7fc8f00d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:56.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.064+0000 7fc8f4a75640 1 --2- 172.21.15.57:0/4233500257 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc8f00cf340 0x7fc8f00d1910 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fc8f015b680 tx=0x7fc8e002bd30 comp rx=0 tx=0).stop 2023-10-19T12:57:56.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.064+0000 7fc8f4a75640 1 -- 172.21.15.57:0/4233500257 shutdown_connections 2023-10-19T12:57:56.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.064+0000 7fc8f4a75640 1 --2- 172.21.15.57:0/4233500257 >> v2:172.21.15.179:3300/0 conn(0x7fc8f00ceb40 0x7fc8f00b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:56.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.064+0000 7fc8f4a75640 1 --2- 172.21.15.57:0/4233500257 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc8f00cf340 0x7fc8f00d1910 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:56.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.064+0000 7fc8f4a75640 1 -- 172.21.15.57:0/4233500257 >> 172.21.15.57:0/4233500257 conn(0x7fc8f00d0d60 msgr2=0x7fc8f00d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:56.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.064+0000 7fc8f4a75640 1 -- 172.21.15.57:0/4233500257 shutdown_connections 2023-10-19T12:57:56.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.064+0000 7fc8f4a75640 1 -- 172.21.15.57:0/4233500257 wait complete. 2023-10-19T12:57:56.072 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.065+0000 7fc8f4a75640 1 Processor -- start 2023-10-19T12:57:56.072 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.066+0000 7fc8f4a75640 1 -- start start 2023-10-19T12:57:56.072 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.066+0000 7fc8f4a75640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc8f00ceb40 0x7fc8f01f54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:56.072 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.066+0000 7fc8f4a75640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc8f00cf340 0x7fc8f01f13f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:56.072 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.066+0000 7fc8f4a75640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc8f015b2a0 0x7fc8f00ca870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:56.073 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.066+0000 7fc8f4a75640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f0158d50 con 0x7fc8f00ceb40 2023-10-19T12:57:56.073 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.066+0000 7fc8f4a75640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f00cb010 con 0x7fc8f015b2a0 2023-10-19T12:57:56.073 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.066+0000 7fc8f4a75640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f00cadb0 con 0x7fc8f00cf340 2023-10-19T12:57:56.073 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.066+0000 7fc8ee59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc8f00ceb40 0x7fc8f01f54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:56.073 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.066+0000 7fc8ee59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc8f00ceb40 0x7fc8f01f54f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44020/0 (socket says 172.21.15.57:44020) 2023-10-19T12:57:56.074 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.066+0000 7fc8ee59c640 1 -- 172.21.15.57:0/3084699224 learned_addr learned my addr 172.21.15.57:0/3084699224 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:57:56.074 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.067+0000 7fc8e7fff640 1 --2- 172.21.15.57:0/3084699224 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc8f00cf340 0x7fc8f01f13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:56.074 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.067+0000 7fc8eed9d640 1 --2- 172.21.15.57:0/3084699224 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc8f015b2a0 0x7fc8f00ca870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:56.074 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.067+0000 7fc8ee59c640 1 -- 172.21.15.57:0/3084699224 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc8f00cf340 msgr2=0x7fc8f01f13f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:56.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.067+0000 7fc8ee59c640 1 --2- 172.21.15.57:0/3084699224 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc8f00cf340 0x7fc8f01f13f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:56.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.067+0000 7fc8ee59c640 1 -- 172.21.15.57:0/3084699224 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc8f015b2a0 msgr2=0x7fc8f00ca870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:56.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.067+0000 7fc8ee59c640 1 --2- 172.21.15.57:0/3084699224 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc8f015b2a0 0x7fc8f00ca870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:56.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.067+0000 7fc8ee59c640 1 -- 172.21.15.57:0/3084699224 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8e0009560 con 0x7fc8f00ceb40 2023-10-19T12:57:56.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.067+0000 7fc8e7fff640 1 --2- 172.21.15.57:0/3084699224 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc8f00cf340 0x7fc8f01f13f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:57:56.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.067+0000 7fc8ee59c640 1 --2- 172.21.15.57:0/3084699224 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc8f00ceb40 0x7fc8f01f54f0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fc8e0036040 tx=0x7fc8e002dd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:56.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.067+0000 7fc8e6ffd640 1 -- 172.21.15.57:0/3084699224 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e0005650 con 0x7fc8f00ceb40 2023-10-19T12:57:56.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.067+0000 7fc8f4a75640 1 -- 172.21.15.57:0/3084699224 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8f00c7fd0 con 0x7fc8f00ceb40 2023-10-19T12:57:56.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.067+0000 7fc8e6ffd640 1 -- 172.21.15.57:0/3084699224 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fc8e00038f0 con 0x7fc8f00ceb40 2023-10-19T12:57:56.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.067+0000 7fc8e6ffd640 1 -- 172.21.15.57:0/3084699224 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e0003ce0 con 0x7fc8f00ceb40 2023-10-19T12:57:56.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.067+0000 7fc8f4a75640 1 -- 172.21.15.57:0/3084699224 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8f00c8550 con 0x7fc8f00ceb40 2023-10-19T12:57:56.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.068+0000 7fc8f4a75640 1 -- 172.21.15.57:0/3084699224 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8f00690e0 con 0x7fc8f00ceb40 2023-10-19T12:57:56.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.068+0000 7fc8e6ffd640 1 -- 172.21.15.57:0/3084699224 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 20) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc8e003e450 con 0x7fc8f00ceb40 2023-10-19T12:57:56.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.069+0000 7fc8e6ffd640 1 --2- 172.21.15.57:0/3084699224 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7fc8d006af60 0x7fc8d006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:56.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.069+0000 7fc8e6ffd640 1 -- 172.21.15.57:0/3084699224 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fc8e00ac070 con 0x7fc8f00ceb40 2023-10-19T12:57:56.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.069+0000 7fc8e7fff640 1 --2- 172.21.15.57:0/3084699224 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7fc8d006af60 0x7fc8d006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:56.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.070+0000 7fc8e7fff640 1 --2- 172.21.15.57:0/3084699224 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7fc8d006af60 0x7fc8d006d420 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc8f00c74d0 tx=0x7fc8d8008040 comp rx=0 tx=0).ready entity=mgr.5809 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:56.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.073+0000 7fc8e6ffd640 1 -- 172.21.15.57:0/3084699224 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc8e00772c0 con 0x7fc8f00ceb40 2023-10-19T12:57:56.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.249+0000 7fc8f4a75640 1 -- 172.21.15.57:0/3084699224 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc8f00b0d00 con 0x7fc8f00ceb40 2023-10-19T12:57:56.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.250+0000 7fc8e6ffd640 1 -- 172.21.15.57:0/3084699224 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v66) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc8e007a8e0 con 0x7fc8f00ceb40 2023-10-19T12:57:56.253 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-19T12:57:56.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.252+0000 7fc8f4a75640 1 -- 172.21.15.57:0/3084699224 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7fc8d006af60 msgr2=0x7fc8d006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:56.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.252+0000 7fc8f4a75640 1 --2- 172.21.15.57:0/3084699224 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7fc8d006af60 0x7fc8d006d420 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc8f00c74d0 tx=0x7fc8d8008040 comp rx=0 tx=0).stop 2023-10-19T12:57:56.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.252+0000 7fc8f4a75640 1 -- 172.21.15.57:0/3084699224 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc8f00ceb40 msgr2=0x7fc8f01f54f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:56.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.252+0000 7fc8f4a75640 1 --2- 172.21.15.57:0/3084699224 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc8f00ceb40 0x7fc8f01f54f0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fc8e0036040 tx=0x7fc8e002dd00 comp rx=0 tx=0).stop 2023-10-19T12:57:56.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.252+0000 7fc8f4a75640 1 -- 172.21.15.57:0/3084699224 shutdown_connections 2023-10-19T12:57:56.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.252+0000 7fc8f4a75640 1 --2- 172.21.15.57:0/3084699224 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7fc8d006af60 0x7fc8d006d420 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:56.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.252+0000 7fc8f4a75640 1 --2- 172.21.15.57:0/3084699224 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc8f015b2a0 0x7fc8f00ca870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:56.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.252+0000 7fc8f4a75640 1 --2- 172.21.15.57:0/3084699224 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc8f00cf340 0x7fc8f01f13f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:56.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.252+0000 7fc8f4a75640 1 --2- 172.21.15.57:0/3084699224 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc8f00ceb40 0x7fc8f01f54f0 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:56.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.252+0000 7fc8f4a75640 1 -- 172.21.15.57:0/3084699224 >> 172.21.15.57:0/3084699224 conn(0x7fc8f00d0d60 msgr2=0x7fc8f00b0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:56.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.252+0000 7fc8f4a75640 1 -- 172.21.15.57:0/3084699224 shutdown_connections 2023-10-19T12:57:56.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.252+0000 7fc8f4a75640 1 -- 172.21.15.57:0/3084699224 wait complete. 2023-10-19T12:57:56.265 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-19T12:57:56.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.371+0000 7f16549f5640 1 -- 172.21.15.57:0/2732378618 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f16500cd9e0 msgr2=0x7f16500cde50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:56.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.371+0000 7f16549f5640 1 --2- 172.21.15.57:0/2732378618 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f16500cd9e0 0x7f16500cde50 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f1650072450 tx=0x7f164802c260 comp rx=0 tx=0).stop 2023-10-19T12:57:56.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.371+0000 7f16549f5640 1 -- 172.21.15.57:0/2732378618 shutdown_connections 2023-10-19T12:57:56.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.371+0000 7f16549f5640 1 --2- 172.21.15.57:0/2732378618 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f16500cd9e0 0x7f16500cde50 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:56.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.371+0000 7f16549f5640 1 --2- 172.21.15.57:0/2732378618 >> v2:172.21.15.179:3300/0 conn(0x7f1650158cf0 0x7f165015c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:56.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.371+0000 7f16549f5640 1 -- 172.21.15.57:0/2732378618 >> 172.21.15.57:0/2732378618 conn(0x7f16500af7a0 msgr2=0x7f16500afb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:56.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.371+0000 7f16549f5640 1 -- 172.21.15.57:0/2732378618 shutdown_connections 2023-10-19T12:57:56.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.371+0000 7f16549f5640 1 -- 172.21.15.57:0/2732378618 wait complete. 2023-10-19T12:57:56.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.373+0000 7f16549f5640 1 Processor -- start 2023-10-19T12:57:56.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.373+0000 7f16549f5640 1 -- start start 2023-10-19T12:57:56.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.374+0000 7f16549f5640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f16500cd9e0 0x7f1650149270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:56.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.374+0000 7f16549f5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1650158910 0x7f1650155110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:56.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.374+0000 7f16549f5640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1650158cf0 0x7f16500ae840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:56.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.374+0000 7f16549f5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1650149c50 con 0x7f1650158910 2023-10-19T12:57:56.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.374+0000 7f16549f5640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f165014a080 con 0x7f16500cd9e0 2023-10-19T12:57:56.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.374+0000 7f16549f5640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16501ef6e0 con 0x7f1650158cf0 2023-10-19T12:57:56.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.374+0000 7f1645d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1650158910 0x7f1650155110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:56.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.374+0000 7f1645d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1650158910 0x7f1650155110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44042/0 (socket says 172.21.15.57:44042) 2023-10-19T12:57:56.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.374+0000 7f1645d9b640 1 -- 172.21.15.57:0/1457346375 learned_addr learned my addr 172.21.15.57:0/1457346375 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:57:56.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.374+0000 7f164e59c640 1 --2- 172.21.15.57:0/1457346375 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f16500cd9e0 0x7f1650149270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:56.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.374+0000 7f164ed9d640 1 --2- 172.21.15.57:0/1457346375 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1650158cf0 0x7f16500ae840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:56.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.374+0000 7f1645d9b640 1 -- 172.21.15.57:0/1457346375 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1650158cf0 msgr2=0x7f16500ae840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:56.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.374+0000 7f1645d9b640 1 --2- 172.21.15.57:0/1457346375 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1650158cf0 0x7f16500ae840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:56.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.374+0000 7f1645d9b640 1 -- 172.21.15.57:0/1457346375 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f16500cd9e0 msgr2=0x7f1650149270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:56.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.374+0000 7f1645d9b640 1 --2- 172.21.15.57:0/1457346375 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f16500cd9e0 0x7f1650149270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:56.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.374+0000 7f1645d9b640 1 -- 172.21.15.57:0/1457346375 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1648009900 con 0x7f1650158910 2023-10-19T12:57:56.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.374+0000 7f164e59c640 1 --2- 172.21.15.57:0/1457346375 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f16500cd9e0 0x7f1650149270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T12:57:56.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.375+0000 7f1645d9b640 1 --2- 172.21.15.57:0/1457346375 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1650158910 0x7f1650155110 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f163800c970 tx=0x7f163800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:56.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.375+0000 7f16477fe640 1 -- 172.21.15.57:0/1457346375 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1638007bb0 con 0x7f1650158910 2023-10-19T12:57:56.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.375+0000 7f16477fe640 1 -- 172.21.15.57:0/1457346375 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f1638007e00 con 0x7f1650158910 2023-10-19T12:57:56.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.375+0000 7f16477fe640 1 -- 172.21.15.57:0/1457346375 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1638005280 con 0x7f1650158910 2023-10-19T12:57:56.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.375+0000 7f16549f5640 1 -- 172.21.15.57:0/1457346375 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16501ef9c0 con 0x7f1650158910 2023-10-19T12:57:56.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.375+0000 7f16549f5640 1 -- 172.21.15.57:0/1457346375 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16501eff80 con 0x7f1650158910 2023-10-19T12:57:56.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.376+0000 7f16549f5640 1 -- 172.21.15.57:0/1457346375 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16500690e0 con 0x7f1650158910 2023-10-19T12:57:56.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.376+0000 7f16477fe640 1 -- 172.21.15.57:0/1457346375 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 20) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f16380053e0 con 0x7f1650158910 2023-10-19T12:57:56.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.377+0000 7f16477fe640 1 --2- 172.21.15.57:0/1457346375 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f163006af60 0x7f163006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:56.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.377+0000 7f16477fe640 1 -- 172.21.15.57:0/1457346375 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f1638089aa0 con 0x7f1650158910 2023-10-19T12:57:56.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.377+0000 7f164e59c640 1 --2- 172.21.15.57:0/1457346375 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f163006af60 0x7f163006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:56.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.377+0000 7f164e59c640 1 --2- 172.21.15.57:0/1457346375 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f163006af60 0x7f163006d420 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1640001bc0 tx=0x7f1640009040 comp rx=0 tx=0).ready entity=mgr.5809 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:56.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.381+0000 7f16477fe640 1 -- 172.21.15.57:0/1457346375 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f1638054cb0 con 0x7f1650158910 2023-10-19T12:57:56.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.559+0000 7f16549f5640 1 -- 172.21.15.57:0/1457346375 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f16500c9ba0 con 0x7f1650158910 2023-10-19T12:57:56.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.560+0000 7f16477fe640 1 -- 172.21.15.57:0/1457346375 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v66) v1 ==== 84+0+65 (secure 0 0 0) 0x7f163801d070 con 0x7f1650158910 2023-10-19T12:57:56.563 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:57:56.563 INFO:teuthology.orchestra.run.smithi057.stdout:[ 2023-10-19T12:57:56.564 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T12:57:56.564 INFO:teuthology.orchestra.run.smithi057.stdout: "poolnum": 1, 2023-10-19T12:57:56.564 INFO:teuthology.orchestra.run.smithi057.stdout: "poolname": ".mgr" 2023-10-19T12:57:56.564 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T12:57:56.564 INFO:teuthology.orchestra.run.smithi057.stdout:] 2023-10-19T12:57:56.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.561+0000 7f16549f5640 1 -- 172.21.15.57:0/1457346375 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f163006af60 msgr2=0x7f163006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:56.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.561+0000 7f16549f5640 1 --2- 172.21.15.57:0/1457346375 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f163006af60 0x7f163006d420 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1640001bc0 tx=0x7f1640009040 comp rx=0 tx=0).stop 2023-10-19T12:57:56.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.561+0000 7f16549f5640 1 -- 172.21.15.57:0/1457346375 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1650158910 msgr2=0x7f1650155110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:56.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.561+0000 7f16549f5640 1 --2- 172.21.15.57:0/1457346375 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1650158910 0x7f1650155110 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f163800c970 tx=0x7f163800ce30 comp rx=0 tx=0).stop 2023-10-19T12:57:56.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.561+0000 7f16549f5640 1 -- 172.21.15.57:0/1457346375 shutdown_connections 2023-10-19T12:57:56.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.561+0000 7f16549f5640 1 --2- 172.21.15.57:0/1457346375 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f163006af60 0x7f163006d420 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:56.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.561+0000 7f16549f5640 1 --2- 172.21.15.57:0/1457346375 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1650158cf0 0x7f16500ae840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:56.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.561+0000 7f16549f5640 1 --2- 172.21.15.57:0/1457346375 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1650158910 0x7f1650155110 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:56.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.561+0000 7f16549f5640 1 --2- 172.21.15.57:0/1457346375 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f16500cd9e0 0x7f1650149270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:56.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.561+0000 7f16549f5640 1 -- 172.21.15.57:0/1457346375 >> 172.21.15.57:0/1457346375 conn(0x7f16500af7a0 msgr2=0x7f16500cc190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:56.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.561+0000 7f16549f5640 1 -- 172.21.15.57:0/1457346375 shutdown_connections 2023-10-19T12:57:56.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.561+0000 7f16549f5640 1 -- 172.21.15.57:0/1457346375 wait complete. 2023-10-19T12:57:56.575 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-19T12:57:56.575 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-19T12:57:56.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.686+0000 7f437ec7c640 1 -- 172.21.15.57:0/1079553821 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4378070350 msgr2=0x7f43780707e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:56.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.686+0000 7f437ec7c640 1 --2- 172.21.15.57:0/1079553821 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4378070350 0x7f43780707e0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f4370005fd0 tx=0x7f437002c2e0 comp rx=0 tx=0).stop 2023-10-19T12:57:56.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.686+0000 7f437ec7c640 1 -- 172.21.15.57:0/1079553821 shutdown_connections 2023-10-19T12:57:56.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.686+0000 7f437ec7c640 1 --2- 172.21.15.57:0/1079553821 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4378070350 0x7f43780707e0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:56.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.686+0000 7f437ec7c640 1 --2- 172.21.15.57:0/1079553821 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f437813d350 0x7f43780cbd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:56.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.686+0000 7f437ec7c640 1 -- 172.21.15.57:0/1079553821 >> 172.21.15.57:0/1079553821 conn(0x7f4378069930 msgr2=0x7f4378069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:56.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.686+0000 7f437ec7c640 1 -- 172.21.15.57:0/1079553821 shutdown_connections 2023-10-19T12:57:56.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.686+0000 7f437ec7c640 1 -- 172.21.15.57:0/1079553821 wait complete. 2023-10-19T12:57:56.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.687+0000 7f437ec7c640 1 Processor -- start 2023-10-19T12:57:56.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.687+0000 7f437ec7c640 1 -- start start 2023-10-19T12:57:56.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.688+0000 7f437ec7c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4378070350 0x7f4378136380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:56.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.688+0000 7f437ec7c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f43780cb980 0x7f4378159140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:56.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.688+0000 7f437ec7c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f437814fc70 0x7f4378150f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:56.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.688+0000 7f437ec7c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43781373a0 con 0x7f437814fc70 2023-10-19T12:57:56.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.688+0000 7f437ec7c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4378137680 con 0x7f4378070350 2023-10-19T12:57:56.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.688+0000 7f437ec7c640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4378137960 con 0x7f43780cb980 2023-10-19T12:57:56.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.688+0000 7f437d219640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f437814fc70 0x7f4378150f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:56.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.688+0000 7f437d219640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f437814fc70 0x7f4378150f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:41584/0 (socket says 172.21.15.57:41584) 2023-10-19T12:57:56.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.688+0000 7f437d219640 1 -- 172.21.15.57:0/3026887653 learned_addr learned my addr 172.21.15.57:0/3026887653 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:57:56.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.688+0000 7f437ca18640 1 --2- 172.21.15.57:0/3026887653 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4378070350 0x7f4378136380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:56.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.688+0000 7f4377fff640 1 --2- 172.21.15.57:0/3026887653 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f43780cb980 0x7f4378159140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:56.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.689+0000 7f437d219640 1 -- 172.21.15.57:0/3026887653 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f43780cb980 msgr2=0x7f4378159140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:56.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.689+0000 7f437d219640 1 --2- 172.21.15.57:0/3026887653 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f43780cb980 0x7f4378159140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:56.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.689+0000 7f437d219640 1 -- 172.21.15.57:0/3026887653 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4378070350 msgr2=0x7f4378136380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:56.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.689+0000 7f437d219640 1 --2- 172.21.15.57:0/3026887653 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4378070350 0x7f4378136380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:56.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.689+0000 7f437d219640 1 -- 172.21.15.57:0/3026887653 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4370009900 con 0x7f437814fc70 2023-10-19T12:57:56.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.689+0000 7f437ca18640 1 --2- 172.21.15.57:0/3026887653 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4378070350 0x7f4378136380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:57:56.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.689+0000 7f437d219640 1 --2- 172.21.15.57:0/3026887653 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f437814fc70 0x7f4378150f70 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f4370009bf0 tx=0x7f4370035b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:56.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.689+0000 7f43757fa640 1 -- 172.21.15.57:0/3026887653 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4370004240 con 0x7f437814fc70 2023-10-19T12:57:56.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.689+0000 7f43757fa640 1 -- 172.21.15.57:0/3026887653 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f4370004490 con 0x7f437814fc70 2023-10-19T12:57:56.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.689+0000 7f437ec7c640 1 -- 172.21.15.57:0/3026887653 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4378137c40 con 0x7f437814fc70 2023-10-19T12:57:56.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.689+0000 7f43757fa640 1 -- 172.21.15.57:0/3026887653 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4370004860 con 0x7f437814fc70 2023-10-19T12:57:56.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.689+0000 7f437ec7c640 1 -- 172.21.15.57:0/3026887653 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4378139bc0 con 0x7f437814fc70 2023-10-19T12:57:56.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.690+0000 7f437ec7c640 1 -- 172.21.15.57:0/3026887653 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43781374e0 con 0x7f437814fc70 2023-10-19T12:57:56.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.690+0000 7f43757fa640 1 -- 172.21.15.57:0/3026887653 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 20) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f437003b050 con 0x7f437814fc70 2023-10-19T12:57:56.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.691+0000 7f43757fa640 1 --2- 172.21.15.57:0/3026887653 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f435c06af60 0x7f435c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:56.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.691+0000 7f43757fa640 1 -- 172.21.15.57:0/3026887653 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f43700ac2d0 con 0x7f437814fc70 2023-10-19T12:57:56.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.691+0000 7f437ca18640 1 --2- 172.21.15.57:0/3026887653 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f435c06af60 0x7f435c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:56.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.692+0000 7f437ca18640 1 --2- 172.21.15.57:0/3026887653 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f435c06af60 0x7f435c06d420 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f436c001700 tx=0x7f436c009040 comp rx=0 tx=0).ready entity=mgr.5809 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:56.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.695+0000 7f43757fa640 1 -- 172.21.15.57:0/3026887653 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f43781374e0 con 0x7f437814fc70 2023-10-19T12:57:56.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:56.869+0000 7f437ec7c640 1 -- 172.21.15.57:0/3026887653 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f43781374e0 con 0x7f437814fc70 2023-10-19T12:57:57.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:57.228+0000 7f43757fa640 1 -- 172.21.15.57:0/3026887653 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v67) v1 ==== 138+0+0 (secure 0 0 0) 0x7f43781374e0 con 0x7f437814fc70 2023-10-19T12:57:57.232 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'cephfs_metadata' created 2023-10-19T12:57:57.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:57.231+0000 7f437ec7c640 1 -- 172.21.15.57:0/3026887653 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f435c06af60 msgr2=0x7f435c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:57.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:57.231+0000 7f437ec7c640 1 --2- 172.21.15.57:0/3026887653 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f435c06af60 0x7f435c06d420 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f436c001700 tx=0x7f436c009040 comp rx=0 tx=0).stop 2023-10-19T12:57:57.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:57.231+0000 7f437ec7c640 1 -- 172.21.15.57:0/3026887653 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f437814fc70 msgr2=0x7f4378150f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:57.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:57.231+0000 7f437ec7c640 1 --2- 172.21.15.57:0/3026887653 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f437814fc70 0x7f4378150f70 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f4370009bf0 tx=0x7f4370035b50 comp rx=0 tx=0).stop 2023-10-19T12:57:57.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:57.231+0000 7f437ec7c640 1 -- 172.21.15.57:0/3026887653 shutdown_connections 2023-10-19T12:57:57.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:57.231+0000 7f437ec7c640 1 --2- 172.21.15.57:0/3026887653 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f435c06af60 0x7f435c06d420 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:57.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:57.231+0000 7f437ec7c640 1 --2- 172.21.15.57:0/3026887653 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f437814fc70 0x7f4378150f70 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:57.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:57.231+0000 7f437ec7c640 1 --2- 172.21.15.57:0/3026887653 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f43780cb980 0x7f4378159140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:57.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:57.231+0000 7f437ec7c640 1 --2- 172.21.15.57:0/3026887653 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4378070350 0x7f4378136380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:57.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:57.231+0000 7f437ec7c640 1 -- 172.21.15.57:0/3026887653 >> 172.21.15.57:0/3026887653 conn(0x7f4378069930 msgr2=0x7f4378156d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:57.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:57.232+0000 7f437ec7c640 1 -- 172.21.15.57:0/3026887653 shutdown_connections 2023-10-19T12:57:57.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:57.232+0000 7f437ec7c640 1 -- 172.21.15.57:0/3026887653 wait complete. 2023-10-19T12:57:57.249 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-19T12:57:57.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:57.364+0000 7f0ad2fd7640 1 -- 172.21.15.57:0/587593309 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0acc0ceb40 msgr2=0x7f0acc0b0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:57.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:57.364+0000 7f0ad2fd7640 1 --2- 172.21.15.57:0/587593309 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0acc0ceb40 0x7f0acc0b0650 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f0acc08a120 tx=0x7f0abc02bdb0 comp rx=0 tx=0).stop 2023-10-19T12:57:57.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:57.364+0000 7f0ad2fd7640 1 -- 172.21.15.57:0/587593309 shutdown_connections 2023-10-19T12:57:57.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:57.364+0000 7f0ad2fd7640 1 --2- 172.21.15.57:0/587593309 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0acc0ceb40 0x7f0acc0b0650 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:57.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:57.364+0000 7f0ad2fd7640 1 --2- 172.21.15.57:0/587593309 >> v2:172.21.15.179:3300/0 conn(0x7f0acc15b2a0 0x7f0acc0d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:57.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:57.364+0000 7f0ad2fd7640 1 -- 172.21.15.57:0/587593309 >> 172.21.15.57:0/587593309 conn(0x7f0acc0d0d60 msgr2=0x7f0acc0d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:57.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:57.364+0000 7f0ad2fd7640 1 -- 172.21.15.57:0/587593309 shutdown_connections 2023-10-19T12:57:57.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:57.364+0000 7f0ad2fd7640 1 -- 172.21.15.57:0/587593309 wait complete. 2023-10-19T12:57:57.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:57.365+0000 7f0ad2fd7640 1 Processor -- start 2023-10-19T12:57:57.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:57.365+0000 7f0ad2fd7640 1 -- start start 2023-10-19T12:57:57.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:57.366+0000 7f0ad2fd7640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0acc0ceb40 0x7f0acc15c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:57.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:57.366+0000 7f0ad2fd7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0acc0cf340 0x7f0acc15b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:57.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:57.366+0000 7f0ad2fd7640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0acc15b2a0 0x7f0acc0b4930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:57.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:57.366+0000 7f0ad2fd7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0acc15c920 con 0x7f0acc0cf340 2023-10-19T12:57:57.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:57.366+0000 7f0ad2fd7640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0acc150c50 con 0x7f0acc0ceb40 2023-10-19T12:57:57.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:57.366+0000 7f0ad2fd7640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0acc15bbc0 con 0x7f0acc15b2a0 2023-10-19T12:57:57.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:57.367+0000 7f0acbfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0acc0cf340 0x7f0acc15b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:57.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:57.367+0000 7f0acbfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0acc0cf340 0x7f0acc15b680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:41598/0 (socket says 172.21.15.57:41598) 2023-10-19T12:57:57.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:57.367+0000 7f0acbfff640 1 -- 172.21.15.57:0/3500683406 learned_addr learned my addr 172.21.15.57:0/3500683406 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:57:57.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:57.367+0000 7f0ad1574640 1 --2- 172.21.15.57:0/3500683406 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0acc15b2a0 0x7f0acc0b4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:57.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:57.367+0000 7f0ad0d73640 1 --2- 172.21.15.57:0/3500683406 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0acc0ceb40 0x7f0acc15c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:57.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:57.367+0000 7f0acbfff640 1 -- 172.21.15.57:0/3500683406 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0acc15b2a0 msgr2=0x7f0acc0b4930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:57.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:57.367+0000 7f0acbfff640 1 --2- 172.21.15.57:0/3500683406 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0acc15b2a0 0x7f0acc0b4930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:57.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:57.367+0000 7f0acbfff640 1 -- 172.21.15.57:0/3500683406 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0acc0ceb40 msgr2=0x7f0acc15c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:57.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:57.367+0000 7f0acbfff640 1 --2- 172.21.15.57:0/3500683406 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0acc0ceb40 0x7f0acc15c3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:57.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:57.367+0000 7f0acbfff640 1 -- 172.21.15.57:0/3500683406 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0abc009560 con 0x7f0acc0cf340 2023-10-19T12:57:57.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:57.367+0000 7f0ad0d73640 1 --2- 172.21.15.57:0/3500683406 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0acc0ceb40 0x7f0acc15c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:57:57.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:57.367+0000 7f0acbfff640 1 --2- 172.21.15.57:0/3500683406 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0acc0cf340 0x7f0acc15b680 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f0ac000d6d0 tx=0x7f0ac000db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:57.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:57.367+0000 7f0ac97fa640 1 -- 172.21.15.57:0/3500683406 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ac0005220 con 0x7f0acc0cf340 2023-10-19T12:57:57.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:57.367+0000 7f0ac97fa640 1 -- 172.21.15.57:0/3500683406 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f0ac0005490 con 0x7f0acc0cf340 2023-10-19T12:57:57.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:57.367+0000 7f0ad2fd7640 1 -- 172.21.15.57:0/3500683406 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0acc0b5180 con 0x7f0acc0cf340 2023-10-19T12:57:57.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:57.367+0000 7f0ac97fa640 1 -- 172.21.15.57:0/3500683406 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ac0009a80 con 0x7f0acc0cf340 2023-10-19T12:57:57.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:57.367+0000 7f0ad2fd7640 1 -- 172.21.15.57:0/3500683406 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0acc0b5740 con 0x7f0acc0cf340 2023-10-19T12:57:57.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:57.368+0000 7f0ad2fd7640 1 -- 172.21.15.57:0/3500683406 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0acc150dc0 con 0x7f0acc0cf340 2023-10-19T12:57:57.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:57.369+0000 7f0ac97fa640 1 -- 172.21.15.57:0/3500683406 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 20) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f0ac0009be0 con 0x7f0acc0cf340 2023-10-19T12:57:57.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:57.369+0000 7f0ac97fa640 1 --2- 172.21.15.57:0/3500683406 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f0ab806af60 0x7f0ab806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:57.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:57.369+0000 7f0ac97fa640 1 -- 172.21.15.57:0/3500683406 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5547+0+0 (secure 0 0 0) 0x7f0ac00898d0 con 0x7f0acc0cf340 2023-10-19T12:57:57.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:57.370+0000 7f0ad0d73640 1 --2- 172.21.15.57:0/3500683406 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f0ab806af60 0x7f0ab806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:57.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:57.370+0000 7f0ad0d73640 1 --2- 172.21.15.57:0/3500683406 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f0ab806af60 0x7f0ab806d420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0ac40012e0 tx=0x7f0ac4009040 comp rx=0 tx=0).ready entity=mgr.5809 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:57.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:57.374+0000 7f0ac97fa640 1 -- 172.21.15.57:0/3500683406 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0ac00548a0 con 0x7f0acc0cf340 2023-10-19T12:57:57.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:57.550+0000 7f0ad2fd7640 1 -- 172.21.15.57:0/3500683406 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f0acc0b0d00 con 0x7f0acc0cf340 2023-10-19T12:57:58.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:58.235+0000 7f0ac97fa640 1 -- 172.21.15.57:0/3500683406 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v68) v1 ==== 170+0+0 (secure 0 0 0) 0x7f0ac0017020 con 0x7f0acc0cf340 2023-10-19T12:57:58.238 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'cephfs_data' created 2023-10-19T12:57:58.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:58.237+0000 7f0ad2fd7640 1 -- 172.21.15.57:0/3500683406 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f0ab806af60 msgr2=0x7f0ab806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:58.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:58.237+0000 7f0ad2fd7640 1 --2- 172.21.15.57:0/3500683406 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f0ab806af60 0x7f0ab806d420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0ac40012e0 tx=0x7f0ac4009040 comp rx=0 tx=0).stop 2023-10-19T12:57:58.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:58.237+0000 7f0ad2fd7640 1 -- 172.21.15.57:0/3500683406 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0acc0cf340 msgr2=0x7f0acc15b680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:58.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:58.237+0000 7f0ad2fd7640 1 --2- 172.21.15.57:0/3500683406 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0acc0cf340 0x7f0acc15b680 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f0ac000d6d0 tx=0x7f0ac000db90 comp rx=0 tx=0).stop 2023-10-19T12:57:58.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:58.237+0000 7f0ad2fd7640 1 -- 172.21.15.57:0/3500683406 shutdown_connections 2023-10-19T12:57:58.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:58.237+0000 7f0ad2fd7640 1 --2- 172.21.15.57:0/3500683406 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f0ab806af60 0x7f0ab806d420 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:58.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:58.237+0000 7f0ad2fd7640 1 --2- 172.21.15.57:0/3500683406 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0acc15b2a0 0x7f0acc0b4930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:58.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:58.237+0000 7f0ad2fd7640 1 --2- 172.21.15.57:0/3500683406 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0acc0cf340 0x7f0acc15b680 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:58.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:58.237+0000 7f0ad2fd7640 1 --2- 172.21.15.57:0/3500683406 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0acc0ceb40 0x7f0acc15c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:58.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:58.237+0000 7f0ad2fd7640 1 -- 172.21.15.57:0/3500683406 >> 172.21.15.57:0/3500683406 conn(0x7f0acc0d0d60 msgr2=0x7f0acc06b500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:58.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:58.238+0000 7f0ad2fd7640 1 -- 172.21.15.57:0/3500683406 shutdown_connections 2023-10-19T12:57:58.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:58.238+0000 7f0ad2fd7640 1 -- 172.21.15.57:0/3500683406 wait complete. 2023-10-19T12:57:58.268 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-19T12:57:58.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:58.381+0000 7f89a39d7640 1 -- 172.21.15.57:0/3367771661 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f89880047c0 msgr2=0x7f8988004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:58.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:58.381+0000 7f89a39d7640 1 --2- 172.21.15.57:0/3367771661 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f89880047c0 0x7f8988004c40 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f898c001680 tx=0x7f89880057f0 comp rx=0 tx=0).stop 2023-10-19T12:57:58.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:58.381+0000 7f89a39d7640 1 -- 172.21.15.57:0/3367771661 shutdown_connections 2023-10-19T12:57:58.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:58.381+0000 7f89a39d7640 1 --2- 172.21.15.57:0/3367771661 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f89880047c0 0x7f8988004c40 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:58.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:58.381+0000 7f89a39d7640 1 -- 172.21.15.57:0/3367771661 >> 172.21.15.57:0/3367771661 conn(0x7f899c0aec40 msgr2=0x7f899c0af050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:58.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:58.381+0000 7f89a39d7640 1 -- 172.21.15.57:0/3367771661 shutdown_connections 2023-10-19T12:57:58.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:58.381+0000 7f89a39d7640 1 -- 172.21.15.57:0/3367771661 wait complete. 2023-10-19T12:57:58.391 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:58.382+0000 7f89a39d7640 1 Processor -- start 2023-10-19T12:57:58.392 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:58.382+0000 7f89a39d7640 1 -- start start 2023-10-19T12:57:58.392 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:58.383+0000 7f89a39d7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f89880047c0 0x7f899c14cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:58.392 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:58.383+0000 7f89a39d7640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f899c1ef4d0 0x7f899c1ef930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:58.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:58.383+0000 7f89a39d7640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f899c15c680 0x7f899c15cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:58.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:58.383+0000 7f89a39d7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f899c15d050 con 0x7f89880047c0 2023-10-19T12:57:58.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:58.383+0000 7f89a39d7640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f899c1f3850 con 0x7f899c15c680 2023-10-19T12:57:58.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:58.383+0000 7f89a39d7640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f899c0aa530 con 0x7f899c1ef4d0 2023-10-19T12:57:58.394 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:58.383+0000 7f89a1773640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f89880047c0 0x7f899c14cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:58.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:58.383+0000 7f89a1773640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f89880047c0 0x7f899c14cb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:41628/0 (socket says 172.21.15.57:41628) 2023-10-19T12:57:58.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:58.383+0000 7f89a1773640 1 -- 172.21.15.57:0/3029525426 learned_addr learned my addr 172.21.15.57:0/3029525426 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:57:58.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:58.383+0000 7f89a0f72640 1 --2- 172.21.15.57:0/3029525426 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f899c1ef4d0 0x7f899c1ef930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:58.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:58.383+0000 7f89a1f74640 1 --2- 172.21.15.57:0/3029525426 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f899c15c680 0x7f899c15cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:58.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:58.383+0000 7f89a1773640 1 -- 172.21.15.57:0/3029525426 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f899c1ef4d0 msgr2=0x7f899c1ef930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:58.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:58.383+0000 7f89a1773640 1 --2- 172.21.15.57:0/3029525426 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f899c1ef4d0 0x7f899c1ef930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:58.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:58.383+0000 7f89a1773640 1 -- 172.21.15.57:0/3029525426 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f899c15c680 msgr2=0x7f899c15cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:58.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:58.383+0000 7f89a1773640 1 --2- 172.21.15.57:0/3029525426 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f899c15c680 0x7f899c15cb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:58.397 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:58.383+0000 7f89a1773640 1 -- 172.21.15.57:0/3029525426 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f899c0cae60 con 0x7f89880047c0 2023-10-19T12:57:58.397 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:58.383+0000 7f89a0f72640 1 --2- 172.21.15.57:0/3029525426 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f899c1ef4d0 0x7f899c1ef930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:57:58.397 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:58.383+0000 7f89a1f74640 1 --2- 172.21.15.57:0/3029525426 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f899c15c680 0x7f899c15cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:57:58.397 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:58.384+0000 7f89a1773640 1 --2- 172.21.15.57:0/3029525426 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f89880047c0 0x7f899c14cb20 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f89980011e0 tx=0x7f89980039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:58.397 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:58.384+0000 7f8991ffb640 1 -- 172.21.15.57:0/3029525426 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8998011900 con 0x7f89880047c0 2023-10-19T12:57:58.398 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:58.384+0000 7f8991ffb640 1 -- 172.21.15.57:0/3029525426 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f8998011a80 con 0x7f89880047c0 2023-10-19T12:57:58.398 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:58.384+0000 7f89a39d7640 1 -- 172.21.15.57:0/3029525426 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f899c0aa810 con 0x7f89880047c0 2023-10-19T12:57:58.398 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:58.384+0000 7f8991ffb640 1 -- 172.21.15.57:0/3029525426 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8998011e90 con 0x7f89880047c0 2023-10-19T12:57:58.398 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:58.384+0000 7f89a39d7640 1 -- 172.21.15.57:0/3029525426 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f899c0aadd0 con 0x7f89880047c0 2023-10-19T12:57:58.398 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:58.385+0000 7f89a39d7640 1 -- 172.21.15.57:0/3029525426 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f899c0678d0 con 0x7f89880047c0 2023-10-19T12:57:58.399 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:58.386+0000 7f8991ffb640 1 -- 172.21.15.57:0/3029525426 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 20) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f89980103c0 con 0x7f89880047c0 2023-10-19T12:57:58.399 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:58.386+0000 7f8991ffb640 1 --2- 172.21.15.57:0/3029525426 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f898006af90 0x7f898006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:58.399 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:58.386+0000 7f8991ffb640 1 -- 172.21.15.57:0/3029525426 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5929+0+0 (secure 0 0 0) 0x7f899808ab80 con 0x7f89880047c0 2023-10-19T12:57:58.399 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:58.387+0000 7f89a0f72640 1 --2- 172.21.15.57:0/3029525426 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f898006af90 0x7f898006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:58.399 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:58.387+0000 7f89a0f72640 1 --2- 172.21.15.57:0/3029525426 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f898006af90 0x7f898006d450 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f898c034000 tx=0x7f898c039040 comp rx=0 tx=0).ready entity=mgr.5809 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:58.400 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:58.391+0000 7f8991ffb640 1 -- 172.21.15.57:0/3029525426 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8998055940 con 0x7f89880047c0 2023-10-19T12:57:58.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:58.604+0000 7f89a39d7640 1 -- 172.21.15.57:0/3029525426 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f899c0690e0 con 0x7f89880047c0 2023-10-19T12:57:59.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.262+0000 7f8991ffb640 1 -- 172.21.15.57:0/3029525426 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 7 and data pool 8 v21) v1 ==== 172+0+0 (secure 0 0 0) 0x7f8998058f60 con 0x7f89880047c0 2023-10-19T12:57:59.272 INFO:teuthology.orchestra.run.smithi057.stderr:new fs with metadata pool 7 and data pool 8 2023-10-19T12:57:59.272 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.270+0000 7f89a39d7640 1 -- 172.21.15.57:0/3029525426 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f898006af90 msgr2=0x7f898006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:59.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.270+0000 7f89a39d7640 1 --2- 172.21.15.57:0/3029525426 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f898006af90 0x7f898006d450 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f898c034000 tx=0x7f898c039040 comp rx=0 tx=0).stop 2023-10-19T12:57:59.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.270+0000 7f89a39d7640 1 -- 172.21.15.57:0/3029525426 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f89880047c0 msgr2=0x7f899c14cb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:59.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.270+0000 7f89a39d7640 1 --2- 172.21.15.57:0/3029525426 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f89880047c0 0x7f899c14cb20 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f89980011e0 tx=0x7f89980039c0 comp rx=0 tx=0).stop 2023-10-19T12:57:59.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.272+0000 7f89a39d7640 1 -- 172.21.15.57:0/3029525426 shutdown_connections 2023-10-19T12:57:59.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.272+0000 7f89a39d7640 1 --2- 172.21.15.57:0/3029525426 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f898006af90 0x7f898006d450 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:59.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.272+0000 7f89a39d7640 1 --2- 172.21.15.57:0/3029525426 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f899c15c680 0x7f899c15cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:59.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.272+0000 7f89a39d7640 1 --2- 172.21.15.57:0/3029525426 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f899c1ef4d0 0x7f899c1ef930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:59.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.272+0000 7f89a39d7640 1 --2- 172.21.15.57:0/3029525426 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f89880047c0 0x7f899c14cb20 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:59.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.272+0000 7f89a39d7640 1 -- 172.21.15.57:0/3029525426 >> 172.21.15.57:0/3029525426 conn(0x7f899c0aec40 msgr2=0x7f899c0b0fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:59.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.275+0000 7f89a39d7640 1 -- 172.21.15.57:0/3029525426 shutdown_connections 2023-10-19T12:57:59.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.275+0000 7f89a39d7640 1 -- 172.21.15.57:0/3029525426 wait complete. 2023-10-19T12:57:59.301 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T12:57:59.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.411+0000 7f6157e7c640 1 -- 172.21.15.57:0/1401918985 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f61500c9ed0 msgr2=0x7f615006c1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:59.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.411+0000 7f6157e7c640 1 --2- 172.21.15.57:0/1401918985 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f61500c9ed0 0x7f615006c1a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6148005fd0 tx=0x7f614802c0e0 comp rx=0 tx=0).stop 2023-10-19T12:57:59.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.411+0000 7f6157e7c640 1 -- 172.21.15.57:0/1401918985 shutdown_connections 2023-10-19T12:57:59.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.411+0000 7f6157e7c640 1 --2- 172.21.15.57:0/1401918985 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f61500c9ed0 0x7f615006c1a0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:59.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.411+0000 7f6157e7c640 1 --2- 172.21.15.57:0/1401918985 >> v2:172.21.15.179:3300/0 conn(0x7f615013d360 0x7f615015a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:59.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.411+0000 7f6157e7c640 1 -- 172.21.15.57:0/1401918985 >> 172.21.15.57:0/1401918985 conn(0x7f6150069930 msgr2=0x7f6150069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:59.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.411+0000 7f6157e7c640 1 -- 172.21.15.57:0/1401918985 shutdown_connections 2023-10-19T12:57:59.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.411+0000 7f6157e7c640 1 -- 172.21.15.57:0/1401918985 wait complete. 2023-10-19T12:57:59.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.412+0000 7f6157e7c640 1 Processor -- start 2023-10-19T12:57:59.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.412+0000 7f6157e7c640 1 -- start start 2023-10-19T12:57:59.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.413+0000 7f6157e7c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f61500c9ed0 0x7f615014ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:59.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.413+0000 7f6157e7c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f615013d360 0x7f6150132a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:59.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.413+0000 7f6157e7c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6150140660 0x7f61500b09a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:59.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.413+0000 7f6157e7c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6150140c70 con 0x7f61500c9ed0 2023-10-19T12:57:59.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.413+0000 7f6157e7c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6150140ec0 con 0x7f615013d360 2023-10-19T12:57:59.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.413+0000 7f6157e7c640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61501411a0 con 0x7f6150140660 2023-10-19T12:57:59.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.413+0000 7f6155c18640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f61500c9ed0 0x7f615014ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:59.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.413+0000 7f6155c18640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f61500c9ed0 0x7f615014ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:41634/0 (socket says 172.21.15.57:41634) 2023-10-19T12:57:59.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.413+0000 7f6155c18640 1 -- 172.21.15.57:0/4262221698 learned_addr learned my addr 172.21.15.57:0/4262221698 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:57:59.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.413+0000 7f6155417640 1 --2- 172.21.15.57:0/4262221698 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f615013d360 0x7f6150132a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:59.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.413+0000 7f6156419640 1 --2- 172.21.15.57:0/4262221698 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6150140660 0x7f61500b09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:59.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.414+0000 7f6155c18640 1 -- 172.21.15.57:0/4262221698 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6150140660 msgr2=0x7f61500b09a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:59.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.414+0000 7f6155c18640 1 --2- 172.21.15.57:0/4262221698 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6150140660 0x7f61500b09a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:59.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.414+0000 7f6155c18640 1 -- 172.21.15.57:0/4262221698 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f615013d360 msgr2=0x7f6150132a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:59.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.414+0000 7f6155c18640 1 --2- 172.21.15.57:0/4262221698 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f615013d360 0x7f6150132a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:59.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.414+0000 7f6155c18640 1 -- 172.21.15.57:0/4262221698 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f614c009560 con 0x7f61500c9ed0 2023-10-19T12:57:59.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.414+0000 7f6155c18640 1 --2- 172.21.15.57:0/4262221698 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f61500c9ed0 0x7f615014ef40 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f614c00e9c0 tx=0x7f614c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:59.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.414+0000 7f61467fc640 1 -- 172.21.15.57:0/4262221698 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f614c00cde0 con 0x7f61500c9ed0 2023-10-19T12:57:59.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.414+0000 7f6157e7c640 1 -- 172.21.15.57:0/4262221698 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6148009840 con 0x7f61500c9ed0 2023-10-19T12:57:59.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.414+0000 7f61467fc640 1 -- 172.21.15.57:0/4262221698 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f614c004020 con 0x7f61500c9ed0 2023-10-19T12:57:59.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.414+0000 7f61467fc640 1 -- 172.21.15.57:0/4262221698 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f614c004410 con 0x7f61500c9ed0 2023-10-19T12:57:59.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.414+0000 7f6157e7c640 1 -- 172.21.15.57:0/4262221698 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6150141760 con 0x7f61500c9ed0 2023-10-19T12:57:59.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.415+0000 7f6157e7c640 1 -- 172.21.15.57:0/4262221698 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f615006f060 con 0x7f61500c9ed0 2023-10-19T12:57:59.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.415+0000 7f61467fc640 1 -- 172.21.15.57:0/4262221698 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 20) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f614c0035f0 con 0x7f61500c9ed0 2023-10-19T12:57:59.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.416+0000 7f61467fc640 1 --2- 172.21.15.57:0/4262221698 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f613406af90 0x7f613406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:59.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.416+0000 7f61467fc640 1 -- 172.21.15.57:0/4262221698 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f614c014070 con 0x7f61500c9ed0 2023-10-19T12:57:59.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.416+0000 7f6155417640 1 --2- 172.21.15.57:0/4262221698 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f613406af90 0x7f613406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:59.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.417+0000 7f6155417640 1 --2- 172.21.15.57:0/4262221698 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f613406af90 0x7f613406d450 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6140007fd0 tx=0x7f6140009040 comp rx=0 tx=0).ready entity=mgr.5809 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:59.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.420+0000 7f61467fc640 1 -- 172.21.15.57:0/4262221698 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f614c0552e0 con 0x7f61500c9ed0 2023-10-19T12:57:59.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.597+0000 7f6157e7c640 1 -- 172.21.15.57:0/4262221698 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f61500690e0 con 0x7f61500c9ed0 2023-10-19T12:57:59.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.599+0000 7f61467fc640 1 -- 172.21.15.57:0/4262221698 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v69) v1 ==== 74+0+16799 (secure 0 0 0) 0x7f614c058900 con 0x7f61500c9ed0 2023-10-19T12:57:59.601 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:57:59.602 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":69,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T12:57:58.606306+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2023-10-19T12:57:56.901606+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2023-10-19T12:57:57.581428+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T12:57:59.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.601+0000 7f6157e7c640 1 -- 172.21.15.57:0/4262221698 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f613406af90 msgr2=0x7f613406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:59.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.601+0000 7f6157e7c640 1 --2- 172.21.15.57:0/4262221698 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f613406af90 0x7f613406d450 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6140007fd0 tx=0x7f6140009040 comp rx=0 tx=0).stop 2023-10-19T12:57:59.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.601+0000 7f6157e7c640 1 -- 172.21.15.57:0/4262221698 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f61500c9ed0 msgr2=0x7f615014ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:59.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.601+0000 7f6157e7c640 1 --2- 172.21.15.57:0/4262221698 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f61500c9ed0 0x7f615014ef40 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f614c00e9c0 tx=0x7f614c00ee80 comp rx=0 tx=0).stop 2023-10-19T12:57:59.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.601+0000 7f6157e7c640 1 -- 172.21.15.57:0/4262221698 shutdown_connections 2023-10-19T12:57:59.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.601+0000 7f6157e7c640 1 --2- 172.21.15.57:0/4262221698 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f613406af90 0x7f613406d450 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:59.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.601+0000 7f6157e7c640 1 --2- 172.21.15.57:0/4262221698 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6150140660 0x7f61500b09a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:59.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.601+0000 7f6157e7c640 1 --2- 172.21.15.57:0/4262221698 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f615013d360 0x7f6150132a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:59.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.601+0000 7f6157e7c640 1 --2- 172.21.15.57:0/4262221698 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f61500c9ed0 0x7f615014ef40 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:59.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.601+0000 7f6157e7c640 1 -- 172.21.15.57:0/4262221698 >> 172.21.15.57:0/4262221698 conn(0x7f6150069930 msgr2=0x7f6150157870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:59.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.601+0000 7f6157e7c640 1 -- 172.21.15.57:0/4262221698 shutdown_connections 2023-10-19T12:57:59.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.601+0000 7f6157e7c640 1 -- 172.21.15.57:0/4262221698 wait complete. 2023-10-19T12:57:59.614 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T12:57:59.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.723+0000 7fec9854c640 1 -- 172.21.15.57:0/831437783 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec7c0047c0 msgr2=0x7fec7c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:59.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.723+0000 7fec9854c640 1 --2- 172.21.15.57:0/831437783 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec7c0047c0 0x7fec7c004c40 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fec80009b50 tx=0x7fec7c0057f0 comp rx=0 tx=0).stop 2023-10-19T12:57:59.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.723+0000 7fec96ae9640 1 -- 172.21.15.57:0/831437783 reap_dead start 2023-10-19T12:57:59.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.723+0000 7fec9854c640 1 -- 172.21.15.57:0/831437783 shutdown_connections 2023-10-19T12:57:59.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.723+0000 7fec9854c640 1 -- 172.21.15.57:0/831437783 >> 172.21.15.57:0/831437783 conn(0x7fec9006fae0 msgr2=0x7fec9006fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:59.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.723+0000 7fec9854c640 1 -- 172.21.15.57:0/831437783 shutdown_connections 2023-10-19T12:57:59.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.723+0000 7fec9854c640 1 -- 172.21.15.57:0/831437783 wait complete. 2023-10-19T12:57:59.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.723+0000 7fec9854c640 1 Processor -- start 2023-10-19T12:57:59.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.723+0000 7fec9854c640 1 -- start start 2023-10-19T12:57:59.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.724+0000 7fec9854c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fec7c0047c0 0x7fec900b1550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:59.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.724+0000 7fec9854c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fec90152640 0x7fec90150c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:59.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.724+0000 7fec9854c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec90144ea0 0x7fec90155410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:59.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.724+0000 7fec9854c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec900b1a90 con 0x7fec90144ea0 2023-10-19T12:57:59.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.724+0000 7fec9854c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec90152dd0 con 0x7fec90152640 2023-10-19T12:57:59.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.724+0000 7fec9854c640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec900aac30 con 0x7fec7c0047c0 2023-10-19T12:57:59.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.724+0000 7fec96ae9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec90144ea0 0x7fec90155410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:59.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.724+0000 7fec96ae9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec90144ea0 0x7fec90155410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:41646/0 (socket says 172.21.15.57:41646) 2023-10-19T12:57:59.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.724+0000 7fec96ae9640 1 -- 172.21.15.57:0/853900659 learned_addr learned my addr 172.21.15.57:0/853900659 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:57:59.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.724+0000 7fec962e8640 1 --2- 172.21.15.57:0/853900659 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fec7c0047c0 0x7fec900b1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:59.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.724+0000 7fec95ae7640 1 --2- 172.21.15.57:0/853900659 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fec90152640 0x7fec90150c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:59.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.724+0000 7fec96ae9640 1 -- 172.21.15.57:0/853900659 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fec7c0047c0 msgr2=0x7fec900b1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:59.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.724+0000 7fec96ae9640 1 --2- 172.21.15.57:0/853900659 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fec7c0047c0 0x7fec900b1550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:59.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.724+0000 7fec96ae9640 1 -- 172.21.15.57:0/853900659 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fec90152640 msgr2=0x7fec90150c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:59.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.724+0000 7fec96ae9640 1 --2- 172.21.15.57:0/853900659 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fec90152640 0x7fec90150c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:59.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.724+0000 7fec96ae9640 1 -- 172.21.15.57:0/853900659 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec900aaf10 con 0x7fec90144ea0 2023-10-19T12:57:59.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.724+0000 7fec96ae9640 1 --2- 172.21.15.57:0/853900659 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec90144ea0 0x7fec90155410 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fec90068f80 tx=0x7fec8c00f3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:59.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.724+0000 7fec86ffd640 1 -- 172.21.15.57:0/853900659 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec8c004b20 con 0x7fec90144ea0 2023-10-19T12:57:59.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.724+0000 7fec86ffd640 1 -- 172.21.15.57:0/853900659 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fec8c004c80 con 0x7fec90144ea0 2023-10-19T12:57:59.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.724+0000 7fec9854c640 1 -- 172.21.15.57:0/853900659 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec900ab1f0 con 0x7fec90144ea0 2023-10-19T12:57:59.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.724+0000 7fec86ffd640 1 -- 172.21.15.57:0/853900659 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec8c003bd0 con 0x7fec90144ea0 2023-10-19T12:57:59.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.724+0000 7fec9854c640 1 -- 172.21.15.57:0/853900659 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec90151170 con 0x7fec90144ea0 2023-10-19T12:57:59.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.726+0000 7fec9854c640 1 -- 172.21.15.57:0/853900659 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec900b1270 con 0x7fec90144ea0 2023-10-19T12:57:59.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.726+0000 7fec86ffd640 1 -- 172.21.15.57:0/853900659 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 20) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fec8c003d30 con 0x7fec90144ea0 2023-10-19T12:57:59.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.726+0000 7fec86ffd640 1 --2- 172.21.15.57:0/853900659 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7fec7406af60 0x7fec7406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:57:59.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.726+0000 7fec86ffd640 1 -- 172.21.15.57:0/853900659 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fec8c089f00 con 0x7fec90144ea0 2023-10-19T12:57:59.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.727+0000 7fec962e8640 1 --2- 172.21.15.57:0/853900659 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7fec7406af60 0x7fec7406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:57:59.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.727+0000 7fec962e8640 1 --2- 172.21.15.57:0/853900659 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7fec7406af60 0x7fec7406d420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fec88001b90 tx=0x7fec88001b20 comp rx=0 tx=0).ready entity=mgr.5809 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:57:59.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.730+0000 7fec86ffd640 1 -- 172.21.15.57:0/853900659 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fec900b1270 con 0x7fec90144ea0 2023-10-19T12:57:59.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.906+0000 7fec9854c640 1 -- 172.21.15.57:0/853900659 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fec900b1270 con 0x7fec90144ea0 2023-10-19T12:57:59.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.907+0000 7fec86ffd640 1 -- 172.21.15.57:0/853900659 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v69) v1 ==== 74+0+16799 (secure 0 0 0) 0x7fec900b1270 con 0x7fec90144ea0 2023-10-19T12:57:59.910 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:57:59.911 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":69,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T12:57:58.606306+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2023-10-19T12:57:56.901606+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2023-10-19T12:57:57.581428+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T12:57:59.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.909+0000 7fec9854c640 1 -- 172.21.15.57:0/853900659 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7fec7406af60 msgr2=0x7fec7406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:59.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.909+0000 7fec9854c640 1 --2- 172.21.15.57:0/853900659 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7fec7406af60 0x7fec7406d420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fec88001b90 tx=0x7fec88001b20 comp rx=0 tx=0).stop 2023-10-19T12:57:59.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.909+0000 7fec9854c640 1 -- 172.21.15.57:0/853900659 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec90144ea0 msgr2=0x7fec90155410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:57:59.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.909+0000 7fec9854c640 1 --2- 172.21.15.57:0/853900659 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec90144ea0 0x7fec90155410 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fec90068f80 tx=0x7fec8c00f3c0 comp rx=0 tx=0).stop 2023-10-19T12:57:59.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.909+0000 7fec9854c640 1 -- 172.21.15.57:0/853900659 shutdown_connections 2023-10-19T12:57:59.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.909+0000 7fec9854c640 1 --2- 172.21.15.57:0/853900659 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7fec7406af60 0x7fec7406d420 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:59.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.909+0000 7fec9854c640 1 --2- 172.21.15.57:0/853900659 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec90144ea0 0x7fec90155410 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:59.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.909+0000 7fec9854c640 1 --2- 172.21.15.57:0/853900659 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fec90152640 0x7fec90150c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:59.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.909+0000 7fec9854c640 1 --2- 172.21.15.57:0/853900659 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fec7c0047c0 0x7fec900b1550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:57:59.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.909+0000 7fec9854c640 1 -- 172.21.15.57:0/853900659 >> 172.21.15.57:0/853900659 conn(0x7fec9006fae0 msgr2=0x7fec90156c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:57:59.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.910+0000 7fec9854c640 1 -- 172.21.15.57:0/853900659 shutdown_connections 2023-10-19T12:57:59.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:57:59.910+0000 7fec9854c640 1 -- 172.21.15.57:0/853900659 wait complete. 2023-10-19T12:57:59.923 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-19T12:58:00.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.034+0000 7ff9e654c640 1 -- 172.21.15.57:0/3713046385 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff9e00cd1c0 msgr2=0x7ff9e015c0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:00.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.034+0000 7ff9e654c640 1 --2- 172.21.15.57:0/3713046385 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff9e00cd1c0 0x7ff9e015c0e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff9d0005fd0 tx=0x7ff9d002be10 comp rx=0 tx=0).stop 2023-10-19T12:58:00.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.034+0000 7ff9e654c640 1 -- 172.21.15.57:0/3713046385 shutdown_connections 2023-10-19T12:58:00.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.034+0000 7ff9e654c640 1 --2- 172.21.15.57:0/3713046385 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff9e00cd1c0 0x7ff9e015c0e0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:00.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.034+0000 7ff9e654c640 1 --2- 172.21.15.57:0/3713046385 >> v2:172.21.15.179:3300/0 conn(0x7ff9e0158910 0x7ff9e0158cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:00.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.034+0000 7ff9e654c640 1 -- 172.21.15.57:0/3713046385 >> 172.21.15.57:0/3713046385 conn(0x7ff9e00af7a0 msgr2=0x7ff9e00afb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:58:00.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.034+0000 7ff9e654c640 1 -- 172.21.15.57:0/3713046385 shutdown_connections 2023-10-19T12:58:00.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.034+0000 7ff9e654c640 1 -- 172.21.15.57:0/3713046385 wait complete. 2023-10-19T12:58:00.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.035+0000 7ff9e654c640 1 Processor -- start 2023-10-19T12:58:00.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.036+0000 7ff9e654c640 1 -- start start 2023-10-19T12:58:00.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.036+0000 7ff9e654c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff9e00cd1c0 0x7ff9e014a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:00.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.036+0000 7ff9e654c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff9e00cd9e0 0x7ff9e0151530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:00.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.036+0000 7ff9e654c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9e0158910 0x7ff9e0151a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:00.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.036+0000 7ff9e654c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e00aab60 con 0x7ff9e0158910 2023-10-19T12:58:00.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.036+0000 7ff9e654c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e0157800 con 0x7ff9e00cd1c0 2023-10-19T12:58:00.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.036+0000 7ff9e654c640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e0156f60 con 0x7ff9e00cd9e0 2023-10-19T12:58:00.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.036+0000 7ff9e4ae9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9e0158910 0x7ff9e0151a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:00.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.036+0000 7ff9e4ae9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9e0158910 0x7ff9e0151a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:41650/0 (socket says 172.21.15.57:41650) 2023-10-19T12:58:00.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.036+0000 7ff9e4ae9640 1 -- 172.21.15.57:0/3712888755 learned_addr learned my addr 172.21.15.57:0/3712888755 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:58:00.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.037+0000 7ff9dffff640 1 --2- 172.21.15.57:0/3712888755 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff9e00cd1c0 0x7ff9e014a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:00.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.037+0000 7ff9df7fe640 1 --2- 172.21.15.57:0/3712888755 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff9e00cd9e0 0x7ff9e0151530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:00.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.037+0000 7ff9e4ae9640 1 -- 172.21.15.57:0/3712888755 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff9e00cd9e0 msgr2=0x7ff9e0151530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:00.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.037+0000 7ff9e4ae9640 1 --2- 172.21.15.57:0/3712888755 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff9e00cd9e0 0x7ff9e0151530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:00.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.037+0000 7ff9e4ae9640 1 -- 172.21.15.57:0/3712888755 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff9e00cd1c0 msgr2=0x7ff9e014a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:00.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.037+0000 7ff9e4ae9640 1 --2- 172.21.15.57:0/3712888755 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff9e00cd1c0 0x7ff9e014a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:00.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.037+0000 7ff9e4ae9640 1 -- 172.21.15.57:0/3712888755 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9d0009560 con 0x7ff9e0158910 2023-10-19T12:58:00.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.037+0000 7ff9e4ae9640 1 --2- 172.21.15.57:0/3712888755 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9e0158910 0x7ff9e0151a70 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7ff9d800c710 tx=0x7ff9d800cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:58:00.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.037+0000 7ff9dcff9640 1 -- 172.21.15.57:0/3712888755 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d8006d50 con 0x7ff9e0158910 2023-10-19T12:58:00.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.037+0000 7ff9dcff9640 1 -- 172.21.15.57:0/3712888755 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7ff9d800ce20 con 0x7ff9e0158910 2023-10-19T12:58:00.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.037+0000 7ff9e654c640 1 -- 172.21.15.57:0/3712888755 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9e0157240 con 0x7ff9e0158910 2023-10-19T12:58:00.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.037+0000 7ff9dcff9640 1 -- 172.21.15.57:0/3712888755 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d8003bd0 con 0x7ff9e0158910 2023-10-19T12:58:00.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.037+0000 7ff9e654c640 1 -- 172.21.15.57:0/3712888755 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9e01561a0 con 0x7ff9e0158910 2023-10-19T12:58:00.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.038+0000 7ff9e654c640 1 -- 172.21.15.57:0/3712888755 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9e00aa8c0 con 0x7ff9e0158910 2023-10-19T12:58:00.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.039+0000 7ff9dcff9640 1 -- 172.21.15.57:0/3712888755 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 20) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff9d8003d30 con 0x7ff9e0158910 2023-10-19T12:58:00.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.039+0000 7ff9dcff9640 1 --2- 172.21.15.57:0/3712888755 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7ff9cc06af60 0x7ff9cc06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:00.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.039+0000 7ff9dcff9640 1 -- 172.21.15.57:0/3712888755 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6029+0+0 (secure 0 0 0) 0x7ff9d8089690 con 0x7ff9e0158910 2023-10-19T12:58:00.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.039+0000 7ff9dffff640 1 --2- 172.21.15.57:0/3712888755 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7ff9cc06af60 0x7ff9cc06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:00.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.040+0000 7ff9dffff640 1 --2- 172.21.15.57:0/3712888755 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7ff9cc06af60 0x7ff9cc06d420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff9e00af4e0 tx=0x7ff9d4005f90 comp rx=0 tx=0).ready entity=mgr.5809 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:58:00.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.043+0000 7ff9dcff9640 1 -- 172.21.15.57:0/3712888755 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff9e00aa8c0 con 0x7ff9e0158910 2023-10-19T12:58:00.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.257+0000 7ff9e654c640 1 -- 172.21.15.57:0/3712888755 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7ff9e00aaca0 con 0x7ff9e0158910 2023-10-19T12:58:00.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.263+0000 7ff9dcff9640 1 -- 172.21.15.57:0/3712888755 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v22) v1 ==== 118+0+0 (secure 0 0 0) 0x7ff9e00aaca0 con 0x7ff9e0158910 2023-10-19T12:58:00.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.265+0000 7ff9e654c640 1 -- 172.21.15.57:0/3712888755 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7ff9cc06af60 msgr2=0x7ff9cc06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:00.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.265+0000 7ff9e654c640 1 --2- 172.21.15.57:0/3712888755 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7ff9cc06af60 0x7ff9cc06d420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff9e00af4e0 tx=0x7ff9d4005f90 comp rx=0 tx=0).stop 2023-10-19T12:58:00.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.265+0000 7ff9e654c640 1 -- 172.21.15.57:0/3712888755 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9e0158910 msgr2=0x7ff9e0151a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:00.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.265+0000 7ff9e654c640 1 --2- 172.21.15.57:0/3712888755 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9e0158910 0x7ff9e0151a70 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7ff9d800c710 tx=0x7ff9d800cbd0 comp rx=0 tx=0).stop 2023-10-19T12:58:00.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.265+0000 7ff9e654c640 1 -- 172.21.15.57:0/3712888755 shutdown_connections 2023-10-19T12:58:00.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.265+0000 7ff9e654c640 1 --2- 172.21.15.57:0/3712888755 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7ff9cc06af60 0x7ff9cc06d420 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:00.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.265+0000 7ff9e654c640 1 --2- 172.21.15.57:0/3712888755 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9e0158910 0x7ff9e0151a70 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:00.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.266+0000 7ff9e654c640 1 --2- 172.21.15.57:0/3712888755 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff9e00cd9e0 0x7ff9e0151530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:00.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.266+0000 7ff9e654c640 1 --2- 172.21.15.57:0/3712888755 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff9e00cd1c0 0x7ff9e014a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:00.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.266+0000 7ff9e654c640 1 -- 172.21.15.57:0/3712888755 >> 172.21.15.57:0/3712888755 conn(0x7ff9e00af7a0 msgr2=0x7ff9e006db10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:58:00.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.266+0000 7ff9e654c640 1 -- 172.21.15.57:0/3712888755 shutdown_connections 2023-10-19T12:58:00.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.266+0000 7ff9e654c640 1 -- 172.21.15.57:0/3712888755 wait complete. 2023-10-19T12:58:00.279 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-19T12:58:00.279 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-19T12:58:00.391 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.389+0000 7f7cd1807640 1 -- 172.21.15.57:0/3159135048 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ccc0cd9e0 msgr2=0x7f7ccc0cde50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:00.392 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.389+0000 7f7cd1807640 1 --2- 172.21.15.57:0/3159135048 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ccc0cd9e0 0x7f7ccc0cde50 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc073460 tx=0x7f7cbc02c2f0 comp rx=0 tx=0).stop 2023-10-19T12:58:00.392 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.389+0000 7f7cd1807640 1 -- 172.21.15.57:0/3159135048 shutdown_connections 2023-10-19T12:58:00.392 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.389+0000 7f7cd1807640 1 --2- 172.21.15.57:0/3159135048 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ccc0cd9e0 0x7f7ccc0cde50 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:00.392 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.389+0000 7f7cd1807640 1 --2- 172.21.15.57:0/3159135048 >> v2:172.21.15.179:3300/0 conn(0x7f7ccc158910 0x7f7ccc158cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:00.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.389+0000 7f7cd1807640 1 -- 172.21.15.57:0/3159135048 >> 172.21.15.57:0/3159135048 conn(0x7f7ccc0af7a0 msgr2=0x7f7ccc0afb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:58:00.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.389+0000 7f7cd1807640 1 -- 172.21.15.57:0/3159135048 shutdown_connections 2023-10-19T12:58:00.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.389+0000 7f7cd1807640 1 -- 172.21.15.57:0/3159135048 wait complete. 2023-10-19T12:58:00.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.390+0000 7f7cd1807640 1 Processor -- start 2023-10-19T12:58:00.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.391+0000 7f7cd1807640 1 -- start start 2023-10-19T12:58:00.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.391+0000 7f7cd1807640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7ccc0cd1c0 0x7f7ccc0b0890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:00.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.391+0000 7f7cd1807640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7ccc0cd9e0 0x7f7ccc0af060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:00.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.391+0000 7f7cd1807640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ccc158910 0x7f7ccc14a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:00.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.391+0000 7f7cd1807640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ccc0af5a0 con 0x7f7ccc158910 2023-10-19T12:58:00.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.391+0000 7f7cd1807640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ccc0b0f20 con 0x7f7ccc0cd9e0 2023-10-19T12:58:00.397 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.391+0000 7f7cd1807640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ccc0b1200 con 0x7f7ccc0cd1c0 2023-10-19T12:58:00.397 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.391+0000 7f7ccb7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ccc158910 0x7f7ccc14a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:00.397 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.392+0000 7f7ccb7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ccc158910 0x7f7ccc14a420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:41660/0 (socket says 172.21.15.57:41660) 2023-10-19T12:58:00.397 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.392+0000 7f7ccb7fe640 1 -- 172.21.15.57:0/3247906894 learned_addr learned my addr 172.21.15.57:0/3247906894 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:58:00.398 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.392+0000 7f7cca7fc640 1 --2- 172.21.15.57:0/3247906894 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7ccc0cd9e0 0x7f7ccc0af060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:00.398 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.392+0000 7f7ccaffd640 1 --2- 172.21.15.57:0/3247906894 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7ccc0cd1c0 0x7f7ccc0b0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:00.398 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.392+0000 7f7ccb7fe640 1 -- 172.21.15.57:0/3247906894 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7ccc0cd1c0 msgr2=0x7f7ccc0b0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:00.398 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.392+0000 7f7ccb7fe640 1 --2- 172.21.15.57:0/3247906894 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7ccc0cd1c0 0x7f7ccc0b0890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:00.399 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.392+0000 7f7ccb7fe640 1 -- 172.21.15.57:0/3247906894 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7ccc0cd9e0 msgr2=0x7f7ccc0af060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:00.399 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.392+0000 7f7ccb7fe640 1 --2- 172.21.15.57:0/3247906894 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7ccc0cd9e0 0x7f7ccc0af060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:00.399 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.392+0000 7f7ccb7fe640 1 -- 172.21.15.57:0/3247906894 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cbc009900 con 0x7f7ccc158910 2023-10-19T12:58:00.399 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.392+0000 7f7ccb7fe640 1 --2- 172.21.15.57:0/3247906894 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ccc158910 0x7f7ccc14a420 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f7cbc009d60 tx=0x7f7cbc035ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:58:00.400 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.392+0000 7f7cc3fff640 1 -- 172.21.15.57:0/3247906894 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cbc004250 con 0x7f7ccc158910 2023-10-19T12:58:00.400 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.392+0000 7f7cc3fff640 1 -- 172.21.15.57:0/3247906894 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f7cbc0044a0 con 0x7f7ccc158910 2023-10-19T12:58:00.400 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.392+0000 7f7cd1807640 1 -- 172.21.15.57:0/3247906894 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ccc15b990 con 0x7f7ccc158910 2023-10-19T12:58:00.400 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.392+0000 7f7cc3fff640 1 -- 172.21.15.57:0/3247906894 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cbc040450 con 0x7f7ccc158910 2023-10-19T12:58:00.401 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.392+0000 7f7cd1807640 1 -- 172.21.15.57:0/3247906894 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ccc0b17c0 con 0x7f7ccc158910 2023-10-19T12:58:00.401 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.393+0000 7f7cd1807640 1 -- 172.21.15.57:0/3247906894 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ccc0690e0 con 0x7f7ccc158910 2023-10-19T12:58:00.401 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.394+0000 7f7cc3fff640 1 -- 172.21.15.57:0/3247906894 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 20) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f7cbc00a040 con 0x7f7ccc158910 2023-10-19T12:58:00.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.394+0000 7f7cc3fff640 1 --2- 172.21.15.57:0/3247906894 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f7cac06af60 0x7f7cac06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:00.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.394+0000 7f7cc3fff640 1 -- 172.21.15.57:0/3247906894 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f7cbc0acb50 con 0x7f7ccc158910 2023-10-19T12:58:00.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.394+0000 7f7ccaffd640 1 --2- 172.21.15.57:0/3247906894 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f7cac06af60 0x7f7cac06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:00.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.395+0000 7f7ccaffd640 1 --2- 172.21.15.57:0/3247906894 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f7cac06af60 0x7f7cac06d420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7cc4005fd0 tx=0x7f7cc4005850 comp rx=0 tx=0).ready entity=mgr.5809 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:58:00.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.398+0000 7f7cc3fff640 1 -- 172.21.15.57:0/3247906894 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7cbc077a40 con 0x7f7ccc158910 2023-10-19T12:58:00.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:00.613+0000 7f7cd1807640 1 -- 172.21.15.57:0/3247906894 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f7ccc0ce5e0 con 0x7f7ccc158910 2023-10-19T12:58:01.278 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.277+0000 7f7cc3fff640 1 -- 172.21.15.57:0/3247906894 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v23) v1 ==== 122+0+0 (secure 0 0 0) 0x7f7cbc039020 con 0x7f7ccc158910 2023-10-19T12:58:01.282 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.279+0000 7f7cd1807640 1 -- 172.21.15.57:0/3247906894 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f7cac06af60 msgr2=0x7f7cac06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:01.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.279+0000 7f7cd1807640 1 --2- 172.21.15.57:0/3247906894 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f7cac06af60 0x7f7cac06d420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7cc4005fd0 tx=0x7f7cc4005850 comp rx=0 tx=0).stop 2023-10-19T12:58:01.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.279+0000 7f7cd1807640 1 -- 172.21.15.57:0/3247906894 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ccc158910 msgr2=0x7f7ccc14a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:01.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.279+0000 7f7cd1807640 1 --2- 172.21.15.57:0/3247906894 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ccc158910 0x7f7ccc14a420 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f7cbc009d60 tx=0x7f7cbc035ed0 comp rx=0 tx=0).stop 2023-10-19T12:58:01.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.280+0000 7f7cd1807640 1 -- 172.21.15.57:0/3247906894 shutdown_connections 2023-10-19T12:58:01.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.280+0000 7f7cd1807640 1 --2- 172.21.15.57:0/3247906894 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f7cac06af60 0x7f7cac06d420 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:01.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.280+0000 7f7cd1807640 1 --2- 172.21.15.57:0/3247906894 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ccc158910 0x7f7ccc14a420 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:01.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.280+0000 7f7cd1807640 1 --2- 172.21.15.57:0/3247906894 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7ccc0cd9e0 0x7f7ccc0af060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:01.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.280+0000 7f7cd1807640 1 --2- 172.21.15.57:0/3247906894 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7ccc0cd1c0 0x7f7ccc0b0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:01.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.280+0000 7f7cd1807640 1 -- 172.21.15.57:0/3247906894 >> 172.21.15.57:0/3247906894 conn(0x7f7ccc0af7a0 msgr2=0x7f7ccc06db10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:58:01.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.280+0000 7f7cd1807640 1 -- 172.21.15.57:0/3247906894 shutdown_connections 2023-10-19T12:58:01.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.280+0000 7f7cd1807640 1 -- 172.21.15.57:0/3247906894 wait complete. 2023-10-19T12:58:01.294 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-19T12:58:01.295 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T12:58:01.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.409+0000 7f7eebe56640 1 -- 172.21.15.57:0/4092932136 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ee40c9d00 msgr2=0x7f7ee406b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:01.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.409+0000 7f7eebe56640 1 --2- 172.21.15.57:0/4092932136 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ee40c9d00 0x7f7ee406b910 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f7ed4005fd0 tx=0x7f7ed402c0e0 comp rx=0 tx=0).stop 2023-10-19T12:58:01.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.409+0000 7f7eebe56640 1 -- 172.21.15.57:0/4092932136 shutdown_connections 2023-10-19T12:58:01.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.409+0000 7f7eebe56640 1 --2- 172.21.15.57:0/4092932136 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7ee4140660 0x7f7ee40c97c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:01.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.409+0000 7f7eebe56640 1 --2- 172.21.15.57:0/4092932136 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ee40c9d00 0x7f7ee406b910 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:01.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.409+0000 7f7eebe56640 1 -- 172.21.15.57:0/4092932136 >> 172.21.15.57:0/4092932136 conn(0x7f7ee4069930 msgr2=0x7f7ee4069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:58:01.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.409+0000 7f7eebe56640 1 -- 172.21.15.57:0/4092932136 shutdown_connections 2023-10-19T12:58:01.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.409+0000 7f7eebe56640 1 -- 172.21.15.57:0/4092932136 wait complete. 2023-10-19T12:58:01.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.410+0000 7f7eebe56640 1 Processor -- start 2023-10-19T12:58:01.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.410+0000 7f7eebe56640 1 -- start start 2023-10-19T12:58:01.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.411+0000 7f7eebe56640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7ee40c9d00 0x7f7ee4156240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:01.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.411+0000 7f7eebe56640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7ee413d360 0x7f7ee413a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:01.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.411+0000 7f7eebe56640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ee4140660 0x7f7ee4134450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:01.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.411+0000 7f7eebe56640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee413b2c0 con 0x7f7ee4140660 2023-10-19T12:58:01.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.411+0000 7f7eebe56640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee412c900 con 0x7f7ee413d360 2023-10-19T12:58:01.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.411+0000 7f7eebe56640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee412cbe0 con 0x7f7ee40c9d00 2023-10-19T12:58:01.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.411+0000 7f7eea3f3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ee4140660 0x7f7ee4134450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:01.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.411+0000 7f7eea3f3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ee4140660 0x7f7ee4134450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:41682/0 (socket says 172.21.15.57:41682) 2023-10-19T12:58:01.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.411+0000 7f7eea3f3640 1 -- 172.21.15.57:0/2170259272 learned_addr learned my addr 172.21.15.57:0/2170259272 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:58:01.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.411+0000 7f7ee93f1640 1 --2- 172.21.15.57:0/2170259272 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7ee413d360 0x7f7ee413a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:01.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.411+0000 7f7ee9bf2640 1 --2- 172.21.15.57:0/2170259272 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7ee40c9d00 0x7f7ee4156240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:01.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.412+0000 7f7eea3f3640 1 -- 172.21.15.57:0/2170259272 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7ee40c9d00 msgr2=0x7f7ee4156240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:01.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.412+0000 7f7eea3f3640 1 --2- 172.21.15.57:0/2170259272 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7ee40c9d00 0x7f7ee4156240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:01.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.412+0000 7f7eea3f3640 1 -- 172.21.15.57:0/2170259272 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7ee413d360 msgr2=0x7f7ee413a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:01.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.412+0000 7f7eea3f3640 1 --2- 172.21.15.57:0/2170259272 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7ee413d360 0x7f7ee413a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:01.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.412+0000 7f7eea3f3640 1 -- 172.21.15.57:0/2170259272 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ed4009840 con 0x7f7ee4140660 2023-10-19T12:58:01.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.412+0000 7f7ee93f1640 1 --2- 172.21.15.57:0/2170259272 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7ee413d360 0x7f7ee413a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:58:01.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.412+0000 7f7eea3f3640 1 --2- 172.21.15.57:0/2170259272 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ee4140660 0x7f7ee4134450 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f7ed4036040 tx=0x7f7ed4004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:58:01.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.412+0000 7f7ee27fc640 1 -- 172.21.15.57:0/2170259272 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ed4004670 con 0x7f7ee4140660 2023-10-19T12:58:01.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.412+0000 7f7ee27fc640 1 -- 172.21.15.57:0/2170259272 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f7ed40345a0 con 0x7f7ee4140660 2023-10-19T12:58:01.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.412+0000 7f7ee27fc640 1 -- 172.21.15.57:0/2170259272 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ed4034970 con 0x7f7ee4140660 2023-10-19T12:58:01.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.412+0000 7f7eebe56640 1 -- 172.21.15.57:0/2170259272 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ee412cec0 con 0x7f7ee4140660 2023-10-19T12:58:01.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.412+0000 7f7eebe56640 1 -- 172.21.15.57:0/2170259272 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ee412d6c0 con 0x7f7ee4140660 2023-10-19T12:58:01.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.413+0000 7f7eebe56640 1 -- 172.21.15.57:0/2170259272 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ee412d3e0 con 0x7f7ee4140660 2023-10-19T12:58:01.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.413+0000 7f7ee27fc640 1 -- 172.21.15.57:0/2170259272 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 20) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f7ed4034ad0 con 0x7f7ee4140660 2023-10-19T12:58:01.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.414+0000 7f7ee27fc640 1 --2- 172.21.15.57:0/2170259272 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f7ed006af60 0x7f7ed006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:01.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.414+0000 7f7ee27fc640 1 -- 172.21.15.57:0/2170259272 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f7ed40acac0 con 0x7f7ee4140660 2023-10-19T12:58:01.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.414+0000 7f7ee9bf2640 1 --2- 172.21.15.57:0/2170259272 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f7ed006af60 0x7f7ed006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:01.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.415+0000 7f7ee9bf2640 1 --2- 172.21.15.57:0/2170259272 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f7ed006af60 0x7f7ed006d420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7edc000f60 tx=0x7f7edc009040 comp rx=0 tx=0).ready entity=mgr.5809 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:58:01.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.418+0000 7f7ee27fc640 1 -- 172.21.15.57:0/2170259272 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7ee412d3e0 con 0x7f7ee4140660 2023-10-19T12:58:01.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.635+0000 7f7eebe56640 1 -- 172.21.15.57:0/2170259272 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7ee412d3e0 con 0x7f7ee4140660 2023-10-19T12:58:01.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.635+0000 7f7ee27fc640 1 -- 172.21.15.57:0/2170259272 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+4913 (secure 0 0 0) 0x7f7ee412d3e0 con 0x7f7ee4140660 2023-10-19T12:58:01.638 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:58:01.638 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":23,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5594,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/3950728949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":3950728949},{"type":"v1","addr":"172.21.15.179:6837","nonce":3950728949}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":5597,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/3158611991","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":3158611991},{"type":"v1","addr":"172.21.15.179:6836","nonce":3158611991}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":6507,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/352867817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":352867817},{"type":"v1","addr":"172.21.15.57:6834","nonce":352867817}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T12:57:58.606618+0000","modified":"2023-10-19T12:58:00.613725+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6510},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6510":{"gid":6510,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.57:6835/243029526","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":243029526},{"type":"v1","addr":"172.21.15.57:6835","nonce":243029526}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-10-19T12:58:01.640 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 23 2023-10-19T12:58:01.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.637+0000 7f7eebe56640 1 -- 172.21.15.57:0/2170259272 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f7ed006af60 msgr2=0x7f7ed006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:01.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.637+0000 7f7eebe56640 1 --2- 172.21.15.57:0/2170259272 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f7ed006af60 0x7f7ed006d420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7edc000f60 tx=0x7f7edc009040 comp rx=0 tx=0).stop 2023-10-19T12:58:01.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.637+0000 7f7eebe56640 1 -- 172.21.15.57:0/2170259272 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ee4140660 msgr2=0x7f7ee4134450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:01.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.637+0000 7f7eebe56640 1 --2- 172.21.15.57:0/2170259272 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ee4140660 0x7f7ee4134450 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f7ed4036040 tx=0x7f7ed4004270 comp rx=0 tx=0).stop 2023-10-19T12:58:01.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.637+0000 7f7eebe56640 1 -- 172.21.15.57:0/2170259272 shutdown_connections 2023-10-19T12:58:01.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.637+0000 7f7eebe56640 1 --2- 172.21.15.57:0/2170259272 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f7ed006af60 0x7f7ed006d420 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:01.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.637+0000 7f7eebe56640 1 --2- 172.21.15.57:0/2170259272 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ee4140660 0x7f7ee4134450 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:01.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.637+0000 7f7eebe56640 1 --2- 172.21.15.57:0/2170259272 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7ee413d360 0x7f7ee413a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:01.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.637+0000 7f7eebe56640 1 --2- 172.21.15.57:0/2170259272 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7ee40c9d00 0x7f7ee4156240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:01.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.637+0000 7f7eebe56640 1 -- 172.21.15.57:0/2170259272 >> 172.21.15.57:0/2170259272 conn(0x7f7ee4069930 msgr2=0x7f7ee4159680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:58:01.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.637+0000 7f7eebe56640 1 -- 172.21.15.57:0/2170259272 shutdown_connections 2023-10-19T12:58:01.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.637+0000 7f7eebe56640 1 -- 172.21.15.57:0/2170259272 wait complete. 2023-10-19T12:58:01.649 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T12:58:01.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.757+0000 7fe50391d640 1 -- 172.21.15.57:0/2952918150 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe4fc0cf340 msgr2=0x7fe4fc0d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:01.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.757+0000 7fe50391d640 1 --2- 172.21.15.57:0/2952918150 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe4fc0cf340 0x7fe4fc0d1910 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fe4fc15b680 tx=0x7fe4f402bd30 comp rx=0 tx=0).stop 2023-10-19T12:58:01.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.757+0000 7fe50391d640 1 -- 172.21.15.57:0/2952918150 shutdown_connections 2023-10-19T12:58:01.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.757+0000 7fe50391d640 1 --2- 172.21.15.57:0/2952918150 >> v2:172.21.15.179:3300/0 conn(0x7fe4fc0ceb40 0x7fe4fc0b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:01.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.757+0000 7fe50391d640 1 --2- 172.21.15.57:0/2952918150 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe4fc0cf340 0x7fe4fc0d1910 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:01.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.757+0000 7fe50391d640 1 -- 172.21.15.57:0/2952918150 >> 172.21.15.57:0/2952918150 conn(0x7fe4fc0d0d60 msgr2=0x7fe4fc0d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:58:01.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.757+0000 7fe50391d640 1 -- 172.21.15.57:0/2952918150 shutdown_connections 2023-10-19T12:58:01.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.757+0000 7fe50391d640 1 -- 172.21.15.57:0/2952918150 wait complete. 2023-10-19T12:58:01.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.758+0000 7fe50391d640 1 Processor -- start 2023-10-19T12:58:01.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.758+0000 7fe50391d640 1 -- start start 2023-10-19T12:58:01.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.759+0000 7fe50391d640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe4fc0ceb40 0x7fe4fc157110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:01.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.759+0000 7fe50391d640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe4fc0cf340 0x7fe4fc159100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:01.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.759+0000 7fe50391d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe4fc15b2a0 0x7fe4fc159640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:01.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.759+0000 7fe50391d640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4fc0cd560 con 0x7fe4fc15b2a0 2023-10-19T12:58:01.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.759+0000 7fe50391d640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4fc0cc8e0 con 0x7fe4fc0ceb40 2023-10-19T12:58:01.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.759+0000 7fe50391d640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4fc0ccbc0 con 0x7fe4fc0cf340 2023-10-19T12:58:01.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.759+0000 7fe501eba640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe4fc15b2a0 0x7fe4fc159640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:01.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.759+0000 7fe501eba640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe4fc15b2a0 0x7fe4fc159640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:41704/0 (socket says 172.21.15.57:41704) 2023-10-19T12:58:01.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.759+0000 7fe501eba640 1 -- 172.21.15.57:0/1840552833 learned_addr learned my addr 172.21.15.57:0/1840552833 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:58:01.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.760+0000 7fe500eb8640 1 --2- 172.21.15.57:0/1840552833 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe4fc0cf340 0x7fe4fc159100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:01.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.760+0000 7fe5016b9640 1 --2- 172.21.15.57:0/1840552833 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe4fc0ceb40 0x7fe4fc157110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:01.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.760+0000 7fe501eba640 1 -- 172.21.15.57:0/1840552833 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe4fc0cf340 msgr2=0x7fe4fc159100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:01.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.760+0000 7fe501eba640 1 --2- 172.21.15.57:0/1840552833 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe4fc0cf340 0x7fe4fc159100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:01.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.760+0000 7fe501eba640 1 -- 172.21.15.57:0/1840552833 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe4fc0ceb40 msgr2=0x7fe4fc157110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:01.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.760+0000 7fe501eba640 1 --2- 172.21.15.57:0/1840552833 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe4fc0ceb40 0x7fe4fc157110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:01.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.760+0000 7fe501eba640 1 -- 172.21.15.57:0/1840552833 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4f4009560 con 0x7fe4fc15b2a0 2023-10-19T12:58:01.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.760+0000 7fe500eb8640 1 --2- 172.21.15.57:0/1840552833 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe4fc0cf340 0x7fe4fc159100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:58:01.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.760+0000 7fe501eba640 1 --2- 172.21.15.57:0/1840552833 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe4fc15b2a0 0x7fe4fc159640 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fe4ec00cad0 tx=0x7fe4ec00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:58:01.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.760+0000 7fe4f9ffb640 1 -- 172.21.15.57:0/1840552833 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4ec00ae20 con 0x7fe4fc15b2a0 2023-10-19T12:58:01.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.760+0000 7fe4f9ffb640 1 -- 172.21.15.57:0/1840552833 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fe4ec004b20 con 0x7fe4fc15b2a0 2023-10-19T12:58:01.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.760+0000 7fe50391d640 1 -- 172.21.15.57:0/1840552833 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4fc0ccea0 con 0x7fe4fc15b2a0 2023-10-19T12:58:01.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.760+0000 7fe4f9ffb640 1 -- 172.21.15.57:0/1840552833 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4ec005020 con 0x7fe4fc15b2a0 2023-10-19T12:58:01.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.760+0000 7fe50391d640 1 -- 172.21.15.57:0/1840552833 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4fc157650 con 0x7fe4fc15b2a0 2023-10-19T12:58:01.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.761+0000 7fe50391d640 1 -- 172.21.15.57:0/1840552833 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4fc0cd6a0 con 0x7fe4fc15b2a0 2023-10-19T12:58:01.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.762+0000 7fe4f9ffb640 1 -- 172.21.15.57:0/1840552833 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 20) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe4ec005180 con 0x7fe4fc15b2a0 2023-10-19T12:58:01.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.762+0000 7fe4f9ffb640 1 --2- 172.21.15.57:0/1840552833 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7fe4e806af60 0x7fe4e806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:01.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.762+0000 7fe4f9ffb640 1 -- 172.21.15.57:0/1840552833 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fe4ec08a250 con 0x7fe4fc15b2a0 2023-10-19T12:58:01.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.762+0000 7fe5016b9640 1 --2- 172.21.15.57:0/1840552833 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7fe4e806af60 0x7fe4e806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:01.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.763+0000 7fe5016b9640 1 --2- 172.21.15.57:0/1840552833 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7fe4e806af60 0x7fe4e806d420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe4f4035000 tx=0x7fe4f4004730 comp rx=0 tx=0).ready entity=mgr.5809 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:58:01.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.766+0000 7fe4f9ffb640 1 -- 172.21.15.57:0/1840552833 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe4fc0cd6a0 con 0x7fe4fc15b2a0 2023-10-19T12:58:01.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.944+0000 7fe50391d640 1 -- 172.21.15.57:0/1840552833 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe4fc0cd6a0 con 0x7fe4fc15b2a0 2023-10-19T12:58:01.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.945+0000 7fe4f9ffb640 1 -- 172.21.15.57:0/1840552833 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v70) v1 ==== 74+0+16786 (secure 0 0 0) 0x7fe4fc0cd6a0 con 0x7fe4fc15b2a0 2023-10-19T12:58:01.947 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:58:01.947 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":70,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T12:58:00.262979+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2023-10-19T12:57:56.901606+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2023-10-19T12:57:57.581428+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T12:58:01.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.947+0000 7fe50391d640 1 -- 172.21.15.57:0/1840552833 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7fe4e806af60 msgr2=0x7fe4e806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:01.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.947+0000 7fe50391d640 1 --2- 172.21.15.57:0/1840552833 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7fe4e806af60 0x7fe4e806d420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe4f4035000 tx=0x7fe4f4004730 comp rx=0 tx=0).stop 2023-10-19T12:58:01.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.947+0000 7fe50391d640 1 -- 172.21.15.57:0/1840552833 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe4fc15b2a0 msgr2=0x7fe4fc159640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:01.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.947+0000 7fe50391d640 1 --2- 172.21.15.57:0/1840552833 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe4fc15b2a0 0x7fe4fc159640 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fe4ec00cad0 tx=0x7fe4ec00cf90 comp rx=0 tx=0).stop 2023-10-19T12:58:01.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.947+0000 7fe50391d640 1 -- 172.21.15.57:0/1840552833 shutdown_connections 2023-10-19T12:58:01.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.947+0000 7fe50391d640 1 --2- 172.21.15.57:0/1840552833 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7fe4e806af60 0x7fe4e806d420 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:01.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.947+0000 7fe50391d640 1 --2- 172.21.15.57:0/1840552833 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe4fc15b2a0 0x7fe4fc159640 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:01.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.947+0000 7fe50391d640 1 --2- 172.21.15.57:0/1840552833 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe4fc0cf340 0x7fe4fc159100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:01.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.947+0000 7fe50391d640 1 --2- 172.21.15.57:0/1840552833 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe4fc0ceb40 0x7fe4fc157110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:01.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.947+0000 7fe50391d640 1 -- 172.21.15.57:0/1840552833 >> 172.21.15.57:0/1840552833 conn(0x7fe4fc0d0d60 msgr2=0x7fe4fc15a5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:58:01.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.947+0000 7fe50391d640 1 -- 172.21.15.57:0/1840552833 shutdown_connections 2023-10-19T12:58:01.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:01.947+0000 7fe50391d640 1 -- 172.21.15.57:0/1840552833 wait complete. 2023-10-19T12:58:01.960 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-19T12:58:01.960 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-19T12:58:02.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.067+0000 7f19a6e75640 1 -- 172.21.15.57:0/1530892514 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f19a0141690 msgr2=0x7f19a00703e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:02.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.067+0000 7f19a6e75640 1 --2- 172.21.15.57:0/1530892514 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f19a0141690 0x7f19a00703e0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f19a00cbd00 tx=0x7f199002c080 comp rx=0 tx=0).stop 2023-10-19T12:58:02.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.067+0000 7f19a6e75640 1 -- 172.21.15.57:0/1530892514 shutdown_connections 2023-10-19T12:58:02.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.067+0000 7f19a6e75640 1 --2- 172.21.15.57:0/1530892514 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f19a0141690 0x7f19a00703e0 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:02.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.067+0000 7f19a6e75640 1 --2- 172.21.15.57:0/1530892514 >> v2:172.21.15.179:3300/0 conn(0x7f19a013e520 0x7f19a006bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:02.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.067+0000 7f19a6e75640 1 -- 172.21.15.57:0/1530892514 >> 172.21.15.57:0/1530892514 conn(0x7f19a0069930 msgr2=0x7f19a0069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:58:02.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.068+0000 7f19a6e75640 1 -- 172.21.15.57:0/1530892514 shutdown_connections 2023-10-19T12:58:02.072 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.068+0000 7f19a6e75640 1 -- 172.21.15.57:0/1530892514 wait complete. 2023-10-19T12:58:02.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.069+0000 7f19a6e75640 1 Processor -- start 2023-10-19T12:58:02.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.069+0000 7f19a6e75640 1 -- start start 2023-10-19T12:58:02.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.070+0000 7f19a6e75640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f19a006c0a0 0x7f19a0157870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:02.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.070+0000 7f19a6e75640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f19a013e520 0x7f19a0158c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:02.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.070+0000 7f19a6e75640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f19a0141690 0x7f19a0162d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:02.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.070+0000 7f19a6e75640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19a0157db0 con 0x7f19a013e520 2023-10-19T12:58:02.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.070+0000 7f19a6e75640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19a01596a0 con 0x7f19a006c0a0 2023-10-19T12:58:02.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.070+0000 7f19a6e75640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19a013ea30 con 0x7f19a0141690 2023-10-19T12:58:02.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.070+0000 7f199ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f19a013e520 0x7f19a0158c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:02.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.070+0000 7f199ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f19a013e520 0x7f19a0158c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:41734/0 (socket says 172.21.15.57:41734) 2023-10-19T12:58:02.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.070+0000 7f199ffff640 1 -- 172.21.15.57:0/19905 learned_addr learned my addr 172.21.15.57:0/19905 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:58:02.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.070+0000 7f19a4c11640 1 --2- 172.21.15.57:0/19905 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f19a006c0a0 0x7f19a0157870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:02.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.070+0000 7f199ffff640 1 -- 172.21.15.57:0/19905 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f19a0141690 msgr2=0x7f19a0162d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:02.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.070+0000 7f19a5412640 1 --2- 172.21.15.57:0/19905 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f19a0141690 0x7f19a0162d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:02.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.070+0000 7f199ffff640 1 --2- 172.21.15.57:0/19905 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f19a0141690 0x7f19a0162d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:02.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.070+0000 7f199ffff640 1 -- 172.21.15.57:0/19905 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f19a006c0a0 msgr2=0x7f19a0157870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:02.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.070+0000 7f199ffff640 1 --2- 172.21.15.57:0/19905 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f19a006c0a0 0x7f19a0157870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:02.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.070+0000 7f199ffff640 1 -- 172.21.15.57:0/19905 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1990009840 con 0x7f19a013e520 2023-10-19T12:58:02.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.070+0000 7f19a4c11640 1 --2- 172.21.15.57:0/19905 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f19a006c0a0 0x7f19a0157870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:58:02.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.070+0000 7f19a5412640 1 --2- 172.21.15.57:0/19905 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f19a0141690 0x7f19a0162d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:58:02.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.070+0000 7f199ffff640 1 --2- 172.21.15.57:0/19905 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f19a013e520 0x7f19a0158c20 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f1990000f80 tx=0x7f1990004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:58:02.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.070+0000 7f199d7fa640 1 -- 172.21.15.57:0/19905 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1990004870 con 0x7f19a013e520 2023-10-19T12:58:02.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.070+0000 7f199d7fa640 1 -- 172.21.15.57:0/19905 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f199003d520 con 0x7f19a013e520 2023-10-19T12:58:02.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.070+0000 7f19a6e75640 1 -- 172.21.15.57:0/19905 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19a013ed10 con 0x7f19a013e520 2023-10-19T12:58:02.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.070+0000 7f199d7fa640 1 -- 172.21.15.57:0/19905 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f199003d910 con 0x7f19a013e520 2023-10-19T12:58:02.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.070+0000 7f19a6e75640 1 -- 172.21.15.57:0/19905 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19a013f2d0 con 0x7f19a013e520 2023-10-19T12:58:02.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.071+0000 7f19a6e75640 1 -- 172.21.15.57:0/19905 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19a00690e0 con 0x7f19a013e520 2023-10-19T12:58:02.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.072+0000 7f199d7fa640 1 -- 172.21.15.57:0/19905 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 20) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f199003da70 con 0x7f19a013e520 2023-10-19T12:58:02.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.072+0000 7f199d7fa640 1 --2- 172.21.15.57:0/19905 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f198006af60 0x7f198006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:02.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.072+0000 7f199d7fa640 1 -- 172.21.15.57:0/19905 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f19900ac780 con 0x7f19a013e520 2023-10-19T12:58:02.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.073+0000 7f19a4c11640 1 --2- 172.21.15.57:0/19905 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f198006af60 0x7f198006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:02.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.073+0000 7f19a4c11640 1 --2- 172.21.15.57:0/19905 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f198006af60 0x7f198006d420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1998009b40 tx=0x7f1998005e50 comp rx=0 tx=0).ready entity=mgr.5809 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:58:02.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.077+0000 7f199d7fa640 1 -- 172.21.15.57:0/19905 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f19900775e0 con 0x7f19a013e520 2023-10-19T12:58:02.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.248+0000 7f19a6e75640 1 -- 172.21.15.57:0/19905 --> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f19a013f490 con 0x7f198006af60 2023-10-19T12:58:02.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.257+0000 7f199d7fa640 1 -- 172.21.15.57:0/19905 <== mgr.5809 v2:172.21.15.179:6832/3449041082 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+280765 (secure 0 0 0) 0x7f19a013f490 con 0x7f198006af60 2023-10-19T12:58:02.261 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:58:02.263 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2023-10-19T12:58:02.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.259+0000 7f19a6e75640 1 -- 172.21.15.57:0/19905 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f198006af60 msgr2=0x7f198006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:02.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.259+0000 7f19a6e75640 1 --2- 172.21.15.57:0/19905 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f198006af60 0x7f198006d420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1998009b40 tx=0x7f1998005e50 comp rx=0 tx=0).stop 2023-10-19T12:58:02.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.259+0000 7f19a6e75640 1 -- 172.21.15.57:0/19905 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f19a013e520 msgr2=0x7f19a0158c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:02.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.259+0000 7f19a6e75640 1 --2- 172.21.15.57:0/19905 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f19a013e520 0x7f19a0158c20 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f1990000f80 tx=0x7f1990004270 comp rx=0 tx=0).stop 2023-10-19T12:58:02.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.260+0000 7f19a6e75640 1 -- 172.21.15.57:0/19905 shutdown_connections 2023-10-19T12:58:02.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.260+0000 7f19a6e75640 1 --2- 172.21.15.57:0/19905 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f198006af60 0x7f198006d420 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:02.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.260+0000 7f19a6e75640 1 --2- 172.21.15.57:0/19905 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f19a0141690 0x7f19a0162d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:02.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.260+0000 7f19a6e75640 1 --2- 172.21.15.57:0/19905 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f19a013e520 0x7f19a0158c20 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:02.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.260+0000 7f19a6e75640 1 --2- 172.21.15.57:0/19905 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f19a006c0a0 0x7f19a0157870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:02.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.260+0000 7f19a6e75640 1 -- 172.21.15.57:0/19905 >> 172.21.15.57:0/19905 conn(0x7f19a0069930 msgr2=0x7f19a0069d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:58:02.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.260+0000 7f19a6e75640 1 -- 172.21.15.57:0/19905 shutdown_connections 2023-10-19T12:58:02.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.260+0000 7f19a6e75640 1 -- 172.21.15.57:0/19905 wait complete. 2023-10-19T12:58:02.272 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2023-10-19T12:58:00.627481+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590424,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":210,"num_write_kb":2424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":80,"acting":80,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":71,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":4,"kb":754974720,"kb_used":309440,"kb_used_data":7240,"kb_used_omap":12,"kb_used_meta":302067,"kb_avail":754665280,"statfs":{"total":773094113280,"available":772777246720,"internally_reserved":0,"allocated":7413760,"data_stored":4251052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":309317177},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":56,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000943"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.276851+0000","last_change":"2023-10-19T12:57:59.284384+0000","last_active":"2023-10-19T12:58:00.276851+0000","last_peered":"2023-10-19T12:58:00.276851+0000","last_clean":"2023-10-19T12:58:00.276851+0000","last_became_active":"2023-10-19T12:57:59.283993+0000","last_became_peered":"2023-10-19T12:57:59.283993+0000","last_unstale":"2023-10-19T12:58:00.276851+0000","last_undegraded":"2023-10-19T12:58:00.276851+0000","last_fullsized":"2023-10-19T12:58:00.276851+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:37:05.744377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:58.229465+0000","last_change":"2023-10-19T12:57:58.229465+0000","last_active":"2023-10-19T12:57:58.229465+0000","last_peered":"2023-10-19T12:57:58.229465+0000","last_clean":"2023-10-19T12:57:58.229465+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:58.229465+0000","last_undegraded":"2023-10-19T12:57:58.229465+0000","last_fullsized":"2023-10-19T12:57:58.229465+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.286509+0000","last_change":"2023-10-19T12:57:59.278020+0000","last_active":"2023-10-19T12:57:59.286509+0000","last_peered":"2023-10-19T12:57:59.286509+0000","last_clean":"2023-10-19T12:57:59.286509+0000","last_became_active":"2023-10-19T12:57:59.277709+0000","last_became_peered":"2023-10-19T12:57:59.277709+0000","last_unstale":"2023-10-19T12:57:59.286509+0000","last_undegraded":"2023-10-19T12:57:59.286509+0000","last_fullsized":"2023-10-19T12:57:59.286509+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:38:38.391082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:58.229465+0000","last_change":"2023-10-19T12:57:58.229465+0000","last_active":"2023-10-19T12:57:58.229465+0000","last_peered":"2023-10-19T12:57:58.229465+0000","last_clean":"2023-10-19T12:57:58.229465+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:58.229465+0000","last_undegraded":"2023-10-19T12:57:58.229465+0000","last_fullsized":"2023-10-19T12:57:58.229465+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:58.229465+0000","last_change":"2023-10-19T12:57:58.229465+0000","last_active":"2023-10-19T12:57:58.229465+0000","last_peered":"2023-10-19T12:57:58.229465+0000","last_clean":"2023-10-19T12:57:58.229465+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:58.229465+0000","last_undegraded":"2023-10-19T12:57:58.229465+0000","last_fullsized":"2023-10-19T12:57:58.229465+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.276766+0000","last_change":"2023-10-19T12:57:59.277273+0000","last_active":"2023-10-19T12:58:00.276766+0000","last_peered":"2023-10-19T12:58:00.276766+0000","last_clean":"2023-10-19T12:58:00.276766+0000","last_became_active":"2023-10-19T12:57:59.276991+0000","last_became_peered":"2023-10-19T12:57:59.276991+0000","last_unstale":"2023-10-19T12:58:00.276766+0000","last_undegraded":"2023-10-19T12:58:00.276766+0000","last_fullsized":"2023-10-19T12:58:00.276766+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:13:10.195317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.286565+0000","last_change":"2023-10-19T12:57:59.280280+0000","last_active":"2023-10-19T12:57:59.286565+0000","last_peered":"2023-10-19T12:57:59.286565+0000","last_clean":"2023-10-19T12:57:59.286565+0000","last_became_active":"2023-10-19T12:57:59.279626+0000","last_became_peered":"2023-10-19T12:57:59.279626+0000","last_unstale":"2023-10-19T12:57:59.286565+0000","last_undegraded":"2023-10-19T12:57:59.286565+0000","last_fullsized":"2023-10-19T12:57:59.286565+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:46:43.361440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:58.229465+0000","last_change":"2023-10-19T12:57:58.229465+0000","last_active":"2023-10-19T12:57:58.229465+0000","last_peered":"2023-10-19T12:57:58.229465+0000","last_clean":"2023-10-19T12:57:58.229465+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:58.229465+0000","last_undegraded":"2023-10-19T12:57:58.229465+0000","last_fullsized":"2023-10-19T12:57:58.229465+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:58.229465+0000","last_change":"2023-10-19T12:57:58.229465+0000","last_active":"2023-10-19T12:57:58.229465+0000","last_peered":"2023-10-19T12:57:58.229465+0000","last_clean":"2023-10-19T12:57:58.229465+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:58.229465+0000","last_undegraded":"2023-10-19T12:57:58.229465+0000","last_fullsized":"2023-10-19T12:57:58.229465+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.276809+0000","last_change":"2023-10-19T12:57:59.281613+0000","last_active":"2023-10-19T12:58:00.276809+0000","last_peered":"2023-10-19T12:58:00.276809+0000","last_clean":"2023-10-19T12:58:00.276809+0000","last_became_active":"2023-10-19T12:57:59.281322+0000","last_became_peered":"2023-10-19T12:57:59.281322+0000","last_unstale":"2023-10-19T12:58:00.276809+0000","last_undegraded":"2023-10-19T12:58:00.276809+0000","last_fullsized":"2023-10-19T12:58:00.276809+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:33:13.743238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.278808+0000","last_change":"2023-10-19T12:57:59.275510+0000","last_active":"2023-10-19T12:57:59.278808+0000","last_peered":"2023-10-19T12:57:59.278808+0000","last_clean":"2023-10-19T12:57:59.278808+0000","last_became_active":"2023-10-19T12:57:59.275193+0000","last_became_peered":"2023-10-19T12:57:59.275193+0000","last_unstale":"2023-10-19T12:57:59.278808+0000","last_undegraded":"2023-10-19T12:57:59.278808+0000","last_fullsized":"2023-10-19T12:57:59.278808+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:20:23.928988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:58.229465+0000","last_change":"2023-10-19T12:57:58.229465+0000","last_active":"2023-10-19T12:57:58.229465+0000","last_peered":"2023-10-19T12:57:58.229465+0000","last_clean":"2023-10-19T12:57:58.229465+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:58.229465+0000","last_undegraded":"2023-10-19T12:57:58.229465+0000","last_fullsized":"2023-10-19T12:57:58.229465+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.276748+0000","last_change":"2023-10-19T12:57:59.279218+0000","last_active":"2023-10-19T12:58:00.276748+0000","last_peered":"2023-10-19T12:58:00.276748+0000","last_clean":"2023-10-19T12:58:00.276748+0000","last_became_active":"2023-10-19T12:57:59.278919+0000","last_became_peered":"2023-10-19T12:57:59.278919+0000","last_unstale":"2023-10-19T12:58:00.276748+0000","last_undegraded":"2023-10-19T12:58:00.276748+0000","last_fullsized":"2023-10-19T12:58:00.276748+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:50:41.250507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.286325+0000","last_change":"2023-10-19T12:57:59.280185+0000","last_active":"2023-10-19T12:57:59.286325+0000","last_peered":"2023-10-19T12:57:59.286325+0000","last_clean":"2023-10-19T12:57:59.286325+0000","last_became_active":"2023-10-19T12:57:59.279558+0000","last_became_peered":"2023-10-19T12:57:59.279558+0000","last_unstale":"2023-10-19T12:57:59.286325+0000","last_undegraded":"2023-10-19T12:57:59.286325+0000","last_fullsized":"2023-10-19T12:57:59.286325+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:27:42.893031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:58.229465+0000","last_change":"2023-10-19T12:57:58.229465+0000","last_active":"2023-10-19T12:57:58.229465+0000","last_peered":"2023-10-19T12:57:58.229465+0000","last_clean":"2023-10-19T12:57:58.229465+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:58.229465+0000","last_undegraded":"2023-10-19T12:57:58.229465+0000","last_fullsized":"2023-10-19T12:57:58.229465+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.276674+0000","last_change":"2023-10-19T12:57:59.277264+0000","last_active":"2023-10-19T12:58:00.276674+0000","last_peered":"2023-10-19T12:58:00.276674+0000","last_clean":"2023-10-19T12:58:00.276674+0000","last_became_active":"2023-10-19T12:57:59.276973+0000","last_became_peered":"2023-10-19T12:57:59.276973+0000","last_unstale":"2023-10-19T12:58:00.276674+0000","last_undegraded":"2023-10-19T12:58:00.276674+0000","last_fullsized":"2023-10-19T12:58:00.276674+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:19:17.421557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.287675+0000","last_change":"2023-10-19T12:57:59.278487+0000","last_active":"2023-10-19T12:57:59.287675+0000","last_peered":"2023-10-19T12:57:59.287675+0000","last_clean":"2023-10-19T12:57:59.287675+0000","last_became_active":"2023-10-19T12:57:59.276322+0000","last_became_peered":"2023-10-19T12:57:59.276322+0000","last_unstale":"2023-10-19T12:57:59.287675+0000","last_undegraded":"2023-10-19T12:57:59.287675+0000","last_fullsized":"2023-10-19T12:57:59.287675+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:32:09.277054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:58.229465+0000","last_change":"2023-10-19T12:57:58.229465+0000","last_active":"2023-10-19T12:57:58.229465+0000","last_peered":"2023-10-19T12:57:58.229465+0000","last_clean":"2023-10-19T12:57:58.229465+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:58.229465+0000","last_undegraded":"2023-10-19T12:57:58.229465+0000","last_fullsized":"2023-10-19T12:57:58.229465+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.286530+0000","last_change":"2023-10-19T12:57:59.280395+0000","last_active":"2023-10-19T12:57:59.286530+0000","last_peered":"2023-10-19T12:57:59.286530+0000","last_clean":"2023-10-19T12:57:59.286530+0000","last_became_active":"2023-10-19T12:57:59.280082+0000","last_became_peered":"2023-10-19T12:57:59.280082+0000","last_unstale":"2023-10-19T12:57:59.286530+0000","last_undegraded":"2023-10-19T12:57:59.286530+0000","last_fullsized":"2023-10-19T12:57:59.286530+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:52:31.351268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:58.229465+0000","last_change":"2023-10-19T12:57:58.229465+0000","last_active":"2023-10-19T12:57:58.229465+0000","last_peered":"2023-10-19T12:57:58.229465+0000","last_clean":"2023-10-19T12:57:58.229465+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:58.229465+0000","last_undegraded":"2023-10-19T12:57:58.229465+0000","last_fullsized":"2023-10-19T12:57:58.229465+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:58.229465+0000","last_change":"2023-10-19T12:57:58.229465+0000","last_active":"2023-10-19T12:57:58.229465+0000","last_peered":"2023-10-19T12:57:58.229465+0000","last_clean":"2023-10-19T12:57:58.229465+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:58.229465+0000","last_undegraded":"2023-10-19T12:57:58.229465+0000","last_fullsized":"2023-10-19T12:57:58.229465+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:58.229465+0000","last_change":"2023-10-19T12:57:58.229465+0000","last_active":"2023-10-19T12:57:58.229465+0000","last_peered":"2023-10-19T12:57:58.229465+0000","last_clean":"2023-10-19T12:57:58.229465+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:58.229465+0000","last_undegraded":"2023-10-19T12:57:58.229465+0000","last_fullsized":"2023-10-19T12:57:58.229465+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.270235+0000","last_change":"2023-10-19T12:57:59.266850+0000","last_active":"2023-10-19T12:57:59.270235+0000","last_peered":"2023-10-19T12:57:59.270235+0000","last_clean":"2023-10-19T12:57:59.270235+0000","last_became_active":"2023-10-19T12:57:59.266576+0000","last_became_peered":"2023-10-19T12:57:59.266576+0000","last_unstale":"2023-10-19T12:57:59.270235+0000","last_undegraded":"2023-10-19T12:57:59.270235+0000","last_fullsized":"2023-10-19T12:57:59.270235+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:51:51.003245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.289064+0000","last_change":"2023-10-19T12:57:59.279954+0000","last_active":"2023-10-19T12:57:59.289064+0000","last_peered":"2023-10-19T12:57:59.289064+0000","last_clean":"2023-10-19T12:57:59.289064+0000","last_became_active":"2023-10-19T12:57:59.279667+0000","last_became_peered":"2023-10-19T12:57:59.279667+0000","last_unstale":"2023-10-19T12:57:59.289064+0000","last_undegraded":"2023-10-19T12:57:59.289064+0000","last_fullsized":"2023-10-19T12:57:59.289064+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:31:43.020739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.293832+0000","last_change":"2023-10-19T12:57:59.292802+0000","last_active":"2023-10-19T12:57:59.293832+0000","last_peered":"2023-10-19T12:57:59.293832+0000","last_clean":"2023-10-19T12:57:59.293832+0000","last_became_active":"2023-10-19T12:57:59.292429+0000","last_became_peered":"2023-10-19T12:57:59.292429+0000","last_unstale":"2023-10-19T12:57:59.293832+0000","last_undegraded":"2023-10-19T12:57:59.293832+0000","last_fullsized":"2023-10-19T12:57:59.293832+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:23:37.280360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:58.229465+0000","last_change":"2023-10-19T12:57:58.229465+0000","last_active":"2023-10-19T12:57:58.229465+0000","last_peered":"2023-10-19T12:57:58.229465+0000","last_clean":"2023-10-19T12:57:58.229465+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:58.229465+0000","last_undegraded":"2023-10-19T12:57:58.229465+0000","last_fullsized":"2023-10-19T12:57:58.229465+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.277061+0000","last_change":"2023-10-19T12:57:59.283185+0000","last_active":"2023-10-19T12:58:00.277061+0000","last_peered":"2023-10-19T12:58:00.277061+0000","last_clean":"2023-10-19T12:58:00.277061+0000","last_became_active":"2023-10-19T12:57:59.282305+0000","last_became_peered":"2023-10-19T12:57:59.282305+0000","last_unstale":"2023-10-19T12:58:00.277061+0000","last_undegraded":"2023-10-19T12:58:00.277061+0000","last_fullsized":"2023-10-19T12:58:00.277061+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:07:58.427104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:58.229465+0000","last_change":"2023-10-19T12:57:58.229465+0000","last_active":"2023-10-19T12:57:58.229465+0000","last_peered":"2023-10-19T12:57:58.229465+0000","last_clean":"2023-10-19T12:57:58.229465+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:58.229465+0000","last_undegraded":"2023-10-19T12:57:58.229465+0000","last_fullsized":"2023-10-19T12:57:58.229465+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.288166+0000","last_change":"2023-10-19T12:57:59.281600+0000","last_active":"2023-10-19T12:57:59.288166+0000","last_peered":"2023-10-19T12:57:59.288166+0000","last_clean":"2023-10-19T12:57:59.288166+0000","last_became_active":"2023-10-19T12:57:59.281239+0000","last_became_peered":"2023-10-19T12:57:59.281239+0000","last_unstale":"2023-10-19T12:57:59.288166+0000","last_undegraded":"2023-10-19T12:57:59.288166+0000","last_fullsized":"2023-10-19T12:57:59.288166+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:47:41.975167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:58.229465+0000","last_change":"2023-10-19T12:57:58.229465+0000","last_active":"2023-10-19T12:57:58.229465+0000","last_peered":"2023-10-19T12:57:58.229465+0000","last_clean":"2023-10-19T12:57:58.229465+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:58.229465+0000","last_undegraded":"2023-10-19T12:57:58.229465+0000","last_fullsized":"2023-10-19T12:57:58.229465+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.286544+0000","last_change":"2023-10-19T12:57:59.280469+0000","last_active":"2023-10-19T12:57:59.286544+0000","last_peered":"2023-10-19T12:57:59.286544+0000","last_clean":"2023-10-19T12:57:59.286544+0000","last_became_active":"2023-10-19T12:57:59.280002+0000","last_became_peered":"2023-10-19T12:57:59.280002+0000","last_unstale":"2023-10-19T12:57:59.286544+0000","last_undegraded":"2023-10-19T12:57:59.286544+0000","last_fullsized":"2023-10-19T12:57:59.286544+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:54:04.339773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:58.229465+0000","last_change":"2023-10-19T12:57:58.229465+0000","last_active":"2023-10-19T12:57:58.229465+0000","last_peered":"2023-10-19T12:57:58.229465+0000","last_clean":"2023-10-19T12:57:58.229465+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:58.229465+0000","last_undegraded":"2023-10-19T12:57:58.229465+0000","last_fullsized":"2023-10-19T12:57:58.229465+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:58.229465+0000","last_change":"2023-10-19T12:57:58.229465+0000","last_active":"2023-10-19T12:57:58.229465+0000","last_peered":"2023-10-19T12:57:58.229465+0000","last_clean":"2023-10-19T12:57:58.229465+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:58.229465+0000","last_undegraded":"2023-10-19T12:57:58.229465+0000","last_fullsized":"2023-10-19T12:57:58.229465+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:57.219060+0000","last_change":"2023-10-19T12:57:57.219060+0000","last_active":"2023-10-19T12:57:57.219060+0000","last_peered":"2023-10-19T12:57:57.219060+0000","last_clean":"2023-10-19T12:57:57.219060+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:57.219060+0000","last_undegraded":"2023-10-19T12:57:57.219060+0000","last_fullsized":"2023-10-19T12:57:57.219060+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:58.229465+0000","last_change":"2023-10-19T12:57:58.229465+0000","last_active":"2023-10-19T12:57:58.229465+0000","last_peered":"2023-10-19T12:57:58.229465+0000","last_clean":"2023-10-19T12:57:58.229465+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:58.229465+0000","last_undegraded":"2023-10-19T12:57:58.229465+0000","last_fullsized":"2023-10-19T12:57:58.229465+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1e","version":"69'1","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.276793+0000","last_change":"2023-10-19T12:57:58.256650+0000","last_active":"2023-10-19T12:58:00.276793+0000","last_peered":"2023-10-19T12:58:00.276793+0000","last_clean":"2023-10-19T12:58:00.276793+0000","last_became_active":"2023-10-19T12:57:58.256210+0000","last_became_peered":"2023-10-19T12:57:58.256210+0000","last_unstale":"2023-10-19T12:58:00.276793+0000","last_undegraded":"2023-10-19T12:58:00.276793+0000","last_fullsized":"2023-10-19T12:58:00.276793+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:19:19.078844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:58.229465+0000","last_change":"2023-10-19T12:57:58.229465+0000","last_active":"2023-10-19T12:57:58.229465+0000","last_peered":"2023-10-19T12:57:58.229465+0000","last_clean":"2023-10-19T12:57:58.229465+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:58.229465+0000","last_undegraded":"2023-10-19T12:57:58.229465+0000","last_fullsized":"2023-10-19T12:57:58.229465+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1d","version":"69'2","reported_seq":14,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.277103+0000","last_change":"2023-10-19T12:57:58.255163+0000","last_active":"2023-10-19T12:58:00.277103+0000","last_peered":"2023-10-19T12:58:00.277103+0000","last_clean":"2023-10-19T12:58:00.277103+0000","last_became_active":"2023-10-19T12:57:58.254897+0000","last_became_peered":"2023-10-19T12:57:58.254897+0000","last_unstale":"2023-10-19T12:58:00.277103+0000","last_undegraded":"2023-10-19T12:58:00.277103+0000","last_fullsized":"2023-10-19T12:58:00.277103+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:48:56.789976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:58.229465+0000","last_change":"2023-10-19T12:57:58.229465+0000","last_active":"2023-10-19T12:57:58.229465+0000","last_peered":"2023-10-19T12:57:58.229465+0000","last_clean":"2023-10-19T12:57:58.229465+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:58.229465+0000","last_undegraded":"2023-10-19T12:57:58.229465+0000","last_fullsized":"2023-10-19T12:57:58.229465+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:57.219060+0000","last_change":"2023-10-19T12:57:57.219060+0000","last_active":"2023-10-19T12:57:57.219060+0000","last_peered":"2023-10-19T12:57:57.219060+0000","last_clean":"2023-10-19T12:57:57.219060+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:57.219060+0000","last_undegraded":"2023-10-19T12:57:57.219060+0000","last_fullsized":"2023-10-19T12:57:57.219060+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:58.229465+0000","last_change":"2023-10-19T12:57:58.229465+0000","last_active":"2023-10-19T12:57:58.229465+0000","last_peered":"2023-10-19T12:57:58.229465+0000","last_clean":"2023-10-19T12:57:58.229465+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:58.229465+0000","last_undegraded":"2023-10-19T12:57:58.229465+0000","last_fullsized":"2023-10-19T12:57:58.229465+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1b","version":"69'1","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.293130+0000","last_change":"2023-10-19T12:57:58.257095+0000","last_active":"2023-10-19T12:57:59.293130+0000","last_peered":"2023-10-19T12:57:59.293130+0000","last_clean":"2023-10-19T12:57:59.293130+0000","last_became_active":"2023-10-19T12:57:58.256684+0000","last_became_peered":"2023-10-19T12:57:58.256684+0000","last_unstale":"2023-10-19T12:57:59.293130+0000","last_undegraded":"2023-10-19T12:57:59.293130+0000","last_fullsized":"2023-10-19T12:57:59.293130+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:39:16.198486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:58.229465+0000","last_change":"2023-10-19T12:57:58.229465+0000","last_active":"2023-10-19T12:57:58.229465+0000","last_peered":"2023-10-19T12:57:58.229465+0000","last_clean":"2023-10-19T12:57:58.229465+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:58.229465+0000","last_undegraded":"2023-10-19T12:57:58.229465+0000","last_fullsized":"2023-10-19T12:57:58.229465+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:57.219060+0000","last_change":"2023-10-19T12:57:57.219060+0000","last_active":"2023-10-19T12:57:57.219060+0000","last_peered":"2023-10-19T12:57:57.219060+0000","last_clean":"2023-10-19T12:57:57.219060+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:57.219060+0000","last_undegraded":"2023-10-19T12:57:57.219060+0000","last_fullsized":"2023-10-19T12:57:57.219060+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:58.229465+0000","last_change":"2023-10-19T12:57:58.229465+0000","last_active":"2023-10-19T12:57:58.229465+0000","last_peered":"2023-10-19T12:57:58.229465+0000","last_clean":"2023-10-19T12:57:58.229465+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:58.229465+0000","last_undegraded":"2023-10-19T12:57:58.229465+0000","last_fullsized":"2023-10-19T12:57:58.229465+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:57.219060+0000","last_change":"2023-10-19T12:57:57.219060+0000","last_active":"2023-10-19T12:57:57.219060+0000","last_peered":"2023-10-19T12:57:57.219060+0000","last_clean":"2023-10-19T12:57:57.219060+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:57.219060+0000","last_undegraded":"2023-10-19T12:57:57.219060+0000","last_fullsized":"2023-10-19T12:57:57.219060+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:58.229465+0000","last_change":"2023-10-19T12:57:58.229465+0000","last_active":"2023-10-19T12:57:58.229465+0000","last_peered":"2023-10-19T12:57:58.229465+0000","last_clean":"2023-10-19T12:57:58.229465+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:58.229465+0000","last_undegraded":"2023-10-19T12:57:58.229465+0000","last_fullsized":"2023-10-19T12:57:58.229465+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":11,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.266101+0000","last_change":"2023-10-19T12:57:58.259037+0000","last_active":"2023-10-19T12:57:59.266101+0000","last_peered":"2023-10-19T12:57:59.266101+0000","last_clean":"2023-10-19T12:57:59.266101+0000","last_became_active":"2023-10-19T12:57:58.258745+0000","last_became_peered":"2023-10-19T12:57:58.258745+0000","last_unstale":"2023-10-19T12:57:59.266101+0000","last_undegraded":"2023-10-19T12:57:59.266101+0000","last_fullsized":"2023-10-19T12:57:59.266101+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:29:50.196136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:58.229465+0000","last_change":"2023-10-19T12:57:58.229465+0000","last_active":"2023-10-19T12:57:58.229465+0000","last_peered":"2023-10-19T12:57:58.229465+0000","last_clean":"2023-10-19T12:57:58.229465+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:58.229465+0000","last_undegraded":"2023-10-19T12:57:58.229465+0000","last_fullsized":"2023-10-19T12:57:58.229465+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:57.219060+0000","last_change":"2023-10-19T12:57:57.219060+0000","last_active":"2023-10-19T12:57:57.219060+0000","last_peered":"2023-10-19T12:57:57.219060+0000","last_clean":"2023-10-19T12:57:57.219060+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:57.219060+0000","last_undegraded":"2023-10-19T12:57:57.219060+0000","last_fullsized":"2023-10-19T12:57:57.219060+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:58.229465+0000","last_change":"2023-10-19T12:57:58.229465+0000","last_active":"2023-10-19T12:57:58.229465+0000","last_peered":"2023-10-19T12:57:58.229465+0000","last_clean":"2023-10-19T12:57:58.229465+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:58.229465+0000","last_undegraded":"2023-10-19T12:57:58.229465+0000","last_fullsized":"2023-10-19T12:57:58.229465+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:57.219060+0000","last_change":"2023-10-19T12:57:57.219060+0000","last_active":"2023-10-19T12:57:57.219060+0000","last_peered":"2023-10-19T12:57:57.219060+0000","last_clean":"2023-10-19T12:57:57.219060+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:57.219060+0000","last_undegraded":"2023-10-19T12:57:57.219060+0000","last_fullsized":"2023-10-19T12:57:57.219060+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.288065+0000","last_change":"2023-10-19T12:57:59.282205+0000","last_active":"2023-10-19T12:57:59.288065+0000","last_peered":"2023-10-19T12:57:59.288065+0000","last_clean":"2023-10-19T12:57:59.288065+0000","last_became_active":"2023-10-19T12:57:59.281923+0000","last_became_peered":"2023-10-19T12:57:59.281923+0000","last_unstale":"2023-10-19T12:57:59.288065+0000","last_undegraded":"2023-10-19T12:57:59.288065+0000","last_fullsized":"2023-10-19T12:57:59.288065+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:32:27.978530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:57.219060+0000","last_change":"2023-10-19T12:57:57.219060+0000","last_active":"2023-10-19T12:57:57.219060+0000","last_peered":"2023-10-19T12:57:57.219060+0000","last_clean":"2023-10-19T12:57:57.219060+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:57.219060+0000","last_undegraded":"2023-10-19T12:57:57.219060+0000","last_fullsized":"2023-10-19T12:57:57.219060+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:58.229465+0000","last_change":"2023-10-19T12:57:58.229465+0000","last_active":"2023-10-19T12:57:58.229465+0000","last_peered":"2023-10-19T12:57:58.229465+0000","last_clean":"2023-10-19T12:57:58.229465+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:58.229465+0000","last_undegraded":"2023-10-19T12:57:58.229465+0000","last_fullsized":"2023-10-19T12:57:58.229465+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:57.219060+0000","last_change":"2023-10-19T12:57:57.219060+0000","last_active":"2023-10-19T12:57:57.219060+0000","last_peered":"2023-10-19T12:57:57.219060+0000","last_clean":"2023-10-19T12:57:57.219060+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:57.219060+0000","last_undegraded":"2023-10-19T12:57:57.219060+0000","last_fullsized":"2023-10-19T12:57:57.219060+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:58.229465+0000","last_change":"2023-10-19T12:57:58.229465+0000","last_active":"2023-10-19T12:57:58.229465+0000","last_peered":"2023-10-19T12:57:58.229465+0000","last_clean":"2023-10-19T12:57:58.229465+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:58.229465+0000","last_undegraded":"2023-10-19T12:57:58.229465+0000","last_fullsized":"2023-10-19T12:57:58.229465+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.13","version":"69'2","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.290333+0000","last_change":"2023-10-19T12:57:58.257253+0000","last_active":"2023-10-19T12:57:59.290333+0000","last_peered":"2023-10-19T12:57:59.290333+0000","last_clean":"2023-10-19T12:57:59.290333+0000","last_became_active":"2023-10-19T12:57:58.256982+0000","last_became_peered":"2023-10-19T12:57:58.256982+0000","last_unstale":"2023-10-19T12:57:59.290333+0000","last_undegraded":"2023-10-19T12:57:59.290333+0000","last_fullsized":"2023-10-19T12:57:59.290333+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:26:46.604704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.288024+0000","last_change":"2023-10-19T12:57:59.282874+0000","last_active":"2023-10-19T12:57:59.288024+0000","last_peered":"2023-10-19T12:57:59.288024+0000","last_clean":"2023-10-19T12:57:59.288024+0000","last_became_active":"2023-10-19T12:57:59.282521+0000","last_became_peered":"2023-10-19T12:57:59.282521+0000","last_unstale":"2023-10-19T12:57:59.288024+0000","last_undegraded":"2023-10-19T12:57:59.288024+0000","last_fullsized":"2023-10-19T12:57:59.288024+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:42:43.566749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:57.219060+0000","last_change":"2023-10-19T12:57:57.219060+0000","last_active":"2023-10-19T12:57:57.219060+0000","last_peered":"2023-10-19T12:57:57.219060+0000","last_clean":"2023-10-19T12:57:57.219060+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:57.219060+0000","last_undegraded":"2023-10-19T12:57:57.219060+0000","last_fullsized":"2023-10-19T12:57:57.219060+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:58.229465+0000","last_change":"2023-10-19T12:57:58.229465+0000","last_active":"2023-10-19T12:57:58.229465+0000","last_peered":"2023-10-19T12:57:58.229465+0000","last_clean":"2023-10-19T12:57:58.229465+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:58.229465+0000","last_undegraded":"2023-10-19T12:57:58.229465+0000","last_fullsized":"2023-10-19T12:57:58.229465+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:57.219060+0000","last_change":"2023-10-19T12:57:57.219060+0000","last_active":"2023-10-19T12:57:57.219060+0000","last_peered":"2023-10-19T12:57:57.219060+0000","last_clean":"2023-10-19T12:57:57.219060+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:57.219060+0000","last_undegraded":"2023-10-19T12:57:57.219060+0000","last_fullsized":"2023-10-19T12:57:57.219060+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.287695+0000","last_change":"2023-10-19T12:57:59.278651+0000","last_active":"2023-10-19T12:57:59.287695+0000","last_peered":"2023-10-19T12:57:59.287695+0000","last_clean":"2023-10-19T12:57:59.287695+0000","last_became_active":"2023-10-19T12:57:59.278113+0000","last_became_peered":"2023-10-19T12:57:59.278113+0000","last_unstale":"2023-10-19T12:57:59.287695+0000","last_undegraded":"2023-10-19T12:57:59.287695+0000","last_fullsized":"2023-10-19T12:57:59.287695+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:12:20.974872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:57.219060+0000","last_change":"2023-10-19T12:57:57.219060+0000","last_active":"2023-10-19T12:57:57.219060+0000","last_peered":"2023-10-19T12:57:57.219060+0000","last_clean":"2023-10-19T12:57:57.219060+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:57.219060+0000","last_undegraded":"2023-10-19T12:57:57.219060+0000","last_fullsized":"2023-10-19T12:57:57.219060+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:58.229465+0000","last_change":"2023-10-19T12:57:58.229465+0000","last_active":"2023-10-19T12:57:58.229465+0000","last_peered":"2023-10-19T12:57:58.229465+0000","last_clean":"2023-10-19T12:57:58.229465+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:58.229465+0000","last_undegraded":"2023-10-19T12:57:58.229465+0000","last_fullsized":"2023-10-19T12:57:58.229465+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.f","version":"69'1","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.292498+0000","last_change":"2023-10-19T12:57:58.257724+0000","last_active":"2023-10-19T12:57:59.292498+0000","last_peered":"2023-10-19T12:57:59.292498+0000","last_clean":"2023-10-19T12:57:59.292498+0000","last_became_active":"2023-10-19T12:57:58.257432+0000","last_became_peered":"2023-10-19T12:57:58.257432+0000","last_unstale":"2023-10-19T12:57:59.292498+0000","last_undegraded":"2023-10-19T12:57:59.292498+0000","last_fullsized":"2023-10-19T12:57:59.292498+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:21:40.593175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.276683+0000","last_change":"2023-10-19T12:57:59.274011+0000","last_active":"2023-10-19T12:58:00.276683+0000","last_peered":"2023-10-19T12:58:00.276683+0000","last_clean":"2023-10-19T12:58:00.276683+0000","last_became_active":"2023-10-19T12:57:59.273742+0000","last_became_peered":"2023-10-19T12:57:59.273742+0000","last_unstale":"2023-10-19T12:58:00.276683+0000","last_undegraded":"2023-10-19T12:58:00.276683+0000","last_fullsized":"2023-10-19T12:58:00.276683+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:36:39.520498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"69'2","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.288309+0000","last_change":"2023-10-19T12:57:58.249377+0000","last_active":"2023-10-19T12:57:59.288309+0000","last_peered":"2023-10-19T12:57:59.288309+0000","last_clean":"2023-10-19T12:57:59.288309+0000","last_became_active":"2023-10-19T12:57:58.249095+0000","last_became_peered":"2023-10-19T12:57:58.249095+0000","last_unstale":"2023-10-19T12:57:59.288309+0000","last_undegraded":"2023-10-19T12:57:59.288309+0000","last_fullsized":"2023-10-19T12:57:59.288309+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:54:31.441800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:58.229465+0000","last_change":"2023-10-19T12:57:58.229465+0000","last_active":"2023-10-19T12:57:58.229465+0000","last_peered":"2023-10-19T12:57:58.229465+0000","last_clean":"2023-10-19T12:57:58.229465+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:58.229465+0000","last_undegraded":"2023-10-19T12:57:58.229465+0000","last_fullsized":"2023-10-19T12:57:58.229465+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:57.219060+0000","last_change":"2023-10-19T12:57:57.219060+0000","last_active":"2023-10-19T12:57:57.219060+0000","last_peered":"2023-10-19T12:57:57.219060+0000","last_clean":"2023-10-19T12:57:57.219060+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:57.219060+0000","last_undegraded":"2023-10-19T12:57:57.219060+0000","last_fullsized":"2023-10-19T12:57:57.219060+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.276881+0000","last_change":"2023-10-19T12:57:59.277839+0000","last_active":"2023-10-19T12:58:00.276881+0000","last_peered":"2023-10-19T12:58:00.276881+0000","last_clean":"2023-10-19T12:58:00.276881+0000","last_became_active":"2023-10-19T12:57:59.277539+0000","last_became_peered":"2023-10-19T12:57:59.277539+0000","last_unstale":"2023-10-19T12:58:00.276881+0000","last_undegraded":"2023-10-19T12:58:00.276881+0000","last_fullsized":"2023-10-19T12:58:00.276881+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:47:49.700629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:57.219060+0000","last_change":"2023-10-19T12:57:57.219060+0000","last_active":"2023-10-19T12:57:57.219060+0000","last_peered":"2023-10-19T12:57:57.219060+0000","last_clean":"2023-10-19T12:57:57.219060+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:57.219060+0000","last_undegraded":"2023-10-19T12:57:57.219060+0000","last_fullsized":"2023-10-19T12:57:57.219060+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:58.229465+0000","last_change":"2023-10-19T12:57:58.229465+0000","last_active":"2023-10-19T12:57:58.229465+0000","last_peered":"2023-10-19T12:57:58.229465+0000","last_clean":"2023-10-19T12:57:58.229465+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:58.229465+0000","last_undegraded":"2023-10-19T12:57:58.229465+0000","last_fullsized":"2023-10-19T12:57:58.229465+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.b","version":"69'2","reported_seq":14,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.276952+0000","last_change":"2023-10-19T12:57:58.254673+0000","last_active":"2023-10-19T12:58:00.276952+0000","last_peered":"2023-10-19T12:58:00.276952+0000","last_clean":"2023-10-19T12:58:00.276952+0000","last_became_active":"2023-10-19T12:57:58.254408+0000","last_became_peered":"2023-10-19T12:57:58.254408+0000","last_unstale":"2023-10-19T12:58:00.276952+0000","last_undegraded":"2023-10-19T12:58:00.276952+0000","last_fullsized":"2023-10-19T12:58:00.276952+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:25:49.650520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:58.229465+0000","last_change":"2023-10-19T12:57:58.229465+0000","last_active":"2023-10-19T12:57:58.229465+0000","last_peered":"2023-10-19T12:57:58.229465+0000","last_clean":"2023-10-19T12:57:58.229465+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:58.229465+0000","last_undegraded":"2023-10-19T12:57:58.229465+0000","last_fullsized":"2023-10-19T12:57:58.229465+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:57.219060+0000","last_change":"2023-10-19T12:57:57.219060+0000","last_active":"2023-10-19T12:57:57.219060+0000","last_peered":"2023-10-19T12:57:57.219060+0000","last_clean":"2023-10-19T12:57:57.219060+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:57.219060+0000","last_undegraded":"2023-10-19T12:57:57.219060+0000","last_fullsized":"2023-10-19T12:57:57.219060+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.289238+0000","last_change":"2023-10-19T12:57:59.275897+0000","last_active":"2023-10-19T12:57:59.289238+0000","last_peered":"2023-10-19T12:57:59.289238+0000","last_clean":"2023-10-19T12:57:59.289238+0000","last_became_active":"2023-10-19T12:57:59.274965+0000","last_became_peered":"2023-10-19T12:57:59.274965+0000","last_unstale":"2023-10-19T12:57:59.289238+0000","last_undegraded":"2023-10-19T12:57:59.289238+0000","last_fullsized":"2023-10-19T12:57:59.289238+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:30:55.579208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":11,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.267824+0000","last_change":"2023-10-19T12:57:58.255960+0000","last_active":"2023-10-19T12:57:59.267824+0000","last_peered":"2023-10-19T12:57:59.267824+0000","last_clean":"2023-10-19T12:57:59.267824+0000","last_became_active":"2023-10-19T12:57:58.255655+0000","last_became_peered":"2023-10-19T12:57:58.255655+0000","last_unstale":"2023-10-19T12:57:59.267824+0000","last_undegraded":"2023-10-19T12:57:59.267824+0000","last_fullsized":"2023-10-19T12:57:59.267824+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:35:32.478337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.276919+0000","last_change":"2023-10-19T12:57:59.284564+0000","last_active":"2023-10-19T12:58:00.276919+0000","last_peered":"2023-10-19T12:58:00.276919+0000","last_clean":"2023-10-19T12:58:00.276919+0000","last_became_active":"2023-10-19T12:57:59.278312+0000","last_became_peered":"2023-10-19T12:57:59.278312+0000","last_unstale":"2023-10-19T12:58:00.276919+0000","last_undegraded":"2023-10-19T12:58:00.276919+0000","last_fullsized":"2023-10-19T12:58:00.276919+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:19:34.001505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":11,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.255502+0000","last_change":"2023-10-19T12:57:58.248153+0000","last_active":"2023-10-19T12:57:59.255502+0000","last_peered":"2023-10-19T12:57:59.255502+0000","last_clean":"2023-10-19T12:57:59.255502+0000","last_became_active":"2023-10-19T12:57:58.247864+0000","last_became_peered":"2023-10-19T12:57:58.247864+0000","last_unstale":"2023-10-19T12:57:59.255502+0000","last_undegraded":"2023-10-19T12:57:59.255502+0000","last_fullsized":"2023-10-19T12:57:59.255502+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:53:21.688950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:58.229465+0000","last_change":"2023-10-19T12:57:58.229465+0000","last_active":"2023-10-19T12:57:58.229465+0000","last_peered":"2023-10-19T12:57:58.229465+0000","last_clean":"2023-10-19T12:57:58.229465+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:58.229465+0000","last_undegraded":"2023-10-19T12:57:58.229465+0000","last_fullsized":"2023-10-19T12:57:58.229465+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:57.219060+0000","last_change":"2023-10-19T12:57:57.219060+0000","last_active":"2023-10-19T12:57:57.219060+0000","last_peered":"2023-10-19T12:57:57.219060+0000","last_clean":"2023-10-19T12:57:57.219060+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:57.219060+0000","last_undegraded":"2023-10-19T12:57:57.219060+0000","last_fullsized":"2023-10-19T12:57:57.219060+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.276817+0000","last_change":"2023-10-19T12:57:59.271932+0000","last_active":"2023-10-19T12:58:00.276817+0000","last_peered":"2023-10-19T12:58:00.276817+0000","last_clean":"2023-10-19T12:58:00.276817+0000","last_became_active":"2023-10-19T12:57:59.271620+0000","last_became_peered":"2023-10-19T12:57:59.271620+0000","last_unstale":"2023-10-19T12:58:00.276817+0000","last_undegraded":"2023-10-19T12:58:00.276817+0000","last_fullsized":"2023-10-19T12:58:00.276817+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:13:06.950010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:57.219060+0000","last_change":"2023-10-19T12:57:57.219060+0000","last_active":"2023-10-19T12:57:57.219060+0000","last_peered":"2023-10-19T12:57:57.219060+0000","last_clean":"2023-10-19T12:57:57.219060+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:57.219060+0000","last_undegraded":"2023-10-19T12:57:57.219060+0000","last_fullsized":"2023-10-19T12:57:57.219060+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:58.229465+0000","last_change":"2023-10-19T12:57:58.229465+0000","last_active":"2023-10-19T12:57:58.229465+0000","last_peered":"2023-10-19T12:57:58.229465+0000","last_clean":"2023-10-19T12:57:58.229465+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:58.229465+0000","last_undegraded":"2023-10-19T12:57:58.229465+0000","last_fullsized":"2023-10-19T12:57:58.229465+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":11,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.265565+0000","last_change":"2023-10-19T12:57:58.253449+0000","last_active":"2023-10-19T12:57:59.265565+0000","last_peered":"2023-10-19T12:57:59.265565+0000","last_clean":"2023-10-19T12:57:59.265565+0000","last_became_active":"2023-10-19T12:57:58.253169+0000","last_became_peered":"2023-10-19T12:57:58.253169+0000","last_unstale":"2023-10-19T12:57:59.265565+0000","last_undegraded":"2023-10-19T12:57:59.265565+0000","last_fullsized":"2023-10-19T12:57:59.265565+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:55:53.005401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.287287+0000","last_change":"2023-10-19T12:57:59.280611+0000","last_active":"2023-10-19T12:57:59.287287+0000","last_peered":"2023-10-19T12:57:59.287287+0000","last_clean":"2023-10-19T12:57:59.287287+0000","last_became_active":"2023-10-19T12:57:59.280321+0000","last_became_peered":"2023-10-19T12:57:59.280321+0000","last_unstale":"2023-10-19T12:57:59.287287+0000","last_undegraded":"2023-10-19T12:57:59.287287+0000","last_fullsized":"2023-10-19T12:57:59.287287+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:13:46.024291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:57.219060+0000","last_change":"2023-10-19T12:57:57.219060+0000","last_active":"2023-10-19T12:57:57.219060+0000","last_peered":"2023-10-19T12:57:57.219060+0000","last_clean":"2023-10-19T12:57:57.219060+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:57.219060+0000","last_undegraded":"2023-10-19T12:57:57.219060+0000","last_fullsized":"2023-10-19T12:57:57.219060+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:58.229465+0000","last_change":"2023-10-19T12:57:58.229465+0000","last_active":"2023-10-19T12:57:58.229465+0000","last_peered":"2023-10-19T12:57:58.229465+0000","last_clean":"2023-10-19T12:57:58.229465+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:58.229465+0000","last_undegraded":"2023-10-19T12:57:58.229465+0000","last_fullsized":"2023-10-19T12:57:58.229465+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:57.219060+0000","last_change":"2023-10-19T12:57:57.219060+0000","last_active":"2023-10-19T12:57:57.219060+0000","last_peered":"2023-10-19T12:57:57.219060+0000","last_clean":"2023-10-19T12:57:57.219060+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:57.219060+0000","last_undegraded":"2023-10-19T12:57:57.219060+0000","last_fullsized":"2023-10-19T12:57:57.219060+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.289255+0000","last_change":"2023-10-19T12:57:59.279293+0000","last_active":"2023-10-19T12:57:59.289255+0000","last_peered":"2023-10-19T12:57:59.289255+0000","last_clean":"2023-10-19T12:57:59.289255+0000","last_became_active":"2023-10-19T12:57:59.278399+0000","last_became_peered":"2023-10-19T12:57:59.278399+0000","last_unstale":"2023-10-19T12:57:59.289255+0000","last_undegraded":"2023-10-19T12:57:59.289255+0000","last_fullsized":"2023-10-19T12:57:59.289255+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:18:02.353710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:57.219060+0000","last_change":"2023-10-19T12:57:57.219060+0000","last_active":"2023-10-19T12:57:57.219060+0000","last_peered":"2023-10-19T12:57:57.219060+0000","last_clean":"2023-10-19T12:57:57.219060+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:57.219060+0000","last_undegraded":"2023-10-19T12:57:57.219060+0000","last_fullsized":"2023-10-19T12:57:57.219060+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:58.229465+0000","last_change":"2023-10-19T12:57:58.229465+0000","last_active":"2023-10-19T12:57:58.229465+0000","last_peered":"2023-10-19T12:57:58.229465+0000","last_clean":"2023-10-19T12:57:58.229465+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:58.229465+0000","last_undegraded":"2023-10-19T12:57:58.229465+0000","last_fullsized":"2023-10-19T12:57:58.229465+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:57.219060+0000","last_change":"2023-10-19T12:57:57.219060+0000","last_active":"2023-10-19T12:57:57.219060+0000","last_peered":"2023-10-19T12:57:57.219060+0000","last_clean":"2023-10-19T12:57:57.219060+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:57.219060+0000","last_undegraded":"2023-10-19T12:57:57.219060+0000","last_fullsized":"2023-10-19T12:57:57.219060+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:58.229465+0000","last_change":"2023-10-19T12:57:58.229465+0000","last_active":"2023-10-19T12:57:58.229465+0000","last_peered":"2023-10-19T12:57:58.229465+0000","last_clean":"2023-10-19T12:57:58.229465+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:58.229465+0000","last_undegraded":"2023-10-19T12:57:58.229465+0000","last_fullsized":"2023-10-19T12:57:58.229465+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":11,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.264710+0000","last_change":"2023-10-19T12:57:58.254016+0000","last_active":"2023-10-19T12:57:59.264710+0000","last_peered":"2023-10-19T12:57:59.264710+0000","last_clean":"2023-10-19T12:57:59.264710+0000","last_became_active":"2023-10-19T12:57:58.253756+0000","last_became_peered":"2023-10-19T12:57:58.253756+0000","last_unstale":"2023-10-19T12:57:59.264710+0000","last_undegraded":"2023-10-19T12:57:59.264710+0000","last_fullsized":"2023-10-19T12:57:59.264710+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:52:42.753982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"58'139","reported_seq":286,"reported_epoch":61,"state":"active+clean","last_fresh":"2023-10-19T12:57:49.219229+0000","last_change":"2023-10-19T12:44:08.636613+0000","last_active":"2023-10-19T12:57:49.219229+0000","last_peered":"2023-10-19T12:57:49.219229+0000","last_clean":"2023-10-19T12:57:49.219229+0000","last_became_active":"2023-10-19T12:44:08.635697+0000","last_became_peered":"2023-10-19T12:44:08.635697+0000","last_unstale":"2023-10-19T12:57:49.219229+0000","last_undegraded":"2023-10-19T12:57:49.219229+0000","last_fullsized":"2023-10-19T12:57:49.219229+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_clean_scrub_stamp":"2023-10-19T12:44:07.399738+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:15:23.685485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":54,"acting":54,"num_store_stats":3},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":56,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":24,"acting":24,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575017,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58856,"kb_used_data":1304,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312984,"statfs":{"total":96636764160,"available":96576495616,"internally_reserved":0,"allocated":1335296,"data_stored":957245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58915275},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:57:38 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.58499999999999996,"15min":0.56100000000000005},"min":{"1min":0.39300000000000002,"5min":0.34499999999999997,"15min":0.248},"max":{"1min":0.67600000000000005,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.59699999999999998,"15min":0.57799999999999996},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.26900000000000002},"max":{"1min":0.76100000000000001,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.59599999999999997}]},{"osd":1,"last update":"Thu Oct 19 12:57:38 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.58699999999999997,"15min":0.56899999999999995},"min":{"1min":0.40300000000000002,"5min":0.371,"15min":0.28199999999999997},"max":{"1min":0.747,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.497},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.59699999999999998,"15min":0.57299999999999995},"min":{"1min":0.46200000000000002,"5min":0.34999999999999998,"15min":0.29299999999999998},"max":{"1min":0.66700000000000004,"5min":1.284,"15min":1.284},"last":0.51100000000000001}]},{"osd":2,"last update":"Thu Oct 19 12:57:38 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.63900000000000001,"15min":0.61799999999999999},"min":{"1min":0.434,"5min":0.41699999999999998,"15min":0.375},"max":{"1min":0.78600000000000003,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.65500000000000003,"15min":0.629},"min":{"1min":0.497,"5min":0.44600000000000001,"15min":0.28399999999999997},"max":{"1min":0.80700000000000005,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.628}]},{"osd":3,"last update":"Thu Oct 19 12:57:38 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.61899999999999999,"15min":0.59499999999999997},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34399999999999997},"max":{"1min":0.79100000000000004,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.66000000000000003,"15min":0.64700000000000002},"min":{"1min":0.45300000000000001,"5min":0.44400000000000001,"15min":0.39700000000000002},"max":{"1min":0.77900000000000003,"5min":1.329,"15min":1.329},"last":0.56499999999999995}]},{"osd":4,"last update":"Thu Oct 19 12:57:38 2023","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.46899999999999997,"15min":0.46600000000000003},"min":{"1min":0.28599999999999998,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.73399999999999999,"5min":0.75800000000000001,"15min":0.80500000000000005},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.46800000000000003,"15min":0.46999999999999997},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.627,"5min":0.71999999999999997,"15min":0.78900000000000003},"last":0.42399999999999999}]},{"osd":5,"last update":"Thu Oct 19 12:57:38 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.48199999999999998,"15min":0.48799999999999999},"min":{"1min":0.30499999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.75800000000000001,"5min":0.80600000000000005,"15min":0.90400000000000003},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.52300000000000002,"15min":0.51400000000000001},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.308},"max":{"1min":0.77400000000000002,"5min":0.81999999999999995,"15min":0.92200000000000004},"last":0.54300000000000004}]},{"osd":6,"last update":"Thu Oct 19 12:57:38 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54500000000000004,"15min":0.54400000000000004},"min":{"1min":0.34100000000000003,"5min":0.308,"15min":0.308},"max":{"1min":0.80400000000000005,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.53400000000000003,"15min":0.53200000000000003},"min":{"1min":0.30299999999999999,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.72599999999999998,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.58199999999999996}]}]},{"osd":6,"up_from":13,"seq":55834575017,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31916,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339924,"statfs":{"total":96636764160,"available":96604082176,"internally_reserved":0,"allocated":753664,"data_stored":372985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:57:21 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60299999999999998,"15min":0.58999999999999997},"min":{"1min":0.311,"5min":0.311,"15min":0.24399999999999999},"max":{"1min":0.76000000000000001,"5min":0.79900000000000004,"15min":0.81999999999999995},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.60499999999999998,"15min":0.58399999999999996},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.77500000000000002,"5min":0.78000000000000003,"15min":0.81200000000000006},"last":0.47999999999999998}]},{"osd":1,"last update":"Thu Oct 19 12:57:21 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.626,"15min":0.60399999999999998},"min":{"1min":0.376,"5min":0.376,"15min":0.34799999999999998},"max":{"1min":0.80500000000000005,"5min":0.83299999999999996,"15min":0.93899999999999995},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.62,"15min":0.60599999999999998},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.28399999999999997},"max":{"1min":0.79400000000000004,"5min":0.82099999999999995,"15min":1.075},"last":0.54000000000000004}]},{"osd":2,"last update":"Thu Oct 19 12:57:21 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60799999999999998,"15min":0.58299999999999996},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.26600000000000001},"max":{"1min":0.748,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.501},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.63200000000000001,"15min":0.60599999999999998},"min":{"1min":0.40500000000000003,"5min":0.36099999999999999,"15min":0.28399999999999997},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.59499999999999997}]},{"osd":3,"last update":"Thu Oct 19 12:57:21 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.65100000000000002,"15min":0.625},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.32600000000000001},"max":{"1min":0.83099999999999996,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.64700000000000002,"15min":0.63500000000000001},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.81499999999999995,"5min":0.81699999999999995,"15min":1.0900000000000001},"last":0.61699999999999999}]},{"osd":4,"last update":"Thu Oct 19 12:57:21 2023","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.46300000000000002,"15min":0.47599999999999998},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":1.02},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.47799999999999998,"15min":0.49099999999999999},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.247},"max":{"1min":0.65400000000000003,"5min":0.752,"15min":0.80000000000000004},"last":0.60699999999999998}]},{"osd":5,"last update":"Thu Oct 19 12:57:21 2023","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.47899999999999998,"15min":0.48399999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.26600000000000001},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.505,"5min":0.51400000000000001,"15min":0.52900000000000003},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.312},"max":{"1min":0.78500000000000003,"5min":0.85699999999999998,"15min":0.98799999999999999},"last":0.67100000000000004}]},{"osd":7,"last update":"Thu Oct 19 12:57:21 2023","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.52600000000000002,"15min":0.53100000000000003},"min":{"1min":0.35699999999999998,"5min":0.314,"15min":0.28199999999999997},"max":{"1min":0.76100000000000001,"5min":0.82199999999999995,"15min":1.036},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.53900000000000003,"15min":0.55200000000000005},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":1.1299999999999999},"last":0.64300000000000002}]}]},{"osd":5,"up_from":13,"seq":55834575017,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27744,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344096,"statfs":{"total":96636764160,"available":96608354304,"internally_reserved":0,"allocated":741376,"data_stored":366877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:57:24 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.59199999999999997,"15min":0.58599999999999997},"min":{"1min":0.39800000000000002,"5min":0.26700000000000002,"15min":0.223},"max":{"1min":0.79400000000000004,"5min":0.82699999999999996,"15min":0.88800000000000001},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.60799999999999998,"15min":0.59099999999999997},"min":{"1min":0.39700000000000002,"5min":0.312,"15min":0.27800000000000002},"max":{"1min":0.72699999999999998,"5min":0.80400000000000005,"15min":0.874},"last":0.50800000000000001}]},{"osd":1,"last update":"Thu Oct 19 12:57:24 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.60499999999999998,"15min":0.59599999999999997},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.32800000000000001},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.876},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.61299999999999999,"15min":0.59999999999999998},"min":{"1min":0.439,"5min":0.34699999999999998,"15min":0.34499999999999997},"max":{"1min":0.79500000000000004,"5min":0.80100000000000005,"15min":0.83499999999999996},"last":0.72699999999999998}]},{"osd":2,"last update":"Thu Oct 19 12:57:24 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59399999999999997,"15min":0.57899999999999996},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.307},"max":{"1min":0.70899999999999996,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.62,"15min":0.61399999999999999},"min":{"1min":0.47399999999999998,"5min":0.41999999999999998,"15min":0.33700000000000002},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.93600000000000005},"last":0.55200000000000005}]},{"osd":3,"last update":"Thu Oct 19 12:57:24 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.60999999999999999,"15min":0.59999999999999998},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.39300000000000002},"max":{"1min":0.73199999999999998,"5min":0.77900000000000003,"15min":0.874},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63900000000000001,"15min":0.63400000000000001},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.41999999999999998},"max":{"1min":0.746,"5min":0.80400000000000005,"15min":0.89500000000000002},"last":0.74299999999999999}]},{"osd":4,"last update":"Thu Oct 19 12:57:24 2023","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.34499999999999997,"5min":0.32100000000000001,"15min":0.28899999999999998},"max":{"1min":0.66100000000000003,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.47099999999999997,"15min":0.47599999999999998},"min":{"1min":0.307,"5min":0.307,"15min":0.23899999999999999},"max":{"1min":0.75900000000000001,"5min":0.79400000000000004,"15min":0.80500000000000005},"last":0.60199999999999998}]},{"osd":6,"last update":"Thu Oct 19 12:57:24 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.47399999999999998,"15min":0.47899999999999998},"min":{"1min":0.34100000000000003,"5min":0.31900000000000001,"15min":0.25900000000000001},"max":{"1min":0.74399999999999999,"5min":0.77500000000000002,"15min":0.79300000000000004},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.50800000000000001,"15min":0.498},"min":{"1min":0.38500000000000001,"5min":0.33600000000000002,"15min":0.30399999999999999},"max":{"1min":0.71899999999999997,"5min":0.79100000000000004,"15min":0.81299999999999994},"last":0.71199999999999997}]},{"osd":7,"last update":"Thu Oct 19 12:57:24 2023","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.47999999999999998,"15min":0.49399999999999999},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.315},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71899999999999997},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.52700000000000002,"15min":0.53400000000000003},"min":{"1min":0.39700000000000002,"5min":0.34300000000000003,"15min":0.27500000000000002},"max":{"1min":0.77900000000000003,"5min":0.78100000000000003,"15min":0.78500000000000003},"last":0.65600000000000003}]}]},{"osd":4,"up_from":13,"seq":55834575017,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31904,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339936,"statfs":{"total":96636764160,"available":96604094464,"internally_reserved":0,"allocated":741376,"data_stored":366877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:57:37 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.54900000000000004,"15min":0.55100000000000005},"min":{"1min":0.36499999999999999,"5min":0.32900000000000001,"15min":0.28899999999999998},"max":{"1min":0.72699999999999998,"5min":0.77400000000000002,"15min":0.84799999999999998},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55900000000000005,"15min":0.54800000000000004},"min":{"1min":0.38100000000000001,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.73999999999999999,"5min":0.77800000000000002,"15min":0.78200000000000003},"last":0.48499999999999999}]},{"osd":1,"last update":"Thu Oct 19 12:57:37 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.61099999999999999,"15min":0.60399999999999998},"min":{"1min":0.377,"5min":0.377,"15min":0.34799999999999998},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.86199999999999999},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.61799999999999999,"15min":0.61099999999999999},"min":{"1min":0.44800000000000001,"5min":0.40200000000000002,"15min":0.35799999999999998},"max":{"1min":0.78900000000000003,"5min":0.79500000000000004,"15min":0.873},"last":0.66400000000000003}]},{"osd":2,"last update":"Thu Oct 19 12:57:37 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.61599999999999999,"15min":0.60899999999999999},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.254},"max":{"1min":0.76000000000000001,"5min":0.85299999999999998,"15min":0.86599999999999999},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.60899999999999999,"15min":0.61699999999999999},"min":{"1min":0.44800000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.75,"5min":0.82999999999999996,"15min":0.85999999999999999},"last":0.74299999999999999}]},{"osd":3,"last update":"Thu Oct 19 12:57:37 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.58499999999999996,"15min":0.57499999999999996},"min":{"1min":0.35199999999999998,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.70199999999999996,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.64400000000000002,"15min":0.63400000000000001},"min":{"1min":0.45700000000000002,"5min":0.41299999999999998,"15min":0.37},"max":{"1min":0.77300000000000002,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.63600000000000001}]},{"osd":5,"last update":"Thu Oct 19 12:57:37 2023","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.437,"15min":0.45200000000000001},"min":{"1min":0.30599999999999999,"5min":0.29099999999999998,"15min":0.249},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.81299999999999994},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.45500000000000002,"15min":0.46400000000000002},"min":{"1min":0.32000000000000001,"5min":0.26400000000000001,"15min":0.23499999999999999},"max":{"1min":0.71999999999999997,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.70499999999999996}]},{"osd":6,"last update":"Thu Oct 19 12:57:37 2023","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45400000000000001,"15min":0.46000000000000002},"min":{"1min":0.32300000000000001,"5min":0.29499999999999998,"15min":0.26400000000000001},"max":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.78600000000000003},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.47899999999999998,"15min":0.505},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.27100000000000002},"max":{"1min":0.76800000000000002,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.77100000000000002}]},{"osd":7,"last update":"Thu Oct 19 12:57:37 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.49199999999999999,"15min":0.50900000000000001},"min":{"1min":0.38600000000000001,"5min":0.307,"15min":0.27500000000000002},"max":{"1min":0.71399999999999997,"5min":0.79100000000000004,"15min":0.80200000000000005},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.53900000000000003,"15min":0.55900000000000005},"min":{"1min":0.41799999999999998,"5min":0.32100000000000001,"15min":0.30199999999999999},"max":{"1min":0.745,"5min":0.80500000000000005,"15min":0.83499999999999996},"last":0.68600000000000005}]}]},{"osd":3,"up_from":13,"seq":55834575016,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":36068,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335772,"statfs":{"total":96636764160,"available":96599830528,"internally_reserved":0,"allocated":741376,"data_stored":366877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:57:22 2023","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.40300000000000002,"15min":0.39500000000000002},"min":{"1min":0.248,"5min":0.23599999999999999,"15min":0.20899999999999999},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":1.04},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.42299999999999999,"15min":0.40699999999999997},"min":{"1min":0.28399999999999997,"5min":0.248,"15min":0.245},"max":{"1min":0.57399999999999995,"5min":0.60999999999999999,"15min":0.64600000000000002},"last":0.26200000000000001}]},{"osd":1,"last update":"Thu Oct 19 12:57:22 2023","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.41599999999999998,"15min":0.40400000000000003},"min":{"1min":0.30599999999999999,"5min":0.26100000000000001,"15min":0.22500000000000001},"max":{"1min":0.58299999999999996,"5min":0.621,"15min":0.69899999999999995},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.46100000000000002,"15min":0.442},"min":{"1min":0.32000000000000001,"5min":0.29899999999999999,"15min":0.215},"max":{"1min":0.65200000000000002,"5min":0.65900000000000003,"15min":0.874},"last":0.312}]},{"osd":2,"last update":"Thu Oct 19 12:57:22 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.46899999999999997,"15min":0.45900000000000002},"min":{"1min":0.34699999999999998,"5min":0.32500000000000001,"15min":0.26100000000000001},"max":{"1min":0.59399999999999997,"5min":0.65800000000000003,"15min":0.73599999999999999},"last":0.375},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.498,"15min":0.48299999999999998},"min":{"1min":0.36299999999999999,"5min":0.34799999999999998,"15min":0.247},"max":{"1min":0.64000000000000001,"5min":0.72599999999999998,"15min":0.96499999999999997},"last":0.38500000000000001}]},{"osd":4,"last update":"Thu Oct 19 12:57:22 2023","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.67700000000000005,"15min":0.65900000000000003},"min":{"1min":0.49399999999999999,"5min":0.45900000000000002,"15min":0.40300000000000002},"max":{"1min":0.90500000000000003,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.70199999999999996,"15min":0.68700000000000006},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.42099999999999999},"max":{"1min":0.91300000000000003,"5min":0.98999999999999999,"15min":0.99399999999999999},"last":0.63100000000000001}]},{"osd":5,"last update":"Thu Oct 19 12:57:22 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.66900000000000004,"15min":0.64900000000000002},"min":{"1min":0.495,"5min":0.495,"15min":0.40500000000000003},"max":{"1min":0.93899999999999995,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.65700000000000003,"15min":0.64500000000000002},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.35899999999999999},"max":{"1min":0.85899999999999999,"5min":0.97299999999999998,"15min":0.99099999999999999},"last":0.57699999999999996}]},{"osd":6,"last update":"Thu Oct 19 12:57:22 2023","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.66500000000000004,"15min":0.65400000000000003},"min":{"1min":0.439,"5min":0.439,"15min":0.41199999999999998},"max":{"1min":0.84699999999999998,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.68100000000000005,"15min":0.66700000000000004},"min":{"1min":0.55200000000000005,"5min":0.496,"15min":0.35899999999999999},"max":{"1min":0.86899999999999999,"5min":1.05,"15min":1.05},"last":0.621}]},{"osd":7,"last update":"Thu Oct 19 12:57:22 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.67400000000000004,"15min":0.66500000000000004},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.39800000000000002},"max":{"1min":0.95699999999999996,"5min":1,"15min":1},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.68400000000000005,"15min":0.67100000000000004},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.88500000000000001,"5min":0.93400000000000005,"15min":0.97599999999999998},"last":0.65700000000000003}]}]},{"osd":2,"up_from":13,"seq":55834575019,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54216,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317624,"statfs":{"total":96636764160,"available":96581246976,"internally_reserved":0,"allocated":839680,"data_stored":411150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:57:25 2023","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.38900000000000001,"15min":0.38500000000000001},"min":{"1min":0.254,"5min":0.20899999999999999,"15min":0.187},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.83999999999999997},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.376,"5min":0.41699999999999998,"15min":0.40100000000000002},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.57799999999999996,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.36199999999999999}]},{"osd":1,"last update":"Thu Oct 19 12:57:25 2023","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.40799999999999997,"15min":0.40000000000000002},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.57899999999999996,"5min":0.71099999999999997,"15min":0.80400000000000005},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.44,"15min":0.43099999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.23400000000000001},"max":{"1min":0.57599999999999996,"5min":0.63,"15min":0.78600000000000003},"last":0.32000000000000001}]},{"osd":3,"last update":"Thu Oct 19 12:57:25 2023","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.46700000000000003,"15min":0.45600000000000002},"min":{"1min":0.25900000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.623,"5min":0.68000000000000005,"15min":0.79600000000000004},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.497,"15min":0.48799999999999999},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.87},"last":0.375}]},{"osd":4,"last update":"Thu Oct 19 12:57:25 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.64800000000000002,"15min":0.64100000000000001},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":1.0569999999999999},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.66400000000000003,"15min":0.65500000000000003},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.32000000000000001},"max":{"1min":0.91200000000000003,"5min":0.92300000000000004,"15min":1.0780000000000001},"last":0.45400000000000001}]},{"osd":5,"last update":"Thu Oct 19 12:57:25 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.68100000000000005,"15min":0.66000000000000003},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.78600000000000003,"5min":0.96499999999999997,"15min":0.99399999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.67500000000000004,"15min":0.65700000000000003},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.90200000000000002,"5min":0.94599999999999995,"15min":0.97099999999999997},"last":0.46999999999999997}]},{"osd":6,"last update":"Thu Oct 19 12:57:25 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.67900000000000005,"15min":0.67200000000000004},"min":{"1min":0.441,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.86799999999999999,"5min":1.008,"15min":1.008},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.68200000000000005,"15min":0.66900000000000004},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.93899999999999995,"5min":0.995,"15min":0.995},"last":0.623}]},{"osd":7,"last update":"Thu Oct 19 12:57:25 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.69499999999999995,"15min":0.68400000000000005},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.36399999999999999},"max":{"1min":0.98999999999999999,"5min":1.0149999999999999,"15min":1.111},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.69799999999999995,"15min":0.68500000000000005},"min":{"1min":0.42099999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.89700000000000002,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.60599999999999998}]}]},{"osd":1,"up_from":13,"seq":55834575018,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32004,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339836,"statfs":{"total":96636764160,"available":96603992064,"internally_reserved":0,"allocated":835584,"data_stored":409698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:57:32 2023","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.39200000000000002,"15min":0.378},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.187},"max":{"1min":0.59099999999999997,"5min":0.61799999999999999,"15min":0.66200000000000003},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.443,"15min":0.42199999999999999},"min":{"1min":0.30499999999999999,"5min":0.28000000000000003,"15min":0.21199999999999999},"max":{"1min":0.63600000000000001,"5min":0.68799999999999994,"15min":0.69799999999999995},"last":0.33900000000000002}]},{"osd":2,"last update":"Thu Oct 19 12:57:32 2023","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.44600000000000001,"15min":0.42099999999999999},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.20100000000000001},"max":{"1min":0.56000000000000005,"5min":0.68999999999999995,"15min":0.69999999999999996},"last":0.374},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.45800000000000002,"15min":0.44400000000000001},"min":{"1min":0.34699999999999998,"5min":0.309,"15min":0.23699999999999999},"max":{"1min":0.60599999999999998,"5min":0.67500000000000004,"15min":0.755},"last":0.41299999999999998}]},{"osd":3,"last update":"Thu Oct 19 12:57:32 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.49099999999999999,"15min":0.47699999999999998},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.26200000000000001},"max":{"1min":0.63500000000000001,"5min":0.65900000000000003,"15min":0.83799999999999997},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.435,"15min":0.42499999999999999},"min":{"1min":0.29499999999999998,"5min":0.28499999999999998,"15min":0.27000000000000002},"max":{"1min":0.56499999999999995,"5min":0.70699999999999996,"15min":0.80500000000000005},"last":0.35399999999999998}]},{"osd":4,"last update":"Thu Oct 19 12:57:32 2023","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.67500000000000004,"15min":0.65700000000000003},"min":{"1min":0.44,"5min":0.44,"15min":0.39300000000000002},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.622},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.66600000000000004,"15min":0.65200000000000002},"min":{"1min":0.495,"5min":0.42599999999999999,"15min":0.314},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":1.1910000000000001},"last":0.68799999999999994}]},{"osd":5,"last update":"Thu Oct 19 12:57:32 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.66700000000000004,"15min":0.65700000000000003},"min":{"1min":0.52300000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.878,"5min":0.878,"15min":0.91800000000000004},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.69099999999999995,"15min":0.67200000000000004},"min":{"1min":0.53700000000000003,"5min":0.47099999999999997,"15min":0.38700000000000001},"max":{"1min":0.88500000000000001,"5min":0.93999999999999995,"15min":1.117},"last":0.58499999999999996}]},{"osd":6,"last update":"Thu Oct 19 12:57:32 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.67700000000000005,"15min":0.66200000000000003},"min":{"1min":0.48399999999999999,"5min":0.36799999999999999,"15min":0.36399999999999999},"max":{"1min":0.92200000000000004,"5min":1.0069999999999999,"15min":1.077},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.69399999999999995,"15min":0.68500000000000005},"min":{"1min":0.47399999999999998,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.995,"5min":0.995,"15min":1.0009999999999999},"last":0.64900000000000002}]},{"osd":7,"last update":"Thu Oct 19 12:57:32 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.69499999999999995,"15min":0.68100000000000005},"min":{"1min":0.54000000000000004,"5min":0.46999999999999997,"15min":0.443},"max":{"1min":0.84499999999999997,"5min":0.97399999999999998,"15min":1.0149999999999999},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.70099999999999996,"15min":0.68300000000000005},"min":{"1min":0.46500000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":1.0489999999999999},"last":0.64300000000000002}]}]},{"osd":0,"up_from":13,"seq":55834575019,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36732,"kb_used_data":1392,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335108,"statfs":{"total":96636764160,"available":96599150592,"internally_reserved":0,"allocated":1425408,"data_stored":999343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Oct 19 12:57:36 2023","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.40100000000000002,"15min":0.38500000000000001},"min":{"1min":0.28699999999999998,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.442,"15min":0.42399999999999999},"min":{"1min":0.25,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.623,"5min":0.68700000000000006,"15min":0.746},"last":0.32500000000000001}]},{"osd":2,"last update":"Thu Oct 19 12:57:36 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.46700000000000003,"15min":0.45000000000000001},"min":{"1min":0.33900000000000002,"5min":0.26200000000000001,"15min":0.25800000000000001},"max":{"1min":0.752,"5min":0.752,"15min":0.76300000000000001},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.45100000000000001,"15min":0.42799999999999999},"min":{"1min":0.27500000000000002,"5min":0.216,"15min":0.20699999999999999},"max":{"1min":0.59399999999999997,"5min":0.68600000000000005,"15min":0.76800000000000002},"last":0.29299999999999998}]},{"osd":3,"last update":"Thu Oct 19 12:57:36 2023","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.42299999999999999,"15min":0.40100000000000002},"min":{"1min":0.24299999999999999,"5min":0.23400000000000001,"15min":0.22800000000000001},"max":{"1min":1.2829999999999999,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.38},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.51000000000000001,"15min":0.48799999999999999},"min":{"1min":0.33300000000000002,"5min":0.29399999999999998,"15min":0.29199999999999998},"max":{"1min":1.4990000000000001,"5min":1.4990000000000001,"15min":1.4990000000000001},"last":0.375}]},{"osd":4,"last update":"Thu Oct 19 12:57:36 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.65000000000000002,"15min":0.63},"min":{"1min":0.46700000000000003,"5min":0.45100000000000001,"15min":0.34999999999999998},"max":{"1min":1.361,"5min":1.361,"15min":1.361},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.65800000000000003,"15min":0.64100000000000001},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.35999999999999999},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.998},"last":0.55300000000000005}]},{"osd":5,"last update":"Thu Oct 19 12:57:36 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.69099999999999995,"15min":0.66800000000000004},"min":{"1min":0.44800000000000001,"5min":0.42099999999999999,"15min":0.38},"max":{"1min":1.325,"5min":1.325,"15min":1.325},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.69699999999999995,"15min":0.67800000000000005},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.318},"max":{"1min":0.88400000000000001,"5min":0.98699999999999999,"15min":1.056},"last":0.51500000000000001}]},{"osd":6,"last update":"Thu Oct 19 12:57:36 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.67000000000000004,"15min":0.65600000000000003},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.38500000000000001},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.438},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.68200000000000005,"15min":0.65800000000000003},"min":{"1min":0.45500000000000002,"5min":0.41999999999999998,"15min":0.38900000000000001},"max":{"1min":0.995,"5min":0.995,"15min":1.109},"last":0.47599999999999998}]},{"osd":7,"last update":"Thu Oct 19 12:57:36 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.69199999999999995,"15min":0.67700000000000005},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.41099999999999998},"max":{"1min":0.92000000000000004,"5min":0.95099999999999996,"15min":1.0089999999999999},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.67300000000000004,"15min":0.65600000000000003},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.389,"5min":1.389,"15min":1.389},"last":0.497}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-19T12:58:02.279 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-19T12:58:02.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.420+0000 7f1cacbe9640 1 -- 172.21.15.57:0/3172509960 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1ca80c9dc0 msgr2=0x7f1ca806b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:02.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.420+0000 7f1cacbe9640 1 --2- 172.21.15.57:0/3172509960 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1ca80c9dc0 0x7f1ca806b2e0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f1ca806d590 tx=0x7f1c9002bd70 comp rx=0 tx=0).stop 2023-10-19T12:58:02.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.421+0000 7f1cacbe9640 1 -- 172.21.15.57:0/3172509960 shutdown_connections 2023-10-19T12:58:02.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.421+0000 7f1cacbe9640 1 --2- 172.21.15.57:0/3172509960 >> v2:172.21.15.179:3300/0 conn(0x7f1ca80ab500 0x7f1ca80ab990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:02.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.421+0000 7f1cacbe9640 1 --2- 172.21.15.57:0/3172509960 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1ca80c9dc0 0x7f1ca806b2e0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:02.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.421+0000 7f1cacbe9640 1 -- 172.21.15.57:0/3172509960 >> 172.21.15.57:0/3172509960 conn(0x7f1ca8069930 msgr2=0x7f1ca8069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:58:02.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.421+0000 7f1cacbe9640 1 -- 172.21.15.57:0/3172509960 shutdown_connections 2023-10-19T12:58:02.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.421+0000 7f1cacbe9640 1 -- 172.21.15.57:0/3172509960 wait complete. 2023-10-19T12:58:02.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.422+0000 7f1cacbe9640 1 Processor -- start 2023-10-19T12:58:02.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.422+0000 7f1cacbe9640 1 -- start start 2023-10-19T12:58:02.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.423+0000 7f1cacbe9640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1ca806af00 0x7f1ca813d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:02.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.423+0000 7f1cacbe9640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1ca80ab500 0x7f1ca80ae820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:02.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.423+0000 7f1cacbe9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1ca80c9dc0 0x7f1ca80aed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:02.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.423+0000 7f1cacbe9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ca813dde0 con 0x7f1ca80c9dc0 2023-10-19T12:58:02.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.423+0000 7f1cacbe9640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ca813d410 con 0x7f1ca80ab500 2023-10-19T12:58:02.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.423+0000 7f1cacbe9640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ca813c800 con 0x7f1ca806af00 2023-10-19T12:58:02.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.423+0000 7f1ca6d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1ca80c9dc0 0x7f1ca80aed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:02.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.423+0000 7f1ca6d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1ca80c9dc0 0x7f1ca80aed60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:41746/0 (socket says 172.21.15.57:41746) 2023-10-19T12:58:02.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.423+0000 7f1ca6d9d640 1 -- 172.21.15.57:0/485193888 learned_addr learned my addr 172.21.15.57:0/485193888 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:58:02.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.423+0000 7f1c9ffff640 1 --2- 172.21.15.57:0/485193888 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1ca80ab500 0x7f1ca80ae820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:02.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.423+0000 7f1ca659c640 1 --2- 172.21.15.57:0/485193888 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1ca806af00 0x7f1ca813d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:02.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.423+0000 7f1ca6d9d640 1 -- 172.21.15.57:0/485193888 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1ca806af00 msgr2=0x7f1ca813d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:02.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.423+0000 7f1ca6d9d640 1 --2- 172.21.15.57:0/485193888 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1ca806af00 0x7f1ca813d8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:02.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.423+0000 7f1ca6d9d640 1 -- 172.21.15.57:0/485193888 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1ca80ab500 msgr2=0x7f1ca80ae820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:02.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.423+0000 7f1ca6d9d640 1 --2- 172.21.15.57:0/485193888 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1ca80ab500 0x7f1ca80ae820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:02.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.423+0000 7f1ca6d9d640 1 -- 172.21.15.57:0/485193888 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c90009490 con 0x7f1ca80c9dc0 2023-10-19T12:58:02.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.423+0000 7f1ca659c640 1 --2- 172.21.15.57:0/485193888 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1ca806af00 0x7f1ca813d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:58:02.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.423+0000 7f1ca6d9d640 1 --2- 172.21.15.57:0/485193888 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1ca80c9dc0 0x7f1ca80aed60 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f1ca000b970 tx=0x7f1ca000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:58:02.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.423+0000 7f1c9effd640 1 -- 172.21.15.57:0/485193888 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ca000cb40 con 0x7f1ca80c9dc0 2023-10-19T12:58:02.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.423+0000 7f1c9effd640 1 -- 172.21.15.57:0/485193888 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f1ca000cdb0 con 0x7f1ca80c9dc0 2023-10-19T12:58:02.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.423+0000 7f1c9effd640 1 -- 172.21.15.57:0/485193888 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ca00146a0 con 0x7f1ca80c9dc0 2023-10-19T12:58:02.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.423+0000 7f1cacbe9640 1 -- 172.21.15.57:0/485193888 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ca8159a20 con 0x7f1ca80c9dc0 2023-10-19T12:58:02.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.423+0000 7f1cacbe9640 1 -- 172.21.15.57:0/485193888 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ca813cdc0 con 0x7f1ca80c9dc0 2023-10-19T12:58:02.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.424+0000 7f1cacbe9640 1 -- 172.21.15.57:0/485193888 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ca813d5c0 con 0x7f1ca80c9dc0 2023-10-19T12:58:02.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.425+0000 7f1c9effd640 1 -- 172.21.15.57:0/485193888 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 20) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f1ca001d3b0 con 0x7f1ca80c9dc0 2023-10-19T12:58:02.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.425+0000 7f1c9effd640 1 --2- 172.21.15.57:0/485193888 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f1c8806af60 0x7f1c8806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:02.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.425+0000 7f1c9effd640 1 -- 172.21.15.57:0/485193888 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f1ca0089ad0 con 0x7f1ca80c9dc0 2023-10-19T12:58:02.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.426+0000 7f1ca659c640 1 --2- 172.21.15.57:0/485193888 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f1c8806af60 0x7f1c8806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:02.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.426+0000 7f1ca659c640 1 --2- 172.21.15.57:0/485193888 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f1c8806af60 0x7f1c8806d420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f1ca8158f00 tx=0x7f1c98009040 comp rx=0 tx=0).ready entity=mgr.5809 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:58:02.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.430+0000 7f1c9effd640 1 -- 172.21.15.57:0/485193888 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f1ca813d5c0 con 0x7f1ca80c9dc0 2023-10-19T12:58:02.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.604+0000 7f1cacbe9640 1 -- 172.21.15.57:0/485193888 --> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1ca80abed0 con 0x7f1c8806af60 2023-10-19T12:58:02.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.612+0000 7f1c9effd640 1 -- 172.21.15.57:0/485193888 <== mgr.5809 v2:172.21.15.179:6832/3449041082 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+280765 (secure 0 0 0) 0x7f1c98001c50 con 0x7f1c8806af60 2023-10-19T12:58:02.617 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:58:02.619 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2023-10-19T12:58:02.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.615+0000 7f1cacbe9640 1 -- 172.21.15.57:0/485193888 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f1c8806af60 msgr2=0x7f1c8806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:02.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.615+0000 7f1cacbe9640 1 --2- 172.21.15.57:0/485193888 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f1c8806af60 0x7f1c8806d420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f1ca8158f00 tx=0x7f1c98009040 comp rx=0 tx=0).stop 2023-10-19T12:58:02.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.615+0000 7f1cacbe9640 1 -- 172.21.15.57:0/485193888 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1ca80c9dc0 msgr2=0x7f1ca80aed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:02.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.615+0000 7f1cacbe9640 1 --2- 172.21.15.57:0/485193888 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1ca80c9dc0 0x7f1ca80aed60 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f1ca000b970 tx=0x7f1ca000be30 comp rx=0 tx=0).stop 2023-10-19T12:58:02.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.615+0000 7f1cacbe9640 1 -- 172.21.15.57:0/485193888 shutdown_connections 2023-10-19T12:58:02.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.615+0000 7f1cacbe9640 1 --2- 172.21.15.57:0/485193888 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f1c8806af60 0x7f1c8806d420 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:02.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.615+0000 7f1cacbe9640 1 --2- 172.21.15.57:0/485193888 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1ca80c9dc0 0x7f1ca80aed60 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:02.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.615+0000 7f1cacbe9640 1 --2- 172.21.15.57:0/485193888 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1ca80ab500 0x7f1ca80ae820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:02.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.615+0000 7f1cacbe9640 1 --2- 172.21.15.57:0/485193888 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1ca806af00 0x7f1ca813d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:02.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.615+0000 7f1cacbe9640 1 -- 172.21.15.57:0/485193888 >> 172.21.15.57:0/485193888 conn(0x7f1ca8069930 msgr2=0x7f1ca80c9680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:58:02.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.615+0000 7f1cacbe9640 1 -- 172.21.15.57:0/485193888 shutdown_connections 2023-10-19T12:58:02.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.615+0000 7f1cacbe9640 1 -- 172.21.15.57:0/485193888 wait complete. 2023-10-19T12:58:02.627 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2023-10-19T12:58:00.627481+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590424,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":210,"num_write_kb":2424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":80,"acting":80,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":71,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":4,"kb":754974720,"kb_used":309440,"kb_used_data":7240,"kb_used_omap":12,"kb_used_meta":302067,"kb_avail":754665280,"statfs":{"total":773094113280,"available":772777246720,"internally_reserved":0,"allocated":7413760,"data_stored":4251052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":309317177},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":56,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000943"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.276851+0000","last_change":"2023-10-19T12:57:59.284384+0000","last_active":"2023-10-19T12:58:00.276851+0000","last_peered":"2023-10-19T12:58:00.276851+0000","last_clean":"2023-10-19T12:58:00.276851+0000","last_became_active":"2023-10-19T12:57:59.283993+0000","last_became_peered":"2023-10-19T12:57:59.283993+0000","last_unstale":"2023-10-19T12:58:00.276851+0000","last_undegraded":"2023-10-19T12:58:00.276851+0000","last_fullsized":"2023-10-19T12:58:00.276851+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:37:05.744377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:58.229465+0000","last_change":"2023-10-19T12:57:58.229465+0000","last_active":"2023-10-19T12:57:58.229465+0000","last_peered":"2023-10-19T12:57:58.229465+0000","last_clean":"2023-10-19T12:57:58.229465+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:58.229465+0000","last_undegraded":"2023-10-19T12:57:58.229465+0000","last_fullsized":"2023-10-19T12:57:58.229465+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.286509+0000","last_change":"2023-10-19T12:57:59.278020+0000","last_active":"2023-10-19T12:57:59.286509+0000","last_peered":"2023-10-19T12:57:59.286509+0000","last_clean":"2023-10-19T12:57:59.286509+0000","last_became_active":"2023-10-19T12:57:59.277709+0000","last_became_peered":"2023-10-19T12:57:59.277709+0000","last_unstale":"2023-10-19T12:57:59.286509+0000","last_undegraded":"2023-10-19T12:57:59.286509+0000","last_fullsized":"2023-10-19T12:57:59.286509+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:38:38.391082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:58.229465+0000","last_change":"2023-10-19T12:57:58.229465+0000","last_active":"2023-10-19T12:57:58.229465+0000","last_peered":"2023-10-19T12:57:58.229465+0000","last_clean":"2023-10-19T12:57:58.229465+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:58.229465+0000","last_undegraded":"2023-10-19T12:57:58.229465+0000","last_fullsized":"2023-10-19T12:57:58.229465+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:58.229465+0000","last_change":"2023-10-19T12:57:58.229465+0000","last_active":"2023-10-19T12:57:58.229465+0000","last_peered":"2023-10-19T12:57:58.229465+0000","last_clean":"2023-10-19T12:57:58.229465+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:58.229465+0000","last_undegraded":"2023-10-19T12:57:58.229465+0000","last_fullsized":"2023-10-19T12:57:58.229465+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.276766+0000","last_change":"2023-10-19T12:57:59.277273+0000","last_active":"2023-10-19T12:58:00.276766+0000","last_peered":"2023-10-19T12:58:00.276766+0000","last_clean":"2023-10-19T12:58:00.276766+0000","last_became_active":"2023-10-19T12:57:59.276991+0000","last_became_peered":"2023-10-19T12:57:59.276991+0000","last_unstale":"2023-10-19T12:58:00.276766+0000","last_undegraded":"2023-10-19T12:58:00.276766+0000","last_fullsized":"2023-10-19T12:58:00.276766+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:13:10.195317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.286565+0000","last_change":"2023-10-19T12:57:59.280280+0000","last_active":"2023-10-19T12:57:59.286565+0000","last_peered":"2023-10-19T12:57:59.286565+0000","last_clean":"2023-10-19T12:57:59.286565+0000","last_became_active":"2023-10-19T12:57:59.279626+0000","last_became_peered":"2023-10-19T12:57:59.279626+0000","last_unstale":"2023-10-19T12:57:59.286565+0000","last_undegraded":"2023-10-19T12:57:59.286565+0000","last_fullsized":"2023-10-19T12:57:59.286565+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:46:43.361440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:58.229465+0000","last_change":"2023-10-19T12:57:58.229465+0000","last_active":"2023-10-19T12:57:58.229465+0000","last_peered":"2023-10-19T12:57:58.229465+0000","last_clean":"2023-10-19T12:57:58.229465+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:58.229465+0000","last_undegraded":"2023-10-19T12:57:58.229465+0000","last_fullsized":"2023-10-19T12:57:58.229465+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:58.229465+0000","last_change":"2023-10-19T12:57:58.229465+0000","last_active":"2023-10-19T12:57:58.229465+0000","last_peered":"2023-10-19T12:57:58.229465+0000","last_clean":"2023-10-19T12:57:58.229465+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:58.229465+0000","last_undegraded":"2023-10-19T12:57:58.229465+0000","last_fullsized":"2023-10-19T12:57:58.229465+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.276809+0000","last_change":"2023-10-19T12:57:59.281613+0000","last_active":"2023-10-19T12:58:00.276809+0000","last_peered":"2023-10-19T12:58:00.276809+0000","last_clean":"2023-10-19T12:58:00.276809+0000","last_became_active":"2023-10-19T12:57:59.281322+0000","last_became_peered":"2023-10-19T12:57:59.281322+0000","last_unstale":"2023-10-19T12:58:00.276809+0000","last_undegraded":"2023-10-19T12:58:00.276809+0000","last_fullsized":"2023-10-19T12:58:00.276809+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:33:13.743238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.278808+0000","last_change":"2023-10-19T12:57:59.275510+0000","last_active":"2023-10-19T12:57:59.278808+0000","last_peered":"2023-10-19T12:57:59.278808+0000","last_clean":"2023-10-19T12:57:59.278808+0000","last_became_active":"2023-10-19T12:57:59.275193+0000","last_became_peered":"2023-10-19T12:57:59.275193+0000","last_unstale":"2023-10-19T12:57:59.278808+0000","last_undegraded":"2023-10-19T12:57:59.278808+0000","last_fullsized":"2023-10-19T12:57:59.278808+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:20:23.928988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:58.229465+0000","last_change":"2023-10-19T12:57:58.229465+0000","last_active":"2023-10-19T12:57:58.229465+0000","last_peered":"2023-10-19T12:57:58.229465+0000","last_clean":"2023-10-19T12:57:58.229465+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:58.229465+0000","last_undegraded":"2023-10-19T12:57:58.229465+0000","last_fullsized":"2023-10-19T12:57:58.229465+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.276748+0000","last_change":"2023-10-19T12:57:59.279218+0000","last_active":"2023-10-19T12:58:00.276748+0000","last_peered":"2023-10-19T12:58:00.276748+0000","last_clean":"2023-10-19T12:58:00.276748+0000","last_became_active":"2023-10-19T12:57:59.278919+0000","last_became_peered":"2023-10-19T12:57:59.278919+0000","last_unstale":"2023-10-19T12:58:00.276748+0000","last_undegraded":"2023-10-19T12:58:00.276748+0000","last_fullsized":"2023-10-19T12:58:00.276748+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:50:41.250507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.286325+0000","last_change":"2023-10-19T12:57:59.280185+0000","last_active":"2023-10-19T12:57:59.286325+0000","last_peered":"2023-10-19T12:57:59.286325+0000","last_clean":"2023-10-19T12:57:59.286325+0000","last_became_active":"2023-10-19T12:57:59.279558+0000","last_became_peered":"2023-10-19T12:57:59.279558+0000","last_unstale":"2023-10-19T12:57:59.286325+0000","last_undegraded":"2023-10-19T12:57:59.286325+0000","last_fullsized":"2023-10-19T12:57:59.286325+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:27:42.893031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:58.229465+0000","last_change":"2023-10-19T12:57:58.229465+0000","last_active":"2023-10-19T12:57:58.229465+0000","last_peered":"2023-10-19T12:57:58.229465+0000","last_clean":"2023-10-19T12:57:58.229465+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:58.229465+0000","last_undegraded":"2023-10-19T12:57:58.229465+0000","last_fullsized":"2023-10-19T12:57:58.229465+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.276674+0000","last_change":"2023-10-19T12:57:59.277264+0000","last_active":"2023-10-19T12:58:00.276674+0000","last_peered":"2023-10-19T12:58:00.276674+0000","last_clean":"2023-10-19T12:58:00.276674+0000","last_became_active":"2023-10-19T12:57:59.276973+0000","last_became_peered":"2023-10-19T12:57:59.276973+0000","last_unstale":"2023-10-19T12:58:00.276674+0000","last_undegraded":"2023-10-19T12:58:00.276674+0000","last_fullsized":"2023-10-19T12:58:00.276674+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:19:17.421557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.287675+0000","last_change":"2023-10-19T12:57:59.278487+0000","last_active":"2023-10-19T12:57:59.287675+0000","last_peered":"2023-10-19T12:57:59.287675+0000","last_clean":"2023-10-19T12:57:59.287675+0000","last_became_active":"2023-10-19T12:57:59.276322+0000","last_became_peered":"2023-10-19T12:57:59.276322+0000","last_unstale":"2023-10-19T12:57:59.287675+0000","last_undegraded":"2023-10-19T12:57:59.287675+0000","last_fullsized":"2023-10-19T12:57:59.287675+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:32:09.277054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:58.229465+0000","last_change":"2023-10-19T12:57:58.229465+0000","last_active":"2023-10-19T12:57:58.229465+0000","last_peered":"2023-10-19T12:57:58.229465+0000","last_clean":"2023-10-19T12:57:58.229465+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:58.229465+0000","last_undegraded":"2023-10-19T12:57:58.229465+0000","last_fullsized":"2023-10-19T12:57:58.229465+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.286530+0000","last_change":"2023-10-19T12:57:59.280395+0000","last_active":"2023-10-19T12:57:59.286530+0000","last_peered":"2023-10-19T12:57:59.286530+0000","last_clean":"2023-10-19T12:57:59.286530+0000","last_became_active":"2023-10-19T12:57:59.280082+0000","last_became_peered":"2023-10-19T12:57:59.280082+0000","last_unstale":"2023-10-19T12:57:59.286530+0000","last_undegraded":"2023-10-19T12:57:59.286530+0000","last_fullsized":"2023-10-19T12:57:59.286530+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:52:31.351268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:58.229465+0000","last_change":"2023-10-19T12:57:58.229465+0000","last_active":"2023-10-19T12:57:58.229465+0000","last_peered":"2023-10-19T12:57:58.229465+0000","last_clean":"2023-10-19T12:57:58.229465+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:58.229465+0000","last_undegraded":"2023-10-19T12:57:58.229465+0000","last_fullsized":"2023-10-19T12:57:58.229465+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:58.229465+0000","last_change":"2023-10-19T12:57:58.229465+0000","last_active":"2023-10-19T12:57:58.229465+0000","last_peered":"2023-10-19T12:57:58.229465+0000","last_clean":"2023-10-19T12:57:58.229465+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:58.229465+0000","last_undegraded":"2023-10-19T12:57:58.229465+0000","last_fullsized":"2023-10-19T12:57:58.229465+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:58.229465+0000","last_change":"2023-10-19T12:57:58.229465+0000","last_active":"2023-10-19T12:57:58.229465+0000","last_peered":"2023-10-19T12:57:58.229465+0000","last_clean":"2023-10-19T12:57:58.229465+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:58.229465+0000","last_undegraded":"2023-10-19T12:57:58.229465+0000","last_fullsized":"2023-10-19T12:57:58.229465+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.270235+0000","last_change":"2023-10-19T12:57:59.266850+0000","last_active":"2023-10-19T12:57:59.270235+0000","last_peered":"2023-10-19T12:57:59.270235+0000","last_clean":"2023-10-19T12:57:59.270235+0000","last_became_active":"2023-10-19T12:57:59.266576+0000","last_became_peered":"2023-10-19T12:57:59.266576+0000","last_unstale":"2023-10-19T12:57:59.270235+0000","last_undegraded":"2023-10-19T12:57:59.270235+0000","last_fullsized":"2023-10-19T12:57:59.270235+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:51:51.003245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.289064+0000","last_change":"2023-10-19T12:57:59.279954+0000","last_active":"2023-10-19T12:57:59.289064+0000","last_peered":"2023-10-19T12:57:59.289064+0000","last_clean":"2023-10-19T12:57:59.289064+0000","last_became_active":"2023-10-19T12:57:59.279667+0000","last_became_peered":"2023-10-19T12:57:59.279667+0000","last_unstale":"2023-10-19T12:57:59.289064+0000","last_undegraded":"2023-10-19T12:57:59.289064+0000","last_fullsized":"2023-10-19T12:57:59.289064+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:31:43.020739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.293832+0000","last_change":"2023-10-19T12:57:59.292802+0000","last_active":"2023-10-19T12:57:59.293832+0000","last_peered":"2023-10-19T12:57:59.293832+0000","last_clean":"2023-10-19T12:57:59.293832+0000","last_became_active":"2023-10-19T12:57:59.292429+0000","last_became_peered":"2023-10-19T12:57:59.292429+0000","last_unstale":"2023-10-19T12:57:59.293832+0000","last_undegraded":"2023-10-19T12:57:59.293832+0000","last_fullsized":"2023-10-19T12:57:59.293832+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:23:37.280360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:58.229465+0000","last_change":"2023-10-19T12:57:58.229465+0000","last_active":"2023-10-19T12:57:58.229465+0000","last_peered":"2023-10-19T12:57:58.229465+0000","last_clean":"2023-10-19T12:57:58.229465+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:58.229465+0000","last_undegraded":"2023-10-19T12:57:58.229465+0000","last_fullsized":"2023-10-19T12:57:58.229465+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.277061+0000","last_change":"2023-10-19T12:57:59.283185+0000","last_active":"2023-10-19T12:58:00.277061+0000","last_peered":"2023-10-19T12:58:00.277061+0000","last_clean":"2023-10-19T12:58:00.277061+0000","last_became_active":"2023-10-19T12:57:59.282305+0000","last_became_peered":"2023-10-19T12:57:59.282305+0000","last_unstale":"2023-10-19T12:58:00.277061+0000","last_undegraded":"2023-10-19T12:58:00.277061+0000","last_fullsized":"2023-10-19T12:58:00.277061+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:07:58.427104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:58.229465+0000","last_change":"2023-10-19T12:57:58.229465+0000","last_active":"2023-10-19T12:57:58.229465+0000","last_peered":"2023-10-19T12:57:58.229465+0000","last_clean":"2023-10-19T12:57:58.229465+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:58.229465+0000","last_undegraded":"2023-10-19T12:57:58.229465+0000","last_fullsized":"2023-10-19T12:57:58.229465+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.288166+0000","last_change":"2023-10-19T12:57:59.281600+0000","last_active":"2023-10-19T12:57:59.288166+0000","last_peered":"2023-10-19T12:57:59.288166+0000","last_clean":"2023-10-19T12:57:59.288166+0000","last_became_active":"2023-10-19T12:57:59.281239+0000","last_became_peered":"2023-10-19T12:57:59.281239+0000","last_unstale":"2023-10-19T12:57:59.288166+0000","last_undegraded":"2023-10-19T12:57:59.288166+0000","last_fullsized":"2023-10-19T12:57:59.288166+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:47:41.975167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:58.229465+0000","last_change":"2023-10-19T12:57:58.229465+0000","last_active":"2023-10-19T12:57:58.229465+0000","last_peered":"2023-10-19T12:57:58.229465+0000","last_clean":"2023-10-19T12:57:58.229465+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:58.229465+0000","last_undegraded":"2023-10-19T12:57:58.229465+0000","last_fullsized":"2023-10-19T12:57:58.229465+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.286544+0000","last_change":"2023-10-19T12:57:59.280469+0000","last_active":"2023-10-19T12:57:59.286544+0000","last_peered":"2023-10-19T12:57:59.286544+0000","last_clean":"2023-10-19T12:57:59.286544+0000","last_became_active":"2023-10-19T12:57:59.280002+0000","last_became_peered":"2023-10-19T12:57:59.280002+0000","last_unstale":"2023-10-19T12:57:59.286544+0000","last_undegraded":"2023-10-19T12:57:59.286544+0000","last_fullsized":"2023-10-19T12:57:59.286544+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:54:04.339773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:58.229465+0000","last_change":"2023-10-19T12:57:58.229465+0000","last_active":"2023-10-19T12:57:58.229465+0000","last_peered":"2023-10-19T12:57:58.229465+0000","last_clean":"2023-10-19T12:57:58.229465+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:58.229465+0000","last_undegraded":"2023-10-19T12:57:58.229465+0000","last_fullsized":"2023-10-19T12:57:58.229465+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:58.229465+0000","last_change":"2023-10-19T12:57:58.229465+0000","last_active":"2023-10-19T12:57:58.229465+0000","last_peered":"2023-10-19T12:57:58.229465+0000","last_clean":"2023-10-19T12:57:58.229465+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:58.229465+0000","last_undegraded":"2023-10-19T12:57:58.229465+0000","last_fullsized":"2023-10-19T12:57:58.229465+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:57.219060+0000","last_change":"2023-10-19T12:57:57.219060+0000","last_active":"2023-10-19T12:57:57.219060+0000","last_peered":"2023-10-19T12:57:57.219060+0000","last_clean":"2023-10-19T12:57:57.219060+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:57.219060+0000","last_undegraded":"2023-10-19T12:57:57.219060+0000","last_fullsized":"2023-10-19T12:57:57.219060+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:58.229465+0000","last_change":"2023-10-19T12:57:58.229465+0000","last_active":"2023-10-19T12:57:58.229465+0000","last_peered":"2023-10-19T12:57:58.229465+0000","last_clean":"2023-10-19T12:57:58.229465+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:58.229465+0000","last_undegraded":"2023-10-19T12:57:58.229465+0000","last_fullsized":"2023-10-19T12:57:58.229465+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1e","version":"69'1","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.276793+0000","last_change":"2023-10-19T12:57:58.256650+0000","last_active":"2023-10-19T12:58:00.276793+0000","last_peered":"2023-10-19T12:58:00.276793+0000","last_clean":"2023-10-19T12:58:00.276793+0000","last_became_active":"2023-10-19T12:57:58.256210+0000","last_became_peered":"2023-10-19T12:57:58.256210+0000","last_unstale":"2023-10-19T12:58:00.276793+0000","last_undegraded":"2023-10-19T12:58:00.276793+0000","last_fullsized":"2023-10-19T12:58:00.276793+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:19:19.078844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:58.229465+0000","last_change":"2023-10-19T12:57:58.229465+0000","last_active":"2023-10-19T12:57:58.229465+0000","last_peered":"2023-10-19T12:57:58.229465+0000","last_clean":"2023-10-19T12:57:58.229465+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:58.229465+0000","last_undegraded":"2023-10-19T12:57:58.229465+0000","last_fullsized":"2023-10-19T12:57:58.229465+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1d","version":"69'2","reported_seq":14,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.277103+0000","last_change":"2023-10-19T12:57:58.255163+0000","last_active":"2023-10-19T12:58:00.277103+0000","last_peered":"2023-10-19T12:58:00.277103+0000","last_clean":"2023-10-19T12:58:00.277103+0000","last_became_active":"2023-10-19T12:57:58.254897+0000","last_became_peered":"2023-10-19T12:57:58.254897+0000","last_unstale":"2023-10-19T12:58:00.277103+0000","last_undegraded":"2023-10-19T12:58:00.277103+0000","last_fullsized":"2023-10-19T12:58:00.277103+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:48:56.789976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:58.229465+0000","last_change":"2023-10-19T12:57:58.229465+0000","last_active":"2023-10-19T12:57:58.229465+0000","last_peered":"2023-10-19T12:57:58.229465+0000","last_clean":"2023-10-19T12:57:58.229465+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:58.229465+0000","last_undegraded":"2023-10-19T12:57:58.229465+0000","last_fullsized":"2023-10-19T12:57:58.229465+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:57.219060+0000","last_change":"2023-10-19T12:57:57.219060+0000","last_active":"2023-10-19T12:57:57.219060+0000","last_peered":"2023-10-19T12:57:57.219060+0000","last_clean":"2023-10-19T12:57:57.219060+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:57.219060+0000","last_undegraded":"2023-10-19T12:57:57.219060+0000","last_fullsized":"2023-10-19T12:57:57.219060+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:58.229465+0000","last_change":"2023-10-19T12:57:58.229465+0000","last_active":"2023-10-19T12:57:58.229465+0000","last_peered":"2023-10-19T12:57:58.229465+0000","last_clean":"2023-10-19T12:57:58.229465+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:58.229465+0000","last_undegraded":"2023-10-19T12:57:58.229465+0000","last_fullsized":"2023-10-19T12:57:58.229465+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1b","version":"69'1","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.293130+0000","last_change":"2023-10-19T12:57:58.257095+0000","last_active":"2023-10-19T12:57:59.293130+0000","last_peered":"2023-10-19T12:57:59.293130+0000","last_clean":"2023-10-19T12:57:59.293130+0000","last_became_active":"2023-10-19T12:57:58.256684+0000","last_became_peered":"2023-10-19T12:57:58.256684+0000","last_unstale":"2023-10-19T12:57:59.293130+0000","last_undegraded":"2023-10-19T12:57:59.293130+0000","last_fullsized":"2023-10-19T12:57:59.293130+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:39:16.198486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:58.229465+0000","last_change":"2023-10-19T12:57:58.229465+0000","last_active":"2023-10-19T12:57:58.229465+0000","last_peered":"2023-10-19T12:57:58.229465+0000","last_clean":"2023-10-19T12:57:58.229465+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:58.229465+0000","last_undegraded":"2023-10-19T12:57:58.229465+0000","last_fullsized":"2023-10-19T12:57:58.229465+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:57.219060+0000","last_change":"2023-10-19T12:57:57.219060+0000","last_active":"2023-10-19T12:57:57.219060+0000","last_peered":"2023-10-19T12:57:57.219060+0000","last_clean":"2023-10-19T12:57:57.219060+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:57.219060+0000","last_undegraded":"2023-10-19T12:57:57.219060+0000","last_fullsized":"2023-10-19T12:57:57.219060+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:58.229465+0000","last_change":"2023-10-19T12:57:58.229465+0000","last_active":"2023-10-19T12:57:58.229465+0000","last_peered":"2023-10-19T12:57:58.229465+0000","last_clean":"2023-10-19T12:57:58.229465+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:58.229465+0000","last_undegraded":"2023-10-19T12:57:58.229465+0000","last_fullsized":"2023-10-19T12:57:58.229465+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:57.219060+0000","last_change":"2023-10-19T12:57:57.219060+0000","last_active":"2023-10-19T12:57:57.219060+0000","last_peered":"2023-10-19T12:57:57.219060+0000","last_clean":"2023-10-19T12:57:57.219060+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:57.219060+0000","last_undegraded":"2023-10-19T12:57:57.219060+0000","last_fullsized":"2023-10-19T12:57:57.219060+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:58.229465+0000","last_change":"2023-10-19T12:57:58.229465+0000","last_active":"2023-10-19T12:57:58.229465+0000","last_peered":"2023-10-19T12:57:58.229465+0000","last_clean":"2023-10-19T12:57:58.229465+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:58.229465+0000","last_undegraded":"2023-10-19T12:57:58.229465+0000","last_fullsized":"2023-10-19T12:57:58.229465+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":11,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.266101+0000","last_change":"2023-10-19T12:57:58.259037+0000","last_active":"2023-10-19T12:57:59.266101+0000","last_peered":"2023-10-19T12:57:59.266101+0000","last_clean":"2023-10-19T12:57:59.266101+0000","last_became_active":"2023-10-19T12:57:58.258745+0000","last_became_peered":"2023-10-19T12:57:58.258745+0000","last_unstale":"2023-10-19T12:57:59.266101+0000","last_undegraded":"2023-10-19T12:57:59.266101+0000","last_fullsized":"2023-10-19T12:57:59.266101+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:29:50.196136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:58.229465+0000","last_change":"2023-10-19T12:57:58.229465+0000","last_active":"2023-10-19T12:57:58.229465+0000","last_peered":"2023-10-19T12:57:58.229465+0000","last_clean":"2023-10-19T12:57:58.229465+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:58.229465+0000","last_undegraded":"2023-10-19T12:57:58.229465+0000","last_fullsized":"2023-10-19T12:57:58.229465+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:57.219060+0000","last_change":"2023-10-19T12:57:57.219060+0000","last_active":"2023-10-19T12:57:57.219060+0000","last_peered":"2023-10-19T12:57:57.219060+0000","last_clean":"2023-10-19T12:57:57.219060+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:57.219060+0000","last_undegraded":"2023-10-19T12:57:57.219060+0000","last_fullsized":"2023-10-19T12:57:57.219060+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:58.229465+0000","last_change":"2023-10-19T12:57:58.229465+0000","last_active":"2023-10-19T12:57:58.229465+0000","last_peered":"2023-10-19T12:57:58.229465+0000","last_clean":"2023-10-19T12:57:58.229465+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:58.229465+0000","last_undegraded":"2023-10-19T12:57:58.229465+0000","last_fullsized":"2023-10-19T12:57:58.229465+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:57.219060+0000","last_change":"2023-10-19T12:57:57.219060+0000","last_active":"2023-10-19T12:57:57.219060+0000","last_peered":"2023-10-19T12:57:57.219060+0000","last_clean":"2023-10-19T12:57:57.219060+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:57.219060+0000","last_undegraded":"2023-10-19T12:57:57.219060+0000","last_fullsized":"2023-10-19T12:57:57.219060+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.288065+0000","last_change":"2023-10-19T12:57:59.282205+0000","last_active":"2023-10-19T12:57:59.288065+0000","last_peered":"2023-10-19T12:57:59.288065+0000","last_clean":"2023-10-19T12:57:59.288065+0000","last_became_active":"2023-10-19T12:57:59.281923+0000","last_became_peered":"2023-10-19T12:57:59.281923+0000","last_unstale":"2023-10-19T12:57:59.288065+0000","last_undegraded":"2023-10-19T12:57:59.288065+0000","last_fullsized":"2023-10-19T12:57:59.288065+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:32:27.978530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:57.219060+0000","last_change":"2023-10-19T12:57:57.219060+0000","last_active":"2023-10-19T12:57:57.219060+0000","last_peered":"2023-10-19T12:57:57.219060+0000","last_clean":"2023-10-19T12:57:57.219060+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:57.219060+0000","last_undegraded":"2023-10-19T12:57:57.219060+0000","last_fullsized":"2023-10-19T12:57:57.219060+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:58.229465+0000","last_change":"2023-10-19T12:57:58.229465+0000","last_active":"2023-10-19T12:57:58.229465+0000","last_peered":"2023-10-19T12:57:58.229465+0000","last_clean":"2023-10-19T12:57:58.229465+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:58.229465+0000","last_undegraded":"2023-10-19T12:57:58.229465+0000","last_fullsized":"2023-10-19T12:57:58.229465+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:57.219060+0000","last_change":"2023-10-19T12:57:57.219060+0000","last_active":"2023-10-19T12:57:57.219060+0000","last_peered":"2023-10-19T12:57:57.219060+0000","last_clean":"2023-10-19T12:57:57.219060+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:57.219060+0000","last_undegraded":"2023-10-19T12:57:57.219060+0000","last_fullsized":"2023-10-19T12:57:57.219060+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:58.229465+0000","last_change":"2023-10-19T12:57:58.229465+0000","last_active":"2023-10-19T12:57:58.229465+0000","last_peered":"2023-10-19T12:57:58.229465+0000","last_clean":"2023-10-19T12:57:58.229465+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:58.229465+0000","last_undegraded":"2023-10-19T12:57:58.229465+0000","last_fullsized":"2023-10-19T12:57:58.229465+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.13","version":"69'2","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.290333+0000","last_change":"2023-10-19T12:57:58.257253+0000","last_active":"2023-10-19T12:57:59.290333+0000","last_peered":"2023-10-19T12:57:59.290333+0000","last_clean":"2023-10-19T12:57:59.290333+0000","last_became_active":"2023-10-19T12:57:58.256982+0000","last_became_peered":"2023-10-19T12:57:58.256982+0000","last_unstale":"2023-10-19T12:57:59.290333+0000","last_undegraded":"2023-10-19T12:57:59.290333+0000","last_fullsized":"2023-10-19T12:57:59.290333+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:26:46.604704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.288024+0000","last_change":"2023-10-19T12:57:59.282874+0000","last_active":"2023-10-19T12:57:59.288024+0000","last_peered":"2023-10-19T12:57:59.288024+0000","last_clean":"2023-10-19T12:57:59.288024+0000","last_became_active":"2023-10-19T12:57:59.282521+0000","last_became_peered":"2023-10-19T12:57:59.282521+0000","last_unstale":"2023-10-19T12:57:59.288024+0000","last_undegraded":"2023-10-19T12:57:59.288024+0000","last_fullsized":"2023-10-19T12:57:59.288024+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:42:43.566749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:57.219060+0000","last_change":"2023-10-19T12:57:57.219060+0000","last_active":"2023-10-19T12:57:57.219060+0000","last_peered":"2023-10-19T12:57:57.219060+0000","last_clean":"2023-10-19T12:57:57.219060+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:57.219060+0000","last_undegraded":"2023-10-19T12:57:57.219060+0000","last_fullsized":"2023-10-19T12:57:57.219060+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:58.229465+0000","last_change":"2023-10-19T12:57:58.229465+0000","last_active":"2023-10-19T12:57:58.229465+0000","last_peered":"2023-10-19T12:57:58.229465+0000","last_clean":"2023-10-19T12:57:58.229465+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:58.229465+0000","last_undegraded":"2023-10-19T12:57:58.229465+0000","last_fullsized":"2023-10-19T12:57:58.229465+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:57.219060+0000","last_change":"2023-10-19T12:57:57.219060+0000","last_active":"2023-10-19T12:57:57.219060+0000","last_peered":"2023-10-19T12:57:57.219060+0000","last_clean":"2023-10-19T12:57:57.219060+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:57.219060+0000","last_undegraded":"2023-10-19T12:57:57.219060+0000","last_fullsized":"2023-10-19T12:57:57.219060+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.287695+0000","last_change":"2023-10-19T12:57:59.278651+0000","last_active":"2023-10-19T12:57:59.287695+0000","last_peered":"2023-10-19T12:57:59.287695+0000","last_clean":"2023-10-19T12:57:59.287695+0000","last_became_active":"2023-10-19T12:57:59.278113+0000","last_became_peered":"2023-10-19T12:57:59.278113+0000","last_unstale":"2023-10-19T12:57:59.287695+0000","last_undegraded":"2023-10-19T12:57:59.287695+0000","last_fullsized":"2023-10-19T12:57:59.287695+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:12:20.974872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:57.219060+0000","last_change":"2023-10-19T12:57:57.219060+0000","last_active":"2023-10-19T12:57:57.219060+0000","last_peered":"2023-10-19T12:57:57.219060+0000","last_clean":"2023-10-19T12:57:57.219060+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:57.219060+0000","last_undegraded":"2023-10-19T12:57:57.219060+0000","last_fullsized":"2023-10-19T12:57:57.219060+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:58.229465+0000","last_change":"2023-10-19T12:57:58.229465+0000","last_active":"2023-10-19T12:57:58.229465+0000","last_peered":"2023-10-19T12:57:58.229465+0000","last_clean":"2023-10-19T12:57:58.229465+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:58.229465+0000","last_undegraded":"2023-10-19T12:57:58.229465+0000","last_fullsized":"2023-10-19T12:57:58.229465+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.f","version":"69'1","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.292498+0000","last_change":"2023-10-19T12:57:58.257724+0000","last_active":"2023-10-19T12:57:59.292498+0000","last_peered":"2023-10-19T12:57:59.292498+0000","last_clean":"2023-10-19T12:57:59.292498+0000","last_became_active":"2023-10-19T12:57:58.257432+0000","last_became_peered":"2023-10-19T12:57:58.257432+0000","last_unstale":"2023-10-19T12:57:59.292498+0000","last_undegraded":"2023-10-19T12:57:59.292498+0000","last_fullsized":"2023-10-19T12:57:59.292498+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:21:40.593175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.276683+0000","last_change":"2023-10-19T12:57:59.274011+0000","last_active":"2023-10-19T12:58:00.276683+0000","last_peered":"2023-10-19T12:58:00.276683+0000","last_clean":"2023-10-19T12:58:00.276683+0000","last_became_active":"2023-10-19T12:57:59.273742+0000","last_became_peered":"2023-10-19T12:57:59.273742+0000","last_unstale":"2023-10-19T12:58:00.276683+0000","last_undegraded":"2023-10-19T12:58:00.276683+0000","last_fullsized":"2023-10-19T12:58:00.276683+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:36:39.520498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"69'2","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.288309+0000","last_change":"2023-10-19T12:57:58.249377+0000","last_active":"2023-10-19T12:57:59.288309+0000","last_peered":"2023-10-19T12:57:59.288309+0000","last_clean":"2023-10-19T12:57:59.288309+0000","last_became_active":"2023-10-19T12:57:58.249095+0000","last_became_peered":"2023-10-19T12:57:58.249095+0000","last_unstale":"2023-10-19T12:57:59.288309+0000","last_undegraded":"2023-10-19T12:57:59.288309+0000","last_fullsized":"2023-10-19T12:57:59.288309+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:54:31.441800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:58.229465+0000","last_change":"2023-10-19T12:57:58.229465+0000","last_active":"2023-10-19T12:57:58.229465+0000","last_peered":"2023-10-19T12:57:58.229465+0000","last_clean":"2023-10-19T12:57:58.229465+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:58.229465+0000","last_undegraded":"2023-10-19T12:57:58.229465+0000","last_fullsized":"2023-10-19T12:57:58.229465+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:57.219060+0000","last_change":"2023-10-19T12:57:57.219060+0000","last_active":"2023-10-19T12:57:57.219060+0000","last_peered":"2023-10-19T12:57:57.219060+0000","last_clean":"2023-10-19T12:57:57.219060+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:57.219060+0000","last_undegraded":"2023-10-19T12:57:57.219060+0000","last_fullsized":"2023-10-19T12:57:57.219060+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.276881+0000","last_change":"2023-10-19T12:57:59.277839+0000","last_active":"2023-10-19T12:58:00.276881+0000","last_peered":"2023-10-19T12:58:00.276881+0000","last_clean":"2023-10-19T12:58:00.276881+0000","last_became_active":"2023-10-19T12:57:59.277539+0000","last_became_peered":"2023-10-19T12:57:59.277539+0000","last_unstale":"2023-10-19T12:58:00.276881+0000","last_undegraded":"2023-10-19T12:58:00.276881+0000","last_fullsized":"2023-10-19T12:58:00.276881+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:47:49.700629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:57.219060+0000","last_change":"2023-10-19T12:57:57.219060+0000","last_active":"2023-10-19T12:57:57.219060+0000","last_peered":"2023-10-19T12:57:57.219060+0000","last_clean":"2023-10-19T12:57:57.219060+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:57.219060+0000","last_undegraded":"2023-10-19T12:57:57.219060+0000","last_fullsized":"2023-10-19T12:57:57.219060+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:58.229465+0000","last_change":"2023-10-19T12:57:58.229465+0000","last_active":"2023-10-19T12:57:58.229465+0000","last_peered":"2023-10-19T12:57:58.229465+0000","last_clean":"2023-10-19T12:57:58.229465+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:58.229465+0000","last_undegraded":"2023-10-19T12:57:58.229465+0000","last_fullsized":"2023-10-19T12:57:58.229465+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.b","version":"69'2","reported_seq":14,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.276952+0000","last_change":"2023-10-19T12:57:58.254673+0000","last_active":"2023-10-19T12:58:00.276952+0000","last_peered":"2023-10-19T12:58:00.276952+0000","last_clean":"2023-10-19T12:58:00.276952+0000","last_became_active":"2023-10-19T12:57:58.254408+0000","last_became_peered":"2023-10-19T12:57:58.254408+0000","last_unstale":"2023-10-19T12:58:00.276952+0000","last_undegraded":"2023-10-19T12:58:00.276952+0000","last_fullsized":"2023-10-19T12:58:00.276952+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:25:49.650520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:58.229465+0000","last_change":"2023-10-19T12:57:58.229465+0000","last_active":"2023-10-19T12:57:58.229465+0000","last_peered":"2023-10-19T12:57:58.229465+0000","last_clean":"2023-10-19T12:57:58.229465+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:58.229465+0000","last_undegraded":"2023-10-19T12:57:58.229465+0000","last_fullsized":"2023-10-19T12:57:58.229465+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:57.219060+0000","last_change":"2023-10-19T12:57:57.219060+0000","last_active":"2023-10-19T12:57:57.219060+0000","last_peered":"2023-10-19T12:57:57.219060+0000","last_clean":"2023-10-19T12:57:57.219060+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:57.219060+0000","last_undegraded":"2023-10-19T12:57:57.219060+0000","last_fullsized":"2023-10-19T12:57:57.219060+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.289238+0000","last_change":"2023-10-19T12:57:59.275897+0000","last_active":"2023-10-19T12:57:59.289238+0000","last_peered":"2023-10-19T12:57:59.289238+0000","last_clean":"2023-10-19T12:57:59.289238+0000","last_became_active":"2023-10-19T12:57:59.274965+0000","last_became_peered":"2023-10-19T12:57:59.274965+0000","last_unstale":"2023-10-19T12:57:59.289238+0000","last_undegraded":"2023-10-19T12:57:59.289238+0000","last_fullsized":"2023-10-19T12:57:59.289238+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:30:55.579208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":11,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.267824+0000","last_change":"2023-10-19T12:57:58.255960+0000","last_active":"2023-10-19T12:57:59.267824+0000","last_peered":"2023-10-19T12:57:59.267824+0000","last_clean":"2023-10-19T12:57:59.267824+0000","last_became_active":"2023-10-19T12:57:58.255655+0000","last_became_peered":"2023-10-19T12:57:58.255655+0000","last_unstale":"2023-10-19T12:57:59.267824+0000","last_undegraded":"2023-10-19T12:57:59.267824+0000","last_fullsized":"2023-10-19T12:57:59.267824+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:35:32.478337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.276919+0000","last_change":"2023-10-19T12:57:59.284564+0000","last_active":"2023-10-19T12:58:00.276919+0000","last_peered":"2023-10-19T12:58:00.276919+0000","last_clean":"2023-10-19T12:58:00.276919+0000","last_became_active":"2023-10-19T12:57:59.278312+0000","last_became_peered":"2023-10-19T12:57:59.278312+0000","last_unstale":"2023-10-19T12:58:00.276919+0000","last_undegraded":"2023-10-19T12:58:00.276919+0000","last_fullsized":"2023-10-19T12:58:00.276919+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:19:34.001505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":11,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.255502+0000","last_change":"2023-10-19T12:57:58.248153+0000","last_active":"2023-10-19T12:57:59.255502+0000","last_peered":"2023-10-19T12:57:59.255502+0000","last_clean":"2023-10-19T12:57:59.255502+0000","last_became_active":"2023-10-19T12:57:58.247864+0000","last_became_peered":"2023-10-19T12:57:58.247864+0000","last_unstale":"2023-10-19T12:57:59.255502+0000","last_undegraded":"2023-10-19T12:57:59.255502+0000","last_fullsized":"2023-10-19T12:57:59.255502+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:53:21.688950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:58.229465+0000","last_change":"2023-10-19T12:57:58.229465+0000","last_active":"2023-10-19T12:57:58.229465+0000","last_peered":"2023-10-19T12:57:58.229465+0000","last_clean":"2023-10-19T12:57:58.229465+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:58.229465+0000","last_undegraded":"2023-10-19T12:57:58.229465+0000","last_fullsized":"2023-10-19T12:57:58.229465+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:57.219060+0000","last_change":"2023-10-19T12:57:57.219060+0000","last_active":"2023-10-19T12:57:57.219060+0000","last_peered":"2023-10-19T12:57:57.219060+0000","last_clean":"2023-10-19T12:57:57.219060+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:57.219060+0000","last_undegraded":"2023-10-19T12:57:57.219060+0000","last_fullsized":"2023-10-19T12:57:57.219060+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.276817+0000","last_change":"2023-10-19T12:57:59.271932+0000","last_active":"2023-10-19T12:58:00.276817+0000","last_peered":"2023-10-19T12:58:00.276817+0000","last_clean":"2023-10-19T12:58:00.276817+0000","last_became_active":"2023-10-19T12:57:59.271620+0000","last_became_peered":"2023-10-19T12:57:59.271620+0000","last_unstale":"2023-10-19T12:58:00.276817+0000","last_undegraded":"2023-10-19T12:58:00.276817+0000","last_fullsized":"2023-10-19T12:58:00.276817+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:13:06.950010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:57.219060+0000","last_change":"2023-10-19T12:57:57.219060+0000","last_active":"2023-10-19T12:57:57.219060+0000","last_peered":"2023-10-19T12:57:57.219060+0000","last_clean":"2023-10-19T12:57:57.219060+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:57.219060+0000","last_undegraded":"2023-10-19T12:57:57.219060+0000","last_fullsized":"2023-10-19T12:57:57.219060+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:58.229465+0000","last_change":"2023-10-19T12:57:58.229465+0000","last_active":"2023-10-19T12:57:58.229465+0000","last_peered":"2023-10-19T12:57:58.229465+0000","last_clean":"2023-10-19T12:57:58.229465+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:58.229465+0000","last_undegraded":"2023-10-19T12:57:58.229465+0000","last_fullsized":"2023-10-19T12:57:58.229465+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":11,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.265565+0000","last_change":"2023-10-19T12:57:58.253449+0000","last_active":"2023-10-19T12:57:59.265565+0000","last_peered":"2023-10-19T12:57:59.265565+0000","last_clean":"2023-10-19T12:57:59.265565+0000","last_became_active":"2023-10-19T12:57:58.253169+0000","last_became_peered":"2023-10-19T12:57:58.253169+0000","last_unstale":"2023-10-19T12:57:59.265565+0000","last_undegraded":"2023-10-19T12:57:59.265565+0000","last_fullsized":"2023-10-19T12:57:59.265565+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:55:53.005401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.287287+0000","last_change":"2023-10-19T12:57:59.280611+0000","last_active":"2023-10-19T12:57:59.287287+0000","last_peered":"2023-10-19T12:57:59.287287+0000","last_clean":"2023-10-19T12:57:59.287287+0000","last_became_active":"2023-10-19T12:57:59.280321+0000","last_became_peered":"2023-10-19T12:57:59.280321+0000","last_unstale":"2023-10-19T12:57:59.287287+0000","last_undegraded":"2023-10-19T12:57:59.287287+0000","last_fullsized":"2023-10-19T12:57:59.287287+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:13:46.024291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:57.219060+0000","last_change":"2023-10-19T12:57:57.219060+0000","last_active":"2023-10-19T12:57:57.219060+0000","last_peered":"2023-10-19T12:57:57.219060+0000","last_clean":"2023-10-19T12:57:57.219060+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:57.219060+0000","last_undegraded":"2023-10-19T12:57:57.219060+0000","last_fullsized":"2023-10-19T12:57:57.219060+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:58.229465+0000","last_change":"2023-10-19T12:57:58.229465+0000","last_active":"2023-10-19T12:57:58.229465+0000","last_peered":"2023-10-19T12:57:58.229465+0000","last_clean":"2023-10-19T12:57:58.229465+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:58.229465+0000","last_undegraded":"2023-10-19T12:57:58.229465+0000","last_fullsized":"2023-10-19T12:57:58.229465+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:57.219060+0000","last_change":"2023-10-19T12:57:57.219060+0000","last_active":"2023-10-19T12:57:57.219060+0000","last_peered":"2023-10-19T12:57:57.219060+0000","last_clean":"2023-10-19T12:57:57.219060+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:57.219060+0000","last_undegraded":"2023-10-19T12:57:57.219060+0000","last_fullsized":"2023-10-19T12:57:57.219060+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.289255+0000","last_change":"2023-10-19T12:57:59.279293+0000","last_active":"2023-10-19T12:57:59.289255+0000","last_peered":"2023-10-19T12:57:59.289255+0000","last_clean":"2023-10-19T12:57:59.289255+0000","last_became_active":"2023-10-19T12:57:59.278399+0000","last_became_peered":"2023-10-19T12:57:59.278399+0000","last_unstale":"2023-10-19T12:57:59.289255+0000","last_undegraded":"2023-10-19T12:57:59.289255+0000","last_fullsized":"2023-10-19T12:57:59.289255+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:18:02.353710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:57.219060+0000","last_change":"2023-10-19T12:57:57.219060+0000","last_active":"2023-10-19T12:57:57.219060+0000","last_peered":"2023-10-19T12:57:57.219060+0000","last_clean":"2023-10-19T12:57:57.219060+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:57.219060+0000","last_undegraded":"2023-10-19T12:57:57.219060+0000","last_fullsized":"2023-10-19T12:57:57.219060+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:58.229465+0000","last_change":"2023-10-19T12:57:58.229465+0000","last_active":"2023-10-19T12:57:58.229465+0000","last_peered":"2023-10-19T12:57:58.229465+0000","last_clean":"2023-10-19T12:57:58.229465+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:58.229465+0000","last_undegraded":"2023-10-19T12:57:58.229465+0000","last_fullsized":"2023-10-19T12:57:58.229465+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:57.219060+0000","last_change":"2023-10-19T12:57:57.219060+0000","last_active":"2023-10-19T12:57:57.219060+0000","last_peered":"2023-10-19T12:57:57.219060+0000","last_clean":"2023-10-19T12:57:57.219060+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:57.219060+0000","last_undegraded":"2023-10-19T12:57:57.219060+0000","last_fullsized":"2023-10-19T12:57:57.219060+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T12:57:58.229465+0000","last_change":"2023-10-19T12:57:58.229465+0000","last_active":"2023-10-19T12:57:58.229465+0000","last_peered":"2023-10-19T12:57:58.229465+0000","last_clean":"2023-10-19T12:57:58.229465+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T12:57:58.229465+0000","last_undegraded":"2023-10-19T12:57:58.229465+0000","last_fullsized":"2023-10-19T12:57:58.229465+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":11,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.264710+0000","last_change":"2023-10-19T12:57:58.254016+0000","last_active":"2023-10-19T12:57:59.264710+0000","last_peered":"2023-10-19T12:57:59.264710+0000","last_clean":"2023-10-19T12:57:59.264710+0000","last_became_active":"2023-10-19T12:57:58.253756+0000","last_became_peered":"2023-10-19T12:57:58.253756+0000","last_unstale":"2023-10-19T12:57:59.264710+0000","last_undegraded":"2023-10-19T12:57:59.264710+0000","last_fullsized":"2023-10-19T12:57:59.264710+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:52:42.753982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"58'139","reported_seq":286,"reported_epoch":61,"state":"active+clean","last_fresh":"2023-10-19T12:57:49.219229+0000","last_change":"2023-10-19T12:44:08.636613+0000","last_active":"2023-10-19T12:57:49.219229+0000","last_peered":"2023-10-19T12:57:49.219229+0000","last_clean":"2023-10-19T12:57:49.219229+0000","last_became_active":"2023-10-19T12:44:08.635697+0000","last_became_peered":"2023-10-19T12:44:08.635697+0000","last_unstale":"2023-10-19T12:57:49.219229+0000","last_undegraded":"2023-10-19T12:57:49.219229+0000","last_fullsized":"2023-10-19T12:57:49.219229+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_clean_scrub_stamp":"2023-10-19T12:44:07.399738+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:15:23.685485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":54,"acting":54,"num_store_stats":3},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":56,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":24,"acting":24,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575017,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58856,"kb_used_data":1304,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312984,"statfs":{"total":96636764160,"available":96576495616,"internally_reserved":0,"allocated":1335296,"data_stored":957245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58915275},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:57:38 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.58499999999999996,"15min":0.56100000000000005},"min":{"1min":0.39300000000000002,"5min":0.34499999999999997,"15min":0.248},"max":{"1min":0.67600000000000005,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.59699999999999998,"15min":0.57799999999999996},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.26900000000000002},"max":{"1min":0.76100000000000001,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.59599999999999997}]},{"osd":1,"last update":"Thu Oct 19 12:57:38 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.58699999999999997,"15min":0.56899999999999995},"min":{"1min":0.40300000000000002,"5min":0.371,"15min":0.28199999999999997},"max":{"1min":0.747,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.497},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.59699999999999998,"15min":0.57299999999999995},"min":{"1min":0.46200000000000002,"5min":0.34999999999999998,"15min":0.29299999999999998},"max":{"1min":0.66700000000000004,"5min":1.284,"15min":1.284},"last":0.51100000000000001}]},{"osd":2,"last update":"Thu Oct 19 12:57:38 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.63900000000000001,"15min":0.61799999999999999},"min":{"1min":0.434,"5min":0.41699999999999998,"15min":0.375},"max":{"1min":0.78600000000000003,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.65500000000000003,"15min":0.629},"min":{"1min":0.497,"5min":0.44600000000000001,"15min":0.28399999999999997},"max":{"1min":0.80700000000000005,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.628}]},{"osd":3,"last update":"Thu Oct 19 12:57:38 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.61899999999999999,"15min":0.59499999999999997},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34399999999999997},"max":{"1min":0.79100000000000004,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.66000000000000003,"15min":0.64700000000000002},"min":{"1min":0.45300000000000001,"5min":0.44400000000000001,"15min":0.39700000000000002},"max":{"1min":0.77900000000000003,"5min":1.329,"15min":1.329},"last":0.56499999999999995}]},{"osd":4,"last update":"Thu Oct 19 12:57:38 2023","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.46899999999999997,"15min":0.46600000000000003},"min":{"1min":0.28599999999999998,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.73399999999999999,"5min":0.75800000000000001,"15min":0.80500000000000005},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.46800000000000003,"15min":0.46999999999999997},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.627,"5min":0.71999999999999997,"15min":0.78900000000000003},"last":0.42399999999999999}]},{"osd":5,"last update":"Thu Oct 19 12:57:38 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.48199999999999998,"15min":0.48799999999999999},"min":{"1min":0.30499999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.75800000000000001,"5min":0.80600000000000005,"15min":0.90400000000000003},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.52300000000000002,"15min":0.51400000000000001},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.308},"max":{"1min":0.77400000000000002,"5min":0.81999999999999995,"15min":0.92200000000000004},"last":0.54300000000000004}]},{"osd":6,"last update":"Thu Oct 19 12:57:38 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54500000000000004,"15min":0.54400000000000004},"min":{"1min":0.34100000000000003,"5min":0.308,"15min":0.308},"max":{"1min":0.80400000000000005,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.53400000000000003,"15min":0.53200000000000003},"min":{"1min":0.30299999999999999,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.72599999999999998,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.58199999999999996}]}]},{"osd":6,"up_from":13,"seq":55834575017,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31916,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339924,"statfs":{"total":96636764160,"available":96604082176,"internally_reserved":0,"allocated":753664,"data_stored":372985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:57:21 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60299999999999998,"15min":0.58999999999999997},"min":{"1min":0.311,"5min":0.311,"15min":0.24399999999999999},"max":{"1min":0.76000000000000001,"5min":0.79900000000000004,"15min":0.81999999999999995},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.60499999999999998,"15min":0.58399999999999996},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.77500000000000002,"5min":0.78000000000000003,"15min":0.81200000000000006},"last":0.47999999999999998}]},{"osd":1,"last update":"Thu Oct 19 12:57:21 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.626,"15min":0.60399999999999998},"min":{"1min":0.376,"5min":0.376,"15min":0.34799999999999998},"max":{"1min":0.80500000000000005,"5min":0.83299999999999996,"15min":0.93899999999999995},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.62,"15min":0.60599999999999998},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.28399999999999997},"max":{"1min":0.79400000000000004,"5min":0.82099999999999995,"15min":1.075},"last":0.54000000000000004}]},{"osd":2,"last update":"Thu Oct 19 12:57:21 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60799999999999998,"15min":0.58299999999999996},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.26600000000000001},"max":{"1min":0.748,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.501},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.63200000000000001,"15min":0.60599999999999998},"min":{"1min":0.40500000000000003,"5min":0.36099999999999999,"15min":0.28399999999999997},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.59499999999999997}]},{"osd":3,"last update":"Thu Oct 19 12:57:21 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.65100000000000002,"15min":0.625},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.32600000000000001},"max":{"1min":0.83099999999999996,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.64700000000000002,"15min":0.63500000000000001},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.81499999999999995,"5min":0.81699999999999995,"15min":1.0900000000000001},"last":0.61699999999999999}]},{"osd":4,"last update":"Thu Oct 19 12:57:21 2023","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.46300000000000002,"15min":0.47599999999999998},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":1.02},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.47799999999999998,"15min":0.49099999999999999},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.247},"max":{"1min":0.65400000000000003,"5min":0.752,"15min":0.80000000000000004},"last":0.60699999999999998}]},{"osd":5,"last update":"Thu Oct 19 12:57:21 2023","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.47899999999999998,"15min":0.48399999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.26600000000000001},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.505,"5min":0.51400000000000001,"15min":0.52900000000000003},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.312},"max":{"1min":0.78500000000000003,"5min":0.85699999999999998,"15min":0.98799999999999999},"last":0.67100000000000004}]},{"osd":7,"last update":"Thu Oct 19 12:57:21 2023","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.52600000000000002,"15min":0.53100000000000003},"min":{"1min":0.35699999999999998,"5min":0.314,"15min":0.28199999999999997},"max":{"1min":0.76100000000000001,"5min":0.82199999999999995,"15min":1.036},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.53900000000000003,"15min":0.55200000000000005},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":1.1299999999999999},"last":0.64300000000000002}]}]},{"osd":5,"up_from":13,"seq":55834575017,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27744,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344096,"statfs":{"total":96636764160,"available":96608354304,"internally_reserved":0,"allocated":741376,"data_stored":366877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:57:24 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.59199999999999997,"15min":0.58599999999999997},"min":{"1min":0.39800000000000002,"5min":0.26700000000000002,"15min":0.223},"max":{"1min":0.79400000000000004,"5min":0.82699999999999996,"15min":0.88800000000000001},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.60799999999999998,"15min":0.59099999999999997},"min":{"1min":0.39700000000000002,"5min":0.312,"15min":0.27800000000000002},"max":{"1min":0.72699999999999998,"5min":0.80400000000000005,"15min":0.874},"last":0.50800000000000001}]},{"osd":1,"last update":"Thu Oct 19 12:57:24 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.60499999999999998,"15min":0.59599999999999997},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.32800000000000001},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.876},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.61299999999999999,"15min":0.59999999999999998},"min":{"1min":0.439,"5min":0.34699999999999998,"15min":0.34499999999999997},"max":{"1min":0.79500000000000004,"5min":0.80100000000000005,"15min":0.83499999999999996},"last":0.72699999999999998}]},{"osd":2,"last update":"Thu Oct 19 12:57:24 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59399999999999997,"15min":0.57899999999999996},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.307},"max":{"1min":0.70899999999999996,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.62,"15min":0.61399999999999999},"min":{"1min":0.47399999999999998,"5min":0.41999999999999998,"15min":0.33700000000000002},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.93600000000000005},"last":0.55200000000000005}]},{"osd":3,"last update":"Thu Oct 19 12:57:24 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.60999999999999999,"15min":0.59999999999999998},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.39300000000000002},"max":{"1min":0.73199999999999998,"5min":0.77900000000000003,"15min":0.874},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63900000000000001,"15min":0.63400000000000001},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.41999999999999998},"max":{"1min":0.746,"5min":0.80400000000000005,"15min":0.89500000000000002},"last":0.74299999999999999}]},{"osd":4,"last update":"Thu Oct 19 12:57:24 2023","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.34499999999999997,"5min":0.32100000000000001,"15min":0.28899999999999998},"max":{"1min":0.66100000000000003,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.47099999999999997,"15min":0.47599999999999998},"min":{"1min":0.307,"5min":0.307,"15min":0.23899999999999999},"max":{"1min":0.75900000000000001,"5min":0.79400000000000004,"15min":0.80500000000000005},"last":0.60199999999999998}]},{"osd":6,"last update":"Thu Oct 19 12:57:24 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.47399999999999998,"15min":0.47899999999999998},"min":{"1min":0.34100000000000003,"5min":0.31900000000000001,"15min":0.25900000000000001},"max":{"1min":0.74399999999999999,"5min":0.77500000000000002,"15min":0.79300000000000004},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.50800000000000001,"15min":0.498},"min":{"1min":0.38500000000000001,"5min":0.33600000000000002,"15min":0.30399999999999999},"max":{"1min":0.71899999999999997,"5min":0.79100000000000004,"15min":0.81299999999999994},"last":0.71199999999999997}]},{"osd":7,"last update":"Thu Oct 19 12:57:24 2023","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.47999999999999998,"15min":0.49399999999999999},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.315},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71899999999999997},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.52700000000000002,"15min":0.53400000000000003},"min":{"1min":0.39700000000000002,"5min":0.34300000000000003,"15min":0.27500000000000002},"max":{"1min":0.77900000000000003,"5min":0.78100000000000003,"15min":0.78500000000000003},"last":0.65600000000000003}]}]},{"osd":4,"up_from":13,"seq":55834575017,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31904,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339936,"statfs":{"total":96636764160,"available":96604094464,"internally_reserved":0,"allocated":741376,"data_stored":366877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:57:37 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.54900000000000004,"15min":0.55100000000000005},"min":{"1min":0.36499999999999999,"5min":0.32900000000000001,"15min":0.28899999999999998},"max":{"1min":0.72699999999999998,"5min":0.77400000000000002,"15min":0.84799999999999998},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55900000000000005,"15min":0.54800000000000004},"min":{"1min":0.38100000000000001,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.73999999999999999,"5min":0.77800000000000002,"15min":0.78200000000000003},"last":0.48499999999999999}]},{"osd":1,"last update":"Thu Oct 19 12:57:37 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.61099999999999999,"15min":0.60399999999999998},"min":{"1min":0.377,"5min":0.377,"15min":0.34799999999999998},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.86199999999999999},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.61799999999999999,"15min":0.61099999999999999},"min":{"1min":0.44800000000000001,"5min":0.40200000000000002,"15min":0.35799999999999998},"max":{"1min":0.78900000000000003,"5min":0.79500000000000004,"15min":0.873},"last":0.66400000000000003}]},{"osd":2,"last update":"Thu Oct 19 12:57:37 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.61599999999999999,"15min":0.60899999999999999},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.254},"max":{"1min":0.76000000000000001,"5min":0.85299999999999998,"15min":0.86599999999999999},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.60899999999999999,"15min":0.61699999999999999},"min":{"1min":0.44800000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.75,"5min":0.82999999999999996,"15min":0.85999999999999999},"last":0.74299999999999999}]},{"osd":3,"last update":"Thu Oct 19 12:57:37 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.58499999999999996,"15min":0.57499999999999996},"min":{"1min":0.35199999999999998,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.70199999999999996,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.64400000000000002,"15min":0.63400000000000001},"min":{"1min":0.45700000000000002,"5min":0.41299999999999998,"15min":0.37},"max":{"1min":0.77300000000000002,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.63600000000000001}]},{"osd":5,"last update":"Thu Oct 19 12:57:37 2023","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.437,"15min":0.45200000000000001},"min":{"1min":0.30599999999999999,"5min":0.29099999999999998,"15min":0.249},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.81299999999999994},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.45500000000000002,"15min":0.46400000000000002},"min":{"1min":0.32000000000000001,"5min":0.26400000000000001,"15min":0.23499999999999999},"max":{"1min":0.71999999999999997,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.70499999999999996}]},{"osd":6,"last update":"Thu Oct 19 12:57:37 2023","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45400000000000001,"15min":0.46000000000000002},"min":{"1min":0.32300000000000001,"5min":0.29499999999999998,"15min":0.26400000000000001},"max":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.78600000000000003},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.47899999999999998,"15min":0.505},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.27100000000000002},"max":{"1min":0.76800000000000002,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.77100000000000002}]},{"osd":7,"last update":"Thu Oct 19 12:57:37 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.49199999999999999,"15min":0.50900000000000001},"min":{"1min":0.38600000000000001,"5min":0.307,"15min":0.27500000000000002},"max":{"1min":0.71399999999999997,"5min":0.79100000000000004,"15min":0.80200000000000005},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.53900000000000003,"15min":0.55900000000000005},"min":{"1min":0.41799999999999998,"5min":0.32100000000000001,"15min":0.30199999999999999},"max":{"1min":0.745,"5min":0.80500000000000005,"15min":0.83499999999999996},"last":0.68600000000000005}]}]},{"osd":3,"up_from":13,"seq":55834575016,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":36068,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335772,"statfs":{"total":96636764160,"available":96599830528,"internally_reserved":0,"allocated":741376,"data_stored":366877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:57:22 2023","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.40300000000000002,"15min":0.39500000000000002},"min":{"1min":0.248,"5min":0.23599999999999999,"15min":0.20899999999999999},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":1.04},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.42299999999999999,"15min":0.40699999999999997},"min":{"1min":0.28399999999999997,"5min":0.248,"15min":0.245},"max":{"1min":0.57399999999999995,"5min":0.60999999999999999,"15min":0.64600000000000002},"last":0.26200000000000001}]},{"osd":1,"last update":"Thu Oct 19 12:57:22 2023","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.41599999999999998,"15min":0.40400000000000003},"min":{"1min":0.30599999999999999,"5min":0.26100000000000001,"15min":0.22500000000000001},"max":{"1min":0.58299999999999996,"5min":0.621,"15min":0.69899999999999995},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.46100000000000002,"15min":0.442},"min":{"1min":0.32000000000000001,"5min":0.29899999999999999,"15min":0.215},"max":{"1min":0.65200000000000002,"5min":0.65900000000000003,"15min":0.874},"last":0.312}]},{"osd":2,"last update":"Thu Oct 19 12:57:22 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.46899999999999997,"15min":0.45900000000000002},"min":{"1min":0.34699999999999998,"5min":0.32500000000000001,"15min":0.26100000000000001},"max":{"1min":0.59399999999999997,"5min":0.65800000000000003,"15min":0.73599999999999999},"last":0.375},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.498,"15min":0.48299999999999998},"min":{"1min":0.36299999999999999,"5min":0.34799999999999998,"15min":0.247},"max":{"1min":0.64000000000000001,"5min":0.72599999999999998,"15min":0.96499999999999997},"last":0.38500000000000001}]},{"osd":4,"last update":"Thu Oct 19 12:57:22 2023","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.67700000000000005,"15min":0.65900000000000003},"min":{"1min":0.49399999999999999,"5min":0.45900000000000002,"15min":0.40300000000000002},"max":{"1min":0.90500000000000003,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.70199999999999996,"15min":0.68700000000000006},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.42099999999999999},"max":{"1min":0.91300000000000003,"5min":0.98999999999999999,"15min":0.99399999999999999},"last":0.63100000000000001}]},{"osd":5,"last update":"Thu Oct 19 12:57:22 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.66900000000000004,"15min":0.64900000000000002},"min":{"1min":0.495,"5min":0.495,"15min":0.40500000000000003},"max":{"1min":0.93899999999999995,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.65700000000000003,"15min":0.64500000000000002},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.35899999999999999},"max":{"1min":0.85899999999999999,"5min":0.97299999999999998,"15min":0.99099999999999999},"last":0.57699999999999996}]},{"osd":6,"last update":"Thu Oct 19 12:57:22 2023","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.66500000000000004,"15min":0.65400000000000003},"min":{"1min":0.439,"5min":0.439,"15min":0.41199999999999998},"max":{"1min":0.84699999999999998,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.68100000000000005,"15min":0.66700000000000004},"min":{"1min":0.55200000000000005,"5min":0.496,"15min":0.35899999999999999},"max":{"1min":0.86899999999999999,"5min":1.05,"15min":1.05},"last":0.621}]},{"osd":7,"last update":"Thu Oct 19 12:57:22 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.67400000000000004,"15min":0.66500000000000004},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.39800000000000002},"max":{"1min":0.95699999999999996,"5min":1,"15min":1},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.68400000000000005,"15min":0.67100000000000004},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.88500000000000001,"5min":0.93400000000000005,"15min":0.97599999999999998},"last":0.65700000000000003}]}]},{"osd":2,"up_from":13,"seq":55834575019,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54216,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317624,"statfs":{"total":96636764160,"available":96581246976,"internally_reserved":0,"allocated":839680,"data_stored":411150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:57:25 2023","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.38900000000000001,"15min":0.38500000000000001},"min":{"1min":0.254,"5min":0.20899999999999999,"15min":0.187},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.83999999999999997},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.376,"5min":0.41699999999999998,"15min":0.40100000000000002},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.57799999999999996,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.36199999999999999}]},{"osd":1,"last update":"Thu Oct 19 12:57:25 2023","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.40799999999999997,"15min":0.40000000000000002},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.57899999999999996,"5min":0.71099999999999997,"15min":0.80400000000000005},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.44,"15min":0.43099999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.23400000000000001},"max":{"1min":0.57599999999999996,"5min":0.63,"15min":0.78600000000000003},"last":0.32000000000000001}]},{"osd":3,"last update":"Thu Oct 19 12:57:25 2023","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.46700000000000003,"15min":0.45600000000000002},"min":{"1min":0.25900000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.623,"5min":0.68000000000000005,"15min":0.79600000000000004},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.497,"15min":0.48799999999999999},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.87},"last":0.375}]},{"osd":4,"last update":"Thu Oct 19 12:57:25 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.64800000000000002,"15min":0.64100000000000001},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":1.0569999999999999},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.66400000000000003,"15min":0.65500000000000003},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.32000000000000001},"max":{"1min":0.91200000000000003,"5min":0.92300000000000004,"15min":1.0780000000000001},"last":0.45400000000000001}]},{"osd":5,"last update":"Thu Oct 19 12:57:25 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.68100000000000005,"15min":0.66000000000000003},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.78600000000000003,"5min":0.96499999999999997,"15min":0.99399999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.67500000000000004,"15min":0.65700000000000003},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.90200000000000002,"5min":0.94599999999999995,"15min":0.97099999999999997},"last":0.46999999999999997}]},{"osd":6,"last update":"Thu Oct 19 12:57:25 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.67900000000000005,"15min":0.67200000000000004},"min":{"1min":0.441,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.86799999999999999,"5min":1.008,"15min":1.008},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.68200000000000005,"15min":0.66900000000000004},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.93899999999999995,"5min":0.995,"15min":0.995},"last":0.623}]},{"osd":7,"last update":"Thu Oct 19 12:57:25 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.69499999999999995,"15min":0.68400000000000005},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.36399999999999999},"max":{"1min":0.98999999999999999,"5min":1.0149999999999999,"15min":1.111},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.69799999999999995,"15min":0.68500000000000005},"min":{"1min":0.42099999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.89700000000000002,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.60599999999999998}]}]},{"osd":1,"up_from":13,"seq":55834575018,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32004,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339836,"statfs":{"total":96636764160,"available":96603992064,"internally_reserved":0,"allocated":835584,"data_stored":409698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:57:32 2023","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.39200000000000002,"15min":0.378},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.187},"max":{"1min":0.59099999999999997,"5min":0.61799999999999999,"15min":0.66200000000000003},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.443,"15min":0.42199999999999999},"min":{"1min":0.30499999999999999,"5min":0.28000000000000003,"15min":0.21199999999999999},"max":{"1min":0.63600000000000001,"5min":0.68799999999999994,"15min":0.69799999999999995},"last":0.33900000000000002}]},{"osd":2,"last update":"Thu Oct 19 12:57:32 2023","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.44600000000000001,"15min":0.42099999999999999},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.20100000000000001},"max":{"1min":0.56000000000000005,"5min":0.68999999999999995,"15min":0.69999999999999996},"last":0.374},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.45800000000000002,"15min":0.44400000000000001},"min":{"1min":0.34699999999999998,"5min":0.309,"15min":0.23699999999999999},"max":{"1min":0.60599999999999998,"5min":0.67500000000000004,"15min":0.755},"last":0.41299999999999998}]},{"osd":3,"last update":"Thu Oct 19 12:57:32 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.49099999999999999,"15min":0.47699999999999998},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.26200000000000001},"max":{"1min":0.63500000000000001,"5min":0.65900000000000003,"15min":0.83799999999999997},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.435,"15min":0.42499999999999999},"min":{"1min":0.29499999999999998,"5min":0.28499999999999998,"15min":0.27000000000000002},"max":{"1min":0.56499999999999995,"5min":0.70699999999999996,"15min":0.80500000000000005},"last":0.35399999999999998}]},{"osd":4,"last update":"Thu Oct 19 12:57:32 2023","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.67500000000000004,"15min":0.65700000000000003},"min":{"1min":0.44,"5min":0.44,"15min":0.39300000000000002},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.622},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.66600000000000004,"15min":0.65200000000000002},"min":{"1min":0.495,"5min":0.42599999999999999,"15min":0.314},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":1.1910000000000001},"last":0.68799999999999994}]},{"osd":5,"last update":"Thu Oct 19 12:57:32 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.66700000000000004,"15min":0.65700000000000003},"min":{"1min":0.52300000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.878,"5min":0.878,"15min":0.91800000000000004},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.69099999999999995,"15min":0.67200000000000004},"min":{"1min":0.53700000000000003,"5min":0.47099999999999997,"15min":0.38700000000000001},"max":{"1min":0.88500000000000001,"5min":0.93999999999999995,"15min":1.117},"last":0.58499999999999996}]},{"osd":6,"last update":"Thu Oct 19 12:57:32 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.67700000000000005,"15min":0.66200000000000003},"min":{"1min":0.48399999999999999,"5min":0.36799999999999999,"15min":0.36399999999999999},"max":{"1min":0.92200000000000004,"5min":1.0069999999999999,"15min":1.077},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.69399999999999995,"15min":0.68500000000000005},"min":{"1min":0.47399999999999998,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.995,"5min":0.995,"15min":1.0009999999999999},"last":0.64900000000000002}]},{"osd":7,"last update":"Thu Oct 19 12:57:32 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.69499999999999995,"15min":0.68100000000000005},"min":{"1min":0.54000000000000004,"5min":0.46999999999999997,"15min":0.443},"max":{"1min":0.84499999999999997,"5min":0.97399999999999998,"15min":1.0149999999999999},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.70099999999999996,"15min":0.68300000000000005},"min":{"1min":0.46500000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":1.0489999999999999},"last":0.64300000000000002}]}]},{"osd":0,"up_from":13,"seq":55834575019,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36732,"kb_used_data":1392,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335108,"statfs":{"total":96636764160,"available":96599150592,"internally_reserved":0,"allocated":1425408,"data_stored":999343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Oct 19 12:57:36 2023","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.40100000000000002,"15min":0.38500000000000001},"min":{"1min":0.28699999999999998,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.442,"15min":0.42399999999999999},"min":{"1min":0.25,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.623,"5min":0.68700000000000006,"15min":0.746},"last":0.32500000000000001}]},{"osd":2,"last update":"Thu Oct 19 12:57:36 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.46700000000000003,"15min":0.45000000000000001},"min":{"1min":0.33900000000000002,"5min":0.26200000000000001,"15min":0.25800000000000001},"max":{"1min":0.752,"5min":0.752,"15min":0.76300000000000001},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.45100000000000001,"15min":0.42799999999999999},"min":{"1min":0.27500000000000002,"5min":0.216,"15min":0.20699999999999999},"max":{"1min":0.59399999999999997,"5min":0.68600000000000005,"15min":0.76800000000000002},"last":0.29299999999999998}]},{"osd":3,"last update":"Thu Oct 19 12:57:36 2023","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.42299999999999999,"15min":0.40100000000000002},"min":{"1min":0.24299999999999999,"5min":0.23400000000000001,"15min":0.22800000000000001},"max":{"1min":1.2829999999999999,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.38},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.51000000000000001,"15min":0.48799999999999999},"min":{"1min":0.33300000000000002,"5min":0.29399999999999998,"15min":0.29199999999999998},"max":{"1min":1.4990000000000001,"5min":1.4990000000000001,"15min":1.4990000000000001},"last":0.375}]},{"osd":4,"last update":"Thu Oct 19 12:57:36 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.65000000000000002,"15min":0.63},"min":{"1min":0.46700000000000003,"5min":0.45100000000000001,"15min":0.34999999999999998},"max":{"1min":1.361,"5min":1.361,"15min":1.361},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.65800000000000003,"15min":0.64100000000000001},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.35999999999999999},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.998},"last":0.55300000000000005}]},{"osd":5,"last update":"Thu Oct 19 12:57:36 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.69099999999999995,"15min":0.66800000000000004},"min":{"1min":0.44800000000000001,"5min":0.42099999999999999,"15min":0.38},"max":{"1min":1.325,"5min":1.325,"15min":1.325},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.69699999999999995,"15min":0.67800000000000005},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.318},"max":{"1min":0.88400000000000001,"5min":0.98699999999999999,"15min":1.056},"last":0.51500000000000001}]},{"osd":6,"last update":"Thu Oct 19 12:57:36 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.67000000000000004,"15min":0.65600000000000003},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.38500000000000001},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.438},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.68200000000000005,"15min":0.65800000000000003},"min":{"1min":0.45500000000000002,"5min":0.41999999999999998,"15min":0.38900000000000001},"max":{"1min":0.995,"5min":0.995,"15min":1.109},"last":0.47599999999999998}]},{"osd":7,"last update":"Thu Oct 19 12:57:36 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.69199999999999995,"15min":0.67700000000000005},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.41099999999999998},"max":{"1min":0.92000000000000004,"5min":0.95099999999999996,"15min":1.0089999999999999},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.67300000000000004,"15min":0.65600000000000003},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.389,"5min":1.389,"15min":1.389},"last":0.497}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-19T12:58:02.633 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-19T12:58:02.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.774+0000 7f5ff8ed4640 1 -- 172.21.15.57:0/1516730010 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5fe0003970 msgr2=0x7f5fe0003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:02.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.774+0000 7f5ff8ed4640 1 --2- 172.21.15.57:0/1516730010 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5fe0003970 0x7f5fe0003d90 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f5fe4006fd0 tx=0x7f5fe00057f0 comp rx=0 tx=0).stop 2023-10-19T12:58:02.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.774+0000 7f5ff8ed4640 1 -- 172.21.15.57:0/1516730010 shutdown_connections 2023-10-19T12:58:02.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.774+0000 7f5ff8ed4640 1 --2- 172.21.15.57:0/1516730010 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5fe0003970 0x7f5fe0003d90 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:02.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.774+0000 7f5ff8ed4640 1 -- 172.21.15.57:0/1516730010 >> 172.21.15.57:0/1516730010 conn(0x7f5ff4069930 msgr2=0x7f5ff4069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:58:02.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.774+0000 7f5ff8ed4640 1 -- 172.21.15.57:0/1516730010 shutdown_connections 2023-10-19T12:58:02.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.774+0000 7f5ff8ed4640 1 -- 172.21.15.57:0/1516730010 wait complete. 2023-10-19T12:58:02.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.776+0000 7f5ff8ed4640 1 Processor -- start 2023-10-19T12:58:02.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.776+0000 7f5ff8ed4640 1 -- start start 2023-10-19T12:58:02.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.776+0000 7f5ff8ed4640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5fe0003970 0x7f5ff413b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:02.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.776+0000 7f5ff8ed4640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5ff413d780 0x7f5ff40ad960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:02.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.776+0000 7f5ff8ed4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5ff40adea0 0x7f5ff413b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:02.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.776+0000 7f5ff8ed4640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ff40ae320 con 0x7f5ff40adea0 2023-10-19T12:58:02.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.776+0000 7f5ff8ed4640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ff413c6e0 con 0x7f5ff413d780 2023-10-19T12:58:02.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.776+0000 7f5ff8ed4640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ff413c9c0 con 0x7f5fe0003970 2023-10-19T12:58:02.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.777+0000 7f5ff2d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5ff40adea0 0x7f5ff413b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:02.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.777+0000 7f5ff2d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5ff40adea0 0x7f5ff413b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:41758/0 (socket says 172.21.15.57:41758) 2023-10-19T12:58:02.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.777+0000 7f5ff2d9d640 1 -- 172.21.15.57:0/2958621140 learned_addr learned my addr 172.21.15.57:0/2958621140 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:58:02.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.777+0000 7f5ff2d9d640 1 -- 172.21.15.57:0/2958621140 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5fe0003970 msgr2=0x7f5ff413b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:02.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.777+0000 7f5ff2d9d640 1 --2- 172.21.15.57:0/2958621140 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5fe0003970 0x7f5ff413b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:02.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.777+0000 7f5ff2d9d640 1 -- 172.21.15.57:0/2958621140 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5ff413d780 msgr2=0x7f5ff40ad960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:02.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.777+0000 7f5ff2d9d640 1 --2- 172.21.15.57:0/2958621140 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5ff413d780 0x7f5ff40ad960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:02.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.777+0000 7f5ff2d9d640 1 -- 172.21.15.57:0/2958621140 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ff413cca0 con 0x7f5ff40adea0 2023-10-19T12:58:02.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.777+0000 7f5ff2d9d640 1 --2- 172.21.15.57:0/2958621140 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5ff40adea0 0x7f5ff413b9d0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f5ff40ca400 tx=0x7f5fec010f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:58:02.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.777+0000 7f5feaffd640 1 -- 172.21.15.57:0/2958621140 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fec003e40 con 0x7f5ff40adea0 2023-10-19T12:58:02.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.777+0000 7f5feaffd640 1 -- 172.21.15.57:0/2958621140 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f5fec00ea90 con 0x7f5ff40adea0 2023-10-19T12:58:02.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.777+0000 7f5ff8ed4640 1 -- 172.21.15.57:0/2958621140 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ff413cf80 con 0x7f5ff40adea0 2023-10-19T12:58:02.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.777+0000 7f5feaffd640 1 -- 172.21.15.57:0/2958621140 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fec00eea0 con 0x7f5ff40adea0 2023-10-19T12:58:02.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.777+0000 7f5ff8ed4640 1 -- 172.21.15.57:0/2958621140 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ff413b630 con 0x7f5ff40adea0 2023-10-19T12:58:02.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.778+0000 7f5ff8ed4640 1 -- 172.21.15.57:0/2958621140 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ff413d4a0 con 0x7f5ff40adea0 2023-10-19T12:58:02.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.779+0000 7f5feaffd640 1 -- 172.21.15.57:0/2958621140 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 20) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f5fec004050 con 0x7f5ff40adea0 2023-10-19T12:58:02.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.779+0000 7f5feaffd640 1 --2- 172.21.15.57:0/2958621140 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f5fd406af90 0x7f5fd406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:02.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.779+0000 7f5feaffd640 1 -- 172.21.15.57:0/2958621140 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f5fec017030 con 0x7f5ff40adea0 2023-10-19T12:58:02.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.779+0000 7f5ff259c640 1 --2- 172.21.15.57:0/2958621140 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f5fd406af90 0x7f5fd406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:02.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.780+0000 7f5ff259c640 1 --2- 172.21.15.57:0/2958621140 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f5fd406af90 0x7f5fd406d450 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5fe4034000 tx=0x7f5fe4039040 comp rx=0 tx=0).ready entity=mgr.5809 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:58:02.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.783+0000 7f5feaffd640 1 -- 172.21.15.57:0/2958621140 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f5ff413d4a0 con 0x7f5ff40adea0 2023-10-19T12:58:02.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.956+0000 7f5ff8ed4640 1 -- 172.21.15.57:0/2958621140 --> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5ff40708c0 con 0x7f5fd406af90 2023-10-19T12:58:02.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.963+0000 7f5feaffd640 1 -- 172.21.15.57:0/2958621140 <== mgr.5809 v2:172.21.15.179:6832/3449041082 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289621 (secure 0 0 0) 0x7f5fe40015c0 con 0x7f5fd406af90 2023-10-19T12:58:02.970 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:58:02.972 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2023-10-19T12:58:02.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.966+0000 7f5ff8ed4640 1 -- 172.21.15.57:0/2958621140 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f5fd406af90 msgr2=0x7f5fd406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:02.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.966+0000 7f5ff8ed4640 1 --2- 172.21.15.57:0/2958621140 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f5fd406af90 0x7f5fd406d450 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5fe4034000 tx=0x7f5fe4039040 comp rx=0 tx=0).stop 2023-10-19T12:58:02.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.966+0000 7f5ff8ed4640 1 -- 172.21.15.57:0/2958621140 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5ff40adea0 msgr2=0x7f5ff413b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:02.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.966+0000 7f5ff8ed4640 1 --2- 172.21.15.57:0/2958621140 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5ff40adea0 0x7f5ff413b9d0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f5ff40ca400 tx=0x7f5fec010f40 comp rx=0 tx=0).stop 2023-10-19T12:58:02.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.966+0000 7f5ff8ed4640 1 -- 172.21.15.57:0/2958621140 shutdown_connections 2023-10-19T12:58:02.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.966+0000 7f5ff8ed4640 1 --2- 172.21.15.57:0/2958621140 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f5fd406af90 0x7f5fd406d450 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:02.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.966+0000 7f5ff8ed4640 1 --2- 172.21.15.57:0/2958621140 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5ff40adea0 0x7f5ff413b9d0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:02.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.966+0000 7f5ff8ed4640 1 --2- 172.21.15.57:0/2958621140 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5ff413d780 0x7f5ff40ad960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:02.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.966+0000 7f5ff8ed4640 1 --2- 172.21.15.57:0/2958621140 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5fe0003970 0x7f5ff413b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:02.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.966+0000 7f5ff8ed4640 1 -- 172.21.15.57:0/2958621140 >> 172.21.15.57:0/2958621140 conn(0x7f5ff4069930 msgr2=0x7f5ff40af1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:58:02.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.966+0000 7f5ff8ed4640 1 -- 172.21.15.57:0/2958621140 shutdown_connections 2023-10-19T12:58:02.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:02.966+0000 7f5ff8ed4640 1 -- 172.21.15.57:0/2958621140 wait complete. 2023-10-19T12:58:02.980 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-10-19T12:58:02.627957+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":243,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":177,"ondisk_log_size":177,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":127,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":314044,"kb_used_data":7684,"kb_used_omap":12,"kb_used_meta":306227,"kb_avail":754660676,"statfs":{"total":773094113280,"available":772772532224,"internally_reserved":0,"allocated":7868416,"data_stored":4461241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":313577017},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.397600"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.276851+0000","last_change":"2023-10-19T12:57:59.284384+0000","last_active":"2023-10-19T12:58:00.276851+0000","last_peered":"2023-10-19T12:58:00.276851+0000","last_clean":"2023-10-19T12:58:00.276851+0000","last_became_active":"2023-10-19T12:57:59.283993+0000","last_became_peered":"2023-10-19T12:57:59.283993+0000","last_unstale":"2023-10-19T12:58:00.276851+0000","last_undegraded":"2023-10-19T12:58:00.276851+0000","last_fullsized":"2023-10-19T12:58:00.276851+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:37:05.744377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.273924+0000","last_change":"2023-10-19T12:57:59.269413+0000","last_active":"2023-10-19T12:57:59.273924+0000","last_peered":"2023-10-19T12:57:59.273924+0000","last_clean":"2023-10-19T12:57:59.273924+0000","last_became_active":"2023-10-19T12:57:59.269167+0000","last_became_peered":"2023-10-19T12:57:59.269167+0000","last_unstale":"2023-10-19T12:57:59.273924+0000","last_undegraded":"2023-10-19T12:57:59.273924+0000","last_fullsized":"2023-10-19T12:57:59.273924+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:35:33.488742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.286509+0000","last_change":"2023-10-19T12:57:59.278020+0000","last_active":"2023-10-19T12:57:59.286509+0000","last_peered":"2023-10-19T12:57:59.286509+0000","last_clean":"2023-10-19T12:57:59.286509+0000","last_became_active":"2023-10-19T12:57:59.277709+0000","last_became_peered":"2023-10-19T12:57:59.277709+0000","last_unstale":"2023-10-19T12:57:59.286509+0000","last_undegraded":"2023-10-19T12:57:59.286509+0000","last_fullsized":"2023-10-19T12:57:59.286509+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:38:38.391082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.279043+0000","last_change":"2023-10-19T12:57:59.277566+0000","last_active":"2023-10-19T12:57:59.279043+0000","last_peered":"2023-10-19T12:57:59.279043+0000","last_clean":"2023-10-19T12:57:59.279043+0000","last_became_active":"2023-10-19T12:57:59.277336+0000","last_became_peered":"2023-10-19T12:57:59.277336+0000","last_unstale":"2023-10-19T12:57:59.279043+0000","last_undegraded":"2023-10-19T12:57:59.279043+0000","last_fullsized":"2023-10-19T12:57:59.279043+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:39:17.208891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.279837+0000","last_change":"2023-10-19T12:57:59.274819+0000","last_active":"2023-10-19T12:57:59.279837+0000","last_peered":"2023-10-19T12:57:59.279837+0000","last_clean":"2023-10-19T12:57:59.279837+0000","last_became_active":"2023-10-19T12:57:59.274476+0000","last_became_peered":"2023-10-19T12:57:59.274476+0000","last_unstale":"2023-10-19T12:57:59.279837+0000","last_undegraded":"2023-10-19T12:57:59.279837+0000","last_fullsized":"2023-10-19T12:57:59.279837+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:55:16.780598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.276766+0000","last_change":"2023-10-19T12:57:59.277273+0000","last_active":"2023-10-19T12:58:00.276766+0000","last_peered":"2023-10-19T12:58:00.276766+0000","last_clean":"2023-10-19T12:58:00.276766+0000","last_became_active":"2023-10-19T12:57:59.276991+0000","last_became_peered":"2023-10-19T12:57:59.276991+0000","last_unstale":"2023-10-19T12:58:00.276766+0000","last_undegraded":"2023-10-19T12:58:00.276766+0000","last_fullsized":"2023-10-19T12:58:00.276766+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:13:10.195317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.286565+0000","last_change":"2023-10-19T12:57:59.280280+0000","last_active":"2023-10-19T12:57:59.286565+0000","last_peered":"2023-10-19T12:57:59.286565+0000","last_clean":"2023-10-19T12:57:59.286565+0000","last_became_active":"2023-10-19T12:57:59.279626+0000","last_became_peered":"2023-10-19T12:57:59.279626+0000","last_unstale":"2023-10-19T12:57:59.286565+0000","last_undegraded":"2023-10-19T12:57:59.286565+0000","last_fullsized":"2023-10-19T12:57:59.286565+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:46:43.361440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.274836+0000","last_change":"2023-10-19T12:57:59.281673+0000","last_active":"2023-10-19T12:58:00.274836+0000","last_peered":"2023-10-19T12:58:00.274836+0000","last_clean":"2023-10-19T12:58:00.274836+0000","last_became_active":"2023-10-19T12:57:59.281404+0000","last_became_peered":"2023-10-19T12:57:59.281404+0000","last_unstale":"2023-10-19T12:58:00.274836+0000","last_undegraded":"2023-10-19T12:58:00.274836+0000","last_fullsized":"2023-10-19T12:58:00.274836+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:03:46.464251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.275183+0000","last_change":"2023-10-19T12:57:59.282312+0000","last_active":"2023-10-19T12:58:00.275183+0000","last_peered":"2023-10-19T12:58:00.275183+0000","last_clean":"2023-10-19T12:58:00.275183+0000","last_became_active":"2023-10-19T12:57:59.282052+0000","last_became_peered":"2023-10-19T12:57:59.282052+0000","last_unstale":"2023-10-19T12:58:00.275183+0000","last_undegraded":"2023-10-19T12:58:00.275183+0000","last_fullsized":"2023-10-19T12:58:00.275183+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:35:30.427138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.276809+0000","last_change":"2023-10-19T12:57:59.281613+0000","last_active":"2023-10-19T12:58:00.276809+0000","last_peered":"2023-10-19T12:58:00.276809+0000","last_clean":"2023-10-19T12:58:00.276809+0000","last_became_active":"2023-10-19T12:57:59.281322+0000","last_became_peered":"2023-10-19T12:57:59.281322+0000","last_unstale":"2023-10-19T12:58:00.276809+0000","last_undegraded":"2023-10-19T12:58:00.276809+0000","last_fullsized":"2023-10-19T12:58:00.276809+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:33:13.743238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.278808+0000","last_change":"2023-10-19T12:57:59.275510+0000","last_active":"2023-10-19T12:57:59.278808+0000","last_peered":"2023-10-19T12:57:59.278808+0000","last_clean":"2023-10-19T12:57:59.278808+0000","last_became_active":"2023-10-19T12:57:59.275193+0000","last_became_peered":"2023-10-19T12:57:59.275193+0000","last_unstale":"2023-10-19T12:57:59.278808+0000","last_undegraded":"2023-10-19T12:57:59.278808+0000","last_fullsized":"2023-10-19T12:57:59.278808+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:20:23.928988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.275242+0000","last_change":"2023-10-19T12:57:59.269953+0000","last_active":"2023-10-19T12:58:00.275242+0000","last_peered":"2023-10-19T12:58:00.275242+0000","last_clean":"2023-10-19T12:58:00.275242+0000","last_became_active":"2023-10-19T12:57:59.269663+0000","last_became_peered":"2023-10-19T12:57:59.269663+0000","last_unstale":"2023-10-19T12:58:00.275242+0000","last_undegraded":"2023-10-19T12:58:00.275242+0000","last_fullsized":"2023-10-19T12:58:00.275242+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:36:18.446379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.276748+0000","last_change":"2023-10-19T12:57:59.279218+0000","last_active":"2023-10-19T12:58:00.276748+0000","last_peered":"2023-10-19T12:58:00.276748+0000","last_clean":"2023-10-19T12:58:00.276748+0000","last_became_active":"2023-10-19T12:57:59.278919+0000","last_became_peered":"2023-10-19T12:57:59.278919+0000","last_unstale":"2023-10-19T12:58:00.276748+0000","last_undegraded":"2023-10-19T12:58:00.276748+0000","last_fullsized":"2023-10-19T12:58:00.276748+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:50:41.250507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.286325+0000","last_change":"2023-10-19T12:57:59.280185+0000","last_active":"2023-10-19T12:57:59.286325+0000","last_peered":"2023-10-19T12:57:59.286325+0000","last_clean":"2023-10-19T12:57:59.286325+0000","last_became_active":"2023-10-19T12:57:59.279558+0000","last_became_peered":"2023-10-19T12:57:59.279558+0000","last_unstale":"2023-10-19T12:57:59.286325+0000","last_undegraded":"2023-10-19T12:57:59.286325+0000","last_fullsized":"2023-10-19T12:57:59.286325+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:27:42.893031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.273812+0000","last_change":"2023-10-19T12:57:59.268237+0000","last_active":"2023-10-19T12:57:59.273812+0000","last_peered":"2023-10-19T12:57:59.273812+0000","last_clean":"2023-10-19T12:57:59.273812+0000","last_became_active":"2023-10-19T12:57:59.267965+0000","last_became_peered":"2023-10-19T12:57:59.267965+0000","last_unstale":"2023-10-19T12:57:59.273812+0000","last_undegraded":"2023-10-19T12:57:59.273812+0000","last_fullsized":"2023-10-19T12:57:59.273812+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:26:47.615109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.276674+0000","last_change":"2023-10-19T12:57:59.277264+0000","last_active":"2023-10-19T12:58:00.276674+0000","last_peered":"2023-10-19T12:58:00.276674+0000","last_clean":"2023-10-19T12:58:00.276674+0000","last_became_active":"2023-10-19T12:57:59.276973+0000","last_became_peered":"2023-10-19T12:57:59.276973+0000","last_unstale":"2023-10-19T12:58:00.276674+0000","last_undegraded":"2023-10-19T12:58:00.276674+0000","last_fullsized":"2023-10-19T12:58:00.276674+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:19:17.421557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.287675+0000","last_change":"2023-10-19T12:57:59.278487+0000","last_active":"2023-10-19T12:57:59.287675+0000","last_peered":"2023-10-19T12:57:59.287675+0000","last_clean":"2023-10-19T12:57:59.287675+0000","last_became_active":"2023-10-19T12:57:59.276322+0000","last_became_peered":"2023-10-19T12:57:59.276322+0000","last_unstale":"2023-10-19T12:57:59.287675+0000","last_undegraded":"2023-10-19T12:57:59.287675+0000","last_fullsized":"2023-10-19T12:57:59.287675+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:32:09.277054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.273906+0000","last_change":"2023-10-19T12:57:59.268961+0000","last_active":"2023-10-19T12:57:59.273906+0000","last_peered":"2023-10-19T12:57:59.273906+0000","last_clean":"2023-10-19T12:57:59.273906+0000","last_became_active":"2023-10-19T12:57:59.268701+0000","last_became_peered":"2023-10-19T12:57:59.268701+0000","last_unstale":"2023-10-19T12:57:59.273906+0000","last_undegraded":"2023-10-19T12:57:59.273906+0000","last_fullsized":"2023-10-19T12:57:59.273906+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:12:20.974872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.286530+0000","last_change":"2023-10-19T12:57:59.280395+0000","last_active":"2023-10-19T12:57:59.286530+0000","last_peered":"2023-10-19T12:57:59.286530+0000","last_clean":"2023-10-19T12:57:59.286530+0000","last_became_active":"2023-10-19T12:57:59.280082+0000","last_became_peered":"2023-10-19T12:57:59.280082+0000","last_unstale":"2023-10-19T12:57:59.286530+0000","last_undegraded":"2023-10-19T12:57:59.286530+0000","last_fullsized":"2023-10-19T12:57:59.286530+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:52:31.351268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.281922+0000","last_change":"2023-10-19T12:57:59.279468+0000","last_active":"2023-10-19T12:57:59.281922+0000","last_peered":"2023-10-19T12:57:59.281922+0000","last_clean":"2023-10-19T12:57:59.281922+0000","last_became_active":"2023-10-19T12:57:59.279204+0000","last_became_peered":"2023-10-19T12:57:59.279204+0000","last_unstale":"2023-10-19T12:57:59.281922+0000","last_undegraded":"2023-10-19T12:57:59.281922+0000","last_fullsized":"2023-10-19T12:57:59.281922+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:19:20.089249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.280032+0000","last_change":"2023-10-19T12:57:59.273672+0000","last_active":"2023-10-19T12:57:59.280032+0000","last_peered":"2023-10-19T12:57:59.280032+0000","last_clean":"2023-10-19T12:57:59.280032+0000","last_became_active":"2023-10-19T12:57:59.273382+0000","last_became_peered":"2023-10-19T12:57:59.273382+0000","last_unstale":"2023-10-19T12:57:59.280032+0000","last_undegraded":"2023-10-19T12:57:59.280032+0000","last_fullsized":"2023-10-19T12:57:59.280032+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:25:50.660925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.274788+0000","last_change":"2023-10-19T12:57:59.275894+0000","last_active":"2023-10-19T12:58:00.274788+0000","last_peered":"2023-10-19T12:58:00.274788+0000","last_clean":"2023-10-19T12:58:00.274788+0000","last_became_active":"2023-10-19T12:57:59.275622+0000","last_became_peered":"2023-10-19T12:57:59.275622+0000","last_unstale":"2023-10-19T12:58:00.274788+0000","last_undegraded":"2023-10-19T12:58:00.274788+0000","last_fullsized":"2023-10-19T12:58:00.274788+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:50:34.298018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.270235+0000","last_change":"2023-10-19T12:57:59.266850+0000","last_active":"2023-10-19T12:57:59.270235+0000","last_peered":"2023-10-19T12:57:59.270235+0000","last_clean":"2023-10-19T12:57:59.270235+0000","last_became_active":"2023-10-19T12:57:59.266576+0000","last_became_peered":"2023-10-19T12:57:59.266576+0000","last_unstale":"2023-10-19T12:57:59.270235+0000","last_undegraded":"2023-10-19T12:57:59.270235+0000","last_fullsized":"2023-10-19T12:57:59.270235+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:51:51.003245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.289064+0000","last_change":"2023-10-19T12:57:59.279954+0000","last_active":"2023-10-19T12:57:59.289064+0000","last_peered":"2023-10-19T12:57:59.289064+0000","last_clean":"2023-10-19T12:57:59.289064+0000","last_became_active":"2023-10-19T12:57:59.279667+0000","last_became_peered":"2023-10-19T12:57:59.279667+0000","last_unstale":"2023-10-19T12:57:59.289064+0000","last_undegraded":"2023-10-19T12:57:59.289064+0000","last_fullsized":"2023-10-19T12:57:59.289064+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:31:43.020739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.293832+0000","last_change":"2023-10-19T12:57:59.292802+0000","last_active":"2023-10-19T12:57:59.293832+0000","last_peered":"2023-10-19T12:57:59.293832+0000","last_clean":"2023-10-19T12:57:59.293832+0000","last_became_active":"2023-10-19T12:57:59.292429+0000","last_became_peered":"2023-10-19T12:57:59.292429+0000","last_unstale":"2023-10-19T12:57:59.293832+0000","last_undegraded":"2023-10-19T12:57:59.293832+0000","last_fullsized":"2023-10-19T12:57:59.293832+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:23:37.280360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.274984+0000","last_change":"2023-10-19T12:57:59.262814+0000","last_active":"2023-10-19T12:58:00.274984+0000","last_peered":"2023-10-19T12:58:00.274984+0000","last_clean":"2023-10-19T12:58:00.274984+0000","last_became_active":"2023-10-19T12:57:59.262506+0000","last_became_peered":"2023-10-19T12:57:59.262506+0000","last_unstale":"2023-10-19T12:58:00.274984+0000","last_undegraded":"2023-10-19T12:58:00.274984+0000","last_fullsized":"2023-10-19T12:58:00.274984+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:32:35.849874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.277061+0000","last_change":"2023-10-19T12:57:59.283185+0000","last_active":"2023-10-19T12:58:00.277061+0000","last_peered":"2023-10-19T12:58:00.277061+0000","last_clean":"2023-10-19T12:58:00.277061+0000","last_became_active":"2023-10-19T12:57:59.282305+0000","last_became_peered":"2023-10-19T12:57:59.282305+0000","last_unstale":"2023-10-19T12:58:00.277061+0000","last_undegraded":"2023-10-19T12:58:00.277061+0000","last_fullsized":"2023-10-19T12:58:00.277061+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:07:58.427104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.280083+0000","last_change":"2023-10-19T12:57:59.274260+0000","last_active":"2023-10-19T12:57:59.280083+0000","last_peered":"2023-10-19T12:57:59.280083+0000","last_clean":"2023-10-19T12:57:59.280083+0000","last_became_active":"2023-10-19T12:57:59.274012+0000","last_became_peered":"2023-10-19T12:57:59.274012+0000","last_unstale":"2023-10-19T12:57:59.280083+0000","last_undegraded":"2023-10-19T12:57:59.280083+0000","last_fullsized":"2023-10-19T12:57:59.280083+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:58:13.879902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.288166+0000","last_change":"2023-10-19T12:57:59.281600+0000","last_active":"2023-10-19T12:57:59.288166+0000","last_peered":"2023-10-19T12:57:59.288166+0000","last_clean":"2023-10-19T12:57:59.288166+0000","last_became_active":"2023-10-19T12:57:59.281239+0000","last_became_peered":"2023-10-19T12:57:59.281239+0000","last_unstale":"2023-10-19T12:57:59.288166+0000","last_undegraded":"2023-10-19T12:57:59.288166+0000","last_fullsized":"2023-10-19T12:57:59.288166+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:47:41.975167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.279019+0000","last_change":"2023-10-19T12:57:59.277338+0000","last_active":"2023-10-19T12:57:59.279019+0000","last_peered":"2023-10-19T12:57:59.279019+0000","last_clean":"2023-10-19T12:57:59.279019+0000","last_became_active":"2023-10-19T12:57:59.277100+0000","last_became_peered":"2023-10-19T12:57:59.277100+0000","last_unstale":"2023-10-19T12:57:59.279019+0000","last_undegraded":"2023-10-19T12:57:59.279019+0000","last_fullsized":"2023-10-19T12:57:59.279019+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:47:41.975167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.286544+0000","last_change":"2023-10-19T12:57:59.280469+0000","last_active":"2023-10-19T12:57:59.286544+0000","last_peered":"2023-10-19T12:57:59.286544+0000","last_clean":"2023-10-19T12:57:59.286544+0000","last_became_active":"2023-10-19T12:57:59.280002+0000","last_became_peered":"2023-10-19T12:57:59.280002+0000","last_unstale":"2023-10-19T12:57:59.286544+0000","last_undegraded":"2023-10-19T12:57:59.286544+0000","last_fullsized":"2023-10-19T12:57:59.286544+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:54:04.339773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.272211+0000","last_change":"2023-10-19T12:57:59.261431+0000","last_active":"2023-10-19T12:58:00.272211+0000","last_peered":"2023-10-19T12:58:00.272211+0000","last_clean":"2023-10-19T12:58:00.272211+0000","last_became_active":"2023-10-19T12:57:59.261169+0000","last_became_peered":"2023-10-19T12:57:59.261169+0000","last_unstale":"2023-10-19T12:58:00.272211+0000","last_undegraded":"2023-10-19T12:58:00.272211+0000","last_fullsized":"2023-10-19T12:58:00.272211+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:50:32.740818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.278389+0000","last_change":"2023-10-19T12:57:59.272988+0000","last_active":"2023-10-19T12:57:59.278389+0000","last_peered":"2023-10-19T12:57:59.278389+0000","last_clean":"2023-10-19T12:57:59.278389+0000","last_became_active":"2023-10-19T12:57:59.272744+0000","last_became_peered":"2023-10-19T12:57:59.272744+0000","last_unstale":"2023-10-19T12:57:59.278389+0000","last_undegraded":"2023-10-19T12:57:59.278389+0000","last_fullsized":"2023-10-19T12:57:59.278389+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:37:57.451855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"69'4","reported_seq":15,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.287757+0000","last_change":"2023-10-19T12:57:58.252331+0000","last_active":"2023-10-19T12:57:59.287757+0000","last_peered":"2023-10-19T12:57:59.287757+0000","last_clean":"2023-10-19T12:57:59.287757+0000","last_became_active":"2023-10-19T12:57:58.251127+0000","last_became_peered":"2023-10-19T12:57:58.251127+0000","last_unstale":"2023-10-19T12:57:59.287757+0000","last_undegraded":"2023-10-19T12:57:59.287757+0000","last_fullsized":"2023-10-19T12:57:59.287757+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:43:25.389215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.272250+0000","last_change":"2023-10-19T12:57:59.264715+0000","last_active":"2023-10-19T12:58:00.272250+0000","last_peered":"2023-10-19T12:58:00.272250+0000","last_clean":"2023-10-19T12:58:00.272250+0000","last_became_active":"2023-10-19T12:57:59.264476+0000","last_became_peered":"2023-10-19T12:57:59.264476+0000","last_unstale":"2023-10-19T12:58:00.272250+0000","last_undegraded":"2023-10-19T12:58:00.272250+0000","last_fullsized":"2023-10-19T12:58:00.272250+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:06:20.535644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"69'1","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.276793+0000","last_change":"2023-10-19T12:57:58.256650+0000","last_active":"2023-10-19T12:58:00.276793+0000","last_peered":"2023-10-19T12:58:00.276793+0000","last_clean":"2023-10-19T12:58:00.276793+0000","last_became_active":"2023-10-19T12:57:58.256210+0000","last_became_peered":"2023-10-19T12:57:58.256210+0000","last_unstale":"2023-10-19T12:58:00.276793+0000","last_undegraded":"2023-10-19T12:58:00.276793+0000","last_fullsized":"2023-10-19T12:58:00.276793+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:19:19.078844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.279681+0000","last_change":"2023-10-19T12:57:59.269502+0000","last_active":"2023-10-19T12:57:59.279681+0000","last_peered":"2023-10-19T12:57:59.279681+0000","last_clean":"2023-10-19T12:57:59.279681+0000","last_became_active":"2023-10-19T12:57:59.268734+0000","last_became_peered":"2023-10-19T12:57:59.268734+0000","last_unstale":"2023-10-19T12:57:59.279681+0000","last_undegraded":"2023-10-19T12:57:59.279681+0000","last_fullsized":"2023-10-19T12:57:59.279681+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:29:08.762149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"69'2","reported_seq":14,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.277103+0000","last_change":"2023-10-19T12:57:58.255163+0000","last_active":"2023-10-19T12:58:00.277103+0000","last_peered":"2023-10-19T12:58:00.277103+0000","last_clean":"2023-10-19T12:58:00.277103+0000","last_became_active":"2023-10-19T12:57:58.254897+0000","last_became_peered":"2023-10-19T12:57:58.254897+0000","last_unstale":"2023-10-19T12:58:00.277103+0000","last_undegraded":"2023-10-19T12:58:00.277103+0000","last_fullsized":"2023-10-19T12:58:00.277103+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:48:56.789976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.274847+0000","last_change":"2023-10-19T12:57:59.261552+0000","last_active":"2023-10-19T12:58:00.274847+0000","last_peered":"2023-10-19T12:58:00.274847+0000","last_clean":"2023-10-19T12:58:00.274847+0000","last_became_active":"2023-10-19T12:57:59.261224+0000","last_became_peered":"2023-10-19T12:57:59.261224+0000","last_unstale":"2023-10-19T12:58:00.274847+0000","last_undegraded":"2023-10-19T12:58:00.274847+0000","last_fullsized":"2023-10-19T12:58:00.274847+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:38:57.879320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"69'2","reported_seq":14,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.272345+0000","last_change":"2023-10-19T12:57:58.255648+0000","last_active":"2023-10-19T12:58:00.272345+0000","last_peered":"2023-10-19T12:58:00.272345+0000","last_clean":"2023-10-19T12:58:00.272345+0000","last_became_active":"2023-10-19T12:57:58.255149+0000","last_became_peered":"2023-10-19T12:57:58.255149+0000","last_unstale":"2023-10-19T12:58:00.272345+0000","last_undegraded":"2023-10-19T12:58:00.272345+0000","last_fullsized":"2023-10-19T12:58:00.272345+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:55:58.608631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.287223+0000","last_change":"2023-10-19T12:57:59.282345+0000","last_active":"2023-10-19T12:57:59.287223+0000","last_peered":"2023-10-19T12:57:59.287223+0000","last_clean":"2023-10-19T12:57:59.287223+0000","last_became_active":"2023-10-19T12:57:59.282022+0000","last_became_peered":"2023-10-19T12:57:59.282022+0000","last_unstale":"2023-10-19T12:57:59.287223+0000","last_undegraded":"2023-10-19T12:57:59.287223+0000","last_fullsized":"2023-10-19T12:57:59.287223+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:50:41.250507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"69'1","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.293130+0000","last_change":"2023-10-19T12:57:58.257095+0000","last_active":"2023-10-19T12:57:59.293130+0000","last_peered":"2023-10-19T12:57:59.293130+0000","last_clean":"2023-10-19T12:57:59.293130+0000","last_became_active":"2023-10-19T12:57:58.256684+0000","last_became_peered":"2023-10-19T12:57:58.256684+0000","last_unstale":"2023-10-19T12:57:59.293130+0000","last_undegraded":"2023-10-19T12:57:59.293130+0000","last_fullsized":"2023-10-19T12:57:59.293130+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:39:16.198486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.279487+0000","last_change":"2023-10-19T12:57:59.278382+0000","last_active":"2023-10-19T12:57:59.279487+0000","last_peered":"2023-10-19T12:57:59.279487+0000","last_clean":"2023-10-19T12:57:59.279487+0000","last_became_active":"2023-10-19T12:57:59.278100+0000","last_became_peered":"2023-10-19T12:57:59.278100+0000","last_unstale":"2023-10-19T12:57:59.279487+0000","last_undegraded":"2023-10-19T12:57:59.279487+0000","last_fullsized":"2023-10-19T12:57:59.279487+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:21:41.603580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"69'1","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.280435+0000","last_change":"2023-10-19T12:57:58.254342+0000","last_active":"2023-10-19T12:57:59.280435+0000","last_peered":"2023-10-19T12:57:59.280435+0000","last_clean":"2023-10-19T12:57:59.280435+0000","last_became_active":"2023-10-19T12:57:58.254053+0000","last_became_peered":"2023-10-19T12:57:58.254053+0000","last_unstale":"2023-10-19T12:57:59.280435+0000","last_undegraded":"2023-10-19T12:57:59.280435+0000","last_fullsized":"2023-10-19T12:57:59.280435+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:33:12.732833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.275130+0000","last_change":"2023-10-19T12:57:59.261739+0000","last_active":"2023-10-19T12:58:00.275130+0000","last_peered":"2023-10-19T12:58:00.275130+0000","last_clean":"2023-10-19T12:58:00.275130+0000","last_became_active":"2023-10-19T12:57:59.261380+0000","last_became_peered":"2023-10-19T12:57:59.261380+0000","last_unstale":"2023-10-19T12:58:00.275130+0000","last_undegraded":"2023-10-19T12:58:00.275130+0000","last_fullsized":"2023-10-19T12:58:00.275130+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:12:33.439060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":11,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.262572+0000","last_change":"2023-10-19T12:57:58.252697+0000","last_active":"2023-10-19T12:57:59.262572+0000","last_peered":"2023-10-19T12:57:59.262572+0000","last_clean":"2023-10-19T12:57:59.262572+0000","last_became_active":"2023-10-19T12:57:58.252422+0000","last_became_peered":"2023-10-19T12:57:58.252422+0000","last_unstale":"2023-10-19T12:57:59.262572+0000","last_undegraded":"2023-10-19T12:57:59.262572+0000","last_fullsized":"2023-10-19T12:57:59.262572+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:26:25.065741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.281544+0000","last_change":"2023-10-19T12:57:59.280236+0000","last_active":"2023-10-19T12:57:59.281544+0000","last_peered":"2023-10-19T12:57:59.281544+0000","last_clean":"2023-10-19T12:57:59.281544+0000","last_became_active":"2023-10-19T12:57:59.279806+0000","last_became_peered":"2023-10-19T12:57:59.279806+0000","last_unstale":"2023-10-19T12:57:59.281544+0000","last_undegraded":"2023-10-19T12:57:59.281544+0000","last_fullsized":"2023-10-19T12:57:59.281544+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:32:09.277054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":11,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.266101+0000","last_change":"2023-10-19T12:57:58.259037+0000","last_active":"2023-10-19T12:57:59.266101+0000","last_peered":"2023-10-19T12:57:59.266101+0000","last_clean":"2023-10-19T12:57:59.266101+0000","last_became_active":"2023-10-19T12:57:58.258745+0000","last_became_peered":"2023-10-19T12:57:58.258745+0000","last_unstale":"2023-10-19T12:57:59.266101+0000","last_undegraded":"2023-10-19T12:57:59.266101+0000","last_fullsized":"2023-10-19T12:57:59.266101+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:29:50.196136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.270176+0000","last_change":"2023-10-19T12:57:59.264886+0000","last_active":"2023-10-19T12:57:59.270176+0000","last_peered":"2023-10-19T12:57:59.270176+0000","last_clean":"2023-10-19T12:57:59.270176+0000","last_became_active":"2023-10-19T12:57:59.264635+0000","last_became_peered":"2023-10-19T12:57:59.264635+0000","last_unstale":"2023-10-19T12:57:59.270176+0000","last_undegraded":"2023-10-19T12:57:59.270176+0000","last_fullsized":"2023-10-19T12:57:59.270176+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:19:20.089249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":11,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.261596+0000","last_change":"2023-10-19T12:57:58.251390+0000","last_active":"2023-10-19T12:57:59.261596+0000","last_peered":"2023-10-19T12:57:59.261596+0000","last_clean":"2023-10-19T12:57:59.261596+0000","last_became_active":"2023-10-19T12:57:58.251120+0000","last_became_peered":"2023-10-19T12:57:58.251120+0000","last_unstale":"2023-10-19T12:57:59.261596+0000","last_undegraded":"2023-10-19T12:57:59.261596+0000","last_fullsized":"2023-10-19T12:57:59.261596+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:23:36.269955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.272416+0000","last_change":"2023-10-19T12:57:59.270585+0000","last_active":"2023-10-19T12:58:00.272416+0000","last_peered":"2023-10-19T12:58:00.272416+0000","last_clean":"2023-10-19T12:58:00.272416+0000","last_became_active":"2023-10-19T12:57:59.270313+0000","last_became_peered":"2023-10-19T12:57:59.270313+0000","last_unstale":"2023-10-19T12:58:00.272416+0000","last_undegraded":"2023-10-19T12:58:00.272416+0000","last_fullsized":"2023-10-19T12:58:00.272416+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:41:54.210777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":11,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.268616+0000","last_change":"2023-10-19T12:57:58.257516+0000","last_active":"2023-10-19T12:57:59.268616+0000","last_peered":"2023-10-19T12:57:59.268616+0000","last_clean":"2023-10-19T12:57:59.268616+0000","last_became_active":"2023-10-19T12:57:58.257217+0000","last_became_peered":"2023-10-19T12:57:58.257217+0000","last_unstale":"2023-10-19T12:57:59.268616+0000","last_undegraded":"2023-10-19T12:57:59.268616+0000","last_fullsized":"2023-10-19T12:57:59.268616+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:13:05.939605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.288065+0000","last_change":"2023-10-19T12:57:59.282205+0000","last_active":"2023-10-19T12:57:59.288065+0000","last_peered":"2023-10-19T12:57:59.288065+0000","last_clean":"2023-10-19T12:57:59.288065+0000","last_became_active":"2023-10-19T12:57:59.281923+0000","last_became_peered":"2023-10-19T12:57:59.281923+0000","last_unstale":"2023-10-19T12:57:59.288065+0000","last_undegraded":"2023-10-19T12:57:59.288065+0000","last_fullsized":"2023-10-19T12:57:59.288065+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:32:27.978530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"69'3","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.289447+0000","last_change":"2023-10-19T12:57:58.255428+0000","last_active":"2023-10-19T12:57:59.289447+0000","last_peered":"2023-10-19T12:57:59.289447+0000","last_clean":"2023-10-19T12:57:59.289447+0000","last_became_active":"2023-10-19T12:57:58.255138+0000","last_became_peered":"2023-10-19T12:57:58.255138+0000","last_unstale":"2023-10-19T12:57:59.289447+0000","last_undegraded":"2023-10-19T12:57:59.289447+0000","last_fullsized":"2023-10-19T12:57:59.289447+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:40:07.021492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.281230+0000","last_change":"2023-10-19T12:57:59.279730+0000","last_active":"2023-10-19T12:57:59.281230+0000","last_peered":"2023-10-19T12:57:59.281230+0000","last_clean":"2023-10-19T12:57:59.281230+0000","last_became_active":"2023-10-19T12:57:59.279463+0000","last_became_peered":"2023-10-19T12:57:59.279463+0000","last_unstale":"2023-10-19T12:57:59.281230+0000","last_undegraded":"2023-10-19T12:57:59.281230+0000","last_fullsized":"2023-10-19T12:57:59.281230+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:36:39.520498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"69'6","reported_seq":18,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.275275+0000","last_change":"2023-10-19T12:57:58.250143+0000","last_active":"2023-10-19T12:58:00.275275+0000","last_peered":"2023-10-19T12:58:00.275275+0000","last_clean":"2023-10-19T12:58:00.275275+0000","last_became_active":"2023-10-19T12:57:58.249866+0000","last_became_peered":"2023-10-19T12:57:58.249866+0000","last_unstale":"2023-10-19T12:58:00.275275+0000","last_undegraded":"2023-10-19T12:58:00.275275+0000","last_fullsized":"2023-10-19T12:58:00.275275+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:29:34.406278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.272574+0000","last_change":"2023-10-19T12:57:59.266797+0000","last_active":"2023-10-19T12:58:00.272574+0000","last_peered":"2023-10-19T12:58:00.272574+0000","last_clean":"2023-10-19T12:58:00.272574+0000","last_became_active":"2023-10-19T12:57:59.266142+0000","last_became_peered":"2023-10-19T12:57:59.266142+0000","last_unstale":"2023-10-19T12:58:00.272574+0000","last_undegraded":"2023-10-19T12:58:00.272574+0000","last_fullsized":"2023-10-19T12:58:00.272574+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:40:50.284709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"69'2","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.290333+0000","last_change":"2023-10-19T12:57:58.257253+0000","last_active":"2023-10-19T12:57:59.290333+0000","last_peered":"2023-10-19T12:57:59.290333+0000","last_clean":"2023-10-19T12:57:59.290333+0000","last_became_active":"2023-10-19T12:57:58.256982+0000","last_became_peered":"2023-10-19T12:57:58.256982+0000","last_unstale":"2023-10-19T12:57:59.290333+0000","last_undegraded":"2023-10-19T12:57:59.290333+0000","last_fullsized":"2023-10-19T12:57:59.290333+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:26:46.604704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.288024+0000","last_change":"2023-10-19T12:57:59.282874+0000","last_active":"2023-10-19T12:57:59.288024+0000","last_peered":"2023-10-19T12:57:59.288024+0000","last_clean":"2023-10-19T12:57:59.288024+0000","last_became_active":"2023-10-19T12:57:59.282521+0000","last_became_peered":"2023-10-19T12:57:59.282521+0000","last_unstale":"2023-10-19T12:57:59.288024+0000","last_undegraded":"2023-10-19T12:57:59.288024+0000","last_fullsized":"2023-10-19T12:57:59.288024+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:42:43.566749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":11,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.260671+0000","last_change":"2023-10-19T12:57:58.259423+0000","last_active":"2023-10-19T12:57:59.260671+0000","last_peered":"2023-10-19T12:57:59.260671+0000","last_clean":"2023-10-19T12:57:59.260671+0000","last_became_active":"2023-10-19T12:57:58.259165+0000","last_became_peered":"2023-10-19T12:57:58.259165+0000","last_unstale":"2023-10-19T12:57:59.260671+0000","last_undegraded":"2023-10-19T12:57:59.260671+0000","last_fullsized":"2023-10-19T12:57:59.260671+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:46:53.191615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.275075+0000","last_change":"2023-10-19T12:57:59.267621+0000","last_active":"2023-10-19T12:58:00.275075+0000","last_peered":"2023-10-19T12:58:00.275075+0000","last_clean":"2023-10-19T12:58:00.275075+0000","last_became_active":"2023-10-19T12:57:59.267351+0000","last_became_peered":"2023-10-19T12:57:59.267351+0000","last_unstale":"2023-10-19T12:58:00.275075+0000","last_undegraded":"2023-10-19T12:58:00.275075+0000","last_fullsized":"2023-10-19T12:58:00.275075+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:53:20.624726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":11,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.261870+0000","last_change":"2023-10-19T12:57:58.254432+0000","last_active":"2023-10-19T12:57:59.261870+0000","last_peered":"2023-10-19T12:57:59.261870+0000","last_clean":"2023-10-19T12:57:59.261870+0000","last_became_active":"2023-10-19T12:57:58.254162+0000","last_became_peered":"2023-10-19T12:57:58.254162+0000","last_unstale":"2023-10-19T12:57:59.261870+0000","last_undegraded":"2023-10-19T12:57:59.261870+0000","last_fullsized":"2023-10-19T12:57:59.261870+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:38:37.380677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.287695+0000","last_change":"2023-10-19T12:57:59.278651+0000","last_active":"2023-10-19T12:57:59.287695+0000","last_peered":"2023-10-19T12:57:59.287695+0000","last_clean":"2023-10-19T12:57:59.287695+0000","last_became_active":"2023-10-19T12:57:59.278113+0000","last_became_peered":"2023-10-19T12:57:59.278113+0000","last_unstale":"2023-10-19T12:57:59.287695+0000","last_undegraded":"2023-10-19T12:57:59.287695+0000","last_fullsized":"2023-10-19T12:57:59.287695+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:12:20.974872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"69'8","reported_seq":20,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.275013+0000","last_change":"2023-10-19T12:57:58.247030+0000","last_active":"2023-10-19T12:58:00.275013+0000","last_peered":"2023-10-19T12:58:00.275013+0000","last_clean":"2023-10-19T12:58:00.275013+0000","last_became_active":"2023-10-19T12:57:58.246777+0000","last_became_peered":"2023-10-19T12:57:58.246777+0000","last_unstale":"2023-10-19T12:58:00.275013+0000","last_undegraded":"2023-10-19T12:58:00.275013+0000","last_fullsized":"2023-10-19T12:58:00.275013+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:24:54.236739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.275193+0000","last_change":"2023-10-19T12:57:59.262682+0000","last_active":"2023-10-19T12:58:00.275193+0000","last_peered":"2023-10-19T12:58:00.275193+0000","last_clean":"2023-10-19T12:58:00.275193+0000","last_became_active":"2023-10-19T12:57:59.262360+0000","last_became_peered":"2023-10-19T12:57:59.262360+0000","last_unstale":"2023-10-19T12:58:00.275193+0000","last_undegraded":"2023-10-19T12:58:00.275193+0000","last_fullsized":"2023-10-19T12:58:00.275193+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:22:59.082914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"69'1","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.292498+0000","last_change":"2023-10-19T12:57:58.257724+0000","last_active":"2023-10-19T12:57:59.292498+0000","last_peered":"2023-10-19T12:57:59.292498+0000","last_clean":"2023-10-19T12:57:59.292498+0000","last_became_active":"2023-10-19T12:57:58.257432+0000","last_became_peered":"2023-10-19T12:57:58.257432+0000","last_unstale":"2023-10-19T12:57:59.292498+0000","last_undegraded":"2023-10-19T12:57:59.292498+0000","last_fullsized":"2023-10-19T12:57:59.292498+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:21:40.593175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.276683+0000","last_change":"2023-10-19T12:57:59.274011+0000","last_active":"2023-10-19T12:58:00.276683+0000","last_peered":"2023-10-19T12:58:00.276683+0000","last_clean":"2023-10-19T12:58:00.276683+0000","last_became_active":"2023-10-19T12:57:59.273742+0000","last_became_peered":"2023-10-19T12:57:59.273742+0000","last_unstale":"2023-10-19T12:58:00.276683+0000","last_undegraded":"2023-10-19T12:58:00.276683+0000","last_fullsized":"2023-10-19T12:58:00.276683+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:36:39.520498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"69'2","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.288309+0000","last_change":"2023-10-19T12:57:58.249377+0000","last_active":"2023-10-19T12:57:59.288309+0000","last_peered":"2023-10-19T12:57:59.288309+0000","last_clean":"2023-10-19T12:57:59.288309+0000","last_became_active":"2023-10-19T12:57:58.249095+0000","last_became_peered":"2023-10-19T12:57:58.249095+0000","last_unstale":"2023-10-19T12:57:59.288309+0000","last_undegraded":"2023-10-19T12:57:59.288309+0000","last_fullsized":"2023-10-19T12:57:59.288309+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:54:31.441800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.287176+0000","last_change":"2023-10-19T12:57:59.282010+0000","last_active":"2023-10-19T12:57:59.287176+0000","last_peered":"2023-10-19T12:57:59.287176+0000","last_clean":"2023-10-19T12:57:59.287176+0000","last_became_active":"2023-10-19T12:57:59.281731+0000","last_became_peered":"2023-10-19T12:57:59.281731+0000","last_unstale":"2023-10-19T12:57:59.287176+0000","last_undegraded":"2023-10-19T12:57:59.287176+0000","last_fullsized":"2023-10-19T12:57:59.287176+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:19:17.421557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"69'1","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.290864+0000","last_change":"2023-10-19T12:57:58.255311+0000","last_active":"2023-10-19T12:57:59.290864+0000","last_peered":"2023-10-19T12:57:59.290864+0000","last_clean":"2023-10-19T12:57:59.290864+0000","last_became_active":"2023-10-19T12:57:58.255029+0000","last_became_peered":"2023-10-19T12:57:58.255029+0000","last_unstale":"2023-10-19T12:57:59.290864+0000","last_undegraded":"2023-10-19T12:57:59.290864+0000","last_fullsized":"2023-10-19T12:57:59.290864+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:48:56.789976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.276881+0000","last_change":"2023-10-19T12:57:59.277839+0000","last_active":"2023-10-19T12:58:00.276881+0000","last_peered":"2023-10-19T12:58:00.276881+0000","last_clean":"2023-10-19T12:58:00.276881+0000","last_became_active":"2023-10-19T12:57:59.277539+0000","last_became_peered":"2023-10-19T12:57:59.277539+0000","last_unstale":"2023-10-19T12:58:00.276881+0000","last_undegraded":"2023-10-19T12:58:00.276881+0000","last_fullsized":"2023-10-19T12:58:00.276881+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:47:49.700629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":11,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.264273+0000","last_change":"2023-10-19T12:57:58.256158+0000","last_active":"2023-10-19T12:57:59.264273+0000","last_peered":"2023-10-19T12:57:59.264273+0000","last_clean":"2023-10-19T12:57:59.264273+0000","last_became_active":"2023-10-19T12:57:58.255870+0000","last_became_peered":"2023-10-19T12:57:58.255870+0000","last_unstale":"2023-10-19T12:57:59.264273+0000","last_undegraded":"2023-10-19T12:57:59.264273+0000","last_fullsized":"2023-10-19T12:57:59.264273+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:55:15.770193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.272249+0000","last_change":"2023-10-19T12:57:59.273663+0000","last_active":"2023-10-19T12:58:00.272249+0000","last_peered":"2023-10-19T12:58:00.272249+0000","last_clean":"2023-10-19T12:58:00.272249+0000","last_became_active":"2023-10-19T12:57:59.273411+0000","last_became_peered":"2023-10-19T12:57:59.273411+0000","last_unstale":"2023-10-19T12:58:00.272249+0000","last_undegraded":"2023-10-19T12:58:00.272249+0000","last_fullsized":"2023-10-19T12:58:00.272249+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:43:58.424456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"69'2","reported_seq":14,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.276952+0000","last_change":"2023-10-19T12:57:58.254673+0000","last_active":"2023-10-19T12:58:00.276952+0000","last_peered":"2023-10-19T12:58:00.276952+0000","last_clean":"2023-10-19T12:58:00.276952+0000","last_became_active":"2023-10-19T12:57:58.254408+0000","last_became_peered":"2023-10-19T12:57:58.254408+0000","last_unstale":"2023-10-19T12:58:00.276952+0000","last_undegraded":"2023-10-19T12:58:00.276952+0000","last_fullsized":"2023-10-19T12:58:00.276952+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:25:49.650520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.281891+0000","last_change":"2023-10-19T12:57:59.279156+0000","last_active":"2023-10-19T12:57:59.281891+0000","last_peered":"2023-10-19T12:57:59.281891+0000","last_clean":"2023-10-19T12:57:59.281891+0000","last_became_active":"2023-10-19T12:57:59.278885+0000","last_became_peered":"2023-10-19T12:57:59.278885+0000","last_unstale":"2023-10-19T12:57:59.281891+0000","last_undegraded":"2023-10-19T12:57:59.281891+0000","last_fullsized":"2023-10-19T12:57:59.281891+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:43:35.936513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.274755+0000","last_change":"2023-10-19T12:57:58.252362+0000","last_active":"2023-10-19T12:58:00.274755+0000","last_peered":"2023-10-19T12:58:00.274755+0000","last_clean":"2023-10-19T12:58:00.274755+0000","last_became_active":"2023-10-19T12:57:58.252084+0000","last_became_peered":"2023-10-19T12:57:58.252084+0000","last_unstale":"2023-10-19T12:58:00.274755+0000","last_undegraded":"2023-10-19T12:58:00.274755+0000","last_fullsized":"2023-10-19T12:58:00.274755+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:56:55.149376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.289238+0000","last_change":"2023-10-19T12:57:59.275897+0000","last_active":"2023-10-19T12:57:59.289238+0000","last_peered":"2023-10-19T12:57:59.289238+0000","last_clean":"2023-10-19T12:57:59.289238+0000","last_became_active":"2023-10-19T12:57:59.274965+0000","last_became_peered":"2023-10-19T12:57:59.274965+0000","last_unstale":"2023-10-19T12:57:59.289238+0000","last_undegraded":"2023-10-19T12:57:59.289238+0000","last_fullsized":"2023-10-19T12:57:59.289238+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:30:55.579208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":11,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.267824+0000","last_change":"2023-10-19T12:57:58.255960+0000","last_active":"2023-10-19T12:57:59.267824+0000","last_peered":"2023-10-19T12:57:59.267824+0000","last_clean":"2023-10-19T12:57:59.267824+0000","last_became_active":"2023-10-19T12:57:58.255655+0000","last_became_peered":"2023-10-19T12:57:58.255655+0000","last_unstale":"2023-10-19T12:57:59.267824+0000","last_undegraded":"2023-10-19T12:57:59.267824+0000","last_fullsized":"2023-10-19T12:57:59.267824+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:35:32.478337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.276919+0000","last_change":"2023-10-19T12:57:59.284564+0000","last_active":"2023-10-19T12:58:00.276919+0000","last_peered":"2023-10-19T12:58:00.276919+0000","last_clean":"2023-10-19T12:58:00.276919+0000","last_became_active":"2023-10-19T12:57:59.278312+0000","last_became_peered":"2023-10-19T12:57:59.278312+0000","last_unstale":"2023-10-19T12:58:00.276919+0000","last_undegraded":"2023-10-19T12:58:00.276919+0000","last_fullsized":"2023-10-19T12:58:00.276919+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:19:34.001505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":11,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.255502+0000","last_change":"2023-10-19T12:57:58.248153+0000","last_active":"2023-10-19T12:57:59.255502+0000","last_peered":"2023-10-19T12:57:59.255502+0000","last_clean":"2023-10-19T12:57:59.255502+0000","last_became_active":"2023-10-19T12:57:58.247864+0000","last_became_peered":"2023-10-19T12:57:58.247864+0000","last_unstale":"2023-10-19T12:57:59.255502+0000","last_undegraded":"2023-10-19T12:57:59.255502+0000","last_fullsized":"2023-10-19T12:57:59.255502+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:53:21.688950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.272507+0000","last_change":"2023-10-19T12:57:59.260621+0000","last_active":"2023-10-19T12:58:00.272507+0000","last_peered":"2023-10-19T12:58:00.272507+0000","last_clean":"2023-10-19T12:58:00.272507+0000","last_became_active":"2023-10-19T12:57:59.260352+0000","last_became_peered":"2023-10-19T12:57:59.260352+0000","last_unstale":"2023-10-19T12:58:00.272507+0000","last_undegraded":"2023-10-19T12:58:00.272507+0000","last_fullsized":"2023-10-19T12:58:00.272507+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:56:56.159781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"69'2","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.290867+0000","last_change":"2023-10-19T12:57:58.259426+0000","last_active":"2023-10-19T12:57:59.290867+0000","last_peered":"2023-10-19T12:57:59.290867+0000","last_clean":"2023-10-19T12:57:59.290867+0000","last_became_active":"2023-10-19T12:57:58.259118+0000","last_became_peered":"2023-10-19T12:57:58.259118+0000","last_unstale":"2023-10-19T12:57:59.290867+0000","last_undegraded":"2023-10-19T12:57:59.290867+0000","last_fullsized":"2023-10-19T12:57:59.290867+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:07:09.641522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.276817+0000","last_change":"2023-10-19T12:57:59.271932+0000","last_active":"2023-10-19T12:58:00.276817+0000","last_peered":"2023-10-19T12:58:00.276817+0000","last_clean":"2023-10-19T12:58:00.276817+0000","last_became_active":"2023-10-19T12:57:59.271620+0000","last_became_peered":"2023-10-19T12:57:59.271620+0000","last_unstale":"2023-10-19T12:58:00.276817+0000","last_undegraded":"2023-10-19T12:58:00.276817+0000","last_fullsized":"2023-10-19T12:58:00.276817+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:13:06.950010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":11,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.262975+0000","last_change":"2023-10-19T12:57:58.253264+0000","last_active":"2023-10-19T12:57:59.262975+0000","last_peered":"2023-10-19T12:57:59.262975+0000","last_clean":"2023-10-19T12:57:59.262975+0000","last_became_active":"2023-10-19T12:57:58.252867+0000","last_became_peered":"2023-10-19T12:57:58.252867+0000","last_unstale":"2023-10-19T12:57:59.262975+0000","last_undegraded":"2023-10-19T12:57:59.262975+0000","last_fullsized":"2023-10-19T12:57:59.262975+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:08:22.862915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.274916+0000","last_change":"2023-10-19T12:57:59.274237+0000","last_active":"2023-10-19T12:58:00.274916+0000","last_peered":"2023-10-19T12:58:00.274916+0000","last_clean":"2023-10-19T12:58:00.274916+0000","last_became_active":"2023-10-19T12:57:59.273873+0000","last_became_peered":"2023-10-19T12:57:59.273873+0000","last_unstale":"2023-10-19T12:58:00.274916+0000","last_undegraded":"2023-10-19T12:58:00.274916+0000","last_fullsized":"2023-10-19T12:58:00.274916+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:31:14.459655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":11,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.265565+0000","last_change":"2023-10-19T12:57:58.253449+0000","last_active":"2023-10-19T12:57:59.265565+0000","last_peered":"2023-10-19T12:57:59.265565+0000","last_clean":"2023-10-19T12:57:59.265565+0000","last_became_active":"2023-10-19T12:57:58.253169+0000","last_became_peered":"2023-10-19T12:57:58.253169+0000","last_unstale":"2023-10-19T12:57:59.265565+0000","last_undegraded":"2023-10-19T12:57:59.265565+0000","last_fullsized":"2023-10-19T12:57:59.265565+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:55:53.005401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.287287+0000","last_change":"2023-10-19T12:57:59.280611+0000","last_active":"2023-10-19T12:57:59.287287+0000","last_peered":"2023-10-19T12:57:59.287287+0000","last_clean":"2023-10-19T12:57:59.287287+0000","last_became_active":"2023-10-19T12:57:59.280321+0000","last_became_peered":"2023-10-19T12:57:59.280321+0000","last_unstale":"2023-10-19T12:57:59.287287+0000","last_undegraded":"2023-10-19T12:57:59.287287+0000","last_fullsized":"2023-10-19T12:57:59.287287+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:13:46.024291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.274940+0000","last_change":"2023-10-19T12:57:58.246012+0000","last_active":"2023-10-19T12:58:00.274940+0000","last_peered":"2023-10-19T12:58:00.274940+0000","last_clean":"2023-10-19T12:58:00.274940+0000","last_became_active":"2023-10-19T12:57:58.245739+0000","last_became_peered":"2023-10-19T12:57:58.245739+0000","last_unstale":"2023-10-19T12:58:00.274940+0000","last_undegraded":"2023-10-19T12:58:00.274940+0000","last_fullsized":"2023-10-19T12:58:00.274940+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:50:31.730413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.272446+0000","last_change":"2023-10-19T12:57:59.265660+0000","last_active":"2023-10-19T12:58:00.272446+0000","last_peered":"2023-10-19T12:58:00.272446+0000","last_clean":"2023-10-19T12:58:00.272446+0000","last_became_active":"2023-10-19T12:57:59.265334+0000","last_became_peered":"2023-10-19T12:57:59.265334+0000","last_unstale":"2023-10-19T12:58:00.272446+0000","last_undegraded":"2023-10-19T12:58:00.272446+0000","last_fullsized":"2023-10-19T12:58:00.272446+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:28:49.136836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.274837+0000","last_change":"2023-10-19T12:57:58.255022+0000","last_active":"2023-10-19T12:58:00.274837+0000","last_peered":"2023-10-19T12:58:00.274837+0000","last_clean":"2023-10-19T12:58:00.274837+0000","last_became_active":"2023-10-19T12:57:58.254655+0000","last_became_peered":"2023-10-19T12:57:58.254655+0000","last_unstale":"2023-10-19T12:58:00.274837+0000","last_undegraded":"2023-10-19T12:58:00.274837+0000","last_fullsized":"2023-10-19T12:58:00.274837+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:11:37.863957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.289255+0000","last_change":"2023-10-19T12:57:59.279293+0000","last_active":"2023-10-19T12:57:59.289255+0000","last_peered":"2023-10-19T12:57:59.289255+0000","last_clean":"2023-10-19T12:57:59.289255+0000","last_became_active":"2023-10-19T12:57:59.278399+0000","last_became_peered":"2023-10-19T12:57:59.278399+0000","last_unstale":"2023-10-19T12:57:59.289255+0000","last_undegraded":"2023-10-19T12:57:59.289255+0000","last_fullsized":"2023-10-19T12:57:59.289255+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:18:02.353710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":11,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.268350+0000","last_change":"2023-10-19T12:57:58.255022+0000","last_active":"2023-10-19T12:57:59.268350+0000","last_peered":"2023-10-19T12:57:59.268350+0000","last_clean":"2023-10-19T12:57:59.268350+0000","last_became_active":"2023-10-19T12:57:58.254610+0000","last_became_peered":"2023-10-19T12:57:58.254610+0000","last_unstale":"2023-10-19T12:57:59.268350+0000","last_undegraded":"2023-10-19T12:57:59.268350+0000","last_fullsized":"2023-10-19T12:57:59.268350+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:13:09.184912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.275180+0000","last_change":"2023-10-19T12:57:59.261263+0000","last_active":"2023-10-19T12:58:00.275180+0000","last_peered":"2023-10-19T12:58:00.275180+0000","last_clean":"2023-10-19T12:58:00.275180+0000","last_became_active":"2023-10-19T12:57:59.260933+0000","last_became_peered":"2023-10-19T12:57:59.260933+0000","last_unstale":"2023-10-19T12:58:00.275180+0000","last_undegraded":"2023-10-19T12:58:00.275180+0000","last_fullsized":"2023-10-19T12:58:00.275180+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:56:19.223989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.272344+0000","last_change":"2023-10-19T12:57:58.255799+0000","last_active":"2023-10-19T12:58:00.272344+0000","last_peered":"2023-10-19T12:58:00.272344+0000","last_clean":"2023-10-19T12:58:00.272344+0000","last_became_active":"2023-10-19T12:57:58.255531+0000","last_became_peered":"2023-10-19T12:57:58.255531+0000","last_unstale":"2023-10-19T12:58:00.272344+0000","last_undegraded":"2023-10-19T12:58:00.272344+0000","last_fullsized":"2023-10-19T12:58:00.272344+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:03:06.233858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.266105+0000","last_change":"2023-10-19T12:57:59.264409+0000","last_active":"2023-10-19T12:57:59.266105+0000","last_peered":"2023-10-19T12:57:59.266105+0000","last_clean":"2023-10-19T12:57:59.266105+0000","last_became_active":"2023-10-19T12:57:59.263748+0000","last_became_peered":"2023-10-19T12:57:59.263748+0000","last_unstale":"2023-10-19T12:57:59.266105+0000","last_undegraded":"2023-10-19T12:57:59.266105+0000","last_fullsized":"2023-10-19T12:57:59.266105+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:03:18.036820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":11,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.264710+0000","last_change":"2023-10-19T12:57:58.254016+0000","last_active":"2023-10-19T12:57:59.264710+0000","last_peered":"2023-10-19T12:57:59.264710+0000","last_clean":"2023-10-19T12:57:59.264710+0000","last_became_active":"2023-10-19T12:57:58.253756+0000","last_became_peered":"2023-10-19T12:57:58.253756+0000","last_unstale":"2023-10-19T12:57:59.264710+0000","last_undegraded":"2023-10-19T12:57:59.264710+0000","last_fullsized":"2023-10-19T12:57:59.264710+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:52:42.753982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"58'139","reported_seq":290,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-19T12:57:58.244379+0000","last_change":"2023-10-19T12:44:08.636613+0000","last_active":"2023-10-19T12:57:58.244379+0000","last_peered":"2023-10-19T12:57:58.244379+0000","last_clean":"2023-10-19T12:57:58.244379+0000","last_became_active":"2023-10-19T12:44:08.635697+0000","last_became_peered":"2023-10-19T12:44:08.635697+0000","last_unstale":"2023-10-19T12:57:58.244379+0000","last_undegraded":"2023-10-19T12:57:58.244379+0000","last_fullsized":"2023-10-19T12:57:58.244379+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_clean_scrub_stamp":"2023-10-19T12:44:07.399738+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:15:23.685485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575018,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58948,"kb_used_data":1396,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312892,"statfs":{"total":96636764160,"available":96576401408,"internally_reserved":0,"allocated":1429504,"data_stored":1000030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58915275},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:57:38 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.58499999999999996,"15min":0.56100000000000005},"min":{"1min":0.39300000000000002,"5min":0.34499999999999997,"15min":0.248},"max":{"1min":0.67600000000000005,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.496},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.59699999999999998,"15min":0.57799999999999996},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.26900000000000002},"max":{"1min":0.76100000000000001,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.47499999999999998}]},{"osd":1,"last update":"Thu Oct 19 12:57:38 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.58699999999999997,"15min":0.56899999999999995},"min":{"1min":0.40300000000000002,"5min":0.371,"15min":0.28199999999999997},"max":{"1min":0.747,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.59699999999999998,"15min":0.57299999999999995},"min":{"1min":0.46200000000000002,"5min":0.34999999999999998,"15min":0.29299999999999998},"max":{"1min":0.66700000000000004,"5min":1.284,"15min":1.284},"last":0.63700000000000001}]},{"osd":2,"last update":"Thu Oct 19 12:57:38 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.63900000000000001,"15min":0.61799999999999999},"min":{"1min":0.434,"5min":0.41699999999999998,"15min":0.375},"max":{"1min":0.78600000000000003,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.65500000000000003,"15min":0.629},"min":{"1min":0.497,"5min":0.44600000000000001,"15min":0.28399999999999997},"max":{"1min":0.80700000000000005,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.59799999999999998}]},{"osd":3,"last update":"Thu Oct 19 12:57:38 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.61899999999999999,"15min":0.59499999999999997},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34399999999999997},"max":{"1min":0.79100000000000004,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.66000000000000003,"15min":0.64700000000000002},"min":{"1min":0.45300000000000001,"5min":0.44400000000000001,"15min":0.39700000000000002},"max":{"1min":0.77900000000000003,"5min":1.329,"15min":1.329},"last":0.68000000000000005}]},{"osd":4,"last update":"Thu Oct 19 12:57:38 2023","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.46899999999999997,"15min":0.46600000000000003},"min":{"1min":0.28599999999999998,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.73399999999999999,"5min":0.75800000000000001,"15min":0.80500000000000005},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.46800000000000003,"15min":0.46999999999999997},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.627,"5min":0.71999999999999997,"15min":0.78900000000000003},"last":0.61799999999999999}]},{"osd":5,"last update":"Thu Oct 19 12:57:38 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.48199999999999998,"15min":0.48799999999999999},"min":{"1min":0.30499999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.75800000000000001,"5min":0.80600000000000005,"15min":0.90400000000000003},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.52300000000000002,"15min":0.51400000000000001},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.308},"max":{"1min":0.77400000000000002,"5min":0.81999999999999995,"15min":0.92200000000000004},"last":0.55400000000000005}]},{"osd":6,"last update":"Thu Oct 19 12:57:38 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54500000000000004,"15min":0.54400000000000004},"min":{"1min":0.34100000000000003,"5min":0.308,"15min":0.308},"max":{"1min":0.80400000000000005,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.53400000000000003,"15min":0.53200000000000003},"min":{"1min":0.30299999999999999,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.72599999999999998,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.64800000000000002}]}]},{"osd":6,"up_from":13,"seq":55834575018,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31996,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339844,"statfs":{"total":96636764160,"available":96604000256,"internally_reserved":0,"allocated":835584,"data_stored":410190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:57:21 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60299999999999998,"15min":0.58999999999999997},"min":{"1min":0.311,"5min":0.311,"15min":0.24399999999999999},"max":{"1min":0.76000000000000001,"5min":0.79900000000000004,"15min":0.81999999999999995},"last":0.621},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.60499999999999998,"15min":0.58399999999999996},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.77500000000000002,"5min":0.78000000000000003,"15min":0.81200000000000006},"last":0.60899999999999999}]},{"osd":1,"last update":"Thu Oct 19 12:57:21 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.626,"15min":0.60399999999999998},"min":{"1min":0.376,"5min":0.376,"15min":0.34799999999999998},"max":{"1min":0.80500000000000005,"5min":0.83299999999999996,"15min":0.93899999999999995},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.62,"15min":0.60599999999999998},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.28399999999999997},"max":{"1min":0.79400000000000004,"5min":0.82099999999999995,"15min":1.075},"last":0.66800000000000004}]},{"osd":2,"last update":"Thu Oct 19 12:57:21 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60799999999999998,"15min":0.58299999999999996},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.26600000000000001},"max":{"1min":0.748,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.63200000000000001,"15min":0.60599999999999998},"min":{"1min":0.40500000000000003,"5min":0.36099999999999999,"15min":0.28399999999999997},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.71299999999999997}]},{"osd":3,"last update":"Thu Oct 19 12:57:21 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.65100000000000002,"15min":0.625},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.32600000000000001},"max":{"1min":0.83099999999999996,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.64700000000000002,"15min":0.63500000000000001},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.81499999999999995,"5min":0.81699999999999995,"15min":1.0900000000000001},"last":0.59199999999999997}]},{"osd":4,"last update":"Thu Oct 19 12:57:21 2023","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.46300000000000002,"15min":0.47599999999999998},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":1.02},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.47799999999999998,"15min":0.49099999999999999},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.247},"max":{"1min":0.65400000000000003,"5min":0.752,"15min":0.80000000000000004},"last":0.49199999999999999}]},{"osd":5,"last update":"Thu Oct 19 12:57:21 2023","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.47899999999999998,"15min":0.48399999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.26600000000000001},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.505,"5min":0.51400000000000001,"15min":0.52900000000000003},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.312},"max":{"1min":0.78500000000000003,"5min":0.85699999999999998,"15min":0.98799999999999999},"last":0.73299999999999998}]},{"osd":7,"last update":"Thu Oct 19 12:57:21 2023","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.52600000000000002,"15min":0.53100000000000003},"min":{"1min":0.35699999999999998,"5min":0.314,"15min":0.28199999999999997},"max":{"1min":0.76100000000000001,"5min":0.82199999999999995,"15min":1.036},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.53900000000000003,"15min":0.55200000000000005},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":1.1299999999999999},"last":0.53300000000000003}]}]},{"osd":5,"up_from":13,"seq":55834575018,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31988,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339852,"statfs":{"total":96636764160,"available":96604008448,"internally_reserved":0,"allocated":827392,"data_stored":409538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:57:24 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.59199999999999997,"15min":0.58599999999999997},"min":{"1min":0.39800000000000002,"5min":0.26700000000000002,"15min":0.223},"max":{"1min":0.79400000000000004,"5min":0.82699999999999996,"15min":0.88800000000000001},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.60799999999999998,"15min":0.59099999999999997},"min":{"1min":0.39700000000000002,"5min":0.312,"15min":0.27800000000000002},"max":{"1min":0.72699999999999998,"5min":0.80400000000000005,"15min":0.874},"last":0.379}]},{"osd":1,"last update":"Thu Oct 19 12:57:24 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.60499999999999998,"15min":0.59599999999999997},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.32800000000000001},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.876},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.61299999999999999,"15min":0.59999999999999998},"min":{"1min":0.439,"5min":0.34699999999999998,"15min":0.34499999999999997},"max":{"1min":0.79500000000000004,"5min":0.80100000000000005,"15min":0.83499999999999996},"last":0.54200000000000004}]},{"osd":2,"last update":"Thu Oct 19 12:57:24 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59399999999999997,"15min":0.57899999999999996},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.307},"max":{"1min":0.70899999999999996,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.62,"15min":0.61399999999999999},"min":{"1min":0.47399999999999998,"5min":0.41999999999999998,"15min":0.33700000000000002},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.93600000000000005},"last":0.505}]},{"osd":3,"last update":"Thu Oct 19 12:57:24 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.60999999999999999,"15min":0.59999999999999998},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.39300000000000002},"max":{"1min":0.73199999999999998,"5min":0.77900000000000003,"15min":0.874},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63900000000000001,"15min":0.63400000000000001},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.41999999999999998},"max":{"1min":0.746,"5min":0.80400000000000005,"15min":0.89500000000000002},"last":0.48899999999999999}]},{"osd":4,"last update":"Thu Oct 19 12:57:24 2023","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.34499999999999997,"5min":0.32100000000000001,"15min":0.28899999999999998},"max":{"1min":0.66100000000000003,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.47099999999999997,"15min":0.47599999999999998},"min":{"1min":0.307,"5min":0.307,"15min":0.23899999999999999},"max":{"1min":0.75900000000000001,"5min":0.79400000000000004,"15min":0.80500000000000005},"last":0.35699999999999998}]},{"osd":6,"last update":"Thu Oct 19 12:57:24 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.47399999999999998,"15min":0.47899999999999998},"min":{"1min":0.34100000000000003,"5min":0.31900000000000001,"15min":0.25900000000000001},"max":{"1min":0.74399999999999999,"5min":0.77500000000000002,"15min":0.79300000000000004},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.50800000000000001,"15min":0.498},"min":{"1min":0.38500000000000001,"5min":0.33600000000000002,"15min":0.30399999999999999},"max":{"1min":0.71899999999999997,"5min":0.79100000000000004,"15min":0.81299999999999994},"last":0.52300000000000002}]},{"osd":7,"last update":"Thu Oct 19 12:57:24 2023","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.47999999999999998,"15min":0.49399999999999999},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.315},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71899999999999997},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.52700000000000002,"15min":0.53400000000000003},"min":{"1min":0.39700000000000002,"5min":0.34300000000000003,"15min":0.27500000000000002},"max":{"1min":0.77900000000000003,"5min":0.78100000000000003,"15min":0.78500000000000003},"last":0.45000000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575018,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31992,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339848,"statfs":{"total":96636764160,"available":96604004352,"internally_reserved":0,"allocated":831488,"data_stored":410120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:57:37 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.54900000000000004,"15min":0.55100000000000005},"min":{"1min":0.36499999999999999,"5min":0.32900000000000001,"15min":0.28899999999999998},"max":{"1min":0.72699999999999998,"5min":0.77400000000000002,"15min":0.84799999999999998},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55900000000000005,"15min":0.54800000000000004},"min":{"1min":0.38100000000000001,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.73999999999999999,"5min":0.77800000000000002,"15min":0.78200000000000003},"last":0.60799999999999998}]},{"osd":1,"last update":"Thu Oct 19 12:57:37 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.61099999999999999,"15min":0.60399999999999998},"min":{"1min":0.377,"5min":0.377,"15min":0.34799999999999998},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.86199999999999999},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.61799999999999999,"15min":0.61099999999999999},"min":{"1min":0.44800000000000001,"5min":0.40200000000000002,"15min":0.35799999999999998},"max":{"1min":0.78900000000000003,"5min":0.79500000000000004,"15min":0.873},"last":0.72799999999999998}]},{"osd":2,"last update":"Thu Oct 19 12:57:37 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.61599999999999999,"15min":0.60899999999999999},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.254},"max":{"1min":0.76000000000000001,"5min":0.85299999999999998,"15min":0.86599999999999999},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.60899999999999999,"15min":0.61699999999999999},"min":{"1min":0.44800000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.75,"5min":0.82999999999999996,"15min":0.85999999999999999},"last":0.70199999999999996}]},{"osd":3,"last update":"Thu Oct 19 12:57:37 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.58499999999999996,"15min":0.57499999999999996},"min":{"1min":0.35199999999999998,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.70199999999999996,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.64400000000000002,"15min":0.63400000000000001},"min":{"1min":0.45700000000000002,"5min":0.41299999999999998,"15min":0.37},"max":{"1min":0.77300000000000002,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.73999999999999999}]},{"osd":5,"last update":"Thu Oct 19 12:57:37 2023","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.437,"15min":0.45200000000000001},"min":{"1min":0.30599999999999999,"5min":0.29099999999999998,"15min":0.249},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.81299999999999994},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.45500000000000002,"15min":0.46400000000000002},"min":{"1min":0.32000000000000001,"5min":0.26400000000000001,"15min":0.23499999999999999},"max":{"1min":0.71999999999999997,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.46000000000000002}]},{"osd":6,"last update":"Thu Oct 19 12:57:37 2023","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45400000000000001,"15min":0.46000000000000002},"min":{"1min":0.32300000000000001,"5min":0.29499999999999998,"15min":0.26400000000000001},"max":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.78600000000000003},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.47899999999999998,"15min":0.505},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.27100000000000002},"max":{"1min":0.76800000000000002,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.57099999999999995}]},{"osd":7,"last update":"Thu Oct 19 12:57:37 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.49199999999999999,"15min":0.50900000000000001},"min":{"1min":0.38600000000000001,"5min":0.307,"15min":0.27500000000000002},"max":{"1min":0.71399999999999997,"5min":0.79100000000000004,"15min":0.80200000000000005},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.53900000000000003,"15min":0.55900000000000005},"min":{"1min":0.41799999999999998,"5min":0.32100000000000001,"15min":0.30199999999999999},"max":{"1min":0.745,"5min":0.80500000000000005,"15min":0.83499999999999996},"last":0.621}]}]},{"osd":3,"up_from":13,"seq":55834575017,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36168,"kb_used_data":824,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335672,"statfs":{"total":96636764160,"available":96599728128,"internally_reserved":0,"allocated":843776,"data_stored":411172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:57:22 2023","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.40300000000000002,"15min":0.39500000000000002},"min":{"1min":0.248,"5min":0.23599999999999999,"15min":0.20899999999999999},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":1.04},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.42299999999999999,"15min":0.40699999999999997},"min":{"1min":0.28399999999999997,"5min":0.248,"15min":0.245},"max":{"1min":0.57399999999999995,"5min":0.60999999999999999,"15min":0.64600000000000002},"last":0.28000000000000003}]},{"osd":1,"last update":"Thu Oct 19 12:57:22 2023","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.41599999999999998,"15min":0.40400000000000003},"min":{"1min":0.30599999999999999,"5min":0.26100000000000001,"15min":0.22500000000000001},"max":{"1min":0.58299999999999996,"5min":0.621,"15min":0.69899999999999995},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.46100000000000002,"15min":0.442},"min":{"1min":0.32000000000000001,"5min":0.29899999999999999,"15min":0.215},"max":{"1min":0.65200000000000002,"5min":0.65900000000000003,"15min":0.874},"last":0.33900000000000002}]},{"osd":2,"last update":"Thu Oct 19 12:57:22 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.46899999999999997,"15min":0.45900000000000002},"min":{"1min":0.34699999999999998,"5min":0.32500000000000001,"15min":0.26100000000000001},"max":{"1min":0.59399999999999997,"5min":0.65800000000000003,"15min":0.73599999999999999},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.498,"15min":0.48299999999999998},"min":{"1min":0.36299999999999999,"5min":0.34799999999999998,"15min":0.247},"max":{"1min":0.64000000000000001,"5min":0.72599999999999998,"15min":0.96499999999999997},"last":0.379}]},{"osd":4,"last update":"Thu Oct 19 12:57:22 2023","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.67700000000000005,"15min":0.65900000000000003},"min":{"1min":0.49399999999999999,"5min":0.45900000000000002,"15min":0.40300000000000002},"max":{"1min":0.90500000000000003,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.70199999999999996,"15min":0.68700000000000006},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.42099999999999999},"max":{"1min":0.91300000000000003,"5min":0.98999999999999999,"15min":0.99399999999999999},"last":0.78200000000000003}]},{"osd":5,"last update":"Thu Oct 19 12:57:22 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.66900000000000004,"15min":0.64900000000000002},"min":{"1min":0.495,"5min":0.495,"15min":0.40500000000000003},"max":{"1min":0.93899999999999995,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.65700000000000003,"15min":0.64500000000000002},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.35899999999999999},"max":{"1min":0.85899999999999999,"5min":0.97299999999999998,"15min":0.99099999999999999},"last":0.58799999999999997}]},{"osd":6,"last update":"Thu Oct 19 12:57:22 2023","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.66500000000000004,"15min":0.65400000000000003},"min":{"1min":0.439,"5min":0.439,"15min":0.41199999999999998},"max":{"1min":0.84699999999999998,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.68100000000000005,"15min":0.66700000000000004},"min":{"1min":0.55200000000000005,"5min":0.496,"15min":0.35899999999999999},"max":{"1min":0.86899999999999999,"5min":1.05,"15min":1.05},"last":0.71199999999999997}]},{"osd":7,"last update":"Thu Oct 19 12:57:22 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.67400000000000004,"15min":0.66500000000000004},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.39800000000000002},"max":{"1min":0.95699999999999996,"5min":1,"15min":1},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.68400000000000005,"15min":0.67100000000000004},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.88500000000000001,"5min":0.93400000000000005,"15min":0.97599999999999998},"last":0.73599999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575019,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54216,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317624,"statfs":{"total":96636764160,"available":96581246976,"internally_reserved":0,"allocated":839680,"data_stored":411150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:57:25 2023","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.38900000000000001,"15min":0.38500000000000001},"min":{"1min":0.254,"5min":0.20899999999999999,"15min":0.187},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.83999999999999997},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.376,"5min":0.41699999999999998,"15min":0.40100000000000002},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.57799999999999996,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.36199999999999999}]},{"osd":1,"last update":"Thu Oct 19 12:57:25 2023","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.40799999999999997,"15min":0.40000000000000002},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.57899999999999996,"5min":0.71099999999999997,"15min":0.80400000000000005},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.44,"15min":0.43099999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.23400000000000001},"max":{"1min":0.57599999999999996,"5min":0.63,"15min":0.78600000000000003},"last":0.32000000000000001}]},{"osd":3,"last update":"Thu Oct 19 12:57:25 2023","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.46700000000000003,"15min":0.45600000000000002},"min":{"1min":0.25900000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.623,"5min":0.68000000000000005,"15min":0.79600000000000004},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.497,"15min":0.48799999999999999},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.87},"last":0.375}]},{"osd":4,"last update":"Thu Oct 19 12:57:25 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.64800000000000002,"15min":0.64100000000000001},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":1.0569999999999999},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.66400000000000003,"15min":0.65500000000000003},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.32000000000000001},"max":{"1min":0.91200000000000003,"5min":0.92300000000000004,"15min":1.0780000000000001},"last":0.45400000000000001}]},{"osd":5,"last update":"Thu Oct 19 12:57:25 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.68100000000000005,"15min":0.66000000000000003},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.78600000000000003,"5min":0.96499999999999997,"15min":0.99399999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.67500000000000004,"15min":0.65700000000000003},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.90200000000000002,"5min":0.94599999999999995,"15min":0.97099999999999997},"last":0.46999999999999997}]},{"osd":6,"last update":"Thu Oct 19 12:57:25 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.67900000000000005,"15min":0.67200000000000004},"min":{"1min":0.441,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.86799999999999999,"5min":1.008,"15min":1.008},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.68200000000000005,"15min":0.66900000000000004},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.93899999999999995,"5min":0.995,"15min":0.995},"last":0.623}]},{"osd":7,"last update":"Thu Oct 19 12:57:25 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.69499999999999995,"15min":0.68400000000000005},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.36399999999999999},"max":{"1min":0.98999999999999999,"5min":1.0149999999999999,"15min":1.111},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.69799999999999995,"15min":0.68500000000000005},"min":{"1min":0.42099999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.89700000000000002,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.60599999999999998}]}]},{"osd":1,"up_from":13,"seq":55834575018,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32004,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339836,"statfs":{"total":96636764160,"available":96603992064,"internally_reserved":0,"allocated":835584,"data_stored":409698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:57:32 2023","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.39200000000000002,"15min":0.378},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.187},"max":{"1min":0.59099999999999997,"5min":0.61799999999999999,"15min":0.66200000000000003},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.443,"15min":0.42199999999999999},"min":{"1min":0.30499999999999999,"5min":0.28000000000000003,"15min":0.21199999999999999},"max":{"1min":0.63600000000000001,"5min":0.68799999999999994,"15min":0.69799999999999995},"last":0.33900000000000002}]},{"osd":2,"last update":"Thu Oct 19 12:57:32 2023","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.44600000000000001,"15min":0.42099999999999999},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.20100000000000001},"max":{"1min":0.56000000000000005,"5min":0.68999999999999995,"15min":0.69999999999999996},"last":0.374},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.45800000000000002,"15min":0.44400000000000001},"min":{"1min":0.34699999999999998,"5min":0.309,"15min":0.23699999999999999},"max":{"1min":0.60599999999999998,"5min":0.67500000000000004,"15min":0.755},"last":0.41299999999999998}]},{"osd":3,"last update":"Thu Oct 19 12:57:32 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.49099999999999999,"15min":0.47699999999999998},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.26200000000000001},"max":{"1min":0.63500000000000001,"5min":0.65900000000000003,"15min":0.83799999999999997},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.435,"15min":0.42499999999999999},"min":{"1min":0.29499999999999998,"5min":0.28499999999999998,"15min":0.27000000000000002},"max":{"1min":0.56499999999999995,"5min":0.70699999999999996,"15min":0.80500000000000005},"last":0.35399999999999998}]},{"osd":4,"last update":"Thu Oct 19 12:57:32 2023","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.67500000000000004,"15min":0.65700000000000003},"min":{"1min":0.44,"5min":0.44,"15min":0.39300000000000002},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.622},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.66600000000000004,"15min":0.65200000000000002},"min":{"1min":0.495,"5min":0.42599999999999999,"15min":0.314},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":1.1910000000000001},"last":0.68799999999999994}]},{"osd":5,"last update":"Thu Oct 19 12:57:32 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.66700000000000004,"15min":0.65700000000000003},"min":{"1min":0.52300000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.878,"5min":0.878,"15min":0.91800000000000004},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.69099999999999995,"15min":0.67200000000000004},"min":{"1min":0.53700000000000003,"5min":0.47099999999999997,"15min":0.38700000000000001},"max":{"1min":0.88500000000000001,"5min":0.93999999999999995,"15min":1.117},"last":0.58499999999999996}]},{"osd":6,"last update":"Thu Oct 19 12:57:32 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.67700000000000005,"15min":0.66200000000000003},"min":{"1min":0.48399999999999999,"5min":0.36799999999999999,"15min":0.36399999999999999},"max":{"1min":0.92200000000000004,"5min":1.0069999999999999,"15min":1.077},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.69399999999999995,"15min":0.68500000000000005},"min":{"1min":0.47399999999999998,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.995,"5min":0.995,"15min":1.0009999999999999},"last":0.64900000000000002}]},{"osd":7,"last update":"Thu Oct 19 12:57:32 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.69499999999999995,"15min":0.68100000000000005},"min":{"1min":0.54000000000000004,"5min":0.46999999999999997,"15min":0.443},"max":{"1min":0.84499999999999997,"5min":0.97399999999999998,"15min":1.0149999999999999},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.70099999999999996,"15min":0.68300000000000005},"min":{"1min":0.46500000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":1.0489999999999999},"last":0.64300000000000002}]}]},{"osd":0,"up_from":13,"seq":55834575019,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36732,"kb_used_data":1392,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335108,"statfs":{"total":96636764160,"available":96599150592,"internally_reserved":0,"allocated":1425408,"data_stored":999343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Oct 19 12:57:36 2023","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.40100000000000002,"15min":0.38500000000000001},"min":{"1min":0.28699999999999998,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.442,"15min":0.42399999999999999},"min":{"1min":0.25,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.623,"5min":0.68700000000000006,"15min":0.746},"last":0.32500000000000001}]},{"osd":2,"last update":"Thu Oct 19 12:57:36 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.46700000000000003,"15min":0.45000000000000001},"min":{"1min":0.33900000000000002,"5min":0.26200000000000001,"15min":0.25800000000000001},"max":{"1min":0.752,"5min":0.752,"15min":0.76300000000000001},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.45100000000000001,"15min":0.42799999999999999},"min":{"1min":0.27500000000000002,"5min":0.216,"15min":0.20699999999999999},"max":{"1min":0.59399999999999997,"5min":0.68600000000000005,"15min":0.76800000000000002},"last":0.29299999999999998}]},{"osd":3,"last update":"Thu Oct 19 12:57:36 2023","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.42299999999999999,"15min":0.40100000000000002},"min":{"1min":0.24299999999999999,"5min":0.23400000000000001,"15min":0.22800000000000001},"max":{"1min":1.2829999999999999,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.38},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.51000000000000001,"15min":0.48799999999999999},"min":{"1min":0.33300000000000002,"5min":0.29399999999999998,"15min":0.29199999999999998},"max":{"1min":1.4990000000000001,"5min":1.4990000000000001,"15min":1.4990000000000001},"last":0.375}]},{"osd":4,"last update":"Thu Oct 19 12:57:36 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.65000000000000002,"15min":0.63},"min":{"1min":0.46700000000000003,"5min":0.45100000000000001,"15min":0.34999999999999998},"max":{"1min":1.361,"5min":1.361,"15min":1.361},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.65800000000000003,"15min":0.64100000000000001},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.35999999999999999},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.998},"last":0.55300000000000005}]},{"osd":5,"last update":"Thu Oct 19 12:57:36 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.69099999999999995,"15min":0.66800000000000004},"min":{"1min":0.44800000000000001,"5min":0.42099999999999999,"15min":0.38},"max":{"1min":1.325,"5min":1.325,"15min":1.325},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.69699999999999995,"15min":0.67800000000000005},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.318},"max":{"1min":0.88400000000000001,"5min":0.98699999999999999,"15min":1.056},"last":0.51500000000000001}]},{"osd":6,"last update":"Thu Oct 19 12:57:36 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.67000000000000004,"15min":0.65600000000000003},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.38500000000000001},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.438},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.68200000000000005,"15min":0.65800000000000003},"min":{"1min":0.45500000000000002,"5min":0.41999999999999998,"15min":0.38900000000000001},"max":{"1min":0.995,"5min":0.995,"15min":1.109},"last":0.47599999999999998}]},{"osd":7,"last update":"Thu Oct 19 12:57:36 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.69199999999999995,"15min":0.67700000000000005},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.41099999999999998},"max":{"1min":0.92000000000000004,"5min":0.95099999999999996,"15min":1.0089999999999999},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.67300000000000004,"15min":0.65600000000000003},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.389,"5min":1.389,"15min":1.389},"last":0.497}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-19T12:58:02.987 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2023-10-19T12:58:03.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.124+0000 7fbcbf264640 1 -- 172.21.15.57:0/1060818512 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbcb815b2a0 msgr2=0x7fbcb80cf720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:03.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.124+0000 7fbcbf264640 1 --2- 172.21.15.57:0/1060818512 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbcb815b2a0 0x7fbcb80cf720 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fbcb808a810 tx=0x7fbca802bd10 comp rx=0 tx=0).stop 2023-10-19T12:58:03.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.124+0000 7fbcbf264640 1 -- 172.21.15.57:0/1060818512 shutdown_connections 2023-10-19T12:58:03.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.124+0000 7fbcbf264640 1 --2- 172.21.15.57:0/1060818512 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbcb80ceb40 0x7fbcb80b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:03.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.124+0000 7fbcbf264640 1 --2- 172.21.15.57:0/1060818512 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbcb815b2a0 0x7fbcb80cf720 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:03.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.124+0000 7fbcbf264640 1 --2- 172.21.15.57:0/1060818512 >> v2:172.21.15.179:3300/0 conn(0x7fbcb80cf340 0x7fbcb80d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:03.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.124+0000 7fbcbf264640 1 -- 172.21.15.57:0/1060818512 >> 172.21.15.57:0/1060818512 conn(0x7fbcb80d0d60 msgr2=0x7fbcb80d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:58:03.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.124+0000 7fbcbf264640 1 -- 172.21.15.57:0/1060818512 shutdown_connections 2023-10-19T12:58:03.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.124+0000 7fbcbf264640 1 -- 172.21.15.57:0/1060818512 wait complete. 2023-10-19T12:58:03.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.126+0000 7fbcbf264640 1 Processor -- start 2023-10-19T12:58:03.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.126+0000 7fbcbf264640 1 -- start start 2023-10-19T12:58:03.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.127+0000 7fbcbf264640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbcb80ceb40 0x7fbcb814fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:03.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.127+0000 7fbcbf264640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbcb80cf340 0x7fbcb806af90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:03.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.127+0000 7fbcbf264640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbcb815b2a0 0x7fbcb806b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:03.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.127+0000 7fbcbf264640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb81503e0 con 0x7fbcb80ceb40 2023-10-19T12:58:03.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.127+0000 7fbcbf264640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb80b6c40 con 0x7fbcb80cf340 2023-10-19T12:58:03.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.127+0000 7fbcbf264640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb81f48a0 con 0x7fbcb815b2a0 2023-10-19T12:58:03.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.127+0000 7fbcbd000640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbcb80ceb40 0x7fbcb814fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:03.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.127+0000 7fbcbd000640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbcb80ceb40 0x7fbcb814fea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:41778/0 (socket says 172.21.15.57:41778) 2023-10-19T12:58:03.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.127+0000 7fbcbd000640 1 -- 172.21.15.57:0/1561964788 learned_addr learned my addr 172.21.15.57:0/1561964788 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:58:03.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.127+0000 7fbcb7fff640 1 --2- 172.21.15.57:0/1561964788 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbcb80cf340 0x7fbcb806af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:03.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.127+0000 7fbcbd801640 1 --2- 172.21.15.57:0/1561964788 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbcb815b2a0 0x7fbcb806b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:03.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.127+0000 7fbcbd000640 1 -- 172.21.15.57:0/1561964788 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbcb815b2a0 msgr2=0x7fbcb806b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:03.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.127+0000 7fbcbd000640 1 --2- 172.21.15.57:0/1561964788 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbcb815b2a0 0x7fbcb806b4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:03.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.127+0000 7fbcbd000640 1 -- 172.21.15.57:0/1561964788 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbcb80cf340 msgr2=0x7fbcb806af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:03.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.127+0000 7fbcbd000640 1 --2- 172.21.15.57:0/1561964788 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbcb80cf340 0x7fbcb806af90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:03.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.127+0000 7fbcbd000640 1 -- 172.21.15.57:0/1561964788 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbca8009560 con 0x7fbcb80ceb40 2023-10-19T12:58:03.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.127+0000 7fbcbd801640 1 --2- 172.21.15.57:0/1561964788 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbcb815b2a0 0x7fbcb806b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:58:03.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.127+0000 7fbcbd000640 1 --2- 172.21.15.57:0/1561964788 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbcb80ceb40 0x7fbcb814fea0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fbcac00c920 tx=0x7fbcac00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:58:03.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.127+0000 7fbcb57fa640 1 -- 172.21.15.57:0/1561964788 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcac007c70 con 0x7fbcb80ceb40 2023-10-19T12:58:03.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.127+0000 7fbcb57fa640 1 -- 172.21.15.57:0/1561964788 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fbcac00f450 con 0x7fbcb80ceb40 2023-10-19T12:58:03.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.127+0000 7fbcbf264640 1 -- 172.21.15.57:0/1561964788 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcb81f4b40 con 0x7fbcb80ceb40 2023-10-19T12:58:03.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.127+0000 7fbcb57fa640 1 -- 172.21.15.57:0/1561964788 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcac00f860 con 0x7fbcb80ceb40 2023-10-19T12:58:03.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.127+0000 7fbcbf264640 1 -- 172.21.15.57:0/1561964788 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcb81f5100 con 0x7fbcb80ceb40 2023-10-19T12:58:03.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.128+0000 7fbcbf264640 1 -- 172.21.15.57:0/1561964788 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcb80690e0 con 0x7fbcb80ceb40 2023-10-19T12:58:03.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.129+0000 7fbcb57fa640 1 -- 172.21.15.57:0/1561964788 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 20) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fbcac00f9c0 con 0x7fbcb80ceb40 2023-10-19T12:58:03.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.129+0000 7fbcb57fa640 1 --2- 172.21.15.57:0/1561964788 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7fbca406af60 0x7fbca406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:03.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.129+0000 7fbcb57fa640 1 -- 172.21.15.57:0/1561964788 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fbcac08a1f0 con 0x7fbcb80ceb40 2023-10-19T12:58:03.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.130+0000 7fbcb7fff640 1 --2- 172.21.15.57:0/1561964788 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7fbca406af60 0x7fbca406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:03.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.130+0000 7fbcb7fff640 1 --2- 172.21.15.57:0/1561964788 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7fbca406af60 0x7fbca406d420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fbca8035000 tx=0x7fbca8004730 comp rx=0 tx=0).ready entity=mgr.5809 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:58:03.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.134+0000 7fbcb57fa640 1 -- 172.21.15.57:0/1561964788 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fbcac054fd0 con 0x7fbcb80ceb40 2023-10-19T12:58:03.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.379+0000 7fbcbf264640 1 -- 172.21.15.57:0/1561964788 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fbcb80d0b40 con 0x7fbcb80ceb40 2023-10-19T12:58:03.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.380+0000 7fbcb57fa640 1 -- 172.21.15.57:0/1561964788 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1012 (secure 0 0 0) 0x7fbcac0585f0 con 0x7fbcb80ceb40 2023-10-19T12:58:03.384 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:58:03.384 INFO:teuthology.orchestra.run.smithi057.stdout:{"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":850,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":70,"num_osds":8,"num_up_osds":8,"osd_up_since":1697719445,"num_in_osds":8,"osd_in_since":1697719436,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":23,"data_bytes":592778,"bytes_used":321581056,"bytes_avail":772772532224,"bytes_total":773094113280,"write_bytes_sec":2655,"read_op_per_sec":0,"write_op_per_sec":8},"fsmap":{"epoch":23,"id":3,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":3,"rank":0,"name":"c","status":"up:active","gid":6510}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2023-10-19T12:46:58.360342+0000","services":{}},"progress_events":{}} 2023-10-19T12:58:03.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.382+0000 7fbcbf264640 1 -- 172.21.15.57:0/1561964788 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7fbca406af60 msgr2=0x7fbca406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:03.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.382+0000 7fbcbf264640 1 --2- 172.21.15.57:0/1561964788 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7fbca406af60 0x7fbca406d420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fbca8035000 tx=0x7fbca8004730 comp rx=0 tx=0).stop 2023-10-19T12:58:03.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.382+0000 7fbcbf264640 1 -- 172.21.15.57:0/1561964788 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbcb80ceb40 msgr2=0x7fbcb814fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:03.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.382+0000 7fbcbf264640 1 --2- 172.21.15.57:0/1561964788 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbcb80ceb40 0x7fbcb814fea0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fbcac00c920 tx=0x7fbcac00cde0 comp rx=0 tx=0).stop 2023-10-19T12:58:03.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.382+0000 7fbcbf264640 1 -- 172.21.15.57:0/1561964788 shutdown_connections 2023-10-19T12:58:03.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.382+0000 7fbcbf264640 1 --2- 172.21.15.57:0/1561964788 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7fbca406af60 0x7fbca406d420 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:03.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.382+0000 7fbcbf264640 1 --2- 172.21.15.57:0/1561964788 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbcb815b2a0 0x7fbcb806b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:03.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.382+0000 7fbcbf264640 1 --2- 172.21.15.57:0/1561964788 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbcb80cf340 0x7fbcb806af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:03.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.382+0000 7fbcbf264640 1 --2- 172.21.15.57:0/1561964788 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbcb80ceb40 0x7fbcb814fea0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:03.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.382+0000 7fbcbf264640 1 -- 172.21.15.57:0/1561964788 >> 172.21.15.57:0/1561964788 conn(0x7fbcb80d0d60 msgr2=0x7fbcb80b1c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:58:03.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.382+0000 7fbcbf264640 1 -- 172.21.15.57:0/1561964788 shutdown_connections 2023-10-19T12:58:03.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.382+0000 7fbcbf264640 1 -- 172.21.15.57:0/1561964788 wait complete. 2023-10-19T12:58:03.396 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2023-10-19T12:58:03.396 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-19T12:58:03.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.502+0000 7fee8dbf5640 1 -- 172.21.15.57:0/1309569077 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fee8815b2a0 msgr2=0x7fee880cf720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:03.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.502+0000 7fee8dbf5640 1 --2- 172.21.15.57:0/1309569077 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fee8815b2a0 0x7fee880cf720 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fee88089ae0 tx=0x7fee7c02bd10 comp rx=0 tx=0).stop 2023-10-19T12:58:03.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.502+0000 7fee8dbf5640 1 -- 172.21.15.57:0/1309569077 shutdown_connections 2023-10-19T12:58:03.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.502+0000 7fee8dbf5640 1 --2- 172.21.15.57:0/1309569077 >> v2:172.21.15.179:3300/0 conn(0x7fee880ceb40 0x7fee880b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:03.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.502+0000 7fee8dbf5640 1 --2- 172.21.15.57:0/1309569077 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fee8815b2a0 0x7fee880cf720 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:03.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.502+0000 7fee8dbf5640 1 --2- 172.21.15.57:0/1309569077 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fee880cf340 0x7fee880d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:03.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.502+0000 7fee8dbf5640 1 -- 172.21.15.57:0/1309569077 >> 172.21.15.57:0/1309569077 conn(0x7fee880d0d60 msgr2=0x7fee880d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:58:03.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.502+0000 7fee8dbf5640 1 -- 172.21.15.57:0/1309569077 shutdown_connections 2023-10-19T12:58:03.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.502+0000 7fee8dbf5640 1 -- 172.21.15.57:0/1309569077 wait complete. 2023-10-19T12:58:03.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.503+0000 7fee8dbf5640 1 Processor -- start 2023-10-19T12:58:03.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.504+0000 7fee8dbf5640 1 -- start start 2023-10-19T12:58:03.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.504+0000 7fee8dbf5640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fee880ceb40 0x7fee880b3980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:03.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.504+0000 7fee8dbf5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fee880cf340 0x7fee88158270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:03.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.504+0000 7fee8dbf5640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fee8815b2a0 0x7fee8815a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:03.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.504+0000 7fee8dbf5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee880b3f60 con 0x7fee880cf340 2023-10-19T12:58:03.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.504+0000 7fee8dbf5640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee880b4240 con 0x7fee8815b2a0 2023-10-19T12:58:03.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.504+0000 7fee8dbf5640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee880b4520 con 0x7fee880ceb40 2023-10-19T12:58:03.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.504+0000 7fee86ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fee880cf340 0x7fee88158270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:03.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.504+0000 7fee86ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fee880cf340 0x7fee88158270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:41788/0 (socket says 172.21.15.57:41788) 2023-10-19T12:58:03.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.504+0000 7fee86ffd640 1 -- 172.21.15.57:0/1685265218 learned_addr learned my addr 172.21.15.57:0/1685265218 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:58:03.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.504+0000 7fee877fe640 1 --2- 172.21.15.57:0/1685265218 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fee880ceb40 0x7fee880b3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:03.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.505+0000 7fee87fff640 1 --2- 172.21.15.57:0/1685265218 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fee8815b2a0 0x7fee8815a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:03.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.505+0000 7fee86ffd640 1 -- 172.21.15.57:0/1685265218 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fee880ceb40 msgr2=0x7fee880b3980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:03.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.505+0000 7fee86ffd640 1 --2- 172.21.15.57:0/1685265218 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fee880ceb40 0x7fee880b3980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:03.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.505+0000 7fee86ffd640 1 -- 172.21.15.57:0/1685265218 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fee8815b2a0 msgr2=0x7fee8815a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:03.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.505+0000 7fee86ffd640 1 --2- 172.21.15.57:0/1685265218 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fee8815b2a0 0x7fee8815a0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:03.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.505+0000 7fee86ffd640 1 -- 172.21.15.57:0/1685265218 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee7c009560 con 0x7fee880cf340 2023-10-19T12:58:03.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.505+0000 7fee87fff640 1 --2- 172.21.15.57:0/1685265218 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fee8815b2a0 0x7fee8815a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:58:03.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.505+0000 7fee877fe640 1 --2- 172.21.15.57:0/1685265218 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fee880ceb40 0x7fee880b3980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:58:03.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.505+0000 7fee86ffd640 1 --2- 172.21.15.57:0/1685265218 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fee880cf340 0x7fee88158270 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fee7c009850 tx=0x7fee7c02da40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:58:03.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.505+0000 7fee6ffff640 1 -- 172.21.15.57:0/1685265218 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee7c003800 con 0x7fee880cf340 2023-10-19T12:58:03.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.505+0000 7fee6ffff640 1 -- 172.21.15.57:0/1685265218 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fee7c003a50 con 0x7fee880cf340 2023-10-19T12:58:03.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.505+0000 7fee8dbf5640 1 -- 172.21.15.57:0/1685265218 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee880b4800 con 0x7fee880cf340 2023-10-19T12:58:03.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.505+0000 7fee6ffff640 1 -- 172.21.15.57:0/1685265218 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee7c003e20 con 0x7fee880cf340 2023-10-19T12:58:03.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.505+0000 7fee8dbf5640 1 -- 172.21.15.57:0/1685265218 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee881587b0 con 0x7fee880cf340 2023-10-19T12:58:03.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.506+0000 7fee8dbf5640 1 -- 172.21.15.57:0/1685265218 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee880d04f0 con 0x7fee880cf340 2023-10-19T12:58:03.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.506+0000 7fee6ffff640 1 -- 172.21.15.57:0/1685265218 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 20) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fee7c03e450 con 0x7fee880cf340 2023-10-19T12:58:03.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.507+0000 7fee6ffff640 1 --2- 172.21.15.57:0/1685265218 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7fee7406af60 0x7fee7406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:03.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.507+0000 7fee6ffff640 1 -- 172.21.15.57:0/1685265218 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fee7c0ac450 con 0x7fee880cf340 2023-10-19T12:58:03.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.507+0000 7fee877fe640 1 --2- 172.21.15.57:0/1685265218 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7fee7406af60 0x7fee7406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:03.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.508+0000 7fee877fe640 1 --2- 172.21.15.57:0/1685265218 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7fee7406af60 0x7fee7406d420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fee78005fd0 tx=0x7fee78005850 comp rx=0 tx=0).ready entity=mgr.5809 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:58:03.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.511+0000 7fee6ffff640 1 -- 172.21.15.57:0/1685265218 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fee7c0772b0 con 0x7fee880cf340 2023-10-19T12:58:03.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.683+0000 7fee8dbf5640 1 -- 172.21.15.57:0/1685265218 --> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fee880cc170 con 0x7fee7406af60 2023-10-19T12:58:03.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.691+0000 7fee6ffff640 1 -- 172.21.15.57:0/1685265218 <== mgr.5809 v2:172.21.15.179:6832/3449041082 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289621 (secure 0 0 0) 0x7fee7800b950 con 0x7fee7406af60 2023-10-19T12:58:03.696 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:58:03.698 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2023-10-19T12:58:03.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.694+0000 7fee8dbf5640 1 -- 172.21.15.57:0/1685265218 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7fee7406af60 msgr2=0x7fee7406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:03.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.694+0000 7fee8dbf5640 1 --2- 172.21.15.57:0/1685265218 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7fee7406af60 0x7fee7406d420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fee78005fd0 tx=0x7fee78005850 comp rx=0 tx=0).stop 2023-10-19T12:58:03.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.694+0000 7fee8dbf5640 1 -- 172.21.15.57:0/1685265218 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fee880cf340 msgr2=0x7fee88158270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:03.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.694+0000 7fee8dbf5640 1 --2- 172.21.15.57:0/1685265218 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fee880cf340 0x7fee88158270 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fee7c009850 tx=0x7fee7c02da40 comp rx=0 tx=0).stop 2023-10-19T12:58:03.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.694+0000 7fee8dbf5640 1 -- 172.21.15.57:0/1685265218 shutdown_connections 2023-10-19T12:58:03.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.694+0000 7fee8dbf5640 1 --2- 172.21.15.57:0/1685265218 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7fee7406af60 0x7fee7406d420 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:03.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.694+0000 7fee8dbf5640 1 --2- 172.21.15.57:0/1685265218 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fee8815b2a0 0x7fee8815a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:03.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.694+0000 7fee8dbf5640 1 --2- 172.21.15.57:0/1685265218 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fee880cf340 0x7fee88158270 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:03.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.694+0000 7fee8dbf5640 1 --2- 172.21.15.57:0/1685265218 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fee880ceb40 0x7fee880b3980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:03.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.694+0000 7fee8dbf5640 1 -- 172.21.15.57:0/1685265218 >> 172.21.15.57:0/1685265218 conn(0x7fee880d0d60 msgr2=0x7fee880b1e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:58:03.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.694+0000 7fee8dbf5640 1 -- 172.21.15.57:0/1685265218 shutdown_connections 2023-10-19T12:58:03.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:03.694+0000 7fee8dbf5640 1 -- 172.21.15.57:0/1685265218 wait complete. 2023-10-19T12:58:03.708 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-10-19T12:58:02.627957+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":243,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":177,"ondisk_log_size":177,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":127,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":314044,"kb_used_data":7684,"kb_used_omap":12,"kb_used_meta":306227,"kb_avail":754660676,"statfs":{"total":773094113280,"available":772772532224,"internally_reserved":0,"allocated":7868416,"data_stored":4461241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":313577017},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.397600"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.276851+0000","last_change":"2023-10-19T12:57:59.284384+0000","last_active":"2023-10-19T12:58:00.276851+0000","last_peered":"2023-10-19T12:58:00.276851+0000","last_clean":"2023-10-19T12:58:00.276851+0000","last_became_active":"2023-10-19T12:57:59.283993+0000","last_became_peered":"2023-10-19T12:57:59.283993+0000","last_unstale":"2023-10-19T12:58:00.276851+0000","last_undegraded":"2023-10-19T12:58:00.276851+0000","last_fullsized":"2023-10-19T12:58:00.276851+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:37:05.744377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.273924+0000","last_change":"2023-10-19T12:57:59.269413+0000","last_active":"2023-10-19T12:57:59.273924+0000","last_peered":"2023-10-19T12:57:59.273924+0000","last_clean":"2023-10-19T12:57:59.273924+0000","last_became_active":"2023-10-19T12:57:59.269167+0000","last_became_peered":"2023-10-19T12:57:59.269167+0000","last_unstale":"2023-10-19T12:57:59.273924+0000","last_undegraded":"2023-10-19T12:57:59.273924+0000","last_fullsized":"2023-10-19T12:57:59.273924+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:35:33.488742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.286509+0000","last_change":"2023-10-19T12:57:59.278020+0000","last_active":"2023-10-19T12:57:59.286509+0000","last_peered":"2023-10-19T12:57:59.286509+0000","last_clean":"2023-10-19T12:57:59.286509+0000","last_became_active":"2023-10-19T12:57:59.277709+0000","last_became_peered":"2023-10-19T12:57:59.277709+0000","last_unstale":"2023-10-19T12:57:59.286509+0000","last_undegraded":"2023-10-19T12:57:59.286509+0000","last_fullsized":"2023-10-19T12:57:59.286509+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:38:38.391082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.279043+0000","last_change":"2023-10-19T12:57:59.277566+0000","last_active":"2023-10-19T12:57:59.279043+0000","last_peered":"2023-10-19T12:57:59.279043+0000","last_clean":"2023-10-19T12:57:59.279043+0000","last_became_active":"2023-10-19T12:57:59.277336+0000","last_became_peered":"2023-10-19T12:57:59.277336+0000","last_unstale":"2023-10-19T12:57:59.279043+0000","last_undegraded":"2023-10-19T12:57:59.279043+0000","last_fullsized":"2023-10-19T12:57:59.279043+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:39:17.208891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.279837+0000","last_change":"2023-10-19T12:57:59.274819+0000","last_active":"2023-10-19T12:57:59.279837+0000","last_peered":"2023-10-19T12:57:59.279837+0000","last_clean":"2023-10-19T12:57:59.279837+0000","last_became_active":"2023-10-19T12:57:59.274476+0000","last_became_peered":"2023-10-19T12:57:59.274476+0000","last_unstale":"2023-10-19T12:57:59.279837+0000","last_undegraded":"2023-10-19T12:57:59.279837+0000","last_fullsized":"2023-10-19T12:57:59.279837+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:55:16.780598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.276766+0000","last_change":"2023-10-19T12:57:59.277273+0000","last_active":"2023-10-19T12:58:00.276766+0000","last_peered":"2023-10-19T12:58:00.276766+0000","last_clean":"2023-10-19T12:58:00.276766+0000","last_became_active":"2023-10-19T12:57:59.276991+0000","last_became_peered":"2023-10-19T12:57:59.276991+0000","last_unstale":"2023-10-19T12:58:00.276766+0000","last_undegraded":"2023-10-19T12:58:00.276766+0000","last_fullsized":"2023-10-19T12:58:00.276766+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:13:10.195317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.286565+0000","last_change":"2023-10-19T12:57:59.280280+0000","last_active":"2023-10-19T12:57:59.286565+0000","last_peered":"2023-10-19T12:57:59.286565+0000","last_clean":"2023-10-19T12:57:59.286565+0000","last_became_active":"2023-10-19T12:57:59.279626+0000","last_became_peered":"2023-10-19T12:57:59.279626+0000","last_unstale":"2023-10-19T12:57:59.286565+0000","last_undegraded":"2023-10-19T12:57:59.286565+0000","last_fullsized":"2023-10-19T12:57:59.286565+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:46:43.361440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.274836+0000","last_change":"2023-10-19T12:57:59.281673+0000","last_active":"2023-10-19T12:58:00.274836+0000","last_peered":"2023-10-19T12:58:00.274836+0000","last_clean":"2023-10-19T12:58:00.274836+0000","last_became_active":"2023-10-19T12:57:59.281404+0000","last_became_peered":"2023-10-19T12:57:59.281404+0000","last_unstale":"2023-10-19T12:58:00.274836+0000","last_undegraded":"2023-10-19T12:58:00.274836+0000","last_fullsized":"2023-10-19T12:58:00.274836+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:03:46.464251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.275183+0000","last_change":"2023-10-19T12:57:59.282312+0000","last_active":"2023-10-19T12:58:00.275183+0000","last_peered":"2023-10-19T12:58:00.275183+0000","last_clean":"2023-10-19T12:58:00.275183+0000","last_became_active":"2023-10-19T12:57:59.282052+0000","last_became_peered":"2023-10-19T12:57:59.282052+0000","last_unstale":"2023-10-19T12:58:00.275183+0000","last_undegraded":"2023-10-19T12:58:00.275183+0000","last_fullsized":"2023-10-19T12:58:00.275183+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:35:30.427138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.276809+0000","last_change":"2023-10-19T12:57:59.281613+0000","last_active":"2023-10-19T12:58:00.276809+0000","last_peered":"2023-10-19T12:58:00.276809+0000","last_clean":"2023-10-19T12:58:00.276809+0000","last_became_active":"2023-10-19T12:57:59.281322+0000","last_became_peered":"2023-10-19T12:57:59.281322+0000","last_unstale":"2023-10-19T12:58:00.276809+0000","last_undegraded":"2023-10-19T12:58:00.276809+0000","last_fullsized":"2023-10-19T12:58:00.276809+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:33:13.743238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.278808+0000","last_change":"2023-10-19T12:57:59.275510+0000","last_active":"2023-10-19T12:57:59.278808+0000","last_peered":"2023-10-19T12:57:59.278808+0000","last_clean":"2023-10-19T12:57:59.278808+0000","last_became_active":"2023-10-19T12:57:59.275193+0000","last_became_peered":"2023-10-19T12:57:59.275193+0000","last_unstale":"2023-10-19T12:57:59.278808+0000","last_undegraded":"2023-10-19T12:57:59.278808+0000","last_fullsized":"2023-10-19T12:57:59.278808+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:20:23.928988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.275242+0000","last_change":"2023-10-19T12:57:59.269953+0000","last_active":"2023-10-19T12:58:00.275242+0000","last_peered":"2023-10-19T12:58:00.275242+0000","last_clean":"2023-10-19T12:58:00.275242+0000","last_became_active":"2023-10-19T12:57:59.269663+0000","last_became_peered":"2023-10-19T12:57:59.269663+0000","last_unstale":"2023-10-19T12:58:00.275242+0000","last_undegraded":"2023-10-19T12:58:00.275242+0000","last_fullsized":"2023-10-19T12:58:00.275242+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:36:18.446379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.276748+0000","last_change":"2023-10-19T12:57:59.279218+0000","last_active":"2023-10-19T12:58:00.276748+0000","last_peered":"2023-10-19T12:58:00.276748+0000","last_clean":"2023-10-19T12:58:00.276748+0000","last_became_active":"2023-10-19T12:57:59.278919+0000","last_became_peered":"2023-10-19T12:57:59.278919+0000","last_unstale":"2023-10-19T12:58:00.276748+0000","last_undegraded":"2023-10-19T12:58:00.276748+0000","last_fullsized":"2023-10-19T12:58:00.276748+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:50:41.250507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.286325+0000","last_change":"2023-10-19T12:57:59.280185+0000","last_active":"2023-10-19T12:57:59.286325+0000","last_peered":"2023-10-19T12:57:59.286325+0000","last_clean":"2023-10-19T12:57:59.286325+0000","last_became_active":"2023-10-19T12:57:59.279558+0000","last_became_peered":"2023-10-19T12:57:59.279558+0000","last_unstale":"2023-10-19T12:57:59.286325+0000","last_undegraded":"2023-10-19T12:57:59.286325+0000","last_fullsized":"2023-10-19T12:57:59.286325+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:27:42.893031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.273812+0000","last_change":"2023-10-19T12:57:59.268237+0000","last_active":"2023-10-19T12:57:59.273812+0000","last_peered":"2023-10-19T12:57:59.273812+0000","last_clean":"2023-10-19T12:57:59.273812+0000","last_became_active":"2023-10-19T12:57:59.267965+0000","last_became_peered":"2023-10-19T12:57:59.267965+0000","last_unstale":"2023-10-19T12:57:59.273812+0000","last_undegraded":"2023-10-19T12:57:59.273812+0000","last_fullsized":"2023-10-19T12:57:59.273812+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:26:47.615109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.276674+0000","last_change":"2023-10-19T12:57:59.277264+0000","last_active":"2023-10-19T12:58:00.276674+0000","last_peered":"2023-10-19T12:58:00.276674+0000","last_clean":"2023-10-19T12:58:00.276674+0000","last_became_active":"2023-10-19T12:57:59.276973+0000","last_became_peered":"2023-10-19T12:57:59.276973+0000","last_unstale":"2023-10-19T12:58:00.276674+0000","last_undegraded":"2023-10-19T12:58:00.276674+0000","last_fullsized":"2023-10-19T12:58:00.276674+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:19:17.421557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.287675+0000","last_change":"2023-10-19T12:57:59.278487+0000","last_active":"2023-10-19T12:57:59.287675+0000","last_peered":"2023-10-19T12:57:59.287675+0000","last_clean":"2023-10-19T12:57:59.287675+0000","last_became_active":"2023-10-19T12:57:59.276322+0000","last_became_peered":"2023-10-19T12:57:59.276322+0000","last_unstale":"2023-10-19T12:57:59.287675+0000","last_undegraded":"2023-10-19T12:57:59.287675+0000","last_fullsized":"2023-10-19T12:57:59.287675+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:32:09.277054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.273906+0000","last_change":"2023-10-19T12:57:59.268961+0000","last_active":"2023-10-19T12:57:59.273906+0000","last_peered":"2023-10-19T12:57:59.273906+0000","last_clean":"2023-10-19T12:57:59.273906+0000","last_became_active":"2023-10-19T12:57:59.268701+0000","last_became_peered":"2023-10-19T12:57:59.268701+0000","last_unstale":"2023-10-19T12:57:59.273906+0000","last_undegraded":"2023-10-19T12:57:59.273906+0000","last_fullsized":"2023-10-19T12:57:59.273906+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:12:20.974872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.286530+0000","last_change":"2023-10-19T12:57:59.280395+0000","last_active":"2023-10-19T12:57:59.286530+0000","last_peered":"2023-10-19T12:57:59.286530+0000","last_clean":"2023-10-19T12:57:59.286530+0000","last_became_active":"2023-10-19T12:57:59.280082+0000","last_became_peered":"2023-10-19T12:57:59.280082+0000","last_unstale":"2023-10-19T12:57:59.286530+0000","last_undegraded":"2023-10-19T12:57:59.286530+0000","last_fullsized":"2023-10-19T12:57:59.286530+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:52:31.351268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.281922+0000","last_change":"2023-10-19T12:57:59.279468+0000","last_active":"2023-10-19T12:57:59.281922+0000","last_peered":"2023-10-19T12:57:59.281922+0000","last_clean":"2023-10-19T12:57:59.281922+0000","last_became_active":"2023-10-19T12:57:59.279204+0000","last_became_peered":"2023-10-19T12:57:59.279204+0000","last_unstale":"2023-10-19T12:57:59.281922+0000","last_undegraded":"2023-10-19T12:57:59.281922+0000","last_fullsized":"2023-10-19T12:57:59.281922+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:19:20.089249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.280032+0000","last_change":"2023-10-19T12:57:59.273672+0000","last_active":"2023-10-19T12:57:59.280032+0000","last_peered":"2023-10-19T12:57:59.280032+0000","last_clean":"2023-10-19T12:57:59.280032+0000","last_became_active":"2023-10-19T12:57:59.273382+0000","last_became_peered":"2023-10-19T12:57:59.273382+0000","last_unstale":"2023-10-19T12:57:59.280032+0000","last_undegraded":"2023-10-19T12:57:59.280032+0000","last_fullsized":"2023-10-19T12:57:59.280032+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:25:50.660925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.274788+0000","last_change":"2023-10-19T12:57:59.275894+0000","last_active":"2023-10-19T12:58:00.274788+0000","last_peered":"2023-10-19T12:58:00.274788+0000","last_clean":"2023-10-19T12:58:00.274788+0000","last_became_active":"2023-10-19T12:57:59.275622+0000","last_became_peered":"2023-10-19T12:57:59.275622+0000","last_unstale":"2023-10-19T12:58:00.274788+0000","last_undegraded":"2023-10-19T12:58:00.274788+0000","last_fullsized":"2023-10-19T12:58:00.274788+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:50:34.298018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.270235+0000","last_change":"2023-10-19T12:57:59.266850+0000","last_active":"2023-10-19T12:57:59.270235+0000","last_peered":"2023-10-19T12:57:59.270235+0000","last_clean":"2023-10-19T12:57:59.270235+0000","last_became_active":"2023-10-19T12:57:59.266576+0000","last_became_peered":"2023-10-19T12:57:59.266576+0000","last_unstale":"2023-10-19T12:57:59.270235+0000","last_undegraded":"2023-10-19T12:57:59.270235+0000","last_fullsized":"2023-10-19T12:57:59.270235+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:51:51.003245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.289064+0000","last_change":"2023-10-19T12:57:59.279954+0000","last_active":"2023-10-19T12:57:59.289064+0000","last_peered":"2023-10-19T12:57:59.289064+0000","last_clean":"2023-10-19T12:57:59.289064+0000","last_became_active":"2023-10-19T12:57:59.279667+0000","last_became_peered":"2023-10-19T12:57:59.279667+0000","last_unstale":"2023-10-19T12:57:59.289064+0000","last_undegraded":"2023-10-19T12:57:59.289064+0000","last_fullsized":"2023-10-19T12:57:59.289064+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:31:43.020739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.293832+0000","last_change":"2023-10-19T12:57:59.292802+0000","last_active":"2023-10-19T12:57:59.293832+0000","last_peered":"2023-10-19T12:57:59.293832+0000","last_clean":"2023-10-19T12:57:59.293832+0000","last_became_active":"2023-10-19T12:57:59.292429+0000","last_became_peered":"2023-10-19T12:57:59.292429+0000","last_unstale":"2023-10-19T12:57:59.293832+0000","last_undegraded":"2023-10-19T12:57:59.293832+0000","last_fullsized":"2023-10-19T12:57:59.293832+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:23:37.280360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.274984+0000","last_change":"2023-10-19T12:57:59.262814+0000","last_active":"2023-10-19T12:58:00.274984+0000","last_peered":"2023-10-19T12:58:00.274984+0000","last_clean":"2023-10-19T12:58:00.274984+0000","last_became_active":"2023-10-19T12:57:59.262506+0000","last_became_peered":"2023-10-19T12:57:59.262506+0000","last_unstale":"2023-10-19T12:58:00.274984+0000","last_undegraded":"2023-10-19T12:58:00.274984+0000","last_fullsized":"2023-10-19T12:58:00.274984+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:32:35.849874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.277061+0000","last_change":"2023-10-19T12:57:59.283185+0000","last_active":"2023-10-19T12:58:00.277061+0000","last_peered":"2023-10-19T12:58:00.277061+0000","last_clean":"2023-10-19T12:58:00.277061+0000","last_became_active":"2023-10-19T12:57:59.282305+0000","last_became_peered":"2023-10-19T12:57:59.282305+0000","last_unstale":"2023-10-19T12:58:00.277061+0000","last_undegraded":"2023-10-19T12:58:00.277061+0000","last_fullsized":"2023-10-19T12:58:00.277061+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:07:58.427104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.280083+0000","last_change":"2023-10-19T12:57:59.274260+0000","last_active":"2023-10-19T12:57:59.280083+0000","last_peered":"2023-10-19T12:57:59.280083+0000","last_clean":"2023-10-19T12:57:59.280083+0000","last_became_active":"2023-10-19T12:57:59.274012+0000","last_became_peered":"2023-10-19T12:57:59.274012+0000","last_unstale":"2023-10-19T12:57:59.280083+0000","last_undegraded":"2023-10-19T12:57:59.280083+0000","last_fullsized":"2023-10-19T12:57:59.280083+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:58:13.879902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.288166+0000","last_change":"2023-10-19T12:57:59.281600+0000","last_active":"2023-10-19T12:57:59.288166+0000","last_peered":"2023-10-19T12:57:59.288166+0000","last_clean":"2023-10-19T12:57:59.288166+0000","last_became_active":"2023-10-19T12:57:59.281239+0000","last_became_peered":"2023-10-19T12:57:59.281239+0000","last_unstale":"2023-10-19T12:57:59.288166+0000","last_undegraded":"2023-10-19T12:57:59.288166+0000","last_fullsized":"2023-10-19T12:57:59.288166+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:47:41.975167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.279019+0000","last_change":"2023-10-19T12:57:59.277338+0000","last_active":"2023-10-19T12:57:59.279019+0000","last_peered":"2023-10-19T12:57:59.279019+0000","last_clean":"2023-10-19T12:57:59.279019+0000","last_became_active":"2023-10-19T12:57:59.277100+0000","last_became_peered":"2023-10-19T12:57:59.277100+0000","last_unstale":"2023-10-19T12:57:59.279019+0000","last_undegraded":"2023-10-19T12:57:59.279019+0000","last_fullsized":"2023-10-19T12:57:59.279019+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:47:41.975167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.286544+0000","last_change":"2023-10-19T12:57:59.280469+0000","last_active":"2023-10-19T12:57:59.286544+0000","last_peered":"2023-10-19T12:57:59.286544+0000","last_clean":"2023-10-19T12:57:59.286544+0000","last_became_active":"2023-10-19T12:57:59.280002+0000","last_became_peered":"2023-10-19T12:57:59.280002+0000","last_unstale":"2023-10-19T12:57:59.286544+0000","last_undegraded":"2023-10-19T12:57:59.286544+0000","last_fullsized":"2023-10-19T12:57:59.286544+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:54:04.339773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.272211+0000","last_change":"2023-10-19T12:57:59.261431+0000","last_active":"2023-10-19T12:58:00.272211+0000","last_peered":"2023-10-19T12:58:00.272211+0000","last_clean":"2023-10-19T12:58:00.272211+0000","last_became_active":"2023-10-19T12:57:59.261169+0000","last_became_peered":"2023-10-19T12:57:59.261169+0000","last_unstale":"2023-10-19T12:58:00.272211+0000","last_undegraded":"2023-10-19T12:58:00.272211+0000","last_fullsized":"2023-10-19T12:58:00.272211+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:50:32.740818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.278389+0000","last_change":"2023-10-19T12:57:59.272988+0000","last_active":"2023-10-19T12:57:59.278389+0000","last_peered":"2023-10-19T12:57:59.278389+0000","last_clean":"2023-10-19T12:57:59.278389+0000","last_became_active":"2023-10-19T12:57:59.272744+0000","last_became_peered":"2023-10-19T12:57:59.272744+0000","last_unstale":"2023-10-19T12:57:59.278389+0000","last_undegraded":"2023-10-19T12:57:59.278389+0000","last_fullsized":"2023-10-19T12:57:59.278389+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:37:57.451855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"69'4","reported_seq":15,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.287757+0000","last_change":"2023-10-19T12:57:58.252331+0000","last_active":"2023-10-19T12:57:59.287757+0000","last_peered":"2023-10-19T12:57:59.287757+0000","last_clean":"2023-10-19T12:57:59.287757+0000","last_became_active":"2023-10-19T12:57:58.251127+0000","last_became_peered":"2023-10-19T12:57:58.251127+0000","last_unstale":"2023-10-19T12:57:59.287757+0000","last_undegraded":"2023-10-19T12:57:59.287757+0000","last_fullsized":"2023-10-19T12:57:59.287757+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:43:25.389215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.272250+0000","last_change":"2023-10-19T12:57:59.264715+0000","last_active":"2023-10-19T12:58:00.272250+0000","last_peered":"2023-10-19T12:58:00.272250+0000","last_clean":"2023-10-19T12:58:00.272250+0000","last_became_active":"2023-10-19T12:57:59.264476+0000","last_became_peered":"2023-10-19T12:57:59.264476+0000","last_unstale":"2023-10-19T12:58:00.272250+0000","last_undegraded":"2023-10-19T12:58:00.272250+0000","last_fullsized":"2023-10-19T12:58:00.272250+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:06:20.535644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"69'1","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.276793+0000","last_change":"2023-10-19T12:57:58.256650+0000","last_active":"2023-10-19T12:58:00.276793+0000","last_peered":"2023-10-19T12:58:00.276793+0000","last_clean":"2023-10-19T12:58:00.276793+0000","last_became_active":"2023-10-19T12:57:58.256210+0000","last_became_peered":"2023-10-19T12:57:58.256210+0000","last_unstale":"2023-10-19T12:58:00.276793+0000","last_undegraded":"2023-10-19T12:58:00.276793+0000","last_fullsized":"2023-10-19T12:58:00.276793+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:19:19.078844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.279681+0000","last_change":"2023-10-19T12:57:59.269502+0000","last_active":"2023-10-19T12:57:59.279681+0000","last_peered":"2023-10-19T12:57:59.279681+0000","last_clean":"2023-10-19T12:57:59.279681+0000","last_became_active":"2023-10-19T12:57:59.268734+0000","last_became_peered":"2023-10-19T12:57:59.268734+0000","last_unstale":"2023-10-19T12:57:59.279681+0000","last_undegraded":"2023-10-19T12:57:59.279681+0000","last_fullsized":"2023-10-19T12:57:59.279681+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:29:08.762149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"69'2","reported_seq":14,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.277103+0000","last_change":"2023-10-19T12:57:58.255163+0000","last_active":"2023-10-19T12:58:00.277103+0000","last_peered":"2023-10-19T12:58:00.277103+0000","last_clean":"2023-10-19T12:58:00.277103+0000","last_became_active":"2023-10-19T12:57:58.254897+0000","last_became_peered":"2023-10-19T12:57:58.254897+0000","last_unstale":"2023-10-19T12:58:00.277103+0000","last_undegraded":"2023-10-19T12:58:00.277103+0000","last_fullsized":"2023-10-19T12:58:00.277103+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:48:56.789976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.274847+0000","last_change":"2023-10-19T12:57:59.261552+0000","last_active":"2023-10-19T12:58:00.274847+0000","last_peered":"2023-10-19T12:58:00.274847+0000","last_clean":"2023-10-19T12:58:00.274847+0000","last_became_active":"2023-10-19T12:57:59.261224+0000","last_became_peered":"2023-10-19T12:57:59.261224+0000","last_unstale":"2023-10-19T12:58:00.274847+0000","last_undegraded":"2023-10-19T12:58:00.274847+0000","last_fullsized":"2023-10-19T12:58:00.274847+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:38:57.879320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"69'2","reported_seq":14,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.272345+0000","last_change":"2023-10-19T12:57:58.255648+0000","last_active":"2023-10-19T12:58:00.272345+0000","last_peered":"2023-10-19T12:58:00.272345+0000","last_clean":"2023-10-19T12:58:00.272345+0000","last_became_active":"2023-10-19T12:57:58.255149+0000","last_became_peered":"2023-10-19T12:57:58.255149+0000","last_unstale":"2023-10-19T12:58:00.272345+0000","last_undegraded":"2023-10-19T12:58:00.272345+0000","last_fullsized":"2023-10-19T12:58:00.272345+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:55:58.608631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.287223+0000","last_change":"2023-10-19T12:57:59.282345+0000","last_active":"2023-10-19T12:57:59.287223+0000","last_peered":"2023-10-19T12:57:59.287223+0000","last_clean":"2023-10-19T12:57:59.287223+0000","last_became_active":"2023-10-19T12:57:59.282022+0000","last_became_peered":"2023-10-19T12:57:59.282022+0000","last_unstale":"2023-10-19T12:57:59.287223+0000","last_undegraded":"2023-10-19T12:57:59.287223+0000","last_fullsized":"2023-10-19T12:57:59.287223+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:50:41.250507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"69'1","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.293130+0000","last_change":"2023-10-19T12:57:58.257095+0000","last_active":"2023-10-19T12:57:59.293130+0000","last_peered":"2023-10-19T12:57:59.293130+0000","last_clean":"2023-10-19T12:57:59.293130+0000","last_became_active":"2023-10-19T12:57:58.256684+0000","last_became_peered":"2023-10-19T12:57:58.256684+0000","last_unstale":"2023-10-19T12:57:59.293130+0000","last_undegraded":"2023-10-19T12:57:59.293130+0000","last_fullsized":"2023-10-19T12:57:59.293130+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:39:16.198486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.279487+0000","last_change":"2023-10-19T12:57:59.278382+0000","last_active":"2023-10-19T12:57:59.279487+0000","last_peered":"2023-10-19T12:57:59.279487+0000","last_clean":"2023-10-19T12:57:59.279487+0000","last_became_active":"2023-10-19T12:57:59.278100+0000","last_became_peered":"2023-10-19T12:57:59.278100+0000","last_unstale":"2023-10-19T12:57:59.279487+0000","last_undegraded":"2023-10-19T12:57:59.279487+0000","last_fullsized":"2023-10-19T12:57:59.279487+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:21:41.603580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"69'1","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.280435+0000","last_change":"2023-10-19T12:57:58.254342+0000","last_active":"2023-10-19T12:57:59.280435+0000","last_peered":"2023-10-19T12:57:59.280435+0000","last_clean":"2023-10-19T12:57:59.280435+0000","last_became_active":"2023-10-19T12:57:58.254053+0000","last_became_peered":"2023-10-19T12:57:58.254053+0000","last_unstale":"2023-10-19T12:57:59.280435+0000","last_undegraded":"2023-10-19T12:57:59.280435+0000","last_fullsized":"2023-10-19T12:57:59.280435+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:33:12.732833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.275130+0000","last_change":"2023-10-19T12:57:59.261739+0000","last_active":"2023-10-19T12:58:00.275130+0000","last_peered":"2023-10-19T12:58:00.275130+0000","last_clean":"2023-10-19T12:58:00.275130+0000","last_became_active":"2023-10-19T12:57:59.261380+0000","last_became_peered":"2023-10-19T12:57:59.261380+0000","last_unstale":"2023-10-19T12:58:00.275130+0000","last_undegraded":"2023-10-19T12:58:00.275130+0000","last_fullsized":"2023-10-19T12:58:00.275130+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:12:33.439060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":11,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.262572+0000","last_change":"2023-10-19T12:57:58.252697+0000","last_active":"2023-10-19T12:57:59.262572+0000","last_peered":"2023-10-19T12:57:59.262572+0000","last_clean":"2023-10-19T12:57:59.262572+0000","last_became_active":"2023-10-19T12:57:58.252422+0000","last_became_peered":"2023-10-19T12:57:58.252422+0000","last_unstale":"2023-10-19T12:57:59.262572+0000","last_undegraded":"2023-10-19T12:57:59.262572+0000","last_fullsized":"2023-10-19T12:57:59.262572+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:26:25.065741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.281544+0000","last_change":"2023-10-19T12:57:59.280236+0000","last_active":"2023-10-19T12:57:59.281544+0000","last_peered":"2023-10-19T12:57:59.281544+0000","last_clean":"2023-10-19T12:57:59.281544+0000","last_became_active":"2023-10-19T12:57:59.279806+0000","last_became_peered":"2023-10-19T12:57:59.279806+0000","last_unstale":"2023-10-19T12:57:59.281544+0000","last_undegraded":"2023-10-19T12:57:59.281544+0000","last_fullsized":"2023-10-19T12:57:59.281544+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:32:09.277054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":11,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.266101+0000","last_change":"2023-10-19T12:57:58.259037+0000","last_active":"2023-10-19T12:57:59.266101+0000","last_peered":"2023-10-19T12:57:59.266101+0000","last_clean":"2023-10-19T12:57:59.266101+0000","last_became_active":"2023-10-19T12:57:58.258745+0000","last_became_peered":"2023-10-19T12:57:58.258745+0000","last_unstale":"2023-10-19T12:57:59.266101+0000","last_undegraded":"2023-10-19T12:57:59.266101+0000","last_fullsized":"2023-10-19T12:57:59.266101+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:29:50.196136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.270176+0000","last_change":"2023-10-19T12:57:59.264886+0000","last_active":"2023-10-19T12:57:59.270176+0000","last_peered":"2023-10-19T12:57:59.270176+0000","last_clean":"2023-10-19T12:57:59.270176+0000","last_became_active":"2023-10-19T12:57:59.264635+0000","last_became_peered":"2023-10-19T12:57:59.264635+0000","last_unstale":"2023-10-19T12:57:59.270176+0000","last_undegraded":"2023-10-19T12:57:59.270176+0000","last_fullsized":"2023-10-19T12:57:59.270176+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:19:20.089249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":11,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.261596+0000","last_change":"2023-10-19T12:57:58.251390+0000","last_active":"2023-10-19T12:57:59.261596+0000","last_peered":"2023-10-19T12:57:59.261596+0000","last_clean":"2023-10-19T12:57:59.261596+0000","last_became_active":"2023-10-19T12:57:58.251120+0000","last_became_peered":"2023-10-19T12:57:58.251120+0000","last_unstale":"2023-10-19T12:57:59.261596+0000","last_undegraded":"2023-10-19T12:57:59.261596+0000","last_fullsized":"2023-10-19T12:57:59.261596+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:23:36.269955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.272416+0000","last_change":"2023-10-19T12:57:59.270585+0000","last_active":"2023-10-19T12:58:00.272416+0000","last_peered":"2023-10-19T12:58:00.272416+0000","last_clean":"2023-10-19T12:58:00.272416+0000","last_became_active":"2023-10-19T12:57:59.270313+0000","last_became_peered":"2023-10-19T12:57:59.270313+0000","last_unstale":"2023-10-19T12:58:00.272416+0000","last_undegraded":"2023-10-19T12:58:00.272416+0000","last_fullsized":"2023-10-19T12:58:00.272416+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:41:54.210777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":11,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.268616+0000","last_change":"2023-10-19T12:57:58.257516+0000","last_active":"2023-10-19T12:57:59.268616+0000","last_peered":"2023-10-19T12:57:59.268616+0000","last_clean":"2023-10-19T12:57:59.268616+0000","last_became_active":"2023-10-19T12:57:58.257217+0000","last_became_peered":"2023-10-19T12:57:58.257217+0000","last_unstale":"2023-10-19T12:57:59.268616+0000","last_undegraded":"2023-10-19T12:57:59.268616+0000","last_fullsized":"2023-10-19T12:57:59.268616+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:13:05.939605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.288065+0000","last_change":"2023-10-19T12:57:59.282205+0000","last_active":"2023-10-19T12:57:59.288065+0000","last_peered":"2023-10-19T12:57:59.288065+0000","last_clean":"2023-10-19T12:57:59.288065+0000","last_became_active":"2023-10-19T12:57:59.281923+0000","last_became_peered":"2023-10-19T12:57:59.281923+0000","last_unstale":"2023-10-19T12:57:59.288065+0000","last_undegraded":"2023-10-19T12:57:59.288065+0000","last_fullsized":"2023-10-19T12:57:59.288065+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:32:27.978530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"69'3","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.289447+0000","last_change":"2023-10-19T12:57:58.255428+0000","last_active":"2023-10-19T12:57:59.289447+0000","last_peered":"2023-10-19T12:57:59.289447+0000","last_clean":"2023-10-19T12:57:59.289447+0000","last_became_active":"2023-10-19T12:57:58.255138+0000","last_became_peered":"2023-10-19T12:57:58.255138+0000","last_unstale":"2023-10-19T12:57:59.289447+0000","last_undegraded":"2023-10-19T12:57:59.289447+0000","last_fullsized":"2023-10-19T12:57:59.289447+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:40:07.021492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.281230+0000","last_change":"2023-10-19T12:57:59.279730+0000","last_active":"2023-10-19T12:57:59.281230+0000","last_peered":"2023-10-19T12:57:59.281230+0000","last_clean":"2023-10-19T12:57:59.281230+0000","last_became_active":"2023-10-19T12:57:59.279463+0000","last_became_peered":"2023-10-19T12:57:59.279463+0000","last_unstale":"2023-10-19T12:57:59.281230+0000","last_undegraded":"2023-10-19T12:57:59.281230+0000","last_fullsized":"2023-10-19T12:57:59.281230+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:36:39.520498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"69'6","reported_seq":18,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.275275+0000","last_change":"2023-10-19T12:57:58.250143+0000","last_active":"2023-10-19T12:58:00.275275+0000","last_peered":"2023-10-19T12:58:00.275275+0000","last_clean":"2023-10-19T12:58:00.275275+0000","last_became_active":"2023-10-19T12:57:58.249866+0000","last_became_peered":"2023-10-19T12:57:58.249866+0000","last_unstale":"2023-10-19T12:58:00.275275+0000","last_undegraded":"2023-10-19T12:58:00.275275+0000","last_fullsized":"2023-10-19T12:58:00.275275+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:29:34.406278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.272574+0000","last_change":"2023-10-19T12:57:59.266797+0000","last_active":"2023-10-19T12:58:00.272574+0000","last_peered":"2023-10-19T12:58:00.272574+0000","last_clean":"2023-10-19T12:58:00.272574+0000","last_became_active":"2023-10-19T12:57:59.266142+0000","last_became_peered":"2023-10-19T12:57:59.266142+0000","last_unstale":"2023-10-19T12:58:00.272574+0000","last_undegraded":"2023-10-19T12:58:00.272574+0000","last_fullsized":"2023-10-19T12:58:00.272574+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:40:50.284709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"69'2","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.290333+0000","last_change":"2023-10-19T12:57:58.257253+0000","last_active":"2023-10-19T12:57:59.290333+0000","last_peered":"2023-10-19T12:57:59.290333+0000","last_clean":"2023-10-19T12:57:59.290333+0000","last_became_active":"2023-10-19T12:57:58.256982+0000","last_became_peered":"2023-10-19T12:57:58.256982+0000","last_unstale":"2023-10-19T12:57:59.290333+0000","last_undegraded":"2023-10-19T12:57:59.290333+0000","last_fullsized":"2023-10-19T12:57:59.290333+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:26:46.604704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.288024+0000","last_change":"2023-10-19T12:57:59.282874+0000","last_active":"2023-10-19T12:57:59.288024+0000","last_peered":"2023-10-19T12:57:59.288024+0000","last_clean":"2023-10-19T12:57:59.288024+0000","last_became_active":"2023-10-19T12:57:59.282521+0000","last_became_peered":"2023-10-19T12:57:59.282521+0000","last_unstale":"2023-10-19T12:57:59.288024+0000","last_undegraded":"2023-10-19T12:57:59.288024+0000","last_fullsized":"2023-10-19T12:57:59.288024+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:42:43.566749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":11,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.260671+0000","last_change":"2023-10-19T12:57:58.259423+0000","last_active":"2023-10-19T12:57:59.260671+0000","last_peered":"2023-10-19T12:57:59.260671+0000","last_clean":"2023-10-19T12:57:59.260671+0000","last_became_active":"2023-10-19T12:57:58.259165+0000","last_became_peered":"2023-10-19T12:57:58.259165+0000","last_unstale":"2023-10-19T12:57:59.260671+0000","last_undegraded":"2023-10-19T12:57:59.260671+0000","last_fullsized":"2023-10-19T12:57:59.260671+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:46:53.191615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.275075+0000","last_change":"2023-10-19T12:57:59.267621+0000","last_active":"2023-10-19T12:58:00.275075+0000","last_peered":"2023-10-19T12:58:00.275075+0000","last_clean":"2023-10-19T12:58:00.275075+0000","last_became_active":"2023-10-19T12:57:59.267351+0000","last_became_peered":"2023-10-19T12:57:59.267351+0000","last_unstale":"2023-10-19T12:58:00.275075+0000","last_undegraded":"2023-10-19T12:58:00.275075+0000","last_fullsized":"2023-10-19T12:58:00.275075+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:53:20.624726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":11,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.261870+0000","last_change":"2023-10-19T12:57:58.254432+0000","last_active":"2023-10-19T12:57:59.261870+0000","last_peered":"2023-10-19T12:57:59.261870+0000","last_clean":"2023-10-19T12:57:59.261870+0000","last_became_active":"2023-10-19T12:57:58.254162+0000","last_became_peered":"2023-10-19T12:57:58.254162+0000","last_unstale":"2023-10-19T12:57:59.261870+0000","last_undegraded":"2023-10-19T12:57:59.261870+0000","last_fullsized":"2023-10-19T12:57:59.261870+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:38:37.380677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.287695+0000","last_change":"2023-10-19T12:57:59.278651+0000","last_active":"2023-10-19T12:57:59.287695+0000","last_peered":"2023-10-19T12:57:59.287695+0000","last_clean":"2023-10-19T12:57:59.287695+0000","last_became_active":"2023-10-19T12:57:59.278113+0000","last_became_peered":"2023-10-19T12:57:59.278113+0000","last_unstale":"2023-10-19T12:57:59.287695+0000","last_undegraded":"2023-10-19T12:57:59.287695+0000","last_fullsized":"2023-10-19T12:57:59.287695+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:12:20.974872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"69'8","reported_seq":20,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.275013+0000","last_change":"2023-10-19T12:57:58.247030+0000","last_active":"2023-10-19T12:58:00.275013+0000","last_peered":"2023-10-19T12:58:00.275013+0000","last_clean":"2023-10-19T12:58:00.275013+0000","last_became_active":"2023-10-19T12:57:58.246777+0000","last_became_peered":"2023-10-19T12:57:58.246777+0000","last_unstale":"2023-10-19T12:58:00.275013+0000","last_undegraded":"2023-10-19T12:58:00.275013+0000","last_fullsized":"2023-10-19T12:58:00.275013+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:24:54.236739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.275193+0000","last_change":"2023-10-19T12:57:59.262682+0000","last_active":"2023-10-19T12:58:00.275193+0000","last_peered":"2023-10-19T12:58:00.275193+0000","last_clean":"2023-10-19T12:58:00.275193+0000","last_became_active":"2023-10-19T12:57:59.262360+0000","last_became_peered":"2023-10-19T12:57:59.262360+0000","last_unstale":"2023-10-19T12:58:00.275193+0000","last_undegraded":"2023-10-19T12:58:00.275193+0000","last_fullsized":"2023-10-19T12:58:00.275193+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:22:59.082914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"69'1","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.292498+0000","last_change":"2023-10-19T12:57:58.257724+0000","last_active":"2023-10-19T12:57:59.292498+0000","last_peered":"2023-10-19T12:57:59.292498+0000","last_clean":"2023-10-19T12:57:59.292498+0000","last_became_active":"2023-10-19T12:57:58.257432+0000","last_became_peered":"2023-10-19T12:57:58.257432+0000","last_unstale":"2023-10-19T12:57:59.292498+0000","last_undegraded":"2023-10-19T12:57:59.292498+0000","last_fullsized":"2023-10-19T12:57:59.292498+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:21:40.593175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.276683+0000","last_change":"2023-10-19T12:57:59.274011+0000","last_active":"2023-10-19T12:58:00.276683+0000","last_peered":"2023-10-19T12:58:00.276683+0000","last_clean":"2023-10-19T12:58:00.276683+0000","last_became_active":"2023-10-19T12:57:59.273742+0000","last_became_peered":"2023-10-19T12:57:59.273742+0000","last_unstale":"2023-10-19T12:58:00.276683+0000","last_undegraded":"2023-10-19T12:58:00.276683+0000","last_fullsized":"2023-10-19T12:58:00.276683+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:36:39.520498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"69'2","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.288309+0000","last_change":"2023-10-19T12:57:58.249377+0000","last_active":"2023-10-19T12:57:59.288309+0000","last_peered":"2023-10-19T12:57:59.288309+0000","last_clean":"2023-10-19T12:57:59.288309+0000","last_became_active":"2023-10-19T12:57:58.249095+0000","last_became_peered":"2023-10-19T12:57:58.249095+0000","last_unstale":"2023-10-19T12:57:59.288309+0000","last_undegraded":"2023-10-19T12:57:59.288309+0000","last_fullsized":"2023-10-19T12:57:59.288309+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:54:31.441800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.287176+0000","last_change":"2023-10-19T12:57:59.282010+0000","last_active":"2023-10-19T12:57:59.287176+0000","last_peered":"2023-10-19T12:57:59.287176+0000","last_clean":"2023-10-19T12:57:59.287176+0000","last_became_active":"2023-10-19T12:57:59.281731+0000","last_became_peered":"2023-10-19T12:57:59.281731+0000","last_unstale":"2023-10-19T12:57:59.287176+0000","last_undegraded":"2023-10-19T12:57:59.287176+0000","last_fullsized":"2023-10-19T12:57:59.287176+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:19:17.421557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"69'1","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.290864+0000","last_change":"2023-10-19T12:57:58.255311+0000","last_active":"2023-10-19T12:57:59.290864+0000","last_peered":"2023-10-19T12:57:59.290864+0000","last_clean":"2023-10-19T12:57:59.290864+0000","last_became_active":"2023-10-19T12:57:58.255029+0000","last_became_peered":"2023-10-19T12:57:58.255029+0000","last_unstale":"2023-10-19T12:57:59.290864+0000","last_undegraded":"2023-10-19T12:57:59.290864+0000","last_fullsized":"2023-10-19T12:57:59.290864+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:48:56.789976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.276881+0000","last_change":"2023-10-19T12:57:59.277839+0000","last_active":"2023-10-19T12:58:00.276881+0000","last_peered":"2023-10-19T12:58:00.276881+0000","last_clean":"2023-10-19T12:58:00.276881+0000","last_became_active":"2023-10-19T12:57:59.277539+0000","last_became_peered":"2023-10-19T12:57:59.277539+0000","last_unstale":"2023-10-19T12:58:00.276881+0000","last_undegraded":"2023-10-19T12:58:00.276881+0000","last_fullsized":"2023-10-19T12:58:00.276881+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:47:49.700629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":11,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.264273+0000","last_change":"2023-10-19T12:57:58.256158+0000","last_active":"2023-10-19T12:57:59.264273+0000","last_peered":"2023-10-19T12:57:59.264273+0000","last_clean":"2023-10-19T12:57:59.264273+0000","last_became_active":"2023-10-19T12:57:58.255870+0000","last_became_peered":"2023-10-19T12:57:58.255870+0000","last_unstale":"2023-10-19T12:57:59.264273+0000","last_undegraded":"2023-10-19T12:57:59.264273+0000","last_fullsized":"2023-10-19T12:57:59.264273+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:55:15.770193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.272249+0000","last_change":"2023-10-19T12:57:59.273663+0000","last_active":"2023-10-19T12:58:00.272249+0000","last_peered":"2023-10-19T12:58:00.272249+0000","last_clean":"2023-10-19T12:58:00.272249+0000","last_became_active":"2023-10-19T12:57:59.273411+0000","last_became_peered":"2023-10-19T12:57:59.273411+0000","last_unstale":"2023-10-19T12:58:00.272249+0000","last_undegraded":"2023-10-19T12:58:00.272249+0000","last_fullsized":"2023-10-19T12:58:00.272249+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:43:58.424456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"69'2","reported_seq":14,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.276952+0000","last_change":"2023-10-19T12:57:58.254673+0000","last_active":"2023-10-19T12:58:00.276952+0000","last_peered":"2023-10-19T12:58:00.276952+0000","last_clean":"2023-10-19T12:58:00.276952+0000","last_became_active":"2023-10-19T12:57:58.254408+0000","last_became_peered":"2023-10-19T12:57:58.254408+0000","last_unstale":"2023-10-19T12:58:00.276952+0000","last_undegraded":"2023-10-19T12:58:00.276952+0000","last_fullsized":"2023-10-19T12:58:00.276952+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:25:49.650520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.281891+0000","last_change":"2023-10-19T12:57:59.279156+0000","last_active":"2023-10-19T12:57:59.281891+0000","last_peered":"2023-10-19T12:57:59.281891+0000","last_clean":"2023-10-19T12:57:59.281891+0000","last_became_active":"2023-10-19T12:57:59.278885+0000","last_became_peered":"2023-10-19T12:57:59.278885+0000","last_unstale":"2023-10-19T12:57:59.281891+0000","last_undegraded":"2023-10-19T12:57:59.281891+0000","last_fullsized":"2023-10-19T12:57:59.281891+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:43:35.936513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.274755+0000","last_change":"2023-10-19T12:57:58.252362+0000","last_active":"2023-10-19T12:58:00.274755+0000","last_peered":"2023-10-19T12:58:00.274755+0000","last_clean":"2023-10-19T12:58:00.274755+0000","last_became_active":"2023-10-19T12:57:58.252084+0000","last_became_peered":"2023-10-19T12:57:58.252084+0000","last_unstale":"2023-10-19T12:58:00.274755+0000","last_undegraded":"2023-10-19T12:58:00.274755+0000","last_fullsized":"2023-10-19T12:58:00.274755+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:56:55.149376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.289238+0000","last_change":"2023-10-19T12:57:59.275897+0000","last_active":"2023-10-19T12:57:59.289238+0000","last_peered":"2023-10-19T12:57:59.289238+0000","last_clean":"2023-10-19T12:57:59.289238+0000","last_became_active":"2023-10-19T12:57:59.274965+0000","last_became_peered":"2023-10-19T12:57:59.274965+0000","last_unstale":"2023-10-19T12:57:59.289238+0000","last_undegraded":"2023-10-19T12:57:59.289238+0000","last_fullsized":"2023-10-19T12:57:59.289238+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:30:55.579208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":11,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.267824+0000","last_change":"2023-10-19T12:57:58.255960+0000","last_active":"2023-10-19T12:57:59.267824+0000","last_peered":"2023-10-19T12:57:59.267824+0000","last_clean":"2023-10-19T12:57:59.267824+0000","last_became_active":"2023-10-19T12:57:58.255655+0000","last_became_peered":"2023-10-19T12:57:58.255655+0000","last_unstale":"2023-10-19T12:57:59.267824+0000","last_undegraded":"2023-10-19T12:57:59.267824+0000","last_fullsized":"2023-10-19T12:57:59.267824+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:35:32.478337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.276919+0000","last_change":"2023-10-19T12:57:59.284564+0000","last_active":"2023-10-19T12:58:00.276919+0000","last_peered":"2023-10-19T12:58:00.276919+0000","last_clean":"2023-10-19T12:58:00.276919+0000","last_became_active":"2023-10-19T12:57:59.278312+0000","last_became_peered":"2023-10-19T12:57:59.278312+0000","last_unstale":"2023-10-19T12:58:00.276919+0000","last_undegraded":"2023-10-19T12:58:00.276919+0000","last_fullsized":"2023-10-19T12:58:00.276919+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:19:34.001505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":11,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.255502+0000","last_change":"2023-10-19T12:57:58.248153+0000","last_active":"2023-10-19T12:57:59.255502+0000","last_peered":"2023-10-19T12:57:59.255502+0000","last_clean":"2023-10-19T12:57:59.255502+0000","last_became_active":"2023-10-19T12:57:58.247864+0000","last_became_peered":"2023-10-19T12:57:58.247864+0000","last_unstale":"2023-10-19T12:57:59.255502+0000","last_undegraded":"2023-10-19T12:57:59.255502+0000","last_fullsized":"2023-10-19T12:57:59.255502+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:53:21.688950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.272507+0000","last_change":"2023-10-19T12:57:59.260621+0000","last_active":"2023-10-19T12:58:00.272507+0000","last_peered":"2023-10-19T12:58:00.272507+0000","last_clean":"2023-10-19T12:58:00.272507+0000","last_became_active":"2023-10-19T12:57:59.260352+0000","last_became_peered":"2023-10-19T12:57:59.260352+0000","last_unstale":"2023-10-19T12:58:00.272507+0000","last_undegraded":"2023-10-19T12:58:00.272507+0000","last_fullsized":"2023-10-19T12:58:00.272507+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:56:56.159781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"69'2","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.290867+0000","last_change":"2023-10-19T12:57:58.259426+0000","last_active":"2023-10-19T12:57:59.290867+0000","last_peered":"2023-10-19T12:57:59.290867+0000","last_clean":"2023-10-19T12:57:59.290867+0000","last_became_active":"2023-10-19T12:57:58.259118+0000","last_became_peered":"2023-10-19T12:57:58.259118+0000","last_unstale":"2023-10-19T12:57:59.290867+0000","last_undegraded":"2023-10-19T12:57:59.290867+0000","last_fullsized":"2023-10-19T12:57:59.290867+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:07:09.641522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.276817+0000","last_change":"2023-10-19T12:57:59.271932+0000","last_active":"2023-10-19T12:58:00.276817+0000","last_peered":"2023-10-19T12:58:00.276817+0000","last_clean":"2023-10-19T12:58:00.276817+0000","last_became_active":"2023-10-19T12:57:59.271620+0000","last_became_peered":"2023-10-19T12:57:59.271620+0000","last_unstale":"2023-10-19T12:58:00.276817+0000","last_undegraded":"2023-10-19T12:58:00.276817+0000","last_fullsized":"2023-10-19T12:58:00.276817+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:13:06.950010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":11,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.262975+0000","last_change":"2023-10-19T12:57:58.253264+0000","last_active":"2023-10-19T12:57:59.262975+0000","last_peered":"2023-10-19T12:57:59.262975+0000","last_clean":"2023-10-19T12:57:59.262975+0000","last_became_active":"2023-10-19T12:57:58.252867+0000","last_became_peered":"2023-10-19T12:57:58.252867+0000","last_unstale":"2023-10-19T12:57:59.262975+0000","last_undegraded":"2023-10-19T12:57:59.262975+0000","last_fullsized":"2023-10-19T12:57:59.262975+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:08:22.862915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.274916+0000","last_change":"2023-10-19T12:57:59.274237+0000","last_active":"2023-10-19T12:58:00.274916+0000","last_peered":"2023-10-19T12:58:00.274916+0000","last_clean":"2023-10-19T12:58:00.274916+0000","last_became_active":"2023-10-19T12:57:59.273873+0000","last_became_peered":"2023-10-19T12:57:59.273873+0000","last_unstale":"2023-10-19T12:58:00.274916+0000","last_undegraded":"2023-10-19T12:58:00.274916+0000","last_fullsized":"2023-10-19T12:58:00.274916+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:31:14.459655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":11,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.265565+0000","last_change":"2023-10-19T12:57:58.253449+0000","last_active":"2023-10-19T12:57:59.265565+0000","last_peered":"2023-10-19T12:57:59.265565+0000","last_clean":"2023-10-19T12:57:59.265565+0000","last_became_active":"2023-10-19T12:57:58.253169+0000","last_became_peered":"2023-10-19T12:57:58.253169+0000","last_unstale":"2023-10-19T12:57:59.265565+0000","last_undegraded":"2023-10-19T12:57:59.265565+0000","last_fullsized":"2023-10-19T12:57:59.265565+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:55:53.005401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.287287+0000","last_change":"2023-10-19T12:57:59.280611+0000","last_active":"2023-10-19T12:57:59.287287+0000","last_peered":"2023-10-19T12:57:59.287287+0000","last_clean":"2023-10-19T12:57:59.287287+0000","last_became_active":"2023-10-19T12:57:59.280321+0000","last_became_peered":"2023-10-19T12:57:59.280321+0000","last_unstale":"2023-10-19T12:57:59.287287+0000","last_undegraded":"2023-10-19T12:57:59.287287+0000","last_fullsized":"2023-10-19T12:57:59.287287+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:13:46.024291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.274940+0000","last_change":"2023-10-19T12:57:58.246012+0000","last_active":"2023-10-19T12:58:00.274940+0000","last_peered":"2023-10-19T12:58:00.274940+0000","last_clean":"2023-10-19T12:58:00.274940+0000","last_became_active":"2023-10-19T12:57:58.245739+0000","last_became_peered":"2023-10-19T12:57:58.245739+0000","last_unstale":"2023-10-19T12:58:00.274940+0000","last_undegraded":"2023-10-19T12:58:00.274940+0000","last_fullsized":"2023-10-19T12:58:00.274940+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:50:31.730413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.272446+0000","last_change":"2023-10-19T12:57:59.265660+0000","last_active":"2023-10-19T12:58:00.272446+0000","last_peered":"2023-10-19T12:58:00.272446+0000","last_clean":"2023-10-19T12:58:00.272446+0000","last_became_active":"2023-10-19T12:57:59.265334+0000","last_became_peered":"2023-10-19T12:57:59.265334+0000","last_unstale":"2023-10-19T12:58:00.272446+0000","last_undegraded":"2023-10-19T12:58:00.272446+0000","last_fullsized":"2023-10-19T12:58:00.272446+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:28:49.136836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.274837+0000","last_change":"2023-10-19T12:57:58.255022+0000","last_active":"2023-10-19T12:58:00.274837+0000","last_peered":"2023-10-19T12:58:00.274837+0000","last_clean":"2023-10-19T12:58:00.274837+0000","last_became_active":"2023-10-19T12:57:58.254655+0000","last_became_peered":"2023-10-19T12:57:58.254655+0000","last_unstale":"2023-10-19T12:58:00.274837+0000","last_undegraded":"2023-10-19T12:58:00.274837+0000","last_fullsized":"2023-10-19T12:58:00.274837+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:11:37.863957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.289255+0000","last_change":"2023-10-19T12:57:59.279293+0000","last_active":"2023-10-19T12:57:59.289255+0000","last_peered":"2023-10-19T12:57:59.289255+0000","last_clean":"2023-10-19T12:57:59.289255+0000","last_became_active":"2023-10-19T12:57:59.278399+0000","last_became_peered":"2023-10-19T12:57:59.278399+0000","last_unstale":"2023-10-19T12:57:59.289255+0000","last_undegraded":"2023-10-19T12:57:59.289255+0000","last_fullsized":"2023-10-19T12:57:59.289255+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:18:02.353710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":11,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.268350+0000","last_change":"2023-10-19T12:57:58.255022+0000","last_active":"2023-10-19T12:57:59.268350+0000","last_peered":"2023-10-19T12:57:59.268350+0000","last_clean":"2023-10-19T12:57:59.268350+0000","last_became_active":"2023-10-19T12:57:58.254610+0000","last_became_peered":"2023-10-19T12:57:58.254610+0000","last_unstale":"2023-10-19T12:57:59.268350+0000","last_undegraded":"2023-10-19T12:57:59.268350+0000","last_fullsized":"2023-10-19T12:57:59.268350+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:13:09.184912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.275180+0000","last_change":"2023-10-19T12:57:59.261263+0000","last_active":"2023-10-19T12:58:00.275180+0000","last_peered":"2023-10-19T12:58:00.275180+0000","last_clean":"2023-10-19T12:58:00.275180+0000","last_became_active":"2023-10-19T12:57:59.260933+0000","last_became_peered":"2023-10-19T12:57:59.260933+0000","last_unstale":"2023-10-19T12:58:00.275180+0000","last_undegraded":"2023-10-19T12:58:00.275180+0000","last_fullsized":"2023-10-19T12:58:00.275180+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:56:19.223989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.272344+0000","last_change":"2023-10-19T12:57:58.255799+0000","last_active":"2023-10-19T12:58:00.272344+0000","last_peered":"2023-10-19T12:58:00.272344+0000","last_clean":"2023-10-19T12:58:00.272344+0000","last_became_active":"2023-10-19T12:57:58.255531+0000","last_became_peered":"2023-10-19T12:57:58.255531+0000","last_unstale":"2023-10-19T12:58:00.272344+0000","last_undegraded":"2023-10-19T12:58:00.272344+0000","last_fullsized":"2023-10-19T12:58:00.272344+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:03:06.233858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.266105+0000","last_change":"2023-10-19T12:57:59.264409+0000","last_active":"2023-10-19T12:57:59.266105+0000","last_peered":"2023-10-19T12:57:59.266105+0000","last_clean":"2023-10-19T12:57:59.266105+0000","last_became_active":"2023-10-19T12:57:59.263748+0000","last_became_peered":"2023-10-19T12:57:59.263748+0000","last_unstale":"2023-10-19T12:57:59.266105+0000","last_undegraded":"2023-10-19T12:57:59.266105+0000","last_fullsized":"2023-10-19T12:57:59.266105+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:03:18.036820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":11,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.264710+0000","last_change":"2023-10-19T12:57:58.254016+0000","last_active":"2023-10-19T12:57:59.264710+0000","last_peered":"2023-10-19T12:57:59.264710+0000","last_clean":"2023-10-19T12:57:59.264710+0000","last_became_active":"2023-10-19T12:57:58.253756+0000","last_became_peered":"2023-10-19T12:57:58.253756+0000","last_unstale":"2023-10-19T12:57:59.264710+0000","last_undegraded":"2023-10-19T12:57:59.264710+0000","last_fullsized":"2023-10-19T12:57:59.264710+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:52:42.753982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"58'139","reported_seq":290,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-19T12:57:58.244379+0000","last_change":"2023-10-19T12:44:08.636613+0000","last_active":"2023-10-19T12:57:58.244379+0000","last_peered":"2023-10-19T12:57:58.244379+0000","last_clean":"2023-10-19T12:57:58.244379+0000","last_became_active":"2023-10-19T12:44:08.635697+0000","last_became_peered":"2023-10-19T12:44:08.635697+0000","last_unstale":"2023-10-19T12:57:58.244379+0000","last_undegraded":"2023-10-19T12:57:58.244379+0000","last_fullsized":"2023-10-19T12:57:58.244379+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_clean_scrub_stamp":"2023-10-19T12:44:07.399738+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:15:23.685485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575018,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58948,"kb_used_data":1396,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312892,"statfs":{"total":96636764160,"available":96576401408,"internally_reserved":0,"allocated":1429504,"data_stored":1000030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58915275},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:57:38 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.58499999999999996,"15min":0.56100000000000005},"min":{"1min":0.39300000000000002,"5min":0.34499999999999997,"15min":0.248},"max":{"1min":0.67600000000000005,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.496},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.59699999999999998,"15min":0.57799999999999996},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.26900000000000002},"max":{"1min":0.76100000000000001,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.47499999999999998}]},{"osd":1,"last update":"Thu Oct 19 12:57:38 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.58699999999999997,"15min":0.56899999999999995},"min":{"1min":0.40300000000000002,"5min":0.371,"15min":0.28199999999999997},"max":{"1min":0.747,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.59699999999999998,"15min":0.57299999999999995},"min":{"1min":0.46200000000000002,"5min":0.34999999999999998,"15min":0.29299999999999998},"max":{"1min":0.66700000000000004,"5min":1.284,"15min":1.284},"last":0.63700000000000001}]},{"osd":2,"last update":"Thu Oct 19 12:57:38 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.63900000000000001,"15min":0.61799999999999999},"min":{"1min":0.434,"5min":0.41699999999999998,"15min":0.375},"max":{"1min":0.78600000000000003,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.65500000000000003,"15min":0.629},"min":{"1min":0.497,"5min":0.44600000000000001,"15min":0.28399999999999997},"max":{"1min":0.80700000000000005,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.59799999999999998}]},{"osd":3,"last update":"Thu Oct 19 12:57:38 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.61899999999999999,"15min":0.59499999999999997},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34399999999999997},"max":{"1min":0.79100000000000004,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.66000000000000003,"15min":0.64700000000000002},"min":{"1min":0.45300000000000001,"5min":0.44400000000000001,"15min":0.39700000000000002},"max":{"1min":0.77900000000000003,"5min":1.329,"15min":1.329},"last":0.68000000000000005}]},{"osd":4,"last update":"Thu Oct 19 12:57:38 2023","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.46899999999999997,"15min":0.46600000000000003},"min":{"1min":0.28599999999999998,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.73399999999999999,"5min":0.75800000000000001,"15min":0.80500000000000005},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.46800000000000003,"15min":0.46999999999999997},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.627,"5min":0.71999999999999997,"15min":0.78900000000000003},"last":0.61799999999999999}]},{"osd":5,"last update":"Thu Oct 19 12:57:38 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.48199999999999998,"15min":0.48799999999999999},"min":{"1min":0.30499999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.75800000000000001,"5min":0.80600000000000005,"15min":0.90400000000000003},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.52300000000000002,"15min":0.51400000000000001},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.308},"max":{"1min":0.77400000000000002,"5min":0.81999999999999995,"15min":0.92200000000000004},"last":0.55400000000000005}]},{"osd":6,"last update":"Thu Oct 19 12:57:38 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54500000000000004,"15min":0.54400000000000004},"min":{"1min":0.34100000000000003,"5min":0.308,"15min":0.308},"max":{"1min":0.80400000000000005,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.53400000000000003,"15min":0.53200000000000003},"min":{"1min":0.30299999999999999,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.72599999999999998,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.64800000000000002}]}]},{"osd":6,"up_from":13,"seq":55834575018,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31996,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339844,"statfs":{"total":96636764160,"available":96604000256,"internally_reserved":0,"allocated":835584,"data_stored":410190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:57:21 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60299999999999998,"15min":0.58999999999999997},"min":{"1min":0.311,"5min":0.311,"15min":0.24399999999999999},"max":{"1min":0.76000000000000001,"5min":0.79900000000000004,"15min":0.81999999999999995},"last":0.621},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.60499999999999998,"15min":0.58399999999999996},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.77500000000000002,"5min":0.78000000000000003,"15min":0.81200000000000006},"last":0.60899999999999999}]},{"osd":1,"last update":"Thu Oct 19 12:57:21 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.626,"15min":0.60399999999999998},"min":{"1min":0.376,"5min":0.376,"15min":0.34799999999999998},"max":{"1min":0.80500000000000005,"5min":0.83299999999999996,"15min":0.93899999999999995},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.62,"15min":0.60599999999999998},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.28399999999999997},"max":{"1min":0.79400000000000004,"5min":0.82099999999999995,"15min":1.075},"last":0.66800000000000004}]},{"osd":2,"last update":"Thu Oct 19 12:57:21 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60799999999999998,"15min":0.58299999999999996},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.26600000000000001},"max":{"1min":0.748,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.63200000000000001,"15min":0.60599999999999998},"min":{"1min":0.40500000000000003,"5min":0.36099999999999999,"15min":0.28399999999999997},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.71299999999999997}]},{"osd":3,"last update":"Thu Oct 19 12:57:21 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.65100000000000002,"15min":0.625},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.32600000000000001},"max":{"1min":0.83099999999999996,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.64700000000000002,"15min":0.63500000000000001},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.81499999999999995,"5min":0.81699999999999995,"15min":1.0900000000000001},"last":0.59199999999999997}]},{"osd":4,"last update":"Thu Oct 19 12:57:21 2023","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.46300000000000002,"15min":0.47599999999999998},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":1.02},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.47799999999999998,"15min":0.49099999999999999},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.247},"max":{"1min":0.65400000000000003,"5min":0.752,"15min":0.80000000000000004},"last":0.49199999999999999}]},{"osd":5,"last update":"Thu Oct 19 12:57:21 2023","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.47899999999999998,"15min":0.48399999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.26600000000000001},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.505,"5min":0.51400000000000001,"15min":0.52900000000000003},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.312},"max":{"1min":0.78500000000000003,"5min":0.85699999999999998,"15min":0.98799999999999999},"last":0.73299999999999998}]},{"osd":7,"last update":"Thu Oct 19 12:57:21 2023","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.52600000000000002,"15min":0.53100000000000003},"min":{"1min":0.35699999999999998,"5min":0.314,"15min":0.28199999999999997},"max":{"1min":0.76100000000000001,"5min":0.82199999999999995,"15min":1.036},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.53900000000000003,"15min":0.55200000000000005},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":1.1299999999999999},"last":0.53300000000000003}]}]},{"osd":5,"up_from":13,"seq":55834575018,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31988,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339852,"statfs":{"total":96636764160,"available":96604008448,"internally_reserved":0,"allocated":827392,"data_stored":409538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:57:24 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.59199999999999997,"15min":0.58599999999999997},"min":{"1min":0.39800000000000002,"5min":0.26700000000000002,"15min":0.223},"max":{"1min":0.79400000000000004,"5min":0.82699999999999996,"15min":0.88800000000000001},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.60799999999999998,"15min":0.59099999999999997},"min":{"1min":0.39700000000000002,"5min":0.312,"15min":0.27800000000000002},"max":{"1min":0.72699999999999998,"5min":0.80400000000000005,"15min":0.874},"last":0.379}]},{"osd":1,"last update":"Thu Oct 19 12:57:24 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.60499999999999998,"15min":0.59599999999999997},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.32800000000000001},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.876},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.61299999999999999,"15min":0.59999999999999998},"min":{"1min":0.439,"5min":0.34699999999999998,"15min":0.34499999999999997},"max":{"1min":0.79500000000000004,"5min":0.80100000000000005,"15min":0.83499999999999996},"last":0.54200000000000004}]},{"osd":2,"last update":"Thu Oct 19 12:57:24 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59399999999999997,"15min":0.57899999999999996},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.307},"max":{"1min":0.70899999999999996,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.62,"15min":0.61399999999999999},"min":{"1min":0.47399999999999998,"5min":0.41999999999999998,"15min":0.33700000000000002},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.93600000000000005},"last":0.505}]},{"osd":3,"last update":"Thu Oct 19 12:57:24 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.60999999999999999,"15min":0.59999999999999998},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.39300000000000002},"max":{"1min":0.73199999999999998,"5min":0.77900000000000003,"15min":0.874},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63900000000000001,"15min":0.63400000000000001},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.41999999999999998},"max":{"1min":0.746,"5min":0.80400000000000005,"15min":0.89500000000000002},"last":0.48899999999999999}]},{"osd":4,"last update":"Thu Oct 19 12:57:24 2023","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.34499999999999997,"5min":0.32100000000000001,"15min":0.28899999999999998},"max":{"1min":0.66100000000000003,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.47099999999999997,"15min":0.47599999999999998},"min":{"1min":0.307,"5min":0.307,"15min":0.23899999999999999},"max":{"1min":0.75900000000000001,"5min":0.79400000000000004,"15min":0.80500000000000005},"last":0.35699999999999998}]},{"osd":6,"last update":"Thu Oct 19 12:57:24 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.47399999999999998,"15min":0.47899999999999998},"min":{"1min":0.34100000000000003,"5min":0.31900000000000001,"15min":0.25900000000000001},"max":{"1min":0.74399999999999999,"5min":0.77500000000000002,"15min":0.79300000000000004},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.50800000000000001,"15min":0.498},"min":{"1min":0.38500000000000001,"5min":0.33600000000000002,"15min":0.30399999999999999},"max":{"1min":0.71899999999999997,"5min":0.79100000000000004,"15min":0.81299999999999994},"last":0.52300000000000002}]},{"osd":7,"last update":"Thu Oct 19 12:57:24 2023","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.47999999999999998,"15min":0.49399999999999999},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.315},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71899999999999997},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.52700000000000002,"15min":0.53400000000000003},"min":{"1min":0.39700000000000002,"5min":0.34300000000000003,"15min":0.27500000000000002},"max":{"1min":0.77900000000000003,"5min":0.78100000000000003,"15min":0.78500000000000003},"last":0.45000000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575018,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31992,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339848,"statfs":{"total":96636764160,"available":96604004352,"internally_reserved":0,"allocated":831488,"data_stored":410120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:57:37 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.54900000000000004,"15min":0.55100000000000005},"min":{"1min":0.36499999999999999,"5min":0.32900000000000001,"15min":0.28899999999999998},"max":{"1min":0.72699999999999998,"5min":0.77400000000000002,"15min":0.84799999999999998},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55900000000000005,"15min":0.54800000000000004},"min":{"1min":0.38100000000000001,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.73999999999999999,"5min":0.77800000000000002,"15min":0.78200000000000003},"last":0.60799999999999998}]},{"osd":1,"last update":"Thu Oct 19 12:57:37 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.61099999999999999,"15min":0.60399999999999998},"min":{"1min":0.377,"5min":0.377,"15min":0.34799999999999998},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.86199999999999999},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.61799999999999999,"15min":0.61099999999999999},"min":{"1min":0.44800000000000001,"5min":0.40200000000000002,"15min":0.35799999999999998},"max":{"1min":0.78900000000000003,"5min":0.79500000000000004,"15min":0.873},"last":0.72799999999999998}]},{"osd":2,"last update":"Thu Oct 19 12:57:37 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.61599999999999999,"15min":0.60899999999999999},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.254},"max":{"1min":0.76000000000000001,"5min":0.85299999999999998,"15min":0.86599999999999999},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.60899999999999999,"15min":0.61699999999999999},"min":{"1min":0.44800000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.75,"5min":0.82999999999999996,"15min":0.85999999999999999},"last":0.70199999999999996}]},{"osd":3,"last update":"Thu Oct 19 12:57:37 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.58499999999999996,"15min":0.57499999999999996},"min":{"1min":0.35199999999999998,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.70199999999999996,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.64400000000000002,"15min":0.63400000000000001},"min":{"1min":0.45700000000000002,"5min":0.41299999999999998,"15min":0.37},"max":{"1min":0.77300000000000002,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.73999999999999999}]},{"osd":5,"last update":"Thu Oct 19 12:57:37 2023","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.437,"15min":0.45200000000000001},"min":{"1min":0.30599999999999999,"5min":0.29099999999999998,"15min":0.249},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.81299999999999994},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.45500000000000002,"15min":0.46400000000000002},"min":{"1min":0.32000000000000001,"5min":0.26400000000000001,"15min":0.23499999999999999},"max":{"1min":0.71999999999999997,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.46000000000000002}]},{"osd":6,"last update":"Thu Oct 19 12:57:37 2023","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45400000000000001,"15min":0.46000000000000002},"min":{"1min":0.32300000000000001,"5min":0.29499999999999998,"15min":0.26400000000000001},"max":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.78600000000000003},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.47899999999999998,"15min":0.505},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.27100000000000002},"max":{"1min":0.76800000000000002,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.57099999999999995}]},{"osd":7,"last update":"Thu Oct 19 12:57:37 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.49199999999999999,"15min":0.50900000000000001},"min":{"1min":0.38600000000000001,"5min":0.307,"15min":0.27500000000000002},"max":{"1min":0.71399999999999997,"5min":0.79100000000000004,"15min":0.80200000000000005},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.53900000000000003,"15min":0.55900000000000005},"min":{"1min":0.41799999999999998,"5min":0.32100000000000001,"15min":0.30199999999999999},"max":{"1min":0.745,"5min":0.80500000000000005,"15min":0.83499999999999996},"last":0.621}]}]},{"osd":3,"up_from":13,"seq":55834575017,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36168,"kb_used_data":824,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335672,"statfs":{"total":96636764160,"available":96599728128,"internally_reserved":0,"allocated":843776,"data_stored":411172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:57:22 2023","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.40300000000000002,"15min":0.39500000000000002},"min":{"1min":0.248,"5min":0.23599999999999999,"15min":0.20899999999999999},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":1.04},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.42299999999999999,"15min":0.40699999999999997},"min":{"1min":0.28399999999999997,"5min":0.248,"15min":0.245},"max":{"1min":0.57399999999999995,"5min":0.60999999999999999,"15min":0.64600000000000002},"last":0.28000000000000003}]},{"osd":1,"last update":"Thu Oct 19 12:57:22 2023","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.41599999999999998,"15min":0.40400000000000003},"min":{"1min":0.30599999999999999,"5min":0.26100000000000001,"15min":0.22500000000000001},"max":{"1min":0.58299999999999996,"5min":0.621,"15min":0.69899999999999995},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.46100000000000002,"15min":0.442},"min":{"1min":0.32000000000000001,"5min":0.29899999999999999,"15min":0.215},"max":{"1min":0.65200000000000002,"5min":0.65900000000000003,"15min":0.874},"last":0.33900000000000002}]},{"osd":2,"last update":"Thu Oct 19 12:57:22 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.46899999999999997,"15min":0.45900000000000002},"min":{"1min":0.34699999999999998,"5min":0.32500000000000001,"15min":0.26100000000000001},"max":{"1min":0.59399999999999997,"5min":0.65800000000000003,"15min":0.73599999999999999},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.498,"15min":0.48299999999999998},"min":{"1min":0.36299999999999999,"5min":0.34799999999999998,"15min":0.247},"max":{"1min":0.64000000000000001,"5min":0.72599999999999998,"15min":0.96499999999999997},"last":0.379}]},{"osd":4,"last update":"Thu Oct 19 12:57:22 2023","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.67700000000000005,"15min":0.65900000000000003},"min":{"1min":0.49399999999999999,"5min":0.45900000000000002,"15min":0.40300000000000002},"max":{"1min":0.90500000000000003,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.70199999999999996,"15min":0.68700000000000006},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.42099999999999999},"max":{"1min":0.91300000000000003,"5min":0.98999999999999999,"15min":0.99399999999999999},"last":0.78200000000000003}]},{"osd":5,"last update":"Thu Oct 19 12:57:22 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.66900000000000004,"15min":0.64900000000000002},"min":{"1min":0.495,"5min":0.495,"15min":0.40500000000000003},"max":{"1min":0.93899999999999995,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.65700000000000003,"15min":0.64500000000000002},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.35899999999999999},"max":{"1min":0.85899999999999999,"5min":0.97299999999999998,"15min":0.99099999999999999},"last":0.58799999999999997}]},{"osd":6,"last update":"Thu Oct 19 12:57:22 2023","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.66500000000000004,"15min":0.65400000000000003},"min":{"1min":0.439,"5min":0.439,"15min":0.41199999999999998},"max":{"1min":0.84699999999999998,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.68100000000000005,"15min":0.66700000000000004},"min":{"1min":0.55200000000000005,"5min":0.496,"15min":0.35899999999999999},"max":{"1min":0.86899999999999999,"5min":1.05,"15min":1.05},"last":0.71199999999999997}]},{"osd":7,"last update":"Thu Oct 19 12:57:22 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.67400000000000004,"15min":0.66500000000000004},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.39800000000000002},"max":{"1min":0.95699999999999996,"5min":1,"15min":1},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.68400000000000005,"15min":0.67100000000000004},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.88500000000000001,"5min":0.93400000000000005,"15min":0.97599999999999998},"last":0.73599999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575019,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54216,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317624,"statfs":{"total":96636764160,"available":96581246976,"internally_reserved":0,"allocated":839680,"data_stored":411150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:57:25 2023","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.38900000000000001,"15min":0.38500000000000001},"min":{"1min":0.254,"5min":0.20899999999999999,"15min":0.187},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.83999999999999997},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.376,"5min":0.41699999999999998,"15min":0.40100000000000002},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.57799999999999996,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.36199999999999999}]},{"osd":1,"last update":"Thu Oct 19 12:57:25 2023","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.40799999999999997,"15min":0.40000000000000002},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.57899999999999996,"5min":0.71099999999999997,"15min":0.80400000000000005},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.44,"15min":0.43099999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.23400000000000001},"max":{"1min":0.57599999999999996,"5min":0.63,"15min":0.78600000000000003},"last":0.32000000000000001}]},{"osd":3,"last update":"Thu Oct 19 12:57:25 2023","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.46700000000000003,"15min":0.45600000000000002},"min":{"1min":0.25900000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.623,"5min":0.68000000000000005,"15min":0.79600000000000004},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.497,"15min":0.48799999999999999},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.87},"last":0.375}]},{"osd":4,"last update":"Thu Oct 19 12:57:25 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.64800000000000002,"15min":0.64100000000000001},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":1.0569999999999999},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.66400000000000003,"15min":0.65500000000000003},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.32000000000000001},"max":{"1min":0.91200000000000003,"5min":0.92300000000000004,"15min":1.0780000000000001},"last":0.45400000000000001}]},{"osd":5,"last update":"Thu Oct 19 12:57:25 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.68100000000000005,"15min":0.66000000000000003},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.78600000000000003,"5min":0.96499999999999997,"15min":0.99399999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.67500000000000004,"15min":0.65700000000000003},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.90200000000000002,"5min":0.94599999999999995,"15min":0.97099999999999997},"last":0.46999999999999997}]},{"osd":6,"last update":"Thu Oct 19 12:57:25 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.67900000000000005,"15min":0.67200000000000004},"min":{"1min":0.441,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.86799999999999999,"5min":1.008,"15min":1.008},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.68200000000000005,"15min":0.66900000000000004},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.93899999999999995,"5min":0.995,"15min":0.995},"last":0.623}]},{"osd":7,"last update":"Thu Oct 19 12:57:25 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.69499999999999995,"15min":0.68400000000000005},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.36399999999999999},"max":{"1min":0.98999999999999999,"5min":1.0149999999999999,"15min":1.111},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.69799999999999995,"15min":0.68500000000000005},"min":{"1min":0.42099999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.89700000000000002,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.60599999999999998}]}]},{"osd":1,"up_from":13,"seq":55834575018,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32004,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339836,"statfs":{"total":96636764160,"available":96603992064,"internally_reserved":0,"allocated":835584,"data_stored":409698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:57:32 2023","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.39200000000000002,"15min":0.378},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.187},"max":{"1min":0.59099999999999997,"5min":0.61799999999999999,"15min":0.66200000000000003},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.443,"15min":0.42199999999999999},"min":{"1min":0.30499999999999999,"5min":0.28000000000000003,"15min":0.21199999999999999},"max":{"1min":0.63600000000000001,"5min":0.68799999999999994,"15min":0.69799999999999995},"last":0.33900000000000002}]},{"osd":2,"last update":"Thu Oct 19 12:57:32 2023","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.44600000000000001,"15min":0.42099999999999999},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.20100000000000001},"max":{"1min":0.56000000000000005,"5min":0.68999999999999995,"15min":0.69999999999999996},"last":0.374},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.45800000000000002,"15min":0.44400000000000001},"min":{"1min":0.34699999999999998,"5min":0.309,"15min":0.23699999999999999},"max":{"1min":0.60599999999999998,"5min":0.67500000000000004,"15min":0.755},"last":0.41299999999999998}]},{"osd":3,"last update":"Thu Oct 19 12:57:32 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.49099999999999999,"15min":0.47699999999999998},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.26200000000000001},"max":{"1min":0.63500000000000001,"5min":0.65900000000000003,"15min":0.83799999999999997},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.435,"15min":0.42499999999999999},"min":{"1min":0.29499999999999998,"5min":0.28499999999999998,"15min":0.27000000000000002},"max":{"1min":0.56499999999999995,"5min":0.70699999999999996,"15min":0.80500000000000005},"last":0.35399999999999998}]},{"osd":4,"last update":"Thu Oct 19 12:57:32 2023","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.67500000000000004,"15min":0.65700000000000003},"min":{"1min":0.44,"5min":0.44,"15min":0.39300000000000002},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.622},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.66600000000000004,"15min":0.65200000000000002},"min":{"1min":0.495,"5min":0.42599999999999999,"15min":0.314},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":1.1910000000000001},"last":0.68799999999999994}]},{"osd":5,"last update":"Thu Oct 19 12:57:32 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.66700000000000004,"15min":0.65700000000000003},"min":{"1min":0.52300000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.878,"5min":0.878,"15min":0.91800000000000004},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.69099999999999995,"15min":0.67200000000000004},"min":{"1min":0.53700000000000003,"5min":0.47099999999999997,"15min":0.38700000000000001},"max":{"1min":0.88500000000000001,"5min":0.93999999999999995,"15min":1.117},"last":0.58499999999999996}]},{"osd":6,"last update":"Thu Oct 19 12:57:32 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.67700000000000005,"15min":0.66200000000000003},"min":{"1min":0.48399999999999999,"5min":0.36799999999999999,"15min":0.36399999999999999},"max":{"1min":0.92200000000000004,"5min":1.0069999999999999,"15min":1.077},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.69399999999999995,"15min":0.68500000000000005},"min":{"1min":0.47399999999999998,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.995,"5min":0.995,"15min":1.0009999999999999},"last":0.64900000000000002}]},{"osd":7,"last update":"Thu Oct 19 12:57:32 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.69499999999999995,"15min":0.68100000000000005},"min":{"1min":0.54000000000000004,"5min":0.46999999999999997,"15min":0.443},"max":{"1min":0.84499999999999997,"5min":0.97399999999999998,"15min":1.0149999999999999},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.70099999999999996,"15min":0.68300000000000005},"min":{"1min":0.46500000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":1.0489999999999999},"last":0.64300000000000002}]}]},{"osd":0,"up_from":13,"seq":55834575019,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36732,"kb_used_data":1392,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335108,"statfs":{"total":96636764160,"available":96599150592,"internally_reserved":0,"allocated":1425408,"data_stored":999343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Oct 19 12:57:36 2023","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.40100000000000002,"15min":0.38500000000000001},"min":{"1min":0.28699999999999998,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.442,"15min":0.42399999999999999},"min":{"1min":0.25,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.623,"5min":0.68700000000000006,"15min":0.746},"last":0.32500000000000001}]},{"osd":2,"last update":"Thu Oct 19 12:57:36 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.46700000000000003,"15min":0.45000000000000001},"min":{"1min":0.33900000000000002,"5min":0.26200000000000001,"15min":0.25800000000000001},"max":{"1min":0.752,"5min":0.752,"15min":0.76300000000000001},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.45100000000000001,"15min":0.42799999999999999},"min":{"1min":0.27500000000000002,"5min":0.216,"15min":0.20699999999999999},"max":{"1min":0.59399999999999997,"5min":0.68600000000000005,"15min":0.76800000000000002},"last":0.29299999999999998}]},{"osd":3,"last update":"Thu Oct 19 12:57:36 2023","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.42299999999999999,"15min":0.40100000000000002},"min":{"1min":0.24299999999999999,"5min":0.23400000000000001,"15min":0.22800000000000001},"max":{"1min":1.2829999999999999,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.38},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.51000000000000001,"15min":0.48799999999999999},"min":{"1min":0.33300000000000002,"5min":0.29399999999999998,"15min":0.29199999999999998},"max":{"1min":1.4990000000000001,"5min":1.4990000000000001,"15min":1.4990000000000001},"last":0.375}]},{"osd":4,"last update":"Thu Oct 19 12:57:36 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.65000000000000002,"15min":0.63},"min":{"1min":0.46700000000000003,"5min":0.45100000000000001,"15min":0.34999999999999998},"max":{"1min":1.361,"5min":1.361,"15min":1.361},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.65800000000000003,"15min":0.64100000000000001},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.35999999999999999},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.998},"last":0.55300000000000005}]},{"osd":5,"last update":"Thu Oct 19 12:57:36 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.69099999999999995,"15min":0.66800000000000004},"min":{"1min":0.44800000000000001,"5min":0.42099999999999999,"15min":0.38},"max":{"1min":1.325,"5min":1.325,"15min":1.325},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.69699999999999995,"15min":0.67800000000000005},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.318},"max":{"1min":0.88400000000000001,"5min":0.98699999999999999,"15min":1.056},"last":0.51500000000000001}]},{"osd":6,"last update":"Thu Oct 19 12:57:36 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.67000000000000004,"15min":0.65600000000000003},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.38500000000000001},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.438},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.68200000000000005,"15min":0.65800000000000003},"min":{"1min":0.45500000000000002,"5min":0.41999999999999998,"15min":0.38900000000000001},"max":{"1min":0.995,"5min":0.995,"15min":1.109},"last":0.47599999999999998}]},{"osd":7,"last update":"Thu Oct 19 12:57:36 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.69199999999999995,"15min":0.67700000000000005},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.41099999999999998},"max":{"1min":0.92000000000000004,"5min":0.95099999999999996,"15min":1.0089999999999999},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.67300000000000004,"15min":0.65600000000000003},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.389,"5min":1.389,"15min":1.389},"last":0.497}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-19T12:58:05.219 INFO:tasks.ceph.mds.c.smithi057.stderr:2023-10-19T12:58:05.216+0000 7f49fc7c1640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-19T12:58:06.715 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-19T12:58:06.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:06.834+0000 7f34e6900640 1 -- 172.21.15.57:0/1497876294 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f34e00c9dc0 msgr2=0x7f34e006b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:06.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:06.834+0000 7f34e6900640 1 --2- 172.21.15.57:0/1497876294 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f34e00c9dc0 0x7f34e006b2e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f34c8009830 tx=0x7f34c802bc20 comp rx=0 tx=0).stop 2023-10-19T12:58:06.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:06.835+0000 7f34e6900640 1 -- 172.21.15.57:0/1497876294 shutdown_connections 2023-10-19T12:58:06.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:06.835+0000 7f34e6900640 1 --2- 172.21.15.57:0/1497876294 >> v2:172.21.15.179:3300/0 conn(0x7f34e00ab500 0x7f34e00ab9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:06.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:06.835+0000 7f34e6900640 1 --2- 172.21.15.57:0/1497876294 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f34e00c9dc0 0x7f34e006b2e0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:06.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:06.835+0000 7f34e6900640 1 -- 172.21.15.57:0/1497876294 >> 172.21.15.57:0/1497876294 conn(0x7f34e0069930 msgr2=0x7f34e0069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:58:06.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:06.835+0000 7f34e6900640 1 -- 172.21.15.57:0/1497876294 shutdown_connections 2023-10-19T12:58:06.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:06.835+0000 7f34e6900640 1 -- 172.21.15.57:0/1497876294 wait complete. 2023-10-19T12:58:06.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:06.836+0000 7f34e6900640 1 Processor -- start 2023-10-19T12:58:06.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:06.836+0000 7f34e6900640 1 -- start start 2023-10-19T12:58:06.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:06.837+0000 7f34e6900640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f34e006af00 0x7f34e01f3f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:06.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:06.837+0000 7f34e6900640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f34e00ab500 0x7f34e00aa8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:06.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:06.837+0000 7f34e6900640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f34e00c9dc0 0x7f34e00aae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:06.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:06.837+0000 7f34e6900640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34e013d9d0 con 0x7f34e006af00 2023-10-19T12:58:06.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:06.837+0000 7f34e6900640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34e0151050 con 0x7f34e00c9dc0 2023-10-19T12:58:06.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:06.837+0000 7f34e6900640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34e01503d0 con 0x7f34e00ab500 2023-10-19T12:58:06.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:06.837+0000 7f34dffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f34e006af00 0x7f34e01f3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:06.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:06.837+0000 7f34dffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f34e006af00 0x7f34e01f3f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59418/0 (socket says 172.21.15.57:59418) 2023-10-19T12:58:06.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:06.837+0000 7f34dffff640 1 -- 172.21.15.57:0/1007935085 learned_addr learned my addr 172.21.15.57:0/1007935085 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:58:06.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:06.837+0000 7f34e4e9d640 1 --2- 172.21.15.57:0/1007935085 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f34e00c9dc0 0x7f34e00aae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:06.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:06.837+0000 7f34df7fe640 1 --2- 172.21.15.57:0/1007935085 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f34e00ab500 0x7f34e00aa8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:06.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:06.837+0000 7f34dffff640 1 -- 172.21.15.57:0/1007935085 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f34e00ab500 msgr2=0x7f34e00aa8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:06.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:06.837+0000 7f34dffff640 1 --2- 172.21.15.57:0/1007935085 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f34e00ab500 0x7f34e00aa8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:06.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:06.837+0000 7f34dffff640 1 -- 172.21.15.57:0/1007935085 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f34e00c9dc0 msgr2=0x7f34e00aae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:06.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:06.837+0000 7f34dffff640 1 --2- 172.21.15.57:0/1007935085 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f34e00c9dc0 0x7f34e00aae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:06.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:06.837+0000 7f34dffff640 1 -- 172.21.15.57:0/1007935085 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34c8009490 con 0x7f34e006af00 2023-10-19T12:58:06.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:06.837+0000 7f34dffff640 1 --2- 172.21.15.57:0/1007935085 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f34e006af00 0x7f34e01f3f30 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f34d000c770 tx=0x7f34d000cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:58:06.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:06.838+0000 7f34dcff9640 1 -- 172.21.15.57:0/1007935085 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34d000a840 con 0x7f34e006af00 2023-10-19T12:58:06.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:06.838+0000 7f34e6900640 1 -- 172.21.15.57:0/1007935085 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34e01506b0 con 0x7f34e006af00 2023-10-19T12:58:06.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:06.838+0000 7f34dcff9640 1 -- 172.21.15.57:0/1007935085 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f34d000aab0 con 0x7f34e006af00 2023-10-19T12:58:06.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:06.838+0000 7f34dcff9640 1 -- 172.21.15.57:0/1007935085 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34d0012420 con 0x7f34e006af00 2023-10-19T12:58:06.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:06.838+0000 7f34e6900640 1 -- 172.21.15.57:0/1007935085 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34e0150c70 con 0x7f34e006af00 2023-10-19T12:58:06.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:06.839+0000 7f34e6900640 1 -- 172.21.15.57:0/1007935085 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34e0151190 con 0x7f34e006af00 2023-10-19T12:58:06.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:06.839+0000 7f34dcff9640 1 -- 172.21.15.57:0/1007935085 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 20) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f34d001a3b0 con 0x7f34e006af00 2023-10-19T12:58:06.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:06.840+0000 7f34dcff9640 1 --2- 172.21.15.57:0/1007935085 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f34c006af60 0x7f34c006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:06.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:06.840+0000 7f34dcff9640 1 -- 172.21.15.57:0/1007935085 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f34d0089bb0 con 0x7f34e006af00 2023-10-19T12:58:06.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:06.840+0000 7f34df7fe640 1 --2- 172.21.15.57:0/1007935085 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f34c006af60 0x7f34c006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:06.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:06.841+0000 7f34df7fe640 1 --2- 172.21.15.57:0/1007935085 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f34c006af60 0x7f34c006d420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f34c8036000 tx=0x7f34c8004730 comp rx=0 tx=0).ready entity=mgr.5809 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:58:06.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:06.844+0000 7f34dcff9640 1 -- 172.21.15.57:0/1007935085 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f34d0054a20 con 0x7f34e006af00 2023-10-19T12:58:07.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.017+0000 7f34e6900640 1 -- 172.21.15.57:0/1007935085 --> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f34e0159c50 con 0x7f34c006af60 2023-10-19T12:58:07.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.025+0000 7f34dcff9640 1 -- 172.21.15.57:0/1007935085 <== mgr.5809 v2:172.21.15.179:6832/3449041082 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289605 (secure 0 0 0) 0x7f34c8004320 con 0x7f34c006af60 2023-10-19T12:58:07.031 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:58:07.032 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2023-10-19T12:58:07.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.028+0000 7f34e6900640 1 -- 172.21.15.57:0/1007935085 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f34c006af60 msgr2=0x7f34c006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:07.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.028+0000 7f34e6900640 1 --2- 172.21.15.57:0/1007935085 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f34c006af60 0x7f34c006d420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f34c8036000 tx=0x7f34c8004730 comp rx=0 tx=0).stop 2023-10-19T12:58:07.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.028+0000 7f34e6900640 1 -- 172.21.15.57:0/1007935085 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f34e006af00 msgr2=0x7f34e01f3f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:07.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.028+0000 7f34e6900640 1 --2- 172.21.15.57:0/1007935085 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f34e006af00 0x7f34e01f3f30 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f34d000c770 tx=0x7f34d000cc30 comp rx=0 tx=0).stop 2023-10-19T12:58:07.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.028+0000 7f34e6900640 1 -- 172.21.15.57:0/1007935085 shutdown_connections 2023-10-19T12:58:07.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.028+0000 7f34e6900640 1 --2- 172.21.15.57:0/1007935085 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f34c006af60 0x7f34c006d420 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:07.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.028+0000 7f34e6900640 1 --2- 172.21.15.57:0/1007935085 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f34e00c9dc0 0x7f34e00aae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:07.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.028+0000 7f34e6900640 1 --2- 172.21.15.57:0/1007935085 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f34e00ab500 0x7f34e00aa8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:07.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.028+0000 7f34e6900640 1 --2- 172.21.15.57:0/1007935085 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f34e006af00 0x7f34e01f3f30 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:07.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.028+0000 7f34e6900640 1 -- 172.21.15.57:0/1007935085 >> 172.21.15.57:0/1007935085 conn(0x7f34e0069930 msgr2=0x7f34e0069d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:58:07.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.028+0000 7f34e6900640 1 -- 172.21.15.57:0/1007935085 shutdown_connections 2023-10-19T12:58:07.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.028+0000 7f34e6900640 1 -- 172.21.15.57:0/1007935085 wait complete. 2023-10-19T12:58:07.041 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2023-10-19T12:58:06.628996+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":244,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":178,"ondisk_log_size":178,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":314140,"kb_used_data":7780,"kb_used_omap":12,"kb_used_meta":306227,"kb_avail":754660580,"statfs":{"total":773094113280,"available":772772433920,"internally_reserved":0,"allocated":7966720,"data_stored":4514201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":313577017},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001883"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.276851+0000","last_change":"2023-10-19T12:57:59.284384+0000","last_active":"2023-10-19T12:58:00.276851+0000","last_peered":"2023-10-19T12:58:00.276851+0000","last_clean":"2023-10-19T12:58:00.276851+0000","last_became_active":"2023-10-19T12:57:59.283993+0000","last_became_peered":"2023-10-19T12:57:59.283993+0000","last_unstale":"2023-10-19T12:58:00.276851+0000","last_undegraded":"2023-10-19T12:58:00.276851+0000","last_fullsized":"2023-10-19T12:58:00.276851+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:37:05.744377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:01.921750+0000","last_change":"2023-10-19T12:57:59.269413+0000","last_active":"2023-10-19T12:58:01.921750+0000","last_peered":"2023-10-19T12:58:01.921750+0000","last_clean":"2023-10-19T12:58:01.921750+0000","last_became_active":"2023-10-19T12:57:59.269167+0000","last_became_peered":"2023-10-19T12:57:59.269167+0000","last_unstale":"2023-10-19T12:58:01.921750+0000","last_undegraded":"2023-10-19T12:58:01.921750+0000","last_fullsized":"2023-10-19T12:58:01.921750+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:23:11.368528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:01.920984+0000","last_change":"2023-10-19T12:57:59.278020+0000","last_active":"2023-10-19T12:58:01.920984+0000","last_peered":"2023-10-19T12:58:01.920984+0000","last_clean":"2023-10-19T12:58:01.920984+0000","last_became_active":"2023-10-19T12:57:59.277709+0000","last_became_peered":"2023-10-19T12:57:59.277709+0000","last_unstale":"2023-10-19T12:58:01.920984+0000","last_undegraded":"2023-10-19T12:58:01.920984+0000","last_fullsized":"2023-10-19T12:58:01.920984+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:55:59.619036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:01.921690+0000","last_change":"2023-10-19T12:57:59.277566+0000","last_active":"2023-10-19T12:58:01.921690+0000","last_peered":"2023-10-19T12:58:01.921690+0000","last_clean":"2023-10-19T12:58:01.921690+0000","last_became_active":"2023-10-19T12:57:59.277336+0000","last_became_peered":"2023-10-19T12:57:59.277336+0000","last_unstale":"2023-10-19T12:58:01.921690+0000","last_undegraded":"2023-10-19T12:58:01.921690+0000","last_fullsized":"2023-10-19T12:58:01.921690+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:04:27.552252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:01.921706+0000","last_change":"2023-10-19T12:57:59.274819+0000","last_active":"2023-10-19T12:58:01.921706+0000","last_peered":"2023-10-19T12:58:01.921706+0000","last_clean":"2023-10-19T12:58:01.921706+0000","last_became_active":"2023-10-19T12:57:59.274476+0000","last_became_peered":"2023-10-19T12:57:59.274476+0000","last_unstale":"2023-10-19T12:58:01.921706+0000","last_undegraded":"2023-10-19T12:58:01.921706+0000","last_fullsized":"2023-10-19T12:58:01.921706+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:47:49.700629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.276766+0000","last_change":"2023-10-19T12:57:59.277273+0000","last_active":"2023-10-19T12:58:00.276766+0000","last_peered":"2023-10-19T12:58:00.276766+0000","last_clean":"2023-10-19T12:58:00.276766+0000","last_became_active":"2023-10-19T12:57:59.276991+0000","last_became_peered":"2023-10-19T12:57:59.276991+0000","last_unstale":"2023-10-19T12:58:00.276766+0000","last_undegraded":"2023-10-19T12:58:00.276766+0000","last_fullsized":"2023-10-19T12:58:00.276766+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:13:10.195317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:01.920743+0000","last_change":"2023-10-19T12:57:59.280280+0000","last_active":"2023-10-19T12:58:01.920743+0000","last_peered":"2023-10-19T12:58:01.920743+0000","last_clean":"2023-10-19T12:58:01.920743+0000","last_became_active":"2023-10-19T12:57:59.279626+0000","last_became_peered":"2023-10-19T12:57:59.279626+0000","last_unstale":"2023-10-19T12:58:01.920743+0000","last_undegraded":"2023-10-19T12:58:01.920743+0000","last_fullsized":"2023-10-19T12:58:01.920743+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:08:23.873320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.274836+0000","last_change":"2023-10-19T12:57:59.281673+0000","last_active":"2023-10-19T12:58:00.274836+0000","last_peered":"2023-10-19T12:58:00.274836+0000","last_clean":"2023-10-19T12:58:00.274836+0000","last_became_active":"2023-10-19T12:57:59.281404+0000","last_became_peered":"2023-10-19T12:57:59.281404+0000","last_unstale":"2023-10-19T12:58:00.274836+0000","last_undegraded":"2023-10-19T12:58:00.274836+0000","last_fullsized":"2023-10-19T12:58:00.274836+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:03:46.464251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.275183+0000","last_change":"2023-10-19T12:57:59.282312+0000","last_active":"2023-10-19T12:58:00.275183+0000","last_peered":"2023-10-19T12:58:00.275183+0000","last_clean":"2023-10-19T12:58:00.275183+0000","last_became_active":"2023-10-19T12:57:59.282052+0000","last_became_peered":"2023-10-19T12:57:59.282052+0000","last_unstale":"2023-10-19T12:58:00.275183+0000","last_undegraded":"2023-10-19T12:58:00.275183+0000","last_fullsized":"2023-10-19T12:58:00.275183+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:35:30.427138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.276809+0000","last_change":"2023-10-19T12:57:59.281613+0000","last_active":"2023-10-19T12:58:00.276809+0000","last_peered":"2023-10-19T12:58:00.276809+0000","last_clean":"2023-10-19T12:58:00.276809+0000","last_became_active":"2023-10-19T12:57:59.281322+0000","last_became_peered":"2023-10-19T12:57:59.281322+0000","last_unstale":"2023-10-19T12:58:00.276809+0000","last_undegraded":"2023-10-19T12:58:00.276809+0000","last_fullsized":"2023-10-19T12:58:00.276809+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:33:13.743238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.855192+0000","last_change":"2023-10-19T12:57:59.275510+0000","last_active":"2023-10-19T12:58:00.855192+0000","last_peered":"2023-10-19T12:58:00.855192+0000","last_clean":"2023-10-19T12:58:00.855192+0000","last_became_active":"2023-10-19T12:57:59.275193+0000","last_became_peered":"2023-10-19T12:57:59.275193+0000","last_unstale":"2023-10-19T12:58:00.855192+0000","last_undegraded":"2023-10-19T12:58:00.855192+0000","last_fullsized":"2023-10-19T12:58:00.855192+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:28:50.912212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.275242+0000","last_change":"2023-10-19T12:57:59.269953+0000","last_active":"2023-10-19T12:58:00.275242+0000","last_peered":"2023-10-19T12:58:00.275242+0000","last_clean":"2023-10-19T12:58:00.275242+0000","last_became_active":"2023-10-19T12:57:59.269663+0000","last_became_peered":"2023-10-19T12:57:59.269663+0000","last_unstale":"2023-10-19T12:58:00.275242+0000","last_undegraded":"2023-10-19T12:58:00.275242+0000","last_fullsized":"2023-10-19T12:58:00.275242+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:36:18.446379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.276748+0000","last_change":"2023-10-19T12:57:59.279218+0000","last_active":"2023-10-19T12:58:00.276748+0000","last_peered":"2023-10-19T12:58:00.276748+0000","last_clean":"2023-10-19T12:58:00.276748+0000","last_became_active":"2023-10-19T12:57:59.278919+0000","last_became_peered":"2023-10-19T12:57:59.278919+0000","last_unstale":"2023-10-19T12:58:00.276748+0000","last_undegraded":"2023-10-19T12:58:00.276748+0000","last_fullsized":"2023-10-19T12:58:00.276748+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:50:41.250507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:01.920845+0000","last_change":"2023-10-19T12:57:59.280185+0000","last_active":"2023-10-19T12:58:01.920845+0000","last_peered":"2023-10-19T12:58:01.920845+0000","last_clean":"2023-10-19T12:58:01.920845+0000","last_became_active":"2023-10-19T12:57:59.279558+0000","last_became_peered":"2023-10-19T12:57:59.279558+0000","last_unstale":"2023-10-19T12:58:01.920845+0000","last_undegraded":"2023-10-19T12:58:01.920845+0000","last_fullsized":"2023-10-19T12:58:01.920845+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:05:13.640074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:01.921426+0000","last_change":"2023-10-19T12:57:59.268237+0000","last_active":"2023-10-19T12:58:01.921426+0000","last_peered":"2023-10-19T12:58:01.921426+0000","last_clean":"2023-10-19T12:58:01.921426+0000","last_became_active":"2023-10-19T12:57:59.267965+0000","last_became_peered":"2023-10-19T12:57:59.267965+0000","last_unstale":"2023-10-19T12:58:01.921426+0000","last_undegraded":"2023-10-19T12:58:01.921426+0000","last_fullsized":"2023-10-19T12:58:01.921426+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:58:49.729860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.276674+0000","last_change":"2023-10-19T12:57:59.277264+0000","last_active":"2023-10-19T12:58:00.276674+0000","last_peered":"2023-10-19T12:58:00.276674+0000","last_clean":"2023-10-19T12:58:00.276674+0000","last_became_active":"2023-10-19T12:57:59.276973+0000","last_became_peered":"2023-10-19T12:57:59.276973+0000","last_unstale":"2023-10-19T12:58:00.276674+0000","last_undegraded":"2023-10-19T12:58:00.276674+0000","last_fullsized":"2023-10-19T12:58:00.276674+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:19:17.421557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:01.921262+0000","last_change":"2023-10-19T12:57:59.278487+0000","last_active":"2023-10-19T12:58:01.921262+0000","last_peered":"2023-10-19T12:58:01.921262+0000","last_clean":"2023-10-19T12:58:01.921262+0000","last_became_active":"2023-10-19T12:57:59.276322+0000","last_became_peered":"2023-10-19T12:57:59.276322+0000","last_unstale":"2023-10-19T12:58:01.921262+0000","last_undegraded":"2023-10-19T12:58:01.921262+0000","last_fullsized":"2023-10-19T12:58:01.921262+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:50:32.740818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:01.921788+0000","last_change":"2023-10-19T12:57:59.268961+0000","last_active":"2023-10-19T12:58:01.921788+0000","last_peered":"2023-10-19T12:58:01.921788+0000","last_clean":"2023-10-19T12:58:01.921788+0000","last_became_active":"2023-10-19T12:57:59.268701+0000","last_became_peered":"2023-10-19T12:57:59.268701+0000","last_unstale":"2023-10-19T12:58:01.921788+0000","last_undegraded":"2023-10-19T12:58:01.921788+0000","last_fullsized":"2023-10-19T12:58:01.921788+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:03:07.244263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:01.921030+0000","last_change":"2023-10-19T12:57:59.280395+0000","last_active":"2023-10-19T12:58:01.921030+0000","last_peered":"2023-10-19T12:58:01.921030+0000","last_clean":"2023-10-19T12:58:01.921030+0000","last_became_active":"2023-10-19T12:57:59.280082+0000","last_became_peered":"2023-10-19T12:57:59.280082+0000","last_unstale":"2023-10-19T12:58:01.921030+0000","last_undegraded":"2023-10-19T12:58:01.921030+0000","last_fullsized":"2023-10-19T12:58:01.921030+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:04:27.552252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:01.922266+0000","last_change":"2023-10-19T12:57:59.279468+0000","last_active":"2023-10-19T12:58:01.922266+0000","last_peered":"2023-10-19T12:58:01.922266+0000","last_clean":"2023-10-19T12:58:01.922266+0000","last_became_active":"2023-10-19T12:57:59.279204+0000","last_became_peered":"2023-10-19T12:57:59.279204+0000","last_unstale":"2023-10-19T12:58:01.922266+0000","last_undegraded":"2023-10-19T12:58:01.922266+0000","last_fullsized":"2023-10-19T12:58:01.922266+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:50:41.250507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:01.922167+0000","last_change":"2023-10-19T12:57:59.273672+0000","last_active":"2023-10-19T12:58:01.922167+0000","last_peered":"2023-10-19T12:58:01.922167+0000","last_clean":"2023-10-19T12:58:01.922167+0000","last_became_active":"2023-10-19T12:57:59.273382+0000","last_became_peered":"2023-10-19T12:57:59.273382+0000","last_unstale":"2023-10-19T12:58:01.922167+0000","last_undegraded":"2023-10-19T12:58:01.922167+0000","last_fullsized":"2023-10-19T12:58:01.922167+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:36:39.520498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.274788+0000","last_change":"2023-10-19T12:57:59.275894+0000","last_active":"2023-10-19T12:58:00.274788+0000","last_peered":"2023-10-19T12:58:00.274788+0000","last_clean":"2023-10-19T12:58:00.274788+0000","last_became_active":"2023-10-19T12:57:59.275622+0000","last_became_peered":"2023-10-19T12:57:59.275622+0000","last_unstale":"2023-10-19T12:58:00.274788+0000","last_undegraded":"2023-10-19T12:58:00.274788+0000","last_fullsized":"2023-10-19T12:58:00.274788+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:50:34.298018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.854983+0000","last_change":"2023-10-19T12:57:59.266850+0000","last_active":"2023-10-19T12:58:00.854983+0000","last_peered":"2023-10-19T12:58:00.854983+0000","last_clean":"2023-10-19T12:58:00.854983+0000","last_became_active":"2023-10-19T12:57:59.266576+0000","last_became_peered":"2023-10-19T12:57:59.266576+0000","last_unstale":"2023-10-19T12:58:00.854983+0000","last_undegraded":"2023-10-19T12:58:00.854983+0000","last_fullsized":"2023-10-19T12:58:00.854983+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:04:25.280981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:01.920691+0000","last_change":"2023-10-19T12:57:59.279954+0000","last_active":"2023-10-19T12:58:01.920691+0000","last_peered":"2023-10-19T12:58:01.920691+0000","last_clean":"2023-10-19T12:58:01.920691+0000","last_became_active":"2023-10-19T12:57:59.279667+0000","last_became_peered":"2023-10-19T12:57:59.279667+0000","last_unstale":"2023-10-19T12:58:01.920691+0000","last_undegraded":"2023-10-19T12:58:01.920691+0000","last_fullsized":"2023-10-19T12:58:01.920691+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:40:08.031897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:01.921361+0000","last_change":"2023-10-19T12:57:59.292802+0000","last_active":"2023-10-19T12:58:01.921361+0000","last_peered":"2023-10-19T12:58:01.921361+0000","last_clean":"2023-10-19T12:58:01.921361+0000","last_became_active":"2023-10-19T12:57:59.292429+0000","last_became_peered":"2023-10-19T12:57:59.292429+0000","last_unstale":"2023-10-19T12:58:01.921361+0000","last_undegraded":"2023-10-19T12:58:01.921361+0000","last_fullsized":"2023-10-19T12:58:01.921361+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:56:56.159781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.274984+0000","last_change":"2023-10-19T12:57:59.262814+0000","last_active":"2023-10-19T12:58:00.274984+0000","last_peered":"2023-10-19T12:58:00.274984+0000","last_clean":"2023-10-19T12:58:00.274984+0000","last_became_active":"2023-10-19T12:57:59.262506+0000","last_became_peered":"2023-10-19T12:57:59.262506+0000","last_unstale":"2023-10-19T12:58:00.274984+0000","last_undegraded":"2023-10-19T12:58:00.274984+0000","last_fullsized":"2023-10-19T12:58:00.274984+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:32:35.849874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.277061+0000","last_change":"2023-10-19T12:57:59.283185+0000","last_active":"2023-10-19T12:58:00.277061+0000","last_peered":"2023-10-19T12:58:00.277061+0000","last_clean":"2023-10-19T12:58:00.277061+0000","last_became_active":"2023-10-19T12:57:59.282305+0000","last_became_peered":"2023-10-19T12:57:59.282305+0000","last_unstale":"2023-10-19T12:58:00.277061+0000","last_undegraded":"2023-10-19T12:58:00.277061+0000","last_fullsized":"2023-10-19T12:58:00.277061+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:07:58.427104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:01.922308+0000","last_change":"2023-10-19T12:57:59.274260+0000","last_active":"2023-10-19T12:58:01.922308+0000","last_peered":"2023-10-19T12:58:01.922308+0000","last_clean":"2023-10-19T12:58:01.922308+0000","last_became_active":"2023-10-19T12:57:59.274012+0000","last_became_peered":"2023-10-19T12:57:59.274012+0000","last_unstale":"2023-10-19T12:58:01.922308+0000","last_undegraded":"2023-10-19T12:58:01.922308+0000","last_fullsized":"2023-10-19T12:58:01.922308+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:13:10.195317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:01.920849+0000","last_change":"2023-10-19T12:57:59.281600+0000","last_active":"2023-10-19T12:58:01.920849+0000","last_peered":"2023-10-19T12:58:01.920849+0000","last_clean":"2023-10-19T12:58:01.920849+0000","last_became_active":"2023-10-19T12:57:59.281239+0000","last_became_peered":"2023-10-19T12:57:59.281239+0000","last_unstale":"2023-10-19T12:58:01.920849+0000","last_undegraded":"2023-10-19T12:58:01.920849+0000","last_fullsized":"2023-10-19T12:58:01.920849+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:58:49.729860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:01.921942+0000","last_change":"2023-10-19T12:57:59.277338+0000","last_active":"2023-10-19T12:58:01.921942+0000","last_peered":"2023-10-19T12:58:01.921942+0000","last_clean":"2023-10-19T12:58:01.921942+0000","last_became_active":"2023-10-19T12:57:59.277100+0000","last_became_peered":"2023-10-19T12:57:59.277100+0000","last_unstale":"2023-10-19T12:58:01.921942+0000","last_undegraded":"2023-10-19T12:58:01.921942+0000","last_fullsized":"2023-10-19T12:58:01.921942+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:55:59.619036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:01.920813+0000","last_change":"2023-10-19T12:57:59.280469+0000","last_active":"2023-10-19T12:58:01.920813+0000","last_peered":"2023-10-19T12:58:01.920813+0000","last_clean":"2023-10-19T12:58:01.920813+0000","last_became_active":"2023-10-19T12:57:59.280002+0000","last_became_peered":"2023-10-19T12:57:59.280002+0000","last_unstale":"2023-10-19T12:58:01.920813+0000","last_undegraded":"2023-10-19T12:58:01.920813+0000","last_fullsized":"2023-10-19T12:58:01.920813+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:37:57.451855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.272211+0000","last_change":"2023-10-19T12:57:59.261431+0000","last_active":"2023-10-19T12:58:00.272211+0000","last_peered":"2023-10-19T12:58:00.272211+0000","last_clean":"2023-10-19T12:58:00.272211+0000","last_became_active":"2023-10-19T12:57:59.261169+0000","last_became_peered":"2023-10-19T12:57:59.261169+0000","last_unstale":"2023-10-19T12:58:00.272211+0000","last_undegraded":"2023-10-19T12:58:00.272211+0000","last_fullsized":"2023-10-19T12:58:00.272211+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:50:32.740818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:01.921629+0000","last_change":"2023-10-19T12:57:59.272988+0000","last_active":"2023-10-19T12:58:01.921629+0000","last_peered":"2023-10-19T12:58:01.921629+0000","last_clean":"2023-10-19T12:58:01.921629+0000","last_became_active":"2023-10-19T12:57:59.272744+0000","last_became_peered":"2023-10-19T12:57:59.272744+0000","last_unstale":"2023-10-19T12:58:01.921629+0000","last_undegraded":"2023-10-19T12:58:01.921629+0000","last_fullsized":"2023-10-19T12:58:01.921629+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:23:02.269273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"69'4","reported_seq":16,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.629388+0000","last_change":"2023-10-19T12:57:58.252331+0000","last_active":"2023-10-19T12:58:00.629388+0000","last_peered":"2023-10-19T12:58:00.629388+0000","last_clean":"2023-10-19T12:58:00.629388+0000","last_became_active":"2023-10-19T12:57:58.251127+0000","last_became_peered":"2023-10-19T12:57:58.251127+0000","last_unstale":"2023-10-19T12:58:00.629388+0000","last_undegraded":"2023-10-19T12:58:00.629388+0000","last_fullsized":"2023-10-19T12:58:00.629388+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:43:25.389215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.272250+0000","last_change":"2023-10-19T12:57:59.264715+0000","last_active":"2023-10-19T12:58:00.272250+0000","last_peered":"2023-10-19T12:58:00.272250+0000","last_clean":"2023-10-19T12:58:00.272250+0000","last_became_active":"2023-10-19T12:57:59.264476+0000","last_became_peered":"2023-10-19T12:57:59.264476+0000","last_unstale":"2023-10-19T12:58:00.272250+0000","last_undegraded":"2023-10-19T12:58:00.272250+0000","last_fullsized":"2023-10-19T12:58:00.272250+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:06:20.535644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"69'1","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.276793+0000","last_change":"2023-10-19T12:57:58.256650+0000","last_active":"2023-10-19T12:58:00.276793+0000","last_peered":"2023-10-19T12:58:00.276793+0000","last_clean":"2023-10-19T12:58:00.276793+0000","last_became_active":"2023-10-19T12:57:58.256210+0000","last_became_peered":"2023-10-19T12:57:58.256210+0000","last_unstale":"2023-10-19T12:58:00.276793+0000","last_undegraded":"2023-10-19T12:58:00.276793+0000","last_fullsized":"2023-10-19T12:58:00.276793+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:19:19.078844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:01.922437+0000","last_change":"2023-10-19T12:57:59.269502+0000","last_active":"2023-10-19T12:58:01.922437+0000","last_peered":"2023-10-19T12:58:01.922437+0000","last_clean":"2023-10-19T12:58:01.922437+0000","last_became_active":"2023-10-19T12:57:59.268734+0000","last_became_peered":"2023-10-19T12:57:59.268734+0000","last_unstale":"2023-10-19T12:58:01.922437+0000","last_undegraded":"2023-10-19T12:58:01.922437+0000","last_fullsized":"2023-10-19T12:58:01.922437+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:13:06.950010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"69'2","reported_seq":14,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.277103+0000","last_change":"2023-10-19T12:57:58.255163+0000","last_active":"2023-10-19T12:58:00.277103+0000","last_peered":"2023-10-19T12:58:00.277103+0000","last_clean":"2023-10-19T12:58:00.277103+0000","last_became_active":"2023-10-19T12:57:58.254897+0000","last_became_peered":"2023-10-19T12:57:58.254897+0000","last_unstale":"2023-10-19T12:58:00.277103+0000","last_undegraded":"2023-10-19T12:58:00.277103+0000","last_fullsized":"2023-10-19T12:58:00.277103+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:48:56.789976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.274847+0000","last_change":"2023-10-19T12:57:59.261552+0000","last_active":"2023-10-19T12:58:00.274847+0000","last_peered":"2023-10-19T12:58:00.274847+0000","last_clean":"2023-10-19T12:58:00.274847+0000","last_became_active":"2023-10-19T12:57:59.261224+0000","last_became_peered":"2023-10-19T12:57:59.261224+0000","last_unstale":"2023-10-19T12:58:00.274847+0000","last_undegraded":"2023-10-19T12:58:00.274847+0000","last_fullsized":"2023-10-19T12:58:00.274847+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:38:57.879320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"69'2","reported_seq":14,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.272345+0000","last_change":"2023-10-19T12:57:58.255648+0000","last_active":"2023-10-19T12:58:00.272345+0000","last_peered":"2023-10-19T12:58:00.272345+0000","last_clean":"2023-10-19T12:58:00.272345+0000","last_became_active":"2023-10-19T12:57:58.255149+0000","last_became_peered":"2023-10-19T12:57:58.255149+0000","last_unstale":"2023-10-19T12:58:00.272345+0000","last_undegraded":"2023-10-19T12:58:00.272345+0000","last_fullsized":"2023-10-19T12:58:00.272345+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:55:58.608631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:01.921825+0000","last_change":"2023-10-19T12:57:59.282345+0000","last_active":"2023-10-19T12:58:01.921825+0000","last_peered":"2023-10-19T12:58:01.921825+0000","last_clean":"2023-10-19T12:58:01.921825+0000","last_became_active":"2023-10-19T12:57:59.282022+0000","last_became_peered":"2023-10-19T12:57:59.282022+0000","last_unstale":"2023-10-19T12:58:01.921825+0000","last_undegraded":"2023-10-19T12:58:01.921825+0000","last_fullsized":"2023-10-19T12:58:01.921825+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:07:58.427104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"69'1","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.624369+0000","last_change":"2023-10-19T12:57:58.257095+0000","last_active":"2023-10-19T12:58:00.624369+0000","last_peered":"2023-10-19T12:58:00.624369+0000","last_clean":"2023-10-19T12:58:00.624369+0000","last_became_active":"2023-10-19T12:57:58.256684+0000","last_became_peered":"2023-10-19T12:57:58.256684+0000","last_unstale":"2023-10-19T12:58:00.624369+0000","last_undegraded":"2023-10-19T12:58:00.624369+0000","last_fullsized":"2023-10-19T12:58:00.624369+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:39:16.198486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:01.921967+0000","last_change":"2023-10-19T12:57:59.278382+0000","last_active":"2023-10-19T12:58:01.921967+0000","last_peered":"2023-10-19T12:58:01.921967+0000","last_clean":"2023-10-19T12:58:01.921967+0000","last_became_active":"2023-10-19T12:57:59.278100+0000","last_became_peered":"2023-10-19T12:57:59.278100+0000","last_unstale":"2023-10-19T12:58:01.921967+0000","last_undegraded":"2023-10-19T12:58:01.921967+0000","last_fullsized":"2023-10-19T12:58:01.921967+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:49:12.889515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"69'1","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.627267+0000","last_change":"2023-10-19T12:57:58.254342+0000","last_active":"2023-10-19T12:58:00.627267+0000","last_peered":"2023-10-19T12:58:00.627267+0000","last_clean":"2023-10-19T12:58:00.627267+0000","last_became_active":"2023-10-19T12:57:58.254053+0000","last_became_peered":"2023-10-19T12:57:58.254053+0000","last_unstale":"2023-10-19T12:58:00.627267+0000","last_undegraded":"2023-10-19T12:58:00.627267+0000","last_fullsized":"2023-10-19T12:58:00.627267+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:33:12.732833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.275130+0000","last_change":"2023-10-19T12:57:59.261739+0000","last_active":"2023-10-19T12:58:00.275130+0000","last_peered":"2023-10-19T12:58:00.275130+0000","last_clean":"2023-10-19T12:58:00.275130+0000","last_became_active":"2023-10-19T12:57:59.261380+0000","last_became_peered":"2023-10-19T12:57:59.261380+0000","last_unstale":"2023-10-19T12:58:00.275130+0000","last_undegraded":"2023-10-19T12:58:00.275130+0000","last_fullsized":"2023-10-19T12:58:00.275130+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:12:33.439060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.629266+0000","last_change":"2023-10-19T12:57:58.252697+0000","last_active":"2023-10-19T12:58:00.629266+0000","last_peered":"2023-10-19T12:58:00.629266+0000","last_clean":"2023-10-19T12:58:00.629266+0000","last_became_active":"2023-10-19T12:57:58.252422+0000","last_became_peered":"2023-10-19T12:57:58.252422+0000","last_unstale":"2023-10-19T12:58:00.629266+0000","last_undegraded":"2023-10-19T12:58:00.629266+0000","last_fullsized":"2023-10-19T12:58:00.629266+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:26:25.065741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:01.921568+0000","last_change":"2023-10-19T12:57:59.280236+0000","last_active":"2023-10-19T12:58:01.921568+0000","last_peered":"2023-10-19T12:58:01.921568+0000","last_clean":"2023-10-19T12:58:01.921568+0000","last_became_active":"2023-10-19T12:57:59.279806+0000","last_became_peered":"2023-10-19T12:57:59.279806+0000","last_unstale":"2023-10-19T12:58:01.921568+0000","last_undegraded":"2023-10-19T12:58:01.921568+0000","last_fullsized":"2023-10-19T12:58:01.921568+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:32:54.591183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.624301+0000","last_change":"2023-10-19T12:57:58.259037+0000","last_active":"2023-10-19T12:58:00.624301+0000","last_peered":"2023-10-19T12:58:00.624301+0000","last_clean":"2023-10-19T12:58:00.624301+0000","last_became_active":"2023-10-19T12:57:58.258745+0000","last_became_peered":"2023-10-19T12:57:58.258745+0000","last_unstale":"2023-10-19T12:58:00.624301+0000","last_undegraded":"2023-10-19T12:58:00.624301+0000","last_fullsized":"2023-10-19T12:58:00.624301+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:29:50.196136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:01.922456+0000","last_change":"2023-10-19T12:57:59.264886+0000","last_active":"2023-10-19T12:58:01.922456+0000","last_peered":"2023-10-19T12:58:01.922456+0000","last_clean":"2023-10-19T12:58:01.922456+0000","last_became_active":"2023-10-19T12:57:59.264635+0000","last_became_peered":"2023-10-19T12:57:59.264635+0000","last_unstale":"2023-10-19T12:58:01.922456+0000","last_undegraded":"2023-10-19T12:58:01.922456+0000","last_fullsized":"2023-10-19T12:58:01.922456+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:11:38.896474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.629141+0000","last_change":"2023-10-19T12:57:58.251390+0000","last_active":"2023-10-19T12:58:00.629141+0000","last_peered":"2023-10-19T12:58:00.629141+0000","last_clean":"2023-10-19T12:58:00.629141+0000","last_became_active":"2023-10-19T12:57:58.251120+0000","last_became_peered":"2023-10-19T12:57:58.251120+0000","last_unstale":"2023-10-19T12:58:00.629141+0000","last_undegraded":"2023-10-19T12:58:00.629141+0000","last_fullsized":"2023-10-19T12:58:00.629141+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:23:36.269955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.272416+0000","last_change":"2023-10-19T12:57:59.270585+0000","last_active":"2023-10-19T12:58:00.272416+0000","last_peered":"2023-10-19T12:58:00.272416+0000","last_clean":"2023-10-19T12:58:00.272416+0000","last_became_active":"2023-10-19T12:57:59.270313+0000","last_became_peered":"2023-10-19T12:57:59.270313+0000","last_unstale":"2023-10-19T12:58:00.272416+0000","last_undegraded":"2023-10-19T12:58:00.272416+0000","last_fullsized":"2023-10-19T12:58:00.272416+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:41:54.210777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.627320+0000","last_change":"2023-10-19T12:57:58.257516+0000","last_active":"2023-10-19T12:58:00.627320+0000","last_peered":"2023-10-19T12:58:00.627320+0000","last_clean":"2023-10-19T12:58:00.627320+0000","last_became_active":"2023-10-19T12:57:58.257217+0000","last_became_peered":"2023-10-19T12:57:58.257217+0000","last_unstale":"2023-10-19T12:58:00.627320+0000","last_undegraded":"2023-10-19T12:58:00.627320+0000","last_fullsized":"2023-10-19T12:58:00.627320+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:13:05.939605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:01.920970+0000","last_change":"2023-10-19T12:57:59.282205+0000","last_active":"2023-10-19T12:58:01.920970+0000","last_peered":"2023-10-19T12:58:01.920970+0000","last_clean":"2023-10-19T12:58:01.920970+0000","last_became_active":"2023-10-19T12:57:59.281923+0000","last_became_peered":"2023-10-19T12:57:59.281923+0000","last_unstale":"2023-10-19T12:58:01.920970+0000","last_undegraded":"2023-10-19T12:58:01.920970+0000","last_fullsized":"2023-10-19T12:58:01.920970+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:32:54.591183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"69'3","reported_seq":15,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.629041+0000","last_change":"2023-10-19T12:57:58.255428+0000","last_active":"2023-10-19T12:58:00.629041+0000","last_peered":"2023-10-19T12:58:00.629041+0000","last_clean":"2023-10-19T12:58:00.629041+0000","last_became_active":"2023-10-19T12:57:58.255138+0000","last_became_peered":"2023-10-19T12:57:58.255138+0000","last_unstale":"2023-10-19T12:58:00.629041+0000","last_undegraded":"2023-10-19T12:58:00.629041+0000","last_fullsized":"2023-10-19T12:58:00.629041+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:40:07.021492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:01.921752+0000","last_change":"2023-10-19T12:57:59.279730+0000","last_active":"2023-10-19T12:58:01.921752+0000","last_peered":"2023-10-19T12:58:01.921752+0000","last_clean":"2023-10-19T12:58:01.921752+0000","last_became_active":"2023-10-19T12:57:59.279463+0000","last_became_peered":"2023-10-19T12:57:59.279463+0000","last_unstale":"2023-10-19T12:58:01.921752+0000","last_undegraded":"2023-10-19T12:58:01.921752+0000","last_fullsized":"2023-10-19T12:58:01.921752+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:19:34.001505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"69'6","reported_seq":18,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.275275+0000","last_change":"2023-10-19T12:57:58.250143+0000","last_active":"2023-10-19T12:58:00.275275+0000","last_peered":"2023-10-19T12:58:00.275275+0000","last_clean":"2023-10-19T12:58:00.275275+0000","last_became_active":"2023-10-19T12:57:58.249866+0000","last_became_peered":"2023-10-19T12:57:58.249866+0000","last_unstale":"2023-10-19T12:58:00.275275+0000","last_undegraded":"2023-10-19T12:58:00.275275+0000","last_fullsized":"2023-10-19T12:58:00.275275+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:29:34.406278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.272574+0000","last_change":"2023-10-19T12:57:59.266797+0000","last_active":"2023-10-19T12:58:00.272574+0000","last_peered":"2023-10-19T12:58:00.272574+0000","last_clean":"2023-10-19T12:58:00.272574+0000","last_became_active":"2023-10-19T12:57:59.266142+0000","last_became_peered":"2023-10-19T12:57:59.266142+0000","last_unstale":"2023-10-19T12:58:00.272574+0000","last_undegraded":"2023-10-19T12:58:00.272574+0000","last_fullsized":"2023-10-19T12:58:00.272574+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:40:50.284709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"69'2","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.624379+0000","last_change":"2023-10-19T12:57:58.257253+0000","last_active":"2023-10-19T12:58:00.624379+0000","last_peered":"2023-10-19T12:58:00.624379+0000","last_clean":"2023-10-19T12:58:00.624379+0000","last_became_active":"2023-10-19T12:57:58.256982+0000","last_became_peered":"2023-10-19T12:57:58.256982+0000","last_unstale":"2023-10-19T12:58:00.624379+0000","last_undegraded":"2023-10-19T12:58:00.624379+0000","last_fullsized":"2023-10-19T12:58:00.624379+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:26:46.604704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:01.921168+0000","last_change":"2023-10-19T12:57:59.282874+0000","last_active":"2023-10-19T12:58:01.921168+0000","last_peered":"2023-10-19T12:58:01.921168+0000","last_clean":"2023-10-19T12:58:01.921168+0000","last_became_active":"2023-10-19T12:57:59.282521+0000","last_became_peered":"2023-10-19T12:57:59.282521+0000","last_unstale":"2023-10-19T12:58:01.921168+0000","last_undegraded":"2023-10-19T12:58:01.921168+0000","last_fullsized":"2023-10-19T12:58:01.921168+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:49:12.889515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.629303+0000","last_change":"2023-10-19T12:57:58.259423+0000","last_active":"2023-10-19T12:58:00.629303+0000","last_peered":"2023-10-19T12:58:00.629303+0000","last_clean":"2023-10-19T12:58:00.629303+0000","last_became_active":"2023-10-19T12:57:58.259165+0000","last_became_peered":"2023-10-19T12:57:58.259165+0000","last_unstale":"2023-10-19T12:58:00.629303+0000","last_undegraded":"2023-10-19T12:58:00.629303+0000","last_fullsized":"2023-10-19T12:58:00.629303+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:46:53.191615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.275075+0000","last_change":"2023-10-19T12:57:59.267621+0000","last_active":"2023-10-19T12:58:00.275075+0000","last_peered":"2023-10-19T12:58:00.275075+0000","last_clean":"2023-10-19T12:58:00.275075+0000","last_became_active":"2023-10-19T12:57:59.267351+0000","last_became_peered":"2023-10-19T12:57:59.267351+0000","last_unstale":"2023-10-19T12:58:00.275075+0000","last_undegraded":"2023-10-19T12:58:00.275075+0000","last_fullsized":"2023-10-19T12:58:00.275075+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:53:20.624726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.628969+0000","last_change":"2023-10-19T12:57:58.254432+0000","last_active":"2023-10-19T12:58:00.628969+0000","last_peered":"2023-10-19T12:58:00.628969+0000","last_clean":"2023-10-19T12:58:00.628969+0000","last_became_active":"2023-10-19T12:57:58.254162+0000","last_became_peered":"2023-10-19T12:57:58.254162+0000","last_unstale":"2023-10-19T12:58:00.628969+0000","last_undegraded":"2023-10-19T12:58:00.628969+0000","last_fullsized":"2023-10-19T12:58:00.628969+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:38:37.380677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:01.921247+0000","last_change":"2023-10-19T12:57:59.278651+0000","last_active":"2023-10-19T12:58:01.921247+0000","last_peered":"2023-10-19T12:58:01.921247+0000","last_clean":"2023-10-19T12:58:01.921247+0000","last_became_active":"2023-10-19T12:57:59.278113+0000","last_became_peered":"2023-10-19T12:57:59.278113+0000","last_unstale":"2023-10-19T12:58:01.921247+0000","last_undegraded":"2023-10-19T12:58:01.921247+0000","last_fullsized":"2023-10-19T12:58:01.921247+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:23:11.368528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"69'8","reported_seq":20,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.275013+0000","last_change":"2023-10-19T12:57:58.247030+0000","last_active":"2023-10-19T12:58:00.275013+0000","last_peered":"2023-10-19T12:58:00.275013+0000","last_clean":"2023-10-19T12:58:00.275013+0000","last_became_active":"2023-10-19T12:57:58.246777+0000","last_became_peered":"2023-10-19T12:57:58.246777+0000","last_unstale":"2023-10-19T12:58:00.275013+0000","last_undegraded":"2023-10-19T12:58:00.275013+0000","last_fullsized":"2023-10-19T12:58:00.275013+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:24:54.236739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.275193+0000","last_change":"2023-10-19T12:57:59.262682+0000","last_active":"2023-10-19T12:58:00.275193+0000","last_peered":"2023-10-19T12:58:00.275193+0000","last_clean":"2023-10-19T12:58:00.275193+0000","last_became_active":"2023-10-19T12:57:59.262360+0000","last_became_peered":"2023-10-19T12:57:59.262360+0000","last_unstale":"2023-10-19T12:58:00.275193+0000","last_undegraded":"2023-10-19T12:58:00.275193+0000","last_fullsized":"2023-10-19T12:58:00.275193+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:22:59.082914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"69'1","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.624494+0000","last_change":"2023-10-19T12:57:58.257724+0000","last_active":"2023-10-19T12:58:00.624494+0000","last_peered":"2023-10-19T12:58:00.624494+0000","last_clean":"2023-10-19T12:58:00.624494+0000","last_became_active":"2023-10-19T12:57:58.257432+0000","last_became_peered":"2023-10-19T12:57:58.257432+0000","last_unstale":"2023-10-19T12:58:00.624494+0000","last_undegraded":"2023-10-19T12:58:00.624494+0000","last_fullsized":"2023-10-19T12:58:00.624494+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:21:40.593175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.276683+0000","last_change":"2023-10-19T12:57:59.274011+0000","last_active":"2023-10-19T12:58:00.276683+0000","last_peered":"2023-10-19T12:58:00.276683+0000","last_clean":"2023-10-19T12:58:00.276683+0000","last_became_active":"2023-10-19T12:57:59.273742+0000","last_became_peered":"2023-10-19T12:57:59.273742+0000","last_unstale":"2023-10-19T12:58:00.276683+0000","last_undegraded":"2023-10-19T12:58:00.276683+0000","last_fullsized":"2023-10-19T12:58:00.276683+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:36:39.520498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"69'2","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.625189+0000","last_change":"2023-10-19T12:57:58.249377+0000","last_active":"2023-10-19T12:58:00.625189+0000","last_peered":"2023-10-19T12:58:00.625189+0000","last_clean":"2023-10-19T12:58:00.625189+0000","last_became_active":"2023-10-19T12:57:58.249095+0000","last_became_peered":"2023-10-19T12:57:58.249095+0000","last_unstale":"2023-10-19T12:58:00.625189+0000","last_undegraded":"2023-10-19T12:58:00.625189+0000","last_fullsized":"2023-10-19T12:58:00.625189+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:54:31.441800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:01.921233+0000","last_change":"2023-10-19T12:57:59.282010+0000","last_active":"2023-10-19T12:58:01.921233+0000","last_peered":"2023-10-19T12:58:01.921233+0000","last_clean":"2023-10-19T12:58:01.921233+0000","last_became_active":"2023-10-19T12:57:59.281731+0000","last_became_peered":"2023-10-19T12:57:59.281731+0000","last_unstale":"2023-10-19T12:58:01.921233+0000","last_undegraded":"2023-10-19T12:58:01.921233+0000","last_fullsized":"2023-10-19T12:58:01.921233+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:37:05.744377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"69'1","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.629377+0000","last_change":"2023-10-19T12:57:58.255311+0000","last_active":"2023-10-19T12:58:00.629377+0000","last_peered":"2023-10-19T12:58:00.629377+0000","last_clean":"2023-10-19T12:58:00.629377+0000","last_became_active":"2023-10-19T12:57:58.255029+0000","last_became_peered":"2023-10-19T12:57:58.255029+0000","last_unstale":"2023-10-19T12:58:00.629377+0000","last_undegraded":"2023-10-19T12:58:00.629377+0000","last_fullsized":"2023-10-19T12:58:00.629377+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:48:56.789976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.276881+0000","last_change":"2023-10-19T12:57:59.277839+0000","last_active":"2023-10-19T12:58:00.276881+0000","last_peered":"2023-10-19T12:58:00.276881+0000","last_clean":"2023-10-19T12:58:00.276881+0000","last_became_active":"2023-10-19T12:57:59.277539+0000","last_became_peered":"2023-10-19T12:57:59.277539+0000","last_unstale":"2023-10-19T12:58:00.276881+0000","last_undegraded":"2023-10-19T12:58:00.276881+0000","last_fullsized":"2023-10-19T12:58:00.276881+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:47:49.700629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.629355+0000","last_change":"2023-10-19T12:57:58.256158+0000","last_active":"2023-10-19T12:58:00.629355+0000","last_peered":"2023-10-19T12:58:00.629355+0000","last_clean":"2023-10-19T12:58:00.629355+0000","last_became_active":"2023-10-19T12:57:58.255870+0000","last_became_peered":"2023-10-19T12:57:58.255870+0000","last_unstale":"2023-10-19T12:58:00.629355+0000","last_undegraded":"2023-10-19T12:58:00.629355+0000","last_fullsized":"2023-10-19T12:58:00.629355+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:55:15.770193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.272249+0000","last_change":"2023-10-19T12:57:59.273663+0000","last_active":"2023-10-19T12:58:00.272249+0000","last_peered":"2023-10-19T12:58:00.272249+0000","last_clean":"2023-10-19T12:58:00.272249+0000","last_became_active":"2023-10-19T12:57:59.273411+0000","last_became_peered":"2023-10-19T12:57:59.273411+0000","last_unstale":"2023-10-19T12:58:00.272249+0000","last_undegraded":"2023-10-19T12:58:00.272249+0000","last_fullsized":"2023-10-19T12:58:00.272249+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:43:58.424456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"69'2","reported_seq":14,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.276952+0000","last_change":"2023-10-19T12:57:58.254673+0000","last_active":"2023-10-19T12:58:00.276952+0000","last_peered":"2023-10-19T12:58:00.276952+0000","last_clean":"2023-10-19T12:58:00.276952+0000","last_became_active":"2023-10-19T12:57:58.254408+0000","last_became_peered":"2023-10-19T12:57:58.254408+0000","last_unstale":"2023-10-19T12:58:00.276952+0000","last_undegraded":"2023-10-19T12:58:00.276952+0000","last_fullsized":"2023-10-19T12:58:00.276952+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:25:49.650520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:01.922410+0000","last_change":"2023-10-19T12:57:59.279156+0000","last_active":"2023-10-19T12:58:01.922410+0000","last_peered":"2023-10-19T12:58:01.922410+0000","last_clean":"2023-10-19T12:58:01.922410+0000","last_became_active":"2023-10-19T12:57:59.278885+0000","last_became_peered":"2023-10-19T12:57:59.278885+0000","last_unstale":"2023-10-19T12:58:01.922410+0000","last_undegraded":"2023-10-19T12:58:01.922410+0000","last_fullsized":"2023-10-19T12:58:01.922410+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:33:13.743238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.274755+0000","last_change":"2023-10-19T12:57:58.252362+0000","last_active":"2023-10-19T12:58:00.274755+0000","last_peered":"2023-10-19T12:58:00.274755+0000","last_clean":"2023-10-19T12:58:00.274755+0000","last_became_active":"2023-10-19T12:57:58.252084+0000","last_became_peered":"2023-10-19T12:57:58.252084+0000","last_unstale":"2023-10-19T12:58:00.274755+0000","last_undegraded":"2023-10-19T12:58:00.274755+0000","last_fullsized":"2023-10-19T12:58:00.274755+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:56:55.149376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:01.921299+0000","last_change":"2023-10-19T12:57:59.275897+0000","last_active":"2023-10-19T12:58:01.921299+0000","last_peered":"2023-10-19T12:58:01.921299+0000","last_clean":"2023-10-19T12:58:01.921299+0000","last_became_active":"2023-10-19T12:57:59.274965+0000","last_became_peered":"2023-10-19T12:57:59.274965+0000","last_unstale":"2023-10-19T12:58:01.921299+0000","last_undegraded":"2023-10-19T12:58:01.921299+0000","last_fullsized":"2023-10-19T12:58:01.921299+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:43:58.424456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.624330+0000","last_change":"2023-10-19T12:57:58.255960+0000","last_active":"2023-10-19T12:58:00.624330+0000","last_peered":"2023-10-19T12:58:00.624330+0000","last_clean":"2023-10-19T12:58:00.624330+0000","last_became_active":"2023-10-19T12:57:58.255655+0000","last_became_peered":"2023-10-19T12:57:58.255655+0000","last_unstale":"2023-10-19T12:58:00.624330+0000","last_undegraded":"2023-10-19T12:58:00.624330+0000","last_fullsized":"2023-10-19T12:58:00.624330+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:35:32.478337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.276919+0000","last_change":"2023-10-19T12:57:59.284564+0000","last_active":"2023-10-19T12:58:00.276919+0000","last_peered":"2023-10-19T12:58:00.276919+0000","last_clean":"2023-10-19T12:58:00.276919+0000","last_became_active":"2023-10-19T12:57:59.278312+0000","last_became_peered":"2023-10-19T12:57:59.278312+0000","last_unstale":"2023-10-19T12:58:00.276919+0000","last_undegraded":"2023-10-19T12:58:00.276919+0000","last_fullsized":"2023-10-19T12:58:00.276919+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:19:34.001505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.625154+0000","last_change":"2023-10-19T12:57:58.248153+0000","last_active":"2023-10-19T12:58:00.625154+0000","last_peered":"2023-10-19T12:58:00.625154+0000","last_clean":"2023-10-19T12:58:00.625154+0000","last_became_active":"2023-10-19T12:57:58.247864+0000","last_became_peered":"2023-10-19T12:57:58.247864+0000","last_unstale":"2023-10-19T12:58:00.625154+0000","last_undegraded":"2023-10-19T12:58:00.625154+0000","last_fullsized":"2023-10-19T12:58:00.625154+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:53:21.688950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.272507+0000","last_change":"2023-10-19T12:57:59.260621+0000","last_active":"2023-10-19T12:58:00.272507+0000","last_peered":"2023-10-19T12:58:00.272507+0000","last_clean":"2023-10-19T12:58:00.272507+0000","last_became_active":"2023-10-19T12:57:59.260352+0000","last_became_peered":"2023-10-19T12:57:59.260352+0000","last_unstale":"2023-10-19T12:58:00.272507+0000","last_undegraded":"2023-10-19T12:58:00.272507+0000","last_fullsized":"2023-10-19T12:58:00.272507+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:56:56.159781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"69'2","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.629404+0000","last_change":"2023-10-19T12:57:58.259426+0000","last_active":"2023-10-19T12:58:00.629404+0000","last_peered":"2023-10-19T12:58:00.629404+0000","last_clean":"2023-10-19T12:58:00.629404+0000","last_became_active":"2023-10-19T12:57:58.259118+0000","last_became_peered":"2023-10-19T12:57:58.259118+0000","last_unstale":"2023-10-19T12:58:00.629404+0000","last_undegraded":"2023-10-19T12:58:00.629404+0000","last_fullsized":"2023-10-19T12:58:00.629404+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:07:09.641522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.276817+0000","last_change":"2023-10-19T12:57:59.271932+0000","last_active":"2023-10-19T12:58:00.276817+0000","last_peered":"2023-10-19T12:58:00.276817+0000","last_clean":"2023-10-19T12:58:00.276817+0000","last_became_active":"2023-10-19T12:57:59.271620+0000","last_became_peered":"2023-10-19T12:57:59.271620+0000","last_unstale":"2023-10-19T12:58:00.276817+0000","last_undegraded":"2023-10-19T12:58:00.276817+0000","last_fullsized":"2023-10-19T12:58:00.276817+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:13:06.950010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.629060+0000","last_change":"2023-10-19T12:57:58.253264+0000","last_active":"2023-10-19T12:58:00.629060+0000","last_peered":"2023-10-19T12:58:00.629060+0000","last_clean":"2023-10-19T12:58:00.629060+0000","last_became_active":"2023-10-19T12:57:58.252867+0000","last_became_peered":"2023-10-19T12:57:58.252867+0000","last_unstale":"2023-10-19T12:58:00.629060+0000","last_undegraded":"2023-10-19T12:58:00.629060+0000","last_fullsized":"2023-10-19T12:58:00.629060+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:08:22.862915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.274916+0000","last_change":"2023-10-19T12:57:59.274237+0000","last_active":"2023-10-19T12:58:00.274916+0000","last_peered":"2023-10-19T12:58:00.274916+0000","last_clean":"2023-10-19T12:58:00.274916+0000","last_became_active":"2023-10-19T12:57:59.273873+0000","last_became_peered":"2023-10-19T12:57:59.273873+0000","last_unstale":"2023-10-19T12:58:00.274916+0000","last_undegraded":"2023-10-19T12:58:00.274916+0000","last_fullsized":"2023-10-19T12:58:00.274916+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:31:14.459655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.624398+0000","last_change":"2023-10-19T12:57:58.253449+0000","last_active":"2023-10-19T12:58:00.624398+0000","last_peered":"2023-10-19T12:58:00.624398+0000","last_clean":"2023-10-19T12:58:00.624398+0000","last_became_active":"2023-10-19T12:57:58.253169+0000","last_became_peered":"2023-10-19T12:57:58.253169+0000","last_unstale":"2023-10-19T12:58:00.624398+0000","last_undegraded":"2023-10-19T12:58:00.624398+0000","last_fullsized":"2023-10-19T12:58:00.624398+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:55:53.005401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:01.920973+0000","last_change":"2023-10-19T12:57:59.280611+0000","last_active":"2023-10-19T12:58:01.920973+0000","last_peered":"2023-10-19T12:58:01.920973+0000","last_clean":"2023-10-19T12:58:01.920973+0000","last_became_active":"2023-10-19T12:57:59.280321+0000","last_became_peered":"2023-10-19T12:57:59.280321+0000","last_unstale":"2023-10-19T12:58:01.920973+0000","last_undegraded":"2023-10-19T12:58:01.920973+0000","last_fullsized":"2023-10-19T12:58:01.920973+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:23:02.269273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.274940+0000","last_change":"2023-10-19T12:57:58.246012+0000","last_active":"2023-10-19T12:58:00.274940+0000","last_peered":"2023-10-19T12:58:00.274940+0000","last_clean":"2023-10-19T12:58:00.274940+0000","last_became_active":"2023-10-19T12:57:58.245739+0000","last_became_peered":"2023-10-19T12:57:58.245739+0000","last_unstale":"2023-10-19T12:58:00.274940+0000","last_undegraded":"2023-10-19T12:58:00.274940+0000","last_fullsized":"2023-10-19T12:58:00.274940+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:50:31.730413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.272446+0000","last_change":"2023-10-19T12:57:59.265660+0000","last_active":"2023-10-19T12:58:00.272446+0000","last_peered":"2023-10-19T12:58:00.272446+0000","last_clean":"2023-10-19T12:58:00.272446+0000","last_became_active":"2023-10-19T12:57:59.265334+0000","last_became_peered":"2023-10-19T12:57:59.265334+0000","last_unstale":"2023-10-19T12:58:00.272446+0000","last_undegraded":"2023-10-19T12:58:00.272446+0000","last_fullsized":"2023-10-19T12:58:00.272446+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:28:49.136836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.274837+0000","last_change":"2023-10-19T12:57:58.255022+0000","last_active":"2023-10-19T12:58:00.274837+0000","last_peered":"2023-10-19T12:58:00.274837+0000","last_clean":"2023-10-19T12:58:00.274837+0000","last_became_active":"2023-10-19T12:57:58.254655+0000","last_became_peered":"2023-10-19T12:57:58.254655+0000","last_unstale":"2023-10-19T12:58:00.274837+0000","last_undegraded":"2023-10-19T12:58:00.274837+0000","last_fullsized":"2023-10-19T12:58:00.274837+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:11:37.863957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:01.921136+0000","last_change":"2023-10-19T12:57:59.279293+0000","last_active":"2023-10-19T12:58:01.921136+0000","last_peered":"2023-10-19T12:58:01.921136+0000","last_clean":"2023-10-19T12:58:01.921136+0000","last_became_active":"2023-10-19T12:57:59.278399+0000","last_became_peered":"2023-10-19T12:57:59.278399+0000","last_unstale":"2023-10-19T12:58:01.921136+0000","last_undegraded":"2023-10-19T12:58:01.921136+0000","last_fullsized":"2023-10-19T12:58:01.921136+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:03:07.244263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.627241+0000","last_change":"2023-10-19T12:57:58.255022+0000","last_active":"2023-10-19T12:58:00.627241+0000","last_peered":"2023-10-19T12:58:00.627241+0000","last_clean":"2023-10-19T12:58:00.627241+0000","last_became_active":"2023-10-19T12:57:58.254610+0000","last_became_peered":"2023-10-19T12:57:58.254610+0000","last_unstale":"2023-10-19T12:58:00.627241+0000","last_undegraded":"2023-10-19T12:58:00.627241+0000","last_fullsized":"2023-10-19T12:58:00.627241+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:13:09.184912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.275180+0000","last_change":"2023-10-19T12:57:59.261263+0000","last_active":"2023-10-19T12:58:00.275180+0000","last_peered":"2023-10-19T12:58:00.275180+0000","last_clean":"2023-10-19T12:58:00.275180+0000","last_became_active":"2023-10-19T12:57:59.260933+0000","last_became_peered":"2023-10-19T12:57:59.260933+0000","last_unstale":"2023-10-19T12:58:00.275180+0000","last_undegraded":"2023-10-19T12:58:00.275180+0000","last_fullsized":"2023-10-19T12:58:00.275180+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:56:19.223989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:00.272344+0000","last_change":"2023-10-19T12:57:58.255799+0000","last_active":"2023-10-19T12:58:00.272344+0000","last_peered":"2023-10-19T12:58:00.272344+0000","last_clean":"2023-10-19T12:58:00.272344+0000","last_became_active":"2023-10-19T12:57:58.255531+0000","last_became_peered":"2023-10-19T12:57:58.255531+0000","last_unstale":"2023-10-19T12:58:00.272344+0000","last_undegraded":"2023-10-19T12:58:00.272344+0000","last_fullsized":"2023-10-19T12:58:00.272344+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:03:06.233858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:01.921231+0000","last_change":"2023-10-19T12:57:59.264409+0000","last_active":"2023-10-19T12:58:01.921231+0000","last_peered":"2023-10-19T12:58:01.921231+0000","last_clean":"2023-10-19T12:58:01.921231+0000","last_became_active":"2023-10-19T12:57:59.263748+0000","last_became_peered":"2023-10-19T12:57:59.263748+0000","last_unstale":"2023-10-19T12:58:01.921231+0000","last_undegraded":"2023-10-19T12:58:01.921231+0000","last_fullsized":"2023-10-19T12:58:01.921231+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:58.229465+0000","last_clean_scrub_stamp":"2023-10-19T12:57:58.229465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:19:17.421557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"70'1","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2023-10-19T12:58:01.921249+0000","last_change":"2023-10-19T12:57:58.254016+0000","last_active":"2023-10-19T12:58:01.921249+0000","last_peered":"2023-10-19T12:58:01.921249+0000","last_clean":"2023-10-19T12:58:01.921249+0000","last_became_active":"2023-10-19T12:57:58.253756+0000","last_became_peered":"2023-10-19T12:57:58.253756+0000","last_unstale":"2023-10-19T12:58:01.921249+0000","last_undegraded":"2023-10-19T12:58:01.921249+0000","last_fullsized":"2023-10-19T12:58:01.921249+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:57:57.219060+0000","last_clean_scrub_stamp":"2023-10-19T12:57:57.219060+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:52:42.753982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"58'139","reported_seq":291,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-19T12:57:59.262495+0000","last_change":"2023-10-19T12:44:08.636613+0000","last_active":"2023-10-19T12:57:59.262495+0000","last_peered":"2023-10-19T12:57:59.262495+0000","last_clean":"2023-10-19T12:57:59.262495+0000","last_became_active":"2023-10-19T12:44:08.635697+0000","last_became_peered":"2023-10-19T12:44:08.635697+0000","last_unstale":"2023-10-19T12:57:59.262495+0000","last_undegraded":"2023-10-19T12:57:59.262495+0000","last_fullsized":"2023-10-19T12:57:59.262495+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_clean_scrub_stamp":"2023-10-19T12:44:07.399738+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:15:23.685485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575019,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58956,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312884,"statfs":{"total":96636764160,"available":96576393216,"internally_reserved":0,"allocated":1441792,"data_stored":1006650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58915275},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:57:38 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.58499999999999996,"15min":0.56100000000000005},"min":{"1min":0.39300000000000002,"5min":0.34499999999999997,"15min":0.248},"max":{"1min":0.67600000000000005,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.59699999999999998,"15min":0.57799999999999996},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.26900000000000002},"max":{"1min":0.76100000000000001,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.39900000000000002}]},{"osd":1,"last update":"Thu Oct 19 12:57:38 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.58699999999999997,"15min":0.56899999999999995},"min":{"1min":0.40300000000000002,"5min":0.371,"15min":0.28199999999999997},"max":{"1min":0.747,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.59699999999999998,"15min":0.57299999999999995},"min":{"1min":0.46200000000000002,"5min":0.34999999999999998,"15min":0.29299999999999998},"max":{"1min":0.66700000000000004,"5min":1.284,"15min":1.284},"last":0.54600000000000004}]},{"osd":2,"last update":"Thu Oct 19 12:57:38 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.63900000000000001,"15min":0.61799999999999999},"min":{"1min":0.434,"5min":0.41699999999999998,"15min":0.375},"max":{"1min":0.78600000000000003,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.65500000000000003,"15min":0.629},"min":{"1min":0.497,"5min":0.44600000000000001,"15min":0.28399999999999997},"max":{"1min":0.80700000000000005,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.50600000000000001}]},{"osd":3,"last update":"Thu Oct 19 12:57:38 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.61899999999999999,"15min":0.59499999999999997},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34399999999999997},"max":{"1min":0.79100000000000004,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.66000000000000003,"15min":0.64700000000000002},"min":{"1min":0.45300000000000001,"5min":0.44400000000000001,"15min":0.39700000000000002},"max":{"1min":0.77900000000000003,"5min":1.329,"15min":1.329},"last":0.52200000000000002}]},{"osd":4,"last update":"Thu Oct 19 12:57:38 2023","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.46899999999999997,"15min":0.46600000000000003},"min":{"1min":0.28599999999999998,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.73399999999999999,"5min":0.75800000000000001,"15min":0.80500000000000005},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.46800000000000003,"15min":0.46999999999999997},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.627,"5min":0.71999999999999997,"15min":0.78900000000000003},"last":0.46500000000000002}]},{"osd":5,"last update":"Thu Oct 19 12:57:38 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.48199999999999998,"15min":0.48799999999999999},"min":{"1min":0.30499999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.75800000000000001,"5min":0.80600000000000005,"15min":0.90400000000000003},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.52300000000000002,"15min":0.51400000000000001},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.308},"max":{"1min":0.77400000000000002,"5min":0.81999999999999995,"15min":0.92200000000000004},"last":0.55900000000000005}]},{"osd":6,"last update":"Thu Oct 19 12:57:38 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54500000000000004,"15min":0.54400000000000004},"min":{"1min":0.34100000000000003,"5min":0.308,"15min":0.308},"max":{"1min":0.80400000000000005,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.53400000000000003,"15min":0.53200000000000003},"min":{"1min":0.30299999999999999,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.72599999999999998,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.36899999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575019,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32012,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339828,"statfs":{"total":96636764160,"available":96603983872,"internally_reserved":0,"allocated":847872,"data_stored":416810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:57:21 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60299999999999998,"15min":0.58999999999999997},"min":{"1min":0.311,"5min":0.311,"15min":0.24399999999999999},"max":{"1min":0.76000000000000001,"5min":0.79900000000000004,"15min":0.81999999999999995},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.60499999999999998,"15min":0.58399999999999996},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.77500000000000002,"5min":0.78000000000000003,"15min":0.81200000000000006},"last":0.45800000000000002}]},{"osd":1,"last update":"Thu Oct 19 12:57:21 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.626,"15min":0.60399999999999998},"min":{"1min":0.376,"5min":0.376,"15min":0.34799999999999998},"max":{"1min":0.80500000000000005,"5min":0.83299999999999996,"15min":0.93899999999999995},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.62,"15min":0.60599999999999998},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.28399999999999997},"max":{"1min":0.79400000000000004,"5min":0.82099999999999995,"15min":1.075},"last":0.503}]},{"osd":2,"last update":"Thu Oct 19 12:57:21 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60799999999999998,"15min":0.58299999999999996},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.26600000000000001},"max":{"1min":0.748,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.63200000000000001,"15min":0.60599999999999998},"min":{"1min":0.40500000000000003,"5min":0.36099999999999999,"15min":0.28399999999999997},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.73299999999999998}]},{"osd":3,"last update":"Thu Oct 19 12:57:21 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.65100000000000002,"15min":0.625},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.32600000000000001},"max":{"1min":0.83099999999999996,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.64700000000000002,"15min":0.63500000000000001},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.81499999999999995,"5min":0.81699999999999995,"15min":1.0900000000000001},"last":0.66300000000000003}]},{"osd":4,"last update":"Thu Oct 19 12:57:21 2023","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.46300000000000002,"15min":0.47599999999999998},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":1.02},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.47799999999999998,"15min":0.49099999999999999},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.247},"max":{"1min":0.65400000000000003,"5min":0.752,"15min":0.80000000000000004},"last":0.67800000000000005}]},{"osd":5,"last update":"Thu Oct 19 12:57:21 2023","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.47899999999999998,"15min":0.48399999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.26600000000000001},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.627},{"interface":"front","average":{"1min":0.505,"5min":0.51400000000000001,"15min":0.52900000000000003},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.312},"max":{"1min":0.78500000000000003,"5min":0.85699999999999998,"15min":0.98799999999999999},"last":0.75700000000000001}]},{"osd":7,"last update":"Thu Oct 19 12:57:21 2023","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.52600000000000002,"15min":0.53100000000000003},"min":{"1min":0.35699999999999998,"5min":0.314,"15min":0.28199999999999997},"max":{"1min":0.76100000000000001,"5min":0.82199999999999995,"15min":1.036},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.53900000000000003,"15min":0.55200000000000005},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":1.1299999999999999},"last":0.71599999999999997}]}]},{"osd":5,"up_from":13,"seq":55834575019,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32004,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339836,"statfs":{"total":96636764160,"available":96603992064,"internally_reserved":0,"allocated":839680,"data_stored":416158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:57:24 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.59199999999999997,"15min":0.58599999999999997},"min":{"1min":0.39800000000000002,"5min":0.26700000000000002,"15min":0.223},"max":{"1min":0.79400000000000004,"5min":0.82699999999999996,"15min":0.88800000000000001},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.60799999999999998,"15min":0.59099999999999997},"min":{"1min":0.39700000000000002,"5min":0.312,"15min":0.27800000000000002},"max":{"1min":0.72699999999999998,"5min":0.80400000000000005,"15min":0.874},"last":0.60399999999999998}]},{"osd":1,"last update":"Thu Oct 19 12:57:24 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.60499999999999998,"15min":0.59599999999999997},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.32800000000000001},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.876},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.61299999999999999,"15min":0.59999999999999998},"min":{"1min":0.439,"5min":0.34699999999999998,"15min":0.34499999999999997},"max":{"1min":0.79500000000000004,"5min":0.80100000000000005,"15min":0.83499999999999996},"last":0.49199999999999999}]},{"osd":2,"last update":"Thu Oct 19 12:57:24 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59399999999999997,"15min":0.57899999999999996},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.307},"max":{"1min":0.70899999999999996,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.62,"15min":0.61399999999999999},"min":{"1min":0.47399999999999998,"5min":0.41999999999999998,"15min":0.33700000000000002},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.93600000000000005},"last":0.45600000000000002}]},{"osd":3,"last update":"Thu Oct 19 12:57:24 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.60999999999999999,"15min":0.59999999999999998},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.39300000000000002},"max":{"1min":0.73199999999999998,"5min":0.77900000000000003,"15min":0.874},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63900000000000001,"15min":0.63400000000000001},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.41999999999999998},"max":{"1min":0.746,"5min":0.80400000000000005,"15min":0.89500000000000002},"last":0.59099999999999997}]},{"osd":4,"last update":"Thu Oct 19 12:57:24 2023","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.34499999999999997,"5min":0.32100000000000001,"15min":0.28899999999999998},"max":{"1min":0.66100000000000003,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.505},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.47099999999999997,"15min":0.47599999999999998},"min":{"1min":0.307,"5min":0.307,"15min":0.23899999999999999},"max":{"1min":0.75900000000000001,"5min":0.79400000000000004,"15min":0.80500000000000005},"last":0.41599999999999998}]},{"osd":6,"last update":"Thu Oct 19 12:57:24 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.47399999999999998,"15min":0.47899999999999998},"min":{"1min":0.34100000000000003,"5min":0.31900000000000001,"15min":0.25900000000000001},"max":{"1min":0.74399999999999999,"5min":0.77500000000000002,"15min":0.79300000000000004},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.50800000000000001,"15min":0.498},"min":{"1min":0.38500000000000001,"5min":0.33600000000000002,"15min":0.30399999999999999},"max":{"1min":0.71899999999999997,"5min":0.79100000000000004,"15min":0.81299999999999994},"last":0.52500000000000002}]},{"osd":7,"last update":"Thu Oct 19 12:57:24 2023","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.47999999999999998,"15min":0.49399999999999999},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.315},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71899999999999997},"last":0.437},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.52700000000000002,"15min":0.53400000000000003},"min":{"1min":0.39700000000000002,"5min":0.34300000000000003,"15min":0.27500000000000002},"max":{"1min":0.77900000000000003,"5min":0.78100000000000003,"15min":0.78500000000000003},"last":0.47399999999999998}]}]},{"osd":4,"up_from":13,"seq":55834575019,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32008,"kb_used_data":824,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339832,"statfs":{"total":96636764160,"available":96603987968,"internally_reserved":0,"allocated":843776,"data_stored":416740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:57:37 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.54900000000000004,"15min":0.55100000000000005},"min":{"1min":0.36499999999999999,"5min":0.32900000000000001,"15min":0.28899999999999998},"max":{"1min":0.72699999999999998,"5min":0.77400000000000002,"15min":0.84799999999999998},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55900000000000005,"15min":0.54800000000000004},"min":{"1min":0.38100000000000001,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.73999999999999999,"5min":0.77800000000000002,"15min":0.78200000000000003},"last":0.45900000000000002}]},{"osd":1,"last update":"Thu Oct 19 12:57:37 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.61099999999999999,"15min":0.60399999999999998},"min":{"1min":0.377,"5min":0.377,"15min":0.34799999999999998},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.86199999999999999},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.61799999999999999,"15min":0.61099999999999999},"min":{"1min":0.44800000000000001,"5min":0.40200000000000002,"15min":0.35799999999999998},"max":{"1min":0.78900000000000003,"5min":0.79500000000000004,"15min":0.873},"last":0.5}]},{"osd":2,"last update":"Thu Oct 19 12:57:37 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.61599999999999999,"15min":0.60899999999999999},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.254},"max":{"1min":0.76000000000000001,"5min":0.85299999999999998,"15min":0.86599999999999999},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.60899999999999999,"15min":0.61699999999999999},"min":{"1min":0.44800000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.75,"5min":0.82999999999999996,"15min":0.85999999999999999},"last":0.54100000000000004}]},{"osd":3,"last update":"Thu Oct 19 12:57:37 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.58499999999999996,"15min":0.57499999999999996},"min":{"1min":0.35199999999999998,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.70199999999999996,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.64400000000000002,"15min":0.63400000000000001},"min":{"1min":0.45700000000000002,"5min":0.41299999999999998,"15min":0.37},"max":{"1min":0.77300000000000002,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.52500000000000002}]},{"osd":5,"last update":"Thu Oct 19 12:57:37 2023","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.437,"15min":0.45200000000000001},"min":{"1min":0.30599999999999999,"5min":0.29099999999999998,"15min":0.249},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.81299999999999994},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.45500000000000002,"15min":0.46400000000000002},"min":{"1min":0.32000000000000001,"5min":0.26400000000000001,"15min":0.23499999999999999},"max":{"1min":0.71999999999999997,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.36399999999999999}]},{"osd":6,"last update":"Thu Oct 19 12:57:37 2023","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45400000000000001,"15min":0.46000000000000002},"min":{"1min":0.32300000000000001,"5min":0.29499999999999998,"15min":0.26400000000000001},"max":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.78600000000000003},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.47899999999999998,"15min":0.505},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.27100000000000002},"max":{"1min":0.76800000000000002,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.33100000000000002}]},{"osd":7,"last update":"Thu Oct 19 12:57:37 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.49199999999999999,"15min":0.50900000000000001},"min":{"1min":0.38600000000000001,"5min":0.307,"15min":0.27500000000000002},"max":{"1min":0.71399999999999997,"5min":0.79100000000000004,"15min":0.80200000000000005},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.53900000000000003,"15min":0.55900000000000005},"min":{"1min":0.41799999999999998,"5min":0.32100000000000001,"15min":0.30199999999999999},"max":{"1min":0.745,"5min":0.80500000000000005,"15min":0.83499999999999996},"last":0.55300000000000005}]}]},{"osd":3,"up_from":13,"seq":55834575018,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36184,"kb_used_data":836,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335656,"statfs":{"total":96636764160,"available":96599711744,"internally_reserved":0,"allocated":856064,"data_stored":417792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:57:22 2023","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.40300000000000002,"15min":0.39500000000000002},"min":{"1min":0.248,"5min":0.23599999999999999,"15min":0.20899999999999999},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":1.04},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.42299999999999999,"15min":0.40699999999999997},"min":{"1min":0.28399999999999997,"5min":0.248,"15min":0.245},"max":{"1min":0.57399999999999995,"5min":0.60999999999999999,"15min":0.64600000000000002},"last":0.33300000000000002}]},{"osd":1,"last update":"Thu Oct 19 12:57:22 2023","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.41599999999999998,"15min":0.40400000000000003},"min":{"1min":0.30599999999999999,"5min":0.26100000000000001,"15min":0.22500000000000001},"max":{"1min":0.58299999999999996,"5min":0.621,"15min":0.69899999999999995},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.46100000000000002,"15min":0.442},"min":{"1min":0.32000000000000001,"5min":0.29899999999999999,"15min":0.215},"max":{"1min":0.65200000000000002,"5min":0.65900000000000003,"15min":0.874},"last":0.48799999999999999}]},{"osd":2,"last update":"Thu Oct 19 12:57:22 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.46899999999999997,"15min":0.45900000000000002},"min":{"1min":0.34699999999999998,"5min":0.32500000000000001,"15min":0.26100000000000001},"max":{"1min":0.59399999999999997,"5min":0.65800000000000003,"15min":0.73599999999999999},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.498,"15min":0.48299999999999998},"min":{"1min":0.36299999999999999,"5min":0.34799999999999998,"15min":0.247},"max":{"1min":0.64000000000000001,"5min":0.72599999999999998,"15min":0.96499999999999997},"last":0.44900000000000001}]},{"osd":4,"last update":"Thu Oct 19 12:57:22 2023","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.67700000000000005,"15min":0.65900000000000003},"min":{"1min":0.49399999999999999,"5min":0.45900000000000002,"15min":0.40300000000000002},"max":{"1min":0.90500000000000003,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.70199999999999996,"15min":0.68700000000000006},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.42099999999999999},"max":{"1min":0.91300000000000003,"5min":0.98999999999999999,"15min":0.99399999999999999},"last":0.77800000000000002}]},{"osd":5,"last update":"Thu Oct 19 12:57:22 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.66900000000000004,"15min":0.64900000000000002},"min":{"1min":0.495,"5min":0.495,"15min":0.40500000000000003},"max":{"1min":0.93899999999999995,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.65700000000000003,"15min":0.64500000000000002},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.35899999999999999},"max":{"1min":0.85899999999999999,"5min":0.97299999999999998,"15min":0.99099999999999999},"last":0.79100000000000004}]},{"osd":6,"last update":"Thu Oct 19 12:57:22 2023","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.66500000000000004,"15min":0.65400000000000003},"min":{"1min":0.439,"5min":0.439,"15min":0.41199999999999998},"max":{"1min":0.84699999999999998,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.68100000000000005,"15min":0.66700000000000004},"min":{"1min":0.55200000000000005,"5min":0.496,"15min":0.35899999999999999},"max":{"1min":0.86899999999999999,"5min":1.05,"15min":1.05},"last":0.80500000000000005}]},{"osd":7,"last update":"Thu Oct 19 12:57:22 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.67400000000000004,"15min":0.66500000000000004},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.39800000000000002},"max":{"1min":0.95699999999999996,"5min":1,"15min":1},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.68400000000000005,"15min":0.67100000000000004},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.88500000000000001,"5min":0.93400000000000005,"15min":0.97599999999999998},"last":0.73999999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575020,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54220,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317620,"statfs":{"total":96636764160,"available":96581242880,"internally_reserved":0,"allocated":851968,"data_stored":417770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:57:25 2023","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.38900000000000001,"15min":0.38500000000000001},"min":{"1min":0.254,"5min":0.20899999999999999,"15min":0.187},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.83999999999999997},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.376,"5min":0.41699999999999998,"15min":0.40100000000000002},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.57799999999999996,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.35999999999999999}]},{"osd":1,"last update":"Thu Oct 19 12:57:25 2023","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.40799999999999997,"15min":0.40000000000000002},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.57899999999999996,"5min":0.71099999999999997,"15min":0.80400000000000005},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.44,"15min":0.43099999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.23400000000000001},"max":{"1min":0.57599999999999996,"5min":0.63,"15min":0.78600000000000003},"last":0.26200000000000001}]},{"osd":3,"last update":"Thu Oct 19 12:57:25 2023","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.46700000000000003,"15min":0.45600000000000002},"min":{"1min":0.25900000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.623,"5min":0.68000000000000005,"15min":0.79600000000000004},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.497,"15min":0.48799999999999999},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.87},"last":0.34699999999999998}]},{"osd":4,"last update":"Thu Oct 19 12:57:25 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.64800000000000002,"15min":0.64100000000000001},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":1.0569999999999999},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.66400000000000003,"15min":0.65500000000000003},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.32000000000000001},"max":{"1min":0.91200000000000003,"5min":0.92300000000000004,"15min":1.0780000000000001},"last":0.59199999999999997}]},{"osd":5,"last update":"Thu Oct 19 12:57:25 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.68100000000000005,"15min":0.66000000000000003},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.78600000000000003,"5min":0.96499999999999997,"15min":0.99399999999999999},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.67500000000000004,"15min":0.65700000000000003},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.90200000000000002,"5min":0.94599999999999995,"15min":0.97099999999999997},"last":0.54500000000000004}]},{"osd":6,"last update":"Thu Oct 19 12:57:25 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.67900000000000005,"15min":0.67200000000000004},"min":{"1min":0.441,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.86799999999999999,"5min":1.008,"15min":1.008},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.68200000000000005,"15min":0.66900000000000004},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.93899999999999995,"5min":0.995,"15min":0.995},"last":0.64100000000000001}]},{"osd":7,"last update":"Thu Oct 19 12:57:25 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.69499999999999995,"15min":0.68400000000000005},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.36399999999999999},"max":{"1min":0.98999999999999999,"5min":1.0149999999999999,"15min":1.111},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.69799999999999995,"15min":0.68500000000000005},"min":{"1min":0.42099999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.89700000000000002,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.66300000000000003}]}]},{"osd":1,"up_from":13,"seq":55834575019,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32008,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339832,"statfs":{"total":96636764160,"available":96603987968,"internally_reserved":0,"allocated":847872,"data_stored":416318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:57:32 2023","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.39200000000000002,"15min":0.378},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.187},"max":{"1min":0.59099999999999997,"5min":0.61799999999999999,"15min":0.66200000000000003},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.443,"15min":0.42199999999999999},"min":{"1min":0.30499999999999999,"5min":0.28000000000000003,"15min":0.21199999999999999},"max":{"1min":0.63600000000000001,"5min":0.68799999999999994,"15min":0.69799999999999995},"last":0.30199999999999999}]},{"osd":2,"last update":"Thu Oct 19 12:57:32 2023","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.44600000000000001,"15min":0.42099999999999999},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.20100000000000001},"max":{"1min":0.56000000000000005,"5min":0.68999999999999995,"15min":0.69999999999999996},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.45800000000000002,"15min":0.44400000000000001},"min":{"1min":0.34699999999999998,"5min":0.309,"15min":0.23699999999999999},"max":{"1min":0.60599999999999998,"5min":0.67500000000000004,"15min":0.755},"last":0.371}]},{"osd":3,"last update":"Thu Oct 19 12:57:32 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.49099999999999999,"15min":0.47699999999999998},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.26200000000000001},"max":{"1min":0.63500000000000001,"5min":0.65900000000000003,"15min":0.83799999999999997},"last":0.436},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.435,"15min":0.42499999999999999},"min":{"1min":0.29499999999999998,"5min":0.28499999999999998,"15min":0.27000000000000002},"max":{"1min":0.56499999999999995,"5min":0.70699999999999996,"15min":0.80500000000000005},"last":0.28399999999999997}]},{"osd":4,"last update":"Thu Oct 19 12:57:32 2023","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.67500000000000004,"15min":0.65700000000000003},"min":{"1min":0.44,"5min":0.44,"15min":0.39300000000000002},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.66600000000000004,"15min":0.65200000000000002},"min":{"1min":0.495,"5min":0.42599999999999999,"15min":0.314},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":1.1910000000000001},"last":0.68799999999999994}]},{"osd":5,"last update":"Thu Oct 19 12:57:32 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.66700000000000004,"15min":0.65700000000000003},"min":{"1min":0.52300000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.878,"5min":0.878,"15min":0.91800000000000004},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.69099999999999995,"15min":0.67200000000000004},"min":{"1min":0.53700000000000003,"5min":0.47099999999999997,"15min":0.38700000000000001},"max":{"1min":0.88500000000000001,"5min":0.93999999999999995,"15min":1.117},"last":0.67300000000000004}]},{"osd":6,"last update":"Thu Oct 19 12:57:32 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.67700000000000005,"15min":0.66200000000000003},"min":{"1min":0.48399999999999999,"5min":0.36799999999999999,"15min":0.36399999999999999},"max":{"1min":0.92200000000000004,"5min":1.0069999999999999,"15min":1.077},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.69399999999999995,"15min":0.68500000000000005},"min":{"1min":0.47399999999999998,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.995,"5min":0.995,"15min":1.0009999999999999},"last":0.63500000000000001}]},{"osd":7,"last update":"Thu Oct 19 12:57:32 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.69499999999999995,"15min":0.68100000000000005},"min":{"1min":0.54000000000000004,"5min":0.46999999999999997,"15min":0.443},"max":{"1min":0.84499999999999997,"5min":0.97399999999999998,"15min":1.0149999999999999},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.70099999999999996,"15min":0.68300000000000005},"min":{"1min":0.46500000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":1.0489999999999999},"last":0.53900000000000003}]}]},{"osd":0,"up_from":13,"seq":55834575020,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36748,"kb_used_data":1404,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335092,"statfs":{"total":96636764160,"available":96599134208,"internally_reserved":0,"allocated":1437696,"data_stored":1005963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Oct 19 12:57:36 2023","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.40100000000000002,"15min":0.38500000000000001},"min":{"1min":0.28699999999999998,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.442,"15min":0.42399999999999999},"min":{"1min":0.25,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.623,"5min":0.68700000000000006,"15min":0.746},"last":0.442}]},{"osd":2,"last update":"Thu Oct 19 12:57:36 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.46700000000000003,"15min":0.45000000000000001},"min":{"1min":0.33900000000000002,"5min":0.26200000000000001,"15min":0.25800000000000001},"max":{"1min":0.752,"5min":0.752,"15min":0.76300000000000001},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.45100000000000001,"15min":0.42799999999999999},"min":{"1min":0.27500000000000002,"5min":0.216,"15min":0.20699999999999999},"max":{"1min":0.59399999999999997,"5min":0.68600000000000005,"15min":0.76800000000000002},"last":0.42599999999999999}]},{"osd":3,"last update":"Thu Oct 19 12:57:36 2023","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.42299999999999999,"15min":0.40100000000000002},"min":{"1min":0.24299999999999999,"5min":0.23400000000000001,"15min":0.22800000000000001},"max":{"1min":1.2829999999999999,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.496},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.51000000000000001,"15min":0.48799999999999999},"min":{"1min":0.33300000000000002,"5min":0.29399999999999998,"15min":0.29199999999999998},"max":{"1min":1.4990000000000001,"5min":1.4990000000000001,"15min":1.4990000000000001},"last":0.56599999999999995}]},{"osd":4,"last update":"Thu Oct 19 12:57:36 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.65000000000000002,"15min":0.63},"min":{"1min":0.46700000000000003,"5min":0.45100000000000001,"15min":0.34999999999999998},"max":{"1min":1.361,"5min":1.361,"15min":1.361},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.65800000000000003,"15min":0.64100000000000001},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.35999999999999999},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.998},"last":0.748}]},{"osd":5,"last update":"Thu Oct 19 12:57:36 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.69099999999999995,"15min":0.66800000000000004},"min":{"1min":0.44800000000000001,"5min":0.42099999999999999,"15min":0.38},"max":{"1min":1.325,"5min":1.325,"15min":1.325},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.69699999999999995,"15min":0.67800000000000005},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.318},"max":{"1min":0.88400000000000001,"5min":0.98699999999999999,"15min":1.056},"last":0.78400000000000003}]},{"osd":6,"last update":"Thu Oct 19 12:57:36 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.67000000000000004,"15min":0.65600000000000003},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.38500000000000001},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.68200000000000005,"15min":0.65800000000000003},"min":{"1min":0.45500000000000002,"5min":0.41999999999999998,"15min":0.38900000000000001},"max":{"1min":0.995,"5min":0.995,"15min":1.109},"last":0.71999999999999997}]},{"osd":7,"last update":"Thu Oct 19 12:57:36 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.69199999999999995,"15min":0.67700000000000005},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.41099999999999998},"max":{"1min":0.92000000000000004,"5min":0.95099999999999996,"15min":1.0089999999999999},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.67300000000000004,"15min":0.65600000000000003},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.389,"5min":1.389,"15min":1.389},"last":0.76500000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-19T12:58:07.047 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-19T12:58:07.048 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-19T12:58:07.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.195+0000 7fbc95010640 1 -- 172.21.15.57:0/2376871907 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc9006af00 msgr2=0x7fbc9006b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:07.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.195+0000 7fbc95010640 1 --2- 172.21.15.57:0/2376871907 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc9006af00 0x7fbc9006b2e0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fbc80009860 tx=0x7fbc8002bd60 comp rx=0 tx=0).stop 2023-10-19T12:58:07.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.195+0000 7fbc95010640 1 -- 172.21.15.57:0/2376871907 shutdown_connections 2023-10-19T12:58:07.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.195+0000 7fbc95010640 1 --2- 172.21.15.57:0/2376871907 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbc900c96b0 0x7fbc900c9b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:07.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.195+0000 7fbc95010640 1 --2- 172.21.15.57:0/2376871907 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc9006af00 0x7fbc9006b2e0 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:07.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.195+0000 7fbc95010640 1 -- 172.21.15.57:0/2376871907 >> 172.21.15.57:0/2376871907 conn(0x7fbc90069930 msgr2=0x7fbc90069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:58:07.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.195+0000 7fbc95010640 1 -- 172.21.15.57:0/2376871907 shutdown_connections 2023-10-19T12:58:07.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.195+0000 7fbc95010640 1 -- 172.21.15.57:0/2376871907 wait complete. 2023-10-19T12:58:07.202 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.196+0000 7fbc95010640 1 Processor -- start 2023-10-19T12:58:07.202 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.196+0000 7fbc95010640 1 -- start start 2023-10-19T12:58:07.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.197+0000 7fbc95010640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbc9006af00 0x7fbc90152a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:07.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.197+0000 7fbc95010640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc900ab500 0x7fbc901399c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:07.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.197+0000 7fbc95010640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbc900c96b0 0x7fbc90139f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:07.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.197+0000 7fbc95010640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc9013a440 con 0x7fbc900ab500 2023-10-19T12:58:07.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.197+0000 7fbc95010640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc900a9fe0 con 0x7fbc9006af00 2023-10-19T12:58:07.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.197+0000 7fbc95010640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc900a6670 con 0x7fbc900c96b0 2023-10-19T12:58:07.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.197+0000 7fbc87fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc900ab500 0x7fbc901399c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:07.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.197+0000 7fbc87fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc900ab500 0x7fbc901399c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59436/0 (socket says 172.21.15.57:59436) 2023-10-19T12:58:07.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.197+0000 7fbc87fff640 1 -- 172.21.15.57:0/1227128257 learned_addr learned my addr 172.21.15.57:0/1227128257 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:58:07.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.197+0000 7fbc8ed9d640 1 --2- 172.21.15.57:0/1227128257 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbc9006af00 0x7fbc90152a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:07.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.197+0000 7fbc8f59e640 1 --2- 172.21.15.57:0/1227128257 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbc900c96b0 0x7fbc90139f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:07.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.197+0000 7fbc87fff640 1 -- 172.21.15.57:0/1227128257 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbc900c96b0 msgr2=0x7fbc90139f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:07.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.197+0000 7fbc87fff640 1 --2- 172.21.15.57:0/1227128257 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbc900c96b0 0x7fbc90139f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:07.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.197+0000 7fbc87fff640 1 -- 172.21.15.57:0/1227128257 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbc9006af00 msgr2=0x7fbc90152a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:07.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.197+0000 7fbc87fff640 1 --2- 172.21.15.57:0/1227128257 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbc9006af00 0x7fbc90152a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:07.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.197+0000 7fbc87fff640 1 -- 172.21.15.57:0/1227128257 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc80009490 con 0x7fbc900ab500 2023-10-19T12:58:07.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.197+0000 7fbc87fff640 1 --2- 172.21.15.57:0/1227128257 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc900ab500 0x7fbc901399c0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fbc7800e790 tx=0x7fbc7800ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:58:07.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.197+0000 7fbc877fe640 1 -- 172.21.15.57:0/1227128257 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc78009580 con 0x7fbc900ab500 2023-10-19T12:58:07.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.197+0000 7fbc877fe640 1 -- 172.21.15.57:0/1227128257 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fbc7800cdb0 con 0x7fbc900ab500 2023-10-19T12:58:07.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.197+0000 7fbc95010640 1 -- 172.21.15.57:0/1227128257 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc900a6950 con 0x7fbc900ab500 2023-10-19T12:58:07.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.198+0000 7fbc877fe640 1 -- 172.21.15.57:0/1227128257 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc78010690 con 0x7fbc900ab500 2023-10-19T12:58:07.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.198+0000 7fbc95010640 1 -- 172.21.15.57:0/1227128257 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc900a6f10 con 0x7fbc900ab500 2023-10-19T12:58:07.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.199+0000 7fbc95010640 1 -- 172.21.15.57:0/1227128257 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc9013baf0 con 0x7fbc900ab500 2023-10-19T12:58:07.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.199+0000 7fbc877fe640 1 -- 172.21.15.57:0/1227128257 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 20) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fbc780107f0 con 0x7fbc900ab500 2023-10-19T12:58:07.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.199+0000 7fbc877fe640 1 --2- 172.21.15.57:0/1227128257 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7fbc7006af60 0x7fbc7006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:07.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.200+0000 7fbc877fe640 1 -- 172.21.15.57:0/1227128257 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fbc78014070 con 0x7fbc900ab500 2023-10-19T12:58:07.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.200+0000 7fbc8ed9d640 1 --2- 172.21.15.57:0/1227128257 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7fbc7006af60 0x7fbc7006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:07.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.200+0000 7fbc8ed9d640 1 --2- 172.21.15.57:0/1227128257 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7fbc7006af60 0x7fbc7006d420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fbc80036000 tx=0x7fbc80004730 comp rx=0 tx=0).ready entity=mgr.5809 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:58:07.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.204+0000 7fbc877fe640 1 -- 172.21.15.57:0/1227128257 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fbc78051ca0 con 0x7fbc900ab500 2023-10-19T12:58:07.452 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.449+0000 7fbc95010640 1 -- 172.21.15.57:0/1227128257 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fbc900690e0 con 0x7fbc900ab500 2023-10-19T12:58:07.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.451+0000 7fbc877fe640 1 -- 172.21.15.57:0/1227128257 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v22) v1 ==== 190+0+147 (secure 0 0 0) 0x7fbc780551e0 con 0x7fbc900ab500 2023-10-19T12:58:07.455 INFO:teuthology.orchestra.run.smithi057.stdout:[client.0] 2023-10-19T12:58:07.455 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQBkJDFlg9yuExAATxiWDfS+p07G1F0pORbJEg== 2023-10-19T12:58:07.455 INFO:teuthology.orchestra.run.smithi057.stdout: caps mds = "allow" 2023-10-19T12:58:07.455 INFO:teuthology.orchestra.run.smithi057.stdout: caps mon = "allow r" 2023-10-19T12:58:07.456 INFO:teuthology.orchestra.run.smithi057.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-19T12:58:07.456 INFO:teuthology.orchestra.run.smithi057.stderr:updated caps for client.0 2023-10-19T12:58:07.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.453+0000 7fbc95010640 1 -- 172.21.15.57:0/1227128257 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7fbc7006af60 msgr2=0x7fbc7006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:07.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.453+0000 7fbc95010640 1 --2- 172.21.15.57:0/1227128257 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7fbc7006af60 0x7fbc7006d420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fbc80036000 tx=0x7fbc80004730 comp rx=0 tx=0).stop 2023-10-19T12:58:07.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.453+0000 7fbc95010640 1 -- 172.21.15.57:0/1227128257 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc900ab500 msgr2=0x7fbc901399c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:07.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.453+0000 7fbc95010640 1 --2- 172.21.15.57:0/1227128257 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc900ab500 0x7fbc901399c0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fbc7800e790 tx=0x7fbc7800ec50 comp rx=0 tx=0).stop 2023-10-19T12:58:07.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.454+0000 7fbc95010640 1 -- 172.21.15.57:0/1227128257 shutdown_connections 2023-10-19T12:58:07.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.454+0000 7fbc95010640 1 --2- 172.21.15.57:0/1227128257 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7fbc7006af60 0x7fbc7006d420 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:07.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.454+0000 7fbc95010640 1 --2- 172.21.15.57:0/1227128257 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbc900c96b0 0x7fbc90139f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:07.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.454+0000 7fbc95010640 1 --2- 172.21.15.57:0/1227128257 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc900ab500 0x7fbc901399c0 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:07.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.454+0000 7fbc95010640 1 --2- 172.21.15.57:0/1227128257 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbc9006af00 0x7fbc90152a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:07.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.454+0000 7fbc95010640 1 -- 172.21.15.57:0/1227128257 >> 172.21.15.57:0/1227128257 conn(0x7fbc90069930 msgr2=0x7fbc900ab990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:58:07.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.454+0000 7fbc95010640 1 -- 172.21.15.57:0/1227128257 shutdown_connections 2023-10-19T12:58:07.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.454+0000 7fbc95010640 1 -- 172.21.15.57:0/1227128257 wait complete. 2023-10-19T12:58:07.467 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T12:58:07.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.574+0000 7f9122640640 1 -- 172.21.15.57:0/3316138372 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f911c06ae40 msgr2=0x7f911c0b0e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:07.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.574+0000 7f9122640640 1 --2- 172.21.15.57:0/3316138372 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f911c06ae40 0x7f911c0b0e90 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f911c0acd30 tx=0x7f910c02bdb0 comp rx=0 tx=0).stop 2023-10-19T12:58:07.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.574+0000 7f9122640640 1 -- 172.21.15.57:0/3316138372 shutdown_connections 2023-10-19T12:58:07.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.574+0000 7f9122640640 1 --2- 172.21.15.57:0/3316138372 >> v2:172.21.15.179:3300/0 conn(0x7f911c15c1f0 0x7f911c157dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:07.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.574+0000 7f9122640640 1 --2- 172.21.15.57:0/3316138372 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f911c06ae40 0x7f911c0b0e90 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:07.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.574+0000 7f9122640640 1 -- 172.21.15.57:0/3316138372 >> 172.21.15.57:0/3316138372 conn(0x7f911c06fae0 msgr2=0x7f911c06fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:58:07.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.574+0000 7f9122640640 1 -- 172.21.15.57:0/3316138372 shutdown_connections 2023-10-19T12:58:07.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.574+0000 7f9122640640 1 -- 172.21.15.57:0/3316138372 wait complete. 2023-10-19T12:58:07.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.575+0000 7f9122640640 1 Processor -- start 2023-10-19T12:58:07.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.575+0000 7f9122640640 1 -- start start 2023-10-19T12:58:07.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.576+0000 7f9122640640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f911c06ae40 0x7f911c0aa8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:07.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.576+0000 7f9122640640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f911c1579f0 0x7f911c146c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:07.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.576+0000 7f9122640640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f911c15c1f0 0x7f911c159490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:07.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.576+0000 7f9122640640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f911c155700 con 0x7f911c15c1f0 2023-10-19T12:58:07.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.576+0000 7f9122640640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f911c147140 con 0x7f911c1579f0 2023-10-19T12:58:07.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.576+0000 7f9122640640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f911c155340 con 0x7f911c06ae40 2023-10-19T12:58:07.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.576+0000 7f9120bdd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f911c15c1f0 0x7f911c159490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:07.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.576+0000 7f9120bdd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f911c15c1f0 0x7f911c159490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59454/0 (socket says 172.21.15.57:59454) 2023-10-19T12:58:07.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.576+0000 7f9120bdd640 1 -- 172.21.15.57:0/1994341869 learned_addr learned my addr 172.21.15.57:0/1994341869 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:58:07.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.576+0000 7f911b7fe640 1 --2- 172.21.15.57:0/1994341869 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f911c1579f0 0x7f911c146c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:07.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.576+0000 7f911bfff640 1 --2- 172.21.15.57:0/1994341869 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f911c06ae40 0x7f911c0aa8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:07.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.576+0000 7f9120bdd640 1 -- 172.21.15.57:0/1994341869 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f911c06ae40 msgr2=0x7f911c0aa8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:07.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.576+0000 7f9120bdd640 1 --2- 172.21.15.57:0/1994341869 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f911c06ae40 0x7f911c0aa8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:07.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.576+0000 7f9120bdd640 1 -- 172.21.15.57:0/1994341869 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f911c1579f0 msgr2=0x7f911c146c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:07.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.576+0000 7f9120bdd640 1 --2- 172.21.15.57:0/1994341869 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f911c1579f0 0x7f911c146c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:07.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.576+0000 7f9120bdd640 1 -- 172.21.15.57:0/1994341869 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f910c009560 con 0x7f911c15c1f0 2023-10-19T12:58:07.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.576+0000 7f911bfff640 1 --2- 172.21.15.57:0/1994341869 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f911c06ae40 0x7f911c0aa8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:58:07.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.576+0000 7f911b7fe640 1 --2- 172.21.15.57:0/1994341869 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f911c1579f0 0x7f911c146c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:58:07.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.576+0000 7f9120bdd640 1 --2- 172.21.15.57:0/1994341869 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f911c15c1f0 0x7f911c159490 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f911400cae0 tx=0x7f9114007220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:58:07.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.576+0000 7f9118ff9640 1 -- 172.21.15.57:0/1994341869 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9114007da0 con 0x7f911c15c1f0 2023-10-19T12:58:07.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.576+0000 7f9118ff9640 1 -- 172.21.15.57:0/1994341869 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f9114004b20 con 0x7f911c15c1f0 2023-10-19T12:58:07.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.576+0000 7f9122640640 1 -- 172.21.15.57:0/1994341869 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f911c0b1de0 con 0x7f911c15c1f0 2023-10-19T12:58:07.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.576+0000 7f9118ff9640 1 -- 172.21.15.57:0/1994341869 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9114005020 con 0x7f911c15c1f0 2023-10-19T12:58:07.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.577+0000 7f9122640640 1 -- 172.21.15.57:0/1994341869 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f911c0add20 con 0x7f911c15c1f0 2023-10-19T12:58:07.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.577+0000 7f9122640640 1 -- 172.21.15.57:0/1994341869 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f911c155840 con 0x7f911c15c1f0 2023-10-19T12:58:07.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.578+0000 7f9118ff9640 1 -- 172.21.15.57:0/1994341869 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 20) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f9114005180 con 0x7f911c15c1f0 2023-10-19T12:58:07.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.578+0000 7f9118ff9640 1 --2- 172.21.15.57:0/1994341869 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f910806af90 0x7f910806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:07.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.578+0000 7f9118ff9640 1 -- 172.21.15.57:0/1994341869 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f911408a3d0 con 0x7f911c15c1f0 2023-10-19T12:58:07.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.579+0000 7f911bfff640 1 --2- 172.21.15.57:0/1994341869 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f910806af90 0x7f910806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:07.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.579+0000 7f911bfff640 1 --2- 172.21.15.57:0/1994341869 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f910806af90 0x7f910806d450 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f911c15a1b0 tx=0x7f9110009040 comp rx=0 tx=0).ready entity=mgr.5809 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:58:07.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.583+0000 7f9118ff9640 1 -- 172.21.15.57:0/1994341869 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f911c155840 con 0x7f911c15c1f0 2023-10-19T12:58:07.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.793+0000 7f9122640640 1 -- 172.21.15.57:0/1994341869 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f911c155840 con 0x7f911c15c1f0 2023-10-19T12:58:07.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.794+0000 7f9118ff9640 1 -- 172.21.15.57:0/1994341869 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+4913 (secure 0 0 0) 0x7f911c155840 con 0x7f911c15c1f0 2023-10-19T12:58:07.797 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:58:07.798 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":23,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5594,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/3950728949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":3950728949},{"type":"v1","addr":"172.21.15.179:6837","nonce":3950728949}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":5597,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/3158611991","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":3158611991},{"type":"v1","addr":"172.21.15.179:6836","nonce":3158611991}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":6507,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/352867817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":352867817},{"type":"v1","addr":"172.21.15.57:6834","nonce":352867817}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T12:57:58.606618+0000","modified":"2023-10-19T12:58:00.613725+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6510},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6510":{"gid":6510,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.57:6835/243029526","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":243029526},{"type":"v1","addr":"172.21.15.57:6835","nonce":243029526}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-10-19T12:58:07.798 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 23 2023-10-19T12:58:07.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.796+0000 7f9122640640 1 -- 172.21.15.57:0/1994341869 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f910806af90 msgr2=0x7f910806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:07.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.796+0000 7f9122640640 1 --2- 172.21.15.57:0/1994341869 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f910806af90 0x7f910806d450 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f911c15a1b0 tx=0x7f9110009040 comp rx=0 tx=0).stop 2023-10-19T12:58:07.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.796+0000 7f9122640640 1 -- 172.21.15.57:0/1994341869 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f911c15c1f0 msgr2=0x7f911c159490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:07.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.796+0000 7f9122640640 1 --2- 172.21.15.57:0/1994341869 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f911c15c1f0 0x7f911c159490 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f911400cae0 tx=0x7f9114007220 comp rx=0 tx=0).stop 2023-10-19T12:58:07.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.796+0000 7f9122640640 1 -- 172.21.15.57:0/1994341869 shutdown_connections 2023-10-19T12:58:07.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.796+0000 7f9122640640 1 --2- 172.21.15.57:0/1994341869 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f910806af90 0x7f910806d450 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:07.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.796+0000 7f9122640640 1 --2- 172.21.15.57:0/1994341869 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f911c15c1f0 0x7f911c159490 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:07.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.796+0000 7f9122640640 1 --2- 172.21.15.57:0/1994341869 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f911c1579f0 0x7f911c146c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:07.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.796+0000 7f9122640640 1 --2- 172.21.15.57:0/1994341869 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f911c06ae40 0x7f911c0aa8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:07.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.796+0000 7f9122640640 1 -- 172.21.15.57:0/1994341869 >> 172.21.15.57:0/1994341869 conn(0x7f911c06fae0 msgr2=0x7f911c150610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:58:07.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.796+0000 7f9122640640 1 -- 172.21.15.57:0/1994341869 shutdown_connections 2023-10-19T12:58:07.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:07.796+0000 7f9122640640 1 -- 172.21.15.57:0/1994341869 wait complete. 2023-10-19T12:58:08.212 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-19T12:57:58.606618+0000', 'modified': '2023-10-19T12:58:00.613725+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6510}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6510': {'gid': 6510, 'name': 'c', 'rank': 0, 'incarnation': 21, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.57:6835/243029526', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6833', 'nonce': 243029526}, {'type': 'v1', 'addr': '172.21.15.57:6835', 'nonce': 243029526}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [8], 'metadata_pool': 7, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-19T12:58:08.213 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-19T12:58:08.213 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-10-19T12:58:08.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.324+0000 7f103b84f640 1 -- 172.21.15.57:0/1259173743 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f10340ceb40 msgr2=0x7f10340b0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:08.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.324+0000 7f103b84f640 1 --2- 172.21.15.57:0/1259173743 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f10340ceb40 0x7f10340b0650 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f103408a810 tx=0x7f102c02bd10 comp rx=0 tx=0).stop 2023-10-19T12:58:08.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.324+0000 7f103b84f640 1 -- 172.21.15.57:0/1259173743 shutdown_connections 2023-10-19T12:58:08.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.324+0000 7f103b84f640 1 --2- 172.21.15.57:0/1259173743 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f10340ceb40 0x7f10340b0650 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:08.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.324+0000 7f103b84f640 1 --2- 172.21.15.57:0/1259173743 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f103415b2a0 0x7f10340cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:08.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.324+0000 7f103b84f640 1 --2- 172.21.15.57:0/1259173743 >> v2:172.21.15.179:3300/0 conn(0x7f10340cf340 0x7f10340d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:08.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.324+0000 7f103b84f640 1 -- 172.21.15.57:0/1259173743 >> 172.21.15.57:0/1259173743 conn(0x7f10340d0d60 msgr2=0x7f10340d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:58:08.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.324+0000 7f103b84f640 1 -- 172.21.15.57:0/1259173743 shutdown_connections 2023-10-19T12:58:08.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.324+0000 7f103b84f640 1 -- 172.21.15.57:0/1259173743 wait complete. 2023-10-19T12:58:08.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.326+0000 7f103b84f640 1 Processor -- start 2023-10-19T12:58:08.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.326+0000 7f103b84f640 1 -- start start 2023-10-19T12:58:08.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.326+0000 7f103b84f640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f10340ceb40 0x7f10340b66a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:08.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.326+0000 7f103b84f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f10340cf340 0x7f1034159530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:08.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.326+0000 7f103b84f640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f103415b2a0 0x7f1034159ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:08.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.326+0000 7f103b84f640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f103415c790 con 0x7f10340cf340 2023-10-19T12:58:08.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.326+0000 7f103b84f640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f103415c180 con 0x7f10340ceb40 2023-10-19T12:58:08.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.326+0000 7f103b84f640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10340b6be0 con 0x7f103415b2a0 2023-10-19T12:58:08.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.326+0000 7f1038dea640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f10340cf340 0x7f1034159530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:08.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.326+0000 7f1038dea640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f10340cf340 0x7f1034159530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59464/0 (socket says 172.21.15.57:59464) 2023-10-19T12:58:08.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.326+0000 7f1038dea640 1 -- 172.21.15.57:0/2578196842 learned_addr learned my addr 172.21.15.57:0/2578196842 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:58:08.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.327+0000 7f1038dea640 1 -- 172.21.15.57:0/2578196842 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f103415b2a0 msgr2=0x7f1034159ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:08.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.327+0000 7f1038dea640 1 --2- 172.21.15.57:0/2578196842 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f103415b2a0 0x7f1034159ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:08.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.327+0000 7f1038dea640 1 -- 172.21.15.57:0/2578196842 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f10340ceb40 msgr2=0x7f10340b66a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:08.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.327+0000 7f1038dea640 1 --2- 172.21.15.57:0/2578196842 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f10340ceb40 0x7f10340b66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:08.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.327+0000 7f1038dea640 1 -- 172.21.15.57:0/2578196842 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f102c009560 con 0x7f10340cf340 2023-10-19T12:58:08.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.327+0000 7f1038dea640 1 --2- 172.21.15.57:0/2578196842 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f10340cf340 0x7f1034159530 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f102800d6d0 tx=0x7f102800db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:58:08.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.327+0000 7f1031ffb640 1 -- 172.21.15.57:0/2578196842 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1028005220 con 0x7f10340cf340 2023-10-19T12:58:08.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.327+0000 7f1031ffb640 1 -- 172.21.15.57:0/2578196842 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f1028005490 con 0x7f10340cf340 2023-10-19T12:58:08.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.327+0000 7f103b84f640 1 -- 172.21.15.57:0/2578196842 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10341e5a40 con 0x7f10340cf340 2023-10-19T12:58:08.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.327+0000 7f1031ffb640 1 -- 172.21.15.57:0/2578196842 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1028009a80 con 0x7f10340cf340 2023-10-19T12:58:08.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.327+0000 7f103b84f640 1 -- 172.21.15.57:0/2578196842 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f103415c420 con 0x7f10340cf340 2023-10-19T12:58:08.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.329+0000 7f1031ffb640 1 -- 172.21.15.57:0/2578196842 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 20) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f10280047a0 con 0x7f10340cf340 2023-10-19T12:58:08.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.329+0000 7f1031ffb640 1 --2- 172.21.15.57:0/2578196842 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f102006af60 0x7f102006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:08.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.329+0000 7f1031ffb640 1 -- 172.21.15.57:0/2578196842 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f1028089e50 con 0x7f10340cf340 2023-10-19T12:58:08.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.329+0000 7f10395eb640 1 --2- 172.21.15.57:0/2578196842 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f102006af60 0x7f102006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:08.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.330+0000 7f10395eb640 1 --2- 172.21.15.57:0/2578196842 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f102006af60 0x7f102006d420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f10240092f0 tx=0x7f1024009220 comp rx=0 tx=0).ready entity=mgr.5809 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:58:08.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.330+0000 7f103e3e6740 1 Processor -- start 2023-10-19T12:58:08.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.330+0000 7f103e3e6740 1 -- start start 2023-10-19T12:58:08.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.331+0000 7f103e3e6740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ade0bf9b40 0x55ade0bccae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:08.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.331+0000 7f103e3e6740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ade0bf9f20 0x55ade0bfc8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:08.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.331+0000 7f103e3e6740 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55ade0bcd020 0x55ade0bfce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:08.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.331+0000 7f103e3e6740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ade0ade730 con 0x55ade0bf9f20 2023-10-19T12:58:08.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.331+0000 7f103e3e6740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ade0b3f780 con 0x55ade0bf9b40 2023-10-19T12:58:08.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.331+0000 7f103e3e6740 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ade0b3ce70 con 0x55ade0bcd020 2023-10-19T12:58:08.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.331+0000 7f1038dea640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ade0bf9f20 0x55ade0bfc8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:08.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.331+0000 7f1038dea640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ade0bf9f20 0x55ade0bfc8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59472/0 (socket says 172.21.15.57:59472) 2023-10-19T12:58:08.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.331+0000 7f1038dea640 1 -- 172.21.15.57:0/2968493203 learned_addr learned my addr 172.21.15.57:0/2968493203 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:58:08.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.331+0000 7f10395eb640 1 --2- 172.21.15.57:0/2968493203 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ade0bf9b40 0x55ade0bccae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:08.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.331+0000 7f1039dec640 1 --2- 172.21.15.57:0/2968493203 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55ade0bcd020 0x55ade0bfce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:08.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.331+0000 7f1038dea640 1 -- 172.21.15.57:0/2968493203 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55ade0bcd020 msgr2=0x55ade0bfce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T12:58:08.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.331+0000 7f1038dea640 1 --2- 172.21.15.57:0/2968493203 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55ade0bcd020 0x55ade0bfce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:08.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.331+0000 7f1038dea640 1 -- 172.21.15.57:0/2968493203 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ade0bf9b40 msgr2=0x55ade0bccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T12:58:08.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.331+0000 7f1038dea640 1 --2- 172.21.15.57:0/2968493203 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ade0bf9b40 0x55ade0bccae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:08.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.332+0000 7f1038dea640 1 -- 172.21.15.57:0/2968493203 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ade0b3d3d0 con 0x55ade0bf9f20 2023-10-19T12:58:08.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.332+0000 7f1038dea640 1 --2- 172.21.15.57:0/2968493203 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ade0bf9f20 0x55ade0bfc8d0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f10280584b0 tx=0x7f102805c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da0528bda95a710c server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:58:08.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.332+0000 7f101affd640 1 -- 172.21.15.57:0/2968493203 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10280568f0 con 0x55ade0bf9f20 2023-10-19T12:58:08.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.332+0000 7f101affd640 1 -- 172.21.15.57:0/2968493203 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f1028005640 con 0x55ade0bf9f20 2023-10-19T12:58:08.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.332+0000 7f101affd640 1 -- 172.21.15.57:0/2968493203 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1028088210 con 0x55ade0bf9f20 2023-10-19T12:58:08.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.332+0000 7f103e3e6740 1 -- 172.21.15.57:0/2968493203 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ade0bf9f20 msgr2=0x55ade0bfc8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:08.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.332+0000 7f103e3e6740 1 --2- 172.21.15.57:0/2968493203 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ade0bf9f20 0x55ade0bfc8d0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f10280584b0 tx=0x7f102805c900 comp rx=0 tx=0).stop 2023-10-19T12:58:08.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.332+0000 7f103e3e6740 1 -- 172.21.15.57:0/2968493203 shutdown_connections 2023-10-19T12:58:08.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.332+0000 7f103e3e6740 1 --2- 172.21.15.57:0/2968493203 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55ade0bcd020 0x55ade0bfce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:08.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.332+0000 7f103e3e6740 1 --2- 172.21.15.57:0/2968493203 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ade0bf9f20 0x55ade0bfc8d0 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:08.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.332+0000 7f103e3e6740 1 --2- 172.21.15.57:0/2968493203 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ade0bf9b40 0x55ade0bccae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:08.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.332+0000 7f103e3e6740 1 -- 172.21.15.57:0/2968493203 >> 172.21.15.57:0/2968493203 conn(0x55ade0bced50 msgr2=0x55ade0c178d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:58:08.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.332+0000 7f103e3e6740 1 -- 172.21.15.57:0/2968493203 shutdown_connections 2023-10-19T12:58:08.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.333+0000 7f103e3e6740 1 -- 172.21.15.57:0/2968493203 wait complete. 2023-10-19T12:58:08.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.333+0000 7f103e3e6740 1 -- start start 2023-10-19T12:58:08.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.333+0000 7f103e3e6740 1 Processor -- start 2023-10-19T12:58:08.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.334+0000 7f103e3e6740 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55ade0bf9b40 0x55ade0be7a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:08.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.334+0000 7f103e3e6740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ade0bf9f20 0x55ade0c11f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:08.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.334+0000 7f103e3e6740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ade0c12440 0x55ade0bff440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:08.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.334+0000 7f103e3e6740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ade0b38760 con 0x55ade0c12440 2023-10-19T12:58:08.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.334+0000 7f103e3e6740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ade0af38f0 con 0x55ade0bf9f20 2023-10-19T12:58:08.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.334+0000 7f103e3e6740 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ade0af3590 con 0x55ade0bf9b40 2023-10-19T12:58:08.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.334+0000 7f103e3e6740 10 client.0 _ll_register_callbacks cb 0x55ade0b2fee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-19T12:58:08.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.334+0000 7f1039dec640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ade0c12440 0x55ade0bff440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:08.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.334+0000 7f1039dec640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ade0c12440 0x55ade0bff440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59480/0 (socket says 172.21.15.57:59480) 2023-10-19T12:58:08.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.334+0000 7f1039dec640 1 -- 172.21.15.57:0/41483619 learned_addr learned my addr 172.21.15.57:0/41483619 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:58:08.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.334+0000 7f1038dea640 1 --2- 172.21.15.57:0/41483619 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ade0bf9f20 0x55ade0c11f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:08.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.334+0000 7f10395eb640 1 --2- 172.21.15.57:0/41483619 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55ade0bf9b40 0x55ade0be7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:08.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.335+0000 7f1039dec640 1 -- 172.21.15.57:0/41483619 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55ade0bf9b40 msgr2=0x55ade0be7a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T12:58:08.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.335+0000 7f1039dec640 1 --2- 172.21.15.57:0/41483619 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55ade0bf9b40 0x55ade0be7a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:08.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.335+0000 7f1039dec640 1 -- 172.21.15.57:0/41483619 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ade0bf9f20 msgr2=0x55ade0c11f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T12:58:08.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.335+0000 7f1039dec640 1 --2- 172.21.15.57:0/41483619 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ade0bf9f20 0x55ade0c11f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:08.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.335+0000 7f1039dec640 1 -- 172.21.15.57:0/41483619 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f103415c420 con 0x55ade0c12440 2023-10-19T12:58:08.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.335+0000 7f1039dec640 1 --2- 172.21.15.57:0/41483619 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ade0c12440 0x55ade0bff440 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f102c036040 tx=0x7f102c005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7f4012c0925ab31 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:58:08.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.335+0000 7f1013fff640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2023-10-19T12:58:08.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.335+0000 7f1013fff640 1 -- 172.21.15.57:0/41483619 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f102c02c8c0 con 0x55ade0c12440 2023-10-19T12:58:08.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.335+0000 7f1013fff640 1 -- 172.21.15.57:0/41483619 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(70..70 src has 1..70) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f102c02ca20 con 0x55ade0c12440 2023-10-19T12:58:08.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.335+0000 7f103e3e6740 1 -- 172.21.15.57:0/41483619 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ade0b31740 con 0x55ade0c12440 2023-10-19T12:58:08.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.335+0000 7f1013fff640 1 -- 172.21.15.57:0/41483619 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f102c02cbe0 con 0x55ade0c12440 2023-10-19T12:58:08.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.335+0000 7f103e3e6740 10 client.6663 fetch_fsmap learned FSMap version 23 2023-10-19T12:58:08.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.335+0000 7f103e3e6740 1 -- 172.21.15.57:0/41483619 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=2+}) v3 -- 0x55ade0b36de0 con 0x55ade0c12440 2023-10-19T12:58:08.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.335+0000 7f1013fff640 1 -- 172.21.15.57:0/41483619 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x55ade0b31740 con 0x55ade0c12440 2023-10-19T12:58:08.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.335+0000 7f1013fff640 1 -- 172.21.15.57:0/41483619 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 23) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f102c0035f0 con 0x55ade0c12440 2023-10-19T12:58:08.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.335+0000 7f103e3e6740 10 client.6663 fetch_fsmap finished waiting for FSMap version 23 2023-10-19T12:58:08.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.335+0000 7f103e3e6740 10 client.6663 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-19T12:58:08.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.335+0000 7f103e3e6740 20 client.6663 populate_metadata read hostname 'smithi057' 2023-10-19T12:58:08.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.335+0000 7f103e3e6740 1 --2- 172.21.15.57:0/41483619 >> [v2:172.21.15.57:6833/243029526,v1:172.21.15.57:6835/243029526] conn(0x55ade0be9080 0x55ade0be9460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:08.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.335+0000 7f103e3e6740 4 client.6663 mds_command: new command op to 6510 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-19T12:58:08.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.335+0000 7f103e3e6740 1 -- 172.21.15.57:0/41483619 --> [v2:172.21.15.57:6833/243029526,v1:172.21.15.57:6835/243029526] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ade0b31460 con 0x55ade0be9080 2023-10-19T12:58:08.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.336+0000 7f10395eb640 1 --2- 172.21.15.57:0/41483619 >> [v2:172.21.15.57:6833/243029526,v1:172.21.15.57:6835/243029526] conn(0x55ade0be9080 0x55ade0be9460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:08.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.336+0000 7f1013fff640 10 client.6663 ms_handle_connect on v2:172.21.15.57:6833/243029526 2023-10-19T12:58:08.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.336+0000 7f10395eb640 1 --2- 172.21.15.57:0/41483619 >> [v2:172.21.15.57:6833/243029526,v1:172.21.15.57:6835/243029526] conn(0x55ade0be9080 0x55ade0be9460 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9a9a115087679ce5 server_cookie=a115793c8d41b369 in_seq=0 out_seq=0 2023-10-19T12:58:08.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.336+0000 7f1013fff640 1 -- 172.21.15.57:0/41483619 <== mds.0 v2:172.21.15.57:6833/243029526 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55ade0b31460 con 0x55ade0be9080 2023-10-19T12:58:08.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.336+0000 7f1013fff640 10 client.6663 handle_command_reply: tid=0 2023-10-19T12:58:08.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.336+0000 7f103e3e6740 1 client.6663 shutdown 2023-10-19T12:58:08.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.337+0000 7f103e3e6740 1 -- 172.21.15.57:0/41483619 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ade0c12440 msgr2=0x55ade0bff440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:08.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.337+0000 7f103e3e6740 1 --2- 172.21.15.57:0/41483619 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ade0c12440 0x55ade0bff440 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f102c036040 tx=0x7f102c005d50 comp rx=0 tx=0).stop 2023-10-19T12:58:08.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.337+0000 7f103e3e6740 1 -- 172.21.15.57:0/41483619 shutdown_connections 2023-10-19T12:58:08.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.337+0000 7f103e3e6740 1 --2- 172.21.15.57:0/41483619 >> [v2:172.21.15.57:6833/243029526,v1:172.21.15.57:6835/243029526] conn(0x55ade0be9080 0x55ade0be9460 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:08.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.337+0000 7f103e3e6740 1 --2- 172.21.15.57:0/41483619 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ade0c12440 0x55ade0bff440 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:08.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.337+0000 7f103e3e6740 1 --2- 172.21.15.57:0/41483619 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ade0bf9f20 0x55ade0c11f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:08.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.337+0000 7f1013fff640 0 client.6663 ms_handle_reset on v2:172.21.15.57:6833/243029526 2023-10-19T12:58:08.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.337+0000 7f103e3e6740 1 --2- 172.21.15.57:0/41483619 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55ade0bf9b40 0x55ade0be7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:08.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.337+0000 7f103e3e6740 1 -- 172.21.15.57:0/41483619 >> 172.21.15.57:0/41483619 conn(0x55ade0bcd020 msgr2=0x55ade0bfc3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:58:08.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.337+0000 7f103e3e6740 1 -- 172.21.15.57:0/41483619 shutdown_connections 2023-10-19T12:58:08.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.337+0000 7f103e3e6740 1 -- 172.21.15.57:0/41483619 wait complete. 2023-10-19T12:58:08.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.337+0000 7f103e3e6740 20 client.6663 trim_cache size 0 max 16384 2023-10-19T12:58:08.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.352+0000 7f103e3e6740 1 Processor -- start 2023-10-19T12:58:08.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.352+0000 7f103e3e6740 1 -- start start 2023-10-19T12:58:08.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.353+0000 7f103e3e6740 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55ade0bf9b40 0x55ade0c1ade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:08.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.353+0000 7f103e3e6740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ade0bf9f20 0x55ade0c1b320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:08.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.353+0000 7f103e3e6740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ade0c12440 0x55ade0be9460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:08.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.353+0000 7f103e3e6740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ade0ab9270 con 0x55ade0c12440 2023-10-19T12:58:08.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.353+0000 7f103e3e6740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ade0aeef10 con 0x55ade0bf9f20 2023-10-19T12:58:08.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.353+0000 7f103e3e6740 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ade0b3a640 con 0x55ade0bf9b40 2023-10-19T12:58:08.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.353+0000 7f1039dec640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ade0c12440 0x55ade0be9460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:08.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.353+0000 7f1039dec640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ade0c12440 0x55ade0be9460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59496/0 (socket says 172.21.15.57:59496) 2023-10-19T12:58:08.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.353+0000 7f1039dec640 1 -- 172.21.15.57:0/3697615719 learned_addr learned my addr 172.21.15.57:0/3697615719 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:58:08.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.353+0000 7f10395eb640 1 --2- 172.21.15.57:0/3697615719 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55ade0bf9b40 0x55ade0c1ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:08.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.353+0000 7f1038dea640 1 --2- 172.21.15.57:0/3697615719 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ade0bf9f20 0x55ade0c1b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:08.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.353+0000 7f1039dec640 1 -- 172.21.15.57:0/3697615719 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55ade0bf9b40 msgr2=0x55ade0c1ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T12:58:08.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.353+0000 7f1039dec640 1 --2- 172.21.15.57:0/3697615719 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55ade0bf9b40 0x55ade0c1ade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:08.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.353+0000 7f1039dec640 1 -- 172.21.15.57:0/3697615719 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ade0bf9f20 msgr2=0x55ade0c1b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T12:58:08.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.353+0000 7f1039dec640 1 --2- 172.21.15.57:0/3697615719 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ade0bf9f20 0x55ade0c1b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:08.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.353+0000 7f1039dec640 1 -- 172.21.15.57:0/3697615719 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ade0ad5d00 con 0x55ade0c12440 2023-10-19T12:58:08.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.353+0000 7f1039dec640 1 --2- 172.21.15.57:0/3697615719 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ade0c12440 0x55ade0be9460 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f102c004530 tx=0x7f102c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32bf0b4f6e03aad5 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:58:08.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.354+0000 7f101b7fe640 1 -- 172.21.15.57:0/3697615719 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f102c003b00 con 0x55ade0c12440 2023-10-19T12:58:08.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.354+0000 7f101b7fe640 1 -- 172.21.15.57:0/3697615719 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f102c003d50 con 0x55ade0c12440 2023-10-19T12:58:08.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.354+0000 7f101b7fe640 1 -- 172.21.15.57:0/3697615719 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f102c02cb20 con 0x55ade0c12440 2023-10-19T12:58:08.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.354+0000 7f103e3e6740 1 -- 172.21.15.57:0/3697615719 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ade0c12440 msgr2=0x55ade0be9460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:08.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.354+0000 7f103e3e6740 1 --2- 172.21.15.57:0/3697615719 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ade0c12440 0x55ade0be9460 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f102c004530 tx=0x7f102c004730 comp rx=0 tx=0).stop 2023-10-19T12:58:08.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.354+0000 7f103e3e6740 1 -- 172.21.15.57:0/3697615719 shutdown_connections 2023-10-19T12:58:08.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.354+0000 7f103e3e6740 1 --2- 172.21.15.57:0/3697615719 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ade0c12440 0x55ade0be9460 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:08.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.354+0000 7f103e3e6740 1 --2- 172.21.15.57:0/3697615719 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ade0bf9f20 0x55ade0c1b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:08.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.354+0000 7f103e3e6740 1 --2- 172.21.15.57:0/3697615719 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55ade0bf9b40 0x55ade0c1ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:08.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.354+0000 7f103e3e6740 1 -- 172.21.15.57:0/3697615719 >> 172.21.15.57:0/3697615719 conn(0x55ade0bcd020 msgr2=0x55ade0bd3b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:58:08.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.354+0000 7f103e3e6740 1 -- 172.21.15.57:0/3697615719 shutdown_connections 2023-10-19T12:58:08.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.354+0000 7f103e3e6740 1 -- 172.21.15.57:0/3697615719 wait complete. 2023-10-19T12:58:08.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.355+0000 7f103e3e6740 1 -- start start 2023-10-19T12:58:08.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.355+0000 7f103e3e6740 1 Processor -- start 2023-10-19T12:58:08.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.356+0000 7f103e3e6740 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55ade0bf9b40 0x55ade0be9b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:08.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.356+0000 7f103e3e6740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ade0bf9f20 0x55ade0c7fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:08.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.356+0000 7f103e3e6740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ade0c12440 0x55ade0be79b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:08.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.356+0000 7f103e3e6740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ade0bea090 con 0x55ade0c12440 2023-10-19T12:58:08.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.356+0000 7f103e3e6740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ade0bd2010 con 0x55ade0bf9f20 2023-10-19T12:58:08.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.356+0000 7f103e3e6740 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ade0c0e090 con 0x55ade0bf9b40 2023-10-19T12:58:08.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.356+0000 7f103e3e6740 10 client.0 _ll_register_callbacks cb 0x7f1028058600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-19T12:58:08.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.356+0000 7f1039dec640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ade0c12440 0x55ade0be79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:08.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.356+0000 7f1039dec640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ade0c12440 0x55ade0be79b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59502/0 (socket says 172.21.15.57:59502) 2023-10-19T12:58:08.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.356+0000 7f1039dec640 1 -- 172.21.15.57:0/1638534707 learned_addr learned my addr 172.21.15.57:0/1638534707 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:58:08.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.356+0000 7f10395eb640 1 --2- 172.21.15.57:0/1638534707 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55ade0bf9b40 0x55ade0be9b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:08.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.356+0000 7f1038dea640 1 --2- 172.21.15.57:0/1638534707 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ade0bf9f20 0x55ade0c7fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:08.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.356+0000 7f1039dec640 1 -- 172.21.15.57:0/1638534707 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55ade0bf9b40 msgr2=0x55ade0be9b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T12:58:08.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.356+0000 7f1039dec640 1 --2- 172.21.15.57:0/1638534707 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55ade0bf9b40 0x55ade0be9b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:08.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.356+0000 7f1039dec640 1 -- 172.21.15.57:0/1638534707 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ade0bf9f20 msgr2=0x55ade0c7fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T12:58:08.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.356+0000 7f1039dec640 1 --2- 172.21.15.57:0/1638534707 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ade0bf9f20 0x55ade0c7fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:08.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.356+0000 7f1039dec640 1 -- 172.21.15.57:0/1638534707 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ade0b36de0 con 0x55ade0c12440 2023-10-19T12:58:08.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.356+0000 7f1013fff640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2023-10-19T12:58:08.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.356+0000 7f1039dec640 1 --2- 172.21.15.57:0/1638534707 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ade0c12440 0x55ade0be79b0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f102c054040 tx=0x7f102c02cef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd68b7b296492658 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:58:08.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.356+0000 7f1013fff640 1 -- 172.21.15.57:0/1638534707 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f102c003b00 con 0x55ade0c12440 2023-10-19T12:58:08.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.356+0000 7f1013fff640 1 -- 172.21.15.57:0/1638534707 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(70..70 src has 1..70) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f102c03f470 con 0x55ade0c12440 2023-10-19T12:58:08.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.356+0000 7f103e3e6740 1 -- 172.21.15.57:0/1638534707 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ade0c0f120 con 0x55ade0c12440 2023-10-19T12:58:08.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.357+0000 7f1013fff640 1 -- 172.21.15.57:0/1638534707 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f102c003ec0 con 0x55ade0c12440 2023-10-19T12:58:08.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.357+0000 7f103e3e6740 10 client.6669 fetch_fsmap learned FSMap version 23 2023-10-19T12:58:08.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.357+0000 7f103e3e6740 1 -- 172.21.15.57:0/1638534707 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=2+}) v3 -- 0x55ade0afb4f0 con 0x55ade0c12440 2023-10-19T12:58:08.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.357+0000 7f1013fff640 1 -- 172.21.15.57:0/1638534707 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x55ade0c0f120 con 0x55ade0c12440 2023-10-19T12:58:08.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.357+0000 7f1013fff640 1 -- 172.21.15.57:0/1638534707 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 23) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f102c05b670 con 0x55ade0c12440 2023-10-19T12:58:08.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.357+0000 7f103e3e6740 10 client.6669 fetch_fsmap finished waiting for FSMap version 23 2023-10-19T12:58:08.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.357+0000 7f103e3e6740 10 client.6669 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-19T12:58:08.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.357+0000 7f103e3e6740 20 client.6669 populate_metadata read hostname 'smithi057' 2023-10-19T12:58:08.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.357+0000 7f103e3e6740 1 --2- 172.21.15.57:0/1638534707 >> [v2:172.21.15.57:6833/243029526,v1:172.21.15.57:6835/243029526] conn(0x55ade0c0d990 0x55ade0c1ade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:08.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.357+0000 7f103e3e6740 4 client.6669 mds_command: new command op to 6510 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-19T12:58:08.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.357+0000 7f103e3e6740 1 -- 172.21.15.57:0/1638534707 --> [v2:172.21.15.57:6833/243029526,v1:172.21.15.57:6835/243029526] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7f1028005640 con 0x55ade0c0d990 2023-10-19T12:58:08.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.357+0000 7f10395eb640 1 --2- 172.21.15.57:0/1638534707 >> [v2:172.21.15.57:6833/243029526,v1:172.21.15.57:6835/243029526] conn(0x55ade0c0d990 0x55ade0c1ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:08.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.357+0000 7f1013fff640 10 client.6669 ms_handle_connect on v2:172.21.15.57:6833/243029526 2023-10-19T12:58:08.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.357+0000 7f10395eb640 1 --2- 172.21.15.57:0/1638534707 >> [v2:172.21.15.57:6833/243029526,v1:172.21.15.57:6835/243029526] conn(0x55ade0c0d990 0x55ade0c1ade0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=74f10ce0274618f2 server_cookie=53bf27d206a35534 in_seq=0 out_seq=0 2023-10-19T12:58:08.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.357+0000 7f1013fff640 1 -- 172.21.15.57:0/1638534707 <== mds.0 v2:172.21.15.57:6833/243029526 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x7f1028005640 con 0x55ade0c0d990 2023-10-19T12:58:08.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.357+0000 7f1013fff640 10 client.6669 handle_command_reply: tid=0 2023-10-19T12:58:08.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.358+0000 7f103e3e6740 1 client.6669 shutdown 2023-10-19T12:58:08.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.358+0000 7f103e3e6740 1 -- 172.21.15.57:0/1638534707 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ade0c12440 msgr2=0x55ade0be79b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:08.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.358+0000 7f103e3e6740 1 --2- 172.21.15.57:0/1638534707 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ade0c12440 0x55ade0be79b0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f102c054040 tx=0x7f102c02cef0 comp rx=0 tx=0).stop 2023-10-19T12:58:08.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.358+0000 7f103e3e6740 1 -- 172.21.15.57:0/1638534707 shutdown_connections 2023-10-19T12:58:08.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.358+0000 7f103e3e6740 1 --2- 172.21.15.57:0/1638534707 >> [v2:172.21.15.57:6833/243029526,v1:172.21.15.57:6835/243029526] conn(0x55ade0c0d990 0x55ade0c1ade0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:08.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.358+0000 7f103e3e6740 1 --2- 172.21.15.57:0/1638534707 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ade0c12440 0x55ade0be79b0 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:08.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.358+0000 7f103e3e6740 1 --2- 172.21.15.57:0/1638534707 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ade0bf9f20 0x55ade0c7fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:08.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.358+0000 7f1013fff640 0 client.6669 ms_handle_reset on v2:172.21.15.57:6833/243029526 2023-10-19T12:58:08.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.358+0000 7f103e3e6740 1 --2- 172.21.15.57:0/1638534707 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55ade0bf9b40 0x55ade0be9b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:08.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.358+0000 7f103e3e6740 1 -- 172.21.15.57:0/1638534707 >> 172.21.15.57:0/1638534707 conn(0x55ade0bcd020 msgr2=0x55ade0bd3b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:58:08.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.358+0000 7f103e3e6740 1 -- 172.21.15.57:0/1638534707 shutdown_connections 2023-10-19T12:58:08.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.358+0000 7f103e3e6740 1 -- 172.21.15.57:0/1638534707 wait complete. 2023-10-19T12:58:08.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.358+0000 7f103e3e6740 20 client.6669 trim_cache size 0 max 16384 2023-10-19T12:58:08.382 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2023-10-19T12:58:08.383 INFO:teuthology.orchestra.run.smithi057.stdout: "cluster_fsid": "1e9beb19-3260-4dd7-85ef-5b399ce774ac", 2023-10-19T12:58:08.383 INFO:teuthology.orchestra.run.smithi057.stdout: "whoami": 0, 2023-10-19T12:58:08.383 INFO:teuthology.orchestra.run.smithi057.stdout: "id": 6510, 2023-10-19T12:58:08.383 INFO:teuthology.orchestra.run.smithi057.stdout: "want_state": "up:active", 2023-10-19T12:58:08.383 INFO:teuthology.orchestra.run.smithi057.stdout: "state": "up:active", 2023-10-19T12:58:08.384 INFO:teuthology.orchestra.run.smithi057.stdout: "fs_name": "cephfs", 2023-10-19T12:58:08.384 INFO:teuthology.orchestra.run.smithi057.stdout: "rank_uptime": 9.0949937330000008, 2023-10-19T12:58:08.384 INFO:teuthology.orchestra.run.smithi057.stdout: "mdsmap_epoch": 23, 2023-10-19T12:58:08.384 INFO:teuthology.orchestra.run.smithi057.stdout: "osdmap_epoch": 70, 2023-10-19T12:58:08.384 INFO:teuthology.orchestra.run.smithi057.stdout: "osdmap_epoch_barrier": 70, 2023-10-19T12:58:08.385 INFO:teuthology.orchestra.run.smithi057.stdout: "uptime": 13.824603575999999 2023-10-19T12:58:08.385 INFO:teuthology.orchestra.run.smithi057.stdout:} 2023-10-19T12:58:08.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.358+0000 7f103b84f640 1 -- 172.21.15.57:0/2578196842 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f102006af60 msgr2=0x7f102006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:08.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.358+0000 7f103b84f640 1 --2- 172.21.15.57:0/2578196842 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f102006af60 0x7f102006d420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f10240092f0 tx=0x7f1024009220 comp rx=0 tx=0).stop 2023-10-19T12:58:08.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.358+0000 7f103b84f640 1 -- 172.21.15.57:0/2578196842 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f10340cf340 msgr2=0x7f1034159530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:08.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.358+0000 7f103b84f640 1 --2- 172.21.15.57:0/2578196842 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f10340cf340 0x7f1034159530 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f102800d6d0 tx=0x7f102800db90 comp rx=0 tx=0).stop 2023-10-19T12:58:08.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.358+0000 7f103b84f640 1 -- 172.21.15.57:0/2578196842 shutdown_connections 2023-10-19T12:58:08.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.358+0000 7f103b84f640 1 --2- 172.21.15.57:0/2578196842 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f102006af60 0x7f102006d420 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:08.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.358+0000 7f103b84f640 1 --2- 172.21.15.57:0/2578196842 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f103415b2a0 0x7f1034159ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:08.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.358+0000 7f103b84f640 1 --2- 172.21.15.57:0/2578196842 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f10340cf340 0x7f1034159530 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:08.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.358+0000 7f103b84f640 1 --2- 172.21.15.57:0/2578196842 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f10340ceb40 0x7f10340b66a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:08.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.358+0000 7f103b84f640 1 -- 172.21.15.57:0/2578196842 >> 172.21.15.57:0/2578196842 conn(0x7f10340d0d60 msgr2=0x7f103406b700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:58:08.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.359+0000 7f103b84f640 1 -- 172.21.15.57:0/2578196842 shutdown_connections 2023-10-19T12:58:08.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.359+0000 7f103b84f640 1 -- 172.21.15.57:0/2578196842 wait complete. 2023-10-19T12:58:08.390 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-19T12:58:08.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.496+0000 7f6857444640 1 -- 172.21.15.57:0/3551773054 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f685015b2a0 msgr2=0x7f68500cf720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:08.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.496+0000 7f6857444640 1 --2- 172.21.15.57:0/3551773054 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f685015b2a0 0x7f68500cf720 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f6850089ae0 tx=0x7f684402bd10 comp rx=0 tx=0).stop 2023-10-19T12:58:08.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.496+0000 7f6857444640 1 -- 172.21.15.57:0/3551773054 shutdown_connections 2023-10-19T12:58:08.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.496+0000 7f6857444640 1 --2- 172.21.15.57:0/3551773054 >> v2:172.21.15.179:3300/0 conn(0x7f68500ceb40 0x7f68500b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:08.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.496+0000 7f6857444640 1 --2- 172.21.15.57:0/3551773054 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f685015b2a0 0x7f68500cf720 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:08.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.496+0000 7f6857444640 1 --2- 172.21.15.57:0/3551773054 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f68500cf340 0x7f68500d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:08.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.496+0000 7f6857444640 1 -- 172.21.15.57:0/3551773054 >> 172.21.15.57:0/3551773054 conn(0x7f68500d0d60 msgr2=0x7f68500d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:58:08.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.496+0000 7f6857444640 1 -- 172.21.15.57:0/3551773054 shutdown_connections 2023-10-19T12:58:08.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.496+0000 7f6857444640 1 -- 172.21.15.57:0/3551773054 wait complete. 2023-10-19T12:58:08.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.498+0000 7f6857444640 1 Processor -- start 2023-10-19T12:58:08.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.498+0000 7f6857444640 1 -- start start 2023-10-19T12:58:08.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.498+0000 7f6857444640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f68500ceb40 0x7f685015c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:08.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.498+0000 7f6857444640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f68500cf340 0x7f6850158140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:08.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.498+0000 7f6857444640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f685015b2a0 0x7f68500b5710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:08.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.498+0000 7f6857444640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f685015d930 con 0x7f68500cf340 2023-10-19T12:58:08.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.498+0000 7f6857444640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f685015b740 con 0x7f68500ceb40 2023-10-19T12:58:08.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.498+0000 7f6857444640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68500b6080 con 0x7f685015b2a0 2023-10-19T12:58:08.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.498+0000 7f68549df640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f68500cf340 0x7f6850158140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:08.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.499+0000 7f68549df640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f68500cf340 0x7f6850158140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59518/0 (socket says 172.21.15.57:59518) 2023-10-19T12:58:08.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.499+0000 7f68549df640 1 -- 172.21.15.57:0/3359719232 learned_addr learned my addr 172.21.15.57:0/3359719232 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:58:08.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.499+0000 7f68559e1640 1 --2- 172.21.15.57:0/3359719232 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f685015b2a0 0x7f68500b5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:08.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.499+0000 7f68551e0640 1 --2- 172.21.15.57:0/3359719232 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f68500ceb40 0x7f685015c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:08.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.499+0000 7f68549df640 1 -- 172.21.15.57:0/3359719232 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f685015b2a0 msgr2=0x7f68500b5710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:08.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.499+0000 7f68549df640 1 --2- 172.21.15.57:0/3359719232 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f685015b2a0 0x7f68500b5710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:08.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.499+0000 7f68549df640 1 -- 172.21.15.57:0/3359719232 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f68500ceb40 msgr2=0x7f685015c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:08.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.499+0000 7f68549df640 1 --2- 172.21.15.57:0/3359719232 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f68500ceb40 0x7f685015c1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:08.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.499+0000 7f68549df640 1 -- 172.21.15.57:0/3359719232 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6844009560 con 0x7f68500cf340 2023-10-19T12:58:08.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.499+0000 7f68559e1640 1 --2- 172.21.15.57:0/3359719232 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f685015b2a0 0x7f68500b5710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:58:08.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.499+0000 7f68549df640 1 --2- 172.21.15.57:0/3359719232 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f68500cf340 0x7f6850158140 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f6844009850 tx=0x7f684402d9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:58:08.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.499+0000 7f684dffb640 1 -- 172.21.15.57:0/3359719232 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f684402dde0 con 0x7f68500cf340 2023-10-19T12:58:08.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.499+0000 7f684dffb640 1 -- 172.21.15.57:0/3359719232 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f68440036e0 con 0x7f68500cf340 2023-10-19T12:58:08.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.499+0000 7f684dffb640 1 -- 172.21.15.57:0/3359719232 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6844003ab0 con 0x7f68500cf340 2023-10-19T12:58:08.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.499+0000 7f6857444640 1 -- 172.21.15.57:0/3359719232 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f685015b9e0 con 0x7f68500cf340 2023-10-19T12:58:08.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.499+0000 7f6857444640 1 -- 172.21.15.57:0/3359719232 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68500b6360 con 0x7f68500cf340 2023-10-19T12:58:08.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.501+0000 7f6857444640 1 -- 172.21.15.57:0/3359719232 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f685015bf00 con 0x7f68500cf340 2023-10-19T12:58:08.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.501+0000 7f684dffb640 1 -- 172.21.15.57:0/3359719232 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 20) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6844003c10 con 0x7f68500cf340 2023-10-19T12:58:08.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.501+0000 7f684dffb640 1 --2- 172.21.15.57:0/3359719232 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f683c06af60 0x7f683c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:08.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.501+0000 7f684dffb640 1 -- 172.21.15.57:0/3359719232 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f68440abf40 con 0x7f68500cf340 2023-10-19T12:58:08.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.502+0000 7f68551e0640 1 --2- 172.21.15.57:0/3359719232 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f683c06af60 0x7f683c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:08.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.502+0000 7f68551e0640 1 --2- 172.21.15.57:0/3359719232 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f683c06af60 0x7f683c06d420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f68500b51e0 tx=0x7f6848005f90 comp rx=0 tx=0).ready entity=mgr.5809 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:58:08.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.506+0000 7f684dffb640 1 -- 172.21.15.57:0/3359719232 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f685015bf00 con 0x7f68500cf340 2023-10-19T12:58:08.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.717+0000 7f6857444640 1 -- 172.21.15.57:0/3359719232 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f685015bf00 con 0x7f68500cf340 2023-10-19T12:58:08.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.717+0000 7f684dffb640 1 -- 172.21.15.57:0/3359719232 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v23) v1 ==== 53+0+73 (secure 0 0 0) 0x7f685015bf00 con 0x7f68500cf340 2023-10-19T12:58:08.721 INFO:teuthology.orchestra.run.smithi057.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-19T12:58:08.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.719+0000 7f6857444640 1 -- 172.21.15.57:0/3359719232 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f683c06af60 msgr2=0x7f683c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:08.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.719+0000 7f6857444640 1 --2- 172.21.15.57:0/3359719232 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f683c06af60 0x7f683c06d420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f68500b51e0 tx=0x7f6848005f90 comp rx=0 tx=0).stop 2023-10-19T12:58:08.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.719+0000 7f6857444640 1 -- 172.21.15.57:0/3359719232 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f68500cf340 msgr2=0x7f6850158140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:08.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.719+0000 7f6857444640 1 --2- 172.21.15.57:0/3359719232 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f68500cf340 0x7f6850158140 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f6844009850 tx=0x7f684402d9e0 comp rx=0 tx=0).stop 2023-10-19T12:58:08.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.719+0000 7f6857444640 1 -- 172.21.15.57:0/3359719232 shutdown_connections 2023-10-19T12:58:08.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.719+0000 7f6857444640 1 --2- 172.21.15.57:0/3359719232 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f683c06af60 0x7f683c06d420 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:08.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.719+0000 7f6857444640 1 --2- 172.21.15.57:0/3359719232 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f685015b2a0 0x7f68500b5710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:08.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.719+0000 7f6857444640 1 --2- 172.21.15.57:0/3359719232 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f68500cf340 0x7f6850158140 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:08.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.719+0000 7f6857444640 1 --2- 172.21.15.57:0/3359719232 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f68500ceb40 0x7f685015c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:08.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.719+0000 7f6857444640 1 -- 172.21.15.57:0/3359719232 >> 172.21.15.57:0/3359719232 conn(0x7f68500d0d60 msgr2=0x7f685006b590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:58:08.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.719+0000 7f6857444640 1 -- 172.21.15.57:0/3359719232 shutdown_connections 2023-10-19T12:58:08.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:08.719+0000 7f6857444640 1 -- 172.21.15.57:0/3359719232 wait complete. 2023-10-19T12:58:08.732 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-10-19T12:58:08.733 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-19T12:58:08.733 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi057.front.sepia.ceph.com 2023-10-19T12:58:08.733 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-19T12:58:08.734 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-19T12:58:08.734 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-10-19T12:58:08.734 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-10-19T12:58:08.735 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-19T12:58:08.735 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-19T12:58:08.736 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:58:08.736 DEBUG:teuthology.orchestra.run.smithi057:> ip addr 2023-10-19T12:58:08.752 INFO:teuthology.orchestra.run.smithi057.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-19T12:58:08.752 INFO:teuthology.orchestra.run.smithi057.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-19T12:58:08.752 INFO:teuthology.orchestra.run.smithi057.stdout: inet 127.0.0.1/8 scope host lo 2023-10-19T12:58:08.752 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T12:58:08.753 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 ::1/128 scope host 2023-10-19T12:58:08.753 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T12:58:08.753 INFO:teuthology.orchestra.run.smithi057.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-19T12:58:08.753 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:6b:ff:86 brd ff:ff:ff:ff:ff:ff 2023-10-19T12:58:08.753 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp7s0f0 2023-10-19T12:58:08.754 INFO:teuthology.orchestra.run.smithi057.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-19T12:58:08.754 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:6b:ff:87 brd ff:ff:ff:ff:ff:ff 2023-10-19T12:58:08.754 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp7s0f1 2023-10-19T12:58:08.754 INFO:teuthology.orchestra.run.smithi057.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-19T12:58:08.754 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:bd:15:62 brd ff:ff:ff:ff:ff:ff 2023-10-19T12:58:08.755 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp1s0f0 2023-10-19T12:58:08.755 INFO:teuthology.orchestra.run.smithi057.stdout: inet 172.21.15.57/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-19T12:58:08.755 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft 41822sec preferred_lft 41822sec 2023-10-19T12:58:08.755 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 fe80::ec4:7aff:febd:1562/64 scope link noprefixroute 2023-10-19T12:58:08.755 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T12:58:08.755 INFO:teuthology.orchestra.run.smithi057.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-19T12:58:08.756 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:bd:15:63 brd ff:ff:ff:ff:ff:ff 2023-10-19T12:58:08.756 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp1s0f1 2023-10-19T12:58:08.756 INFO:teuthology.orchestra.run.smithi057.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-19T12:58:08.756 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0e:fe:f1:b5:41:59 brd ff:ff:ff:ff:ff:ff 2023-10-19T12:58:08.756 INFO:teuthology.orchestra.run.smithi057.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-19T12:58:08.757 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T12:58:08.757 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 fe80::64b1:a2ff:feb3:b08/64 scope link 2023-10-19T12:58:08.757 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T12:58:08.757 INFO:teuthology.orchestra.run.smithi057.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-19T12:58:08.757 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0e:fe:f1:b5:41:59 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T12:58:08.758 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 fe80::cfe:f1ff:feb5:4159/64 scope link 2023-10-19T12:58:08.758 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T12:58:08.758 INFO:teuthology.orchestra.run.smithi057.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-19T12:58:08.758 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 62:53:72:9d:52:71 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-19T12:58:08.758 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 fe80::6053:72ff:fe9d:5271/64 scope link 2023-10-19T12:58:08.759 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T12:58:08.759 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:58:08.759 DEBUG:teuthology.orchestra.run.smithi057:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-19T12:58:08.822 INFO:teuthology.orchestra.run.smithi057.stdout:1 2023-10-19T12:58:08.823 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:58:08.824 DEBUG:teuthology.orchestra.run.smithi057:> route 2023-10-19T12:58:08.880 INFO:teuthology.orchestra.run.smithi057.stdout:Kernel IP routing table 2023-10-19T12:58:08.881 INFO:teuthology.orchestra.run.smithi057.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-19T12:58:08.881 INFO:teuthology.orchestra.run.smithi057.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-19T12:58:08.881 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-19T12:58:08.881 INFO:teuthology.orchestra.run.smithi057.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-19T12:58:08.882 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:58:08.883 DEBUG:teuthology.orchestra.run.smithi057:> (cd / && exec bash -c ' 2023-10-19T12:58:08.883 DEBUG:teuthology.orchestra.run.smithi057:> set -e 2023-10-19T12:58:08.883 DEBUG:teuthology.orchestra.run.smithi057:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-19T12:58:08.884 DEBUG:teuthology.orchestra.run.smithi057:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-19T12:58:08.884 DEBUG:teuthology.orchestra.run.smithi057:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-19T12:58:08.884 DEBUG:teuthology.orchestra.run.smithi057:> ') 2023-10-19T12:58:08.987 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:58:08.988 DEBUG:teuthology.orchestra.run.smithi057:> ip netns list 2023-10-19T12:58:09.003 INFO:teuthology.orchestra.run.smithi057.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-19T12:58:09.004 INFO:teuthology.orchestra.run.smithi057.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-19T12:58:09.004 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-19T12:58:09.004 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:58:09.005 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip link set brx.0 up 2023-10-19T12:58:09.074 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-10-19T12:58:09.075 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-19T12:58:09.075 DEBUG:teuthology.orchestra.run.smithi057:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-19T12:58:09.129 INFO:teuthology.orchestra.run.smithi057.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-19T12:58:09.130 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-19T12:58:09.130 DEBUG:teuthology.orchestra.run.smithi057:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-19T12:58:09.184 DEBUG:teuthology.orchestra.run.smithi057:> sudo modprobe fuse 2023-10-19T12:58:09.255 DEBUG:teuthology.orchestra.run.smithi057:> cat /proc/self/mounts | awk '{print $2}' 2023-10-19T12:58:09.311 INFO:teuthology.orchestra.run.smithi057.stdout:/proc 2023-10-19T12:58:09.312 INFO:teuthology.orchestra.run.smithi057.stdout:/sys 2023-10-19T12:58:09.312 INFO:teuthology.orchestra.run.smithi057.stdout:/dev 2023-10-19T12:58:09.312 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/security 2023-10-19T12:58:09.312 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/shm 2023-10-19T12:58:09.312 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/pts 2023-10-19T12:58:09.313 INFO:teuthology.orchestra.run.smithi057.stdout:/run 2023-10-19T12:58:09.313 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/cgroup 2023-10-19T12:58:09.313 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/pstore 2023-10-19T12:58:09.313 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/bpf 2023-10-19T12:58:09.313 INFO:teuthology.orchestra.run.smithi057.stdout:/ 2023-10-19T12:58:09.314 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/selinux 2023-10-19T12:58:09.314 INFO:teuthology.orchestra.run.smithi057.stdout:/proc/sys/fs/binfmt_misc 2023-10-19T12:58:09.314 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/hugepages 2023-10-19T12:58:09.314 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/mqueue 2023-10-19T12:58:09.314 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/debug 2023-10-19T12:58:09.315 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/tracing 2023-10-19T12:58:09.315 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/fuse/connections 2023-10-19T12:58:09.315 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/config 2023-10-19T12:58:09.315 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-sysctl.service 2023-10-19T12:58:09.315 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-10-19T12:58:09.316 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-10-19T12:58:09.316 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/nfs/rpc_pipefs 2023-10-19T12:58:09.316 INFO:teuthology.orchestra.run.smithi057.stdout:/run/user/1000 2023-10-19T12:58:09.316 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph 2023-10-19T12:58:09.316 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-0 2023-10-19T12:58:09.317 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-1 2023-10-19T12:58:09.317 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-2 2023-10-19T12:58:09.317 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-3 2023-10-19T12:58:09.317 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns 2023-10-19T12:58:09.318 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T12:58:09.318 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T12:58:09.318 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-19T12:58:09.318 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-19T12:58:09.319 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:58:09.319 DEBUG:teuthology.orchestra.run.smithi057:> ls /sys/fs/fuse/connections 2023-10-19T12:58:09.366 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-10-19T12:58:09.367 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-10-19T12:58:09.409 DEBUG:teuthology.orchestra.run.smithi057:> sudo modprobe fuse 2023-10-19T12:58:09.440 DEBUG:teuthology.orchestra.run.smithi057:> cat /proc/self/mounts | awk '{print $2}' 2023-10-19T12:58:09.495 INFO:teuthology.orchestra.run.smithi057.stdout:/proc 2023-10-19T12:58:09.495 INFO:teuthology.orchestra.run.smithi057.stdout:/sys 2023-10-19T12:58:09.496 INFO:teuthology.orchestra.run.smithi057.stdout:/dev 2023-10-19T12:58:09.496 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/security 2023-10-19T12:58:09.496 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/shm 2023-10-19T12:58:09.496 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/pts 2023-10-19T12:58:09.496 INFO:teuthology.orchestra.run.smithi057.stdout:/run 2023-10-19T12:58:09.497 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/cgroup 2023-10-19T12:58:09.497 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/pstore 2023-10-19T12:58:09.497 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/bpf 2023-10-19T12:58:09.497 INFO:teuthology.orchestra.run.smithi057.stdout:/ 2023-10-19T12:58:09.497 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/selinux 2023-10-19T12:58:09.497 INFO:teuthology.orchestra.run.smithi057.stdout:/proc/sys/fs/binfmt_misc 2023-10-19T12:58:09.498 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/hugepages 2023-10-19T12:58:09.498 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/mqueue 2023-10-19T12:58:09.498 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/debug 2023-10-19T12:58:09.498 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/tracing 2023-10-19T12:58:09.498 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/fuse/connections 2023-10-19T12:58:09.499 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/config 2023-10-19T12:58:09.499 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-sysctl.service 2023-10-19T12:58:09.499 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-10-19T12:58:09.499 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-10-19T12:58:09.499 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/nfs/rpc_pipefs 2023-10-19T12:58:09.500 INFO:teuthology.orchestra.run.smithi057.stdout:/run/user/1000 2023-10-19T12:58:09.500 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph 2023-10-19T12:58:09.500 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-0 2023-10-19T12:58:09.500 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-1 2023-10-19T12:58:09.501 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-2 2023-10-19T12:58:09.501 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-3 2023-10-19T12:58:09.501 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns 2023-10-19T12:58:09.501 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T12:58:09.501 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T12:58:09.502 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-19T12:58:09.502 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-19T12:58:09.502 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:58:09.502 DEBUG:teuthology.orchestra.run.smithi057:> ls /sys/fs/fuse/connections 2023-10-19T12:58:09.504 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi057.stderr:2023-10-19T12:58:09.498+0000 7fb9fcc3e480 -1 init, newargv = 0x558208f74b10 newargc=15 2023-10-19T12:58:09.504 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi057.stderr:ceph-fuse[113835]: starting ceph client 2023-10-19T12:58:09.506 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi057.stderr:ceph-fuse[113835]: starting fuse 2023-10-19T12:58:09.518 INFO:teuthology.orchestra.run.smithi057.stdout:43 2023-10-19T12:58:09.518 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2023-10-19T12:58:09.519 DEBUG:teuthology.orchestra.run.smithi057:> sudo stdin-killer -- python3 -c ' 2023-10-19T12:58:09.519 DEBUG:teuthology.orchestra.run.smithi057:> import glob 2023-10-19T12:58:09.520 DEBUG:teuthology.orchestra.run.smithi057:> import re 2023-10-19T12:58:09.520 DEBUG:teuthology.orchestra.run.smithi057:> import os 2023-10-19T12:58:09.520 DEBUG:teuthology.orchestra.run.smithi057:> import subprocess 2023-10-19T12:58:09.521 DEBUG:teuthology.orchestra.run.smithi057:> 2023-10-19T12:58:09.521 DEBUG:teuthology.orchestra.run.smithi057:> def _find_admin_socket(client_name): 2023-10-19T12:58:09.521 DEBUG:teuthology.orchestra.run.smithi057:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-10-19T12:58:09.522 DEBUG:teuthology.orchestra.run.smithi057:> files = glob.glob(asok_path) 2023-10-19T12:58:09.522 DEBUG:teuthology.orchestra.run.smithi057:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-10-19T12:58:09.523 DEBUG:teuthology.orchestra.run.smithi057:> 2023-10-19T12:58:09.523 DEBUG:teuthology.orchestra.run.smithi057:> # Given a non-glob path, it better be there 2023-10-19T12:58:09.523 DEBUG:teuthology.orchestra.run.smithi057:> if "*" not in asok_path: 2023-10-19T12:58:09.524 DEBUG:teuthology.orchestra.run.smithi057:> assert(len(files) == 1) 2023-10-19T12:58:09.524 DEBUG:teuthology.orchestra.run.smithi057:> return files[0] 2023-10-19T12:58:09.524 DEBUG:teuthology.orchestra.run.smithi057:> 2023-10-19T12:58:09.525 DEBUG:teuthology.orchestra.run.smithi057:> for f in files: 2023-10-19T12:58:09.525 DEBUG:teuthology.orchestra.run.smithi057:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-10-19T12:58:09.525 DEBUG:teuthology.orchestra.run.smithi057:> if os.path.exists("/proc/{0}".format(pid)): 2023-10-19T12:58:09.526 DEBUG:teuthology.orchestra.run.smithi057:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-10-19T12:58:09.526 DEBUG:teuthology.orchestra.run.smithi057:> contents = proc_f.read() 2023-10-19T12:58:09.526 DEBUG:teuthology.orchestra.run.smithi057:> if mountpoint in contents: 2023-10-19T12:58:09.527 DEBUG:teuthology.orchestra.run.smithi057:> return f 2023-10-19T12:58:09.527 DEBUG:teuthology.orchestra.run.smithi057:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-10-19T12:58:09.527 DEBUG:teuthology.orchestra.run.smithi057:> 2023-10-19T12:58:09.528 DEBUG:teuthology.orchestra.run.smithi057:> print(_find_admin_socket("client.0")) 2023-10-19T12:58:09.528 DEBUG:teuthology.orchestra.run.smithi057:> ' 2023-10-19T12:58:09.643 INFO:teuthology.orchestra.run.smithi057.stdout:/var/run/ceph/ceph-client.0.113835.asok 2023-10-19T12:58:09.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:09 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2023-10-19T12:58:09.653 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.113835.asok 2023-10-19T12:58:09.653 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:58:09.654 DEBUG:teuthology.orchestra.run.smithi057:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.113835.asok status 2023-10-19T12:58:09.794 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2023-10-19T12:58:09.795 INFO:teuthology.orchestra.run.smithi057.stdout: "metadata": { 2023-10-19T12:58:09.795 INFO:teuthology.orchestra.run.smithi057.stdout: "ceph_sha1": "f7420d7f8e55ceea786dcbf536c338ce920c21eb", 2023-10-19T12:58:09.795 INFO:teuthology.orchestra.run.smithi057.stdout: "ceph_version": "ceph version 18.0.0-6786-gf7420d7f (f7420d7f8e55ceea786dcbf536c338ce920c21eb) reef (dev)", 2023-10-19T12:58:09.795 INFO:teuthology.orchestra.run.smithi057.stdout: "entity_id": "0", 2023-10-19T12:58:09.795 INFO:teuthology.orchestra.run.smithi057.stdout: "hostname": "smithi057", 2023-10-19T12:58:09.796 INFO:teuthology.orchestra.run.smithi057.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-10-19T12:58:09.796 INFO:teuthology.orchestra.run.smithi057.stdout: "pid": "113835", 2023-10-19T12:58:09.796 INFO:teuthology.orchestra.run.smithi057.stdout: "root": "/" 2023-10-19T12:58:09.796 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T12:58:09.796 INFO:teuthology.orchestra.run.smithi057.stdout: "dentry_count": 0, 2023-10-19T12:58:09.797 INFO:teuthology.orchestra.run.smithi057.stdout: "dentry_pinned_count": 0, 2023-10-19T12:58:09.797 INFO:teuthology.orchestra.run.smithi057.stdout: "id": 6681, 2023-10-19T12:58:09.797 INFO:teuthology.orchestra.run.smithi057.stdout: "inst": { 2023-10-19T12:58:09.797 INFO:teuthology.orchestra.run.smithi057.stdout: "name": { 2023-10-19T12:58:09.797 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "client", 2023-10-19T12:58:09.797 INFO:teuthology.orchestra.run.smithi057.stdout: "num": 6681 2023-10-19T12:58:09.798 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T12:58:09.798 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": { 2023-10-19T12:58:09.798 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "v1", 2023-10-19T12:58:09.798 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": "192.168.0.1:0", 2023-10-19T12:58:09.798 INFO:teuthology.orchestra.run.smithi057.stdout: "nonce": 368012968 2023-10-19T12:58:09.799 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T12:58:09.799 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T12:58:09.799 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": { 2023-10-19T12:58:09.799 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "v1", 2023-10-19T12:58:09.799 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": "192.168.0.1:0", 2023-10-19T12:58:09.800 INFO:teuthology.orchestra.run.smithi057.stdout: "nonce": 368012968 2023-10-19T12:58:09.800 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T12:58:09.800 INFO:teuthology.orchestra.run.smithi057.stdout: "inst_str": "client.6681 192.168.0.1:0/368012968", 2023-10-19T12:58:09.800 INFO:teuthology.orchestra.run.smithi057.stdout: "addr_str": "192.168.0.1:0/368012968", 2023-10-19T12:58:09.800 INFO:teuthology.orchestra.run.smithi057.stdout: "inode_count": 1, 2023-10-19T12:58:09.800 INFO:teuthology.orchestra.run.smithi057.stdout: "mds_epoch": 23, 2023-10-19T12:58:09.801 INFO:teuthology.orchestra.run.smithi057.stdout: "osd_epoch": 70, 2023-10-19T12:58:09.801 INFO:teuthology.orchestra.run.smithi057.stdout: "osd_epoch_barrier": 0, 2023-10-19T12:58:09.801 INFO:teuthology.orchestra.run.smithi057.stdout: "blocklisted": false, 2023-10-19T12:58:09.801 INFO:teuthology.orchestra.run.smithi057.stdout: "fs_name": "cephfs" 2023-10-19T12:58:09.801 INFO:teuthology.orchestra.run.smithi057.stdout:} 2023-10-19T12:58:09.802 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:58:09.803 DEBUG:teuthology.orchestra.run.smithi057:> stat --file-system '--printf=%T 2023-10-19T12:58:09.803 DEBUG:teuthology.orchestra.run.smithi057:> ' -- /home/ubuntu/cephtest/mnt.0 2023-10-19T12:58:09.819 INFO:teuthology.orchestra.run.smithi057.stdout:fuseblk 2023-10-19T12:58:09.819 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-10-19T12:58:09.820 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:58:09.820 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-19T12:58:09.891 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-19T12:58:10.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.038+0000 7f8c19e83640 1 -- 172.21.15.57:0/2265195008 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8c1413e520 msgr2=0x7f8c1406bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:10.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.038+0000 7f8c19e83640 1 --2- 172.21.15.57:0/2265195008 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8c1413e520 0x7f8c1406bb60 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f8c08005fd0 tx=0x7f8c0802bd10 comp rx=0 tx=0).stop 2023-10-19T12:58:10.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.038+0000 7f8c19e83640 1 -- 172.21.15.57:0/2265195008 shutdown_connections 2023-10-19T12:58:10.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.038+0000 7f8c19e83640 1 --2- 172.21.15.57:0/2265195008 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8c14070af0 0x7f8c140affe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:10.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.038+0000 7f8c19e83640 1 --2- 172.21.15.57:0/2265195008 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8c1413e520 0x7f8c1406bb60 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:10.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.038+0000 7f8c19e83640 1 -- 172.21.15.57:0/2265195008 >> 172.21.15.57:0/2265195008 conn(0x7f8c14069930 msgr2=0x7f8c14069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:58:10.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.038+0000 7f8c19e83640 1 -- 172.21.15.57:0/2265195008 shutdown_connections 2023-10-19T12:58:10.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.038+0000 7f8c19e83640 1 -- 172.21.15.57:0/2265195008 wait complete. 2023-10-19T12:58:10.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.040+0000 7f8c19e83640 1 Processor -- start 2023-10-19T12:58:10.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.040+0000 7f8c19e83640 1 -- start start 2023-10-19T12:58:10.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.040+0000 7f8c19e83640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8c14070af0 0x7f8c140ca870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:10.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.040+0000 7f8c19e83640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8c1413e520 0x7f8c140cadb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:10.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.040+0000 7f8c19e83640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8c14141690 0x7f8c141e0510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:10.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.040+0000 7f8c19e83640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c141f1950 con 0x7f8c1413e520 2023-10-19T12:58:10.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.040+0000 7f8c19e83640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c141f1c30 con 0x7f8c14070af0 2023-10-19T12:58:10.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.040+0000 7f8c19e83640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c141f1f10 con 0x7f8c14141690 2023-10-19T12:58:10.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.041+0000 7f8c12ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8c1413e520 0x7f8c140cadb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:10.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.041+0000 7f8c12ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8c1413e520 0x7f8c140cadb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59544/0 (socket says 172.21.15.57:59544) 2023-10-19T12:58:10.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.041+0000 7f8c12ffd640 1 -- 172.21.15.57:0/3313056877 learned_addr learned my addr 172.21.15.57:0/3313056877 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:58:10.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.041+0000 7f8c13fff640 1 --2- 172.21.15.57:0/3313056877 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8c14141690 0x7f8c141e0510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:10.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.041+0000 7f8c137fe640 1 --2- 172.21.15.57:0/3313056877 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8c14070af0 0x7f8c140ca870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:10.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.041+0000 7f8c12ffd640 1 -- 172.21.15.57:0/3313056877 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8c14141690 msgr2=0x7f8c141e0510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:10.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.041+0000 7f8c12ffd640 1 --2- 172.21.15.57:0/3313056877 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8c14141690 0x7f8c141e0510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:10.049 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.041+0000 7f8c12ffd640 1 -- 172.21.15.57:0/3313056877 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8c14070af0 msgr2=0x7f8c140ca870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:10.049 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.041+0000 7f8c12ffd640 1 --2- 172.21.15.57:0/3313056877 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8c14070af0 0x7f8c140ca870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:10.049 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.041+0000 7f8c12ffd640 1 -- 172.21.15.57:0/3313056877 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c08009560 con 0x7f8c1413e520 2023-10-19T12:58:10.049 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.041+0000 7f8c137fe640 1 --2- 172.21.15.57:0/3313056877 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8c14070af0 0x7f8c140ca870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:58:10.049 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.041+0000 7f8c13fff640 1 --2- 172.21.15.57:0/3313056877 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8c14141690 0x7f8c141e0510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:58:10.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.041+0000 7f8c12ffd640 1 --2- 172.21.15.57:0/3313056877 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8c1413e520 0x7f8c140cadb0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f8c0400b3f0 tx=0x7f8c0400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:58:10.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.041+0000 7f8bfbfff640 1 -- 172.21.15.57:0/3313056877 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c04011830 con 0x7f8c1413e520 2023-10-19T12:58:10.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.041+0000 7f8bfbfff640 1 -- 172.21.15.57:0/3313056877 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f8c04011a80 con 0x7f8c1413e520 2023-10-19T12:58:10.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.041+0000 7f8c19e83640 1 -- 172.21.15.57:0/3313056877 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c141f21f0 con 0x7f8c1413e520 2023-10-19T12:58:10.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.041+0000 7f8bfbfff640 1 -- 172.21.15.57:0/3313056877 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c04011e90 con 0x7f8c1413e520 2023-10-19T12:58:10.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.041+0000 7f8c19e83640 1 -- 172.21.15.57:0/3313056877 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c141f2710 con 0x7f8c1413e520 2023-10-19T12:58:10.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.042+0000 7f8c19e83640 1 -- 172.21.15.57:0/3313056877 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c141f1a90 con 0x7f8c1413e520 2023-10-19T12:58:10.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.043+0000 7f8bfbfff640 1 -- 172.21.15.57:0/3313056877 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 20) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f8c0401a400 con 0x7f8c1413e520 2023-10-19T12:58:10.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.043+0000 7f8bfbfff640 1 --2- 172.21.15.57:0/3313056877 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f8c0006af60 0x7f8c0006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:10.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.043+0000 7f8bfbfff640 1 -- 172.21.15.57:0/3313056877 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f8c04089990 con 0x7f8c1413e520 2023-10-19T12:58:10.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.043+0000 7f8c137fe640 1 --2- 172.21.15.57:0/3313056877 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f8c0006af60 0x7f8c0006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:10.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.044+0000 7f8c137fe640 1 --2- 172.21.15.57:0/3313056877 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f8c0006af60 0x7f8c0006d420 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8c08035000 tx=0x7f8c08004730 comp rx=0 tx=0).ready entity=mgr.5809 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:58:10.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.047+0000 7f8bfbfff640 1 -- 172.21.15.57:0/3313056877 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8c141f1a90 con 0x7f8c1413e520 2023-10-19T12:58:10.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.244+0000 7f8c19e83640 1 -- 172.21.15.57:0/3313056877 --> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f8c141f2430 con 0x7f8c0006af60 2023-10-19T12:58:10.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.246+0000 7f8bfbfff640 1 -- 172.21.15.57:0/3313056877 <== mgr.5809 v2:172.21.15.179:6832/3449041082 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f8c141f2430 con 0x7f8c0006af60 2023-10-19T12:58:10.249 INFO:teuthology.orchestra.run.smithi057.stdout:[ 2023-10-19T12:58:10.249 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T12:58:10.249 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "cephfs" 2023-10-19T12:58:10.249 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T12:58:10.250 INFO:teuthology.orchestra.run.smithi057.stdout:] 2023-10-19T12:58:10.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.248+0000 7f8c19e83640 1 -- 172.21.15.57:0/3313056877 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f8c0006af60 msgr2=0x7f8c0006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:10.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.248+0000 7f8c19e83640 1 --2- 172.21.15.57:0/3313056877 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f8c0006af60 0x7f8c0006d420 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8c08035000 tx=0x7f8c08004730 comp rx=0 tx=0).stop 2023-10-19T12:58:10.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.248+0000 7f8c19e83640 1 -- 172.21.15.57:0/3313056877 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8c1413e520 msgr2=0x7f8c140cadb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:10.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.248+0000 7f8c19e83640 1 --2- 172.21.15.57:0/3313056877 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8c1413e520 0x7f8c140cadb0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f8c0400b3f0 tx=0x7f8c0400b8b0 comp rx=0 tx=0).stop 2023-10-19T12:58:10.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.248+0000 7f8c19e83640 1 -- 172.21.15.57:0/3313056877 shutdown_connections 2023-10-19T12:58:10.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.248+0000 7f8c19e83640 1 --2- 172.21.15.57:0/3313056877 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f8c0006af60 0x7f8c0006d420 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:10.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.248+0000 7f8c19e83640 1 --2- 172.21.15.57:0/3313056877 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8c14141690 0x7f8c141e0510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:10.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.248+0000 7f8c19e83640 1 --2- 172.21.15.57:0/3313056877 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8c1413e520 0x7f8c140cadb0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:10.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.248+0000 7f8c19e83640 1 --2- 172.21.15.57:0/3313056877 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8c14070af0 0x7f8c140ca870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:10.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.248+0000 7f8c19e83640 1 -- 172.21.15.57:0/3313056877 >> 172.21.15.57:0/3313056877 conn(0x7f8c14069930 msgr2=0x7f8c14069d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:58:10.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.248+0000 7f8c19e83640 1 -- 172.21.15.57:0/3313056877 shutdown_connections 2023-10-19T12:58:10.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.248+0000 7f8c19e83640 1 -- 172.21.15.57:0/3313056877 wait complete. 2023-10-19T12:58:10.261 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-19T12:58:10.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.369+0000 7fa57b810640 1 -- 172.21.15.57:0/1789305638 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa5740ceb40 msgr2=0x7fa5740b0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:10.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.369+0000 7fa57b810640 1 --2- 172.21.15.57:0/1789305638 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa5740ceb40 0x7fa5740b0650 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fa56c005fd0 tx=0x7fa56c02bd90 comp rx=0 tx=0).stop 2023-10-19T12:58:10.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.369+0000 7fa57b810640 1 -- 172.21.15.57:0/1789305638 shutdown_connections 2023-10-19T12:58:10.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.369+0000 7fa57b810640 1 --2- 172.21.15.57:0/1789305638 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa57415b2a0 0x7fa5740d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:10.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.369+0000 7fa57b810640 1 --2- 172.21.15.57:0/1789305638 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa5740ceb40 0x7fa5740b0650 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:10.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.369+0000 7fa57b810640 1 -- 172.21.15.57:0/1789305638 >> 172.21.15.57:0/1789305638 conn(0x7fa5740d0d60 msgr2=0x7fa5740d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:58:10.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.369+0000 7fa57b810640 1 -- 172.21.15.57:0/1789305638 shutdown_connections 2023-10-19T12:58:10.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.369+0000 7fa57b810640 1 -- 172.21.15.57:0/1789305638 wait complete. 2023-10-19T12:58:10.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.370+0000 7fa57b810640 1 Processor -- start 2023-10-19T12:58:10.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.371+0000 7fa57b810640 1 -- start start 2023-10-19T12:58:10.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.371+0000 7fa57b810640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa5740ceb40 0x7fa574150db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:10.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.371+0000 7fa57b810640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa5740cf340 0x7fa5740c97a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:10.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.371+0000 7fa57b810640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa57415b2a0 0x7fa5741f47a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:10.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.371+0000 7fa57b810640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5740cafd0 con 0x7fa57415b2a0 2023-10-19T12:58:10.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.371+0000 7fa57b810640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa57415d330 con 0x7fa5740cf340 2023-10-19T12:58:10.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.371+0000 7fa57b810640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5740c7e30 con 0x7fa5740ceb40 2023-10-19T12:58:10.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.371+0000 7fa579dad640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa57415b2a0 0x7fa5741f47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:10.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.371+0000 7fa579dad640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa57415b2a0 0x7fa5741f47a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59560/0 (socket says 172.21.15.57:59560) 2023-10-19T12:58:10.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.371+0000 7fa579dad640 1 -- 172.21.15.57:0/1681134611 learned_addr learned my addr 172.21.15.57:0/1681134611 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:58:10.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.372+0000 7fa579dad640 1 -- 172.21.15.57:0/1681134611 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa5740ceb40 msgr2=0x7fa574150db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:10.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.372+0000 7fa578dab640 1 --2- 172.21.15.57:0/1681134611 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa5740cf340 0x7fa5740c97a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:10.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.372+0000 7fa5795ac640 1 --2- 172.21.15.57:0/1681134611 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa5740ceb40 0x7fa574150db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:10.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.372+0000 7fa579dad640 1 --2- 172.21.15.57:0/1681134611 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa5740ceb40 0x7fa574150db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:10.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.372+0000 7fa579dad640 1 -- 172.21.15.57:0/1681134611 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa5740cf340 msgr2=0x7fa5740c97a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:10.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.372+0000 7fa579dad640 1 --2- 172.21.15.57:0/1681134611 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa5740cf340 0x7fa5740c97a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:10.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.372+0000 7fa579dad640 1 -- 172.21.15.57:0/1681134611 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa56c009560 con 0x7fa57415b2a0 2023-10-19T12:58:10.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.372+0000 7fa578dab640 1 --2- 172.21.15.57:0/1681134611 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa5740cf340 0x7fa5740c97a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:58:10.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.372+0000 7fa5795ac640 1 --2- 172.21.15.57:0/1681134611 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa5740ceb40 0x7fa574150db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T12:58:10.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.372+0000 7fa579dad640 1 --2- 172.21.15.57:0/1681134611 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa57415b2a0 0x7fa5741f47a0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fa56c006010 tx=0x7fa56c02da60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:58:10.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.372+0000 7fa571ffb640 1 -- 172.21.15.57:0/1681134611 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa56c02de60 con 0x7fa57415b2a0 2023-10-19T12:58:10.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.372+0000 7fa571ffb640 1 -- 172.21.15.57:0/1681134611 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fa56c003700 con 0x7fa57415b2a0 2023-10-19T12:58:10.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.372+0000 7fa57b810640 1 -- 172.21.15.57:0/1681134611 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5740c9ce0 con 0x7fa57415b2a0 2023-10-19T12:58:10.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.372+0000 7fa571ffb640 1 -- 172.21.15.57:0/1681134611 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa56c003b10 con 0x7fa57415b2a0 2023-10-19T12:58:10.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.372+0000 7fa57b810640 1 -- 172.21.15.57:0/1681134611 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa57415ca90 con 0x7fa57415b2a0 2023-10-19T12:58:10.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.373+0000 7fa57b810640 1 -- 172.21.15.57:0/1681134611 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5740c7fa0 con 0x7fa57415b2a0 2023-10-19T12:58:10.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.374+0000 7fa571ffb640 1 -- 172.21.15.57:0/1681134611 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 20) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fa56c003c70 con 0x7fa57415b2a0 2023-10-19T12:58:10.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.374+0000 7fa571ffb640 1 --2- 172.21.15.57:0/1681134611 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7fa56006af90 0x7fa56006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:10.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.374+0000 7fa571ffb640 1 -- 172.21.15.57:0/1681134611 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fa56c0ac3d0 con 0x7fa57415b2a0 2023-10-19T12:58:10.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.374+0000 7fa5795ac640 1 --2- 172.21.15.57:0/1681134611 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7fa56006af90 0x7fa56006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:10.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.375+0000 7fa5795ac640 1 --2- 172.21.15.57:0/1681134611 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7fa56006af90 0x7fa56006d450 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa5640012e0 tx=0x7fa564009040 comp rx=0 tx=0).ready entity=mgr.5809 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:58:10.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.378+0000 7fa571ffb640 1 -- 172.21.15.57:0/1681134611 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa5740c7fa0 con 0x7fa57415b2a0 2023-10-19T12:58:10.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:10.601+0000 7fa57b810640 1 -- 172.21.15.57:0/1681134611 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7fa5740c7fa0 con 0x7fa57415b2a0 2023-10-19T12:58:11.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.186+0000 7fa571ffb640 1 -- 172.21.15.57:0/1681134611 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v21) v1 ==== 92+0+0 (secure 0 0 0) 0x7fa5740c7fa0 con 0x7fa57415b2a0 2023-10-19T12:58:11.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.189+0000 7fa57b810640 1 -- 172.21.15.57:0/1681134611 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7fa56006af90 msgr2=0x7fa56006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:11.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.189+0000 7fa57b810640 1 --2- 172.21.15.57:0/1681134611 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7fa56006af90 0x7fa56006d450 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa5640012e0 tx=0x7fa564009040 comp rx=0 tx=0).stop 2023-10-19T12:58:11.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.189+0000 7fa57b810640 1 -- 172.21.15.57:0/1681134611 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa57415b2a0 msgr2=0x7fa5741f47a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:11.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.189+0000 7fa57b810640 1 --2- 172.21.15.57:0/1681134611 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa57415b2a0 0x7fa5741f47a0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fa56c006010 tx=0x7fa56c02da60 comp rx=0 tx=0).stop 2023-10-19T12:58:11.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.189+0000 7fa57b810640 1 -- 172.21.15.57:0/1681134611 shutdown_connections 2023-10-19T12:58:11.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.189+0000 7fa57b810640 1 --2- 172.21.15.57:0/1681134611 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7fa56006af90 0x7fa56006d450 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:11.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.189+0000 7fa57b810640 1 --2- 172.21.15.57:0/1681134611 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa57415b2a0 0x7fa5741f47a0 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:11.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.189+0000 7fa57b810640 1 --2- 172.21.15.57:0/1681134611 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa5740cf340 0x7fa5740c97a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:11.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.189+0000 7fa57b810640 1 --2- 172.21.15.57:0/1681134611 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa5740ceb40 0x7fa574150db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:11.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.189+0000 7fa57b810640 1 -- 172.21.15.57:0/1681134611 >> 172.21.15.57:0/1681134611 conn(0x7fa5740d0d60 msgr2=0x7fa5740b0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:58:11.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.190+0000 7fa57b810640 1 -- 172.21.15.57:0/1681134611 shutdown_connections 2023-10-19T12:58:11.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.190+0000 7fa57b810640 1 -- 172.21.15.57:0/1681134611 wait complete. 2023-10-19T12:58:11.206 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-19T12:58:11.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.320+0000 7fe339324640 1 -- 172.21.15.57:0/3200426127 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe33415e5e0 msgr2=0x7fe3340ce9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:11.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.320+0000 7fe339324640 1 --2- 172.21.15.57:0/3200426127 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe33415e5e0 0x7fe3340ce9a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fe328009830 tx=0x7fe32802bd60 comp rx=0 tx=0).stop 2023-10-19T12:58:11.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.321+0000 7fe339324640 1 -- 172.21.15.57:0/3200426127 shutdown_connections 2023-10-19T12:58:11.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.321+0000 7fe339324640 1 --2- 172.21.15.57:0/3200426127 >> v2:172.21.15.179:3300/0 conn(0x7fe3340b1ca0 0x7fe3340cf1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:11.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.321+0000 7fe339324640 1 --2- 172.21.15.57:0/3200426127 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe33415e5e0 0x7fe3340ce9a0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:11.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.321+0000 7fe339324640 1 -- 172.21.15.57:0/3200426127 >> 172.21.15.57:0/3200426127 conn(0x7fe3340d0c50 msgr2=0x7fe3340d1060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:58:11.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.321+0000 7fe339324640 1 -- 172.21.15.57:0/3200426127 shutdown_connections 2023-10-19T12:58:11.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.321+0000 7fe339324640 1 -- 172.21.15.57:0/3200426127 wait complete. 2023-10-19T12:58:11.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.322+0000 7fe339324640 1 Processor -- start 2023-10-19T12:58:11.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.322+0000 7fe339324640 1 -- start start 2023-10-19T12:58:11.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.323+0000 7fe339324640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe3340b1ca0 0x7fe33409ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:11.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.323+0000 7fe339324640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe3340cf6e0 0x7fe3340b13a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:11.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.323+0000 7fe339324640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe33415e5e0 0x7fe33409e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:11.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.323+0000 7fe339324640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3340b18e0 con 0x7fe33415e5e0 2023-10-19T12:58:11.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.323+0000 7fe339324640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe33409ea70 con 0x7fe3340b1ca0 2023-10-19T12:58:11.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.323+0000 7fe339324640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3340aea00 con 0x7fe3340cf6e0 2023-10-19T12:58:11.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.323+0000 7fe3337fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe33415e5e0 0x7fe33409e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:11.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.323+0000 7fe3327fc640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe3340cf6e0 0x7fe3340b13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:11.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.323+0000 7fe332ffd640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe3340b1ca0 0x7fe33409ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:11.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.323+0000 7fe3337fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe33415e5e0 0x7fe33409e530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59592/0 (socket says 172.21.15.57:59592) 2023-10-19T12:58:11.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.323+0000 7fe3337fe640 1 -- 172.21.15.57:0/590048675 learned_addr learned my addr 172.21.15.57:0/590048675 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:58:11.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.323+0000 7fe3327fc640 1 -- 172.21.15.57:0/590048675 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe3340b1ca0 msgr2=0x7fe33409ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:11.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.323+0000 7fe3327fc640 1 --2- 172.21.15.57:0/590048675 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe3340b1ca0 0x7fe33409ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:11.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.323+0000 7fe3327fc640 1 -- 172.21.15.57:0/590048675 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe33415e5e0 msgr2=0x7fe33409e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:11.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.323+0000 7fe3327fc640 1 --2- 172.21.15.57:0/590048675 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe33415e5e0 0x7fe33409e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:11.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.323+0000 7fe3327fc640 1 -- 172.21.15.57:0/590048675 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe324009490 con 0x7fe3340cf6e0 2023-10-19T12:58:11.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.323+0000 7fe332ffd640 1 --2- 172.21.15.57:0/590048675 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe3340b1ca0 0x7fe33409ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T12:58:11.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.323+0000 7fe3327fc640 1 --2- 172.21.15.57:0/590048675 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe3340cf6e0 0x7fe3340b13a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe32400e9f0 tx=0x7fe32400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:58:11.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.324+0000 7fe31b7fe640 1 -- 172.21.15.57:0/590048675 <== mon.2 v2:172.21.15.179:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe32400cd80 con 0x7fe3340cf6e0 2023-10-19T12:58:11.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.324+0000 7fe31b7fe640 1 -- 172.21.15.57:0/590048675 <== mon.2 v2:172.21.15.179:3301/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fe324004020 con 0x7fe3340cf6e0 2023-10-19T12:58:11.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.324+0000 7fe339324640 1 -- 172.21.15.57:0/590048675 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe328009490 con 0x7fe3340cf6e0 2023-10-19T12:58:11.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.324+0000 7fe31b7fe640 1 -- 172.21.15.57:0/590048675 <== mon.2 v2:172.21.15.179:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe324004410 con 0x7fe3340cf6e0 2023-10-19T12:58:11.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.324+0000 7fe339324640 1 -- 172.21.15.57:0/590048675 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3340aeef0 con 0x7fe3340cf6e0 2023-10-19T12:58:11.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.325+0000 7fe339324640 1 -- 172.21.15.57:0/590048675 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3340690e0 con 0x7fe3340cf6e0 2023-10-19T12:58:11.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.325+0000 7fe31b7fe640 1 -- 172.21.15.57:0/590048675 <== mon.2 v2:172.21.15.179:3301/0 4 ==== mgrmap(e 21) v1 ==== 89348+0+0 (secure 0 0 0) 0x7fe3240035f0 con 0x7fe3340cf6e0 2023-10-19T12:58:11.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.326+0000 7fe31b7fe640 1 --2- 172.21.15.57:0/590048675 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7fe32006afb0 0x7fe32006d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:11.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.326+0000 7fe31b7fe640 1 -- 172.21.15.57:0/590048675 <== mon.2 v2:172.21.15.179:3301/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fe324014070 con 0x7fe3340cf6e0 2023-10-19T12:58:11.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.326+0000 7fe332ffd640 1 -- 172.21.15.57:0/590048675 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7fe32006afb0 msgr2=0x7fe32006d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/3449041082 2023-10-19T12:58:11.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.326+0000 7fe332ffd640 1 --2- 172.21.15.57:0/590048675 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7fe32006afb0 0x7fe32006d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T12:58:11.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.330+0000 7fe31b7fe640 1 -- 172.21.15.57:0/590048675 <== mon.2 v2:172.21.15.179:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe324055250 con 0x7fe3340cf6e0 2023-10-19T12:58:11.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.508+0000 7fe339324640 1 -- 172.21.15.57:0/590048675 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7fe3340b1a30 con 0x7fe3340cf6e0 2023-10-19T12:58:11.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.513+0000 7fe31b7fe640 1 -- 172.21.15.57:0/590048675 <== mon.2 v2:172.21.15.179:3301/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v36)=0 v36) v1 ==== 138+0+0 (secure 0 0 0) 0x7fe324058870 con 0x7fe3340cf6e0 2023-10-19T12:58:11.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.515+0000 7fe339324640 1 -- 172.21.15.57:0/590048675 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7fe32006afb0 msgr2=0x7fe32006d470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T12:58:11.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.515+0000 7fe339324640 1 --2- 172.21.15.57:0/590048675 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7fe32006afb0 0x7fe32006d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:11.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.515+0000 7fe339324640 1 -- 172.21.15.57:0/590048675 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe3340cf6e0 msgr2=0x7fe3340b13a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:11.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.515+0000 7fe339324640 1 --2- 172.21.15.57:0/590048675 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe3340cf6e0 0x7fe3340b13a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe32400e9f0 tx=0x7fe32400eeb0 comp rx=0 tx=0).stop 2023-10-19T12:58:11.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.515+0000 7fe339324640 1 -- 172.21.15.57:0/590048675 shutdown_connections 2023-10-19T12:58:11.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.515+0000 7fe339324640 1 --2- 172.21.15.57:0/590048675 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7fe32006afb0 0x7fe32006d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:11.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.515+0000 7fe339324640 1 --2- 172.21.15.57:0/590048675 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe33415e5e0 0x7fe33409e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:11.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.515+0000 7fe339324640 1 --2- 172.21.15.57:0/590048675 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe3340cf6e0 0x7fe3340b13a0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:11.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.515+0000 7fe339324640 1 --2- 172.21.15.57:0/590048675 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe3340b1ca0 0x7fe33409ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:11.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.515+0000 7fe339324640 1 -- 172.21.15.57:0/590048675 >> 172.21.15.57:0/590048675 conn(0x7fe3340d0c50 msgr2=0x7fe3340b26d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:58:11.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.515+0000 7fe339324640 1 -- 172.21.15.57:0/590048675 shutdown_connections 2023-10-19T12:58:11.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.515+0000 7fe339324640 1 -- 172.21.15.57:0/590048675 wait complete. 2023-10-19T12:58:11.530 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-19T12:58:11.569 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:58:11.568+0000 7f36cf612040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-19T12:58:11.590 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:58:11.588+0000 7fc894012040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-19T12:58:11.614 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:58:11.612+0000 7f36cf612040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-19T12:58:11.638 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:58:11.635+0000 7fc894012040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-19T12:58:11.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.649+0000 7f4ed9410640 1 -- 172.21.15.57:0/3919439954 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4ed406d2d0 msgr2=0x7f4ed406d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:11.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.649+0000 7f4ed9410640 1 --2- 172.21.15.57:0/3919439954 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4ed406d2d0 0x7f4ed406d6b0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f4ed40c9200 tx=0x7f4ec402be40 comp rx=0 tx=0).stop 2023-10-19T12:58:11.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.649+0000 7f4ed9410640 1 -- 172.21.15.57:0/3919439954 shutdown_connections 2023-10-19T12:58:11.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.649+0000 7f4ed9410640 1 --2- 172.21.15.57:0/3919439954 >> v2:172.21.15.179:3300/0 conn(0x7f4ed40ad370 0x7f4ed40ad800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:11.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.649+0000 7f4ed9410640 1 --2- 172.21.15.57:0/3919439954 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4ed406d2d0 0x7f4ed406d6b0 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:11.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.649+0000 7f4ed9410640 1 -- 172.21.15.57:0/3919439954 >> 172.21.15.57:0/3919439954 conn(0x7f4ed40b0670 msgr2=0x7f4ed40b0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:58:11.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.649+0000 7f4ed9410640 1 -- 172.21.15.57:0/3919439954 shutdown_connections 2023-10-19T12:58:11.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.649+0000 7f4ed9410640 1 -- 172.21.15.57:0/3919439954 wait complete. 2023-10-19T12:58:11.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.650+0000 7f4ed9410640 1 Processor -- start 2023-10-19T12:58:11.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.650+0000 7f4ed9410640 1 -- start start 2023-10-19T12:58:11.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.651+0000 7f4ed9410640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4ed406d2d0 0x7f4ed414e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:11.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.651+0000 7f4ed9410640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4ed406f0a0 0x7f4ed40c7e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:11.658 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.651+0000 7f4ed9410640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4ed40ad370 0x7f4ed40af4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:11.658 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.651+0000 7f4ed9410640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed4151150 con 0x7f4ed40ad370 2023-10-19T12:58:11.658 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.651+0000 7f4ed9410640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed414e760 con 0x7f4ed406f0a0 2023-10-19T12:58:11.658 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.651+0000 7f4ed9410640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed40c89d0 con 0x7f4ed406d2d0 2023-10-19T12:58:11.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.651+0000 7f4ed37fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4ed40ad370 0x7f4ed40af4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:11.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.651+0000 7f4ed37fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4ed40ad370 0x7f4ed40af4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59600/0 (socket says 172.21.15.57:59600) 2023-10-19T12:58:11.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.651+0000 7f4ed37fe640 1 -- 172.21.15.57:0/2574816464 learned_addr learned my addr 172.21.15.57:0/2574816464 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:58:11.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.651+0000 7f4ecbfff640 1 --2- 172.21.15.57:0/2574816464 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4ed406f0a0 0x7f4ed40c7e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:11.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.651+0000 7f4ed2ffd640 1 --2- 172.21.15.57:0/2574816464 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4ed406d2d0 0x7f4ed414e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:11.660 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.651+0000 7f4ed37fe640 1 -- 172.21.15.57:0/2574816464 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4ed406d2d0 msgr2=0x7f4ed414e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:11.660 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.651+0000 7f4ed37fe640 1 --2- 172.21.15.57:0/2574816464 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4ed406d2d0 0x7f4ed414e220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:11.660 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.651+0000 7f4ed37fe640 1 -- 172.21.15.57:0/2574816464 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4ed406f0a0 msgr2=0x7f4ed40c7e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:11.660 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.651+0000 7f4ed37fe640 1 --2- 172.21.15.57:0/2574816464 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4ed406f0a0 0x7f4ed40c7e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:11.660 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.651+0000 7f4ed37fe640 1 -- 172.21.15.57:0/2574816464 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ec4009560 con 0x7f4ed40ad370 2023-10-19T12:58:11.661 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.651+0000 7f4ed37fe640 1 --2- 172.21.15.57:0/2574816464 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4ed40ad370 0x7f4ed40af4a0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f4ecc00eab0 tx=0x7f4ecc00ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:58:11.661 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.652+0000 7f4ecb7fe640 1 -- 172.21.15.57:0/2574816464 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ecc00cea0 con 0x7f4ed40ad370 2023-10-19T12:58:11.661 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.652+0000 7f4ecb7fe640 1 -- 172.21.15.57:0/2574816464 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f4ecc004b20 con 0x7f4ed40ad370 2023-10-19T12:58:11.661 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.652+0000 7f4ecb7fe640 1 -- 172.21.15.57:0/2574816464 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ecc005020 con 0x7f4ed40ad370 2023-10-19T12:58:11.661 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.652+0000 7f4ed9410640 1 -- 172.21.15.57:0/2574816464 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ed40c86f0 con 0x7f4ed40ad370 2023-10-19T12:58:11.661 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.652+0000 7f4ed9410640 1 -- 172.21.15.57:0/2574816464 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ed4150de0 con 0x7f4ed40ad370 2023-10-19T12:58:11.662 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.653+0000 7f4ed9410640 1 -- 172.21.15.57:0/2574816464 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ed40ce1d0 con 0x7f4ed40ad370 2023-10-19T12:58:11.662 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.653+0000 7f4ecb7fe640 1 -- 172.21.15.57:0/2574816464 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 21) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f4ecc005180 con 0x7f4ed40ad370 2023-10-19T12:58:11.662 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.654+0000 7f4ecb7fe640 1 --2- 172.21.15.57:0/2574816464 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f4eb406afb0 0x7f4eb406d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:11.662 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.654+0000 7f4ed2ffd640 1 -- 172.21.15.57:0/2574816464 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f4eb406afb0 msgr2=0x7f4eb406d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/3449041082 2023-10-19T12:58:11.662 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.654+0000 7f4ecb7fe640 1 -- 172.21.15.57:0/2574816464 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f4ecc014070 con 0x7f4ed40ad370 2023-10-19T12:58:11.663 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.654+0000 7f4ed2ffd640 1 --2- 172.21.15.57:0/2574816464 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f4eb406afb0 0x7f4eb406d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T12:58:11.663 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.658+0000 7f4ecb7fe640 1 -- 172.21.15.57:0/2574816464 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4ecc0553a0 con 0x7f4ed40ad370 2023-10-19T12:58:11.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.835+0000 7f4ed9410640 1 -- 172.21.15.57:0/2574816464 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f4ed40690e0 con 0x7f4ed40ad370 2023-10-19T12:58:11.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.843+0000 7f4ecb7fe640 1 -- 172.21.15.57:0/2574816464 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v37)=0 v37) v1 ==== 133+0+0 (secure 0 0 0) 0x7f4ecc0589c0 con 0x7f4ed40ad370 2023-10-19T12:58:11.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.845+0000 7f4ed9410640 1 -- 172.21.15.57:0/2574816464 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f4eb406afb0 msgr2=0x7f4eb406d470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T12:58:11.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.845+0000 7f4ed9410640 1 --2- 172.21.15.57:0/2574816464 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f4eb406afb0 0x7f4eb406d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:11.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.845+0000 7f4ed9410640 1 -- 172.21.15.57:0/2574816464 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4ed40ad370 msgr2=0x7f4ed40af4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:11.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.845+0000 7f4ed9410640 1 --2- 172.21.15.57:0/2574816464 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4ed40ad370 0x7f4ed40af4a0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f4ecc00eab0 tx=0x7f4ecc00ef70 comp rx=0 tx=0).stop 2023-10-19T12:58:11.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.845+0000 7f4ed9410640 1 -- 172.21.15.57:0/2574816464 shutdown_connections 2023-10-19T12:58:11.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.845+0000 7f4ed9410640 1 --2- 172.21.15.57:0/2574816464 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f4eb406afb0 0x7f4eb406d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:11.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.845+0000 7f4ed9410640 1 --2- 172.21.15.57:0/2574816464 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4ed40ad370 0x7f4ed40af4a0 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:11.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.845+0000 7f4ed9410640 1 --2- 172.21.15.57:0/2574816464 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4ed406f0a0 0x7f4ed40c7e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:11.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.845+0000 7f4ed9410640 1 --2- 172.21.15.57:0/2574816464 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4ed406d2d0 0x7f4ed414e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:11.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.845+0000 7f4ed9410640 1 -- 172.21.15.57:0/2574816464 >> 172.21.15.57:0/2574816464 conn(0x7f4ed40b0670 msgr2=0x7f4ed40ab950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:58:11.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.845+0000 7f4ed9410640 1 -- 172.21.15.57:0/2574816464 shutdown_connections 2023-10-19T12:58:11.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:11.845+0000 7f4ed9410640 1 -- 172.21.15.57:0/2574816464 wait complete. 2023-10-19T12:58:11.859 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:58:11.860 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-10-19T12:58:11.887 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 1m 2023-10-19T12:58:12.003 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:58:12.001+0000 7f36cf612040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-19T12:58:12.046 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:58:12.044+0000 7f36cf612040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-19T12:58:12.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:12.044+0000 7f866eda9640 1 -- 172.21.15.57:0/1045046202 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f86680b0550 msgr2=0x7f86680cd6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:12.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:12.044+0000 7f866eda9640 1 --2- 172.21.15.57:0/1045046202 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f86680b0550 0x7f86680cd6c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f8654009860 tx=0x7f865402bde0 comp rx=0 tx=0).stop 2023-10-19T12:58:12.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:12.044+0000 7f866eda9640 1 -- 172.21.15.57:0/1045046202 shutdown_connections 2023-10-19T12:58:12.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:12.044+0000 7f866eda9640 1 --2- 172.21.15.57:0/1045046202 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f86680b0550 0x7f86680cd6c0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:12.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:12.044+0000 7f866eda9640 1 -- 172.21.15.57:0/1045046202 >> 172.21.15.57:0/1045046202 conn(0x7f866809a060 msgr2=0x7f86680d0d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:58:12.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:12.044+0000 7f866eda9640 1 -- 172.21.15.57:0/1045046202 shutdown_connections 2023-10-19T12:58:12.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:12.044+0000 7f866eda9640 1 -- 172.21.15.57:0/1045046202 wait complete. 2023-10-19T12:58:12.049 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:12.046+0000 7f866eda9640 1 Processor -- start 2023-10-19T12:58:12.049 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:12.046+0000 7f866eda9640 1 -- start start 2023-10-19T12:58:12.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:12.047+0000 7f866eda9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f86680b0550 0x7f866815caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:12.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:12.047+0000 7f866eda9640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f86680b09b0 0x7f86681f1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:12.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:12.047+0000 7f866eda9640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f866815e930 0x7f86680b27f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:12.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:12.047+0000 7f866eda9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8668151070 con 0x7f86680b0550 2023-10-19T12:58:12.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:12.047+0000 7f866eda9640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f866814d8d0 con 0x7f86680b09b0 2023-10-19T12:58:12.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:12.047+0000 7f866eda9640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f866814dbb0 con 0x7f866815e930 2023-10-19T12:58:12.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:12.048+0000 7f866cb45640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f86680b0550 0x7f866815caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:12.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:12.048+0000 7f866cb45640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f86680b0550 0x7f866815caf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59610/0 (socket says 172.21.15.57:59610) 2023-10-19T12:58:12.054 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:12.048+0000 7f866cb45640 1 -- 172.21.15.57:0/1272236459 learned_addr learned my addr 172.21.15.57:0/1272236459 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:58:12.054 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:12.048+0000 7f866d346640 1 --2- 172.21.15.57:0/1272236459 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f866815e930 0x7f86680b27f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:12.054 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:12.048+0000 7f8667fff640 1 --2- 172.21.15.57:0/1272236459 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f86680b09b0 0x7f86681f1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:12.055 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:12.048+0000 7f866cb45640 1 -- 172.21.15.57:0/1272236459 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f866815e930 msgr2=0x7f86680b27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:12.055 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:12.048+0000 7f866cb45640 1 --2- 172.21.15.57:0/1272236459 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f866815e930 0x7f86680b27f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:12.055 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:12.048+0000 7f866cb45640 1 -- 172.21.15.57:0/1272236459 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f86680b09b0 msgr2=0x7f86681f1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:12.055 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:12.048+0000 7f866cb45640 1 --2- 172.21.15.57:0/1272236459 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f86680b09b0 0x7f86681f1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:12.055 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:12.048+0000 7f866cb45640 1 -- 172.21.15.57:0/1272236459 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8654009490 con 0x7f86680b0550 2023-10-19T12:58:12.056 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:12.048+0000 7f866d346640 1 --2- 172.21.15.57:0/1272236459 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f866815e930 0x7f86680b27f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T12:58:12.056 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:12.048+0000 7f866cb45640 1 --2- 172.21.15.57:0/1272236459 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f86680b0550 0x7f866815caf0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f865c00c430 tx=0x7f865c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:58:12.056 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:12.048+0000 7f86657fa640 1 -- 172.21.15.57:0/1272236459 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f865c011790 con 0x7f86680b0550 2023-10-19T12:58:12.056 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:12.048+0000 7f86657fa640 1 -- 172.21.15.57:0/1272236459 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f865c011a00 con 0x7f86680b0550 2023-10-19T12:58:12.057 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:12.048+0000 7f866eda9640 1 -- 172.21.15.57:0/1272236459 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f866815c010 con 0x7f86680b0550 2023-10-19T12:58:12.057 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:12.048+0000 7f86657fa640 1 -- 172.21.15.57:0/1272236459 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f865c011e10 con 0x7f86680b0550 2023-10-19T12:58:12.057 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:12.048+0000 7f866eda9640 1 -- 172.21.15.57:0/1272236459 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f866815c5d0 con 0x7f86680b0550 2023-10-19T12:58:12.058 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:12.050+0000 7f866eda9640 1 -- 172.21.15.57:0/1272236459 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8668066960 con 0x7f86680b0550 2023-10-19T12:58:12.058 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:12.050+0000 7f86657fa640 1 -- 172.21.15.57:0/1272236459 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 21) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f865c01c460 con 0x7f86680b0550 2023-10-19T12:58:12.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:12.050+0000 7f86657fa640 1 --2- 172.21.15.57:0/1272236459 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f864c06afb0 0x7f864c06d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:12.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:12.050+0000 7f8667fff640 1 -- 172.21.15.57:0/1272236459 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f864c06afb0 msgr2=0x7f864c06d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/3449041082 2023-10-19T12:58:12.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:12.051+0000 7f86657fa640 1 -- 172.21.15.57:0/1272236459 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f865c089a80 con 0x7f86680b0550 2023-10-19T12:58:12.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:12.051+0000 7f8667fff640 1 --2- 172.21.15.57:0/1272236459 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f864c06afb0 0x7f864c06d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T12:58:12.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:12.054+0000 7f86657fa640 1 -- 172.21.15.57:0/1272236459 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f865c0548d0 con 0x7f86680b0550 2023-10-19T12:58:12.061 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:58:12.059+0000 7fc894012040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-19T12:58:12.089 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:58:12.087+0000 7f36cf612040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-19T12:58:12.108 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:58:12.105+0000 7fc894012040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-19T12:58:12.140 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:58:12.138+0000 7f36cf612040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-19T12:58:12.153 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:58:12.150+0000 7fc894012040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-19T12:58:12.181 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:58:12.179+0000 7f36cf612040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-19T12:58:12.208 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:58:12.206+0000 7fc894012040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-19T12:58:12.252 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:58:12.249+0000 7fc894012040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-19T12:58:12.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:12.251+0000 7f866eda9640 1 -- 172.21.15.57:0/1272236459 --> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f866809ed40 con 0x7f864c06afb0 2023-10-19T12:58:12.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:12.252+0000 7f8667fff640 1 -- 172.21.15.57:0/1272236459 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f864c06afb0 msgr2=0x7f864c06d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/3449041082 2023-10-19T12:58:12.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:12.252+0000 7f8667fff640 1 --2- 172.21.15.57:0/1272236459 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f864c06afb0 0x7f864c06d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-19T12:58:12.280 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:58:12.279+0000 7f36cf612040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-19T12:58:12.333 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:58:12.331+0000 7f36cf612040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-19T12:58:12.355 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:58:12.352+0000 7fc894012040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-19T12:58:12.375 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:58:12.373+0000 7f36cf612040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-19T12:58:12.410 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:58:12.407+0000 7fc894012040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-19T12:58:12.422 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:58:12.421+0000 7f36cf612040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-19T12:58:12.453 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:58:12.450+0000 7fc894012040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-19T12:58:12.465 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:58:12.463+0000 7f36cf612040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-19T12:58:12.503 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:58:12.500+0000 7fc894012040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-19T12:58:12.547 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:58:12.545+0000 7fc894012040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-19T12:58:12.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:12.653+0000 7f8667fff640 1 -- 172.21.15.57:0/1272236459 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f864c06afb0 msgr2=0x7f864c06d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/3449041082 2023-10-19T12:58:12.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:12.653+0000 7f8667fff640 1 --2- 172.21.15.57:0/1272236459 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f864c06afb0 0x7f864c06d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-19T12:58:12.777 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:58:12.775+0000 7f36cf612040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-19T12:58:12.877 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:58:12.874+0000 7fc894012040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-19T12:58:12.905 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:58:12.903+0000 7f36cf612040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-19T12:58:12.997 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:58:12.995+0000 7f36cf612040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-19T12:58:13.016 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:58:13.013+0000 7fc894012040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-19T12:58:13.040 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:58:13.038+0000 7f36cf612040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-19T12:58:13.111 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:58:13.109+0000 7fc894012040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-19T12:58:13.134 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:58:13.132+0000 7f36cf612040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-19T12:58:13.158 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:58:13.155+0000 7fc894012040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-19T12:58:13.225 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:58:13.223+0000 7f36cf612040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-19T12:58:13.256 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:58:13.254+0000 7fc894012040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-19T12:58:13.351 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:58:13.349+0000 7fc894012040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-19T12:58:13.361 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:58:13.359+0000 7f36cf612040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-19T12:58:13.404 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:58:13.402+0000 7f36cf612040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-19T12:58:13.456 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:13.454+0000 7f8667fff640 1 -- 172.21.15.57:0/1272236459 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f864c06afb0 msgr2=0x7f864c06d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/3449041082 2023-10-19T12:58:13.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:13.454+0000 7f8667fff640 1 --2- 172.21.15.57:0/1272236459 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f864c06afb0 0x7f864c06d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-19T12:58:13.493 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:58:13.491+0000 7fc894012040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-19T12:58:13.508 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:58:13.506+0000 7f36cf612040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-19T12:58:13.537 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:58:13.534+0000 7fc894012040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-19T12:58:13.646 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:58:13.643+0000 7fc894012040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-19T12:58:14.115 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:58:14.114+0000 7f36cf612040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-19T12:58:14.286 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:58:14.283+0000 7fc894012040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-19T12:58:14.379 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:58:14.378+0000 7f36cf612040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-19T12:58:14.428 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:58:14.426+0000 7f36cf612040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-19T12:58:14.527 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:58:14.525+0000 7f36cf612040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-19T12:58:14.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:14.546+0000 7f86657fa640 1 -- 172.21.15.57:0/1272236459 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mgrmap(e 22) v1 ==== 88957+0+0 (secure 0 0 0) 0x7f865c054ca0 con 0x7f86680b0550 2023-10-19T12:58:14.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:14.546+0000 7f86657fa640 1 -- 172.21.15.57:0/1272236459 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f864c06afb0 msgr2=0x7f864c06d470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T12:58:14.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:14.546+0000 7f86657fa640 1 --2- 172.21.15.57:0/1272236459 >> [v2:172.21.15.179:6832/3449041082,v1:172.21.15.179:6833/3449041082] conn(0x7f864c06afb0 0x7f864c06d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:14.562 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:58:14.560+0000 7fc894012040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-19T12:58:14.617 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:58:14.615+0000 7fc894012040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-19T12:58:14.651 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:58:14.648+0000 7f364bc83640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T12:58:14.651 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:58:14.648+0000 7f364bc83640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T12:58:14.651 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:58:14.648+0000 7f364bc83640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T12:58:14.652 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:58:14.648+0000 7f364bc83640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T12:58:14.652 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:58:14.648+0000 7f364bc83640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T12:58:14.652 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:58:14.648+0000 7f364ecc9640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T12:58:14.652 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:58:14.648+0000 7f364ecc9640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T12:58:14.652 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:58:14.648+0000 7f364ecc9640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T12:58:14.653 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:58:14.648+0000 7f364ecc9640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T12:58:14.653 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:58:14.648+0000 7f364ecc9640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T12:58:14.726 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T12:58:14.723+0000 7fc894012040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-19T12:58:15.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:15.553+0000 7f86657fa640 1 -- 172.21.15.57:0/1272236459 <== mon.0 v2:172.21.15.57:3300/0 8 ==== mgrmap(e 23) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f865c060400 con 0x7f86680b0550 2023-10-19T12:58:15.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:15.553+0000 7f86657fa640 1 --2- 172.21.15.57:0/1272236459 >> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] conn(0x7f864c0358b0 0x7f864c075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:15.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:15.553+0000 7f86657fa640 1 -- 172.21.15.57:0/1272236459 --> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f866809ed40 con 0x7f864c0358b0 2023-10-19T12:58:15.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:15.556+0000 7f8667fff640 1 --2- 172.21.15.57:0/1272236459 >> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] conn(0x7f864c0358b0 0x7f864c075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:15.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:15.558+0000 7f8667fff640 1 --2- 172.21.15.57:0/1272236459 >> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] conn(0x7f864c0358b0 0x7f864c075380 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f8654036000 tx=0x7f8654004730 comp rx=0 tx=0).ready entity=mgr.5690 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:58:15.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:15.646+0000 7f86657fa640 1 -- 172.21.15.57:0/1272236459 <== mgr.5690 v2:172.21.15.179:6832/2207781045 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f866809ed40 con 0x7f864c0358b0 2023-10-19T12:58:15.650 INFO:teuthology.orchestra.run.smithi057.stdout:Schedule set for path snap_test_dir1 2023-10-19T12:58:15.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:15.648+0000 7f866eda9640 1 -- 172.21.15.57:0/1272236459 >> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] conn(0x7f864c0358b0 msgr2=0x7f864c075380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:15.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:15.648+0000 7f866eda9640 1 --2- 172.21.15.57:0/1272236459 >> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] conn(0x7f864c0358b0 0x7f864c075380 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f8654036000 tx=0x7f8654004730 comp rx=0 tx=0).stop 2023-10-19T12:58:15.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:15.648+0000 7f866eda9640 1 -- 172.21.15.57:0/1272236459 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f86680b0550 msgr2=0x7f866815caf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:15.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:15.648+0000 7f866eda9640 1 --2- 172.21.15.57:0/1272236459 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f86680b0550 0x7f866815caf0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f865c00c430 tx=0x7f865c00c8f0 comp rx=0 tx=0).stop 2023-10-19T12:58:15.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:15.648+0000 7f866d346640 1 -- 172.21.15.57:0/1272236459 reap_dead start 2023-10-19T12:58:15.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:15.648+0000 7f866eda9640 1 -- 172.21.15.57:0/1272236459 shutdown_connections 2023-10-19T12:58:15.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:15.648+0000 7f866eda9640 1 -- 172.21.15.57:0/1272236459 >> 172.21.15.57:0/1272236459 conn(0x7f866809a060 msgr2=0x7f866806aef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:58:15.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:15.648+0000 7f866eda9640 1 -- 172.21.15.57:0/1272236459 shutdown_connections 2023-10-19T12:58:15.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:15.648+0000 7f866eda9640 1 -- 172.21.15.57:0/1272236459 wait complete. 2023-10-19T12:58:15.663 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2023-10-19T12:58:15.664 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 2m 2023-10-19T12:58:15.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:15.776+0000 7fc9b00da640 1 -- 172.21.15.57:0/1916166609 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc9a806fab0 msgr2=0x7fc9a80c97a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:15.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:15.776+0000 7fc9b00da640 1 --2- 172.21.15.57:0/1916166609 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc9a806fab0 0x7fc9a80c97a0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fc9a0005fd0 tx=0x7fc9a002c0e0 comp rx=0 tx=0).stop 2023-10-19T12:58:15.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:15.776+0000 7fc9b00da640 1 -- 172.21.15.57:0/1916166609 shutdown_connections 2023-10-19T12:58:15.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:15.776+0000 7fc9b00da640 1 --2- 172.21.15.57:0/1916166609 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc9a806fab0 0x7fc9a80c97a0 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:15.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:15.776+0000 7fc9b00da640 1 --2- 172.21.15.57:0/1916166609 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc9a8140620 0x7fc9a80703e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:15.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:15.776+0000 7fc9b00da640 1 -- 172.21.15.57:0/1916166609 >> 172.21.15.57:0/1916166609 conn(0x7fc9a8069930 msgr2=0x7fc9a8069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:58:15.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:15.776+0000 7fc9b00da640 1 -- 172.21.15.57:0/1916166609 shutdown_connections 2023-10-19T12:58:15.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:15.777+0000 7fc9b00da640 1 -- 172.21.15.57:0/1916166609 wait complete. 2023-10-19T12:58:15.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:15.778+0000 7fc9b00da640 1 Processor -- start 2023-10-19T12:58:15.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:15.778+0000 7fc9b00da640 1 -- start start 2023-10-19T12:58:15.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:15.778+0000 7fc9b00da640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc9a806fab0 0x7fc9a8130490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:15.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:15.778+0000 7fc9b00da640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc9a8140620 0x7fc9a813f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:15.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:15.778+0000 7fc9b00da640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc9a8140a00 0x7fc9a80a9980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:15.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:15.778+0000 7fc9b00da640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9a80a9ec0 con 0x7fc9a8140a00 2023-10-19T12:58:15.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:15.778+0000 7fc9b00da640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9a81316b0 con 0x7fc9a806fab0 2023-10-19T12:58:15.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:15.778+0000 7fc9b00da640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9a8130aa0 con 0x7fc9a8140620 2023-10-19T12:58:15.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:15.778+0000 7fc9ae677640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc9a8140a00 0x7fc9a80a9980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:15.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:15.779+0000 7fc9ae677640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc9a8140a00 0x7fc9a80a9980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59624/0 (socket says 172.21.15.57:59624) 2023-10-19T12:58:15.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:15.779+0000 7fc9ae677640 1 -- 172.21.15.57:0/3663592693 learned_addr learned my addr 172.21.15.57:0/3663592693 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:58:15.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:15.779+0000 7fc9ae677640 1 -- 172.21.15.57:0/3663592693 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc9a8140620 msgr2=0x7fc9a813f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:15.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:15.779+0000 7fc9ad675640 1 --2- 172.21.15.57:0/3663592693 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc9a8140620 0x7fc9a813f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:15.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:15.779+0000 7fc9ade76640 1 --2- 172.21.15.57:0/3663592693 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc9a806fab0 0x7fc9a8130490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:15.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:15.779+0000 7fc9ae677640 1 --2- 172.21.15.57:0/3663592693 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc9a8140620 0x7fc9a813f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:15.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:15.779+0000 7fc9ae677640 1 -- 172.21.15.57:0/3663592693 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc9a806fab0 msgr2=0x7fc9a8130490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:15.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:15.779+0000 7fc9ae677640 1 --2- 172.21.15.57:0/3663592693 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc9a806fab0 0x7fc9a8130490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:15.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:15.779+0000 7fc9ae677640 1 -- 172.21.15.57:0/3663592693 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9a0009840 con 0x7fc9a8140a00 2023-10-19T12:58:15.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:15.779+0000 7fc9ae677640 1 --2- 172.21.15.57:0/3663592693 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc9a8140a00 0x7fc9a80a9980 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fc9a0009cd0 tx=0x7fc9a0004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:58:15.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:15.779+0000 7fc99e7fc640 1 -- 172.21.15.57:0/3663592693 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9a0004760 con 0x7fc9a8140a00 2023-10-19T12:58:15.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:15.779+0000 7fc99e7fc640 1 -- 172.21.15.57:0/3663592693 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fc9a00345a0 con 0x7fc9a8140a00 2023-10-19T12:58:15.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:15.779+0000 7fc9b00da640 1 -- 172.21.15.57:0/3663592693 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9a8130d80 con 0x7fc9a8140a00 2023-10-19T12:58:15.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:15.779+0000 7fc99e7fc640 1 -- 172.21.15.57:0/3663592693 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9a0034990 con 0x7fc9a8140a00 2023-10-19T12:58:15.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:15.779+0000 7fc9b00da640 1 -- 172.21.15.57:0/3663592693 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9a8131340 con 0x7fc9a8140a00 2023-10-19T12:58:15.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:15.780+0000 7fc9b00da640 1 -- 172.21.15.57:0/3663592693 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9a806aea0 con 0x7fc9a8140a00 2023-10-19T12:58:15.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:15.781+0000 7fc99e7fc640 1 -- 172.21.15.57:0/3663592693 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 23) v1 ==== 89084+0+0 (secure 0 0 0) 0x7fc9a0034af0 con 0x7fc9a8140a00 2023-10-19T12:58:15.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:15.781+0000 7fc99e7fc640 1 --2- 172.21.15.57:0/3663592693 >> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] conn(0x7fc98c06ace0 0x7fc98c06d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:15.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:15.781+0000 7fc99e7fc640 1 -- 172.21.15.57:0/3663592693 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6373+0+0 (secure 0 0 0) 0x7fc9a00ac830 con 0x7fc9a8140a00 2023-10-19T12:58:15.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:15.781+0000 7fc9ade76640 1 --2- 172.21.15.57:0/3663592693 >> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] conn(0x7fc98c06ace0 0x7fc98c06d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:15.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:15.781+0000 7fc9ade76640 1 --2- 172.21.15.57:0/3663592693 >> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] conn(0x7fc98c06ace0 0x7fc98c06d1a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc9a400ba40 tx=0x7fc9a4005cc0 comp rx=0 tx=0).ready entity=mgr.5690 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:58:15.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:15.785+0000 7fc99e7fc640 1 -- 172.21.15.57:0/3663592693 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc9a0077980 con 0x7fc9a8140a00 2023-10-19T12:58:15.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:15.984+0000 7fc9b00da640 1 -- 172.21.15.57:0/3663592693 --> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "2m", "target": ["mon-mgr", ""]}) v1 -- 0x7fc9a806ca20 con 0x7fc98c06ace0 2023-10-19T12:58:16.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:16.034+0000 7fc99e7fc640 1 -- 172.21.15.57:0/3663592693 <== mgr.5690 v2:172.21.15.179:6832/2207781045 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fc9a400da80 con 0x7fc98c06ace0 2023-10-19T12:58:16.037 INFO:teuthology.orchestra.run.smithi057.stdout:Schedule set for path snap_test_dir1 2023-10-19T12:58:16.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:16.036+0000 7fc9b00da640 1 -- 172.21.15.57:0/3663592693 >> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] conn(0x7fc98c06ace0 msgr2=0x7fc98c06d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:16.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:16.036+0000 7fc9b00da640 1 --2- 172.21.15.57:0/3663592693 >> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] conn(0x7fc98c06ace0 0x7fc98c06d1a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc9a400ba40 tx=0x7fc9a4005cc0 comp rx=0 tx=0).stop 2023-10-19T12:58:16.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:16.036+0000 7fc9b00da640 1 -- 172.21.15.57:0/3663592693 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc9a8140a00 msgr2=0x7fc9a80a9980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:16.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:16.036+0000 7fc9b00da640 1 --2- 172.21.15.57:0/3663592693 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc9a8140a00 0x7fc9a80a9980 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fc9a0009cd0 tx=0x7fc9a0004360 comp rx=0 tx=0).stop 2023-10-19T12:58:16.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:16.036+0000 7fc9b00da640 1 -- 172.21.15.57:0/3663592693 shutdown_connections 2023-10-19T12:58:16.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:16.036+0000 7fc9b00da640 1 --2- 172.21.15.57:0/3663592693 >> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] conn(0x7fc98c06ace0 0x7fc98c06d1a0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:16.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:16.036+0000 7fc9b00da640 1 --2- 172.21.15.57:0/3663592693 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc9a8140a00 0x7fc9a80a9980 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:16.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:16.036+0000 7fc9b00da640 1 --2- 172.21.15.57:0/3663592693 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc9a8140620 0x7fc9a813f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:16.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:16.036+0000 7fc9b00da640 1 --2- 172.21.15.57:0/3663592693 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc9a806fab0 0x7fc9a8130490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:16.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:16.036+0000 7fc9b00da640 1 -- 172.21.15.57:0/3663592693 >> 172.21.15.57:0/3663592693 conn(0x7fc9a8069930 msgr2=0x7fc9a813bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:58:16.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:16.036+0000 7fc9b00da640 1 -- 172.21.15.57:0/3663592693 shutdown_connections 2023-10-19T12:58:16.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:16.036+0000 7fc9b00da640 1 -- 172.21.15.57:0/3663592693 wait complete. 2023-10-19T12:58:16.049 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2023-10-19T12:58:16.050 DEBUG:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2023-10-19-12_59_00 in ~44s... 2023-10-19T12:58:16.050 DEBUG:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2023-10-19-13_00_00 in ~104s... 2023-10-19T12:58:16.051 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1m', '2m'], expected_retention: [] 2023-10-19T12:58:16.051 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --format json snap_test_dir1 2023-10-19T12:58:16.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:16.159+0000 7f807eb16640 1 -- 172.21.15.57:0/1843601533 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f80781579f0 msgr2=0x7f80780b0e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:16.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:16.159+0000 7f807eb16640 1 --2- 172.21.15.57:0/1843601533 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f80781579f0 0x7f80780b0e90 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f80780b0160 tx=0x7f806c02bd70 comp rx=0 tx=0).stop 2023-10-19T12:58:16.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:16.159+0000 7f807eb16640 1 -- 172.21.15.57:0/1843601533 shutdown_connections 2023-10-19T12:58:16.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:16.159+0000 7f807eb16640 1 --2- 172.21.15.57:0/1843601533 >> v2:172.21.15.179:3300/0 conn(0x7f807806ae40 0x7f8078157dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:16.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:16.159+0000 7f807eb16640 1 --2- 172.21.15.57:0/1843601533 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f80781579f0 0x7f80780b0e90 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:16.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:16.160+0000 7f807eb16640 1 -- 172.21.15.57:0/1843601533 >> 172.21.15.57:0/1843601533 conn(0x7f807806fae0 msgr2=0x7f807806fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:58:16.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:16.160+0000 7f807eb16640 1 -- 172.21.15.57:0/1843601533 shutdown_connections 2023-10-19T12:58:16.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:16.160+0000 7f807eb16640 1 -- 172.21.15.57:0/1843601533 wait complete. 2023-10-19T12:58:16.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:16.161+0000 7f807eb16640 1 Processor -- start 2023-10-19T12:58:16.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:16.161+0000 7f807eb16640 1 -- start start 2023-10-19T12:58:16.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:16.162+0000 7f807eb16640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f807806ae40 0x7f807814eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:16.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:16.162+0000 7f807eb16640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f80781579f0 0x7f80780c9cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:16.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:16.162+0000 7f807eb16640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f807815c1f0 0x7f807806b980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:16.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:16.162+0000 7f807eb16640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f807814f530 con 0x7f807806ae40 2023-10-19T12:58:16.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:16.162+0000 7f807eb16640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80781e24e0 con 0x7f80781579f0 2023-10-19T12:58:16.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:16.162+0000 7f807eb16640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80781ef600 con 0x7f807815c1f0 2023-10-19T12:58:16.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:16.162+0000 7f807c8b2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f807806ae40 0x7f807814eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:16.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:16.162+0000 7f807c8b2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f807806ae40 0x7f807814eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59630/0 (socket says 172.21.15.57:59630) 2023-10-19T12:58:16.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:16.162+0000 7f807c8b2640 1 -- 172.21.15.57:0/64561177 learned_addr learned my addr 172.21.15.57:0/64561177 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T12:58:16.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:16.162+0000 7f8077fff640 1 --2- 172.21.15.57:0/64561177 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f80781579f0 0x7f80780c9cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:16.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:16.162+0000 7f807d0b3640 1 --2- 172.21.15.57:0/64561177 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f807815c1f0 0x7f807806b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:16.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:16.162+0000 7f807c8b2640 1 -- 172.21.15.57:0/64561177 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f807815c1f0 msgr2=0x7f807806b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:16.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:16.162+0000 7f807c8b2640 1 --2- 172.21.15.57:0/64561177 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f807815c1f0 0x7f807806b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:16.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:16.162+0000 7f807c8b2640 1 -- 172.21.15.57:0/64561177 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f80781579f0 msgr2=0x7f80780c9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:16.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:16.162+0000 7f807c8b2640 1 --2- 172.21.15.57:0/64561177 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f80781579f0 0x7f80780c9cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:16.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:16.162+0000 7f807c8b2640 1 -- 172.21.15.57:0/64561177 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f806c009490 con 0x7f807806ae40 2023-10-19T12:58:16.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:16.162+0000 7f807c8b2640 1 --2- 172.21.15.57:0/64561177 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f807806ae40 0x7f807814eff0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f806c000c00 tx=0x7f806c0038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:58:16.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:16.162+0000 7f80757fa640 1 -- 172.21.15.57:0/64561177 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f806c003ea0 con 0x7f807806ae40 2023-10-19T12:58:16.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:16.162+0000 7f80757fa640 1 -- 172.21.15.57:0/64561177 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f806c034550 con 0x7f807806ae40 2023-10-19T12:58:16.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:16.162+0000 7f807eb16640 1 -- 172.21.15.57:0/64561177 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80781ef8e0 con 0x7f807806ae40 2023-10-19T12:58:16.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:16.162+0000 7f80757fa640 1 -- 172.21.15.57:0/64561177 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f806c034960 con 0x7f807806ae40 2023-10-19T12:58:16.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:16.162+0000 7f807eb16640 1 -- 172.21.15.57:0/64561177 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80781efea0 con 0x7f807806ae40 2023-10-19T12:58:16.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:16.163+0000 7f807eb16640 1 -- 172.21.15.57:0/64561177 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80780690e0 con 0x7f807806ae40 2023-10-19T12:58:16.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:16.164+0000 7f80757fa640 1 -- 172.21.15.57:0/64561177 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 23) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f806c03b040 con 0x7f807806ae40 2023-10-19T12:58:16.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:16.164+0000 7f80757fa640 1 --2- 172.21.15.57:0/64561177 >> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] conn(0x7f805c06ace0 0x7f805c06d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T12:58:16.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:16.164+0000 7f80757fa640 1 -- 172.21.15.57:0/64561177 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f806c0aba40 con 0x7f807806ae40 2023-10-19T12:58:16.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:16.164+0000 7f8077fff640 1 --2- 172.21.15.57:0/64561177 >> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] conn(0x7f805c06ace0 0x7f805c06d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T12:58:16.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:16.165+0000 7f8077fff640 1 --2- 172.21.15.57:0/64561177 >> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] conn(0x7f805c06ace0 0x7f805c06d1a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f80780aee80 tx=0x7f8064007eb0 comp rx=0 tx=0).ready entity=mgr.5690 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T12:58:16.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:16.168+0000 7f80757fa640 1 -- 172.21.15.57:0/64561177 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f806c076b90 con 0x7f807806ae40 2023-10-19T12:58:16.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:16.365+0000 7f807eb16640 1 -- 172.21.15.57:0/64561177 --> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f80780ca210 con 0x7f805c06ace0 2023-10-19T12:58:16.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:16.373+0000 7f80757fa640 1 -- 172.21.15.57:0/64561177 <== mgr.5690 v2:172.21.15.179:6832/2207781045 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+76 (secure 0 0 0) 0x7f80780ca210 con 0x7f805c06ace0 2023-10-19T12:58:16.376 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T12:58:16.376 INFO:teuthology.orchestra.run.smithi057.stdout:{"path": "snap_test_dir1", "schedule": ["1m", "2m"], "retention": [{}, {}]} 2023-10-19T12:58:16.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:16.375+0000 7f807eb16640 1 -- 172.21.15.57:0/64561177 >> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] conn(0x7f805c06ace0 msgr2=0x7f805c06d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:16.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:16.375+0000 7f807eb16640 1 --2- 172.21.15.57:0/64561177 >> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] conn(0x7f805c06ace0 0x7f805c06d1a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f80780aee80 tx=0x7f8064007eb0 comp rx=0 tx=0).stop 2023-10-19T12:58:16.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:16.375+0000 7f807eb16640 1 -- 172.21.15.57:0/64561177 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f807806ae40 msgr2=0x7f807814eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T12:58:16.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:16.375+0000 7f807eb16640 1 --2- 172.21.15.57:0/64561177 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f807806ae40 0x7f807814eff0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f806c000c00 tx=0x7f806c0038a0 comp rx=0 tx=0).stop 2023-10-19T12:58:16.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:16.375+0000 7f807eb16640 1 -- 172.21.15.57:0/64561177 shutdown_connections 2023-10-19T12:58:16.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:16.375+0000 7f807eb16640 1 --2- 172.21.15.57:0/64561177 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f807815c1f0 0x7f807806b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:16.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:16.375+0000 7f807eb16640 1 --2- 172.21.15.57:0/64561177 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f80781579f0 0x7f80780c9cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:16.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:16.375+0000 7f807eb16640 1 --2- 172.21.15.57:0/64561177 >> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] conn(0x7f805c06ace0 0x7f805c06d1a0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:16.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:16.375+0000 7f807eb16640 1 --2- 172.21.15.57:0/64561177 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f807806ae40 0x7f807814eff0 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T12:58:16.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:16.375+0000 7f807eb16640 1 -- 172.21.15.57:0/64561177 >> 172.21.15.57:0/64561177 conn(0x7f807806fae0 msgr2=0x7f80780cb980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T12:58:16.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:16.375+0000 7f807eb16640 1 -- 172.21.15.57:0/64561177 shutdown_connections 2023-10-19T12:58:16.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T12:58:16.375+0000 7f807eb16640 1 -- 172.21.15.57:0/64561177 wait complete. 2023-10-19T12:58:16.389 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "snap_test_dir1", "schedule": ["1m", "2m"], "retention": [{}, {}]} 2023-10-19T12:58:16.389 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': 'snap_test_dir1', 'schedule': ['1m', '2m'], 'retention': [{}, {}]} 2023-10-19T12:58:16.390 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:58:16.390 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:58:16.410 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T12:58:16.410 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:58:16.410 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:58:17.025 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T12:58:17.028 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T12:58:17.412 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:58:17.412 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:58:17.436 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T12:58:17.436 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:58:17.437 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:58:18.438 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:58:18.439 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:58:18.462 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T12:58:18.463 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:58:18.463 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:58:19.464 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:58:19.464 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:58:19.488 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T12:58:19.489 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:58:19.489 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:58:20.490 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:58:20.491 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:58:20.515 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T12:58:20.515 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:58:20.517 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:58:21.518 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:58:21.519 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:58:21.543 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T12:58:21.543 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:58:21.544 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:58:22.545 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:58:22.546 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:58:22.569 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T12:58:22.570 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:58:22.570 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:58:23.572 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:58:23.573 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:58:23.596 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T12:58:23.597 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:58:23.597 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:58:24.601 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:58:24.603 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:58:24.626 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T12:58:24.627 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:58:24.627 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:58:25.629 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:58:25.629 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:58:25.654 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T12:58:25.654 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:58:25.655 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:58:26.657 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:58:26.657 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:58:26.681 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T12:58:26.681 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:58:26.681 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:58:27.682 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:58:27.683 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:58:27.707 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T12:58:27.707 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:58:27.707 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:58:28.708 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:58:28.709 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:58:28.733 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T12:58:28.733 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:58:28.733 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:58:29.735 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:58:29.735 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:58:29.759 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T12:58:29.759 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:58:29.760 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:58:30.760 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:58:30.761 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:58:30.784 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T12:58:30.785 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:58:30.785 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:58:31.787 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:58:31.787 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:58:31.811 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T12:58:31.811 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:58:31.811 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:58:32.813 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:58:32.813 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:58:32.837 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T12:58:32.838 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:58:32.838 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:58:33.839 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:58:33.840 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:58:33.863 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T12:58:33.864 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:58:33.864 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:58:34.865 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:58:34.866 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:58:34.890 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T12:58:34.890 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:58:34.891 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:58:35.892 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:58:35.892 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:58:35.916 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T12:58:35.916 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:58:35.917 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:58:36.918 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:58:36.919 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:58:36.942 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T12:58:36.943 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:58:36.943 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:58:37.943 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:58:37.944 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:58:37.967 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T12:58:37.968 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:58:37.968 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:58:38.969 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:58:38.969 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:58:38.993 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T12:58:38.994 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:58:38.994 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:58:39.996 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:58:39.996 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:58:40.020 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T12:58:40.020 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:58:40.021 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:58:41.022 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:58:41.022 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:58:41.046 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T12:58:41.046 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:58:41.047 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:58:42.048 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:58:42.048 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:58:42.072 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T12:58:42.073 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:58:42.073 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:58:43.074 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:58:43.074 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:58:43.098 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T12:58:43.098 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:58:43.099 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:58:44.100 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:58:44.101 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:58:44.125 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T12:58:44.125 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:58:44.125 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:58:45.127 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:58:45.127 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:58:45.150 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T12:58:45.151 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:58:45.151 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:58:46.152 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:58:46.152 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:58:46.176 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T12:58:46.176 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:58:46.177 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:58:47.092 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T12:58:47.095 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T12:58:47.178 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:58:47.179 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:58:47.200 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T12:58:47.201 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:58:47.201 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:58:48.202 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:58:48.203 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:58:48.226 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T12:58:48.226 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:58:48.227 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:58:49.227 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:58:49.228 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:58:49.252 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T12:58:49.252 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:58:49.253 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:58:50.254 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:58:50.254 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:58:50.278 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T12:58:50.279 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:58:50.279 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:58:51.280 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:58:51.281 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:58:51.305 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T12:58:51.305 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:58:51.305 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:58:52.306 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:58:52.307 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:58:52.331 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T12:58:52.331 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:58:52.332 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:58:53.333 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:58:53.334 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:58:53.358 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T12:58:53.359 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:58:53.359 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:58:54.360 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:58:54.361 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:58:54.385 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T12:58:54.385 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:58:54.385 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:58:55.387 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:58:55.388 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:58:55.413 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T12:58:55.413 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:58:55.414 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:58:56.415 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:58:56.416 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:58:56.440 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T12:58:56.441 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:58:56.441 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:58:57.443 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:58:57.443 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:58:57.466 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T12:58:57.467 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:58:57.467 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:58:58.468 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:58:58.469 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:58:58.493 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T12:58:58.493 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:58:58.493 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:58:59.495 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:58:59.495 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:58:59.519 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T12:58:59.520 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:58:59.520 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:59:00.047 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:59:00.044+0000 7f363dc67640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T12:59:00.048 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:59:00.044+0000 7f363dc67640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T12:59:00.048 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:59:00.044+0000 7f363dc67640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T12:59:00.048 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:59:00.044+0000 7f363dc67640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T12:59:00.048 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T12:59:00.044+0000 7f363dc67640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T12:59:00.521 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:59:00.522 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:59:00.545 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_59_00_UTC 2023-10-19T12:59:00.546 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-12_59_00_UTC'} 2023-10-19T12:59:00.546 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2023-10-19-12_59_00_UTC'} 2023-10-19T12:59:00.547 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:59:00.547 DEBUG:tasks.cephfs.test_snap_schedules:snapshots added=['scheduled-2023-10-19-12_59_00_UTC'] 2023-10-19T12:59:00.547 DEBUG:tasks.cephfs.test_snap_schedules:exec=1697720296.0503266, now = 1697720340.5476406, timo = 44 2023-10-19T12:59:01.549 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:59:01.549 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:59:01.573 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_59_00_UTC 2023-10-19T12:59:01.573 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-12_59_00_UTC'} 2023-10-19T12:59:01.573 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:59:01.574 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:59:02.574 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:59:02.575 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:59:02.598 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_59_00_UTC 2023-10-19T12:59:02.599 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-12_59_00_UTC'} 2023-10-19T12:59:02.599 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:59:02.599 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:59:03.601 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:59:03.601 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:59:03.625 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_59_00_UTC 2023-10-19T12:59:03.626 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-12_59_00_UTC'} 2023-10-19T12:59:03.626 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:59:03.626 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:59:04.627 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:59:04.628 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:59:04.651 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_59_00_UTC 2023-10-19T12:59:04.652 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-12_59_00_UTC'} 2023-10-19T12:59:04.652 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:59:04.652 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:59:05.653 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:59:05.654 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:59:05.678 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_59_00_UTC 2023-10-19T12:59:05.678 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-12_59_00_UTC'} 2023-10-19T12:59:05.679 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:59:05.679 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:59:06.680 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:59:06.681 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:59:06.704 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_59_00_UTC 2023-10-19T12:59:06.705 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-12_59_00_UTC'} 2023-10-19T12:59:06.706 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:59:06.706 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:59:07.707 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:59:07.707 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:59:07.730 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_59_00_UTC 2023-10-19T12:59:07.731 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-12_59_00_UTC'} 2023-10-19T12:59:07.732 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:59:07.732 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:59:08.733 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:59:08.734 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:59:08.757 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_59_00_UTC 2023-10-19T12:59:08.758 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-12_59_00_UTC'} 2023-10-19T12:59:08.759 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:59:08.759 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:59:09.761 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:59:09.761 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:59:09.786 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_59_00_UTC 2023-10-19T12:59:09.786 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-12_59_00_UTC'} 2023-10-19T12:59:09.787 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:59:09.788 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:59:10.789 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:59:10.790 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:59:10.812 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_59_00_UTC 2023-10-19T12:59:10.813 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-12_59_00_UTC'} 2023-10-19T12:59:10.814 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:59:10.814 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:59:11.815 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:59:11.816 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:59:11.839 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_59_00_UTC 2023-10-19T12:59:11.840 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-12_59_00_UTC'} 2023-10-19T12:59:11.840 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:59:11.841 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:59:12.841 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:59:12.842 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:59:12.866 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_59_00_UTC 2023-10-19T12:59:12.866 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-12_59_00_UTC'} 2023-10-19T12:59:12.866 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:59:12.867 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:59:13.868 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:59:13.868 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:59:13.892 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_59_00_UTC 2023-10-19T12:59:13.892 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-12_59_00_UTC'} 2023-10-19T12:59:13.892 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:59:13.893 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:59:14.894 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:59:14.894 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:59:14.919 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_59_00_UTC 2023-10-19T12:59:14.919 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-12_59_00_UTC'} 2023-10-19T12:59:14.920 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:59:14.920 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:59:15.921 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:59:15.921 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:59:15.945 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_59_00_UTC 2023-10-19T12:59:15.945 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-12_59_00_UTC'} 2023-10-19T12:59:15.946 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:59:15.946 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:59:16.947 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:59:16.948 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:59:16.970 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_59_00_UTC 2023-10-19T12:59:16.972 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-12_59_00_UTC'} 2023-10-19T12:59:16.972 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:59:16.972 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:59:17.232 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T12:59:17.235 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T12:59:17.974 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:59:17.975 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:59:17.998 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_59_00_UTC 2023-10-19T12:59:17.998 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-12_59_00_UTC'} 2023-10-19T12:59:17.999 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:59:17.999 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:59:19.000 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:59:19.000 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:59:19.024 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_59_00_UTC 2023-10-19T12:59:19.024 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-12_59_00_UTC'} 2023-10-19T12:59:19.024 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:59:19.025 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:59:20.026 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:59:20.027 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:59:20.050 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_59_00_UTC 2023-10-19T12:59:20.050 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-12_59_00_UTC'} 2023-10-19T12:59:20.051 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:59:20.051 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:59:21.053 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:59:21.053 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:59:21.076 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_59_00_UTC 2023-10-19T12:59:21.077 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-12_59_00_UTC'} 2023-10-19T12:59:21.077 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:59:21.078 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:59:22.079 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:59:22.079 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:59:22.102 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_59_00_UTC 2023-10-19T12:59:22.103 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-12_59_00_UTC'} 2023-10-19T12:59:22.104 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:59:22.104 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:59:23.105 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:59:23.106 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:59:23.129 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_59_00_UTC 2023-10-19T12:59:23.130 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-12_59_00_UTC'} 2023-10-19T12:59:23.130 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:59:23.130 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:59:24.131 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:59:24.132 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:59:24.155 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_59_00_UTC 2023-10-19T12:59:24.156 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-12_59_00_UTC'} 2023-10-19T12:59:24.156 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:59:24.157 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:59:25.158 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:59:25.158 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:59:25.182 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_59_00_UTC 2023-10-19T12:59:25.183 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-12_59_00_UTC'} 2023-10-19T12:59:25.183 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:59:25.183 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:59:26.185 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:59:26.185 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:59:26.209 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_59_00_UTC 2023-10-19T12:59:26.209 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-12_59_00_UTC'} 2023-10-19T12:59:26.209 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:59:26.210 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:59:27.210 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:59:27.211 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:59:27.234 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_59_00_UTC 2023-10-19T12:59:27.235 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-12_59_00_UTC'} 2023-10-19T12:59:27.235 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:59:27.235 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:59:28.236 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:59:28.237 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:59:28.261 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_59_00_UTC 2023-10-19T12:59:28.261 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-12_59_00_UTC'} 2023-10-19T12:59:28.262 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:59:28.262 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:59:29.262 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:59:29.263 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:59:29.287 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_59_00_UTC 2023-10-19T12:59:29.288 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-12_59_00_UTC'} 2023-10-19T12:59:29.288 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:59:29.288 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:59:30.290 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:59:30.291 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:59:30.314 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_59_00_UTC 2023-10-19T12:59:30.315 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-12_59_00_UTC'} 2023-10-19T12:59:30.316 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:59:30.316 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:59:31.317 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:59:31.318 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:59:31.341 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_59_00_UTC 2023-10-19T12:59:31.342 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-12_59_00_UTC'} 2023-10-19T12:59:31.343 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:59:31.343 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:59:32.344 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:59:32.345 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:59:32.370 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_59_00_UTC 2023-10-19T12:59:32.370 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-12_59_00_UTC'} 2023-10-19T12:59:32.371 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:59:32.371 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:59:33.372 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:59:33.373 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:59:33.398 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_59_00_UTC 2023-10-19T12:59:33.399 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-12_59_00_UTC'} 2023-10-19T12:59:33.399 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:59:33.399 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:59:34.400 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:59:34.401 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:59:34.424 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_59_00_UTC 2023-10-19T12:59:34.426 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-12_59_00_UTC'} 2023-10-19T12:59:34.426 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:59:34.426 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:59:35.428 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:59:35.428 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:59:35.451 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_59_00_UTC 2023-10-19T12:59:35.452 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-12_59_00_UTC'} 2023-10-19T12:59:35.453 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:59:35.453 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:59:36.454 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:59:36.455 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:59:36.479 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_59_00_UTC 2023-10-19T12:59:36.480 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-12_59_00_UTC'} 2023-10-19T12:59:36.480 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:59:36.480 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:59:37.481 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:59:37.483 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:59:37.506 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_59_00_UTC 2023-10-19T12:59:37.507 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-12_59_00_UTC'} 2023-10-19T12:59:37.507 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:59:37.507 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:59:38.509 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:59:38.510 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:59:38.534 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_59_00_UTC 2023-10-19T12:59:38.535 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-12_59_00_UTC'} 2023-10-19T12:59:38.535 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:59:38.536 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:59:39.538 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:59:39.538 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:59:39.561 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_59_00_UTC 2023-10-19T12:59:39.562 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-12_59_00_UTC'} 2023-10-19T12:59:39.562 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:59:39.563 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:59:40.564 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:59:40.565 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:59:40.588 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_59_00_UTC 2023-10-19T12:59:40.589 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-12_59_00_UTC'} 2023-10-19T12:59:40.589 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:59:40.590 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:59:41.591 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:59:41.591 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:59:41.614 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_59_00_UTC 2023-10-19T12:59:41.615 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-12_59_00_UTC'} 2023-10-19T12:59:41.616 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:59:41.616 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:59:42.617 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:59:42.617 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:59:42.640 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_59_00_UTC 2023-10-19T12:59:42.641 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-12_59_00_UTC'} 2023-10-19T12:59:42.642 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:59:42.642 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:59:43.643 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:59:43.644 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:59:43.667 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_59_00_UTC 2023-10-19T12:59:43.668 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-12_59_00_UTC'} 2023-10-19T12:59:43.668 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:59:43.668 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:59:44.669 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:59:44.670 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:59:44.693 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_59_00_UTC 2023-10-19T12:59:44.694 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-12_59_00_UTC'} 2023-10-19T12:59:44.694 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:59:44.694 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:59:45.695 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:59:45.696 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:59:45.719 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_59_00_UTC 2023-10-19T12:59:45.720 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-12_59_00_UTC'} 2023-10-19T12:59:45.720 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:59:45.720 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:59:46.721 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:59:46.722 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:59:46.745 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_59_00_UTC 2023-10-19T12:59:46.746 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-12_59_00_UTC'} 2023-10-19T12:59:46.746 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:59:46.746 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:59:47.300 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T12:59:47.303 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T12:59:47.748 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:59:47.748 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:59:47.771 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_59_00_UTC 2023-10-19T12:59:47.772 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-12_59_00_UTC'} 2023-10-19T12:59:47.772 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:59:47.772 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:59:48.774 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:59:48.774 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:59:48.797 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_59_00_UTC 2023-10-19T12:59:48.798 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-12_59_00_UTC'} 2023-10-19T12:59:48.798 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:59:48.798 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:59:49.800 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:59:49.800 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:59:49.824 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_59_00_UTC 2023-10-19T12:59:49.824 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-12_59_00_UTC'} 2023-10-19T12:59:49.824 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:59:49.825 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:59:50.826 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:59:50.827 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:59:50.850 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_59_00_UTC 2023-10-19T12:59:50.851 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-12_59_00_UTC'} 2023-10-19T12:59:50.851 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:59:50.851 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:59:51.853 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:59:51.853 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:59:51.876 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_59_00_UTC 2023-10-19T12:59:51.877 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-12_59_00_UTC'} 2023-10-19T12:59:51.878 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:59:51.878 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:59:52.879 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:59:52.880 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:59:52.902 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_59_00_UTC 2023-10-19T12:59:52.903 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-12_59_00_UTC'} 2023-10-19T12:59:52.904 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:59:52.904 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:59:53.905 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:59:53.906 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:59:53.929 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_59_00_UTC 2023-10-19T12:59:53.929 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-12_59_00_UTC'} 2023-10-19T12:59:53.930 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:59:53.930 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:59:54.931 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:59:54.931 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:59:54.955 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_59_00_UTC 2023-10-19T12:59:54.955 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-12_59_00_UTC'} 2023-10-19T12:59:54.956 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:59:54.956 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:59:55.957 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:59:55.958 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:59:55.980 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_59_00_UTC 2023-10-19T12:59:55.981 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-12_59_00_UTC'} 2023-10-19T12:59:55.982 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:59:55.982 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:59:56.984 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:59:56.985 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:59:57.008 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_59_00_UTC 2023-10-19T12:59:57.009 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-12_59_00_UTC'} 2023-10-19T12:59:57.009 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:59:57.009 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:59:58.010 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:59:58.011 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:59:58.034 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_59_00_UTC 2023-10-19T12:59:58.035 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-12_59_00_UTC'} 2023-10-19T12:59:58.035 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:59:58.035 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T12:59:59.036 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T12:59:59.037 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T12:59:59.060 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_59_00_UTC 2023-10-19T12:59:59.061 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-12_59_00_UTC'} 2023-10-19T12:59:59.061 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T12:59:59.061 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T13:00:00.063 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:00:00.063 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T13:00:00.086 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_59_00_UTC 2023-10-19T13:00:00.087 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-12_59_00_UTC'} 2023-10-19T13:00:00.087 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T13:00:00.088 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T13:00:01.089 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:00:01.089 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T13:00:01.112 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_59_00_UTC 2023-10-19T13:00:01.113 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_00_00_UTC 2023-10-19T13:00:01.113 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-12_59_00_UTC', 'scheduled-2023-10-19-13_00_00_UTC'} 2023-10-19T13:00:01.114 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2023-10-19-13_00_00_UTC'} 2023-10-19T13:00:01.114 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T13:00:01.114 DEBUG:tasks.cephfs.test_snap_schedules:snapshots added=['scheduled-2023-10-19-13_00_00_UTC'] 2023-10-19T13:00:01.115 DEBUG:tasks.cephfs.test_snap_schedules:exec=1697720296.0503266, now = 1697720401.115053, timo = 104 2023-10-19T13:00:02.115 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove snap_test_dir1 2023-10-19T13:00:02.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.237+0000 7f6eeb1d7640 1 -- 172.21.15.57:0/3629313999 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6ee413e520 msgr2=0x7f6ee406bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:02.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.237+0000 7f6eeb1d7640 1 --2- 172.21.15.57:0/3629313999 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6ee413e520 0x7f6ee406bb60 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f6ee40cbdc0 tx=0x7f6ed802bd10 comp rx=0 tx=0).stop 2023-10-19T13:00:02.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.237+0000 7f6eeb1d7640 1 -- 172.21.15.57:0/3629313999 shutdown_connections 2023-10-19T13:00:02.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.237+0000 7f6eeb1d7640 1 --2- 172.21.15.57:0/3629313999 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6ee406c0a0 0x7f6ee4070920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:02.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.237+0000 7f6eeb1d7640 1 --2- 172.21.15.57:0/3629313999 >> v2:172.21.15.179:3300/0 conn(0x7f6ee4141690 0x7f6ee40703e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:02.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.237+0000 7f6eeb1d7640 1 --2- 172.21.15.57:0/3629313999 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6ee413e520 0x7f6ee406bb60 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:02.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.237+0000 7f6eeb1d7640 1 -- 172.21.15.57:0/3629313999 >> 172.21.15.57:0/3629313999 conn(0x7f6ee4069930 msgr2=0x7f6ee4069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:02.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.237+0000 7f6eeb1d7640 1 -- 172.21.15.57:0/3629313999 shutdown_connections 2023-10-19T13:00:02.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.237+0000 7f6eeb1d7640 1 -- 172.21.15.57:0/3629313999 wait complete. 2023-10-19T13:00:02.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.238+0000 7f6eeb1d7640 1 Processor -- start 2023-10-19T13:00:02.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.238+0000 7f6eeb1d7640 1 -- start start 2023-10-19T13:00:02.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.239+0000 7f6eeb1d7640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6ee406c0a0 0x7f6ee40a7080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:02.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.239+0000 7f6eeb1d7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6ee413e520 0x7f6ee40a75c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:02.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.239+0000 7f6eeb1d7640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6ee4141690 0x7f6ee41eee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:02.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.239+0000 7f6eeb1d7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee413b660 con 0x7f6ee413e520 2023-10-19T13:00:02.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.239+0000 7f6eeb1d7640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee41f0130 con 0x7f6ee406c0a0 2023-10-19T13:00:02.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.239+0000 7f6eeb1d7640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee41f0410 con 0x7f6ee4141690 2023-10-19T13:00:02.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.239+0000 7f6ee3fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6ee413e520 0x7f6ee40a75c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:02.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.239+0000 7f6ee3fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6ee413e520 0x7f6ee40a75c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44666/0 (socket says 172.21.15.57:44666) 2023-10-19T13:00:02.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.239+0000 7f6ee3fff640 1 -- 172.21.15.57:0/536185774 learned_addr learned my addr 172.21.15.57:0/536185774 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:02.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.239+0000 7f6ee9774640 1 --2- 172.21.15.57:0/536185774 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6ee4141690 0x7f6ee41eee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:02.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.239+0000 7f6ee8f73640 1 --2- 172.21.15.57:0/536185774 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6ee406c0a0 0x7f6ee40a7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:02.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.239+0000 7f6ee3fff640 1 -- 172.21.15.57:0/536185774 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6ee4141690 msgr2=0x7f6ee41eee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:02.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.239+0000 7f6ee3fff640 1 --2- 172.21.15.57:0/536185774 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6ee4141690 0x7f6ee41eee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:02.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.239+0000 7f6ee3fff640 1 -- 172.21.15.57:0/536185774 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6ee406c0a0 msgr2=0x7f6ee40a7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:02.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.239+0000 7f6ee3fff640 1 --2- 172.21.15.57:0/536185774 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6ee406c0a0 0x7f6ee40a7080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:02.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.239+0000 7f6ee3fff640 1 -- 172.21.15.57:0/536185774 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ed8009560 con 0x7f6ee413e520 2023-10-19T13:00:02.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.240+0000 7f6ee3fff640 1 --2- 172.21.15.57:0/536185774 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6ee413e520 0x7f6ee40a75c0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f6ed000d740 tx=0x7f6ed000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:02.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.240+0000 7f6ee17fa640 1 -- 172.21.15.57:0/536185774 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ed0009c30 con 0x7f6ee413e520 2023-10-19T13:00:02.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.240+0000 7f6ee17fa640 1 -- 172.21.15.57:0/536185774 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f6ed0009e80 con 0x7f6ee413e520 2023-10-19T13:00:02.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.240+0000 7f6eeb1d7640 1 -- 172.21.15.57:0/536185774 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ee41f06f0 con 0x7f6ee413e520 2023-10-19T13:00:02.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.240+0000 7f6ee17fa640 1 -- 172.21.15.57:0/536185774 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ed00106c0 con 0x7f6ee413e520 2023-10-19T13:00:02.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.240+0000 7f6eeb1d7640 1 -- 172.21.15.57:0/536185774 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ee40a7b00 con 0x7f6ee413e520 2023-10-19T13:00:02.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.241+0000 7f6eeb1d7640 1 -- 172.21.15.57:0/536185774 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ee40690e0 con 0x7f6ee413e520 2023-10-19T13:00:02.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.241+0000 7f6ee17fa640 1 -- 172.21.15.57:0/536185774 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 27) v1 ==== 89360+0+0 (secure 0 0 0) 0x7f6ed0010820 con 0x7f6ee413e520 2023-10-19T13:00:02.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.242+0000 7f6ee17fa640 1 --2- 172.21.15.57:0/536185774 >> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] conn(0x7f6ec806afb0 0x7f6ec806d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:02.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.242+0000 7f6ee17fa640 1 -- 172.21.15.57:0/536185774 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f6ed008a060 con 0x7f6ee413e520 2023-10-19T13:00:02.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.242+0000 7f6ee8f73640 1 --2- 172.21.15.57:0/536185774 >> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] conn(0x7f6ec806afb0 0x7f6ec806d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:02.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.243+0000 7f6ee8f73640 1 --2- 172.21.15.57:0/536185774 >> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] conn(0x7f6ec806afb0 0x7f6ec806d470 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f6ed8035000 tx=0x7f6ed8004730 comp rx=0 tx=0).ready entity=mgr.5690 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:02.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.246+0000 7f6ee17fa640 1 -- 172.21.15.57:0/536185774 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6ed0054cb0 con 0x7f6ee413e520 2023-10-19T13:00:02.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.444+0000 7f6eeb1d7640 1 -- 172.21.15.57:0/536185774 --> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f6ee406d1d0 con 0x7f6ec806afb0 2023-10-19T13:00:02.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.514+0000 7f6ee17fa640 1 -- 172.21.15.57:0/536185774 <== mgr.5690 v2:172.21.15.179:6832/2207781045 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f6ee406d1d0 con 0x7f6ec806afb0 2023-10-19T13:00:02.518 INFO:teuthology.orchestra.run.smithi057.stdout:Schedule removed for path snap_test_dir1 2023-10-19T13:00:02.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.516+0000 7f6eeb1d7640 1 -- 172.21.15.57:0/536185774 >> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] conn(0x7f6ec806afb0 msgr2=0x7f6ec806d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:02.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.516+0000 7f6eeb1d7640 1 --2- 172.21.15.57:0/536185774 >> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] conn(0x7f6ec806afb0 0x7f6ec806d470 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f6ed8035000 tx=0x7f6ed8004730 comp rx=0 tx=0).stop 2023-10-19T13:00:02.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.516+0000 7f6eeb1d7640 1 -- 172.21.15.57:0/536185774 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6ee413e520 msgr2=0x7f6ee40a75c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:02.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.516+0000 7f6eeb1d7640 1 --2- 172.21.15.57:0/536185774 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6ee413e520 0x7f6ee40a75c0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f6ed000d740 tx=0x7f6ed000dc00 comp rx=0 tx=0).stop 2023-10-19T13:00:02.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.517+0000 7f6eeb1d7640 1 -- 172.21.15.57:0/536185774 shutdown_connections 2023-10-19T13:00:02.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.517+0000 7f6eeb1d7640 1 --2- 172.21.15.57:0/536185774 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6ee4141690 0x7f6ee41eee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:02.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.517+0000 7f6eeb1d7640 1 --2- 172.21.15.57:0/536185774 >> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] conn(0x7f6ec806afb0 0x7f6ec806d470 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:02.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.517+0000 7f6eeb1d7640 1 --2- 172.21.15.57:0/536185774 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6ee413e520 0x7f6ee40a75c0 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:02.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.517+0000 7f6eeb1d7640 1 --2- 172.21.15.57:0/536185774 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6ee406c0a0 0x7f6ee40a7080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:02.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.517+0000 7f6eeb1d7640 1 -- 172.21.15.57:0/536185774 >> 172.21.15.57:0/536185774 conn(0x7f6ee4069930 msgr2=0x7f6ee41e06e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:02.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.517+0000 7f6eeb1d7640 1 -- 172.21.15.57:0/536185774 shutdown_connections 2023-10-19T13:00:02.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.517+0000 7f6eeb1d7640 1 -- 172.21.15.57:0/536185774 wait complete. 2023-10-19T13:00:02.530 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2023-10-19T13:00:02.531 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:00:02.531 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T13:00:02.550 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-12_59_00_UTC 2023-10-19T13:00:02.550 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_00_00_UTC 2023-10-19T13:00:02.550 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-19-12_59_00_UTC 2023-10-19T13:00:02.551 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:00:02.551 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-19-12_59_00_UTC) 2023-10-19T13:00:02.612 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-19-13_00_00_UTC 2023-10-19T13:00:02.613 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:00:02.613 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-19-13_00_00_UTC) 2023-10-19T13:00:02.674 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:00:02.674 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2023-10-19T13:00:02.733 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-19T13:00:02.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.880+0000 7fd8e2ba9640 1 -- 172.21.15.57:0/2888491242 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd8dc0b0530 msgr2=0x7fd8dc0b09c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:02.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.880+0000 7fd8e2ba9640 1 --2- 172.21.15.57:0/2888491242 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd8dc0b0530 0x7fd8dc0b09c0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fd8c4009830 tx=0x7fd8c402bd60 comp rx=0 tx=0).stop 2023-10-19T13:00:02.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.880+0000 7fd8e2ba9640 1 -- 172.21.15.57:0/2888491242 shutdown_connections 2023-10-19T13:00:02.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.880+0000 7fd8e2ba9640 1 --2- 172.21.15.57:0/2888491242 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd8dc0afa20 0x7fd8dc158300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:02.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.880+0000 7fd8e2ba9640 1 --2- 172.21.15.57:0/2888491242 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd8dc0b0530 0x7fd8dc0b09c0 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:02.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.880+0000 7fd8e2ba9640 1 -- 172.21.15.57:0/2888491242 >> 172.21.15.57:0/2888491242 conn(0x7fd8dc069930 msgr2=0x7fd8dc069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:02.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.880+0000 7fd8e2ba9640 1 -- 172.21.15.57:0/2888491242 shutdown_connections 2023-10-19T13:00:02.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.880+0000 7fd8e2ba9640 1 -- 172.21.15.57:0/2888491242 wait complete. 2023-10-19T13:00:02.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.881+0000 7fd8e2ba9640 1 Processor -- start 2023-10-19T13:00:02.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.882+0000 7fd8e2ba9640 1 -- start start 2023-10-19T13:00:02.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.882+0000 7fd8e2ba9640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd8dc0afa20 0x7fd8dc1ef2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:02.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.882+0000 7fd8e2ba9640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd8dc0b0530 0x7fd8dc13eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:02.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.882+0000 7fd8e2ba9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd8dc0cbd20 0x7fd8dc14c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:02.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.882+0000 7fd8e2ba9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8dc1ef7f0 con 0x7fd8dc0cbd20 2023-10-19T13:00:02.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.882+0000 7fd8e2ba9640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8dc141ff0 con 0x7fd8dc0b0530 2023-10-19T13:00:02.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.882+0000 7fd8e2ba9640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8dc144750 con 0x7fd8dc0afa20 2023-10-19T13:00:02.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.882+0000 7fd8e1146640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd8dc0cbd20 0x7fd8dc14c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:02.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.882+0000 7fd8e1146640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd8dc0cbd20 0x7fd8dc14c870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44692/0 (socket says 172.21.15.57:44692) 2023-10-19T13:00:02.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.882+0000 7fd8e1146640 1 -- 172.21.15.57:0/921583865 learned_addr learned my addr 172.21.15.57:0/921583865 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:02.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.883+0000 7fd8e0945640 1 --2- 172.21.15.57:0/921583865 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd8dc0afa20 0x7fd8dc1ef2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:02.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.883+0000 7fd8dbfff640 1 --2- 172.21.15.57:0/921583865 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd8dc0b0530 0x7fd8dc13eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:02.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.883+0000 7fd8e1146640 1 -- 172.21.15.57:0/921583865 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd8dc0afa20 msgr2=0x7fd8dc1ef2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:02.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.883+0000 7fd8e1146640 1 --2- 172.21.15.57:0/921583865 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd8dc0afa20 0x7fd8dc1ef2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:02.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.883+0000 7fd8e1146640 1 -- 172.21.15.57:0/921583865 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd8dc0b0530 msgr2=0x7fd8dc13eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:02.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.883+0000 7fd8e1146640 1 --2- 172.21.15.57:0/921583865 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd8dc0b0530 0x7fd8dc13eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:02.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.883+0000 7fd8e1146640 1 -- 172.21.15.57:0/921583865 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8c4009490 con 0x7fd8dc0cbd20 2023-10-19T13:00:02.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.883+0000 7fd8dbfff640 1 --2- 172.21.15.57:0/921583865 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd8dc0b0530 0x7fd8dc13eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T13:00:02.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.883+0000 7fd8e1146640 1 --2- 172.21.15.57:0/921583865 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd8dc0cbd20 0x7fd8dc14c870 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7fd8c402c370 tx=0x7fd8c40038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:02.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.883+0000 7fd8d97fa640 1 -- 172.21.15.57:0/921583865 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8c4003ea0 con 0x7fd8dc0cbd20 2023-10-19T13:00:02.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.883+0000 7fd8d97fa640 1 -- 172.21.15.57:0/921583865 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fd8c4005020 con 0x7fd8dc0cbd20 2023-10-19T13:00:02.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.883+0000 7fd8e2ba9640 1 -- 172.21.15.57:0/921583865 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8dc143eb0 con 0x7fd8dc0cbd20 2023-10-19T13:00:02.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.883+0000 7fd8d97fa640 1 -- 172.21.15.57:0/921583865 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8c40053b0 con 0x7fd8dc0cbd20 2023-10-19T13:00:02.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.883+0000 7fd8e2ba9640 1 -- 172.21.15.57:0/921583865 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8dc144470 con 0x7fd8dc0cbd20 2023-10-19T13:00:02.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.884+0000 7fd8e2ba9640 1 -- 172.21.15.57:0/921583865 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8dc0b0910 con 0x7fd8dc0cbd20 2023-10-19T13:00:02.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.885+0000 7fd8d97fa640 1 -- 172.21.15.57:0/921583865 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 27) v1 ==== 89360+0+0 (secure 0 0 0) 0x7fd8c4005510 con 0x7fd8dc0cbd20 2023-10-19T13:00:02.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.885+0000 7fd8d97fa640 1 --2- 172.21.15.57:0/921583865 >> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] conn(0x7fd8bc06afb0 0x7fd8bc06d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:02.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.885+0000 7fd8d97fa640 1 -- 172.21.15.57:0/921583865 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fd8c40ac360 con 0x7fd8dc0cbd20 2023-10-19T13:00:02.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.885+0000 7fd8e0945640 1 --2- 172.21.15.57:0/921583865 >> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] conn(0x7fd8bc06afb0 0x7fd8bc06d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:02.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.886+0000 7fd8e0945640 1 --2- 172.21.15.57:0/921583865 >> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] conn(0x7fd8bc06afb0 0x7fd8bc06d470 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd8d40012e0 tx=0x7fd8d4009040 comp rx=0 tx=0).ready entity=mgr.5690 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:02.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:02.889+0000 7fd8d97fa640 1 -- 172.21.15.57:0/921583865 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd8c4077030 con 0x7fd8dc0cbd20 2023-10-19T13:00:03.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:03.113+0000 7fd8e2ba9640 1 -- 172.21.15.57:0/921583865 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7fd8dc0690e0 con 0x7fd8dc0cbd20 2023-10-19T13:00:03.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:03.644+0000 7fd8d97fa640 1 -- 172.21.15.57:0/921583865 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v28) v1 ==== 93+0+0 (secure 0 0 0) 0x7fd8c407a650 con 0x7fd8dc0cbd20 2023-10-19T13:00:03.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:03.646+0000 7fd8e2ba9640 1 -- 172.21.15.57:0/921583865 >> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] conn(0x7fd8bc06afb0 msgr2=0x7fd8bc06d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:03.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:03.646+0000 7fd8e2ba9640 1 --2- 172.21.15.57:0/921583865 >> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] conn(0x7fd8bc06afb0 0x7fd8bc06d470 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd8d40012e0 tx=0x7fd8d4009040 comp rx=0 tx=0).stop 2023-10-19T13:00:03.650 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:03.646+0000 7fd8e2ba9640 1 -- 172.21.15.57:0/921583865 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd8dc0cbd20 msgr2=0x7fd8dc14c870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:03.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:03.646+0000 7fd8e2ba9640 1 --2- 172.21.15.57:0/921583865 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd8dc0cbd20 0x7fd8dc14c870 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7fd8c402c370 tx=0x7fd8c40038a0 comp rx=0 tx=0).stop 2023-10-19T13:00:03.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:03.647+0000 7fd8e2ba9640 1 -- 172.21.15.57:0/921583865 shutdown_connections 2023-10-19T13:00:03.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:03.647+0000 7fd8e2ba9640 1 --2- 172.21.15.57:0/921583865 >> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] conn(0x7fd8bc06afb0 0x7fd8bc06d470 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:03.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:03.647+0000 7fd8e2ba9640 1 --2- 172.21.15.57:0/921583865 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd8dc0cbd20 0x7fd8dc14c870 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:03.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:03.647+0000 7fd8e2ba9640 1 --2- 172.21.15.57:0/921583865 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd8dc0b0530 0x7fd8dc13eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:03.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:03.647+0000 7fd8e2ba9640 1 --2- 172.21.15.57:0/921583865 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd8dc0afa20 0x7fd8dc1ef2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:03.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:03.647+0000 7fd8e2ba9640 1 -- 172.21.15.57:0/921583865 >> 172.21.15.57:0/921583865 conn(0x7fd8dc069930 msgr2=0x7fd8dc069d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:03.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:03.647+0000 7fd8e2ba9640 1 -- 172.21.15.57:0/921583865 shutdown_connections 2023-10-19T13:00:03.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:03.647+0000 7fd8e2ba9640 1 -- 172.21.15.57:0/921583865 wait complete. 2023-10-19T13:00:03.665 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-19T13:00:03.665 DEBUG:teuthology.orchestra.run.smithi057:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-19T13:00:03.717 DEBUG:teuthology.orchestra.run.smithi179:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-19T13:00:03.763 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-19T13:00:03.764 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T13:00:03.764 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T13:00:03.781 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T13:00:03.932 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:03.929+0000 7fd58e010640 1 -- 172.21.15.57:0/3646366195 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5880cf340 msgr2=0x7fd5880d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:03.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:03.929+0000 7fd58e010640 1 --2- 172.21.15.57:0/3646366195 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5880cf340 0x7fd5880d1910 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fd578005fd0 tx=0x7fd57802bd90 comp rx=0 tx=0).stop 2023-10-19T13:00:03.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:03.929+0000 7fd58e010640 1 -- 172.21.15.57:0/3646366195 shutdown_connections 2023-10-19T13:00:03.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:03.929+0000 7fd58e010640 1 --2- 172.21.15.57:0/3646366195 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd5880ceb40 0x7fd5880b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:03.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:03.929+0000 7fd58e010640 1 --2- 172.21.15.57:0/3646366195 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5880cf340 0x7fd5880d1910 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:03.934 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:03.929+0000 7fd58e010640 1 -- 172.21.15.57:0/3646366195 >> 172.21.15.57:0/3646366195 conn(0x7fd5880d0d60 msgr2=0x7fd5880d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:03.934 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:03.929+0000 7fd58e010640 1 -- 172.21.15.57:0/3646366195 shutdown_connections 2023-10-19T13:00:03.934 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:03.929+0000 7fd58e010640 1 -- 172.21.15.57:0/3646366195 wait complete. 2023-10-19T13:00:03.936 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:03.930+0000 7fd58e010640 1 Processor -- start 2023-10-19T13:00:03.936 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:03.931+0000 7fd58e010640 1 -- start start 2023-10-19T13:00:03.936 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:03.931+0000 7fd58e010640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd5880ceb40 0x7fd58809e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:03.937 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:03.931+0000 7fd58e010640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5880cf340 0x7fd5881573f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:03.937 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:03.931+0000 7fd58e010640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd58815b2a0 0x7fd5881f1f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:03.937 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:03.931+0000 7fd58e010640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd58815cb70 con 0x7fd5880cf340 2023-10-19T13:00:03.937 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:03.931+0000 7fd58e010640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd58809e9b0 con 0x7fd58815b2a0 2023-10-19T13:00:03.937 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:03.931+0000 7fd58e010640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd588157930 con 0x7fd5880ceb40 2023-10-19T13:00:03.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:03.931+0000 7fd586ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5880cf340 0x7fd5881573f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:03.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:03.932+0000 7fd586ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5880cf340 0x7fd5881573f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44726/0 (socket says 172.21.15.57:44726) 2023-10-19T13:00:03.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:03.932+0000 7fd586ffd640 1 -- 172.21.15.57:0/1126070004 learned_addr learned my addr 172.21.15.57:0/1126070004 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:03.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:03.932+0000 7fd5877fe640 1 --2- 172.21.15.57:0/1126070004 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd5880ceb40 0x7fd58809e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:03.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:03.932+0000 7fd587fff640 1 --2- 172.21.15.57:0/1126070004 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd58815b2a0 0x7fd5881f1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:03.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:03.932+0000 7fd586ffd640 1 -- 172.21.15.57:0/1126070004 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd5880ceb40 msgr2=0x7fd58809e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:03.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:03.932+0000 7fd586ffd640 1 --2- 172.21.15.57:0/1126070004 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd5880ceb40 0x7fd58809e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:03.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:03.932+0000 7fd586ffd640 1 -- 172.21.15.57:0/1126070004 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd58815b2a0 msgr2=0x7fd5881f1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:03.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:03.932+0000 7fd586ffd640 1 --2- 172.21.15.57:0/1126070004 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd58815b2a0 0x7fd5881f1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:03.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:03.932+0000 7fd586ffd640 1 -- 172.21.15.57:0/1126070004 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd578009560 con 0x7fd5880cf340 2023-10-19T13:00:03.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:03.932+0000 7fd586ffd640 1 --2- 172.21.15.57:0/1126070004 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5880cf340 0x7fd5881573f0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fd57000c400 tx=0x7fd57000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:03.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:03.932+0000 7fd57ffff640 1 -- 172.21.15.57:0/1126070004 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5700052d0 con 0x7fd5880cf340 2023-10-19T13:00:03.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:03.932+0000 7fd57ffff640 1 -- 172.21.15.57:0/1126070004 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fd5700035f0 con 0x7fd5880cf340 2023-10-19T13:00:03.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:03.932+0000 7fd58e010640 1 -- 172.21.15.57:0/1126070004 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5880b0d50 con 0x7fd5880cf340 2023-10-19T13:00:03.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:03.932+0000 7fd57ffff640 1 -- 172.21.15.57:0/1126070004 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd570003a00 con 0x7fd5880cf340 2023-10-19T13:00:03.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:03.932+0000 7fd58e010640 1 -- 172.21.15.57:0/1126070004 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5880c86d0 con 0x7fd5880cf340 2023-10-19T13:00:03.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:03.933+0000 7fd58e010640 1 -- 172.21.15.57:0/1126070004 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5880d1ac0 con 0x7fd5880cf340 2023-10-19T13:00:03.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:03.934+0000 7fd57ffff640 1 -- 172.21.15.57:0/1126070004 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 28) v1 ==== 89343+0+0 (secure 0 0 0) 0x7fd570003b60 con 0x7fd5880cf340 2023-10-19T13:00:03.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:03.934+0000 7fd57ffff640 1 --2- 172.21.15.57:0/1126070004 >> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] conn(0x7fd56806af60 0x7fd56806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:03.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:03.934+0000 7fd5877fe640 1 -- 172.21.15.57:0/1126070004 >> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] conn(0x7fd56806af60 msgr2=0x7fd56806d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/2207781045 2023-10-19T13:00:03.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:03.934+0000 7fd57ffff640 1 -- 172.21.15.57:0/1126070004 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fd5700892e0 con 0x7fd5880cf340 2023-10-19T13:00:03.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:03.934+0000 7fd5877fe640 1 --2- 172.21.15.57:0/1126070004 >> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] conn(0x7fd56806af60 0x7fd56806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T13:00:03.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:03.943+0000 7fd57ffff640 1 -- 172.21.15.57:0/1126070004 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd570053f40 con 0x7fd5880cf340 2023-10-19T13:00:04.035 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:04.032+0000 7fa9e0439040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-19T13:00:04.057 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:00:04.054+0000 7f282504c040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-19T13:00:04.080 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:04.077+0000 7fa9e0439040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-19T13:00:04.104 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:00:04.102+0000 7f282504c040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-19T13:00:04.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.121+0000 7fd58e010640 1 -- 172.21.15.57:0/1126070004 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd5880690e0 con 0x7fd5880cf340 2023-10-19T13:00:04.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.123+0000 7fd57ffff640 1 -- 172.21.15.57:0/1126070004 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v73) v1 ==== 74+0+17387 (secure 0 0 0) 0x7fd570057560 con 0x7fd5880cf340 2023-10-19T13:00:04.126 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:00:04.126 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":73,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T12:59:16.241313+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2023-10-19T12:57:56.901606+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2023-10-19T12:57:57.581428+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.2a","mappings":[{"from":0,"to":6}]},{"pgid":"8.38","mappings":[{"from":0,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.179:0/70135229":"2023-10-20T12:58:14.538805+0000","172.21.15.179:6832/3449041082":"2023-10-20T12:58:14.538805+0000","172.21.15.179:0/1553007782":"2023-10-20T12:58:14.538805+0000","172.21.15.179:0/2777182879":"2023-10-20T12:58:14.538805+0000","172.21.15.179:6833/3449041082":"2023-10-20T12:58:14.538805+0000","172.21.15.179:0/64734340":"2023-10-20T12:58:14.538805+0000","172.21.15.179:0/1129452163":"2023-10-20T12:58:14.538805+0000","172.21.15.179:0/3893958736":"2023-10-20T12:58:14.538805+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T13:00:04.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.124+0000 7fd58e010640 1 -- 172.21.15.57:0/1126070004 >> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] conn(0x7fd56806af60 msgr2=0x7fd56806d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T13:00:04.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.124+0000 7fd58e010640 1 --2- 172.21.15.57:0/1126070004 >> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] conn(0x7fd56806af60 0x7fd56806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:04.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.124+0000 7fd58e010640 1 -- 172.21.15.57:0/1126070004 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5880cf340 msgr2=0x7fd5881573f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:04.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.124+0000 7fd58e010640 1 --2- 172.21.15.57:0/1126070004 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5880cf340 0x7fd5881573f0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fd57000c400 tx=0x7fd57000c8c0 comp rx=0 tx=0).stop 2023-10-19T13:00:04.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.125+0000 7fd58e010640 1 -- 172.21.15.57:0/1126070004 shutdown_connections 2023-10-19T13:00:04.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.125+0000 7fd58e010640 1 --2- 172.21.15.57:0/1126070004 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd58815b2a0 0x7fd5881f1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:04.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.125+0000 7fd58e010640 1 --2- 172.21.15.57:0/1126070004 >> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] conn(0x7fd56806af60 0x7fd56806d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:04.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.125+0000 7fd58e010640 1 --2- 172.21.15.57:0/1126070004 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5880cf340 0x7fd5881573f0 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:04.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.125+0000 7fd58e010640 1 --2- 172.21.15.57:0/1126070004 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd5880ceb40 0x7fd58809e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:04.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.125+0000 7fd58e010640 1 -- 172.21.15.57:0/1126070004 >> 172.21.15.57:0/1126070004 conn(0x7fd5880d0d60 msgr2=0x7fd5880ad8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:04.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.125+0000 7fd58e010640 1 -- 172.21.15.57:0/1126070004 shutdown_connections 2023-10-19T13:00:04.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.125+0000 7fd58e010640 1 -- 172.21.15.57:0/1126070004 wait complete. 2023-10-19T13:00:04.139 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-19T12:44:07.371186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-19T12:57:56.901606+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2023-10-19T12:57:57.581428+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-19T13:00:04.140 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-19T13:00:04.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.250+0000 7fadc5a1a640 1 -- 172.21.15.57:0/1586815357 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fadc00cf340 msgr2=0x7fadc00d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:04.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.250+0000 7fadc5a1a640 1 --2- 172.21.15.57:0/1586815357 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fadc00cf340 0x7fadc00d1910 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fadb4009830 tx=0x7fadb402bcd0 comp rx=0 tx=0).stop 2023-10-19T13:00:04.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.250+0000 7fadc5a1a640 1 -- 172.21.15.57:0/1586815357 shutdown_connections 2023-10-19T13:00:04.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.250+0000 7fadc5a1a640 1 --2- 172.21.15.57:0/1586815357 >> v2:172.21.15.179:3300/0 conn(0x7fadc015b2a0 0x7fadc00cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:04.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.250+0000 7fadc5a1a640 1 --2- 172.21.15.57:0/1586815357 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fadc00cf340 0x7fadc00d1910 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:04.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.250+0000 7fadc5a1a640 1 -- 172.21.15.57:0/1586815357 >> 172.21.15.57:0/1586815357 conn(0x7fadc00d0d60 msgr2=0x7fadc00d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:04.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.250+0000 7fadc5a1a640 1 -- 172.21.15.57:0/1586815357 shutdown_connections 2023-10-19T13:00:04.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.250+0000 7fadc5a1a640 1 -- 172.21.15.57:0/1586815357 wait complete. 2023-10-19T13:00:04.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.251+0000 7fadc5a1a640 1 Processor -- start 2023-10-19T13:00:04.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.252+0000 7fadc5a1a640 1 -- start start 2023-10-19T13:00:04.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.252+0000 7fadc5a1a640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fadc00ceb40 0x7fadc0156130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:04.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.252+0000 7fadc5a1a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fadc00cf340 0x7fadc015da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:04.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.252+0000 7fadc5a1a640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fadc015b2a0 0x7fadc014a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:04.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.252+0000 7fadc5a1a640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadc015d640 con 0x7fadc00cf340 2023-10-19T13:00:04.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.252+0000 7fadc5a1a640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadc00c7450 con 0x7fadc00ceb40 2023-10-19T13:00:04.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.252+0000 7fadc5a1a640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadc00c7f60 con 0x7fadc015b2a0 2023-10-19T13:00:04.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.253+0000 7fadbe7fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fadc00cf340 0x7fadc015da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:04.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.253+0000 7fadbe7fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fadc00cf340 0x7fadc015da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44740/0 (socket says 172.21.15.57:44740) 2023-10-19T13:00:04.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.253+0000 7fadbe7fc640 1 -- 172.21.15.57:0/2516110913 learned_addr learned my addr 172.21.15.57:0/2516110913 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:04.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.253+0000 7fadbf7fe640 1 --2- 172.21.15.57:0/2516110913 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fadc015b2a0 0x7fadc014a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:04.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.253+0000 7fadbeffd640 1 --2- 172.21.15.57:0/2516110913 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fadc00ceb40 0x7fadc0156130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:04.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.253+0000 7fadbe7fc640 1 -- 172.21.15.57:0/2516110913 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fadc015b2a0 msgr2=0x7fadc014a140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:04.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.253+0000 7fadbe7fc640 1 --2- 172.21.15.57:0/2516110913 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fadc015b2a0 0x7fadc014a140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:04.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.253+0000 7fadbe7fc640 1 -- 172.21.15.57:0/2516110913 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fadc00ceb40 msgr2=0x7fadc0156130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:04.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.253+0000 7fadbe7fc640 1 --2- 172.21.15.57:0/2516110913 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fadc00ceb40 0x7fadc0156130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:04.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.253+0000 7fadbe7fc640 1 -- 172.21.15.57:0/2516110913 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadb4009490 con 0x7fadc00cf340 2023-10-19T13:00:04.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.253+0000 7fadbe7fc640 1 --2- 172.21.15.57:0/2516110913 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fadc00cf340 0x7fadc015da60 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fadb0009fd0 tx=0x7fadb000ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:04.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.253+0000 7fada7fff640 1 -- 172.21.15.57:0/2516110913 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadb0009580 con 0x7fadc00cf340 2023-10-19T13:00:04.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.253+0000 7fada7fff640 1 -- 172.21.15.57:0/2516110913 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fadb000cc60 con 0x7fadc00cf340 2023-10-19T13:00:04.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.253+0000 7fadc5a1a640 1 -- 172.21.15.57:0/2516110913 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadc00c8240 con 0x7fadc00cf340 2023-10-19T13:00:04.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.253+0000 7fada7fff640 1 -- 172.21.15.57:0/2516110913 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadb0005280 con 0x7fadc00cf340 2023-10-19T13:00:04.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.253+0000 7fadc5a1a640 1 -- 172.21.15.57:0/2516110913 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadc00c8520 con 0x7fadc00cf340 2023-10-19T13:00:04.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.254+0000 7fadc5a1a640 1 -- 172.21.15.57:0/2516110913 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadc00690e0 con 0x7fadc00cf340 2023-10-19T13:00:04.262 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.255+0000 7fada7fff640 1 -- 172.21.15.57:0/2516110913 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 28) v1 ==== 89343+0+0 (secure 0 0 0) 0x7fadb00053e0 con 0x7fadc00cf340 2023-10-19T13:00:04.262 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.255+0000 7fada7fff640 1 --2- 172.21.15.57:0/2516110913 >> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] conn(0x7fadac06af90 0x7fadac06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:04.263 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.255+0000 7fadbeffd640 1 -- 172.21.15.57:0/2516110913 >> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] conn(0x7fadac06af90 msgr2=0x7fadac06d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/2207781045 2023-10-19T13:00:04.263 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.255+0000 7fada7fff640 1 -- 172.21.15.57:0/2516110913 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fadb0014070 con 0x7fadc00cf340 2023-10-19T13:00:04.263 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.255+0000 7fadbeffd640 1 --2- 172.21.15.57:0/2516110913 >> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] conn(0x7fadac06af90 0x7fadac06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T13:00:04.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.263+0000 7fada7fff640 1 -- 172.21.15.57:0/2516110913 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fadb0058a00 con 0x7fadc00cf340 2023-10-19T13:00:04.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.441+0000 7fadc5a1a640 1 -- 172.21.15.57:0/2516110913 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fadc00d0b40 con 0x7fadc00cf340 2023-10-19T13:00:04.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.441+0000 7fada7fff640 1 -- 172.21.15.57:0/2516110913 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v73) v1 ==== 93+0+10 (secure 0 0 0) 0x7fadb0058590 con 0x7fadc00cf340 2023-10-19T13:00:04.444 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-19T13:00:04.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.443+0000 7fadc5a1a640 1 -- 172.21.15.57:0/2516110913 >> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] conn(0x7fadac06af90 msgr2=0x7fadac06d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T13:00:04.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.443+0000 7fadc5a1a640 1 --2- 172.21.15.57:0/2516110913 >> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] conn(0x7fadac06af90 0x7fadac06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:04.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.443+0000 7fadc5a1a640 1 -- 172.21.15.57:0/2516110913 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fadc00cf340 msgr2=0x7fadc015da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:04.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.443+0000 7fadc5a1a640 1 --2- 172.21.15.57:0/2516110913 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fadc00cf340 0x7fadc015da60 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fadb0009fd0 tx=0x7fadb000ec20 comp rx=0 tx=0).stop 2023-10-19T13:00:04.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.443+0000 7fadc5a1a640 1 -- 172.21.15.57:0/2516110913 shutdown_connections 2023-10-19T13:00:04.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.443+0000 7fadc5a1a640 1 --2- 172.21.15.57:0/2516110913 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fadc015b2a0 0x7fadc014a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:04.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.443+0000 7fadc5a1a640 1 --2- 172.21.15.57:0/2516110913 >> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] conn(0x7fadac06af90 0x7fadac06d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:04.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.443+0000 7fadc5a1a640 1 --2- 172.21.15.57:0/2516110913 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fadc00cf340 0x7fadc015da60 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:04.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.443+0000 7fadc5a1a640 1 --2- 172.21.15.57:0/2516110913 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fadc00ceb40 0x7fadc0156130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:04.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.443+0000 7fadc5a1a640 1 -- 172.21.15.57:0/2516110913 >> 172.21.15.57:0/2516110913 conn(0x7fadc00d0d60 msgr2=0x7fadc00af020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:04.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.443+0000 7fadc5a1a640 1 -- 172.21.15.57:0/2516110913 shutdown_connections 2023-10-19T13:00:04.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.443+0000 7fadc5a1a640 1 -- 172.21.15.57:0/2516110913 wait complete. 2023-10-19T13:00:04.457 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-19T13:00:04.468 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:04.465+0000 7fa9e0439040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-19T13:00:04.511 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:04.509+0000 7fa9e0439040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-19T13:00:04.528 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:00:04.525+0000 7f282504c040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-19T13:00:04.554 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:04.551+0000 7fa9e0439040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-19T13:00:04.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.568+0000 7f207ffd0640 1 -- 172.21.15.57:0/1852796373 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f20780cf340 msgr2=0x7f20780d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:04.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.568+0000 7f207ffd0640 1 --2- 172.21.15.57:0/1852796373 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f20780cf340 0x7f20780d1910 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f207808a120 tx=0x7f207402bdb0 comp rx=0 tx=0).stop 2023-10-19T13:00:04.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.569+0000 7f207ffd0640 1 -- 172.21.15.57:0/1852796373 shutdown_connections 2023-10-19T13:00:04.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.569+0000 7f207ffd0640 1 --2- 172.21.15.57:0/1852796373 >> v2:172.21.15.179:3300/0 conn(0x7f207815b2a0 0x7f20780cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:04.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.569+0000 7f207ffd0640 1 --2- 172.21.15.57:0/1852796373 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f20780cf340 0x7f20780d1910 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:04.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.569+0000 7f207ffd0640 1 -- 172.21.15.57:0/1852796373 >> 172.21.15.57:0/1852796373 conn(0x7f20780d0d60 msgr2=0x7f20780d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:04.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.569+0000 7f207ffd0640 1 -- 172.21.15.57:0/1852796373 shutdown_connections 2023-10-19T13:00:04.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.569+0000 7f207ffd0640 1 -- 172.21.15.57:0/1852796373 wait complete. 2023-10-19T13:00:04.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.570+0000 7f207ffd0640 1 Processor -- start 2023-10-19T13:00:04.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.570+0000 7f207ffd0640 1 -- start start 2023-10-19T13:00:04.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.571+0000 7f207ffd0640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f20780ceb40 0x7f20780c87e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:04.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.571+0000 7f207ffd0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f20780cf340 0x7f207806b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:04.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.571+0000 7f207ffd0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f207815b2a0 0x7f207814bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:04.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.571+0000 7f207ffd0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20780ca0d0 con 0x7f207815b2a0 2023-10-19T13:00:04.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.571+0000 7f207ffd0640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20780ca3b0 con 0x7f20780cf340 2023-10-19T13:00:04.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.571+0000 7f207ffd0640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20780ca690 con 0x7f20780ceb40 2023-10-19T13:00:04.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.571+0000 7f207e56d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f207815b2a0 0x7f207814bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:04.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.571+0000 7f207e56d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f207815b2a0 0x7f207814bd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44756/0 (socket says 172.21.15.57:44756) 2023-10-19T13:00:04.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.571+0000 7f207e56d640 1 -- 172.21.15.57:0/103882656 learned_addr learned my addr 172.21.15.57:0/103882656 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:04.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.571+0000 7f207d56b640 1 --2- 172.21.15.57:0/103882656 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f20780cf340 0x7f207806b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:04.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.571+0000 7f207dd6c640 1 --2- 172.21.15.57:0/103882656 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f20780ceb40 0x7f20780c87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:04.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.571+0000 7f207e56d640 1 -- 172.21.15.57:0/103882656 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f20780ceb40 msgr2=0x7f20780c87e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:04.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.571+0000 7f207e56d640 1 --2- 172.21.15.57:0/103882656 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f20780ceb40 0x7f20780c87e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:04.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.571+0000 7f207e56d640 1 -- 172.21.15.57:0/103882656 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f20780cf340 msgr2=0x7f207806b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:04.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.571+0000 7f207e56d640 1 --2- 172.21.15.57:0/103882656 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f20780cf340 0x7f207806b3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:04.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.571+0000 7f207e56d640 1 -- 172.21.15.57:0/103882656 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2074009560 con 0x7f207815b2a0 2023-10-19T13:00:04.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.571+0000 7f207e56d640 1 --2- 172.21.15.57:0/103882656 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f207815b2a0 0x7f207814bd80 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f207000c730 tx=0x7f207000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:04.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.571+0000 7f206e7fc640 1 -- 172.21.15.57:0/103882656 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2070003b90 con 0x7f207815b2a0 2023-10-19T13:00:04.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.571+0000 7f206e7fc640 1 -- 172.21.15.57:0/103882656 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f2070003cf0 con 0x7f207815b2a0 2023-10-19T13:00:04.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.571+0000 7f206e7fc640 1 -- 172.21.15.57:0/103882656 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f207000aab0 con 0x7f207815b2a0 2023-10-19T13:00:04.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.572+0000 7f207ffd0640 1 -- 172.21.15.57:0/103882656 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20780ca970 con 0x7f207815b2a0 2023-10-19T13:00:04.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.572+0000 7f207ffd0640 1 -- 172.21.15.57:0/103882656 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f207806b8e0 con 0x7f207815b2a0 2023-10-19T13:00:04.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.573+0000 7f207ffd0640 1 -- 172.21.15.57:0/103882656 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20780b3950 con 0x7f207815b2a0 2023-10-19T13:00:04.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.573+0000 7f206e7fc640 1 -- 172.21.15.57:0/103882656 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 28) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f207000ac10 con 0x7f207815b2a0 2023-10-19T13:00:04.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.573+0000 7f206e7fc640 1 --2- 172.21.15.57:0/103882656 >> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] conn(0x7f205c06af60 0x7f205c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:04.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.574+0000 7f207dd6c640 1 -- 172.21.15.57:0/103882656 >> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] conn(0x7f205c06af60 msgr2=0x7f205c06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/2207781045 2023-10-19T13:00:04.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.574+0000 7f206e7fc640 1 -- 172.21.15.57:0/103882656 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f20700898a0 con 0x7f207815b2a0 2023-10-19T13:00:04.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.574+0000 7f207dd6c640 1 --2- 172.21.15.57:0/103882656 >> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] conn(0x7f205c06af60 0x7f205c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T13:00:04.582 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:00:04.574+0000 7f282504c040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-19T13:00:04.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.578+0000 7f206e7fc640 1 -- 172.21.15.57:0/103882656 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f20780b3950 con 0x7f207815b2a0 2023-10-19T13:00:04.606 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:04.603+0000 7fa9e0439040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-19T13:00:04.622 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:00:04.620+0000 7f282504c040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-19T13:00:04.648 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:04.645+0000 7fa9e0439040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-19T13:00:04.679 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:00:04.677+0000 7f282504c040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-19T13:00:04.723 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:00:04.720+0000 7f282504c040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-19T13:00:04.749 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:04.746+0000 7fa9e0439040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-19T13:00:04.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.756+0000 7f207ffd0640 1 -- 172.21.15.57:0/103882656 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f20780b3950 con 0x7f207815b2a0 2023-10-19T13:00:04.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.757+0000 7f206e7fc640 1 -- 172.21.15.57:0/103882656 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v74) v1 ==== 104+0+11 (secure 0 0 0) 0x7f20780b3950 con 0x7f207815b2a0 2023-10-19T13:00:04.760 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2023-10-19T13:00:04.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.759+0000 7f207ffd0640 1 -- 172.21.15.57:0/103882656 >> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] conn(0x7f205c06af60 msgr2=0x7f205c06d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T13:00:04.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.759+0000 7f207ffd0640 1 --2- 172.21.15.57:0/103882656 >> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] conn(0x7f205c06af60 0x7f205c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:04.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.759+0000 7f207ffd0640 1 -- 172.21.15.57:0/103882656 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f207815b2a0 msgr2=0x7f207814bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:04.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.759+0000 7f207ffd0640 1 --2- 172.21.15.57:0/103882656 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f207815b2a0 0x7f207814bd80 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f207000c730 tx=0x7f207000cbf0 comp rx=0 tx=0).stop 2023-10-19T13:00:04.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.759+0000 7f207ffd0640 1 -- 172.21.15.57:0/103882656 shutdown_connections 2023-10-19T13:00:04.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.759+0000 7f207ffd0640 1 --2- 172.21.15.57:0/103882656 >> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] conn(0x7f205c06af60 0x7f205c06d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:04.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.759+0000 7f207ffd0640 1 --2- 172.21.15.57:0/103882656 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f207815b2a0 0x7f207814bd80 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:04.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.759+0000 7f207ffd0640 1 --2- 172.21.15.57:0/103882656 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f20780cf340 0x7f207806b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:04.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.759+0000 7f207ffd0640 1 --2- 172.21.15.57:0/103882656 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f20780ceb40 0x7f20780c87e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:04.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.759+0000 7f207ffd0640 1 -- 172.21.15.57:0/103882656 >> 172.21.15.57:0/103882656 conn(0x7f20780d0d60 msgr2=0x7f20780aee40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:04.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.759+0000 7f207ffd0640 1 -- 172.21.15.57:0/103882656 shutdown_connections 2023-10-19T13:00:04.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.759+0000 7f207ffd0640 1 -- 172.21.15.57:0/103882656 wait complete. 2023-10-19T13:00:04.772 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-19T13:00:04.801 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:04.799+0000 7fa9e0439040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-19T13:00:04.828 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:00:04.826+0000 7f282504c040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-19T13:00:04.842 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:04.840+0000 7fa9e0439040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-19T13:00:04.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.882+0000 7f115ef9e640 1 -- 172.21.15.57:0/2763412581 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f11580cf850 msgr2=0x7f11580cc9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:04.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.882+0000 7f115ef9e640 1 --2- 172.21.15.57:0/2763412581 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f11580cf850 0x7f11580cc9f0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f11580894b0 tx=0x7f115002c2f0 comp rx=0 tx=0).stop 2023-10-19T13:00:04.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.882+0000 7f115ef9e640 1 -- 172.21.15.57:0/2763412581 shutdown_connections 2023-10-19T13:00:04.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.882+0000 7f115ef9e640 1 --2- 172.21.15.57:0/2763412581 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f11580cf850 0x7f11580cc9f0 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:04.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.882+0000 7f115ef9e640 1 --2- 172.21.15.57:0/2763412581 >> v2:172.21.15.179:3300/0 conn(0x7f115815b260 0x7f11580cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:04.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.882+0000 7f115ef9e640 1 -- 172.21.15.57:0/2763412581 >> 172.21.15.57:0/2763412581 conn(0x7f115809a030 msgr2=0x7f11580d0d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:04.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.882+0000 7f115ef9e640 1 -- 172.21.15.57:0/2763412581 shutdown_connections 2023-10-19T13:00:04.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.882+0000 7f115ef9e640 1 -- 172.21.15.57:0/2763412581 wait complete. 2023-10-19T13:00:04.887 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:00:04.883+0000 7f282504c040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-19T13:00:04.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.884+0000 7f115ef9e640 1 Processor -- start 2023-10-19T13:00:04.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.884+0000 7f115ef9e640 1 -- start start 2023-10-19T13:00:04.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.884+0000 7f115ef9e640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f11580cf850 0x7f1158157940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:04.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.885+0000 7f115ef9e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f115815b260 0x7f11580b6510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:04.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.885+0000 7f115ef9e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f115815b640 0x7f115806b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:04.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.885+0000 7f115ef9e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11580b5bb0 con 0x7f115815b260 2023-10-19T13:00:04.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.885+0000 7f115ef9e640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11581594f0 con 0x7f115815b640 2023-10-19T13:00:04.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.885+0000 7f115ef9e640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11580b4fa0 con 0x7f11580cf850 2023-10-19T13:00:04.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.885+0000 7f1157fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f115815b260 0x7f11580b6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:04.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.885+0000 7f1157fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f115815b260 0x7f11580b6510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44776/0 (socket says 172.21.15.57:44776) 2023-10-19T13:00:04.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.885+0000 7f1157fff640 1 -- 172.21.15.57:0/623271759 learned_addr learned my addr 172.21.15.57:0/623271759 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:04.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.885+0000 7f115d53b640 1 --2- 172.21.15.57:0/623271759 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f115815b640 0x7f115806b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:04.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.885+0000 7f115cd3a640 1 --2- 172.21.15.57:0/623271759 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f11580cf850 0x7f1158157940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:04.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.885+0000 7f1157fff640 1 -- 172.21.15.57:0/623271759 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f11580cf850 msgr2=0x7f1158157940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:04.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.885+0000 7f1157fff640 1 --2- 172.21.15.57:0/623271759 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f11580cf850 0x7f1158157940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:04.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.885+0000 7f1157fff640 1 -- 172.21.15.57:0/623271759 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f115815b640 msgr2=0x7f115806b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:04.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.885+0000 7f1157fff640 1 --2- 172.21.15.57:0/623271759 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f115815b640 0x7f115806b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:04.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.885+0000 7f1157fff640 1 -- 172.21.15.57:0/623271759 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1150009900 con 0x7f115815b260 2023-10-19T13:00:04.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.885+0000 7f115d53b640 1 --2- 172.21.15.57:0/623271759 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f115815b640 0x7f115806b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T13:00:04.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.885+0000 7f1157fff640 1 --2- 172.21.15.57:0/623271759 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f115815b260 0x7f11580b6510 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f114400c400 tx=0x7f114400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:04.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.885+0000 7f11557fa640 1 -- 172.21.15.57:0/623271759 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11440052d0 con 0x7f115815b260 2023-10-19T13:00:04.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.886+0000 7f11557fa640 1 -- 172.21.15.57:0/623271759 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f11440035f0 con 0x7f115815b260 2023-10-19T13:00:04.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.886+0000 7f115ef9e640 1 -- 172.21.15.57:0/623271759 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11580b5280 con 0x7f115815b260 2023-10-19T13:00:04.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.886+0000 7f11557fa640 1 -- 172.21.15.57:0/623271759 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1144003a00 con 0x7f115815b260 2023-10-19T13:00:04.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.886+0000 7f115ef9e640 1 -- 172.21.15.57:0/623271759 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11580b5840 con 0x7f115815b260 2023-10-19T13:00:04.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.886+0000 7f115ef9e640 1 -- 172.21.15.57:0/623271759 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11580675f0 con 0x7f115815b260 2023-10-19T13:00:04.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.887+0000 7f11557fa640 1 -- 172.21.15.57:0/623271759 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 28) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f1144003b60 con 0x7f115815b260 2023-10-19T13:00:04.895 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:04.888+0000 7fa9e0439040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-19T13:00:04.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.888+0000 7f11557fa640 1 --2- 172.21.15.57:0/623271759 >> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] conn(0x7f113c06af60 0x7f113c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:04.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.888+0000 7f115cd3a640 1 -- 172.21.15.57:0/623271759 >> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] conn(0x7f113c06af60 msgr2=0x7f113c06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/2207781045 2023-10-19T13:00:04.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.888+0000 7f11557fa640 1 -- 172.21.15.57:0/623271759 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6519+0+0 (secure 0 0 0) 0x7f1144089320 con 0x7f115815b260 2023-10-19T13:00:04.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.888+0000 7f115cd3a640 1 --2- 172.21.15.57:0/623271759 >> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] conn(0x7f113c06af60 0x7f113c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T13:00:04.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:04.892+0000 7f11557fa640 1 -- 172.21.15.57:0/623271759 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f1144053f30 con 0x7f115815b260 2023-10-19T13:00:04.931 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:00:04.928+0000 7f282504c040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-19T13:00:04.933 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:04.930+0000 7fa9e0439040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-19T13:00:04.987 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:00:04.984+0000 7f282504c040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-19T13:00:05.034 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:00:05.032+0000 7f282504c040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-19T13:00:05.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.078+0000 7f115ef9e640 1 -- 172.21.15.57:0/623271759 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f11580aeb50 con 0x7f115815b260 2023-10-19T13:00:05.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.078+0000 7f11557fa640 1 -- 172.21.15.57:0/623271759 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v74) v1 ==== 100+0+11 (secure 0 0 0) 0x7f1144057550 con 0x7f115815b260 2023-10-19T13:00:05.081 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 64 2023-10-19T13:00:05.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.080+0000 7f115ef9e640 1 -- 172.21.15.57:0/623271759 >> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] conn(0x7f113c06af60 msgr2=0x7f113c06d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T13:00:05.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.080+0000 7f115ef9e640 1 --2- 172.21.15.57:0/623271759 >> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] conn(0x7f113c06af60 0x7f113c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:05.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.080+0000 7f115ef9e640 1 -- 172.21.15.57:0/623271759 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f115815b260 msgr2=0x7f11580b6510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:05.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.080+0000 7f115ef9e640 1 --2- 172.21.15.57:0/623271759 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f115815b260 0x7f11580b6510 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f114400c400 tx=0x7f114400c8c0 comp rx=0 tx=0).stop 2023-10-19T13:00:05.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.081+0000 7f115ef9e640 1 -- 172.21.15.57:0/623271759 shutdown_connections 2023-10-19T13:00:05.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.081+0000 7f115ef9e640 1 --2- 172.21.15.57:0/623271759 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f115815b640 0x7f115806b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:05.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.081+0000 7f115ef9e640 1 --2- 172.21.15.57:0/623271759 >> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] conn(0x7f113c06af60 0x7f113c06d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:05.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.081+0000 7f115ef9e640 1 --2- 172.21.15.57:0/623271759 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f115815b260 0x7f11580b6510 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:05.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.081+0000 7f115ef9e640 1 --2- 172.21.15.57:0/623271759 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f11580cf850 0x7f1158157940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:05.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.081+0000 7f115ef9e640 1 -- 172.21.15.57:0/623271759 >> 172.21.15.57:0/623271759 conn(0x7f115809a030 msgr2=0x7f115806a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:05.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.081+0000 7f115ef9e640 1 -- 172.21.15.57:0/623271759 shutdown_connections 2023-10-19T13:00:05.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.081+0000 7f115ef9e640 1 -- 172.21.15.57:0/623271759 wait complete. 2023-10-19T13:00:05.094 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-19T13:00:05.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.203+0000 7f572787c640 1 -- 172.21.15.57:0/4047526273 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f57200d1910 msgr2=0x7f57200b0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:05.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.203+0000 7f572787c640 1 --2- 172.21.15.57:0/4047526273 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f57200d1910 0x7f57200b0650 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f5718005fd0 tx=0x7f571802bfc0 comp rx=0 tx=0).stop 2023-10-19T13:00:05.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.203+0000 7f572787c640 1 -- 172.21.15.57:0/4047526273 shutdown_connections 2023-10-19T13:00:05.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.203+0000 7f572787c640 1 --2- 172.21.15.57:0/4047526273 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f57200d1910 0x7f57200b0650 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:05.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.203+0000 7f572787c640 1 -- 172.21.15.57:0/4047526273 >> 172.21.15.57:0/4047526273 conn(0x7f57200d0d60 msgr2=0x7f57200d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:05.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.203+0000 7f572787c640 1 -- 172.21.15.57:0/4047526273 shutdown_connections 2023-10-19T13:00:05.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.203+0000 7f572787c640 1 -- 172.21.15.57:0/4047526273 wait complete. 2023-10-19T13:00:05.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.204+0000 7f572787c640 1 Processor -- start 2023-10-19T13:00:05.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.205+0000 7f572787c640 1 -- start start 2023-10-19T13:00:05.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.205+0000 7f572787c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f57200cf340 0x7f57201f1060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:05.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.205+0000 7f572787c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f57200d1910 0x7f57200c8860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:05.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.205+0000 7f572787c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f572015b2a0 0x7f572014ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:05.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.205+0000 7f572787c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f572014ad30 con 0x7f572015b2a0 2023-10-19T13:00:05.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.205+0000 7f572787c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f572014f2c0 con 0x7f57200cf340 2023-10-19T13:00:05.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.205+0000 7f572787c640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f572014bb20 con 0x7f57200d1910 2023-10-19T13:00:05.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.205+0000 7f5725e19640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f572015b2a0 0x7f572014ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:05.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.205+0000 7f5725e19640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f572015b2a0 0x7f572014ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44790/0 (socket says 172.21.15.57:44790) 2023-10-19T13:00:05.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.205+0000 7f5725e19640 1 -- 172.21.15.57:0/2773192680 learned_addr learned my addr 172.21.15.57:0/2773192680 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:05.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.205+0000 7f5725618640 1 --2- 172.21.15.57:0/2773192680 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f57200cf340 0x7f57201f1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:05.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.205+0000 7f5724e17640 1 --2- 172.21.15.57:0/2773192680 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f57200d1910 0x7f57200c8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:05.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.206+0000 7f5725e19640 1 -- 172.21.15.57:0/2773192680 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f57200d1910 msgr2=0x7f57200c8860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:05.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.206+0000 7f5725e19640 1 --2- 172.21.15.57:0/2773192680 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f57200d1910 0x7f57200c8860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:05.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.206+0000 7f5725e19640 1 -- 172.21.15.57:0/2773192680 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f57200cf340 msgr2=0x7f57201f1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:05.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.206+0000 7f5725e19640 1 --2- 172.21.15.57:0/2773192680 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f57200cf340 0x7f57201f1060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:05.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.206+0000 7f5725e19640 1 -- 172.21.15.57:0/2773192680 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5718009840 con 0x7f572015b2a0 2023-10-19T13:00:05.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.206+0000 7f5724e17640 1 --2- 172.21.15.57:0/2773192680 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f57200d1910 0x7f57200c8860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T13:00:05.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.206+0000 7f5725618640 1 --2- 172.21.15.57:0/2773192680 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f57200cf340 0x7f57201f1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T13:00:05.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.206+0000 7f5725e19640 1 --2- 172.21.15.57:0/2773192680 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f572015b2a0 0x7f572014ed80 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f5718036040 tx=0x7f5718003f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:05.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.206+0000 7f5715ffb640 1 -- 172.21.15.57:0/2773192680 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57180043d0 con 0x7f572015b2a0 2023-10-19T13:00:05.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.206+0000 7f5715ffb640 1 -- 172.21.15.57:0/2773192680 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f5718004640 con 0x7f572015b2a0 2023-10-19T13:00:05.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.206+0000 7f572787c640 1 -- 172.21.15.57:0/2773192680 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5720148a20 con 0x7f572015b2a0 2023-10-19T13:00:05.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.206+0000 7f5715ffb640 1 -- 172.21.15.57:0/2773192680 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f571803e450 con 0x7f572015b2a0 2023-10-19T13:00:05.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.207+0000 7f572787c640 1 -- 172.21.15.57:0/2773192680 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f572014b1f0 con 0x7f572015b2a0 2023-10-19T13:00:05.217 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.208+0000 7f572787c640 1 -- 172.21.15.57:0/2773192680 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57200690e0 con 0x7f572015b2a0 2023-10-19T13:00:05.217 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.208+0000 7f5715ffb640 1 -- 172.21.15.57:0/2773192680 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 28) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f571803e5b0 con 0x7f572015b2a0 2023-10-19T13:00:05.217 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.208+0000 7f5715ffb640 1 --2- 172.21.15.57:0/2773192680 >> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] conn(0x7f570406af60 0x7f570406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:05.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.208+0000 7f5715ffb640 1 -- 172.21.15.57:0/2773192680 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6519+0+0 (secure 0 0 0) 0x7f57180ac8a0 con 0x7f572015b2a0 2023-10-19T13:00:05.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.208+0000 7f5725618640 1 -- 172.21.15.57:0/2773192680 >> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] conn(0x7f570406af60 msgr2=0x7f570406d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/2207781045 2023-10-19T13:00:05.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.209+0000 7f5725618640 1 --2- 172.21.15.57:0/2773192680 >> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] conn(0x7f570406af60 0x7f570406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T13:00:05.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.213+0000 7f5715ffb640 1 -- 172.21.15.57:0/2773192680 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f57180776e0 con 0x7f572015b2a0 2023-10-19T13:00:05.254 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:05.251+0000 7fa9e0439040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-19T13:00:05.375 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:00:05.372+0000 7f282504c040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-19T13:00:05.384 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:05.382+0000 7fa9e0439040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-19T13:00:05.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.402+0000 7f572787c640 1 -- 172.21.15.57:0/2773192680 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f572014bc60 con 0x7f572015b2a0 2023-10-19T13:00:05.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.402+0000 7f5715ffb640 1 -- 172.21.15.57:0/2773192680 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v74) v1 ==== 80+0+474 (secure 0 0 0) 0x7f5718043080 con 0x7f572015b2a0 2023-10-19T13:00:05.405 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/70135229 2023-10-20T12:58:14.538805+0000 2023-10-19T13:00:05.406 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:6832/3449041082 2023-10-20T12:58:14.538805+0000 2023-10-19T13:00:05.406 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/1553007782 2023-10-20T12:58:14.538805+0000 2023-10-19T13:00:05.406 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/2777182879 2023-10-20T12:58:14.538805+0000 2023-10-19T13:00:05.406 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:6833/3449041082 2023-10-20T12:58:14.538805+0000 2023-10-19T13:00:05.406 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/64734340 2023-10-20T12:58:14.538805+0000 2023-10-19T13:00:05.407 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/1129452163 2023-10-20T12:58:14.538805+0000 2023-10-19T13:00:05.407 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/3893958736 2023-10-20T12:58:14.538805+0000 2023-10-19T13:00:05.407 INFO:teuthology.orchestra.run.smithi057.stderr:listed 8 entries 2023-10-19T13:00:05.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.404+0000 7f572787c640 1 -- 172.21.15.57:0/2773192680 >> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] conn(0x7f570406af60 msgr2=0x7f570406d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T13:00:05.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.404+0000 7f572787c640 1 --2- 172.21.15.57:0/2773192680 >> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] conn(0x7f570406af60 0x7f570406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:05.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.404+0000 7f572787c640 1 -- 172.21.15.57:0/2773192680 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f572015b2a0 msgr2=0x7f572014ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:05.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.404+0000 7f572787c640 1 --2- 172.21.15.57:0/2773192680 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f572015b2a0 0x7f572014ed80 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f5718036040 tx=0x7f5718003f20 comp rx=0 tx=0).stop 2023-10-19T13:00:05.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.404+0000 7f572787c640 1 -- 172.21.15.57:0/2773192680 shutdown_connections 2023-10-19T13:00:05.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.404+0000 7f572787c640 1 --2- 172.21.15.57:0/2773192680 >> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] conn(0x7f570406af60 0x7f570406d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:05.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.404+0000 7f572787c640 1 --2- 172.21.15.57:0/2773192680 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f572015b2a0 0x7f572014ed80 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:05.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.404+0000 7f572787c640 1 --2- 172.21.15.57:0/2773192680 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f57200d1910 0x7f57200c8860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:05.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.404+0000 7f572787c640 1 --2- 172.21.15.57:0/2773192680 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f57200cf340 0x7f57201f1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:05.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.404+0000 7f572787c640 1 -- 172.21.15.57:0/2773192680 >> 172.21.15.57:0/2773192680 conn(0x7f57200d0d60 msgr2=0x7f57200b0da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:05.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.404+0000 7f572787c640 1 -- 172.21.15.57:0/2773192680 shutdown_connections 2023-10-19T13:00:05.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.404+0000 7f572787c640 1 -- 172.21.15.57:0/2773192680 wait complete. 2023-10-19T13:00:05.417 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi057.front.sepia.ceph.com... 2023-10-19T13:00:05.418 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:00:05.418 DEBUG:teuthology.orchestra.run.smithi057:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-10-19T13:00:05.447 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi057.stderr:ceph-fuse[113835]: fuse finished with error 0 and tester_r 0 2023-10-19T13:00:05.449 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi057.front.sepia.ceph.com 2023-10-19T13:00:05.450 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:00:05.450 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-19T13:00:05.475 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:05.472+0000 7fa9e0439040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-19T13:00:05.516 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:00:05.514+0000 7f282504c040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-19T13:00:05.517 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:05.514+0000 7fa9e0439040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-19T13:00:05.610 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:05.608+0000 7fa9e0439040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-19T13:00:05.614 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:00:05.611+0000 7f282504c040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-19T13:00:05.615 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-19T13:00:05.615 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T13:00:05.631 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054833,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=284 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3247632k,nr_inodes=811908,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-19T13:00:05.631 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi179.front.sepia.ceph.com 2023-10-19T13:00:05.632 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:00:05.632 DEBUG:teuthology.orchestra.run.smithi179:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-10-19T13:00:05.663 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:00:05.660+0000 7f282504c040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-19T13:00:05.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-19T13:00:05.687 INFO:teuthology.orchestra.run.smithi179.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2023-10-19T13:00:05.688 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T13:00:05.688 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T13:00:05.703 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055614,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=358 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248260k,nr_inodes=812065,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-19T13:00:05.704 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi057.front.sepia.ceph.com 2023-10-19T13:00:05.704 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:00:05.705 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-10-19T13:00:05.708 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:05.706+0000 7fa9e0439040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-19T13:00:05.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-19T13:00:05.759 INFO:teuthology.orchestra.run.smithi057.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2023-10-19T13:00:05.759 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-19T13:00:05.760 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T13:00:05.764 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:00:05.762+0000 7f282504c040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-19T13:00:05.775 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054833,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=284 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3247632k,nr_inodes=811908,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-19T13:00:05.775 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi179.front.sepia.ceph.com 2023-10-19T13:00:05.776 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:00:05.776 DEBUG:teuthology.orchestra.run.smithi179:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-10-19T13:00:05.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-19T13:00:05.834 INFO:teuthology.orchestra.run.smithi179.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2023-10-19T13:00:05.835 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T13:00:05.853 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:05.851+0000 7fa9e0439040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-19T13:00:05.864 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:00:05.861+0000 7f282504c040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-19T13:00:05.896 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:05.894+0000 7fa9e0439040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-19T13:00:05.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.942+0000 7f4eeb7d0640 1 -- 172.21.15.57:0/3510371941 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4ee40cf340 msgr2=0x7f4ee40d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:05.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.942+0000 7f4eeb7d0640 1 --2- 172.21.15.57:0/3510371941 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4ee40cf340 0x7f4ee40d1910 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f4ee0005fd0 tx=0x7f4ee002bd90 comp rx=0 tx=0).stop 2023-10-19T13:00:05.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.942+0000 7f4eeb7d0640 1 -- 172.21.15.57:0/3510371941 shutdown_connections 2023-10-19T13:00:05.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.942+0000 7f4eeb7d0640 1 --2- 172.21.15.57:0/3510371941 >> v2:172.21.15.179:3300/0 conn(0x7f4ee40ceb40 0x7f4ee40b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:05.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.942+0000 7f4eeb7d0640 1 --2- 172.21.15.57:0/3510371941 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4ee40cf340 0x7f4ee40d1910 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:05.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.942+0000 7f4eeb7d0640 1 -- 172.21.15.57:0/3510371941 >> 172.21.15.57:0/3510371941 conn(0x7f4ee40d0d60 msgr2=0x7f4ee40d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:05.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.942+0000 7f4eeb7d0640 1 -- 172.21.15.57:0/3510371941 shutdown_connections 2023-10-19T13:00:05.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.942+0000 7f4eeb7d0640 1 -- 172.21.15.57:0/3510371941 wait complete. 2023-10-19T13:00:05.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.943+0000 7f4eeb7d0640 1 Processor -- start 2023-10-19T13:00:05.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.943+0000 7f4eeb7d0640 1 -- start start 2023-10-19T13:00:05.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.944+0000 7f4eeb7d0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4ee40ceb40 0x7f4ee40c7000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:05.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.944+0000 7f4eeb7d0640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4ee40cf340 0x7f4ee406a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:05.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.944+0000 7f4eeb7d0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4ee415b2a0 0x7f4ee406abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:05.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.944+0000 7f4eeb7d0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ee415e9e0 con 0x7f4ee415b2a0 2023-10-19T13:00:05.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.944+0000 7f4eeb7d0640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ee40c7540 con 0x7f4ee40ceb40 2023-10-19T13:00:05.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.944+0000 7f4eeb7d0640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ee415e700 con 0x7f4ee40cf340 2023-10-19T13:00:05.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.944+0000 7f4ee9d6d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4ee415b2a0 0x7f4ee406abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:05.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.944+0000 7f4ee9d6d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4ee415b2a0 0x7f4ee406abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44798/0 (socket says 172.21.15.57:44798) 2023-10-19T13:00:05.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.944+0000 7f4ee9d6d640 1 -- 172.21.15.57:0/2656207936 learned_addr learned my addr 172.21.15.57:0/2656207936 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:05.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.944+0000 7f4ee956c640 1 --2- 172.21.15.57:0/2656207936 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4ee40ceb40 0x7f4ee40c7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:05.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.944+0000 7f4ee8d6b640 1 --2- 172.21.15.57:0/2656207936 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4ee40cf340 0x7f4ee406a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:05.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.944+0000 7f4ee9d6d640 1 -- 172.21.15.57:0/2656207936 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4ee40cf340 msgr2=0x7f4ee406a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:05.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.944+0000 7f4ee9d6d640 1 --2- 172.21.15.57:0/2656207936 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4ee40cf340 0x7f4ee406a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:05.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.944+0000 7f4ee9d6d640 1 -- 172.21.15.57:0/2656207936 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4ee40ceb40 msgr2=0x7f4ee40c7000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:05.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.944+0000 7f4ee9d6d640 1 --2- 172.21.15.57:0/2656207936 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4ee40ceb40 0x7f4ee40c7000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:05.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.944+0000 7f4ee9d6d640 1 -- 172.21.15.57:0/2656207936 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4edc009840 con 0x7f4ee415b2a0 2023-10-19T13:00:05.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.945+0000 7f4ee9d6d640 1 --2- 172.21.15.57:0/2656207936 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4ee415b2a0 0x7f4ee406abb0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f4edc00efd0 tx=0x7f4edc00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:05.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.945+0000 7f4ed9ffb640 1 -- 172.21.15.57:0/2656207936 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4edc009b40 con 0x7f4ee415b2a0 2023-10-19T13:00:05.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.945+0000 7f4ed9ffb640 1 -- 172.21.15.57:0/2656207936 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f4edc003c60 con 0x7f4ee415b2a0 2023-10-19T13:00:05.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.945+0000 7f4eeb7d0640 1 -- 172.21.15.57:0/2656207936 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ee0009560 con 0x7f4ee415b2a0 2023-10-19T13:00:05.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.945+0000 7f4ed9ffb640 1 -- 172.21.15.57:0/2656207936 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4edc004020 con 0x7f4ee415b2a0 2023-10-19T13:00:05.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.945+0000 7f4eeb7d0640 1 -- 172.21.15.57:0/2656207936 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ee415bb50 con 0x7f4ee415b2a0 2023-10-19T13:00:05.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.946+0000 7f4eeb7d0640 1 -- 172.21.15.57:0/2656207936 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ee415b6d0 con 0x7f4ee415b2a0 2023-10-19T13:00:05.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.947+0000 7f4ed9ffb640 1 -- 172.21.15.57:0/2656207936 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 28) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f4edc0041a0 con 0x7f4ee415b2a0 2023-10-19T13:00:05.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.947+0000 7f4ed9ffb640 1 --2- 172.21.15.57:0/2656207936 >> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] conn(0x7f4ec806af60 0x7f4ec806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:05.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.947+0000 7f4ed9ffb640 1 -- 172.21.15.57:0/2656207936 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6503+0+0 (secure 0 0 0) 0x7f4edc014070 con 0x7f4ee415b2a0 2023-10-19T13:00:05.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.947+0000 7f4ee956c640 1 -- 172.21.15.57:0/2656207936 >> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] conn(0x7f4ec806af60 msgr2=0x7f4ec806d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/2207781045 2023-10-19T13:00:05.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.947+0000 7f4ee956c640 1 --2- 172.21.15.57:0/2656207936 >> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] conn(0x7f4ec806af60 0x7f4ec806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T13:00:05.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:05.951+0000 7f4ed9ffb640 1 -- 172.21.15.57:0/2656207936 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4ee415b6d0 con 0x7f4ee415b2a0 2023-10-19T13:00:06.001 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:05.998+0000 7fa9e0439040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-19T13:00:06.011 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:00:06.008+0000 7f282504c040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-19T13:00:06.055 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:00:06.053+0000 7f282504c040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-19T13:00:06.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.148+0000 7f4ee956c640 1 -- 172.21.15.57:0/2656207936 >> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] conn(0x7f4ec806af60 msgr2=0x7f4ec806d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/2207781045 2023-10-19T13:00:06.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.148+0000 7f4ee956c640 1 --2- 172.21.15.57:0/2656207936 >> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] conn(0x7f4ec806af60 0x7f4ec806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-19T13:00:06.166 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:00:06.163+0000 7f282504c040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-19T13:00:06.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.163+0000 7f4eeb7d0640 1 -- 172.21.15.57:0/2656207936 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4ee415b6d0 con 0x7f4ee415b2a0 2023-10-19T13:00:06.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.164+0000 7f4ed9ffb640 1 -- 172.21.15.57:0/2656207936 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+4913 (secure 0 0 0) 0x7f4ee415b6d0 con 0x7f4ee415b2a0 2023-10-19T13:00:06.167 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:00:06.167 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":23,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5594,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/3950728949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":3950728949},{"type":"v1","addr":"172.21.15.179:6837","nonce":3950728949}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":5597,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/3158611991","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":3158611991},{"type":"v1","addr":"172.21.15.179:6836","nonce":3158611991}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":6507,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/352867817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":352867817},{"type":"v1","addr":"172.21.15.57:6834","nonce":352867817}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T12:57:58.606618+0000","modified":"2023-10-19T12:58:00.613725+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6510},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6510":{"gid":6510,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.57:6835/243029526","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":243029526},{"type":"v1","addr":"172.21.15.57:6835","nonce":243029526}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-10-19T13:00:06.168 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 23 2023-10-19T13:00:06.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.166+0000 7f4eeb7d0640 1 -- 172.21.15.57:0/2656207936 >> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] conn(0x7f4ec806af60 msgr2=0x7f4ec806d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T13:00:06.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.166+0000 7f4eeb7d0640 1 --2- 172.21.15.57:0/2656207936 >> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] conn(0x7f4ec806af60 0x7f4ec806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:06.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.166+0000 7f4eeb7d0640 1 -- 172.21.15.57:0/2656207936 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4ee415b2a0 msgr2=0x7f4ee406abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:06.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.166+0000 7f4eeb7d0640 1 --2- 172.21.15.57:0/2656207936 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4ee415b2a0 0x7f4ee406abb0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f4edc00efd0 tx=0x7f4edc00c6a0 comp rx=0 tx=0).stop 2023-10-19T13:00:06.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.166+0000 7f4eeb7d0640 1 -- 172.21.15.57:0/2656207936 shutdown_connections 2023-10-19T13:00:06.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.166+0000 7f4eeb7d0640 1 --2- 172.21.15.57:0/2656207936 >> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] conn(0x7f4ec806af60 0x7f4ec806d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:06.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.166+0000 7f4eeb7d0640 1 --2- 172.21.15.57:0/2656207936 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4ee415b2a0 0x7f4ee406abb0 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:06.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.166+0000 7f4eeb7d0640 1 --2- 172.21.15.57:0/2656207936 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4ee40cf340 0x7f4ee406a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:06.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.166+0000 7f4eeb7d0640 1 --2- 172.21.15.57:0/2656207936 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4ee40ceb40 0x7f4ee40c7000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:06.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.166+0000 7f4eeb7d0640 1 -- 172.21.15.57:0/2656207936 >> 172.21.15.57:0/2656207936 conn(0x7f4ee40d0d60 msgr2=0x7f4ee415a050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:06.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.166+0000 7f4eeb7d0640 1 -- 172.21.15.57:0/2656207936 shutdown_connections 2023-10-19T13:00:06.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.166+0000 7f4eeb7d0640 1 -- 172.21.15.57:0/2656207936 wait complete. 2023-10-19T13:00:06.179 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T13:00:06.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.329+0000 7f7245456640 1 -- 172.21.15.57:0/3323677095 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f724015b260 msgr2=0x7f72400cf310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:06.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.329+0000 7f7245456640 1 --2- 172.21.15.57:0/3323677095 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f724015b260 0x7f72400cf310 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f7230005fd0 tx=0x7f723002c2e0 comp rx=0 tx=0).stop 2023-10-19T13:00:06.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.329+0000 7f7245456640 1 -- 172.21.15.57:0/3323677095 shutdown_connections 2023-10-19T13:00:06.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.329+0000 7f7245456640 1 --2- 172.21.15.57:0/3323677095 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f72400cf850 0x7f72400cc9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:06.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.329+0000 7f7245456640 1 --2- 172.21.15.57:0/3323677095 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f724015b260 0x7f72400cf310 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:06.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.329+0000 7f7245456640 1 -- 172.21.15.57:0/3323677095 >> 172.21.15.57:0/3323677095 conn(0x7f724009a030 msgr2=0x7f72400d0d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:06.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.329+0000 7f7245456640 1 -- 172.21.15.57:0/3323677095 shutdown_connections 2023-10-19T13:00:06.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.329+0000 7f7245456640 1 -- 172.21.15.57:0/3323677095 wait complete. 2023-10-19T13:00:06.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.331+0000 7f7245456640 1 Processor -- start 2023-10-19T13:00:06.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.331+0000 7f7245456640 1 -- start start 2023-10-19T13:00:06.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.332+0000 7f7245456640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f72400cf850 0x7f72400b63f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:06.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.332+0000 7f7245456640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f724015b260 0x7f72400b22d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:06.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.332+0000 7f7245456640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f724015b640 0x7f724015c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:06.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.332+0000 7f7245456640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72400c97a0 con 0x7f724015b260 2023-10-19T13:00:06.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.332+0000 7f7245456640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72400b2d00 con 0x7f72400cf850 2023-10-19T13:00:06.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.332+0000 7f7245456640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72400c9150 con 0x7f724015b640 2023-10-19T13:00:06.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.332+0000 7f723e7fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f724015b260 0x7f72400b22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:06.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.332+0000 7f723e7fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f724015b260 0x7f72400b22d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44812/0 (socket says 172.21.15.57:44812) 2023-10-19T13:00:06.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.332+0000 7f723e7fc640 1 -- 172.21.15.57:0/322212542 learned_addr learned my addr 172.21.15.57:0/322212542 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:06.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.332+0000 7f723effd640 1 --2- 172.21.15.57:0/322212542 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f72400cf850 0x7f72400b63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:06.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.332+0000 7f723f7fe640 1 --2- 172.21.15.57:0/322212542 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f724015b640 0x7f724015c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:06.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.332+0000 7f723e7fc640 1 -- 172.21.15.57:0/322212542 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f724015b640 msgr2=0x7f724015c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:06.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.332+0000 7f723e7fc640 1 --2- 172.21.15.57:0/322212542 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f724015b640 0x7f724015c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:06.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.332+0000 7f723e7fc640 1 -- 172.21.15.57:0/322212542 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f72400cf850 msgr2=0x7f72400b63f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:06.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.332+0000 7f723e7fc640 1 --2- 172.21.15.57:0/322212542 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f72400cf850 0x7f72400b63f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:06.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.332+0000 7f723e7fc640 1 -- 172.21.15.57:0/322212542 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7230009900 con 0x7f724015b260 2023-10-19T13:00:06.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.332+0000 7f723e7fc640 1 --2- 172.21.15.57:0/322212542 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f724015b260 0x7f72400b22d0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f723400c400 tx=0x7f723400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:06.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.332+0000 7f72277fe640 1 -- 172.21.15.57:0/322212542 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72340052d0 con 0x7f724015b260 2023-10-19T13:00:06.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.332+0000 7f72277fe640 1 -- 172.21.15.57:0/322212542 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f72340035f0 con 0x7f724015b260 2023-10-19T13:00:06.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.332+0000 7f7245456640 1 -- 172.21.15.57:0/322212542 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72400c9430 con 0x7f724015b260 2023-10-19T13:00:06.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.332+0000 7f72277fe640 1 -- 172.21.15.57:0/322212542 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7234003a00 con 0x7f724015b260 2023-10-19T13:00:06.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.333+0000 7f7245456640 1 -- 172.21.15.57:0/322212542 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72400b3940 con 0x7f724015b260 2023-10-19T13:00:06.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.333+0000 7f7245456640 1 -- 172.21.15.57:0/322212542 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72400c9910 con 0x7f724015b260 2023-10-19T13:00:06.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.334+0000 7f72277fe640 1 -- 172.21.15.57:0/322212542 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 28) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f7234003b60 con 0x7f724015b260 2023-10-19T13:00:06.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.334+0000 7f72277fe640 1 --2- 172.21.15.57:0/322212542 >> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] conn(0x7f722c06af60 0x7f722c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:06.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.334+0000 7f72277fe640 1 -- 172.21.15.57:0/322212542 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6503+0+0 (secure 0 0 0) 0x7f7234089160 con 0x7f724015b260 2023-10-19T13:00:06.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.334+0000 7f723effd640 1 -- 172.21.15.57:0/322212542 >> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] conn(0x7f722c06af60 msgr2=0x7f722c06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/2207781045 2023-10-19T13:00:06.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.334+0000 7f723effd640 1 --2- 172.21.15.57:0/322212542 >> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] conn(0x7f722c06af60 0x7f722c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T13:00:06.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.339+0000 7f72277fe640 1 -- 172.21.15.57:0/322212542 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7234053f20 con 0x7f724015b260 2023-10-19T13:00:06.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.518+0000 7f7245456640 1 -- 172.21.15.57:0/322212542 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f72400675f0 con 0x7f724015b260 2023-10-19T13:00:06.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.519+0000 7f72277fe640 1 -- 172.21.15.57:0/322212542 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v75) v1 ==== 74+0+17518 (secure 0 0 0) 0x7f7234057540 con 0x7f724015b260 2023-10-19T13:00:06.522 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:00:06.522 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":75,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T13:00:05.646260+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2023-10-19T12:57:56.901606+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2023-10-19T12:57:57.581428+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":75,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.2a","mappings":[{"from":0,"to":6}]},{"pgid":"8.38","mappings":[{"from":0,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.179:0/70135229":"2023-10-20T12:58:14.538805+0000","172.21.15.179:6832/3449041082":"2023-10-20T12:58:14.538805+0000","172.21.15.179:0/1553007782":"2023-10-20T12:58:14.538805+0000","172.21.15.179:0/2777182879":"2023-10-20T12:58:14.538805+0000","172.21.15.179:6833/3449041082":"2023-10-20T12:58:14.538805+0000","172.21.15.179:0/64734340":"2023-10-20T12:58:14.538805+0000","172.21.15.179:0/1129452163":"2023-10-20T12:58:14.538805+0000","172.21.15.179:0/3893958736":"2023-10-20T12:58:14.538805+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[{"pool":8,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T13:00:06.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.521+0000 7f7245456640 1 -- 172.21.15.57:0/322212542 >> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] conn(0x7f722c06af60 msgr2=0x7f722c06d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T13:00:06.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.521+0000 7f7245456640 1 --2- 172.21.15.57:0/322212542 >> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] conn(0x7f722c06af60 0x7f722c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:06.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.521+0000 7f7245456640 1 -- 172.21.15.57:0/322212542 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f724015b260 msgr2=0x7f72400b22d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:06.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.521+0000 7f7245456640 1 --2- 172.21.15.57:0/322212542 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f724015b260 0x7f72400b22d0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f723400c400 tx=0x7f723400c8c0 comp rx=0 tx=0).stop 2023-10-19T13:00:06.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.521+0000 7f7245456640 1 -- 172.21.15.57:0/322212542 shutdown_connections 2023-10-19T13:00:06.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.521+0000 7f7245456640 1 --2- 172.21.15.57:0/322212542 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f724015b640 0x7f724015c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:06.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.521+0000 7f7245456640 1 --2- 172.21.15.57:0/322212542 >> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] conn(0x7f722c06af60 0x7f722c06d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:06.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.521+0000 7f7245456640 1 --2- 172.21.15.57:0/322212542 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f724015b260 0x7f72400b22d0 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:06.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.521+0000 7f7245456640 1 --2- 172.21.15.57:0/322212542 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f72400cf850 0x7f72400b63f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:06.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.521+0000 7f7245456640 1 -- 172.21.15.57:0/322212542 >> 172.21.15.57:0/322212542 conn(0x7f724009a030 msgr2=0x7f7240069f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:06.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.522+0000 7f7245456640 1 -- 172.21.15.57:0/322212542 shutdown_connections 2023-10-19T13:00:06.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.522+0000 7f7245456640 1 -- 172.21.15.57:0/322212542 wait complete. 2023-10-19T13:00:06.535 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-19T12:44:07.371186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-19T12:57:56.901606+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2023-10-19T12:57:57.581428+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 5, 'snap_epoch': 75, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-19T13:00:06.536 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-19T13:00:06.607 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:06.605+0000 7fa9e0439040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-19T13:00:06.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.648+0000 7fac23844640 1 -- 172.21.15.57:0/3962960393 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac1c15b2a0 msgr2=0x7fac1c0d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:06.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.648+0000 7fac23844640 1 --2- 172.21.15.57:0/3962960393 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac1c15b2a0 0x7fac1c0d1910 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fac10009830 tx=0x7fac1002bd60 comp rx=0 tx=0).stop 2023-10-19T13:00:06.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.648+0000 7fac23844640 1 -- 172.21.15.57:0/3962960393 shutdown_connections 2023-10-19T13:00:06.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.648+0000 7fac23844640 1 --2- 172.21.15.57:0/3962960393 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac1c15b2a0 0x7fac1c0d1910 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:06.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.648+0000 7fac23844640 1 -- 172.21.15.57:0/3962960393 >> 172.21.15.57:0/3962960393 conn(0x7fac1c0d0d60 msgr2=0x7fac1c0d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:06.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.648+0000 7fac23844640 1 -- 172.21.15.57:0/3962960393 shutdown_connections 2023-10-19T13:00:06.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.648+0000 7fac23844640 1 -- 172.21.15.57:0/3962960393 wait complete. 2023-10-19T13:00:06.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.649+0000 7fac23844640 1 Processor -- start 2023-10-19T13:00:06.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.650+0000 7fac23844640 1 -- start start 2023-10-19T13:00:06.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.650+0000 7fac23844640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fac1c0ceb40 0x7fac1c0c87c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:06.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.650+0000 7fac23844640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac1c0cf340 0x7fac1c1573f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:06.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.650+0000 7fac23844640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fac1c15b2a0 0x7fac1c15c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:06.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.650+0000 7fac23844640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac1c15e670 con 0x7fac1c0cf340 2023-10-19T13:00:06.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.650+0000 7fac23844640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac1c15de60 con 0x7fac1c15b2a0 2023-10-19T13:00:06.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.650+0000 7fac23844640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac1c15e0b0 con 0x7fac1c0ceb40 2023-10-19T13:00:06.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.650+0000 7fac20ddf640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac1c0cf340 0x7fac1c1573f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:06.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.650+0000 7fac20ddf640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac1c0cf340 0x7fac1c1573f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39734/0 (socket says 172.21.15.57:39734) 2023-10-19T13:00:06.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.650+0000 7fac20ddf640 1 -- 172.21.15.57:0/713189062 learned_addr learned my addr 172.21.15.57:0/713189062 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:06.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.650+0000 7fac21de1640 1 --2- 172.21.15.57:0/713189062 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fac1c15b2a0 0x7fac1c15c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:06.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.650+0000 7fac215e0640 1 --2- 172.21.15.57:0/713189062 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fac1c0ceb40 0x7fac1c0c87c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:06.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.651+0000 7fac20ddf640 1 -- 172.21.15.57:0/713189062 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fac1c0ceb40 msgr2=0x7fac1c0c87c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:06.658 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.651+0000 7fac20ddf640 1 --2- 172.21.15.57:0/713189062 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fac1c0ceb40 0x7fac1c0c87c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:06.658 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.651+0000 7fac20ddf640 1 -- 172.21.15.57:0/713189062 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fac1c15b2a0 msgr2=0x7fac1c15c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:06.658 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.651+0000 7fac20ddf640 1 --2- 172.21.15.57:0/713189062 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fac1c15b2a0 0x7fac1c15c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:06.658 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.651+0000 7fac20ddf640 1 -- 172.21.15.57:0/713189062 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac10009490 con 0x7fac1c0cf340 2023-10-19T13:00:06.658 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.651+0000 7fac20ddf640 1 --2- 172.21.15.57:0/713189062 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac1c0cf340 0x7fac1c1573f0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fac10009950 tx=0x7fac10003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:06.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.651+0000 7fac0dffb640 1 -- 172.21.15.57:0/713189062 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac10003e20 con 0x7fac1c0cf340 2023-10-19T13:00:06.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.651+0000 7fac0dffb640 1 -- 172.21.15.57:0/713189062 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fac10005110 con 0x7fac1c0cf340 2023-10-19T13:00:06.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.651+0000 7fac23844640 1 -- 172.21.15.57:0/713189062 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac1c15e390 con 0x7fac1c0cf340 2023-10-19T13:00:06.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.651+0000 7fac0dffb640 1 -- 172.21.15.57:0/713189062 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac10005520 con 0x7fac1c0cf340 2023-10-19T13:00:06.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.651+0000 7fac23844640 1 -- 172.21.15.57:0/713189062 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac1c0c8d00 con 0x7fac1c0cf340 2023-10-19T13:00:06.660 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.652+0000 7fac23844640 1 -- 172.21.15.57:0/713189062 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac1c15b6b0 con 0x7fac1c0cf340 2023-10-19T13:00:06.660 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.654+0000 7fac0dffb640 1 -- 172.21.15.57:0/713189062 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 28) v1 ==== 89343+0+0 (secure 0 0 0) 0x7fac100047a0 con 0x7fac1c0cf340 2023-10-19T13:00:06.661 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.654+0000 7fac0dffb640 1 --2- 172.21.15.57:0/713189062 >> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] conn(0x7fac0006af60 0x7fac0006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:06.661 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.654+0000 7fac0dffb640 1 -- 172.21.15.57:0/713189062 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6503+0+0 (secure 0 0 0) 0x7fac100ac5c0 con 0x7fac1c0cf340 2023-10-19T13:00:06.661 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.654+0000 7fac215e0640 1 -- 172.21.15.57:0/713189062 >> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] conn(0x7fac0006af60 msgr2=0x7fac0006d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/2207781045 2023-10-19T13:00:06.661 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.654+0000 7fac215e0640 1 --2- 172.21.15.57:0/713189062 >> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] conn(0x7fac0006af60 0x7fac0006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T13:00:06.662 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.657+0000 7fac0dffb640 1 -- 172.21.15.57:0/713189062 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fac100774f0 con 0x7fac1c0cf340 2023-10-19T13:00:06.834 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:00:06.832+0000 7f282504c040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-19T13:00:06.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.835+0000 7fac23844640 1 -- 172.21.15.57:0/713189062 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fac1c15b6b0 con 0x7fac1c0cf340 2023-10-19T13:00:06.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.836+0000 7fac0dffb640 1 -- 172.21.15.57:0/713189062 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v75) v1 ==== 93+0+10 (secure 0 0 0) 0x7fac10039020 con 0x7fac1c0cf340 2023-10-19T13:00:06.839 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-19T13:00:06.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.838+0000 7fac23844640 1 -- 172.21.15.57:0/713189062 >> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] conn(0x7fac0006af60 msgr2=0x7fac0006d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T13:00:06.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.838+0000 7fac23844640 1 --2- 172.21.15.57:0/713189062 >> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] conn(0x7fac0006af60 0x7fac0006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:06.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.838+0000 7fac23844640 1 -- 172.21.15.57:0/713189062 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac1c0cf340 msgr2=0x7fac1c1573f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:06.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.838+0000 7fac23844640 1 --2- 172.21.15.57:0/713189062 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac1c0cf340 0x7fac1c1573f0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fac10009950 tx=0x7fac10003820 comp rx=0 tx=0).stop 2023-10-19T13:00:06.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.838+0000 7fac23844640 1 -- 172.21.15.57:0/713189062 shutdown_connections 2023-10-19T13:00:06.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.838+0000 7fac23844640 1 --2- 172.21.15.57:0/713189062 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fac1c15b2a0 0x7fac1c15c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:06.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.838+0000 7fac23844640 1 --2- 172.21.15.57:0/713189062 >> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] conn(0x7fac0006af60 0x7fac0006d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:06.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.838+0000 7fac23844640 1 --2- 172.21.15.57:0/713189062 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac1c0cf340 0x7fac1c1573f0 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:06.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.838+0000 7fac23844640 1 --2- 172.21.15.57:0/713189062 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fac1c0ceb40 0x7fac1c0c87c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:06.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.838+0000 7fac23844640 1 -- 172.21.15.57:0/713189062 >> 172.21.15.57:0/713189062 conn(0x7fac1c0d0d60 msgr2=0x7fac1c0ad8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:06.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.838+0000 7fac23844640 1 -- 172.21.15.57:0/713189062 shutdown_connections 2023-10-19T13:00:06.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.838+0000 7fac23844640 1 -- 172.21.15.57:0/713189062 wait complete. 2023-10-19T13:00:06.852 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-19T13:00:06.867 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:06.865+0000 7fa9e0439040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-19T13:00:06.916 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:06.913+0000 7fa9e0439040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-19T13:00:06.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.962+0000 7fe5ab1d7640 1 -- 172.21.15.57:0/3764701699 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe594004d00 msgr2=0x7fe594005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:06.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.962+0000 7fe5ab1d7640 1 --2- 172.21.15.57:0/3764701699 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe594004d00 0x7fe594005180 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7fe59c009f00 tx=0x7fe5940057f0 comp rx=0 tx=0).stop 2023-10-19T13:00:06.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.962+0000 7fe5a9774640 1 -- 172.21.15.57:0/3764701699 reap_dead start 2023-10-19T13:00:06.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.962+0000 7fe5ab1d7640 1 -- 172.21.15.57:0/3764701699 shutdown_connections 2023-10-19T13:00:06.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.962+0000 7fe5ab1d7640 1 -- 172.21.15.57:0/3764701699 >> 172.21.15.57:0/3764701699 conn(0x7fe5a40d0d60 msgr2=0x7fe5a40d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:06.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.962+0000 7fe5ab1d7640 1 -- 172.21.15.57:0/3764701699 shutdown_connections 2023-10-19T13:00:06.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.962+0000 7fe5ab1d7640 1 -- 172.21.15.57:0/3764701699 wait complete. 2023-10-19T13:00:06.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.963+0000 7fe5ab1d7640 1 Processor -- start 2023-10-19T13:00:06.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.963+0000 7fe5ab1d7640 1 -- start start 2023-10-19T13:00:06.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.964+0000 7fe5ab1d7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe5a4157e90 0x7fe5a415c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:06.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.964+0000 7fe5ab1d7640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe5a415de70 0x7fe5a40aea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:06.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.964+0000 7fe5ab1d7640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe5a4158270 0x7fe5a40aef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:06.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.964+0000 7fe5ab1d7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a415e630 con 0x7fe5a4157e90 2023-10-19T13:00:06.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.964+0000 7fe5ab1d7640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a415e350 con 0x7fe5a415de70 2023-10-19T13:00:06.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.964+0000 7fe5ab1d7640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a40c7f40 con 0x7fe5a4158270 2023-10-19T13:00:06.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.964+0000 7fe5a8f73640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe5a4157e90 0x7fe5a415c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:06.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.964+0000 7fe5a8f73640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe5a4157e90 0x7fe5a415c090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39746/0 (socket says 172.21.15.57:39746) 2023-10-19T13:00:06.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.964+0000 7fe5a8f73640 1 -- 172.21.15.57:0/2044075324 learned_addr learned my addr 172.21.15.57:0/2044075324 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:06.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.964+0000 7fe5a9774640 1 --2- 172.21.15.57:0/2044075324 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe5a4158270 0x7fe5a40aef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:06.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.964+0000 7fe5a3fff640 1 --2- 172.21.15.57:0/2044075324 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe5a415de70 0x7fe5a40aea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:06.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.965+0000 7fe5a8f73640 1 -- 172.21.15.57:0/2044075324 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe5a4158270 msgr2=0x7fe5a40aef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:06.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.965+0000 7fe5a8f73640 1 --2- 172.21.15.57:0/2044075324 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe5a4158270 0x7fe5a40aef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:06.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.965+0000 7fe5a8f73640 1 -- 172.21.15.57:0/2044075324 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe5a415de70 msgr2=0x7fe5a40aea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:06.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.965+0000 7fe5a8f73640 1 --2- 172.21.15.57:0/2044075324 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe5a415de70 0x7fe5a40aea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:06.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.965+0000 7fe5a8f73640 1 -- 172.21.15.57:0/2044075324 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5a40c8220 con 0x7fe5a4157e90 2023-10-19T13:00:06.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.965+0000 7fe5a8f73640 1 --2- 172.21.15.57:0/2044075324 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe5a4157e90 0x7fe5a415c090 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fe598001aa0 tx=0x7fe5980039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:06.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.965+0000 7fe5a17fa640 1 -- 172.21.15.57:0/2044075324 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe598011900 con 0x7fe5a4157e90 2023-10-19T13:00:06.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.965+0000 7fe5a17fa640 1 -- 172.21.15.57:0/2044075324 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fe598011a80 con 0x7fe5a4157e90 2023-10-19T13:00:06.973 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.965+0000 7fe5ab1d7640 1 -- 172.21.15.57:0/2044075324 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5a40c8500 con 0x7fe5a4157e90 2023-10-19T13:00:06.973 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.965+0000 7fe5a17fa640 1 -- 172.21.15.57:0/2044075324 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe598011e90 con 0x7fe5a4157e90 2023-10-19T13:00:06.973 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.965+0000 7fe5ab1d7640 1 -- 172.21.15.57:0/2044075324 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5a41f4660 con 0x7fe5a4157e90 2023-10-19T13:00:06.973 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.966+0000 7fe5ab1d7640 1 -- 172.21.15.57:0/2044075324 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5a40c8740 con 0x7fe5a4157e90 2023-10-19T13:00:06.974 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.967+0000 7fe5a17fa640 1 -- 172.21.15.57:0/2044075324 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 28) v1 ==== 89343+0+0 (secure 0 0 0) 0x7fe598010420 con 0x7fe5a4157e90 2023-10-19T13:00:06.974 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.967+0000 7fe5a17fa640 1 --2- 172.21.15.57:0/2044075324 >> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] conn(0x7fe58806af60 0x7fe58806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:06.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.967+0000 7fe5a17fa640 1 -- 172.21.15.57:0/2044075324 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6503+0+0 (secure 0 0 0) 0x7fe59808a8b0 con 0x7fe5a4157e90 2023-10-19T13:00:06.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.967+0000 7fe5a3fff640 1 -- 172.21.15.57:0/2044075324 >> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] conn(0x7fe58806af60 msgr2=0x7fe58806d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/2207781045 2023-10-19T13:00:06.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.967+0000 7fe5a3fff640 1 --2- 172.21.15.57:0/2044075324 >> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] conn(0x7fe58806af60 0x7fe58806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T13:00:06.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:06.971+0000 7fe5a17fa640 1 -- 172.21.15.57:0/2044075324 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe598055560 con 0x7fe5a4157e90 2023-10-19T13:00:07.017 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:07.014+0000 7fa9e0439040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-19T13:00:07.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.045+0000 7fe5a17fa640 1 -- 172.21.15.57:0/2044075324 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mgrmap(e 29) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fe598088ab0 con 0x7fe5a4157e90 2023-10-19T13:00:07.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.045+0000 7fe5a17fa640 1 -- 172.21.15.57:0/2044075324 >> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] conn(0x7fe58806af60 msgr2=0x7fe58806d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T13:00:07.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.045+0000 7fe5a17fa640 1 --2- 172.21.15.57:0/2044075324 >> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] conn(0x7fe58806af60 0x7fe58806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:07.096 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:07.093+0000 7fa960ee3640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:00:07.097 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:07.093+0000 7fa960ee3640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:00:07.097 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:07.093+0000 7fa960ee3640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:00:07.097 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:07.093+0000 7fa960ee3640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:00:07.097 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:07.093+0000 7fa960ee3640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:00:07.098 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:07.094+0000 7fa964eeb640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:00:07.098 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:07.094+0000 7fa964eeb640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:00:07.098 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:07.094+0000 7fa964eeb640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:00:07.098 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:07.094+0000 7fa964eeb640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:00:07.098 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:07.094+0000 7fa964eeb640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:00:07.121 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:00:07.119+0000 7f282504c040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-19T13:00:07.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.153+0000 7fe5ab1d7640 1 -- 172.21.15.57:0/2044075324 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe5a40690e0 con 0x7fe5a4157e90 2023-10-19T13:00:07.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.154+0000 7fe5a17fa640 1 -- 172.21.15.57:0/2044075324 <== mon.0 v2:172.21.15.57:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v76) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe598071de0 con 0x7fe5a4157e90 2023-10-19T13:00:07.157 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2023-10-19T13:00:07.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.156+0000 7fe5ab1d7640 1 -- 172.21.15.57:0/2044075324 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe5a4157e90 msgr2=0x7fe5a415c090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:07.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.156+0000 7fe5ab1d7640 1 --2- 172.21.15.57:0/2044075324 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe5a4157e90 0x7fe5a415c090 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fe598001aa0 tx=0x7fe5980039c0 comp rx=0 tx=0).stop 2023-10-19T13:00:07.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.156+0000 7fe5ab1d7640 1 -- 172.21.15.57:0/2044075324 shutdown_connections 2023-10-19T13:00:07.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.156+0000 7fe5ab1d7640 1 --2- 172.21.15.57:0/2044075324 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe5a4158270 0x7fe5a40aef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:07.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.156+0000 7fe5ab1d7640 1 --2- 172.21.15.57:0/2044075324 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe5a415de70 0x7fe5a40aea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:07.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.156+0000 7fe5ab1d7640 1 --2- 172.21.15.57:0/2044075324 >> [v2:172.21.15.179:6832/2207781045,v1:172.21.15.179:6833/2207781045] conn(0x7fe58806af60 0x7fe58806d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:07.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.156+0000 7fe5ab1d7640 1 --2- 172.21.15.57:0/2044075324 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe5a4157e90 0x7fe5a415c090 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:07.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.156+0000 7fe5ab1d7640 1 -- 172.21.15.57:0/2044075324 >> 172.21.15.57:0/2044075324 conn(0x7fe5a40d0d60 msgr2=0x7fe5a40b0e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:07.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.156+0000 7fe5ab1d7640 1 -- 172.21.15.57:0/2044075324 shutdown_connections 2023-10-19T13:00:07.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.156+0000 7fe5ab1d7640 1 -- 172.21.15.57:0/2044075324 wait complete. 2023-10-19T13:00:07.170 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-19T13:00:07.174 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:00:07.171+0000 7f282504c040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-19T13:00:07.283 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:00:07.281+0000 7f282504c040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-19T13:00:07.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.281+0000 7f7eeb6d7640 1 -- 172.21.15.57:0/1495771397 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ee40ceb40 msgr2=0x7f7ee40b0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:07.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.281+0000 7f7eeb6d7640 1 --2- 172.21.15.57:0/1495771397 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ee40ceb40 0x7f7ee40b0650 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f7edc005fd0 tx=0x7f7edc02c050 comp rx=0 tx=0).stop 2023-10-19T13:00:07.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.281+0000 7f7eeb6d7640 1 -- 172.21.15.57:0/1495771397 shutdown_connections 2023-10-19T13:00:07.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.281+0000 7f7eeb6d7640 1 --2- 172.21.15.57:0/1495771397 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ee40ceb40 0x7f7ee40b0650 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:07.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.281+0000 7f7eeb6d7640 1 --2- 172.21.15.57:0/1495771397 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7ee40cf340 0x7f7ee40d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:07.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.281+0000 7f7eeb6d7640 1 -- 172.21.15.57:0/1495771397 >> 172.21.15.57:0/1495771397 conn(0x7f7ee40d0d60 msgr2=0x7f7ee40d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:07.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.281+0000 7f7eeb6d7640 1 -- 172.21.15.57:0/1495771397 shutdown_connections 2023-10-19T13:00:07.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.281+0000 7f7eeb6d7640 1 -- 172.21.15.57:0/1495771397 wait complete. 2023-10-19T13:00:07.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.282+0000 7f7eeb6d7640 1 Processor -- start 2023-10-19T13:00:07.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.283+0000 7f7eeb6d7640 1 -- start start 2023-10-19T13:00:07.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.283+0000 7f7eeb6d7640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7ee40ceb40 0x7f7ee4156be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:07.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.283+0000 7f7eeb6d7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ee40cf340 0x7f7ee409e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:07.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.283+0000 7f7eeb6d7640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7ee415b2a0 0x7f7ee40c9e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:07.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.283+0000 7f7eeb6d7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee4157120 con 0x7f7ee40cf340 2023-10-19T13:00:07.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.283+0000 7f7eeb6d7640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee41f13c0 con 0x7f7ee415b2a0 2023-10-19T13:00:07.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.283+0000 7f7eeb6d7640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee40b64f0 con 0x7f7ee40ceb40 2023-10-19T13:00:07.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.283+0000 7f7ee8c72640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ee40cf340 0x7f7ee409e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:07.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.283+0000 7f7ee9473640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7ee40ceb40 0x7f7ee4156be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:07.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.283+0000 7f7ee8c72640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ee40cf340 0x7f7ee409e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39766/0 (socket says 172.21.15.57:39766) 2023-10-19T13:00:07.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.283+0000 7f7ee8c72640 1 -- 172.21.15.57:0/256391110 learned_addr learned my addr 172.21.15.57:0/256391110 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:07.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.284+0000 7f7ee9c74640 1 --2- 172.21.15.57:0/256391110 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7ee415b2a0 0x7f7ee40c9e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:07.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.284+0000 7f7ee8c72640 1 -- 172.21.15.57:0/256391110 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7ee40ceb40 msgr2=0x7f7ee4156be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:07.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.284+0000 7f7ee8c72640 1 --2- 172.21.15.57:0/256391110 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7ee40ceb40 0x7f7ee4156be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:07.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.284+0000 7f7ee8c72640 1 -- 172.21.15.57:0/256391110 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7ee415b2a0 msgr2=0x7f7ee40c9e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:07.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.284+0000 7f7ee8c72640 1 --2- 172.21.15.57:0/256391110 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7ee415b2a0 0x7f7ee40c9e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:07.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.284+0000 7f7ee8c72640 1 -- 172.21.15.57:0/256391110 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7edc009840 con 0x7f7ee40cf340 2023-10-19T13:00:07.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.284+0000 7f7ee9473640 1 --2- 172.21.15.57:0/256391110 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7ee40ceb40 0x7f7ee4156be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T13:00:07.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.284+0000 7f7ee8c72640 1 --2- 172.21.15.57:0/256391110 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ee40cf340 0x7f7ee409e420 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f7ed400c430 tx=0x7f7ed400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:07.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.284+0000 7f7ed9ffb640 1 -- 172.21.15.57:0/256391110 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ed4011790 con 0x7f7ee40cf340 2023-10-19T13:00:07.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.284+0000 7f7ed9ffb640 1 -- 172.21.15.57:0/256391110 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f7ed4011a00 con 0x7f7ee40cf340 2023-10-19T13:00:07.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.284+0000 7f7eeb6d7640 1 -- 172.21.15.57:0/256391110 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ee40b67d0 con 0x7f7ee40cf340 2023-10-19T13:00:07.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.284+0000 7f7ed9ffb640 1 -- 172.21.15.57:0/256391110 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ed4011e50 con 0x7f7ee40cf340 2023-10-19T13:00:07.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.284+0000 7f7eeb6d7640 1 -- 172.21.15.57:0/256391110 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ee415c930 con 0x7f7ee40cf340 2023-10-19T13:00:07.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.285+0000 7f7eeb6d7640 1 -- 172.21.15.57:0/256391110 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ee40690e0 con 0x7f7ee40cf340 2023-10-19T13:00:07.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.286+0000 7f7ed9ffb640 1 -- 172.21.15.57:0/256391110 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 29) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f7ed401c460 con 0x7f7ee40cf340 2023-10-19T13:00:07.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.286+0000 7f7ed9ffb640 1 -- 172.21.15.57:0/256391110 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6803+0+0 (secure 0 0 0) 0x7f7ed40894c0 con 0x7f7ee40cf340 2023-10-19T13:00:07.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.291+0000 7f7ed9ffb640 1 -- 172.21.15.57:0/256391110 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7ed4054620 con 0x7f7ee40cf340 2023-10-19T13:00:07.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.465+0000 7f7eeb6d7640 1 -- 172.21.15.57:0/256391110 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7ee40cb9b0 con 0x7f7ee40cf340 2023-10-19T13:00:07.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.466+0000 7f7ed9ffb640 1 -- 172.21.15.57:0/256391110 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v76) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7ed4057c40 con 0x7f7ee40cf340 2023-10-19T13:00:07.469 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 64 2023-10-19T13:00:07.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.468+0000 7f7eeb6d7640 1 -- 172.21.15.57:0/256391110 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ee40cf340 msgr2=0x7f7ee409e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:07.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.468+0000 7f7eeb6d7640 1 --2- 172.21.15.57:0/256391110 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ee40cf340 0x7f7ee409e420 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f7ed400c430 tx=0x7f7ed400c8f0 comp rx=0 tx=0).stop 2023-10-19T13:00:07.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.468+0000 7f7eeb6d7640 1 -- 172.21.15.57:0/256391110 shutdown_connections 2023-10-19T13:00:07.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.468+0000 7f7eeb6d7640 1 --2- 172.21.15.57:0/256391110 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7ee415b2a0 0x7f7ee40c9e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:07.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.468+0000 7f7eeb6d7640 1 --2- 172.21.15.57:0/256391110 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ee40cf340 0x7f7ee409e420 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:07.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.468+0000 7f7eeb6d7640 1 --2- 172.21.15.57:0/256391110 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7ee40ceb40 0x7f7ee4156be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:07.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.468+0000 7f7eeb6d7640 1 -- 172.21.15.57:0/256391110 >> 172.21.15.57:0/256391110 conn(0x7f7ee40d0d60 msgr2=0x7f7ee4157b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:07.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.468+0000 7f7eeb6d7640 1 -- 172.21.15.57:0/256391110 shutdown_connections 2023-10-19T13:00:07.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.468+0000 7f7eeb6d7640 1 -- 172.21.15.57:0/256391110 wait complete. 2023-10-19T13:00:07.481 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T13:00:07.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.628+0000 7fa58dbf5640 1 -- 172.21.15.57:0/4172965325 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa5880c9dc0 msgr2=0x7fa58806b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:07.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.628+0000 7fa58dbf5640 1 --2- 172.21.15.57:0/4172965325 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa5880c9dc0 0x7fa58806b2e0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fa57c005fd0 tx=0x7fa57c02be10 comp rx=0 tx=0).stop 2023-10-19T13:00:07.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.628+0000 7fa58dbf5640 1 -- 172.21.15.57:0/4172965325 shutdown_connections 2023-10-19T13:00:07.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.628+0000 7fa58dbf5640 1 --2- 172.21.15.57:0/4172965325 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa5880ab500 0x7fa5880ab990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:07.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.628+0000 7fa58dbf5640 1 --2- 172.21.15.57:0/4172965325 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa5880c9dc0 0x7fa58806b2e0 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:07.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.628+0000 7fa58dbf5640 1 -- 172.21.15.57:0/4172965325 >> 172.21.15.57:0/4172965325 conn(0x7fa588069930 msgr2=0x7fa588069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:07.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.628+0000 7fa58dbf5640 1 -- 172.21.15.57:0/4172965325 shutdown_connections 2023-10-19T13:00:07.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.628+0000 7fa58dbf5640 1 -- 172.21.15.57:0/4172965325 wait complete. 2023-10-19T13:00:07.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.629+0000 7fa58dbf5640 1 Processor -- start 2023-10-19T13:00:07.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.629+0000 7fa58dbf5640 1 -- start start 2023-10-19T13:00:07.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.631+0000 7fa58dbf5640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa58806af00 0x7fa58813a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:07.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.631+0000 7fa58dbf5640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa5880ab500 0x7fa5880a6ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:07.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.631+0000 7fa58dbf5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa5880c9dc0 0x7fa5880ae700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:07.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.631+0000 7fa58dbf5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa58813aa20 con 0x7fa5880c9dc0 2023-10-19T13:00:07.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.631+0000 7fa58dbf5640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5880a7400 con 0x7fa5880ab500 2023-10-19T13:00:07.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.631+0000 7fa58dbf5640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5880a9860 con 0x7fa58806af00 2023-10-19T13:00:07.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.631+0000 7fa587fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa5880c9dc0 0x7fa5880ae700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:07.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.631+0000 7fa587fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa5880c9dc0 0x7fa5880ae700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39782/0 (socket says 172.21.15.57:39782) 2023-10-19T13:00:07.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.631+0000 7fa587fff640 1 -- 172.21.15.57:0/3899060204 learned_addr learned my addr 172.21.15.57:0/3899060204 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:07.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.631+0000 7fa586ffd640 1 --2- 172.21.15.57:0/3899060204 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa5880ab500 0x7fa5880a6ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:07.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.631+0000 7fa5877fe640 1 --2- 172.21.15.57:0/3899060204 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa58806af00 0x7fa58813a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:07.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.631+0000 7fa587fff640 1 -- 172.21.15.57:0/3899060204 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa58806af00 msgr2=0x7fa58813a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:07.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.631+0000 7fa587fff640 1 --2- 172.21.15.57:0/3899060204 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa58806af00 0x7fa58813a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:07.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.631+0000 7fa587fff640 1 -- 172.21.15.57:0/3899060204 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa5880ab500 msgr2=0x7fa5880a6ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:07.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.631+0000 7fa587fff640 1 --2- 172.21.15.57:0/3899060204 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa5880ab500 0x7fa5880a6ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:07.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.631+0000 7fa587fff640 1 -- 172.21.15.57:0/3899060204 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa57c009560 con 0x7fa5880c9dc0 2023-10-19T13:00:07.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.631+0000 7fa586ffd640 1 --2- 172.21.15.57:0/3899060204 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa5880ab500 0x7fa5880a6ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:00:07.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.631+0000 7fa5877fe640 1 --2- 172.21.15.57:0/3899060204 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa58806af00 0x7fa58813a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:00:07.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.631+0000 7fa587fff640 1 --2- 172.21.15.57:0/3899060204 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa5880c9dc0 0x7fa5880ae700 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fa58000eaa0 tx=0x7fa58000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:07.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.631+0000 7fa56ffff640 1 -- 172.21.15.57:0/3899060204 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa58000cdb0 con 0x7fa5880c9dc0 2023-10-19T13:00:07.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.631+0000 7fa56ffff640 1 -- 172.21.15.57:0/3899060204 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fa580004b20 con 0x7fa5880c9dc0 2023-10-19T13:00:07.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.631+0000 7fa58dbf5640 1 -- 172.21.15.57:0/3899060204 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5880a9b40 con 0x7fa5880c9dc0 2023-10-19T13:00:07.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.631+0000 7fa56ffff640 1 -- 172.21.15.57:0/3899060204 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa580005020 con 0x7fa5880c9dc0 2023-10-19T13:00:07.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.631+0000 7fa58dbf5640 1 -- 172.21.15.57:0/3899060204 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5880aa100 con 0x7fa5880c9dc0 2023-10-19T13:00:07.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.632+0000 7fa58dbf5640 1 -- 172.21.15.57:0/3899060204 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5880690e0 con 0x7fa5880c9dc0 2023-10-19T13:00:07.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.632+0000 7fa56ffff640 1 -- 172.21.15.57:0/3899060204 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 29) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fa580005180 con 0x7fa5880c9dc0 2023-10-19T13:00:07.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.633+0000 7fa56ffff640 1 -- 172.21.15.57:0/3899060204 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6803+0+0 (secure 0 0 0) 0x7fa580089f50 con 0x7fa5880c9dc0 2023-10-19T13:00:07.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.637+0000 7fa56ffff640 1 -- 172.21.15.57:0/3899060204 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa5800bcca0 con 0x7fa5880c9dc0 2023-10-19T13:00:07.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.849+0000 7fa58dbf5640 1 -- 172.21.15.57:0/3899060204 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa5880cc420 con 0x7fa5880c9dc0 2023-10-19T13:00:07.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.849+0000 7fa56ffff640 1 -- 172.21.15.57:0/3899060204 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+4913 (secure 0 0 0) 0x7fa58001f020 con 0x7fa5880c9dc0 2023-10-19T13:00:07.852 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:00:07.852 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":23,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5594,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/3950728949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":3950728949},{"type":"v1","addr":"172.21.15.179:6837","nonce":3950728949}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":5597,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/3158611991","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":3158611991},{"type":"v1","addr":"172.21.15.179:6836","nonce":3158611991}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":6507,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/352867817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":352867817},{"type":"v1","addr":"172.21.15.57:6834","nonce":352867817}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T12:57:58.606618+0000","modified":"2023-10-19T12:58:00.613725+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6510},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6510":{"gid":6510,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.57:6835/243029526","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":243029526},{"type":"v1","addr":"172.21.15.57:6835","nonce":243029526}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-10-19T13:00:07.853 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 23 2023-10-19T13:00:07.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.852+0000 7fa58dbf5640 1 -- 172.21.15.57:0/3899060204 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa5880c9dc0 msgr2=0x7fa5880ae700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:07.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.852+0000 7fa58dbf5640 1 --2- 172.21.15.57:0/3899060204 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa5880c9dc0 0x7fa5880ae700 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fa58000eaa0 tx=0x7fa58000ef60 comp rx=0 tx=0).stop 2023-10-19T13:00:07.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.852+0000 7fa58dbf5640 1 -- 172.21.15.57:0/3899060204 shutdown_connections 2023-10-19T13:00:07.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.852+0000 7fa58dbf5640 1 --2- 172.21.15.57:0/3899060204 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa5880c9dc0 0x7fa5880ae700 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:07.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.852+0000 7fa58dbf5640 1 --2- 172.21.15.57:0/3899060204 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa5880ab500 0x7fa5880a6ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:07.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.852+0000 7fa58dbf5640 1 --2- 172.21.15.57:0/3899060204 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa58806af00 0x7fa58813a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:07.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.852+0000 7fa58dbf5640 1 -- 172.21.15.57:0/3899060204 >> 172.21.15.57:0/3899060204 conn(0x7fa588069930 msgr2=0x7fa58815ce70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:07.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.852+0000 7fa58dbf5640 1 -- 172.21.15.57:0/3899060204 shutdown_connections 2023-10-19T13:00:07.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.852+0000 7fa58dbf5640 1 -- 172.21.15.57:0/3899060204 wait complete. 2023-10-19T13:00:07.865 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T13:00:07.974 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.971+0000 7fbdd3e64640 1 -- 172.21.15.57:0/1789295331 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbdcc0cf340 msgr2=0x7fbdcc0d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:07.974 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.971+0000 7fbdd3e64640 1 --2- 172.21.15.57:0/1789295331 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbdcc0cf340 0x7fbdcc0d1910 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fbdc8009860 tx=0x7fbdc802bd60 comp rx=0 tx=0).stop 2023-10-19T13:00:07.974 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.971+0000 7fbdd3e64640 1 -- 172.21.15.57:0/1789295331 shutdown_connections 2023-10-19T13:00:07.974 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.971+0000 7fbdd3e64640 1 --2- 172.21.15.57:0/1789295331 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbdcc0cf340 0x7fbdcc0d1910 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:07.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.971+0000 7fbdd3e64640 1 -- 172.21.15.57:0/1789295331 >> 172.21.15.57:0/1789295331 conn(0x7fbdcc0d0d60 msgr2=0x7fbdcc0d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:07.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.971+0000 7fbdd3e64640 1 -- 172.21.15.57:0/1789295331 shutdown_connections 2023-10-19T13:00:07.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.971+0000 7fbdd3e64640 1 -- 172.21.15.57:0/1789295331 wait complete. 2023-10-19T13:00:07.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.972+0000 7fbdd3e64640 1 Processor -- start 2023-10-19T13:00:07.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.972+0000 7fbdd3e64640 1 -- start start 2023-10-19T13:00:07.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.974+0000 7fbdd3e64640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbdcc0ceb40 0x7fbdcc1582c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:07.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.974+0000 7fbdd3e64640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbdcc0cf340 0x7fbdcc1e5e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:07.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.974+0000 7fbdd3e64640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbdcc15b2a0 0x7fbdcc0b2af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:07.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.974+0000 7fbdd3e64640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdcc1f2530 con 0x7fbdcc15b2a0 2023-10-19T13:00:07.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.974+0000 7fbdd3e64640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdcc158800 con 0x7fbdcc0ceb40 2023-10-19T13:00:07.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.974+0000 7fbdd3e64640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdcc1f18f0 con 0x7fbdcc0cf340 2023-10-19T13:00:07.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.974+0000 7fbdd2401640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbdcc15b2a0 0x7fbdcc0b2af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:07.979 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.974+0000 7fbdd2401640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbdcc15b2a0 0x7fbdcc0b2af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39784/0 (socket says 172.21.15.57:39784) 2023-10-19T13:00:07.979 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.974+0000 7fbdd2401640 1 -- 172.21.15.57:0/3325790097 learned_addr learned my addr 172.21.15.57:0/3325790097 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:07.979 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.974+0000 7fbdd1c00640 1 --2- 172.21.15.57:0/3325790097 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbdcc0ceb40 0x7fbdcc1582c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:07.979 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.974+0000 7fbdd13ff640 1 --2- 172.21.15.57:0/3325790097 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbdcc0cf340 0x7fbdcc1e5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:07.979 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.974+0000 7fbdd2401640 1 -- 172.21.15.57:0/3325790097 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbdcc0cf340 msgr2=0x7fbdcc1e5e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:07.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.974+0000 7fbdd2401640 1 --2- 172.21.15.57:0/3325790097 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbdcc0cf340 0x7fbdcc1e5e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:07.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.974+0000 7fbdd2401640 1 -- 172.21.15.57:0/3325790097 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbdcc0ceb40 msgr2=0x7fbdcc1582c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:07.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.974+0000 7fbdd2401640 1 --2- 172.21.15.57:0/3325790097 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbdcc0ceb40 0x7fbdcc1582c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:07.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.974+0000 7fbdd2401640 1 -- 172.21.15.57:0/3325790097 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdc8009490 con 0x7fbdcc15b2a0 2023-10-19T13:00:07.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.974+0000 7fbdd13ff640 1 --2- 172.21.15.57:0/3325790097 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbdcc0cf340 0x7fbdcc1e5e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:00:07.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.974+0000 7fbdd2401640 1 --2- 172.21.15.57:0/3325790097 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbdcc15b2a0 0x7fbdcc0b2af0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7fbdc400cbc0 tx=0x7fbdc400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:07.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.974+0000 7fbdc27fc640 1 -- 172.21.15.57:0/3325790097 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdc400ad40 con 0x7fbdcc15b2a0 2023-10-19T13:00:07.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.974+0000 7fbdc27fc640 1 -- 172.21.15.57:0/3325790097 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fbdc4012420 con 0x7fbdcc15b2a0 2023-10-19T13:00:07.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.974+0000 7fbdd3e64640 1 -- 172.21.15.57:0/3325790097 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdcc1f1b90 con 0x7fbdcc15b2a0 2023-10-19T13:00:07.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.974+0000 7fbdc27fc640 1 -- 172.21.15.57:0/3325790097 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdc4012850 con 0x7fbdcc15b2a0 2023-10-19T13:00:07.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.974+0000 7fbdd3e64640 1 -- 172.21.15.57:0/3325790097 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdcc1f2150 con 0x7fbdcc15b2a0 2023-10-19T13:00:07.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.975+0000 7fbdd3e64640 1 -- 172.21.15.57:0/3325790097 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdcc1f2670 con 0x7fbdcc15b2a0 2023-10-19T13:00:07.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.975+0000 7fbdc27fc640 1 -- 172.21.15.57:0/3325790097 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 29) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fbdc401a3b0 con 0x7fbdcc15b2a0 2023-10-19T13:00:07.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.976+0000 7fbdc27fc640 1 -- 172.21.15.57:0/3325790097 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6803+0+0 (secure 0 0 0) 0x7fbdc4089d00 con 0x7fbdcc15b2a0 2023-10-19T13:00:07.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:07.980+0000 7fbdc27fc640 1 -- 172.21.15.57:0/3325790097 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fbdcc1f2670 con 0x7fbdcc15b2a0 2023-10-19T13:00:08.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.040+0000 7fbdc27fc640 1 -- 172.21.15.57:0/3325790097 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mgrmap(e 30) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fbdc40547e0 con 0x7fbdcc15b2a0 2023-10-19T13:00:08.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.040+0000 7fbdc27fc640 1 --2- 172.21.15.57:0/3325790097 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7fbdb0070c00 0x7fbdb0072ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:08.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.041+0000 7fbdd1c00640 1 --2- 172.21.15.57:0/3325790097 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7fbdb0070c00 0x7fbdb0072ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:08.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.044+0000 7fbdd1c00640 1 --2- 172.21.15.57:0/3325790097 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7fbdb0070c00 0x7fbdb0072ff0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fbdc8036000 tx=0x7fbdc8004730 comp rx=0 tx=0).ready entity=mgr.6256 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:08.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.159+0000 7fbdd3e64640 1 -- 172.21.15.57:0/3325790097 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbdcc1f2670 con 0x7fbdcc15b2a0 2023-10-19T13:00:08.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.161+0000 7fbdc27fc640 1 -- 172.21.15.57:0/3325790097 <== mon.0 v2:172.21.15.57:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v76) v1 ==== 74+0+17962 (secure 0 0 0) 0x7fbdcc1f2670 con 0x7fbdcc15b2a0 2023-10-19T13:00:08.163 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:00:08.163 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":76,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T13:00:07.028692+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2023-10-19T12:57:56.901606+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2023-10-19T12:57:57.581428+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":75,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.2a","mappings":[{"from":0,"to":6}]},{"pgid":"8.38","mappings":[{"from":0,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.179:6833/2207781045":"2023-10-20T13:00:07.028653+0000","172.21.15.179:0/3759354760":"2023-10-20T13:00:07.028653+0000","172.21.15.179:0/1939092102":"2023-10-20T13:00:07.028653+0000","172.21.15.179:0/1372200045":"2023-10-20T13:00:07.028653+0000","172.21.15.179:0/1174466246":"2023-10-20T13:00:07.028653+0000","172.21.15.179:0/1234281500":"2023-10-20T13:00:07.028653+0000","172.21.15.179:6832/2207781045":"2023-10-20T13:00:07.028653+0000","172.21.15.179:0/2482595099":"2023-10-20T13:00:07.028653+0000","172.21.15.179:0/70135229":"2023-10-20T12:58:14.538805+0000","172.21.15.179:6832/3449041082":"2023-10-20T12:58:14.538805+0000","172.21.15.179:0/1553007782":"2023-10-20T12:58:14.538805+0000","172.21.15.179:0/2777182879":"2023-10-20T12:58:14.538805+0000","172.21.15.179:6833/3449041082":"2023-10-20T12:58:14.538805+0000","172.21.15.179:0/64734340":"2023-10-20T12:58:14.538805+0000","172.21.15.179:0/1129452163":"2023-10-20T12:58:14.538805+0000","172.21.15.179:0/3893958736":"2023-10-20T12:58:14.538805+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T13:00:08.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.163+0000 7fbdd3e64640 1 -- 172.21.15.57:0/3325790097 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7fbdb0070c00 msgr2=0x7fbdb0072ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:08.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.163+0000 7fbdd3e64640 1 --2- 172.21.15.57:0/3325790097 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7fbdb0070c00 0x7fbdb0072ff0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fbdc8036000 tx=0x7fbdc8004730 comp rx=0 tx=0).stop 2023-10-19T13:00:08.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.163+0000 7fbdd3e64640 1 -- 172.21.15.57:0/3325790097 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbdcc15b2a0 msgr2=0x7fbdcc0b2af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:08.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.163+0000 7fbdd3e64640 1 --2- 172.21.15.57:0/3325790097 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbdcc15b2a0 0x7fbdcc0b2af0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7fbdc400cbc0 tx=0x7fbdc400a430 comp rx=0 tx=0).stop 2023-10-19T13:00:08.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.163+0000 7fbdd3e64640 1 -- 172.21.15.57:0/3325790097 shutdown_connections 2023-10-19T13:00:08.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.163+0000 7fbdd3e64640 1 --2- 172.21.15.57:0/3325790097 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7fbdb0070c00 0x7fbdb0072ff0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:08.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.163+0000 7fbdd3e64640 1 --2- 172.21.15.57:0/3325790097 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbdcc15b2a0 0x7fbdcc0b2af0 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:08.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.163+0000 7fbdd3e64640 1 --2- 172.21.15.57:0/3325790097 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbdcc0cf340 0x7fbdcc1e5e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:08.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.163+0000 7fbdd3e64640 1 --2- 172.21.15.57:0/3325790097 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbdcc0ceb40 0x7fbdcc1582c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:08.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.163+0000 7fbdd3e64640 1 -- 172.21.15.57:0/3325790097 >> 172.21.15.57:0/3325790097 conn(0x7fbdcc0d0d60 msgr2=0x7fbdcc06b2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:08.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.163+0000 7fbdd3e64640 1 -- 172.21.15.57:0/3325790097 shutdown_connections 2023-10-19T13:00:08.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.163+0000 7fbdd3e64640 1 -- 172.21.15.57:0/3325790097 wait complete. 2023-10-19T13:00:08.175 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-19T13:00:08.176 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T13:00:08.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.282+0000 7ff7cc61a640 1 -- 172.21.15.57:0/3030131009 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff7c4156790 msgr2=0x7ff7c40cc990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:08.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.282+0000 7ff7cc61a640 1 --2- 172.21.15.57:0/3030131009 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff7c4156790 0x7ff7c40cc990 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7ff7c4065ed0 tx=0x7ff7c002be20 comp rx=0 tx=0).stop 2023-10-19T13:00:08.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.283+0000 7ff7cc61a640 1 -- 172.21.15.57:0/3030131009 shutdown_connections 2023-10-19T13:00:08.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.283+0000 7ff7cc61a640 1 --2- 172.21.15.57:0/3030131009 >> v2:172.21.15.179:3300/0 conn(0x7ff7c406a730 0x7ff7c406abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:08.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.283+0000 7ff7cc61a640 1 --2- 172.21.15.57:0/3030131009 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff7c4156790 0x7ff7c40cc990 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:08.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.283+0000 7ff7cc61a640 1 -- 172.21.15.57:0/3030131009 >> 172.21.15.57:0/3030131009 conn(0x7ff7c406fae0 msgr2=0x7ff7c406fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:08.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.283+0000 7ff7cc61a640 1 -- 172.21.15.57:0/3030131009 shutdown_connections 2023-10-19T13:00:08.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.283+0000 7ff7cc61a640 1 -- 172.21.15.57:0/3030131009 wait complete. 2023-10-19T13:00:08.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.284+0000 7ff7cc61a640 1 Processor -- start 2023-10-19T13:00:08.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.284+0000 7ff7cc61a640 1 -- start start 2023-10-19T13:00:08.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.284+0000 7ff7cc61a640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff7c406a730 0x7ff7c40b1100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:08.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.284+0000 7ff7cc61a640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff7c40ac870 0x7ff7c4154ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:08.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.285+0000 7ff7cc61a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff7c4156790 0x7ff7c40b18d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:08.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.285+0000 7ff7cc61a640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7c40b0c70 con 0x7ff7c4156790 2023-10-19T13:00:08.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.285+0000 7ff7cc61a640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7c40b1640 con 0x7ff7c406a730 2023-10-19T13:00:08.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.285+0000 7ff7cc61a640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7c40b2270 con 0x7ff7c40ac870 2023-10-19T13:00:08.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.285+0000 7ff7cabb7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff7c4156790 0x7ff7c40b18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:08.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.285+0000 7ff7cabb7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff7c4156790 0x7ff7c40b18d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39802/0 (socket says 172.21.15.57:39802) 2023-10-19T13:00:08.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.285+0000 7ff7cabb7640 1 -- 172.21.15.57:0/2751661567 learned_addr learned my addr 172.21.15.57:0/2751661567 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:08.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.285+0000 7ff7c9bb5640 1 --2- 172.21.15.57:0/2751661567 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff7c40ac870 0x7ff7c4154ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:08.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.285+0000 7ff7ca3b6640 1 --2- 172.21.15.57:0/2751661567 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff7c406a730 0x7ff7c40b1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:08.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.285+0000 7ff7cabb7640 1 -- 172.21.15.57:0/2751661567 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff7c40ac870 msgr2=0x7ff7c4154ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:08.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.285+0000 7ff7cabb7640 1 --2- 172.21.15.57:0/2751661567 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff7c40ac870 0x7ff7c4154ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:08.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.285+0000 7ff7cabb7640 1 -- 172.21.15.57:0/2751661567 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff7c406a730 msgr2=0x7ff7c40b1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:08.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.285+0000 7ff7cabb7640 1 --2- 172.21.15.57:0/2751661567 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff7c406a730 0x7ff7c40b1100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:08.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.285+0000 7ff7cabb7640 1 -- 172.21.15.57:0/2751661567 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7c0009560 con 0x7ff7c4156790 2023-10-19T13:00:08.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.285+0000 7ff7cabb7640 1 --2- 172.21.15.57:0/2751661567 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff7c4156790 0x7ff7c40b18d0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7ff7bc00e960 tx=0x7ff7bc00ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:08.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.285+0000 7ff7baffd640 1 -- 172.21.15.57:0/2751661567 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7bc004d20 con 0x7ff7c4156790 2023-10-19T13:00:08.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.285+0000 7ff7baffd640 1 -- 172.21.15.57:0/2751661567 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7ff7bc00cb20 con 0x7ff7c4156790 2023-10-19T13:00:08.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.285+0000 7ff7baffd640 1 -- 172.21.15.57:0/2751661567 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7bc005020 con 0x7ff7c4156790 2023-10-19T13:00:08.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.285+0000 7ff7cc61a640 1 -- 172.21.15.57:0/2751661567 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7c40b2550 con 0x7ff7c4156790 2023-10-19T13:00:08.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.286+0000 7ff7cc61a640 1 -- 172.21.15.57:0/2751661567 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7c40b0900 con 0x7ff7c4156790 2023-10-19T13:00:08.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.286+0000 7ff7cc61a640 1 -- 172.21.15.57:0/2751661567 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7c40b0e20 con 0x7ff7c4156790 2023-10-19T13:00:08.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.287+0000 7ff7baffd640 1 -- 172.21.15.57:0/2751661567 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 30) v1 ==== 89067+0+0 (secure 0 0 0) 0x7ff7bc005180 con 0x7ff7c4156790 2023-10-19T13:00:08.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.287+0000 7ff7baffd640 1 --2- 172.21.15.57:0/2751661567 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7ff7a806ac90 0x7ff7a806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:08.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.288+0000 7ff7baffd640 1 -- 172.21.15.57:0/2751661567 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6803+0+0 (secure 0 0 0) 0x7ff7bc089f20 con 0x7ff7c4156790 2023-10-19T13:00:08.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.288+0000 7ff7ca3b6640 1 --2- 172.21.15.57:0/2751661567 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7ff7a806ac90 0x7ff7a806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:08.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.288+0000 7ff7ca3b6640 1 --2- 172.21.15.57:0/2751661567 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7ff7a806ac90 0x7ff7a806d150 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff7c0035000 tx=0x7ff7c0004730 comp rx=0 tx=0).ready entity=mgr.6256 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:08.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.292+0000 7ff7baffd640 1 -- 172.21.15.57:0/2751661567 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff7c40b0e20 con 0x7ff7c4156790 2023-10-19T13:00:08.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.504+0000 7ff7cc61a640 1 -- 172.21.15.57:0/2751661567 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff7c40690e0 con 0x7ff7c4156790 2023-10-19T13:00:08.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.504+0000 7ff7baffd640 1 -- 172.21.15.57:0/2751661567 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+4913 (secure 0 0 0) 0x7ff7bc01f070 con 0x7ff7c4156790 2023-10-19T13:00:08.507 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:00:08.507 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":23,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5594,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/3950728949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":3950728949},{"type":"v1","addr":"172.21.15.179:6837","nonce":3950728949}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":5597,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/3158611991","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":3158611991},{"type":"v1","addr":"172.21.15.179:6836","nonce":3158611991}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":6507,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/352867817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":352867817},{"type":"v1","addr":"172.21.15.57:6834","nonce":352867817}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T12:57:58.606618+0000","modified":"2023-10-19T12:58:00.613725+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6510},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6510":{"gid":6510,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.57:6835/243029526","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":243029526},{"type":"v1","addr":"172.21.15.57:6835","nonce":243029526}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-10-19T13:00:08.508 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 23 2023-10-19T13:00:08.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.506+0000 7ff7cc61a640 1 -- 172.21.15.57:0/2751661567 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7ff7a806ac90 msgr2=0x7ff7a806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:08.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.506+0000 7ff7cc61a640 1 --2- 172.21.15.57:0/2751661567 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7ff7a806ac90 0x7ff7a806d150 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff7c0035000 tx=0x7ff7c0004730 comp rx=0 tx=0).stop 2023-10-19T13:00:08.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.506+0000 7ff7cc61a640 1 -- 172.21.15.57:0/2751661567 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff7c4156790 msgr2=0x7ff7c40b18d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:08.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.506+0000 7ff7cc61a640 1 --2- 172.21.15.57:0/2751661567 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff7c4156790 0x7ff7c40b18d0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7ff7bc00e960 tx=0x7ff7bc00ee20 comp rx=0 tx=0).stop 2023-10-19T13:00:08.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.506+0000 7ff7cc61a640 1 -- 172.21.15.57:0/2751661567 shutdown_connections 2023-10-19T13:00:08.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.506+0000 7ff7cc61a640 1 --2- 172.21.15.57:0/2751661567 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7ff7a806ac90 0x7ff7a806d150 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:08.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.506+0000 7ff7cc61a640 1 --2- 172.21.15.57:0/2751661567 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff7c4156790 0x7ff7c40b18d0 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:08.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.506+0000 7ff7cc61a640 1 --2- 172.21.15.57:0/2751661567 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff7c40ac870 0x7ff7c4154ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:08.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.506+0000 7ff7cc61a640 1 --2- 172.21.15.57:0/2751661567 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff7c406a730 0x7ff7c40b1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:08.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.506+0000 7ff7cc61a640 1 -- 172.21.15.57:0/2751661567 >> 172.21.15.57:0/2751661567 conn(0x7ff7c406fae0 msgr2=0x7ff7c40c9670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:08.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.506+0000 7ff7cc61a640 1 -- 172.21.15.57:0/2751661567 shutdown_connections 2023-10-19T13:00:08.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.506+0000 7ff7cc61a640 1 -- 172.21.15.57:0/2751661567 wait complete. 2023-10-19T13:00:08.518 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T13:00:08.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.626+0000 7f249b141640 1 -- 172.21.15.57:0/1020642087 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f24840047c0 msgr2=0x7f2484004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:08.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.626+0000 7f249b141640 1 --2- 172.21.15.57:0/1020642087 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f24840047c0 0x7f2484004c40 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f2480006fa0 tx=0x7f24840057f0 comp rx=0 tx=0).stop 2023-10-19T13:00:08.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.626+0000 7f249b141640 1 -- 172.21.15.57:0/1020642087 shutdown_connections 2023-10-19T13:00:08.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.626+0000 7f249b141640 1 --2- 172.21.15.57:0/1020642087 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f24840047c0 0x7f2484004c40 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:08.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.626+0000 7f249b141640 1 -- 172.21.15.57:0/1020642087 >> 172.21.15.57:0/1020642087 conn(0x7f24940d0d60 msgr2=0x7f24940d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:08.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.626+0000 7f249b141640 1 -- 172.21.15.57:0/1020642087 shutdown_connections 2023-10-19T13:00:08.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.626+0000 7f249b141640 1 -- 172.21.15.57:0/1020642087 wait complete. 2023-10-19T13:00:08.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.627+0000 7f249b141640 1 Processor -- start 2023-10-19T13:00:08.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.627+0000 7f249b141640 1 -- start start 2023-10-19T13:00:08.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.628+0000 7f249b141640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f24840047c0 0x7f24941f4a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:08.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.628+0000 7f249b141640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f249415d920 0x7f249415b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:08.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.628+0000 7f249b141640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f24940b2850 0x7f24940b2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:08.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.628+0000 7f249b141640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2494152cf0 con 0x7f249415d920 2023-10-19T13:00:08.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.628+0000 7f249b141640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24940b3210 con 0x7f24940b2850 2023-10-19T13:00:08.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.628+0000 7f249b141640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24940b4ad0 con 0x7f24840047c0 2023-10-19T13:00:08.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.628+0000 7f2493fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f249415d920 0x7f249415b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:08.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.628+0000 7f2493fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f249415d920 0x7f249415b680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39818/0 (socket says 172.21.15.57:39818) 2023-10-19T13:00:08.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.628+0000 7f2493fff640 1 -- 172.21.15.57:0/891144781 learned_addr learned my addr 172.21.15.57:0/891144781 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:08.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.628+0000 7f2498edd640 1 --2- 172.21.15.57:0/891144781 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f24840047c0 0x7f24941f4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:08.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.628+0000 7f24996de640 1 --2- 172.21.15.57:0/891144781 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f24940b2850 0x7f24940b2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:08.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.628+0000 7f2493fff640 1 -- 172.21.15.57:0/891144781 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f24840047c0 msgr2=0x7f24941f4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:08.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.628+0000 7f2493fff640 1 --2- 172.21.15.57:0/891144781 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f24840047c0 0x7f24941f4a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:08.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.628+0000 7f2493fff640 1 -- 172.21.15.57:0/891144781 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f24940b2850 msgr2=0x7f24940b2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:08.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.628+0000 7f2493fff640 1 --2- 172.21.15.57:0/891144781 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f24940b2850 0x7f24940b2cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:08.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.628+0000 7f2493fff640 1 -- 172.21.15.57:0/891144781 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24941f5930 con 0x7f249415d920 2023-10-19T13:00:08.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.628+0000 7f2498edd640 1 --2- 172.21.15.57:0/891144781 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f24840047c0 0x7f24941f4a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T13:00:08.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.628+0000 7f2493fff640 1 --2- 172.21.15.57:0/891144781 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f249415d920 0x7f249415b680 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f248002c8a0 tx=0x7f2480001720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:08.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.629+0000 7f24917fa640 1 -- 172.21.15.57:0/891144781 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f248003c040 con 0x7f249415d920 2023-10-19T13:00:08.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.629+0000 7f24917fa640 1 -- 172.21.15.57:0/891144781 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f24800037f0 con 0x7f249415d920 2023-10-19T13:00:08.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.629+0000 7f249b141640 1 -- 172.21.15.57:0/891144781 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24941f5090 con 0x7f249415d920 2023-10-19T13:00:08.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.629+0000 7f24917fa640 1 -- 172.21.15.57:0/891144781 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2480003bc0 con 0x7f249415d920 2023-10-19T13:00:08.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.629+0000 7f249b141640 1 -- 172.21.15.57:0/891144781 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24941f5650 con 0x7f249415d920 2023-10-19T13:00:08.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.630+0000 7f249b141640 1 -- 172.21.15.57:0/891144781 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24940690e0 con 0x7f249415d920 2023-10-19T13:00:08.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.630+0000 7f24917fa640 1 -- 172.21.15.57:0/891144781 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 30) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f2480032070 con 0x7f249415d920 2023-10-19T13:00:08.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.630+0000 7f24917fa640 1 --2- 172.21.15.57:0/891144781 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f247806ac90 0x7f247806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:08.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.630+0000 7f24917fa640 1 -- 172.21.15.57:0/891144781 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6803+0+0 (secure 0 0 0) 0x7f24800aad10 con 0x7f249415d920 2023-10-19T13:00:08.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.631+0000 7f2498edd640 1 --2- 172.21.15.57:0/891144781 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f247806ac90 0x7f247806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:08.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.631+0000 7f2498edd640 1 --2- 172.21.15.57:0/891144781 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f247806ac90 0x7f247806d150 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f24940682a0 tx=0x7f2488005ef0 comp rx=0 tx=0).ready entity=mgr.6256 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:08.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.634+0000 7f24917fa640 1 -- 172.21.15.57:0/891144781 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f24800b0020 con 0x7f249415d920 2023-10-19T13:00:08.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.850+0000 7f249b141640 1 -- 172.21.15.57:0/891144781 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f24940d0b40 con 0x7f249415d920 2023-10-19T13:00:08.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.850+0000 7f24917fa640 1 -- 172.21.15.57:0/891144781 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+4913 (secure 0 0 0) 0x7f2480046020 con 0x7f249415d920 2023-10-19T13:00:08.852 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:00:08.853 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":23,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5594,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/3950728949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":3950728949},{"type":"v1","addr":"172.21.15.179:6837","nonce":3950728949}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":5597,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/3158611991","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":3158611991},{"type":"v1","addr":"172.21.15.179:6836","nonce":3158611991}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":6507,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/352867817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":352867817},{"type":"v1","addr":"172.21.15.57:6834","nonce":352867817}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T12:57:58.606618+0000","modified":"2023-10-19T12:58:00.613725+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6510},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6510":{"gid":6510,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.57:6835/243029526","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":243029526},{"type":"v1","addr":"172.21.15.57:6835","nonce":243029526}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-10-19T13:00:08.853 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 23 2023-10-19T13:00:08.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.852+0000 7f249b141640 1 -- 172.21.15.57:0/891144781 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f247806ac90 msgr2=0x7f247806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:08.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.852+0000 7f249b141640 1 --2- 172.21.15.57:0/891144781 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f247806ac90 0x7f247806d150 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f24940682a0 tx=0x7f2488005ef0 comp rx=0 tx=0).stop 2023-10-19T13:00:08.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.852+0000 7f249b141640 1 -- 172.21.15.57:0/891144781 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f249415d920 msgr2=0x7f249415b680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:08.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.852+0000 7f249b141640 1 --2- 172.21.15.57:0/891144781 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f249415d920 0x7f249415b680 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f248002c8a0 tx=0x7f2480001720 comp rx=0 tx=0).stop 2023-10-19T13:00:08.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.852+0000 7f249b141640 1 -- 172.21.15.57:0/891144781 shutdown_connections 2023-10-19T13:00:08.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.852+0000 7f249b141640 1 --2- 172.21.15.57:0/891144781 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f247806ac90 0x7f247806d150 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:08.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.852+0000 7f249b141640 1 --2- 172.21.15.57:0/891144781 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f24940b2850 0x7f24940b2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:08.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.852+0000 7f249b141640 1 --2- 172.21.15.57:0/891144781 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f249415d920 0x7f249415b680 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:08.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.852+0000 7f249b141640 1 --2- 172.21.15.57:0/891144781 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f24840047c0 0x7f24941f4a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:08.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.852+0000 7f249b141640 1 -- 172.21.15.57:0/891144781 >> 172.21.15.57:0/891144781 conn(0x7f24940d0d60 msgr2=0x7f24940af480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:08.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.852+0000 7f249b141640 1 -- 172.21.15.57:0/891144781 shutdown_connections 2023-10-19T13:00:08.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.852+0000 7f249b141640 1 -- 172.21.15.57:0/891144781 wait complete. 2023-10-19T13:00:08.864 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T13:00:08.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.972+0000 7f3e24290640 1 -- 172.21.15.57:0/44326831 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3e1c0afa20 msgr2=0x7f3e1c158300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:08.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.972+0000 7f3e24290640 1 --2- 172.21.15.57:0/44326831 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3e1c0afa20 0x7f3e1c158300 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f3e10009830 tx=0x7f3e1002bd60 comp rx=0 tx=0).stop 2023-10-19T13:00:08.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.972+0000 7f3e24290640 1 -- 172.21.15.57:0/44326831 shutdown_connections 2023-10-19T13:00:08.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.972+0000 7f3e24290640 1 --2- 172.21.15.57:0/44326831 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3e1c0b0530 0x7f3e1c0b09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:08.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.972+0000 7f3e24290640 1 --2- 172.21.15.57:0/44326831 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3e1c0afa20 0x7f3e1c158300 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:08.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.972+0000 7f3e24290640 1 -- 172.21.15.57:0/44326831 >> 172.21.15.57:0/44326831 conn(0x7f3e1c069930 msgr2=0x7f3e1c069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:08.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.973+0000 7f3e24290640 1 -- 172.21.15.57:0/44326831 shutdown_connections 2023-10-19T13:00:08.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.973+0000 7f3e24290640 1 -- 172.21.15.57:0/44326831 wait complete. 2023-10-19T13:00:08.979 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.974+0000 7f3e24290640 1 Processor -- start 2023-10-19T13:00:08.979 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.974+0000 7f3e24290640 1 -- start start 2023-10-19T13:00:08.979 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.974+0000 7f3e24290640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3e1c0afa20 0x7f3e1c142440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:08.979 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.974+0000 7f3e24290640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3e1c0b0530 0x7f3e1c1474b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:08.979 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.974+0000 7f3e24290640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3e1c0cbd20 0x7f3e1c143210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:08.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.974+0000 7f3e24290640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e1c148930 con 0x7f3e1c0cbd20 2023-10-19T13:00:08.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.974+0000 7f3e24290640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e1c14e390 con 0x7f3e1c0afa20 2023-10-19T13:00:08.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.974+0000 7f3e24290640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e1c147db0 con 0x7f3e1c0b0530 2023-10-19T13:00:08.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.974+0000 7f3e2282d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3e1c0cbd20 0x7f3e1c143210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:08.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.974+0000 7f3e2282d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3e1c0cbd20 0x7f3e1c143210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39832/0 (socket says 172.21.15.57:39832) 2023-10-19T13:00:08.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.974+0000 7f3e2282d640 1 -- 172.21.15.57:0/1329704458 learned_addr learned my addr 172.21.15.57:0/1329704458 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:08.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.975+0000 7f3e2182b640 1 --2- 172.21.15.57:0/1329704458 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3e1c0b0530 0x7f3e1c1474b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:08.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.975+0000 7f3e2202c640 1 --2- 172.21.15.57:0/1329704458 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3e1c0afa20 0x7f3e1c142440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:08.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.975+0000 7f3e2282d640 1 -- 172.21.15.57:0/1329704458 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3e1c0b0530 msgr2=0x7f3e1c1474b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:08.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.975+0000 7f3e2282d640 1 --2- 172.21.15.57:0/1329704458 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3e1c0b0530 0x7f3e1c1474b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:08.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.975+0000 7f3e2282d640 1 -- 172.21.15.57:0/1329704458 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3e1c0afa20 msgr2=0x7f3e1c142440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:08.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.975+0000 7f3e2282d640 1 --2- 172.21.15.57:0/1329704458 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3e1c0afa20 0x7f3e1c142440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:08.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.975+0000 7f3e2282d640 1 -- 172.21.15.57:0/1329704458 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e14009490 con 0x7f3e1c0cbd20 2023-10-19T13:00:08.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.975+0000 7f3e2202c640 1 --2- 172.21.15.57:0/1329704458 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3e1c0afa20 0x7f3e1c142440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T13:00:08.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.975+0000 7f3e2282d640 1 --2- 172.21.15.57:0/1329704458 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3e1c0cbd20 0x7f3e1c143210 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f3e1400e9b0 tx=0x7f3e1400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:08.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.975+0000 7f3e1affd640 1 -- 172.21.15.57:0/1329704458 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e1400cd80 con 0x7f3e1c0cbd20 2023-10-19T13:00:08.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.975+0000 7f3e1affd640 1 -- 172.21.15.57:0/1329704458 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f3e14004020 con 0x7f3e1c0cbd20 2023-10-19T13:00:08.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.975+0000 7f3e24290640 1 -- 172.21.15.57:0/1329704458 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e10009490 con 0x7f3e1c0cbd20 2023-10-19T13:00:08.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.975+0000 7f3e1affd640 1 -- 172.21.15.57:0/1329704458 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e1400cee0 con 0x7f3e1c0cbd20 2023-10-19T13:00:08.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.975+0000 7f3e24290640 1 -- 172.21.15.57:0/1329704458 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e1c148370 con 0x7f3e1c0cbd20 2023-10-19T13:00:08.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.976+0000 7f3e24290640 1 -- 172.21.15.57:0/1329704458 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e1c14e550 con 0x7f3e1c0cbd20 2023-10-19T13:00:08.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.977+0000 7f3e1affd640 1 -- 172.21.15.57:0/1329704458 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 30) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f3e140035f0 con 0x7f3e1c0cbd20 2023-10-19T13:00:08.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.977+0000 7f3e1affd640 1 --2- 172.21.15.57:0/1329704458 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f3e0806ac90 0x7f3e0806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:08.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.977+0000 7f3e1affd640 1 -- 172.21.15.57:0/1329704458 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6803+0+0 (secure 0 0 0) 0x7f3e14014070 con 0x7f3e1c0cbd20 2023-10-19T13:00:08.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.977+0000 7f3e2202c640 1 --2- 172.21.15.57:0/1329704458 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f3e0806ac90 0x7f3e0806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:08.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.978+0000 7f3e2202c640 1 --2- 172.21.15.57:0/1329704458 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f3e0806ac90 0x7f3e0806d150 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f3e0c004a30 tx=0x7f3e0c009040 comp rx=0 tx=0).ready entity=mgr.6256 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:08.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:08.981+0000 7f3e1affd640 1 -- 172.21.15.57:0/1329704458 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3e14058d00 con 0x7f3e1c0cbd20 2023-10-19T13:00:09.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.162+0000 7f3e24290640 1 -- 172.21.15.57:0/1329704458 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3e1c0700d0 con 0x7f3e1c0cbd20 2023-10-19T13:00:09.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.163+0000 7f3e1affd640 1 -- 172.21.15.57:0/1329704458 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v76) v1 ==== 74+0+17962 (secure 0 0 0) 0x7f3e14058890 con 0x7f3e1c0cbd20 2023-10-19T13:00:09.165 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:00:09.165 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":76,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T13:00:07.028692+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2023-10-19T12:57:56.901606+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2023-10-19T12:57:57.581428+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":75,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.2a","mappings":[{"from":0,"to":6}]},{"pgid":"8.38","mappings":[{"from":0,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.179:6833/2207781045":"2023-10-20T13:00:07.028653+0000","172.21.15.179:0/3759354760":"2023-10-20T13:00:07.028653+0000","172.21.15.179:0/1939092102":"2023-10-20T13:00:07.028653+0000","172.21.15.179:0/1372200045":"2023-10-20T13:00:07.028653+0000","172.21.15.179:0/1174466246":"2023-10-20T13:00:07.028653+0000","172.21.15.179:0/1234281500":"2023-10-20T13:00:07.028653+0000","172.21.15.179:6832/2207781045":"2023-10-20T13:00:07.028653+0000","172.21.15.179:0/2482595099":"2023-10-20T13:00:07.028653+0000","172.21.15.179:0/70135229":"2023-10-20T12:58:14.538805+0000","172.21.15.179:6832/3449041082":"2023-10-20T12:58:14.538805+0000","172.21.15.179:0/1553007782":"2023-10-20T12:58:14.538805+0000","172.21.15.179:0/2777182879":"2023-10-20T12:58:14.538805+0000","172.21.15.179:6833/3449041082":"2023-10-20T12:58:14.538805+0000","172.21.15.179:0/64734340":"2023-10-20T12:58:14.538805+0000","172.21.15.179:0/1129452163":"2023-10-20T12:58:14.538805+0000","172.21.15.179:0/3893958736":"2023-10-20T12:58:14.538805+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T13:00:09.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.165+0000 7f3e24290640 1 -- 172.21.15.57:0/1329704458 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f3e0806ac90 msgr2=0x7f3e0806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:09.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.165+0000 7f3e24290640 1 --2- 172.21.15.57:0/1329704458 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f3e0806ac90 0x7f3e0806d150 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f3e0c004a30 tx=0x7f3e0c009040 comp rx=0 tx=0).stop 2023-10-19T13:00:09.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.165+0000 7f3e24290640 1 -- 172.21.15.57:0/1329704458 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3e1c0cbd20 msgr2=0x7f3e1c143210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:09.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.165+0000 7f3e24290640 1 --2- 172.21.15.57:0/1329704458 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3e1c0cbd20 0x7f3e1c143210 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f3e1400e9b0 tx=0x7f3e1400ee70 comp rx=0 tx=0).stop 2023-10-19T13:00:09.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.165+0000 7f3e24290640 1 -- 172.21.15.57:0/1329704458 shutdown_connections 2023-10-19T13:00:09.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.165+0000 7f3e24290640 1 --2- 172.21.15.57:0/1329704458 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f3e0806ac90 0x7f3e0806d150 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:09.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.165+0000 7f3e24290640 1 --2- 172.21.15.57:0/1329704458 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3e1c0cbd20 0x7f3e1c143210 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:09.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.165+0000 7f3e24290640 1 --2- 172.21.15.57:0/1329704458 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3e1c0b0530 0x7f3e1c1474b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:09.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.165+0000 7f3e24290640 1 --2- 172.21.15.57:0/1329704458 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3e1c0afa20 0x7f3e1c142440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:09.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.165+0000 7f3e24290640 1 -- 172.21.15.57:0/1329704458 >> 172.21.15.57:0/1329704458 conn(0x7f3e1c069930 msgr2=0x7f3e1c0acc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:09.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.165+0000 7f3e24290640 1 -- 172.21.15.57:0/1329704458 shutdown_connections 2023-10-19T13:00:09.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.165+0000 7f3e24290640 1 -- 172.21.15.57:0/1329704458 wait complete. 2023-10-19T13:00:09.179 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-19T13:00:09.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.287+0000 7f991f105640 1 -- 172.21.15.57:0/1891252133 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f99180ceb40 msgr2=0x7f99180b0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:09.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.287+0000 7f991f105640 1 --2- 172.21.15.57:0/1891252133 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f99180ceb40 0x7f99180b0650 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f9910005fd0 tx=0x7f991002c240 comp rx=0 tx=0).stop 2023-10-19T13:00:09.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.287+0000 7f991f105640 1 -- 172.21.15.57:0/1891252133 shutdown_connections 2023-10-19T13:00:09.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.287+0000 7f991f105640 1 --2- 172.21.15.57:0/1891252133 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f99180ceb40 0x7f99180b0650 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:09.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.287+0000 7f991f105640 1 --2- 172.21.15.57:0/1891252133 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f99180cf340 0x7f99180d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:09.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.287+0000 7f991f105640 1 -- 172.21.15.57:0/1891252133 >> 172.21.15.57:0/1891252133 conn(0x7f99180d0d60 msgr2=0x7f99180d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:09.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.287+0000 7f991f105640 1 -- 172.21.15.57:0/1891252133 shutdown_connections 2023-10-19T13:00:09.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.287+0000 7f991f105640 1 -- 172.21.15.57:0/1891252133 wait complete. 2023-10-19T13:00:09.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.288+0000 7f991f105640 1 Processor -- start 2023-10-19T13:00:09.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.288+0000 7f991f105640 1 -- start start 2023-10-19T13:00:09.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.289+0000 7f991f105640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f99180ceb40 0x7f99180b3950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:09.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.289+0000 7f991f105640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f99180cf340 0x7f99180b62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:09.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.289+0000 7f991f105640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f991815b2a0 0x7f99180b6810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:09.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.289+0000 7f991f105640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99180c7e30 con 0x7f99180ceb40 2023-10-19T13:00:09.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.289+0000 7f991f105640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99180c8080 con 0x7f99180cf340 2023-10-19T13:00:09.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.289+0000 7f991f105640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99180c8360 con 0x7f991815b2a0 2023-10-19T13:00:09.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.289+0000 7f991cea1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f99180ceb40 0x7f99180b3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:09.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.289+0000 7f991cea1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f99180ceb40 0x7f99180b3950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39850/0 (socket says 172.21.15.57:39850) 2023-10-19T13:00:09.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.289+0000 7f991cea1640 1 -- 172.21.15.57:0/3328558270 learned_addr learned my addr 172.21.15.57:0/3328558270 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:09.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.290+0000 7f991cea1640 1 -- 172.21.15.57:0/3328558270 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f991815b2a0 msgr2=0x7f99180b6810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:09.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.290+0000 7f991cea1640 1 --2- 172.21.15.57:0/3328558270 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f991815b2a0 0x7f99180b6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:09.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.290+0000 7f991cea1640 1 -- 172.21.15.57:0/3328558270 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f99180cf340 msgr2=0x7f99180b62d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:09.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.290+0000 7f991cea1640 1 --2- 172.21.15.57:0/3328558270 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f99180cf340 0x7f99180b62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:09.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.290+0000 7f991cea1640 1 -- 172.21.15.57:0/3328558270 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9910009900 con 0x7f99180ceb40 2023-10-19T13:00:09.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.290+0000 7f991cea1640 1 --2- 172.21.15.57:0/3328558270 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f99180ceb40 0x7f99180b3950 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f990800d770 tx=0x7f990800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:09.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.290+0000 7f99157fa640 1 -- 172.21.15.57:0/3328558270 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9908009c30 con 0x7f99180ceb40 2023-10-19T13:00:09.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.290+0000 7f99157fa640 1 -- 172.21.15.57:0/3328558270 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f990800de80 con 0x7f99180ceb40 2023-10-19T13:00:09.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.290+0000 7f991f105640 1 -- 172.21.15.57:0/3328558270 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99180c8640 con 0x7f99180ceb40 2023-10-19T13:00:09.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.290+0000 7f99157fa640 1 -- 172.21.15.57:0/3328558270 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9908005280 con 0x7f99180ceb40 2023-10-19T13:00:09.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.290+0000 7f991f105640 1 -- 172.21.15.57:0/3328558270 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99180c8c00 con 0x7f99180ceb40 2023-10-19T13:00:09.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.291+0000 7f991f105640 1 -- 172.21.15.57:0/3328558270 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99180690e0 con 0x7f99180ceb40 2023-10-19T13:00:09.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.291+0000 7f99157fa640 1 -- 172.21.15.57:0/3328558270 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 30) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f99080053e0 con 0x7f99180ceb40 2023-10-19T13:00:09.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.292+0000 7f99157fa640 1 --2- 172.21.15.57:0/3328558270 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f990406acc0 0x7f990406d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:09.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.292+0000 7f99157fa640 1 -- 172.21.15.57:0/3328558270 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6803+0+0 (secure 0 0 0) 0x7f9908089870 con 0x7f99180ceb40 2023-10-19T13:00:09.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.292+0000 7f9917fff640 1 --2- 172.21.15.57:0/3328558270 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f990406acc0 0x7f990406d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:09.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.293+0000 7f9917fff640 1 --2- 172.21.15.57:0/3328558270 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f990406acc0 0x7f990406d180 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f990c000f60 tx=0x7f990c0079b0 comp rx=0 tx=0).ready entity=mgr.6256 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:09.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.296+0000 7f99157fa640 1 -- 172.21.15.57:0/3328558270 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9908054810 con 0x7f99180ceb40 2023-10-19T13:00:09.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.510+0000 7f991f105640 1 -- 172.21.15.57:0/3328558270 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f9918158870 con 0x7f99180ceb40 2023-10-19T13:00:09.512 INFO:tasks.ceph.mon.a.smithi057.stderr:2023-10-19T13:00:09.510+0000 7f1253ff3640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-19T13:00:09.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.512+0000 7f99157fa640 1 -- 172.21.15.57:0/3328558270 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v24) v1 ==== 161+0+0 (secure 0 0 0) 0x7f9908017090 con 0x7f99180ceb40 2023-10-19T13:00:09.514 INFO:teuthology.orchestra.run.smithi057.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-19T13:00:09.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.514+0000 7f991f105640 1 -- 172.21.15.57:0/3328558270 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f990406acc0 msgr2=0x7f990406d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:09.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.514+0000 7f991f105640 1 --2- 172.21.15.57:0/3328558270 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f990406acc0 0x7f990406d180 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f990c000f60 tx=0x7f990c0079b0 comp rx=0 tx=0).stop 2023-10-19T13:00:09.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.514+0000 7f991f105640 1 -- 172.21.15.57:0/3328558270 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f99180ceb40 msgr2=0x7f99180b3950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:09.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.514+0000 7f991f105640 1 --2- 172.21.15.57:0/3328558270 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f99180ceb40 0x7f99180b3950 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f990800d770 tx=0x7f990800dc30 comp rx=0 tx=0).stop 2023-10-19T13:00:09.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.514+0000 7f991f105640 1 -- 172.21.15.57:0/3328558270 shutdown_connections 2023-10-19T13:00:09.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.514+0000 7f991f105640 1 --2- 172.21.15.57:0/3328558270 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f990406acc0 0x7f990406d180 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:09.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.514+0000 7f991f105640 1 --2- 172.21.15.57:0/3328558270 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f991815b2a0 0x7f99180b6810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:09.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.514+0000 7f991f105640 1 --2- 172.21.15.57:0/3328558270 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f99180cf340 0x7f99180b62d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:09.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.514+0000 7f991f105640 1 --2- 172.21.15.57:0/3328558270 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f99180ceb40 0x7f99180b3950 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:09.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.514+0000 7f991f105640 1 -- 172.21.15.57:0/3328558270 >> 172.21.15.57:0/3328558270 conn(0x7f99180d0d60 msgr2=0x7f991815a180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:09.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.514+0000 7f991f105640 1 -- 172.21.15.57:0/3328558270 shutdown_connections 2023-10-19T13:00:09.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.514+0000 7f991f105640 1 -- 172.21.15.57:0/3328558270 wait complete. 2023-10-19T13:00:09.528 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-19T13:00:09.553 INFO:tasks.ceph.mds.c.smithi057.stdout:starting mds.c at 2023-10-19T13:00:09.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.639+0000 7f249ea87640 1 -- 172.21.15.57:0/4253029522 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f249815b3e0 msgr2=0x7f24980cf2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:09.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.639+0000 7f249ea87640 1 --2- 172.21.15.57:0/4253029522 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f249815b3e0 0x7f24980cf2d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f248c005fd0 tx=0x7f248c02be10 comp rx=0 tx=0).stop 2023-10-19T13:00:09.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.639+0000 7f249ea87640 1 -- 172.21.15.57:0/4253029522 shutdown_connections 2023-10-19T13:00:09.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.639+0000 7f249ea87640 1 --2- 172.21.15.57:0/4253029522 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f249815b3e0 0x7f24980cf2d0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:09.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.639+0000 7f249ea87640 1 --2- 172.21.15.57:0/4253029522 >> v2:172.21.15.179:3300/0 conn(0x7f249815b000 0x7f24980d1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:09.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.639+0000 7f249ea87640 1 -- 172.21.15.57:0/4253029522 >> 172.21.15.57:0/4253029522 conn(0x7f24980d0cf0 msgr2=0x7f24980d1100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:09.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.640+0000 7f249ea87640 1 -- 172.21.15.57:0/4253029522 shutdown_connections 2023-10-19T13:00:09.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.640+0000 7f249ea87640 1 -- 172.21.15.57:0/4253029522 wait complete. 2023-10-19T13:00:09.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.641+0000 7f249ea87640 1 Processor -- start 2023-10-19T13:00:09.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.641+0000 7f249ea87640 1 -- start start 2023-10-19T13:00:09.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.642+0000 7f249ea87640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f24980afdc0 0x7f24980ca960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:09.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.642+0000 7f249ea87640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f249815b000 0x7f24981f1130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:09.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.642+0000 7f249ea87640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f249815b3e0 0x7f2498147910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:09.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.642+0000 7f249ea87640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24980caea0 con 0x7f24980afdc0 2023-10-19T13:00:09.650 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.642+0000 7f249ea87640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24981f1670 con 0x7f249815b3e0 2023-10-19T13:00:09.650 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.642+0000 7f249ea87640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24980c98c0 con 0x7f249815b000 2023-10-19T13:00:09.650 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.642+0000 7f249c823640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f24980afdc0 0x7f24980ca960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:09.650 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.642+0000 7f249c823640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f24980afdc0 0x7f24980ca960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39892/0 (socket says 172.21.15.57:39892) 2023-10-19T13:00:09.650 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.642+0000 7f249c823640 1 -- 172.21.15.57:0/2547158290 learned_addr learned my addr 172.21.15.57:0/2547158290 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:09.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.642+0000 7f249d024640 1 --2- 172.21.15.57:0/2547158290 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f249815b3e0 0x7f2498147910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:09.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.642+0000 7f2497fff640 1 --2- 172.21.15.57:0/2547158290 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f249815b000 0x7f24981f1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:09.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.642+0000 7f249c823640 1 -- 172.21.15.57:0/2547158290 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f249815b000 msgr2=0x7f24981f1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:09.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.642+0000 7f249c823640 1 --2- 172.21.15.57:0/2547158290 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f249815b000 0x7f24981f1130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:09.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.642+0000 7f249c823640 1 -- 172.21.15.57:0/2547158290 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f249815b3e0 msgr2=0x7f2498147910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:09.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.642+0000 7f249c823640 1 --2- 172.21.15.57:0/2547158290 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f249815b3e0 0x7f2498147910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:09.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.642+0000 7f249c823640 1 -- 172.21.15.57:0/2547158290 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f248c009560 con 0x7f24980afdc0 2023-10-19T13:00:09.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.642+0000 7f249c823640 1 --2- 172.21.15.57:0/2547158290 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f24980afdc0 0x7f24980ca960 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f248800e9f0 tx=0x7f248800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:09.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.642+0000 7f24957fa640 1 -- 172.21.15.57:0/2547158290 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2488009700 con 0x7f24980afdc0 2023-10-19T13:00:09.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.642+0000 7f24957fa640 1 -- 172.21.15.57:0/2547158290 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f248800ce70 con 0x7f24980afdc0 2023-10-19T13:00:09.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.642+0000 7f249ea87640 1 -- 172.21.15.57:0/2547158290 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24980c9ba0 con 0x7f24980afdc0 2023-10-19T13:00:09.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.642+0000 7f24957fa640 1 -- 172.21.15.57:0/2547158290 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2488010670 con 0x7f24980afdc0 2023-10-19T13:00:09.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.642+0000 7f249ea87640 1 -- 172.21.15.57:0/2547158290 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24980ca160 con 0x7f24980afdc0 2023-10-19T13:00:09.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.643+0000 7f249ea87640 1 -- 172.21.15.57:0/2547158290 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24980690e0 con 0x7f24980afdc0 2023-10-19T13:00:09.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.644+0000 7f24957fa640 1 -- 172.21.15.57:0/2547158290 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 30) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f24880107d0 con 0x7f24980afdc0 2023-10-19T13:00:09.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.644+0000 7f24957fa640 1 --2- 172.21.15.57:0/2547158290 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f248406ac90 0x7f248406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:09.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.645+0000 7f24957fa640 1 -- 172.21.15.57:0/2547158290 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6889+0+0 (secure 0 0 0) 0x7f2488014070 con 0x7f24980afdc0 2023-10-19T13:00:09.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.645+0000 7f2497fff640 1 --2- 172.21.15.57:0/2547158290 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f248406ac90 0x7f248406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:09.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.645+0000 7f2497fff640 1 --2- 172.21.15.57:0/2547158290 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f248406ac90 0x7f248406d150 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f248c035000 tx=0x7f248c004730 comp rx=0 tx=0).ready entity=mgr.6256 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:09.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.649+0000 7f24957fa640 1 -- 172.21.15.57:0/2547158290 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2488051ca0 con 0x7f24980afdc0 2023-10-19T13:00:09.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:09.862+0000 7f249ea87640 1 -- 172.21.15.57:0/2547158290 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f24980b1f00 con 0x7f24980afdc0 2023-10-19T13:00:10.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:10.058+0000 7f24957fa640 1 -- 172.21.15.57:0/2547158290 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mgrmap(e 31) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f248805f0c0 con 0x7f24980afdc0 2023-10-19T13:00:10.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:10.517+0000 7f24957fa640 1 -- 172.21.15.57:0/2547158290 <== mon.0 v2:172.21.15.57:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v25) v1 ==== 104+0+0 (secure 0 0 0) 0x7f24880849f0 con 0x7f24980afdc0 2023-10-19T13:00:10.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:10.520+0000 7f249ea87640 1 -- 172.21.15.57:0/2547158290 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f248406ac90 msgr2=0x7f248406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:10.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:10.520+0000 7f249ea87640 1 --2- 172.21.15.57:0/2547158290 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f248406ac90 0x7f248406d150 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f248c035000 tx=0x7f248c004730 comp rx=0 tx=0).stop 2023-10-19T13:00:10.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:10.520+0000 7f249ea87640 1 -- 172.21.15.57:0/2547158290 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f24980afdc0 msgr2=0x7f24980ca960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:10.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:10.520+0000 7f249ea87640 1 --2- 172.21.15.57:0/2547158290 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f24980afdc0 0x7f24980ca960 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f248800e9f0 tx=0x7f248800eeb0 comp rx=0 tx=0).stop 2023-10-19T13:00:10.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:10.520+0000 7f249ea87640 1 -- 172.21.15.57:0/2547158290 shutdown_connections 2023-10-19T13:00:10.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:10.520+0000 7f249ea87640 1 --2- 172.21.15.57:0/2547158290 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f248406ac90 0x7f248406d150 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:10.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:10.520+0000 7f249ea87640 1 --2- 172.21.15.57:0/2547158290 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f249815b3e0 0x7f2498147910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:10.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:10.520+0000 7f249ea87640 1 --2- 172.21.15.57:0/2547158290 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f249815b000 0x7f24981f1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:10.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:10.520+0000 7f249ea87640 1 --2- 172.21.15.57:0/2547158290 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f24980afdc0 0x7f24980ca960 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:10.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:10.520+0000 7f249ea87640 1 -- 172.21.15.57:0/2547158290 >> 172.21.15.57:0/2547158290 conn(0x7f24980d0cf0 msgr2=0x7f2498159df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:10.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:10.520+0000 7f249ea87640 1 -- 172.21.15.57:0/2547158290 shutdown_connections 2023-10-19T13:00:10.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:10.520+0000 7f249ea87640 1 -- 172.21.15.57:0/2547158290 wait complete. 2023-10-19T13:00:10.534 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-19T13:00:10.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:10.693+0000 7f714844f640 1 -- 172.21.15.57:0/3883437718 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f71400cf340 msgr2=0x7f71400d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:10.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:10.693+0000 7f714844f640 1 --2- 172.21.15.57:0/3883437718 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f71400cf340 0x7f71400d1910 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f713c009830 tx=0x7f713c02bcd0 comp rx=0 tx=0).stop 2023-10-19T13:00:10.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:10.693+0000 7f714844f640 1 -- 172.21.15.57:0/3883437718 shutdown_connections 2023-10-19T13:00:10.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:10.693+0000 7f714844f640 1 --2- 172.21.15.57:0/3883437718 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f71400ceb40 0x7f71400b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:10.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:10.693+0000 7f714844f640 1 --2- 172.21.15.57:0/3883437718 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f71400cf340 0x7f71400d1910 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:10.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:10.693+0000 7f714844f640 1 -- 172.21.15.57:0/3883437718 >> 172.21.15.57:0/3883437718 conn(0x7f71400d0d60 msgr2=0x7f71400d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:10.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:10.694+0000 7f714844f640 1 -- 172.21.15.57:0/3883437718 shutdown_connections 2023-10-19T13:00:10.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:10.694+0000 7f714844f640 1 -- 172.21.15.57:0/3883437718 wait complete. 2023-10-19T13:00:10.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:10.695+0000 7f714844f640 1 Processor -- start 2023-10-19T13:00:10.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:10.695+0000 7f714844f640 1 -- start start 2023-10-19T13:00:10.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:10.695+0000 7f714844f640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f71400ceb40 0x7f71400b6020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:10.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:10.695+0000 7f714844f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f71400cf340 0x7f71400b42f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:10.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:10.695+0000 7f714844f640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f714015b2a0 0x7f71401f0c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:10.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:10.695+0000 7f714844f640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71400b4830 con 0x7f71400cf340 2023-10-19T13:00:10.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:10.695+0000 7f714844f640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7140156da0 con 0x7f714015b2a0 2023-10-19T13:00:10.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:10.695+0000 7f714844f640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71400b52e0 con 0x7f71400ceb40 2023-10-19T13:00:10.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:10.696+0000 7f71459ea640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f71400cf340 0x7f71400b42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:10.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:10.696+0000 7f71459ea640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f71400cf340 0x7f71400b42f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39918/0 (socket says 172.21.15.57:39918) 2023-10-19T13:00:10.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:10.696+0000 7f71459ea640 1 -- 172.21.15.57:0/1517964899 learned_addr learned my addr 172.21.15.57:0/1517964899 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:10.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:10.696+0000 7f71461eb640 1 --2- 172.21.15.57:0/1517964899 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f71400ceb40 0x7f71400b6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:10.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:10.696+0000 7f71469ec640 1 --2- 172.21.15.57:0/1517964899 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f714015b2a0 0x7f71401f0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:10.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:10.696+0000 7f71459ea640 1 -- 172.21.15.57:0/1517964899 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f71400ceb40 msgr2=0x7f71400b6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:10.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:10.696+0000 7f71459ea640 1 --2- 172.21.15.57:0/1517964899 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f71400ceb40 0x7f71400b6020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:10.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:10.696+0000 7f71459ea640 1 -- 172.21.15.57:0/1517964899 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f714015b2a0 msgr2=0x7f71401f0c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:10.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:10.696+0000 7f71459ea640 1 --2- 172.21.15.57:0/1517964899 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f714015b2a0 0x7f71401f0c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:10.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:10.696+0000 7f71459ea640 1 -- 172.21.15.57:0/1517964899 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f713c009490 con 0x7f71400cf340 2023-10-19T13:00:10.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:10.696+0000 7f71469ec640 1 --2- 172.21.15.57:0/1517964899 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f714015b2a0 0x7f71401f0c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:00:10.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:10.696+0000 7f71459ea640 1 --2- 172.21.15.57:0/1517964899 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f71400cf340 0x7f71400b42f0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f713000c440 tx=0x7f713000c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:10.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:10.696+0000 7f7136ffd640 1 -- 172.21.15.57:0/1517964899 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7130005480 con 0x7f71400cf340 2023-10-19T13:00:10.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:10.696+0000 7f7136ffd640 1 -- 172.21.15.57:0/1517964899 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f71300035f0 con 0x7f71400cf340 2023-10-19T13:00:10.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:10.696+0000 7f714844f640 1 -- 172.21.15.57:0/1517964899 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71400b4a40 con 0x7f71400cf340 2023-10-19T13:00:10.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:10.696+0000 7f7136ffd640 1 -- 172.21.15.57:0/1517964899 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7130003a00 con 0x7f71400cf340 2023-10-19T13:00:10.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:10.696+0000 7f714844f640 1 -- 172.21.15.57:0/1517964899 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71400b5000 con 0x7f71400cf340 2023-10-19T13:00:10.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:10.697+0000 7f714844f640 1 -- 172.21.15.57:0/1517964899 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71400690e0 con 0x7f71400cf340 2023-10-19T13:00:10.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:10.698+0000 7f7136ffd640 1 -- 172.21.15.57:0/1517964899 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 31) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f7130003b60 con 0x7f71400cf340 2023-10-19T13:00:10.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:10.698+0000 7f7136ffd640 1 --2- 172.21.15.57:0/1517964899 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f712406aed0 0x7f712406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:10.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:10.698+0000 7f7136ffd640 1 -- 172.21.15.57:0/1517964899 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6889+0+0 (secure 0 0 0) 0x7f7130089180 con 0x7f71400cf340 2023-10-19T13:00:10.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:10.698+0000 7f71461eb640 1 --2- 172.21.15.57:0/1517964899 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f712406aed0 0x7f712406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:10.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:10.699+0000 7f71461eb640 1 --2- 172.21.15.57:0/1517964899 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f712406aed0 0x7f712406d390 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f713c036000 tx=0x7f713c004730 comp rx=0 tx=0).ready entity=mgr.6256 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:10.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:10.702+0000 7f7136ffd640 1 -- 172.21.15.57:0/1517964899 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f71300578e0 con 0x7f71400cf340 2023-10-19T13:00:10.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:10.878+0000 7f714844f640 1 -- 172.21.15.57:0/1517964899 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f71400b0cd0 con 0x7f71400cf340 2023-10-19T13:00:11.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:11.684+0000 7f7136ffd640 1 -- 172.21.15.57:0/1517964899 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v78) v1 ==== 181+0+0 (secure 0 0 0) 0x7f7130017020 con 0x7f71400cf340 2023-10-19T13:00:11.688 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'cephfs_metadata' removed 2023-10-19T13:00:11.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:11.687+0000 7f714844f640 1 -- 172.21.15.57:0/1517964899 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f712406aed0 msgr2=0x7f712406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:11.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:11.687+0000 7f714844f640 1 --2- 172.21.15.57:0/1517964899 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f712406aed0 0x7f712406d390 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f713c036000 tx=0x7f713c004730 comp rx=0 tx=0).stop 2023-10-19T13:00:11.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:11.687+0000 7f714844f640 1 -- 172.21.15.57:0/1517964899 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f71400cf340 msgr2=0x7f71400b42f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:11.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:11.687+0000 7f714844f640 1 --2- 172.21.15.57:0/1517964899 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f71400cf340 0x7f71400b42f0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f713000c440 tx=0x7f713000c900 comp rx=0 tx=0).stop 2023-10-19T13:00:11.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:11.687+0000 7f714844f640 1 -- 172.21.15.57:0/1517964899 shutdown_connections 2023-10-19T13:00:11.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:11.687+0000 7f714844f640 1 --2- 172.21.15.57:0/1517964899 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f712406aed0 0x7f712406d390 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:11.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:11.687+0000 7f714844f640 1 --2- 172.21.15.57:0/1517964899 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f714015b2a0 0x7f71401f0c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:11.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:11.687+0000 7f714844f640 1 --2- 172.21.15.57:0/1517964899 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f71400cf340 0x7f71400b42f0 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:11.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:11.687+0000 7f714844f640 1 --2- 172.21.15.57:0/1517964899 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f71400ceb40 0x7f71400b6020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:11.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:11.687+0000 7f714844f640 1 -- 172.21.15.57:0/1517964899 >> 172.21.15.57:0/1517964899 conn(0x7f71400d0d60 msgr2=0x7f71400b1770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:11.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:11.687+0000 7f714844f640 1 -- 172.21.15.57:0/1517964899 shutdown_connections 2023-10-19T13:00:11.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:11.687+0000 7f714844f640 1 -- 172.21.15.57:0/1517964899 wait complete. 2023-10-19T13:00:11.710 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-19T13:00:11.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:11.827+0000 7ff8f5075640 1 -- 172.21.15.57:0/1631168844 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff8dc0047c0 msgr2=0x7ff8dc004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:11.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:11.827+0000 7ff8f5075640 1 --2- 172.21.15.57:0/1631168844 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff8dc0047c0 0x7ff8dc004c40 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7ff8d8006f30 tx=0x7ff8dc0057f0 comp rx=0 tx=0).stop 2023-10-19T13:00:11.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:11.827+0000 7ff8ef59e640 1 -- 172.21.15.57:0/1631168844 reap_dead start 2023-10-19T13:00:11.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:11.827+0000 7ff8f5075640 1 -- 172.21.15.57:0/1631168844 shutdown_connections 2023-10-19T13:00:11.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:11.827+0000 7ff8f5075640 1 -- 172.21.15.57:0/1631168844 >> 172.21.15.57:0/1631168844 conn(0x7ff8f00d0d60 msgr2=0x7ff8f00d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:11.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:11.827+0000 7ff8f5075640 1 -- 172.21.15.57:0/1631168844 shutdown_connections 2023-10-19T13:00:11.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:11.827+0000 7ff8f5075640 1 -- 172.21.15.57:0/1631168844 wait complete. 2023-10-19T13:00:11.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:11.828+0000 7ff8f5075640 1 Processor -- start 2023-10-19T13:00:11.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:11.828+0000 7ff8f5075640 1 -- start start 2023-10-19T13:00:11.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:11.829+0000 7ff8f5075640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff8dc0047c0 0x7ff8f01e63d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:11.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:11.829+0000 7ff8f5075640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff8f01584a0 0x7ff8f014c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:11.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:11.829+0000 7ff8f5075640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff8f00b3950 0x7ff8f00b4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:11.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:11.829+0000 7ff8f5075640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f00b2670 con 0x7ff8dc0047c0 2023-10-19T13:00:11.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:11.829+0000 7ff8f5075640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f01e5f40 con 0x7ff8f00b3950 2023-10-19T13:00:11.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:11.829+0000 7ff8f5075640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f01e5330 con 0x7ff8f01584a0 2023-10-19T13:00:11.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:11.829+0000 7ff8eed9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff8dc0047c0 0x7ff8f01e63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:11.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:11.829+0000 7ff8eed9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff8dc0047c0 0x7ff8f01e63d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39948/0 (socket says 172.21.15.57:39948) 2023-10-19T13:00:11.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:11.829+0000 7ff8eed9d640 1 -- 172.21.15.57:0/1413696929 learned_addr learned my addr 172.21.15.57:0/1413696929 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:11.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:11.829+0000 7ff8ef59e640 1 --2- 172.21.15.57:0/1413696929 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff8f00b3950 0x7ff8f00b4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:11.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:11.829+0000 7ff8ee59c640 1 --2- 172.21.15.57:0/1413696929 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff8f01584a0 0x7ff8f014c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:11.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:11.829+0000 7ff8eed9d640 1 -- 172.21.15.57:0/1413696929 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff8f01584a0 msgr2=0x7ff8f014c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:11.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:11.829+0000 7ff8eed9d640 1 --2- 172.21.15.57:0/1413696929 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff8f01584a0 0x7ff8f014c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:11.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:11.829+0000 7ff8eed9d640 1 -- 172.21.15.57:0/1413696929 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff8f00b3950 msgr2=0x7ff8f00b4600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:11.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:11.829+0000 7ff8eed9d640 1 --2- 172.21.15.57:0/1413696929 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff8f00b3950 0x7ff8f00b4600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:11.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:11.829+0000 7ff8eed9d640 1 -- 172.21.15.57:0/1413696929 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8f01e5610 con 0x7ff8dc0047c0 2023-10-19T13:00:11.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:11.829+0000 7ff8ef59e640 1 --2- 172.21.15.57:0/1413696929 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff8f00b3950 0x7ff8f00b4600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T13:00:11.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:11.829+0000 7ff8ee59c640 1 --2- 172.21.15.57:0/1413696929 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff8f01584a0 0x7ff8f014c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T13:00:11.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:11.830+0000 7ff8eed9d640 1 --2- 172.21.15.57:0/1413696929 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff8dc0047c0 0x7ff8f01e63d0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7ff8e80119a0 tx=0x7ff8e8011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:11.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:11.830+0000 7ff8e77fe640 1 -- 172.21.15.57:0/1413696929 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8e800fe70 con 0x7ff8dc0047c0 2023-10-19T13:00:11.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:11.830+0000 7ff8e77fe640 1 -- 172.21.15.57:0/1413696929 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7ff8e8004140 con 0x7ff8dc0047c0 2023-10-19T13:00:11.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:11.830+0000 7ff8f5075640 1 -- 172.21.15.57:0/1413696929 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8f01e58f0 con 0x7ff8dc0047c0 2023-10-19T13:00:11.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:11.830+0000 7ff8e77fe640 1 -- 172.21.15.57:0/1413696929 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8e80044f0 con 0x7ff8dc0047c0 2023-10-19T13:00:11.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:11.830+0000 7ff8f5075640 1 -- 172.21.15.57:0/1413696929 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8f00b3e00 con 0x7ff8dc0047c0 2023-10-19T13:00:11.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:11.831+0000 7ff8f5075640 1 -- 172.21.15.57:0/1413696929 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8f00690e0 con 0x7ff8dc0047c0 2023-10-19T13:00:11.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:11.831+0000 7ff8e77fe640 1 -- 172.21.15.57:0/1413696929 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 32) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff8e80035f0 con 0x7ff8dc0047c0 2023-10-19T13:00:11.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:11.832+0000 7ff8e77fe640 1 --2- 172.21.15.57:0/1413696929 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7ff8d006af90 0x7ff8d006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:11.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:11.832+0000 7ff8e77fe640 1 -- 172.21.15.57:0/1413696929 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6451+0+0 (secure 0 0 0) 0x7ff8e808aad0 con 0x7ff8dc0047c0 2023-10-19T13:00:11.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:11.832+0000 7ff8ee59c640 1 --2- 172.21.15.57:0/1413696929 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7ff8d006af90 0x7ff8d006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:11.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:11.832+0000 7ff8ee59c640 1 --2- 172.21.15.57:0/1413696929 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7ff8d006af90 0x7ff8d006d450 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff8d8034000 tx=0x7ff8d8039040 comp rx=0 tx=0).ready entity=mgr.6256 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:11.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:11.836+0000 7ff8e77fe640 1 -- 172.21.15.57:0/1413696929 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff8e8055700 con 0x7ff8dc0047c0 2023-10-19T13:00:12.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:12.015+0000 7ff8f5075640 1 -- 172.21.15.57:0/1413696929 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7ff8f00d0b40 con 0x7ff8dc0047c0 2023-10-19T13:00:12.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:12.686+0000 7ff8e77fe640 1 -- 172.21.15.57:0/1413696929 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v79) v1 ==== 169+0+0 (secure 0 0 0) 0x7ff8e801e0d0 con 0x7ff8dc0047c0 2023-10-19T13:00:12.689 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'cephfs_data' removed 2023-10-19T13:00:12.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:12.689+0000 7ff8f5075640 1 -- 172.21.15.57:0/1413696929 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7ff8d006af90 msgr2=0x7ff8d006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:12.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:12.689+0000 7ff8f5075640 1 --2- 172.21.15.57:0/1413696929 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7ff8d006af90 0x7ff8d006d450 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff8d8034000 tx=0x7ff8d8039040 comp rx=0 tx=0).stop 2023-10-19T13:00:12.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:12.689+0000 7ff8f5075640 1 -- 172.21.15.57:0/1413696929 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff8dc0047c0 msgr2=0x7ff8f01e63d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:12.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:12.689+0000 7ff8f5075640 1 --2- 172.21.15.57:0/1413696929 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff8dc0047c0 0x7ff8f01e63d0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7ff8e80119a0 tx=0x7ff8e8011e60 comp rx=0 tx=0).stop 2023-10-19T13:00:12.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:12.689+0000 7ff8f5075640 1 -- 172.21.15.57:0/1413696929 shutdown_connections 2023-10-19T13:00:12.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:12.689+0000 7ff8f5075640 1 --2- 172.21.15.57:0/1413696929 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7ff8d006af90 0x7ff8d006d450 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:12.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:12.689+0000 7ff8f5075640 1 --2- 172.21.15.57:0/1413696929 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff8f00b3950 0x7ff8f00b4600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:12.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:12.689+0000 7ff8f5075640 1 --2- 172.21.15.57:0/1413696929 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff8f01584a0 0x7ff8f014c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:12.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:12.689+0000 7ff8f5075640 1 --2- 172.21.15.57:0/1413696929 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff8dc0047c0 0x7ff8f01e63d0 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:12.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:12.689+0000 7ff8f5075640 1 -- 172.21.15.57:0/1413696929 >> 172.21.15.57:0/1413696929 conn(0x7ff8f00d0d60 msgr2=0x7ff8f015a1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:12.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:12.689+0000 7ff8f5075640 1 -- 172.21.15.57:0/1413696929 shutdown_connections 2023-10-19T13:00:12.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:12.689+0000 7ff8f5075640 1 -- 172.21.15.57:0/1413696929 wait complete. 2023-10-19T13:00:12.709 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 30 2023-10-19T13:00:12.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:12.823+0000 7fc52b525640 1 -- 172.21.15.57:0/415079372 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc510005000 msgr2=0x7fc5100058d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:12.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:12.823+0000 7fc52b525640 1 --2- 172.21.15.57:0/415079372 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc510005000 0x7fc5100058d0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7fc514009b50 tx=0x7fc510006900 comp rx=0 tx=0).stop 2023-10-19T13:00:12.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:12.823+0000 7fc529ac2640 1 -- 172.21.15.57:0/415079372 reap_dead start 2023-10-19T13:00:12.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:12.823+0000 7fc52b525640 1 -- 172.21.15.57:0/415079372 shutdown_connections 2023-10-19T13:00:12.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:12.823+0000 7fc52b525640 1 -- 172.21.15.57:0/415079372 >> 172.21.15.57:0/415079372 conn(0x7fc524069930 msgr2=0x7fc524069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:12.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:12.823+0000 7fc52b525640 1 -- 172.21.15.57:0/415079372 shutdown_connections 2023-10-19T13:00:12.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:12.823+0000 7fc52b525640 1 -- 172.21.15.57:0/415079372 wait complete. 2023-10-19T13:00:12.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:12.824+0000 7fc52b525640 1 Processor -- start 2023-10-19T13:00:12.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:12.824+0000 7fc52b525640 1 -- start start 2023-10-19T13:00:12.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:12.825+0000 7fc52b525640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc510005000 0x7fc524137e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:12.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:12.825+0000 7fc52b525640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc52406ba80 0x7fc52406bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:12.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:12.825+0000 7fc52b525640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc524138960 0x7fc524138de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:12.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:12.825+0000 7fc52b525640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc52413a310 con 0x7fc52406ba80 2023-10-19T13:00:12.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:12.825+0000 7fc52b525640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc52406c440 con 0x7fc510005000 2023-10-19T13:00:12.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:12.825+0000 7fc52b525640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc524139690 con 0x7fc524138960 2023-10-19T13:00:12.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:12.825+0000 7fc528ac0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc52406ba80 0x7fc52406bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:12.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:12.825+0000 7fc528ac0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc52406ba80 0x7fc52406bf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39972/0 (socket says 172.21.15.57:39972) 2023-10-19T13:00:12.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:12.825+0000 7fc528ac0640 1 -- 172.21.15.57:0/606647438 learned_addr learned my addr 172.21.15.57:0/606647438 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:12.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:12.825+0000 7fc5292c1640 1 --2- 172.21.15.57:0/606647438 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc510005000 0x7fc524137e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:12.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:12.825+0000 7fc529ac2640 1 --2- 172.21.15.57:0/606647438 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc524138960 0x7fc524138de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:12.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:12.825+0000 7fc528ac0640 1 -- 172.21.15.57:0/606647438 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc524138960 msgr2=0x7fc524138de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:12.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:12.825+0000 7fc528ac0640 1 --2- 172.21.15.57:0/606647438 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc524138960 0x7fc524138de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:12.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:12.825+0000 7fc528ac0640 1 -- 172.21.15.57:0/606647438 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc510005000 msgr2=0x7fc524137e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:12.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:12.825+0000 7fc528ac0640 1 --2- 172.21.15.57:0/606647438 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc510005000 0x7fc524137e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:12.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:12.825+0000 7fc528ac0640 1 -- 172.21.15.57:0/606647438 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc524070cf0 con 0x7fc52406ba80 2023-10-19T13:00:12.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:12.825+0000 7fc528ac0640 1 --2- 172.21.15.57:0/606647438 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc52406ba80 0x7fc52406bf00 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fc51402cc20 tx=0x7fc514033c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:12.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:12.825+0000 7fc519ffb640 1 -- 172.21.15.57:0/606647438 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc514004220 con 0x7fc52406ba80 2023-10-19T13:00:12.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:12.826+0000 7fc519ffb640 1 -- 172.21.15.57:0/606647438 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fc514004380 con 0x7fc52406ba80 2023-10-19T13:00:12.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:12.826+0000 7fc52b525640 1 -- 172.21.15.57:0/606647438 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc524139970 con 0x7fc52406ba80 2023-10-19T13:00:12.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:12.826+0000 7fc519ffb640 1 -- 172.21.15.57:0/606647438 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc51403d880 con 0x7fc52406ba80 2023-10-19T13:00:12.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:12.826+0000 7fc52b525640 1 -- 172.21.15.57:0/606647438 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc524139f30 con 0x7fc52406ba80 2023-10-19T13:00:12.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:12.827+0000 7fc52b525640 1 -- 172.21.15.57:0/606647438 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5240690e0 con 0x7fc52406ba80 2023-10-19T13:00:12.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:12.827+0000 7fc519ffb640 1 -- 172.21.15.57:0/606647438 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 32) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc51403d9e0 con 0x7fc52406ba80 2023-10-19T13:00:12.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:12.828+0000 7fc519ffb640 1 --2- 172.21.15.57:0/606647438 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7fc50806af60 0x7fc50806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:12.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:12.828+0000 7fc519ffb640 1 -- 172.21.15.57:0/606647438 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 5979+0+0 (secure 0 0 0) 0x7fc5140ac5b0 con 0x7fc52406ba80 2023-10-19T13:00:12.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:12.828+0000 7fc5292c1640 1 --2- 172.21.15.57:0/606647438 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7fc50806af60 0x7fc50806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:12.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:12.829+0000 7fc5292c1640 1 --2- 172.21.15.57:0/606647438 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7fc50806af60 0x7fc50806d420 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc51c009b70 tx=0x7fc51c005f90 comp rx=0 tx=0).ready entity=mgr.6256 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:12.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:12.832+0000 7fc519ffb640 1 -- 172.21.15.57:0/606647438 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc514077470 con 0x7fc52406ba80 2023-10-19T13:00:13.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.013+0000 7fc52b525640 1 -- 172.21.15.57:0/606647438 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "config reset", "num": 30} v 0) v1 -- 0x7fc52406f700 con 0x7fc52406ba80 2023-10-19T13:00:13.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.016+0000 7fc519ffb640 1 -- 172.21.15.57:0/606647438 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 30}]=0 v38) v1 ==== 71+0+0 (secure 0 0 0) 0x7fc51407aa90 con 0x7fc52406ba80 2023-10-19T13:00:13.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.018+0000 7fc52b525640 1 -- 172.21.15.57:0/606647438 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7fc50806af60 msgr2=0x7fc50806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:13.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.018+0000 7fc52b525640 1 --2- 172.21.15.57:0/606647438 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7fc50806af60 0x7fc50806d420 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc51c009b70 tx=0x7fc51c005f90 comp rx=0 tx=0).stop 2023-10-19T13:00:13.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.018+0000 7fc52b525640 1 -- 172.21.15.57:0/606647438 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc52406ba80 msgr2=0x7fc52406bf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:13.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.018+0000 7fc52b525640 1 --2- 172.21.15.57:0/606647438 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc52406ba80 0x7fc52406bf00 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fc51402cc20 tx=0x7fc514033c30 comp rx=0 tx=0).stop 2023-10-19T13:00:13.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.018+0000 7fc52b525640 1 -- 172.21.15.57:0/606647438 shutdown_connections 2023-10-19T13:00:13.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.018+0000 7fc52b525640 1 --2- 172.21.15.57:0/606647438 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7fc50806af60 0x7fc50806d420 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:13.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.018+0000 7fc52b525640 1 --2- 172.21.15.57:0/606647438 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc524138960 0x7fc524138de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:13.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.018+0000 7fc52b525640 1 --2- 172.21.15.57:0/606647438 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc52406ba80 0x7fc52406bf00 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:13.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.018+0000 7fc52b525640 1 --2- 172.21.15.57:0/606647438 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc510005000 0x7fc524137e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:13.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.018+0000 7fc52b525640 1 -- 172.21.15.57:0/606647438 >> 172.21.15.57:0/606647438 conn(0x7fc524069930 msgr2=0x7fc5240ae790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:13.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.019+0000 7fc52b525640 1 -- 172.21.15.57:0/606647438 shutdown_connections 2023-10-19T13:00:13.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.019+0000 7fc52b525640 1 -- 172.21.15.57:0/606647438 wait complete. 2023-10-19T13:00:13.032 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule' 2023-10-19T13:00:13.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.139+0000 7fd922a10640 1 -- 172.21.15.57:0/120665832 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd9040047c0 msgr2=0x7fd904004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:13.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.139+0000 7fd922a10640 1 --2- 172.21.15.57:0/120665832 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd9040047c0 0x7fd904004c40 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fd91000c290 tx=0x7fd9040057f0 comp rx=0 tx=0).stop 2023-10-19T13:00:13.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.139+0000 7fd920fad640 1 -- 172.21.15.57:0/120665832 reap_dead start 2023-10-19T13:00:13.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.139+0000 7fd922a10640 1 -- 172.21.15.57:0/120665832 shutdown_connections 2023-10-19T13:00:13.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.139+0000 7fd922a10640 1 -- 172.21.15.57:0/120665832 >> 172.21.15.57:0/120665832 conn(0x7fd91c0d0d60 msgr2=0x7fd91c0d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:13.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.139+0000 7fd922a10640 1 -- 172.21.15.57:0/120665832 shutdown_connections 2023-10-19T13:00:13.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.139+0000 7fd922a10640 1 -- 172.21.15.57:0/120665832 wait complete. 2023-10-19T13:00:13.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.140+0000 7fd922a10640 1 Processor -- start 2023-10-19T13:00:13.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.140+0000 7fd922a10640 1 -- start start 2023-10-19T13:00:13.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.141+0000 7fd922a10640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd9040047c0 0x7fd91c1501d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:13.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.141+0000 7fd922a10640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd91c15d130 0x7fd91c15b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:13.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.141+0000 7fd922a10640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd91c15bc10 0x7fd91c1f4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:13.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.141+0000 7fd922a10640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd91c150710 con 0x7fd91c15bc10 2023-10-19T13:00:13.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.141+0000 7fd922a10640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd91c15c090 con 0x7fd91c15d130 2023-10-19T13:00:13.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.141+0000 7fd922a10640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd91c15c370 con 0x7fd9040047c0 2023-10-19T13:00:13.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.141+0000 7fd920fad640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd91c15bc10 0x7fd91c1f4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:13.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.141+0000 7fd920fad640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd91c15bc10 0x7fd91c1f4cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39980/0 (socket says 172.21.15.57:39980) 2023-10-19T13:00:13.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.141+0000 7fd920fad640 1 -- 172.21.15.57:0/2910014527 learned_addr learned my addr 172.21.15.57:0/2910014527 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:13.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.141+0000 7fd91b7fe640 1 --2- 172.21.15.57:0/2910014527 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd91c15d130 0x7fd91c15b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:13.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.141+0000 7fd91bfff640 1 --2- 172.21.15.57:0/2910014527 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd9040047c0 0x7fd91c1501d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:13.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.141+0000 7fd920fad640 1 -- 172.21.15.57:0/2910014527 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd9040047c0 msgr2=0x7fd91c1501d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:13.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.141+0000 7fd920fad640 1 --2- 172.21.15.57:0/2910014527 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd9040047c0 0x7fd91c1501d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:13.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.141+0000 7fd920fad640 1 -- 172.21.15.57:0/2910014527 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd91c15d130 msgr2=0x7fd91c15b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:13.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.141+0000 7fd920fad640 1 --2- 172.21.15.57:0/2910014527 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd91c15d130 0x7fd91c15b6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:13.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.141+0000 7fd920fad640 1 -- 172.21.15.57:0/2910014527 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd91c15c650 con 0x7fd91c15bc10 2023-10-19T13:00:13.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.141+0000 7fd91bfff640 1 --2- 172.21.15.57:0/2910014527 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd9040047c0 0x7fd91c1501d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:00:13.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.141+0000 7fd920fad640 1 --2- 172.21.15.57:0/2910014527 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd91c15bc10 0x7fd91c1f4cf0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fd91c0b6e10 tx=0x7fd91400f3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:13.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.142+0000 7fd918ff9640 1 -- 172.21.15.57:0/2910014527 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd914004b20 con 0x7fd91c15bc10 2023-10-19T13:00:13.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.142+0000 7fd918ff9640 1 -- 172.21.15.57:0/2910014527 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fd914004c80 con 0x7fd91c15bc10 2023-10-19T13:00:13.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.142+0000 7fd918ff9640 1 -- 172.21.15.57:0/2910014527 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd914003bd0 con 0x7fd91c15bc10 2023-10-19T13:00:13.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.142+0000 7fd922a10640 1 -- 172.21.15.57:0/2910014527 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd91c15c930 con 0x7fd91c15bc10 2023-10-19T13:00:13.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.142+0000 7fd922a10640 1 -- 172.21.15.57:0/2910014527 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd91c1f5970 con 0x7fd91c15bc10 2023-10-19T13:00:13.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.143+0000 7fd922a10640 1 -- 172.21.15.57:0/2910014527 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd91c15ce50 con 0x7fd91c15bc10 2023-10-19T13:00:13.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.143+0000 7fd918ff9640 1 -- 172.21.15.57:0/2910014527 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 32) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd914003d30 con 0x7fd91c15bc10 2023-10-19T13:00:13.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.143+0000 7fd918ff9640 1 --2- 172.21.15.57:0/2910014527 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7fd90806af90 0x7fd90806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:13.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.144+0000 7fd918ff9640 1 -- 172.21.15.57:0/2910014527 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 5979+0+0 (secure 0 0 0) 0x7fd914089d40 con 0x7fd91c15bc10 2023-10-19T13:00:13.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.144+0000 7fd91bfff640 1 --2- 172.21.15.57:0/2910014527 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7fd90806af90 0x7fd90806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:13.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.144+0000 7fd91bfff640 1 --2- 172.21.15.57:0/2910014527 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7fd90806af90 0x7fd90806d450 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd90c009c50 tx=0x7fd90c005f90 comp rx=0 tx=0).ready entity=mgr.6256 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:13.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.148+0000 7fd918ff9640 1 -- 172.21.15.57:0/2910014527 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd91c15ce50 con 0x7fd91c15bc10 2023-10-19T13:00:13.399 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.396+0000 7fd922a10640 1 -- 172.21.15.57:0/2910014527 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule"]} v 0) v1 -- 0x7fd91c15ce50 con 0x7fd91c15bc10 2023-10-19T13:00:13.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.699+0000 7fd918ff9640 1 -- 172.21.15.57:0/2910014527 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule"]}]=0 v582) v1 ==== 154+0+0 (secure 0 0 0) 0x7fd91c15ce50 con 0x7fd91c15bc10 2023-10-19T13:00:13.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.702+0000 7fd922a10640 1 -- 172.21.15.57:0/2910014527 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7fd90806af90 msgr2=0x7fd90806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:13.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.702+0000 7fd922a10640 1 --2- 172.21.15.57:0/2910014527 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7fd90806af90 0x7fd90806d450 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd90c009c50 tx=0x7fd90c005f90 comp rx=0 tx=0).stop 2023-10-19T13:00:13.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.702+0000 7fd922a10640 1 -- 172.21.15.57:0/2910014527 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd91c15bc10 msgr2=0x7fd91c1f4cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:13.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.702+0000 7fd922a10640 1 --2- 172.21.15.57:0/2910014527 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd91c15bc10 0x7fd91c1f4cf0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fd91c0b6e10 tx=0x7fd91400f3c0 comp rx=0 tx=0).stop 2023-10-19T13:00:13.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.702+0000 7fd922a10640 1 -- 172.21.15.57:0/2910014527 shutdown_connections 2023-10-19T13:00:13.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.702+0000 7fd922a10640 1 --2- 172.21.15.57:0/2910014527 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7fd90806af90 0x7fd90806d450 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:13.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.702+0000 7fd922a10640 1 --2- 172.21.15.57:0/2910014527 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd91c15bc10 0x7fd91c1f4cf0 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:13.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.702+0000 7fd922a10640 1 --2- 172.21.15.57:0/2910014527 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd91c15d130 0x7fd91c15b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:13.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.702+0000 7fd922a10640 1 --2- 172.21.15.57:0/2910014527 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd9040047c0 0x7fd91c1501d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:13.707 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.702+0000 7fd922a10640 1 -- 172.21.15.57:0/2910014527 >> 172.21.15.57:0/2910014527 conn(0x7fd91c0d0d60 msgr2=0x7fd91c0af390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:13.707 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.702+0000 7fd922a10640 1 -- 172.21.15.57:0/2910014527 shutdown_connections 2023-10-19T13:00:13.707 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.702+0000 7fd922a10640 1 -- 172.21.15.57:0/2910014527 wait complete. 2023-10-19T13:00:13.716 INFO:tasks.cephfs_test_runner:Test exisitence of multiple scheduled snapshots ... ok 2023-10-19T13:00:13.716 INFO:tasks.cephfs_test_runner:Starting test: test_non_existent_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test listing non-existing snap schedules failure 2023-10-19T13:00:13.716 INFO:tasks.cephfs_test_runner:test_non_existent_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-10-19T13:00:13.717 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T13:00:13.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.825+0000 7fe15b4e1640 1 -- 172.21.15.57:0/4126719213 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe15415b2a0 msgr2=0x7fe1540d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:13.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.825+0000 7fe15b4e1640 1 --2- 172.21.15.57:0/4126719213 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe15415b2a0 0x7fe1540d1910 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fe148009830 tx=0x7fe14802bcd0 comp rx=0 tx=0).stop 2023-10-19T13:00:13.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.825+0000 7fe15b4e1640 1 -- 172.21.15.57:0/4126719213 shutdown_connections 2023-10-19T13:00:13.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.825+0000 7fe15b4e1640 1 --2- 172.21.15.57:0/4126719213 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe1540ceb40 0x7fe1540b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:13.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.825+0000 7fe15b4e1640 1 --2- 172.21.15.57:0/4126719213 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe15415b2a0 0x7fe1540d1910 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:13.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.825+0000 7fe15b4e1640 1 -- 172.21.15.57:0/4126719213 >> 172.21.15.57:0/4126719213 conn(0x7fe1540d0d60 msgr2=0x7fe1540d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:13.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.825+0000 7fe15b4e1640 1 -- 172.21.15.57:0/4126719213 shutdown_connections 2023-10-19T13:00:13.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.825+0000 7fe15b4e1640 1 -- 172.21.15.57:0/4126719213 wait complete. 2023-10-19T13:00:13.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.826+0000 7fe15b4e1640 1 Processor -- start 2023-10-19T13:00:13.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.826+0000 7fe15b4e1640 1 -- start start 2023-10-19T13:00:13.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.827+0000 7fe15b4e1640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe1540ceb40 0x7fe154151150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:13.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.827+0000 7fe15b4e1640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe1540cf340 0x7fe1540b4900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:13.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.827+0000 7fe15b4e1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe15415b2a0 0x7fe15406a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:13.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.827+0000 7fe15b4e1640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1540b52e0 con 0x7fe15415b2a0 2023-10-19T13:00:13.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.827+0000 7fe15b4e1640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1540b55c0 con 0x7fe1540cf340 2023-10-19T13:00:13.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.827+0000 7fe15b4e1640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1540b58a0 con 0x7fe1540ceb40 2023-10-19T13:00:13.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.827+0000 7fe159a7e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe15415b2a0 0x7fe15406a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:13.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.827+0000 7fe158a7c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe1540cf340 0x7fe1540b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:13.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.827+0000 7fe159a7e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe15415b2a0 0x7fe15406a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39990/0 (socket says 172.21.15.57:39990) 2023-10-19T13:00:13.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.827+0000 7fe159a7e640 1 -- 172.21.15.57:0/2494206317 learned_addr learned my addr 172.21.15.57:0/2494206317 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:13.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.827+0000 7fe15927d640 1 --2- 172.21.15.57:0/2494206317 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe1540ceb40 0x7fe154151150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:13.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.827+0000 7fe159a7e640 1 -- 172.21.15.57:0/2494206317 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe1540ceb40 msgr2=0x7fe154151150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:13.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.827+0000 7fe159a7e640 1 --2- 172.21.15.57:0/2494206317 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe1540ceb40 0x7fe154151150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:13.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.827+0000 7fe159a7e640 1 -- 172.21.15.57:0/2494206317 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe1540cf340 msgr2=0x7fe1540b4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:13.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.827+0000 7fe159a7e640 1 --2- 172.21.15.57:0/2494206317 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe1540cf340 0x7fe1540b4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:13.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.827+0000 7fe159a7e640 1 -- 172.21.15.57:0/2494206317 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe14c009560 con 0x7fe15415b2a0 2023-10-19T13:00:13.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.827+0000 7fe158a7c640 1 --2- 172.21.15.57:0/2494206317 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe1540cf340 0x7fe1540b4900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T13:00:13.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.827+0000 7fe159a7e640 1 --2- 172.21.15.57:0/2494206317 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe15415b2a0 0x7fe15406a1f0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fe14c00e940 tx=0x7fe14c00ee00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:13.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.828+0000 7fe151ffb640 1 -- 172.21.15.57:0/2494206317 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe14c00cd80 con 0x7fe15415b2a0 2023-10-19T13:00:13.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.828+0000 7fe151ffb640 1 -- 172.21.15.57:0/2494206317 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fe14c004030 con 0x7fe15415b2a0 2023-10-19T13:00:13.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.828+0000 7fe15b4e1640 1 -- 172.21.15.57:0/2494206317 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe148009490 con 0x7fe15415b2a0 2023-10-19T13:00:13.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.828+0000 7fe151ffb640 1 -- 172.21.15.57:0/2494206317 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe14c004440 con 0x7fe15415b2a0 2023-10-19T13:00:13.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.828+0000 7fe15b4e1640 1 -- 172.21.15.57:0/2494206317 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1540b5e60 con 0x7fe15415b2a0 2023-10-19T13:00:13.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.829+0000 7fe15b4e1640 1 -- 172.21.15.57:0/2494206317 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1540677b0 con 0x7fe15415b2a0 2023-10-19T13:00:13.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.829+0000 7fe151ffb640 1 -- 172.21.15.57:0/2494206317 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 32) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe14c0035f0 con 0x7fe15415b2a0 2023-10-19T13:00:13.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.829+0000 7fe151ffb640 1 --2- 172.21.15.57:0/2494206317 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7fe14006af60 0x7fe14006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:13.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.829+0000 7fe151ffb640 1 -- 172.21.15.57:0/2494206317 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 5979+0+0 (secure 0 0 0) 0x7fe14c014070 con 0x7fe15415b2a0 2023-10-19T13:00:13.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.830+0000 7fe15927d640 1 --2- 172.21.15.57:0/2494206317 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7fe14006af60 0x7fe14006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:13.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.830+0000 7fe15927d640 1 --2- 172.21.15.57:0/2494206317 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7fe14006af60 0x7fe14006d420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe144001ea0 tx=0x7fe144009040 comp rx=0 tx=0).ready entity=mgr.6256 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:13.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:13.834+0000 7fe151ffb640 1 -- 172.21.15.57:0/2494206317 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe14c055280 con 0x7fe15415b2a0 2023-10-19T13:00:14.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.010+0000 7fe15b4e1640 1 -- 172.21.15.57:0/2494206317 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe1540b0cd0 con 0x7fe15415b2a0 2023-10-19T13:00:14.014 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.011+0000 7fe151ffb640 1 -- 172.21.15.57:0/2494206317 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) v1 ==== 74+0+14260 (secure 0 0 0) 0x7fe14c0588a0 con 0x7fe15415b2a0 2023-10-19T13:00:14.014 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:00:14.015 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":79,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T13:00:12.685846+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.57:6835/243029526":"2023-10-20T13:00:09.510516+0000","172.21.15.179:6833/2207781045":"2023-10-20T13:00:07.028653+0000","172.21.15.179:0/3759354760":"2023-10-20T13:00:07.028653+0000","172.21.15.179:0/1939092102":"2023-10-20T13:00:07.028653+0000","172.21.15.179:0/1372200045":"2023-10-20T13:00:07.028653+0000","172.21.15.179:0/1174466246":"2023-10-20T13:00:07.028653+0000","172.21.15.179:0/1234281500":"2023-10-20T13:00:07.028653+0000","172.21.15.179:6832/2207781045":"2023-10-20T13:00:07.028653+0000","172.21.15.179:0/2482595099":"2023-10-20T13:00:07.028653+0000","172.21.15.179:0/70135229":"2023-10-20T12:58:14.538805+0000","172.21.15.179:6832/3449041082":"2023-10-20T12:58:14.538805+0000","172.21.15.179:0/1553007782":"2023-10-20T12:58:14.538805+0000","172.21.15.179:0/2777182879":"2023-10-20T12:58:14.538805+0000","172.21.15.179:6833/3449041082":"2023-10-20T12:58:14.538805+0000","172.21.15.179:0/64734340":"2023-10-20T12:58:14.538805+0000","172.21.15.179:0/1129452163":"2023-10-20T12:58:14.538805+0000","172.21.15.57:6833/243029526":"2023-10-20T13:00:09.510516+0000","172.21.15.179:0/3893958736":"2023-10-20T12:58:14.538805+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T13:00:14.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.014+0000 7fe15b4e1640 1 -- 172.21.15.57:0/2494206317 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7fe14006af60 msgr2=0x7fe14006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:14.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.014+0000 7fe15b4e1640 1 --2- 172.21.15.57:0/2494206317 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7fe14006af60 0x7fe14006d420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe144001ea0 tx=0x7fe144009040 comp rx=0 tx=0).stop 2023-10-19T13:00:14.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.014+0000 7fe15b4e1640 1 -- 172.21.15.57:0/2494206317 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe15415b2a0 msgr2=0x7fe15406a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:14.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.014+0000 7fe15b4e1640 1 --2- 172.21.15.57:0/2494206317 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe15415b2a0 0x7fe15406a1f0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fe14c00e940 tx=0x7fe14c00ee00 comp rx=0 tx=0).stop 2023-10-19T13:00:14.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.014+0000 7fe15b4e1640 1 -- 172.21.15.57:0/2494206317 shutdown_connections 2023-10-19T13:00:14.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.014+0000 7fe15b4e1640 1 --2- 172.21.15.57:0/2494206317 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7fe14006af60 0x7fe14006d420 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:14.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.014+0000 7fe15b4e1640 1 --2- 172.21.15.57:0/2494206317 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe15415b2a0 0x7fe15406a1f0 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:14.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.014+0000 7fe15b4e1640 1 --2- 172.21.15.57:0/2494206317 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe1540cf340 0x7fe1540b4900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:14.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.014+0000 7fe15b4e1640 1 --2- 172.21.15.57:0/2494206317 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe1540ceb40 0x7fe154151150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:14.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.014+0000 7fe15b4e1640 1 -- 172.21.15.57:0/2494206317 >> 172.21.15.57:0/2494206317 conn(0x7fe1540d0d60 msgr2=0x7fe154159100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:14.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.014+0000 7fe15b4e1640 1 -- 172.21.15.57:0/2494206317 shutdown_connections 2023-10-19T13:00:14.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.014+0000 7fe15b4e1640 1 -- 172.21.15.57:0/2494206317 wait complete. 2023-10-19T13:00:14.027 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-19T12:44:07.371186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-19T13:00:14.028 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-19T13:00:14.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.137+0000 7ff3cef9e640 1 -- 172.21.15.57:0/1956223102 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff3c80cb980 msgr2=0x7ff3c813db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:14.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.137+0000 7ff3cef9e640 1 --2- 172.21.15.57:0/1956223102 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff3c80cb980 0x7ff3c813db80 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc005fd0 tx=0x7ff3bc02be10 comp rx=0 tx=0).stop 2023-10-19T13:00:14.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.137+0000 7ff3cef9e640 1 -- 172.21.15.57:0/1956223102 shutdown_connections 2023-10-19T13:00:14.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.137+0000 7ff3cef9e640 1 --2- 172.21.15.57:0/1956223102 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff3c8070350 0x7ff3c80707c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:14.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.137+0000 7ff3cef9e640 1 --2- 172.21.15.57:0/1956223102 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff3c80cb980 0x7ff3c813db80 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:14.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.137+0000 7ff3cef9e640 1 -- 172.21.15.57:0/1956223102 >> 172.21.15.57:0/1956223102 conn(0x7ff3c8069930 msgr2=0x7ff3c8069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:14.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.137+0000 7ff3cef9e640 1 -- 172.21.15.57:0/1956223102 shutdown_connections 2023-10-19T13:00:14.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.137+0000 7ff3cef9e640 1 -- 172.21.15.57:0/1956223102 wait complete. 2023-10-19T13:00:14.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.138+0000 7ff3cef9e640 1 Processor -- start 2023-10-19T13:00:14.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.138+0000 7ff3cef9e640 1 -- start start 2023-10-19T13:00:14.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.139+0000 7ff3cef9e640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff3c8070350 0x7ff3c8137bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:14.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.139+0000 7ff3cef9e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff3c80cb980 0x7ff3c81f2a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:14.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.139+0000 7ff3cef9e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff3c81553c0 0x7ff3c81f2f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:14.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.139+0000 7ff3cef9e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c8138510 con 0x7ff3c80cb980 2023-10-19T13:00:14.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.139+0000 7ff3cef9e640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c81531d0 con 0x7ff3c81553c0 2023-10-19T13:00:14.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.139+0000 7ff3cef9e640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c81528a0 con 0x7ff3c8070350 2023-10-19T13:00:14.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.139+0000 7ff3c7fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff3c80cb980 0x7ff3c81f2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:14.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.139+0000 7ff3c7fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff3c80cb980 0x7ff3c81f2a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40002/0 (socket says 172.21.15.57:40002) 2023-10-19T13:00:14.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.139+0000 7ff3c7fff640 1 -- 172.21.15.57:0/1326716916 learned_addr learned my addr 172.21.15.57:0/1326716916 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:14.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.139+0000 7ff3ccd3a640 1 --2- 172.21.15.57:0/1326716916 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff3c8070350 0x7ff3c8137bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:14.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.139+0000 7ff3cd53b640 1 --2- 172.21.15.57:0/1326716916 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff3c81553c0 0x7ff3c81f2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:14.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.139+0000 7ff3c7fff640 1 -- 172.21.15.57:0/1326716916 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff3c8070350 msgr2=0x7ff3c8137bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:14.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.139+0000 7ff3c7fff640 1 --2- 172.21.15.57:0/1326716916 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff3c8070350 0x7ff3c8137bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:14.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.139+0000 7ff3c7fff640 1 -- 172.21.15.57:0/1326716916 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff3c81553c0 msgr2=0x7ff3c81f2f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:14.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.139+0000 7ff3c7fff640 1 --2- 172.21.15.57:0/1326716916 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff3c81553c0 0x7ff3c81f2f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:14.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.139+0000 7ff3c7fff640 1 -- 172.21.15.57:0/1326716916 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3bc009560 con 0x7ff3c80cb980 2023-10-19T13:00:14.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.139+0000 7ff3c7fff640 1 --2- 172.21.15.57:0/1326716916 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff3c80cb980 0x7ff3c81f2a00 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7ff3b400c400 tx=0x7ff3b400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:14.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.140+0000 7ff3c57fa640 1 -- 172.21.15.57:0/1326716916 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3b40052d0 con 0x7ff3c80cb980 2023-10-19T13:00:14.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.140+0000 7ff3c57fa640 1 -- 172.21.15.57:0/1326716916 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7ff3b40035f0 con 0x7ff3c80cb980 2023-10-19T13:00:14.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.140+0000 7ff3c57fa640 1 -- 172.21.15.57:0/1326716916 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3b4003a00 con 0x7ff3c80cb980 2023-10-19T13:00:14.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.140+0000 7ff3cef9e640 1 -- 172.21.15.57:0/1326716916 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3c8152b80 con 0x7ff3c80cb980 2023-10-19T13:00:14.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.140+0000 7ff3cef9e640 1 -- 172.21.15.57:0/1326716916 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3c814e640 con 0x7ff3c80cb980 2023-10-19T13:00:14.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.141+0000 7ff3cef9e640 1 -- 172.21.15.57:0/1326716916 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3c8071290 con 0x7ff3c80cb980 2023-10-19T13:00:14.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.141+0000 7ff3c57fa640 1 -- 172.21.15.57:0/1326716916 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 32) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff3b4003b60 con 0x7ff3c80cb980 2023-10-19T13:00:14.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.142+0000 7ff3c57fa640 1 --2- 172.21.15.57:0/1326716916 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7ff3ac06af60 0x7ff3ac06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:14.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.142+0000 7ff3c57fa640 1 -- 172.21.15.57:0/1326716916 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 5979+0+0 (secure 0 0 0) 0x7ff3b4088ff0 con 0x7ff3c80cb980 2023-10-19T13:00:14.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.142+0000 7ff3ccd3a640 1 --2- 172.21.15.57:0/1326716916 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7ff3ac06af60 0x7ff3ac06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:14.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.143+0000 7ff3ccd3a640 1 --2- 172.21.15.57:0/1326716916 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7ff3ac06af60 0x7ff3ac06d420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc035000 tx=0x7ff3bc004730 comp rx=0 tx=0).ready entity=mgr.6256 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:14.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.146+0000 7ff3c57fa640 1 -- 172.21.15.57:0/1326716916 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff3b4053f20 con 0x7ff3c80cb980 2023-10-19T13:00:14.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.320+0000 7ff3cef9e640 1 -- 172.21.15.57:0/1326716916 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff3c8153340 con 0x7ff3c80cb980 2023-10-19T13:00:14.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.320+0000 7ff3c57fa640 1 -- 172.21.15.57:0/1326716916 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v79) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff3b40170a0 con 0x7ff3c80cb980 2023-10-19T13:00:14.323 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-19T13:00:14.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.322+0000 7ff3cef9e640 1 -- 172.21.15.57:0/1326716916 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7ff3ac06af60 msgr2=0x7ff3ac06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:14.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.322+0000 7ff3cef9e640 1 --2- 172.21.15.57:0/1326716916 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7ff3ac06af60 0x7ff3ac06d420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc035000 tx=0x7ff3bc004730 comp rx=0 tx=0).stop 2023-10-19T13:00:14.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.322+0000 7ff3cef9e640 1 -- 172.21.15.57:0/1326716916 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff3c80cb980 msgr2=0x7ff3c81f2a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:14.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.322+0000 7ff3cef9e640 1 --2- 172.21.15.57:0/1326716916 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff3c80cb980 0x7ff3c81f2a00 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7ff3b400c400 tx=0x7ff3b400c8c0 comp rx=0 tx=0).stop 2023-10-19T13:00:14.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.322+0000 7ff3cef9e640 1 -- 172.21.15.57:0/1326716916 shutdown_connections 2023-10-19T13:00:14.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.322+0000 7ff3cef9e640 1 --2- 172.21.15.57:0/1326716916 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7ff3ac06af60 0x7ff3ac06d420 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:14.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.322+0000 7ff3cef9e640 1 --2- 172.21.15.57:0/1326716916 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff3c81553c0 0x7ff3c81f2f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:14.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.322+0000 7ff3cef9e640 1 --2- 172.21.15.57:0/1326716916 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff3c80cb980 0x7ff3c81f2a00 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:14.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.322+0000 7ff3cef9e640 1 --2- 172.21.15.57:0/1326716916 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff3c8070350 0x7ff3c8137bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:14.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.322+0000 7ff3cef9e640 1 -- 172.21.15.57:0/1326716916 >> 172.21.15.57:0/1326716916 conn(0x7ff3c8069930 msgr2=0x7ff3c8158100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:14.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.322+0000 7ff3cef9e640 1 -- 172.21.15.57:0/1326716916 shutdown_connections 2023-10-19T13:00:14.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.322+0000 7ff3cef9e640 1 -- 172.21.15.57:0/1326716916 wait complete. 2023-10-19T13:00:14.335 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule' 2023-10-19T13:00:14.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.442+0000 7f8811139640 1 -- 172.21.15.57:0/1346903472 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f880c0c9dc0 msgr2=0x7f880c06b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:14.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.442+0000 7f8811139640 1 --2- 172.21.15.57:0/1346903472 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f880c0c9dc0 0x7f880c06b2e0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f87f4009830 tx=0x7f87f402bdf0 comp rx=0 tx=0).stop 2023-10-19T13:00:14.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.442+0000 7f8811139640 1 -- 172.21.15.57:0/1346903472 shutdown_connections 2023-10-19T13:00:14.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.442+0000 7f8811139640 1 --2- 172.21.15.57:0/1346903472 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f880c0c9dc0 0x7f880c06b2e0 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:14.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.442+0000 7f8811139640 1 -- 172.21.15.57:0/1346903472 >> 172.21.15.57:0/1346903472 conn(0x7f880c069930 msgr2=0x7f880c069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:14.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.442+0000 7f8811139640 1 -- 172.21.15.57:0/1346903472 shutdown_connections 2023-10-19T13:00:14.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.442+0000 7f8811139640 1 -- 172.21.15.57:0/1346903472 wait complete. 2023-10-19T13:00:14.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.443+0000 7f8811139640 1 Processor -- start 2023-10-19T13:00:14.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.443+0000 7f8811139640 1 -- start start 2023-10-19T13:00:14.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.444+0000 7f8811139640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f880c06af00 0x7f880c0acbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:14.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.444+0000 7f8811139640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f880c0ab500 0x7f880c0b0240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:14.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.444+0000 7f8811139640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f880c0c9dc0 0x7f880c13a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:14.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.444+0000 7f8811139640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f880c0b0780 con 0x7f880c0c9dc0 2023-10-19T13:00:14.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.444+0000 7f8811139640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f880c13ac50 con 0x7f880c06af00 2023-10-19T13:00:14.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.444+0000 7f8811139640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f880c0a8820 con 0x7f880c0ab500 2023-10-19T13:00:14.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.444+0000 7f880b59e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f880c0c9dc0 0x7f880c13a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:14.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.444+0000 7f880b59e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f880c0c9dc0 0x7f880c13a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40010/0 (socket says 172.21.15.57:40010) 2023-10-19T13:00:14.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.444+0000 7f880b59e640 1 -- 172.21.15.57:0/3525514445 learned_addr learned my addr 172.21.15.57:0/3525514445 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:14.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.444+0000 7f880ad9d640 1 --2- 172.21.15.57:0/3525514445 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f880c06af00 0x7f880c0acbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:14.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.444+0000 7f880259c640 1 --2- 172.21.15.57:0/3525514445 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f880c0ab500 0x7f880c0b0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:14.452 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.444+0000 7f880b59e640 1 -- 172.21.15.57:0/3525514445 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f880c0ab500 msgr2=0x7f880c0b0240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:14.452 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.444+0000 7f880b59e640 1 --2- 172.21.15.57:0/3525514445 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f880c0ab500 0x7f880c0b0240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:14.452 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.444+0000 7f880b59e640 1 -- 172.21.15.57:0/3525514445 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f880c06af00 msgr2=0x7f880c0acbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:14.452 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.444+0000 7f880b59e640 1 --2- 172.21.15.57:0/3525514445 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f880c06af00 0x7f880c0acbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:14.452 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.444+0000 7f880b59e640 1 -- 172.21.15.57:0/3525514445 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87f4009490 con 0x7f880c0c9dc0 2023-10-19T13:00:14.452 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.444+0000 7f880b59e640 1 --2- 172.21.15.57:0/3525514445 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f880c0c9dc0 0x7f880c13a710 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f880400cb40 tx=0x7f880400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:14.453 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.445+0000 7f8803fff640 1 -- 172.21.15.57:0/3525514445 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f880400aca0 con 0x7f880c0c9dc0 2023-10-19T13:00:14.453 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.445+0000 7f8803fff640 1 -- 172.21.15.57:0/3525514445 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f8804012420 con 0x7f880c0c9dc0 2023-10-19T13:00:14.453 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.445+0000 7f8811139640 1 -- 172.21.15.57:0/3525514445 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f880c0b1240 con 0x7f880c0c9dc0 2023-10-19T13:00:14.453 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.445+0000 7f8803fff640 1 -- 172.21.15.57:0/3525514445 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8804012810 con 0x7f880c0c9dc0 2023-10-19T13:00:14.453 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.445+0000 7f8811139640 1 -- 172.21.15.57:0/3525514445 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f880c0b0c80 con 0x7f880c0c9dc0 2023-10-19T13:00:14.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.446+0000 7f8811139640 1 -- 172.21.15.57:0/3525514445 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f880c0a85f0 con 0x7f880c0c9dc0 2023-10-19T13:00:14.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.446+0000 7f8803fff640 1 -- 172.21.15.57:0/3525514445 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 32) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f880401b400 con 0x7f880c0c9dc0 2023-10-19T13:00:14.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.447+0000 7f8803fff640 1 --2- 172.21.15.57:0/3525514445 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f87ec06af90 0x7f87ec06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:14.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.447+0000 7f8803fff640 1 -- 172.21.15.57:0/3525514445 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 5979+0+0 (secure 0 0 0) 0x7f880408a270 con 0x7f880c0c9dc0 2023-10-19T13:00:14.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.447+0000 7f880ad9d640 1 --2- 172.21.15.57:0/3525514445 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f87ec06af90 0x7f87ec06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:14.456 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.448+0000 7f880ad9d640 1 --2- 172.21.15.57:0/3525514445 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f87ec06af90 0x7f87ec06d450 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f87fc000f60 tx=0x7f87fc009040 comp rx=0 tx=0).ready entity=mgr.6256 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:14.456 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.451+0000 7f8803fff640 1 -- 172.21.15.57:0/3525514445 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f880c0a85f0 con 0x7f880c0c9dc0 2023-10-19T13:00:14.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.693+0000 7f8811139640 1 -- 172.21.15.57:0/3525514445 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule"]} v 0) v1 -- 0x7f880c0a85f0 con 0x7f880c0c9dc0 2023-10-19T13:00:14.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.701+0000 7f8803fff640 1 -- 172.21.15.57:0/3525514445 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule"]}]=0 v583) v1 ==== 159+0+0 (secure 0 0 0) 0x7f880c0a85f0 con 0x7f880c0c9dc0 2023-10-19T13:00:14.707 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.704+0000 7f8811139640 1 -- 172.21.15.57:0/3525514445 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f87ec06af90 msgr2=0x7f87ec06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:14.707 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.704+0000 7f8811139640 1 --2- 172.21.15.57:0/3525514445 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f87ec06af90 0x7f87ec06d450 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f87fc000f60 tx=0x7f87fc009040 comp rx=0 tx=0).stop 2023-10-19T13:00:14.707 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.704+0000 7f8811139640 1 -- 172.21.15.57:0/3525514445 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f880c0c9dc0 msgr2=0x7f880c13a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:14.707 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.704+0000 7f8811139640 1 --2- 172.21.15.57:0/3525514445 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f880c0c9dc0 0x7f880c13a710 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f880400cb40 tx=0x7f880400a060 comp rx=0 tx=0).stop 2023-10-19T13:00:14.707 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.704+0000 7f8811139640 1 -- 172.21.15.57:0/3525514445 shutdown_connections 2023-10-19T13:00:14.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.704+0000 7f8811139640 1 --2- 172.21.15.57:0/3525514445 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f87ec06af90 0x7f87ec06d450 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:14.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.704+0000 7f8811139640 1 --2- 172.21.15.57:0/3525514445 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f880c0c9dc0 0x7f880c13a710 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:14.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.704+0000 7f8811139640 1 --2- 172.21.15.57:0/3525514445 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f880c0ab500 0x7f880c0b0240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:14.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.704+0000 7f8811139640 1 --2- 172.21.15.57:0/3525514445 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f880c06af00 0x7f880c0acbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:14.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.704+0000 7f8811139640 1 -- 172.21.15.57:0/3525514445 >> 172.21.15.57:0/3525514445 conn(0x7f880c069930 msgr2=0x7f880c15a3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:14.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.704+0000 7f8811139640 1 -- 172.21.15.57:0/3525514445 shutdown_connections 2023-10-19T13:00:14.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.704+0000 7f8811139640 1 -- 172.21.15.57:0/3525514445 wait complete. 2023-10-19T13:00:14.717 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2023-10-19T13:00:14.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.824+0000 7f9eae632640 1 -- 172.21.15.57:0/2249720550 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9e90004d00 msgr2=0x7f9e90005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:14.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.824+0000 7f9eae632640 1 --2- 172.21.15.57:0/2249720550 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9e90004d00 0x7f9e90005180 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f9ea000c640 tx=0x7f9e900057f0 comp rx=0 tx=0).stop 2023-10-19T13:00:14.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.824+0000 7f9eacbcf640 1 -- 172.21.15.57:0/2249720550 reap_dead start 2023-10-19T13:00:14.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.824+0000 7f9eae632640 1 -- 172.21.15.57:0/2249720550 shutdown_connections 2023-10-19T13:00:14.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.824+0000 7f9eae632640 1 -- 172.21.15.57:0/2249720550 >> 172.21.15.57:0/2249720550 conn(0x7f9ea80af7a0 msgr2=0x7f9ea80afb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:14.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.824+0000 7f9eae632640 1 -- 172.21.15.57:0/2249720550 shutdown_connections 2023-10-19T13:00:14.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.824+0000 7f9eae632640 1 -- 172.21.15.57:0/2249720550 wait complete. 2023-10-19T13:00:14.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.826+0000 7f9eae632640 1 Processor -- start 2023-10-19T13:00:14.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.826+0000 7f9eae632640 1 -- start start 2023-10-19T13:00:14.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.826+0000 7f9eae632640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9e90004d00 0x7f9ea81502b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:14.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.827+0000 7f9eae632640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9ea8140c80 0x7f9ea814e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:14.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.827+0000 7f9eae632640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9ea80b2330 0x7f9ea80b2790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:14.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.827+0000 7f9eae632640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea81515f0 con 0x7f9ea80b2330 2023-10-19T13:00:14.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.827+0000 7f9eae632640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea8141100 con 0x7f9e90004d00 2023-10-19T13:00:14.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.827+0000 7f9eae632640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea8150990 con 0x7f9ea8140c80 2023-10-19T13:00:14.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.827+0000 7f9eacbcf640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9ea80b2330 0x7f9ea80b2790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:14.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.827+0000 7f9eacbcf640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9ea80b2330 0x7f9ea80b2790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40024/0 (socket says 172.21.15.57:40024) 2023-10-19T13:00:14.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.827+0000 7f9eacbcf640 1 -- 172.21.15.57:0/3645262287 learned_addr learned my addr 172.21.15.57:0/3645262287 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:14.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.827+0000 7f9ea7fff640 1 --2- 172.21.15.57:0/3645262287 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9e90004d00 0x7f9ea81502b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:14.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.827+0000 7f9ea77fe640 1 --2- 172.21.15.57:0/3645262287 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9ea8140c80 0x7f9ea814e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:14.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.827+0000 7f9eacbcf640 1 -- 172.21.15.57:0/3645262287 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9ea8140c80 msgr2=0x7f9ea814e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:14.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.827+0000 7f9eacbcf640 1 --2- 172.21.15.57:0/3645262287 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9ea8140c80 0x7f9ea814e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:14.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.827+0000 7f9eacbcf640 1 -- 172.21.15.57:0/3645262287 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9e90004d00 msgr2=0x7f9ea81502b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:14.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.827+0000 7f9eacbcf640 1 --2- 172.21.15.57:0/3645262287 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9e90004d00 0x7f9ea81502b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:14.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.827+0000 7f9eacbcf640 1 -- 172.21.15.57:0/3645262287 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ea8150c70 con 0x7f9ea80b2330 2023-10-19T13:00:14.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.827+0000 7f9eacbcf640 1 --2- 172.21.15.57:0/3645262287 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9ea80b2330 0x7f9ea80b2790 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f9ea000c640 tx=0x7f9ea0002170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:14.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.827+0000 7f9ea4ff9640 1 -- 172.21.15.57:0/3645262287 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ea00018b0 con 0x7f9ea80b2330 2023-10-19T13:00:14.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.827+0000 7f9ea4ff9640 1 -- 172.21.15.57:0/3645262287 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f9ea0003970 con 0x7f9ea80b2330 2023-10-19T13:00:14.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.827+0000 7f9eae632640 1 -- 172.21.15.57:0/3645262287 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ea8150f50 con 0x7f9ea80b2330 2023-10-19T13:00:14.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.827+0000 7f9ea4ff9640 1 -- 172.21.15.57:0/3645262287 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ea0003d80 con 0x7f9ea80b2330 2023-10-19T13:00:14.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.828+0000 7f9eae632640 1 -- 172.21.15.57:0/3645262287 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ea80b2cd0 con 0x7f9ea80b2330 2023-10-19T13:00:14.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.828+0000 7f9eae632640 1 -- 172.21.15.57:0/3645262287 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ea81517a0 con 0x7f9ea80b2330 2023-10-19T13:00:14.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.829+0000 7f9ea4ff9640 1 -- 172.21.15.57:0/3645262287 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 32) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f9ea0031030 con 0x7f9ea80b2330 2023-10-19T13:00:14.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.829+0000 7f9ea4ff9640 1 --2- 172.21.15.57:0/3645262287 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f9e9406af90 0x7f9e9406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:14.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.829+0000 7f9ea4ff9640 1 -- 172.21.15.57:0/3645262287 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 5979+0+0 (secure 0 0 0) 0x7f9ea00aa6b0 con 0x7f9ea80b2330 2023-10-19T13:00:14.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.830+0000 7f9ea7fff640 1 --2- 172.21.15.57:0/3645262287 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f9e9406af90 0x7f9e9406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:14.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.830+0000 7f9ea7fff640 1 --2- 172.21.15.57:0/3645262287 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f9e9406af90 0x7f9e9406d450 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9ea81f0520 tx=0x7f9e9c005f90 comp rx=0 tx=0).ready entity=mgr.6256 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:14.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:14.834+0000 7f9ea4ff9640 1 -- 172.21.15.57:0/3645262287 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9ea81517a0 con 0x7f9ea80b2330 2023-10-19T13:00:15.011 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.008+0000 7f9eae632640 1 -- 172.21.15.57:0/3645262287 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "config dump"} v 0) v1 -- 0x7f9ea80690e0 con 0x7f9ea80b2330 2023-10-19T13:00:15.012 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.009+0000 7f9ea4ff9640 1 -- 172.21.15.57:0/3645262287 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v38) v1 ==== 59+0+2220 (secure 0 0 0) 0x7f9ea0043080 con 0x7f9ea80b2330 2023-10-19T13:00:15.012 INFO:teuthology.orchestra.run.smithi057.stdout:WHO MASK LEVEL OPTION VALUE RO 2023-10-19T13:00:15.012 INFO:teuthology.orchestra.run.smithi057.stdout:mon advanced mon_op_complaint_time 120 2023-10-19T13:00:15.013 INFO:teuthology.orchestra.run.smithi057.stdout:osd advanced osd_op_complaint_time 180.000000 2023-10-19T13:00:15.013 INFO:teuthology.orchestra.run.smithi057.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 4775.995373 2023-10-19T13:00:15.013 INFO:teuthology.orchestra.run.smithi057.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 4208.118410 2023-10-19T13:00:15.013 INFO:teuthology.orchestra.run.smithi057.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 4541.123030 2023-10-19T13:00:15.013 INFO:teuthology.orchestra.run.smithi057.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 4877.592206 2023-10-19T13:00:15.014 INFO:teuthology.orchestra.run.smithi057.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 5019.748032 2023-10-19T13:00:15.014 INFO:teuthology.orchestra.run.smithi057.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 5000.074376 2023-10-19T13:00:15.014 INFO:teuthology.orchestra.run.smithi057.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 5153.479439 2023-10-19T13:00:15.014 INFO:teuthology.orchestra.run.smithi057.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 4977.363845 2023-10-19T13:00:15.014 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced debug_mds 20/20 2023-10-19T13:00:15.015 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced debug_mds_balancer 20/20 2023-10-19T13:00:15.015 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced debug_ms 1/1 2023-10-19T13:00:15.015 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced mds_bal_fragment_size_max 10000 2023-10-19T13:00:15.015 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced mds_bal_merge_size 5 2023-10-19T13:00:15.015 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced mds_bal_split_bits 3 2023-10-19T13:00:15.016 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced mds_bal_split_size 100 2023-10-19T13:00:15.016 INFO:teuthology.orchestra.run.smithi057.stdout:mds dev mds_debug_frag true 2023-10-19T13:00:15.016 INFO:teuthology.orchestra.run.smithi057.stdout:mds dev mds_debug_scatterstat true 2023-10-19T13:00:15.016 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced mds_op_complaint_time 180.000000 2023-10-19T13:00:15.016 INFO:teuthology.orchestra.run.smithi057.stdout:mds dev mds_verify_scatter true 2023-10-19T13:00:15.016 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced osd_op_complaint_time 180.000000 2023-10-19T13:00:15.017 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced rados_mon_op_timeout 900 2023-10-19T13:00:15.017 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced rados_osd_op_timeout 900 2023-10-19T13:00:15.017 INFO:teuthology.orchestra.run.smithi057.stdout:client advanced client_mount_timeout 600 2023-10-19T13:00:15.017 INFO:teuthology.orchestra.run.smithi057.stdout:client advanced debug_client 20/20 2023-10-19T13:00:15.017 INFO:teuthology.orchestra.run.smithi057.stdout:client advanced debug_ms 1/1 2023-10-19T13:00:15.017 INFO:teuthology.orchestra.run.smithi057.stdout:client advanced rados_mon_op_timeout 900 2023-10-19T13:00:15.018 INFO:teuthology.orchestra.run.smithi057.stdout:client advanced rados_osd_op_timeout 900 2023-10-19T13:00:15.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.010+0000 7f9eae632640 1 -- 172.21.15.57:0/3645262287 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f9e9406af90 msgr2=0x7f9e9406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:15.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.010+0000 7f9eae632640 1 --2- 172.21.15.57:0/3645262287 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f9e9406af90 0x7f9e9406d450 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9ea81f0520 tx=0x7f9e9c005f90 comp rx=0 tx=0).stop 2023-10-19T13:00:15.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.010+0000 7f9eae632640 1 -- 172.21.15.57:0/3645262287 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9ea80b2330 msgr2=0x7f9ea80b2790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:15.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.010+0000 7f9eae632640 1 --2- 172.21.15.57:0/3645262287 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9ea80b2330 0x7f9ea80b2790 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f9ea000c640 tx=0x7f9ea0002170 comp rx=0 tx=0).stop 2023-10-19T13:00:15.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.010+0000 7f9eae632640 1 -- 172.21.15.57:0/3645262287 shutdown_connections 2023-10-19T13:00:15.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.010+0000 7f9eae632640 1 --2- 172.21.15.57:0/3645262287 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f9e9406af90 0x7f9e9406d450 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:15.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.010+0000 7f9eae632640 1 --2- 172.21.15.57:0/3645262287 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9ea80b2330 0x7f9ea80b2790 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:15.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.010+0000 7f9eae632640 1 --2- 172.21.15.57:0/3645262287 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9ea8140c80 0x7f9ea814e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:15.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.010+0000 7f9eae632640 1 --2- 172.21.15.57:0/3645262287 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9e90004d00 0x7f9ea81502b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:15.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.011+0000 7f9eae632640 1 -- 172.21.15.57:0/3645262287 >> 172.21.15.57:0/3645262287 conn(0x7f9ea80af7a0 msgr2=0x7f9ea8159b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:15.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.011+0000 7f9eae632640 1 -- 172.21.15.57:0/3645262287 shutdown_connections 2023-10-19T13:00:15.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.011+0000 7f9eae632640 1 -- 172.21.15.57:0/3645262287 wait complete. 2023-10-19T13:00:15.024 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-19T13:00:15.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.133+0000 7f7ec5d6c640 1 -- 172.21.15.57:0/752057952 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ec00cd1c0 msgr2=0x7f7ec015c0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:15.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.133+0000 7f7ec5d6c640 1 --2- 172.21.15.57:0/752057952 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ec00cd1c0 0x7f7ec015c0e0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f7eb4009830 tx=0x7f7eb402bd60 comp rx=0 tx=0).stop 2023-10-19T13:00:15.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.133+0000 7f7ec5d6c640 1 -- 172.21.15.57:0/752057952 shutdown_connections 2023-10-19T13:00:15.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.133+0000 7f7ec5d6c640 1 --2- 172.21.15.57:0/752057952 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ec00cd1c0 0x7f7ec015c0e0 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:15.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.133+0000 7f7ec5d6c640 1 --2- 172.21.15.57:0/752057952 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7ec0158910 0x7f7ec0158cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:15.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.133+0000 7f7ec5d6c640 1 -- 172.21.15.57:0/752057952 >> 172.21.15.57:0/752057952 conn(0x7f7ec00af7a0 msgr2=0x7f7ec00afb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:15.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.133+0000 7f7ec5d6c640 1 -- 172.21.15.57:0/752057952 shutdown_connections 2023-10-19T13:00:15.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.133+0000 7f7ec5d6c640 1 -- 172.21.15.57:0/752057952 wait complete. 2023-10-19T13:00:15.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.134+0000 7f7ec5d6c640 1 Processor -- start 2023-10-19T13:00:15.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.134+0000 7f7ec5d6c640 1 -- start start 2023-10-19T13:00:15.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.135+0000 7f7ec5d6c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7ec00cd1c0 0x7f7ec01f35c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:15.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.135+0000 7f7ec5d6c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7ec00cd9e0 0x7f7ec01eed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:15.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.135+0000 7f7ec5d6c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ec0158910 0x7f7ec013dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:15.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.135+0000 7f7ec5d6c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec014e7f0 con 0x7f7ec0158910 2023-10-19T13:00:15.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.135+0000 7f7ec5d6c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec013e5d0 con 0x7f7ec00cd9e0 2023-10-19T13:00:15.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.135+0000 7f7ec5d6c640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec013e8b0 con 0x7f7ec00cd1c0 2023-10-19T13:00:15.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.135+0000 7f7ebffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ec0158910 0x7f7ec013dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:15.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.135+0000 7f7ebffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ec0158910 0x7f7ec013dd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40048/0 (socket says 172.21.15.57:40048) 2023-10-19T13:00:15.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.135+0000 7f7ebffff640 1 -- 172.21.15.57:0/2903978204 learned_addr learned my addr 172.21.15.57:0/2903978204 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:15.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.135+0000 7f7ebf7fe640 1 --2- 172.21.15.57:0/2903978204 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7ec00cd1c0 0x7f7ec01f35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:15.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.135+0000 7f7ebeffd640 1 --2- 172.21.15.57:0/2903978204 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7ec00cd9e0 0x7f7ec01eed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:15.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.136+0000 7f7ebffff640 1 -- 172.21.15.57:0/2903978204 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7ec00cd1c0 msgr2=0x7f7ec01f35c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:15.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.136+0000 7f7ebffff640 1 --2- 172.21.15.57:0/2903978204 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7ec00cd1c0 0x7f7ec01f35c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:15.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.136+0000 7f7ebffff640 1 -- 172.21.15.57:0/2903978204 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7ec00cd9e0 msgr2=0x7f7ec01eed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:15.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.136+0000 7f7ebffff640 1 --2- 172.21.15.57:0/2903978204 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7ec00cd9e0 0x7f7ec01eed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:15.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.136+0000 7f7ebffff640 1 -- 172.21.15.57:0/2903978204 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7eb4009490 con 0x7f7ec0158910 2023-10-19T13:00:15.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.136+0000 7f7ebffff640 1 --2- 172.21.15.57:0/2903978204 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ec0158910 0x7f7ec013dd20 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f7eb800cb40 tx=0x7f7eb800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:15.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.136+0000 7f7ea7fff640 1 -- 172.21.15.57:0/2903978204 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eb800aca0 con 0x7f7ec0158910 2023-10-19T13:00:15.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.136+0000 7f7ea7fff640 1 -- 172.21.15.57:0/2903978204 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f7eb8012420 con 0x7f7ec0158910 2023-10-19T13:00:15.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.136+0000 7f7ea7fff640 1 -- 172.21.15.57:0/2903978204 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eb8012850 con 0x7f7ec0158910 2023-10-19T13:00:15.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.136+0000 7f7ec5d6c640 1 -- 172.21.15.57:0/2903978204 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ec013eb90 con 0x7f7ec0158910 2023-10-19T13:00:15.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.136+0000 7f7ec5d6c640 1 -- 172.21.15.57:0/2903978204 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ec014e410 con 0x7f7ec0158910 2023-10-19T13:00:15.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.137+0000 7f7ec5d6c640 1 -- 172.21.15.57:0/2903978204 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ec0070f40 con 0x7f7ec0158910 2023-10-19T13:00:15.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.138+0000 7f7ea7fff640 1 -- 172.21.15.57:0/2903978204 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 32) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f7eb801a3b0 con 0x7f7ec0158910 2023-10-19T13:00:15.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.138+0000 7f7ea7fff640 1 --2- 172.21.15.57:0/2903978204 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f7eac06af90 0x7f7eac06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:15.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.138+0000 7f7ea7fff640 1 -- 172.21.15.57:0/2903978204 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 5979+0+0 (secure 0 0 0) 0x7f7eb8089fa0 con 0x7f7ec0158910 2023-10-19T13:00:15.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.138+0000 7f7ebf7fe640 1 --2- 172.21.15.57:0/2903978204 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f7eac06af90 0x7f7eac06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:15.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.139+0000 7f7ebf7fe640 1 --2- 172.21.15.57:0/2903978204 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f7eac06af90 0x7f7eac06d450 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7eb0005fd0 tx=0x7f7eb0005850 comp rx=0 tx=0).ready entity=mgr.6256 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:15.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.142+0000 7f7ea7fff640 1 -- 172.21.15.57:0/2903978204 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7eb80bcca0 con 0x7f7ec0158910 2023-10-19T13:00:15.321 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.318+0000 7f7ec5d6c640 1 -- 172.21.15.57:0/2903978204 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f7ec00690e0 con 0x7f7ec0158910 2023-10-19T13:00:15.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.323+0000 7f7ea7fff640 1 -- 172.21.15.57:0/2903978204 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v39)=0 v39) v1 ==== 122+0+0 (secure 0 0 0) 0x7f7eb8054ec0 con 0x7f7ec0158910 2023-10-19T13:00:15.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.325+0000 7f7ec5d6c640 1 -- 172.21.15.57:0/2903978204 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f7eac06af90 msgr2=0x7f7eac06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:15.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.325+0000 7f7ec5d6c640 1 --2- 172.21.15.57:0/2903978204 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f7eac06af90 0x7f7eac06d450 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7eb0005fd0 tx=0x7f7eb0005850 comp rx=0 tx=0).stop 2023-10-19T13:00:15.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.325+0000 7f7ec5d6c640 1 -- 172.21.15.57:0/2903978204 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ec0158910 msgr2=0x7f7ec013dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:15.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.325+0000 7f7ec5d6c640 1 --2- 172.21.15.57:0/2903978204 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ec0158910 0x7f7ec013dd20 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f7eb800cb40 tx=0x7f7eb800a060 comp rx=0 tx=0).stop 2023-10-19T13:00:15.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.325+0000 7f7ec5d6c640 1 -- 172.21.15.57:0/2903978204 shutdown_connections 2023-10-19T13:00:15.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.325+0000 7f7ec5d6c640 1 --2- 172.21.15.57:0/2903978204 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f7eac06af90 0x7f7eac06d450 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:15.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.325+0000 7f7ec5d6c640 1 --2- 172.21.15.57:0/2903978204 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ec0158910 0x7f7ec013dd20 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:15.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.325+0000 7f7ec5d6c640 1 --2- 172.21.15.57:0/2903978204 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7ec00cd9e0 0x7f7ec01eed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:15.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.325+0000 7f7ec5d6c640 1 --2- 172.21.15.57:0/2903978204 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7ec00cd1c0 0x7f7ec01f35c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:15.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.325+0000 7f7ec5d6c640 1 -- 172.21.15.57:0/2903978204 >> 172.21.15.57:0/2903978204 conn(0x7f7ec00af7a0 msgr2=0x7f7ec006db10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:15.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.325+0000 7f7ec5d6c640 1 -- 172.21.15.57:0/2903978204 shutdown_connections 2023-10-19T13:00:15.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.325+0000 7f7ec5d6c640 1 -- 172.21.15.57:0/2903978204 wait complete. 2023-10-19T13:00:15.338 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-19T13:00:15.339 DEBUG:teuthology.orchestra.run.smithi057:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-19T13:00:15.395 DEBUG:teuthology.orchestra.run.smithi179:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-19T13:00:15.441 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-19T13:00:15.441 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T13:00:15.442 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T13:00:15.457 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055614,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=358 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248260k,nr_inodes=812065,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-19T13:00:15.458 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-19T13:00:15.458 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T13:00:15.475 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054833,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=284 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3247632k,nr_inodes=811908,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-19T13:00:15.476 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T13:00:15.476 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T13:00:15.512 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055614,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=358 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248260k,nr_inodes=812065,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-19T13:00:15.513 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-19T13:00:15.513 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T13:00:15.532 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054833,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=284 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3247632k,nr_inodes=811908,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-19T13:00:15.532 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-19T13:00:15.533 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T13:00:15.662 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.659+0000 7efde024f640 1 -- 172.21.15.57:0/550754257 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efdc0003f10 msgr2=0x7efdc00047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:15.662 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.659+0000 7efde024f640 1 --2- 172.21.15.57:0/550754257 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efdc0003f10 0x7efdc00047c0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7efdcc009b50 tx=0x7efdc00057f0 comp rx=0 tx=0).stop 2023-10-19T13:00:15.662 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.659+0000 7efdde7ec640 1 -- 172.21.15.57:0/550754257 reap_dead start 2023-10-19T13:00:15.663 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.659+0000 7efde024f640 1 -- 172.21.15.57:0/550754257 shutdown_connections 2023-10-19T13:00:15.663 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.659+0000 7efde024f640 1 -- 172.21.15.57:0/550754257 >> 172.21.15.57:0/550754257 conn(0x7efdd8069930 msgr2=0x7efdd8069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:15.663 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.659+0000 7efde024f640 1 -- 172.21.15.57:0/550754257 shutdown_connections 2023-10-19T13:00:15.663 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.659+0000 7efde024f640 1 -- 172.21.15.57:0/550754257 wait complete. 2023-10-19T13:00:15.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.660+0000 7efde024f640 1 Processor -- start 2023-10-19T13:00:15.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.660+0000 7efde024f640 1 -- start start 2023-10-19T13:00:15.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.661+0000 7efde024f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efdc0003f10 0x7efdd806f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:15.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.661+0000 7efde024f640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efdd8070d40 0x7efdd80aab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:15.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.661+0000 7efde024f640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7efdd81394c0 0x7efdd814f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:15.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.661+0000 7efde024f640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdd8070920 con 0x7efdc0003f10 2023-10-19T13:00:15.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.661+0000 7efde024f640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdd806fca0 con 0x7efdd8070d40 2023-10-19T13:00:15.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.661+0000 7efde024f640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdd806ff80 con 0x7efdd81394c0 2023-10-19T13:00:15.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.661+0000 7efdddfeb640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efdc0003f10 0x7efdd806f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:15.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.661+0000 7efdddfeb640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efdc0003f10 0x7efdd806f470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40064/0 (socket says 172.21.15.57:40064) 2023-10-19T13:00:15.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.661+0000 7efdddfeb640 1 -- 172.21.15.57:0/2169652627 learned_addr learned my addr 172.21.15.57:0/2169652627 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:15.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.661+0000 7efddd7ea640 1 --2- 172.21.15.57:0/2169652627 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efdd8070d40 0x7efdd80aab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:15.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.661+0000 7efdde7ec640 1 --2- 172.21.15.57:0/2169652627 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7efdd81394c0 0x7efdd814f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:15.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.661+0000 7efdddfeb640 1 -- 172.21.15.57:0/2169652627 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7efdd81394c0 msgr2=0x7efdd814f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:15.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.661+0000 7efdddfeb640 1 --2- 172.21.15.57:0/2169652627 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7efdd81394c0 0x7efdd814f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:15.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.661+0000 7efdddfeb640 1 -- 172.21.15.57:0/2169652627 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efdd8070d40 msgr2=0x7efdd80aab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:15.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.661+0000 7efdddfeb640 1 --2- 172.21.15.57:0/2169652627 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efdd8070d40 0x7efdd80aab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:15.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.661+0000 7efdddfeb640 1 -- 172.21.15.57:0/2169652627 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdd8070260 con 0x7efdc0003f10 2023-10-19T13:00:15.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.661+0000 7efddd7ea640 1 --2- 172.21.15.57:0/2169652627 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efdd8070d40 0x7efdd80aab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:00:15.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.661+0000 7efdddfeb640 1 --2- 172.21.15.57:0/2169652627 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efdc0003f10 0x7efdd806f470 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7efdd00066b0 tx=0x7efdd00039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:15.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.662+0000 7efdd67fc640 1 -- 172.21.15.57:0/2169652627 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdd000f960 con 0x7efdc0003f10 2023-10-19T13:00:15.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.662+0000 7efdd67fc640 1 -- 172.21.15.57:0/2169652627 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7efdd000fac0 con 0x7efdc0003f10 2023-10-19T13:00:15.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.662+0000 7efde024f640 1 -- 172.21.15.57:0/2169652627 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdd8070540 con 0x7efdc0003f10 2023-10-19T13:00:15.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.662+0000 7efdd67fc640 1 -- 172.21.15.57:0/2169652627 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdd000fed0 con 0x7efdc0003f10 2023-10-19T13:00:15.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.662+0000 7efde024f640 1 -- 172.21.15.57:0/2169652627 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdd80ab070 con 0x7efdc0003f10 2023-10-19T13:00:15.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.663+0000 7efde024f640 1 -- 172.21.15.57:0/2169652627 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdd8070b90 con 0x7efdc0003f10 2023-10-19T13:00:15.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.663+0000 7efdd67fc640 1 -- 172.21.15.57:0/2169652627 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 32) v1 ==== 89331+0+0 (secure 0 0 0) 0x7efdd0011430 con 0x7efdc0003f10 2023-10-19T13:00:15.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.663+0000 7efdd67fc640 1 --2- 172.21.15.57:0/2169652627 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7efdc406af90 0x7efdc406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:15.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.664+0000 7efdd67fc640 1 -- 172.21.15.57:0/2169652627 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 5979+0+0 (secure 0 0 0) 0x7efdd008a7d0 con 0x7efdc0003f10 2023-10-19T13:00:15.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.664+0000 7efddd7ea640 1 --2- 172.21.15.57:0/2169652627 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7efdc406af90 0x7efdc406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:15.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.664+0000 7efddd7ea640 1 --2- 172.21.15.57:0/2169652627 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7efdc406af90 0x7efdc406d450 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7efdcc036000 tx=0x7efdcc035040 comp rx=0 tx=0).ready entity=mgr.6256 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:15.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.668+0000 7efdd67fc640 1 -- 172.21.15.57:0/2169652627 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7efdd00554c0 con 0x7efdc0003f10 2023-10-19T13:00:15.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.879+0000 7efde024f640 1 -- 172.21.15.57:0/2169652627 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efdd806d3e0 con 0x7efdc0003f10 2023-10-19T13:00:15.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.879+0000 7efdd67fc640 1 -- 172.21.15.57:0/2169652627 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+3702 (secure 0 0 0) 0x7efdd0020070 con 0x7efdc0003f10 2023-10-19T13:00:15.883 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:00:15.883 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":25,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5594,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/3950728949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":3950728949},{"type":"v1","addr":"172.21.15.179:6837","nonce":3950728949}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":5597,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/3158611991","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":3158611991},{"type":"v1","addr":"172.21.15.179:6836","nonce":3158611991}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":6507,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/352867817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":352867817},{"type":"v1","addr":"172.21.15.57:6834","nonce":352867817}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":6954,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/2748097680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2748097680},{"type":"v1","addr":"172.21.15.57:6835","nonce":2748097680}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":25}],"filesystems":[]} 2023-10-19T13:00:15.884 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 25 2023-10-19T13:00:15.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.881+0000 7efde024f640 1 -- 172.21.15.57:0/2169652627 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7efdc406af90 msgr2=0x7efdc406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:15.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.881+0000 7efde024f640 1 --2- 172.21.15.57:0/2169652627 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7efdc406af90 0x7efdc406d450 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7efdcc036000 tx=0x7efdcc035040 comp rx=0 tx=0).stop 2023-10-19T13:00:15.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.881+0000 7efde024f640 1 -- 172.21.15.57:0/2169652627 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efdc0003f10 msgr2=0x7efdd806f470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:15.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.881+0000 7efde024f640 1 --2- 172.21.15.57:0/2169652627 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efdc0003f10 0x7efdd806f470 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7efdd00066b0 tx=0x7efdd00039c0 comp rx=0 tx=0).stop 2023-10-19T13:00:15.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.881+0000 7efde024f640 1 -- 172.21.15.57:0/2169652627 shutdown_connections 2023-10-19T13:00:15.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.881+0000 7efde024f640 1 --2- 172.21.15.57:0/2169652627 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7efdc406af90 0x7efdc406d450 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:15.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.881+0000 7efde024f640 1 --2- 172.21.15.57:0/2169652627 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7efdd81394c0 0x7efdd814f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:15.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.881+0000 7efde024f640 1 --2- 172.21.15.57:0/2169652627 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efdd8070d40 0x7efdd80aab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:15.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.881+0000 7efde024f640 1 --2- 172.21.15.57:0/2169652627 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efdc0003f10 0x7efdd806f470 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:15.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.881+0000 7efde024f640 1 -- 172.21.15.57:0/2169652627 >> 172.21.15.57:0/2169652627 conn(0x7efdd8069930 msgr2=0x7efdd815b500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:15.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.881+0000 7efde024f640 1 -- 172.21.15.57:0/2169652627 shutdown_connections 2023-10-19T13:00:15.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:15.881+0000 7efde024f640 1 -- 172.21.15.57:0/2169652627 wait complete. 2023-10-19T13:00:15.895 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-19T13:00:15.895 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-19T13:00:15.895 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-19T13:00:15.896 INFO:teuthology.orchestra.run:waiting for 300 2023-10-19T13:00:15.896 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-19T13:00:15.896 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-19T13:00:15.896 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-19T13:00:15.897 INFO:teuthology.orchestra.run:waiting for 300 2023-10-19T13:00:15.897 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-19T13:00:15.897 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-19T13:00:15.898 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-19T13:00:15.898 INFO:teuthology.orchestra.run:waiting for 300 2023-10-19T13:00:15.898 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-19T13:00:15.898 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-19T13:00:15.899 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-19T13:00:15.899 INFO:teuthology.orchestra.run:waiting for 300 2023-10-19T13:00:15.904 INFO:tasks.ceph.mds.a:Stopped 2023-10-19T13:00:15.905 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-19T13:00:15.906 INFO:tasks.ceph.mds.c:Stopped 2023-10-19T13:00:15.907 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-19T13:00:15.912 INFO:tasks.ceph.mds.b:Stopped 2023-10-19T13:00:15.912 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-19T13:00:15.913 INFO:tasks.ceph.mds.d:Stopped 2023-10-19T13:00:15.913 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-19T13:00:15.948 INFO:tasks.ceph.mds.a:Started 2023-10-19T13:00:15.949 DEBUG:teuthology.parallel:result is None 2023-10-19T13:00:15.950 INFO:tasks.ceph.mds.c:Started 2023-10-19T13:00:15.950 DEBUG:teuthology.parallel:result is None 2023-10-19T13:00:15.955 INFO:tasks.ceph.mds.b:Started 2023-10-19T13:00:15.955 DEBUG:teuthology.parallel:result is None 2023-10-19T13:00:15.956 INFO:tasks.ceph.mds.d:Started 2023-10-19T13:00:15.956 DEBUG:teuthology.parallel:result is None 2023-10-19T13:00:15.957 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-19T13:00:16.028 INFO:tasks.ceph.mds.c.smithi057.stdout:starting mds.c at 2023-10-19T13:00:16.029 INFO:tasks.ceph.mds.a.smithi057.stdout:starting mds.a at 2023-10-19T13:00:16.039 INFO:tasks.ceph.mds.b.smithi179.stdout:starting mds.b at 2023-10-19T13:00:16.039 INFO:tasks.ceph.mds.d.smithi179.stdout:starting mds.d at 2023-10-19T13:00:16.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.067+0000 7f5074b39640 1 -- 172.21.15.57:0/682258614 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f507015b2a0 msgr2=0x7f50700d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:16.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.067+0000 7f5074b39640 1 --2- 172.21.15.57:0/682258614 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f507015b2a0 0x7f50700d1910 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f5058009830 tx=0x7f505802bcd0 comp rx=0 tx=0).stop 2023-10-19T13:00:16.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.067+0000 7f5074b39640 1 -- 172.21.15.57:0/682258614 shutdown_connections 2023-10-19T13:00:16.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.067+0000 7f5074b39640 1 --2- 172.21.15.57:0/682258614 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f50700ceb40 0x7f50700b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:16.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.067+0000 7f5074b39640 1 --2- 172.21.15.57:0/682258614 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f507015b2a0 0x7f50700d1910 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:16.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.067+0000 7f5074b39640 1 -- 172.21.15.57:0/682258614 >> 172.21.15.57:0/682258614 conn(0x7f50700d0d60 msgr2=0x7f50700d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:16.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.067+0000 7f5074b39640 1 -- 172.21.15.57:0/682258614 shutdown_connections 2023-10-19T13:00:16.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.067+0000 7f5074b39640 1 -- 172.21.15.57:0/682258614 wait complete. 2023-10-19T13:00:16.074 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.068+0000 7f5074b39640 1 Processor -- start 2023-10-19T13:00:16.074 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.069+0000 7f5074b39640 1 -- start start 2023-10-19T13:00:16.074 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.069+0000 7f5074b39640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f50700ceb40 0x7f507015cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:16.074 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.069+0000 7f5074b39640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f50700cf340 0x7f50701578d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:16.074 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.069+0000 7f5074b39640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f507015b2a0 0x7f50700c9930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:16.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.069+0000 7f5074b39640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f507015e0d0 con 0x7f50700ceb40 2023-10-19T13:00:16.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.069+0000 7f5074b39640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50700b5110 con 0x7f50700cf340 2023-10-19T13:00:16.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.069+0000 7f5074b39640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f507015d1b0 con 0x7f507015b2a0 2023-10-19T13:00:16.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.069+0000 7f506e59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f50700ceb40 0x7f507015cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:16.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.069+0000 7f506e59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f50700ceb40 0x7f507015cc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40110/0 (socket says 172.21.15.57:40110) 2023-10-19T13:00:16.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.069+0000 7f506e59c640 1 -- 172.21.15.57:0/635125991 learned_addr learned my addr 172.21.15.57:0/635125991 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:16.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.070+0000 7f506e59c640 1 -- 172.21.15.57:0/635125991 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f507015b2a0 msgr2=0x7f50700c9930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:16.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.070+0000 7f5067fff640 1 --2- 172.21.15.57:0/635125991 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f50700cf340 0x7f50701578d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:16.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.070+0000 7f506ed9d640 1 --2- 172.21.15.57:0/635125991 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f507015b2a0 0x7f50700c9930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:16.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.070+0000 7f506e59c640 1 --2- 172.21.15.57:0/635125991 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f507015b2a0 0x7f50700c9930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:16.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.070+0000 7f506e59c640 1 -- 172.21.15.57:0/635125991 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f50700cf340 msgr2=0x7f50701578d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:16.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.070+0000 7f506e59c640 1 --2- 172.21.15.57:0/635125991 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f50700cf340 0x7f50701578d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:16.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.070+0000 7f506e59c640 1 -- 172.21.15.57:0/635125991 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5058009490 con 0x7f50700ceb40 2023-10-19T13:00:16.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.070+0000 7f5067fff640 1 --2- 172.21.15.57:0/635125991 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f50700cf340 0x7f50701578d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T13:00:16.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.070+0000 7f506e59c640 1 --2- 172.21.15.57:0/635125991 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f50700ceb40 0x7f507015cc70 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f506000c500 tx=0x7f506000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:16.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.070+0000 7f5066ffd640 1 -- 172.21.15.57:0/635125991 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5060012830 con 0x7f50700ceb40 2023-10-19T13:00:16.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.070+0000 7f5066ffd640 1 -- 172.21.15.57:0/635125991 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f5060012a80 con 0x7f50700ceb40 2023-10-19T13:00:16.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.070+0000 7f5074b39640 1 -- 172.21.15.57:0/635125991 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5070157e10 con 0x7f50700ceb40 2023-10-19T13:00:16.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.070+0000 7f5066ffd640 1 -- 172.21.15.57:0/635125991 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5060012e90 con 0x7f50700ceb40 2023-10-19T13:00:16.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.070+0000 7f5074b39640 1 -- 172.21.15.57:0/635125991 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f507015d730 con 0x7f50700ceb40 2023-10-19T13:00:16.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.071+0000 7f5074b39640 1 -- 172.21.15.57:0/635125991 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50700690e0 con 0x7f50700ceb40 2023-10-19T13:00:16.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.071+0000 7f5066ffd640 1 -- 172.21.15.57:0/635125991 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 32) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f506001a3b0 con 0x7f50700ceb40 2023-10-19T13:00:16.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.072+0000 7f5066ffd640 1 --2- 172.21.15.57:0/635125991 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f505006af60 0x7f505006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:16.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.072+0000 7f5066ffd640 1 -- 172.21.15.57:0/635125991 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 5979+0+0 (secure 0 0 0) 0x7f5060089a30 con 0x7f50700ceb40 2023-10-19T13:00:16.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.072+0000 7f5067fff640 1 --2- 172.21.15.57:0/635125991 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f505006af60 0x7f505006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:16.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.073+0000 7f5067fff640 1 --2- 172.21.15.57:0/635125991 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f505006af60 0x7f505006d420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5058036000 tx=0x7f5058004730 comp rx=0 tx=0).ready entity=mgr.6256 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:16.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.076+0000 7f5066ffd640 1 -- 172.21.15.57:0/635125991 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f50600548d0 con 0x7f50700ceb40 2023-10-19T13:00:16.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.263+0000 7f5074b39640 1 -- 172.21.15.57:0/635125991 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f50700b14f0 con 0x7f50700ceb40 2023-10-19T13:00:16.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.721+0000 7f5066ffd640 1 -- 172.21.15.57:0/635125991 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v81) v1 ==== 97+0+0 (secure 0 0 0) 0x7f5060057ef0 con 0x7f50700ceb40 2023-10-19T13:00:16.725 INFO:teuthology.orchestra.run.smithi057.stderr: removed all blocklist entries 2023-10-19T13:00:16.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.724+0000 7f5074b39640 1 -- 172.21.15.57:0/635125991 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f505006af60 msgr2=0x7f505006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:16.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.724+0000 7f5074b39640 1 --2- 172.21.15.57:0/635125991 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f505006af60 0x7f505006d420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5058036000 tx=0x7f5058004730 comp rx=0 tx=0).stop 2023-10-19T13:00:16.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.724+0000 7f5074b39640 1 -- 172.21.15.57:0/635125991 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f50700ceb40 msgr2=0x7f507015cc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:16.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.724+0000 7f5074b39640 1 --2- 172.21.15.57:0/635125991 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f50700ceb40 0x7f507015cc70 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f506000c500 tx=0x7f506000c9c0 comp rx=0 tx=0).stop 2023-10-19T13:00:16.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.724+0000 7f5074b39640 1 -- 172.21.15.57:0/635125991 shutdown_connections 2023-10-19T13:00:16.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.724+0000 7f5074b39640 1 --2- 172.21.15.57:0/635125991 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f505006af60 0x7f505006d420 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:16.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.724+0000 7f5074b39640 1 --2- 172.21.15.57:0/635125991 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f507015b2a0 0x7f50700c9930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:16.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.724+0000 7f5074b39640 1 --2- 172.21.15.57:0/635125991 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f50700cf340 0x7f50701578d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:16.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.724+0000 7f5074b39640 1 --2- 172.21.15.57:0/635125991 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f50700ceb40 0x7f507015cc70 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:16.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.724+0000 7f5074b39640 1 -- 172.21.15.57:0/635125991 >> 172.21.15.57:0/635125991 conn(0x7f50700d0d60 msgr2=0x7f507015a5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:16.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.724+0000 7f5074b39640 1 -- 172.21.15.57:0/635125991 shutdown_connections 2023-10-19T13:00:16.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.724+0000 7f5074b39640 1 -- 172.21.15.57:0/635125991 wait complete. 2023-10-19T13:00:16.740 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-19T13:00:16.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.848+0000 7f9fb55f5640 1 -- 172.21.15.57:0/3335377864 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9fb0070af0 msgr2=0x7f9fb01628b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:16.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.848+0000 7f9fb55f5640 1 --2- 172.21.15.57:0/3335377864 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9fb0070af0 0x7f9fb01628b0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f9fa0005fd0 tx=0x7f9fa002c250 comp rx=0 tx=0).stop 2023-10-19T13:00:16.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.848+0000 7f9fb55f5640 1 -- 172.21.15.57:0/3335377864 shutdown_connections 2023-10-19T13:00:16.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.848+0000 7f9fb55f5640 1 --2- 172.21.15.57:0/3335377864 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9fb0070af0 0x7f9fb01628b0 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:16.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.848+0000 7f9fb55f5640 1 -- 172.21.15.57:0/3335377864 >> 172.21.15.57:0/3335377864 conn(0x7f9fb0069930 msgr2=0x7f9fb0069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:16.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.848+0000 7f9fb55f5640 1 -- 172.21.15.57:0/3335377864 shutdown_connections 2023-10-19T13:00:16.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.848+0000 7f9fb55f5640 1 -- 172.21.15.57:0/3335377864 wait complete. 2023-10-19T13:00:16.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.850+0000 7f9fb55f5640 1 Processor -- start 2023-10-19T13:00:16.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.850+0000 7f9fb55f5640 1 -- start start 2023-10-19T13:00:16.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.850+0000 7f9fb55f5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9fb0070af0 0x7f9fb0159770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:16.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.850+0000 7f9fb55f5640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9fb013e520 0x7f9fb00cae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:16.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.850+0000 7f9fb55f5640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9fb0141690 0x7f9fb00cb350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:16.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.850+0000 7f9fb55f5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fb0152d90 con 0x7f9fb0070af0 2023-10-19T13:00:16.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.850+0000 7f9fb55f5640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fb0152460 con 0x7f9fb0141690 2023-10-19T13:00:16.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.850+0000 7f9fb55f5640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fb0152740 con 0x7f9fb013e520 2023-10-19T13:00:16.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.850+0000 7f9faeffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9fb0070af0 0x7f9fb0159770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:16.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.850+0000 7f9faeffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9fb0070af0 0x7f9fb0159770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:49624/0 (socket says 172.21.15.57:49624) 2023-10-19T13:00:16.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.850+0000 7f9faeffd640 1 -- 172.21.15.57:0/2872202054 learned_addr learned my addr 172.21.15.57:0/2872202054 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:16.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.851+0000 7f9faf7fe640 1 --2- 172.21.15.57:0/2872202054 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9fb0141690 0x7f9fb00cb350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:16.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.851+0000 7f9faeffd640 1 -- 172.21.15.57:0/2872202054 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9fb013e520 msgr2=0x7f9fb00cae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:16.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.851+0000 7f9fae7fc640 1 --2- 172.21.15.57:0/2872202054 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9fb013e520 0x7f9fb00cae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:16.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.851+0000 7f9faeffd640 1 --2- 172.21.15.57:0/2872202054 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9fb013e520 0x7f9fb00cae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:16.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.851+0000 7f9faeffd640 1 -- 172.21.15.57:0/2872202054 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9fb0141690 msgr2=0x7f9fb00cb350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:16.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.851+0000 7f9faeffd640 1 --2- 172.21.15.57:0/2872202054 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9fb0141690 0x7f9fb00cb350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:16.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.851+0000 7f9faeffd640 1 -- 172.21.15.57:0/2872202054 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fa0009900 con 0x7f9fb0070af0 2023-10-19T13:00:16.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.851+0000 7f9faf7fe640 1 --2- 172.21.15.57:0/2872202054 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9fb0141690 0x7f9fb00cb350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T13:00:16.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.851+0000 7f9fae7fc640 1 --2- 172.21.15.57:0/2872202054 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9fb013e520 0x7f9fb00cae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T13:00:16.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.851+0000 7f9faeffd640 1 --2- 172.21.15.57:0/2872202054 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9fb0070af0 0x7f9fb0159770 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f9fa800c430 tx=0x7f9fa800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:16.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.851+0000 7f9fa77fe640 1 -- 172.21.15.57:0/2872202054 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fa8011790 con 0x7f9fb0070af0 2023-10-19T13:00:16.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.851+0000 7f9fa77fe640 1 -- 172.21.15.57:0/2872202054 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f9fa80119e0 con 0x7f9fb0070af0 2023-10-19T13:00:16.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.851+0000 7f9fb55f5640 1 -- 172.21.15.57:0/2872202054 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fb0152a20 con 0x7f9fb0070af0 2023-10-19T13:00:16.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.851+0000 7f9fa77fe640 1 -- 172.21.15.57:0/2872202054 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fa8011e30 con 0x7f9fb0070af0 2023-10-19T13:00:16.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.851+0000 7f9fb55f5640 1 -- 172.21.15.57:0/2872202054 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fb0159cb0 con 0x7f9fb0070af0 2023-10-19T13:00:16.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.852+0000 7f9fb55f5640 1 -- 172.21.15.57:0/2872202054 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fb00b1f20 con 0x7f9fb0070af0 2023-10-19T13:00:16.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.852+0000 7f9fa77fe640 1 -- 172.21.15.57:0/2872202054 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 32) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f9fa801c460 con 0x7f9fb0070af0 2023-10-19T13:00:16.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.853+0000 7f9fa77fe640 1 --2- 172.21.15.57:0/2872202054 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f9f9006af60 0x7f9f9006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:16.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.853+0000 7f9fa77fe640 1 -- 172.21.15.57:0/2872202054 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f9fa8089800 con 0x7f9fb0070af0 2023-10-19T13:00:16.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.853+0000 7f9fae7fc640 1 --2- 172.21.15.57:0/2872202054 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f9f9006af60 0x7f9f9006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:16.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.854+0000 7f9fae7fc640 1 --2- 172.21.15.57:0/2872202054 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f9f9006af60 0x7f9f9006d420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f9fb00c9870 tx=0x7f9f98009040 comp rx=0 tx=0).ready entity=mgr.6256 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:16.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:16.857+0000 7f9fa77fe640 1 -- 172.21.15.57:0/2872202054 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9fa8054990 con 0x7f9fb0070af0 2023-10-19T13:00:17.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.101+0000 7f9fb55f5640 1 -- 172.21.15.57:0/2872202054 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f9fb013e060 con 0x7f9fb0070af0 2023-10-19T13:00:17.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.101+0000 7f9fa77fe640 1 -- 172.21.15.57:0/2872202054 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v23) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f9fa8057fb0 con 0x7f9fb0070af0 2023-10-19T13:00:17.104 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:00:17.104 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2023-10-19T13:00:17.105 INFO:teuthology.orchestra.run.smithi057.stdout: "auth_dump": [ 2023-10-19T13:00:17.105 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:00:17.105 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "mds.a", 2023-10-19T13:00:17.105 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBjJDFlmbVANhAAP/ynHlU/OxaRq5lQoAa8Ng==", 2023-10-19T13:00:17.105 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:00:17.106 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T13:00:17.106 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow *", 2023-10-19T13:00:17.106 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow *", 2023-10-19T13:00:17.106 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:00:17.106 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:00:17.107 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:00:17.107 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:00:17.107 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "mds.b", 2023-10-19T13:00:17.107 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFl9l7RBxAAC0geI2tJ1SIZZr1++kS/BA==", 2023-10-19T13:00:17.107 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:00:17.108 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T13:00:17.108 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow *", 2023-10-19T13:00:17.108 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow *", 2023-10-19T13:00:17.108 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:00:17.108 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:00:17.109 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:00:17.109 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:00:17.109 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "mds.c", 2023-10-19T13:00:17.109 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFlFkaoAhAA/PBizkJ4bm958oP4CmPuIQ==", 2023-10-19T13:00:17.109 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:00:17.110 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T13:00:17.110 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow *", 2023-10-19T13:00:17.110 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow *", 2023-10-19T13:00:17.110 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:00:17.110 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:00:17.111 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:00:17.111 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:00:17.111 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "mds.d", 2023-10-19T13:00:17.111 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFlx8skDxAAAfCHTYtYJgJik0WHSWeQfQ==", 2023-10-19T13:00:17.111 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:00:17.112 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T13:00:17.112 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow *", 2023-10-19T13:00:17.112 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow *", 2023-10-19T13:00:17.112 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:00:17.112 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:00:17.113 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:00:17.113 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:00:17.113 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.0", 2023-10-19T13:00:17.113 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBrJDFlCo9sBxAAQN2cID4nCQgRwGaGFEV8LQ==", 2023-10-19T13:00:17.113 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:00:17.113 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T13:00:17.114 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T13:00:17.114 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:00:17.114 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:00:17.114 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:00:17.114 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:00:17.115 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.1", 2023-10-19T13:00:17.115 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBuJDFl4fsLCBAAp3Pi80yZtc5sju0GisEuVA==", 2023-10-19T13:00:17.115 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:00:17.115 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T13:00:17.115 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T13:00:17.116 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:00:17.116 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:00:17.116 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:00:17.116 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:00:17.116 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.2", 2023-10-19T13:00:17.116 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBxJDFl+KAKCBAAd4N9cluARxPVfBLU+7Hd7A==", 2023-10-19T13:00:17.117 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:00:17.117 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T13:00:17.117 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T13:00:17.117 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:00:17.117 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:00:17.118 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:00:17.118 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:00:17.118 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.3", 2023-10-19T13:00:17.118 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQB0JDFlU/toCBAAP0Ou1zcHKNx5SfbybJkh4Q==", 2023-10-19T13:00:17.118 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:00:17.119 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T13:00:17.119 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T13:00:17.119 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:00:17.119 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:00:17.119 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:00:17.119 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:00:17.120 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.4", 2023-10-19T13:00:17.120 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQB6JDFlc6stEBAALkgNyxoiCTzZnKpBSPxI6w==", 2023-10-19T13:00:17.120 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:00:17.120 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T13:00:17.120 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T13:00:17.121 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:00:17.121 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:00:17.121 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:00:17.121 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:00:17.121 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.5", 2023-10-19T13:00:17.121 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQB9JDFlI3cNDhAAuG3tHJ6RvPwAzPE76wTIVQ==", 2023-10-19T13:00:17.122 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:00:17.122 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T13:00:17.122 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T13:00:17.122 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:00:17.122 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:00:17.123 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:00:17.123 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:00:17.123 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.6", 2023-10-19T13:00:17.123 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQCAJDFlLPumCxAA/iy08t/2N9rXfn71KtbSQw==", 2023-10-19T13:00:17.123 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:00:17.123 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T13:00:17.124 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T13:00:17.124 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:00:17.124 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:00:17.124 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:00:17.124 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:00:17.125 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.7", 2023-10-19T13:00:17.125 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQCDJDFlHmrbCRAA+hpWygGIJydfU6cHWlfx/Q==", 2023-10-19T13:00:17.125 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:00:17.125 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T13:00:17.125 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T13:00:17.126 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:00:17.126 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:00:17.126 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:00:17.126 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:00:17.126 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "client.0", 2023-10-19T13:00:17.127 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFlg9yuExAATxiWDfS+p07G1F0pORbJEg==", 2023-10-19T13:00:17.127 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:00:17.127 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T13:00:17.127 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow r", 2023-10-19T13:00:17.127 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-19T13:00:17.128 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:00:17.128 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:00:17.128 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:00:17.128 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "client.1", 2023-10-19T13:00:17.128 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFl9IwYGxAAHPituiJSPkHDuD0QyG4Hww==", 2023-10-19T13:00:17.128 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:00:17.129 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T13:00:17.129 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow r", 2023-10-19T13:00:17.129 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow rw", 2023-10-19T13:00:17.129 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow rwx" 2023-10-19T13:00:17.129 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:00:17.130 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:00:17.130 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:00:17.130 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "client.2", 2023-10-19T13:00:17.130 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFlMd+AFxAAS/UTESDt76GmrhhN2o/Fdw==", 2023-10-19T13:00:17.130 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:00:17.131 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T13:00:17.131 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow r", 2023-10-19T13:00:17.131 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow rw", 2023-10-19T13:00:17.131 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow rwx" 2023-10-19T13:00:17.131 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:00:17.131 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:00:17.132 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:00:17.132 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "client.3", 2023-10-19T13:00:17.132 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFlHZ3fIBAA42B7n8opLS5smXIwJyVYzw==", 2023-10-19T13:00:17.132 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:00:17.132 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T13:00:17.132 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow r", 2023-10-19T13:00:17.133 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow rw", 2023-10-19T13:00:17.133 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow rwx" 2023-10-19T13:00:17.133 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:00:17.133 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:00:17.133 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:00:17.134 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "client.admin", 2023-10-19T13:00:17.134 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBjJDFlBgasFxAAMA24j8KszcXMEpYRQDXj4g==", 2023-10-19T13:00:17.134 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:00:17.134 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow *", 2023-10-19T13:00:17.134 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow *", 2023-10-19T13:00:17.135 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow *", 2023-10-19T13:00:17.135 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:00:17.135 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:00:17.135 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:00:17.135 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:00:17.135 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "mgr.x", 2023-10-19T13:00:17.136 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBjJDFllFBfMhAA5+KPXWmYZqcESmLObaUbBA==", 2023-10-19T13:00:17.136 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:00:17.136 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow *", 2023-10-19T13:00:17.136 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile mgr", 2023-10-19T13:00:17.136 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:00:17.136 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:00:17.137 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:00:17.137 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:00:17.137 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "mgr.y", 2023-10-19T13:00:17.137 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBjJDFlxRzJLRAAGjG58o7qXAHvIEIRYwl6QA==", 2023-10-19T13:00:17.137 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:00:17.137 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow *", 2023-10-19T13:00:17.138 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile mgr", 2023-10-19T13:00:17.138 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:00:17.138 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:00:17.138 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:00:17.138 INFO:teuthology.orchestra.run.smithi057.stdout: ] 2023-10-19T13:00:17.138 INFO:teuthology.orchestra.run.smithi057.stdout:} 2023-10-19T13:00:17.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.103+0000 7f9fb55f5640 1 -- 172.21.15.57:0/2872202054 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f9f9006af60 msgr2=0x7f9f9006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:17.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.103+0000 7f9fb55f5640 1 --2- 172.21.15.57:0/2872202054 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f9f9006af60 0x7f9f9006d420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f9fb00c9870 tx=0x7f9f98009040 comp rx=0 tx=0).stop 2023-10-19T13:00:17.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.103+0000 7f9fb55f5640 1 -- 172.21.15.57:0/2872202054 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9fb0070af0 msgr2=0x7f9fb0159770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:17.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.103+0000 7f9fb55f5640 1 --2- 172.21.15.57:0/2872202054 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9fb0070af0 0x7f9fb0159770 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f9fa800c430 tx=0x7f9fa800c8f0 comp rx=0 tx=0).stop 2023-10-19T13:00:17.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.103+0000 7f9fb55f5640 1 -- 172.21.15.57:0/2872202054 shutdown_connections 2023-10-19T13:00:17.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.103+0000 7f9fb55f5640 1 --2- 172.21.15.57:0/2872202054 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f9f9006af60 0x7f9f9006d420 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:17.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.103+0000 7f9fb55f5640 1 --2- 172.21.15.57:0/2872202054 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9fb0141690 0x7f9fb00cb350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:17.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.103+0000 7f9fb55f5640 1 --2- 172.21.15.57:0/2872202054 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9fb013e520 0x7f9fb00cae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:17.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.103+0000 7f9fb55f5640 1 --2- 172.21.15.57:0/2872202054 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9fb0070af0 0x7f9fb0159770 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:17.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.103+0000 7f9fb55f5640 1 -- 172.21.15.57:0/2872202054 >> 172.21.15.57:0/2872202054 conn(0x7f9fb0069930 msgr2=0x7f9fb015ad40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:17.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.103+0000 7f9fb55f5640 1 -- 172.21.15.57:0/2872202054 shutdown_connections 2023-10-19T13:00:17.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.103+0000 7f9fb55f5640 1 -- 172.21.15.57:0/2872202054 wait complete. 2023-10-19T13:00:17.143 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T13:00:17.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.249+0000 7fe258664640 1 -- 172.21.15.57:0/2716931705 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe250158cf0 msgr2=0x7fe25015c0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:17.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.249+0000 7fe258664640 1 --2- 172.21.15.57:0/2716931705 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe250158cf0 0x7fe25015c0e0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7fe244005fd0 tx=0x7fe24402bd90 comp rx=0 tx=0).stop 2023-10-19T13:00:17.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.250+0000 7fe258664640 1 -- 172.21.15.57:0/2716931705 shutdown_connections 2023-10-19T13:00:17.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.250+0000 7fe258664640 1 --2- 172.21.15.57:0/2716931705 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe2500cd9e0 0x7fe2500cde70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:17.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.250+0000 7fe258664640 1 --2- 172.21.15.57:0/2716931705 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe250158cf0 0x7fe25015c0e0 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:17.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.250+0000 7fe258664640 1 -- 172.21.15.57:0/2716931705 >> 172.21.15.57:0/2716931705 conn(0x7fe2500af7a0 msgr2=0x7fe2500afb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:17.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.250+0000 7fe258664640 1 -- 172.21.15.57:0/2716931705 shutdown_connections 2023-10-19T13:00:17.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.250+0000 7fe258664640 1 -- 172.21.15.57:0/2716931705 wait complete. 2023-10-19T13:00:17.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.251+0000 7fe258664640 1 Processor -- start 2023-10-19T13:00:17.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.251+0000 7fe258664640 1 -- start start 2023-10-19T13:00:17.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.251+0000 7fe258664640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe2500cd9e0 0x7fe2500b12a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:17.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.251+0000 7fe258664640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe250158910 0x7fe250149760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:17.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.251+0000 7fe258664640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe250158cf0 0x7fe250159640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:17.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.251+0000 7fe258664640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2500ad380 con 0x7fe250158910 2023-10-19T13:00:17.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.251+0000 7fe258664640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2500ac800 con 0x7fe250158cf0 2023-10-19T13:00:17.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.251+0000 7fe258664640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2500acae0 con 0x7fe2500cd9e0 2023-10-19T13:00:17.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.252+0000 7fe255bff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe250158910 0x7fe250149760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:17.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.252+0000 7fe255bff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe250158910 0x7fe250149760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:49644/0 (socket says 172.21.15.57:49644) 2023-10-19T13:00:17.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.252+0000 7fe255bff640 1 -- 172.21.15.57:0/1680132693 learned_addr learned my addr 172.21.15.57:0/1680132693 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:17.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.252+0000 7fe256c01640 1 --2- 172.21.15.57:0/1680132693 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe250158cf0 0x7fe250159640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:17.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.252+0000 7fe256400640 1 --2- 172.21.15.57:0/1680132693 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe2500cd9e0 0x7fe2500b12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:17.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.252+0000 7fe255bff640 1 -- 172.21.15.57:0/1680132693 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe2500cd9e0 msgr2=0x7fe2500b12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:17.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.252+0000 7fe255bff640 1 --2- 172.21.15.57:0/1680132693 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe2500cd9e0 0x7fe2500b12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:17.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.252+0000 7fe255bff640 1 -- 172.21.15.57:0/1680132693 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe250158cf0 msgr2=0x7fe250159640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:17.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.252+0000 7fe255bff640 1 --2- 172.21.15.57:0/1680132693 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe250158cf0 0x7fe250159640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:17.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.252+0000 7fe255bff640 1 -- 172.21.15.57:0/1680132693 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe244009560 con 0x7fe250158910 2023-10-19T13:00:17.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.252+0000 7fe256c01640 1 --2- 172.21.15.57:0/1680132693 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe250158cf0 0x7fe250159640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:00:17.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.252+0000 7fe255bff640 1 --2- 172.21.15.57:0/1680132693 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe250158910 0x7fe250149760 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fe244006010 tx=0x7fe24402dd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:17.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.252+0000 7fe24effd640 1 -- 172.21.15.57:0/1680132693 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe244005650 con 0x7fe250158910 2023-10-19T13:00:17.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.252+0000 7fe24effd640 1 -- 172.21.15.57:0/1680132693 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fe2440038f0 con 0x7fe250158910 2023-10-19T13:00:17.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.252+0000 7fe258664640 1 -- 172.21.15.57:0/1680132693 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2500acdc0 con 0x7fe250158910 2023-10-19T13:00:17.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.252+0000 7fe24effd640 1 -- 172.21.15.57:0/1680132693 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe244003d00 con 0x7fe250158910 2023-10-19T13:00:17.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.252+0000 7fe258664640 1 -- 172.21.15.57:0/1680132693 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2500b24e0 con 0x7fe250158910 2023-10-19T13:00:17.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.253+0000 7fe258664640 1 -- 172.21.15.57:0/1680132693 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe250153fd0 con 0x7fe250158910 2023-10-19T13:00:17.262 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.254+0000 7fe24effd640 1 -- 172.21.15.57:0/1680132693 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 32) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe24403e450 con 0x7fe250158910 2023-10-19T13:00:17.262 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.254+0000 7fe24effd640 1 --2- 172.21.15.57:0/1680132693 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7fe23c06af60 0x7fe23c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:17.262 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.254+0000 7fe24effd640 1 -- 172.21.15.57:0/1680132693 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5205+0+0 (secure 0 0 0) 0x7fe2440ac1c0 con 0x7fe250158910 2023-10-19T13:00:17.262 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.255+0000 7fe256400640 1 --2- 172.21.15.57:0/1680132693 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7fe23c06af60 0x7fe23c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:17.263 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.255+0000 7fe256400640 1 --2- 172.21.15.57:0/1680132693 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7fe23c06af60 0x7fe23c06d420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe2500c9e00 tx=0x7fe240009040 comp rx=0 tx=0).ready entity=mgr.6256 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:17.263 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.258+0000 7fe24effd640 1 -- 172.21.15.57:0/1680132693 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe2440775f0 con 0x7fe250158910 2023-10-19T13:00:17.369 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:00:17.371 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:00:17.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.437+0000 7fe258664640 1 -- 172.21.15.57:0/1680132693 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe2500690e0 con 0x7fe250158910 2023-10-19T13:00:17.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.438+0000 7fe24effd640 1 -- 172.21.15.57:0/1680132693 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+13117 (secure 0 0 0) 0x7fe244003e60 con 0x7fe250158910 2023-10-19T13:00:17.441 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:00:17.441 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":83,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T13:00:16.733821+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T13:00:17.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.440+0000 7fe258664640 1 -- 172.21.15.57:0/1680132693 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7fe23c06af60 msgr2=0x7fe23c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:17.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.440+0000 7fe258664640 1 --2- 172.21.15.57:0/1680132693 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7fe23c06af60 0x7fe23c06d420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe2500c9e00 tx=0x7fe240009040 comp rx=0 tx=0).stop 2023-10-19T13:00:17.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.440+0000 7fe258664640 1 -- 172.21.15.57:0/1680132693 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe250158910 msgr2=0x7fe250149760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:17.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.440+0000 7fe258664640 1 --2- 172.21.15.57:0/1680132693 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe250158910 0x7fe250149760 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fe244006010 tx=0x7fe24402dd60 comp rx=0 tx=0).stop 2023-10-19T13:00:17.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.440+0000 7fe258664640 1 -- 172.21.15.57:0/1680132693 shutdown_connections 2023-10-19T13:00:17.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.440+0000 7fe258664640 1 --2- 172.21.15.57:0/1680132693 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7fe23c06af60 0x7fe23c06d420 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:17.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.440+0000 7fe258664640 1 --2- 172.21.15.57:0/1680132693 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe250158cf0 0x7fe250159640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:17.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.440+0000 7fe258664640 1 --2- 172.21.15.57:0/1680132693 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe250158910 0x7fe250149760 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:17.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.440+0000 7fe258664640 1 --2- 172.21.15.57:0/1680132693 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe2500cd9e0 0x7fe2500b12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:17.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.440+0000 7fe258664640 1 -- 172.21.15.57:0/1680132693 >> 172.21.15.57:0/1680132693 conn(0x7fe2500af7a0 msgr2=0x7fe25015ad20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:17.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.440+0000 7fe258664640 1 -- 172.21.15.57:0/1680132693 shutdown_connections 2023-10-19T13:00:17.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.440+0000 7fe258664640 1 -- 172.21.15.57:0/1680132693 wait complete. 2023-10-19T13:00:17.454 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-19T12:44:07.371186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-19T13:00:17.454 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-19T13:00:17.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.603+0000 7ff310f4c640 1 -- 172.21.15.57:0/1938180964 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff2f8003970 msgr2=0x7ff2f8003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:17.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.603+0000 7ff310f4c640 1 --2- 172.21.15.57:0/1938180964 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff2f8003970 0x7ff2f8003d90 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7ff2fc00bbb0 tx=0x7ff2f80057f0 comp rx=0 tx=0).stop 2023-10-19T13:00:17.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.603+0000 7ff30ad9d640 1 -- 172.21.15.57:0/1938180964 reap_dead start 2023-10-19T13:00:17.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.603+0000 7ff310f4c640 1 -- 172.21.15.57:0/1938180964 shutdown_connections 2023-10-19T13:00:17.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.603+0000 7ff310f4c640 1 -- 172.21.15.57:0/1938180964 >> 172.21.15.57:0/1938180964 conn(0x7ff30c069930 msgr2=0x7ff30c069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:17.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.603+0000 7ff310f4c640 1 -- 172.21.15.57:0/1938180964 shutdown_connections 2023-10-19T13:00:17.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.603+0000 7ff310f4c640 1 -- 172.21.15.57:0/1938180964 wait complete. 2023-10-19T13:00:17.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.605+0000 7ff310f4c640 1 Processor -- start 2023-10-19T13:00:17.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.605+0000 7ff310f4c640 1 -- start start 2023-10-19T13:00:17.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.605+0000 7ff310f4c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff2f8003970 0x7ff30c1db450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:17.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.605+0000 7ff310f4c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff30c13e990 0x7ff30c1e20e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:17.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.605+0000 7ff310f4c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff30c1e2620 0x7ff30c0ca870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:17.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.605+0000 7ff310f4c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff30c1e31d0 con 0x7ff30c1e2620 2023-10-19T13:00:17.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.605+0000 7ff310f4c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff30c0aadd0 con 0x7ff2f8003970 2023-10-19T13:00:17.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.605+0000 7ff310f4c640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff30c1db990 con 0x7ff30c13e990 2023-10-19T13:00:17.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.605+0000 7ff30ad9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff30c1e2620 0x7ff30c0ca870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:17.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.606+0000 7ff30ad9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff30c1e2620 0x7ff30c0ca870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:49666/0 (socket says 172.21.15.57:49666) 2023-10-19T13:00:17.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.606+0000 7ff30ad9d640 1 -- 172.21.15.57:0/1725774656 learned_addr learned my addr 172.21.15.57:0/1725774656 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:17.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.606+0000 7ff303fff640 1 --2- 172.21.15.57:0/1725774656 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff30c13e990 0x7ff30c1e20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:17.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.606+0000 7ff30ad9d640 1 -- 172.21.15.57:0/1725774656 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff30c13e990 msgr2=0x7ff30c1e20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:17.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.606+0000 7ff30ad9d640 1 --2- 172.21.15.57:0/1725774656 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff30c13e990 0x7ff30c1e20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:17.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.606+0000 7ff30a59c640 1 --2- 172.21.15.57:0/1725774656 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff2f8003970 0x7ff30c1db450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:17.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.606+0000 7ff30ad9d640 1 -- 172.21.15.57:0/1725774656 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff2f8003970 msgr2=0x7ff30c1db450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:17.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.606+0000 7ff30ad9d640 1 --2- 172.21.15.57:0/1725774656 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff2f8003970 0x7ff30c1db450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:17.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.606+0000 7ff30ad9d640 1 -- 172.21.15.57:0/1725774656 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff30c1e2c10 con 0x7ff30c1e2620 2023-10-19T13:00:17.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.606+0000 7ff30ad9d640 1 --2- 172.21.15.57:0/1725774656 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff30c1e2620 0x7ff30c0ca870 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7ff304002230 tx=0x7ff30400ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:17.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.606+0000 7ff302ffd640 1 -- 172.21.15.57:0/1725774656 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff30400c940 con 0x7ff30c1e2620 2023-10-19T13:00:17.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.606+0000 7ff302ffd640 1 -- 172.21.15.57:0/1725774656 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7ff30400cbb0 con 0x7ff30c1e2620 2023-10-19T13:00:17.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.606+0000 7ff310f4c640 1 -- 172.21.15.57:0/1725774656 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff30c1e2ef0 con 0x7ff30c1e2620 2023-10-19T13:00:17.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.606+0000 7ff302ffd640 1 -- 172.21.15.57:0/1725774656 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff304011460 con 0x7ff30c1e2620 2023-10-19T13:00:17.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.606+0000 7ff310f4c640 1 -- 172.21.15.57:0/1725774656 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff30c13ee10 con 0x7ff30c1e2620 2023-10-19T13:00:17.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.607+0000 7ff310f4c640 1 -- 172.21.15.57:0/1725774656 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff30c0aa8c0 con 0x7ff30c1e2620 2023-10-19T13:00:17.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.608+0000 7ff302ffd640 1 -- 172.21.15.57:0/1725774656 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 32) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff3040115c0 con 0x7ff30c1e2620 2023-10-19T13:00:17.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.608+0000 7ff302ffd640 1 --2- 172.21.15.57:0/1725774656 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7ff2ec06af60 0x7ff2ec06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:17.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.608+0000 7ff302ffd640 1 -- 172.21.15.57:0/1725774656 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5205+0+0 (secure 0 0 0) 0x7ff304089cc0 con 0x7ff30c1e2620 2023-10-19T13:00:17.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.608+0000 7ff30a59c640 1 --2- 172.21.15.57:0/1725774656 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7ff2ec06af60 0x7ff2ec06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:17.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.609+0000 7ff30a59c640 1 --2- 172.21.15.57:0/1725774656 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7ff2ec06af60 0x7ff2ec06d420 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff2fc037000 tx=0x7ff2fc02e040 comp rx=0 tx=0).ready entity=mgr.6256 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:17.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.612+0000 7ff302ffd640 1 -- 172.21.15.57:0/1725774656 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff30c0aa8c0 con 0x7ff30c1e2620 2023-10-19T13:00:17.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.793+0000 7ff310f4c640 1 -- 172.21.15.57:0/1725774656 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff30c0aa8c0 con 0x7ff30c1e2620 2023-10-19T13:00:17.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.794+0000 7ff302ffd640 1 -- 172.21.15.57:0/1725774656 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v83) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff30c0aa8c0 con 0x7ff30c1e2620 2023-10-19T13:00:17.797 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-19T13:00:17.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.796+0000 7ff310f4c640 1 -- 172.21.15.57:0/1725774656 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7ff2ec06af60 msgr2=0x7ff2ec06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:17.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.796+0000 7ff310f4c640 1 --2- 172.21.15.57:0/1725774656 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7ff2ec06af60 0x7ff2ec06d420 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff2fc037000 tx=0x7ff2fc02e040 comp rx=0 tx=0).stop 2023-10-19T13:00:17.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.796+0000 7ff310f4c640 1 -- 172.21.15.57:0/1725774656 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff30c1e2620 msgr2=0x7ff30c0ca870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:17.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.796+0000 7ff310f4c640 1 --2- 172.21.15.57:0/1725774656 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff30c1e2620 0x7ff30c0ca870 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7ff304002230 tx=0x7ff30400ed10 comp rx=0 tx=0).stop 2023-10-19T13:00:17.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.796+0000 7ff310f4c640 1 -- 172.21.15.57:0/1725774656 shutdown_connections 2023-10-19T13:00:17.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.796+0000 7ff310f4c640 1 --2- 172.21.15.57:0/1725774656 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7ff2ec06af60 0x7ff2ec06d420 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:17.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.796+0000 7ff310f4c640 1 --2- 172.21.15.57:0/1725774656 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff30c1e2620 0x7ff30c0ca870 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:17.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.796+0000 7ff310f4c640 1 --2- 172.21.15.57:0/1725774656 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff30c13e990 0x7ff30c1e20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:17.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.796+0000 7ff310f4c640 1 --2- 172.21.15.57:0/1725774656 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff2f8003970 0x7ff30c1db450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:17.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.796+0000 7ff310f4c640 1 -- 172.21.15.57:0/1725774656 >> 172.21.15.57:0/1725774656 conn(0x7ff30c069930 msgr2=0x7ff30c1df640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:17.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.796+0000 7ff310f4c640 1 -- 172.21.15.57:0/1725774656 shutdown_connections 2023-10-19T13:00:17.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.796+0000 7ff310f4c640 1 -- 172.21.15.57:0/1725774656 wait complete. 2023-10-19T13:00:17.810 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-19T13:00:17.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.916+0000 7f9d83490640 1 -- 172.21.15.57:0/3616257717 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9d680047c0 msgr2=0x7f9d68004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:17.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.916+0000 7f9d83490640 1 --2- 172.21.15.57:0/3616257717 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9d680047c0 0x7f9d68004c40 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c001680 tx=0x7f9d680057f0 comp rx=0 tx=0).stop 2023-10-19T13:00:17.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.916+0000 7f9d83490640 1 -- 172.21.15.57:0/3616257717 shutdown_connections 2023-10-19T13:00:17.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.916+0000 7f9d83490640 1 --2- 172.21.15.57:0/3616257717 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9d680047c0 0x7f9d68004c40 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:17.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.916+0000 7f9d83490640 1 -- 172.21.15.57:0/3616257717 >> 172.21.15.57:0/3616257717 conn(0x7f9d7c0b0670 msgr2=0x7f9d7c0b0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:17.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.916+0000 7f9d83490640 1 -- 172.21.15.57:0/3616257717 shutdown_connections 2023-10-19T13:00:17.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.916+0000 7f9d83490640 1 -- 172.21.15.57:0/3616257717 wait complete. 2023-10-19T13:00:17.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.918+0000 7f9d83490640 1 Processor -- start 2023-10-19T13:00:17.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.918+0000 7f9d83490640 1 -- start start 2023-10-19T13:00:17.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.918+0000 7f9d83490640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9d680047c0 0x7f9d7c0cc080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:17.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.918+0000 7f9d83490640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9d7c1e4890 0x7f9d7c1eefd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:17.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.918+0000 7f9d83490640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9d7c15ae50 0x7f9d7c15b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:17.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.918+0000 7f9d83490640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d7c0c98d0 con 0x7f9d7c15ae50 2023-10-19T13:00:17.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.918+0000 7f9d83490640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d7c15b7d0 con 0x7f9d7c1e4890 2023-10-19T13:00:17.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.918+0000 7f9d83490640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d7c0c9580 con 0x7f9d680047c0 2023-10-19T13:00:17.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.918+0000 7f9d81a2d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9d7c15ae50 0x7f9d7c15b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:17.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.918+0000 7f9d81a2d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9d7c15ae50 0x7f9d7c15b290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:49676/0 (socket says 172.21.15.57:49676) 2023-10-19T13:00:17.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.918+0000 7f9d81a2d640 1 -- 172.21.15.57:0/2801124273 learned_addr learned my addr 172.21.15.57:0/2801124273 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:17.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.919+0000 7f9d8122c640 1 --2- 172.21.15.57:0/2801124273 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9d680047c0 0x7f9d7c0cc080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:17.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.919+0000 7f9d80a2b640 1 --2- 172.21.15.57:0/2801124273 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9d7c1e4890 0x7f9d7c1eefd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:17.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.919+0000 7f9d81a2d640 1 -- 172.21.15.57:0/2801124273 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9d680047c0 msgr2=0x7f9d7c0cc080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:17.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.919+0000 7f9d81a2d640 1 --2- 172.21.15.57:0/2801124273 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9d680047c0 0x7f9d7c0cc080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:17.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.919+0000 7f9d81a2d640 1 -- 172.21.15.57:0/2801124273 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9d7c1e4890 msgr2=0x7f9d7c1eefd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:17.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.919+0000 7f9d81a2d640 1 --2- 172.21.15.57:0/2801124273 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9d7c1e4890 0x7f9d7c1eefd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:17.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.919+0000 7f9d81a2d640 1 -- 172.21.15.57:0/2801124273 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d7c1e50e0 con 0x7f9d7c15ae50 2023-10-19T13:00:17.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.919+0000 7f9d80a2b640 1 --2- 172.21.15.57:0/2801124273 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9d7c1e4890 0x7f9d7c1eefd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:00:17.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.919+0000 7f9d81a2d640 1 --2- 172.21.15.57:0/2801124273 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9d7c15ae50 0x7f9d7c15b290 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f9d74000f80 tx=0x7f9d74006720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:17.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.919+0000 7f9d71ffb640 1 -- 172.21.15.57:0/2801124273 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d7400fa20 con 0x7f9d7c15ae50 2023-10-19T13:00:17.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.919+0000 7f9d71ffb640 1 -- 172.21.15.57:0/2801124273 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f9d7400fb80 con 0x7f9d7c15ae50 2023-10-19T13:00:17.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.919+0000 7f9d71ffb640 1 -- 172.21.15.57:0/2801124273 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d74011430 con 0x7f9d7c15ae50 2023-10-19T13:00:17.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.919+0000 7f9d83490640 1 -- 172.21.15.57:0/2801124273 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d7c1e53c0 con 0x7f9d7c15ae50 2023-10-19T13:00:17.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.919+0000 7f9d83490640 1 -- 172.21.15.57:0/2801124273 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d7c1ef510 con 0x7f9d7c15ae50 2023-10-19T13:00:17.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.920+0000 7f9d83490640 1 -- 172.21.15.57:0/2801124273 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d7c0ce1d0 con 0x7f9d7c15ae50 2023-10-19T13:00:17.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.921+0000 7f9d71ffb640 1 -- 172.21.15.57:0/2801124273 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 32) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f9d74011590 con 0x7f9d7c15ae50 2023-10-19T13:00:17.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.921+0000 7f9d71ffb640 1 --2- 172.21.15.57:0/2801124273 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f9d6006af60 0x7f9d6006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:17.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.921+0000 7f9d71ffb640 1 -- 172.21.15.57:0/2801124273 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f9d7408a260 con 0x7f9d7c15ae50 2023-10-19T13:00:17.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.921+0000 7f9d8122c640 1 --2- 172.21.15.57:0/2801124273 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f9d6006af60 0x7f9d6006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:17.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.922+0000 7f9d8122c640 1 --2- 172.21.15.57:0/2801124273 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f9d6006af60 0x7f9d6006d420 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f9d7800bbc0 tx=0x7f9d7800b2e0 comp rx=0 tx=0).ready entity=mgr.6256 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:17.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:17.925+0000 7f9d71ffb640 1 -- 172.21.15.57:0/2801124273 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9d740551b0 con 0x7f9d7c15ae50 2023-10-19T13:00:18.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:18.104+0000 7f9d83490640 1 -- 172.21.15.57:0/2801124273 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f9d7c0690e0 con 0x7f9d7c15ae50 2023-10-19T13:00:18.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:18.104+0000 7f9d71ffb640 1 -- 172.21.15.57:0/2801124273 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v83) v1 ==== 84+0+65 (secure 0 0 0) 0x7f9d740587d0 con 0x7f9d7c15ae50 2023-10-19T13:00:18.107 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:00:18.108 INFO:teuthology.orchestra.run.smithi057.stdout:[ 2023-10-19T13:00:18.108 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:00:18.108 INFO:teuthology.orchestra.run.smithi057.stdout: "poolnum": 1, 2023-10-19T13:00:18.108 INFO:teuthology.orchestra.run.smithi057.stdout: "poolname": ".mgr" 2023-10-19T13:00:18.108 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:00:18.109 INFO:teuthology.orchestra.run.smithi057.stdout:] 2023-10-19T13:00:18.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:18.106+0000 7f9d83490640 1 -- 172.21.15.57:0/2801124273 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f9d6006af60 msgr2=0x7f9d6006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:18.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:18.106+0000 7f9d83490640 1 --2- 172.21.15.57:0/2801124273 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f9d6006af60 0x7f9d6006d420 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f9d7800bbc0 tx=0x7f9d7800b2e0 comp rx=0 tx=0).stop 2023-10-19T13:00:18.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:18.106+0000 7f9d83490640 1 -- 172.21.15.57:0/2801124273 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9d7c15ae50 msgr2=0x7f9d7c15b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:18.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:18.106+0000 7f9d83490640 1 --2- 172.21.15.57:0/2801124273 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9d7c15ae50 0x7f9d7c15b290 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f9d74000f80 tx=0x7f9d74006720 comp rx=0 tx=0).stop 2023-10-19T13:00:18.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:18.106+0000 7f9d83490640 1 -- 172.21.15.57:0/2801124273 shutdown_connections 2023-10-19T13:00:18.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:18.106+0000 7f9d83490640 1 --2- 172.21.15.57:0/2801124273 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f9d6006af60 0x7f9d6006d420 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:18.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:18.106+0000 7f9d83490640 1 --2- 172.21.15.57:0/2801124273 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9d7c15ae50 0x7f9d7c15b290 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:18.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:18.106+0000 7f9d83490640 1 --2- 172.21.15.57:0/2801124273 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9d7c1e4890 0x7f9d7c1eefd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:18.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:18.106+0000 7f9d83490640 1 --2- 172.21.15.57:0/2801124273 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9d680047c0 0x7f9d7c0cc080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:18.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:18.106+0000 7f9d83490640 1 -- 172.21.15.57:0/2801124273 >> 172.21.15.57:0/2801124273 conn(0x7f9d7c0b0670 msgr2=0x7f9d7c0afcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:18.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:18.106+0000 7f9d83490640 1 -- 172.21.15.57:0/2801124273 shutdown_connections 2023-10-19T13:00:18.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:18.106+0000 7f9d83490640 1 -- 172.21.15.57:0/2801124273 wait complete. 2023-10-19T13:00:18.119 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-19T13:00:18.120 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-19T13:00:18.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:18.231+0000 7f255a2cf640 1 -- 172.21.15.57:0/3494190689 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f255406d2d0 msgr2=0x7f255406d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:18.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:18.231+0000 7f255a2cf640 1 --2- 172.21.15.57:0/3494190689 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f255406d2d0 0x7f255406d6b0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f2544005fd0 tx=0x7f254402bd90 comp rx=0 tx=0).stop 2023-10-19T13:00:18.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:18.231+0000 7f255a2cf640 1 -- 172.21.15.57:0/3494190689 shutdown_connections 2023-10-19T13:00:18.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:18.231+0000 7f255a2cf640 1 --2- 172.21.15.57:0/3494190689 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f25540ad370 0x7f25540ad800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:18.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:18.231+0000 7f255a2cf640 1 --2- 172.21.15.57:0/3494190689 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f255406d2d0 0x7f255406d6b0 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:18.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:18.231+0000 7f255a2cf640 1 -- 172.21.15.57:0/3494190689 >> 172.21.15.57:0/3494190689 conn(0x7f25540b0670 msgr2=0x7f25540b0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:18.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:18.231+0000 7f255a2cf640 1 -- 172.21.15.57:0/3494190689 shutdown_connections 2023-10-19T13:00:18.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:18.232+0000 7f255a2cf640 1 -- 172.21.15.57:0/3494190689 wait complete. 2023-10-19T13:00:18.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:18.233+0000 7f255a2cf640 1 Processor -- start 2023-10-19T13:00:18.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:18.233+0000 7f255a2cf640 1 -- start start 2023-10-19T13:00:18.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:18.234+0000 7f255a2cf640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f255406d2d0 0x7f25541ef0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:18.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:18.234+0000 7f255a2cf640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f255406f0a0 0x7f25541524f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:18.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:18.234+0000 7f255a2cf640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25540ad370 0x7f255415b180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:18.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:18.234+0000 7f255a2cf640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25541ef620 con 0x7f25540ad370 2023-10-19T13:00:18.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:18.234+0000 7f255a2cf640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2554152a30 con 0x7f255406f0a0 2023-10-19T13:00:18.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:18.234+0000 7f255a2cf640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25540cb4d0 con 0x7f255406d2d0 2023-10-19T13:00:18.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:18.234+0000 7f255886c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25540ad370 0x7f255415b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:18.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:18.234+0000 7f255886c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25540ad370 0x7f255415b180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:49700/0 (socket says 172.21.15.57:49700) 2023-10-19T13:00:18.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:18.234+0000 7f255886c640 1 -- 172.21.15.57:0/3949433348 learned_addr learned my addr 172.21.15.57:0/3949433348 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:18.243 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:18.234+0000 7f2553fff640 1 --2- 172.21.15.57:0/3949433348 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f255406d2d0 0x7f25541ef0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:18.243 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:18.234+0000 7f25537fe640 1 --2- 172.21.15.57:0/3949433348 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f255406f0a0 0x7f25541524f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:18.243 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:18.234+0000 7f255886c640 1 -- 172.21.15.57:0/3949433348 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f255406d2d0 msgr2=0x7f25541ef0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:18.243 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:18.234+0000 7f255886c640 1 --2- 172.21.15.57:0/3949433348 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f255406d2d0 0x7f25541ef0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:18.243 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:18.234+0000 7f255886c640 1 -- 172.21.15.57:0/3949433348 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f255406f0a0 msgr2=0x7f25541524f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:18.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:18.234+0000 7f255886c640 1 --2- 172.21.15.57:0/3949433348 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f255406f0a0 0x7f25541524f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:18.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:18.234+0000 7f255886c640 1 -- 172.21.15.57:0/3949433348 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2544009560 con 0x7f25540ad370 2023-10-19T13:00:18.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:18.234+0000 7f255886c640 1 --2- 172.21.15.57:0/3949433348 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25540ad370 0x7f255415b180 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f254c00b970 tx=0x7f254c00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:18.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:18.234+0000 7f2550ff9640 1 -- 172.21.15.57:0/3949433348 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f254c00cb40 con 0x7f25540ad370 2023-10-19T13:00:18.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:18.234+0000 7f2550ff9640 1 -- 172.21.15.57:0/3949433348 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f254c00cdb0 con 0x7f25540ad370 2023-10-19T13:00:18.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:18.234+0000 7f255a2cf640 1 -- 172.21.15.57:0/3949433348 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25540cb7b0 con 0x7f25540ad370 2023-10-19T13:00:18.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:18.234+0000 7f2550ff9640 1 -- 172.21.15.57:0/3949433348 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f254c0146a0 con 0x7f25540ad370 2023-10-19T13:00:18.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:18.234+0000 7f255a2cf640 1 -- 172.21.15.57:0/3949433348 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25540cbd70 con 0x7f25540ad370 2023-10-19T13:00:18.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:18.235+0000 7f255a2cf640 1 -- 172.21.15.57:0/3949433348 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25540ce1d0 con 0x7f25540ad370 2023-10-19T13:00:18.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:18.236+0000 7f2550ff9640 1 -- 172.21.15.57:0/3949433348 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 32) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f254c01d3b0 con 0x7f25540ad370 2023-10-19T13:00:18.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:18.236+0000 7f2550ff9640 1 --2- 172.21.15.57:0/3949433348 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f254006af60 0x7f254006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:18.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:18.237+0000 7f2550ff9640 1 -- 172.21.15.57:0/3949433348 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f254c089650 con 0x7f25540ad370 2023-10-19T13:00:18.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:18.237+0000 7f2553fff640 1 --2- 172.21.15.57:0/3949433348 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f254006af60 0x7f254006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:18.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:18.238+0000 7f2553fff640 1 --2- 172.21.15.57:0/3949433348 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f254006af60 0x7f254006d420 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2544035000 tx=0x7f2544004730 comp rx=0 tx=0).ready entity=mgr.6256 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:18.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:18.241+0000 7f2550ff9640 1 -- 172.21.15.57:0/3949433348 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f254c054920 con 0x7f25540ad370 2023-10-19T13:00:18.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:18.420+0000 7f255a2cf640 1 -- 172.21.15.57:0/3949433348 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f25540c25b0 con 0x7f25540ad370 2023-10-19T13:00:19.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:19.171+0000 7f2550ff9640 1 -- 172.21.15.57:0/3949433348 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v84) v1 ==== 138+0+0 (secure 0 0 0) 0x7f254c0190b0 con 0x7f25540ad370 2023-10-19T13:00:19.174 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'cephfs_metadata' created 2023-10-19T13:00:19.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:19.174+0000 7f255a2cf640 1 -- 172.21.15.57:0/3949433348 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f254006af60 msgr2=0x7f254006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:19.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:19.174+0000 7f255a2cf640 1 --2- 172.21.15.57:0/3949433348 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f254006af60 0x7f254006d420 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2544035000 tx=0x7f2544004730 comp rx=0 tx=0).stop 2023-10-19T13:00:19.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:19.174+0000 7f255a2cf640 1 -- 172.21.15.57:0/3949433348 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25540ad370 msgr2=0x7f255415b180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:19.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:19.174+0000 7f255a2cf640 1 --2- 172.21.15.57:0/3949433348 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25540ad370 0x7f255415b180 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f254c00b970 tx=0x7f254c00be30 comp rx=0 tx=0).stop 2023-10-19T13:00:19.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:19.175+0000 7f255a2cf640 1 -- 172.21.15.57:0/3949433348 shutdown_connections 2023-10-19T13:00:19.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:19.175+0000 7f255a2cf640 1 --2- 172.21.15.57:0/3949433348 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f254006af60 0x7f254006d420 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:19.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:19.175+0000 7f255a2cf640 1 --2- 172.21.15.57:0/3949433348 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25540ad370 0x7f255415b180 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:19.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:19.175+0000 7f255a2cf640 1 --2- 172.21.15.57:0/3949433348 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f255406f0a0 0x7f25541524f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:19.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:19.175+0000 7f255a2cf640 1 --2- 172.21.15.57:0/3949433348 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f255406d2d0 0x7f25541ef0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:19.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:19.175+0000 7f255a2cf640 1 -- 172.21.15.57:0/3949433348 >> 172.21.15.57:0/3949433348 conn(0x7f25540b0670 msgr2=0x7f25541591c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:19.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:19.175+0000 7f255a2cf640 1 -- 172.21.15.57:0/3949433348 shutdown_connections 2023-10-19T13:00:19.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:19.176+0000 7f255a2cf640 1 -- 172.21.15.57:0/3949433348 wait complete. 2023-10-19T13:00:19.194 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-19T13:00:19.317 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:19.313+0000 7f6a6d444640 1 -- 172.21.15.57:0/815559012 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6a680c96b0 msgr2=0x7f6a680c9b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:19.317 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:19.313+0000 7f6a6d444640 1 --2- 172.21.15.57:0/815559012 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6a680c96b0 0x7f6a680c9b60 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f6a50005fd0 tx=0x7f6a5002bd90 comp rx=0 tx=0).stop 2023-10-19T13:00:19.317 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:19.314+0000 7f6a6d444640 1 -- 172.21.15.57:0/815559012 shutdown_connections 2023-10-19T13:00:19.317 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:19.314+0000 7f6a6d444640 1 --2- 172.21.15.57:0/815559012 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6a680ab500 0x7f6a680ab990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:19.318 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:19.314+0000 7f6a6d444640 1 --2- 172.21.15.57:0/815559012 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6a680c96b0 0x7f6a680c9b60 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:19.318 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:19.314+0000 7f6a6d444640 1 --2- 172.21.15.57:0/815559012 >> v2:172.21.15.179:3300/0 conn(0x7f6a6806af00 0x7f6a6806b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:19.318 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:19.314+0000 7f6a6d444640 1 -- 172.21.15.57:0/815559012 >> 172.21.15.57:0/815559012 conn(0x7f6a68069930 msgr2=0x7f6a68069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:19.318 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:19.314+0000 7f6a6d444640 1 -- 172.21.15.57:0/815559012 shutdown_connections 2023-10-19T13:00:19.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:19.314+0000 7f6a6d444640 1 -- 172.21.15.57:0/815559012 wait complete. 2023-10-19T13:00:19.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:19.315+0000 7f6a6d444640 1 Processor -- start 2023-10-19T13:00:19.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:19.315+0000 7f6a6d444640 1 -- start start 2023-10-19T13:00:19.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:19.316+0000 7f6a6d444640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6a6806af00 0x7f6a6812f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:19.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:19.316+0000 7f6a6d444640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6a680ab500 0x7f6a680aa610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:19.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:19.316+0000 7f6a6d444640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6a680c96b0 0x7f6a680a8520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:19.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:19.316+0000 7f6a6d444640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a680a94b0 con 0x7f6a680ab500 2023-10-19T13:00:19.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:19.316+0000 7f6a6d444640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a6812fae0 con 0x7f6a680c96b0 2023-10-19T13:00:19.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:19.316+0000 7f6a6d444640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a680ab090 con 0x7f6a6806af00 2023-10-19T13:00:19.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:19.316+0000 7f6a5e7fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6a680ab500 0x7f6a680aa610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:19.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:19.316+0000 7f6a5e7fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6a680ab500 0x7f6a680aa610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:49714/0 (socket says 172.21.15.57:49714) 2023-10-19T13:00:19.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:19.316+0000 7f6a5e7fc640 1 -- 172.21.15.57:0/3119968322 learned_addr learned my addr 172.21.15.57:0/3119968322 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:19.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:19.316+0000 7f6a66ffd640 1 --2- 172.21.15.57:0/3119968322 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6a6806af00 0x7f6a6812f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:19.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:19.316+0000 7f6a677fe640 1 --2- 172.21.15.57:0/3119968322 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6a680c96b0 0x7f6a680a8520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:19.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:19.316+0000 7f6a5e7fc640 1 -- 172.21.15.57:0/3119968322 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6a6806af00 msgr2=0x7f6a6812f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:19.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:19.316+0000 7f6a5e7fc640 1 --2- 172.21.15.57:0/3119968322 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6a6806af00 0x7f6a6812f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:19.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:19.316+0000 7f6a5e7fc640 1 -- 172.21.15.57:0/3119968322 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6a680c96b0 msgr2=0x7f6a680a8520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:19.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:19.316+0000 7f6a5e7fc640 1 --2- 172.21.15.57:0/3119968322 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6a680c96b0 0x7f6a680a8520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:19.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:19.316+0000 7f6a5e7fc640 1 -- 172.21.15.57:0/3119968322 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a50009560 con 0x7f6a680ab500 2023-10-19T13:00:19.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:19.316+0000 7f6a66ffd640 1 --2- 172.21.15.57:0/3119968322 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6a6806af00 0x7f6a6812f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:00:19.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:19.316+0000 7f6a5e7fc640 1 --2- 172.21.15.57:0/3119968322 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6a680ab500 0x7f6a680aa610 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f6a500099f0 tx=0x7f6a5002db30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:19.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:19.316+0000 7f6a5ffff640 1 -- 172.21.15.57:0/3119968322 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a50003800 con 0x7f6a680ab500 2023-10-19T13:00:19.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:19.316+0000 7f6a5ffff640 1 -- 172.21.15.57:0/3119968322 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f6a50003a50 con 0x7f6a680ab500 2023-10-19T13:00:19.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:19.316+0000 7f6a6d444640 1 -- 172.21.15.57:0/3119968322 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a680ae880 con 0x7f6a680ab500 2023-10-19T13:00:19.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:19.316+0000 7f6a5ffff640 1 -- 172.21.15.57:0/3119968322 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a50003e60 con 0x7f6a680ab500 2023-10-19T13:00:19.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:19.317+0000 7f6a6d444640 1 -- 172.21.15.57:0/3119968322 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a680aee40 con 0x7f6a680ab500 2023-10-19T13:00:19.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:19.318+0000 7f6a6d444640 1 -- 172.21.15.57:0/3119968322 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a680af030 con 0x7f6a680ab500 2023-10-19T13:00:19.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:19.318+0000 7f6a5ffff640 1 -- 172.21.15.57:0/3119968322 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 32) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6a5003e450 con 0x7f6a680ab500 2023-10-19T13:00:19.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:19.319+0000 7f6a5ffff640 1 --2- 172.21.15.57:0/3119968322 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f6a4806af60 0x7f6a4806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:19.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:19.319+0000 7f6a5ffff640 1 -- 172.21.15.57:0/3119968322 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5575+0+0 (secure 0 0 0) 0x7f6a500ac660 con 0x7f6a680ab500 2023-10-19T13:00:19.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:19.319+0000 7f6a66ffd640 1 --2- 172.21.15.57:0/3119968322 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f6a4806af60 0x7f6a4806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:19.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:19.320+0000 7f6a66ffd640 1 --2- 172.21.15.57:0/3119968322 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f6a4806af60 0x7f6a4806d420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f6a580058c0 tx=0x7f6a58005850 comp rx=0 tx=0).ready entity=mgr.6256 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:19.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:19.323+0000 7f6a5ffff640 1 -- 172.21.15.57:0/3119968322 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6a500775e0 con 0x7f6a680ab500 2023-10-19T13:00:19.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:19.499+0000 7f6a6d444640 1 -- 172.21.15.57:0/3119968322 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f6a680abe50 con 0x7f6a680ab500 2023-10-19T13:00:20.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:20.177+0000 7f6a5ffff640 1 -- 172.21.15.57:0/3119968322 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v85) v1 ==== 170+0+0 (secure 0 0 0) 0x7f6a500420d0 con 0x7f6a680ab500 2023-10-19T13:00:20.180 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'cephfs_data' created 2023-10-19T13:00:20.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:20.181+0000 7f6a6d444640 1 -- 172.21.15.57:0/3119968322 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f6a4806af60 msgr2=0x7f6a4806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:20.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:20.181+0000 7f6a6d444640 1 --2- 172.21.15.57:0/3119968322 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f6a4806af60 0x7f6a4806d420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f6a580058c0 tx=0x7f6a58005850 comp rx=0 tx=0).stop 2023-10-19T13:00:20.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:20.181+0000 7f6a6d444640 1 -- 172.21.15.57:0/3119968322 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6a680ab500 msgr2=0x7f6a680aa610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:20.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:20.181+0000 7f6a6d444640 1 --2- 172.21.15.57:0/3119968322 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6a680ab500 0x7f6a680aa610 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f6a500099f0 tx=0x7f6a5002db30 comp rx=0 tx=0).stop 2023-10-19T13:00:20.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:20.182+0000 7f6a6d444640 1 -- 172.21.15.57:0/3119968322 shutdown_connections 2023-10-19T13:00:20.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:20.182+0000 7f6a6d444640 1 --2- 172.21.15.57:0/3119968322 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f6a4806af60 0x7f6a4806d420 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:20.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:20.182+0000 7f6a6d444640 1 --2- 172.21.15.57:0/3119968322 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6a680c96b0 0x7f6a680a8520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:20.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:20.182+0000 7f6a6d444640 1 --2- 172.21.15.57:0/3119968322 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6a680ab500 0x7f6a680aa610 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:20.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:20.182+0000 7f6a6d444640 1 --2- 172.21.15.57:0/3119968322 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6a6806af00 0x7f6a6812f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:20.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:20.182+0000 7f6a6d444640 1 -- 172.21.15.57:0/3119968322 >> 172.21.15.57:0/3119968322 conn(0x7f6a68069930 msgr2=0x7f6a6813b1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:20.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:20.183+0000 7f6a6d444640 1 -- 172.21.15.57:0/3119968322 shutdown_connections 2023-10-19T13:00:20.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:20.184+0000 7f6a6d444640 1 -- 172.21.15.57:0/3119968322 wait complete. 2023-10-19T13:00:20.210 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-19T13:00:20.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:20.324+0000 7f752ba21640 1 -- 172.21.15.57:0/2955603245 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7524158d00 msgr2=0x7f752406daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:20.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:20.324+0000 7f752ba21640 1 --2- 172.21.15.57:0/2955603245 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7524158d00 0x7f752406daf0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f75240714f0 tx=0x7f751802bdb0 comp rx=0 tx=0).stop 2023-10-19T13:00:20.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:20.324+0000 7f752ba21640 1 -- 172.21.15.57:0/2955603245 shutdown_connections 2023-10-19T13:00:20.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:20.324+0000 7f752ba21640 1 --2- 172.21.15.57:0/2955603245 >> v2:172.21.15.179:3300/0 conn(0x7f752413b490 0x7f752413b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:20.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:20.324+0000 7f752ba21640 1 --2- 172.21.15.57:0/2955603245 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7524158d00 0x7f752406daf0 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:20.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:20.324+0000 7f752ba21640 1 -- 172.21.15.57:0/2955603245 >> 172.21.15.57:0/2955603245 conn(0x7f75240af7a0 msgr2=0x7f75240afb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:20.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:20.324+0000 7f752ba21640 1 -- 172.21.15.57:0/2955603245 shutdown_connections 2023-10-19T13:00:20.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:20.324+0000 7f752ba21640 1 -- 172.21.15.57:0/2955603245 wait complete. 2023-10-19T13:00:20.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:20.326+0000 7f752ba21640 1 Processor -- start 2023-10-19T13:00:20.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:20.326+0000 7f752ba21640 1 -- start start 2023-10-19T13:00:20.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:20.326+0000 7f752ba21640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f752413ad40 0x7f75240ca300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:20.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:20.326+0000 7f752ba21640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f752413b490 0x7f75240ab250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:20.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:20.326+0000 7f752ba21640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7524158d00 0x7f75241f1620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:20.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:20.326+0000 7f752ba21640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75240ad1c0 con 0x7f7524158d00 2023-10-19T13:00:20.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:20.326+0000 7f752ba21640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75240b1340 con 0x7f752413ad40 2023-10-19T13:00:20.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:20.326+0000 7f752ba21640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75240ca840 con 0x7f752413b490 2023-10-19T13:00:20.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:20.326+0000 7f7529fbe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7524158d00 0x7f75241f1620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:20.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:20.326+0000 7f7529fbe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7524158d00 0x7f75241f1620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:49728/0 (socket says 172.21.15.57:49728) 2023-10-19T13:00:20.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:20.326+0000 7f7529fbe640 1 -- 172.21.15.57:0/2044607943 learned_addr learned my addr 172.21.15.57:0/2044607943 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:20.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:20.327+0000 7f7528fbc640 1 --2- 172.21.15.57:0/2044607943 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f752413b490 0x7f75240ab250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:20.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:20.327+0000 7f75297bd640 1 --2- 172.21.15.57:0/2044607943 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f752413ad40 0x7f75240ca300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:20.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:20.327+0000 7f7529fbe640 1 -- 172.21.15.57:0/2044607943 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f752413b490 msgr2=0x7f75240ab250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:20.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:20.327+0000 7f7529fbe640 1 --2- 172.21.15.57:0/2044607943 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f752413b490 0x7f75240ab250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:20.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:20.327+0000 7f7529fbe640 1 -- 172.21.15.57:0/2044607943 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f752413ad40 msgr2=0x7f75240ca300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:20.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:20.327+0000 7f7529fbe640 1 --2- 172.21.15.57:0/2044607943 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f752413ad40 0x7f75240ca300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:20.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:20.327+0000 7f7529fbe640 1 -- 172.21.15.57:0/2044607943 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7518009560 con 0x7f7524158d00 2023-10-19T13:00:20.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:20.327+0000 7f75297bd640 1 --2- 172.21.15.57:0/2044607943 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f752413ad40 0x7f75240ca300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:00:20.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:20.327+0000 7f7529fbe640 1 --2- 172.21.15.57:0/2044607943 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7524158d00 0x7f75241f1620 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f751c009fd0 tx=0x7f751c00b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:20.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:20.327+0000 7f7521ffb640 1 -- 172.21.15.57:0/2044607943 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f751c011830 con 0x7f7524158d00 2023-10-19T13:00:20.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:20.327+0000 7f7521ffb640 1 -- 172.21.15.57:0/2044607943 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f751c011a80 con 0x7f7524158d00 2023-10-19T13:00:20.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:20.327+0000 7f752ba21640 1 -- 172.21.15.57:0/2044607943 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75240aac80 con 0x7f7524158d00 2023-10-19T13:00:20.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:20.327+0000 7f7521ffb640 1 -- 172.21.15.57:0/2044607943 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f751c011e70 con 0x7f7524158d00 2023-10-19T13:00:20.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:20.327+0000 7f752ba21640 1 -- 172.21.15.57:0/2044607943 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75240b18c0 con 0x7f7524158d00 2023-10-19T13:00:20.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:20.328+0000 7f752ba21640 1 -- 172.21.15.57:0/2044607943 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75240b1ab0 con 0x7f7524158d00 2023-10-19T13:00:20.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:20.329+0000 7f7521ffb640 1 -- 172.21.15.57:0/2044607943 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 32) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f751c01b460 con 0x7f7524158d00 2023-10-19T13:00:20.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:20.329+0000 7f7521ffb640 1 --2- 172.21.15.57:0/2044607943 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f751006af60 0x7f751006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:20.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:20.330+0000 7f7521ffb640 1 -- 172.21.15.57:0/2044607943 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5957+0+0 (secure 0 0 0) 0x7f751c089880 con 0x7f7524158d00 2023-10-19T13:00:20.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:20.330+0000 7f75297bd640 1 --2- 172.21.15.57:0/2044607943 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f751006af60 0x7f751006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:20.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:20.330+0000 7f75297bd640 1 --2- 172.21.15.57:0/2044607943 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f751006af60 0x7f751006d420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f7514001e50 tx=0x7f7514009040 comp rx=0 tx=0).ready entity=mgr.6256 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:20.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:20.334+0000 7f7521ffb640 1 -- 172.21.15.57:0/2044607943 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f751c0581b0 con 0x7f7524158d00 2023-10-19T13:00:20.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:20.547+0000 7f752ba21640 1 -- 172.21.15.57:0/2044607943 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f75240690e0 con 0x7f7524158d00 2023-10-19T13:00:21.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.208+0000 7f7521ffb640 1 -- 172.21.15.57:0/2044607943 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 9 and data pool 10 v28) v1 ==== 173+0+0 (secure 0 0 0) 0x7f751c057d40 con 0x7f7524158d00 2023-10-19T13:00:21.211 INFO:teuthology.orchestra.run.smithi057.stderr:new fs with metadata pool 9 and data pool 10 2023-10-19T13:00:21.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.221+0000 7f752ba21640 1 -- 172.21.15.57:0/2044607943 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f751006af60 msgr2=0x7f751006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:21.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.221+0000 7f752ba21640 1 --2- 172.21.15.57:0/2044607943 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f751006af60 0x7f751006d420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f7514001e50 tx=0x7f7514009040 comp rx=0 tx=0).stop 2023-10-19T13:00:21.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.222+0000 7f752ba21640 1 -- 172.21.15.57:0/2044607943 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7524158d00 msgr2=0x7f75241f1620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:21.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.222+0000 7f752ba21640 1 --2- 172.21.15.57:0/2044607943 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7524158d00 0x7f75241f1620 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f751c009fd0 tx=0x7f751c00b8e0 comp rx=0 tx=0).stop 2023-10-19T13:00:21.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.222+0000 7f752ba21640 1 -- 172.21.15.57:0/2044607943 shutdown_connections 2023-10-19T13:00:21.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.223+0000 7f752ba21640 1 --2- 172.21.15.57:0/2044607943 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f751006af60 0x7f751006d420 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:21.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.223+0000 7f752ba21640 1 --2- 172.21.15.57:0/2044607943 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7524158d00 0x7f75241f1620 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:21.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.223+0000 7f752ba21640 1 --2- 172.21.15.57:0/2044607943 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f752413b490 0x7f75240ab250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:21.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.223+0000 7f752ba21640 1 --2- 172.21.15.57:0/2044607943 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f752413ad40 0x7f75240ca300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:21.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.223+0000 7f752ba21640 1 -- 172.21.15.57:0/2044607943 >> 172.21.15.57:0/2044607943 conn(0x7f75240af7a0 msgr2=0x7f752413ccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:21.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.224+0000 7f752ba21640 1 -- 172.21.15.57:0/2044607943 shutdown_connections 2023-10-19T13:00:21.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.225+0000 7f752ba21640 1 -- 172.21.15.57:0/2044607943 wait complete. 2023-10-19T13:00:21.246 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T13:00:21.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.356+0000 7fa866fe7640 1 -- 172.21.15.57:0/1154872307 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa8600b0530 msgr2=0x7fa8600b09c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:21.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.356+0000 7fa866fe7640 1 --2- 172.21.15.57:0/1154872307 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa8600b0530 0x7fa8600b09c0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7fa858005fd0 tx=0x7fa85802c2e0 comp rx=0 tx=0).stop 2023-10-19T13:00:21.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.356+0000 7fa866fe7640 1 -- 172.21.15.57:0/1154872307 shutdown_connections 2023-10-19T13:00:21.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.356+0000 7fa866fe7640 1 --2- 172.21.15.57:0/1154872307 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa8600b0530 0x7fa8600b09c0 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:21.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.356+0000 7fa866fe7640 1 --2- 172.21.15.57:0/1154872307 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa8600afa20 0x7fa860158300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:21.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.356+0000 7fa866fe7640 1 -- 172.21.15.57:0/1154872307 >> 172.21.15.57:0/1154872307 conn(0x7fa860069930 msgr2=0x7fa860069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:21.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.356+0000 7fa866fe7640 1 -- 172.21.15.57:0/1154872307 shutdown_connections 2023-10-19T13:00:21.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.356+0000 7fa866fe7640 1 -- 172.21.15.57:0/1154872307 wait complete. 2023-10-19T13:00:21.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.357+0000 7fa866fe7640 1 Processor -- start 2023-10-19T13:00:21.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.357+0000 7fa866fe7640 1 -- start start 2023-10-19T13:00:21.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.358+0000 7fa866fe7640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa8600afa20 0x7fa86013b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:21.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.358+0000 7fa866fe7640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa8600b0530 0x7fa86014daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:21.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.358+0000 7fa866fe7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa8600cbd20 0x7fa86014fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:21.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.358+0000 7fa866fe7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa86013b680 con 0x7fa8600cbd20 2023-10-19T13:00:21.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.358+0000 7fa866fe7640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa86014e3f0 con 0x7fa8600afa20 2023-10-19T13:00:21.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.358+0000 7fa866fe7640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa86014e6d0 con 0x7fa8600b0530 2023-10-19T13:00:21.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.358+0000 7fa865584640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa8600cbd20 0x7fa86014fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:21.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.358+0000 7fa865584640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa8600cbd20 0x7fa86014fd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:49752/0 (socket says 172.21.15.57:49752) 2023-10-19T13:00:21.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.358+0000 7fa865584640 1 -- 172.21.15.57:0/2916369477 learned_addr learned my addr 172.21.15.57:0/2916369477 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:21.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.358+0000 7fa864d83640 1 --2- 172.21.15.57:0/2916369477 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa8600afa20 0x7fa86013b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:21.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.358+0000 7fa857fff640 1 --2- 172.21.15.57:0/2916369477 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa8600b0530 0x7fa86014daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:21.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.358+0000 7fa865584640 1 -- 172.21.15.57:0/2916369477 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa8600b0530 msgr2=0x7fa86014daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:21.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.358+0000 7fa865584640 1 --2- 172.21.15.57:0/2916369477 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa8600b0530 0x7fa86014daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:21.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.358+0000 7fa865584640 1 -- 172.21.15.57:0/2916369477 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa8600afa20 msgr2=0x7fa86013b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:21.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.358+0000 7fa865584640 1 --2- 172.21.15.57:0/2916369477 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa8600afa20 0x7fa86013b140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:21.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.358+0000 7fa865584640 1 -- 172.21.15.57:0/2916369477 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa858009900 con 0x7fa8600cbd20 2023-10-19T13:00:21.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.358+0000 7fa865584640 1 --2- 172.21.15.57:0/2916369477 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa8600cbd20 0x7fa86014fd30 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7fa858037040 tx=0x7fa858035ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:21.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.358+0000 7fa8557fa640 1 -- 172.21.15.57:0/2916369477 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa858004240 con 0x7fa8600cbd20 2023-10-19T13:00:21.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.358+0000 7fa8557fa640 1 -- 172.21.15.57:0/2916369477 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fa858004490 con 0x7fa8600cbd20 2023-10-19T13:00:21.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.358+0000 7fa866fe7640 1 -- 172.21.15.57:0/2916369477 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa86014e9b0 con 0x7fa8600cbd20 2023-10-19T13:00:21.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.358+0000 7fa8557fa640 1 -- 172.21.15.57:0/2916369477 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa858004820 con 0x7fa8600cbd20 2023-10-19T13:00:21.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.359+0000 7fa866fe7640 1 -- 172.21.15.57:0/2916369477 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa86014ef70 con 0x7fa8600cbd20 2023-10-19T13:00:21.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.359+0000 7fa866fe7640 1 -- 172.21.15.57:0/2916369477 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8600690e0 con 0x7fa8600cbd20 2023-10-19T13:00:21.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.360+0000 7fa8557fa640 1 -- 172.21.15.57:0/2916369477 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 32) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fa85803b040 con 0x7fa8600cbd20 2023-10-19T13:00:21.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.360+0000 7fa8557fa640 1 --2- 172.21.15.57:0/2916369477 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7fa84c06af60 0x7fa84c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:21.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.360+0000 7fa8557fa640 1 -- 172.21.15.57:0/2916369477 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fa8580ac420 con 0x7fa8600cbd20 2023-10-19T13:00:21.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.361+0000 7fa864d83640 1 --2- 172.21.15.57:0/2916369477 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7fa84c06af60 0x7fa84c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:21.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.361+0000 7fa864d83640 1 --2- 172.21.15.57:0/2916369477 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7fa84c06af60 0x7fa84c06d420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa85c000f60 tx=0x7fa85c009040 comp rx=0 tx=0).ready entity=mgr.6256 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:21.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.365+0000 7fa8557fa640 1 -- 172.21.15.57:0/2916369477 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa8580773a0 con 0x7fa8600cbd20 2023-10-19T13:00:21.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.540+0000 7fa866fe7640 1 -- 172.21.15.57:0/2916369477 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa8600700d0 con 0x7fa8600cbd20 2023-10-19T13:00:21.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.541+0000 7fa8557fa640 1 -- 172.21.15.57:0/2916369477 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+16784 (secure 0 0 0) 0x7fa85807a9c0 con 0x7fa8600cbd20 2023-10-19T13:00:21.545 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:00:21.545 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":86,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T13:00:20.548947+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2023-10-19T13:00:18.452198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2023-10-19T13:00:19.531525+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T13:00:21.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.543+0000 7fa866fe7640 1 -- 172.21.15.57:0/2916369477 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7fa84c06af60 msgr2=0x7fa84c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:21.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.543+0000 7fa866fe7640 1 --2- 172.21.15.57:0/2916369477 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7fa84c06af60 0x7fa84c06d420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa85c000f60 tx=0x7fa85c009040 comp rx=0 tx=0).stop 2023-10-19T13:00:21.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.543+0000 7fa866fe7640 1 -- 172.21.15.57:0/2916369477 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa8600cbd20 msgr2=0x7fa86014fd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:21.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.543+0000 7fa866fe7640 1 --2- 172.21.15.57:0/2916369477 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa8600cbd20 0x7fa86014fd30 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7fa858037040 tx=0x7fa858035ad0 comp rx=0 tx=0).stop 2023-10-19T13:00:21.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.543+0000 7fa866fe7640 1 -- 172.21.15.57:0/2916369477 shutdown_connections 2023-10-19T13:00:21.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.543+0000 7fa866fe7640 1 --2- 172.21.15.57:0/2916369477 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7fa84c06af60 0x7fa84c06d420 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:21.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.543+0000 7fa866fe7640 1 --2- 172.21.15.57:0/2916369477 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa8600cbd20 0x7fa86014fd30 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:21.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.543+0000 7fa866fe7640 1 --2- 172.21.15.57:0/2916369477 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa8600b0530 0x7fa86014daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:21.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.543+0000 7fa866fe7640 1 --2- 172.21.15.57:0/2916369477 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa8600afa20 0x7fa86013b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:21.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.543+0000 7fa866fe7640 1 -- 172.21.15.57:0/2916369477 >> 172.21.15.57:0/2916369477 conn(0x7fa860069930 msgr2=0x7fa8600ac930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:21.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.543+0000 7fa866fe7640 1 -- 172.21.15.57:0/2916369477 shutdown_connections 2023-10-19T13:00:21.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.543+0000 7fa866fe7640 1 -- 172.21.15.57:0/2916369477 wait complete. 2023-10-19T13:00:21.557 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T13:00:21.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.664+0000 7fa58efa0640 1 -- 172.21.15.57:0/2677024827 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa588070870 msgr2=0x7fa58806ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:21.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.664+0000 7fa58efa0640 1 --2- 172.21.15.57:0/2677024827 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa588070870 0x7fa58806ccd0 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7fa5880713f0 tx=0x7fa57802c2f0 comp rx=0 tx=0).stop 2023-10-19T13:00:21.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.664+0000 7fa58efa0640 1 -- 172.21.15.57:0/2677024827 shutdown_connections 2023-10-19T13:00:21.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.664+0000 7fa58efa0640 1 --2- 172.21.15.57:0/2677024827 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa588070870 0x7fa58806ccd0 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:21.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.664+0000 7fa58efa0640 1 --2- 172.21.15.57:0/2677024827 >> v2:172.21.15.179:3300/0 conn(0x7fa588156ff0 0x7fa5880b0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:21.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.664+0000 7fa58efa0640 1 -- 172.21.15.57:0/2677024827 >> 172.21.15.57:0/2677024827 conn(0x7fa5880ae5d0 msgr2=0x7fa5880ae9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:21.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.664+0000 7fa58efa0640 1 -- 172.21.15.57:0/2677024827 shutdown_connections 2023-10-19T13:00:21.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.664+0000 7fa58efa0640 1 -- 172.21.15.57:0/2677024827 wait complete. 2023-10-19T13:00:21.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.665+0000 7fa58efa0640 1 Processor -- start 2023-10-19T13:00:21.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.665+0000 7fa58efa0640 1 -- start start 2023-10-19T13:00:21.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.666+0000 7fa58efa0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa58806c840 0x7fa5881490a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:21.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.666+0000 7fa58efa0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa588070870 0x7fa5880aa8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:21.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.666+0000 7fa58efa0640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa588156ff0 0x7fa5880aae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:21.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.666+0000 7fa58efa0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5881495e0 con 0x7fa588070870 2023-10-19T13:00:21.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.666+0000 7fa58efa0640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa588158530 con 0x7fa58806c840 2023-10-19T13:00:21.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.666+0000 7fa58efa0640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa588158810 con 0x7fa588156ff0 2023-10-19T13:00:21.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.666+0000 7fa58d79d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa588070870 0x7fa5880aa8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:21.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.666+0000 7fa58d79d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa588070870 0x7fa5880aa8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:49776/0 (socket says 172.21.15.57:49776) 2023-10-19T13:00:21.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.666+0000 7fa58d79d640 1 -- 172.21.15.57:0/508755768 learned_addr learned my addr 172.21.15.57:0/508755768 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:21.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.666+0000 7fa58df9e640 1 --2- 172.21.15.57:0/508755768 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa58806c840 0x7fa5881490a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:21.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.666+0000 7fa58d79d640 1 -- 172.21.15.57:0/508755768 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa588156ff0 msgr2=0x7fa5880aae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:21.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.666+0000 7fa58e79f640 1 --2- 172.21.15.57:0/508755768 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa588156ff0 0x7fa5880aae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:21.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.666+0000 7fa58d79d640 1 --2- 172.21.15.57:0/508755768 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa588156ff0 0x7fa5880aae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:21.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.666+0000 7fa58d79d640 1 -- 172.21.15.57:0/508755768 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa58806c840 msgr2=0x7fa5881490a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:21.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.666+0000 7fa58d79d640 1 --2- 172.21.15.57:0/508755768 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa58806c840 0x7fa5881490a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:21.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.666+0000 7fa58d79d640 1 -- 172.21.15.57:0/508755768 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa578009900 con 0x7fa588070870 2023-10-19T13:00:21.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.667+0000 7fa58df9e640 1 --2- 172.21.15.57:0/508755768 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa58806c840 0x7fa5881490a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T13:00:21.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.667+0000 7fa58d79d640 1 --2- 172.21.15.57:0/508755768 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa588070870 0x7fa5880aa8c0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7fa57c00c450 tx=0x7fa57c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:21.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.667+0000 7fa5867fc640 1 -- 172.21.15.57:0/508755768 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa57c0052d0 con 0x7fa588070870 2023-10-19T13:00:21.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.667+0000 7fa5867fc640 1 -- 172.21.15.57:0/508755768 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fa57c005430 con 0x7fa588070870 2023-10-19T13:00:21.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.667+0000 7fa58efa0640 1 -- 172.21.15.57:0/508755768 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa588158af0 con 0x7fa588070870 2023-10-19T13:00:21.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.667+0000 7fa5867fc640 1 -- 172.21.15.57:0/508755768 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa57c012890 con 0x7fa588070870 2023-10-19T13:00:21.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.667+0000 7fa58efa0640 1 -- 172.21.15.57:0/508755768 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5881590b0 con 0x7fa588070870 2023-10-19T13:00:21.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.668+0000 7fa58efa0640 1 -- 172.21.15.57:0/508755768 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5881592d0 con 0x7fa588070870 2023-10-19T13:00:21.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.668+0000 7fa5867fc640 1 -- 172.21.15.57:0/508755768 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 32) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fa57c0035f0 con 0x7fa588070870 2023-10-19T13:00:21.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.669+0000 7fa5867fc640 1 --2- 172.21.15.57:0/508755768 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7fa57406af60 0x7fa57406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:21.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.669+0000 7fa5867fc640 1 -- 172.21.15.57:0/508755768 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fa57c089670 con 0x7fa588070870 2023-10-19T13:00:21.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.669+0000 7fa58df9e640 1 --2- 172.21.15.57:0/508755768 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7fa57406af60 0x7fa57406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:21.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.670+0000 7fa58df9e640 1 --2- 172.21.15.57:0/508755768 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7fa57406af60 0x7fa57406d420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa58000cb50 tx=0x7fa580005db0 comp rx=0 tx=0).ready entity=mgr.6256 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:21.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.673+0000 7fa5867fc640 1 -- 172.21.15.57:0/508755768 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa57c054410 con 0x7fa588070870 2023-10-19T13:00:21.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.853+0000 7fa58efa0640 1 -- 172.21.15.57:0/508755768 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa58804da70 con 0x7fa588070870 2023-10-19T13:00:21.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.854+0000 7fa5867fc640 1 -- 172.21.15.57:0/508755768 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+16784 (secure 0 0 0) 0x7fa57c057a30 con 0x7fa588070870 2023-10-19T13:00:21.857 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:00:21.857 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":86,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T13:00:20.548947+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2023-10-19T13:00:18.452198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2023-10-19T13:00:19.531525+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T13:00:21.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.856+0000 7fa58efa0640 1 -- 172.21.15.57:0/508755768 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7fa57406af60 msgr2=0x7fa57406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:21.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.856+0000 7fa58efa0640 1 --2- 172.21.15.57:0/508755768 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7fa57406af60 0x7fa57406d420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa58000cb50 tx=0x7fa580005db0 comp rx=0 tx=0).stop 2023-10-19T13:00:21.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.856+0000 7fa58efa0640 1 -- 172.21.15.57:0/508755768 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa588070870 msgr2=0x7fa5880aa8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:21.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.856+0000 7fa58efa0640 1 --2- 172.21.15.57:0/508755768 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa588070870 0x7fa5880aa8c0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7fa57c00c450 tx=0x7fa57c00c910 comp rx=0 tx=0).stop 2023-10-19T13:00:21.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.856+0000 7fa58efa0640 1 -- 172.21.15.57:0/508755768 shutdown_connections 2023-10-19T13:00:21.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.856+0000 7fa58efa0640 1 --2- 172.21.15.57:0/508755768 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7fa57406af60 0x7fa57406d420 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:21.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.856+0000 7fa58efa0640 1 --2- 172.21.15.57:0/508755768 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa588156ff0 0x7fa5880aae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:21.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.856+0000 7fa58efa0640 1 --2- 172.21.15.57:0/508755768 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa588070870 0x7fa5880aa8c0 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:21.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.856+0000 7fa58efa0640 1 --2- 172.21.15.57:0/508755768 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa58806c840 0x7fa5881490a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:21.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.856+0000 7fa58efa0640 1 -- 172.21.15.57:0/508755768 >> 172.21.15.57:0/508755768 conn(0x7fa5880ae5d0 msgr2=0x7fa58806f870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:21.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.856+0000 7fa58efa0640 1 -- 172.21.15.57:0/508755768 shutdown_connections 2023-10-19T13:00:21.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.856+0000 7fa58efa0640 1 -- 172.21.15.57:0/508755768 wait complete. 2023-10-19T13:00:21.870 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-19T13:00:21.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.980+0000 7f18ac051640 1 -- 172.21.15.57:0/1176974329 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f188c004d00 msgr2=0x7f188c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:21.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.980+0000 7f18ac051640 1 --2- 172.21.15.57:0/1176974329 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f188c004d00 0x7f188c005180 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f189400c610 tx=0x7f188c0057f0 comp rx=0 tx=0).stop 2023-10-19T13:00:21.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.980+0000 7f18aa5ee640 1 -- 172.21.15.57:0/1176974329 reap_dead start 2023-10-19T13:00:21.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.980+0000 7f18ac051640 1 -- 172.21.15.57:0/1176974329 shutdown_connections 2023-10-19T13:00:21.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.980+0000 7f18ac051640 1 -- 172.21.15.57:0/1176974329 >> 172.21.15.57:0/1176974329 conn(0x7f18a40d0d60 msgr2=0x7f18a40d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:21.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.980+0000 7f18ac051640 1 -- 172.21.15.57:0/1176974329 shutdown_connections 2023-10-19T13:00:21.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.980+0000 7f18ac051640 1 -- 172.21.15.57:0/1176974329 wait complete. 2023-10-19T13:00:21.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.981+0000 7f18ac051640 1 Processor -- start 2023-10-19T13:00:21.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.981+0000 7f18ac051640 1 -- start start 2023-10-19T13:00:21.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.982+0000 7f18ac051640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f18a40b5fa0 0x7f18a41f56d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:21.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.982+0000 7f18ac051640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f18a4152100 0x7f18a41f0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:21.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.982+0000 7f18ac051640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f18a40b6380 0x7f18a41f1000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:21.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.982+0000 7f18ac051640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18a415e130 con 0x7f18a40b6380 2023-10-19T13:00:21.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.982+0000 7f18ac051640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18a41531b0 con 0x7f18a4152100 2023-10-19T13:00:21.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.982+0000 7f18ac051640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18a41f5c10 con 0x7f18a40b5fa0 2023-10-19T13:00:21.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.982+0000 7f18aa5ee640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f18a40b6380 0x7f18a41f1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:21.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.982+0000 7f18aa5ee640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f18a40b6380 0x7f18a41f1000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:49784/0 (socket says 172.21.15.57:49784) 2023-10-19T13:00:21.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.982+0000 7f18aa5ee640 1 -- 172.21.15.57:0/2746374201 learned_addr learned my addr 172.21.15.57:0/2746374201 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:21.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.982+0000 7f18a95ec640 1 --2- 172.21.15.57:0/2746374201 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f18a4152100 0x7f18a41f0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:21.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.982+0000 7f18a9ded640 1 --2- 172.21.15.57:0/2746374201 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f18a40b5fa0 0x7f18a41f56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:21.989 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.982+0000 7f18aa5ee640 1 -- 172.21.15.57:0/2746374201 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f18a40b5fa0 msgr2=0x7f18a41f56d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:21.989 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.982+0000 7f18aa5ee640 1 --2- 172.21.15.57:0/2746374201 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f18a40b5fa0 0x7f18a41f56d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:21.989 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.982+0000 7f18aa5ee640 1 -- 172.21.15.57:0/2746374201 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f18a4152100 msgr2=0x7f18a41f0ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:21.989 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.982+0000 7f18aa5ee640 1 --2- 172.21.15.57:0/2746374201 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f18a4152100 0x7f18a41f0ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:21.989 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.982+0000 7f18aa5ee640 1 -- 172.21.15.57:0/2746374201 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18a41f1540 con 0x7f18a40b6380 2023-10-19T13:00:21.989 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.982+0000 7f18a95ec640 1 --2- 172.21.15.57:0/2746374201 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f18a4152100 0x7f18a41f0ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:00:21.990 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.982+0000 7f18aa5ee640 1 --2- 172.21.15.57:0/2746374201 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f18a40b6380 0x7f18a41f1000 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f189402cbe0 tx=0x7f18940020a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:21.990 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.983+0000 7f18a27fc640 1 -- 172.21.15.57:0/2746374201 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1894003a50 con 0x7f18a40b6380 2023-10-19T13:00:21.990 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.983+0000 7f18a27fc640 1 -- 172.21.15.57:0/2746374201 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f1894003bb0 con 0x7f18a40b6380 2023-10-19T13:00:21.990 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.983+0000 7f18ac051640 1 -- 172.21.15.57:0/2746374201 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18a41525a0 con 0x7f18a40b6380 2023-10-19T13:00:21.990 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.983+0000 7f18a27fc640 1 -- 172.21.15.57:0/2746374201 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18940060e0 con 0x7f18a40b6380 2023-10-19T13:00:21.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.983+0000 7f18ac051640 1 -- 172.21.15.57:0/2746374201 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18a4152b60 con 0x7f18a40b6380 2023-10-19T13:00:21.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.984+0000 7f18ac051640 1 -- 172.21.15.57:0/2746374201 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18a40690e0 con 0x7f18a40b6380 2023-10-19T13:00:21.992 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.984+0000 7f18a27fc640 1 -- 172.21.15.57:0/2746374201 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 32) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f1894031030 con 0x7f18a40b6380 2023-10-19T13:00:21.992 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.985+0000 7f18a27fc640 1 --2- 172.21.15.57:0/2746374201 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f189006af60 0x7f189006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:21.992 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.985+0000 7f18a27fc640 1 -- 172.21.15.57:0/2746374201 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f18940aa5f0 con 0x7f18a40b6380 2023-10-19T13:00:21.992 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.985+0000 7f18a9ded640 1 --2- 172.21.15.57:0/2746374201 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f189006af60 0x7f189006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:21.992 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.985+0000 7f18a9ded640 1 --2- 172.21.15.57:0/2746374201 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f189006af60 0x7f189006d420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f189c009c10 tx=0x7f189c005f90 comp rx=0 tx=0).ready entity=mgr.6256 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:21.993 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:21.989+0000 7f18a27fc640 1 -- 172.21.15.57:0/2746374201 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f1894078db0 con 0x7f18a40b6380 2023-10-19T13:00:22.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:22.201+0000 7f18ac051640 1 -- 172.21.15.57:0/2746374201 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f18a40d0b40 con 0x7f18a40b6380 2023-10-19T13:00:22.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:22.208+0000 7f18a27fc640 1 -- 172.21.15.57:0/2746374201 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v29) v1 ==== 118+0+0 (secure 0 0 0) 0x7f1894078940 con 0x7f18a40b6380 2023-10-19T13:00:22.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:22.210+0000 7f18ac051640 1 -- 172.21.15.57:0/2746374201 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f189006af60 msgr2=0x7f189006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:22.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:22.210+0000 7f18ac051640 1 --2- 172.21.15.57:0/2746374201 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f189006af60 0x7f189006d420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f189c009c10 tx=0x7f189c005f90 comp rx=0 tx=0).stop 2023-10-19T13:00:22.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:22.210+0000 7f18ac051640 1 -- 172.21.15.57:0/2746374201 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f18a40b6380 msgr2=0x7f18a41f1000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:22.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:22.210+0000 7f18ac051640 1 --2- 172.21.15.57:0/2746374201 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f18a40b6380 0x7f18a41f1000 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f189402cbe0 tx=0x7f18940020a0 comp rx=0 tx=0).stop 2023-10-19T13:00:22.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:22.210+0000 7f18ac051640 1 -- 172.21.15.57:0/2746374201 shutdown_connections 2023-10-19T13:00:22.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:22.210+0000 7f18ac051640 1 --2- 172.21.15.57:0/2746374201 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f189006af60 0x7f189006d420 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:22.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:22.210+0000 7f18ac051640 1 --2- 172.21.15.57:0/2746374201 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f18a40b6380 0x7f18a41f1000 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:22.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:22.210+0000 7f18ac051640 1 --2- 172.21.15.57:0/2746374201 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f18a4152100 0x7f18a41f0ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:22.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:22.210+0000 7f18ac051640 1 --2- 172.21.15.57:0/2746374201 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f18a40b5fa0 0x7f18a41f56d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:22.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:22.210+0000 7f18ac051640 1 -- 172.21.15.57:0/2746374201 >> 172.21.15.57:0/2746374201 conn(0x7f18a40d0d60 msgr2=0x7f18a40b0630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:22.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:22.210+0000 7f18ac051640 1 -- 172.21.15.57:0/2746374201 shutdown_connections 2023-10-19T13:00:22.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:22.210+0000 7f18ac051640 1 -- 172.21.15.57:0/2746374201 wait complete. 2023-10-19T13:00:22.224 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-19T13:00:22.225 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-19T13:00:22.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:22.335+0000 7fdac3456640 1 -- 172.21.15.57:0/162041326 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdabc0cd1c0 msgr2=0x7fdabc15c0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:22.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:22.335+0000 7fdac3456640 1 --2- 172.21.15.57:0/162041326 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdabc0cd1c0 0x7fdabc15c0e0 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7fdab0005fd0 tx=0x7fdab002be10 comp rx=0 tx=0).stop 2023-10-19T13:00:22.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:22.335+0000 7fdac3456640 1 -- 172.21.15.57:0/162041326 shutdown_connections 2023-10-19T13:00:22.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:22.335+0000 7fdac3456640 1 --2- 172.21.15.57:0/162041326 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdabc0cd1c0 0x7fdabc15c0e0 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:22.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:22.335+0000 7fdac3456640 1 --2- 172.21.15.57:0/162041326 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdabc158910 0x7fdabc158cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:22.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:22.335+0000 7fdac3456640 1 -- 172.21.15.57:0/162041326 >> 172.21.15.57:0/162041326 conn(0x7fdabc0af7a0 msgr2=0x7fdabc0afb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:22.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:22.335+0000 7fdac3456640 1 -- 172.21.15.57:0/162041326 shutdown_connections 2023-10-19T13:00:22.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:22.335+0000 7fdac3456640 1 -- 172.21.15.57:0/162041326 wait complete. 2023-10-19T13:00:22.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:22.336+0000 7fdac3456640 1 Processor -- start 2023-10-19T13:00:22.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:22.336+0000 7fdac3456640 1 -- start start 2023-10-19T13:00:22.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:22.337+0000 7fdac3456640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdabc0cd1c0 0x7fdabc14a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:22.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:22.337+0000 7fdac3456640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdabc0cd9e0 0x7fdabc14b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:22.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:22.337+0000 7fdac3456640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdabc158910 0x7fdabc070110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:22.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:22.337+0000 7fdac3456640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdabc0acf70 con 0x7fdabc0cd9e0 2023-10-19T13:00:22.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:22.337+0000 7fdac3456640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdabc0ad250 con 0x7fdabc0cd1c0 2023-10-19T13:00:22.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:22.337+0000 7fdac3456640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdabc14a9b0 con 0x7fdabc158910 2023-10-19T13:00:22.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:22.337+0000 7fdac09f1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdabc0cd9e0 0x7fdabc14b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:22.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:22.337+0000 7fdac09f1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdabc0cd9e0 0x7fdabc14b490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:49808/0 (socket says 172.21.15.57:49808) 2023-10-19T13:00:22.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:22.337+0000 7fdac09f1640 1 -- 172.21.15.57:0/2009590555 learned_addr learned my addr 172.21.15.57:0/2009590555 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:22.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:22.337+0000 7fdac11f2640 1 --2- 172.21.15.57:0/2009590555 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdabc0cd1c0 0x7fdabc14a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:22.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:22.337+0000 7fdac19f3640 1 --2- 172.21.15.57:0/2009590555 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdabc158910 0x7fdabc070110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:22.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:22.337+0000 7fdac09f1640 1 -- 172.21.15.57:0/2009590555 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdabc158910 msgr2=0x7fdabc070110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:22.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:22.337+0000 7fdac09f1640 1 --2- 172.21.15.57:0/2009590555 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdabc158910 0x7fdabc070110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:22.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:22.337+0000 7fdac09f1640 1 -- 172.21.15.57:0/2009590555 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdabc0cd1c0 msgr2=0x7fdabc14a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:22.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:22.337+0000 7fdac09f1640 1 --2- 172.21.15.57:0/2009590555 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdabc0cd1c0 0x7fdabc14a3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:22.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:22.337+0000 7fdac09f1640 1 -- 172.21.15.57:0/2009590555 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdab0009560 con 0x7fdabc0cd9e0 2023-10-19T13:00:22.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:22.337+0000 7fdac09f1640 1 --2- 172.21.15.57:0/2009590555 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdabc0cd9e0 0x7fdabc14b490 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7fdab0006010 tx=0x7fdab002dbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:22.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:22.337+0000 7fdab9ffb640 1 -- 172.21.15.57:0/2009590555 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdab00035f0 con 0x7fdabc0cd9e0 2023-10-19T13:00:22.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:22.337+0000 7fdab9ffb640 1 -- 172.21.15.57:0/2009590555 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fdab0003840 con 0x7fdabc0cd9e0 2023-10-19T13:00:22.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:22.337+0000 7fdac3456640 1 -- 172.21.15.57:0/2009590555 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdabc14ac90 con 0x7fdabc0cd9e0 2023-10-19T13:00:22.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:22.337+0000 7fdab9ffb640 1 -- 172.21.15.57:0/2009590555 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdab0003c50 con 0x7fdabc0cd9e0 2023-10-19T13:00:22.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:22.338+0000 7fdac3456640 1 -- 172.21.15.57:0/2009590555 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdabc14b9d0 con 0x7fdabc0cd9e0 2023-10-19T13:00:22.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:22.338+0000 7fdac3456640 1 -- 172.21.15.57:0/2009590555 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdabc14bc20 con 0x7fdabc0cd9e0 2023-10-19T13:00:22.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:22.339+0000 7fdab9ffb640 1 -- 172.21.15.57:0/2009590555 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 32) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fdab003e450 con 0x7fdabc0cd9e0 2023-10-19T13:00:22.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:22.339+0000 7fdab9ffb640 1 --2- 172.21.15.57:0/2009590555 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7fdaa806af60 0x7fdaa806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:22.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:22.339+0000 7fdab9ffb640 1 -- 172.21.15.57:0/2009590555 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fdab00ac400 con 0x7fdabc0cd9e0 2023-10-19T13:00:22.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:22.340+0000 7fdac11f2640 1 --2- 172.21.15.57:0/2009590555 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7fdaa806af60 0x7fdaa806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:22.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:22.340+0000 7fdac11f2640 1 --2- 172.21.15.57:0/2009590555 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7fdaa806af60 0x7fdaa806d420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fdabc0b2290 tx=0x7fdaac0095e0 comp rx=0 tx=0).ready entity=mgr.6256 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:22.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:22.344+0000 7fdab9ffb640 1 -- 172.21.15.57:0/2009590555 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fdab0077400 con 0x7fdabc0cd9e0 2023-10-19T13:00:22.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:22.559+0000 7fdac3456640 1 -- 172.21.15.57:0/2009590555 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fdabc0690e0 con 0x7fdabc0cd9e0 2023-10-19T13:00:23.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.224+0000 7fdab9ffb640 1 -- 172.21.15.57:0/2009590555 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v30) v1 ==== 122+0+0 (secure 0 0 0) 0x7fdab007aa20 con 0x7fdabc0cd9e0 2023-10-19T13:00:23.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.227+0000 7fdac3456640 1 -- 172.21.15.57:0/2009590555 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7fdaa806af60 msgr2=0x7fdaa806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:23.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.227+0000 7fdac3456640 1 --2- 172.21.15.57:0/2009590555 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7fdaa806af60 0x7fdaa806d420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fdabc0b2290 tx=0x7fdaac0095e0 comp rx=0 tx=0).stop 2023-10-19T13:00:23.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.227+0000 7fdac3456640 1 -- 172.21.15.57:0/2009590555 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdabc0cd9e0 msgr2=0x7fdabc14b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:23.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.227+0000 7fdac3456640 1 --2- 172.21.15.57:0/2009590555 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdabc0cd9e0 0x7fdabc14b490 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7fdab0006010 tx=0x7fdab002dbb0 comp rx=0 tx=0).stop 2023-10-19T13:00:23.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.227+0000 7fdac3456640 1 -- 172.21.15.57:0/2009590555 shutdown_connections 2023-10-19T13:00:23.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.227+0000 7fdac3456640 1 --2- 172.21.15.57:0/2009590555 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7fdaa806af60 0x7fdaa806d420 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:23.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.227+0000 7fdac3456640 1 --2- 172.21.15.57:0/2009590555 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdabc158910 0x7fdabc070110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:23.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.227+0000 7fdac3456640 1 --2- 172.21.15.57:0/2009590555 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdabc0cd9e0 0x7fdabc14b490 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:23.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.227+0000 7fdac3456640 1 --2- 172.21.15.57:0/2009590555 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdabc0cd1c0 0x7fdabc14a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:23.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.227+0000 7fdac3456640 1 -- 172.21.15.57:0/2009590555 >> 172.21.15.57:0/2009590555 conn(0x7fdabc0af7a0 msgr2=0x7fdabc06db10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:23.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.227+0000 7fdac3456640 1 -- 172.21.15.57:0/2009590555 shutdown_connections 2023-10-19T13:00:23.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.227+0000 7fdac3456640 1 -- 172.21.15.57:0/2009590555 wait complete. 2023-10-19T13:00:23.242 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-19T13:00:23.243 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T13:00:23.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.355+0000 7f31aead7640 1 -- 172.21.15.57:0/1952261104 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f31a80cf340 msgr2=0x7f31a80d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:23.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.355+0000 7f31aead7640 1 --2- 172.21.15.57:0/1952261104 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f31a80cf340 0x7f31a80d1910 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f3198009830 tx=0x7f319802bcd0 comp rx=0 tx=0).stop 2023-10-19T13:00:23.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.355+0000 7f31aead7640 1 -- 172.21.15.57:0/1952261104 shutdown_connections 2023-10-19T13:00:23.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.355+0000 7f31aead7640 1 --2- 172.21.15.57:0/1952261104 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f31a80cf340 0x7f31a80d1910 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:23.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.355+0000 7f31aead7640 1 -- 172.21.15.57:0/1952261104 >> 172.21.15.57:0/1952261104 conn(0x7f31a80d0d60 msgr2=0x7f31a80d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:23.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.355+0000 7f31aead7640 1 -- 172.21.15.57:0/1952261104 shutdown_connections 2023-10-19T13:00:23.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.355+0000 7f31aead7640 1 -- 172.21.15.57:0/1952261104 wait complete. 2023-10-19T13:00:23.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.356+0000 7f31aead7640 1 Processor -- start 2023-10-19T13:00:23.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.356+0000 7f31aead7640 1 -- start start 2023-10-19T13:00:23.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.357+0000 7f31aead7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f31a80ceb40 0x7f31a8158c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:23.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.357+0000 7f31aead7640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f31a80cf340 0x7f31a80b4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:23.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.357+0000 7f31aead7640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f31a815b2a0 0x7f31a80b60c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:23.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.357+0000 7f31aead7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31a81f17a0 con 0x7f31a80ceb40 2023-10-19T13:00:23.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.357+0000 7f31aead7640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31a8159140 con 0x7f31a815b2a0 2023-10-19T13:00:23.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.357+0000 7f31aead7640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31a80b3c10 con 0x7f31a80cf340 2023-10-19T13:00:23.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.357+0000 7f31ac873640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f31a80ceb40 0x7f31a8158c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:23.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.357+0000 7f31ac873640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f31a80ceb40 0x7f31a8158c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:49812/0 (socket says 172.21.15.57:49812) 2023-10-19T13:00:23.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.358+0000 7f31ac873640 1 -- 172.21.15.57:0/1908486249 learned_addr learned my addr 172.21.15.57:0/1908486249 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:23.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.358+0000 7f31a7fff640 1 --2- 172.21.15.57:0/1908486249 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f31a80cf340 0x7f31a80b4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:23.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.358+0000 7f31ad074640 1 --2- 172.21.15.57:0/1908486249 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f31a815b2a0 0x7f31a80b60c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:23.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.358+0000 7f31ac873640 1 -- 172.21.15.57:0/1908486249 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f31a80cf340 msgr2=0x7f31a80b4cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:23.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.358+0000 7f31ac873640 1 --2- 172.21.15.57:0/1908486249 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f31a80cf340 0x7f31a80b4cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:23.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.358+0000 7f31ac873640 1 -- 172.21.15.57:0/1908486249 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f31a815b2a0 msgr2=0x7f31a80b60c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:23.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.358+0000 7f31ac873640 1 --2- 172.21.15.57:0/1908486249 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f31a815b2a0 0x7f31a80b60c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:23.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.358+0000 7f31ac873640 1 -- 172.21.15.57:0/1908486249 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3198009490 con 0x7f31a80ceb40 2023-10-19T13:00:23.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.358+0000 7f31a7fff640 1 --2- 172.21.15.57:0/1908486249 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f31a80cf340 0x7f31a80b4cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T13:00:23.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.358+0000 7f31ad074640 1 --2- 172.21.15.57:0/1908486249 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f31a815b2a0 0x7f31a80b60c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T13:00:23.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.358+0000 7f31ac873640 1 --2- 172.21.15.57:0/1908486249 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f31a80ceb40 0x7f31a8158c00 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f3198009780 tx=0x7f31980038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:23.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.358+0000 7f31a57fa640 1 -- 172.21.15.57:0/1908486249 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3198003ea0 con 0x7f31a80ceb40 2023-10-19T13:00:23.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.358+0000 7f31a57fa640 1 -- 172.21.15.57:0/1908486249 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f3198034550 con 0x7f31a80ceb40 2023-10-19T13:00:23.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.358+0000 7f31aead7640 1 -- 172.21.15.57:0/1908486249 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31a80b3ef0 con 0x7f31a80ceb40 2023-10-19T13:00:23.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.358+0000 7f31a57fa640 1 -- 172.21.15.57:0/1908486249 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3198034960 con 0x7f31a80ceb40 2023-10-19T13:00:23.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.358+0000 7f31aead7640 1 -- 172.21.15.57:0/1908486249 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31a80b44b0 con 0x7f31a80ceb40 2023-10-19T13:00:23.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.359+0000 7f31aead7640 1 -- 172.21.15.57:0/1908486249 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31a80d07d0 con 0x7f31a80ceb40 2023-10-19T13:00:23.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.360+0000 7f31a57fa640 1 -- 172.21.15.57:0/1908486249 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 32) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f319803b040 con 0x7f31a80ceb40 2023-10-19T13:00:23.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.360+0000 7f31a57fa640 1 --2- 172.21.15.57:0/1908486249 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f319406af90 0x7f319406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:23.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.360+0000 7f31a57fa640 1 -- 172.21.15.57:0/1908486249 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f31980abeb0 con 0x7f31a80ceb40 2023-10-19T13:00:23.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.360+0000 7f31a7fff640 1 --2- 172.21.15.57:0/1908486249 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f319406af90 0x7f319406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:23.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.361+0000 7f31a7fff640 1 --2- 172.21.15.57:0/1908486249 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f319406af90 0x7f319406d450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f31a80b3220 tx=0x7f319c008040 comp rx=0 tx=0).ready entity=mgr.6256 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:23.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.364+0000 7f31a57fa640 1 -- 172.21.15.57:0/1908486249 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3198076d80 con 0x7f31a80ceb40 2023-10-19T13:00:23.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.581+0000 7f31aead7640 1 -- 172.21.15.57:0/1908486249 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f31a80690e0 con 0x7f31a80ceb40 2023-10-19T13:00:23.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.582+0000 7f31a57fa640 1 -- 172.21.15.57:0/1908486249 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+4917 (secure 0 0 0) 0x7f3198039090 con 0x7f31a80ceb40 2023-10-19T13:00:23.585 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:00:23.586 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":30,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5966,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/765319531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":765319531},{"type":"v1","addr":"172.21.15.179:6837","nonce":765319531}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27},{"gid":5969,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/1388526553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":1388526553},{"type":"v1","addr":"172.21.15.179:6836","nonce":1388526553}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27},{"gid":7035,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/3237434259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":3237434259},{"type":"v1","addr":"172.21.15.57:6834","nonce":3237434259}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T13:00:20.549275+0000","modified":"2023-10-19T13:00:22.561271+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7038},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7038":{"gid":7038,"name":"a","rank":0,"incarnation":28,"state":"up:active","state_seq":3,"addr":"172.21.15.57:6835/2257211800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2257211800},{"type":"v1","addr":"172.21.15.57:6835","nonce":2257211800}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-10-19T13:00:23.586 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 30 2023-10-19T13:00:23.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.584+0000 7f31aead7640 1 -- 172.21.15.57:0/1908486249 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f319406af90 msgr2=0x7f319406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:23.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.584+0000 7f31aead7640 1 --2- 172.21.15.57:0/1908486249 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f319406af90 0x7f319406d450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f31a80b3220 tx=0x7f319c008040 comp rx=0 tx=0).stop 2023-10-19T13:00:23.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.584+0000 7f31aead7640 1 -- 172.21.15.57:0/1908486249 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f31a80ceb40 msgr2=0x7f31a8158c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:23.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.584+0000 7f31aead7640 1 --2- 172.21.15.57:0/1908486249 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f31a80ceb40 0x7f31a8158c00 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f3198009780 tx=0x7f31980038a0 comp rx=0 tx=0).stop 2023-10-19T13:00:23.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.584+0000 7f31aead7640 1 -- 172.21.15.57:0/1908486249 shutdown_connections 2023-10-19T13:00:23.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.584+0000 7f31aead7640 1 --2- 172.21.15.57:0/1908486249 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f319406af90 0x7f319406d450 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:23.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.584+0000 7f31aead7640 1 --2- 172.21.15.57:0/1908486249 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f31a815b2a0 0x7f31a80b60c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:23.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.584+0000 7f31aead7640 1 --2- 172.21.15.57:0/1908486249 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f31a80cf340 0x7f31a80b4cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:23.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.584+0000 7f31aead7640 1 --2- 172.21.15.57:0/1908486249 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f31a80ceb40 0x7f31a8158c00 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:23.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.584+0000 7f31aead7640 1 -- 172.21.15.57:0/1908486249 >> 172.21.15.57:0/1908486249 conn(0x7f31a80d0d60 msgr2=0x7f31a806b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:23.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.584+0000 7f31aead7640 1 -- 172.21.15.57:0/1908486249 shutdown_connections 2023-10-19T13:00:23.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.584+0000 7f31aead7640 1 -- 172.21.15.57:0/1908486249 wait complete. 2023-10-19T13:00:23.599 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T13:00:23.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.705+0000 7f92c8d33640 1 -- 172.21.15.57:0/3392689627 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f92c40cf340 msgr2=0x7f92c40d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:23.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.705+0000 7f92c8d33640 1 --2- 172.21.15.57:0/3392689627 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f92c40cf340 0x7f92c40d1910 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f92c408a810 tx=0x7f92b402bd10 comp rx=0 tx=0).stop 2023-10-19T13:00:23.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.705+0000 7f92c8d33640 1 -- 172.21.15.57:0/3392689627 shutdown_connections 2023-10-19T13:00:23.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.705+0000 7f92c8d33640 1 --2- 172.21.15.57:0/3392689627 >> v2:172.21.15.179:3300/0 conn(0x7f92c415b2a0 0x7f92c40cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:23.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.705+0000 7f92c8d33640 1 --2- 172.21.15.57:0/3392689627 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f92c40cf340 0x7f92c40d1910 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:23.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.705+0000 7f92c8d33640 1 -- 172.21.15.57:0/3392689627 >> 172.21.15.57:0/3392689627 conn(0x7f92c40d0d60 msgr2=0x7f92c40d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:23.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.705+0000 7f92c8d33640 1 -- 172.21.15.57:0/3392689627 shutdown_connections 2023-10-19T13:00:23.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.705+0000 7f92c8d33640 1 -- 172.21.15.57:0/3392689627 wait complete. 2023-10-19T13:00:23.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.706+0000 7f92c8d33640 1 Processor -- start 2023-10-19T13:00:23.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.706+0000 7f92c8d33640 1 -- start start 2023-10-19T13:00:23.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.707+0000 7f92c8d33640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f92c40ceb40 0x7f92c4156920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:23.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.707+0000 7f92c8d33640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f92c40cf340 0x7f92c40c8940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:23.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.707+0000 7f92c8d33640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f92c415b2a0 0x7f92c41551d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:23.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.707+0000 7f92c8d33640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92c415e8b0 con 0x7f92c415b2a0 2023-10-19T13:00:23.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.707+0000 7f92c8d33640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92c4156e60 con 0x7f92c40cf340 2023-10-19T13:00:23.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.707+0000 7f92c8d33640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92c415bec0 con 0x7f92c40ceb40 2023-10-19T13:00:23.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.707+0000 7f92c2d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f92c415b2a0 0x7f92c41551d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:23.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.707+0000 7f92c2d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f92c415b2a0 0x7f92c41551d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:49828/0 (socket says 172.21.15.57:49828) 2023-10-19T13:00:23.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.707+0000 7f92c2d9d640 1 -- 172.21.15.57:0/3494808336 learned_addr learned my addr 172.21.15.57:0/3494808336 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:23.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.708+0000 7f92b9d9b640 1 --2- 172.21.15.57:0/3494808336 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f92c40cf340 0x7f92c40c8940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:23.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.708+0000 7f92c259c640 1 --2- 172.21.15.57:0/3494808336 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f92c40ceb40 0x7f92c4156920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:23.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.708+0000 7f92c2d9d640 1 -- 172.21.15.57:0/3494808336 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f92c40ceb40 msgr2=0x7f92c4156920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:23.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.708+0000 7f92c2d9d640 1 --2- 172.21.15.57:0/3494808336 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f92c40ceb40 0x7f92c4156920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:23.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.708+0000 7f92c2d9d640 1 -- 172.21.15.57:0/3494808336 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f92c40cf340 msgr2=0x7f92c40c8940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:23.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.708+0000 7f92c2d9d640 1 --2- 172.21.15.57:0/3494808336 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f92c40cf340 0x7f92c40c8940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:23.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.708+0000 7f92c2d9d640 1 -- 172.21.15.57:0/3494808336 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92b4009560 con 0x7f92c415b2a0 2023-10-19T13:00:23.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.708+0000 7f92c259c640 1 --2- 172.21.15.57:0/3494808336 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f92c40ceb40 0x7f92c4156920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:00:23.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.708+0000 7f92b9d9b640 1 --2- 172.21.15.57:0/3494808336 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f92c40cf340 0x7f92c40c8940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:00:23.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.708+0000 7f92c2d9d640 1 --2- 172.21.15.57:0/3494808336 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f92c415b2a0 0x7f92c41551d0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f92bc00c780 tx=0x7f92bc00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:23.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.708+0000 7f92bb7fe640 1 -- 172.21.15.57:0/3494808336 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92bc003d80 con 0x7f92c415b2a0 2023-10-19T13:00:23.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.708+0000 7f92bb7fe640 1 -- 172.21.15.57:0/3494808336 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f92bc00ce90 con 0x7f92c415b2a0 2023-10-19T13:00:23.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.708+0000 7f92c8d33640 1 -- 172.21.15.57:0/3494808336 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92c41586a0 con 0x7f92c415b2a0 2023-10-19T13:00:23.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.708+0000 7f92bb7fe640 1 -- 172.21.15.57:0/3494808336 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92bc0052d0 con 0x7f92c415b2a0 2023-10-19T13:00:23.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.708+0000 7f92c8d33640 1 -- 172.21.15.57:0/3494808336 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92c415e5d0 con 0x7f92c415b2a0 2023-10-19T13:00:23.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.709+0000 7f92c8d33640 1 -- 172.21.15.57:0/3494808336 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92c40690e0 con 0x7f92c415b2a0 2023-10-19T13:00:23.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.710+0000 7f92bb7fe640 1 -- 172.21.15.57:0/3494808336 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 32) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f92bc005430 con 0x7f92c415b2a0 2023-10-19T13:00:23.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.710+0000 7f92bb7fe640 1 --2- 172.21.15.57:0/3494808336 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f92a406af60 0x7f92a406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:23.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.710+0000 7f92bb7fe640 1 -- 172.21.15.57:0/3494808336 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f92bc089400 con 0x7f92c415b2a0 2023-10-19T13:00:23.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.710+0000 7f92c259c640 1 --2- 172.21.15.57:0/3494808336 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f92a406af60 0x7f92a406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:23.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.711+0000 7f92c259c640 1 --2- 172.21.15.57:0/3494808336 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f92a406af60 0x7f92a406d420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f92b4035000 tx=0x7f92b4004730 comp rx=0 tx=0).ready entity=mgr.6256 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:23.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.714+0000 7f92bb7fe640 1 -- 172.21.15.57:0/3494808336 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f92bc054240 con 0x7f92c415b2a0 2023-10-19T13:00:23.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.890+0000 7f92c8d33640 1 -- 172.21.15.57:0/3494808336 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f92c40cb9d0 con 0x7f92c415b2a0 2023-10-19T13:00:23.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.891+0000 7f92bb7fe640 1 -- 172.21.15.57:0/3494808336 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+16771 (secure 0 0 0) 0x7f92bc057860 con 0x7f92c415b2a0 2023-10-19T13:00:23.894 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:00:23.894 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":87,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T13:00:22.208490+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2023-10-19T13:00:18.452198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2023-10-19T13:00:19.531525+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T13:00:23.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.893+0000 7f92c8d33640 1 -- 172.21.15.57:0/3494808336 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f92a406af60 msgr2=0x7f92a406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:23.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.893+0000 7f92c8d33640 1 --2- 172.21.15.57:0/3494808336 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f92a406af60 0x7f92a406d420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f92b4035000 tx=0x7f92b4004730 comp rx=0 tx=0).stop 2023-10-19T13:00:23.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.893+0000 7f92c8d33640 1 -- 172.21.15.57:0/3494808336 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f92c415b2a0 msgr2=0x7f92c41551d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:23.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.893+0000 7f92c8d33640 1 --2- 172.21.15.57:0/3494808336 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f92c415b2a0 0x7f92c41551d0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f92bc00c780 tx=0x7f92bc00cc40 comp rx=0 tx=0).stop 2023-10-19T13:00:23.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.893+0000 7f92c8d33640 1 -- 172.21.15.57:0/3494808336 shutdown_connections 2023-10-19T13:00:23.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.893+0000 7f92c8d33640 1 --2- 172.21.15.57:0/3494808336 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f92a406af60 0x7f92a406d420 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:23.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.893+0000 7f92c8d33640 1 --2- 172.21.15.57:0/3494808336 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f92c415b2a0 0x7f92c41551d0 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:23.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.893+0000 7f92c8d33640 1 --2- 172.21.15.57:0/3494808336 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f92c40cf340 0x7f92c40c8940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:23.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.893+0000 7f92c8d33640 1 --2- 172.21.15.57:0/3494808336 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f92c40ceb40 0x7f92c4156920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:23.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.893+0000 7f92c8d33640 1 -- 172.21.15.57:0/3494808336 >> 172.21.15.57:0/3494808336 conn(0x7f92c40d0d60 msgr2=0x7f92c415a5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:23.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.893+0000 7f92c8d33640 1 -- 172.21.15.57:0/3494808336 shutdown_connections 2023-10-19T13:00:23.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:23.893+0000 7f92c8d33640 1 -- 172.21.15.57:0/3494808336 wait complete. 2023-10-19T13:00:23.907 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-19T13:00:23.907 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-19T13:00:24.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.013+0000 7f0536407640 1 -- 172.21.15.57:0/299666839 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f05300cbd20 msgr2=0x7f0530158300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:24.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.013+0000 7f0536407640 1 --2- 172.21.15.57:0/299666839 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f05300cbd20 0x7f0530158300 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f05300b10f0 tx=0x7f052402bd70 comp rx=0 tx=0).stop 2023-10-19T13:00:24.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.013+0000 7f0536407640 1 -- 172.21.15.57:0/299666839 shutdown_connections 2023-10-19T13:00:24.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.013+0000 7f0536407640 1 --2- 172.21.15.57:0/299666839 >> v2:172.21.15.179:3300/0 conn(0x7f05300b0530 0x7f05300b09a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:24.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.013+0000 7f0536407640 1 --2- 172.21.15.57:0/299666839 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f05300cbd20 0x7f0530158300 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:24.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.013+0000 7f0536407640 1 -- 172.21.15.57:0/299666839 >> 172.21.15.57:0/299666839 conn(0x7f0530069930 msgr2=0x7f0530069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:24.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.013+0000 7f0536407640 1 -- 172.21.15.57:0/299666839 shutdown_connections 2023-10-19T13:00:24.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.013+0000 7f0536407640 1 -- 172.21.15.57:0/299666839 wait complete. 2023-10-19T13:00:24.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.014+0000 7f0536407640 1 Processor -- start 2023-10-19T13:00:24.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.015+0000 7f0536407640 1 -- start start 2023-10-19T13:00:24.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.015+0000 7f0536407640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f05300afa20 0x7f05301460d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:24.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.015+0000 7f0536407640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f05300b0530 0x7f053013e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:24.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.015+0000 7f0536407640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f05300cbd20 0x7f0530165bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:24.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.015+0000 7f0536407640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05301f1560 con 0x7f05300afa20 2023-10-19T13:00:24.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.015+0000 7f0536407640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0530148100 con 0x7f05300b0530 2023-10-19T13:00:24.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.015+0000 7f0536407640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0530146610 con 0x7f05300cbd20 2023-10-19T13:00:24.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.015+0000 7f052ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f05300afa20 0x7f05301460d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:24.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.015+0000 7f052ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f05300afa20 0x7f05301460d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:49858/0 (socket says 172.21.15.57:49858) 2023-10-19T13:00:24.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.015+0000 7f052ffff640 1 -- 172.21.15.57:0/3459366305 learned_addr learned my addr 172.21.15.57:0/3459366305 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:24.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.016+0000 7f05349a4640 1 --2- 172.21.15.57:0/3459366305 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f05300cbd20 0x7f0530165bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:24.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.016+0000 7f052f7fe640 1 --2- 172.21.15.57:0/3459366305 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f05300b0530 0x7f053013e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:24.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.016+0000 7f052ffff640 1 -- 172.21.15.57:0/3459366305 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f05300cbd20 msgr2=0x7f0530165bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:24.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.016+0000 7f052ffff640 1 --2- 172.21.15.57:0/3459366305 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f05300cbd20 0x7f0530165bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:24.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.016+0000 7f052ffff640 1 -- 172.21.15.57:0/3459366305 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f05300b0530 msgr2=0x7f053013e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:24.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.016+0000 7f052ffff640 1 --2- 172.21.15.57:0/3459366305 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f05300b0530 0x7f053013e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:24.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.016+0000 7f052ffff640 1 -- 172.21.15.57:0/3459366305 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0524009490 con 0x7f05300afa20 2023-10-19T13:00:24.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.016+0000 7f052ffff640 1 --2- 172.21.15.57:0/3459366305 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f05300afa20 0x7f05301460d0 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f0524000c00 tx=0x7f05240038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:24.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.016+0000 7f052cff9640 1 -- 172.21.15.57:0/3459366305 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0524003ea0 con 0x7f05300afa20 2023-10-19T13:00:24.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.016+0000 7f052cff9640 1 -- 172.21.15.57:0/3459366305 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f0524034550 con 0x7f05300afa20 2023-10-19T13:00:24.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.016+0000 7f052cff9640 1 -- 172.21.15.57:0/3459366305 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0524034960 con 0x7f05300afa20 2023-10-19T13:00:24.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.016+0000 7f0536407640 1 -- 172.21.15.57:0/3459366305 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f053013eab0 con 0x7f05300afa20 2023-10-19T13:00:24.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.016+0000 7f0536407640 1 -- 172.21.15.57:0/3459366305 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0530148350 con 0x7f05300afa20 2023-10-19T13:00:24.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.017+0000 7f0536407640 1 -- 172.21.15.57:0/3459366305 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0530151c10 con 0x7f05300afa20 2023-10-19T13:00:24.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.018+0000 7f052cff9640 1 -- 172.21.15.57:0/3459366305 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 32) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f052403b040 con 0x7f05300afa20 2023-10-19T13:00:24.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.018+0000 7f052cff9640 1 --2- 172.21.15.57:0/3459366305 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f051c06af90 0x7f051c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:24.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.018+0000 7f052cff9640 1 -- 172.21.15.57:0/3459366305 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f05240abec0 con 0x7f05300afa20 2023-10-19T13:00:24.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.018+0000 7f052f7fe640 1 --2- 172.21.15.57:0/3459366305 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f051c06af90 0x7f051c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:24.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.019+0000 7f052f7fe640 1 --2- 172.21.15.57:0/3459366305 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f051c06af90 0x7f051c06d450 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f053014b530 tx=0x7f0520009040 comp rx=0 tx=0).ready entity=mgr.6256 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:24.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.022+0000 7f052cff9640 1 -- 172.21.15.57:0/3459366305 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0524076c70 con 0x7f05300afa20 2023-10-19T13:00:24.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.195+0000 7f0536407640 1 -- 172.21.15.57:0/3459366305 --> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f05300cc100 con 0x7f051c06af90 2023-10-19T13:00:24.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.203+0000 7f052cff9640 1 -- 172.21.15.57:0/3459366305 <== mgr.6256 v2:172.21.15.179:6832/3815793798 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+279504 (secure 0 0 0) 0x7f0520001c70 con 0x7f051c06af90 2023-10-19T13:00:24.208 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:00:24.210 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2023-10-19T13:00:24.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.206+0000 7f0536407640 1 -- 172.21.15.57:0/3459366305 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f051c06af90 msgr2=0x7f051c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:24.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.206+0000 7f0536407640 1 --2- 172.21.15.57:0/3459366305 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f051c06af90 0x7f051c06d450 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f053014b530 tx=0x7f0520009040 comp rx=0 tx=0).stop 2023-10-19T13:00:24.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.206+0000 7f0536407640 1 -- 172.21.15.57:0/3459366305 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f05300afa20 msgr2=0x7f05301460d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:24.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.206+0000 7f0536407640 1 --2- 172.21.15.57:0/3459366305 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f05300afa20 0x7f05301460d0 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f0524000c00 tx=0x7f05240038a0 comp rx=0 tx=0).stop 2023-10-19T13:00:24.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.206+0000 7f0536407640 1 -- 172.21.15.57:0/3459366305 shutdown_connections 2023-10-19T13:00:24.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.206+0000 7f0536407640 1 --2- 172.21.15.57:0/3459366305 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f051c06af90 0x7f051c06d450 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:24.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.206+0000 7f0536407640 1 --2- 172.21.15.57:0/3459366305 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f05300cbd20 0x7f0530165bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:24.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.206+0000 7f0536407640 1 --2- 172.21.15.57:0/3459366305 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f05300b0530 0x7f053013e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:24.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.206+0000 7f0536407640 1 --2- 172.21.15.57:0/3459366305 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f05300afa20 0x7f05301460d0 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:24.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.206+0000 7f0536407640 1 -- 172.21.15.57:0/3459366305 >> 172.21.15.57:0/3459366305 conn(0x7f0530069930 msgr2=0x7f05300ca870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:24.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.206+0000 7f0536407640 1 -- 172.21.15.57:0/3459366305 shutdown_connections 2023-10-19T13:00:24.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.206+0000 7f0536407640 1 -- 172.21.15.57:0/3459366305 wait complete. 2023-10-19T13:00:24.219 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2023-10-19T13:00:23.049618+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592562,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":256,"num_write_kb":2420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":178,"ondisk_log_size":178,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":43,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":4,"kb":754974720,"kb_used":315544,"kb_used_data":9164,"kb_used_omap":12,"kb_used_meta":306227,"kb_avail":754659176,"statfs":{"total":773094113280,"available":772770996224,"internally_reserved":0,"allocated":9383936,"data_stored":5262013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":313577017},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2194,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000992"},"pg_stats":[{"pgid":"10.3f","version":"0'0","reported_seq":9,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-19T13:00:21.237877+0000","last_change":"2023-10-19T13:00:21.228303+0000","last_active":"2023-10-19T13:00:21.237877+0000","last_peered":"2023-10-19T13:00:21.237877+0000","last_clean":"2023-10-19T13:00:21.237877+0000","last_became_active":"2023-10-19T13:00:21.227974+0000","last_became_peered":"2023-10-19T13:00:21.227974+0000","last_unstale":"2023-10-19T13:00:21.237877+0000","last_undegraded":"2023-10-19T13:00:21.237877+0000","last_fullsized":"2023-10-19T13:00:21.237877+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:37:52.369486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3e","version":"0'0","reported_seq":9,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-19T13:00:21.233058+0000","last_change":"2023-10-19T13:00:21.227759+0000","last_active":"2023-10-19T13:00:21.233058+0000","last_peered":"2023-10-19T13:00:21.233058+0000","last_clean":"2023-10-19T13:00:21.233058+0000","last_became_active":"2023-10-19T13:00:21.227466+0000","last_became_peered":"2023-10-19T13:00:21.227466+0000","last_unstale":"2023-10-19T13:00:21.233058+0000","last_undegraded":"2023-10-19T13:00:21.233058+0000","last_fullsized":"2023-10-19T13:00:21.233058+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:55:42.567075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.3a","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.221076+0000","last_change":"2023-10-19T13:00:21.219325+0000","last_active":"2023-10-19T13:00:22.221076+0000","last_peered":"2023-10-19T13:00:22.221076+0000","last_clean":"2023-10-19T13:00:22.221076+0000","last_became_active":"2023-10-19T13:00:21.218458+0000","last_became_peered":"2023-10-19T13:00:21.218458+0000","last_unstale":"2023-10-19T13:00:22.221076+0000","last_undegraded":"2023-10-19T13:00:22.221076+0000","last_fullsized":"2023-10-19T13:00:22.221076+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:31:39.478057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.39","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:23.044542+0000","last_change":"2023-10-19T13:00:21.220590+0000","last_active":"2023-10-19T13:00:23.044542+0000","last_peered":"2023-10-19T13:00:23.044542+0000","last_clean":"2023-10-19T13:00:23.044542+0000","last_became_active":"2023-10-19T13:00:21.220337+0000","last_became_peered":"2023-10-19T13:00:21.220337+0000","last_unstale":"2023-10-19T13:00:23.044542+0000","last_undegraded":"2023-10-19T13:00:23.044542+0000","last_fullsized":"2023-10-19T13:00:23.044542+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:25:25.779991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.30","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:23.044519+0000","last_change":"2023-10-19T13:00:21.219839+0000","last_active":"2023-10-19T13:00:23.044519+0000","last_peered":"2023-10-19T13:00:23.044519+0000","last_clean":"2023-10-19T13:00:23.044519+0000","last_became_active":"2023-10-19T13:00:21.219524+0000","last_became_peered":"2023-10-19T13:00:21.219524+0000","last_unstale":"2023-10-19T13:00:23.044519+0000","last_undegraded":"2023-10-19T13:00:23.044519+0000","last_fullsized":"2023-10-19T13:00:23.044519+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:16:36.020089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2f","version":"0'0","reported_seq":9,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-19T13:00:21.237892+0000","last_change":"2023-10-19T13:00:21.228480+0000","last_active":"2023-10-19T13:00:21.237892+0000","last_peered":"2023-10-19T13:00:21.237892+0000","last_clean":"2023-10-19T13:00:21.237892+0000","last_became_active":"2023-10-19T13:00:21.227687+0000","last_became_peered":"2023-10-19T13:00:21.227687+0000","last_unstale":"2023-10-19T13:00:21.237892+0000","last_undegraded":"2023-10-19T13:00:21.237892+0000","last_fullsized":"2023-10-19T13:00:21.237892+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:58:41.166338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.24","version":"0'0","reported_seq":9,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-19T13:00:21.233957+0000","last_change":"2023-10-19T13:00:21.227180+0000","last_active":"2023-10-19T13:00:21.233957+0000","last_peered":"2023-10-19T13:00:21.233957+0000","last_clean":"2023-10-19T13:00:21.233957+0000","last_became_active":"2023-10-19T13:00:21.226763+0000","last_became_peered":"2023-10-19T13:00:21.226763+0000","last_unstale":"2023-10-19T13:00:21.233957+0000","last_undegraded":"2023-10-19T13:00:21.233957+0000","last_fullsized":"2023-10-19T13:00:21.233957+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:41:19.821669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.20","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.220680+0000","last_change":"2023-10-19T13:00:21.217955+0000","last_active":"2023-10-19T13:00:22.220680+0000","last_peered":"2023-10-19T13:00:22.220680+0000","last_clean":"2023-10-19T13:00:22.220680+0000","last_became_active":"2023-10-19T13:00:21.217649+0000","last_became_peered":"2023-10-19T13:00:21.217649+0000","last_unstale":"2023-10-19T13:00:22.220680+0000","last_undegraded":"2023-10-19T13:00:22.220680+0000","last_fullsized":"2023-10-19T13:00:22.220680+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:55:17.988760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1f","version":"86'4","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-19T13:00:21.238495+0000","last_change":"2023-10-19T13:00:20.197586+0000","last_active":"2023-10-19T13:00:21.238495+0000","last_peered":"2023-10-19T13:00:21.238495+0000","last_clean":"2023-10-19T13:00:21.238495+0000","last_became_active":"2023-10-19T13:00:20.197286+0000","last_became_peered":"2023-10-19T13:00:20.197286+0000","last_unstale":"2023-10-19T13:00:21.238495+0000","last_undegraded":"2023-10-19T13:00:21.238495+0000","last_fullsized":"2023-10-19T13:00:21.238495+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:45:06.777683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1e","version":"86'1","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.221295+0000","last_change":"2023-10-19T13:00:20.194934+0000","last_active":"2023-10-19T13:00:22.221295+0000","last_peered":"2023-10-19T13:00:22.221295+0000","last_clean":"2023-10-19T13:00:22.221295+0000","last_became_active":"2023-10-19T13:00:20.194657+0000","last_became_peered":"2023-10-19T13:00:20.194657+0000","last_unstale":"2023-10-19T13:00:22.221295+0000","last_undegraded":"2023-10-19T13:00:22.221295+0000","last_fullsized":"2023-10-19T13:00:22.221295+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:12:57.142360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.221258+0000","last_change":"2023-10-19T13:00:21.218060+0000","last_active":"2023-10-19T13:00:22.221258+0000","last_peered":"2023-10-19T13:00:22.221258+0000","last_clean":"2023-10-19T13:00:22.221258+0000","last_became_active":"2023-10-19T13:00:21.217766+0000","last_became_peered":"2023-10-19T13:00:21.217766+0000","last_unstale":"2023-10-19T13:00:22.221258+0000","last_undegraded":"2023-10-19T13:00:22.221258+0000","last_fullsized":"2023-10-19T13:00:22.221258+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:50:26.459690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"86'2","reported_seq":14,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.220954+0000","last_change":"2023-10-19T13:00:20.188312+0000","last_active":"2023-10-19T13:00:22.220954+0000","last_peered":"2023-10-19T13:00:22.220954+0000","last_clean":"2023-10-19T13:00:22.220954+0000","last_became_active":"2023-10-19T13:00:20.187673+0000","last_became_peered":"2023-10-19T13:00:20.187673+0000","last_unstale":"2023-10-19T13:00:22.220954+0000","last_undegraded":"2023-10-19T13:00:22.220954+0000","last_fullsized":"2023-10-19T13:00:22.220954+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:22:13.247630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1c","version":"86'2","reported_seq":14,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.221016+0000","last_change":"2023-10-19T13:00:20.197923+0000","last_active":"2023-10-19T13:00:22.221016+0000","last_peered":"2023-10-19T13:00:22.221016+0000","last_clean":"2023-10-19T13:00:22.221016+0000","last_became_active":"2023-10-19T13:00:20.197546+0000","last_became_peered":"2023-10-19T13:00:20.197546+0000","last_unstale":"2023-10-19T13:00:22.221016+0000","last_undegraded":"2023-10-19T13:00:22.221016+0000","last_fullsized":"2023-10-19T13:00:22.221016+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:27:45.352182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:23.044381+0000","last_change":"2023-10-19T13:00:21.229195+0000","last_active":"2023-10-19T13:00:23.044381+0000","last_peered":"2023-10-19T13:00:23.044381+0000","last_clean":"2023-10-19T13:00:23.044381+0000","last_became_active":"2023-10-19T13:00:21.228898+0000","last_became_peered":"2023-10-19T13:00:21.228898+0000","last_unstale":"2023-10-19T13:00:23.044381+0000","last_undegraded":"2023-10-19T13:00:23.044381+0000","last_fullsized":"2023-10-19T13:00:23.044381+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:28:07.228859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:19.169402+0000","last_change":"2023-10-19T13:00:19.169402+0000","last_active":"2023-10-19T13:00:19.169402+0000","last_peered":"2023-10-19T13:00:19.169402+0000","last_clean":"2023-10-19T13:00:19.169402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:19.169402+0000","last_undegraded":"2023-10-19T13:00:19.169402+0000","last_fullsized":"2023-10-19T13:00:19.169402+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":9,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-19T13:00:21.232881+0000","last_change":"2023-10-19T13:00:21.226995+0000","last_active":"2023-10-19T13:00:21.232881+0000","last_peered":"2023-10-19T13:00:21.232881+0000","last_clean":"2023-10-19T13:00:21.232881+0000","last_became_active":"2023-10-19T13:00:21.226673+0000","last_became_peered":"2023-10-19T13:00:21.226673+0000","last_unstale":"2023-10-19T13:00:21.232881+0000","last_undegraded":"2023-10-19T13:00:21.232881+0000","last_fullsized":"2023-10-19T13:00:21.232881+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:52:56.240367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:19.169402+0000","last_change":"2023-10-19T13:00:19.169402+0000","last_active":"2023-10-19T13:00:19.169402+0000","last_peered":"2023-10-19T13:00:19.169402+0000","last_clean":"2023-10-19T13:00:19.169402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:19.169402+0000","last_undegraded":"2023-10-19T13:00:19.169402+0000","last_fullsized":"2023-10-19T13:00:19.169402+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:19.169402+0000","last_change":"2023-10-19T13:00:19.169402+0000","last_active":"2023-10-19T13:00:19.169402+0000","last_peered":"2023-10-19T13:00:19.169402+0000","last_clean":"2023-10-19T13:00:19.169402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:19.169402+0000","last_undegraded":"2023-10-19T13:00:19.169402+0000","last_fullsized":"2023-10-19T13:00:19.169402+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:19.169402+0000","last_change":"2023-10-19T13:00:19.169402+0000","last_active":"2023-10-19T13:00:19.169402+0000","last_peered":"2023-10-19T13:00:19.169402+0000","last_clean":"2023-10-19T13:00:19.169402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:19.169402+0000","last_undegraded":"2023-10-19T13:00:19.169402+0000","last_fullsized":"2023-10-19T13:00:19.169402+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":9,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-19T13:00:21.233976+0000","last_change":"2023-10-19T13:00:21.227321+0000","last_active":"2023-10-19T13:00:21.233976+0000","last_peered":"2023-10-19T13:00:21.233976+0000","last_clean":"2023-10-19T13:00:21.233976+0000","last_became_active":"2023-10-19T13:00:21.227028+0000","last_became_peered":"2023-10-19T13:00:21.227028+0000","last_unstale":"2023-10-19T13:00:21.233976+0000","last_undegraded":"2023-10-19T13:00:21.233976+0000","last_fullsized":"2023-10-19T13:00:21.233976+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:06:08.406600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:19.169402+0000","last_change":"2023-10-19T13:00:19.169402+0000","last_active":"2023-10-19T13:00:19.169402+0000","last_peered":"2023-10-19T13:00:19.169402+0000","last_clean":"2023-10-19T13:00:19.169402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:19.169402+0000","last_undegraded":"2023-10-19T13:00:19.169402+0000","last_fullsized":"2023-10-19T13:00:19.169402+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:19.169402+0000","last_change":"2023-10-19T13:00:19.169402+0000","last_active":"2023-10-19T13:00:19.169402+0000","last_peered":"2023-10-19T13:00:19.169402+0000","last_clean":"2023-10-19T13:00:19.169402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:19.169402+0000","last_undegraded":"2023-10-19T13:00:19.169402+0000","last_fullsized":"2023-10-19T13:00:19.169402+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:19.169402+0000","last_change":"2023-10-19T13:00:19.169402+0000","last_active":"2023-10-19T13:00:19.169402+0000","last_peered":"2023-10-19T13:00:19.169402+0000","last_clean":"2023-10-19T13:00:19.169402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:19.169402+0000","last_undegraded":"2023-10-19T13:00:19.169402+0000","last_fullsized":"2023-10-19T13:00:19.169402+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.14","version":"86'6","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-19T13:00:21.240123+0000","last_change":"2023-10-19T13:00:20.198504+0000","last_active":"2023-10-19T13:00:21.240123+0000","last_peered":"2023-10-19T13:00:21.240123+0000","last_clean":"2023-10-19T13:00:21.240123+0000","last_became_active":"2023-10-19T13:00:20.197655+0000","last_became_peered":"2023-10-19T13:00:20.197655+0000","last_unstale":"2023-10-19T13:00:21.240123+0000","last_undegraded":"2023-10-19T13:00:21.240123+0000","last_fullsized":"2023-10-19T13:00:21.240123+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:25:20.022851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:19.169402+0000","last_change":"2023-10-19T13:00:19.169402+0000","last_active":"2023-10-19T13:00:19.169402+0000","last_peered":"2023-10-19T13:00:19.169402+0000","last_clean":"2023-10-19T13:00:19.169402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:19.169402+0000","last_undegraded":"2023-10-19T13:00:19.169402+0000","last_fullsized":"2023-10-19T13:00:19.169402+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:19.169402+0000","last_change":"2023-10-19T13:00:19.169402+0000","last_active":"2023-10-19T13:00:19.169402+0000","last_peered":"2023-10-19T13:00:19.169402+0000","last_clean":"2023-10-19T13:00:19.169402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:19.169402+0000","last_undegraded":"2023-10-19T13:00:19.169402+0000","last_fullsized":"2023-10-19T13:00:19.169402+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:19.169402+0000","last_change":"2023-10-19T13:00:19.169402+0000","last_active":"2023-10-19T13:00:19.169402+0000","last_peered":"2023-10-19T13:00:19.169402+0000","last_clean":"2023-10-19T13:00:19.169402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:19.169402+0000","last_undegraded":"2023-10-19T13:00:19.169402+0000","last_fullsized":"2023-10-19T13:00:19.169402+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":9,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-19T13:00:21.232936+0000","last_change":"2023-10-19T13:00:21.228024+0000","last_active":"2023-10-19T13:00:21.232936+0000","last_peered":"2023-10-19T13:00:21.232936+0000","last_clean":"2023-10-19T13:00:21.232936+0000","last_became_active":"2023-10-19T13:00:21.227727+0000","last_became_peered":"2023-10-19T13:00:21.227727+0000","last_unstale":"2023-10-19T13:00:21.232936+0000","last_undegraded":"2023-10-19T13:00:21.232936+0000","last_fullsized":"2023-10-19T13:00:21.232936+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:14:55.381409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:19.169402+0000","last_change":"2023-10-19T13:00:19.169402+0000","last_active":"2023-10-19T13:00:19.169402+0000","last_peered":"2023-10-19T13:00:19.169402+0000","last_clean":"2023-10-19T13:00:19.169402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:19.169402+0000","last_undegraded":"2023-10-19T13:00:19.169402+0000","last_fullsized":"2023-10-19T13:00:19.169402+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:19.169402+0000","last_change":"2023-10-19T13:00:19.169402+0000","last_active":"2023-10-19T13:00:19.169402+0000","last_peered":"2023-10-19T13:00:19.169402+0000","last_clean":"2023-10-19T13:00:19.169402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:19.169402+0000","last_undegraded":"2023-10-19T13:00:19.169402+0000","last_fullsized":"2023-10-19T13:00:19.169402+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:19.169402+0000","last_change":"2023-10-19T13:00:19.169402+0000","last_active":"2023-10-19T13:00:19.169402+0000","last_peered":"2023-10-19T13:00:19.169402+0000","last_clean":"2023-10-19T13:00:19.169402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:19.169402+0000","last_undegraded":"2023-10-19T13:00:19.169402+0000","last_fullsized":"2023-10-19T13:00:19.169402+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:19.169402+0000","last_change":"2023-10-19T13:00:19.169402+0000","last_active":"2023-10-19T13:00:19.169402+0000","last_peered":"2023-10-19T13:00:19.169402+0000","last_clean":"2023-10-19T13:00:19.169402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:19.169402+0000","last_undegraded":"2023-10-19T13:00:19.169402+0000","last_fullsized":"2023-10-19T13:00:19.169402+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:19.169402+0000","last_change":"2023-10-19T13:00:19.169402+0000","last_active":"2023-10-19T13:00:19.169402+0000","last_peered":"2023-10-19T13:00:19.169402+0000","last_clean":"2023-10-19T13:00:19.169402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:19.169402+0000","last_undegraded":"2023-10-19T13:00:19.169402+0000","last_fullsized":"2023-10-19T13:00:19.169402+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.220733+0000","last_change":"2023-10-19T13:00:21.217141+0000","last_active":"2023-10-19T13:00:22.220733+0000","last_peered":"2023-10-19T13:00:22.220733+0000","last_clean":"2023-10-19T13:00:22.220733+0000","last_became_active":"2023-10-19T13:00:21.216807+0000","last_became_peered":"2023-10-19T13:00:21.216807+0000","last_unstale":"2023-10-19T13:00:22.220733+0000","last_undegraded":"2023-10-19T13:00:22.220733+0000","last_fullsized":"2023-10-19T13:00:22.220733+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:39:34.098405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"86'2","reported_seq":14,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.220911+0000","last_change":"2023-10-19T13:00:20.195565+0000","last_active":"2023-10-19T13:00:22.220911+0000","last_peered":"2023-10-19T13:00:22.220911+0000","last_clean":"2023-10-19T13:00:22.220911+0000","last_became_active":"2023-10-19T13:00:20.195256+0000","last_became_peered":"2023-10-19T13:00:20.195256+0000","last_unstale":"2023-10-19T13:00:22.220911+0000","last_undegraded":"2023-10-19T13:00:22.220911+0000","last_fullsized":"2023-10-19T13:00:22.220911+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:18:37.647497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:19.169402+0000","last_change":"2023-10-19T13:00:19.169402+0000","last_active":"2023-10-19T13:00:19.169402+0000","last_peered":"2023-10-19T13:00:19.169402+0000","last_clean":"2023-10-19T13:00:19.169402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:19.169402+0000","last_undegraded":"2023-10-19T13:00:19.169402+0000","last_fullsized":"2023-10-19T13:00:19.169402+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.220731+0000","last_change":"2023-10-19T13:00:20.194028+0000","last_active":"2023-10-19T13:00:22.220731+0000","last_peered":"2023-10-19T13:00:22.220731+0000","last_clean":"2023-10-19T13:00:22.220731+0000","last_became_active":"2023-10-19T13:00:20.193758+0000","last_became_peered":"2023-10-19T13:00:20.193758+0000","last_unstale":"2023-10-19T13:00:22.220731+0000","last_undegraded":"2023-10-19T13:00:22.220731+0000","last_fullsized":"2023-10-19T13:00:22.220731+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:41:30.219455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.221024+0000","last_change":"2023-10-19T13:00:21.221247+0000","last_active":"2023-10-19T13:00:22.221024+0000","last_peered":"2023-10-19T13:00:22.221024+0000","last_clean":"2023-10-19T13:00:22.221024+0000","last_became_active":"2023-10-19T13:00:21.220182+0000","last_became_peered":"2023-10-19T13:00:21.220182+0000","last_unstale":"2023-10-19T13:00:22.221024+0000","last_undegraded":"2023-10-19T13:00:22.221024+0000","last_fullsized":"2023-10-19T13:00:22.221024+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:47:08.944926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:19.169402+0000","last_change":"2023-10-19T13:00:19.169402+0000","last_active":"2023-10-19T13:00:19.169402+0000","last_peered":"2023-10-19T13:00:19.169402+0000","last_clean":"2023-10-19T13:00:19.169402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:19.169402+0000","last_undegraded":"2023-10-19T13:00:19.169402+0000","last_fullsized":"2023-10-19T13:00:19.169402+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:23.044679+0000","last_change":"2023-10-19T13:00:21.222511+0000","last_active":"2023-10-19T13:00:23.044679+0000","last_peered":"2023-10-19T13:00:23.044679+0000","last_clean":"2023-10-19T13:00:23.044679+0000","last_became_active":"2023-10-19T13:00:21.222236+0000","last_became_peered":"2023-10-19T13:00:21.222236+0000","last_unstale":"2023-10-19T13:00:23.044679+0000","last_undegraded":"2023-10-19T13:00:23.044679+0000","last_fullsized":"2023-10-19T13:00:23.044679+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:50:46.028354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:19.169402+0000","last_change":"2023-10-19T13:00:19.169402+0000","last_active":"2023-10-19T13:00:19.169402+0000","last_peered":"2023-10-19T13:00:19.169402+0000","last_clean":"2023-10-19T13:00:19.169402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:19.169402+0000","last_undegraded":"2023-10-19T13:00:19.169402+0000","last_fullsized":"2023-10-19T13:00:19.169402+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.221031+0000","last_change":"2023-10-19T13:00:20.188530+0000","last_active":"2023-10-19T13:00:22.221031+0000","last_peered":"2023-10-19T13:00:22.221031+0000","last_clean":"2023-10-19T13:00:22.221031+0000","last_became_active":"2023-10-19T13:00:20.188074+0000","last_became_peered":"2023-10-19T13:00:20.188074+0000","last_unstale":"2023-10-19T13:00:22.221031+0000","last_undegraded":"2023-10-19T13:00:22.221031+0000","last_fullsized":"2023-10-19T13:00:22.221031+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:08:44.211054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-19T13:00:21.216870+0000","last_change":"2023-10-19T13:00:20.198569+0000","last_active":"2023-10-19T13:00:21.216870+0000","last_peered":"2023-10-19T13:00:21.216870+0000","last_clean":"2023-10-19T13:00:21.216870+0000","last_became_active":"2023-10-19T13:00:20.197088+0000","last_became_peered":"2023-10-19T13:00:20.197088+0000","last_unstale":"2023-10-19T13:00:21.216870+0000","last_undegraded":"2023-10-19T13:00:21.216870+0000","last_fullsized":"2023-10-19T13:00:21.216870+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:38:39.386315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:19.169402+0000","last_change":"2023-10-19T13:00:19.169402+0000","last_active":"2023-10-19T13:00:19.169402+0000","last_peered":"2023-10-19T13:00:19.169402+0000","last_clean":"2023-10-19T13:00:19.169402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:19.169402+0000","last_undegraded":"2023-10-19T13:00:19.169402+0000","last_fullsized":"2023-10-19T13:00:19.169402+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:19.169402+0000","last_change":"2023-10-19T13:00:19.169402+0000","last_active":"2023-10-19T13:00:19.169402+0000","last_peered":"2023-10-19T13:00:19.169402+0000","last_clean":"2023-10-19T13:00:19.169402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:19.169402+0000","last_undegraded":"2023-10-19T13:00:19.169402+0000","last_fullsized":"2023-10-19T13:00:19.169402+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":9,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-19T13:00:21.227472+0000","last_change":"2023-10-19T13:00:21.226366+0000","last_active":"2023-10-19T13:00:21.227472+0000","last_peered":"2023-10-19T13:00:21.227472+0000","last_clean":"2023-10-19T13:00:21.227472+0000","last_became_active":"2023-10-19T13:00:21.226075+0000","last_became_peered":"2023-10-19T13:00:21.226075+0000","last_unstale":"2023-10-19T13:00:21.227472+0000","last_undegraded":"2023-10-19T13:00:21.227472+0000","last_fullsized":"2023-10-19T13:00:21.227472+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:33:36.402003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:19.169402+0000","last_change":"2023-10-19T13:00:19.169402+0000","last_active":"2023-10-19T13:00:19.169402+0000","last_peered":"2023-10-19T13:00:19.169402+0000","last_clean":"2023-10-19T13:00:19.169402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:19.169402+0000","last_undegraded":"2023-10-19T13:00:19.169402+0000","last_fullsized":"2023-10-19T13:00:19.169402+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:23.044464+0000","last_change":"2023-10-19T13:00:21.219143+0000","last_active":"2023-10-19T13:00:23.044464+0000","last_peered":"2023-10-19T13:00:23.044464+0000","last_clean":"2023-10-19T13:00:23.044464+0000","last_became_active":"2023-10-19T13:00:21.218834+0000","last_became_peered":"2023-10-19T13:00:21.218834+0000","last_unstale":"2023-10-19T13:00:23.044464+0000","last_undegraded":"2023-10-19T13:00:23.044464+0000","last_fullsized":"2023-10-19T13:00:23.044464+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:29:26.836263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-19T13:00:21.215556+0000","last_change":"2023-10-19T13:00:20.198235+0000","last_active":"2023-10-19T13:00:21.215556+0000","last_peered":"2023-10-19T13:00:21.215556+0000","last_clean":"2023-10-19T13:00:21.215556+0000","last_became_active":"2023-10-19T13:00:20.197920+0000","last_became_peered":"2023-10-19T13:00:20.197920+0000","last_unstale":"2023-10-19T13:00:21.215556+0000","last_undegraded":"2023-10-19T13:00:21.215556+0000","last_fullsized":"2023-10-19T13:00:21.215556+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:34:56.789811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:19.169402+0000","last_change":"2023-10-19T13:00:19.169402+0000","last_active":"2023-10-19T13:00:19.169402+0000","last_peered":"2023-10-19T13:00:19.169402+0000","last_clean":"2023-10-19T13:00:19.169402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:19.169402+0000","last_undegraded":"2023-10-19T13:00:19.169402+0000","last_fullsized":"2023-10-19T13:00:19.169402+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"76'161","reported_seq":381,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-19T13:00:16.745788+0000","last_change":"2023-10-19T12:44:08.636613+0000","last_active":"2023-10-19T13:00:16.745788+0000","last_peered":"2023-10-19T13:00:16.745788+0000","last_clean":"2023-10-19T13:00:16.745788+0000","last_became_active":"2023-10-19T12:44:08.635697+0000","last_became_peered":"2023-10-19T12:44:08.635697+0000","last_unstale":"2023-10-19T13:00:16.745788+0000","last_undegraded":"2023-10-19T13:00:16.745788+0000","last_fullsized":"2023-10-19T13:00:16.745788+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_clean_scrub_stamp":"2023-10-19T12:44:07.399738+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:15:23.685485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":36,"acting":36,"num_store_stats":3},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":2194,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":20,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575045,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59112,"kb_used_data":1564,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312728,"statfs":{"total":96636764160,"available":96576233472,"internally_reserved":0,"allocated":1601536,"data_stored":1092689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58915275},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:59:44 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.57499999999999996,"15min":0.56999999999999995},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.248},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.60099999999999998,"15min":0.58799999999999997},"min":{"1min":0.45400000000000001,"5min":0.33200000000000002,"15min":0.26900000000000002},"max":{"1min":0.78200000000000003,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.50800000000000001}]},{"osd":1,"last update":"Thu Oct 19 12:59:44 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.58199999999999996,"15min":0.57599999999999996},"min":{"1min":0.41299999999999998,"5min":0.40300000000000002,"15min":0.28199999999999997},"max":{"1min":0.84299999999999997,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.623,"5min":0.59999999999999998,"15min":0.58799999999999997},"min":{"1min":0.51700000000000002,"5min":0.43099999999999999,"15min":0.29299999999999998},"max":{"1min":0.82699999999999996,"5min":1.284,"15min":1.284},"last":0.60399999999999998}]},{"osd":2,"last update":"Thu Oct 19 12:59:44 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.64100000000000001,"15min":0.629},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.375},"max":{"1min":0.79300000000000004,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.64600000000000002,"15min":0.63300000000000001},"min":{"1min":0.55300000000000005,"5min":0.41599999999999998,"15min":0.28399999999999997},"max":{"1min":0.81200000000000006,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.47299999999999998}]},{"osd":3,"last update":"Thu Oct 19 12:59:44 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60699999999999998,"15min":0.60099999999999998},"min":{"1min":0.47199999999999998,"5min":0.34999999999999998,"15min":0.34399999999999997},"max":{"1min":0.78900000000000003,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.67000000000000004,"15min":0.65500000000000003},"min":{"1min":0.58199999999999996,"5min":0.45300000000000001,"15min":0.39700000000000002},"max":{"1min":0.92000000000000004,"5min":1.329,"15min":1.329},"last":0.65400000000000003}]},{"osd":4,"last update":"Thu Oct 19 12:59:44 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.46800000000000003,"15min":0.46700000000000003},"min":{"1min":0.36899999999999999,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.72599999999999998,"5min":0.75800000000000001,"15min":0.80500000000000005},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.33600000000000002,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.623,"5min":0.75600000000000001,"15min":0.78900000000000003},"last":0.52400000000000002}]},{"osd":5,"last update":"Thu Oct 19 12:59:44 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.48299999999999998,"15min":0.48499999999999999},"min":{"1min":0.34399999999999997,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.68600000000000005,"5min":0.75800000000000001,"15min":0.90400000000000003},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.52200000000000002,"15min":0.51600000000000001},"min":{"1min":0.38,"5min":0.314,"15min":0.308},"max":{"1min":0.71499999999999997,"5min":0.78700000000000003,"15min":0.92200000000000004},"last":0.624}]},{"osd":6,"last update":"Thu Oct 19 12:59:44 2023","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.54200000000000004,"15min":0.54700000000000004},"min":{"1min":0.379,"5min":0.308,"15min":0.308},"max":{"1min":0.749,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.52100000000000002,"15min":0.52900000000000003},"min":{"1min":0.36899999999999999,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.76900000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.58799999999999997}]}]},{"osd":6,"up_from":13,"seq":55834575045,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32168,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339672,"statfs":{"total":96636764160,"available":96603824128,"internally_reserved":0,"allocated":1007616,"data_stored":502321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:59:28 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.57999999999999996,"15min":0.58699999999999997},"min":{"1min":0.41899999999999998,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.755,"5min":0.79900000000000004,"15min":0.81999999999999995},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.41999999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.56399999999999995}]},{"osd":1,"last update":"Thu Oct 19 12:59:28 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.60299999999999998,"15min":0.59999999999999998},"min":{"1min":0.39900000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.71999999999999997,"5min":0.83299999999999996,"15min":0.93899999999999995},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.59699999999999998,"15min":0.59999999999999998},"min":{"1min":0.42599999999999999,"5min":0.34300000000000003,"15min":0.28399999999999997},"max":{"1min":0.78200000000000003,"5min":0.82099999999999995,"15min":1.075},"last":0.68700000000000006}]},{"osd":2,"last update":"Thu Oct 19 12:59:28 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.42799999999999999,"5min":0.307,"15min":0.26600000000000001},"max":{"1min":0.84699999999999998,"5min":0.85299999999999998,"15min":0.85699999999999998},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.624,"5min":0.61299999999999999,"15min":0.60799999999999998},"min":{"1min":0.48499999999999999,"5min":0.35999999999999999,"15min":0.28399999999999997},"max":{"1min":0.82199999999999995,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.38900000000000001}]},{"osd":3,"last update":"Thu Oct 19 12:59:28 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.625,"15min":0.623},"min":{"1min":0.46400000000000002,"5min":0.38500000000000001,"15min":0.32600000000000001},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.61799999999999999,"15min":0.628},"min":{"1min":0.44900000000000001,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":1.0900000000000001},"last":0.49199999999999999}]},{"osd":4,"last update":"Thu Oct 19 12:59:28 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.46100000000000002,"15min":0.46999999999999997},"min":{"1min":0.32900000000000001,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.64400000000000002,"5min":0.77900000000000003,"15min":1.02},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.47999999999999998,"15min":0.48499999999999999},"min":{"1min":0.33200000000000002,"5min":0.32800000000000001,"15min":0.247},"max":{"1min":0.69099999999999995,"5min":0.752,"15min":0.80000000000000004},"last":0.73799999999999999}]},{"osd":5,"last update":"Thu Oct 19 12:59:28 2023","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.46800000000000003,"15min":0.47699999999999998},"min":{"1min":0.32900000000000001,"5min":0.30199999999999999,"15min":0.26600000000000001},"max":{"1min":0.61899999999999999,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.442},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.50800000000000001,"15min":0.51900000000000002},"min":{"1min":0.35699999999999998,"5min":0.32100000000000001,"15min":0.312},"max":{"1min":0.75600000000000001,"5min":0.85699999999999998,"15min":0.98799999999999999},"last":0.63300000000000001}]},{"osd":7,"last update":"Thu Oct 19 12:59:28 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.51800000000000002,"15min":0.52300000000000002},"min":{"1min":0.32500000000000001,"5min":0.29299999999999998,"15min":0.28199999999999997},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":1.036},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53100000000000003,"15min":0.54100000000000004},"min":{"1min":0.36499999999999999,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":1.1299999999999999},"last":0.65100000000000002}]}]},{"osd":5,"up_from":13,"seq":55834575045,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32176,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339664,"statfs":{"total":96636764160,"available":96603815936,"internally_reserved":0,"allocated":1007616,"data_stored":502321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:59:29 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.58799999999999997,"15min":0.58899999999999997},"min":{"1min":0.44700000000000001,"5min":0.28999999999999998,"15min":0.223},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.88800000000000001},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.63,"5min":0.58999999999999997,"15min":0.59599999999999997},"min":{"1min":0.42499999999999999,"5min":0.36099999999999999,"15min":0.27800000000000002},"max":{"1min":0.81399999999999995,"5min":0.84299999999999997,"15min":0.874},"last":0.82999999999999996}]},{"osd":1,"last update":"Thu Oct 19 12:59:29 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.57899999999999996,"15min":0.59399999999999997},"min":{"1min":0.46899999999999997,"5min":0.34599999999999997,"15min":0.32800000000000001},"max":{"1min":0.77500000000000002,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.59999999999999998,"15min":0.60599999999999998},"min":{"1min":0.49199999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83499999999999996},"last":0.70099999999999996}]},{"osd":2,"last update":"Thu Oct 19 12:59:29 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.57299999999999995,"15min":0.58299999999999996},"min":{"1min":0.48899999999999999,"5min":0.34100000000000003,"15min":0.307},"max":{"1min":0.73999999999999999,"5min":0.78600000000000003,"15min":0.80700000000000005},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.60499999999999998,"15min":0.61899999999999999},"min":{"1min":0.50900000000000001,"5min":0.38400000000000001,"15min":0.33700000000000002},"max":{"1min":0.80900000000000005,"5min":0.88100000000000001,"15min":0.93600000000000005},"last":0.79300000000000004}]},{"osd":3,"last update":"Thu Oct 19 12:59:29 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.59299999999999997,"15min":0.60199999999999998},"min":{"1min":0.48499999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.751,"5min":0.79900000000000004,"15min":0.874},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.625,"15min":0.63400000000000001},"min":{"1min":0.50900000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.89500000000000002},"last":0.71799999999999997}]},{"osd":4,"last update":"Thu Oct 19 12:59:29 2023","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.36099999999999999,"5min":0.29099999999999998,"15min":0.28899999999999998},"max":{"1min":0.61399999999999999,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.46700000000000003,"15min":0.47199999999999998},"min":{"1min":0.35899999999999999,"5min":0.251,"15min":0.23899999999999999},"max":{"1min":0.58299999999999996,"5min":0.79400000000000004,"15min":0.80500000000000005},"last":0.36199999999999999}]},{"osd":6,"last update":"Thu Oct 19 12:59:29 2023","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46500000000000002,"15min":0.47299999999999998},"min":{"1min":0.34499999999999997,"5min":0.27200000000000002,"15min":0.25900000000000001},"max":{"1min":0.56299999999999994,"5min":0.74399999999999999,"15min":0.79300000000000004},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.497,"5min":0.50900000000000001,"15min":0.502},"min":{"1min":0.373,"5min":0.34200000000000003,"15min":0.30399999999999999},"max":{"1min":0.64800000000000002,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.60499999999999998}]},{"osd":7,"last update":"Thu Oct 19 12:59:29 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.48599999999999999,"15min":0.49299999999999999},"min":{"1min":0.39400000000000002,"5min":0.32000000000000001,"15min":0.315},"max":{"1min":0.64200000000000002,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.624},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.53900000000000003,"15min":0.53700000000000003},"min":{"1min":0.46100000000000002,"5min":0.311,"15min":0.27500000000000002},"max":{"1min":0.66900000000000004,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.68100000000000005}]}]},{"osd":4,"up_from":13,"seq":55834575045,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32152,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339688,"statfs":{"total":96636764160,"available":96603840512,"internally_reserved":0,"allocated":995328,"data_stored":496916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56100000000000005,"15min":0.55900000000000005},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.28899999999999998},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.55800000000000005,"15min":0.55500000000000005},"min":{"1min":0.45400000000000001,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.77100000000000002,"5min":0.77800000000000002,"15min":0.78200000000000003},"last":0.39400000000000002}]},{"osd":1,"last update":"Thu Oct 19 12:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.60399999999999998,"15min":0.60799999999999998},"min":{"1min":0.33000000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.78600000000000003,"5min":0.80600000000000005,"15min":0.86199999999999999},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61899999999999999,"15min":0.61599999999999999},"min":{"1min":0.40899999999999997,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.77300000000000002,"5min":0.79900000000000004,"15min":0.873},"last":0.54700000000000004}]},{"osd":2,"last update":"Thu Oct 19 12:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.61199999999999999,"15min":0.61299999999999999},"min":{"1min":0.375,"5min":0.32200000000000001,"15min":0.254},"max":{"1min":0.72199999999999998,"5min":0.85299999999999998,"15min":0.86599999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61699999999999999,"15min":0.61599999999999999},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.34499999999999997},"max":{"1min":0.78500000000000003,"5min":0.82999999999999996,"15min":0.85999999999999999},"last":0.42099999999999999}]},{"osd":3,"last update":"Thu Oct 19 12:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.59099999999999997,"15min":0.58399999999999996},"min":{"1min":0.41999999999999998,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.73499999999999999,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.64600000000000002,"15min":0.63700000000000001},"min":{"1min":0.52500000000000002,"5min":0.42599999999999999,"15min":0.37},"max":{"1min":0.73899999999999999,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.624}]},{"osd":5,"last update":"Thu Oct 19 12:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.45600000000000002,"15min":0.45200000000000001},"min":{"1min":0.309,"5min":0.29599999999999999,"15min":0.249},"max":{"1min":0.63500000000000001,"5min":0.79600000000000004,"15min":0.81299999999999994},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.47499999999999998,"15min":0.46200000000000002},"min":{"1min":0.34699999999999998,"5min":0.26400000000000001,"15min":0.23499999999999999},"max":{"1min":0.623,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.505}]},{"osd":6,"last update":"Thu Oct 19 12:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.45900000000000002,"15min":0.45600000000000002},"min":{"1min":0.31,"5min":0.29499999999999998,"15min":0.26400000000000001},"max":{"1min":0.52800000000000002,"5min":0.72599999999999998,"15min":0.78600000000000003},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.48599999999999999,"15min":0.497},"min":{"1min":0.34300000000000003,"5min":0.32100000000000001,"15min":0.27100000000000002},"max":{"1min":0.66700000000000004,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.63600000000000001}]},{"osd":7,"last update":"Thu Oct 19 12:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.501,"15min":0.503},"min":{"1min":0.33700000000000002,"5min":0.307,"15min":0.27500000000000002},"max":{"1min":0.68300000000000005,"5min":0.79100000000000004,"15min":0.80200000000000005},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.503,"5min":0.54200000000000004,"15min":0.54800000000000004},"min":{"1min":0.39100000000000001,"5min":0.35799999999999998,"15min":0.30199999999999999},"max":{"1min":0.69399999999999995,"5min":0.80500000000000005,"15min":0.83499999999999996},"last":0.67200000000000004}]}]},{"osd":3,"up_from":13,"seq":55834575044,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":36332,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335508,"statfs":{"total":96636764160,"available":96599560192,"internally_reserved":0,"allocated":1007616,"data_stored":502321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:59:24 2023","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.41799999999999998,"15min":0.39900000000000002},"min":{"1min":0.27100000000000002,"5min":0.23599999999999999,"15min":0.20899999999999999},"max":{"1min":0.69099999999999995,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.252},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.41199999999999998,"15min":0.41299999999999998},"min":{"1min":0.316,"5min":0.223,"15min":0.223},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.28699999999999998}]},{"osd":1,"last update":"Thu Oct 19 12:59:24 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.40600000000000003,"15min":0.40699999999999997},"min":{"1min":0.29099999999999998,"5min":0.26100000000000001,"15min":0.22500000000000001},"max":{"1min":0.63500000000000001,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.433},{"interface":"front","average":{"1min":0.505,"5min":0.48299999999999998,"15min":0.45700000000000002},"min":{"1min":0.27600000000000002,"5min":0.24099999999999999,"15min":0.215},"max":{"1min":0.66700000000000004,"5min":2.9399999999999999,"15min":2.9399999999999999},"last":0.32700000000000001}]},{"osd":2,"last update":"Thu Oct 19 12:59:24 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.46899999999999997,"15min":0.46500000000000002},"min":{"1min":0.33800000000000002,"5min":0.314,"15min":0.26100000000000001},"max":{"1min":0.69399999999999995,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.48099999999999998,"15min":0.48499999999999999},"min":{"1min":0.35099999999999998,"5min":0.29999999999999999,"15min":0.247},"max":{"1min":0.754,"5min":0.754,"15min":0.96499999999999997},"last":0.39000000000000001}]},{"osd":4,"last update":"Thu Oct 19 12:59:24 2023","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.65700000000000003,"15min":0.66400000000000003},"min":{"1min":0.50700000000000001,"5min":0.42299999999999999,"15min":0.40300000000000002},"max":{"1min":0.90200000000000002,"5min":0.90500000000000003,"15min":1.0069999999999999},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.69499999999999995,"15min":0.69199999999999995},"min":{"1min":0.59499999999999997,"5min":0.46899999999999997,"15min":0.42099999999999999},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.99399999999999999},"last":0.67900000000000005}]},{"osd":5,"last update":"Thu Oct 19 12:59:24 2023","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.67600000000000005,"15min":0.65900000000000003},"min":{"1min":0.56000000000000005,"5min":0.45500000000000002,"15min":0.40500000000000003},"max":{"1min":0.93600000000000005,"5min":2.9990000000000001,"15min":2.9990000000000001},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.52100000000000002,"5min":0.40899999999999997,"15min":0.35899999999999999},"max":{"1min":0.81899999999999995,"5min":1.774,"15min":1.774},"last":0.59699999999999998}]},{"osd":6,"last update":"Thu Oct 19 12:59:24 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.66100000000000003,"15min":0.65400000000000003},"min":{"1min":0.52300000000000002,"5min":0.433,"15min":0.41199999999999998},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.96799999999999997},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.67500000000000004,"15min":0.67400000000000004},"min":{"1min":0.60299999999999998,"5min":0.377,"15min":0.35899999999999999},"max":{"1min":0.93700000000000006,"5min":1.167,"15min":1.167},"last":0.64500000000000002}]},{"osd":7,"last update":"Thu Oct 19 12:59:24 2023","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.65200000000000002,"15min":0.66500000000000004},"min":{"1min":0.45500000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.94299999999999995,"5min":1.042,"15min":1.042},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.69299999999999995,"15min":0.67900000000000005},"min":{"1min":0.54200000000000004,"5min":0.432,"15min":0.432},"max":{"1min":0.89200000000000002,"5min":2.8149999999999999,"15min":2.8149999999999999},"last":0.61299999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575047,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54416,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317424,"statfs":{"total":96636764160,"available":96581042176,"internally_reserved":0,"allocated":1044480,"data_stored":522224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:59:33 2023","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.377,"15min":0.38100000000000001},"min":{"1min":0.27500000000000002,"5min":0.20899999999999999,"15min":0.187},"max":{"1min":0.54000000000000004,"5min":0.67700000000000005,"15min":0.83999999999999997},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.436,"5min":0.40999999999999998,"15min":0.40600000000000003},"min":{"1min":0.32900000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.58299999999999996,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.56000000000000005}]},{"osd":1,"last update":"Thu Oct 19 12:59:33 2023","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.39400000000000002,"15min":0.39800000000000002},"min":{"1min":0.318,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.80400000000000005},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.41999999999999998,"15min":0.42999999999999999},"min":{"1min":0.38200000000000001,"5min":0.22,"15min":0.22},"max":{"1min":0.59499999999999997,"5min":0.60499999999999998,"15min":0.78600000000000003},"last":0.61299999999999999}]},{"osd":3,"last update":"Thu Oct 19 12:59:33 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.46300000000000002,"15min":0.45700000000000002},"min":{"1min":0.33400000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.498},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.48099999999999998,"15min":0.48499999999999999},"min":{"1min":0.40300000000000002,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.72799999999999998,"5min":0.77800000000000002,"15min":0.87},"last":0.57499999999999996}]},{"osd":4,"last update":"Thu Oct 19 12:59:33 2023","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.63700000000000001,"15min":0.64200000000000002},"min":{"1min":0.497,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.89500000000000002,"5min":0.93999999999999995,"15min":1.0569999999999999},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.65400000000000003,"15min":0.65700000000000003},"min":{"1min":0.54300000000000004,"5min":0.35099999999999998,"15min":0.32000000000000001},"max":{"1min":0.83399999999999996,"5min":0.92300000000000004,"15min":1.0780000000000001},"last":0.70999999999999996}]},{"osd":5,"last update":"Thu Oct 19 12:59:33 2023","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.67400000000000004,"15min":0.66200000000000003},"min":{"1min":0.58299999999999996,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.84299999999999997,"5min":0.96499999999999997,"15min":0.99399999999999999},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.65200000000000002,"15min":0.65800000000000003},"min":{"1min":0.56899999999999995,"5min":0.375,"15min":0.375},"max":{"1min":0.872,"5min":0.94599999999999995,"15min":0.97099999999999997},"last":0.64000000000000001}]},{"osd":6,"last update":"Thu Oct 19 12:59:33 2023","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.65000000000000002,"15min":0.66700000000000004},"min":{"1min":0.47199999999999998,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.82399999999999995,"5min":1.008,"15min":1.008},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.67800000000000005,"15min":0.67400000000000004},"min":{"1min":0.52000000000000002,"5min":0.371,"15min":0.371},"max":{"1min":0.88100000000000001,"5min":0.995,"15min":0.995},"last":0.63}]},{"osd":7,"last update":"Thu Oct 19 12:59:33 2023","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.67500000000000004,"15min":0.68500000000000005},"min":{"1min":0.59699999999999998,"5min":0.39300000000000002,"15min":0.36399999999999999},"max":{"1min":0.86099999999999999,"5min":1.0149999999999999,"15min":1.111},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.68899999999999995,"15min":0.68300000000000005},"min":{"1min":0.53000000000000003,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.89800000000000002,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.65300000000000002}]}]},{"osd":1,"up_from":13,"seq":55834575046,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32236,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339604,"statfs":{"total":96636764160,"available":96603754496,"internally_reserved":0,"allocated":1073152,"data_stored":524600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:59:33 2023","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.39200000000000002,"15min":0.38500000000000001},"min":{"1min":0.23999999999999999,"5min":0.21299999999999999,"15min":0.187},"max":{"1min":0.55800000000000005,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.442,"15min":0.42799999999999999},"min":{"1min":0.29499999999999998,"5min":0.23899999999999999,"15min":0.21199999999999999},"max":{"1min":0.58799999999999997,"5min":2.5800000000000001,"15min":2.5800000000000001},"last":0.52700000000000002}]},{"osd":2,"last update":"Thu Oct 19 12:59:33 2023","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.42499999999999999,"15min":0.42099999999999999},"min":{"1min":0.309,"5min":0.21299999999999999,"15min":0.20100000000000001},"max":{"1min":0.53300000000000003,"5min":0.60499999999999998,"15min":0.69999999999999996},"last":0.502},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.47099999999999997,"15min":0.45300000000000001},"min":{"1min":0.34999999999999998,"5min":0.313,"15min":0.23699999999999999},"max":{"1min":0.625,"5min":2.6230000000000002,"15min":2.6230000000000002},"last":0.57999999999999996}]},{"osd":3,"last update":"Thu Oct 19 12:59:33 2023","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.51600000000000001,"15min":0.48599999999999999},"min":{"1min":0.38900000000000001,"5min":0.307,"15min":0.26200000000000001},"max":{"1min":0.621,"5min":3.9740000000000002,"15min":3.9740000000000002},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.43099999999999999,"15min":0.42299999999999999},"min":{"1min":0.35999999999999999,"5min":0.27700000000000002,"15min":0.27000000000000002},"max":{"1min":0.59299999999999997,"5min":0.63200000000000001,"15min":0.80500000000000005},"last":0.51400000000000001}]},{"osd":4,"last update":"Thu Oct 19 12:59:33 2023","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.65600000000000003,"15min":0.65200000000000002},"min":{"1min":0.501,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.85699999999999998,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.66600000000000004,"15min":0.64800000000000002},"min":{"1min":0.47399999999999998,"5min":0.42599999999999999,"15min":0.314},"max":{"1min":0.85399999999999998,"5min":1.0860000000000001,"15min":1.1910000000000001},"last":0.76300000000000001}]},{"osd":5,"last update":"Thu Oct 19 12:59:33 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.65700000000000003,"15min":0.65300000000000002},"min":{"1min":0.499,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.83599999999999997,"5min":1.1200000000000001,"15min":1.1200000000000001},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.69599999999999995,"15min":0.67300000000000004},"min":{"1min":0.54300000000000004,"5min":0.42299999999999999,"15min":0.38700000000000001},"max":{"1min":0.81799999999999995,"5min":2.7120000000000002,"15min":2.7120000000000002},"last":0.77900000000000003}]},{"osd":6,"last update":"Thu Oct 19 12:59:33 2023","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.65800000000000003,"15min":0.65700000000000003},"min":{"1min":0.53800000000000003,"5min":0.36799999999999999,"15min":0.36399999999999999},"max":{"1min":0.91300000000000003,"5min":1.0069999999999999,"15min":1.077},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67400000000000004,"15min":0.67800000000000005},"min":{"1min":0.55800000000000005,"5min":0.376,"15min":0.376},"max":{"1min":0.77500000000000002,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.72099999999999997}]},{"osd":7,"last update":"Thu Oct 19 12:59:33 2023","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.69799999999999995,"15min":0.67900000000000005},"min":{"1min":0.54300000000000004,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.84999999999999998,"5min":2.786,"15min":2.786},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.68200000000000005,"15min":0.67900000000000005},"min":{"1min":0.47499999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":1.0489999999999999},"last":0.79500000000000004}]}]},{"osd":0,"up_from":13,"seq":55834575047,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36952,"kb_used_data":1608,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334888,"statfs":{"total":96636764160,"available":96598925312,"internally_reserved":0,"allocated":1646592,"data_stored":1118621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Oct 19 12:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.40300000000000002,"15min":0.39100000000000001},"min":{"1min":0.26500000000000001,"5min":0.182,"15min":0.182},"max":{"1min":0.58899999999999997,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.434,"15min":0.432},"min":{"1min":0.34599999999999997,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.746},"last":0.38200000000000001}]},{"osd":2,"last update":"Thu Oct 19 12:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.46700000000000003,"15min":0.45500000000000002},"min":{"1min":0.33200000000000002,"5min":0.29499999999999998,"15min":0.25800000000000001},"max":{"1min":0.58599999999999997,"5min":0.752,"15min":0.76300000000000001},"last":0.309},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.437,"15min":0.434},"min":{"1min":0.26000000000000001,"5min":0.216,"15min":0.20699999999999999},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.76800000000000002},"last":0.432}]},{"osd":3,"last update":"Thu Oct 19 12:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.42199999999999999,"15min":0.40899999999999997},"min":{"1min":0.34000000000000002,"5min":0.23400000000000001,"15min":0.22800000000000001},"max":{"1min":0.53500000000000003,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.505,"5min":0.504,"15min":0.49099999999999999},"min":{"1min":0.33600000000000002,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.66300000000000003,"5min":1.4990000000000001,"15min":1.4990000000000001},"last":0.34799999999999998}]},{"osd":4,"last update":"Thu Oct 19 12:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.63100000000000001,"15min":0.63600000000000001},"min":{"1min":0.46600000000000003,"5min":0.40000000000000002,"15min":0.34999999999999998},"max":{"1min":0.749,"5min":1.361,"15min":1.361},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.65700000000000003,"15min":0.65000000000000002},"min":{"1min":0.52900000000000003,"5min":0.41899999999999998,"15min":0.35999999999999999},"max":{"1min":0.871,"5min":0.98999999999999999,"15min":0.998},"last":0.57699999999999996}]},{"osd":5,"last update":"Thu Oct 19 12:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.68999999999999995,"15min":0.68000000000000005},"min":{"1min":0.50700000000000001,"5min":0.378,"15min":0.378},"max":{"1min":0.94599999999999995,"5min":1.325,"15min":1.325},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.69299999999999995,"15min":0.68600000000000005},"min":{"1min":0.45900000000000002,"5min":0.36299999999999999,"15min":0.318},"max":{"1min":1.02,"5min":1.024,"15min":1.056},"last":0.625}]},{"osd":6,"last update":"Thu Oct 19 12:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.67200000000000004,"15min":0.66400000000000003},"min":{"1min":0.59999999999999998,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.873,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.67600000000000005,"15min":0.66600000000000004},"min":{"1min":0.57199999999999995,"5min":0.41999999999999998,"15min":0.38900000000000001},"max":{"1min":0.92000000000000004,"5min":0.995,"15min":1.109},"last":0.65600000000000003}]},{"osd":7,"last update":"Thu Oct 19 12:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.68700000000000006,"15min":0.68400000000000005},"min":{"1min":0.53500000000000003,"5min":0.41199999999999998,"15min":0.41099999999999998},"max":{"1min":0.89900000000000002,"5min":0.98799999999999999,"15min":1.0089999999999999},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.67300000000000004,"15min":0.66300000000000003},"min":{"1min":0.58799999999999997,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.90400000000000003,"5min":1.389,"15min":1.389},"last":0.67100000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-19T13:00:24.225 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-19T13:00:24.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.367+0000 7fa79e697640 1 -- 172.21.15.57:0/3233610091 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa780003970 msgr2=0x7fa780003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:24.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.367+0000 7fa79e697640 1 --2- 172.21.15.57:0/3233610091 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa780003970 0x7fa780003d90 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fa78800c260 tx=0x7fa7800057f0 comp rx=0 tx=0).stop 2023-10-19T13:00:24.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.367+0000 7fa79cc34640 1 -- 172.21.15.57:0/3233610091 reap_dead start 2023-10-19T13:00:24.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.367+0000 7fa79e697640 1 -- 172.21.15.57:0/3233610091 shutdown_connections 2023-10-19T13:00:24.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.367+0000 7fa79e697640 1 -- 172.21.15.57:0/3233610091 >> 172.21.15.57:0/3233610091 conn(0x7fa798069930 msgr2=0x7fa798069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:24.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.367+0000 7fa79e697640 1 -- 172.21.15.57:0/3233610091 shutdown_connections 2023-10-19T13:00:24.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.367+0000 7fa79e697640 1 -- 172.21.15.57:0/3233610091 wait complete. 2023-10-19T13:00:24.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.369+0000 7fa79e697640 1 Processor -- start 2023-10-19T13:00:24.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.369+0000 7fa79e697640 1 -- start start 2023-10-19T13:00:24.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.370+0000 7fa79e697640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa780003970 0x7fa798156ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:24.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.370+0000 7fa79e697640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa7981e50a0 0x7fa7981e5520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:24.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.370+0000 7fa79e697640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa7980ad0c0 0x7fa7980ad570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:24.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.370+0000 7fa79e697640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7981388b0 con 0x7fa7980ad0c0 2023-10-19T13:00:24.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.370+0000 7fa79e697640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7981e5a60 con 0x7fa780003970 2023-10-19T13:00:24.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.370+0000 7fa79e697640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7980ade20 con 0x7fa7981e50a0 2023-10-19T13:00:24.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.370+0000 7fa79cc34640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa7980ad0c0 0x7fa7980ad570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:24.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.370+0000 7fa79cc34640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa7980ad0c0 0x7fa7980ad570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:49878/0 (socket says 172.21.15.57:49878) 2023-10-19T13:00:24.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.370+0000 7fa79cc34640 1 -- 172.21.15.57:0/2950488911 learned_addr learned my addr 172.21.15.57:0/2950488911 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:24.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.370+0000 7fa7977fe640 1 --2- 172.21.15.57:0/2950488911 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa7981e50a0 0x7fa7981e5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:24.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.370+0000 7fa797fff640 1 --2- 172.21.15.57:0/2950488911 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa780003970 0x7fa798156ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:24.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.370+0000 7fa79cc34640 1 -- 172.21.15.57:0/2950488911 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa7981e50a0 msgr2=0x7fa7981e5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:24.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.370+0000 7fa79cc34640 1 --2- 172.21.15.57:0/2950488911 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa7981e50a0 0x7fa7981e5520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:24.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.370+0000 7fa79cc34640 1 -- 172.21.15.57:0/2950488911 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa780003970 msgr2=0x7fa798156ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:24.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.370+0000 7fa79cc34640 1 --2- 172.21.15.57:0/2950488911 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa780003970 0x7fa798156ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:24.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.370+0000 7fa79cc34640 1 -- 172.21.15.57:0/2950488911 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7981381f0 con 0x7fa7980ad0c0 2023-10-19T13:00:24.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.370+0000 7fa79cc34640 1 --2- 172.21.15.57:0/2950488911 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa7980ad0c0 0x7fa7980ad570 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7fa79806f410 tx=0x7fa7900103d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:24.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.370+0000 7fa794ff9640 1 -- 172.21.15.57:0/2950488911 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa790003b70 con 0x7fa7980ad0c0 2023-10-19T13:00:24.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.370+0000 7fa794ff9640 1 -- 172.21.15.57:0/2950488911 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fa790003cd0 con 0x7fa7980ad0c0 2023-10-19T13:00:24.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.370+0000 7fa79e697640 1 -- 172.21.15.57:0/2950488911 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7981384d0 con 0x7fa7980ad0c0 2023-10-19T13:00:24.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.370+0000 7fa794ff9640 1 -- 172.21.15.57:0/2950488911 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa790004290 con 0x7fa7980ad0c0 2023-10-19T13:00:24.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.370+0000 7fa79e697640 1 -- 172.21.15.57:0/2950488911 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7980adab0 con 0x7fa7980ad0c0 2023-10-19T13:00:24.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.371+0000 7fa79e697640 1 -- 172.21.15.57:0/2950488911 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7981389f0 con 0x7fa7980ad0c0 2023-10-19T13:00:24.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.372+0000 7fa794ff9640 1 -- 172.21.15.57:0/2950488911 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 32) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fa790018020 con 0x7fa7980ad0c0 2023-10-19T13:00:24.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.372+0000 7fa794ff9640 1 --2- 172.21.15.57:0/2950488911 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7fa78406af90 0x7fa78406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:24.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.372+0000 7fa794ff9640 1 -- 172.21.15.57:0/2950488911 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fa79008a890 con 0x7fa7980ad0c0 2023-10-19T13:00:24.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.372+0000 7fa797fff640 1 --2- 172.21.15.57:0/2950488911 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7fa78406af90 0x7fa78406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:24.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.373+0000 7fa797fff640 1 --2- 172.21.15.57:0/2950488911 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7fa78406af90 0x7fa78406d450 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa788034000 tx=0x7fa788001a70 comp rx=0 tx=0).ready entity=mgr.6256 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:24.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.376+0000 7fa794ff9640 1 -- 172.21.15.57:0/2950488911 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa7981389f0 con 0x7fa7980ad0c0 2023-10-19T13:00:24.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.548+0000 7fa79e697640 1 -- 172.21.15.57:0/2950488911 --> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa7980cc100 con 0x7fa78406af90 2023-10-19T13:00:24.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.555+0000 7fa794ff9640 1 -- 172.21.15.57:0/2950488911 <== mgr.6256 v2:172.21.15.179:6832/3815793798 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+279504 (secure 0 0 0) 0x7fa788039030 con 0x7fa78406af90 2023-10-19T13:00:24.561 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:00:24.562 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2023-10-19T13:00:24.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.558+0000 7fa79e697640 1 -- 172.21.15.57:0/2950488911 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7fa78406af90 msgr2=0x7fa78406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:24.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.558+0000 7fa79e697640 1 --2- 172.21.15.57:0/2950488911 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7fa78406af90 0x7fa78406d450 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa788034000 tx=0x7fa788001a70 comp rx=0 tx=0).stop 2023-10-19T13:00:24.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.558+0000 7fa79e697640 1 -- 172.21.15.57:0/2950488911 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa7980ad0c0 msgr2=0x7fa7980ad570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:24.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.558+0000 7fa79e697640 1 --2- 172.21.15.57:0/2950488911 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa7980ad0c0 0x7fa7980ad570 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7fa79806f410 tx=0x7fa7900103d0 comp rx=0 tx=0).stop 2023-10-19T13:00:24.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.558+0000 7fa79e697640 1 -- 172.21.15.57:0/2950488911 shutdown_connections 2023-10-19T13:00:24.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.558+0000 7fa79e697640 1 --2- 172.21.15.57:0/2950488911 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7fa78406af90 0x7fa78406d450 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:24.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.558+0000 7fa79e697640 1 --2- 172.21.15.57:0/2950488911 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa7980ad0c0 0x7fa7980ad570 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:24.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.558+0000 7fa79e697640 1 --2- 172.21.15.57:0/2950488911 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa7981e50a0 0x7fa7981e5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:24.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.558+0000 7fa79e697640 1 --2- 172.21.15.57:0/2950488911 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa780003970 0x7fa798156ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:24.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.558+0000 7fa79e697640 1 -- 172.21.15.57:0/2950488911 >> 172.21.15.57:0/2950488911 conn(0x7fa798069930 msgr2=0x7fa7980ca870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:24.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.558+0000 7fa79e697640 1 -- 172.21.15.57:0/2950488911 shutdown_connections 2023-10-19T13:00:24.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.558+0000 7fa79e697640 1 -- 172.21.15.57:0/2950488911 wait complete. 2023-10-19T13:00:24.572 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2023-10-19T13:00:23.049618+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592562,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":256,"num_write_kb":2420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":178,"ondisk_log_size":178,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":43,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":4,"kb":754974720,"kb_used":315544,"kb_used_data":9164,"kb_used_omap":12,"kb_used_meta":306227,"kb_avail":754659176,"statfs":{"total":773094113280,"available":772770996224,"internally_reserved":0,"allocated":9383936,"data_stored":5262013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":313577017},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2194,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000992"},"pg_stats":[{"pgid":"10.3f","version":"0'0","reported_seq":9,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-19T13:00:21.237877+0000","last_change":"2023-10-19T13:00:21.228303+0000","last_active":"2023-10-19T13:00:21.237877+0000","last_peered":"2023-10-19T13:00:21.237877+0000","last_clean":"2023-10-19T13:00:21.237877+0000","last_became_active":"2023-10-19T13:00:21.227974+0000","last_became_peered":"2023-10-19T13:00:21.227974+0000","last_unstale":"2023-10-19T13:00:21.237877+0000","last_undegraded":"2023-10-19T13:00:21.237877+0000","last_fullsized":"2023-10-19T13:00:21.237877+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:37:52.369486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3e","version":"0'0","reported_seq":9,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-19T13:00:21.233058+0000","last_change":"2023-10-19T13:00:21.227759+0000","last_active":"2023-10-19T13:00:21.233058+0000","last_peered":"2023-10-19T13:00:21.233058+0000","last_clean":"2023-10-19T13:00:21.233058+0000","last_became_active":"2023-10-19T13:00:21.227466+0000","last_became_peered":"2023-10-19T13:00:21.227466+0000","last_unstale":"2023-10-19T13:00:21.233058+0000","last_undegraded":"2023-10-19T13:00:21.233058+0000","last_fullsized":"2023-10-19T13:00:21.233058+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:55:42.567075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.3a","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.221076+0000","last_change":"2023-10-19T13:00:21.219325+0000","last_active":"2023-10-19T13:00:22.221076+0000","last_peered":"2023-10-19T13:00:22.221076+0000","last_clean":"2023-10-19T13:00:22.221076+0000","last_became_active":"2023-10-19T13:00:21.218458+0000","last_became_peered":"2023-10-19T13:00:21.218458+0000","last_unstale":"2023-10-19T13:00:22.221076+0000","last_undegraded":"2023-10-19T13:00:22.221076+0000","last_fullsized":"2023-10-19T13:00:22.221076+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:31:39.478057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.39","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:23.044542+0000","last_change":"2023-10-19T13:00:21.220590+0000","last_active":"2023-10-19T13:00:23.044542+0000","last_peered":"2023-10-19T13:00:23.044542+0000","last_clean":"2023-10-19T13:00:23.044542+0000","last_became_active":"2023-10-19T13:00:21.220337+0000","last_became_peered":"2023-10-19T13:00:21.220337+0000","last_unstale":"2023-10-19T13:00:23.044542+0000","last_undegraded":"2023-10-19T13:00:23.044542+0000","last_fullsized":"2023-10-19T13:00:23.044542+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:25:25.779991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.30","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:23.044519+0000","last_change":"2023-10-19T13:00:21.219839+0000","last_active":"2023-10-19T13:00:23.044519+0000","last_peered":"2023-10-19T13:00:23.044519+0000","last_clean":"2023-10-19T13:00:23.044519+0000","last_became_active":"2023-10-19T13:00:21.219524+0000","last_became_peered":"2023-10-19T13:00:21.219524+0000","last_unstale":"2023-10-19T13:00:23.044519+0000","last_undegraded":"2023-10-19T13:00:23.044519+0000","last_fullsized":"2023-10-19T13:00:23.044519+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:16:36.020089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2f","version":"0'0","reported_seq":9,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-19T13:00:21.237892+0000","last_change":"2023-10-19T13:00:21.228480+0000","last_active":"2023-10-19T13:00:21.237892+0000","last_peered":"2023-10-19T13:00:21.237892+0000","last_clean":"2023-10-19T13:00:21.237892+0000","last_became_active":"2023-10-19T13:00:21.227687+0000","last_became_peered":"2023-10-19T13:00:21.227687+0000","last_unstale":"2023-10-19T13:00:21.237892+0000","last_undegraded":"2023-10-19T13:00:21.237892+0000","last_fullsized":"2023-10-19T13:00:21.237892+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:58:41.166338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.24","version":"0'0","reported_seq":9,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-19T13:00:21.233957+0000","last_change":"2023-10-19T13:00:21.227180+0000","last_active":"2023-10-19T13:00:21.233957+0000","last_peered":"2023-10-19T13:00:21.233957+0000","last_clean":"2023-10-19T13:00:21.233957+0000","last_became_active":"2023-10-19T13:00:21.226763+0000","last_became_peered":"2023-10-19T13:00:21.226763+0000","last_unstale":"2023-10-19T13:00:21.233957+0000","last_undegraded":"2023-10-19T13:00:21.233957+0000","last_fullsized":"2023-10-19T13:00:21.233957+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:41:19.821669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.20","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.220680+0000","last_change":"2023-10-19T13:00:21.217955+0000","last_active":"2023-10-19T13:00:22.220680+0000","last_peered":"2023-10-19T13:00:22.220680+0000","last_clean":"2023-10-19T13:00:22.220680+0000","last_became_active":"2023-10-19T13:00:21.217649+0000","last_became_peered":"2023-10-19T13:00:21.217649+0000","last_unstale":"2023-10-19T13:00:22.220680+0000","last_undegraded":"2023-10-19T13:00:22.220680+0000","last_fullsized":"2023-10-19T13:00:22.220680+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:55:17.988760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1f","version":"86'4","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-19T13:00:21.238495+0000","last_change":"2023-10-19T13:00:20.197586+0000","last_active":"2023-10-19T13:00:21.238495+0000","last_peered":"2023-10-19T13:00:21.238495+0000","last_clean":"2023-10-19T13:00:21.238495+0000","last_became_active":"2023-10-19T13:00:20.197286+0000","last_became_peered":"2023-10-19T13:00:20.197286+0000","last_unstale":"2023-10-19T13:00:21.238495+0000","last_undegraded":"2023-10-19T13:00:21.238495+0000","last_fullsized":"2023-10-19T13:00:21.238495+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:45:06.777683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1e","version":"86'1","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.221295+0000","last_change":"2023-10-19T13:00:20.194934+0000","last_active":"2023-10-19T13:00:22.221295+0000","last_peered":"2023-10-19T13:00:22.221295+0000","last_clean":"2023-10-19T13:00:22.221295+0000","last_became_active":"2023-10-19T13:00:20.194657+0000","last_became_peered":"2023-10-19T13:00:20.194657+0000","last_unstale":"2023-10-19T13:00:22.221295+0000","last_undegraded":"2023-10-19T13:00:22.221295+0000","last_fullsized":"2023-10-19T13:00:22.221295+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:12:57.142360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.221258+0000","last_change":"2023-10-19T13:00:21.218060+0000","last_active":"2023-10-19T13:00:22.221258+0000","last_peered":"2023-10-19T13:00:22.221258+0000","last_clean":"2023-10-19T13:00:22.221258+0000","last_became_active":"2023-10-19T13:00:21.217766+0000","last_became_peered":"2023-10-19T13:00:21.217766+0000","last_unstale":"2023-10-19T13:00:22.221258+0000","last_undegraded":"2023-10-19T13:00:22.221258+0000","last_fullsized":"2023-10-19T13:00:22.221258+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:50:26.459690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"86'2","reported_seq":14,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.220954+0000","last_change":"2023-10-19T13:00:20.188312+0000","last_active":"2023-10-19T13:00:22.220954+0000","last_peered":"2023-10-19T13:00:22.220954+0000","last_clean":"2023-10-19T13:00:22.220954+0000","last_became_active":"2023-10-19T13:00:20.187673+0000","last_became_peered":"2023-10-19T13:00:20.187673+0000","last_unstale":"2023-10-19T13:00:22.220954+0000","last_undegraded":"2023-10-19T13:00:22.220954+0000","last_fullsized":"2023-10-19T13:00:22.220954+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:22:13.247630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1c","version":"86'2","reported_seq":14,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.221016+0000","last_change":"2023-10-19T13:00:20.197923+0000","last_active":"2023-10-19T13:00:22.221016+0000","last_peered":"2023-10-19T13:00:22.221016+0000","last_clean":"2023-10-19T13:00:22.221016+0000","last_became_active":"2023-10-19T13:00:20.197546+0000","last_became_peered":"2023-10-19T13:00:20.197546+0000","last_unstale":"2023-10-19T13:00:22.221016+0000","last_undegraded":"2023-10-19T13:00:22.221016+0000","last_fullsized":"2023-10-19T13:00:22.221016+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:27:45.352182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:23.044381+0000","last_change":"2023-10-19T13:00:21.229195+0000","last_active":"2023-10-19T13:00:23.044381+0000","last_peered":"2023-10-19T13:00:23.044381+0000","last_clean":"2023-10-19T13:00:23.044381+0000","last_became_active":"2023-10-19T13:00:21.228898+0000","last_became_peered":"2023-10-19T13:00:21.228898+0000","last_unstale":"2023-10-19T13:00:23.044381+0000","last_undegraded":"2023-10-19T13:00:23.044381+0000","last_fullsized":"2023-10-19T13:00:23.044381+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:28:07.228859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:19.169402+0000","last_change":"2023-10-19T13:00:19.169402+0000","last_active":"2023-10-19T13:00:19.169402+0000","last_peered":"2023-10-19T13:00:19.169402+0000","last_clean":"2023-10-19T13:00:19.169402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:19.169402+0000","last_undegraded":"2023-10-19T13:00:19.169402+0000","last_fullsized":"2023-10-19T13:00:19.169402+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":9,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-19T13:00:21.232881+0000","last_change":"2023-10-19T13:00:21.226995+0000","last_active":"2023-10-19T13:00:21.232881+0000","last_peered":"2023-10-19T13:00:21.232881+0000","last_clean":"2023-10-19T13:00:21.232881+0000","last_became_active":"2023-10-19T13:00:21.226673+0000","last_became_peered":"2023-10-19T13:00:21.226673+0000","last_unstale":"2023-10-19T13:00:21.232881+0000","last_undegraded":"2023-10-19T13:00:21.232881+0000","last_fullsized":"2023-10-19T13:00:21.232881+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:52:56.240367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:19.169402+0000","last_change":"2023-10-19T13:00:19.169402+0000","last_active":"2023-10-19T13:00:19.169402+0000","last_peered":"2023-10-19T13:00:19.169402+0000","last_clean":"2023-10-19T13:00:19.169402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:19.169402+0000","last_undegraded":"2023-10-19T13:00:19.169402+0000","last_fullsized":"2023-10-19T13:00:19.169402+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:19.169402+0000","last_change":"2023-10-19T13:00:19.169402+0000","last_active":"2023-10-19T13:00:19.169402+0000","last_peered":"2023-10-19T13:00:19.169402+0000","last_clean":"2023-10-19T13:00:19.169402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:19.169402+0000","last_undegraded":"2023-10-19T13:00:19.169402+0000","last_fullsized":"2023-10-19T13:00:19.169402+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:19.169402+0000","last_change":"2023-10-19T13:00:19.169402+0000","last_active":"2023-10-19T13:00:19.169402+0000","last_peered":"2023-10-19T13:00:19.169402+0000","last_clean":"2023-10-19T13:00:19.169402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:19.169402+0000","last_undegraded":"2023-10-19T13:00:19.169402+0000","last_fullsized":"2023-10-19T13:00:19.169402+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":9,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-19T13:00:21.233976+0000","last_change":"2023-10-19T13:00:21.227321+0000","last_active":"2023-10-19T13:00:21.233976+0000","last_peered":"2023-10-19T13:00:21.233976+0000","last_clean":"2023-10-19T13:00:21.233976+0000","last_became_active":"2023-10-19T13:00:21.227028+0000","last_became_peered":"2023-10-19T13:00:21.227028+0000","last_unstale":"2023-10-19T13:00:21.233976+0000","last_undegraded":"2023-10-19T13:00:21.233976+0000","last_fullsized":"2023-10-19T13:00:21.233976+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:06:08.406600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:19.169402+0000","last_change":"2023-10-19T13:00:19.169402+0000","last_active":"2023-10-19T13:00:19.169402+0000","last_peered":"2023-10-19T13:00:19.169402+0000","last_clean":"2023-10-19T13:00:19.169402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:19.169402+0000","last_undegraded":"2023-10-19T13:00:19.169402+0000","last_fullsized":"2023-10-19T13:00:19.169402+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:19.169402+0000","last_change":"2023-10-19T13:00:19.169402+0000","last_active":"2023-10-19T13:00:19.169402+0000","last_peered":"2023-10-19T13:00:19.169402+0000","last_clean":"2023-10-19T13:00:19.169402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:19.169402+0000","last_undegraded":"2023-10-19T13:00:19.169402+0000","last_fullsized":"2023-10-19T13:00:19.169402+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:19.169402+0000","last_change":"2023-10-19T13:00:19.169402+0000","last_active":"2023-10-19T13:00:19.169402+0000","last_peered":"2023-10-19T13:00:19.169402+0000","last_clean":"2023-10-19T13:00:19.169402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:19.169402+0000","last_undegraded":"2023-10-19T13:00:19.169402+0000","last_fullsized":"2023-10-19T13:00:19.169402+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.14","version":"86'6","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-19T13:00:21.240123+0000","last_change":"2023-10-19T13:00:20.198504+0000","last_active":"2023-10-19T13:00:21.240123+0000","last_peered":"2023-10-19T13:00:21.240123+0000","last_clean":"2023-10-19T13:00:21.240123+0000","last_became_active":"2023-10-19T13:00:20.197655+0000","last_became_peered":"2023-10-19T13:00:20.197655+0000","last_unstale":"2023-10-19T13:00:21.240123+0000","last_undegraded":"2023-10-19T13:00:21.240123+0000","last_fullsized":"2023-10-19T13:00:21.240123+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:25:20.022851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:19.169402+0000","last_change":"2023-10-19T13:00:19.169402+0000","last_active":"2023-10-19T13:00:19.169402+0000","last_peered":"2023-10-19T13:00:19.169402+0000","last_clean":"2023-10-19T13:00:19.169402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:19.169402+0000","last_undegraded":"2023-10-19T13:00:19.169402+0000","last_fullsized":"2023-10-19T13:00:19.169402+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:19.169402+0000","last_change":"2023-10-19T13:00:19.169402+0000","last_active":"2023-10-19T13:00:19.169402+0000","last_peered":"2023-10-19T13:00:19.169402+0000","last_clean":"2023-10-19T13:00:19.169402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:19.169402+0000","last_undegraded":"2023-10-19T13:00:19.169402+0000","last_fullsized":"2023-10-19T13:00:19.169402+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:19.169402+0000","last_change":"2023-10-19T13:00:19.169402+0000","last_active":"2023-10-19T13:00:19.169402+0000","last_peered":"2023-10-19T13:00:19.169402+0000","last_clean":"2023-10-19T13:00:19.169402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:19.169402+0000","last_undegraded":"2023-10-19T13:00:19.169402+0000","last_fullsized":"2023-10-19T13:00:19.169402+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":9,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-19T13:00:21.232936+0000","last_change":"2023-10-19T13:00:21.228024+0000","last_active":"2023-10-19T13:00:21.232936+0000","last_peered":"2023-10-19T13:00:21.232936+0000","last_clean":"2023-10-19T13:00:21.232936+0000","last_became_active":"2023-10-19T13:00:21.227727+0000","last_became_peered":"2023-10-19T13:00:21.227727+0000","last_unstale":"2023-10-19T13:00:21.232936+0000","last_undegraded":"2023-10-19T13:00:21.232936+0000","last_fullsized":"2023-10-19T13:00:21.232936+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:14:55.381409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:19.169402+0000","last_change":"2023-10-19T13:00:19.169402+0000","last_active":"2023-10-19T13:00:19.169402+0000","last_peered":"2023-10-19T13:00:19.169402+0000","last_clean":"2023-10-19T13:00:19.169402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:19.169402+0000","last_undegraded":"2023-10-19T13:00:19.169402+0000","last_fullsized":"2023-10-19T13:00:19.169402+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:19.169402+0000","last_change":"2023-10-19T13:00:19.169402+0000","last_active":"2023-10-19T13:00:19.169402+0000","last_peered":"2023-10-19T13:00:19.169402+0000","last_clean":"2023-10-19T13:00:19.169402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:19.169402+0000","last_undegraded":"2023-10-19T13:00:19.169402+0000","last_fullsized":"2023-10-19T13:00:19.169402+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:19.169402+0000","last_change":"2023-10-19T13:00:19.169402+0000","last_active":"2023-10-19T13:00:19.169402+0000","last_peered":"2023-10-19T13:00:19.169402+0000","last_clean":"2023-10-19T13:00:19.169402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:19.169402+0000","last_undegraded":"2023-10-19T13:00:19.169402+0000","last_fullsized":"2023-10-19T13:00:19.169402+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:19.169402+0000","last_change":"2023-10-19T13:00:19.169402+0000","last_active":"2023-10-19T13:00:19.169402+0000","last_peered":"2023-10-19T13:00:19.169402+0000","last_clean":"2023-10-19T13:00:19.169402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:19.169402+0000","last_undegraded":"2023-10-19T13:00:19.169402+0000","last_fullsized":"2023-10-19T13:00:19.169402+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:19.169402+0000","last_change":"2023-10-19T13:00:19.169402+0000","last_active":"2023-10-19T13:00:19.169402+0000","last_peered":"2023-10-19T13:00:19.169402+0000","last_clean":"2023-10-19T13:00:19.169402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:19.169402+0000","last_undegraded":"2023-10-19T13:00:19.169402+0000","last_fullsized":"2023-10-19T13:00:19.169402+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.220733+0000","last_change":"2023-10-19T13:00:21.217141+0000","last_active":"2023-10-19T13:00:22.220733+0000","last_peered":"2023-10-19T13:00:22.220733+0000","last_clean":"2023-10-19T13:00:22.220733+0000","last_became_active":"2023-10-19T13:00:21.216807+0000","last_became_peered":"2023-10-19T13:00:21.216807+0000","last_unstale":"2023-10-19T13:00:22.220733+0000","last_undegraded":"2023-10-19T13:00:22.220733+0000","last_fullsized":"2023-10-19T13:00:22.220733+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:39:34.098405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"86'2","reported_seq":14,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.220911+0000","last_change":"2023-10-19T13:00:20.195565+0000","last_active":"2023-10-19T13:00:22.220911+0000","last_peered":"2023-10-19T13:00:22.220911+0000","last_clean":"2023-10-19T13:00:22.220911+0000","last_became_active":"2023-10-19T13:00:20.195256+0000","last_became_peered":"2023-10-19T13:00:20.195256+0000","last_unstale":"2023-10-19T13:00:22.220911+0000","last_undegraded":"2023-10-19T13:00:22.220911+0000","last_fullsized":"2023-10-19T13:00:22.220911+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:18:37.647497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:19.169402+0000","last_change":"2023-10-19T13:00:19.169402+0000","last_active":"2023-10-19T13:00:19.169402+0000","last_peered":"2023-10-19T13:00:19.169402+0000","last_clean":"2023-10-19T13:00:19.169402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:19.169402+0000","last_undegraded":"2023-10-19T13:00:19.169402+0000","last_fullsized":"2023-10-19T13:00:19.169402+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.220731+0000","last_change":"2023-10-19T13:00:20.194028+0000","last_active":"2023-10-19T13:00:22.220731+0000","last_peered":"2023-10-19T13:00:22.220731+0000","last_clean":"2023-10-19T13:00:22.220731+0000","last_became_active":"2023-10-19T13:00:20.193758+0000","last_became_peered":"2023-10-19T13:00:20.193758+0000","last_unstale":"2023-10-19T13:00:22.220731+0000","last_undegraded":"2023-10-19T13:00:22.220731+0000","last_fullsized":"2023-10-19T13:00:22.220731+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:41:30.219455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.221024+0000","last_change":"2023-10-19T13:00:21.221247+0000","last_active":"2023-10-19T13:00:22.221024+0000","last_peered":"2023-10-19T13:00:22.221024+0000","last_clean":"2023-10-19T13:00:22.221024+0000","last_became_active":"2023-10-19T13:00:21.220182+0000","last_became_peered":"2023-10-19T13:00:21.220182+0000","last_unstale":"2023-10-19T13:00:22.221024+0000","last_undegraded":"2023-10-19T13:00:22.221024+0000","last_fullsized":"2023-10-19T13:00:22.221024+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:47:08.944926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:19.169402+0000","last_change":"2023-10-19T13:00:19.169402+0000","last_active":"2023-10-19T13:00:19.169402+0000","last_peered":"2023-10-19T13:00:19.169402+0000","last_clean":"2023-10-19T13:00:19.169402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:19.169402+0000","last_undegraded":"2023-10-19T13:00:19.169402+0000","last_fullsized":"2023-10-19T13:00:19.169402+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:23.044679+0000","last_change":"2023-10-19T13:00:21.222511+0000","last_active":"2023-10-19T13:00:23.044679+0000","last_peered":"2023-10-19T13:00:23.044679+0000","last_clean":"2023-10-19T13:00:23.044679+0000","last_became_active":"2023-10-19T13:00:21.222236+0000","last_became_peered":"2023-10-19T13:00:21.222236+0000","last_unstale":"2023-10-19T13:00:23.044679+0000","last_undegraded":"2023-10-19T13:00:23.044679+0000","last_fullsized":"2023-10-19T13:00:23.044679+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:50:46.028354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:19.169402+0000","last_change":"2023-10-19T13:00:19.169402+0000","last_active":"2023-10-19T13:00:19.169402+0000","last_peered":"2023-10-19T13:00:19.169402+0000","last_clean":"2023-10-19T13:00:19.169402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:19.169402+0000","last_undegraded":"2023-10-19T13:00:19.169402+0000","last_fullsized":"2023-10-19T13:00:19.169402+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.221031+0000","last_change":"2023-10-19T13:00:20.188530+0000","last_active":"2023-10-19T13:00:22.221031+0000","last_peered":"2023-10-19T13:00:22.221031+0000","last_clean":"2023-10-19T13:00:22.221031+0000","last_became_active":"2023-10-19T13:00:20.188074+0000","last_became_peered":"2023-10-19T13:00:20.188074+0000","last_unstale":"2023-10-19T13:00:22.221031+0000","last_undegraded":"2023-10-19T13:00:22.221031+0000","last_fullsized":"2023-10-19T13:00:22.221031+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:08:44.211054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-19T13:00:21.216870+0000","last_change":"2023-10-19T13:00:20.198569+0000","last_active":"2023-10-19T13:00:21.216870+0000","last_peered":"2023-10-19T13:00:21.216870+0000","last_clean":"2023-10-19T13:00:21.216870+0000","last_became_active":"2023-10-19T13:00:20.197088+0000","last_became_peered":"2023-10-19T13:00:20.197088+0000","last_unstale":"2023-10-19T13:00:21.216870+0000","last_undegraded":"2023-10-19T13:00:21.216870+0000","last_fullsized":"2023-10-19T13:00:21.216870+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:38:39.386315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:19.169402+0000","last_change":"2023-10-19T13:00:19.169402+0000","last_active":"2023-10-19T13:00:19.169402+0000","last_peered":"2023-10-19T13:00:19.169402+0000","last_clean":"2023-10-19T13:00:19.169402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:19.169402+0000","last_undegraded":"2023-10-19T13:00:19.169402+0000","last_fullsized":"2023-10-19T13:00:19.169402+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:19.169402+0000","last_change":"2023-10-19T13:00:19.169402+0000","last_active":"2023-10-19T13:00:19.169402+0000","last_peered":"2023-10-19T13:00:19.169402+0000","last_clean":"2023-10-19T13:00:19.169402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:19.169402+0000","last_undegraded":"2023-10-19T13:00:19.169402+0000","last_fullsized":"2023-10-19T13:00:19.169402+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":9,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-19T13:00:21.227472+0000","last_change":"2023-10-19T13:00:21.226366+0000","last_active":"2023-10-19T13:00:21.227472+0000","last_peered":"2023-10-19T13:00:21.227472+0000","last_clean":"2023-10-19T13:00:21.227472+0000","last_became_active":"2023-10-19T13:00:21.226075+0000","last_became_peered":"2023-10-19T13:00:21.226075+0000","last_unstale":"2023-10-19T13:00:21.227472+0000","last_undegraded":"2023-10-19T13:00:21.227472+0000","last_fullsized":"2023-10-19T13:00:21.227472+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:33:36.402003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:19.169402+0000","last_change":"2023-10-19T13:00:19.169402+0000","last_active":"2023-10-19T13:00:19.169402+0000","last_peered":"2023-10-19T13:00:19.169402+0000","last_clean":"2023-10-19T13:00:19.169402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:19.169402+0000","last_undegraded":"2023-10-19T13:00:19.169402+0000","last_fullsized":"2023-10-19T13:00:19.169402+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:23.044464+0000","last_change":"2023-10-19T13:00:21.219143+0000","last_active":"2023-10-19T13:00:23.044464+0000","last_peered":"2023-10-19T13:00:23.044464+0000","last_clean":"2023-10-19T13:00:23.044464+0000","last_became_active":"2023-10-19T13:00:21.218834+0000","last_became_peered":"2023-10-19T13:00:21.218834+0000","last_unstale":"2023-10-19T13:00:23.044464+0000","last_undegraded":"2023-10-19T13:00:23.044464+0000","last_fullsized":"2023-10-19T13:00:23.044464+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:29:26.836263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-19T13:00:21.215556+0000","last_change":"2023-10-19T13:00:20.198235+0000","last_active":"2023-10-19T13:00:21.215556+0000","last_peered":"2023-10-19T13:00:21.215556+0000","last_clean":"2023-10-19T13:00:21.215556+0000","last_became_active":"2023-10-19T13:00:20.197920+0000","last_became_peered":"2023-10-19T13:00:20.197920+0000","last_unstale":"2023-10-19T13:00:21.215556+0000","last_undegraded":"2023-10-19T13:00:21.215556+0000","last_fullsized":"2023-10-19T13:00:21.215556+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:34:56.789811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:19.169402+0000","last_change":"2023-10-19T13:00:19.169402+0000","last_active":"2023-10-19T13:00:19.169402+0000","last_peered":"2023-10-19T13:00:19.169402+0000","last_clean":"2023-10-19T13:00:19.169402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:19.169402+0000","last_undegraded":"2023-10-19T13:00:19.169402+0000","last_fullsized":"2023-10-19T13:00:19.169402+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"76'161","reported_seq":381,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-19T13:00:16.745788+0000","last_change":"2023-10-19T12:44:08.636613+0000","last_active":"2023-10-19T13:00:16.745788+0000","last_peered":"2023-10-19T13:00:16.745788+0000","last_clean":"2023-10-19T13:00:16.745788+0000","last_became_active":"2023-10-19T12:44:08.635697+0000","last_became_peered":"2023-10-19T12:44:08.635697+0000","last_unstale":"2023-10-19T13:00:16.745788+0000","last_undegraded":"2023-10-19T13:00:16.745788+0000","last_fullsized":"2023-10-19T13:00:16.745788+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_clean_scrub_stamp":"2023-10-19T12:44:07.399738+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:15:23.685485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":36,"acting":36,"num_store_stats":3},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":2194,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":20,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575045,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59112,"kb_used_data":1564,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312728,"statfs":{"total":96636764160,"available":96576233472,"internally_reserved":0,"allocated":1601536,"data_stored":1092689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58915275},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:59:44 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.57499999999999996,"15min":0.56999999999999995},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.248},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.60099999999999998,"15min":0.58799999999999997},"min":{"1min":0.45400000000000001,"5min":0.33200000000000002,"15min":0.26900000000000002},"max":{"1min":0.78200000000000003,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.50800000000000001}]},{"osd":1,"last update":"Thu Oct 19 12:59:44 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.58199999999999996,"15min":0.57599999999999996},"min":{"1min":0.41299999999999998,"5min":0.40300000000000002,"15min":0.28199999999999997},"max":{"1min":0.84299999999999997,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.623,"5min":0.59999999999999998,"15min":0.58799999999999997},"min":{"1min":0.51700000000000002,"5min":0.43099999999999999,"15min":0.29299999999999998},"max":{"1min":0.82699999999999996,"5min":1.284,"15min":1.284},"last":0.60399999999999998}]},{"osd":2,"last update":"Thu Oct 19 12:59:44 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.64100000000000001,"15min":0.629},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.375},"max":{"1min":0.79300000000000004,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.64600000000000002,"15min":0.63300000000000001},"min":{"1min":0.55300000000000005,"5min":0.41599999999999998,"15min":0.28399999999999997},"max":{"1min":0.81200000000000006,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.47299999999999998}]},{"osd":3,"last update":"Thu Oct 19 12:59:44 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60699999999999998,"15min":0.60099999999999998},"min":{"1min":0.47199999999999998,"5min":0.34999999999999998,"15min":0.34399999999999997},"max":{"1min":0.78900000000000003,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.67000000000000004,"15min":0.65500000000000003},"min":{"1min":0.58199999999999996,"5min":0.45300000000000001,"15min":0.39700000000000002},"max":{"1min":0.92000000000000004,"5min":1.329,"15min":1.329},"last":0.65400000000000003}]},{"osd":4,"last update":"Thu Oct 19 12:59:44 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.46800000000000003,"15min":0.46700000000000003},"min":{"1min":0.36899999999999999,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.72599999999999998,"5min":0.75800000000000001,"15min":0.80500000000000005},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.33600000000000002,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.623,"5min":0.75600000000000001,"15min":0.78900000000000003},"last":0.52400000000000002}]},{"osd":5,"last update":"Thu Oct 19 12:59:44 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.48299999999999998,"15min":0.48499999999999999},"min":{"1min":0.34399999999999997,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.68600000000000005,"5min":0.75800000000000001,"15min":0.90400000000000003},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.52200000000000002,"15min":0.51600000000000001},"min":{"1min":0.38,"5min":0.314,"15min":0.308},"max":{"1min":0.71499999999999997,"5min":0.78700000000000003,"15min":0.92200000000000004},"last":0.624}]},{"osd":6,"last update":"Thu Oct 19 12:59:44 2023","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.54200000000000004,"15min":0.54700000000000004},"min":{"1min":0.379,"5min":0.308,"15min":0.308},"max":{"1min":0.749,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.52100000000000002,"15min":0.52900000000000003},"min":{"1min":0.36899999999999999,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.76900000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.58799999999999997}]}]},{"osd":6,"up_from":13,"seq":55834575045,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32168,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339672,"statfs":{"total":96636764160,"available":96603824128,"internally_reserved":0,"allocated":1007616,"data_stored":502321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:59:28 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.57999999999999996,"15min":0.58699999999999997},"min":{"1min":0.41899999999999998,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.755,"5min":0.79900000000000004,"15min":0.81999999999999995},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.41999999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.56399999999999995}]},{"osd":1,"last update":"Thu Oct 19 12:59:28 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.60299999999999998,"15min":0.59999999999999998},"min":{"1min":0.39900000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.71999999999999997,"5min":0.83299999999999996,"15min":0.93899999999999995},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.59699999999999998,"15min":0.59999999999999998},"min":{"1min":0.42599999999999999,"5min":0.34300000000000003,"15min":0.28399999999999997},"max":{"1min":0.78200000000000003,"5min":0.82099999999999995,"15min":1.075},"last":0.68700000000000006}]},{"osd":2,"last update":"Thu Oct 19 12:59:28 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.42799999999999999,"5min":0.307,"15min":0.26600000000000001},"max":{"1min":0.84699999999999998,"5min":0.85299999999999998,"15min":0.85699999999999998},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.624,"5min":0.61299999999999999,"15min":0.60799999999999998},"min":{"1min":0.48499999999999999,"5min":0.35999999999999999,"15min":0.28399999999999997},"max":{"1min":0.82199999999999995,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.38900000000000001}]},{"osd":3,"last update":"Thu Oct 19 12:59:28 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.625,"15min":0.623},"min":{"1min":0.46400000000000002,"5min":0.38500000000000001,"15min":0.32600000000000001},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.61799999999999999,"15min":0.628},"min":{"1min":0.44900000000000001,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":1.0900000000000001},"last":0.49199999999999999}]},{"osd":4,"last update":"Thu Oct 19 12:59:28 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.46100000000000002,"15min":0.46999999999999997},"min":{"1min":0.32900000000000001,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.64400000000000002,"5min":0.77900000000000003,"15min":1.02},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.47999999999999998,"15min":0.48499999999999999},"min":{"1min":0.33200000000000002,"5min":0.32800000000000001,"15min":0.247},"max":{"1min":0.69099999999999995,"5min":0.752,"15min":0.80000000000000004},"last":0.73799999999999999}]},{"osd":5,"last update":"Thu Oct 19 12:59:28 2023","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.46800000000000003,"15min":0.47699999999999998},"min":{"1min":0.32900000000000001,"5min":0.30199999999999999,"15min":0.26600000000000001},"max":{"1min":0.61899999999999999,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.442},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.50800000000000001,"15min":0.51900000000000002},"min":{"1min":0.35699999999999998,"5min":0.32100000000000001,"15min":0.312},"max":{"1min":0.75600000000000001,"5min":0.85699999999999998,"15min":0.98799999999999999},"last":0.63300000000000001}]},{"osd":7,"last update":"Thu Oct 19 12:59:28 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.51800000000000002,"15min":0.52300000000000002},"min":{"1min":0.32500000000000001,"5min":0.29299999999999998,"15min":0.28199999999999997},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":1.036},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53100000000000003,"15min":0.54100000000000004},"min":{"1min":0.36499999999999999,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":1.1299999999999999},"last":0.65100000000000002}]}]},{"osd":5,"up_from":13,"seq":55834575045,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32176,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339664,"statfs":{"total":96636764160,"available":96603815936,"internally_reserved":0,"allocated":1007616,"data_stored":502321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:59:29 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.58799999999999997,"15min":0.58899999999999997},"min":{"1min":0.44700000000000001,"5min":0.28999999999999998,"15min":0.223},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.88800000000000001},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.63,"5min":0.58999999999999997,"15min":0.59599999999999997},"min":{"1min":0.42499999999999999,"5min":0.36099999999999999,"15min":0.27800000000000002},"max":{"1min":0.81399999999999995,"5min":0.84299999999999997,"15min":0.874},"last":0.82999999999999996}]},{"osd":1,"last update":"Thu Oct 19 12:59:29 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.57899999999999996,"15min":0.59399999999999997},"min":{"1min":0.46899999999999997,"5min":0.34599999999999997,"15min":0.32800000000000001},"max":{"1min":0.77500000000000002,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.59999999999999998,"15min":0.60599999999999998},"min":{"1min":0.49199999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83499999999999996},"last":0.70099999999999996}]},{"osd":2,"last update":"Thu Oct 19 12:59:29 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.57299999999999995,"15min":0.58299999999999996},"min":{"1min":0.48899999999999999,"5min":0.34100000000000003,"15min":0.307},"max":{"1min":0.73999999999999999,"5min":0.78600000000000003,"15min":0.80700000000000005},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.60499999999999998,"15min":0.61899999999999999},"min":{"1min":0.50900000000000001,"5min":0.38400000000000001,"15min":0.33700000000000002},"max":{"1min":0.80900000000000005,"5min":0.88100000000000001,"15min":0.93600000000000005},"last":0.79300000000000004}]},{"osd":3,"last update":"Thu Oct 19 12:59:29 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.59299999999999997,"15min":0.60199999999999998},"min":{"1min":0.48499999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.751,"5min":0.79900000000000004,"15min":0.874},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.625,"15min":0.63400000000000001},"min":{"1min":0.50900000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.89500000000000002},"last":0.71799999999999997}]},{"osd":4,"last update":"Thu Oct 19 12:59:29 2023","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.36099999999999999,"5min":0.29099999999999998,"15min":0.28899999999999998},"max":{"1min":0.61399999999999999,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.46700000000000003,"15min":0.47199999999999998},"min":{"1min":0.35899999999999999,"5min":0.251,"15min":0.23899999999999999},"max":{"1min":0.58299999999999996,"5min":0.79400000000000004,"15min":0.80500000000000005},"last":0.36199999999999999}]},{"osd":6,"last update":"Thu Oct 19 12:59:29 2023","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46500000000000002,"15min":0.47299999999999998},"min":{"1min":0.34499999999999997,"5min":0.27200000000000002,"15min":0.25900000000000001},"max":{"1min":0.56299999999999994,"5min":0.74399999999999999,"15min":0.79300000000000004},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.497,"5min":0.50900000000000001,"15min":0.502},"min":{"1min":0.373,"5min":0.34200000000000003,"15min":0.30399999999999999},"max":{"1min":0.64800000000000002,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.60499999999999998}]},{"osd":7,"last update":"Thu Oct 19 12:59:29 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.48599999999999999,"15min":0.49299999999999999},"min":{"1min":0.39400000000000002,"5min":0.32000000000000001,"15min":0.315},"max":{"1min":0.64200000000000002,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.624},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.53900000000000003,"15min":0.53700000000000003},"min":{"1min":0.46100000000000002,"5min":0.311,"15min":0.27500000000000002},"max":{"1min":0.66900000000000004,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.68100000000000005}]}]},{"osd":4,"up_from":13,"seq":55834575045,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32152,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339688,"statfs":{"total":96636764160,"available":96603840512,"internally_reserved":0,"allocated":995328,"data_stored":496916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56100000000000005,"15min":0.55900000000000005},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.28899999999999998},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.55800000000000005,"15min":0.55500000000000005},"min":{"1min":0.45400000000000001,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.77100000000000002,"5min":0.77800000000000002,"15min":0.78200000000000003},"last":0.39400000000000002}]},{"osd":1,"last update":"Thu Oct 19 12:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.60399999999999998,"15min":0.60799999999999998},"min":{"1min":0.33000000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.78600000000000003,"5min":0.80600000000000005,"15min":0.86199999999999999},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61899999999999999,"15min":0.61599999999999999},"min":{"1min":0.40899999999999997,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.77300000000000002,"5min":0.79900000000000004,"15min":0.873},"last":0.54700000000000004}]},{"osd":2,"last update":"Thu Oct 19 12:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.61199999999999999,"15min":0.61299999999999999},"min":{"1min":0.375,"5min":0.32200000000000001,"15min":0.254},"max":{"1min":0.72199999999999998,"5min":0.85299999999999998,"15min":0.86599999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61699999999999999,"15min":0.61599999999999999},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.34499999999999997},"max":{"1min":0.78500000000000003,"5min":0.82999999999999996,"15min":0.85999999999999999},"last":0.42099999999999999}]},{"osd":3,"last update":"Thu Oct 19 12:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.59099999999999997,"15min":0.58399999999999996},"min":{"1min":0.41999999999999998,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.73499999999999999,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.64600000000000002,"15min":0.63700000000000001},"min":{"1min":0.52500000000000002,"5min":0.42599999999999999,"15min":0.37},"max":{"1min":0.73899999999999999,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.624}]},{"osd":5,"last update":"Thu Oct 19 12:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.45600000000000002,"15min":0.45200000000000001},"min":{"1min":0.309,"5min":0.29599999999999999,"15min":0.249},"max":{"1min":0.63500000000000001,"5min":0.79600000000000004,"15min":0.81299999999999994},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.47499999999999998,"15min":0.46200000000000002},"min":{"1min":0.34699999999999998,"5min":0.26400000000000001,"15min":0.23499999999999999},"max":{"1min":0.623,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.505}]},{"osd":6,"last update":"Thu Oct 19 12:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.45900000000000002,"15min":0.45600000000000002},"min":{"1min":0.31,"5min":0.29499999999999998,"15min":0.26400000000000001},"max":{"1min":0.52800000000000002,"5min":0.72599999999999998,"15min":0.78600000000000003},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.48599999999999999,"15min":0.497},"min":{"1min":0.34300000000000003,"5min":0.32100000000000001,"15min":0.27100000000000002},"max":{"1min":0.66700000000000004,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.63600000000000001}]},{"osd":7,"last update":"Thu Oct 19 12:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.501,"15min":0.503},"min":{"1min":0.33700000000000002,"5min":0.307,"15min":0.27500000000000002},"max":{"1min":0.68300000000000005,"5min":0.79100000000000004,"15min":0.80200000000000005},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.503,"5min":0.54200000000000004,"15min":0.54800000000000004},"min":{"1min":0.39100000000000001,"5min":0.35799999999999998,"15min":0.30199999999999999},"max":{"1min":0.69399999999999995,"5min":0.80500000000000005,"15min":0.83499999999999996},"last":0.67200000000000004}]}]},{"osd":3,"up_from":13,"seq":55834575044,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":36332,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335508,"statfs":{"total":96636764160,"available":96599560192,"internally_reserved":0,"allocated":1007616,"data_stored":502321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:59:24 2023","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.41799999999999998,"15min":0.39900000000000002},"min":{"1min":0.27100000000000002,"5min":0.23599999999999999,"15min":0.20899999999999999},"max":{"1min":0.69099999999999995,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.252},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.41199999999999998,"15min":0.41299999999999998},"min":{"1min":0.316,"5min":0.223,"15min":0.223},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.28699999999999998}]},{"osd":1,"last update":"Thu Oct 19 12:59:24 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.40600000000000003,"15min":0.40699999999999997},"min":{"1min":0.29099999999999998,"5min":0.26100000000000001,"15min":0.22500000000000001},"max":{"1min":0.63500000000000001,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.433},{"interface":"front","average":{"1min":0.505,"5min":0.48299999999999998,"15min":0.45700000000000002},"min":{"1min":0.27600000000000002,"5min":0.24099999999999999,"15min":0.215},"max":{"1min":0.66700000000000004,"5min":2.9399999999999999,"15min":2.9399999999999999},"last":0.32700000000000001}]},{"osd":2,"last update":"Thu Oct 19 12:59:24 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.46899999999999997,"15min":0.46500000000000002},"min":{"1min":0.33800000000000002,"5min":0.314,"15min":0.26100000000000001},"max":{"1min":0.69399999999999995,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.48099999999999998,"15min":0.48499999999999999},"min":{"1min":0.35099999999999998,"5min":0.29999999999999999,"15min":0.247},"max":{"1min":0.754,"5min":0.754,"15min":0.96499999999999997},"last":0.39000000000000001}]},{"osd":4,"last update":"Thu Oct 19 12:59:24 2023","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.65700000000000003,"15min":0.66400000000000003},"min":{"1min":0.50700000000000001,"5min":0.42299999999999999,"15min":0.40300000000000002},"max":{"1min":0.90200000000000002,"5min":0.90500000000000003,"15min":1.0069999999999999},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.69499999999999995,"15min":0.69199999999999995},"min":{"1min":0.59499999999999997,"5min":0.46899999999999997,"15min":0.42099999999999999},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.99399999999999999},"last":0.67900000000000005}]},{"osd":5,"last update":"Thu Oct 19 12:59:24 2023","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.67600000000000005,"15min":0.65900000000000003},"min":{"1min":0.56000000000000005,"5min":0.45500000000000002,"15min":0.40500000000000003},"max":{"1min":0.93600000000000005,"5min":2.9990000000000001,"15min":2.9990000000000001},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.52100000000000002,"5min":0.40899999999999997,"15min":0.35899999999999999},"max":{"1min":0.81899999999999995,"5min":1.774,"15min":1.774},"last":0.59699999999999998}]},{"osd":6,"last update":"Thu Oct 19 12:59:24 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.66100000000000003,"15min":0.65400000000000003},"min":{"1min":0.52300000000000002,"5min":0.433,"15min":0.41199999999999998},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.96799999999999997},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.67500000000000004,"15min":0.67400000000000004},"min":{"1min":0.60299999999999998,"5min":0.377,"15min":0.35899999999999999},"max":{"1min":0.93700000000000006,"5min":1.167,"15min":1.167},"last":0.64500000000000002}]},{"osd":7,"last update":"Thu Oct 19 12:59:24 2023","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.65200000000000002,"15min":0.66500000000000004},"min":{"1min":0.45500000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.94299999999999995,"5min":1.042,"15min":1.042},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.69299999999999995,"15min":0.67900000000000005},"min":{"1min":0.54200000000000004,"5min":0.432,"15min":0.432},"max":{"1min":0.89200000000000002,"5min":2.8149999999999999,"15min":2.8149999999999999},"last":0.61299999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575047,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54416,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317424,"statfs":{"total":96636764160,"available":96581042176,"internally_reserved":0,"allocated":1044480,"data_stored":522224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:59:33 2023","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.377,"15min":0.38100000000000001},"min":{"1min":0.27500000000000002,"5min":0.20899999999999999,"15min":0.187},"max":{"1min":0.54000000000000004,"5min":0.67700000000000005,"15min":0.83999999999999997},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.436,"5min":0.40999999999999998,"15min":0.40600000000000003},"min":{"1min":0.32900000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.58299999999999996,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.56000000000000005}]},{"osd":1,"last update":"Thu Oct 19 12:59:33 2023","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.39400000000000002,"15min":0.39800000000000002},"min":{"1min":0.318,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.80400000000000005},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.41999999999999998,"15min":0.42999999999999999},"min":{"1min":0.38200000000000001,"5min":0.22,"15min":0.22},"max":{"1min":0.59499999999999997,"5min":0.60499999999999998,"15min":0.78600000000000003},"last":0.61299999999999999}]},{"osd":3,"last update":"Thu Oct 19 12:59:33 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.46300000000000002,"15min":0.45700000000000002},"min":{"1min":0.33400000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.498},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.48099999999999998,"15min":0.48499999999999999},"min":{"1min":0.40300000000000002,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.72799999999999998,"5min":0.77800000000000002,"15min":0.87},"last":0.57499999999999996}]},{"osd":4,"last update":"Thu Oct 19 12:59:33 2023","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.63700000000000001,"15min":0.64200000000000002},"min":{"1min":0.497,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.89500000000000002,"5min":0.93999999999999995,"15min":1.0569999999999999},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.65400000000000003,"15min":0.65700000000000003},"min":{"1min":0.54300000000000004,"5min":0.35099999999999998,"15min":0.32000000000000001},"max":{"1min":0.83399999999999996,"5min":0.92300000000000004,"15min":1.0780000000000001},"last":0.70999999999999996}]},{"osd":5,"last update":"Thu Oct 19 12:59:33 2023","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.67400000000000004,"15min":0.66200000000000003},"min":{"1min":0.58299999999999996,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.84299999999999997,"5min":0.96499999999999997,"15min":0.99399999999999999},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.65200000000000002,"15min":0.65800000000000003},"min":{"1min":0.56899999999999995,"5min":0.375,"15min":0.375},"max":{"1min":0.872,"5min":0.94599999999999995,"15min":0.97099999999999997},"last":0.64000000000000001}]},{"osd":6,"last update":"Thu Oct 19 12:59:33 2023","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.65000000000000002,"15min":0.66700000000000004},"min":{"1min":0.47199999999999998,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.82399999999999995,"5min":1.008,"15min":1.008},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.67800000000000005,"15min":0.67400000000000004},"min":{"1min":0.52000000000000002,"5min":0.371,"15min":0.371},"max":{"1min":0.88100000000000001,"5min":0.995,"15min":0.995},"last":0.63}]},{"osd":7,"last update":"Thu Oct 19 12:59:33 2023","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.67500000000000004,"15min":0.68500000000000005},"min":{"1min":0.59699999999999998,"5min":0.39300000000000002,"15min":0.36399999999999999},"max":{"1min":0.86099999999999999,"5min":1.0149999999999999,"15min":1.111},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.68899999999999995,"15min":0.68300000000000005},"min":{"1min":0.53000000000000003,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.89800000000000002,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.65300000000000002}]}]},{"osd":1,"up_from":13,"seq":55834575046,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32236,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339604,"statfs":{"total":96636764160,"available":96603754496,"internally_reserved":0,"allocated":1073152,"data_stored":524600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:59:33 2023","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.39200000000000002,"15min":0.38500000000000001},"min":{"1min":0.23999999999999999,"5min":0.21299999999999999,"15min":0.187},"max":{"1min":0.55800000000000005,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.442,"15min":0.42799999999999999},"min":{"1min":0.29499999999999998,"5min":0.23899999999999999,"15min":0.21199999999999999},"max":{"1min":0.58799999999999997,"5min":2.5800000000000001,"15min":2.5800000000000001},"last":0.52700000000000002}]},{"osd":2,"last update":"Thu Oct 19 12:59:33 2023","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.42499999999999999,"15min":0.42099999999999999},"min":{"1min":0.309,"5min":0.21299999999999999,"15min":0.20100000000000001},"max":{"1min":0.53300000000000003,"5min":0.60499999999999998,"15min":0.69999999999999996},"last":0.502},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.47099999999999997,"15min":0.45300000000000001},"min":{"1min":0.34999999999999998,"5min":0.313,"15min":0.23699999999999999},"max":{"1min":0.625,"5min":2.6230000000000002,"15min":2.6230000000000002},"last":0.57999999999999996}]},{"osd":3,"last update":"Thu Oct 19 12:59:33 2023","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.51600000000000001,"15min":0.48599999999999999},"min":{"1min":0.38900000000000001,"5min":0.307,"15min":0.26200000000000001},"max":{"1min":0.621,"5min":3.9740000000000002,"15min":3.9740000000000002},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.43099999999999999,"15min":0.42299999999999999},"min":{"1min":0.35999999999999999,"5min":0.27700000000000002,"15min":0.27000000000000002},"max":{"1min":0.59299999999999997,"5min":0.63200000000000001,"15min":0.80500000000000005},"last":0.51400000000000001}]},{"osd":4,"last update":"Thu Oct 19 12:59:33 2023","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.65600000000000003,"15min":0.65200000000000002},"min":{"1min":0.501,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.85699999999999998,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.66600000000000004,"15min":0.64800000000000002},"min":{"1min":0.47399999999999998,"5min":0.42599999999999999,"15min":0.314},"max":{"1min":0.85399999999999998,"5min":1.0860000000000001,"15min":1.1910000000000001},"last":0.76300000000000001}]},{"osd":5,"last update":"Thu Oct 19 12:59:33 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.65700000000000003,"15min":0.65300000000000002},"min":{"1min":0.499,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.83599999999999997,"5min":1.1200000000000001,"15min":1.1200000000000001},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.69599999999999995,"15min":0.67300000000000004},"min":{"1min":0.54300000000000004,"5min":0.42299999999999999,"15min":0.38700000000000001},"max":{"1min":0.81799999999999995,"5min":2.7120000000000002,"15min":2.7120000000000002},"last":0.77900000000000003}]},{"osd":6,"last update":"Thu Oct 19 12:59:33 2023","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.65800000000000003,"15min":0.65700000000000003},"min":{"1min":0.53800000000000003,"5min":0.36799999999999999,"15min":0.36399999999999999},"max":{"1min":0.91300000000000003,"5min":1.0069999999999999,"15min":1.077},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67400000000000004,"15min":0.67800000000000005},"min":{"1min":0.55800000000000005,"5min":0.376,"15min":0.376},"max":{"1min":0.77500000000000002,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.72099999999999997}]},{"osd":7,"last update":"Thu Oct 19 12:59:33 2023","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.69799999999999995,"15min":0.67900000000000005},"min":{"1min":0.54300000000000004,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.84999999999999998,"5min":2.786,"15min":2.786},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.68200000000000005,"15min":0.67900000000000005},"min":{"1min":0.47499999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":1.0489999999999999},"last":0.79500000000000004}]}]},{"osd":0,"up_from":13,"seq":55834575047,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36952,"kb_used_data":1608,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334888,"statfs":{"total":96636764160,"available":96598925312,"internally_reserved":0,"allocated":1646592,"data_stored":1118621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Oct 19 12:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.40300000000000002,"15min":0.39100000000000001},"min":{"1min":0.26500000000000001,"5min":0.182,"15min":0.182},"max":{"1min":0.58899999999999997,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.434,"15min":0.432},"min":{"1min":0.34599999999999997,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.746},"last":0.38200000000000001}]},{"osd":2,"last update":"Thu Oct 19 12:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.46700000000000003,"15min":0.45500000000000002},"min":{"1min":0.33200000000000002,"5min":0.29499999999999998,"15min":0.25800000000000001},"max":{"1min":0.58599999999999997,"5min":0.752,"15min":0.76300000000000001},"last":0.309},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.437,"15min":0.434},"min":{"1min":0.26000000000000001,"5min":0.216,"15min":0.20699999999999999},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.76800000000000002},"last":0.432}]},{"osd":3,"last update":"Thu Oct 19 12:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.42199999999999999,"15min":0.40899999999999997},"min":{"1min":0.34000000000000002,"5min":0.23400000000000001,"15min":0.22800000000000001},"max":{"1min":0.53500000000000003,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.505,"5min":0.504,"15min":0.49099999999999999},"min":{"1min":0.33600000000000002,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.66300000000000003,"5min":1.4990000000000001,"15min":1.4990000000000001},"last":0.34799999999999998}]},{"osd":4,"last update":"Thu Oct 19 12:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.63100000000000001,"15min":0.63600000000000001},"min":{"1min":0.46600000000000003,"5min":0.40000000000000002,"15min":0.34999999999999998},"max":{"1min":0.749,"5min":1.361,"15min":1.361},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.65700000000000003,"15min":0.65000000000000002},"min":{"1min":0.52900000000000003,"5min":0.41899999999999998,"15min":0.35999999999999999},"max":{"1min":0.871,"5min":0.98999999999999999,"15min":0.998},"last":0.57699999999999996}]},{"osd":5,"last update":"Thu Oct 19 12:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.68999999999999995,"15min":0.68000000000000005},"min":{"1min":0.50700000000000001,"5min":0.378,"15min":0.378},"max":{"1min":0.94599999999999995,"5min":1.325,"15min":1.325},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.69299999999999995,"15min":0.68600000000000005},"min":{"1min":0.45900000000000002,"5min":0.36299999999999999,"15min":0.318},"max":{"1min":1.02,"5min":1.024,"15min":1.056},"last":0.625}]},{"osd":6,"last update":"Thu Oct 19 12:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.67200000000000004,"15min":0.66400000000000003},"min":{"1min":0.59999999999999998,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.873,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.67600000000000005,"15min":0.66600000000000004},"min":{"1min":0.57199999999999995,"5min":0.41999999999999998,"15min":0.38900000000000001},"max":{"1min":0.92000000000000004,"5min":0.995,"15min":1.109},"last":0.65600000000000003}]},{"osd":7,"last update":"Thu Oct 19 12:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.68700000000000006,"15min":0.68400000000000005},"min":{"1min":0.53500000000000003,"5min":0.41199999999999998,"15min":0.41099999999999998},"max":{"1min":0.89900000000000002,"5min":0.98799999999999999,"15min":1.0089999999999999},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.67300000000000004,"15min":0.66300000000000003},"min":{"1min":0.58799999999999997,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.90400000000000003,"5min":1.389,"15min":1.389},"last":0.67100000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-19T13:00:24.578 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-19T13:00:24.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.719+0000 7fd727461640 1 -- 172.21.15.57:0/3808195313 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd7200afa20 msgr2=0x7fd7200705a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:24.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.719+0000 7fd727461640 1 --2- 172.21.15.57:0/3808195313 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd7200afa20 0x7fd7200705a0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fd714009830 tx=0x7fd71402bd60 comp rx=0 tx=0).stop 2023-10-19T13:00:24.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.719+0000 7fd727461640 1 -- 172.21.15.57:0/3808195313 shutdown_connections 2023-10-19T13:00:24.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.719+0000 7fd727461640 1 --2- 172.21.15.57:0/3808195313 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd7200afa20 0x7fd7200705a0 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:24.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.719+0000 7fd727461640 1 --2- 172.21.15.57:0/3808195313 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd7200cbd20 0x7fd720158300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:24.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.719+0000 7fd727461640 1 -- 172.21.15.57:0/3808195313 >> 172.21.15.57:0/3808195313 conn(0x7fd720069930 msgr2=0x7fd720069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:24.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.719+0000 7fd727461640 1 -- 172.21.15.57:0/3808195313 shutdown_connections 2023-10-19T13:00:24.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.719+0000 7fd727461640 1 -- 172.21.15.57:0/3808195313 wait complete. 2023-10-19T13:00:24.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.720+0000 7fd727461640 1 Processor -- start 2023-10-19T13:00:24.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.720+0000 7fd727461640 1 -- start start 2023-10-19T13:00:24.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.721+0000 7fd727461640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd7200afa20 0x7fd72013c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:24.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.721+0000 7fd727461640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd7200b0530 0x7fd7200ca870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:24.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.721+0000 7fd727461640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd7200cbd20 0x7fd7200cadb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:24.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.721+0000 7fd727461640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd72013c770 con 0x7fd7200b0530 2023-10-19T13:00:24.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.721+0000 7fd727461640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd72014c6c0 con 0x7fd7200cbd20 2023-10-19T13:00:24.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.721+0000 7fd727461640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd720153500 con 0x7fd7200afa20 2023-10-19T13:00:24.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.721+0000 7fd7249fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd7200b0530 0x7fd7200ca870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:24.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.721+0000 7fd7249fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd7200b0530 0x7fd7200ca870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:49894/0 (socket says 172.21.15.57:49894) 2023-10-19T13:00:24.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.721+0000 7fd7249fc640 1 -- 172.21.15.57:0/3089282187 learned_addr learned my addr 172.21.15.57:0/3089282187 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:24.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.721+0000 7fd7259fe640 1 --2- 172.21.15.57:0/3089282187 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd7200cbd20 0x7fd7200cadb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:24.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.721+0000 7fd7251fd640 1 --2- 172.21.15.57:0/3089282187 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd7200afa20 0x7fd72013c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:24.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.721+0000 7fd7249fc640 1 -- 172.21.15.57:0/3089282187 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd7200afa20 msgr2=0x7fd72013c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:24.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.721+0000 7fd7249fc640 1 --2- 172.21.15.57:0/3089282187 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd7200afa20 0x7fd72013c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:24.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.721+0000 7fd7249fc640 1 -- 172.21.15.57:0/3089282187 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd7200cbd20 msgr2=0x7fd7200cadb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:24.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.721+0000 7fd7249fc640 1 --2- 172.21.15.57:0/3089282187 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd7200cbd20 0x7fd7200cadb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:24.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.721+0000 7fd7249fc640 1 -- 172.21.15.57:0/3089282187 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd714009490 con 0x7fd7200b0530 2023-10-19T13:00:24.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.722+0000 7fd7249fc640 1 --2- 172.21.15.57:0/3089282187 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd7200b0530 0x7fd7200ca870 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7fd714000c00 tx=0x7fd7140038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:24.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.722+0000 7fd71dffb640 1 -- 172.21.15.57:0/3089282187 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd714003ea0 con 0x7fd7200b0530 2023-10-19T13:00:24.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.722+0000 7fd71dffb640 1 -- 172.21.15.57:0/3089282187 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fd714034530 con 0x7fd7200b0530 2023-10-19T13:00:24.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.722+0000 7fd71dffb640 1 -- 172.21.15.57:0/3089282187 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd714034940 con 0x7fd7200b0530 2023-10-19T13:00:24.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.722+0000 7fd727461640 1 -- 172.21.15.57:0/3089282187 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7201537e0 con 0x7fd7200b0530 2023-10-19T13:00:24.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.722+0000 7fd727461640 1 -- 172.21.15.57:0/3089282187 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd720153da0 con 0x7fd7200b0530 2023-10-19T13:00:24.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.723+0000 7fd727461640 1 -- 172.21.15.57:0/3089282187 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7200690e0 con 0x7fd7200b0530 2023-10-19T13:00:24.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.723+0000 7fd71dffb640 1 -- 172.21.15.57:0/3089282187 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 32) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd714034aa0 con 0x7fd7200b0530 2023-10-19T13:00:24.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.724+0000 7fd71dffb640 1 --2- 172.21.15.57:0/3089282187 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7fd70c06af90 0x7fd70c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:24.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.724+0000 7fd71dffb640 1 -- 172.21.15.57:0/3089282187 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fd7140ac260 con 0x7fd7200b0530 2023-10-19T13:00:24.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.724+0000 7fd7251fd640 1 --2- 172.21.15.57:0/3089282187 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7fd70c06af90 0x7fd70c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:24.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.725+0000 7fd7251fd640 1 --2- 172.21.15.57:0/3089282187 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7fd70c06af90 0x7fd70c06d450 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd718005d70 tx=0x7fd718005d00 comp rx=0 tx=0).ready entity=mgr.6256 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:24.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.728+0000 7fd71dffb640 1 -- 172.21.15.57:0/3089282187 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd7140770a0 con 0x7fd7200b0530 2023-10-19T13:00:24.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.899+0000 7fd727461640 1 -- 172.21.15.57:0/3089282187 --> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd72006b910 con 0x7fd70c06af90 2023-10-19T13:00:24.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.906+0000 7fd71dffb640 1 -- 172.21.15.57:0/3089282187 <== mgr.6256 v2:172.21.15.179:6832/3815793798 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+279504 (secure 0 0 0) 0x7fd71800c740 con 0x7fd70c06af90 2023-10-19T13:00:24.912 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:00:24.913 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2023-10-19T13:00:24.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.909+0000 7fd727461640 1 -- 172.21.15.57:0/3089282187 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7fd70c06af90 msgr2=0x7fd70c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:24.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.909+0000 7fd727461640 1 --2- 172.21.15.57:0/3089282187 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7fd70c06af90 0x7fd70c06d450 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd718005d70 tx=0x7fd718005d00 comp rx=0 tx=0).stop 2023-10-19T13:00:24.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.909+0000 7fd727461640 1 -- 172.21.15.57:0/3089282187 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd7200b0530 msgr2=0x7fd7200ca870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:24.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.909+0000 7fd727461640 1 --2- 172.21.15.57:0/3089282187 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd7200b0530 0x7fd7200ca870 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7fd714000c00 tx=0x7fd7140038a0 comp rx=0 tx=0).stop 2023-10-19T13:00:24.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.909+0000 7fd727461640 1 -- 172.21.15.57:0/3089282187 shutdown_connections 2023-10-19T13:00:24.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.909+0000 7fd727461640 1 --2- 172.21.15.57:0/3089282187 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7fd70c06af90 0x7fd70c06d450 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:24.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.909+0000 7fd727461640 1 --2- 172.21.15.57:0/3089282187 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd7200cbd20 0x7fd7200cadb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:24.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.909+0000 7fd727461640 1 --2- 172.21.15.57:0/3089282187 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd7200b0530 0x7fd7200ca870 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:24.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.909+0000 7fd727461640 1 --2- 172.21.15.57:0/3089282187 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd7200afa20 0x7fd72013c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:24.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.909+0000 7fd727461640 1 -- 172.21.15.57:0/3089282187 >> 172.21.15.57:0/3089282187 conn(0x7fd720069930 msgr2=0x7fd72015a510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:24.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.909+0000 7fd727461640 1 -- 172.21.15.57:0/3089282187 shutdown_connections 2023-10-19T13:00:24.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:24.909+0000 7fd727461640 1 -- 172.21.15.57:0/3089282187 wait complete. 2023-10-19T13:00:24.923 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2023-10-19T13:00:23.049618+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592562,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":256,"num_write_kb":2420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":178,"ondisk_log_size":178,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":43,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":4,"kb":754974720,"kb_used":315544,"kb_used_data":9164,"kb_used_omap":12,"kb_used_meta":306227,"kb_avail":754659176,"statfs":{"total":773094113280,"available":772770996224,"internally_reserved":0,"allocated":9383936,"data_stored":5262013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":313577017},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2194,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000992"},"pg_stats":[{"pgid":"10.3f","version":"0'0","reported_seq":9,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-19T13:00:21.237877+0000","last_change":"2023-10-19T13:00:21.228303+0000","last_active":"2023-10-19T13:00:21.237877+0000","last_peered":"2023-10-19T13:00:21.237877+0000","last_clean":"2023-10-19T13:00:21.237877+0000","last_became_active":"2023-10-19T13:00:21.227974+0000","last_became_peered":"2023-10-19T13:00:21.227974+0000","last_unstale":"2023-10-19T13:00:21.237877+0000","last_undegraded":"2023-10-19T13:00:21.237877+0000","last_fullsized":"2023-10-19T13:00:21.237877+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:37:52.369486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3e","version":"0'0","reported_seq":9,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-19T13:00:21.233058+0000","last_change":"2023-10-19T13:00:21.227759+0000","last_active":"2023-10-19T13:00:21.233058+0000","last_peered":"2023-10-19T13:00:21.233058+0000","last_clean":"2023-10-19T13:00:21.233058+0000","last_became_active":"2023-10-19T13:00:21.227466+0000","last_became_peered":"2023-10-19T13:00:21.227466+0000","last_unstale":"2023-10-19T13:00:21.233058+0000","last_undegraded":"2023-10-19T13:00:21.233058+0000","last_fullsized":"2023-10-19T13:00:21.233058+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:55:42.567075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.3a","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.221076+0000","last_change":"2023-10-19T13:00:21.219325+0000","last_active":"2023-10-19T13:00:22.221076+0000","last_peered":"2023-10-19T13:00:22.221076+0000","last_clean":"2023-10-19T13:00:22.221076+0000","last_became_active":"2023-10-19T13:00:21.218458+0000","last_became_peered":"2023-10-19T13:00:21.218458+0000","last_unstale":"2023-10-19T13:00:22.221076+0000","last_undegraded":"2023-10-19T13:00:22.221076+0000","last_fullsized":"2023-10-19T13:00:22.221076+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:31:39.478057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.39","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:23.044542+0000","last_change":"2023-10-19T13:00:21.220590+0000","last_active":"2023-10-19T13:00:23.044542+0000","last_peered":"2023-10-19T13:00:23.044542+0000","last_clean":"2023-10-19T13:00:23.044542+0000","last_became_active":"2023-10-19T13:00:21.220337+0000","last_became_peered":"2023-10-19T13:00:21.220337+0000","last_unstale":"2023-10-19T13:00:23.044542+0000","last_undegraded":"2023-10-19T13:00:23.044542+0000","last_fullsized":"2023-10-19T13:00:23.044542+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:25:25.779991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.30","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:23.044519+0000","last_change":"2023-10-19T13:00:21.219839+0000","last_active":"2023-10-19T13:00:23.044519+0000","last_peered":"2023-10-19T13:00:23.044519+0000","last_clean":"2023-10-19T13:00:23.044519+0000","last_became_active":"2023-10-19T13:00:21.219524+0000","last_became_peered":"2023-10-19T13:00:21.219524+0000","last_unstale":"2023-10-19T13:00:23.044519+0000","last_undegraded":"2023-10-19T13:00:23.044519+0000","last_fullsized":"2023-10-19T13:00:23.044519+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:16:36.020089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2f","version":"0'0","reported_seq":9,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-19T13:00:21.237892+0000","last_change":"2023-10-19T13:00:21.228480+0000","last_active":"2023-10-19T13:00:21.237892+0000","last_peered":"2023-10-19T13:00:21.237892+0000","last_clean":"2023-10-19T13:00:21.237892+0000","last_became_active":"2023-10-19T13:00:21.227687+0000","last_became_peered":"2023-10-19T13:00:21.227687+0000","last_unstale":"2023-10-19T13:00:21.237892+0000","last_undegraded":"2023-10-19T13:00:21.237892+0000","last_fullsized":"2023-10-19T13:00:21.237892+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:58:41.166338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.24","version":"0'0","reported_seq":9,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-19T13:00:21.233957+0000","last_change":"2023-10-19T13:00:21.227180+0000","last_active":"2023-10-19T13:00:21.233957+0000","last_peered":"2023-10-19T13:00:21.233957+0000","last_clean":"2023-10-19T13:00:21.233957+0000","last_became_active":"2023-10-19T13:00:21.226763+0000","last_became_peered":"2023-10-19T13:00:21.226763+0000","last_unstale":"2023-10-19T13:00:21.233957+0000","last_undegraded":"2023-10-19T13:00:21.233957+0000","last_fullsized":"2023-10-19T13:00:21.233957+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:41:19.821669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.20","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.220680+0000","last_change":"2023-10-19T13:00:21.217955+0000","last_active":"2023-10-19T13:00:22.220680+0000","last_peered":"2023-10-19T13:00:22.220680+0000","last_clean":"2023-10-19T13:00:22.220680+0000","last_became_active":"2023-10-19T13:00:21.217649+0000","last_became_peered":"2023-10-19T13:00:21.217649+0000","last_unstale":"2023-10-19T13:00:22.220680+0000","last_undegraded":"2023-10-19T13:00:22.220680+0000","last_fullsized":"2023-10-19T13:00:22.220680+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:55:17.988760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1f","version":"86'4","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-19T13:00:21.238495+0000","last_change":"2023-10-19T13:00:20.197586+0000","last_active":"2023-10-19T13:00:21.238495+0000","last_peered":"2023-10-19T13:00:21.238495+0000","last_clean":"2023-10-19T13:00:21.238495+0000","last_became_active":"2023-10-19T13:00:20.197286+0000","last_became_peered":"2023-10-19T13:00:20.197286+0000","last_unstale":"2023-10-19T13:00:21.238495+0000","last_undegraded":"2023-10-19T13:00:21.238495+0000","last_fullsized":"2023-10-19T13:00:21.238495+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:45:06.777683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1e","version":"86'1","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.221295+0000","last_change":"2023-10-19T13:00:20.194934+0000","last_active":"2023-10-19T13:00:22.221295+0000","last_peered":"2023-10-19T13:00:22.221295+0000","last_clean":"2023-10-19T13:00:22.221295+0000","last_became_active":"2023-10-19T13:00:20.194657+0000","last_became_peered":"2023-10-19T13:00:20.194657+0000","last_unstale":"2023-10-19T13:00:22.221295+0000","last_undegraded":"2023-10-19T13:00:22.221295+0000","last_fullsized":"2023-10-19T13:00:22.221295+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:12:57.142360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.221258+0000","last_change":"2023-10-19T13:00:21.218060+0000","last_active":"2023-10-19T13:00:22.221258+0000","last_peered":"2023-10-19T13:00:22.221258+0000","last_clean":"2023-10-19T13:00:22.221258+0000","last_became_active":"2023-10-19T13:00:21.217766+0000","last_became_peered":"2023-10-19T13:00:21.217766+0000","last_unstale":"2023-10-19T13:00:22.221258+0000","last_undegraded":"2023-10-19T13:00:22.221258+0000","last_fullsized":"2023-10-19T13:00:22.221258+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:50:26.459690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"86'2","reported_seq":14,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.220954+0000","last_change":"2023-10-19T13:00:20.188312+0000","last_active":"2023-10-19T13:00:22.220954+0000","last_peered":"2023-10-19T13:00:22.220954+0000","last_clean":"2023-10-19T13:00:22.220954+0000","last_became_active":"2023-10-19T13:00:20.187673+0000","last_became_peered":"2023-10-19T13:00:20.187673+0000","last_unstale":"2023-10-19T13:00:22.220954+0000","last_undegraded":"2023-10-19T13:00:22.220954+0000","last_fullsized":"2023-10-19T13:00:22.220954+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:22:13.247630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1c","version":"86'2","reported_seq":14,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.221016+0000","last_change":"2023-10-19T13:00:20.197923+0000","last_active":"2023-10-19T13:00:22.221016+0000","last_peered":"2023-10-19T13:00:22.221016+0000","last_clean":"2023-10-19T13:00:22.221016+0000","last_became_active":"2023-10-19T13:00:20.197546+0000","last_became_peered":"2023-10-19T13:00:20.197546+0000","last_unstale":"2023-10-19T13:00:22.221016+0000","last_undegraded":"2023-10-19T13:00:22.221016+0000","last_fullsized":"2023-10-19T13:00:22.221016+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:27:45.352182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:23.044381+0000","last_change":"2023-10-19T13:00:21.229195+0000","last_active":"2023-10-19T13:00:23.044381+0000","last_peered":"2023-10-19T13:00:23.044381+0000","last_clean":"2023-10-19T13:00:23.044381+0000","last_became_active":"2023-10-19T13:00:21.228898+0000","last_became_peered":"2023-10-19T13:00:21.228898+0000","last_unstale":"2023-10-19T13:00:23.044381+0000","last_undegraded":"2023-10-19T13:00:23.044381+0000","last_fullsized":"2023-10-19T13:00:23.044381+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:28:07.228859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:19.169402+0000","last_change":"2023-10-19T13:00:19.169402+0000","last_active":"2023-10-19T13:00:19.169402+0000","last_peered":"2023-10-19T13:00:19.169402+0000","last_clean":"2023-10-19T13:00:19.169402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:19.169402+0000","last_undegraded":"2023-10-19T13:00:19.169402+0000","last_fullsized":"2023-10-19T13:00:19.169402+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":9,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-19T13:00:21.232881+0000","last_change":"2023-10-19T13:00:21.226995+0000","last_active":"2023-10-19T13:00:21.232881+0000","last_peered":"2023-10-19T13:00:21.232881+0000","last_clean":"2023-10-19T13:00:21.232881+0000","last_became_active":"2023-10-19T13:00:21.226673+0000","last_became_peered":"2023-10-19T13:00:21.226673+0000","last_unstale":"2023-10-19T13:00:21.232881+0000","last_undegraded":"2023-10-19T13:00:21.232881+0000","last_fullsized":"2023-10-19T13:00:21.232881+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:52:56.240367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:19.169402+0000","last_change":"2023-10-19T13:00:19.169402+0000","last_active":"2023-10-19T13:00:19.169402+0000","last_peered":"2023-10-19T13:00:19.169402+0000","last_clean":"2023-10-19T13:00:19.169402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:19.169402+0000","last_undegraded":"2023-10-19T13:00:19.169402+0000","last_fullsized":"2023-10-19T13:00:19.169402+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:19.169402+0000","last_change":"2023-10-19T13:00:19.169402+0000","last_active":"2023-10-19T13:00:19.169402+0000","last_peered":"2023-10-19T13:00:19.169402+0000","last_clean":"2023-10-19T13:00:19.169402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:19.169402+0000","last_undegraded":"2023-10-19T13:00:19.169402+0000","last_fullsized":"2023-10-19T13:00:19.169402+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:19.169402+0000","last_change":"2023-10-19T13:00:19.169402+0000","last_active":"2023-10-19T13:00:19.169402+0000","last_peered":"2023-10-19T13:00:19.169402+0000","last_clean":"2023-10-19T13:00:19.169402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:19.169402+0000","last_undegraded":"2023-10-19T13:00:19.169402+0000","last_fullsized":"2023-10-19T13:00:19.169402+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":9,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-19T13:00:21.233976+0000","last_change":"2023-10-19T13:00:21.227321+0000","last_active":"2023-10-19T13:00:21.233976+0000","last_peered":"2023-10-19T13:00:21.233976+0000","last_clean":"2023-10-19T13:00:21.233976+0000","last_became_active":"2023-10-19T13:00:21.227028+0000","last_became_peered":"2023-10-19T13:00:21.227028+0000","last_unstale":"2023-10-19T13:00:21.233976+0000","last_undegraded":"2023-10-19T13:00:21.233976+0000","last_fullsized":"2023-10-19T13:00:21.233976+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:06:08.406600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:19.169402+0000","last_change":"2023-10-19T13:00:19.169402+0000","last_active":"2023-10-19T13:00:19.169402+0000","last_peered":"2023-10-19T13:00:19.169402+0000","last_clean":"2023-10-19T13:00:19.169402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:19.169402+0000","last_undegraded":"2023-10-19T13:00:19.169402+0000","last_fullsized":"2023-10-19T13:00:19.169402+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:19.169402+0000","last_change":"2023-10-19T13:00:19.169402+0000","last_active":"2023-10-19T13:00:19.169402+0000","last_peered":"2023-10-19T13:00:19.169402+0000","last_clean":"2023-10-19T13:00:19.169402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:19.169402+0000","last_undegraded":"2023-10-19T13:00:19.169402+0000","last_fullsized":"2023-10-19T13:00:19.169402+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:19.169402+0000","last_change":"2023-10-19T13:00:19.169402+0000","last_active":"2023-10-19T13:00:19.169402+0000","last_peered":"2023-10-19T13:00:19.169402+0000","last_clean":"2023-10-19T13:00:19.169402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:19.169402+0000","last_undegraded":"2023-10-19T13:00:19.169402+0000","last_fullsized":"2023-10-19T13:00:19.169402+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.14","version":"86'6","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-19T13:00:21.240123+0000","last_change":"2023-10-19T13:00:20.198504+0000","last_active":"2023-10-19T13:00:21.240123+0000","last_peered":"2023-10-19T13:00:21.240123+0000","last_clean":"2023-10-19T13:00:21.240123+0000","last_became_active":"2023-10-19T13:00:20.197655+0000","last_became_peered":"2023-10-19T13:00:20.197655+0000","last_unstale":"2023-10-19T13:00:21.240123+0000","last_undegraded":"2023-10-19T13:00:21.240123+0000","last_fullsized":"2023-10-19T13:00:21.240123+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:25:20.022851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:19.169402+0000","last_change":"2023-10-19T13:00:19.169402+0000","last_active":"2023-10-19T13:00:19.169402+0000","last_peered":"2023-10-19T13:00:19.169402+0000","last_clean":"2023-10-19T13:00:19.169402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:19.169402+0000","last_undegraded":"2023-10-19T13:00:19.169402+0000","last_fullsized":"2023-10-19T13:00:19.169402+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:19.169402+0000","last_change":"2023-10-19T13:00:19.169402+0000","last_active":"2023-10-19T13:00:19.169402+0000","last_peered":"2023-10-19T13:00:19.169402+0000","last_clean":"2023-10-19T13:00:19.169402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:19.169402+0000","last_undegraded":"2023-10-19T13:00:19.169402+0000","last_fullsized":"2023-10-19T13:00:19.169402+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:19.169402+0000","last_change":"2023-10-19T13:00:19.169402+0000","last_active":"2023-10-19T13:00:19.169402+0000","last_peered":"2023-10-19T13:00:19.169402+0000","last_clean":"2023-10-19T13:00:19.169402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:19.169402+0000","last_undegraded":"2023-10-19T13:00:19.169402+0000","last_fullsized":"2023-10-19T13:00:19.169402+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":9,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-19T13:00:21.232936+0000","last_change":"2023-10-19T13:00:21.228024+0000","last_active":"2023-10-19T13:00:21.232936+0000","last_peered":"2023-10-19T13:00:21.232936+0000","last_clean":"2023-10-19T13:00:21.232936+0000","last_became_active":"2023-10-19T13:00:21.227727+0000","last_became_peered":"2023-10-19T13:00:21.227727+0000","last_unstale":"2023-10-19T13:00:21.232936+0000","last_undegraded":"2023-10-19T13:00:21.232936+0000","last_fullsized":"2023-10-19T13:00:21.232936+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:14:55.381409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:19.169402+0000","last_change":"2023-10-19T13:00:19.169402+0000","last_active":"2023-10-19T13:00:19.169402+0000","last_peered":"2023-10-19T13:00:19.169402+0000","last_clean":"2023-10-19T13:00:19.169402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:19.169402+0000","last_undegraded":"2023-10-19T13:00:19.169402+0000","last_fullsized":"2023-10-19T13:00:19.169402+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:19.169402+0000","last_change":"2023-10-19T13:00:19.169402+0000","last_active":"2023-10-19T13:00:19.169402+0000","last_peered":"2023-10-19T13:00:19.169402+0000","last_clean":"2023-10-19T13:00:19.169402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:19.169402+0000","last_undegraded":"2023-10-19T13:00:19.169402+0000","last_fullsized":"2023-10-19T13:00:19.169402+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:19.169402+0000","last_change":"2023-10-19T13:00:19.169402+0000","last_active":"2023-10-19T13:00:19.169402+0000","last_peered":"2023-10-19T13:00:19.169402+0000","last_clean":"2023-10-19T13:00:19.169402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:19.169402+0000","last_undegraded":"2023-10-19T13:00:19.169402+0000","last_fullsized":"2023-10-19T13:00:19.169402+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:19.169402+0000","last_change":"2023-10-19T13:00:19.169402+0000","last_active":"2023-10-19T13:00:19.169402+0000","last_peered":"2023-10-19T13:00:19.169402+0000","last_clean":"2023-10-19T13:00:19.169402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:19.169402+0000","last_undegraded":"2023-10-19T13:00:19.169402+0000","last_fullsized":"2023-10-19T13:00:19.169402+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:19.169402+0000","last_change":"2023-10-19T13:00:19.169402+0000","last_active":"2023-10-19T13:00:19.169402+0000","last_peered":"2023-10-19T13:00:19.169402+0000","last_clean":"2023-10-19T13:00:19.169402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:19.169402+0000","last_undegraded":"2023-10-19T13:00:19.169402+0000","last_fullsized":"2023-10-19T13:00:19.169402+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.220733+0000","last_change":"2023-10-19T13:00:21.217141+0000","last_active":"2023-10-19T13:00:22.220733+0000","last_peered":"2023-10-19T13:00:22.220733+0000","last_clean":"2023-10-19T13:00:22.220733+0000","last_became_active":"2023-10-19T13:00:21.216807+0000","last_became_peered":"2023-10-19T13:00:21.216807+0000","last_unstale":"2023-10-19T13:00:22.220733+0000","last_undegraded":"2023-10-19T13:00:22.220733+0000","last_fullsized":"2023-10-19T13:00:22.220733+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:39:34.098405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"86'2","reported_seq":14,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.220911+0000","last_change":"2023-10-19T13:00:20.195565+0000","last_active":"2023-10-19T13:00:22.220911+0000","last_peered":"2023-10-19T13:00:22.220911+0000","last_clean":"2023-10-19T13:00:22.220911+0000","last_became_active":"2023-10-19T13:00:20.195256+0000","last_became_peered":"2023-10-19T13:00:20.195256+0000","last_unstale":"2023-10-19T13:00:22.220911+0000","last_undegraded":"2023-10-19T13:00:22.220911+0000","last_fullsized":"2023-10-19T13:00:22.220911+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:18:37.647497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:19.169402+0000","last_change":"2023-10-19T13:00:19.169402+0000","last_active":"2023-10-19T13:00:19.169402+0000","last_peered":"2023-10-19T13:00:19.169402+0000","last_clean":"2023-10-19T13:00:19.169402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:19.169402+0000","last_undegraded":"2023-10-19T13:00:19.169402+0000","last_fullsized":"2023-10-19T13:00:19.169402+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.220731+0000","last_change":"2023-10-19T13:00:20.194028+0000","last_active":"2023-10-19T13:00:22.220731+0000","last_peered":"2023-10-19T13:00:22.220731+0000","last_clean":"2023-10-19T13:00:22.220731+0000","last_became_active":"2023-10-19T13:00:20.193758+0000","last_became_peered":"2023-10-19T13:00:20.193758+0000","last_unstale":"2023-10-19T13:00:22.220731+0000","last_undegraded":"2023-10-19T13:00:22.220731+0000","last_fullsized":"2023-10-19T13:00:22.220731+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:41:30.219455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.221024+0000","last_change":"2023-10-19T13:00:21.221247+0000","last_active":"2023-10-19T13:00:22.221024+0000","last_peered":"2023-10-19T13:00:22.221024+0000","last_clean":"2023-10-19T13:00:22.221024+0000","last_became_active":"2023-10-19T13:00:21.220182+0000","last_became_peered":"2023-10-19T13:00:21.220182+0000","last_unstale":"2023-10-19T13:00:22.221024+0000","last_undegraded":"2023-10-19T13:00:22.221024+0000","last_fullsized":"2023-10-19T13:00:22.221024+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:47:08.944926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:19.169402+0000","last_change":"2023-10-19T13:00:19.169402+0000","last_active":"2023-10-19T13:00:19.169402+0000","last_peered":"2023-10-19T13:00:19.169402+0000","last_clean":"2023-10-19T13:00:19.169402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:19.169402+0000","last_undegraded":"2023-10-19T13:00:19.169402+0000","last_fullsized":"2023-10-19T13:00:19.169402+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:23.044679+0000","last_change":"2023-10-19T13:00:21.222511+0000","last_active":"2023-10-19T13:00:23.044679+0000","last_peered":"2023-10-19T13:00:23.044679+0000","last_clean":"2023-10-19T13:00:23.044679+0000","last_became_active":"2023-10-19T13:00:21.222236+0000","last_became_peered":"2023-10-19T13:00:21.222236+0000","last_unstale":"2023-10-19T13:00:23.044679+0000","last_undegraded":"2023-10-19T13:00:23.044679+0000","last_fullsized":"2023-10-19T13:00:23.044679+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:50:46.028354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:19.169402+0000","last_change":"2023-10-19T13:00:19.169402+0000","last_active":"2023-10-19T13:00:19.169402+0000","last_peered":"2023-10-19T13:00:19.169402+0000","last_clean":"2023-10-19T13:00:19.169402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:19.169402+0000","last_undegraded":"2023-10-19T13:00:19.169402+0000","last_fullsized":"2023-10-19T13:00:19.169402+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.221031+0000","last_change":"2023-10-19T13:00:20.188530+0000","last_active":"2023-10-19T13:00:22.221031+0000","last_peered":"2023-10-19T13:00:22.221031+0000","last_clean":"2023-10-19T13:00:22.221031+0000","last_became_active":"2023-10-19T13:00:20.188074+0000","last_became_peered":"2023-10-19T13:00:20.188074+0000","last_unstale":"2023-10-19T13:00:22.221031+0000","last_undegraded":"2023-10-19T13:00:22.221031+0000","last_fullsized":"2023-10-19T13:00:22.221031+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:08:44.211054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-19T13:00:21.216870+0000","last_change":"2023-10-19T13:00:20.198569+0000","last_active":"2023-10-19T13:00:21.216870+0000","last_peered":"2023-10-19T13:00:21.216870+0000","last_clean":"2023-10-19T13:00:21.216870+0000","last_became_active":"2023-10-19T13:00:20.197088+0000","last_became_peered":"2023-10-19T13:00:20.197088+0000","last_unstale":"2023-10-19T13:00:21.216870+0000","last_undegraded":"2023-10-19T13:00:21.216870+0000","last_fullsized":"2023-10-19T13:00:21.216870+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:38:39.386315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:19.169402+0000","last_change":"2023-10-19T13:00:19.169402+0000","last_active":"2023-10-19T13:00:19.169402+0000","last_peered":"2023-10-19T13:00:19.169402+0000","last_clean":"2023-10-19T13:00:19.169402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:19.169402+0000","last_undegraded":"2023-10-19T13:00:19.169402+0000","last_fullsized":"2023-10-19T13:00:19.169402+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:19.169402+0000","last_change":"2023-10-19T13:00:19.169402+0000","last_active":"2023-10-19T13:00:19.169402+0000","last_peered":"2023-10-19T13:00:19.169402+0000","last_clean":"2023-10-19T13:00:19.169402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:19.169402+0000","last_undegraded":"2023-10-19T13:00:19.169402+0000","last_fullsized":"2023-10-19T13:00:19.169402+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":9,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-19T13:00:21.227472+0000","last_change":"2023-10-19T13:00:21.226366+0000","last_active":"2023-10-19T13:00:21.227472+0000","last_peered":"2023-10-19T13:00:21.227472+0000","last_clean":"2023-10-19T13:00:21.227472+0000","last_became_active":"2023-10-19T13:00:21.226075+0000","last_became_peered":"2023-10-19T13:00:21.226075+0000","last_unstale":"2023-10-19T13:00:21.227472+0000","last_undegraded":"2023-10-19T13:00:21.227472+0000","last_fullsized":"2023-10-19T13:00:21.227472+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:33:36.402003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:19.169402+0000","last_change":"2023-10-19T13:00:19.169402+0000","last_active":"2023-10-19T13:00:19.169402+0000","last_peered":"2023-10-19T13:00:19.169402+0000","last_clean":"2023-10-19T13:00:19.169402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:19.169402+0000","last_undegraded":"2023-10-19T13:00:19.169402+0000","last_fullsized":"2023-10-19T13:00:19.169402+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:23.044464+0000","last_change":"2023-10-19T13:00:21.219143+0000","last_active":"2023-10-19T13:00:23.044464+0000","last_peered":"2023-10-19T13:00:23.044464+0000","last_clean":"2023-10-19T13:00:23.044464+0000","last_became_active":"2023-10-19T13:00:21.218834+0000","last_became_peered":"2023-10-19T13:00:21.218834+0000","last_unstale":"2023-10-19T13:00:23.044464+0000","last_undegraded":"2023-10-19T13:00:23.044464+0000","last_fullsized":"2023-10-19T13:00:23.044464+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:29:26.836263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-19T13:00:21.215556+0000","last_change":"2023-10-19T13:00:20.198235+0000","last_active":"2023-10-19T13:00:21.215556+0000","last_peered":"2023-10-19T13:00:21.215556+0000","last_clean":"2023-10-19T13:00:21.215556+0000","last_became_active":"2023-10-19T13:00:20.197920+0000","last_became_peered":"2023-10-19T13:00:20.197920+0000","last_unstale":"2023-10-19T13:00:21.215556+0000","last_undegraded":"2023-10-19T13:00:21.215556+0000","last_fullsized":"2023-10-19T13:00:21.215556+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:34:56.789811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:20.171814+0000","last_change":"2023-10-19T13:00:20.171814+0000","last_active":"2023-10-19T13:00:20.171814+0000","last_peered":"2023-10-19T13:00:20.171814+0000","last_clean":"2023-10-19T13:00:20.171814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:20.171814+0000","last_undegraded":"2023-10-19T13:00:20.171814+0000","last_fullsized":"2023-10-19T13:00:20.171814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:00:19.169402+0000","last_change":"2023-10-19T13:00:19.169402+0000","last_active":"2023-10-19T13:00:19.169402+0000","last_peered":"2023-10-19T13:00:19.169402+0000","last_clean":"2023-10-19T13:00:19.169402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:00:19.169402+0000","last_undegraded":"2023-10-19T13:00:19.169402+0000","last_fullsized":"2023-10-19T13:00:19.169402+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"76'161","reported_seq":381,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-10-19T13:00:16.745788+0000","last_change":"2023-10-19T12:44:08.636613+0000","last_active":"2023-10-19T13:00:16.745788+0000","last_peered":"2023-10-19T13:00:16.745788+0000","last_clean":"2023-10-19T13:00:16.745788+0000","last_became_active":"2023-10-19T12:44:08.635697+0000","last_became_peered":"2023-10-19T12:44:08.635697+0000","last_unstale":"2023-10-19T13:00:16.745788+0000","last_undegraded":"2023-10-19T13:00:16.745788+0000","last_fullsized":"2023-10-19T13:00:16.745788+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_clean_scrub_stamp":"2023-10-19T12:44:07.399738+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:15:23.685485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":36,"acting":36,"num_store_stats":3},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":2194,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":20,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575045,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59112,"kb_used_data":1564,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312728,"statfs":{"total":96636764160,"available":96576233472,"internally_reserved":0,"allocated":1601536,"data_stored":1092689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58915275},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:59:44 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.57499999999999996,"15min":0.56999999999999995},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.248},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.60099999999999998,"15min":0.58799999999999997},"min":{"1min":0.45400000000000001,"5min":0.33200000000000002,"15min":0.26900000000000002},"max":{"1min":0.78200000000000003,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.50800000000000001}]},{"osd":1,"last update":"Thu Oct 19 12:59:44 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.58199999999999996,"15min":0.57599999999999996},"min":{"1min":0.41299999999999998,"5min":0.40300000000000002,"15min":0.28199999999999997},"max":{"1min":0.84299999999999997,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.623,"5min":0.59999999999999998,"15min":0.58799999999999997},"min":{"1min":0.51700000000000002,"5min":0.43099999999999999,"15min":0.29299999999999998},"max":{"1min":0.82699999999999996,"5min":1.284,"15min":1.284},"last":0.60399999999999998}]},{"osd":2,"last update":"Thu Oct 19 12:59:44 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.64100000000000001,"15min":0.629},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.375},"max":{"1min":0.79300000000000004,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.64600000000000002,"15min":0.63300000000000001},"min":{"1min":0.55300000000000005,"5min":0.41599999999999998,"15min":0.28399999999999997},"max":{"1min":0.81200000000000006,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.47299999999999998}]},{"osd":3,"last update":"Thu Oct 19 12:59:44 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60699999999999998,"15min":0.60099999999999998},"min":{"1min":0.47199999999999998,"5min":0.34999999999999998,"15min":0.34399999999999997},"max":{"1min":0.78900000000000003,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.67000000000000004,"15min":0.65500000000000003},"min":{"1min":0.58199999999999996,"5min":0.45300000000000001,"15min":0.39700000000000002},"max":{"1min":0.92000000000000004,"5min":1.329,"15min":1.329},"last":0.65400000000000003}]},{"osd":4,"last update":"Thu Oct 19 12:59:44 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.46800000000000003,"15min":0.46700000000000003},"min":{"1min":0.36899999999999999,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.72599999999999998,"5min":0.75800000000000001,"15min":0.80500000000000005},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.33600000000000002,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.623,"5min":0.75600000000000001,"15min":0.78900000000000003},"last":0.52400000000000002}]},{"osd":5,"last update":"Thu Oct 19 12:59:44 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.48299999999999998,"15min":0.48499999999999999},"min":{"1min":0.34399999999999997,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.68600000000000005,"5min":0.75800000000000001,"15min":0.90400000000000003},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.52200000000000002,"15min":0.51600000000000001},"min":{"1min":0.38,"5min":0.314,"15min":0.308},"max":{"1min":0.71499999999999997,"5min":0.78700000000000003,"15min":0.92200000000000004},"last":0.624}]},{"osd":6,"last update":"Thu Oct 19 12:59:44 2023","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.54200000000000004,"15min":0.54700000000000004},"min":{"1min":0.379,"5min":0.308,"15min":0.308},"max":{"1min":0.749,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.52100000000000002,"15min":0.52900000000000003},"min":{"1min":0.36899999999999999,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.76900000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.58799999999999997}]}]},{"osd":6,"up_from":13,"seq":55834575045,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32168,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339672,"statfs":{"total":96636764160,"available":96603824128,"internally_reserved":0,"allocated":1007616,"data_stored":502321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:59:28 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.57999999999999996,"15min":0.58699999999999997},"min":{"1min":0.41899999999999998,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.755,"5min":0.79900000000000004,"15min":0.81999999999999995},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.41999999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.56399999999999995}]},{"osd":1,"last update":"Thu Oct 19 12:59:28 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.60299999999999998,"15min":0.59999999999999998},"min":{"1min":0.39900000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.71999999999999997,"5min":0.83299999999999996,"15min":0.93899999999999995},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.59699999999999998,"15min":0.59999999999999998},"min":{"1min":0.42599999999999999,"5min":0.34300000000000003,"15min":0.28399999999999997},"max":{"1min":0.78200000000000003,"5min":0.82099999999999995,"15min":1.075},"last":0.68700000000000006}]},{"osd":2,"last update":"Thu Oct 19 12:59:28 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.42799999999999999,"5min":0.307,"15min":0.26600000000000001},"max":{"1min":0.84699999999999998,"5min":0.85299999999999998,"15min":0.85699999999999998},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.624,"5min":0.61299999999999999,"15min":0.60799999999999998},"min":{"1min":0.48499999999999999,"5min":0.35999999999999999,"15min":0.28399999999999997},"max":{"1min":0.82199999999999995,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.38900000000000001}]},{"osd":3,"last update":"Thu Oct 19 12:59:28 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.625,"15min":0.623},"min":{"1min":0.46400000000000002,"5min":0.38500000000000001,"15min":0.32600000000000001},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.61799999999999999,"15min":0.628},"min":{"1min":0.44900000000000001,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":1.0900000000000001},"last":0.49199999999999999}]},{"osd":4,"last update":"Thu Oct 19 12:59:28 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.46100000000000002,"15min":0.46999999999999997},"min":{"1min":0.32900000000000001,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.64400000000000002,"5min":0.77900000000000003,"15min":1.02},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.47999999999999998,"15min":0.48499999999999999},"min":{"1min":0.33200000000000002,"5min":0.32800000000000001,"15min":0.247},"max":{"1min":0.69099999999999995,"5min":0.752,"15min":0.80000000000000004},"last":0.73799999999999999}]},{"osd":5,"last update":"Thu Oct 19 12:59:28 2023","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.46800000000000003,"15min":0.47699999999999998},"min":{"1min":0.32900000000000001,"5min":0.30199999999999999,"15min":0.26600000000000001},"max":{"1min":0.61899999999999999,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.442},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.50800000000000001,"15min":0.51900000000000002},"min":{"1min":0.35699999999999998,"5min":0.32100000000000001,"15min":0.312},"max":{"1min":0.75600000000000001,"5min":0.85699999999999998,"15min":0.98799999999999999},"last":0.63300000000000001}]},{"osd":7,"last update":"Thu Oct 19 12:59:28 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.51800000000000002,"15min":0.52300000000000002},"min":{"1min":0.32500000000000001,"5min":0.29299999999999998,"15min":0.28199999999999997},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":1.036},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53100000000000003,"15min":0.54100000000000004},"min":{"1min":0.36499999999999999,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":1.1299999999999999},"last":0.65100000000000002}]}]},{"osd":5,"up_from":13,"seq":55834575045,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32176,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339664,"statfs":{"total":96636764160,"available":96603815936,"internally_reserved":0,"allocated":1007616,"data_stored":502321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:59:29 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.58799999999999997,"15min":0.58899999999999997},"min":{"1min":0.44700000000000001,"5min":0.28999999999999998,"15min":0.223},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.88800000000000001},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.63,"5min":0.58999999999999997,"15min":0.59599999999999997},"min":{"1min":0.42499999999999999,"5min":0.36099999999999999,"15min":0.27800000000000002},"max":{"1min":0.81399999999999995,"5min":0.84299999999999997,"15min":0.874},"last":0.82999999999999996}]},{"osd":1,"last update":"Thu Oct 19 12:59:29 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.57899999999999996,"15min":0.59399999999999997},"min":{"1min":0.46899999999999997,"5min":0.34599999999999997,"15min":0.32800000000000001},"max":{"1min":0.77500000000000002,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.59999999999999998,"15min":0.60599999999999998},"min":{"1min":0.49199999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83499999999999996},"last":0.70099999999999996}]},{"osd":2,"last update":"Thu Oct 19 12:59:29 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.57299999999999995,"15min":0.58299999999999996},"min":{"1min":0.48899999999999999,"5min":0.34100000000000003,"15min":0.307},"max":{"1min":0.73999999999999999,"5min":0.78600000000000003,"15min":0.80700000000000005},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.60499999999999998,"15min":0.61899999999999999},"min":{"1min":0.50900000000000001,"5min":0.38400000000000001,"15min":0.33700000000000002},"max":{"1min":0.80900000000000005,"5min":0.88100000000000001,"15min":0.93600000000000005},"last":0.79300000000000004}]},{"osd":3,"last update":"Thu Oct 19 12:59:29 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.59299999999999997,"15min":0.60199999999999998},"min":{"1min":0.48499999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.751,"5min":0.79900000000000004,"15min":0.874},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.625,"15min":0.63400000000000001},"min":{"1min":0.50900000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.89500000000000002},"last":0.71799999999999997}]},{"osd":4,"last update":"Thu Oct 19 12:59:29 2023","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.36099999999999999,"5min":0.29099999999999998,"15min":0.28899999999999998},"max":{"1min":0.61399999999999999,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.46700000000000003,"15min":0.47199999999999998},"min":{"1min":0.35899999999999999,"5min":0.251,"15min":0.23899999999999999},"max":{"1min":0.58299999999999996,"5min":0.79400000000000004,"15min":0.80500000000000005},"last":0.36199999999999999}]},{"osd":6,"last update":"Thu Oct 19 12:59:29 2023","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46500000000000002,"15min":0.47299999999999998},"min":{"1min":0.34499999999999997,"5min":0.27200000000000002,"15min":0.25900000000000001},"max":{"1min":0.56299999999999994,"5min":0.74399999999999999,"15min":0.79300000000000004},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.497,"5min":0.50900000000000001,"15min":0.502},"min":{"1min":0.373,"5min":0.34200000000000003,"15min":0.30399999999999999},"max":{"1min":0.64800000000000002,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.60499999999999998}]},{"osd":7,"last update":"Thu Oct 19 12:59:29 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.48599999999999999,"15min":0.49299999999999999},"min":{"1min":0.39400000000000002,"5min":0.32000000000000001,"15min":0.315},"max":{"1min":0.64200000000000002,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.624},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.53900000000000003,"15min":0.53700000000000003},"min":{"1min":0.46100000000000002,"5min":0.311,"15min":0.27500000000000002},"max":{"1min":0.66900000000000004,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.68100000000000005}]}]},{"osd":4,"up_from":13,"seq":55834575045,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32152,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339688,"statfs":{"total":96636764160,"available":96603840512,"internally_reserved":0,"allocated":995328,"data_stored":496916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56100000000000005,"15min":0.55900000000000005},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.28899999999999998},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.55800000000000005,"15min":0.55500000000000005},"min":{"1min":0.45400000000000001,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.77100000000000002,"5min":0.77800000000000002,"15min":0.78200000000000003},"last":0.39400000000000002}]},{"osd":1,"last update":"Thu Oct 19 12:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.60399999999999998,"15min":0.60799999999999998},"min":{"1min":0.33000000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.78600000000000003,"5min":0.80600000000000005,"15min":0.86199999999999999},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61899999999999999,"15min":0.61599999999999999},"min":{"1min":0.40899999999999997,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.77300000000000002,"5min":0.79900000000000004,"15min":0.873},"last":0.54700000000000004}]},{"osd":2,"last update":"Thu Oct 19 12:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.61199999999999999,"15min":0.61299999999999999},"min":{"1min":0.375,"5min":0.32200000000000001,"15min":0.254},"max":{"1min":0.72199999999999998,"5min":0.85299999999999998,"15min":0.86599999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61699999999999999,"15min":0.61599999999999999},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.34499999999999997},"max":{"1min":0.78500000000000003,"5min":0.82999999999999996,"15min":0.85999999999999999},"last":0.42099999999999999}]},{"osd":3,"last update":"Thu Oct 19 12:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.59099999999999997,"15min":0.58399999999999996},"min":{"1min":0.41999999999999998,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.73499999999999999,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.64600000000000002,"15min":0.63700000000000001},"min":{"1min":0.52500000000000002,"5min":0.42599999999999999,"15min":0.37},"max":{"1min":0.73899999999999999,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.624}]},{"osd":5,"last update":"Thu Oct 19 12:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.45600000000000002,"15min":0.45200000000000001},"min":{"1min":0.309,"5min":0.29599999999999999,"15min":0.249},"max":{"1min":0.63500000000000001,"5min":0.79600000000000004,"15min":0.81299999999999994},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.47499999999999998,"15min":0.46200000000000002},"min":{"1min":0.34699999999999998,"5min":0.26400000000000001,"15min":0.23499999999999999},"max":{"1min":0.623,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.505}]},{"osd":6,"last update":"Thu Oct 19 12:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.45900000000000002,"15min":0.45600000000000002},"min":{"1min":0.31,"5min":0.29499999999999998,"15min":0.26400000000000001},"max":{"1min":0.52800000000000002,"5min":0.72599999999999998,"15min":0.78600000000000003},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.48599999999999999,"15min":0.497},"min":{"1min":0.34300000000000003,"5min":0.32100000000000001,"15min":0.27100000000000002},"max":{"1min":0.66700000000000004,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.63600000000000001}]},{"osd":7,"last update":"Thu Oct 19 12:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.501,"15min":0.503},"min":{"1min":0.33700000000000002,"5min":0.307,"15min":0.27500000000000002},"max":{"1min":0.68300000000000005,"5min":0.79100000000000004,"15min":0.80200000000000005},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.503,"5min":0.54200000000000004,"15min":0.54800000000000004},"min":{"1min":0.39100000000000001,"5min":0.35799999999999998,"15min":0.30199999999999999},"max":{"1min":0.69399999999999995,"5min":0.80500000000000005,"15min":0.83499999999999996},"last":0.67200000000000004}]}]},{"osd":3,"up_from":13,"seq":55834575044,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":36332,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335508,"statfs":{"total":96636764160,"available":96599560192,"internally_reserved":0,"allocated":1007616,"data_stored":502321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:59:24 2023","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.41799999999999998,"15min":0.39900000000000002},"min":{"1min":0.27100000000000002,"5min":0.23599999999999999,"15min":0.20899999999999999},"max":{"1min":0.69099999999999995,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.252},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.41199999999999998,"15min":0.41299999999999998},"min":{"1min":0.316,"5min":0.223,"15min":0.223},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.28699999999999998}]},{"osd":1,"last update":"Thu Oct 19 12:59:24 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.40600000000000003,"15min":0.40699999999999997},"min":{"1min":0.29099999999999998,"5min":0.26100000000000001,"15min":0.22500000000000001},"max":{"1min":0.63500000000000001,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.433},{"interface":"front","average":{"1min":0.505,"5min":0.48299999999999998,"15min":0.45700000000000002},"min":{"1min":0.27600000000000002,"5min":0.24099999999999999,"15min":0.215},"max":{"1min":0.66700000000000004,"5min":2.9399999999999999,"15min":2.9399999999999999},"last":0.32700000000000001}]},{"osd":2,"last update":"Thu Oct 19 12:59:24 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.46899999999999997,"15min":0.46500000000000002},"min":{"1min":0.33800000000000002,"5min":0.314,"15min":0.26100000000000001},"max":{"1min":0.69399999999999995,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.48099999999999998,"15min":0.48499999999999999},"min":{"1min":0.35099999999999998,"5min":0.29999999999999999,"15min":0.247},"max":{"1min":0.754,"5min":0.754,"15min":0.96499999999999997},"last":0.39000000000000001}]},{"osd":4,"last update":"Thu Oct 19 12:59:24 2023","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.65700000000000003,"15min":0.66400000000000003},"min":{"1min":0.50700000000000001,"5min":0.42299999999999999,"15min":0.40300000000000002},"max":{"1min":0.90200000000000002,"5min":0.90500000000000003,"15min":1.0069999999999999},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.69499999999999995,"15min":0.69199999999999995},"min":{"1min":0.59499999999999997,"5min":0.46899999999999997,"15min":0.42099999999999999},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.99399999999999999},"last":0.67900000000000005}]},{"osd":5,"last update":"Thu Oct 19 12:59:24 2023","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.67600000000000005,"15min":0.65900000000000003},"min":{"1min":0.56000000000000005,"5min":0.45500000000000002,"15min":0.40500000000000003},"max":{"1min":0.93600000000000005,"5min":2.9990000000000001,"15min":2.9990000000000001},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.52100000000000002,"5min":0.40899999999999997,"15min":0.35899999999999999},"max":{"1min":0.81899999999999995,"5min":1.774,"15min":1.774},"last":0.59699999999999998}]},{"osd":6,"last update":"Thu Oct 19 12:59:24 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.66100000000000003,"15min":0.65400000000000003},"min":{"1min":0.52300000000000002,"5min":0.433,"15min":0.41199999999999998},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.96799999999999997},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.67500000000000004,"15min":0.67400000000000004},"min":{"1min":0.60299999999999998,"5min":0.377,"15min":0.35899999999999999},"max":{"1min":0.93700000000000006,"5min":1.167,"15min":1.167},"last":0.64500000000000002}]},{"osd":7,"last update":"Thu Oct 19 12:59:24 2023","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.65200000000000002,"15min":0.66500000000000004},"min":{"1min":0.45500000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.94299999999999995,"5min":1.042,"15min":1.042},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.69299999999999995,"15min":0.67900000000000005},"min":{"1min":0.54200000000000004,"5min":0.432,"15min":0.432},"max":{"1min":0.89200000000000002,"5min":2.8149999999999999,"15min":2.8149999999999999},"last":0.61299999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575047,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54416,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317424,"statfs":{"total":96636764160,"available":96581042176,"internally_reserved":0,"allocated":1044480,"data_stored":522224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:59:33 2023","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.377,"15min":0.38100000000000001},"min":{"1min":0.27500000000000002,"5min":0.20899999999999999,"15min":0.187},"max":{"1min":0.54000000000000004,"5min":0.67700000000000005,"15min":0.83999999999999997},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.436,"5min":0.40999999999999998,"15min":0.40600000000000003},"min":{"1min":0.32900000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.58299999999999996,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.56000000000000005}]},{"osd":1,"last update":"Thu Oct 19 12:59:33 2023","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.39400000000000002,"15min":0.39800000000000002},"min":{"1min":0.318,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.80400000000000005},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.41999999999999998,"15min":0.42999999999999999},"min":{"1min":0.38200000000000001,"5min":0.22,"15min":0.22},"max":{"1min":0.59499999999999997,"5min":0.60499999999999998,"15min":0.78600000000000003},"last":0.61299999999999999}]},{"osd":3,"last update":"Thu Oct 19 12:59:33 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.46300000000000002,"15min":0.45700000000000002},"min":{"1min":0.33400000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.498},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.48099999999999998,"15min":0.48499999999999999},"min":{"1min":0.40300000000000002,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.72799999999999998,"5min":0.77800000000000002,"15min":0.87},"last":0.57499999999999996}]},{"osd":4,"last update":"Thu Oct 19 12:59:33 2023","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.63700000000000001,"15min":0.64200000000000002},"min":{"1min":0.497,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.89500000000000002,"5min":0.93999999999999995,"15min":1.0569999999999999},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.65400000000000003,"15min":0.65700000000000003},"min":{"1min":0.54300000000000004,"5min":0.35099999999999998,"15min":0.32000000000000001},"max":{"1min":0.83399999999999996,"5min":0.92300000000000004,"15min":1.0780000000000001},"last":0.70999999999999996}]},{"osd":5,"last update":"Thu Oct 19 12:59:33 2023","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.67400000000000004,"15min":0.66200000000000003},"min":{"1min":0.58299999999999996,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.84299999999999997,"5min":0.96499999999999997,"15min":0.99399999999999999},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.65200000000000002,"15min":0.65800000000000003},"min":{"1min":0.56899999999999995,"5min":0.375,"15min":0.375},"max":{"1min":0.872,"5min":0.94599999999999995,"15min":0.97099999999999997},"last":0.64000000000000001}]},{"osd":6,"last update":"Thu Oct 19 12:59:33 2023","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.65000000000000002,"15min":0.66700000000000004},"min":{"1min":0.47199999999999998,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.82399999999999995,"5min":1.008,"15min":1.008},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.67800000000000005,"15min":0.67400000000000004},"min":{"1min":0.52000000000000002,"5min":0.371,"15min":0.371},"max":{"1min":0.88100000000000001,"5min":0.995,"15min":0.995},"last":0.63}]},{"osd":7,"last update":"Thu Oct 19 12:59:33 2023","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.67500000000000004,"15min":0.68500000000000005},"min":{"1min":0.59699999999999998,"5min":0.39300000000000002,"15min":0.36399999999999999},"max":{"1min":0.86099999999999999,"5min":1.0149999999999999,"15min":1.111},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.68899999999999995,"15min":0.68300000000000005},"min":{"1min":0.53000000000000003,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.89800000000000002,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.65300000000000002}]}]},{"osd":1,"up_from":13,"seq":55834575046,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32236,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339604,"statfs":{"total":96636764160,"available":96603754496,"internally_reserved":0,"allocated":1073152,"data_stored":524600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:59:33 2023","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.39200000000000002,"15min":0.38500000000000001},"min":{"1min":0.23999999999999999,"5min":0.21299999999999999,"15min":0.187},"max":{"1min":0.55800000000000005,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.442,"15min":0.42799999999999999},"min":{"1min":0.29499999999999998,"5min":0.23899999999999999,"15min":0.21199999999999999},"max":{"1min":0.58799999999999997,"5min":2.5800000000000001,"15min":2.5800000000000001},"last":0.52700000000000002}]},{"osd":2,"last update":"Thu Oct 19 12:59:33 2023","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.42499999999999999,"15min":0.42099999999999999},"min":{"1min":0.309,"5min":0.21299999999999999,"15min":0.20100000000000001},"max":{"1min":0.53300000000000003,"5min":0.60499999999999998,"15min":0.69999999999999996},"last":0.502},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.47099999999999997,"15min":0.45300000000000001},"min":{"1min":0.34999999999999998,"5min":0.313,"15min":0.23699999999999999},"max":{"1min":0.625,"5min":2.6230000000000002,"15min":2.6230000000000002},"last":0.57999999999999996}]},{"osd":3,"last update":"Thu Oct 19 12:59:33 2023","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.51600000000000001,"15min":0.48599999999999999},"min":{"1min":0.38900000000000001,"5min":0.307,"15min":0.26200000000000001},"max":{"1min":0.621,"5min":3.9740000000000002,"15min":3.9740000000000002},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.43099999999999999,"15min":0.42299999999999999},"min":{"1min":0.35999999999999999,"5min":0.27700000000000002,"15min":0.27000000000000002},"max":{"1min":0.59299999999999997,"5min":0.63200000000000001,"15min":0.80500000000000005},"last":0.51400000000000001}]},{"osd":4,"last update":"Thu Oct 19 12:59:33 2023","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.65600000000000003,"15min":0.65200000000000002},"min":{"1min":0.501,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.85699999999999998,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.66600000000000004,"15min":0.64800000000000002},"min":{"1min":0.47399999999999998,"5min":0.42599999999999999,"15min":0.314},"max":{"1min":0.85399999999999998,"5min":1.0860000000000001,"15min":1.1910000000000001},"last":0.76300000000000001}]},{"osd":5,"last update":"Thu Oct 19 12:59:33 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.65700000000000003,"15min":0.65300000000000002},"min":{"1min":0.499,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.83599999999999997,"5min":1.1200000000000001,"15min":1.1200000000000001},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.69599999999999995,"15min":0.67300000000000004},"min":{"1min":0.54300000000000004,"5min":0.42299999999999999,"15min":0.38700000000000001},"max":{"1min":0.81799999999999995,"5min":2.7120000000000002,"15min":2.7120000000000002},"last":0.77900000000000003}]},{"osd":6,"last update":"Thu Oct 19 12:59:33 2023","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.65800000000000003,"15min":0.65700000000000003},"min":{"1min":0.53800000000000003,"5min":0.36799999999999999,"15min":0.36399999999999999},"max":{"1min":0.91300000000000003,"5min":1.0069999999999999,"15min":1.077},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67400000000000004,"15min":0.67800000000000005},"min":{"1min":0.55800000000000005,"5min":0.376,"15min":0.376},"max":{"1min":0.77500000000000002,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.72099999999999997}]},{"osd":7,"last update":"Thu Oct 19 12:59:33 2023","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.69799999999999995,"15min":0.67900000000000005},"min":{"1min":0.54300000000000004,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.84999999999999998,"5min":2.786,"15min":2.786},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.68200000000000005,"15min":0.67900000000000005},"min":{"1min":0.47499999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":1.0489999999999999},"last":0.79500000000000004}]}]},{"osd":0,"up_from":13,"seq":55834575047,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36952,"kb_used_data":1608,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334888,"statfs":{"total":96636764160,"available":96598925312,"internally_reserved":0,"allocated":1646592,"data_stored":1118621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Oct 19 12:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.40300000000000002,"15min":0.39100000000000001},"min":{"1min":0.26500000000000001,"5min":0.182,"15min":0.182},"max":{"1min":0.58899999999999997,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.434,"15min":0.432},"min":{"1min":0.34599999999999997,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.746},"last":0.38200000000000001}]},{"osd":2,"last update":"Thu Oct 19 12:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.46700000000000003,"15min":0.45500000000000002},"min":{"1min":0.33200000000000002,"5min":0.29499999999999998,"15min":0.25800000000000001},"max":{"1min":0.58599999999999997,"5min":0.752,"15min":0.76300000000000001},"last":0.309},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.437,"15min":0.434},"min":{"1min":0.26000000000000001,"5min":0.216,"15min":0.20699999999999999},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.76800000000000002},"last":0.432}]},{"osd":3,"last update":"Thu Oct 19 12:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.42199999999999999,"15min":0.40899999999999997},"min":{"1min":0.34000000000000002,"5min":0.23400000000000001,"15min":0.22800000000000001},"max":{"1min":0.53500000000000003,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.505,"5min":0.504,"15min":0.49099999999999999},"min":{"1min":0.33600000000000002,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.66300000000000003,"5min":1.4990000000000001,"15min":1.4990000000000001},"last":0.34799999999999998}]},{"osd":4,"last update":"Thu Oct 19 12:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.63100000000000001,"15min":0.63600000000000001},"min":{"1min":0.46600000000000003,"5min":0.40000000000000002,"15min":0.34999999999999998},"max":{"1min":0.749,"5min":1.361,"15min":1.361},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.65700000000000003,"15min":0.65000000000000002},"min":{"1min":0.52900000000000003,"5min":0.41899999999999998,"15min":0.35999999999999999},"max":{"1min":0.871,"5min":0.98999999999999999,"15min":0.998},"last":0.57699999999999996}]},{"osd":5,"last update":"Thu Oct 19 12:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.68999999999999995,"15min":0.68000000000000005},"min":{"1min":0.50700000000000001,"5min":0.378,"15min":0.378},"max":{"1min":0.94599999999999995,"5min":1.325,"15min":1.325},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.69299999999999995,"15min":0.68600000000000005},"min":{"1min":0.45900000000000002,"5min":0.36299999999999999,"15min":0.318},"max":{"1min":1.02,"5min":1.024,"15min":1.056},"last":0.625}]},{"osd":6,"last update":"Thu Oct 19 12:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.67200000000000004,"15min":0.66400000000000003},"min":{"1min":0.59999999999999998,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.873,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.67600000000000005,"15min":0.66600000000000004},"min":{"1min":0.57199999999999995,"5min":0.41999999999999998,"15min":0.38900000000000001},"max":{"1min":0.92000000000000004,"5min":0.995,"15min":1.109},"last":0.65600000000000003}]},{"osd":7,"last update":"Thu Oct 19 12:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.68700000000000006,"15min":0.68400000000000005},"min":{"1min":0.53500000000000003,"5min":0.41199999999999998,"15min":0.41099999999999998},"max":{"1min":0.89900000000000002,"5min":0.98799999999999999,"15min":1.0089999999999999},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.67300000000000004,"15min":0.66300000000000003},"min":{"1min":0.58799999999999997,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.90400000000000003,"5min":1.389,"15min":1.389},"last":0.67100000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-19T13:00:24.929 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.3d is not active+clean 2023-10-19T13:00:24.929 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.3d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T13:00:20.171814+0000', 'last_change': '2023-10-19T13:00:20.171814+0000', 'last_active': '2023-10-19T13:00:20.171814+0000', 'last_peered': '2023-10-19T13:00:20.171814+0000', 'last_clean': '2023-10-19T13:00:20.171814+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T13:00:20.171814+0000', 'last_undegraded': '2023-10-19T13:00:20.171814+0000', 'last_fullsized': '2023-10-19T13:00:20.171814+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_clean_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T13:00:24.930 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.3c is not active+clean 2023-10-19T13:00:24.930 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.3c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T13:00:20.171814+0000', 'last_change': '2023-10-19T13:00:20.171814+0000', 'last_active': '2023-10-19T13:00:20.171814+0000', 'last_peered': '2023-10-19T13:00:20.171814+0000', 'last_clean': '2023-10-19T13:00:20.171814+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T13:00:20.171814+0000', 'last_undegraded': '2023-10-19T13:00:20.171814+0000', 'last_fullsized': '2023-10-19T13:00:20.171814+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_clean_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T13:00:24.930 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.3b is not active+clean 2023-10-19T13:00:24.931 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.3b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T13:00:20.171814+0000', 'last_change': '2023-10-19T13:00:20.171814+0000', 'last_active': '2023-10-19T13:00:20.171814+0000', 'last_peered': '2023-10-19T13:00:20.171814+0000', 'last_clean': '2023-10-19T13:00:20.171814+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T13:00:20.171814+0000', 'last_undegraded': '2023-10-19T13:00:20.171814+0000', 'last_fullsized': '2023-10-19T13:00:20.171814+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_clean_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T13:00:24.931 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.38 is not active+clean 2023-10-19T13:00:24.932 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.38', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T13:00:20.171814+0000', 'last_change': '2023-10-19T13:00:20.171814+0000', 'last_active': '2023-10-19T13:00:20.171814+0000', 'last_peered': '2023-10-19T13:00:20.171814+0000', 'last_clean': '2023-10-19T13:00:20.171814+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T13:00:20.171814+0000', 'last_undegraded': '2023-10-19T13:00:20.171814+0000', 'last_fullsized': '2023-10-19T13:00:20.171814+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_clean_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T13:00:24.932 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.37 is not active+clean 2023-10-19T13:00:24.933 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.37', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T13:00:20.171814+0000', 'last_change': '2023-10-19T13:00:20.171814+0000', 'last_active': '2023-10-19T13:00:20.171814+0000', 'last_peered': '2023-10-19T13:00:20.171814+0000', 'last_clean': '2023-10-19T13:00:20.171814+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T13:00:20.171814+0000', 'last_undegraded': '2023-10-19T13:00:20.171814+0000', 'last_fullsized': '2023-10-19T13:00:20.171814+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_clean_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T13:00:24.933 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.36 is not active+clean 2023-10-19T13:00:24.934 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.36', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T13:00:20.171814+0000', 'last_change': '2023-10-19T13:00:20.171814+0000', 'last_active': '2023-10-19T13:00:20.171814+0000', 'last_peered': '2023-10-19T13:00:20.171814+0000', 'last_clean': '2023-10-19T13:00:20.171814+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T13:00:20.171814+0000', 'last_undegraded': '2023-10-19T13:00:20.171814+0000', 'last_fullsized': '2023-10-19T13:00:20.171814+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_clean_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T13:00:24.934 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.35 is not active+clean 2023-10-19T13:00:24.934 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.35', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T13:00:20.171814+0000', 'last_change': '2023-10-19T13:00:20.171814+0000', 'last_active': '2023-10-19T13:00:20.171814+0000', 'last_peered': '2023-10-19T13:00:20.171814+0000', 'last_clean': '2023-10-19T13:00:20.171814+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T13:00:20.171814+0000', 'last_undegraded': '2023-10-19T13:00:20.171814+0000', 'last_fullsized': '2023-10-19T13:00:20.171814+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_clean_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T13:00:24.935 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.34 is not active+clean 2023-10-19T13:00:24.935 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.34', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T13:00:20.171814+0000', 'last_change': '2023-10-19T13:00:20.171814+0000', 'last_active': '2023-10-19T13:00:20.171814+0000', 'last_peered': '2023-10-19T13:00:20.171814+0000', 'last_clean': '2023-10-19T13:00:20.171814+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T13:00:20.171814+0000', 'last_undegraded': '2023-10-19T13:00:20.171814+0000', 'last_fullsized': '2023-10-19T13:00:20.171814+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_clean_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T13:00:24.936 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.33 is not active+clean 2023-10-19T13:00:24.936 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.33', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T13:00:20.171814+0000', 'last_change': '2023-10-19T13:00:20.171814+0000', 'last_active': '2023-10-19T13:00:20.171814+0000', 'last_peered': '2023-10-19T13:00:20.171814+0000', 'last_clean': '2023-10-19T13:00:20.171814+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T13:00:20.171814+0000', 'last_undegraded': '2023-10-19T13:00:20.171814+0000', 'last_fullsized': '2023-10-19T13:00:20.171814+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_clean_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T13:00:24.937 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.32 is not active+clean 2023-10-19T13:00:24.937 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.32', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T13:00:20.171814+0000', 'last_change': '2023-10-19T13:00:20.171814+0000', 'last_active': '2023-10-19T13:00:20.171814+0000', 'last_peered': '2023-10-19T13:00:20.171814+0000', 'last_clean': '2023-10-19T13:00:20.171814+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T13:00:20.171814+0000', 'last_undegraded': '2023-10-19T13:00:20.171814+0000', 'last_fullsized': '2023-10-19T13:00:20.171814+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_clean_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T13:00:24.937 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.31 is not active+clean 2023-10-19T13:00:24.938 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.31', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T13:00:20.171814+0000', 'last_change': '2023-10-19T13:00:20.171814+0000', 'last_active': '2023-10-19T13:00:20.171814+0000', 'last_peered': '2023-10-19T13:00:20.171814+0000', 'last_clean': '2023-10-19T13:00:20.171814+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T13:00:20.171814+0000', 'last_undegraded': '2023-10-19T13:00:20.171814+0000', 'last_fullsized': '2023-10-19T13:00:20.171814+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_clean_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T13:00:24.938 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.2e is not active+clean 2023-10-19T13:00:24.939 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.2e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T13:00:20.171814+0000', 'last_change': '2023-10-19T13:00:20.171814+0000', 'last_active': '2023-10-19T13:00:20.171814+0000', 'last_peered': '2023-10-19T13:00:20.171814+0000', 'last_clean': '2023-10-19T13:00:20.171814+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T13:00:20.171814+0000', 'last_undegraded': '2023-10-19T13:00:20.171814+0000', 'last_fullsized': '2023-10-19T13:00:20.171814+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_clean_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T13:00:24.939 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.2d is not active+clean 2023-10-19T13:00:24.940 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.2d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T13:00:20.171814+0000', 'last_change': '2023-10-19T13:00:20.171814+0000', 'last_active': '2023-10-19T13:00:20.171814+0000', 'last_peered': '2023-10-19T13:00:20.171814+0000', 'last_clean': '2023-10-19T13:00:20.171814+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T13:00:20.171814+0000', 'last_undegraded': '2023-10-19T13:00:20.171814+0000', 'last_fullsized': '2023-10-19T13:00:20.171814+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_clean_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T13:00:24.940 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.2c is not active+clean 2023-10-19T13:00:24.941 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.2c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T13:00:20.171814+0000', 'last_change': '2023-10-19T13:00:20.171814+0000', 'last_active': '2023-10-19T13:00:20.171814+0000', 'last_peered': '2023-10-19T13:00:20.171814+0000', 'last_clean': '2023-10-19T13:00:20.171814+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T13:00:20.171814+0000', 'last_undegraded': '2023-10-19T13:00:20.171814+0000', 'last_fullsized': '2023-10-19T13:00:20.171814+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_clean_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T13:00:24.941 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.2b is not active+clean 2023-10-19T13:00:24.942 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.2b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T13:00:20.171814+0000', 'last_change': '2023-10-19T13:00:20.171814+0000', 'last_active': '2023-10-19T13:00:20.171814+0000', 'last_peered': '2023-10-19T13:00:20.171814+0000', 'last_clean': '2023-10-19T13:00:20.171814+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T13:00:20.171814+0000', 'last_undegraded': '2023-10-19T13:00:20.171814+0000', 'last_fullsized': '2023-10-19T13:00:20.171814+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_clean_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T13:00:24.942 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.2a is not active+clean 2023-10-19T13:00:24.942 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.2a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T13:00:20.171814+0000', 'last_change': '2023-10-19T13:00:20.171814+0000', 'last_active': '2023-10-19T13:00:20.171814+0000', 'last_peered': '2023-10-19T13:00:20.171814+0000', 'last_clean': '2023-10-19T13:00:20.171814+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T13:00:20.171814+0000', 'last_undegraded': '2023-10-19T13:00:20.171814+0000', 'last_fullsized': '2023-10-19T13:00:20.171814+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_clean_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T13:00:24.943 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.29 is not active+clean 2023-10-19T13:00:24.943 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.29', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T13:00:20.171814+0000', 'last_change': '2023-10-19T13:00:20.171814+0000', 'last_active': '2023-10-19T13:00:20.171814+0000', 'last_peered': '2023-10-19T13:00:20.171814+0000', 'last_clean': '2023-10-19T13:00:20.171814+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T13:00:20.171814+0000', 'last_undegraded': '2023-10-19T13:00:20.171814+0000', 'last_fullsized': '2023-10-19T13:00:20.171814+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_clean_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T13:00:24.944 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.28 is not active+clean 2023-10-19T13:00:24.944 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.28', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T13:00:20.171814+0000', 'last_change': '2023-10-19T13:00:20.171814+0000', 'last_active': '2023-10-19T13:00:20.171814+0000', 'last_peered': '2023-10-19T13:00:20.171814+0000', 'last_clean': '2023-10-19T13:00:20.171814+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T13:00:20.171814+0000', 'last_undegraded': '2023-10-19T13:00:20.171814+0000', 'last_fullsized': '2023-10-19T13:00:20.171814+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_clean_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T13:00:24.944 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.27 is not active+clean 2023-10-19T13:00:24.945 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.27', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T13:00:20.171814+0000', 'last_change': '2023-10-19T13:00:20.171814+0000', 'last_active': '2023-10-19T13:00:20.171814+0000', 'last_peered': '2023-10-19T13:00:20.171814+0000', 'last_clean': '2023-10-19T13:00:20.171814+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T13:00:20.171814+0000', 'last_undegraded': '2023-10-19T13:00:20.171814+0000', 'last_fullsized': '2023-10-19T13:00:20.171814+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_clean_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T13:00:24.945 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.26 is not active+clean 2023-10-19T13:00:24.945 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.26', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T13:00:20.171814+0000', 'last_change': '2023-10-19T13:00:20.171814+0000', 'last_active': '2023-10-19T13:00:20.171814+0000', 'last_peered': '2023-10-19T13:00:20.171814+0000', 'last_clean': '2023-10-19T13:00:20.171814+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T13:00:20.171814+0000', 'last_undegraded': '2023-10-19T13:00:20.171814+0000', 'last_fullsized': '2023-10-19T13:00:20.171814+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_clean_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T13:00:24.946 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.25 is not active+clean 2023-10-19T13:00:24.946 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.25', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T13:00:20.171814+0000', 'last_change': '2023-10-19T13:00:20.171814+0000', 'last_active': '2023-10-19T13:00:20.171814+0000', 'last_peered': '2023-10-19T13:00:20.171814+0000', 'last_clean': '2023-10-19T13:00:20.171814+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T13:00:20.171814+0000', 'last_undegraded': '2023-10-19T13:00:20.171814+0000', 'last_fullsized': '2023-10-19T13:00:20.171814+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_clean_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T13:00:24.947 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.23 is not active+clean 2023-10-19T13:00:24.947 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.23', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T13:00:20.171814+0000', 'last_change': '2023-10-19T13:00:20.171814+0000', 'last_active': '2023-10-19T13:00:20.171814+0000', 'last_peered': '2023-10-19T13:00:20.171814+0000', 'last_clean': '2023-10-19T13:00:20.171814+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T13:00:20.171814+0000', 'last_undegraded': '2023-10-19T13:00:20.171814+0000', 'last_fullsized': '2023-10-19T13:00:20.171814+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_clean_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T13:00:24.948 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.22 is not active+clean 2023-10-19T13:00:24.948 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.22', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T13:00:20.171814+0000', 'last_change': '2023-10-19T13:00:20.171814+0000', 'last_active': '2023-10-19T13:00:20.171814+0000', 'last_peered': '2023-10-19T13:00:20.171814+0000', 'last_clean': '2023-10-19T13:00:20.171814+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T13:00:20.171814+0000', 'last_undegraded': '2023-10-19T13:00:20.171814+0000', 'last_fullsized': '2023-10-19T13:00:20.171814+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_clean_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T13:00:24.948 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.21 is not active+clean 2023-10-19T13:00:24.949 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.21', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T13:00:20.171814+0000', 'last_change': '2023-10-19T13:00:20.171814+0000', 'last_active': '2023-10-19T13:00:20.171814+0000', 'last_peered': '2023-10-19T13:00:20.171814+0000', 'last_clean': '2023-10-19T13:00:20.171814+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T13:00:20.171814+0000', 'last_undegraded': '2023-10-19T13:00:20.171814+0000', 'last_fullsized': '2023-10-19T13:00:20.171814+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_clean_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T13:00:24.949 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.1c is not active+clean 2023-10-19T13:00:24.951 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.1c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T13:00:20.171814+0000', 'last_change': '2023-10-19T13:00:20.171814+0000', 'last_active': '2023-10-19T13:00:20.171814+0000', 'last_peered': '2023-10-19T13:00:20.171814+0000', 'last_clean': '2023-10-19T13:00:20.171814+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T13:00:20.171814+0000', 'last_undegraded': '2023-10-19T13:00:20.171814+0000', 'last_fullsized': '2023-10-19T13:00:20.171814+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_clean_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T13:00:24.951 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.1d is not active+clean 2023-10-19T13:00:24.951 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.1d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T13:00:20.171814+0000', 'last_change': '2023-10-19T13:00:20.171814+0000', 'last_active': '2023-10-19T13:00:20.171814+0000', 'last_peered': '2023-10-19T13:00:20.171814+0000', 'last_clean': '2023-10-19T13:00:20.171814+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T13:00:20.171814+0000', 'last_undegraded': '2023-10-19T13:00:20.171814+0000', 'last_fullsized': '2023-10-19T13:00:20.171814+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_clean_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T13:00:24.952 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.1f is not active+clean 2023-10-19T13:00:24.952 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.1f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T13:00:20.171814+0000', 'last_change': '2023-10-19T13:00:20.171814+0000', 'last_active': '2023-10-19T13:00:20.171814+0000', 'last_peered': '2023-10-19T13:00:20.171814+0000', 'last_clean': '2023-10-19T13:00:20.171814+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T13:00:20.171814+0000', 'last_undegraded': '2023-10-19T13:00:20.171814+0000', 'last_fullsized': '2023-10-19T13:00:20.171814+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_clean_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T13:00:24.953 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.1b is not active+clean 2023-10-19T13:00:24.953 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.1b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T13:00:19.169402+0000', 'last_change': '2023-10-19T13:00:19.169402+0000', 'last_active': '2023-10-19T13:00:19.169402+0000', 'last_peered': '2023-10-19T13:00:19.169402+0000', 'last_clean': '2023-10-19T13:00:19.169402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T13:00:19.169402+0000', 'last_undegraded': '2023-10-19T13:00:19.169402+0000', 'last_fullsized': '2023-10-19T13:00:19.169402+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T13:00:19.169402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T13:00:19.169402+0000', 'last_clean_scrub_stamp': '2023-10-19T13:00:19.169402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T13:00:24.954 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.1a is not active+clean 2023-10-19T13:00:24.954 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.1a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T13:00:19.169402+0000', 'last_change': '2023-10-19T13:00:19.169402+0000', 'last_active': '2023-10-19T13:00:19.169402+0000', 'last_peered': '2023-10-19T13:00:19.169402+0000', 'last_clean': '2023-10-19T13:00:19.169402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T13:00:19.169402+0000', 'last_undegraded': '2023-10-19T13:00:19.169402+0000', 'last_fullsized': '2023-10-19T13:00:19.169402+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T13:00:19.169402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T13:00:19.169402+0000', 'last_clean_scrub_stamp': '2023-10-19T13:00:19.169402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T13:00:24.955 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.1a is not active+clean 2023-10-19T13:00:24.955 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.1a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T13:00:20.171814+0000', 'last_change': '2023-10-19T13:00:20.171814+0000', 'last_active': '2023-10-19T13:00:20.171814+0000', 'last_peered': '2023-10-19T13:00:20.171814+0000', 'last_clean': '2023-10-19T13:00:20.171814+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T13:00:20.171814+0000', 'last_undegraded': '2023-10-19T13:00:20.171814+0000', 'last_fullsized': '2023-10-19T13:00:20.171814+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_clean_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T13:00:24.956 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.19 is not active+clean 2023-10-19T13:00:24.956 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.19', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T13:00:19.169402+0000', 'last_change': '2023-10-19T13:00:19.169402+0000', 'last_active': '2023-10-19T13:00:19.169402+0000', 'last_peered': '2023-10-19T13:00:19.169402+0000', 'last_clean': '2023-10-19T13:00:19.169402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T13:00:19.169402+0000', 'last_undegraded': '2023-10-19T13:00:19.169402+0000', 'last_fullsized': '2023-10-19T13:00:19.169402+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T13:00:19.169402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T13:00:19.169402+0000', 'last_clean_scrub_stamp': '2023-10-19T13:00:19.169402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T13:00:24.956 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.1b is not active+clean 2023-10-19T13:00:24.957 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.1b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T13:00:20.171814+0000', 'last_change': '2023-10-19T13:00:20.171814+0000', 'last_active': '2023-10-19T13:00:20.171814+0000', 'last_peered': '2023-10-19T13:00:20.171814+0000', 'last_clean': '2023-10-19T13:00:20.171814+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T13:00:20.171814+0000', 'last_undegraded': '2023-10-19T13:00:20.171814+0000', 'last_fullsized': '2023-10-19T13:00:20.171814+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_clean_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T13:00:24.957 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.18 is not active+clean 2023-10-19T13:00:24.958 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.18', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T13:00:19.169402+0000', 'last_change': '2023-10-19T13:00:19.169402+0000', 'last_active': '2023-10-19T13:00:19.169402+0000', 'last_peered': '2023-10-19T13:00:19.169402+0000', 'last_clean': '2023-10-19T13:00:19.169402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T13:00:19.169402+0000', 'last_undegraded': '2023-10-19T13:00:19.169402+0000', 'last_fullsized': '2023-10-19T13:00:19.169402+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T13:00:19.169402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T13:00:19.169402+0000', 'last_clean_scrub_stamp': '2023-10-19T13:00:19.169402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T13:00:24.958 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.17 is not active+clean 2023-10-19T13:00:24.958 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.17', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T13:00:19.169402+0000', 'last_change': '2023-10-19T13:00:19.169402+0000', 'last_active': '2023-10-19T13:00:19.169402+0000', 'last_peered': '2023-10-19T13:00:19.169402+0000', 'last_clean': '2023-10-19T13:00:19.169402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T13:00:19.169402+0000', 'last_undegraded': '2023-10-19T13:00:19.169402+0000', 'last_fullsized': '2023-10-19T13:00:19.169402+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T13:00:19.169402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T13:00:19.169402+0000', 'last_clean_scrub_stamp': '2023-10-19T13:00:19.169402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T13:00:24.959 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.15 is not active+clean 2023-10-19T13:00:24.959 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.15', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T13:00:20.171814+0000', 'last_change': '2023-10-19T13:00:20.171814+0000', 'last_active': '2023-10-19T13:00:20.171814+0000', 'last_peered': '2023-10-19T13:00:20.171814+0000', 'last_clean': '2023-10-19T13:00:20.171814+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T13:00:20.171814+0000', 'last_undegraded': '2023-10-19T13:00:20.171814+0000', 'last_fullsized': '2023-10-19T13:00:20.171814+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_clean_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T13:00:24.960 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.16 is not active+clean 2023-10-19T13:00:24.960 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.16', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T13:00:19.169402+0000', 'last_change': '2023-10-19T13:00:19.169402+0000', 'last_active': '2023-10-19T13:00:19.169402+0000', 'last_peered': '2023-10-19T13:00:19.169402+0000', 'last_clean': '2023-10-19T13:00:19.169402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T13:00:19.169402+0000', 'last_undegraded': '2023-10-19T13:00:19.169402+0000', 'last_fullsized': '2023-10-19T13:00:19.169402+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T13:00:19.169402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T13:00:19.169402+0000', 'last_clean_scrub_stamp': '2023-10-19T13:00:19.169402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T13:00:24.960 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.16 is not active+clean 2023-10-19T13:00:24.961 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.16', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T13:00:20.171814+0000', 'last_change': '2023-10-19T13:00:20.171814+0000', 'last_active': '2023-10-19T13:00:20.171814+0000', 'last_peered': '2023-10-19T13:00:20.171814+0000', 'last_clean': '2023-10-19T13:00:20.171814+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T13:00:20.171814+0000', 'last_undegraded': '2023-10-19T13:00:20.171814+0000', 'last_fullsized': '2023-10-19T13:00:20.171814+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_clean_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T13:00:24.961 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.15 is not active+clean 2023-10-19T13:00:24.962 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.15', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T13:00:19.169402+0000', 'last_change': '2023-10-19T13:00:19.169402+0000', 'last_active': '2023-10-19T13:00:19.169402+0000', 'last_peered': '2023-10-19T13:00:19.169402+0000', 'last_clean': '2023-10-19T13:00:19.169402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T13:00:19.169402+0000', 'last_undegraded': '2023-10-19T13:00:19.169402+0000', 'last_fullsized': '2023-10-19T13:00:19.169402+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T13:00:19.169402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T13:00:19.169402+0000', 'last_clean_scrub_stamp': '2023-10-19T13:00:19.169402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T13:00:24.962 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.17 is not active+clean 2023-10-19T13:00:24.962 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.17', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T13:00:20.171814+0000', 'last_change': '2023-10-19T13:00:20.171814+0000', 'last_active': '2023-10-19T13:00:20.171814+0000', 'last_peered': '2023-10-19T13:00:20.171814+0000', 'last_clean': '2023-10-19T13:00:20.171814+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T13:00:20.171814+0000', 'last_undegraded': '2023-10-19T13:00:20.171814+0000', 'last_fullsized': '2023-10-19T13:00:20.171814+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_clean_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T13:00:24.963 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.10 is not active+clean 2023-10-19T13:00:24.963 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.10', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T13:00:20.171814+0000', 'last_change': '2023-10-19T13:00:20.171814+0000', 'last_active': '2023-10-19T13:00:20.171814+0000', 'last_peered': '2023-10-19T13:00:20.171814+0000', 'last_clean': '2023-10-19T13:00:20.171814+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T13:00:20.171814+0000', 'last_undegraded': '2023-10-19T13:00:20.171814+0000', 'last_fullsized': '2023-10-19T13:00:20.171814+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_clean_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T13:00:24.963 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.13 is not active+clean 2023-10-19T13:00:24.964 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.13', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T13:00:19.169402+0000', 'last_change': '2023-10-19T13:00:19.169402+0000', 'last_active': '2023-10-19T13:00:19.169402+0000', 'last_peered': '2023-10-19T13:00:19.169402+0000', 'last_clean': '2023-10-19T13:00:19.169402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T13:00:19.169402+0000', 'last_undegraded': '2023-10-19T13:00:19.169402+0000', 'last_fullsized': '2023-10-19T13:00:19.169402+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T13:00:19.169402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T13:00:19.169402+0000', 'last_clean_scrub_stamp': '2023-10-19T13:00:19.169402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T13:00:24.964 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.11 is not active+clean 2023-10-19T13:00:24.964 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.11', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T13:00:20.171814+0000', 'last_change': '2023-10-19T13:00:20.171814+0000', 'last_active': '2023-10-19T13:00:20.171814+0000', 'last_peered': '2023-10-19T13:00:20.171814+0000', 'last_clean': '2023-10-19T13:00:20.171814+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T13:00:20.171814+0000', 'last_undegraded': '2023-10-19T13:00:20.171814+0000', 'last_fullsized': '2023-10-19T13:00:20.171814+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_clean_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T13:00:24.965 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.12 is not active+clean 2023-10-19T13:00:24.965 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.12', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T13:00:19.169402+0000', 'last_change': '2023-10-19T13:00:19.169402+0000', 'last_active': '2023-10-19T13:00:19.169402+0000', 'last_peered': '2023-10-19T13:00:19.169402+0000', 'last_clean': '2023-10-19T13:00:19.169402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T13:00:19.169402+0000', 'last_undegraded': '2023-10-19T13:00:19.169402+0000', 'last_fullsized': '2023-10-19T13:00:19.169402+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T13:00:19.169402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T13:00:19.169402+0000', 'last_clean_scrub_stamp': '2023-10-19T13:00:19.169402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T13:00:24.966 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.12 is not active+clean 2023-10-19T13:00:24.966 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.12', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T13:00:20.171814+0000', 'last_change': '2023-10-19T13:00:20.171814+0000', 'last_active': '2023-10-19T13:00:20.171814+0000', 'last_peered': '2023-10-19T13:00:20.171814+0000', 'last_clean': '2023-10-19T13:00:20.171814+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T13:00:20.171814+0000', 'last_undegraded': '2023-10-19T13:00:20.171814+0000', 'last_fullsized': '2023-10-19T13:00:20.171814+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_clean_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T13:00:24.967 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.11 is not active+clean 2023-10-19T13:00:24.967 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.11', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T13:00:19.169402+0000', 'last_change': '2023-10-19T13:00:19.169402+0000', 'last_active': '2023-10-19T13:00:19.169402+0000', 'last_peered': '2023-10-19T13:00:19.169402+0000', 'last_clean': '2023-10-19T13:00:19.169402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T13:00:19.169402+0000', 'last_undegraded': '2023-10-19T13:00:19.169402+0000', 'last_fullsized': '2023-10-19T13:00:19.169402+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T13:00:19.169402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T13:00:19.169402+0000', 'last_clean_scrub_stamp': '2023-10-19T13:00:19.169402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T13:00:24.967 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.10 is not active+clean 2023-10-19T13:00:24.968 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.10', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T13:00:19.169402+0000', 'last_change': '2023-10-19T13:00:19.169402+0000', 'last_active': '2023-10-19T13:00:19.169402+0000', 'last_peered': '2023-10-19T13:00:19.169402+0000', 'last_clean': '2023-10-19T13:00:19.169402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T13:00:19.169402+0000', 'last_undegraded': '2023-10-19T13:00:19.169402+0000', 'last_fullsized': '2023-10-19T13:00:19.169402+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T13:00:19.169402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T13:00:19.169402+0000', 'last_clean_scrub_stamp': '2023-10-19T13:00:19.169402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T13:00:24.968 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.c is not active+clean 2023-10-19T13:00:24.968 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T13:00:20.171814+0000', 'last_change': '2023-10-19T13:00:20.171814+0000', 'last_active': '2023-10-19T13:00:20.171814+0000', 'last_peered': '2023-10-19T13:00:20.171814+0000', 'last_clean': '2023-10-19T13:00:20.171814+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T13:00:20.171814+0000', 'last_undegraded': '2023-10-19T13:00:20.171814+0000', 'last_fullsized': '2023-10-19T13:00:20.171814+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_clean_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T13:00:24.969 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.f is not active+clean 2023-10-19T13:00:24.969 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T13:00:19.169402+0000', 'last_change': '2023-10-19T13:00:19.169402+0000', 'last_active': '2023-10-19T13:00:19.169402+0000', 'last_peered': '2023-10-19T13:00:19.169402+0000', 'last_clean': '2023-10-19T13:00:19.169402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T13:00:19.169402+0000', 'last_undegraded': '2023-10-19T13:00:19.169402+0000', 'last_fullsized': '2023-10-19T13:00:19.169402+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T13:00:19.169402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T13:00:19.169402+0000', 'last_clean_scrub_stamp': '2023-10-19T13:00:19.169402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T13:00:24.970 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.d is not active+clean 2023-10-19T13:00:24.970 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T13:00:20.171814+0000', 'last_change': '2023-10-19T13:00:20.171814+0000', 'last_active': '2023-10-19T13:00:20.171814+0000', 'last_peered': '2023-10-19T13:00:20.171814+0000', 'last_clean': '2023-10-19T13:00:20.171814+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T13:00:20.171814+0000', 'last_undegraded': '2023-10-19T13:00:20.171814+0000', 'last_fullsized': '2023-10-19T13:00:20.171814+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_clean_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T13:00:24.970 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.e is not active+clean 2023-10-19T13:00:24.971 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T13:00:19.169402+0000', 'last_change': '2023-10-19T13:00:19.169402+0000', 'last_active': '2023-10-19T13:00:19.169402+0000', 'last_peered': '2023-10-19T13:00:19.169402+0000', 'last_clean': '2023-10-19T13:00:19.169402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T13:00:19.169402+0000', 'last_undegraded': '2023-10-19T13:00:19.169402+0000', 'last_fullsized': '2023-10-19T13:00:19.169402+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T13:00:19.169402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T13:00:19.169402+0000', 'last_clean_scrub_stamp': '2023-10-19T13:00:19.169402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T13:00:24.971 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.e is not active+clean 2023-10-19T13:00:24.971 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T13:00:20.171814+0000', 'last_change': '2023-10-19T13:00:20.171814+0000', 'last_active': '2023-10-19T13:00:20.171814+0000', 'last_peered': '2023-10-19T13:00:20.171814+0000', 'last_clean': '2023-10-19T13:00:20.171814+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T13:00:20.171814+0000', 'last_undegraded': '2023-10-19T13:00:20.171814+0000', 'last_fullsized': '2023-10-19T13:00:20.171814+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_clean_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T13:00:24.972 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.d is not active+clean 2023-10-19T13:00:24.972 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T13:00:19.169402+0000', 'last_change': '2023-10-19T13:00:19.169402+0000', 'last_active': '2023-10-19T13:00:19.169402+0000', 'last_peered': '2023-10-19T13:00:19.169402+0000', 'last_clean': '2023-10-19T13:00:19.169402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T13:00:19.169402+0000', 'last_undegraded': '2023-10-19T13:00:19.169402+0000', 'last_fullsized': '2023-10-19T13:00:19.169402+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T13:00:19.169402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T13:00:19.169402+0000', 'last_clean_scrub_stamp': '2023-10-19T13:00:19.169402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T13:00:24.973 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.f is not active+clean 2023-10-19T13:00:24.973 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T13:00:20.171814+0000', 'last_change': '2023-10-19T13:00:20.171814+0000', 'last_active': '2023-10-19T13:00:20.171814+0000', 'last_peered': '2023-10-19T13:00:20.171814+0000', 'last_clean': '2023-10-19T13:00:20.171814+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T13:00:20.171814+0000', 'last_undegraded': '2023-10-19T13:00:20.171814+0000', 'last_fullsized': '2023-10-19T13:00:20.171814+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_clean_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T13:00:24.973 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.c is not active+clean 2023-10-19T13:00:24.974 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T13:00:19.169402+0000', 'last_change': '2023-10-19T13:00:19.169402+0000', 'last_active': '2023-10-19T13:00:19.169402+0000', 'last_peered': '2023-10-19T13:00:19.169402+0000', 'last_clean': '2023-10-19T13:00:19.169402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T13:00:19.169402+0000', 'last_undegraded': '2023-10-19T13:00:19.169402+0000', 'last_fullsized': '2023-10-19T13:00:19.169402+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T13:00:19.169402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T13:00:19.169402+0000', 'last_clean_scrub_stamp': '2023-10-19T13:00:19.169402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T13:00:24.974 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.9 is not active+clean 2023-10-19T13:00:24.975 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.9', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T13:00:20.171814+0000', 'last_change': '2023-10-19T13:00:20.171814+0000', 'last_active': '2023-10-19T13:00:20.171814+0000', 'last_peered': '2023-10-19T13:00:20.171814+0000', 'last_clean': '2023-10-19T13:00:20.171814+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T13:00:20.171814+0000', 'last_undegraded': '2023-10-19T13:00:20.171814+0000', 'last_fullsized': '2023-10-19T13:00:20.171814+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_clean_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T13:00:24.975 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.a is not active+clean 2023-10-19T13:00:24.975 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T13:00:19.169402+0000', 'last_change': '2023-10-19T13:00:19.169402+0000', 'last_active': '2023-10-19T13:00:19.169402+0000', 'last_peered': '2023-10-19T13:00:19.169402+0000', 'last_clean': '2023-10-19T13:00:19.169402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T13:00:19.169402+0000', 'last_undegraded': '2023-10-19T13:00:19.169402+0000', 'last_fullsized': '2023-10-19T13:00:19.169402+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T13:00:19.169402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T13:00:19.169402+0000', 'last_clean_scrub_stamp': '2023-10-19T13:00:19.169402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T13:00:24.976 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.a is not active+clean 2023-10-19T13:00:24.976 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T13:00:20.171814+0000', 'last_change': '2023-10-19T13:00:20.171814+0000', 'last_active': '2023-10-19T13:00:20.171814+0000', 'last_peered': '2023-10-19T13:00:20.171814+0000', 'last_clean': '2023-10-19T13:00:20.171814+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T13:00:20.171814+0000', 'last_undegraded': '2023-10-19T13:00:20.171814+0000', 'last_fullsized': '2023-10-19T13:00:20.171814+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_clean_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T13:00:24.977 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.7 is not active+clean 2023-10-19T13:00:24.977 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.7', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T13:00:19.169402+0000', 'last_change': '2023-10-19T13:00:19.169402+0000', 'last_active': '2023-10-19T13:00:19.169402+0000', 'last_peered': '2023-10-19T13:00:19.169402+0000', 'last_clean': '2023-10-19T13:00:19.169402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T13:00:19.169402+0000', 'last_undegraded': '2023-10-19T13:00:19.169402+0000', 'last_fullsized': '2023-10-19T13:00:19.169402+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T13:00:19.169402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T13:00:19.169402+0000', 'last_clean_scrub_stamp': '2023-10-19T13:00:19.169402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T13:00:24.978 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.6 is not active+clean 2023-10-19T13:00:24.978 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.6', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T13:00:19.169402+0000', 'last_change': '2023-10-19T13:00:19.169402+0000', 'last_active': '2023-10-19T13:00:19.169402+0000', 'last_peered': '2023-10-19T13:00:19.169402+0000', 'last_clean': '2023-10-19T13:00:19.169402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T13:00:19.169402+0000', 'last_undegraded': '2023-10-19T13:00:19.169402+0000', 'last_fullsized': '2023-10-19T13:00:19.169402+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T13:00:19.169402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T13:00:19.169402+0000', 'last_clean_scrub_stamp': '2023-10-19T13:00:19.169402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T13:00:24.979 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.6 is not active+clean 2023-10-19T13:00:24.979 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.6', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T13:00:20.171814+0000', 'last_change': '2023-10-19T13:00:20.171814+0000', 'last_active': '2023-10-19T13:00:20.171814+0000', 'last_peered': '2023-10-19T13:00:20.171814+0000', 'last_clean': '2023-10-19T13:00:20.171814+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T13:00:20.171814+0000', 'last_undegraded': '2023-10-19T13:00:20.171814+0000', 'last_fullsized': '2023-10-19T13:00:20.171814+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_clean_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T13:00:24.979 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.7 is not active+clean 2023-10-19T13:00:24.980 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.7', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T13:00:20.171814+0000', 'last_change': '2023-10-19T13:00:20.171814+0000', 'last_active': '2023-10-19T13:00:20.171814+0000', 'last_peered': '2023-10-19T13:00:20.171814+0000', 'last_clean': '2023-10-19T13:00:20.171814+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T13:00:20.171814+0000', 'last_undegraded': '2023-10-19T13:00:20.171814+0000', 'last_fullsized': '2023-10-19T13:00:20.171814+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_clean_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T13:00:24.980 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.0 is not active+clean 2023-10-19T13:00:24.981 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.0', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T13:00:20.171814+0000', 'last_change': '2023-10-19T13:00:20.171814+0000', 'last_active': '2023-10-19T13:00:20.171814+0000', 'last_peered': '2023-10-19T13:00:20.171814+0000', 'last_clean': '2023-10-19T13:00:20.171814+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T13:00:20.171814+0000', 'last_undegraded': '2023-10-19T13:00:20.171814+0000', 'last_fullsized': '2023-10-19T13:00:20.171814+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_clean_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T13:00:24.981 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.3 is not active+clean 2023-10-19T13:00:24.981 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.3', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T13:00:19.169402+0000', 'last_change': '2023-10-19T13:00:19.169402+0000', 'last_active': '2023-10-19T13:00:19.169402+0000', 'last_peered': '2023-10-19T13:00:19.169402+0000', 'last_clean': '2023-10-19T13:00:19.169402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T13:00:19.169402+0000', 'last_undegraded': '2023-10-19T13:00:19.169402+0000', 'last_fullsized': '2023-10-19T13:00:19.169402+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T13:00:19.169402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T13:00:19.169402+0000', 'last_clean_scrub_stamp': '2023-10-19T13:00:19.169402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T13:00:24.982 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.1 is not active+clean 2023-10-19T13:00:24.982 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.1', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T13:00:20.171814+0000', 'last_change': '2023-10-19T13:00:20.171814+0000', 'last_active': '2023-10-19T13:00:20.171814+0000', 'last_peered': '2023-10-19T13:00:20.171814+0000', 'last_clean': '2023-10-19T13:00:20.171814+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T13:00:20.171814+0000', 'last_undegraded': '2023-10-19T13:00:20.171814+0000', 'last_fullsized': '2023-10-19T13:00:20.171814+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_clean_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T13:00:24.983 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.2 is not active+clean 2023-10-19T13:00:24.983 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.2', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T13:00:19.169402+0000', 'last_change': '2023-10-19T13:00:19.169402+0000', 'last_active': '2023-10-19T13:00:19.169402+0000', 'last_peered': '2023-10-19T13:00:19.169402+0000', 'last_clean': '2023-10-19T13:00:19.169402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T13:00:19.169402+0000', 'last_undegraded': '2023-10-19T13:00:19.169402+0000', 'last_fullsized': '2023-10-19T13:00:19.169402+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T13:00:19.169402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T13:00:19.169402+0000', 'last_clean_scrub_stamp': '2023-10-19T13:00:19.169402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T13:00:24.983 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.1 is not active+clean 2023-10-19T13:00:24.984 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.1', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T13:00:19.169402+0000', 'last_change': '2023-10-19T13:00:19.169402+0000', 'last_active': '2023-10-19T13:00:19.169402+0000', 'last_peered': '2023-10-19T13:00:19.169402+0000', 'last_clean': '2023-10-19T13:00:19.169402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T13:00:19.169402+0000', 'last_undegraded': '2023-10-19T13:00:19.169402+0000', 'last_fullsized': '2023-10-19T13:00:19.169402+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T13:00:19.169402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T13:00:19.169402+0000', 'last_clean_scrub_stamp': '2023-10-19T13:00:19.169402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T13:00:24.984 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.b is not active+clean 2023-10-19T13:00:24.984 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T13:00:20.171814+0000', 'last_change': '2023-10-19T13:00:20.171814+0000', 'last_active': '2023-10-19T13:00:20.171814+0000', 'last_peered': '2023-10-19T13:00:20.171814+0000', 'last_clean': '2023-10-19T13:00:20.171814+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T13:00:20.171814+0000', 'last_undegraded': '2023-10-19T13:00:20.171814+0000', 'last_fullsized': '2023-10-19T13:00:20.171814+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'last_clean_scrub_stamp': '2023-10-19T13:00:20.171814+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T13:00:24.985 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.8 is not active+clean 2023-10-19T13:00:24.985 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.8', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T13:00:19.169402+0000', 'last_change': '2023-10-19T13:00:19.169402+0000', 'last_active': '2023-10-19T13:00:19.169402+0000', 'last_peered': '2023-10-19T13:00:19.169402+0000', 'last_clean': '2023-10-19T13:00:19.169402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T13:00:19.169402+0000', 'last_undegraded': '2023-10-19T13:00:19.169402+0000', 'last_fullsized': '2023-10-19T13:00:19.169402+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T13:00:19.169402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T13:00:19.169402+0000', 'last_clean_scrub_stamp': '2023-10-19T13:00:19.169402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T13:00:24.986 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2023-10-19T13:00:25.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.092+0000 7f45199b0640 1 -- 172.21.15.57:0/484313340 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4500004d00 msgr2=0x7f4500005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:25.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.092+0000 7f45199b0640 1 --2- 172.21.15.57:0/484313340 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4500004d00 0x7f4500005180 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f4504001d80 tx=0x7f45000057f0 comp rx=0 tx=0).stop 2023-10-19T13:00:25.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.092+0000 7f45137fe640 1 -- 172.21.15.57:0/484313340 reap_dead start 2023-10-19T13:00:25.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.093+0000 7f45199b0640 1 -- 172.21.15.57:0/484313340 shutdown_connections 2023-10-19T13:00:25.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.093+0000 7f45199b0640 1 -- 172.21.15.57:0/484313340 >> 172.21.15.57:0/484313340 conn(0x7f45140b0670 msgr2=0x7f45140b0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:25.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.093+0000 7f45199b0640 1 -- 172.21.15.57:0/484313340 shutdown_connections 2023-10-19T13:00:25.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.093+0000 7f45199b0640 1 -- 172.21.15.57:0/484313340 wait complete. 2023-10-19T13:00:25.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.094+0000 7f45199b0640 1 Processor -- start 2023-10-19T13:00:25.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.094+0000 7f45199b0640 1 -- start start 2023-10-19T13:00:25.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.095+0000 7f45199b0640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4500004d00 0x7f4514152af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:25.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.095+0000 7f45199b0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f45141ef0e0 0x7f4514157440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:25.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.095+0000 7f45199b0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4514150940 0x7f4514150dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:25.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.095+0000 7f45199b0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4514151300 con 0x7f45141ef0e0 2023-10-19T13:00:25.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.095+0000 7f45199b0640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45141588a0 con 0x7f4514150940 2023-10-19T13:00:25.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.095+0000 7f45199b0640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4514157c90 con 0x7f4500004d00 2023-10-19T13:00:25.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.095+0000 7f45127fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f45141ef0e0 0x7f4514157440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:25.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.095+0000 7f45127fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f45141ef0e0 0x7f4514157440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:49902/0 (socket says 172.21.15.57:49902) 2023-10-19T13:00:25.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.095+0000 7f45127fc640 1 -- 172.21.15.57:0/2391795789 learned_addr learned my addr 172.21.15.57:0/2391795789 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:25.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.095+0000 7f4512ffd640 1 --2- 172.21.15.57:0/2391795789 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4500004d00 0x7f4514152af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:25.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.095+0000 7f45137fe640 1 --2- 172.21.15.57:0/2391795789 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4514150940 0x7f4514150dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:25.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.095+0000 7f45127fc640 1 -- 172.21.15.57:0/2391795789 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4500004d00 msgr2=0x7f4514152af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:25.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.095+0000 7f45127fc640 1 --2- 172.21.15.57:0/2391795789 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4500004d00 0x7f4514152af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:25.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.095+0000 7f45127fc640 1 -- 172.21.15.57:0/2391795789 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4514150940 msgr2=0x7f4514150dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:25.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.095+0000 7f45127fc640 1 --2- 172.21.15.57:0/2391795789 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4514150940 0x7f4514150dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:25.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.095+0000 7f45127fc640 1 -- 172.21.15.57:0/2391795789 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45140cd310 con 0x7f45141ef0e0 2023-10-19T13:00:25.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.095+0000 7f4512ffd640 1 --2- 172.21.15.57:0/2391795789 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4500004d00 0x7f4514152af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:00:25.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.095+0000 7f45127fc640 1 --2- 172.21.15.57:0/2391795789 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f45141ef0e0 0x7f4514157440 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f44fc00e690 tx=0x7f44fc00eb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:25.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.095+0000 7f450bfff640 1 -- 172.21.15.57:0/2391795789 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44fc004460 con 0x7f45141ef0e0 2023-10-19T13:00:25.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.096+0000 7f450bfff640 1 -- 172.21.15.57:0/2391795789 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f44fc00c840 con 0x7f45141ef0e0 2023-10-19T13:00:25.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.096+0000 7f45199b0640 1 -- 172.21.15.57:0/2391795789 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4514157f70 con 0x7f45141ef0e0 2023-10-19T13:00:25.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.096+0000 7f450bfff640 1 -- 172.21.15.57:0/2391795789 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44fc00cc50 con 0x7f45141ef0e0 2023-10-19T13:00:25.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.096+0000 7f45199b0640 1 -- 172.21.15.57:0/2391795789 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4514158530 con 0x7f45141ef0e0 2023-10-19T13:00:25.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.096+0000 7f45199b0640 1 -- 172.21.15.57:0/2391795789 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45140ce1d0 con 0x7f45141ef0e0 2023-10-19T13:00:25.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.097+0000 7f450bfff640 1 -- 172.21.15.57:0/2391795789 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 32) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f44fc0035f0 con 0x7f45141ef0e0 2023-10-19T13:00:25.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.097+0000 7f450bfff640 1 --2- 172.21.15.57:0/2391795789 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f44f406af60 0x7f44f406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:25.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.097+0000 7f450bfff640 1 -- 172.21.15.57:0/2391795789 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f44fc08a470 con 0x7f45141ef0e0 2023-10-19T13:00:25.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.098+0000 7f4512ffd640 1 --2- 172.21.15.57:0/2391795789 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f44f406af60 0x7f44f406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:25.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.098+0000 7f4512ffd640 1 --2- 172.21.15.57:0/2391795789 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f44f406af60 0x7f44f406d420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f450c009c50 tx=0x7f450c005f90 comp rx=0 tx=0).ready entity=mgr.6256 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:25.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.102+0000 7f450bfff640 1 -- 172.21.15.57:0/2391795789 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f44fc055230 con 0x7f45141ef0e0 2023-10-19T13:00:25.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.344+0000 7f45199b0640 1 -- 172.21.15.57:0/2391795789 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f4514158a10 con 0x7f45141ef0e0 2023-10-19T13:00:25.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.344+0000 7f450bfff640 1 -- 172.21.15.57:0/2391795789 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1012 (secure 0 0 0) 0x7f44fc058850 con 0x7f45141ef0e0 2023-10-19T13:00:25.347 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:00:25.348 INFO:teuthology.orchestra.run.smithi057.stdout:{"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":992,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":87,"num_osds":8,"num_up_osds":8,"osd_up_since":1697719445,"num_in_osds":8,"osd_in_since":1697719436,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":23,"data_bytes":592778,"bytes_used":323391488,"bytes_avail":772770721792,"bytes_total":773094113280,"write_bytes_sec":2439,"read_op_per_sec":0,"write_op_per_sec":7},"fsmap":{"epoch":30,"id":4,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":4,"rank":0,"name":"a","status":"up:active","gid":7038}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2023-10-19T12:59:16.567883+0000","services":{}},"progress_events":{}} 2023-10-19T13:00:25.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.346+0000 7f45199b0640 1 -- 172.21.15.57:0/2391795789 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f44f406af60 msgr2=0x7f44f406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:25.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.346+0000 7f45199b0640 1 --2- 172.21.15.57:0/2391795789 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f44f406af60 0x7f44f406d420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f450c009c50 tx=0x7f450c005f90 comp rx=0 tx=0).stop 2023-10-19T13:00:25.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.346+0000 7f45199b0640 1 -- 172.21.15.57:0/2391795789 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f45141ef0e0 msgr2=0x7f4514157440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:25.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.346+0000 7f45199b0640 1 --2- 172.21.15.57:0/2391795789 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f45141ef0e0 0x7f4514157440 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f44fc00e690 tx=0x7f44fc00eb50 comp rx=0 tx=0).stop 2023-10-19T13:00:25.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.346+0000 7f45199b0640 1 -- 172.21.15.57:0/2391795789 shutdown_connections 2023-10-19T13:00:25.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.346+0000 7f45199b0640 1 --2- 172.21.15.57:0/2391795789 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f44f406af60 0x7f44f406d420 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:25.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.346+0000 7f45199b0640 1 --2- 172.21.15.57:0/2391795789 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4514150940 0x7f4514150dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:25.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.346+0000 7f45199b0640 1 --2- 172.21.15.57:0/2391795789 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f45141ef0e0 0x7f4514157440 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:25.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.346+0000 7f45199b0640 1 --2- 172.21.15.57:0/2391795789 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4500004d00 0x7f4514152af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:25.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.346+0000 7f45199b0640 1 -- 172.21.15.57:0/2391795789 >> 172.21.15.57:0/2391795789 conn(0x7f45140b0670 msgr2=0x7f45140acb10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:25.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.346+0000 7f45199b0640 1 -- 172.21.15.57:0/2391795789 shutdown_connections 2023-10-19T13:00:25.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.346+0000 7f45199b0640 1 -- 172.21.15.57:0/2391795789 wait complete. 2023-10-19T13:00:25.359 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2023-10-19T13:00:25.360 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-19T13:00:25.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.466+0000 7f664ac26640 1 -- 172.21.15.57:0/142447538 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f66440cb2f0 msgr2=0x7f6644070520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:25.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.466+0000 7f664ac26640 1 --2- 172.21.15.57:0/142447538 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f66440cb2f0 0x7f6644070520 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f6630005fd0 tx=0x7f663002bcf0 comp rx=0 tx=0).stop 2023-10-19T13:00:25.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.466+0000 7f664ac26640 1 -- 172.21.15.57:0/142447538 shutdown_connections 2023-10-19T13:00:25.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.466+0000 7f664ac26640 1 --2- 172.21.15.57:0/142447538 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f66440cb2f0 0x7f6644070520 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:25.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.466+0000 7f664ac26640 1 --2- 172.21.15.57:0/142447538 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6644157c00 0x7f66440c98b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:25.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.466+0000 7f664ac26640 1 -- 172.21.15.57:0/142447538 >> 172.21.15.57:0/142447538 conn(0x7f66440aec40 msgr2=0x7f66440af050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:25.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.466+0000 7f664ac26640 1 -- 172.21.15.57:0/142447538 shutdown_connections 2023-10-19T13:00:25.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.466+0000 7f664ac26640 1 -- 172.21.15.57:0/142447538 wait complete. 2023-10-19T13:00:25.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.467+0000 7f664ac26640 1 Processor -- start 2023-10-19T13:00:25.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.467+0000 7f664ac26640 1 -- start start 2023-10-19T13:00:25.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.468+0000 7f664ac26640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6644070a60 0x7f6644146190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:25.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.468+0000 7f664ac26640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f66440cb2f0 0x7f664415aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:25.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.468+0000 7f664ac26640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6644157c00 0x7f6644146700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:25.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.468+0000 7f664ac26640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f664415a8a0 con 0x7f6644070a60 2023-10-19T13:00:25.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.468+0000 7f664ac26640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f664414b3e0 con 0x7f66440cb2f0 2023-10-19T13:00:25.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.468+0000 7f664ac26640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f664415afb0 con 0x7f6644157c00 2023-10-19T13:00:25.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.468+0000 7f66489c2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6644070a60 0x7f6644146190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:25.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.468+0000 7f66489c2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6644070a60 0x7f6644146190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:49924/0 (socket says 172.21.15.57:49924) 2023-10-19T13:00:25.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.468+0000 7f66489c2640 1 -- 172.21.15.57:0/2722804685 learned_addr learned my addr 172.21.15.57:0/2722804685 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:25.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.468+0000 7f6643fff640 1 --2- 172.21.15.57:0/2722804685 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f66440cb2f0 0x7f664415aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:25.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.468+0000 7f66491c3640 1 --2- 172.21.15.57:0/2722804685 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6644157c00 0x7f6644146700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:25.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.468+0000 7f66489c2640 1 -- 172.21.15.57:0/2722804685 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6644157c00 msgr2=0x7f6644146700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:25.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.468+0000 7f66489c2640 1 --2- 172.21.15.57:0/2722804685 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6644157c00 0x7f6644146700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:25.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.468+0000 7f66489c2640 1 -- 172.21.15.57:0/2722804685 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f66440cb2f0 msgr2=0x7f664415aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:25.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.468+0000 7f66489c2640 1 --2- 172.21.15.57:0/2722804685 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f66440cb2f0 0x7f664415aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:25.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.468+0000 7f66489c2640 1 -- 172.21.15.57:0/2722804685 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6630009560 con 0x7f6644070a60 2023-10-19T13:00:25.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.469+0000 7f66489c2640 1 --2- 172.21.15.57:0/2722804685 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6644070a60 0x7f6644146190 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f663800d740 tx=0x7f663800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:25.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.469+0000 7f66417fa640 1 -- 172.21.15.57:0/2722804685 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6638009bd0 con 0x7f6644070a60 2023-10-19T13:00:25.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.469+0000 7f66417fa640 1 -- 172.21.15.57:0/2722804685 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f6638009e20 con 0x7f6644070a60 2023-10-19T13:00:25.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.469+0000 7f66417fa640 1 -- 172.21.15.57:0/2722804685 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66380106c0 con 0x7f6644070a60 2023-10-19T13:00:25.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.469+0000 7f664ac26640 1 -- 172.21.15.57:0/2722804685 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f664415a020 con 0x7f6644070a60 2023-10-19T13:00:25.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.469+0000 7f664ac26640 1 -- 172.21.15.57:0/2722804685 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f664415a2e0 con 0x7f6644070a60 2023-10-19T13:00:25.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.470+0000 7f664ac26640 1 -- 172.21.15.57:0/2722804685 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66440690e0 con 0x7f6644070a60 2023-10-19T13:00:25.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.470+0000 7f66417fa640 1 -- 172.21.15.57:0/2722804685 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 32) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6638010820 con 0x7f6644070a60 2023-10-19T13:00:25.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.471+0000 7f66417fa640 1 --2- 172.21.15.57:0/2722804685 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f662806af60 0x7f662806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:25.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.471+0000 7f66417fa640 1 -- 172.21.15.57:0/2722804685 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f6638089eb0 con 0x7f6644070a60 2023-10-19T13:00:25.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.471+0000 7f6643fff640 1 --2- 172.21.15.57:0/2722804685 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f662806af60 0x7f662806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:25.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.471+0000 7f6643fff640 1 --2- 172.21.15.57:0/2722804685 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f662806af60 0x7f662806d420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6630035000 tx=0x7f6630004730 comp rx=0 tx=0).ready entity=mgr.6256 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:25.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.475+0000 7f66417fa640 1 -- 172.21.15.57:0/2722804685 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6638054bf0 con 0x7f6644070a60 2023-10-19T13:00:25.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.645+0000 7f664ac26640 1 -- 172.21.15.57:0/2722804685 --> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f664415bd20 con 0x7f662806af60 2023-10-19T13:00:25.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.653+0000 7f66417fa640 1 -- 172.21.15.57:0/2722804685 <== mgr.6256 v2:172.21.15.179:6832/3815793798 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289637 (secure 0 0 0) 0x7f66300055e0 con 0x7f662806af60 2023-10-19T13:00:25.659 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:00:25.660 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2023-10-19T13:00:25.662 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.656+0000 7f664ac26640 1 -- 172.21.15.57:0/2722804685 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f662806af60 msgr2=0x7f662806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:25.662 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.656+0000 7f664ac26640 1 --2- 172.21.15.57:0/2722804685 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f662806af60 0x7f662806d420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6630035000 tx=0x7f6630004730 comp rx=0 tx=0).stop 2023-10-19T13:00:25.662 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.656+0000 7f664ac26640 1 -- 172.21.15.57:0/2722804685 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6644070a60 msgr2=0x7f6644146190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:25.662 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.656+0000 7f664ac26640 1 --2- 172.21.15.57:0/2722804685 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6644070a60 0x7f6644146190 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f663800d740 tx=0x7f663800dc00 comp rx=0 tx=0).stop 2023-10-19T13:00:25.663 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.656+0000 7f664ac26640 1 -- 172.21.15.57:0/2722804685 shutdown_connections 2023-10-19T13:00:25.663 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.656+0000 7f664ac26640 1 --2- 172.21.15.57:0/2722804685 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f662806af60 0x7f662806d420 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:25.663 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.656+0000 7f664ac26640 1 --2- 172.21.15.57:0/2722804685 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6644157c00 0x7f6644146700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:25.663 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.656+0000 7f664ac26640 1 --2- 172.21.15.57:0/2722804685 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f66440cb2f0 0x7f664415aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:25.663 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.656+0000 7f664ac26640 1 --2- 172.21.15.57:0/2722804685 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6644070a60 0x7f6644146190 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:25.664 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.656+0000 7f664ac26640 1 -- 172.21.15.57:0/2722804685 >> 172.21.15.57:0/2722804685 conn(0x7f66440aec40 msgr2=0x7f664414baa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:25.664 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.656+0000 7f664ac26640 1 -- 172.21.15.57:0/2722804685 shutdown_connections 2023-10-19T13:00:25.664 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:25.656+0000 7f664ac26640 1 -- 172.21.15.57:0/2722804685 wait complete. 2023-10-19T13:00:25.669 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2023-10-19T13:00:25.050166+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":283,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":199,"ondisk_log_size":199,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":153,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":315812,"kb_used_data":9428,"kb_used_omap":12,"kb_used_meta":306227,"kb_avail":754658908,"statfs":{"total":773094113280,"available":772770721792,"internally_reserved":0,"allocated":9654272,"data_stored":5389321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":313577017},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.876206"},"pg_stats":[{"pgid":"10.3f","version":"0'0","reported_seq":9,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-19T13:00:21.237877+0000","last_change":"2023-10-19T13:00:21.228303+0000","last_active":"2023-10-19T13:00:21.237877+0000","last_peered":"2023-10-19T13:00:21.237877+0000","last_clean":"2023-10-19T13:00:21.237877+0000","last_became_active":"2023-10-19T13:00:21.227974+0000","last_became_peered":"2023-10-19T13:00:21.227974+0000","last_unstale":"2023-10-19T13:00:21.237877+0000","last_undegraded":"2023-10-19T13:00:21.237877+0000","last_fullsized":"2023-10-19T13:00:21.237877+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:37:52.369486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3e","version":"0'0","reported_seq":9,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-19T13:00:21.233058+0000","last_change":"2023-10-19T13:00:21.227759+0000","last_active":"2023-10-19T13:00:21.233058+0000","last_peered":"2023-10-19T13:00:21.233058+0000","last_clean":"2023-10-19T13:00:21.233058+0000","last_became_active":"2023-10-19T13:00:21.227466+0000","last_became_peered":"2023-10-19T13:00:21.227466+0000","last_unstale":"2023-10-19T13:00:21.233058+0000","last_undegraded":"2023-10-19T13:00:21.233058+0000","last_fullsized":"2023-10-19T13:00:21.233058+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:55:42.567075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3d","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.222826+0000","last_change":"2023-10-19T13:00:21.224369+0000","last_active":"2023-10-19T13:00:22.222826+0000","last_peered":"2023-10-19T13:00:22.222826+0000","last_clean":"2023-10-19T13:00:22.222826+0000","last_became_active":"2023-10-19T13:00:21.224115+0000","last_became_peered":"2023-10-19T13:00:21.224115+0000","last_unstale":"2023-10-19T13:00:22.222826+0000","last_undegraded":"2023-10-19T13:00:22.222826+0000","last_fullsized":"2023-10-19T13:00:22.222826+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:27:46.354594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.3c","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.384886+0000","last_change":"2023-10-19T13:00:21.221167+0000","last_active":"2023-10-19T13:00:22.384886+0000","last_peered":"2023-10-19T13:00:22.384886+0000","last_clean":"2023-10-19T13:00:22.384886+0000","last_became_active":"2023-10-19T13:00:21.220894+0000","last_became_peered":"2023-10-19T13:00:21.220894+0000","last_unstale":"2023-10-19T13:00:22.384886+0000","last_undegraded":"2023-10-19T13:00:22.384886+0000","last_fullsized":"2023-10-19T13:00:22.384886+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:34:57.792223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3b","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.423410+0000","last_change":"2023-10-19T13:00:21.214718+0000","last_active":"2023-10-19T13:00:22.423410+0000","last_peered":"2023-10-19T13:00:22.423410+0000","last_clean":"2023-10-19T13:00:22.423410+0000","last_became_active":"2023-10-19T13:00:21.214364+0000","last_became_peered":"2023-10-19T13:00:21.214364+0000","last_unstale":"2023-10-19T13:00:22.423410+0000","last_undegraded":"2023-10-19T13:00:22.423410+0000","last_fullsized":"2023-10-19T13:00:22.423410+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:52:56.240367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.3a","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.221076+0000","last_change":"2023-10-19T13:00:21.219325+0000","last_active":"2023-10-19T13:00:22.221076+0000","last_peered":"2023-10-19T13:00:22.221076+0000","last_clean":"2023-10-19T13:00:22.221076+0000","last_became_active":"2023-10-19T13:00:21.218458+0000","last_became_peered":"2023-10-19T13:00:21.218458+0000","last_unstale":"2023-10-19T13:00:22.221076+0000","last_undegraded":"2023-10-19T13:00:22.221076+0000","last_fullsized":"2023-10-19T13:00:22.221076+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:31:39.478057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.39","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:23.044542+0000","last_change":"2023-10-19T13:00:21.220590+0000","last_active":"2023-10-19T13:00:23.044542+0000","last_peered":"2023-10-19T13:00:23.044542+0000","last_clean":"2023-10-19T13:00:23.044542+0000","last_became_active":"2023-10-19T13:00:21.220337+0000","last_became_peered":"2023-10-19T13:00:21.220337+0000","last_unstale":"2023-10-19T13:00:23.044542+0000","last_undegraded":"2023-10-19T13:00:23.044542+0000","last_fullsized":"2023-10-19T13:00:23.044542+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:25:25.779991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.38","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:23.044542+0000","last_change":"2023-10-19T13:00:21.214103+0000","last_active":"2023-10-19T13:00:23.044542+0000","last_peered":"2023-10-19T13:00:23.044542+0000","last_clean":"2023-10-19T13:00:23.044542+0000","last_became_active":"2023-10-19T13:00:21.213851+0000","last_became_peered":"2023-10-19T13:00:21.213851+0000","last_unstale":"2023-10-19T13:00:23.044542+0000","last_undegraded":"2023-10-19T13:00:23.044542+0000","last_fullsized":"2023-10-19T13:00:23.044542+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:30:23.953334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.37","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.223081+0000","last_change":"2023-10-19T13:00:21.226394+0000","last_active":"2023-10-19T13:00:22.223081+0000","last_peered":"2023-10-19T13:00:22.223081+0000","last_clean":"2023-10-19T13:00:22.223081+0000","last_became_active":"2023-10-19T13:00:21.226136+0000","last_became_peered":"2023-10-19T13:00:21.226136+0000","last_unstale":"2023-10-19T13:00:22.223081+0000","last_undegraded":"2023-10-19T13:00:22.223081+0000","last_fullsized":"2023-10-19T13:00:22.223081+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:55:17.988760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.36","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.223037+0000","last_change":"2023-10-19T13:00:21.231437+0000","last_active":"2023-10-19T13:00:22.223037+0000","last_peered":"2023-10-19T13:00:22.223037+0000","last_clean":"2023-10-19T13:00:22.223037+0000","last_became_active":"2023-10-19T13:00:21.231133+0000","last_became_peered":"2023-10-19T13:00:21.231133+0000","last_unstale":"2023-10-19T13:00:22.223037+0000","last_undegraded":"2023-10-19T13:00:22.223037+0000","last_fullsized":"2023-10-19T13:00:22.223037+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:08:45.213465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.35","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.384804+0000","last_change":"2023-10-19T13:00:21.230427+0000","last_active":"2023-10-19T13:00:22.384804+0000","last_peered":"2023-10-19T13:00:22.384804+0000","last_clean":"2023-10-19T13:00:22.384804+0000","last_became_active":"2023-10-19T13:00:21.230143+0000","last_became_peered":"2023-10-19T13:00:21.230143+0000","last_unstale":"2023-10-19T13:00:22.384804+0000","last_undegraded":"2023-10-19T13:00:22.384804+0000","last_fullsized":"2023-10-19T13:00:22.384804+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:33:36.402003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.34","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:23.044878+0000","last_change":"2023-10-19T13:00:21.218737+0000","last_active":"2023-10-19T13:00:23.044878+0000","last_peered":"2023-10-19T13:00:23.044878+0000","last_clean":"2023-10-19T13:00:23.044878+0000","last_became_active":"2023-10-19T13:00:21.218347+0000","last_became_peered":"2023-10-19T13:00:21.218347+0000","last_unstale":"2023-10-19T13:00:23.044878+0000","last_undegraded":"2023-10-19T13:00:23.044878+0000","last_fullsized":"2023-10-19T13:00:23.044878+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:22:14.250042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.33","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:23.044789+0000","last_change":"2023-10-19T13:00:21.216810+0000","last_active":"2023-10-19T13:00:23.044789+0000","last_peered":"2023-10-19T13:00:23.044789+0000","last_clean":"2023-10-19T13:00:23.044789+0000","last_became_active":"2023-10-19T13:00:21.216536+0000","last_became_peered":"2023-10-19T13:00:21.216536+0000","last_unstale":"2023-10-19T13:00:23.044789+0000","last_undegraded":"2023-10-19T13:00:23.044789+0000","last_fullsized":"2023-10-19T13:00:23.044789+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:12:58.144772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.32","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.222482+0000","last_change":"2023-10-19T13:00:21.224986+0000","last_active":"2023-10-19T13:00:22.222482+0000","last_peered":"2023-10-19T13:00:22.222482+0000","last_clean":"2023-10-19T13:00:22.222482+0000","last_became_active":"2023-10-19T13:00:21.224703+0000","last_became_peered":"2023-10-19T13:00:21.224703+0000","last_unstale":"2023-10-19T13:00:22.222482+0000","last_undegraded":"2023-10-19T13:00:22.222482+0000","last_fullsized":"2023-10-19T13:00:22.222482+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:14:30.933348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.31","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.222437+0000","last_change":"2023-10-19T13:00:21.230937+0000","last_active":"2023-10-19T13:00:22.222437+0000","last_peered":"2023-10-19T13:00:22.222437+0000","last_clean":"2023-10-19T13:00:22.222437+0000","last_became_active":"2023-10-19T13:00:21.230649+0000","last_became_peered":"2023-10-19T13:00:21.230649+0000","last_unstale":"2023-10-19T13:00:22.222437+0000","last_undegraded":"2023-10-19T13:00:22.222437+0000","last_fullsized":"2023-10-19T13:00:22.222437+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:00:42.269401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.30","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:23.044519+0000","last_change":"2023-10-19T13:00:21.219839+0000","last_active":"2023-10-19T13:00:23.044519+0000","last_peered":"2023-10-19T13:00:23.044519+0000","last_clean":"2023-10-19T13:00:23.044519+0000","last_became_active":"2023-10-19T13:00:21.219524+0000","last_became_peered":"2023-10-19T13:00:21.219524+0000","last_unstale":"2023-10-19T13:00:23.044519+0000","last_undegraded":"2023-10-19T13:00:23.044519+0000","last_fullsized":"2023-10-19T13:00:23.044519+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:16:36.020089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2f","version":"0'0","reported_seq":9,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-19T13:00:21.237892+0000","last_change":"2023-10-19T13:00:21.228480+0000","last_active":"2023-10-19T13:00:21.237892+0000","last_peered":"2023-10-19T13:00:21.237892+0000","last_clean":"2023-10-19T13:00:21.237892+0000","last_became_active":"2023-10-19T13:00:21.227687+0000","last_became_peered":"2023-10-19T13:00:21.227687+0000","last_unstale":"2023-10-19T13:00:21.237892+0000","last_undegraded":"2023-10-19T13:00:21.237892+0000","last_fullsized":"2023-10-19T13:00:21.237892+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:58:41.166338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2e","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.384934+0000","last_change":"2023-10-19T13:00:21.221940+0000","last_active":"2023-10-19T13:00:22.384934+0000","last_peered":"2023-10-19T13:00:22.384934+0000","last_clean":"2023-10-19T13:00:22.384934+0000","last_became_active":"2023-10-19T13:00:21.221511+0000","last_became_peered":"2023-10-19T13:00:21.221511+0000","last_unstale":"2023-10-19T13:00:22.384934+0000","last_undegraded":"2023-10-19T13:00:22.384934+0000","last_fullsized":"2023-10-19T13:00:22.384934+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:14:55.381409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2d","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.423674+0000","last_change":"2023-10-19T13:00:21.208462+0000","last_active":"2023-10-19T13:00:22.423674+0000","last_peered":"2023-10-19T13:00:22.423674+0000","last_clean":"2023-10-19T13:00:22.423674+0000","last_became_active":"2023-10-19T13:00:21.208176+0000","last_became_peered":"2023-10-19T13:00:21.208176+0000","last_unstale":"2023-10-19T13:00:22.423674+0000","last_undegraded":"2023-10-19T13:00:22.423674+0000","last_fullsized":"2023-10-19T13:00:22.423674+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:34:57.792223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2c","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.222039+0000","last_change":"2023-10-19T13:00:21.221939+0000","last_active":"2023-10-19T13:00:22.222039+0000","last_peered":"2023-10-19T13:00:22.222039+0000","last_clean":"2023-10-19T13:00:22.222039+0000","last_became_active":"2023-10-19T13:00:21.221665+0000","last_became_peered":"2023-10-19T13:00:21.221665+0000","last_unstale":"2023-10-19T13:00:22.222039+0000","last_undegraded":"2023-10-19T13:00:22.222039+0000","last_fullsized":"2023-10-19T13:00:22.222039+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:17:57.872305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2b","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:23.044845+0000","last_change":"2023-10-19T13:00:21.229853+0000","last_active":"2023-10-19T13:00:23.044845+0000","last_peered":"2023-10-19T13:00:23.044845+0000","last_clean":"2023-10-19T13:00:23.044845+0000","last_became_active":"2023-10-19T13:00:21.229515+0000","last_became_peered":"2023-10-19T13:00:21.229515+0000","last_unstale":"2023-10-19T13:00:23.044845+0000","last_undegraded":"2023-10-19T13:00:23.044845+0000","last_fullsized":"2023-10-19T13:00:23.044845+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:27:46.354594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.2a","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.221862+0000","last_change":"2023-10-19T13:00:21.215974+0000","last_active":"2023-10-19T13:00:22.221862+0000","last_peered":"2023-10-19T13:00:22.221862+0000","last_clean":"2023-10-19T13:00:22.221862+0000","last_became_active":"2023-10-19T13:00:21.215688+0000","last_became_peered":"2023-10-19T13:00:21.215688+0000","last_unstale":"2023-10-19T13:00:22.221862+0000","last_undegraded":"2023-10-19T13:00:22.221862+0000","last_fullsized":"2023-10-19T13:00:22.221862+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:18:50.120828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.29","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.221799+0000","last_change":"2023-10-19T13:00:21.221347+0000","last_active":"2023-10-19T13:00:22.221799+0000","last_peered":"2023-10-19T13:00:22.221799+0000","last_clean":"2023-10-19T13:00:22.221799+0000","last_became_active":"2023-10-19T13:00:21.220982+0000","last_became_peered":"2023-10-19T13:00:21.220982+0000","last_unstale":"2023-10-19T13:00:22.221799+0000","last_undegraded":"2023-10-19T13:00:22.221799+0000","last_fullsized":"2023-10-19T13:00:22.221799+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:36:46.822232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.28","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.222424+0000","last_change":"2023-10-19T13:00:21.226183+0000","last_active":"2023-10-19T13:00:22.222424+0000","last_peered":"2023-10-19T13:00:22.222424+0000","last_clean":"2023-10-19T13:00:22.222424+0000","last_became_active":"2023-10-19T13:00:21.225901+0000","last_became_peered":"2023-10-19T13:00:21.225901+0000","last_unstale":"2023-10-19T13:00:22.222424+0000","last_undegraded":"2023-10-19T13:00:22.222424+0000","last_fullsized":"2023-10-19T13:00:22.222424+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:00:49.808529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.27","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:23.045013+0000","last_change":"2023-10-19T13:00:21.226532+0000","last_active":"2023-10-19T13:00:23.045013+0000","last_peered":"2023-10-19T13:00:23.045013+0000","last_clean":"2023-10-19T13:00:23.045013+0000","last_became_active":"2023-10-19T13:00:21.226239+0000","last_became_peered":"2023-10-19T13:00:21.226239+0000","last_unstale":"2023-10-19T13:00:23.045013+0000","last_undegraded":"2023-10-19T13:00:23.045013+0000","last_fullsized":"2023-10-19T13:00:23.045013+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:39:34.098405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.26","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:23.044982+0000","last_change":"2023-10-19T13:00:21.222721+0000","last_active":"2023-10-19T13:00:23.044982+0000","last_peered":"2023-10-19T13:00:23.044982+0000","last_clean":"2023-10-19T13:00:23.044982+0000","last_became_active":"2023-10-19T13:00:21.222414+0000","last_became_peered":"2023-10-19T13:00:21.222414+0000","last_unstale":"2023-10-19T13:00:23.044982+0000","last_undegraded":"2023-10-19T13:00:23.044982+0000","last_fullsized":"2023-10-19T13:00:23.044982+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:08:45.213465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.25","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.384972+0000","last_change":"2023-10-19T13:00:21.212641+0000","last_active":"2023-10-19T13:00:22.384972+0000","last_peered":"2023-10-19T13:00:22.384972+0000","last_clean":"2023-10-19T13:00:22.384972+0000","last_became_active":"2023-10-19T13:00:21.212380+0000","last_became_peered":"2023-10-19T13:00:21.212380+0000","last_unstale":"2023-10-19T13:00:22.384972+0000","last_undegraded":"2023-10-19T13:00:22.384972+0000","last_fullsized":"2023-10-19T13:00:22.384972+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:58:41.166338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.24","version":"0'0","reported_seq":9,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-19T13:00:21.233957+0000","last_change":"2023-10-19T13:00:21.227180+0000","last_active":"2023-10-19T13:00:21.233957+0000","last_peered":"2023-10-19T13:00:21.233957+0000","last_clean":"2023-10-19T13:00:21.233957+0000","last_became_active":"2023-10-19T13:00:21.226763+0000","last_became_peered":"2023-10-19T13:00:21.226763+0000","last_unstale":"2023-10-19T13:00:21.233957+0000","last_undegraded":"2023-10-19T13:00:21.233957+0000","last_fullsized":"2023-10-19T13:00:21.233957+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:41:19.821669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.23","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.222316+0000","last_change":"2023-10-19T13:00:21.224996+0000","last_active":"2023-10-19T13:00:22.222316+0000","last_peered":"2023-10-19T13:00:22.222316+0000","last_clean":"2023-10-19T13:00:22.222316+0000","last_became_active":"2023-10-19T13:00:21.224677+0000","last_became_peered":"2023-10-19T13:00:21.224677+0000","last_unstale":"2023-10-19T13:00:22.222316+0000","last_undegraded":"2023-10-19T13:00:22.222316+0000","last_fullsized":"2023-10-19T13:00:22.222316+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:30:23.953334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.22","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:23.044596+0000","last_change":"2023-10-19T13:00:21.219461+0000","last_active":"2023-10-19T13:00:23.044596+0000","last_peered":"2023-10-19T13:00:23.044596+0000","last_clean":"2023-10-19T13:00:23.044596+0000","last_became_active":"2023-10-19T13:00:21.219209+0000","last_became_peered":"2023-10-19T13:00:21.219209+0000","last_unstale":"2023-10-19T13:00:23.044596+0000","last_undegraded":"2023-10-19T13:00:23.044596+0000","last_fullsized":"2023-10-19T13:00:23.044596+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:14:30.933348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.21","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.384507+0000","last_change":"2023-10-19T13:00:21.212876+0000","last_active":"2023-10-19T13:00:22.384507+0000","last_peered":"2023-10-19T13:00:22.384507+0000","last_clean":"2023-10-19T13:00:22.384507+0000","last_became_active":"2023-10-19T13:00:21.212607+0000","last_became_peered":"2023-10-19T13:00:21.212607+0000","last_unstale":"2023-10-19T13:00:22.384507+0000","last_undegraded":"2023-10-19T13:00:22.384507+0000","last_fullsized":"2023-10-19T13:00:22.384507+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:52:56.240367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.20","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.220680+0000","last_change":"2023-10-19T13:00:21.217955+0000","last_active":"2023-10-19T13:00:22.220680+0000","last_peered":"2023-10-19T13:00:22.220680+0000","last_clean":"2023-10-19T13:00:22.220680+0000","last_became_active":"2023-10-19T13:00:21.217649+0000","last_became_peered":"2023-10-19T13:00:21.217649+0000","last_unstale":"2023-10-19T13:00:22.220680+0000","last_undegraded":"2023-10-19T13:00:22.220680+0000","last_fullsized":"2023-10-19T13:00:22.220680+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:55:17.988760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.423650+0000","last_change":"2023-10-19T13:00:21.231512+0000","last_active":"2023-10-19T13:00:22.423650+0000","last_peered":"2023-10-19T13:00:22.423650+0000","last_clean":"2023-10-19T13:00:22.423650+0000","last_became_active":"2023-10-19T13:00:21.231224+0000","last_became_peered":"2023-10-19T13:00:21.231224+0000","last_unstale":"2023-10-19T13:00:22.423650+0000","last_undegraded":"2023-10-19T13:00:22.423650+0000","last_fullsized":"2023-10-19T13:00:22.423650+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:33:36.402003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1f","version":"86'4","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-19T13:00:21.238495+0000","last_change":"2023-10-19T13:00:20.197586+0000","last_active":"2023-10-19T13:00:21.238495+0000","last_peered":"2023-10-19T13:00:21.238495+0000","last_clean":"2023-10-19T13:00:21.238495+0000","last_became_active":"2023-10-19T13:00:20.197286+0000","last_became_peered":"2023-10-19T13:00:20.197286+0000","last_unstale":"2023-10-19T13:00:21.238495+0000","last_undegraded":"2023-10-19T13:00:21.238495+0000","last_fullsized":"2023-10-19T13:00:21.238495+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:45:06.777683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.385057+0000","last_change":"2023-10-19T13:00:21.213911+0000","last_active":"2023-10-19T13:00:22.385057+0000","last_peered":"2023-10-19T13:00:22.385057+0000","last_clean":"2023-10-19T13:00:22.385057+0000","last_became_active":"2023-10-19T13:00:21.213624+0000","last_became_peered":"2023-10-19T13:00:21.213624+0000","last_unstale":"2023-10-19T13:00:22.385057+0000","last_undegraded":"2023-10-19T13:00:22.385057+0000","last_fullsized":"2023-10-19T13:00:22.385057+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:37:52.369486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"86'1","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.221295+0000","last_change":"2023-10-19T13:00:20.194934+0000","last_active":"2023-10-19T13:00:22.221295+0000","last_peered":"2023-10-19T13:00:22.221295+0000","last_clean":"2023-10-19T13:00:22.221295+0000","last_became_active":"2023-10-19T13:00:20.194657+0000","last_became_peered":"2023-10-19T13:00:20.194657+0000","last_unstale":"2023-10-19T13:00:22.221295+0000","last_undegraded":"2023-10-19T13:00:22.221295+0000","last_fullsized":"2023-10-19T13:00:22.221295+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:12:57.142360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.221258+0000","last_change":"2023-10-19T13:00:21.218060+0000","last_active":"2023-10-19T13:00:22.221258+0000","last_peered":"2023-10-19T13:00:22.221258+0000","last_clean":"2023-10-19T13:00:22.221258+0000","last_became_active":"2023-10-19T13:00:21.217766+0000","last_became_peered":"2023-10-19T13:00:21.217766+0000","last_unstale":"2023-10-19T13:00:22.221258+0000","last_undegraded":"2023-10-19T13:00:22.221258+0000","last_fullsized":"2023-10-19T13:00:22.221258+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:50:26.459690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"86'2","reported_seq":14,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.220954+0000","last_change":"2023-10-19T13:00:20.188312+0000","last_active":"2023-10-19T13:00:22.220954+0000","last_peered":"2023-10-19T13:00:22.220954+0000","last_clean":"2023-10-19T13:00:22.220954+0000","last_became_active":"2023-10-19T13:00:20.187673+0000","last_became_peered":"2023-10-19T13:00:20.187673+0000","last_unstale":"2023-10-19T13:00:22.220954+0000","last_undegraded":"2023-10-19T13:00:22.220954+0000","last_fullsized":"2023-10-19T13:00:22.220954+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:22:13.247630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.222925+0000","last_change":"2023-10-19T13:00:21.232599+0000","last_active":"2023-10-19T13:00:22.222925+0000","last_peered":"2023-10-19T13:00:22.222925+0000","last_clean":"2023-10-19T13:00:22.222925+0000","last_became_active":"2023-10-19T13:00:21.231174+0000","last_became_peered":"2023-10-19T13:00:21.231174+0000","last_unstale":"2023-10-19T13:00:22.222925+0000","last_undegraded":"2023-10-19T13:00:22.222925+0000","last_fullsized":"2023-10-19T13:00:22.222925+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:22:14.250042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"86'2","reported_seq":14,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.221016+0000","last_change":"2023-10-19T13:00:20.197923+0000","last_active":"2023-10-19T13:00:22.221016+0000","last_peered":"2023-10-19T13:00:22.221016+0000","last_clean":"2023-10-19T13:00:22.221016+0000","last_became_active":"2023-10-19T13:00:20.197546+0000","last_became_peered":"2023-10-19T13:00:20.197546+0000","last_unstale":"2023-10-19T13:00:22.221016+0000","last_undegraded":"2023-10-19T13:00:22.221016+0000","last_fullsized":"2023-10-19T13:00:22.221016+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:27:45.352182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:23.044381+0000","last_change":"2023-10-19T13:00:21.229195+0000","last_active":"2023-10-19T13:00:23.044381+0000","last_peered":"2023-10-19T13:00:23.044381+0000","last_clean":"2023-10-19T13:00:23.044381+0000","last_became_active":"2023-10-19T13:00:21.228898+0000","last_became_peered":"2023-10-19T13:00:21.228898+0000","last_unstale":"2023-10-19T13:00:23.044381+0000","last_undegraded":"2023-10-19T13:00:23.044381+0000","last_fullsized":"2023-10-19T13:00:23.044381+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:28:07.228859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"86'1","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.222099+0000","last_change":"2023-10-19T13:00:20.187648+0000","last_active":"2023-10-19T13:00:22.222099+0000","last_peered":"2023-10-19T13:00:22.222099+0000","last_clean":"2023-10-19T13:00:22.222099+0000","last_became_active":"2023-10-19T13:00:20.187366+0000","last_became_peered":"2023-10-19T13:00:20.187366+0000","last_unstale":"2023-10-19T13:00:22.222099+0000","last_undegraded":"2023-10-19T13:00:22.222099+0000","last_fullsized":"2023-10-19T13:00:22.222099+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:04:54.169253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":9,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-19T13:00:21.232881+0000","last_change":"2023-10-19T13:00:21.226995+0000","last_active":"2023-10-19T13:00:21.232881+0000","last_peered":"2023-10-19T13:00:21.232881+0000","last_clean":"2023-10-19T13:00:21.232881+0000","last_became_active":"2023-10-19T13:00:21.226673+0000","last_became_peered":"2023-10-19T13:00:21.226673+0000","last_unstale":"2023-10-19T13:00:21.232881+0000","last_undegraded":"2023-10-19T13:00:21.232881+0000","last_fullsized":"2023-10-19T13:00:21.232881+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:52:56.240367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1a","version":"86'1","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.221892+0000","last_change":"2023-10-19T13:00:20.187521+0000","last_active":"2023-10-19T13:00:22.221892+0000","last_peered":"2023-10-19T13:00:22.221892+0000","last_clean":"2023-10-19T13:00:22.221892+0000","last_became_active":"2023-10-19T13:00:20.186630+0000","last_became_peered":"2023-10-19T13:00:20.186630+0000","last_unstale":"2023-10-19T13:00:22.221892+0000","last_undegraded":"2023-10-19T13:00:22.221892+0000","last_fullsized":"2023-10-19T13:00:22.221892+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:37:14.044000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.423329+0000","last_change":"2023-10-19T13:00:21.219987+0000","last_active":"2023-10-19T13:00:22.423329+0000","last_peered":"2023-10-19T13:00:22.423329+0000","last_clean":"2023-10-19T13:00:22.423329+0000","last_became_active":"2023-10-19T13:00:21.218983+0000","last_became_peered":"2023-10-19T13:00:21.218983+0000","last_unstale":"2023-10-19T13:00:22.423329+0000","last_undegraded":"2023-10-19T13:00:22.423329+0000","last_fullsized":"2023-10-19T13:00:22.423329+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:31:57.359031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.221811+0000","last_change":"2023-10-19T13:00:20.196675+0000","last_active":"2023-10-19T13:00:22.221811+0000","last_peered":"2023-10-19T13:00:22.221811+0000","last_clean":"2023-10-19T13:00:22.221811+0000","last_became_active":"2023-10-19T13:00:20.196319+0000","last_became_peered":"2023-10-19T13:00:20.196319+0000","last_unstale":"2023-10-19T13:00:22.221811+0000","last_undegraded":"2023-10-19T13:00:22.221811+0000","last_fullsized":"2023-10-19T13:00:22.221811+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:28:05.111808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.222072+0000","last_change":"2023-10-19T13:00:21.222186+0000","last_active":"2023-10-19T13:00:22.222072+0000","last_peered":"2023-10-19T13:00:22.222072+0000","last_clean":"2023-10-19T13:00:22.222072+0000","last_became_active":"2023-10-19T13:00:21.221912+0000","last_became_peered":"2023-10-19T13:00:21.221912+0000","last_unstale":"2023-10-19T13:00:22.222072+0000","last_undegraded":"2023-10-19T13:00:22.222072+0000","last_fullsized":"2023-10-19T13:00:22.222072+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:48:53.902328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.423270+0000","last_change":"2023-10-19T13:00:20.198229+0000","last_active":"2023-10-19T13:00:22.423270+0000","last_peered":"2023-10-19T13:00:22.423270+0000","last_clean":"2023-10-19T13:00:22.423270+0000","last_became_active":"2023-10-19T13:00:20.197850+0000","last_became_peered":"2023-10-19T13:00:20.197850+0000","last_unstale":"2023-10-19T13:00:22.423270+0000","last_undegraded":"2023-10-19T13:00:22.423270+0000","last_fullsized":"2023-10-19T13:00:22.423270+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:58:20.558973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":9,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-19T13:00:21.233976+0000","last_change":"2023-10-19T13:00:21.227321+0000","last_active":"2023-10-19T13:00:21.233976+0000","last_peered":"2023-10-19T13:00:21.233976+0000","last_clean":"2023-10-19T13:00:21.233976+0000","last_became_active":"2023-10-19T13:00:21.227028+0000","last_became_peered":"2023-10-19T13:00:21.227028+0000","last_unstale":"2023-10-19T13:00:21.233976+0000","last_undegraded":"2023-10-19T13:00:21.233976+0000","last_fullsized":"2023-10-19T13:00:21.233976+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:06:08.406600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.223055+0000","last_change":"2023-10-19T13:00:20.202547+0000","last_active":"2023-10-19T13:00:22.223055+0000","last_peered":"2023-10-19T13:00:22.223055+0000","last_clean":"2023-10-19T13:00:22.223055+0000","last_became_active":"2023-10-19T13:00:20.202218+0000","last_became_peered":"2023-10-19T13:00:20.202218+0000","last_unstale":"2023-10-19T13:00:22.223055+0000","last_undegraded":"2023-10-19T13:00:22.223055+0000","last_fullsized":"2023-10-19T13:00:22.223055+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:41:30.219455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.221899+0000","last_change":"2023-10-19T13:00:21.218934+0000","last_active":"2023-10-19T13:00:22.221899+0000","last_peered":"2023-10-19T13:00:22.221899+0000","last_clean":"2023-10-19T13:00:22.221899+0000","last_became_active":"2023-10-19T13:00:21.218623+0000","last_became_peered":"2023-10-19T13:00:21.218623+0000","last_unstale":"2023-10-19T13:00:22.221899+0000","last_undegraded":"2023-10-19T13:00:22.221899+0000","last_fullsized":"2023-10-19T13:00:22.221899+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:08:57.610716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.385036+0000","last_change":"2023-10-19T13:00:20.198305+0000","last_active":"2023-10-19T13:00:22.385036+0000","last_peered":"2023-10-19T13:00:22.385036+0000","last_clean":"2023-10-19T13:00:22.385036+0000","last_became_active":"2023-10-19T13:00:20.198010+0000","last_became_peered":"2023-10-19T13:00:20.198010+0000","last_unstale":"2023-10-19T13:00:22.385036+0000","last_undegraded":"2023-10-19T13:00:22.385036+0000","last_fullsized":"2023-10-19T13:00:22.385036+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:13:59.814299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.423705+0000","last_change":"2023-10-19T13:00:21.217676+0000","last_active":"2023-10-19T13:00:22.423705+0000","last_peered":"2023-10-19T13:00:22.423705+0000","last_clean":"2023-10-19T13:00:22.423705+0000","last_became_active":"2023-10-19T13:00:21.217413+0000","last_became_peered":"2023-10-19T13:00:21.217413+0000","last_unstale":"2023-10-19T13:00:22.423705+0000","last_undegraded":"2023-10-19T13:00:22.423705+0000","last_fullsized":"2023-10-19T13:00:22.423705+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:55:42.567075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.15","version":"86'3","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.222755+0000","last_change":"2023-10-19T13:00:20.202410+0000","last_active":"2023-10-19T13:00:22.222755+0000","last_peered":"2023-10-19T13:00:22.222755+0000","last_clean":"2023-10-19T13:00:22.222755+0000","last_became_active":"2023-10-19T13:00:20.202053+0000","last_became_peered":"2023-10-19T13:00:20.202053+0000","last_unstale":"2023-10-19T13:00:22.222755+0000","last_undegraded":"2023-10-19T13:00:22.222755+0000","last_fullsized":"2023-10-19T13:00:22.222755+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:20:39.435252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.385090+0000","last_change":"2023-10-19T13:00:21.224440+0000","last_active":"2023-10-19T13:00:22.385090+0000","last_peered":"2023-10-19T13:00:22.385090+0000","last_clean":"2023-10-19T13:00:22.385090+0000","last_became_active":"2023-10-19T13:00:21.224089+0000","last_became_peered":"2023-10-19T13:00:21.224089+0000","last_unstale":"2023-10-19T13:00:22.385090+0000","last_undegraded":"2023-10-19T13:00:22.385090+0000","last_fullsized":"2023-10-19T13:00:22.385090+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:25:21.025263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"86'6","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-19T13:00:21.240123+0000","last_change":"2023-10-19T13:00:20.198504+0000","last_active":"2023-10-19T13:00:21.240123+0000","last_peered":"2023-10-19T13:00:21.240123+0000","last_clean":"2023-10-19T13:00:21.240123+0000","last_became_active":"2023-10-19T13:00:20.197655+0000","last_became_peered":"2023-10-19T13:00:20.197655+0000","last_unstale":"2023-10-19T13:00:21.240123+0000","last_undegraded":"2023-10-19T13:00:21.240123+0000","last_fullsized":"2023-10-19T13:00:21.240123+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:25:20.022851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.384529+0000","last_change":"2023-10-19T13:00:21.214443+0000","last_active":"2023-10-19T13:00:22.384529+0000","last_peered":"2023-10-19T13:00:22.384529+0000","last_clean":"2023-10-19T13:00:22.384529+0000","last_became_active":"2023-10-19T13:00:21.214184+0000","last_became_peered":"2023-10-19T13:00:21.214184+0000","last_unstale":"2023-10-19T13:00:22.384529+0000","last_undegraded":"2023-10-19T13:00:22.384529+0000","last_fullsized":"2023-10-19T13:00:22.384529+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:06:08.406600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.13","version":"86'2","reported_seq":14,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.222611+0000","last_change":"2023-10-19T13:00:20.196998+0000","last_active":"2023-10-19T13:00:22.222611+0000","last_peered":"2023-10-19T13:00:22.222611+0000","last_clean":"2023-10-19T13:00:22.222611+0000","last_became_active":"2023-10-19T13:00:20.196205+0000","last_became_peered":"2023-10-19T13:00:20.196205+0000","last_unstale":"2023-10-19T13:00:22.222611+0000","last_undegraded":"2023-10-19T13:00:22.222611+0000","last_fullsized":"2023-10-19T13:00:22.222611+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:29:02.236981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:23.044542+0000","last_change":"2023-10-19T13:00:21.218889+0000","last_active":"2023-10-19T13:00:23.044542+0000","last_peered":"2023-10-19T13:00:23.044542+0000","last_clean":"2023-10-19T13:00:23.044542+0000","last_became_active":"2023-10-19T13:00:21.218604+0000","last_became_peered":"2023-10-19T13:00:21.218604+0000","last_unstale":"2023-10-19T13:00:23.044542+0000","last_undegraded":"2023-10-19T13:00:23.044542+0000","last_fullsized":"2023-10-19T13:00:23.044542+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:00:49.808529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:23.044575+0000","last_change":"2023-10-19T13:00:20.197561+0000","last_active":"2023-10-19T13:00:23.044575+0000","last_peered":"2023-10-19T13:00:23.044575+0000","last_clean":"2023-10-19T13:00:23.044575+0000","last_became_active":"2023-10-19T13:00:20.197265+0000","last_became_peered":"2023-10-19T13:00:20.197265+0000","last_unstale":"2023-10-19T13:00:23.044575+0000","last_undegraded":"2023-10-19T13:00:23.044575+0000","last_fullsized":"2023-10-19T13:00:23.044575+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:20:48.999042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.423360+0000","last_change":"2023-10-19T13:00:21.207696+0000","last_active":"2023-10-19T13:00:22.423360+0000","last_peered":"2023-10-19T13:00:22.423360+0000","last_clean":"2023-10-19T13:00:22.423360+0000","last_became_active":"2023-10-19T13:00:21.207276+0000","last_became_peered":"2023-10-19T13:00:21.207276+0000","last_unstale":"2023-10-19T13:00:22.423360+0000","last_undegraded":"2023-10-19T13:00:22.423360+0000","last_fullsized":"2023-10-19T13:00:22.423360+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:03:44.562412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.222526+0000","last_change":"2023-10-19T13:00:20.197213+0000","last_active":"2023-10-19T13:00:22.222526+0000","last_peered":"2023-10-19T13:00:22.222526+0000","last_clean":"2023-10-19T13:00:22.222526+0000","last_became_active":"2023-10-19T13:00:20.196943+0000","last_became_peered":"2023-10-19T13:00:20.196943+0000","last_unstale":"2023-10-19T13:00:22.222526+0000","last_undegraded":"2023-10-19T13:00:22.222526+0000","last_fullsized":"2023-10-19T13:00:22.222526+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:05:35.715244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":9,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-19T13:00:21.232936+0000","last_change":"2023-10-19T13:00:21.228024+0000","last_active":"2023-10-19T13:00:21.232936+0000","last_peered":"2023-10-19T13:00:21.232936+0000","last_clean":"2023-10-19T13:00:21.232936+0000","last_became_active":"2023-10-19T13:00:21.227727+0000","last_became_peered":"2023-10-19T13:00:21.227727+0000","last_unstale":"2023-10-19T13:00:21.232936+0000","last_undegraded":"2023-10-19T13:00:21.232936+0000","last_fullsized":"2023-10-19T13:00:21.232936+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:14:55.381409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.10","version":"86'8","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.384437+0000","last_change":"2023-10-19T13:00:20.195915+0000","last_active":"2023-10-19T13:00:22.384437+0000","last_peered":"2023-10-19T13:00:22.384437+0000","last_clean":"2023-10-19T13:00:22.384437+0000","last_became_active":"2023-10-19T13:00:20.195641+0000","last_became_peered":"2023-10-19T13:00:20.195641+0000","last_unstale":"2023-10-19T13:00:22.384437+0000","last_undegraded":"2023-10-19T13:00:22.384437+0000","last_fullsized":"2023-10-19T13:00:22.384437+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:52:53.680755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.222779+0000","last_change":"2023-10-19T13:00:21.226358+0000","last_active":"2023-10-19T13:00:22.222779+0000","last_peered":"2023-10-19T13:00:22.222779+0000","last_clean":"2023-10-19T13:00:22.222779+0000","last_became_active":"2023-10-19T13:00:21.226094+0000","last_became_peered":"2023-10-19T13:00:21.226094+0000","last_unstale":"2023-10-19T13:00:22.222779+0000","last_undegraded":"2023-10-19T13:00:22.222779+0000","last_fullsized":"2023-10-19T13:00:22.222779+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:12:58.144772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"86'1","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:23.044984+0000","last_change":"2023-10-19T13:00:20.198629+0000","last_active":"2023-10-19T13:00:23.044984+0000","last_peered":"2023-10-19T13:00:23.044984+0000","last_clean":"2023-10-19T13:00:23.044984+0000","last_became_active":"2023-10-19T13:00:20.198381+0000","last_became_peered":"2023-10-19T13:00:20.198381+0000","last_unstale":"2023-10-19T13:00:23.044984+0000","last_undegraded":"2023-10-19T13:00:23.044984+0000","last_fullsized":"2023-10-19T13:00:23.044984+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:18:37.647497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.423742+0000","last_change":"2023-10-19T13:00:21.215295+0000","last_active":"2023-10-19T13:00:22.423742+0000","last_peered":"2023-10-19T13:00:22.423742+0000","last_clean":"2023-10-19T13:00:22.423742+0000","last_became_active":"2023-10-19T13:00:21.214996+0000","last_became_peered":"2023-10-19T13:00:21.214996+0000","last_unstale":"2023-10-19T13:00:22.423742+0000","last_undegraded":"2023-10-19T13:00:22.423742+0000","last_fullsized":"2023-10-19T13:00:22.423742+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:58:41.166338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"86'2","reported_seq":14,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.222910+0000","last_change":"2023-10-19T13:00:20.196057+0000","last_active":"2023-10-19T13:00:22.222910+0000","last_peered":"2023-10-19T13:00:22.222910+0000","last_clean":"2023-10-19T13:00:22.222910+0000","last_became_active":"2023-10-19T13:00:20.195717+0000","last_became_peered":"2023-10-19T13:00:20.195717+0000","last_unstale":"2023-10-19T13:00:22.222910+0000","last_undegraded":"2023-10-19T13:00:22.222910+0000","last_fullsized":"2023-10-19T13:00:22.222910+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:11:52.601802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.384933+0000","last_change":"2023-10-19T13:00:21.213116+0000","last_active":"2023-10-19T13:00:22.384933+0000","last_peered":"2023-10-19T13:00:22.384933+0000","last_clean":"2023-10-19T13:00:22.384933+0000","last_became_active":"2023-10-19T13:00:21.212842+0000","last_became_peered":"2023-10-19T13:00:21.212842+0000","last_unstale":"2023-10-19T13:00:22.384933+0000","last_undegraded":"2023-10-19T13:00:22.384933+0000","last_fullsized":"2023-10-19T13:00:22.384933+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:55:42.567075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"86'1","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.222864+0000","last_change":"2023-10-19T13:00:20.202706+0000","last_active":"2023-10-19T13:00:22.222864+0000","last_peered":"2023-10-19T13:00:22.222864+0000","last_clean":"2023-10-19T13:00:22.222864+0000","last_became_active":"2023-10-19T13:00:20.202398+0000","last_became_peered":"2023-10-19T13:00:20.202398+0000","last_unstale":"2023-10-19T13:00:22.222864+0000","last_undegraded":"2023-10-19T13:00:22.222864+0000","last_fullsized":"2023-10-19T13:00:22.222864+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:24:33.951829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.423732+0000","last_change":"2023-10-19T13:00:21.223843+0000","last_active":"2023-10-19T13:00:22.423732+0000","last_peered":"2023-10-19T13:00:22.423732+0000","last_clean":"2023-10-19T13:00:22.423732+0000","last_became_active":"2023-10-19T13:00:21.223493+0000","last_became_peered":"2023-10-19T13:00:21.223493+0000","last_unstale":"2023-10-19T13:00:22.423732+0000","last_undegraded":"2023-10-19T13:00:22.423732+0000","last_fullsized":"2023-10-19T13:00:22.423732+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:14:55.381409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.222072+0000","last_change":"2023-10-19T13:00:20.193764+0000","last_active":"2023-10-19T13:00:22.222072+0000","last_peered":"2023-10-19T13:00:22.222072+0000","last_clean":"2023-10-19T13:00:22.222072+0000","last_became_active":"2023-10-19T13:00:20.193512+0000","last_became_peered":"2023-10-19T13:00:20.193512+0000","last_unstale":"2023-10-19T13:00:22.222072+0000","last_undegraded":"2023-10-19T13:00:22.222072+0000","last_fullsized":"2023-10-19T13:00:22.222072+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:02:09.714383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.220733+0000","last_change":"2023-10-19T13:00:21.217141+0000","last_active":"2023-10-19T13:00:22.220733+0000","last_peered":"2023-10-19T13:00:22.220733+0000","last_clean":"2023-10-19T13:00:22.220733+0000","last_became_active":"2023-10-19T13:00:21.216807+0000","last_became_peered":"2023-10-19T13:00:21.216807+0000","last_unstale":"2023-10-19T13:00:22.220733+0000","last_undegraded":"2023-10-19T13:00:22.220733+0000","last_fullsized":"2023-10-19T13:00:22.220733+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:39:34.098405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"86'2","reported_seq":14,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.220911+0000","last_change":"2023-10-19T13:00:20.195565+0000","last_active":"2023-10-19T13:00:22.220911+0000","last_peered":"2023-10-19T13:00:22.220911+0000","last_clean":"2023-10-19T13:00:22.220911+0000","last_became_active":"2023-10-19T13:00:20.195256+0000","last_became_peered":"2023-10-19T13:00:20.195256+0000","last_unstale":"2023-10-19T13:00:22.220911+0000","last_undegraded":"2023-10-19T13:00:22.220911+0000","last_fullsized":"2023-10-19T13:00:22.220911+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:18:37.647497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:23.044579+0000","last_change":"2023-10-19T13:00:21.220030+0000","last_active":"2023-10-19T13:00:23.044579+0000","last_peered":"2023-10-19T13:00:23.044579+0000","last_clean":"2023-10-19T13:00:23.044579+0000","last_became_active":"2023-10-19T13:00:21.219790+0000","last_became_peered":"2023-10-19T13:00:21.219790+0000","last_unstale":"2023-10-19T13:00:23.044579+0000","last_undegraded":"2023-10-19T13:00:23.044579+0000","last_fullsized":"2023-10-19T13:00:23.044579+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:00:42.269401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:23.044670+0000","last_change":"2023-10-19T13:00:20.197800+0000","last_active":"2023-10-19T13:00:23.044670+0000","last_peered":"2023-10-19T13:00:23.044670+0000","last_clean":"2023-10-19T13:00:23.044670+0000","last_became_active":"2023-10-19T13:00:20.197482+0000","last_became_peered":"2023-10-19T13:00:20.197482+0000","last_unstale":"2023-10-19T13:00:23.044670+0000","last_undegraded":"2023-10-19T13:00:23.044670+0000","last_fullsized":"2023-10-19T13:00:23.044670+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:11:52.601802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.423437+0000","last_change":"2023-10-19T13:00:21.220368+0000","last_active":"2023-10-19T13:00:22.423437+0000","last_peered":"2023-10-19T13:00:22.423437+0000","last_clean":"2023-10-19T13:00:22.423437+0000","last_became_active":"2023-10-19T13:00:21.219726+0000","last_became_peered":"2023-10-19T13:00:21.219726+0000","last_unstale":"2023-10-19T13:00:22.423437+0000","last_undegraded":"2023-10-19T13:00:22.423437+0000","last_fullsized":"2023-10-19T13:00:22.423437+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:06:08.406600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.220731+0000","last_change":"2023-10-19T13:00:20.194028+0000","last_active":"2023-10-19T13:00:22.220731+0000","last_peered":"2023-10-19T13:00:22.220731+0000","last_clean":"2023-10-19T13:00:22.220731+0000","last_became_active":"2023-10-19T13:00:20.193758+0000","last_became_peered":"2023-10-19T13:00:20.193758+0000","last_unstale":"2023-10-19T13:00:22.220731+0000","last_undegraded":"2023-10-19T13:00:22.220731+0000","last_fullsized":"2023-10-19T13:00:22.220731+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:41:30.219455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.221024+0000","last_change":"2023-10-19T13:00:21.221247+0000","last_active":"2023-10-19T13:00:22.221024+0000","last_peered":"2023-10-19T13:00:22.221024+0000","last_clean":"2023-10-19T13:00:22.221024+0000","last_became_active":"2023-10-19T13:00:21.220182+0000","last_became_peered":"2023-10-19T13:00:21.220182+0000","last_unstale":"2023-10-19T13:00:22.221024+0000","last_undegraded":"2023-10-19T13:00:22.221024+0000","last_fullsized":"2023-10-19T13:00:22.221024+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:47:08.944926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"86'2","reported_seq":14,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:23.045126+0000","last_change":"2023-10-19T13:00:20.193049+0000","last_active":"2023-10-19T13:00:23.045126+0000","last_peered":"2023-10-19T13:00:23.045126+0000","last_clean":"2023-10-19T13:00:23.045126+0000","last_became_active":"2023-10-19T13:00:20.192788+0000","last_became_peered":"2023-10-19T13:00:20.192788+0000","last_unstale":"2023-10-19T13:00:23.045126+0000","last_undegraded":"2023-10-19T13:00:23.045126+0000","last_fullsized":"2023-10-19T13:00:23.045126+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:07:06.063303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:23.044679+0000","last_change":"2023-10-19T13:00:21.222511+0000","last_active":"2023-10-19T13:00:23.044679+0000","last_peered":"2023-10-19T13:00:23.044679+0000","last_clean":"2023-10-19T13:00:23.044679+0000","last_became_active":"2023-10-19T13:00:21.222236+0000","last_became_peered":"2023-10-19T13:00:21.222236+0000","last_unstale":"2023-10-19T13:00:23.044679+0000","last_undegraded":"2023-10-19T13:00:23.044679+0000","last_fullsized":"2023-10-19T13:00:23.044679+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:50:46.028354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:23.045102+0000","last_change":"2023-10-19T13:00:20.186527+0000","last_active":"2023-10-19T13:00:23.045102+0000","last_peered":"2023-10-19T13:00:23.045102+0000","last_clean":"2023-10-19T13:00:23.045102+0000","last_became_active":"2023-10-19T13:00:20.186271+0000","last_became_peered":"2023-10-19T13:00:20.186271+0000","last_unstale":"2023-10-19T13:00:23.045102+0000","last_undegraded":"2023-10-19T13:00:23.045102+0000","last_fullsized":"2023-10-19T13:00:23.045102+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:26:39.899099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:23.044992+0000","last_change":"2023-10-19T13:00:21.223718+0000","last_active":"2023-10-19T13:00:23.044992+0000","last_peered":"2023-10-19T13:00:23.044992+0000","last_clean":"2023-10-19T13:00:23.044992+0000","last_became_active":"2023-10-19T13:00:21.223259+0000","last_became_peered":"2023-10-19T13:00:21.223259+0000","last_unstale":"2023-10-19T13:00:23.044992+0000","last_undegraded":"2023-10-19T13:00:23.044992+0000","last_fullsized":"2023-10-19T13:00:23.044992+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:55:17.988760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.221031+0000","last_change":"2023-10-19T13:00:20.188530+0000","last_active":"2023-10-19T13:00:22.221031+0000","last_peered":"2023-10-19T13:00:22.221031+0000","last_clean":"2023-10-19T13:00:22.221031+0000","last_became_active":"2023-10-19T13:00:20.188074+0000","last_became_peered":"2023-10-19T13:00:20.188074+0000","last_unstale":"2023-10-19T13:00:22.221031+0000","last_undegraded":"2023-10-19T13:00:22.221031+0000","last_fullsized":"2023-10-19T13:00:22.221031+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:08:44.211054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.423799+0000","last_change":"2023-10-19T13:00:21.223166+0000","last_active":"2023-10-19T13:00:22.423799+0000","last_peered":"2023-10-19T13:00:22.423799+0000","last_clean":"2023-10-19T13:00:22.423799+0000","last_became_active":"2023-10-19T13:00:21.222906+0000","last_became_peered":"2023-10-19T13:00:21.222906+0000","last_unstale":"2023-10-19T13:00:22.423799+0000","last_undegraded":"2023-10-19T13:00:22.423799+0000","last_fullsized":"2023-10-19T13:00:22.423799+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:37:52.369486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-19T13:00:21.216870+0000","last_change":"2023-10-19T13:00:20.198569+0000","last_active":"2023-10-19T13:00:21.216870+0000","last_peered":"2023-10-19T13:00:21.216870+0000","last_clean":"2023-10-19T13:00:21.216870+0000","last_became_active":"2023-10-19T13:00:20.197088+0000","last_became_peered":"2023-10-19T13:00:20.197088+0000","last_unstale":"2023-10-19T13:00:21.216870+0000","last_undegraded":"2023-10-19T13:00:21.216870+0000","last_fullsized":"2023-10-19T13:00:21.216870+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:38:39.386315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.423286+0000","last_change":"2023-10-19T13:00:21.232429+0000","last_active":"2023-10-19T13:00:22.423286+0000","last_peered":"2023-10-19T13:00:22.423286+0000","last_clean":"2023-10-19T13:00:22.423286+0000","last_became_active":"2023-10-19T13:00:21.232143+0000","last_became_peered":"2023-10-19T13:00:21.232143+0000","last_unstale":"2023-10-19T13:00:22.423286+0000","last_undegraded":"2023-10-19T13:00:22.423286+0000","last_fullsized":"2023-10-19T13:00:22.423286+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:27:17.189492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.222621+0000","last_change":"2023-10-19T13:00:20.198187+0000","last_active":"2023-10-19T13:00:22.222621+0000","last_peered":"2023-10-19T13:00:22.222621+0000","last_clean":"2023-10-19T13:00:22.222621+0000","last_became_active":"2023-10-19T13:00:20.196844+0000","last_became_peered":"2023-10-19T13:00:20.196844+0000","last_unstale":"2023-10-19T13:00:22.222621+0000","last_undegraded":"2023-10-19T13:00:22.222621+0000","last_fullsized":"2023-10-19T13:00:22.222621+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:20:48.999042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.384657+0000","last_change":"2023-10-19T13:00:21.223216+0000","last_active":"2023-10-19T13:00:22.384657+0000","last_peered":"2023-10-19T13:00:22.384657+0000","last_clean":"2023-10-19T13:00:22.384657+0000","last_became_active":"2023-10-19T13:00:21.222881+0000","last_became_peered":"2023-10-19T13:00:21.222881+0000","last_unstale":"2023-10-19T13:00:22.384657+0000","last_undegraded":"2023-10-19T13:00:22.384657+0000","last_fullsized":"2023-10-19T13:00:22.384657+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:41:19.821669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.221947+0000","last_change":"2023-10-19T13:00:20.192942+0000","last_active":"2023-10-19T13:00:22.221947+0000","last_peered":"2023-10-19T13:00:22.221947+0000","last_clean":"2023-10-19T13:00:22.221947+0000","last_became_active":"2023-10-19T13:00:20.192688+0000","last_became_peered":"2023-10-19T13:00:20.192688+0000","last_unstale":"2023-10-19T13:00:22.221947+0000","last_undegraded":"2023-10-19T13:00:22.221947+0000","last_fullsized":"2023-10-19T13:00:22.221947+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:16:31.679428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":9,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-19T13:00:21.227472+0000","last_change":"2023-10-19T13:00:21.226366+0000","last_active":"2023-10-19T13:00:21.227472+0000","last_peered":"2023-10-19T13:00:21.227472+0000","last_clean":"2023-10-19T13:00:21.227472+0000","last_became_active":"2023-10-19T13:00:21.226075+0000","last_became_peered":"2023-10-19T13:00:21.226075+0000","last_unstale":"2023-10-19T13:00:21.227472+0000","last_undegraded":"2023-10-19T13:00:21.227472+0000","last_fullsized":"2023-10-19T13:00:21.227472+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:33:36.402003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.221906+0000","last_change":"2023-10-19T13:00:20.195720+0000","last_active":"2023-10-19T13:00:22.221906+0000","last_peered":"2023-10-19T13:00:22.221906+0000","last_clean":"2023-10-19T13:00:22.221906+0000","last_became_active":"2023-10-19T13:00:20.195451+0000","last_became_peered":"2023-10-19T13:00:20.195451+0000","last_unstale":"2023-10-19T13:00:22.221906+0000","last_undegraded":"2023-10-19T13:00:22.221906+0000","last_fullsized":"2023-10-19T13:00:22.221906+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:35:48.984686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:23.044464+0000","last_change":"2023-10-19T13:00:21.219143+0000","last_active":"2023-10-19T13:00:23.044464+0000","last_peered":"2023-10-19T13:00:23.044464+0000","last_clean":"2023-10-19T13:00:23.044464+0000","last_became_active":"2023-10-19T13:00:21.218834+0000","last_became_peered":"2023-10-19T13:00:21.218834+0000","last_unstale":"2023-10-19T13:00:23.044464+0000","last_undegraded":"2023-10-19T13:00:23.044464+0000","last_fullsized":"2023-10-19T13:00:23.044464+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:29:26.836263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-19T13:00:21.215556+0000","last_change":"2023-10-19T13:00:20.198235+0000","last_active":"2023-10-19T13:00:21.215556+0000","last_peered":"2023-10-19T13:00:21.215556+0000","last_clean":"2023-10-19T13:00:21.215556+0000","last_became_active":"2023-10-19T13:00:20.197920+0000","last_became_peered":"2023-10-19T13:00:20.197920+0000","last_unstale":"2023-10-19T13:00:21.215556+0000","last_undegraded":"2023-10-19T13:00:21.215556+0000","last_fullsized":"2023-10-19T13:00:21.215556+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:34:56.789811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.423517+0000","last_change":"2023-10-19T13:00:21.231904+0000","last_active":"2023-10-19T13:00:22.423517+0000","last_peered":"2023-10-19T13:00:22.423517+0000","last_clean":"2023-10-19T13:00:22.423517+0000","last_became_active":"2023-10-19T13:00:21.231629+0000","last_became_peered":"2023-10-19T13:00:21.231629+0000","last_unstale":"2023-10-19T13:00:22.423517+0000","last_undegraded":"2023-10-19T13:00:22.423517+0000","last_fullsized":"2023-10-19T13:00:22.423517+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:41:19.821669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.221722+0000","last_change":"2023-10-19T13:00:20.187569+0000","last_active":"2023-10-19T13:00:22.221722+0000","last_peered":"2023-10-19T13:00:22.221722+0000","last_clean":"2023-10-19T13:00:22.221722+0000","last_became_active":"2023-10-19T13:00:20.187271+0000","last_became_peered":"2023-10-19T13:00:20.187271+0000","last_unstale":"2023-10-19T13:00:22.221722+0000","last_undegraded":"2023-10-19T13:00:22.221722+0000","last_fullsized":"2023-10-19T13:00:22.221722+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:51:04.372619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"76'161","reported_seq":385,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-19T13:00:21.215648+0000","last_change":"2023-10-19T12:44:08.636613+0000","last_active":"2023-10-19T13:00:21.215648+0000","last_peered":"2023-10-19T13:00:21.215648+0000","last_clean":"2023-10-19T13:00:21.215648+0000","last_became_active":"2023-10-19T12:44:08.635697+0000","last_became_peered":"2023-10-19T12:44:08.635697+0000","last_unstale":"2023-10-19T13:00:21.215648+0000","last_undegraded":"2023-10-19T13:00:21.215648+0000","last_fullsized":"2023-10-19T13:00:21.215648+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_clean_scrub_stamp":"2023-10-19T12:44:07.399738+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:15:23.685485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575046,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59184,"kb_used_data":1628,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312656,"statfs":{"total":96636764160,"available":96576159744,"internally_reserved":0,"allocated":1667072,"data_stored":1120016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58915275},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:59:44 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.57499999999999996,"15min":0.56999999999999995},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.248},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.60099999999999998,"15min":0.58799999999999997},"min":{"1min":0.45400000000000001,"5min":0.33200000000000002,"15min":0.26900000000000002},"max":{"1min":0.78200000000000003,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.36499999999999999}]},{"osd":1,"last update":"Thu Oct 19 12:59:44 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.58199999999999996,"15min":0.57599999999999996},"min":{"1min":0.41299999999999998,"5min":0.40300000000000002,"15min":0.28199999999999997},"max":{"1min":0.84299999999999997,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.623,"5min":0.59999999999999998,"15min":0.58799999999999997},"min":{"1min":0.51700000000000002,"5min":0.43099999999999999,"15min":0.29299999999999998},"max":{"1min":0.82699999999999996,"5min":1.284,"15min":1.284},"last":0.34399999999999997}]},{"osd":2,"last update":"Thu Oct 19 12:59:44 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.64100000000000001,"15min":0.629},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.375},"max":{"1min":0.79300000000000004,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.64600000000000002,"15min":0.63300000000000001},"min":{"1min":0.55300000000000005,"5min":0.41599999999999998,"15min":0.28399999999999997},"max":{"1min":0.81200000000000006,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.5}]},{"osd":3,"last update":"Thu Oct 19 12:59:44 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60699999999999998,"15min":0.60099999999999998},"min":{"1min":0.47199999999999998,"5min":0.34999999999999998,"15min":0.34399999999999997},"max":{"1min":0.78900000000000003,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.67000000000000004,"15min":0.65500000000000003},"min":{"1min":0.58199999999999996,"5min":0.45300000000000001,"15min":0.39700000000000002},"max":{"1min":0.92000000000000004,"5min":1.329,"15min":1.329},"last":0.51900000000000002}]},{"osd":4,"last update":"Thu Oct 19 12:59:44 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.46800000000000003,"15min":0.46700000000000003},"min":{"1min":0.36899999999999999,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.72599999999999998,"5min":0.75800000000000001,"15min":0.80500000000000005},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.33600000000000002,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.623,"5min":0.75600000000000001,"15min":0.78900000000000003},"last":0.57699999999999996}]},{"osd":5,"last update":"Thu Oct 19 12:59:44 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.48299999999999998,"15min":0.48499999999999999},"min":{"1min":0.34399999999999997,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.68600000000000005,"5min":0.75800000000000001,"15min":0.90400000000000003},"last":0.438},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.52200000000000002,"15min":0.51600000000000001},"min":{"1min":0.38,"5min":0.314,"15min":0.308},"max":{"1min":0.71499999999999997,"5min":0.78700000000000003,"15min":0.92200000000000004},"last":0.45000000000000001}]},{"osd":6,"last update":"Thu Oct 19 12:59:44 2023","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.54200000000000004,"15min":0.54700000000000004},"min":{"1min":0.379,"5min":0.308,"15min":0.308},"max":{"1min":0.749,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.52100000000000002,"15min":0.52900000000000003},"min":{"1min":0.36899999999999999,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.76900000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.54100000000000004}]}]},{"osd":6,"up_from":13,"seq":55834575046,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32212,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339628,"statfs":{"total":96636764160,"available":96603779072,"internally_reserved":0,"allocated":1048576,"data_stored":522258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:59:28 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.57999999999999996,"15min":0.58699999999999997},"min":{"1min":0.41899999999999998,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.755,"5min":0.79900000000000004,"15min":0.81999999999999995},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.41999999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.52200000000000002}]},{"osd":1,"last update":"Thu Oct 19 12:59:28 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.60299999999999998,"15min":0.59999999999999998},"min":{"1min":0.39900000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.71999999999999997,"5min":0.83299999999999996,"15min":0.93899999999999995},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.59699999999999998,"15min":0.59999999999999998},"min":{"1min":0.42599999999999999,"5min":0.34300000000000003,"15min":0.28399999999999997},"max":{"1min":0.78200000000000003,"5min":0.82099999999999995,"15min":1.075},"last":0.45000000000000001}]},{"osd":2,"last update":"Thu Oct 19 12:59:28 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.42799999999999999,"5min":0.307,"15min":0.26600000000000001},"max":{"1min":0.84699999999999998,"5min":0.85299999999999998,"15min":0.85699999999999998},"last":0.499},{"interface":"front","average":{"1min":0.624,"5min":0.61299999999999999,"15min":0.60799999999999998},"min":{"1min":0.48499999999999999,"5min":0.35999999999999999,"15min":0.28399999999999997},"max":{"1min":0.82199999999999995,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.72199999999999998}]},{"osd":3,"last update":"Thu Oct 19 12:59:28 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.625,"15min":0.623},"min":{"1min":0.46400000000000002,"5min":0.38500000000000001,"15min":0.32600000000000001},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.61799999999999999,"15min":0.628},"min":{"1min":0.44900000000000001,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":1.0900000000000001},"last":0.65100000000000002}]},{"osd":4,"last update":"Thu Oct 19 12:59:28 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.46100000000000002,"15min":0.46999999999999997},"min":{"1min":0.32900000000000001,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.64400000000000002,"5min":0.77900000000000003,"15min":1.02},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.47999999999999998,"15min":0.48499999999999999},"min":{"1min":0.33200000000000002,"5min":0.32800000000000001,"15min":0.247},"max":{"1min":0.69099999999999995,"5min":0.752,"15min":0.80000000000000004},"last":0.67300000000000004}]},{"osd":5,"last update":"Thu Oct 19 12:59:28 2023","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.46800000000000003,"15min":0.47699999999999998},"min":{"1min":0.32900000000000001,"5min":0.30199999999999999,"15min":0.26600000000000001},"max":{"1min":0.61899999999999999,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.433},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.50800000000000001,"15min":0.51900000000000002},"min":{"1min":0.35699999999999998,"5min":0.32100000000000001,"15min":0.312},"max":{"1min":0.75600000000000001,"5min":0.85699999999999998,"15min":0.98799999999999999},"last":0.747}]},{"osd":7,"last update":"Thu Oct 19 12:59:28 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.51800000000000002,"15min":0.52300000000000002},"min":{"1min":0.32500000000000001,"5min":0.29299999999999998,"15min":0.28199999999999997},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":1.036},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53100000000000003,"15min":0.54100000000000004},"min":{"1min":0.36499999999999999,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":1.1299999999999999},"last":0.58899999999999997}]}]},{"osd":5,"up_from":13,"seq":55834575046,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32204,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339636,"statfs":{"total":96636764160,"available":96603787264,"internally_reserved":0,"allocated":1048576,"data_stored":522246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:59:29 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.58799999999999997,"15min":0.58899999999999997},"min":{"1min":0.44700000000000001,"5min":0.28999999999999998,"15min":0.223},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.88800000000000001},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.63,"5min":0.58999999999999997,"15min":0.59599999999999997},"min":{"1min":0.42499999999999999,"5min":0.36099999999999999,"15min":0.27800000000000002},"max":{"1min":0.81399999999999995,"5min":0.84299999999999997,"15min":0.874},"last":0.47099999999999997}]},{"osd":1,"last update":"Thu Oct 19 12:59:29 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.57899999999999996,"15min":0.59399999999999997},"min":{"1min":0.46899999999999997,"5min":0.34599999999999997,"15min":0.32800000000000001},"max":{"1min":0.77500000000000002,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.59999999999999998,"15min":0.60599999999999998},"min":{"1min":0.49199999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83499999999999996},"last":0.55000000000000004}]},{"osd":2,"last update":"Thu Oct 19 12:59:29 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.57299999999999995,"15min":0.58299999999999996},"min":{"1min":0.48899999999999999,"5min":0.34100000000000003,"15min":0.307},"max":{"1min":0.73999999999999999,"5min":0.78600000000000003,"15min":0.80700000000000005},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.60499999999999998,"15min":0.61899999999999999},"min":{"1min":0.50900000000000001,"5min":0.38400000000000001,"15min":0.33700000000000002},"max":{"1min":0.80900000000000005,"5min":0.88100000000000001,"15min":0.93600000000000005},"last":0.753}]},{"osd":3,"last update":"Thu Oct 19 12:59:29 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.59299999999999997,"15min":0.60199999999999998},"min":{"1min":0.48499999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.751,"5min":0.79900000000000004,"15min":0.874},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.625,"15min":0.63400000000000001},"min":{"1min":0.50900000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.89500000000000002},"last":0.53700000000000003}]},{"osd":4,"last update":"Thu Oct 19 12:59:29 2023","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.36099999999999999,"5min":0.29099999999999998,"15min":0.28899999999999998},"max":{"1min":0.61399999999999999,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.46700000000000003,"15min":0.47199999999999998},"min":{"1min":0.35899999999999999,"5min":0.251,"15min":0.23899999999999999},"max":{"1min":0.58299999999999996,"5min":0.79400000000000004,"15min":0.80500000000000005},"last":0.69199999999999995}]},{"osd":6,"last update":"Thu Oct 19 12:59:29 2023","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46500000000000002,"15min":0.47299999999999998},"min":{"1min":0.34499999999999997,"5min":0.27200000000000002,"15min":0.25900000000000001},"max":{"1min":0.56299999999999994,"5min":0.74399999999999999,"15min":0.79300000000000004},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.497,"5min":0.50900000000000001,"15min":0.502},"min":{"1min":0.373,"5min":0.34200000000000003,"15min":0.30399999999999999},"max":{"1min":0.64800000000000002,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.626}]},{"osd":7,"last update":"Thu Oct 19 12:59:29 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.48599999999999999,"15min":0.49299999999999999},"min":{"1min":0.39400000000000002,"5min":0.32000000000000001,"15min":0.315},"max":{"1min":0.64200000000000002,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.53900000000000003,"15min":0.53700000000000003},"min":{"1min":0.46100000000000002,"5min":0.311,"15min":0.27500000000000002},"max":{"1min":0.66900000000000004,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.74099999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575046,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32228,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339612,"statfs":{"total":96636764160,"available":96603762688,"internally_reserved":0,"allocated":1069056,"data_stored":530484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56100000000000005,"15min":0.55900000000000005},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.28899999999999998},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.626},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.55800000000000005,"15min":0.55500000000000005},"min":{"1min":0.45400000000000001,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.77100000000000002,"5min":0.77800000000000002,"15min":0.78200000000000003},"last":0.60099999999999998}]},{"osd":1,"last update":"Thu Oct 19 12:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.60399999999999998,"15min":0.60799999999999998},"min":{"1min":0.33000000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.78600000000000003,"5min":0.80600000000000005,"15min":0.86199999999999999},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61899999999999999,"15min":0.61599999999999999},"min":{"1min":0.40899999999999997,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.77300000000000002,"5min":0.79900000000000004,"15min":0.873},"last":0.54500000000000004}]},{"osd":2,"last update":"Thu Oct 19 12:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.61199999999999999,"15min":0.61299999999999999},"min":{"1min":0.375,"5min":0.32200000000000001,"15min":0.254},"max":{"1min":0.72199999999999998,"5min":0.85299999999999998,"15min":0.86599999999999999},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61699999999999999,"15min":0.61599999999999999},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.34499999999999997},"max":{"1min":0.78500000000000003,"5min":0.82999999999999996,"15min":0.85999999999999999},"last":0.72999999999999998}]},{"osd":3,"last update":"Thu Oct 19 12:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.59099999999999997,"15min":0.58399999999999996},"min":{"1min":0.41999999999999998,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.73499999999999999,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.64600000000000002,"15min":0.63700000000000001},"min":{"1min":0.52500000000000002,"5min":0.42599999999999999,"15min":0.37},"max":{"1min":0.73899999999999999,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.64800000000000002}]},{"osd":5,"last update":"Thu Oct 19 12:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.45600000000000002,"15min":0.45200000000000001},"min":{"1min":0.309,"5min":0.29599999999999999,"15min":0.249},"max":{"1min":0.63500000000000001,"5min":0.79600000000000004,"15min":0.81299999999999994},"last":0.44},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.47499999999999998,"15min":0.46200000000000002},"min":{"1min":0.34699999999999998,"5min":0.26400000000000001,"15min":0.23499999999999999},"max":{"1min":0.623,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.35699999999999998}]},{"osd":6,"last update":"Thu Oct 19 12:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.45900000000000002,"15min":0.45600000000000002},"min":{"1min":0.31,"5min":0.29499999999999998,"15min":0.26400000000000001},"max":{"1min":0.52800000000000002,"5min":0.72599999999999998,"15min":0.78600000000000003},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.48599999999999999,"15min":0.497},"min":{"1min":0.34300000000000003,"5min":0.32100000000000001,"15min":0.27100000000000002},"max":{"1min":0.66700000000000004,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.68400000000000005}]},{"osd":7,"last update":"Thu Oct 19 12:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.501,"15min":0.503},"min":{"1min":0.33700000000000002,"5min":0.307,"15min":0.27500000000000002},"max":{"1min":0.68300000000000005,"5min":0.79100000000000004,"15min":0.80200000000000005},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.503,"5min":0.54200000000000004,"15min":0.54800000000000004},"min":{"1min":0.39100000000000001,"5min":0.35799999999999998,"15min":0.30199999999999999},"max":{"1min":0.69399999999999995,"5min":0.80500000000000005,"15min":0.83499999999999996},"last":0.70799999999999996}]}]},{"osd":3,"up_from":13,"seq":55834575045,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36380,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335460,"statfs":{"total":96636764160,"available":96599511040,"internally_reserved":0,"allocated":1056768,"data_stored":528872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:59:24 2023","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.41799999999999998,"15min":0.39900000000000002},"min":{"1min":0.27100000000000002,"5min":0.23599999999999999,"15min":0.20899999999999999},"max":{"1min":0.69099999999999995,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.41199999999999998,"15min":0.41299999999999998},"min":{"1min":0.316,"5min":0.223,"15min":0.223},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.45500000000000002}]},{"osd":1,"last update":"Thu Oct 19 12:59:24 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.40600000000000003,"15min":0.40699999999999997},"min":{"1min":0.29099999999999998,"5min":0.26100000000000001,"15min":0.22500000000000001},"max":{"1min":0.63500000000000001,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.505,"5min":0.48299999999999998,"15min":0.45700000000000002},"min":{"1min":0.27600000000000002,"5min":0.24099999999999999,"15min":0.215},"max":{"1min":0.66700000000000004,"5min":2.9399999999999999,"15min":2.9399999999999999},"last":0.41799999999999998}]},{"osd":2,"last update":"Thu Oct 19 12:59:24 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.46899999999999997,"15min":0.46500000000000002},"min":{"1min":0.33800000000000002,"5min":0.314,"15min":0.26100000000000001},"max":{"1min":0.69399999999999995,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.48099999999999998,"15min":0.48499999999999999},"min":{"1min":0.35099999999999998,"5min":0.29999999999999999,"15min":0.247},"max":{"1min":0.754,"5min":0.754,"15min":0.96499999999999997},"last":0.51000000000000001}]},{"osd":4,"last update":"Thu Oct 19 12:59:24 2023","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.65700000000000003,"15min":0.66400000000000003},"min":{"1min":0.50700000000000001,"5min":0.42299999999999999,"15min":0.40300000000000002},"max":{"1min":0.90200000000000002,"5min":0.90500000000000003,"15min":1.0069999999999999},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.69499999999999995,"15min":0.69199999999999995},"min":{"1min":0.59499999999999997,"5min":0.46899999999999997,"15min":0.42099999999999999},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.99399999999999999},"last":0.57899999999999996}]},{"osd":5,"last update":"Thu Oct 19 12:59:24 2023","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.67600000000000005,"15min":0.65900000000000003},"min":{"1min":0.56000000000000005,"5min":0.45500000000000002,"15min":0.40500000000000003},"max":{"1min":0.93600000000000005,"5min":2.9990000000000001,"15min":2.9990000000000001},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.52100000000000002,"5min":0.40899999999999997,"15min":0.35899999999999999},"max":{"1min":0.81899999999999995,"5min":1.774,"15min":1.774},"last":0.77300000000000002}]},{"osd":6,"last update":"Thu Oct 19 12:59:24 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.66100000000000003,"15min":0.65400000000000003},"min":{"1min":0.52300000000000002,"5min":0.433,"15min":0.41199999999999998},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.96799999999999997},"last":0.752},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.67500000000000004,"15min":0.67400000000000004},"min":{"1min":0.60299999999999998,"5min":0.377,"15min":0.35899999999999999},"max":{"1min":0.93700000000000006,"5min":1.167,"15min":1.167},"last":0.64700000000000002}]},{"osd":7,"last update":"Thu Oct 19 12:59:24 2023","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.65200000000000002,"15min":0.66500000000000004},"min":{"1min":0.45500000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.94299999999999995,"5min":1.042,"15min":1.042},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.69299999999999995,"15min":0.67900000000000005},"min":{"1min":0.54200000000000004,"5min":0.432,"15min":0.432},"max":{"1min":0.89200000000000002,"5min":2.8149999999999999,"15min":2.8149999999999999},"last":0.66400000000000003}]}]},{"osd":2,"up_from":13,"seq":55834575047,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54416,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317424,"statfs":{"total":96636764160,"available":96581042176,"internally_reserved":0,"allocated":1044480,"data_stored":522224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:59:33 2023","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.377,"15min":0.38100000000000001},"min":{"1min":0.27500000000000002,"5min":0.20899999999999999,"15min":0.187},"max":{"1min":0.54000000000000004,"5min":0.67700000000000005,"15min":0.83999999999999997},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.436,"5min":0.40999999999999998,"15min":0.40600000000000003},"min":{"1min":0.32900000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.58299999999999996,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.56000000000000005}]},{"osd":1,"last update":"Thu Oct 19 12:59:33 2023","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.39400000000000002,"15min":0.39800000000000002},"min":{"1min":0.318,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.80400000000000005},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.41999999999999998,"15min":0.42999999999999999},"min":{"1min":0.38200000000000001,"5min":0.22,"15min":0.22},"max":{"1min":0.59499999999999997,"5min":0.60499999999999998,"15min":0.78600000000000003},"last":0.61299999999999999}]},{"osd":3,"last update":"Thu Oct 19 12:59:33 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.46300000000000002,"15min":0.45700000000000002},"min":{"1min":0.33400000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.498},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.48099999999999998,"15min":0.48499999999999999},"min":{"1min":0.40300000000000002,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.72799999999999998,"5min":0.77800000000000002,"15min":0.87},"last":0.57499999999999996}]},{"osd":4,"last update":"Thu Oct 19 12:59:33 2023","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.63700000000000001,"15min":0.64200000000000002},"min":{"1min":0.497,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.89500000000000002,"5min":0.93999999999999995,"15min":1.0569999999999999},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.65400000000000003,"15min":0.65700000000000003},"min":{"1min":0.54300000000000004,"5min":0.35099999999999998,"15min":0.32000000000000001},"max":{"1min":0.83399999999999996,"5min":0.92300000000000004,"15min":1.0780000000000001},"last":0.70999999999999996}]},{"osd":5,"last update":"Thu Oct 19 12:59:33 2023","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.67400000000000004,"15min":0.66200000000000003},"min":{"1min":0.58299999999999996,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.84299999999999997,"5min":0.96499999999999997,"15min":0.99399999999999999},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.65200000000000002,"15min":0.65800000000000003},"min":{"1min":0.56899999999999995,"5min":0.375,"15min":0.375},"max":{"1min":0.872,"5min":0.94599999999999995,"15min":0.97099999999999997},"last":0.64000000000000001}]},{"osd":6,"last update":"Thu Oct 19 12:59:33 2023","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.65000000000000002,"15min":0.66700000000000004},"min":{"1min":0.47199999999999998,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.82399999999999995,"5min":1.008,"15min":1.008},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.67800000000000005,"15min":0.67400000000000004},"min":{"1min":0.52000000000000002,"5min":0.371,"15min":0.371},"max":{"1min":0.88100000000000001,"5min":0.995,"15min":0.995},"last":0.63}]},{"osd":7,"last update":"Thu Oct 19 12:59:33 2023","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.67500000000000004,"15min":0.68500000000000005},"min":{"1min":0.59699999999999998,"5min":0.39300000000000002,"15min":0.36399999999999999},"max":{"1min":0.86099999999999999,"5min":1.0149999999999999,"15min":1.111},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.68899999999999995,"15min":0.68300000000000005},"min":{"1min":0.53000000000000003,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.89800000000000002,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.65300000000000002}]}]},{"osd":1,"up_from":13,"seq":55834575046,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32236,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339604,"statfs":{"total":96636764160,"available":96603754496,"internally_reserved":0,"allocated":1073152,"data_stored":524600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:59:33 2023","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.39200000000000002,"15min":0.38500000000000001},"min":{"1min":0.23999999999999999,"5min":0.21299999999999999,"15min":0.187},"max":{"1min":0.55800000000000005,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.442,"15min":0.42799999999999999},"min":{"1min":0.29499999999999998,"5min":0.23899999999999999,"15min":0.21199999999999999},"max":{"1min":0.58799999999999997,"5min":2.5800000000000001,"15min":2.5800000000000001},"last":0.52700000000000002}]},{"osd":2,"last update":"Thu Oct 19 12:59:33 2023","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.42499999999999999,"15min":0.42099999999999999},"min":{"1min":0.309,"5min":0.21299999999999999,"15min":0.20100000000000001},"max":{"1min":0.53300000000000003,"5min":0.60499999999999998,"15min":0.69999999999999996},"last":0.502},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.47099999999999997,"15min":0.45300000000000001},"min":{"1min":0.34999999999999998,"5min":0.313,"15min":0.23699999999999999},"max":{"1min":0.625,"5min":2.6230000000000002,"15min":2.6230000000000002},"last":0.57999999999999996}]},{"osd":3,"last update":"Thu Oct 19 12:59:33 2023","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.51600000000000001,"15min":0.48599999999999999},"min":{"1min":0.38900000000000001,"5min":0.307,"15min":0.26200000000000001},"max":{"1min":0.621,"5min":3.9740000000000002,"15min":3.9740000000000002},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.43099999999999999,"15min":0.42299999999999999},"min":{"1min":0.35999999999999999,"5min":0.27700000000000002,"15min":0.27000000000000002},"max":{"1min":0.59299999999999997,"5min":0.63200000000000001,"15min":0.80500000000000005},"last":0.51400000000000001}]},{"osd":4,"last update":"Thu Oct 19 12:59:33 2023","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.65600000000000003,"15min":0.65200000000000002},"min":{"1min":0.501,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.85699999999999998,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.66600000000000004,"15min":0.64800000000000002},"min":{"1min":0.47399999999999998,"5min":0.42599999999999999,"15min":0.314},"max":{"1min":0.85399999999999998,"5min":1.0860000000000001,"15min":1.1910000000000001},"last":0.76300000000000001}]},{"osd":5,"last update":"Thu Oct 19 12:59:33 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.65700000000000003,"15min":0.65300000000000002},"min":{"1min":0.499,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.83599999999999997,"5min":1.1200000000000001,"15min":1.1200000000000001},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.69599999999999995,"15min":0.67300000000000004},"min":{"1min":0.54300000000000004,"5min":0.42299999999999999,"15min":0.38700000000000001},"max":{"1min":0.81799999999999995,"5min":2.7120000000000002,"15min":2.7120000000000002},"last":0.77900000000000003}]},{"osd":6,"last update":"Thu Oct 19 12:59:33 2023","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.65800000000000003,"15min":0.65700000000000003},"min":{"1min":0.53800000000000003,"5min":0.36799999999999999,"15min":0.36399999999999999},"max":{"1min":0.91300000000000003,"5min":1.0069999999999999,"15min":1.077},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67400000000000004,"15min":0.67800000000000005},"min":{"1min":0.55800000000000005,"5min":0.376,"15min":0.376},"max":{"1min":0.77500000000000002,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.72099999999999997}]},{"osd":7,"last update":"Thu Oct 19 12:59:33 2023","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.69799999999999995,"15min":0.67900000000000005},"min":{"1min":0.54300000000000004,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.84999999999999998,"5min":2.786,"15min":2.786},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.68200000000000005,"15min":0.67900000000000005},"min":{"1min":0.47499999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":1.0489999999999999},"last":0.79500000000000004}]}]},{"osd":0,"up_from":13,"seq":55834575047,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36952,"kb_used_data":1608,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334888,"statfs":{"total":96636764160,"available":96598925312,"internally_reserved":0,"allocated":1646592,"data_stored":1118621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Oct 19 12:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.40300000000000002,"15min":0.39100000000000001},"min":{"1min":0.26500000000000001,"5min":0.182,"15min":0.182},"max":{"1min":0.58899999999999997,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.434,"15min":0.432},"min":{"1min":0.34599999999999997,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.746},"last":0.38200000000000001}]},{"osd":2,"last update":"Thu Oct 19 12:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.46700000000000003,"15min":0.45500000000000002},"min":{"1min":0.33200000000000002,"5min":0.29499999999999998,"15min":0.25800000000000001},"max":{"1min":0.58599999999999997,"5min":0.752,"15min":0.76300000000000001},"last":0.309},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.437,"15min":0.434},"min":{"1min":0.26000000000000001,"5min":0.216,"15min":0.20699999999999999},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.76800000000000002},"last":0.432}]},{"osd":3,"last update":"Thu Oct 19 12:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.42199999999999999,"15min":0.40899999999999997},"min":{"1min":0.34000000000000002,"5min":0.23400000000000001,"15min":0.22800000000000001},"max":{"1min":0.53500000000000003,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.505,"5min":0.504,"15min":0.49099999999999999},"min":{"1min":0.33600000000000002,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.66300000000000003,"5min":1.4990000000000001,"15min":1.4990000000000001},"last":0.34799999999999998}]},{"osd":4,"last update":"Thu Oct 19 12:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.63100000000000001,"15min":0.63600000000000001},"min":{"1min":0.46600000000000003,"5min":0.40000000000000002,"15min":0.34999999999999998},"max":{"1min":0.749,"5min":1.361,"15min":1.361},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.65700000000000003,"15min":0.65000000000000002},"min":{"1min":0.52900000000000003,"5min":0.41899999999999998,"15min":0.35999999999999999},"max":{"1min":0.871,"5min":0.98999999999999999,"15min":0.998},"last":0.57699999999999996}]},{"osd":5,"last update":"Thu Oct 19 12:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.68999999999999995,"15min":0.68000000000000005},"min":{"1min":0.50700000000000001,"5min":0.378,"15min":0.378},"max":{"1min":0.94599999999999995,"5min":1.325,"15min":1.325},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.69299999999999995,"15min":0.68600000000000005},"min":{"1min":0.45900000000000002,"5min":0.36299999999999999,"15min":0.318},"max":{"1min":1.02,"5min":1.024,"15min":1.056},"last":0.625}]},{"osd":6,"last update":"Thu Oct 19 12:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.67200000000000004,"15min":0.66400000000000003},"min":{"1min":0.59999999999999998,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.873,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.67600000000000005,"15min":0.66600000000000004},"min":{"1min":0.57199999999999995,"5min":0.41999999999999998,"15min":0.38900000000000001},"max":{"1min":0.92000000000000004,"5min":0.995,"15min":1.109},"last":0.65600000000000003}]},{"osd":7,"last update":"Thu Oct 19 12:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.68700000000000006,"15min":0.68400000000000005},"min":{"1min":0.53500000000000003,"5min":0.41199999999999998,"15min":0.41099999999999998},"max":{"1min":0.89900000000000002,"5min":0.98799999999999999,"15min":1.0089999999999999},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.67300000000000004,"15min":0.66300000000000003},"min":{"1min":0.58799999999999997,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.90400000000000003,"5min":1.389,"15min":1.389},"last":0.67100000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-19T13:00:26.719 INFO:tasks.ceph.mds.a.smithi057.stderr:2023-10-19T13:00:26.717+0000 7fbae915a640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-19T13:00:28.676 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-19T13:00:28.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:28.795+0000 7fb0da2c2640 1 -- 172.21.15.57:0/1320917074 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb0d40ceb40 msgr2=0x7fb0d40b0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:28.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:28.795+0000 7fb0da2c2640 1 --2- 172.21.15.57:0/1320917074 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb0d40ceb40 0x7fb0d40b0650 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7fb0d408a810 tx=0x7fb0cc02c080 comp rx=0 tx=0).stop 2023-10-19T13:00:28.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:28.795+0000 7fb0da2c2640 1 -- 172.21.15.57:0/1320917074 shutdown_connections 2023-10-19T13:00:28.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:28.796+0000 7fb0da2c2640 1 --2- 172.21.15.57:0/1320917074 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb0d40ceb40 0x7fb0d40b0650 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:28.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:28.796+0000 7fb0da2c2640 1 --2- 172.21.15.57:0/1320917074 >> v2:172.21.15.179:3300/0 conn(0x7fb0d40cf340 0x7fb0d40d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:28.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:28.796+0000 7fb0da2c2640 1 -- 172.21.15.57:0/1320917074 >> 172.21.15.57:0/1320917074 conn(0x7fb0d40d0d60 msgr2=0x7fb0d40d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:28.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:28.796+0000 7fb0da2c2640 1 -- 172.21.15.57:0/1320917074 shutdown_connections 2023-10-19T13:00:28.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:28.796+0000 7fb0da2c2640 1 -- 172.21.15.57:0/1320917074 wait complete. 2023-10-19T13:00:28.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:28.797+0000 7fb0da2c2640 1 Processor -- start 2023-10-19T13:00:28.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:28.797+0000 7fb0da2c2640 1 -- start start 2023-10-19T13:00:28.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:28.798+0000 7fb0da2c2640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb0d40ceb40 0x7fb0d4156430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:28.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:28.798+0000 7fb0da2c2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb0d40cf340 0x7fb0d4157e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:28.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:28.798+0000 7fb0da2c2640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb0d415b2a0 0x7fb0d409e200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:28.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:28.798+0000 7fb0da2c2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d4156b90 con 0x7fb0d40cf340 2023-10-19T13:00:28.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:28.798+0000 7fb0da2c2640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d4156970 con 0x7fb0d40ceb40 2023-10-19T13:00:28.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:28.798+0000 7fb0da2c2640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d4156de0 con 0x7fb0d415b2a0 2023-10-19T13:00:28.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:28.798+0000 7fb0d37fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb0d40cf340 0x7fb0d4157e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:28.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:28.798+0000 7fb0d37fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb0d40cf340 0x7fb0d4157e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56338/0 (socket says 172.21.15.57:56338) 2023-10-19T13:00:28.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:28.798+0000 7fb0d37fe640 1 -- 172.21.15.57:0/1581916622 learned_addr learned my addr 172.21.15.57:0/1581916622 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:28.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:28.798+0000 7fb0d3fff640 1 --2- 172.21.15.57:0/1581916622 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb0d40ceb40 0x7fb0d4156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:28.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:28.798+0000 7fb0d885f640 1 --2- 172.21.15.57:0/1581916622 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb0d415b2a0 0x7fb0d409e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:28.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:28.798+0000 7fb0d37fe640 1 -- 172.21.15.57:0/1581916622 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb0d415b2a0 msgr2=0x7fb0d409e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:28.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:28.798+0000 7fb0d37fe640 1 --2- 172.21.15.57:0/1581916622 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb0d415b2a0 0x7fb0d409e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:28.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:28.798+0000 7fb0d37fe640 1 -- 172.21.15.57:0/1581916622 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb0d40ceb40 msgr2=0x7fb0d4156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:28.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:28.798+0000 7fb0d37fe640 1 --2- 172.21.15.57:0/1581916622 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb0d40ceb40 0x7fb0d4156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:28.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:28.798+0000 7fb0d37fe640 1 -- 172.21.15.57:0/1581916622 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0cc009840 con 0x7fb0d40cf340 2023-10-19T13:00:28.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:28.798+0000 7fb0d3fff640 1 --2- 172.21.15.57:0/1581916622 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb0d40ceb40 0x7fb0d4156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:00:28.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:28.798+0000 7fb0d37fe640 1 --2- 172.21.15.57:0/1581916622 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb0d40cf340 0x7fb0d4157e80 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7fb0c800c3c0 tx=0x7fb0c800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:28.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:28.799+0000 7fb0d0ff9640 1 -- 172.21.15.57:0/1581916622 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0c8004230 con 0x7fb0d40cf340 2023-10-19T13:00:28.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:28.799+0000 7fb0d0ff9640 1 -- 172.21.15.57:0/1581916622 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fb0c80044a0 con 0x7fb0d40cf340 2023-10-19T13:00:28.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:28.799+0000 7fb0da2c2640 1 -- 172.21.15.57:0/1581916622 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0d41570c0 con 0x7fb0d40cf340 2023-10-19T13:00:28.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:28.799+0000 7fb0d0ff9640 1 -- 172.21.15.57:0/1581916622 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0c8003850 con 0x7fb0d40cf340 2023-10-19T13:00:28.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:28.799+0000 7fb0da2c2640 1 -- 172.21.15.57:0/1581916622 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0d4157680 con 0x7fb0d40cf340 2023-10-19T13:00:28.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:28.800+0000 7fb0da2c2640 1 -- 172.21.15.57:0/1581916622 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0d40690e0 con 0x7fb0d40cf340 2023-10-19T13:00:28.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:28.800+0000 7fb0d0ff9640 1 -- 172.21.15.57:0/1581916622 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 32) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb0c80039b0 con 0x7fb0d40cf340 2023-10-19T13:00:28.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:28.800+0000 7fb0d0ff9640 1 --2- 172.21.15.57:0/1581916622 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7fb0c006af60 0x7fb0c006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:28.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:28.801+0000 7fb0d0ff9640 1 -- 172.21.15.57:0/1581916622 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fb0c8089180 con 0x7fb0d40cf340 2023-10-19T13:00:28.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:28.801+0000 7fb0d3fff640 1 --2- 172.21.15.57:0/1581916622 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7fb0c006af60 0x7fb0c006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:28.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:28.801+0000 7fb0d3fff640 1 --2- 172.21.15.57:0/1581916622 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7fb0c006af60 0x7fb0c006d420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb0c40058c0 tx=0x7fb0c4005850 comp rx=0 tx=0).ready entity=mgr.6256 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:28.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:28.805+0000 7fb0d0ff9640 1 -- 172.21.15.57:0/1581916622 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb0c8053e90 con 0x7fb0d40cf340 2023-10-19T13:00:28.979 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:28.977+0000 7fb0da2c2640 1 -- 172.21.15.57:0/1581916622 --> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb0d40b0d00 con 0x7fb0c006af60 2023-10-19T13:00:28.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:28.985+0000 7fb0d0ff9640 1 -- 172.21.15.57:0/1581916622 <== mgr.6256 v2:172.21.15.179:6832/3815793798 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289637 (secure 0 0 0) 0x7fb0d40b0d00 con 0x7fb0c006af60 2023-10-19T13:00:28.990 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:00:28.992 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2023-10-19T13:00:28.993 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:28.987+0000 7fb0da2c2640 1 -- 172.21.15.57:0/1581916622 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7fb0c006af60 msgr2=0x7fb0c006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:28.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:28.987+0000 7fb0da2c2640 1 --2- 172.21.15.57:0/1581916622 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7fb0c006af60 0x7fb0c006d420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb0c40058c0 tx=0x7fb0c4005850 comp rx=0 tx=0).stop 2023-10-19T13:00:28.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:28.987+0000 7fb0da2c2640 1 -- 172.21.15.57:0/1581916622 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb0d40cf340 msgr2=0x7fb0d4157e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:28.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:28.987+0000 7fb0da2c2640 1 --2- 172.21.15.57:0/1581916622 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb0d40cf340 0x7fb0d4157e80 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7fb0c800c3c0 tx=0x7fb0c800c880 comp rx=0 tx=0).stop 2023-10-19T13:00:28.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:28.988+0000 7fb0da2c2640 1 -- 172.21.15.57:0/1581916622 shutdown_connections 2023-10-19T13:00:28.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:28.988+0000 7fb0da2c2640 1 --2- 172.21.15.57:0/1581916622 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7fb0c006af60 0x7fb0c006d420 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:28.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:28.988+0000 7fb0da2c2640 1 --2- 172.21.15.57:0/1581916622 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb0d415b2a0 0x7fb0d409e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:28.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:28.988+0000 7fb0da2c2640 1 --2- 172.21.15.57:0/1581916622 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb0d40cf340 0x7fb0d4157e80 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:28.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:28.988+0000 7fb0da2c2640 1 --2- 172.21.15.57:0/1581916622 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb0d40ceb40 0x7fb0d4156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:28.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:28.988+0000 7fb0da2c2640 1 -- 172.21.15.57:0/1581916622 >> 172.21.15.57:0/1581916622 conn(0x7fb0d40d0d60 msgr2=0x7fb0d4159550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:28.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:28.988+0000 7fb0da2c2640 1 -- 172.21.15.57:0/1581916622 shutdown_connections 2023-10-19T13:00:28.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:28.988+0000 7fb0da2c2640 1 -- 172.21.15.57:0/1581916622 wait complete. 2023-10-19T13:00:29.001 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-10-19T13:00:27.050786+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":283,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":199,"ondisk_log_size":199,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":153,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":315812,"kb_used_data":9428,"kb_used_omap":12,"kb_used_meta":306227,"kb_avail":754658908,"statfs":{"total":773094113280,"available":772770721792,"internally_reserved":0,"allocated":9654272,"data_stored":5389321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":313577017},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.870505"},"pg_stats":[{"pgid":"10.3f","version":"0'0","reported_seq":9,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-19T13:00:21.237877+0000","last_change":"2023-10-19T13:00:21.228303+0000","last_active":"2023-10-19T13:00:21.237877+0000","last_peered":"2023-10-19T13:00:21.237877+0000","last_clean":"2023-10-19T13:00:21.237877+0000","last_became_active":"2023-10-19T13:00:21.227974+0000","last_became_peered":"2023-10-19T13:00:21.227974+0000","last_unstale":"2023-10-19T13:00:21.237877+0000","last_undegraded":"2023-10-19T13:00:21.237877+0000","last_fullsized":"2023-10-19T13:00:21.237877+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:37:52.369486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3e","version":"0'0","reported_seq":9,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-19T13:00:21.233058+0000","last_change":"2023-10-19T13:00:21.227759+0000","last_active":"2023-10-19T13:00:21.233058+0000","last_peered":"2023-10-19T13:00:21.233058+0000","last_clean":"2023-10-19T13:00:21.233058+0000","last_became_active":"2023-10-19T13:00:21.227466+0000","last_became_peered":"2023-10-19T13:00:21.227466+0000","last_unstale":"2023-10-19T13:00:21.233058+0000","last_undegraded":"2023-10-19T13:00:21.233058+0000","last_fullsized":"2023-10-19T13:00:21.233058+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:55:42.567075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3d","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.222826+0000","last_change":"2023-10-19T13:00:21.224369+0000","last_active":"2023-10-19T13:00:22.222826+0000","last_peered":"2023-10-19T13:00:22.222826+0000","last_clean":"2023-10-19T13:00:22.222826+0000","last_became_active":"2023-10-19T13:00:21.224115+0000","last_became_peered":"2023-10-19T13:00:21.224115+0000","last_unstale":"2023-10-19T13:00:22.222826+0000","last_undegraded":"2023-10-19T13:00:22.222826+0000","last_fullsized":"2023-10-19T13:00:22.222826+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:27:46.354594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.3c","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.384886+0000","last_change":"2023-10-19T13:00:21.221167+0000","last_active":"2023-10-19T13:00:22.384886+0000","last_peered":"2023-10-19T13:00:22.384886+0000","last_clean":"2023-10-19T13:00:22.384886+0000","last_became_active":"2023-10-19T13:00:21.220894+0000","last_became_peered":"2023-10-19T13:00:21.220894+0000","last_unstale":"2023-10-19T13:00:22.384886+0000","last_undegraded":"2023-10-19T13:00:22.384886+0000","last_fullsized":"2023-10-19T13:00:22.384886+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:34:57.792223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3b","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.423410+0000","last_change":"2023-10-19T13:00:21.214718+0000","last_active":"2023-10-19T13:00:22.423410+0000","last_peered":"2023-10-19T13:00:22.423410+0000","last_clean":"2023-10-19T13:00:22.423410+0000","last_became_active":"2023-10-19T13:00:21.214364+0000","last_became_peered":"2023-10-19T13:00:21.214364+0000","last_unstale":"2023-10-19T13:00:22.423410+0000","last_undegraded":"2023-10-19T13:00:22.423410+0000","last_fullsized":"2023-10-19T13:00:22.423410+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:52:56.240367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.3a","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.221076+0000","last_change":"2023-10-19T13:00:21.219325+0000","last_active":"2023-10-19T13:00:22.221076+0000","last_peered":"2023-10-19T13:00:22.221076+0000","last_clean":"2023-10-19T13:00:22.221076+0000","last_became_active":"2023-10-19T13:00:21.218458+0000","last_became_peered":"2023-10-19T13:00:21.218458+0000","last_unstale":"2023-10-19T13:00:22.221076+0000","last_undegraded":"2023-10-19T13:00:22.221076+0000","last_fullsized":"2023-10-19T13:00:22.221076+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:31:39.478057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.39","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:23.044542+0000","last_change":"2023-10-19T13:00:21.220590+0000","last_active":"2023-10-19T13:00:23.044542+0000","last_peered":"2023-10-19T13:00:23.044542+0000","last_clean":"2023-10-19T13:00:23.044542+0000","last_became_active":"2023-10-19T13:00:21.220337+0000","last_became_peered":"2023-10-19T13:00:21.220337+0000","last_unstale":"2023-10-19T13:00:23.044542+0000","last_undegraded":"2023-10-19T13:00:23.044542+0000","last_fullsized":"2023-10-19T13:00:23.044542+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:25:25.779991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.38","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:23.044542+0000","last_change":"2023-10-19T13:00:21.214103+0000","last_active":"2023-10-19T13:00:23.044542+0000","last_peered":"2023-10-19T13:00:23.044542+0000","last_clean":"2023-10-19T13:00:23.044542+0000","last_became_active":"2023-10-19T13:00:21.213851+0000","last_became_peered":"2023-10-19T13:00:21.213851+0000","last_unstale":"2023-10-19T13:00:23.044542+0000","last_undegraded":"2023-10-19T13:00:23.044542+0000","last_fullsized":"2023-10-19T13:00:23.044542+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:30:23.953334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.37","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.223081+0000","last_change":"2023-10-19T13:00:21.226394+0000","last_active":"2023-10-19T13:00:22.223081+0000","last_peered":"2023-10-19T13:00:22.223081+0000","last_clean":"2023-10-19T13:00:22.223081+0000","last_became_active":"2023-10-19T13:00:21.226136+0000","last_became_peered":"2023-10-19T13:00:21.226136+0000","last_unstale":"2023-10-19T13:00:22.223081+0000","last_undegraded":"2023-10-19T13:00:22.223081+0000","last_fullsized":"2023-10-19T13:00:22.223081+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:55:17.988760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.36","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.223037+0000","last_change":"2023-10-19T13:00:21.231437+0000","last_active":"2023-10-19T13:00:22.223037+0000","last_peered":"2023-10-19T13:00:22.223037+0000","last_clean":"2023-10-19T13:00:22.223037+0000","last_became_active":"2023-10-19T13:00:21.231133+0000","last_became_peered":"2023-10-19T13:00:21.231133+0000","last_unstale":"2023-10-19T13:00:22.223037+0000","last_undegraded":"2023-10-19T13:00:22.223037+0000","last_fullsized":"2023-10-19T13:00:22.223037+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:08:45.213465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.35","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.384804+0000","last_change":"2023-10-19T13:00:21.230427+0000","last_active":"2023-10-19T13:00:22.384804+0000","last_peered":"2023-10-19T13:00:22.384804+0000","last_clean":"2023-10-19T13:00:22.384804+0000","last_became_active":"2023-10-19T13:00:21.230143+0000","last_became_peered":"2023-10-19T13:00:21.230143+0000","last_unstale":"2023-10-19T13:00:22.384804+0000","last_undegraded":"2023-10-19T13:00:22.384804+0000","last_fullsized":"2023-10-19T13:00:22.384804+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:33:36.402003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.34","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:23.044878+0000","last_change":"2023-10-19T13:00:21.218737+0000","last_active":"2023-10-19T13:00:23.044878+0000","last_peered":"2023-10-19T13:00:23.044878+0000","last_clean":"2023-10-19T13:00:23.044878+0000","last_became_active":"2023-10-19T13:00:21.218347+0000","last_became_peered":"2023-10-19T13:00:21.218347+0000","last_unstale":"2023-10-19T13:00:23.044878+0000","last_undegraded":"2023-10-19T13:00:23.044878+0000","last_fullsized":"2023-10-19T13:00:23.044878+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:22:14.250042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.33","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:23.044789+0000","last_change":"2023-10-19T13:00:21.216810+0000","last_active":"2023-10-19T13:00:23.044789+0000","last_peered":"2023-10-19T13:00:23.044789+0000","last_clean":"2023-10-19T13:00:23.044789+0000","last_became_active":"2023-10-19T13:00:21.216536+0000","last_became_peered":"2023-10-19T13:00:21.216536+0000","last_unstale":"2023-10-19T13:00:23.044789+0000","last_undegraded":"2023-10-19T13:00:23.044789+0000","last_fullsized":"2023-10-19T13:00:23.044789+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:12:58.144772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.32","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.222482+0000","last_change":"2023-10-19T13:00:21.224986+0000","last_active":"2023-10-19T13:00:22.222482+0000","last_peered":"2023-10-19T13:00:22.222482+0000","last_clean":"2023-10-19T13:00:22.222482+0000","last_became_active":"2023-10-19T13:00:21.224703+0000","last_became_peered":"2023-10-19T13:00:21.224703+0000","last_unstale":"2023-10-19T13:00:22.222482+0000","last_undegraded":"2023-10-19T13:00:22.222482+0000","last_fullsized":"2023-10-19T13:00:22.222482+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:14:30.933348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.31","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.222437+0000","last_change":"2023-10-19T13:00:21.230937+0000","last_active":"2023-10-19T13:00:22.222437+0000","last_peered":"2023-10-19T13:00:22.222437+0000","last_clean":"2023-10-19T13:00:22.222437+0000","last_became_active":"2023-10-19T13:00:21.230649+0000","last_became_peered":"2023-10-19T13:00:21.230649+0000","last_unstale":"2023-10-19T13:00:22.222437+0000","last_undegraded":"2023-10-19T13:00:22.222437+0000","last_fullsized":"2023-10-19T13:00:22.222437+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:00:42.269401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.30","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:23.044519+0000","last_change":"2023-10-19T13:00:21.219839+0000","last_active":"2023-10-19T13:00:23.044519+0000","last_peered":"2023-10-19T13:00:23.044519+0000","last_clean":"2023-10-19T13:00:23.044519+0000","last_became_active":"2023-10-19T13:00:21.219524+0000","last_became_peered":"2023-10-19T13:00:21.219524+0000","last_unstale":"2023-10-19T13:00:23.044519+0000","last_undegraded":"2023-10-19T13:00:23.044519+0000","last_fullsized":"2023-10-19T13:00:23.044519+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:16:36.020089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2f","version":"0'0","reported_seq":9,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-19T13:00:21.237892+0000","last_change":"2023-10-19T13:00:21.228480+0000","last_active":"2023-10-19T13:00:21.237892+0000","last_peered":"2023-10-19T13:00:21.237892+0000","last_clean":"2023-10-19T13:00:21.237892+0000","last_became_active":"2023-10-19T13:00:21.227687+0000","last_became_peered":"2023-10-19T13:00:21.227687+0000","last_unstale":"2023-10-19T13:00:21.237892+0000","last_undegraded":"2023-10-19T13:00:21.237892+0000","last_fullsized":"2023-10-19T13:00:21.237892+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:58:41.166338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2e","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.384934+0000","last_change":"2023-10-19T13:00:21.221940+0000","last_active":"2023-10-19T13:00:22.384934+0000","last_peered":"2023-10-19T13:00:22.384934+0000","last_clean":"2023-10-19T13:00:22.384934+0000","last_became_active":"2023-10-19T13:00:21.221511+0000","last_became_peered":"2023-10-19T13:00:21.221511+0000","last_unstale":"2023-10-19T13:00:22.384934+0000","last_undegraded":"2023-10-19T13:00:22.384934+0000","last_fullsized":"2023-10-19T13:00:22.384934+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:14:55.381409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2d","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.423674+0000","last_change":"2023-10-19T13:00:21.208462+0000","last_active":"2023-10-19T13:00:22.423674+0000","last_peered":"2023-10-19T13:00:22.423674+0000","last_clean":"2023-10-19T13:00:22.423674+0000","last_became_active":"2023-10-19T13:00:21.208176+0000","last_became_peered":"2023-10-19T13:00:21.208176+0000","last_unstale":"2023-10-19T13:00:22.423674+0000","last_undegraded":"2023-10-19T13:00:22.423674+0000","last_fullsized":"2023-10-19T13:00:22.423674+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:34:57.792223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2c","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.222039+0000","last_change":"2023-10-19T13:00:21.221939+0000","last_active":"2023-10-19T13:00:22.222039+0000","last_peered":"2023-10-19T13:00:22.222039+0000","last_clean":"2023-10-19T13:00:22.222039+0000","last_became_active":"2023-10-19T13:00:21.221665+0000","last_became_peered":"2023-10-19T13:00:21.221665+0000","last_unstale":"2023-10-19T13:00:22.222039+0000","last_undegraded":"2023-10-19T13:00:22.222039+0000","last_fullsized":"2023-10-19T13:00:22.222039+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:17:57.872305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2b","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:23.044845+0000","last_change":"2023-10-19T13:00:21.229853+0000","last_active":"2023-10-19T13:00:23.044845+0000","last_peered":"2023-10-19T13:00:23.044845+0000","last_clean":"2023-10-19T13:00:23.044845+0000","last_became_active":"2023-10-19T13:00:21.229515+0000","last_became_peered":"2023-10-19T13:00:21.229515+0000","last_unstale":"2023-10-19T13:00:23.044845+0000","last_undegraded":"2023-10-19T13:00:23.044845+0000","last_fullsized":"2023-10-19T13:00:23.044845+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:27:46.354594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.2a","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.221862+0000","last_change":"2023-10-19T13:00:21.215974+0000","last_active":"2023-10-19T13:00:22.221862+0000","last_peered":"2023-10-19T13:00:22.221862+0000","last_clean":"2023-10-19T13:00:22.221862+0000","last_became_active":"2023-10-19T13:00:21.215688+0000","last_became_peered":"2023-10-19T13:00:21.215688+0000","last_unstale":"2023-10-19T13:00:22.221862+0000","last_undegraded":"2023-10-19T13:00:22.221862+0000","last_fullsized":"2023-10-19T13:00:22.221862+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:18:50.120828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.29","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.221799+0000","last_change":"2023-10-19T13:00:21.221347+0000","last_active":"2023-10-19T13:00:22.221799+0000","last_peered":"2023-10-19T13:00:22.221799+0000","last_clean":"2023-10-19T13:00:22.221799+0000","last_became_active":"2023-10-19T13:00:21.220982+0000","last_became_peered":"2023-10-19T13:00:21.220982+0000","last_unstale":"2023-10-19T13:00:22.221799+0000","last_undegraded":"2023-10-19T13:00:22.221799+0000","last_fullsized":"2023-10-19T13:00:22.221799+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:36:46.822232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.28","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.222424+0000","last_change":"2023-10-19T13:00:21.226183+0000","last_active":"2023-10-19T13:00:22.222424+0000","last_peered":"2023-10-19T13:00:22.222424+0000","last_clean":"2023-10-19T13:00:22.222424+0000","last_became_active":"2023-10-19T13:00:21.225901+0000","last_became_peered":"2023-10-19T13:00:21.225901+0000","last_unstale":"2023-10-19T13:00:22.222424+0000","last_undegraded":"2023-10-19T13:00:22.222424+0000","last_fullsized":"2023-10-19T13:00:22.222424+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:00:49.808529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.27","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:23.045013+0000","last_change":"2023-10-19T13:00:21.226532+0000","last_active":"2023-10-19T13:00:23.045013+0000","last_peered":"2023-10-19T13:00:23.045013+0000","last_clean":"2023-10-19T13:00:23.045013+0000","last_became_active":"2023-10-19T13:00:21.226239+0000","last_became_peered":"2023-10-19T13:00:21.226239+0000","last_unstale":"2023-10-19T13:00:23.045013+0000","last_undegraded":"2023-10-19T13:00:23.045013+0000","last_fullsized":"2023-10-19T13:00:23.045013+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:39:34.098405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.26","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:23.044982+0000","last_change":"2023-10-19T13:00:21.222721+0000","last_active":"2023-10-19T13:00:23.044982+0000","last_peered":"2023-10-19T13:00:23.044982+0000","last_clean":"2023-10-19T13:00:23.044982+0000","last_became_active":"2023-10-19T13:00:21.222414+0000","last_became_peered":"2023-10-19T13:00:21.222414+0000","last_unstale":"2023-10-19T13:00:23.044982+0000","last_undegraded":"2023-10-19T13:00:23.044982+0000","last_fullsized":"2023-10-19T13:00:23.044982+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:08:45.213465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.25","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.384972+0000","last_change":"2023-10-19T13:00:21.212641+0000","last_active":"2023-10-19T13:00:22.384972+0000","last_peered":"2023-10-19T13:00:22.384972+0000","last_clean":"2023-10-19T13:00:22.384972+0000","last_became_active":"2023-10-19T13:00:21.212380+0000","last_became_peered":"2023-10-19T13:00:21.212380+0000","last_unstale":"2023-10-19T13:00:22.384972+0000","last_undegraded":"2023-10-19T13:00:22.384972+0000","last_fullsized":"2023-10-19T13:00:22.384972+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:58:41.166338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.24","version":"0'0","reported_seq":9,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-19T13:00:21.233957+0000","last_change":"2023-10-19T13:00:21.227180+0000","last_active":"2023-10-19T13:00:21.233957+0000","last_peered":"2023-10-19T13:00:21.233957+0000","last_clean":"2023-10-19T13:00:21.233957+0000","last_became_active":"2023-10-19T13:00:21.226763+0000","last_became_peered":"2023-10-19T13:00:21.226763+0000","last_unstale":"2023-10-19T13:00:21.233957+0000","last_undegraded":"2023-10-19T13:00:21.233957+0000","last_fullsized":"2023-10-19T13:00:21.233957+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:41:19.821669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.23","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.222316+0000","last_change":"2023-10-19T13:00:21.224996+0000","last_active":"2023-10-19T13:00:22.222316+0000","last_peered":"2023-10-19T13:00:22.222316+0000","last_clean":"2023-10-19T13:00:22.222316+0000","last_became_active":"2023-10-19T13:00:21.224677+0000","last_became_peered":"2023-10-19T13:00:21.224677+0000","last_unstale":"2023-10-19T13:00:22.222316+0000","last_undegraded":"2023-10-19T13:00:22.222316+0000","last_fullsized":"2023-10-19T13:00:22.222316+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:30:23.953334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.22","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:23.044596+0000","last_change":"2023-10-19T13:00:21.219461+0000","last_active":"2023-10-19T13:00:23.044596+0000","last_peered":"2023-10-19T13:00:23.044596+0000","last_clean":"2023-10-19T13:00:23.044596+0000","last_became_active":"2023-10-19T13:00:21.219209+0000","last_became_peered":"2023-10-19T13:00:21.219209+0000","last_unstale":"2023-10-19T13:00:23.044596+0000","last_undegraded":"2023-10-19T13:00:23.044596+0000","last_fullsized":"2023-10-19T13:00:23.044596+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:14:30.933348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.21","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.384507+0000","last_change":"2023-10-19T13:00:21.212876+0000","last_active":"2023-10-19T13:00:22.384507+0000","last_peered":"2023-10-19T13:00:22.384507+0000","last_clean":"2023-10-19T13:00:22.384507+0000","last_became_active":"2023-10-19T13:00:21.212607+0000","last_became_peered":"2023-10-19T13:00:21.212607+0000","last_unstale":"2023-10-19T13:00:22.384507+0000","last_undegraded":"2023-10-19T13:00:22.384507+0000","last_fullsized":"2023-10-19T13:00:22.384507+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:52:56.240367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.20","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.220680+0000","last_change":"2023-10-19T13:00:21.217955+0000","last_active":"2023-10-19T13:00:22.220680+0000","last_peered":"2023-10-19T13:00:22.220680+0000","last_clean":"2023-10-19T13:00:22.220680+0000","last_became_active":"2023-10-19T13:00:21.217649+0000","last_became_peered":"2023-10-19T13:00:21.217649+0000","last_unstale":"2023-10-19T13:00:22.220680+0000","last_undegraded":"2023-10-19T13:00:22.220680+0000","last_fullsized":"2023-10-19T13:00:22.220680+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:55:17.988760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.423650+0000","last_change":"2023-10-19T13:00:21.231512+0000","last_active":"2023-10-19T13:00:22.423650+0000","last_peered":"2023-10-19T13:00:22.423650+0000","last_clean":"2023-10-19T13:00:22.423650+0000","last_became_active":"2023-10-19T13:00:21.231224+0000","last_became_peered":"2023-10-19T13:00:21.231224+0000","last_unstale":"2023-10-19T13:00:22.423650+0000","last_undegraded":"2023-10-19T13:00:22.423650+0000","last_fullsized":"2023-10-19T13:00:22.423650+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:33:36.402003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1f","version":"86'4","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-19T13:00:21.238495+0000","last_change":"2023-10-19T13:00:20.197586+0000","last_active":"2023-10-19T13:00:21.238495+0000","last_peered":"2023-10-19T13:00:21.238495+0000","last_clean":"2023-10-19T13:00:21.238495+0000","last_became_active":"2023-10-19T13:00:20.197286+0000","last_became_peered":"2023-10-19T13:00:20.197286+0000","last_unstale":"2023-10-19T13:00:21.238495+0000","last_undegraded":"2023-10-19T13:00:21.238495+0000","last_fullsized":"2023-10-19T13:00:21.238495+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:45:06.777683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.385057+0000","last_change":"2023-10-19T13:00:21.213911+0000","last_active":"2023-10-19T13:00:22.385057+0000","last_peered":"2023-10-19T13:00:22.385057+0000","last_clean":"2023-10-19T13:00:22.385057+0000","last_became_active":"2023-10-19T13:00:21.213624+0000","last_became_peered":"2023-10-19T13:00:21.213624+0000","last_unstale":"2023-10-19T13:00:22.385057+0000","last_undegraded":"2023-10-19T13:00:22.385057+0000","last_fullsized":"2023-10-19T13:00:22.385057+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:37:52.369486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"86'1","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.221295+0000","last_change":"2023-10-19T13:00:20.194934+0000","last_active":"2023-10-19T13:00:22.221295+0000","last_peered":"2023-10-19T13:00:22.221295+0000","last_clean":"2023-10-19T13:00:22.221295+0000","last_became_active":"2023-10-19T13:00:20.194657+0000","last_became_peered":"2023-10-19T13:00:20.194657+0000","last_unstale":"2023-10-19T13:00:22.221295+0000","last_undegraded":"2023-10-19T13:00:22.221295+0000","last_fullsized":"2023-10-19T13:00:22.221295+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:12:57.142360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.221258+0000","last_change":"2023-10-19T13:00:21.218060+0000","last_active":"2023-10-19T13:00:22.221258+0000","last_peered":"2023-10-19T13:00:22.221258+0000","last_clean":"2023-10-19T13:00:22.221258+0000","last_became_active":"2023-10-19T13:00:21.217766+0000","last_became_peered":"2023-10-19T13:00:21.217766+0000","last_unstale":"2023-10-19T13:00:22.221258+0000","last_undegraded":"2023-10-19T13:00:22.221258+0000","last_fullsized":"2023-10-19T13:00:22.221258+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:50:26.459690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"86'2","reported_seq":14,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.220954+0000","last_change":"2023-10-19T13:00:20.188312+0000","last_active":"2023-10-19T13:00:22.220954+0000","last_peered":"2023-10-19T13:00:22.220954+0000","last_clean":"2023-10-19T13:00:22.220954+0000","last_became_active":"2023-10-19T13:00:20.187673+0000","last_became_peered":"2023-10-19T13:00:20.187673+0000","last_unstale":"2023-10-19T13:00:22.220954+0000","last_undegraded":"2023-10-19T13:00:22.220954+0000","last_fullsized":"2023-10-19T13:00:22.220954+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:22:13.247630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.222925+0000","last_change":"2023-10-19T13:00:21.232599+0000","last_active":"2023-10-19T13:00:22.222925+0000","last_peered":"2023-10-19T13:00:22.222925+0000","last_clean":"2023-10-19T13:00:22.222925+0000","last_became_active":"2023-10-19T13:00:21.231174+0000","last_became_peered":"2023-10-19T13:00:21.231174+0000","last_unstale":"2023-10-19T13:00:22.222925+0000","last_undegraded":"2023-10-19T13:00:22.222925+0000","last_fullsized":"2023-10-19T13:00:22.222925+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:22:14.250042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"86'2","reported_seq":14,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.221016+0000","last_change":"2023-10-19T13:00:20.197923+0000","last_active":"2023-10-19T13:00:22.221016+0000","last_peered":"2023-10-19T13:00:22.221016+0000","last_clean":"2023-10-19T13:00:22.221016+0000","last_became_active":"2023-10-19T13:00:20.197546+0000","last_became_peered":"2023-10-19T13:00:20.197546+0000","last_unstale":"2023-10-19T13:00:22.221016+0000","last_undegraded":"2023-10-19T13:00:22.221016+0000","last_fullsized":"2023-10-19T13:00:22.221016+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:27:45.352182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:23.044381+0000","last_change":"2023-10-19T13:00:21.229195+0000","last_active":"2023-10-19T13:00:23.044381+0000","last_peered":"2023-10-19T13:00:23.044381+0000","last_clean":"2023-10-19T13:00:23.044381+0000","last_became_active":"2023-10-19T13:00:21.228898+0000","last_became_peered":"2023-10-19T13:00:21.228898+0000","last_unstale":"2023-10-19T13:00:23.044381+0000","last_undegraded":"2023-10-19T13:00:23.044381+0000","last_fullsized":"2023-10-19T13:00:23.044381+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:28:07.228859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"86'1","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.222099+0000","last_change":"2023-10-19T13:00:20.187648+0000","last_active":"2023-10-19T13:00:22.222099+0000","last_peered":"2023-10-19T13:00:22.222099+0000","last_clean":"2023-10-19T13:00:22.222099+0000","last_became_active":"2023-10-19T13:00:20.187366+0000","last_became_peered":"2023-10-19T13:00:20.187366+0000","last_unstale":"2023-10-19T13:00:22.222099+0000","last_undegraded":"2023-10-19T13:00:22.222099+0000","last_fullsized":"2023-10-19T13:00:22.222099+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:04:54.169253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":9,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-19T13:00:21.232881+0000","last_change":"2023-10-19T13:00:21.226995+0000","last_active":"2023-10-19T13:00:21.232881+0000","last_peered":"2023-10-19T13:00:21.232881+0000","last_clean":"2023-10-19T13:00:21.232881+0000","last_became_active":"2023-10-19T13:00:21.226673+0000","last_became_peered":"2023-10-19T13:00:21.226673+0000","last_unstale":"2023-10-19T13:00:21.232881+0000","last_undegraded":"2023-10-19T13:00:21.232881+0000","last_fullsized":"2023-10-19T13:00:21.232881+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:52:56.240367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1a","version":"86'1","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.221892+0000","last_change":"2023-10-19T13:00:20.187521+0000","last_active":"2023-10-19T13:00:22.221892+0000","last_peered":"2023-10-19T13:00:22.221892+0000","last_clean":"2023-10-19T13:00:22.221892+0000","last_became_active":"2023-10-19T13:00:20.186630+0000","last_became_peered":"2023-10-19T13:00:20.186630+0000","last_unstale":"2023-10-19T13:00:22.221892+0000","last_undegraded":"2023-10-19T13:00:22.221892+0000","last_fullsized":"2023-10-19T13:00:22.221892+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:37:14.044000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.423329+0000","last_change":"2023-10-19T13:00:21.219987+0000","last_active":"2023-10-19T13:00:22.423329+0000","last_peered":"2023-10-19T13:00:22.423329+0000","last_clean":"2023-10-19T13:00:22.423329+0000","last_became_active":"2023-10-19T13:00:21.218983+0000","last_became_peered":"2023-10-19T13:00:21.218983+0000","last_unstale":"2023-10-19T13:00:22.423329+0000","last_undegraded":"2023-10-19T13:00:22.423329+0000","last_fullsized":"2023-10-19T13:00:22.423329+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:31:57.359031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.221811+0000","last_change":"2023-10-19T13:00:20.196675+0000","last_active":"2023-10-19T13:00:22.221811+0000","last_peered":"2023-10-19T13:00:22.221811+0000","last_clean":"2023-10-19T13:00:22.221811+0000","last_became_active":"2023-10-19T13:00:20.196319+0000","last_became_peered":"2023-10-19T13:00:20.196319+0000","last_unstale":"2023-10-19T13:00:22.221811+0000","last_undegraded":"2023-10-19T13:00:22.221811+0000","last_fullsized":"2023-10-19T13:00:22.221811+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:28:05.111808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.222072+0000","last_change":"2023-10-19T13:00:21.222186+0000","last_active":"2023-10-19T13:00:22.222072+0000","last_peered":"2023-10-19T13:00:22.222072+0000","last_clean":"2023-10-19T13:00:22.222072+0000","last_became_active":"2023-10-19T13:00:21.221912+0000","last_became_peered":"2023-10-19T13:00:21.221912+0000","last_unstale":"2023-10-19T13:00:22.222072+0000","last_undegraded":"2023-10-19T13:00:22.222072+0000","last_fullsized":"2023-10-19T13:00:22.222072+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:48:53.902328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.423270+0000","last_change":"2023-10-19T13:00:20.198229+0000","last_active":"2023-10-19T13:00:22.423270+0000","last_peered":"2023-10-19T13:00:22.423270+0000","last_clean":"2023-10-19T13:00:22.423270+0000","last_became_active":"2023-10-19T13:00:20.197850+0000","last_became_peered":"2023-10-19T13:00:20.197850+0000","last_unstale":"2023-10-19T13:00:22.423270+0000","last_undegraded":"2023-10-19T13:00:22.423270+0000","last_fullsized":"2023-10-19T13:00:22.423270+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:58:20.558973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":9,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-19T13:00:21.233976+0000","last_change":"2023-10-19T13:00:21.227321+0000","last_active":"2023-10-19T13:00:21.233976+0000","last_peered":"2023-10-19T13:00:21.233976+0000","last_clean":"2023-10-19T13:00:21.233976+0000","last_became_active":"2023-10-19T13:00:21.227028+0000","last_became_peered":"2023-10-19T13:00:21.227028+0000","last_unstale":"2023-10-19T13:00:21.233976+0000","last_undegraded":"2023-10-19T13:00:21.233976+0000","last_fullsized":"2023-10-19T13:00:21.233976+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:06:08.406600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.223055+0000","last_change":"2023-10-19T13:00:20.202547+0000","last_active":"2023-10-19T13:00:22.223055+0000","last_peered":"2023-10-19T13:00:22.223055+0000","last_clean":"2023-10-19T13:00:22.223055+0000","last_became_active":"2023-10-19T13:00:20.202218+0000","last_became_peered":"2023-10-19T13:00:20.202218+0000","last_unstale":"2023-10-19T13:00:22.223055+0000","last_undegraded":"2023-10-19T13:00:22.223055+0000","last_fullsized":"2023-10-19T13:00:22.223055+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:41:30.219455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.221899+0000","last_change":"2023-10-19T13:00:21.218934+0000","last_active":"2023-10-19T13:00:22.221899+0000","last_peered":"2023-10-19T13:00:22.221899+0000","last_clean":"2023-10-19T13:00:22.221899+0000","last_became_active":"2023-10-19T13:00:21.218623+0000","last_became_peered":"2023-10-19T13:00:21.218623+0000","last_unstale":"2023-10-19T13:00:22.221899+0000","last_undegraded":"2023-10-19T13:00:22.221899+0000","last_fullsized":"2023-10-19T13:00:22.221899+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:08:57.610716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.385036+0000","last_change":"2023-10-19T13:00:20.198305+0000","last_active":"2023-10-19T13:00:22.385036+0000","last_peered":"2023-10-19T13:00:22.385036+0000","last_clean":"2023-10-19T13:00:22.385036+0000","last_became_active":"2023-10-19T13:00:20.198010+0000","last_became_peered":"2023-10-19T13:00:20.198010+0000","last_unstale":"2023-10-19T13:00:22.385036+0000","last_undegraded":"2023-10-19T13:00:22.385036+0000","last_fullsized":"2023-10-19T13:00:22.385036+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:13:59.814299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.423705+0000","last_change":"2023-10-19T13:00:21.217676+0000","last_active":"2023-10-19T13:00:22.423705+0000","last_peered":"2023-10-19T13:00:22.423705+0000","last_clean":"2023-10-19T13:00:22.423705+0000","last_became_active":"2023-10-19T13:00:21.217413+0000","last_became_peered":"2023-10-19T13:00:21.217413+0000","last_unstale":"2023-10-19T13:00:22.423705+0000","last_undegraded":"2023-10-19T13:00:22.423705+0000","last_fullsized":"2023-10-19T13:00:22.423705+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:55:42.567075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.15","version":"86'3","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.222755+0000","last_change":"2023-10-19T13:00:20.202410+0000","last_active":"2023-10-19T13:00:22.222755+0000","last_peered":"2023-10-19T13:00:22.222755+0000","last_clean":"2023-10-19T13:00:22.222755+0000","last_became_active":"2023-10-19T13:00:20.202053+0000","last_became_peered":"2023-10-19T13:00:20.202053+0000","last_unstale":"2023-10-19T13:00:22.222755+0000","last_undegraded":"2023-10-19T13:00:22.222755+0000","last_fullsized":"2023-10-19T13:00:22.222755+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:20:39.435252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.385090+0000","last_change":"2023-10-19T13:00:21.224440+0000","last_active":"2023-10-19T13:00:22.385090+0000","last_peered":"2023-10-19T13:00:22.385090+0000","last_clean":"2023-10-19T13:00:22.385090+0000","last_became_active":"2023-10-19T13:00:21.224089+0000","last_became_peered":"2023-10-19T13:00:21.224089+0000","last_unstale":"2023-10-19T13:00:22.385090+0000","last_undegraded":"2023-10-19T13:00:22.385090+0000","last_fullsized":"2023-10-19T13:00:22.385090+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:25:21.025263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"86'6","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-19T13:00:21.240123+0000","last_change":"2023-10-19T13:00:20.198504+0000","last_active":"2023-10-19T13:00:21.240123+0000","last_peered":"2023-10-19T13:00:21.240123+0000","last_clean":"2023-10-19T13:00:21.240123+0000","last_became_active":"2023-10-19T13:00:20.197655+0000","last_became_peered":"2023-10-19T13:00:20.197655+0000","last_unstale":"2023-10-19T13:00:21.240123+0000","last_undegraded":"2023-10-19T13:00:21.240123+0000","last_fullsized":"2023-10-19T13:00:21.240123+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:25:20.022851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.384529+0000","last_change":"2023-10-19T13:00:21.214443+0000","last_active":"2023-10-19T13:00:22.384529+0000","last_peered":"2023-10-19T13:00:22.384529+0000","last_clean":"2023-10-19T13:00:22.384529+0000","last_became_active":"2023-10-19T13:00:21.214184+0000","last_became_peered":"2023-10-19T13:00:21.214184+0000","last_unstale":"2023-10-19T13:00:22.384529+0000","last_undegraded":"2023-10-19T13:00:22.384529+0000","last_fullsized":"2023-10-19T13:00:22.384529+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:06:08.406600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.13","version":"86'2","reported_seq":14,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.222611+0000","last_change":"2023-10-19T13:00:20.196998+0000","last_active":"2023-10-19T13:00:22.222611+0000","last_peered":"2023-10-19T13:00:22.222611+0000","last_clean":"2023-10-19T13:00:22.222611+0000","last_became_active":"2023-10-19T13:00:20.196205+0000","last_became_peered":"2023-10-19T13:00:20.196205+0000","last_unstale":"2023-10-19T13:00:22.222611+0000","last_undegraded":"2023-10-19T13:00:22.222611+0000","last_fullsized":"2023-10-19T13:00:22.222611+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:29:02.236981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:23.044542+0000","last_change":"2023-10-19T13:00:21.218889+0000","last_active":"2023-10-19T13:00:23.044542+0000","last_peered":"2023-10-19T13:00:23.044542+0000","last_clean":"2023-10-19T13:00:23.044542+0000","last_became_active":"2023-10-19T13:00:21.218604+0000","last_became_peered":"2023-10-19T13:00:21.218604+0000","last_unstale":"2023-10-19T13:00:23.044542+0000","last_undegraded":"2023-10-19T13:00:23.044542+0000","last_fullsized":"2023-10-19T13:00:23.044542+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:00:49.808529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:23.044575+0000","last_change":"2023-10-19T13:00:20.197561+0000","last_active":"2023-10-19T13:00:23.044575+0000","last_peered":"2023-10-19T13:00:23.044575+0000","last_clean":"2023-10-19T13:00:23.044575+0000","last_became_active":"2023-10-19T13:00:20.197265+0000","last_became_peered":"2023-10-19T13:00:20.197265+0000","last_unstale":"2023-10-19T13:00:23.044575+0000","last_undegraded":"2023-10-19T13:00:23.044575+0000","last_fullsized":"2023-10-19T13:00:23.044575+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:20:48.999042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.423360+0000","last_change":"2023-10-19T13:00:21.207696+0000","last_active":"2023-10-19T13:00:22.423360+0000","last_peered":"2023-10-19T13:00:22.423360+0000","last_clean":"2023-10-19T13:00:22.423360+0000","last_became_active":"2023-10-19T13:00:21.207276+0000","last_became_peered":"2023-10-19T13:00:21.207276+0000","last_unstale":"2023-10-19T13:00:22.423360+0000","last_undegraded":"2023-10-19T13:00:22.423360+0000","last_fullsized":"2023-10-19T13:00:22.423360+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:03:44.562412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.222526+0000","last_change":"2023-10-19T13:00:20.197213+0000","last_active":"2023-10-19T13:00:22.222526+0000","last_peered":"2023-10-19T13:00:22.222526+0000","last_clean":"2023-10-19T13:00:22.222526+0000","last_became_active":"2023-10-19T13:00:20.196943+0000","last_became_peered":"2023-10-19T13:00:20.196943+0000","last_unstale":"2023-10-19T13:00:22.222526+0000","last_undegraded":"2023-10-19T13:00:22.222526+0000","last_fullsized":"2023-10-19T13:00:22.222526+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:05:35.715244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":9,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-19T13:00:21.232936+0000","last_change":"2023-10-19T13:00:21.228024+0000","last_active":"2023-10-19T13:00:21.232936+0000","last_peered":"2023-10-19T13:00:21.232936+0000","last_clean":"2023-10-19T13:00:21.232936+0000","last_became_active":"2023-10-19T13:00:21.227727+0000","last_became_peered":"2023-10-19T13:00:21.227727+0000","last_unstale":"2023-10-19T13:00:21.232936+0000","last_undegraded":"2023-10-19T13:00:21.232936+0000","last_fullsized":"2023-10-19T13:00:21.232936+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:14:55.381409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.10","version":"86'8","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.384437+0000","last_change":"2023-10-19T13:00:20.195915+0000","last_active":"2023-10-19T13:00:22.384437+0000","last_peered":"2023-10-19T13:00:22.384437+0000","last_clean":"2023-10-19T13:00:22.384437+0000","last_became_active":"2023-10-19T13:00:20.195641+0000","last_became_peered":"2023-10-19T13:00:20.195641+0000","last_unstale":"2023-10-19T13:00:22.384437+0000","last_undegraded":"2023-10-19T13:00:22.384437+0000","last_fullsized":"2023-10-19T13:00:22.384437+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:52:53.680755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.222779+0000","last_change":"2023-10-19T13:00:21.226358+0000","last_active":"2023-10-19T13:00:22.222779+0000","last_peered":"2023-10-19T13:00:22.222779+0000","last_clean":"2023-10-19T13:00:22.222779+0000","last_became_active":"2023-10-19T13:00:21.226094+0000","last_became_peered":"2023-10-19T13:00:21.226094+0000","last_unstale":"2023-10-19T13:00:22.222779+0000","last_undegraded":"2023-10-19T13:00:22.222779+0000","last_fullsized":"2023-10-19T13:00:22.222779+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:12:58.144772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"86'1","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:23.044984+0000","last_change":"2023-10-19T13:00:20.198629+0000","last_active":"2023-10-19T13:00:23.044984+0000","last_peered":"2023-10-19T13:00:23.044984+0000","last_clean":"2023-10-19T13:00:23.044984+0000","last_became_active":"2023-10-19T13:00:20.198381+0000","last_became_peered":"2023-10-19T13:00:20.198381+0000","last_unstale":"2023-10-19T13:00:23.044984+0000","last_undegraded":"2023-10-19T13:00:23.044984+0000","last_fullsized":"2023-10-19T13:00:23.044984+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:18:37.647497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.423742+0000","last_change":"2023-10-19T13:00:21.215295+0000","last_active":"2023-10-19T13:00:22.423742+0000","last_peered":"2023-10-19T13:00:22.423742+0000","last_clean":"2023-10-19T13:00:22.423742+0000","last_became_active":"2023-10-19T13:00:21.214996+0000","last_became_peered":"2023-10-19T13:00:21.214996+0000","last_unstale":"2023-10-19T13:00:22.423742+0000","last_undegraded":"2023-10-19T13:00:22.423742+0000","last_fullsized":"2023-10-19T13:00:22.423742+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:58:41.166338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"86'2","reported_seq":14,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.222910+0000","last_change":"2023-10-19T13:00:20.196057+0000","last_active":"2023-10-19T13:00:22.222910+0000","last_peered":"2023-10-19T13:00:22.222910+0000","last_clean":"2023-10-19T13:00:22.222910+0000","last_became_active":"2023-10-19T13:00:20.195717+0000","last_became_peered":"2023-10-19T13:00:20.195717+0000","last_unstale":"2023-10-19T13:00:22.222910+0000","last_undegraded":"2023-10-19T13:00:22.222910+0000","last_fullsized":"2023-10-19T13:00:22.222910+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:11:52.601802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.384933+0000","last_change":"2023-10-19T13:00:21.213116+0000","last_active":"2023-10-19T13:00:22.384933+0000","last_peered":"2023-10-19T13:00:22.384933+0000","last_clean":"2023-10-19T13:00:22.384933+0000","last_became_active":"2023-10-19T13:00:21.212842+0000","last_became_peered":"2023-10-19T13:00:21.212842+0000","last_unstale":"2023-10-19T13:00:22.384933+0000","last_undegraded":"2023-10-19T13:00:22.384933+0000","last_fullsized":"2023-10-19T13:00:22.384933+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:55:42.567075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"86'1","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.222864+0000","last_change":"2023-10-19T13:00:20.202706+0000","last_active":"2023-10-19T13:00:22.222864+0000","last_peered":"2023-10-19T13:00:22.222864+0000","last_clean":"2023-10-19T13:00:22.222864+0000","last_became_active":"2023-10-19T13:00:20.202398+0000","last_became_peered":"2023-10-19T13:00:20.202398+0000","last_unstale":"2023-10-19T13:00:22.222864+0000","last_undegraded":"2023-10-19T13:00:22.222864+0000","last_fullsized":"2023-10-19T13:00:22.222864+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:24:33.951829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.423732+0000","last_change":"2023-10-19T13:00:21.223843+0000","last_active":"2023-10-19T13:00:22.423732+0000","last_peered":"2023-10-19T13:00:22.423732+0000","last_clean":"2023-10-19T13:00:22.423732+0000","last_became_active":"2023-10-19T13:00:21.223493+0000","last_became_peered":"2023-10-19T13:00:21.223493+0000","last_unstale":"2023-10-19T13:00:22.423732+0000","last_undegraded":"2023-10-19T13:00:22.423732+0000","last_fullsized":"2023-10-19T13:00:22.423732+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:14:55.381409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.222072+0000","last_change":"2023-10-19T13:00:20.193764+0000","last_active":"2023-10-19T13:00:22.222072+0000","last_peered":"2023-10-19T13:00:22.222072+0000","last_clean":"2023-10-19T13:00:22.222072+0000","last_became_active":"2023-10-19T13:00:20.193512+0000","last_became_peered":"2023-10-19T13:00:20.193512+0000","last_unstale":"2023-10-19T13:00:22.222072+0000","last_undegraded":"2023-10-19T13:00:22.222072+0000","last_fullsized":"2023-10-19T13:00:22.222072+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:02:09.714383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.220733+0000","last_change":"2023-10-19T13:00:21.217141+0000","last_active":"2023-10-19T13:00:22.220733+0000","last_peered":"2023-10-19T13:00:22.220733+0000","last_clean":"2023-10-19T13:00:22.220733+0000","last_became_active":"2023-10-19T13:00:21.216807+0000","last_became_peered":"2023-10-19T13:00:21.216807+0000","last_unstale":"2023-10-19T13:00:22.220733+0000","last_undegraded":"2023-10-19T13:00:22.220733+0000","last_fullsized":"2023-10-19T13:00:22.220733+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:39:34.098405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"86'2","reported_seq":14,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.220911+0000","last_change":"2023-10-19T13:00:20.195565+0000","last_active":"2023-10-19T13:00:22.220911+0000","last_peered":"2023-10-19T13:00:22.220911+0000","last_clean":"2023-10-19T13:00:22.220911+0000","last_became_active":"2023-10-19T13:00:20.195256+0000","last_became_peered":"2023-10-19T13:00:20.195256+0000","last_unstale":"2023-10-19T13:00:22.220911+0000","last_undegraded":"2023-10-19T13:00:22.220911+0000","last_fullsized":"2023-10-19T13:00:22.220911+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:18:37.647497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:23.044579+0000","last_change":"2023-10-19T13:00:21.220030+0000","last_active":"2023-10-19T13:00:23.044579+0000","last_peered":"2023-10-19T13:00:23.044579+0000","last_clean":"2023-10-19T13:00:23.044579+0000","last_became_active":"2023-10-19T13:00:21.219790+0000","last_became_peered":"2023-10-19T13:00:21.219790+0000","last_unstale":"2023-10-19T13:00:23.044579+0000","last_undegraded":"2023-10-19T13:00:23.044579+0000","last_fullsized":"2023-10-19T13:00:23.044579+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:00:42.269401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:23.044670+0000","last_change":"2023-10-19T13:00:20.197800+0000","last_active":"2023-10-19T13:00:23.044670+0000","last_peered":"2023-10-19T13:00:23.044670+0000","last_clean":"2023-10-19T13:00:23.044670+0000","last_became_active":"2023-10-19T13:00:20.197482+0000","last_became_peered":"2023-10-19T13:00:20.197482+0000","last_unstale":"2023-10-19T13:00:23.044670+0000","last_undegraded":"2023-10-19T13:00:23.044670+0000","last_fullsized":"2023-10-19T13:00:23.044670+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:11:52.601802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.423437+0000","last_change":"2023-10-19T13:00:21.220368+0000","last_active":"2023-10-19T13:00:22.423437+0000","last_peered":"2023-10-19T13:00:22.423437+0000","last_clean":"2023-10-19T13:00:22.423437+0000","last_became_active":"2023-10-19T13:00:21.219726+0000","last_became_peered":"2023-10-19T13:00:21.219726+0000","last_unstale":"2023-10-19T13:00:22.423437+0000","last_undegraded":"2023-10-19T13:00:22.423437+0000","last_fullsized":"2023-10-19T13:00:22.423437+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:06:08.406600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.220731+0000","last_change":"2023-10-19T13:00:20.194028+0000","last_active":"2023-10-19T13:00:22.220731+0000","last_peered":"2023-10-19T13:00:22.220731+0000","last_clean":"2023-10-19T13:00:22.220731+0000","last_became_active":"2023-10-19T13:00:20.193758+0000","last_became_peered":"2023-10-19T13:00:20.193758+0000","last_unstale":"2023-10-19T13:00:22.220731+0000","last_undegraded":"2023-10-19T13:00:22.220731+0000","last_fullsized":"2023-10-19T13:00:22.220731+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:41:30.219455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.221024+0000","last_change":"2023-10-19T13:00:21.221247+0000","last_active":"2023-10-19T13:00:22.221024+0000","last_peered":"2023-10-19T13:00:22.221024+0000","last_clean":"2023-10-19T13:00:22.221024+0000","last_became_active":"2023-10-19T13:00:21.220182+0000","last_became_peered":"2023-10-19T13:00:21.220182+0000","last_unstale":"2023-10-19T13:00:22.221024+0000","last_undegraded":"2023-10-19T13:00:22.221024+0000","last_fullsized":"2023-10-19T13:00:22.221024+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:47:08.944926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"86'2","reported_seq":14,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:23.045126+0000","last_change":"2023-10-19T13:00:20.193049+0000","last_active":"2023-10-19T13:00:23.045126+0000","last_peered":"2023-10-19T13:00:23.045126+0000","last_clean":"2023-10-19T13:00:23.045126+0000","last_became_active":"2023-10-19T13:00:20.192788+0000","last_became_peered":"2023-10-19T13:00:20.192788+0000","last_unstale":"2023-10-19T13:00:23.045126+0000","last_undegraded":"2023-10-19T13:00:23.045126+0000","last_fullsized":"2023-10-19T13:00:23.045126+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:07:06.063303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:23.044679+0000","last_change":"2023-10-19T13:00:21.222511+0000","last_active":"2023-10-19T13:00:23.044679+0000","last_peered":"2023-10-19T13:00:23.044679+0000","last_clean":"2023-10-19T13:00:23.044679+0000","last_became_active":"2023-10-19T13:00:21.222236+0000","last_became_peered":"2023-10-19T13:00:21.222236+0000","last_unstale":"2023-10-19T13:00:23.044679+0000","last_undegraded":"2023-10-19T13:00:23.044679+0000","last_fullsized":"2023-10-19T13:00:23.044679+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:50:46.028354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:23.045102+0000","last_change":"2023-10-19T13:00:20.186527+0000","last_active":"2023-10-19T13:00:23.045102+0000","last_peered":"2023-10-19T13:00:23.045102+0000","last_clean":"2023-10-19T13:00:23.045102+0000","last_became_active":"2023-10-19T13:00:20.186271+0000","last_became_peered":"2023-10-19T13:00:20.186271+0000","last_unstale":"2023-10-19T13:00:23.045102+0000","last_undegraded":"2023-10-19T13:00:23.045102+0000","last_fullsized":"2023-10-19T13:00:23.045102+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:26:39.899099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:23.044992+0000","last_change":"2023-10-19T13:00:21.223718+0000","last_active":"2023-10-19T13:00:23.044992+0000","last_peered":"2023-10-19T13:00:23.044992+0000","last_clean":"2023-10-19T13:00:23.044992+0000","last_became_active":"2023-10-19T13:00:21.223259+0000","last_became_peered":"2023-10-19T13:00:21.223259+0000","last_unstale":"2023-10-19T13:00:23.044992+0000","last_undegraded":"2023-10-19T13:00:23.044992+0000","last_fullsized":"2023-10-19T13:00:23.044992+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:55:17.988760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.221031+0000","last_change":"2023-10-19T13:00:20.188530+0000","last_active":"2023-10-19T13:00:22.221031+0000","last_peered":"2023-10-19T13:00:22.221031+0000","last_clean":"2023-10-19T13:00:22.221031+0000","last_became_active":"2023-10-19T13:00:20.188074+0000","last_became_peered":"2023-10-19T13:00:20.188074+0000","last_unstale":"2023-10-19T13:00:22.221031+0000","last_undegraded":"2023-10-19T13:00:22.221031+0000","last_fullsized":"2023-10-19T13:00:22.221031+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:08:44.211054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.423799+0000","last_change":"2023-10-19T13:00:21.223166+0000","last_active":"2023-10-19T13:00:22.423799+0000","last_peered":"2023-10-19T13:00:22.423799+0000","last_clean":"2023-10-19T13:00:22.423799+0000","last_became_active":"2023-10-19T13:00:21.222906+0000","last_became_peered":"2023-10-19T13:00:21.222906+0000","last_unstale":"2023-10-19T13:00:22.423799+0000","last_undegraded":"2023-10-19T13:00:22.423799+0000","last_fullsized":"2023-10-19T13:00:22.423799+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:37:52.369486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-19T13:00:21.216870+0000","last_change":"2023-10-19T13:00:20.198569+0000","last_active":"2023-10-19T13:00:21.216870+0000","last_peered":"2023-10-19T13:00:21.216870+0000","last_clean":"2023-10-19T13:00:21.216870+0000","last_became_active":"2023-10-19T13:00:20.197088+0000","last_became_peered":"2023-10-19T13:00:20.197088+0000","last_unstale":"2023-10-19T13:00:21.216870+0000","last_undegraded":"2023-10-19T13:00:21.216870+0000","last_fullsized":"2023-10-19T13:00:21.216870+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:38:39.386315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.423286+0000","last_change":"2023-10-19T13:00:21.232429+0000","last_active":"2023-10-19T13:00:22.423286+0000","last_peered":"2023-10-19T13:00:22.423286+0000","last_clean":"2023-10-19T13:00:22.423286+0000","last_became_active":"2023-10-19T13:00:21.232143+0000","last_became_peered":"2023-10-19T13:00:21.232143+0000","last_unstale":"2023-10-19T13:00:22.423286+0000","last_undegraded":"2023-10-19T13:00:22.423286+0000","last_fullsized":"2023-10-19T13:00:22.423286+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:27:17.189492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.222621+0000","last_change":"2023-10-19T13:00:20.198187+0000","last_active":"2023-10-19T13:00:22.222621+0000","last_peered":"2023-10-19T13:00:22.222621+0000","last_clean":"2023-10-19T13:00:22.222621+0000","last_became_active":"2023-10-19T13:00:20.196844+0000","last_became_peered":"2023-10-19T13:00:20.196844+0000","last_unstale":"2023-10-19T13:00:22.222621+0000","last_undegraded":"2023-10-19T13:00:22.222621+0000","last_fullsized":"2023-10-19T13:00:22.222621+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:20:48.999042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.384657+0000","last_change":"2023-10-19T13:00:21.223216+0000","last_active":"2023-10-19T13:00:22.384657+0000","last_peered":"2023-10-19T13:00:22.384657+0000","last_clean":"2023-10-19T13:00:22.384657+0000","last_became_active":"2023-10-19T13:00:21.222881+0000","last_became_peered":"2023-10-19T13:00:21.222881+0000","last_unstale":"2023-10-19T13:00:22.384657+0000","last_undegraded":"2023-10-19T13:00:22.384657+0000","last_fullsized":"2023-10-19T13:00:22.384657+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:41:19.821669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.221947+0000","last_change":"2023-10-19T13:00:20.192942+0000","last_active":"2023-10-19T13:00:22.221947+0000","last_peered":"2023-10-19T13:00:22.221947+0000","last_clean":"2023-10-19T13:00:22.221947+0000","last_became_active":"2023-10-19T13:00:20.192688+0000","last_became_peered":"2023-10-19T13:00:20.192688+0000","last_unstale":"2023-10-19T13:00:22.221947+0000","last_undegraded":"2023-10-19T13:00:22.221947+0000","last_fullsized":"2023-10-19T13:00:22.221947+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:16:31.679428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":9,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-19T13:00:21.227472+0000","last_change":"2023-10-19T13:00:21.226366+0000","last_active":"2023-10-19T13:00:21.227472+0000","last_peered":"2023-10-19T13:00:21.227472+0000","last_clean":"2023-10-19T13:00:21.227472+0000","last_became_active":"2023-10-19T13:00:21.226075+0000","last_became_peered":"2023-10-19T13:00:21.226075+0000","last_unstale":"2023-10-19T13:00:21.227472+0000","last_undegraded":"2023-10-19T13:00:21.227472+0000","last_fullsized":"2023-10-19T13:00:21.227472+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:33:36.402003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.221906+0000","last_change":"2023-10-19T13:00:20.195720+0000","last_active":"2023-10-19T13:00:22.221906+0000","last_peered":"2023-10-19T13:00:22.221906+0000","last_clean":"2023-10-19T13:00:22.221906+0000","last_became_active":"2023-10-19T13:00:20.195451+0000","last_became_peered":"2023-10-19T13:00:20.195451+0000","last_unstale":"2023-10-19T13:00:22.221906+0000","last_undegraded":"2023-10-19T13:00:22.221906+0000","last_fullsized":"2023-10-19T13:00:22.221906+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:35:48.984686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:23.044464+0000","last_change":"2023-10-19T13:00:21.219143+0000","last_active":"2023-10-19T13:00:23.044464+0000","last_peered":"2023-10-19T13:00:23.044464+0000","last_clean":"2023-10-19T13:00:23.044464+0000","last_became_active":"2023-10-19T13:00:21.218834+0000","last_became_peered":"2023-10-19T13:00:21.218834+0000","last_unstale":"2023-10-19T13:00:23.044464+0000","last_undegraded":"2023-10-19T13:00:23.044464+0000","last_fullsized":"2023-10-19T13:00:23.044464+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:29:26.836263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-19T13:00:21.215556+0000","last_change":"2023-10-19T13:00:20.198235+0000","last_active":"2023-10-19T13:00:21.215556+0000","last_peered":"2023-10-19T13:00:21.215556+0000","last_clean":"2023-10-19T13:00:21.215556+0000","last_became_active":"2023-10-19T13:00:20.197920+0000","last_became_peered":"2023-10-19T13:00:20.197920+0000","last_unstale":"2023-10-19T13:00:21.215556+0000","last_undegraded":"2023-10-19T13:00:21.215556+0000","last_fullsized":"2023-10-19T13:00:21.215556+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:34:56.789811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.423517+0000","last_change":"2023-10-19T13:00:21.231904+0000","last_active":"2023-10-19T13:00:22.423517+0000","last_peered":"2023-10-19T13:00:22.423517+0000","last_clean":"2023-10-19T13:00:22.423517+0000","last_became_active":"2023-10-19T13:00:21.231629+0000","last_became_peered":"2023-10-19T13:00:21.231629+0000","last_unstale":"2023-10-19T13:00:22.423517+0000","last_undegraded":"2023-10-19T13:00:22.423517+0000","last_fullsized":"2023-10-19T13:00:22.423517+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:20.171814+0000","last_clean_scrub_stamp":"2023-10-19T13:00:20.171814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:41:19.821669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-19T13:00:22.221722+0000","last_change":"2023-10-19T13:00:20.187569+0000","last_active":"2023-10-19T13:00:22.221722+0000","last_peered":"2023-10-19T13:00:22.221722+0000","last_clean":"2023-10-19T13:00:22.221722+0000","last_became_active":"2023-10-19T13:00:20.187271+0000","last_became_peered":"2023-10-19T13:00:20.187271+0000","last_unstale":"2023-10-19T13:00:22.221722+0000","last_undegraded":"2023-10-19T13:00:22.221722+0000","last_fullsized":"2023-10-19T13:00:22.221722+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:19.169402+0000","last_clean_scrub_stamp":"2023-10-19T13:00:19.169402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:51:04.372619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"76'161","reported_seq":385,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-19T13:00:21.215648+0000","last_change":"2023-10-19T12:44:08.636613+0000","last_active":"2023-10-19T13:00:21.215648+0000","last_peered":"2023-10-19T13:00:21.215648+0000","last_clean":"2023-10-19T13:00:21.215648+0000","last_became_active":"2023-10-19T12:44:08.635697+0000","last_became_peered":"2023-10-19T12:44:08.635697+0000","last_unstale":"2023-10-19T13:00:21.215648+0000","last_undegraded":"2023-10-19T13:00:21.215648+0000","last_fullsized":"2023-10-19T13:00:21.215648+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_clean_scrub_stamp":"2023-10-19T12:44:07.399738+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:15:23.685485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575046,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59184,"kb_used_data":1628,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312656,"statfs":{"total":96636764160,"available":96576159744,"internally_reserved":0,"allocated":1667072,"data_stored":1120016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58915275},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:59:44 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.57499999999999996,"15min":0.56999999999999995},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.248},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.60099999999999998,"15min":0.58799999999999997},"min":{"1min":0.45400000000000001,"5min":0.33200000000000002,"15min":0.26900000000000002},"max":{"1min":0.78200000000000003,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.36499999999999999}]},{"osd":1,"last update":"Thu Oct 19 12:59:44 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.58199999999999996,"15min":0.57599999999999996},"min":{"1min":0.41299999999999998,"5min":0.40300000000000002,"15min":0.28199999999999997},"max":{"1min":0.84299999999999997,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.623,"5min":0.59999999999999998,"15min":0.58799999999999997},"min":{"1min":0.51700000000000002,"5min":0.43099999999999999,"15min":0.29299999999999998},"max":{"1min":0.82699999999999996,"5min":1.284,"15min":1.284},"last":0.34399999999999997}]},{"osd":2,"last update":"Thu Oct 19 12:59:44 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.64100000000000001,"15min":0.629},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.375},"max":{"1min":0.79300000000000004,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.64600000000000002,"15min":0.63300000000000001},"min":{"1min":0.55300000000000005,"5min":0.41599999999999998,"15min":0.28399999999999997},"max":{"1min":0.81200000000000006,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.5}]},{"osd":3,"last update":"Thu Oct 19 12:59:44 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60699999999999998,"15min":0.60099999999999998},"min":{"1min":0.47199999999999998,"5min":0.34999999999999998,"15min":0.34399999999999997},"max":{"1min":0.78900000000000003,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.67000000000000004,"15min":0.65500000000000003},"min":{"1min":0.58199999999999996,"5min":0.45300000000000001,"15min":0.39700000000000002},"max":{"1min":0.92000000000000004,"5min":1.329,"15min":1.329},"last":0.51900000000000002}]},{"osd":4,"last update":"Thu Oct 19 12:59:44 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.46800000000000003,"15min":0.46700000000000003},"min":{"1min":0.36899999999999999,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.72599999999999998,"5min":0.75800000000000001,"15min":0.80500000000000005},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.33600000000000002,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.623,"5min":0.75600000000000001,"15min":0.78900000000000003},"last":0.57699999999999996}]},{"osd":5,"last update":"Thu Oct 19 12:59:44 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.48299999999999998,"15min":0.48499999999999999},"min":{"1min":0.34399999999999997,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.68600000000000005,"5min":0.75800000000000001,"15min":0.90400000000000003},"last":0.438},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.52200000000000002,"15min":0.51600000000000001},"min":{"1min":0.38,"5min":0.314,"15min":0.308},"max":{"1min":0.71499999999999997,"5min":0.78700000000000003,"15min":0.92200000000000004},"last":0.45000000000000001}]},{"osd":6,"last update":"Thu Oct 19 12:59:44 2023","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.54200000000000004,"15min":0.54700000000000004},"min":{"1min":0.379,"5min":0.308,"15min":0.308},"max":{"1min":0.749,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.52100000000000002,"15min":0.52900000000000003},"min":{"1min":0.36899999999999999,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.76900000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.54100000000000004}]}]},{"osd":6,"up_from":13,"seq":55834575046,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32212,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339628,"statfs":{"total":96636764160,"available":96603779072,"internally_reserved":0,"allocated":1048576,"data_stored":522258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:59:28 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.57999999999999996,"15min":0.58699999999999997},"min":{"1min":0.41899999999999998,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.755,"5min":0.79900000000000004,"15min":0.81999999999999995},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.41999999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.52200000000000002}]},{"osd":1,"last update":"Thu Oct 19 12:59:28 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.60299999999999998,"15min":0.59999999999999998},"min":{"1min":0.39900000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.71999999999999997,"5min":0.83299999999999996,"15min":0.93899999999999995},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.59699999999999998,"15min":0.59999999999999998},"min":{"1min":0.42599999999999999,"5min":0.34300000000000003,"15min":0.28399999999999997},"max":{"1min":0.78200000000000003,"5min":0.82099999999999995,"15min":1.075},"last":0.45000000000000001}]},{"osd":2,"last update":"Thu Oct 19 12:59:28 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.42799999999999999,"5min":0.307,"15min":0.26600000000000001},"max":{"1min":0.84699999999999998,"5min":0.85299999999999998,"15min":0.85699999999999998},"last":0.499},{"interface":"front","average":{"1min":0.624,"5min":0.61299999999999999,"15min":0.60799999999999998},"min":{"1min":0.48499999999999999,"5min":0.35999999999999999,"15min":0.28399999999999997},"max":{"1min":0.82199999999999995,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.72199999999999998}]},{"osd":3,"last update":"Thu Oct 19 12:59:28 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.625,"15min":0.623},"min":{"1min":0.46400000000000002,"5min":0.38500000000000001,"15min":0.32600000000000001},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.61799999999999999,"15min":0.628},"min":{"1min":0.44900000000000001,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":1.0900000000000001},"last":0.65100000000000002}]},{"osd":4,"last update":"Thu Oct 19 12:59:28 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.46100000000000002,"15min":0.46999999999999997},"min":{"1min":0.32900000000000001,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.64400000000000002,"5min":0.77900000000000003,"15min":1.02},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.47999999999999998,"15min":0.48499999999999999},"min":{"1min":0.33200000000000002,"5min":0.32800000000000001,"15min":0.247},"max":{"1min":0.69099999999999995,"5min":0.752,"15min":0.80000000000000004},"last":0.67300000000000004}]},{"osd":5,"last update":"Thu Oct 19 12:59:28 2023","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.46800000000000003,"15min":0.47699999999999998},"min":{"1min":0.32900000000000001,"5min":0.30199999999999999,"15min":0.26600000000000001},"max":{"1min":0.61899999999999999,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.433},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.50800000000000001,"15min":0.51900000000000002},"min":{"1min":0.35699999999999998,"5min":0.32100000000000001,"15min":0.312},"max":{"1min":0.75600000000000001,"5min":0.85699999999999998,"15min":0.98799999999999999},"last":0.747}]},{"osd":7,"last update":"Thu Oct 19 12:59:28 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.51800000000000002,"15min":0.52300000000000002},"min":{"1min":0.32500000000000001,"5min":0.29299999999999998,"15min":0.28199999999999997},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":1.036},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53100000000000003,"15min":0.54100000000000004},"min":{"1min":0.36499999999999999,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":1.1299999999999999},"last":0.58899999999999997}]}]},{"osd":5,"up_from":13,"seq":55834575046,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32204,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339636,"statfs":{"total":96636764160,"available":96603787264,"internally_reserved":0,"allocated":1048576,"data_stored":522246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:59:29 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.58799999999999997,"15min":0.58899999999999997},"min":{"1min":0.44700000000000001,"5min":0.28999999999999998,"15min":0.223},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.88800000000000001},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.63,"5min":0.58999999999999997,"15min":0.59599999999999997},"min":{"1min":0.42499999999999999,"5min":0.36099999999999999,"15min":0.27800000000000002},"max":{"1min":0.81399999999999995,"5min":0.84299999999999997,"15min":0.874},"last":0.47099999999999997}]},{"osd":1,"last update":"Thu Oct 19 12:59:29 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.57899999999999996,"15min":0.59399999999999997},"min":{"1min":0.46899999999999997,"5min":0.34599999999999997,"15min":0.32800000000000001},"max":{"1min":0.77500000000000002,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.59999999999999998,"15min":0.60599999999999998},"min":{"1min":0.49199999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83499999999999996},"last":0.55000000000000004}]},{"osd":2,"last update":"Thu Oct 19 12:59:29 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.57299999999999995,"15min":0.58299999999999996},"min":{"1min":0.48899999999999999,"5min":0.34100000000000003,"15min":0.307},"max":{"1min":0.73999999999999999,"5min":0.78600000000000003,"15min":0.80700000000000005},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.60499999999999998,"15min":0.61899999999999999},"min":{"1min":0.50900000000000001,"5min":0.38400000000000001,"15min":0.33700000000000002},"max":{"1min":0.80900000000000005,"5min":0.88100000000000001,"15min":0.93600000000000005},"last":0.753}]},{"osd":3,"last update":"Thu Oct 19 12:59:29 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.59299999999999997,"15min":0.60199999999999998},"min":{"1min":0.48499999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.751,"5min":0.79900000000000004,"15min":0.874},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.625,"15min":0.63400000000000001},"min":{"1min":0.50900000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.89500000000000002},"last":0.53700000000000003}]},{"osd":4,"last update":"Thu Oct 19 12:59:29 2023","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.36099999999999999,"5min":0.29099999999999998,"15min":0.28899999999999998},"max":{"1min":0.61399999999999999,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.46700000000000003,"15min":0.47199999999999998},"min":{"1min":0.35899999999999999,"5min":0.251,"15min":0.23899999999999999},"max":{"1min":0.58299999999999996,"5min":0.79400000000000004,"15min":0.80500000000000005},"last":0.69199999999999995}]},{"osd":6,"last update":"Thu Oct 19 12:59:29 2023","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46500000000000002,"15min":0.47299999999999998},"min":{"1min":0.34499999999999997,"5min":0.27200000000000002,"15min":0.25900000000000001},"max":{"1min":0.56299999999999994,"5min":0.74399999999999999,"15min":0.79300000000000004},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.497,"5min":0.50900000000000001,"15min":0.502},"min":{"1min":0.373,"5min":0.34200000000000003,"15min":0.30399999999999999},"max":{"1min":0.64800000000000002,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.626}]},{"osd":7,"last update":"Thu Oct 19 12:59:29 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.48599999999999999,"15min":0.49299999999999999},"min":{"1min":0.39400000000000002,"5min":0.32000000000000001,"15min":0.315},"max":{"1min":0.64200000000000002,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.53900000000000003,"15min":0.53700000000000003},"min":{"1min":0.46100000000000002,"5min":0.311,"15min":0.27500000000000002},"max":{"1min":0.66900000000000004,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.74099999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575046,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32228,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339612,"statfs":{"total":96636764160,"available":96603762688,"internally_reserved":0,"allocated":1069056,"data_stored":530484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56100000000000005,"15min":0.55900000000000005},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.28899999999999998},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.626},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.55800000000000005,"15min":0.55500000000000005},"min":{"1min":0.45400000000000001,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.77100000000000002,"5min":0.77800000000000002,"15min":0.78200000000000003},"last":0.60099999999999998}]},{"osd":1,"last update":"Thu Oct 19 12:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.60399999999999998,"15min":0.60799999999999998},"min":{"1min":0.33000000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.78600000000000003,"5min":0.80600000000000005,"15min":0.86199999999999999},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61899999999999999,"15min":0.61599999999999999},"min":{"1min":0.40899999999999997,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.77300000000000002,"5min":0.79900000000000004,"15min":0.873},"last":0.54500000000000004}]},{"osd":2,"last update":"Thu Oct 19 12:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.61199999999999999,"15min":0.61299999999999999},"min":{"1min":0.375,"5min":0.32200000000000001,"15min":0.254},"max":{"1min":0.72199999999999998,"5min":0.85299999999999998,"15min":0.86599999999999999},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61699999999999999,"15min":0.61599999999999999},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.34499999999999997},"max":{"1min":0.78500000000000003,"5min":0.82999999999999996,"15min":0.85999999999999999},"last":0.72999999999999998}]},{"osd":3,"last update":"Thu Oct 19 12:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.59099999999999997,"15min":0.58399999999999996},"min":{"1min":0.41999999999999998,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.73499999999999999,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.64600000000000002,"15min":0.63700000000000001},"min":{"1min":0.52500000000000002,"5min":0.42599999999999999,"15min":0.37},"max":{"1min":0.73899999999999999,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.64800000000000002}]},{"osd":5,"last update":"Thu Oct 19 12:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.45600000000000002,"15min":0.45200000000000001},"min":{"1min":0.309,"5min":0.29599999999999999,"15min":0.249},"max":{"1min":0.63500000000000001,"5min":0.79600000000000004,"15min":0.81299999999999994},"last":0.44},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.47499999999999998,"15min":0.46200000000000002},"min":{"1min":0.34699999999999998,"5min":0.26400000000000001,"15min":0.23499999999999999},"max":{"1min":0.623,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.35699999999999998}]},{"osd":6,"last update":"Thu Oct 19 12:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.45900000000000002,"15min":0.45600000000000002},"min":{"1min":0.31,"5min":0.29499999999999998,"15min":0.26400000000000001},"max":{"1min":0.52800000000000002,"5min":0.72599999999999998,"15min":0.78600000000000003},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.48599999999999999,"15min":0.497},"min":{"1min":0.34300000000000003,"5min":0.32100000000000001,"15min":0.27100000000000002},"max":{"1min":0.66700000000000004,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.68400000000000005}]},{"osd":7,"last update":"Thu Oct 19 12:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.501,"15min":0.503},"min":{"1min":0.33700000000000002,"5min":0.307,"15min":0.27500000000000002},"max":{"1min":0.68300000000000005,"5min":0.79100000000000004,"15min":0.80200000000000005},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.503,"5min":0.54200000000000004,"15min":0.54800000000000004},"min":{"1min":0.39100000000000001,"5min":0.35799999999999998,"15min":0.30199999999999999},"max":{"1min":0.69399999999999995,"5min":0.80500000000000005,"15min":0.83499999999999996},"last":0.70799999999999996}]}]},{"osd":3,"up_from":13,"seq":55834575045,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36380,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335460,"statfs":{"total":96636764160,"available":96599511040,"internally_reserved":0,"allocated":1056768,"data_stored":528872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:59:24 2023","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.41799999999999998,"15min":0.39900000000000002},"min":{"1min":0.27100000000000002,"5min":0.23599999999999999,"15min":0.20899999999999999},"max":{"1min":0.69099999999999995,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.41199999999999998,"15min":0.41299999999999998},"min":{"1min":0.316,"5min":0.223,"15min":0.223},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.45500000000000002}]},{"osd":1,"last update":"Thu Oct 19 12:59:24 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.40600000000000003,"15min":0.40699999999999997},"min":{"1min":0.29099999999999998,"5min":0.26100000000000001,"15min":0.22500000000000001},"max":{"1min":0.63500000000000001,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.505,"5min":0.48299999999999998,"15min":0.45700000000000002},"min":{"1min":0.27600000000000002,"5min":0.24099999999999999,"15min":0.215},"max":{"1min":0.66700000000000004,"5min":2.9399999999999999,"15min":2.9399999999999999},"last":0.41799999999999998}]},{"osd":2,"last update":"Thu Oct 19 12:59:24 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.46899999999999997,"15min":0.46500000000000002},"min":{"1min":0.33800000000000002,"5min":0.314,"15min":0.26100000000000001},"max":{"1min":0.69399999999999995,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.48099999999999998,"15min":0.48499999999999999},"min":{"1min":0.35099999999999998,"5min":0.29999999999999999,"15min":0.247},"max":{"1min":0.754,"5min":0.754,"15min":0.96499999999999997},"last":0.51000000000000001}]},{"osd":4,"last update":"Thu Oct 19 12:59:24 2023","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.65700000000000003,"15min":0.66400000000000003},"min":{"1min":0.50700000000000001,"5min":0.42299999999999999,"15min":0.40300000000000002},"max":{"1min":0.90200000000000002,"5min":0.90500000000000003,"15min":1.0069999999999999},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.69499999999999995,"15min":0.69199999999999995},"min":{"1min":0.59499999999999997,"5min":0.46899999999999997,"15min":0.42099999999999999},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.99399999999999999},"last":0.57899999999999996}]},{"osd":5,"last update":"Thu Oct 19 12:59:24 2023","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.67600000000000005,"15min":0.65900000000000003},"min":{"1min":0.56000000000000005,"5min":0.45500000000000002,"15min":0.40500000000000003},"max":{"1min":0.93600000000000005,"5min":2.9990000000000001,"15min":2.9990000000000001},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.52100000000000002,"5min":0.40899999999999997,"15min":0.35899999999999999},"max":{"1min":0.81899999999999995,"5min":1.774,"15min":1.774},"last":0.77300000000000002}]},{"osd":6,"last update":"Thu Oct 19 12:59:24 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.66100000000000003,"15min":0.65400000000000003},"min":{"1min":0.52300000000000002,"5min":0.433,"15min":0.41199999999999998},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.96799999999999997},"last":0.752},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.67500000000000004,"15min":0.67400000000000004},"min":{"1min":0.60299999999999998,"5min":0.377,"15min":0.35899999999999999},"max":{"1min":0.93700000000000006,"5min":1.167,"15min":1.167},"last":0.64700000000000002}]},{"osd":7,"last update":"Thu Oct 19 12:59:24 2023","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.65200000000000002,"15min":0.66500000000000004},"min":{"1min":0.45500000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.94299999999999995,"5min":1.042,"15min":1.042},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.69299999999999995,"15min":0.67900000000000005},"min":{"1min":0.54200000000000004,"5min":0.432,"15min":0.432},"max":{"1min":0.89200000000000002,"5min":2.8149999999999999,"15min":2.8149999999999999},"last":0.66400000000000003}]}]},{"osd":2,"up_from":13,"seq":55834575047,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54416,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317424,"statfs":{"total":96636764160,"available":96581042176,"internally_reserved":0,"allocated":1044480,"data_stored":522224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:59:33 2023","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.377,"15min":0.38100000000000001},"min":{"1min":0.27500000000000002,"5min":0.20899999999999999,"15min":0.187},"max":{"1min":0.54000000000000004,"5min":0.67700000000000005,"15min":0.83999999999999997},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.436,"5min":0.40999999999999998,"15min":0.40600000000000003},"min":{"1min":0.32900000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.58299999999999996,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.56000000000000005}]},{"osd":1,"last update":"Thu Oct 19 12:59:33 2023","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.39400000000000002,"15min":0.39800000000000002},"min":{"1min":0.318,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.80400000000000005},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.41999999999999998,"15min":0.42999999999999999},"min":{"1min":0.38200000000000001,"5min":0.22,"15min":0.22},"max":{"1min":0.59499999999999997,"5min":0.60499999999999998,"15min":0.78600000000000003},"last":0.61299999999999999}]},{"osd":3,"last update":"Thu Oct 19 12:59:33 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.46300000000000002,"15min":0.45700000000000002},"min":{"1min":0.33400000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.498},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.48099999999999998,"15min":0.48499999999999999},"min":{"1min":0.40300000000000002,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.72799999999999998,"5min":0.77800000000000002,"15min":0.87},"last":0.57499999999999996}]},{"osd":4,"last update":"Thu Oct 19 12:59:33 2023","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.63700000000000001,"15min":0.64200000000000002},"min":{"1min":0.497,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.89500000000000002,"5min":0.93999999999999995,"15min":1.0569999999999999},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.65400000000000003,"15min":0.65700000000000003},"min":{"1min":0.54300000000000004,"5min":0.35099999999999998,"15min":0.32000000000000001},"max":{"1min":0.83399999999999996,"5min":0.92300000000000004,"15min":1.0780000000000001},"last":0.70999999999999996}]},{"osd":5,"last update":"Thu Oct 19 12:59:33 2023","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.67400000000000004,"15min":0.66200000000000003},"min":{"1min":0.58299999999999996,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.84299999999999997,"5min":0.96499999999999997,"15min":0.99399999999999999},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.65200000000000002,"15min":0.65800000000000003},"min":{"1min":0.56899999999999995,"5min":0.375,"15min":0.375},"max":{"1min":0.872,"5min":0.94599999999999995,"15min":0.97099999999999997},"last":0.64000000000000001}]},{"osd":6,"last update":"Thu Oct 19 12:59:33 2023","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.65000000000000002,"15min":0.66700000000000004},"min":{"1min":0.47199999999999998,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.82399999999999995,"5min":1.008,"15min":1.008},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.67800000000000005,"15min":0.67400000000000004},"min":{"1min":0.52000000000000002,"5min":0.371,"15min":0.371},"max":{"1min":0.88100000000000001,"5min":0.995,"15min":0.995},"last":0.63}]},{"osd":7,"last update":"Thu Oct 19 12:59:33 2023","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.67500000000000004,"15min":0.68500000000000005},"min":{"1min":0.59699999999999998,"5min":0.39300000000000002,"15min":0.36399999999999999},"max":{"1min":0.86099999999999999,"5min":1.0149999999999999,"15min":1.111},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.68899999999999995,"15min":0.68300000000000005},"min":{"1min":0.53000000000000003,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.89800000000000002,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.65300000000000002}]}]},{"osd":1,"up_from":13,"seq":55834575046,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32236,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339604,"statfs":{"total":96636764160,"available":96603754496,"internally_reserved":0,"allocated":1073152,"data_stored":524600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 12:59:33 2023","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.39200000000000002,"15min":0.38500000000000001},"min":{"1min":0.23999999999999999,"5min":0.21299999999999999,"15min":0.187},"max":{"1min":0.55800000000000005,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.442,"15min":0.42799999999999999},"min":{"1min":0.29499999999999998,"5min":0.23899999999999999,"15min":0.21199999999999999},"max":{"1min":0.58799999999999997,"5min":2.5800000000000001,"15min":2.5800000000000001},"last":0.52700000000000002}]},{"osd":2,"last update":"Thu Oct 19 12:59:33 2023","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.42499999999999999,"15min":0.42099999999999999},"min":{"1min":0.309,"5min":0.21299999999999999,"15min":0.20100000000000001},"max":{"1min":0.53300000000000003,"5min":0.60499999999999998,"15min":0.69999999999999996},"last":0.502},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.47099999999999997,"15min":0.45300000000000001},"min":{"1min":0.34999999999999998,"5min":0.313,"15min":0.23699999999999999},"max":{"1min":0.625,"5min":2.6230000000000002,"15min":2.6230000000000002},"last":0.57999999999999996}]},{"osd":3,"last update":"Thu Oct 19 12:59:33 2023","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.51600000000000001,"15min":0.48599999999999999},"min":{"1min":0.38900000000000001,"5min":0.307,"15min":0.26200000000000001},"max":{"1min":0.621,"5min":3.9740000000000002,"15min":3.9740000000000002},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.43099999999999999,"15min":0.42299999999999999},"min":{"1min":0.35999999999999999,"5min":0.27700000000000002,"15min":0.27000000000000002},"max":{"1min":0.59299999999999997,"5min":0.63200000000000001,"15min":0.80500000000000005},"last":0.51400000000000001}]},{"osd":4,"last update":"Thu Oct 19 12:59:33 2023","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.65600000000000003,"15min":0.65200000000000002},"min":{"1min":0.501,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.85699999999999998,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.66600000000000004,"15min":0.64800000000000002},"min":{"1min":0.47399999999999998,"5min":0.42599999999999999,"15min":0.314},"max":{"1min":0.85399999999999998,"5min":1.0860000000000001,"15min":1.1910000000000001},"last":0.76300000000000001}]},{"osd":5,"last update":"Thu Oct 19 12:59:33 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.65700000000000003,"15min":0.65300000000000002},"min":{"1min":0.499,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.83599999999999997,"5min":1.1200000000000001,"15min":1.1200000000000001},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.69599999999999995,"15min":0.67300000000000004},"min":{"1min":0.54300000000000004,"5min":0.42299999999999999,"15min":0.38700000000000001},"max":{"1min":0.81799999999999995,"5min":2.7120000000000002,"15min":2.7120000000000002},"last":0.77900000000000003}]},{"osd":6,"last update":"Thu Oct 19 12:59:33 2023","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.65800000000000003,"15min":0.65700000000000003},"min":{"1min":0.53800000000000003,"5min":0.36799999999999999,"15min":0.36399999999999999},"max":{"1min":0.91300000000000003,"5min":1.0069999999999999,"15min":1.077},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67400000000000004,"15min":0.67800000000000005},"min":{"1min":0.55800000000000005,"5min":0.376,"15min":0.376},"max":{"1min":0.77500000000000002,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.72099999999999997}]},{"osd":7,"last update":"Thu Oct 19 12:59:33 2023","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.69799999999999995,"15min":0.67900000000000005},"min":{"1min":0.54300000000000004,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.84999999999999998,"5min":2.786,"15min":2.786},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.68200000000000005,"15min":0.67900000000000005},"min":{"1min":0.47499999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":1.0489999999999999},"last":0.79500000000000004}]}]},{"osd":0,"up_from":13,"seq":55834575047,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36952,"kb_used_data":1608,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334888,"statfs":{"total":96636764160,"available":96598925312,"internally_reserved":0,"allocated":1646592,"data_stored":1118621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Oct 19 12:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.40300000000000002,"15min":0.39100000000000001},"min":{"1min":0.26500000000000001,"5min":0.182,"15min":0.182},"max":{"1min":0.58899999999999997,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.434,"15min":0.432},"min":{"1min":0.34599999999999997,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.746},"last":0.38200000000000001}]},{"osd":2,"last update":"Thu Oct 19 12:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.46700000000000003,"15min":0.45500000000000002},"min":{"1min":0.33200000000000002,"5min":0.29499999999999998,"15min":0.25800000000000001},"max":{"1min":0.58599999999999997,"5min":0.752,"15min":0.76300000000000001},"last":0.309},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.437,"15min":0.434},"min":{"1min":0.26000000000000001,"5min":0.216,"15min":0.20699999999999999},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.76800000000000002},"last":0.432}]},{"osd":3,"last update":"Thu Oct 19 12:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.42199999999999999,"15min":0.40899999999999997},"min":{"1min":0.34000000000000002,"5min":0.23400000000000001,"15min":0.22800000000000001},"max":{"1min":0.53500000000000003,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.505,"5min":0.504,"15min":0.49099999999999999},"min":{"1min":0.33600000000000002,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.66300000000000003,"5min":1.4990000000000001,"15min":1.4990000000000001},"last":0.34799999999999998}]},{"osd":4,"last update":"Thu Oct 19 12:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.63100000000000001,"15min":0.63600000000000001},"min":{"1min":0.46600000000000003,"5min":0.40000000000000002,"15min":0.34999999999999998},"max":{"1min":0.749,"5min":1.361,"15min":1.361},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.65700000000000003,"15min":0.65000000000000002},"min":{"1min":0.52900000000000003,"5min":0.41899999999999998,"15min":0.35999999999999999},"max":{"1min":0.871,"5min":0.98999999999999999,"15min":0.998},"last":0.57699999999999996}]},{"osd":5,"last update":"Thu Oct 19 12:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.68999999999999995,"15min":0.68000000000000005},"min":{"1min":0.50700000000000001,"5min":0.378,"15min":0.378},"max":{"1min":0.94599999999999995,"5min":1.325,"15min":1.325},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.69299999999999995,"15min":0.68600000000000005},"min":{"1min":0.45900000000000002,"5min":0.36299999999999999,"15min":0.318},"max":{"1min":1.02,"5min":1.024,"15min":1.056},"last":0.625}]},{"osd":6,"last update":"Thu Oct 19 12:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.67200000000000004,"15min":0.66400000000000003},"min":{"1min":0.59999999999999998,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.873,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.67600000000000005,"15min":0.66600000000000004},"min":{"1min":0.57199999999999995,"5min":0.41999999999999998,"15min":0.38900000000000001},"max":{"1min":0.92000000000000004,"5min":0.995,"15min":1.109},"last":0.65600000000000003}]},{"osd":7,"last update":"Thu Oct 19 12:59:41 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.68700000000000006,"15min":0.68400000000000005},"min":{"1min":0.53500000000000003,"5min":0.41199999999999998,"15min":0.41099999999999998},"max":{"1min":0.89900000000000002,"5min":0.98799999999999999,"15min":1.0089999999999999},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.67300000000000004,"15min":0.66300000000000003},"min":{"1min":0.58799999999999997,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.90400000000000003,"5min":1.389,"15min":1.389},"last":0.67100000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-19T13:00:29.007 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-19T13:00:29.008 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-19T13:00:29.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.155+0000 7fc9a5610640 1 -- 172.21.15.57:0/294044154 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc9a006ffb0 msgr2=0x7fc9a00c96b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:29.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.155+0000 7fc9a5610640 1 --2- 172.21.15.57:0/294044154 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc9a006ffb0 0x7fc9a00c96b0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7fc9a006ced0 tx=0x7fc98802bda0 comp rx=0 tx=0).stop 2023-10-19T13:00:29.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.155+0000 7fc9a5610640 1 -- 172.21.15.57:0/294044154 shutdown_connections 2023-10-19T13:00:29.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.155+0000 7fc9a5610640 1 --2- 172.21.15.57:0/294044154 >> v2:172.21.15.179:3300/0 conn(0x7fc9a00c9bf0 0x7fc9a0154af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:29.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.155+0000 7fc9a5610640 1 --2- 172.21.15.57:0/294044154 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc9a006ffb0 0x7fc9a00c96b0 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:29.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.155+0000 7fc9a5610640 1 --2- 172.21.15.57:0/294044154 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc9a00b1920 0x7fc9a006fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:29.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.155+0000 7fc9a5610640 1 -- 172.21.15.57:0/294044154 >> 172.21.15.57:0/294044154 conn(0x7fc9a0069930 msgr2=0x7fc9a0069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:29.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.155+0000 7fc9a5610640 1 -- 172.21.15.57:0/294044154 shutdown_connections 2023-10-19T13:00:29.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.155+0000 7fc9a5610640 1 -- 172.21.15.57:0/294044154 wait complete. 2023-10-19T13:00:29.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.156+0000 7fc9a5610640 1 Processor -- start 2023-10-19T13:00:29.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.156+0000 7fc9a5610640 1 -- start start 2023-10-19T13:00:29.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.157+0000 7fc9a5610640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc9a006ffb0 0x7fc9a01516c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:29.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.157+0000 7fc9a5610640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc9a00b1920 0x7fc9a015d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:29.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.157+0000 7fc9a5610640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc9a00c9bf0 0x7fc9a01f1790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:29.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.157+0000 7fc9a5610640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9a015c580 con 0x7fc9a00c9bf0 2023-10-19T13:00:29.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.157+0000 7fc9a5610640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9a0151c00 con 0x7fc9a006ffb0 2023-10-19T13:00:29.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.157+0000 7fc9a5610640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9a01581a0 con 0x7fc9a00b1920 2023-10-19T13:00:29.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.157+0000 7fc99f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc9a00c9bf0 0x7fc9a01f1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:29.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.157+0000 7fc99f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc9a00c9bf0 0x7fc9a01f1790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56362/0 (socket says 172.21.15.57:56362) 2023-10-19T13:00:29.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.157+0000 7fc99f7fe640 1 -- 172.21.15.57:0/4079932848 learned_addr learned my addr 172.21.15.57:0/4079932848 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:29.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.157+0000 7fc99e7fc640 1 --2- 172.21.15.57:0/4079932848 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc9a00b1920 0x7fc9a015d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:29.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.157+0000 7fc99effd640 1 --2- 172.21.15.57:0/4079932848 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc9a006ffb0 0x7fc9a01516c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:29.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.157+0000 7fc99f7fe640 1 -- 172.21.15.57:0/4079932848 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc9a00b1920 msgr2=0x7fc9a015d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:29.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.157+0000 7fc99f7fe640 1 --2- 172.21.15.57:0/4079932848 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc9a00b1920 0x7fc9a015d270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:29.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.157+0000 7fc99f7fe640 1 -- 172.21.15.57:0/4079932848 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc9a006ffb0 msgr2=0x7fc9a01516c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:29.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.157+0000 7fc99f7fe640 1 --2- 172.21.15.57:0/4079932848 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc9a006ffb0 0x7fc9a01516c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:29.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.157+0000 7fc99f7fe640 1 -- 172.21.15.57:0/4079932848 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc988009560 con 0x7fc9a00c9bf0 2023-10-19T13:00:29.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.158+0000 7fc99f7fe640 1 --2- 172.21.15.57:0/4079932848 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc9a00c9bf0 0x7fc9a01f1790 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7fc99000e960 tx=0x7fc99000ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:29.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.158+0000 7fc9977fe640 1 -- 172.21.15.57:0/4079932848 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc990004d20 con 0x7fc9a00c9bf0 2023-10-19T13:00:29.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.158+0000 7fc9977fe640 1 -- 172.21.15.57:0/4079932848 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fc99000cb20 con 0x7fc9a00c9bf0 2023-10-19T13:00:29.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.158+0000 7fc9a5610640 1 -- 172.21.15.57:0/4079932848 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9a015c790 con 0x7fc9a00c9bf0 2023-10-19T13:00:29.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.158+0000 7fc9977fe640 1 -- 172.21.15.57:0/4079932848 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc990005020 con 0x7fc9a00c9bf0 2023-10-19T13:00:29.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.158+0000 7fc9a5610640 1 -- 172.21.15.57:0/4079932848 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9a015cd50 con 0x7fc9a00c9bf0 2023-10-19T13:00:29.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.159+0000 7fc9a5610640 1 -- 172.21.15.57:0/4079932848 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9a015cf40 con 0x7fc9a00c9bf0 2023-10-19T13:00:29.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.159+0000 7fc9977fe640 1 -- 172.21.15.57:0/4079932848 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 32) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc990005180 con 0x7fc9a00c9bf0 2023-10-19T13:00:29.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.160+0000 7fc9977fe640 1 --2- 172.21.15.57:0/4079932848 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7fc98006af60 0x7fc98006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:29.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.160+0000 7fc9977fe640 1 -- 172.21.15.57:0/4079932848 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fc99008a230 con 0x7fc9a00c9bf0 2023-10-19T13:00:29.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.160+0000 7fc99effd640 1 --2- 172.21.15.57:0/4079932848 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7fc98006af60 0x7fc98006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:29.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.161+0000 7fc99effd640 1 --2- 172.21.15.57:0/4079932848 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7fc98006af60 0x7fc98006d420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc99800bbe0 tx=0x7fc99800b390 comp rx=0 tx=0).ready entity=mgr.6256 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:29.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.164+0000 7fc9977fe640 1 -- 172.21.15.57:0/4079932848 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc990055010 con 0x7fc9a00c9bf0 2023-10-19T13:00:29.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.408+0000 7fc9a5610640 1 -- 172.21.15.57:0/4079932848 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fc9a00690e0 con 0x7fc9a00c9bf0 2023-10-19T13:00:29.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.411+0000 7fc9977fe640 1 -- 172.21.15.57:0/4079932848 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v24) v1 ==== 190+0+147 (secure 0 0 0) 0x7fc990058630 con 0x7fc9a00c9bf0 2023-10-19T13:00:29.414 INFO:teuthology.orchestra.run.smithi057.stdout:[client.0] 2023-10-19T13:00:29.414 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQBkJDFlg9yuExAATxiWDfS+p07G1F0pORbJEg== 2023-10-19T13:00:29.415 INFO:teuthology.orchestra.run.smithi057.stdout: caps mds = "allow" 2023-10-19T13:00:29.415 INFO:teuthology.orchestra.run.smithi057.stdout: caps mon = "allow r" 2023-10-19T13:00:29.415 INFO:teuthology.orchestra.run.smithi057.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-19T13:00:29.416 INFO:teuthology.orchestra.run.smithi057.stderr:updated caps for client.0 2023-10-19T13:00:29.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.413+0000 7fc9a5610640 1 -- 172.21.15.57:0/4079932848 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7fc98006af60 msgr2=0x7fc98006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:29.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.413+0000 7fc9a5610640 1 --2- 172.21.15.57:0/4079932848 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7fc98006af60 0x7fc98006d420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc99800bbe0 tx=0x7fc99800b390 comp rx=0 tx=0).stop 2023-10-19T13:00:29.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.413+0000 7fc9a5610640 1 -- 172.21.15.57:0/4079932848 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc9a00c9bf0 msgr2=0x7fc9a01f1790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:29.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.413+0000 7fc9a5610640 1 --2- 172.21.15.57:0/4079932848 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc9a00c9bf0 0x7fc9a01f1790 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7fc99000e960 tx=0x7fc99000ee20 comp rx=0 tx=0).stop 2023-10-19T13:00:29.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.413+0000 7fc9a5610640 1 -- 172.21.15.57:0/4079932848 shutdown_connections 2023-10-19T13:00:29.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.413+0000 7fc9a5610640 1 --2- 172.21.15.57:0/4079932848 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7fc98006af60 0x7fc98006d420 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:29.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.413+0000 7fc9a5610640 1 --2- 172.21.15.57:0/4079932848 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc9a00c9bf0 0x7fc9a01f1790 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:29.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.413+0000 7fc9a5610640 1 --2- 172.21.15.57:0/4079932848 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc9a00b1920 0x7fc9a015d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:29.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.413+0000 7fc9a5610640 1 --2- 172.21.15.57:0/4079932848 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc9a006ffb0 0x7fc9a01516c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:29.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.413+0000 7fc9a5610640 1 -- 172.21.15.57:0/4079932848 >> 172.21.15.57:0/4079932848 conn(0x7fc9a0069930 msgr2=0x7fc9a01538c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:29.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.413+0000 7fc9a5610640 1 -- 172.21.15.57:0/4079932848 shutdown_connections 2023-10-19T13:00:29.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.413+0000 7fc9a5610640 1 -- 172.21.15.57:0/4079932848 wait complete. 2023-10-19T13:00:29.427 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T13:00:29.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.533+0000 7f7e8bae5640 1 -- 172.21.15.57:0/3793621709 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e84142850 msgr2=0x7f7e84159440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:29.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.533+0000 7f7e8bae5640 1 --2- 172.21.15.57:0/3793621709 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e84142850 0x7f7e84159440 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f7e80009830 tx=0x7f7e8002bcd0 comp rx=0 tx=0).stop 2023-10-19T13:00:29.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.534+0000 7f7e8bae5640 1 -- 172.21.15.57:0/3793621709 shutdown_connections 2023-10-19T13:00:29.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.534+0000 7f7e8bae5640 1 --2- 172.21.15.57:0/3793621709 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7e840cd6d0 0x7f7e840cdb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:29.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.534+0000 7f7e8bae5640 1 --2- 172.21.15.57:0/3793621709 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e84142850 0x7f7e84159440 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:29.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.534+0000 7f7e8bae5640 1 -- 172.21.15.57:0/3793621709 >> 172.21.15.57:0/3793621709 conn(0x7f7e840af7a0 msgr2=0x7f7e840afb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:29.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.534+0000 7f7e8bae5640 1 -- 172.21.15.57:0/3793621709 shutdown_connections 2023-10-19T13:00:29.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.534+0000 7f7e8bae5640 1 -- 172.21.15.57:0/3793621709 wait complete. 2023-10-19T13:00:29.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.535+0000 7f7e8bae5640 1 Processor -- start 2023-10-19T13:00:29.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.535+0000 7f7e8bae5640 1 -- start start 2023-10-19T13:00:29.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.535+0000 7f7e8bae5640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7e84069720 0x7f7e84158740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:29.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.535+0000 7f7e8bae5640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7e840cd6d0 0x7f7e840ac950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:29.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.536+0000 7f7e8bae5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e84142850 0x7f7e840ace90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:29.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.536+0000 7f7e8bae5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e8413c050 con 0x7f7e84142850 2023-10-19T13:00:29.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.536+0000 7f7e8bae5640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e8413c330 con 0x7f7e840cd6d0 2023-10-19T13:00:29.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.536+0000 7f7e8bae5640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e8413c610 con 0x7f7e84069720 2023-10-19T13:00:29.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.536+0000 7f7e8a082640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e84142850 0x7f7e840ace90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:29.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.536+0000 7f7e8a082640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e84142850 0x7f7e840ace90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56380/0 (socket says 172.21.15.57:56380) 2023-10-19T13:00:29.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.536+0000 7f7e8a082640 1 -- 172.21.15.57:0/818057599 learned_addr learned my addr 172.21.15.57:0/818057599 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:29.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.536+0000 7f7e89881640 1 --2- 172.21.15.57:0/818057599 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7e84069720 0x7f7e84158740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:29.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.536+0000 7f7e89080640 1 --2- 172.21.15.57:0/818057599 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7e840cd6d0 0x7f7e840ac950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:29.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.536+0000 7f7e8a082640 1 -- 172.21.15.57:0/818057599 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7e84069720 msgr2=0x7f7e84158740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:29.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.536+0000 7f7e8a082640 1 --2- 172.21.15.57:0/818057599 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7e84069720 0x7f7e84158740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:29.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.536+0000 7f7e8a082640 1 -- 172.21.15.57:0/818057599 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7e840cd6d0 msgr2=0x7f7e840ac950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:29.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.536+0000 7f7e8a082640 1 --2- 172.21.15.57:0/818057599 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7e840cd6d0 0x7f7e840ac950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:29.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.536+0000 7f7e8a082640 1 -- 172.21.15.57:0/818057599 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e80009490 con 0x7f7e84142850 2023-10-19T13:00:29.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.536+0000 7f7e89080640 1 --2- 172.21.15.57:0/818057599 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7e840cd6d0 0x7f7e840ac950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:00:29.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.536+0000 7f7e8a082640 1 --2- 172.21.15.57:0/818057599 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e84142850 0x7f7e840ace90 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f7e7c00caa0 tx=0x7f7e7c00cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:29.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.536+0000 7f7e7a7fc640 1 -- 172.21.15.57:0/818057599 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e7c00ac00 con 0x7f7e84142850 2023-10-19T13:00:29.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.536+0000 7f7e7a7fc640 1 -- 172.21.15.57:0/818057599 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f7e7c00ae70 con 0x7f7e84142850 2023-10-19T13:00:29.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.536+0000 7f7e8bae5640 1 -- 172.21.15.57:0/818057599 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e8413c8f0 con 0x7f7e84142850 2023-10-19T13:00:29.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.536+0000 7f7e7a7fc640 1 -- 172.21.15.57:0/818057599 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e7c012660 con 0x7f7e84142850 2023-10-19T13:00:29.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.536+0000 7f7e8bae5640 1 -- 172.21.15.57:0/818057599 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e84158c80 con 0x7f7e84142850 2023-10-19T13:00:29.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.537+0000 7f7e8bae5640 1 -- 172.21.15.57:0/818057599 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e8413c750 con 0x7f7e84142850 2023-10-19T13:00:29.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.538+0000 7f7e7a7fc640 1 -- 172.21.15.57:0/818057599 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 32) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f7e7c01b400 con 0x7f7e84142850 2023-10-19T13:00:29.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.538+0000 7f7e7a7fc640 1 --2- 172.21.15.57:0/818057599 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f7e6806af60 0x7f7e6806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:29.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.539+0000 7f7e7a7fc640 1 -- 172.21.15.57:0/818057599 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f7e7c08a3b0 con 0x7f7e84142850 2023-10-19T13:00:29.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.539+0000 7f7e89881640 1 --2- 172.21.15.57:0/818057599 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f7e6806af60 0x7f7e6806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:29.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.539+0000 7f7e89881640 1 --2- 172.21.15.57:0/818057599 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f7e6806af60 0x7f7e6806d420 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7e80036000 tx=0x7f7e80004730 comp rx=0 tx=0).ready entity=mgr.6256 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:29.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.543+0000 7f7e7a7fc640 1 -- 172.21.15.57:0/818057599 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7e8413c750 con 0x7f7e84142850 2023-10-19T13:00:29.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.755+0000 7f7e8bae5640 1 -- 172.21.15.57:0/818057599 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7e8413ce10 con 0x7f7e84142850 2023-10-19T13:00:29.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.755+0000 7f7e7a7fc640 1 -- 172.21.15.57:0/818057599 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+4917 (secure 0 0 0) 0x7f7e8413ce10 con 0x7f7e84142850 2023-10-19T13:00:29.758 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:00:29.759 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":30,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5966,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/765319531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":765319531},{"type":"v1","addr":"172.21.15.179:6837","nonce":765319531}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27},{"gid":5969,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/1388526553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":1388526553},{"type":"v1","addr":"172.21.15.179:6836","nonce":1388526553}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27},{"gid":7035,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/3237434259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":3237434259},{"type":"v1","addr":"172.21.15.57:6834","nonce":3237434259}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T13:00:20.549275+0000","modified":"2023-10-19T13:00:22.561271+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7038},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7038":{"gid":7038,"name":"a","rank":0,"incarnation":28,"state":"up:active","state_seq":3,"addr":"172.21.15.57:6835/2257211800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2257211800},{"type":"v1","addr":"172.21.15.57:6835","nonce":2257211800}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-10-19T13:00:29.759 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 30 2023-10-19T13:00:29.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.757+0000 7f7e8bae5640 1 -- 172.21.15.57:0/818057599 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f7e6806af60 msgr2=0x7f7e6806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:29.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.757+0000 7f7e8bae5640 1 --2- 172.21.15.57:0/818057599 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f7e6806af60 0x7f7e6806d420 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7e80036000 tx=0x7f7e80004730 comp rx=0 tx=0).stop 2023-10-19T13:00:29.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.757+0000 7f7e8bae5640 1 -- 172.21.15.57:0/818057599 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e84142850 msgr2=0x7f7e840ace90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:29.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.757+0000 7f7e8bae5640 1 --2- 172.21.15.57:0/818057599 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e84142850 0x7f7e840ace90 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f7e7c00caa0 tx=0x7f7e7c00cf60 comp rx=0 tx=0).stop 2023-10-19T13:00:29.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.757+0000 7f7e8bae5640 1 -- 172.21.15.57:0/818057599 shutdown_connections 2023-10-19T13:00:29.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.757+0000 7f7e8bae5640 1 --2- 172.21.15.57:0/818057599 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f7e6806af60 0x7f7e6806d420 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:29.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.757+0000 7f7e8bae5640 1 --2- 172.21.15.57:0/818057599 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e84142850 0x7f7e840ace90 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:29.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.757+0000 7f7e8bae5640 1 --2- 172.21.15.57:0/818057599 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7e840cd6d0 0x7f7e840ac950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:29.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.757+0000 7f7e8bae5640 1 --2- 172.21.15.57:0/818057599 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7e84069720 0x7f7e84158740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:29.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.757+0000 7f7e8bae5640 1 -- 172.21.15.57:0/818057599 >> 172.21.15.57:0/818057599 conn(0x7f7e840af7a0 msgr2=0x7f7e8413fdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:29.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.757+0000 7f7e8bae5640 1 -- 172.21.15.57:0/818057599 shutdown_connections 2023-10-19T13:00:29.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:29.757+0000 7f7e8bae5640 1 -- 172.21.15.57:0/818057599 wait complete. 2023-10-19T13:00:30.174 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-19T13:00:20.549275+0000', 'modified': '2023-10-19T13:00:22.561271+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7038}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7038': {'gid': 7038, 'name': 'a', 'rank': 0, 'incarnation': 28, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.57:6835/2257211800', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6833', 'nonce': 2257211800}, {'type': 'v1', 'addr': '172.21.15.57:6835', 'nonce': 2257211800}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [10], 'metadata_pool': 9, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-19T13:00:30.174 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-19T13:00:30.175 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2023-10-19T13:00:30.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.286+0000 7f5412b39640 1 -- 172.21.15.57:0/2200401560 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f540c1579f0 msgr2=0x7f540c0b0e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:30.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.286+0000 7f5412b39640 1 --2- 172.21.15.57:0/2200401560 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f540c1579f0 0x7f540c0b0e90 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f540c0acd30 tx=0x7f540002bda0 comp rx=0 tx=0).stop 2023-10-19T13:00:30.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.286+0000 7f5412b39640 1 -- 172.21.15.57:0/2200401560 shutdown_connections 2023-10-19T13:00:30.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.286+0000 7f5412b39640 1 --2- 172.21.15.57:0/2200401560 >> v2:172.21.15.179:3300/0 conn(0x7f540c06ac70 0x7f540c06ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:30.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.286+0000 7f5412b39640 1 --2- 172.21.15.57:0/2200401560 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f540c15c1f0 0x7f540c06a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:30.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.286+0000 7f5412b39640 1 --2- 172.21.15.57:0/2200401560 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f540c1579f0 0x7f540c0b0e90 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:30.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.286+0000 7f5412b39640 1 -- 172.21.15.57:0/2200401560 >> 172.21.15.57:0/2200401560 conn(0x7f540c06fae0 msgr2=0x7f540c06fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:30.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.287+0000 7f5412b39640 1 -- 172.21.15.57:0/2200401560 shutdown_connections 2023-10-19T13:00:30.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.287+0000 7f5412b39640 1 -- 172.21.15.57:0/2200401560 wait complete. 2023-10-19T13:00:30.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.288+0000 7f5412b39640 1 Processor -- start 2023-10-19T13:00:30.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.288+0000 7f5412b39640 1 -- start start 2023-10-19T13:00:30.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.288+0000 7f5412b39640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f540c06ac70 0x7f540c152aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:30.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.288+0000 7f5412b39640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f540c1579f0 0x7f540c147d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:30.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.288+0000 7f5412b39640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f540c15c1f0 0x7f540c1482c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:30.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.289+0000 7f5412b39640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f540c153f50 con 0x7f540c1579f0 2023-10-19T13:00:30.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.289+0000 7f5412b39640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f540c14f890 con 0x7f540c15c1f0 2023-10-19T13:00:30.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.289+0000 7f5412b39640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f540c156dc0 con 0x7f540c06ac70 2023-10-19T13:00:30.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.289+0000 7f540bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f540c1579f0 0x7f540c147d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:30.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.289+0000 7f540bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f540c1579f0 0x7f540c147d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56396/0 (socket says 172.21.15.57:56396) 2023-10-19T13:00:30.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.289+0000 7f540bfff640 1 -- 172.21.15.57:0/3131434690 learned_addr learned my addr 172.21.15.57:0/3131434690 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:30.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.289+0000 7f540bfff640 1 -- 172.21.15.57:0/3131434690 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f540c06ac70 msgr2=0x7f540c152aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:30.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.289+0000 7f54108d5640 1 --2- 172.21.15.57:0/3131434690 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f540c06ac70 0x7f540c152aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:30.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.289+0000 7f54110d6640 1 --2- 172.21.15.57:0/3131434690 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f540c15c1f0 0x7f540c1482c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:30.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.289+0000 7f540bfff640 1 --2- 172.21.15.57:0/3131434690 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f540c06ac70 0x7f540c152aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:30.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.289+0000 7f540bfff640 1 -- 172.21.15.57:0/3131434690 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f540c15c1f0 msgr2=0x7f540c1482c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:30.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.289+0000 7f540bfff640 1 --2- 172.21.15.57:0/3131434690 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f540c15c1f0 0x7f540c1482c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:30.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.289+0000 7f540bfff640 1 -- 172.21.15.57:0/3131434690 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5400009560 con 0x7f540c1579f0 2023-10-19T13:00:30.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.289+0000 7f540bfff640 1 --2- 172.21.15.57:0/3131434690 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f540c1579f0 0x7f540c147d80 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f53f800d6d0 tx=0x7f53f800db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:30.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.289+0000 7f54097fa640 1 -- 172.21.15.57:0/3131434690 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53f8004220 con 0x7f540c1579f0 2023-10-19T13:00:30.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.289+0000 7f54097fa640 1 -- 172.21.15.57:0/3131434690 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f53f8004470 con 0x7f540c1579f0 2023-10-19T13:00:30.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.289+0000 7f5412b39640 1 -- 172.21.15.57:0/3131434690 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f540c0b2330 con 0x7f540c1579f0 2023-10-19T13:00:30.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.289+0000 7f54097fa640 1 -- 172.21.15.57:0/3131434690 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53f8009a80 con 0x7f540c1579f0 2023-10-19T13:00:30.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.289+0000 7f5412b39640 1 -- 172.21.15.57:0/3131434690 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f540c152fe0 con 0x7f540c1579f0 2023-10-19T13:00:30.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.291+0000 7f54097fa640 1 -- 172.21.15.57:0/3131434690 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 32) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f53f80035f0 con 0x7f540c1579f0 2023-10-19T13:00:30.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.291+0000 7f54097fa640 1 --2- 172.21.15.57:0/3131434690 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f53f006af90 0x7f53f006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:30.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.291+0000 7f54097fa640 1 -- 172.21.15.57:0/3131434690 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f53f8089cc0 con 0x7f540c1579f0 2023-10-19T13:00:30.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.292+0000 7f54108d5640 1 --2- 172.21.15.57:0/3131434690 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f53f006af90 0x7f53f006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:30.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.292+0000 7f54108d5640 1 --2- 172.21.15.57:0/3131434690 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f53f006af90 0x7f53f006d450 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5400035000 tx=0x7f5400004730 comp rx=0 tx=0).ready entity=mgr.6256 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:30.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.292+0000 7f54157a1740 1 Processor -- start 2023-10-19T13:00:30.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.292+0000 7f54157a1740 1 -- start start 2023-10-19T13:00:30.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.293+0000 7f54157a1740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562180766b40 0x562180739ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:30.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.293+0000 7f54157a1740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562180766f20 0x5621807698d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:30.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.293+0000 7f54157a1740 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x56218073a020 0x562180769e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:30.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.293+0000 7f54157a1740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x56218064b730 con 0x562180766b40 2023-10-19T13:00:30.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.293+0000 7f54157a1740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621806ac780 con 0x562180766f20 2023-10-19T13:00:30.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.293+0000 7f54157a1740 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x5621806a9e70 con 0x56218073a020 2023-10-19T13:00:30.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.293+0000 7f54108d5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562180766b40 0x562180739ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:30.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.293+0000 7f54108d5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562180766b40 0x562180739ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56404/0 (socket says 172.21.15.57:56404) 2023-10-19T13:00:30.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.293+0000 7f54108d5640 1 -- 172.21.15.57:0/119812281 learned_addr learned my addr 172.21.15.57:0/119812281 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:30.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.293+0000 7f540bfff640 1 --2- 172.21.15.57:0/119812281 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562180766f20 0x5621807698d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:30.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.293+0000 7f54110d6640 1 --2- 172.21.15.57:0/119812281 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x56218073a020 0x562180769e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:30.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.293+0000 7f54108d5640 1 -- 172.21.15.57:0/119812281 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x56218073a020 msgr2=0x562180769e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T13:00:30.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.293+0000 7f54108d5640 1 --2- 172.21.15.57:0/119812281 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x56218073a020 0x562180769e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:30.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.293+0000 7f54108d5640 1 -- 172.21.15.57:0/119812281 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562180766f20 msgr2=0x5621807698d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T13:00:30.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.294+0000 7f54108d5640 1 --2- 172.21.15.57:0/119812281 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562180766f20 0x5621807698d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:30.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.294+0000 7f54108d5640 1 -- 172.21.15.57:0/119812281 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5621806aa3d0 con 0x562180766b40 2023-10-19T13:00:30.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.294+0000 7f54108d5640 1 --2- 172.21.15.57:0/119812281 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562180766b40 0x562180739ae0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f5400009230 tx=0x7f540002e050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=287b694bda8ebf25 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:30.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.294+0000 7f53f67fc640 1 -- 172.21.15.57:0/119812281 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5400047070 con 0x562180766b40 2023-10-19T13:00:30.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.294+0000 7f53f67fc640 1 -- 172.21.15.57:0/119812281 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f540002cc10 con 0x562180766b40 2023-10-19T13:00:30.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.294+0000 7f53f67fc640 1 -- 172.21.15.57:0/119812281 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5400004520 con 0x562180766b40 2023-10-19T13:00:30.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.294+0000 7f54157a1740 1 -- 172.21.15.57:0/119812281 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562180766b40 msgr2=0x562180739ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:30.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.294+0000 7f54157a1740 1 --2- 172.21.15.57:0/119812281 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562180766b40 0x562180739ae0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f5400009230 tx=0x7f540002e050 comp rx=0 tx=0).stop 2023-10-19T13:00:30.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.294+0000 7f54157a1740 1 -- 172.21.15.57:0/119812281 shutdown_connections 2023-10-19T13:00:30.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.294+0000 7f54157a1740 1 --2- 172.21.15.57:0/119812281 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x56218073a020 0x562180769e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:30.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.294+0000 7f54157a1740 1 --2- 172.21.15.57:0/119812281 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562180766f20 0x5621807698d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:30.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.294+0000 7f54157a1740 1 --2- 172.21.15.57:0/119812281 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562180766b40 0x562180739ae0 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:30.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.294+0000 7f54157a1740 1 -- 172.21.15.57:0/119812281 >> 172.21.15.57:0/119812281 conn(0x56218073bd50 msgr2=0x5621807848d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:30.312 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.295+0000 7f54157a1740 1 -- 172.21.15.57:0/119812281 shutdown_connections 2023-10-19T13:00:30.312 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.295+0000 7f54157a1740 1 -- 172.21.15.57:0/119812281 wait complete. 2023-10-19T13:00:30.312 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.295+0000 7f54157a1740 1 -- start start 2023-10-19T13:00:30.312 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.295+0000 7f54157a1740 1 Processor -- start 2023-10-19T13:00:30.312 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.296+0000 7f54157a1740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562180766b40 0x562180782450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:30.313 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.296+0000 7f54157a1740 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x562180766f20 0x562180786690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:30.313 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.296+0000 7f54157a1740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562180786bd0 0x562180754450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:30.313 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.296+0000 7f54157a1740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621806a5760 con 0x562180786bd0 2023-10-19T13:00:30.313 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.296+0000 7f54157a1740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621806608f0 con 0x562180766b40 2023-10-19T13:00:30.313 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.296+0000 7f54157a1740 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x562180660590 con 0x562180766f20 2023-10-19T13:00:30.314 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.296+0000 7f54157a1740 10 client.0 _ll_register_callbacks cb 0x56218069cee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-19T13:00:30.314 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.296+0000 7f54110d6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562180786bd0 0x562180754450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:30.314 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.296+0000 7f54110d6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562180786bd0 0x562180754450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56416/0 (socket says 172.21.15.57:56416) 2023-10-19T13:00:30.314 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.296+0000 7f54110d6640 1 -- 172.21.15.57:0/3552571305 learned_addr learned my addr 172.21.15.57:0/3552571305 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:30.314 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.296+0000 7f540bfff640 1 --2- 172.21.15.57:0/3552571305 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x562180766f20 0x562180786690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:30.315 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.296+0000 7f54108d5640 1 --2- 172.21.15.57:0/3552571305 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562180766b40 0x562180782450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:30.315 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.296+0000 7f54110d6640 1 -- 172.21.15.57:0/3552571305 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x562180766f20 msgr2=0x562180786690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T13:00:30.315 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.296+0000 7f54110d6640 1 --2- 172.21.15.57:0/3552571305 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x562180766f20 0x562180786690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:30.315 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.296+0000 7f54110d6640 1 -- 172.21.15.57:0/3552571305 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562180766b40 msgr2=0x562180782450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T13:00:30.315 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.296+0000 7f54110d6640 1 --2- 172.21.15.57:0/3552571305 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562180766b40 0x562180782450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:30.315 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.296+0000 7f54110d6640 1 -- 172.21.15.57:0/3552571305 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f540c0b13d0 con 0x562180786bd0 2023-10-19T13:00:30.316 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.297+0000 7f53eb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2023-10-19T13:00:30.316 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.297+0000 7f54110d6640 1 --2- 172.21.15.57:0/3552571305 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562180786bd0 0x562180754450 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f540400f410 tx=0x7f540400f810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=680acf739b49c4da server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:30.316 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.297+0000 7f53eb7fe640 1 -- 172.21.15.57:0/3552571305 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5404011850 con 0x562180786bd0 2023-10-19T13:00:30.316 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.297+0000 7f53eb7fe640 1 -- 172.21.15.57:0/3552571305 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f5404015070 con 0x562180786bd0 2023-10-19T13:00:30.316 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.297+0000 7f54157a1740 1 -- 172.21.15.57:0/3552571305 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56218069e740 con 0x562180786bd0 2023-10-19T13:00:30.317 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.297+0000 7f53eb7fe640 1 -- 172.21.15.57:0/3552571305 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f54040119b0 con 0x562180786bd0 2023-10-19T13:00:30.317 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.297+0000 7f54157a1740 10 client.7218 fetch_fsmap learned FSMap version 30 2023-10-19T13:00:30.317 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.297+0000 7f54157a1740 1 -- 172.21.15.57:0/3552571305 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=2+}) v3 -- 0x5621806a3de0 con 0x562180786bd0 2023-10-19T13:00:30.317 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.297+0000 7f53eb7fe640 1 -- 172.21.15.57:0/3552571305 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x56218069e740 con 0x562180786bd0 2023-10-19T13:00:30.317 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.297+0000 7f53eb7fe640 1 -- 172.21.15.57:0/3552571305 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 30) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f5404011c00 con 0x562180786bd0 2023-10-19T13:00:30.318 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.297+0000 7f54157a1740 10 client.7218 fetch_fsmap finished waiting for FSMap version 30 2023-10-19T13:00:30.318 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.297+0000 7f54157a1740 10 client.7218 resolve_mds: resolved name 'a' to daemon mds.a 2023-10-19T13:00:30.318 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.297+0000 7f54157a1740 20 client.7218 populate_metadata read hostname 'smithi057' 2023-10-19T13:00:30.318 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.297+0000 7f54157a1740 1 --2- 172.21.15.57:0/3552571305 >> [v2:172.21.15.57:6833/2257211800,v1:172.21.15.57:6835/2257211800] conn(0x562180784640 0x562180784a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:30.318 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.297+0000 7f54157a1740 4 client.7218 mds_command: new command op to 7038 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-19T13:00:30.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.297+0000 7f54157a1740 1 -- 172.21.15.57:0/3552571305 --> [v2:172.21.15.57:6833/2257211800,v1:172.21.15.57:6835/2257211800] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56218069e460 con 0x562180784640 2023-10-19T13:00:30.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.297+0000 7f54108d5640 1 --2- 172.21.15.57:0/3552571305 >> [v2:172.21.15.57:6833/2257211800,v1:172.21.15.57:6835/2257211800] conn(0x562180784640 0x562180784a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:30.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.298+0000 7f54108d5640 1 --2- 172.21.15.57:0/3552571305 >> [v2:172.21.15.57:6833/2257211800,v1:172.21.15.57:6835/2257211800] conn(0x562180784640 0x562180784a20 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=56bd92c494e859df server_cookie=c0f205d71ef60b8d in_seq=0 out_seq=0 2023-10-19T13:00:30.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.298+0000 7f53eb7fe640 10 client.7218 ms_handle_connect on v2:172.21.15.57:6833/2257211800 2023-10-19T13:00:30.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.298+0000 7f53eb7fe640 1 -- 172.21.15.57:0/3552571305 <== mds.0 v2:172.21.15.57:6833/2257211800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x56218069e460 con 0x562180784640 2023-10-19T13:00:30.320 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.298+0000 7f53eb7fe640 10 client.7218 handle_command_reply: tid=0 2023-10-19T13:00:30.320 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.298+0000 7f54157a1740 1 client.7218 shutdown 2023-10-19T13:00:30.320 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.298+0000 7f54157a1740 1 -- 172.21.15.57:0/3552571305 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562180786bd0 msgr2=0x562180754450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:30.320 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.299+0000 7f54157a1740 1 --2- 172.21.15.57:0/3552571305 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562180786bd0 0x562180754450 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f540400f410 tx=0x7f540400f810 comp rx=0 tx=0).stop 2023-10-19T13:00:30.320 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.299+0000 7f54157a1740 1 -- 172.21.15.57:0/3552571305 shutdown_connections 2023-10-19T13:00:30.321 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.299+0000 7f54157a1740 1 --2- 172.21.15.57:0/3552571305 >> [v2:172.21.15.57:6833/2257211800,v1:172.21.15.57:6835/2257211800] conn(0x562180784640 0x562180784a20 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:30.321 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.299+0000 7f54157a1740 1 --2- 172.21.15.57:0/3552571305 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562180786bd0 0x562180754450 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:30.321 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.299+0000 7f54157a1740 1 --2- 172.21.15.57:0/3552571305 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x562180766f20 0x562180786690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:30.321 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.299+0000 7f53eb7fe640 0 client.7218 ms_handle_reset on v2:172.21.15.57:6833/2257211800 2023-10-19T13:00:30.321 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.299+0000 7f54157a1740 1 --2- 172.21.15.57:0/3552571305 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562180766b40 0x562180782450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:30.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.299+0000 7f54157a1740 1 -- 172.21.15.57:0/3552571305 >> 172.21.15.57:0/3552571305 conn(0x56218073a020 msgr2=0x562180785b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:30.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.299+0000 7f54157a1740 1 -- 172.21.15.57:0/3552571305 shutdown_connections 2023-10-19T13:00:30.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.299+0000 7f54157a1740 1 -- 172.21.15.57:0/3552571305 wait complete. 2023-10-19T13:00:30.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.299+0000 7f54157a1740 20 client.7218 trim_cache size 0 max 16384 2023-10-19T13:00:30.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.314+0000 7f54157a1740 1 Processor -- start 2023-10-19T13:00:30.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.314+0000 7f54157a1740 1 -- start start 2023-10-19T13:00:30.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.315+0000 7f54157a1740 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x562180766b40 0x5621807551d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:30.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.315+0000 7f54157a1740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562180766f20 0x562180755710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:30.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.315+0000 7f54157a1740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562180786bd0 0x562180769310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:30.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.315+0000 7f54157a1740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x562180626270 con 0x562180786bd0 2023-10-19T13:00:30.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.315+0000 7f54157a1740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621807392e0 con 0x562180766f20 2023-10-19T13:00:30.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.315+0000 7f54157a1740 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x56218073fdc0 con 0x562180766b40 2023-10-19T13:00:30.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.315+0000 7f54110d6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562180786bd0 0x562180769310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:30.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.315+0000 7f54110d6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562180786bd0 0x562180769310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56432/0 (socket says 172.21.15.57:56432) 2023-10-19T13:00:30.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.315+0000 7f54110d6640 1 -- 172.21.15.57:0/528533105 learned_addr learned my addr 172.21.15.57:0/528533105 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:30.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.315+0000 7f54108d5640 1 --2- 172.21.15.57:0/528533105 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x562180766b40 0x5621807551d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:30.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.315+0000 7f540bfff640 1 --2- 172.21.15.57:0/528533105 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562180766f20 0x562180755710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:30.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.315+0000 7f54110d6640 1 -- 172.21.15.57:0/528533105 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x562180766b40 msgr2=0x5621807551d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T13:00:30.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.315+0000 7f54110d6640 1 --2- 172.21.15.57:0/528533105 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x562180766b40 0x5621807551d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:30.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.315+0000 7f54110d6640 1 -- 172.21.15.57:0/528533105 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562180766f20 msgr2=0x562180755710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T13:00:30.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.315+0000 7f54110d6640 1 --2- 172.21.15.57:0/528533105 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562180766f20 0x562180755710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:30.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.315+0000 7f54110d6640 1 -- 172.21.15.57:0/528533105 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562180642d00 con 0x562180786bd0 2023-10-19T13:00:30.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.315+0000 7f54110d6640 1 --2- 172.21.15.57:0/528533105 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562180786bd0 0x562180769310 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f5404009640 tx=0x7f5404004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b346be9306218be server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:30.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.315+0000 7f53f6ffd640 1 -- 172.21.15.57:0/528533105 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f540401a4d0 con 0x562180786bd0 2023-10-19T13:00:30.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.315+0000 7f53f6ffd640 1 -- 172.21.15.57:0/528533105 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f540400fd10 con 0x562180786bd0 2023-10-19T13:00:30.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.315+0000 7f53f6ffd640 1 -- 172.21.15.57:0/528533105 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54040118b0 con 0x562180786bd0 2023-10-19T13:00:30.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.316+0000 7f54157a1740 1 -- 172.21.15.57:0/528533105 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562180786bd0 msgr2=0x562180769310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:30.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.316+0000 7f54157a1740 1 --2- 172.21.15.57:0/528533105 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562180786bd0 0x562180769310 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f5404009640 tx=0x7f5404004ab0 comp rx=0 tx=0).stop 2023-10-19T13:00:30.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.316+0000 7f54157a1740 1 -- 172.21.15.57:0/528533105 shutdown_connections 2023-10-19T13:00:30.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.316+0000 7f54157a1740 1 --2- 172.21.15.57:0/528533105 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562180786bd0 0x562180769310 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:30.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.316+0000 7f54157a1740 1 --2- 172.21.15.57:0/528533105 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562180766f20 0x562180755710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:30.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.316+0000 7f54157a1740 1 --2- 172.21.15.57:0/528533105 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x562180766b40 0x5621807551d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:30.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.316+0000 7f54157a1740 1 -- 172.21.15.57:0/528533105 >> 172.21.15.57:0/528533105 conn(0x56218073a020 msgr2=0x562180784a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:30.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.316+0000 7f54157a1740 1 -- 172.21.15.57:0/528533105 shutdown_connections 2023-10-19T13:00:30.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.316+0000 7f54157a1740 1 -- 172.21.15.57:0/528533105 wait complete. 2023-10-19T13:00:30.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.316+0000 7f54157a1740 1 -- start start 2023-10-19T13:00:30.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.317+0000 7f54157a1740 1 Processor -- start 2023-10-19T13:00:30.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.317+0000 7f54157a1740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562180766b40 0x5621807e3c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:30.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.317+0000 7f54157a1740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562180766f20 0x5621807e4190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:30.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.317+0000 7f54157a1740 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x562180786bd0 0x562180759fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:30.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.317+0000 7f54157a1740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x56218076cec0 con 0x562180766b40 2023-10-19T13:00:30.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.317+0000 7f54157a1740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x56218073f470 con 0x562180766f20 2023-10-19T13:00:30.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.318+0000 7f54157a1740 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x562180769830 con 0x562180786bd0 2023-10-19T13:00:30.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.318+0000 7f54157a1740 10 client.0 _ll_register_callbacks cb 0x7f5400009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-19T13:00:30.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.318+0000 7f54108d5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562180766b40 0x5621807e3c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:30.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.318+0000 7f54108d5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562180766b40 0x5621807e3c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56434/0 (socket says 172.21.15.57:56434) 2023-10-19T13:00:30.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.318+0000 7f54108d5640 1 -- 172.21.15.57:0/3828121078 learned_addr learned my addr 172.21.15.57:0/3828121078 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:30.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.318+0000 7f54110d6640 1 --2- 172.21.15.57:0/3828121078 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x562180786bd0 0x562180759fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:30.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.318+0000 7f540bfff640 1 --2- 172.21.15.57:0/3828121078 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562180766f20 0x5621807e4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:30.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.318+0000 7f54108d5640 1 -- 172.21.15.57:0/3828121078 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x562180786bd0 msgr2=0x562180759fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T13:00:30.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.318+0000 7f54108d5640 1 --2- 172.21.15.57:0/3828121078 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x562180786bd0 0x562180759fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:30.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.318+0000 7f54108d5640 1 -- 172.21.15.57:0/3828121078 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562180766f20 msgr2=0x5621807e4190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T13:00:30.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.318+0000 7f54108d5640 1 --2- 172.21.15.57:0/3828121078 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562180766f20 0x5621807e4190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:30.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.318+0000 7f54108d5640 1 -- 172.21.15.57:0/3828121078 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5621806a3de0 con 0x562180766b40 2023-10-19T13:00:30.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.318+0000 7f540bfff640 1 --2- 172.21.15.57:0/3828121078 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562180766f20 0x5621807e4190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T13:00:30.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.318+0000 7f54108d5640 1 --2- 172.21.15.57:0/3828121078 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562180766b40 0x5621807e3c50 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f54000099c0 tx=0x7f5400056b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cbcd1518db2edd7 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:30.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.318+0000 7f53eb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2023-10-19T13:00:30.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.318+0000 7f53eb7fe640 1 -- 172.21.15.57:0/3828121078 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5400047040 con 0x562180766b40 2023-10-19T13:00:30.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.318+0000 7f53eb7fe640 1 -- 172.21.15.57:0/3828121078 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f5400056e40 con 0x562180766b40 2023-10-19T13:00:30.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.318+0000 7f54157a1740 1 -- 172.21.15.57:0/3828121078 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f540002cc10 con 0x562180766b40 2023-10-19T13:00:30.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.318+0000 7f53eb7fe640 1 -- 172.21.15.57:0/3828121078 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5400056110 con 0x562180766b40 2023-10-19T13:00:30.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.318+0000 7f54157a1740 10 client.7224 fetch_fsmap learned FSMap version 30 2023-10-19T13:00:30.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.318+0000 7f54157a1740 1 -- 172.21.15.57:0/3828121078 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=2+}) v3 -- 0x5621806684f0 con 0x562180766b40 2023-10-19T13:00:30.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.319+0000 7f53eb7fe640 1 -- 172.21.15.57:0/3828121078 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f540002cc10 con 0x562180766b40 2023-10-19T13:00:30.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.319+0000 7f53eb7fe640 1 -- 172.21.15.57:0/3828121078 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 30) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f540003c040 con 0x562180766b40 2023-10-19T13:00:30.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.319+0000 7f54157a1740 10 client.7224 fetch_fsmap finished waiting for FSMap version 30 2023-10-19T13:00:30.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.319+0000 7f54157a1740 10 client.7224 resolve_mds: resolved name 'a' to daemon mds.a 2023-10-19T13:00:30.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.319+0000 7f54157a1740 20 client.7224 populate_metadata read hostname 'smithi057' 2023-10-19T13:00:30.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.319+0000 7f54157a1740 1 --2- 172.21.15.57:0/3828121078 >> [v2:172.21.15.57:6833/2257211800,v1:172.21.15.57:6835/2257211800] conn(0x5621807dc810 0x56218075a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:30.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.319+0000 7f54157a1740 4 client.7224 mds_command: new command op to 7038 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-19T13:00:30.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.319+0000 7f54157a1740 1 -- 172.21.15.57:0/3828121078 --> [v2:172.21.15.57:6833/2257211800,v1:172.21.15.57:6835/2257211800] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5621803a76c0 con 0x5621807dc810 2023-10-19T13:00:30.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.319+0000 7f540bfff640 1 --2- 172.21.15.57:0/3828121078 >> [v2:172.21.15.57:6833/2257211800,v1:172.21.15.57:6835/2257211800] conn(0x5621807dc810 0x56218075a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:30.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.319+0000 7f53eb7fe640 10 client.7224 ms_handle_connect on v2:172.21.15.57:6833/2257211800 2023-10-19T13:00:30.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.319+0000 7f540bfff640 1 --2- 172.21.15.57:0/3828121078 >> [v2:172.21.15.57:6833/2257211800,v1:172.21.15.57:6835/2257211800] conn(0x5621807dc810 0x56218075a4f0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e49fff3075d9522b server_cookie=72fec2169af75c34 in_seq=0 out_seq=0 2023-10-19T13:00:30.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.319+0000 7f53eb7fe640 1 -- 172.21.15.57:0/3828121078 <== mds.0 v2:172.21.15.57:6833/2257211800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x5621803a76c0 con 0x5621807dc810 2023-10-19T13:00:30.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.319+0000 7f53eb7fe640 10 client.7224 handle_command_reply: tid=0 2023-10-19T13:00:30.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.319+0000 7f54157a1740 1 client.7224 shutdown 2023-10-19T13:00:30.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.320+0000 7f54157a1740 1 -- 172.21.15.57:0/3828121078 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562180766b40 msgr2=0x5621807e3c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:30.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.320+0000 7f54157a1740 1 --2- 172.21.15.57:0/3828121078 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562180766b40 0x5621807e3c50 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f54000099c0 tx=0x7f5400056b40 comp rx=0 tx=0).stop 2023-10-19T13:00:30.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.320+0000 7f54157a1740 1 -- 172.21.15.57:0/3828121078 shutdown_connections 2023-10-19T13:00:30.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.320+0000 7f54157a1740 1 --2- 172.21.15.57:0/3828121078 >> [v2:172.21.15.57:6833/2257211800,v1:172.21.15.57:6835/2257211800] conn(0x5621807dc810 0x56218075a4f0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:30.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.320+0000 7f54157a1740 1 --2- 172.21.15.57:0/3828121078 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x562180786bd0 0x562180759fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:30.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.320+0000 7f54157a1740 1 --2- 172.21.15.57:0/3828121078 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562180766f20 0x5621807e4190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:30.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.320+0000 7f53eb7fe640 0 client.7224 ms_handle_reset on v2:172.21.15.57:6833/2257211800 2023-10-19T13:00:30.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.320+0000 7f54157a1740 1 --2- 172.21.15.57:0/3828121078 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562180766b40 0x5621807e3c50 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:30.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.320+0000 7f54157a1740 1 -- 172.21.15.57:0/3828121078 >> 172.21.15.57:0/3828121078 conn(0x56218073a020 msgr2=0x562180787de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:30.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.320+0000 7f54157a1740 1 -- 172.21.15.57:0/3828121078 shutdown_connections 2023-10-19T13:00:30.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.320+0000 7f54157a1740 1 -- 172.21.15.57:0/3828121078 wait complete. 2023-10-19T13:00:30.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.320+0000 7f54157a1740 20 client.7224 trim_cache size 0 max 16384 2023-10-19T13:00:30.345 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2023-10-19T13:00:30.345 INFO:teuthology.orchestra.run.smithi057.stdout: "cluster_fsid": "1e9beb19-3260-4dd7-85ef-5b399ce774ac", 2023-10-19T13:00:30.345 INFO:teuthology.orchestra.run.smithi057.stdout: "whoami": 0, 2023-10-19T13:00:30.345 INFO:teuthology.orchestra.run.smithi057.stdout: "id": 7038, 2023-10-19T13:00:30.345 INFO:teuthology.orchestra.run.smithi057.stdout: "want_state": "up:active", 2023-10-19T13:00:30.346 INFO:teuthology.orchestra.run.smithi057.stdout: "state": "up:active", 2023-10-19T13:00:30.346 INFO:teuthology.orchestra.run.smithi057.stdout: "fs_name": "cephfs", 2023-10-19T13:00:30.346 INFO:teuthology.orchestra.run.smithi057.stdout: "rank_uptime": 9.1097930040000001, 2023-10-19T13:00:30.346 INFO:teuthology.orchestra.run.smithi057.stdout: "mdsmap_epoch": 30, 2023-10-19T13:00:30.346 INFO:teuthology.orchestra.run.smithi057.stdout: "osdmap_epoch": 87, 2023-10-19T13:00:30.347 INFO:teuthology.orchestra.run.smithi057.stdout: "osdmap_epoch_barrier": 87, 2023-10-19T13:00:30.347 INFO:teuthology.orchestra.run.smithi057.stdout: "uptime": 14.292259205000001 2023-10-19T13:00:30.347 INFO:teuthology.orchestra.run.smithi057.stdout:} 2023-10-19T13:00:30.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.320+0000 7f5412b39640 1 -- 172.21.15.57:0/3131434690 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f53f006af90 msgr2=0x7f53f006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:30.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.320+0000 7f5412b39640 1 --2- 172.21.15.57:0/3131434690 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f53f006af90 0x7f53f006d450 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5400035000 tx=0x7f5400004730 comp rx=0 tx=0).stop 2023-10-19T13:00:30.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.320+0000 7f5412b39640 1 -- 172.21.15.57:0/3131434690 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f540c1579f0 msgr2=0x7f540c147d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:30.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.320+0000 7f5412b39640 1 --2- 172.21.15.57:0/3131434690 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f540c1579f0 0x7f540c147d80 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f53f800d6d0 tx=0x7f53f800db90 comp rx=0 tx=0).stop 2023-10-19T13:00:30.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.320+0000 7f5412b39640 1 -- 172.21.15.57:0/3131434690 shutdown_connections 2023-10-19T13:00:30.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.320+0000 7f5412b39640 1 --2- 172.21.15.57:0/3131434690 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f53f006af90 0x7f53f006d450 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:30.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.320+0000 7f5412b39640 1 --2- 172.21.15.57:0/3131434690 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f540c15c1f0 0x7f540c1482c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:30.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.320+0000 7f5412b39640 1 --2- 172.21.15.57:0/3131434690 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f540c1579f0 0x7f540c147d80 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:30.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.320+0000 7f5412b39640 1 --2- 172.21.15.57:0/3131434690 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f540c06ac70 0x7f540c152aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:30.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.320+0000 7f5412b39640 1 -- 172.21.15.57:0/3131434690 >> 172.21.15.57:0/3131434690 conn(0x7f540c06fae0 msgr2=0x7f540c153610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:30.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.321+0000 7f5412b39640 1 -- 172.21.15.57:0/3131434690 shutdown_connections 2023-10-19T13:00:30.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.321+0000 7f5412b39640 1 -- 172.21.15.57:0/3131434690 wait complete. 2023-10-19T13:00:30.351 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-19T13:00:30.462 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.459+0000 7f30260e5640 1 -- 172.21.15.57:0/677046311 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f30080047c0 msgr2=0x7f3008004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:30.462 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.459+0000 7f30260e5640 1 --2- 172.21.15.57:0/677046311 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f30080047c0 0x7f3008004c40 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f3014001680 tx=0x7f30080057f0 comp rx=0 tx=0).stop 2023-10-19T13:00:30.462 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.459+0000 7f30260e5640 1 -- 172.21.15.57:0/677046311 shutdown_connections 2023-10-19T13:00:30.462 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.459+0000 7f30260e5640 1 --2- 172.21.15.57:0/677046311 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f30080047c0 0x7f3008004c40 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:30.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.459+0000 7f30260e5640 1 -- 172.21.15.57:0/677046311 >> 172.21.15.57:0/677046311 conn(0x7f3020069930 msgr2=0x7f3020069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:30.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.459+0000 7f30260e5640 1 -- 172.21.15.57:0/677046311 shutdown_connections 2023-10-19T13:00:30.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.459+0000 7f30260e5640 1 -- 172.21.15.57:0/677046311 wait complete. 2023-10-19T13:00:30.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.460+0000 7f30260e5640 1 Processor -- start 2023-10-19T13:00:30.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.460+0000 7f30260e5640 1 -- start start 2023-10-19T13:00:30.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.461+0000 7f30260e5640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f30080047c0 0x7f302013a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:30.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.461+0000 7f30260e5640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f30200a7270 0x7f30200b0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:30.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.461+0000 7f30260e5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f30200b0ba0 0x7f30200aa600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:30.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.461+0000 7f30260e5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30200b1020 con 0x7f30200b0ba0 2023-10-19T13:00:30.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.461+0000 7f30260e5640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30200ab330 con 0x7f30080047c0 2023-10-19T13:00:30.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.461+0000 7f30260e5640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30200a64b0 con 0x7f30200a7270 2023-10-19T13:00:30.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.461+0000 7f301ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f30200b0ba0 0x7f30200aa600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:30.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.461+0000 7f301ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f30200b0ba0 0x7f30200aa600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56452/0 (socket says 172.21.15.57:56452) 2023-10-19T13:00:30.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.461+0000 7f301ffff640 1 -- 172.21.15.57:0/2971898295 learned_addr learned my addr 172.21.15.57:0/2971898295 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:30.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.461+0000 7f301f7fe640 1 --2- 172.21.15.57:0/2971898295 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f30080047c0 0x7f302013a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:30.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.461+0000 7f301effd640 1 --2- 172.21.15.57:0/2971898295 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f30200a7270 0x7f30200b0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:30.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.461+0000 7f301ffff640 1 -- 172.21.15.57:0/2971898295 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f30200a7270 msgr2=0x7f30200b0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:30.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.461+0000 7f301ffff640 1 --2- 172.21.15.57:0/2971898295 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f30200a7270 0x7f30200b0660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:30.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.461+0000 7f301ffff640 1 -- 172.21.15.57:0/2971898295 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f30080047c0 msgr2=0x7f302013a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:30.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.461+0000 7f301ffff640 1 --2- 172.21.15.57:0/2971898295 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f30080047c0 0x7f302013a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:30.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.461+0000 7f301ffff640 1 -- 172.21.15.57:0/2971898295 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30200a6790 con 0x7f30200b0ba0 2023-10-19T13:00:30.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.461+0000 7f301ffff640 1 --2- 172.21.15.57:0/2971898295 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f30200b0ba0 0x7f30200aa600 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f30200cc430 tx=0x7f3018010f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:30.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.461+0000 7f3007fff640 1 -- 172.21.15.57:0/2971898295 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f301800ee10 con 0x7f30200b0ba0 2023-10-19T13:00:30.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.462+0000 7f3007fff640 1 -- 172.21.15.57:0/2971898295 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f3018003a60 con 0x7f30200b0ba0 2023-10-19T13:00:30.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.462+0000 7f3007fff640 1 -- 172.21.15.57:0/2971898295 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3018003e70 con 0x7f30200b0ba0 2023-10-19T13:00:30.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.462+0000 7f30260e5640 1 -- 172.21.15.57:0/2971898295 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30200a6a70 con 0x7f30200b0ba0 2023-10-19T13:00:30.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.462+0000 7f30260e5640 1 -- 172.21.15.57:0/2971898295 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f302013a640 con 0x7f30200b0ba0 2023-10-19T13:00:30.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.462+0000 7f30260e5640 1 -- 172.21.15.57:0/2971898295 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30200a6f90 con 0x7f30200b0ba0 2023-10-19T13:00:30.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.463+0000 7f3007fff640 1 -- 172.21.15.57:0/2971898295 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 32) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f3018019030 con 0x7f30200b0ba0 2023-10-19T13:00:30.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.463+0000 7f3007fff640 1 --2- 172.21.15.57:0/2971898295 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f300c06af60 0x7f300c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:30.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.463+0000 7f3007fff640 1 -- 172.21.15.57:0/2971898295 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f3018017070 con 0x7f30200b0ba0 2023-10-19T13:00:30.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.464+0000 7f301f7fe640 1 --2- 172.21.15.57:0/2971898295 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f300c06af60 0x7f300c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:30.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.464+0000 7f301f7fe640 1 --2- 172.21.15.57:0/2971898295 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f300c06af60 0x7f300c06d420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3010001390 tx=0x7f3010000fa0 comp rx=0 tx=0).ready entity=mgr.6256 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:30.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.467+0000 7f3007fff640 1 -- 172.21.15.57:0/2971898295 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f30200a6f90 con 0x7f30200b0ba0 2023-10-19T13:00:30.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.679+0000 7f30260e5640 1 -- 172.21.15.57:0/2971898295 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f30200a6f90 con 0x7f30200b0ba0 2023-10-19T13:00:30.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.679+0000 7f3007fff640 1 -- 172.21.15.57:0/2971898295 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v30) v1 ==== 53+0+73 (secure 0 0 0) 0x7f30200a6f90 con 0x7f30200b0ba0 2023-10-19T13:00:30.682 INFO:teuthology.orchestra.run.smithi057.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-19T13:00:30.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.681+0000 7f30260e5640 1 -- 172.21.15.57:0/2971898295 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f300c06af60 msgr2=0x7f300c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:30.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.681+0000 7f30260e5640 1 --2- 172.21.15.57:0/2971898295 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f300c06af60 0x7f300c06d420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3010001390 tx=0x7f3010000fa0 comp rx=0 tx=0).stop 2023-10-19T13:00:30.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.681+0000 7f30260e5640 1 -- 172.21.15.57:0/2971898295 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f30200b0ba0 msgr2=0x7f30200aa600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:30.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.681+0000 7f30260e5640 1 --2- 172.21.15.57:0/2971898295 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f30200b0ba0 0x7f30200aa600 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f30200cc430 tx=0x7f3018010f40 comp rx=0 tx=0).stop 2023-10-19T13:00:30.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.681+0000 7f30260e5640 1 -- 172.21.15.57:0/2971898295 shutdown_connections 2023-10-19T13:00:30.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.681+0000 7f30260e5640 1 --2- 172.21.15.57:0/2971898295 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f300c06af60 0x7f300c06d420 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:30.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.681+0000 7f30260e5640 1 --2- 172.21.15.57:0/2971898295 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f30200b0ba0 0x7f30200aa600 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:30.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.681+0000 7f30260e5640 1 --2- 172.21.15.57:0/2971898295 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f30200a7270 0x7f30200b0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:30.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.681+0000 7f30260e5640 1 --2- 172.21.15.57:0/2971898295 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f30080047c0 0x7f302013a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:30.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.681+0000 7f30260e5640 1 -- 172.21.15.57:0/2971898295 >> 172.21.15.57:0/2971898295 conn(0x7f3020069930 msgr2=0x7f302013b1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:30.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.681+0000 7f30260e5640 1 -- 172.21.15.57:0/2971898295 shutdown_connections 2023-10-19T13:00:30.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:30.681+0000 7f30260e5640 1 -- 172.21.15.57:0/2971898295 wait complete. 2023-10-19T13:00:30.694 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-10-19T13:00:30.695 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-19T13:00:30.695 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi057.front.sepia.ceph.com 2023-10-19T13:00:30.695 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-19T13:00:30.696 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-19T13:00:30.696 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-10-19T13:00:30.696 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-10-19T13:00:30.697 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-19T13:00:30.697 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-19T13:00:30.697 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:00:30.698 DEBUG:teuthology.orchestra.run.smithi057:> ip addr 2023-10-19T13:00:30.714 INFO:teuthology.orchestra.run.smithi057.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-19T13:00:30.714 INFO:teuthology.orchestra.run.smithi057.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-19T13:00:30.714 INFO:teuthology.orchestra.run.smithi057.stdout: inet 127.0.0.1/8 scope host lo 2023-10-19T13:00:30.714 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T13:00:30.715 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 ::1/128 scope host 2023-10-19T13:00:30.715 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T13:00:30.715 INFO:teuthology.orchestra.run.smithi057.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-19T13:00:30.715 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:6b:ff:86 brd ff:ff:ff:ff:ff:ff 2023-10-19T13:00:30.715 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp7s0f0 2023-10-19T13:00:30.716 INFO:teuthology.orchestra.run.smithi057.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-19T13:00:30.716 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:6b:ff:87 brd ff:ff:ff:ff:ff:ff 2023-10-19T13:00:30.716 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp7s0f1 2023-10-19T13:00:30.716 INFO:teuthology.orchestra.run.smithi057.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-19T13:00:30.716 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:bd:15:62 brd ff:ff:ff:ff:ff:ff 2023-10-19T13:00:30.717 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp1s0f0 2023-10-19T13:00:30.717 INFO:teuthology.orchestra.run.smithi057.stdout: inet 172.21.15.57/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-19T13:00:30.717 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft 41680sec preferred_lft 41680sec 2023-10-19T13:00:30.717 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 fe80::ec4:7aff:febd:1562/64 scope link noprefixroute 2023-10-19T13:00:30.717 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T13:00:30.717 INFO:teuthology.orchestra.run.smithi057.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-19T13:00:30.718 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:bd:15:63 brd ff:ff:ff:ff:ff:ff 2023-10-19T13:00:30.718 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp1s0f1 2023-10-19T13:00:30.718 INFO:teuthology.orchestra.run.smithi057.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-19T13:00:30.718 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0e:fe:f1:b5:41:59 brd ff:ff:ff:ff:ff:ff 2023-10-19T13:00:30.718 INFO:teuthology.orchestra.run.smithi057.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-19T13:00:30.719 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T13:00:30.719 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 fe80::64b1:a2ff:feb3:b08/64 scope link 2023-10-19T13:00:30.719 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T13:00:30.719 INFO:teuthology.orchestra.run.smithi057.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-19T13:00:30.719 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0e:fe:f1:b5:41:59 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T13:00:30.720 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 fe80::cfe:f1ff:feb5:4159/64 scope link 2023-10-19T13:00:30.720 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T13:00:30.720 INFO:teuthology.orchestra.run.smithi057.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-19T13:00:30.720 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 62:53:72:9d:52:71 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-19T13:00:30.720 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 fe80::6053:72ff:fe9d:5271/64 scope link 2023-10-19T13:00:30.720 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T13:00:30.721 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:00:30.721 DEBUG:teuthology.orchestra.run.smithi057:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-19T13:00:30.784 INFO:teuthology.orchestra.run.smithi057.stdout:1 2023-10-19T13:00:30.785 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:00:30.785 DEBUG:teuthology.orchestra.run.smithi057:> route 2023-10-19T13:00:30.842 INFO:teuthology.orchestra.run.smithi057.stdout:Kernel IP routing table 2023-10-19T13:00:30.842 INFO:teuthology.orchestra.run.smithi057.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-19T13:00:30.842 INFO:teuthology.orchestra.run.smithi057.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-19T13:00:30.843 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-19T13:00:30.843 INFO:teuthology.orchestra.run.smithi057.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-19T13:00:30.843 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:00:30.844 DEBUG:teuthology.orchestra.run.smithi057:> (cd / && exec bash -c ' 2023-10-19T13:00:30.844 DEBUG:teuthology.orchestra.run.smithi057:> set -e 2023-10-19T13:00:30.844 DEBUG:teuthology.orchestra.run.smithi057:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-19T13:00:30.845 DEBUG:teuthology.orchestra.run.smithi057:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-19T13:00:30.845 DEBUG:teuthology.orchestra.run.smithi057:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-19T13:00:30.845 DEBUG:teuthology.orchestra.run.smithi057:> ') 2023-10-19T13:00:30.948 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:00:30.948 DEBUG:teuthology.orchestra.run.smithi057:> ip netns list 2023-10-19T13:00:30.964 INFO:teuthology.orchestra.run.smithi057.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-19T13:00:30.964 INFO:teuthology.orchestra.run.smithi057.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-19T13:00:30.965 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-19T13:00:30.965 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:00:30.965 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip link set brx.0 up 2023-10-19T13:00:31.035 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-10-19T13:00:31.036 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-19T13:00:31.036 DEBUG:teuthology.orchestra.run.smithi057:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-19T13:00:31.090 INFO:teuthology.orchestra.run.smithi057.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-19T13:00:31.091 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-19T13:00:31.091 DEBUG:teuthology.orchestra.run.smithi057:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-19T13:00:31.146 DEBUG:teuthology.orchestra.run.smithi057:> sudo modprobe fuse 2023-10-19T13:00:31.217 DEBUG:teuthology.orchestra.run.smithi057:> cat /proc/self/mounts | awk '{print $2}' 2023-10-19T13:00:31.273 INFO:teuthology.orchestra.run.smithi057.stdout:/proc 2023-10-19T13:00:31.274 INFO:teuthology.orchestra.run.smithi057.stdout:/sys 2023-10-19T13:00:31.274 INFO:teuthology.orchestra.run.smithi057.stdout:/dev 2023-10-19T13:00:31.274 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/security 2023-10-19T13:00:31.274 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/shm 2023-10-19T13:00:31.275 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/pts 2023-10-19T13:00:31.275 INFO:teuthology.orchestra.run.smithi057.stdout:/run 2023-10-19T13:00:31.275 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/cgroup 2023-10-19T13:00:31.275 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/pstore 2023-10-19T13:00:31.275 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/bpf 2023-10-19T13:00:31.275 INFO:teuthology.orchestra.run.smithi057.stdout:/ 2023-10-19T13:00:31.276 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/selinux 2023-10-19T13:00:31.276 INFO:teuthology.orchestra.run.smithi057.stdout:/proc/sys/fs/binfmt_misc 2023-10-19T13:00:31.276 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/hugepages 2023-10-19T13:00:31.276 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/mqueue 2023-10-19T13:00:31.276 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/debug 2023-10-19T13:00:31.277 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/tracing 2023-10-19T13:00:31.277 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/fuse/connections 2023-10-19T13:00:31.277 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/config 2023-10-19T13:00:31.277 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-sysctl.service 2023-10-19T13:00:31.277 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-10-19T13:00:31.277 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-10-19T13:00:31.278 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/nfs/rpc_pipefs 2023-10-19T13:00:31.278 INFO:teuthology.orchestra.run.smithi057.stdout:/run/user/1000 2023-10-19T13:00:31.278 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph 2023-10-19T13:00:31.278 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-0 2023-10-19T13:00:31.278 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-1 2023-10-19T13:00:31.279 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-2 2023-10-19T13:00:31.279 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-3 2023-10-19T13:00:31.279 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns 2023-10-19T13:00:31.279 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T13:00:31.280 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T13:00:31.280 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-19T13:00:31.280 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-19T13:00:31.280 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:00:31.281 DEBUG:teuthology.orchestra.run.smithi057:> ls /sys/fs/fuse/connections 2023-10-19T13:00:31.328 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-10-19T13:00:31.329 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-10-19T13:00:31.370 DEBUG:teuthology.orchestra.run.smithi057:> sudo modprobe fuse 2023-10-19T13:00:31.401 DEBUG:teuthology.orchestra.run.smithi057:> cat /proc/self/mounts | awk '{print $2}' 2023-10-19T13:00:31.457 INFO:teuthology.orchestra.run.smithi057.stdout:/proc 2023-10-19T13:00:31.457 INFO:teuthology.orchestra.run.smithi057.stdout:/sys 2023-10-19T13:00:31.458 INFO:teuthology.orchestra.run.smithi057.stdout:/dev 2023-10-19T13:00:31.458 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/security 2023-10-19T13:00:31.458 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/shm 2023-10-19T13:00:31.458 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/pts 2023-10-19T13:00:31.458 INFO:teuthology.orchestra.run.smithi057.stdout:/run 2023-10-19T13:00:31.459 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/cgroup 2023-10-19T13:00:31.459 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/pstore 2023-10-19T13:00:31.459 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/bpf 2023-10-19T13:00:31.459 INFO:teuthology.orchestra.run.smithi057.stdout:/ 2023-10-19T13:00:31.459 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/selinux 2023-10-19T13:00:31.459 INFO:teuthology.orchestra.run.smithi057.stdout:/proc/sys/fs/binfmt_misc 2023-10-19T13:00:31.460 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/hugepages 2023-10-19T13:00:31.460 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/mqueue 2023-10-19T13:00:31.460 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/debug 2023-10-19T13:00:31.460 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/tracing 2023-10-19T13:00:31.460 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/fuse/connections 2023-10-19T13:00:31.460 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/config 2023-10-19T13:00:31.461 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-sysctl.service 2023-10-19T13:00:31.461 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-10-19T13:00:31.461 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-10-19T13:00:31.461 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/nfs/rpc_pipefs 2023-10-19T13:00:31.461 INFO:teuthology.orchestra.run.smithi057.stdout:/run/user/1000 2023-10-19T13:00:31.462 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph 2023-10-19T13:00:31.462 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-0 2023-10-19T13:00:31.462 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-1 2023-10-19T13:00:31.462 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-2 2023-10-19T13:00:31.462 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-3 2023-10-19T13:00:31.462 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns 2023-10-19T13:00:31.463 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T13:00:31.463 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T13:00:31.463 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-19T13:00:31.463 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-19T13:00:31.463 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:00:31.464 DEBUG:teuthology.orchestra.run.smithi057:> ls /sys/fs/fuse/connections 2023-10-19T13:00:31.465 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi057.stderr:2023-10-19T13:00:31.460+0000 7fd4c8c29480 -1 init, newargv = 0x55f9f5408c80 newargc=15 2023-10-19T13:00:31.465 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi057.stderr:ceph-fuse[119022]: starting ceph client 2023-10-19T13:00:31.468 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi057.stderr:ceph-fuse[119022]: starting fuse 2023-10-19T13:00:31.479 INFO:teuthology.orchestra.run.smithi057.stdout:43 2023-10-19T13:00:31.480 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2023-10-19T13:00:31.480 DEBUG:teuthology.orchestra.run.smithi057:> sudo stdin-killer -- python3 -c ' 2023-10-19T13:00:31.480 DEBUG:teuthology.orchestra.run.smithi057:> import glob 2023-10-19T13:00:31.481 DEBUG:teuthology.orchestra.run.smithi057:> import re 2023-10-19T13:00:31.481 DEBUG:teuthology.orchestra.run.smithi057:> import os 2023-10-19T13:00:31.481 DEBUG:teuthology.orchestra.run.smithi057:> import subprocess 2023-10-19T13:00:31.482 DEBUG:teuthology.orchestra.run.smithi057:> 2023-10-19T13:00:31.482 DEBUG:teuthology.orchestra.run.smithi057:> def _find_admin_socket(client_name): 2023-10-19T13:00:31.483 DEBUG:teuthology.orchestra.run.smithi057:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-10-19T13:00:31.483 DEBUG:teuthology.orchestra.run.smithi057:> files = glob.glob(asok_path) 2023-10-19T13:00:31.483 DEBUG:teuthology.orchestra.run.smithi057:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-10-19T13:00:31.484 DEBUG:teuthology.orchestra.run.smithi057:> 2023-10-19T13:00:31.484 DEBUG:teuthology.orchestra.run.smithi057:> # Given a non-glob path, it better be there 2023-10-19T13:00:31.484 DEBUG:teuthology.orchestra.run.smithi057:> if "*" not in asok_path: 2023-10-19T13:00:31.485 DEBUG:teuthology.orchestra.run.smithi057:> assert(len(files) == 1) 2023-10-19T13:00:31.485 DEBUG:teuthology.orchestra.run.smithi057:> return files[0] 2023-10-19T13:00:31.485 DEBUG:teuthology.orchestra.run.smithi057:> 2023-10-19T13:00:31.486 DEBUG:teuthology.orchestra.run.smithi057:> for f in files: 2023-10-19T13:00:31.486 DEBUG:teuthology.orchestra.run.smithi057:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-10-19T13:00:31.486 DEBUG:teuthology.orchestra.run.smithi057:> if os.path.exists("/proc/{0}".format(pid)): 2023-10-19T13:00:31.487 DEBUG:teuthology.orchestra.run.smithi057:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-10-19T13:00:31.487 DEBUG:teuthology.orchestra.run.smithi057:> contents = proc_f.read() 2023-10-19T13:00:31.487 DEBUG:teuthology.orchestra.run.smithi057:> if mountpoint in contents: 2023-10-19T13:00:31.488 DEBUG:teuthology.orchestra.run.smithi057:> return f 2023-10-19T13:00:31.488 DEBUG:teuthology.orchestra.run.smithi057:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-10-19T13:00:31.488 DEBUG:teuthology.orchestra.run.smithi057:> 2023-10-19T13:00:31.489 DEBUG:teuthology.orchestra.run.smithi057:> print(_find_admin_socket("client.0")) 2023-10-19T13:00:31.489 DEBUG:teuthology.orchestra.run.smithi057:> ' 2023-10-19T13:00:31.604 INFO:teuthology.orchestra.run.smithi057.stdout:/var/run/ceph/ceph-client.0.119022.asok 2023-10-19T13:00:31.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:31 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2023-10-19T13:00:31.614 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.119022.asok 2023-10-19T13:00:31.614 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:00:31.615 DEBUG:teuthology.orchestra.run.smithi057:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.119022.asok status 2023-10-19T13:00:31.757 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2023-10-19T13:00:31.758 INFO:teuthology.orchestra.run.smithi057.stdout: "metadata": { 2023-10-19T13:00:31.758 INFO:teuthology.orchestra.run.smithi057.stdout: "ceph_sha1": "f7420d7f8e55ceea786dcbf536c338ce920c21eb", 2023-10-19T13:00:31.758 INFO:teuthology.orchestra.run.smithi057.stdout: "ceph_version": "ceph version 18.0.0-6786-gf7420d7f (f7420d7f8e55ceea786dcbf536c338ce920c21eb) reef (dev)", 2023-10-19T13:00:31.758 INFO:teuthology.orchestra.run.smithi057.stdout: "entity_id": "0", 2023-10-19T13:00:31.759 INFO:teuthology.orchestra.run.smithi057.stdout: "hostname": "smithi057", 2023-10-19T13:00:31.759 INFO:teuthology.orchestra.run.smithi057.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-10-19T13:00:31.759 INFO:teuthology.orchestra.run.smithi057.stdout: "pid": "119022", 2023-10-19T13:00:31.759 INFO:teuthology.orchestra.run.smithi057.stdout: "root": "/" 2023-10-19T13:00:31.759 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:00:31.760 INFO:teuthology.orchestra.run.smithi057.stdout: "dentry_count": 0, 2023-10-19T13:00:31.760 INFO:teuthology.orchestra.run.smithi057.stdout: "dentry_pinned_count": 0, 2023-10-19T13:00:31.760 INFO:teuthology.orchestra.run.smithi057.stdout: "id": 7236, 2023-10-19T13:00:31.760 INFO:teuthology.orchestra.run.smithi057.stdout: "inst": { 2023-10-19T13:00:31.760 INFO:teuthology.orchestra.run.smithi057.stdout: "name": { 2023-10-19T13:00:31.761 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "client", 2023-10-19T13:00:31.761 INFO:teuthology.orchestra.run.smithi057.stdout: "num": 7236 2023-10-19T13:00:31.761 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:00:31.761 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": { 2023-10-19T13:00:31.761 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "v1", 2023-10-19T13:00:31.761 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": "192.168.0.1:0", 2023-10-19T13:00:31.762 INFO:teuthology.orchestra.run.smithi057.stdout: "nonce": 3751097283 2023-10-19T13:00:31.762 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:00:31.762 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:00:31.762 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": { 2023-10-19T13:00:31.763 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "v1", 2023-10-19T13:00:31.763 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": "192.168.0.1:0", 2023-10-19T13:00:31.763 INFO:teuthology.orchestra.run.smithi057.stdout: "nonce": 3751097283 2023-10-19T13:00:31.763 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:00:31.763 INFO:teuthology.orchestra.run.smithi057.stdout: "inst_str": "client.7236 192.168.0.1:0/3751097283", 2023-10-19T13:00:31.764 INFO:teuthology.orchestra.run.smithi057.stdout: "addr_str": "192.168.0.1:0/3751097283", 2023-10-19T13:00:31.764 INFO:teuthology.orchestra.run.smithi057.stdout: "inode_count": 1, 2023-10-19T13:00:31.764 INFO:teuthology.orchestra.run.smithi057.stdout: "mds_epoch": 30, 2023-10-19T13:00:31.764 INFO:teuthology.orchestra.run.smithi057.stdout: "osd_epoch": 87, 2023-10-19T13:00:31.764 INFO:teuthology.orchestra.run.smithi057.stdout: "osd_epoch_barrier": 0, 2023-10-19T13:00:31.764 INFO:teuthology.orchestra.run.smithi057.stdout: "blocklisted": false, 2023-10-19T13:00:31.765 INFO:teuthology.orchestra.run.smithi057.stdout: "fs_name": "cephfs" 2023-10-19T13:00:31.765 INFO:teuthology.orchestra.run.smithi057.stdout:} 2023-10-19T13:00:31.766 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:00:31.766 DEBUG:teuthology.orchestra.run.smithi057:> stat --file-system '--printf=%T 2023-10-19T13:00:31.767 DEBUG:teuthology.orchestra.run.smithi057:> ' -- /home/ubuntu/cephtest/mnt.0 2023-10-19T13:00:31.782 INFO:teuthology.orchestra.run.smithi057.stdout:fuseblk 2023-10-19T13:00:31.783 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-10-19T13:00:31.783 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:00:31.783 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-19T13:00:31.853 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-19T13:00:32.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.000+0000 7fabbe96e640 1 -- 172.21.15.57:0/926660137 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fabb80b0530 msgr2=0x7fabb80b09c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:32.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.000+0000 7fabbe96e640 1 --2- 172.21.15.57:0/926660137 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fabb80b0530 0x7fabb80b09c0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7fabb0005fd0 tx=0x7fabb002c240 comp rx=0 tx=0).stop 2023-10-19T13:00:32.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.000+0000 7fabbe96e640 1 -- 172.21.15.57:0/926660137 shutdown_connections 2023-10-19T13:00:32.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.000+0000 7fabbe96e640 1 --2- 172.21.15.57:0/926660137 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fabb80b0530 0x7fabb80b09c0 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:32.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.000+0000 7fabbe96e640 1 --2- 172.21.15.57:0/926660137 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fabb80afa20 0x7fabb8158300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:32.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.000+0000 7fabbe96e640 1 -- 172.21.15.57:0/926660137 >> 172.21.15.57:0/926660137 conn(0x7fabb8069930 msgr2=0x7fabb8069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:32.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.000+0000 7fabbe96e640 1 -- 172.21.15.57:0/926660137 shutdown_connections 2023-10-19T13:00:32.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.000+0000 7fabbe96e640 1 -- 172.21.15.57:0/926660137 wait complete. 2023-10-19T13:00:32.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.001+0000 7fabbe96e640 1 Processor -- start 2023-10-19T13:00:32.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.001+0000 7fabbe96e640 1 -- start start 2023-10-19T13:00:32.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.002+0000 7fabbe96e640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fabb80afa20 0x7fabb8137c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:32.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.002+0000 7fabbe96e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fabb80b0530 0x7fabb8137060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:32.008 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.002+0000 7fabbe96e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fabb80cbd20 0x7fabb813f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:32.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.002+0000 7fabbe96e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabb813cad0 con 0x7fabb80cbd20 2023-10-19T13:00:32.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.002+0000 7fabbe96e640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabb8146510 con 0x7fabb80b0530 2023-10-19T13:00:32.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.002+0000 7fabbe96e640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabb81467f0 con 0x7fabb80afa20 2023-10-19T13:00:32.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.002+0000 7fabbcf0b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fabb80cbd20 0x7fabb813f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:32.010 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.002+0000 7fabbcf0b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fabb80cbd20 0x7fabb813f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56480/0 (socket says 172.21.15.57:56480) 2023-10-19T13:00:32.010 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.002+0000 7fabbcf0b640 1 -- 172.21.15.57:0/1824387655 learned_addr learned my addr 172.21.15.57:0/1824387655 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:32.010 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.002+0000 7fabb7fff640 1 --2- 172.21.15.57:0/1824387655 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fabb80afa20 0x7fabb8137c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:32.010 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.002+0000 7fabb77fe640 1 --2- 172.21.15.57:0/1824387655 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fabb80b0530 0x7fabb8137060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:32.011 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.002+0000 7fabbcf0b640 1 -- 172.21.15.57:0/1824387655 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fabb80afa20 msgr2=0x7fabb8137c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:32.011 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.002+0000 7fabbcf0b640 1 --2- 172.21.15.57:0/1824387655 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fabb80afa20 0x7fabb8137c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:32.011 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.002+0000 7fabbcf0b640 1 -- 172.21.15.57:0/1824387655 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fabb80b0530 msgr2=0x7fabb8137060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:32.011 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.002+0000 7fabbcf0b640 1 --2- 172.21.15.57:0/1824387655 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fabb80b0530 0x7fabb8137060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:32.011 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.002+0000 7fabbcf0b640 1 -- 172.21.15.57:0/1824387655 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabb0009900 con 0x7fabb80cbd20 2023-10-19T13:00:32.012 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.002+0000 7fabb77fe640 1 --2- 172.21.15.57:0/1824387655 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fabb80b0530 0x7fabb8137060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T13:00:32.012 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.002+0000 7fabbcf0b640 1 --2- 172.21.15.57:0/1824387655 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fabb80cbd20 0x7fabb813f790 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7fabb0009bf0 tx=0x7fabb0035e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:32.012 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.003+0000 7fabb4ff9640 1 -- 172.21.15.57:0/1824387655 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabb0034440 con 0x7fabb80cbd20 2023-10-19T13:00:32.012 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.003+0000 7fabb4ff9640 1 -- 172.21.15.57:0/1824387655 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fabb0004250 con 0x7fabb80cbd20 2023-10-19T13:00:32.012 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.003+0000 7fabbe96e640 1 -- 172.21.15.57:0/1824387655 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabb8146ad0 con 0x7fabb80cbd20 2023-10-19T13:00:32.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.003+0000 7fabb4ff9640 1 -- 172.21.15.57:0/1824387655 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabb0004600 con 0x7fabb80cbd20 2023-10-19T13:00:32.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.003+0000 7fabbe96e640 1 -- 172.21.15.57:0/1824387655 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabb8138180 con 0x7fabb80cbd20 2023-10-19T13:00:32.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.004+0000 7fabbe96e640 1 -- 172.21.15.57:0/1824387655 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabb8146ff0 con 0x7fabb80cbd20 2023-10-19T13:00:32.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.004+0000 7fabb4ff9640 1 -- 172.21.15.57:0/1824387655 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 32) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fabb000a040 con 0x7fabb80cbd20 2023-10-19T13:00:32.014 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.005+0000 7fabb4ff9640 1 --2- 172.21.15.57:0/1824387655 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7faba406af60 0x7faba406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:32.014 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.005+0000 7fabb4ff9640 1 -- 172.21.15.57:0/1824387655 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fabb00ac8e0 con 0x7fabb80cbd20 2023-10-19T13:00:32.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.005+0000 7fabb7fff640 1 --2- 172.21.15.57:0/1824387655 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7faba406af60 0x7faba406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:32.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.005+0000 7fabb7fff640 1 --2- 172.21.15.57:0/1824387655 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7faba406af60 0x7faba406d420 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fabac006fd0 tx=0x7fabac008040 comp rx=0 tx=0).ready entity=mgr.6256 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:32.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.009+0000 7fabb4ff9640 1 -- 172.21.15.57:0/1824387655 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fabb8146ff0 con 0x7fabb80cbd20 2023-10-19T13:00:32.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.206+0000 7fabbe96e640 1 -- 172.21.15.57:0/1824387655 --> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fabb80cc100 con 0x7faba406af60 2023-10-19T13:00:32.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.208+0000 7fabb4ff9640 1 -- 172.21.15.57:0/1824387655 <== mgr.6256 v2:172.21.15.179:6832/3815793798 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fabac00bda0 con 0x7faba406af60 2023-10-19T13:00:32.211 INFO:teuthology.orchestra.run.smithi057.stdout:[ 2023-10-19T13:00:32.212 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:00:32.212 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "cephfs" 2023-10-19T13:00:32.212 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:00:32.212 INFO:teuthology.orchestra.run.smithi057.stdout:] 2023-10-19T13:00:32.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.210+0000 7fabbe96e640 1 -- 172.21.15.57:0/1824387655 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7faba406af60 msgr2=0x7faba406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:32.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.210+0000 7fabbe96e640 1 --2- 172.21.15.57:0/1824387655 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7faba406af60 0x7faba406d420 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fabac006fd0 tx=0x7fabac008040 comp rx=0 tx=0).stop 2023-10-19T13:00:32.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.210+0000 7fabbe96e640 1 -- 172.21.15.57:0/1824387655 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fabb80cbd20 msgr2=0x7fabb813f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:32.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.210+0000 7fabbe96e640 1 --2- 172.21.15.57:0/1824387655 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fabb80cbd20 0x7fabb813f790 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7fabb0009bf0 tx=0x7fabb0035e30 comp rx=0 tx=0).stop 2023-10-19T13:00:32.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.210+0000 7fabbe96e640 1 -- 172.21.15.57:0/1824387655 shutdown_connections 2023-10-19T13:00:32.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.210+0000 7fabbe96e640 1 --2- 172.21.15.57:0/1824387655 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7faba406af60 0x7faba406d420 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:32.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.210+0000 7fabbe96e640 1 --2- 172.21.15.57:0/1824387655 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fabb80cbd20 0x7fabb813f790 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:32.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.210+0000 7fabbe96e640 1 --2- 172.21.15.57:0/1824387655 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fabb80b0530 0x7fabb8137060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:32.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.210+0000 7fabbe96e640 1 --2- 172.21.15.57:0/1824387655 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fabb80afa20 0x7fabb8137c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:32.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.210+0000 7fabbe96e640 1 -- 172.21.15.57:0/1824387655 >> 172.21.15.57:0/1824387655 conn(0x7fabb8069930 msgr2=0x7fabb80ae700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:32.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.210+0000 7fabbe96e640 1 -- 172.21.15.57:0/1824387655 shutdown_connections 2023-10-19T13:00:32.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.210+0000 7fabbe96e640 1 -- 172.21.15.57:0/1824387655 wait complete. 2023-10-19T13:00:32.223 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-19T13:00:32.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.332+0000 7feb19325640 1 -- 172.21.15.57:0/356833452 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feb000039a0 msgr2=0x7feb00003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:32.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.332+0000 7feb19325640 1 --2- 172.21.15.57:0/356833452 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feb000039a0 0x7feb00003dc0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7feb04009b20 tx=0x7feb000057f0 comp rx=0 tx=0).stop 2023-10-19T13:00:32.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.332+0000 7feb137fe640 1 -- 172.21.15.57:0/356833452 reap_dead start 2023-10-19T13:00:32.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.332+0000 7feb19325640 1 -- 172.21.15.57:0/356833452 shutdown_connections 2023-10-19T13:00:32.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.332+0000 7feb19325640 1 -- 172.21.15.57:0/356833452 >> 172.21.15.57:0/356833452 conn(0x7feb14069930 msgr2=0x7feb14069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:32.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.332+0000 7feb19325640 1 -- 172.21.15.57:0/356833452 shutdown_connections 2023-10-19T13:00:32.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.332+0000 7feb19325640 1 -- 172.21.15.57:0/356833452 wait complete. 2023-10-19T13:00:32.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.334+0000 7feb19325640 1 Processor -- start 2023-10-19T13:00:32.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.334+0000 7feb19325640 1 -- start start 2023-10-19T13:00:32.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.334+0000 7feb19325640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feb000039a0 0x7feb140b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:32.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.334+0000 7feb19325640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7feb14070540 0x7feb14070980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:32.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.334+0000 7feb19325640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feb14138c60 0x7feb141390e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:32.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.334+0000 7feb19325640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb141f2160 con 0x7feb000039a0 2023-10-19T13:00:32.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.334+0000 7feb19325640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb141f17c0 con 0x7feb14138c60 2023-10-19T13:00:32.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.334+0000 7feb19325640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb141f1aa0 con 0x7feb14070540 2023-10-19T13:00:32.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.334+0000 7feb12ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feb000039a0 0x7feb140b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:32.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.334+0000 7feb12ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feb000039a0 0x7feb140b0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56504/0 (socket says 172.21.15.57:56504) 2023-10-19T13:00:32.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.334+0000 7feb12ffd640 1 -- 172.21.15.57:0/2007454810 learned_addr learned my addr 172.21.15.57:0/2007454810 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:32.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.335+0000 7feb12ffd640 1 -- 172.21.15.57:0/2007454810 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7feb14070540 msgr2=0x7feb14070980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:32.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.335+0000 7feb0bfff640 1 --2- 172.21.15.57:0/2007454810 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7feb14070540 0x7feb14070980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:32.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.335+0000 7feb137fe640 1 --2- 172.21.15.57:0/2007454810 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feb14138c60 0x7feb141390e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:32.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.335+0000 7feb12ffd640 1 --2- 172.21.15.57:0/2007454810 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7feb14070540 0x7feb14070980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:32.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.335+0000 7feb12ffd640 1 -- 172.21.15.57:0/2007454810 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feb14138c60 msgr2=0x7feb141390e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:32.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.335+0000 7feb12ffd640 1 --2- 172.21.15.57:0/2007454810 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feb14138c60 0x7feb141390e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:32.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.335+0000 7feb12ffd640 1 -- 172.21.15.57:0/2007454810 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb141f1d80 con 0x7feb000039a0 2023-10-19T13:00:32.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.335+0000 7feb137fe640 1 --2- 172.21.15.57:0/2007454810 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feb14138c60 0x7feb141390e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T13:00:32.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.335+0000 7feb12ffd640 1 --2- 172.21.15.57:0/2007454810 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feb000039a0 0x7feb140b0cf0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7feb04009b20 tx=0x7feb04033c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:32.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.335+0000 7feb0b7fe640 1 -- 172.21.15.57:0/2007454810 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb04004440 con 0x7feb000039a0 2023-10-19T13:00:32.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.335+0000 7feb0b7fe640 1 -- 172.21.15.57:0/2007454810 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7feb04033e60 con 0x7feb000039a0 2023-10-19T13:00:32.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.335+0000 7feb19325640 1 -- 172.21.15.57:0/2007454810 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb141f1470 con 0x7feb000039a0 2023-10-19T13:00:32.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.335+0000 7feb0b7fe640 1 -- 172.21.15.57:0/2007454810 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb04003830 con 0x7feb000039a0 2023-10-19T13:00:32.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.335+0000 7feb19325640 1 -- 172.21.15.57:0/2007454810 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb14070ec0 con 0x7feb000039a0 2023-10-19T13:00:32.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.336+0000 7feb19325640 1 -- 172.21.15.57:0/2007454810 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb140690e0 con 0x7feb000039a0 2023-10-19T13:00:32.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.337+0000 7feb0b7fe640 1 -- 172.21.15.57:0/2007454810 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 32) v1 ==== 89331+0+0 (secure 0 0 0) 0x7feb04003990 con 0x7feb000039a0 2023-10-19T13:00:32.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.337+0000 7feb0b7fe640 1 --2- 172.21.15.57:0/2007454810 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7feaf406af60 0x7feaf406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:32.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.337+0000 7feb0b7fe640 1 -- 172.21.15.57:0/2007454810 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6057+0+0 (secure 0 0 0) 0x7feb040ac480 con 0x7feb000039a0 2023-10-19T13:00:32.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.337+0000 7feb0bfff640 1 --2- 172.21.15.57:0/2007454810 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7feaf406af60 0x7feaf406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:32.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.338+0000 7feb0bfff640 1 --2- 172.21.15.57:0/2007454810 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7feaf406af60 0x7feaf406d420 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7feafc001c10 tx=0x7feafc001b40 comp rx=0 tx=0).ready entity=mgr.6256 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:32.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.341+0000 7feb0b7fe640 1 -- 172.21.15.57:0/2007454810 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7feb04077240 con 0x7feb000039a0 2023-10-19T13:00:32.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:32.564+0000 7feb19325640 1 -- 172.21.15.57:0/2007454810 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7feb1415bbc0 con 0x7feb000039a0 2023-10-19T13:00:33.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.271+0000 7feb0b7fe640 1 -- 172.21.15.57:0/2007454810 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v33) v1 ==== 92+0+0 (secure 0 0 0) 0x7feb0407a860 con 0x7feb000039a0 2023-10-19T13:00:33.278 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.275+0000 7feb19325640 1 -- 172.21.15.57:0/2007454810 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7feaf406af60 msgr2=0x7feaf406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:33.279 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.275+0000 7feb19325640 1 --2- 172.21.15.57:0/2007454810 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7feaf406af60 0x7feaf406d420 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7feafc001c10 tx=0x7feafc001b40 comp rx=0 tx=0).stop 2023-10-19T13:00:33.279 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.275+0000 7feb19325640 1 -- 172.21.15.57:0/2007454810 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feb000039a0 msgr2=0x7feb140b0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:33.279 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.275+0000 7feb19325640 1 --2- 172.21.15.57:0/2007454810 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feb000039a0 0x7feb140b0cf0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7feb04009b20 tx=0x7feb04033c30 comp rx=0 tx=0).stop 2023-10-19T13:00:33.279 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.276+0000 7feb19325640 1 -- 172.21.15.57:0/2007454810 shutdown_connections 2023-10-19T13:00:33.279 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.276+0000 7feb19325640 1 --2- 172.21.15.57:0/2007454810 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7feaf406af60 0x7feaf406d420 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:33.280 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.276+0000 7feb19325640 1 --2- 172.21.15.57:0/2007454810 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feb14138c60 0x7feb141390e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:33.280 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.276+0000 7feb19325640 1 --2- 172.21.15.57:0/2007454810 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7feb14070540 0x7feb14070980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:33.280 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.276+0000 7feb19325640 1 --2- 172.21.15.57:0/2007454810 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feb000039a0 0x7feb140b0cf0 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:33.280 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.276+0000 7feb19325640 1 -- 172.21.15.57:0/2007454810 >> 172.21.15.57:0/2007454810 conn(0x7feb14069930 msgr2=0x7feb141525e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:33.280 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.276+0000 7feb19325640 1 -- 172.21.15.57:0/2007454810 shutdown_connections 2023-10-19T13:00:33.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.276+0000 7feb19325640 1 -- 172.21.15.57:0/2007454810 wait complete. 2023-10-19T13:00:33.292 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-19T13:00:33.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.411+0000 7ff964c39640 1 -- 172.21.15.57:0/1342161906 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff9600cf340 msgr2=0x7ff9600d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:33.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.411+0000 7ff964c39640 1 --2- 172.21.15.57:0/1342161906 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff9600cf340 0x7ff9600d1910 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7ff950009830 tx=0x7ff95002bd60 comp rx=0 tx=0).stop 2023-10-19T13:00:33.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.411+0000 7ff964c39640 1 -- 172.21.15.57:0/1342161906 shutdown_connections 2023-10-19T13:00:33.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.411+0000 7ff964c39640 1 --2- 172.21.15.57:0/1342161906 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff9600cf340 0x7ff9600d1910 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:33.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.411+0000 7ff964c39640 1 -- 172.21.15.57:0/1342161906 >> 172.21.15.57:0/1342161906 conn(0x7ff9600d0d60 msgr2=0x7ff9600d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:33.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.411+0000 7ff964c39640 1 -- 172.21.15.57:0/1342161906 shutdown_connections 2023-10-19T13:00:33.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.411+0000 7ff964c39640 1 -- 172.21.15.57:0/1342161906 wait complete. 2023-10-19T13:00:33.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.413+0000 7ff964c39640 1 Processor -- start 2023-10-19T13:00:33.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.413+0000 7ff964c39640 1 -- start start 2023-10-19T13:00:33.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.414+0000 7ff964c39640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff9600ceb40 0x7ff960158730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:33.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.414+0000 7ff964c39640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9600cf340 0x7ff9600ca6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:33.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.414+0000 7ff964c39640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff96015b2a0 0x7ff96015dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:33.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.414+0000 7ff964c39640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff960158c70 con 0x7ff9600cf340 2023-10-19T13:00:33.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.414+0000 7ff964c39640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff96009e450 con 0x7ff96015b2a0 2023-10-19T13:00:33.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.414+0000 7ff964c39640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9600c9e00 con 0x7ff9600ceb40 2023-10-19T13:00:33.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.414+0000 7ff95dd9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9600cf340 0x7ff9600ca6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:33.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.414+0000 7ff95e59c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff9600ceb40 0x7ff960158730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:33.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.414+0000 7ff95dd9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9600cf340 0x7ff9600ca6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56526/0 (socket says 172.21.15.57:56526) 2023-10-19T13:00:33.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.414+0000 7ff95dd9b640 1 -- 172.21.15.57:0/283022551 learned_addr learned my addr 172.21.15.57:0/283022551 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:33.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.414+0000 7ff95ed9d640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff96015b2a0 0x7ff96015dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:33.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.414+0000 7ff95dd9b640 1 -- 172.21.15.57:0/283022551 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff9600ceb40 msgr2=0x7ff960158730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:33.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.414+0000 7ff95dd9b640 1 --2- 172.21.15.57:0/283022551 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff9600ceb40 0x7ff960158730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:33.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.414+0000 7ff95dd9b640 1 -- 172.21.15.57:0/283022551 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff96015b2a0 msgr2=0x7ff96015dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:33.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.414+0000 7ff95dd9b640 1 --2- 172.21.15.57:0/283022551 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff96015b2a0 0x7ff96015dc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:33.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.414+0000 7ff95dd9b640 1 -- 172.21.15.57:0/283022551 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff950009490 con 0x7ff9600cf340 2023-10-19T13:00:33.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.414+0000 7ff95e59c640 1 --2- 172.21.15.57:0/283022551 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff9600ceb40 0x7ff960158730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T13:00:33.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.414+0000 7ff95ed9d640 1 --2- 172.21.15.57:0/283022551 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff96015b2a0 0x7ff96015dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T13:00:33.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.414+0000 7ff95dd9b640 1 --2- 172.21.15.57:0/283022551 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9600cf340 0x7ff9600ca6b0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7ff95400c410 tx=0x7ff95400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:33.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.415+0000 7ff94effd640 1 -- 172.21.15.57:0/283022551 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9540035f0 con 0x7ff9600cf340 2023-10-19T13:00:33.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.415+0000 7ff94effd640 1 -- 172.21.15.57:0/283022551 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7ff954003770 con 0x7ff9600cf340 2023-10-19T13:00:33.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.415+0000 7ff964c39640 1 -- 172.21.15.57:0/283022551 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9600c91f0 con 0x7ff9600cf340 2023-10-19T13:00:33.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.415+0000 7ff94effd640 1 -- 172.21.15.57:0/283022551 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff954003b80 con 0x7ff9600cf340 2023-10-19T13:00:33.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.415+0000 7ff964c39640 1 -- 172.21.15.57:0/283022551 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9600c97b0 con 0x7ff9600cf340 2023-10-19T13:00:33.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.416+0000 7ff964c39640 1 -- 172.21.15.57:0/283022551 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9600d04f0 con 0x7ff9600cf340 2023-10-19T13:00:33.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.417+0000 7ff94effd640 1 -- 172.21.15.57:0/283022551 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 33) v1 ==== 89348+0+0 (secure 0 0 0) 0x7ff954003ce0 con 0x7ff9600cf340 2023-10-19T13:00:33.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.417+0000 7ff94effd640 1 --2- 172.21.15.57:0/283022551 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7ff94006afe0 0x7ff94006d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:33.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.417+0000 7ff95e59c640 1 -- 172.21.15.57:0/283022551 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7ff94006afe0 msgr2=0x7ff94006d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/3815793798 2023-10-19T13:00:33.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.417+0000 7ff94effd640 1 -- 172.21.15.57:0/283022551 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6057+0+0 (secure 0 0 0) 0x7ff954089170 con 0x7ff9600cf340 2023-10-19T13:00:33.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.417+0000 7ff95e59c640 1 --2- 172.21.15.57:0/283022551 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7ff94006afe0 0x7ff94006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T13:00:33.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.422+0000 7ff94effd640 1 -- 172.21.15.57:0/283022551 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff954053f40 con 0x7ff9600cf340 2023-10-19T13:00:33.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.600+0000 7ff964c39640 1 -- 172.21.15.57:0/283022551 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7ff9600cca20 con 0x7ff9600cf340 2023-10-19T13:00:33.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.604+0000 7ff94effd640 1 -- 172.21.15.57:0/283022551 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v40)=0 v40) v1 ==== 138+0+0 (secure 0 0 0) 0x7ff954057560 con 0x7ff9600cf340 2023-10-19T13:00:33.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.606+0000 7ff964c39640 1 -- 172.21.15.57:0/283022551 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7ff94006afe0 msgr2=0x7ff94006d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T13:00:33.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.606+0000 7ff964c39640 1 --2- 172.21.15.57:0/283022551 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7ff94006afe0 0x7ff94006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:33.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.606+0000 7ff964c39640 1 -- 172.21.15.57:0/283022551 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9600cf340 msgr2=0x7ff9600ca6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:33.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.606+0000 7ff964c39640 1 --2- 172.21.15.57:0/283022551 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9600cf340 0x7ff9600ca6b0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7ff95400c410 tx=0x7ff95400c8d0 comp rx=0 tx=0).stop 2023-10-19T13:00:33.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.606+0000 7ff964c39640 1 -- 172.21.15.57:0/283022551 shutdown_connections 2023-10-19T13:00:33.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.606+0000 7ff964c39640 1 --2- 172.21.15.57:0/283022551 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7ff94006afe0 0x7ff94006d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:33.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.606+0000 7ff964c39640 1 --2- 172.21.15.57:0/283022551 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff96015b2a0 0x7ff96015dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:33.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.606+0000 7ff964c39640 1 --2- 172.21.15.57:0/283022551 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9600cf340 0x7ff9600ca6b0 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:33.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.606+0000 7ff964c39640 1 --2- 172.21.15.57:0/283022551 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff9600ceb40 0x7ff960158730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:33.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.606+0000 7ff964c39640 1 -- 172.21.15.57:0/283022551 >> 172.21.15.57:0/283022551 conn(0x7ff9600d0d60 msgr2=0x7ff9600b2850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:33.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.606+0000 7ff964c39640 1 -- 172.21.15.57:0/283022551 shutdown_connections 2023-10-19T13:00:33.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.606+0000 7ff964c39640 1 -- 172.21.15.57:0/283022551 wait complete. 2023-10-19T13:00:33.620 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-19T13:00:33.657 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:33.654+0000 7f310603e040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-19T13:00:33.686 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:00:33.683+0000 7f213e017040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-19T13:00:33.701 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:33.698+0000 7f310603e040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-19T13:00:33.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.730+0000 7f8174e90640 1 -- 172.21.15.57:0/1907444620 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f81700cf850 msgr2=0x7f81700cc9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:33.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.730+0000 7f8174e90640 1 --2- 172.21.15.57:0/1907444620 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f81700cf850 0x7f81700cc9f0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f81700894b0 tx=0x7f816802c110 comp rx=0 tx=0).stop 2023-10-19T13:00:33.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.731+0000 7f8174e90640 1 -- 172.21.15.57:0/1907444620 shutdown_connections 2023-10-19T13:00:33.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.731+0000 7f8174e90640 1 --2- 172.21.15.57:0/1907444620 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f81700cf850 0x7f81700cc9f0 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:33.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.731+0000 7f8174e90640 1 --2- 172.21.15.57:0/1907444620 >> v2:172.21.15.179:3300/0 conn(0x7f817015b260 0x7f81700cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:33.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.731+0000 7f8174e90640 1 -- 172.21.15.57:0/1907444620 >> 172.21.15.57:0/1907444620 conn(0x7f817009a030 msgr2=0x7f81700d0d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:33.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.731+0000 7f8174e90640 1 -- 172.21.15.57:0/1907444620 shutdown_connections 2023-10-19T13:00:33.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.731+0000 7f8174e90640 1 -- 172.21.15.57:0/1907444620 wait complete. 2023-10-19T13:00:33.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.732+0000 7f8174e90640 1 Processor -- start 2023-10-19T13:00:33.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.732+0000 7f8174e90640 1 -- start start 2023-10-19T13:00:33.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.733+0000 7f8174e90640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f81700cf850 0x7f817014d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:33.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.733+0000 7f8174e90640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f817015b260 0x7f81700c9cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:33.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.733+0000 7f8174e90640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f817015b640 0x7f81700aec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:33.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.733+0000 7f8174e90640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81700c7f90 con 0x7f81700cf850 2023-10-19T13:00:33.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.733+0000 7f8174e90640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81700c8230 con 0x7f817015b260 2023-10-19T13:00:33.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.733+0000 7f8174e90640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81700c8510 con 0x7f817015b640 2023-10-19T13:00:33.738 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:00:33.733+0000 7f213e017040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-19T13:00:33.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.733+0000 7f816e59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f81700cf850 0x7f817014d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:33.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.733+0000 7f816e59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f81700cf850 0x7f817014d0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56546/0 (socket says 172.21.15.57:56546) 2023-10-19T13:00:33.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.733+0000 7f816e59c640 1 -- 172.21.15.57:0/3540278046 learned_addr learned my addr 172.21.15.57:0/3540278046 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:33.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.733+0000 7f816dd9b640 1 --2- 172.21.15.57:0/3540278046 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f817015b260 0x7f81700c9cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:33.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.733+0000 7f816ed9d640 1 --2- 172.21.15.57:0/3540278046 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f817015b640 0x7f81700aec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:33.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.733+0000 7f816e59c640 1 -- 172.21.15.57:0/3540278046 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f817015b640 msgr2=0x7f81700aec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:33.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.733+0000 7f816e59c640 1 --2- 172.21.15.57:0/3540278046 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f817015b640 0x7f81700aec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:33.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.733+0000 7f816e59c640 1 -- 172.21.15.57:0/3540278046 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f817015b260 msgr2=0x7f81700c9cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:33.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.733+0000 7f816e59c640 1 --2- 172.21.15.57:0/3540278046 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f817015b260 0x7f81700c9cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:33.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.733+0000 7f816e59c640 1 -- 172.21.15.57:0/3540278046 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8168009840 con 0x7f81700cf850 2023-10-19T13:00:33.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.734+0000 7f816e59c640 1 --2- 172.21.15.57:0/3540278046 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f81700cf850 0x7f817014d0f0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f816000e9f0 tx=0x7f816000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:33.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.734+0000 7f815effd640 1 -- 172.21.15.57:0/3540278046 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8160009700 con 0x7f81700cf850 2023-10-19T13:00:33.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.734+0000 7f815effd640 1 -- 172.21.15.57:0/3540278046 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f816000ce70 con 0x7f81700cf850 2023-10-19T13:00:33.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.734+0000 7f815effd640 1 -- 172.21.15.57:0/3540278046 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8160010670 con 0x7f81700cf850 2023-10-19T13:00:33.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.734+0000 7f8174e90640 1 -- 172.21.15.57:0/3540278046 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81700c87f0 con 0x7f81700cf850 2023-10-19T13:00:33.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.734+0000 7f8174e90640 1 -- 172.21.15.57:0/3540278046 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f817014d630 con 0x7f81700cf850 2023-10-19T13:00:33.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.735+0000 7f8174e90640 1 -- 172.21.15.57:0/3540278046 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81700675f0 con 0x7f81700cf850 2023-10-19T13:00:33.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.736+0000 7f815effd640 1 -- 172.21.15.57:0/3540278046 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 33) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f81600107d0 con 0x7f81700cf850 2023-10-19T13:00:33.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.736+0000 7f815effd640 1 --2- 172.21.15.57:0/3540278046 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f815006afe0 0x7f815006d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:33.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.736+0000 7f815effd640 1 -- 172.21.15.57:0/3540278046 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f8160014070 con 0x7f81700cf850 2023-10-19T13:00:33.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.736+0000 7f816dd9b640 1 -- 172.21.15.57:0/3540278046 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f815006afe0 msgr2=0x7f815006d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/3815793798 2023-10-19T13:00:33.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.736+0000 7f816dd9b640 1 --2- 172.21.15.57:0/3540278046 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f815006afe0 0x7f815006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T13:00:33.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.740+0000 7f815effd640 1 -- 172.21.15.57:0/3540278046 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8160051ca0 con 0x7f81700cf850 2023-10-19T13:00:33.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.917+0000 7f8174e90640 1 -- 172.21.15.57:0/3540278046 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f81700affb0 con 0x7f81700cf850 2023-10-19T13:00:33.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.921+0000 7f815effd640 1 -- 172.21.15.57:0/3540278046 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v41)=0 v41) v1 ==== 133+0+0 (secure 0 0 0) 0x7f8160058ea0 con 0x7f81700cf850 2023-10-19T13:00:33.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.923+0000 7f8174e90640 1 -- 172.21.15.57:0/3540278046 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f815006afe0 msgr2=0x7f815006d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T13:00:33.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.923+0000 7f8174e90640 1 --2- 172.21.15.57:0/3540278046 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f815006afe0 0x7f815006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:33.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.923+0000 7f8174e90640 1 -- 172.21.15.57:0/3540278046 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f81700cf850 msgr2=0x7f817014d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:33.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.923+0000 7f8174e90640 1 --2- 172.21.15.57:0/3540278046 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f81700cf850 0x7f817014d0f0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f816000e9f0 tx=0x7f816000eeb0 comp rx=0 tx=0).stop 2023-10-19T13:00:33.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.923+0000 7f8174e90640 1 -- 172.21.15.57:0/3540278046 shutdown_connections 2023-10-19T13:00:33.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.923+0000 7f8174e90640 1 --2- 172.21.15.57:0/3540278046 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f815006afe0 0x7f815006d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:33.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.923+0000 7f8174e90640 1 --2- 172.21.15.57:0/3540278046 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f817015b640 0x7f81700aec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:33.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.923+0000 7f8174e90640 1 --2- 172.21.15.57:0/3540278046 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f817015b260 0x7f81700c9cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:33.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.923+0000 7f8174e90640 1 --2- 172.21.15.57:0/3540278046 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f81700cf850 0x7f817014d0f0 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:33.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.923+0000 7f8174e90640 1 -- 172.21.15.57:0/3540278046 >> 172.21.15.57:0/3540278046 conn(0x7f817009a030 msgr2=0x7f81700b18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:33.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.923+0000 7f8174e90640 1 -- 172.21.15.57:0/3540278046 shutdown_connections 2023-10-19T13:00:33.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:33.923+0000 7f8174e90640 1 -- 172.21.15.57:0/3540278046 wait complete. 2023-10-19T13:00:33.937 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:00:33.937 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-10-19T13:00:33.958 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list snap_test_dir1 2023-10-19T13:00:34.091 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:34.088+0000 7f310603e040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-19T13:00:34.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:34.110+0000 7f5b85bb0640 1 -- 172.21.15.57:0/1118114934 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5b800cddc0 msgr2=0x7f5b80165c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:34.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:34.110+0000 7f5b85bb0640 1 --2- 172.21.15.57:0/1118114934 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5b800cddc0 0x7f5b80165c20 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f5b78005fd0 tx=0x7f5b7802c0e0 comp rx=0 tx=0).stop 2023-10-19T13:00:34.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:34.110+0000 7f5b85bb0640 1 -- 172.21.15.57:0/1118114934 shutdown_connections 2023-10-19T13:00:34.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:34.110+0000 7f5b85bb0640 1 --2- 172.21.15.57:0/1118114934 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5b800cddc0 0x7f5b80165c20 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:34.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:34.110+0000 7f5b85bb0640 1 -- 172.21.15.57:0/1118114934 >> 172.21.15.57:0/1118114934 conn(0x7f5b80099ff0 msgr2=0x7f5b800d0d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:34.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:34.110+0000 7f5b85bb0640 1 -- 172.21.15.57:0/1118114934 shutdown_connections 2023-10-19T13:00:34.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:34.110+0000 7f5b85bb0640 1 -- 172.21.15.57:0/1118114934 wait complete. 2023-10-19T13:00:34.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:34.111+0000 7f5b85bb0640 1 Processor -- start 2023-10-19T13:00:34.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:34.112+0000 7f5b85bb0640 1 -- start start 2023-10-19T13:00:34.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:34.112+0000 7f5b85bb0640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5b800cddc0 0x7f5b800b0310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:34.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:34.112+0000 7f5b85bb0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5b800ce420 0x7f5b800b53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:34.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:34.112+0000 7f5b85bb0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5b8015b180 0x7f5b800b5930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:34.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:34.112+0000 7f5b85bb0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b800c9030 con 0x7f5b800ce420 2023-10-19T13:00:34.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:34.112+0000 7f5b85bb0640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b800b5e70 con 0x7f5b8015b180 2023-10-19T13:00:34.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:34.112+0000 7f5b85bb0640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b800b6120 con 0x7f5b800cddc0 2023-10-19T13:00:34.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:34.112+0000 7f5b7effd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5b800ce420 0x7f5b800b53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:34.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:34.113+0000 7f5b7effd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5b800ce420 0x7f5b800b53f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56552/0 (socket says 172.21.15.57:56552) 2023-10-19T13:00:34.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:34.113+0000 7f5b7effd640 1 -- 172.21.15.57:0/286214753 learned_addr learned my addr 172.21.15.57:0/286214753 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:34.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:34.113+0000 7f5b7ffff640 1 --2- 172.21.15.57:0/286214753 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5b8015b180 0x7f5b800b5930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:34.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:34.113+0000 7f5b7f7fe640 1 --2- 172.21.15.57:0/286214753 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5b800cddc0 0x7f5b800b0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:34.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:34.113+0000 7f5b7effd640 1 -- 172.21.15.57:0/286214753 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5b800cddc0 msgr2=0x7f5b800b0310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:34.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:34.113+0000 7f5b7effd640 1 --2- 172.21.15.57:0/286214753 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5b800cddc0 0x7f5b800b0310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:34.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:34.113+0000 7f5b7effd640 1 -- 172.21.15.57:0/286214753 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5b8015b180 msgr2=0x7f5b800b5930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:34.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:34.113+0000 7f5b7effd640 1 --2- 172.21.15.57:0/286214753 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5b8015b180 0x7f5b800b5930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:34.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:34.113+0000 7f5b7effd640 1 -- 172.21.15.57:0/286214753 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b78009840 con 0x7f5b800ce420 2023-10-19T13:00:34.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:34.113+0000 7f5b7ffff640 1 --2- 172.21.15.57:0/286214753 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5b8015b180 0x7f5b800b5930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T13:00:34.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:34.113+0000 7f5b7effd640 1 --2- 172.21.15.57:0/286214753 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5b800ce420 0x7f5b800b53f0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f5b7400c780 tx=0x7f5b7400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:34.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:34.113+0000 7f5b67fff640 1 -- 172.21.15.57:0/286214753 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b7400a840 con 0x7f5b800ce420 2023-10-19T13:00:34.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:34.113+0000 7f5b67fff640 1 -- 172.21.15.57:0/286214753 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f5b7400aab0 con 0x7f5b800ce420 2023-10-19T13:00:34.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:34.113+0000 7f5b67fff640 1 -- 172.21.15.57:0/286214753 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b74012420 con 0x7f5b800ce420 2023-10-19T13:00:34.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:34.113+0000 7f5b85bb0640 1 -- 172.21.15.57:0/286214753 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b800c8970 con 0x7f5b800ce420 2023-10-19T13:00:34.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:34.113+0000 7f5b85bb0640 1 -- 172.21.15.57:0/286214753 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b800b0850 con 0x7f5b800ce420 2023-10-19T13:00:34.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:34.114+0000 7f5b85bb0640 1 -- 172.21.15.57:0/286214753 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b80066960 con 0x7f5b800ce420 2023-10-19T13:00:34.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:34.115+0000 7f5b67fff640 1 -- 172.21.15.57:0/286214753 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 33) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f5b7401a3b0 con 0x7f5b800ce420 2023-10-19T13:00:34.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:34.115+0000 7f5b67fff640 1 --2- 172.21.15.57:0/286214753 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f5b6c06afb0 0x7f5b6c06d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:34.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:34.116+0000 7f5b7f7fe640 1 -- 172.21.15.57:0/286214753 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f5b6c06afb0 msgr2=0x7f5b6c06d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/3815793798 2023-10-19T13:00:34.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:34.116+0000 7f5b67fff640 1 -- 172.21.15.57:0/286214753 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f5b74089b10 con 0x7f5b800ce420 2023-10-19T13:00:34.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:34.116+0000 7f5b7f7fe640 1 --2- 172.21.15.57:0/286214753 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f5b6c06afb0 0x7f5b6c06d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T13:00:34.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:34.120+0000 7f5b67fff640 1 -- 172.21.15.57:0/286214753 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f5b74054a00 con 0x7f5b800ce420 2023-10-19T13:00:34.135 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:34.132+0000 7f310603e040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-19T13:00:34.156 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:00:34.154+0000 7f213e017040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-19T13:00:34.177 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:34.174+0000 7f310603e040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-19T13:00:34.203 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:00:34.201+0000 7f213e017040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-19T13:00:34.227 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:34.225+0000 7f310603e040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-19T13:00:34.249 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:00:34.246+0000 7f213e017040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-19T13:00:34.268 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:34.265+0000 7f310603e040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-19T13:00:34.304 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:00:34.302+0000 7f213e017040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-19T13:00:34.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:34.317+0000 7f5b7f7fe640 1 -- 172.21.15.57:0/286214753 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f5b6c06afb0 msgr2=0x7f5b6c06d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/3815793798 2023-10-19T13:00:34.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:34.317+0000 7f5b7f7fe640 1 --2- 172.21.15.57:0/286214753 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f5b6c06afb0 0x7f5b6c06d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-19T13:00:34.321 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:34.319+0000 7f5b85bb0640 1 -- 172.21.15.57:0/286214753 --> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f5b800c87f0 con 0x7f5b6c06afb0 2023-10-19T13:00:34.348 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:00:34.346+0000 7f213e017040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-19T13:00:34.367 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:34.364+0000 7f310603e040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-19T13:00:34.418 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:34.416+0000 7f310603e040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-19T13:00:34.452 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:00:34.449+0000 7f213e017040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-19T13:00:34.459 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:34.457+0000 7f310603e040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-19T13:00:34.507 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:34.504+0000 7f310603e040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-19T13:00:34.508 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:00:34.505+0000 7f213e017040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-19T13:00:34.548 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:34.546+0000 7f310603e040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-19T13:00:34.552 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:00:34.549+0000 7f213e017040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-19T13:00:34.603 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:00:34.600+0000 7f213e017040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-19T13:00:34.647 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:00:34.645+0000 7f213e017040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-19T13:00:34.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:34.718+0000 7f5b7f7fe640 1 -- 172.21.15.57:0/286214753 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f5b6c06afb0 msgr2=0x7f5b6c06d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/3815793798 2023-10-19T13:00:34.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:34.718+0000 7f5b7f7fe640 1 --2- 172.21.15.57:0/286214753 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f5b6c06afb0 0x7f5b6c06d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-19T13:00:34.859 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:34.856+0000 7f310603e040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-19T13:00:34.983 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:00:34.981+0000 7f213e017040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-19T13:00:34.987 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:34.984+0000 7f310603e040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-19T13:00:35.077 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:35.074+0000 7f310603e040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-19T13:00:35.118 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:35.116+0000 7f310603e040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-19T13:00:35.122 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:00:35.120+0000 7f213e017040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-19T13:00:35.212 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:35.209+0000 7f310603e040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-19T13:00:35.219 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:00:35.217+0000 7f213e017040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-19T13:00:35.264 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:00:35.262+0000 7f213e017040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-19T13:00:35.300 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:35.297+0000 7f310603e040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-19T13:00:35.366 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:00:35.363+0000 7f213e017040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-19T13:00:35.436 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:35.433+0000 7f310603e040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-19T13:00:35.462 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:00:35.460+0000 7f213e017040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-19T13:00:35.477 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:35.475+0000 7f310603e040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-19T13:00:35.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:35.519+0000 7f5b7f7fe640 1 -- 172.21.15.57:0/286214753 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f5b6c06afb0 msgr2=0x7f5b6c06d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/3815793798 2023-10-19T13:00:35.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:35.519+0000 7f5b7f7fe640 1 --2- 172.21.15.57:0/286214753 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f5b6c06afb0 0x7f5b6c06d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-19T13:00:35.581 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:35.578+0000 7f310603e040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-19T13:00:35.607 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:00:35.604+0000 7f213e017040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-19T13:00:35.651 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:00:35.648+0000 7f213e017040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-19T13:00:35.761 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:00:35.758+0000 7f213e017040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-19T13:00:36.191 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:36.188+0000 7f310603e040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-19T13:00:36.410 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:00:36.407+0000 7f213e017040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-19T13:00:36.452 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:36.450+0000 7f310603e040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-19T13:00:36.500 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:36.498+0000 7f310603e040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-19T13:00:36.599 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:36.596+0000 7f310603e040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-19T13:00:36.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:36.618+0000 7f5b67fff640 1 -- 172.21.15.57:0/286214753 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mgrmap(e 34) v1 ==== 88957+0+0 (secure 0 0 0) 0x7f5b74087ed0 con 0x7f5b800ce420 2023-10-19T13:00:36.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:36.618+0000 7f5b67fff640 1 -- 172.21.15.57:0/286214753 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f5b6c06afb0 msgr2=0x7f5b6c06d470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T13:00:36.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:36.618+0000 7f5b67fff640 1 --2- 172.21.15.57:0/286214753 >> [v2:172.21.15.179:6832/3815793798,v1:172.21.15.179:6833/3815793798] conn(0x7f5b6c06afb0 0x7f5b6c06d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:36.694 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:00:36.692+0000 7f213e017040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-19T13:00:36.726 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:36.723+0000 7f3085b66640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:00:36.726 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:36.723+0000 7f3085b66640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:00:36.727 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:36.723+0000 7f3085b66640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:00:36.727 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:36.723+0000 7f3085b66640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:00:36.727 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:36.723+0000 7f3085b66640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:00:36.727 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:36.723+0000 7f3082b20640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:00:36.727 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:36.723+0000 7f3082b20640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:00:36.727 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:36.723+0000 7f3082b20640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:00:36.728 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:36.723+0000 7f3082b20640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:00:36.728 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:36.723+0000 7f3082b20640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:00:36.749 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:00:36.747+0000 7f213e017040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-19T13:00:36.855 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:00:36.852+0000 7f213e017040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-19T13:00:37.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:37.622+0000 7f5b67fff640 1 -- 172.21.15.57:0/286214753 <== mon.0 v2:172.21.15.57:3300/0 8 ==== mgrmap(e 35) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f5b74054dd0 con 0x7f5b800ce420 2023-10-19T13:00:37.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:37.622+0000 7f5b67fff640 1 --2- 172.21.15.57:0/286214753 >> [v2:172.21.15.179:6832/137854273,v1:172.21.15.179:6833/137854273] conn(0x7f5b6c035850 0x7f5b6c0753e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:37.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:37.622+0000 7f5b67fff640 1 -- 172.21.15.57:0/286214753 --> [v2:172.21.15.179:6832/137854273,v1:172.21.15.179:6833/137854273] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f5b6c035c30 con 0x7f5b6c035850 2023-10-19T13:00:37.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:37.625+0000 7f5b7f7fe640 1 --2- 172.21.15.57:0/286214753 >> [v2:172.21.15.179:6832/137854273,v1:172.21.15.179:6833/137854273] conn(0x7f5b6c035850 0x7f5b6c0753e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:37.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:37.627+0000 7f5b7f7fe640 1 --2- 172.21.15.57:0/286214753 >> [v2:172.21.15.179:6832/137854273,v1:172.21.15.179:6833/137854273] conn(0x7f5b6c035850 0x7f5b6c0753e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f5b70007fd0 tx=0x7f5b70007d20 comp rx=0 tx=0).ready entity=mgr.6592 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:37.637 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:37.635+0000 7f308cb74640 -1 mgr.server reply reply (2) No such file or directory SnapSchedule for snap_test_dir1 not found 2023-10-19T13:00:37.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:37.635+0000 7f5b67fff640 1 -- 172.21.15.57:0/286214753 <== mgr.6592 v2:172.21.15.179:6832/137854273 1 ==== mgr_command_reply(tid 0: -2 SnapSchedule for snap_test_dir1 not found) v1 ==== 49+0+0 (secure 0 0 0) 0x7f5b6c035c30 con 0x7f5b6c035850 2023-10-19T13:00:37.638 INFO:teuthology.orchestra.run.smithi057.stderr:Error ENOENT: SnapSchedule for snap_test_dir1 not found 2023-10-19T13:00:37.643 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-10-19T13:00:37.644 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:00:37.644 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2023-10-19T13:00:37.665 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-19T13:00:37.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:37.816+0000 7fe323056640 1 -- 172.21.15.57:0/2610367676 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe31c06d2d0 msgr2=0x7fe31c06d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:37.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:37.816+0000 7fe323056640 1 --2- 172.21.15.57:0/2610367676 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe31c06d2d0 0x7fe31c06d6b0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fe30c009830 tx=0x7fe30c02bd60 comp rx=0 tx=0).stop 2023-10-19T13:00:37.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:37.816+0000 7fe323056640 1 -- 172.21.15.57:0/2610367676 shutdown_connections 2023-10-19T13:00:37.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:37.816+0000 7fe323056640 1 --2- 172.21.15.57:0/2610367676 >> v2:172.21.15.179:3300/0 conn(0x7fe31c0ad370 0x7fe31c0ad800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:37.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:37.816+0000 7fe323056640 1 --2- 172.21.15.57:0/2610367676 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe31c06d2d0 0x7fe31c06d6b0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:37.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:37.816+0000 7fe323056640 1 -- 172.21.15.57:0/2610367676 >> 172.21.15.57:0/2610367676 conn(0x7fe31c0b0670 msgr2=0x7fe31c0b0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:37.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:37.816+0000 7fe323056640 1 -- 172.21.15.57:0/2610367676 shutdown_connections 2023-10-19T13:00:37.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:37.816+0000 7fe323056640 1 -- 172.21.15.57:0/2610367676 wait complete. 2023-10-19T13:00:37.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:37.817+0000 7fe323056640 1 Processor -- start 2023-10-19T13:00:37.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:37.817+0000 7fe323056640 1 -- start start 2023-10-19T13:00:37.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:37.818+0000 7fe323056640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe31c06d2d0 0x7fe31c0c3280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:37.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:37.818+0000 7fe323056640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe31c06f0a0 0x7fe31c151110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:37.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:37.818+0000 7fe323056640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe31c0ad370 0x7fe31c15af70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:37.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:37.818+0000 7fe323056640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe31c0c37c0 con 0x7fe31c0ad370 2023-10-19T13:00:37.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:37.818+0000 7fe323056640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe31c0c79b0 con 0x7fe31c06f0a0 2023-10-19T13:00:37.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:37.818+0000 7fe323056640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe31c0c7c90 con 0x7fe31c06d2d0 2023-10-19T13:00:37.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:37.818+0000 7fe3215f3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe31c0ad370 0x7fe31c15af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:37.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:37.818+0000 7fe3215f3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe31c0ad370 0x7fe31c15af70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:47524/0 (socket says 172.21.15.57:47524) 2023-10-19T13:00:37.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:37.818+0000 7fe3215f3640 1 -- 172.21.15.57:0/629224725 learned_addr learned my addr 172.21.15.57:0/629224725 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:37.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:37.818+0000 7fe320df2640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe31c06d2d0 0x7fe31c0c3280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:37.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:37.818+0000 7fe31bfff640 1 --2- 172.21.15.57:0/629224725 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe31c06f0a0 0x7fe31c151110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:37.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:37.818+0000 7fe3215f3640 1 -- 172.21.15.57:0/629224725 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe31c06d2d0 msgr2=0x7fe31c0c3280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:37.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:37.818+0000 7fe3215f3640 1 --2- 172.21.15.57:0/629224725 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe31c06d2d0 0x7fe31c0c3280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:37.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:37.818+0000 7fe3215f3640 1 -- 172.21.15.57:0/629224725 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe31c06f0a0 msgr2=0x7fe31c151110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:37.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:37.818+0000 7fe3215f3640 1 --2- 172.21.15.57:0/629224725 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe31c06f0a0 0x7fe31c151110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:37.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:37.818+0000 7fe3215f3640 1 -- 172.21.15.57:0/629224725 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe314009840 con 0x7fe31c0ad370 2023-10-19T13:00:37.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:37.818+0000 7fe3215f3640 1 --2- 172.21.15.57:0/629224725 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe31c0ad370 0x7fe31c15af70 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7fe31400ec20 tx=0x7fe31400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:37.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:37.819+0000 7fe3197fa640 1 -- 172.21.15.57:0/629224725 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe314009aa0 con 0x7fe31c0ad370 2023-10-19T13:00:37.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:37.819+0000 7fe323056640 1 -- 172.21.15.57:0/629224725 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe30c009490 con 0x7fe31c0ad370 2023-10-19T13:00:37.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:37.819+0000 7fe3197fa640 1 -- 172.21.15.57:0/629224725 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fe314003c60 con 0x7fe31c0ad370 2023-10-19T13:00:37.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:37.819+0000 7fe3197fa640 1 -- 172.21.15.57:0/629224725 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe314004020 con 0x7fe31c0ad370 2023-10-19T13:00:37.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:37.819+0000 7fe323056640 1 -- 172.21.15.57:0/629224725 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe31c0c8250 con 0x7fe31c0ad370 2023-10-19T13:00:37.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:37.820+0000 7fe323056640 1 -- 172.21.15.57:0/629224725 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe31c0ce1d0 con 0x7fe31c0ad370 2023-10-19T13:00:37.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:37.820+0000 7fe3197fa640 1 -- 172.21.15.57:0/629224725 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 35) v1 ==== 89084+0+0 (secure 0 0 0) 0x7fe3140041a0 con 0x7fe31c0ad370 2023-10-19T13:00:37.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:37.820+0000 7fe3197fa640 1 --2- 172.21.15.57:0/629224725 >> [v2:172.21.15.179:6832/137854273,v1:172.21.15.179:6833/137854273] conn(0x7fe30806ace0 0x7fe30806d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:37.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:37.821+0000 7fe3197fa640 1 -- 172.21.15.57:0/629224725 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6401+0+0 (secure 0 0 0) 0x7fe314014070 con 0x7fe31c0ad370 2023-10-19T13:00:37.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:37.821+0000 7fe320df2640 1 --2- 172.21.15.57:0/629224725 >> [v2:172.21.15.179:6832/137854273,v1:172.21.15.179:6833/137854273] conn(0x7fe30806ace0 0x7fe30806d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:37.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:37.821+0000 7fe320df2640 1 --2- 172.21.15.57:0/629224725 >> [v2:172.21.15.179:6832/137854273,v1:172.21.15.179:6833/137854273] conn(0x7fe30806ace0 0x7fe30806d1a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fe30c036000 tx=0x7fe30c004730 comp rx=0 tx=0).ready entity=mgr.6592 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:37.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:37.825+0000 7fe3197fa640 1 -- 172.21.15.57:0/629224725 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe3140556e0 con 0x7fe31c0ad370 2023-10-19T13:00:38.049 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:38.046+0000 7fe323056640 1 -- 172.21.15.57:0/629224725 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7fe31c0690e0 con 0x7fe31c0ad370 2023-10-19T13:00:38.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:38.675+0000 7fe3197fa640 1 -- 172.21.15.57:0/629224725 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v36) v1 ==== 93+0+0 (secure 0 0 0) 0x7fe314058d00 con 0x7fe31c0ad370 2023-10-19T13:00:38.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:38.679+0000 7fe323056640 1 -- 172.21.15.57:0/629224725 >> [v2:172.21.15.179:6832/137854273,v1:172.21.15.179:6833/137854273] conn(0x7fe30806ace0 msgr2=0x7fe30806d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:38.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:38.679+0000 7fe323056640 1 --2- 172.21.15.57:0/629224725 >> [v2:172.21.15.179:6832/137854273,v1:172.21.15.179:6833/137854273] conn(0x7fe30806ace0 0x7fe30806d1a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fe30c036000 tx=0x7fe30c004730 comp rx=0 tx=0).stop 2023-10-19T13:00:38.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:38.679+0000 7fe323056640 1 -- 172.21.15.57:0/629224725 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe31c0ad370 msgr2=0x7fe31c15af70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:38.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:38.679+0000 7fe3197fa640 1 -- 172.21.15.57:0/629224725 <== mon.0 v2:172.21.15.57:3300/0 8 ==== mgrmap(e 36) v1 ==== 89332+0+0 (secure 0 0 0) 0x7fe314055ab0 con 0x7fe31c0ad370 2023-10-19T13:00:38.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:38.679+0000 7fe323056640 1 --2- 172.21.15.57:0/629224725 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe31c0ad370 0x7fe31c15af70 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7fe31400ec20 tx=0x7fe31400c6a0 comp rx=0 tx=0).stop 2023-10-19T13:00:38.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:38.679+0000 7fe323056640 1 -- 172.21.15.57:0/629224725 shutdown_connections 2023-10-19T13:00:38.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:38.679+0000 7fe323056640 1 --2- 172.21.15.57:0/629224725 >> [v2:172.21.15.179:6832/137854273,v1:172.21.15.179:6833/137854273] conn(0x7fe30806ace0 0x7fe30806d1a0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:38.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:38.679+0000 7fe323056640 1 --2- 172.21.15.57:0/629224725 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe31c0ad370 0x7fe31c15af70 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:38.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:38.679+0000 7fe323056640 1 --2- 172.21.15.57:0/629224725 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe31c06f0a0 0x7fe31c151110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:38.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:38.679+0000 7fe323056640 1 --2- 172.21.15.57:0/629224725 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe31c06d2d0 0x7fe31c0c3280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:38.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:38.679+0000 7fe323056640 1 -- 172.21.15.57:0/629224725 >> 172.21.15.57:0/629224725 conn(0x7fe31c0b0670 msgr2=0x7fe31c0aecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:38.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:38.680+0000 7fe323056640 1 -- 172.21.15.57:0/629224725 shutdown_connections 2023-10-19T13:00:38.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:38.680+0000 7fe323056640 1 -- 172.21.15.57:0/629224725 wait complete. 2023-10-19T13:00:38.695 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-19T13:00:38.696 DEBUG:teuthology.orchestra.run.smithi057:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-19T13:00:38.738 DEBUG:teuthology.orchestra.run.smithi179:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-19T13:00:38.781 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-19T13:00:38.781 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T13:00:38.782 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T13:00:38.797 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T13:00:38.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:38.945+0000 7f9001db6640 1 -- 172.21.15.57:0/2858384009 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ffc15b200 msgr2=0x7f8ffc0b00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:38.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:38.945+0000 7f9001db6640 1 --2- 172.21.15.57:0/2858384009 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ffc15b200 0x7f8ffc0b00f0 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f8fec005fd0 tx=0x7f8fec02bea0 comp rx=0 tx=0).stop 2023-10-19T13:00:38.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:38.945+0000 7f9001db6640 1 -- 172.21.15.57:0/2858384009 shutdown_connections 2023-10-19T13:00:38.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:38.945+0000 7f9001db6640 1 --2- 172.21.15.57:0/2858384009 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8ffc0cf2a0 0x7f8ffc0cf730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:38.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:38.945+0000 7f9001db6640 1 --2- 172.21.15.57:0/2858384009 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ffc15b200 0x7f8ffc0b00f0 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:38.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:38.945+0000 7f9001db6640 1 -- 172.21.15.57:0/2858384009 >> 172.21.15.57:0/2858384009 conn(0x7f8ffc08a650 msgr2=0x7f8ffc099f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:38.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:38.945+0000 7f9001db6640 1 -- 172.21.15.57:0/2858384009 shutdown_connections 2023-10-19T13:00:38.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:38.945+0000 7f9001db6640 1 -- 172.21.15.57:0/2858384009 wait complete. 2023-10-19T13:00:38.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:38.947+0000 7f9001db6640 1 Processor -- start 2023-10-19T13:00:38.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:38.947+0000 7f9001db6640 1 -- start start 2023-10-19T13:00:38.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:38.947+0000 7f9001db6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ffc0cf2a0 0x7f8ffc06b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:38.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:38.947+0000 7f9001db6640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8ffc15b200 0x7f8ffc0c47d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:38.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:38.948+0000 7f9001db6640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8ffc15eba0 0x7f8ffc0c69f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:38.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:38.948+0000 7f9001db6640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ffc1f56e0 con 0x7f8ffc0cf2a0 2023-10-19T13:00:38.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:38.948+0000 7f9001db6640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ffc0c5220 con 0x7f8ffc15eba0 2023-10-19T13:00:38.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:38.948+0000 7f9001db6640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ffc1f5090 con 0x7f8ffc15b200 2023-10-19T13:00:38.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:38.948+0000 7f8ffb7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ffc0cf2a0 0x7f8ffc06b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:38.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:38.948+0000 7f8ffb7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ffc0cf2a0 0x7f8ffc06b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:47560/0 (socket says 172.21.15.57:47560) 2023-10-19T13:00:38.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:38.948+0000 7f8ffb7fe640 1 -- 172.21.15.57:0/3792278338 learned_addr learned my addr 172.21.15.57:0/3792278338 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:38.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:38.948+0000 7f8ffbfff640 1 --2- 172.21.15.57:0/3792278338 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8ffc15eba0 0x7f8ffc0c69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:38.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:38.948+0000 7f8ffaffd640 1 --2- 172.21.15.57:0/3792278338 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8ffc15b200 0x7f8ffc0c47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:38.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:38.948+0000 7f8ffb7fe640 1 -- 172.21.15.57:0/3792278338 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8ffc15b200 msgr2=0x7f8ffc0c47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:38.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:38.948+0000 7f8ffb7fe640 1 --2- 172.21.15.57:0/3792278338 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8ffc15b200 0x7f8ffc0c47d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:38.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:38.948+0000 7f8ffb7fe640 1 -- 172.21.15.57:0/3792278338 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8ffc15eba0 msgr2=0x7f8ffc0c69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:38.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:38.948+0000 7f8ffb7fe640 1 --2- 172.21.15.57:0/3792278338 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8ffc15eba0 0x7f8ffc0c69f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:38.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:38.948+0000 7f8ffb7fe640 1 -- 172.21.15.57:0/3792278338 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fec009560 con 0x7f8ffc0cf2a0 2023-10-19T13:00:38.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:38.948+0000 7f8ffb7fe640 1 --2- 172.21.15.57:0/3792278338 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ffc0cf2a0 0x7f8ffc06b900 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f8fec009850 tx=0x7f8fec0038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:38.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:38.948+0000 7f8ff3fff640 1 -- 172.21.15.57:0/3792278338 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fec005d50 con 0x7f8ffc0cf2a0 2023-10-19T13:00:38.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:38.948+0000 7f8ff3fff640 1 -- 172.21.15.57:0/3792278338 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f8fec003d90 con 0x7f8ffc0cf2a0 2023-10-19T13:00:38.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:38.948+0000 7f9001db6640 1 -- 172.21.15.57:0/3792278338 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ffc1f5370 con 0x7f8ffc0cf2a0 2023-10-19T13:00:38.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:38.948+0000 7f8ff3fff640 1 -- 172.21.15.57:0/3792278338 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fec03d6a0 con 0x7f8ffc0cf2a0 2023-10-19T13:00:38.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:38.948+0000 7f9001db6640 1 -- 172.21.15.57:0/3792278338 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ffc150330 con 0x7f8ffc0cf2a0 2023-10-19T13:00:38.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:38.949+0000 7f9001db6640 1 -- 172.21.15.57:0/3792278338 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ffc1f5530 con 0x7f8ffc0cf2a0 2023-10-19T13:00:38.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:38.950+0000 7f8ff3fff640 1 -- 172.21.15.57:0/3792278338 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 36) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f8fec03d800 con 0x7f8ffc0cf2a0 2023-10-19T13:00:38.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:38.951+0000 7f8ff3fff640 1 --2- 172.21.15.57:0/3792278338 >> [v2:172.21.15.179:6832/137854273,v1:172.21.15.179:6833/137854273] conn(0x7f8fdc06af60 0x7f8fdc06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:38.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:38.951+0000 7f8ffaffd640 1 -- 172.21.15.57:0/3792278338 >> [v2:172.21.15.179:6832/137854273,v1:172.21.15.179:6833/137854273] conn(0x7f8fdc06af60 msgr2=0x7f8fdc06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/137854273 2023-10-19T13:00:38.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:38.951+0000 7f8ff3fff640 1 -- 172.21.15.57:0/3792278338 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f8fec0ac4f0 con 0x7f8ffc0cf2a0 2023-10-19T13:00:38.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:38.951+0000 7f8ffaffd640 1 --2- 172.21.15.57:0/3792278338 >> [v2:172.21.15.179:6832/137854273,v1:172.21.15.179:6833/137854273] conn(0x7f8fdc06af60 0x7f8fdc06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T13:00:38.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:38.956+0000 7f8ff3fff640 1 -- 172.21.15.57:0/3792278338 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8fec077280 con 0x7f8ffc0cf2a0 2023-10-19T13:00:39.064 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:39.062+0000 7f842ea41040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-19T13:00:39.078 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:00:39.076+0000 7f918bc30040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-19T13:00:39.109 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:39.106+0000 7f842ea41040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-19T13:00:39.126 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:00:39.123+0000 7f918bc30040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-19T13:00:39.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.135+0000 7f9001db6640 1 -- 172.21.15.57:0/3792278338 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8ffc1f51d0 con 0x7f8ffc0cf2a0 2023-10-19T13:00:39.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.136+0000 7f8ff3fff640 1 -- 172.21.15.57:0/3792278338 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+17280 (secure 0 0 0) 0x7f8ffc1f51d0 con 0x7f8ffc0cf2a0 2023-10-19T13:00:39.139 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:00:39.139 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":88,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T13:00:36.611926+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2023-10-19T13:00:18.452198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2023-10-19T13:00:19.531525+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.179:0/4093253624":"2023-10-20T13:00:36.611894+0000","172.21.15.179:6833/3815793798":"2023-10-20T13:00:36.611894+0000","172.21.15.179:0/2495886194":"2023-10-20T13:00:36.611894+0000","172.21.15.179:6832/3815793798":"2023-10-20T13:00:36.611894+0000","172.21.15.179:0/2414172521":"2023-10-20T13:00:36.611894+0000","172.21.15.179:0/2931605339":"2023-10-20T13:00:36.611894+0000","172.21.15.179:0/1988099910":"2023-10-20T13:00:36.611894+0000","172.21.15.179:0/4264007972":"2023-10-20T13:00:36.611894+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T13:00:39.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.138+0000 7f9001db6640 1 -- 172.21.15.57:0/3792278338 >> [v2:172.21.15.179:6832/137854273,v1:172.21.15.179:6833/137854273] conn(0x7f8fdc06af60 msgr2=0x7f8fdc06d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T13:00:39.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.138+0000 7f9001db6640 1 --2- 172.21.15.57:0/3792278338 >> [v2:172.21.15.179:6832/137854273,v1:172.21.15.179:6833/137854273] conn(0x7f8fdc06af60 0x7f8fdc06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:39.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.138+0000 7f9001db6640 1 -- 172.21.15.57:0/3792278338 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ffc0cf2a0 msgr2=0x7f8ffc06b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:39.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.138+0000 7f9001db6640 1 --2- 172.21.15.57:0/3792278338 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ffc0cf2a0 0x7f8ffc06b900 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f8fec009850 tx=0x7f8fec0038a0 comp rx=0 tx=0).stop 2023-10-19T13:00:39.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.138+0000 7f9001db6640 1 -- 172.21.15.57:0/3792278338 shutdown_connections 2023-10-19T13:00:39.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.138+0000 7f9001db6640 1 --2- 172.21.15.57:0/3792278338 >> [v2:172.21.15.179:6832/137854273,v1:172.21.15.179:6833/137854273] conn(0x7f8fdc06af60 0x7f8fdc06d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:39.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.138+0000 7f9001db6640 1 --2- 172.21.15.57:0/3792278338 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8ffc15eba0 0x7f8ffc0c69f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:39.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.138+0000 7f9001db6640 1 --2- 172.21.15.57:0/3792278338 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8ffc15b200 0x7f8ffc0c47d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:39.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.138+0000 7f9001db6640 1 --2- 172.21.15.57:0/3792278338 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ffc0cf2a0 0x7f8ffc06b900 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:39.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.138+0000 7f9001db6640 1 -- 172.21.15.57:0/3792278338 >> 172.21.15.57:0/3792278338 conn(0x7f8ffc08a650 msgr2=0x7f8ffc0d0b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:39.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.139+0000 7f9001db6640 1 -- 172.21.15.57:0/3792278338 shutdown_connections 2023-10-19T13:00:39.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.139+0000 7f9001db6640 1 -- 172.21.15.57:0/3792278338 wait complete. 2023-10-19T13:00:39.153 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-19T12:44:07.371186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-19T13:00:18.452198+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_data', 'create_time': '2023-10-19T13:00:19.531525+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-19T13:00:39.153 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-19T13:00:39.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.263+0000 7fece6f4c640 1 -- 172.21.15.57:0/2603721559 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fece00ceb40 msgr2=0x7fece00b0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:39.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.263+0000 7fece6f4c640 1 --2- 172.21.15.57:0/2603721559 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fece00ceb40 0x7fece00b0650 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fecd8005fd0 tx=0x7fecd802c050 comp rx=0 tx=0).stop 2023-10-19T13:00:39.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.264+0000 7fece6f4c640 1 -- 172.21.15.57:0/2603721559 shutdown_connections 2023-10-19T13:00:39.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.264+0000 7fece6f4c640 1 --2- 172.21.15.57:0/2603721559 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fece00ceb40 0x7fece00b0650 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:39.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.264+0000 7fece6f4c640 1 --2- 172.21.15.57:0/2603721559 >> v2:172.21.15.179:3300/0 conn(0x7fece015b2a0 0x7fece00d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:39.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.264+0000 7fece6f4c640 1 -- 172.21.15.57:0/2603721559 >> 172.21.15.57:0/2603721559 conn(0x7fece00d0d60 msgr2=0x7fece00d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:39.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.264+0000 7fece6f4c640 1 -- 172.21.15.57:0/2603721559 shutdown_connections 2023-10-19T13:00:39.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.264+0000 7fece6f4c640 1 -- 172.21.15.57:0/2603721559 wait complete. 2023-10-19T13:00:39.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.265+0000 7fece6f4c640 1 Processor -- start 2023-10-19T13:00:39.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.265+0000 7fece6f4c640 1 -- start start 2023-10-19T13:00:39.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.266+0000 7fece6f4c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fece00ceb40 0x7fece00b5330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:39.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.266+0000 7fece6f4c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fece00cf340 0x7fece01573f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:39.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.266+0000 7fece6f4c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fece015b2a0 0x7fece015b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:39.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.266+0000 7fece6f4c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fece00b5870 con 0x7fece015b2a0 2023-10-19T13:00:39.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.266+0000 7fece6f4c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fece00b4c50 con 0x7fece00ceb40 2023-10-19T13:00:39.272 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.266+0000 7fece6f4c640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fece0157930 con 0x7fece00cf340 2023-10-19T13:00:39.272 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.266+0000 7fece54e9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fece015b2a0 0x7fece015b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:39.272 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.266+0000 7fecdffff640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fece00cf340 0x7fece01573f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:39.272 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.266+0000 7fece4ce8640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fece00ceb40 0x7fece00b5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:39.272 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.266+0000 7fecdffff640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fece00cf340 0x7fece01573f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3301/0 says I am v2:172.21.15.57:41252/0 (socket says 172.21.15.57:41252) 2023-10-19T13:00:39.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.266+0000 7fece54e9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fece015b2a0 0x7fece015b6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:47572/0 (socket says 172.21.15.57:47572) 2023-10-19T13:00:39.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.266+0000 7fecdffff640 1 -- 172.21.15.57:0/440474592 learned_addr learned my addr 172.21.15.57:0/440474592 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:39.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.266+0000 7fece54e9640 1 -- 172.21.15.57:0/440474592 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fece00cf340 msgr2=0x7fece01573f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:39.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.266+0000 7fece54e9640 1 --2- 172.21.15.57:0/440474592 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fece00cf340 0x7fece01573f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:39.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.266+0000 7fece54e9640 1 -- 172.21.15.57:0/440474592 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fece00ceb40 msgr2=0x7fece00b5330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:39.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.266+0000 7fece54e9640 1 --2- 172.21.15.57:0/440474592 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fece00ceb40 0x7fece00b5330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:39.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.266+0000 7fece54e9640 1 -- 172.21.15.57:0/440474592 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feccc009490 con 0x7fece015b2a0 2023-10-19T13:00:39.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.266+0000 7fecdffff640 1 --2- 172.21.15.57:0/440474592 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fece00cf340 0x7fece01573f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T13:00:39.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.266+0000 7fece54e9640 1 --2- 172.21.15.57:0/440474592 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fece015b2a0 0x7fece015b6d0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7fecd8009000 tx=0x7fecd80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:39.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.267+0000 7fecdd7fa640 1 -- 172.21.15.57:0/440474592 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecd8003e80 con 0x7fece015b2a0 2023-10-19T13:00:39.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.267+0000 7fecdd7fa640 1 -- 172.21.15.57:0/440474592 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fecd803e460 con 0x7fece015b2a0 2023-10-19T13:00:39.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.267+0000 7fece6f4c640 1 -- 172.21.15.57:0/440474592 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecd8009840 con 0x7fece015b2a0 2023-10-19T13:00:39.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.267+0000 7fecdd7fa640 1 -- 172.21.15.57:0/440474592 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecd803e7d0 con 0x7fece015b2a0 2023-10-19T13:00:39.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.267+0000 7fece6f4c640 1 -- 172.21.15.57:0/440474592 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fece00b0d50 con 0x7fece015b2a0 2023-10-19T13:00:39.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.268+0000 7fece6f4c640 1 -- 172.21.15.57:0/440474592 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fece00690e0 con 0x7fece015b2a0 2023-10-19T13:00:39.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.268+0000 7fecdd7fa640 1 -- 172.21.15.57:0/440474592 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 36) v1 ==== 89332+0+0 (secure 0 0 0) 0x7fecd803e930 con 0x7fece015b2a0 2023-10-19T13:00:39.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.269+0000 7fecdd7fa640 1 --2- 172.21.15.57:0/440474592 >> [v2:172.21.15.179:6832/137854273,v1:172.21.15.179:6833/137854273] conn(0x7fecc406af60 0x7fecc406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:39.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.269+0000 7fece4ce8640 1 -- 172.21.15.57:0/440474592 >> [v2:172.21.15.179:6832/137854273,v1:172.21.15.179:6833/137854273] conn(0x7fecc406af60 msgr2=0x7fecc406d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/137854273 2023-10-19T13:00:39.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.269+0000 7fecdd7fa640 1 -- 172.21.15.57:0/440474592 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6401+0+0 (secure 0 0 0) 0x7fecd80ac840 con 0x7fece015b2a0 2023-10-19T13:00:39.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.269+0000 7fece4ce8640 1 --2- 172.21.15.57:0/440474592 >> [v2:172.21.15.179:6832/137854273,v1:172.21.15.179:6833/137854273] conn(0x7fecc406af60 0x7fecc406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T13:00:39.279 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.277+0000 7fecdd7fa640 1 -- 172.21.15.57:0/440474592 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fecd80775e0 con 0x7fece015b2a0 2023-10-19T13:00:39.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.453+0000 7fece6f4c640 1 -- 172.21.15.57:0/440474592 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fece00b2110 con 0x7fece015b2a0 2023-10-19T13:00:39.456 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.453+0000 7fecdd7fa640 1 -- 172.21.15.57:0/440474592 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) v1 ==== 93+0+10 (secure 0 0 0) 0x7fecd807ac00 con 0x7fece015b2a0 2023-10-19T13:00:39.456 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-19T13:00:39.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.455+0000 7fece6f4c640 1 -- 172.21.15.57:0/440474592 >> [v2:172.21.15.179:6832/137854273,v1:172.21.15.179:6833/137854273] conn(0x7fecc406af60 msgr2=0x7fecc406d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T13:00:39.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.455+0000 7fece6f4c640 1 --2- 172.21.15.57:0/440474592 >> [v2:172.21.15.179:6832/137854273,v1:172.21.15.179:6833/137854273] conn(0x7fecc406af60 0x7fecc406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:39.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.455+0000 7fece6f4c640 1 -- 172.21.15.57:0/440474592 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fece015b2a0 msgr2=0x7fece015b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:39.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.455+0000 7fece6f4c640 1 --2- 172.21.15.57:0/440474592 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fece015b2a0 0x7fece015b6d0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7fecd8009000 tx=0x7fecd80042f0 comp rx=0 tx=0).stop 2023-10-19T13:00:39.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.455+0000 7fece6f4c640 1 -- 172.21.15.57:0/440474592 shutdown_connections 2023-10-19T13:00:39.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.455+0000 7fece6f4c640 1 --2- 172.21.15.57:0/440474592 >> [v2:172.21.15.179:6832/137854273,v1:172.21.15.179:6833/137854273] conn(0x7fecc406af60 0x7fecc406d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:39.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.455+0000 7fece6f4c640 1 --2- 172.21.15.57:0/440474592 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fece015b2a0 0x7fece015b6d0 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:39.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.455+0000 7fece6f4c640 1 --2- 172.21.15.57:0/440474592 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fece00cf340 0x7fece01573f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:39.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.455+0000 7fece6f4c640 1 --2- 172.21.15.57:0/440474592 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fece00ceb40 0x7fece00b5330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:39.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.455+0000 7fece6f4c640 1 -- 172.21.15.57:0/440474592 >> 172.21.15.57:0/440474592 conn(0x7fece00d0d60 msgr2=0x7fece006b440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:39.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.455+0000 7fece6f4c640 1 -- 172.21.15.57:0/440474592 shutdown_connections 2023-10-19T13:00:39.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.455+0000 7fece6f4c640 1 -- 172.21.15.57:0/440474592 wait complete. 2023-10-19T13:00:39.469 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-19T13:00:39.499 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:39.497+0000 7f842ea41040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-19T13:00:39.542 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:39.540+0000 7f842ea41040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-19T13:00:39.548 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:00:39.545+0000 7f918bc30040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-19T13:00:39.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.579+0000 7f6dd24c9640 1 -- 172.21.15.57:0/3553478765 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6dcc15b2a0 msgr2=0x7f6dcc0d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:39.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.579+0000 7f6dd24c9640 1 --2- 172.21.15.57:0/3553478765 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6dcc15b2a0 0x7f6dcc0d1910 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f6dc0005fd0 tx=0x7f6dc002bd90 comp rx=0 tx=0).stop 2023-10-19T13:00:39.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.579+0000 7f6dd24c9640 1 -- 172.21.15.57:0/3553478765 shutdown_connections 2023-10-19T13:00:39.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.579+0000 7f6dd24c9640 1 --2- 172.21.15.57:0/3553478765 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6dcc0ceb40 0x7f6dcc0b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:39.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.579+0000 7f6dd24c9640 1 --2- 172.21.15.57:0/3553478765 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6dcc15b2a0 0x7f6dcc0d1910 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:39.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.579+0000 7f6dd24c9640 1 -- 172.21.15.57:0/3553478765 >> 172.21.15.57:0/3553478765 conn(0x7f6dcc0d0d60 msgr2=0x7f6dcc0d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:39.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.579+0000 7f6dd24c9640 1 -- 172.21.15.57:0/3553478765 shutdown_connections 2023-10-19T13:00:39.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.579+0000 7f6dd24c9640 1 -- 172.21.15.57:0/3553478765 wait complete. 2023-10-19T13:00:39.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.580+0000 7f6dd24c9640 1 Processor -- start 2023-10-19T13:00:39.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.581+0000 7f6dd24c9640 1 -- start start 2023-10-19T13:00:39.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.581+0000 7f6dd24c9640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6dcc0ceb40 0x7f6dcc1582e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:39.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.581+0000 7f6dd24c9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6dcc0cf340 0x7f6dcc15b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:39.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.581+0000 7f6dd24c9640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6dcc15b2a0 0x7f6dcc1e5db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:39.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.581+0000 7f6dd24c9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dcc1e5990 con 0x7f6dcc0cf340 2023-10-19T13:00:39.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.581+0000 7f6dd24c9640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dcc1e4610 con 0x7f6dcc0ceb40 2023-10-19T13:00:39.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.581+0000 7f6dd24c9640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6dcc1e52d0 con 0x7f6dcc15b2a0 2023-10-19T13:00:39.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.581+0000 7f6dcb7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6dcc0cf340 0x7f6dcc15b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:39.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.581+0000 7f6dcb7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6dcc0cf340 0x7f6dcc15b6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:47588/0 (socket says 172.21.15.57:47588) 2023-10-19T13:00:39.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.582+0000 7f6dcb7fe640 1 -- 172.21.15.57:0/3880404019 learned_addr learned my addr 172.21.15.57:0/3880404019 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:39.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.582+0000 7f6dd0a66640 1 --2- 172.21.15.57:0/3880404019 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6dcc15b2a0 0x7f6dcc1e5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:39.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.582+0000 7f6dcbfff640 1 --2- 172.21.15.57:0/3880404019 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6dcc0ceb40 0x7f6dcc1582e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:39.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.582+0000 7f6dcb7fe640 1 -- 172.21.15.57:0/3880404019 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6dcc15b2a0 msgr2=0x7f6dcc1e5db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:39.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.582+0000 7f6dcb7fe640 1 --2- 172.21.15.57:0/3880404019 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6dcc15b2a0 0x7f6dcc1e5db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:39.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.582+0000 7f6dcb7fe640 1 -- 172.21.15.57:0/3880404019 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6dcc0ceb40 msgr2=0x7f6dcc1582e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:39.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.582+0000 7f6dcb7fe640 1 --2- 172.21.15.57:0/3880404019 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6dcc0ceb40 0x7f6dcc1582e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:39.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.582+0000 7f6dcb7fe640 1 -- 172.21.15.57:0/3880404019 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dc0009560 con 0x7f6dcc0cf340 2023-10-19T13:00:39.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.582+0000 7f6dcb7fe640 1 --2- 172.21.15.57:0/3880404019 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6dcc0cf340 0x7f6dcc15b6d0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f6dc0006010 tx=0x7f6dc002db30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:39.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.582+0000 7f6dc8ff9640 1 -- 172.21.15.57:0/3880404019 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dc00035f0 con 0x7f6dcc0cf340 2023-10-19T13:00:39.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.582+0000 7f6dd24c9640 1 -- 172.21.15.57:0/3880404019 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6dcc0b0d50 con 0x7f6dcc0cf340 2023-10-19T13:00:39.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.582+0000 7f6dc8ff9640 1 -- 172.21.15.57:0/3880404019 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f6dc0003840 con 0x7f6dcc0cf340 2023-10-19T13:00:39.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.582+0000 7f6dc8ff9640 1 -- 172.21.15.57:0/3880404019 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dc0003c50 con 0x7f6dcc0cf340 2023-10-19T13:00:39.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.582+0000 7f6dd24c9640 1 -- 172.21.15.57:0/3880404019 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dcc158820 con 0x7f6dcc0cf340 2023-10-19T13:00:39.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.583+0000 7f6dd24c9640 1 -- 172.21.15.57:0/3880404019 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6dcc1e5130 con 0x7f6dcc0cf340 2023-10-19T13:00:39.591 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:39.582+0000 7f842ea41040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-19T13:00:39.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.584+0000 7f6dc8ff9640 1 -- 172.21.15.57:0/3880404019 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 36) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f6dc003e450 con 0x7f6dcc0cf340 2023-10-19T13:00:39.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.584+0000 7f6dc8ff9640 1 --2- 172.21.15.57:0/3880404019 >> [v2:172.21.15.179:6832/137854273,v1:172.21.15.179:6833/137854273] conn(0x7f6db806af60 0x7f6db806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:39.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.584+0000 7f6dc8ff9640 1 -- 172.21.15.57:0/3880404019 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f6dc00ac490 con 0x7f6dcc0cf340 2023-10-19T13:00:39.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.584+0000 7f6dcbfff640 1 -- 172.21.15.57:0/3880404019 >> [v2:172.21.15.179:6832/137854273,v1:172.21.15.179:6833/137854273] conn(0x7f6db806af60 msgr2=0x7f6db806d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/137854273 2023-10-19T13:00:39.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.584+0000 7f6dcbfff640 1 --2- 172.21.15.57:0/3880404019 >> [v2:172.21.15.179:6832/137854273,v1:172.21.15.179:6833/137854273] conn(0x7f6db806af60 0x7f6db806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T13:00:39.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.588+0000 7f6dc8ff9640 1 -- 172.21.15.57:0/3880404019 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6dcc1e5130 con 0x7f6dcc0cf340 2023-10-19T13:00:39.596 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:00:39.594+0000 7f918bc30040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-19T13:00:39.637 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:39.635+0000 7f842ea41040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-19T13:00:39.642 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:00:39.639+0000 7f918bc30040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-19T13:00:39.678 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:39.676+0000 7f842ea41040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-19T13:00:39.697 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:00:39.694+0000 7f918bc30040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-19T13:00:39.741 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:00:39.738+0000 7f918bc30040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-19T13:00:39.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.764+0000 7f6dd24c9640 1 -- 172.21.15.57:0/3880404019 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f6dcc1e5130 con 0x7f6dcc0cf340 2023-10-19T13:00:39.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.765+0000 7f6dc8ff9640 1 -- 172.21.15.57:0/3880404019 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6dcc1e5130 con 0x7f6dcc0cf340 2023-10-19T13:00:39.768 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2023-10-19T13:00:39.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.767+0000 7f6dd24c9640 1 -- 172.21.15.57:0/3880404019 >> [v2:172.21.15.179:6832/137854273,v1:172.21.15.179:6833/137854273] conn(0x7f6db806af60 msgr2=0x7f6db806d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T13:00:39.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.767+0000 7f6dd24c9640 1 --2- 172.21.15.57:0/3880404019 >> [v2:172.21.15.179:6832/137854273,v1:172.21.15.179:6833/137854273] conn(0x7f6db806af60 0x7f6db806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:39.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.767+0000 7f6dd24c9640 1 -- 172.21.15.57:0/3880404019 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6dcc0cf340 msgr2=0x7f6dcc15b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:39.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.767+0000 7f6dd24c9640 1 --2- 172.21.15.57:0/3880404019 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6dcc0cf340 0x7f6dcc15b6d0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f6dc0006010 tx=0x7f6dc002db30 comp rx=0 tx=0).stop 2023-10-19T13:00:39.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.767+0000 7f6dd24c9640 1 -- 172.21.15.57:0/3880404019 shutdown_connections 2023-10-19T13:00:39.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.767+0000 7f6dd24c9640 1 --2- 172.21.15.57:0/3880404019 >> [v2:172.21.15.179:6832/137854273,v1:172.21.15.179:6833/137854273] conn(0x7f6db806af60 0x7f6db806d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:39.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.767+0000 7f6dd24c9640 1 --2- 172.21.15.57:0/3880404019 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6dcc15b2a0 0x7f6dcc1e5db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:39.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.767+0000 7f6dd24c9640 1 --2- 172.21.15.57:0/3880404019 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6dcc0cf340 0x7f6dcc15b6d0 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:39.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.767+0000 7f6dd24c9640 1 --2- 172.21.15.57:0/3880404019 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6dcc0ceb40 0x7f6dcc1582e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:39.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.767+0000 7f6dd24c9640 1 -- 172.21.15.57:0/3880404019 >> 172.21.15.57:0/3880404019 conn(0x7f6dcc0d0d60 msgr2=0x7f6dcc06b7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:39.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.767+0000 7f6dd24c9640 1 -- 172.21.15.57:0/3880404019 shutdown_connections 2023-10-19T13:00:39.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.767+0000 7f6dd24c9640 1 -- 172.21.15.57:0/3880404019 wait complete. 2023-10-19T13:00:39.777 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:39.775+0000 7f842ea41040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-19T13:00:39.781 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-19T13:00:39.829 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:39.827+0000 7f842ea41040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-19T13:00:39.846 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:00:39.843+0000 7f918bc30040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-19T13:00:39.870 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:39.868+0000 7f842ea41040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-19T13:00:39.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.891+0000 7f7df3c39640 1 -- 172.21.15.57:0/2411678205 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7dec0cf340 msgr2=0x7f7dec0d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:39.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.891+0000 7f7df3c39640 1 --2- 172.21.15.57:0/2411678205 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7dec0cf340 0x7f7dec0d1910 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f7dec06c850 tx=0x7f7de802bd10 comp rx=0 tx=0).stop 2023-10-19T13:00:39.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.891+0000 7f7df3c39640 1 -- 172.21.15.57:0/2411678205 shutdown_connections 2023-10-19T13:00:39.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.891+0000 7f7df3c39640 1 --2- 172.21.15.57:0/2411678205 >> v2:172.21.15.179:3300/0 conn(0x7f7dec0ceb40 0x7f7dec0b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:39.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.891+0000 7f7df3c39640 1 --2- 172.21.15.57:0/2411678205 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7dec15b2a0 0x7f7dec0cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:39.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.891+0000 7f7df3c39640 1 --2- 172.21.15.57:0/2411678205 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7dec0cf340 0x7f7dec0d1910 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:39.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.891+0000 7f7df3c39640 1 -- 172.21.15.57:0/2411678205 >> 172.21.15.57:0/2411678205 conn(0x7f7dec0d0d60 msgr2=0x7f7dec0d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:39.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.891+0000 7f7df3c39640 1 -- 172.21.15.57:0/2411678205 shutdown_connections 2023-10-19T13:00:39.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.891+0000 7f7df3c39640 1 -- 172.21.15.57:0/2411678205 wait complete. 2023-10-19T13:00:39.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.892+0000 7f7df3c39640 1 Processor -- start 2023-10-19T13:00:39.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.892+0000 7f7df3c39640 1 -- start start 2023-10-19T13:00:39.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.893+0000 7f7df3c39640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7dec0ceb40 0x7f7dec1f1110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:39.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.893+0000 7f7df3c39640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7dec0cf340 0x7f7dec0b5af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:39.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.893+0000 7f7df3c39640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7dec15b2a0 0x7f7dec157270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:39.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.893+0000 7f7df3c39640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dec153b60 con 0x7f7dec0ceb40 2023-10-19T13:00:39.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.893+0000 7f7df3c39640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dec1585e0 con 0x7f7dec0cf340 2023-10-19T13:00:39.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.893+0000 7f7df3c39640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dec1577b0 con 0x7f7dec15b2a0 2023-10-19T13:00:39.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.893+0000 7f7df19d5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7dec0ceb40 0x7f7dec1f1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:39.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.893+0000 7f7df19d5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7dec0ceb40 0x7f7dec1f1110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:47608/0 (socket says 172.21.15.57:47608) 2023-10-19T13:00:39.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.893+0000 7f7df19d5640 1 -- 172.21.15.57:0/3358044176 learned_addr learned my addr 172.21.15.57:0/3358044176 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:39.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.893+0000 7f7df11d4640 1 --2- 172.21.15.57:0/3358044176 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7dec0cf340 0x7f7dec0b5af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:39.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.893+0000 7f7df21d6640 1 --2- 172.21.15.57:0/3358044176 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7dec15b2a0 0x7f7dec157270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:39.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.894+0000 7f7df19d5640 1 -- 172.21.15.57:0/3358044176 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7dec15b2a0 msgr2=0x7f7dec157270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:39.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.894+0000 7f7df19d5640 1 --2- 172.21.15.57:0/3358044176 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7dec15b2a0 0x7f7dec157270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:39.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.894+0000 7f7df19d5640 1 -- 172.21.15.57:0/3358044176 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7dec0cf340 msgr2=0x7f7dec0b5af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:39.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.894+0000 7f7df19d5640 1 --2- 172.21.15.57:0/3358044176 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7dec0cf340 0x7f7dec0b5af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:39.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.894+0000 7f7df19d5640 1 -- 172.21.15.57:0/3358044176 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7de8009560 con 0x7f7dec0ceb40 2023-10-19T13:00:39.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.894+0000 7f7df19d5640 1 --2- 172.21.15.57:0/3358044176 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7dec0ceb40 0x7f7dec1f1110 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f7de80099c0 tx=0x7f7de802dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:39.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.894+0000 7f7de27fc640 1 -- 172.21.15.57:0/3358044176 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7de8003800 con 0x7f7dec0ceb40 2023-10-19T13:00:39.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.894+0000 7f7de27fc640 1 -- 172.21.15.57:0/3358044176 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f7de8003a50 con 0x7f7dec0ceb40 2023-10-19T13:00:39.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.894+0000 7f7de27fc640 1 -- 172.21.15.57:0/3358044176 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7de8003e60 con 0x7f7dec0ceb40 2023-10-19T13:00:39.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.894+0000 7f7df3c39640 1 -- 172.21.15.57:0/3358044176 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dec157a60 con 0x7f7dec0ceb40 2023-10-19T13:00:39.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.894+0000 7f7df3c39640 1 -- 172.21.15.57:0/3358044176 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dec158020 con 0x7f7dec0ceb40 2023-10-19T13:00:39.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.895+0000 7f7df3c39640 1 -- 172.21.15.57:0/3358044176 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dec0690e0 con 0x7f7dec0ceb40 2023-10-19T13:00:39.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.896+0000 7f7de27fc640 1 -- 172.21.15.57:0/3358044176 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 36) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f7de803e450 con 0x7f7dec0ceb40 2023-10-19T13:00:39.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.897+0000 7f7de27fc640 1 --2- 172.21.15.57:0/3358044176 >> [v2:172.21.15.179:6832/137854273,v1:172.21.15.179:6833/137854273] conn(0x7f7dd006af60 0x7f7dd006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:39.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.897+0000 7f7de27fc640 1 -- 172.21.15.57:0/3358044176 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f7de80ac350 con 0x7f7dec0ceb40 2023-10-19T13:00:39.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.897+0000 7f7df11d4640 1 -- 172.21.15.57:0/3358044176 >> [v2:172.21.15.179:6832/137854273,v1:172.21.15.179:6833/137854273] conn(0x7f7dd006af60 msgr2=0x7f7dd006d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/137854273 2023-10-19T13:00:39.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.897+0000 7f7df11d4640 1 --2- 172.21.15.57:0/3358044176 >> [v2:172.21.15.179:6832/137854273,v1:172.21.15.179:6833/137854273] conn(0x7f7dd006af60 0x7f7dd006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T13:00:39.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:39.901+0000 7f7de27fc640 1 -- 172.21.15.57:0/3358044176 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7de8077290 con 0x7f7dec0ceb40 2023-10-19T13:00:39.905 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:00:39.901+0000 7f918bc30040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-19T13:00:39.919 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:39.916+0000 7f842ea41040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-19T13:00:39.948 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:00:39.946+0000 7f918bc30040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-19T13:00:39.961 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:39.958+0000 7f842ea41040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-19T13:00:39.999 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:00:39.997+0000 7f918bc30040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-19T13:00:40.044 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:00:40.041+0000 7f918bc30040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-19T13:00:40.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.077+0000 7f7df3c39640 1 -- 172.21.15.57:0/3358044176 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7dec0d0b40 con 0x7f7dec0ceb40 2023-10-19T13:00:40.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.078+0000 7f7de27fc640 1 -- 172.21.15.57:0/3358044176 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7de807a8b0 con 0x7f7dec0ceb40 2023-10-19T13:00:40.081 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 64 2023-10-19T13:00:40.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.080+0000 7f7df3c39640 1 -- 172.21.15.57:0/3358044176 >> [v2:172.21.15.179:6832/137854273,v1:172.21.15.179:6833/137854273] conn(0x7f7dd006af60 msgr2=0x7f7dd006d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T13:00:40.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.080+0000 7f7df3c39640 1 --2- 172.21.15.57:0/3358044176 >> [v2:172.21.15.179:6832/137854273,v1:172.21.15.179:6833/137854273] conn(0x7f7dd006af60 0x7f7dd006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:40.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.080+0000 7f7df3c39640 1 -- 172.21.15.57:0/3358044176 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7dec0ceb40 msgr2=0x7f7dec1f1110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:40.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.080+0000 7f7df3c39640 1 --2- 172.21.15.57:0/3358044176 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7dec0ceb40 0x7f7dec1f1110 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f7de80099c0 tx=0x7f7de802dab0 comp rx=0 tx=0).stop 2023-10-19T13:00:40.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.080+0000 7f7df3c39640 1 -- 172.21.15.57:0/3358044176 shutdown_connections 2023-10-19T13:00:40.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.080+0000 7f7df3c39640 1 --2- 172.21.15.57:0/3358044176 >> [v2:172.21.15.179:6832/137854273,v1:172.21.15.179:6833/137854273] conn(0x7f7dd006af60 0x7f7dd006d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:40.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.080+0000 7f7df3c39640 1 --2- 172.21.15.57:0/3358044176 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7dec15b2a0 0x7f7dec157270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:40.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.080+0000 7f7df3c39640 1 --2- 172.21.15.57:0/3358044176 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7dec0cf340 0x7f7dec0b5af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:40.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.080+0000 7f7df3c39640 1 --2- 172.21.15.57:0/3358044176 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7dec0ceb40 0x7f7dec1f1110 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:40.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.080+0000 7f7df3c39640 1 -- 172.21.15.57:0/3358044176 >> 172.21.15.57:0/3358044176 conn(0x7f7dec0d0d60 msgr2=0x7f7dec0b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:40.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.080+0000 7f7df3c39640 1 -- 172.21.15.57:0/3358044176 shutdown_connections 2023-10-19T13:00:40.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.080+0000 7f7df3c39640 1 -- 172.21.15.57:0/3358044176 wait complete. 2023-10-19T13:00:40.094 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-19T13:00:40.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.202+0000 7f27e5fa9640 1 -- 172.21.15.57:0/4293870281 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f27e00cf340 msgr2=0x7f27e00d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:40.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.202+0000 7f27e5fa9640 1 --2- 172.21.15.57:0/4293870281 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f27e00cf340 0x7f27e00d1910 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f27e008a120 tx=0x7f27c802bdb0 comp rx=0 tx=0).stop 2023-10-19T13:00:40.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.202+0000 7f27e5fa9640 1 -- 172.21.15.57:0/4293870281 shutdown_connections 2023-10-19T13:00:40.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.202+0000 7f27e5fa9640 1 --2- 172.21.15.57:0/4293870281 >> v2:172.21.15.179:3300/0 conn(0x7f27e015b2a0 0x7f27e00cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:40.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.202+0000 7f27e5fa9640 1 --2- 172.21.15.57:0/4293870281 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f27e00cf340 0x7f27e00d1910 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:40.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.202+0000 7f27e5fa9640 1 -- 172.21.15.57:0/4293870281 >> 172.21.15.57:0/4293870281 conn(0x7f27e00d0d60 msgr2=0x7f27e00d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:40.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.202+0000 7f27e5fa9640 1 -- 172.21.15.57:0/4293870281 shutdown_connections 2023-10-19T13:00:40.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.202+0000 7f27e5fa9640 1 -- 172.21.15.57:0/4293870281 wait complete. 2023-10-19T13:00:40.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.203+0000 7f27e5fa9640 1 Processor -- start 2023-10-19T13:00:40.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.203+0000 7f27e5fa9640 1 -- start start 2023-10-19T13:00:40.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.204+0000 7f27e5fa9640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f27e00ceb40 0x7f27e0157eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:40.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.204+0000 7f27e5fa9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f27e00cf340 0x7f27e00cad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:40.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.204+0000 7f27e5fa9640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f27e015b2a0 0x7f27e00ca130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:40.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.204+0000 7f27e5fa9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27e01e64d0 con 0x7f27e00cf340 2023-10-19T13:00:40.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.204+0000 7f27e5fa9640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27e00c9650 con 0x7f27e00ceb40 2023-10-19T13:00:40.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.204+0000 7f27e5fa9640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27e00c9930 con 0x7f27e015b2a0 2023-10-19T13:00:40.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.204+0000 7f27deffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f27e00cf340 0x7f27e00cad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:40.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.204+0000 7f27deffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f27e00cf340 0x7f27e00cad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:47612/0 (socket says 172.21.15.57:47612) 2023-10-19T13:00:40.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.204+0000 7f27deffd640 1 -- 172.21.15.57:0/2397153476 learned_addr learned my addr 172.21.15.57:0/2397153476 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:40.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.204+0000 7f27df7fe640 1 --2- 172.21.15.57:0/2397153476 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f27e00ceb40 0x7f27e0157eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:40.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.204+0000 7f27dffff640 1 --2- 172.21.15.57:0/2397153476 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f27e015b2a0 0x7f27e00ca130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:40.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.205+0000 7f27deffd640 1 -- 172.21.15.57:0/2397153476 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f27e015b2a0 msgr2=0x7f27e00ca130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:40.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.205+0000 7f27deffd640 1 --2- 172.21.15.57:0/2397153476 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f27e015b2a0 0x7f27e00ca130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:40.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.205+0000 7f27deffd640 1 -- 172.21.15.57:0/2397153476 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f27e00ceb40 msgr2=0x7f27e0157eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:40.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.205+0000 7f27deffd640 1 --2- 172.21.15.57:0/2397153476 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f27e00ceb40 0x7f27e0157eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:40.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.205+0000 7f27deffd640 1 -- 172.21.15.57:0/2397153476 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27c8009560 con 0x7f27e00cf340 2023-10-19T13:00:40.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.205+0000 7f27deffd640 1 --2- 172.21.15.57:0/2397153476 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f27e00cf340 0x7f27e00cad70 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f27d000e790 tx=0x7f27d000ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:40.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.205+0000 7f27d7fff640 1 -- 172.21.15.57:0/2397153476 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27d0009580 con 0x7f27e00cf340 2023-10-19T13:00:40.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.205+0000 7f27d7fff640 1 -- 172.21.15.57:0/2397153476 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f27d000cdb0 con 0x7f27e00cf340 2023-10-19T13:00:40.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.205+0000 7f27e5fa9640 1 -- 172.21.15.57:0/2397153476 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27e00c9c10 con 0x7f27e00cf340 2023-10-19T13:00:40.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.205+0000 7f27d7fff640 1 -- 172.21.15.57:0/2397153476 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27d0010690 con 0x7f27e00cf340 2023-10-19T13:00:40.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.205+0000 7f27e5fa9640 1 -- 172.21.15.57:0/2397153476 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27e00c9300 con 0x7f27e00cf340 2023-10-19T13:00:40.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.206+0000 7f27e5fa9640 1 -- 172.21.15.57:0/2397153476 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27e00690e0 con 0x7f27e00cf340 2023-10-19T13:00:40.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.207+0000 7f27d7fff640 1 -- 172.21.15.57:0/2397153476 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 36) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f27d00107f0 con 0x7f27e00cf340 2023-10-19T13:00:40.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.207+0000 7f27d7fff640 1 --2- 172.21.15.57:0/2397153476 >> [v2:172.21.15.179:6832/137854273,v1:172.21.15.179:6833/137854273] conn(0x7f27c006af90 0x7f27c006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:40.216 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.207+0000 7f27d7fff640 1 -- 172.21.15.57:0/2397153476 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f27d0014070 con 0x7f27e00cf340 2023-10-19T13:00:40.216 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.207+0000 7f27df7fe640 1 -- 172.21.15.57:0/2397153476 >> [v2:172.21.15.179:6832/137854273,v1:172.21.15.179:6833/137854273] conn(0x7f27c006af90 msgr2=0x7f27c006d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/137854273 2023-10-19T13:00:40.216 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.207+0000 7f27df7fe640 1 --2- 172.21.15.57:0/2397153476 >> [v2:172.21.15.179:6832/137854273,v1:172.21.15.179:6833/137854273] conn(0x7f27c006af90 0x7f27c006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T13:00:40.216 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.211+0000 7f27d7fff640 1 -- 172.21.15.57:0/2397153476 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f27d0051ca0 con 0x7f27e00cf340 2023-10-19T13:00:40.275 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:40.273+0000 7f842ea41040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-19T13:00:40.383 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:00:40.380+0000 7f918bc30040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-19T13:00:40.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.401+0000 7f27e5fa9640 1 -- 172.21.15.57:0/2397153476 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f27e00cb9d0 con 0x7f27e00cf340 2023-10-19T13:00:40.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.402+0000 7f27d7fff640 1 -- 172.21.15.57:0/2397153476 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v88) v1 ==== 80+0+478 (secure 0 0 0) 0x7f27d0055190 con 0x7f27e00cf340 2023-10-19T13:00:40.405 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/4093253624 2023-10-20T13:00:36.611894+0000 2023-10-19T13:00:40.405 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:6833/3815793798 2023-10-20T13:00:36.611894+0000 2023-10-19T13:00:40.405 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/2495886194 2023-10-20T13:00:36.611894+0000 2023-10-19T13:00:40.406 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:6832/3815793798 2023-10-20T13:00:36.611894+0000 2023-10-19T13:00:40.406 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/2414172521 2023-10-20T13:00:36.611894+0000 2023-10-19T13:00:40.406 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/2931605339 2023-10-20T13:00:36.611894+0000 2023-10-19T13:00:40.406 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/1988099910 2023-10-20T13:00:36.611894+0000 2023-10-19T13:00:40.407 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/4264007972 2023-10-20T13:00:36.611894+0000 2023-10-19T13:00:40.407 INFO:teuthology.orchestra.run.smithi057.stderr:listed 8 entries 2023-10-19T13:00:40.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.404+0000 7f27e5fa9640 1 -- 172.21.15.57:0/2397153476 >> [v2:172.21.15.179:6832/137854273,v1:172.21.15.179:6833/137854273] conn(0x7f27c006af90 msgr2=0x7f27c006d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T13:00:40.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.404+0000 7f27e5fa9640 1 --2- 172.21.15.57:0/2397153476 >> [v2:172.21.15.179:6832/137854273,v1:172.21.15.179:6833/137854273] conn(0x7f27c006af90 0x7f27c006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:40.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.404+0000 7f27e5fa9640 1 -- 172.21.15.57:0/2397153476 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f27e00cf340 msgr2=0x7f27e00cad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:40.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.404+0000 7f27e5fa9640 1 --2- 172.21.15.57:0/2397153476 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f27e00cf340 0x7f27e00cad70 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f27d000e790 tx=0x7f27d000ec50 comp rx=0 tx=0).stop 2023-10-19T13:00:40.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.404+0000 7f27e5fa9640 1 -- 172.21.15.57:0/2397153476 shutdown_connections 2023-10-19T13:00:40.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.404+0000 7f27e5fa9640 1 --2- 172.21.15.57:0/2397153476 >> [v2:172.21.15.179:6832/137854273,v1:172.21.15.179:6833/137854273] conn(0x7f27c006af90 0x7f27c006d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:40.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.404+0000 7f27e5fa9640 1 --2- 172.21.15.57:0/2397153476 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f27e015b2a0 0x7f27e00ca130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:40.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.404+0000 7f27e5fa9640 1 --2- 172.21.15.57:0/2397153476 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f27e00cf340 0x7f27e00cad70 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:40.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.404+0000 7f27e5fa9640 1 --2- 172.21.15.57:0/2397153476 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f27e00ceb40 0x7f27e0157eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:40.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.404+0000 7f27e5fa9640 1 -- 172.21.15.57:0/2397153476 >> 172.21.15.57:0/2397153476 conn(0x7f27e00d0d60 msgr2=0x7f27e00b1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:40.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.404+0000 7f27e5fa9640 1 -- 172.21.15.57:0/2397153476 shutdown_connections 2023-10-19T13:00:40.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.404+0000 7f27e5fa9640 1 -- 172.21.15.57:0/2397153476 wait complete. 2023-10-19T13:00:40.411 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:40.404+0000 7f842ea41040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-19T13:00:40.417 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi057.front.sepia.ceph.com... 2023-10-19T13:00:40.418 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:00:40.418 DEBUG:teuthology.orchestra.run.smithi057:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-10-19T13:00:40.447 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi057.stderr:ceph-fuse[119022]: fuse finished with error 0 and tester_r 0 2023-10-19T13:00:40.449 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi057.front.sepia.ceph.com 2023-10-19T13:00:40.449 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:00:40.450 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-19T13:00:40.497 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:40.495+0000 7f842ea41040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-19T13:00:40.525 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:00:40.523+0000 7f918bc30040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-19T13:00:40.539 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:40.537+0000 7f842ea41040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-19T13:00:40.614 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-19T13:00:40.615 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T13:00:40.622 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:00:40.619+0000 7f918bc30040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-19T13:00:40.629 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054833,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=284 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3247632k,nr_inodes=811908,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-19T13:00:40.630 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi179.front.sepia.ceph.com 2023-10-19T13:00:40.630 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:00:40.631 DEBUG:teuthology.orchestra.run.smithi179:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-10-19T13:00:40.635 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:40.632+0000 7f842ea41040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-19T13:00:40.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-19T13:00:40.649 INFO:teuthology.orchestra.run.smithi179.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2023-10-19T13:00:40.649 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T13:00:40.649 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T13:00:40.665 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055614,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=358 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248260k,nr_inodes=812065,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-19T13:00:40.665 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi057.front.sepia.ceph.com 2023-10-19T13:00:40.666 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:00:40.666 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-10-19T13:00:40.667 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:00:40.664+0000 7f918bc30040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-19T13:00:40.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-19T13:00:40.682 INFO:teuthology.orchestra.run.smithi057.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2023-10-19T13:00:40.682 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-19T13:00:40.682 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T13:00:40.703 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054833,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=284 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3247632k,nr_inodes=811908,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-19T13:00:40.704 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi179.front.sepia.ceph.com 2023-10-19T13:00:40.704 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:00:40.704 DEBUG:teuthology.orchestra.run.smithi179:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-10-19T13:00:40.732 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:40.729+0000 7f842ea41040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-19T13:00:40.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-19T13:00:40.747 INFO:teuthology.orchestra.run.smithi179.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2023-10-19T13:00:40.747 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T13:00:40.768 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:00:40.765+0000 7f918bc30040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-19T13:00:40.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.854+0000 7f7e88cec640 1 -- 172.21.15.57:0/586220001 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e8415b2a0 msgr2=0x7f7e840cf720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:40.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.854+0000 7f7e88cec640 1 --2- 172.21.15.57:0/586220001 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e8415b2a0 0x7f7e840cf720 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f7e84089ae0 tx=0x7f7e7802bd10 comp rx=0 tx=0).stop 2023-10-19T13:00:40.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.855+0000 7f7e88cec640 1 -- 172.21.15.57:0/586220001 shutdown_connections 2023-10-19T13:00:40.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.855+0000 7f7e88cec640 1 --2- 172.21.15.57:0/586220001 >> v2:172.21.15.179:3300/0 conn(0x7f7e840ceb40 0x7f7e840b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:40.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.855+0000 7f7e88cec640 1 --2- 172.21.15.57:0/586220001 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e8415b2a0 0x7f7e840cf720 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:40.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.855+0000 7f7e88cec640 1 --2- 172.21.15.57:0/586220001 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7e840cf340 0x7f7e840d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:40.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.855+0000 7f7e88cec640 1 -- 172.21.15.57:0/586220001 >> 172.21.15.57:0/586220001 conn(0x7f7e840d0d60 msgr2=0x7f7e840d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:40.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.855+0000 7f7e88cec640 1 -- 172.21.15.57:0/586220001 shutdown_connections 2023-10-19T13:00:40.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.855+0000 7f7e88cec640 1 -- 172.21.15.57:0/586220001 wait complete. 2023-10-19T13:00:40.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.856+0000 7f7e88cec640 1 Processor -- start 2023-10-19T13:00:40.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.856+0000 7f7e88cec640 1 -- start start 2023-10-19T13:00:40.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.857+0000 7f7e88cec640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7e840ceb40 0x7f7e8415c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:40.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.857+0000 7f7e88cec640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e840cf340 0x7f7e84158140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:40.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.857+0000 7f7e88cec640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7e8415b2a0 0x7f7e840b5710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:40.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.857+0000 7f7e88cec640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e8415d930 con 0x7f7e840cf340 2023-10-19T13:00:40.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.857+0000 7f7e88cec640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e8415b740 con 0x7f7e840ceb40 2023-10-19T13:00:40.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.857+0000 7f7e88cec640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e840b6080 con 0x7f7e8415b2a0 2023-10-19T13:00:40.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.857+0000 7f7e81d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e840cf340 0x7f7e84158140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:40.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.857+0000 7f7e81d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e840cf340 0x7f7e84158140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:47640/0 (socket says 172.21.15.57:47640) 2023-10-19T13:00:40.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.857+0000 7f7e81d9b640 1 -- 172.21.15.57:0/541006230 learned_addr learned my addr 172.21.15.57:0/541006230 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:40.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.857+0000 7f7e8259c640 1 --2- 172.21.15.57:0/541006230 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7e840ceb40 0x7f7e8415c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:40.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.857+0000 7f7e82d9d640 1 --2- 172.21.15.57:0/541006230 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7e8415b2a0 0x7f7e840b5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:40.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.857+0000 7f7e81d9b640 1 -- 172.21.15.57:0/541006230 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7e8415b2a0 msgr2=0x7f7e840b5710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:40.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.857+0000 7f7e81d9b640 1 --2- 172.21.15.57:0/541006230 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7e8415b2a0 0x7f7e840b5710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:40.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.857+0000 7f7e81d9b640 1 -- 172.21.15.57:0/541006230 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7e840ceb40 msgr2=0x7f7e8415c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:40.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.857+0000 7f7e81d9b640 1 --2- 172.21.15.57:0/541006230 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7e840ceb40 0x7f7e8415c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:40.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.857+0000 7f7e81d9b640 1 -- 172.21.15.57:0/541006230 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e78009560 con 0x7f7e840cf340 2023-10-19T13:00:40.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.857+0000 7f7e8259c640 1 --2- 172.21.15.57:0/541006230 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7e840ceb40 0x7f7e8415c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T13:00:40.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.857+0000 7f7e81d9b640 1 --2- 172.21.15.57:0/541006230 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e840cf340 0x7f7e84158140 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f7e78009850 tx=0x7f7e7802da40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:40.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.858+0000 7f7e72ffd640 1 -- 172.21.15.57:0/541006230 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e78003800 con 0x7f7e840cf340 2023-10-19T13:00:40.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.858+0000 7f7e72ffd640 1 -- 172.21.15.57:0/541006230 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f7e78003a50 con 0x7f7e840cf340 2023-10-19T13:00:40.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.858+0000 7f7e88cec640 1 -- 172.21.15.57:0/541006230 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e8415b9e0 con 0x7f7e840cf340 2023-10-19T13:00:40.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.858+0000 7f7e72ffd640 1 -- 172.21.15.57:0/541006230 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e78003e20 con 0x7f7e840cf340 2023-10-19T13:00:40.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.858+0000 7f7e88cec640 1 -- 172.21.15.57:0/541006230 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e840b6360 con 0x7f7e840cf340 2023-10-19T13:00:40.868 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.859+0000 7f7e88cec640 1 -- 172.21.15.57:0/541006230 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e8415bf00 con 0x7f7e840cf340 2023-10-19T13:00:40.868 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.859+0000 7f7e72ffd640 1 -- 172.21.15.57:0/541006230 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 36) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f7e7803e450 con 0x7f7e840cf340 2023-10-19T13:00:40.868 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.860+0000 7f7e72ffd640 1 --2- 172.21.15.57:0/541006230 >> [v2:172.21.15.179:6832/137854273,v1:172.21.15.179:6833/137854273] conn(0x7f7e6406af60 0x7f7e6406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:40.868 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.860+0000 7f7e8259c640 1 -- 172.21.15.57:0/541006230 >> [v2:172.21.15.179:6832/137854273,v1:172.21.15.179:6833/137854273] conn(0x7f7e6406af60 msgr2=0x7f7e6406d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/137854273 2023-10-19T13:00:40.869 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.860+0000 7f7e72ffd640 1 -- 172.21.15.57:0/541006230 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f7e780ac3f0 con 0x7f7e840cf340 2023-10-19T13:00:40.869 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.860+0000 7f7e8259c640 1 --2- 172.21.15.57:0/541006230 >> [v2:172.21.15.179:6832/137854273,v1:172.21.15.179:6833/137854273] conn(0x7f7e6406af60 0x7f7e6406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T13:00:40.869 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:00:40.864+0000 7f918bc30040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-19T13:00:40.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:40.864+0000 7f7e72ffd640 1 -- 172.21.15.57:0/541006230 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7e8415bf00 con 0x7f7e840cf340 2023-10-19T13:00:40.872 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:40.869+0000 7f842ea41040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-19T13:00:40.914 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:40.912+0000 7f842ea41040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-19T13:00:41.011 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:00:41.008+0000 7f918bc30040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-19T13:00:41.019 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:41.016+0000 7f842ea41040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-19T13:00:41.055 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:00:41.052+0000 7f918bc30040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-19T13:00:41.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.061+0000 7f7e8259c640 1 -- 172.21.15.57:0/541006230 >> [v2:172.21.15.179:6832/137854273,v1:172.21.15.179:6833/137854273] conn(0x7f7e6406af60 msgr2=0x7f7e6406d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/137854273 2023-10-19T13:00:41.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.061+0000 7f7e8259c640 1 --2- 172.21.15.57:0/541006230 >> [v2:172.21.15.179:6832/137854273,v1:172.21.15.179:6833/137854273] conn(0x7f7e6406af60 0x7f7e6406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-19T13:00:41.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.075+0000 7f7e88cec640 1 -- 172.21.15.57:0/541006230 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7e8415bf00 con 0x7f7e840cf340 2023-10-19T13:00:41.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.076+0000 7f7e72ffd640 1 -- 172.21.15.57:0/541006230 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+4917 (secure 0 0 0) 0x7f7e8415bf00 con 0x7f7e840cf340 2023-10-19T13:00:41.079 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:00:41.079 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":30,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5966,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/765319531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":765319531},{"type":"v1","addr":"172.21.15.179:6837","nonce":765319531}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27},{"gid":5969,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/1388526553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":1388526553},{"type":"v1","addr":"172.21.15.179:6836","nonce":1388526553}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27},{"gid":7035,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/3237434259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":3237434259},{"type":"v1","addr":"172.21.15.57:6834","nonce":3237434259}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T13:00:20.549275+0000","modified":"2023-10-19T13:00:22.561271+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7038},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7038":{"gid":7038,"name":"a","rank":0,"incarnation":28,"state":"up:active","state_seq":3,"addr":"172.21.15.57:6835/2257211800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2257211800},{"type":"v1","addr":"172.21.15.57:6835","nonce":2257211800}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-10-19T13:00:41.080 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 30 2023-10-19T13:00:41.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.078+0000 7f7e88cec640 1 -- 172.21.15.57:0/541006230 >> [v2:172.21.15.179:6832/137854273,v1:172.21.15.179:6833/137854273] conn(0x7f7e6406af60 msgr2=0x7f7e6406d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T13:00:41.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.078+0000 7f7e88cec640 1 --2- 172.21.15.57:0/541006230 >> [v2:172.21.15.179:6832/137854273,v1:172.21.15.179:6833/137854273] conn(0x7f7e6406af60 0x7f7e6406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:41.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.078+0000 7f7e88cec640 1 -- 172.21.15.57:0/541006230 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e840cf340 msgr2=0x7f7e84158140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:41.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.078+0000 7f7e88cec640 1 --2- 172.21.15.57:0/541006230 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e840cf340 0x7f7e84158140 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f7e78009850 tx=0x7f7e7802da40 comp rx=0 tx=0).stop 2023-10-19T13:00:41.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.078+0000 7f7e88cec640 1 -- 172.21.15.57:0/541006230 shutdown_connections 2023-10-19T13:00:41.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.078+0000 7f7e88cec640 1 --2- 172.21.15.57:0/541006230 >> [v2:172.21.15.179:6832/137854273,v1:172.21.15.179:6833/137854273] conn(0x7f7e6406af60 0x7f7e6406d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:41.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.078+0000 7f7e88cec640 1 --2- 172.21.15.57:0/541006230 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7e8415b2a0 0x7f7e840b5710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:41.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.078+0000 7f7e88cec640 1 --2- 172.21.15.57:0/541006230 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e840cf340 0x7f7e84158140 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:41.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.078+0000 7f7e88cec640 1 --2- 172.21.15.57:0/541006230 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7e840ceb40 0x7f7e8415c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:41.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.078+0000 7f7e88cec640 1 -- 172.21.15.57:0/541006230 >> 172.21.15.57:0/541006230 conn(0x7f7e840d0d60 msgr2=0x7f7e8406b590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:41.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.078+0000 7f7e88cec640 1 -- 172.21.15.57:0/541006230 shutdown_connections 2023-10-19T13:00:41.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.078+0000 7f7e88cec640 1 -- 172.21.15.57:0/541006230 wait complete. 2023-10-19T13:00:41.092 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T13:00:41.169 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:00:41.166+0000 7f918bc30040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-19T13:00:41.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.242+0000 7f37c278a640 1 -- 172.21.15.57:0/978046326 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f37bc159060 msgr2=0x7f37bc06d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:41.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.242+0000 7f37c278a640 1 --2- 172.21.15.57:0/978046326 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f37bc159060 0x7f37bc06d260 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f37b0005fd0 tx=0x7f37b002bd90 comp rx=0 tx=0).stop 2023-10-19T13:00:41.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.242+0000 7f37c278a640 1 -- 172.21.15.57:0/978046326 shutdown_connections 2023-10-19T13:00:41.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.242+0000 7f37c278a640 1 --2- 172.21.15.57:0/978046326 >> v2:172.21.15.179:3300/0 conn(0x7f37bc15d400 0x7f37bc0cdfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:41.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.242+0000 7f37c278a640 1 --2- 172.21.15.57:0/978046326 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f37bc159060 0x7f37bc06d260 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:41.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.242+0000 7f37c278a640 1 -- 172.21.15.57:0/978046326 >> 172.21.15.57:0/978046326 conn(0x7f37bc0b0670 msgr2=0x7f37bc0b0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:41.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.242+0000 7f37c278a640 1 -- 172.21.15.57:0/978046326 shutdown_connections 2023-10-19T13:00:41.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.242+0000 7f37c278a640 1 -- 172.21.15.57:0/978046326 wait complete. 2023-10-19T13:00:41.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.244+0000 7f37c278a640 1 Processor -- start 2023-10-19T13:00:41.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.244+0000 7f37c278a640 1 -- start start 2023-10-19T13:00:41.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.244+0000 7f37c278a640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f37bc0cdb30 0x7f37bc0c1820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:41.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.244+0000 7f37c278a640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f37bc159060 0x7f37bc06c110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:41.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.244+0000 7f37c278a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f37bc15d400 0x7f37bc0af460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:41.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.245+0000 7f37c278a640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37bc0c84a0 con 0x7f37bc15d400 2023-10-19T13:00:41.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.245+0000 7f37c278a640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37bc151920 con 0x7f37bc0cdb30 2023-10-19T13:00:41.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.245+0000 7f37c278a640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37bc0c7920 con 0x7f37bc159060 2023-10-19T13:00:41.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.245+0000 7f37c0d27640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f37bc15d400 0x7f37bc0af460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:41.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.245+0000 7f37c0d27640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f37bc15d400 0x7f37bc0af460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:47652/0 (socket says 172.21.15.57:47652) 2023-10-19T13:00:41.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.245+0000 7f37c0d27640 1 -- 172.21.15.57:0/3747931425 learned_addr learned my addr 172.21.15.57:0/3747931425 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:41.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.245+0000 7f37bbfff640 1 --2- 172.21.15.57:0/3747931425 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f37bc0cdb30 0x7f37bc0c1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:41.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.245+0000 7f37bb7fe640 1 --2- 172.21.15.57:0/3747931425 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f37bc159060 0x7f37bc06c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:41.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.245+0000 7f37c0d27640 1 -- 172.21.15.57:0/3747931425 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f37bc159060 msgr2=0x7f37bc06c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:41.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.245+0000 7f37c0d27640 1 --2- 172.21.15.57:0/3747931425 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f37bc159060 0x7f37bc06c110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:41.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.245+0000 7f37c0d27640 1 -- 172.21.15.57:0/3747931425 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f37bc0cdb30 msgr2=0x7f37bc0c1820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:41.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.245+0000 7f37c0d27640 1 --2- 172.21.15.57:0/3747931425 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f37bc0cdb30 0x7f37bc0c1820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:41.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.245+0000 7f37c0d27640 1 -- 172.21.15.57:0/3747931425 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37b0009560 con 0x7f37bc15d400 2023-10-19T13:00:41.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.245+0000 7f37c0d27640 1 --2- 172.21.15.57:0/3747931425 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f37bc15d400 0x7f37bc0af460 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f37b400da90 tx=0x7f37b400df50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:41.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.245+0000 7f37b8ff9640 1 -- 172.21.15.57:0/3747931425 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37b400bdd0 con 0x7f37bc15d400 2023-10-19T13:00:41.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.245+0000 7f37b8ff9640 1 -- 172.21.15.57:0/3747931425 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f37b4003970 con 0x7f37bc15d400 2023-10-19T13:00:41.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.245+0000 7f37c278a640 1 -- 172.21.15.57:0/3747931425 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37bc0c7c00 con 0x7f37bc15d400 2023-10-19T13:00:41.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.245+0000 7f37b8ff9640 1 -- 172.21.15.57:0/3747931425 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37b4003d80 con 0x7f37bc15d400 2023-10-19T13:00:41.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.246+0000 7f37c278a640 1 -- 172.21.15.57:0/3747931425 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37bc0c81c0 con 0x7f37bc15d400 2023-10-19T13:00:41.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.247+0000 7f37c278a640 1 -- 172.21.15.57:0/3747931425 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37bc0675f0 con 0x7f37bc15d400 2023-10-19T13:00:41.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.247+0000 7f37b8ff9640 1 -- 172.21.15.57:0/3747931425 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 36) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f37b4003ee0 con 0x7f37bc15d400 2023-10-19T13:00:41.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.248+0000 7f37b8ff9640 1 --2- 172.21.15.57:0/3747931425 >> [v2:172.21.15.179:6832/137854273,v1:172.21.15.179:6833/137854273] conn(0x7f37a806af60 0x7f37a806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:41.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.248+0000 7f37b8ff9640 1 -- 172.21.15.57:0/3747931425 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f37b4089f30 con 0x7f37bc15d400 2023-10-19T13:00:41.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.248+0000 7f37bbfff640 1 -- 172.21.15.57:0/3747931425 >> [v2:172.21.15.179:6832/137854273,v1:172.21.15.179:6833/137854273] conn(0x7f37a806af60 msgr2=0x7f37a806d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/137854273 2023-10-19T13:00:41.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.248+0000 7f37bbfff640 1 --2- 172.21.15.57:0/3747931425 >> [v2:172.21.15.179:6832/137854273,v1:172.21.15.179:6833/137854273] conn(0x7f37a806af60 0x7f37a806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T13:00:41.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.252+0000 7f37b8ff9640 1 -- 172.21.15.57:0/3747931425 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f37b4054bb0 con 0x7f37bc15d400 2023-10-19T13:00:41.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.428+0000 7f37c278a640 1 -- 172.21.15.57:0/3747931425 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f37bc0b4aa0 con 0x7f37bc15d400 2023-10-19T13:00:41.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.429+0000 7f37b8ff9640 1 -- 172.21.15.57:0/3747931425 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+17280 (secure 0 0 0) 0x7f37b4017090 con 0x7f37bc15d400 2023-10-19T13:00:41.432 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:00:41.432 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":88,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T13:00:36.611926+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2023-10-19T13:00:18.452198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2023-10-19T13:00:19.531525+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.179:0/4093253624":"2023-10-20T13:00:36.611894+0000","172.21.15.179:6833/3815793798":"2023-10-20T13:00:36.611894+0000","172.21.15.179:0/2495886194":"2023-10-20T13:00:36.611894+0000","172.21.15.179:6832/3815793798":"2023-10-20T13:00:36.611894+0000","172.21.15.179:0/2414172521":"2023-10-20T13:00:36.611894+0000","172.21.15.179:0/2931605339":"2023-10-20T13:00:36.611894+0000","172.21.15.179:0/1988099910":"2023-10-20T13:00:36.611894+0000","172.21.15.179:0/4264007972":"2023-10-20T13:00:36.611894+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T13:00:41.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.431+0000 7f37c278a640 1 -- 172.21.15.57:0/3747931425 >> [v2:172.21.15.179:6832/137854273,v1:172.21.15.179:6833/137854273] conn(0x7f37a806af60 msgr2=0x7f37a806d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T13:00:41.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.431+0000 7f37c278a640 1 --2- 172.21.15.57:0/3747931425 >> [v2:172.21.15.179:6832/137854273,v1:172.21.15.179:6833/137854273] conn(0x7f37a806af60 0x7f37a806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:41.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.431+0000 7f37c278a640 1 -- 172.21.15.57:0/3747931425 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f37bc15d400 msgr2=0x7f37bc0af460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:41.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.431+0000 7f37c278a640 1 --2- 172.21.15.57:0/3747931425 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f37bc15d400 0x7f37bc0af460 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f37b400da90 tx=0x7f37b400df50 comp rx=0 tx=0).stop 2023-10-19T13:00:41.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.431+0000 7f37c278a640 1 -- 172.21.15.57:0/3747931425 shutdown_connections 2023-10-19T13:00:41.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.431+0000 7f37c278a640 1 --2- 172.21.15.57:0/3747931425 >> [v2:172.21.15.179:6832/137854273,v1:172.21.15.179:6833/137854273] conn(0x7f37a806af60 0x7f37a806d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:41.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.431+0000 7f37c278a640 1 --2- 172.21.15.57:0/3747931425 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f37bc15d400 0x7f37bc0af460 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:41.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.431+0000 7f37c278a640 1 --2- 172.21.15.57:0/3747931425 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f37bc159060 0x7f37bc06c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:41.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.431+0000 7f37c278a640 1 --2- 172.21.15.57:0/3747931425 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f37bc0cdb30 0x7f37bc0c1820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:41.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.431+0000 7f37c278a640 1 -- 172.21.15.57:0/3747931425 >> 172.21.15.57:0/3747931425 conn(0x7f37bc0b0670 msgr2=0x7f37bc0b0a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:41.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.431+0000 7f37c278a640 1 -- 172.21.15.57:0/3747931425 shutdown_connections 2023-10-19T13:00:41.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.431+0000 7f37c278a640 1 -- 172.21.15.57:0/3747931425 wait complete. 2023-10-19T13:00:41.445 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-19T12:44:07.371186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-19T13:00:18.452198+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_data', 'create_time': '2023-10-19T13:00:19.531525+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-19T13:00:41.445 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-19T13:00:41.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.595+0000 7f58ed4f2640 1 -- 172.21.15.57:0/3021752970 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f58e80b1980 msgr2=0x7f58e8070080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:41.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.595+0000 7f58ed4f2640 1 --2- 172.21.15.57:0/3021752970 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f58e80b1980 0x7f58e8070080 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f58e815c9d0 tx=0x7f58d002bda0 comp rx=0 tx=0).stop 2023-10-19T13:00:41.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.596+0000 7f58ed4f2640 1 -- 172.21.15.57:0/3021752970 shutdown_connections 2023-10-19T13:00:41.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.596+0000 7f58ed4f2640 1 --2- 172.21.15.57:0/3021752970 >> v2:172.21.15.179:3300/0 conn(0x7f58e80705c0 0x7f58e815b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:41.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.596+0000 7f58ed4f2640 1 --2- 172.21.15.57:0/3021752970 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f58e80b1980 0x7f58e8070080 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:41.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.596+0000 7f58ed4f2640 1 --2- 172.21.15.57:0/3021752970 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f58e81416f0 0x7f58e80b1440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:41.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.596+0000 7f58ed4f2640 1 -- 172.21.15.57:0/3021752970 >> 172.21.15.57:0/3021752970 conn(0x7f58e80af730 msgr2=0x7f58e80afb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:41.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.596+0000 7f58ed4f2640 1 -- 172.21.15.57:0/3021752970 shutdown_connections 2023-10-19T13:00:41.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.596+0000 7f58ed4f2640 1 -- 172.21.15.57:0/3021752970 wait complete. 2023-10-19T13:00:41.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.597+0000 7f58ed4f2640 1 Processor -- start 2023-10-19T13:00:41.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.597+0000 7f58ed4f2640 1 -- start start 2023-10-19T13:00:41.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.598+0000 7f58ed4f2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f58e80705c0 0x7f58e81ef880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:41.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.598+0000 7f58ed4f2640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f58e80b1980 0x7f58e815a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:41.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.598+0000 7f58ed4f2640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f58e81416f0 0x7f58e81f2ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:41.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.598+0000 7f58ed4f2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58e813dbb0 con 0x7f58e80705c0 2023-10-19T13:00:41.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.598+0000 7f58ed4f2640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58e81f3410 con 0x7f58e80b1980 2023-10-19T13:00:41.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.598+0000 7f58ed4f2640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58e81f3650 con 0x7f58e81416f0 2023-10-19T13:00:41.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.598+0000 7f58e6ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f58e80705c0 0x7f58e81ef880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:41.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.598+0000 7f58e6ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f58e80705c0 0x7f58e81ef880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:47664/0 (socket says 172.21.15.57:47664) 2023-10-19T13:00:41.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.598+0000 7f58e6ffd640 1 -- 172.21.15.57:0/957361087 learned_addr learned my addr 172.21.15.57:0/957361087 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:41.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.599+0000 7f58e77fe640 1 --2- 172.21.15.57:0/957361087 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f58e81416f0 0x7f58e81f2ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:41.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.599+0000 7f58dffff640 1 --2- 172.21.15.57:0/957361087 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f58e80b1980 0x7f58e815a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:41.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.599+0000 7f58e6ffd640 1 -- 172.21.15.57:0/957361087 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f58e81416f0 msgr2=0x7f58e81f2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:41.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.599+0000 7f58e6ffd640 1 --2- 172.21.15.57:0/957361087 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f58e81416f0 0x7f58e81f2ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:41.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.599+0000 7f58e6ffd640 1 -- 172.21.15.57:0/957361087 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f58e80b1980 msgr2=0x7f58e815a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:41.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.599+0000 7f58e6ffd640 1 --2- 172.21.15.57:0/957361087 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f58e80b1980 0x7f58e815a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:41.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.599+0000 7f58e6ffd640 1 -- 172.21.15.57:0/957361087 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58d0009560 con 0x7f58e80705c0 2023-10-19T13:00:41.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.599+0000 7f58e6ffd640 1 --2- 172.21.15.57:0/957361087 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f58e80705c0 0x7f58e81ef880 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f58d800d770 tx=0x7f58d800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:41.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.599+0000 7f58df7fe640 1 -- 172.21.15.57:0/957361087 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58d8004220 con 0x7f58e80705c0 2023-10-19T13:00:41.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.599+0000 7f58df7fe640 1 -- 172.21.15.57:0/957361087 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f58d8004470 con 0x7f58e80705c0 2023-10-19T13:00:41.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.599+0000 7f58ed4f2640 1 -- 172.21.15.57:0/957361087 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58e81f3930 con 0x7f58e80705c0 2023-10-19T13:00:41.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.599+0000 7f58df7fe640 1 -- 172.21.15.57:0/957361087 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58d8009c10 con 0x7f58e80705c0 2023-10-19T13:00:41.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.599+0000 7f58ed4f2640 1 -- 172.21.15.57:0/957361087 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58e813dd90 con 0x7f58e80705c0 2023-10-19T13:00:41.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.601+0000 7f58ed4f2640 1 -- 172.21.15.57:0/957361087 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58e81f3e50 con 0x7f58e80705c0 2023-10-19T13:00:41.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.601+0000 7f58df7fe640 1 -- 172.21.15.57:0/957361087 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 36) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f58d8005020 con 0x7f58e80705c0 2023-10-19T13:00:41.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.601+0000 7f58df7fe640 1 --2- 172.21.15.57:0/957361087 >> [v2:172.21.15.179:6832/137854273,v1:172.21.15.179:6833/137854273] conn(0x7f58c806af90 0x7f58c806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:41.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.601+0000 7f58df7fe640 1 -- 172.21.15.57:0/957361087 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f58d8089d50 con 0x7f58e80705c0 2023-10-19T13:00:41.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.601+0000 7f58dffff640 1 -- 172.21.15.57:0/957361087 >> [v2:172.21.15.179:6832/137854273,v1:172.21.15.179:6833/137854273] conn(0x7f58c806af90 msgr2=0x7f58c806d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/137854273 2023-10-19T13:00:41.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.601+0000 7f58dffff640 1 --2- 172.21.15.57:0/957361087 >> [v2:172.21.15.179:6832/137854273,v1:172.21.15.179:6833/137854273] conn(0x7f58c806af90 0x7f58c806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T13:00:41.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.605+0000 7f58df7fe640 1 -- 172.21.15.57:0/957361087 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f58d808d030 con 0x7f58e80705c0 2023-10-19T13:00:41.635 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:41.633+0000 7f842ea41040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-19T13:00:41.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.802+0000 7f58dffff640 1 -- 172.21.15.57:0/957361087 >> [v2:172.21.15.179:6832/137854273,v1:172.21.15.179:6833/137854273] conn(0x7f58c806af90 msgr2=0x7f58c806d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/137854273 2023-10-19T13:00:41.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.802+0000 7f58dffff640 1 --2- 172.21.15.57:0/957361087 >> [v2:172.21.15.179:6832/137854273,v1:172.21.15.179:6833/137854273] conn(0x7f58c806af90 0x7f58c806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-19T13:00:41.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.830+0000 7f58ed4f2640 1 -- 172.21.15.57:0/957361087 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f58e80675f0 con 0x7f58e80705c0 2023-10-19T13:00:41.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.831+0000 7f58df7fe640 1 -- 172.21.15.57:0/957361087 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) v1 ==== 93+0+10 (secure 0 0 0) 0x7f58d8017020 con 0x7f58e80705c0 2023-10-19T13:00:41.834 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-19T13:00:41.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.833+0000 7f58ed4f2640 1 -- 172.21.15.57:0/957361087 >> [v2:172.21.15.179:6832/137854273,v1:172.21.15.179:6833/137854273] conn(0x7f58c806af90 msgr2=0x7f58c806d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T13:00:41.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.833+0000 7f58ed4f2640 1 --2- 172.21.15.57:0/957361087 >> [v2:172.21.15.179:6832/137854273,v1:172.21.15.179:6833/137854273] conn(0x7f58c806af90 0x7f58c806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:41.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.833+0000 7f58ed4f2640 1 -- 172.21.15.57:0/957361087 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f58e80705c0 msgr2=0x7f58e81ef880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:41.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.833+0000 7f58ed4f2640 1 --2- 172.21.15.57:0/957361087 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f58e80705c0 0x7f58e81ef880 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f58d800d770 tx=0x7f58d800dc30 comp rx=0 tx=0).stop 2023-10-19T13:00:41.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.833+0000 7f58ed4f2640 1 -- 172.21.15.57:0/957361087 shutdown_connections 2023-10-19T13:00:41.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.833+0000 7f58ed4f2640 1 --2- 172.21.15.57:0/957361087 >> [v2:172.21.15.179:6832/137854273,v1:172.21.15.179:6833/137854273] conn(0x7f58c806af90 0x7f58c806d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:41.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.833+0000 7f58ed4f2640 1 --2- 172.21.15.57:0/957361087 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f58e81416f0 0x7f58e81f2ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:41.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.833+0000 7f58ed4f2640 1 --2- 172.21.15.57:0/957361087 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f58e80b1980 0x7f58e815a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:41.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.833+0000 7f58ed4f2640 1 --2- 172.21.15.57:0/957361087 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f58e80705c0 0x7f58e81ef880 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:41.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.833+0000 7f58ed4f2640 1 -- 172.21.15.57:0/957361087 >> 172.21.15.57:0/957361087 conn(0x7f58e80af730 msgr2=0x7f58e8069fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:41.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.833+0000 7f58ed4f2640 1 -- 172.21.15.57:0/957361087 shutdown_connections 2023-10-19T13:00:41.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.833+0000 7f58ed4f2640 1 -- 172.21.15.57:0/957361087 wait complete. 2023-10-19T13:00:41.847 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-19T13:00:41.872 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:00:41.870+0000 7f918bc30040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-19T13:00:41.898 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:41.895+0000 7f842ea41040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-19T13:00:41.946 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:41.944+0000 7f842ea41040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-19T13:00:41.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.956+0000 7f6231687640 1 -- 172.21.15.57:0/797977013 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f622c13e520 msgr2=0x7f622c06bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:41.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.956+0000 7f6231687640 1 --2- 172.21.15.57:0/797977013 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f622c13e520 0x7f622c06bb60 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f622c0b20f0 tx=0x7f621402bda0 comp rx=0 tx=0).stop 2023-10-19T13:00:41.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.956+0000 7f6231687640 1 -- 172.21.15.57:0/797977013 shutdown_connections 2023-10-19T13:00:41.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.956+0000 7f6231687640 1 --2- 172.21.15.57:0/797977013 >> v2:172.21.15.179:3300/0 conn(0x7f622c070af0 0x7f622c0affe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:41.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.956+0000 7f6231687640 1 --2- 172.21.15.57:0/797977013 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f622c13e520 0x7f622c06bb60 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:41.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.956+0000 7f6231687640 1 -- 172.21.15.57:0/797977013 >> 172.21.15.57:0/797977013 conn(0x7f622c069930 msgr2=0x7f622c069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:41.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.956+0000 7f6231687640 1 -- 172.21.15.57:0/797977013 shutdown_connections 2023-10-19T13:00:41.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.956+0000 7f6231687640 1 -- 172.21.15.57:0/797977013 wait complete. 2023-10-19T13:00:41.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.958+0000 7f6231687640 1 Processor -- start 2023-10-19T13:00:41.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.958+0000 7f6231687640 1 -- start start 2023-10-19T13:00:41.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.958+0000 7f6231687640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f622c070af0 0x7f622c0adad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:41.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.958+0000 7f6231687640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f622c13e520 0x7f622c13b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:41.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.958+0000 7f6231687640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f622c141690 0x7f622c13d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:41.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.958+0000 7f6231687640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f622c0ae3c0 con 0x7f622c13e520 2023-10-19T13:00:41.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.958+0000 7f6231687640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f622c1e2a80 con 0x7f622c141690 2023-10-19T13:00:41.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.958+0000 7f6231687640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f622c1e2d60 con 0x7f622c070af0 2023-10-19T13:00:41.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.958+0000 7f622a7fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f622c13e520 0x7f622c13b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:41.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.959+0000 7f622a7fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f622c13e520 0x7f622c13b3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:47694/0 (socket says 172.21.15.57:47694) 2023-10-19T13:00:41.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.959+0000 7f622a7fc640 1 -- 172.21.15.57:0/3126826383 learned_addr learned my addr 172.21.15.57:0/3126826383 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:41.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.959+0000 7f622b7fe640 1 --2- 172.21.15.57:0/3126826383 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f622c141690 0x7f622c13d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:41.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.959+0000 7f622affd640 1 --2- 172.21.15.57:0/3126826383 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f622c070af0 0x7f622c0adad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:41.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.959+0000 7f622a7fc640 1 -- 172.21.15.57:0/3126826383 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f622c070af0 msgr2=0x7f622c0adad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:41.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.959+0000 7f622a7fc640 1 --2- 172.21.15.57:0/3126826383 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f622c070af0 0x7f622c0adad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:41.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.959+0000 7f622a7fc640 1 -- 172.21.15.57:0/3126826383 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f622c141690 msgr2=0x7f622c13d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:41.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.959+0000 7f622a7fc640 1 --2- 172.21.15.57:0/3126826383 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f622c141690 0x7f622c13d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:41.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.959+0000 7f622a7fc640 1 -- 172.21.15.57:0/3126826383 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6214009560 con 0x7f622c13e520 2023-10-19T13:00:41.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.959+0000 7f622a7fc640 1 --2- 172.21.15.57:0/3126826383 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f622c13e520 0x7f622c13b3d0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f621c00c400 tx=0x7f621c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:41.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.959+0000 7f62237fe640 1 -- 172.21.15.57:0/3126826383 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f621c0052d0 con 0x7f622c13e520 2023-10-19T13:00:41.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.959+0000 7f62237fe640 1 -- 172.21.15.57:0/3126826383 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f621c0035f0 con 0x7f622c13e520 2023-10-19T13:00:41.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.959+0000 7f6231687640 1 -- 172.21.15.57:0/3126826383 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f622c1e3040 con 0x7f622c13e520 2023-10-19T13:00:41.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.959+0000 7f62237fe640 1 -- 172.21.15.57:0/3126826383 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f621c003a00 con 0x7f622c13e520 2023-10-19T13:00:41.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.959+0000 7f6231687640 1 -- 172.21.15.57:0/3126826383 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f622c13d940 con 0x7f622c13e520 2023-10-19T13:00:41.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.960+0000 7f6231687640 1 -- 172.21.15.57:0/3126826383 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f622c1e3560 con 0x7f622c13e520 2023-10-19T13:00:41.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.961+0000 7f62237fe640 1 -- 172.21.15.57:0/3126826383 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 36) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f621c003b60 con 0x7f622c13e520 2023-10-19T13:00:41.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.961+0000 7f62237fe640 1 --2- 172.21.15.57:0/3126826383 >> [v2:172.21.15.179:6832/137854273,v1:172.21.15.179:6833/137854273] conn(0x7f620c06af60 0x7f620c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:41.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.961+0000 7f622affd640 1 -- 172.21.15.57:0/3126826383 >> [v2:172.21.15.179:6832/137854273,v1:172.21.15.179:6833/137854273] conn(0x7f620c06af60 msgr2=0x7f620c06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/137854273 2023-10-19T13:00:41.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.961+0000 7f62237fe640 1 -- 172.21.15.57:0/3126826383 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f621c089190 con 0x7f622c13e520 2023-10-19T13:00:41.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.961+0000 7f622affd640 1 --2- 172.21.15.57:0/3126826383 >> [v2:172.21.15.179:6832/137854273,v1:172.21.15.179:6833/137854273] conn(0x7f620c06af60 0x7f620c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T13:00:41.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:41.970+0000 7f62237fe640 1 -- 172.21.15.57:0/3126826383 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f622c1e3560 con 0x7f622c13e520 2023-10-19T13:00:42.047 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:42.045+0000 7f842ea41040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-19T13:00:42.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.068+0000 7f62237fe640 1 -- 172.21.15.57:0/3126826383 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mgrmap(e 37) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f621c0873f0 con 0x7f622c13e520 2023-10-19T13:00:42.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.069+0000 7f62237fe640 1 -- 172.21.15.57:0/3126826383 >> [v2:172.21.15.179:6832/137854273,v1:172.21.15.179:6833/137854273] conn(0x7f620c06af60 msgr2=0x7f620c06d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T13:00:42.072 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.069+0000 7f62237fe640 1 --2- 172.21.15.57:0/3126826383 >> [v2:172.21.15.179:6832/137854273,v1:172.21.15.179:6833/137854273] conn(0x7f620c06af60 0x7f620c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:42.129 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:42.126+0000 7f83afcca640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:00:42.129 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:42.126+0000 7f83afcca640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:00:42.130 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:42.126+0000 7f83afcca640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:00:42.130 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:42.126+0000 7f83afcca640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:00:42.130 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:42.126+0000 7f83afcca640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:00:42.130 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:42.126+0000 7f83b34d1640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:00:42.131 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:42.126+0000 7f83b34d1640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:00:42.131 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:42.126+0000 7f83b34d1640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:00:42.131 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:42.126+0000 7f83b34d1640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:00:42.131 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:00:42.126+0000 7f83b34d1640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:00:42.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.152+0000 7f6231687640 1 -- 172.21.15.57:0/3126826383 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f622c1e3560 con 0x7f622c13e520 2023-10-19T13:00:42.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.152+0000 7f62237fe640 1 -- 172.21.15.57:0/3126826383 <== mon.0 v2:172.21.15.57:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) v1 ==== 104+0+11 (secure 0 0 0) 0x7f622c1e3560 con 0x7f622c13e520 2023-10-19T13:00:42.155 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2023-10-19T13:00:42.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.154+0000 7f6231687640 1 -- 172.21.15.57:0/3126826383 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f622c13e520 msgr2=0x7f622c13b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:42.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.154+0000 7f6231687640 1 --2- 172.21.15.57:0/3126826383 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f622c13e520 0x7f622c13b3d0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f621c00c400 tx=0x7f621c00c8c0 comp rx=0 tx=0).stop 2023-10-19T13:00:42.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.154+0000 7f6231687640 1 -- 172.21.15.57:0/3126826383 shutdown_connections 2023-10-19T13:00:42.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.154+0000 7f6231687640 1 --2- 172.21.15.57:0/3126826383 >> [v2:172.21.15.179:6832/137854273,v1:172.21.15.179:6833/137854273] conn(0x7f620c06af60 0x7f620c06d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:42.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.154+0000 7f6231687640 1 --2- 172.21.15.57:0/3126826383 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f622c141690 0x7f622c13d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:42.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.154+0000 7f6231687640 1 --2- 172.21.15.57:0/3126826383 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f622c13e520 0x7f622c13b3d0 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:42.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.154+0000 7f6231687640 1 --2- 172.21.15.57:0/3126826383 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f622c070af0 0x7f622c0adad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:42.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.154+0000 7f6231687640 1 -- 172.21.15.57:0/3126826383 >> 172.21.15.57:0/3126826383 conn(0x7f622c069930 msgr2=0x7f622c0ae6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:42.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.154+0000 7f6231687640 1 -- 172.21.15.57:0/3126826383 shutdown_connections 2023-10-19T13:00:42.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.154+0000 7f6231687640 1 -- 172.21.15.57:0/3126826383 wait complete. 2023-10-19T13:00:42.164 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:00:42.161+0000 7f918bc30040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-19T13:00:42.167 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-19T13:00:42.215 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:00:42.213+0000 7f918bc30040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-19T13:00:42.321 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.318+0000 7f0049007640 1 -- 172.21.15.57:0/1539581795 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f00440d1910 msgr2=0x7f00440b0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:42.321 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.318+0000 7f0049007640 1 --2- 172.21.15.57:0/1539581795 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f00440d1910 0x7f00440b0650 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f003c009be0 tx=0x7f003c02c0c0 comp rx=0 tx=0).stop 2023-10-19T13:00:42.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.318+0000 7f0049007640 1 -- 172.21.15.57:0/1539581795 shutdown_connections 2023-10-19T13:00:42.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.318+0000 7f0049007640 1 --2- 172.21.15.57:0/1539581795 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f00440d1910 0x7f00440b0650 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:42.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.318+0000 7f0049007640 1 -- 172.21.15.57:0/1539581795 >> 172.21.15.57:0/1539581795 conn(0x7f00440d0d60 msgr2=0x7f00440d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:42.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.318+0000 7f0049007640 1 -- 172.21.15.57:0/1539581795 shutdown_connections 2023-10-19T13:00:42.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.318+0000 7f0049007640 1 -- 172.21.15.57:0/1539581795 wait complete. 2023-10-19T13:00:42.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.320+0000 7f0049007640 1 Processor -- start 2023-10-19T13:00:42.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.320+0000 7f0049007640 1 -- start start 2023-10-19T13:00:42.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.320+0000 7f0049007640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f00440cf340 0x7f004409d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:42.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.320+0000 7f0049007640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f00440d1910 0x7f004409b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:42.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.321+0000 7f0049007640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f004415b2a0 0x7f004409dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:42.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.321+0000 7f0049007640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f004415e670 con 0x7f00440cf340 2023-10-19T13:00:42.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.321+0000 7f0049007640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f004409dc20 con 0x7f004415b2a0 2023-10-19T13:00:42.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.321+0000 7f0049007640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f004409bc60 con 0x7f00440d1910 2023-10-19T13:00:42.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.321+0000 7f0042d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f00440cf340 0x7f004409d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:42.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.321+0000 7f0042d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f00440cf340 0x7f004409d6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:47704/0 (socket says 172.21.15.57:47704) 2023-10-19T13:00:42.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.321+0000 7f004259c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f00440d1910 0x7f004409b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:42.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.321+0000 7f004359e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f004415b2a0 0x7f004409dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:42.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.321+0000 7f0042d9d640 1 -- 172.21.15.57:0/3651859028 learned_addr learned my addr 172.21.15.57:0/3651859028 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:42.327 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:00:42.321+0000 7f918bc30040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-19T13:00:42.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.321+0000 7f004259c640 1 -- 172.21.15.57:0/3651859028 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f004415b2a0 msgr2=0x7f004409dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:42.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.321+0000 7f004259c640 1 --2- 172.21.15.57:0/3651859028 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f004415b2a0 0x7f004409dfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:42.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.321+0000 7f004259c640 1 -- 172.21.15.57:0/3651859028 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f00440cf340 msgr2=0x7f004409d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:42.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.321+0000 7f004259c640 1 --2- 172.21.15.57:0/3651859028 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f00440cf340 0x7f004409d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:42.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.321+0000 7f004259c640 1 -- 172.21.15.57:0/3651859028 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f003c009840 con 0x7f00440d1910 2023-10-19T13:00:42.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.321+0000 7f0042d9d640 1 --2- 172.21.15.57:0/3651859028 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f00440cf340 0x7f004409d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T13:00:42.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.321+0000 7f004259c640 1 --2- 172.21.15.57:0/3651859028 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f00440d1910 0x7f004409b720 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f003800c390 tx=0x7f0038001e70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:42.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.321+0000 7f002b7fe640 1 -- 172.21.15.57:0/3651859028 <== mon.2 v2:172.21.15.179:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00380052d0 con 0x7f00440d1910 2023-10-19T13:00:42.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.321+0000 7f002b7fe640 1 -- 172.21.15.57:0/3651859028 <== mon.2 v2:172.21.15.179:3301/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f00380035f0 con 0x7f00440d1910 2023-10-19T13:00:42.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.322+0000 7f0049007640 1 -- 172.21.15.57:0/3651859028 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f004415e390 con 0x7f00440d1910 2023-10-19T13:00:42.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.322+0000 7f002b7fe640 1 -- 172.21.15.57:0/3651859028 <== mon.2 v2:172.21.15.179:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0038003a40 con 0x7f00440d1910 2023-10-19T13:00:42.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.322+0000 7f0049007640 1 -- 172.21.15.57:0/3651859028 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f004415bb50 con 0x7f00440d1910 2023-10-19T13:00:42.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.323+0000 7f0049007640 1 -- 172.21.15.57:0/3651859028 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f004415b6d0 con 0x7f00440d1910 2023-10-19T13:00:42.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.323+0000 7f002b7fe640 1 -- 172.21.15.57:0/3651859028 <== mon.2 v2:172.21.15.179:3301/0 4 ==== mgrmap(e 37) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f0038003ba0 con 0x7f00440d1910 2023-10-19T13:00:42.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.323+0000 7f002b7fe640 1 -- 172.21.15.57:0/3651859028 <== mon.2 v2:172.21.15.179:3301/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6745+0+0 (secure 0 0 0) 0x7f0038088be0 con 0x7f00440d1910 2023-10-19T13:00:42.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.328+0000 7f002b7fe640 1 -- 172.21.15.57:0/3651859028 <== mon.2 v2:172.21.15.179:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f004415b6d0 con 0x7f00440d1910 2023-10-19T13:00:42.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.504+0000 7f0049007640 1 -- 172.21.15.57:0/3651859028 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f004415b6d0 con 0x7f00440d1910 2023-10-19T13:00:42.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.505+0000 7f002b7fe640 1 -- 172.21.15.57:0/3651859028 <== mon.2 v2:172.21.15.179:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) v1 ==== 100+0+11 (secure 0 0 0) 0x7f004415b6d0 con 0x7f00440d1910 2023-10-19T13:00:42.508 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 64 2023-10-19T13:00:42.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.506+0000 7f0049007640 1 -- 172.21.15.57:0/3651859028 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f00440d1910 msgr2=0x7f004409b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:42.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.506+0000 7f0049007640 1 --2- 172.21.15.57:0/3651859028 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f00440d1910 0x7f004409b720 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f003800c390 tx=0x7f0038001e70 comp rx=0 tx=0).stop 2023-10-19T13:00:42.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.507+0000 7f0049007640 1 -- 172.21.15.57:0/3651859028 shutdown_connections 2023-10-19T13:00:42.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.507+0000 7f0049007640 1 --2- 172.21.15.57:0/3651859028 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f004415b2a0 0x7f004409dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:42.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.507+0000 7f0049007640 1 --2- 172.21.15.57:0/3651859028 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f00440d1910 0x7f004409b720 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:42.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.507+0000 7f0049007640 1 --2- 172.21.15.57:0/3651859028 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f00440cf340 0x7f004409d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:42.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.507+0000 7f0049007640 1 -- 172.21.15.57:0/3651859028 >> 172.21.15.57:0/3651859028 conn(0x7f00440d0d60 msgr2=0x7f004406b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:42.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.507+0000 7f0049007640 1 -- 172.21.15.57:0/3651859028 shutdown_connections 2023-10-19T13:00:42.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.507+0000 7f0049007640 1 -- 172.21.15.57:0/3651859028 wait complete. 2023-10-19T13:00:42.520 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T13:00:42.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.666+0000 7f46d5573640 1 -- 172.21.15.57:0/194408804 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f46b80039a0 msgr2=0x7f46b8003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:42.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.666+0000 7f46d5573640 1 --2- 172.21.15.57:0/194408804 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f46b80039a0 0x7f46b8003dc0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f46c8009b80 tx=0x7f46b80057f0 comp rx=0 tx=0).stop 2023-10-19T13:00:42.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.666+0000 7f46cf7fe640 1 -- 172.21.15.57:0/194408804 reap_dead start 2023-10-19T13:00:42.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.666+0000 7f46d5573640 1 -- 172.21.15.57:0/194408804 shutdown_connections 2023-10-19T13:00:42.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.666+0000 7f46d5573640 1 -- 172.21.15.57:0/194408804 >> 172.21.15.57:0/194408804 conn(0x7f46d0069930 msgr2=0x7f46d0069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:42.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.666+0000 7f46d5573640 1 -- 172.21.15.57:0/194408804 shutdown_connections 2023-10-19T13:00:42.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.666+0000 7f46d5573640 1 -- 172.21.15.57:0/194408804 wait complete. 2023-10-19T13:00:42.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.667+0000 7f46d5573640 1 Processor -- start 2023-10-19T13:00:42.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.667+0000 7f46d5573640 1 -- start start 2023-10-19T13:00:42.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.668+0000 7f46d5573640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f46b80039a0 0x7f46d0139290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:42.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.668+0000 7f46d5573640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f46d00b1c20 0x7f46d01579a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:42.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.668+0000 7f46d5573640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f46d01556c0 0x7f46d0157ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:42.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.668+0000 7f46d5573640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46d0158420 con 0x7f46d01556c0 2023-10-19T13:00:42.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.668+0000 7f46d5573640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46d0154af0 con 0x7f46b80039a0 2023-10-19T13:00:42.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.668+0000 7f46d5573640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46d0153ad0 con 0x7f46d00b1c20 2023-10-19T13:00:42.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.668+0000 7f46cf7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f46d01556c0 0x7f46d0157ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:42.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.668+0000 7f46cf7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f46d01556c0 0x7f46d0157ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:47714/0 (socket says 172.21.15.57:47714) 2023-10-19T13:00:42.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.668+0000 7f46cf7fe640 1 -- 172.21.15.57:0/1376061868 learned_addr learned my addr 172.21.15.57:0/1376061868 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:42.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.668+0000 7f46ceffd640 1 --2- 172.21.15.57:0/1376061868 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f46b80039a0 0x7f46d0139290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:42.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.668+0000 7f46ce7fc640 1 --2- 172.21.15.57:0/1376061868 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f46d00b1c20 0x7f46d01579a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:42.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.668+0000 7f46cf7fe640 1 -- 172.21.15.57:0/1376061868 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f46d00b1c20 msgr2=0x7f46d01579a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:42.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.668+0000 7f46cf7fe640 1 --2- 172.21.15.57:0/1376061868 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f46d00b1c20 0x7f46d01579a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:42.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.668+0000 7f46cf7fe640 1 -- 172.21.15.57:0/1376061868 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f46b80039a0 msgr2=0x7f46d0139290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:42.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.668+0000 7f46cf7fe640 1 --2- 172.21.15.57:0/1376061868 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f46b80039a0 0x7f46d0139290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:42.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.668+0000 7f46cf7fe640 1 -- 172.21.15.57:0/1376061868 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46d0153ee0 con 0x7f46d01556c0 2023-10-19T13:00:42.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.669+0000 7f46cf7fe640 1 --2- 172.21.15.57:0/1376061868 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f46d01556c0 0x7f46d0157ee0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f46d006d560 tx=0x7f46c000f3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:42.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.669+0000 7f46b77fe640 1 -- 172.21.15.57:0/1376061868 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c0004b20 con 0x7f46d01556c0 2023-10-19T13:00:42.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.669+0000 7f46b77fe640 1 -- 172.21.15.57:0/1376061868 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f46c0004c80 con 0x7f46d01556c0 2023-10-19T13:00:42.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.669+0000 7f46d5573640 1 -- 172.21.15.57:0/1376061868 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46d01541c0 con 0x7f46d01556c0 2023-10-19T13:00:42.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.669+0000 7f46b77fe640 1 -- 172.21.15.57:0/1376061868 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c0003bd0 con 0x7f46d01556c0 2023-10-19T13:00:42.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.669+0000 7f46d5573640 1 -- 172.21.15.57:0/1376061868 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46d0154780 con 0x7f46d01556c0 2023-10-19T13:00:42.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.670+0000 7f46d5573640 1 -- 172.21.15.57:0/1376061868 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46d0153670 con 0x7f46d01556c0 2023-10-19T13:00:42.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.671+0000 7f46b77fe640 1 -- 172.21.15.57:0/1376061868 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 37) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f46c0003d30 con 0x7f46d01556c0 2023-10-19T13:00:42.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.671+0000 7f46b77fe640 1 -- 172.21.15.57:0/1376061868 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6745+0+0 (secure 0 0 0) 0x7f46c0089c20 con 0x7f46d01556c0 2023-10-19T13:00:42.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.675+0000 7f46b77fe640 1 -- 172.21.15.57:0/1376061868 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f46c0054dc0 con 0x7f46d01556c0 2023-10-19T13:00:42.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.887+0000 7f46d5573640 1 -- 172.21.15.57:0/1376061868 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f46d01599c0 con 0x7f46d01556c0 2023-10-19T13:00:42.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.887+0000 7f46b77fe640 1 -- 172.21.15.57:0/1376061868 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+4917 (secure 0 0 0) 0x7f46c00583e0 con 0x7f46d01556c0 2023-10-19T13:00:42.891 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:00:42.891 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":30,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5966,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/765319531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":765319531},{"type":"v1","addr":"172.21.15.179:6837","nonce":765319531}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27},{"gid":5969,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/1388526553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":1388526553},{"type":"v1","addr":"172.21.15.179:6836","nonce":1388526553}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27},{"gid":7035,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/3237434259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":3237434259},{"type":"v1","addr":"172.21.15.57:6834","nonce":3237434259}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T13:00:20.549275+0000","modified":"2023-10-19T13:00:22.561271+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7038},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7038":{"gid":7038,"name":"a","rank":0,"incarnation":28,"state":"up:active","state_seq":3,"addr":"172.21.15.57:6835/2257211800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2257211800},{"type":"v1","addr":"172.21.15.57:6835","nonce":2257211800}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-10-19T13:00:42.891 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 30 2023-10-19T13:00:42.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.889+0000 7f46d5573640 1 -- 172.21.15.57:0/1376061868 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f46d01556c0 msgr2=0x7f46d0157ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:42.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.889+0000 7f46d5573640 1 --2- 172.21.15.57:0/1376061868 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f46d01556c0 0x7f46d0157ee0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f46d006d560 tx=0x7f46c000f3c0 comp rx=0 tx=0).stop 2023-10-19T13:00:42.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.890+0000 7f46d5573640 1 -- 172.21.15.57:0/1376061868 shutdown_connections 2023-10-19T13:00:42.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.890+0000 7f46d5573640 1 --2- 172.21.15.57:0/1376061868 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f46d01556c0 0x7f46d0157ee0 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:42.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.890+0000 7f46d5573640 1 --2- 172.21.15.57:0/1376061868 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f46d00b1c20 0x7f46d01579a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:42.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.890+0000 7f46d5573640 1 --2- 172.21.15.57:0/1376061868 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f46b80039a0 0x7f46d0139290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:42.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.890+0000 7f46d5573640 1 -- 172.21.15.57:0/1376061868 >> 172.21.15.57:0/1376061868 conn(0x7f46d0069930 msgr2=0x7f46d015d030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:42.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.890+0000 7f46d5573640 1 -- 172.21.15.57:0/1376061868 shutdown_connections 2023-10-19T13:00:42.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:42.890+0000 7f46d5573640 1 -- 172.21.15.57:0/1376061868 wait complete. 2023-10-19T13:00:42.903 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T13:00:43.012 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.009+0000 7febf8610640 1 -- 172.21.15.57:0/1433509533 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7febf0141690 msgr2=0x7febf00703e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:43.012 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.009+0000 7febf8610640 1 --2- 172.21.15.57:0/1433509533 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7febf0141690 0x7febf00703e0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7febe4005fd0 tx=0x7febe402bc60 comp rx=0 tx=0).stop 2023-10-19T13:00:43.012 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.009+0000 7febf8610640 1 -- 172.21.15.57:0/1433509533 shutdown_connections 2023-10-19T13:00:43.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.009+0000 7febf8610640 1 --2- 172.21.15.57:0/1433509533 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7febf0141690 0x7febf00703e0 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:43.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.009+0000 7febf8610640 1 --2- 172.21.15.57:0/1433509533 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7febf013e520 0x7febf006bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:43.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.009+0000 7febf8610640 1 -- 172.21.15.57:0/1433509533 >> 172.21.15.57:0/1433509533 conn(0x7febf0069930 msgr2=0x7febf0069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:43.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.009+0000 7febf8610640 1 -- 172.21.15.57:0/1433509533 shutdown_connections 2023-10-19T13:00:43.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.009+0000 7febf8610640 1 -- 172.21.15.57:0/1433509533 wait complete. 2023-10-19T13:00:43.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.010+0000 7febf8610640 1 Processor -- start 2023-10-19T13:00:43.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.010+0000 7febf8610640 1 -- start start 2023-10-19T13:00:43.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.011+0000 7febf8610640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7febf006c0a0 0x7febf00abc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:43.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.011+0000 7febf8610640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7febf013e520 0x7febf01e2a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:43.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.011+0000 7febf8610640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7febf0141690 0x7febf01de9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:43.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.011+0000 7febf8610640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febf01e0fd0 con 0x7febf006c0a0 2023-10-19T13:00:43.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.011+0000 7febf8610640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febf00ab770 con 0x7febf0141690 2023-10-19T13:00:43.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.011+0000 7febf8610640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febf00aa8c0 con 0x7febf013e520 2023-10-19T13:00:43.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.011+0000 7febf63ac640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7febf006c0a0 0x7febf00abc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:43.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.011+0000 7febf63ac640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7febf006c0a0 0x7febf00abc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:47730/0 (socket says 172.21.15.57:47730) 2023-10-19T13:00:43.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.011+0000 7febf63ac640 1 -- 172.21.15.57:0/3229418999 learned_addr learned my addr 172.21.15.57:0/3229418999 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:43.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.011+0000 7febf6bad640 1 --2- 172.21.15.57:0/3229418999 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7febf0141690 0x7febf01de9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:43.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.011+0000 7febf5bab640 1 --2- 172.21.15.57:0/3229418999 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7febf013e520 0x7febf01e2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:43.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.011+0000 7febf63ac640 1 -- 172.21.15.57:0/3229418999 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7febf013e520 msgr2=0x7febf01e2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:43.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.011+0000 7febf63ac640 1 --2- 172.21.15.57:0/3229418999 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7febf013e520 0x7febf01e2a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:43.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.011+0000 7febf63ac640 1 -- 172.21.15.57:0/3229418999 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7febf0141690 msgr2=0x7febf01de9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:43.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.011+0000 7febf63ac640 1 --2- 172.21.15.57:0/3229418999 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7febf0141690 0x7febf01de9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:43.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.011+0000 7febf63ac640 1 -- 172.21.15.57:0/3229418999 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febe4009560 con 0x7febf006c0a0 2023-10-19T13:00:43.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.011+0000 7febf6bad640 1 --2- 172.21.15.57:0/3229418999 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7febf0141690 0x7febf01de9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:00:43.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.012+0000 7febf63ac640 1 --2- 172.21.15.57:0/3229418999 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7febf006c0a0 0x7febf00abc00 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7febe800c920 tx=0x7febe800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:43.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.012+0000 7febeeffd640 1 -- 172.21.15.57:0/3229418999 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febe8007c70 con 0x7febf006c0a0 2023-10-19T13:00:43.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.012+0000 7febeeffd640 1 -- 172.21.15.57:0/3229418999 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7febe800f450 con 0x7febf006c0a0 2023-10-19T13:00:43.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.012+0000 7febf8610640 1 -- 172.21.15.57:0/3229418999 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febf00aab60 con 0x7febf006c0a0 2023-10-19T13:00:43.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.012+0000 7febeeffd640 1 -- 172.21.15.57:0/3229418999 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febe800f860 con 0x7febf006c0a0 2023-10-19T13:00:43.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.012+0000 7febf8610640 1 -- 172.21.15.57:0/3229418999 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febf00ab400 con 0x7febf006c0a0 2023-10-19T13:00:43.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.013+0000 7febf8610640 1 -- 172.21.15.57:0/3229418999 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febf00690e0 con 0x7febf006c0a0 2023-10-19T13:00:43.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.013+0000 7febeeffd640 1 -- 172.21.15.57:0/3229418999 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 37) v1 ==== 88940+0+0 (secure 0 0 0) 0x7febe800f9c0 con 0x7febf006c0a0 2023-10-19T13:00:43.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.014+0000 7febeeffd640 1 -- 172.21.15.57:0/3229418999 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6745+0+0 (secure 0 0 0) 0x7febe8089e70 con 0x7febf006c0a0 2023-10-19T13:00:43.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.018+0000 7febeeffd640 1 -- 172.21.15.57:0/3229418999 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7febe8054e70 con 0x7febf006c0a0 2023-10-19T13:00:43.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.069+0000 7febeeffd640 1 -- 172.21.15.57:0/3229418999 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mgrmap(e 38) v1 ==== 89067+0+0 (secure 0 0 0) 0x7febe8054920 con 0x7febf006c0a0 2023-10-19T13:00:43.072 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.069+0000 7febeeffd640 1 --2- 172.21.15.57:0/3229418999 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7febdc070b80 0x7febdc072f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:43.074 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.072+0000 7febf5bab640 1 --2- 172.21.15.57:0/3229418999 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7febdc070b80 0x7febdc072f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:43.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.075+0000 7febf5bab640 1 --2- 172.21.15.57:0/3229418999 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7febdc070b80 0x7febdc072f70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7febe4035000 tx=0x7febe4004730 comp rx=0 tx=0).ready entity=mgr.6679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:43.202 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.199+0000 7febf8610640 1 -- 172.21.15.57:0/3229418999 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7febf01e34e0 con 0x7febf006c0a0 2023-10-19T13:00:43.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.201+0000 7febeeffd640 1 -- 172.21.15.57:0/3229418999 <== mon.0 v2:172.21.15.57:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+17787 (secure 0 0 0) 0x7febe8073e60 con 0x7febf006c0a0 2023-10-19T13:00:43.204 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:00:43.204 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":89,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T13:00:42.059971+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2023-10-19T13:00:18.452198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2023-10-19T13:00:19.531525+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.179:0/4135533550":"2023-10-20T13:00:42.059915+0000","172.21.15.179:0/3560014978":"2023-10-20T13:00:42.059915+0000","172.21.15.179:6833/137854273":"2023-10-20T13:00:42.059915+0000","172.21.15.179:6832/137854273":"2023-10-20T13:00:42.059915+0000","172.21.15.179:0/4157624230":"2023-10-20T13:00:42.059915+0000","172.21.15.179:0/3436591908":"2023-10-20T13:00:42.059915+0000","172.21.15.179:0/911511055":"2023-10-20T13:00:42.059915+0000","172.21.15.179:0/4093253624":"2023-10-20T13:00:36.611894+0000","172.21.15.179:6833/3815793798":"2023-10-20T13:00:36.611894+0000","172.21.15.179:0/2495886194":"2023-10-20T13:00:36.611894+0000","172.21.15.179:6832/3815793798":"2023-10-20T13:00:36.611894+0000","172.21.15.179:0/2414172521":"2023-10-20T13:00:36.611894+0000","172.21.15.179:0/4199202801":"2023-10-20T13:00:42.059915+0000","172.21.15.179:0/2931605339":"2023-10-20T13:00:36.611894+0000","172.21.15.179:0/1988099910":"2023-10-20T13:00:36.611894+0000","172.21.15.179:0/4264007972":"2023-10-20T13:00:36.611894+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T13:00:43.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.203+0000 7febf8610640 1 -- 172.21.15.57:0/3229418999 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7febdc070b80 msgr2=0x7febdc072f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:43.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.203+0000 7febf8610640 1 --2- 172.21.15.57:0/3229418999 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7febdc070b80 0x7febdc072f70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7febe4035000 tx=0x7febe4004730 comp rx=0 tx=0).stop 2023-10-19T13:00:43.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.203+0000 7febf8610640 1 -- 172.21.15.57:0/3229418999 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7febf006c0a0 msgr2=0x7febf00abc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:43.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.203+0000 7febf8610640 1 --2- 172.21.15.57:0/3229418999 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7febf006c0a0 0x7febf00abc00 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7febe800c920 tx=0x7febe800cde0 comp rx=0 tx=0).stop 2023-10-19T13:00:43.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.203+0000 7febf8610640 1 -- 172.21.15.57:0/3229418999 shutdown_connections 2023-10-19T13:00:43.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.203+0000 7febf8610640 1 --2- 172.21.15.57:0/3229418999 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7febdc070b80 0x7febdc072f70 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:43.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.203+0000 7febf8610640 1 --2- 172.21.15.57:0/3229418999 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7febf0141690 0x7febf01de9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:43.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.203+0000 7febf8610640 1 --2- 172.21.15.57:0/3229418999 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7febf013e520 0x7febf01e2a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:43.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.203+0000 7febf8610640 1 --2- 172.21.15.57:0/3229418999 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7febf006c0a0 0x7febf00abc00 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:43.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.203+0000 7febf8610640 1 -- 172.21.15.57:0/3229418999 >> 172.21.15.57:0/3229418999 conn(0x7febf0069930 msgr2=0x7febf01dfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:43.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.203+0000 7febf8610640 1 -- 172.21.15.57:0/3229418999 shutdown_connections 2023-10-19T13:00:43.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.203+0000 7febf8610640 1 -- 172.21.15.57:0/3229418999 wait complete. 2023-10-19T13:00:43.216 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-19T13:00:43.217 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T13:00:43.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.323+0000 7fd249161640 1 -- 172.21.15.57:0/3479011862 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd2440d1910 msgr2=0x7fd2440b0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:43.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.323+0000 7fd249161640 1 --2- 172.21.15.57:0/3479011862 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd2440d1910 0x7fd2440b0650 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7fd23c009830 tx=0x7fd23c02bd60 comp rx=0 tx=0).stop 2023-10-19T13:00:43.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.323+0000 7fd249161640 1 -- 172.21.15.57:0/3479011862 shutdown_connections 2023-10-19T13:00:43.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.323+0000 7fd249161640 1 --2- 172.21.15.57:0/3479011862 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd2440d1910 0x7fd2440b0650 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:43.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.323+0000 7fd249161640 1 -- 172.21.15.57:0/3479011862 >> 172.21.15.57:0/3479011862 conn(0x7fd2440d0d60 msgr2=0x7fd2440d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:43.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.323+0000 7fd249161640 1 -- 172.21.15.57:0/3479011862 shutdown_connections 2023-10-19T13:00:43.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.323+0000 7fd249161640 1 -- 172.21.15.57:0/3479011862 wait complete. 2023-10-19T13:00:43.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.324+0000 7fd249161640 1 Processor -- start 2023-10-19T13:00:43.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.324+0000 7fd249161640 1 -- start start 2023-10-19T13:00:43.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.325+0000 7fd249161640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd2440cf340 0x7fd24414f230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:43.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.325+0000 7fd249161640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd2440d1910 0x7fd2440ca440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:43.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.325+0000 7fd249161640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd24415b2a0 0x7fd2440b0da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:43.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.325+0000 7fd249161640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd244157a50 con 0x7fd2440cf340 2023-10-19T13:00:43.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.325+0000 7fd249161640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd24414f770 con 0x7fd24415b2a0 2023-10-19T13:00:43.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.325+0000 7fd249161640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd24415ccc0 con 0x7fd2440d1910 2023-10-19T13:00:43.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.325+0000 7fd242d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd2440cf340 0x7fd24414f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:43.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.325+0000 7fd242d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd2440cf340 0x7fd24414f230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:47740/0 (socket says 172.21.15.57:47740) 2023-10-19T13:00:43.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.325+0000 7fd242d9d640 1 -- 172.21.15.57:0/418351603 learned_addr learned my addr 172.21.15.57:0/418351603 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:43.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.325+0000 7fd24359e640 1 --2- 172.21.15.57:0/418351603 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd24415b2a0 0x7fd2440b0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:43.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.325+0000 7fd24259c640 1 --2- 172.21.15.57:0/418351603 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd2440d1910 0x7fd2440ca440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:43.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.326+0000 7fd242d9d640 1 -- 172.21.15.57:0/418351603 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd2440d1910 msgr2=0x7fd2440ca440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:43.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.326+0000 7fd242d9d640 1 --2- 172.21.15.57:0/418351603 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd2440d1910 0x7fd2440ca440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:43.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.326+0000 7fd242d9d640 1 -- 172.21.15.57:0/418351603 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd24415b2a0 msgr2=0x7fd2440b0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:43.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.326+0000 7fd242d9d640 1 --2- 172.21.15.57:0/418351603 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd24415b2a0 0x7fd2440b0da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:43.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.326+0000 7fd242d9d640 1 -- 172.21.15.57:0/418351603 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd23c009490 con 0x7fd2440cf340 2023-10-19T13:00:43.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.326+0000 7fd242d9d640 1 --2- 172.21.15.57:0/418351603 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd2440cf340 0x7fd24414f230 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7fd23400c7c0 tx=0x7fd23400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:43.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.326+0000 7fd23b7fe640 1 -- 172.21.15.57:0/418351603 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd234006d50 con 0x7fd2440cf340 2023-10-19T13:00:43.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.326+0000 7fd23b7fe640 1 -- 172.21.15.57:0/418351603 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fd234003970 con 0x7fd2440cf340 2023-10-19T13:00:43.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.326+0000 7fd249161640 1 -- 172.21.15.57:0/418351603 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2440ca980 con 0x7fd2440cf340 2023-10-19T13:00:43.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.326+0000 7fd23b7fe640 1 -- 172.21.15.57:0/418351603 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd234003d80 con 0x7fd2440cf340 2023-10-19T13:00:43.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.326+0000 7fd249161640 1 -- 172.21.15.57:0/418351603 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2440caf20 con 0x7fd2440cf340 2023-10-19T13:00:43.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.327+0000 7fd249161640 1 -- 172.21.15.57:0/418351603 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2440690e0 con 0x7fd2440cf340 2023-10-19T13:00:43.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.327+0000 7fd23b7fe640 1 -- 172.21.15.57:0/418351603 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 38) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fd23400a850 con 0x7fd2440cf340 2023-10-19T13:00:43.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.328+0000 7fd23b7fe640 1 --2- 172.21.15.57:0/418351603 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fd22406ac90 0x7fd22406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:43.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.328+0000 7fd23b7fe640 1 -- 172.21.15.57:0/418351603 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6745+0+0 (secure 0 0 0) 0x7fd2340893c0 con 0x7fd2440cf340 2023-10-19T13:00:43.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.328+0000 7fd24259c640 1 --2- 172.21.15.57:0/418351603 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fd22406ac90 0x7fd22406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:43.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.329+0000 7fd24259c640 1 --2- 172.21.15.57:0/418351603 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fd22406ac90 0x7fd22406d150 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fd2440b2310 tx=0x7fd22c007d20 comp rx=0 tx=0).ready entity=mgr.6679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:43.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.332+0000 7fd23b7fe640 1 -- 172.21.15.57:0/418351603 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd234054450 con 0x7fd2440cf340 2023-10-19T13:00:43.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.548+0000 7fd249161640 1 -- 172.21.15.57:0/418351603 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd2440d0b40 con 0x7fd2440cf340 2023-10-19T13:00:43.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.548+0000 7fd23b7fe640 1 -- 172.21.15.57:0/418351603 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+4917 (secure 0 0 0) 0x7fd234057a70 con 0x7fd2440cf340 2023-10-19T13:00:43.552 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:00:43.552 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":30,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5966,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/765319531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":765319531},{"type":"v1","addr":"172.21.15.179:6837","nonce":765319531}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27},{"gid":5969,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/1388526553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":1388526553},{"type":"v1","addr":"172.21.15.179:6836","nonce":1388526553}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27},{"gid":7035,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/3237434259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":3237434259},{"type":"v1","addr":"172.21.15.57:6834","nonce":3237434259}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T13:00:20.549275+0000","modified":"2023-10-19T13:00:22.561271+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7038},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7038":{"gid":7038,"name":"a","rank":0,"incarnation":28,"state":"up:active","state_seq":3,"addr":"172.21.15.57:6835/2257211800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2257211800},{"type":"v1","addr":"172.21.15.57:6835","nonce":2257211800}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-10-19T13:00:43.552 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 30 2023-10-19T13:00:43.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.550+0000 7fd249161640 1 -- 172.21.15.57:0/418351603 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fd22406ac90 msgr2=0x7fd22406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:43.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.550+0000 7fd249161640 1 --2- 172.21.15.57:0/418351603 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fd22406ac90 0x7fd22406d150 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fd2440b2310 tx=0x7fd22c007d20 comp rx=0 tx=0).stop 2023-10-19T13:00:43.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.550+0000 7fd249161640 1 -- 172.21.15.57:0/418351603 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd2440cf340 msgr2=0x7fd24414f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:43.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.550+0000 7fd249161640 1 --2- 172.21.15.57:0/418351603 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd2440cf340 0x7fd24414f230 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7fd23400c7c0 tx=0x7fd23400cc80 comp rx=0 tx=0).stop 2023-10-19T13:00:43.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.550+0000 7fd249161640 1 -- 172.21.15.57:0/418351603 shutdown_connections 2023-10-19T13:00:43.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.550+0000 7fd249161640 1 --2- 172.21.15.57:0/418351603 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fd22406ac90 0x7fd22406d150 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:43.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.550+0000 7fd249161640 1 --2- 172.21.15.57:0/418351603 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd24415b2a0 0x7fd2440b0da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:43.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.550+0000 7fd249161640 1 --2- 172.21.15.57:0/418351603 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd2440d1910 0x7fd2440ca440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:43.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.550+0000 7fd249161640 1 --2- 172.21.15.57:0/418351603 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd2440cf340 0x7fd24414f230 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:43.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.550+0000 7fd249161640 1 -- 172.21.15.57:0/418351603 >> 172.21.15.57:0/418351603 conn(0x7fd2440d0d60 msgr2=0x7fd24406b5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:43.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.551+0000 7fd249161640 1 -- 172.21.15.57:0/418351603 shutdown_connections 2023-10-19T13:00:43.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.551+0000 7fd249161640 1 -- 172.21.15.57:0/418351603 wait complete. 2023-10-19T13:00:43.564 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T13:00:43.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.670+0000 7ff679c64640 1 -- 172.21.15.57:0/1419206643 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff6740b0c40 msgr2=0x7ff674158300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:43.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.670+0000 7ff679c64640 1 --2- 172.21.15.57:0/1419206643 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff6740b0c40 0x7ff674158300 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7ff66c005fd0 tx=0x7ff66c02c2e0 comp rx=0 tx=0).stop 2023-10-19T13:00:43.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.670+0000 7ff679c64640 1 -- 172.21.15.57:0/1419206643 shutdown_connections 2023-10-19T13:00:43.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.670+0000 7ff679c64640 1 --2- 172.21.15.57:0/1419206643 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff6740b0c40 0x7ff674158300 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:43.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.670+0000 7ff679c64640 1 -- 172.21.15.57:0/1419206643 >> 172.21.15.57:0/1419206643 conn(0x7ff674069930 msgr2=0x7ff674069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:43.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.670+0000 7ff679c64640 1 -- 172.21.15.57:0/1419206643 shutdown_connections 2023-10-19T13:00:43.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.670+0000 7ff679c64640 1 -- 172.21.15.57:0/1419206643 wait complete. 2023-10-19T13:00:43.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.671+0000 7ff679c64640 1 Processor -- start 2023-10-19T13:00:43.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.671+0000 7ff679c64640 1 -- start start 2023-10-19T13:00:43.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.672+0000 7ff679c64640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff6740afa20 0x7ff67413eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:43.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.672+0000 7ff679c64640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff6740b0c40 0x7ff674142bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:43.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.672+0000 7ff679c64640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff6740cbd20 0x7ff67413fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:43.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.672+0000 7ff679c64640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff674143130 con 0x7ff6740afa20 2023-10-19T13:00:43.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.672+0000 7ff679c64640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff674145a90 con 0x7ff6740cbd20 2023-10-19T13:00:43.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.672+0000 7ff679c64640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff674146610 con 0x7ff6740b0c40 2023-10-19T13:00:43.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.672+0000 7ff6737fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff6740afa20 0x7ff67413eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:43.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.672+0000 7ff6737fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff6740afa20 0x7ff67413eda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:47758/0 (socket says 172.21.15.57:47758) 2023-10-19T13:00:43.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.672+0000 7ff6737fe640 1 -- 172.21.15.57:0/2650286573 learned_addr learned my addr 172.21.15.57:0/2650286573 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:43.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.673+0000 7ff6737fe640 1 -- 172.21.15.57:0/2650286573 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff6740b0c40 msgr2=0x7ff674142bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:43.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.673+0000 7ff673fff640 1 --2- 172.21.15.57:0/2650286573 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff6740cbd20 0x7ff67413fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:43.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.673+0000 7ff672ffd640 1 --2- 172.21.15.57:0/2650286573 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff6740b0c40 0x7ff674142bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:43.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.673+0000 7ff6737fe640 1 --2- 172.21.15.57:0/2650286573 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff6740b0c40 0x7ff674142bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:43.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.673+0000 7ff6737fe640 1 -- 172.21.15.57:0/2650286573 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff6740cbd20 msgr2=0x7ff67413fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:43.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.673+0000 7ff6737fe640 1 --2- 172.21.15.57:0/2650286573 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff6740cbd20 0x7ff67413fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:43.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.673+0000 7ff6737fe640 1 -- 172.21.15.57:0/2650286573 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff66c009900 con 0x7ff6740afa20 2023-10-19T13:00:43.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.673+0000 7ff673fff640 1 --2- 172.21.15.57:0/2650286573 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff6740cbd20 0x7ff67413fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T13:00:43.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.673+0000 7ff6737fe640 1 --2- 172.21.15.57:0/2650286573 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff6740afa20 0x7ff67413eda0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7ff66400d990 tx=0x7ff66400de50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:43.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.673+0000 7ff65bfff640 1 -- 172.21.15.57:0/2650286573 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff66400ed40 con 0x7ff6740afa20 2023-10-19T13:00:43.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.673+0000 7ff65bfff640 1 -- 172.21.15.57:0/2650286573 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7ff664016440 con 0x7ff6740afa20 2023-10-19T13:00:43.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.673+0000 7ff679c64640 1 -- 172.21.15.57:0/2650286573 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff674145ce0 con 0x7ff6740afa20 2023-10-19T13:00:43.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.673+0000 7ff65bfff640 1 -- 172.21.15.57:0/2650286573 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff664016850 con 0x7ff6740afa20 2023-10-19T13:00:43.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.673+0000 7ff679c64640 1 -- 172.21.15.57:0/2650286573 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6741462a0 con 0x7ff6740afa20 2023-10-19T13:00:43.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.674+0000 7ff679c64640 1 -- 172.21.15.57:0/2650286573 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6740690e0 con 0x7ff6740afa20 2023-10-19T13:00:43.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.674+0000 7ff65bfff640 1 -- 172.21.15.57:0/2650286573 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 38) v1 ==== 89067+0+0 (secure 0 0 0) 0x7ff66401f3b0 con 0x7ff6740afa20 2023-10-19T13:00:43.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.675+0000 7ff65bfff640 1 --2- 172.21.15.57:0/2650286573 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7ff66006ac90 0x7ff66006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:43.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.675+0000 7ff65bfff640 1 -- 172.21.15.57:0/2650286573 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6745+0+0 (secure 0 0 0) 0x7ff66408b910 con 0x7ff6740afa20 2023-10-19T13:00:43.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.675+0000 7ff672ffd640 1 --2- 172.21.15.57:0/2650286573 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7ff66006ac90 0x7ff66006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:43.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.676+0000 7ff672ffd640 1 --2- 172.21.15.57:0/2650286573 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7ff66006ac90 0x7ff66006d150 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff67414b940 tx=0x7ff668009040 comp rx=0 tx=0).ready entity=mgr.6679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:43.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.679+0000 7ff65bfff640 1 -- 172.21.15.57:0/2650286573 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff6640568f0 con 0x7ff6740afa20 2023-10-19T13:00:43.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.891+0000 7ff679c64640 1 -- 172.21.15.57:0/2650286573 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff67415a7b0 con 0x7ff6740afa20 2023-10-19T13:00:43.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.892+0000 7ff65bfff640 1 -- 172.21.15.57:0/2650286573 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+4917 (secure 0 0 0) 0x7ff66401b090 con 0x7ff6740afa20 2023-10-19T13:00:43.895 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:00:43.895 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":30,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5966,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/765319531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":765319531},{"type":"v1","addr":"172.21.15.179:6837","nonce":765319531}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27},{"gid":5969,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/1388526553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":1388526553},{"type":"v1","addr":"172.21.15.179:6836","nonce":1388526553}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27},{"gid":7035,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/3237434259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":3237434259},{"type":"v1","addr":"172.21.15.57:6834","nonce":3237434259}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T13:00:20.549275+0000","modified":"2023-10-19T13:00:22.561271+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7038},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7038":{"gid":7038,"name":"a","rank":0,"incarnation":28,"state":"up:active","state_seq":3,"addr":"172.21.15.57:6835/2257211800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2257211800},{"type":"v1","addr":"172.21.15.57:6835","nonce":2257211800}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-10-19T13:00:43.897 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 30 2023-10-19T13:00:43.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.893+0000 7ff679c64640 1 -- 172.21.15.57:0/2650286573 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7ff66006ac90 msgr2=0x7ff66006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:43.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.893+0000 7ff679c64640 1 --2- 172.21.15.57:0/2650286573 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7ff66006ac90 0x7ff66006d150 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff67414b940 tx=0x7ff668009040 comp rx=0 tx=0).stop 2023-10-19T13:00:43.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.893+0000 7ff679c64640 1 -- 172.21.15.57:0/2650286573 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff6740afa20 msgr2=0x7ff67413eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:43.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.893+0000 7ff679c64640 1 --2- 172.21.15.57:0/2650286573 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff6740afa20 0x7ff67413eda0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7ff66400d990 tx=0x7ff66400de50 comp rx=0 tx=0).stop 2023-10-19T13:00:43.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.894+0000 7ff679c64640 1 -- 172.21.15.57:0/2650286573 shutdown_connections 2023-10-19T13:00:43.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.894+0000 7ff679c64640 1 --2- 172.21.15.57:0/2650286573 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7ff66006ac90 0x7ff66006d150 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:43.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.894+0000 7ff679c64640 1 --2- 172.21.15.57:0/2650286573 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff6740cbd20 0x7ff67413fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:43.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.894+0000 7ff679c64640 1 --2- 172.21.15.57:0/2650286573 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff6740b0c40 0x7ff674142bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:43.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.894+0000 7ff679c64640 1 --2- 172.21.15.57:0/2650286573 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff6740afa20 0x7ff67413eda0 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:43.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.894+0000 7ff679c64640 1 -- 172.21.15.57:0/2650286573 >> 172.21.15.57:0/2650286573 conn(0x7ff674069930 msgr2=0x7ff674069d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:43.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.894+0000 7ff679c64640 1 -- 172.21.15.57:0/2650286573 shutdown_connections 2023-10-19T13:00:43.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:43.894+0000 7ff679c64640 1 -- 172.21.15.57:0/2650286573 wait complete. 2023-10-19T13:00:43.907 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T13:00:44.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.013+0000 7f196e14c640 1 -- 172.21.15.57:0/2967752896 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f196815b2a0 msgr2=0x7f19680d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:44.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.013+0000 7f196e14c640 1 --2- 172.21.15.57:0/2967752896 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f196815b2a0 0x7f19680d1910 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f195c009830 tx=0x7f195c02bcd0 comp rx=0 tx=0).stop 2023-10-19T13:00:44.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.014+0000 7f196e14c640 1 -- 172.21.15.57:0/2967752896 shutdown_connections 2023-10-19T13:00:44.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.014+0000 7f196e14c640 1 --2- 172.21.15.57:0/2967752896 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f19680ceb40 0x7f19680b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:44.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.014+0000 7f196e14c640 1 --2- 172.21.15.57:0/2967752896 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f196815b2a0 0x7f19680d1910 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:44.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.014+0000 7f196e14c640 1 -- 172.21.15.57:0/2967752896 >> 172.21.15.57:0/2967752896 conn(0x7f19680d0d60 msgr2=0x7f19680d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:44.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.014+0000 7f196e14c640 1 -- 172.21.15.57:0/2967752896 shutdown_connections 2023-10-19T13:00:44.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.014+0000 7f196e14c640 1 -- 172.21.15.57:0/2967752896 wait complete. 2023-10-19T13:00:44.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.015+0000 7f196e14c640 1 Processor -- start 2023-10-19T13:00:44.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.015+0000 7f196e14c640 1 -- start start 2023-10-19T13:00:44.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.016+0000 7f196e14c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f19680ceb40 0x7f1968151150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:44.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.016+0000 7f196e14c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f19680cf340 0x7f19680b4900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:44.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.016+0000 7f196e14c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f196815b2a0 0x7f196806a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:44.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.016+0000 7f196e14c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19680b52e0 con 0x7f19680cf340 2023-10-19T13:00:44.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.016+0000 7f196e14c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19680b55c0 con 0x7f196815b2a0 2023-10-19T13:00:44.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.016+0000 7f196e14c640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19680b58a0 con 0x7f19680ceb40 2023-10-19T13:00:44.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.016+0000 7f1966ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f19680cf340 0x7f19680b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:44.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.016+0000 7f1966ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f19680cf340 0x7f19680b4900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:47764/0 (socket says 172.21.15.57:47764) 2023-10-19T13:00:44.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.016+0000 7f1966ffd640 1 -- 172.21.15.57:0/3433500851 learned_addr learned my addr 172.21.15.57:0/3433500851 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:44.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.016+0000 7f19677fe640 1 --2- 172.21.15.57:0/3433500851 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f19680ceb40 0x7f1968151150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:44.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.016+0000 7f1967fff640 1 --2- 172.21.15.57:0/3433500851 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f196815b2a0 0x7f196806a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:44.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.016+0000 7f1966ffd640 1 -- 172.21.15.57:0/3433500851 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f19680ceb40 msgr2=0x7f1968151150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:44.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.016+0000 7f1966ffd640 1 --2- 172.21.15.57:0/3433500851 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f19680ceb40 0x7f1968151150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:44.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.016+0000 7f1966ffd640 1 -- 172.21.15.57:0/3433500851 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f196815b2a0 msgr2=0x7f196806a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:44.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.016+0000 7f1966ffd640 1 --2- 172.21.15.57:0/3433500851 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f196815b2a0 0x7f196806a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:44.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.016+0000 7f1966ffd640 1 -- 172.21.15.57:0/3433500851 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1960009900 con 0x7f19680cf340 2023-10-19T13:00:44.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.016+0000 7f1966ffd640 1 --2- 172.21.15.57:0/3433500851 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f19680cf340 0x7f19680b4900 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f195c000c00 tx=0x7f195c003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:44.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.016+0000 7f194ffff640 1 -- 172.21.15.57:0/3433500851 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f195c003e20 con 0x7f19680cf340 2023-10-19T13:00:44.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.016+0000 7f196e14c640 1 -- 172.21.15.57:0/3433500851 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f195c009490 con 0x7f19680cf340 2023-10-19T13:00:44.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.016+0000 7f194ffff640 1 -- 172.21.15.57:0/3433500851 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f195c034440 con 0x7f19680cf340 2023-10-19T13:00:44.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.016+0000 7f194ffff640 1 -- 172.21.15.57:0/3433500851 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f195c034850 con 0x7f19680cf340 2023-10-19T13:00:44.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.017+0000 7f196e14c640 1 -- 172.21.15.57:0/3433500851 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19680b5e60 con 0x7f19680cf340 2023-10-19T13:00:44.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.017+0000 7f196e14c640 1 -- 172.21.15.57:0/3433500851 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19680677b0 con 0x7f19680cf340 2023-10-19T13:00:44.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.018+0000 7f194ffff640 1 -- 172.21.15.57:0/3433500851 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 38) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f195c0349b0 con 0x7f19680cf340 2023-10-19T13:00:44.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.018+0000 7f194ffff640 1 --2- 172.21.15.57:0/3433500851 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f195406ac90 0x7f195406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:44.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.018+0000 7f194ffff640 1 -- 172.21.15.57:0/3433500851 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6745+0+0 (secure 0 0 0) 0x7f195c0abea0 con 0x7f19680cf340 2023-10-19T13:00:44.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.019+0000 7f19677fe640 1 --2- 172.21.15.57:0/3433500851 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f195406ac90 0x7f195406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:44.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.019+0000 7f19677fe640 1 --2- 172.21.15.57:0/3433500851 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f195406ac90 0x7f195406d150 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f1958001650 tx=0x7f1958009040 comp rx=0 tx=0).ready entity=mgr.6679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:44.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.023+0000 7f194ffff640 1 -- 172.21.15.57:0/3433500851 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f195c076e00 con 0x7f19680cf340 2023-10-19T13:00:44.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.204+0000 7f196e14c640 1 -- 172.21.15.57:0/3433500851 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f19680b0cd0 con 0x7f19680cf340 2023-10-19T13:00:44.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.205+0000 7f194ffff640 1 -- 172.21.15.57:0/3433500851 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+17787 (secure 0 0 0) 0x7f195c039090 con 0x7f19680cf340 2023-10-19T13:00:44.209 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:00:44.209 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":89,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T13:00:42.059971+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2023-10-19T13:00:18.452198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2023-10-19T13:00:19.531525+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.179:0/4135533550":"2023-10-20T13:00:42.059915+0000","172.21.15.179:0/3560014978":"2023-10-20T13:00:42.059915+0000","172.21.15.179:6833/137854273":"2023-10-20T13:00:42.059915+0000","172.21.15.179:6832/137854273":"2023-10-20T13:00:42.059915+0000","172.21.15.179:0/4157624230":"2023-10-20T13:00:42.059915+0000","172.21.15.179:0/3436591908":"2023-10-20T13:00:42.059915+0000","172.21.15.179:0/911511055":"2023-10-20T13:00:42.059915+0000","172.21.15.179:0/4093253624":"2023-10-20T13:00:36.611894+0000","172.21.15.179:6833/3815793798":"2023-10-20T13:00:36.611894+0000","172.21.15.179:0/2495886194":"2023-10-20T13:00:36.611894+0000","172.21.15.179:6832/3815793798":"2023-10-20T13:00:36.611894+0000","172.21.15.179:0/2414172521":"2023-10-20T13:00:36.611894+0000","172.21.15.179:0/4199202801":"2023-10-20T13:00:42.059915+0000","172.21.15.179:0/2931605339":"2023-10-20T13:00:36.611894+0000","172.21.15.179:0/1988099910":"2023-10-20T13:00:36.611894+0000","172.21.15.179:0/4264007972":"2023-10-20T13:00:36.611894+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T13:00:44.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.207+0000 7f196e14c640 1 -- 172.21.15.57:0/3433500851 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f195406ac90 msgr2=0x7f195406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:44.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.207+0000 7f196e14c640 1 --2- 172.21.15.57:0/3433500851 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f195406ac90 0x7f195406d150 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f1958001650 tx=0x7f1958009040 comp rx=0 tx=0).stop 2023-10-19T13:00:44.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.207+0000 7f196e14c640 1 -- 172.21.15.57:0/3433500851 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f19680cf340 msgr2=0x7f19680b4900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:44.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.207+0000 7f196e14c640 1 --2- 172.21.15.57:0/3433500851 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f19680cf340 0x7f19680b4900 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f195c000c00 tx=0x7f195c003820 comp rx=0 tx=0).stop 2023-10-19T13:00:44.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.208+0000 7f196e14c640 1 -- 172.21.15.57:0/3433500851 shutdown_connections 2023-10-19T13:00:44.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.208+0000 7f196e14c640 1 --2- 172.21.15.57:0/3433500851 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f195406ac90 0x7f195406d150 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:44.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.208+0000 7f196e14c640 1 --2- 172.21.15.57:0/3433500851 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f196815b2a0 0x7f196806a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:44.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.208+0000 7f196e14c640 1 --2- 172.21.15.57:0/3433500851 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f19680cf340 0x7f19680b4900 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:44.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.208+0000 7f196e14c640 1 --2- 172.21.15.57:0/3433500851 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f19680ceb40 0x7f1968151150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:44.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.208+0000 7f196e14c640 1 -- 172.21.15.57:0/3433500851 >> 172.21.15.57:0/3433500851 conn(0x7f19680d0d60 msgr2=0x7f1968159100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:44.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.208+0000 7f196e14c640 1 -- 172.21.15.57:0/3433500851 shutdown_connections 2023-10-19T13:00:44.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.208+0000 7f196e14c640 1 -- 172.21.15.57:0/3433500851 wait complete. 2023-10-19T13:00:44.221 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-19T13:00:44.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.328+0000 7fced214c640 1 -- 172.21.15.57:0/2055446742 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcecc0ab500 msgr2=0x7fcecc0ab990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:44.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.328+0000 7fced214c640 1 --2- 172.21.15.57:0/2055446742 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcecc0ab500 0x7fcecc0ab990 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7fcecc0713a0 tx=0x7fcec402c250 comp rx=0 tx=0).stop 2023-10-19T13:00:44.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.328+0000 7fced214c640 1 -- 172.21.15.57:0/2055446742 shutdown_connections 2023-10-19T13:00:44.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.328+0000 7fced214c640 1 --2- 172.21.15.57:0/2055446742 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcecc0ab500 0x7fcecc0ab990 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:44.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.328+0000 7fced214c640 1 --2- 172.21.15.57:0/2055446742 >> v2:172.21.15.179:3300/0 conn(0x7fcecc0c96b0 0x7fcecc0c9b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:44.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.328+0000 7fced214c640 1 --2- 172.21.15.57:0/2055446742 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcecc06af00 0x7fcecc06b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:44.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.328+0000 7fced214c640 1 -- 172.21.15.57:0/2055446742 >> 172.21.15.57:0/2055446742 conn(0x7fcecc069930 msgr2=0x7fcecc069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:44.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.328+0000 7fced214c640 1 -- 172.21.15.57:0/2055446742 shutdown_connections 2023-10-19T13:00:44.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.328+0000 7fced214c640 1 -- 172.21.15.57:0/2055446742 wait complete. 2023-10-19T13:00:44.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.329+0000 7fced214c640 1 Processor -- start 2023-10-19T13:00:44.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.329+0000 7fced214c640 1 -- start start 2023-10-19T13:00:44.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.330+0000 7fced214c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcecc06af00 0x7fcecc0aac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:44.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.330+0000 7fced214c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcecc0ab500 0x7fcecc15b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:44.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.330+0000 7fced214c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcecc0c96b0 0x7fcecc15b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:44.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.330+0000 7fced214c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcecc131cb0 con 0x7fcecc06af00 2023-10-19T13:00:44.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.330+0000 7fced214c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcecc0a83f0 con 0x7fcecc0c96b0 2023-10-19T13:00:44.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.330+0000 7fced214c640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcecc12e500 con 0x7fcecc0ab500 2023-10-19T13:00:44.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.330+0000 7fcecb7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcecc06af00 0x7fcecc0aac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:44.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.330+0000 7fcecb7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcecc06af00 0x7fcecc0aac50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:47776/0 (socket says 172.21.15.57:47776) 2023-10-19T13:00:44.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.330+0000 7fcecb7fe640 1 -- 172.21.15.57:0/2550326057 learned_addr learned my addr 172.21.15.57:0/2550326057 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:44.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.330+0000 7fcecaffd640 1 --2- 172.21.15.57:0/2550326057 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcecc0ab500 0x7fcecc15b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:44.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.330+0000 7fcecbfff640 1 --2- 172.21.15.57:0/2550326057 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcecc0c96b0 0x7fcecc15b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:44.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.330+0000 7fcecb7fe640 1 -- 172.21.15.57:0/2550326057 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcecc0ab500 msgr2=0x7fcecc15b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:44.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.330+0000 7fcecb7fe640 1 --2- 172.21.15.57:0/2550326057 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcecc0ab500 0x7fcecc15b160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:44.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.330+0000 7fcecb7fe640 1 -- 172.21.15.57:0/2550326057 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcecc0c96b0 msgr2=0x7fcecc15b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:44.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.330+0000 7fcecb7fe640 1 --2- 172.21.15.57:0/2550326057 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcecc0c96b0 0x7fcecc15b6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:44.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.330+0000 7fcecb7fe640 1 -- 172.21.15.57:0/2550326057 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcec4009900 con 0x7fcecc06af00 2023-10-19T13:00:44.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.330+0000 7fcecbfff640 1 --2- 172.21.15.57:0/2550326057 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcecc0c96b0 0x7fcecc15b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T13:00:44.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.331+0000 7fcecb7fe640 1 --2- 172.21.15.57:0/2550326057 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcecc06af00 0x7fcecc0aac50 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7fcebc00c920 tx=0x7fcebc00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:44.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.331+0000 7fceb3fff640 1 -- 172.21.15.57:0/2550326057 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcebc007c70 con 0x7fcecc06af00 2023-10-19T13:00:44.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.331+0000 7fceb3fff640 1 -- 172.21.15.57:0/2550326057 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fcebc00f450 con 0x7fcecc06af00 2023-10-19T13:00:44.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.331+0000 7fced214c640 1 -- 172.21.15.57:0/2550326057 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcecc12e7e0 con 0x7fcecc06af00 2023-10-19T13:00:44.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.331+0000 7fceb3fff640 1 -- 172.21.15.57:0/2550326057 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcebc00f860 con 0x7fcecc06af00 2023-10-19T13:00:44.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.331+0000 7fced214c640 1 -- 172.21.15.57:0/2550326057 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcecc0a8080 con 0x7fcecc06af00 2023-10-19T13:00:44.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.332+0000 7fced214c640 1 -- 172.21.15.57:0/2550326057 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcecc0690e0 con 0x7fcecc06af00 2023-10-19T13:00:44.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.332+0000 7fceb3fff640 1 -- 172.21.15.57:0/2550326057 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 38) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fcebc00f9c0 con 0x7fcecc06af00 2023-10-19T13:00:44.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.333+0000 7fceb3fff640 1 --2- 172.21.15.57:0/2550326057 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fceb806ac90 0x7fceb806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:44.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.333+0000 7fceb3fff640 1 -- 172.21.15.57:0/2550326057 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6745+0+0 (secure 0 0 0) 0x7fcebc089ed0 con 0x7fcecc06af00 2023-10-19T13:00:44.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.333+0000 7fcecaffd640 1 --2- 172.21.15.57:0/2550326057 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fceb806ac90 0x7fceb806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:44.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.333+0000 7fcecaffd640 1 --2- 172.21.15.57:0/2550326057 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fceb806ac90 0x7fceb806d150 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fcec0005fd0 tx=0x7fcec0005850 comp rx=0 tx=0).ready entity=mgr.6679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:44.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.337+0000 7fceb3fff640 1 -- 172.21.15.57:0/2550326057 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fcebc054eb0 con 0x7fcecc06af00 2023-10-19T13:00:44.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.547+0000 7fced214c640 1 -- 172.21.15.57:0/2550326057 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fcecc0cc420 con 0x7fcecc06af00 2023-10-19T13:00:44.550 INFO:tasks.ceph.mon.a.smithi057.stderr:2023-10-19T13:00:44.548+0000 7f1253ff3640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-19T13:00:44.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.549+0000 7fceb3fff640 1 -- 172.21.15.57:0/2550326057 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v31) v1 ==== 161+0+0 (secure 0 0 0) 0x7fcebc01d020 con 0x7fcecc06af00 2023-10-19T13:00:44.552 INFO:teuthology.orchestra.run.smithi057.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-19T13:00:44.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.552+0000 7fced214c640 1 -- 172.21.15.57:0/2550326057 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fceb806ac90 msgr2=0x7fceb806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:44.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.552+0000 7fced214c640 1 --2- 172.21.15.57:0/2550326057 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fceb806ac90 0x7fceb806d150 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fcec0005fd0 tx=0x7fcec0005850 comp rx=0 tx=0).stop 2023-10-19T13:00:44.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.552+0000 7fced214c640 1 -- 172.21.15.57:0/2550326057 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcecc06af00 msgr2=0x7fcecc0aac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:44.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.552+0000 7fced214c640 1 --2- 172.21.15.57:0/2550326057 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcecc06af00 0x7fcecc0aac50 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7fcebc00c920 tx=0x7fcebc00cde0 comp rx=0 tx=0).stop 2023-10-19T13:00:44.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.552+0000 7fced214c640 1 -- 172.21.15.57:0/2550326057 shutdown_connections 2023-10-19T13:00:44.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.552+0000 7fced214c640 1 --2- 172.21.15.57:0/2550326057 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fceb806ac90 0x7fceb806d150 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:44.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.552+0000 7fced214c640 1 --2- 172.21.15.57:0/2550326057 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcecc0c96b0 0x7fcecc15b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:44.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.552+0000 7fced214c640 1 --2- 172.21.15.57:0/2550326057 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcecc0ab500 0x7fcecc15b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:44.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.552+0000 7fced214c640 1 --2- 172.21.15.57:0/2550326057 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcecc06af00 0x7fcecc0aac50 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:44.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.552+0000 7fced214c640 1 -- 172.21.15.57:0/2550326057 >> 172.21.15.57:0/2550326057 conn(0x7fcecc069930 msgr2=0x7fcecc069d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:44.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.552+0000 7fced214c640 1 -- 172.21.15.57:0/2550326057 shutdown_connections 2023-10-19T13:00:44.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.552+0000 7fced214c640 1 -- 172.21.15.57:0/2550326057 wait complete. 2023-10-19T13:00:44.557 INFO:tasks.ceph.mds.a.smithi057.stderr: -2818> 2023-10-19T13:00:26.717+0000 7fbae915a640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-19T13:00:44.566 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-19T13:00:44.584 INFO:tasks.ceph.mds.a.smithi057.stdout:starting mds.a at 2023-10-19T13:00:44.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.675+0000 7fc46d41a640 1 -- 172.21.15.57:0/439416250 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4680b0530 msgr2=0x7fc4680b09a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:44.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.675+0000 7fc46d41a640 1 --2- 172.21.15.57:0/439416250 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4680b0530 0x7fc4680b09a0 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7fc4680cc7e0 tx=0x7fc46002c2f0 comp rx=0 tx=0).stop 2023-10-19T13:00:44.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.675+0000 7fc46d41a640 1 -- 172.21.15.57:0/439416250 shutdown_connections 2023-10-19T13:00:44.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.675+0000 7fc46d41a640 1 --2- 172.21.15.57:0/439416250 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4680b0530 0x7fc4680b09a0 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:44.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.675+0000 7fc46d41a640 1 --2- 172.21.15.57:0/439416250 >> v2:172.21.15.179:3300/0 conn(0x7fc4680cbd20 0x7fc468158300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:44.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.675+0000 7fc46d41a640 1 -- 172.21.15.57:0/439416250 >> 172.21.15.57:0/439416250 conn(0x7fc468069930 msgr2=0x7fc468069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:44.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.675+0000 7fc46d41a640 1 -- 172.21.15.57:0/439416250 shutdown_connections 2023-10-19T13:00:44.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.675+0000 7fc46d41a640 1 -- 172.21.15.57:0/439416250 wait complete. 2023-10-19T13:00:44.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.676+0000 7fc46d41a640 1 Processor -- start 2023-10-19T13:00:44.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.676+0000 7fc46d41a640 1 -- start start 2023-10-19T13:00:44.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.677+0000 7fc46d41a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4680afa20 0x7fc468147d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:44.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.677+0000 7fc46d41a640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc4680b0530 0x7fc46812d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:44.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.677+0000 7fc46d41a640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc4680cbd20 0x7fc468141db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:44.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.677+0000 7fc46d41a640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc468143da0 con 0x7fc4680afa20 2023-10-19T13:00:44.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.677+0000 7fc46d41a640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc46812de50 con 0x7fc4680cbd20 2023-10-19T13:00:44.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.677+0000 7fc46d41a640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc468148300 con 0x7fc4680b0530 2023-10-19T13:00:44.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.677+0000 7fc466ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4680afa20 0x7fc468147d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:44.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.677+0000 7fc466ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4680afa20 0x7fc468147d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:47812/0 (socket says 172.21.15.57:47812) 2023-10-19T13:00:44.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.677+0000 7fc466ffd640 1 -- 172.21.15.57:0/274392104 learned_addr learned my addr 172.21.15.57:0/274392104 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:44.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.677+0000 7fc45e7fc640 1 --2- 172.21.15.57:0/274392104 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc4680b0530 0x7fc46812d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:44.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.677+0000 7fc4677fe640 1 --2- 172.21.15.57:0/274392104 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc4680cbd20 0x7fc468141db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:44.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.677+0000 7fc466ffd640 1 -- 172.21.15.57:0/274392104 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc4680b0530 msgr2=0x7fc46812d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:44.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.677+0000 7fc466ffd640 1 --2- 172.21.15.57:0/274392104 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc4680b0530 0x7fc46812d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:44.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.677+0000 7fc466ffd640 1 -- 172.21.15.57:0/274392104 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc4680cbd20 msgr2=0x7fc468141db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:44.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.677+0000 7fc466ffd640 1 --2- 172.21.15.57:0/274392104 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc4680cbd20 0x7fc468141db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:44.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.677+0000 7fc466ffd640 1 -- 172.21.15.57:0/274392104 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc460009900 con 0x7fc4680afa20 2023-10-19T13:00:44.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.677+0000 7fc466ffd640 1 --2- 172.21.15.57:0/274392104 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4680afa20 0x7fc468147d20 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7fc45800d750 tx=0x7fc45800dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:44.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.678+0000 7fc45ffff640 1 -- 172.21.15.57:0/274392104 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc458004220 con 0x7fc4680afa20 2023-10-19T13:00:44.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.678+0000 7fc45ffff640 1 -- 172.21.15.57:0/274392104 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fc458004470 con 0x7fc4680afa20 2023-10-19T13:00:44.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.678+0000 7fc46d41a640 1 -- 172.21.15.57:0/274392104 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4681485e0 con 0x7fc4680afa20 2023-10-19T13:00:44.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.678+0000 7fc45ffff640 1 -- 172.21.15.57:0/274392104 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc458009c30 con 0x7fc4680afa20 2023-10-19T13:00:44.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.678+0000 7fc46d41a640 1 -- 172.21.15.57:0/274392104 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc468148ba0 con 0x7fc4680afa20 2023-10-19T13:00:44.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.678+0000 7fc46d41a640 1 -- 172.21.15.57:0/274392104 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4680c9da0 con 0x7fc4680afa20 2023-10-19T13:00:44.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.679+0000 7fc45ffff640 1 -- 172.21.15.57:0/274392104 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 38) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fc458005020 con 0x7fc4680afa20 2023-10-19T13:00:44.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.679+0000 7fc45ffff640 1 --2- 172.21.15.57:0/274392104 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fc44806ac90 0x7fc44806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:44.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.679+0000 7fc45ffff640 1 -- 172.21.15.57:0/274392104 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6831+0+0 (secure 0 0 0) 0x7fc458089b00 con 0x7fc4680afa20 2023-10-19T13:00:44.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.680+0000 7fc45e7fc640 1 --2- 172.21.15.57:0/274392104 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fc44806ac90 0x7fc44806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:44.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.680+0000 7fc45e7fc640 1 --2- 172.21.15.57:0/274392104 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fc44806ac90 0x7fc44806d150 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc46806caf0 tx=0x7fc450009040 comp rx=0 tx=0).ready entity=mgr.6679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:44.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.684+0000 7fc45ffff640 1 -- 172.21.15.57:0/274392104 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc458054a90 con 0x7fc4680afa20 2023-10-19T13:00:44.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:44.896+0000 7fc46d41a640 1 -- 172.21.15.57:0/274392104 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fc46806f700 con 0x7fc4680afa20 2023-10-19T13:00:45.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:45.077+0000 7fc45ffff640 1 -- 172.21.15.57:0/274392104 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mgrmap(e 39) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fc458054e60 con 0x7fc4680afa20 2023-10-19T13:00:45.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:45.555+0000 7fc45ffff640 1 -- 172.21.15.57:0/274392104 <== mon.0 v2:172.21.15.57:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v32) v1 ==== 104+0+0 (secure 0 0 0) 0x7fc4580879a0 con 0x7fc4680afa20 2023-10-19T13:00:45.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:45.558+0000 7fc46d41a640 1 -- 172.21.15.57:0/274392104 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fc44806ac90 msgr2=0x7fc44806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:45.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:45.558+0000 7fc46d41a640 1 --2- 172.21.15.57:0/274392104 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fc44806ac90 0x7fc44806d150 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc46806caf0 tx=0x7fc450009040 comp rx=0 tx=0).stop 2023-10-19T13:00:45.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:45.558+0000 7fc46d41a640 1 -- 172.21.15.57:0/274392104 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4680afa20 msgr2=0x7fc468147d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:45.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:45.558+0000 7fc46d41a640 1 --2- 172.21.15.57:0/274392104 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4680afa20 0x7fc468147d20 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7fc45800d750 tx=0x7fc45800dc10 comp rx=0 tx=0).stop 2023-10-19T13:00:45.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:45.558+0000 7fc46d41a640 1 -- 172.21.15.57:0/274392104 shutdown_connections 2023-10-19T13:00:45.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:45.558+0000 7fc46d41a640 1 --2- 172.21.15.57:0/274392104 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fc44806ac90 0x7fc44806d150 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:45.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:45.558+0000 7fc46d41a640 1 --2- 172.21.15.57:0/274392104 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc4680cbd20 0x7fc468141db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:45.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:45.558+0000 7fc46d41a640 1 --2- 172.21.15.57:0/274392104 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc4680b0530 0x7fc46812d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:45.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:45.558+0000 7fc46d41a640 1 --2- 172.21.15.57:0/274392104 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4680afa20 0x7fc468147d20 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:45.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:45.558+0000 7fc46d41a640 1 -- 172.21.15.57:0/274392104 >> 172.21.15.57:0/274392104 conn(0x7fc468069930 msgr2=0x7fc46806bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:45.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:45.558+0000 7fc46d41a640 1 -- 172.21.15.57:0/274392104 shutdown_connections 2023-10-19T13:00:45.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:45.558+0000 7fc46d41a640 1 -- 172.21.15.57:0/274392104 wait complete. 2023-10-19T13:00:45.573 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-19T13:00:45.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:45.729+0000 7f36219b6640 1 -- 172.21.15.57:0/1425325222 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3604003970 msgr2=0x7f3604003d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:45.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:45.729+0000 7f36219b6640 1 --2- 172.21.15.57:0/1425325222 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3604003970 0x7f3604003d50 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f360c00c290 tx=0x7f36040057f0 comp rx=0 tx=0).stop 2023-10-19T13:00:45.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:45.729+0000 7f36219b6640 1 -- 172.21.15.57:0/1425325222 shutdown_connections 2023-10-19T13:00:45.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:45.729+0000 7f36219b6640 1 --2- 172.21.15.57:0/1425325222 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3604003970 0x7f3604003d50 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:45.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:45.729+0000 7f36219b6640 1 -- 172.21.15.57:0/1425325222 >> 172.21.15.57:0/1425325222 conn(0x7f361c069930 msgr2=0x7f361c069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:45.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:45.729+0000 7f36219b6640 1 -- 172.21.15.57:0/1425325222 shutdown_connections 2023-10-19T13:00:45.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:45.729+0000 7f36219b6640 1 -- 172.21.15.57:0/1425325222 wait complete. 2023-10-19T13:00:45.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:45.730+0000 7f36219b6640 1 Processor -- start 2023-10-19T13:00:45.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:45.730+0000 7f36219b6640 1 -- start start 2023-10-19T13:00:45.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:45.731+0000 7f36219b6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3604003970 0x7f361c1582c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:45.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:45.731+0000 7f36219b6640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f361c13b010 0x7f361c1e37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:45.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:45.731+0000 7f36219b6640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f361c1e3d20 0x7f361c0af750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:45.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:45.731+0000 7f36219b6640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f361c1e41a0 con 0x7f3604003970 2023-10-19T13:00:45.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:45.731+0000 7f36219b6640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f361c0aa980 con 0x7f361c13b010 2023-10-19T13:00:45.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:45.731+0000 7f36219b6640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f361c0aac60 con 0x7f361c1e3d20 2023-10-19T13:00:45.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:45.731+0000 7f361affd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3604003970 0x7f361c1582c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:45.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:45.731+0000 7f361affd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3604003970 0x7f361c1582c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:47826/0 (socket says 172.21.15.57:47826) 2023-10-19T13:00:45.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:45.731+0000 7f361affd640 1 -- 172.21.15.57:0/623540603 learned_addr learned my addr 172.21.15.57:0/623540603 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:45.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:45.731+0000 7f361b7fe640 1 --2- 172.21.15.57:0/623540603 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f361c1e3d20 0x7f361c0af750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:45.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:45.731+0000 7f361affd640 1 -- 172.21.15.57:0/623540603 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f361c1e3d20 msgr2=0x7f361c0af750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:45.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:45.731+0000 7f361affd640 1 --2- 172.21.15.57:0/623540603 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f361c1e3d20 0x7f361c0af750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:45.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:45.731+0000 7f361affd640 1 -- 172.21.15.57:0/623540603 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f361c13b010 msgr2=0x7f361c1e37e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:45.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:45.731+0000 7f361affd640 1 --2- 172.21.15.57:0/623540603 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f361c13b010 0x7f361c1e37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:45.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:45.731+0000 7f361affd640 1 -- 172.21.15.57:0/623540603 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f361c0aaf40 con 0x7f3604003970 2023-10-19T13:00:45.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:45.732+0000 7f361b7fe640 1 --2- 172.21.15.57:0/623540603 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f361c1e3d20 0x7f361c0af750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T13:00:45.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:45.732+0000 7f361affd640 1 --2- 172.21.15.57:0/623540603 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3604003970 0x7f361c1582c0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f360c00c3b0 tx=0x7f360c001010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:45.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:45.732+0000 7f3603fff640 1 -- 172.21.15.57:0/623540603 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f360c001670 con 0x7f3604003970 2023-10-19T13:00:45.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:45.732+0000 7f3603fff640 1 -- 172.21.15.57:0/623540603 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f360c006270 con 0x7f3604003970 2023-10-19T13:00:45.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:45.732+0000 7f36219b6640 1 -- 172.21.15.57:0/623540603 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f361c0ab220 con 0x7f3604003970 2023-10-19T13:00:45.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:45.732+0000 7f3603fff640 1 -- 172.21.15.57:0/623540603 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f360c0035f0 con 0x7f3604003970 2023-10-19T13:00:45.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:45.732+0000 7f36219b6640 1 -- 172.21.15.57:0/623540603 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f361c0b17d0 con 0x7f3604003970 2023-10-19T13:00:45.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:45.733+0000 7f36219b6640 1 -- 172.21.15.57:0/623540603 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f361c0690e0 con 0x7f3604003970 2023-10-19T13:00:45.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:45.733+0000 7f3603fff640 1 -- 172.21.15.57:0/623540603 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 39) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f360c031030 con 0x7f3604003970 2023-10-19T13:00:45.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:45.734+0000 7f3603fff640 1 --2- 172.21.15.57:0/623540603 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f360806af00 0x7f360806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:45.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:45.734+0000 7f3603fff640 1 -- 172.21.15.57:0/623540603 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6831+0+0 (secure 0 0 0) 0x7f360c0ab240 con 0x7f3604003970 2023-10-19T13:00:45.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:45.734+0000 7f361a7fc640 1 --2- 172.21.15.57:0/623540603 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f360806af00 0x7f360806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:45.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:45.734+0000 7f361a7fc640 1 --2- 172.21.15.57:0/623540603 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f360806af00 0x7f360806d3c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f361c1f2610 tx=0x7f3610005b30 comp rx=0 tx=0).ready entity=mgr.6679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:45.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:45.738+0000 7f3603fff640 1 -- 172.21.15.57:0/623540603 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f360c075d90 con 0x7f3604003970 2023-10-19T13:00:45.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:45.915+0000 7f36219b6640 1 -- 172.21.15.57:0/623540603 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f361c0cc420 con 0x7f3604003970 2023-10-19T13:00:46.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:46.705+0000 7f3603fff640 1 -- 172.21.15.57:0/623540603 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v91) v1 ==== 181+0+0 (secure 0 0 0) 0x7f360c0793b0 con 0x7f3604003970 2023-10-19T13:00:46.708 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'cephfs_metadata' removed 2023-10-19T13:00:46.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:46.709+0000 7f36219b6640 1 -- 172.21.15.57:0/623540603 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f360806af00 msgr2=0x7f360806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:46.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:46.709+0000 7f36219b6640 1 --2- 172.21.15.57:0/623540603 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f360806af00 0x7f360806d3c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f361c1f2610 tx=0x7f3610005b30 comp rx=0 tx=0).stop 2023-10-19T13:00:46.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:46.709+0000 7f36219b6640 1 -- 172.21.15.57:0/623540603 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3604003970 msgr2=0x7f361c1582c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:46.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:46.709+0000 7f36219b6640 1 --2- 172.21.15.57:0/623540603 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3604003970 0x7f361c1582c0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f360c00c3b0 tx=0x7f360c001010 comp rx=0 tx=0).stop 2023-10-19T13:00:46.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:46.712+0000 7f36219b6640 1 -- 172.21.15.57:0/623540603 shutdown_connections 2023-10-19T13:00:46.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:46.712+0000 7f36219b6640 1 --2- 172.21.15.57:0/623540603 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f360806af00 0x7f360806d3c0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:46.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:46.712+0000 7f36219b6640 1 --2- 172.21.15.57:0/623540603 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f361c1e3d20 0x7f361c0af750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:46.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:46.712+0000 7f36219b6640 1 --2- 172.21.15.57:0/623540603 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f361c13b010 0x7f361c1e37e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:46.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:46.712+0000 7f36219b6640 1 --2- 172.21.15.57:0/623540603 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3604003970 0x7f361c1582c0 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:46.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:46.713+0000 7f36219b6640 1 -- 172.21.15.57:0/623540603 >> 172.21.15.57:0/623540603 conn(0x7f361c069930 msgr2=0x7f361c13e3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:46.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:46.713+0000 7f36219b6640 1 -- 172.21.15.57:0/623540603 shutdown_connections 2023-10-19T13:00:46.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:46.713+0000 7f36219b6640 1 -- 172.21.15.57:0/623540603 wait complete. 2023-10-19T13:00:46.730 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-19T13:00:46.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:46.847+0000 7ff014e44640 1 -- 172.21.15.57:0/2592654809 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff010159060 msgr2=0x7ff01006d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:46.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:46.847+0000 7ff014e44640 1 --2- 172.21.15.57:0/2592654809 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff010159060 0x7ff01006d260 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7ff000009900 tx=0x7ff00002bd90 comp rx=0 tx=0).stop 2023-10-19T13:00:46.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:46.848+0000 7ff014e44640 1 -- 172.21.15.57:0/2592654809 shutdown_connections 2023-10-19T13:00:46.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:46.848+0000 7ff014e44640 1 --2- 172.21.15.57:0/2592654809 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff010159060 0x7ff01006d260 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:46.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:46.848+0000 7ff014e44640 1 -- 172.21.15.57:0/2592654809 >> 172.21.15.57:0/2592654809 conn(0x7ff0100b0670 msgr2=0x7ff0100b0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:46.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:46.848+0000 7ff014e44640 1 -- 172.21.15.57:0/2592654809 shutdown_connections 2023-10-19T13:00:46.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:46.848+0000 7ff014e44640 1 -- 172.21.15.57:0/2592654809 wait complete. 2023-10-19T13:00:46.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:46.849+0000 7ff014e44640 1 Processor -- start 2023-10-19T13:00:46.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:46.849+0000 7ff014e44640 1 -- start start 2023-10-19T13:00:46.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:46.850+0000 7ff014e44640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff0100cdb30 0x7ff01015bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:46.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:46.850+0000 7ff014e44640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff010159060 0x7ff010150bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:46.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:46.850+0000 7ff014e44640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff01015d400 0x7ff0101500d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:46.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:46.850+0000 7ff014e44640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff01015c3b0 con 0x7ff01015d400 2023-10-19T13:00:46.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:46.850+0000 7ff014e44640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff01015c690 con 0x7ff0100cdb30 2023-10-19T13:00:46.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:46.850+0000 7ff014e44640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0100c34c0 con 0x7ff010159060 2023-10-19T13:00:46.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:46.850+0000 7ff00ed9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff01015d400 0x7ff0101500d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:46.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:46.850+0000 7ff00ed9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff01015d400 0x7ff0101500d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51306/0 (socket says 172.21.15.57:51306) 2023-10-19T13:00:46.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:46.851+0000 7ff00ed9d640 1 -- 172.21.15.57:0/2732728875 learned_addr learned my addr 172.21.15.57:0/2732728875 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:46.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:46.851+0000 7ff00e59c640 1 --2- 172.21.15.57:0/2732728875 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff0100cdb30 0x7ff01015bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:46.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:46.851+0000 7ff005d9b640 1 --2- 172.21.15.57:0/2732728875 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff010159060 0x7ff010150bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:46.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:46.851+0000 7ff00ed9d640 1 -- 172.21.15.57:0/2732728875 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff010159060 msgr2=0x7ff010150bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:46.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:46.851+0000 7ff00ed9d640 1 --2- 172.21.15.57:0/2732728875 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff010159060 0x7ff010150bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:46.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:46.851+0000 7ff00ed9d640 1 -- 172.21.15.57:0/2732728875 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff0100cdb30 msgr2=0x7ff01015bd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:46.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:46.851+0000 7ff00ed9d640 1 --2- 172.21.15.57:0/2732728875 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff0100cdb30 0x7ff01015bd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:46.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:46.851+0000 7ff00ed9d640 1 -- 172.21.15.57:0/2732728875 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff000009560 con 0x7ff01015d400 2023-10-19T13:00:46.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:46.851+0000 7ff00e59c640 1 --2- 172.21.15.57:0/2732728875 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff0100cdb30 0x7ff01015bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:00:46.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:46.851+0000 7ff00ed9d640 1 --2- 172.21.15.57:0/2732728875 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff01015d400 0x7ff0101500d0 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7ff00800c6b0 tx=0x7ff00800cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:46.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:46.851+0000 7ff0077fe640 1 -- 172.21.15.57:0/2732728875 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff008003b70 con 0x7ff01015d400 2023-10-19T13:00:46.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:46.851+0000 7ff014e44640 1 -- 172.21.15.57:0/2732728875 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0100c37a0 con 0x7ff01015d400 2023-10-19T13:00:46.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:46.851+0000 7ff0077fe640 1 -- 172.21.15.57:0/2732728875 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7ff008003cf0 con 0x7ff01015d400 2023-10-19T13:00:46.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:46.851+0000 7ff0077fe640 1 -- 172.21.15.57:0/2732728875 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff008005330 con 0x7ff01015d400 2023-10-19T13:00:46.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:46.851+0000 7ff014e44640 1 -- 172.21.15.57:0/2732728875 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0101510f0 con 0x7ff01015d400 2023-10-19T13:00:46.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:46.852+0000 7ff014e44640 1 -- 172.21.15.57:0/2732728875 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0100675f0 con 0x7ff01015d400 2023-10-19T13:00:46.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:46.853+0000 7ff0077fe640 1 -- 172.21.15.57:0/2732728875 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 40) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff008005490 con 0x7ff01015d400 2023-10-19T13:00:46.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:46.853+0000 7ff0077fe640 1 --2- 172.21.15.57:0/2732728875 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7feff006af60 0x7feff006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:46.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:46.853+0000 7ff0077fe640 1 -- 172.21.15.57:0/2732728875 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6393+0+0 (secure 0 0 0) 0x7ff008089950 con 0x7ff01015d400 2023-10-19T13:00:46.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:46.853+0000 7ff00e59c640 1 --2- 172.21.15.57:0/2732728875 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7feff006af60 0x7feff006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:46.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:46.854+0000 7ff00e59c640 1 --2- 172.21.15.57:0/2732728875 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7feff006af60 0x7feff006d420 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff000035000 tx=0x7ff000004730 comp rx=0 tx=0).ready entity=mgr.6679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:46.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:46.857+0000 7ff0077fe640 1 -- 172.21.15.57:0/2732728875 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff008054650 con 0x7ff01015d400 2023-10-19T13:00:47.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:47.033+0000 7ff014e44640 1 -- 172.21.15.57:0/2732728875 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7ff0100b4aa0 con 0x7ff01015d400 2023-10-19T13:00:47.558 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:00:47.561 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:00:47.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:47.719+0000 7ff0077fe640 1 -- 172.21.15.57:0/2732728875 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v92) v1 ==== 169+0+0 (secure 0 0 0) 0x7ff008057c70 con 0x7ff01015d400 2023-10-19T13:00:47.722 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'cephfs_data' removed 2023-10-19T13:00:47.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:47.721+0000 7ff014e44640 1 -- 172.21.15.57:0/2732728875 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7feff006af60 msgr2=0x7feff006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:47.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:47.721+0000 7ff014e44640 1 --2- 172.21.15.57:0/2732728875 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7feff006af60 0x7feff006d420 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff000035000 tx=0x7ff000004730 comp rx=0 tx=0).stop 2023-10-19T13:00:47.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:47.721+0000 7ff014e44640 1 -- 172.21.15.57:0/2732728875 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff01015d400 msgr2=0x7ff0101500d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:47.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:47.721+0000 7ff014e44640 1 --2- 172.21.15.57:0/2732728875 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff01015d400 0x7ff0101500d0 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7ff00800c6b0 tx=0x7ff00800cb70 comp rx=0 tx=0).stop 2023-10-19T13:00:47.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:47.721+0000 7ff014e44640 1 -- 172.21.15.57:0/2732728875 shutdown_connections 2023-10-19T13:00:47.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:47.721+0000 7ff014e44640 1 --2- 172.21.15.57:0/2732728875 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7feff006af60 0x7feff006d420 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:47.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:47.721+0000 7ff014e44640 1 --2- 172.21.15.57:0/2732728875 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff01015d400 0x7ff0101500d0 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:47.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:47.721+0000 7ff014e44640 1 --2- 172.21.15.57:0/2732728875 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff010159060 0x7ff010150bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:47.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:47.721+0000 7ff014e44640 1 --2- 172.21.15.57:0/2732728875 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff0100cdb30 0x7ff01015bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:47.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:47.721+0000 7ff014e44640 1 -- 172.21.15.57:0/2732728875 >> 172.21.15.57:0/2732728875 conn(0x7ff0100b0670 msgr2=0x7ff0100cca20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:47.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:47.721+0000 7ff014e44640 1 -- 172.21.15.57:0/2732728875 shutdown_connections 2023-10-19T13:00:47.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:47.721+0000 7ff014e44640 1 -- 172.21.15.57:0/2732728875 wait complete. 2023-10-19T13:00:47.739 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 30 2023-10-19T13:00:47.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:47.847+0000 7f7e23094640 1 -- 172.21.15.57:0/3576597450 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e1c0afa20 msgr2=0x7f7e1c158300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:47.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:47.847+0000 7f7e23094640 1 --2- 172.21.15.57:0/3576597450 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e1c0afa20 0x7f7e1c158300 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f7e10009830 tx=0x7f7e1002bd60 comp rx=0 tx=0).stop 2023-10-19T13:00:47.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:47.847+0000 7f7e23094640 1 -- 172.21.15.57:0/3576597450 shutdown_connections 2023-10-19T13:00:47.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:47.847+0000 7f7e23094640 1 --2- 172.21.15.57:0/3576597450 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7e1c0b0530 0x7f7e1c0b09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:47.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:47.847+0000 7f7e23094640 1 --2- 172.21.15.57:0/3576597450 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e1c0afa20 0x7f7e1c158300 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:47.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:47.847+0000 7f7e23094640 1 -- 172.21.15.57:0/3576597450 >> 172.21.15.57:0/3576597450 conn(0x7f7e1c069930 msgr2=0x7f7e1c069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:47.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:47.847+0000 7f7e23094640 1 -- 172.21.15.57:0/3576597450 shutdown_connections 2023-10-19T13:00:47.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:47.847+0000 7f7e23094640 1 -- 172.21.15.57:0/3576597450 wait complete. 2023-10-19T13:00:47.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:47.848+0000 7f7e23094640 1 Processor -- start 2023-10-19T13:00:47.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:47.848+0000 7f7e23094640 1 -- start start 2023-10-19T13:00:47.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:47.849+0000 7f7e23094640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7e1c0afa20 0x7f7e1c1e1f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:47.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:47.849+0000 7f7e23094640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7e1c0b0530 0x7f7e1c13d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:47.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:47.849+0000 7f7e23094640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e1c0cbd20 0x7f7e1c14bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:47.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:47.849+0000 7f7e23094640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e1c13d830 con 0x7f7e1c0cbd20 2023-10-19T13:00:47.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:47.849+0000 7f7e23094640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e1c1e0be0 con 0x7f7e1c0b0530 2023-10-19T13:00:47.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:47.849+0000 7f7e23094640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e1c1e0060 con 0x7f7e1c0afa20 2023-10-19T13:00:47.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:47.849+0000 7f7e21631640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e1c0cbd20 0x7f7e1c14bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:47.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:47.849+0000 7f7e21631640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e1c0cbd20 0x7f7e1c14bd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51332/0 (socket says 172.21.15.57:51332) 2023-10-19T13:00:47.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:47.849+0000 7f7e21631640 1 -- 172.21.15.57:0/2960740402 learned_addr learned my addr 172.21.15.57:0/2960740402 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:47.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:47.849+0000 7f7e20e30640 1 --2- 172.21.15.57:0/2960740402 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7e1c0afa20 0x7f7e1c1e1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:47.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:47.849+0000 7f7e1bfff640 1 --2- 172.21.15.57:0/2960740402 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7e1c0b0530 0x7f7e1c13d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:47.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:47.850+0000 7f7e21631640 1 -- 172.21.15.57:0/2960740402 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7e1c0afa20 msgr2=0x7f7e1c1e1f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:47.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:47.850+0000 7f7e21631640 1 --2- 172.21.15.57:0/2960740402 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7e1c0afa20 0x7f7e1c1e1f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:47.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:47.850+0000 7f7e21631640 1 -- 172.21.15.57:0/2960740402 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7e1c0b0530 msgr2=0x7f7e1c13d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:47.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:47.850+0000 7f7e21631640 1 --2- 172.21.15.57:0/2960740402 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7e1c0b0530 0x7f7e1c13d2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:47.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:47.850+0000 7f7e21631640 1 -- 172.21.15.57:0/2960740402 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e10009490 con 0x7f7e1c0cbd20 2023-10-19T13:00:47.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:47.850+0000 7f7e1bfff640 1 --2- 172.21.15.57:0/2960740402 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7e1c0b0530 0x7f7e1c13d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:00:47.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:47.850+0000 7f7e21631640 1 --2- 172.21.15.57:0/2960740402 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e1c0cbd20 0x7f7e1c14bd30 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f7e1400dd70 tx=0x7f7e1400b730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:47.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:47.850+0000 7f7e197fa640 1 -- 172.21.15.57:0/2960740402 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e14004d20 con 0x7f7e1c0cbd20 2023-10-19T13:00:47.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:47.850+0000 7f7e197fa640 1 -- 172.21.15.57:0/2960740402 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f7e14004e80 con 0x7f7e1c0cbd20 2023-10-19T13:00:47.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:47.850+0000 7f7e23094640 1 -- 172.21.15.57:0/2960740402 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e1c1e0340 con 0x7f7e1c0cbd20 2023-10-19T13:00:47.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:47.850+0000 7f7e197fa640 1 -- 172.21.15.57:0/2960740402 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e1400e8a0 con 0x7f7e1c0cbd20 2023-10-19T13:00:47.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:47.850+0000 7f7e23094640 1 -- 172.21.15.57:0/2960740402 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e1c1e0900 con 0x7f7e1c0cbd20 2023-10-19T13:00:47.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:47.851+0000 7f7e23094640 1 -- 172.21.15.57:0/2960740402 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e1c0690e0 con 0x7f7e1c0cbd20 2023-10-19T13:00:47.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:47.851+0000 7f7e197fa640 1 -- 172.21.15.57:0/2960740402 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 40) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f7e1400ea00 con 0x7f7e1c0cbd20 2023-10-19T13:00:47.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:47.852+0000 7f7e197fa640 1 --2- 172.21.15.57:0/2960740402 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f7e0806af60 0x7f7e0806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:47.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:47.852+0000 7f7e197fa640 1 -- 172.21.15.57:0/2960740402 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5979+0+0 (secure 0 0 0) 0x7f7e1408a580 con 0x7f7e1c0cbd20 2023-10-19T13:00:47.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:47.852+0000 7f7e20e30640 1 --2- 172.21.15.57:0/2960740402 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f7e0806af60 0x7f7e0806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:47.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:47.853+0000 7f7e20e30640 1 --2- 172.21.15.57:0/2960740402 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f7e0806af60 0x7f7e0806d420 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f7e0c0046b0 tx=0x7f7e0c007eb0 comp rx=0 tx=0).ready entity=mgr.6679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:47.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:47.856+0000 7f7e197fa640 1 -- 172.21.15.57:0/2960740402 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7e14055410 con 0x7f7e1c0cbd20 2023-10-19T13:00:48.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.036+0000 7f7e23094640 1 -- 172.21.15.57:0/2960740402 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "config reset", "num": 30} v 0) v1 -- 0x7f7e1c0700d0 con 0x7f7e1c0cbd20 2023-10-19T13:00:48.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.039+0000 7f7e197fa640 1 -- 172.21.15.57:0/2960740402 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 30}]=0 v42) v1 ==== 71+0+0 (secure 0 0 0) 0x7f7e1400ecb0 con 0x7f7e1c0cbd20 2023-10-19T13:00:48.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.041+0000 7f7e23094640 1 -- 172.21.15.57:0/2960740402 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f7e0806af60 msgr2=0x7f7e0806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:48.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.041+0000 7f7e23094640 1 --2- 172.21.15.57:0/2960740402 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f7e0806af60 0x7f7e0806d420 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f7e0c0046b0 tx=0x7f7e0c007eb0 comp rx=0 tx=0).stop 2023-10-19T13:00:48.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.041+0000 7f7e23094640 1 -- 172.21.15.57:0/2960740402 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e1c0cbd20 msgr2=0x7f7e1c14bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:48.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.041+0000 7f7e23094640 1 --2- 172.21.15.57:0/2960740402 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e1c0cbd20 0x7f7e1c14bd30 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f7e1400dd70 tx=0x7f7e1400b730 comp rx=0 tx=0).stop 2023-10-19T13:00:48.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.041+0000 7f7e23094640 1 -- 172.21.15.57:0/2960740402 shutdown_connections 2023-10-19T13:00:48.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.041+0000 7f7e23094640 1 --2- 172.21.15.57:0/2960740402 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f7e0806af60 0x7f7e0806d420 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:48.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.041+0000 7f7e23094640 1 --2- 172.21.15.57:0/2960740402 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e1c0cbd20 0x7f7e1c14bd30 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:48.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.041+0000 7f7e23094640 1 --2- 172.21.15.57:0/2960740402 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7e1c0b0530 0x7f7e1c13d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:48.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.041+0000 7f7e23094640 1 --2- 172.21.15.57:0/2960740402 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7e1c0afa20 0x7f7e1c1e1f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:48.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.041+0000 7f7e23094640 1 -- 172.21.15.57:0/2960740402 >> 172.21.15.57:0/2960740402 conn(0x7f7e1c069930 msgr2=0x7f7e1c150550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:48.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.041+0000 7f7e23094640 1 -- 172.21.15.57:0/2960740402 shutdown_connections 2023-10-19T13:00:48.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.041+0000 7f7e23094640 1 -- 172.21.15.57:0/2960740402 wait complete. 2023-10-19T13:00:48.055 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule' 2023-10-19T13:00:48.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.161+0000 7fa0f511d640 1 -- 172.21.15.57:0/1323471490 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0f00afa20 msgr2=0x7fa0f00705a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:48.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.161+0000 7fa0f511d640 1 --2- 172.21.15.57:0/1323471490 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0f00afa20 0x7fa0f00705a0 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7fa0f00cc7e0 tx=0x7fa0d802bdb0 comp rx=0 tx=0).stop 2023-10-19T13:00:48.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.162+0000 7fa0f511d640 1 -- 172.21.15.57:0/1323471490 shutdown_connections 2023-10-19T13:00:48.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.162+0000 7fa0f511d640 1 --2- 172.21.15.57:0/1323471490 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0f00afa20 0x7fa0f00705a0 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:48.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.162+0000 7fa0f511d640 1 --2- 172.21.15.57:0/1323471490 >> v2:172.21.15.179:3300/0 conn(0x7fa0f00cbd20 0x7fa0f0158300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:48.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.162+0000 7fa0f511d640 1 -- 172.21.15.57:0/1323471490 >> 172.21.15.57:0/1323471490 conn(0x7fa0f0069930 msgr2=0x7fa0f0069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:48.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.162+0000 7fa0f511d640 1 -- 172.21.15.57:0/1323471490 shutdown_connections 2023-10-19T13:00:48.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.162+0000 7fa0f511d640 1 -- 172.21.15.57:0/1323471490 wait complete. 2023-10-19T13:00:48.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.163+0000 7fa0f511d640 1 Processor -- start 2023-10-19T13:00:48.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.163+0000 7fa0f511d640 1 -- start start 2023-10-19T13:00:48.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.164+0000 7fa0f511d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0f00afa20 0x7fa0f0155dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:48.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.164+0000 7fa0f511d640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa0f00b0530 0x7fa0f01e36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:48.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.164+0000 7fa0f511d640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa0f00cbd20 0x7fa0f01538b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:48.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.164+0000 7fa0f511d640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f01e2f40 con 0x7fa0f00afa20 2023-10-19T13:00:48.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.164+0000 7fa0f511d640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f0156790 con 0x7fa0f00b0530 2023-10-19T13:00:48.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.164+0000 7fa0f511d640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f0156a70 con 0x7fa0f00cbd20 2023-10-19T13:00:48.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.164+0000 7fa0eed9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0f00afa20 0x7fa0f0155dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:48.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.164+0000 7fa0eed9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0f00afa20 0x7fa0f0155dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51350/0 (socket says 172.21.15.57:51350) 2023-10-19T13:00:48.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.164+0000 7fa0eed9d640 1 -- 172.21.15.57:0/2375503573 learned_addr learned my addr 172.21.15.57:0/2375503573 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:48.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.164+0000 7fa0e7fff640 1 --2- 172.21.15.57:0/2375503573 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa0f00b0530 0x7fa0f01e36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:48.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.164+0000 7fa0ef59e640 1 --2- 172.21.15.57:0/2375503573 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa0f00cbd20 0x7fa0f01538b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:48.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.164+0000 7fa0eed9d640 1 -- 172.21.15.57:0/2375503573 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa0f00cbd20 msgr2=0x7fa0f01538b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:48.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.164+0000 7fa0eed9d640 1 --2- 172.21.15.57:0/2375503573 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa0f00cbd20 0x7fa0f01538b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:48.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.164+0000 7fa0eed9d640 1 -- 172.21.15.57:0/2375503573 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa0f00b0530 msgr2=0x7fa0f01e36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:48.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.164+0000 7fa0eed9d640 1 --2- 172.21.15.57:0/2375503573 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa0f00b0530 0x7fa0f01e36d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:48.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.164+0000 7fa0eed9d640 1 -- 172.21.15.57:0/2375503573 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0d8009560 con 0x7fa0f00afa20 2023-10-19T13:00:48.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.164+0000 7fa0e7fff640 1 --2- 172.21.15.57:0/2375503573 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa0f00b0530 0x7fa0f01e36d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T13:00:48.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.164+0000 7fa0ef59e640 1 --2- 172.21.15.57:0/2375503573 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa0f00cbd20 0x7fa0f01538b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T13:00:48.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.164+0000 7fa0eed9d640 1 --2- 172.21.15.57:0/2375503573 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0f00afa20 0x7fa0f0155dd0 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7fa0e000c970 tx=0x7fa0e000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:48.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.164+0000 7fa0e77fe640 1 -- 172.21.15.57:0/2375503573 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0e0007bb0 con 0x7fa0f00afa20 2023-10-19T13:00:48.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.164+0000 7fa0e77fe640 1 -- 172.21.15.57:0/2375503573 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fa0e0007e20 con 0x7fa0f00afa20 2023-10-19T13:00:48.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.164+0000 7fa0f511d640 1 -- 172.21.15.57:0/2375503573 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0f01e31b0 con 0x7fa0f00afa20 2023-10-19T13:00:48.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.164+0000 7fa0e77fe640 1 -- 172.21.15.57:0/2375503573 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0e0005280 con 0x7fa0f00afa20 2023-10-19T13:00:48.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.164+0000 7fa0f511d640 1 -- 172.21.15.57:0/2375503573 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0f0156310 con 0x7fa0f00afa20 2023-10-19T13:00:48.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.165+0000 7fa0f511d640 1 -- 172.21.15.57:0/2375503573 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0f0156560 con 0x7fa0f00afa20 2023-10-19T13:00:48.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.166+0000 7fa0e77fe640 1 -- 172.21.15.57:0/2375503573 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 40) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fa0e00053e0 con 0x7fa0f00afa20 2023-10-19T13:00:48.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.166+0000 7fa0e77fe640 1 --2- 172.21.15.57:0/2375503573 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fa0d006af60 0x7fa0d006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:48.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.166+0000 7fa0e77fe640 1 -- 172.21.15.57:0/2375503573 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5979+0+0 (secure 0 0 0) 0x7fa0e0089e80 con 0x7fa0f00afa20 2023-10-19T13:00:48.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.167+0000 7fa0e7fff640 1 --2- 172.21.15.57:0/2375503573 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fa0d006af60 0x7fa0d006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:48.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.167+0000 7fa0e7fff640 1 --2- 172.21.15.57:0/2375503573 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fa0d006af60 0x7fa0d006d420 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fa0d8035000 tx=0x7fa0d8004730 comp rx=0 tx=0).ready entity=mgr.6679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:48.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.170+0000 7fa0e77fe640 1 -- 172.21.15.57:0/2375503573 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa0e0054c10 con 0x7fa0f00afa20 2023-10-19T13:00:48.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.411+0000 7fa0f511d640 1 -- 172.21.15.57:0/2375503573 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule"]} v 0) v1 -- 0x7fa0f00690e0 con 0x7fa0f00afa20 2023-10-19T13:00:48.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.725+0000 7fa0e77fe640 1 -- 172.21.15.57:0/2375503573 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule"]}]=0 v613) v1 ==== 156+0+0 (secure 0 0 0) 0x7fa0e001d0e0 con 0x7fa0f00afa20 2023-10-19T13:00:48.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.727+0000 7fa0f511d640 1 -- 172.21.15.57:0/2375503573 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fa0d006af60 msgr2=0x7fa0d006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:48.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.727+0000 7fa0f511d640 1 --2- 172.21.15.57:0/2375503573 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fa0d006af60 0x7fa0d006d420 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fa0d8035000 tx=0x7fa0d8004730 comp rx=0 tx=0).stop 2023-10-19T13:00:48.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.727+0000 7fa0f511d640 1 -- 172.21.15.57:0/2375503573 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0f00afa20 msgr2=0x7fa0f0155dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:48.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.727+0000 7fa0f511d640 1 --2- 172.21.15.57:0/2375503573 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0f00afa20 0x7fa0f0155dd0 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7fa0e000c970 tx=0x7fa0e000ce30 comp rx=0 tx=0).stop 2023-10-19T13:00:48.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.727+0000 7fa0f511d640 1 -- 172.21.15.57:0/2375503573 shutdown_connections 2023-10-19T13:00:48.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.727+0000 7fa0f511d640 1 --2- 172.21.15.57:0/2375503573 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fa0d006af60 0x7fa0d006d420 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:48.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.727+0000 7fa0f511d640 1 --2- 172.21.15.57:0/2375503573 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa0f00cbd20 0x7fa0f01538b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:48.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.727+0000 7fa0f511d640 1 --2- 172.21.15.57:0/2375503573 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa0f00b0530 0x7fa0f01e36d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:48.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.727+0000 7fa0f511d640 1 --2- 172.21.15.57:0/2375503573 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0f00afa20 0x7fa0f0155dd0 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:48.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.727+0000 7fa0f511d640 1 -- 172.21.15.57:0/2375503573 >> 172.21.15.57:0/2375503573 conn(0x7fa0f0069930 msgr2=0x7fa0f0069d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:48.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.727+0000 7fa0f511d640 1 -- 172.21.15.57:0/2375503573 shutdown_connections 2023-10-19T13:00:48.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.727+0000 7fa0f511d640 1 -- 172.21.15.57:0/2375503573 wait complete. 2023-10-19T13:00:48.741 INFO:tasks.cephfs_test_runner:Test listing non-existing snap schedules failure ... ok 2023-10-19T13:00:48.742 INFO:tasks.cephfs_test_runner:Starting test: test_non_existent_snap_schedule_list (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test listing snap schedules on a non-existing filesystem path failure 2023-10-19T13:00:48.742 INFO:tasks.cephfs_test_runner:test_non_existent_snap_schedule_list (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-10-19T13:00:48.743 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T13:00:48.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.850+0000 7f6e18683640 1 -- 172.21.15.57:0/1847998848 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6dfc005e10 msgr2=0x7f6dfc006290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:48.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.850+0000 7f6e18683640 1 --2- 172.21.15.57:0/1847998848 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6dfc005e10 0x7f6dfc006290 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f6e0800c610 tx=0x7f6dfc006900 comp rx=0 tx=0).stop 2023-10-19T13:00:48.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.850+0000 7f6e16c20640 1 -- 172.21.15.57:0/1847998848 reap_dead start 2023-10-19T13:00:48.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.850+0000 7f6e18683640 1 -- 172.21.15.57:0/1847998848 shutdown_connections 2023-10-19T13:00:48.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.850+0000 7f6e18683640 1 -- 172.21.15.57:0/1847998848 >> 172.21.15.57:0/1847998848 conn(0x7f6e10069930 msgr2=0x7f6e10069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:48.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.850+0000 7f6e18683640 1 -- 172.21.15.57:0/1847998848 shutdown_connections 2023-10-19T13:00:48.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.850+0000 7f6e18683640 1 -- 172.21.15.57:0/1847998848 wait complete. 2023-10-19T13:00:48.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.851+0000 7f6e18683640 1 Processor -- start 2023-10-19T13:00:48.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.851+0000 7f6e18683640 1 -- start start 2023-10-19T13:00:48.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.852+0000 7f6e18683640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6dfc005e10 0x7f6e1013c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:48.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.852+0000 7f6e18683640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6e10148320 0x7f6e101f0420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:48.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.852+0000 7f6e18683640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6e100ad840 0x7f6e100adcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:48.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.852+0000 7f6e18683640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e100ae230 con 0x7f6e10148320 2023-10-19T13:00:48.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.852+0000 7f6e18683640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e101f1350 con 0x7f6dfc005e10 2023-10-19T13:00:48.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.852+0000 7f6e18683640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e1013cb40 con 0x7f6e100ad840 2023-10-19T13:00:48.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.852+0000 7f6e15c1e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6e10148320 0x7f6e101f0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:48.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.852+0000 7f6e15c1e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6e10148320 0x7f6e101f0420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51368/0 (socket says 172.21.15.57:51368) 2023-10-19T13:00:48.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.852+0000 7f6e15c1e640 1 -- 172.21.15.57:0/3129021084 learned_addr learned my addr 172.21.15.57:0/3129021084 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:48.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.852+0000 7f6e1641f640 1 --2- 172.21.15.57:0/3129021084 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6dfc005e10 0x7f6e1013c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:48.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.852+0000 7f6e16c20640 1 --2- 172.21.15.57:0/3129021084 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6e100ad840 0x7f6e100adcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:48.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.853+0000 7f6e15c1e640 1 -- 172.21.15.57:0/3129021084 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6e100ad840 msgr2=0x7f6e100adcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:48.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.853+0000 7f6e15c1e640 1 --2- 172.21.15.57:0/3129021084 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6e100ad840 0x7f6e100adcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:48.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.853+0000 7f6e15c1e640 1 -- 172.21.15.57:0/3129021084 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6dfc005e10 msgr2=0x7f6e1013c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:48.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.853+0000 7f6e15c1e640 1 --2- 172.21.15.57:0/3129021084 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6dfc005e10 0x7f6e1013c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:48.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.853+0000 7f6e15c1e640 1 -- 172.21.15.57:0/3129021084 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e101f0d90 con 0x7f6e10148320 2023-10-19T13:00:48.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.853+0000 7f6e15c1e640 1 --2- 172.21.15.57:0/3129021084 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6e10148320 0x7f6e101f0420 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f6e00011bf0 tx=0x7f6e0000e780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:48.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.853+0000 7f6e06ffd640 1 -- 172.21.15.57:0/3129021084 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e00016070 con 0x7f6e10148320 2023-10-19T13:00:48.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.853+0000 7f6e06ffd640 1 -- 172.21.15.57:0/3129021084 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f6e0000ee30 con 0x7f6e10148320 2023-10-19T13:00:48.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.853+0000 7f6e18683640 1 -- 172.21.15.57:0/3129021084 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e101f1070 con 0x7f6e10148320 2023-10-19T13:00:48.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.853+0000 7f6e06ffd640 1 -- 172.21.15.57:0/3129021084 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e000044a0 con 0x7f6e10148320 2023-10-19T13:00:48.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.853+0000 7f6e18683640 1 -- 172.21.15.57:0/3129021084 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e1014be80 con 0x7f6e10148320 2023-10-19T13:00:48.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.854+0000 7f6e18683640 1 -- 172.21.15.57:0/3129021084 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e100690e0 con 0x7f6e10148320 2023-10-19T13:00:48.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.855+0000 7f6e06ffd640 1 -- 172.21.15.57:0/3129021084 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 40) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6e000035f0 con 0x7f6e10148320 2023-10-19T13:00:48.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.855+0000 7f6e06ffd640 1 --2- 172.21.15.57:0/3129021084 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f6df406af90 0x7f6df406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:48.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.855+0000 7f6e06ffd640 1 -- 172.21.15.57:0/3129021084 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5979+0+0 (secure 0 0 0) 0x7f6e0008c750 con 0x7f6e10148320 2023-10-19T13:00:48.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.855+0000 7f6e1641f640 1 --2- 172.21.15.57:0/3129021084 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f6df406af90 0x7f6df406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:48.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.856+0000 7f6e1641f640 1 --2- 172.21.15.57:0/3129021084 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f6df406af90 0x7f6df406d450 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f6e0c006600 tx=0x7f6e0c006b30 comp rx=0 tx=0).ready entity=mgr.6679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:48.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:48.859+0000 7f6e06ffd640 1 -- 172.21.15.57:0/3129021084 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6e0005b0b0 con 0x7f6e10148320 2023-10-19T13:00:49.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.039+0000 7f6e18683640 1 -- 172.21.15.57:0/3129021084 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6e100aeef0 con 0x7f6e10148320 2023-10-19T13:00:49.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.041+0000 7f6e06ffd640 1 -- 172.21.15.57:0/3129021084 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) v1 ==== 74+0+14264 (secure 0 0 0) 0x7f6e00061070 con 0x7f6e10148320 2023-10-19T13:00:49.044 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:00:49.044 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":92,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T13:00:47.706662+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.57:6833/2257211800":"2023-10-20T13:00:44.548793+0000","172.21.15.179:0/4135533550":"2023-10-20T13:00:42.059915+0000","172.21.15.179:0/3560014978":"2023-10-20T13:00:42.059915+0000","172.21.15.179:6833/137854273":"2023-10-20T13:00:42.059915+0000","172.21.15.179:6832/137854273":"2023-10-20T13:00:42.059915+0000","172.21.15.179:0/4157624230":"2023-10-20T13:00:42.059915+0000","172.21.15.179:0/3436591908":"2023-10-20T13:00:42.059915+0000","172.21.15.57:6835/2257211800":"2023-10-20T13:00:44.548793+0000","172.21.15.179:0/911511055":"2023-10-20T13:00:42.059915+0000","172.21.15.179:0/4093253624":"2023-10-20T13:00:36.611894+0000","172.21.15.179:6833/3815793798":"2023-10-20T13:00:36.611894+0000","172.21.15.179:0/2495886194":"2023-10-20T13:00:36.611894+0000","172.21.15.179:6832/3815793798":"2023-10-20T13:00:36.611894+0000","172.21.15.179:0/2414172521":"2023-10-20T13:00:36.611894+0000","172.21.15.179:0/4199202801":"2023-10-20T13:00:42.059915+0000","172.21.15.179:0/2931605339":"2023-10-20T13:00:36.611894+0000","172.21.15.179:0/1988099910":"2023-10-20T13:00:36.611894+0000","172.21.15.179:0/4264007972":"2023-10-20T13:00:36.611894+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T13:00:49.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.043+0000 7f6e18683640 1 -- 172.21.15.57:0/3129021084 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f6df406af90 msgr2=0x7f6df406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:49.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.043+0000 7f6e18683640 1 --2- 172.21.15.57:0/3129021084 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f6df406af90 0x7f6df406d450 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f6e0c006600 tx=0x7f6e0c006b30 comp rx=0 tx=0).stop 2023-10-19T13:00:49.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.043+0000 7f6e18683640 1 -- 172.21.15.57:0/3129021084 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6e10148320 msgr2=0x7f6e101f0420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:49.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.043+0000 7f6e18683640 1 --2- 172.21.15.57:0/3129021084 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6e10148320 0x7f6e101f0420 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f6e00011bf0 tx=0x7f6e0000e780 comp rx=0 tx=0).stop 2023-10-19T13:00:49.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.043+0000 7f6e18683640 1 -- 172.21.15.57:0/3129021084 shutdown_connections 2023-10-19T13:00:49.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.043+0000 7f6e18683640 1 --2- 172.21.15.57:0/3129021084 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f6df406af90 0x7f6df406d450 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:49.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.043+0000 7f6e18683640 1 --2- 172.21.15.57:0/3129021084 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6e100ad840 0x7f6e100adcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:49.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.043+0000 7f6e18683640 1 --2- 172.21.15.57:0/3129021084 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6e10148320 0x7f6e101f0420 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:49.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.043+0000 7f6e18683640 1 --2- 172.21.15.57:0/3129021084 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6dfc005e10 0x7f6e1013c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:49.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.043+0000 7f6e18683640 1 -- 172.21.15.57:0/3129021084 >> 172.21.15.57:0/3129021084 conn(0x7f6e10069930 msgr2=0x7f6e100ace20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:49.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.043+0000 7f6e18683640 1 -- 172.21.15.57:0/3129021084 shutdown_connections 2023-10-19T13:00:49.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.043+0000 7f6e18683640 1 -- 172.21.15.57:0/3129021084 wait complete. 2023-10-19T13:00:49.056 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-19T12:44:07.371186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-19T13:00:49.057 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-19T13:00:49.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.167+0000 7ff20ce83640 1 -- 172.21.15.57:0/851168002 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff208070af0 msgr2=0x7ff2080affe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:49.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.167+0000 7ff20ce83640 1 --2- 172.21.15.57:0/851168002 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff208070af0 0x7ff2080affe0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff1f8005fd0 tx=0x7ff1f802c240 comp rx=0 tx=0).stop 2023-10-19T13:00:49.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.167+0000 7ff20ce83640 1 -- 172.21.15.57:0/851168002 shutdown_connections 2023-10-19T13:00:49.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.167+0000 7ff20ce83640 1 --2- 172.21.15.57:0/851168002 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff208070af0 0x7ff2080affe0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:49.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.167+0000 7ff20ce83640 1 --2- 172.21.15.57:0/851168002 >> v2:172.21.15.179:3300/0 conn(0x7ff20813e520 0x7ff20806bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:49.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.167+0000 7ff20ce83640 1 -- 172.21.15.57:0/851168002 >> 172.21.15.57:0/851168002 conn(0x7ff208069930 msgr2=0x7ff208069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:49.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.167+0000 7ff20ce83640 1 -- 172.21.15.57:0/851168002 shutdown_connections 2023-10-19T13:00:49.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.167+0000 7ff20ce83640 1 -- 172.21.15.57:0/851168002 wait complete. 2023-10-19T13:00:49.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.169+0000 7ff20ce83640 1 Processor -- start 2023-10-19T13:00:49.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.169+0000 7ff20ce83640 1 -- start start 2023-10-19T13:00:49.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.169+0000 7ff20ce83640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff208070af0 0x7ff2080c5da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:49.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.169+0000 7ff20ce83640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff20813e520 0x7ff20813b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:49.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.169+0000 7ff20ce83640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff208141690 0x7ff20806f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:49.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.169+0000 7ff20ce83640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2081f1110 con 0x7ff208070af0 2023-10-19T13:00:49.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.169+0000 7ff20ce83640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2080c65f0 con 0x7ff20813e520 2023-10-19T13:00:49.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.169+0000 7ff20ce83640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2081f20c0 con 0x7ff208141690 2023-10-19T13:00:49.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.170+0000 7ff20659c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff208070af0 0x7ff2080c5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:49.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.170+0000 7ff20659c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff208070af0 0x7ff2080c5da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51378/0 (socket says 172.21.15.57:51378) 2023-10-19T13:00:49.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.170+0000 7ff20659c640 1 -- 172.21.15.57:0/3420651089 learned_addr learned my addr 172.21.15.57:0/3420651089 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:49.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.170+0000 7ff206d9d640 1 --2- 172.21.15.57:0/3420651089 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff208141690 0x7ff20806f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:49.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.170+0000 7ff205d9b640 1 --2- 172.21.15.57:0/3420651089 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff20813e520 0x7ff20813b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:49.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.170+0000 7ff20659c640 1 -- 172.21.15.57:0/3420651089 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff208141690 msgr2=0x7ff20806f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:49.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.170+0000 7ff20659c640 1 --2- 172.21.15.57:0/3420651089 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff208141690 0x7ff20806f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:49.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.170+0000 7ff20659c640 1 -- 172.21.15.57:0/3420651089 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff20813e520 msgr2=0x7ff20813b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:49.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.170+0000 7ff20659c640 1 --2- 172.21.15.57:0/3420651089 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff20813e520 0x7ff20813b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:49.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.170+0000 7ff20659c640 1 -- 172.21.15.57:0/3420651089 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1f8009900 con 0x7ff208070af0 2023-10-19T13:00:49.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.170+0000 7ff205d9b640 1 --2- 172.21.15.57:0/3420651089 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff20813e520 0x7ff20813b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:00:49.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.170+0000 7ff20659c640 1 --2- 172.21.15.57:0/3420651089 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff208070af0 0x7ff2080c5da0 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7ff20000e9f0 tx=0x7ff20000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:49.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.170+0000 7ff1feffd640 1 -- 172.21.15.57:0/3420651089 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff200009700 con 0x7ff208070af0 2023-10-19T13:00:49.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.170+0000 7ff1feffd640 1 -- 172.21.15.57:0/3420651089 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7ff20000ce10 con 0x7ff208070af0 2023-10-19T13:00:49.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.170+0000 7ff1feffd640 1 -- 172.21.15.57:0/3420651089 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff200010690 con 0x7ff208070af0 2023-10-19T13:00:49.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.170+0000 7ff20ce83640 1 -- 172.21.15.57:0/3420651089 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2081f1300 con 0x7ff208070af0 2023-10-19T13:00:49.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.170+0000 7ff20ce83640 1 -- 172.21.15.57:0/3420651089 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2081f18c0 con 0x7ff208070af0 2023-10-19T13:00:49.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.171+0000 7ff20ce83640 1 -- 172.21.15.57:0/3420651089 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2080abf90 con 0x7ff208070af0 2023-10-19T13:00:49.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.172+0000 7ff1feffd640 1 -- 172.21.15.57:0/3420651089 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 40) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff2000107f0 con 0x7ff208070af0 2023-10-19T13:00:49.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.172+0000 7ff1feffd640 1 --2- 172.21.15.57:0/3420651089 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7ff1e806af60 0x7ff1e806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:49.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.172+0000 7ff1feffd640 1 -- 172.21.15.57:0/3420651089 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5979+0+0 (secure 0 0 0) 0x7ff200014070 con 0x7ff208070af0 2023-10-19T13:00:49.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.173+0000 7ff205d9b640 1 --2- 172.21.15.57:0/3420651089 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7ff1e806af60 0x7ff1e806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:49.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.173+0000 7ff205d9b640 1 --2- 172.21.15.57:0/3420651089 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7ff1e806af60 0x7ff1e806d420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ff2080a7900 tx=0x7ff1f0009040 comp rx=0 tx=0).ready entity=mgr.6679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:49.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.176+0000 7ff1feffd640 1 -- 172.21.15.57:0/3420651089 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff200051ca0 con 0x7ff208070af0 2023-10-19T13:00:49.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.349+0000 7ff20ce83640 1 -- 172.21.15.57:0/3420651089 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff2080690e0 con 0x7ff208070af0 2023-10-19T13:00:49.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.349+0000 7ff1feffd640 1 -- 172.21.15.57:0/3420651089 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v92) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff200058e10 con 0x7ff208070af0 2023-10-19T13:00:49.353 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-19T13:00:49.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.351+0000 7ff20ce83640 1 -- 172.21.15.57:0/3420651089 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7ff1e806af60 msgr2=0x7ff1e806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:49.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.351+0000 7ff20ce83640 1 --2- 172.21.15.57:0/3420651089 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7ff1e806af60 0x7ff1e806d420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ff2080a7900 tx=0x7ff1f0009040 comp rx=0 tx=0).stop 2023-10-19T13:00:49.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.351+0000 7ff20ce83640 1 -- 172.21.15.57:0/3420651089 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff208070af0 msgr2=0x7ff2080c5da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:49.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.351+0000 7ff20ce83640 1 --2- 172.21.15.57:0/3420651089 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff208070af0 0x7ff2080c5da0 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7ff20000e9f0 tx=0x7ff20000eeb0 comp rx=0 tx=0).stop 2023-10-19T13:00:49.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.351+0000 7ff20ce83640 1 -- 172.21.15.57:0/3420651089 shutdown_connections 2023-10-19T13:00:49.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.351+0000 7ff20ce83640 1 --2- 172.21.15.57:0/3420651089 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7ff1e806af60 0x7ff1e806d420 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:49.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.351+0000 7ff20ce83640 1 --2- 172.21.15.57:0/3420651089 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff208141690 0x7ff20806f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:49.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.351+0000 7ff20ce83640 1 --2- 172.21.15.57:0/3420651089 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff20813e520 0x7ff20813b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:49.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.351+0000 7ff20ce83640 1 --2- 172.21.15.57:0/3420651089 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff208070af0 0x7ff2080c5da0 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:49.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.351+0000 7ff20ce83640 1 -- 172.21.15.57:0/3420651089 >> 172.21.15.57:0/3420651089 conn(0x7ff208069930 msgr2=0x7ff2080703e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:49.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.351+0000 7ff20ce83640 1 -- 172.21.15.57:0/3420651089 shutdown_connections 2023-10-19T13:00:49.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.351+0000 7ff20ce83640 1 -- 172.21.15.57:0/3420651089 wait complete. 2023-10-19T13:00:49.365 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list' 2023-10-19T13:00:49.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.471+0000 7fb3bfa40640 1 -- 172.21.15.57:0/4284576427 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3b80cd6d0 msgr2=0x7fb3b80cdb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:49.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.471+0000 7fb3bfa40640 1 --2- 172.21.15.57:0/4284576427 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3b80cd6d0 0x7fb3b80cdb30 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7fb3b8072420 tx=0x7fb3ac02bd30 comp rx=0 tx=0).stop 2023-10-19T13:00:49.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.471+0000 7fb3bfa40640 1 -- 172.21.15.57:0/4284576427 shutdown_connections 2023-10-19T13:00:49.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.471+0000 7fb3bfa40640 1 --2- 172.21.15.57:0/4284576427 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3b80cd6d0 0x7fb3b80cdb30 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:49.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.471+0000 7fb3bfa40640 1 --2- 172.21.15.57:0/4284576427 >> v2:172.21.15.179:3300/0 conn(0x7fb3b8142850 0x7fb3b8159440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:49.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.471+0000 7fb3bfa40640 1 -- 172.21.15.57:0/4284576427 >> 172.21.15.57:0/4284576427 conn(0x7fb3b80af7a0 msgr2=0x7fb3b80afb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:49.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.471+0000 7fb3bfa40640 1 -- 172.21.15.57:0/4284576427 shutdown_connections 2023-10-19T13:00:49.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.471+0000 7fb3bfa40640 1 -- 172.21.15.57:0/4284576427 wait complete. 2023-10-19T13:00:49.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.472+0000 7fb3bfa40640 1 Processor -- start 2023-10-19T13:00:49.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.472+0000 7fb3bfa40640 1 -- start start 2023-10-19T13:00:49.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.473+0000 7fb3bfa40640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3b8069720 0x7fb3b8157ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:49.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.473+0000 7fb3bfa40640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb3b80cd6d0 0x7fb3b81e16a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:49.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.473+0000 7fb3bfa40640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb3b8142850 0x7fb3b81e1be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:49.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.473+0000 7fb3bfa40640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3b81e3fd0 con 0x7fb3b8069720 2023-10-19T13:00:49.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.473+0000 7fb3bfa40640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3b8158af0 con 0x7fb3b8142850 2023-10-19T13:00:49.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.473+0000 7fb3bfa40640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3b81e33c0 con 0x7fb3b80cd6d0 2023-10-19T13:00:49.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.473+0000 7fb3bd7dc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3b8069720 0x7fb3b8157ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:49.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.473+0000 7fb3bd7dc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3b8069720 0x7fb3b8157ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51398/0 (socket says 172.21.15.57:51398) 2023-10-19T13:00:49.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.473+0000 7fb3bd7dc640 1 -- 172.21.15.57:0/555123311 learned_addr learned my addr 172.21.15.57:0/555123311 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:49.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.473+0000 7fb3bd7dc640 1 -- 172.21.15.57:0/555123311 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb3b80cd6d0 msgr2=0x7fb3b81e16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:49.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.473+0000 7fb3bd7dc640 1 --2- 172.21.15.57:0/555123311 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb3b80cd6d0 0x7fb3b81e16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:49.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.473+0000 7fb3bd7dc640 1 -- 172.21.15.57:0/555123311 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb3b8142850 msgr2=0x7fb3b81e1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:49.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.473+0000 7fb3bdfdd640 1 --2- 172.21.15.57:0/555123311 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb3b8142850 0x7fb3b81e1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:49.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.473+0000 7fb3bd7dc640 1 --2- 172.21.15.57:0/555123311 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb3b8142850 0x7fb3b81e1be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:49.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.474+0000 7fb3bd7dc640 1 -- 172.21.15.57:0/555123311 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3ac009560 con 0x7fb3b8069720 2023-10-19T13:00:49.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.474+0000 7fb3bd7dc640 1 --2- 172.21.15.57:0/555123311 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3b8069720 0x7fb3b8157ec0 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fb3b000d6d0 tx=0x7fb3b000db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:49.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.474+0000 7fb3b5ffb640 1 -- 172.21.15.57:0/555123311 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3b0005220 con 0x7fb3b8069720 2023-10-19T13:00:49.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.474+0000 7fb3b5ffb640 1 -- 172.21.15.57:0/555123311 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fb3b0005490 con 0x7fb3b8069720 2023-10-19T13:00:49.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.474+0000 7fb3bfa40640 1 -- 172.21.15.57:0/555123311 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3b81e36a0 con 0x7fb3b8069720 2023-10-19T13:00:49.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.474+0000 7fb3b5ffb640 1 -- 172.21.15.57:0/555123311 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3b0009a80 con 0x7fb3b8069720 2023-10-19T13:00:49.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.474+0000 7fb3bfa40640 1 -- 172.21.15.57:0/555123311 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3b81e3c60 con 0x7fb3b8069720 2023-10-19T13:00:49.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.475+0000 7fb3bfa40640 1 -- 172.21.15.57:0/555123311 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3b80690e0 con 0x7fb3b8069720 2023-10-19T13:00:49.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.475+0000 7fb3b5ffb640 1 -- 172.21.15.57:0/555123311 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 40) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb3b00047a0 con 0x7fb3b8069720 2023-10-19T13:00:49.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.476+0000 7fb3b5ffb640 1 --2- 172.21.15.57:0/555123311 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fb3a406af60 0x7fb3a406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:49.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.476+0000 7fb3b5ffb640 1 -- 172.21.15.57:0/555123311 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5979+0+0 (secure 0 0 0) 0x7fb3b0089e40 con 0x7fb3b8069720 2023-10-19T13:00:49.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.476+0000 7fb3bcfdb640 1 --2- 172.21.15.57:0/555123311 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fb3a406af60 0x7fb3a406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:49.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.477+0000 7fb3bcfdb640 1 --2- 172.21.15.57:0/555123311 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fb3a406af60 0x7fb3a406d420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb3ac035000 tx=0x7fb3ac0046c0 comp rx=0 tx=0).ready entity=mgr.6679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:49.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.480+0000 7fb3b5ffb640 1 -- 172.21.15.57:0/555123311 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb3b0054b30 con 0x7fb3b8069720 2023-10-19T13:00:49.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.724+0000 7fb3bfa40640 1 -- 172.21.15.57:0/555123311 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list"]} v 0) v1 -- 0x7fb3b8159a60 con 0x7fb3b8069720 2023-10-19T13:00:49.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.726+0000 7fb3b5ffb640 1 -- 172.21.15.57:0/555123311 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list"]}]=0 v614) v1 ==== 169+0+0 (secure 0 0 0) 0x7fb3b0017070 con 0x7fb3b8069720 2023-10-19T13:00:49.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.728+0000 7fb3bfa40640 1 -- 172.21.15.57:0/555123311 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fb3a406af60 msgr2=0x7fb3a406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:49.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.728+0000 7fb3bfa40640 1 --2- 172.21.15.57:0/555123311 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fb3a406af60 0x7fb3a406d420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb3ac035000 tx=0x7fb3ac0046c0 comp rx=0 tx=0).stop 2023-10-19T13:00:49.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.728+0000 7fb3bfa40640 1 -- 172.21.15.57:0/555123311 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3b8069720 msgr2=0x7fb3b8157ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:49.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.728+0000 7fb3bfa40640 1 --2- 172.21.15.57:0/555123311 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3b8069720 0x7fb3b8157ec0 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fb3b000d6d0 tx=0x7fb3b000db90 comp rx=0 tx=0).stop 2023-10-19T13:00:49.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.728+0000 7fb3bfa40640 1 -- 172.21.15.57:0/555123311 shutdown_connections 2023-10-19T13:00:49.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.728+0000 7fb3bfa40640 1 --2- 172.21.15.57:0/555123311 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fb3a406af60 0x7fb3a406d420 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:49.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.728+0000 7fb3bfa40640 1 --2- 172.21.15.57:0/555123311 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb3b8142850 0x7fb3b81e1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:49.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.728+0000 7fb3bfa40640 1 --2- 172.21.15.57:0/555123311 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb3b80cd6d0 0x7fb3b81e16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:49.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.728+0000 7fb3bfa40640 1 --2- 172.21.15.57:0/555123311 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3b8069720 0x7fb3b8157ec0 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:49.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.728+0000 7fb3bfa40640 1 -- 172.21.15.57:0/555123311 >> 172.21.15.57:0/555123311 conn(0x7fb3b80af7a0 msgr2=0x7fb3b813f160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:49.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.728+0000 7fb3bfa40640 1 -- 172.21.15.57:0/555123311 shutdown_connections 2023-10-19T13:00:49.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.728+0000 7fb3bfa40640 1 -- 172.21.15.57:0/555123311 wait complete. 2023-10-19T13:00:49.741 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2023-10-19T13:00:49.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.848+0000 7fa4aaf5e640 1 -- 172.21.15.57:0/3000011800 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa48c003970 msgr2=0x7fa48c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:49.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.848+0000 7fa4aaf5e640 1 --2- 172.21.15.57:0/3000011800 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa48c003970 0x7fa48c003d90 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7fa494009b50 tx=0x7fa48c0057f0 comp rx=0 tx=0).stop 2023-10-19T13:00:49.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.848+0000 7fa4aaf5e640 1 -- 172.21.15.57:0/3000011800 shutdown_connections 2023-10-19T13:00:49.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.848+0000 7fa4aaf5e640 1 --2- 172.21.15.57:0/3000011800 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa48c003970 0x7fa48c003d90 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:49.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.848+0000 7fa4aaf5e640 1 -- 172.21.15.57:0/3000011800 >> 172.21.15.57:0/3000011800 conn(0x7fa4a406fae0 msgr2=0x7fa4a406fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:49.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.848+0000 7fa4aaf5e640 1 -- 172.21.15.57:0/3000011800 shutdown_connections 2023-10-19T13:00:49.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.848+0000 7fa4aaf5e640 1 -- 172.21.15.57:0/3000011800 wait complete. 2023-10-19T13:00:49.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.849+0000 7fa4aaf5e640 1 Processor -- start 2023-10-19T13:00:49.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.849+0000 7fa4aaf5e640 1 -- start start 2023-10-19T13:00:49.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.850+0000 7fa4aaf5e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa48c003970 0x7fa4a414c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:49.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.850+0000 7fa4aaf5e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa4a414caa0 0x7fa4a406ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:49.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.850+0000 7fa4aaf5e640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa4a414cff0 0x7fa4a406d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:49.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.850+0000 7fa4aaf5e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a4159650 con 0x7fa48c003970 2023-10-19T13:00:49.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.850+0000 7fa4aaf5e640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a414fa40 con 0x7fa4a414caa0 2023-10-19T13:00:49.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.850+0000 7fa4aaf5e640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a41502e0 con 0x7fa4a414cff0 2023-10-19T13:00:49.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.850+0000 7fa4a8cfa640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa48c003970 0x7fa4a414c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:49.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.850+0000 7fa4a8cfa640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa48c003970 0x7fa4a414c380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51408/0 (socket says 172.21.15.57:51408) 2023-10-19T13:00:49.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.850+0000 7fa4a8cfa640 1 -- 172.21.15.57:0/369041559 learned_addr learned my addr 172.21.15.57:0/369041559 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:49.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.850+0000 7fa4a8cfa640 1 -- 172.21.15.57:0/369041559 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa4a414cff0 msgr2=0x7fa4a406d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:49.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.850+0000 7fa4a3fff640 1 --2- 172.21.15.57:0/369041559 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa4a414caa0 0x7fa4a406ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:49.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.850+0000 7fa4a8cfa640 1 --2- 172.21.15.57:0/369041559 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa4a414cff0 0x7fa4a406d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:49.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.850+0000 7fa4a8cfa640 1 -- 172.21.15.57:0/369041559 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa4a414caa0 msgr2=0x7fa4a406ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:49.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.850+0000 7fa4a8cfa640 1 --2- 172.21.15.57:0/369041559 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa4a414caa0 0x7fa4a406ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:49.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.850+0000 7fa4a8cfa640 1 -- 172.21.15.57:0/369041559 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4a414c8c0 con 0x7fa48c003970 2023-10-19T13:00:49.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.850+0000 7fa4a3fff640 1 --2- 172.21.15.57:0/369041559 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa4a414caa0 0x7fa4a406ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T13:00:49.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.851+0000 7fa4a8cfa640 1 --2- 172.21.15.57:0/369041559 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa48c003970 0x7fa4a414c380 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7fa494009b50 tx=0x7fa494033c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:49.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.851+0000 7fa4a17fa640 1 -- 172.21.15.57:0/369041559 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa494004460 con 0x7fa48c003970 2023-10-19T13:00:49.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.851+0000 7fa4a17fa640 1 -- 172.21.15.57:0/369041559 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fa494033e40 con 0x7fa48c003970 2023-10-19T13:00:49.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.851+0000 7fa4aaf5e640 1 -- 172.21.15.57:0/369041559 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4a415d0c0 con 0x7fa48c003970 2023-10-19T13:00:49.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.851+0000 7fa4a17fa640 1 -- 172.21.15.57:0/369041559 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa494003850 con 0x7fa48c003970 2023-10-19T13:00:49.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.851+0000 7fa4aaf5e640 1 -- 172.21.15.57:0/369041559 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4a414fc90 con 0x7fa48c003970 2023-10-19T13:00:49.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.852+0000 7fa4aaf5e640 1 -- 172.21.15.57:0/369041559 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4a4159650 con 0x7fa48c003970 2023-10-19T13:00:49.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.852+0000 7fa4a17fa640 1 -- 172.21.15.57:0/369041559 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 40) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fa4940039b0 con 0x7fa48c003970 2023-10-19T13:00:49.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.853+0000 7fa4a17fa640 1 --2- 172.21.15.57:0/369041559 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fa49006af60 0x7fa49006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:49.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.853+0000 7fa4a17fa640 1 -- 172.21.15.57:0/369041559 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5979+0+0 (secure 0 0 0) 0x7fa4940acb80 con 0x7fa48c003970 2023-10-19T13:00:49.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.853+0000 7fa4a3fff640 1 --2- 172.21.15.57:0/369041559 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fa49006af60 0x7fa49006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:49.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.853+0000 7fa4a3fff640 1 --2- 172.21.15.57:0/369041559 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fa49006af60 0x7fa49006d420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa4980011e0 tx=0x7fa498001150 comp rx=0 tx=0).ready entity=mgr.6679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:49.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:49.857+0000 7fa4a17fa640 1 -- 172.21.15.57:0/369041559 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa494077ab0 con 0x7fa48c003970 2023-10-19T13:00:50.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.033+0000 7fa4aaf5e640 1 -- 172.21.15.57:0/369041559 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "config dump"} v 0) v1 -- 0x7fa4a40690e0 con 0x7fa48c003970 2023-10-19T13:00:50.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.033+0000 7fa4a17fa640 1 -- 172.21.15.57:0/369041559 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v42) v1 ==== 59+0+2220 (secure 0 0 0) 0x7fa49404b070 con 0x7fa48c003970 2023-10-19T13:00:50.037 INFO:teuthology.orchestra.run.smithi057.stdout:WHO MASK LEVEL OPTION VALUE RO 2023-10-19T13:00:50.037 INFO:teuthology.orchestra.run.smithi057.stdout:mon advanced mon_op_complaint_time 120 2023-10-19T13:00:50.037 INFO:teuthology.orchestra.run.smithi057.stdout:osd advanced osd_op_complaint_time 180.000000 2023-10-19T13:00:50.037 INFO:teuthology.orchestra.run.smithi057.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 4775.995373 2023-10-19T13:00:50.038 INFO:teuthology.orchestra.run.smithi057.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 4208.118410 2023-10-19T13:00:50.038 INFO:teuthology.orchestra.run.smithi057.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 4541.123030 2023-10-19T13:00:50.038 INFO:teuthology.orchestra.run.smithi057.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 4877.592206 2023-10-19T13:00:50.038 INFO:teuthology.orchestra.run.smithi057.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 5019.748032 2023-10-19T13:00:50.038 INFO:teuthology.orchestra.run.smithi057.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 5000.074376 2023-10-19T13:00:50.039 INFO:teuthology.orchestra.run.smithi057.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 5153.479439 2023-10-19T13:00:50.039 INFO:teuthology.orchestra.run.smithi057.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 4977.363845 2023-10-19T13:00:50.039 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced debug_mds 20/20 2023-10-19T13:00:50.039 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced debug_mds_balancer 20/20 2023-10-19T13:00:50.039 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced debug_ms 1/1 2023-10-19T13:00:50.040 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced mds_bal_fragment_size_max 10000 2023-10-19T13:00:50.040 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced mds_bal_merge_size 5 2023-10-19T13:00:50.040 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced mds_bal_split_bits 3 2023-10-19T13:00:50.040 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced mds_bal_split_size 100 2023-10-19T13:00:50.040 INFO:teuthology.orchestra.run.smithi057.stdout:mds dev mds_debug_frag true 2023-10-19T13:00:50.040 INFO:teuthology.orchestra.run.smithi057.stdout:mds dev mds_debug_scatterstat true 2023-10-19T13:00:50.041 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced mds_op_complaint_time 180.000000 2023-10-19T13:00:50.041 INFO:teuthology.orchestra.run.smithi057.stdout:mds dev mds_verify_scatter true 2023-10-19T13:00:50.041 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced osd_op_complaint_time 180.000000 2023-10-19T13:00:50.041 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced rados_mon_op_timeout 900 2023-10-19T13:00:50.041 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced rados_osd_op_timeout 900 2023-10-19T13:00:50.041 INFO:teuthology.orchestra.run.smithi057.stdout:client advanced client_mount_timeout 600 2023-10-19T13:00:50.042 INFO:teuthology.orchestra.run.smithi057.stdout:client advanced debug_client 20/20 2023-10-19T13:00:50.042 INFO:teuthology.orchestra.run.smithi057.stdout:client advanced debug_ms 1/1 2023-10-19T13:00:50.042 INFO:teuthology.orchestra.run.smithi057.stdout:client advanced rados_mon_op_timeout 900 2023-10-19T13:00:50.042 INFO:teuthology.orchestra.run.smithi057.stdout:client advanced rados_osd_op_timeout 900 2023-10-19T13:00:50.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.035+0000 7fa4aaf5e640 1 -- 172.21.15.57:0/369041559 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fa49006af60 msgr2=0x7fa49006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:50.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.035+0000 7fa4aaf5e640 1 --2- 172.21.15.57:0/369041559 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fa49006af60 0x7fa49006d420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa4980011e0 tx=0x7fa498001150 comp rx=0 tx=0).stop 2023-10-19T13:00:50.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.035+0000 7fa4aaf5e640 1 -- 172.21.15.57:0/369041559 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa48c003970 msgr2=0x7fa4a414c380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:50.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.035+0000 7fa4aaf5e640 1 --2- 172.21.15.57:0/369041559 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa48c003970 0x7fa4a414c380 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7fa494009b50 tx=0x7fa494033c10 comp rx=0 tx=0).stop 2023-10-19T13:00:50.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.035+0000 7fa4aaf5e640 1 -- 172.21.15.57:0/369041559 shutdown_connections 2023-10-19T13:00:50.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.035+0000 7fa4aaf5e640 1 --2- 172.21.15.57:0/369041559 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fa49006af60 0x7fa49006d420 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:50.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.035+0000 7fa4aaf5e640 1 --2- 172.21.15.57:0/369041559 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa4a414cff0 0x7fa4a406d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:50.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.035+0000 7fa4aaf5e640 1 --2- 172.21.15.57:0/369041559 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa4a414caa0 0x7fa4a406ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:50.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.035+0000 7fa4aaf5e640 1 --2- 172.21.15.57:0/369041559 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa48c003970 0x7fa4a414c380 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:50.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.035+0000 7fa4aaf5e640 1 -- 172.21.15.57:0/369041559 >> 172.21.15.57:0/369041559 conn(0x7fa4a406fae0 msgr2=0x7fa4a40acff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:50.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.035+0000 7fa4aaf5e640 1 -- 172.21.15.57:0/369041559 shutdown_connections 2023-10-19T13:00:50.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.035+0000 7fa4aaf5e640 1 -- 172.21.15.57:0/369041559 wait complete. 2023-10-19T13:00:50.049 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-19T13:00:50.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.159+0000 7f32de105640 1 -- 172.21.15.57:0/389489647 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f32c0004d00 msgr2=0x7f32c0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:50.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.159+0000 7f32de105640 1 --2- 172.21.15.57:0/389489647 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f32c0004d00 0x7f32c0005180 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f32d000c640 tx=0x7f32c00057f0 comp rx=0 tx=0).stop 2023-10-19T13:00:50.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.159+0000 7f32d7fff640 1 -- 172.21.15.57:0/389489647 reap_dead start 2023-10-19T13:00:50.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.159+0000 7f32de105640 1 -- 172.21.15.57:0/389489647 shutdown_connections 2023-10-19T13:00:50.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.159+0000 7f32de105640 1 -- 172.21.15.57:0/389489647 >> 172.21.15.57:0/389489647 conn(0x7f32d809a030 msgr2=0x7f32d80d0d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:50.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.159+0000 7f32de105640 1 -- 172.21.15.57:0/389489647 shutdown_connections 2023-10-19T13:00:50.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.159+0000 7f32de105640 1 -- 172.21.15.57:0/389489647 wait complete. 2023-10-19T13:00:50.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.160+0000 7f32de105640 1 Processor -- start 2023-10-19T13:00:50.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.160+0000 7f32de105640 1 -- start start 2023-10-19T13:00:50.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.161+0000 7f32de105640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f32c0004d00 0x7f32d814efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:50.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.161+0000 7f32de105640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f32d80ca090 0x7f32d80ca510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:50.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.161+0000 7f32de105640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f32d80c7df0 0x7f32d80c8270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:50.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.161+0000 7f32de105640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32d80c87b0 con 0x7f32c0004d00 2023-10-19T13:00:50.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.161+0000 7f32de105640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32d815e8f0 con 0x7f32d80c7df0 2023-10-19T13:00:50.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.161+0000 7f32de105640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32d815e330 con 0x7f32d80ca090 2023-10-19T13:00:50.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.161+0000 7f32d77fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f32c0004d00 0x7f32d814efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:50.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.161+0000 7f32d77fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f32c0004d00 0x7f32d814efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51436/0 (socket says 172.21.15.57:51436) 2023-10-19T13:00:50.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.161+0000 7f32d77fe640 1 -- 172.21.15.57:0/4131388111 learned_addr learned my addr 172.21.15.57:0/4131388111 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:50.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.161+0000 7f32d7fff640 1 --2- 172.21.15.57:0/4131388111 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f32d80c7df0 0x7f32d80c8270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:50.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.161+0000 7f32d6ffd640 1 --2- 172.21.15.57:0/4131388111 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f32d80ca090 0x7f32d80ca510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:50.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.161+0000 7f32d77fe640 1 -- 172.21.15.57:0/4131388111 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f32d80ca090 msgr2=0x7f32d80ca510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:50.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.161+0000 7f32d77fe640 1 --2- 172.21.15.57:0/4131388111 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f32d80ca090 0x7f32d80ca510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:50.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.161+0000 7f32d77fe640 1 -- 172.21.15.57:0/4131388111 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f32d80c7df0 msgr2=0x7f32d80c8270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:50.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.161+0000 7f32d77fe640 1 --2- 172.21.15.57:0/4131388111 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f32d80c7df0 0x7f32d80c8270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:50.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.161+0000 7f32d77fe640 1 -- 172.21.15.57:0/4131388111 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32d815e610 con 0x7f32c0004d00 2023-10-19T13:00:50.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.162+0000 7f32d77fe640 1 --2- 172.21.15.57:0/4131388111 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f32c0004d00 0x7f32d814efe0 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f32c8011ae0 tx=0x7f32c8009e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:50.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.162+0000 7f32bffff640 1 -- 172.21.15.57:0/4131388111 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32c8004460 con 0x7f32c0004d00 2023-10-19T13:00:50.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.162+0000 7f32bffff640 1 -- 172.21.15.57:0/4131388111 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f32c800fa00 con 0x7f32c0004d00 2023-10-19T13:00:50.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.162+0000 7f32bffff640 1 -- 172.21.15.57:0/4131388111 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32c800fe10 con 0x7f32c0004d00 2023-10-19T13:00:50.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.162+0000 7f32de105640 1 -- 172.21.15.57:0/4131388111 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32d815bc00 con 0x7f32c0004d00 2023-10-19T13:00:50.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.162+0000 7f32de105640 1 -- 172.21.15.57:0/4131388111 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32d814f520 con 0x7f32c0004d00 2023-10-19T13:00:50.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.163+0000 7f32de105640 1 -- 172.21.15.57:0/4131388111 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32d815ba20 con 0x7f32c0004d00 2023-10-19T13:00:50.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.163+0000 7f32bffff640 1 -- 172.21.15.57:0/4131388111 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 40) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f32c80035f0 con 0x7f32c0004d00 2023-10-19T13:00:50.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.164+0000 7f32bffff640 1 --2- 172.21.15.57:0/4131388111 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f32c406af60 0x7f32c406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:50.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.164+0000 7f32bffff640 1 -- 172.21.15.57:0/4131388111 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5979+0+0 (secure 0 0 0) 0x7f32c808a650 con 0x7f32c0004d00 2023-10-19T13:00:50.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.164+0000 7f32d6ffd640 1 --2- 172.21.15.57:0/4131388111 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f32c406af60 0x7f32c406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:50.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.164+0000 7f32d6ffd640 1 --2- 172.21.15.57:0/4131388111 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f32c406af60 0x7f32c406d420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f32d81f4a10 tx=0x7f32cc005f90 comp rx=0 tx=0).ready entity=mgr.6679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:50.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.168+0000 7f32bffff640 1 -- 172.21.15.57:0/4131388111 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f32c80555d0 con 0x7f32c0004d00 2023-10-19T13:00:50.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.346+0000 7f32de105640 1 -- 172.21.15.57:0/4131388111 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f32d80675f0 con 0x7f32c0004d00 2023-10-19T13:00:50.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.349+0000 7f32bffff640 1 -- 172.21.15.57:0/4131388111 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v43)=0 v43) v1 ==== 122+0+0 (secure 0 0 0) 0x7f32c8058bf0 con 0x7f32c0004d00 2023-10-19T13:00:50.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.352+0000 7f32de105640 1 -- 172.21.15.57:0/4131388111 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f32c406af60 msgr2=0x7f32c406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:50.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.352+0000 7f32de105640 1 --2- 172.21.15.57:0/4131388111 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f32c406af60 0x7f32c406d420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f32d81f4a10 tx=0x7f32cc005f90 comp rx=0 tx=0).stop 2023-10-19T13:00:50.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.352+0000 7f32de105640 1 -- 172.21.15.57:0/4131388111 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f32c0004d00 msgr2=0x7f32d814efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:50.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.352+0000 7f32de105640 1 --2- 172.21.15.57:0/4131388111 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f32c0004d00 0x7f32d814efe0 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f32c8011ae0 tx=0x7f32c8009e90 comp rx=0 tx=0).stop 2023-10-19T13:00:50.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.352+0000 7f32de105640 1 -- 172.21.15.57:0/4131388111 shutdown_connections 2023-10-19T13:00:50.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.352+0000 7f32de105640 1 --2- 172.21.15.57:0/4131388111 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f32c406af60 0x7f32c406d420 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:50.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.352+0000 7f32de105640 1 --2- 172.21.15.57:0/4131388111 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f32d80c7df0 0x7f32d80c8270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:50.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.352+0000 7f32de105640 1 --2- 172.21.15.57:0/4131388111 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f32d80ca090 0x7f32d80ca510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:50.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.352+0000 7f32de105640 1 --2- 172.21.15.57:0/4131388111 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f32c0004d00 0x7f32d814efe0 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:50.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.352+0000 7f32de105640 1 -- 172.21.15.57:0/4131388111 >> 172.21.15.57:0/4131388111 conn(0x7f32d809a030 msgr2=0x7f32d815a040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:50.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.352+0000 7f32de105640 1 -- 172.21.15.57:0/4131388111 shutdown_connections 2023-10-19T13:00:50.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.352+0000 7f32de105640 1 -- 172.21.15.57:0/4131388111 wait complete. 2023-10-19T13:00:50.365 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-19T13:00:50.365 DEBUG:teuthology.orchestra.run.smithi057:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-19T13:00:50.416 DEBUG:teuthology.orchestra.run.smithi179:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-19T13:00:50.467 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-19T13:00:50.468 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T13:00:50.468 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T13:00:50.483 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055614,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=358 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248260k,nr_inodes=812065,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-19T13:00:50.484 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-19T13:00:50.484 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T13:00:50.501 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054833,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=284 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3247632k,nr_inodes=811908,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-19T13:00:50.502 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T13:00:50.502 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T13:00:50.539 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055614,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=358 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248260k,nr_inodes=812065,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-19T13:00:50.539 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-19T13:00:50.539 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T13:00:50.558 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054833,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=284 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3247632k,nr_inodes=811908,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-19T13:00:50.559 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-19T13:00:50.559 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T13:00:50.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.684+0000 7faa3bcc2640 1 -- 172.21.15.57:0/1800979620 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa340cb980 msgr2=0x7faa3413db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:50.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.684+0000 7faa3bcc2640 1 --2- 172.21.15.57:0/1800979620 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa340cb980 0x7faa3413db80 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7faa30009830 tx=0x7faa3002bd60 comp rx=0 tx=0).stop 2023-10-19T13:00:50.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.684+0000 7faa3bcc2640 1 -- 172.21.15.57:0/1800979620 shutdown_connections 2023-10-19T13:00:50.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.684+0000 7faa3bcc2640 1 --2- 172.21.15.57:0/1800979620 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa340cb980 0x7faa3413db80 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:50.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.684+0000 7faa3bcc2640 1 -- 172.21.15.57:0/1800979620 >> 172.21.15.57:0/1800979620 conn(0x7faa34069930 msgr2=0x7faa34069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:50.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.685+0000 7faa3bcc2640 1 -- 172.21.15.57:0/1800979620 shutdown_connections 2023-10-19T13:00:50.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.685+0000 7faa3bcc2640 1 -- 172.21.15.57:0/1800979620 wait complete. 2023-10-19T13:00:50.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.686+0000 7faa3bcc2640 1 Processor -- start 2023-10-19T13:00:50.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.686+0000 7faa3bcc2640 1 -- start start 2023-10-19T13:00:50.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.686+0000 7faa3bcc2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa34070350 0x7faa341f4870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:50.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.686+0000 7faa3bcc2640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faa340cb980 0x7faa3413fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:50.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.686+0000 7faa3bcc2640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7faa340acfa0 0x7faa341401e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:50.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.686+0000 7faa3bcc2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa340ad460 con 0x7faa34070350 2023-10-19T13:00:50.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.686+0000 7faa3bcc2640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa340ad740 con 0x7faa340cb980 2023-10-19T13:00:50.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.686+0000 7faa3bcc2640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa340ada20 con 0x7faa340acfa0 2023-10-19T13:00:50.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.686+0000 7faa39a5e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa34070350 0x7faa341f4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:50.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.686+0000 7faa39a5e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa34070350 0x7faa341f4870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51446/0 (socket says 172.21.15.57:51446) 2023-10-19T13:00:50.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.686+0000 7faa39a5e640 1 -- 172.21.15.57:0/1540784802 learned_addr learned my addr 172.21.15.57:0/1540784802 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:50.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.687+0000 7faa3a25f640 1 --2- 172.21.15.57:0/1540784802 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7faa340acfa0 0x7faa341401e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:50.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.687+0000 7faa3925d640 1 --2- 172.21.15.57:0/1540784802 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faa340cb980 0x7faa3413fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:50.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.687+0000 7faa39a5e640 1 -- 172.21.15.57:0/1540784802 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7faa340acfa0 msgr2=0x7faa341401e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:50.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.687+0000 7faa39a5e640 1 --2- 172.21.15.57:0/1540784802 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7faa340acfa0 0x7faa341401e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:50.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.687+0000 7faa39a5e640 1 -- 172.21.15.57:0/1540784802 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faa340cb980 msgr2=0x7faa3413fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:50.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.687+0000 7faa39a5e640 1 --2- 172.21.15.57:0/1540784802 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faa340cb980 0x7faa3413fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:50.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.687+0000 7faa39a5e640 1 -- 172.21.15.57:0/1540784802 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa30009490 con 0x7faa34070350 2023-10-19T13:00:50.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.687+0000 7faa3a25f640 1 --2- 172.21.15.57:0/1540784802 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7faa340acfa0 0x7faa341401e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:00:50.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.687+0000 7faa3925d640 1 --2- 172.21.15.57:0/1540784802 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faa340cb980 0x7faa3413fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:00:50.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.687+0000 7faa39a5e640 1 --2- 172.21.15.57:0/1540784802 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa34070350 0x7faa341f4870 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7faa30009780 tx=0x7faa300038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:50.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.687+0000 7faa2a7fc640 1 -- 172.21.15.57:0/1540784802 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa30003ea0 con 0x7faa34070350 2023-10-19T13:00:50.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.687+0000 7faa2a7fc640 1 -- 172.21.15.57:0/1540784802 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7faa30034550 con 0x7faa34070350 2023-10-19T13:00:50.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.687+0000 7faa3bcc2640 1 -- 172.21.15.57:0/1540784802 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa340add00 con 0x7faa34070350 2023-10-19T13:00:50.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.687+0000 7faa2a7fc640 1 -- 172.21.15.57:0/1540784802 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa30034960 con 0x7faa34070350 2023-10-19T13:00:50.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.687+0000 7faa3bcc2640 1 -- 172.21.15.57:0/1540784802 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa34140f00 con 0x7faa34070350 2023-10-19T13:00:50.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.688+0000 7faa3bcc2640 1 -- 172.21.15.57:0/1540784802 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa3415b610 con 0x7faa34070350 2023-10-19T13:00:50.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.689+0000 7faa2a7fc640 1 -- 172.21.15.57:0/1540784802 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 40) v1 ==== 89331+0+0 (secure 0 0 0) 0x7faa3003b040 con 0x7faa34070350 2023-10-19T13:00:50.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.689+0000 7faa2a7fc640 1 --2- 172.21.15.57:0/1540784802 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7faa1806af60 0x7faa1806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:50.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.689+0000 7faa2a7fc640 1 -- 172.21.15.57:0/1540784802 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5979+0+0 (secure 0 0 0) 0x7faa300abe90 con 0x7faa34070350 2023-10-19T13:00:50.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.689+0000 7faa3925d640 1 --2- 172.21.15.57:0/1540784802 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7faa1806af60 0x7faa1806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:50.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.690+0000 7faa3925d640 1 --2- 172.21.15.57:0/1540784802 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7faa1806af60 0x7faa1806d420 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7faa34157d50 tx=0x7faa24009040 comp rx=0 tx=0).ready entity=mgr.6679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:50.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.693+0000 7faa2a7fc640 1 -- 172.21.15.57:0/1540784802 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7faa30076db0 con 0x7faa34070350 2023-10-19T13:00:50.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.908+0000 7faa3bcc2640 1 -- 172.21.15.57:0/1540784802 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faa340ac270 con 0x7faa34070350 2023-10-19T13:00:50.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.908+0000 7faa2a7fc640 1 -- 172.21.15.57:0/1540784802 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+3702 (secure 0 0 0) 0x7faa30039090 con 0x7faa34070350 2023-10-19T13:00:50.911 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:00:50.912 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":32,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5966,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/765319531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":765319531},{"type":"v1","addr":"172.21.15.179:6837","nonce":765319531}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27},{"gid":5969,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/1388526553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":1388526553},{"type":"v1","addr":"172.21.15.179:6836","nonce":1388526553}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27},{"gid":7035,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/3237434259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":3237434259},{"type":"v1","addr":"172.21.15.57:6834","nonce":3237434259}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27},{"gid":7458,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/3603007686","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":3603007686},{"type":"v1","addr":"172.21.15.57:6835","nonce":3603007686}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":32}],"filesystems":[]} 2023-10-19T13:00:50.912 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 32 2023-10-19T13:00:50.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.910+0000 7faa3bcc2640 1 -- 172.21.15.57:0/1540784802 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7faa1806af60 msgr2=0x7faa1806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:50.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.910+0000 7faa3bcc2640 1 --2- 172.21.15.57:0/1540784802 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7faa1806af60 0x7faa1806d420 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7faa34157d50 tx=0x7faa24009040 comp rx=0 tx=0).stop 2023-10-19T13:00:50.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.910+0000 7faa3bcc2640 1 -- 172.21.15.57:0/1540784802 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa34070350 msgr2=0x7faa341f4870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:50.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.910+0000 7faa3bcc2640 1 --2- 172.21.15.57:0/1540784802 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa34070350 0x7faa341f4870 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7faa30009780 tx=0x7faa300038a0 comp rx=0 tx=0).stop 2023-10-19T13:00:50.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.910+0000 7faa3bcc2640 1 -- 172.21.15.57:0/1540784802 shutdown_connections 2023-10-19T13:00:50.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.910+0000 7faa3bcc2640 1 --2- 172.21.15.57:0/1540784802 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7faa1806af60 0x7faa1806d420 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:50.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.910+0000 7faa3bcc2640 1 --2- 172.21.15.57:0/1540784802 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7faa340acfa0 0x7faa341401e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:50.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.910+0000 7faa3bcc2640 1 --2- 172.21.15.57:0/1540784802 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faa340cb980 0x7faa3413fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:50.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.910+0000 7faa3bcc2640 1 --2- 172.21.15.57:0/1540784802 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa34070350 0x7faa341f4870 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:50.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.910+0000 7faa3bcc2640 1 -- 172.21.15.57:0/1540784802 >> 172.21.15.57:0/1540784802 conn(0x7faa34069930 msgr2=0x7faa340ca870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:50.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.910+0000 7faa3bcc2640 1 -- 172.21.15.57:0/1540784802 shutdown_connections 2023-10-19T13:00:50.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:50.910+0000 7faa3bcc2640 1 -- 172.21.15.57:0/1540784802 wait complete. 2023-10-19T13:00:50.924 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-19T13:00:50.924 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-19T13:00:50.924 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-19T13:00:50.925 INFO:teuthology.orchestra.run:waiting for 300 2023-10-19T13:00:50.925 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-19T13:00:50.925 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-19T13:00:50.926 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-19T13:00:50.926 INFO:teuthology.orchestra.run:waiting for 300 2023-10-19T13:00:50.926 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-19T13:00:50.926 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-19T13:00:50.927 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-19T13:00:50.927 INFO:teuthology.orchestra.run:waiting for 300 2023-10-19T13:00:50.927 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-19T13:00:50.928 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-19T13:00:50.928 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-19T13:00:50.928 INFO:teuthology.orchestra.run:waiting for 300 2023-10-19T13:00:50.933 INFO:tasks.ceph.mds.a:Stopped 2023-10-19T13:00:50.934 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-19T13:00:50.935 INFO:tasks.ceph.mds.c:Stopped 2023-10-19T13:00:50.936 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-19T13:00:50.941 INFO:tasks.ceph.mds.b:Stopped 2023-10-19T13:00:50.941 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-19T13:00:50.942 INFO:tasks.ceph.mds.d:Stopped 2023-10-19T13:00:50.943 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-19T13:00:50.977 INFO:tasks.ceph.mds.a:Started 2023-10-19T13:00:50.978 DEBUG:teuthology.parallel:result is None 2023-10-19T13:00:50.979 INFO:tasks.ceph.mds.c:Started 2023-10-19T13:00:50.979 DEBUG:teuthology.parallel:result is None 2023-10-19T13:00:50.984 INFO:tasks.ceph.mds.b:Started 2023-10-19T13:00:50.984 DEBUG:teuthology.parallel:result is None 2023-10-19T13:00:50.985 INFO:tasks.ceph.mds.d:Started 2023-10-19T13:00:50.985 DEBUG:teuthology.parallel:result is None 2023-10-19T13:00:50.986 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-19T13:00:51.058 INFO:tasks.ceph.mds.a.smithi057.stdout:starting mds.a at 2023-10-19T13:00:51.059 INFO:tasks.ceph.mds.c.smithi057.stdout:starting mds.c at 2023-10-19T13:00:51.069 INFO:tasks.ceph.mds.d.smithi179.stdout:starting mds.d at 2023-10-19T13:00:51.069 INFO:tasks.ceph.mds.b.smithi179.stdout:starting mds.b at 2023-10-19T13:00:51.070 INFO:tasks.ceph.mds.d.smithi179.stderr:2023-10-19T13:00:51.067+0000 7f25128bd640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6834/0: (98) Address already in use 2023-10-19T13:00:51.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.097+0000 7f5ae23d0640 1 -- 172.21.15.57:0/4198672232 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5adc0ceb40 msgr2=0x7f5adc0b0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:51.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.097+0000 7f5ae23d0640 1 --2- 172.21.15.57:0/4198672232 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5adc0ceb40 0x7f5adc0b0650 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f5ad4005fd0 tx=0x7f5ad402c050 comp rx=0 tx=0).stop 2023-10-19T13:00:51.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.097+0000 7f5ae23d0640 1 -- 172.21.15.57:0/4198672232 shutdown_connections 2023-10-19T13:00:51.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.097+0000 7f5ae23d0640 1 --2- 172.21.15.57:0/4198672232 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5adc0ceb40 0x7f5adc0b0650 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:51.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.097+0000 7f5ae23d0640 1 --2- 172.21.15.57:0/4198672232 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5adc0cf340 0x7f5adc0d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:51.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.097+0000 7f5ae23d0640 1 -- 172.21.15.57:0/4198672232 >> 172.21.15.57:0/4198672232 conn(0x7f5adc0d0d60 msgr2=0x7f5adc0d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:51.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.097+0000 7f5ae23d0640 1 -- 172.21.15.57:0/4198672232 shutdown_connections 2023-10-19T13:00:51.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.097+0000 7f5ae23d0640 1 -- 172.21.15.57:0/4198672232 wait complete. 2023-10-19T13:00:51.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.098+0000 7f5ae23d0640 1 Processor -- start 2023-10-19T13:00:51.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.098+0000 7f5ae23d0640 1 -- start start 2023-10-19T13:00:51.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.099+0000 7f5ae23d0640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5adc0ceb40 0x7f5adc1f5960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:51.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.099+0000 7f5ae23d0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5adc0cf340 0x7f5adc0b4080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:51.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.099+0000 7f5ae23d0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5adc15b2a0 0x7f5adc1582e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:51.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.099+0000 7f5ae23d0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5adc1f5ea0 con 0x7f5adc0cf340 2023-10-19T13:00:51.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.099+0000 7f5ae23d0640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5adc15d7b0 con 0x7f5adc15b2a0 2023-10-19T13:00:51.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.099+0000 7f5ae23d0640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5adc0b45c0 con 0x7f5adc0ceb40 2023-10-19T13:00:51.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.099+0000 7f5adb7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5adc0cf340 0x7f5adc0b4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:51.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.099+0000 7f5adb7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5adc0cf340 0x7f5adc0b4080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51520/0 (socket says 172.21.15.57:51520) 2023-10-19T13:00:51.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.099+0000 7f5adb7fe640 1 -- 172.21.15.57:0/826530670 learned_addr learned my addr 172.21.15.57:0/826530670 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:51.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.099+0000 7f5adbfff640 1 --2- 172.21.15.57:0/826530670 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5adc0ceb40 0x7f5adc1f5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:51.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.099+0000 7f5ae096d640 1 --2- 172.21.15.57:0/826530670 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5adc15b2a0 0x7f5adc1582e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:51.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.099+0000 7f5adb7fe640 1 -- 172.21.15.57:0/826530670 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5adc0ceb40 msgr2=0x7f5adc1f5960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:51.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.099+0000 7f5adb7fe640 1 --2- 172.21.15.57:0/826530670 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5adc0ceb40 0x7f5adc1f5960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:51.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.099+0000 7f5adb7fe640 1 -- 172.21.15.57:0/826530670 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5adc15b2a0 msgr2=0x7f5adc1582e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:51.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.099+0000 7f5adb7fe640 1 --2- 172.21.15.57:0/826530670 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5adc15b2a0 0x7f5adc1582e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:51.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.099+0000 7f5adb7fe640 1 -- 172.21.15.57:0/826530670 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ad4009840 con 0x7f5adc0cf340 2023-10-19T13:00:51.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.099+0000 7f5ae096d640 1 --2- 172.21.15.57:0/826530670 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5adc15b2a0 0x7f5adc1582e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:00:51.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.100+0000 7f5adb7fe640 1 --2- 172.21.15.57:0/826530670 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5adc0cf340 0x7f5adc0b4080 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f5ad000c440 tx=0x7f5ad000c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:51.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.100+0000 7f5ad8ff9640 1 -- 172.21.15.57:0/826530670 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ad0004230 con 0x7f5adc0cf340 2023-10-19T13:00:51.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.100+0000 7f5ad8ff9640 1 -- 172.21.15.57:0/826530670 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f5ad0004480 con 0x7f5adc0cf340 2023-10-19T13:00:51.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.100+0000 7f5ae23d0640 1 -- 172.21.15.57:0/826530670 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5adc15cc30 con 0x7f5adc0cf340 2023-10-19T13:00:51.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.100+0000 7f5ad8ff9640 1 -- 172.21.15.57:0/826530670 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ad0003850 con 0x7f5adc0cf340 2023-10-19T13:00:51.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.100+0000 7f5ae23d0640 1 -- 172.21.15.57:0/826530670 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5adc15d1f0 con 0x7f5adc0cf340 2023-10-19T13:00:51.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.101+0000 7f5ae23d0640 1 -- 172.21.15.57:0/826530670 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5adc0c9610 con 0x7f5adc0cf340 2023-10-19T13:00:51.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.101+0000 7f5ad8ff9640 1 -- 172.21.15.57:0/826530670 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 40) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f5ad0003a70 con 0x7f5adc0cf340 2023-10-19T13:00:51.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.102+0000 7f5ad8ff9640 1 --2- 172.21.15.57:0/826530670 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f5ac806af60 0x7f5ac806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:51.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.102+0000 7f5ad8ff9640 1 -- 172.21.15.57:0/826530670 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5979+0+0 (secure 0 0 0) 0x7f5ad0088fe0 con 0x7f5adc0cf340 2023-10-19T13:00:51.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.102+0000 7f5adbfff640 1 --2- 172.21.15.57:0/826530670 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f5ac806af60 0x7f5ac806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:51.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.102+0000 7f5adbfff640 1 --2- 172.21.15.57:0/826530670 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f5ac806af60 0x7f5ac806d420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5acc005e10 tx=0x7f5acc005da0 comp rx=0 tx=0).ready entity=mgr.6679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:51.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.106+0000 7f5ad8ff9640 1 -- 172.21.15.57:0/826530670 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f5adc0c9610 con 0x7f5adc0cf340 2023-10-19T13:00:51.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.293+0000 7f5ae23d0640 1 -- 172.21.15.57:0/826530670 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f5adc0cb9b0 con 0x7f5adc0cf340 2023-10-19T13:00:51.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.764+0000 7f5ad8ff9640 1 -- 172.21.15.57:0/826530670 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v94) v1 ==== 97+0+0 (secure 0 0 0) 0x7f5ad0017020 con 0x7f5adc0cf340 2023-10-19T13:00:51.768 INFO:teuthology.orchestra.run.smithi057.stderr: removed all blocklist entries 2023-10-19T13:00:51.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.767+0000 7f5ae23d0640 1 -- 172.21.15.57:0/826530670 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f5ac806af60 msgr2=0x7f5ac806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:51.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.767+0000 7f5ae23d0640 1 --2- 172.21.15.57:0/826530670 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f5ac806af60 0x7f5ac806d420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5acc005e10 tx=0x7f5acc005da0 comp rx=0 tx=0).stop 2023-10-19T13:00:51.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.767+0000 7f5ae23d0640 1 -- 172.21.15.57:0/826530670 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5adc0cf340 msgr2=0x7f5adc0b4080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:51.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.767+0000 7f5ae23d0640 1 --2- 172.21.15.57:0/826530670 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5adc0cf340 0x7f5adc0b4080 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f5ad000c440 tx=0x7f5ad000c900 comp rx=0 tx=0).stop 2023-10-19T13:00:51.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.767+0000 7f5ae23d0640 1 -- 172.21.15.57:0/826530670 shutdown_connections 2023-10-19T13:00:51.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.767+0000 7f5ae23d0640 1 --2- 172.21.15.57:0/826530670 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f5ac806af60 0x7f5ac806d420 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:51.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.767+0000 7f5ae23d0640 1 --2- 172.21.15.57:0/826530670 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5adc15b2a0 0x7f5adc1582e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:51.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.767+0000 7f5ae23d0640 1 --2- 172.21.15.57:0/826530670 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5adc0cf340 0x7f5adc0b4080 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:51.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.767+0000 7f5ae23d0640 1 --2- 172.21.15.57:0/826530670 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5adc0ceb40 0x7f5adc1f5960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:51.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.767+0000 7f5ae23d0640 1 -- 172.21.15.57:0/826530670 >> 172.21.15.57:0/826530670 conn(0x7f5adc0d0d60 msgr2=0x7f5adc0cf720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:51.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.767+0000 7f5ae23d0640 1 -- 172.21.15.57:0/826530670 shutdown_connections 2023-10-19T13:00:51.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.767+0000 7f5ae23d0640 1 -- 172.21.15.57:0/826530670 wait complete. 2023-10-19T13:00:51.784 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-19T13:00:51.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.893+0000 7f13acdd0640 1 -- 172.21.15.57:0/837162815 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f13a8141690 msgr2=0x7f13a80703e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:51.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.893+0000 7f13acdd0640 1 --2- 172.21.15.57:0/837162815 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f13a8141690 0x7f13a80703e0 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f1390009830 tx=0x7f139002bcd0 comp rx=0 tx=0).stop 2023-10-19T13:00:51.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.893+0000 7f13acdd0640 1 -- 172.21.15.57:0/837162815 shutdown_connections 2023-10-19T13:00:51.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.893+0000 7f13acdd0640 1 --2- 172.21.15.57:0/837162815 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f13a8141690 0x7f13a80703e0 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:51.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.893+0000 7f13acdd0640 1 -- 172.21.15.57:0/837162815 >> 172.21.15.57:0/837162815 conn(0x7f13a8069930 msgr2=0x7f13a8069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:51.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.893+0000 7f13acdd0640 1 -- 172.21.15.57:0/837162815 shutdown_connections 2023-10-19T13:00:51.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.893+0000 7f13acdd0640 1 -- 172.21.15.57:0/837162815 wait complete. 2023-10-19T13:00:51.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.895+0000 7f13acdd0640 1 Processor -- start 2023-10-19T13:00:51.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.895+0000 7f13acdd0640 1 -- start start 2023-10-19T13:00:51.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.895+0000 7f13acdd0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f13a8070920 0x7f13a81e1b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:51.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.895+0000 7f13acdd0640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f13a813e520 0x7f13a80a86c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:51.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.895+0000 7f13acdd0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f13a8141690 0x7f13a81e3c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:51.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.895+0000 7f13acdd0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13a80a9e00 con 0x7f13a8070920 2023-10-19T13:00:51.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.895+0000 7f13acdd0640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13a80a9180 con 0x7f13a8141690 2023-10-19T13:00:51.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.895+0000 7f13acdd0640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13a80a9460 con 0x7f13a813e520 2023-10-19T13:00:51.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.895+0000 7f13a659c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f13a8070920 0x7f13a81e1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:51.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.896+0000 7f13a659c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f13a8070920 0x7f13a81e1b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51534/0 (socket says 172.21.15.57:51534) 2023-10-19T13:00:51.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.896+0000 7f13a659c640 1 -- 172.21.15.57:0/2761171802 learned_addr learned my addr 172.21.15.57:0/2761171802 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:51.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.896+0000 7f13a659c640 1 -- 172.21.15.57:0/2761171802 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f13a813e520 msgr2=0x7f13a80a86c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:51.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.896+0000 7f13a659c640 1 --2- 172.21.15.57:0/2761171802 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f13a813e520 0x7f13a80a86c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:51.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.896+0000 7f13a659c640 1 -- 172.21.15.57:0/2761171802 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f13a8141690 msgr2=0x7f13a81e3c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:51.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.896+0000 7f13a659c640 1 --2- 172.21.15.57:0/2761171802 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f13a8141690 0x7f13a81e3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:51.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.896+0000 7f13a659c640 1 -- 172.21.15.57:0/2761171802 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1390009490 con 0x7f13a8070920 2023-10-19T13:00:51.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.896+0000 7f13a659c640 1 --2- 172.21.15.57:0/2761171802 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f13a8070920 0x7f13a81e1b00 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f139800c410 tx=0x7f139800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:51.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.896+0000 7f139effd640 1 -- 172.21.15.57:0/2761171802 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13980035f0 con 0x7f13a8070920 2023-10-19T13:00:51.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.896+0000 7f139effd640 1 -- 172.21.15.57:0/2761171802 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f1398003750 con 0x7f13a8070920 2023-10-19T13:00:51.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.896+0000 7f13acdd0640 1 -- 172.21.15.57:0/2761171802 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13a80a9740 con 0x7f13a8070920 2023-10-19T13:00:51.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.896+0000 7f139effd640 1 -- 172.21.15.57:0/2761171802 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1398003ba0 con 0x7f13a8070920 2023-10-19T13:00:51.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.896+0000 7f13acdd0640 1 -- 172.21.15.57:0/2761171802 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13a81e2590 con 0x7f13a8070920 2023-10-19T13:00:51.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.897+0000 7f13acdd0640 1 -- 172.21.15.57:0/2761171802 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13a80a9f40 con 0x7f13a8070920 2023-10-19T13:00:51.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.898+0000 7f139effd640 1 -- 172.21.15.57:0/2761171802 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 40) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f1398003d00 con 0x7f13a8070920 2023-10-19T13:00:51.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.898+0000 7f139effd640 1 --2- 172.21.15.57:0/2761171802 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f138806af60 0x7f138806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:51.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.898+0000 7f139effd640 1 -- 172.21.15.57:0/2761171802 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f1398088d70 con 0x7f13a8070920 2023-10-19T13:00:51.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.898+0000 7f139ffff640 1 --2- 172.21.15.57:0/2761171802 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f138806af60 0x7f138806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:51.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.899+0000 7f139ffff640 1 --2- 172.21.15.57:0/2761171802 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f138806af60 0x7f138806d420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f1390036000 tx=0x7f13900046c0 comp rx=0 tx=0).ready entity=mgr.6679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:51.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:51.902+0000 7f139effd640 1 -- 172.21.15.57:0/2761171802 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f1398053f90 con 0x7f13a8070920 2023-10-19T13:00:52.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.143+0000 7f13acdd0640 1 -- 172.21.15.57:0/2761171802 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f13a80690e0 con 0x7f13a8070920 2023-10-19T13:00:52.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.143+0000 7f139effd640 1 -- 172.21.15.57:0/2761171802 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v24) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f1398017070 con 0x7f13a8070920 2023-10-19T13:00:52.147 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:00:52.147 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2023-10-19T13:00:52.147 INFO:teuthology.orchestra.run.smithi057.stdout: "auth_dump": [ 2023-10-19T13:00:52.148 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:00:52.148 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "mds.a", 2023-10-19T13:00:52.148 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBjJDFlmbVANhAAP/ynHlU/OxaRq5lQoAa8Ng==", 2023-10-19T13:00:52.148 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:00:52.148 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T13:00:52.149 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow *", 2023-10-19T13:00:52.149 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow *", 2023-10-19T13:00:52.149 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:00:52.149 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:00:52.149 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:00:52.150 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:00:52.150 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "mds.b", 2023-10-19T13:00:52.150 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFl9l7RBxAAC0geI2tJ1SIZZr1++kS/BA==", 2023-10-19T13:00:52.150 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:00:52.150 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T13:00:52.151 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow *", 2023-10-19T13:00:52.151 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow *", 2023-10-19T13:00:52.151 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:00:52.151 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:00:52.151 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:00:52.152 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:00:52.152 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "mds.c", 2023-10-19T13:00:52.152 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFlFkaoAhAA/PBizkJ4bm958oP4CmPuIQ==", 2023-10-19T13:00:52.152 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:00:52.152 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T13:00:52.153 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow *", 2023-10-19T13:00:52.153 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow *", 2023-10-19T13:00:52.153 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:00:52.153 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:00:52.153 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:00:52.154 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:00:52.154 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "mds.d", 2023-10-19T13:00:52.154 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFlx8skDxAAAfCHTYtYJgJik0WHSWeQfQ==", 2023-10-19T13:00:52.154 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:00:52.154 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T13:00:52.155 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow *", 2023-10-19T13:00:52.155 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow *", 2023-10-19T13:00:52.155 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:00:52.155 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:00:52.155 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:00:52.156 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:00:52.156 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.0", 2023-10-19T13:00:52.156 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBrJDFlCo9sBxAAQN2cID4nCQgRwGaGFEV8LQ==", 2023-10-19T13:00:52.156 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:00:52.156 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T13:00:52.156 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T13:00:52.157 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:00:52.157 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:00:52.157 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:00:52.157 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:00:52.157 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.1", 2023-10-19T13:00:52.158 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBuJDFl4fsLCBAAp3Pi80yZtc5sju0GisEuVA==", 2023-10-19T13:00:52.158 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:00:52.158 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T13:00:52.158 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T13:00:52.158 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:00:52.159 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:00:52.159 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:00:52.159 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:00:52.159 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.2", 2023-10-19T13:00:52.159 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBxJDFl+KAKCBAAd4N9cluARxPVfBLU+7Hd7A==", 2023-10-19T13:00:52.160 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:00:52.160 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T13:00:52.160 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T13:00:52.160 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:00:52.160 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:00:52.161 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:00:52.161 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:00:52.161 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.3", 2023-10-19T13:00:52.161 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQB0JDFlU/toCBAAP0Ou1zcHKNx5SfbybJkh4Q==", 2023-10-19T13:00:52.161 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:00:52.162 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T13:00:52.162 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T13:00:52.162 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:00:52.162 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:00:52.162 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:00:52.163 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:00:52.163 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.4", 2023-10-19T13:00:52.163 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQB6JDFlc6stEBAALkgNyxoiCTzZnKpBSPxI6w==", 2023-10-19T13:00:52.163 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:00:52.163 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T13:00:52.164 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T13:00:52.164 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:00:52.164 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:00:52.164 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:00:52.164 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:00:52.164 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.5", 2023-10-19T13:00:52.165 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQB9JDFlI3cNDhAAuG3tHJ6RvPwAzPE76wTIVQ==", 2023-10-19T13:00:52.165 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:00:52.165 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T13:00:52.165 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T13:00:52.165 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:00:52.166 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:00:52.166 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:00:52.166 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:00:52.166 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.6", 2023-10-19T13:00:52.166 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQCAJDFlLPumCxAA/iy08t/2N9rXfn71KtbSQw==", 2023-10-19T13:00:52.167 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:00:52.167 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T13:00:52.167 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T13:00:52.167 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:00:52.167 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:00:52.168 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:00:52.168 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:00:52.168 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.7", 2023-10-19T13:00:52.168 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQCDJDFlHmrbCRAA+hpWygGIJydfU6cHWlfx/Q==", 2023-10-19T13:00:52.168 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:00:52.169 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T13:00:52.169 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T13:00:52.169 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:00:52.169 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:00:52.169 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:00:52.169 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:00:52.170 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "client.0", 2023-10-19T13:00:52.170 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFlg9yuExAATxiWDfS+p07G1F0pORbJEg==", 2023-10-19T13:00:52.170 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:00:52.170 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T13:00:52.171 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow r", 2023-10-19T13:00:52.171 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-19T13:00:52.171 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:00:52.171 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:00:52.171 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:00:52.171 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "client.1", 2023-10-19T13:00:52.172 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFl9IwYGxAAHPituiJSPkHDuD0QyG4Hww==", 2023-10-19T13:00:52.172 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:00:52.172 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T13:00:52.172 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow r", 2023-10-19T13:00:52.172 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow rw", 2023-10-19T13:00:52.173 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow rwx" 2023-10-19T13:00:52.173 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:00:52.173 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:00:52.173 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:00:52.173 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "client.2", 2023-10-19T13:00:52.173 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFlMd+AFxAAS/UTESDt76GmrhhN2o/Fdw==", 2023-10-19T13:00:52.174 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:00:52.174 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T13:00:52.174 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow r", 2023-10-19T13:00:52.174 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow rw", 2023-10-19T13:00:52.174 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow rwx" 2023-10-19T13:00:52.175 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:00:52.175 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:00:52.175 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:00:52.175 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "client.3", 2023-10-19T13:00:52.175 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFlHZ3fIBAA42B7n8opLS5smXIwJyVYzw==", 2023-10-19T13:00:52.176 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:00:52.176 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T13:00:52.176 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow r", 2023-10-19T13:00:52.176 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow rw", 2023-10-19T13:00:52.176 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow rwx" 2023-10-19T13:00:52.177 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:00:52.177 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:00:52.177 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:00:52.177 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "client.admin", 2023-10-19T13:00:52.177 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBjJDFlBgasFxAAMA24j8KszcXMEpYRQDXj4g==", 2023-10-19T13:00:52.178 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:00:52.178 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow *", 2023-10-19T13:00:52.178 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow *", 2023-10-19T13:00:52.178 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow *", 2023-10-19T13:00:52.178 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:00:52.179 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:00:52.179 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:00:52.179 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:00:52.179 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "mgr.x", 2023-10-19T13:00:52.179 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBjJDFllFBfMhAA5+KPXWmYZqcESmLObaUbBA==", 2023-10-19T13:00:52.179 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:00:52.180 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow *", 2023-10-19T13:00:52.180 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile mgr", 2023-10-19T13:00:52.180 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:00:52.180 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:00:52.180 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:00:52.180 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:00:52.181 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "mgr.y", 2023-10-19T13:00:52.181 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBjJDFlxRzJLRAAGjG58o7qXAHvIEIRYwl6QA==", 2023-10-19T13:00:52.181 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:00:52.181 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow *", 2023-10-19T13:00:52.181 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile mgr", 2023-10-19T13:00:52.182 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:00:52.182 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:00:52.182 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:00:52.182 INFO:teuthology.orchestra.run.smithi057.stdout: ] 2023-10-19T13:00:52.182 INFO:teuthology.orchestra.run.smithi057.stdout:} 2023-10-19T13:00:52.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.145+0000 7f13acdd0640 1 -- 172.21.15.57:0/2761171802 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f138806af60 msgr2=0x7f138806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:52.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.145+0000 7f13acdd0640 1 --2- 172.21.15.57:0/2761171802 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f138806af60 0x7f138806d420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f1390036000 tx=0x7f13900046c0 comp rx=0 tx=0).stop 2023-10-19T13:00:52.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.145+0000 7f13acdd0640 1 -- 172.21.15.57:0/2761171802 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f13a8070920 msgr2=0x7f13a81e1b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:52.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.145+0000 7f13acdd0640 1 --2- 172.21.15.57:0/2761171802 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f13a8070920 0x7f13a81e1b00 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f139800c410 tx=0x7f139800c8d0 comp rx=0 tx=0).stop 2023-10-19T13:00:52.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.145+0000 7f13acdd0640 1 -- 172.21.15.57:0/2761171802 shutdown_connections 2023-10-19T13:00:52.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.145+0000 7f13acdd0640 1 --2- 172.21.15.57:0/2761171802 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f138806af60 0x7f138806d420 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:52.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.145+0000 7f13acdd0640 1 --2- 172.21.15.57:0/2761171802 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f13a8141690 0x7f13a81e3c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:52.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.145+0000 7f13acdd0640 1 --2- 172.21.15.57:0/2761171802 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f13a813e520 0x7f13a80a86c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:52.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.145+0000 7f13acdd0640 1 --2- 172.21.15.57:0/2761171802 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f13a8070920 0x7f13a81e1b00 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:52.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.145+0000 7f13acdd0640 1 -- 172.21.15.57:0/2761171802 >> 172.21.15.57:0/2761171802 conn(0x7f13a8069930 msgr2=0x7f13a806bb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:52.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.145+0000 7f13acdd0640 1 -- 172.21.15.57:0/2761171802 shutdown_connections 2023-10-19T13:00:52.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.145+0000 7f13acdd0640 1 -- 172.21.15.57:0/2761171802 wait complete. 2023-10-19T13:00:52.187 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T13:00:52.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.293+0000 7efd034d7640 1 -- 172.21.15.57:0/3126365353 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efcfc0b0c40 msgr2=0x7efcfc158300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:52.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.293+0000 7efd034d7640 1 --2- 172.21.15.57:0/3126365353 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efcfc0b0c40 0x7efcfc158300 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7efcf8005fd0 tx=0x7efcf802d4a0 comp rx=0 tx=0).stop 2023-10-19T13:00:52.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.293+0000 7efd034d7640 1 -- 172.21.15.57:0/3126365353 shutdown_connections 2023-10-19T13:00:52.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.293+0000 7efd034d7640 1 --2- 172.21.15.57:0/3126365353 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efcfc0b0c40 0x7efcfc158300 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:52.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.293+0000 7efd034d7640 1 -- 172.21.15.57:0/3126365353 >> 172.21.15.57:0/3126365353 conn(0x7efcfc069930 msgr2=0x7efcfc069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:52.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.293+0000 7efd034d7640 1 -- 172.21.15.57:0/3126365353 shutdown_connections 2023-10-19T13:00:52.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.293+0000 7efd034d7640 1 -- 172.21.15.57:0/3126365353 wait complete. 2023-10-19T13:00:52.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.295+0000 7efd034d7640 1 Processor -- start 2023-10-19T13:00:52.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.295+0000 7efd034d7640 1 -- start start 2023-10-19T13:00:52.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.295+0000 7efd034d7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efcfc0afa20 0x7efcfc13e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:52.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.295+0000 7efd034d7640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7efcfc0b0c40 0x7efcfc1e50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:52.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.295+0000 7efd034d7640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efcfc0cbd20 0x7efcfc1e55e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:52.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.295+0000 7efd034d7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcfc15aa50 con 0x7efcfc0afa20 2023-10-19T13:00:52.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.295+0000 7efd034d7640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcfc13e940 con 0x7efcfc0cbd20 2023-10-19T13:00:52.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.295+0000 7efd034d7640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcfc149e70 con 0x7efcfc0b0c40 2023-10-19T13:00:52.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.295+0000 7efd01273640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efcfc0afa20 0x7efcfc13e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:52.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.295+0000 7efd01273640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efcfc0afa20 0x7efcfc13e400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51556/0 (socket says 172.21.15.57:51556) 2023-10-19T13:00:52.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.295+0000 7efd01273640 1 -- 172.21.15.57:0/2721769470 learned_addr learned my addr 172.21.15.57:0/2721769470 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:52.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.296+0000 7efd01a74640 1 --2- 172.21.15.57:0/2721769470 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efcfc0cbd20 0x7efcfc1e55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:52.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.296+0000 7efd00a72640 1 --2- 172.21.15.57:0/2721769470 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7efcfc0b0c40 0x7efcfc1e50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:52.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.296+0000 7efd01273640 1 -- 172.21.15.57:0/2721769470 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7efcfc0b0c40 msgr2=0x7efcfc1e50a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:52.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.296+0000 7efd01273640 1 --2- 172.21.15.57:0/2721769470 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7efcfc0b0c40 0x7efcfc1e50a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:52.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.296+0000 7efd01273640 1 -- 172.21.15.57:0/2721769470 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efcfc0cbd20 msgr2=0x7efcfc1e55e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:52.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.296+0000 7efd01273640 1 --2- 172.21.15.57:0/2721769470 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efcfc0cbd20 0x7efcfc1e55e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:52.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.296+0000 7efd01273640 1 -- 172.21.15.57:0/2721769470 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcf800a950 con 0x7efcfc0afa20 2023-10-19T13:00:52.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.296+0000 7efd01a74640 1 --2- 172.21.15.57:0/2721769470 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efcfc0cbd20 0x7efcfc1e55e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T13:00:52.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.296+0000 7efd00a72640 1 --2- 172.21.15.57:0/2721769470 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7efcfc0b0c40 0x7efcfc1e50a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T13:00:52.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.296+0000 7efd01273640 1 --2- 172.21.15.57:0/2721769470 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efcfc0afa20 0x7efcfc13e400 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7efcf400c3c0 tx=0x7efcf400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:52.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.296+0000 7efcf1ffb640 1 -- 172.21.15.57:0/2721769470 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcf40035f0 con 0x7efcfc0afa20 2023-10-19T13:00:52.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.296+0000 7efcf1ffb640 1 -- 172.21.15.57:0/2721769470 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7efcf4003770 con 0x7efcfc0afa20 2023-10-19T13:00:52.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.296+0000 7efd034d7640 1 -- 172.21.15.57:0/2721769470 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcfc14a150 con 0x7efcfc0afa20 2023-10-19T13:00:52.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.296+0000 7efcf1ffb640 1 -- 172.21.15.57:0/2721769470 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcf4003b80 con 0x7efcfc0afa20 2023-10-19T13:00:52.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.296+0000 7efd034d7640 1 -- 172.21.15.57:0/2721769470 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcfc14a710 con 0x7efcfc0afa20 2023-10-19T13:00:52.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.297+0000 7efd034d7640 1 -- 172.21.15.57:0/2721769470 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcfc14a930 con 0x7efcfc0afa20 2023-10-19T13:00:52.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.298+0000 7efcf1ffb640 1 -- 172.21.15.57:0/2721769470 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 40) v1 ==== 89331+0+0 (secure 0 0 0) 0x7efcf4003ce0 con 0x7efcfc0afa20 2023-10-19T13:00:52.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.298+0000 7efcf1ffb640 1 --2- 172.21.15.57:0/2721769470 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7efce006af60 0x7efce006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:52.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.298+0000 7efcf1ffb640 1 -- 172.21.15.57:0/2721769470 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5205+0+0 (secure 0 0 0) 0x7efcf4088d20 con 0x7efcfc0afa20 2023-10-19T13:00:52.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.298+0000 7efd00a72640 1 --2- 172.21.15.57:0/2721769470 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7efce006af60 0x7efce006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:52.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.299+0000 7efd00a72640 1 --2- 172.21.15.57:0/2721769470 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7efce006af60 0x7efce006d420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7efcfc1e5d80 tx=0x7efcec009040 comp rx=0 tx=0).ready entity=mgr.6679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:52.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.302+0000 7efcf1ffb640 1 -- 172.21.15.57:0/2721769470 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7efcf4053e30 con 0x7efcfc0afa20 2023-10-19T13:00:52.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.478+0000 7efd034d7640 1 -- 172.21.15.57:0/2721769470 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efcfc0690e0 con 0x7efcfc0afa20 2023-10-19T13:00:52.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.479+0000 7efcf1ffb640 1 -- 172.21.15.57:0/2721769470 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) v1 ==== 74+0+13118 (secure 0 0 0) 0x7efcf4017020 con 0x7efcfc0afa20 2023-10-19T13:00:52.482 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:00:52.483 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":96,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T13:00:51.778501+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T13:00:52.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.481+0000 7efd034d7640 1 -- 172.21.15.57:0/2721769470 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7efce006af60 msgr2=0x7efce006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:52.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.481+0000 7efd034d7640 1 --2- 172.21.15.57:0/2721769470 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7efce006af60 0x7efce006d420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7efcfc1e5d80 tx=0x7efcec009040 comp rx=0 tx=0).stop 2023-10-19T13:00:52.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.481+0000 7efd034d7640 1 -- 172.21.15.57:0/2721769470 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efcfc0afa20 msgr2=0x7efcfc13e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:52.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.481+0000 7efd034d7640 1 --2- 172.21.15.57:0/2721769470 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efcfc0afa20 0x7efcfc13e400 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7efcf400c3c0 tx=0x7efcf400c880 comp rx=0 tx=0).stop 2023-10-19T13:00:52.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.481+0000 7efd034d7640 1 -- 172.21.15.57:0/2721769470 shutdown_connections 2023-10-19T13:00:52.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.481+0000 7efd034d7640 1 --2- 172.21.15.57:0/2721769470 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7efce006af60 0x7efce006d420 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:52.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.481+0000 7efd034d7640 1 --2- 172.21.15.57:0/2721769470 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efcfc0cbd20 0x7efcfc1e55e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:52.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.481+0000 7efd034d7640 1 --2- 172.21.15.57:0/2721769470 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7efcfc0b0c40 0x7efcfc1e50a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:52.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.481+0000 7efd034d7640 1 --2- 172.21.15.57:0/2721769470 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efcfc0afa20 0x7efcfc13e400 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:52.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.481+0000 7efd034d7640 1 -- 172.21.15.57:0/2721769470 >> 172.21.15.57:0/2721769470 conn(0x7efcfc069930 msgr2=0x7efcfc0ae910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:52.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.481+0000 7efd034d7640 1 -- 172.21.15.57:0/2721769470 shutdown_connections 2023-10-19T13:00:52.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.481+0000 7efd034d7640 1 -- 172.21.15.57:0/2721769470 wait complete. 2023-10-19T13:00:52.495 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-19T12:44:07.371186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-19T13:00:52.495 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-19T13:00:52.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.606+0000 7fea1d9d7640 1 -- 172.21.15.57:0/1572020674 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fea1813e520 msgr2=0x7fea1806bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:52.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.606+0000 7fea1d9d7640 1 --2- 172.21.15.57:0/1572020674 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fea1813e520 0x7fea1806bb60 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7fea08009830 tx=0x7fea0802bcd0 comp rx=0 tx=0).stop 2023-10-19T13:00:52.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.606+0000 7fea1d9d7640 1 -- 172.21.15.57:0/1572020674 shutdown_connections 2023-10-19T13:00:52.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.606+0000 7fea1d9d7640 1 --2- 172.21.15.57:0/1572020674 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fea18070af0 0x7fea180affe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:52.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.606+0000 7fea1d9d7640 1 --2- 172.21.15.57:0/1572020674 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fea1813e520 0x7fea1806bb60 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:52.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.606+0000 7fea1d9d7640 1 -- 172.21.15.57:0/1572020674 >> 172.21.15.57:0/1572020674 conn(0x7fea18069930 msgr2=0x7fea18069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:52.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.606+0000 7fea1d9d7640 1 -- 172.21.15.57:0/1572020674 shutdown_connections 2023-10-19T13:00:52.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.606+0000 7fea1d9d7640 1 -- 172.21.15.57:0/1572020674 wait complete. 2023-10-19T13:00:52.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.607+0000 7fea1d9d7640 1 Processor -- start 2023-10-19T13:00:52.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.607+0000 7fea1d9d7640 1 -- start start 2023-10-19T13:00:52.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.608+0000 7fea1d9d7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fea18070af0 0x7fea1813b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:52.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.608+0000 7fea1d9d7640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fea1813e520 0x7fea1813c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:52.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.608+0000 7fea1d9d7640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fea18141690 0x7fea180caae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:52.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.608+0000 7fea1d9d7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea180ae520 con 0x7fea18070af0 2023-10-19T13:00:52.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.608+0000 7fea1d9d7640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea1813d9b0 con 0x7fea1813e520 2023-10-19T13:00:52.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.608+0000 7fea1d9d7640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea1813d6d0 con 0x7fea18141690 2023-10-19T13:00:52.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.608+0000 7fea16ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fea18070af0 0x7fea1813b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:52.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.608+0000 7fea16ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fea18070af0 0x7fea1813b3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51578/0 (socket says 172.21.15.57:51578) 2023-10-19T13:00:52.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.608+0000 7fea16ffd640 1 -- 172.21.15.57:0/93403866 learned_addr learned my addr 172.21.15.57:0/93403866 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:52.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.608+0000 7fea16ffd640 1 -- 172.21.15.57:0/93403866 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fea18141690 msgr2=0x7fea180caae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:52.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.608+0000 7fea177fe640 1 --2- 172.21.15.57:0/93403866 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fea18141690 0x7fea180caae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:52.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.608+0000 7fea16ffd640 1 --2- 172.21.15.57:0/93403866 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fea18141690 0x7fea180caae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:52.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.608+0000 7fea16ffd640 1 -- 172.21.15.57:0/93403866 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fea1813e520 msgr2=0x7fea1813c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:52.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.608+0000 7fea16ffd640 1 --2- 172.21.15.57:0/93403866 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fea1813e520 0x7fea1813c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:52.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.608+0000 7fea16ffd640 1 -- 172.21.15.57:0/93403866 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea08009490 con 0x7fea18070af0 2023-10-19T13:00:52.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.608+0000 7fea16ffd640 1 --2- 172.21.15.57:0/93403866 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fea18070af0 0x7fea1813b3f0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7fea08000c00 tx=0x7fea080038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:52.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.608+0000 7fea14ff9640 1 -- 172.21.15.57:0/93403866 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea08003ea0 con 0x7fea18070af0 2023-10-19T13:00:52.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.608+0000 7fea14ff9640 1 -- 172.21.15.57:0/93403866 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fea08034550 con 0x7fea18070af0 2023-10-19T13:00:52.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.608+0000 7fea1d9d7640 1 -- 172.21.15.57:0/93403866 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea1813ba70 con 0x7fea18070af0 2023-10-19T13:00:52.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.608+0000 7fea14ff9640 1 -- 172.21.15.57:0/93403866 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea08034960 con 0x7fea18070af0 2023-10-19T13:00:52.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.608+0000 7fea1d9d7640 1 -- 172.21.15.57:0/93403866 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea1813c030 con 0x7fea18070af0 2023-10-19T13:00:52.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.609+0000 7fea1d9d7640 1 -- 172.21.15.57:0/93403866 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea1813c220 con 0x7fea18070af0 2023-10-19T13:00:52.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.610+0000 7fea14ff9640 1 -- 172.21.15.57:0/93403866 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 40) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fea08034ac0 con 0x7fea18070af0 2023-10-19T13:00:52.618 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.610+0000 7fea14ff9640 1 --2- 172.21.15.57:0/93403866 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fe9f806af60 0x7fe9f806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:52.618 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.610+0000 7fea14ff9640 1 -- 172.21.15.57:0/93403866 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5205+0+0 (secure 0 0 0) 0x7fea080abee0 con 0x7fea18070af0 2023-10-19T13:00:52.618 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.610+0000 7fea0ffff640 1 --2- 172.21.15.57:0/93403866 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fe9f806af60 0x7fe9f806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:52.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.611+0000 7fea0ffff640 1 --2- 172.21.15.57:0/93403866 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fe9f806af60 0x7fe9f806d420 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fea000046b0 tx=0x7fea00007a70 comp rx=0 tx=0).ready entity=mgr.6679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:52.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.614+0000 7fea14ff9640 1 -- 172.21.15.57:0/93403866 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fea08077070 con 0x7fea18070af0 2023-10-19T13:00:52.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.790+0000 7fea1d9d7640 1 -- 172.21.15.57:0/93403866 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fea180690e0 con 0x7fea18070af0 2023-10-19T13:00:52.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.790+0000 7fea14ff9640 1 -- 172.21.15.57:0/93403866 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v96) v1 ==== 93+0+10 (secure 0 0 0) 0x7fea08039020 con 0x7fea18070af0 2023-10-19T13:00:52.794 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-19T13:00:52.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.792+0000 7fea1d9d7640 1 -- 172.21.15.57:0/93403866 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fe9f806af60 msgr2=0x7fe9f806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:52.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.792+0000 7fea1d9d7640 1 --2- 172.21.15.57:0/93403866 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fe9f806af60 0x7fe9f806d420 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fea000046b0 tx=0x7fea00007a70 comp rx=0 tx=0).stop 2023-10-19T13:00:52.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.792+0000 7fea1d9d7640 1 -- 172.21.15.57:0/93403866 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fea18070af0 msgr2=0x7fea1813b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:52.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.792+0000 7fea1d9d7640 1 --2- 172.21.15.57:0/93403866 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fea18070af0 0x7fea1813b3f0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7fea08000c00 tx=0x7fea080038a0 comp rx=0 tx=0).stop 2023-10-19T13:00:52.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.792+0000 7fea1d9d7640 1 -- 172.21.15.57:0/93403866 shutdown_connections 2023-10-19T13:00:52.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.792+0000 7fea1d9d7640 1 --2- 172.21.15.57:0/93403866 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fe9f806af60 0x7fe9f806d420 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:52.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.792+0000 7fea1d9d7640 1 --2- 172.21.15.57:0/93403866 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fea18141690 0x7fea180caae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:52.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.792+0000 7fea1d9d7640 1 --2- 172.21.15.57:0/93403866 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fea1813e520 0x7fea1813c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:52.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.792+0000 7fea1d9d7640 1 --2- 172.21.15.57:0/93403866 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fea18070af0 0x7fea1813b3f0 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:52.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.792+0000 7fea1d9d7640 1 -- 172.21.15.57:0/93403866 >> 172.21.15.57:0/93403866 conn(0x7fea18069930 msgr2=0x7fea180acf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:52.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.792+0000 7fea1d9d7640 1 -- 172.21.15.57:0/93403866 shutdown_connections 2023-10-19T13:00:52.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.792+0000 7fea1d9d7640 1 -- 172.21.15.57:0/93403866 wait complete. 2023-10-19T13:00:52.806 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-19T13:00:52.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.912+0000 7fb82e9ce640 1 -- 172.21.15.57:0/2072988060 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb82815b2a0 msgr2=0x7fb8280cf720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:52.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.912+0000 7fb82e9ce640 1 --2- 172.21.15.57:0/2072988060 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb82815b2a0 0x7fb8280cf720 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7fb828089ae0 tx=0x7fb81802bd10 comp rx=0 tx=0).stop 2023-10-19T13:00:52.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.912+0000 7fb82e9ce640 1 -- 172.21.15.57:0/2072988060 shutdown_connections 2023-10-19T13:00:52.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.912+0000 7fb82e9ce640 1 --2- 172.21.15.57:0/2072988060 >> v2:172.21.15.179:3300/0 conn(0x7fb8280ceb40 0x7fb8280b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:52.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.912+0000 7fb82e9ce640 1 --2- 172.21.15.57:0/2072988060 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb82815b2a0 0x7fb8280cf720 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:52.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.912+0000 7fb82e9ce640 1 --2- 172.21.15.57:0/2072988060 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb8280cf340 0x7fb8280d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:52.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.912+0000 7fb82e9ce640 1 -- 172.21.15.57:0/2072988060 >> 172.21.15.57:0/2072988060 conn(0x7fb8280d0d60 msgr2=0x7fb8280d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:52.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.912+0000 7fb82e9ce640 1 -- 172.21.15.57:0/2072988060 shutdown_connections 2023-10-19T13:00:52.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.912+0000 7fb82e9ce640 1 -- 172.21.15.57:0/2072988060 wait complete. 2023-10-19T13:00:52.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.914+0000 7fb82e9ce640 1 Processor -- start 2023-10-19T13:00:52.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.914+0000 7fb82e9ce640 1 -- start start 2023-10-19T13:00:52.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.914+0000 7fb82e9ce640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8280ceb40 0x7fb8280b5b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:52.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.914+0000 7fb82e9ce640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb8280cf340 0x7fb8281e5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:52.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.914+0000 7fb82e9ce640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb82815b2a0 0x7fb8280c7e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:52.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.914+0000 7fb82e9ce640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb828156930 con 0x7fb8280ceb40 2023-10-19T13:00:52.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.914+0000 7fb82e9ce640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8280b4da0 con 0x7fb82815b2a0 2023-10-19T13:00:52.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.914+0000 7fb82e9ce640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8280b5080 con 0x7fb8280cf340 2023-10-19T13:00:52.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.915+0000 7fb827fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8280ceb40 0x7fb8280b5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:52.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.915+0000 7fb827fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8280ceb40 0x7fb8280b5b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51582/0 (socket says 172.21.15.57:51582) 2023-10-19T13:00:52.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.915+0000 7fb827fff640 1 -- 172.21.15.57:0/1489322049 learned_addr learned my addr 172.21.15.57:0/1489322049 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:52.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.915+0000 7fb8277fe640 1 --2- 172.21.15.57:0/1489322049 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb8280cf340 0x7fb8281e5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:52.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.915+0000 7fb82cf6b640 1 --2- 172.21.15.57:0/1489322049 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb82815b2a0 0x7fb8280c7e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:52.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.915+0000 7fb827fff640 1 -- 172.21.15.57:0/1489322049 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb8280cf340 msgr2=0x7fb8281e5a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:52.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.915+0000 7fb827fff640 1 --2- 172.21.15.57:0/1489322049 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb8280cf340 0x7fb8281e5a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:52.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.915+0000 7fb827fff640 1 -- 172.21.15.57:0/1489322049 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb82815b2a0 msgr2=0x7fb8280c7e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:52.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.915+0000 7fb827fff640 1 --2- 172.21.15.57:0/1489322049 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb82815b2a0 0x7fb8280c7e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:52.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.915+0000 7fb827fff640 1 -- 172.21.15.57:0/1489322049 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb818009560 con 0x7fb8280ceb40 2023-10-19T13:00:52.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.915+0000 7fb8277fe640 1 --2- 172.21.15.57:0/1489322049 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb8280cf340 0x7fb8281e5a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T13:00:52.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.915+0000 7fb827fff640 1 --2- 172.21.15.57:0/1489322049 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8280ceb40 0x7fb8280b5b60 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7fb81c00e9f0 tx=0x7fb81c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:52.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.915+0000 7fb824ff9640 1 -- 172.21.15.57:0/1489322049 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb81c009720 con 0x7fb8280ceb40 2023-10-19T13:00:52.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.915+0000 7fb824ff9640 1 -- 172.21.15.57:0/1489322049 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fb81c00cd80 con 0x7fb8280ceb40 2023-10-19T13:00:52.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.915+0000 7fb82e9ce640 1 -- 172.21.15.57:0/1489322049 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8280b5360 con 0x7fb8280ceb40 2023-10-19T13:00:52.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.915+0000 7fb824ff9640 1 -- 172.21.15.57:0/1489322049 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb81c005280 con 0x7fb8280ceb40 2023-10-19T13:00:52.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.915+0000 7fb82e9ce640 1 -- 172.21.15.57:0/1489322049 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8281e6570 con 0x7fb8280ceb40 2023-10-19T13:00:52.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.916+0000 7fb82e9ce640 1 -- 172.21.15.57:0/1489322049 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8280690e0 con 0x7fb8280ceb40 2023-10-19T13:00:52.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.917+0000 7fb824ff9640 1 -- 172.21.15.57:0/1489322049 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 40) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb81c0053e0 con 0x7fb8280ceb40 2023-10-19T13:00:52.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.917+0000 7fb824ff9640 1 --2- 172.21.15.57:0/1489322049 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fb81406af60 0x7fb81406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:52.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.917+0000 7fb824ff9640 1 -- 172.21.15.57:0/1489322049 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5205+0+0 (secure 0 0 0) 0x7fb81c014070 con 0x7fb8280ceb40 2023-10-19T13:00:52.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.918+0000 7fb8277fe640 1 --2- 172.21.15.57:0/1489322049 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fb81406af60 0x7fb81406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:52.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.918+0000 7fb8277fe640 1 --2- 172.21.15.57:0/1489322049 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fb81406af60 0x7fb81406d420 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb818035000 tx=0x7fb818004730 comp rx=0 tx=0).ready entity=mgr.6679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:52.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:52.921+0000 7fb824ff9640 1 -- 172.21.15.57:0/1489322049 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb81c0553f0 con 0x7fb8280ceb40 2023-10-19T13:00:53.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:53.099+0000 7fb82e9ce640 1 -- 172.21.15.57:0/1489322049 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fb8280d0b40 con 0x7fb8280ceb40 2023-10-19T13:00:53.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:53.099+0000 7fb824ff9640 1 -- 172.21.15.57:0/1489322049 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v96) v1 ==== 84+0+65 (secure 0 0 0) 0x7fb81c058a10 con 0x7fb8280ceb40 2023-10-19T13:00:53.102 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:00:53.103 INFO:teuthology.orchestra.run.smithi057.stdout:[ 2023-10-19T13:00:53.103 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:00:53.103 INFO:teuthology.orchestra.run.smithi057.stdout: "poolnum": 1, 2023-10-19T13:00:53.103 INFO:teuthology.orchestra.run.smithi057.stdout: "poolname": ".mgr" 2023-10-19T13:00:53.103 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:00:53.104 INFO:teuthology.orchestra.run.smithi057.stdout:] 2023-10-19T13:00:53.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:53.101+0000 7fb82e9ce640 1 -- 172.21.15.57:0/1489322049 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fb81406af60 msgr2=0x7fb81406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:53.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:53.101+0000 7fb82e9ce640 1 --2- 172.21.15.57:0/1489322049 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fb81406af60 0x7fb81406d420 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb818035000 tx=0x7fb818004730 comp rx=0 tx=0).stop 2023-10-19T13:00:53.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:53.101+0000 7fb82e9ce640 1 -- 172.21.15.57:0/1489322049 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8280ceb40 msgr2=0x7fb8280b5b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:53.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:53.101+0000 7fb82e9ce640 1 --2- 172.21.15.57:0/1489322049 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8280ceb40 0x7fb8280b5b60 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7fb81c00e9f0 tx=0x7fb81c00edf0 comp rx=0 tx=0).stop 2023-10-19T13:00:53.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:53.101+0000 7fb82e9ce640 1 -- 172.21.15.57:0/1489322049 shutdown_connections 2023-10-19T13:00:53.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:53.101+0000 7fb82e9ce640 1 --2- 172.21.15.57:0/1489322049 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fb81406af60 0x7fb81406d420 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:53.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:53.101+0000 7fb82e9ce640 1 --2- 172.21.15.57:0/1489322049 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb82815b2a0 0x7fb8280c7e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:53.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:53.101+0000 7fb82e9ce640 1 --2- 172.21.15.57:0/1489322049 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb8280cf340 0x7fb8281e5a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:53.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:53.101+0000 7fb82e9ce640 1 --2- 172.21.15.57:0/1489322049 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8280ceb40 0x7fb8280b5b60 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:53.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:53.101+0000 7fb82e9ce640 1 -- 172.21.15.57:0/1489322049 >> 172.21.15.57:0/1489322049 conn(0x7fb8280d0d60 msgr2=0x7fb8280b0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:53.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:53.101+0000 7fb82e9ce640 1 -- 172.21.15.57:0/1489322049 shutdown_connections 2023-10-19T13:00:53.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:53.101+0000 7fb82e9ce640 1 -- 172.21.15.57:0/1489322049 wait complete. 2023-10-19T13:00:53.114 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-19T13:00:53.115 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-19T13:00:53.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:53.226+0000 7f3e3a36e640 1 -- 172.21.15.57:0/2830113100 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3e20004d00 msgr2=0x7f3e20005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:53.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:53.226+0000 7f3e3a36e640 1 --2- 172.21.15.57:0/2830113100 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3e20004d00 0x7f3e20005180 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c009ed0 tx=0x7f3e200057f0 comp rx=0 tx=0).stop 2023-10-19T13:00:53.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:53.226+0000 7f3e3890b640 1 -- 172.21.15.57:0/2830113100 reap_dead start 2023-10-19T13:00:53.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:53.226+0000 7f3e3a36e640 1 -- 172.21.15.57:0/2830113100 shutdown_connections 2023-10-19T13:00:53.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:53.226+0000 7f3e3a36e640 1 -- 172.21.15.57:0/2830113100 >> 172.21.15.57:0/2830113100 conn(0x7f3e340aec40 msgr2=0x7f3e340af050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:53.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:53.226+0000 7f3e3a36e640 1 -- 172.21.15.57:0/2830113100 shutdown_connections 2023-10-19T13:00:53.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:53.226+0000 7f3e3a36e640 1 -- 172.21.15.57:0/2830113100 wait complete. 2023-10-19T13:00:53.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:53.227+0000 7f3e3a36e640 1 Processor -- start 2023-10-19T13:00:53.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:53.227+0000 7f3e3a36e640 1 -- start start 2023-10-19T13:00:53.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:53.228+0000 7f3e3a36e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3e20004d00 0x7f3e341f2350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:53.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:53.228+0000 7f3e3a36e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3e341498d0 0x7f3e340c9c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:53.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:53.228+0000 7f3e3a36e640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3e34149d50 0x7f3e340ca180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:53.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:53.228+0000 7f3e3a36e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e3415aee0 con 0x7f3e20004d00 2023-10-19T13:00:53.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:53.228+0000 7f3e3a36e640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e341f2890 con 0x7f3e341498d0 2023-10-19T13:00:53.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:53.228+0000 7f3e3a36e640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e3415ac00 con 0x7f3e34149d50 2023-10-19T13:00:53.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:53.228+0000 7f3e33fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3e20004d00 0x7f3e341f2350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:53.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:53.228+0000 7f3e33fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3e20004d00 0x7f3e341f2350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51586/0 (socket says 172.21.15.57:51586) 2023-10-19T13:00:53.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:53.228+0000 7f3e33fff640 1 -- 172.21.15.57:0/467788665 learned_addr learned my addr 172.21.15.57:0/467788665 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:53.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:53.228+0000 7f3e337fe640 1 --2- 172.21.15.57:0/467788665 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3e341498d0 0x7f3e340c9c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:53.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:53.228+0000 7f3e3890b640 1 --2- 172.21.15.57:0/467788665 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3e34149d50 0x7f3e340ca180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:53.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:53.228+0000 7f3e33fff640 1 -- 172.21.15.57:0/467788665 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3e34149d50 msgr2=0x7f3e340ca180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:53.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:53.228+0000 7f3e33fff640 1 --2- 172.21.15.57:0/467788665 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3e34149d50 0x7f3e340ca180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:53.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:53.228+0000 7f3e33fff640 1 -- 172.21.15.57:0/467788665 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3e341498d0 msgr2=0x7f3e340c9c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:53.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:53.228+0000 7f3e33fff640 1 --2- 172.21.15.57:0/467788665 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3e341498d0 0x7f3e340c9c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:53.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:53.228+0000 7f3e33fff640 1 -- 172.21.15.57:0/467788665 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e3406ba60 con 0x7f3e20004d00 2023-10-19T13:00:53.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:53.228+0000 7f3e337fe640 1 --2- 172.21.15.57:0/467788665 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3e341498d0 0x7f3e340c9c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T13:00:53.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:53.228+0000 7f3e3890b640 1 --2- 172.21.15.57:0/467788665 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3e34149d50 0x7f3e340ca180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T13:00:53.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:53.229+0000 7f3e33fff640 1 --2- 172.21.15.57:0/467788665 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3e20004d00 0x7f3e341f2350 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f3e24006670 tx=0x7f3e240039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:53.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:53.229+0000 7f3e30ff9640 1 -- 172.21.15.57:0/467788665 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e2400f960 con 0x7f3e20004d00 2023-10-19T13:00:53.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:53.229+0000 7f3e30ff9640 1 -- 172.21.15.57:0/467788665 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f3e2400fae0 con 0x7f3e20004d00 2023-10-19T13:00:53.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:53.229+0000 7f3e30ff9640 1 -- 172.21.15.57:0/467788665 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e24011430 con 0x7f3e20004d00 2023-10-19T13:00:53.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:53.229+0000 7f3e3a36e640 1 -- 172.21.15.57:0/467788665 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e3406bd40 con 0x7f3e20004d00 2023-10-19T13:00:53.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:53.229+0000 7f3e3a36e640 1 -- 172.21.15.57:0/467788665 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e34159e90 con 0x7f3e20004d00 2023-10-19T13:00:53.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:53.230+0000 7f3e3a36e640 1 -- 172.21.15.57:0/467788665 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e3406b8c0 con 0x7f3e20004d00 2023-10-19T13:00:53.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:53.230+0000 7f3e30ff9640 1 -- 172.21.15.57:0/467788665 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 40) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f3e24011590 con 0x7f3e20004d00 2023-10-19T13:00:53.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:53.231+0000 7f3e30ff9640 1 --2- 172.21.15.57:0/467788665 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f3e1406af60 0x7f3e1406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:53.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:53.231+0000 7f3e30ff9640 1 -- 172.21.15.57:0/467788665 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f3e2408a5a0 con 0x7f3e20004d00 2023-10-19T13:00:53.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:53.231+0000 7f3e337fe640 1 --2- 172.21.15.57:0/467788665 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f3e1406af60 0x7f3e1406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:53.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:53.231+0000 7f3e337fe640 1 --2- 172.21.15.57:0/467788665 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f3e1406af60 0x7f3e1406d420 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3e340aea60 tx=0x7f3e1c005f90 comp rx=0 tx=0).ready entity=mgr.6679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:53.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:53.235+0000 7f3e30ff9640 1 -- 172.21.15.57:0/467788665 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3e3406b8c0 con 0x7f3e20004d00 2023-10-19T13:00:53.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:53.414+0000 7f3e3a36e640 1 -- 172.21.15.57:0/467788665 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f3e3406c260 con 0x7f3e20004d00 2023-10-19T13:00:54.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:54.180+0000 7f3e30ff9640 1 -- 172.21.15.57:0/467788665 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v97) v1 ==== 138+0+0 (secure 0 0 0) 0x7f3e3406c260 con 0x7f3e20004d00 2023-10-19T13:00:54.183 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'cephfs_metadata' created 2023-10-19T13:00:54.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:54.184+0000 7f3e3a36e640 1 -- 172.21.15.57:0/467788665 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f3e1406af60 msgr2=0x7f3e1406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:54.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:54.184+0000 7f3e3a36e640 1 --2- 172.21.15.57:0/467788665 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f3e1406af60 0x7f3e1406d420 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3e340aea60 tx=0x7f3e1c005f90 comp rx=0 tx=0).stop 2023-10-19T13:00:54.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:54.184+0000 7f3e3a36e640 1 -- 172.21.15.57:0/467788665 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3e20004d00 msgr2=0x7f3e341f2350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:54.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:54.184+0000 7f3e3a36e640 1 --2- 172.21.15.57:0/467788665 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3e20004d00 0x7f3e341f2350 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f3e24006670 tx=0x7f3e240039c0 comp rx=0 tx=0).stop 2023-10-19T13:00:54.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:54.184+0000 7f3e3a36e640 1 -- 172.21.15.57:0/467788665 shutdown_connections 2023-10-19T13:00:54.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:54.184+0000 7f3e3a36e640 1 --2- 172.21.15.57:0/467788665 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f3e1406af60 0x7f3e1406d420 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:54.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:54.184+0000 7f3e3a36e640 1 --2- 172.21.15.57:0/467788665 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3e34149d50 0x7f3e340ca180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:54.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:54.184+0000 7f3e3a36e640 1 --2- 172.21.15.57:0/467788665 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3e341498d0 0x7f3e340c9c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:54.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:54.184+0000 7f3e3a36e640 1 --2- 172.21.15.57:0/467788665 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3e20004d00 0x7f3e341f2350 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:54.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:54.184+0000 7f3e3a36e640 1 -- 172.21.15.57:0/467788665 >> 172.21.15.57:0/467788665 conn(0x7f3e340aec40 msgr2=0x7f3e34070350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:54.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:54.184+0000 7f3e3a36e640 1 -- 172.21.15.57:0/467788665 shutdown_connections 2023-10-19T13:00:54.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:54.184+0000 7f3e3a36e640 1 -- 172.21.15.57:0/467788665 wait complete. 2023-10-19T13:00:54.203 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-19T13:00:54.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:54.322+0000 7fd628525640 1 -- 172.21.15.57:0/73353514 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd62015b2a0 msgr2=0x7fd6200d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:54.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:54.322+0000 7fd628525640 1 --2- 172.21.15.57:0/73353514 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd62015b2a0 0x7fd6200d1910 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7fd614005fd0 tx=0x7fd61402bd90 comp rx=0 tx=0).stop 2023-10-19T13:00:54.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:54.322+0000 7fd628525640 1 -- 172.21.15.57:0/73353514 shutdown_connections 2023-10-19T13:00:54.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:54.322+0000 7fd628525640 1 --2- 172.21.15.57:0/73353514 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd6200ceb40 0x7fd6200b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:54.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:54.322+0000 7fd628525640 1 --2- 172.21.15.57:0/73353514 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd62015b2a0 0x7fd6200d1910 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:54.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:54.322+0000 7fd628525640 1 -- 172.21.15.57:0/73353514 >> 172.21.15.57:0/73353514 conn(0x7fd6200d0d60 msgr2=0x7fd6200d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:54.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:54.322+0000 7fd628525640 1 -- 172.21.15.57:0/73353514 shutdown_connections 2023-10-19T13:00:54.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:54.322+0000 7fd628525640 1 -- 172.21.15.57:0/73353514 wait complete. 2023-10-19T13:00:54.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:54.324+0000 7fd628525640 1 Processor -- start 2023-10-19T13:00:54.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:54.324+0000 7fd628525640 1 -- start start 2023-10-19T13:00:54.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:54.324+0000 7fd628525640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd6200ceb40 0x7fd620157080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:54.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:54.324+0000 7fd628525640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd6200cf340 0x7fd62006afe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:54.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:54.324+0000 7fd628525640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd62015b2a0 0x7fd62006b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:54.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:54.324+0000 7fd628525640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6201f1560 con 0x7fd62015b2a0 2023-10-19T13:00:54.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:54.324+0000 7fd628525640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6201f1750 con 0x7fd6200ceb40 2023-10-19T13:00:54.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:54.324+0000 7fd628525640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6201f19f0 con 0x7fd6200cf340 2023-10-19T13:00:54.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:54.324+0000 7fd626ac2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd62015b2a0 0x7fd62006b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:54.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:54.324+0000 7fd626ac2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd62015b2a0 0x7fd62006b520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51606/0 (socket says 172.21.15.57:51606) 2023-10-19T13:00:54.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:54.324+0000 7fd626ac2640 1 -- 172.21.15.57:0/2523638427 learned_addr learned my addr 172.21.15.57:0/2523638427 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:54.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:54.325+0000 7fd625ac0640 1 --2- 172.21.15.57:0/2523638427 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd6200cf340 0x7fd62006afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:54.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:54.325+0000 7fd6262c1640 1 --2- 172.21.15.57:0/2523638427 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd6200ceb40 0x7fd620157080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:54.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:54.325+0000 7fd626ac2640 1 -- 172.21.15.57:0/2523638427 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd6200cf340 msgr2=0x7fd62006afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:54.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:54.325+0000 7fd626ac2640 1 --2- 172.21.15.57:0/2523638427 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd6200cf340 0x7fd62006afe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:54.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:54.325+0000 7fd626ac2640 1 -- 172.21.15.57:0/2523638427 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd6200ceb40 msgr2=0x7fd620157080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:54.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:54.325+0000 7fd626ac2640 1 --2- 172.21.15.57:0/2523638427 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd6200ceb40 0x7fd620157080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:54.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:54.325+0000 7fd626ac2640 1 -- 172.21.15.57:0/2523638427 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd614009560 con 0x7fd62015b2a0 2023-10-19T13:00:54.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:54.325+0000 7fd625ac0640 1 --2- 172.21.15.57:0/2523638427 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd6200cf340 0x7fd62006afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T13:00:54.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:54.325+0000 7fd6262c1640 1 --2- 172.21.15.57:0/2523638427 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd6200ceb40 0x7fd620157080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T13:00:54.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:54.325+0000 7fd626ac2640 1 --2- 172.21.15.57:0/2523638427 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd62015b2a0 0x7fd62006b520 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fd61000cad0 tx=0x7fd61000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:54.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:54.325+0000 7fd61effd640 1 -- 172.21.15.57:0/2523638427 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd61000ae20 con 0x7fd62015b2a0 2023-10-19T13:00:54.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:54.325+0000 7fd61effd640 1 -- 172.21.15.57:0/2523638427 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fd610004b20 con 0x7fd62015b2a0 2023-10-19T13:00:54.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:54.325+0000 7fd628525640 1 -- 172.21.15.57:0/2523638427 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6200b0d50 con 0x7fd62015b2a0 2023-10-19T13:00:54.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:54.325+0000 7fd61effd640 1 -- 172.21.15.57:0/2523638427 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd610005020 con 0x7fd62015b2a0 2023-10-19T13:00:54.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:54.325+0000 7fd628525640 1 -- 172.21.15.57:0/2523638427 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6201f1fb0 con 0x7fd62015b2a0 2023-10-19T13:00:54.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:54.326+0000 7fd628525640 1 -- 172.21.15.57:0/2523638427 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6200677b0 con 0x7fd62015b2a0 2023-10-19T13:00:54.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:54.327+0000 7fd61effd640 1 -- 172.21.15.57:0/2523638427 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 40) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd610005180 con 0x7fd62015b2a0 2023-10-19T13:00:54.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:54.327+0000 7fd61effd640 1 --2- 172.21.15.57:0/2523638427 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fd60c06af60 0x7fd60c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:54.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:54.327+0000 7fd61effd640 1 -- 172.21.15.57:0/2523638427 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5575+0+0 (secure 0 0 0) 0x7fd61008a0a0 con 0x7fd62015b2a0 2023-10-19T13:00:54.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:54.327+0000 7fd6262c1640 1 --2- 172.21.15.57:0/2523638427 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fd60c06af60 0x7fd60c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:54.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:54.328+0000 7fd6262c1640 1 --2- 172.21.15.57:0/2523638427 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fd60c06af60 0x7fd60c06d420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd618000f60 tx=0x7fd618009040 comp rx=0 tx=0).ready entity=mgr.6679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:54.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:54.331+0000 7fd61effd640 1 -- 172.21.15.57:0/2523638427 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd610054fb0 con 0x7fd62015b2a0 2023-10-19T13:00:54.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:54.509+0000 7fd628525640 1 -- 172.21.15.57:0/2523638427 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fd6200690e0 con 0x7fd62015b2a0 2023-10-19T13:00:55.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:55.182+0000 7fd61effd640 1 -- 172.21.15.57:0/2523638427 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v98) v1 ==== 170+0+0 (secure 0 0 0) 0x7fd61001d0d0 con 0x7fd62015b2a0 2023-10-19T13:00:55.186 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'cephfs_data' created 2023-10-19T13:00:55.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:55.185+0000 7fd628525640 1 -- 172.21.15.57:0/2523638427 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fd60c06af60 msgr2=0x7fd60c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:55.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:55.185+0000 7fd628525640 1 --2- 172.21.15.57:0/2523638427 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fd60c06af60 0x7fd60c06d420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd618000f60 tx=0x7fd618009040 comp rx=0 tx=0).stop 2023-10-19T13:00:55.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:55.185+0000 7fd628525640 1 -- 172.21.15.57:0/2523638427 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd62015b2a0 msgr2=0x7fd62006b520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:55.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:55.185+0000 7fd628525640 1 --2- 172.21.15.57:0/2523638427 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd62015b2a0 0x7fd62006b520 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fd61000cad0 tx=0x7fd61000cf90 comp rx=0 tx=0).stop 2023-10-19T13:00:55.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:55.185+0000 7fd628525640 1 -- 172.21.15.57:0/2523638427 shutdown_connections 2023-10-19T13:00:55.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:55.185+0000 7fd628525640 1 --2- 172.21.15.57:0/2523638427 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fd60c06af60 0x7fd60c06d420 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:55.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:55.185+0000 7fd628525640 1 --2- 172.21.15.57:0/2523638427 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd62015b2a0 0x7fd62006b520 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:55.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:55.185+0000 7fd628525640 1 --2- 172.21.15.57:0/2523638427 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd6200cf340 0x7fd62006afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:55.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:55.185+0000 7fd628525640 1 --2- 172.21.15.57:0/2523638427 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd6200ceb40 0x7fd620157080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:55.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:55.185+0000 7fd628525640 1 -- 172.21.15.57:0/2523638427 >> 172.21.15.57:0/2523638427 conn(0x7fd6200d0d60 msgr2=0x7fd6200b2850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:55.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:55.185+0000 7fd628525640 1 -- 172.21.15.57:0/2523638427 shutdown_connections 2023-10-19T13:00:55.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:55.185+0000 7fd628525640 1 -- 172.21.15.57:0/2523638427 wait complete. 2023-10-19T13:00:55.209 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-19T13:00:55.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:55.321+0000 7f7ae9f39640 1 -- 172.21.15.57:0/973880918 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ae4158770 msgr2=0x7f7ae40c9560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:55.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:55.321+0000 7f7ae9f39640 1 --2- 172.21.15.57:0/973880918 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ae4158770 0x7f7ae40c9560 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f7adc005fd0 tx=0x7f7adc02c0e0 comp rx=0 tx=0).stop 2023-10-19T13:00:55.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:55.321+0000 7f7ae9f39640 1 -- 172.21.15.57:0/973880918 shutdown_connections 2023-10-19T13:00:55.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:55.321+0000 7f7ae9f39640 1 --2- 172.21.15.57:0/973880918 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ae4158770 0x7f7ae40c9560 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:55.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:55.321+0000 7f7ae9f39640 1 -- 172.21.15.57:0/973880918 >> 172.21.15.57:0/973880918 conn(0x7f7ae406fae0 msgr2=0x7f7ae406fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:55.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:55.321+0000 7f7ae9f39640 1 -- 172.21.15.57:0/973880918 shutdown_connections 2023-10-19T13:00:55.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:55.321+0000 7f7ae9f39640 1 -- 172.21.15.57:0/973880918 wait complete. 2023-10-19T13:00:55.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:55.322+0000 7f7ae9f39640 1 Processor -- start 2023-10-19T13:00:55.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:55.322+0000 7f7ae9f39640 1 -- start start 2023-10-19T13:00:55.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:55.323+0000 7f7ae9f39640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7ae406ae40 0x7f7ae4154b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:55.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:55.323+0000 7f7ae9f39640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7ae414eba0 0x7f7ae4153860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:55.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:55.323+0000 7f7ae9f39640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ae4158770 0x7f7ae414bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:55.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:55.323+0000 7f7ae9f39640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ae4152570 con 0x7f7ae4158770 2023-10-19T13:00:55.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:55.323+0000 7f7ae9f39640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ae40aca20 con 0x7f7ae414eba0 2023-10-19T13:00:55.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:55.323+0000 7f7ae9f39640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ae41527c0 con 0x7f7ae406ae40 2023-10-19T13:00:55.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:55.323+0000 7f7ae3fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ae4158770 0x7f7ae414bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:55.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:55.323+0000 7f7ae3fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ae4158770 0x7f7ae414bb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51626/0 (socket says 172.21.15.57:51626) 2023-10-19T13:00:55.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:55.323+0000 7f7ae3fff640 1 -- 172.21.15.57:0/4130931610 learned_addr learned my addr 172.21.15.57:0/4130931610 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:55.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:55.323+0000 7f7ae37fe640 1 --2- 172.21.15.57:0/4130931610 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7ae406ae40 0x7f7ae4154b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:55.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:55.323+0000 7f7ae2ffd640 1 --2- 172.21.15.57:0/4130931610 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7ae414eba0 0x7f7ae4153860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:55.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:55.323+0000 7f7ae3fff640 1 -- 172.21.15.57:0/4130931610 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7ae406ae40 msgr2=0x7f7ae4154b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:55.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:55.323+0000 7f7ae3fff640 1 --2- 172.21.15.57:0/4130931610 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7ae406ae40 0x7f7ae4154b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:55.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:55.323+0000 7f7ae3fff640 1 -- 172.21.15.57:0/4130931610 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7ae414eba0 msgr2=0x7f7ae4153860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:55.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:55.323+0000 7f7ae3fff640 1 --2- 172.21.15.57:0/4130931610 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7ae414eba0 0x7f7ae4153860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:55.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:55.323+0000 7f7ae3fff640 1 -- 172.21.15.57:0/4130931610 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7adc009840 con 0x7f7ae4158770 2023-10-19T13:00:55.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:55.323+0000 7f7ae3fff640 1 --2- 172.21.15.57:0/4130931610 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ae4158770 0x7f7ae414bb10 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f7adc005fd0 tx=0x7f7adc004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:55.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:55.324+0000 7f7acbfff640 1 -- 172.21.15.57:0/4130931610 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7adc004760 con 0x7f7ae4158770 2023-10-19T13:00:55.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:55.324+0000 7f7ae9f39640 1 -- 172.21.15.57:0/4130931610 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ae4152aa0 con 0x7f7ae4158770 2023-10-19T13:00:55.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:55.324+0000 7f7acbfff640 1 -- 172.21.15.57:0/4130931610 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f7adc0345a0 con 0x7f7ae4158770 2023-10-19T13:00:55.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:55.324+0000 7f7acbfff640 1 -- 172.21.15.57:0/4130931610 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7adc034990 con 0x7f7ae4158770 2023-10-19T13:00:55.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:55.324+0000 7f7ae9f39640 1 -- 172.21.15.57:0/4130931610 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ae4153060 con 0x7f7ae4158770 2023-10-19T13:00:55.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:55.325+0000 7f7ae9f39640 1 -- 172.21.15.57:0/4130931610 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ae40690e0 con 0x7f7ae4158770 2023-10-19T13:00:55.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:55.325+0000 7f7acbfff640 1 -- 172.21.15.57:0/4130931610 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 40) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f7adc034af0 con 0x7f7ae4158770 2023-10-19T13:00:55.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:55.326+0000 7f7acbfff640 1 --2- 172.21.15.57:0/4130931610 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f7ad006af60 0x7f7ad006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:55.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:55.326+0000 7f7acbfff640 1 -- 172.21.15.57:0/4130931610 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5957+0+0 (secure 0 0 0) 0x7f7adc0acc30 con 0x7f7ae4158770 2023-10-19T13:00:55.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:55.326+0000 7f7ae37fe640 1 --2- 172.21.15.57:0/4130931610 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f7ad006af60 0x7f7ad006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:55.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:55.327+0000 7f7ae37fe640 1 --2- 172.21.15.57:0/4130931610 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f7ad006af60 0x7f7ad006d420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f7ad4004ea0 tx=0x7f7ad4009040 comp rx=0 tx=0).ready entity=mgr.6679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:55.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:55.330+0000 7f7acbfff640 1 -- 172.21.15.57:0/4130931610 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7adc077a40 con 0x7f7ae4158770 2023-10-19T13:00:55.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:55.546+0000 7f7ae9f39640 1 -- 172.21.15.57:0/4130931610 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f7ae40cc920 con 0x7f7ae4158770 2023-10-19T13:00:56.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.217+0000 7f7acbfff640 1 -- 172.21.15.57:0/4130931610 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 11 and data pool 12 v35) v1 ==== 174+0+0 (secure 0 0 0) 0x7f7adc07b060 con 0x7f7ae4158770 2023-10-19T13:00:56.221 INFO:teuthology.orchestra.run.smithi057.stderr:new fs with metadata pool 11 and data pool 12 2023-10-19T13:00:56.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.228+0000 7f7ae9f39640 1 -- 172.21.15.57:0/4130931610 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f7ad006af60 msgr2=0x7f7ad006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:56.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.228+0000 7f7ae9f39640 1 --2- 172.21.15.57:0/4130931610 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f7ad006af60 0x7f7ad006d420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f7ad4004ea0 tx=0x7f7ad4009040 comp rx=0 tx=0).stop 2023-10-19T13:00:56.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.228+0000 7f7ae9f39640 1 -- 172.21.15.57:0/4130931610 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ae4158770 msgr2=0x7f7ae414bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:56.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.228+0000 7f7ae9f39640 1 --2- 172.21.15.57:0/4130931610 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ae4158770 0x7f7ae414bb10 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f7adc005fd0 tx=0x7f7adc004360 comp rx=0 tx=0).stop 2023-10-19T13:00:56.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.231+0000 7f7ae9f39640 1 -- 172.21.15.57:0/4130931610 shutdown_connections 2023-10-19T13:00:56.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.231+0000 7f7ae9f39640 1 --2- 172.21.15.57:0/4130931610 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f7ad006af60 0x7f7ad006d420 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:56.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.231+0000 7f7ae9f39640 1 --2- 172.21.15.57:0/4130931610 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ae4158770 0x7f7ae414bb10 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:56.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.231+0000 7f7ae9f39640 1 --2- 172.21.15.57:0/4130931610 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7ae414eba0 0x7f7ae4153860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:56.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.231+0000 7f7ae9f39640 1 --2- 172.21.15.57:0/4130931610 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7ae406ae40 0x7f7ae4154b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:56.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.231+0000 7f7ae9f39640 1 -- 172.21.15.57:0/4130931610 >> 172.21.15.57:0/4130931610 conn(0x7f7ae406fae0 msgr2=0x7f7ae40b1800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:56.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.233+0000 7f7ae9f39640 1 -- 172.21.15.57:0/4130931610 shutdown_connections 2023-10-19T13:00:56.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.233+0000 7f7ae9f39640 1 -- 172.21.15.57:0/4130931610 wait complete. 2023-10-19T13:00:56.253 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T13:00:56.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.365+0000 7fabffd16640 1 -- 172.21.15.57:0/3195906339 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fabe00047c0 msgr2=0x7fabe0004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:56.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.365+0000 7fabffd16640 1 --2- 172.21.15.57:0/3195906339 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fabe00047c0 0x7fabe0004c40 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7fabec001650 tx=0x7fabe00057f0 comp rx=0 tx=0).stop 2023-10-19T13:00:56.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.365+0000 7fabfe2b3640 1 -- 172.21.15.57:0/3195906339 reap_dead start 2023-10-19T13:00:56.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.365+0000 7fabffd16640 1 -- 172.21.15.57:0/3195906339 shutdown_connections 2023-10-19T13:00:56.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.365+0000 7fabffd16640 1 -- 172.21.15.57:0/3195906339 >> 172.21.15.57:0/3195906339 conn(0x7fabf8069930 msgr2=0x7fabf8069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:56.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.365+0000 7fabffd16640 1 -- 172.21.15.57:0/3195906339 shutdown_connections 2023-10-19T13:00:56.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.365+0000 7fabffd16640 1 -- 172.21.15.57:0/3195906339 wait complete. 2023-10-19T13:00:56.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.366+0000 7fabffd16640 1 Processor -- start 2023-10-19T13:00:56.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.366+0000 7fabffd16640 1 -- start start 2023-10-19T13:00:56.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.367+0000 7fabffd16640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fabe00047c0 0x7fabf81eb240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:56.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.367+0000 7fabffd16640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fabf8159bd0 0x7fabf80acb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:56.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.367+0000 7fabffd16640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fabf81f12a0 0x7fabf81562c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:56.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.367+0000 7fabffd16640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabf80a8500 con 0x7fabf81f12a0 2023-10-19T13:00:56.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.367+0000 7fabffd16640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabf81eb780 con 0x7fabe00047c0 2023-10-19T13:00:56.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.367+0000 7fabffd16640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabf81f48c0 con 0x7fabf8159bd0 2023-10-19T13:00:56.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.367+0000 7fabfe2b3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fabf81f12a0 0x7fabf81562c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:56.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.367+0000 7fabfe2b3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fabf81f12a0 0x7fabf81562c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51650/0 (socket says 172.21.15.57:51650) 2023-10-19T13:00:56.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.367+0000 7fabfe2b3640 1 -- 172.21.15.57:0/3593997597 learned_addr learned my addr 172.21.15.57:0/3593997597 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:56.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.367+0000 7fabfd2b1640 1 --2- 172.21.15.57:0/3593997597 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fabf8159bd0 0x7fabf80acb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:56.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.367+0000 7fabfdab2640 1 --2- 172.21.15.57:0/3593997597 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fabe00047c0 0x7fabf81eb240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:56.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.367+0000 7fabfe2b3640 1 -- 172.21.15.57:0/3593997597 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fabf8159bd0 msgr2=0x7fabf80acb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:56.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.367+0000 7fabfe2b3640 1 --2- 172.21.15.57:0/3593997597 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fabf8159bd0 0x7fabf80acb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:56.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.367+0000 7fabfe2b3640 1 -- 172.21.15.57:0/3593997597 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fabe00047c0 msgr2=0x7fabf81eb240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:56.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.367+0000 7fabfe2b3640 1 --2- 172.21.15.57:0/3593997597 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fabe00047c0 0x7fabf81eb240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:56.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.367+0000 7fabfe2b3640 1 -- 172.21.15.57:0/3593997597 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabf815a090 con 0x7fabf81f12a0 2023-10-19T13:00:56.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.367+0000 7fabfe2b3640 1 --2- 172.21.15.57:0/3593997597 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fabf81f12a0 0x7fabf81562c0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7fabf806f410 tx=0x7fabf00039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:56.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.368+0000 7fabf67fc640 1 -- 172.21.15.57:0/3593997597 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabf0011840 con 0x7fabf81f12a0 2023-10-19T13:00:56.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.368+0000 7fabf67fc640 1 -- 172.21.15.57:0/3593997597 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fabf00119c0 con 0x7fabf81f12a0 2023-10-19T13:00:56.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.368+0000 7fabffd16640 1 -- 172.21.15.57:0/3593997597 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabf815a370 con 0x7fabf81f12a0 2023-10-19T13:00:56.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.368+0000 7fabf67fc640 1 -- 172.21.15.57:0/3593997597 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabf0011dd0 con 0x7fabf81f12a0 2023-10-19T13:00:56.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.368+0000 7fabffd16640 1 -- 172.21.15.57:0/3593997597 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabf815a930 con 0x7fabf81f12a0 2023-10-19T13:00:56.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.369+0000 7fabffd16640 1 -- 172.21.15.57:0/3593997597 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabf80690e0 con 0x7fabf81f12a0 2023-10-19T13:00:56.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.369+0000 7fabf67fc640 1 -- 172.21.15.57:0/3593997597 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 40) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fabf001d460 con 0x7fabf81f12a0 2023-10-19T13:00:56.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.369+0000 7fabf67fc640 1 --2- 172.21.15.57:0/3593997597 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fabe406af60 0x7fabe406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:56.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.370+0000 7fabf67fc640 1 -- 172.21.15.57:0/3593997597 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fabf008a310 con 0x7fabf81f12a0 2023-10-19T13:00:56.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.370+0000 7fabfdab2640 1 --2- 172.21.15.57:0/3593997597 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fabe406af60 0x7fabe406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:56.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.370+0000 7fabfdab2640 1 --2- 172.21.15.57:0/3593997597 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fabe406af60 0x7fabe406d420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fabf8151030 tx=0x7fabe800a610 comp rx=0 tx=0).ready entity=mgr.6679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:56.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.374+0000 7fabf67fc640 1 -- 172.21.15.57:0/3593997597 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fabf00550f0 con 0x7fabf81f12a0 2023-10-19T13:00:56.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.551+0000 7fabffd16640 1 -- 172.21.15.57:0/3593997597 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fabf80ac840 con 0x7fabf81f12a0 2023-10-19T13:00:56.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.552+0000 7fabf67fc640 1 -- 172.21.15.57:0/3593997597 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v99) v1 ==== 74+0+16845 (secure 0 0 0) 0x7fabf0058710 con 0x7fabf81f12a0 2023-10-19T13:00:56.555 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:00:56.556 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":99,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T13:00:55.547491+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2023-10-19T13:00:53.446697+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2023-10-19T13:00:54.540962+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T13:00:56.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.554+0000 7fabffd16640 1 -- 172.21.15.57:0/3593997597 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fabe406af60 msgr2=0x7fabe406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:56.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.554+0000 7fabffd16640 1 --2- 172.21.15.57:0/3593997597 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fabe406af60 0x7fabe406d420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fabf8151030 tx=0x7fabe800a610 comp rx=0 tx=0).stop 2023-10-19T13:00:56.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.554+0000 7fabffd16640 1 -- 172.21.15.57:0/3593997597 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fabf81f12a0 msgr2=0x7fabf81562c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:56.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.554+0000 7fabffd16640 1 --2- 172.21.15.57:0/3593997597 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fabf81f12a0 0x7fabf81562c0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7fabf806f410 tx=0x7fabf00039c0 comp rx=0 tx=0).stop 2023-10-19T13:00:56.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.554+0000 7fabffd16640 1 -- 172.21.15.57:0/3593997597 shutdown_connections 2023-10-19T13:00:56.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.554+0000 7fabffd16640 1 --2- 172.21.15.57:0/3593997597 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fabe406af60 0x7fabe406d420 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:56.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.554+0000 7fabffd16640 1 --2- 172.21.15.57:0/3593997597 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fabf81f12a0 0x7fabf81562c0 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:56.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.554+0000 7fabffd16640 1 --2- 172.21.15.57:0/3593997597 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fabf8159bd0 0x7fabf80acb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:56.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.554+0000 7fabffd16640 1 --2- 172.21.15.57:0/3593997597 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fabe00047c0 0x7fabf81eb240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:56.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.554+0000 7fabffd16640 1 -- 172.21.15.57:0/3593997597 >> 172.21.15.57:0/3593997597 conn(0x7fabf8069930 msgr2=0x7fabf8150790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:56.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.554+0000 7fabffd16640 1 -- 172.21.15.57:0/3593997597 shutdown_connections 2023-10-19T13:00:56.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.554+0000 7fabffd16640 1 -- 172.21.15.57:0/3593997597 wait complete. 2023-10-19T13:00:56.568 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T13:00:56.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.674+0000 7fb08e99e640 1 -- 172.21.15.57:0/1302999119 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb0880cf340 msgr2=0x7fb0880d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:56.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.674+0000 7fb08e99e640 1 --2- 172.21.15.57:0/1302999119 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb0880cf340 0x7fb0880d1910 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7fb078009830 tx=0x7fb07802bd60 comp rx=0 tx=0).stop 2023-10-19T13:00:56.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.674+0000 7fb08e99e640 1 -- 172.21.15.57:0/1302999119 shutdown_connections 2023-10-19T13:00:56.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.674+0000 7fb08e99e640 1 --2- 172.21.15.57:0/1302999119 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb0880cf340 0x7fb0880d1910 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:56.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.674+0000 7fb08e99e640 1 -- 172.21.15.57:0/1302999119 >> 172.21.15.57:0/1302999119 conn(0x7fb0880d0d60 msgr2=0x7fb0880d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:56.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.674+0000 7fb08e99e640 1 -- 172.21.15.57:0/1302999119 shutdown_connections 2023-10-19T13:00:56.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.674+0000 7fb08e99e640 1 -- 172.21.15.57:0/1302999119 wait complete. 2023-10-19T13:00:56.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.675+0000 7fb08e99e640 1 Processor -- start 2023-10-19T13:00:56.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.676+0000 7fb08e99e640 1 -- start start 2023-10-19T13:00:56.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.676+0000 7fb08e99e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb0880ceb40 0x7fb0880c8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:56.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.676+0000 7fb08e99e640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb0880cf340 0x7fb0880c9b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:56.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.676+0000 7fb08e99e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb08815b2a0 0x7fb0881f3e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:56.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.676+0000 7fb08e99e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb088158d30 con 0x7fb08815b2a0 2023-10-19T13:00:56.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.676+0000 7fb08e99e640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0880b53c0 con 0x7fb0880ceb40 2023-10-19T13:00:56.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.676+0000 7fb08e99e640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0880b4840 con 0x7fb0880cf340 2023-10-19T13:00:56.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.676+0000 7fb08cf3b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb08815b2a0 0x7fb0881f3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:56.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.677+0000 7fb08cf3b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb08815b2a0 0x7fb0881f3e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37968/0 (socket says 172.21.15.57:37968) 2023-10-19T13:00:56.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.677+0000 7fb08cf3b640 1 -- 172.21.15.57:0/1188342149 learned_addr learned my addr 172.21.15.57:0/1188342149 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:56.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.677+0000 7fb087fff640 1 --2- 172.21.15.57:0/1188342149 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb0880ceb40 0x7fb0880c8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:56.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.677+0000 7fb0877fe640 1 --2- 172.21.15.57:0/1188342149 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb0880cf340 0x7fb0880c9b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:56.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.677+0000 7fb08cf3b640 1 -- 172.21.15.57:0/1188342149 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb0880cf340 msgr2=0x7fb0880c9b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:56.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.677+0000 7fb08cf3b640 1 --2- 172.21.15.57:0/1188342149 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb0880cf340 0x7fb0880c9b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:56.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.677+0000 7fb08cf3b640 1 -- 172.21.15.57:0/1188342149 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb0880ceb40 msgr2=0x7fb0880c8eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:56.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.677+0000 7fb08cf3b640 1 --2- 172.21.15.57:0/1188342149 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb0880ceb40 0x7fb0880c8eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:56.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.677+0000 7fb08cf3b640 1 -- 172.21.15.57:0/1188342149 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb078009490 con 0x7fb08815b2a0 2023-10-19T13:00:56.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.677+0000 7fb0877fe640 1 --2- 172.21.15.57:0/1188342149 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb0880cf340 0x7fb0880c9b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:00:56.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.677+0000 7fb08cf3b640 1 --2- 172.21.15.57:0/1188342149 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb08815b2a0 0x7fb0881f3e10 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7fb08000cb40 tx=0x7fb08000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:56.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.677+0000 7fb084ff9640 1 -- 172.21.15.57:0/1188342149 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb08000aca0 con 0x7fb08815b2a0 2023-10-19T13:00:56.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.677+0000 7fb084ff9640 1 -- 172.21.15.57:0/1188342149 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fb080012420 con 0x7fb08815b2a0 2023-10-19T13:00:56.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.677+0000 7fb08e99e640 1 -- 172.21.15.57:0/1188342149 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0880b4b20 con 0x7fb08815b2a0 2023-10-19T13:00:56.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.677+0000 7fb084ff9640 1 -- 172.21.15.57:0/1188342149 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb080012850 con 0x7fb08815b2a0 2023-10-19T13:00:56.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.677+0000 7fb08e99e640 1 -- 172.21.15.57:0/1188342149 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0880b50e0 con 0x7fb08815b2a0 2023-10-19T13:00:56.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.678+0000 7fb08e99e640 1 -- 172.21.15.57:0/1188342149 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0881f1480 con 0x7fb08815b2a0 2023-10-19T13:00:56.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.679+0000 7fb084ff9640 1 -- 172.21.15.57:0/1188342149 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 40) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb08001a3b0 con 0x7fb08815b2a0 2023-10-19T13:00:56.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.679+0000 7fb084ff9640 1 --2- 172.21.15.57:0/1188342149 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fb07406af60 0x7fb07406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:56.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.679+0000 7fb084ff9640 1 -- 172.21.15.57:0/1188342149 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fb08008a080 con 0x7fb08815b2a0 2023-10-19T13:00:56.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.679+0000 7fb087fff640 1 --2- 172.21.15.57:0/1188342149 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fb07406af60 0x7fb07406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:56.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.680+0000 7fb087fff640 1 --2- 172.21.15.57:0/1188342149 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fb07406af60 0x7fb07406d420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb078036000 tx=0x7fb078004730 comp rx=0 tx=0).ready entity=mgr.6679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:56.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.683+0000 7fb084ff9640 1 -- 172.21.15.57:0/1188342149 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb080054ec0 con 0x7fb08815b2a0 2023-10-19T13:00:56.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.859+0000 7fb08e99e640 1 -- 172.21.15.57:0/1188342149 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb0880690e0 con 0x7fb08815b2a0 2023-10-19T13:00:56.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.860+0000 7fb084ff9640 1 -- 172.21.15.57:0/1188342149 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v99) v1 ==== 74+0+16845 (secure 0 0 0) 0x7fb0800584e0 con 0x7fb08815b2a0 2023-10-19T13:00:56.863 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:00:56.864 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":99,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T13:00:55.547491+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2023-10-19T13:00:53.446697+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2023-10-19T13:00:54.540962+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T13:00:56.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.862+0000 7fb08e99e640 1 -- 172.21.15.57:0/1188342149 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fb07406af60 msgr2=0x7fb07406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:56.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.862+0000 7fb08e99e640 1 --2- 172.21.15.57:0/1188342149 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fb07406af60 0x7fb07406d420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb078036000 tx=0x7fb078004730 comp rx=0 tx=0).stop 2023-10-19T13:00:56.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.862+0000 7fb08e99e640 1 -- 172.21.15.57:0/1188342149 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb08815b2a0 msgr2=0x7fb0881f3e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:56.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.862+0000 7fb08e99e640 1 --2- 172.21.15.57:0/1188342149 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb08815b2a0 0x7fb0881f3e10 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7fb08000cb40 tx=0x7fb08000a060 comp rx=0 tx=0).stop 2023-10-19T13:00:56.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.862+0000 7fb08e99e640 1 -- 172.21.15.57:0/1188342149 shutdown_connections 2023-10-19T13:00:56.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.862+0000 7fb08e99e640 1 --2- 172.21.15.57:0/1188342149 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fb07406af60 0x7fb07406d420 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:56.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.862+0000 7fb08e99e640 1 --2- 172.21.15.57:0/1188342149 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb08815b2a0 0x7fb0881f3e10 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:56.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.862+0000 7fb08e99e640 1 --2- 172.21.15.57:0/1188342149 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb0880cf340 0x7fb0880c9b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:56.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.862+0000 7fb08e99e640 1 --2- 172.21.15.57:0/1188342149 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb0880ceb40 0x7fb0880c8eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:56.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.862+0000 7fb08e99e640 1 -- 172.21.15.57:0/1188342149 >> 172.21.15.57:0/1188342149 conn(0x7fb0880d0d60 msgr2=0x7fb08806b450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:56.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.862+0000 7fb08e99e640 1 -- 172.21.15.57:0/1188342149 shutdown_connections 2023-10-19T13:00:56.868 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.862+0000 7fb08e99e640 1 -- 172.21.15.57:0/1188342149 wait complete. 2023-10-19T13:00:56.876 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-19T13:00:56.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.985+0000 7fea19efc640 1 -- 172.21.15.57:0/3297801914 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fea140c9dc0 msgr2=0x7fea1406b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:56.989 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.985+0000 7fea19efc640 1 --2- 172.21.15.57:0/3297801914 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fea140c9dc0 0x7fea1406b2e0 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7fea08009830 tx=0x7fea0802bdf0 comp rx=0 tx=0).stop 2023-10-19T13:00:56.989 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.985+0000 7fea19efc640 1 -- 172.21.15.57:0/3297801914 shutdown_connections 2023-10-19T13:00:56.989 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.985+0000 7fea19efc640 1 --2- 172.21.15.57:0/3297801914 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fea140c9dc0 0x7fea1406b2e0 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:56.989 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.985+0000 7fea19efc640 1 -- 172.21.15.57:0/3297801914 >> 172.21.15.57:0/3297801914 conn(0x7fea14069930 msgr2=0x7fea14069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:56.990 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.985+0000 7fea19efc640 1 -- 172.21.15.57:0/3297801914 shutdown_connections 2023-10-19T13:00:56.990 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.985+0000 7fea19efc640 1 -- 172.21.15.57:0/3297801914 wait complete. 2023-10-19T13:00:56.993 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.986+0000 7fea19efc640 1 Processor -- start 2023-10-19T13:00:56.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.987+0000 7fea19efc640 1 -- start start 2023-10-19T13:00:56.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.987+0000 7fea19efc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fea1406af00 0x7fea14070180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:56.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.987+0000 7fea19efc640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fea140ab500 0x7fea1413a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:56.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.987+0000 7fea19efc640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fea140c9dc0 0x7fea140aa8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:56.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.987+0000 7fea19efc640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea1406fd60 con 0x7fea1406af00 2023-10-19T13:00:56.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.987+0000 7fea19efc640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea1406f120 con 0x7fea140c9dc0 2023-10-19T13:00:56.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.987+0000 7fea19efc640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea1406f3c0 con 0x7fea140ab500 2023-10-19T13:00:56.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.987+0000 7fea137fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fea1406af00 0x7fea14070180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:56.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.987+0000 7fea137fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fea1406af00 0x7fea14070180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37976/0 (socket says 172.21.15.57:37976) 2023-10-19T13:00:56.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.987+0000 7fea137fe640 1 -- 172.21.15.57:0/3556323882 learned_addr learned my addr 172.21.15.57:0/3556323882 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:56.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.988+0000 7fea12ffd640 1 --2- 172.21.15.57:0/3556323882 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fea140ab500 0x7fea1413a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:56.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.988+0000 7fea13fff640 1 --2- 172.21.15.57:0/3556323882 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fea140c9dc0 0x7fea140aa8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:56.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.988+0000 7fea137fe640 1 -- 172.21.15.57:0/3556323882 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fea140ab500 msgr2=0x7fea1413a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:56.997 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.988+0000 7fea137fe640 1 --2- 172.21.15.57:0/3556323882 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fea140ab500 0x7fea1413a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:56.997 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.988+0000 7fea137fe640 1 -- 172.21.15.57:0/3556323882 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fea140c9dc0 msgr2=0x7fea140aa8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:56.997 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.988+0000 7fea137fe640 1 --2- 172.21.15.57:0/3556323882 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fea140c9dc0 0x7fea140aa8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:56.997 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.988+0000 7fea137fe640 1 -- 172.21.15.57:0/3556323882 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea08009490 con 0x7fea1406af00 2023-10-19T13:00:56.997 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.988+0000 7fea137fe640 1 --2- 172.21.15.57:0/3556323882 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fea1406af00 0x7fea14070180 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7fea0400c460 tx=0x7fea0400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:56.998 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.988+0000 7fe9fbfff640 1 -- 172.21.15.57:0/3556323882 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea04005480 con 0x7fea1406af00 2023-10-19T13:00:56.998 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.988+0000 7fe9fbfff640 1 -- 172.21.15.57:0/3556323882 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fea040035f0 con 0x7fea1406af00 2023-10-19T13:00:56.998 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.988+0000 7fea19efc640 1 -- 172.21.15.57:0/3556323882 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea14159a20 con 0x7fea1406af00 2023-10-19T13:00:56.998 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.988+0000 7fe9fbfff640 1 -- 172.21.15.57:0/3556323882 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea04003a00 con 0x7fea1406af00 2023-10-19T13:00:56.999 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.988+0000 7fea19efc640 1 -- 172.21.15.57:0/3556323882 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea1406f980 con 0x7fea1406af00 2023-10-19T13:00:56.999 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.989+0000 7fea19efc640 1 -- 172.21.15.57:0/3556323882 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea1406fea0 con 0x7fea1406af00 2023-10-19T13:00:56.999 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.990+0000 7fe9fbfff640 1 -- 172.21.15.57:0/3556323882 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 40) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fea04003b60 con 0x7fea1406af00 2023-10-19T13:00:56.999 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.990+0000 7fe9fbfff640 1 --2- 172.21.15.57:0/3556323882 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fea0006af90 0x7fea0006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:56.999 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.990+0000 7fe9fbfff640 1 -- 172.21.15.57:0/3556323882 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fea04089500 con 0x7fea1406af00 2023-10-19T13:00:57.000 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.990+0000 7fea12ffd640 1 --2- 172.21.15.57:0/3556323882 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fea0006af90 0x7fea0006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:57.000 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.991+0000 7fea12ffd640 1 --2- 172.21.15.57:0/3556323882 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fea0006af90 0x7fea0006d450 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fea08036000 tx=0x7fea08004730 comp rx=0 tx=0).ready entity=mgr.6679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:57.001 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:56.994+0000 7fe9fbfff640 1 -- 172.21.15.57:0/3556323882 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fea1406fea0 con 0x7fea1406af00 2023-10-19T13:00:57.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:57.205+0000 7fea19efc640 1 -- 172.21.15.57:0/3556323882 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fea1406fea0 con 0x7fea1406af00 2023-10-19T13:00:57.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:57.219+0000 7fe9fbfff640 1 -- 172.21.15.57:0/3556323882 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v36) v1 ==== 118+0+0 (secure 0 0 0) 0x7fea1406fea0 con 0x7fea1406af00 2023-10-19T13:00:57.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:57.221+0000 7fea19efc640 1 -- 172.21.15.57:0/3556323882 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fea0006af90 msgr2=0x7fea0006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:57.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:57.221+0000 7fea19efc640 1 --2- 172.21.15.57:0/3556323882 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fea0006af90 0x7fea0006d450 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fea08036000 tx=0x7fea08004730 comp rx=0 tx=0).stop 2023-10-19T13:00:57.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:57.221+0000 7fea19efc640 1 -- 172.21.15.57:0/3556323882 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fea1406af00 msgr2=0x7fea14070180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:57.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:57.221+0000 7fea19efc640 1 --2- 172.21.15.57:0/3556323882 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fea1406af00 0x7fea14070180 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7fea0400c460 tx=0x7fea0400c920 comp rx=0 tx=0).stop 2023-10-19T13:00:57.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:57.221+0000 7fea19efc640 1 -- 172.21.15.57:0/3556323882 shutdown_connections 2023-10-19T13:00:57.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:57.221+0000 7fea19efc640 1 --2- 172.21.15.57:0/3556323882 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fea0006af90 0x7fea0006d450 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:57.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:57.221+0000 7fea19efc640 1 --2- 172.21.15.57:0/3556323882 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fea140c9dc0 0x7fea140aa8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:57.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:57.221+0000 7fea19efc640 1 --2- 172.21.15.57:0/3556323882 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fea140ab500 0x7fea1413a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:57.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:57.221+0000 7fea19efc640 1 --2- 172.21.15.57:0/3556323882 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fea1406af00 0x7fea14070180 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:57.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:57.221+0000 7fea19efc640 1 -- 172.21.15.57:0/3556323882 >> 172.21.15.57:0/3556323882 conn(0x7fea14069930 msgr2=0x7fea140ac370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:57.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:57.221+0000 7fea19efc640 1 -- 172.21.15.57:0/3556323882 shutdown_connections 2023-10-19T13:00:57.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:57.221+0000 7fea19efc640 1 -- 172.21.15.57:0/3556323882 wait complete. 2023-10-19T13:00:57.235 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-19T13:00:57.236 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-19T13:00:57.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:57.348+0000 7f23cc59e640 1 -- 172.21.15.57:0/363937743 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f23b0004d00 msgr2=0x7f23b0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:57.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:57.348+0000 7f23cc59e640 1 --2- 172.21.15.57:0/363937743 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f23b0004d00 0x7f23b0005180 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f23bc005fd0 tx=0x7f23b00057f0 comp rx=0 tx=0).stop 2023-10-19T13:00:57.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:57.348+0000 7f23cab3b640 1 -- 172.21.15.57:0/363937743 reap_dead start 2023-10-19T13:00:57.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:57.348+0000 7f23cc59e640 1 -- 172.21.15.57:0/363937743 shutdown_connections 2023-10-19T13:00:57.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:57.348+0000 7f23cc59e640 1 -- 172.21.15.57:0/363937743 >> 172.21.15.57:0/363937743 conn(0x7f23c40d0d60 msgr2=0x7f23c40d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:57.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:57.348+0000 7f23cc59e640 1 -- 172.21.15.57:0/363937743 shutdown_connections 2023-10-19T13:00:57.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:57.348+0000 7f23cc59e640 1 -- 172.21.15.57:0/363937743 wait complete. 2023-10-19T13:00:57.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:57.349+0000 7f23cc59e640 1 Processor -- start 2023-10-19T13:00:57.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:57.349+0000 7f23cc59e640 1 -- start start 2023-10-19T13:00:57.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:57.350+0000 7f23cc59e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f23b0004d00 0x7f23c40c7e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:57.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:57.350+0000 7f23cc59e640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f23c414f590 0x7f23c40ca600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:57.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:57.350+0000 7f23cc59e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f23c40cab40 0x7f23c41e56e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:57.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:57.350+0000 7f23cc59e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23c41e3f90 con 0x7f23b0004d00 2023-10-19T13:00:57.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:57.350+0000 7f23cc59e640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23c41e4de0 con 0x7f23c40cab40 2023-10-19T13:00:57.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:57.350+0000 7f23cc59e640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23c41e41a0 con 0x7f23c414f590 2023-10-19T13:00:57.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:57.350+0000 7f23ca33a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f23b0004d00 0x7f23c40c7e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:57.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:57.350+0000 7f23ca33a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f23b0004d00 0x7f23c40c7e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37982/0 (socket says 172.21.15.57:37982) 2023-10-19T13:00:57.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:57.350+0000 7f23ca33a640 1 -- 172.21.15.57:0/2996518788 learned_addr learned my addr 172.21.15.57:0/2996518788 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:57.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:57.350+0000 7f23c9b39640 1 --2- 172.21.15.57:0/2996518788 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f23c414f590 0x7f23c40ca600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:57.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:57.350+0000 7f23cab3b640 1 --2- 172.21.15.57:0/2996518788 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f23c40cab40 0x7f23c41e56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:57.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:57.350+0000 7f23ca33a640 1 -- 172.21.15.57:0/2996518788 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f23c414f590 msgr2=0x7f23c40ca600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:57.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:57.350+0000 7f23ca33a640 1 --2- 172.21.15.57:0/2996518788 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f23c414f590 0x7f23c40ca600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:57.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:57.350+0000 7f23ca33a640 1 -- 172.21.15.57:0/2996518788 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f23c40cab40 msgr2=0x7f23c41e56e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:57.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:57.350+0000 7f23ca33a640 1 --2- 172.21.15.57:0/2996518788 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f23c40cab40 0x7f23c41e56e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:57.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:57.350+0000 7f23ca33a640 1 -- 172.21.15.57:0/2996518788 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23c41e4480 con 0x7f23b0004d00 2023-10-19T13:00:57.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:57.350+0000 7f23ca33a640 1 --2- 172.21.15.57:0/2996518788 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f23b0004d00 0x7f23c40c7e30 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f23c0006650 tx=0x7f23c0003a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:57.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:57.351+0000 7f23baffd640 1 -- 172.21.15.57:0/2996518788 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23c000f9e0 con 0x7f23b0004d00 2023-10-19T13:00:57.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:57.351+0000 7f23baffd640 1 -- 172.21.15.57:0/2996518788 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f23c000fb40 con 0x7f23b0004d00 2023-10-19T13:00:57.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:57.351+0000 7f23baffd640 1 -- 172.21.15.57:0/2996518788 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23c0011430 con 0x7f23b0004d00 2023-10-19T13:00:57.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:57.351+0000 7f23cc59e640 1 -- 172.21.15.57:0/2996518788 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23c41e4760 con 0x7f23b0004d00 2023-10-19T13:00:57.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:57.351+0000 7f23cc59e640 1 -- 172.21.15.57:0/2996518788 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23c415c7f0 con 0x7f23b0004d00 2023-10-19T13:00:57.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:57.352+0000 7f23cc59e640 1 -- 172.21.15.57:0/2996518788 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23c415cd40 con 0x7f23b0004d00 2023-10-19T13:00:57.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:57.352+0000 7f23baffd640 1 -- 172.21.15.57:0/2996518788 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 40) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f23c0011590 con 0x7f23b0004d00 2023-10-19T13:00:57.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:57.353+0000 7f23baffd640 1 --2- 172.21.15.57:0/2996518788 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f23a806af90 0x7f23a806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:57.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:57.353+0000 7f23baffd640 1 -- 172.21.15.57:0/2996518788 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f23c008a8a0 con 0x7f23b0004d00 2023-10-19T13:00:57.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:57.353+0000 7f23c9b39640 1 --2- 172.21.15.57:0/2996518788 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f23a806af90 0x7f23a806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:57.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:57.353+0000 7f23c9b39640 1 --2- 172.21.15.57:0/2996518788 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f23a806af90 0x7f23a806d450 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f23c40b0fe0 tx=0x7f23b4006b70 comp rx=0 tx=0).ready entity=mgr.6679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:57.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:57.357+0000 7f23baffd640 1 -- 172.21.15.57:0/2996518788 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f23c415cd40 con 0x7f23b0004d00 2023-10-19T13:00:57.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:57.570+0000 7f23cc59e640 1 -- 172.21.15.57:0/2996518788 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f23c415cd40 con 0x7f23b0004d00 2023-10-19T13:00:58.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.231+0000 7f23baffd640 1 -- 172.21.15.57:0/2996518788 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v37) v1 ==== 122+0+0 (secure 0 0 0) 0x7f23c415cd40 con 0x7f23b0004d00 2023-10-19T13:00:58.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.234+0000 7f23cc59e640 1 -- 172.21.15.57:0/2996518788 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f23a806af90 msgr2=0x7f23a806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:58.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.234+0000 7f23cc59e640 1 --2- 172.21.15.57:0/2996518788 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f23a806af90 0x7f23a806d450 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f23c40b0fe0 tx=0x7f23b4006b70 comp rx=0 tx=0).stop 2023-10-19T13:00:58.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.234+0000 7f23cc59e640 1 -- 172.21.15.57:0/2996518788 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f23b0004d00 msgr2=0x7f23c40c7e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:58.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.234+0000 7f23cc59e640 1 --2- 172.21.15.57:0/2996518788 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f23b0004d00 0x7f23c40c7e30 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f23c0006650 tx=0x7f23c0003a80 comp rx=0 tx=0).stop 2023-10-19T13:00:58.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.234+0000 7f23cc59e640 1 -- 172.21.15.57:0/2996518788 shutdown_connections 2023-10-19T13:00:58.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.234+0000 7f23cc59e640 1 --2- 172.21.15.57:0/2996518788 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f23a806af90 0x7f23a806d450 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:58.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.234+0000 7f23cc59e640 1 --2- 172.21.15.57:0/2996518788 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f23c40cab40 0x7f23c41e56e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:58.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.234+0000 7f23cc59e640 1 --2- 172.21.15.57:0/2996518788 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f23c414f590 0x7f23c40ca600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:58.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.234+0000 7f23cc59e640 1 --2- 172.21.15.57:0/2996518788 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f23b0004d00 0x7f23c40c7e30 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:58.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.234+0000 7f23cc59e640 1 -- 172.21.15.57:0/2996518788 >> 172.21.15.57:0/2996518788 conn(0x7f23c40d0d60 msgr2=0x7f23c40cf720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:58.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.234+0000 7f23cc59e640 1 -- 172.21.15.57:0/2996518788 shutdown_connections 2023-10-19T13:00:58.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.234+0000 7f23cc59e640 1 -- 172.21.15.57:0/2996518788 wait complete. 2023-10-19T13:00:58.248 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-19T13:00:58.249 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T13:00:58.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.359+0000 7f71295a5640 1 -- 172.21.15.57:0/3654142161 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f710c0039c0 msgr2=0x7f710c003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:58.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.359+0000 7f71295a5640 1 --2- 172.21.15.57:0/3654142161 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f710c0039c0 0x7f710c003de0 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f7114006f00 tx=0x7f710c0057f0 comp rx=0 tx=0).stop 2023-10-19T13:00:58.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.359+0000 7f71295a5640 1 -- 172.21.15.57:0/3654142161 shutdown_connections 2023-10-19T13:00:58.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.359+0000 7f71295a5640 1 --2- 172.21.15.57:0/3654142161 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f710c0039c0 0x7f710c003de0 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:58.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.359+0000 7f71295a5640 1 -- 172.21.15.57:0/3654142161 >> 172.21.15.57:0/3654142161 conn(0x7f7124069930 msgr2=0x7f7124069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:58.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.359+0000 7f71295a5640 1 -- 172.21.15.57:0/3654142161 shutdown_connections 2023-10-19T13:00:58.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.359+0000 7f71295a5640 1 -- 172.21.15.57:0/3654142161 wait complete. 2023-10-19T13:00:58.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.360+0000 7f71295a5640 1 Processor -- start 2023-10-19T13:00:58.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.360+0000 7f71295a5640 1 -- start start 2023-10-19T13:00:58.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.361+0000 7f71295a5640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f710c0039c0 0x7f71241390f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:58.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.361+0000 7f71295a5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7124156600 0x7f712414c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:58.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.361+0000 7f71295a5640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7124156a60 0x7f712414c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:58.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.361+0000 7f71295a5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f712414b950 con 0x7f7124156600 2023-10-19T13:00:58.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.361+0000 7f71295a5640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f712414acd0 con 0x7f7124156a60 2023-10-19T13:00:58.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.361+0000 7f71295a5640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f712414afb0 con 0x7f710c0039c0 2023-10-19T13:00:58.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.361+0000 7f71227fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7124156600 0x7f712414c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:58.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.361+0000 7f71227fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7124156600 0x7f712414c0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38012/0 (socket says 172.21.15.57:38012) 2023-10-19T13:00:58.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.361+0000 7f71227fc640 1 -- 172.21.15.57:0/1378779034 learned_addr learned my addr 172.21.15.57:0/1378779034 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:58.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.361+0000 7f71237fe640 1 --2- 172.21.15.57:0/1378779034 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7124156a60 0x7f712414c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:58.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.361+0000 7f7122ffd640 1 --2- 172.21.15.57:0/1378779034 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f710c0039c0 0x7f71241390f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:58.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.361+0000 7f71227fc640 1 -- 172.21.15.57:0/1378779034 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f710c0039c0 msgr2=0x7f71241390f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:58.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.361+0000 7f71227fc640 1 --2- 172.21.15.57:0/1378779034 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f710c0039c0 0x7f71241390f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:58.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.361+0000 7f71227fc640 1 -- 172.21.15.57:0/1378779034 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7124156a60 msgr2=0x7f712414c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:58.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.361+0000 7f71227fc640 1 --2- 172.21.15.57:0/1378779034 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7124156a60 0x7f712414c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:58.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.361+0000 7f71227fc640 1 -- 172.21.15.57:0/1378779034 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f712414b290 con 0x7f7124156600 2023-10-19T13:00:58.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.361+0000 7f71237fe640 1 --2- 172.21.15.57:0/1378779034 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7124156a60 0x7f712414c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:00:58.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.362+0000 7f71227fc640 1 --2- 172.21.15.57:0/1378779034 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7124156600 0x7f712414c0a0 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f7118003e40 tx=0x7f7118001630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:58.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.362+0000 7f710b7fe640 1 -- 172.21.15.57:0/1378779034 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7118011840 con 0x7f7124156600 2023-10-19T13:00:58.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.362+0000 7f710b7fe640 1 -- 172.21.15.57:0/1378779034 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f71180119a0 con 0x7f7124156600 2023-10-19T13:00:58.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.362+0000 7f71295a5640 1 -- 172.21.15.57:0/1378779034 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f712414b570 con 0x7f7124156600 2023-10-19T13:00:58.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.362+0000 7f710b7fe640 1 -- 172.21.15.57:0/1378779034 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7118011d90 con 0x7f7124156600 2023-10-19T13:00:58.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.362+0000 7f71295a5640 1 -- 172.21.15.57:0/1378779034 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f712414a8e0 con 0x7f7124156600 2023-10-19T13:00:58.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.363+0000 7f71295a5640 1 -- 172.21.15.57:0/1378779034 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71240690e0 con 0x7f7124156600 2023-10-19T13:00:58.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.363+0000 7f710b7fe640 1 -- 172.21.15.57:0/1378779034 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 40) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f71180103c0 con 0x7f7124156600 2023-10-19T13:00:58.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.364+0000 7f710b7fe640 1 --2- 172.21.15.57:0/1378779034 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f711006af60 0x7f711006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:58.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.364+0000 7f710b7fe640 1 -- 172.21.15.57:0/1378779034 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f711808a8d0 con 0x7f7124156600 2023-10-19T13:00:58.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.364+0000 7f7122ffd640 1 --2- 172.21.15.57:0/1378779034 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f711006af60 0x7f711006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:58.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.364+0000 7f7122ffd640 1 --2- 172.21.15.57:0/1378779034 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f711006af60 0x7f711006d420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7114034000 tx=0x7f7114039040 comp rx=0 tx=0).ready entity=mgr.6679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:58.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.368+0000 7f710b7fe640 1 -- 172.21.15.57:0/1378779034 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7118055690 con 0x7f7124156600 2023-10-19T13:00:58.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.579+0000 7f71295a5640 1 -- 172.21.15.57:0/1378779034 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7124150f70 con 0x7f7124156600 2023-10-19T13:00:58.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.580+0000 7f710b7fe640 1 -- 172.21.15.57:0/1378779034 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+4921 (secure 0 0 0) 0x7f711801f070 con 0x7f7124156600 2023-10-19T13:00:58.583 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:00:58.583 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":37,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6305,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/2631343514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":2631343514},{"type":"v1","addr":"172.21.15.179:6837","nonce":2631343514}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":34},{"gid":6883,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/4008617188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":4008617188},{"type":"v1","addr":"172.21.15.179:6836","nonce":4008617188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":34},{"gid":7542,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/2010253133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2010253133},{"type":"v1","addr":"172.21.15.57:6834","nonce":2010253133}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T13:00:55.547801+0000","modified":"2023-10-19T13:00:57.571937+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7545},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7545":{"gid":7545,"name":"a","rank":0,"incarnation":35,"state":"up:active","state_seq":3,"addr":"172.21.15.57:6835/2583001580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2583001580},{"type":"v1","addr":"172.21.15.57:6835","nonce":2583001580}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-10-19T13:00:58.583 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 37 2023-10-19T13:00:58.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.581+0000 7f71295a5640 1 -- 172.21.15.57:0/1378779034 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f711006af60 msgr2=0x7f711006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:58.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.581+0000 7f71295a5640 1 --2- 172.21.15.57:0/1378779034 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f711006af60 0x7f711006d420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7114034000 tx=0x7f7114039040 comp rx=0 tx=0).stop 2023-10-19T13:00:58.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.581+0000 7f71295a5640 1 -- 172.21.15.57:0/1378779034 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7124156600 msgr2=0x7f712414c0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:58.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.581+0000 7f71295a5640 1 --2- 172.21.15.57:0/1378779034 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7124156600 0x7f712414c0a0 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f7118003e40 tx=0x7f7118001630 comp rx=0 tx=0).stop 2023-10-19T13:00:58.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.581+0000 7f71295a5640 1 -- 172.21.15.57:0/1378779034 shutdown_connections 2023-10-19T13:00:58.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.581+0000 7f71295a5640 1 --2- 172.21.15.57:0/1378779034 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f711006af60 0x7f711006d420 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:58.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.581+0000 7f71295a5640 1 --2- 172.21.15.57:0/1378779034 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7124156a60 0x7f712414c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:58.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.581+0000 7f71295a5640 1 --2- 172.21.15.57:0/1378779034 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7124156600 0x7f712414c0a0 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:58.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.581+0000 7f71295a5640 1 --2- 172.21.15.57:0/1378779034 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f710c0039c0 0x7f71241390f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:58.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.581+0000 7f71295a5640 1 -- 172.21.15.57:0/1378779034 >> 172.21.15.57:0/1378779034 conn(0x7f7124069930 msgr2=0x7f712406c130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:58.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.582+0000 7f71295a5640 1 -- 172.21.15.57:0/1378779034 shutdown_connections 2023-10-19T13:00:58.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.582+0000 7f71295a5640 1 -- 172.21.15.57:0/1378779034 wait complete. 2023-10-19T13:00:58.595 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T13:00:58.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.701+0000 7fe2fe141640 1 -- 172.21.15.57:0/413880450 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe2f806bec0 msgr2=0x7fe2f806a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:58.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.701+0000 7fe2fe141640 1 --2- 172.21.15.57:0/413880450 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe2f806bec0 0x7fe2f806a730 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7fe2f80cc700 tx=0x7fe2f002c0e0 comp rx=0 tx=0).stop 2023-10-19T13:00:58.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.701+0000 7fe2fe141640 1 -- 172.21.15.57:0/413880450 shutdown_connections 2023-10-19T13:00:58.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.701+0000 7fe2fe141640 1 --2- 172.21.15.57:0/413880450 >> v2:172.21.15.179:3300/0 conn(0x7fe2f80ac870 0x7fe2f806b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:58.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.701+0000 7fe2fe141640 1 --2- 172.21.15.57:0/413880450 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe2f806bec0 0x7fe2f806a730 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:58.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.701+0000 7fe2fe141640 1 -- 172.21.15.57:0/413880450 >> 172.21.15.57:0/413880450 conn(0x7fe2f806fae0 msgr2=0x7fe2f806fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:58.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.701+0000 7fe2fe141640 1 -- 172.21.15.57:0/413880450 shutdown_connections 2023-10-19T13:00:58.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.701+0000 7fe2fe141640 1 -- 172.21.15.57:0/413880450 wait complete. 2023-10-19T13:00:58.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.703+0000 7fe2fe141640 1 Processor -- start 2023-10-19T13:00:58.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.703+0000 7fe2fe141640 1 -- start start 2023-10-19T13:00:58.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.703+0000 7fe2fe141640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe2f806bec0 0x7fe2f81ef990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:58.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.703+0000 7fe2fe141640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe2f80ac870 0x7fe2f814ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:58.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.703+0000 7fe2fe141640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe2f8156790 0x7fe2f814d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:58.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.704+0000 7fe2fe141640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f814ccb0 con 0x7fe2f8156790 2023-10-19T13:00:58.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.704+0000 7fe2fe141640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f814d1e0 con 0x7fe2f806bec0 2023-10-19T13:00:58.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.704+0000 7fe2fe141640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f814cf00 con 0x7fe2f80ac870 2023-10-19T13:00:58.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.704+0000 7fe2f7fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe2f8156790 0x7fe2f814d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:58.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.704+0000 7fe2f7fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe2f8156790 0x7fe2f814d5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38032/0 (socket says 172.21.15.57:38032) 2023-10-19T13:00:58.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.704+0000 7fe2f7fff640 1 -- 172.21.15.57:0/464762962 learned_addr learned my addr 172.21.15.57:0/464762962 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:58.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.704+0000 7fe2f77fe640 1 --2- 172.21.15.57:0/464762962 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe2f806bec0 0x7fe2f81ef990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:58.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.704+0000 7fe2f6ffd640 1 --2- 172.21.15.57:0/464762962 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe2f80ac870 0x7fe2f814ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:58.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.704+0000 7fe2f7fff640 1 -- 172.21.15.57:0/464762962 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe2f80ac870 msgr2=0x7fe2f814ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:58.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.704+0000 7fe2f7fff640 1 --2- 172.21.15.57:0/464762962 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe2f80ac870 0x7fe2f814ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:58.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.704+0000 7fe2f7fff640 1 -- 172.21.15.57:0/464762962 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe2f806bec0 msgr2=0x7fe2f81ef990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:58.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.704+0000 7fe2f7fff640 1 --2- 172.21.15.57:0/464762962 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe2f806bec0 0x7fe2f81ef990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:58.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.704+0000 7fe2f7fff640 1 -- 172.21.15.57:0/464762962 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2f0009840 con 0x7fe2f8156790 2023-10-19T13:00:58.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.704+0000 7fe2f7fff640 1 --2- 172.21.15.57:0/464762962 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe2f8156790 0x7fe2f814d5b0 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7fe2f0036040 tx=0x7fe2f0004300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:58.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.704+0000 7fe2dffff640 1 -- 172.21.15.57:0/464762962 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2f00047c0 con 0x7fe2f8156790 2023-10-19T13:00:58.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.704+0000 7fe2dffff640 1 -- 172.21.15.57:0/464762962 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fe2f0034690 con 0x7fe2f8156790 2023-10-19T13:00:58.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.704+0000 7fe2fe141640 1 -- 172.21.15.57:0/464762962 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2f814ecf0 con 0x7fe2f8156790 2023-10-19T13:00:58.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.704+0000 7fe2dffff640 1 -- 172.21.15.57:0/464762962 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2f0034a60 con 0x7fe2f8156790 2023-10-19T13:00:58.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.705+0000 7fe2fe141640 1 -- 172.21.15.57:0/464762962 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2f81517d0 con 0x7fe2f8156790 2023-10-19T13:00:58.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.706+0000 7fe2fe141640 1 -- 172.21.15.57:0/464762962 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2f814f210 con 0x7fe2f8156790 2023-10-19T13:00:58.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.706+0000 7fe2dffff640 1 -- 172.21.15.57:0/464762962 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 40) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe2f0034bc0 con 0x7fe2f8156790 2023-10-19T13:00:58.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.706+0000 7fe2dffff640 1 --2- 172.21.15.57:0/464762962 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fe2e406af60 0x7fe2e406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:58.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.706+0000 7fe2dffff640 1 -- 172.21.15.57:0/464762962 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fe2f00acbb0 con 0x7fe2f8156790 2023-10-19T13:00:58.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.707+0000 7fe2f77fe640 1 --2- 172.21.15.57:0/464762962 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fe2e406af60 0x7fe2e406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:58.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.707+0000 7fe2f77fe640 1 --2- 172.21.15.57:0/464762962 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fe2e406af60 0x7fe2e406d420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe2e80046b0 tx=0x7fe2e8009040 comp rx=0 tx=0).ready entity=mgr.6679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:58.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.710+0000 7fe2dffff640 1 -- 172.21.15.57:0/464762962 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe2f814f210 con 0x7fe2f8156790 2023-10-19T13:00:58.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.887+0000 7fe2fe141640 1 -- 172.21.15.57:0/464762962 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe2f814f210 con 0x7fe2f8156790 2023-10-19T13:00:58.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.889+0000 7fe2dffff640 1 -- 172.21.15.57:0/464762962 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v100) v1 ==== 74+0+16834 (secure 0 0 0) 0x7fe2f814f210 con 0x7fe2f8156790 2023-10-19T13:00:58.892 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:00:58.892 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":100,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T13:00:57.204074+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2023-10-19T13:00:53.446697+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2023-10-19T13:00:54.540962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T13:00:58.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.890+0000 7fe2fe141640 1 -- 172.21.15.57:0/464762962 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fe2e406af60 msgr2=0x7fe2e406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:58.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.890+0000 7fe2fe141640 1 --2- 172.21.15.57:0/464762962 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fe2e406af60 0x7fe2e406d420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe2e80046b0 tx=0x7fe2e8009040 comp rx=0 tx=0).stop 2023-10-19T13:00:58.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.890+0000 7fe2fe141640 1 -- 172.21.15.57:0/464762962 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe2f8156790 msgr2=0x7fe2f814d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:58.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.890+0000 7fe2fe141640 1 --2- 172.21.15.57:0/464762962 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe2f8156790 0x7fe2f814d5b0 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7fe2f0036040 tx=0x7fe2f0004300 comp rx=0 tx=0).stop 2023-10-19T13:00:58.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.890+0000 7fe2fe141640 1 -- 172.21.15.57:0/464762962 shutdown_connections 2023-10-19T13:00:58.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.890+0000 7fe2fe141640 1 --2- 172.21.15.57:0/464762962 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fe2e406af60 0x7fe2e406d420 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:58.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.890+0000 7fe2fe141640 1 --2- 172.21.15.57:0/464762962 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe2f8156790 0x7fe2f814d5b0 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:58.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.890+0000 7fe2fe141640 1 --2- 172.21.15.57:0/464762962 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe2f80ac870 0x7fe2f814ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:58.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.890+0000 7fe2fe141640 1 --2- 172.21.15.57:0/464762962 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe2f806bec0 0x7fe2f81ef990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:58.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.890+0000 7fe2fe141640 1 -- 172.21.15.57:0/464762962 >> 172.21.15.57:0/464762962 conn(0x7fe2f806fae0 msgr2=0x7fe2f80cc950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:58.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.891+0000 7fe2fe141640 1 -- 172.21.15.57:0/464762962 shutdown_connections 2023-10-19T13:00:58.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:58.891+0000 7fe2fe141640 1 -- 172.21.15.57:0/464762962 wait complete. 2023-10-19T13:00:58.904 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-19T13:00:58.904 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-19T13:00:59.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.010+0000 7f5870f4c640 1 -- 172.21.15.57:0/1832683591 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f586c0b0530 msgr2=0x7f586c0b09c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:59.014 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.010+0000 7f5870f4c640 1 --2- 172.21.15.57:0/1832683591 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f586c0b0530 0x7f586c0b09c0 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f585c005fd0 tx=0x7f585c02c240 comp rx=0 tx=0).stop 2023-10-19T13:00:59.014 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.010+0000 7f5870f4c640 1 -- 172.21.15.57:0/1832683591 shutdown_connections 2023-10-19T13:00:59.014 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.010+0000 7f5870f4c640 1 --2- 172.21.15.57:0/1832683591 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f586c0b0530 0x7f586c0b09c0 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:59.014 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.010+0000 7f5870f4c640 1 --2- 172.21.15.57:0/1832683591 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f586c0afa20 0x7f586c158300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:59.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.010+0000 7f5870f4c640 1 -- 172.21.15.57:0/1832683591 >> 172.21.15.57:0/1832683591 conn(0x7f586c069930 msgr2=0x7f586c069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:59.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.010+0000 7f5870f4c640 1 -- 172.21.15.57:0/1832683591 shutdown_connections 2023-10-19T13:00:59.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.010+0000 7f5870f4c640 1 -- 172.21.15.57:0/1832683591 wait complete. 2023-10-19T13:00:59.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.012+0000 7f5870f4c640 1 Processor -- start 2023-10-19T13:00:59.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.012+0000 7f5870f4c640 1 -- start start 2023-10-19T13:00:59.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.012+0000 7f5870f4c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f586c0afa20 0x7f586c146b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:59.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.012+0000 7f5870f4c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f586c0b0530 0x7f586c145380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:59.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.012+0000 7f5870f4c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f586c0cbd20 0x7f586c1427e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:59.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.012+0000 7f5870f4c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f586c145a60 con 0x7f586c0afa20 2023-10-19T13:00:59.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.012+0000 7f5870f4c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f586c145d40 con 0x7f586c0cbd20 2023-10-19T13:00:59.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.012+0000 7f5870f4c640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f586c146020 con 0x7f586c0b0530 2023-10-19T13:00:59.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.013+0000 7f586a59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f586c0afa20 0x7f586c146b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:59.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.013+0000 7f586a59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f586c0afa20 0x7f586c146b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38042/0 (socket says 172.21.15.57:38042) 2023-10-19T13:00:59.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.013+0000 7f586a59c640 1 -- 172.21.15.57:0/4182110492 learned_addr learned my addr 172.21.15.57:0/4182110492 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:59.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.013+0000 7f586a59c640 1 -- 172.21.15.57:0/4182110492 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f586c0b0530 msgr2=0x7f586c145380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:59.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.013+0000 7f586a59c640 1 --2- 172.21.15.57:0/4182110492 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f586c0b0530 0x7f586c145380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:59.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.013+0000 7f586a59c640 1 -- 172.21.15.57:0/4182110492 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f586c0cbd20 msgr2=0x7f586c1427e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:59.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.013+0000 7f586ad9d640 1 --2- 172.21.15.57:0/4182110492 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f586c0cbd20 0x7f586c1427e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:59.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.013+0000 7f586a59c640 1 --2- 172.21.15.57:0/4182110492 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f586c0cbd20 0x7f586c1427e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:59.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.013+0000 7f586a59c640 1 -- 172.21.15.57:0/4182110492 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f585c009900 con 0x7f586c0afa20 2023-10-19T13:00:59.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.013+0000 7f586a59c640 1 --2- 172.21.15.57:0/4182110492 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f586c0afa20 0x7f586c146b00 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f586400b3f0 tx=0x7f586400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:59.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.013+0000 7f585affd640 1 -- 172.21.15.57:0/4182110492 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5864011830 con 0x7f586c0afa20 2023-10-19T13:00:59.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.013+0000 7f585affd640 1 -- 172.21.15.57:0/4182110492 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f5864011a80 con 0x7f586c0afa20 2023-10-19T13:00:59.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.013+0000 7f5870f4c640 1 -- 172.21.15.57:0/4182110492 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f586c146300 con 0x7f586c0afa20 2023-10-19T13:00:59.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.013+0000 7f585affd640 1 -- 172.21.15.57:0/4182110492 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5864011e90 con 0x7f586c0afa20 2023-10-19T13:00:59.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.013+0000 7f5870f4c640 1 -- 172.21.15.57:0/4182110492 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f586c142d20 con 0x7f586c0afa20 2023-10-19T13:00:59.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.014+0000 7f5870f4c640 1 -- 172.21.15.57:0/4182110492 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f586c146820 con 0x7f586c0afa20 2023-10-19T13:00:59.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.015+0000 7f585affd640 1 -- 172.21.15.57:0/4182110492 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 40) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f586401a400 con 0x7f586c0afa20 2023-10-19T13:00:59.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.015+0000 7f585affd640 1 --2- 172.21.15.57:0/4182110492 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f584c06af60 0x7f584c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:59.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.015+0000 7f585affd640 1 -- 172.21.15.57:0/4182110492 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f58640899e0 con 0x7f586c0afa20 2023-10-19T13:00:59.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.015+0000 7f5869d9b640 1 --2- 172.21.15.57:0/4182110492 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f584c06af60 0x7f584c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:59.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.016+0000 7f5869d9b640 1 --2- 172.21.15.57:0/4182110492 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f584c06af60 0x7f584c06d420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f586c1434c0 tx=0x7f5860005c70 comp rx=0 tx=0).ready entity=mgr.6679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:59.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.019+0000 7f585affd640 1 -- 172.21.15.57:0/4182110492 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f586c146820 con 0x7f586c0afa20 2023-10-19T13:00:59.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.191+0000 7f5870f4c640 1 -- 172.21.15.57:0/4182110492 --> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f586c145eb0 con 0x7f584c06af60 2023-10-19T13:00:59.202 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.200+0000 7f585affd640 1 -- 172.21.15.57:0/4182110492 <== mgr.6679 v2:172.21.15.179:6832/1413162726 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289681 (secure 0 0 0) 0x7f586c145eb0 con 0x7f584c06af60 2023-10-19T13:00:59.206 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:00:59.208 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2023-10-19T13:00:59.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.202+0000 7f5870f4c640 1 -- 172.21.15.57:0/4182110492 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f584c06af60 msgr2=0x7f584c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:59.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.202+0000 7f5870f4c640 1 --2- 172.21.15.57:0/4182110492 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f584c06af60 0x7f584c06d420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f586c1434c0 tx=0x7f5860005c70 comp rx=0 tx=0).stop 2023-10-19T13:00:59.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.202+0000 7f5870f4c640 1 -- 172.21.15.57:0/4182110492 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f586c0afa20 msgr2=0x7f586c146b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:59.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.202+0000 7f5870f4c640 1 --2- 172.21.15.57:0/4182110492 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f586c0afa20 0x7f586c146b00 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f586400b3f0 tx=0x7f586400b8b0 comp rx=0 tx=0).stop 2023-10-19T13:00:59.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.202+0000 7f5870f4c640 1 -- 172.21.15.57:0/4182110492 shutdown_connections 2023-10-19T13:00:59.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.202+0000 7f5870f4c640 1 --2- 172.21.15.57:0/4182110492 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f584c06af60 0x7f584c06d420 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:59.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.202+0000 7f5870f4c640 1 --2- 172.21.15.57:0/4182110492 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f586c0cbd20 0x7f586c1427e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:59.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.202+0000 7f5870f4c640 1 --2- 172.21.15.57:0/4182110492 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f586c0b0530 0x7f586c145380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:59.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.202+0000 7f5870f4c640 1 --2- 172.21.15.57:0/4182110492 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f586c0afa20 0x7f586c146b00 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:59.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.202+0000 7f5870f4c640 1 -- 172.21.15.57:0/4182110492 >> 172.21.15.57:0/4182110492 conn(0x7f586c069930 msgr2=0x7f586c0ad8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:59.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.202+0000 7f5870f4c640 1 -- 172.21.15.57:0/4182110492 shutdown_connections 2023-10-19T13:00:59.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.202+0000 7f5870f4c640 1 -- 172.21.15.57:0/4182110492 wait complete. 2023-10-19T13:00:59.216 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2023-10-19T13:00:58.082904+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":674,"num_read_kb":1321,"num_write":319,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":219,"ondisk_log_size":219,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":157,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":335204,"kb_used_data":10788,"kb_used_omap":12,"kb_used_meta":324275,"kb_avail":754639516,"statfs":{"total":773094113280,"available":772750864384,"internally_reserved":0,"allocated":11046912,"data_stored":6092505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":332058169},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001047"},"pg_stats":[{"pgid":"12.3f","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289249+0000","last_change":"2023-10-19T13:00:56.233432+0000","last_active":"2023-10-19T13:00:57.289249+0000","last_peered":"2023-10-19T13:00:57.289249+0000","last_clean":"2023-10-19T13:00:57.289249+0000","last_became_active":"2023-10-19T13:00:56.233030+0000","last_became_peered":"2023-10-19T13:00:56.233030+0000","last_unstale":"2023-10-19T13:00:57.289249+0000","last_undegraded":"2023-10-19T13:00:57.289249+0000","last_fullsized":"2023-10-19T13:00:57.289249+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:12:00.991308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3e","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289472+0000","last_change":"2023-10-19T13:00:56.214716+0000","last_active":"2023-10-19T13:00:57.289472+0000","last_peered":"2023-10-19T13:00:57.289472+0000","last_clean":"2023-10-19T13:00:57.289472+0000","last_became_active":"2023-10-19T13:00:56.214379+0000","last_became_peered":"2023-10-19T13:00:56.214379+0000","last_unstale":"2023-10-19T13:00:57.289472+0000","last_undegraded":"2023-10-19T13:00:57.289472+0000","last_fullsized":"2023-10-19T13:00:57.289472+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:23:42.593585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.3d","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289480+0000","last_change":"2023-10-19T13:00:56.230243+0000","last_active":"2023-10-19T13:00:57.289480+0000","last_peered":"2023-10-19T13:00:57.289480+0000","last_clean":"2023-10-19T13:00:57.289480+0000","last_became_active":"2023-10-19T13:00:56.229904+0000","last_became_peered":"2023-10-19T13:00:56.229904+0000","last_unstale":"2023-10-19T13:00:57.289480+0000","last_undegraded":"2023-10-19T13:00:57.289480+0000","last_fullsized":"2023-10-19T13:00:57.289480+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:10:14.580813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3c","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.210319+0000","last_change":"2023-10-19T13:00:56.229454+0000","last_active":"2023-10-19T13:00:57.210319+0000","last_peered":"2023-10-19T13:00:57.210319+0000","last_clean":"2023-10-19T13:00:57.210319+0000","last_became_active":"2023-10-19T13:00:56.229188+0000","last_became_peered":"2023-10-19T13:00:56.229188+0000","last_unstale":"2023-10-19T13:00:57.210319+0000","last_undegraded":"2023-10-19T13:00:57.210319+0000","last_fullsized":"2023-10-19T13:00:57.210319+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:25:33.751644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3b","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.209004+0000","last_change":"2023-10-19T13:00:56.229151+0000","last_active":"2023-10-19T13:00:57.209004+0000","last_peered":"2023-10-19T13:00:57.209004+0000","last_clean":"2023-10-19T13:00:57.209004+0000","last_became_active":"2023-10-19T13:00:56.228845+0000","last_became_peered":"2023-10-19T13:00:56.228845+0000","last_unstale":"2023-10-19T13:00:57.209004+0000","last_undegraded":"2023-10-19T13:00:57.209004+0000","last_fullsized":"2023-10-19T13:00:57.209004+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:58:56.570918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.3a","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.290067+0000","last_change":"2023-10-19T13:00:56.233538+0000","last_active":"2023-10-19T13:00:57.290067+0000","last_peered":"2023-10-19T13:00:57.290067+0000","last_clean":"2023-10-19T13:00:57.290067+0000","last_became_active":"2023-10-19T13:00:56.233274+0000","last_became_peered":"2023-10-19T13:00:56.233274+0000","last_unstale":"2023-10-19T13:00:57.290067+0000","last_undegraded":"2023-10-19T13:00:57.290067+0000","last_fullsized":"2023-10-19T13:00:57.290067+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:59:59.026551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.39","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289571+0000","last_change":"2023-10-19T13:00:56.230649+0000","last_active":"2023-10-19T13:00:57.289571+0000","last_peered":"2023-10-19T13:00:57.289571+0000","last_clean":"2023-10-19T13:00:57.289571+0000","last_became_active":"2023-10-19T13:00:56.230364+0000","last_became_peered":"2023-10-19T13:00:56.230364+0000","last_unstale":"2023-10-19T13:00:57.289571+0000","last_undegraded":"2023-10-19T13:00:57.289571+0000","last_fullsized":"2023-10-19T13:00:57.289571+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:03:01.128637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.38","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.208747+0000","last_change":"2023-10-19T13:00:56.229391+0000","last_active":"2023-10-19T13:00:57.208747+0000","last_peered":"2023-10-19T13:00:57.208747+0000","last_clean":"2023-10-19T13:00:57.208747+0000","last_became_active":"2023-10-19T13:00:56.229119+0000","last_became_peered":"2023-10-19T13:00:56.229119+0000","last_unstale":"2023-10-19T13:00:57.208747+0000","last_undegraded":"2023-10-19T13:00:57.208747+0000","last_fullsized":"2023-10-19T13:00:57.208747+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:35:51.543065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.37","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.210571+0000","last_change":"2023-10-19T13:00:56.233060+0000","last_active":"2023-10-19T13:00:57.210571+0000","last_peered":"2023-10-19T13:00:57.210571+0000","last_clean":"2023-10-19T13:00:57.210571+0000","last_became_active":"2023-10-19T13:00:56.232791+0000","last_became_peered":"2023-10-19T13:00:56.232791+0000","last_unstale":"2023-10-19T13:00:57.210571+0000","last_undegraded":"2023-10-19T13:00:57.210571+0000","last_fullsized":"2023-10-19T13:00:57.210571+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:47:54.774879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.36","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.210054+0000","last_change":"2023-10-19T13:00:56.230825+0000","last_active":"2023-10-19T13:00:57.210054+0000","last_peered":"2023-10-19T13:00:57.210054+0000","last_clean":"2023-10-19T13:00:57.210054+0000","last_became_active":"2023-10-19T13:00:56.230569+0000","last_became_peered":"2023-10-19T13:00:56.230569+0000","last_unstale":"2023-10-19T13:00:57.210054+0000","last_undegraded":"2023-10-19T13:00:57.210054+0000","last_fullsized":"2023-10-19T13:00:57.210054+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:22:40.229902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.35","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.290236+0000","last_change":"2023-10-19T13:00:56.216939+0000","last_active":"2023-10-19T13:00:57.290236+0000","last_peered":"2023-10-19T13:00:57.290236+0000","last_clean":"2023-10-19T13:00:57.290236+0000","last_became_active":"2023-10-19T13:00:56.216579+0000","last_became_peered":"2023-10-19T13:00:56.216579+0000","last_unstale":"2023-10-19T13:00:57.290236+0000","last_undegraded":"2023-10-19T13:00:57.290236+0000","last_fullsized":"2023-10-19T13:00:57.290236+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:51:40.006891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.34","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.210446+0000","last_change":"2023-10-19T13:00:56.222785+0000","last_active":"2023-10-19T13:00:57.210446+0000","last_peered":"2023-10-19T13:00:57.210446+0000","last_clean":"2023-10-19T13:00:57.210446+0000","last_became_active":"2023-10-19T13:00:56.222553+0000","last_became_peered":"2023-10-19T13:00:56.222553+0000","last_unstale":"2023-10-19T13:00:57.210446+0000","last_undegraded":"2023-10-19T13:00:57.210446+0000","last_fullsized":"2023-10-19T13:00:57.210446+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:41:18.979132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.33","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.209083+0000","last_change":"2023-10-19T13:00:56.226771+0000","last_active":"2023-10-19T13:00:57.209083+0000","last_peered":"2023-10-19T13:00:57.209083+0000","last_clean":"2023-10-19T13:00:57.209083+0000","last_became_active":"2023-10-19T13:00:56.226479+0000","last_became_peered":"2023-10-19T13:00:56.226479+0000","last_unstale":"2023-10-19T13:00:57.209083+0000","last_undegraded":"2023-10-19T13:00:57.209083+0000","last_fullsized":"2023-10-19T13:00:57.209083+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:52:09.841397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.32","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.290189+0000","last_change":"2023-10-19T13:00:56.233427+0000","last_active":"2023-10-19T13:00:57.290189+0000","last_peered":"2023-10-19T13:00:57.290189+0000","last_clean":"2023-10-19T13:00:57.290189+0000","last_became_active":"2023-10-19T13:00:56.233116+0000","last_became_peered":"2023-10-19T13:00:56.233116+0000","last_unstale":"2023-10-19T13:00:57.290189+0000","last_undegraded":"2023-10-19T13:00:57.290189+0000","last_fullsized":"2023-10-19T13:00:57.290189+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:35:55.138938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.31","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289012+0000","last_change":"2023-10-19T13:00:56.216064+0000","last_active":"2023-10-19T13:00:57.289012+0000","last_peered":"2023-10-19T13:00:57.289012+0000","last_clean":"2023-10-19T13:00:57.289012+0000","last_became_active":"2023-10-19T13:00:56.215751+0000","last_became_peered":"2023-10-19T13:00:56.215751+0000","last_unstale":"2023-10-19T13:00:57.289012+0000","last_undegraded":"2023-10-19T13:00:57.289012+0000","last_fullsized":"2023-10-19T13:00:57.289012+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:12:00.991308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.30","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.288930+0000","last_change":"2023-10-19T13:00:56.232927+0000","last_active":"2023-10-19T13:00:57.288930+0000","last_peered":"2023-10-19T13:00:57.288930+0000","last_clean":"2023-10-19T13:00:57.288930+0000","last_became_active":"2023-10-19T13:00:56.231096+0000","last_became_peered":"2023-10-19T13:00:56.231096+0000","last_unstale":"2023-10-19T13:00:57.288930+0000","last_undegraded":"2023-10-19T13:00:57.288930+0000","last_fullsized":"2023-10-19T13:00:57.288930+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:12:38.285374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.2f","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.209316+0000","last_change":"2023-10-19T13:00:56.238408+0000","last_active":"2023-10-19T13:00:57.209316+0000","last_peered":"2023-10-19T13:00:57.209316+0000","last_clean":"2023-10-19T13:00:57.209316+0000","last_became_active":"2023-10-19T13:00:56.238142+0000","last_became_peered":"2023-10-19T13:00:56.238142+0000","last_unstale":"2023-10-19T13:00:57.209316+0000","last_undegraded":"2023-10-19T13:00:57.209316+0000","last_fullsized":"2023-10-19T13:00:57.209316+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:46:55.376338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2e","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289920+0000","last_change":"2023-10-19T13:00:56.232174+0000","last_active":"2023-10-19T13:00:57.289920+0000","last_peered":"2023-10-19T13:00:57.289920+0000","last_clean":"2023-10-19T13:00:57.289920+0000","last_became_active":"2023-10-19T13:00:56.231904+0000","last_became_peered":"2023-10-19T13:00:56.231904+0000","last_unstale":"2023-10-19T13:00:57.289920+0000","last_undegraded":"2023-10-19T13:00:57.289920+0000","last_fullsized":"2023-10-19T13:00:57.289920+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:20:06.228901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2d","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289727+0000","last_change":"2023-10-19T13:00:56.235934+0000","last_active":"2023-10-19T13:00:57.289727+0000","last_peered":"2023-10-19T13:00:57.289727+0000","last_clean":"2023-10-19T13:00:57.289727+0000","last_became_active":"2023-10-19T13:00:56.235640+0000","last_became_peered":"2023-10-19T13:00:56.235640+0000","last_unstale":"2023-10-19T13:00:57.289727+0000","last_undegraded":"2023-10-19T13:00:57.289727+0000","last_fullsized":"2023-10-19T13:00:57.289727+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:10:09.955885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2c","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.209168+0000","last_change":"2023-10-19T13:00:56.216977+0000","last_active":"2023-10-19T13:00:57.209168+0000","last_peered":"2023-10-19T13:00:57.209168+0000","last_clean":"2023-10-19T13:00:57.209168+0000","last_became_active":"2023-10-19T13:00:56.216572+0000","last_became_peered":"2023-10-19T13:00:56.216572+0000","last_unstale":"2023-10-19T13:00:57.209168+0000","last_undegraded":"2023-10-19T13:00:57.209168+0000","last_fullsized":"2023-10-19T13:00:57.209168+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:53:29.692700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2b","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.210552+0000","last_change":"2023-10-19T13:00:56.230703+0000","last_active":"2023-10-19T13:00:57.210552+0000","last_peered":"2023-10-19T13:00:57.210552+0000","last_clean":"2023-10-19T13:00:57.210552+0000","last_became_active":"2023-10-19T13:00:56.230444+0000","last_became_peered":"2023-10-19T13:00:56.230444+0000","last_unstale":"2023-10-19T13:00:57.210552+0000","last_undegraded":"2023-10-19T13:00:57.210552+0000","last_fullsized":"2023-10-19T13:00:57.210552+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:04:40.134114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.2a","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.208797+0000","last_change":"2023-10-19T13:00:56.222547+0000","last_active":"2023-10-19T13:00:57.208797+0000","last_peered":"2023-10-19T13:00:57.208797+0000","last_clean":"2023-10-19T13:00:57.208797+0000","last_became_active":"2023-10-19T13:00:56.222285+0000","last_became_peered":"2023-10-19T13:00:56.222285+0000","last_unstale":"2023-10-19T13:00:57.208797+0000","last_undegraded":"2023-10-19T13:00:57.208797+0000","last_fullsized":"2023-10-19T13:00:57.208797+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:07:24.504133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.29","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.210666+0000","last_change":"2023-10-19T13:00:56.228216+0000","last_active":"2023-10-19T13:00:57.210666+0000","last_peered":"2023-10-19T13:00:57.210666+0000","last_clean":"2023-10-19T13:00:57.210666+0000","last_became_active":"2023-10-19T13:00:56.227905+0000","last_became_peered":"2023-10-19T13:00:56.227905+0000","last_unstale":"2023-10-19T13:00:57.210666+0000","last_undegraded":"2023-10-19T13:00:57.210666+0000","last_fullsized":"2023-10-19T13:00:57.210666+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:21:47.277500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.28","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289030+0000","last_change":"2023-10-19T13:00:56.225141+0000","last_active":"2023-10-19T13:00:57.289030+0000","last_peered":"2023-10-19T13:00:57.289030+0000","last_clean":"2023-10-19T13:00:57.289030+0000","last_became_active":"2023-10-19T13:00:56.224875+0000","last_became_peered":"2023-10-19T13:00:56.224875+0000","last_unstale":"2023-10-19T13:00:57.289030+0000","last_undegraded":"2023-10-19T13:00:57.289030+0000","last_fullsized":"2023-10-19T13:00:57.289030+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:03:05.773151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.27","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289803+0000","last_change":"2023-10-19T13:00:56.223381+0000","last_active":"2023-10-19T13:00:57.289803+0000","last_peered":"2023-10-19T13:00:57.289803+0000","last_clean":"2023-10-19T13:00:57.289803+0000","last_became_active":"2023-10-19T13:00:56.223093+0000","last_became_peered":"2023-10-19T13:00:56.223093+0000","last_unstale":"2023-10-19T13:00:57.289803+0000","last_undegraded":"2023-10-19T13:00:57.289803+0000","last_fullsized":"2023-10-19T13:00:57.289803+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:37:15.788197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.26","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.210216+0000","last_change":"2023-10-19T13:00:56.222205+0000","last_active":"2023-10-19T13:00:57.210216+0000","last_peered":"2023-10-19T13:00:57.210216+0000","last_clean":"2023-10-19T13:00:57.210216+0000","last_became_active":"2023-10-19T13:00:56.221861+0000","last_became_peered":"2023-10-19T13:00:56.221861+0000","last_unstale":"2023-10-19T13:00:57.210216+0000","last_undegraded":"2023-10-19T13:00:57.210216+0000","last_fullsized":"2023-10-19T13:00:57.210216+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:46:21.162186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.25","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.288915+0000","last_change":"2023-10-19T13:00:56.229084+0000","last_active":"2023-10-19T13:00:57.288915+0000","last_peered":"2023-10-19T13:00:57.288915+0000","last_clean":"2023-10-19T13:00:57.288915+0000","last_became_active":"2023-10-19T13:00:56.228827+0000","last_became_peered":"2023-10-19T13:00:56.228827+0000","last_unstale":"2023-10-19T13:00:57.288915+0000","last_undegraded":"2023-10-19T13:00:57.288915+0000","last_fullsized":"2023-10-19T13:00:57.288915+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:04:47.678215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.24","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.209160+0000","last_change":"2023-10-19T13:00:56.214805+0000","last_active":"2023-10-19T13:00:57.209160+0000","last_peered":"2023-10-19T13:00:57.209160+0000","last_clean":"2023-10-19T13:00:57.209160+0000","last_became_active":"2023-10-19T13:00:56.214391+0000","last_became_peered":"2023-10-19T13:00:56.214391+0000","last_unstale":"2023-10-19T13:00:57.209160+0000","last_undegraded":"2023-10-19T13:00:57.209160+0000","last_fullsized":"2023-10-19T13:00:57.209160+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:26:08.320410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.23","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289332+0000","last_change":"2023-10-19T13:00:56.231492+0000","last_active":"2023-10-19T13:00:57.289332+0000","last_peered":"2023-10-19T13:00:57.289332+0000","last_clean":"2023-10-19T13:00:57.289332+0000","last_became_active":"2023-10-19T13:00:56.231206+0000","last_became_peered":"2023-10-19T13:00:56.231206+0000","last_unstale":"2023-10-19T13:00:57.289332+0000","last_undegraded":"2023-10-19T13:00:57.289332+0000","last_fullsized":"2023-10-19T13:00:57.289332+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:02:46.194505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.22","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289007+0000","last_change":"2023-10-19T13:00:56.230353+0000","last_active":"2023-10-19T13:00:57.289007+0000","last_peered":"2023-10-19T13:00:57.289007+0000","last_clean":"2023-10-19T13:00:57.289007+0000","last_became_active":"2023-10-19T13:00:56.230030+0000","last_became_peered":"2023-10-19T13:00:56.230030+0000","last_unstale":"2023-10-19T13:00:57.289007+0000","last_undegraded":"2023-10-19T13:00:57.289007+0000","last_fullsized":"2023-10-19T13:00:57.289007+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:54:53.966112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.21","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289300+0000","last_change":"2023-10-19T13:00:56.235310+0000","last_active":"2023-10-19T13:00:57.289300+0000","last_peered":"2023-10-19T13:00:57.289300+0000","last_clean":"2023-10-19T13:00:57.289300+0000","last_became_active":"2023-10-19T13:00:56.234965+0000","last_became_peered":"2023-10-19T13:00:56.234965+0000","last_unstale":"2023-10-19T13:00:57.289300+0000","last_undegraded":"2023-10-19T13:00:57.289300+0000","last_fullsized":"2023-10-19T13:00:57.289300+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:41:21.345551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.20","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.288924+0000","last_change":"2023-10-19T13:00:56.235826+0000","last_active":"2023-10-19T13:00:57.288924+0000","last_peered":"2023-10-19T13:00:57.288924+0000","last_clean":"2023-10-19T13:00:57.288924+0000","last_became_active":"2023-10-19T13:00:56.235340+0000","last_became_peered":"2023-10-19T13:00:56.235340+0000","last_unstale":"2023-10-19T13:00:57.288924+0000","last_undegraded":"2023-10-19T13:00:57.288924+0000","last_fullsized":"2023-10-19T13:00:57.288924+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:17:54.044666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.288979+0000","last_change":"2023-10-19T13:00:56.229351+0000","last_active":"2023-10-19T13:00:57.288979+0000","last_peered":"2023-10-19T13:00:57.288979+0000","last_clean":"2023-10-19T13:00:57.288979+0000","last_became_active":"2023-10-19T13:00:56.229094+0000","last_became_peered":"2023-10-19T13:00:56.229094+0000","last_unstale":"2023-10-19T13:00:57.288979+0000","last_undegraded":"2023-10-19T13:00:57.288979+0000","last_fullsized":"2023-10-19T13:00:57.288979+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:46:48.130929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1f","version":"99'4","reported_seq":16,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.209350+0000","last_change":"2023-10-19T13:00:55.202919+0000","last_active":"2023-10-19T13:00:57.209350+0000","last_peered":"2023-10-19T13:00:57.209350+0000","last_clean":"2023-10-19T13:00:57.209350+0000","last_became_active":"2023-10-19T13:00:55.202608+0000","last_became_peered":"2023-10-19T13:00:55.202608+0000","last_unstale":"2023-10-19T13:00:57.209350+0000","last_undegraded":"2023-10-19T13:00:57.209350+0000","last_fullsized":"2023-10-19T13:00:57.209350+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_clean_scrub_stamp":"2023-10-19T13:00:54.175433+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:11:19.819288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289081+0000","last_change":"2023-10-19T13:00:56.215503+0000","last_active":"2023-10-19T13:00:57.289081+0000","last_peered":"2023-10-19T13:00:57.289081+0000","last_clean":"2023-10-19T13:00:57.289081+0000","last_became_active":"2023-10-19T13:00:56.214927+0000","last_became_peered":"2023-10-19T13:00:56.214927+0000","last_unstale":"2023-10-19T13:00:57.289081+0000","last_undegraded":"2023-10-19T13:00:57.289081+0000","last_fullsized":"2023-10-19T13:00:57.289081+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:07:30.083579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"99'1","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289093+0000","last_change":"2023-10-19T13:00:55.201299+0000","last_active":"2023-10-19T13:00:57.289093+0000","last_peered":"2023-10-19T13:00:57.289093+0000","last_clean":"2023-10-19T13:00:57.289093+0000","last_became_active":"2023-10-19T13:00:55.201009+0000","last_became_peered":"2023-10-19T13:00:55.201009+0000","last_unstale":"2023-10-19T13:00:57.289093+0000","last_undegraded":"2023-10-19T13:00:57.289093+0000","last_fullsized":"2023-10-19T13:00:57.289093+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_clean_scrub_stamp":"2023-10-19T13:00:54.175433+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:37:20.825852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.210709+0000","last_change":"2023-10-19T13:00:56.235410+0000","last_active":"2023-10-19T13:00:57.210709+0000","last_peered":"2023-10-19T13:00:57.210709+0000","last_clean":"2023-10-19T13:00:57.210709+0000","last_became_active":"2023-10-19T13:00:56.233880+0000","last_became_peered":"2023-10-19T13:00:56.233880+0000","last_unstale":"2023-10-19T13:00:57.210709+0000","last_undegraded":"2023-10-19T13:00:57.210709+0000","last_fullsized":"2023-10-19T13:00:57.210709+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:03:01.128637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"99'2","reported_seq":14,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289055+0000","last_change":"2023-10-19T13:00:55.207421+0000","last_active":"2023-10-19T13:00:57.289055+0000","last_peered":"2023-10-19T13:00:57.289055+0000","last_clean":"2023-10-19T13:00:57.289055+0000","last_became_active":"2023-10-19T13:00:55.207085+0000","last_became_peered":"2023-10-19T13:00:55.207085+0000","last_unstale":"2023-10-19T13:00:57.289055+0000","last_undegraded":"2023-10-19T13:00:57.289055+0000","last_fullsized":"2023-10-19T13:00:57.289055+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_clean_scrub_stamp":"2023-10-19T13:00:54.175433+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:19:24.124448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.210717+0000","last_change":"2023-10-19T13:00:56.230156+0000","last_active":"2023-10-19T13:00:57.210717+0000","last_peered":"2023-10-19T13:00:57.210717+0000","last_clean":"2023-10-19T13:00:57.210717+0000","last_became_active":"2023-10-19T13:00:56.229837+0000","last_became_peered":"2023-10-19T13:00:56.229837+0000","last_unstale":"2023-10-19T13:00:57.210717+0000","last_undegraded":"2023-10-19T13:00:57.210717+0000","last_fullsized":"2023-10-19T13:00:57.210717+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:25:26.972552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1c","version":"99'2","reported_seq":14,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289683+0000","last_change":"2023-10-19T13:00:55.200397+0000","last_active":"2023-10-19T13:00:57.289683+0000","last_peered":"2023-10-19T13:00:57.289683+0000","last_clean":"2023-10-19T13:00:57.289683+0000","last_became_active":"2023-10-19T13:00:55.200138+0000","last_became_peered":"2023-10-19T13:00:55.200138+0000","last_unstale":"2023-10-19T13:00:57.289683+0000","last_undegraded":"2023-10-19T13:00:57.289683+0000","last_fullsized":"2023-10-19T13:00:57.289683+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_clean_scrub_stamp":"2023-10-19T13:00:54.175433+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:04:39.128200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.290121+0000","last_change":"2023-10-19T13:00:56.216180+0000","last_active":"2023-10-19T13:00:57.290121+0000","last_peered":"2023-10-19T13:00:57.290121+0000","last_clean":"2023-10-19T13:00:57.290121+0000","last_became_active":"2023-10-19T13:00:56.215725+0000","last_became_peered":"2023-10-19T13:00:56.215725+0000","last_unstale":"2023-10-19T13:00:57.290121+0000","last_undegraded":"2023-10-19T13:00:57.290121+0000","last_fullsized":"2023-10-19T13:00:57.290121+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:41:50.788069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"99'1","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.210799+0000","last_change":"2023-10-19T13:00:55.200646+0000","last_active":"2023-10-19T13:00:57.210799+0000","last_peered":"2023-10-19T13:00:57.210799+0000","last_clean":"2023-10-19T13:00:57.210799+0000","last_became_active":"2023-10-19T13:00:55.200395+0000","last_became_peered":"2023-10-19T13:00:55.200395+0000","last_unstale":"2023-10-19T13:00:57.210799+0000","last_undegraded":"2023-10-19T13:00:57.210799+0000","last_fullsized":"2023-10-19T13:00:57.210799+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_clean_scrub_stamp":"2023-10-19T13:00:54.175433+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:33:54.831822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.290120+0000","last_change":"2023-10-19T13:00:56.228533+0000","last_active":"2023-10-19T13:00:57.290120+0000","last_peered":"2023-10-19T13:00:57.290120+0000","last_clean":"2023-10-19T13:00:57.290120+0000","last_became_active":"2023-10-19T13:00:56.228198+0000","last_became_peered":"2023-10-19T13:00:56.228198+0000","last_unstale":"2023-10-19T13:00:57.290120+0000","last_undegraded":"2023-10-19T13:00:57.290120+0000","last_fullsized":"2023-10-19T13:00:57.290120+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:49:31.042667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"99'1","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.209888+0000","last_change":"2023-10-19T13:00:55.192914+0000","last_active":"2023-10-19T13:00:57.209888+0000","last_peered":"2023-10-19T13:00:57.209888+0000","last_clean":"2023-10-19T13:00:57.209888+0000","last_became_active":"2023-10-19T13:00:55.192577+0000","last_became_peered":"2023-10-19T13:00:55.192577+0000","last_unstale":"2023-10-19T13:00:57.209888+0000","last_undegraded":"2023-10-19T13:00:57.209888+0000","last_fullsized":"2023-10-19T13:00:57.209888+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_clean_scrub_stamp":"2023-10-19T13:00:54.175433+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:19:24.124448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.210511+0000","last_change":"2023-10-19T13:00:56.235761+0000","last_active":"2023-10-19T13:00:57.210511+0000","last_peered":"2023-10-19T13:00:57.210511+0000","last_clean":"2023-10-19T13:00:57.210511+0000","last_became_active":"2023-10-19T13:00:56.235508+0000","last_became_peered":"2023-10-19T13:00:56.235508+0000","last_unstale":"2023-10-19T13:00:57.210511+0000","last_undegraded":"2023-10-19T13:00:57.210511+0000","last_fullsized":"2023-10-19T13:00:57.210511+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:52:32.379639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":12,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289167+0000","last_change":"2023-10-19T13:00:55.205226+0000","last_active":"2023-10-19T13:00:57.289167+0000","last_peered":"2023-10-19T13:00:57.289167+0000","last_clean":"2023-10-19T13:00:57.289167+0000","last_became_active":"2023-10-19T13:00:55.204316+0000","last_became_peered":"2023-10-19T13:00:55.204316+0000","last_unstale":"2023-10-19T13:00:57.289167+0000","last_undegraded":"2023-10-19T13:00:57.289167+0000","last_fullsized":"2023-10-19T13:00:57.289167+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_clean_scrub_stamp":"2023-10-19T13:00:54.175433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:04:46.672302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.290001+0000","last_change":"2023-10-19T13:00:56.231969+0000","last_active":"2023-10-19T13:00:57.290001+0000","last_peered":"2023-10-19T13:00:57.290001+0000","last_clean":"2023-10-19T13:00:57.290001+0000","last_became_active":"2023-10-19T13:00:56.231658+0000","last_became_peered":"2023-10-19T13:00:56.231658+0000","last_unstale":"2023-10-19T13:00:57.290001+0000","last_undegraded":"2023-10-19T13:00:57.290001+0000","last_fullsized":"2023-10-19T13:00:57.290001+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:54:37.924182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":12,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.208757+0000","last_change":"2023-10-19T13:00:55.200476+0000","last_active":"2023-10-19T13:00:57.208757+0000","last_peered":"2023-10-19T13:00:57.208757+0000","last_clean":"2023-10-19T13:00:57.208757+0000","last_became_active":"2023-10-19T13:00:55.200204+0000","last_became_peered":"2023-10-19T13:00:55.200204+0000","last_unstale":"2023-10-19T13:00:57.208757+0000","last_undegraded":"2023-10-19T13:00:57.208757+0000","last_fullsized":"2023-10-19T13:00:57.208757+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_clean_scrub_stamp":"2023-10-19T13:00:54.175433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:57:00.285741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289004+0000","last_change":"2023-10-19T13:00:56.235688+0000","last_active":"2023-10-19T13:00:57.289004+0000","last_peered":"2023-10-19T13:00:57.289004+0000","last_clean":"2023-10-19T13:00:57.289004+0000","last_became_active":"2023-10-19T13:00:56.235370+0000","last_became_peered":"2023-10-19T13:00:56.235370+0000","last_unstale":"2023-10-19T13:00:57.289004+0000","last_undegraded":"2023-10-19T13:00:57.289004+0000","last_fullsized":"2023-10-19T13:00:57.289004+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:56:34.119416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":12,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289245+0000","last_change":"2023-10-19T13:00:55.200716+0000","last_active":"2023-10-19T13:00:57.289245+0000","last_peered":"2023-10-19T13:00:57.289245+0000","last_clean":"2023-10-19T13:00:57.289245+0000","last_became_active":"2023-10-19T13:00:55.200457+0000","last_became_peered":"2023-10-19T13:00:55.200457+0000","last_unstale":"2023-10-19T13:00:57.289245+0000","last_undegraded":"2023-10-19T13:00:57.289245+0000","last_fullsized":"2023-10-19T13:00:57.289245+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_clean_scrub_stamp":"2023-10-19T13:00:54.175433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:18:53.973512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.210415+0000","last_change":"2023-10-19T13:00:56.229746+0000","last_active":"2023-10-19T13:00:57.210415+0000","last_peered":"2023-10-19T13:00:57.210415+0000","last_clean":"2023-10-19T13:00:57.210415+0000","last_became_active":"2023-10-19T13:00:56.229483+0000","last_became_peered":"2023-10-19T13:00:56.229483+0000","last_unstale":"2023-10-19T13:00:57.210415+0000","last_undegraded":"2023-10-19T13:00:57.210415+0000","last_fullsized":"2023-10-19T13:00:57.210415+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:31:05.271554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":12,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.210314+0000","last_change":"2023-10-19T13:00:55.201725+0000","last_active":"2023-10-19T13:00:57.210314+0000","last_peered":"2023-10-19T13:00:57.210314+0000","last_clean":"2023-10-19T13:00:57.210314+0000","last_became_active":"2023-10-19T13:00:55.201444+0000","last_became_peered":"2023-10-19T13:00:55.201444+0000","last_unstale":"2023-10-19T13:00:57.210314+0000","last_undegraded":"2023-10-19T13:00:57.210314+0000","last_fullsized":"2023-10-19T13:00:57.210314+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_clean_scrub_stamp":"2023-10-19T13:00:54.175433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:18:53.973512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.290282+0000","last_change":"2023-10-19T13:00:56.225547+0000","last_active":"2023-10-19T13:00:57.290282+0000","last_peered":"2023-10-19T13:00:57.290282+0000","last_clean":"2023-10-19T13:00:57.290282+0000","last_became_active":"2023-10-19T13:00:56.225270+0000","last_became_peered":"2023-10-19T13:00:56.225270+0000","last_unstale":"2023-10-19T13:00:57.290282+0000","last_undegraded":"2023-10-19T13:00:57.290282+0000","last_fullsized":"2023-10-19T13:00:57.290282+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:58:43.686419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"99'3","reported_seq":15,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.209326+0000","last_change":"2023-10-19T13:00:55.207091+0000","last_active":"2023-10-19T13:00:57.209326+0000","last_peered":"2023-10-19T13:00:57.209326+0000","last_clean":"2023-10-19T13:00:57.209326+0000","last_became_active":"2023-10-19T13:00:55.206828+0000","last_became_peered":"2023-10-19T13:00:55.206828+0000","last_unstale":"2023-10-19T13:00:57.209326+0000","last_undegraded":"2023-10-19T13:00:57.209326+0000","last_fullsized":"2023-10-19T13:00:57.209326+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_clean_scrub_stamp":"2023-10-19T13:00:54.175433+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:15:16.920840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289516+0000","last_change":"2023-10-19T13:00:56.224907+0000","last_active":"2023-10-19T13:00:57.289516+0000","last_peered":"2023-10-19T13:00:57.289516+0000","last_clean":"2023-10-19T13:00:57.289516+0000","last_became_active":"2023-10-19T13:00:56.224629+0000","last_became_peered":"2023-10-19T13:00:56.224629+0000","last_unstale":"2023-10-19T13:00:57.289516+0000","last_undegraded":"2023-10-19T13:00:57.289516+0000","last_fullsized":"2023-10-19T13:00:57.289516+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:12:40.484752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"99'6","reported_seq":18,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289725+0000","last_change":"2023-10-19T13:00:55.198821+0000","last_active":"2023-10-19T13:00:57.289725+0000","last_peered":"2023-10-19T13:00:57.289725+0000","last_clean":"2023-10-19T13:00:57.289725+0000","last_became_active":"2023-10-19T13:00:55.198475+0000","last_became_peered":"2023-10-19T13:00:55.198475+0000","last_unstale":"2023-10-19T13:00:57.289725+0000","last_undegraded":"2023-10-19T13:00:57.289725+0000","last_fullsized":"2023-10-19T13:00:57.289725+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_clean_scrub_stamp":"2023-10-19T13:00:54.175433+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:47:53.768966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.210567+0000","last_change":"2023-10-19T13:00:56.221947+0000","last_active":"2023-10-19T13:00:57.210567+0000","last_peered":"2023-10-19T13:00:57.210567+0000","last_clean":"2023-10-19T13:00:57.210567+0000","last_became_active":"2023-10-19T13:00:56.221660+0000","last_became_peered":"2023-10-19T13:00:56.221660+0000","last_unstale":"2023-10-19T13:00:57.210567+0000","last_undegraded":"2023-10-19T13:00:57.210567+0000","last_fullsized":"2023-10-19T13:00:57.210567+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:04:47.678215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.13","version":"99'2","reported_seq":14,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.210007+0000","last_change":"2023-10-19T13:00:55.193949+0000","last_active":"2023-10-19T13:00:57.210007+0000","last_peered":"2023-10-19T13:00:57.210007+0000","last_clean":"2023-10-19T13:00:57.210007+0000","last_became_active":"2023-10-19T13:00:55.193676+0000","last_became_peered":"2023-10-19T13:00:55.193676+0000","last_unstale":"2023-10-19T13:00:57.210007+0000","last_undegraded":"2023-10-19T13:00:57.210007+0000","last_fullsized":"2023-10-19T13:00:57.210007+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_clean_scrub_stamp":"2023-10-19T13:00:54.175433+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:37:20.825852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289806+0000","last_change":"2023-10-19T13:00:56.231757+0000","last_active":"2023-10-19T13:00:57.289806+0000","last_peered":"2023-10-19T13:00:57.289806+0000","last_clean":"2023-10-19T13:00:57.289806+0000","last_became_active":"2023-10-19T13:00:56.231488+0000","last_became_peered":"2023-10-19T13:00:56.231488+0000","last_unstale":"2023-10-19T13:00:57.289806+0000","last_undegraded":"2023-10-19T13:00:57.289806+0000","last_fullsized":"2023-10-19T13:00:57.289806+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:46:48.130929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":12,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289087+0000","last_change":"2023-10-19T13:00:55.202887+0000","last_active":"2023-10-19T13:00:57.289087+0000","last_peered":"2023-10-19T13:00:57.289087+0000","last_clean":"2023-10-19T13:00:57.289087+0000","last_became_active":"2023-10-19T13:00:55.202560+0000","last_became_peered":"2023-10-19T13:00:55.202560+0000","last_unstale":"2023-10-19T13:00:57.289087+0000","last_undegraded":"2023-10-19T13:00:57.289087+0000","last_fullsized":"2023-10-19T13:00:57.289087+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_clean_scrub_stamp":"2023-10-19T13:00:54.175433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:25:32.745730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289882+0000","last_change":"2023-10-19T13:00:56.235887+0000","last_active":"2023-10-19T13:00:57.289882+0000","last_peered":"2023-10-19T13:00:57.289882+0000","last_clean":"2023-10-19T13:00:57.289882+0000","last_became_active":"2023-10-19T13:00:56.235636+0000","last_became_peered":"2023-10-19T13:00:56.235636+0000","last_unstale":"2023-10-19T13:00:57.289882+0000","last_undegraded":"2023-10-19T13:00:57.289882+0000","last_fullsized":"2023-10-19T13:00:57.289882+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:10:14.580813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":12,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.208793+0000","last_change":"2023-10-19T13:00:55.201275+0000","last_active":"2023-10-19T13:00:57.208793+0000","last_peered":"2023-10-19T13:00:57.208793+0000","last_clean":"2023-10-19T13:00:57.208793+0000","last_became_active":"2023-10-19T13:00:55.201017+0000","last_became_peered":"2023-10-19T13:00:55.201017+0000","last_unstale":"2023-10-19T13:00:57.208793+0000","last_undegraded":"2023-10-19T13:00:57.208793+0000","last_fullsized":"2023-10-19T13:00:57.208793+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_clean_scrub_stamp":"2023-10-19T13:00:54.175433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:49:39.307408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289972+0000","last_change":"2023-10-19T13:00:56.234225+0000","last_active":"2023-10-19T13:00:57.289972+0000","last_peered":"2023-10-19T13:00:57.289972+0000","last_clean":"2023-10-19T13:00:57.289972+0000","last_became_active":"2023-10-19T13:00:56.233546+0000","last_became_peered":"2023-10-19T13:00:56.233546+0000","last_unstale":"2023-10-19T13:00:57.289972+0000","last_undegraded":"2023-10-19T13:00:57.289972+0000","last_fullsized":"2023-10-19T13:00:57.289972+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:12:40.484752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"99'8","reported_seq":20,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289107+0000","last_change":"2023-10-19T13:00:55.197229+0000","last_active":"2023-10-19T13:00:57.289107+0000","last_peered":"2023-10-19T13:00:57.289107+0000","last_clean":"2023-10-19T13:00:57.289107+0000","last_became_active":"2023-10-19T13:00:55.196966+0000","last_became_peered":"2023-10-19T13:00:55.196966+0000","last_unstale":"2023-10-19T13:00:57.289107+0000","last_undegraded":"2023-10-19T13:00:57.289107+0000","last_fullsized":"2023-10-19T13:00:57.289107+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_clean_scrub_stamp":"2023-10-19T13:00:54.175433+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:12:37.279461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289682+0000","last_change":"2023-10-19T13:00:56.229823+0000","last_active":"2023-10-19T13:00:57.289682+0000","last_peered":"2023-10-19T13:00:57.289682+0000","last_clean":"2023-10-19T13:00:57.289682+0000","last_became_active":"2023-10-19T13:00:56.229475+0000","last_became_peered":"2023-10-19T13:00:56.229475+0000","last_unstale":"2023-10-19T13:00:57.289682+0000","last_undegraded":"2023-10-19T13:00:57.289682+0000","last_fullsized":"2023-10-19T13:00:57.289682+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:18:17.438467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"99'1","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289306+0000","last_change":"2023-10-19T13:00:55.208633+0000","last_active":"2023-10-19T13:00:57.289306+0000","last_peered":"2023-10-19T13:00:57.289306+0000","last_clean":"2023-10-19T13:00:57.289306+0000","last_became_active":"2023-10-19T13:00:55.208365+0000","last_became_peered":"2023-10-19T13:00:55.208365+0000","last_unstale":"2023-10-19T13:00:57.289306+0000","last_undegraded":"2023-10-19T13:00:57.289306+0000","last_fullsized":"2023-10-19T13:00:57.289306+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_clean_scrub_stamp":"2023-10-19T13:00:54.175433+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:22:39.223988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.208758+0000","last_change":"2023-10-19T13:00:56.237212+0000","last_active":"2023-10-19T13:00:57.208758+0000","last_peered":"2023-10-19T13:00:57.208758+0000","last_clean":"2023-10-19T13:00:57.208758+0000","last_became_active":"2023-10-19T13:00:56.236953+0000","last_became_peered":"2023-10-19T13:00:56.236953+0000","last_unstale":"2023-10-19T13:00:57.208758+0000","last_undegraded":"2023-10-19T13:00:57.208758+0000","last_fullsized":"2023-10-19T13:00:57.208758+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:25:59.221155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"99'2","reported_seq":14,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.210239+0000","last_change":"2023-10-19T13:00:55.204299+0000","last_active":"2023-10-19T13:00:57.210239+0000","last_peered":"2023-10-19T13:00:57.210239+0000","last_clean":"2023-10-19T13:00:57.210239+0000","last_became_active":"2023-10-19T13:00:55.203996+0000","last_became_peered":"2023-10-19T13:00:55.203996+0000","last_unstale":"2023-10-19T13:00:57.210239+0000","last_undegraded":"2023-10-19T13:00:57.210239+0000","last_fullsized":"2023-10-19T13:00:57.210239+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_clean_scrub_stamp":"2023-10-19T13:00:54.175433+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:11:31.723870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.208964+0000","last_change":"2023-10-19T13:00:56.214727+0000","last_active":"2023-10-19T13:00:57.208964+0000","last_peered":"2023-10-19T13:00:57.208964+0000","last_clean":"2023-10-19T13:00:57.208964+0000","last_became_active":"2023-10-19T13:00:56.214204+0000","last_became_peered":"2023-10-19T13:00:56.214204+0000","last_unstale":"2023-10-19T13:00:57.208964+0000","last_undegraded":"2023-10-19T13:00:57.208964+0000","last_fullsized":"2023-10-19T13:00:57.208964+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:06:04.196145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"99'1","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.209384+0000","last_change":"2023-10-19T13:00:55.199355+0000","last_active":"2023-10-19T13:00:57.209384+0000","last_peered":"2023-10-19T13:00:57.209384+0000","last_clean":"2023-10-19T13:00:57.209384+0000","last_became_active":"2023-10-19T13:00:55.199080+0000","last_became_peered":"2023-10-19T13:00:55.199080+0000","last_unstale":"2023-10-19T13:00:57.209384+0000","last_undegraded":"2023-10-19T13:00:57.209384+0000","last_fullsized":"2023-10-19T13:00:57.209384+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_clean_scrub_stamp":"2023-10-19T13:00:54.175433+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:40:53.397823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289642+0000","last_change":"2023-10-19T13:00:56.230576+0000","last_active":"2023-10-19T13:00:57.289642+0000","last_peered":"2023-10-19T13:00:57.289642+0000","last_clean":"2023-10-19T13:00:57.289642+0000","last_became_active":"2023-10-19T13:00:56.230253+0000","last_became_peered":"2023-10-19T13:00:56.230253+0000","last_unstale":"2023-10-19T13:00:57.289642+0000","last_undegraded":"2023-10-19T13:00:57.289642+0000","last_fullsized":"2023-10-19T13:00:57.289642+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:51:21.587669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":12,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289306+0000","last_change":"2023-10-19T13:00:55.208450+0000","last_active":"2023-10-19T13:00:57.289306+0000","last_peered":"2023-10-19T13:00:57.289306+0000","last_clean":"2023-10-19T13:00:57.289306+0000","last_became_active":"2023-10-19T13:00:55.208166+0000","last_became_peered":"2023-10-19T13:00:55.208166+0000","last_unstale":"2023-10-19T13:00:57.289306+0000","last_undegraded":"2023-10-19T13:00:57.289306+0000","last_fullsized":"2023-10-19T13:00:57.289306+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_clean_scrub_stamp":"2023-10-19T13:00:54.175433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:25:25.966638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289969+0000","last_change":"2023-10-19T13:00:56.215279+0000","last_active":"2023-10-19T13:00:57.289969+0000","last_peered":"2023-10-19T13:00:57.289969+0000","last_clean":"2023-10-19T13:00:57.289969+0000","last_became_active":"2023-10-19T13:00:56.214996+0000","last_became_peered":"2023-10-19T13:00:56.214996+0000","last_unstale":"2023-10-19T13:00:57.289969+0000","last_undegraded":"2023-10-19T13:00:57.289969+0000","last_fullsized":"2023-10-19T13:00:57.289969+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:51:28.535991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"99'2","reported_seq":14,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289739+0000","last_change":"2023-10-19T13:00:55.204983+0000","last_active":"2023-10-19T13:00:57.289739+0000","last_peered":"2023-10-19T13:00:57.289739+0000","last_clean":"2023-10-19T13:00:57.289739+0000","last_became_active":"2023-10-19T13:00:55.204734+0000","last_became_peered":"2023-10-19T13:00:55.204734+0000","last_unstale":"2023-10-19T13:00:57.289739+0000","last_undegraded":"2023-10-19T13:00:57.289739+0000","last_fullsized":"2023-10-19T13:00:57.289739+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_clean_scrub_stamp":"2023-10-19T13:00:54.175433+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:41:49.782155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289593+0000","last_change":"2023-10-19T13:00:56.232216+0000","last_active":"2023-10-19T13:00:57.289593+0000","last_peered":"2023-10-19T13:00:57.289593+0000","last_clean":"2023-10-19T13:00:57.289593+0000","last_became_active":"2023-10-19T13:00:56.231956+0000","last_became_peered":"2023-10-19T13:00:56.231956+0000","last_unstale":"2023-10-19T13:00:57.289593+0000","last_undegraded":"2023-10-19T13:00:57.289593+0000","last_fullsized":"2023-10-19T13:00:57.289593+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:14:01.227382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":12,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289576+0000","last_change":"2023-10-19T13:00:55.198778+0000","last_active":"2023-10-19T13:00:57.289576+0000","last_peered":"2023-10-19T13:00:57.289576+0000","last_clean":"2023-10-19T13:00:57.289576+0000","last_became_active":"2023-10-19T13:00:55.196076+0000","last_became_peered":"2023-10-19T13:00:55.196076+0000","last_unstale":"2023-10-19T13:00:57.289576+0000","last_undegraded":"2023-10-19T13:00:57.289576+0000","last_fullsized":"2023-10-19T13:00:57.289576+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_clean_scrub_stamp":"2023-10-19T13:00:54.175433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:49:30.036753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.290048+0000","last_change":"2023-10-19T13:00:56.232810+0000","last_active":"2023-10-19T13:00:57.290048+0000","last_peered":"2023-10-19T13:00:57.290048+0000","last_clean":"2023-10-19T13:00:57.290048+0000","last_became_active":"2023-10-19T13:00:56.232551+0000","last_became_peered":"2023-10-19T13:00:56.232551+0000","last_unstale":"2023-10-19T13:00:57.290048+0000","last_undegraded":"2023-10-19T13:00:57.290048+0000","last_fullsized":"2023-10-19T13:00:57.290048+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:20:06.228901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":12,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289761+0000","last_change":"2023-10-19T13:00:55.199409+0000","last_active":"2023-10-19T13:00:57.289761+0000","last_peered":"2023-10-19T13:00:57.289761+0000","last_clean":"2023-10-19T13:00:57.289761+0000","last_became_active":"2023-10-19T13:00:55.198868+0000","last_became_peered":"2023-10-19T13:00:55.198868+0000","last_unstale":"2023-10-19T13:00:57.289761+0000","last_undegraded":"2023-10-19T13:00:57.289761+0000","last_fullsized":"2023-10-19T13:00:57.289761+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_clean_scrub_stamp":"2023-10-19T13:00:54.175433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:51:27.530077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.210496+0000","last_change":"2023-10-19T13:00:56.232955+0000","last_active":"2023-10-19T13:00:57.210496+0000","last_peered":"2023-10-19T13:00:57.210496+0000","last_clean":"2023-10-19T13:00:57.210496+0000","last_became_active":"2023-10-19T13:00:56.232696+0000","last_became_peered":"2023-10-19T13:00:56.232696+0000","last_unstale":"2023-10-19T13:00:57.210496+0000","last_undegraded":"2023-10-19T13:00:57.210496+0000","last_fullsized":"2023-10-19T13:00:57.210496+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:30:02.513725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"99'2","reported_seq":14,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.209942+0000","last_change":"2023-10-19T13:00:55.208659+0000","last_active":"2023-10-19T13:00:57.209942+0000","last_peered":"2023-10-19T13:00:57.209942+0000","last_clean":"2023-10-19T13:00:57.209942+0000","last_became_active":"2023-10-19T13:00:55.208333+0000","last_became_peered":"2023-10-19T13:00:55.208333+0000","last_unstale":"2023-10-19T13:00:57.209942+0000","last_undegraded":"2023-10-19T13:00:57.209942+0000","last_fullsized":"2023-10-19T13:00:57.209942+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_clean_scrub_stamp":"2023-10-19T13:00:54.175433+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:18:31.875925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289570+0000","last_change":"2023-10-19T13:00:56.229810+0000","last_active":"2023-10-19T13:00:57.289570+0000","last_peered":"2023-10-19T13:00:57.289570+0000","last_clean":"2023-10-19T13:00:57.289570+0000","last_became_active":"2023-10-19T13:00:56.229381+0000","last_became_peered":"2023-10-19T13:00:56.229381+0000","last_unstale":"2023-10-19T13:00:57.289570+0000","last_undegraded":"2023-10-19T13:00:57.289570+0000","last_fullsized":"2023-10-19T13:00:57.289570+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:25:26.972552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":12,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289673+0000","last_change":"2023-10-19T13:00:55.205267+0000","last_active":"2023-10-19T13:00:57.289673+0000","last_peered":"2023-10-19T13:00:57.289673+0000","last_clean":"2023-10-19T13:00:57.289673+0000","last_became_active":"2023-10-19T13:00:55.204979+0000","last_became_peered":"2023-10-19T13:00:55.204979+0000","last_unstale":"2023-10-19T13:00:57.289673+0000","last_undegraded":"2023-10-19T13:00:57.289673+0000","last_fullsized":"2023-10-19T13:00:57.289673+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_clean_scrub_stamp":"2023-10-19T13:00:54.175433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:17:53.038752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.210532+0000","last_change":"2023-10-19T13:00:56.233768+0000","last_active":"2023-10-19T13:00:57.210532+0000","last_peered":"2023-10-19T13:00:57.210532+0000","last_clean":"2023-10-19T13:00:57.210532+0000","last_became_active":"2023-10-19T13:00:56.233427+0000","last_became_peered":"2023-10-19T13:00:56.233427+0000","last_unstale":"2023-10-19T13:00:57.210532+0000","last_undegraded":"2023-10-19T13:00:57.210532+0000","last_fullsized":"2023-10-19T13:00:57.210532+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:07:12.067905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":12,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289197+0000","last_change":"2023-10-19T13:00:55.200925+0000","last_active":"2023-10-19T13:00:57.289197+0000","last_peered":"2023-10-19T13:00:57.289197+0000","last_clean":"2023-10-19T13:00:57.289197+0000","last_became_active":"2023-10-19T13:00:55.200653+0000","last_became_peered":"2023-10-19T13:00:55.200653+0000","last_unstale":"2023-10-19T13:00:57.289197+0000","last_undegraded":"2023-10-19T13:00:57.289197+0000","last_fullsized":"2023-10-19T13:00:57.289197+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_clean_scrub_stamp":"2023-10-19T13:00:54.175433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:10:01.251030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289454+0000","last_change":"2023-10-19T13:00:56.230487+0000","last_active":"2023-10-19T13:00:57.289454+0000","last_peered":"2023-10-19T13:00:57.289454+0000","last_clean":"2023-10-19T13:00:57.289454+0000","last_became_active":"2023-10-19T13:00:56.230232+0000","last_became_peered":"2023-10-19T13:00:56.230232+0000","last_unstale":"2023-10-19T13:00:57.289454+0000","last_undegraded":"2023-10-19T13:00:57.289454+0000","last_fullsized":"2023-10-19T13:00:57.289454+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:40:00.647749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":12,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289102+0000","last_change":"2023-10-19T13:00:55.203346+0000","last_active":"2023-10-19T13:00:57.289102+0000","last_peered":"2023-10-19T13:00:57.289102+0000","last_clean":"2023-10-19T13:00:57.289102+0000","last_became_active":"2023-10-19T13:00:55.203048+0000","last_became_peered":"2023-10-19T13:00:55.203048+0000","last_unstale":"2023-10-19T13:00:57.289102+0000","last_undegraded":"2023-10-19T13:00:57.289102+0000","last_fullsized":"2023-10-19T13:00:57.289102+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_clean_scrub_stamp":"2023-10-19T13:00:54.175433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:02:44.720415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289537+0000","last_change":"2023-10-19T13:00:56.234907+0000","last_active":"2023-10-19T13:00:57.289537+0000","last_peered":"2023-10-19T13:00:57.289537+0000","last_clean":"2023-10-19T13:00:57.289537+0000","last_became_active":"2023-10-19T13:00:56.234653+0000","last_became_peered":"2023-10-19T13:00:56.234653+0000","last_unstale":"2023-10-19T13:00:57.289537+0000","last_undegraded":"2023-10-19T13:00:57.289537+0000","last_fullsized":"2023-10-19T13:00:57.289537+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:21:47.277500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":12,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289429+0000","last_change":"2023-10-19T13:00:55.202434+0000","last_active":"2023-10-19T13:00:57.289429+0000","last_peered":"2023-10-19T13:00:57.289429+0000","last_clean":"2023-10-19T13:00:57.289429+0000","last_became_active":"2023-10-19T13:00:55.202018+0000","last_became_peered":"2023-10-19T13:00:55.202018+0000","last_unstale":"2023-10-19T13:00:57.289429+0000","last_undegraded":"2023-10-19T13:00:57.289429+0000","last_fullsized":"2023-10-19T13:00:57.289429+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_clean_scrub_stamp":"2023-10-19T13:00:54.175433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:08:07.333292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289833+0000","last_change":"2023-10-19T13:00:56.233959+0000","last_active":"2023-10-19T13:00:57.289833+0000","last_peered":"2023-10-19T13:00:57.289833+0000","last_clean":"2023-10-19T13:00:57.289833+0000","last_became_active":"2023-10-19T13:00:56.233404+0000","last_became_peered":"2023-10-19T13:00:56.233404+0000","last_unstale":"2023-10-19T13:00:57.289833+0000","last_undegraded":"2023-10-19T13:00:57.289833+0000","last_fullsized":"2023-10-19T13:00:57.289833+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:41:21.345551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":12,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.208909+0000","last_change":"2023-10-19T13:00:55.201999+0000","last_active":"2023-10-19T13:00:57.208909+0000","last_peered":"2023-10-19T13:00:57.208909+0000","last_clean":"2023-10-19T13:00:57.208909+0000","last_became_active":"2023-10-19T13:00:55.201737+0000","last_became_peered":"2023-10-19T13:00:55.201737+0000","last_unstale":"2023-10-19T13:00:57.208909+0000","last_undegraded":"2023-10-19T13:00:57.208909+0000","last_fullsized":"2023-10-19T13:00:57.208909+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_clean_scrub_stamp":"2023-10-19T13:00:54.175433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:29:43.561077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.290038+0000","last_change":"2023-10-19T13:00:56.222137+0000","last_active":"2023-10-19T13:00:57.290038+0000","last_peered":"2023-10-19T13:00:57.290038+0000","last_clean":"2023-10-19T13:00:57.290038+0000","last_became_active":"2023-10-19T13:00:56.221851+0000","last_became_peered":"2023-10-19T13:00:56.221851+0000","last_unstale":"2023-10-19T13:00:57.290038+0000","last_undegraded":"2023-10-19T13:00:57.290038+0000","last_fullsized":"2023-10-19T13:00:57.290038+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:45:02.161382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":12,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289630+0000","last_change":"2023-10-19T13:00:55.200490+0000","last_active":"2023-10-19T13:00:57.289630+0000","last_peered":"2023-10-19T13:00:57.289630+0000","last_clean":"2023-10-19T13:00:57.289630+0000","last_became_active":"2023-10-19T13:00:55.200218+0000","last_became_peered":"2023-10-19T13:00:55.200218+0000","last_unstale":"2023-10-19T13:00:57.289630+0000","last_undegraded":"2023-10-19T13:00:57.289630+0000","last_fullsized":"2023-10-19T13:00:57.289630+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_clean_scrub_stamp":"2023-10-19T13:00:54.175433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:52:31.373725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289847+0000","last_change":"2023-10-19T13:00:56.230863+0000","last_active":"2023-10-19T13:00:57.289847+0000","last_peered":"2023-10-19T13:00:57.289847+0000","last_clean":"2023-10-19T13:00:57.289847+0000","last_became_active":"2023-10-19T13:00:56.230572+0000","last_became_peered":"2023-10-19T13:00:56.230572+0000","last_unstale":"2023-10-19T13:00:57.289847+0000","last_undegraded":"2023-10-19T13:00:57.289847+0000","last_fullsized":"2023-10-19T13:00:57.289847+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:54:37.924182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":12,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289105+0000","last_change":"2023-10-19T13:00:55.209095+0000","last_active":"2023-10-19T13:00:57.289105+0000","last_peered":"2023-10-19T13:00:57.289105+0000","last_clean":"2023-10-19T13:00:57.289105+0000","last_became_active":"2023-10-19T13:00:55.208807+0000","last_became_peered":"2023-10-19T13:00:55.208807+0000","last_unstale":"2023-10-19T13:00:57.289105+0000","last_undegraded":"2023-10-19T13:00:57.289105+0000","last_fullsized":"2023-10-19T13:00:57.289105+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_clean_scrub_stamp":"2023-10-19T13:00:54.175433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:47:53.768966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289205+0000","last_change":"2023-10-19T13:00:56.234690+0000","last_active":"2023-10-19T13:00:57.289205+0000","last_peered":"2023-10-19T13:00:57.289205+0000","last_clean":"2023-10-19T13:00:57.289205+0000","last_became_active":"2023-10-19T13:00:56.234405+0000","last_became_peered":"2023-10-19T13:00:56.234405+0000","last_unstale":"2023-10-19T13:00:57.289205+0000","last_undegraded":"2023-10-19T13:00:57.289205+0000","last_fullsized":"2023-10-19T13:00:57.289205+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:56:34.119416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":12,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.290378+0000","last_change":"2023-10-19T13:00:55.199843+0000","last_active":"2023-10-19T13:00:57.290378+0000","last_peered":"2023-10-19T13:00:57.290378+0000","last_clean":"2023-10-19T13:00:57.290378+0000","last_became_active":"2023-10-19T13:00:55.199018+0000","last_became_peered":"2023-10-19T13:00:55.199018+0000","last_unstale":"2023-10-19T13:00:57.290378+0000","last_undegraded":"2023-10-19T13:00:57.290378+0000","last_fullsized":"2023-10-19T13:00:57.290378+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_clean_scrub_stamp":"2023-10-19T13:00:54.175433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:51:39.000977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"89'181","reported_seq":467,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-19T13:00:56.221874+0000","last_change":"2023-10-19T12:44:08.636613+0000","last_active":"2023-10-19T13:00:56.221874+0000","last_peered":"2023-10-19T13:00:56.221874+0000","last_clean":"2023-10-19T13:00:56.221874+0000","last_became_active":"2023-10-19T12:44:08.635697+0000","last_became_peered":"2023-10-19T12:44:08.635697+0000","last_unstale":"2023-10-19T13:00:56.221874+0000","last_undegraded":"2023-10-19T13:00:56.221874+0000","last_fullsized":"2023-10-19T13:00:56.221874+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_clean_scrub_stamp":"2023-10-19T12:44:07.399738+0000","objects_scrubbed":0,"log_size":181,"log_dups_size":0,"ondisk_log_size":181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:15:23.685485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":674,"num_read_kb":1321,"num_write":275,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":674,"num_read_kb":1321,"num_write":275,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":181,"ondisk_log_size":181,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575053,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59332,"kb_used_data":1780,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312508,"statfs":{"total":96636764160,"available":96576008192,"internally_reserved":0,"allocated":1822720,"data_stored":1203836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58915275},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 13:00:44 2023","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.55200000000000005,"15min":0.56899999999999995},"min":{"1min":0.34100000000000003,"5min":0.30299999999999999,"15min":0.26400000000000001},"max":{"1min":0.74099999999999999,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.58199999999999996,"15min":0.58899999999999997},"min":{"1min":0.36499999999999999,"5min":0.33200000000000002,"15min":0.29599999999999999},"max":{"1min":0.75600000000000001,"5min":0.83799999999999997,"15min":0.93700000000000006},"last":0.46800000000000003}]},{"osd":1,"last update":"Thu Oct 19 13:00:44 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.56599999999999995,"15min":0.57699999999999996},"min":{"1min":0.41299999999999998,"5min":0.40300000000000002,"15min":0.28199999999999997},"max":{"1min":0.68100000000000005,"5min":0.86299999999999999,"15min":1.3129999999999999},"last":0.441},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.57599999999999996,"15min":0.58999999999999997},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.32700000000000001},"max":{"1min":0.69299999999999995,"5min":0.82699999999999996,"15min":1.284},"last":0.41599999999999998}]},{"osd":2,"last update":"Thu Oct 19 13:00:44 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.623,"15min":0.63},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.76900000000000002,"5min":0.82699999999999996,"15min":0.98599999999999999},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.61899999999999999,"15min":0.63},"min":{"1min":0.46200000000000002,"5min":0.41599999999999998,"15min":0.28399999999999997},"max":{"1min":0.78300000000000003,"5min":0.81200000000000006,"15min":1.2589999999999999},"last":0.54500000000000004}]},{"osd":3,"last update":"Thu Oct 19 13:00:44 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.58999999999999997,"15min":0.59999999999999998},"min":{"1min":0.39700000000000002,"5min":0.34999999999999998,"15min":0.34399999999999997},"max":{"1min":0.71599999999999997,"5min":0.81200000000000006,"15min":0.96899999999999997},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.621,"5min":0.65400000000000003,"15min":0.65600000000000003},"min":{"1min":0.45500000000000002,"5min":0.45300000000000001,"15min":0.39700000000000002},"max":{"1min":0.79800000000000004,"5min":0.92000000000000004,"15min":1.329},"last":0.59299999999999997}]},{"osd":4,"last update":"Thu Oct 19 13:00:44 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.47499999999999998,"15min":0.46999999999999997},"min":{"1min":0.315,"5min":0.27400000000000002,"15min":0.25600000000000001},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.80500000000000005},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.47399999999999998,"15min":0.47199999999999998},"min":{"1min":0.30199999999999999,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.67100000000000004,"5min":0.75600000000000001,"15min":0.78900000000000003},"last":0.38800000000000001}]},{"osd":5,"last update":"Thu Oct 19 13:00:44 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.48899999999999999,"15min":0.48499999999999999},"min":{"1min":0.32000000000000001,"5min":0.30499999999999999,"15min":0.29099999999999998},"max":{"1min":0.74099999999999999,"5min":0.75800000000000001,"15min":0.90400000000000003},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52000000000000002,"15min":0.51700000000000002},"min":{"1min":0.34000000000000002,"5min":0.314,"15min":0.308},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.92200000000000004},"last":0.69299999999999995}]},{"osd":6,"last update":"Thu Oct 19 13:00:44 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.54000000000000004,"15min":0.54800000000000004},"min":{"1min":0.378,"5min":0.34100000000000003,"15min":0.308},"max":{"1min":0.749,"5min":0.81299999999999994,"15min":0.88900000000000001},"last":0.62},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52000000000000002,"15min":0.52800000000000002},"min":{"1min":0.33400000000000002,"5min":0.30299999999999999,"15min":0.27300000000000002},"max":{"1min":0.68300000000000005,"5min":0.83999999999999997,"15min":0.90300000000000002},"last":0.60799999999999998}]}]},{"osd":6,"up_from":13,"seq":55834575053,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32396,"kb_used_data":1208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339444,"statfs":{"total":96636764160,"available":96603590656,"internally_reserved":0,"allocated":1236992,"data_stored":615058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 13:00:32 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.57199999999999995,"15min":0.58599999999999997},"min":{"1min":0.373,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.75800000000000001,"5min":0.76000000000000001,"15min":0.81999999999999995},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.57799999999999996,"15min":0.58499999999999996},"min":{"1min":0.36399999999999999,"5min":0.247,"15min":0.247},"max":{"1min":0.75600000000000001,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.57499999999999996}]},{"osd":1,"last update":"Thu Oct 19 13:00:32 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.35299999999999998,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.76000000000000001,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.59699999999999998,"15min":0.59899999999999998},"min":{"1min":0.42099999999999999,"5min":0.34300000000000003,"15min":0.28399999999999997},"max":{"1min":0.749,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.439}]},{"osd":2,"last update":"Thu Oct 19 13:00:32 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.57299999999999995,"15min":0.58199999999999996},"min":{"1min":0.36899999999999999,"5min":0.307,"15min":0.26600000000000001},"max":{"1min":0.755,"5min":0.84699999999999998,"15min":0.85699999999999998},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.61299999999999999,"15min":0.60999999999999999},"min":{"1min":0.38900000000000001,"5min":0.35999999999999999,"15min":0.28399999999999997},"max":{"1min":0.79400000000000004,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.56299999999999994}]},{"osd":3,"last update":"Thu Oct 19 13:00:32 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61899999999999999,"15min":0.624},"min":{"1min":0.38800000000000001,"5min":0.38500000000000001,"15min":0.32600000000000001},"max":{"1min":0.81299999999999994,"5min":0.871,"15min":0.871},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.61899999999999999,"15min":0.624},"min":{"1min":0.40100000000000002,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.74099999999999999,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.48699999999999999}]},{"osd":4,"last update":"Thu Oct 19 13:00:32 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.45900000000000002,"15min":0.46600000000000003},"min":{"1min":0.29099999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.76700000000000002,"5min":0.77900000000000003,"15min":0.79300000000000004},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.47699999999999998,"15min":0.47999999999999998},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.247},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.77600000000000002},"last":0.501}]},{"osd":5,"last update":"Thu Oct 19 13:00:32 2023","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.46100000000000002,"15min":0.47099999999999997},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.26600000000000001},"max":{"1min":0.55700000000000005,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.51400000000000001,"15min":0.51600000000000001},"min":{"1min":0.375,"5min":0.32100000000000001,"15min":0.312},"max":{"1min":0.747,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.53100000000000003}]},{"osd":7,"last update":"Thu Oct 19 13:00:32 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.52500000000000002,"15min":0.52100000000000002},"min":{"1min":0.32400000000000001,"5min":0.29299999999999998,"15min":0.28199999999999997},"max":{"1min":0.77600000000000002,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.53500000000000003},"min":{"1min":0.27900000000000003,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.66700000000000004,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.52000000000000002}]}]},{"osd":5,"up_from":13,"seq":55834575053,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50428,"kb_used_data":1200,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321412,"statfs":{"total":96636764160,"available":96585125888,"internally_reserved":0,"allocated":1228800,"data_stored":613480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 13:00:29 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58799999999999997,"15min":0.59099999999999997},"min":{"1min":0.307,"5min":0.28999999999999998,"15min":0.223},"max":{"1min":0.75900000000000001,"5min":0.79900000000000004,"15min":0.82699999999999996},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58599999999999997,"15min":0.59799999999999998},"min":{"1min":0.41099999999999998,"5min":0.36099999999999999,"15min":0.27800000000000002},"max":{"1min":0.82999999999999996,"5min":0.84299999999999997,"15min":0.874},"last":0.51700000000000002}]},{"osd":1,"last update":"Thu Oct 19 13:00:29 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.57799999999999996,"15min":0.59499999999999997},"min":{"1min":0.40400000000000003,"5min":0.34599999999999997,"15min":0.32800000000000001},"max":{"1min":0.71299999999999997,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.60199999999999998,"15min":0.61199999999999999},"min":{"1min":0.378,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.77200000000000002,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.57499999999999996}]},{"osd":2,"last update":"Thu Oct 19 13:00:29 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.56399999999999995,"15min":0.58499999999999996},"min":{"1min":0.41199999999999998,"5min":0.34100000000000003,"15min":0.307},"max":{"1min":0.71699999999999997,"5min":0.78600000000000003,"15min":0.80700000000000005},"last":0.495},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.61199999999999999,"15min":0.624},"min":{"1min":0.47099999999999997,"5min":0.38400000000000001,"15min":0.36599999999999999},"max":{"1min":0.79300000000000004,"5min":0.88100000000000001,"15min":0.93600000000000005},"last":0.64500000000000002}]},{"osd":3,"last update":"Thu Oct 19 13:00:29 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.58899999999999997,"15min":0.60399999999999998},"min":{"1min":0.41699999999999998,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.752,"5min":0.79900000000000004,"15min":0.81299999999999994},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.61499999999999999,"15min":0.63400000000000001},"min":{"1min":0.501,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.73999999999999999,"5min":0.81000000000000005,"15min":0.89500000000000002},"last":0.55400000000000005}]},{"osd":4,"last update":"Thu Oct 19 13:00:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.47599999999999998,"15min":0.47499999999999998},"min":{"1min":0.35399999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.73599999999999999,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.46800000000000003,"15min":0.46999999999999997},"min":{"1min":0.33900000000000002,"5min":0.251,"15min":0.251},"max":{"1min":0.69199999999999995,"5min":0.75900000000000001,"15min":0.80500000000000005},"last":0.59999999999999998}]},{"osd":6,"last update":"Thu Oct 19 13:00:29 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.34100000000000003,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.75},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.51100000000000001,"15min":0.504},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.33000000000000002},"max":{"1min":0.67200000000000004,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.69399999999999995}]},{"osd":7,"last update":"Thu Oct 19 13:00:29 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.498,"15min":0.49399999999999999},"min":{"1min":0.39800000000000002,"5min":0.32000000000000001,"15min":0.315},"max":{"1min":0.72399999999999998,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.54900000000000004,"15min":0.53900000000000003},"min":{"1min":0.42599999999999999,"5min":0.311,"15min":0.311},"max":{"1min":0.74099999999999999,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.63400000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575053,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32396,"kb_used_data":1216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339444,"statfs":{"total":96636764160,"available":96603590656,"internally_reserved":0,"allocated":1245184,"data_stored":615162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 13:00:43 2023","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.55200000000000005,"15min":0.56000000000000005},"min":{"1min":0.36099999999999999,"5min":0.29499999999999998,"15min":0.28899999999999998},"max":{"1min":0.72299999999999998,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.54700000000000004,"15min":0.55700000000000005},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.27600000000000002},"max":{"1min":0.70099999999999996,"5min":0.77800000000000002,"15min":0.78200000000000003},"last":0.50800000000000001}]},{"osd":1,"last update":"Thu Oct 19 13:00:43 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.58099999999999996,"15min":0.60699999999999998},"min":{"1min":0.36599999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.72699999999999998,"5min":0.80600000000000005,"15min":0.86199999999999999},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.60299999999999998,"15min":0.61499999999999999},"min":{"1min":0.433,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.73699999999999999,"5min":0.79900000000000004,"15min":0.873},"last":0.66400000000000003}]},{"osd":2,"last update":"Thu Oct 19 13:00:43 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.59399999999999997,"15min":0.61099999999999999},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.254},"max":{"1min":0.79300000000000004,"5min":0.85299999999999998,"15min":0.86599999999999999},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.59699999999999998,"15min":0.61299999999999999},"min":{"1min":0.38700000000000001,"5min":0.35799999999999998,"15min":0.34499999999999997},"max":{"1min":0.75800000000000001,"5min":0.82999999999999996,"15min":0.85999999999999999},"last":0.63500000000000001}]},{"osd":3,"last update":"Thu Oct 19 13:00:43 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.57499999999999996,"15min":0.58499999999999996},"min":{"1min":0.36199999999999999,"5min":0.35199999999999998,"15min":0.32400000000000001},"max":{"1min":0.75,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.63,"15min":0.63700000000000001},"min":{"1min":0.44800000000000001,"5min":0.42599999999999999,"15min":0.37},"max":{"1min":0.76500000000000001,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.59899999999999998}]},{"osd":5,"last update":"Thu Oct 19 13:00:43 2023","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.45100000000000001,"15min":0.44800000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.249},"max":{"1min":0.63200000000000001,"5min":0.79600000000000004,"15min":0.81299999999999994},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.46899999999999997,"15min":0.45900000000000002},"min":{"1min":0.30099999999999999,"5min":0.26400000000000001,"15min":0.23499999999999999},"max":{"1min":0.69499999999999995,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.36899999999999999}]},{"osd":6,"last update":"Thu Oct 19 13:00:43 2023","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.45500000000000002,"15min":0.45400000000000001},"min":{"1min":0.30099999999999999,"5min":0.29499999999999998,"15min":0.26400000000000001},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.78600000000000003},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.48799999999999999,"15min":0.49399999999999999},"min":{"1min":0.308,"5min":0.308,"15min":0.27100000000000002},"max":{"1min":0.70699999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.41799999999999998}]},{"osd":7,"last update":"Thu Oct 19 13:00:43 2023","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.505,"15min":0.501},"min":{"1min":0.33200000000000002,"5min":0.307,"15min":0.27500000000000002},"max":{"1min":0.72799999999999998,"5min":0.79100000000000004,"15min":0.80200000000000005},"last":0.442},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54200000000000004,"15min":0.54400000000000004},"min":{"1min":0.371,"5min":0.35799999999999998,"15min":0.30199999999999999},"max":{"1min":0.70799999999999996,"5min":0.80500000000000005,"15min":0.83499999999999996},"last":0.46100000000000002}]}]},{"osd":3,"up_from":13,"seq":55834575052,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36544,"kb_used_data":1200,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335296,"statfs":{"total":96636764160,"available":96599343104,"internally_reserved":0,"allocated":1228800,"data_stored":614028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 13:00:27 2023","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.42199999999999999,"15min":0.40000000000000002},"min":{"1min":0.252,"5min":0.23599999999999999,"15min":0.20899999999999999},"max":{"1min":0.60399999999999998,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.41799999999999998,"15min":0.41599999999999998},"min":{"1min":0.28699999999999998,"5min":0.223,"15min":0.223},"max":{"1min":0.59899999999999998,"5min":0.75,"15min":0.75},"last":0.52800000000000002}]},{"osd":1,"last update":"Thu Oct 19 13:00:27 2023","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.41999999999999998,"15min":0.41199999999999998},"min":{"1min":0.26900000000000002,"5min":0.26100000000000001,"15min":0.246},"max":{"1min":0.628,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.48599999999999999,"15min":0.45900000000000002},"min":{"1min":0.30099999999999999,"5min":0.24099999999999999,"15min":0.215},"max":{"1min":0.64200000000000002,"5min":2.9399999999999999,"15min":2.9399999999999999},"last":0.46000000000000002}]},{"osd":2,"last update":"Thu Oct 19 13:00:27 2023","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.47099999999999997,"15min":0.46800000000000003},"min":{"1min":0.317,"5min":0.314,"15min":0.26100000000000001},"max":{"1min":0.62,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.496,"5min":0.48299999999999998,"15min":0.48699999999999999},"min":{"1min":0.34499999999999997,"5min":0.29999999999999999,"15min":0.247},"max":{"1min":0.66900000000000004,"5min":0.754,"15min":0.754},"last":0.50800000000000001}]},{"osd":4,"last update":"Thu Oct 19 13:00:27 2023","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.66000000000000003,"15min":0.66700000000000004},"min":{"1min":0.44700000000000001,"5min":0.42299999999999999,"15min":0.40300000000000002},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":1.0069999999999999},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.68799999999999994,"15min":0.69199999999999995},"min":{"1min":0.48899999999999999,"5min":0.46899999999999997,"15min":0.42099999999999999},"max":{"1min":0.91400000000000003,"5min":0.93000000000000005,"15min":0.99399999999999999},"last":0.84999999999999998}]},{"osd":5,"last update":"Thu Oct 19 13:00:27 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.68500000000000005,"15min":0.66000000000000003},"min":{"1min":0.47499999999999998,"5min":0.45500000000000002,"15min":0.41099999999999998},"max":{"1min":0.90600000000000003,"5min":2.9990000000000001,"15min":2.9990000000000001},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.65000000000000002,"15min":0.65100000000000002},"min":{"1min":0.47699999999999998,"5min":0.40899999999999997,"15min":0.35899999999999999},"max":{"1min":0.873,"5min":1.774,"15min":1.774},"last":0.86099999999999999}]},{"osd":6,"last update":"Thu Oct 19 13:00:27 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.66300000000000003,"15min":0.65400000000000003},"min":{"1min":0.503,"5min":0.433,"15min":0.41199999999999998},"max":{"1min":0.84499999999999997,"5min":0.93700000000000006,"15min":0.96799999999999997},"last":0.90100000000000002},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.68000000000000005,"15min":0.67600000000000005},"min":{"1min":0.501,"5min":0.377,"15min":0.377},"max":{"1min":0.94399999999999995,"5min":1.167,"15min":1.167},"last":0.80400000000000005}]},{"osd":7,"last update":"Thu Oct 19 13:00:27 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.65900000000000003,"15min":0.66500000000000004},"min":{"1min":0.47999999999999998,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.90700000000000003,"5min":1.042,"15min":1.042},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.69499999999999995,"15min":0.68100000000000005},"min":{"1min":0.51100000000000001,"5min":0.432,"15min":0.432},"max":{"1min":0.88300000000000001,"5min":2.8149999999999999,"15min":2.8149999999999999},"last":0.88}]}]},{"osd":2,"up_from":13,"seq":55834575054,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54604,"kb_used_data":1208,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317236,"statfs":{"total":96636764160,"available":96580849664,"internally_reserved":0,"allocated":1236992,"data_stored":614188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 13:00:37 2023","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.36899999999999999,"15min":0.378},"min":{"1min":0.215,"5min":0.20899999999999999,"15min":0.187},"max":{"1min":0.52900000000000003,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.314},{"interface":"front","average":{"1min":0.374,"5min":0.39700000000000002,"15min":0.40699999999999997},"min":{"1min":0.28000000000000003,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.56000000000000005,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.33300000000000002}]},{"osd":1,"last update":"Thu Oct 19 13:00:37 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.38200000000000001,"15min":0.39600000000000002},"min":{"1min":0.26100000000000001,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.51100000000000001,"5min":0.63900000000000001,"15min":0.71099999999999997},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.41099999999999998,"15min":0.42799999999999999},"min":{"1min":0.26900000000000002,"5min":0.22,"15min":0.22},"max":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.68100000000000005},"last":0.38500000000000001}]},{"osd":3,"last update":"Thu Oct 19 13:00:37 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.45300000000000001,"15min":0.45500000000000002},"min":{"1min":0.33700000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.50700000000000001,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.432},{"interface":"front","average":{"1min":0.439,"5min":0.46899999999999997,"15min":0.48199999999999998},"min":{"1min":0.34100000000000003,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.57499999999999996,"5min":0.77800000000000002,"15min":0.82999999999999996},"last":0.41599999999999998}]},{"osd":4,"last update":"Thu Oct 19 13:00:37 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.61899999999999999,"15min":0.64000000000000001},"min":{"1min":0.40100000000000002,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.68700000000000006,"5min":0.93999999999999995,"15min":0.98999999999999999},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.63400000000000001,"15min":0.65500000000000003},"min":{"1min":0.42999999999999999,"5min":0.35099999999999998,"15min":0.32000000000000001},"max":{"1min":0.78000000000000003,"5min":0.91200000000000003,"15min":1.046},"last":0.45900000000000002}]},{"osd":5,"last update":"Thu Oct 19 13:00:37 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.65000000000000002,"15min":0.66100000000000003},"min":{"1min":0.46899999999999997,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.73999999999999999,"5min":0.96499999999999997,"15min":0.99399999999999999},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.63200000000000001,"15min":0.65600000000000003},"min":{"1min":0.39100000000000001,"5min":0.375,"15min":0.375},"max":{"1min":0.753,"5min":0.90200000000000002,"15min":0.95799999999999996},"last":0.48299999999999998}]},{"osd":6,"last update":"Thu Oct 19 13:00:37 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.63700000000000001,"15min":0.66300000000000003},"min":{"1min":0.48799999999999999,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.73599999999999999,"5min":1.008,"15min":1.008},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.65600000000000003,"15min":0.67400000000000004},"min":{"1min":0.46300000000000002,"5min":0.371,"15min":0.371},"max":{"1min":0.80500000000000005,"5min":0.995,"15min":0.995},"last":0.59399999999999997}]},{"osd":7,"last update":"Thu Oct 19 13:00:37 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.65400000000000003,"15min":0.68000000000000005},"min":{"1min":0.39800000000000002,"5min":0.39300000000000002,"15min":0.36399999999999999},"max":{"1min":0.71799999999999997,"5min":0.98999999999999999,"15min":1.0149999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.66800000000000004,"15min":0.68100000000000005},"min":{"1min":0.47199999999999998,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.80000000000000004,"5min":1.02,"15min":1.0329999999999999},"last":0.61899999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575053,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32392,"kb_used_data":1204,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339448,"statfs":{"total":96636764160,"available":96603594752,"internally_reserved":0,"allocated":1232896,"data_stored":613558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 13:00:36 2023","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.39000000000000001,"15min":0.38600000000000001},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.187},"max":{"1min":0.57799999999999996,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.245},{"interface":"front","average":{"1min":0.438,"5min":0.441,"15min":0.43099999999999999},"min":{"1min":0.217,"5min":0.217,"15min":0.21199999999999999},"max":{"1min":0.65800000000000003,"5min":2.5800000000000001,"15min":2.5800000000000001},"last":0.314}]},{"osd":2,"last update":"Thu Oct 19 13:00:36 2023","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.41799999999999998,"15min":0.42199999999999999},"min":{"1min":0.24199999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.69999999999999996},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.47599999999999998,"15min":0.45500000000000002},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.23699999999999999},"max":{"1min":0.75600000000000001,"5min":2.6230000000000002,"15min":2.6230000000000002},"last":0.34399999999999997}]},{"osd":3,"last update":"Thu Oct 19 13:00:36 2023","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.50700000000000001,"15min":0.48499999999999999},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.26200000000000001},"max":{"1min":0.69599999999999995,"5min":3.9740000000000002,"15min":3.9740000000000002},"last":0.371},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.43099999999999999,"15min":0.42199999999999999},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27000000000000002},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.80500000000000005},"last":0.35499999999999998}]},{"osd":4,"last update":"Thu Oct 19 13:00:36 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.44900000000000001,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.86899999999999999,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.627,"5min":0.66100000000000003,"15min":0.64500000000000002},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.314},"max":{"1min":0.86899999999999999,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.51500000000000001}]},{"osd":5,"last update":"Thu Oct 19 13:00:36 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.65300000000000002,"15min":0.65300000000000002},"min":{"1min":0.55300000000000005,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.85299999999999998,"5min":1.1200000000000001,"15min":1.1200000000000001},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.69499999999999995,"15min":0.67300000000000004},"min":{"1min":0.436,"5min":0.42299999999999999,"15min":0.38700000000000001},"max":{"1min":0.84499999999999997,"5min":2.7120000000000002,"15min":2.7120000000000002},"last":0.48499999999999999}]},{"osd":6,"last update":"Thu Oct 19 13:00:36 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.64600000000000002,"15min":0.65400000000000003},"min":{"1min":0.46999999999999997,"5min":0.36799999999999999,"15min":0.36399999999999999},"max":{"1min":0.83799999999999997,"5min":0.92200000000000004,"15min":1.0069999999999999},"last":0.495},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.67800000000000005,"15min":0.67700000000000005},"min":{"1min":0.47299999999999998,"5min":0.376,"15min":0.376},"max":{"1min":1.129,"5min":1.129,"15min":1.129},"last":0.58899999999999997}]},{"osd":7,"last update":"Thu Oct 19 13:00:36 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.69199999999999995,"15min":0.67500000000000004},"min":{"1min":0.52900000000000003,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.80300000000000005,"5min":2.786,"15min":2.786},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.67400000000000004,"15min":0.67600000000000005},"min":{"1min":0.45200000000000001,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.79500000000000004,"5min":0.94199999999999995,"15min":1.034},"last":0.53000000000000003}]}]},{"osd":0,"up_from":13,"seq":55834575054,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37112,"kb_used_data":1772,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334728,"statfs":{"total":96636764160,"available":96598761472,"internally_reserved":0,"allocated":1814528,"data_stored":1203195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Oct 19 13:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.38700000000000001,"15min":0.39000000000000001},"min":{"1min":0.20899999999999999,"5min":0.182,"15min":0.182},"max":{"1min":0.52800000000000002,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.372,"5min":0.42199999999999999,"15min":0.43099999999999999},"min":{"1min":0.248,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.58999999999999997,"5min":0.67600000000000005,"15min":0.746},"last":0.36099999999999999}]},{"osd":2,"last update":"Thu Oct 19 13:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.45100000000000001,"15min":0.45300000000000001},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.25800000000000001},"max":{"1min":0.64100000000000001,"5min":0.752,"15min":0.76300000000000001},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.42499999999999999,"15min":0.434},"min":{"1min":0.246,"5min":0.216,"15min":0.20699999999999999},"max":{"1min":0.623,"5min":0.72999999999999998,"15min":0.76800000000000002},"last":0.377}]},{"osd":3,"last update":"Thu Oct 19 13:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.40999999999999998,"15min":0.40899999999999997},"min":{"1min":0.26600000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.56599999999999995,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.308},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.318,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.67500000000000004,"5min":1.4990000000000001,"15min":1.4990000000000001},"last":0.32700000000000001}]},{"osd":4,"last update":"Thu Oct 19 13:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.61399999999999999,"15min":0.63300000000000001},"min":{"1min":0.40300000000000002,"5min":0.40000000000000002,"15min":0.34999999999999998},"max":{"1min":0.71299999999999997,"5min":1.361,"15min":1.361},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.64500000000000002,"15min":0.64900000000000002},"min":{"1min":0.42899999999999999,"5min":0.41899999999999998,"15min":0.35999999999999999},"max":{"1min":0.80900000000000005,"5min":0.98999999999999999,"15min":0.998},"last":0.60299999999999998}]},{"osd":5,"last update":"Thu Oct 19 13:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.67900000000000005,"15min":0.68100000000000005},"min":{"1min":0.49199999999999999,"5min":0.378,"15min":0.378},"max":{"1min":0.90000000000000002,"5min":1.325,"15min":1.325},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.62,"5min":0.67900000000000005,"15min":0.68300000000000005},"min":{"1min":0.437,"5min":0.36299999999999999,"15min":0.318},"max":{"1min":0.88,"5min":1.024,"15min":1.024},"last":0.68100000000000005}]},{"osd":6,"last update":"Thu Oct 19 13:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.65600000000000003,"15min":0.66200000000000003},"min":{"1min":0.42499999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.79300000000000004,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.62},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.65200000000000002,"15min":0.66200000000000003},"min":{"1min":0.45300000000000001,"5min":0.41999999999999998,"15min":0.38900000000000001},"max":{"1min":0.73499999999999999,"5min":0.995,"15min":1.089},"last":0.56699999999999995}]},{"osd":7,"last update":"Thu Oct 19 13:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.66200000000000003,"15min":0.67900000000000005},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.82799999999999996,"5min":0.98799999999999999,"15min":1.0089999999999999},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.65900000000000003,"15min":0.66100000000000003},"min":{"1min":0.442,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.94499999999999995,"5min":1.389,"15min":1.389},"last":0.58299999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-19T13:00:59.223 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-19T13:00:59.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.363+0000 7f6d60b89640 1 -- 172.21.15.57:0/3720948716 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6d48003ef0 msgr2=0x7f6d480047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:59.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.363+0000 7f6d60b89640 1 --2- 172.21.15.57:0/3720948716 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6d48003ef0 0x7f6d480047c0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f6d4400c260 tx=0x7f6d480057f0 comp rx=0 tx=0).stop 2023-10-19T13:00:59.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.363+0000 7f6d5ad9d640 1 -- 172.21.15.57:0/3720948716 reap_dead start 2023-10-19T13:00:59.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.363+0000 7f6d60b89640 1 -- 172.21.15.57:0/3720948716 shutdown_connections 2023-10-19T13:00:59.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.363+0000 7f6d60b89640 1 -- 172.21.15.57:0/3720948716 >> 172.21.15.57:0/3720948716 conn(0x7f6d5c0d0d60 msgr2=0x7f6d5c0d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:59.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.363+0000 7f6d60b89640 1 -- 172.21.15.57:0/3720948716 shutdown_connections 2023-10-19T13:00:59.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.363+0000 7f6d60b89640 1 -- 172.21.15.57:0/3720948716 wait complete. 2023-10-19T13:00:59.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.364+0000 7f6d60b89640 1 Processor -- start 2023-10-19T13:00:59.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.364+0000 7f6d60b89640 1 -- start start 2023-10-19T13:00:59.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.365+0000 7f6d60b89640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6d48003ef0 0x7f6d5c0c7e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:59.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.365+0000 7f6d60b89640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6d5c0caaa0 0x7f6d5c1f15e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:59.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.365+0000 7f6d60b89640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6d5c14fdc0 0x7f6d5c150240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:59.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.365+0000 7f6d60b89640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d5c150780 con 0x7f6d5c14fdc0 2023-10-19T13:00:59.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.365+0000 7f6d60b89640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d5c0c97d0 con 0x7f6d5c0caaa0 2023-10-19T13:00:59.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.365+0000 7f6d60b89640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d5c0c8730 con 0x7f6d48003ef0 2023-10-19T13:00:59.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.365+0000 7f6d5ad9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6d5c14fdc0 0x7f6d5c150240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:59.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.365+0000 7f6d5ad9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6d5c14fdc0 0x7f6d5c150240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38054/0 (socket says 172.21.15.57:38054) 2023-10-19T13:00:59.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.365+0000 7f6d5ad9d640 1 -- 172.21.15.57:0/290206514 learned_addr learned my addr 172.21.15.57:0/290206514 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:59.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.365+0000 7f6d53fff640 1 --2- 172.21.15.57:0/290206514 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6d5c0caaa0 0x7f6d5c1f15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:59.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.365+0000 7f6d5a59c640 1 --2- 172.21.15.57:0/290206514 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6d48003ef0 0x7f6d5c0c7e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:59.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.366+0000 7f6d5ad9d640 1 -- 172.21.15.57:0/290206514 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6d48003ef0 msgr2=0x7f6d5c0c7e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:59.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.366+0000 7f6d5ad9d640 1 --2- 172.21.15.57:0/290206514 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6d48003ef0 0x7f6d5c0c7e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:59.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.366+0000 7f6d5ad9d640 1 -- 172.21.15.57:0/290206514 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6d5c0caaa0 msgr2=0x7f6d5c1f15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:59.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.366+0000 7f6d5ad9d640 1 --2- 172.21.15.57:0/290206514 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6d5c0caaa0 0x7f6d5c1f15e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:59.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.366+0000 7f6d5ad9d640 1 -- 172.21.15.57:0/290206514 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d5c0c8a10 con 0x7f6d5c14fdc0 2023-10-19T13:00:59.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.366+0000 7f6d5a59c640 1 --2- 172.21.15.57:0/290206514 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6d48003ef0 0x7f6d5c0c7e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:00:59.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.366+0000 7f6d5ad9d640 1 --2- 172.21.15.57:0/290206514 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6d5c14fdc0 0x7f6d5c150240 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f6d5c08a120 tx=0x7f6d5400f3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:59.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.366+0000 7f6d52ffd640 1 -- 172.21.15.57:0/290206514 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d54004b20 con 0x7f6d5c14fdc0 2023-10-19T13:00:59.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.366+0000 7f6d52ffd640 1 -- 172.21.15.57:0/290206514 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f6d54004c80 con 0x7f6d5c14fdc0 2023-10-19T13:00:59.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.366+0000 7f6d52ffd640 1 -- 172.21.15.57:0/290206514 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d54003bd0 con 0x7f6d5c14fdc0 2023-10-19T13:00:59.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.366+0000 7f6d60b89640 1 -- 172.21.15.57:0/290206514 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d5c0c8cf0 con 0x7f6d5c14fdc0 2023-10-19T13:00:59.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.366+0000 7f6d60b89640 1 -- 172.21.15.57:0/290206514 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d5c0c92b0 con 0x7f6d5c14fdc0 2023-10-19T13:00:59.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.367+0000 7f6d60b89640 1 -- 172.21.15.57:0/290206514 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d5c0c94d0 con 0x7f6d5c14fdc0 2023-10-19T13:00:59.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.367+0000 7f6d52ffd640 1 -- 172.21.15.57:0/290206514 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 40) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6d54003d30 con 0x7f6d5c14fdc0 2023-10-19T13:00:59.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.368+0000 7f6d52ffd640 1 --2- 172.21.15.57:0/290206514 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f6d3c06af60 0x7f6d3c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:59.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.368+0000 7f6d52ffd640 1 -- 172.21.15.57:0/290206514 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f6d54089e10 con 0x7f6d5c14fdc0 2023-10-19T13:00:59.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.368+0000 7f6d5a59c640 1 --2- 172.21.15.57:0/290206514 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f6d3c06af60 0x7f6d3c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:59.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.369+0000 7f6d5a59c640 1 --2- 172.21.15.57:0/290206514 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f6d3c06af60 0x7f6d3c06d420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6d4c009b70 tx=0x7f6d4c005f90 comp rx=0 tx=0).ready entity=mgr.6679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:59.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.372+0000 7f6d52ffd640 1 -- 172.21.15.57:0/290206514 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6d54054bf0 con 0x7f6d5c14fdc0 2023-10-19T13:00:59.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.543+0000 7f6d60b89640 1 -- 172.21.15.57:0/290206514 --> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6d5c0d1910 con 0x7f6d3c06af60 2023-10-19T13:00:59.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.552+0000 7f6d52ffd640 1 -- 172.21.15.57:0/290206514 <== mgr.6679 v2:172.21.15.179:6832/1413162726 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289681 (secure 0 0 0) 0x7f6d4c005a70 con 0x7f6d3c06af60 2023-10-19T13:00:59.557 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:00:59.559 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2023-10-19T13:00:59.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.554+0000 7f6d60b89640 1 -- 172.21.15.57:0/290206514 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f6d3c06af60 msgr2=0x7f6d3c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:59.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.554+0000 7f6d60b89640 1 --2- 172.21.15.57:0/290206514 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f6d3c06af60 0x7f6d3c06d420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6d4c009b70 tx=0x7f6d4c005f90 comp rx=0 tx=0).stop 2023-10-19T13:00:59.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.554+0000 7f6d60b89640 1 -- 172.21.15.57:0/290206514 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6d5c14fdc0 msgr2=0x7f6d5c150240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:59.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.554+0000 7f6d60b89640 1 --2- 172.21.15.57:0/290206514 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6d5c14fdc0 0x7f6d5c150240 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f6d5c08a120 tx=0x7f6d5400f3c0 comp rx=0 tx=0).stop 2023-10-19T13:00:59.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.554+0000 7f6d60b89640 1 -- 172.21.15.57:0/290206514 shutdown_connections 2023-10-19T13:00:59.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.554+0000 7f6d60b89640 1 --2- 172.21.15.57:0/290206514 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f6d3c06af60 0x7f6d3c06d420 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:59.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.554+0000 7f6d60b89640 1 --2- 172.21.15.57:0/290206514 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6d5c14fdc0 0x7f6d5c150240 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:59.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.555+0000 7f6d60b89640 1 --2- 172.21.15.57:0/290206514 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6d5c0caaa0 0x7f6d5c1f15e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:59.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.555+0000 7f6d60b89640 1 --2- 172.21.15.57:0/290206514 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6d48003ef0 0x7f6d5c0c7e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:59.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.555+0000 7f6d60b89640 1 -- 172.21.15.57:0/290206514 >> 172.21.15.57:0/290206514 conn(0x7f6d5c0d0d60 msgr2=0x7f6d5c0b1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:59.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.555+0000 7f6d60b89640 1 -- 172.21.15.57:0/290206514 shutdown_connections 2023-10-19T13:00:59.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.555+0000 7f6d60b89640 1 -- 172.21.15.57:0/290206514 wait complete. 2023-10-19T13:00:59.568 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2023-10-19T13:00:58.082904+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":674,"num_read_kb":1321,"num_write":319,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":219,"ondisk_log_size":219,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":157,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":335204,"kb_used_data":10788,"kb_used_omap":12,"kb_used_meta":324275,"kb_avail":754639516,"statfs":{"total":773094113280,"available":772750864384,"internally_reserved":0,"allocated":11046912,"data_stored":6092505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":332058169},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001047"},"pg_stats":[{"pgid":"12.3f","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289249+0000","last_change":"2023-10-19T13:00:56.233432+0000","last_active":"2023-10-19T13:00:57.289249+0000","last_peered":"2023-10-19T13:00:57.289249+0000","last_clean":"2023-10-19T13:00:57.289249+0000","last_became_active":"2023-10-19T13:00:56.233030+0000","last_became_peered":"2023-10-19T13:00:56.233030+0000","last_unstale":"2023-10-19T13:00:57.289249+0000","last_undegraded":"2023-10-19T13:00:57.289249+0000","last_fullsized":"2023-10-19T13:00:57.289249+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:12:00.991308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3e","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289472+0000","last_change":"2023-10-19T13:00:56.214716+0000","last_active":"2023-10-19T13:00:57.289472+0000","last_peered":"2023-10-19T13:00:57.289472+0000","last_clean":"2023-10-19T13:00:57.289472+0000","last_became_active":"2023-10-19T13:00:56.214379+0000","last_became_peered":"2023-10-19T13:00:56.214379+0000","last_unstale":"2023-10-19T13:00:57.289472+0000","last_undegraded":"2023-10-19T13:00:57.289472+0000","last_fullsized":"2023-10-19T13:00:57.289472+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:23:42.593585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.3d","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289480+0000","last_change":"2023-10-19T13:00:56.230243+0000","last_active":"2023-10-19T13:00:57.289480+0000","last_peered":"2023-10-19T13:00:57.289480+0000","last_clean":"2023-10-19T13:00:57.289480+0000","last_became_active":"2023-10-19T13:00:56.229904+0000","last_became_peered":"2023-10-19T13:00:56.229904+0000","last_unstale":"2023-10-19T13:00:57.289480+0000","last_undegraded":"2023-10-19T13:00:57.289480+0000","last_fullsized":"2023-10-19T13:00:57.289480+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:10:14.580813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3c","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.210319+0000","last_change":"2023-10-19T13:00:56.229454+0000","last_active":"2023-10-19T13:00:57.210319+0000","last_peered":"2023-10-19T13:00:57.210319+0000","last_clean":"2023-10-19T13:00:57.210319+0000","last_became_active":"2023-10-19T13:00:56.229188+0000","last_became_peered":"2023-10-19T13:00:56.229188+0000","last_unstale":"2023-10-19T13:00:57.210319+0000","last_undegraded":"2023-10-19T13:00:57.210319+0000","last_fullsized":"2023-10-19T13:00:57.210319+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:25:33.751644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3b","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.209004+0000","last_change":"2023-10-19T13:00:56.229151+0000","last_active":"2023-10-19T13:00:57.209004+0000","last_peered":"2023-10-19T13:00:57.209004+0000","last_clean":"2023-10-19T13:00:57.209004+0000","last_became_active":"2023-10-19T13:00:56.228845+0000","last_became_peered":"2023-10-19T13:00:56.228845+0000","last_unstale":"2023-10-19T13:00:57.209004+0000","last_undegraded":"2023-10-19T13:00:57.209004+0000","last_fullsized":"2023-10-19T13:00:57.209004+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:58:56.570918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.3a","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.290067+0000","last_change":"2023-10-19T13:00:56.233538+0000","last_active":"2023-10-19T13:00:57.290067+0000","last_peered":"2023-10-19T13:00:57.290067+0000","last_clean":"2023-10-19T13:00:57.290067+0000","last_became_active":"2023-10-19T13:00:56.233274+0000","last_became_peered":"2023-10-19T13:00:56.233274+0000","last_unstale":"2023-10-19T13:00:57.290067+0000","last_undegraded":"2023-10-19T13:00:57.290067+0000","last_fullsized":"2023-10-19T13:00:57.290067+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:59:59.026551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.39","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289571+0000","last_change":"2023-10-19T13:00:56.230649+0000","last_active":"2023-10-19T13:00:57.289571+0000","last_peered":"2023-10-19T13:00:57.289571+0000","last_clean":"2023-10-19T13:00:57.289571+0000","last_became_active":"2023-10-19T13:00:56.230364+0000","last_became_peered":"2023-10-19T13:00:56.230364+0000","last_unstale":"2023-10-19T13:00:57.289571+0000","last_undegraded":"2023-10-19T13:00:57.289571+0000","last_fullsized":"2023-10-19T13:00:57.289571+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:03:01.128637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.38","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.208747+0000","last_change":"2023-10-19T13:00:56.229391+0000","last_active":"2023-10-19T13:00:57.208747+0000","last_peered":"2023-10-19T13:00:57.208747+0000","last_clean":"2023-10-19T13:00:57.208747+0000","last_became_active":"2023-10-19T13:00:56.229119+0000","last_became_peered":"2023-10-19T13:00:56.229119+0000","last_unstale":"2023-10-19T13:00:57.208747+0000","last_undegraded":"2023-10-19T13:00:57.208747+0000","last_fullsized":"2023-10-19T13:00:57.208747+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:35:51.543065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.37","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.210571+0000","last_change":"2023-10-19T13:00:56.233060+0000","last_active":"2023-10-19T13:00:57.210571+0000","last_peered":"2023-10-19T13:00:57.210571+0000","last_clean":"2023-10-19T13:00:57.210571+0000","last_became_active":"2023-10-19T13:00:56.232791+0000","last_became_peered":"2023-10-19T13:00:56.232791+0000","last_unstale":"2023-10-19T13:00:57.210571+0000","last_undegraded":"2023-10-19T13:00:57.210571+0000","last_fullsized":"2023-10-19T13:00:57.210571+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:47:54.774879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.36","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.210054+0000","last_change":"2023-10-19T13:00:56.230825+0000","last_active":"2023-10-19T13:00:57.210054+0000","last_peered":"2023-10-19T13:00:57.210054+0000","last_clean":"2023-10-19T13:00:57.210054+0000","last_became_active":"2023-10-19T13:00:56.230569+0000","last_became_peered":"2023-10-19T13:00:56.230569+0000","last_unstale":"2023-10-19T13:00:57.210054+0000","last_undegraded":"2023-10-19T13:00:57.210054+0000","last_fullsized":"2023-10-19T13:00:57.210054+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:22:40.229902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.35","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.290236+0000","last_change":"2023-10-19T13:00:56.216939+0000","last_active":"2023-10-19T13:00:57.290236+0000","last_peered":"2023-10-19T13:00:57.290236+0000","last_clean":"2023-10-19T13:00:57.290236+0000","last_became_active":"2023-10-19T13:00:56.216579+0000","last_became_peered":"2023-10-19T13:00:56.216579+0000","last_unstale":"2023-10-19T13:00:57.290236+0000","last_undegraded":"2023-10-19T13:00:57.290236+0000","last_fullsized":"2023-10-19T13:00:57.290236+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:51:40.006891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.34","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.210446+0000","last_change":"2023-10-19T13:00:56.222785+0000","last_active":"2023-10-19T13:00:57.210446+0000","last_peered":"2023-10-19T13:00:57.210446+0000","last_clean":"2023-10-19T13:00:57.210446+0000","last_became_active":"2023-10-19T13:00:56.222553+0000","last_became_peered":"2023-10-19T13:00:56.222553+0000","last_unstale":"2023-10-19T13:00:57.210446+0000","last_undegraded":"2023-10-19T13:00:57.210446+0000","last_fullsized":"2023-10-19T13:00:57.210446+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:41:18.979132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.33","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.209083+0000","last_change":"2023-10-19T13:00:56.226771+0000","last_active":"2023-10-19T13:00:57.209083+0000","last_peered":"2023-10-19T13:00:57.209083+0000","last_clean":"2023-10-19T13:00:57.209083+0000","last_became_active":"2023-10-19T13:00:56.226479+0000","last_became_peered":"2023-10-19T13:00:56.226479+0000","last_unstale":"2023-10-19T13:00:57.209083+0000","last_undegraded":"2023-10-19T13:00:57.209083+0000","last_fullsized":"2023-10-19T13:00:57.209083+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:52:09.841397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.32","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.290189+0000","last_change":"2023-10-19T13:00:56.233427+0000","last_active":"2023-10-19T13:00:57.290189+0000","last_peered":"2023-10-19T13:00:57.290189+0000","last_clean":"2023-10-19T13:00:57.290189+0000","last_became_active":"2023-10-19T13:00:56.233116+0000","last_became_peered":"2023-10-19T13:00:56.233116+0000","last_unstale":"2023-10-19T13:00:57.290189+0000","last_undegraded":"2023-10-19T13:00:57.290189+0000","last_fullsized":"2023-10-19T13:00:57.290189+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:35:55.138938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.31","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289012+0000","last_change":"2023-10-19T13:00:56.216064+0000","last_active":"2023-10-19T13:00:57.289012+0000","last_peered":"2023-10-19T13:00:57.289012+0000","last_clean":"2023-10-19T13:00:57.289012+0000","last_became_active":"2023-10-19T13:00:56.215751+0000","last_became_peered":"2023-10-19T13:00:56.215751+0000","last_unstale":"2023-10-19T13:00:57.289012+0000","last_undegraded":"2023-10-19T13:00:57.289012+0000","last_fullsized":"2023-10-19T13:00:57.289012+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:12:00.991308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.30","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.288930+0000","last_change":"2023-10-19T13:00:56.232927+0000","last_active":"2023-10-19T13:00:57.288930+0000","last_peered":"2023-10-19T13:00:57.288930+0000","last_clean":"2023-10-19T13:00:57.288930+0000","last_became_active":"2023-10-19T13:00:56.231096+0000","last_became_peered":"2023-10-19T13:00:56.231096+0000","last_unstale":"2023-10-19T13:00:57.288930+0000","last_undegraded":"2023-10-19T13:00:57.288930+0000","last_fullsized":"2023-10-19T13:00:57.288930+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:12:38.285374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.2f","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.209316+0000","last_change":"2023-10-19T13:00:56.238408+0000","last_active":"2023-10-19T13:00:57.209316+0000","last_peered":"2023-10-19T13:00:57.209316+0000","last_clean":"2023-10-19T13:00:57.209316+0000","last_became_active":"2023-10-19T13:00:56.238142+0000","last_became_peered":"2023-10-19T13:00:56.238142+0000","last_unstale":"2023-10-19T13:00:57.209316+0000","last_undegraded":"2023-10-19T13:00:57.209316+0000","last_fullsized":"2023-10-19T13:00:57.209316+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:46:55.376338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2e","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289920+0000","last_change":"2023-10-19T13:00:56.232174+0000","last_active":"2023-10-19T13:00:57.289920+0000","last_peered":"2023-10-19T13:00:57.289920+0000","last_clean":"2023-10-19T13:00:57.289920+0000","last_became_active":"2023-10-19T13:00:56.231904+0000","last_became_peered":"2023-10-19T13:00:56.231904+0000","last_unstale":"2023-10-19T13:00:57.289920+0000","last_undegraded":"2023-10-19T13:00:57.289920+0000","last_fullsized":"2023-10-19T13:00:57.289920+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:20:06.228901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2d","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289727+0000","last_change":"2023-10-19T13:00:56.235934+0000","last_active":"2023-10-19T13:00:57.289727+0000","last_peered":"2023-10-19T13:00:57.289727+0000","last_clean":"2023-10-19T13:00:57.289727+0000","last_became_active":"2023-10-19T13:00:56.235640+0000","last_became_peered":"2023-10-19T13:00:56.235640+0000","last_unstale":"2023-10-19T13:00:57.289727+0000","last_undegraded":"2023-10-19T13:00:57.289727+0000","last_fullsized":"2023-10-19T13:00:57.289727+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:10:09.955885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2c","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.209168+0000","last_change":"2023-10-19T13:00:56.216977+0000","last_active":"2023-10-19T13:00:57.209168+0000","last_peered":"2023-10-19T13:00:57.209168+0000","last_clean":"2023-10-19T13:00:57.209168+0000","last_became_active":"2023-10-19T13:00:56.216572+0000","last_became_peered":"2023-10-19T13:00:56.216572+0000","last_unstale":"2023-10-19T13:00:57.209168+0000","last_undegraded":"2023-10-19T13:00:57.209168+0000","last_fullsized":"2023-10-19T13:00:57.209168+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:53:29.692700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2b","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.210552+0000","last_change":"2023-10-19T13:00:56.230703+0000","last_active":"2023-10-19T13:00:57.210552+0000","last_peered":"2023-10-19T13:00:57.210552+0000","last_clean":"2023-10-19T13:00:57.210552+0000","last_became_active":"2023-10-19T13:00:56.230444+0000","last_became_peered":"2023-10-19T13:00:56.230444+0000","last_unstale":"2023-10-19T13:00:57.210552+0000","last_undegraded":"2023-10-19T13:00:57.210552+0000","last_fullsized":"2023-10-19T13:00:57.210552+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:04:40.134114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.2a","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.208797+0000","last_change":"2023-10-19T13:00:56.222547+0000","last_active":"2023-10-19T13:00:57.208797+0000","last_peered":"2023-10-19T13:00:57.208797+0000","last_clean":"2023-10-19T13:00:57.208797+0000","last_became_active":"2023-10-19T13:00:56.222285+0000","last_became_peered":"2023-10-19T13:00:56.222285+0000","last_unstale":"2023-10-19T13:00:57.208797+0000","last_undegraded":"2023-10-19T13:00:57.208797+0000","last_fullsized":"2023-10-19T13:00:57.208797+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:07:24.504133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.29","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.210666+0000","last_change":"2023-10-19T13:00:56.228216+0000","last_active":"2023-10-19T13:00:57.210666+0000","last_peered":"2023-10-19T13:00:57.210666+0000","last_clean":"2023-10-19T13:00:57.210666+0000","last_became_active":"2023-10-19T13:00:56.227905+0000","last_became_peered":"2023-10-19T13:00:56.227905+0000","last_unstale":"2023-10-19T13:00:57.210666+0000","last_undegraded":"2023-10-19T13:00:57.210666+0000","last_fullsized":"2023-10-19T13:00:57.210666+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:21:47.277500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.28","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289030+0000","last_change":"2023-10-19T13:00:56.225141+0000","last_active":"2023-10-19T13:00:57.289030+0000","last_peered":"2023-10-19T13:00:57.289030+0000","last_clean":"2023-10-19T13:00:57.289030+0000","last_became_active":"2023-10-19T13:00:56.224875+0000","last_became_peered":"2023-10-19T13:00:56.224875+0000","last_unstale":"2023-10-19T13:00:57.289030+0000","last_undegraded":"2023-10-19T13:00:57.289030+0000","last_fullsized":"2023-10-19T13:00:57.289030+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:03:05.773151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.27","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289803+0000","last_change":"2023-10-19T13:00:56.223381+0000","last_active":"2023-10-19T13:00:57.289803+0000","last_peered":"2023-10-19T13:00:57.289803+0000","last_clean":"2023-10-19T13:00:57.289803+0000","last_became_active":"2023-10-19T13:00:56.223093+0000","last_became_peered":"2023-10-19T13:00:56.223093+0000","last_unstale":"2023-10-19T13:00:57.289803+0000","last_undegraded":"2023-10-19T13:00:57.289803+0000","last_fullsized":"2023-10-19T13:00:57.289803+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:37:15.788197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.26","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.210216+0000","last_change":"2023-10-19T13:00:56.222205+0000","last_active":"2023-10-19T13:00:57.210216+0000","last_peered":"2023-10-19T13:00:57.210216+0000","last_clean":"2023-10-19T13:00:57.210216+0000","last_became_active":"2023-10-19T13:00:56.221861+0000","last_became_peered":"2023-10-19T13:00:56.221861+0000","last_unstale":"2023-10-19T13:00:57.210216+0000","last_undegraded":"2023-10-19T13:00:57.210216+0000","last_fullsized":"2023-10-19T13:00:57.210216+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:46:21.162186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.25","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.288915+0000","last_change":"2023-10-19T13:00:56.229084+0000","last_active":"2023-10-19T13:00:57.288915+0000","last_peered":"2023-10-19T13:00:57.288915+0000","last_clean":"2023-10-19T13:00:57.288915+0000","last_became_active":"2023-10-19T13:00:56.228827+0000","last_became_peered":"2023-10-19T13:00:56.228827+0000","last_unstale":"2023-10-19T13:00:57.288915+0000","last_undegraded":"2023-10-19T13:00:57.288915+0000","last_fullsized":"2023-10-19T13:00:57.288915+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:04:47.678215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.24","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.209160+0000","last_change":"2023-10-19T13:00:56.214805+0000","last_active":"2023-10-19T13:00:57.209160+0000","last_peered":"2023-10-19T13:00:57.209160+0000","last_clean":"2023-10-19T13:00:57.209160+0000","last_became_active":"2023-10-19T13:00:56.214391+0000","last_became_peered":"2023-10-19T13:00:56.214391+0000","last_unstale":"2023-10-19T13:00:57.209160+0000","last_undegraded":"2023-10-19T13:00:57.209160+0000","last_fullsized":"2023-10-19T13:00:57.209160+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:26:08.320410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.23","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289332+0000","last_change":"2023-10-19T13:00:56.231492+0000","last_active":"2023-10-19T13:00:57.289332+0000","last_peered":"2023-10-19T13:00:57.289332+0000","last_clean":"2023-10-19T13:00:57.289332+0000","last_became_active":"2023-10-19T13:00:56.231206+0000","last_became_peered":"2023-10-19T13:00:56.231206+0000","last_unstale":"2023-10-19T13:00:57.289332+0000","last_undegraded":"2023-10-19T13:00:57.289332+0000","last_fullsized":"2023-10-19T13:00:57.289332+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:02:46.194505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.22","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289007+0000","last_change":"2023-10-19T13:00:56.230353+0000","last_active":"2023-10-19T13:00:57.289007+0000","last_peered":"2023-10-19T13:00:57.289007+0000","last_clean":"2023-10-19T13:00:57.289007+0000","last_became_active":"2023-10-19T13:00:56.230030+0000","last_became_peered":"2023-10-19T13:00:56.230030+0000","last_unstale":"2023-10-19T13:00:57.289007+0000","last_undegraded":"2023-10-19T13:00:57.289007+0000","last_fullsized":"2023-10-19T13:00:57.289007+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:54:53.966112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.21","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289300+0000","last_change":"2023-10-19T13:00:56.235310+0000","last_active":"2023-10-19T13:00:57.289300+0000","last_peered":"2023-10-19T13:00:57.289300+0000","last_clean":"2023-10-19T13:00:57.289300+0000","last_became_active":"2023-10-19T13:00:56.234965+0000","last_became_peered":"2023-10-19T13:00:56.234965+0000","last_unstale":"2023-10-19T13:00:57.289300+0000","last_undegraded":"2023-10-19T13:00:57.289300+0000","last_fullsized":"2023-10-19T13:00:57.289300+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:41:21.345551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.20","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.288924+0000","last_change":"2023-10-19T13:00:56.235826+0000","last_active":"2023-10-19T13:00:57.288924+0000","last_peered":"2023-10-19T13:00:57.288924+0000","last_clean":"2023-10-19T13:00:57.288924+0000","last_became_active":"2023-10-19T13:00:56.235340+0000","last_became_peered":"2023-10-19T13:00:56.235340+0000","last_unstale":"2023-10-19T13:00:57.288924+0000","last_undegraded":"2023-10-19T13:00:57.288924+0000","last_fullsized":"2023-10-19T13:00:57.288924+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:17:54.044666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.288979+0000","last_change":"2023-10-19T13:00:56.229351+0000","last_active":"2023-10-19T13:00:57.288979+0000","last_peered":"2023-10-19T13:00:57.288979+0000","last_clean":"2023-10-19T13:00:57.288979+0000","last_became_active":"2023-10-19T13:00:56.229094+0000","last_became_peered":"2023-10-19T13:00:56.229094+0000","last_unstale":"2023-10-19T13:00:57.288979+0000","last_undegraded":"2023-10-19T13:00:57.288979+0000","last_fullsized":"2023-10-19T13:00:57.288979+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:46:48.130929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1f","version":"99'4","reported_seq":16,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.209350+0000","last_change":"2023-10-19T13:00:55.202919+0000","last_active":"2023-10-19T13:00:57.209350+0000","last_peered":"2023-10-19T13:00:57.209350+0000","last_clean":"2023-10-19T13:00:57.209350+0000","last_became_active":"2023-10-19T13:00:55.202608+0000","last_became_peered":"2023-10-19T13:00:55.202608+0000","last_unstale":"2023-10-19T13:00:57.209350+0000","last_undegraded":"2023-10-19T13:00:57.209350+0000","last_fullsized":"2023-10-19T13:00:57.209350+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_clean_scrub_stamp":"2023-10-19T13:00:54.175433+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:11:19.819288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289081+0000","last_change":"2023-10-19T13:00:56.215503+0000","last_active":"2023-10-19T13:00:57.289081+0000","last_peered":"2023-10-19T13:00:57.289081+0000","last_clean":"2023-10-19T13:00:57.289081+0000","last_became_active":"2023-10-19T13:00:56.214927+0000","last_became_peered":"2023-10-19T13:00:56.214927+0000","last_unstale":"2023-10-19T13:00:57.289081+0000","last_undegraded":"2023-10-19T13:00:57.289081+0000","last_fullsized":"2023-10-19T13:00:57.289081+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:07:30.083579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"99'1","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289093+0000","last_change":"2023-10-19T13:00:55.201299+0000","last_active":"2023-10-19T13:00:57.289093+0000","last_peered":"2023-10-19T13:00:57.289093+0000","last_clean":"2023-10-19T13:00:57.289093+0000","last_became_active":"2023-10-19T13:00:55.201009+0000","last_became_peered":"2023-10-19T13:00:55.201009+0000","last_unstale":"2023-10-19T13:00:57.289093+0000","last_undegraded":"2023-10-19T13:00:57.289093+0000","last_fullsized":"2023-10-19T13:00:57.289093+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_clean_scrub_stamp":"2023-10-19T13:00:54.175433+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:37:20.825852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.210709+0000","last_change":"2023-10-19T13:00:56.235410+0000","last_active":"2023-10-19T13:00:57.210709+0000","last_peered":"2023-10-19T13:00:57.210709+0000","last_clean":"2023-10-19T13:00:57.210709+0000","last_became_active":"2023-10-19T13:00:56.233880+0000","last_became_peered":"2023-10-19T13:00:56.233880+0000","last_unstale":"2023-10-19T13:00:57.210709+0000","last_undegraded":"2023-10-19T13:00:57.210709+0000","last_fullsized":"2023-10-19T13:00:57.210709+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:03:01.128637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"99'2","reported_seq":14,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289055+0000","last_change":"2023-10-19T13:00:55.207421+0000","last_active":"2023-10-19T13:00:57.289055+0000","last_peered":"2023-10-19T13:00:57.289055+0000","last_clean":"2023-10-19T13:00:57.289055+0000","last_became_active":"2023-10-19T13:00:55.207085+0000","last_became_peered":"2023-10-19T13:00:55.207085+0000","last_unstale":"2023-10-19T13:00:57.289055+0000","last_undegraded":"2023-10-19T13:00:57.289055+0000","last_fullsized":"2023-10-19T13:00:57.289055+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_clean_scrub_stamp":"2023-10-19T13:00:54.175433+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:19:24.124448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.210717+0000","last_change":"2023-10-19T13:00:56.230156+0000","last_active":"2023-10-19T13:00:57.210717+0000","last_peered":"2023-10-19T13:00:57.210717+0000","last_clean":"2023-10-19T13:00:57.210717+0000","last_became_active":"2023-10-19T13:00:56.229837+0000","last_became_peered":"2023-10-19T13:00:56.229837+0000","last_unstale":"2023-10-19T13:00:57.210717+0000","last_undegraded":"2023-10-19T13:00:57.210717+0000","last_fullsized":"2023-10-19T13:00:57.210717+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:25:26.972552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1c","version":"99'2","reported_seq":14,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289683+0000","last_change":"2023-10-19T13:00:55.200397+0000","last_active":"2023-10-19T13:00:57.289683+0000","last_peered":"2023-10-19T13:00:57.289683+0000","last_clean":"2023-10-19T13:00:57.289683+0000","last_became_active":"2023-10-19T13:00:55.200138+0000","last_became_peered":"2023-10-19T13:00:55.200138+0000","last_unstale":"2023-10-19T13:00:57.289683+0000","last_undegraded":"2023-10-19T13:00:57.289683+0000","last_fullsized":"2023-10-19T13:00:57.289683+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_clean_scrub_stamp":"2023-10-19T13:00:54.175433+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:04:39.128200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.290121+0000","last_change":"2023-10-19T13:00:56.216180+0000","last_active":"2023-10-19T13:00:57.290121+0000","last_peered":"2023-10-19T13:00:57.290121+0000","last_clean":"2023-10-19T13:00:57.290121+0000","last_became_active":"2023-10-19T13:00:56.215725+0000","last_became_peered":"2023-10-19T13:00:56.215725+0000","last_unstale":"2023-10-19T13:00:57.290121+0000","last_undegraded":"2023-10-19T13:00:57.290121+0000","last_fullsized":"2023-10-19T13:00:57.290121+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:41:50.788069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"99'1","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.210799+0000","last_change":"2023-10-19T13:00:55.200646+0000","last_active":"2023-10-19T13:00:57.210799+0000","last_peered":"2023-10-19T13:00:57.210799+0000","last_clean":"2023-10-19T13:00:57.210799+0000","last_became_active":"2023-10-19T13:00:55.200395+0000","last_became_peered":"2023-10-19T13:00:55.200395+0000","last_unstale":"2023-10-19T13:00:57.210799+0000","last_undegraded":"2023-10-19T13:00:57.210799+0000","last_fullsized":"2023-10-19T13:00:57.210799+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_clean_scrub_stamp":"2023-10-19T13:00:54.175433+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:33:54.831822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.290120+0000","last_change":"2023-10-19T13:00:56.228533+0000","last_active":"2023-10-19T13:00:57.290120+0000","last_peered":"2023-10-19T13:00:57.290120+0000","last_clean":"2023-10-19T13:00:57.290120+0000","last_became_active":"2023-10-19T13:00:56.228198+0000","last_became_peered":"2023-10-19T13:00:56.228198+0000","last_unstale":"2023-10-19T13:00:57.290120+0000","last_undegraded":"2023-10-19T13:00:57.290120+0000","last_fullsized":"2023-10-19T13:00:57.290120+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:49:31.042667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"99'1","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.209888+0000","last_change":"2023-10-19T13:00:55.192914+0000","last_active":"2023-10-19T13:00:57.209888+0000","last_peered":"2023-10-19T13:00:57.209888+0000","last_clean":"2023-10-19T13:00:57.209888+0000","last_became_active":"2023-10-19T13:00:55.192577+0000","last_became_peered":"2023-10-19T13:00:55.192577+0000","last_unstale":"2023-10-19T13:00:57.209888+0000","last_undegraded":"2023-10-19T13:00:57.209888+0000","last_fullsized":"2023-10-19T13:00:57.209888+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_clean_scrub_stamp":"2023-10-19T13:00:54.175433+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:19:24.124448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.210511+0000","last_change":"2023-10-19T13:00:56.235761+0000","last_active":"2023-10-19T13:00:57.210511+0000","last_peered":"2023-10-19T13:00:57.210511+0000","last_clean":"2023-10-19T13:00:57.210511+0000","last_became_active":"2023-10-19T13:00:56.235508+0000","last_became_peered":"2023-10-19T13:00:56.235508+0000","last_unstale":"2023-10-19T13:00:57.210511+0000","last_undegraded":"2023-10-19T13:00:57.210511+0000","last_fullsized":"2023-10-19T13:00:57.210511+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:52:32.379639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":12,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289167+0000","last_change":"2023-10-19T13:00:55.205226+0000","last_active":"2023-10-19T13:00:57.289167+0000","last_peered":"2023-10-19T13:00:57.289167+0000","last_clean":"2023-10-19T13:00:57.289167+0000","last_became_active":"2023-10-19T13:00:55.204316+0000","last_became_peered":"2023-10-19T13:00:55.204316+0000","last_unstale":"2023-10-19T13:00:57.289167+0000","last_undegraded":"2023-10-19T13:00:57.289167+0000","last_fullsized":"2023-10-19T13:00:57.289167+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_clean_scrub_stamp":"2023-10-19T13:00:54.175433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:04:46.672302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.290001+0000","last_change":"2023-10-19T13:00:56.231969+0000","last_active":"2023-10-19T13:00:57.290001+0000","last_peered":"2023-10-19T13:00:57.290001+0000","last_clean":"2023-10-19T13:00:57.290001+0000","last_became_active":"2023-10-19T13:00:56.231658+0000","last_became_peered":"2023-10-19T13:00:56.231658+0000","last_unstale":"2023-10-19T13:00:57.290001+0000","last_undegraded":"2023-10-19T13:00:57.290001+0000","last_fullsized":"2023-10-19T13:00:57.290001+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:54:37.924182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":12,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.208757+0000","last_change":"2023-10-19T13:00:55.200476+0000","last_active":"2023-10-19T13:00:57.208757+0000","last_peered":"2023-10-19T13:00:57.208757+0000","last_clean":"2023-10-19T13:00:57.208757+0000","last_became_active":"2023-10-19T13:00:55.200204+0000","last_became_peered":"2023-10-19T13:00:55.200204+0000","last_unstale":"2023-10-19T13:00:57.208757+0000","last_undegraded":"2023-10-19T13:00:57.208757+0000","last_fullsized":"2023-10-19T13:00:57.208757+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_clean_scrub_stamp":"2023-10-19T13:00:54.175433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:57:00.285741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289004+0000","last_change":"2023-10-19T13:00:56.235688+0000","last_active":"2023-10-19T13:00:57.289004+0000","last_peered":"2023-10-19T13:00:57.289004+0000","last_clean":"2023-10-19T13:00:57.289004+0000","last_became_active":"2023-10-19T13:00:56.235370+0000","last_became_peered":"2023-10-19T13:00:56.235370+0000","last_unstale":"2023-10-19T13:00:57.289004+0000","last_undegraded":"2023-10-19T13:00:57.289004+0000","last_fullsized":"2023-10-19T13:00:57.289004+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:56:34.119416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":12,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289245+0000","last_change":"2023-10-19T13:00:55.200716+0000","last_active":"2023-10-19T13:00:57.289245+0000","last_peered":"2023-10-19T13:00:57.289245+0000","last_clean":"2023-10-19T13:00:57.289245+0000","last_became_active":"2023-10-19T13:00:55.200457+0000","last_became_peered":"2023-10-19T13:00:55.200457+0000","last_unstale":"2023-10-19T13:00:57.289245+0000","last_undegraded":"2023-10-19T13:00:57.289245+0000","last_fullsized":"2023-10-19T13:00:57.289245+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_clean_scrub_stamp":"2023-10-19T13:00:54.175433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:18:53.973512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.210415+0000","last_change":"2023-10-19T13:00:56.229746+0000","last_active":"2023-10-19T13:00:57.210415+0000","last_peered":"2023-10-19T13:00:57.210415+0000","last_clean":"2023-10-19T13:00:57.210415+0000","last_became_active":"2023-10-19T13:00:56.229483+0000","last_became_peered":"2023-10-19T13:00:56.229483+0000","last_unstale":"2023-10-19T13:00:57.210415+0000","last_undegraded":"2023-10-19T13:00:57.210415+0000","last_fullsized":"2023-10-19T13:00:57.210415+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:31:05.271554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":12,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.210314+0000","last_change":"2023-10-19T13:00:55.201725+0000","last_active":"2023-10-19T13:00:57.210314+0000","last_peered":"2023-10-19T13:00:57.210314+0000","last_clean":"2023-10-19T13:00:57.210314+0000","last_became_active":"2023-10-19T13:00:55.201444+0000","last_became_peered":"2023-10-19T13:00:55.201444+0000","last_unstale":"2023-10-19T13:00:57.210314+0000","last_undegraded":"2023-10-19T13:00:57.210314+0000","last_fullsized":"2023-10-19T13:00:57.210314+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_clean_scrub_stamp":"2023-10-19T13:00:54.175433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:18:53.973512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.290282+0000","last_change":"2023-10-19T13:00:56.225547+0000","last_active":"2023-10-19T13:00:57.290282+0000","last_peered":"2023-10-19T13:00:57.290282+0000","last_clean":"2023-10-19T13:00:57.290282+0000","last_became_active":"2023-10-19T13:00:56.225270+0000","last_became_peered":"2023-10-19T13:00:56.225270+0000","last_unstale":"2023-10-19T13:00:57.290282+0000","last_undegraded":"2023-10-19T13:00:57.290282+0000","last_fullsized":"2023-10-19T13:00:57.290282+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:58:43.686419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"99'3","reported_seq":15,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.209326+0000","last_change":"2023-10-19T13:00:55.207091+0000","last_active":"2023-10-19T13:00:57.209326+0000","last_peered":"2023-10-19T13:00:57.209326+0000","last_clean":"2023-10-19T13:00:57.209326+0000","last_became_active":"2023-10-19T13:00:55.206828+0000","last_became_peered":"2023-10-19T13:00:55.206828+0000","last_unstale":"2023-10-19T13:00:57.209326+0000","last_undegraded":"2023-10-19T13:00:57.209326+0000","last_fullsized":"2023-10-19T13:00:57.209326+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_clean_scrub_stamp":"2023-10-19T13:00:54.175433+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:15:16.920840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289516+0000","last_change":"2023-10-19T13:00:56.224907+0000","last_active":"2023-10-19T13:00:57.289516+0000","last_peered":"2023-10-19T13:00:57.289516+0000","last_clean":"2023-10-19T13:00:57.289516+0000","last_became_active":"2023-10-19T13:00:56.224629+0000","last_became_peered":"2023-10-19T13:00:56.224629+0000","last_unstale":"2023-10-19T13:00:57.289516+0000","last_undegraded":"2023-10-19T13:00:57.289516+0000","last_fullsized":"2023-10-19T13:00:57.289516+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:12:40.484752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"99'6","reported_seq":18,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289725+0000","last_change":"2023-10-19T13:00:55.198821+0000","last_active":"2023-10-19T13:00:57.289725+0000","last_peered":"2023-10-19T13:00:57.289725+0000","last_clean":"2023-10-19T13:00:57.289725+0000","last_became_active":"2023-10-19T13:00:55.198475+0000","last_became_peered":"2023-10-19T13:00:55.198475+0000","last_unstale":"2023-10-19T13:00:57.289725+0000","last_undegraded":"2023-10-19T13:00:57.289725+0000","last_fullsized":"2023-10-19T13:00:57.289725+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_clean_scrub_stamp":"2023-10-19T13:00:54.175433+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:47:53.768966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.210567+0000","last_change":"2023-10-19T13:00:56.221947+0000","last_active":"2023-10-19T13:00:57.210567+0000","last_peered":"2023-10-19T13:00:57.210567+0000","last_clean":"2023-10-19T13:00:57.210567+0000","last_became_active":"2023-10-19T13:00:56.221660+0000","last_became_peered":"2023-10-19T13:00:56.221660+0000","last_unstale":"2023-10-19T13:00:57.210567+0000","last_undegraded":"2023-10-19T13:00:57.210567+0000","last_fullsized":"2023-10-19T13:00:57.210567+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:04:47.678215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.13","version":"99'2","reported_seq":14,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.210007+0000","last_change":"2023-10-19T13:00:55.193949+0000","last_active":"2023-10-19T13:00:57.210007+0000","last_peered":"2023-10-19T13:00:57.210007+0000","last_clean":"2023-10-19T13:00:57.210007+0000","last_became_active":"2023-10-19T13:00:55.193676+0000","last_became_peered":"2023-10-19T13:00:55.193676+0000","last_unstale":"2023-10-19T13:00:57.210007+0000","last_undegraded":"2023-10-19T13:00:57.210007+0000","last_fullsized":"2023-10-19T13:00:57.210007+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_clean_scrub_stamp":"2023-10-19T13:00:54.175433+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:37:20.825852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289806+0000","last_change":"2023-10-19T13:00:56.231757+0000","last_active":"2023-10-19T13:00:57.289806+0000","last_peered":"2023-10-19T13:00:57.289806+0000","last_clean":"2023-10-19T13:00:57.289806+0000","last_became_active":"2023-10-19T13:00:56.231488+0000","last_became_peered":"2023-10-19T13:00:56.231488+0000","last_unstale":"2023-10-19T13:00:57.289806+0000","last_undegraded":"2023-10-19T13:00:57.289806+0000","last_fullsized":"2023-10-19T13:00:57.289806+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:46:48.130929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":12,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289087+0000","last_change":"2023-10-19T13:00:55.202887+0000","last_active":"2023-10-19T13:00:57.289087+0000","last_peered":"2023-10-19T13:00:57.289087+0000","last_clean":"2023-10-19T13:00:57.289087+0000","last_became_active":"2023-10-19T13:00:55.202560+0000","last_became_peered":"2023-10-19T13:00:55.202560+0000","last_unstale":"2023-10-19T13:00:57.289087+0000","last_undegraded":"2023-10-19T13:00:57.289087+0000","last_fullsized":"2023-10-19T13:00:57.289087+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_clean_scrub_stamp":"2023-10-19T13:00:54.175433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:25:32.745730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289882+0000","last_change":"2023-10-19T13:00:56.235887+0000","last_active":"2023-10-19T13:00:57.289882+0000","last_peered":"2023-10-19T13:00:57.289882+0000","last_clean":"2023-10-19T13:00:57.289882+0000","last_became_active":"2023-10-19T13:00:56.235636+0000","last_became_peered":"2023-10-19T13:00:56.235636+0000","last_unstale":"2023-10-19T13:00:57.289882+0000","last_undegraded":"2023-10-19T13:00:57.289882+0000","last_fullsized":"2023-10-19T13:00:57.289882+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:10:14.580813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":12,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.208793+0000","last_change":"2023-10-19T13:00:55.201275+0000","last_active":"2023-10-19T13:00:57.208793+0000","last_peered":"2023-10-19T13:00:57.208793+0000","last_clean":"2023-10-19T13:00:57.208793+0000","last_became_active":"2023-10-19T13:00:55.201017+0000","last_became_peered":"2023-10-19T13:00:55.201017+0000","last_unstale":"2023-10-19T13:00:57.208793+0000","last_undegraded":"2023-10-19T13:00:57.208793+0000","last_fullsized":"2023-10-19T13:00:57.208793+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_clean_scrub_stamp":"2023-10-19T13:00:54.175433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:49:39.307408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289972+0000","last_change":"2023-10-19T13:00:56.234225+0000","last_active":"2023-10-19T13:00:57.289972+0000","last_peered":"2023-10-19T13:00:57.289972+0000","last_clean":"2023-10-19T13:00:57.289972+0000","last_became_active":"2023-10-19T13:00:56.233546+0000","last_became_peered":"2023-10-19T13:00:56.233546+0000","last_unstale":"2023-10-19T13:00:57.289972+0000","last_undegraded":"2023-10-19T13:00:57.289972+0000","last_fullsized":"2023-10-19T13:00:57.289972+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:12:40.484752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"99'8","reported_seq":20,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289107+0000","last_change":"2023-10-19T13:00:55.197229+0000","last_active":"2023-10-19T13:00:57.289107+0000","last_peered":"2023-10-19T13:00:57.289107+0000","last_clean":"2023-10-19T13:00:57.289107+0000","last_became_active":"2023-10-19T13:00:55.196966+0000","last_became_peered":"2023-10-19T13:00:55.196966+0000","last_unstale":"2023-10-19T13:00:57.289107+0000","last_undegraded":"2023-10-19T13:00:57.289107+0000","last_fullsized":"2023-10-19T13:00:57.289107+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_clean_scrub_stamp":"2023-10-19T13:00:54.175433+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:12:37.279461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289682+0000","last_change":"2023-10-19T13:00:56.229823+0000","last_active":"2023-10-19T13:00:57.289682+0000","last_peered":"2023-10-19T13:00:57.289682+0000","last_clean":"2023-10-19T13:00:57.289682+0000","last_became_active":"2023-10-19T13:00:56.229475+0000","last_became_peered":"2023-10-19T13:00:56.229475+0000","last_unstale":"2023-10-19T13:00:57.289682+0000","last_undegraded":"2023-10-19T13:00:57.289682+0000","last_fullsized":"2023-10-19T13:00:57.289682+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:18:17.438467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"99'1","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289306+0000","last_change":"2023-10-19T13:00:55.208633+0000","last_active":"2023-10-19T13:00:57.289306+0000","last_peered":"2023-10-19T13:00:57.289306+0000","last_clean":"2023-10-19T13:00:57.289306+0000","last_became_active":"2023-10-19T13:00:55.208365+0000","last_became_peered":"2023-10-19T13:00:55.208365+0000","last_unstale":"2023-10-19T13:00:57.289306+0000","last_undegraded":"2023-10-19T13:00:57.289306+0000","last_fullsized":"2023-10-19T13:00:57.289306+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_clean_scrub_stamp":"2023-10-19T13:00:54.175433+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:22:39.223988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.208758+0000","last_change":"2023-10-19T13:00:56.237212+0000","last_active":"2023-10-19T13:00:57.208758+0000","last_peered":"2023-10-19T13:00:57.208758+0000","last_clean":"2023-10-19T13:00:57.208758+0000","last_became_active":"2023-10-19T13:00:56.236953+0000","last_became_peered":"2023-10-19T13:00:56.236953+0000","last_unstale":"2023-10-19T13:00:57.208758+0000","last_undegraded":"2023-10-19T13:00:57.208758+0000","last_fullsized":"2023-10-19T13:00:57.208758+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:25:59.221155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"99'2","reported_seq":14,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.210239+0000","last_change":"2023-10-19T13:00:55.204299+0000","last_active":"2023-10-19T13:00:57.210239+0000","last_peered":"2023-10-19T13:00:57.210239+0000","last_clean":"2023-10-19T13:00:57.210239+0000","last_became_active":"2023-10-19T13:00:55.203996+0000","last_became_peered":"2023-10-19T13:00:55.203996+0000","last_unstale":"2023-10-19T13:00:57.210239+0000","last_undegraded":"2023-10-19T13:00:57.210239+0000","last_fullsized":"2023-10-19T13:00:57.210239+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_clean_scrub_stamp":"2023-10-19T13:00:54.175433+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:11:31.723870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.208964+0000","last_change":"2023-10-19T13:00:56.214727+0000","last_active":"2023-10-19T13:00:57.208964+0000","last_peered":"2023-10-19T13:00:57.208964+0000","last_clean":"2023-10-19T13:00:57.208964+0000","last_became_active":"2023-10-19T13:00:56.214204+0000","last_became_peered":"2023-10-19T13:00:56.214204+0000","last_unstale":"2023-10-19T13:00:57.208964+0000","last_undegraded":"2023-10-19T13:00:57.208964+0000","last_fullsized":"2023-10-19T13:00:57.208964+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:06:04.196145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"99'1","reported_seq":13,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.209384+0000","last_change":"2023-10-19T13:00:55.199355+0000","last_active":"2023-10-19T13:00:57.209384+0000","last_peered":"2023-10-19T13:00:57.209384+0000","last_clean":"2023-10-19T13:00:57.209384+0000","last_became_active":"2023-10-19T13:00:55.199080+0000","last_became_peered":"2023-10-19T13:00:55.199080+0000","last_unstale":"2023-10-19T13:00:57.209384+0000","last_undegraded":"2023-10-19T13:00:57.209384+0000","last_fullsized":"2023-10-19T13:00:57.209384+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_clean_scrub_stamp":"2023-10-19T13:00:54.175433+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:40:53.397823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289642+0000","last_change":"2023-10-19T13:00:56.230576+0000","last_active":"2023-10-19T13:00:57.289642+0000","last_peered":"2023-10-19T13:00:57.289642+0000","last_clean":"2023-10-19T13:00:57.289642+0000","last_became_active":"2023-10-19T13:00:56.230253+0000","last_became_peered":"2023-10-19T13:00:56.230253+0000","last_unstale":"2023-10-19T13:00:57.289642+0000","last_undegraded":"2023-10-19T13:00:57.289642+0000","last_fullsized":"2023-10-19T13:00:57.289642+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:51:21.587669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":12,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289306+0000","last_change":"2023-10-19T13:00:55.208450+0000","last_active":"2023-10-19T13:00:57.289306+0000","last_peered":"2023-10-19T13:00:57.289306+0000","last_clean":"2023-10-19T13:00:57.289306+0000","last_became_active":"2023-10-19T13:00:55.208166+0000","last_became_peered":"2023-10-19T13:00:55.208166+0000","last_unstale":"2023-10-19T13:00:57.289306+0000","last_undegraded":"2023-10-19T13:00:57.289306+0000","last_fullsized":"2023-10-19T13:00:57.289306+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_clean_scrub_stamp":"2023-10-19T13:00:54.175433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:25:25.966638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289969+0000","last_change":"2023-10-19T13:00:56.215279+0000","last_active":"2023-10-19T13:00:57.289969+0000","last_peered":"2023-10-19T13:00:57.289969+0000","last_clean":"2023-10-19T13:00:57.289969+0000","last_became_active":"2023-10-19T13:00:56.214996+0000","last_became_peered":"2023-10-19T13:00:56.214996+0000","last_unstale":"2023-10-19T13:00:57.289969+0000","last_undegraded":"2023-10-19T13:00:57.289969+0000","last_fullsized":"2023-10-19T13:00:57.289969+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:51:28.535991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"99'2","reported_seq":14,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289739+0000","last_change":"2023-10-19T13:00:55.204983+0000","last_active":"2023-10-19T13:00:57.289739+0000","last_peered":"2023-10-19T13:00:57.289739+0000","last_clean":"2023-10-19T13:00:57.289739+0000","last_became_active":"2023-10-19T13:00:55.204734+0000","last_became_peered":"2023-10-19T13:00:55.204734+0000","last_unstale":"2023-10-19T13:00:57.289739+0000","last_undegraded":"2023-10-19T13:00:57.289739+0000","last_fullsized":"2023-10-19T13:00:57.289739+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_clean_scrub_stamp":"2023-10-19T13:00:54.175433+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:41:49.782155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289593+0000","last_change":"2023-10-19T13:00:56.232216+0000","last_active":"2023-10-19T13:00:57.289593+0000","last_peered":"2023-10-19T13:00:57.289593+0000","last_clean":"2023-10-19T13:00:57.289593+0000","last_became_active":"2023-10-19T13:00:56.231956+0000","last_became_peered":"2023-10-19T13:00:56.231956+0000","last_unstale":"2023-10-19T13:00:57.289593+0000","last_undegraded":"2023-10-19T13:00:57.289593+0000","last_fullsized":"2023-10-19T13:00:57.289593+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:14:01.227382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":12,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289576+0000","last_change":"2023-10-19T13:00:55.198778+0000","last_active":"2023-10-19T13:00:57.289576+0000","last_peered":"2023-10-19T13:00:57.289576+0000","last_clean":"2023-10-19T13:00:57.289576+0000","last_became_active":"2023-10-19T13:00:55.196076+0000","last_became_peered":"2023-10-19T13:00:55.196076+0000","last_unstale":"2023-10-19T13:00:57.289576+0000","last_undegraded":"2023-10-19T13:00:57.289576+0000","last_fullsized":"2023-10-19T13:00:57.289576+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_clean_scrub_stamp":"2023-10-19T13:00:54.175433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:49:30.036753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.290048+0000","last_change":"2023-10-19T13:00:56.232810+0000","last_active":"2023-10-19T13:00:57.290048+0000","last_peered":"2023-10-19T13:00:57.290048+0000","last_clean":"2023-10-19T13:00:57.290048+0000","last_became_active":"2023-10-19T13:00:56.232551+0000","last_became_peered":"2023-10-19T13:00:56.232551+0000","last_unstale":"2023-10-19T13:00:57.290048+0000","last_undegraded":"2023-10-19T13:00:57.290048+0000","last_fullsized":"2023-10-19T13:00:57.290048+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:20:06.228901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":12,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289761+0000","last_change":"2023-10-19T13:00:55.199409+0000","last_active":"2023-10-19T13:00:57.289761+0000","last_peered":"2023-10-19T13:00:57.289761+0000","last_clean":"2023-10-19T13:00:57.289761+0000","last_became_active":"2023-10-19T13:00:55.198868+0000","last_became_peered":"2023-10-19T13:00:55.198868+0000","last_unstale":"2023-10-19T13:00:57.289761+0000","last_undegraded":"2023-10-19T13:00:57.289761+0000","last_fullsized":"2023-10-19T13:00:57.289761+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_clean_scrub_stamp":"2023-10-19T13:00:54.175433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:51:27.530077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.210496+0000","last_change":"2023-10-19T13:00:56.232955+0000","last_active":"2023-10-19T13:00:57.210496+0000","last_peered":"2023-10-19T13:00:57.210496+0000","last_clean":"2023-10-19T13:00:57.210496+0000","last_became_active":"2023-10-19T13:00:56.232696+0000","last_became_peered":"2023-10-19T13:00:56.232696+0000","last_unstale":"2023-10-19T13:00:57.210496+0000","last_undegraded":"2023-10-19T13:00:57.210496+0000","last_fullsized":"2023-10-19T13:00:57.210496+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:30:02.513725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"99'2","reported_seq":14,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.209942+0000","last_change":"2023-10-19T13:00:55.208659+0000","last_active":"2023-10-19T13:00:57.209942+0000","last_peered":"2023-10-19T13:00:57.209942+0000","last_clean":"2023-10-19T13:00:57.209942+0000","last_became_active":"2023-10-19T13:00:55.208333+0000","last_became_peered":"2023-10-19T13:00:55.208333+0000","last_unstale":"2023-10-19T13:00:57.209942+0000","last_undegraded":"2023-10-19T13:00:57.209942+0000","last_fullsized":"2023-10-19T13:00:57.209942+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_clean_scrub_stamp":"2023-10-19T13:00:54.175433+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:18:31.875925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289570+0000","last_change":"2023-10-19T13:00:56.229810+0000","last_active":"2023-10-19T13:00:57.289570+0000","last_peered":"2023-10-19T13:00:57.289570+0000","last_clean":"2023-10-19T13:00:57.289570+0000","last_became_active":"2023-10-19T13:00:56.229381+0000","last_became_peered":"2023-10-19T13:00:56.229381+0000","last_unstale":"2023-10-19T13:00:57.289570+0000","last_undegraded":"2023-10-19T13:00:57.289570+0000","last_fullsized":"2023-10-19T13:00:57.289570+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:25:26.972552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":12,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289673+0000","last_change":"2023-10-19T13:00:55.205267+0000","last_active":"2023-10-19T13:00:57.289673+0000","last_peered":"2023-10-19T13:00:57.289673+0000","last_clean":"2023-10-19T13:00:57.289673+0000","last_became_active":"2023-10-19T13:00:55.204979+0000","last_became_peered":"2023-10-19T13:00:55.204979+0000","last_unstale":"2023-10-19T13:00:57.289673+0000","last_undegraded":"2023-10-19T13:00:57.289673+0000","last_fullsized":"2023-10-19T13:00:57.289673+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_clean_scrub_stamp":"2023-10-19T13:00:54.175433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:17:53.038752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.210532+0000","last_change":"2023-10-19T13:00:56.233768+0000","last_active":"2023-10-19T13:00:57.210532+0000","last_peered":"2023-10-19T13:00:57.210532+0000","last_clean":"2023-10-19T13:00:57.210532+0000","last_became_active":"2023-10-19T13:00:56.233427+0000","last_became_peered":"2023-10-19T13:00:56.233427+0000","last_unstale":"2023-10-19T13:00:57.210532+0000","last_undegraded":"2023-10-19T13:00:57.210532+0000","last_fullsized":"2023-10-19T13:00:57.210532+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:07:12.067905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":12,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289197+0000","last_change":"2023-10-19T13:00:55.200925+0000","last_active":"2023-10-19T13:00:57.289197+0000","last_peered":"2023-10-19T13:00:57.289197+0000","last_clean":"2023-10-19T13:00:57.289197+0000","last_became_active":"2023-10-19T13:00:55.200653+0000","last_became_peered":"2023-10-19T13:00:55.200653+0000","last_unstale":"2023-10-19T13:00:57.289197+0000","last_undegraded":"2023-10-19T13:00:57.289197+0000","last_fullsized":"2023-10-19T13:00:57.289197+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_clean_scrub_stamp":"2023-10-19T13:00:54.175433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:10:01.251030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289454+0000","last_change":"2023-10-19T13:00:56.230487+0000","last_active":"2023-10-19T13:00:57.289454+0000","last_peered":"2023-10-19T13:00:57.289454+0000","last_clean":"2023-10-19T13:00:57.289454+0000","last_became_active":"2023-10-19T13:00:56.230232+0000","last_became_peered":"2023-10-19T13:00:56.230232+0000","last_unstale":"2023-10-19T13:00:57.289454+0000","last_undegraded":"2023-10-19T13:00:57.289454+0000","last_fullsized":"2023-10-19T13:00:57.289454+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:40:00.647749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":12,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289102+0000","last_change":"2023-10-19T13:00:55.203346+0000","last_active":"2023-10-19T13:00:57.289102+0000","last_peered":"2023-10-19T13:00:57.289102+0000","last_clean":"2023-10-19T13:00:57.289102+0000","last_became_active":"2023-10-19T13:00:55.203048+0000","last_became_peered":"2023-10-19T13:00:55.203048+0000","last_unstale":"2023-10-19T13:00:57.289102+0000","last_undegraded":"2023-10-19T13:00:57.289102+0000","last_fullsized":"2023-10-19T13:00:57.289102+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_clean_scrub_stamp":"2023-10-19T13:00:54.175433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:02:44.720415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289537+0000","last_change":"2023-10-19T13:00:56.234907+0000","last_active":"2023-10-19T13:00:57.289537+0000","last_peered":"2023-10-19T13:00:57.289537+0000","last_clean":"2023-10-19T13:00:57.289537+0000","last_became_active":"2023-10-19T13:00:56.234653+0000","last_became_peered":"2023-10-19T13:00:56.234653+0000","last_unstale":"2023-10-19T13:00:57.289537+0000","last_undegraded":"2023-10-19T13:00:57.289537+0000","last_fullsized":"2023-10-19T13:00:57.289537+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:21:47.277500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":12,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289429+0000","last_change":"2023-10-19T13:00:55.202434+0000","last_active":"2023-10-19T13:00:57.289429+0000","last_peered":"2023-10-19T13:00:57.289429+0000","last_clean":"2023-10-19T13:00:57.289429+0000","last_became_active":"2023-10-19T13:00:55.202018+0000","last_became_peered":"2023-10-19T13:00:55.202018+0000","last_unstale":"2023-10-19T13:00:57.289429+0000","last_undegraded":"2023-10-19T13:00:57.289429+0000","last_fullsized":"2023-10-19T13:00:57.289429+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_clean_scrub_stamp":"2023-10-19T13:00:54.175433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:08:07.333292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289833+0000","last_change":"2023-10-19T13:00:56.233959+0000","last_active":"2023-10-19T13:00:57.289833+0000","last_peered":"2023-10-19T13:00:57.289833+0000","last_clean":"2023-10-19T13:00:57.289833+0000","last_became_active":"2023-10-19T13:00:56.233404+0000","last_became_peered":"2023-10-19T13:00:56.233404+0000","last_unstale":"2023-10-19T13:00:57.289833+0000","last_undegraded":"2023-10-19T13:00:57.289833+0000","last_fullsized":"2023-10-19T13:00:57.289833+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:41:21.345551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":12,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.208909+0000","last_change":"2023-10-19T13:00:55.201999+0000","last_active":"2023-10-19T13:00:57.208909+0000","last_peered":"2023-10-19T13:00:57.208909+0000","last_clean":"2023-10-19T13:00:57.208909+0000","last_became_active":"2023-10-19T13:00:55.201737+0000","last_became_peered":"2023-10-19T13:00:55.201737+0000","last_unstale":"2023-10-19T13:00:57.208909+0000","last_undegraded":"2023-10-19T13:00:57.208909+0000","last_fullsized":"2023-10-19T13:00:57.208909+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_clean_scrub_stamp":"2023-10-19T13:00:54.175433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:29:43.561077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.290038+0000","last_change":"2023-10-19T13:00:56.222137+0000","last_active":"2023-10-19T13:00:57.290038+0000","last_peered":"2023-10-19T13:00:57.290038+0000","last_clean":"2023-10-19T13:00:57.290038+0000","last_became_active":"2023-10-19T13:00:56.221851+0000","last_became_peered":"2023-10-19T13:00:56.221851+0000","last_unstale":"2023-10-19T13:00:57.290038+0000","last_undegraded":"2023-10-19T13:00:57.290038+0000","last_fullsized":"2023-10-19T13:00:57.290038+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:45:02.161382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":12,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289630+0000","last_change":"2023-10-19T13:00:55.200490+0000","last_active":"2023-10-19T13:00:57.289630+0000","last_peered":"2023-10-19T13:00:57.289630+0000","last_clean":"2023-10-19T13:00:57.289630+0000","last_became_active":"2023-10-19T13:00:55.200218+0000","last_became_peered":"2023-10-19T13:00:55.200218+0000","last_unstale":"2023-10-19T13:00:57.289630+0000","last_undegraded":"2023-10-19T13:00:57.289630+0000","last_fullsized":"2023-10-19T13:00:57.289630+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_clean_scrub_stamp":"2023-10-19T13:00:54.175433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:52:31.373725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289847+0000","last_change":"2023-10-19T13:00:56.230863+0000","last_active":"2023-10-19T13:00:57.289847+0000","last_peered":"2023-10-19T13:00:57.289847+0000","last_clean":"2023-10-19T13:00:57.289847+0000","last_became_active":"2023-10-19T13:00:56.230572+0000","last_became_peered":"2023-10-19T13:00:56.230572+0000","last_unstale":"2023-10-19T13:00:57.289847+0000","last_undegraded":"2023-10-19T13:00:57.289847+0000","last_fullsized":"2023-10-19T13:00:57.289847+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:54:37.924182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":12,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289105+0000","last_change":"2023-10-19T13:00:55.209095+0000","last_active":"2023-10-19T13:00:57.289105+0000","last_peered":"2023-10-19T13:00:57.289105+0000","last_clean":"2023-10-19T13:00:57.289105+0000","last_became_active":"2023-10-19T13:00:55.208807+0000","last_became_peered":"2023-10-19T13:00:55.208807+0000","last_unstale":"2023-10-19T13:00:57.289105+0000","last_undegraded":"2023-10-19T13:00:57.289105+0000","last_fullsized":"2023-10-19T13:00:57.289105+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_clean_scrub_stamp":"2023-10-19T13:00:54.175433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:47:53.768966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.289205+0000","last_change":"2023-10-19T13:00:56.234690+0000","last_active":"2023-10-19T13:00:57.289205+0000","last_peered":"2023-10-19T13:00:57.289205+0000","last_clean":"2023-10-19T13:00:57.289205+0000","last_became_active":"2023-10-19T13:00:56.234405+0000","last_became_peered":"2023-10-19T13:00:56.234405+0000","last_unstale":"2023-10-19T13:00:57.289205+0000","last_undegraded":"2023-10-19T13:00:57.289205+0000","last_fullsized":"2023-10-19T13:00:57.289205+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:55.181347+0000","last_clean_scrub_stamp":"2023-10-19T13:00:55.181347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:56:34.119416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":12,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-10-19T13:00:57.290378+0000","last_change":"2023-10-19T13:00:55.199843+0000","last_active":"2023-10-19T13:00:57.290378+0000","last_peered":"2023-10-19T13:00:57.290378+0000","last_clean":"2023-10-19T13:00:57.290378+0000","last_became_active":"2023-10-19T13:00:55.199018+0000","last_became_peered":"2023-10-19T13:00:55.199018+0000","last_unstale":"2023-10-19T13:00:57.290378+0000","last_undegraded":"2023-10-19T13:00:57.290378+0000","last_fullsized":"2023-10-19T13:00:57.290378+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:00:54.175433+0000","last_clean_scrub_stamp":"2023-10-19T13:00:54.175433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:51:39.000977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"89'181","reported_seq":467,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-19T13:00:56.221874+0000","last_change":"2023-10-19T12:44:08.636613+0000","last_active":"2023-10-19T13:00:56.221874+0000","last_peered":"2023-10-19T13:00:56.221874+0000","last_clean":"2023-10-19T13:00:56.221874+0000","last_became_active":"2023-10-19T12:44:08.635697+0000","last_became_peered":"2023-10-19T12:44:08.635697+0000","last_unstale":"2023-10-19T13:00:56.221874+0000","last_undegraded":"2023-10-19T13:00:56.221874+0000","last_fullsized":"2023-10-19T13:00:56.221874+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_clean_scrub_stamp":"2023-10-19T12:44:07.399738+0000","objects_scrubbed":0,"log_size":181,"log_dups_size":0,"ondisk_log_size":181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:15:23.685485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":674,"num_read_kb":1321,"num_write":275,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":674,"num_read_kb":1321,"num_write":275,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":181,"ondisk_log_size":181,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575053,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59332,"kb_used_data":1780,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312508,"statfs":{"total":96636764160,"available":96576008192,"internally_reserved":0,"allocated":1822720,"data_stored":1203836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58915275},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 13:00:44 2023","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.55200000000000005,"15min":0.56899999999999995},"min":{"1min":0.34100000000000003,"5min":0.30299999999999999,"15min":0.26400000000000001},"max":{"1min":0.74099999999999999,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.58199999999999996,"15min":0.58899999999999997},"min":{"1min":0.36499999999999999,"5min":0.33200000000000002,"15min":0.29599999999999999},"max":{"1min":0.75600000000000001,"5min":0.83799999999999997,"15min":0.93700000000000006},"last":0.46800000000000003}]},{"osd":1,"last update":"Thu Oct 19 13:00:44 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.56599999999999995,"15min":0.57699999999999996},"min":{"1min":0.41299999999999998,"5min":0.40300000000000002,"15min":0.28199999999999997},"max":{"1min":0.68100000000000005,"5min":0.86299999999999999,"15min":1.3129999999999999},"last":0.441},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.57599999999999996,"15min":0.58999999999999997},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.32700000000000001},"max":{"1min":0.69299999999999995,"5min":0.82699999999999996,"15min":1.284},"last":0.41599999999999998}]},{"osd":2,"last update":"Thu Oct 19 13:00:44 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.623,"15min":0.63},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.76900000000000002,"5min":0.82699999999999996,"15min":0.98599999999999999},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.61899999999999999,"15min":0.63},"min":{"1min":0.46200000000000002,"5min":0.41599999999999998,"15min":0.28399999999999997},"max":{"1min":0.78300000000000003,"5min":0.81200000000000006,"15min":1.2589999999999999},"last":0.54500000000000004}]},{"osd":3,"last update":"Thu Oct 19 13:00:44 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.58999999999999997,"15min":0.59999999999999998},"min":{"1min":0.39700000000000002,"5min":0.34999999999999998,"15min":0.34399999999999997},"max":{"1min":0.71599999999999997,"5min":0.81200000000000006,"15min":0.96899999999999997},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.621,"5min":0.65400000000000003,"15min":0.65600000000000003},"min":{"1min":0.45500000000000002,"5min":0.45300000000000001,"15min":0.39700000000000002},"max":{"1min":0.79800000000000004,"5min":0.92000000000000004,"15min":1.329},"last":0.59299999999999997}]},{"osd":4,"last update":"Thu Oct 19 13:00:44 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.47499999999999998,"15min":0.46999999999999997},"min":{"1min":0.315,"5min":0.27400000000000002,"15min":0.25600000000000001},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.80500000000000005},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.47399999999999998,"15min":0.47199999999999998},"min":{"1min":0.30199999999999999,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.67100000000000004,"5min":0.75600000000000001,"15min":0.78900000000000003},"last":0.38800000000000001}]},{"osd":5,"last update":"Thu Oct 19 13:00:44 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.48899999999999999,"15min":0.48499999999999999},"min":{"1min":0.32000000000000001,"5min":0.30499999999999999,"15min":0.29099999999999998},"max":{"1min":0.74099999999999999,"5min":0.75800000000000001,"15min":0.90400000000000003},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52000000000000002,"15min":0.51700000000000002},"min":{"1min":0.34000000000000002,"5min":0.314,"15min":0.308},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.92200000000000004},"last":0.69299999999999995}]},{"osd":6,"last update":"Thu Oct 19 13:00:44 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.54000000000000004,"15min":0.54800000000000004},"min":{"1min":0.378,"5min":0.34100000000000003,"15min":0.308},"max":{"1min":0.749,"5min":0.81299999999999994,"15min":0.88900000000000001},"last":0.62},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52000000000000002,"15min":0.52800000000000002},"min":{"1min":0.33400000000000002,"5min":0.30299999999999999,"15min":0.27300000000000002},"max":{"1min":0.68300000000000005,"5min":0.83999999999999997,"15min":0.90300000000000002},"last":0.60799999999999998}]}]},{"osd":6,"up_from":13,"seq":55834575053,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32396,"kb_used_data":1208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339444,"statfs":{"total":96636764160,"available":96603590656,"internally_reserved":0,"allocated":1236992,"data_stored":615058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 13:00:32 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.57199999999999995,"15min":0.58599999999999997},"min":{"1min":0.373,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.75800000000000001,"5min":0.76000000000000001,"15min":0.81999999999999995},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.57799999999999996,"15min":0.58499999999999996},"min":{"1min":0.36399999999999999,"5min":0.247,"15min":0.247},"max":{"1min":0.75600000000000001,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.57499999999999996}]},{"osd":1,"last update":"Thu Oct 19 13:00:32 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.35299999999999998,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.76000000000000001,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.59699999999999998,"15min":0.59899999999999998},"min":{"1min":0.42099999999999999,"5min":0.34300000000000003,"15min":0.28399999999999997},"max":{"1min":0.749,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.439}]},{"osd":2,"last update":"Thu Oct 19 13:00:32 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.57299999999999995,"15min":0.58199999999999996},"min":{"1min":0.36899999999999999,"5min":0.307,"15min":0.26600000000000001},"max":{"1min":0.755,"5min":0.84699999999999998,"15min":0.85699999999999998},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.61299999999999999,"15min":0.60999999999999999},"min":{"1min":0.38900000000000001,"5min":0.35999999999999999,"15min":0.28399999999999997},"max":{"1min":0.79400000000000004,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.56299999999999994}]},{"osd":3,"last update":"Thu Oct 19 13:00:32 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61899999999999999,"15min":0.624},"min":{"1min":0.38800000000000001,"5min":0.38500000000000001,"15min":0.32600000000000001},"max":{"1min":0.81299999999999994,"5min":0.871,"15min":0.871},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.61899999999999999,"15min":0.624},"min":{"1min":0.40100000000000002,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.74099999999999999,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.48699999999999999}]},{"osd":4,"last update":"Thu Oct 19 13:00:32 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.45900000000000002,"15min":0.46600000000000003},"min":{"1min":0.29099999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.76700000000000002,"5min":0.77900000000000003,"15min":0.79300000000000004},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.47699999999999998,"15min":0.47999999999999998},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.247},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.77600000000000002},"last":0.501}]},{"osd":5,"last update":"Thu Oct 19 13:00:32 2023","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.46100000000000002,"15min":0.47099999999999997},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.26600000000000001},"max":{"1min":0.55700000000000005,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.51400000000000001,"15min":0.51600000000000001},"min":{"1min":0.375,"5min":0.32100000000000001,"15min":0.312},"max":{"1min":0.747,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.53100000000000003}]},{"osd":7,"last update":"Thu Oct 19 13:00:32 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.52500000000000002,"15min":0.52100000000000002},"min":{"1min":0.32400000000000001,"5min":0.29299999999999998,"15min":0.28199999999999997},"max":{"1min":0.77600000000000002,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.53500000000000003},"min":{"1min":0.27900000000000003,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.66700000000000004,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.52000000000000002}]}]},{"osd":5,"up_from":13,"seq":55834575053,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50428,"kb_used_data":1200,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321412,"statfs":{"total":96636764160,"available":96585125888,"internally_reserved":0,"allocated":1228800,"data_stored":613480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 13:00:29 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58799999999999997,"15min":0.59099999999999997},"min":{"1min":0.307,"5min":0.28999999999999998,"15min":0.223},"max":{"1min":0.75900000000000001,"5min":0.79900000000000004,"15min":0.82699999999999996},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58599999999999997,"15min":0.59799999999999998},"min":{"1min":0.41099999999999998,"5min":0.36099999999999999,"15min":0.27800000000000002},"max":{"1min":0.82999999999999996,"5min":0.84299999999999997,"15min":0.874},"last":0.51700000000000002}]},{"osd":1,"last update":"Thu Oct 19 13:00:29 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.57799999999999996,"15min":0.59499999999999997},"min":{"1min":0.40400000000000003,"5min":0.34599999999999997,"15min":0.32800000000000001},"max":{"1min":0.71299999999999997,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.60199999999999998,"15min":0.61199999999999999},"min":{"1min":0.378,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.77200000000000002,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.57499999999999996}]},{"osd":2,"last update":"Thu Oct 19 13:00:29 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.56399999999999995,"15min":0.58499999999999996},"min":{"1min":0.41199999999999998,"5min":0.34100000000000003,"15min":0.307},"max":{"1min":0.71699999999999997,"5min":0.78600000000000003,"15min":0.80700000000000005},"last":0.495},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.61199999999999999,"15min":0.624},"min":{"1min":0.47099999999999997,"5min":0.38400000000000001,"15min":0.36599999999999999},"max":{"1min":0.79300000000000004,"5min":0.88100000000000001,"15min":0.93600000000000005},"last":0.64500000000000002}]},{"osd":3,"last update":"Thu Oct 19 13:00:29 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.58899999999999997,"15min":0.60399999999999998},"min":{"1min":0.41699999999999998,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.752,"5min":0.79900000000000004,"15min":0.81299999999999994},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.61499999999999999,"15min":0.63400000000000001},"min":{"1min":0.501,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.73999999999999999,"5min":0.81000000000000005,"15min":0.89500000000000002},"last":0.55400000000000005}]},{"osd":4,"last update":"Thu Oct 19 13:00:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.47599999999999998,"15min":0.47499999999999998},"min":{"1min":0.35399999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.73599999999999999,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.46800000000000003,"15min":0.46999999999999997},"min":{"1min":0.33900000000000002,"5min":0.251,"15min":0.251},"max":{"1min":0.69199999999999995,"5min":0.75900000000000001,"15min":0.80500000000000005},"last":0.59999999999999998}]},{"osd":6,"last update":"Thu Oct 19 13:00:29 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.34100000000000003,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.75},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.51100000000000001,"15min":0.504},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.33000000000000002},"max":{"1min":0.67200000000000004,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.69399999999999995}]},{"osd":7,"last update":"Thu Oct 19 13:00:29 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.498,"15min":0.49399999999999999},"min":{"1min":0.39800000000000002,"5min":0.32000000000000001,"15min":0.315},"max":{"1min":0.72399999999999998,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.54900000000000004,"15min":0.53900000000000003},"min":{"1min":0.42599999999999999,"5min":0.311,"15min":0.311},"max":{"1min":0.74099999999999999,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.63400000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575053,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32396,"kb_used_data":1216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339444,"statfs":{"total":96636764160,"available":96603590656,"internally_reserved":0,"allocated":1245184,"data_stored":615162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 13:00:43 2023","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.55200000000000005,"15min":0.56000000000000005},"min":{"1min":0.36099999999999999,"5min":0.29499999999999998,"15min":0.28899999999999998},"max":{"1min":0.72299999999999998,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.54700000000000004,"15min":0.55700000000000005},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.27600000000000002},"max":{"1min":0.70099999999999996,"5min":0.77800000000000002,"15min":0.78200000000000003},"last":0.50800000000000001}]},{"osd":1,"last update":"Thu Oct 19 13:00:43 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.58099999999999996,"15min":0.60699999999999998},"min":{"1min":0.36599999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.72699999999999998,"5min":0.80600000000000005,"15min":0.86199999999999999},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.60299999999999998,"15min":0.61499999999999999},"min":{"1min":0.433,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.73699999999999999,"5min":0.79900000000000004,"15min":0.873},"last":0.66400000000000003}]},{"osd":2,"last update":"Thu Oct 19 13:00:43 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.59399999999999997,"15min":0.61099999999999999},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.254},"max":{"1min":0.79300000000000004,"5min":0.85299999999999998,"15min":0.86599999999999999},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.59699999999999998,"15min":0.61299999999999999},"min":{"1min":0.38700000000000001,"5min":0.35799999999999998,"15min":0.34499999999999997},"max":{"1min":0.75800000000000001,"5min":0.82999999999999996,"15min":0.85999999999999999},"last":0.63500000000000001}]},{"osd":3,"last update":"Thu Oct 19 13:00:43 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.57499999999999996,"15min":0.58499999999999996},"min":{"1min":0.36199999999999999,"5min":0.35199999999999998,"15min":0.32400000000000001},"max":{"1min":0.75,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.63,"15min":0.63700000000000001},"min":{"1min":0.44800000000000001,"5min":0.42599999999999999,"15min":0.37},"max":{"1min":0.76500000000000001,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.59899999999999998}]},{"osd":5,"last update":"Thu Oct 19 13:00:43 2023","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.45100000000000001,"15min":0.44800000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.249},"max":{"1min":0.63200000000000001,"5min":0.79600000000000004,"15min":0.81299999999999994},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.46899999999999997,"15min":0.45900000000000002},"min":{"1min":0.30099999999999999,"5min":0.26400000000000001,"15min":0.23499999999999999},"max":{"1min":0.69499999999999995,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.36899999999999999}]},{"osd":6,"last update":"Thu Oct 19 13:00:43 2023","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.45500000000000002,"15min":0.45400000000000001},"min":{"1min":0.30099999999999999,"5min":0.29499999999999998,"15min":0.26400000000000001},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.78600000000000003},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.48799999999999999,"15min":0.49399999999999999},"min":{"1min":0.308,"5min":0.308,"15min":0.27100000000000002},"max":{"1min":0.70699999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.41799999999999998}]},{"osd":7,"last update":"Thu Oct 19 13:00:43 2023","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.505,"15min":0.501},"min":{"1min":0.33200000000000002,"5min":0.307,"15min":0.27500000000000002},"max":{"1min":0.72799999999999998,"5min":0.79100000000000004,"15min":0.80200000000000005},"last":0.442},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54200000000000004,"15min":0.54400000000000004},"min":{"1min":0.371,"5min":0.35799999999999998,"15min":0.30199999999999999},"max":{"1min":0.70799999999999996,"5min":0.80500000000000005,"15min":0.83499999999999996},"last":0.46100000000000002}]}]},{"osd":3,"up_from":13,"seq":55834575052,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36544,"kb_used_data":1200,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335296,"statfs":{"total":96636764160,"available":96599343104,"internally_reserved":0,"allocated":1228800,"data_stored":614028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 13:00:27 2023","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.42199999999999999,"15min":0.40000000000000002},"min":{"1min":0.252,"5min":0.23599999999999999,"15min":0.20899999999999999},"max":{"1min":0.60399999999999998,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.41799999999999998,"15min":0.41599999999999998},"min":{"1min":0.28699999999999998,"5min":0.223,"15min":0.223},"max":{"1min":0.59899999999999998,"5min":0.75,"15min":0.75},"last":0.52800000000000002}]},{"osd":1,"last update":"Thu Oct 19 13:00:27 2023","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.41999999999999998,"15min":0.41199999999999998},"min":{"1min":0.26900000000000002,"5min":0.26100000000000001,"15min":0.246},"max":{"1min":0.628,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.48599999999999999,"15min":0.45900000000000002},"min":{"1min":0.30099999999999999,"5min":0.24099999999999999,"15min":0.215},"max":{"1min":0.64200000000000002,"5min":2.9399999999999999,"15min":2.9399999999999999},"last":0.46000000000000002}]},{"osd":2,"last update":"Thu Oct 19 13:00:27 2023","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.47099999999999997,"15min":0.46800000000000003},"min":{"1min":0.317,"5min":0.314,"15min":0.26100000000000001},"max":{"1min":0.62,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.496,"5min":0.48299999999999998,"15min":0.48699999999999999},"min":{"1min":0.34499999999999997,"5min":0.29999999999999999,"15min":0.247},"max":{"1min":0.66900000000000004,"5min":0.754,"15min":0.754},"last":0.50800000000000001}]},{"osd":4,"last update":"Thu Oct 19 13:00:27 2023","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.66000000000000003,"15min":0.66700000000000004},"min":{"1min":0.44700000000000001,"5min":0.42299999999999999,"15min":0.40300000000000002},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":1.0069999999999999},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.68799999999999994,"15min":0.69199999999999995},"min":{"1min":0.48899999999999999,"5min":0.46899999999999997,"15min":0.42099999999999999},"max":{"1min":0.91400000000000003,"5min":0.93000000000000005,"15min":0.99399999999999999},"last":0.84999999999999998}]},{"osd":5,"last update":"Thu Oct 19 13:00:27 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.68500000000000005,"15min":0.66000000000000003},"min":{"1min":0.47499999999999998,"5min":0.45500000000000002,"15min":0.41099999999999998},"max":{"1min":0.90600000000000003,"5min":2.9990000000000001,"15min":2.9990000000000001},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.65000000000000002,"15min":0.65100000000000002},"min":{"1min":0.47699999999999998,"5min":0.40899999999999997,"15min":0.35899999999999999},"max":{"1min":0.873,"5min":1.774,"15min":1.774},"last":0.86099999999999999}]},{"osd":6,"last update":"Thu Oct 19 13:00:27 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.66300000000000003,"15min":0.65400000000000003},"min":{"1min":0.503,"5min":0.433,"15min":0.41199999999999998},"max":{"1min":0.84499999999999997,"5min":0.93700000000000006,"15min":0.96799999999999997},"last":0.90100000000000002},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.68000000000000005,"15min":0.67600000000000005},"min":{"1min":0.501,"5min":0.377,"15min":0.377},"max":{"1min":0.94399999999999995,"5min":1.167,"15min":1.167},"last":0.80400000000000005}]},{"osd":7,"last update":"Thu Oct 19 13:00:27 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.65900000000000003,"15min":0.66500000000000004},"min":{"1min":0.47999999999999998,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.90700000000000003,"5min":1.042,"15min":1.042},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.69499999999999995,"15min":0.68100000000000005},"min":{"1min":0.51100000000000001,"5min":0.432,"15min":0.432},"max":{"1min":0.88300000000000001,"5min":2.8149999999999999,"15min":2.8149999999999999},"last":0.88}]}]},{"osd":2,"up_from":13,"seq":55834575054,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54604,"kb_used_data":1208,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317236,"statfs":{"total":96636764160,"available":96580849664,"internally_reserved":0,"allocated":1236992,"data_stored":614188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 13:00:37 2023","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.36899999999999999,"15min":0.378},"min":{"1min":0.215,"5min":0.20899999999999999,"15min":0.187},"max":{"1min":0.52900000000000003,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.314},{"interface":"front","average":{"1min":0.374,"5min":0.39700000000000002,"15min":0.40699999999999997},"min":{"1min":0.28000000000000003,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.56000000000000005,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.33300000000000002}]},{"osd":1,"last update":"Thu Oct 19 13:00:37 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.38200000000000001,"15min":0.39600000000000002},"min":{"1min":0.26100000000000001,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.51100000000000001,"5min":0.63900000000000001,"15min":0.71099999999999997},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.41099999999999998,"15min":0.42799999999999999},"min":{"1min":0.26900000000000002,"5min":0.22,"15min":0.22},"max":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.68100000000000005},"last":0.38500000000000001}]},{"osd":3,"last update":"Thu Oct 19 13:00:37 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.45300000000000001,"15min":0.45500000000000002},"min":{"1min":0.33700000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.50700000000000001,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.432},{"interface":"front","average":{"1min":0.439,"5min":0.46899999999999997,"15min":0.48199999999999998},"min":{"1min":0.34100000000000003,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.57499999999999996,"5min":0.77800000000000002,"15min":0.82999999999999996},"last":0.41599999999999998}]},{"osd":4,"last update":"Thu Oct 19 13:00:37 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.61899999999999999,"15min":0.64000000000000001},"min":{"1min":0.40100000000000002,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.68700000000000006,"5min":0.93999999999999995,"15min":0.98999999999999999},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.63400000000000001,"15min":0.65500000000000003},"min":{"1min":0.42999999999999999,"5min":0.35099999999999998,"15min":0.32000000000000001},"max":{"1min":0.78000000000000003,"5min":0.91200000000000003,"15min":1.046},"last":0.45900000000000002}]},{"osd":5,"last update":"Thu Oct 19 13:00:37 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.65000000000000002,"15min":0.66100000000000003},"min":{"1min":0.46899999999999997,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.73999999999999999,"5min":0.96499999999999997,"15min":0.99399999999999999},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.63200000000000001,"15min":0.65600000000000003},"min":{"1min":0.39100000000000001,"5min":0.375,"15min":0.375},"max":{"1min":0.753,"5min":0.90200000000000002,"15min":0.95799999999999996},"last":0.48299999999999998}]},{"osd":6,"last update":"Thu Oct 19 13:00:37 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.63700000000000001,"15min":0.66300000000000003},"min":{"1min":0.48799999999999999,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.73599999999999999,"5min":1.008,"15min":1.008},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.65600000000000003,"15min":0.67400000000000004},"min":{"1min":0.46300000000000002,"5min":0.371,"15min":0.371},"max":{"1min":0.80500000000000005,"5min":0.995,"15min":0.995},"last":0.59399999999999997}]},{"osd":7,"last update":"Thu Oct 19 13:00:37 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.65400000000000003,"15min":0.68000000000000005},"min":{"1min":0.39800000000000002,"5min":0.39300000000000002,"15min":0.36399999999999999},"max":{"1min":0.71799999999999997,"5min":0.98999999999999999,"15min":1.0149999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.66800000000000004,"15min":0.68100000000000005},"min":{"1min":0.47199999999999998,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.80000000000000004,"5min":1.02,"15min":1.0329999999999999},"last":0.61899999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575053,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32392,"kb_used_data":1204,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339448,"statfs":{"total":96636764160,"available":96603594752,"internally_reserved":0,"allocated":1232896,"data_stored":613558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 13:00:36 2023","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.39000000000000001,"15min":0.38600000000000001},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.187},"max":{"1min":0.57799999999999996,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.245},{"interface":"front","average":{"1min":0.438,"5min":0.441,"15min":0.43099999999999999},"min":{"1min":0.217,"5min":0.217,"15min":0.21199999999999999},"max":{"1min":0.65800000000000003,"5min":2.5800000000000001,"15min":2.5800000000000001},"last":0.314}]},{"osd":2,"last update":"Thu Oct 19 13:00:36 2023","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.41799999999999998,"15min":0.42199999999999999},"min":{"1min":0.24199999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.69999999999999996},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.47599999999999998,"15min":0.45500000000000002},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.23699999999999999},"max":{"1min":0.75600000000000001,"5min":2.6230000000000002,"15min":2.6230000000000002},"last":0.34399999999999997}]},{"osd":3,"last update":"Thu Oct 19 13:00:36 2023","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.50700000000000001,"15min":0.48499999999999999},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.26200000000000001},"max":{"1min":0.69599999999999995,"5min":3.9740000000000002,"15min":3.9740000000000002},"last":0.371},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.43099999999999999,"15min":0.42199999999999999},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27000000000000002},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.80500000000000005},"last":0.35499999999999998}]},{"osd":4,"last update":"Thu Oct 19 13:00:36 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.44900000000000001,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.86899999999999999,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.627,"5min":0.66100000000000003,"15min":0.64500000000000002},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.314},"max":{"1min":0.86899999999999999,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.51500000000000001}]},{"osd":5,"last update":"Thu Oct 19 13:00:36 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.65300000000000002,"15min":0.65300000000000002},"min":{"1min":0.55300000000000005,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.85299999999999998,"5min":1.1200000000000001,"15min":1.1200000000000001},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.69499999999999995,"15min":0.67300000000000004},"min":{"1min":0.436,"5min":0.42299999999999999,"15min":0.38700000000000001},"max":{"1min":0.84499999999999997,"5min":2.7120000000000002,"15min":2.7120000000000002},"last":0.48499999999999999}]},{"osd":6,"last update":"Thu Oct 19 13:00:36 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.64600000000000002,"15min":0.65400000000000003},"min":{"1min":0.46999999999999997,"5min":0.36799999999999999,"15min":0.36399999999999999},"max":{"1min":0.83799999999999997,"5min":0.92200000000000004,"15min":1.0069999999999999},"last":0.495},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.67800000000000005,"15min":0.67700000000000005},"min":{"1min":0.47299999999999998,"5min":0.376,"15min":0.376},"max":{"1min":1.129,"5min":1.129,"15min":1.129},"last":0.58899999999999997}]},{"osd":7,"last update":"Thu Oct 19 13:00:36 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.69199999999999995,"15min":0.67500000000000004},"min":{"1min":0.52900000000000003,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.80300000000000005,"5min":2.786,"15min":2.786},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.67400000000000004,"15min":0.67600000000000005},"min":{"1min":0.45200000000000001,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.79500000000000004,"5min":0.94199999999999995,"15min":1.034},"last":0.53000000000000003}]}]},{"osd":0,"up_from":13,"seq":55834575054,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37112,"kb_used_data":1772,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334728,"statfs":{"total":96636764160,"available":96598761472,"internally_reserved":0,"allocated":1814528,"data_stored":1203195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Oct 19 13:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.38700000000000001,"15min":0.39000000000000001},"min":{"1min":0.20899999999999999,"5min":0.182,"15min":0.182},"max":{"1min":0.52800000000000002,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.372,"5min":0.42199999999999999,"15min":0.43099999999999999},"min":{"1min":0.248,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.58999999999999997,"5min":0.67600000000000005,"15min":0.746},"last":0.36099999999999999}]},{"osd":2,"last update":"Thu Oct 19 13:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.45100000000000001,"15min":0.45300000000000001},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.25800000000000001},"max":{"1min":0.64100000000000001,"5min":0.752,"15min":0.76300000000000001},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.42499999999999999,"15min":0.434},"min":{"1min":0.246,"5min":0.216,"15min":0.20699999999999999},"max":{"1min":0.623,"5min":0.72999999999999998,"15min":0.76800000000000002},"last":0.377}]},{"osd":3,"last update":"Thu Oct 19 13:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.40999999999999998,"15min":0.40899999999999997},"min":{"1min":0.26600000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.56599999999999995,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.308},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.318,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.67500000000000004,"5min":1.4990000000000001,"15min":1.4990000000000001},"last":0.32700000000000001}]},{"osd":4,"last update":"Thu Oct 19 13:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.61399999999999999,"15min":0.63300000000000001},"min":{"1min":0.40300000000000002,"5min":0.40000000000000002,"15min":0.34999999999999998},"max":{"1min":0.71299999999999997,"5min":1.361,"15min":1.361},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.64500000000000002,"15min":0.64900000000000002},"min":{"1min":0.42899999999999999,"5min":0.41899999999999998,"15min":0.35999999999999999},"max":{"1min":0.80900000000000005,"5min":0.98999999999999999,"15min":0.998},"last":0.60299999999999998}]},{"osd":5,"last update":"Thu Oct 19 13:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.67900000000000005,"15min":0.68100000000000005},"min":{"1min":0.49199999999999999,"5min":0.378,"15min":0.378},"max":{"1min":0.90000000000000002,"5min":1.325,"15min":1.325},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.62,"5min":0.67900000000000005,"15min":0.68300000000000005},"min":{"1min":0.437,"5min":0.36299999999999999,"15min":0.318},"max":{"1min":0.88,"5min":1.024,"15min":1.024},"last":0.68100000000000005}]},{"osd":6,"last update":"Thu Oct 19 13:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.65600000000000003,"15min":0.66200000000000003},"min":{"1min":0.42499999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.79300000000000004,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.62},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.65200000000000002,"15min":0.66200000000000003},"min":{"1min":0.45300000000000001,"5min":0.41999999999999998,"15min":0.38900000000000001},"max":{"1min":0.73499999999999999,"5min":0.995,"15min":1.089},"last":0.56699999999999995}]},{"osd":7,"last update":"Thu Oct 19 13:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.66200000000000003,"15min":0.67900000000000005},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.82799999999999996,"5min":0.98799999999999999,"15min":1.0089999999999999},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.65900000000000003,"15min":0.66100000000000003},"min":{"1min":0.442,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.94499999999999995,"5min":1.389,"15min":1.389},"last":0.58299999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-19T13:00:59.573 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-19T13:00:59.574 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-19T13:00:59.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.722+0000 7fc92650c640 1 -- 172.21.15.57:0/2057929207 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc908004d00 msgr2=0x7fc908005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:59.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.722+0000 7fc92650c640 1 --2- 172.21.15.57:0/2057929207 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc908004d00 0x7fc908005180 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7fc91800c610 tx=0x7fc9080057f0 comp rx=0 tx=0).stop 2023-10-19T13:00:59.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.722+0000 7fc924aa9640 1 -- 172.21.15.57:0/2057929207 reap_dead start 2023-10-19T13:00:59.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.722+0000 7fc92650c640 1 -- 172.21.15.57:0/2057929207 shutdown_connections 2023-10-19T13:00:59.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.722+0000 7fc92650c640 1 -- 172.21.15.57:0/2057929207 >> 172.21.15.57:0/2057929207 conn(0x7fc920069930 msgr2=0x7fc920069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:59.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.722+0000 7fc92650c640 1 -- 172.21.15.57:0/2057929207 shutdown_connections 2023-10-19T13:00:59.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.722+0000 7fc92650c640 1 -- 172.21.15.57:0/2057929207 wait complete. 2023-10-19T13:00:59.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.723+0000 7fc92650c640 1 Processor -- start 2023-10-19T13:00:59.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.723+0000 7fc92650c640 1 -- start start 2023-10-19T13:00:59.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.724+0000 7fc92650c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc9201562c0 0x7fc9200ae5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:59.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.724+0000 7fc92650c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc920164c30 0x7fc9201566a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:59.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.724+0000 7fc92650c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc9201f4c60 0x7fc9201f50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:59.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.724+0000 7fc92650c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9200ae190 con 0x7fc920164c30 2023-10-19T13:00:59.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.724+0000 7fc92650c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9201650b0 con 0x7fc9201562c0 2023-10-19T13:00:59.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.724+0000 7fc92650c640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9200cb720 con 0x7fc9201f4c60 2023-10-19T13:00:59.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.724+0000 7fc91f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc920164c30 0x7fc9201566a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:59.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.724+0000 7fc91f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc920164c30 0x7fc9201566a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38070/0 (socket says 172.21.15.57:38070) 2023-10-19T13:00:59.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.724+0000 7fc91f7fe640 1 -- 172.21.15.57:0/2669296972 learned_addr learned my addr 172.21.15.57:0/2669296972 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:00:59.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.724+0000 7fc91ffff640 1 --2- 172.21.15.57:0/2669296972 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc9201562c0 0x7fc9200ae5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:59.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.724+0000 7fc924aa9640 1 --2- 172.21.15.57:0/2669296972 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc9201f4c60 0x7fc9201f50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:59.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.724+0000 7fc91f7fe640 1 -- 172.21.15.57:0/2669296972 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc9201f4c60 msgr2=0x7fc9201f50e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:59.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.724+0000 7fc91f7fe640 1 --2- 172.21.15.57:0/2669296972 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc9201f4c60 0x7fc9201f50e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:59.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.724+0000 7fc91f7fe640 1 -- 172.21.15.57:0/2669296972 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc9201562c0 msgr2=0x7fc9200ae5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:59.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.724+0000 7fc91f7fe640 1 --2- 172.21.15.57:0/2669296972 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc9201562c0 0x7fc9200ae5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:59.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.724+0000 7fc91f7fe640 1 -- 172.21.15.57:0/2669296972 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc92013b4f0 con 0x7fc920164c30 2023-10-19T13:00:59.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.724+0000 7fc924aa9640 1 --2- 172.21.15.57:0/2669296972 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc9201f4c60 0x7fc9201f50e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:00:59.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.724+0000 7fc91f7fe640 1 --2- 172.21.15.57:0/2669296972 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc920164c30 0x7fc9201566a0 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7fc914005f60 tx=0x7fc914011cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:59.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.725+0000 7fc91cff9640 1 -- 172.21.15.57:0/2669296972 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc91400fd50 con 0x7fc920164c30 2023-10-19T13:00:59.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.725+0000 7fc91cff9640 1 -- 172.21.15.57:0/2669296972 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fc914012450 con 0x7fc920164c30 2023-10-19T13:00:59.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.725+0000 7fc92650c640 1 -- 172.21.15.57:0/2669296972 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc920156be0 con 0x7fc920164c30 2023-10-19T13:00:59.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.725+0000 7fc91cff9640 1 -- 172.21.15.57:0/2669296972 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc914012860 con 0x7fc920164c30 2023-10-19T13:00:59.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.725+0000 7fc92650c640 1 -- 172.21.15.57:0/2669296972 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9200ac6c0 con 0x7fc920164c30 2023-10-19T13:00:59.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.726+0000 7fc92650c640 1 -- 172.21.15.57:0/2669296972 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9200cb860 con 0x7fc920164c30 2023-10-19T13:00:59.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.726+0000 7fc91cff9640 1 -- 172.21.15.57:0/2669296972 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 40) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc9140129c0 con 0x7fc920164c30 2023-10-19T13:00:59.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.726+0000 7fc91cff9640 1 --2- 172.21.15.57:0/2669296972 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fc90c06af60 0x7fc90c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:00:59.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.726+0000 7fc91cff9640 1 -- 172.21.15.57:0/2669296972 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fc91408a670 con 0x7fc920164c30 2023-10-19T13:00:59.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.727+0000 7fc91ffff640 1 --2- 172.21.15.57:0/2669296972 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fc90c06af60 0x7fc90c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:00:59.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.727+0000 7fc91ffff640 1 --2- 172.21.15.57:0/2669296972 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fc90c06af60 0x7fc90c06d420 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc910009bd0 tx=0x7fc910005f90 comp rx=0 tx=0).ready entity=mgr.6679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:00:59.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.731+0000 7fc91cff9640 1 -- 172.21.15.57:0/2669296972 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc9140555f0 con 0x7fc920164c30 2023-10-19T13:00:59.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.977+0000 7fc92650c640 1 -- 172.21.15.57:0/2669296972 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fc9200690e0 con 0x7fc920164c30 2023-10-19T13:00:59.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.980+0000 7fc91cff9640 1 -- 172.21.15.57:0/2669296972 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v25) v1 ==== 190+0+147 (secure 0 0 0) 0x7fc914088e60 con 0x7fc920164c30 2023-10-19T13:00:59.983 INFO:teuthology.orchestra.run.smithi057.stdout:[client.0] 2023-10-19T13:00:59.983 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQBkJDFlg9yuExAATxiWDfS+p07G1F0pORbJEg== 2023-10-19T13:00:59.983 INFO:teuthology.orchestra.run.smithi057.stdout: caps mds = "allow" 2023-10-19T13:00:59.983 INFO:teuthology.orchestra.run.smithi057.stdout: caps mon = "allow r" 2023-10-19T13:00:59.984 INFO:teuthology.orchestra.run.smithi057.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-19T13:00:59.984 INFO:teuthology.orchestra.run.smithi057.stderr:updated caps for client.0 2023-10-19T13:00:59.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.982+0000 7fc92650c640 1 -- 172.21.15.57:0/2669296972 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fc90c06af60 msgr2=0x7fc90c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:59.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.982+0000 7fc92650c640 1 --2- 172.21.15.57:0/2669296972 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fc90c06af60 0x7fc90c06d420 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc910009bd0 tx=0x7fc910005f90 comp rx=0 tx=0).stop 2023-10-19T13:00:59.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.982+0000 7fc92650c640 1 -- 172.21.15.57:0/2669296972 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc920164c30 msgr2=0x7fc9201566a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:00:59.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.982+0000 7fc92650c640 1 --2- 172.21.15.57:0/2669296972 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc920164c30 0x7fc9201566a0 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7fc914005f60 tx=0x7fc914011cb0 comp rx=0 tx=0).stop 2023-10-19T13:00:59.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.982+0000 7fc92650c640 1 -- 172.21.15.57:0/2669296972 shutdown_connections 2023-10-19T13:00:59.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.982+0000 7fc92650c640 1 --2- 172.21.15.57:0/2669296972 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fc90c06af60 0x7fc90c06d420 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:59.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.982+0000 7fc92650c640 1 --2- 172.21.15.57:0/2669296972 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc9201f4c60 0x7fc9201f50e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:59.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.982+0000 7fc92650c640 1 --2- 172.21.15.57:0/2669296972 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc920164c30 0x7fc9201566a0 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:59.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.982+0000 7fc92650c640 1 --2- 172.21.15.57:0/2669296972 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc9201562c0 0x7fc9200ae5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:00:59.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.982+0000 7fc92650c640 1 -- 172.21.15.57:0/2669296972 >> 172.21.15.57:0/2669296972 conn(0x7fc920069930 msgr2=0x7fc92015a1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:00:59.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.982+0000 7fc92650c640 1 -- 172.21.15.57:0/2669296972 shutdown_connections 2023-10-19T13:00:59.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:00:59.982+0000 7fc92650c640 1 -- 172.21.15.57:0/2669296972 wait complete. 2023-10-19T13:00:59.995 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T13:01:00.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.102+0000 7ff298b55640 1 -- 172.21.15.57:0/1767831579 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff2940c9d00 msgr2=0x7ff29406b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:00.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.102+0000 7ff298b55640 1 --2- 172.21.15.57:0/1767831579 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff2940c9d00 0x7ff29406b910 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7ff28c005fd0 tx=0x7ff28c02c7a0 comp rx=0 tx=0).stop 2023-10-19T13:01:00.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.102+0000 7ff298b55640 1 -- 172.21.15.57:0/1767831579 shutdown_connections 2023-10-19T13:01:00.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.102+0000 7ff298b55640 1 --2- 172.21.15.57:0/1767831579 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff2940c9d00 0x7ff29406b910 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:00.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.102+0000 7ff298b55640 1 --2- 172.21.15.57:0/1767831579 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff294140660 0x7ff2940c97c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:00.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.102+0000 7ff298b55640 1 -- 172.21.15.57:0/1767831579 >> 172.21.15.57:0/1767831579 conn(0x7ff294069930 msgr2=0x7ff294069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:00.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.102+0000 7ff298b55640 1 -- 172.21.15.57:0/1767831579 shutdown_connections 2023-10-19T13:01:00.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.102+0000 7ff298b55640 1 -- 172.21.15.57:0/1767831579 wait complete. 2023-10-19T13:01:00.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.103+0000 7ff298b55640 1 Processor -- start 2023-10-19T13:01:00.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.103+0000 7ff298b55640 1 -- start start 2023-10-19T13:01:00.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.104+0000 7ff298b55640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff2940c9d00 0x7ff2941f1c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:00.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.104+0000 7ff298b55640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff29413d360 0x7ff294148760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:00.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.104+0000 7ff298b55640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff294140660 0x7ff2941f4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:00.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.104+0000 7ff298b55640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2941490a0 con 0x7ff294140660 2023-10-19T13:01:00.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.104+0000 7ff298b55640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff294149380 con 0x7ff2940c9d00 2023-10-19T13:01:00.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.104+0000 7ff298b55640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff294149660 con 0x7ff29413d360 2023-10-19T13:01:00.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.104+0000 7ff292d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff294140660 0x7ff2941f4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:00.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.104+0000 7ff292d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff294140660 0x7ff2941f4f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38084/0 (socket says 172.21.15.57:38084) 2023-10-19T13:01:00.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.104+0000 7ff292d9d640 1 -- 172.21.15.57:0/767123171 learned_addr learned my addr 172.21.15.57:0/767123171 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:01:00.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.104+0000 7ff28bfff640 1 --2- 172.21.15.57:0/767123171 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff29413d360 0x7ff294148760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:00.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.104+0000 7ff292d9d640 1 -- 172.21.15.57:0/767123171 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff29413d360 msgr2=0x7ff294148760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:00.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.104+0000 7ff292d9d640 1 --2- 172.21.15.57:0/767123171 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff29413d360 0x7ff294148760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:00.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.104+0000 7ff29259c640 1 --2- 172.21.15.57:0/767123171 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff2940c9d00 0x7ff2941f1c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:00.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.104+0000 7ff292d9d640 1 -- 172.21.15.57:0/767123171 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff2940c9d00 msgr2=0x7ff2941f1c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:00.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.104+0000 7ff292d9d640 1 --2- 172.21.15.57:0/767123171 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff2940c9d00 0x7ff2941f1c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:00.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.104+0000 7ff292d9d640 1 -- 172.21.15.57:0/767123171 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff28c00c4a0 con 0x7ff294140660 2023-10-19T13:01:00.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.104+0000 7ff29259c640 1 --2- 172.21.15.57:0/767123171 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff2940c9d00 0x7ff2941f1c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T13:01:00.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.104+0000 7ff292d9d640 1 --2- 172.21.15.57:0/767123171 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff294140660 0x7ff2941f4f70 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7ff28c00c790 tx=0x7ff28c032e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:00.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.105+0000 7ff28affd640 1 -- 172.21.15.57:0/767123171 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff28c009570 con 0x7ff294140660 2023-10-19T13:01:00.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.105+0000 7ff28affd640 1 -- 172.21.15.57:0/767123171 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7ff28c0096d0 con 0x7ff294140660 2023-10-19T13:01:00.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.105+0000 7ff298b55640 1 -- 172.21.15.57:0/767123171 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2941f1460 con 0x7ff294140660 2023-10-19T13:01:00.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.105+0000 7ff28affd640 1 -- 172.21.15.57:0/767123171 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff28c009ae0 con 0x7ff294140660 2023-10-19T13:01:00.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.105+0000 7ff298b55640 1 -- 172.21.15.57:0/767123171 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2941f21a0 con 0x7ff294140660 2023-10-19T13:01:00.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.106+0000 7ff298b55640 1 -- 172.21.15.57:0/767123171 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2941f1980 con 0x7ff294140660 2023-10-19T13:01:00.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.106+0000 7ff28affd640 1 -- 172.21.15.57:0/767123171 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 40) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff28c00a040 con 0x7ff294140660 2023-10-19T13:01:00.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.107+0000 7ff28affd640 1 --2- 172.21.15.57:0/767123171 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7ff27406af60 0x7ff27406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:00.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.107+0000 7ff28affd640 1 -- 172.21.15.57:0/767123171 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6057+0+0 (secure 0 0 0) 0x7ff28c0ac720 con 0x7ff294140660 2023-10-19T13:01:00.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.107+0000 7ff29259c640 1 --2- 172.21.15.57:0/767123171 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7ff27406af60 0x7ff27406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:00.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.107+0000 7ff29259c640 1 --2- 172.21.15.57:0/767123171 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7ff27406af60 0x7ff27406d420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff284003eb0 tx=0x7ff284009040 comp rx=0 tx=0).ready entity=mgr.6679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:00.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.111+0000 7ff28affd640 1 -- 172.21.15.57:0/767123171 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff2941f1980 con 0x7ff294140660 2023-10-19T13:01:00.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.324+0000 7ff298b55640 1 -- 172.21.15.57:0/767123171 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff2941f1980 con 0x7ff294140660 2023-10-19T13:01:00.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.325+0000 7ff28affd640 1 -- 172.21.15.57:0/767123171 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+4921 (secure 0 0 0) 0x7ff2941f1980 con 0x7ff294140660 2023-10-19T13:01:00.328 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:01:00.328 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":37,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6305,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/2631343514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":2631343514},{"type":"v1","addr":"172.21.15.179:6837","nonce":2631343514}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":34},{"gid":6883,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/4008617188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":4008617188},{"type":"v1","addr":"172.21.15.179:6836","nonce":4008617188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":34},{"gid":7542,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/2010253133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2010253133},{"type":"v1","addr":"172.21.15.57:6834","nonce":2010253133}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T13:00:55.547801+0000","modified":"2023-10-19T13:00:57.571937+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7545},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7545":{"gid":7545,"name":"a","rank":0,"incarnation":35,"state":"up:active","state_seq":3,"addr":"172.21.15.57:6835/2583001580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2583001580},{"type":"v1","addr":"172.21.15.57:6835","nonce":2583001580}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-10-19T13:01:00.329 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 37 2023-10-19T13:01:00.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.327+0000 7ff298b55640 1 -- 172.21.15.57:0/767123171 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7ff27406af60 msgr2=0x7ff27406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:00.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.327+0000 7ff298b55640 1 --2- 172.21.15.57:0/767123171 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7ff27406af60 0x7ff27406d420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff284003eb0 tx=0x7ff284009040 comp rx=0 tx=0).stop 2023-10-19T13:01:00.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.327+0000 7ff298b55640 1 -- 172.21.15.57:0/767123171 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff294140660 msgr2=0x7ff2941f4f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:00.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.327+0000 7ff298b55640 1 --2- 172.21.15.57:0/767123171 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff294140660 0x7ff2941f4f70 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7ff28c00c790 tx=0x7ff28c032e10 comp rx=0 tx=0).stop 2023-10-19T13:01:00.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.327+0000 7ff298b55640 1 -- 172.21.15.57:0/767123171 shutdown_connections 2023-10-19T13:01:00.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.327+0000 7ff298b55640 1 --2- 172.21.15.57:0/767123171 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7ff27406af60 0x7ff27406d420 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:00.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.327+0000 7ff298b55640 1 --2- 172.21.15.57:0/767123171 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff294140660 0x7ff2941f4f70 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:00.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.327+0000 7ff298b55640 1 --2- 172.21.15.57:0/767123171 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff29413d360 0x7ff294148760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:00.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.327+0000 7ff298b55640 1 --2- 172.21.15.57:0/767123171 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff2940c9d00 0x7ff2941f1c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:00.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.327+0000 7ff298b55640 1 -- 172.21.15.57:0/767123171 >> 172.21.15.57:0/767123171 conn(0x7ff294069930 msgr2=0x7ff29406c390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:00.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.327+0000 7ff298b55640 1 -- 172.21.15.57:0/767123171 shutdown_connections 2023-10-19T13:01:00.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.327+0000 7ff298b55640 1 -- 172.21.15.57:0/767123171 wait complete. 2023-10-19T13:01:00.742 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 37, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-19T13:00:55.547801+0000', 'modified': '2023-10-19T13:00:57.571937+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7545}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7545': {'gid': 7545, 'name': 'a', 'rank': 0, 'incarnation': 35, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.57:6835/2583001580', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6833', 'nonce': 2583001580}, {'type': 'v1', 'addr': '172.21.15.57:6835', 'nonce': 2583001580}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [12], 'metadata_pool': 11, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-19T13:01:00.743 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-19T13:01:00.743 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2023-10-19T13:01:00.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.856+0000 7f6d74d3c640 1 -- 172.21.15.57:0/206056771 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6d700c96b0 msgr2=0x7f6d700c9b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:00.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.856+0000 7f6d74d3c640 1 --2- 172.21.15.57:0/206056771 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6d700c96b0 0x7f6d700c9b30 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f6d58009860 tx=0x7f6d5802bd60 comp rx=0 tx=0).stop 2023-10-19T13:01:00.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.856+0000 7f6d74d3c640 1 -- 172.21.15.57:0/206056771 shutdown_connections 2023-10-19T13:01:00.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.856+0000 7f6d74d3c640 1 --2- 172.21.15.57:0/206056771 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6d700c96b0 0x7f6d700c9b30 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:00.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.856+0000 7f6d74d3c640 1 --2- 172.21.15.57:0/206056771 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6d7006af00 0x7f6d7006b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:00.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.856+0000 7f6d74d3c640 1 -- 172.21.15.57:0/206056771 >> 172.21.15.57:0/206056771 conn(0x7f6d70069930 msgr2=0x7f6d70069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:00.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.856+0000 7f6d74d3c640 1 -- 172.21.15.57:0/206056771 shutdown_connections 2023-10-19T13:01:00.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.856+0000 7f6d74d3c640 1 -- 172.21.15.57:0/206056771 wait complete. 2023-10-19T13:01:00.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.858+0000 7f6d74d3c640 1 Processor -- start 2023-10-19T13:01:00.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.858+0000 7f6d74d3c640 1 -- start start 2023-10-19T13:01:00.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.858+0000 7f6d74d3c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6d7006af00 0x7f6d7013a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:00.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.858+0000 7f6d74d3c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6d700ab500 0x7f6d7015b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:00.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.858+0000 7f6d74d3c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6d700c96b0 0x7f6d700aa8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:00.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.858+0000 7f6d74d3c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d7013aa70 con 0x7f6d700ab500 2023-10-19T13:01:00.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.858+0000 7f6d74d3c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d70155910 con 0x7f6d7006af00 2023-10-19T13:01:00.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.858+0000 7f6d74d3c640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d70070670 con 0x7f6d700c96b0 2023-10-19T13:01:00.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.859+0000 7f6d6dd9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6d700ab500 0x7f6d7015b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:00.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.859+0000 7f6d6dd9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6d700ab500 0x7f6d7015b970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38094/0 (socket says 172.21.15.57:38094) 2023-10-19T13:01:00.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.859+0000 7f6d6dd9b640 1 -- 172.21.15.57:0/2809534064 learned_addr learned my addr 172.21.15.57:0/2809534064 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:01:00.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.859+0000 7f6d6ed9d640 1 --2- 172.21.15.57:0/2809534064 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6d700c96b0 0x7f6d700aa8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:00.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.859+0000 7f6d6e59c640 1 --2- 172.21.15.57:0/2809534064 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6d7006af00 0x7f6d7013a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:00.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.859+0000 7f6d6dd9b640 1 -- 172.21.15.57:0/2809534064 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6d700c96b0 msgr2=0x7f6d700aa8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:00.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.859+0000 7f6d6dd9b640 1 --2- 172.21.15.57:0/2809534064 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6d700c96b0 0x7f6d700aa8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:00.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.859+0000 7f6d6dd9b640 1 -- 172.21.15.57:0/2809534064 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6d7006af00 msgr2=0x7f6d7013a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:00.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.859+0000 7f6d6dd9b640 1 --2- 172.21.15.57:0/2809534064 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6d7006af00 0x7f6d7013a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:00.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.859+0000 7f6d6dd9b640 1 -- 172.21.15.57:0/2809534064 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d68009490 con 0x7f6d700ab500 2023-10-19T13:01:00.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.859+0000 7f6d6dd9b640 1 --2- 172.21.15.57:0/2809534064 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6d700ab500 0x7f6d7015b970 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f6d58009980 tx=0x7f6d580038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:00.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.859+0000 7f6d66ffd640 1 -- 172.21.15.57:0/2809534064 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d58003ea0 con 0x7f6d700ab500 2023-10-19T13:01:00.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.859+0000 7f6d66ffd640 1 -- 172.21.15.57:0/2809534064 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f6d58034440 con 0x7f6d700ab500 2023-10-19T13:01:00.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.859+0000 7f6d74d3c640 1 -- 172.21.15.57:0/2809534064 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d58009490 con 0x7f6d700ab500 2023-10-19T13:01:00.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.859+0000 7f6d66ffd640 1 -- 172.21.15.57:0/2809534064 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d58034850 con 0x7f6d700ab500 2023-10-19T13:01:00.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.859+0000 7f6d74d3c640 1 -- 172.21.15.57:0/2809534064 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d70070c30 con 0x7f6d700ab500 2023-10-19T13:01:00.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.861+0000 7f6d66ffd640 1 -- 172.21.15.57:0/2809534064 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 40) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6d5803b040 con 0x7f6d700ab500 2023-10-19T13:01:00.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.861+0000 7f6d66ffd640 1 --2- 172.21.15.57:0/2809534064 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f6d5006af60 0x7f6d5006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:00.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.861+0000 7f6d66ffd640 1 -- 172.21.15.57:0/2809534064 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f6d580abe10 con 0x7f6d700ab500 2023-10-19T13:01:00.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.862+0000 7f6d6e59c640 1 --2- 172.21.15.57:0/2809534064 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f6d5006af60 0x7f6d5006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:00.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.862+0000 7f6d6e59c640 1 --2- 172.21.15.57:0/2809534064 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f6d5006af60 0x7f6d5006d420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6d70152830 tx=0x7f6d68005f90 comp rx=0 tx=0).ready entity=mgr.6679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:00.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.862+0000 7f6d779ac740 1 Processor -- start 2023-10-19T13:01:00.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.863+0000 7f6d779ac740 1 -- start start 2023-10-19T13:01:00.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.863+0000 7f6d779ac740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55cd7ae98b40 0x55cd7ae6bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:00.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.863+0000 7f6d779ac740 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55cd7ae98f20 0x55cd7ae9b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:00.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.863+0000 7f6d779ac740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cd7ae6c020 0x55cd7ae9be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:00.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.863+0000 7f6d779ac740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd7ad7d730 con 0x55cd7ae98b40 2023-10-19T13:01:00.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.863+0000 7f6d779ac740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd7adde780 con 0x55cd7ae6c020 2023-10-19T13:01:00.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.863+0000 7f6d779ac740 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cd7addbe70 con 0x55cd7ae98f20 2023-10-19T13:01:00.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.863+0000 7f6d6e59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55cd7ae98b40 0x55cd7ae6bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:00.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.863+0000 7f6d6e59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55cd7ae98b40 0x55cd7ae6bae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38110/0 (socket says 172.21.15.57:38110) 2023-10-19T13:01:00.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.863+0000 7f6d6e59c640 1 -- 172.21.15.57:0/1404453575 learned_addr learned my addr 172.21.15.57:0/1404453575 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:01:00.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.864+0000 7f6d6dd9b640 1 --2- 172.21.15.57:0/1404453575 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55cd7ae98f20 0x55cd7ae9b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:00.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.864+0000 7f6d6ed9d640 1 --2- 172.21.15.57:0/1404453575 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cd7ae6c020 0x55cd7ae9be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:00.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.864+0000 7f6d6e59c640 1 -- 172.21.15.57:0/1404453575 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55cd7ae98f20 msgr2=0x55cd7ae9b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T13:01:00.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.864+0000 7f6d6e59c640 1 --2- 172.21.15.57:0/1404453575 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55cd7ae98f20 0x55cd7ae9b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:00.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.864+0000 7f6d6e59c640 1 -- 172.21.15.57:0/1404453575 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cd7ae6c020 msgr2=0x55cd7ae9be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T13:01:00.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.864+0000 7f6d6e59c640 1 --2- 172.21.15.57:0/1404453575 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cd7ae6c020 0x55cd7ae9be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:00.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.864+0000 7f6d6e59c640 1 -- 172.21.15.57:0/1404453575 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cd7addc3d0 con 0x55cd7ae98b40 2023-10-19T13:01:00.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.864+0000 7f6d6e59c640 1 --2- 172.21.15.57:0/1404453575 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55cd7ae98b40 0x55cd7ae6bae0 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f6d68012530 tx=0x7f6d68014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bae345605247716 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:00.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.864+0000 7f6d4ffff640 1 -- 172.21.15.57:0/1404453575 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d68005220 con 0x55cd7ae98b40 2023-10-19T13:01:00.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.864+0000 7f6d4ffff640 1 -- 172.21.15.57:0/1404453575 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f6d68016070 con 0x55cd7ae98b40 2023-10-19T13:01:00.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.864+0000 7f6d4ffff640 1 -- 172.21.15.57:0/1404453575 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d680047a0 con 0x55cd7ae98b40 2023-10-19T13:01:00.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.865+0000 7f6d779ac740 1 -- 172.21.15.57:0/1404453575 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55cd7ae98b40 msgr2=0x55cd7ae6bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:00.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.865+0000 7f6d779ac740 1 --2- 172.21.15.57:0/1404453575 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55cd7ae98b40 0x55cd7ae6bae0 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f6d68012530 tx=0x7f6d68014000 comp rx=0 tx=0).stop 2023-10-19T13:01:00.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.865+0000 7f6d779ac740 1 -- 172.21.15.57:0/1404453575 shutdown_connections 2023-10-19T13:01:00.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.865+0000 7f6d779ac740 1 --2- 172.21.15.57:0/1404453575 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cd7ae6c020 0x55cd7ae9be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:00.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.865+0000 7f6d779ac740 1 --2- 172.21.15.57:0/1404453575 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55cd7ae98f20 0x55cd7ae9b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:00.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.865+0000 7f6d779ac740 1 --2- 172.21.15.57:0/1404453575 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55cd7ae98b40 0x55cd7ae6bae0 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:00.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.865+0000 7f6d779ac740 1 -- 172.21.15.57:0/1404453575 >> 172.21.15.57:0/1404453575 conn(0x55cd7ae6dd50 msgr2=0x55cd7aeb68d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:00.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.865+0000 7f6d779ac740 1 -- 172.21.15.57:0/1404453575 shutdown_connections 2023-10-19T13:01:00.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.865+0000 7f6d779ac740 1 -- 172.21.15.57:0/1404453575 wait complete. 2023-10-19T13:01:00.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.865+0000 7f6d779ac740 1 -- start start 2023-10-19T13:01:00.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.866+0000 7f6d779ac740 1 Processor -- start 2023-10-19T13:01:00.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.866+0000 7f6d779ac740 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55cd7ae98b40 0x55cd7ae9e320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:00.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.866+0000 7f6d779ac740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cd7ae98f20 0x55cd7aead3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:00.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.866+0000 7f6d779ac740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55cd7ae88ef0 0x55cd7aead930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:00.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.866+0000 7f6d779ac740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd7add7760 con 0x55cd7ae88ef0 2023-10-19T13:01:00.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.866+0000 7f6d779ac740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd7ad928f0 con 0x55cd7ae98f20 2023-10-19T13:01:00.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.866+0000 7f6d779ac740 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cd7ad92590 con 0x55cd7ae98b40 2023-10-19T13:01:00.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.867+0000 7f6d779ac740 10 client.0 _ll_register_callbacks cb 0x55cd7adceee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-19T13:01:00.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.867+0000 7f6d6ed9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55cd7ae88ef0 0x55cd7aead930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:00.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.867+0000 7f6d6ed9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55cd7ae88ef0 0x55cd7aead930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38118/0 (socket says 172.21.15.57:38118) 2023-10-19T13:01:00.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.867+0000 7f6d6ed9d640 1 -- 172.21.15.57:0/63976519 learned_addr learned my addr 172.21.15.57:0/63976519 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:01:00.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.867+0000 7f6d6e59c640 1 --2- 172.21.15.57:0/63976519 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55cd7ae98b40 0x55cd7ae9e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:00.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.867+0000 7f6d6dd9b640 1 --2- 172.21.15.57:0/63976519 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cd7ae98f20 0x55cd7aead3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:00.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.867+0000 7f6d6ed9d640 1 -- 172.21.15.57:0/63976519 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55cd7ae98b40 msgr2=0x55cd7ae9e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T13:01:00.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.867+0000 7f6d6ed9d640 1 --2- 172.21.15.57:0/63976519 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55cd7ae98b40 0x55cd7ae9e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:00.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.867+0000 7f6d6ed9d640 1 -- 172.21.15.57:0/63976519 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cd7ae98f20 msgr2=0x55cd7aead3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T13:01:00.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.867+0000 7f6d6ed9d640 1 --2- 172.21.15.57:0/63976519 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cd7ae98f20 0x55cd7aead3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:00.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.867+0000 7f6d6ed9d640 1 -- 172.21.15.57:0/63976519 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d68009db0 con 0x55cd7ae88ef0 2023-10-19T13:01:00.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.867+0000 7f6d6ed9d640 1 --2- 172.21.15.57:0/63976519 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55cd7ae88ef0 0x55cd7aead930 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f6d60011970 tx=0x7f6d60011d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46aa8a50b2147bda server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:00.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.867+0000 7f6d4d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2023-10-19T13:01:00.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.867+0000 7f6d4d7fa640 1 -- 172.21.15.57:0/63976519 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d6000f9c0 con 0x55cd7ae88ef0 2023-10-19T13:01:00.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.867+0000 7f6d4d7fa640 1 -- 172.21.15.57:0/63976519 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f6d6000fb20 con 0x55cd7ae88ef0 2023-10-19T13:01:00.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.867+0000 7f6d779ac740 1 -- 172.21.15.57:0/63976519 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55cd7add0740 con 0x55cd7ae88ef0 2023-10-19T13:01:00.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.867+0000 7f6d4d7fa640 1 -- 172.21.15.57:0/63976519 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6d6000fce0 con 0x55cd7ae88ef0 2023-10-19T13:01:00.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.867+0000 7f6d779ac740 10 client.7701 fetch_fsmap learned FSMap version 37 2023-10-19T13:01:00.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.867+0000 7f6d779ac740 1 -- 172.21.15.57:0/63976519 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=2+}) v3 -- 0x55cd7add5de0 con 0x55cd7ae88ef0 2023-10-19T13:01:00.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.868+0000 7f6d4d7fa640 1 -- 172.21.15.57:0/63976519 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x55cd7add0740 con 0x55cd7ae88ef0 2023-10-19T13:01:00.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.868+0000 7f6d4d7fa640 1 -- 172.21.15.57:0/63976519 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 37) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f6d6001e440 con 0x55cd7ae88ef0 2023-10-19T13:01:00.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.868+0000 7f6d779ac740 10 client.7701 fetch_fsmap finished waiting for FSMap version 37 2023-10-19T13:01:00.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.868+0000 7f6d779ac740 10 client.7701 resolve_mds: resolved name 'a' to daemon mds.a 2023-10-19T13:01:00.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.868+0000 7f6d779ac740 20 client.7701 populate_metadata read hostname 'smithi057' 2023-10-19T13:01:00.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.868+0000 7f6d779ac740 1 --2- 172.21.15.57:0/63976519 >> [v2:172.21.15.57:6833/2583001580,v1:172.21.15.57:6835/2583001580] conn(0x55cd7ae963a0 0x55cd7ae96780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:00.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.868+0000 7f6d779ac740 4 client.7701 mds_command: new command op to 7545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-19T13:01:00.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.868+0000 7f6d779ac740 1 -- 172.21.15.57:0/63976519 --> [v2:172.21.15.57:6833/2583001580,v1:172.21.15.57:6835/2583001580] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cd7add0460 con 0x55cd7ae963a0 2023-10-19T13:01:00.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.868+0000 7f6d6e59c640 1 --2- 172.21.15.57:0/63976519 >> [v2:172.21.15.57:6833/2583001580,v1:172.21.15.57:6835/2583001580] conn(0x55cd7ae963a0 0x55cd7ae96780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:00.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.868+0000 7f6d4d7fa640 10 client.7701 ms_handle_connect on v2:172.21.15.57:6833/2583001580 2023-10-19T13:01:00.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.868+0000 7f6d6e59c640 1 --2- 172.21.15.57:0/63976519 >> [v2:172.21.15.57:6833/2583001580,v1:172.21.15.57:6835/2583001580] conn(0x55cd7ae963a0 0x55cd7ae96780 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=234dfdb79d60e50e server_cookie=74f30cf92de72cd4 in_seq=0 out_seq=0 2023-10-19T13:01:00.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.869+0000 7f6d4d7fa640 1 -- 172.21.15.57:0/63976519 <== mds.0 v2:172.21.15.57:6833/2583001580 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55cd7add0460 con 0x55cd7ae963a0 2023-10-19T13:01:00.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.869+0000 7f6d4d7fa640 10 client.7701 handle_command_reply: tid=0 2023-10-19T13:01:00.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.869+0000 7f6d779ac740 1 client.7701 shutdown 2023-10-19T13:01:00.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.869+0000 7f6d779ac740 1 -- 172.21.15.57:0/63976519 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55cd7ae88ef0 msgr2=0x55cd7aead930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:00.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.869+0000 7f6d779ac740 1 --2- 172.21.15.57:0/63976519 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55cd7ae88ef0 0x55cd7aead930 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f6d60011970 tx=0x7f6d60011d70 comp rx=0 tx=0).stop 2023-10-19T13:01:00.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.869+0000 7f6d779ac740 1 -- 172.21.15.57:0/63976519 shutdown_connections 2023-10-19T13:01:00.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.869+0000 7f6d779ac740 1 --2- 172.21.15.57:0/63976519 >> [v2:172.21.15.57:6833/2583001580,v1:172.21.15.57:6835/2583001580] conn(0x55cd7ae963a0 0x55cd7ae96780 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:00.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.869+0000 7f6d779ac740 1 --2- 172.21.15.57:0/63976519 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55cd7ae88ef0 0x55cd7aead930 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:00.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.869+0000 7f6d779ac740 1 --2- 172.21.15.57:0/63976519 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cd7ae98f20 0x55cd7aead3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:00.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.869+0000 7f6d779ac740 1 --2- 172.21.15.57:0/63976519 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55cd7ae98b40 0x55cd7ae9e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:00.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.869+0000 7f6d4d7fa640 0 client.7701 ms_handle_reset on v2:172.21.15.57:6833/2583001580 2023-10-19T13:01:00.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.869+0000 7f6d779ac740 1 -- 172.21.15.57:0/63976519 >> 172.21.15.57:0/63976519 conn(0x55cd7ae6c020 msgr2=0x55cd7ae6fa60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:00.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.869+0000 7f6d779ac740 1 -- 172.21.15.57:0/63976519 shutdown_connections 2023-10-19T13:01:00.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.869+0000 7f6d779ac740 1 -- 172.21.15.57:0/63976519 wait complete. 2023-10-19T13:01:00.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.869+0000 7f6d779ac740 20 client.7701 trim_cache size 0 max 16384 2023-10-19T13:01:00.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.884+0000 7f6d779ac740 1 Processor -- start 2023-10-19T13:01:00.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.885+0000 7f6d779ac740 1 -- start start 2023-10-19T13:01:00.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.885+0000 7f6d779ac740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55cd7ae88ef0 0x55cd7aeb8f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:00.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.885+0000 7f6d779ac740 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55cd7ae98b40 0x55cd7ae9bd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:00.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.885+0000 7f6d779ac740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cd7ae98f20 0x55cd7aeb9de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:00.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.885+0000 7f6d779ac740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd7ad58270 con 0x55cd7ae88ef0 2023-10-19T13:01:00.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.885+0000 7f6d779ac740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd7add6900 con 0x55cd7ae98f20 2023-10-19T13:01:00.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.885+0000 7f6d779ac740 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cd7ad8de70 con 0x55cd7ae98b40 2023-10-19T13:01:00.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.885+0000 7f6d6e59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55cd7ae88ef0 0x55cd7aeb8f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:00.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.885+0000 7f6d6e59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55cd7ae88ef0 0x55cd7aeb8f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38134/0 (socket says 172.21.15.57:38134) 2023-10-19T13:01:00.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.885+0000 7f6d6e59c640 1 -- 172.21.15.57:0/3165144058 learned_addr learned my addr 172.21.15.57:0/3165144058 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:01:00.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.885+0000 7f6d6ed9d640 1 --2- 172.21.15.57:0/3165144058 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cd7ae98f20 0x55cd7aeb9de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:00.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.886+0000 7f6d6e59c640 1 -- 172.21.15.57:0/3165144058 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55cd7ae98b40 msgr2=0x55cd7ae9bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T13:01:00.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.886+0000 7f6d6dd9b640 1 --2- 172.21.15.57:0/3165144058 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55cd7ae98b40 0x55cd7ae9bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:00.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.886+0000 7f6d6e59c640 1 --2- 172.21.15.57:0/3165144058 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55cd7ae98b40 0x55cd7ae9bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:00.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.886+0000 7f6d6e59c640 1 -- 172.21.15.57:0/3165144058 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cd7ae98f20 msgr2=0x55cd7aeb9de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T13:01:00.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.886+0000 7f6d6e59c640 1 --2- 172.21.15.57:0/3165144058 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cd7ae98f20 0x55cd7aeb9de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:00.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.886+0000 7f6d6e59c640 1 -- 172.21.15.57:0/3165144058 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cd7ad74d00 con 0x55cd7ae88ef0 2023-10-19T13:01:00.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.886+0000 7f6d6ed9d640 1 --2- 172.21.15.57:0/3165144058 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cd7ae98f20 0x55cd7aeb9de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T13:01:00.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.886+0000 7f6d6dd9b640 1 --2- 172.21.15.57:0/3165144058 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55cd7ae98b40 0x55cd7ae9bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T13:01:00.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.886+0000 7f6d6e59c640 1 --2- 172.21.15.57:0/3165144058 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55cd7ae88ef0 0x55cd7aeb8f50 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f6d68012500 tx=0x7f6d68004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=670b48a10f6f137b server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:00.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.886+0000 7f6d64ff9640 1 -- 172.21.15.57:0/3165144058 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d6802c040 con 0x55cd7ae88ef0 2023-10-19T13:01:00.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.886+0000 7f6d64ff9640 1 -- 172.21.15.57:0/3165144058 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f6d6802d040 con 0x55cd7ae88ef0 2023-10-19T13:01:00.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.886+0000 7f6d64ff9640 1 -- 172.21.15.57:0/3165144058 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d68036e80 con 0x55cd7ae88ef0 2023-10-19T13:01:00.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.886+0000 7f6d779ac740 1 -- 172.21.15.57:0/3165144058 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55cd7ae88ef0 msgr2=0x55cd7aeb8f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:00.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.886+0000 7f6d779ac740 1 --2- 172.21.15.57:0/3165144058 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55cd7ae88ef0 0x55cd7aeb8f50 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f6d68012500 tx=0x7f6d68004730 comp rx=0 tx=0).stop 2023-10-19T13:01:00.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.886+0000 7f6d779ac740 1 -- 172.21.15.57:0/3165144058 shutdown_connections 2023-10-19T13:01:00.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.886+0000 7f6d779ac740 1 --2- 172.21.15.57:0/3165144058 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cd7ae98f20 0x55cd7aeb9de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:00.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.886+0000 7f6d779ac740 1 --2- 172.21.15.57:0/3165144058 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55cd7ae98b40 0x55cd7ae9bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:00.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.886+0000 7f6d779ac740 1 --2- 172.21.15.57:0/3165144058 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55cd7ae88ef0 0x55cd7aeb8f50 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:00.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.886+0000 7f6d779ac740 1 -- 172.21.15.57:0/3165144058 >> 172.21.15.57:0/3165144058 conn(0x55cd7ae6c020 msgr2=0x55cd7ae933f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:00.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.887+0000 7f6d779ac740 1 -- 172.21.15.57:0/3165144058 shutdown_connections 2023-10-19T13:01:00.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.887+0000 7f6d779ac740 1 -- 172.21.15.57:0/3165144058 wait complete. 2023-10-19T13:01:00.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.887+0000 7f6d779ac740 1 -- start start 2023-10-19T13:01:00.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.887+0000 7f6d779ac740 1 Processor -- start 2023-10-19T13:01:00.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.888+0000 7f6d779ac740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55cd7ae88ef0 0x55cd7af1eda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:00.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.888+0000 7f6d779ac740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cd7ae98b40 0x55cd7aeb9de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:00.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.888+0000 7f6d779ac740 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55cd7ae98f20 0x55cd7aeba320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:00.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.888+0000 7f6d779ac740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd7ae9b440 con 0x55cd7ae88ef0 2023-10-19T13:01:00.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.888+0000 7f6d779ac740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd7aeca8c0 con 0x55cd7ae98b40 2023-10-19T13:01:00.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.888+0000 7f6d779ac740 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cd7ae86ba0 con 0x55cd7ae98f20 2023-10-19T13:01:00.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.888+0000 7f6d779ac740 10 client.0 _ll_register_callbacks cb 0x7f6d68009960 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-19T13:01:00.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.888+0000 7f6d6e59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55cd7ae88ef0 0x55cd7af1eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:00.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.888+0000 7f6d6e59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55cd7ae88ef0 0x55cd7af1eda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38140/0 (socket says 172.21.15.57:38140) 2023-10-19T13:01:00.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.888+0000 7f6d6e59c640 1 -- 172.21.15.57:0/3112879944 learned_addr learned my addr 172.21.15.57:0/3112879944 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:01:00.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.888+0000 7f6d6dd9b640 1 --2- 172.21.15.57:0/3112879944 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cd7ae98b40 0x55cd7aeb9de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:00.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.888+0000 7f6d6ed9d640 1 --2- 172.21.15.57:0/3112879944 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55cd7ae98f20 0x55cd7aeba320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:00.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.888+0000 7f6d6e59c640 1 -- 172.21.15.57:0/3112879944 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55cd7ae98f20 msgr2=0x55cd7aeba320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T13:01:00.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.888+0000 7f6d6e59c640 1 --2- 172.21.15.57:0/3112879944 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55cd7ae98f20 0x55cd7aeba320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:00.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.888+0000 7f6d6e59c640 1 -- 172.21.15.57:0/3112879944 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cd7ae98b40 msgr2=0x55cd7aeb9de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T13:01:00.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.888+0000 7f6d6e59c640 1 --2- 172.21.15.57:0/3112879944 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cd7ae98b40 0x55cd7aeb9de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:00.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.888+0000 7f6d6e59c640 1 -- 172.21.15.57:0/3112879944 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cd7addc3d0 con 0x55cd7ae88ef0 2023-10-19T13:01:00.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.888+0000 7f6d6ed9d640 1 --2- 172.21.15.57:0/3112879944 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55cd7ae98f20 0x55cd7aeba320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:01:00.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.889+0000 7f6d6e59c640 1 --2- 172.21.15.57:0/3112879944 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55cd7ae88ef0 0x55cd7af1eda0 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f6d68023000 tx=0x7f6d68036410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c910f7414d711a85 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:00.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.889+0000 7f6d4d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2023-10-19T13:01:00.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.889+0000 7f6d4d7fa640 1 -- 172.21.15.57:0/3112879944 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d6802c040 con 0x55cd7ae88ef0 2023-10-19T13:01:00.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.889+0000 7f6d4d7fa640 1 -- 172.21.15.57:0/3112879944 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f6d68018070 con 0x55cd7ae88ef0 2023-10-19T13:01:00.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.889+0000 7f6d779ac740 1 -- 172.21.15.57:0/3112879944 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6d68016070 con 0x55cd7ae88ef0 2023-10-19T13:01:00.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.889+0000 7f6d4d7fa640 1 -- 172.21.15.57:0/3112879944 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6d68036540 con 0x55cd7ae88ef0 2023-10-19T13:01:00.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.889+0000 7f6d779ac740 10 client.7707 fetch_fsmap learned FSMap version 37 2023-10-19T13:01:00.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.889+0000 7f6d779ac740 1 -- 172.21.15.57:0/3112879944 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=2+}) v3 -- 0x55cd7ad9a4f0 con 0x55cd7ae88ef0 2023-10-19T13:01:00.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.889+0000 7f6d4d7fa640 1 -- 172.21.15.57:0/3112879944 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f6d68016070 con 0x55cd7ae88ef0 2023-10-19T13:01:00.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.889+0000 7f6d4d7fa640 1 -- 172.21.15.57:0/3112879944 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 37) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f6d68025030 con 0x55cd7ae88ef0 2023-10-19T13:01:00.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.889+0000 7f6d779ac740 10 client.7707 fetch_fsmap finished waiting for FSMap version 37 2023-10-19T13:01:00.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.889+0000 7f6d779ac740 10 client.7707 resolve_mds: resolved name 'a' to daemon mds.a 2023-10-19T13:01:00.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.889+0000 7f6d779ac740 20 client.7707 populate_metadata read hostname 'smithi057' 2023-10-19T13:01:00.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.889+0000 7f6d779ac740 1 --2- 172.21.15.57:0/3112879944 >> [v2:172.21.15.57:6833/2583001580,v1:172.21.15.57:6835/2583001580] conn(0x55cd7aec5420 0x55cd7aec5800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:00.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.889+0000 7f6d779ac740 4 client.7707 mds_command: new command op to 7545 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-19T13:01:00.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.889+0000 7f6d779ac740 1 -- 172.21.15.57:0/3112879944 --> [v2:172.21.15.57:6833/2583001580,v1:172.21.15.57:6835/2583001580] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55cd7aad96c0 con 0x55cd7aec5420 2023-10-19T13:01:00.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.889+0000 7f6d6dd9b640 1 --2- 172.21.15.57:0/3112879944 >> [v2:172.21.15.57:6833/2583001580,v1:172.21.15.57:6835/2583001580] conn(0x55cd7aec5420 0x55cd7aec5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:00.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.890+0000 7f6d4d7fa640 10 client.7707 ms_handle_connect on v2:172.21.15.57:6833/2583001580 2023-10-19T13:01:00.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.890+0000 7f6d6dd9b640 1 --2- 172.21.15.57:0/3112879944 >> [v2:172.21.15.57:6833/2583001580,v1:172.21.15.57:6835/2583001580] conn(0x55cd7aec5420 0x55cd7aec5800 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=183a36e1cae9ddd5 server_cookie=4f416cc06391490a in_seq=0 out_seq=0 2023-10-19T13:01:00.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.890+0000 7f6d4d7fa640 1 -- 172.21.15.57:0/3112879944 <== mds.0 v2:172.21.15.57:6833/2583001580 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x55cd7aad96c0 con 0x55cd7aec5420 2023-10-19T13:01:00.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.890+0000 7f6d4d7fa640 10 client.7707 handle_command_reply: tid=0 2023-10-19T13:01:00.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.890+0000 7f6d779ac740 1 client.7707 shutdown 2023-10-19T13:01:00.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.890+0000 7f6d779ac740 1 -- 172.21.15.57:0/3112879944 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55cd7ae88ef0 msgr2=0x55cd7af1eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:00.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.890+0000 7f6d779ac740 1 --2- 172.21.15.57:0/3112879944 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55cd7ae88ef0 0x55cd7af1eda0 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f6d68023000 tx=0x7f6d68036410 comp rx=0 tx=0).stop 2023-10-19T13:01:00.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.890+0000 7f6d779ac740 1 -- 172.21.15.57:0/3112879944 shutdown_connections 2023-10-19T13:01:00.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.890+0000 7f6d779ac740 1 --2- 172.21.15.57:0/3112879944 >> [v2:172.21.15.57:6833/2583001580,v1:172.21.15.57:6835/2583001580] conn(0x55cd7aec5420 0x55cd7aec5800 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:00.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.890+0000 7f6d779ac740 1 --2- 172.21.15.57:0/3112879944 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55cd7ae98f20 0x55cd7aeba320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:00.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.890+0000 7f6d779ac740 1 --2- 172.21.15.57:0/3112879944 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cd7ae98b40 0x55cd7aeb9de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:00.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.890+0000 7f6d4d7fa640 0 client.7707 ms_handle_reset on v2:172.21.15.57:6833/2583001580 2023-10-19T13:01:00.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.890+0000 7f6d779ac740 1 --2- 172.21.15.57:0/3112879944 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55cd7ae88ef0 0x55cd7af1eda0 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:00.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.890+0000 7f6d779ac740 1 -- 172.21.15.57:0/3112879944 >> 172.21.15.57:0/3112879944 conn(0x55cd7ae6c020 msgr2=0x55cd7ae9bbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:00.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.890+0000 7f6d779ac740 1 -- 172.21.15.57:0/3112879944 shutdown_connections 2023-10-19T13:01:00.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.890+0000 7f6d779ac740 1 -- 172.21.15.57:0/3112879944 wait complete. 2023-10-19T13:01:00.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.890+0000 7f6d779ac740 20 client.7707 trim_cache size 0 max 16384 2023-10-19T13:01:00.914 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2023-10-19T13:01:00.915 INFO:teuthology.orchestra.run.smithi057.stdout: "cluster_fsid": "1e9beb19-3260-4dd7-85ef-5b399ce774ac", 2023-10-19T13:01:00.915 INFO:teuthology.orchestra.run.smithi057.stdout: "whoami": 0, 2023-10-19T13:01:00.915 INFO:teuthology.orchestra.run.smithi057.stdout: "id": 7545, 2023-10-19T13:01:00.915 INFO:teuthology.orchestra.run.smithi057.stdout: "want_state": "up:active", 2023-10-19T13:01:00.916 INFO:teuthology.orchestra.run.smithi057.stdout: "state": "up:active", 2023-10-19T13:01:00.916 INFO:teuthology.orchestra.run.smithi057.stdout: "fs_name": "cephfs", 2023-10-19T13:01:00.916 INFO:teuthology.orchestra.run.smithi057.stdout: "rank_uptime": 4.6723230249999999, 2023-10-19T13:01:00.916 INFO:teuthology.orchestra.run.smithi057.stdout: "mdsmap_epoch": 37, 2023-10-19T13:01:00.916 INFO:teuthology.orchestra.run.smithi057.stdout: "osdmap_epoch": 100, 2023-10-19T13:01:00.916 INFO:teuthology.orchestra.run.smithi057.stdout: "osdmap_epoch_barrier": 100, 2023-10-19T13:01:00.917 INFO:teuthology.orchestra.run.smithi057.stdout: "uptime": 9.8325589069999992 2023-10-19T13:01:00.917 INFO:teuthology.orchestra.run.smithi057.stdout:} 2023-10-19T13:01:00.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.891+0000 7f6d74d3c640 1 -- 172.21.15.57:0/2809534064 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f6d5006af60 msgr2=0x7f6d5006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:00.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.891+0000 7f6d74d3c640 1 --2- 172.21.15.57:0/2809534064 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f6d5006af60 0x7f6d5006d420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6d70152830 tx=0x7f6d68005f90 comp rx=0 tx=0).stop 2023-10-19T13:01:00.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.891+0000 7f6d74d3c640 1 -- 172.21.15.57:0/2809534064 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6d700ab500 msgr2=0x7f6d7015b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:00.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.891+0000 7f6d74d3c640 1 --2- 172.21.15.57:0/2809534064 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6d700ab500 0x7f6d7015b970 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f6d58009980 tx=0x7f6d580038a0 comp rx=0 tx=0).stop 2023-10-19T13:01:00.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.891+0000 7f6d74d3c640 1 -- 172.21.15.57:0/2809534064 shutdown_connections 2023-10-19T13:01:00.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.891+0000 7f6d74d3c640 1 --2- 172.21.15.57:0/2809534064 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f6d5006af60 0x7f6d5006d420 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:00.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.891+0000 7f6d74d3c640 1 --2- 172.21.15.57:0/2809534064 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6d700c96b0 0x7f6d700aa8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:00.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.891+0000 7f6d74d3c640 1 --2- 172.21.15.57:0/2809534064 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6d700ab500 0x7f6d7015b970 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:00.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.891+0000 7f6d74d3c640 1 --2- 172.21.15.57:0/2809534064 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6d7006af00 0x7f6d7013a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:00.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.891+0000 7f6d74d3c640 1 -- 172.21.15.57:0/2809534064 >> 172.21.15.57:0/2809534064 conn(0x7f6d70069930 msgr2=0x7f6d70069d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:00.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.891+0000 7f6d74d3c640 1 -- 172.21.15.57:0/2809534064 shutdown_connections 2023-10-19T13:01:00.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:00.891+0000 7f6d74d3c640 1 -- 172.21.15.57:0/2809534064 wait complete. 2023-10-19T13:01:00.921 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-19T13:01:01.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:01.029+0000 7f1c6c55e640 1 -- 172.21.15.57:0/2741880411 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1c6415b2a0 msgr2=0x7f1c640cf720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:01.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:01.029+0000 7f1c6c55e640 1 --2- 172.21.15.57:0/2741880411 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1c6415b2a0 0x7f1c640cf720 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f1c54009830 tx=0x7f1c5402bcd0 comp rx=0 tx=0).stop 2023-10-19T13:01:01.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:01.029+0000 7f1c6c55e640 1 -- 172.21.15.57:0/2741880411 shutdown_connections 2023-10-19T13:01:01.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:01.029+0000 7f1c6c55e640 1 --2- 172.21.15.57:0/2741880411 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1c6415b2a0 0x7f1c640cf720 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:01.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:01.029+0000 7f1c6c55e640 1 --2- 172.21.15.57:0/2741880411 >> v2:172.21.15.179:3300/0 conn(0x7f1c640cf340 0x7f1c640d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:01.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:01.029+0000 7f1c6c55e640 1 -- 172.21.15.57:0/2741880411 >> 172.21.15.57:0/2741880411 conn(0x7f1c640d0d60 msgr2=0x7f1c640d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:01.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:01.029+0000 7f1c6c55e640 1 -- 172.21.15.57:0/2741880411 shutdown_connections 2023-10-19T13:01:01.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:01.029+0000 7f1c6c55e640 1 -- 172.21.15.57:0/2741880411 wait complete. 2023-10-19T13:01:01.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:01.030+0000 7f1c6c55e640 1 Processor -- start 2023-10-19T13:01:01.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:01.030+0000 7f1c6c55e640 1 -- start start 2023-10-19T13:01:01.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:01.031+0000 7f1c6c55e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1c640ceb40 0x7f1c641570c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:01.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:01.031+0000 7f1c6c55e640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1c640cf340 0x7f1c640cac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:01.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:01.031+0000 7f1c6c55e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1c6415b2a0 0x7f1c64159f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:01.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:01.031+0000 7f1c6c55e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c641576a0 con 0x7f1c640ceb40 2023-10-19T13:01:01.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:01.031+0000 7f1c6c55e640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c64157980 con 0x7f1c6415b2a0 2023-10-19T13:01:01.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:01.031+0000 7f1c6c55e640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c64157c60 con 0x7f1c640cf340 2023-10-19T13:01:01.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:01.031+0000 7f1c6a2fa640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1c640ceb40 0x7f1c641570c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:01.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:01.031+0000 7f1c6a2fa640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1c640ceb40 0x7f1c641570c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38156/0 (socket says 172.21.15.57:38156) 2023-10-19T13:01:01.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:01.031+0000 7f1c6a2fa640 1 -- 172.21.15.57:0/3903807250 learned_addr learned my addr 172.21.15.57:0/3903807250 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:01:01.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:01.031+0000 7f1c69af9640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1c640cf340 0x7f1c640cac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:01.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:01.031+0000 7f1c6aafb640 1 --2- 172.21.15.57:0/3903807250 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1c6415b2a0 0x7f1c64159f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:01.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:01.031+0000 7f1c6a2fa640 1 -- 172.21.15.57:0/3903807250 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1c640cf340 msgr2=0x7f1c640cac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:01.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:01.031+0000 7f1c6a2fa640 1 --2- 172.21.15.57:0/3903807250 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1c640cf340 0x7f1c640cac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:01.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:01.031+0000 7f1c6a2fa640 1 -- 172.21.15.57:0/3903807250 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1c6415b2a0 msgr2=0x7f1c64159f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:01.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:01.031+0000 7f1c6a2fa640 1 --2- 172.21.15.57:0/3903807250 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1c6415b2a0 0x7f1c64159f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:01.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:01.031+0000 7f1c6a2fa640 1 -- 172.21.15.57:0/3903807250 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c60009490 con 0x7f1c640ceb40 2023-10-19T13:01:01.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:01.031+0000 7f1c6a2fa640 1 --2- 172.21.15.57:0/3903807250 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1c640ceb40 0x7f1c641570c0 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f1c6000ea60 tx=0x7f1c6000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:01.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:01.031+0000 7f1c5affd640 1 -- 172.21.15.57:0/3903807250 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c6000cde0 con 0x7f1c640ceb40 2023-10-19T13:01:01.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:01.031+0000 7f1c6c55e640 1 -- 172.21.15.57:0/3903807250 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c54009490 con 0x7f1c640ceb40 2023-10-19T13:01:01.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:01.032+0000 7f1c5affd640 1 -- 172.21.15.57:0/3903807250 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f1c60004020 con 0x7f1c640ceb40 2023-10-19T13:01:01.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:01.032+0000 7f1c5affd640 1 -- 172.21.15.57:0/3903807250 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c60004410 con 0x7f1c640ceb40 2023-10-19T13:01:01.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:01.032+0000 7f1c6c55e640 1 -- 172.21.15.57:0/3903807250 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c64158220 con 0x7f1c640ceb40 2023-10-19T13:01:01.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:01.033+0000 7f1c6c55e640 1 -- 172.21.15.57:0/3903807250 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c640690e0 con 0x7f1c640ceb40 2023-10-19T13:01:01.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:01.033+0000 7f1c5affd640 1 -- 172.21.15.57:0/3903807250 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 40) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f1c600035f0 con 0x7f1c640ceb40 2023-10-19T13:01:01.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:01.033+0000 7f1c5affd640 1 --2- 172.21.15.57:0/3903807250 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f1c4806af60 0x7f1c4806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:01.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:01.034+0000 7f1c5affd640 1 -- 172.21.15.57:0/3903807250 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f1c60014070 con 0x7f1c640ceb40 2023-10-19T13:01:01.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:01.034+0000 7f1c69af9640 1 --2- 172.21.15.57:0/3903807250 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f1c4806af60 0x7f1c4806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:01.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:01.034+0000 7f1c69af9640 1 --2- 172.21.15.57:0/3903807250 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f1c4806af60 0x7f1c4806d420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1c54036000 tx=0x7f1c54004730 comp rx=0 tx=0).ready entity=mgr.6679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:01.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:01.038+0000 7f1c5affd640 1 -- 172.21.15.57:0/3903807250 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f1c600552f0 con 0x7f1c640ceb40 2023-10-19T13:01:01.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:01.252+0000 7f1c6c55e640 1 -- 172.21.15.57:0/3903807250 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f1c640d01d0 con 0x7f1c640ceb40 2023-10-19T13:01:01.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:01.252+0000 7f1c5affd640 1 -- 172.21.15.57:0/3903807250 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v37) v1 ==== 53+0+73 (secure 0 0 0) 0x7f1c60058910 con 0x7f1c640ceb40 2023-10-19T13:01:01.256 INFO:teuthology.orchestra.run.smithi057.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-19T13:01:01.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:01.254+0000 7f1c6c55e640 1 -- 172.21.15.57:0/3903807250 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f1c4806af60 msgr2=0x7f1c4806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:01.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:01.254+0000 7f1c6c55e640 1 --2- 172.21.15.57:0/3903807250 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f1c4806af60 0x7f1c4806d420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1c54036000 tx=0x7f1c54004730 comp rx=0 tx=0).stop 2023-10-19T13:01:01.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:01.254+0000 7f1c6c55e640 1 -- 172.21.15.57:0/3903807250 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1c640ceb40 msgr2=0x7f1c641570c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:01.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:01.254+0000 7f1c6c55e640 1 --2- 172.21.15.57:0/3903807250 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1c640ceb40 0x7f1c641570c0 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f1c6000ea60 tx=0x7f1c6000ef20 comp rx=0 tx=0).stop 2023-10-19T13:01:01.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:01.254+0000 7f1c6c55e640 1 -- 172.21.15.57:0/3903807250 shutdown_connections 2023-10-19T13:01:01.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:01.254+0000 7f1c6c55e640 1 --2- 172.21.15.57:0/3903807250 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f1c4806af60 0x7f1c4806d420 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:01.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:01.254+0000 7f1c6c55e640 1 --2- 172.21.15.57:0/3903807250 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1c6415b2a0 0x7f1c64159f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:01.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:01.254+0000 7f1c6c55e640 1 --2- 172.21.15.57:0/3903807250 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1c640cf340 0x7f1c640cac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:01.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:01.254+0000 7f1c6c55e640 1 --2- 172.21.15.57:0/3903807250 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1c640ceb40 0x7f1c641570c0 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:01.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:01.254+0000 7f1c6c55e640 1 -- 172.21.15.57:0/3903807250 >> 172.21.15.57:0/3903807250 conn(0x7f1c640d0d60 msgr2=0x7f1c6415d6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:01.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:01.254+0000 7f1c6c55e640 1 -- 172.21.15.57:0/3903807250 shutdown_connections 2023-10-19T13:01:01.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:01.254+0000 7f1c6c55e640 1 -- 172.21.15.57:0/3903807250 wait complete. 2023-10-19T13:01:01.267 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-10-19T13:01:01.268 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-19T13:01:01.268 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi057.front.sepia.ceph.com 2023-10-19T13:01:01.268 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-19T13:01:01.269 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-19T13:01:01.269 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-10-19T13:01:01.269 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-10-19T13:01:01.270 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-19T13:01:01.270 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-19T13:01:01.270 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:01:01.271 DEBUG:teuthology.orchestra.run.smithi057:> ip addr 2023-10-19T13:01:01.286 INFO:teuthology.orchestra.run.smithi057.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-19T13:01:01.286 INFO:teuthology.orchestra.run.smithi057.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-19T13:01:01.287 INFO:teuthology.orchestra.run.smithi057.stdout: inet 127.0.0.1/8 scope host lo 2023-10-19T13:01:01.287 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T13:01:01.287 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 ::1/128 scope host 2023-10-19T13:01:01.287 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T13:01:01.287 INFO:teuthology.orchestra.run.smithi057.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-19T13:01:01.288 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:6b:ff:86 brd ff:ff:ff:ff:ff:ff 2023-10-19T13:01:01.288 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp7s0f0 2023-10-19T13:01:01.288 INFO:teuthology.orchestra.run.smithi057.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-19T13:01:01.288 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:6b:ff:87 brd ff:ff:ff:ff:ff:ff 2023-10-19T13:01:01.288 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp7s0f1 2023-10-19T13:01:01.288 INFO:teuthology.orchestra.run.smithi057.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-19T13:01:01.289 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:bd:15:62 brd ff:ff:ff:ff:ff:ff 2023-10-19T13:01:01.289 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp1s0f0 2023-10-19T13:01:01.289 INFO:teuthology.orchestra.run.smithi057.stdout: inet 172.21.15.57/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-19T13:01:01.289 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft 41649sec preferred_lft 41649sec 2023-10-19T13:01:01.289 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 fe80::ec4:7aff:febd:1562/64 scope link noprefixroute 2023-10-19T13:01:01.289 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T13:01:01.290 INFO:teuthology.orchestra.run.smithi057.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-19T13:01:01.290 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:bd:15:63 brd ff:ff:ff:ff:ff:ff 2023-10-19T13:01:01.290 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp1s0f1 2023-10-19T13:01:01.290 INFO:teuthology.orchestra.run.smithi057.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-19T13:01:01.290 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0e:fe:f1:b5:41:59 brd ff:ff:ff:ff:ff:ff 2023-10-19T13:01:01.291 INFO:teuthology.orchestra.run.smithi057.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-19T13:01:01.291 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T13:01:01.291 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 fe80::64b1:a2ff:feb3:b08/64 scope link 2023-10-19T13:01:01.291 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T13:01:01.291 INFO:teuthology.orchestra.run.smithi057.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-19T13:01:01.292 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0e:fe:f1:b5:41:59 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T13:01:01.292 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 fe80::cfe:f1ff:feb5:4159/64 scope link 2023-10-19T13:01:01.292 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T13:01:01.292 INFO:teuthology.orchestra.run.smithi057.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-19T13:01:01.292 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 62:53:72:9d:52:71 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-19T13:01:01.292 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 fe80::6053:72ff:fe9d:5271/64 scope link 2023-10-19T13:01:01.293 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T13:01:01.293 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:01:01.293 DEBUG:teuthology.orchestra.run.smithi057:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-19T13:01:01.355 INFO:teuthology.orchestra.run.smithi057.stdout:1 2023-10-19T13:01:01.357 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:01:01.357 DEBUG:teuthology.orchestra.run.smithi057:> route 2023-10-19T13:01:01.414 INFO:teuthology.orchestra.run.smithi057.stdout:Kernel IP routing table 2023-10-19T13:01:01.414 INFO:teuthology.orchestra.run.smithi057.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-19T13:01:01.414 INFO:teuthology.orchestra.run.smithi057.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-19T13:01:01.414 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-19T13:01:01.415 INFO:teuthology.orchestra.run.smithi057.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-19T13:01:01.415 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:01:01.415 DEBUG:teuthology.orchestra.run.smithi057:> (cd / && exec bash -c ' 2023-10-19T13:01:01.416 DEBUG:teuthology.orchestra.run.smithi057:> set -e 2023-10-19T13:01:01.416 DEBUG:teuthology.orchestra.run.smithi057:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-19T13:01:01.417 DEBUG:teuthology.orchestra.run.smithi057:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-19T13:01:01.417 DEBUG:teuthology.orchestra.run.smithi057:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-19T13:01:01.417 DEBUG:teuthology.orchestra.run.smithi057:> ') 2023-10-19T13:01:01.520 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:01:01.520 DEBUG:teuthology.orchestra.run.smithi057:> ip netns list 2023-10-19T13:01:01.535 INFO:teuthology.orchestra.run.smithi057.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-19T13:01:01.536 INFO:teuthology.orchestra.run.smithi057.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-19T13:01:01.536 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-19T13:01:01.537 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:01:01.537 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip link set brx.0 up 2023-10-19T13:01:01.606 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-10-19T13:01:01.607 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-19T13:01:01.607 DEBUG:teuthology.orchestra.run.smithi057:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-19T13:01:01.662 INFO:teuthology.orchestra.run.smithi057.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-19T13:01:01.662 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-19T13:01:01.663 DEBUG:teuthology.orchestra.run.smithi057:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-19T13:01:01.717 DEBUG:teuthology.orchestra.run.smithi057:> sudo modprobe fuse 2023-10-19T13:01:01.766 INFO:tasks.ceph.mds.a.smithi057.stderr:2023-10-19T13:01:01.764+0000 7f9809bd3640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-19T13:01:01.788 DEBUG:teuthology.orchestra.run.smithi057:> cat /proc/self/mounts | awk '{print $2}' 2023-10-19T13:01:01.844 INFO:teuthology.orchestra.run.smithi057.stdout:/proc 2023-10-19T13:01:01.845 INFO:teuthology.orchestra.run.smithi057.stdout:/sys 2023-10-19T13:01:01.845 INFO:teuthology.orchestra.run.smithi057.stdout:/dev 2023-10-19T13:01:01.845 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/security 2023-10-19T13:01:01.845 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/shm 2023-10-19T13:01:01.845 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/pts 2023-10-19T13:01:01.846 INFO:teuthology.orchestra.run.smithi057.stdout:/run 2023-10-19T13:01:01.846 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/cgroup 2023-10-19T13:01:01.846 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/pstore 2023-10-19T13:01:01.846 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/bpf 2023-10-19T13:01:01.846 INFO:teuthology.orchestra.run.smithi057.stdout:/ 2023-10-19T13:01:01.847 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/selinux 2023-10-19T13:01:01.847 INFO:teuthology.orchestra.run.smithi057.stdout:/proc/sys/fs/binfmt_misc 2023-10-19T13:01:01.847 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/hugepages 2023-10-19T13:01:01.847 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/mqueue 2023-10-19T13:01:01.847 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/debug 2023-10-19T13:01:01.848 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/tracing 2023-10-19T13:01:01.848 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/fuse/connections 2023-10-19T13:01:01.848 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/config 2023-10-19T13:01:01.848 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-sysctl.service 2023-10-19T13:01:01.848 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-10-19T13:01:01.849 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-10-19T13:01:01.849 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/nfs/rpc_pipefs 2023-10-19T13:01:01.849 INFO:teuthology.orchestra.run.smithi057.stdout:/run/user/1000 2023-10-19T13:01:01.849 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph 2023-10-19T13:01:01.849 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-0 2023-10-19T13:01:01.850 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-1 2023-10-19T13:01:01.850 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-2 2023-10-19T13:01:01.850 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-3 2023-10-19T13:01:01.850 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns 2023-10-19T13:01:01.850 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T13:01:01.851 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T13:01:01.851 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-19T13:01:01.851 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-19T13:01:01.851 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:01:01.852 DEBUG:teuthology.orchestra.run.smithi057:> ls /sys/fs/fuse/connections 2023-10-19T13:01:01.899 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-10-19T13:01:01.899 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-10-19T13:01:01.942 DEBUG:teuthology.orchestra.run.smithi057:> sudo modprobe fuse 2023-10-19T13:01:01.972 DEBUG:teuthology.orchestra.run.smithi057:> cat /proc/self/mounts | awk '{print $2}' 2023-10-19T13:01:02.028 INFO:teuthology.orchestra.run.smithi057.stdout:/proc 2023-10-19T13:01:02.029 INFO:teuthology.orchestra.run.smithi057.stdout:/sys 2023-10-19T13:01:02.029 INFO:teuthology.orchestra.run.smithi057.stdout:/dev 2023-10-19T13:01:02.029 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/security 2023-10-19T13:01:02.029 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/shm 2023-10-19T13:01:02.030 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/pts 2023-10-19T13:01:02.030 INFO:teuthology.orchestra.run.smithi057.stdout:/run 2023-10-19T13:01:02.030 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/cgroup 2023-10-19T13:01:02.030 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/pstore 2023-10-19T13:01:02.030 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/bpf 2023-10-19T13:01:02.031 INFO:teuthology.orchestra.run.smithi057.stdout:/ 2023-10-19T13:01:02.031 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/selinux 2023-10-19T13:01:02.031 INFO:teuthology.orchestra.run.smithi057.stdout:/proc/sys/fs/binfmt_misc 2023-10-19T13:01:02.031 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/hugepages 2023-10-19T13:01:02.031 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/mqueue 2023-10-19T13:01:02.032 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/debug 2023-10-19T13:01:02.032 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/tracing 2023-10-19T13:01:02.032 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/fuse/connections 2023-10-19T13:01:02.032 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/config 2023-10-19T13:01:02.032 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-sysctl.service 2023-10-19T13:01:02.033 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-10-19T13:01:02.033 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-10-19T13:01:02.033 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/nfs/rpc_pipefs 2023-10-19T13:01:02.033 INFO:teuthology.orchestra.run.smithi057.stdout:/run/user/1000 2023-10-19T13:01:02.033 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph 2023-10-19T13:01:02.034 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-0 2023-10-19T13:01:02.034 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-1 2023-10-19T13:01:02.034 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-2 2023-10-19T13:01:02.034 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-3 2023-10-19T13:01:02.034 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns 2023-10-19T13:01:02.035 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T13:01:02.035 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T13:01:02.035 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-19T13:01:02.035 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-19T13:01:02.035 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:01:02.036 DEBUG:teuthology.orchestra.run.smithi057:> ls /sys/fs/fuse/connections 2023-10-19T13:01:02.037 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi057.stderr:2023-10-19T13:01:02.033+0000 7f5e5b768480 -1 init, newargv = 0x5595fb46ac30 newargc=15 2023-10-19T13:01:02.037 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi057.stderr:ceph-fuse[122054]: starting ceph client 2023-10-19T13:01:02.039 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi057.stderr:ceph-fuse[122054]: starting fuse 2023-10-19T13:01:02.052 INFO:teuthology.orchestra.run.smithi057.stdout:43 2023-10-19T13:01:02.052 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2023-10-19T13:01:02.053 DEBUG:teuthology.orchestra.run.smithi057:> sudo stdin-killer -- python3 -c ' 2023-10-19T13:01:02.053 DEBUG:teuthology.orchestra.run.smithi057:> import glob 2023-10-19T13:01:02.054 DEBUG:teuthology.orchestra.run.smithi057:> import re 2023-10-19T13:01:02.054 DEBUG:teuthology.orchestra.run.smithi057:> import os 2023-10-19T13:01:02.054 DEBUG:teuthology.orchestra.run.smithi057:> import subprocess 2023-10-19T13:01:02.055 DEBUG:teuthology.orchestra.run.smithi057:> 2023-10-19T13:01:02.055 DEBUG:teuthology.orchestra.run.smithi057:> def _find_admin_socket(client_name): 2023-10-19T13:01:02.056 DEBUG:teuthology.orchestra.run.smithi057:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-10-19T13:01:02.056 DEBUG:teuthology.orchestra.run.smithi057:> files = glob.glob(asok_path) 2023-10-19T13:01:02.056 DEBUG:teuthology.orchestra.run.smithi057:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-10-19T13:01:02.057 DEBUG:teuthology.orchestra.run.smithi057:> 2023-10-19T13:01:02.057 DEBUG:teuthology.orchestra.run.smithi057:> # Given a non-glob path, it better be there 2023-10-19T13:01:02.058 DEBUG:teuthology.orchestra.run.smithi057:> if "*" not in asok_path: 2023-10-19T13:01:02.058 DEBUG:teuthology.orchestra.run.smithi057:> assert(len(files) == 1) 2023-10-19T13:01:02.058 DEBUG:teuthology.orchestra.run.smithi057:> return files[0] 2023-10-19T13:01:02.059 DEBUG:teuthology.orchestra.run.smithi057:> 2023-10-19T13:01:02.059 DEBUG:teuthology.orchestra.run.smithi057:> for f in files: 2023-10-19T13:01:02.059 DEBUG:teuthology.orchestra.run.smithi057:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-10-19T13:01:02.060 DEBUG:teuthology.orchestra.run.smithi057:> if os.path.exists("/proc/{0}".format(pid)): 2023-10-19T13:01:02.060 DEBUG:teuthology.orchestra.run.smithi057:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-10-19T13:01:02.061 DEBUG:teuthology.orchestra.run.smithi057:> contents = proc_f.read() 2023-10-19T13:01:02.061 DEBUG:teuthology.orchestra.run.smithi057:> if mountpoint in contents: 2023-10-19T13:01:02.061 DEBUG:teuthology.orchestra.run.smithi057:> return f 2023-10-19T13:01:02.062 DEBUG:teuthology.orchestra.run.smithi057:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-10-19T13:01:02.062 DEBUG:teuthology.orchestra.run.smithi057:> 2023-10-19T13:01:02.063 DEBUG:teuthology.orchestra.run.smithi057:> print(_find_admin_socket("client.0")) 2023-10-19T13:01:02.063 DEBUG:teuthology.orchestra.run.smithi057:> ' 2023-10-19T13:01:02.177 INFO:teuthology.orchestra.run.smithi057.stdout:/var/run/ceph/ceph-client.0.122054.asok 2023-10-19T13:01:02.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2023-10-19T13:01:02.188 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.122054.asok 2023-10-19T13:01:02.188 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:01:02.189 DEBUG:teuthology.orchestra.run.smithi057:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.122054.asok status 2023-10-19T13:01:02.330 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2023-10-19T13:01:02.330 INFO:teuthology.orchestra.run.smithi057.stdout: "metadata": { 2023-10-19T13:01:02.331 INFO:teuthology.orchestra.run.smithi057.stdout: "ceph_sha1": "f7420d7f8e55ceea786dcbf536c338ce920c21eb", 2023-10-19T13:01:02.331 INFO:teuthology.orchestra.run.smithi057.stdout: "ceph_version": "ceph version 18.0.0-6786-gf7420d7f (f7420d7f8e55ceea786dcbf536c338ce920c21eb) reef (dev)", 2023-10-19T13:01:02.331 INFO:teuthology.orchestra.run.smithi057.stdout: "entity_id": "0", 2023-10-19T13:01:02.331 INFO:teuthology.orchestra.run.smithi057.stdout: "hostname": "smithi057", 2023-10-19T13:01:02.331 INFO:teuthology.orchestra.run.smithi057.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-10-19T13:01:02.332 INFO:teuthology.orchestra.run.smithi057.stdout: "pid": "122054", 2023-10-19T13:01:02.332 INFO:teuthology.orchestra.run.smithi057.stdout: "root": "/" 2023-10-19T13:01:02.332 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:01:02.332 INFO:teuthology.orchestra.run.smithi057.stdout: "dentry_count": 0, 2023-10-19T13:01:02.332 INFO:teuthology.orchestra.run.smithi057.stdout: "dentry_pinned_count": 0, 2023-10-19T13:01:02.333 INFO:teuthology.orchestra.run.smithi057.stdout: "id": 7713, 2023-10-19T13:01:02.333 INFO:teuthology.orchestra.run.smithi057.stdout: "inst": { 2023-10-19T13:01:02.333 INFO:teuthology.orchestra.run.smithi057.stdout: "name": { 2023-10-19T13:01:02.333 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "client", 2023-10-19T13:01:02.333 INFO:teuthology.orchestra.run.smithi057.stdout: "num": 7713 2023-10-19T13:01:02.334 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:01:02.334 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": { 2023-10-19T13:01:02.334 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "v1", 2023-10-19T13:01:02.334 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": "192.168.0.1:0", 2023-10-19T13:01:02.335 INFO:teuthology.orchestra.run.smithi057.stdout: "nonce": 3368125258 2023-10-19T13:01:02.335 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:01:02.335 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:01:02.335 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": { 2023-10-19T13:01:02.335 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "v1", 2023-10-19T13:01:02.335 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": "192.168.0.1:0", 2023-10-19T13:01:02.336 INFO:teuthology.orchestra.run.smithi057.stdout: "nonce": 3368125258 2023-10-19T13:01:02.336 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:01:02.336 INFO:teuthology.orchestra.run.smithi057.stdout: "inst_str": "client.7713 192.168.0.1:0/3368125258", 2023-10-19T13:01:02.336 INFO:teuthology.orchestra.run.smithi057.stdout: "addr_str": "192.168.0.1:0/3368125258", 2023-10-19T13:01:02.336 INFO:teuthology.orchestra.run.smithi057.stdout: "inode_count": 1, 2023-10-19T13:01:02.337 INFO:teuthology.orchestra.run.smithi057.stdout: "mds_epoch": 37, 2023-10-19T13:01:02.337 INFO:teuthology.orchestra.run.smithi057.stdout: "osd_epoch": 100, 2023-10-19T13:01:02.337 INFO:teuthology.orchestra.run.smithi057.stdout: "osd_epoch_barrier": 0, 2023-10-19T13:01:02.337 INFO:teuthology.orchestra.run.smithi057.stdout: "blocklisted": false, 2023-10-19T13:01:02.337 INFO:teuthology.orchestra.run.smithi057.stdout: "fs_name": "cephfs" 2023-10-19T13:01:02.338 INFO:teuthology.orchestra.run.smithi057.stdout:} 2023-10-19T13:01:02.339 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:01:02.339 DEBUG:teuthology.orchestra.run.smithi057:> stat --file-system '--printf=%T 2023-10-19T13:01:02.340 DEBUG:teuthology.orchestra.run.smithi057:> ' -- /home/ubuntu/cephtest/mnt.0 2023-10-19T13:01:02.355 INFO:teuthology.orchestra.run.smithi057.stdout:fuseblk 2023-10-19T13:01:02.355 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-10-19T13:01:02.355 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:01:02.356 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-19T13:01:02.427 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-19T13:01:02.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.573+0000 7fefa29f5640 1 -- 172.21.15.57:0/1827003338 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fef9c15b2a0 msgr2=0x7fef9c0cf720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:02.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.573+0000 7fefa29f5640 1 --2- 172.21.15.57:0/1827003338 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fef9c15b2a0 0x7fef9c0cf720 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7fef90005fd0 tx=0x7fef9002bd90 comp rx=0 tx=0).stop 2023-10-19T13:01:02.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.573+0000 7fefa29f5640 1 -- 172.21.15.57:0/1827003338 shutdown_connections 2023-10-19T13:01:02.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.573+0000 7fefa29f5640 1 --2- 172.21.15.57:0/1827003338 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fef9c15b2a0 0x7fef9c0cf720 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:02.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.573+0000 7fefa29f5640 1 --2- 172.21.15.57:0/1827003338 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fef9c0cf340 0x7fef9c0d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:02.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.573+0000 7fefa29f5640 1 -- 172.21.15.57:0/1827003338 >> 172.21.15.57:0/1827003338 conn(0x7fef9c0d0d60 msgr2=0x7fef9c0d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:02.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.573+0000 7fefa29f5640 1 -- 172.21.15.57:0/1827003338 shutdown_connections 2023-10-19T13:01:02.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.573+0000 7fefa29f5640 1 -- 172.21.15.57:0/1827003338 wait complete. 2023-10-19T13:01:02.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.574+0000 7fefa29f5640 1 Processor -- start 2023-10-19T13:01:02.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.575+0000 7fefa29f5640 1 -- start start 2023-10-19T13:01:02.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.575+0000 7fefa29f5640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fef9c0ceb40 0x7fef9c158990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:02.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.575+0000 7fefa29f5640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fef9c0cf340 0x7fef9c0c8840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:02.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.575+0000 7fefa29f5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fef9c15b2a0 0x7fef9c1f1550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:02.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.575+0000 7fefa29f5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef9c158ed0 con 0x7fef9c15b2a0 2023-10-19T13:01:02.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.575+0000 7fefa29f5640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef9c0c92c0 con 0x7fef9c0ceb40 2023-10-19T13:01:02.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.575+0000 7fefa29f5640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef9c0c95a0 con 0x7fef9c0cf340 2023-10-19T13:01:02.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.575+0000 7fefa0f92640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fef9c15b2a0 0x7fef9c1f1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:02.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.575+0000 7fefa0f92640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fef9c15b2a0 0x7fef9c1f1550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38162/0 (socket says 172.21.15.57:38162) 2023-10-19T13:01:02.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.575+0000 7fefa0f92640 1 -- 172.21.15.57:0/3278802431 learned_addr learned my addr 172.21.15.57:0/3278802431 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:01:02.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.576+0000 7fef9b7fe640 1 --2- 172.21.15.57:0/3278802431 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fef9c0cf340 0x7fef9c0c8840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:02.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.576+0000 7fef9bfff640 1 --2- 172.21.15.57:0/3278802431 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fef9c0ceb40 0x7fef9c158990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:02.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.576+0000 7fefa0f92640 1 -- 172.21.15.57:0/3278802431 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fef9c0cf340 msgr2=0x7fef9c0c8840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:02.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.576+0000 7fefa0f92640 1 --2- 172.21.15.57:0/3278802431 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fef9c0cf340 0x7fef9c0c8840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:02.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.576+0000 7fefa0f92640 1 -- 172.21.15.57:0/3278802431 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fef9c0ceb40 msgr2=0x7fef9c158990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:02.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.576+0000 7fefa0f92640 1 --2- 172.21.15.57:0/3278802431 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fef9c0ceb40 0x7fef9c158990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:02.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.576+0000 7fefa0f92640 1 -- 172.21.15.57:0/3278802431 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef90009560 con 0x7fef9c15b2a0 2023-10-19T13:01:02.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.576+0000 7fefa0f92640 1 --2- 172.21.15.57:0/3278802431 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fef9c15b2a0 0x7fef9c1f1550 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7fef9400c7a0 tx=0x7fef9400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:02.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.576+0000 7fef98ff9640 1 -- 172.21.15.57:0/3278802431 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef94003b70 con 0x7fef9c15b2a0 2023-10-19T13:01:02.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.576+0000 7fef98ff9640 1 -- 172.21.15.57:0/3278802431 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fef94003cf0 con 0x7fef9c15b2a0 2023-10-19T13:01:02.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.576+0000 7fefa29f5640 1 -- 172.21.15.57:0/3278802431 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef9c0c9880 con 0x7fef9c15b2a0 2023-10-19T13:01:02.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.576+0000 7fef98ff9640 1 -- 172.21.15.57:0/3278802431 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef9400aab0 con 0x7fef9c15b2a0 2023-10-19T13:01:02.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.576+0000 7fefa29f5640 1 -- 172.21.15.57:0/3278802431 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef9c0c9e40 con 0x7fef9c15b2a0 2023-10-19T13:01:02.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.577+0000 7fefa29f5640 1 -- 172.21.15.57:0/3278802431 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef9c0690e0 con 0x7fef9c15b2a0 2023-10-19T13:01:02.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.578+0000 7fef98ff9640 1 -- 172.21.15.57:0/3278802431 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 40) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fef9400ac10 con 0x7fef9c15b2a0 2023-10-19T13:01:02.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.578+0000 7fef98ff9640 1 --2- 172.21.15.57:0/3278802431 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fef8806af90 0x7fef8806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:02.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.578+0000 7fef98ff9640 1 -- 172.21.15.57:0/3278802431 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fef940898d0 con 0x7fef9c15b2a0 2023-10-19T13:01:02.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.578+0000 7fef9bfff640 1 --2- 172.21.15.57:0/3278802431 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fef8806af90 0x7fef8806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:02.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.579+0000 7fef9bfff640 1 --2- 172.21.15.57:0/3278802431 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fef8806af90 0x7fef8806d450 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fef8c005fd0 tx=0x7fef8c005850 comp rx=0 tx=0).ready entity=mgr.6679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:02.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.582+0000 7fef98ff9640 1 -- 172.21.15.57:0/3278802431 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fef94054710 con 0x7fef9c15b2a0 2023-10-19T13:01:02.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.781+0000 7fefa29f5640 1 -- 172.21.15.57:0/3278802431 --> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fef9c0cc170 con 0x7fef8806af90 2023-10-19T13:01:02.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.784+0000 7fef98ff9640 1 -- 172.21.15.57:0/3278802431 <== mgr.6679 v2:172.21.15.179:6832/1413162726 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fef8c00b360 con 0x7fef8806af90 2023-10-19T13:01:02.786 INFO:teuthology.orchestra.run.smithi057.stdout:[ 2023-10-19T13:01:02.786 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:01:02.787 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "cephfs" 2023-10-19T13:01:02.787 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:01:02.787 INFO:teuthology.orchestra.run.smithi057.stdout:] 2023-10-19T13:01:02.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.786+0000 7fefa29f5640 1 -- 172.21.15.57:0/3278802431 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fef8806af90 msgr2=0x7fef8806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:02.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.786+0000 7fefa29f5640 1 --2- 172.21.15.57:0/3278802431 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fef8806af90 0x7fef8806d450 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fef8c005fd0 tx=0x7fef8c005850 comp rx=0 tx=0).stop 2023-10-19T13:01:02.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.786+0000 7fefa29f5640 1 -- 172.21.15.57:0/3278802431 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fef9c15b2a0 msgr2=0x7fef9c1f1550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:02.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.786+0000 7fefa29f5640 1 --2- 172.21.15.57:0/3278802431 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fef9c15b2a0 0x7fef9c1f1550 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7fef9400c7a0 tx=0x7fef9400cc60 comp rx=0 tx=0).stop 2023-10-19T13:01:02.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.786+0000 7fefa29f5640 1 -- 172.21.15.57:0/3278802431 shutdown_connections 2023-10-19T13:01:02.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.786+0000 7fefa29f5640 1 --2- 172.21.15.57:0/3278802431 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7fef8806af90 0x7fef8806d450 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:02.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.786+0000 7fefa29f5640 1 --2- 172.21.15.57:0/3278802431 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fef9c15b2a0 0x7fef9c1f1550 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:02.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.786+0000 7fefa29f5640 1 --2- 172.21.15.57:0/3278802431 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fef9c0cf340 0x7fef9c0c8840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:02.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.786+0000 7fefa29f5640 1 --2- 172.21.15.57:0/3278802431 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fef9c0ceb40 0x7fef9c158990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:02.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.786+0000 7fefa29f5640 1 -- 172.21.15.57:0/3278802431 >> 172.21.15.57:0/3278802431 conn(0x7fef9c0d0d60 msgr2=0x7fef9c06a9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:02.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.786+0000 7fefa29f5640 1 -- 172.21.15.57:0/3278802431 shutdown_connections 2023-10-19T13:01:02.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.786+0000 7fefa29f5640 1 -- 172.21.15.57:0/3278802431 wait complete. 2023-10-19T13:01:02.799 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-19T13:01:02.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.907+0000 7f42b5061640 1 -- 172.21.15.57:0/2277191157 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f42b0070920 msgr2=0x7f42b00affe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:02.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.907+0000 7f42b5061640 1 --2- 172.21.15.57:0/2277191157 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f42b0070920 0x7f42b00affe0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f42a0005fd0 tx=0x7f42a002c1b0 comp rx=0 tx=0).stop 2023-10-19T13:01:02.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.907+0000 7f42b5061640 1 -- 172.21.15.57:0/2277191157 shutdown_connections 2023-10-19T13:01:02.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.907+0000 7f42b5061640 1 --2- 172.21.15.57:0/2277191157 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f42b0070920 0x7f42b00affe0 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:02.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.907+0000 7f42b5061640 1 --2- 172.21.15.57:0/2277191157 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f42b0141690 0x7f42b00703e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:02.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.907+0000 7f42b5061640 1 -- 172.21.15.57:0/2277191157 >> 172.21.15.57:0/2277191157 conn(0x7f42b0069930 msgr2=0x7f42b0069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:02.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.907+0000 7f42b5061640 1 -- 172.21.15.57:0/2277191157 shutdown_connections 2023-10-19T13:01:02.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.907+0000 7f42b5061640 1 -- 172.21.15.57:0/2277191157 wait complete. 2023-10-19T13:01:02.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.908+0000 7f42b5061640 1 Processor -- start 2023-10-19T13:01:02.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.908+0000 7f42b5061640 1 -- start start 2023-10-19T13:01:02.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.909+0000 7f42b5061640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f42b0070920 0x7f42b013adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:02.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.909+0000 7f42b5061640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f42b013e520 0x7f42b01ef4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:02.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.909+0000 7f42b5061640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f42b0141690 0x7f42b00abbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:02.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.909+0000 7f42b5061640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42b01e1660 con 0x7f42b0070920 2023-10-19T13:01:02.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.909+0000 7f42b5061640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42b01e1310 con 0x7f42b0141690 2023-10-19T13:01:02.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.909+0000 7f42b5061640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42b01f2810 con 0x7f42b013e520 2023-10-19T13:01:02.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.909+0000 7f42aed9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f42b0070920 0x7f42b013adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:02.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.909+0000 7f42aed9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f42b0070920 0x7f42b013adf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38180/0 (socket says 172.21.15.57:38180) 2023-10-19T13:01:02.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.909+0000 7f42aed9d640 1 -- 172.21.15.57:0/3426156063 learned_addr learned my addr 172.21.15.57:0/3426156063 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:01:02.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.909+0000 7f42af59e640 1 --2- 172.21.15.57:0/3426156063 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f42b0141690 0x7f42b00abbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:02.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.909+0000 7f42ae59c640 1 --2- 172.21.15.57:0/3426156063 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f42b013e520 0x7f42b01ef4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:02.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.909+0000 7f42aed9d640 1 -- 172.21.15.57:0/3426156063 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f42b013e520 msgr2=0x7f42b01ef4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:02.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.909+0000 7f42aed9d640 1 --2- 172.21.15.57:0/3426156063 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f42b013e520 0x7f42b01ef4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:02.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.909+0000 7f42aed9d640 1 -- 172.21.15.57:0/3426156063 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f42b0141690 msgr2=0x7f42b00abbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:02.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.909+0000 7f42aed9d640 1 --2- 172.21.15.57:0/3426156063 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f42b0141690 0x7f42b00abbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:02.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.909+0000 7f42aed9d640 1 -- 172.21.15.57:0/3426156063 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42a0009900 con 0x7f42b0070920 2023-10-19T13:01:02.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.909+0000 7f42ae59c640 1 --2- 172.21.15.57:0/3426156063 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f42b013e520 0x7f42b01ef4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:01:02.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.910+0000 7f42aed9d640 1 --2- 172.21.15.57:0/3426156063 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f42b0070920 0x7f42b013adf0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f42a800c430 tx=0x7f42a800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:02.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.910+0000 7f42a77fe640 1 -- 172.21.15.57:0/3426156063 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42a8011790 con 0x7f42b0070920 2023-10-19T13:01:02.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.910+0000 7f42a77fe640 1 -- 172.21.15.57:0/3426156063 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f42a80119e0 con 0x7f42b0070920 2023-10-19T13:01:02.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.910+0000 7f42a77fe640 1 -- 172.21.15.57:0/3426156063 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42a8011e30 con 0x7f42b0070920 2023-10-19T13:01:02.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.910+0000 7f42b5061640 1 -- 172.21.15.57:0/3426156063 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42b013b330 con 0x7f42b0070920 2023-10-19T13:01:02.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.910+0000 7f42b5061640 1 -- 172.21.15.57:0/3426156063 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42b01dff60 con 0x7f42b0070920 2023-10-19T13:01:02.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.911+0000 7f42b5061640 1 -- 172.21.15.57:0/3426156063 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42b00690e0 con 0x7f42b0070920 2023-10-19T13:01:02.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.911+0000 7f42a77fe640 1 -- 172.21.15.57:0/3426156063 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 40) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f42a801c460 con 0x7f42b0070920 2023-10-19T13:01:02.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.912+0000 7f42a77fe640 1 --2- 172.21.15.57:0/3426156063 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f429006af60 0x7f429006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:02.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.912+0000 7f42a77fe640 1 -- 172.21.15.57:0/3426156063 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f42a8089b10 con 0x7f42b0070920 2023-10-19T13:01:02.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.912+0000 7f42ae59c640 1 --2- 172.21.15.57:0/3426156063 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f429006af60 0x7f429006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:02.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.913+0000 7f42ae59c640 1 --2- 172.21.15.57:0/3426156063 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f429006af60 0x7f429006d420 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f42b013de30 tx=0x7f4298008040 comp rx=0 tx=0).ready entity=mgr.6679 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:02.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:02.916+0000 7f42a77fe640 1 -- 172.21.15.57:0/3426156063 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f42a80548d0 con 0x7f42b0070920 2023-10-19T13:01:03.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.141+0000 7f42b5061640 1 -- 172.21.15.57:0/3426156063 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f42b00703e0 con 0x7f42b0070920 2023-10-19T13:01:03.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.244+0000 7f42a77fe640 1 -- 172.21.15.57:0/3426156063 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v41) v1 ==== 92+0+0 (secure 0 0 0) 0x7f42a8057ef0 con 0x7f42b0070920 2023-10-19T13:01:03.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.247+0000 7f42b5061640 1 -- 172.21.15.57:0/3426156063 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f429006af60 msgr2=0x7f429006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:03.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.247+0000 7f42b5061640 1 --2- 172.21.15.57:0/3426156063 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f429006af60 0x7f429006d420 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f42b013de30 tx=0x7f4298008040 comp rx=0 tx=0).stop 2023-10-19T13:01:03.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.247+0000 7f42b5061640 1 -- 172.21.15.57:0/3426156063 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f42b0070920 msgr2=0x7f42b013adf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:03.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.247+0000 7f42b5061640 1 --2- 172.21.15.57:0/3426156063 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f42b0070920 0x7f42b013adf0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f42a800c430 tx=0x7f42a800c8f0 comp rx=0 tx=0).stop 2023-10-19T13:01:03.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.247+0000 7f42b5061640 1 -- 172.21.15.57:0/3426156063 shutdown_connections 2023-10-19T13:01:03.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.247+0000 7f42b5061640 1 --2- 172.21.15.57:0/3426156063 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f429006af60 0x7f429006d420 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:03.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.247+0000 7f42b5061640 1 --2- 172.21.15.57:0/3426156063 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f42b0141690 0x7f42b00abbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:03.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.247+0000 7f42b5061640 1 --2- 172.21.15.57:0/3426156063 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f42b013e520 0x7f42b01ef4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:03.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.247+0000 7f42b5061640 1 --2- 172.21.15.57:0/3426156063 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f42b0070920 0x7f42b013adf0 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:03.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.247+0000 7f42b5061640 1 -- 172.21.15.57:0/3426156063 >> 172.21.15.57:0/3426156063 conn(0x7f42b0069930 msgr2=0x7f42b0069d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:03.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.248+0000 7f42b5061640 1 -- 172.21.15.57:0/3426156063 shutdown_connections 2023-10-19T13:01:03.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.248+0000 7f42b5061640 1 -- 172.21.15.57:0/3426156063 wait complete. 2023-10-19T13:01:03.262 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-19T13:01:03.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.374+0000 7ff095f25640 1 -- 172.21.15.57:0/793838366 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff0900cfa20 msgr2=0x7ff0900cb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:03.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.374+0000 7ff095f25640 1 --2- 172.21.15.57:0/793838366 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff0900cfa20 0x7ff0900cb9e0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7ff084009830 tx=0x7ff08402bdf0 comp rx=0 tx=0).stop 2023-10-19T13:01:03.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.374+0000 7ff095f25640 1 -- 172.21.15.57:0/793838366 shutdown_connections 2023-10-19T13:01:03.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.374+0000 7ff095f25640 1 --2- 172.21.15.57:0/793838366 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff0900cfa20 0x7ff0900cb9e0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:03.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.374+0000 7ff095f25640 1 -- 172.21.15.57:0/793838366 >> 172.21.15.57:0/793838366 conn(0x7ff09009a030 msgr2=0x7ff0900d0d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:03.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.374+0000 7ff095f25640 1 -- 172.21.15.57:0/793838366 shutdown_connections 2023-10-19T13:01:03.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.374+0000 7ff095f25640 1 -- 172.21.15.57:0/793838366 wait complete. 2023-10-19T13:01:03.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.376+0000 7ff095f25640 1 Processor -- start 2023-10-19T13:01:03.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.376+0000 7ff095f25640 1 -- start start 2023-10-19T13:01:03.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.377+0000 7ff095f25640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff0900cfa20 0x7ff0900ca650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:03.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.377+0000 7ff095f25640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff09015b260 0x7ff0901522d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:03.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.377+0000 7ff095f25640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff09015b640 0x7ff09015ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:03.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.377+0000 7ff095f25640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0900b5a10 con 0x7ff09015b260 2023-10-19T13:01:03.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.377+0000 7ff095f25640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0900b4bc0 con 0x7ff09015b640 2023-10-19T13:01:03.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.377+0000 7ff095f25640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0900b4e40 con 0x7ff0900cfa20 2023-10-19T13:01:03.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.377+0000 7ff08effd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff09015b260 0x7ff0901522d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:03.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.377+0000 7ff08effd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff09015b260 0x7ff0901522d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38200/0 (socket says 172.21.15.57:38200) 2023-10-19T13:01:03.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.377+0000 7ff08effd640 1 -- 172.21.15.57:0/1098997977 learned_addr learned my addr 172.21.15.57:0/1098997977 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:01:03.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.377+0000 7ff08ffff640 1 --2- 172.21.15.57:0/1098997977 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff09015b640 0x7ff09015ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:03.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.377+0000 7ff08f7fe640 1 --2- 172.21.15.57:0/1098997977 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff0900cfa20 0x7ff0900ca650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:03.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.377+0000 7ff08effd640 1 -- 172.21.15.57:0/1098997977 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff0900cfa20 msgr2=0x7ff0900ca650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:03.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.377+0000 7ff08effd640 1 --2- 172.21.15.57:0/1098997977 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff0900cfa20 0x7ff0900ca650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:03.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.377+0000 7ff08effd640 1 -- 172.21.15.57:0/1098997977 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff09015b640 msgr2=0x7ff09015ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:03.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.377+0000 7ff08effd640 1 --2- 172.21.15.57:0/1098997977 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff09015b640 0x7ff09015ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:03.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.377+0000 7ff08effd640 1 -- 172.21.15.57:0/1098997977 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff084009490 con 0x7ff09015b260 2023-10-19T13:01:03.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.377+0000 7ff08ffff640 1 --2- 172.21.15.57:0/1098997977 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff09015b640 0x7ff09015ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T13:01:03.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.377+0000 7ff08effd640 1 --2- 172.21.15.57:0/1098997977 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff09015b260 0x7ff0901522d0 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7ff084009950 tx=0x7ff084003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:03.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.377+0000 7ff077fff640 1 -- 172.21.15.57:0/1098997977 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff084003e20 con 0x7ff09015b260 2023-10-19T13:01:03.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.377+0000 7ff077fff640 1 -- 172.21.15.57:0/1098997977 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7ff084034440 con 0x7ff09015b260 2023-10-19T13:01:03.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.377+0000 7ff095f25640 1 -- 172.21.15.57:0/1098997977 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0900b50e0 con 0x7ff09015b260 2023-10-19T13:01:03.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.377+0000 7ff077fff640 1 -- 172.21.15.57:0/1098997977 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff084034850 con 0x7ff09015b260 2023-10-19T13:01:03.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.378+0000 7ff095f25640 1 -- 172.21.15.57:0/1098997977 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0900b56a0 con 0x7ff09015b260 2023-10-19T13:01:03.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.379+0000 7ff095f25640 1 -- 172.21.15.57:0/1098997977 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0900675f0 con 0x7ff09015b260 2023-10-19T13:01:03.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.380+0000 7ff077fff640 1 -- 172.21.15.57:0/1098997977 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 41) v1 ==== 89348+0+0 (secure 0 0 0) 0x7ff0840349b0 con 0x7ff09015b260 2023-10-19T13:01:03.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.380+0000 7ff077fff640 1 --2- 172.21.15.57:0/1098997977 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7ff07c06afb0 0x7ff07c06d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:03.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.380+0000 7ff077fff640 1 -- 172.21.15.57:0/1098997977 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6057+0+0 (secure 0 0 0) 0x7ff0840abf70 con 0x7ff09015b260 2023-10-19T13:01:03.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.380+0000 7ff08f7fe640 1 -- 172.21.15.57:0/1098997977 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7ff07c06afb0 msgr2=0x7ff07c06d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/1413162726 2023-10-19T13:01:03.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.380+0000 7ff08f7fe640 1 --2- 172.21.15.57:0/1098997977 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7ff07c06afb0 0x7ff07c06d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T13:01:03.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.385+0000 7ff077fff640 1 -- 172.21.15.57:0/1098997977 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff0840deca0 con 0x7ff09015b260 2023-10-19T13:01:03.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.563+0000 7ff095f25640 1 -- 172.21.15.57:0/1098997977 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7ff0900af790 con 0x7ff09015b260 2023-10-19T13:01:03.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.571+0000 7ff077fff640 1 -- 172.21.15.57:0/1098997977 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v44)=0 v44) v1 ==== 138+0+0 (secure 0 0 0) 0x7ff0840390d0 con 0x7ff09015b260 2023-10-19T13:01:03.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.573+0000 7ff095f25640 1 -- 172.21.15.57:0/1098997977 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7ff07c06afb0 msgr2=0x7ff07c06d470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T13:01:03.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.573+0000 7ff095f25640 1 --2- 172.21.15.57:0/1098997977 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7ff07c06afb0 0x7ff07c06d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:03.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.573+0000 7ff095f25640 1 -- 172.21.15.57:0/1098997977 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff09015b260 msgr2=0x7ff0901522d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:03.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.573+0000 7ff095f25640 1 --2- 172.21.15.57:0/1098997977 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff09015b260 0x7ff0901522d0 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7ff084009950 tx=0x7ff084003820 comp rx=0 tx=0).stop 2023-10-19T13:01:03.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.573+0000 7ff095f25640 1 -- 172.21.15.57:0/1098997977 shutdown_connections 2023-10-19T13:01:03.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.573+0000 7ff095f25640 1 --2- 172.21.15.57:0/1098997977 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7ff07c06afb0 0x7ff07c06d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:03.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.573+0000 7ff095f25640 1 --2- 172.21.15.57:0/1098997977 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff09015b640 0x7ff09015ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:03.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.573+0000 7ff095f25640 1 --2- 172.21.15.57:0/1098997977 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff09015b260 0x7ff0901522d0 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:03.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.573+0000 7ff095f25640 1 --2- 172.21.15.57:0/1098997977 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff0900cfa20 0x7ff0900ca650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:03.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.573+0000 7ff095f25640 1 -- 172.21.15.57:0/1098997977 >> 172.21.15.57:0/1098997977 conn(0x7ff09009a030 msgr2=0x7ff090154380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:03.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.573+0000 7ff095f25640 1 -- 172.21.15.57:0/1098997977 shutdown_connections 2023-10-19T13:01:03.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.573+0000 7ff095f25640 1 -- 172.21.15.57:0/1098997977 wait complete. 2023-10-19T13:01:03.587 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-19T13:01:03.636 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:03.634+0000 7f896e20e040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-19T13:01:03.651 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:01:03.648+0000 7f990941d040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-19T13:01:03.681 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:03.678+0000 7f896e20e040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-19T13:01:03.699 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:01:03.697+0000 7f990941d040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-19T13:01:03.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.699+0000 7f3ba7c07640 1 -- 172.21.15.57:0/1201014457 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3b8c004360 msgr2=0x7f3b8c004840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:03.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.699+0000 7f3ba7c07640 1 --2- 172.21.15.57:0/1201014457 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3b8c004360 0x7f3b8c004840 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f3b9c00c260 tx=0x7f3b8c0057f0 comp rx=0 tx=0).stop 2023-10-19T13:01:03.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.699+0000 7f3ba61a4640 1 -- 172.21.15.57:0/1201014457 reap_dead start 2023-10-19T13:01:03.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.699+0000 7f3ba7c07640 1 -- 172.21.15.57:0/1201014457 shutdown_connections 2023-10-19T13:01:03.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.699+0000 7f3ba7c07640 1 -- 172.21.15.57:0/1201014457 >> 172.21.15.57:0/1201014457 conn(0x7f3ba00d0d60 msgr2=0x7f3ba00d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:03.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.699+0000 7f3ba7c07640 1 -- 172.21.15.57:0/1201014457 shutdown_connections 2023-10-19T13:01:03.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.699+0000 7f3ba7c07640 1 -- 172.21.15.57:0/1201014457 wait complete. 2023-10-19T13:01:03.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.700+0000 7f3ba7c07640 1 Processor -- start 2023-10-19T13:01:03.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.700+0000 7f3ba7c07640 1 -- start start 2023-10-19T13:01:03.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.701+0000 7f3ba7c07640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3b8c004360 0x7f3ba00c98d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:03.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.701+0000 7f3ba7c07640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3ba00c8e40 0x7f3ba00ca4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:03.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.701+0000 7f3ba7c07640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3ba00caa20 0x7f3ba0158990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:03.707 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.701+0000 7f3ba7c07640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ba00c7e30 con 0x7f3ba00caa20 2023-10-19T13:01:03.707 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.701+0000 7f3ba7c07640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ba00c8080 con 0x7f3ba00c8e40 2023-10-19T13:01:03.707 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.701+0000 7f3ba7c07640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ba00c8360 con 0x7f3b8c004360 2023-10-19T13:01:03.707 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.701+0000 7f3ba61a4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3ba00caa20 0x7f3ba0158990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:03.707 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.701+0000 7f3ba61a4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3ba00caa20 0x7f3ba0158990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38208/0 (socket says 172.21.15.57:38208) 2023-10-19T13:01:03.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.701+0000 7f3ba61a4640 1 -- 172.21.15.57:0/3194941820 learned_addr learned my addr 172.21.15.57:0/3194941820 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:01:03.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.701+0000 7f3ba59a3640 1 --2- 172.21.15.57:0/3194941820 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3b8c004360 0x7f3ba00c98d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:03.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.701+0000 7f3ba51a2640 1 --2- 172.21.15.57:0/3194941820 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3ba00c8e40 0x7f3ba00ca4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:03.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.702+0000 7f3ba61a4640 1 -- 172.21.15.57:0/3194941820 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3b8c004360 msgr2=0x7f3ba00c98d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:03.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.702+0000 7f3ba61a4640 1 --2- 172.21.15.57:0/3194941820 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3b8c004360 0x7f3ba00c98d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:03.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.702+0000 7f3ba61a4640 1 -- 172.21.15.57:0/3194941820 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3ba00c8e40 msgr2=0x7f3ba00ca4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:03.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.702+0000 7f3ba61a4640 1 --2- 172.21.15.57:0/3194941820 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3ba00c8e40 0x7f3ba00ca4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:03.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.702+0000 7f3ba61a4640 1 -- 172.21.15.57:0/3194941820 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ba00c8640 con 0x7f3ba00caa20 2023-10-19T13:01:03.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.702+0000 7f3ba61a4640 1 --2- 172.21.15.57:0/3194941820 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3ba00caa20 0x7f3ba0158990 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f3ba00cef20 tx=0x7f3b9800f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:03.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.702+0000 7f3b927fc640 1 -- 172.21.15.57:0/3194941820 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b98003d80 con 0x7f3ba00caa20 2023-10-19T13:01:03.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.702+0000 7f3b927fc640 1 -- 172.21.15.57:0/3194941820 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f3b9800feb0 con 0x7f3ba00caa20 2023-10-19T13:01:03.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.702+0000 7f3ba7c07640 1 -- 172.21.15.57:0/3194941820 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ba00c8920 con 0x7f3ba00caa20 2023-10-19T13:01:03.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.702+0000 7f3b927fc640 1 -- 172.21.15.57:0/3194941820 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b98004d80 con 0x7f3ba00caa20 2023-10-19T13:01:03.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.702+0000 7f3ba7c07640 1 -- 172.21.15.57:0/3194941820 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ba00c9ff0 con 0x7f3ba00caa20 2023-10-19T13:01:03.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.703+0000 7f3ba7c07640 1 -- 172.21.15.57:0/3194941820 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ba00690e0 con 0x7f3ba00caa20 2023-10-19T13:01:03.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.704+0000 7f3b927fc640 1 -- 172.21.15.57:0/3194941820 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 41) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f3b98004030 con 0x7f3ba00caa20 2023-10-19T13:01:03.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.704+0000 7f3b927fc640 1 --2- 172.21.15.57:0/3194941820 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f3b8406afb0 0x7f3b8406d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:03.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.704+0000 7f3b927fc640 1 -- 172.21.15.57:0/3194941820 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f3b9808a5d0 con 0x7f3ba00caa20 2023-10-19T13:01:03.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.704+0000 7f3ba59a3640 1 -- 172.21.15.57:0/3194941820 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f3b8406afb0 msgr2=0x7f3b8406d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/1413162726 2023-10-19T13:01:03.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.704+0000 7f3ba59a3640 1 --2- 172.21.15.57:0/3194941820 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f3b8406afb0 0x7f3b8406d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T13:01:03.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.708+0000 7f3b927fc640 1 -- 172.21.15.57:0/3194941820 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3b98055390 con 0x7f3ba00caa20 2023-10-19T13:01:03.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.886+0000 7f3ba7c07640 1 -- 172.21.15.57:0/3194941820 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f3ba00cd8f0 con 0x7f3ba00caa20 2023-10-19T13:01:03.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.893+0000 7f3b927fc640 1 -- 172.21.15.57:0/3194941820 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v45)=0 v45) v1 ==== 133+0+0 (secure 0 0 0) 0x7f3b980589b0 con 0x7f3ba00caa20 2023-10-19T13:01:03.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.895+0000 7f3ba7c07640 1 -- 172.21.15.57:0/3194941820 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f3b8406afb0 msgr2=0x7f3b8406d470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T13:01:03.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.895+0000 7f3ba7c07640 1 --2- 172.21.15.57:0/3194941820 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f3b8406afb0 0x7f3b8406d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:03.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.895+0000 7f3ba7c07640 1 -- 172.21.15.57:0/3194941820 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3ba00caa20 msgr2=0x7f3ba0158990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:03.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.895+0000 7f3ba7c07640 1 --2- 172.21.15.57:0/3194941820 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3ba00caa20 0x7f3ba0158990 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f3ba00cef20 tx=0x7f3b9800f790 comp rx=0 tx=0).stop 2023-10-19T13:01:03.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.895+0000 7f3ba7c07640 1 -- 172.21.15.57:0/3194941820 shutdown_connections 2023-10-19T13:01:03.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.895+0000 7f3ba7c07640 1 --2- 172.21.15.57:0/3194941820 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f3b8406afb0 0x7f3b8406d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:03.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.895+0000 7f3ba7c07640 1 --2- 172.21.15.57:0/3194941820 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3ba00caa20 0x7f3ba0158990 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:03.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.895+0000 7f3ba7c07640 1 --2- 172.21.15.57:0/3194941820 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3ba00c8e40 0x7f3ba00ca4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:03.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.895+0000 7f3ba7c07640 1 --2- 172.21.15.57:0/3194941820 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3b8c004360 0x7f3ba00c98d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:03.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.895+0000 7f3ba7c07640 1 -- 172.21.15.57:0/3194941820 >> 172.21.15.57:0/3194941820 conn(0x7f3ba00d0d60 msgr2=0x7f3ba015a5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:03.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.895+0000 7f3ba7c07640 1 -- 172.21.15.57:0/3194941820 shutdown_connections 2023-10-19T13:01:03.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:03.895+0000 7f3ba7c07640 1 -- 172.21.15.57:0/3194941820 wait complete. 2023-10-19T13:01:03.909 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list snap_test_dir1 2023-10-19T13:01:04.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:04.019+0000 7f2588ee5640 1 -- 172.21.15.57:0/1388423780 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f25840b1200 msgr2=0x7f25840cd120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:04.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:04.019+0000 7f2588ee5640 1 --2- 172.21.15.57:0/1388423780 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f25840b1200 0x7f25840cd120 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f257c009be0 tx=0x7f257c02c030 comp rx=0 tx=0).stop 2023-10-19T13:01:04.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:04.019+0000 7f2588ee5640 1 -- 172.21.15.57:0/1388423780 shutdown_connections 2023-10-19T13:01:04.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:04.019+0000 7f2588ee5640 1 --2- 172.21.15.57:0/1388423780 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f25840b1200 0x7f25840cd120 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:04.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:04.019+0000 7f2588ee5640 1 -- 172.21.15.57:0/1388423780 >> 172.21.15.57:0/1388423780 conn(0x7f25840b06e0 msgr2=0x7f25840b0af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:04.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:04.019+0000 7f2588ee5640 1 -- 172.21.15.57:0/1388423780 shutdown_connections 2023-10-19T13:01:04.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:04.019+0000 7f2588ee5640 1 -- 172.21.15.57:0/1388423780 wait complete. 2023-10-19T13:01:04.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:04.020+0000 7f2588ee5640 1 Processor -- start 2023-10-19T13:01:04.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:04.021+0000 7f2588ee5640 1 -- start start 2023-10-19T13:01:04.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:04.021+0000 7f2588ee5640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f258406d890 0x7f2584155bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:04.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:04.021+0000 7f2588ee5640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f25840b1200 0x7f25840c6f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:04.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:04.021+0000 7f2588ee5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2584159150 0x7f25840c7820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:04.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:04.021+0000 7f2588ee5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25840b4220 con 0x7f2584159150 2023-10-19T13:01:04.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:04.021+0000 7f2588ee5640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25840b4750 con 0x7f25840b1200 2023-10-19T13:01:04.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:04.021+0000 7f2588ee5640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25840b4470 con 0x7f258406d890 2023-10-19T13:01:04.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:04.021+0000 7f2582d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2584159150 0x7f25840c7820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:04.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:04.021+0000 7f2582d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2584159150 0x7f25840c7820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38222/0 (socket says 172.21.15.57:38222) 2023-10-19T13:01:04.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:04.021+0000 7f2582d9d640 1 -- 172.21.15.57:0/780245166 learned_addr learned my addr 172.21.15.57:0/780245166 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:01:04.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:04.021+0000 7f258259c640 1 --2- 172.21.15.57:0/780245166 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f258406d890 0x7f2584155bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:04.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:04.021+0000 7f2581d9b640 1 --2- 172.21.15.57:0/780245166 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f25840b1200 0x7f25840c6f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:04.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:04.022+0000 7f2582d9d640 1 -- 172.21.15.57:0/780245166 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f258406d890 msgr2=0x7f2584155bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:04.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:04.022+0000 7f2582d9d640 1 --2- 172.21.15.57:0/780245166 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f258406d890 0x7f2584155bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:04.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:04.022+0000 7f2582d9d640 1 -- 172.21.15.57:0/780245166 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f25840b1200 msgr2=0x7f25840c6f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:04.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:04.022+0000 7f2582d9d640 1 --2- 172.21.15.57:0/780245166 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f25840b1200 0x7f25840c6f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:04.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:04.022+0000 7f2582d9d640 1 -- 172.21.15.57:0/780245166 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f257c009840 con 0x7f2584159150 2023-10-19T13:01:04.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:04.022+0000 7f2582d9d640 1 --2- 172.21.15.57:0/780245166 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2584159150 0x7f25840c7820 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f257c036040 tx=0x7f257c0346e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:04.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:04.022+0000 7f2572ffd640 1 -- 172.21.15.57:0/780245166 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f257c034c80 con 0x7f2584159150 2023-10-19T13:01:04.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:04.022+0000 7f2588ee5640 1 -- 172.21.15.57:0/780245166 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25840c6720 con 0x7f2584159150 2023-10-19T13:01:04.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:04.022+0000 7f2572ffd640 1 -- 172.21.15.57:0/780245166 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f257c004040 con 0x7f2584159150 2023-10-19T13:01:04.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:04.022+0000 7f2572ffd640 1 -- 172.21.15.57:0/780245166 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f257c004450 con 0x7f2584159150 2023-10-19T13:01:04.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:04.022+0000 7f2588ee5640 1 -- 172.21.15.57:0/780245166 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25840c7460 con 0x7f2584159150 2023-10-19T13:01:04.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:04.023+0000 7f2588ee5640 1 -- 172.21.15.57:0/780245166 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25840c6c40 con 0x7f2584159150 2023-10-19T13:01:04.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:04.024+0000 7f2572ffd640 1 -- 172.21.15.57:0/780245166 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 41) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f257c0045b0 con 0x7f2584159150 2023-10-19T13:01:04.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:04.024+0000 7f2572ffd640 1 --2- 172.21.15.57:0/780245166 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f256406afb0 0x7f256406d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:04.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:04.024+0000 7f258259c640 1 -- 172.21.15.57:0/780245166 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f256406afb0 msgr2=0x7f256406d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/1413162726 2023-10-19T13:01:04.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:04.024+0000 7f2572ffd640 1 -- 172.21.15.57:0/780245166 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f257c0ac760 con 0x7f2584159150 2023-10-19T13:01:04.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:04.024+0000 7f258259c640 1 --2- 172.21.15.57:0/780245166 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f256406afb0 0x7f256406d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T13:01:04.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:04.028+0000 7f2572ffd640 1 -- 172.21.15.57:0/780245166 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f25840c6c40 con 0x7f2584159150 2023-10-19T13:01:04.073 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:04.071+0000 7f896e20e040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-19T13:01:04.116 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:01:04.114+0000 7f990941d040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-19T13:01:04.117 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:04.114+0000 7f896e20e040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-19T13:01:04.159 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:04.157+0000 7f896e20e040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-19T13:01:04.162 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:01:04.160+0000 7f990941d040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-19T13:01:04.207 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:01:04.205+0000 7f990941d040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-19T13:01:04.211 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:04.208+0000 7f896e20e040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-19T13:01:04.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:04.225+0000 7f258259c640 1 -- 172.21.15.57:0/780245166 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f256406afb0 msgr2=0x7f256406d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/1413162726 2023-10-19T13:01:04.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:04.225+0000 7f258259c640 1 --2- 172.21.15.57:0/780245166 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f256406afb0 0x7f256406d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-19T13:01:04.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:04.229+0000 7f2588ee5640 1 -- 172.21.15.57:0/780245166 --> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f25840c6c40 con 0x7f256406afb0 2023-10-19T13:01:04.255 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:04.252+0000 7f896e20e040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-19T13:01:04.264 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:01:04.262+0000 7f990941d040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-19T13:01:04.308 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:01:04.305+0000 7f990941d040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-19T13:01:04.356 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:04.353+0000 7f896e20e040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-19T13:01:04.409 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:04.406+0000 7f896e20e040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-19T13:01:04.410 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:01:04.408+0000 7f990941d040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-19T13:01:04.450 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:04.448+0000 7f896e20e040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-19T13:01:04.466 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:01:04.463+0000 7f990941d040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-19T13:01:04.498 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:04.496+0000 7f896e20e040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-19T13:01:04.509 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:01:04.507+0000 7f990941d040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-19T13:01:04.541 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:04.538+0000 7f896e20e040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-19T13:01:04.561 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:01:04.557+0000 7f990941d040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-19T13:01:04.604 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:01:04.601+0000 7f990941d040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-19T13:01:04.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:04.626+0000 7f258259c640 1 -- 172.21.15.57:0/780245166 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f256406afb0 msgr2=0x7f256406d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/1413162726 2023-10-19T13:01:04.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:04.626+0000 7f258259c640 1 --2- 172.21.15.57:0/780245166 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f256406afb0 0x7f256406d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-19T13:01:04.853 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:04.851+0000 7f896e20e040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-19T13:01:04.936 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:01:04.933+0000 7f990941d040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-19T13:01:04.982 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:04.979+0000 7f896e20e040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-19T13:01:05.073 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:05.070+0000 7f896e20e040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-19T13:01:05.074 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:01:05.071+0000 7f990941d040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-19T13:01:05.115 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:05.112+0000 7f896e20e040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-19T13:01:05.170 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:01:05.167+0000 7f990941d040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-19T13:01:05.208 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:05.205+0000 7f896e20e040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-19T13:01:05.215 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:01:05.212+0000 7f990941d040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-19T13:01:05.298 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:05.295+0000 7f896e20e040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-19T13:01:05.314 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:01:05.311+0000 7f990941d040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-19T13:01:05.409 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:01:05.406+0000 7f990941d040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-19T13:01:05.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:05.427+0000 7f258259c640 1 -- 172.21.15.57:0/780245166 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f256406afb0 msgr2=0x7f256406d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/1413162726 2023-10-19T13:01:05.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:05.427+0000 7f258259c640 1 --2- 172.21.15.57:0/780245166 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f256406afb0 0x7f256406d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-19T13:01:05.432 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:05.429+0000 7f896e20e040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-19T13:01:05.473 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:05.470+0000 7f896e20e040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-19T13:01:05.552 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:01:05.549+0000 7f990941d040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-19T13:01:05.576 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:05.573+0000 7f896e20e040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-19T13:01:05.596 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:01:05.593+0000 7f990941d040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-19T13:01:05.705 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:01:05.702+0000 7f990941d040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-19T13:01:06.182 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:06.180+0000 7f896e20e040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-19T13:01:06.347 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:01:06.344+0000 7f990941d040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-19T13:01:06.443 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:06.440+0000 7f896e20e040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-19T13:01:06.491 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:06.488+0000 7f896e20e040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-19T13:01:06.591 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:06.588+0000 7f896e20e040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-19T13:01:06.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:06.613+0000 7f2572ffd640 1 -- 172.21.15.57:0/780245166 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mgrmap(e 42) v1 ==== 88957+0+0 (secure 0 0 0) 0x7f257c077a30 con 0x7f2584159150 2023-10-19T13:01:06.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:06.613+0000 7f2572ffd640 1 -- 172.21.15.57:0/780245166 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f256406afb0 msgr2=0x7f256406d470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T13:01:06.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:06.613+0000 7f2572ffd640 1 --2- 172.21.15.57:0/780245166 >> [v2:172.21.15.179:6832/1413162726,v1:172.21.15.179:6833/1413162726] conn(0x7f256406afb0 0x7f256406d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:06.624 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:01:06.622+0000 7f990941d040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-19T13:01:06.681 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:01:06.678+0000 7f990941d040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-19T13:01:06.718 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:06.715+0000 7f88ea880640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:01:06.719 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:06.715+0000 7f88ea880640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:01:06.719 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:06.715+0000 7f88ea880640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:01:06.719 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:06.715+0000 7f88ea880640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:01:06.719 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:06.715+0000 7f88ea880640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:01:06.719 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:06.715+0000 7f88ed8c6640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:01:06.719 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:06.715+0000 7f88ed8c6640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:01:06.720 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:06.715+0000 7f88ed8c6640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:01:06.720 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:06.715+0000 7f88ed8c6640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:01:06.720 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:06.715+0000 7f88ed8c6640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:01:06.792 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:01:06.790+0000 7f990941d040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-19T13:01:07.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:07.617+0000 7f2572ffd640 1 -- 172.21.15.57:0/780245166 <== mon.0 v2:172.21.15.57:3300/0 8 ==== mgrmap(e 43) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f257c0811a0 con 0x7f2584159150 2023-10-19T13:01:07.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:07.618+0000 7f2572ffd640 1 --2- 172.21.15.57:0/780245166 >> [v2:172.21.15.179:6832/124009847,v1:172.21.15.179:6833/124009847] conn(0x7f2564035850 0x7f25640753e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:07.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:07.618+0000 7f2572ffd640 1 -- 172.21.15.57:0/780245166 --> [v2:172.21.15.179:6832/124009847,v1:172.21.15.179:6833/124009847] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f25840c6c40 con 0x7f2564035850 2023-10-19T13:01:07.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:07.619+0000 7f258259c640 1 --2- 172.21.15.57:0/780245166 >> [v2:172.21.15.179:6832/124009847,v1:172.21.15.179:6833/124009847] conn(0x7f2564035850 0x7f25640753e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:07.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:07.620+0000 7f258259c640 1 --2- 172.21.15.57:0/780245166 >> [v2:172.21.15.179:6832/124009847,v1:172.21.15.179:6833/124009847] conn(0x7f2564035850 0x7f25640753e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f257400b010 tx=0x7f2574007d10 comp rx=0 tx=0).ready entity=mgr.7003 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:07.631 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:07.629+0000 7f88f48d4640 -1 mgr.server reply reply (2) No such file or directory SnapSchedule for snap_test_dir1 not found 2023-10-19T13:01:07.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:07.629+0000 7f2572ffd640 1 -- 172.21.15.57:0/780245166 <== mgr.7003 v2:172.21.15.179:6832/124009847 1 ==== mgr_command_reply(tid 0: -2 SnapSchedule for snap_test_dir1 not found) v1 ==== 49+0+0 (secure 0 0 0) 0x7f25840c6c40 con 0x7f2564035850 2023-10-19T13:01:07.632 INFO:teuthology.orchestra.run.smithi057.stderr:Error ENOENT: SnapSchedule for snap_test_dir1 not found 2023-10-19T13:01:07.638 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-10-19T13:01:07.639 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-19T13:01:07.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:07.755+0000 7f115f0e5640 1 -- 172.21.15.57:0/2140723498 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1144003f20 msgr2=0x7f11440047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:07.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:07.755+0000 7f115f0e5640 1 --2- 172.21.15.57:0/2140723498 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1144003f20 0x7f11440047c0 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f1148006fd0 tx=0x7f11440057f0 comp rx=0 tx=0).stop 2023-10-19T13:01:07.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:07.755+0000 7f115d682640 1 -- 172.21.15.57:0/2140723498 reap_dead start 2023-10-19T13:01:07.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:07.755+0000 7f115f0e5640 1 -- 172.21.15.57:0/2140723498 shutdown_connections 2023-10-19T13:01:07.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:07.755+0000 7f115f0e5640 1 -- 172.21.15.57:0/2140723498 >> 172.21.15.57:0/2140723498 conn(0x7f1158069930 msgr2=0x7f1158069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:07.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:07.755+0000 7f115f0e5640 1 -- 172.21.15.57:0/2140723498 shutdown_connections 2023-10-19T13:01:07.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:07.755+0000 7f115f0e5640 1 -- 172.21.15.57:0/2140723498 wait complete. 2023-10-19T13:01:07.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:07.756+0000 7f115f0e5640 1 Processor -- start 2023-10-19T13:01:07.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:07.756+0000 7f115f0e5640 1 -- start start 2023-10-19T13:01:07.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:07.757+0000 7f115f0e5640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1144003f20 0x7f1158141380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:07.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:07.757+0000 7f115f0e5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f115806b040 0x7f11580ad3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:07.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:07.757+0000 7f115f0e5640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f115806b4c0 0x7f11580ad930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:07.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:07.757+0000 7f115f0e5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11581418c0 con 0x7f115806b040 2023-10-19T13:01:07.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:07.757+0000 7f115f0e5640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1158153fa0 con 0x7f115806b4c0 2023-10-19T13:01:07.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:07.757+0000 7f115f0e5640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1158154280 con 0x7f1144003f20 2023-10-19T13:01:07.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:07.757+0000 7f1157fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f115806b040 0x7f11580ad3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:07.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:07.757+0000 7f1157fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f115806b040 0x7f11580ad3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56012/0 (socket says 172.21.15.57:56012) 2023-10-19T13:01:07.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:07.757+0000 7f1157fff640 1 -- 172.21.15.57:0/4086954818 learned_addr learned my addr 172.21.15.57:0/4086954818 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:01:07.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:07.757+0000 7f115d682640 1 --2- 172.21.15.57:0/4086954818 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f115806b4c0 0x7f11580ad930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:07.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:07.757+0000 7f115ce81640 1 --2- 172.21.15.57:0/4086954818 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1144003f20 0x7f1158141380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:07.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:07.757+0000 7f1157fff640 1 -- 172.21.15.57:0/4086954818 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1144003f20 msgr2=0x7f1158141380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:07.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:07.757+0000 7f1157fff640 1 --2- 172.21.15.57:0/4086954818 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1144003f20 0x7f1158141380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:07.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:07.757+0000 7f1157fff640 1 -- 172.21.15.57:0/4086954818 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f115806b4c0 msgr2=0x7f11580ad930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:07.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:07.757+0000 7f1157fff640 1 --2- 172.21.15.57:0/4086954818 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f115806b4c0 0x7f11580ad930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:07.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:07.757+0000 7f1157fff640 1 -- 172.21.15.57:0/4086954818 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1158070300 con 0x7f115806b040 2023-10-19T13:01:07.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:07.757+0000 7f1157fff640 1 --2- 172.21.15.57:0/4086954818 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f115806b040 0x7f11580ad3f0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f114802c900 tx=0x7f1148001560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:07.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:07.758+0000 7f11557fa640 1 -- 172.21.15.57:0/4086954818 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1148006230 con 0x7f115806b040 2023-10-19T13:01:07.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:07.758+0000 7f11557fa640 1 -- 172.21.15.57:0/4086954818 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f1148006390 con 0x7f115806b040 2023-10-19T13:01:07.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:07.758+0000 7f115f0e5640 1 -- 172.21.15.57:0/4086954818 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1158154560 con 0x7f115806b040 2023-10-19T13:01:07.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:07.758+0000 7f11557fa640 1 -- 172.21.15.57:0/4086954818 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1148001750 con 0x7f115806b040 2023-10-19T13:01:07.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:07.758+0000 7f115f0e5640 1 -- 172.21.15.57:0/4086954818 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1158154d60 con 0x7f115806b040 2023-10-19T13:01:07.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:07.759+0000 7f115f0e5640 1 -- 172.21.15.57:0/4086954818 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11580690e0 con 0x7f115806b040 2023-10-19T13:01:07.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:07.759+0000 7f11557fa640 1 -- 172.21.15.57:0/4086954818 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 43) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f1148031030 con 0x7f115806b040 2023-10-19T13:01:07.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:07.759+0000 7f11557fa640 1 --2- 172.21.15.57:0/4086954818 >> [v2:172.21.15.179:6832/124009847,v1:172.21.15.179:6833/124009847] conn(0x7f113806ad10 0x7f113806d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:07.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:07.760+0000 7f11557fa640 1 -- 172.21.15.57:0/4086954818 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f11480aa2d0 con 0x7f115806b040 2023-10-19T13:01:07.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:07.760+0000 7f115ce81640 1 --2- 172.21.15.57:0/4086954818 >> [v2:172.21.15.179:6832/124009847,v1:172.21.15.179:6833/124009847] conn(0x7f113806ad10 0x7f113806d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:07.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:07.760+0000 7f115ce81640 1 --2- 172.21.15.57:0/4086954818 >> [v2:172.21.15.179:6832/124009847,v1:172.21.15.179:6833/124009847] conn(0x7f113806ad10 0x7f113806d1d0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f1150006180 tx=0x7f1150006eb0 comp rx=0 tx=0).ready entity=mgr.7003 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:07.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:07.764+0000 7f11557fa640 1 -- 172.21.15.57:0/4086954818 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f1148078ec0 con 0x7f115806b040 2023-10-19T13:01:07.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:07.988+0000 7f115f0e5640 1 -- 172.21.15.57:0/4086954818 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f11580cb830 con 0x7f115806b040 2023-10-19T13:01:08.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:08.664+0000 7f11557fa640 1 -- 172.21.15.57:0/4086954818 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v44) v1 ==== 93+0+0 (secure 0 0 0) 0x7f1148078a50 con 0x7f115806b040 2023-10-19T13:01:08.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:08.667+0000 7f115f0e5640 1 -- 172.21.15.57:0/4086954818 >> [v2:172.21.15.179:6832/124009847,v1:172.21.15.179:6833/124009847] conn(0x7f113806ad10 msgr2=0x7f113806d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:08.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:08.667+0000 7f115f0e5640 1 --2- 172.21.15.57:0/4086954818 >> [v2:172.21.15.179:6832/124009847,v1:172.21.15.179:6833/124009847] conn(0x7f113806ad10 0x7f113806d1d0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f1150006180 tx=0x7f1150006eb0 comp rx=0 tx=0).stop 2023-10-19T13:01:08.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:08.667+0000 7f115f0e5640 1 -- 172.21.15.57:0/4086954818 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f115806b040 msgr2=0x7f11580ad3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:08.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:08.667+0000 7f115f0e5640 1 --2- 172.21.15.57:0/4086954818 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f115806b040 0x7f11580ad3f0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f114802c900 tx=0x7f1148001560 comp rx=0 tx=0).stop 2023-10-19T13:01:08.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:08.668+0000 7f115f0e5640 1 -- 172.21.15.57:0/4086954818 shutdown_connections 2023-10-19T13:01:08.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:08.668+0000 7f115f0e5640 1 --2- 172.21.15.57:0/4086954818 >> [v2:172.21.15.179:6832/124009847,v1:172.21.15.179:6833/124009847] conn(0x7f113806ad10 0x7f113806d1d0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:08.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:08.668+0000 7f115f0e5640 1 --2- 172.21.15.57:0/4086954818 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f115806b4c0 0x7f11580ad930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:08.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:08.668+0000 7f115f0e5640 1 --2- 172.21.15.57:0/4086954818 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f115806b040 0x7f11580ad3f0 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:08.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:08.668+0000 7f115f0e5640 1 --2- 172.21.15.57:0/4086954818 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1144003f20 0x7f1158141380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:08.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:08.668+0000 7f115f0e5640 1 -- 172.21.15.57:0/4086954818 >> 172.21.15.57:0/4086954818 conn(0x7f1158069930 msgr2=0x7f11580b0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:08.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:08.668+0000 7f115f0e5640 1 -- 172.21.15.57:0/4086954818 shutdown_connections 2023-10-19T13:01:08.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:08.668+0000 7f115f0e5640 1 -- 172.21.15.57:0/4086954818 wait complete. 2023-10-19T13:01:08.685 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-19T13:01:08.686 DEBUG:teuthology.orchestra.run.smithi057:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-19T13:01:08.740 DEBUG:teuthology.orchestra.run.smithi179:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-19T13:01:08.780 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-19T13:01:08.781 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T13:01:08.781 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T13:01:08.798 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T13:01:08.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:08.946+0000 7f9d3404f640 1 -- 172.21.15.57:0/3886268495 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9d2c0c9b60 msgr2=0x7f9d2c155200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:08.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:08.946+0000 7f9d3404f640 1 --2- 172.21.15.57:0/3886268495 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9d2c0c9b60 0x7f9d2c155200 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f9d24005fd0 tx=0x7f9d2402c170 comp rx=0 tx=0).stop 2023-10-19T13:01:08.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:08.946+0000 7f9d3404f640 1 -- 172.21.15.57:0/3886268495 shutdown_connections 2023-10-19T13:01:08.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:08.946+0000 7f9d3404f640 1 --2- 172.21.15.57:0/3886268495 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9d2c0c9b60 0x7f9d2c155200 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:08.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:08.946+0000 7f9d3404f640 1 -- 172.21.15.57:0/3886268495 >> 172.21.15.57:0/3886268495 conn(0x7f9d2c069930 msgr2=0x7f9d2c069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:08.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:08.946+0000 7f9d3404f640 1 -- 172.21.15.57:0/3886268495 shutdown_connections 2023-10-19T13:01:08.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:08.946+0000 7f9d3404f640 1 -- 172.21.15.57:0/3886268495 wait complete. 2023-10-19T13:01:08.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:08.948+0000 7f9d3404f640 1 Processor -- start 2023-10-19T13:01:08.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:08.948+0000 7f9d3404f640 1 -- start start 2023-10-19T13:01:08.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:08.948+0000 7f9d3404f640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9d2c0b1920 0x7f9d2c1582f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:08.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:08.948+0000 7f9d3404f640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9d2c0c96b0 0x7f9d2c153550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:08.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:08.948+0000 7f9d3404f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9d2c0c9b60 0x7f9d2c0aeaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:08.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:08.948+0000 7f9d3404f640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d2c157250 con 0x7f9d2c0c9b60 2023-10-19T13:01:08.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:08.948+0000 7f9d3404f640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d2c157530 con 0x7f9d2c0b1920 2023-10-19T13:01:08.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:08.948+0000 7f9d3404f640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d2c157810 con 0x7f9d2c0c96b0 2023-10-19T13:01:08.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:08.949+0000 7f9d325ec640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9d2c0c9b60 0x7f9d2c0aeaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:08.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:08.949+0000 7f9d325ec640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9d2c0c9b60 0x7f9d2c0aeaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56046/0 (socket says 172.21.15.57:56046) 2023-10-19T13:01:08.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:08.949+0000 7f9d325ec640 1 -- 172.21.15.57:0/2282042753 learned_addr learned my addr 172.21.15.57:0/2282042753 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:01:08.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:08.949+0000 7f9d31deb640 1 --2- 172.21.15.57:0/2282042753 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9d2c0b1920 0x7f9d2c1582f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:08.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:08.949+0000 7f9d315ea640 1 --2- 172.21.15.57:0/2282042753 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9d2c0c96b0 0x7f9d2c153550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:08.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:08.949+0000 7f9d325ec640 1 -- 172.21.15.57:0/2282042753 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9d2c0c96b0 msgr2=0x7f9d2c153550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:08.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:08.949+0000 7f9d325ec640 1 --2- 172.21.15.57:0/2282042753 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9d2c0c96b0 0x7f9d2c153550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:08.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:08.949+0000 7f9d325ec640 1 -- 172.21.15.57:0/2282042753 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9d2c0b1920 msgr2=0x7f9d2c1582f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:08.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:08.949+0000 7f9d325ec640 1 --2- 172.21.15.57:0/2282042753 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9d2c0b1920 0x7f9d2c1582f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:08.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:08.949+0000 7f9d325ec640 1 -- 172.21.15.57:0/2282042753 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d24009840 con 0x7f9d2c0c9b60 2023-10-19T13:01:08.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:08.949+0000 7f9d31deb640 1 --2- 172.21.15.57:0/2282042753 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9d2c0b1920 0x7f9d2c1582f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T13:01:08.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:08.949+0000 7f9d325ec640 1 --2- 172.21.15.57:0/2282042753 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9d2c0c9b60 0x7f9d2c0aeaa0 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f9d24005fd0 tx=0x7f9d240042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:08.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:08.949+0000 7f9d227fc640 1 -- 172.21.15.57:0/2282042753 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d24003e80 con 0x7f9d2c0c9b60 2023-10-19T13:01:08.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:08.949+0000 7f9d227fc640 1 -- 172.21.15.57:0/2282042753 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f9d240047e0 con 0x7f9d2c0c9b60 2023-10-19T13:01:08.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:08.949+0000 7f9d227fc640 1 -- 172.21.15.57:0/2282042753 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d24034800 con 0x7f9d2c0c9b60 2023-10-19T13:01:08.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:08.950+0000 7f9d3404f640 1 -- 172.21.15.57:0/2282042753 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d2c070300 con 0x7f9d2c0c9b60 2023-10-19T13:01:08.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:08.950+0000 7f9d3404f640 1 -- 172.21.15.57:0/2282042753 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d2c157dd0 con 0x7f9d2c0c9b60 2023-10-19T13:01:08.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:08.951+0000 7f9d3404f640 1 -- 172.21.15.57:0/2282042753 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d2c0b1480 con 0x7f9d2c0c9b60 2023-10-19T13:01:08.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:08.951+0000 7f9d227fc640 1 -- 172.21.15.57:0/2282042753 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 44) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f9d24034960 con 0x7f9d2c0c9b60 2023-10-19T13:01:08.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:08.951+0000 7f9d227fc640 1 --2- 172.21.15.57:0/2282042753 >> [v2:172.21.15.179:6832/124009847,v1:172.21.15.179:6833/124009847] conn(0x7f9d1006af90 0x7f9d1006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:08.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:08.952+0000 7f9d227fc640 1 -- 172.21.15.57:0/2282042753 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f9d240aca90 con 0x7f9d2c0c9b60 2023-10-19T13:01:08.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:08.952+0000 7f9d31deb640 1 -- 172.21.15.57:0/2282042753 >> [v2:172.21.15.179:6832/124009847,v1:172.21.15.179:6833/124009847] conn(0x7f9d1006af90 msgr2=0x7f9d1006d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/124009847 2023-10-19T13:01:08.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:08.952+0000 7f9d31deb640 1 --2- 172.21.15.57:0/2282042753 >> [v2:172.21.15.179:6832/124009847,v1:172.21.15.179:6833/124009847] conn(0x7f9d1006af90 0x7f9d1006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T13:01:08.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:08.956+0000 7f9d227fc640 1 -- 172.21.15.57:0/2282042753 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9d24077820 con 0x7f9d2c0c9b60 2023-10-19T13:01:09.057 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:09.054+0000 7fb56ee2f040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-19T13:01:09.076 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:01:09.073+0000 7f14f6411040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-19T13:01:09.101 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:09.099+0000 7fb56ee2f040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-19T13:01:09.123 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:01:09.121+0000 7f14f6411040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-19T13:01:09.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.133+0000 7f9d3404f640 1 -- 172.21.15.57:0/2282042753 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9d2c0690e0 con 0x7f9d2c0c9b60 2023-10-19T13:01:09.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.134+0000 7f9d227fc640 1 -- 172.21.15.57:0/2282042753 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v101) v1 ==== 74+0+17343 (secure 0 0 0) 0x7f9d24043090 con 0x7f9d2c0c9b60 2023-10-19T13:01:09.137 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:01:09.138 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":101,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T13:01:06.603844+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2023-10-19T13:00:53.446697+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2023-10-19T13:00:54.540962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.179:6833/1413162726":"2023-10-20T13:01:06.603811+0000","172.21.15.179:6832/1413162726":"2023-10-20T13:01:06.603811+0000","172.21.15.179:0/3401895301":"2023-10-20T13:01:06.603811+0000","172.21.15.179:0/2499848570":"2023-10-20T13:01:06.603811+0000","172.21.15.179:0/3622569807":"2023-10-20T13:01:06.603811+0000","172.21.15.179:0/2835518029":"2023-10-20T13:01:06.603811+0000","172.21.15.179:0/1643279404":"2023-10-20T13:01:06.603811+0000","172.21.15.179:0/4219879451":"2023-10-20T13:01:06.603811+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T13:01:09.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.136+0000 7f9d3404f640 1 -- 172.21.15.57:0/2282042753 >> [v2:172.21.15.179:6832/124009847,v1:172.21.15.179:6833/124009847] conn(0x7f9d1006af90 msgr2=0x7f9d1006d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T13:01:09.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.136+0000 7f9d3404f640 1 --2- 172.21.15.57:0/2282042753 >> [v2:172.21.15.179:6832/124009847,v1:172.21.15.179:6833/124009847] conn(0x7f9d1006af90 0x7f9d1006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:09.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.136+0000 7f9d3404f640 1 -- 172.21.15.57:0/2282042753 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9d2c0c9b60 msgr2=0x7f9d2c0aeaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:09.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.136+0000 7f9d3404f640 1 --2- 172.21.15.57:0/2282042753 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9d2c0c9b60 0x7f9d2c0aeaa0 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f9d24005fd0 tx=0x7f9d240042f0 comp rx=0 tx=0).stop 2023-10-19T13:01:09.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.136+0000 7f9d3404f640 1 -- 172.21.15.57:0/2282042753 shutdown_connections 2023-10-19T13:01:09.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.136+0000 7f9d3404f640 1 --2- 172.21.15.57:0/2282042753 >> [v2:172.21.15.179:6832/124009847,v1:172.21.15.179:6833/124009847] conn(0x7f9d1006af90 0x7f9d1006d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:09.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.136+0000 7f9d3404f640 1 --2- 172.21.15.57:0/2282042753 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9d2c0c9b60 0x7f9d2c0aeaa0 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:09.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.136+0000 7f9d3404f640 1 --2- 172.21.15.57:0/2282042753 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9d2c0c96b0 0x7f9d2c153550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:09.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.136+0000 7f9d3404f640 1 --2- 172.21.15.57:0/2282042753 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9d2c0b1920 0x7f9d2c1582f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:09.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.136+0000 7f9d3404f640 1 -- 172.21.15.57:0/2282042753 >> 172.21.15.57:0/2282042753 conn(0x7f9d2c069930 msgr2=0x7f9d2c06f2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:09.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.136+0000 7f9d3404f640 1 -- 172.21.15.57:0/2282042753 shutdown_connections 2023-10-19T13:01:09.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.136+0000 7f9d3404f640 1 -- 172.21.15.57:0/2282042753 wait complete. 2023-10-19T13:01:09.150 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-19T12:44:07.371186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-19T13:00:53.446697+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '99', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_data', 'create_time': '2023-10-19T13:00:54.540962+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '100', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-19T13:01:09.151 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-19T13:01:09.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.261+0000 7f8c7e139640 1 -- 172.21.15.57:0/159871925 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8c781416f0 msgr2=0x7f8c780b1440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:09.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.261+0000 7f8c7e139640 1 --2- 172.21.15.57:0/159871925 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8c781416f0 0x7f8c780b1440 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f8c780cea60 tx=0x7f8c6802bd70 comp rx=0 tx=0).stop 2023-10-19T13:01:09.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.261+0000 7f8c7e139640 1 -- 172.21.15.57:0/159871925 shutdown_connections 2023-10-19T13:01:09.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.261+0000 7f8c7e139640 1 --2- 172.21.15.57:0/159871925 >> v2:172.21.15.179:3300/0 conn(0x7f8c78070790 0x7f8c780cc200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:09.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.261+0000 7f8c7e139640 1 --2- 172.21.15.57:0/159871925 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8c781416f0 0x7f8c780b1440 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:09.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.261+0000 7f8c7e139640 1 -- 172.21.15.57:0/159871925 >> 172.21.15.57:0/159871925 conn(0x7f8c780af730 msgr2=0x7f8c780afb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:09.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.261+0000 7f8c7e139640 1 -- 172.21.15.57:0/159871925 shutdown_connections 2023-10-19T13:01:09.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.261+0000 7f8c7e139640 1 -- 172.21.15.57:0/159871925 wait complete. 2023-10-19T13:01:09.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.262+0000 7f8c7e139640 1 Processor -- start 2023-10-19T13:01:09.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.262+0000 7f8c7e139640 1 -- start start 2023-10-19T13:01:09.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.263+0000 7f8c7e139640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8c78070790 0x7f8c78159480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:09.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.263+0000 7f8c7e139640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8c780b1980 0x7f8c7815b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:09.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.263+0000 7f8c7e139640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8c781416f0 0x7f8c780acca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:09.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.263+0000 7f8c7e139640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c7815a670 con 0x7f8c781416f0 2023-10-19T13:01:09.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.263+0000 7f8c7e139640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c7815be90 con 0x7f8c78070790 2023-10-19T13:01:09.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.263+0000 7f8c7e139640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c780ad1e0 con 0x7f8c780b1980 2023-10-19T13:01:09.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.263+0000 7f8c77fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8c781416f0 0x7f8c780acca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:09.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.263+0000 7f8c77fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8c781416f0 0x7f8c780acca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56064/0 (socket says 172.21.15.57:56064) 2023-10-19T13:01:09.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.263+0000 7f8c77fff640 1 -- 172.21.15.57:0/3385928886 learned_addr learned my addr 172.21.15.57:0/3385928886 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:01:09.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.263+0000 7f8c777fe640 1 --2- 172.21.15.57:0/3385928886 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8c78070790 0x7f8c78159480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:09.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.263+0000 7f8c76ffd640 1 --2- 172.21.15.57:0/3385928886 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8c780b1980 0x7f8c7815b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:09.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.264+0000 7f8c77fff640 1 -- 172.21.15.57:0/3385928886 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8c780b1980 msgr2=0x7f8c7815b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:09.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.264+0000 7f8c77fff640 1 --2- 172.21.15.57:0/3385928886 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8c780b1980 0x7f8c7815b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:09.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.264+0000 7f8c77fff640 1 -- 172.21.15.57:0/3385928886 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8c78070790 msgr2=0x7f8c78159480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:09.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.264+0000 7f8c77fff640 1 --2- 172.21.15.57:0/3385928886 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8c78070790 0x7f8c78159480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:09.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.264+0000 7f8c77fff640 1 -- 172.21.15.57:0/3385928886 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c68009490 con 0x7f8c781416f0 2023-10-19T13:01:09.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.264+0000 7f8c77fff640 1 --2- 172.21.15.57:0/3385928886 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8c781416f0 0x7f8c780acca0 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f8c7000eb10 tx=0x7f8c7000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:09.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.264+0000 7f8c5ffff640 1 -- 172.21.15.57:0/3385928886 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c7000ce80 con 0x7f8c781416f0 2023-10-19T13:01:09.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.264+0000 7f8c5ffff640 1 -- 172.21.15.57:0/3385928886 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f8c70004c10 con 0x7f8c781416f0 2023-10-19T13:01:09.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.264+0000 7f8c7e139640 1 -- 172.21.15.57:0/3385928886 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c781f1600 con 0x7f8c781416f0 2023-10-19T13:01:09.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.264+0000 7f8c5ffff640 1 -- 172.21.15.57:0/3385928886 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c70005020 con 0x7f8c781416f0 2023-10-19T13:01:09.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.264+0000 7f8c7e139640 1 -- 172.21.15.57:0/3385928886 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c78159dd0 con 0x7f8c781416f0 2023-10-19T13:01:09.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.265+0000 7f8c7e139640 1 -- 172.21.15.57:0/3385928886 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c780675f0 con 0x7f8c781416f0 2023-10-19T13:01:09.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.265+0000 7f8c5ffff640 1 -- 172.21.15.57:0/3385928886 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 44) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f8c70005180 con 0x7f8c781416f0 2023-10-19T13:01:09.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.266+0000 7f8c5ffff640 1 --2- 172.21.15.57:0/3385928886 >> [v2:172.21.15.179:6832/124009847,v1:172.21.15.179:6833/124009847] conn(0x7f8c6406af60 0x7f8c6406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:09.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.266+0000 7f8c777fe640 1 -- 172.21.15.57:0/3385928886 >> [v2:172.21.15.179:6832/124009847,v1:172.21.15.179:6833/124009847] conn(0x7f8c6406af60 msgr2=0x7f8c6406d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/124009847 2023-10-19T13:01:09.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.266+0000 7f8c5ffff640 1 -- 172.21.15.57:0/3385928886 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f8c70014070 con 0x7f8c781416f0 2023-10-19T13:01:09.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.266+0000 7f8c777fe640 1 --2- 172.21.15.57:0/3385928886 >> [v2:172.21.15.179:6832/124009847,v1:172.21.15.179:6833/124009847] conn(0x7f8c6406af60 0x7f8c6406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T13:01:09.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.270+0000 7f8c5ffff640 1 -- 172.21.15.57:0/3385928886 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8c70055410 con 0x7f8c781416f0 2023-10-19T13:01:09.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.445+0000 7f8c7e139640 1 -- 172.21.15.57:0/3385928886 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8c780b2150 con 0x7f8c781416f0 2023-10-19T13:01:09.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.446+0000 7f8c5ffff640 1 -- 172.21.15.57:0/3385928886 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v101) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8c70058a30 con 0x7f8c781416f0 2023-10-19T13:01:09.449 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-19T13:01:09.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.447+0000 7f8c7e139640 1 -- 172.21.15.57:0/3385928886 >> [v2:172.21.15.179:6832/124009847,v1:172.21.15.179:6833/124009847] conn(0x7f8c6406af60 msgr2=0x7f8c6406d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T13:01:09.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.447+0000 7f8c7e139640 1 --2- 172.21.15.57:0/3385928886 >> [v2:172.21.15.179:6832/124009847,v1:172.21.15.179:6833/124009847] conn(0x7f8c6406af60 0x7f8c6406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:09.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.447+0000 7f8c7e139640 1 -- 172.21.15.57:0/3385928886 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8c781416f0 msgr2=0x7f8c780acca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:09.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.447+0000 7f8c7e139640 1 --2- 172.21.15.57:0/3385928886 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8c781416f0 0x7f8c780acca0 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f8c7000eb10 tx=0x7f8c7000c2d0 comp rx=0 tx=0).stop 2023-10-19T13:01:09.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.448+0000 7f8c7e139640 1 -- 172.21.15.57:0/3385928886 shutdown_connections 2023-10-19T13:01:09.452 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.448+0000 7f8c7e139640 1 --2- 172.21.15.57:0/3385928886 >> [v2:172.21.15.179:6832/124009847,v1:172.21.15.179:6833/124009847] conn(0x7f8c6406af60 0x7f8c6406d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:09.452 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.448+0000 7f8c7e139640 1 --2- 172.21.15.57:0/3385928886 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8c781416f0 0x7f8c780acca0 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:09.452 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.448+0000 7f8c7e139640 1 --2- 172.21.15.57:0/3385928886 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8c780b1980 0x7f8c7815b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:09.452 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.448+0000 7f8c7e139640 1 --2- 172.21.15.57:0/3385928886 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8c78070790 0x7f8c78159480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:09.452 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.448+0000 7f8c7e139640 1 -- 172.21.15.57:0/3385928886 >> 172.21.15.57:0/3385928886 conn(0x7f8c780af730 msgr2=0x7f8c78140940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:09.453 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.448+0000 7f8c7e139640 1 -- 172.21.15.57:0/3385928886 shutdown_connections 2023-10-19T13:01:09.453 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.448+0000 7f8c7e139640 1 -- 172.21.15.57:0/3385928886 wait complete. 2023-10-19T13:01:09.461 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-19T13:01:09.488 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:09.486+0000 7fb56ee2f040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-19T13:01:09.532 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:09.529+0000 7fb56ee2f040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-19T13:01:09.542 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:01:09.540+0000 7f14f6411040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-19T13:01:09.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.572+0000 7f21f3439640 1 -- 172.21.15.57:0/738052639 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f21ec0ceb40 msgr2=0x7f21ec0b0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:09.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.572+0000 7f21f3439640 1 --2- 172.21.15.57:0/738052639 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f21ec0ceb40 0x7f21ec0b0650 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f21e4005fd0 tx=0x7f21e402c050 comp rx=0 tx=0).stop 2023-10-19T13:01:09.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.572+0000 7f21f3439640 1 -- 172.21.15.57:0/738052639 shutdown_connections 2023-10-19T13:01:09.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.572+0000 7f21f3439640 1 --2- 172.21.15.57:0/738052639 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f21ec0ceb40 0x7f21ec0b0650 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:09.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.572+0000 7f21f3439640 1 --2- 172.21.15.57:0/738052639 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f21ec15b2a0 0x7f21ec0d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:09.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.572+0000 7f21f3439640 1 -- 172.21.15.57:0/738052639 >> 172.21.15.57:0/738052639 conn(0x7f21ec0d0d60 msgr2=0x7f21ec0d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:09.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.572+0000 7f21f3439640 1 -- 172.21.15.57:0/738052639 shutdown_connections 2023-10-19T13:01:09.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.572+0000 7f21f3439640 1 -- 172.21.15.57:0/738052639 wait complete. 2023-10-19T13:01:09.577 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:09.572+0000 7fb56ee2f040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-19T13:01:09.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.573+0000 7f21f3439640 1 Processor -- start 2023-10-19T13:01:09.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.573+0000 7f21f3439640 1 -- start start 2023-10-19T13:01:09.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.574+0000 7f21f3439640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f21ec0ceb40 0x7f21ec14f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:09.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.574+0000 7f21f3439640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f21ec0cf340 0x7f21ec0c9e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:09.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.574+0000 7f21f3439640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f21ec15b2a0 0x7f21ec0ca380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:09.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.574+0000 7f21f3439640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21ec15c3b0 con 0x7f21ec15b2a0 2023-10-19T13:01:09.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.574+0000 7f21f3439640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21ec15b770 con 0x7f21ec0cf340 2023-10-19T13:01:09.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.574+0000 7f21f3439640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21ec15ba10 con 0x7f21ec0ceb40 2023-10-19T13:01:09.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.574+0000 7f21f19d6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f21ec15b2a0 0x7f21ec0ca380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:09.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.574+0000 7f21f19d6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f21ec15b2a0 0x7f21ec0ca380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56076/0 (socket says 172.21.15.57:56076) 2023-10-19T13:01:09.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.574+0000 7f21f19d6640 1 -- 172.21.15.57:0/3387180656 learned_addr learned my addr 172.21.15.57:0/3387180656 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:01:09.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.574+0000 7f21f09d4640 1 --2- 172.21.15.57:0/3387180656 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f21ec0cf340 0x7f21ec0c9e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:09.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.574+0000 7f21f11d5640 1 --2- 172.21.15.57:0/3387180656 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f21ec0ceb40 0x7f21ec14f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:09.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.574+0000 7f21f19d6640 1 -- 172.21.15.57:0/3387180656 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f21ec0ceb40 msgr2=0x7f21ec14f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:09.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.574+0000 7f21f19d6640 1 --2- 172.21.15.57:0/3387180656 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f21ec0ceb40 0x7f21ec14f170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:09.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.574+0000 7f21f19d6640 1 -- 172.21.15.57:0/3387180656 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f21ec0cf340 msgr2=0x7f21ec0c9e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:09.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.574+0000 7f21f19d6640 1 --2- 172.21.15.57:0/3387180656 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f21ec0cf340 0x7f21ec0c9e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:09.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.574+0000 7f21f19d6640 1 -- 172.21.15.57:0/3387180656 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21e8009490 con 0x7f21ec15b2a0 2023-10-19T13:01:09.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.574+0000 7f21f09d4640 1 --2- 172.21.15.57:0/3387180656 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f21ec0cf340 0x7f21ec0c9e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T13:01:09.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.574+0000 7f21f19d6640 1 --2- 172.21.15.57:0/3387180656 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f21ec15b2a0 0x7f21ec0ca380 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f21e4009b30 tx=0x7f21e4004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:09.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.575+0000 7f21e1ffb640 1 -- 172.21.15.57:0/3387180656 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4004870 con 0x7f21ec15b2a0 2023-10-19T13:01:09.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.575+0000 7f21e1ffb640 1 -- 172.21.15.57:0/3387180656 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f21e403e550 con 0x7f21ec15b2a0 2023-10-19T13:01:09.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.575+0000 7f21e1ffb640 1 -- 172.21.15.57:0/3387180656 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e403e960 con 0x7f21ec15b2a0 2023-10-19T13:01:09.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.575+0000 7f21f3439640 1 -- 172.21.15.57:0/3387180656 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21e4009840 con 0x7f21ec15b2a0 2023-10-19T13:01:09.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.575+0000 7f21f3439640 1 -- 172.21.15.57:0/3387180656 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21ec15bfd0 con 0x7f21ec15b2a0 2023-10-19T13:01:09.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.576+0000 7f21f3439640 1 -- 172.21.15.57:0/3387180656 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21ec0690e0 con 0x7f21ec15b2a0 2023-10-19T13:01:09.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.576+0000 7f21e1ffb640 1 -- 172.21.15.57:0/3387180656 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 44) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f21e403eac0 con 0x7f21ec15b2a0 2023-10-19T13:01:09.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.577+0000 7f21e1ffb640 1 --2- 172.21.15.57:0/3387180656 >> [v2:172.21.15.179:6832/124009847,v1:172.21.15.179:6833/124009847] conn(0x7f21d006af60 0x7f21d006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:09.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.577+0000 7f21e1ffb640 1 -- 172.21.15.57:0/3387180656 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f21e40ac950 con 0x7f21ec15b2a0 2023-10-19T13:01:09.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.577+0000 7f21f11d5640 1 -- 172.21.15.57:0/3387180656 >> [v2:172.21.15.179:6832/124009847,v1:172.21.15.179:6833/124009847] conn(0x7f21d006af60 msgr2=0x7f21d006d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/124009847 2023-10-19T13:01:09.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.577+0000 7f21f11d5640 1 --2- 172.21.15.57:0/3387180656 >> [v2:172.21.15.179:6832/124009847,v1:172.21.15.179:6833/124009847] conn(0x7f21d006af60 0x7f21d006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T13:01:09.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.581+0000 7f21e1ffb640 1 -- 172.21.15.57:0/3387180656 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f21e4077660 con 0x7f21ec15b2a0 2023-10-19T13:01:09.591 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:01:09.588+0000 7f14f6411040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-19T13:01:09.626 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:09.624+0000 7fb56ee2f040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-19T13:01:09.636 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:01:09.634+0000 7f14f6411040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-19T13:01:09.667 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:09.665+0000 7fb56ee2f040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-19T13:01:09.691 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:01:09.689+0000 7f14f6411040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-19T13:01:09.735 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:01:09.732+0000 7f14f6411040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-19T13:01:09.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.761+0000 7f21f3439640 1 -- 172.21.15.57:0/3387180656 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f21ec0d0b40 con 0x7f21ec15b2a0 2023-10-19T13:01:09.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.761+0000 7f21e1ffb640 1 -- 172.21.15.57:0/3387180656 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v101) v1 ==== 104+0+11 (secure 0 0 0) 0x7f21e407ac80 con 0x7f21ec15b2a0 2023-10-19T13:01:09.764 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2023-10-19T13:01:09.765 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:09.763+0000 7fb56ee2f040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-19T13:01:09.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.764+0000 7f21f3439640 1 -- 172.21.15.57:0/3387180656 >> [v2:172.21.15.179:6832/124009847,v1:172.21.15.179:6833/124009847] conn(0x7f21d006af60 msgr2=0x7f21d006d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T13:01:09.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.764+0000 7f21f3439640 1 --2- 172.21.15.57:0/3387180656 >> [v2:172.21.15.179:6832/124009847,v1:172.21.15.179:6833/124009847] conn(0x7f21d006af60 0x7f21d006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:09.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.764+0000 7f21f3439640 1 -- 172.21.15.57:0/3387180656 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f21ec15b2a0 msgr2=0x7f21ec0ca380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:09.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.764+0000 7f21f3439640 1 --2- 172.21.15.57:0/3387180656 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f21ec15b2a0 0x7f21ec0ca380 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f21e4009b30 tx=0x7f21e4004270 comp rx=0 tx=0).stop 2023-10-19T13:01:09.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.764+0000 7f21f3439640 1 -- 172.21.15.57:0/3387180656 shutdown_connections 2023-10-19T13:01:09.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.764+0000 7f21f3439640 1 --2- 172.21.15.57:0/3387180656 >> [v2:172.21.15.179:6832/124009847,v1:172.21.15.179:6833/124009847] conn(0x7f21d006af60 0x7f21d006d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:09.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.764+0000 7f21f3439640 1 --2- 172.21.15.57:0/3387180656 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f21ec15b2a0 0x7f21ec0ca380 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:09.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.764+0000 7f21f3439640 1 --2- 172.21.15.57:0/3387180656 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f21ec0cf340 0x7f21ec0c9e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:09.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.764+0000 7f21f3439640 1 --2- 172.21.15.57:0/3387180656 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f21ec0ceb40 0x7f21ec14f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:09.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.764+0000 7f21f3439640 1 -- 172.21.15.57:0/3387180656 >> 172.21.15.57:0/3387180656 conn(0x7f21ec0d0d60 msgr2=0x7f21ec159ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:09.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.764+0000 7f21f3439640 1 -- 172.21.15.57:0/3387180656 shutdown_connections 2023-10-19T13:01:09.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.764+0000 7f21f3439640 1 -- 172.21.15.57:0/3387180656 wait complete. 2023-10-19T13:01:09.778 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-19T13:01:09.817 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:09.815+0000 7fb56ee2f040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-19T13:01:09.842 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:01:09.839+0000 7f14f6411040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-19T13:01:09.858 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:09.856+0000 7fb56ee2f040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-19T13:01:09.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.889+0000 7fe809705640 1 -- 172.21.15.57:0/1856351588 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe8040acd40 msgr2=0x7fe8041597c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:09.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.889+0000 7fe809705640 1 --2- 172.21.15.57:0/1856351588 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe8040acd40 0x7fe8041597c0 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7fe7fc005fd0 tx=0x7fe7fc02c240 comp rx=0 tx=0).stop 2023-10-19T13:01:09.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.889+0000 7fe809705640 1 -- 172.21.15.57:0/1856351588 shutdown_connections 2023-10-19T13:01:09.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.889+0000 7fe809705640 1 --2- 172.21.15.57:0/1856351588 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe8040acd40 0x7fe8041597c0 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:09.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.889+0000 7fe809705640 1 --2- 172.21.15.57:0/1856351588 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe8040b24d0 0x7fe8040b28b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:09.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.889+0000 7fe809705640 1 -- 172.21.15.57:0/1856351588 >> 172.21.15.57:0/1856351588 conn(0x7fe8040b3f00 msgr2=0x7fe8040b4310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:09.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.889+0000 7fe809705640 1 -- 172.21.15.57:0/1856351588 shutdown_connections 2023-10-19T13:01:09.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.889+0000 7fe809705640 1 -- 172.21.15.57:0/1856351588 wait complete. 2023-10-19T13:01:09.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.890+0000 7fe809705640 1 Processor -- start 2023-10-19T13:01:09.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.890+0000 7fe809705640 1 -- start start 2023-10-19T13:01:09.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.891+0000 7fe809705640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe8040acd40 0x7fe80414d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:09.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.891+0000 7fe809705640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe8040b24d0 0x7fe804152490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:09.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.891+0000 7fe809705640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe8040b3070 0x7fe8041529d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:09.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.891+0000 7fe809705640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe80414eed0 con 0x7fe8040b24d0 2023-10-19T13:01:09.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.891+0000 7fe809705640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe80414de30 con 0x7fe8040b3070 2023-10-19T13:01:09.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.891+0000 7fe809705640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe80414e110 con 0x7fe8040acd40 2023-10-19T13:01:09.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.891+0000 7fe8027fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe8040b24d0 0x7fe804152490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:09.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.891+0000 7fe8027fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe8040b24d0 0x7fe804152490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56102/0 (socket says 172.21.15.57:56102) 2023-10-19T13:01:09.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.891+0000 7fe8027fc640 1 -- 172.21.15.57:0/958514374 learned_addr learned my addr 172.21.15.57:0/958514374 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:01:09.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.891+0000 7fe802ffd640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe8040acd40 0x7fe80414d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:09.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.891+0000 7fe8037fe640 1 --2- 172.21.15.57:0/958514374 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe8040b3070 0x7fe8041529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:09.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.891+0000 7fe8027fc640 1 -- 172.21.15.57:0/958514374 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe8040acd40 msgr2=0x7fe80414d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:09.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.891+0000 7fe8027fc640 1 --2- 172.21.15.57:0/958514374 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe8040acd40 0x7fe80414d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:09.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.891+0000 7fe8027fc640 1 -- 172.21.15.57:0/958514374 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe8040b3070 msgr2=0x7fe8041529d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:09.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.891+0000 7fe8027fc640 1 --2- 172.21.15.57:0/958514374 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe8040b3070 0x7fe8041529d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:09.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.891+0000 7fe8027fc640 1 -- 172.21.15.57:0/958514374 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7fc009900 con 0x7fe8040b24d0 2023-10-19T13:01:09.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.891+0000 7fe8037fe640 1 --2- 172.21.15.57:0/958514374 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe8040b3070 0x7fe8041529d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T13:01:09.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.891+0000 7fe8027fc640 1 --2- 172.21.15.57:0/958514374 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe8040b24d0 0x7fe804152490 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7fe7f800c400 tx=0x7fe7f800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:09.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.892+0000 7fe7eb7fe640 1 -- 172.21.15.57:0/958514374 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7f80052d0 con 0x7fe8040b24d0 2023-10-19T13:01:09.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.892+0000 7fe7eb7fe640 1 -- 172.21.15.57:0/958514374 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fe7f80035f0 con 0x7fe8040b24d0 2023-10-19T13:01:09.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.892+0000 7fe809705640 1 -- 172.21.15.57:0/958514374 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe80414e3f0 con 0x7fe8040b24d0 2023-10-19T13:01:09.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.892+0000 7fe7eb7fe640 1 -- 172.21.15.57:0/958514374 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7f8003a00 con 0x7fe8040b24d0 2023-10-19T13:01:09.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.892+0000 7fe809705640 1 -- 172.21.15.57:0/958514374 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe80414e9b0 con 0x7fe8040b24d0 2023-10-19T13:01:09.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.893+0000 7fe809705640 1 -- 172.21.15.57:0/958514374 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8040b22c0 con 0x7fe8040b24d0 2023-10-19T13:01:09.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.893+0000 7fe7eb7fe640 1 -- 172.21.15.57:0/958514374 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 44) v1 ==== 89332+0+0 (secure 0 0 0) 0x7fe7f8003b60 con 0x7fe8040b24d0 2023-10-19T13:01:09.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.894+0000 7fe7eb7fe640 1 --2- 172.21.15.57:0/958514374 >> [v2:172.21.15.179:6832/124009847,v1:172.21.15.179:6833/124009847] conn(0x7fe7f006af60 0x7fe7f006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:09.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.894+0000 7fe802ffd640 1 -- 172.21.15.57:0/958514374 >> [v2:172.21.15.179:6832/124009847,v1:172.21.15.179:6833/124009847] conn(0x7fe7f006af60 msgr2=0x7fe7f006d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/124009847 2023-10-19T13:01:09.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.894+0000 7fe7eb7fe640 1 -- 172.21.15.57:0/958514374 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6401+0+0 (secure 0 0 0) 0x7fe7f8089190 con 0x7fe8040b24d0 2023-10-19T13:01:09.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.894+0000 7fe802ffd640 1 --2- 172.21.15.57:0/958514374 >> [v2:172.21.15.179:6832/124009847,v1:172.21.15.179:6833/124009847] conn(0x7fe7f006af60 0x7fe7f006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T13:01:09.902 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:01:09.897+0000 7f14f6411040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-19T13:01:09.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:09.898+0000 7fe7eb7fe640 1 -- 172.21.15.57:0/958514374 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe7f8053f20 con 0x7fe8040b24d0 2023-10-19T13:01:09.907 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:09.904+0000 7fb56ee2f040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-19T13:01:09.945 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:01:09.942+0000 7f14f6411040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-19T13:01:09.949 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:09.946+0000 7fb56ee2f040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-19T13:01:09.996 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:01:09.993+0000 7f14f6411040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-19T13:01:10.040 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:01:10.038+0000 7f14f6411040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-19T13:01:10.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.073+0000 7fe809705640 1 -- 172.21.15.57:0/958514374 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe8040abb60 con 0x7fe8040b24d0 2023-10-19T13:01:10.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.074+0000 7fe7eb7fe640 1 -- 172.21.15.57:0/958514374 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v101) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe7f8057540 con 0x7fe8040b24d0 2023-10-19T13:01:10.077 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 64 2023-10-19T13:01:10.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.076+0000 7fe809705640 1 -- 172.21.15.57:0/958514374 >> [v2:172.21.15.179:6832/124009847,v1:172.21.15.179:6833/124009847] conn(0x7fe7f006af60 msgr2=0x7fe7f006d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T13:01:10.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.076+0000 7fe809705640 1 --2- 172.21.15.57:0/958514374 >> [v2:172.21.15.179:6832/124009847,v1:172.21.15.179:6833/124009847] conn(0x7fe7f006af60 0x7fe7f006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:10.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.076+0000 7fe809705640 1 -- 172.21.15.57:0/958514374 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe8040b24d0 msgr2=0x7fe804152490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:10.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.076+0000 7fe809705640 1 --2- 172.21.15.57:0/958514374 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe8040b24d0 0x7fe804152490 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7fe7f800c400 tx=0x7fe7f800c8c0 comp rx=0 tx=0).stop 2023-10-19T13:01:10.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.076+0000 7fe809705640 1 -- 172.21.15.57:0/958514374 shutdown_connections 2023-10-19T13:01:10.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.076+0000 7fe809705640 1 --2- 172.21.15.57:0/958514374 >> [v2:172.21.15.179:6832/124009847,v1:172.21.15.179:6833/124009847] conn(0x7fe7f006af60 0x7fe7f006d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:10.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.076+0000 7fe809705640 1 --2- 172.21.15.57:0/958514374 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe8040b3070 0x7fe8041529d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:10.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.076+0000 7fe809705640 1 --2- 172.21.15.57:0/958514374 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe8040b24d0 0x7fe804152490 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:10.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.076+0000 7fe809705640 1 --2- 172.21.15.57:0/958514374 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe8040acd40 0x7fe80414d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:10.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.076+0000 7fe809705640 1 -- 172.21.15.57:0/958514374 >> 172.21.15.57:0/958514374 conn(0x7fe8040b3f00 msgr2=0x7fe8040b42e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:10.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.076+0000 7fe809705640 1 -- 172.21.15.57:0/958514374 shutdown_connections 2023-10-19T13:01:10.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.076+0000 7fe809705640 1 -- 172.21.15.57:0/958514374 wait complete. 2023-10-19T13:01:10.090 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-19T13:01:10.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.200+0000 7f6fce505640 1 -- 172.21.15.57:0/1536066482 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6fc815b2a0 msgr2=0x7f6fc80d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:10.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.200+0000 7f6fce505640 1 --2- 172.21.15.57:0/1536066482 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6fc815b2a0 0x7f6fc80d1910 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f6fb8009830 tx=0x7f6fb802bd60 comp rx=0 tx=0).stop 2023-10-19T13:01:10.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.200+0000 7f6fce505640 1 -- 172.21.15.57:0/1536066482 shutdown_connections 2023-10-19T13:01:10.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.200+0000 7f6fce505640 1 --2- 172.21.15.57:0/1536066482 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6fc815b2a0 0x7f6fc80d1910 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:10.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.200+0000 7f6fce505640 1 -- 172.21.15.57:0/1536066482 >> 172.21.15.57:0/1536066482 conn(0x7f6fc80d0d60 msgr2=0x7f6fc80d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:10.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.200+0000 7f6fce505640 1 -- 172.21.15.57:0/1536066482 shutdown_connections 2023-10-19T13:01:10.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.200+0000 7f6fce505640 1 -- 172.21.15.57:0/1536066482 wait complete. 2023-10-19T13:01:10.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.201+0000 7f6fce505640 1 Processor -- start 2023-10-19T13:01:10.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.201+0000 7f6fce505640 1 -- start start 2023-10-19T13:01:10.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.202+0000 7f6fce505640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6fc80ceb40 0x7f6fc815d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:10.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.202+0000 7f6fce505640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6fc80cf340 0x7f6fc815cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:10.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.202+0000 7f6fce505640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6fc815b2a0 0x7f6fc80ae3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:10.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.202+0000 7f6fce505640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc81f5940 con 0x7f6fc80ceb40 2023-10-19T13:01:10.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.202+0000 7f6fce505640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc81f4d30 con 0x7f6fc80cf340 2023-10-19T13:01:10.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.202+0000 7f6fce505640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc81f5010 con 0x7f6fc815b2a0 2023-10-19T13:01:10.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.202+0000 7f6fc7fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6fc80ceb40 0x7f6fc815d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:10.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.202+0000 7f6fc7fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6fc80ceb40 0x7f6fc815d5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56126/0 (socket says 172.21.15.57:56126) 2023-10-19T13:01:10.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.202+0000 7f6fc7fff640 1 -- 172.21.15.57:0/1049148818 learned_addr learned my addr 172.21.15.57:0/1049148818 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:01:10.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.202+0000 7f6fc77fe640 1 --2- 172.21.15.57:0/1049148818 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6fc80cf340 0x7f6fc815cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:10.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.202+0000 7f6fccaa2640 1 --2- 172.21.15.57:0/1049148818 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6fc815b2a0 0x7f6fc80ae3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:10.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.202+0000 7f6fc7fff640 1 -- 172.21.15.57:0/1049148818 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6fc815b2a0 msgr2=0x7f6fc80ae3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:10.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.202+0000 7f6fc7fff640 1 --2- 172.21.15.57:0/1049148818 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6fc815b2a0 0x7f6fc80ae3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:10.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.202+0000 7f6fc7fff640 1 -- 172.21.15.57:0/1049148818 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6fc80cf340 msgr2=0x7f6fc815cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:10.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.202+0000 7f6fc7fff640 1 --2- 172.21.15.57:0/1049148818 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6fc80cf340 0x7f6fc815cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:10.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.202+0000 7f6fc7fff640 1 -- 172.21.15.57:0/1049148818 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fb8009490 con 0x7f6fc80ceb40 2023-10-19T13:01:10.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.203+0000 7f6fc7fff640 1 --2- 172.21.15.57:0/1049148818 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6fc80ceb40 0x7f6fc815d5f0 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc00c500 tx=0x7f6fbc00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:10.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.203+0000 7f6fc4ff9640 1 -- 172.21.15.57:0/1049148818 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc0052d0 con 0x7f6fc80ceb40 2023-10-19T13:01:10.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.203+0000 7f6fc4ff9640 1 -- 172.21.15.57:0/1049148818 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f6fbc005450 con 0x7f6fc80ceb40 2023-10-19T13:01:10.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.203+0000 7f6fc4ff9640 1 -- 172.21.15.57:0/1049148818 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc012890 con 0x7f6fc80ceb40 2023-10-19T13:01:10.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.203+0000 7f6fce505640 1 -- 172.21.15.57:0/1049148818 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fc81f52f0 con 0x7f6fc80ceb40 2023-10-19T13:01:10.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.203+0000 7f6fce505640 1 -- 172.21.15.57:0/1049148818 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fc815d1a0 con 0x7f6fc80ceb40 2023-10-19T13:01:10.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.204+0000 7f6fce505640 1 -- 172.21.15.57:0/1049148818 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fc80690e0 con 0x7f6fc80ceb40 2023-10-19T13:01:10.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.205+0000 7f6fc4ff9640 1 -- 172.21.15.57:0/1049148818 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 44) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f6fbc0035f0 con 0x7f6fc80ceb40 2023-10-19T13:01:10.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.205+0000 7f6fc4ff9640 1 --2- 172.21.15.57:0/1049148818 >> [v2:172.21.15.179:6832/124009847,v1:172.21.15.179:6833/124009847] conn(0x7f6fb406af90 0x7f6fb406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:10.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.205+0000 7f6fc77fe640 1 -- 172.21.15.57:0/1049148818 >> [v2:172.21.15.179:6832/124009847,v1:172.21.15.179:6833/124009847] conn(0x7f6fb406af90 msgr2=0x7f6fb406d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/124009847 2023-10-19T13:01:10.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.205+0000 7f6fc4ff9640 1 -- 172.21.15.57:0/1049148818 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f6fbc089780 con 0x7f6fc80ceb40 2023-10-19T13:01:10.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.205+0000 7f6fc77fe640 1 --2- 172.21.15.57:0/1049148818 >> [v2:172.21.15.179:6832/124009847,v1:172.21.15.179:6833/124009847] conn(0x7f6fb406af90 0x7f6fb406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T13:01:10.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.209+0000 7f6fc4ff9640 1 -- 172.21.15.57:0/1049148818 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6fbc054470 con 0x7f6fc80ceb40 2023-10-19T13:01:10.265 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:10.262+0000 7fb56ee2f040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-19T13:01:10.394 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:10.391+0000 7fb56ee2f040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-19T13:01:10.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.406+0000 7f6fc77fe640 1 -- 172.21.15.57:0/1049148818 >> [v2:172.21.15.179:6832/124009847,v1:172.21.15.179:6833/124009847] conn(0x7f6fb406af90 msgr2=0x7f6fb406d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/124009847 2023-10-19T13:01:10.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.406+0000 7f6fc77fe640 1 --2- 172.21.15.57:0/1049148818 >> [v2:172.21.15.179:6832/124009847,v1:172.21.15.179:6833/124009847] conn(0x7f6fb406af90 0x7f6fb406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-19T13:01:10.415 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:01:10.412+0000 7f14f6411040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-19T13:01:10.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.443+0000 7f6fce505640 1 -- 172.21.15.57:0/1049148818 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f6fc81f5a80 con 0x7f6fc80ceb40 2023-10-19T13:01:10.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.443+0000 7f6fc4ff9640 1 -- 172.21.15.57:0/1049148818 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v101) v1 ==== 80+0+478 (secure 0 0 0) 0x7f6fbc057a90 con 0x7f6fc80ceb40 2023-10-19T13:01:10.446 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:6833/1413162726 2023-10-20T13:01:06.603811+0000 2023-10-19T13:01:10.447 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:6832/1413162726 2023-10-20T13:01:06.603811+0000 2023-10-19T13:01:10.447 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/3401895301 2023-10-20T13:01:06.603811+0000 2023-10-19T13:01:10.447 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/2499848570 2023-10-20T13:01:06.603811+0000 2023-10-19T13:01:10.448 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/3622569807 2023-10-20T13:01:06.603811+0000 2023-10-19T13:01:10.448 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/2835518029 2023-10-20T13:01:06.603811+0000 2023-10-19T13:01:10.448 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/1643279404 2023-10-20T13:01:06.603811+0000 2023-10-19T13:01:10.448 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/4219879451 2023-10-20T13:01:06.603811+0000 2023-10-19T13:01:10.449 INFO:teuthology.orchestra.run.smithi057.stderr:listed 8 entries 2023-10-19T13:01:10.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.445+0000 7f6fce505640 1 -- 172.21.15.57:0/1049148818 >> [v2:172.21.15.179:6832/124009847,v1:172.21.15.179:6833/124009847] conn(0x7f6fb406af90 msgr2=0x7f6fb406d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T13:01:10.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.445+0000 7f6fce505640 1 --2- 172.21.15.57:0/1049148818 >> [v2:172.21.15.179:6832/124009847,v1:172.21.15.179:6833/124009847] conn(0x7f6fb406af90 0x7f6fb406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:10.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.445+0000 7f6fce505640 1 -- 172.21.15.57:0/1049148818 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6fc80ceb40 msgr2=0x7f6fc815d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:10.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.445+0000 7f6fce505640 1 --2- 172.21.15.57:0/1049148818 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6fc80ceb40 0x7f6fc815d5f0 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc00c500 tx=0x7f6fbc00c9c0 comp rx=0 tx=0).stop 2023-10-19T13:01:10.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.445+0000 7f6fce505640 1 -- 172.21.15.57:0/1049148818 shutdown_connections 2023-10-19T13:01:10.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.445+0000 7f6fce505640 1 --2- 172.21.15.57:0/1049148818 >> [v2:172.21.15.179:6832/124009847,v1:172.21.15.179:6833/124009847] conn(0x7f6fb406af90 0x7f6fb406d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:10.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.445+0000 7f6fce505640 1 --2- 172.21.15.57:0/1049148818 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6fc815b2a0 0x7f6fc80ae3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:10.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.445+0000 7f6fce505640 1 --2- 172.21.15.57:0/1049148818 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6fc80cf340 0x7f6fc815cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:10.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.445+0000 7f6fce505640 1 --2- 172.21.15.57:0/1049148818 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6fc80ceb40 0x7f6fc815d5f0 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:10.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.445+0000 7f6fce505640 1 -- 172.21.15.57:0/1049148818 >> 172.21.15.57:0/1049148818 conn(0x7f6fc80d0d60 msgr2=0x7f6fc80ad8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:10.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.445+0000 7f6fce505640 1 -- 172.21.15.57:0/1049148818 shutdown_connections 2023-10-19T13:01:10.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.445+0000 7f6fce505640 1 -- 172.21.15.57:0/1049148818 wait complete. 2023-10-19T13:01:10.459 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi057.front.sepia.ceph.com... 2023-10-19T13:01:10.459 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:01:10.459 DEBUG:teuthology.orchestra.run.smithi057:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-10-19T13:01:10.485 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:10.482+0000 7fb56ee2f040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-19T13:01:10.488 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi057.stderr:ceph-fuse[122054]: fuse finished with error 0 and tester_r 0 2023-10-19T13:01:10.490 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi057.front.sepia.ceph.com 2023-10-19T13:01:10.491 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:01:10.491 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-19T13:01:10.528 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:10.526+0000 7fb56ee2f040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-19T13:01:10.557 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:01:10.554+0000 7f14f6411040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-19T13:01:10.623 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:10.620+0000 7fb56ee2f040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-19T13:01:10.634 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-19T13:01:10.634 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T13:01:10.650 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054833,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=284 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3247632k,nr_inodes=811908,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-19T13:01:10.650 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi179.front.sepia.ceph.com 2023-10-19T13:01:10.650 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:01:10.651 DEBUG:teuthology.orchestra.run.smithi179:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-10-19T13:01:10.654 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:01:10.651+0000 7f14f6411040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-19T13:01:10.699 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:01:10.697+0000 7f14f6411040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-19T13:01:10.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-19T13:01:10.704 INFO:teuthology.orchestra.run.smithi179.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2023-10-19T13:01:10.705 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T13:01:10.705 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T13:01:10.717 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:10.715+0000 7fb56ee2f040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-19T13:01:10.722 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055614,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=358 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248260k,nr_inodes=812065,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-19T13:01:10.722 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi057.front.sepia.ceph.com 2023-10-19T13:01:10.722 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:01:10.723 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-10-19T13:01:10.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-19T13:01:10.777 INFO:teuthology.orchestra.run.smithi057.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2023-10-19T13:01:10.777 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-19T13:01:10.778 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T13:01:10.793 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054833,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=284 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3247632k,nr_inodes=811908,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-19T13:01:10.794 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi179.front.sepia.ceph.com 2023-10-19T13:01:10.794 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:01:10.795 DEBUG:teuthology.orchestra.run.smithi179:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-10-19T13:01:10.801 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:01:10.799+0000 7f14f6411040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-19T13:01:10.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-19T13:01:10.848 INFO:teuthology.orchestra.run.smithi179.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2023-10-19T13:01:10.849 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T13:01:10.858 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:10.855+0000 7fb56ee2f040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-19T13:01:10.899 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:01:10.896+0000 7f14f6411040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-19T13:01:10.900 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:10.898+0000 7fb56ee2f040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-19T13:01:10.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.957+0000 7ffb32c83640 1 -- 172.21.15.57:0/1840830375 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ffb2c15b260 msgr2=0x7ffb2c0cf310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:10.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.957+0000 7ffb32c83640 1 --2- 172.21.15.57:0/1840830375 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ffb2c15b260 0x7ffb2c0cf310 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ffb20005fd0 tx=0x7ffb2002be10 comp rx=0 tx=0).stop 2023-10-19T13:01:10.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.957+0000 7ffb32c83640 1 -- 172.21.15.57:0/1840830375 shutdown_connections 2023-10-19T13:01:10.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.957+0000 7ffb32c83640 1 --2- 172.21.15.57:0/1840830375 >> v2:172.21.15.179:3300/0 conn(0x7ffb2c0cf850 0x7ffb2c0cc9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:10.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.957+0000 7ffb32c83640 1 --2- 172.21.15.57:0/1840830375 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ffb2c15b260 0x7ffb2c0cf310 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:10.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.957+0000 7ffb32c83640 1 -- 172.21.15.57:0/1840830375 >> 172.21.15.57:0/1840830375 conn(0x7ffb2c09a030 msgr2=0x7ffb2c0d0d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:10.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.957+0000 7ffb32c83640 1 -- 172.21.15.57:0/1840830375 shutdown_connections 2023-10-19T13:01:10.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.957+0000 7ffb32c83640 1 -- 172.21.15.57:0/1840830375 wait complete. 2023-10-19T13:01:10.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.958+0000 7ffb32c83640 1 Processor -- start 2023-10-19T13:01:10.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.958+0000 7ffb32c83640 1 -- start start 2023-10-19T13:01:10.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.959+0000 7ffb32c83640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffb2c0cf850 0x7ffb2c0ca910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:10.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.959+0000 7ffb32c83640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ffb2c15b260 0x7ffb2c0c5ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:10.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.959+0000 7ffb32c83640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffb2c15b640 0x7ffb2c0c8060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:10.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.959+0000 7ffb32c83640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb2c0c6f50 con 0x7ffb2c15b640 2023-10-19T13:01:10.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.959+0000 7ffb32c83640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb2c0c6010 con 0x7ffb2c0cf850 2023-10-19T13:01:10.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.959+0000 7ffb32c83640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb2c0c62d0 con 0x7ffb2c15b260 2023-10-19T13:01:10.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.959+0000 7ffb31220640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffb2c15b640 0x7ffb2c0c8060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:10.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.959+0000 7ffb31220640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffb2c15b640 0x7ffb2c0c8060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56140/0 (socket says 172.21.15.57:56140) 2023-10-19T13:01:10.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.959+0000 7ffb31220640 1 -- 172.21.15.57:0/588837887 learned_addr learned my addr 172.21.15.57:0/588837887 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:01:10.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.959+0000 7ffb2bfff640 1 --2- 172.21.15.57:0/588837887 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ffb2c15b260 0x7ffb2c0c5ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:10.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.959+0000 7ffb30a1f640 1 --2- 172.21.15.57:0/588837887 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffb2c0cf850 0x7ffb2c0ca910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:10.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.960+0000 7ffb31220640 1 -- 172.21.15.57:0/588837887 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ffb2c15b260 msgr2=0x7ffb2c0c5ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:10.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.960+0000 7ffb31220640 1 --2- 172.21.15.57:0/588837887 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ffb2c15b260 0x7ffb2c0c5ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:10.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.960+0000 7ffb31220640 1 -- 172.21.15.57:0/588837887 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffb2c0cf850 msgr2=0x7ffb2c0ca910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:10.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.960+0000 7ffb31220640 1 --2- 172.21.15.57:0/588837887 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffb2c0cf850 0x7ffb2c0ca910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:10.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.960+0000 7ffb31220640 1 -- 172.21.15.57:0/588837887 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb24009840 con 0x7ffb2c15b640 2023-10-19T13:01:10.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.960+0000 7ffb31220640 1 --2- 172.21.15.57:0/588837887 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffb2c15b640 0x7ffb2c0c8060 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7ffb2400efd0 tx=0x7ffb2400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:10.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.960+0000 7ffb297fa640 1 -- 172.21.15.57:0/588837887 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb24009b40 con 0x7ffb2c15b640 2023-10-19T13:01:10.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.960+0000 7ffb297fa640 1 -- 172.21.15.57:0/588837887 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7ffb24003c60 con 0x7ffb2c15b640 2023-10-19T13:01:10.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.960+0000 7ffb32c83640 1 -- 172.21.15.57:0/588837887 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb20009560 con 0x7ffb2c15b640 2023-10-19T13:01:10.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.960+0000 7ffb297fa640 1 -- 172.21.15.57:0/588837887 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb24004020 con 0x7ffb2c15b640 2023-10-19T13:01:10.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.960+0000 7ffb32c83640 1 -- 172.21.15.57:0/588837887 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb2c0c6890 con 0x7ffb2c15b640 2023-10-19T13:01:10.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.961+0000 7ffb32c83640 1 -- 172.21.15.57:0/588837887 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb2c0d19f0 con 0x7ffb2c15b640 2023-10-19T13:01:10.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.962+0000 7ffb297fa640 1 -- 172.21.15.57:0/588837887 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 44) v1 ==== 89332+0+0 (secure 0 0 0) 0x7ffb240041a0 con 0x7ffb2c15b640 2023-10-19T13:01:10.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.962+0000 7ffb297fa640 1 --2- 172.21.15.57:0/588837887 >> [v2:172.21.15.179:6832/124009847,v1:172.21.15.179:6833/124009847] conn(0x7ffb1006af60 0x7ffb1006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:10.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.963+0000 7ffb30a1f640 1 -- 172.21.15.57:0/588837887 >> [v2:172.21.15.179:6832/124009847,v1:172.21.15.179:6833/124009847] conn(0x7ffb1006af60 msgr2=0x7ffb1006d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/124009847 2023-10-19T13:01:10.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.963+0000 7ffb297fa640 1 -- 172.21.15.57:0/588837887 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6401+0+0 (secure 0 0 0) 0x7ffb24014070 con 0x7ffb2c15b640 2023-10-19T13:01:10.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.963+0000 7ffb30a1f640 1 --2- 172.21.15.57:0/588837887 >> [v2:172.21.15.179:6832/124009847,v1:172.21.15.179:6833/124009847] conn(0x7ffb1006af60 0x7ffb1006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T13:01:10.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:10.967+0000 7ffb297fa640 1 -- 172.21.15.57:0/588837887 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ffb240557d0 con 0x7ffb2c15b640 2023-10-19T13:01:11.007 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:11.004+0000 7fb56ee2f040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-19T13:01:11.046 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:01:11.044+0000 7f14f6411040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-19T13:01:11.090 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:01:11.088+0000 7f14f6411040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-19T13:01:11.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.164+0000 7ffb30a1f640 1 -- 172.21.15.57:0/588837887 >> [v2:172.21.15.179:6832/124009847,v1:172.21.15.179:6833/124009847] conn(0x7ffb1006af60 msgr2=0x7ffb1006d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/124009847 2023-10-19T13:01:11.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.164+0000 7ffb30a1f640 1 --2- 172.21.15.57:0/588837887 >> [v2:172.21.15.179:6832/124009847,v1:172.21.15.179:6833/124009847] conn(0x7ffb1006af60 0x7ffb1006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-19T13:01:11.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.179+0000 7ffb32c83640 1 -- 172.21.15.57:0/588837887 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffb2c0675f0 con 0x7ffb2c15b640 2023-10-19T13:01:11.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.179+0000 7ffb297fa640 1 -- 172.21.15.57:0/588837887 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+4921 (secure 0 0 0) 0x7ffb24058df0 con 0x7ffb2c15b640 2023-10-19T13:01:11.183 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:01:11.183 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":37,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6305,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/2631343514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":2631343514},{"type":"v1","addr":"172.21.15.179:6837","nonce":2631343514}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":34},{"gid":6883,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/4008617188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":4008617188},{"type":"v1","addr":"172.21.15.179:6836","nonce":4008617188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":34},{"gid":7542,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/2010253133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2010253133},{"type":"v1","addr":"172.21.15.57:6834","nonce":2010253133}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T13:00:55.547801+0000","modified":"2023-10-19T13:00:57.571937+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7545},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7545":{"gid":7545,"name":"a","rank":0,"incarnation":35,"state":"up:active","state_seq":3,"addr":"172.21.15.57:6835/2583001580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2583001580},{"type":"v1","addr":"172.21.15.57:6835","nonce":2583001580}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-10-19T13:01:11.183 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 37 2023-10-19T13:01:11.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.181+0000 7ffb32c83640 1 -- 172.21.15.57:0/588837887 >> [v2:172.21.15.179:6832/124009847,v1:172.21.15.179:6833/124009847] conn(0x7ffb1006af60 msgr2=0x7ffb1006d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T13:01:11.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.181+0000 7ffb32c83640 1 --2- 172.21.15.57:0/588837887 >> [v2:172.21.15.179:6832/124009847,v1:172.21.15.179:6833/124009847] conn(0x7ffb1006af60 0x7ffb1006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:11.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.181+0000 7ffb32c83640 1 -- 172.21.15.57:0/588837887 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffb2c15b640 msgr2=0x7ffb2c0c8060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:11.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.181+0000 7ffb32c83640 1 --2- 172.21.15.57:0/588837887 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffb2c15b640 0x7ffb2c0c8060 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7ffb2400efd0 tx=0x7ffb2400c6a0 comp rx=0 tx=0).stop 2023-10-19T13:01:11.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.181+0000 7ffb32c83640 1 -- 172.21.15.57:0/588837887 shutdown_connections 2023-10-19T13:01:11.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.181+0000 7ffb32c83640 1 --2- 172.21.15.57:0/588837887 >> [v2:172.21.15.179:6832/124009847,v1:172.21.15.179:6833/124009847] conn(0x7ffb1006af60 0x7ffb1006d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:11.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.181+0000 7ffb32c83640 1 --2- 172.21.15.57:0/588837887 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffb2c15b640 0x7ffb2c0c8060 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:11.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.181+0000 7ffb32c83640 1 --2- 172.21.15.57:0/588837887 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ffb2c15b260 0x7ffb2c0c5ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:11.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.181+0000 7ffb32c83640 1 --2- 172.21.15.57:0/588837887 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffb2c0cf850 0x7ffb2c0ca910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:11.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.182+0000 7ffb32c83640 1 -- 172.21.15.57:0/588837887 >> 172.21.15.57:0/588837887 conn(0x7ffb2c09a030 msgr2=0x7ffb2c06b220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:11.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.182+0000 7ffb32c83640 1 -- 172.21.15.57:0/588837887 shutdown_connections 2023-10-19T13:01:11.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.182+0000 7ffb32c83640 1 -- 172.21.15.57:0/588837887 wait complete. 2023-10-19T13:01:11.195 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T13:01:11.202 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:01:11.200+0000 7f14f6411040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-19T13:01:11.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.303+0000 7fdcba3fc640 1 -- 172.21.15.57:0/2590273912 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdcb415b2a0 msgr2=0x7fdcb40cf720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:11.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.303+0000 7fdcba3fc640 1 --2- 172.21.15.57:0/2590273912 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdcb415b2a0 0x7fdcb40cf720 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7fdcb4089ae0 tx=0x7fdca402bd10 comp rx=0 tx=0).stop 2023-10-19T13:01:11.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.303+0000 7fdcba3fc640 1 -- 172.21.15.57:0/2590273912 shutdown_connections 2023-10-19T13:01:11.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.303+0000 7fdcba3fc640 1 --2- 172.21.15.57:0/2590273912 >> v2:172.21.15.179:3300/0 conn(0x7fdcb40ceb40 0x7fdcb40b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:11.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.303+0000 7fdcba3fc640 1 --2- 172.21.15.57:0/2590273912 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdcb415b2a0 0x7fdcb40cf720 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:11.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.303+0000 7fdcba3fc640 1 --2- 172.21.15.57:0/2590273912 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdcb40cf340 0x7fdcb40d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:11.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.303+0000 7fdcba3fc640 1 -- 172.21.15.57:0/2590273912 >> 172.21.15.57:0/2590273912 conn(0x7fdcb40d0d60 msgr2=0x7fdcb40d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:11.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.303+0000 7fdcba3fc640 1 -- 172.21.15.57:0/2590273912 shutdown_connections 2023-10-19T13:01:11.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.303+0000 7fdcba3fc640 1 -- 172.21.15.57:0/2590273912 wait complete. 2023-10-19T13:01:11.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.305+0000 7fdcba3fc640 1 Processor -- start 2023-10-19T13:01:11.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.305+0000 7fdcba3fc640 1 -- start start 2023-10-19T13:01:11.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.305+0000 7fdcba3fc640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdcb40ceb40 0x7fdcb40b5b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:11.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.305+0000 7fdcba3fc640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdcb40cf340 0x7fdcb41e5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:11.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.306+0000 7fdcba3fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdcb415b2a0 0x7fdcb40c7e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:11.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.306+0000 7fdcba3fc640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb4156930 con 0x7fdcb415b2a0 2023-10-19T13:01:11.312 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.306+0000 7fdcba3fc640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb40b4da0 con 0x7fdcb40cf340 2023-10-19T13:01:11.312 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.306+0000 7fdcba3fc640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb40b5080 con 0x7fdcb40ceb40 2023-10-19T13:01:11.312 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.306+0000 7fdcb8999640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdcb415b2a0 0x7fdcb40c7e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:11.312 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.306+0000 7fdcb8999640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdcb415b2a0 0x7fdcb40c7e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56150/0 (socket says 172.21.15.57:56150) 2023-10-19T13:01:11.312 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.306+0000 7fdcb8999640 1 -- 172.21.15.57:0/2779049833 learned_addr learned my addr 172.21.15.57:0/2779049833 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:01:11.313 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.306+0000 7fdcb37fe640 1 --2- 172.21.15.57:0/2779049833 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdcb40cf340 0x7fdcb41e5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:11.313 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.306+0000 7fdcb3fff640 1 --2- 172.21.15.57:0/2779049833 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdcb40ceb40 0x7fdcb40b5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:11.313 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.306+0000 7fdcb8999640 1 -- 172.21.15.57:0/2779049833 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdcb40ceb40 msgr2=0x7fdcb40b5b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:11.313 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.306+0000 7fdcb8999640 1 --2- 172.21.15.57:0/2779049833 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdcb40ceb40 0x7fdcb40b5b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:11.313 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.306+0000 7fdcb8999640 1 -- 172.21.15.57:0/2779049833 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdcb40cf340 msgr2=0x7fdcb41e5a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:11.314 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.306+0000 7fdcb8999640 1 --2- 172.21.15.57:0/2779049833 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdcb40cf340 0x7fdcb41e5a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:11.314 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.306+0000 7fdcb8999640 1 -- 172.21.15.57:0/2779049833 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdca4009560 con 0x7fdcb415b2a0 2023-10-19T13:01:11.314 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.306+0000 7fdcb8999640 1 --2- 172.21.15.57:0/2779049833 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdcb415b2a0 0x7fdcb40c7e30 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7fdcac00da90 tx=0x7fdcac00df50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:11.314 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.306+0000 7fdcb0ff9640 1 -- 172.21.15.57:0/2779049833 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcac00bdd0 con 0x7fdcb415b2a0 2023-10-19T13:01:11.314 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.306+0000 7fdcb0ff9640 1 -- 172.21.15.57:0/2779049833 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fdcac003970 con 0x7fdcb415b2a0 2023-10-19T13:01:11.315 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.306+0000 7fdcba3fc640 1 -- 172.21.15.57:0/2779049833 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcb40b5360 con 0x7fdcb415b2a0 2023-10-19T13:01:11.315 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.306+0000 7fdcb0ff9640 1 -- 172.21.15.57:0/2779049833 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcac003d40 con 0x7fdcb415b2a0 2023-10-19T13:01:11.315 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.307+0000 7fdcba3fc640 1 -- 172.21.15.57:0/2779049833 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcb41e6570 con 0x7fdcb415b2a0 2023-10-19T13:01:11.315 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.307+0000 7fdcba3fc640 1 -- 172.21.15.57:0/2779049833 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcb40b5880 con 0x7fdcb415b2a0 2023-10-19T13:01:11.316 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.308+0000 7fdcb0ff9640 1 -- 172.21.15.57:0/2779049833 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 44) v1 ==== 89332+0+0 (secure 0 0 0) 0x7fdcac003ea0 con 0x7fdcb415b2a0 2023-10-19T13:01:11.316 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.308+0000 7fdcb0ff9640 1 --2- 172.21.15.57:0/2779049833 >> [v2:172.21.15.179:6832/124009847,v1:172.21.15.179:6833/124009847] conn(0x7fdc9406af60 0x7fdc9406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:11.317 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.309+0000 7fdcb3fff640 1 -- 172.21.15.57:0/2779049833 >> [v2:172.21.15.179:6832/124009847,v1:172.21.15.179:6833/124009847] conn(0x7fdc9406af60 msgr2=0x7fdc9406d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/124009847 2023-10-19T13:01:11.317 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.309+0000 7fdcb0ff9640 1 -- 172.21.15.57:0/2779049833 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6401+0+0 (secure 0 0 0) 0x7fdcac089fc0 con 0x7fdcb415b2a0 2023-10-19T13:01:11.317 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.309+0000 7fdcb3fff640 1 --2- 172.21.15.57:0/2779049833 >> [v2:172.21.15.179:6832/124009847,v1:172.21.15.179:6833/124009847] conn(0x7fdc9406af60 0x7fdc9406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T13:01:11.317 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.313+0000 7fdcb0ff9640 1 -- 172.21.15.57:0/2779049833 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fdcb40b5880 con 0x7fdcb415b2a0 2023-10-19T13:01:11.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.491+0000 7fdcba3fc640 1 -- 172.21.15.57:0/2779049833 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdcb40b5880 con 0x7fdcb415b2a0 2023-10-19T13:01:11.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.492+0000 7fdcb0ff9640 1 -- 172.21.15.57:0/2779049833 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v101) v1 ==== 74+0+17343 (secure 0 0 0) 0x7fdcb40b5880 con 0x7fdcb415b2a0 2023-10-19T13:01:11.495 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:01:11.496 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":101,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T13:01:06.603844+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2023-10-19T13:00:53.446697+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2023-10-19T13:00:54.540962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.179:6833/1413162726":"2023-10-20T13:01:06.603811+0000","172.21.15.179:6832/1413162726":"2023-10-20T13:01:06.603811+0000","172.21.15.179:0/3401895301":"2023-10-20T13:01:06.603811+0000","172.21.15.179:0/2499848570":"2023-10-20T13:01:06.603811+0000","172.21.15.179:0/3622569807":"2023-10-20T13:01:06.603811+0000","172.21.15.179:0/2835518029":"2023-10-20T13:01:06.603811+0000","172.21.15.179:0/1643279404":"2023-10-20T13:01:06.603811+0000","172.21.15.179:0/4219879451":"2023-10-20T13:01:06.603811+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T13:01:11.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.494+0000 7fdcba3fc640 1 -- 172.21.15.57:0/2779049833 >> [v2:172.21.15.179:6832/124009847,v1:172.21.15.179:6833/124009847] conn(0x7fdc9406af60 msgr2=0x7fdc9406d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T13:01:11.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.494+0000 7fdcba3fc640 1 --2- 172.21.15.57:0/2779049833 >> [v2:172.21.15.179:6832/124009847,v1:172.21.15.179:6833/124009847] conn(0x7fdc9406af60 0x7fdc9406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:11.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.494+0000 7fdcba3fc640 1 -- 172.21.15.57:0/2779049833 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdcb415b2a0 msgr2=0x7fdcb40c7e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:11.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.494+0000 7fdcba3fc640 1 --2- 172.21.15.57:0/2779049833 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdcb415b2a0 0x7fdcb40c7e30 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7fdcac00da90 tx=0x7fdcac00df50 comp rx=0 tx=0).stop 2023-10-19T13:01:11.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.495+0000 7fdcba3fc640 1 -- 172.21.15.57:0/2779049833 shutdown_connections 2023-10-19T13:01:11.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.495+0000 7fdcba3fc640 1 --2- 172.21.15.57:0/2779049833 >> [v2:172.21.15.179:6832/124009847,v1:172.21.15.179:6833/124009847] conn(0x7fdc9406af60 0x7fdc9406d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:11.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.495+0000 7fdcba3fc640 1 --2- 172.21.15.57:0/2779049833 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdcb415b2a0 0x7fdcb40c7e30 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:11.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.495+0000 7fdcba3fc640 1 --2- 172.21.15.57:0/2779049833 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdcb40cf340 0x7fdcb41e5a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:11.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.495+0000 7fdcba3fc640 1 --2- 172.21.15.57:0/2779049833 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdcb40ceb40 0x7fdcb40b5b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:11.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.495+0000 7fdcba3fc640 1 -- 172.21.15.57:0/2779049833 >> 172.21.15.57:0/2779049833 conn(0x7fdcb40d0d60 msgr2=0x7fdcb40b0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:11.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.495+0000 7fdcba3fc640 1 -- 172.21.15.57:0/2779049833 shutdown_connections 2023-10-19T13:01:11.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.495+0000 7fdcba3fc640 1 -- 172.21.15.57:0/2779049833 wait complete. 2023-10-19T13:01:11.508 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-19T12:44:07.371186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-19T13:00:53.446697+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '99', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_data', 'create_time': '2023-10-19T13:00:54.540962+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '100', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-19T13:01:11.509 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-19T13:01:11.627 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:11.625+0000 7fb56ee2f040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-19T13:01:11.664 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.662+0000 7fb2c54d7640 1 -- 172.21.15.57:0/2781123592 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb2c01416f0 msgr2=0x7fb2c00b1440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:11.665 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.662+0000 7fb2c54d7640 1 --2- 172.21.15.57:0/2781123592 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb2c01416f0 0x7fb2c00b1440 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fb2b0009830 tx=0x7fb2b002bcd0 comp rx=0 tx=0).stop 2023-10-19T13:01:11.665 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.662+0000 7fb2c54d7640 1 -- 172.21.15.57:0/2781123592 shutdown_connections 2023-10-19T13:01:11.665 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.662+0000 7fb2c54d7640 1 --2- 172.21.15.57:0/2781123592 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb2c01416f0 0x7fb2c00b1440 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:11.665 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.662+0000 7fb2c54d7640 1 -- 172.21.15.57:0/2781123592 >> 172.21.15.57:0/2781123592 conn(0x7fb2c00af730 msgr2=0x7fb2c00afb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:11.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.662+0000 7fb2c54d7640 1 -- 172.21.15.57:0/2781123592 shutdown_connections 2023-10-19T13:01:11.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.662+0000 7fb2c54d7640 1 -- 172.21.15.57:0/2781123592 wait complete. 2023-10-19T13:01:11.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.663+0000 7fb2c54d7640 1 Processor -- start 2023-10-19T13:01:11.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.663+0000 7fb2c54d7640 1 -- start start 2023-10-19T13:01:11.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.664+0000 7fb2c54d7640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb2c0070790 0x7fb2c006cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:11.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.664+0000 7fb2c54d7640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb2c00b1980 0x7fb2c013c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:11.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.664+0000 7fb2c54d7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb2c01416f0 0x7fb2c013cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:11.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.664+0000 7fb2c54d7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2c0137c70 con 0x7fb2c01416f0 2023-10-19T13:01:11.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.664+0000 7fb2c54d7640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2c006c020 con 0x7fb2c0070790 2023-10-19T13:01:11.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.664+0000 7fb2c54d7640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2c01f5c30 con 0x7fb2c00b1980 2023-10-19T13:01:11.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.664+0000 7fb2bf7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb2c01416f0 0x7fb2c013cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:11.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.664+0000 7fb2bf7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb2c01416f0 0x7fb2c013cb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56160/0 (socket says 172.21.15.57:56160) 2023-10-19T13:01:11.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.664+0000 7fb2beffd640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb2c0070790 0x7fb2c006cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:11.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.664+0000 7fb2bf7fe640 1 -- 172.21.15.57:0/1091202757 learned_addr learned my addr 172.21.15.57:0/1091202757 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:01:11.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.664+0000 7fb2b7fff640 1 --2- 172.21.15.57:0/1091202757 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb2c00b1980 0x7fb2c013c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:11.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.664+0000 7fb2bf7fe640 1 -- 172.21.15.57:0/1091202757 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb2c00b1980 msgr2=0x7fb2c013c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:11.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.664+0000 7fb2bf7fe640 1 --2- 172.21.15.57:0/1091202757 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb2c00b1980 0x7fb2c013c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:11.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.664+0000 7fb2bf7fe640 1 -- 172.21.15.57:0/1091202757 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb2c0070790 msgr2=0x7fb2c006cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:11.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.664+0000 7fb2bf7fe640 1 --2- 172.21.15.57:0/1091202757 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb2c0070790 0x7fb2c006cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:11.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.664+0000 7fb2bf7fe640 1 -- 172.21.15.57:0/1091202757 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2b0009490 con 0x7fb2c01416f0 2023-10-19T13:01:11.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.664+0000 7fb2b7fff640 1 --2- 172.21.15.57:0/1091202757 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb2c00b1980 0x7fb2c013c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T13:01:11.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.664+0000 7fb2beffd640 1 --2- 172.21.15.57:0/1091202757 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb2c0070790 0x7fb2c006cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T13:01:11.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.664+0000 7fb2bf7fe640 1 --2- 172.21.15.57:0/1091202757 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb2c01416f0 0x7fb2c013cb30 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7fb2b800cc70 tx=0x7fb2b800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:11.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.665+0000 7fb2b77fe640 1 -- 172.21.15.57:0/1091202757 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2b800ce80 con 0x7fb2c01416f0 2023-10-19T13:01:11.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.665+0000 7fb2b77fe640 1 -- 172.21.15.57:0/1091202757 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fb2b8012420 con 0x7fb2c01416f0 2023-10-19T13:01:11.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.665+0000 7fb2c54d7640 1 -- 172.21.15.57:0/1091202757 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2c006c230 con 0x7fb2c01416f0 2023-10-19T13:01:11.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.665+0000 7fb2b77fe640 1 -- 172.21.15.57:0/1091202757 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2b8012850 con 0x7fb2c01416f0 2023-10-19T13:01:11.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.665+0000 7fb2c54d7640 1 -- 172.21.15.57:0/1091202757 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2c006c7f0 con 0x7fb2c01416f0 2023-10-19T13:01:11.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.666+0000 7fb2c54d7640 1 -- 172.21.15.57:0/1091202757 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2c006ca10 con 0x7fb2c01416f0 2023-10-19T13:01:11.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.666+0000 7fb2b77fe640 1 -- 172.21.15.57:0/1091202757 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 44) v1 ==== 89332+0+0 (secure 0 0 0) 0x7fb2b80129b0 con 0x7fb2c01416f0 2023-10-19T13:01:11.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.667+0000 7fb2b77fe640 1 --2- 172.21.15.57:0/1091202757 >> [v2:172.21.15.179:6832/124009847,v1:172.21.15.179:6833/124009847] conn(0x7fb2a006af60 0x7fb2a006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:11.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.667+0000 7fb2b77fe640 1 -- 172.21.15.57:0/1091202757 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6401+0+0 (secure 0 0 0) 0x7fb2b808a2a0 con 0x7fb2c01416f0 2023-10-19T13:01:11.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.667+0000 7fb2beffd640 1 -- 172.21.15.57:0/1091202757 >> [v2:172.21.15.179:6832/124009847,v1:172.21.15.179:6833/124009847] conn(0x7fb2a006af60 msgr2=0x7fb2a006d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/124009847 2023-10-19T13:01:11.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.667+0000 7fb2beffd640 1 --2- 172.21.15.57:0/1091202757 >> [v2:172.21.15.179:6832/124009847,v1:172.21.15.179:6833/124009847] conn(0x7fb2a006af60 0x7fb2a006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T13:01:11.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.671+0000 7fb2b77fe640 1 -- 172.21.15.57:0/1091202757 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb2b8054fb0 con 0x7fb2c01416f0 2023-10-19T13:01:11.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.847+0000 7fb2c54d7640 1 -- 172.21.15.57:0/1091202757 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb2c00675f0 con 0x7fb2c01416f0 2023-10-19T13:01:11.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.848+0000 7fb2b77fe640 1 -- 172.21.15.57:0/1091202757 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v101) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb2b80585d0 con 0x7fb2c01416f0 2023-10-19T13:01:11.851 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-19T13:01:11.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.850+0000 7fb2c54d7640 1 -- 172.21.15.57:0/1091202757 >> [v2:172.21.15.179:6832/124009847,v1:172.21.15.179:6833/124009847] conn(0x7fb2a006af60 msgr2=0x7fb2a006d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T13:01:11.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.850+0000 7fb2c54d7640 1 --2- 172.21.15.57:0/1091202757 >> [v2:172.21.15.179:6832/124009847,v1:172.21.15.179:6833/124009847] conn(0x7fb2a006af60 0x7fb2a006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:11.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.850+0000 7fb2c54d7640 1 -- 172.21.15.57:0/1091202757 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb2c01416f0 msgr2=0x7fb2c013cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:11.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.850+0000 7fb2c54d7640 1 --2- 172.21.15.57:0/1091202757 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb2c01416f0 0x7fb2c013cb30 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7fb2b800cc70 tx=0x7fb2b800a430 comp rx=0 tx=0).stop 2023-10-19T13:01:11.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.850+0000 7fb2c54d7640 1 -- 172.21.15.57:0/1091202757 shutdown_connections 2023-10-19T13:01:11.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.850+0000 7fb2c54d7640 1 --2- 172.21.15.57:0/1091202757 >> [v2:172.21.15.179:6832/124009847,v1:172.21.15.179:6833/124009847] conn(0x7fb2a006af60 0x7fb2a006d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:11.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.850+0000 7fb2c54d7640 1 --2- 172.21.15.57:0/1091202757 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb2c01416f0 0x7fb2c013cb30 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:11.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.850+0000 7fb2c54d7640 1 --2- 172.21.15.57:0/1091202757 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb2c00b1980 0x7fb2c013c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:11.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.850+0000 7fb2c54d7640 1 --2- 172.21.15.57:0/1091202757 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb2c0070790 0x7fb2c006cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:11.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.850+0000 7fb2c54d7640 1 -- 172.21.15.57:0/1091202757 >> 172.21.15.57:0/1091202757 conn(0x7fb2c00af730 msgr2=0x7fb2c00ade90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:11.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.850+0000 7fb2c54d7640 1 -- 172.21.15.57:0/1091202757 shutdown_connections 2023-10-19T13:01:11.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.850+0000 7fb2c54d7640 1 -- 172.21.15.57:0/1091202757 wait complete. 2023-10-19T13:01:11.864 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-19T13:01:11.874 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:01:11.871+0000 7f14f6411040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-19T13:01:11.891 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:11.889+0000 7fb56ee2f040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-19T13:01:11.939 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:11.937+0000 7fb56ee2f040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-19T13:01:11.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.976+0000 7f2b42bd0640 1 -- 172.21.15.57:0/573901018 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2b2c004d00 msgr2=0x7f2b2c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:11.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.976+0000 7f2b42bd0640 1 --2- 172.21.15.57:0/573901018 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2b2c004d00 0x7f2b2c005180 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f2b34009ed0 tx=0x7f2b2c0057f0 comp rx=0 tx=0).stop 2023-10-19T13:01:11.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.976+0000 7f2b4116d640 1 -- 172.21.15.57:0/573901018 reap_dead start 2023-10-19T13:01:11.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.976+0000 7f2b42bd0640 1 -- 172.21.15.57:0/573901018 shutdown_connections 2023-10-19T13:01:11.979 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.976+0000 7f2b42bd0640 1 -- 172.21.15.57:0/573901018 >> 172.21.15.57:0/573901018 conn(0x7f2b3c0d0d60 msgr2=0x7f2b3c0d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:11.979 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.976+0000 7f2b42bd0640 1 -- 172.21.15.57:0/573901018 shutdown_connections 2023-10-19T13:01:11.979 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.976+0000 7f2b42bd0640 1 -- 172.21.15.57:0/573901018 wait complete. 2023-10-19T13:01:11.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.977+0000 7f2b42bd0640 1 Processor -- start 2023-10-19T13:01:11.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.977+0000 7f2b42bd0640 1 -- start start 2023-10-19T13:01:11.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.978+0000 7f2b42bd0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2b3c06a870 0x7f2b3c1f1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:11.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.978+0000 7f2b42bd0640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2b3c06ac50 0x7f2b3c1f2940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:11.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.978+0000 7f2b42bd0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2b3c15e200 0x7f2b3c1f2e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:11.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.978+0000 7f2b42bd0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b3c1f1e70 con 0x7f2b3c15e200 2023-10-19T13:01:11.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.978+0000 7f2b42bd0640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b3c0cb060 con 0x7f2b3c06a870 2023-10-19T13:01:11.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.978+0000 7f2b42bd0640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b3c1f20a0 con 0x7f2b3c06ac50 2023-10-19T13:01:11.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.978+0000 7f2b4116d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2b3c15e200 0x7f2b3c1f2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:11.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.978+0000 7f2b4116d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2b3c15e200 0x7f2b3c1f2e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56184/0 (socket says 172.21.15.57:56184) 2023-10-19T13:01:11.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.978+0000 7f2b4116d640 1 -- 172.21.15.57:0/220625863 learned_addr learned my addr 172.21.15.57:0/220625863 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:01:11.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.978+0000 7f2b4096c640 1 --2- 172.21.15.57:0/220625863 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2b3c06a870 0x7f2b3c1f1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:11.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.978+0000 7f2b3bfff640 1 --2- 172.21.15.57:0/220625863 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2b3c06ac50 0x7f2b3c1f2940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:11.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.978+0000 7f2b4116d640 1 -- 172.21.15.57:0/220625863 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2b3c06ac50 msgr2=0x7f2b3c1f2940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:11.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.978+0000 7f2b4116d640 1 --2- 172.21.15.57:0/220625863 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2b3c06ac50 0x7f2b3c1f2940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:11.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.978+0000 7f2b4116d640 1 -- 172.21.15.57:0/220625863 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2b3c06a870 msgr2=0x7f2b3c1f1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:11.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.979+0000 7f2b4116d640 1 --2- 172.21.15.57:0/220625863 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2b3c06a870 0x7f2b3c1f1930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:11.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.979+0000 7f2b4116d640 1 -- 172.21.15.57:0/220625863 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b3c1f2380 con 0x7f2b3c15e200 2023-10-19T13:01:11.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.979+0000 7f2b4116d640 1 --2- 172.21.15.57:0/220625863 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2b3c15e200 0x7f2b3c1f2e80 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f2b34009ed0 tx=0x7f2b34031750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:11.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.979+0000 7f2b397fa640 1 -- 172.21.15.57:0/220625863 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b34040070 con 0x7f2b3c15e200 2023-10-19T13:01:11.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.979+0000 7f2b42bd0640 1 -- 172.21.15.57:0/220625863 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b3c1f2660 con 0x7f2b3c15e200 2023-10-19T13:01:11.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.979+0000 7f2b397fa640 1 -- 172.21.15.57:0/220625863 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f2b34009200 con 0x7f2b3c15e200 2023-10-19T13:01:11.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.979+0000 7f2b397fa640 1 -- 172.21.15.57:0/220625863 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b34003bb0 con 0x7f2b3c15e200 2023-10-19T13:01:11.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.979+0000 7f2b42bd0640 1 -- 172.21.15.57:0/220625863 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b3c0c84f0 con 0x7f2b3c15e200 2023-10-19T13:01:11.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.980+0000 7f2b42bd0640 1 -- 172.21.15.57:0/220625863 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b3c0c7e30 con 0x7f2b3c15e200 2023-10-19T13:01:11.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.980+0000 7f2b397fa640 1 -- 172.21.15.57:0/220625863 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 44) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f2b34003d10 con 0x7f2b3c15e200 2023-10-19T13:01:11.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.981+0000 7f2b397fa640 1 --2- 172.21.15.57:0/220625863 >> [v2:172.21.15.179:6832/124009847,v1:172.21.15.179:6833/124009847] conn(0x7f2b2006af60 0x7f2b2006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:11.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.981+0000 7f2b4096c640 1 -- 172.21.15.57:0/220625863 >> [v2:172.21.15.179:6832/124009847,v1:172.21.15.179:6833/124009847] conn(0x7f2b2006af60 msgr2=0x7f2b2006d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/124009847 2023-10-19T13:01:11.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.981+0000 7f2b397fa640 1 -- 172.21.15.57:0/220625863 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f2b340ac1a0 con 0x7f2b3c15e200 2023-10-19T13:01:11.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.981+0000 7f2b4096c640 1 --2- 172.21.15.57:0/220625863 >> [v2:172.21.15.179:6832/124009847,v1:172.21.15.179:6833/124009847] conn(0x7f2b2006af60 0x7f2b2006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T13:01:11.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:11.985+0000 7f2b397fa640 1 -- 172.21.15.57:0/220625863 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2b3c0c7e30 con 0x7f2b3c15e200 2023-10-19T13:01:12.040 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:12.038+0000 7fb56ee2f040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-19T13:01:12.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.062+0000 7f2b397fa640 1 -- 172.21.15.57:0/220625863 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mgrmap(e 45) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f2b340aa400 con 0x7f2b3c15e200 2023-10-19T13:01:12.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.063+0000 7f2b397fa640 1 -- 172.21.15.57:0/220625863 >> [v2:172.21.15.179:6832/124009847,v1:172.21.15.179:6833/124009847] conn(0x7f2b2006af60 msgr2=0x7f2b2006d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T13:01:12.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.063+0000 7f2b397fa640 1 --2- 172.21.15.57:0/220625863 >> [v2:172.21.15.179:6832/124009847,v1:172.21.15.179:6833/124009847] conn(0x7f2b2006af60 0x7f2b2006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:12.116 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:12.113+0000 7fb4f08b4640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:01:12.116 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:12.113+0000 7fb4f08b4640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:01:12.116 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:12.113+0000 7fb4f08b4640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:01:12.117 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:12.113+0000 7fb4f08b4640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:01:12.117 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:12.113+0000 7fb4f08b4640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:01:12.118 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:12.114+0000 7fb4f38ba640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:01:12.118 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:12.114+0000 7fb4f38ba640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:01:12.118 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:12.114+0000 7fb4f38ba640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:01:12.118 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:12.114+0000 7fb4f38ba640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:01:12.118 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:12.114+0000 7fb4f38ba640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:01:12.159 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:01:12.157+0000 7f14f6411040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-19T13:01:12.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.168+0000 7f2b42bd0640 1 -- 172.21.15.57:0/220625863 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2b3c1f1e70 con 0x7f2b3c15e200 2023-10-19T13:01:12.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.168+0000 7f2b397fa640 1 -- 172.21.15.57:0/220625863 <== mon.0 v2:172.21.15.57:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v102) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2b34097000 con 0x7f2b3c15e200 2023-10-19T13:01:12.170 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2023-10-19T13:01:12.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.170+0000 7f2b42bd0640 1 -- 172.21.15.57:0/220625863 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2b3c15e200 msgr2=0x7f2b3c1f2e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:12.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.170+0000 7f2b42bd0640 1 --2- 172.21.15.57:0/220625863 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2b3c15e200 0x7f2b3c1f2e80 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f2b34009ed0 tx=0x7f2b34031750 comp rx=0 tx=0).stop 2023-10-19T13:01:12.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.170+0000 7f2b42bd0640 1 -- 172.21.15.57:0/220625863 shutdown_connections 2023-10-19T13:01:12.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.170+0000 7f2b42bd0640 1 --2- 172.21.15.57:0/220625863 >> [v2:172.21.15.179:6832/124009847,v1:172.21.15.179:6833/124009847] conn(0x7f2b2006af60 0x7f2b2006d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:12.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.170+0000 7f2b42bd0640 1 --2- 172.21.15.57:0/220625863 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2b3c15e200 0x7f2b3c1f2e80 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:12.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.170+0000 7f2b42bd0640 1 --2- 172.21.15.57:0/220625863 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2b3c06ac50 0x7f2b3c1f2940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:12.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.170+0000 7f2b42bd0640 1 --2- 172.21.15.57:0/220625863 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2b3c06a870 0x7f2b3c1f1930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:12.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.170+0000 7f2b42bd0640 1 -- 172.21.15.57:0/220625863 >> 172.21.15.57:0/220625863 conn(0x7f2b3c0d0d60 msgr2=0x7f2b3c0cf720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:12.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.170+0000 7f2b42bd0640 1 -- 172.21.15.57:0/220625863 shutdown_connections 2023-10-19T13:01:12.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.170+0000 7f2b42bd0640 1 -- 172.21.15.57:0/220625863 wait complete. 2023-10-19T13:01:12.183 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-19T13:01:12.211 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:01:12.210+0000 7f14f6411040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-19T13:01:12.322 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:01:12.320+0000 7f14f6411040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-19T13:01:12.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.323+0000 7f7a55487640 1 -- 172.21.15.57:0/1038839105 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7a500cf340 msgr2=0x7f7a500d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:12.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.323+0000 7f7a55487640 1 --2- 172.21.15.57:0/1038839105 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7a500cf340 0x7f7a500d1910 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f7a5015b680 tx=0x7f7a4002bd30 comp rx=0 tx=0).stop 2023-10-19T13:01:12.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.323+0000 7f7a55487640 1 -- 172.21.15.57:0/1038839105 shutdown_connections 2023-10-19T13:01:12.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.323+0000 7f7a55487640 1 --2- 172.21.15.57:0/1038839105 >> v2:172.21.15.179:3300/0 conn(0x7f7a500ceb40 0x7f7a500b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:12.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.323+0000 7f7a55487640 1 --2- 172.21.15.57:0/1038839105 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7a500cf340 0x7f7a500d1910 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:12.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.323+0000 7f7a55487640 1 -- 172.21.15.57:0/1038839105 >> 172.21.15.57:0/1038839105 conn(0x7f7a500d0d60 msgr2=0x7f7a500d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:12.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.323+0000 7f7a55487640 1 -- 172.21.15.57:0/1038839105 shutdown_connections 2023-10-19T13:01:12.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.323+0000 7f7a55487640 1 -- 172.21.15.57:0/1038839105 wait complete. 2023-10-19T13:01:12.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.324+0000 7f7a55487640 1 Processor -- start 2023-10-19T13:01:12.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.325+0000 7f7a55487640 1 -- start start 2023-10-19T13:01:12.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.325+0000 7f7a55487640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7a500ceb40 0x7f7a500caac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:12.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.325+0000 7f7a55487640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7a500cf340 0x7f7a501f1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:12.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.325+0000 7f7a55487640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7a5015b2a0 0x7f7a5015c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:12.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.325+0000 7f7a55487640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a500b54c0 con 0x7f7a5015b2a0 2023-10-19T13:01:12.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.325+0000 7f7a55487640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a500b4b20 con 0x7f7a500ceb40 2023-10-19T13:01:12.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.325+0000 7f7a55487640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a500b4e00 con 0x7f7a500cf340 2023-10-19T13:01:12.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.325+0000 7f7a4f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7a5015b2a0 0x7f7a5015c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:12.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.325+0000 7f7a4f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7a5015b2a0 0x7f7a5015c0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56204/0 (socket says 172.21.15.57:56204) 2023-10-19T13:01:12.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.325+0000 7f7a4f7fe640 1 -- 172.21.15.57:0/2492383067 learned_addr learned my addr 172.21.15.57:0/2492383067 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:01:12.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.325+0000 7f7a47fff640 1 --2- 172.21.15.57:0/2492383067 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7a500cf340 0x7f7a501f1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:12.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.325+0000 7f7a4effd640 1 --2- 172.21.15.57:0/2492383067 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7a500ceb40 0x7f7a500caac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:12.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.326+0000 7f7a4f7fe640 1 -- 172.21.15.57:0/2492383067 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7a500cf340 msgr2=0x7f7a501f1ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:12.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.326+0000 7f7a4f7fe640 1 --2- 172.21.15.57:0/2492383067 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7a500cf340 0x7f7a501f1ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:12.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.326+0000 7f7a4f7fe640 1 -- 172.21.15.57:0/2492383067 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7a500ceb40 msgr2=0x7f7a500caac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:12.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.326+0000 7f7a4f7fe640 1 --2- 172.21.15.57:0/2492383067 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7a500ceb40 0x7f7a500caac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:12.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.326+0000 7f7a4f7fe640 1 -- 172.21.15.57:0/2492383067 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a40009560 con 0x7f7a5015b2a0 2023-10-19T13:01:12.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.326+0000 7f7a4f7fe640 1 --2- 172.21.15.57:0/2492383067 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7a5015b2a0 0x7f7a5015c0e0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f7a4800cad0 tx=0x7f7a4800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:12.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.326+0000 7f7a477fe640 1 -- 172.21.15.57:0/2492383067 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a4800ae20 con 0x7f7a5015b2a0 2023-10-19T13:01:12.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.326+0000 7f7a477fe640 1 -- 172.21.15.57:0/2492383067 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f7a48004b20 con 0x7f7a5015b2a0 2023-10-19T13:01:12.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.326+0000 7f7a55487640 1 -- 172.21.15.57:0/2492383067 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a500b50e0 con 0x7f7a5015b2a0 2023-10-19T13:01:12.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.326+0000 7f7a477fe640 1 -- 172.21.15.57:0/2492383067 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a48005020 con 0x7f7a5015b2a0 2023-10-19T13:01:12.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.326+0000 7f7a55487640 1 -- 172.21.15.57:0/2492383067 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a5015c620 con 0x7f7a5015b2a0 2023-10-19T13:01:12.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.327+0000 7f7a55487640 1 -- 172.21.15.57:0/2492383067 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a500b5600 con 0x7f7a5015b2a0 2023-10-19T13:01:12.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.328+0000 7f7a477fe640 1 -- 172.21.15.57:0/2492383067 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 45) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f7a48005180 con 0x7f7a5015b2a0 2023-10-19T13:01:12.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.328+0000 7f7a477fe640 1 -- 172.21.15.57:0/2492383067 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 6745+0+0 (secure 0 0 0) 0x7f7a48089cc0 con 0x7f7a5015b2a0 2023-10-19T13:01:12.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.332+0000 7f7a477fe640 1 -- 172.21.15.57:0/2492383067 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7a48054e60 con 0x7f7a5015b2a0 2023-10-19T13:01:12.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.512+0000 7f7a55487640 1 -- 172.21.15.57:0/2492383067 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7a500b0d00 con 0x7f7a5015b2a0 2023-10-19T13:01:12.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.512+0000 7f7a477fe640 1 -- 172.21.15.57:0/2492383067 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v102) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7a4801d070 con 0x7f7a5015b2a0 2023-10-19T13:01:12.515 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 64 2023-10-19T13:01:12.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.514+0000 7f7a55487640 1 -- 172.21.15.57:0/2492383067 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7a5015b2a0 msgr2=0x7f7a5015c0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:12.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.514+0000 7f7a55487640 1 --2- 172.21.15.57:0/2492383067 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7a5015b2a0 0x7f7a5015c0e0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f7a4800cad0 tx=0x7f7a4800cf90 comp rx=0 tx=0).stop 2023-10-19T13:01:12.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.514+0000 7f7a55487640 1 -- 172.21.15.57:0/2492383067 shutdown_connections 2023-10-19T13:01:12.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.514+0000 7f7a55487640 1 --2- 172.21.15.57:0/2492383067 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7a5015b2a0 0x7f7a5015c0e0 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:12.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.514+0000 7f7a55487640 1 --2- 172.21.15.57:0/2492383067 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7a500cf340 0x7f7a501f1ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:12.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.514+0000 7f7a55487640 1 --2- 172.21.15.57:0/2492383067 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7a500ceb40 0x7f7a500caac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:12.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.514+0000 7f7a55487640 1 -- 172.21.15.57:0/2492383067 >> 172.21.15.57:0/2492383067 conn(0x7f7a500d0d60 msgr2=0x7f7a5006a930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:12.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.514+0000 7f7a55487640 1 -- 172.21.15.57:0/2492383067 shutdown_connections 2023-10-19T13:01:12.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.514+0000 7f7a55487640 1 -- 172.21.15.57:0/2492383067 wait complete. 2023-10-19T13:01:12.527 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T13:01:12.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.634+0000 7f2e75ef2640 1 -- 172.21.15.57:0/839169995 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2e700ad370 msgr2=0x7f2e700ad800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:12.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.634+0000 7f2e75ef2640 1 --2- 172.21.15.57:0/839169995 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2e700ad370 0x7f2e700ad800 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f2e7006fc50 tx=0x7f2e6802c2f0 comp rx=0 tx=0).stop 2023-10-19T13:01:12.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.634+0000 7f2e75ef2640 1 -- 172.21.15.57:0/839169995 shutdown_connections 2023-10-19T13:01:12.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.634+0000 7f2e75ef2640 1 --2- 172.21.15.57:0/839169995 >> v2:172.21.15.179:3300/0 conn(0x7f2e7006f0a0 0x7f2e7006ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:12.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.634+0000 7f2e75ef2640 1 --2- 172.21.15.57:0/839169995 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2e700ad370 0x7f2e700ad800 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:12.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.634+0000 7f2e75ef2640 1 -- 172.21.15.57:0/839169995 >> 172.21.15.57:0/839169995 conn(0x7f2e700b0670 msgr2=0x7f2e700b0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:12.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.634+0000 7f2e75ef2640 1 -- 172.21.15.57:0/839169995 shutdown_connections 2023-10-19T13:01:12.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.634+0000 7f2e75ef2640 1 -- 172.21.15.57:0/839169995 wait complete. 2023-10-19T13:01:12.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.635+0000 7f2e75ef2640 1 Processor -- start 2023-10-19T13:01:12.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.635+0000 7f2e75ef2640 1 -- start start 2023-10-19T13:01:12.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.636+0000 7f2e75ef2640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2e7006d2d0 0x7f2e7014ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:12.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.636+0000 7f2e75ef2640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2e7006f0a0 0x7f2e700b31c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:12.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.636+0000 7f2e75ef2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2e700ad370 0x7f2e700b3700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:12.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.636+0000 7f2e75ef2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e70150c40 con 0x7f2e700ad370 2023-10-19T13:01:12.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.636+0000 7f2e75ef2640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e70150520 con 0x7f2e7006d2d0 2023-10-19T13:01:12.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.636+0000 7f2e75ef2640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e700b2e40 con 0x7f2e7006f0a0 2023-10-19T13:01:12.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.636+0000 7f2e6ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2e700ad370 0x7f2e700b3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:12.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.636+0000 7f2e6ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2e700ad370 0x7f2e700b3700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56228/0 (socket says 172.21.15.57:56228) 2023-10-19T13:01:12.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.636+0000 7f2e6ffff640 1 -- 172.21.15.57:0/1964405281 learned_addr learned my addr 172.21.15.57:0/1964405281 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:01:12.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.636+0000 7f2e6effd640 1 --2- 172.21.15.57:0/1964405281 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2e7006f0a0 0x7f2e700b31c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:12.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.636+0000 7f2e6f7fe640 1 --2- 172.21.15.57:0/1964405281 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2e7006d2d0 0x7f2e7014ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:12.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.636+0000 7f2e6ffff640 1 -- 172.21.15.57:0/1964405281 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2e7006f0a0 msgr2=0x7f2e700b31c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:12.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.636+0000 7f2e6ffff640 1 --2- 172.21.15.57:0/1964405281 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2e7006f0a0 0x7f2e700b31c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:12.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.636+0000 7f2e6ffff640 1 -- 172.21.15.57:0/1964405281 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2e7006d2d0 msgr2=0x7f2e7014ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:12.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.636+0000 7f2e6ffff640 1 --2- 172.21.15.57:0/1964405281 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2e7006d2d0 0x7f2e7014ffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:12.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.636+0000 7f2e6ffff640 1 -- 172.21.15.57:0/1964405281 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e68009900 con 0x7f2e700ad370 2023-10-19T13:01:12.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.637+0000 7f2e6ffff640 1 --2- 172.21.15.57:0/1964405281 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2e700ad370 0x7f2e700b3700 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f2e68009d60 tx=0x7f2e68035e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:12.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.637+0000 7f2e57fff640 1 -- 172.21.15.57:0/1964405281 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e6802cea0 con 0x7f2e700ad370 2023-10-19T13:01:12.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.637+0000 7f2e57fff640 1 -- 172.21.15.57:0/1964405281 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f2e68004250 con 0x7f2e700ad370 2023-10-19T13:01:12.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.637+0000 7f2e75ef2640 1 -- 172.21.15.57:0/1964405281 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e700b23b0 con 0x7f2e700ad370 2023-10-19T13:01:12.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.637+0000 7f2e57fff640 1 -- 172.21.15.57:0/1964405281 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e68004640 con 0x7f2e700ad370 2023-10-19T13:01:12.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.637+0000 7f2e75ef2640 1 -- 172.21.15.57:0/1964405281 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e700c9620 con 0x7f2e700ad370 2023-10-19T13:01:12.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.638+0000 7f2e75ef2640 1 -- 172.21.15.57:0/1964405281 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e700ce1d0 con 0x7f2e700ad370 2023-10-19T13:01:12.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.638+0000 7f2e57fff640 1 -- 172.21.15.57:0/1964405281 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 45) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f2e6803b040 con 0x7f2e700ad370 2023-10-19T13:01:12.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.639+0000 7f2e57fff640 1 -- 172.21.15.57:0/1964405281 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 6745+0+0 (secure 0 0 0) 0x7f2e680abf20 con 0x7f2e700ad370 2023-10-19T13:01:12.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.643+0000 7f2e57fff640 1 -- 172.21.15.57:0/1964405281 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2e68076f20 con 0x7f2e700ad370 2023-10-19T13:01:12.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.857+0000 7f2e75ef2640 1 -- 172.21.15.57:0/1964405281 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2e700690e0 con 0x7f2e700ad370 2023-10-19T13:01:12.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.857+0000 7f2e57fff640 1 -- 172.21.15.57:0/1964405281 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+4921 (secure 0 0 0) 0x7f2e6804b090 con 0x7f2e700ad370 2023-10-19T13:01:12.860 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:01:12.860 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":37,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6305,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/2631343514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":2631343514},{"type":"v1","addr":"172.21.15.179:6837","nonce":2631343514}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":34},{"gid":6883,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/4008617188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":4008617188},{"type":"v1","addr":"172.21.15.179:6836","nonce":4008617188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":34},{"gid":7542,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/2010253133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2010253133},{"type":"v1","addr":"172.21.15.57:6834","nonce":2010253133}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T13:00:55.547801+0000","modified":"2023-10-19T13:00:57.571937+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7545},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7545":{"gid":7545,"name":"a","rank":0,"incarnation":35,"state":"up:active","state_seq":3,"addr":"172.21.15.57:6835/2583001580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2583001580},{"type":"v1","addr":"172.21.15.57:6835","nonce":2583001580}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-10-19T13:01:12.861 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 37 2023-10-19T13:01:12.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.859+0000 7f2e75ef2640 1 -- 172.21.15.57:0/1964405281 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2e700ad370 msgr2=0x7f2e700b3700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:12.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.859+0000 7f2e75ef2640 1 --2- 172.21.15.57:0/1964405281 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2e700ad370 0x7f2e700b3700 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f2e68009d60 tx=0x7f2e68035e50 comp rx=0 tx=0).stop 2023-10-19T13:01:12.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.859+0000 7f2e75ef2640 1 -- 172.21.15.57:0/1964405281 shutdown_connections 2023-10-19T13:01:12.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.859+0000 7f2e75ef2640 1 --2- 172.21.15.57:0/1964405281 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2e700ad370 0x7f2e700b3700 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:12.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.859+0000 7f2e75ef2640 1 --2- 172.21.15.57:0/1964405281 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2e7006f0a0 0x7f2e700b31c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:12.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.859+0000 7f2e75ef2640 1 --2- 172.21.15.57:0/1964405281 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2e7006d2d0 0x7f2e7014ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:12.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.859+0000 7f2e75ef2640 1 -- 172.21.15.57:0/1964405281 >> 172.21.15.57:0/1964405281 conn(0x7f2e700b0670 msgr2=0x7f2e70158ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:12.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.859+0000 7f2e75ef2640 1 -- 172.21.15.57:0/1964405281 shutdown_connections 2023-10-19T13:01:12.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.859+0000 7f2e75ef2640 1 -- 172.21.15.57:0/1964405281 wait complete. 2023-10-19T13:01:12.871 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T13:01:12.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.979+0000 7fc85719e640 1 -- 172.21.15.57:0/100339755 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc850141c80 msgr2=0x7fc8500cbba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:12.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.979+0000 7fc85719e640 1 --2- 172.21.15.57:0/100339755 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc850141c80 0x7fc8500cbba0 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7fc848005fd0 tx=0x7fc84802bd90 comp rx=0 tx=0).stop 2023-10-19T13:01:12.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.979+0000 7fc85719e640 1 -- 172.21.15.57:0/100339755 shutdown_connections 2023-10-19T13:01:12.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.979+0000 7fc85719e640 1 --2- 172.21.15.57:0/100339755 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc8500cc0e0 0x7fc85006bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:12.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.979+0000 7fc85719e640 1 --2- 172.21.15.57:0/100339755 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc850141c80 0x7fc8500cbba0 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:12.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.979+0000 7fc85719e640 1 -- 172.21.15.57:0/100339755 >> 172.21.15.57:0/100339755 conn(0x7fc8500ae5d0 msgr2=0x7fc8500ae9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:12.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.979+0000 7fc85719e640 1 -- 172.21.15.57:0/100339755 shutdown_connections 2023-10-19T13:01:12.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.979+0000 7fc85719e640 1 -- 172.21.15.57:0/100339755 wait complete. 2023-10-19T13:01:12.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.981+0000 7fc85719e640 1 Processor -- start 2023-10-19T13:01:12.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.981+0000 7fc85719e640 1 -- start start 2023-10-19T13:01:12.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.981+0000 7fc85719e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc85006c4f0 0x7fc8500c8230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:12.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.981+0000 7fc85719e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc8500cc0e0 0x7fc85015a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:12.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.981+0000 7fc85719e640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc850141c80 0x7fc850157130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:12.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.981+0000 7fc85719e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8500c8770 con 0x7fc85006c4f0 2023-10-19T13:01:12.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.981+0000 7fc85719e640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc85015b150 con 0x7fc8500cc0e0 2023-10-19T13:01:12.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.981+0000 7fc85719e640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc85015a600 con 0x7fc850141c80 2023-10-19T13:01:12.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.982+0000 7fc85619c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc85006c4f0 0x7fc8500c8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:12.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.982+0000 7fc85619c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc85006c4f0 0x7fc8500c8230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56240/0 (socket says 172.21.15.57:56240) 2023-10-19T13:01:12.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.982+0000 7fc85619c640 1 -- 172.21.15.57:0/3028226206 learned_addr learned my addr 172.21.15.57:0/3028226206 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:01:12.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.982+0000 7fc85699d640 1 --2- 172.21.15.57:0/3028226206 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc850141c80 0x7fc850157130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:12.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.982+0000 7fc85599b640 1 --2- 172.21.15.57:0/3028226206 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc8500cc0e0 0x7fc85015a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:12.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.982+0000 7fc85619c640 1 -- 172.21.15.57:0/3028226206 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc850141c80 msgr2=0x7fc850157130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:12.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.982+0000 7fc85619c640 1 --2- 172.21.15.57:0/3028226206 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc850141c80 0x7fc850157130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:12.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.982+0000 7fc85619c640 1 -- 172.21.15.57:0/3028226206 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc8500cc0e0 msgr2=0x7fc85015a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:12.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.982+0000 7fc85619c640 1 --2- 172.21.15.57:0/3028226206 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc8500cc0e0 0x7fc85015a0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:12.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.982+0000 7fc85619c640 1 -- 172.21.15.57:0/3028226206 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc848009560 con 0x7fc85006c4f0 2023-10-19T13:01:12.989 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.982+0000 7fc85599b640 1 --2- 172.21.15.57:0/3028226206 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc8500cc0e0 0x7fc85015a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T13:01:12.989 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.982+0000 7fc85619c640 1 --2- 172.21.15.57:0/3028226206 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc85006c4f0 0x7fc8500c8230 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7fc848036040 tx=0x7fc84802db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:12.989 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.982+0000 7fc84effd640 1 -- 172.21.15.57:0/3028226206 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc848003800 con 0x7fc85006c4f0 2023-10-19T13:01:12.989 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.982+0000 7fc84effd640 1 -- 172.21.15.57:0/3028226206 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fc848003a50 con 0x7fc85006c4f0 2023-10-19T13:01:12.989 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.982+0000 7fc85719e640 1 -- 172.21.15.57:0/3028226206 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc85015a8b0 con 0x7fc85006c4f0 2023-10-19T13:01:12.990 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.982+0000 7fc84effd640 1 -- 172.21.15.57:0/3028226206 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc848003e00 con 0x7fc85006c4f0 2023-10-19T13:01:12.990 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.982+0000 7fc85719e640 1 -- 172.21.15.57:0/3028226206 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc85015ae70 con 0x7fc85006c4f0 2023-10-19T13:01:12.990 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.983+0000 7fc85719e640 1 -- 172.21.15.57:0/3028226206 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc850146ff0 con 0x7fc85006c4f0 2023-10-19T13:01:12.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.984+0000 7fc84effd640 1 -- 172.21.15.57:0/3028226206 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 45) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fc84803e450 con 0x7fc85006c4f0 2023-10-19T13:01:12.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.984+0000 7fc84effd640 1 -- 172.21.15.57:0/3028226206 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 6745+0+0 (secure 0 0 0) 0x7fc8480ac010 con 0x7fc85006c4f0 2023-10-19T13:01:12.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:12.988+0000 7fc84effd640 1 -- 172.21.15.57:0/3028226206 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc848077100 con 0x7fc85006c4f0 2023-10-19T13:01:13.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.066+0000 7fc84effd640 1 -- 172.21.15.57:0/3028226206 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mgrmap(e 46) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fc848076bb0 con 0x7fc85006c4f0 2023-10-19T13:01:13.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.067+0000 7fc84effd640 1 --2- 172.21.15.57:0/3028226206 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7fc83c070b50 0x7fc83c072f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:13.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.067+0000 7fc85599b640 1 --2- 172.21.15.57:0/3028226206 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7fc83c070b50 0x7fc83c072f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:13.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.069+0000 7fc85599b640 1 --2- 172.21.15.57:0/3028226206 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7fc83c070b50 0x7fc83c072f40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fc8500ae530 tx=0x7fc844005d90 comp rx=0 tx=0).ready entity=mgr.7072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:13.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.167+0000 7fc85719e640 1 -- 172.21.15.57:0/3028226206 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc85004da70 con 0x7fc85006c4f0 2023-10-19T13:01:13.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.168+0000 7fc84effd640 1 -- 172.21.15.57:0/3028226206 <== mon.0 v2:172.21.15.57:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v102) v1 ==== 74+0+17850 (secure 0 0 0) 0x7fc8480a56a0 con 0x7fc85006c4f0 2023-10-19T13:01:13.170 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:01:13.170 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":102,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T13:01:12.052858+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2023-10-19T13:00:53.446697+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2023-10-19T13:00:54.540962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.179:0/1870731741":"2023-10-20T13:01:12.052824+0000","172.21.15.179:0/582382550":"2023-10-20T13:01:12.052824+0000","172.21.15.179:6833/124009847":"2023-10-20T13:01:12.052824+0000","172.21.15.179:0/3304791412":"2023-10-20T13:01:12.052824+0000","172.21.15.179:0/3356740410":"2023-10-20T13:01:12.052824+0000","172.21.15.179:0/1965409844":"2023-10-20T13:01:12.052824+0000","172.21.15.179:6833/1413162726":"2023-10-20T13:01:06.603811+0000","172.21.15.179:0/1810025399":"2023-10-20T13:01:12.052824+0000","172.21.15.179:6832/1413162726":"2023-10-20T13:01:06.603811+0000","172.21.15.179:0/3401895301":"2023-10-20T13:01:06.603811+0000","172.21.15.179:0/2499848570":"2023-10-20T13:01:06.603811+0000","172.21.15.179:0/3622569807":"2023-10-20T13:01:06.603811+0000","172.21.15.179:0/2835518029":"2023-10-20T13:01:06.603811+0000","172.21.15.179:0/1643279404":"2023-10-20T13:01:06.603811+0000","172.21.15.179:6832/124009847":"2023-10-20T13:01:12.052824+0000","172.21.15.179:0/4219879451":"2023-10-20T13:01:06.603811+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T13:01:13.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.170+0000 7fc85719e640 1 -- 172.21.15.57:0/3028226206 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7fc83c070b50 msgr2=0x7fc83c072f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:13.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.170+0000 7fc85719e640 1 --2- 172.21.15.57:0/3028226206 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7fc83c070b50 0x7fc83c072f40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fc8500ae530 tx=0x7fc844005d90 comp rx=0 tx=0).stop 2023-10-19T13:01:13.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.170+0000 7fc85719e640 1 -- 172.21.15.57:0/3028226206 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc85006c4f0 msgr2=0x7fc8500c8230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:13.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.170+0000 7fc85719e640 1 --2- 172.21.15.57:0/3028226206 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc85006c4f0 0x7fc8500c8230 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7fc848036040 tx=0x7fc84802db10 comp rx=0 tx=0).stop 2023-10-19T13:01:13.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.170+0000 7fc85719e640 1 -- 172.21.15.57:0/3028226206 shutdown_connections 2023-10-19T13:01:13.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.170+0000 7fc85719e640 1 --2- 172.21.15.57:0/3028226206 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7fc83c070b50 0x7fc83c072f40 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:13.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.170+0000 7fc85719e640 1 --2- 172.21.15.57:0/3028226206 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc850141c80 0x7fc850157130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:13.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.170+0000 7fc85719e640 1 --2- 172.21.15.57:0/3028226206 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc8500cc0e0 0x7fc85015a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:13.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.170+0000 7fc85719e640 1 --2- 172.21.15.57:0/3028226206 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc85006c4f0 0x7fc8500c8230 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:13.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.170+0000 7fc85719e640 1 -- 172.21.15.57:0/3028226206 >> 172.21.15.57:0/3028226206 conn(0x7fc8500ae5d0 msgr2=0x7fc850159080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:13.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.170+0000 7fc85719e640 1 -- 172.21.15.57:0/3028226206 shutdown_connections 2023-10-19T13:01:13.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.170+0000 7fc85719e640 1 -- 172.21.15.57:0/3028226206 wait complete. 2023-10-19T13:01:13.183 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-19T13:01:13.183 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T13:01:13.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.293+0000 7f8ec55d7640 1 -- 172.21.15.57:0/2821324405 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ea80047c0 msgr2=0x7f8ea8004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:13.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.293+0000 7f8ec55d7640 1 --2- 172.21.15.57:0/2821324405 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ea80047c0 0x7f8ea8004c40 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f8eb4001680 tx=0x7f8ea80057f0 comp rx=0 tx=0).stop 2023-10-19T13:01:13.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.293+0000 7f8ec55d7640 1 -- 172.21.15.57:0/2821324405 shutdown_connections 2023-10-19T13:01:13.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.293+0000 7f8ec55d7640 1 --2- 172.21.15.57:0/2821324405 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ea80047c0 0x7f8ea8004c40 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:13.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.293+0000 7f8ec55d7640 1 -- 172.21.15.57:0/2821324405 >> 172.21.15.57:0/2821324405 conn(0x7f8ec009a030 msgr2=0x7f8ec00d0d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:13.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.293+0000 7f8ec55d7640 1 -- 172.21.15.57:0/2821324405 shutdown_connections 2023-10-19T13:01:13.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.293+0000 7f8ec55d7640 1 -- 172.21.15.57:0/2821324405 wait complete. 2023-10-19T13:01:13.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.294+0000 7f8ec55d7640 1 Processor -- start 2023-10-19T13:01:13.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.294+0000 7f8ec55d7640 1 -- start start 2023-10-19T13:01:13.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.295+0000 7f8ec55d7640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8ea80047c0 0x7f8ec00c4850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:13.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.295+0000 7f8ec55d7640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8ec0159080 0x7f8ec00c60d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:13.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.295+0000 7f8ec55d7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ec006b270 0x7f8ec006b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:13.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.295+0000 7f8ec55d7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ec006bc30 con 0x7f8ec006b270 2023-10-19T13:01:13.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.295+0000 7f8ec55d7640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ec00c75a0 con 0x7f8ea80047c0 2023-10-19T13:01:13.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.295+0000 7f8ec55d7640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ec0159500 con 0x7f8ec0159080 2023-10-19T13:01:13.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.295+0000 7f8ebf7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ec006b270 0x7f8ec006b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:13.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.295+0000 7f8ebf7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ec006b270 0x7f8ec006b6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56252/0 (socket says 172.21.15.57:56252) 2023-10-19T13:01:13.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.295+0000 7f8ebf7fe640 1 -- 172.21.15.57:0/3174541129 learned_addr learned my addr 172.21.15.57:0/3174541129 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:01:13.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.295+0000 7f8ebe7fc640 1 --2- 172.21.15.57:0/3174541129 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8ec0159080 0x7f8ec00c60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:13.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.295+0000 7f8ebeffd640 1 --2- 172.21.15.57:0/3174541129 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8ea80047c0 0x7f8ec00c4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:13.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.295+0000 7f8ebf7fe640 1 -- 172.21.15.57:0/3174541129 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8ec0159080 msgr2=0x7f8ec00c60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:13.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.295+0000 7f8ebf7fe640 1 --2- 172.21.15.57:0/3174541129 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8ec0159080 0x7f8ec00c60d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:13.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.295+0000 7f8ebf7fe640 1 -- 172.21.15.57:0/3174541129 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8ea80047c0 msgr2=0x7f8ec00c4850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:13.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.295+0000 7f8ebf7fe640 1 --2- 172.21.15.57:0/3174541129 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8ea80047c0 0x7f8ec00c4850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:13.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.295+0000 7f8ebf7fe640 1 -- 172.21.15.57:0/3174541129 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ec00c6990 con 0x7f8ec006b270 2023-10-19T13:01:13.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.295+0000 7f8ebeffd640 1 --2- 172.21.15.57:0/3174541129 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8ea80047c0 0x7f8ec00c4850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:01:13.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.295+0000 7f8ebf7fe640 1 --2- 172.21.15.57:0/3174541129 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ec006b270 0x7f8ec006b6f0 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f8ec0089fb0 tx=0x7f8eb0011f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:13.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.296+0000 7f8ea77fe640 1 -- 172.21.15.57:0/3174541129 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eb0004b20 con 0x7f8ec006b270 2023-10-19T13:01:13.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.296+0000 7f8ea77fe640 1 -- 172.21.15.57:0/3174541129 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f8eb0004c80 con 0x7f8ec006b270 2023-10-19T13:01:13.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.296+0000 7f8ec55d7640 1 -- 172.21.15.57:0/3174541129 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ec00c6c70 con 0x7f8ec006b270 2023-10-19T13:01:13.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.296+0000 7f8ea77fe640 1 -- 172.21.15.57:0/3174541129 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eb0003bd0 con 0x7f8ec006b270 2023-10-19T13:01:13.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.296+0000 7f8ec55d7640 1 -- 172.21.15.57:0/3174541129 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ec00c7230 con 0x7f8ec006b270 2023-10-19T13:01:13.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.297+0000 7f8ec55d7640 1 -- 172.21.15.57:0/3174541129 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ec00675f0 con 0x7f8ec006b270 2023-10-19T13:01:13.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.297+0000 7f8ea77fe640 1 -- 172.21.15.57:0/3174541129 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 46) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f8eb0003d30 con 0x7f8ec006b270 2023-10-19T13:01:13.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.297+0000 7f8ea77fe640 1 --2- 172.21.15.57:0/3174541129 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f8eac06acc0 0x7f8eac06d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:13.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.297+0000 7f8ea77fe640 1 -- 172.21.15.57:0/3174541129 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 6745+0+0 (secure 0 0 0) 0x7f8eb0089f70 con 0x7f8ec006b270 2023-10-19T13:01:13.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.298+0000 7f8ebeffd640 1 --2- 172.21.15.57:0/3174541129 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f8eac06acc0 0x7f8eac06d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:13.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.298+0000 7f8ebeffd640 1 --2- 172.21.15.57:0/3174541129 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f8eac06acc0 0x7f8eac06d180 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f8eb8000f60 tx=0x7f8eb8000ed0 comp rx=0 tx=0).ready entity=mgr.7072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:13.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.302+0000 7f8ea77fe640 1 -- 172.21.15.57:0/3174541129 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8eb0054f50 con 0x7f8ec006b270 2023-10-19T13:01:13.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.513+0000 7f8ec55d7640 1 -- 172.21.15.57:0/3174541129 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8ec00c7710 con 0x7f8ec006b270 2023-10-19T13:01:13.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.513+0000 7f8ea77fe640 1 -- 172.21.15.57:0/3174541129 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+4921 (secure 0 0 0) 0x7f8eb001d090 con 0x7f8ec006b270 2023-10-19T13:01:13.516 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:01:13.516 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":37,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6305,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/2631343514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":2631343514},{"type":"v1","addr":"172.21.15.179:6837","nonce":2631343514}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":34},{"gid":6883,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/4008617188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":4008617188},{"type":"v1","addr":"172.21.15.179:6836","nonce":4008617188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":34},{"gid":7542,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/2010253133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2010253133},{"type":"v1","addr":"172.21.15.57:6834","nonce":2010253133}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T13:00:55.547801+0000","modified":"2023-10-19T13:00:57.571937+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7545},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7545":{"gid":7545,"name":"a","rank":0,"incarnation":35,"state":"up:active","state_seq":3,"addr":"172.21.15.57:6835/2583001580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2583001580},{"type":"v1","addr":"172.21.15.57:6835","nonce":2583001580}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-10-19T13:01:13.516 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 37 2023-10-19T13:01:13.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.516+0000 7f8ec55d7640 1 -- 172.21.15.57:0/3174541129 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f8eac06acc0 msgr2=0x7f8eac06d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:13.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.516+0000 7f8ec55d7640 1 --2- 172.21.15.57:0/3174541129 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f8eac06acc0 0x7f8eac06d180 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f8eb8000f60 tx=0x7f8eb8000ed0 comp rx=0 tx=0).stop 2023-10-19T13:01:13.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.516+0000 7f8ec55d7640 1 -- 172.21.15.57:0/3174541129 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ec006b270 msgr2=0x7f8ec006b6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:13.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.516+0000 7f8ec55d7640 1 --2- 172.21.15.57:0/3174541129 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ec006b270 0x7f8ec006b6f0 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f8ec0089fb0 tx=0x7f8eb0011f40 comp rx=0 tx=0).stop 2023-10-19T13:01:13.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.516+0000 7f8ec55d7640 1 -- 172.21.15.57:0/3174541129 shutdown_connections 2023-10-19T13:01:13.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.516+0000 7f8ec55d7640 1 --2- 172.21.15.57:0/3174541129 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f8eac06acc0 0x7f8eac06d180 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:13.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.516+0000 7f8ec55d7640 1 --2- 172.21.15.57:0/3174541129 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ec006b270 0x7f8ec006b6f0 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:13.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.516+0000 7f8ec55d7640 1 --2- 172.21.15.57:0/3174541129 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8ec0159080 0x7f8ec00c60d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:13.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.516+0000 7f8ec55d7640 1 --2- 172.21.15.57:0/3174541129 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8ea80047c0 0x7f8ec00c4850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:13.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.516+0000 7f8ec55d7640 1 -- 172.21.15.57:0/3174541129 >> 172.21.15.57:0/3174541129 conn(0x7f8ec009a030 msgr2=0x7f8ec015da30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:13.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.516+0000 7f8ec55d7640 1 -- 172.21.15.57:0/3174541129 shutdown_connections 2023-10-19T13:01:13.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.516+0000 7f8ec55d7640 1 -- 172.21.15.57:0/3174541129 wait complete. 2023-10-19T13:01:13.528 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T13:01:13.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.636+0000 7fd7d35d0640 1 -- 172.21.15.57:0/984462652 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd7cc0cf340 msgr2=0x7fd7cc0d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:13.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.636+0000 7fd7d35d0640 1 --2- 172.21.15.57:0/984462652 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd7cc0cf340 0x7fd7cc0d1910 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd7c8009830 tx=0x7fd7c802bcd0 comp rx=0 tx=0).stop 2023-10-19T13:01:13.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.636+0000 7fd7d35d0640 1 -- 172.21.15.57:0/984462652 shutdown_connections 2023-10-19T13:01:13.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.636+0000 7fd7d35d0640 1 --2- 172.21.15.57:0/984462652 >> v2:172.21.15.179:3300/0 conn(0x7fd7cc15b2a0 0x7fd7cc0cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:13.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.636+0000 7fd7d35d0640 1 --2- 172.21.15.57:0/984462652 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd7cc0cf340 0x7fd7cc0d1910 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:13.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.636+0000 7fd7d35d0640 1 -- 172.21.15.57:0/984462652 >> 172.21.15.57:0/984462652 conn(0x7fd7cc0d0d60 msgr2=0x7fd7cc0d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:13.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.636+0000 7fd7d35d0640 1 -- 172.21.15.57:0/984462652 shutdown_connections 2023-10-19T13:01:13.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.636+0000 7fd7d35d0640 1 -- 172.21.15.57:0/984462652 wait complete. 2023-10-19T13:01:13.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.637+0000 7fd7d35d0640 1 Processor -- start 2023-10-19T13:01:13.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.637+0000 7fd7d35d0640 1 -- start start 2023-10-19T13:01:13.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.638+0000 7fd7d35d0640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd7cc0ceb40 0x7fd7cc0ca270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:13.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.638+0000 7fd7d35d0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd7cc0cf340 0x7fd7cc154050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:13.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.638+0000 7fd7d35d0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd7cc15b2a0 0x7fd7cc154590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:13.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.638+0000 7fd7d35d0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7cc159510 con 0x7fd7cc0cf340 2023-10-19T13:01:13.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.638+0000 7fd7d35d0640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7cc06bd90 con 0x7fd7cc15b2a0 2023-10-19T13:01:13.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.638+0000 7fd7d35d0640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7cc0b64e0 con 0x7fd7cc0ceb40 2023-10-19T13:01:13.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.638+0000 7fd7d0b6b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd7cc0cf340 0x7fd7cc154050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:13.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.638+0000 7fd7d0b6b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd7cc0cf340 0x7fd7cc154050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56264/0 (socket says 172.21.15.57:56264) 2023-10-19T13:01:13.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.638+0000 7fd7d0b6b640 1 -- 172.21.15.57:0/1235684132 learned_addr learned my addr 172.21.15.57:0/1235684132 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:01:13.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.638+0000 7fd7d136c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd7cc0ceb40 0x7fd7cc0ca270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:13.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.638+0000 7fd7d1b6d640 1 --2- 172.21.15.57:0/1235684132 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd7cc15b2a0 0x7fd7cc154590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:13.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.638+0000 7fd7d0b6b640 1 -- 172.21.15.57:0/1235684132 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd7cc0ceb40 msgr2=0x7fd7cc0ca270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:13.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.638+0000 7fd7d0b6b640 1 --2- 172.21.15.57:0/1235684132 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd7cc0ceb40 0x7fd7cc0ca270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:13.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.638+0000 7fd7d0b6b640 1 -- 172.21.15.57:0/1235684132 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd7cc15b2a0 msgr2=0x7fd7cc154590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:13.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.638+0000 7fd7d0b6b640 1 --2- 172.21.15.57:0/1235684132 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd7cc15b2a0 0x7fd7cc154590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:13.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.638+0000 7fd7d0b6b640 1 -- 172.21.15.57:0/1235684132 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7bc009490 con 0x7fd7cc0cf340 2023-10-19T13:01:13.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.638+0000 7fd7d136c640 1 --2- 172.21.15.57:0/1235684132 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd7cc0ceb40 0x7fd7cc0ca270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T13:01:13.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.638+0000 7fd7d0b6b640 1 --2- 172.21.15.57:0/1235684132 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd7cc0cf340 0x7fd7cc154050 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7fd7bc00e9f0 tx=0x7fd7bc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:13.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.638+0000 7fd7c1ffb640 1 -- 172.21.15.57:0/1235684132 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7bc00cd80 con 0x7fd7cc0cf340 2023-10-19T13:01:13.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.639+0000 7fd7c1ffb640 1 -- 172.21.15.57:0/1235684132 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fd7bc004020 con 0x7fd7cc0cf340 2023-10-19T13:01:13.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.639+0000 7fd7d35d0640 1 -- 172.21.15.57:0/1235684132 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7c8009490 con 0x7fd7cc0cf340 2023-10-19T13:01:13.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.639+0000 7fd7c1ffb640 1 -- 172.21.15.57:0/1235684132 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7bc004410 con 0x7fd7cc0cf340 2023-10-19T13:01:13.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.639+0000 7fd7d35d0640 1 -- 172.21.15.57:0/1235684132 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7cc0b6a10 con 0x7fd7cc0cf340 2023-10-19T13:01:13.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.640+0000 7fd7d35d0640 1 -- 172.21.15.57:0/1235684132 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7cc0690e0 con 0x7fd7cc0cf340 2023-10-19T13:01:13.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.640+0000 7fd7c1ffb640 1 -- 172.21.15.57:0/1235684132 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 46) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fd7bc0035f0 con 0x7fd7cc0cf340 2023-10-19T13:01:13.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.640+0000 7fd7c1ffb640 1 --2- 172.21.15.57:0/1235684132 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7fd7b006ac90 0x7fd7b006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:13.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.641+0000 7fd7c1ffb640 1 -- 172.21.15.57:0/1235684132 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 6745+0+0 (secure 0 0 0) 0x7fd7bc014070 con 0x7fd7cc0cf340 2023-10-19T13:01:13.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.641+0000 7fd7d136c640 1 --2- 172.21.15.57:0/1235684132 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7fd7b006ac90 0x7fd7b006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:13.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.641+0000 7fd7d136c640 1 --2- 172.21.15.57:0/1235684132 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7fd7b006ac90 0x7fd7b006d150 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd7c8036000 tx=0x7fd7c8004730 comp rx=0 tx=0).ready entity=mgr.7072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:13.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.645+0000 7fd7c1ffb640 1 -- 172.21.15.57:0/1235684132 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd7bc055120 con 0x7fd7cc0cf340 2023-10-19T13:01:13.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.856+0000 7fd7d35d0640 1 -- 172.21.15.57:0/1235684132 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd7cc0d0b40 con 0x7fd7cc0cf340 2023-10-19T13:01:13.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.857+0000 7fd7c1ffb640 1 -- 172.21.15.57:0/1235684132 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+4921 (secure 0 0 0) 0x7fd7bc058740 con 0x7fd7cc0cf340 2023-10-19T13:01:13.859 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:01:13.860 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":37,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6305,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/2631343514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":2631343514},{"type":"v1","addr":"172.21.15.179:6837","nonce":2631343514}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":34},{"gid":6883,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/4008617188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":4008617188},{"type":"v1","addr":"172.21.15.179:6836","nonce":4008617188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":34},{"gid":7542,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/2010253133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2010253133},{"type":"v1","addr":"172.21.15.57:6834","nonce":2010253133}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T13:00:55.547801+0000","modified":"2023-10-19T13:00:57.571937+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7545},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7545":{"gid":7545,"name":"a","rank":0,"incarnation":35,"state":"up:active","state_seq":3,"addr":"172.21.15.57:6835/2583001580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2583001580},{"type":"v1","addr":"172.21.15.57:6835","nonce":2583001580}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-10-19T13:01:13.861 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 37 2023-10-19T13:01:13.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.858+0000 7fd7d35d0640 1 -- 172.21.15.57:0/1235684132 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7fd7b006ac90 msgr2=0x7fd7b006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:13.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.858+0000 7fd7d35d0640 1 --2- 172.21.15.57:0/1235684132 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7fd7b006ac90 0x7fd7b006d150 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd7c8036000 tx=0x7fd7c8004730 comp rx=0 tx=0).stop 2023-10-19T13:01:13.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.858+0000 7fd7d35d0640 1 -- 172.21.15.57:0/1235684132 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd7cc0cf340 msgr2=0x7fd7cc154050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:13.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.858+0000 7fd7d35d0640 1 --2- 172.21.15.57:0/1235684132 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd7cc0cf340 0x7fd7cc154050 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7fd7bc00e9f0 tx=0x7fd7bc00eeb0 comp rx=0 tx=0).stop 2023-10-19T13:01:13.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.858+0000 7fd7d35d0640 1 -- 172.21.15.57:0/1235684132 shutdown_connections 2023-10-19T13:01:13.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.858+0000 7fd7d35d0640 1 --2- 172.21.15.57:0/1235684132 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7fd7b006ac90 0x7fd7b006d150 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:13.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.858+0000 7fd7d35d0640 1 --2- 172.21.15.57:0/1235684132 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd7cc15b2a0 0x7fd7cc154590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:13.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.858+0000 7fd7d35d0640 1 --2- 172.21.15.57:0/1235684132 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd7cc0cf340 0x7fd7cc154050 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:13.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.858+0000 7fd7d35d0640 1 --2- 172.21.15.57:0/1235684132 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd7cc0ceb40 0x7fd7cc0ca270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:13.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.859+0000 7fd7d35d0640 1 -- 172.21.15.57:0/1235684132 >> 172.21.15.57:0/1235684132 conn(0x7fd7cc0d0d60 msgr2=0x7fd7cc1e5860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:13.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.859+0000 7fd7d35d0640 1 -- 172.21.15.57:0/1235684132 shutdown_connections 2023-10-19T13:01:13.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.859+0000 7fd7d35d0640 1 -- 172.21.15.57:0/1235684132 wait complete. 2023-10-19T13:01:13.871 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T13:01:13.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.978+0000 7f671da44640 1 -- 172.21.15.57:0/3772237713 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f67180cbd20 msgr2=0x7f6718158300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:13.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.978+0000 7f671da44640 1 --2- 172.21.15.57:0/3772237713 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f67180cbd20 0x7f6718158300 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f67180b10f0 tx=0x7f670802c110 comp rx=0 tx=0).stop 2023-10-19T13:01:13.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.978+0000 7f671da44640 1 -- 172.21.15.57:0/3772237713 shutdown_connections 2023-10-19T13:01:13.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.978+0000 7f671da44640 1 --2- 172.21.15.57:0/3772237713 >> v2:172.21.15.179:3300/0 conn(0x7f67180b0530 0x7f67180b09a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:13.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.978+0000 7f671da44640 1 --2- 172.21.15.57:0/3772237713 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f67180cbd20 0x7f6718158300 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:13.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.978+0000 7f671da44640 1 -- 172.21.15.57:0/3772237713 >> 172.21.15.57:0/3772237713 conn(0x7f6718069930 msgr2=0x7f6718069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:13.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.978+0000 7f671da44640 1 -- 172.21.15.57:0/3772237713 shutdown_connections 2023-10-19T13:01:13.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.978+0000 7f671da44640 1 -- 172.21.15.57:0/3772237713 wait complete. 2023-10-19T13:01:13.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.980+0000 7f671da44640 1 Processor -- start 2023-10-19T13:01:13.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.980+0000 7f671da44640 1 -- start start 2023-10-19T13:01:13.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.980+0000 7f671da44640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f67180afa20 0x7f6718140750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:13.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.980+0000 7f671da44640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f67180b0530 0x7f6718149890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:13.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.980+0000 7f671da44640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f67180cbd20 0x7f671806b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:13.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.980+0000 7f671da44640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6718149650 con 0x7f67180cbd20 2023-10-19T13:01:13.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.980+0000 7f671da44640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67181427c0 con 0x7f67180afa20 2023-10-19T13:01:13.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.980+0000 7f671da44640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6718149090 con 0x7f67180b0530 2023-10-19T13:01:13.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.980+0000 7f67177fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f67180cbd20 0x7f671806b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:13.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.981+0000 7f67177fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f67180cbd20 0x7f671806b1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56292/0 (socket says 172.21.15.57:56292) 2023-10-19T13:01:13.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.981+0000 7f67177fe640 1 -- 172.21.15.57:0/940458527 learned_addr learned my addr 172.21.15.57:0/940458527 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:01:13.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.981+0000 7f6716ffd640 1 --2- 172.21.15.57:0/940458527 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f67180afa20 0x7f6718140750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:13.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.981+0000 7f67167fc640 1 --2- 172.21.15.57:0/940458527 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f67180b0530 0x7f6718149890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:13.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.981+0000 7f67177fe640 1 -- 172.21.15.57:0/940458527 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f67180b0530 msgr2=0x7f6718149890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:13.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.981+0000 7f67177fe640 1 --2- 172.21.15.57:0/940458527 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f67180b0530 0x7f6718149890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:13.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.981+0000 7f67177fe640 1 -- 172.21.15.57:0/940458527 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f67180afa20 msgr2=0x7f6718140750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:13.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.981+0000 7f67177fe640 1 --2- 172.21.15.57:0/940458527 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f67180afa20 0x7f6718140750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:13.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.981+0000 7f67177fe640 1 -- 172.21.15.57:0/940458527 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6708009840 con 0x7f67180cbd20 2023-10-19T13:01:13.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.981+0000 7f6716ffd640 1 --2- 172.21.15.57:0/940458527 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f67180afa20 0x7f6718140750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T13:01:13.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.981+0000 7f67167fc640 1 --2- 172.21.15.57:0/940458527 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f67180b0530 0x7f6718149890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T13:01:13.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.981+0000 7f67177fe640 1 --2- 172.21.15.57:0/940458527 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f67180cbd20 0x7f671806b1c0 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f671000c970 tx=0x7f671000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:13.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.981+0000 7f670ffff640 1 -- 172.21.15.57:0/940458527 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6710007bb0 con 0x7f67180cbd20 2023-10-19T13:01:13.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.981+0000 7f670ffff640 1 -- 172.21.15.57:0/940458527 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f6710007e20 con 0x7f67180cbd20 2023-10-19T13:01:13.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.981+0000 7f671da44640 1 -- 172.21.15.57:0/940458527 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6718149370 con 0x7f67180cbd20 2023-10-19T13:01:13.989 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.981+0000 7f670ffff640 1 -- 172.21.15.57:0/940458527 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f671000f6b0 con 0x7f67180cbd20 2023-10-19T13:01:13.989 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.981+0000 7f671da44640 1 -- 172.21.15.57:0/940458527 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67181422a0 con 0x7f67180cbd20 2023-10-19T13:01:13.989 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.982+0000 7f671da44640 1 -- 172.21.15.57:0/940458527 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67180690e0 con 0x7f67180cbd20 2023-10-19T13:01:13.989 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.983+0000 7f670ffff640 1 -- 172.21.15.57:0/940458527 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 46) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f671000f810 con 0x7f67180cbd20 2023-10-19T13:01:13.990 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.983+0000 7f670ffff640 1 --2- 172.21.15.57:0/940458527 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f66f806ac90 0x7f66f806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:13.990 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.983+0000 7f670ffff640 1 -- 172.21.15.57:0/940458527 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 6745+0+0 (secure 0 0 0) 0x7f6710089f40 con 0x7f67180cbd20 2023-10-19T13:01:13.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.983+0000 7f6716ffd640 1 --2- 172.21.15.57:0/940458527 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f66f806ac90 0x7f66f806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:13.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.984+0000 7f6716ffd640 1 --2- 172.21.15.57:0/940458527 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f66f806ac90 0x7f66f806d150 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f6708035000 tx=0x7f6708004ab0 comp rx=0 tx=0).ready entity=mgr.7072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:13.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:13.987+0000 7f670ffff640 1 -- 172.21.15.57:0/940458527 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6710054e90 con 0x7f67180cbd20 2023-10-19T13:01:14.099 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:14.096+0000 7fb508163640 -1 client.7141: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.7030:f55302c0-37aa-4235-8ccc-9775bc542785:v1:172.21.15.179:0/3304791412] 2023-10-19T13:01:14.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.168+0000 7f671da44640 1 -- 172.21.15.57:0/940458527 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f671806f700 con 0x7f67180cbd20 2023-10-19T13:01:14.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.169+0000 7f670ffff640 1 -- 172.21.15.57:0/940458527 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v102) v1 ==== 74+0+17850 (secure 0 0 0) 0x7f671001d070 con 0x7f67180cbd20 2023-10-19T13:01:14.171 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:01:14.172 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":102,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T13:01:12.052858+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2023-10-19T13:00:53.446697+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2023-10-19T13:00:54.540962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.179:0/1870731741":"2023-10-20T13:01:12.052824+0000","172.21.15.179:0/582382550":"2023-10-20T13:01:12.052824+0000","172.21.15.179:6833/124009847":"2023-10-20T13:01:12.052824+0000","172.21.15.179:0/3304791412":"2023-10-20T13:01:12.052824+0000","172.21.15.179:0/3356740410":"2023-10-20T13:01:12.052824+0000","172.21.15.179:0/1965409844":"2023-10-20T13:01:12.052824+0000","172.21.15.179:6833/1413162726":"2023-10-20T13:01:06.603811+0000","172.21.15.179:0/1810025399":"2023-10-20T13:01:12.052824+0000","172.21.15.179:6832/1413162726":"2023-10-20T13:01:06.603811+0000","172.21.15.179:0/3401895301":"2023-10-20T13:01:06.603811+0000","172.21.15.179:0/2499848570":"2023-10-20T13:01:06.603811+0000","172.21.15.179:0/3622569807":"2023-10-20T13:01:06.603811+0000","172.21.15.179:0/2835518029":"2023-10-20T13:01:06.603811+0000","172.21.15.179:0/1643279404":"2023-10-20T13:01:06.603811+0000","172.21.15.179:6832/124009847":"2023-10-20T13:01:12.052824+0000","172.21.15.179:0/4219879451":"2023-10-20T13:01:06.603811+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T13:01:14.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.171+0000 7f671da44640 1 -- 172.21.15.57:0/940458527 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f66f806ac90 msgr2=0x7f66f806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:14.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.171+0000 7f671da44640 1 --2- 172.21.15.57:0/940458527 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f66f806ac90 0x7f66f806d150 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f6708035000 tx=0x7f6708004ab0 comp rx=0 tx=0).stop 2023-10-19T13:01:14.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.171+0000 7f671da44640 1 -- 172.21.15.57:0/940458527 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f67180cbd20 msgr2=0x7f671806b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:14.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.171+0000 7f671da44640 1 --2- 172.21.15.57:0/940458527 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f67180cbd20 0x7f671806b1c0 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f671000c970 tx=0x7f671000ce30 comp rx=0 tx=0).stop 2023-10-19T13:01:14.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.171+0000 7f671da44640 1 -- 172.21.15.57:0/940458527 shutdown_connections 2023-10-19T13:01:14.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.171+0000 7f671da44640 1 --2- 172.21.15.57:0/940458527 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f66f806ac90 0x7f66f806d150 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:14.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.171+0000 7f671da44640 1 --2- 172.21.15.57:0/940458527 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f67180cbd20 0x7f671806b1c0 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:14.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.171+0000 7f671da44640 1 --2- 172.21.15.57:0/940458527 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f67180b0530 0x7f6718149890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:14.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.171+0000 7f671da44640 1 --2- 172.21.15.57:0/940458527 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f67180afa20 0x7f6718140750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:14.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.171+0000 7f671da44640 1 -- 172.21.15.57:0/940458527 >> 172.21.15.57:0/940458527 conn(0x7f6718069930 msgr2=0x7f671806bf70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:14.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.172+0000 7f671da44640 1 -- 172.21.15.57:0/940458527 shutdown_connections 2023-10-19T13:01:14.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.172+0000 7f671da44640 1 -- 172.21.15.57:0/940458527 wait complete. 2023-10-19T13:01:14.184 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-19T13:01:14.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.294+0000 7f8445b05640 1 -- 172.21.15.57:0/561345681 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f844013e520 msgr2=0x7f844006bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:14.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.294+0000 7f8445b05640 1 --2- 172.21.15.57:0/561345681 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f844013e520 0x7f844006bb60 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f8434009830 tx=0x7f843402bc30 comp rx=0 tx=0).stop 2023-10-19T13:01:14.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.294+0000 7f8445b05640 1 -- 172.21.15.57:0/561345681 shutdown_connections 2023-10-19T13:01:14.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.294+0000 7f8445b05640 1 --2- 172.21.15.57:0/561345681 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8440141690 0x7f84400703e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:14.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.294+0000 7f8445b05640 1 --2- 172.21.15.57:0/561345681 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f844013e520 0x7f844006bb60 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:14.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.294+0000 7f8445b05640 1 -- 172.21.15.57:0/561345681 >> 172.21.15.57:0/561345681 conn(0x7f8440069930 msgr2=0x7f8440069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:14.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.294+0000 7f8445b05640 1 -- 172.21.15.57:0/561345681 shutdown_connections 2023-10-19T13:01:14.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.294+0000 7f8445b05640 1 -- 172.21.15.57:0/561345681 wait complete. 2023-10-19T13:01:14.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.295+0000 7f8445b05640 1 Processor -- start 2023-10-19T13:01:14.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.295+0000 7f8445b05640 1 -- start start 2023-10-19T13:01:14.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.296+0000 7f8445b05640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f844006c0a0 0x7f844013b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:14.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.296+0000 7f8445b05640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f844013e520 0x7f84400acb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:14.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.296+0000 7f8445b05640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8440141690 0x7f84400ca870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:14.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.296+0000 7f8445b05640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84400a7150 con 0x7f8440141690 2023-10-19T13:01:14.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.296+0000 7f8445b05640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84400a7430 con 0x7f844006c0a0 2023-10-19T13:01:14.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.296+0000 7f8445b05640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84400a7710 con 0x7f844013e520 2023-10-19T13:01:14.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.296+0000 7f843ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8440141690 0x7f84400ca870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:14.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.296+0000 7f843ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8440141690 0x7f84400ca870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56308/0 (socket says 172.21.15.57:56308) 2023-10-19T13:01:14.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.296+0000 7f843ffff640 1 -- 172.21.15.57:0/809104693 learned_addr learned my addr 172.21.15.57:0/809104693 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:01:14.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.296+0000 7f843effd640 1 --2- 172.21.15.57:0/809104693 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f844013e520 0x7f84400acb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:14.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.296+0000 7f843ffff640 1 -- 172.21.15.57:0/809104693 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f844013e520 msgr2=0x7f84400acb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:14.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.296+0000 7f843ffff640 1 --2- 172.21.15.57:0/809104693 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f844013e520 0x7f84400acb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:14.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.296+0000 7f843ffff640 1 -- 172.21.15.57:0/809104693 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f844006c0a0 msgr2=0x7f844013b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:14.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.296+0000 7f843ffff640 1 --2- 172.21.15.57:0/809104693 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f844006c0a0 0x7f844013b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:14.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.296+0000 7f843ffff640 1 -- 172.21.15.57:0/809104693 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8434009490 con 0x7f8440141690 2023-10-19T13:01:14.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.297+0000 7f843ffff640 1 --2- 172.21.15.57:0/809104693 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8440141690 0x7f84400ca870 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f843800cbc0 tx=0x7f843800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:14.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.297+0000 7f8427fff640 1 -- 172.21.15.57:0/809104693 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f843800ad40 con 0x7f8440141690 2023-10-19T13:01:14.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.297+0000 7f8427fff640 1 -- 172.21.15.57:0/809104693 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f8438012420 con 0x7f8440141690 2023-10-19T13:01:14.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.297+0000 7f8445b05640 1 -- 172.21.15.57:0/809104693 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84400a79f0 con 0x7f8440141690 2023-10-19T13:01:14.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.297+0000 7f8427fff640 1 -- 172.21.15.57:0/809104693 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8438012850 con 0x7f8440141690 2023-10-19T13:01:14.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.297+0000 7f8445b05640 1 -- 172.21.15.57:0/809104693 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84400a90e0 con 0x7f8440141690 2023-10-19T13:01:14.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.298+0000 7f8445b05640 1 -- 172.21.15.57:0/809104693 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84400ac400 con 0x7f8440141690 2023-10-19T13:01:14.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.298+0000 7f8427fff640 1 -- 172.21.15.57:0/809104693 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 46) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f843801a3b0 con 0x7f8440141690 2023-10-19T13:01:14.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.299+0000 7f8427fff640 1 --2- 172.21.15.57:0/809104693 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f842c06ac90 0x7f842c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:14.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.299+0000 7f8427fff640 1 -- 172.21.15.57:0/809104693 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 6745+0+0 (secure 0 0 0) 0x7f8438089f30 con 0x7f8440141690 2023-10-19T13:01:14.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.299+0000 7f843f7fe640 1 --2- 172.21.15.57:0/809104693 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f842c06ac90 0x7f842c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:14.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.299+0000 7f843f7fe640 1 --2- 172.21.15.57:0/809104693 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f842c06ac90 0x7f842c06d150 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f8434036000 tx=0x7f84340046c0 comp rx=0 tx=0).ready entity=mgr.7072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:14.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.303+0000 7f8427fff640 1 -- 172.21.15.57:0/809104693 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8438054f10 con 0x7f8440141690 2023-10-19T13:01:14.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.519+0000 7f8445b05640 1 -- 172.21.15.57:0/809104693 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f84400a9360 con 0x7f8440141690 2023-10-19T13:01:14.522 INFO:tasks.ceph.mon.a.smithi057.stderr:2023-10-19T13:01:14.519+0000 7f1253ff3640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-19T13:01:14.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.521+0000 7f8427fff640 1 -- 172.21.15.57:0/809104693 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v38) v1 ==== 161+0+0 (secure 0 0 0) 0x7f8438016090 con 0x7f8440141690 2023-10-19T13:01:14.523 INFO:teuthology.orchestra.run.smithi057.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-19T13:01:14.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.523+0000 7f8445b05640 1 -- 172.21.15.57:0/809104693 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f842c06ac90 msgr2=0x7f842c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:14.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.523+0000 7f8445b05640 1 --2- 172.21.15.57:0/809104693 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f842c06ac90 0x7f842c06d150 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f8434036000 tx=0x7f84340046c0 comp rx=0 tx=0).stop 2023-10-19T13:01:14.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.523+0000 7f8445b05640 1 -- 172.21.15.57:0/809104693 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8440141690 msgr2=0x7f84400ca870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:14.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.523+0000 7f8445b05640 1 --2- 172.21.15.57:0/809104693 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8440141690 0x7f84400ca870 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f843800cbc0 tx=0x7f843800a430 comp rx=0 tx=0).stop 2023-10-19T13:01:14.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.524+0000 7f8445b05640 1 -- 172.21.15.57:0/809104693 shutdown_connections 2023-10-19T13:01:14.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.524+0000 7f8445b05640 1 --2- 172.21.15.57:0/809104693 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f842c06ac90 0x7f842c06d150 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:14.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.524+0000 7f8445b05640 1 --2- 172.21.15.57:0/809104693 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8440141690 0x7f84400ca870 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:14.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.524+0000 7f8445b05640 1 --2- 172.21.15.57:0/809104693 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f844013e520 0x7f84400acb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:14.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.524+0000 7f8445b05640 1 --2- 172.21.15.57:0/809104693 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f844006c0a0 0x7f844013b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:14.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.524+0000 7f8445b05640 1 -- 172.21.15.57:0/809104693 >> 172.21.15.57:0/809104693 conn(0x7f8440069930 msgr2=0x7f84400ae910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:14.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.524+0000 7f8445b05640 1 -- 172.21.15.57:0/809104693 shutdown_connections 2023-10-19T13:01:14.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.524+0000 7f8445b05640 1 -- 172.21.15.57:0/809104693 wait complete. 2023-10-19T13:01:14.529 INFO:tasks.ceph.mds.a.smithi057.stderr: -1331> 2023-10-19T13:01:01.764+0000 7f9809bd3640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-19T13:01:14.538 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-19T13:01:14.553 INFO:tasks.ceph.mds.a.smithi057.stdout:starting mds.a at 2023-10-19T13:01:14.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.649+0000 7fc9e4d6c640 1 -- 172.21.15.57:0/2849155107 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc9cc003ef0 msgr2=0x7fc9cc0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:14.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.649+0000 7fc9e4d6c640 1 --2- 172.21.15.57:0/2849155107 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc9cc003ef0 0x7fc9cc0047c0 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7fc9c8009b50 tx=0x7fc9cc0057f0 comp rx=0 tx=0).stop 2023-10-19T13:01:14.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.649+0000 7fc9ded9d640 1 -- 172.21.15.57:0/2849155107 reap_dead start 2023-10-19T13:01:14.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.649+0000 7fc9e4d6c640 1 -- 172.21.15.57:0/2849155107 shutdown_connections 2023-10-19T13:01:14.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.649+0000 7fc9e4d6c640 1 -- 172.21.15.57:0/2849155107 >> 172.21.15.57:0/2849155107 conn(0x7fc9e0069930 msgr2=0x7fc9e0069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:14.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.649+0000 7fc9e4d6c640 1 -- 172.21.15.57:0/2849155107 shutdown_connections 2023-10-19T13:01:14.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.649+0000 7fc9e4d6c640 1 -- 172.21.15.57:0/2849155107 wait complete. 2023-10-19T13:01:14.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.651+0000 7fc9e4d6c640 1 Processor -- start 2023-10-19T13:01:14.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.651+0000 7fc9e4d6c640 1 -- start start 2023-10-19T13:01:14.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.651+0000 7fc9e4d6c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc9cc003ef0 0x7fc9e0134260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:14.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.651+0000 7fc9e4d6c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc9e015cbc0 0x7fc9e015d040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:14.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.651+0000 7fc9e4d6c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc9e0134e00 0x7fc9e01352b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:14.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.651+0000 7fc9e4d6c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9e015d580 con 0x7fc9e0134e00 2023-10-19T13:01:14.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.651+0000 7fc9e4d6c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9e013ccb0 con 0x7fc9e015cbc0 2023-10-19T13:01:14.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.651+0000 7fc9e4d6c640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9e013cf90 con 0x7fc9cc003ef0 2023-10-19T13:01:14.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.652+0000 7fc9ded9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc9e0134e00 0x7fc9e01352b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:14.658 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.652+0000 7fc9ded9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc9e0134e00 0x7fc9e01352b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56346/0 (socket says 172.21.15.57:56346) 2023-10-19T13:01:14.658 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.652+0000 7fc9ded9d640 1 -- 172.21.15.57:0/589010833 learned_addr learned my addr 172.21.15.57:0/589010833 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:01:14.658 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.652+0000 7fc9de59c640 1 --2- 172.21.15.57:0/589010833 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc9cc003ef0 0x7fc9e0134260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:14.658 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.652+0000 7fc9d5d9b640 1 --2- 172.21.15.57:0/589010833 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc9e015cbc0 0x7fc9e015d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:14.658 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.652+0000 7fc9ded9d640 1 -- 172.21.15.57:0/589010833 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc9cc003ef0 msgr2=0x7fc9e0134260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:14.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.652+0000 7fc9ded9d640 1 --2- 172.21.15.57:0/589010833 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc9cc003ef0 0x7fc9e0134260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:14.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.652+0000 7fc9ded9d640 1 -- 172.21.15.57:0/589010833 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc9e015cbc0 msgr2=0x7fc9e015d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:14.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.652+0000 7fc9ded9d640 1 --2- 172.21.15.57:0/589010833 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc9e015cbc0 0x7fc9e015d040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:14.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.652+0000 7fc9ded9d640 1 -- 172.21.15.57:0/589010833 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9e013d270 con 0x7fc9e0134e00 2023-10-19T13:01:14.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.652+0000 7fc9d5d9b640 1 --2- 172.21.15.57:0/589010833 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc9e015cbc0 0x7fc9e015d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:01:14.660 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.652+0000 7fc9de59c640 1 --2- 172.21.15.57:0/589010833 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc9cc003ef0 0x7fc9e0134260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:01:14.660 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.652+0000 7fc9ded9d640 1 --2- 172.21.15.57:0/589010833 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc9e0134e00 0x7fc9e01352b0 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7fc9e00cc150 tx=0x7fc9d8010f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:14.660 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.652+0000 7fc9d77fe640 1 -- 172.21.15.57:0/589010833 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9d800ee70 con 0x7fc9e0134e00 2023-10-19T13:01:14.660 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.652+0000 7fc9d77fe640 1 -- 172.21.15.57:0/589010833 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fc9d8003a80 con 0x7fc9e0134e00 2023-10-19T13:01:14.660 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.652+0000 7fc9e4d6c640 1 -- 172.21.15.57:0/589010833 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9e013d550 con 0x7fc9e0134e00 2023-10-19T13:01:14.661 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.652+0000 7fc9d77fe640 1 -- 172.21.15.57:0/589010833 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9d8003e90 con 0x7fc9e0134e00 2023-10-19T13:01:14.661 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.652+0000 7fc9e4d6c640 1 -- 172.21.15.57:0/589010833 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9e01347a0 con 0x7fc9e0134e00 2023-10-19T13:01:14.661 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.653+0000 7fc9e4d6c640 1 -- 172.21.15.57:0/589010833 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9e00690e0 con 0x7fc9e0134e00 2023-10-19T13:01:14.662 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.654+0000 7fc9d77fe640 1 -- 172.21.15.57:0/589010833 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 46) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fc9d8019030 con 0x7fc9e0134e00 2023-10-19T13:01:14.662 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.654+0000 7fc9d77fe640 1 --2- 172.21.15.57:0/589010833 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7fc9c006acc0 0x7fc9c006d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:14.662 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.654+0000 7fc9d77fe640 1 -- 172.21.15.57:0/589010833 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 6831+0+0 (secure 0 0 0) 0x7fc9d8017070 con 0x7fc9e0134e00 2023-10-19T13:01:14.663 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.654+0000 7fc9de59c640 1 --2- 172.21.15.57:0/589010833 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7fc9c006acc0 0x7fc9c006d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:14.663 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.655+0000 7fc9de59c640 1 --2- 172.21.15.57:0/589010833 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7fc9c006acc0 0x7fc9c006d180 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc9d0001050 tx=0x7fc9d0001e70 comp rx=0 tx=0).ready entity=mgr.7072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:14.663 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.659+0000 7fc9d77fe640 1 -- 172.21.15.57:0/589010833 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc9d8054fd0 con 0x7fc9e0134e00 2023-10-19T13:01:14.763 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:14.761+0000 7fb508163640 -1 client.7141: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.7030:f55302c0-37aa-4235-8ccc-9775bc542785:v1:172.21.15.179:0/3304791412] 2023-10-19T13:01:14.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:14.875+0000 7fc9e4d6c640 1 -- 172.21.15.57:0/589010833 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fc9e00cc420 con 0x7fc9e0134e00 2023-10-19T13:01:15.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:15.076+0000 7fc9d77fe640 1 -- 172.21.15.57:0/589010833 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mgrmap(e 47) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fc9d80553a0 con 0x7fc9e0134e00 2023-10-19T13:01:15.435 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:15.433+0000 7fb508163640 -1 client.7141: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.7030:f55302c0-37aa-4235-8ccc-9775bc542785:v1:172.21.15.179:0/3304791412] 2023-10-19T13:01:15.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:15.522+0000 7fc9d77fe640 1 -- 172.21.15.57:0/589010833 <== mon.0 v2:172.21.15.57:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v39) v1 ==== 104+0+0 (secure 0 0 0) 0x7fc9d8071e20 con 0x7fc9e0134e00 2023-10-19T13:01:15.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:15.524+0000 7fc9e4d6c640 1 -- 172.21.15.57:0/589010833 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7fc9c006acc0 msgr2=0x7fc9c006d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:15.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:15.524+0000 7fc9e4d6c640 1 --2- 172.21.15.57:0/589010833 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7fc9c006acc0 0x7fc9c006d180 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc9d0001050 tx=0x7fc9d0001e70 comp rx=0 tx=0).stop 2023-10-19T13:01:15.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:15.524+0000 7fc9e4d6c640 1 -- 172.21.15.57:0/589010833 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc9e0134e00 msgr2=0x7fc9e01352b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:15.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:15.524+0000 7fc9e4d6c640 1 --2- 172.21.15.57:0/589010833 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc9e0134e00 0x7fc9e01352b0 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7fc9e00cc150 tx=0x7fc9d8010f40 comp rx=0 tx=0).stop 2023-10-19T13:01:15.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:15.524+0000 7fc9e4d6c640 1 -- 172.21.15.57:0/589010833 shutdown_connections 2023-10-19T13:01:15.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:15.524+0000 7fc9e4d6c640 1 --2- 172.21.15.57:0/589010833 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7fc9c006acc0 0x7fc9c006d180 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:15.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:15.524+0000 7fc9e4d6c640 1 --2- 172.21.15.57:0/589010833 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc9e0134e00 0x7fc9e01352b0 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:15.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:15.524+0000 7fc9e4d6c640 1 --2- 172.21.15.57:0/589010833 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc9e015cbc0 0x7fc9e015d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:15.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:15.524+0000 7fc9e4d6c640 1 --2- 172.21.15.57:0/589010833 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc9cc003ef0 0x7fc9e0134260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:15.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:15.524+0000 7fc9e4d6c640 1 -- 172.21.15.57:0/589010833 >> 172.21.15.57:0/589010833 conn(0x7fc9e0069930 msgr2=0x7fc9e0152bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:15.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:15.524+0000 7fc9e4d6c640 1 -- 172.21.15.57:0/589010833 shutdown_connections 2023-10-19T13:01:15.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:15.524+0000 7fc9e4d6c640 1 -- 172.21.15.57:0/589010833 wait complete. 2023-10-19T13:01:15.538 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-19T13:01:15.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:15.691+0000 7fc645955640 1 -- 172.21.15.57:0/4210827894 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc64015b3e0 msgr2=0x7fc6400d1810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:15.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:15.691+0000 7fc645955640 1 --2- 172.21.15.57:0/4210827894 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc64015b3e0 0x7fc6400d1810 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7fc638009830 tx=0x7fc63802bdf0 comp rx=0 tx=0).stop 2023-10-19T13:01:15.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:15.691+0000 7fc645955640 1 -- 172.21.15.57:0/4210827894 shutdown_connections 2023-10-19T13:01:15.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:15.691+0000 7fc645955640 1 --2- 172.21.15.57:0/4210827894 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc64015b3e0 0x7fc6400d1810 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:15.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:15.691+0000 7fc645955640 1 -- 172.21.15.57:0/4210827894 >> 172.21.15.57:0/4210827894 conn(0x7fc6400d0cf0 msgr2=0x7fc6400d1100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:15.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:15.691+0000 7fc645955640 1 -- 172.21.15.57:0/4210827894 shutdown_connections 2023-10-19T13:01:15.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:15.691+0000 7fc645955640 1 -- 172.21.15.57:0/4210827894 wait complete. 2023-10-19T13:01:15.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:15.692+0000 7fc645955640 1 Processor -- start 2023-10-19T13:01:15.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:15.692+0000 7fc645955640 1 -- start start 2023-10-19T13:01:15.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:15.693+0000 7fc645955640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc6400cf9e0 0x7fc640154ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:15.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:15.693+0000 7fc645955640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc64015b000 0x7fc6400c6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:15.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:15.693+0000 7fc645955640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc64015b3e0 0x7fc64015b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:15.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:15.693+0000 7fc645955640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6400b46c0 con 0x7fc64015b3e0 2023-10-19T13:01:15.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:15.693+0000 7fc645955640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6401553e0 con 0x7fc64015b000 2023-10-19T13:01:15.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:15.693+0000 7fc645955640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6400b3a40 con 0x7fc6400cf9e0 2023-10-19T13:01:15.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:15.693+0000 7fc63f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc64015b3e0 0x7fc64015b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:15.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:15.693+0000 7fc63f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc64015b3e0 0x7fc64015b890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56360/0 (socket says 172.21.15.57:56360) 2023-10-19T13:01:15.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:15.693+0000 7fc63f7fe640 1 -- 172.21.15.57:0/3143954862 learned_addr learned my addr 172.21.15.57:0/3143954862 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:01:15.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:15.693+0000 7fc63e7fc640 1 --2- 172.21.15.57:0/3143954862 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc64015b000 0x7fc6400c6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:15.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:15.693+0000 7fc63effd640 1 --2- 172.21.15.57:0/3143954862 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc6400cf9e0 0x7fc640154ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:15.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:15.693+0000 7fc63f7fe640 1 -- 172.21.15.57:0/3143954862 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc6400cf9e0 msgr2=0x7fc640154ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:15.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:15.694+0000 7fc63f7fe640 1 --2- 172.21.15.57:0/3143954862 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc6400cf9e0 0x7fc640154ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:15.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:15.694+0000 7fc63f7fe640 1 -- 172.21.15.57:0/3143954862 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc64015b000 msgr2=0x7fc6400c6070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:15.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:15.694+0000 7fc63f7fe640 1 --2- 172.21.15.57:0/3143954862 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc64015b000 0x7fc6400c6070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:15.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:15.694+0000 7fc63f7fe640 1 -- 172.21.15.57:0/3143954862 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc638009490 con 0x7fc64015b3e0 2023-10-19T13:01:15.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:15.694+0000 7fc63f7fe640 1 --2- 172.21.15.57:0/3143954862 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc64015b3e0 0x7fc64015b890 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7fc638009950 tx=0x7fc638003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:15.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:15.694+0000 7fc637fff640 1 -- 172.21.15.57:0/3143954862 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc638005cb0 con 0x7fc64015b3e0 2023-10-19T13:01:15.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:15.694+0000 7fc637fff640 1 -- 172.21.15.57:0/3143954862 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fc638034440 con 0x7fc64015b3e0 2023-10-19T13:01:15.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:15.694+0000 7fc645955640 1 -- 172.21.15.57:0/3143954862 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6400b3d20 con 0x7fc64015b3e0 2023-10-19T13:01:15.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:15.694+0000 7fc637fff640 1 -- 172.21.15.57:0/3143954862 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc638034850 con 0x7fc64015b3e0 2023-10-19T13:01:15.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:15.694+0000 7fc645955640 1 -- 172.21.15.57:0/3143954862 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6400b42e0 con 0x7fc64015b3e0 2023-10-19T13:01:15.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:15.695+0000 7fc645955640 1 -- 172.21.15.57:0/3143954862 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6400b4800 con 0x7fc64015b3e0 2023-10-19T13:01:15.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:15.695+0000 7fc637fff640 1 -- 172.21.15.57:0/3143954862 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 47) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fc63803b040 con 0x7fc64015b3e0 2023-10-19T13:01:15.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:15.696+0000 7fc637fff640 1 --2- 172.21.15.57:0/3143954862 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7fc62006aed0 0x7fc62006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:15.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:15.696+0000 7fc637fff640 1 -- 172.21.15.57:0/3143954862 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 6831+0+0 (secure 0 0 0) 0x7fc6380abec0 con 0x7fc64015b3e0 2023-10-19T13:01:15.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:15.696+0000 7fc63effd640 1 --2- 172.21.15.57:0/3143954862 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7fc62006aed0 0x7fc62006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:15.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:15.696+0000 7fc63effd640 1 --2- 172.21.15.57:0/3143954862 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7fc62006aed0 0x7fc62006d390 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc630001560 tx=0x7fc630009040 comp rx=0 tx=0).ready entity=mgr.7072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:15.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:15.700+0000 7fc637fff640 1 -- 172.21.15.57:0/3143954862 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc638076bb0 con 0x7fc64015b3e0 2023-10-19T13:01:15.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:15.879+0000 7fc645955640 1 -- 172.21.15.57:0/3143954862 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc6400690e0 con 0x7fc64015b3e0 2023-10-19T13:01:16.113 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:16.111+0000 7fb508163640 -1 client.7141: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.7030:f55302c0-37aa-4235-8ccc-9775bc542785:v1:172.21.15.179:0/3304791412] 2023-10-19T13:01:16.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:16.684+0000 7fc637fff640 1 -- 172.21.15.57:0/3143954862 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v104) v1 ==== 181+0+0 (secure 0 0 0) 0x7fc6380390d0 con 0x7fc64015b3e0 2023-10-19T13:01:16.687 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'cephfs_metadata' removed 2023-10-19T13:01:16.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:16.687+0000 7fc645955640 1 -- 172.21.15.57:0/3143954862 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7fc62006aed0 msgr2=0x7fc62006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:16.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:16.687+0000 7fc645955640 1 --2- 172.21.15.57:0/3143954862 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7fc62006aed0 0x7fc62006d390 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc630001560 tx=0x7fc630009040 comp rx=0 tx=0).stop 2023-10-19T13:01:16.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:16.687+0000 7fc645955640 1 -- 172.21.15.57:0/3143954862 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc64015b3e0 msgr2=0x7fc64015b890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:16.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:16.687+0000 7fc645955640 1 --2- 172.21.15.57:0/3143954862 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc64015b3e0 0x7fc64015b890 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7fc638009950 tx=0x7fc638003820 comp rx=0 tx=0).stop 2023-10-19T13:01:16.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:16.687+0000 7fc645955640 1 -- 172.21.15.57:0/3143954862 shutdown_connections 2023-10-19T13:01:16.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:16.687+0000 7fc645955640 1 --2- 172.21.15.57:0/3143954862 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7fc62006aed0 0x7fc62006d390 secure :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc630001560 tx=0x7fc630009040 comp rx=0 tx=0).stop 2023-10-19T13:01:16.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:16.687+0000 7fc645955640 1 --2- 172.21.15.57:0/3143954862 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc64015b3e0 0x7fc64015b890 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:16.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:16.687+0000 7fc645955640 1 --2- 172.21.15.57:0/3143954862 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc64015b000 0x7fc6400c6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:16.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:16.687+0000 7fc645955640 1 --2- 172.21.15.57:0/3143954862 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc6400cf9e0 0x7fc640154ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:16.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:16.687+0000 7fc645955640 1 -- 172.21.15.57:0/3143954862 >> 172.21.15.57:0/3143954862 conn(0x7fc6400d0cf0 msgr2=0x7fc6400d12b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:16.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:16.688+0000 7fc645955640 1 -- 172.21.15.57:0/3143954862 shutdown_connections 2023-10-19T13:01:16.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:16.688+0000 7fc645955640 1 -- 172.21.15.57:0/3143954862 wait complete. 2023-10-19T13:01:16.703 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-19T13:01:16.792 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:16.789+0000 7fb508163640 -1 client.7141: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.7030:f55302c0-37aa-4235-8ccc-9775bc542785:v1:172.21.15.179:0/3304791412] 2023-10-19T13:01:16.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:16.816+0000 7fa326497640 1 -- 172.21.15.57:0/4121173577 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa320156790 msgr2=0x7fa3200cc990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:16.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:16.816+0000 7fa326497640 1 --2- 172.21.15.57:0/4121173577 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa320156790 0x7fa3200cc990 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7fa320061510 tx=0x7fa30802be20 comp rx=0 tx=0).stop 2023-10-19T13:01:16.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:16.816+0000 7fa326497640 1 -- 172.21.15.57:0/4121173577 shutdown_connections 2023-10-19T13:01:16.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:16.816+0000 7fa326497640 1 --2- 172.21.15.57:0/4121173577 >> v2:172.21.15.179:3300/0 conn(0x7fa3200ac870 0x7fa32006b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:16.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:16.816+0000 7fa326497640 1 --2- 172.21.15.57:0/4121173577 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa320156790 0x7fa3200cc990 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:16.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:16.816+0000 7fa326497640 1 -- 172.21.15.57:0/4121173577 >> 172.21.15.57:0/4121173577 conn(0x7fa32006fae0 msgr2=0x7fa32006fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:16.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:16.816+0000 7fa326497640 1 -- 172.21.15.57:0/4121173577 shutdown_connections 2023-10-19T13:01:16.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:16.816+0000 7fa326497640 1 -- 172.21.15.57:0/4121173577 wait complete. 2023-10-19T13:01:16.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:16.817+0000 7fa326497640 1 Processor -- start 2023-10-19T13:01:16.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:16.817+0000 7fa326497640 1 -- start start 2023-10-19T13:01:16.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:16.818+0000 7fa326497640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa32006bec0 0x7fa320141ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:16.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:16.818+0000 7fa326497640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa3200ac870 0x7fa3200a8010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:16.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:16.818+0000 7fa326497640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa320156790 0x7fa3200a8550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:16.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:16.818+0000 7fa326497640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3200aa520 con 0x7fa3200ac870 2023-10-19T13:01:16.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:16.818+0000 7fa326497640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa320142220 con 0x7fa32006bec0 2023-10-19T13:01:16.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:16.818+0000 7fa326497640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa32006cf00 con 0x7fa320156790 2023-10-19T13:01:16.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:16.818+0000 7fa31f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa3200ac870 0x7fa3200a8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:16.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:16.818+0000 7fa31f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa3200ac870 0x7fa3200a8010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:58416/0 (socket says 172.21.15.57:58416) 2023-10-19T13:01:16.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:16.818+0000 7fa31f7fe640 1 -- 172.21.15.57:0/2489114008 learned_addr learned my addr 172.21.15.57:0/2489114008 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:01:16.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:16.818+0000 7fa324a34640 1 --2- 172.21.15.57:0/2489114008 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa320156790 0x7fa3200a8550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:16.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:16.818+0000 7fa31ffff640 1 --2- 172.21.15.57:0/2489114008 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa32006bec0 0x7fa320141ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:16.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:16.818+0000 7fa31f7fe640 1 -- 172.21.15.57:0/2489114008 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa320156790 msgr2=0x7fa3200a8550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:16.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:16.818+0000 7fa31f7fe640 1 --2- 172.21.15.57:0/2489114008 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa320156790 0x7fa3200a8550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:16.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:16.818+0000 7fa31f7fe640 1 -- 172.21.15.57:0/2489114008 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa32006bec0 msgr2=0x7fa320141ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:16.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:16.818+0000 7fa31f7fe640 1 --2- 172.21.15.57:0/2489114008 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa32006bec0 0x7fa320141ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:16.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:16.818+0000 7fa31f7fe640 1 -- 172.21.15.57:0/2489114008 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa308009560 con 0x7fa3200ac870 2023-10-19T13:01:16.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:16.818+0000 7fa31f7fe640 1 --2- 172.21.15.57:0/2489114008 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa3200ac870 0x7fa3200a8010 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7fa31000d950 tx=0x7fa31000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:16.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:16.819+0000 7fa31cff9640 1 -- 172.21.15.57:0/2489114008 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa310009c30 con 0x7fa3200ac870 2023-10-19T13:01:16.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:16.819+0000 7fa31cff9640 1 -- 172.21.15.57:0/2489114008 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fa310009e80 con 0x7fa3200ac870 2023-10-19T13:01:16.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:16.819+0000 7fa326497640 1 -- 172.21.15.57:0/2489114008 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa32006d1e0 con 0x7fa3200ac870 2023-10-19T13:01:16.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:16.819+0000 7fa31cff9640 1 -- 172.21.15.57:0/2489114008 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3100106c0 con 0x7fa3200ac870 2023-10-19T13:01:16.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:16.819+0000 7fa326497640 1 -- 172.21.15.57:0/2489114008 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa32006d7a0 con 0x7fa3200ac870 2023-10-19T13:01:16.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:16.820+0000 7fa326497640 1 -- 172.21.15.57:0/2489114008 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa32006d9c0 con 0x7fa3200ac870 2023-10-19T13:01:16.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:16.820+0000 7fa31cff9640 1 -- 172.21.15.57:0/2489114008 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 47) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fa310010820 con 0x7fa3200ac870 2023-10-19T13:01:16.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:16.821+0000 7fa31cff9640 1 --2- 172.21.15.57:0/2489114008 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7fa30006af00 0x7fa30006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:16.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:16.821+0000 7fa31cff9640 1 -- 172.21.15.57:0/2489114008 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6393+0+0 (secure 0 0 0) 0x7fa310089f10 con 0x7fa3200ac870 2023-10-19T13:01:16.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:16.821+0000 7fa31ffff640 1 --2- 172.21.15.57:0/2489114008 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7fa30006af00 0x7fa30006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:16.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:16.821+0000 7fa31ffff640 1 --2- 172.21.15.57:0/2489114008 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7fa30006af00 0x7fa30006d3c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa308035000 tx=0x7fa308004730 comp rx=0 tx=0).ready entity=mgr.7072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:16.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:16.825+0000 7fa31cff9640 1 -- 172.21.15.57:0/2489114008 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa310054b90 con 0x7fa3200ac870 2023-10-19T13:01:17.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:17.006+0000 7fa326497640 1 -- 172.21.15.57:0/2489114008 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fa3200665b0 con 0x7fa3200ac870 2023-10-19T13:01:17.469 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:17.466+0000 7fb508163640 -1 client.7141: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.7030:f55302c0-37aa-4235-8ccc-9775bc542785:v1:172.21.15.179:0/3304791412] 2023-10-19T13:01:17.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:17.685+0000 7fa31cff9640 1 -- 172.21.15.57:0/2489114008 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v105) v1 ==== 169+0+0 (secure 0 0 0) 0x7fa3100581b0 con 0x7fa3200ac870 2023-10-19T13:01:17.689 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'cephfs_data' removed 2023-10-19T13:01:17.689 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:01:17.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:17.688+0000 7fa326497640 1 -- 172.21.15.57:0/2489114008 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7fa30006af00 msgr2=0x7fa30006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:17.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:17.688+0000 7fa326497640 1 --2- 172.21.15.57:0/2489114008 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7fa30006af00 0x7fa30006d3c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa308035000 tx=0x7fa308004730 comp rx=0 tx=0).stop 2023-10-19T13:01:17.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:17.688+0000 7fa326497640 1 -- 172.21.15.57:0/2489114008 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa3200ac870 msgr2=0x7fa3200a8010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:17.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:17.688+0000 7fa326497640 1 --2- 172.21.15.57:0/2489114008 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa3200ac870 0x7fa3200a8010 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7fa31000d950 tx=0x7fa31000de10 comp rx=0 tx=0).stop 2023-10-19T13:01:17.692 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:01:17.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:17.688+0000 7fa326497640 1 -- 172.21.15.57:0/2489114008 shutdown_connections 2023-10-19T13:01:17.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:17.688+0000 7fa326497640 1 --2- 172.21.15.57:0/2489114008 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7fa30006af00 0x7fa30006d3c0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:17.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:17.688+0000 7fa326497640 1 --2- 172.21.15.57:0/2489114008 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa320156790 0x7fa3200a8550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:17.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:17.688+0000 7fa326497640 1 --2- 172.21.15.57:0/2489114008 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa3200ac870 0x7fa3200a8010 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:17.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:17.688+0000 7fa326497640 1 --2- 172.21.15.57:0/2489114008 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa32006bec0 0x7fa320141ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:17.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:17.688+0000 7fa326497640 1 -- 172.21.15.57:0/2489114008 >> 172.21.15.57:0/2489114008 conn(0x7fa32006fae0 msgr2=0x7fa320156c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:17.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:17.688+0000 7fa326497640 1 -- 172.21.15.57:0/2489114008 shutdown_connections 2023-10-19T13:01:17.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:17.689+0000 7fa326497640 1 -- 172.21.15.57:0/2489114008 wait complete. 2023-10-19T13:01:17.709 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 30 2023-10-19T13:01:17.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:17.892+0000 7f4fdc973640 1 -- 172.21.15.57:0/2450350515 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4fd806af00 msgr2=0x7f4fd806b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:17.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:17.892+0000 7f4fdc973640 1 --2- 172.21.15.57:0/2450350515 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4fd806af00 0x7f4fd806b2e0 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f4fc8005fd0 tx=0x7f4fc802bd90 comp rx=0 tx=0).stop 2023-10-19T13:01:17.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:17.892+0000 7f4fdc973640 1 -- 172.21.15.57:0/2450350515 shutdown_connections 2023-10-19T13:01:17.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:17.892+0000 7f4fdc973640 1 --2- 172.21.15.57:0/2450350515 >> v2:172.21.15.179:3300/0 conn(0x7f4fd80c96b0 0x7f4fd80c9b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:17.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:17.892+0000 7f4fdc973640 1 --2- 172.21.15.57:0/2450350515 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4fd806af00 0x7f4fd806b2e0 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:17.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:17.892+0000 7f4fdc973640 1 -- 172.21.15.57:0/2450350515 >> 172.21.15.57:0/2450350515 conn(0x7f4fd8069930 msgr2=0x7f4fd8069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:17.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:17.892+0000 7f4fdc973640 1 -- 172.21.15.57:0/2450350515 shutdown_connections 2023-10-19T13:01:17.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:17.892+0000 7f4fdc973640 1 -- 172.21.15.57:0/2450350515 wait complete. 2023-10-19T13:01:17.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:17.893+0000 7f4fdc973640 1 Processor -- start 2023-10-19T13:01:17.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:17.893+0000 7f4fdc973640 1 -- start start 2023-10-19T13:01:17.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:17.894+0000 7f4fdc973640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4fd806af00 0x7f4fd81efc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:17.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:17.894+0000 7f4fdc973640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4fd80ab500 0x7f4fd80a8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:17.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:17.894+0000 7f4fdc973640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4fd80c96b0 0x7f4fd80aa510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:17.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:17.894+0000 7f4fdc973640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fd80ab2d0 con 0x7f4fd80c96b0 2023-10-19T13:01:17.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:17.894+0000 7f4fdc973640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fd80aaff0 con 0x7f4fd80ab500 2023-10-19T13:01:17.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:17.894+0000 7f4fdc973640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fd81ef160 con 0x7f4fd806af00 2023-10-19T13:01:17.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:17.894+0000 7f4fd6d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4fd80c96b0 0x7f4fd80aa510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:17.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:17.894+0000 7f4fd6d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4fd80c96b0 0x7f4fd80aa510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:58430/0 (socket says 172.21.15.57:58430) 2023-10-19T13:01:17.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:17.894+0000 7f4fd6d9d640 1 -- 172.21.15.57:0/1811850576 learned_addr learned my addr 172.21.15.57:0/1811850576 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:01:17.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:17.894+0000 7f4fcdd9b640 1 --2- 172.21.15.57:0/1811850576 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4fd80ab500 0x7f4fd80a8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:17.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:17.894+0000 7f4fd659c640 1 --2- 172.21.15.57:0/1811850576 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4fd806af00 0x7f4fd81efc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:17.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:17.894+0000 7f4fd6d9d640 1 -- 172.21.15.57:0/1811850576 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4fd806af00 msgr2=0x7f4fd81efc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:17.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:17.894+0000 7f4fd6d9d640 1 --2- 172.21.15.57:0/1811850576 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4fd806af00 0x7f4fd81efc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:17.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:17.894+0000 7f4fd6d9d640 1 -- 172.21.15.57:0/1811850576 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4fd80ab500 msgr2=0x7f4fd80a8aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:17.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:17.894+0000 7f4fd6d9d640 1 --2- 172.21.15.57:0/1811850576 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4fd80ab500 0x7f4fd80a8aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:17.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:17.894+0000 7f4fd6d9d640 1 -- 172.21.15.57:0/1811850576 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fc8009560 con 0x7f4fd80c96b0 2023-10-19T13:01:17.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:17.895+0000 7f4fd6d9d640 1 --2- 172.21.15.57:0/1811850576 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4fd80c96b0 0x7f4fd80aa510 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f4fd000c7a0 tx=0x7f4fd000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:17.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:17.895+0000 7f4fcf7fe640 1 -- 172.21.15.57:0/1811850576 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fd0003b70 con 0x7f4fd80c96b0 2023-10-19T13:01:17.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:17.895+0000 7f4fcf7fe640 1 -- 172.21.15.57:0/1811850576 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f4fd0003cf0 con 0x7f4fd80c96b0 2023-10-19T13:01:17.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:17.895+0000 7f4fcf7fe640 1 -- 172.21.15.57:0/1811850576 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fd000aab0 con 0x7f4fd80c96b0 2023-10-19T13:01:17.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:17.895+0000 7f4fdc973640 1 -- 172.21.15.57:0/1811850576 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fd81ef440 con 0x7f4fd80c96b0 2023-10-19T13:01:17.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:17.895+0000 7f4fdc973640 1 -- 172.21.15.57:0/1811850576 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fd80a8fe0 con 0x7f4fd80c96b0 2023-10-19T13:01:17.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:17.896+0000 7f4fdc973640 1 -- 172.21.15.57:0/1811850576 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fd81ef960 con 0x7f4fd80c96b0 2023-10-19T13:01:17.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:17.896+0000 7f4fcf7fe640 1 -- 172.21.15.57:0/1811850576 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 47) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f4fd000ac10 con 0x7f4fd80c96b0 2023-10-19T13:01:17.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:17.897+0000 7f4fcf7fe640 1 --2- 172.21.15.57:0/1811850576 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f4fb806aed0 0x7f4fb806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:17.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:17.897+0000 7f4fcf7fe640 1 -- 172.21.15.57:0/1811850576 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5979+0+0 (secure 0 0 0) 0x7f4fd0089770 con 0x7f4fd80c96b0 2023-10-19T13:01:17.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:17.897+0000 7f4fd659c640 1 --2- 172.21.15.57:0/1811850576 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f4fb806aed0 0x7f4fb806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:17.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:17.897+0000 7f4fd659c640 1 --2- 172.21.15.57:0/1811850576 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f4fb806aed0 0x7f4fb806d390 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f4fc8035000 tx=0x7f4fc8004730 comp rx=0 tx=0).ready entity=mgr.7072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:17.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:17.901+0000 7f4fcf7fe640 1 -- 172.21.15.57:0/1811850576 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4fd81ef960 con 0x7f4fd80c96b0 2023-10-19T13:01:18.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.089+0000 7f4fdc973640 1 -- 172.21.15.57:0/1811850576 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "config reset", "num": 30} v 0) v1 -- 0x7f4fd81ef960 con 0x7f4fd80c96b0 2023-10-19T13:01:18.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.092+0000 7f4fcf7fe640 1 -- 172.21.15.57:0/1811850576 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 30}]=0 v46) v1 ==== 71+0+0 (secure 0 0 0) 0x7f4fd81ef960 con 0x7f4fd80c96b0 2023-10-19T13:01:18.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.094+0000 7f4fdc973640 1 -- 172.21.15.57:0/1811850576 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f4fb806aed0 msgr2=0x7f4fb806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:18.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.094+0000 7f4fdc973640 1 --2- 172.21.15.57:0/1811850576 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f4fb806aed0 0x7f4fb806d390 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f4fc8035000 tx=0x7f4fc8004730 comp rx=0 tx=0).stop 2023-10-19T13:01:18.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.094+0000 7f4fdc973640 1 -- 172.21.15.57:0/1811850576 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4fd80c96b0 msgr2=0x7f4fd80aa510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:18.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.094+0000 7f4fdc973640 1 --2- 172.21.15.57:0/1811850576 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4fd80c96b0 0x7f4fd80aa510 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f4fd000c7a0 tx=0x7f4fd000cc60 comp rx=0 tx=0).stop 2023-10-19T13:01:18.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.094+0000 7f4fdc973640 1 -- 172.21.15.57:0/1811850576 shutdown_connections 2023-10-19T13:01:18.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.094+0000 7f4fdc973640 1 --2- 172.21.15.57:0/1811850576 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f4fb806aed0 0x7f4fb806d390 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:18.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.094+0000 7f4fdc973640 1 --2- 172.21.15.57:0/1811850576 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4fd80c96b0 0x7f4fd80aa510 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:18.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.094+0000 7f4fdc973640 1 --2- 172.21.15.57:0/1811850576 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4fd80ab500 0x7f4fd80a8aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:18.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.094+0000 7f4fdc973640 1 --2- 172.21.15.57:0/1811850576 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4fd806af00 0x7f4fd81efc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:18.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.094+0000 7f4fdc973640 1 -- 172.21.15.57:0/1811850576 >> 172.21.15.57:0/1811850576 conn(0x7f4fd8069930 msgr2=0x7f4fd8069d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:18.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.094+0000 7f4fdc973640 1 -- 172.21.15.57:0/1811850576 shutdown_connections 2023-10-19T13:01:18.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.094+0000 7f4fdc973640 1 -- 172.21.15.57:0/1811850576 wait complete. 2023-10-19T13:01:18.107 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list' 2023-10-19T13:01:18.161 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:18.158+0000 7fb508163640 -1 client.7141: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.7030:f55302c0-37aa-4235-8ccc-9775bc542785:v1:172.21.15.179:0/3304791412] 2023-10-19T13:01:18.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.217+0000 7f7625d55640 1 -- 172.21.15.57:0/4005257265 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f76200d1910 msgr2=0x7f76200b0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:18.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.217+0000 7f7625d55640 1 --2- 172.21.15.57:0/4005257265 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f76200d1910 0x7f76200b0650 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f7618009be0 tx=0x7f761802c0c0 comp rx=0 tx=0).stop 2023-10-19T13:01:18.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.217+0000 7f7625d55640 1 -- 172.21.15.57:0/4005257265 shutdown_connections 2023-10-19T13:01:18.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.217+0000 7f7625d55640 1 --2- 172.21.15.57:0/4005257265 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f76200d1910 0x7f76200b0650 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:18.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.217+0000 7f7625d55640 1 -- 172.21.15.57:0/4005257265 >> 172.21.15.57:0/4005257265 conn(0x7f76200d0d60 msgr2=0x7f76200d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:18.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.217+0000 7f7625d55640 1 -- 172.21.15.57:0/4005257265 shutdown_connections 2023-10-19T13:01:18.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.218+0000 7f7625d55640 1 -- 172.21.15.57:0/4005257265 wait complete. 2023-10-19T13:01:18.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.219+0000 7f7625d55640 1 Processor -- start 2023-10-19T13:01:18.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.219+0000 7f7625d55640 1 -- start start 2023-10-19T13:01:18.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.219+0000 7f7625d55640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f76200cf340 0x7f762009e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:18.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.219+0000 7f7625d55640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f76200d1910 0x7f7620156b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:18.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.219+0000 7f7625d55640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f762015b2a0 0x7f7620157040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:18.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.219+0000 7f7625d55640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f762009e690 con 0x7f76200cf340 2023-10-19T13:01:18.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.219+0000 7f7625d55640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76200b5250 con 0x7f762015b2a0 2023-10-19T13:01:18.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.219+0000 7f7625d55640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76200b56e0 con 0x7f76200d1910 2023-10-19T13:01:18.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.220+0000 7f761f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f76200cf340 0x7f762009e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:18.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.220+0000 7f761f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f76200cf340 0x7f762009e150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:58440/0 (socket says 172.21.15.57:58440) 2023-10-19T13:01:18.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.220+0000 7f761f7fe640 1 -- 172.21.15.57:0/1370673019 learned_addr learned my addr 172.21.15.57:0/1370673019 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:01:18.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.220+0000 7f761effd640 1 --2- 172.21.15.57:0/1370673019 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f76200d1910 0x7f7620156b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:18.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.220+0000 7f761ffff640 1 --2- 172.21.15.57:0/1370673019 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f762015b2a0 0x7f7620157040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:18.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.220+0000 7f761f7fe640 1 -- 172.21.15.57:0/1370673019 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f76200d1910 msgr2=0x7f7620156b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:18.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.220+0000 7f761f7fe640 1 --2- 172.21.15.57:0/1370673019 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f76200d1910 0x7f7620156b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:18.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.220+0000 7f761f7fe640 1 -- 172.21.15.57:0/1370673019 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f762015b2a0 msgr2=0x7f7620157040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:18.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.220+0000 7f761f7fe640 1 --2- 172.21.15.57:0/1370673019 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f762015b2a0 0x7f7620157040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:18.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.220+0000 7f761f7fe640 1 -- 172.21.15.57:0/1370673019 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7618009840 con 0x7f76200cf340 2023-10-19T13:01:18.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.220+0000 7f761f7fe640 1 --2- 172.21.15.57:0/1370673019 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f76200cf340 0x7f762009e150 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f761000c4c0 tx=0x7f761000c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:18.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.220+0000 7f7607fff640 1 -- 172.21.15.57:0/1370673019 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76100052d0 con 0x7f76200cf340 2023-10-19T13:01:18.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.220+0000 7f7607fff640 1 -- 172.21.15.57:0/1370673019 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f7610005430 con 0x7f76200cf340 2023-10-19T13:01:18.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.220+0000 7f7625d55640 1 -- 172.21.15.57:0/1370673019 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76200b4360 con 0x7f76200cf340 2023-10-19T13:01:18.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.220+0000 7f7607fff640 1 -- 172.21.15.57:0/1370673019 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7610012890 con 0x7f76200cf340 2023-10-19T13:01:18.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.221+0000 7f7625d55640 1 -- 172.21.15.57:0/1370673019 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76200b4920 con 0x7f76200cf340 2023-10-19T13:01:18.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.221+0000 7f7625d55640 1 -- 172.21.15.57:0/1370673019 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76200b6c60 con 0x7f76200cf340 2023-10-19T13:01:18.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.222+0000 7f7607fff640 1 -- 172.21.15.57:0/1370673019 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 47) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f76100035f0 con 0x7f76200cf340 2023-10-19T13:01:18.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.222+0000 7f7607fff640 1 --2- 172.21.15.57:0/1370673019 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f760c06aed0 0x7f760c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:18.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.222+0000 7f7607fff640 1 -- 172.21.15.57:0/1370673019 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5979+0+0 (secure 0 0 0) 0x7f76100892f0 con 0x7f76200cf340 2023-10-19T13:01:18.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.222+0000 7f761effd640 1 --2- 172.21.15.57:0/1370673019 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f760c06aed0 0x7f760c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:18.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.223+0000 7f761effd640 1 --2- 172.21.15.57:0/1370673019 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f760c06aed0 0x7f760c06d390 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f76201f48b0 tx=0x7f7614007d20 comp rx=0 tx=0).ready entity=mgr.7072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:18.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.227+0000 7f7607fff640 1 -- 172.21.15.57:0/1370673019 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7610054350 con 0x7f76200cf340 2023-10-19T13:01:18.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.472+0000 7f7625d55640 1 -- 172.21.15.57:0/1370673019 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list"]} v 0) v1 -- 0x7f76200690e0 con 0x7f76200cf340 2023-10-19T13:01:18.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.697+0000 7f7607fff640 1 -- 172.21.15.57:0/1370673019 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list"]}]=0 v641) v1 ==== 166+0+0 (secure 0 0 0) 0x7f7610057970 con 0x7f76200cf340 2023-10-19T13:01:18.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.699+0000 7f7625d55640 1 -- 172.21.15.57:0/1370673019 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f760c06aed0 msgr2=0x7f760c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:18.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.699+0000 7f7625d55640 1 --2- 172.21.15.57:0/1370673019 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f760c06aed0 0x7f760c06d390 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f76201f48b0 tx=0x7f7614007d20 comp rx=0 tx=0).stop 2023-10-19T13:01:18.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.699+0000 7f7625d55640 1 -- 172.21.15.57:0/1370673019 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f76200cf340 msgr2=0x7f762009e150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:18.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.699+0000 7f7625d55640 1 --2- 172.21.15.57:0/1370673019 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f76200cf340 0x7f762009e150 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f761000c4c0 tx=0x7f761000c980 comp rx=0 tx=0).stop 2023-10-19T13:01:18.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.699+0000 7f7625d55640 1 -- 172.21.15.57:0/1370673019 shutdown_connections 2023-10-19T13:01:18.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.699+0000 7f7625d55640 1 --2- 172.21.15.57:0/1370673019 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f760c06aed0 0x7f760c06d390 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:18.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.699+0000 7f7625d55640 1 --2- 172.21.15.57:0/1370673019 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f762015b2a0 0x7f7620157040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:18.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.699+0000 7f7625d55640 1 --2- 172.21.15.57:0/1370673019 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f76200d1910 0x7f7620156b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:18.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.699+0000 7f7625d55640 1 --2- 172.21.15.57:0/1370673019 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f76200cf340 0x7f762009e150 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:18.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.699+0000 7f7625d55640 1 -- 172.21.15.57:0/1370673019 >> 172.21.15.57:0/1370673019 conn(0x7f76200d0d60 msgr2=0x7f762015a560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:18.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.699+0000 7f7625d55640 1 -- 172.21.15.57:0/1370673019 shutdown_connections 2023-10-19T13:01:18.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.699+0000 7f7625d55640 1 -- 172.21.15.57:0/1370673019 wait complete. 2023-10-19T13:01:18.713 INFO:tasks.cephfs_test_runner:Test listing snap schedules on a non-existing filesystem path failure ... ok 2023-10-19T13:01:18.713 INFO:tasks.cephfs_test_runner:Starting test: test_schedule_auto_deactivation_for_non_existent_path (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test that a non-existent path leads to schedule deactivation after a few retries. 2023-10-19T13:01:18.713 INFO:tasks.cephfs_test_runner:test_schedule_auto_deactivation_for_non_existent_path (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-10-19T13:01:18.714 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T13:01:18.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.822+0000 7f0682133640 1 -- 172.21.15.57:0/1232496797 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f067c158aa0 msgr2=0x7f067c0cc370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:18.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.822+0000 7f0682133640 1 --2- 172.21.15.57:0/1232496797 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f067c158aa0 0x7f067c0cc370 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7f066c009830 tx=0x7f066c02bd60 comp rx=0 tx=0).stop 2023-10-19T13:01:18.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.822+0000 7f0682133640 1 -- 172.21.15.57:0/1232496797 shutdown_connections 2023-10-19T13:01:18.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.822+0000 7f0682133640 1 --2- 172.21.15.57:0/1232496797 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f067c070c30 0x7f067c0b0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:18.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.822+0000 7f0682133640 1 --2- 172.21.15.57:0/1232496797 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f067c158aa0 0x7f067c0cc370 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:18.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.822+0000 7f0682133640 1 -- 172.21.15.57:0/1232496797 >> 172.21.15.57:0/1232496797 conn(0x7f067c0aec40 msgr2=0x7f067c0af050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:18.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.822+0000 7f0682133640 1 -- 172.21.15.57:0/1232496797 shutdown_connections 2023-10-19T13:01:18.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.822+0000 7f0682133640 1 -- 172.21.15.57:0/1232496797 wait complete. 2023-10-19T13:01:18.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.823+0000 7f0682133640 1 Processor -- start 2023-10-19T13:01:18.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.823+0000 7f0682133640 1 -- start start 2023-10-19T13:01:18.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.824+0000 7f0682133640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f067c070c30 0x7f067c14f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:18.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.824+0000 7f0682133640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f067c0c9250 0x7f067c14c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:18.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.824+0000 7f0682133640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f067c158aa0 0x7f067c153c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:18.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.824+0000 7f0682133640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f067c15bca0 con 0x7f067c0c9250 2023-10-19T13:01:18.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.824+0000 7f0682133640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f067c14e240 con 0x7f067c070c30 2023-10-19T13:01:18.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.824+0000 7f0682133640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f067c14e520 con 0x7f067c158aa0 2023-10-19T13:01:18.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.824+0000 7f067affd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f067c0c9250 0x7f067c14c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:18.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.824+0000 7f067affd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f067c0c9250 0x7f067c14c5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:58460/0 (socket says 172.21.15.57:58460) 2023-10-19T13:01:18.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.824+0000 7f067affd640 1 -- 172.21.15.57:0/266867575 learned_addr learned my addr 172.21.15.57:0/266867575 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:01:18.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.824+0000 7f067bfff640 1 --2- 172.21.15.57:0/266867575 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f067c158aa0 0x7f067c153c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:18.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.824+0000 7f067b7fe640 1 --2- 172.21.15.57:0/266867575 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f067c070c30 0x7f067c14f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:18.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.825+0000 7f067affd640 1 -- 172.21.15.57:0/266867575 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f067c158aa0 msgr2=0x7f067c153c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:18.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.825+0000 7f067affd640 1 --2- 172.21.15.57:0/266867575 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f067c158aa0 0x7f067c153c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:18.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.825+0000 7f067affd640 1 -- 172.21.15.57:0/266867575 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f067c070c30 msgr2=0x7f067c14f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:18.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.825+0000 7f067affd640 1 --2- 172.21.15.57:0/266867575 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f067c070c30 0x7f067c14f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:18.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.825+0000 7f067affd640 1 -- 172.21.15.57:0/266867575 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f066c009490 con 0x7f067c0c9250 2023-10-19T13:01:18.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.825+0000 7f067bfff640 1 --2- 172.21.15.57:0/266867575 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f067c158aa0 0x7f067c153c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T13:01:18.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.825+0000 7f067affd640 1 --2- 172.21.15.57:0/266867575 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f067c0c9250 0x7f067c14c5d0 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f067000c4b0 tx=0x7f067000c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:18.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.825+0000 7f0663fff640 1 -- 172.21.15.57:0/266867575 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0670005480 con 0x7f067c0c9250 2023-10-19T13:01:18.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.825+0000 7f0663fff640 1 -- 172.21.15.57:0/266867575 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f06700036e0 con 0x7f067c0c9250 2023-10-19T13:01:18.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.825+0000 7f0663fff640 1 -- 172.21.15.57:0/266867575 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0670003b30 con 0x7f067c0c9250 2023-10-19T13:01:18.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.825+0000 7f0682133640 1 -- 172.21.15.57:0/266867575 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f067c14e800 con 0x7f067c0c9250 2023-10-19T13:01:18.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.825+0000 7f0682133640 1 -- 172.21.15.57:0/266867575 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f067c14cb10 con 0x7f067c0c9250 2023-10-19T13:01:18.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.826+0000 7f0682133640 1 -- 172.21.15.57:0/266867575 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f067c0690e0 con 0x7f067c0c9250 2023-10-19T13:01:18.835 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:18.832+0000 7fb508163640 -1 client.7141: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.7030:f55302c0-37aa-4235-8ccc-9775bc542785:v1:172.21.15.179:0/3304791412] 2023-10-19T13:01:18.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.827+0000 7f0663fff640 1 -- 172.21.15.57:0/266867575 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 47) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f0670003c90 con 0x7f067c0c9250 2023-10-19T13:01:18.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.827+0000 7f0663fff640 1 --2- 172.21.15.57:0/266867575 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f066806f2e0 0x7f06680717a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:18.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.827+0000 7f0663fff640 1 -- 172.21.15.57:0/266867575 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5979+0+0 (secure 0 0 0) 0x7f0670088e10 con 0x7f067c0c9250 2023-10-19T13:01:18.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.827+0000 7f067b7fe640 1 --2- 172.21.15.57:0/266867575 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f066806f2e0 0x7f06680717a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:18.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.828+0000 7f067b7fe640 1 --2- 172.21.15.57:0/266867575 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f066806f2e0 0x7f06680717a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f066c036000 tx=0x7f066c004730 comp rx=0 tx=0).ready entity=mgr.7072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:18.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:18.831+0000 7f0663fff640 1 -- 172.21.15.57:0/266867575 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0670053f00 con 0x7f067c0c9250 2023-10-19T13:01:19.014 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.011+0000 7f0682133640 1 -- 172.21.15.57:0/266867575 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f067c071120 con 0x7f067c0c9250 2023-10-19T13:01:19.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.012+0000 7f0663fff640 1 -- 172.21.15.57:0/266867575 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v105) v1 ==== 74+0+14265 (secure 0 0 0) 0x7f0670017070 con 0x7f067c0c9250 2023-10-19T13:01:19.015 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:01:19.016 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":105,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T13:01:17.685536+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.57:6835/2583001580":"2023-10-20T13:01:14.520452+0000","172.21.15.179:0/1870731741":"2023-10-20T13:01:12.052824+0000","172.21.15.179:0/582382550":"2023-10-20T13:01:12.052824+0000","172.21.15.179:6833/124009847":"2023-10-20T13:01:12.052824+0000","172.21.15.179:0/3304791412":"2023-10-20T13:01:12.052824+0000","172.21.15.179:0/3356740410":"2023-10-20T13:01:12.052824+0000","172.21.15.179:0/1965409844":"2023-10-20T13:01:12.052824+0000","172.21.15.179:6833/1413162726":"2023-10-20T13:01:06.603811+0000","172.21.15.179:0/1810025399":"2023-10-20T13:01:12.052824+0000","172.21.15.179:6832/1413162726":"2023-10-20T13:01:06.603811+0000","172.21.15.179:0/3401895301":"2023-10-20T13:01:06.603811+0000","172.21.15.57:6833/2583001580":"2023-10-20T13:01:14.520452+0000","172.21.15.179:0/2499848570":"2023-10-20T13:01:06.603811+0000","172.21.15.179:0/3622569807":"2023-10-20T13:01:06.603811+0000","172.21.15.179:0/2835518029":"2023-10-20T13:01:06.603811+0000","172.21.15.179:0/1643279404":"2023-10-20T13:01:06.603811+0000","172.21.15.179:6832/124009847":"2023-10-20T13:01:12.052824+0000","172.21.15.179:0/4219879451":"2023-10-20T13:01:06.603811+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T13:01:19.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.014+0000 7f0682133640 1 -- 172.21.15.57:0/266867575 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f066806f2e0 msgr2=0x7f06680717a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:19.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.014+0000 7f0682133640 1 --2- 172.21.15.57:0/266867575 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f066806f2e0 0x7f06680717a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f066c036000 tx=0x7f066c004730 comp rx=0 tx=0).stop 2023-10-19T13:01:19.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.014+0000 7f0682133640 1 -- 172.21.15.57:0/266867575 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f067c0c9250 msgr2=0x7f067c14c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:19.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.014+0000 7f0682133640 1 --2- 172.21.15.57:0/266867575 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f067c0c9250 0x7f067c14c5d0 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f067000c4b0 tx=0x7f067000c970 comp rx=0 tx=0).stop 2023-10-19T13:01:19.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.014+0000 7f0682133640 1 -- 172.21.15.57:0/266867575 shutdown_connections 2023-10-19T13:01:19.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.014+0000 7f0682133640 1 --2- 172.21.15.57:0/266867575 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f066806f2e0 0x7f06680717a0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:19.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.014+0000 7f0682133640 1 --2- 172.21.15.57:0/266867575 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f067c158aa0 0x7f067c153c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:19.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.014+0000 7f0682133640 1 --2- 172.21.15.57:0/266867575 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f067c0c9250 0x7f067c14c5d0 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:19.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.014+0000 7f0682133640 1 --2- 172.21.15.57:0/266867575 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f067c070c30 0x7f067c14f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:19.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.014+0000 7f0682133640 1 -- 172.21.15.57:0/266867575 >> 172.21.15.57:0/266867575 conn(0x7f067c0aec40 msgr2=0x7f067c0adad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:19.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.014+0000 7f0682133640 1 -- 172.21.15.57:0/266867575 shutdown_connections 2023-10-19T13:01:19.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.015+0000 7f0682133640 1 -- 172.21.15.57:0/266867575 wait complete. 2023-10-19T13:01:19.029 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-19T12:44:07.371186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-19T13:01:19.029 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-19T13:01:19.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.141+0000 7f740dbd7640 1 -- 172.21.15.57:0/376917066 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f74080cf340 msgr2=0x7f74080d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:19.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.141+0000 7f740dbd7640 1 --2- 172.21.15.57:0/376917066 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f74080cf340 0x7f74080d1910 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f73f8009830 tx=0x7f73f802bcd0 comp rx=0 tx=0).stop 2023-10-19T13:01:19.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.141+0000 7f740dbd7640 1 -- 172.21.15.57:0/376917066 shutdown_connections 2023-10-19T13:01:19.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.141+0000 7f740dbd7640 1 --2- 172.21.15.57:0/376917066 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f740815b2a0 0x7f74080cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:19.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.141+0000 7f740dbd7640 1 --2- 172.21.15.57:0/376917066 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f74080cf340 0x7f74080d1910 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:19.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.141+0000 7f740dbd7640 1 -- 172.21.15.57:0/376917066 >> 172.21.15.57:0/376917066 conn(0x7f74080d0d60 msgr2=0x7f74080d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:19.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.141+0000 7f740dbd7640 1 -- 172.21.15.57:0/376917066 shutdown_connections 2023-10-19T13:01:19.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.141+0000 7f740dbd7640 1 -- 172.21.15.57:0/376917066 wait complete. 2023-10-19T13:01:19.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.142+0000 7f740dbd7640 1 Processor -- start 2023-10-19T13:01:19.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.142+0000 7f740dbd7640 1 -- start start 2023-10-19T13:01:19.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.143+0000 7f740dbd7640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f74080ceb40 0x7f7408158560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:19.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.143+0000 7f740dbd7640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f74080cf340 0x7f74080b42f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:19.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.143+0000 7f740dbd7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f740815b2a0 0x7f7408156e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:19.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.143+0000 7f740dbd7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7408158140 con 0x7f740815b2a0 2023-10-19T13:01:19.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.143+0000 7f740dbd7640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74080c7fd0 con 0x7f74080ceb40 2023-10-19T13:01:19.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.143+0000 7f740dbd7640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74081577a0 con 0x7f74080cf340 2023-10-19T13:01:19.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.143+0000 7f7407fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f740815b2a0 0x7f7408156e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:19.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.143+0000 7f7407fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f740815b2a0 0x7f7408156e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:58462/0 (socket says 172.21.15.57:58462) 2023-10-19T13:01:19.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.143+0000 7f7407fff640 1 -- 172.21.15.57:0/3258295536 learned_addr learned my addr 172.21.15.57:0/3258295536 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:01:19.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.143+0000 7f7406ffd640 1 --2- 172.21.15.57:0/3258295536 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f74080cf340 0x7f74080b42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:19.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.143+0000 7f74077fe640 1 --2- 172.21.15.57:0/3258295536 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f74080ceb40 0x7f7408158560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:19.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.144+0000 7f7407fff640 1 -- 172.21.15.57:0/3258295536 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f74080cf340 msgr2=0x7f74080b42f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:19.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.144+0000 7f7407fff640 1 --2- 172.21.15.57:0/3258295536 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f74080cf340 0x7f74080b42f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:19.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.144+0000 7f7407fff640 1 -- 172.21.15.57:0/3258295536 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f74080ceb40 msgr2=0x7f7408158560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:19.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.144+0000 7f7407fff640 1 --2- 172.21.15.57:0/3258295536 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f74080ceb40 0x7f7408158560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:19.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.144+0000 7f7407fff640 1 -- 172.21.15.57:0/3258295536 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73fc009490 con 0x7f740815b2a0 2023-10-19T13:01:19.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.144+0000 7f7407fff640 1 --2- 172.21.15.57:0/3258295536 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f740815b2a0 0x7f7408156e10 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f740000cb10 tx=0x7f740000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:19.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.144+0000 7f73effff640 1 -- 172.21.15.57:0/3258295536 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f740000ac70 con 0x7f740815b2a0 2023-10-19T13:01:19.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.144+0000 7f740dbd7640 1 -- 172.21.15.57:0/3258295536 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73f8009490 con 0x7f740815b2a0 2023-10-19T13:01:19.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.144+0000 7f73effff640 1 -- 172.21.15.57:0/3258295536 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f7400003970 con 0x7f740815b2a0 2023-10-19T13:01:19.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.144+0000 7f73effff640 1 -- 172.21.15.57:0/3258295536 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7400003d80 con 0x7f740815b2a0 2023-10-19T13:01:19.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.144+0000 7f740dbd7640 1 -- 172.21.15.57:0/3258295536 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7408157d60 con 0x7f740815b2a0 2023-10-19T13:01:19.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.145+0000 7f740dbd7640 1 -- 172.21.15.57:0/3258295536 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f740815a960 con 0x7f740815b2a0 2023-10-19T13:01:19.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.146+0000 7f73effff640 1 -- 172.21.15.57:0/3258295536 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 47) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f7400004030 con 0x7f740815b2a0 2023-10-19T13:01:19.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.146+0000 7f73effff640 1 --2- 172.21.15.57:0/3258295536 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f73f406aed0 0x7f73f406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:19.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.146+0000 7f73effff640 1 -- 172.21.15.57:0/3258295536 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5979+0+0 (secure 0 0 0) 0x7f740008a020 con 0x7f740815b2a0 2023-10-19T13:01:19.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.146+0000 7f74077fe640 1 --2- 172.21.15.57:0/3258295536 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f73f406aed0 0x7f73f406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:19.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.147+0000 7f74077fe640 1 --2- 172.21.15.57:0/3258295536 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f73f406aed0 0x7f73f406d390 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f73f8036000 tx=0x7f73f8004730 comp rx=0 tx=0).ready entity=mgr.7072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:19.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.150+0000 7f73effff640 1 -- 172.21.15.57:0/3258295536 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7400055090 con 0x7f740815b2a0 2023-10-19T13:01:19.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.331+0000 7f740dbd7640 1 -- 172.21.15.57:0/3258295536 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f74080690e0 con 0x7f740815b2a0 2023-10-19T13:01:19.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.331+0000 7f73effff640 1 -- 172.21.15.57:0/3258295536 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v105) v1 ==== 93+0+10 (secure 0 0 0) 0x7f74000586b0 con 0x7f740815b2a0 2023-10-19T13:01:19.334 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-19T13:01:19.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.333+0000 7f740dbd7640 1 -- 172.21.15.57:0/3258295536 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f73f406aed0 msgr2=0x7f73f406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:19.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.333+0000 7f740dbd7640 1 --2- 172.21.15.57:0/3258295536 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f73f406aed0 0x7f73f406d390 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f73f8036000 tx=0x7f73f8004730 comp rx=0 tx=0).stop 2023-10-19T13:01:19.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.333+0000 7f740dbd7640 1 -- 172.21.15.57:0/3258295536 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f740815b2a0 msgr2=0x7f7408156e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:19.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.333+0000 7f740dbd7640 1 --2- 172.21.15.57:0/3258295536 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f740815b2a0 0x7f7408156e10 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f740000cb10 tx=0x7f740000a060 comp rx=0 tx=0).stop 2023-10-19T13:01:19.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.333+0000 7f740dbd7640 1 -- 172.21.15.57:0/3258295536 shutdown_connections 2023-10-19T13:01:19.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.333+0000 7f740dbd7640 1 --2- 172.21.15.57:0/3258295536 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f73f406aed0 0x7f73f406d390 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:19.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.333+0000 7f740dbd7640 1 --2- 172.21.15.57:0/3258295536 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f740815b2a0 0x7f7408156e10 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:19.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.333+0000 7f740dbd7640 1 --2- 172.21.15.57:0/3258295536 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f74080cf340 0x7f74080b42f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:19.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.333+0000 7f740dbd7640 1 --2- 172.21.15.57:0/3258295536 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f74080ceb40 0x7f7408158560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:19.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.333+0000 7f740dbd7640 1 -- 172.21.15.57:0/3258295536 >> 172.21.15.57:0/3258295536 conn(0x7f74080d0d60 msgr2=0x7f740806b610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:19.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.333+0000 7f740dbd7640 1 -- 172.21.15.57:0/3258295536 shutdown_connections 2023-10-19T13:01:19.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.333+0000 7f740dbd7640 1 -- 172.21.15.57:0/3258295536 wait complete. 2023-10-19T13:01:19.347 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path' 2023-10-19T13:01:19.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.455+0000 7f8fb464b640 1 -- 172.21.15.57:0/973418916 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8fac15b2a0 msgr2=0x7f8fac0d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:19.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.455+0000 7f8fb464b640 1 --2- 172.21.15.57:0/973418916 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8fac15b2a0 0x7f8fac0d1910 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f8fac15b680 tx=0x7f8f9c02bcf0 comp rx=0 tx=0).stop 2023-10-19T13:01:19.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.455+0000 7f8fb464b640 1 -- 172.21.15.57:0/973418916 shutdown_connections 2023-10-19T13:01:19.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.455+0000 7f8fb464b640 1 --2- 172.21.15.57:0/973418916 >> v2:172.21.15.179:3300/0 conn(0x7f8fac0ceb40 0x7f8fac0b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:19.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.455+0000 7f8fb464b640 1 --2- 172.21.15.57:0/973418916 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8fac15b2a0 0x7f8fac0d1910 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:19.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.455+0000 7f8fb464b640 1 -- 172.21.15.57:0/973418916 >> 172.21.15.57:0/973418916 conn(0x7f8fac0d0d60 msgr2=0x7f8fac0d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:19.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.455+0000 7f8fb464b640 1 -- 172.21.15.57:0/973418916 shutdown_connections 2023-10-19T13:01:19.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.455+0000 7f8fb464b640 1 -- 172.21.15.57:0/973418916 wait complete. 2023-10-19T13:01:19.462 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.456+0000 7f8fb464b640 1 Processor -- start 2023-10-19T13:01:19.462 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.456+0000 7f8fb464b640 1 -- start start 2023-10-19T13:01:19.462 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.457+0000 7f8fb464b640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8fac0ceb40 0x7f8fac0c82f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:19.462 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.457+0000 7f8fb464b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8fac0cf340 0x7f8fac151300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:19.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.457+0000 7f8fb464b640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8fac15b2a0 0x7f8fac0ae610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:19.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.457+0000 7f8fb464b640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fac0c95b0 con 0x7f8fac0cf340 2023-10-19T13:01:19.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.457+0000 7f8fb464b640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fac15bb40 con 0x7f8fac15b2a0 2023-10-19T13:01:19.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.457+0000 7f8fb464b640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fac15be20 con 0x7f8fac0ceb40 2023-10-19T13:01:19.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.457+0000 7f8fb1be6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8fac0cf340 0x7f8fac151300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:19.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.457+0000 7f8fb1be6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8fac0cf340 0x7f8fac151300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:58476/0 (socket says 172.21.15.57:58476) 2023-10-19T13:01:19.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.457+0000 7f8fb1be6640 1 -- 172.21.15.57:0/4078747189 learned_addr learned my addr 172.21.15.57:0/4078747189 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:01:19.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.457+0000 7f8fb1be6640 1 -- 172.21.15.57:0/4078747189 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8fac0ceb40 msgr2=0x7f8fac0c82f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:19.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.457+0000 7f8fb23e7640 1 --2- 172.21.15.57:0/4078747189 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8fac0ceb40 0x7f8fac0c82f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:19.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.457+0000 7f8fb2be8640 1 --2- 172.21.15.57:0/4078747189 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8fac15b2a0 0x7f8fac0ae610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:19.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.457+0000 7f8fb1be6640 1 --2- 172.21.15.57:0/4078747189 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8fac0ceb40 0x7f8fac0c82f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:19.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.457+0000 7f8fb1be6640 1 -- 172.21.15.57:0/4078747189 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8fac15b2a0 msgr2=0x7f8fac0ae610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:19.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.457+0000 7f8fb1be6640 1 --2- 172.21.15.57:0/4078747189 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8fac15b2a0 0x7f8fac0ae610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:19.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.457+0000 7f8fb1be6640 1 -- 172.21.15.57:0/4078747189 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f9c009490 con 0x7f8fac0cf340 2023-10-19T13:01:19.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.458+0000 7f8fb1be6640 1 --2- 172.21.15.57:0/4078747189 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8fac0cf340 0x7f8fac151300 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f8f9c000c00 tx=0x7f8f9c02c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:19.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.458+0000 7f8fa2ffd640 1 -- 172.21.15.57:0/4078747189 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f9c003b80 con 0x7f8fac0cf340 2023-10-19T13:01:19.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.458+0000 7f8fa2ffd640 1 -- 172.21.15.57:0/4078747189 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f8f9c003df0 con 0x7f8fac0cf340 2023-10-19T13:01:19.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.458+0000 7f8fb464b640 1 -- 172.21.15.57:0/4078747189 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fac15c100 con 0x7f8fac0cf340 2023-10-19T13:01:19.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.458+0000 7f8fa2ffd640 1 -- 172.21.15.57:0/4078747189 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f9c0346a0 con 0x7f8fac0cf340 2023-10-19T13:01:19.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.458+0000 7f8fb464b640 1 -- 172.21.15.57:0/4078747189 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fac15c6c0 con 0x7f8fac0cf340 2023-10-19T13:01:19.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.459+0000 7f8fb464b640 1 -- 172.21.15.57:0/4078747189 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fac15c8b0 con 0x7f8fac0cf340 2023-10-19T13:01:19.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.459+0000 7f8fa2ffd640 1 -- 172.21.15.57:0/4078747189 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 47) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f8f9c03b040 con 0x7f8fac0cf340 2023-10-19T13:01:19.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.460+0000 7f8fa2ffd640 1 --2- 172.21.15.57:0/4078747189 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f8f9006aed0 0x7f8f9006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:19.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.460+0000 7f8fa2ffd640 1 -- 172.21.15.57:0/4078747189 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5979+0+0 (secure 0 0 0) 0x7f8f9c0aba60 con 0x7f8fac0cf340 2023-10-19T13:01:19.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.460+0000 7f8fb23e7640 1 --2- 172.21.15.57:0/4078747189 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f8f9006aed0 0x7f8f9006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:19.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.460+0000 7f8fb23e7640 1 --2- 172.21.15.57:0/4078747189 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f8f9006aed0 0x7f8f9006d390 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f8fa80046b0 tx=0x7f8fa8009040 comp rx=0 tx=0).ready entity=mgr.7072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:19.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.464+0000 7f8fa2ffd640 1 -- 172.21.15.57:0/4078747189 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8f9c076a10 con 0x7f8fac0cf340 2023-10-19T13:01:19.510 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:19.507+0000 7fb508163640 -1 client.7141: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.7030:f55302c0-37aa-4235-8ccc-9775bc542785:v1:172.21.15.179:0/3304791412] 2023-10-19T13:01:19.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:19.719+0000 7f8fb464b640 1 -- 172.21.15.57:0/4078747189 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path"]} v 0) v1 -- 0x7f8fac0b0d00 con 0x7f8fac0cf340 2023-10-19T13:01:20.206 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:20.203+0000 7fb508163640 -1 client.7141: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.7030:f55302c0-37aa-4235-8ccc-9775bc542785:v1:172.21.15.179:0/3304791412] 2023-10-19T13:01:20.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:20.708+0000 7f8fa2ffd640 1 -- 172.21.15.57:0/4078747189 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path"]}]=0 v643) v1 ==== 186+0+0 (secure 0 0 0) 0x7f8f9c041100 con 0x7f8fac0cf340 2023-10-19T13:01:20.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:20.710+0000 7f8fb464b640 1 -- 172.21.15.57:0/4078747189 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f8f9006aed0 msgr2=0x7f8f9006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:20.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:20.710+0000 7f8fb464b640 1 --2- 172.21.15.57:0/4078747189 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f8f9006aed0 0x7f8f9006d390 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f8fa80046b0 tx=0x7f8fa8009040 comp rx=0 tx=0).stop 2023-10-19T13:01:20.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:20.710+0000 7f8fb464b640 1 -- 172.21.15.57:0/4078747189 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8fac0cf340 msgr2=0x7f8fac151300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:20.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:20.710+0000 7f8fb464b640 1 --2- 172.21.15.57:0/4078747189 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8fac0cf340 0x7f8fac151300 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f8f9c000c00 tx=0x7f8f9c02c920 comp rx=0 tx=0).stop 2023-10-19T13:01:20.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:20.710+0000 7f8fb464b640 1 -- 172.21.15.57:0/4078747189 shutdown_connections 2023-10-19T13:01:20.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:20.710+0000 7f8fb464b640 1 --2- 172.21.15.57:0/4078747189 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f8f9006aed0 0x7f8f9006d390 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:20.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:20.710+0000 7f8fb464b640 1 --2- 172.21.15.57:0/4078747189 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8fac15b2a0 0x7f8fac0ae610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:20.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:20.710+0000 7f8fb464b640 1 --2- 172.21.15.57:0/4078747189 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8fac0cf340 0x7f8fac151300 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:20.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:20.710+0000 7f8fb464b640 1 --2- 172.21.15.57:0/4078747189 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8fac0ceb40 0x7f8fac0c82f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:20.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:20.710+0000 7f8fb464b640 1 -- 172.21.15.57:0/4078747189 >> 172.21.15.57:0/4078747189 conn(0x7f8fac0d0d60 msgr2=0x7f8fac06b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:20.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:20.710+0000 7f8fb464b640 1 -- 172.21.15.57:0/4078747189 shutdown_connections 2023-10-19T13:01:20.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:20.710+0000 7f8fb464b640 1 -- 172.21.15.57:0/4078747189 wait complete. 2023-10-19T13:01:20.724 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2023-10-19T13:01:20.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:20.833+0000 7ff0f4a5d640 1 -- 172.21.15.57:0/1392754052 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff0dc0047c0 msgr2=0x7ff0dc004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:20.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:20.833+0000 7ff0f4a5d640 1 --2- 172.21.15.57:0/1392754052 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff0dc0047c0 0x7ff0dc004c40 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7ff0d800bbb0 tx=0x7ff0dc0057f0 comp rx=0 tx=0).stop 2023-10-19T13:01:20.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:20.833+0000 7ff0eed9d640 1 -- 172.21.15.57:0/1392754052 reap_dead start 2023-10-19T13:01:20.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:20.833+0000 7ff0f4a5d640 1 -- 172.21.15.57:0/1392754052 shutdown_connections 2023-10-19T13:01:20.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:20.833+0000 7ff0f4a5d640 1 -- 172.21.15.57:0/1392754052 >> 172.21.15.57:0/1392754052 conn(0x7ff0f0069930 msgr2=0x7ff0f0069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:20.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:20.833+0000 7ff0f4a5d640 1 -- 172.21.15.57:0/1392754052 shutdown_connections 2023-10-19T13:01:20.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:20.833+0000 7ff0f4a5d640 1 -- 172.21.15.57:0/1392754052 wait complete. 2023-10-19T13:01:20.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:20.834+0000 7ff0f4a5d640 1 Processor -- start 2023-10-19T13:01:20.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:20.834+0000 7ff0f4a5d640 1 -- start start 2023-10-19T13:01:20.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:20.835+0000 7ff0f4a5d640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff0dc0047c0 0x7ff0f013ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:20.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:20.835+0000 7ff0f4a5d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff0f014c360 0x7ff0f00b0d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:20.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:20.835+0000 7ff0f4a5d640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff0f014c7e0 0x7ff0f00b1290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:20.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:20.835+0000 7ff0f4a5d640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0f014a600 con 0x7ff0f014c360 2023-10-19T13:01:20.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:20.835+0000 7ff0f4a5d640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0f013b3a0 con 0x7ff0dc0047c0 2023-10-19T13:01:20.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:20.835+0000 7ff0f4a5d640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0f014cdb0 con 0x7ff0f014c7e0 2023-10-19T13:01:20.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:20.835+0000 7ff0edd9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff0f014c360 0x7ff0f00b0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:20.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:20.835+0000 7ff0edd9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff0f014c360 0x7ff0f00b0d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:58482/0 (socket says 172.21.15.57:58482) 2023-10-19T13:01:20.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:20.835+0000 7ff0edd9b640 1 -- 172.21.15.57:0/1616347786 learned_addr learned my addr 172.21.15.57:0/1616347786 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:01:20.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:20.835+0000 7ff0ee59c640 1 --2- 172.21.15.57:0/1616347786 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff0dc0047c0 0x7ff0f013ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:20.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:20.835+0000 7ff0eed9d640 1 --2- 172.21.15.57:0/1616347786 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff0f014c7e0 0x7ff0f00b1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:20.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:20.836+0000 7ff0edd9b640 1 -- 172.21.15.57:0/1616347786 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff0f014c7e0 msgr2=0x7ff0f00b1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:20.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:20.836+0000 7ff0edd9b640 1 --2- 172.21.15.57:0/1616347786 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff0f014c7e0 0x7ff0f00b1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:20.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:20.836+0000 7ff0edd9b640 1 -- 172.21.15.57:0/1616347786 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff0dc0047c0 msgr2=0x7ff0f013ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:20.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:20.836+0000 7ff0edd9b640 1 --2- 172.21.15.57:0/1616347786 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff0dc0047c0 0x7ff0f013ae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:20.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:20.836+0000 7ff0edd9b640 1 -- 172.21.15.57:0/1616347786 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0f014d090 con 0x7ff0f014c360 2023-10-19T13:01:20.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:20.836+0000 7ff0edd9b640 1 --2- 172.21.15.57:0/1616347786 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff0f014c360 0x7ff0f00b0d50 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7ff0d800bb00 tx=0x7ff0d8036820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:20.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:20.836+0000 7ff0e6ffd640 1 -- 172.21.15.57:0/1616347786 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0d8036e20 con 0x7ff0f014c360 2023-10-19T13:01:20.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:20.836+0000 7ff0e6ffd640 1 -- 172.21.15.57:0/1616347786 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7ff0d803e440 con 0x7ff0f014c360 2023-10-19T13:01:20.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:20.836+0000 7ff0f4a5d640 1 -- 172.21.15.57:0/1616347786 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0f014d370 con 0x7ff0f014c360 2023-10-19T13:01:20.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:20.836+0000 7ff0e6ffd640 1 -- 172.21.15.57:0/1616347786 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0d803e850 con 0x7ff0f014c360 2023-10-19T13:01:20.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:20.836+0000 7ff0f4a5d640 1 -- 172.21.15.57:0/1616347786 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0f00ac830 con 0x7ff0f014c360 2023-10-19T13:01:20.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:20.837+0000 7ff0f4a5d640 1 -- 172.21.15.57:0/1616347786 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0f014d890 con 0x7ff0f014c360 2023-10-19T13:01:20.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:20.838+0000 7ff0e6ffd640 1 -- 172.21.15.57:0/1616347786 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 47) v1 ==== 89275+0+0 (secure 0 0 0) 0x7ff0d8048440 con 0x7ff0f014c360 2023-10-19T13:01:20.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:20.838+0000 7ff0e6ffd640 1 --2- 172.21.15.57:0/1616347786 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7ff0d006aed0 0x7ff0d006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:20.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:20.838+0000 7ff0e6ffd640 1 -- 172.21.15.57:0/1616347786 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5979+0+0 (secure 0 0 0) 0x7ff0d80aaff0 con 0x7ff0f014c360 2023-10-19T13:01:20.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:20.838+0000 7ff0ee59c640 1 --2- 172.21.15.57:0/1616347786 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7ff0d006aed0 0x7ff0d006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:20.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:20.839+0000 7ff0ee59c640 1 --2- 172.21.15.57:0/1616347786 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7ff0d006aed0 0x7ff0d006d390 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff0f00b1ad0 tx=0x7ff0e8006720 comp rx=0 tx=0).ready entity=mgr.7072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:20.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:20.842+0000 7ff0e6ffd640 1 -- 172.21.15.57:0/1616347786 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff0f014d890 con 0x7ff0f014c360 2023-10-19T13:01:20.898 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:20.895+0000 7fb508163640 -1 client.7141: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.7030:f55302c0-37aa-4235-8ccc-9775bc542785:v1:172.21.15.179:0/3304791412] 2023-10-19T13:01:21.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.025+0000 7ff0f4a5d640 1 -- 172.21.15.57:0/1616347786 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "config dump"} v 0) v1 -- 0x7ff0f014d890 con 0x7ff0f014c360 2023-10-19T13:01:21.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.025+0000 7ff0e6ffd640 1 -- 172.21.15.57:0/1616347786 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v46) v1 ==== 59+0+2220 (secure 0 0 0) 0x7ff0d80438c0 con 0x7ff0f014c360 2023-10-19T13:01:21.029 INFO:teuthology.orchestra.run.smithi057.stdout:WHO MASK LEVEL OPTION VALUE RO 2023-10-19T13:01:21.029 INFO:teuthology.orchestra.run.smithi057.stdout:mon advanced mon_op_complaint_time 120 2023-10-19T13:01:21.029 INFO:teuthology.orchestra.run.smithi057.stdout:osd advanced osd_op_complaint_time 180.000000 2023-10-19T13:01:21.029 INFO:teuthology.orchestra.run.smithi057.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 4775.995373 2023-10-19T13:01:21.030 INFO:teuthology.orchestra.run.smithi057.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 4208.118410 2023-10-19T13:01:21.030 INFO:teuthology.orchestra.run.smithi057.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 4541.123030 2023-10-19T13:01:21.030 INFO:teuthology.orchestra.run.smithi057.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 4877.592206 2023-10-19T13:01:21.030 INFO:teuthology.orchestra.run.smithi057.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 5019.748032 2023-10-19T13:01:21.031 INFO:teuthology.orchestra.run.smithi057.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 5000.074376 2023-10-19T13:01:21.031 INFO:teuthology.orchestra.run.smithi057.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 5153.479439 2023-10-19T13:01:21.031 INFO:teuthology.orchestra.run.smithi057.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 4977.363845 2023-10-19T13:01:21.031 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced debug_mds 20/20 2023-10-19T13:01:21.032 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced debug_mds_balancer 20/20 2023-10-19T13:01:21.032 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced debug_ms 1/1 2023-10-19T13:01:21.032 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced mds_bal_fragment_size_max 10000 2023-10-19T13:01:21.032 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced mds_bal_merge_size 5 2023-10-19T13:01:21.032 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced mds_bal_split_bits 3 2023-10-19T13:01:21.033 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced mds_bal_split_size 100 2023-10-19T13:01:21.033 INFO:teuthology.orchestra.run.smithi057.stdout:mds dev mds_debug_frag true 2023-10-19T13:01:21.033 INFO:teuthology.orchestra.run.smithi057.stdout:mds dev mds_debug_scatterstat true 2023-10-19T13:01:21.033 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced mds_op_complaint_time 180.000000 2023-10-19T13:01:21.033 INFO:teuthology.orchestra.run.smithi057.stdout:mds dev mds_verify_scatter true 2023-10-19T13:01:21.034 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced osd_op_complaint_time 180.000000 2023-10-19T13:01:21.034 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced rados_mon_op_timeout 900 2023-10-19T13:01:21.034 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced rados_osd_op_timeout 900 2023-10-19T13:01:21.034 INFO:teuthology.orchestra.run.smithi057.stdout:client advanced client_mount_timeout 600 2023-10-19T13:01:21.035 INFO:teuthology.orchestra.run.smithi057.stdout:client advanced debug_client 20/20 2023-10-19T13:01:21.035 INFO:teuthology.orchestra.run.smithi057.stdout:client advanced debug_ms 1/1 2023-10-19T13:01:21.035 INFO:teuthology.orchestra.run.smithi057.stdout:client advanced rados_mon_op_timeout 900 2023-10-19T13:01:21.035 INFO:teuthology.orchestra.run.smithi057.stdout:client advanced rados_osd_op_timeout 900 2023-10-19T13:01:21.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.028+0000 7ff0f4a5d640 1 -- 172.21.15.57:0/1616347786 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7ff0d006aed0 msgr2=0x7ff0d006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:21.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.028+0000 7ff0f4a5d640 1 --2- 172.21.15.57:0/1616347786 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7ff0d006aed0 0x7ff0d006d390 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff0f00b1ad0 tx=0x7ff0e8006720 comp rx=0 tx=0).stop 2023-10-19T13:01:21.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.028+0000 7ff0f4a5d640 1 -- 172.21.15.57:0/1616347786 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff0f014c360 msgr2=0x7ff0f00b0d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:21.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.028+0000 7ff0f4a5d640 1 --2- 172.21.15.57:0/1616347786 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff0f014c360 0x7ff0f00b0d50 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7ff0d800bb00 tx=0x7ff0d8036820 comp rx=0 tx=0).stop 2023-10-19T13:01:21.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.028+0000 7ff0f4a5d640 1 -- 172.21.15.57:0/1616347786 shutdown_connections 2023-10-19T13:01:21.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.028+0000 7ff0f4a5d640 1 --2- 172.21.15.57:0/1616347786 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7ff0d006aed0 0x7ff0d006d390 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:21.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.028+0000 7ff0f4a5d640 1 --2- 172.21.15.57:0/1616347786 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff0f014c7e0 0x7ff0f00b1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:21.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.028+0000 7ff0f4a5d640 1 --2- 172.21.15.57:0/1616347786 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff0f014c360 0x7ff0f00b0d50 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:21.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.028+0000 7ff0f4a5d640 1 --2- 172.21.15.57:0/1616347786 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff0dc0047c0 0x7ff0f013ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:21.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.028+0000 7ff0f4a5d640 1 -- 172.21.15.57:0/1616347786 >> 172.21.15.57:0/1616347786 conn(0x7ff0f0069930 msgr2=0x7ff0f0152bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:21.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.028+0000 7ff0f4a5d640 1 -- 172.21.15.57:0/1616347786 shutdown_connections 2023-10-19T13:01:21.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.028+0000 7ff0f4a5d640 1 -- 172.21.15.57:0/1616347786 wait complete. 2023-10-19T13:01:21.041 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-19T13:01:21.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.151+0000 7f8874aa2640 1 -- 172.21.15.57:0/2267755857 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f887015c1f0 msgr2=0x7f887006a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:21.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.151+0000 7f8874aa2640 1 --2- 172.21.15.57:0/2267755857 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f887015c1f0 0x7f887006a730 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f88700662b0 tx=0x7f885802bd70 comp rx=0 tx=0).stop 2023-10-19T13:01:21.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.152+0000 7f8874aa2640 1 -- 172.21.15.57:0/2267755857 shutdown_connections 2023-10-19T13:01:21.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.152+0000 7f8874aa2640 1 --2- 172.21.15.57:0/2267755857 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f887015c1f0 0x7f887006a730 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:21.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.152+0000 7f8874aa2640 1 --2- 172.21.15.57:0/2267755857 >> v2:172.21.15.179:3300/0 conn(0x7f88701579f0 0x7f88700b0e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:21.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.152+0000 7f8874aa2640 1 -- 172.21.15.57:0/2267755857 >> 172.21.15.57:0/2267755857 conn(0x7f887006fae0 msgr2=0x7f887006fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:21.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.152+0000 7f8874aa2640 1 -- 172.21.15.57:0/2267755857 shutdown_connections 2023-10-19T13:01:21.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.152+0000 7f8874aa2640 1 -- 172.21.15.57:0/2267755857 wait complete. 2023-10-19T13:01:21.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.153+0000 7f8874aa2640 1 Processor -- start 2023-10-19T13:01:21.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.153+0000 7f8874aa2640 1 -- start start 2023-10-19T13:01:21.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.154+0000 7f8874aa2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f887006ac70 0x7f8870146c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:21.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.154+0000 7f8874aa2640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f88701579f0 0x7f8870156520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:21.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.154+0000 7f8874aa2640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f887015c1f0 0x7f8870156a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:21.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.154+0000 7f8874aa2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88700ac590 con 0x7f887006ac70 2023-10-19T13:01:21.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.154+0000 7f8874aa2640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88701471b0 con 0x7f88701579f0 2023-10-19T13:01:21.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.154+0000 7f8874aa2640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88700a97b0 con 0x7f887015c1f0 2023-10-19T13:01:21.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.154+0000 7f886e59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f887006ac70 0x7f8870146c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:21.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.154+0000 7f886e59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f887006ac70 0x7f8870146c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:58512/0 (socket says 172.21.15.57:58512) 2023-10-19T13:01:21.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.154+0000 7f886e59c640 1 -- 172.21.15.57:0/3075423099 learned_addr learned my addr 172.21.15.57:0/3075423099 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:01:21.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.154+0000 7f8865d9b640 1 --2- 172.21.15.57:0/3075423099 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f88701579f0 0x7f8870156520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:21.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.154+0000 7f886ed9d640 1 --2- 172.21.15.57:0/3075423099 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f887015c1f0 0x7f8870156a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:21.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.154+0000 7f886e59c640 1 -- 172.21.15.57:0/3075423099 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f887015c1f0 msgr2=0x7f8870156a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:21.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.154+0000 7f886e59c640 1 --2- 172.21.15.57:0/3075423099 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f887015c1f0 0x7f8870156a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:21.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.154+0000 7f886e59c640 1 -- 172.21.15.57:0/3075423099 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f88701579f0 msgr2=0x7f8870156520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:21.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.154+0000 7f886e59c640 1 --2- 172.21.15.57:0/3075423099 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f88701579f0 0x7f8870156520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:21.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.154+0000 7f886e59c640 1 -- 172.21.15.57:0/3075423099 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8858009490 con 0x7f887006ac70 2023-10-19T13:01:21.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.154+0000 7f886e59c640 1 --2- 172.21.15.57:0/3075423099 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f887006ac70 0x7f8870146c70 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f886000e9f0 tx=0x7f886000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:21.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.155+0000 7f88677fe640 1 -- 172.21.15.57:0/3075423099 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8860009700 con 0x7f887006ac70 2023-10-19T13:01:21.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.155+0000 7f88677fe640 1 -- 172.21.15.57:0/3075423099 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f886000ce70 con 0x7f887006ac70 2023-10-19T13:01:21.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.155+0000 7f8874aa2640 1 -- 172.21.15.57:0/3075423099 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88700a9a90 con 0x7f887006ac70 2023-10-19T13:01:21.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.155+0000 7f88677fe640 1 -- 172.21.15.57:0/3075423099 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8860010670 con 0x7f887006ac70 2023-10-19T13:01:21.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.155+0000 7f8874aa2640 1 -- 172.21.15.57:0/3075423099 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88700ac220 con 0x7f887006ac70 2023-10-19T13:01:21.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.156+0000 7f8874aa2640 1 -- 172.21.15.57:0/3075423099 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88700690e0 con 0x7f887006ac70 2023-10-19T13:01:21.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.156+0000 7f88677fe640 1 -- 172.21.15.57:0/3075423099 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 47) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f88600107d0 con 0x7f887006ac70 2023-10-19T13:01:21.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.157+0000 7f88677fe640 1 --2- 172.21.15.57:0/3075423099 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f885006aed0 0x7f885006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:21.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.157+0000 7f88677fe640 1 -- 172.21.15.57:0/3075423099 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5979+0+0 (secure 0 0 0) 0x7f8860014070 con 0x7f887006ac70 2023-10-19T13:01:21.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.157+0000 7f8865d9b640 1 --2- 172.21.15.57:0/3075423099 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f885006aed0 0x7f885006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:21.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.157+0000 7f8865d9b640 1 --2- 172.21.15.57:0/3075423099 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f885006aed0 0x7f885006d390 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f8858036000 tx=0x7f8858004730 comp rx=0 tx=0).ready entity=mgr.7072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:21.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.161+0000 7f88677fe640 1 -- 172.21.15.57:0/3075423099 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8860051ca0 con 0x7f887006ac70 2023-10-19T13:01:21.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.337+0000 7f8874aa2640 1 -- 172.21.15.57:0/3075423099 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f88700ac8a0 con 0x7f887006ac70 2023-10-19T13:01:21.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.340+0000 7f88677fe640 1 -- 172.21.15.57:0/3075423099 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v47)=0 v47) v1 ==== 122+0+0 (secure 0 0 0) 0x7f8860055430 con 0x7f887006ac70 2023-10-19T13:01:21.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.342+0000 7f8874aa2640 1 -- 172.21.15.57:0/3075423099 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f885006aed0 msgr2=0x7f885006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:21.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.342+0000 7f8874aa2640 1 --2- 172.21.15.57:0/3075423099 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f885006aed0 0x7f885006d390 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f8858036000 tx=0x7f8858004730 comp rx=0 tx=0).stop 2023-10-19T13:01:21.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.342+0000 7f8874aa2640 1 -- 172.21.15.57:0/3075423099 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f887006ac70 msgr2=0x7f8870146c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:21.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.342+0000 7f8874aa2640 1 --2- 172.21.15.57:0/3075423099 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f887006ac70 0x7f8870146c70 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f886000e9f0 tx=0x7f886000eeb0 comp rx=0 tx=0).stop 2023-10-19T13:01:21.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.342+0000 7f8874aa2640 1 -- 172.21.15.57:0/3075423099 shutdown_connections 2023-10-19T13:01:21.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.342+0000 7f8874aa2640 1 --2- 172.21.15.57:0/3075423099 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f885006aed0 0x7f885006d390 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:21.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.342+0000 7f8874aa2640 1 --2- 172.21.15.57:0/3075423099 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f887015c1f0 0x7f8870156a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:21.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.342+0000 7f8874aa2640 1 --2- 172.21.15.57:0/3075423099 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f88701579f0 0x7f8870156520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:21.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.342+0000 7f8874aa2640 1 --2- 172.21.15.57:0/3075423099 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f887006ac70 0x7f8870146c70 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:21.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.342+0000 7f8874aa2640 1 -- 172.21.15.57:0/3075423099 >> 172.21.15.57:0/3075423099 conn(0x7f887006fae0 msgr2=0x7f8870069720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:21.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.342+0000 7f8874aa2640 1 -- 172.21.15.57:0/3075423099 shutdown_connections 2023-10-19T13:01:21.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.342+0000 7f8874aa2640 1 -- 172.21.15.57:0/3075423099 wait complete. 2023-10-19T13:01:21.355 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-19T13:01:21.356 DEBUG:teuthology.orchestra.run.smithi057:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-19T13:01:21.411 DEBUG:teuthology.orchestra.run.smithi179:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-19T13:01:21.456 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-19T13:01:21.457 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T13:01:21.457 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T13:01:21.472 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055614,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=358 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248260k,nr_inodes=812065,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-19T13:01:21.472 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-19T13:01:21.473 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T13:01:21.488 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054833,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=284 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3247632k,nr_inodes=811908,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-19T13:01:21.488 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T13:01:21.489 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T13:01:21.528 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055614,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=358 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248260k,nr_inodes=812065,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-19T13:01:21.528 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-19T13:01:21.529 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T13:01:21.543 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054833,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=284 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3247632k,nr_inodes=811908,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-19T13:01:21.544 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-19T13:01:21.544 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T13:01:21.588 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:21.585+0000 7fb508163640 -1 client.7141: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.7030:f55302c0-37aa-4235-8ccc-9775bc542785:v1:172.21.15.179:0/3304791412] 2023-10-19T13:01:21.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.676+0000 7efc5ea44640 1 -- 172.21.15.57:0/1192563346 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efc580b0530 msgr2=0x7efc580b09a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:21.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.676+0000 7efc5ea44640 1 --2- 172.21.15.57:0/1192563346 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efc580b0530 0x7efc580b09a0 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7efc580cc7e0 tx=0x7efc5002c2f0 comp rx=0 tx=0).stop 2023-10-19T13:01:21.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.676+0000 7efc5ea44640 1 -- 172.21.15.57:0/1192563346 shutdown_connections 2023-10-19T13:01:21.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.676+0000 7efc5ea44640 1 --2- 172.21.15.57:0/1192563346 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efc580b0530 0x7efc580b09a0 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:21.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.676+0000 7efc5ea44640 1 --2- 172.21.15.57:0/1192563346 >> v2:172.21.15.179:3300/0 conn(0x7efc580cbd20 0x7efc58158300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:21.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.676+0000 7efc5ea44640 1 -- 172.21.15.57:0/1192563346 >> 172.21.15.57:0/1192563346 conn(0x7efc58069930 msgr2=0x7efc58069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:21.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.676+0000 7efc5ea44640 1 -- 172.21.15.57:0/1192563346 shutdown_connections 2023-10-19T13:01:21.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.676+0000 7efc5ea44640 1 -- 172.21.15.57:0/1192563346 wait complete. 2023-10-19T13:01:21.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.677+0000 7efc5ea44640 1 Processor -- start 2023-10-19T13:01:21.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.677+0000 7efc5ea44640 1 -- start start 2023-10-19T13:01:21.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.678+0000 7efc5ea44640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efc580afa20 0x7efc58158980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:21.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.678+0000 7efc5ea44640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efc580b0530 0x7efc581338e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:21.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.678+0000 7efc5ea44640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7efc580cbd20 0x7efc58164220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:21.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.678+0000 7efc5ea44640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc581458e0 con 0x7efc580afa20 2023-10-19T13:01:21.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.678+0000 7efc5ea44640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc5813e6c0 con 0x7efc580b0530 2023-10-19T13:01:21.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.678+0000 7efc5ea44640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc5813e9a0 con 0x7efc580cbd20 2023-10-19T13:01:21.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.678+0000 7efc57fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efc580afa20 0x7efc58158980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:21.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.678+0000 7efc57fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efc580afa20 0x7efc58158980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:58540/0 (socket says 172.21.15.57:58540) 2023-10-19T13:01:21.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.678+0000 7efc57fff640 1 -- 172.21.15.57:0/1677412904 learned_addr learned my addr 172.21.15.57:0/1677412904 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:01:21.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.678+0000 7efc577fe640 1 --2- 172.21.15.57:0/1677412904 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efc580b0530 0x7efc581338e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:21.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.678+0000 7efc5cfe1640 1 --2- 172.21.15.57:0/1677412904 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7efc580cbd20 0x7efc58164220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:21.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.678+0000 7efc57fff640 1 -- 172.21.15.57:0/1677412904 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7efc580cbd20 msgr2=0x7efc58164220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:21.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.678+0000 7efc57fff640 1 --2- 172.21.15.57:0/1677412904 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7efc580cbd20 0x7efc58164220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:21.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.678+0000 7efc57fff640 1 -- 172.21.15.57:0/1677412904 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efc580b0530 msgr2=0x7efc581338e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:21.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.678+0000 7efc57fff640 1 --2- 172.21.15.57:0/1677412904 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efc580b0530 0x7efc581338e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:21.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.678+0000 7efc57fff640 1 -- 172.21.15.57:0/1677412904 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc50009900 con 0x7efc580afa20 2023-10-19T13:01:21.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.678+0000 7efc57fff640 1 --2- 172.21.15.57:0/1677412904 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efc580afa20 0x7efc58158980 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7efc4800d740 tx=0x7efc4800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:21.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.679+0000 7efc54ff9640 1 -- 172.21.15.57:0/1677412904 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc48009c30 con 0x7efc580afa20 2023-10-19T13:01:21.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.679+0000 7efc54ff9640 1 -- 172.21.15.57:0/1677412904 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7efc48009e80 con 0x7efc580afa20 2023-10-19T13:01:21.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.679+0000 7efc5ea44640 1 -- 172.21.15.57:0/1677412904 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc5813ec80 con 0x7efc580afa20 2023-10-19T13:01:21.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.679+0000 7efc54ff9640 1 -- 172.21.15.57:0/1677412904 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc480106c0 con 0x7efc580afa20 2023-10-19T13:01:21.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.679+0000 7efc5ea44640 1 -- 172.21.15.57:0/1677412904 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc5813f240 con 0x7efc580afa20 2023-10-19T13:01:21.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.680+0000 7efc5ea44640 1 -- 172.21.15.57:0/1677412904 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc5813f400 con 0x7efc580afa20 2023-10-19T13:01:21.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.680+0000 7efc54ff9640 1 -- 172.21.15.57:0/1677412904 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 47) v1 ==== 89275+0+0 (secure 0 0 0) 0x7efc48010820 con 0x7efc580afa20 2023-10-19T13:01:21.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.681+0000 7efc54ff9640 1 --2- 172.21.15.57:0/1677412904 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7efc4406aed0 0x7efc4406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:21.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.681+0000 7efc54ff9640 1 -- 172.21.15.57:0/1677412904 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5979+0+0 (secure 0 0 0) 0x7efc48089dd0 con 0x7efc580afa20 2023-10-19T13:01:21.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.681+0000 7efc577fe640 1 --2- 172.21.15.57:0/1677412904 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7efc4406aed0 0x7efc4406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:21.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.681+0000 7efc577fe640 1 --2- 172.21.15.57:0/1677412904 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7efc4406aed0 0x7efc4406d390 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7efc5806c690 tx=0x7efc4c009040 comp rx=0 tx=0).ready entity=mgr.7072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:21.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.685+0000 7efc54ff9640 1 -- 172.21.15.57:0/1677412904 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7efc48054c20 con 0x7efc580afa20 2023-10-19T13:01:21.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.896+0000 7efc5ea44640 1 -- 172.21.15.57:0/1677412904 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efc580690e0 con 0x7efc580afa20 2023-10-19T13:01:21.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.897+0000 7efc54ff9640 1 -- 172.21.15.57:0/1677412904 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) v1 ==== 94+0+3705 (secure 0 0 0) 0x7efc48021090 con 0x7efc580afa20 2023-10-19T13:01:21.900 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:01:21.900 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":39,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6305,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/2631343514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":2631343514},{"type":"v1","addr":"172.21.15.179:6837","nonce":2631343514}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":34},{"gid":6883,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/4008617188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":4008617188},{"type":"v1","addr":"172.21.15.179:6836","nonce":4008617188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":34},{"gid":7542,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/2010253133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2010253133},{"type":"v1","addr":"172.21.15.57:6834","nonce":2010253133}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":34},{"gid":7938,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/4073481087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":4073481087},{"type":"v1","addr":"172.21.15.57:6835","nonce":4073481087}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39}],"filesystems":[]} 2023-10-19T13:01:21.901 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 39 2023-10-19T13:01:21.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.899+0000 7efc5ea44640 1 -- 172.21.15.57:0/1677412904 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7efc4406aed0 msgr2=0x7efc4406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:21.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.899+0000 7efc5ea44640 1 --2- 172.21.15.57:0/1677412904 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7efc4406aed0 0x7efc4406d390 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7efc5806c690 tx=0x7efc4c009040 comp rx=0 tx=0).stop 2023-10-19T13:01:21.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.899+0000 7efc5ea44640 1 -- 172.21.15.57:0/1677412904 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efc580afa20 msgr2=0x7efc58158980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:21.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.899+0000 7efc5ea44640 1 --2- 172.21.15.57:0/1677412904 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efc580afa20 0x7efc58158980 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7efc4800d740 tx=0x7efc4800dc00 comp rx=0 tx=0).stop 2023-10-19T13:01:21.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.899+0000 7efc5ea44640 1 -- 172.21.15.57:0/1677412904 shutdown_connections 2023-10-19T13:01:21.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.899+0000 7efc5ea44640 1 --2- 172.21.15.57:0/1677412904 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7efc4406aed0 0x7efc4406d390 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:21.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.899+0000 7efc5ea44640 1 --2- 172.21.15.57:0/1677412904 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7efc580cbd20 0x7efc58164220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:21.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.899+0000 7efc5ea44640 1 --2- 172.21.15.57:0/1677412904 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efc580b0530 0x7efc581338e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:21.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.899+0000 7efc5ea44640 1 --2- 172.21.15.57:0/1677412904 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efc580afa20 0x7efc58158980 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:21.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.899+0000 7efc5ea44640 1 -- 172.21.15.57:0/1677412904 >> 172.21.15.57:0/1677412904 conn(0x7efc58069930 msgr2=0x7efc580aef90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:21.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.899+0000 7efc5ea44640 1 -- 172.21.15.57:0/1677412904 shutdown_connections 2023-10-19T13:01:21.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:21.899+0000 7efc5ea44640 1 -- 172.21.15.57:0/1677412904 wait complete. 2023-10-19T13:01:21.912 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-19T13:01:21.913 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-19T13:01:21.913 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-19T13:01:21.913 INFO:teuthology.orchestra.run:waiting for 300 2023-10-19T13:01:21.914 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-19T13:01:21.914 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-19T13:01:21.914 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-19T13:01:21.914 INFO:teuthology.orchestra.run:waiting for 300 2023-10-19T13:01:21.915 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-19T13:01:21.915 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-19T13:01:21.915 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-19T13:01:21.915 INFO:teuthology.orchestra.run:waiting for 300 2023-10-19T13:01:21.916 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-19T13:01:21.916 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-19T13:01:21.916 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-19T13:01:21.917 INFO:teuthology.orchestra.run:waiting for 300 2023-10-19T13:01:21.924 INFO:tasks.ceph.mds.a:Stopped 2023-10-19T13:01:21.924 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-19T13:01:21.926 INFO:tasks.ceph.mds.b:Stopped 2023-10-19T13:01:21.926 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-19T13:01:21.927 INFO:tasks.ceph.mds.c:Stopped 2023-10-19T13:01:21.927 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-19T13:01:21.928 INFO:tasks.ceph.mds.d:Stopped 2023-10-19T13:01:21.928 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-19T13:01:21.968 INFO:tasks.ceph.mds.b:Started 2023-10-19T13:01:21.968 DEBUG:teuthology.parallel:result is None 2023-10-19T13:01:21.969 INFO:tasks.ceph.mds.a:Started 2023-10-19T13:01:21.970 INFO:tasks.ceph.mds.d:Started 2023-10-19T13:01:21.970 DEBUG:teuthology.parallel:result is None 2023-10-19T13:01:21.970 DEBUG:teuthology.parallel:result is None 2023-10-19T13:01:21.971 INFO:tasks.ceph.mds.c:Started 2023-10-19T13:01:21.972 DEBUG:teuthology.parallel:result is None 2023-10-19T13:01:21.972 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-19T13:01:22.043 INFO:tasks.ceph.mds.b.smithi179.stdout:starting mds.b at 2023-10-19T13:01:22.043 INFO:tasks.ceph.mds.d.smithi179.stdout:starting mds.d at 2023-10-19T13:01:22.050 INFO:tasks.ceph.mds.a.smithi057.stdout:starting mds.a at 2023-10-19T13:01:22.051 INFO:tasks.ceph.mds.c.smithi057.stdout:starting mds.c at 2023-10-19T13:01:22.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.086+0000 7f47c5d2c640 1 -- 172.21.15.57:0/1739669879 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f47c015b2a0 msgr2=0x7f47c00d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:22.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.086+0000 7f47c5d2c640 1 --2- 172.21.15.57:0/1739669879 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f47c015b2a0 0x7f47c00d1910 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f47c015b680 tx=0x7f47b402bcf0 comp rx=0 tx=0).stop 2023-10-19T13:01:22.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.086+0000 7f47c5d2c640 1 -- 172.21.15.57:0/1739669879 shutdown_connections 2023-10-19T13:01:22.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.086+0000 7f47c5d2c640 1 --2- 172.21.15.57:0/1739669879 >> v2:172.21.15.179:3300/0 conn(0x7f47c00ceb40 0x7f47c00b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:22.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.086+0000 7f47c5d2c640 1 --2- 172.21.15.57:0/1739669879 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f47c015b2a0 0x7f47c00d1910 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:22.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.086+0000 7f47c5d2c640 1 -- 172.21.15.57:0/1739669879 >> 172.21.15.57:0/1739669879 conn(0x7f47c00d0d60 msgr2=0x7f47c00d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:22.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.086+0000 7f47c5d2c640 1 -- 172.21.15.57:0/1739669879 shutdown_connections 2023-10-19T13:01:22.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.086+0000 7f47c5d2c640 1 -- 172.21.15.57:0/1739669879 wait complete. 2023-10-19T13:01:22.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.087+0000 7f47c5d2c640 1 Processor -- start 2023-10-19T13:01:22.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.088+0000 7f47c5d2c640 1 -- start start 2023-10-19T13:01:22.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.088+0000 7f47c5d2c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f47c00ceb40 0x7f47c006a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:22.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.088+0000 7f47c5d2c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f47c00cf340 0x7f47c00b38e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:22.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.088+0000 7f47c5d2c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f47c015b2a0 0x7f47c015e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:22.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.088+0000 7f47c5d2c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47c0153600 con 0x7f47c015b2a0 2023-10-19T13:01:22.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.088+0000 7f47c5d2c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47c006a6e0 con 0x7f47c00ceb40 2023-10-19T13:01:22.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.088+0000 7f47c5d2c640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47c00b3e20 con 0x7f47c00cf340 2023-10-19T13:01:22.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.088+0000 7f47bffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f47c015b2a0 0x7f47c015e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:22.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.088+0000 7f47bffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f47c015b2a0 0x7f47c015e0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:58584/0 (socket says 172.21.15.57:58584) 2023-10-19T13:01:22.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.088+0000 7f47bffff640 1 -- 172.21.15.57:0/1253170719 learned_addr learned my addr 172.21.15.57:0/1253170719 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:01:22.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.088+0000 7f47bf7fe640 1 --2- 172.21.15.57:0/1253170719 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f47c00ceb40 0x7f47c006a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:22.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.088+0000 7f47beffd640 1 --2- 172.21.15.57:0/1253170719 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f47c00cf340 0x7f47c00b38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:22.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.089+0000 7f47bffff640 1 -- 172.21.15.57:0/1253170719 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f47c00cf340 msgr2=0x7f47c00b38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:22.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.089+0000 7f47bffff640 1 --2- 172.21.15.57:0/1253170719 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f47c00cf340 0x7f47c00b38e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:22.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.089+0000 7f47bffff640 1 -- 172.21.15.57:0/1253170719 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f47c00ceb40 msgr2=0x7f47c006a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:22.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.089+0000 7f47bffff640 1 --2- 172.21.15.57:0/1253170719 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f47c00ceb40 0x7f47c006a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:22.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.089+0000 7f47bffff640 1 -- 172.21.15.57:0/1253170719 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47b4009490 con 0x7f47c015b2a0 2023-10-19T13:01:22.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.089+0000 7f47beffd640 1 --2- 172.21.15.57:0/1253170719 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f47c00cf340 0x7f47c00b38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:01:22.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.089+0000 7f47bf7fe640 1 --2- 172.21.15.57:0/1253170719 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f47c00ceb40 0x7f47c006a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:01:22.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.089+0000 7f47bffff640 1 --2- 172.21.15.57:0/1253170719 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f47c015b2a0 0x7f47c015e0c0 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f47b800eb10 tx=0x7f47b800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:22.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.089+0000 7f47a7fff640 1 -- 172.21.15.57:0/1253170719 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47b8004b20 con 0x7f47c015b2a0 2023-10-19T13:01:22.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.089+0000 7f47a7fff640 1 -- 172.21.15.57:0/1253170719 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f47b8004ca0 con 0x7f47c015b2a0 2023-10-19T13:01:22.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.089+0000 7f47c5d2c640 1 -- 172.21.15.57:0/1253170719 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47c015c590 con 0x7f47c015b2a0 2023-10-19T13:01:22.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.089+0000 7f47a7fff640 1 -- 172.21.15.57:0/1253170719 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47b8005280 con 0x7f47c015b2a0 2023-10-19T13:01:22.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.089+0000 7f47c5d2c640 1 -- 172.21.15.57:0/1253170719 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47c0153220 con 0x7f47c015b2a0 2023-10-19T13:01:22.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.090+0000 7f47c5d2c640 1 -- 172.21.15.57:0/1253170719 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47c0153740 con 0x7f47c015b2a0 2023-10-19T13:01:22.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.091+0000 7f47a7fff640 1 -- 172.21.15.57:0/1253170719 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 47) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f47b80053e0 con 0x7f47c015b2a0 2023-10-19T13:01:22.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.091+0000 7f47a7fff640 1 --2- 172.21.15.57:0/1253170719 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f47ac06aed0 0x7f47ac06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:22.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.091+0000 7f47a7fff640 1 -- 172.21.15.57:0/1253170719 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5979+0+0 (secure 0 0 0) 0x7f47b8014070 con 0x7f47c015b2a0 2023-10-19T13:01:22.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.091+0000 7f47bf7fe640 1 --2- 172.21.15.57:0/1253170719 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f47ac06aed0 0x7f47ac06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:22.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.092+0000 7f47bf7fe640 1 --2- 172.21.15.57:0/1253170719 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f47ac06aed0 0x7f47ac06d390 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f47b00046b0 tx=0x7f47b0009040 comp rx=0 tx=0).ready entity=mgr.7072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:22.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.095+0000 7f47a7fff640 1 -- 172.21.15.57:0/1253170719 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f47c0153740 con 0x7f47c015b2a0 2023-10-19T13:01:22.286 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:22.283+0000 7fb508163640 -1 client.7141: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.7030:f55302c0-37aa-4235-8ccc-9775bc542785:v1:172.21.15.179:0/3304791412] 2023-10-19T13:01:22.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.289+0000 7f47c5d2c640 1 -- 172.21.15.57:0/1253170719 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f47c0153740 con 0x7f47c015b2a0 2023-10-19T13:01:22.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.738+0000 7f47a7fff640 1 -- 172.21.15.57:0/1253170719 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v107) v1 ==== 97+0+0 (secure 0 0 0) 0x7f47c0153740 con 0x7f47c015b2a0 2023-10-19T13:01:22.741 INFO:teuthology.orchestra.run.smithi057.stderr: removed all blocklist entries 2023-10-19T13:01:22.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.741+0000 7f47c5d2c640 1 -- 172.21.15.57:0/1253170719 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f47ac06aed0 msgr2=0x7f47ac06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:22.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.741+0000 7f47c5d2c640 1 --2- 172.21.15.57:0/1253170719 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f47ac06aed0 0x7f47ac06d390 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f47b00046b0 tx=0x7f47b0009040 comp rx=0 tx=0).stop 2023-10-19T13:01:22.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.741+0000 7f47c5d2c640 1 -- 172.21.15.57:0/1253170719 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f47c015b2a0 msgr2=0x7f47c015e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:22.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.741+0000 7f47c5d2c640 1 --2- 172.21.15.57:0/1253170719 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f47c015b2a0 0x7f47c015e0c0 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f47b800eb10 tx=0x7f47b800c2d0 comp rx=0 tx=0).stop 2023-10-19T13:01:22.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.741+0000 7f47c5d2c640 1 -- 172.21.15.57:0/1253170719 shutdown_connections 2023-10-19T13:01:22.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.741+0000 7f47c5d2c640 1 --2- 172.21.15.57:0/1253170719 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f47ac06aed0 0x7f47ac06d390 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:22.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.741+0000 7f47c5d2c640 1 --2- 172.21.15.57:0/1253170719 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f47c015b2a0 0x7f47c015e0c0 unknown :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:22.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.741+0000 7f47c5d2c640 1 --2- 172.21.15.57:0/1253170719 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f47c00cf340 0x7f47c00b38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:22.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.741+0000 7f47c5d2c640 1 --2- 172.21.15.57:0/1253170719 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f47c00ceb40 0x7f47c006a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:22.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.741+0000 7f47c5d2c640 1 -- 172.21.15.57:0/1253170719 >> 172.21.15.57:0/1253170719 conn(0x7f47c00d0d60 msgr2=0x7f47c00cc800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:22.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.741+0000 7f47c5d2c640 1 -- 172.21.15.57:0/1253170719 shutdown_connections 2023-10-19T13:01:22.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.741+0000 7f47c5d2c640 1 -- 172.21.15.57:0/1253170719 wait complete. 2023-10-19T13:01:22.755 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-19T13:01:22.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.864+0000 7f5fecb7d640 1 -- 172.21.15.57:0/4175320092 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5fe815b2a0 msgr2=0x7f5fe80cf720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:22.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.864+0000 7f5fecb7d640 1 --2- 172.21.15.57:0/4175320092 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5fe815b2a0 0x7f5fe80cf720 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f5fd0005fd0 tx=0x7f5fd002bd90 comp rx=0 tx=0).stop 2023-10-19T13:01:22.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.864+0000 7f5fecb7d640 1 -- 172.21.15.57:0/4175320092 shutdown_connections 2023-10-19T13:01:22.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.864+0000 7f5fecb7d640 1 --2- 172.21.15.57:0/4175320092 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5fe815b2a0 0x7f5fe80cf720 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:22.868 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.864+0000 7f5fecb7d640 1 --2- 172.21.15.57:0/4175320092 >> v2:172.21.15.179:3300/0 conn(0x7f5fe80cf340 0x7f5fe80d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:22.868 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.864+0000 7f5fecb7d640 1 -- 172.21.15.57:0/4175320092 >> 172.21.15.57:0/4175320092 conn(0x7f5fe80d0d60 msgr2=0x7f5fe80d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:22.868 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.864+0000 7f5fecb7d640 1 -- 172.21.15.57:0/4175320092 shutdown_connections 2023-10-19T13:01:22.868 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.864+0000 7f5fecb7d640 1 -- 172.21.15.57:0/4175320092 wait complete. 2023-10-19T13:01:22.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.865+0000 7f5fecb7d640 1 Processor -- start 2023-10-19T13:01:22.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.865+0000 7f5fecb7d640 1 -- start start 2023-10-19T13:01:22.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.866+0000 7f5fecb7d640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5fe80ceb40 0x7f5fe814c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:22.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.866+0000 7f5fecb7d640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5fe80cf340 0x7f5fe814cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:22.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.866+0000 7f5fecb7d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5fe815b2a0 0x7f5fe815df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:22.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.866+0000 7f5fecb7d640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe80c8e90 con 0x7f5fe815b2a0 2023-10-19T13:01:22.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.866+0000 7f5fecb7d640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe814c810 con 0x7f5fe80ceb40 2023-10-19T13:01:22.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.866+0000 7f5fecb7d640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe80c8210 con 0x7f5fe80cf340 2023-10-19T13:01:22.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.866+0000 7f5fe659c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5fe80ceb40 0x7f5fe814c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:22.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.866+0000 7f5fe5d9b640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5fe80cf340 0x7f5fe814cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:22.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.866+0000 7f5fe6d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5fe815b2a0 0x7f5fe815df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:22.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.866+0000 7f5fe5d9b640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5fe80cf340 0x7f5fe814cd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3301/0 says I am v2:172.21.15.57:39996/0 (socket says 172.21.15.57:39996) 2023-10-19T13:01:22.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.866+0000 7f5fe659c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5fe80ceb40 0x7f5fe814c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.57:39832/0 (socket says 172.21.15.57:39832) 2023-10-19T13:01:22.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.866+0000 7f5fe5d9b640 1 -- 172.21.15.57:0/2348957392 learned_addr learned my addr 172.21.15.57:0/2348957392 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:01:22.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.866+0000 7f5fe5d9b640 1 -- 172.21.15.57:0/2348957392 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5fe80ceb40 msgr2=0x7f5fe814c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:22.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.866+0000 7f5fe5d9b640 1 --2- 172.21.15.57:0/2348957392 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5fe80ceb40 0x7f5fe814c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:22.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.866+0000 7f5fe5d9b640 1 -- 172.21.15.57:0/2348957392 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5fe815b2a0 msgr2=0x7f5fe815df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:22.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.866+0000 7f5fe5d9b640 1 --2- 172.21.15.57:0/2348957392 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5fe815b2a0 0x7f5fe815df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:22.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.866+0000 7f5fe5d9b640 1 -- 172.21.15.57:0/2348957392 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fd8009840 con 0x7f5fe80cf340 2023-10-19T13:01:22.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.866+0000 7f5fe6d9d640 1 --2- 172.21.15.57:0/2348957392 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5fe815b2a0 0x7f5fe815df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T13:01:22.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.867+0000 7f5fe5d9b640 1 --2- 172.21.15.57:0/2348957392 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5fe80cf340 0x7f5fe814cd20 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f5fd00099f0 tx=0x7f5fd002d9f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:22.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.867+0000 7f5fdeffd640 1 -- 172.21.15.57:0/2348957392 <== mon.2 v2:172.21.15.179:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fd002ddf0 con 0x7f5fe80cf340 2023-10-19T13:01:22.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.867+0000 7f5fdeffd640 1 -- 172.21.15.57:0/2348957392 <== mon.2 v2:172.21.15.179:3301/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f5fd00036e0 con 0x7f5fe80cf340 2023-10-19T13:01:22.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.867+0000 7f5fdeffd640 1 -- 172.21.15.57:0/2348957392 <== mon.2 v2:172.21.15.179:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fd0003af0 con 0x7f5fe80cf340 2023-10-19T13:01:22.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.867+0000 7f5fecb7d640 1 -- 172.21.15.57:0/2348957392 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fd0009560 con 0x7f5fe80cf340 2023-10-19T13:01:22.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.867+0000 7f5fecb7d640 1 -- 172.21.15.57:0/2348957392 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fe80c87d0 con 0x7f5fe80cf340 2023-10-19T13:01:22.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.868+0000 7f5fecb7d640 1 -- 172.21.15.57:0/2348957392 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fe80690e0 con 0x7f5fe80cf340 2023-10-19T13:01:22.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.868+0000 7f5fdeffd640 1 -- 172.21.15.57:0/2348957392 <== mon.2 v2:172.21.15.179:3301/0 4 ==== mgrmap(e 47) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f5fd0003c50 con 0x7f5fe80cf340 2023-10-19T13:01:22.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.869+0000 7f5fdeffd640 1 --2- 172.21.15.57:0/2348957392 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f5fc806aed0 0x7f5fc806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:22.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.869+0000 7f5fdeffd640 1 -- 172.21.15.57:0/2348957392 <== mon.2 v2:172.21.15.179:3301/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f5fd00abbc0 con 0x7f5fe80cf340 2023-10-19T13:01:22.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.869+0000 7f5fe659c640 1 --2- 172.21.15.57:0/2348957392 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f5fc806aed0 0x7f5fc806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:22.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.869+0000 7f5fe659c640 1 --2- 172.21.15.57:0/2348957392 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f5fc806aed0 0x7f5fc806d390 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5fd8005fd0 tx=0x7f5fd8009500 comp rx=0 tx=0).ready entity=mgr.7072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:22.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:22.873+0000 7f5fdeffd640 1 -- 172.21.15.57:0/2348957392 <== mon.2 v2:172.21.15.179:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f5fd0076f30 con 0x7f5fe80cf340 2023-10-19T13:01:22.988 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:22.985+0000 7fb508163640 -1 client.7141: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.7030:f55302c0-37aa-4235-8ccc-9775bc542785:v1:172.21.15.179:0/3304791412] 2023-10-19T13:01:23.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.121+0000 7f5fecb7d640 1 -- 172.21.15.57:0/2348957392 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f5fe80d0b40 con 0x7f5fe80cf340 2023-10-19T13:01:23.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.122+0000 7f5fdeffd640 1 -- 172.21.15.57:0/2348957392 <== mon.2 v2:172.21.15.179:3301/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v25) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f5fd0042020 con 0x7f5fe80cf340 2023-10-19T13:01:23.125 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:01:23.125 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2023-10-19T13:01:23.125 INFO:teuthology.orchestra.run.smithi057.stdout: "auth_dump": [ 2023-10-19T13:01:23.125 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:01:23.126 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "mds.a", 2023-10-19T13:01:23.126 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBjJDFlmbVANhAAP/ynHlU/OxaRq5lQoAa8Ng==", 2023-10-19T13:01:23.126 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:01:23.126 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T13:01:23.126 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow *", 2023-10-19T13:01:23.127 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow *", 2023-10-19T13:01:23.127 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:01:23.127 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:01:23.127 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:01:23.127 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:01:23.128 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "mds.b", 2023-10-19T13:01:23.128 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFl9l7RBxAAC0geI2tJ1SIZZr1++kS/BA==", 2023-10-19T13:01:23.128 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:01:23.128 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T13:01:23.128 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow *", 2023-10-19T13:01:23.129 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow *", 2023-10-19T13:01:23.129 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:01:23.129 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:01:23.129 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:01:23.129 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:01:23.130 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "mds.c", 2023-10-19T13:01:23.130 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFlFkaoAhAA/PBizkJ4bm958oP4CmPuIQ==", 2023-10-19T13:01:23.130 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:01:23.130 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T13:01:23.130 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow *", 2023-10-19T13:01:23.131 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow *", 2023-10-19T13:01:23.131 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:01:23.131 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:01:23.131 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:01:23.131 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:01:23.132 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "mds.d", 2023-10-19T13:01:23.132 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFlx8skDxAAAfCHTYtYJgJik0WHSWeQfQ==", 2023-10-19T13:01:23.132 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:01:23.132 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T13:01:23.132 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow *", 2023-10-19T13:01:23.133 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow *", 2023-10-19T13:01:23.133 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:01:23.133 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:01:23.133 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:01:23.133 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:01:23.133 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.0", 2023-10-19T13:01:23.134 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBrJDFlCo9sBxAAQN2cID4nCQgRwGaGFEV8LQ==", 2023-10-19T13:01:23.134 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:01:23.134 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T13:01:23.134 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T13:01:23.134 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:01:23.135 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:01:23.135 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:01:23.135 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:01:23.135 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.1", 2023-10-19T13:01:23.135 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBuJDFl4fsLCBAAp3Pi80yZtc5sju0GisEuVA==", 2023-10-19T13:01:23.136 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:01:23.136 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T13:01:23.136 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T13:01:23.136 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:01:23.136 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:01:23.136 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:01:23.137 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:01:23.137 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.2", 2023-10-19T13:01:23.137 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBxJDFl+KAKCBAAd4N9cluARxPVfBLU+7Hd7A==", 2023-10-19T13:01:23.137 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:01:23.137 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T13:01:23.137 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T13:01:23.138 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:01:23.138 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:01:23.138 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:01:23.138 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:01:23.139 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.3", 2023-10-19T13:01:23.139 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQB0JDFlU/toCBAAP0Ou1zcHKNx5SfbybJkh4Q==", 2023-10-19T13:01:23.139 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:01:23.139 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T13:01:23.139 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T13:01:23.139 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:01:23.140 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:01:23.140 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:01:23.140 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:01:23.140 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.4", 2023-10-19T13:01:23.140 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQB6JDFlc6stEBAALkgNyxoiCTzZnKpBSPxI6w==", 2023-10-19T13:01:23.141 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:01:23.141 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T13:01:23.141 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T13:01:23.141 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:01:23.141 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:01:23.141 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:01:23.142 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:01:23.142 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.5", 2023-10-19T13:01:23.142 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQB9JDFlI3cNDhAAuG3tHJ6RvPwAzPE76wTIVQ==", 2023-10-19T13:01:23.142 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:01:23.142 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T13:01:23.143 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T13:01:23.143 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:01:23.143 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:01:23.143 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:01:23.143 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:01:23.144 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.6", 2023-10-19T13:01:23.144 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQCAJDFlLPumCxAA/iy08t/2N9rXfn71KtbSQw==", 2023-10-19T13:01:23.144 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:01:23.144 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T13:01:23.144 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T13:01:23.145 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:01:23.145 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:01:23.145 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:01:23.145 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:01:23.145 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.7", 2023-10-19T13:01:23.145 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQCDJDFlHmrbCRAA+hpWygGIJydfU6cHWlfx/Q==", 2023-10-19T13:01:23.146 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:01:23.146 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T13:01:23.146 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T13:01:23.146 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:01:23.146 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:01:23.147 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:01:23.147 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:01:23.147 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "client.0", 2023-10-19T13:01:23.147 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFlg9yuExAATxiWDfS+p07G1F0pORbJEg==", 2023-10-19T13:01:23.147 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:01:23.147 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T13:01:23.148 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow r", 2023-10-19T13:01:23.148 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-19T13:01:23.148 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:01:23.148 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:01:23.148 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:01:23.149 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "client.1", 2023-10-19T13:01:23.149 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFl9IwYGxAAHPituiJSPkHDuD0QyG4Hww==", 2023-10-19T13:01:23.149 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:01:23.149 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T13:01:23.149 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow r", 2023-10-19T13:01:23.149 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow rw", 2023-10-19T13:01:23.150 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow rwx" 2023-10-19T13:01:23.150 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:01:23.150 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:01:23.150 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:01:23.150 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "client.2", 2023-10-19T13:01:23.151 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFlMd+AFxAAS/UTESDt76GmrhhN2o/Fdw==", 2023-10-19T13:01:23.151 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:01:23.151 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T13:01:23.151 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow r", 2023-10-19T13:01:23.151 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow rw", 2023-10-19T13:01:23.151 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow rwx" 2023-10-19T13:01:23.152 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:01:23.152 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:01:23.152 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:01:23.152 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "client.3", 2023-10-19T13:01:23.152 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFlHZ3fIBAA42B7n8opLS5smXIwJyVYzw==", 2023-10-19T13:01:23.152 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:01:23.153 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T13:01:23.153 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow r", 2023-10-19T13:01:23.153 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow rw", 2023-10-19T13:01:23.153 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow rwx" 2023-10-19T13:01:23.153 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:01:23.154 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:01:23.154 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:01:23.154 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "client.admin", 2023-10-19T13:01:23.154 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBjJDFlBgasFxAAMA24j8KszcXMEpYRQDXj4g==", 2023-10-19T13:01:23.154 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:01:23.155 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow *", 2023-10-19T13:01:23.155 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow *", 2023-10-19T13:01:23.155 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow *", 2023-10-19T13:01:23.155 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:01:23.155 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:01:23.155 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:01:23.156 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:01:23.156 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "mgr.x", 2023-10-19T13:01:23.156 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBjJDFllFBfMhAA5+KPXWmYZqcESmLObaUbBA==", 2023-10-19T13:01:23.156 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:01:23.156 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow *", 2023-10-19T13:01:23.157 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile mgr", 2023-10-19T13:01:23.157 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:01:23.157 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:01:23.157 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:01:23.157 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:01:23.158 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "mgr.y", 2023-10-19T13:01:23.158 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBjJDFlxRzJLRAAGjG58o7qXAHvIEIRYwl6QA==", 2023-10-19T13:01:23.158 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:01:23.158 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow *", 2023-10-19T13:01:23.158 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile mgr", 2023-10-19T13:01:23.159 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:01:23.159 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:01:23.159 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:01:23.159 INFO:teuthology.orchestra.run.smithi057.stdout: ] 2023-10-19T13:01:23.159 INFO:teuthology.orchestra.run.smithi057.stdout:} 2023-10-19T13:01:23.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.124+0000 7f5fecb7d640 1 -- 172.21.15.57:0/2348957392 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f5fc806aed0 msgr2=0x7f5fc806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:23.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.124+0000 7f5fecb7d640 1 --2- 172.21.15.57:0/2348957392 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f5fc806aed0 0x7f5fc806d390 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5fd8005fd0 tx=0x7f5fd8009500 comp rx=0 tx=0).stop 2023-10-19T13:01:23.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.124+0000 7f5fecb7d640 1 -- 172.21.15.57:0/2348957392 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5fe80cf340 msgr2=0x7f5fe814cd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:23.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.124+0000 7f5fecb7d640 1 --2- 172.21.15.57:0/2348957392 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5fe80cf340 0x7f5fe814cd20 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f5fd00099f0 tx=0x7f5fd002d9f0 comp rx=0 tx=0).stop 2023-10-19T13:01:23.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.124+0000 7f5fecb7d640 1 -- 172.21.15.57:0/2348957392 shutdown_connections 2023-10-19T13:01:23.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.124+0000 7f5fecb7d640 1 --2- 172.21.15.57:0/2348957392 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f5fc806aed0 0x7f5fc806d390 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:23.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.124+0000 7f5fecb7d640 1 --2- 172.21.15.57:0/2348957392 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5fe815b2a0 0x7f5fe815df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:23.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.124+0000 7f5fecb7d640 1 --2- 172.21.15.57:0/2348957392 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5fe80cf340 0x7f5fe814cd20 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:23.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.124+0000 7f5fecb7d640 1 --2- 172.21.15.57:0/2348957392 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5fe80ceb40 0x7f5fe814c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:23.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.124+0000 7f5fecb7d640 1 -- 172.21.15.57:0/2348957392 >> 172.21.15.57:0/2348957392 conn(0x7f5fe80d0d60 msgr2=0x7f5fe815a110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:23.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.124+0000 7f5fecb7d640 1 -- 172.21.15.57:0/2348957392 shutdown_connections 2023-10-19T13:01:23.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.124+0000 7f5fecb7d640 1 -- 172.21.15.57:0/2348957392 wait complete. 2023-10-19T13:01:23.164 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T13:01:23.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.271+0000 7f6cebc40640 1 -- 172.21.15.57:0/2323056208 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6ce415b000 msgr2=0x7f6ce40d1810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:23.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.271+0000 7f6cebc40640 1 --2- 172.21.15.57:0/2323056208 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6ce415b000 0x7f6ce40d1810 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f6ce40b37f0 tx=0x7f6ce002be40 comp rx=0 tx=0).stop 2023-10-19T13:01:23.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.272+0000 7f6cebc40640 1 -- 172.21.15.57:0/2323056208 shutdown_connections 2023-10-19T13:01:23.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.272+0000 7f6cebc40640 1 --2- 172.21.15.57:0/2323056208 >> v2:172.21.15.179:3300/0 conn(0x7f6ce415b3e0 0x7f6ce40cf2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:23.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.272+0000 7f6cebc40640 1 --2- 172.21.15.57:0/2323056208 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6ce415b000 0x7f6ce40d1810 unknown :-1 s=CLOSED pgs=1259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:23.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.272+0000 7f6cebc40640 1 -- 172.21.15.57:0/2323056208 >> 172.21.15.57:0/2323056208 conn(0x7f6ce40d0cf0 msgr2=0x7f6ce40d1100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:23.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.272+0000 7f6cebc40640 1 -- 172.21.15.57:0/2323056208 shutdown_connections 2023-10-19T13:01:23.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.272+0000 7f6cebc40640 1 -- 172.21.15.57:0/2323056208 wait complete. 2023-10-19T13:01:23.279 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.273+0000 7f6cebc40640 1 Processor -- start 2023-10-19T13:01:23.279 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.273+0000 7f6cebc40640 1 -- start start 2023-10-19T13:01:23.280 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.274+0000 7f6cebc40640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6ce40afdc0 0x7f6ce40d1340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:23.280 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.274+0000 7f6cebc40640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6ce415b000 0x7f6ce415dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:23.280 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.274+0000 7f6cebc40640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6ce415b3e0 0x7f6ce40c7dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:23.280 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.274+0000 7f6cebc40640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce40ca2c0 con 0x7f6ce40afdc0 2023-10-19T13:01:23.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.274+0000 7f6cebc40640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce40c9640 con 0x7f6ce415b3e0 2023-10-19T13:01:23.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.274+0000 7f6cebc40640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce40c9920 con 0x7f6ce415b000 2023-10-19T13:01:23.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.274+0000 7f6ce99dc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6ce40afdc0 0x7f6ce40d1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:23.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.274+0000 7f6ce99dc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6ce40afdc0 0x7f6ce40d1340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:58614/0 (socket says 172.21.15.57:58614) 2023-10-19T13:01:23.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.274+0000 7f6ce99dc640 1 -- 172.21.15.57:0/453878360 learned_addr learned my addr 172.21.15.57:0/453878360 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:01:23.282 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.274+0000 7f6ce91db640 1 --2- 172.21.15.57:0/453878360 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6ce415b000 0x7f6ce415dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:23.282 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.274+0000 7f6cea1dd640 1 --2- 172.21.15.57:0/453878360 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6ce415b3e0 0x7f6ce40c7dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:23.282 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.274+0000 7f6ce99dc640 1 -- 172.21.15.57:0/453878360 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6ce415b000 msgr2=0x7f6ce415dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:23.282 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.274+0000 7f6ce99dc640 1 --2- 172.21.15.57:0/453878360 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6ce415b000 0x7f6ce415dec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:23.282 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.274+0000 7f6ce99dc640 1 -- 172.21.15.57:0/453878360 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6ce415b3e0 msgr2=0x7f6ce40c7dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:23.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.274+0000 7f6ce99dc640 1 --2- 172.21.15.57:0/453878360 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6ce415b3e0 0x7f6ce40c7dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:23.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.274+0000 7f6ce99dc640 1 -- 172.21.15.57:0/453878360 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ce0009560 con 0x7f6ce40afdc0 2023-10-19T13:01:23.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.274+0000 7f6cea1dd640 1 --2- 172.21.15.57:0/453878360 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6ce415b3e0 0x7f6ce40c7dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:01:23.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.274+0000 7f6ce99dc640 1 --2- 172.21.15.57:0/453878360 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6ce40afdc0 0x7f6ce40d1340 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f6ce0009850 tx=0x7f6ce002db70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:23.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.275+0000 7f6cda7fc640 1 -- 172.21.15.57:0/453878360 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce0003800 con 0x7f6ce40afdc0 2023-10-19T13:01:23.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.275+0000 7f6cda7fc640 1 -- 172.21.15.57:0/453878360 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f6ce0003a50 con 0x7f6ce40afdc0 2023-10-19T13:01:23.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.275+0000 7f6cebc40640 1 -- 172.21.15.57:0/453878360 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ce40c9c00 con 0x7f6ce40afdc0 2023-10-19T13:01:23.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.275+0000 7f6cda7fc640 1 -- 172.21.15.57:0/453878360 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce0003e20 con 0x7f6ce40afdc0 2023-10-19T13:01:23.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.275+0000 7f6cebc40640 1 -- 172.21.15.57:0/453878360 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ce415dba0 con 0x7f6ce40afdc0 2023-10-19T13:01:23.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.276+0000 7f6cebc40640 1 -- 172.21.15.57:0/453878360 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ce40690e0 con 0x7f6ce40afdc0 2023-10-19T13:01:23.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.276+0000 7f6cda7fc640 1 -- 172.21.15.57:0/453878360 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 47) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f6ce003e450 con 0x7f6ce40afdc0 2023-10-19T13:01:23.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.276+0000 7f6cda7fc640 1 --2- 172.21.15.57:0/453878360 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f6cc806aed0 0x7f6cc806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:23.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.277+0000 7f6cda7fc640 1 -- 172.21.15.57:0/453878360 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f6ce00abf40 con 0x7f6ce40afdc0 2023-10-19T13:01:23.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.277+0000 7f6ce91db640 1 --2- 172.21.15.57:0/453878360 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f6cc806aed0 0x7f6cc806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:23.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.277+0000 7f6ce91db640 1 --2- 172.21.15.57:0/453878360 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f6cc806aed0 0x7f6cc806d390 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6ce40b6d40 tx=0x7f6cd400b2e0 comp rx=0 tx=0).ready entity=mgr.7072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:23.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.281+0000 7f6cda7fc640 1 -- 172.21.15.57:0/453878360 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6ce0077180 con 0x7f6ce40afdc0 2023-10-19T13:01:23.462 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.459+0000 7f6cebc40640 1 -- 172.21.15.57:0/453878360 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6ce40b1ef0 con 0x7f6ce40afdc0 2023-10-19T13:01:23.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.460+0000 7f6cda7fc640 1 -- 172.21.15.57:0/453878360 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v109) v1 ==== 74+0+13119 (secure 0 0 0) 0x7f6ce007a7a0 con 0x7f6ce40afdc0 2023-10-19T13:01:23.463 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:01:23.463 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":109,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T13:01:22.756177+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T13:01:23.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.462+0000 7f6cebc40640 1 -- 172.21.15.57:0/453878360 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f6cc806aed0 msgr2=0x7f6cc806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:23.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.462+0000 7f6cebc40640 1 --2- 172.21.15.57:0/453878360 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f6cc806aed0 0x7f6cc806d390 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6ce40b6d40 tx=0x7f6cd400b2e0 comp rx=0 tx=0).stop 2023-10-19T13:01:23.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.462+0000 7f6cebc40640 1 -- 172.21.15.57:0/453878360 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6ce40afdc0 msgr2=0x7f6ce40d1340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:23.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.462+0000 7f6cebc40640 1 --2- 172.21.15.57:0/453878360 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6ce40afdc0 0x7f6ce40d1340 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f6ce0009850 tx=0x7f6ce002db70 comp rx=0 tx=0).stop 2023-10-19T13:01:23.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.462+0000 7f6cebc40640 1 -- 172.21.15.57:0/453878360 shutdown_connections 2023-10-19T13:01:23.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.462+0000 7f6cebc40640 1 --2- 172.21.15.57:0/453878360 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f6cc806aed0 0x7f6cc806d390 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:23.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.462+0000 7f6cebc40640 1 --2- 172.21.15.57:0/453878360 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6ce415b3e0 0x7f6ce40c7dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:23.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.462+0000 7f6cebc40640 1 --2- 172.21.15.57:0/453878360 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6ce415b000 0x7f6ce415dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:23.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.462+0000 7f6cebc40640 1 --2- 172.21.15.57:0/453878360 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6ce40afdc0 0x7f6ce40d1340 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:23.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.462+0000 7f6cebc40640 1 -- 172.21.15.57:0/453878360 >> 172.21.15.57:0/453878360 conn(0x7f6ce40d0cf0 msgr2=0x7f6ce41e5a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:23.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.462+0000 7f6cebc40640 1 -- 172.21.15.57:0/453878360 shutdown_connections 2023-10-19T13:01:23.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.462+0000 7f6cebc40640 1 -- 172.21.15.57:0/453878360 wait complete. 2023-10-19T13:01:23.476 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-19T12:44:07.371186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-19T13:01:23.476 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-19T13:01:23.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.587+0000 7f0523d7d640 1 -- 172.21.15.57:0/3212429784 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f051c06af00 msgr2=0x7f051c06b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:23.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.587+0000 7f0523d7d640 1 --2- 172.21.15.57:0/3212429784 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f051c06af00 0x7f051c06b2e0 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7f0518009900 tx=0x7f051802be10 comp rx=0 tx=0).stop 2023-10-19T13:01:23.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.587+0000 7f0523d7d640 1 -- 172.21.15.57:0/3212429784 shutdown_connections 2023-10-19T13:01:23.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.587+0000 7f0523d7d640 1 --2- 172.21.15.57:0/3212429784 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f051c06af00 0x7f051c06b2e0 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:23.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.587+0000 7f0523d7d640 1 -- 172.21.15.57:0/3212429784 >> 172.21.15.57:0/3212429784 conn(0x7f051c069930 msgr2=0x7f051c069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:23.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.588+0000 7f0523d7d640 1 -- 172.21.15.57:0/3212429784 shutdown_connections 2023-10-19T13:01:23.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.588+0000 7f0523d7d640 1 -- 172.21.15.57:0/3212429784 wait complete. 2023-10-19T13:01:23.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.589+0000 7f0523d7d640 1 Processor -- start 2023-10-19T13:01:23.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.589+0000 7f0523d7d640 1 -- start start 2023-10-19T13:01:23.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.589+0000 7f0523d7d640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f051c06af00 0x7f051c151c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:23.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.589+0000 7f0523d7d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f051c0abc10 0x7f051c15aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:23.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.589+0000 7f0523d7d640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f051c0c96b0 0x7f051c15aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:23.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.589+0000 7f0523d7d640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f051c0a6ee0 con 0x7f051c0abc10 2023-10-19T13:01:23.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.589+0000 7f0523d7d640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f051c0a7180 con 0x7f051c06af00 2023-10-19T13:01:23.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.589+0000 7f0523d7d640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f051c0a7460 con 0x7f051c0c96b0 2023-10-19T13:01:23.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.590+0000 7f0521318640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f051c0abc10 0x7f051c15aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:23.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.590+0000 7f0521318640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f051c0abc10 0x7f051c15aab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:58630/0 (socket says 172.21.15.57:58630) 2023-10-19T13:01:23.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.590+0000 7f0521318640 1 -- 172.21.15.57:0/1458149431 learned_addr learned my addr 172.21.15.57:0/1458149431 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:01:23.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.590+0000 7f0521b19640 1 --2- 172.21.15.57:0/1458149431 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f051c06af00 0x7f051c151c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:23.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.590+0000 7f052231a640 1 --2- 172.21.15.57:0/1458149431 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f051c0c96b0 0x7f051c15aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:23.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.590+0000 7f0521318640 1 -- 172.21.15.57:0/1458149431 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f051c0c96b0 msgr2=0x7f051c15aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:23.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.590+0000 7f0521318640 1 --2- 172.21.15.57:0/1458149431 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f051c0c96b0 0x7f051c15aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:23.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.590+0000 7f0521318640 1 -- 172.21.15.57:0/1458149431 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f051c06af00 msgr2=0x7f051c151c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:23.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.590+0000 7f0521318640 1 --2- 172.21.15.57:0/1458149431 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f051c06af00 0x7f051c151c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:23.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.590+0000 7f0521318640 1 -- 172.21.15.57:0/1458149431 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0518009560 con 0x7f051c0abc10 2023-10-19T13:01:23.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.590+0000 7f0521318640 1 --2- 172.21.15.57:0/1458149431 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f051c0abc10 0x7f051c15aab0 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7f050c00c3c0 tx=0x7f050c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:23.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.590+0000 7f05127fc640 1 -- 172.21.15.57:0/1458149431 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f050c0035f0 con 0x7f051c0abc10 2023-10-19T13:01:23.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.590+0000 7f05127fc640 1 -- 172.21.15.57:0/1458149431 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f050c003770 con 0x7f051c0abc10 2023-10-19T13:01:23.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.590+0000 7f0523d7d640 1 -- 172.21.15.57:0/1458149431 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f051c0a7740 con 0x7f051c0abc10 2023-10-19T13:01:23.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.590+0000 7f05127fc640 1 -- 172.21.15.57:0/1458149431 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f050c003b80 con 0x7f051c0abc10 2023-10-19T13:01:23.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.590+0000 7f0523d7d640 1 -- 172.21.15.57:0/1458149431 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f051c1521d0 con 0x7f051c0abc10 2023-10-19T13:01:23.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.591+0000 7f0523d7d640 1 -- 172.21.15.57:0/1458149431 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f051c0a7c60 con 0x7f051c0abc10 2023-10-19T13:01:23.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.592+0000 7f05127fc640 1 -- 172.21.15.57:0/1458149431 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 47) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f050c003ce0 con 0x7f051c0abc10 2023-10-19T13:01:23.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.592+0000 7f05127fc640 1 --2- 172.21.15.57:0/1458149431 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f050006af00 0x7f050006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:23.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.592+0000 7f05127fc640 1 -- 172.21.15.57:0/1458149431 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f050c088aa0 con 0x7f051c0abc10 2023-10-19T13:01:23.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.592+0000 7f0521b19640 1 --2- 172.21.15.57:0/1458149431 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f050006af00 0x7f050006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:23.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.593+0000 7f0521b19640 1 --2- 172.21.15.57:0/1458149431 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f050006af00 0x7f050006d3c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0518035000 tx=0x7f0518004730 comp rx=0 tx=0).ready entity=mgr.7072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:23.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.596+0000 7f05127fc640 1 -- 172.21.15.57:0/1458149431 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f051c0a7c60 con 0x7f051c0abc10 2023-10-19T13:01:23.687 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:23.684+0000 7fb508163640 -1 client.7141: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.7030:f55302c0-37aa-4235-8ccc-9775bc542785:v1:172.21.15.179:0/3304791412] 2023-10-19T13:01:23.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.779+0000 7f0523d7d640 1 -- 172.21.15.57:0/1458149431 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f051c0a7c60 con 0x7f051c0abc10 2023-10-19T13:01:23.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.779+0000 7f05127fc640 1 -- 172.21.15.57:0/1458149431 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v109) v1 ==== 93+0+10 (secure 0 0 0) 0x7f051c0a7c60 con 0x7f051c0abc10 2023-10-19T13:01:23.782 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-19T13:01:23.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.781+0000 7f0523d7d640 1 -- 172.21.15.57:0/1458149431 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f050006af00 msgr2=0x7f050006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:23.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.781+0000 7f0523d7d640 1 --2- 172.21.15.57:0/1458149431 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f050006af00 0x7f050006d3c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0518035000 tx=0x7f0518004730 comp rx=0 tx=0).stop 2023-10-19T13:01:23.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.781+0000 7f0523d7d640 1 -- 172.21.15.57:0/1458149431 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f051c0abc10 msgr2=0x7f051c15aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:23.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.781+0000 7f0523d7d640 1 --2- 172.21.15.57:0/1458149431 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f051c0abc10 0x7f051c15aab0 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7f050c00c3c0 tx=0x7f050c00c880 comp rx=0 tx=0).stop 2023-10-19T13:01:23.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.781+0000 7f0523d7d640 1 -- 172.21.15.57:0/1458149431 shutdown_connections 2023-10-19T13:01:23.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.781+0000 7f0523d7d640 1 --2- 172.21.15.57:0/1458149431 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f050006af00 0x7f050006d3c0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:23.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.781+0000 7f0523d7d640 1 --2- 172.21.15.57:0/1458149431 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f051c0c96b0 0x7f051c15aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:23.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.781+0000 7f0523d7d640 1 --2- 172.21.15.57:0/1458149431 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f051c0abc10 0x7f051c15aab0 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:23.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.781+0000 7f0523d7d640 1 --2- 172.21.15.57:0/1458149431 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f051c06af00 0x7f051c151c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:23.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.781+0000 7f0523d7d640 1 -- 172.21.15.57:0/1458149431 >> 172.21.15.57:0/1458149431 conn(0x7f051c069930 msgr2=0x7f051c152820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:23.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.781+0000 7f0523d7d640 1 -- 172.21.15.57:0/1458149431 shutdown_connections 2023-10-19T13:01:23.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.781+0000 7f0523d7d640 1 -- 172.21.15.57:0/1458149431 wait complete. 2023-10-19T13:01:23.794 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-19T13:01:23.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.901+0000 7f4469032640 1 -- 172.21.15.57:0/681407832 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f44640ceb40 msgr2=0x7f44640b0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:23.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.901+0000 7f4469032640 1 --2- 172.21.15.57:0/681407832 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f44640ceb40 0x7f44640b0650 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f446408a120 tx=0x7f445c02c260 comp rx=0 tx=0).stop 2023-10-19T13:01:23.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.901+0000 7f4469032640 1 -- 172.21.15.57:0/681407832 shutdown_connections 2023-10-19T13:01:23.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.901+0000 7f4469032640 1 --2- 172.21.15.57:0/681407832 >> v2:172.21.15.179:3300/0 conn(0x7f446415b2a0 0x7f44640d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:23.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.901+0000 7f4469032640 1 --2- 172.21.15.57:0/681407832 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f44640ceb40 0x7f44640b0650 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:23.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.901+0000 7f4469032640 1 -- 172.21.15.57:0/681407832 >> 172.21.15.57:0/681407832 conn(0x7f44640d0d60 msgr2=0x7f44640d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:23.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.901+0000 7f4469032640 1 -- 172.21.15.57:0/681407832 shutdown_connections 2023-10-19T13:01:23.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.901+0000 7f4469032640 1 -- 172.21.15.57:0/681407832 wait complete. 2023-10-19T13:01:23.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.902+0000 7f4469032640 1 Processor -- start 2023-10-19T13:01:23.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.902+0000 7f4469032640 1 -- start start 2023-10-19T13:01:23.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.903+0000 7f4469032640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f44640ceb40 0x7f4464158140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:23.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.903+0000 7f4469032640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f44640cf340 0x7f446409e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:23.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.903+0000 7f4469032640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f446415b2a0 0x7f44640b5110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:23.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.903+0000 7f4469032640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4464158680 con 0x7f44640cf340 2023-10-19T13:01:23.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.903+0000 7f4469032640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44640b4390 con 0x7f446415b2a0 2023-10-19T13:01:23.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.903+0000 7f4469032640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44640b4630 con 0x7f44640ceb40 2023-10-19T13:01:23.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.903+0000 7f445a59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f44640cf340 0x7f446409e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:23.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.903+0000 7f445a59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f44640cf340 0x7f446409e4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:58640/0 (socket says 172.21.15.57:58640) 2023-10-19T13:01:23.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.903+0000 7f445a59c640 1 -- 172.21.15.57:0/459877091 learned_addr learned my addr 172.21.15.57:0/459877091 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:01:23.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.903+0000 7f446359e640 1 --2- 172.21.15.57:0/459877091 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f446415b2a0 0x7f44640b5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:23.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.903+0000 7f4462d9d640 1 --2- 172.21.15.57:0/459877091 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f44640ceb40 0x7f4464158140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:23.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.904+0000 7f445a59c640 1 -- 172.21.15.57:0/459877091 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f44640ceb40 msgr2=0x7f4464158140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:23.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.904+0000 7f445a59c640 1 --2- 172.21.15.57:0/459877091 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f44640ceb40 0x7f4464158140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:23.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.904+0000 7f445a59c640 1 -- 172.21.15.57:0/459877091 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f446415b2a0 msgr2=0x7f44640b5110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:23.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.904+0000 7f445a59c640 1 --2- 172.21.15.57:0/459877091 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f446415b2a0 0x7f44640b5110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:23.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.904+0000 7f445a59c640 1 -- 172.21.15.57:0/459877091 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f445c009900 con 0x7f44640cf340 2023-10-19T13:01:23.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.904+0000 7f445a59c640 1 --2- 172.21.15.57:0/459877091 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f44640cf340 0x7f446409e4c0 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f444c00e790 tx=0x7f444c00ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:23.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.904+0000 7f445bfff640 1 -- 172.21.15.57:0/459877091 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f444c009580 con 0x7f44640cf340 2023-10-19T13:01:23.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.904+0000 7f445bfff640 1 -- 172.21.15.57:0/459877091 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f444c00cdb0 con 0x7f44640cf340 2023-10-19T13:01:23.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.904+0000 7f4469032640 1 -- 172.21.15.57:0/459877091 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44640b4910 con 0x7f44640cf340 2023-10-19T13:01:23.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.904+0000 7f445bfff640 1 -- 172.21.15.57:0/459877091 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f444c010690 con 0x7f44640cf340 2023-10-19T13:01:23.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.904+0000 7f4469032640 1 -- 172.21.15.57:0/459877091 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f446409ea00 con 0x7f44640cf340 2023-10-19T13:01:23.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.905+0000 7f4469032640 1 -- 172.21.15.57:0/459877091 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44640690e0 con 0x7f44640cf340 2023-10-19T13:01:23.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.906+0000 7f445bfff640 1 -- 172.21.15.57:0/459877091 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 47) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f444c0107f0 con 0x7f44640cf340 2023-10-19T13:01:23.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.906+0000 7f445bfff640 1 --2- 172.21.15.57:0/459877091 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f444406aed0 0x7f444406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:23.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.906+0000 7f445bfff640 1 -- 172.21.15.57:0/459877091 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f444c014070 con 0x7f44640cf340 2023-10-19T13:01:23.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.906+0000 7f4462d9d640 1 --2- 172.21.15.57:0/459877091 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f444406aed0 0x7f444406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:23.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.907+0000 7f4462d9d640 1 --2- 172.21.15.57:0/459877091 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f444406aed0 0x7f444406d390 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4454000f60 tx=0x7f4454009040 comp rx=0 tx=0).ready entity=mgr.7072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:23.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:23.910+0000 7f445bfff640 1 -- 172.21.15.57:0/459877091 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f444c051ca0 con 0x7f44640cf340 2023-10-19T13:01:24.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.093+0000 7f4469032640 1 -- 172.21.15.57:0/459877091 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f44640b0d00 con 0x7f44640cf340 2023-10-19T13:01:24.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.094+0000 7f445bfff640 1 -- 172.21.15.57:0/459877091 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v109) v1 ==== 84+0+65 (secure 0 0 0) 0x7f444c055170 con 0x7f44640cf340 2023-10-19T13:01:24.097 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:01:24.097 INFO:teuthology.orchestra.run.smithi057.stdout:[ 2023-10-19T13:01:24.098 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:01:24.098 INFO:teuthology.orchestra.run.smithi057.stdout: "poolnum": 1, 2023-10-19T13:01:24.098 INFO:teuthology.orchestra.run.smithi057.stdout: "poolname": ".mgr" 2023-10-19T13:01:24.099 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:01:24.099 INFO:teuthology.orchestra.run.smithi057.stdout:] 2023-10-19T13:01:24.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.095+0000 7f4469032640 1 -- 172.21.15.57:0/459877091 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f444406aed0 msgr2=0x7f444406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:24.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.095+0000 7f4469032640 1 --2- 172.21.15.57:0/459877091 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f444406aed0 0x7f444406d390 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4454000f60 tx=0x7f4454009040 comp rx=0 tx=0).stop 2023-10-19T13:01:24.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.095+0000 7f4469032640 1 -- 172.21.15.57:0/459877091 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f44640cf340 msgr2=0x7f446409e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:24.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.095+0000 7f4469032640 1 --2- 172.21.15.57:0/459877091 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f44640cf340 0x7f446409e4c0 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f444c00e790 tx=0x7f444c00ec50 comp rx=0 tx=0).stop 2023-10-19T13:01:24.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.095+0000 7f4469032640 1 -- 172.21.15.57:0/459877091 shutdown_connections 2023-10-19T13:01:24.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.095+0000 7f4469032640 1 --2- 172.21.15.57:0/459877091 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f444406aed0 0x7f444406d390 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:24.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.095+0000 7f4469032640 1 --2- 172.21.15.57:0/459877091 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f446415b2a0 0x7f44640b5110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:24.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.095+0000 7f4469032640 1 --2- 172.21.15.57:0/459877091 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f44640cf340 0x7f446409e4c0 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:24.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.095+0000 7f4469032640 1 --2- 172.21.15.57:0/459877091 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f44640ceb40 0x7f4464158140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:24.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.095+0000 7f4469032640 1 -- 172.21.15.57:0/459877091 >> 172.21.15.57:0/459877091 conn(0x7f44640d0d60 msgr2=0x7f446406b550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:24.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.096+0000 7f4469032640 1 -- 172.21.15.57:0/459877091 shutdown_connections 2023-10-19T13:01:24.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.096+0000 7f4469032640 1 -- 172.21.15.57:0/459877091 wait complete. 2023-10-19T13:01:24.109 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-19T13:01:24.109 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-19T13:01:24.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.223+0000 7f6815056640 1 -- 172.21.15.57:0/2010324154 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f681006f0a0 msgr2=0x7f681006ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:24.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.223+0000 7f6815056640 1 --2- 172.21.15.57:0/2010324154 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f681006f0a0 0x7f681006ddc0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f67f8009860 tx=0x7f67f802bde0 comp rx=0 tx=0).stop 2023-10-19T13:01:24.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.223+0000 7f6815056640 1 -- 172.21.15.57:0/2010324154 shutdown_connections 2023-10-19T13:01:24.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.223+0000 7f6815056640 1 --2- 172.21.15.57:0/2010324154 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f681006f0a0 0x7f681006ddc0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:24.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.223+0000 7f6815056640 1 -- 172.21.15.57:0/2010324154 >> 172.21.15.57:0/2010324154 conn(0x7f68100b0670 msgr2=0x7f68100b0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:24.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.223+0000 7f6815056640 1 -- 172.21.15.57:0/2010324154 shutdown_connections 2023-10-19T13:01:24.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.223+0000 7f6815056640 1 -- 172.21.15.57:0/2010324154 wait complete. 2023-10-19T13:01:24.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.224+0000 7f6815056640 1 Processor -- start 2023-10-19T13:01:24.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.224+0000 7f6815056640 1 -- start start 2023-10-19T13:01:24.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.225+0000 7f6815056640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f681006d2d0 0x7f681014aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:24.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.225+0000 7f6815056640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f681006f0a0 0x7f68101e1790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:24.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.225+0000 7f6815056640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f68100ad370 0x7f68101e1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:24.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.225+0000 7f6815056640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68101e2b80 con 0x7f681006f0a0 2023-10-19T13:01:24.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.225+0000 7f6815056640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68100b2030 con 0x7f68100ad370 2023-10-19T13:01:24.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.225+0000 7f6815056640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68100b1d20 con 0x7f681006d2d0 2023-10-19T13:01:24.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.225+0000 7f6807fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f681006f0a0 0x7f68101e1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:24.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.225+0000 7f6807fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f681006f0a0 0x7f68101e1790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:58646/0 (socket says 172.21.15.57:58646) 2023-10-19T13:01:24.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.225+0000 7f6807fff640 1 -- 172.21.15.57:0/1644001304 learned_addr learned my addr 172.21.15.57:0/1644001304 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:01:24.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.226+0000 7f680ed9d640 1 --2- 172.21.15.57:0/1644001304 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f681006d2d0 0x7f681014aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:24.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.226+0000 7f680f59e640 1 --2- 172.21.15.57:0/1644001304 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f68100ad370 0x7f68101e1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:24.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.226+0000 7f6807fff640 1 -- 172.21.15.57:0/1644001304 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f681006d2d0 msgr2=0x7f681014aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:24.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.226+0000 7f6807fff640 1 --2- 172.21.15.57:0/1644001304 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f681006d2d0 0x7f681014aeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:24.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.226+0000 7f6807fff640 1 -- 172.21.15.57:0/1644001304 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f68100ad370 msgr2=0x7f68101e1cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:24.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.226+0000 7f6807fff640 1 --2- 172.21.15.57:0/1644001304 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f68100ad370 0x7f68101e1cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:24.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.226+0000 7f6807fff640 1 -- 172.21.15.57:0/1644001304 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67f8009490 con 0x7f681006f0a0 2023-10-19T13:01:24.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.226+0000 7f680f59e640 1 --2- 172.21.15.57:0/1644001304 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f68100ad370 0x7f68101e1cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T13:01:24.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.226+0000 7f6807fff640 1 --2- 172.21.15.57:0/1644001304 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f681006f0a0 0x7f68101e1790 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f67f8009980 tx=0x7f67f80038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:24.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.226+0000 7f68077fe640 1 -- 172.21.15.57:0/1644001304 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67f8003ea0 con 0x7f681006f0a0 2023-10-19T13:01:24.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.226+0000 7f68077fe640 1 -- 172.21.15.57:0/1644001304 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f67f8005020 con 0x7f681006f0a0 2023-10-19T13:01:24.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.226+0000 7f6815056640 1 -- 172.21.15.57:0/1644001304 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68100cb580 con 0x7f681006f0a0 2023-10-19T13:01:24.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.226+0000 7f68077fe640 1 -- 172.21.15.57:0/1644001304 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67f8005430 con 0x7f681006f0a0 2023-10-19T13:01:24.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.226+0000 7f6815056640 1 -- 172.21.15.57:0/1644001304 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68101e25c0 con 0x7f681006f0a0 2023-10-19T13:01:24.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.227+0000 7f6815056640 1 -- 172.21.15.57:0/1644001304 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68100ce1d0 con 0x7f681006f0a0 2023-10-19T13:01:24.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.227+0000 7f68077fe640 1 -- 172.21.15.57:0/1644001304 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 47) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f67f802dd50 con 0x7f681006f0a0 2023-10-19T13:01:24.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.228+0000 7f68077fe640 1 --2- 172.21.15.57:0/1644001304 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f67f006aed0 0x7f67f006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:24.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.228+0000 7f68077fe640 1 -- 172.21.15.57:0/1644001304 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f67f80ab520 con 0x7f681006f0a0 2023-10-19T13:01:24.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.228+0000 7f680ed9d640 1 --2- 172.21.15.57:0/1644001304 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f67f006aed0 0x7f67f006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:24.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.228+0000 7f680ed9d640 1 --2- 172.21.15.57:0/1644001304 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f67f006aed0 0x7f67f006d390 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f6800007fd0 tx=0x7f6800009040 comp rx=0 tx=0).ready entity=mgr.7072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:24.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.232+0000 7f68077fe640 1 -- 172.21.15.57:0/1644001304 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f67f80767d0 con 0x7f681006f0a0 2023-10-19T13:01:24.391 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:24.388+0000 7fb508163640 -1 client.7141: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.7030:f55302c0-37aa-4235-8ccc-9775bc542785:v1:172.21.15.179:0/3304791412] 2023-10-19T13:01:24.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.410+0000 7f6815056640 1 -- 172.21.15.57:0/1644001304 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f68100690e0 con 0x7f681006f0a0 2023-10-19T13:01:24.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.746+0000 7f68077fe640 1 -- 172.21.15.57:0/1644001304 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v110) v1 ==== 138+0+0 (secure 0 0 0) 0x7f67f8079df0 con 0x7f681006f0a0 2023-10-19T13:01:24.749 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'cephfs_metadata' created 2023-10-19T13:01:24.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.748+0000 7f6815056640 1 -- 172.21.15.57:0/1644001304 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f67f006aed0 msgr2=0x7f67f006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:24.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.748+0000 7f6815056640 1 --2- 172.21.15.57:0/1644001304 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f67f006aed0 0x7f67f006d390 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f6800007fd0 tx=0x7f6800009040 comp rx=0 tx=0).stop 2023-10-19T13:01:24.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.748+0000 7f6815056640 1 -- 172.21.15.57:0/1644001304 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f681006f0a0 msgr2=0x7f68101e1790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:24.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.748+0000 7f6815056640 1 --2- 172.21.15.57:0/1644001304 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f681006f0a0 0x7f68101e1790 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f67f8009980 tx=0x7f67f80038a0 comp rx=0 tx=0).stop 2023-10-19T13:01:24.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.748+0000 7f6815056640 1 -- 172.21.15.57:0/1644001304 shutdown_connections 2023-10-19T13:01:24.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.748+0000 7f6815056640 1 --2- 172.21.15.57:0/1644001304 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f67f006aed0 0x7f67f006d390 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:24.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.748+0000 7f6815056640 1 --2- 172.21.15.57:0/1644001304 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f68100ad370 0x7f68101e1cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:24.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.748+0000 7f6815056640 1 --2- 172.21.15.57:0/1644001304 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f681006f0a0 0x7f68101e1790 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:24.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.748+0000 7f6815056640 1 --2- 172.21.15.57:0/1644001304 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f681006d2d0 0x7f681014aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:24.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.748+0000 7f6815056640 1 -- 172.21.15.57:0/1644001304 >> 172.21.15.57:0/1644001304 conn(0x7f68100b0670 msgr2=0x7f681006d6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:24.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.748+0000 7f6815056640 1 -- 172.21.15.57:0/1644001304 shutdown_connections 2023-10-19T13:01:24.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.748+0000 7f6815056640 1 -- 172.21.15.57:0/1644001304 wait complete. 2023-10-19T13:01:24.764 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-19T13:01:24.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.880+0000 7ff839e07640 1 -- 172.21.15.57:0/2276191629 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff83415c1f0 msgr2=0x7ff8340b0e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:24.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.880+0000 7ff839e07640 1 --2- 172.21.15.57:0/2276191629 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff83415c1f0 0x7ff8340b0e90 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7ff82c005fd0 tx=0x7ff82c02c370 comp rx=0 tx=0).stop 2023-10-19T13:01:24.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.880+0000 7ff839e07640 1 -- 172.21.15.57:0/2276191629 shutdown_connections 2023-10-19T13:01:24.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.880+0000 7ff839e07640 1 --2- 172.21.15.57:0/2276191629 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff83415c1f0 0x7ff8340b0e90 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:24.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.880+0000 7ff839e07640 1 -- 172.21.15.57:0/2276191629 >> 172.21.15.57:0/2276191629 conn(0x7ff83406fae0 msgr2=0x7ff83406fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:24.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.880+0000 7ff839e07640 1 -- 172.21.15.57:0/2276191629 shutdown_connections 2023-10-19T13:01:24.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.880+0000 7ff839e07640 1 -- 172.21.15.57:0/2276191629 wait complete. 2023-10-19T13:01:24.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.881+0000 7ff839e07640 1 Processor -- start 2023-10-19T13:01:24.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.881+0000 7ff839e07640 1 -- start start 2023-10-19T13:01:24.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.882+0000 7ff839e07640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff83406ae40 0x7ff8340ae6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:24.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.882+0000 7ff839e07640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff8341579f0 0x7ff8340abdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:24.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.882+0000 7ff839e07640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff83415c1f0 0x7ff834141fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:24.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.882+0000 7ff839e07640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8341415a0 con 0x7ff8341579f0 2023-10-19T13:01:24.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.882+0000 7ff839e07640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8340ac300 con 0x7ff83406ae40 2023-10-19T13:01:24.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.882+0000 7ff839e07640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8340acc20 con 0x7ff83415c1f0 2023-10-19T13:01:24.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.882+0000 7ff832ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff8341579f0 0x7ff8340abdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:24.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.882+0000 7ff832ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff8341579f0 0x7ff8340abdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:58668/0 (socket says 172.21.15.57:58668) 2023-10-19T13:01:24.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.882+0000 7ff832ffd640 1 -- 172.21.15.57:0/2172236936 learned_addr learned my addr 172.21.15.57:0/2172236936 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:01:24.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.882+0000 7ff8337fe640 1 --2- 172.21.15.57:0/2172236936 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff83406ae40 0x7ff8340ae6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:24.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.882+0000 7ff833fff640 1 --2- 172.21.15.57:0/2172236936 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff83415c1f0 0x7ff834141fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:24.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.882+0000 7ff832ffd640 1 -- 172.21.15.57:0/2172236936 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff83415c1f0 msgr2=0x7ff834141fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:24.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.882+0000 7ff832ffd640 1 --2- 172.21.15.57:0/2172236936 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff83415c1f0 0x7ff834141fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:24.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.882+0000 7ff832ffd640 1 -- 172.21.15.57:0/2172236936 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff83406ae40 msgr2=0x7ff8340ae6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:24.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.882+0000 7ff832ffd640 1 --2- 172.21.15.57:0/2172236936 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff83406ae40 0x7ff8340ae6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:24.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.882+0000 7ff832ffd640 1 -- 172.21.15.57:0/2172236936 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff82c009900 con 0x7ff8341579f0 2023-10-19T13:01:24.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.882+0000 7ff8337fe640 1 --2- 172.21.15.57:0/2172236936 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff83406ae40 0x7ff8340ae6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:01:24.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.882+0000 7ff833fff640 1 --2- 172.21.15.57:0/2172236936 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff83415c1f0 0x7ff834141fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:01:24.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.883+0000 7ff832ffd640 1 --2- 172.21.15.57:0/2172236936 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff8341579f0 0x7ff8340abdc0 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7ff82800c400 tx=0x7ff82800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:24.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.883+0000 7ff81bfff640 1 -- 172.21.15.57:0/2172236936 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8280052d0 con 0x7ff8341579f0 2023-10-19T13:01:24.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.883+0000 7ff839e07640 1 -- 172.21.15.57:0/2172236936 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8340acf00 con 0x7ff8341579f0 2023-10-19T13:01:24.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.883+0000 7ff81bfff640 1 -- 172.21.15.57:0/2172236936 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7ff828007020 con 0x7ff8341579f0 2023-10-19T13:01:24.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.883+0000 7ff81bfff640 1 -- 172.21.15.57:0/2172236936 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff828007470 con 0x7ff8341579f0 2023-10-19T13:01:24.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.883+0000 7ff839e07640 1 -- 172.21.15.57:0/2172236936 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff834141ac0 con 0x7ff8341579f0 2023-10-19T13:01:24.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.884+0000 7ff839e07640 1 -- 172.21.15.57:0/2172236936 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8340690e0 con 0x7ff8341579f0 2023-10-19T13:01:24.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.884+0000 7ff81bfff640 1 -- 172.21.15.57:0/2172236936 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 47) v1 ==== 89275+0+0 (secure 0 0 0) 0x7ff8280067d0 con 0x7ff8341579f0 2023-10-19T13:01:24.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.885+0000 7ff81bfff640 1 --2- 172.21.15.57:0/2172236936 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7ff82006aed0 0x7ff82006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:24.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.885+0000 7ff81bfff640 1 -- 172.21.15.57:0/2172236936 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5575+0+0 (secure 0 0 0) 0x7ff828088c30 con 0x7ff8341579f0 2023-10-19T13:01:24.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.885+0000 7ff8337fe640 1 --2- 172.21.15.57:0/2172236936 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7ff82006aed0 0x7ff82006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:24.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.885+0000 7ff8337fe640 1 --2- 172.21.15.57:0/2172236936 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7ff82006aed0 0x7ff82006d390 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff824007fd0 tx=0x7ff824009040 comp rx=0 tx=0).ready entity=mgr.7072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:24.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:24.889+0000 7ff81bfff640 1 -- 172.21.15.57:0/2172236936 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff828053d80 con 0x7ff8341579f0 2023-10-19T13:01:25.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:25.069+0000 7ff839e07640 1 -- 172.21.15.57:0/2172236936 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7ff8340b0e60 con 0x7ff8341579f0 2023-10-19T13:01:25.101 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:25.099+0000 7fb508163640 -1 client.7141: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.7030:f55302c0-37aa-4235-8ccc-9775bc542785:v1:172.21.15.179:0/3304791412] 2023-10-19T13:01:25.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:25.752+0000 7ff81bfff640 1 -- 172.21.15.57:0/2172236936 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v111) v1 ==== 170+0+0 (secure 0 0 0) 0x7ff8280573a0 con 0x7ff8341579f0 2023-10-19T13:01:25.755 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'cephfs_data' created 2023-10-19T13:01:25.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:25.755+0000 7ff839e07640 1 -- 172.21.15.57:0/2172236936 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7ff82006aed0 msgr2=0x7ff82006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:25.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:25.755+0000 7ff839e07640 1 --2- 172.21.15.57:0/2172236936 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7ff82006aed0 0x7ff82006d390 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff824007fd0 tx=0x7ff824009040 comp rx=0 tx=0).stop 2023-10-19T13:01:25.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:25.755+0000 7ff839e07640 1 -- 172.21.15.57:0/2172236936 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff8341579f0 msgr2=0x7ff8340abdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:25.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:25.755+0000 7ff839e07640 1 --2- 172.21.15.57:0/2172236936 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff8341579f0 0x7ff8340abdc0 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7ff82800c400 tx=0x7ff82800c8c0 comp rx=0 tx=0).stop 2023-10-19T13:01:25.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:25.755+0000 7ff839e07640 1 -- 172.21.15.57:0/2172236936 shutdown_connections 2023-10-19T13:01:25.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:25.755+0000 7ff839e07640 1 --2- 172.21.15.57:0/2172236936 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7ff82006aed0 0x7ff82006d390 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:25.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:25.755+0000 7ff839e07640 1 --2- 172.21.15.57:0/2172236936 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff83415c1f0 0x7ff834141fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:25.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:25.755+0000 7ff839e07640 1 --2- 172.21.15.57:0/2172236936 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff8341579f0 0x7ff8340abdc0 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:25.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:25.755+0000 7ff839e07640 1 --2- 172.21.15.57:0/2172236936 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff83406ae40 0x7ff8340ae6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:25.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:25.755+0000 7ff839e07640 1 -- 172.21.15.57:0/2172236936 >> 172.21.15.57:0/2172236936 conn(0x7ff83406fae0 msgr2=0x7ff834153480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:25.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:25.755+0000 7ff839e07640 1 -- 172.21.15.57:0/2172236936 shutdown_connections 2023-10-19T13:01:25.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:25.755+0000 7ff839e07640 1 -- 172.21.15.57:0/2172236936 wait complete. 2023-10-19T13:01:25.778 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-19T13:01:25.831 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:25.828+0000 7fb508163640 -1 client.7141: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.7030:f55302c0-37aa-4235-8ccc-9775bc542785:v1:172.21.15.179:0/3304791412] 2023-10-19T13:01:25.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:25.889+0000 7f02fc856640 1 -- 172.21.15.57:0/2011242800 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f02f815b180 msgr2=0x7f02f80cf350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:25.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:25.889+0000 7f02fc856640 1 --2- 172.21.15.57:0/2011242800 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f02f815b180 0x7f02f80cf350 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f02e8009be0 tx=0x7f02e802c160 comp rx=0 tx=0).stop 2023-10-19T13:01:25.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:25.889+0000 7f02fc856640 1 -- 172.21.15.57:0/2011242800 shutdown_connections 2023-10-19T13:01:25.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:25.889+0000 7f02fc856640 1 --2- 172.21.15.57:0/2011242800 >> v2:172.21.15.179:3300/0 conn(0x7f02f80ce420 0x7f02f80cd6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:25.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:25.889+0000 7f02fc856640 1 --2- 172.21.15.57:0/2011242800 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f02f815b180 0x7f02f80cf350 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:25.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:25.889+0000 7f02fc856640 1 -- 172.21.15.57:0/2011242800 >> 172.21.15.57:0/2011242800 conn(0x7f02f8099ff0 msgr2=0x7f02f80d0d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:25.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:25.889+0000 7f02fc856640 1 -- 172.21.15.57:0/2011242800 shutdown_connections 2023-10-19T13:01:25.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:25.889+0000 7f02fc856640 1 -- 172.21.15.57:0/2011242800 wait complete. 2023-10-19T13:01:25.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:25.890+0000 7f02fc856640 1 Processor -- start 2023-10-19T13:01:25.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:25.890+0000 7f02fc856640 1 -- start start 2023-10-19T13:01:25.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:25.891+0000 7f02fc856640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f02f80cdbf0 0x7f02f80b0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:25.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:25.891+0000 7f02fc856640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f02f80ce420 0x7f02f81f4250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:25.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:25.891+0000 7f02fc856640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f02f815b180 0x7f02f80b28d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:25.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:25.891+0000 7f02fc856640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02f80b2e10 con 0x7f02f80ce420 2023-10-19T13:01:25.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:25.891+0000 7f02fc856640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02f80b3040 con 0x7f02f815b180 2023-10-19T13:01:25.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:25.891+0000 7f02fc856640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02f80b3a30 con 0x7f02f80cdbf0 2023-10-19T13:01:25.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:25.891+0000 7f02f5d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f02f80ce420 0x7f02f81f4250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:25.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:25.891+0000 7f02f5d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f02f80ce420 0x7f02f81f4250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:58678/0 (socket says 172.21.15.57:58678) 2023-10-19T13:01:25.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:25.891+0000 7f02f5d9b640 1 -- 172.21.15.57:0/1325434473 learned_addr learned my addr 172.21.15.57:0/1325434473 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:01:25.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:25.891+0000 7f02f659c640 1 --2- 172.21.15.57:0/1325434473 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f02f80cdbf0 0x7f02f80b0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:25.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:25.891+0000 7f02f6d9d640 1 --2- 172.21.15.57:0/1325434473 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f02f815b180 0x7f02f80b28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:25.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:25.892+0000 7f02f5d9b640 1 -- 172.21.15.57:0/1325434473 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f02f80cdbf0 msgr2=0x7f02f80b0ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:25.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:25.892+0000 7f02f5d9b640 1 --2- 172.21.15.57:0/1325434473 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f02f80cdbf0 0x7f02f80b0ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:25.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:25.892+0000 7f02f5d9b640 1 -- 172.21.15.57:0/1325434473 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f02f815b180 msgr2=0x7f02f80b28d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:25.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:25.892+0000 7f02f5d9b640 1 --2- 172.21.15.57:0/1325434473 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f02f815b180 0x7f02f80b28d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:25.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:25.892+0000 7f02f5d9b640 1 -- 172.21.15.57:0/1325434473 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02e8009840 con 0x7f02f80ce420 2023-10-19T13:01:25.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:25.892+0000 7f02f5d9b640 1 --2- 172.21.15.57:0/1325434473 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f02f80ce420 0x7f02f81f4250 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f02ec00e9f0 tx=0x7f02ec00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:25.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:25.892+0000 7f02e6ffd640 1 -- 172.21.15.57:0/1325434473 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02ec009700 con 0x7f02f80ce420 2023-10-19T13:01:25.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:25.892+0000 7f02e6ffd640 1 -- 172.21.15.57:0/1325434473 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f02ec00ce70 con 0x7f02f80ce420 2023-10-19T13:01:25.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:25.892+0000 7f02fc856640 1 -- 172.21.15.57:0/1325434473 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02f80b3d10 con 0x7f02f80ce420 2023-10-19T13:01:25.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:25.892+0000 7f02e6ffd640 1 -- 172.21.15.57:0/1325434473 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02ec010670 con 0x7f02f80ce420 2023-10-19T13:01:25.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:25.892+0000 7f02fc856640 1 -- 172.21.15.57:0/1325434473 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02f80b42d0 con 0x7f02f80ce420 2023-10-19T13:01:25.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:25.893+0000 7f02fc856640 1 -- 172.21.15.57:0/1325434473 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02f8066960 con 0x7f02f80ce420 2023-10-19T13:01:25.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:25.894+0000 7f02e6ffd640 1 -- 172.21.15.57:0/1325434473 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 47) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f02ec0107d0 con 0x7f02f80ce420 2023-10-19T13:01:25.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:25.894+0000 7f02e6ffd640 1 --2- 172.21.15.57:0/1325434473 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f02d806aed0 0x7f02d806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:25.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:25.894+0000 7f02e6ffd640 1 -- 172.21.15.57:0/1325434473 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5957+0+0 (secure 0 0 0) 0x7f02ec014070 con 0x7f02f80ce420 2023-10-19T13:01:25.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:25.894+0000 7f02f659c640 1 --2- 172.21.15.57:0/1325434473 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f02d806aed0 0x7f02d806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:25.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:25.895+0000 7f02f659c640 1 --2- 172.21.15.57:0/1325434473 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f02d806aed0 0x7f02d806d390 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f02e8009be0 tx=0x7f02e802e040 comp rx=0 tx=0).ready entity=mgr.7072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:25.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:25.899+0000 7f02e6ffd640 1 -- 172.21.15.57:0/1325434473 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f02ec051ca0 con 0x7f02f80ce420 2023-10-19T13:01:26.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:26.117+0000 7f02fc856640 1 -- 172.21.15.57:0/1325434473 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f02f80690e0 con 0x7f02f80ce420 2023-10-19T13:01:26.547 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:26.544+0000 7fb508163640 -1 client.7141: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.7030:f55302c0-37aa-4235-8ccc-9775bc542785:v1:172.21.15.179:0/3304791412] 2023-10-19T13:01:26.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:26.764+0000 7f02e6ffd640 1 -- 172.21.15.57:0/1325434473 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 13 and data pool 14 v42) v1 ==== 174+0+0 (secure 0 0 0) 0x7f02ec058e70 con 0x7f02f80ce420 2023-10-19T13:01:26.770 INFO:teuthology.orchestra.run.smithi057.stderr:new fs with metadata pool 13 and data pool 14 2023-10-19T13:01:26.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:26.768+0000 7f02fc856640 1 -- 172.21.15.57:0/1325434473 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f02d806aed0 msgr2=0x7f02d806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:26.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:26.768+0000 7f02fc856640 1 --2- 172.21.15.57:0/1325434473 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f02d806aed0 0x7f02d806d390 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f02e8009be0 tx=0x7f02e802e040 comp rx=0 tx=0).stop 2023-10-19T13:01:26.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:26.768+0000 7f02fc856640 1 -- 172.21.15.57:0/1325434473 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f02f80ce420 msgr2=0x7f02f81f4250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:26.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:26.769+0000 7f02fc856640 1 --2- 172.21.15.57:0/1325434473 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f02f80ce420 0x7f02f81f4250 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f02ec00e9f0 tx=0x7f02ec00eeb0 comp rx=0 tx=0).stop 2023-10-19T13:01:26.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:26.769+0000 7f02fc856640 1 -- 172.21.15.57:0/1325434473 shutdown_connections 2023-10-19T13:01:26.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:26.770+0000 7f02fc856640 1 --2- 172.21.15.57:0/1325434473 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f02d806aed0 0x7f02d806d390 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:26.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:26.770+0000 7f02fc856640 1 --2- 172.21.15.57:0/1325434473 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f02f815b180 0x7f02f80b28d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:26.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:26.770+0000 7f02fc856640 1 --2- 172.21.15.57:0/1325434473 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f02f80ce420 0x7f02f81f4250 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:26.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:26.770+0000 7f02fc856640 1 --2- 172.21.15.57:0/1325434473 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f02f80cdbf0 0x7f02f80b0ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:26.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:26.770+0000 7f02fc856640 1 -- 172.21.15.57:0/1325434473 >> 172.21.15.57:0/1325434473 conn(0x7f02f8099ff0 msgr2=0x7f02f8159dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:26.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:26.772+0000 7f02fc856640 1 -- 172.21.15.57:0/1325434473 shutdown_connections 2023-10-19T13:01:26.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:26.772+0000 7f02fc856640 1 -- 172.21.15.57:0/1325434473 wait complete. 2023-10-19T13:01:26.794 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T13:01:26.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:26.901+0000 7f4f721d0640 1 -- 172.21.15.57:0/635082263 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4f6c0ceb40 msgr2=0x7f4f6c0b0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:26.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:26.901+0000 7f4f721d0640 1 --2- 172.21.15.57:0/635082263 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4f6c0ceb40 0x7f4f6c0b0650 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f4f64005fd0 tx=0x7f4f6402c240 comp rx=0 tx=0).stop 2023-10-19T13:01:26.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:26.901+0000 7f4f721d0640 1 -- 172.21.15.57:0/635082263 shutdown_connections 2023-10-19T13:01:26.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:26.901+0000 7f4f721d0640 1 --2- 172.21.15.57:0/635082263 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4f6c0ceb40 0x7f4f6c0b0650 unknown :-1 s=CLOSED pgs=1274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:26.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:26.901+0000 7f4f721d0640 1 --2- 172.21.15.57:0/635082263 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4f6c15b2a0 0x7f4f6c0d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:26.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:26.901+0000 7f4f721d0640 1 -- 172.21.15.57:0/635082263 >> 172.21.15.57:0/635082263 conn(0x7f4f6c0d0d60 msgr2=0x7f4f6c0d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:26.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:26.901+0000 7f4f721d0640 1 -- 172.21.15.57:0/635082263 shutdown_connections 2023-10-19T13:01:26.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:26.901+0000 7f4f721d0640 1 -- 172.21.15.57:0/635082263 wait complete. 2023-10-19T13:01:26.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:26.903+0000 7f4f721d0640 1 Processor -- start 2023-10-19T13:01:26.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:26.903+0000 7f4f721d0640 1 -- start start 2023-10-19T13:01:26.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:26.903+0000 7f4f721d0640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4f6c0ceb40 0x7f4f6c0ca8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:26.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:26.903+0000 7f4f721d0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4f6c0cf340 0x7f4f6c06a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:26.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:26.903+0000 7f4f721d0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4f6c15b2a0 0x7f4f6c1f0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:26.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:26.903+0000 7f4f721d0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f6c0cae10 con 0x7f4f6c0cf340 2023-10-19T13:01:26.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:26.903+0000 7f4f721d0640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f6c0c96b0 con 0x7f4f6c15b2a0 2023-10-19T13:01:26.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:26.903+0000 7f4f721d0640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f6c0c89e0 con 0x7f4f6c0ceb40 2023-10-19T13:01:26.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:26.904+0000 7f4f6affd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4f6c0cf340 0x7f4f6c06a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:26.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:26.904+0000 7f4f6affd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4f6c0cf340 0x7f4f6c06a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40018/0 (socket says 172.21.15.57:40018) 2023-10-19T13:01:26.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:26.904+0000 7f4f6affd640 1 -- 172.21.15.57:0/1269968728 learned_addr learned my addr 172.21.15.57:0/1269968728 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:01:26.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:26.904+0000 7f4f6bfff640 1 --2- 172.21.15.57:0/1269968728 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4f6c15b2a0 0x7f4f6c1f0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:26.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:26.904+0000 7f4f6b7fe640 1 --2- 172.21.15.57:0/1269968728 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4f6c0ceb40 0x7f4f6c0ca8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:26.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:26.904+0000 7f4f6affd640 1 -- 172.21.15.57:0/1269968728 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4f6c0ceb40 msgr2=0x7f4f6c0ca8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:26.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:26.904+0000 7f4f6affd640 1 --2- 172.21.15.57:0/1269968728 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4f6c0ceb40 0x7f4f6c0ca8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:26.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:26.904+0000 7f4f6affd640 1 -- 172.21.15.57:0/1269968728 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4f6c15b2a0 msgr2=0x7f4f6c1f0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:26.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:26.904+0000 7f4f6affd640 1 --2- 172.21.15.57:0/1269968728 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4f6c15b2a0 0x7f4f6c1f0ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:26.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:26.904+0000 7f4f6affd640 1 -- 172.21.15.57:0/1269968728 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f64009900 con 0x7f4f6c0cf340 2023-10-19T13:01:26.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:26.904+0000 7f4f6affd640 1 --2- 172.21.15.57:0/1269968728 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4f6c0cf340 0x7f4f6c06a200 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f4f6000d6d0 tx=0x7f4f6000db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:26.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:26.904+0000 7f4f53fff640 1 -- 172.21.15.57:0/1269968728 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f60005220 con 0x7f4f6c0cf340 2023-10-19T13:01:26.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:26.904+0000 7f4f53fff640 1 -- 172.21.15.57:0/1269968728 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f4f60005490 con 0x7f4f6c0cf340 2023-10-19T13:01:26.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:26.904+0000 7f4f721d0640 1 -- 172.21.15.57:0/1269968728 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f6c0b0d50 con 0x7f4f6c0cf340 2023-10-19T13:01:26.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:26.904+0000 7f4f53fff640 1 -- 172.21.15.57:0/1269968728 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f60009a80 con 0x7f4f6c0cf340 2023-10-19T13:01:26.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:26.904+0000 7f4f721d0640 1 -- 172.21.15.57:0/1269968728 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f6c0c9c30 con 0x7f4f6c0cf340 2023-10-19T13:01:26.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:26.905+0000 7f4f721d0640 1 -- 172.21.15.57:0/1269968728 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f6c0690e0 con 0x7f4f6c0cf340 2023-10-19T13:01:26.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:26.906+0000 7f4f53fff640 1 -- 172.21.15.57:0/1269968728 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 47) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f4f60009be0 con 0x7f4f6c0cf340 2023-10-19T13:01:26.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:26.906+0000 7f4f53fff640 1 --2- 172.21.15.57:0/1269968728 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f4f5806af00 0x7f4f5806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:26.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:26.906+0000 7f4f53fff640 1 -- 172.21.15.57:0/1269968728 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f4f600898d0 con 0x7f4f6c0cf340 2023-10-19T13:01:26.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:26.906+0000 7f4f6b7fe640 1 --2- 172.21.15.57:0/1269968728 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f4f5806af00 0x7f4f5806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:26.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:26.907+0000 7f4f6b7fe640 1 --2- 172.21.15.57:0/1269968728 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f4f5806af00 0x7f4f5806d3c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4f6c155170 tx=0x7f4f5c009040 comp rx=0 tx=0).ready entity=mgr.7072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:26.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:26.910+0000 7f4f53fff640 1 -- 172.21.15.57:0/1269968728 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4f60054860 con 0x7f4f6c0cf340 2023-10-19T13:01:27.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.093+0000 7f4f721d0640 1 -- 172.21.15.57:0/1269968728 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4f6c0d0b40 con 0x7f4f6c0cf340 2023-10-19T13:01:27.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.094+0000 7f4f53fff640 1 -- 172.21.15.57:0/1269968728 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v112) v1 ==== 74+0+16796 (secure 0 0 0) 0x7f4f60017090 con 0x7f4f6c0cf340 2023-10-19T13:01:27.098 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:01:27.098 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":112,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T13:01:26.118644+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2023-10-19T13:01:24.443125+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2023-10-19T13:01:25.101276+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T13:01:27.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.096+0000 7f4f721d0640 1 -- 172.21.15.57:0/1269968728 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f4f5806af00 msgr2=0x7f4f5806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:27.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.096+0000 7f4f721d0640 1 --2- 172.21.15.57:0/1269968728 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f4f5806af00 0x7f4f5806d3c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4f6c155170 tx=0x7f4f5c009040 comp rx=0 tx=0).stop 2023-10-19T13:01:27.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.096+0000 7f4f721d0640 1 -- 172.21.15.57:0/1269968728 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4f6c0cf340 msgr2=0x7f4f6c06a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:27.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.096+0000 7f4f721d0640 1 --2- 172.21.15.57:0/1269968728 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4f6c0cf340 0x7f4f6c06a200 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f4f6000d6d0 tx=0x7f4f6000db90 comp rx=0 tx=0).stop 2023-10-19T13:01:27.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.096+0000 7f4f721d0640 1 -- 172.21.15.57:0/1269968728 shutdown_connections 2023-10-19T13:01:27.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.096+0000 7f4f721d0640 1 --2- 172.21.15.57:0/1269968728 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f4f5806af00 0x7f4f5806d3c0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:27.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.096+0000 7f4f721d0640 1 --2- 172.21.15.57:0/1269968728 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4f6c15b2a0 0x7f4f6c1f0ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:27.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.096+0000 7f4f721d0640 1 --2- 172.21.15.57:0/1269968728 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4f6c0cf340 0x7f4f6c06a200 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:27.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.096+0000 7f4f721d0640 1 --2- 172.21.15.57:0/1269968728 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4f6c0ceb40 0x7f4f6c0ca8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:27.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.096+0000 7f4f721d0640 1 -- 172.21.15.57:0/1269968728 >> 172.21.15.57:0/1269968728 conn(0x7f4f6c0d0d60 msgr2=0x7f4f6c15a0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:27.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.097+0000 7f4f721d0640 1 -- 172.21.15.57:0/1269968728 shutdown_connections 2023-10-19T13:01:27.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.097+0000 7f4f721d0640 1 -- 172.21.15.57:0/1269968728 wait complete. 2023-10-19T13:01:27.110 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T13:01:27.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.218+0000 7fc4d3fc0640 1 -- 172.21.15.57:0/3423067316 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4cc06ae40 msgr2=0x7fc4cc0b0e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:27.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.218+0000 7fc4d3fc0640 1 --2- 172.21.15.57:0/3423067316 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4cc06ae40 0x7fc4cc0b0e90 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7fc4bc009830 tx=0x7fc4bc02bdf0 comp rx=0 tx=0).stop 2023-10-19T13:01:27.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.218+0000 7fc4d3fc0640 1 -- 172.21.15.57:0/3423067316 shutdown_connections 2023-10-19T13:01:27.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.218+0000 7fc4d3fc0640 1 --2- 172.21.15.57:0/3423067316 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4cc06ae40 0x7fc4cc0b0e90 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:27.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.218+0000 7fc4d3fc0640 1 -- 172.21.15.57:0/3423067316 >> 172.21.15.57:0/3423067316 conn(0x7fc4cc06fae0 msgr2=0x7fc4cc06fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:27.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.218+0000 7fc4d3fc0640 1 -- 172.21.15.57:0/3423067316 shutdown_connections 2023-10-19T13:01:27.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.218+0000 7fc4d3fc0640 1 -- 172.21.15.57:0/3423067316 wait complete. 2023-10-19T13:01:27.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.219+0000 7fc4d3fc0640 1 Processor -- start 2023-10-19T13:01:27.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.219+0000 7fc4d3fc0640 1 -- start start 2023-10-19T13:01:27.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.220+0000 7fc4d3fc0640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc4cc06ae40 0x7fc4cc156790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:27.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.220+0000 7fc4d3fc0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc4cc1579f0 0x7fc4cc143db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:27.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.220+0000 7fc4d3fc0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4cc15c1f0 0x7fc4cc15a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:27.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.220+0000 7fc4d3fc0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4cc148720 con 0x7fc4cc15c1f0 2023-10-19T13:01:27.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.220+0000 7fc4d3fc0640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4cc156cd0 con 0x7fc4cc1579f0 2023-10-19T13:01:27.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.220+0000 7fc4d3fc0640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4cc154470 con 0x7fc4cc06ae40 2023-10-19T13:01:27.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.220+0000 7fc4d255d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4cc15c1f0 0x7fc4cc15a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:27.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.220+0000 7fc4d255d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4cc15c1f0 0x7fc4cc15a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40028/0 (socket says 172.21.15.57:40028) 2023-10-19T13:01:27.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.220+0000 7fc4d255d640 1 -- 172.21.15.57:0/1204242834 learned_addr learned my addr 172.21.15.57:0/1204242834 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:01:27.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.220+0000 7fc4d155b640 1 --2- 172.21.15.57:0/1204242834 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc4cc1579f0 0x7fc4cc143db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:27.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.220+0000 7fc4d1d5c640 1 --2- 172.21.15.57:0/1204242834 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc4cc06ae40 0x7fc4cc156790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:27.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.220+0000 7fc4d255d640 1 -- 172.21.15.57:0/1204242834 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc4cc06ae40 msgr2=0x7fc4cc156790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:27.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.220+0000 7fc4d255d640 1 --2- 172.21.15.57:0/1204242834 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc4cc06ae40 0x7fc4cc156790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:27.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.220+0000 7fc4d255d640 1 -- 172.21.15.57:0/1204242834 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc4cc1579f0 msgr2=0x7fc4cc143db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:27.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.220+0000 7fc4d255d640 1 --2- 172.21.15.57:0/1204242834 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc4cc1579f0 0x7fc4cc143db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:27.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.220+0000 7fc4d255d640 1 -- 172.21.15.57:0/1204242834 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4bc009490 con 0x7fc4cc15c1f0 2023-10-19T13:01:27.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.220+0000 7fc4d1d5c640 1 --2- 172.21.15.57:0/1204242834 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc4cc06ae40 0x7fc4cc156790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T13:01:27.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.221+0000 7fc4d255d640 1 --2- 172.21.15.57:0/1204242834 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4cc15c1f0 0x7fc4cc15a280 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7fc4c400c780 tx=0x7fc4c400cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:27.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.221+0000 7fc4c27fc640 1 -- 172.21.15.57:0/1204242834 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4c4003b70 con 0x7fc4cc15c1f0 2023-10-19T13:01:27.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.221+0000 7fc4c27fc640 1 -- 172.21.15.57:0/1204242834 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fc4c4003cd0 con 0x7fc4cc15c1f0 2023-10-19T13:01:27.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.221+0000 7fc4d3fc0640 1 -- 172.21.15.57:0/1204242834 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4cc144700 con 0x7fc4cc15c1f0 2023-10-19T13:01:27.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.221+0000 7fc4c27fc640 1 -- 172.21.15.57:0/1204242834 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4c40052d0 con 0x7fc4cc15c1f0 2023-10-19T13:01:27.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.221+0000 7fc4d3fc0640 1 -- 172.21.15.57:0/1204242834 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4cc144cc0 con 0x7fc4cc15c1f0 2023-10-19T13:01:27.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.222+0000 7fc4d3fc0640 1 -- 172.21.15.57:0/1204242834 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4cc144eb0 con 0x7fc4cc15c1f0 2023-10-19T13:01:27.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.222+0000 7fc4c27fc640 1 -- 172.21.15.57:0/1204242834 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 47) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fc4c4005430 con 0x7fc4cc15c1f0 2023-10-19T13:01:27.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.223+0000 7fc4c27fc640 1 --2- 172.21.15.57:0/1204242834 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7fc4b006af00 0x7fc4b006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:27.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.223+0000 7fc4c27fc640 1 -- 172.21.15.57:0/1204242834 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fc4c4089700 con 0x7fc4cc15c1f0 2023-10-19T13:01:27.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.223+0000 7fc4d1d5c640 1 --2- 172.21.15.57:0/1204242834 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7fc4b006af00 0x7fc4b006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:27.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.223+0000 7fc4d1d5c640 1 --2- 172.21.15.57:0/1204242834 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7fc4b006af00 0x7fc4b006d3c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc4c8000f60 tx=0x7fc4c8009040 comp rx=0 tx=0).ready entity=mgr.7072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:27.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.227+0000 7fc4c27fc640 1 -- 172.21.15.57:0/1204242834 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc4c40546f0 con 0x7fc4cc15c1f0 2023-10-19T13:01:27.286 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:27.284+0000 7fb508163640 -1 client.7141: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.7030:f55302c0-37aa-4235-8ccc-9775bc542785:v1:172.21.15.179:0/3304791412] 2023-10-19T13:01:27.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.404+0000 7fc4d3fc0640 1 -- 172.21.15.57:0/1204242834 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc4cc0b1d80 con 0x7fc4cc15c1f0 2023-10-19T13:01:27.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.405+0000 7fc4c27fc640 1 -- 172.21.15.57:0/1204242834 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v112) v1 ==== 74+0+16796 (secure 0 0 0) 0x7fc4c4001730 con 0x7fc4cc15c1f0 2023-10-19T13:01:27.409 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:01:27.409 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":112,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T13:01:26.118644+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2023-10-19T13:01:24.443125+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2023-10-19T13:01:25.101276+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T13:01:27.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.407+0000 7fc4d3fc0640 1 -- 172.21.15.57:0/1204242834 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7fc4b006af00 msgr2=0x7fc4b006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:27.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.407+0000 7fc4d3fc0640 1 --2- 172.21.15.57:0/1204242834 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7fc4b006af00 0x7fc4b006d3c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc4c8000f60 tx=0x7fc4c8009040 comp rx=0 tx=0).stop 2023-10-19T13:01:27.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.407+0000 7fc4d3fc0640 1 -- 172.21.15.57:0/1204242834 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4cc15c1f0 msgr2=0x7fc4cc15a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:27.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.407+0000 7fc4d3fc0640 1 --2- 172.21.15.57:0/1204242834 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4cc15c1f0 0x7fc4cc15a280 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7fc4c400c780 tx=0x7fc4c400cb80 comp rx=0 tx=0).stop 2023-10-19T13:01:27.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.407+0000 7fc4d3fc0640 1 -- 172.21.15.57:0/1204242834 shutdown_connections 2023-10-19T13:01:27.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.407+0000 7fc4d3fc0640 1 --2- 172.21.15.57:0/1204242834 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7fc4b006af00 0x7fc4b006d3c0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:27.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.407+0000 7fc4d3fc0640 1 --2- 172.21.15.57:0/1204242834 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4cc15c1f0 0x7fc4cc15a280 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:27.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.407+0000 7fc4d3fc0640 1 --2- 172.21.15.57:0/1204242834 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc4cc1579f0 0x7fc4cc143db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:27.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.407+0000 7fc4d3fc0640 1 --2- 172.21.15.57:0/1204242834 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc4cc06ae40 0x7fc4cc156790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:27.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.407+0000 7fc4d3fc0640 1 -- 172.21.15.57:0/1204242834 >> 172.21.15.57:0/1204242834 conn(0x7fc4cc06fae0 msgr2=0x7fc4cc0ae890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:27.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.407+0000 7fc4d3fc0640 1 -- 172.21.15.57:0/1204242834 shutdown_connections 2023-10-19T13:01:27.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.408+0000 7fc4d3fc0640 1 -- 172.21.15.57:0/1204242834 wait complete. 2023-10-19T13:01:27.422 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-19T13:01:27.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.531+0000 7f2251f89640 1 -- 172.21.15.57:0/3461514235 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f224c0ceb40 msgr2=0x7f224c0b0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:27.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.531+0000 7f2251f89640 1 --2- 172.21.15.57:0/3461514235 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f224c0ceb40 0x7f224c0b0650 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f2244005fd0 tx=0x7f224402c1b0 comp rx=0 tx=0).stop 2023-10-19T13:01:27.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.531+0000 7f2251f89640 1 -- 172.21.15.57:0/3461514235 shutdown_connections 2023-10-19T13:01:27.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.531+0000 7f2251f89640 1 --2- 172.21.15.57:0/3461514235 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f224c0ceb40 0x7f224c0b0650 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:27.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.531+0000 7f2251f89640 1 --2- 172.21.15.57:0/3461514235 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f224c0cf340 0x7f224c0d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:27.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.531+0000 7f2251f89640 1 -- 172.21.15.57:0/3461514235 >> 172.21.15.57:0/3461514235 conn(0x7f224c0d0d60 msgr2=0x7f224c0d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:27.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.531+0000 7f2251f89640 1 -- 172.21.15.57:0/3461514235 shutdown_connections 2023-10-19T13:01:27.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.531+0000 7f2251f89640 1 -- 172.21.15.57:0/3461514235 wait complete. 2023-10-19T13:01:27.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.532+0000 7f2251f89640 1 Processor -- start 2023-10-19T13:01:27.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.533+0000 7f2251f89640 1 -- start start 2023-10-19T13:01:27.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.533+0000 7f2251f89640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f224c0ceb40 0x7f224c158590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:27.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.533+0000 7f2251f89640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f224c0cf340 0x7f224c15d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:27.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.533+0000 7f2251f89640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f224c15b2a0 0x7f224c0b63f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:27.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.533+0000 7f2251f89640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f224c15cd50 con 0x7f224c0cf340 2023-10-19T13:01:27.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.533+0000 7f2251f89640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f224c15c180 con 0x7f224c15b2a0 2023-10-19T13:01:27.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.533+0000 7f2251f89640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f224c15c420 con 0x7f224c0ceb40 2023-10-19T13:01:27.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.533+0000 7f224affd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f224c0cf340 0x7f224c15d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:27.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.533+0000 7f224affd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f224c0cf340 0x7f224c15d1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40040/0 (socket says 172.21.15.57:40040) 2023-10-19T13:01:27.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.533+0000 7f224affd640 1 -- 172.21.15.57:0/3859989489 learned_addr learned my addr 172.21.15.57:0/3859989489 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:01:27.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.534+0000 7f224b7fe640 1 --2- 172.21.15.57:0/3859989489 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f224c0ceb40 0x7f224c158590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:27.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.534+0000 7f224bfff640 1 --2- 172.21.15.57:0/3859989489 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f224c15b2a0 0x7f224c0b63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:27.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.534+0000 7f224affd640 1 -- 172.21.15.57:0/3859989489 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f224c0ceb40 msgr2=0x7f224c158590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:27.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.534+0000 7f224affd640 1 --2- 172.21.15.57:0/3859989489 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f224c0ceb40 0x7f224c158590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:27.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.534+0000 7f224affd640 1 -- 172.21.15.57:0/3859989489 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f224c15b2a0 msgr2=0x7f224c0b63f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:27.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.534+0000 7f224affd640 1 --2- 172.21.15.57:0/3859989489 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f224c15b2a0 0x7f224c0b63f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:27.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.534+0000 7f224affd640 1 -- 172.21.15.57:0/3859989489 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2244009900 con 0x7f224c0cf340 2023-10-19T13:01:27.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.534+0000 7f224bfff640 1 --2- 172.21.15.57:0/3859989489 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f224c15b2a0 0x7f224c0b63f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T13:01:27.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.534+0000 7f224affd640 1 --2- 172.21.15.57:0/3859989489 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f224c0cf340 0x7f224c15d1e0 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f224000b3c0 tx=0x7f224000b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:27.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.534+0000 7f2233fff640 1 -- 172.21.15.57:0/3859989489 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2240011830 con 0x7f224c0cf340 2023-10-19T13:01:27.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.534+0000 7f2233fff640 1 -- 172.21.15.57:0/3859989489 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f2240011a80 con 0x7f224c0cf340 2023-10-19T13:01:27.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.534+0000 7f2251f89640 1 -- 172.21.15.57:0/3859989489 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f224c15c700 con 0x7f224c0cf340 2023-10-19T13:01:27.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.534+0000 7f2233fff640 1 -- 172.21.15.57:0/3859989489 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2240010380 con 0x7f224c0cf340 2023-10-19T13:01:27.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.534+0000 7f2251f89640 1 -- 172.21.15.57:0/3859989489 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f224c1e5e80 con 0x7f224c0cf340 2023-10-19T13:01:27.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.535+0000 7f2251f89640 1 -- 172.21.15.57:0/3859989489 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f224c1e6070 con 0x7f224c0cf340 2023-10-19T13:01:27.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.536+0000 7f2233fff640 1 -- 172.21.15.57:0/3859989489 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 47) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f22400104e0 con 0x7f224c0cf340 2023-10-19T13:01:27.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.536+0000 7f2233fff640 1 --2- 172.21.15.57:0/3859989489 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f223806aed0 0x7f223806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:27.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.536+0000 7f2233fff640 1 -- 172.21.15.57:0/3859989489 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f2240089640 con 0x7f224c0cf340 2023-10-19T13:01:27.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.536+0000 7f224b7fe640 1 --2- 172.21.15.57:0/3859989489 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f223806aed0 0x7f223806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:27.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.537+0000 7f224b7fe640 1 --2- 172.21.15.57:0/3859989489 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f223806aed0 0x7f223806d390 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f223c009aa0 tx=0x7f223c005f90 comp rx=0 tx=0).ready entity=mgr.7072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:27.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.540+0000 7f2233fff640 1 -- 172.21.15.57:0/3859989489 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2240054510 con 0x7f224c0cf340 2023-10-19T13:01:27.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.756+0000 7f2251f89640 1 -- 172.21.15.57:0/3859989489 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f224c0690e0 con 0x7f224c0cf340 2023-10-19T13:01:27.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.769+0000 7f2233fff640 1 -- 172.21.15.57:0/3859989489 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v43) v1 ==== 118+0+0 (secure 0 0 0) 0x7f2240057b30 con 0x7f224c0cf340 2023-10-19T13:01:27.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.772+0000 7f2251f89640 1 -- 172.21.15.57:0/3859989489 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f223806aed0 msgr2=0x7f223806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:27.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.772+0000 7f2251f89640 1 --2- 172.21.15.57:0/3859989489 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f223806aed0 0x7f223806d390 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f223c009aa0 tx=0x7f223c005f90 comp rx=0 tx=0).stop 2023-10-19T13:01:27.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.772+0000 7f2251f89640 1 -- 172.21.15.57:0/3859989489 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f224c0cf340 msgr2=0x7f224c15d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:27.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.772+0000 7f2251f89640 1 --2- 172.21.15.57:0/3859989489 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f224c0cf340 0x7f224c15d1e0 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f224000b3c0 tx=0x7f224000b880 comp rx=0 tx=0).stop 2023-10-19T13:01:27.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.772+0000 7f2251f89640 1 -- 172.21.15.57:0/3859989489 shutdown_connections 2023-10-19T13:01:27.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.772+0000 7f2251f89640 1 --2- 172.21.15.57:0/3859989489 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f223806aed0 0x7f223806d390 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:27.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.772+0000 7f2251f89640 1 --2- 172.21.15.57:0/3859989489 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f224c15b2a0 0x7f224c0b63f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:27.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.772+0000 7f2251f89640 1 --2- 172.21.15.57:0/3859989489 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f224c0cf340 0x7f224c15d1e0 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:27.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.772+0000 7f2251f89640 1 --2- 172.21.15.57:0/3859989489 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f224c0ceb40 0x7f224c158590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:27.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.772+0000 7f2251f89640 1 -- 172.21.15.57:0/3859989489 >> 172.21.15.57:0/3859989489 conn(0x7f224c0d0d60 msgr2=0x7f224c0ae180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:27.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.772+0000 7f2251f89640 1 -- 172.21.15.57:0/3859989489 shutdown_connections 2023-10-19T13:01:27.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.772+0000 7f2251f89640 1 -- 172.21.15.57:0/3859989489 wait complete. 2023-10-19T13:01:27.786 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-19T13:01:27.787 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-19T13:01:27.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.899+0000 7fbc3e61a640 1 -- 172.21.15.57:0/1343459900 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc20003f20 msgr2=0x7fbc200047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:27.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.899+0000 7fbc3e61a640 1 --2- 172.21.15.57:0/1343459900 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc20003f20 0x7fbc200047c0 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7fbc2c009b50 tx=0x7fbc200057f0 comp rx=0 tx=0).stop 2023-10-19T13:01:27.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.899+0000 7fbc3cbb7640 1 -- 172.21.15.57:0/1343459900 reap_dead start 2023-10-19T13:01:27.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.899+0000 7fbc3e61a640 1 -- 172.21.15.57:0/1343459900 shutdown_connections 2023-10-19T13:01:27.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.899+0000 7fbc3e61a640 1 -- 172.21.15.57:0/1343459900 >> 172.21.15.57:0/1343459900 conn(0x7fbc38069930 msgr2=0x7fbc38069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:27.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.899+0000 7fbc3e61a640 1 -- 172.21.15.57:0/1343459900 shutdown_connections 2023-10-19T13:01:27.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.899+0000 7fbc3e61a640 1 -- 172.21.15.57:0/1343459900 wait complete. 2023-10-19T13:01:27.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.901+0000 7fbc3e61a640 1 Processor -- start 2023-10-19T13:01:27.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.901+0000 7fbc3e61a640 1 -- start start 2023-10-19T13:01:27.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.901+0000 7fbc3e61a640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbc20003f20 0x7fbc3814ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:27.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.901+0000 7fbc3e61a640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbc3813e990 0x7fbc3813edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:27.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.901+0000 7fbc3e61a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc38157c80 0x7fbc38158100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:27.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.901+0000 7fbc3e61a640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc3813f330 con 0x7fbc38157c80 2023-10-19T13:01:27.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.901+0000 7fbc3e61a640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc3814efc0 con 0x7fbc20003f20 2023-10-19T13:01:27.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.901+0000 7fbc3e61a640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc38139d60 con 0x7fbc3813e990 2023-10-19T13:01:27.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.902+0000 7fbc3cbb7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc38157c80 0x7fbc38158100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:27.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.902+0000 7fbc3cbb7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc38157c80 0x7fbc38158100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40068/0 (socket says 172.21.15.57:40068) 2023-10-19T13:01:27.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.902+0000 7fbc3cbb7640 1 -- 172.21.15.57:0/3951373778 learned_addr learned my addr 172.21.15.57:0/3951373778 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:01:27.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.902+0000 7fbc37fff640 1 --2- 172.21.15.57:0/3951373778 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbc20003f20 0x7fbc3814ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:27.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.902+0000 7fbc377fe640 1 --2- 172.21.15.57:0/3951373778 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbc3813e990 0x7fbc3813edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:27.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.902+0000 7fbc3cbb7640 1 -- 172.21.15.57:0/3951373778 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbc3813e990 msgr2=0x7fbc3813edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:27.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.902+0000 7fbc3cbb7640 1 --2- 172.21.15.57:0/3951373778 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbc3813e990 0x7fbc3813edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:27.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.902+0000 7fbc3cbb7640 1 -- 172.21.15.57:0/3951373778 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbc20003f20 msgr2=0x7fbc3814ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:27.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.902+0000 7fbc3cbb7640 1 --2- 172.21.15.57:0/3951373778 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbc20003f20 0x7fbc3814ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:27.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.902+0000 7fbc3cbb7640 1 -- 172.21.15.57:0/3951373778 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc3813a040 con 0x7fbc38157c80 2023-10-19T13:01:27.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.902+0000 7fbc3cbb7640 1 --2- 172.21.15.57:0/3951373778 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc38157c80 0x7fbc38158100 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7fbc3806d570 tx=0x7fbc30001c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:27.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.902+0000 7fbc34ff9640 1 -- 172.21.15.57:0/3951373778 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc30011900 con 0x7fbc38157c80 2023-10-19T13:01:27.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.902+0000 7fbc34ff9640 1 -- 172.21.15.57:0/3951373778 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fbc30011a80 con 0x7fbc38157c80 2023-10-19T13:01:27.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.902+0000 7fbc3e61a640 1 -- 172.21.15.57:0/3951373778 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc3813a320 con 0x7fbc38157c80 2023-10-19T13:01:27.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.902+0000 7fbc34ff9640 1 -- 172.21.15.57:0/3951373778 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc30011e90 con 0x7fbc38157c80 2023-10-19T13:01:27.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.902+0000 7fbc3e61a640 1 -- 172.21.15.57:0/3951373778 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc3813a8e0 con 0x7fbc38157c80 2023-10-19T13:01:27.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.903+0000 7fbc3e61a640 1 -- 172.21.15.57:0/3951373778 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc3813ab00 con 0x7fbc38157c80 2023-10-19T13:01:27.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.904+0000 7fbc34ff9640 1 -- 172.21.15.57:0/3951373778 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 47) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fbc3001d460 con 0x7fbc38157c80 2023-10-19T13:01:27.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.904+0000 7fbc34ff9640 1 --2- 172.21.15.57:0/3951373778 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7fbc2406aed0 0x7fbc2406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:27.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.904+0000 7fbc34ff9640 1 -- 172.21.15.57:0/3951373778 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fbc3008a490 con 0x7fbc38157c80 2023-10-19T13:01:27.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.904+0000 7fbc37fff640 1 --2- 172.21.15.57:0/3951373778 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7fbc2406aed0 0x7fbc2406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:27.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.905+0000 7fbc37fff640 1 --2- 172.21.15.57:0/3951373778 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7fbc2406aed0 0x7fbc2406d390 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fbc28001050 tx=0x7fbc28001170 comp rx=0 tx=0).ready entity=mgr.7072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:27.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:27.908+0000 7fbc34ff9640 1 -- 172.21.15.57:0/3951373778 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fbc30055250 con 0x7fbc38157c80 2023-10-19T13:01:28.017 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:28.015+0000 7fb508163640 -1 client.7141: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.7030:f55302c0-37aa-4235-8ccc-9775bc542785:v1:172.21.15.179:0/3304791412] 2023-10-19T13:01:28.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:28.132+0000 7fbc3e61a640 1 -- 172.21.15.57:0/3951373778 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fbc380690e0 con 0x7fbc38157c80 2023-10-19T13:01:28.760 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:28.757+0000 7fb508163640 -1 client.7141: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.7030:f55302c0-37aa-4235-8ccc-9775bc542785:v1:172.21.15.179:0/3304791412] 2023-10-19T13:01:28.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:28.774+0000 7fbc34ff9640 1 -- 172.21.15.57:0/3951373778 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v44) v1 ==== 122+0+0 (secure 0 0 0) 0x7fbc30058870 con 0x7fbc38157c80 2023-10-19T13:01:28.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:28.776+0000 7fbc3e61a640 1 -- 172.21.15.57:0/3951373778 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7fbc2406aed0 msgr2=0x7fbc2406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:28.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:28.776+0000 7fbc3e61a640 1 --2- 172.21.15.57:0/3951373778 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7fbc2406aed0 0x7fbc2406d390 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fbc28001050 tx=0x7fbc28001170 comp rx=0 tx=0).stop 2023-10-19T13:01:28.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:28.776+0000 7fbc3e61a640 1 -- 172.21.15.57:0/3951373778 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc38157c80 msgr2=0x7fbc38158100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:28.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:28.776+0000 7fbc3e61a640 1 --2- 172.21.15.57:0/3951373778 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc38157c80 0x7fbc38158100 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7fbc3806d570 tx=0x7fbc30001c30 comp rx=0 tx=0).stop 2023-10-19T13:01:28.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:28.776+0000 7fbc3e61a640 1 -- 172.21.15.57:0/3951373778 shutdown_connections 2023-10-19T13:01:28.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:28.776+0000 7fbc3e61a640 1 --2- 172.21.15.57:0/3951373778 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7fbc2406aed0 0x7fbc2406d390 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:28.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:28.776+0000 7fbc3e61a640 1 --2- 172.21.15.57:0/3951373778 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc38157c80 0x7fbc38158100 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:28.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:28.776+0000 7fbc3e61a640 1 --2- 172.21.15.57:0/3951373778 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbc3813e990 0x7fbc3813edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:28.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:28.776+0000 7fbc3e61a640 1 --2- 172.21.15.57:0/3951373778 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbc20003f20 0x7fbc3814ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:28.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:28.776+0000 7fbc3e61a640 1 -- 172.21.15.57:0/3951373778 >> 172.21.15.57:0/3951373778 conn(0x7fbc38069930 msgr2=0x7fbc38140060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:28.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:28.776+0000 7fbc3e61a640 1 -- 172.21.15.57:0/3951373778 shutdown_connections 2023-10-19T13:01:28.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:28.776+0000 7fbc3e61a640 1 -- 172.21.15.57:0/3951373778 wait complete. 2023-10-19T13:01:28.790 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-19T13:01:28.791 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T13:01:28.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:28.898+0000 7ff1d1415640 1 -- 172.21.15.57:0/1132099011 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff1cc0cd9e0 msgr2=0x7ff1cc0cde70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:28.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:28.898+0000 7ff1d1415640 1 --2- 172.21.15.57:0/1132099011 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff1cc0cd9e0 0x7ff1cc0cde70 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7ff1cc061510 tx=0x7ff1bc02c250 comp rx=0 tx=0).stop 2023-10-19T13:01:28.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:28.898+0000 7ff1d1415640 1 -- 172.21.15.57:0/1132099011 shutdown_connections 2023-10-19T13:01:28.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:28.898+0000 7ff1d1415640 1 --2- 172.21.15.57:0/1132099011 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff1cc0cd9e0 0x7ff1cc0cde70 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:28.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:28.898+0000 7ff1d1415640 1 --2- 172.21.15.57:0/1132099011 >> v2:172.21.15.179:3300/0 conn(0x7ff1cc0cd1c0 0x7ff1cc15c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:28.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:28.898+0000 7ff1d1415640 1 --2- 172.21.15.57:0/1132099011 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff1cc158910 0x7ff1cc158cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:28.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:28.898+0000 7ff1d1415640 1 -- 172.21.15.57:0/1132099011 >> 172.21.15.57:0/1132099011 conn(0x7ff1cc0af7a0 msgr2=0x7ff1cc0afb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:28.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:28.898+0000 7ff1d1415640 1 -- 172.21.15.57:0/1132099011 shutdown_connections 2023-10-19T13:01:28.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:28.898+0000 7ff1d1415640 1 -- 172.21.15.57:0/1132099011 wait complete. 2023-10-19T13:01:28.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:28.900+0000 7ff1d1415640 1 Processor -- start 2023-10-19T13:01:28.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:28.900+0000 7ff1d1415640 1 -- start start 2023-10-19T13:01:28.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:28.900+0000 7ff1d1415640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff1cc0cd1c0 0x7ff1cc154a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:28.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:28.900+0000 7ff1d1415640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff1cc0cd9e0 0x7ff1cc14fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:28.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:28.900+0000 7ff1d1415640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff1cc158910 0x7ff1cc150080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:28.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:28.900+0000 7ff1d1415640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1cc152210 con 0x7ff1cc158910 2023-10-19T13:01:28.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:28.900+0000 7ff1d1415640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1cc14b2a0 con 0x7ff1cc0cd9e0 2023-10-19T13:01:28.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:28.900+0000 7ff1d1415640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1cc14aa90 con 0x7ff1cc0cd1c0 2023-10-19T13:01:28.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:28.901+0000 7ff1cb7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff1cc158910 0x7ff1cc150080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:28.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:28.901+0000 7ff1caffd640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff1cc0cd1c0 0x7ff1cc154a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:28.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:28.901+0000 7ff1cb7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff1cc158910 0x7ff1cc150080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40086/0 (socket says 172.21.15.57:40086) 2023-10-19T13:01:28.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:28.901+0000 7ff1cb7fe640 1 -- 172.21.15.57:0/2381081874 learned_addr learned my addr 172.21.15.57:0/2381081874 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:01:28.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:28.901+0000 7ff1ca7fc640 1 --2- 172.21.15.57:0/2381081874 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff1cc0cd9e0 0x7ff1cc14fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:28.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:28.901+0000 7ff1cb7fe640 1 -- 172.21.15.57:0/2381081874 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff1cc0cd1c0 msgr2=0x7ff1cc154a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:28.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:28.901+0000 7ff1cb7fe640 1 --2- 172.21.15.57:0/2381081874 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff1cc0cd1c0 0x7ff1cc154a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:28.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:28.901+0000 7ff1cb7fe640 1 -- 172.21.15.57:0/2381081874 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff1cc0cd9e0 msgr2=0x7ff1cc14fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:28.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:28.901+0000 7ff1cb7fe640 1 --2- 172.21.15.57:0/2381081874 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff1cc0cd9e0 0x7ff1cc14fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:28.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:28.901+0000 7ff1cb7fe640 1 -- 172.21.15.57:0/2381081874 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1bc009900 con 0x7ff1cc158910 2023-10-19T13:01:28.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:28.901+0000 7ff1ca7fc640 1 --2- 172.21.15.57:0/2381081874 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff1cc0cd9e0 0x7ff1cc14fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T13:01:28.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:28.901+0000 7ff1cb7fe640 1 --2- 172.21.15.57:0/2381081874 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff1cc158910 0x7ff1cc150080 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc02c860 tx=0x7ff1bc035d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:28.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:28.901+0000 7ff1c37fe640 1 -- 172.21.15.57:0/2381081874 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1bc034440 con 0x7ff1cc158910 2023-10-19T13:01:28.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:28.901+0000 7ff1c37fe640 1 -- 172.21.15.57:0/2381081874 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7ff1bc004540 con 0x7ff1cc158910 2023-10-19T13:01:28.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:28.901+0000 7ff1d1415640 1 -- 172.21.15.57:0/2381081874 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1cc156010 con 0x7ff1cc158910 2023-10-19T13:01:28.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:28.901+0000 7ff1c37fe640 1 -- 172.21.15.57:0/2381081874 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1bc003e60 con 0x7ff1cc158910 2023-10-19T13:01:28.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:28.901+0000 7ff1d1415640 1 -- 172.21.15.57:0/2381081874 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1cc14afc0 con 0x7ff1cc158910 2023-10-19T13:01:28.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:28.902+0000 7ff1d1415640 1 -- 172.21.15.57:0/2381081874 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1cc155e70 con 0x7ff1cc158910 2023-10-19T13:01:28.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:28.903+0000 7ff1c37fe640 1 -- 172.21.15.57:0/2381081874 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 47) v1 ==== 89275+0+0 (secure 0 0 0) 0x7ff1bc00a040 con 0x7ff1cc158910 2023-10-19T13:01:28.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:28.903+0000 7ff1c37fe640 1 --2- 172.21.15.57:0/2381081874 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7ff1ac06aed0 0x7ff1ac06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:28.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:28.903+0000 7ff1c37fe640 1 -- 172.21.15.57:0/2381081874 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6057+0+0 (secure 0 0 0) 0x7ff1bc0ac480 con 0x7ff1cc158910 2023-10-19T13:01:28.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:28.904+0000 7ff1caffd640 1 --2- 172.21.15.57:0/2381081874 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7ff1ac06aed0 0x7ff1ac06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:28.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:28.904+0000 7ff1caffd640 1 --2- 172.21.15.57:0/2381081874 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7ff1ac06aed0 0x7ff1ac06d390 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff1c4009b30 tx=0x7ff1c4005f90 comp rx=0 tx=0).ready entity=mgr.7072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:28.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:28.908+0000 7ff1c37fe640 1 -- 172.21.15.57:0/2381081874 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff1cc155e70 con 0x7ff1cc158910 2023-10-19T13:01:29.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.125+0000 7ff1d1415640 1 -- 172.21.15.57:0/2381081874 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff1cc155e70 con 0x7ff1cc158910 2023-10-19T13:01:29.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.126+0000 7ff1c37fe640 1 -- 172.21.15.57:0/2381081874 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+4921 (secure 0 0 0) 0x7ff1cc155e70 con 0x7ff1cc158910 2023-10-19T13:01:29.129 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:01:29.129 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":44,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6611,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/2708395824","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2708395824},{"type":"v1","addr":"172.21.15.179:6836","nonce":2708395824}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":41},{"gid":7270,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/3803632191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":3803632191},{"type":"v1","addr":"172.21.15.179:6837","nonce":3803632191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":41},{"gid":7282,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6833/2259979700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2259979700},{"type":"v1","addr":"172.21.15.57:6833","nonce":2259979700}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T13:01:26.118951+0000","modified":"2023-10-19T13:01:28.134083+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8046},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8046":{"gid":8046,"name":"c","rank":0,"incarnation":42,"state":"up:active","state_seq":3,"addr":"172.21.15.57:6835/2556427177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":2556427177},{"type":"v1","addr":"172.21.15.57:6835","nonce":2556427177}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-10-19T13:01:29.130 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 44 2023-10-19T13:01:29.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.128+0000 7ff1d1415640 1 -- 172.21.15.57:0/2381081874 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7ff1ac06aed0 msgr2=0x7ff1ac06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:29.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.128+0000 7ff1d1415640 1 --2- 172.21.15.57:0/2381081874 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7ff1ac06aed0 0x7ff1ac06d390 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff1c4009b30 tx=0x7ff1c4005f90 comp rx=0 tx=0).stop 2023-10-19T13:01:29.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.128+0000 7ff1d1415640 1 -- 172.21.15.57:0/2381081874 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff1cc158910 msgr2=0x7ff1cc150080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:29.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.128+0000 7ff1d1415640 1 --2- 172.21.15.57:0/2381081874 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff1cc158910 0x7ff1cc150080 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc02c860 tx=0x7ff1bc035d80 comp rx=0 tx=0).stop 2023-10-19T13:01:29.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.128+0000 7ff1d1415640 1 -- 172.21.15.57:0/2381081874 shutdown_connections 2023-10-19T13:01:29.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.128+0000 7ff1d1415640 1 --2- 172.21.15.57:0/2381081874 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7ff1ac06aed0 0x7ff1ac06d390 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:29.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.128+0000 7ff1d1415640 1 --2- 172.21.15.57:0/2381081874 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff1cc158910 0x7ff1cc150080 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:29.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.128+0000 7ff1d1415640 1 --2- 172.21.15.57:0/2381081874 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff1cc0cd9e0 0x7ff1cc14fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:29.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.128+0000 7ff1d1415640 1 --2- 172.21.15.57:0/2381081874 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff1cc0cd1c0 0x7ff1cc154a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:29.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.128+0000 7ff1d1415640 1 -- 172.21.15.57:0/2381081874 >> 172.21.15.57:0/2381081874 conn(0x7ff1cc0af7a0 msgr2=0x7ff1cc06a0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:29.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.128+0000 7ff1d1415640 1 -- 172.21.15.57:0/2381081874 shutdown_connections 2023-10-19T13:01:29.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.128+0000 7ff1d1415640 1 -- 172.21.15.57:0/2381081874 wait complete. 2023-10-19T13:01:29.142 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T13:01:29.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.250+0000 7facf5a90640 1 -- 172.21.15.57:0/1760082334 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7facf015b2a0 msgr2=0x7facf00cf720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:29.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.250+0000 7facf5a90640 1 --2- 172.21.15.57:0/1760082334 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7facf015b2a0 0x7facf00cf720 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7facd8005fd0 tx=0x7facd802bcf0 comp rx=0 tx=0).stop 2023-10-19T13:01:29.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.250+0000 7facf5a90640 1 -- 172.21.15.57:0/1760082334 shutdown_connections 2023-10-19T13:01:29.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.250+0000 7facf5a90640 1 --2- 172.21.15.57:0/1760082334 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7facf015b2a0 0x7facf00cf720 unknown :-1 s=CLOSED pgs=1286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:29.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.250+0000 7facf5a90640 1 --2- 172.21.15.57:0/1760082334 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7facf00cf340 0x7facf00d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:29.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.250+0000 7facf5a90640 1 -- 172.21.15.57:0/1760082334 >> 172.21.15.57:0/1760082334 conn(0x7facf00d0d60 msgr2=0x7facf00d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:29.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.250+0000 7facf5a90640 1 -- 172.21.15.57:0/1760082334 shutdown_connections 2023-10-19T13:01:29.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.250+0000 7facf5a90640 1 -- 172.21.15.57:0/1760082334 wait complete. 2023-10-19T13:01:29.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.252+0000 7facf5a90640 1 Processor -- start 2023-10-19T13:01:29.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.252+0000 7facf5a90640 1 -- start start 2023-10-19T13:01:29.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.253+0000 7facf5a90640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7facf00ceb40 0x7facf015cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:29.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.253+0000 7facf5a90640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7facf00cf340 0x7facf015d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:29.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.253+0000 7facf5a90640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7facf015b2a0 0x7facf00c8fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:29.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.253+0000 7facf5a90640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facf0069f90 con 0x7facf00cf340 2023-10-19T13:01:29.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.253+0000 7facf5a90640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facf006a270 con 0x7facf015b2a0 2023-10-19T13:01:29.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.253+0000 7facf5a90640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facf006a550 con 0x7facf00ceb40 2023-10-19T13:01:29.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.253+0000 7faceeffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7facf00cf340 0x7facf015d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:29.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.253+0000 7faceeffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7facf00cf340 0x7facf015d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40116/0 (socket says 172.21.15.57:40116) 2023-10-19T13:01:29.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.253+0000 7faceeffd640 1 -- 172.21.15.57:0/2682178951 learned_addr learned my addr 172.21.15.57:0/2682178951 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:01:29.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.253+0000 7facef7fe640 1 --2- 172.21.15.57:0/2682178951 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7facf00ceb40 0x7facf015cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:29.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.253+0000 7faceffff640 1 --2- 172.21.15.57:0/2682178951 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7facf015b2a0 0x7facf00c8fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:29.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.253+0000 7faceeffd640 1 -- 172.21.15.57:0/2682178951 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7facf00ceb40 msgr2=0x7facf015cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:29.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.253+0000 7faceeffd640 1 --2- 172.21.15.57:0/2682178951 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7facf00ceb40 0x7facf015cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:29.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.253+0000 7faceeffd640 1 -- 172.21.15.57:0/2682178951 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7facf015b2a0 msgr2=0x7facf00c8fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:29.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.253+0000 7faceeffd640 1 --2- 172.21.15.57:0/2682178951 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7facf015b2a0 0x7facf00c8fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:29.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.253+0000 7faceeffd640 1 -- 172.21.15.57:0/2682178951 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facd8009560 con 0x7facf00cf340 2023-10-19T13:01:29.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.254+0000 7faceeffd640 1 --2- 172.21.15.57:0/2682178951 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7facf00cf340 0x7facf015d440 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7facd8009850 tx=0x7facd802da70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:29.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.254+0000 7face7fff640 1 -- 172.21.15.57:0/2682178951 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facd802de70 con 0x7facf00cf340 2023-10-19T13:01:29.262 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.254+0000 7face7fff640 1 -- 172.21.15.57:0/2682178951 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7facd80036e0 con 0x7facf00cf340 2023-10-19T13:01:29.262 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.254+0000 7facf5a90640 1 -- 172.21.15.57:0/2682178951 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facf006a830 con 0x7facf00cf340 2023-10-19T13:01:29.262 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.254+0000 7face7fff640 1 -- 172.21.15.57:0/2682178951 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facd8003a90 con 0x7facf00cf340 2023-10-19T13:01:29.262 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.254+0000 7facf5a90640 1 -- 172.21.15.57:0/2682178951 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facf01f14f0 con 0x7facf00cf340 2023-10-19T13:01:29.263 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.255+0000 7facf5a90640 1 -- 172.21.15.57:0/2682178951 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facf00690e0 con 0x7facf00cf340 2023-10-19T13:01:29.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.255+0000 7face7fff640 1 -- 172.21.15.57:0/2682178951 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 47) v1 ==== 89275+0+0 (secure 0 0 0) 0x7facd8003bf0 con 0x7facf00cf340 2023-10-19T13:01:29.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.256+0000 7face7fff640 1 --2- 172.21.15.57:0/2682178951 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7facd006aed0 0x7facd006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:29.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.256+0000 7face7fff640 1 -- 172.21.15.57:0/2682178951 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6057+0+0 (secure 0 0 0) 0x7facd80ac340 con 0x7facf00cf340 2023-10-19T13:01:29.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.256+0000 7facef7fe640 1 --2- 172.21.15.57:0/2682178951 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7facd006aed0 0x7facd006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:29.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.257+0000 7facef7fe640 1 --2- 172.21.15.57:0/2682178951 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7facd006aed0 0x7facd006d390 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7face00058c0 tx=0x7face000a2d0 comp rx=0 tx=0).ready entity=mgr.7072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:29.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.260+0000 7face7fff640 1 -- 172.21.15.57:0/2682178951 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7facd8077380 con 0x7facf00cf340 2023-10-19T13:01:29.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.440+0000 7facf5a90640 1 -- 172.21.15.57:0/2682178951 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7facf00d0b40 con 0x7facf00cf340 2023-10-19T13:01:29.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.441+0000 7face7fff640 1 -- 172.21.15.57:0/2682178951 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v113) v1 ==== 74+0+16783 (secure 0 0 0) 0x7facd807a9a0 con 0x7facf00cf340 2023-10-19T13:01:29.444 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:01:29.445 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":113,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T13:01:27.764897+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2023-10-19T13:01:24.443125+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2023-10-19T13:01:25.101276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"113","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T13:01:29.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.444+0000 7facf5a90640 1 -- 172.21.15.57:0/2682178951 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7facd006aed0 msgr2=0x7facd006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:29.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.444+0000 7facf5a90640 1 --2- 172.21.15.57:0/2682178951 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7facd006aed0 0x7facd006d390 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7face00058c0 tx=0x7face000a2d0 comp rx=0 tx=0).stop 2023-10-19T13:01:29.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.444+0000 7facf5a90640 1 -- 172.21.15.57:0/2682178951 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7facf00cf340 msgr2=0x7facf015d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:29.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.444+0000 7facf5a90640 1 --2- 172.21.15.57:0/2682178951 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7facf00cf340 0x7facf015d440 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7facd8009850 tx=0x7facd802da70 comp rx=0 tx=0).stop 2023-10-19T13:01:29.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.444+0000 7facf5a90640 1 -- 172.21.15.57:0/2682178951 shutdown_connections 2023-10-19T13:01:29.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.444+0000 7facf5a90640 1 --2- 172.21.15.57:0/2682178951 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7facd006aed0 0x7facd006d390 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:29.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.444+0000 7facf5a90640 1 --2- 172.21.15.57:0/2682178951 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7facf015b2a0 0x7facf00c8fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:29.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.444+0000 7facf5a90640 1 --2- 172.21.15.57:0/2682178951 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7facf00cf340 0x7facf015d440 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:29.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.444+0000 7facf5a90640 1 --2- 172.21.15.57:0/2682178951 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7facf00ceb40 0x7facf015cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:29.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.444+0000 7facf5a90640 1 -- 172.21.15.57:0/2682178951 >> 172.21.15.57:0/2682178951 conn(0x7facf00d0d60 msgr2=0x7facf00ccf40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:29.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.444+0000 7facf5a90640 1 -- 172.21.15.57:0/2682178951 shutdown_connections 2023-10-19T13:01:29.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.444+0000 7facf5a90640 1 -- 172.21.15.57:0/2682178951 wait complete. 2023-10-19T13:01:29.458 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-19T13:01:29.458 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-19T13:01:29.502 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:29.499+0000 7fb508163640 -1 client.7141: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.7030:f55302c0-37aa-4235-8ccc-9775bc542785:v1:172.21.15.179:0/3304791412] 2023-10-19T13:01:29.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.566+0000 7fe743d0f640 1 -- 172.21.15.57:0/1449132763 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe73c06a730 msgr2=0x7fe73c06abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:29.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.566+0000 7fe743d0f640 1 --2- 172.21.15.57:0/1449132763 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe73c06a730 0x7fe73c06abc0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fe734005fd0 tx=0x7fe73402c170 comp rx=0 tx=0).stop 2023-10-19T13:01:29.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.566+0000 7fe743d0f640 1 -- 172.21.15.57:0/1449132763 shutdown_connections 2023-10-19T13:01:29.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.566+0000 7fe743d0f640 1 --2- 172.21.15.57:0/1449132763 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe73c06a730 0x7fe73c06abc0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:29.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.566+0000 7fe743d0f640 1 -- 172.21.15.57:0/1449132763 >> 172.21.15.57:0/1449132763 conn(0x7fe73c06fae0 msgr2=0x7fe73c06fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:29.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.566+0000 7fe743d0f640 1 -- 172.21.15.57:0/1449132763 shutdown_connections 2023-10-19T13:01:29.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.566+0000 7fe743d0f640 1 -- 172.21.15.57:0/1449132763 wait complete. 2023-10-19T13:01:29.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.567+0000 7fe743d0f640 1 Processor -- start 2023-10-19T13:01:29.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.567+0000 7fe743d0f640 1 -- start start 2023-10-19T13:01:29.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.568+0000 7fe743d0f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe73c06a730 0x7fe73c06c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:29.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.568+0000 7fe743d0f640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe73c0ac870 0x7fe73c0a98a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:29.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.568+0000 7fe743d0f640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe73c156790 0x7fe73c0a9de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:29.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.568+0000 7fe743d0f640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe73c0ab520 con 0x7fe73c06a730 2023-10-19T13:01:29.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.568+0000 7fe743d0f640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe73c0ab090 con 0x7fe73c156790 2023-10-19T13:01:29.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.568+0000 7fe743d0f640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe73c0c2720 con 0x7fe73c0ac870 2023-10-19T13:01:29.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.568+0000 7fe741aab640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe73c06a730 0x7fe73c06c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:29.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.568+0000 7fe741aab640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe73c06a730 0x7fe73c06c520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40118/0 (socket says 172.21.15.57:40118) 2023-10-19T13:01:29.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.568+0000 7fe741aab640 1 -- 172.21.15.57:0/3771758238 learned_addr learned my addr 172.21.15.57:0/3771758238 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:01:29.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.568+0000 7fe7422ac640 1 --2- 172.21.15.57:0/3771758238 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe73c156790 0x7fe73c0a9de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:29.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.568+0000 7fe7412aa640 1 --2- 172.21.15.57:0/3771758238 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe73c0ac870 0x7fe73c0a98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:29.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.568+0000 7fe741aab640 1 -- 172.21.15.57:0/3771758238 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe73c0ac870 msgr2=0x7fe73c0a98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:29.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.568+0000 7fe741aab640 1 --2- 172.21.15.57:0/3771758238 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe73c0ac870 0x7fe73c0a98a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:29.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.568+0000 7fe741aab640 1 -- 172.21.15.57:0/3771758238 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe73c156790 msgr2=0x7fe73c0a9de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:29.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.568+0000 7fe741aab640 1 --2- 172.21.15.57:0/3771758238 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe73c156790 0x7fe73c0a9de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:29.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.568+0000 7fe741aab640 1 -- 172.21.15.57:0/3771758238 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe734009840 con 0x7fe73c06a730 2023-10-19T13:01:29.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.568+0000 7fe741aab640 1 --2- 172.21.15.57:0/3771758238 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe73c06a730 0x7fe73c06c520 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7fe73800c4b0 tx=0x7fe73800c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:29.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.569+0000 7fe7327fc640 1 -- 172.21.15.57:0/3771758238 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe738005480 con 0x7fe73c06a730 2023-10-19T13:01:29.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.569+0000 7fe7327fc640 1 -- 172.21.15.57:0/3771758238 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fe7380036e0 con 0x7fe73c06a730 2023-10-19T13:01:29.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.569+0000 7fe743d0f640 1 -- 172.21.15.57:0/3771758238 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe73c0aa480 con 0x7fe73c06a730 2023-10-19T13:01:29.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.569+0000 7fe7327fc640 1 -- 172.21.15.57:0/3771758238 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe738003b30 con 0x7fe73c06a730 2023-10-19T13:01:29.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.569+0000 7fe743d0f640 1 -- 172.21.15.57:0/3771758238 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe73c0aaa40 con 0x7fe73c06a730 2023-10-19T13:01:29.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.570+0000 7fe743d0f640 1 -- 172.21.15.57:0/3771758238 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe73c0690e0 con 0x7fe73c06a730 2023-10-19T13:01:29.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.570+0000 7fe7327fc640 1 -- 172.21.15.57:0/3771758238 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 47) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fe738003c90 con 0x7fe73c06a730 2023-10-19T13:01:29.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.570+0000 7fe7327fc640 1 --2- 172.21.15.57:0/3771758238 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7fe72006af00 0x7fe72006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:29.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.571+0000 7fe7327fc640 1 -- 172.21.15.57:0/3771758238 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fe738088ec0 con 0x7fe73c06a730 2023-10-19T13:01:29.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.571+0000 7fe7412aa640 1 --2- 172.21.15.57:0/3771758238 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7fe72006af00 0x7fe72006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:29.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.571+0000 7fe7412aa640 1 --2- 172.21.15.57:0/3771758238 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7fe72006af00 0x7fe72006d3c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe73c0c9d70 tx=0x7fe72c009040 comp rx=0 tx=0).ready entity=mgr.7072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:29.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.575+0000 7fe7327fc640 1 -- 172.21.15.57:0/3771758238 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe738053ed0 con 0x7fe73c06a730 2023-10-19T13:01:29.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.756+0000 7fe743d0f640 1 -- 172.21.15.57:0/3771758238 --> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe73c0ab1d0 con 0x7fe72006af00 2023-10-19T13:01:29.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.764+0000 7fe7327fc640 1 -- 172.21.15.57:0/3771758238 <== mgr.7072 v2:172.21.15.179:6832/3281457486 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+279394 (secure 0 0 0) 0x7fe72c001330 con 0x7fe72006af00 2023-10-19T13:01:29.770 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:01:29.772 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2023-10-19T13:01:29.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.767+0000 7fe743d0f640 1 -- 172.21.15.57:0/3771758238 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7fe72006af00 msgr2=0x7fe72006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:29.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.767+0000 7fe743d0f640 1 --2- 172.21.15.57:0/3771758238 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7fe72006af00 0x7fe72006d3c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe73c0c9d70 tx=0x7fe72c009040 comp rx=0 tx=0).stop 2023-10-19T13:01:29.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.767+0000 7fe743d0f640 1 -- 172.21.15.57:0/3771758238 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe73c06a730 msgr2=0x7fe73c06c520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:29.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.767+0000 7fe743d0f640 1 --2- 172.21.15.57:0/3771758238 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe73c06a730 0x7fe73c06c520 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7fe73800c4b0 tx=0x7fe73800c970 comp rx=0 tx=0).stop 2023-10-19T13:01:29.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.767+0000 7fe743d0f640 1 -- 172.21.15.57:0/3771758238 shutdown_connections 2023-10-19T13:01:29.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.767+0000 7fe743d0f640 1 --2- 172.21.15.57:0/3771758238 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7fe72006af00 0x7fe72006d3c0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:29.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.767+0000 7fe743d0f640 1 --2- 172.21.15.57:0/3771758238 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe73c156790 0x7fe73c0a9de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:29.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.767+0000 7fe743d0f640 1 --2- 172.21.15.57:0/3771758238 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe73c0ac870 0x7fe73c0a98a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:29.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.767+0000 7fe743d0f640 1 --2- 172.21.15.57:0/3771758238 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe73c06a730 0x7fe73c06c520 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:29.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.767+0000 7fe743d0f640 1 -- 172.21.15.57:0/3771758238 >> 172.21.15.57:0/3771758238 conn(0x7fe73c06fae0 msgr2=0x7fe73c1574f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:29.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.767+0000 7fe743d0f640 1 -- 172.21.15.57:0/3771758238 shutdown_connections 2023-10-19T13:01:29.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.767+0000 7fe743d0f640 1 -- 172.21.15.57:0/3771758238 wait complete. 2023-10-19T13:01:29.781 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2023-10-19T13:01:28.070377+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":591980,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":755,"num_read_kb":1517,"num_write":307,"num_write_kb":2419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1518,"ondisk_log_size":1518,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":3,"kb":754974720,"kb_used":315128,"kb_used_data":11524,"kb_used_omap":12,"kb_used_meta":303475,"kb_avail":754659592,"statfs":{"total":773094113280,"available":772771422208,"internally_reserved":0,"allocated":11800576,"data_stored":6485543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":310758969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1612,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000685"},"pg_stats":[{"pgid":"14.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:01:25.747321+0000","last_change":"2023-10-19T13:01:25.747321+0000","last_active":"2023-10-19T13:01:25.747321+0000","last_peered":"2023-10-19T13:01:25.747321+0000","last_clean":"2023-10-19T13:01:25.747321+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:01:25.747321+0000","last_undegraded":"2023-10-19T13:01:25.747321+0000","last_fullsized":"2023-10-19T13:01:25.747321+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:01:25.747321+0000","last_change":"2023-10-19T13:01:25.747321+0000","last_active":"2023-10-19T13:01:25.747321+0000","last_peered":"2023-10-19T13:01:25.747321+0000","last_clean":"2023-10-19T13:01:25.747321+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:01:25.747321+0000","last_undegraded":"2023-10-19T13:01:25.747321+0000","last_fullsized":"2023-10-19T13:01:25.747321+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:01:25.747321+0000","last_change":"2023-10-19T13:01:25.747321+0000","last_active":"2023-10-19T13:01:25.747321+0000","last_peered":"2023-10-19T13:01:25.747321+0000","last_clean":"2023-10-19T13:01:25.747321+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:01:25.747321+0000","last_undegraded":"2023-10-19T13:01:25.747321+0000","last_fullsized":"2023-10-19T13:01:25.747321+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.3c","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-19T13:01:27.773210+0000","last_change":"2023-10-19T13:01:26.776308+0000","last_active":"2023-10-19T13:01:27.773210+0000","last_peered":"2023-10-19T13:01:27.773210+0000","last_clean":"2023-10-19T13:01:27.773210+0000","last_became_active":"2023-10-19T13:01:26.776045+0000","last_became_peered":"2023-10-19T13:01:26.776045+0000","last_unstale":"2023-10-19T13:01:27.773210+0000","last_undegraded":"2023-10-19T13:01:27.773210+0000","last_fullsized":"2023-10-19T13:01:27.773210+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:40:39.673912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:01:25.747321+0000","last_change":"2023-10-19T13:01:25.747321+0000","last_active":"2023-10-19T13:01:25.747321+0000","last_peered":"2023-10-19T13:01:25.747321+0000","last_clean":"2023-10-19T13:01:25.747321+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:01:25.747321+0000","last_undegraded":"2023-10-19T13:01:25.747321+0000","last_fullsized":"2023-10-19T13:01:25.747321+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.3a","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-19T13:01:26.785125+0000","last_change":"2023-10-19T13:01:26.775715+0000","last_active":"2023-10-19T13:01:26.785125+0000","last_peered":"2023-10-19T13:01:26.785125+0000","last_clean":"2023-10-19T13:01:26.785125+0000","last_became_active":"2023-10-19T13:01:26.775344+0000","last_became_peered":"2023-10-19T13:01:26.775344+0000","last_unstale":"2023-10-19T13:01:26.785125+0000","last_undegraded":"2023-10-19T13:01:26.785125+0000","last_fullsized":"2023-10-19T13:01:26.785125+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:18:30.670180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:01:25.747321+0000","last_change":"2023-10-19T13:01:25.747321+0000","last_active":"2023-10-19T13:01:25.747321+0000","last_peered":"2023-10-19T13:01:25.747321+0000","last_clean":"2023-10-19T13:01:25.747321+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:01:25.747321+0000","last_undegraded":"2023-10-19T13:01:25.747321+0000","last_fullsized":"2023-10-19T13:01:25.747321+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.38","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-19T13:01:27.772777+0000","last_change":"2023-10-19T13:01:26.770141+0000","last_active":"2023-10-19T13:01:27.772777+0000","last_peered":"2023-10-19T13:01:27.772777+0000","last_clean":"2023-10-19T13:01:27.772777+0000","last_became_active":"2023-10-19T13:01:26.769179+0000","last_became_peered":"2023-10-19T13:01:26.769179+0000","last_unstale":"2023-10-19T13:01:27.772777+0000","last_undegraded":"2023-10-19T13:01:27.772777+0000","last_fullsized":"2023-10-19T13:01:27.772777+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:15:36.508855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:01:25.747321+0000","last_change":"2023-10-19T13:01:25.747321+0000","last_active":"2023-10-19T13:01:25.747321+0000","last_peered":"2023-10-19T13:01:25.747321+0000","last_clean":"2023-10-19T13:01:25.747321+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:01:25.747321+0000","last_undegraded":"2023-10-19T13:01:25.747321+0000","last_fullsized":"2023-10-19T13:01:25.747321+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:01:25.747321+0000","last_change":"2023-10-19T13:01:25.747321+0000","last_active":"2023-10-19T13:01:25.747321+0000","last_peered":"2023-10-19T13:01:25.747321+0000","last_clean":"2023-10-19T13:01:25.747321+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:01:25.747321+0000","last_undegraded":"2023-10-19T13:01:25.747321+0000","last_fullsized":"2023-10-19T13:01:25.747321+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:01:25.747321+0000","last_change":"2023-10-19T13:01:25.747321+0000","last_active":"2023-10-19T13:01:25.747321+0000","last_peered":"2023-10-19T13:01:25.747321+0000","last_clean":"2023-10-19T13:01:25.747321+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:01:25.747321+0000","last_undegraded":"2023-10-19T13:01:25.747321+0000","last_fullsized":"2023-10-19T13:01:25.747321+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:01:25.747321+0000","last_change":"2023-10-19T13:01:25.747321+0000","last_active":"2023-10-19T13:01:25.747321+0000","last_peered":"2023-10-19T13:01:25.747321+0000","last_clean":"2023-10-19T13:01:25.747321+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:01:25.747321+0000","last_undegraded":"2023-10-19T13:01:25.747321+0000","last_fullsized":"2023-10-19T13:01:25.747321+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.33","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-19T13:01:27.772916+0000","last_change":"2023-10-19T13:01:26.775962+0000","last_active":"2023-10-19T13:01:27.772916+0000","last_peered":"2023-10-19T13:01:27.772916+0000","last_clean":"2023-10-19T13:01:27.772916+0000","last_became_active":"2023-10-19T13:01:26.775624+0000","last_became_peered":"2023-10-19T13:01:26.775624+0000","last_unstale":"2023-10-19T13:01:27.772916+0000","last_undegraded":"2023-10-19T13:01:27.772916+0000","last_fullsized":"2023-10-19T13:01:27.772916+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:28:51.930101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:01:25.747321+0000","last_change":"2023-10-19T13:01:25.747321+0000","last_active":"2023-10-19T13:01:25.747321+0000","last_peered":"2023-10-19T13:01:25.747321+0000","last_clean":"2023-10-19T13:01:25.747321+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:01:25.747321+0000","last_undegraded":"2023-10-19T13:01:25.747321+0000","last_fullsized":"2023-10-19T13:01:25.747321+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.31","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-19T13:01:27.772998+0000","last_change":"2023-10-19T13:01:26.775387+0000","last_active":"2023-10-19T13:01:27.772998+0000","last_peered":"2023-10-19T13:01:27.772998+0000","last_clean":"2023-10-19T13:01:27.772998+0000","last_became_active":"2023-10-19T13:01:26.775117+0000","last_became_peered":"2023-10-19T13:01:26.775117+0000","last_unstale":"2023-10-19T13:01:27.772998+0000","last_undegraded":"2023-10-19T13:01:27.772998+0000","last_fullsized":"2023-10-19T13:01:27.772998+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:23:19.825549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.30","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-19T13:01:26.787259+0000","last_change":"2023-10-19T13:01:26.775007+0000","last_active":"2023-10-19T13:01:26.787259+0000","last_peered":"2023-10-19T13:01:26.787259+0000","last_clean":"2023-10-19T13:01:26.787259+0000","last_became_active":"2023-10-19T13:01:26.774583+0000","last_became_peered":"2023-10-19T13:01:26.774583+0000","last_unstale":"2023-10-19T13:01:26.787259+0000","last_undegraded":"2023-10-19T13:01:26.787259+0000","last_fullsized":"2023-10-19T13:01:26.787259+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:30:29.811626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2f","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-19T13:01:26.785825+0000","last_change":"2023-10-19T13:01:26.770700+0000","last_active":"2023-10-19T13:01:26.785825+0000","last_peered":"2023-10-19T13:01:26.785825+0000","last_clean":"2023-10-19T13:01:26.785825+0000","last_became_active":"2023-10-19T13:01:26.770096+0000","last_became_peered":"2023-10-19T13:01:26.770096+0000","last_unstale":"2023-10-19T13:01:26.785825+0000","last_undegraded":"2023-10-19T13:01:26.785825+0000","last_fullsized":"2023-10-19T13:01:26.785825+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:24:28.596390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:01:25.747321+0000","last_change":"2023-10-19T13:01:25.747321+0000","last_active":"2023-10-19T13:01:25.747321+0000","last_peered":"2023-10-19T13:01:25.747321+0000","last_clean":"2023-10-19T13:01:25.747321+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:01:25.747321+0000","last_undegraded":"2023-10-19T13:01:25.747321+0000","last_fullsized":"2023-10-19T13:01:25.747321+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:01:25.747321+0000","last_change":"2023-10-19T13:01:25.747321+0000","last_active":"2023-10-19T13:01:25.747321+0000","last_peered":"2023-10-19T13:01:25.747321+0000","last_clean":"2023-10-19T13:01:25.747321+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:01:25.747321+0000","last_undegraded":"2023-10-19T13:01:25.747321+0000","last_fullsized":"2023-10-19T13:01:25.747321+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.2c","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-19T13:01:27.773284+0000","last_change":"2023-10-19T13:01:26.776034+0000","last_active":"2023-10-19T13:01:27.773284+0000","last_peered":"2023-10-19T13:01:27.773284+0000","last_clean":"2023-10-19T13:01:27.773284+0000","last_became_active":"2023-10-19T13:01:26.775709+0000","last_became_peered":"2023-10-19T13:01:26.775709+0000","last_unstale":"2023-10-19T13:01:27.773284+0000","last_undegraded":"2023-10-19T13:01:27.773284+0000","last_fullsized":"2023-10-19T13:01:27.773284+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:32:45.053563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:01:25.747321+0000","last_change":"2023-10-19T13:01:25.747321+0000","last_active":"2023-10-19T13:01:25.747321+0000","last_peered":"2023-10-19T13:01:25.747321+0000","last_clean":"2023-10-19T13:01:25.747321+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:01:25.747321+0000","last_undegraded":"2023-10-19T13:01:25.747321+0000","last_fullsized":"2023-10-19T13:01:25.747321+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:01:25.747321+0000","last_change":"2023-10-19T13:01:25.747321+0000","last_active":"2023-10-19T13:01:25.747321+0000","last_peered":"2023-10-19T13:01:25.747321+0000","last_clean":"2023-10-19T13:01:25.747321+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:01:25.747321+0000","last_undegraded":"2023-10-19T13:01:25.747321+0000","last_fullsized":"2023-10-19T13:01:25.747321+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:01:25.747321+0000","last_change":"2023-10-19T13:01:25.747321+0000","last_active":"2023-10-19T13:01:25.747321+0000","last_peered":"2023-10-19T13:01:25.747321+0000","last_clean":"2023-10-19T13:01:25.747321+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:01:25.747321+0000","last_undegraded":"2023-10-19T13:01:25.747321+0000","last_fullsized":"2023-10-19T13:01:25.747321+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:01:25.747321+0000","last_change":"2023-10-19T13:01:25.747321+0000","last_active":"2023-10-19T13:01:25.747321+0000","last_peered":"2023-10-19T13:01:25.747321+0000","last_clean":"2023-10-19T13:01:25.747321+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:01:25.747321+0000","last_undegraded":"2023-10-19T13:01:25.747321+0000","last_fullsized":"2023-10-19T13:01:25.747321+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:01:25.747321+0000","last_change":"2023-10-19T13:01:25.747321+0000","last_active":"2023-10-19T13:01:25.747321+0000","last_peered":"2023-10-19T13:01:25.747321+0000","last_clean":"2023-10-19T13:01:25.747321+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:01:25.747321+0000","last_undegraded":"2023-10-19T13:01:25.747321+0000","last_fullsized":"2023-10-19T13:01:25.747321+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:01:25.747321+0000","last_change":"2023-10-19T13:01:25.747321+0000","last_active":"2023-10-19T13:01:25.747321+0000","last_peered":"2023-10-19T13:01:25.747321+0000","last_clean":"2023-10-19T13:01:25.747321+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:01:25.747321+0000","last_undegraded":"2023-10-19T13:01:25.747321+0000","last_fullsized":"2023-10-19T13:01:25.747321+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:01:25.747321+0000","last_change":"2023-10-19T13:01:25.747321+0000","last_active":"2023-10-19T13:01:25.747321+0000","last_peered":"2023-10-19T13:01:25.747321+0000","last_clean":"2023-10-19T13:01:25.747321+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:01:25.747321+0000","last_undegraded":"2023-10-19T13:01:25.747321+0000","last_fullsized":"2023-10-19T13:01:25.747321+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.24","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-19T13:01:27.773344+0000","last_change":"2023-10-19T13:01:26.776210+0000","last_active":"2023-10-19T13:01:27.773344+0000","last_peered":"2023-10-19T13:01:27.773344+0000","last_clean":"2023-10-19T13:01:27.773344+0000","last_became_active":"2023-10-19T13:01:26.775771+0000","last_became_peered":"2023-10-19T13:01:26.775771+0000","last_unstale":"2023-10-19T13:01:27.773344+0000","last_undegraded":"2023-10-19T13:01:27.773344+0000","last_fullsized":"2023-10-19T13:01:27.773344+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:51:32.035197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:01:25.747321+0000","last_change":"2023-10-19T13:01:25.747321+0000","last_active":"2023-10-19T13:01:25.747321+0000","last_peered":"2023-10-19T13:01:25.747321+0000","last_clean":"2023-10-19T13:01:25.747321+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:01:25.747321+0000","last_undegraded":"2023-10-19T13:01:25.747321+0000","last_fullsized":"2023-10-19T13:01:25.747321+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:01:25.747321+0000","last_change":"2023-10-19T13:01:25.747321+0000","last_active":"2023-10-19T13:01:25.747321+0000","last_peered":"2023-10-19T13:01:25.747321+0000","last_clean":"2023-10-19T13:01:25.747321+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:01:25.747321+0000","last_undegraded":"2023-10-19T13:01:25.747321+0000","last_fullsized":"2023-10-19T13:01:25.747321+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:01:25.747321+0000","last_change":"2023-10-19T13:01:25.747321+0000","last_active":"2023-10-19T13:01:25.747321+0000","last_peered":"2023-10-19T13:01:25.747321+0000","last_clean":"2023-10-19T13:01:25.747321+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:01:25.747321+0000","last_undegraded":"2023-10-19T13:01:25.747321+0000","last_fullsized":"2023-10-19T13:01:25.747321+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.20","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-19T13:01:26.787250+0000","last_change":"2023-10-19T13:01:26.772839+0000","last_active":"2023-10-19T13:01:26.787250+0000","last_peered":"2023-10-19T13:01:26.787250+0000","last_clean":"2023-10-19T13:01:26.787250+0000","last_became_active":"2023-10-19T13:01:26.772247+0000","last_became_peered":"2023-10-19T13:01:26.772247+0000","last_unstale":"2023-10-19T13:01:26.787250+0000","last_undegraded":"2023-10-19T13:01:26.787250+0000","last_fullsized":"2023-10-19T13:01:26.787250+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:14:09.608228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-19T13:01:26.786235+0000","last_change":"2023-10-19T13:01:26.774468+0000","last_active":"2023-10-19T13:01:26.786235+0000","last_peered":"2023-10-19T13:01:26.786235+0000","last_clean":"2023-10-19T13:01:26.786235+0000","last_became_active":"2023-10-19T13:01:26.774207+0000","last_became_peered":"2023-10-19T13:01:26.774207+0000","last_unstale":"2023-10-19T13:01:26.786235+0000","last_undegraded":"2023-10-19T13:01:26.786235+0000","last_fullsized":"2023-10-19T13:01:26.786235+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:05:44.654555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:01:24.734993+0000","last_change":"2023-10-19T13:01:24.734993+0000","last_active":"2023-10-19T13:01:24.734993+0000","last_peered":"2023-10-19T13:01:24.734993+0000","last_clean":"2023-10-19T13:01:24.734993+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:01:24.734993+0000","last_undegraded":"2023-10-19T13:01:24.734993+0000","last_fullsized":"2023-10-19T13:01:24.734993+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_clean_scrub_stamp":"2023-10-19T13:01:24.734993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-19T13:01:26.782898+0000","last_change":"2023-10-19T13:01:26.767579+0000","last_active":"2023-10-19T13:01:26.782898+0000","last_peered":"2023-10-19T13:01:26.782898+0000","last_clean":"2023-10-19T13:01:26.782898+0000","last_became_active":"2023-10-19T13:01:26.767330+0000","last_became_peered":"2023-10-19T13:01:26.767330+0000","last_unstale":"2023-10-19T13:01:26.782898+0000","last_undegraded":"2023-10-19T13:01:26.782898+0000","last_fullsized":"2023-10-19T13:01:26.782898+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:31:58.385639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:01:24.734993+0000","last_change":"2023-10-19T13:01:24.734993+0000","last_active":"2023-10-19T13:01:24.734993+0000","last_peered":"2023-10-19T13:01:24.734993+0000","last_clean":"2023-10-19T13:01:24.734993+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:01:24.734993+0000","last_undegraded":"2023-10-19T13:01:24.734993+0000","last_fullsized":"2023-10-19T13:01:24.734993+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_clean_scrub_stamp":"2023-10-19T13:01:24.734993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:01:25.747321+0000","last_change":"2023-10-19T13:01:25.747321+0000","last_active":"2023-10-19T13:01:25.747321+0000","last_peered":"2023-10-19T13:01:25.747321+0000","last_clean":"2023-10-19T13:01:25.747321+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:01:25.747321+0000","last_undegraded":"2023-10-19T13:01:25.747321+0000","last_fullsized":"2023-10-19T13:01:25.747321+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:01:24.734993+0000","last_change":"2023-10-19T13:01:24.734993+0000","last_active":"2023-10-19T13:01:24.734993+0000","last_peered":"2023-10-19T13:01:24.734993+0000","last_clean":"2023-10-19T13:01:24.734993+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:01:24.734993+0000","last_undegraded":"2023-10-19T13:01:24.734993+0000","last_fullsized":"2023-10-19T13:01:24.734993+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_clean_scrub_stamp":"2023-10-19T13:01:24.734993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-19T13:01:26.785846+0000","last_change":"2023-10-19T13:01:26.771109+0000","last_active":"2023-10-19T13:01:26.785846+0000","last_peered":"2023-10-19T13:01:26.785846+0000","last_clean":"2023-10-19T13:01:26.785846+0000","last_became_active":"2023-10-19T13:01:26.768995+0000","last_became_peered":"2023-10-19T13:01:26.768995+0000","last_unstale":"2023-10-19T13:01:26.785846+0000","last_undegraded":"2023-10-19T13:01:26.785846+0000","last_fullsized":"2023-10-19T13:01:26.785846+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:26:39.188174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:01:24.734993+0000","last_change":"2023-10-19T13:01:24.734993+0000","last_active":"2023-10-19T13:01:24.734993+0000","last_peered":"2023-10-19T13:01:24.734993+0000","last_clean":"2023-10-19T13:01:24.734993+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:01:24.734993+0000","last_undegraded":"2023-10-19T13:01:24.734993+0000","last_fullsized":"2023-10-19T13:01:24.734993+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_clean_scrub_stamp":"2023-10-19T13:01:24.734993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-19T13:01:27.772769+0000","last_change":"2023-10-19T13:01:26.771485+0000","last_active":"2023-10-19T13:01:27.772769+0000","last_peered":"2023-10-19T13:01:27.772769+0000","last_clean":"2023-10-19T13:01:27.772769+0000","last_became_active":"2023-10-19T13:01:26.771235+0000","last_became_peered":"2023-10-19T13:01:26.771235+0000","last_unstale":"2023-10-19T13:01:27.772769+0000","last_undegraded":"2023-10-19T13:01:27.772769+0000","last_fullsized":"2023-10-19T13:01:27.772769+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:14:03.720278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:01:24.734993+0000","last_change":"2023-10-19T13:01:24.734993+0000","last_active":"2023-10-19T13:01:24.734993+0000","last_peered":"2023-10-19T13:01:24.734993+0000","last_clean":"2023-10-19T13:01:24.734993+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:01:24.734993+0000","last_undegraded":"2023-10-19T13:01:24.734993+0000","last_fullsized":"2023-10-19T13:01:24.734993+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_clean_scrub_stamp":"2023-10-19T13:01:24.734993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:01:25.747321+0000","last_change":"2023-10-19T13:01:25.747321+0000","last_active":"2023-10-19T13:01:25.747321+0000","last_peered":"2023-10-19T13:01:25.747321+0000","last_clean":"2023-10-19T13:01:25.747321+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:01:25.747321+0000","last_undegraded":"2023-10-19T13:01:25.747321+0000","last_fullsized":"2023-10-19T13:01:25.747321+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.1a","version":"112'1","reported_seq":13,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-19T13:01:27.773314+0000","last_change":"2023-10-19T13:01:25.764864+0000","last_active":"2023-10-19T13:01:27.773314+0000","last_peered":"2023-10-19T13:01:27.773314+0000","last_clean":"2023-10-19T13:01:27.773314+0000","last_became_active":"2023-10-19T13:01:25.764586+0000","last_became_peered":"2023-10-19T13:01:25.764586+0000","last_unstale":"2023-10-19T13:01:27.773314+0000","last_undegraded":"2023-10-19T13:01:27.773314+0000","last_fullsized":"2023-10-19T13:01:27.773314+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_clean_scrub_stamp":"2023-10-19T13:01:24.734993+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:25:39.517420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-19T13:01:26.783708+0000","last_change":"2023-10-19T13:01:26.771422+0000","last_active":"2023-10-19T13:01:26.783708+0000","last_peered":"2023-10-19T13:01:26.783708+0000","last_clean":"2023-10-19T13:01:26.783708+0000","last_became_active":"2023-10-19T13:01:26.770660+0000","last_became_peered":"2023-10-19T13:01:26.770660+0000","last_unstale":"2023-10-19T13:01:26.783708+0000","last_undegraded":"2023-10-19T13:01:26.783708+0000","last_fullsized":"2023-10-19T13:01:26.783708+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:43:04.195600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:01:24.734993+0000","last_change":"2023-10-19T13:01:24.734993+0000","last_active":"2023-10-19T13:01:24.734993+0000","last_peered":"2023-10-19T13:01:24.734993+0000","last_clean":"2023-10-19T13:01:24.734993+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:01:24.734993+0000","last_undegraded":"2023-10-19T13:01:24.734993+0000","last_fullsized":"2023-10-19T13:01:24.734993+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_clean_scrub_stamp":"2023-10-19T13:01:24.734993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-19T13:01:27.773049+0000","last_change":"2023-10-19T13:01:26.775878+0000","last_active":"2023-10-19T13:01:27.773049+0000","last_peered":"2023-10-19T13:01:27.773049+0000","last_clean":"2023-10-19T13:01:27.773049+0000","last_became_active":"2023-10-19T13:01:26.775535+0000","last_became_peered":"2023-10-19T13:01:26.775535+0000","last_unstale":"2023-10-19T13:01:27.773049+0000","last_undegraded":"2023-10-19T13:01:27.773049+0000","last_fullsized":"2023-10-19T13:01:27.773049+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:56:23.564267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:01:24.734993+0000","last_change":"2023-10-19T13:01:24.734993+0000","last_active":"2023-10-19T13:01:24.734993+0000","last_peered":"2023-10-19T13:01:24.734993+0000","last_clean":"2023-10-19T13:01:24.734993+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:01:24.734993+0000","last_undegraded":"2023-10-19T13:01:24.734993+0000","last_fullsized":"2023-10-19T13:01:24.734993+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_clean_scrub_stamp":"2023-10-19T13:01:24.734993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:01:25.747321+0000","last_change":"2023-10-19T13:01:25.747321+0000","last_active":"2023-10-19T13:01:25.747321+0000","last_peered":"2023-10-19T13:01:25.747321+0000","last_clean":"2023-10-19T13:01:25.747321+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:01:25.747321+0000","last_undegraded":"2023-10-19T13:01:25.747321+0000","last_fullsized":"2023-10-19T13:01:25.747321+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:01:24.734993+0000","last_change":"2023-10-19T13:01:24.734993+0000","last_active":"2023-10-19T13:01:24.734993+0000","last_peered":"2023-10-19T13:01:24.734993+0000","last_clean":"2023-10-19T13:01:24.734993+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:01:24.734993+0000","last_undegraded":"2023-10-19T13:01:24.734993+0000","last_fullsized":"2023-10-19T13:01:24.734993+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_clean_scrub_stamp":"2023-10-19T13:01:24.734993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:01:25.747321+0000","last_change":"2023-10-19T13:01:25.747321+0000","last_active":"2023-10-19T13:01:25.747321+0000","last_peered":"2023-10-19T13:01:25.747321+0000","last_clean":"2023-10-19T13:01:25.747321+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:01:25.747321+0000","last_undegraded":"2023-10-19T13:01:25.747321+0000","last_fullsized":"2023-10-19T13:01:25.747321+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:01:24.734993+0000","last_change":"2023-10-19T13:01:24.734993+0000","last_active":"2023-10-19T13:01:24.734993+0000","last_peered":"2023-10-19T13:01:24.734993+0000","last_clean":"2023-10-19T13:01:24.734993+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:01:24.734993+0000","last_undegraded":"2023-10-19T13:01:24.734993+0000","last_fullsized":"2023-10-19T13:01:24.734993+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_clean_scrub_stamp":"2023-10-19T13:01:24.734993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:01:25.747321+0000","last_change":"2023-10-19T13:01:25.747321+0000","last_active":"2023-10-19T13:01:25.747321+0000","last_peered":"2023-10-19T13:01:25.747321+0000","last_clean":"2023-10-19T13:01:25.747321+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:01:25.747321+0000","last_undegraded":"2023-10-19T13:01:25.747321+0000","last_fullsized":"2023-10-19T13:01:25.747321+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:01:24.734993+0000","last_change":"2023-10-19T13:01:24.734993+0000","last_active":"2023-10-19T13:01:24.734993+0000","last_peered":"2023-10-19T13:01:24.734993+0000","last_clean":"2023-10-19T13:01:24.734993+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:01:24.734993+0000","last_undegraded":"2023-10-19T13:01:24.734993+0000","last_fullsized":"2023-10-19T13:01:24.734993+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_clean_scrub_stamp":"2023-10-19T13:01:24.734993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-19T13:01:27.773011+0000","last_change":"2023-10-19T13:01:26.776514+0000","last_active":"2023-10-19T13:01:27.773011+0000","last_peered":"2023-10-19T13:01:27.773011+0000","last_clean":"2023-10-19T13:01:27.773011+0000","last_became_active":"2023-10-19T13:01:26.775394+0000","last_became_peered":"2023-10-19T13:01:26.775394+0000","last_unstale":"2023-10-19T13:01:27.773011+0000","last_undegraded":"2023-10-19T13:01:27.773011+0000","last_fullsized":"2023-10-19T13:01:27.773011+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:09:50.788972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.14","version":"112'6","reported_seq":17,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-19T13:01:26.788973+0000","last_change":"2023-10-19T13:01:25.772773+0000","last_active":"2023-10-19T13:01:26.788973+0000","last_peered":"2023-10-19T13:01:26.788973+0000","last_clean":"2023-10-19T13:01:26.788973+0000","last_became_active":"2023-10-19T13:01:25.772300+0000","last_became_peered":"2023-10-19T13:01:25.772300+0000","last_unstale":"2023-10-19T13:01:26.788973+0000","last_undegraded":"2023-10-19T13:01:26.788973+0000","last_fullsized":"2023-10-19T13:01:26.788973+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_clean_scrub_stamp":"2023-10-19T13:01:24.734993+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:42:04.249097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-19T13:01:27.773253+0000","last_change":"2023-10-19T13:01:26.766977+0000","last_active":"2023-10-19T13:01:27.773253+0000","last_peered":"2023-10-19T13:01:27.773253+0000","last_clean":"2023-10-19T13:01:27.773253+0000","last_became_active":"2023-10-19T13:01:26.766753+0000","last_became_peered":"2023-10-19T13:01:26.766753+0000","last_unstale":"2023-10-19T13:01:27.773253+0000","last_undegraded":"2023-10-19T13:01:27.773253+0000","last_fullsized":"2023-10-19T13:01:27.773253+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:48:14.520433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:01:24.734993+0000","last_change":"2023-10-19T13:01:24.734993+0000","last_active":"2023-10-19T13:01:24.734993+0000","last_peered":"2023-10-19T13:01:24.734993+0000","last_clean":"2023-10-19T13:01:24.734993+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:01:24.734993+0000","last_undegraded":"2023-10-19T13:01:24.734993+0000","last_fullsized":"2023-10-19T13:01:24.734993+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_clean_scrub_stamp":"2023-10-19T13:01:24.734993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:01:25.747321+0000","last_change":"2023-10-19T13:01:25.747321+0000","last_active":"2023-10-19T13:01:25.747321+0000","last_peered":"2023-10-19T13:01:25.747321+0000","last_clean":"2023-10-19T13:01:25.747321+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:01:25.747321+0000","last_undegraded":"2023-10-19T13:01:25.747321+0000","last_fullsized":"2023-10-19T13:01:25.747321+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-19T13:01:26.757234+0000","last_change":"2023-10-19T13:01:25.773615+0000","last_active":"2023-10-19T13:01:26.757234+0000","last_peered":"2023-10-19T13:01:26.757234+0000","last_clean":"2023-10-19T13:01:26.757234+0000","last_became_active":"2023-10-19T13:01:25.773260+0000","last_became_peered":"2023-10-19T13:01:25.773260+0000","last_unstale":"2023-10-19T13:01:26.757234+0000","last_undegraded":"2023-10-19T13:01:26.757234+0000","last_fullsized":"2023-10-19T13:01:26.757234+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_clean_scrub_stamp":"2023-10-19T13:01:24.734993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:20:43.498132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:01:25.747321+0000","last_change":"2023-10-19T13:01:25.747321+0000","last_active":"2023-10-19T13:01:25.747321+0000","last_peered":"2023-10-19T13:01:25.747321+0000","last_clean":"2023-10-19T13:01:25.747321+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:01:25.747321+0000","last_undegraded":"2023-10-19T13:01:25.747321+0000","last_fullsized":"2023-10-19T13:01:25.747321+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":12,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-19T13:01:27.772870+0000","last_change":"2023-10-19T13:01:25.769579+0000","last_active":"2023-10-19T13:01:27.772870+0000","last_peered":"2023-10-19T13:01:27.772870+0000","last_clean":"2023-10-19T13:01:27.772870+0000","last_became_active":"2023-10-19T13:01:25.769328+0000","last_became_peered":"2023-10-19T13:01:25.769328+0000","last_unstale":"2023-10-19T13:01:27.772870+0000","last_undegraded":"2023-10-19T13:01:27.772870+0000","last_fullsized":"2023-10-19T13:01:27.772870+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_clean_scrub_stamp":"2023-10-19T13:01:24.734993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:33:16.385580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:01:25.747321+0000","last_change":"2023-10-19T13:01:25.747321+0000","last_active":"2023-10-19T13:01:25.747321+0000","last_peered":"2023-10-19T13:01:25.747321+0000","last_clean":"2023-10-19T13:01:25.747321+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:01:25.747321+0000","last_undegraded":"2023-10-19T13:01:25.747321+0000","last_fullsized":"2023-10-19T13:01:25.747321+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.10","version":"112'8","reported_seq":20,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-19T13:01:27.772869+0000","last_change":"2023-10-19T13:01:25.764911+0000","last_active":"2023-10-19T13:01:27.772869+0000","last_peered":"2023-10-19T13:01:27.772869+0000","last_clean":"2023-10-19T13:01:27.772869+0000","last_became_active":"2023-10-19T13:01:25.764640+0000","last_became_peered":"2023-10-19T13:01:25.764640+0000","last_unstale":"2023-10-19T13:01:27.772869+0000","last_undegraded":"2023-10-19T13:01:27.772869+0000","last_fullsized":"2023-10-19T13:01:27.772869+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_clean_scrub_stamp":"2023-10-19T13:01:24.734993+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:12:58.167393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-19T13:01:26.780644+0000","last_change":"2023-10-19T13:01:26.768802+0000","last_active":"2023-10-19T13:01:26.780644+0000","last_peered":"2023-10-19T13:01:26.780644+0000","last_clean":"2023-10-19T13:01:26.780644+0000","last_became_active":"2023-10-19T13:01:26.768409+0000","last_became_peered":"2023-10-19T13:01:26.768409+0000","last_unstale":"2023-10-19T13:01:26.780644+0000","last_undegraded":"2023-10-19T13:01:26.780644+0000","last_fullsized":"2023-10-19T13:01:26.780644+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:58:01.254793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:01:24.734993+0000","last_change":"2023-10-19T13:01:24.734993+0000","last_active":"2023-10-19T13:01:24.734993+0000","last_peered":"2023-10-19T13:01:24.734993+0000","last_clean":"2023-10-19T13:01:24.734993+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:01:24.734993+0000","last_undegraded":"2023-10-19T13:01:24.734993+0000","last_fullsized":"2023-10-19T13:01:24.734993+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_clean_scrub_stamp":"2023-10-19T13:01:24.734993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:01:25.747321+0000","last_change":"2023-10-19T13:01:25.747321+0000","last_active":"2023-10-19T13:01:25.747321+0000","last_peered":"2023-10-19T13:01:25.747321+0000","last_clean":"2023-10-19T13:01:25.747321+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:01:25.747321+0000","last_undegraded":"2023-10-19T13:01:25.747321+0000","last_fullsized":"2023-10-19T13:01:25.747321+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:01:24.734993+0000","last_change":"2023-10-19T13:01:24.734993+0000","last_active":"2023-10-19T13:01:24.734993+0000","last_peered":"2023-10-19T13:01:24.734993+0000","last_clean":"2023-10-19T13:01:24.734993+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:01:24.734993+0000","last_undegraded":"2023-10-19T13:01:24.734993+0000","last_fullsized":"2023-10-19T13:01:24.734993+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_clean_scrub_stamp":"2023-10-19T13:01:24.734993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:01:25.747321+0000","last_change":"2023-10-19T13:01:25.747321+0000","last_active":"2023-10-19T13:01:25.747321+0000","last_peered":"2023-10-19T13:01:25.747321+0000","last_clean":"2023-10-19T13:01:25.747321+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:01:25.747321+0000","last_undegraded":"2023-10-19T13:01:25.747321+0000","last_fullsized":"2023-10-19T13:01:25.747321+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:01:24.734993+0000","last_change":"2023-10-19T13:01:24.734993+0000","last_active":"2023-10-19T13:01:24.734993+0000","last_peered":"2023-10-19T13:01:24.734993+0000","last_clean":"2023-10-19T13:01:24.734993+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:01:24.734993+0000","last_undegraded":"2023-10-19T13:01:24.734993+0000","last_fullsized":"2023-10-19T13:01:24.734993+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_clean_scrub_stamp":"2023-10-19T13:01:24.734993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-19T13:01:26.787238+0000","last_change":"2023-10-19T13:01:26.772645+0000","last_active":"2023-10-19T13:01:26.787238+0000","last_peered":"2023-10-19T13:01:26.787238+0000","last_clean":"2023-10-19T13:01:26.787238+0000","last_became_active":"2023-10-19T13:01:26.767998+0000","last_became_peered":"2023-10-19T13:01:26.767998+0000","last_unstale":"2023-10-19T13:01:26.787238+0000","last_undegraded":"2023-10-19T13:01:26.787238+0000","last_fullsized":"2023-10-19T13:01:26.787238+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:44:55.208778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:01:24.734993+0000","last_change":"2023-10-19T13:01:24.734993+0000","last_active":"2023-10-19T13:01:24.734993+0000","last_peered":"2023-10-19T13:01:24.734993+0000","last_clean":"2023-10-19T13:01:24.734993+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:01:24.734993+0000","last_undegraded":"2023-10-19T13:01:24.734993+0000","last_fullsized":"2023-10-19T13:01:24.734993+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_clean_scrub_stamp":"2023-10-19T13:01:24.734993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:01:25.747321+0000","last_change":"2023-10-19T13:01:25.747321+0000","last_active":"2023-10-19T13:01:25.747321+0000","last_peered":"2023-10-19T13:01:25.747321+0000","last_clean":"2023-10-19T13:01:25.747321+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:01:25.747321+0000","last_undegraded":"2023-10-19T13:01:25.747321+0000","last_fullsized":"2023-10-19T13:01:25.747321+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:01:24.734993+0000","last_change":"2023-10-19T13:01:24.734993+0000","last_active":"2023-10-19T13:01:24.734993+0000","last_peered":"2023-10-19T13:01:24.734993+0000","last_clean":"2023-10-19T13:01:24.734993+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:01:24.734993+0000","last_undegraded":"2023-10-19T13:01:24.734993+0000","last_fullsized":"2023-10-19T13:01:24.734993+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_clean_scrub_stamp":"2023-10-19T13:01:24.734993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:01:25.747321+0000","last_change":"2023-10-19T13:01:25.747321+0000","last_active":"2023-10-19T13:01:25.747321+0000","last_peered":"2023-10-19T13:01:25.747321+0000","last_clean":"2023-10-19T13:01:25.747321+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:01:25.747321+0000","last_undegraded":"2023-10-19T13:01:25.747321+0000","last_fullsized":"2023-10-19T13:01:25.747321+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:01:24.734993+0000","last_change":"2023-10-19T13:01:24.734993+0000","last_active":"2023-10-19T13:01:24.734993+0000","last_peered":"2023-10-19T13:01:24.734993+0000","last_clean":"2023-10-19T13:01:24.734993+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:01:24.734993+0000","last_undegraded":"2023-10-19T13:01:24.734993+0000","last_fullsized":"2023-10-19T13:01:24.734993+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_clean_scrub_stamp":"2023-10-19T13:01:24.734993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-19T13:01:26.778147+0000","last_change":"2023-10-19T13:01:26.766902+0000","last_active":"2023-10-19T13:01:26.778147+0000","last_peered":"2023-10-19T13:01:26.778147+0000","last_clean":"2023-10-19T13:01:26.778147+0000","last_became_active":"2023-10-19T13:01:26.766632+0000","last_became_peered":"2023-10-19T13:01:26.766632+0000","last_unstale":"2023-10-19T13:01:26.778147+0000","last_undegraded":"2023-10-19T13:01:26.778147+0000","last_fullsized":"2023-10-19T13:01:26.778147+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:43:31.922698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:01:24.734993+0000","last_change":"2023-10-19T13:01:24.734993+0000","last_active":"2023-10-19T13:01:24.734993+0000","last_peered":"2023-10-19T13:01:24.734993+0000","last_clean":"2023-10-19T13:01:24.734993+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:01:24.734993+0000","last_undegraded":"2023-10-19T13:01:24.734993+0000","last_fullsized":"2023-10-19T13:01:24.734993+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_clean_scrub_stamp":"2023-10-19T13:01:24.734993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:01:25.747321+0000","last_change":"2023-10-19T13:01:25.747321+0000","last_active":"2023-10-19T13:01:25.747321+0000","last_peered":"2023-10-19T13:01:25.747321+0000","last_clean":"2023-10-19T13:01:25.747321+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:01:25.747321+0000","last_undegraded":"2023-10-19T13:01:25.747321+0000","last_fullsized":"2023-10-19T13:01:25.747321+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:01:24.734993+0000","last_change":"2023-10-19T13:01:24.734993+0000","last_active":"2023-10-19T13:01:24.734993+0000","last_peered":"2023-10-19T13:01:24.734993+0000","last_clean":"2023-10-19T13:01:24.734993+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:01:24.734993+0000","last_undegraded":"2023-10-19T13:01:24.734993+0000","last_fullsized":"2023-10-19T13:01:24.734993+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_clean_scrub_stamp":"2023-10-19T13:01:24.734993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-19T13:01:26.782917+0000","last_change":"2023-10-19T13:01:26.768607+0000","last_active":"2023-10-19T13:01:26.782917+0000","last_peered":"2023-10-19T13:01:26.782917+0000","last_clean":"2023-10-19T13:01:26.782917+0000","last_became_active":"2023-10-19T13:01:26.768359+0000","last_became_peered":"2023-10-19T13:01:26.768359+0000","last_unstale":"2023-10-19T13:01:26.782917+0000","last_undegraded":"2023-10-19T13:01:26.782917+0000","last_fullsized":"2023-10-19T13:01:26.782917+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:23:00.868126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:01:24.734993+0000","last_change":"2023-10-19T13:01:24.734993+0000","last_active":"2023-10-19T13:01:24.734993+0000","last_peered":"2023-10-19T13:01:24.734993+0000","last_clean":"2023-10-19T13:01:24.734993+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:01:24.734993+0000","last_undegraded":"2023-10-19T13:01:24.734993+0000","last_fullsized":"2023-10-19T13:01:24.734993+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_clean_scrub_stamp":"2023-10-19T13:01:24.734993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-19T13:01:26.778249+0000","last_change":"2023-10-19T13:01:26.767051+0000","last_active":"2023-10-19T13:01:26.778249+0000","last_peered":"2023-10-19T13:01:26.778249+0000","last_clean":"2023-10-19T13:01:26.778249+0000","last_became_active":"2023-10-19T13:01:26.766794+0000","last_became_peered":"2023-10-19T13:01:26.766794+0000","last_unstale":"2023-10-19T13:01:26.778249+0000","last_undegraded":"2023-10-19T13:01:26.778249+0000","last_fullsized":"2023-10-19T13:01:26.778249+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:24:50.102865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:01:24.734993+0000","last_change":"2023-10-19T13:01:24.734993+0000","last_active":"2023-10-19T13:01:24.734993+0000","last_peered":"2023-10-19T13:01:24.734993+0000","last_clean":"2023-10-19T13:01:24.734993+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:01:24.734993+0000","last_undegraded":"2023-10-19T13:01:24.734993+0000","last_fullsized":"2023-10-19T13:01:24.734993+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_clean_scrub_stamp":"2023-10-19T13:01:24.734993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:01:25.747321+0000","last_change":"2023-10-19T13:01:25.747321+0000","last_active":"2023-10-19T13:01:25.747321+0000","last_peered":"2023-10-19T13:01:25.747321+0000","last_clean":"2023-10-19T13:01:25.747321+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:01:25.747321+0000","last_undegraded":"2023-10-19T13:01:25.747321+0000","last_fullsized":"2023-10-19T13:01:25.747321+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:01:24.734993+0000","last_change":"2023-10-19T13:01:24.734993+0000","last_active":"2023-10-19T13:01:24.734993+0000","last_peered":"2023-10-19T13:01:24.734993+0000","last_clean":"2023-10-19T13:01:24.734993+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:01:24.734993+0000","last_undegraded":"2023-10-19T13:01:24.734993+0000","last_fullsized":"2023-10-19T13:01:24.734993+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_clean_scrub_stamp":"2023-10-19T13:01:24.734993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:01:25.747321+0000","last_change":"2023-10-19T13:01:25.747321+0000","last_active":"2023-10-19T13:01:25.747321+0000","last_peered":"2023-10-19T13:01:25.747321+0000","last_clean":"2023-10-19T13:01:25.747321+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:01:25.747321+0000","last_undegraded":"2023-10-19T13:01:25.747321+0000","last_fullsized":"2023-10-19T13:01:25.747321+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-19T13:01:26.758217+0000","last_change":"2023-10-19T13:01:25.773851+0000","last_active":"2023-10-19T13:01:26.758217+0000","last_peered":"2023-10-19T13:01:26.758217+0000","last_clean":"2023-10-19T13:01:26.758217+0000","last_became_active":"2023-10-19T13:01:25.773581+0000","last_became_peered":"2023-10-19T13:01:25.773581+0000","last_unstale":"2023-10-19T13:01:26.758217+0000","last_undegraded":"2023-10-19T13:01:26.758217+0000","last_fullsized":"2023-10-19T13:01:26.758217+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_clean_scrub_stamp":"2023-10-19T13:01:24.734993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:34:04.264994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:01:25.747321+0000","last_change":"2023-10-19T13:01:25.747321+0000","last_active":"2023-10-19T13:01:25.747321+0000","last_peered":"2023-10-19T13:01:25.747321+0000","last_clean":"2023-10-19T13:01:25.747321+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:01:25.747321+0000","last_undegraded":"2023-10-19T13:01:25.747321+0000","last_fullsized":"2023-10-19T13:01:25.747321+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:01:24.734993+0000","last_change":"2023-10-19T13:01:24.734993+0000","last_active":"2023-10-19T13:01:24.734993+0000","last_peered":"2023-10-19T13:01:24.734993+0000","last_clean":"2023-10-19T13:01:24.734993+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:01:24.734993+0000","last_undegraded":"2023-10-19T13:01:24.734993+0000","last_fullsized":"2023-10-19T13:01:24.734993+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_clean_scrub_stamp":"2023-10-19T13:01:24.734993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:01:25.747321+0000","last_change":"2023-10-19T13:01:25.747321+0000","last_active":"2023-10-19T13:01:25.747321+0000","last_peered":"2023-10-19T13:01:25.747321+0000","last_clean":"2023-10-19T13:01:25.747321+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:01:25.747321+0000","last_undegraded":"2023-10-19T13:01:25.747321+0000","last_fullsized":"2023-10-19T13:01:25.747321+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:01:24.734993+0000","last_change":"2023-10-19T13:01:24.734993+0000","last_active":"2023-10-19T13:01:24.734993+0000","last_peered":"2023-10-19T13:01:24.734993+0000","last_clean":"2023-10-19T13:01:24.734993+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:01:24.734993+0000","last_undegraded":"2023-10-19T13:01:24.734993+0000","last_fullsized":"2023-10-19T13:01:24.734993+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_clean_scrub_stamp":"2023-10-19T13:01:24.734993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-19T13:01:26.785392+0000","last_change":"2023-10-19T13:01:26.771343+0000","last_active":"2023-10-19T13:01:26.785392+0000","last_peered":"2023-10-19T13:01:26.785392+0000","last_clean":"2023-10-19T13:01:26.785392+0000","last_became_active":"2023-10-19T13:01:26.770786+0000","last_became_peered":"2023-10-19T13:01:26.770786+0000","last_unstale":"2023-10-19T13:01:26.785392+0000","last_undegraded":"2023-10-19T13:01:26.785392+0000","last_fullsized":"2023-10-19T13:01:26.785392+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:46:37.116113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:01:24.734993+0000","last_change":"2023-10-19T13:01:24.734993+0000","last_active":"2023-10-19T13:01:24.734993+0000","last_peered":"2023-10-19T13:01:24.734993+0000","last_clean":"2023-10-19T13:01:24.734993+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:01:24.734993+0000","last_undegraded":"2023-10-19T13:01:24.734993+0000","last_fullsized":"2023-10-19T13:01:24.734993+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_clean_scrub_stamp":"2023-10-19T13:01:24.734993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:01:25.747321+0000","last_change":"2023-10-19T13:01:25.747321+0000","last_active":"2023-10-19T13:01:25.747321+0000","last_peered":"2023-10-19T13:01:25.747321+0000","last_clean":"2023-10-19T13:01:25.747321+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:01:25.747321+0000","last_undegraded":"2023-10-19T13:01:25.747321+0000","last_fullsized":"2023-10-19T13:01:25.747321+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":12,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-19T13:01:27.773340+0000","last_change":"2023-10-19T13:01:25.768820+0000","last_active":"2023-10-19T13:01:27.773340+0000","last_peered":"2023-10-19T13:01:27.773340+0000","last_clean":"2023-10-19T13:01:27.773340+0000","last_became_active":"2023-10-19T13:01:25.768556+0000","last_became_peered":"2023-10-19T13:01:25.768556+0000","last_unstale":"2023-10-19T13:01:27.773340+0000","last_undegraded":"2023-10-19T13:01:27.773340+0000","last_fullsized":"2023-10-19T13:01:27.773340+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_clean_scrub_stamp":"2023-10-19T13:01:24.734993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:01:54.371707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"109'1503","reported_seq":1829,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-19T13:01:23.062548+0000","last_change":"2023-10-19T12:44:08.636613+0000","last_active":"2023-10-19T13:01:23.062548+0000","last_peered":"2023-10-19T13:01:23.062548+0000","last_clean":"2023-10-19T13:01:23.062548+0000","last_became_active":"2023-10-19T12:44:08.635697+0000","last_became_peered":"2023-10-19T12:44:08.635697+0000","last_unstale":"2023-10-19T13:01:23.062548+0000","last_undegraded":"2023-10-19T13:01:23.062548+0000","last_fullsized":"2023-10-19T13:01:23.062548+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_clean_scrub_stamp":"2023-10-19T12:44:07.399738+0000","objects_scrubbed":0,"log_size":1503,"log_dups_size":0,"ondisk_log_size":1503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:15:23.685485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":755,"num_read_kb":1517,"num_write":289,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":32,"stat_sum":{"num_bytes":1612,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15,"ondisk_log_size":15,"up":14,"acting":14,"num_store_stats":2},{"poolid":14,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":48,"acting":48,"num_store_stats":2},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":755,"num_read_kb":1517,"num_write":289,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1503,"ondisk_log_size":1503,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575058,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46908,"kb_used_data":1844,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324932,"statfs":{"total":96636764160,"available":96588730368,"internally_reserved":0,"allocated":1888256,"data_stored":1238805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":46135755},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 13:00:44 2023","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.55200000000000005,"15min":0.56899999999999995},"min":{"1min":0.34100000000000003,"5min":0.30299999999999999,"15min":0.26400000000000001},"max":{"1min":0.74099999999999999,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.58199999999999996,"15min":0.58899999999999997},"min":{"1min":0.36499999999999999,"5min":0.33200000000000002,"15min":0.29599999999999999},"max":{"1min":0.75600000000000001,"5min":0.83799999999999997,"15min":0.93700000000000006},"last":0.58499999999999996}]},{"osd":1,"last update":"Thu Oct 19 13:00:44 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.56599999999999995,"15min":0.57699999999999996},"min":{"1min":0.41299999999999998,"5min":0.40300000000000002,"15min":0.28199999999999997},"max":{"1min":0.68100000000000005,"5min":0.86299999999999999,"15min":1.3129999999999999},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.57599999999999996,"15min":0.58999999999999997},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.32700000000000001},"max":{"1min":0.69299999999999995,"5min":0.82699999999999996,"15min":1.284},"last":0.56799999999999995}]},{"osd":2,"last update":"Thu Oct 19 13:00:44 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.623,"15min":0.63},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.76900000000000002,"5min":0.82699999999999996,"15min":0.98599999999999999},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.61899999999999999,"15min":0.63},"min":{"1min":0.46200000000000002,"5min":0.41599999999999998,"15min":0.28399999999999997},"max":{"1min":0.78300000000000003,"5min":0.81200000000000006,"15min":1.2589999999999999},"last":0.61899999999999999}]},{"osd":3,"last update":"Thu Oct 19 13:00:44 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.58999999999999997,"15min":0.59999999999999998},"min":{"1min":0.39700000000000002,"5min":0.34999999999999998,"15min":0.34399999999999997},"max":{"1min":0.71599999999999997,"5min":0.81200000000000006,"15min":0.96899999999999997},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.621,"5min":0.65400000000000003,"15min":0.65600000000000003},"min":{"1min":0.45500000000000002,"5min":0.45300000000000001,"15min":0.39700000000000002},"max":{"1min":0.79800000000000004,"5min":0.92000000000000004,"15min":1.329},"last":0.629}]},{"osd":4,"last update":"Thu Oct 19 13:00:44 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.47499999999999998,"15min":0.46999999999999997},"min":{"1min":0.315,"5min":0.27400000000000002,"15min":0.25600000000000001},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.80500000000000005},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.47399999999999998,"15min":0.47199999999999998},"min":{"1min":0.30199999999999999,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.67100000000000004,"5min":0.75600000000000001,"15min":0.78900000000000003},"last":0.41499999999999998}]},{"osd":5,"last update":"Thu Oct 19 13:00:44 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.48899999999999999,"15min":0.48499999999999999},"min":{"1min":0.32000000000000001,"5min":0.30499999999999999,"15min":0.29099999999999998},"max":{"1min":0.74099999999999999,"5min":0.75800000000000001,"15min":0.90400000000000003},"last":0.435},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52000000000000002,"15min":0.51700000000000002},"min":{"1min":0.34000000000000002,"5min":0.314,"15min":0.308},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.92200000000000004},"last":0.45400000000000001}]},{"osd":6,"last update":"Thu Oct 19 13:00:44 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.54000000000000004,"15min":0.54800000000000004},"min":{"1min":0.378,"5min":0.34100000000000003,"15min":0.308},"max":{"1min":0.749,"5min":0.81299999999999994,"15min":0.88900000000000001},"last":0.501},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52000000000000002,"15min":0.52800000000000002},"min":{"1min":0.33400000000000002,"5min":0.30299999999999999,"15min":0.27300000000000002},"max":{"1min":0.68300000000000005,"5min":0.83999999999999997,"15min":0.90300000000000002},"last":0.47399999999999998}]}]},{"osd":6,"up_from":13,"seq":55834575058,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32448,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339392,"statfs":{"total":96636764160,"available":96603537408,"internally_reserved":0,"allocated":1294336,"data_stored":648437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 13:00:32 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.57199999999999995,"15min":0.58599999999999997},"min":{"1min":0.373,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.75800000000000001,"5min":0.76000000000000001,"15min":0.81999999999999995},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.57799999999999996,"15min":0.58499999999999996},"min":{"1min":0.36399999999999999,"5min":0.247,"15min":0.247},"max":{"1min":0.75600000000000001,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.58299999999999996}]},{"osd":1,"last update":"Thu Oct 19 13:00:32 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.35299999999999998,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.76000000000000001,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.59699999999999998,"15min":0.59899999999999998},"min":{"1min":0.42099999999999999,"5min":0.34300000000000003,"15min":0.28399999999999997},"max":{"1min":0.749,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.47299999999999998}]},{"osd":2,"last update":"Thu Oct 19 13:00:32 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.57299999999999995,"15min":0.58199999999999996},"min":{"1min":0.36899999999999999,"5min":0.307,"15min":0.26600000000000001},"max":{"1min":0.755,"5min":0.84699999999999998,"15min":0.85699999999999998},"last":0.5},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.61299999999999999,"15min":0.60999999999999999},"min":{"1min":0.38900000000000001,"5min":0.35999999999999999,"15min":0.28399999999999997},"max":{"1min":0.79400000000000004,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.51800000000000002}]},{"osd":3,"last update":"Thu Oct 19 13:00:32 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61899999999999999,"15min":0.624},"min":{"1min":0.38800000000000001,"5min":0.38500000000000001,"15min":0.32600000000000001},"max":{"1min":0.81299999999999994,"5min":0.871,"15min":0.871},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.61899999999999999,"15min":0.624},"min":{"1min":0.40100000000000002,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.74099999999999999,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.60599999999999998}]},{"osd":4,"last update":"Thu Oct 19 13:00:32 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.45900000000000002,"15min":0.46600000000000003},"min":{"1min":0.29099999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.76700000000000002,"5min":0.77900000000000003,"15min":0.79300000000000004},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.47699999999999998,"15min":0.47999999999999998},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.247},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.77600000000000002},"last":0.65400000000000003}]},{"osd":5,"last update":"Thu Oct 19 13:00:32 2023","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.46100000000000002,"15min":0.47099999999999997},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.26600000000000001},"max":{"1min":0.55700000000000005,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.51400000000000001,"15min":0.51600000000000001},"min":{"1min":0.375,"5min":0.32100000000000001,"15min":0.312},"max":{"1min":0.747,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.61699999999999999}]},{"osd":7,"last update":"Thu Oct 19 13:00:32 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.52500000000000002,"15min":0.52100000000000002},"min":{"1min":0.32400000000000001,"5min":0.29299999999999998,"15min":0.28199999999999997},"max":{"1min":0.77600000000000002,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.53500000000000003},"min":{"1min":0.27900000000000003,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.66700000000000004,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.57299999999999995}]}]},{"osd":5,"up_from":13,"seq":55834575058,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":50552,"kb_used_data":1312,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321288,"statfs":{"total":96636764160,"available":96584998912,"internally_reserved":0,"allocated":1343488,"data_stored":671461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 13:00:29 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58799999999999997,"15min":0.59099999999999997},"min":{"1min":0.307,"5min":0.28999999999999998,"15min":0.223},"max":{"1min":0.75900000000000001,"5min":0.79900000000000004,"15min":0.82699999999999996},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58599999999999997,"15min":0.59799999999999998},"min":{"1min":0.41099999999999998,"5min":0.36099999999999999,"15min":0.27800000000000002},"max":{"1min":0.82999999999999996,"5min":0.84299999999999997,"15min":0.874},"last":0.48799999999999999}]},{"osd":1,"last update":"Thu Oct 19 13:00:29 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.57799999999999996,"15min":0.59499999999999997},"min":{"1min":0.40400000000000003,"5min":0.34599999999999997,"15min":0.32800000000000001},"max":{"1min":0.71299999999999997,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.60199999999999998,"15min":0.61199999999999999},"min":{"1min":0.378,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.77200000000000002,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.52700000000000002}]},{"osd":2,"last update":"Thu Oct 19 13:00:29 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.56399999999999995,"15min":0.58499999999999996},"min":{"1min":0.41199999999999998,"5min":0.34100000000000003,"15min":0.307},"max":{"1min":0.71699999999999997,"5min":0.78600000000000003,"15min":0.80700000000000005},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.61199999999999999,"15min":0.624},"min":{"1min":0.47099999999999997,"5min":0.38400000000000001,"15min":0.36599999999999999},"max":{"1min":0.79300000000000004,"5min":0.88100000000000001,"15min":0.93600000000000005},"last":0.51000000000000001}]},{"osd":3,"last update":"Thu Oct 19 13:00:29 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.58899999999999997,"15min":0.60399999999999998},"min":{"1min":0.41699999999999998,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.752,"5min":0.79900000000000004,"15min":0.81299999999999994},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.61499999999999999,"15min":0.63400000000000001},"min":{"1min":0.501,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.73999999999999999,"5min":0.81000000000000005,"15min":0.89500000000000002},"last":0.5}]},{"osd":4,"last update":"Thu Oct 19 13:00:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.47599999999999998,"15min":0.47499999999999998},"min":{"1min":0.35399999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.73599999999999999,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.46800000000000003,"15min":0.46999999999999997},"min":{"1min":0.33900000000000002,"5min":0.251,"15min":0.251},"max":{"1min":0.69199999999999995,"5min":0.75900000000000001,"15min":0.80500000000000005},"last":0.377}]},{"osd":6,"last update":"Thu Oct 19 13:00:29 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.34100000000000003,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.51100000000000001,"15min":0.504},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.33000000000000002},"max":{"1min":0.67200000000000004,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.435}]},{"osd":7,"last update":"Thu Oct 19 13:00:29 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.498,"15min":0.49399999999999999},"min":{"1min":0.39800000000000002,"5min":0.32000000000000001,"15min":0.315},"max":{"1min":0.72399999999999998,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.54900000000000004,"15min":0.53900000000000003},"min":{"1min":0.42599999999999999,"5min":0.311,"15min":0.311},"max":{"1min":0.74099999999999999,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.40899999999999997}]}]},{"osd":4,"up_from":13,"seq":55834575058,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32444,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339396,"statfs":{"total":96636764160,"available":96603541504,"internally_reserved":0,"allocated":1294336,"data_stored":648437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 13:00:43 2023","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.55200000000000005,"15min":0.56000000000000005},"min":{"1min":0.36099999999999999,"5min":0.29499999999999998,"15min":0.28899999999999998},"max":{"1min":0.72299999999999998,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.247},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.54700000000000004,"15min":0.55700000000000005},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.27600000000000002},"max":{"1min":0.70099999999999996,"5min":0.77800000000000002,"15min":0.78200000000000003},"last":0.28199999999999997}]},{"osd":1,"last update":"Thu Oct 19 13:00:43 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.58099999999999996,"15min":0.60699999999999998},"min":{"1min":0.36599999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.72699999999999998,"5min":0.80600000000000005,"15min":0.86199999999999999},"last":0.371},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.60299999999999998,"15min":0.61499999999999999},"min":{"1min":0.433,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.73699999999999999,"5min":0.79900000000000004,"15min":0.873},"last":0.38200000000000001}]},{"osd":2,"last update":"Thu Oct 19 13:00:43 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.59399999999999997,"15min":0.61099999999999999},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.254},"max":{"1min":0.79300000000000004,"5min":0.85299999999999998,"15min":0.86599999999999999},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.59699999999999998,"15min":0.61299999999999999},"min":{"1min":0.38700000000000001,"5min":0.35799999999999998,"15min":0.34499999999999997},"max":{"1min":0.75800000000000001,"5min":0.82999999999999996,"15min":0.85999999999999999},"last":0.435}]},{"osd":3,"last update":"Thu Oct 19 13:00:43 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.57499999999999996,"15min":0.58499999999999996},"min":{"1min":0.36199999999999999,"5min":0.35199999999999998,"15min":0.32400000000000001},"max":{"1min":0.75,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.443},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.63,"15min":0.63700000000000001},"min":{"1min":0.44800000000000001,"5min":0.42599999999999999,"15min":0.37},"max":{"1min":0.76500000000000001,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.45600000000000002}]},{"osd":5,"last update":"Thu Oct 19 13:00:43 2023","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.45100000000000001,"15min":0.44800000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.249},"max":{"1min":0.63200000000000001,"5min":0.79600000000000004,"15min":0.81299999999999994},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.46899999999999997,"15min":0.45900000000000002},"min":{"1min":0.30099999999999999,"5min":0.26400000000000001,"15min":0.23499999999999999},"max":{"1min":0.69499999999999995,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.39900000000000002}]},{"osd":6,"last update":"Thu Oct 19 13:00:43 2023","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.45500000000000002,"15min":0.45400000000000001},"min":{"1min":0.30099999999999999,"5min":0.29499999999999998,"15min":0.26400000000000001},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.78600000000000003},"last":0.316},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.48799999999999999,"15min":0.49399999999999999},"min":{"1min":0.308,"5min":0.308,"15min":0.27100000000000002},"max":{"1min":0.70699999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.41499999999999998}]},{"osd":7,"last update":"Thu Oct 19 13:00:43 2023","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.505,"15min":0.501},"min":{"1min":0.33200000000000002,"5min":0.307,"15min":0.27500000000000002},"max":{"1min":0.72799999999999998,"5min":0.79100000000000004,"15min":0.80200000000000005},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54200000000000004,"15min":0.54400000000000004},"min":{"1min":0.371,"5min":0.35799999999999998,"15min":0.30199999999999999},"max":{"1min":0.70799999999999996,"5min":0.80500000000000005,"15min":0.83499999999999996},"last":0.42399999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575057,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":36608,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335232,"statfs":{"total":96636764160,"available":96599277568,"internally_reserved":0,"allocated":1294336,"data_stored":648437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 13:00:27 2023","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.42199999999999999,"15min":0.40000000000000002},"min":{"1min":0.252,"5min":0.23599999999999999,"15min":0.20899999999999999},"max":{"1min":0.60399999999999998,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.41799999999999998,"15min":0.41599999999999998},"min":{"1min":0.28699999999999998,"5min":0.223,"15min":0.223},"max":{"1min":0.59899999999999998,"5min":0.75,"15min":0.75},"last":0.20399999999999999}]},{"osd":1,"last update":"Thu Oct 19 13:00:27 2023","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.41999999999999998,"15min":0.41199999999999998},"min":{"1min":0.26900000000000002,"5min":0.26100000000000001,"15min":0.246},"max":{"1min":0.628,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.314},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.48599999999999999,"15min":0.45900000000000002},"min":{"1min":0.30099999999999999,"5min":0.24099999999999999,"15min":0.215},"max":{"1min":0.64200000000000002,"5min":2.9399999999999999,"15min":2.9399999999999999},"last":0.28699999999999998}]},{"osd":2,"last update":"Thu Oct 19 13:00:27 2023","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.47099999999999997,"15min":0.46800000000000003},"min":{"1min":0.317,"5min":0.314,"15min":0.26100000000000001},"max":{"1min":0.62,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.496,"5min":0.48299999999999998,"15min":0.48699999999999999},"min":{"1min":0.34499999999999997,"5min":0.29999999999999999,"15min":0.247},"max":{"1min":0.66900000000000004,"5min":0.754,"15min":0.754},"last":0.35699999999999998}]},{"osd":4,"last update":"Thu Oct 19 13:00:27 2023","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.66000000000000003,"15min":0.66700000000000004},"min":{"1min":0.44700000000000001,"5min":0.42299999999999999,"15min":0.40300000000000002},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":1.0069999999999999},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.68799999999999994,"15min":0.69199999999999995},"min":{"1min":0.48899999999999999,"5min":0.46899999999999997,"15min":0.42099999999999999},"max":{"1min":0.91400000000000003,"5min":0.93000000000000005,"15min":0.99399999999999999},"last":0.40699999999999997}]},{"osd":5,"last update":"Thu Oct 19 13:00:27 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.68500000000000005,"15min":0.66000000000000003},"min":{"1min":0.47499999999999998,"5min":0.45500000000000002,"15min":0.41099999999999998},"max":{"1min":0.90600000000000003,"5min":2.9990000000000001,"15min":2.9990000000000001},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.65000000000000002,"15min":0.65100000000000002},"min":{"1min":0.47699999999999998,"5min":0.40899999999999997,"15min":0.35899999999999999},"max":{"1min":0.873,"5min":1.774,"15min":1.774},"last":0.50600000000000001}]},{"osd":6,"last update":"Thu Oct 19 13:00:27 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.66300000000000003,"15min":0.65400000000000003},"min":{"1min":0.503,"5min":0.433,"15min":0.41199999999999998},"max":{"1min":0.84499999999999997,"5min":0.93700000000000006,"15min":0.96799999999999997},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.68000000000000005,"15min":0.67600000000000005},"min":{"1min":0.501,"5min":0.377,"15min":0.377},"max":{"1min":0.94399999999999995,"5min":1.167,"15min":1.167},"last":0.51800000000000002}]},{"osd":7,"last update":"Thu Oct 19 13:00:27 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.65900000000000003,"15min":0.66500000000000004},"min":{"1min":0.47999999999999998,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.90700000000000003,"5min":1.042,"15min":1.042},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.69499999999999995,"15min":0.68100000000000005},"min":{"1min":0.51100000000000001,"5min":0.432,"15min":0.432},"max":{"1min":0.88300000000000001,"5min":2.8149999999999999,"15min":2.8149999999999999},"last":0.45300000000000001}]}]},{"osd":2,"up_from":13,"seq":55834575059,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":54656,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317184,"statfs":{"total":96636764160,"available":96580796416,"internally_reserved":0,"allocated":1294336,"data_stored":648437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 13:00:37 2023","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.36899999999999999,"15min":0.378},"min":{"1min":0.215,"5min":0.20899999999999999,"15min":0.187},"max":{"1min":0.52900000000000003,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.433},{"interface":"front","average":{"1min":0.374,"5min":0.39700000000000002,"15min":0.40699999999999997},"min":{"1min":0.28000000000000003,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.56000000000000005,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.27400000000000002}]},{"osd":1,"last update":"Thu Oct 19 13:00:37 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.38200000000000001,"15min":0.39600000000000002},"min":{"1min":0.26100000000000001,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.51100000000000001,"5min":0.63900000000000001,"15min":0.71099999999999997},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.41099999999999998,"15min":0.42799999999999999},"min":{"1min":0.26900000000000002,"5min":0.22,"15min":0.22},"max":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.68100000000000005},"last":0.33200000000000002}]},{"osd":3,"last update":"Thu Oct 19 13:00:37 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.45300000000000001,"15min":0.45500000000000002},"min":{"1min":0.33700000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.50700000000000001,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.439,"5min":0.46899999999999997,"15min":0.48199999999999998},"min":{"1min":0.34100000000000003,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.57499999999999996,"5min":0.77800000000000002,"15min":0.82999999999999996},"last":0.41499999999999998}]},{"osd":4,"last update":"Thu Oct 19 13:00:37 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.61899999999999999,"15min":0.64000000000000001},"min":{"1min":0.40100000000000002,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.68700000000000006,"5min":0.93999999999999995,"15min":0.98999999999999999},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.63400000000000001,"15min":0.65500000000000003},"min":{"1min":0.42999999999999999,"5min":0.35099999999999998,"15min":0.32000000000000001},"max":{"1min":0.78000000000000003,"5min":0.91200000000000003,"15min":1.046},"last":0.55700000000000005}]},{"osd":5,"last update":"Thu Oct 19 13:00:37 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.65000000000000002,"15min":0.66100000000000003},"min":{"1min":0.46899999999999997,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.73999999999999999,"5min":0.96499999999999997,"15min":0.99399999999999999},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.63200000000000001,"15min":0.65600000000000003},"min":{"1min":0.39100000000000001,"5min":0.375,"15min":0.375},"max":{"1min":0.753,"5min":0.90200000000000002,"15min":0.95799999999999996},"last":0.47599999999999998}]},{"osd":6,"last update":"Thu Oct 19 13:00:37 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.63700000000000001,"15min":0.66300000000000003},"min":{"1min":0.48799999999999999,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.73599999999999999,"5min":1.008,"15min":1.008},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.65600000000000003,"15min":0.67400000000000004},"min":{"1min":0.46300000000000002,"5min":0.371,"15min":0.371},"max":{"1min":0.80500000000000005,"5min":0.995,"15min":0.995},"last":0.50600000000000001}]},{"osd":7,"last update":"Thu Oct 19 13:00:37 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.65400000000000003,"15min":0.68000000000000005},"min":{"1min":0.39800000000000002,"5min":0.39300000000000002,"15min":0.36399999999999999},"max":{"1min":0.71799999999999997,"5min":0.98999999999999999,"15min":1.0149999999999999},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.66800000000000004,"15min":0.68100000000000005},"min":{"1min":0.47199999999999998,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.80000000000000004,"5min":1.02,"15min":1.0329999999999999},"last":0.54500000000000004}]}]},{"osd":1,"up_from":13,"seq":55834575059,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32544,"kb_used_data":1364,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339296,"statfs":{"total":96636764160,"available":96603439104,"internally_reserved":0,"allocated":1396736,"data_stored":698090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 13:00:36 2023","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.39000000000000001,"15min":0.38600000000000001},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.187},"max":{"1min":0.57799999999999996,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.20399999999999999},{"interface":"front","average":{"1min":0.438,"5min":0.441,"15min":0.43099999999999999},"min":{"1min":0.217,"5min":0.217,"15min":0.21199999999999999},"max":{"1min":0.65800000000000003,"5min":2.5800000000000001,"15min":2.5800000000000001},"last":0.34599999999999997}]},{"osd":2,"last update":"Thu Oct 19 13:00:36 2023","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.41799999999999998,"15min":0.42199999999999999},"min":{"1min":0.24199999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.69999999999999996},"last":0.245},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.47599999999999998,"15min":0.45500000000000002},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.23699999999999999},"max":{"1min":0.75600000000000001,"5min":2.6230000000000002,"15min":2.6230000000000002},"last":0.27800000000000002}]},{"osd":3,"last update":"Thu Oct 19 13:00:36 2023","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.50700000000000001,"15min":0.48499999999999999},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.26200000000000001},"max":{"1min":0.69599999999999995,"5min":3.9740000000000002,"15min":3.9740000000000002},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.43099999999999999,"15min":0.42199999999999999},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27000000000000002},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.80500000000000005},"last":0.30199999999999999}]},{"osd":4,"last update":"Thu Oct 19 13:00:36 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.44900000000000001,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.86899999999999999,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":1.0409999999999999},{"interface":"front","average":{"1min":0.627,"5min":0.66100000000000003,"15min":0.64500000000000002},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.314},"max":{"1min":0.86899999999999999,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.55300000000000005}]},{"osd":5,"last update":"Thu Oct 19 13:00:36 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.65300000000000002,"15min":0.65300000000000002},"min":{"1min":0.55300000000000005,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.85299999999999998,"5min":1.1200000000000001,"15min":1.1200000000000001},"last":1.0549999999999999},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.69499999999999995,"15min":0.67300000000000004},"min":{"1min":0.436,"5min":0.42299999999999999,"15min":0.38700000000000001},"max":{"1min":0.84499999999999997,"5min":2.7120000000000002,"15min":2.7120000000000002},"last":1.1020000000000001}]},{"osd":6,"last update":"Thu Oct 19 13:00:36 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.64600000000000002,"15min":0.65400000000000003},"min":{"1min":0.46999999999999997,"5min":0.36799999999999999,"15min":0.36399999999999999},"max":{"1min":0.83799999999999997,"5min":0.92200000000000004,"15min":1.0069999999999999},"last":0.5},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.67800000000000005,"15min":0.67700000000000005},"min":{"1min":0.47299999999999998,"5min":0.376,"15min":0.376},"max":{"1min":1.129,"5min":1.129,"15min":1.129},"last":0.57699999999999996}]},{"osd":7,"last update":"Thu Oct 19 13:00:36 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.69199999999999995,"15min":0.67500000000000004},"min":{"1min":0.52900000000000003,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.80300000000000005,"5min":2.786,"15min":2.786},"last":1.1439999999999999},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.67400000000000004,"15min":0.67600000000000005},"min":{"1min":0.45200000000000001,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.79500000000000004,"5min":0.94199999999999995,"15min":1.034},"last":0.52600000000000002}]}]},{"osd":0,"up_from":13,"seq":55834575060,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28968,"kb_used_data":1948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342872,"statfs":{"total":96636764160,"available":96607100928,"internally_reserved":0,"allocated":1994752,"data_stored":1283439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Oct 19 13:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.38700000000000001,"15min":0.39000000000000001},"min":{"1min":0.20899999999999999,"5min":0.182,"15min":0.182},"max":{"1min":0.52800000000000002,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.372,"5min":0.42199999999999999,"15min":0.43099999999999999},"min":{"1min":0.248,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.58999999999999997,"5min":0.67600000000000005,"15min":0.746},"last":0.27700000000000002}]},{"osd":2,"last update":"Thu Oct 19 13:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.45100000000000001,"15min":0.45300000000000001},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.25800000000000001},"max":{"1min":0.64100000000000001,"5min":0.752,"15min":0.76300000000000001},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.42499999999999999,"15min":0.434},"min":{"1min":0.246,"5min":0.216,"15min":0.20699999999999999},"max":{"1min":0.623,"5min":0.72999999999999998,"15min":0.76800000000000002},"last":0.29999999999999999}]},{"osd":3,"last update":"Thu Oct 19 13:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.40999999999999998,"15min":0.40899999999999997},"min":{"1min":0.26600000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.56599999999999995,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.311},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.318,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.67500000000000004,"5min":1.4990000000000001,"15min":1.4990000000000001},"last":0.34699999999999998}]},{"osd":4,"last update":"Thu Oct 19 13:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.61399999999999999,"15min":0.63300000000000001},"min":{"1min":0.40300000000000002,"5min":0.40000000000000002,"15min":0.34999999999999998},"max":{"1min":0.71299999999999997,"5min":1.361,"15min":1.361},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.64500000000000002,"15min":0.64900000000000002},"min":{"1min":0.42899999999999999,"5min":0.41899999999999998,"15min":0.35999999999999999},"max":{"1min":0.80900000000000005,"5min":0.98999999999999999,"15min":0.998},"last":0.66500000000000004}]},{"osd":5,"last update":"Thu Oct 19 13:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.67900000000000005,"15min":0.68100000000000005},"min":{"1min":0.49199999999999999,"5min":0.378,"15min":0.378},"max":{"1min":0.90000000000000002,"5min":1.325,"15min":1.325},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.62,"5min":0.67900000000000005,"15min":0.68300000000000005},"min":{"1min":0.437,"5min":0.36299999999999999,"15min":0.318},"max":{"1min":0.88,"5min":1.024,"15min":1.024},"last":0.61499999999999999}]},{"osd":6,"last update":"Thu Oct 19 13:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.65600000000000003,"15min":0.66200000000000003},"min":{"1min":0.42499999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.79300000000000004,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.65200000000000002,"15min":0.66200000000000003},"min":{"1min":0.45300000000000001,"5min":0.41999999999999998,"15min":0.38900000000000001},"max":{"1min":0.73499999999999999,"5min":0.995,"15min":1.089},"last":0.70299999999999996}]},{"osd":7,"last update":"Thu Oct 19 13:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.66200000000000003,"15min":0.67900000000000005},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.82799999999999996,"5min":0.98799999999999999,"15min":1.0089999999999999},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.65900000000000003,"15min":0.66100000000000003},"min":{"1min":0.442,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.94499999999999995,"5min":1.389,"15min":1.389},"last":0.68100000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-19T13:01:29.787 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-19T13:01:29.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.930+0000 7f2c75583640 1 -- 172.21.15.57:0/2428688961 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2c58005180 msgr2=0x7f2c5800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:29.934 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.930+0000 7f2c75583640 1 --2- 172.21.15.57:0/2428688961 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2c58005180 0x7f2c5800b980 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7f2c6800c640 tx=0x7f2c580057f0 comp rx=0 tx=0).stop 2023-10-19T13:01:29.934 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.930+0000 7f2c75583640 1 -- 172.21.15.57:0/2428688961 shutdown_connections 2023-10-19T13:01:29.934 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.930+0000 7f2c75583640 1 --2- 172.21.15.57:0/2428688961 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2c58005180 0x7f2c5800b980 unknown :-1 s=CLOSED pgs=1291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:29.934 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.930+0000 7f2c75583640 1 -- 172.21.15.57:0/2428688961 >> 172.21.15.57:0/2428688961 conn(0x7f2c700b3f00 msgr2=0x7f2c700b4310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:29.934 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.930+0000 7f2c75583640 1 -- 172.21.15.57:0/2428688961 shutdown_connections 2023-10-19T13:01:29.935 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.930+0000 7f2c75583640 1 -- 172.21.15.57:0/2428688961 wait complete. 2023-10-19T13:01:29.937 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.932+0000 7f2c75583640 1 Processor -- start 2023-10-19T13:01:29.937 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.932+0000 7f2c75583640 1 -- start start 2023-10-19T13:01:29.937 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.932+0000 7f2c75583640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2c58005180 0x7f2c7014c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:29.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.932+0000 7f2c75583640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2c7014e700 0x7f2c70154fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:29.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.932+0000 7f2c75583640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2c7014eb80 0x7f2c701554e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:29.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.932+0000 7f2c75583640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c7014cdb0 con 0x7f2c7014eb80 2023-10-19T13:01:29.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.932+0000 7f2c75583640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c700c7e00 con 0x7f2c7014e700 2023-10-19T13:01:29.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.932+0000 7f2c75583640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c7006b610 con 0x7f2c58005180 2023-10-19T13:01:29.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.933+0000 7f2c6f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2c7014eb80 0x7f2c701554e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:29.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.933+0000 7f2c6f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2c7014eb80 0x7f2c701554e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40130/0 (socket says 172.21.15.57:40130) 2023-10-19T13:01:29.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.933+0000 7f2c6f7fe640 1 -- 172.21.15.57:0/49958404 learned_addr learned my addr 172.21.15.57:0/49958404 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:01:29.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.933+0000 7f2c6effd640 1 --2- 172.21.15.57:0/49958404 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2c58005180 0x7f2c7014c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:29.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.933+0000 7f2c6e7fc640 1 --2- 172.21.15.57:0/49958404 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2c7014e700 0x7f2c70154fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:29.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.933+0000 7f2c6f7fe640 1 -- 172.21.15.57:0/49958404 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2c58005180 msgr2=0x7f2c7014c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:29.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.933+0000 7f2c6f7fe640 1 --2- 172.21.15.57:0/49958404 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2c58005180 0x7f2c7014c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:29.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.933+0000 7f2c6f7fe640 1 -- 172.21.15.57:0/49958404 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2c7014e700 msgr2=0x7f2c70154fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:29.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.933+0000 7f2c6f7fe640 1 --2- 172.21.15.57:0/49958404 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2c7014e700 0x7f2c70154fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:29.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.933+0000 7f2c6f7fe640 1 -- 172.21.15.57:0/49958404 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c700c80a0 con 0x7f2c7014eb80 2023-10-19T13:01:29.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.933+0000 7f2c6f7fe640 1 --2- 172.21.15.57:0/49958404 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2c7014eb80 0x7f2c701554e0 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f2c6800c760 tx=0x7f2c680012e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:29.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.933+0000 7f2c577fe640 1 -- 172.21.15.57:0/49958404 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c680018b0 con 0x7f2c7014eb80 2023-10-19T13:01:29.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.933+0000 7f2c577fe640 1 -- 172.21.15.57:0/49958404 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f2c68003970 con 0x7f2c7014eb80 2023-10-19T13:01:29.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.933+0000 7f2c75583640 1 -- 172.21.15.57:0/49958404 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c700c8380 con 0x7f2c7014eb80 2023-10-19T13:01:29.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.933+0000 7f2c577fe640 1 -- 172.21.15.57:0/49958404 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c68003d60 con 0x7f2c7014eb80 2023-10-19T13:01:29.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.933+0000 7f2c75583640 1 -- 172.21.15.57:0/49958404 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c700c88a0 con 0x7f2c7014eb80 2023-10-19T13:01:29.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.934+0000 7f2c75583640 1 -- 172.21.15.57:0/49958404 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c700b42e0 con 0x7f2c7014eb80 2023-10-19T13:01:29.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.935+0000 7f2c577fe640 1 -- 172.21.15.57:0/49958404 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 47) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f2c68031030 con 0x7f2c7014eb80 2023-10-19T13:01:29.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.935+0000 7f2c577fe640 1 --2- 172.21.15.57:0/49958404 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f2c5c06aed0 0x7f2c5c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:29.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.935+0000 7f2c577fe640 1 -- 172.21.15.57:0/49958404 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f2c680aa1d0 con 0x7f2c7014eb80 2023-10-19T13:01:29.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.935+0000 7f2c6effd640 1 --2- 172.21.15.57:0/49958404 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f2c5c06aed0 0x7f2c5c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:29.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.936+0000 7f2c6effd640 1 --2- 172.21.15.57:0/49958404 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f2c5c06aed0 0x7f2c5c06d390 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2c701544a0 tx=0x7f2c60005f90 comp rx=0 tx=0).ready entity=mgr.7072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:29.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:29.940+0000 7f2c577fe640 1 -- 172.21.15.57:0/49958404 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2c68075020 con 0x7f2c7014eb80 2023-10-19T13:01:30.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.118+0000 7f2c75583640 1 -- 172.21.15.57:0/49958404 --> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2c700adde0 con 0x7f2c5c06aed0 2023-10-19T13:01:30.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.126+0000 7f2c577fe640 1 -- 172.21.15.57:0/49958404 <== mgr.7072 v2:172.21.15.179:6832/3281457486 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289854 (secure 0 0 0) 0x7f2c60012bd0 con 0x7f2c5c06aed0 2023-10-19T13:01:30.131 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:01:30.133 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2023-10-19T13:01:30.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.129+0000 7f2c75583640 1 -- 172.21.15.57:0/49958404 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f2c5c06aed0 msgr2=0x7f2c5c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:30.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.129+0000 7f2c75583640 1 --2- 172.21.15.57:0/49958404 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f2c5c06aed0 0x7f2c5c06d390 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2c701544a0 tx=0x7f2c60005f90 comp rx=0 tx=0).stop 2023-10-19T13:01:30.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.129+0000 7f2c75583640 1 -- 172.21.15.57:0/49958404 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2c7014eb80 msgr2=0x7f2c701554e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:30.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.129+0000 7f2c75583640 1 --2- 172.21.15.57:0/49958404 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2c7014eb80 0x7f2c701554e0 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f2c6800c760 tx=0x7f2c680012e0 comp rx=0 tx=0).stop 2023-10-19T13:01:30.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.129+0000 7f2c75583640 1 -- 172.21.15.57:0/49958404 shutdown_connections 2023-10-19T13:01:30.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.129+0000 7f2c75583640 1 --2- 172.21.15.57:0/49958404 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f2c5c06aed0 0x7f2c5c06d390 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:30.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.129+0000 7f2c75583640 1 --2- 172.21.15.57:0/49958404 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2c7014eb80 0x7f2c701554e0 unknown :-1 s=CLOSED pgs=1292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:30.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.129+0000 7f2c75583640 1 --2- 172.21.15.57:0/49958404 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2c7014e700 0x7f2c70154fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:30.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.129+0000 7f2c75583640 1 --2- 172.21.15.57:0/49958404 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2c58005180 0x7f2c7014c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:30.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.129+0000 7f2c75583640 1 -- 172.21.15.57:0/49958404 >> 172.21.15.57:0/49958404 conn(0x7f2c700b3f00 msgr2=0x7f2c70158640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:30.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.129+0000 7f2c75583640 1 -- 172.21.15.57:0/49958404 shutdown_connections 2023-10-19T13:01:30.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.129+0000 7f2c75583640 1 -- 172.21.15.57:0/49958404 wait complete. 2023-10-19T13:01:30.142 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2023-10-19T13:01:30.070689+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":762,"num_read_kb":1524,"num_write":333,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2237,"ondisk_log_size":2237,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":154,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":319800,"kb_used_data":12028,"kb_used_omap":12,"kb_used_meta":307635,"kb_avail":754654920,"statfs":{"total":773094113280,"available":772766638080,"internally_reserved":0,"allocated":12316672,"data_stored":6729209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":315018809},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.323200"},"pg_stats":[{"pgid":"14.3f","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-19T13:01:27.777475+0000","last_change":"2023-10-19T13:01:26.766994+0000","last_active":"2023-10-19T13:01:27.777475+0000","last_peered":"2023-10-19T13:01:27.777475+0000","last_clean":"2023-10-19T13:01:27.777475+0000","last_became_active":"2023-10-19T13:01:26.766740+0000","last_became_peered":"2023-10-19T13:01:26.766740+0000","last_unstale":"2023-10-19T13:01:27.777475+0000","last_undegraded":"2023-10-19T13:01:27.777475+0000","last_fullsized":"2023-10-19T13:01:27.777475+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:40:23.125246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3e","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-19T13:01:27.777250+0000","last_change":"2023-10-19T13:01:26.768883+0000","last_active":"2023-10-19T13:01:27.777250+0000","last_peered":"2023-10-19T13:01:27.777250+0000","last_clean":"2023-10-19T13:01:27.777250+0000","last_became_active":"2023-10-19T13:01:26.768624+0000","last_became_peered":"2023-10-19T13:01:26.768624+0000","last_unstale":"2023-10-19T13:01:27.777250+0000","last_undegraded":"2023-10-19T13:01:27.777250+0000","last_fullsized":"2023-10-19T13:01:27.777250+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:01:58.744461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.3d","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-19T13:01:27.777307+0000","last_change":"2023-10-19T13:01:26.772578+0000","last_active":"2023-10-19T13:01:27.777307+0000","last_peered":"2023-10-19T13:01:27.777307+0000","last_clean":"2023-10-19T13:01:27.777307+0000","last_became_active":"2023-10-19T13:01:26.772320+0000","last_became_peered":"2023-10-19T13:01:26.772320+0000","last_unstale":"2023-10-19T13:01:27.777307+0000","last_undegraded":"2023-10-19T13:01:27.777307+0000","last_fullsized":"2023-10-19T13:01:27.777307+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:23:00.868126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3c","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-19T13:01:27.773210+0000","last_change":"2023-10-19T13:01:26.776308+0000","last_active":"2023-10-19T13:01:27.773210+0000","last_peered":"2023-10-19T13:01:27.773210+0000","last_clean":"2023-10-19T13:01:27.773210+0000","last_became_active":"2023-10-19T13:01:26.776045+0000","last_became_peered":"2023-10-19T13:01:26.776045+0000","last_unstale":"2023-10-19T13:01:27.773210+0000","last_undegraded":"2023-10-19T13:01:27.773210+0000","last_fullsized":"2023-10-19T13:01:27.773210+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:40:39.673912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.3b","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-19T13:01:27.793207+0000","last_change":"2023-10-19T13:01:26.769643+0000","last_active":"2023-10-19T13:01:27.793207+0000","last_peered":"2023-10-19T13:01:27.793207+0000","last_clean":"2023-10-19T13:01:27.793207+0000","last_became_active":"2023-10-19T13:01:26.769388+0000","last_became_peered":"2023-10-19T13:01:26.769388+0000","last_unstale":"2023-10-19T13:01:27.793207+0000","last_undegraded":"2023-10-19T13:01:27.793207+0000","last_fullsized":"2023-10-19T13:01:27.793207+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:56:23.564267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.3a","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-19T13:01:26.785125+0000","last_change":"2023-10-19T13:01:26.775715+0000","last_active":"2023-10-19T13:01:26.785125+0000","last_peered":"2023-10-19T13:01:26.785125+0000","last_clean":"2023-10-19T13:01:26.785125+0000","last_became_active":"2023-10-19T13:01:26.775344+0000","last_became_peered":"2023-10-19T13:01:26.775344+0000","last_unstale":"2023-10-19T13:01:26.785125+0000","last_undegraded":"2023-10-19T13:01:26.785125+0000","last_fullsized":"2023-10-19T13:01:26.785125+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:18:30.670180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.39","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-19T13:01:26.774562+0000","last_change":"2023-10-19T13:01:26.774292+0000","last_active":"2023-10-19T13:01:26.774562+0000","last_peered":"2023-10-19T13:01:26.774562+0000","last_clean":"2023-10-19T13:01:26.774562+0000","last_became_active":"2023-10-19T13:01:26.774087+0000","last_became_peered":"2023-10-19T13:01:26.774087+0000","last_unstale":"2023-10-19T13:01:26.774562+0000","last_undegraded":"2023-10-19T13:01:26.774562+0000","last_fullsized":"2023-10-19T13:01:26.774562+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:36:25.704911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.38","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-19T13:01:27.772777+0000","last_change":"2023-10-19T13:01:26.770141+0000","last_active":"2023-10-19T13:01:27.772777+0000","last_peered":"2023-10-19T13:01:27.772777+0000","last_clean":"2023-10-19T13:01:27.772777+0000","last_became_active":"2023-10-19T13:01:26.769179+0000","last_became_peered":"2023-10-19T13:01:26.769179+0000","last_unstale":"2023-10-19T13:01:27.772777+0000","last_undegraded":"2023-10-19T13:01:27.772777+0000","last_fullsized":"2023-10-19T13:01:27.772777+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:15:36.508855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.37","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-19T13:01:27.793458+0000","last_change":"2023-10-19T13:01:26.770893+0000","last_active":"2023-10-19T13:01:27.793458+0000","last_peered":"2023-10-19T13:01:27.793458+0000","last_clean":"2023-10-19T13:01:27.793458+0000","last_became_active":"2023-10-19T13:01:26.770634+0000","last_became_peered":"2023-10-19T13:01:26.770634+0000","last_unstale":"2023-10-19T13:01:27.793458+0000","last_undegraded":"2023-10-19T13:01:27.793458+0000","last_fullsized":"2023-10-19T13:01:27.793458+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:09:37.554360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.36","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-19T13:01:26.778751+0000","last_change":"2023-10-19T13:01:26.777161+0000","last_active":"2023-10-19T13:01:26.778751+0000","last_peered":"2023-10-19T13:01:26.778751+0000","last_clean":"2023-10-19T13:01:26.778751+0000","last_became_active":"2023-10-19T13:01:26.776865+0000","last_became_peered":"2023-10-19T13:01:26.776865+0000","last_unstale":"2023-10-19T13:01:26.778751+0000","last_undegraded":"2023-10-19T13:01:26.778751+0000","last_fullsized":"2023-10-19T13:01:26.778751+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:59:46.741844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.35","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-19T13:01:27.793366+0000","last_change":"2023-10-19T13:01:26.772798+0000","last_active":"2023-10-19T13:01:27.793366+0000","last_peered":"2023-10-19T13:01:27.793366+0000","last_clean":"2023-10-19T13:01:27.793366+0000","last_became_active":"2023-10-19T13:01:26.772579+0000","last_became_peered":"2023-10-19T13:01:26.772579+0000","last_unstale":"2023-10-19T13:01:27.793366+0000","last_undegraded":"2023-10-19T13:01:27.793366+0000","last_fullsized":"2023-10-19T13:01:27.793366+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:40:05.822071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.34","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-19T13:01:26.775191+0000","last_change":"2023-10-19T13:01:26.773627+0000","last_active":"2023-10-19T13:01:26.775191+0000","last_peered":"2023-10-19T13:01:26.775191+0000","last_clean":"2023-10-19T13:01:26.775191+0000","last_became_active":"2023-10-19T13:01:26.773381+0000","last_became_peered":"2023-10-19T13:01:26.773381+0000","last_unstale":"2023-10-19T13:01:26.775191+0000","last_undegraded":"2023-10-19T13:01:26.775191+0000","last_fullsized":"2023-10-19T13:01:26.775191+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:46:13.355602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.33","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-19T13:01:27.772916+0000","last_change":"2023-10-19T13:01:26.775962+0000","last_active":"2023-10-19T13:01:27.772916+0000","last_peered":"2023-10-19T13:01:27.772916+0000","last_clean":"2023-10-19T13:01:27.772916+0000","last_became_active":"2023-10-19T13:01:26.775624+0000","last_became_peered":"2023-10-19T13:01:26.775624+0000","last_unstale":"2023-10-19T13:01:27.772916+0000","last_undegraded":"2023-10-19T13:01:27.772916+0000","last_fullsized":"2023-10-19T13:01:27.772916+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:28:51.930101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.32","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-19T13:01:27.793058+0000","last_change":"2023-10-19T13:01:26.779673+0000","last_active":"2023-10-19T13:01:27.793058+0000","last_peered":"2023-10-19T13:01:27.793058+0000","last_clean":"2023-10-19T13:01:27.793058+0000","last_became_active":"2023-10-19T13:01:26.779404+0000","last_became_peered":"2023-10-19T13:01:26.779404+0000","last_unstale":"2023-10-19T13:01:27.793058+0000","last_undegraded":"2023-10-19T13:01:27.793058+0000","last_fullsized":"2023-10-19T13:01:27.793058+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:23:19.825549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.31","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-19T13:01:27.772998+0000","last_change":"2023-10-19T13:01:26.775387+0000","last_active":"2023-10-19T13:01:27.772998+0000","last_peered":"2023-10-19T13:01:27.772998+0000","last_clean":"2023-10-19T13:01:27.772998+0000","last_became_active":"2023-10-19T13:01:26.775117+0000","last_became_peered":"2023-10-19T13:01:26.775117+0000","last_unstale":"2023-10-19T13:01:27.772998+0000","last_undegraded":"2023-10-19T13:01:27.772998+0000","last_fullsized":"2023-10-19T13:01:27.772998+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:23:19.825549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.30","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-19T13:01:26.787259+0000","last_change":"2023-10-19T13:01:26.775007+0000","last_active":"2023-10-19T13:01:26.787259+0000","last_peered":"2023-10-19T13:01:26.787259+0000","last_clean":"2023-10-19T13:01:26.787259+0000","last_became_active":"2023-10-19T13:01:26.774583+0000","last_became_peered":"2023-10-19T13:01:26.774583+0000","last_unstale":"2023-10-19T13:01:26.787259+0000","last_undegraded":"2023-10-19T13:01:26.787259+0000","last_fullsized":"2023-10-19T13:01:26.787259+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:30:29.811626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2f","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-19T13:01:26.785825+0000","last_change":"2023-10-19T13:01:26.770700+0000","last_active":"2023-10-19T13:01:26.785825+0000","last_peered":"2023-10-19T13:01:26.785825+0000","last_clean":"2023-10-19T13:01:26.785825+0000","last_became_active":"2023-10-19T13:01:26.770096+0000","last_became_peered":"2023-10-19T13:01:26.770096+0000","last_unstale":"2023-10-19T13:01:26.785825+0000","last_undegraded":"2023-10-19T13:01:26.785825+0000","last_fullsized":"2023-10-19T13:01:26.785825+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:24:28.596390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2e","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-19T13:01:27.777382+0000","last_change":"2023-10-19T13:01:26.764205+0000","last_active":"2023-10-19T13:01:27.777382+0000","last_peered":"2023-10-19T13:01:27.777382+0000","last_clean":"2023-10-19T13:01:27.777382+0000","last_became_active":"2023-10-19T13:01:26.763925+0000","last_became_peered":"2023-10-19T13:01:26.763925+0000","last_unstale":"2023-10-19T13:01:27.777382+0000","last_undegraded":"2023-10-19T13:01:27.777382+0000","last_fullsized":"2023-10-19T13:01:27.777382+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:24:50.102865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.2d","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-19T13:01:27.793375+0000","last_change":"2023-10-19T13:01:26.775891+0000","last_active":"2023-10-19T13:01:27.793375+0000","last_peered":"2023-10-19T13:01:27.793375+0000","last_clean":"2023-10-19T13:01:27.793375+0000","last_became_active":"2023-10-19T13:01:26.775646+0000","last_became_peered":"2023-10-19T13:01:26.775646+0000","last_unstale":"2023-10-19T13:01:27.793375+0000","last_undegraded":"2023-10-19T13:01:27.793375+0000","last_fullsized":"2023-10-19T13:01:27.793375+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:00:48.453255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2c","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-19T13:01:27.773284+0000","last_change":"2023-10-19T13:01:26.776034+0000","last_active":"2023-10-19T13:01:27.773284+0000","last_peered":"2023-10-19T13:01:27.773284+0000","last_clean":"2023-10-19T13:01:27.773284+0000","last_became_active":"2023-10-19T13:01:26.775709+0000","last_became_peered":"2023-10-19T13:01:26.775709+0000","last_unstale":"2023-10-19T13:01:27.773284+0000","last_undegraded":"2023-10-19T13:01:27.773284+0000","last_fullsized":"2023-10-19T13:01:27.773284+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:32:45.053563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.2b","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-19T13:01:27.793329+0000","last_change":"2023-10-19T13:01:26.769609+0000","last_active":"2023-10-19T13:01:27.793329+0000","last_peered":"2023-10-19T13:01:27.793329+0000","last_clean":"2023-10-19T13:01:27.793329+0000","last_became_active":"2023-10-19T13:01:26.769364+0000","last_became_peered":"2023-10-19T13:01:26.769364+0000","last_unstale":"2023-10-19T13:01:27.793329+0000","last_undegraded":"2023-10-19T13:01:27.793329+0000","last_fullsized":"2023-10-19T13:01:27.793329+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:09:50.788972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2a","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-19T13:01:27.793034+0000","last_change":"2023-10-19T13:01:26.780118+0000","last_active":"2023-10-19T13:01:27.793034+0000","last_peered":"2023-10-19T13:01:27.793034+0000","last_clean":"2023-10-19T13:01:27.793034+0000","last_became_active":"2023-10-19T13:01:26.779841+0000","last_became_peered":"2023-10-19T13:01:26.779841+0000","last_unstale":"2023-10-19T13:01:27.793034+0000","last_undegraded":"2023-10-19T13:01:27.793034+0000","last_fullsized":"2023-10-19T13:01:27.793034+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:28:51.930101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.29","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-19T13:01:26.778962+0000","last_change":"2023-10-19T13:01:26.777136+0000","last_active":"2023-10-19T13:01:26.778962+0000","last_peered":"2023-10-19T13:01:26.778962+0000","last_clean":"2023-10-19T13:01:26.778962+0000","last_became_active":"2023-10-19T13:01:26.776864+0000","last_became_peered":"2023-10-19T13:01:26.776864+0000","last_unstale":"2023-10-19T13:01:26.778962+0000","last_undegraded":"2023-10-19T13:01:26.778962+0000","last_fullsized":"2023-10-19T13:01:26.778962+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:12:59.179721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.28","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-19T13:01:27.777206+0000","last_change":"2023-10-19T13:01:26.765183+0000","last_active":"2023-10-19T13:01:27.777206+0000","last_peered":"2023-10-19T13:01:27.777206+0000","last_clean":"2023-10-19T13:01:27.777206+0000","last_became_active":"2023-10-19T13:01:26.764388+0000","last_became_peered":"2023-10-19T13:01:26.764388+0000","last_unstale":"2023-10-19T13:01:27.777206+0000","last_undegraded":"2023-10-19T13:01:27.777206+0000","last_fullsized":"2023-10-19T13:01:27.777206+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:34:05.277322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.27","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-19T13:01:27.777255+0000","last_change":"2023-10-19T13:01:26.764219+0000","last_active":"2023-10-19T13:01:27.777255+0000","last_peered":"2023-10-19T13:01:27.777255+0000","last_clean":"2023-10-19T13:01:27.777255+0000","last_became_active":"2023-10-19T13:01:26.763887+0000","last_became_peered":"2023-10-19T13:01:26.763887+0000","last_unstale":"2023-10-19T13:01:27.777255+0000","last_undegraded":"2023-10-19T13:01:27.777255+0000","last_fullsized":"2023-10-19T13:01:27.777255+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:31:21.716676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.26","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-19T13:01:27.793286+0000","last_change":"2023-10-19T13:01:26.776918+0000","last_active":"2023-10-19T13:01:27.793286+0000","last_peered":"2023-10-19T13:01:27.793286+0000","last_clean":"2023-10-19T13:01:27.793286+0000","last_became_active":"2023-10-19T13:01:26.776663+0000","last_became_peered":"2023-10-19T13:01:26.776663+0000","last_unstale":"2023-10-19T13:01:27.793286+0000","last_undegraded":"2023-10-19T13:01:27.793286+0000","last_fullsized":"2023-10-19T13:01:27.793286+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:48:14.520433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.25","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-19T13:01:26.772527+0000","last_change":"2023-10-19T13:01:26.769718+0000","last_active":"2023-10-19T13:01:26.772527+0000","last_peered":"2023-10-19T13:01:26.772527+0000","last_clean":"2023-10-19T13:01:26.772527+0000","last_became_active":"2023-10-19T13:01:26.769439+0000","last_became_peered":"2023-10-19T13:01:26.769439+0000","last_unstale":"2023-10-19T13:01:26.772527+0000","last_undegraded":"2023-10-19T13:01:26.772527+0000","last_fullsized":"2023-10-19T13:01:26.772527+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:27:18.101164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.24","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-19T13:01:27.773344+0000","last_change":"2023-10-19T13:01:26.776210+0000","last_active":"2023-10-19T13:01:27.773344+0000","last_peered":"2023-10-19T13:01:27.773344+0000","last_clean":"2023-10-19T13:01:27.773344+0000","last_became_active":"2023-10-19T13:01:26.775771+0000","last_became_peered":"2023-10-19T13:01:26.775771+0000","last_unstale":"2023-10-19T13:01:27.773344+0000","last_undegraded":"2023-10-19T13:01:27.773344+0000","last_fullsized":"2023-10-19T13:01:27.773344+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:51:32.035197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.23","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-19T13:01:26.773994+0000","last_change":"2023-10-19T13:01:26.773709+0000","last_active":"2023-10-19T13:01:26.773994+0000","last_peered":"2023-10-19T13:01:26.773994+0000","last_clean":"2023-10-19T13:01:26.773994+0000","last_became_active":"2023-10-19T13:01:26.773411+0000","last_became_peered":"2023-10-19T13:01:26.773411+0000","last_unstale":"2023-10-19T13:01:26.773994+0000","last_undegraded":"2023-10-19T13:01:26.773994+0000","last_fullsized":"2023-10-19T13:01:26.773994+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:20:15.281632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.22","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-19T13:01:27.793219+0000","last_change":"2023-10-19T13:01:26.770258+0000","last_active":"2023-10-19T13:01:27.793219+0000","last_peered":"2023-10-19T13:01:27.793219+0000","last_clean":"2023-10-19T13:01:27.793219+0000","last_became_active":"2023-10-19T13:01:26.770009+0000","last_became_peered":"2023-10-19T13:01:26.770009+0000","last_unstale":"2023-10-19T13:01:27.793219+0000","last_undegraded":"2023-10-19T13:01:27.793219+0000","last_fullsized":"2023-10-19T13:01:27.793219+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:40:39.673912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.21","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-19T13:01:27.777002+0000","last_change":"2023-10-19T13:01:26.777761+0000","last_active":"2023-10-19T13:01:27.777002+0000","last_peered":"2023-10-19T13:01:27.777002+0000","last_clean":"2023-10-19T13:01:27.777002+0000","last_became_active":"2023-10-19T13:01:26.777495+0000","last_became_peered":"2023-10-19T13:01:26.777495+0000","last_unstale":"2023-10-19T13:01:27.777002+0000","last_undegraded":"2023-10-19T13:01:27.777002+0000","last_fullsized":"2023-10-19T13:01:27.777002+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:59:14.252392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.20","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-19T13:01:26.787250+0000","last_change":"2023-10-19T13:01:26.772839+0000","last_active":"2023-10-19T13:01:26.787250+0000","last_peered":"2023-10-19T13:01:26.787250+0000","last_clean":"2023-10-19T13:01:26.787250+0000","last_became_active":"2023-10-19T13:01:26.772247+0000","last_became_peered":"2023-10-19T13:01:26.772247+0000","last_unstale":"2023-10-19T13:01:26.787250+0000","last_undegraded":"2023-10-19T13:01:26.787250+0000","last_fullsized":"2023-10-19T13:01:26.787250+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:14:09.608228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-19T13:01:26.786235+0000","last_change":"2023-10-19T13:01:26.774468+0000","last_active":"2023-10-19T13:01:26.786235+0000","last_peered":"2023-10-19T13:01:26.786235+0000","last_clean":"2023-10-19T13:01:26.786235+0000","last_became_active":"2023-10-19T13:01:26.774207+0000","last_became_peered":"2023-10-19T13:01:26.774207+0000","last_unstale":"2023-10-19T13:01:26.786235+0000","last_undegraded":"2023-10-19T13:01:26.786235+0000","last_fullsized":"2023-10-19T13:01:26.786235+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:05:44.654555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1f","version":"112'4","reported_seq":15,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-19T13:01:26.787674+0000","last_change":"2023-10-19T13:01:25.774201+0000","last_active":"2023-10-19T13:01:26.787674+0000","last_peered":"2023-10-19T13:01:26.787674+0000","last_clean":"2023-10-19T13:01:26.787674+0000","last_became_active":"2023-10-19T13:01:25.773861+0000","last_became_peered":"2023-10-19T13:01:25.773861+0000","last_unstale":"2023-10-19T13:01:26.787674+0000","last_undegraded":"2023-10-19T13:01:26.787674+0000","last_fullsized":"2023-10-19T13:01:26.787674+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_clean_scrub_stamp":"2023-10-19T13:01:24.734993+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:27:45.464690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-19T13:01:26.782898+0000","last_change":"2023-10-19T13:01:26.767579+0000","last_active":"2023-10-19T13:01:26.782898+0000","last_peered":"2023-10-19T13:01:26.782898+0000","last_clean":"2023-10-19T13:01:26.782898+0000","last_became_active":"2023-10-19T13:01:26.767330+0000","last_became_peered":"2023-10-19T13:01:26.767330+0000","last_unstale":"2023-10-19T13:01:26.782898+0000","last_undegraded":"2023-10-19T13:01:26.782898+0000","last_fullsized":"2023-10-19T13:01:26.782898+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:31:58.385639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1e","version":"112'1","reported_seq":13,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-19T13:01:27.793406+0000","last_change":"2023-10-19T13:01:25.766773+0000","last_active":"2023-10-19T13:01:27.793406+0000","last_peered":"2023-10-19T13:01:27.793406+0000","last_clean":"2023-10-19T13:01:27.793406+0000","last_became_active":"2023-10-19T13:01:25.766512+0000","last_became_peered":"2023-10-19T13:01:25.766512+0000","last_unstale":"2023-10-19T13:01:27.793406+0000","last_undegraded":"2023-10-19T13:01:27.793406+0000","last_fullsized":"2023-10-19T13:01:27.793406+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_clean_scrub_stamp":"2023-10-19T13:01:24.734993+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:08:11.628893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-19T13:01:26.784239+0000","last_change":"2023-10-19T13:01:26.783913+0000","last_active":"2023-10-19T13:01:26.784239+0000","last_peered":"2023-10-19T13:01:26.784239+0000","last_clean":"2023-10-19T13:01:26.784239+0000","last_became_active":"2023-10-19T13:01:26.783701+0000","last_became_peered":"2023-10-19T13:01:26.783701+0000","last_unstale":"2023-10-19T13:01:26.784239+0000","last_undegraded":"2023-10-19T13:01:26.784239+0000","last_fullsized":"2023-10-19T13:01:26.784239+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:01:58.744461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.1d","version":"112'2","reported_seq":13,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-19T13:01:26.781666+0000","last_change":"2023-10-19T13:01:25.777454+0000","last_active":"2023-10-19T13:01:26.781666+0000","last_peered":"2023-10-19T13:01:26.781666+0000","last_clean":"2023-10-19T13:01:26.781666+0000","last_became_active":"2023-10-19T13:01:25.777163+0000","last_became_peered":"2023-10-19T13:01:25.777163+0000","last_unstale":"2023-10-19T13:01:26.781666+0000","last_undegraded":"2023-10-19T13:01:26.781666+0000","last_fullsized":"2023-10-19T13:01:26.781666+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_clean_scrub_stamp":"2023-10-19T13:01:24.734993+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:54:24.490682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-19T13:01:26.785846+0000","last_change":"2023-10-19T13:01:26.771109+0000","last_active":"2023-10-19T13:01:26.785846+0000","last_peered":"2023-10-19T13:01:26.785846+0000","last_clean":"2023-10-19T13:01:26.785846+0000","last_became_active":"2023-10-19T13:01:26.768995+0000","last_became_peered":"2023-10-19T13:01:26.768995+0000","last_unstale":"2023-10-19T13:01:26.785846+0000","last_undegraded":"2023-10-19T13:01:26.785846+0000","last_fullsized":"2023-10-19T13:01:26.785846+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:26:39.188174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1c","version":"112'2","reported_seq":13,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-19T13:01:26.785885+0000","last_change":"2023-10-19T13:01:25.770917+0000","last_active":"2023-10-19T13:01:26.785885+0000","last_peered":"2023-10-19T13:01:26.785885+0000","last_clean":"2023-10-19T13:01:26.785885+0000","last_became_active":"2023-10-19T13:01:25.770706+0000","last_became_peered":"2023-10-19T13:01:25.770706+0000","last_unstale":"2023-10-19T13:01:26.785885+0000","last_undegraded":"2023-10-19T13:01:26.785885+0000","last_fullsized":"2023-10-19T13:01:26.785885+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_clean_scrub_stamp":"2023-10-19T13:01:24.734993+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:19:43.213087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-19T13:01:27.772769+0000","last_change":"2023-10-19T13:01:26.771485+0000","last_active":"2023-10-19T13:01:27.772769+0000","last_peered":"2023-10-19T13:01:27.772769+0000","last_clean":"2023-10-19T13:01:27.772769+0000","last_became_active":"2023-10-19T13:01:26.771235+0000","last_became_peered":"2023-10-19T13:01:26.771235+0000","last_unstale":"2023-10-19T13:01:27.772769+0000","last_undegraded":"2023-10-19T13:01:27.772769+0000","last_fullsized":"2023-10-19T13:01:27.772769+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:14:03.720278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.1b","version":"112'1","reported_seq":13,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-19T13:01:27.793292+0000","last_change":"2023-10-19T13:01:25.764777+0000","last_active":"2023-10-19T13:01:27.793292+0000","last_peered":"2023-10-19T13:01:27.793292+0000","last_clean":"2023-10-19T13:01:27.793292+0000","last_became_active":"2023-10-19T13:01:25.763783+0000","last_became_peered":"2023-10-19T13:01:25.763783+0000","last_unstale":"2023-10-19T13:01:27.793292+0000","last_undegraded":"2023-10-19T13:01:27.793292+0000","last_fullsized":"2023-10-19T13:01:27.793292+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_clean_scrub_stamp":"2023-10-19T13:01:24.734993+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:01:46.832580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-19T13:01:26.785360+0000","last_change":"2023-10-19T13:01:26.785030+0000","last_active":"2023-10-19T13:01:26.785360+0000","last_peered":"2023-10-19T13:01:26.785360+0000","last_clean":"2023-10-19T13:01:26.785360+0000","last_became_active":"2023-10-19T13:01:26.784790+0000","last_became_peered":"2023-10-19T13:01:26.784790+0000","last_unstale":"2023-10-19T13:01:26.785360+0000","last_undegraded":"2023-10-19T13:01:26.785360+0000","last_fullsized":"2023-10-19T13:01:26.785360+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:21:55.576961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1a","version":"112'1","reported_seq":13,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-19T13:01:27.773314+0000","last_change":"2023-10-19T13:01:25.764864+0000","last_active":"2023-10-19T13:01:27.773314+0000","last_peered":"2023-10-19T13:01:27.773314+0000","last_clean":"2023-10-19T13:01:27.773314+0000","last_became_active":"2023-10-19T13:01:25.764586+0000","last_became_peered":"2023-10-19T13:01:25.764586+0000","last_unstale":"2023-10-19T13:01:27.773314+0000","last_undegraded":"2023-10-19T13:01:27.773314+0000","last_fullsized":"2023-10-19T13:01:27.773314+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_clean_scrub_stamp":"2023-10-19T13:01:24.734993+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:25:39.517420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-19T13:01:26.783708+0000","last_change":"2023-10-19T13:01:26.771422+0000","last_active":"2023-10-19T13:01:26.783708+0000","last_peered":"2023-10-19T13:01:26.783708+0000","last_clean":"2023-10-19T13:01:26.783708+0000","last_became_active":"2023-10-19T13:01:26.770660+0000","last_became_peered":"2023-10-19T13:01:26.770660+0000","last_unstale":"2023-10-19T13:01:26.783708+0000","last_undegraded":"2023-10-19T13:01:26.783708+0000","last_fullsized":"2023-10-19T13:01:26.783708+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:43:04.195600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":12,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-19T13:01:27.777309+0000","last_change":"2023-10-19T13:01:25.772204+0000","last_active":"2023-10-19T13:01:27.777309+0000","last_peered":"2023-10-19T13:01:27.777309+0000","last_clean":"2023-10-19T13:01:27.777309+0000","last_became_active":"2023-10-19T13:01:25.771937+0000","last_became_peered":"2023-10-19T13:01:25.771937+0000","last_unstale":"2023-10-19T13:01:27.777309+0000","last_undegraded":"2023-10-19T13:01:27.777309+0000","last_fullsized":"2023-10-19T13:01:27.777309+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_clean_scrub_stamp":"2023-10-19T13:01:24.734993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:44:54.196450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-19T13:01:27.773049+0000","last_change":"2023-10-19T13:01:26.775878+0000","last_active":"2023-10-19T13:01:27.773049+0000","last_peered":"2023-10-19T13:01:27.773049+0000","last_clean":"2023-10-19T13:01:27.773049+0000","last_became_active":"2023-10-19T13:01:26.775535+0000","last_became_peered":"2023-10-19T13:01:26.775535+0000","last_unstale":"2023-10-19T13:01:27.773049+0000","last_undegraded":"2023-10-19T13:01:27.773049+0000","last_fullsized":"2023-10-19T13:01:27.773049+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:56:23.564267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-19T13:01:26.766828+0000","last_change":"2023-10-19T13:01:25.778057+0000","last_active":"2023-10-19T13:01:26.766828+0000","last_peered":"2023-10-19T13:01:26.766828+0000","last_clean":"2023-10-19T13:01:26.766828+0000","last_became_active":"2023-10-19T13:01:25.777812+0000","last_became_peered":"2023-10-19T13:01:25.777812+0000","last_unstale":"2023-10-19T13:01:26.766828+0000","last_undegraded":"2023-10-19T13:01:26.766828+0000","last_fullsized":"2023-10-19T13:01:26.766828+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_clean_scrub_stamp":"2023-10-19T13:01:24.734993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:59:13.240064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-19T13:01:27.793329+0000","last_change":"2023-10-19T13:01:26.770474+0000","last_active":"2023-10-19T13:01:27.793329+0000","last_peered":"2023-10-19T13:01:27.793329+0000","last_clean":"2023-10-19T13:01:27.793329+0000","last_became_active":"2023-10-19T13:01:26.770205+0000","last_became_peered":"2023-10-19T13:01:26.770205+0000","last_unstale":"2023-10-19T13:01:27.793329+0000","last_undegraded":"2023-10-19T13:01:27.793329+0000","last_fullsized":"2023-10-19T13:01:27.793329+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:32:45.053563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":12,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-19T13:01:27.777390+0000","last_change":"2023-10-19T13:01:25.770400+0000","last_active":"2023-10-19T13:01:27.777390+0000","last_peered":"2023-10-19T13:01:27.777390+0000","last_clean":"2023-10-19T13:01:27.777390+0000","last_became_active":"2023-10-19T13:01:25.770134+0000","last_became_peered":"2023-10-19T13:01:25.770134+0000","last_unstale":"2023-10-19T13:01:27.777390+0000","last_undegraded":"2023-10-19T13:01:27.777390+0000","last_fullsized":"2023-10-19T13:01:27.777390+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_clean_scrub_stamp":"2023-10-19T13:01:24.734993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:36:24.692583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-19T13:01:26.772387+0000","last_change":"2023-10-19T13:01:26.771797+0000","last_active":"2023-10-19T13:01:26.772387+0000","last_peered":"2023-10-19T13:01:26.772387+0000","last_clean":"2023-10-19T13:01:26.772387+0000","last_became_active":"2023-10-19T13:01:26.771541+0000","last_became_peered":"2023-10-19T13:01:26.771541+0000","last_unstale":"2023-10-19T13:01:26.772387+0000","last_undegraded":"2023-10-19T13:01:26.772387+0000","last_fullsized":"2023-10-19T13:01:26.772387+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:09:36.668121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":12,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-19T13:01:27.777435+0000","last_change":"2023-10-19T13:01:25.771171+0000","last_active":"2023-10-19T13:01:27.777435+0000","last_peered":"2023-10-19T13:01:27.777435+0000","last_clean":"2023-10-19T13:01:27.777435+0000","last_became_active":"2023-10-19T13:01:25.770900+0000","last_became_peered":"2023-10-19T13:01:25.770900+0000","last_unstale":"2023-10-19T13:01:27.777435+0000","last_undegraded":"2023-10-19T13:01:27.777435+0000","last_fullsized":"2023-10-19T13:01:27.777435+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_clean_scrub_stamp":"2023-10-19T13:01:24.734993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:42:04.249097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-19T13:01:27.777431+0000","last_change":"2023-10-19T13:01:26.767777+0000","last_active":"2023-10-19T13:01:27.777431+0000","last_peered":"2023-10-19T13:01:27.777431+0000","last_clean":"2023-10-19T13:01:27.777431+0000","last_became_active":"2023-10-19T13:01:26.767520+0000","last_became_peered":"2023-10-19T13:01:26.767520+0000","last_unstale":"2023-10-19T13:01:27.777431+0000","last_undegraded":"2023-10-19T13:01:27.777431+0000","last_fullsized":"2023-10-19T13:01:27.777431+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:37:03.522414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.15","version":"112'3","reported_seq":15,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-19T13:01:27.793462+0000","last_change":"2023-10-19T13:01:25.777169+0000","last_active":"2023-10-19T13:01:27.793462+0000","last_peered":"2023-10-19T13:01:27.793462+0000","last_clean":"2023-10-19T13:01:27.793462+0000","last_became_active":"2023-10-19T13:01:25.776912+0000","last_became_peered":"2023-10-19T13:01:25.776912+0000","last_unstale":"2023-10-19T13:01:27.793462+0000","last_undegraded":"2023-10-19T13:01:27.793462+0000","last_fullsized":"2023-10-19T13:01:27.793462+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_clean_scrub_stamp":"2023-10-19T13:01:24.734993+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:19:43.213087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-19T13:01:27.773011+0000","last_change":"2023-10-19T13:01:26.776514+0000","last_active":"2023-10-19T13:01:27.773011+0000","last_peered":"2023-10-19T13:01:27.773011+0000","last_clean":"2023-10-19T13:01:27.773011+0000","last_became_active":"2023-10-19T13:01:26.775394+0000","last_became_peered":"2023-10-19T13:01:26.775394+0000","last_unstale":"2023-10-19T13:01:27.773011+0000","last_undegraded":"2023-10-19T13:01:27.773011+0000","last_fullsized":"2023-10-19T13:01:27.773011+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:09:50.788972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.14","version":"112'6","reported_seq":17,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-19T13:01:26.788973+0000","last_change":"2023-10-19T13:01:25.772773+0000","last_active":"2023-10-19T13:01:26.788973+0000","last_peered":"2023-10-19T13:01:26.788973+0000","last_clean":"2023-10-19T13:01:26.788973+0000","last_became_active":"2023-10-19T13:01:25.772300+0000","last_became_peered":"2023-10-19T13:01:25.772300+0000","last_unstale":"2023-10-19T13:01:26.788973+0000","last_undegraded":"2023-10-19T13:01:26.788973+0000","last_fullsized":"2023-10-19T13:01:26.788973+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_clean_scrub_stamp":"2023-10-19T13:01:24.734993+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:42:04.249097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-19T13:01:27.773253+0000","last_change":"2023-10-19T13:01:26.766977+0000","last_active":"2023-10-19T13:01:27.773253+0000","last_peered":"2023-10-19T13:01:27.773253+0000","last_clean":"2023-10-19T13:01:27.773253+0000","last_became_active":"2023-10-19T13:01:26.766753+0000","last_became_peered":"2023-10-19T13:01:26.766753+0000","last_unstale":"2023-10-19T13:01:27.773253+0000","last_undegraded":"2023-10-19T13:01:27.773253+0000","last_fullsized":"2023-10-19T13:01:27.773253+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:48:14.520433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"112'2","reported_seq":13,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-19T13:01:26.787711+0000","last_change":"2023-10-19T13:01:25.778084+0000","last_active":"2023-10-19T13:01:26.787711+0000","last_peered":"2023-10-19T13:01:26.787711+0000","last_clean":"2023-10-19T13:01:26.787711+0000","last_became_active":"2023-10-19T13:01:25.777754+0000","last_became_peered":"2023-10-19T13:01:25.777754+0000","last_unstale":"2023-10-19T13:01:26.787711+0000","last_undegraded":"2023-10-19T13:01:26.787711+0000","last_fullsized":"2023-10-19T13:01:26.787711+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_clean_scrub_stamp":"2023-10-19T13:01:24.734993+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:39:50.471500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-19T13:01:27.793023+0000","last_change":"2023-10-19T13:01:26.764521+0000","last_active":"2023-10-19T13:01:27.793023+0000","last_peered":"2023-10-19T13:01:27.793023+0000","last_clean":"2023-10-19T13:01:27.793023+0000","last_became_active":"2023-10-19T13:01:26.764270+0000","last_became_peered":"2023-10-19T13:01:26.764270+0000","last_unstale":"2023-10-19T13:01:27.793023+0000","last_undegraded":"2023-10-19T13:01:27.793023+0000","last_fullsized":"2023-10-19T13:01:27.793023+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:14:03.720278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-19T13:01:26.757234+0000","last_change":"2023-10-19T13:01:25.773615+0000","last_active":"2023-10-19T13:01:26.757234+0000","last_peered":"2023-10-19T13:01:26.757234+0000","last_clean":"2023-10-19T13:01:26.757234+0000","last_became_active":"2023-10-19T13:01:25.773260+0000","last_became_peered":"2023-10-19T13:01:25.773260+0000","last_unstale":"2023-10-19T13:01:26.757234+0000","last_undegraded":"2023-10-19T13:01:26.757234+0000","last_fullsized":"2023-10-19T13:01:26.757234+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_clean_scrub_stamp":"2023-10-19T13:01:24.734993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:20:43.498132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-19T13:01:26.785890+0000","last_change":"2023-10-19T13:01:26.784964+0000","last_active":"2023-10-19T13:01:26.785890+0000","last_peered":"2023-10-19T13:01:26.785890+0000","last_clean":"2023-10-19T13:01:26.785890+0000","last_became_active":"2023-10-19T13:01:26.784668+0000","last_became_peered":"2023-10-19T13:01:26.784668+0000","last_unstale":"2023-10-19T13:01:26.785890+0000","last_undegraded":"2023-10-19T13:01:26.785890+0000","last_fullsized":"2023-10-19T13:01:26.785890+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:16:00.956915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":12,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-19T13:01:27.772870+0000","last_change":"2023-10-19T13:01:25.769579+0000","last_active":"2023-10-19T13:01:27.772870+0000","last_peered":"2023-10-19T13:01:27.772870+0000","last_clean":"2023-10-19T13:01:27.772870+0000","last_became_active":"2023-10-19T13:01:25.769328+0000","last_became_peered":"2023-10-19T13:01:25.769328+0000","last_unstale":"2023-10-19T13:01:27.772870+0000","last_undegraded":"2023-10-19T13:01:27.772870+0000","last_fullsized":"2023-10-19T13:01:27.772870+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_clean_scrub_stamp":"2023-10-19T13:01:24.734993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:33:16.385580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-19T13:01:26.772992+0000","last_change":"2023-10-19T13:01:26.771809+0000","last_active":"2023-10-19T13:01:26.772992+0000","last_peered":"2023-10-19T13:01:26.772992+0000","last_clean":"2023-10-19T13:01:26.772992+0000","last_became_active":"2023-10-19T13:01:26.771574+0000","last_became_peered":"2023-10-19T13:01:26.771574+0000","last_unstale":"2023-10-19T13:01:26.772992+0000","last_undegraded":"2023-10-19T13:01:26.772992+0000","last_fullsized":"2023-10-19T13:01:26.772992+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:39:45.964234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.10","version":"112'8","reported_seq":20,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-19T13:01:27.772869+0000","last_change":"2023-10-19T13:01:25.764911+0000","last_active":"2023-10-19T13:01:27.772869+0000","last_peered":"2023-10-19T13:01:27.772869+0000","last_clean":"2023-10-19T13:01:27.772869+0000","last_became_active":"2023-10-19T13:01:25.764640+0000","last_became_peered":"2023-10-19T13:01:25.764640+0000","last_unstale":"2023-10-19T13:01:27.772869+0000","last_undegraded":"2023-10-19T13:01:27.772869+0000","last_fullsized":"2023-10-19T13:01:27.772869+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_clean_scrub_stamp":"2023-10-19T13:01:24.734993+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:12:58.167393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-19T13:01:26.780644+0000","last_change":"2023-10-19T13:01:26.768802+0000","last_active":"2023-10-19T13:01:26.780644+0000","last_peered":"2023-10-19T13:01:26.780644+0000","last_clean":"2023-10-19T13:01:26.780644+0000","last_became_active":"2023-10-19T13:01:26.768409+0000","last_became_peered":"2023-10-19T13:01:26.768409+0000","last_unstale":"2023-10-19T13:01:26.780644+0000","last_undegraded":"2023-10-19T13:01:26.780644+0000","last_fullsized":"2023-10-19T13:01:26.780644+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:58:01.254793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.f","version":"112'1","reported_seq":12,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-19T13:01:26.780622+0000","last_change":"2023-10-19T13:01:25.773066+0000","last_active":"2023-10-19T13:01:26.780622+0000","last_peered":"2023-10-19T13:01:26.780622+0000","last_clean":"2023-10-19T13:01:26.780622+0000","last_became_active":"2023-10-19T13:01:25.772817+0000","last_became_peered":"2023-10-19T13:01:25.772817+0000","last_unstale":"2023-10-19T13:01:26.780622+0000","last_undegraded":"2023-10-19T13:01:26.780622+0000","last_fullsized":"2023-10-19T13:01:26.780622+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_clean_scrub_stamp":"2023-10-19T13:01:24.734993+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:33:50.263526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-19T13:01:26.765748+0000","last_change":"2023-10-19T13:01:26.764788+0000","last_active":"2023-10-19T13:01:26.765748+0000","last_peered":"2023-10-19T13:01:26.765748+0000","last_clean":"2023-10-19T13:01:26.765748+0000","last_became_active":"2023-10-19T13:01:26.764550+0000","last_became_peered":"2023-10-19T13:01:26.764550+0000","last_unstale":"2023-10-19T13:01:26.765748+0000","last_undegraded":"2023-10-19T13:01:26.765748+0000","last_fullsized":"2023-10-19T13:01:26.765748+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:08:45.606703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.e","version":"112'2","reported_seq":13,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-19T13:01:26.781978+0000","last_change":"2023-10-19T13:01:25.773394+0000","last_active":"2023-10-19T13:01:26.781978+0000","last_peered":"2023-10-19T13:01:26.781978+0000","last_clean":"2023-10-19T13:01:26.781978+0000","last_became_active":"2023-10-19T13:01:25.773029+0000","last_became_peered":"2023-10-19T13:01:25.773029+0000","last_unstale":"2023-10-19T13:01:26.781978+0000","last_undegraded":"2023-10-19T13:01:26.781978+0000","last_fullsized":"2023-10-19T13:01:26.781978+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_clean_scrub_stamp":"2023-10-19T13:01:24.734993+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:31:20.704348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-19T13:01:27.793378+0000","last_change":"2023-10-19T13:01:26.770031+0000","last_active":"2023-10-19T13:01:27.793378+0000","last_peered":"2023-10-19T13:01:27.793378+0000","last_clean":"2023-10-19T13:01:27.793378+0000","last_became_active":"2023-10-19T13:01:26.769781+0000","last_became_peered":"2023-10-19T13:01:26.769781+0000","last_unstale":"2023-10-19T13:01:27.793378+0000","last_undegraded":"2023-10-19T13:01:27.793378+0000","last_fullsized":"2023-10-19T13:01:27.793378+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:51:32.035197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.d","version":"112'1","reported_seq":13,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-19T13:01:27.793472+0000","last_change":"2023-10-19T13:01:25.770251+0000","last_active":"2023-10-19T13:01:27.793472+0000","last_peered":"2023-10-19T13:01:27.793472+0000","last_clean":"2023-10-19T13:01:27.793472+0000","last_became_active":"2023-10-19T13:01:25.770010+0000","last_became_peered":"2023-10-19T13:01:25.770010+0000","last_unstale":"2023-10-19T13:01:27.793472+0000","last_undegraded":"2023-10-19T13:01:27.793472+0000","last_fullsized":"2023-10-19T13:01:27.793472+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_clean_scrub_stamp":"2023-10-19T13:01:24.734993+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:42:35.785045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-19T13:01:26.787238+0000","last_change":"2023-10-19T13:01:26.772645+0000","last_active":"2023-10-19T13:01:26.787238+0000","last_peered":"2023-10-19T13:01:26.787238+0000","last_clean":"2023-10-19T13:01:26.787238+0000","last_became_active":"2023-10-19T13:01:26.767998+0000","last_became_peered":"2023-10-19T13:01:26.767998+0000","last_unstale":"2023-10-19T13:01:26.787238+0000","last_undegraded":"2023-10-19T13:01:26.787238+0000","last_fullsized":"2023-10-19T13:01:26.787238+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:44:55.208778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.b","version":"112'2","reported_seq":14,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-19T13:01:27.777042+0000","last_change":"2023-10-19T13:01:25.769384+0000","last_active":"2023-10-19T13:01:27.777042+0000","last_peered":"2023-10-19T13:01:27.777042+0000","last_clean":"2023-10-19T13:01:27.777042+0000","last_became_active":"2023-10-19T13:01:25.769135+0000","last_became_peered":"2023-10-19T13:01:25.769135+0000","last_unstale":"2023-10-19T13:01:27.777042+0000","last_undegraded":"2023-10-19T13:01:27.777042+0000","last_fullsized":"2023-10-19T13:01:27.777042+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_clean_scrub_stamp":"2023-10-19T13:01:24.734993+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:51:58.089637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-19T13:01:27.777012+0000","last_change":"2023-10-19T13:01:26.769060+0000","last_active":"2023-10-19T13:01:27.777012+0000","last_peered":"2023-10-19T13:01:27.777012+0000","last_clean":"2023-10-19T13:01:27.777012+0000","last_became_active":"2023-10-19T13:01:26.768679+0000","last_became_peered":"2023-10-19T13:01:26.768679+0000","last_unstale":"2023-10-19T13:01:27.777012+0000","last_undegraded":"2023-10-19T13:01:27.777012+0000","last_fullsized":"2023-10-19T13:01:27.777012+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:09:36.668121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-19T13:01:26.761029+0000","last_change":"2023-10-19T13:01:25.763996+0000","last_active":"2023-10-19T13:01:26.761029+0000","last_peered":"2023-10-19T13:01:26.761029+0000","last_clean":"2023-10-19T13:01:26.761029+0000","last_became_active":"2023-10-19T13:01:25.763741+0000","last_became_peered":"2023-10-19T13:01:25.763741+0000","last_unstale":"2023-10-19T13:01:26.761029+0000","last_undegraded":"2023-10-19T13:01:26.761029+0000","last_fullsized":"2023-10-19T13:01:26.761029+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_clean_scrub_stamp":"2023-10-19T13:01:24.734993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:26:25.588442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-19T13:01:27.777180+0000","last_change":"2023-10-19T13:01:26.767228+0000","last_active":"2023-10-19T13:01:27.777180+0000","last_peered":"2023-10-19T13:01:27.777180+0000","last_clean":"2023-10-19T13:01:27.777180+0000","last_became_active":"2023-10-19T13:01:26.766946+0000","last_became_peered":"2023-10-19T13:01:26.766946+0000","last_unstale":"2023-10-19T13:01:27.777180+0000","last_undegraded":"2023-10-19T13:01:27.777180+0000","last_fullsized":"2023-10-19T13:01:27.777180+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:54:25.503010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":12,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-19T13:01:27.777175+0000","last_change":"2023-10-19T13:01:25.772467+0000","last_active":"2023-10-19T13:01:27.777175+0000","last_peered":"2023-10-19T13:01:27.777175+0000","last_clean":"2023-10-19T13:01:27.777175+0000","last_became_active":"2023-10-19T13:01:25.772222+0000","last_became_peered":"2023-10-19T13:01:25.772222+0000","last_unstale":"2023-10-19T13:01:27.777175+0000","last_undegraded":"2023-10-19T13:01:27.777175+0000","last_fullsized":"2023-10-19T13:01:27.777175+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_clean_scrub_stamp":"2023-10-19T13:01:24.734993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:00:28.580196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-19T13:01:26.778147+0000","last_change":"2023-10-19T13:01:26.766902+0000","last_active":"2023-10-19T13:01:26.778147+0000","last_peered":"2023-10-19T13:01:26.778147+0000","last_clean":"2023-10-19T13:01:26.778147+0000","last_became_active":"2023-10-19T13:01:26.766632+0000","last_became_peered":"2023-10-19T13:01:26.766632+0000","last_unstale":"2023-10-19T13:01:26.778147+0000","last_undegraded":"2023-10-19T13:01:26.778147+0000","last_fullsized":"2023-10-19T13:01:26.778147+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:43:31.922698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-19T13:01:26.760964+0000","last_change":"2023-10-19T13:01:25.767709+0000","last_active":"2023-10-19T13:01:26.760964+0000","last_peered":"2023-10-19T13:01:26.760964+0000","last_clean":"2023-10-19T13:01:26.760964+0000","last_became_active":"2023-10-19T13:01:25.767462+0000","last_became_peered":"2023-10-19T13:01:25.767462+0000","last_unstale":"2023-10-19T13:01:26.760964+0000","last_undegraded":"2023-10-19T13:01:26.760964+0000","last_fullsized":"2023-10-19T13:01:26.760964+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_clean_scrub_stamp":"2023-10-19T13:01:24.734993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:38:56.932665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-19T13:01:26.770229+0000","last_change":"2023-10-19T13:01:26.768158+0000","last_active":"2023-10-19T13:01:26.770229+0000","last_peered":"2023-10-19T13:01:26.770229+0000","last_clean":"2023-10-19T13:01:26.770229+0000","last_became_active":"2023-10-19T13:01:26.767916+0000","last_became_peered":"2023-10-19T13:01:26.767916+0000","last_unstale":"2023-10-19T13:01:26.770229+0000","last_undegraded":"2023-10-19T13:01:26.770229+0000","last_fullsized":"2023-10-19T13:01:26.770229+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:25:40.529748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.7","version":"112'2","reported_seq":13,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-19T13:01:26.782229+0000","last_change":"2023-10-19T13:01:25.773977+0000","last_active":"2023-10-19T13:01:26.782229+0000","last_peered":"2023-10-19T13:01:26.782229+0000","last_clean":"2023-10-19T13:01:26.782229+0000","last_became_active":"2023-10-19T13:01:25.773727+0000","last_became_peered":"2023-10-19T13:01:25.773727+0000","last_unstale":"2023-10-19T13:01:26.782229+0000","last_undegraded":"2023-10-19T13:01:26.782229+0000","last_fullsized":"2023-10-19T13:01:26.782229+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_clean_scrub_stamp":"2023-10-19T13:01:24.734993+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:01:30.794534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-19T13:01:26.782917+0000","last_change":"2023-10-19T13:01:26.768607+0000","last_active":"2023-10-19T13:01:26.782917+0000","last_peered":"2023-10-19T13:01:26.782917+0000","last_clean":"2023-10-19T13:01:26.782917+0000","last_became_active":"2023-10-19T13:01:26.768359+0000","last_became_peered":"2023-10-19T13:01:26.768359+0000","last_unstale":"2023-10-19T13:01:26.782917+0000","last_undegraded":"2023-10-19T13:01:26.782917+0000","last_fullsized":"2023-10-19T13:01:26.782917+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:23:00.868126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":12,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-19T13:01:27.793461+0000","last_change":"2023-10-19T13:01:25.766847+0000","last_active":"2023-10-19T13:01:27.793461+0000","last_peered":"2023-10-19T13:01:27.793461+0000","last_clean":"2023-10-19T13:01:27.793461+0000","last_became_active":"2023-10-19T13:01:25.766596+0000","last_became_peered":"2023-10-19T13:01:25.766596+0000","last_unstale":"2023-10-19T13:01:27.793461+0000","last_undegraded":"2023-10-19T13:01:27.793461+0000","last_fullsized":"2023-10-19T13:01:27.793461+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_clean_scrub_stamp":"2023-10-19T13:01:24.734993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:01:54.371707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-19T13:01:26.778249+0000","last_change":"2023-10-19T13:01:26.767051+0000","last_active":"2023-10-19T13:01:26.778249+0000","last_peered":"2023-10-19T13:01:26.778249+0000","last_clean":"2023-10-19T13:01:26.778249+0000","last_became_active":"2023-10-19T13:01:26.766794+0000","last_became_peered":"2023-10-19T13:01:26.766794+0000","last_unstale":"2023-10-19T13:01:26.778249+0000","last_undegraded":"2023-10-19T13:01:26.778249+0000","last_fullsized":"2023-10-19T13:01:26.778249+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:24:50.102865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":12,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-19T13:01:27.793516+0000","last_change":"2023-10-19T13:01:25.774353+0000","last_active":"2023-10-19T13:01:27.793516+0000","last_peered":"2023-10-19T13:01:27.793516+0000","last_clean":"2023-10-19T13:01:27.793516+0000","last_became_active":"2023-10-19T13:01:25.774064+0000","last_became_peered":"2023-10-19T13:01:25.774064+0000","last_unstale":"2023-10-19T13:01:27.793516+0000","last_undegraded":"2023-10-19T13:01:27.793516+0000","last_fullsized":"2023-10-19T13:01:27.793516+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_clean_scrub_stamp":"2023-10-19T13:01:24.734993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:31:28.516513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-19T13:01:27.777371+0000","last_change":"2023-10-19T13:01:26.768108+0000","last_active":"2023-10-19T13:01:27.777371+0000","last_peered":"2023-10-19T13:01:27.777371+0000","last_clean":"2023-10-19T13:01:27.777371+0000","last_became_active":"2023-10-19T13:01:26.767855+0000","last_became_peered":"2023-10-19T13:01:26.767855+0000","last_unstale":"2023-10-19T13:01:27.777371+0000","last_undegraded":"2023-10-19T13:01:27.777371+0000","last_fullsized":"2023-10-19T13:01:27.777371+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:39:51.483828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-19T13:01:26.762963+0000","last_change":"2023-10-19T13:01:25.767430+0000","last_active":"2023-10-19T13:01:26.762963+0000","last_peered":"2023-10-19T13:01:26.762963+0000","last_clean":"2023-10-19T13:01:26.762963+0000","last_became_active":"2023-10-19T13:01:25.767136+0000","last_became_peered":"2023-10-19T13:01:25.767136+0000","last_unstale":"2023-10-19T13:01:26.762963+0000","last_undegraded":"2023-10-19T13:01:26.762963+0000","last_fullsized":"2023-10-19T13:01:26.762963+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_clean_scrub_stamp":"2023-10-19T13:01:24.734993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:08:11.628893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-19T13:01:26.772590+0000","last_change":"2023-10-19T13:01:26.769715+0000","last_active":"2023-10-19T13:01:26.772590+0000","last_peered":"2023-10-19T13:01:26.772590+0000","last_clean":"2023-10-19T13:01:26.772590+0000","last_became_active":"2023-10-19T13:01:26.769478+0000","last_became_peered":"2023-10-19T13:01:26.769478+0000","last_unstale":"2023-10-19T13:01:26.772590+0000","last_undegraded":"2023-10-19T13:01:26.772590+0000","last_fullsized":"2023-10-19T13:01:26.772590+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:56:48.142582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-19T13:01:26.758217+0000","last_change":"2023-10-19T13:01:25.773851+0000","last_active":"2023-10-19T13:01:26.758217+0000","last_peered":"2023-10-19T13:01:26.758217+0000","last_clean":"2023-10-19T13:01:26.758217+0000","last_became_active":"2023-10-19T13:01:25.773581+0000","last_became_peered":"2023-10-19T13:01:25.773581+0000","last_unstale":"2023-10-19T13:01:26.758217+0000","last_undegraded":"2023-10-19T13:01:26.758217+0000","last_fullsized":"2023-10-19T13:01:26.758217+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_clean_scrub_stamp":"2023-10-19T13:01:24.734993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:34:04.264994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":11,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-19T13:01:27.777230+0000","last_change":"2023-10-19T13:01:26.767813+0000","last_active":"2023-10-19T13:01:27.777230+0000","last_peered":"2023-10-19T13:01:27.777230+0000","last_clean":"2023-10-19T13:01:27.777230+0000","last_became_active":"2023-10-19T13:01:26.767504+0000","last_became_peered":"2023-10-19T13:01:26.767504+0000","last_unstale":"2023-10-19T13:01:27.777230+0000","last_undegraded":"2023-10-19T13:01:27.777230+0000","last_fullsized":"2023-10-19T13:01:27.777230+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:46:37.116113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":12,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-19T13:01:27.777291+0000","last_change":"2023-10-19T13:01:25.769381+0000","last_active":"2023-10-19T13:01:27.777291+0000","last_peered":"2023-10-19T13:01:27.777291+0000","last_clean":"2023-10-19T13:01:27.777291+0000","last_became_active":"2023-10-19T13:01:25.769125+0000","last_became_peered":"2023-10-19T13:01:25.769125+0000","last_unstale":"2023-10-19T13:01:27.777291+0000","last_undegraded":"2023-10-19T13:01:27.777291+0000","last_fullsized":"2023-10-19T13:01:27.777291+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_clean_scrub_stamp":"2023-10-19T13:01:24.734993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:18:46.992112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-19T13:01:26.780260+0000","last_change":"2023-10-19T13:01:26.778785+0000","last_active":"2023-10-19T13:01:26.780260+0000","last_peered":"2023-10-19T13:01:26.780260+0000","last_clean":"2023-10-19T13:01:26.780260+0000","last_became_active":"2023-10-19T13:01:26.778527+0000","last_became_peered":"2023-10-19T13:01:26.778527+0000","last_unstale":"2023-10-19T13:01:26.780260+0000","last_undegraded":"2023-10-19T13:01:26.780260+0000","last_fullsized":"2023-10-19T13:01:26.780260+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:21:46.013171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-19T13:01:26.762862+0000","last_change":"2023-10-19T13:01:25.767432+0000","last_active":"2023-10-19T13:01:26.762862+0000","last_peered":"2023-10-19T13:01:26.762862+0000","last_clean":"2023-10-19T13:01:26.762862+0000","last_became_active":"2023-10-19T13:01:25.767155+0000","last_became_peered":"2023-10-19T13:01:25.767155+0000","last_unstale":"2023-10-19T13:01:26.762862+0000","last_undegraded":"2023-10-19T13:01:26.762862+0000","last_fullsized":"2023-10-19T13:01:26.762862+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_clean_scrub_stamp":"2023-10-19T13:01:24.734993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:33:16.385580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-19T13:01:26.785392+0000","last_change":"2023-10-19T13:01:26.771343+0000","last_active":"2023-10-19T13:01:26.785392+0000","last_peered":"2023-10-19T13:01:26.785392+0000","last_clean":"2023-10-19T13:01:26.785392+0000","last_became_active":"2023-10-19T13:01:26.770786+0000","last_became_peered":"2023-10-19T13:01:26.770786+0000","last_unstale":"2023-10-19T13:01:26.785392+0000","last_undegraded":"2023-10-19T13:01:26.785392+0000","last_fullsized":"2023-10-19T13:01:26.785392+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:46:37.116113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":11,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-19T13:01:26.766282+0000","last_change":"2023-10-19T13:01:25.771982+0000","last_active":"2023-10-19T13:01:26.766282+0000","last_peered":"2023-10-19T13:01:26.766282+0000","last_clean":"2023-10-19T13:01:26.766282+0000","last_became_active":"2023-10-19T13:01:25.771724+0000","last_became_peered":"2023-10-19T13:01:25.771724+0000","last_unstale":"2023-10-19T13:01:26.766282+0000","last_undegraded":"2023-10-19T13:01:26.766282+0000","last_fullsized":"2023-10-19T13:01:26.766282+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_clean_scrub_stamp":"2023-10-19T13:01:24.734993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:00:28.580196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":9,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-19T13:01:26.774592+0000","last_change":"2023-10-19T13:01:26.774302+0000","last_active":"2023-10-19T13:01:26.774592+0000","last_peered":"2023-10-19T13:01:26.774592+0000","last_clean":"2023-10-19T13:01:26.774592+0000","last_became_active":"2023-10-19T13:01:26.774093+0000","last_became_peered":"2023-10-19T13:01:26.774093+0000","last_unstale":"2023-10-19T13:01:26.774592+0000","last_undegraded":"2023-10-19T13:01:26.774592+0000","last_fullsized":"2023-10-19T13:01:26.774592+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:25.747321+0000","last_clean_scrub_stamp":"2023-10-19T13:01:25.747321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:26:37.314738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":12,"reported_epoch":113,"state":"active+clean","last_fresh":"2023-10-19T13:01:27.773340+0000","last_change":"2023-10-19T13:01:25.768820+0000","last_active":"2023-10-19T13:01:27.773340+0000","last_peered":"2023-10-19T13:01:27.773340+0000","last_clean":"2023-10-19T13:01:27.773340+0000","last_became_active":"2023-10-19T13:01:25.768556+0000","last_became_peered":"2023-10-19T13:01:25.768556+0000","last_unstale":"2023-10-19T13:01:27.773340+0000","last_undegraded":"2023-10-19T13:01:27.773340+0000","last_fullsized":"2023-10-19T13:01:27.773340+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:01:24.734993+0000","last_clean_scrub_stamp":"2023-10-19T13:01:24.734993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:01:54.371707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"112'2199","reported_seq":2535,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-19T13:01:28.068164+0000","last_change":"2023-10-19T12:44:08.636613+0000","last_active":"2023-10-19T13:01:28.068164+0000","last_peered":"2023-10-19T13:01:28.068164+0000","last_clean":"2023-10-19T13:01:28.068164+0000","last_became_active":"2023-10-19T12:44:08.635697+0000","last_became_peered":"2023-10-19T12:44:08.635697+0000","last_unstale":"2023-10-19T13:01:28.068164+0000","last_undegraded":"2023-10-19T13:01:28.068164+0000","last_fullsized":"2023-10-19T13:01:28.068164+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_clean_scrub_stamp":"2023-10-19T12:44:07.399738+0000","objects_scrubbed":0,"log_size":2199,"log_dups_size":0,"ondisk_log_size":2199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:15:23.685485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":762,"num_read_kb":1524,"num_write":289,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":14,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":762,"num_read_kb":1524,"num_write":289,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2199,"ondisk_log_size":2199,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575059,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51168,"kb_used_data":1936,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320672,"statfs":{"total":96636764160,"available":96584368128,"internally_reserved":0,"allocated":1982464,"data_stored":1281796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":50395595},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 13:00:44 2023","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.55200000000000005,"15min":0.56899999999999995},"min":{"1min":0.34100000000000003,"5min":0.30299999999999999,"15min":0.26400000000000001},"max":{"1min":0.74099999999999999,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.58199999999999996,"15min":0.58899999999999997},"min":{"1min":0.36499999999999999,"5min":0.33200000000000002,"15min":0.29599999999999999},"max":{"1min":0.75600000000000001,"5min":0.83799999999999997,"15min":0.93700000000000006},"last":0.33200000000000002}]},{"osd":1,"last update":"Thu Oct 19 13:00:44 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.56599999999999995,"15min":0.57699999999999996},"min":{"1min":0.41299999999999998,"5min":0.40300000000000002,"15min":0.28199999999999997},"max":{"1min":0.68100000000000005,"5min":0.86299999999999999,"15min":1.3129999999999999},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.57599999999999996,"15min":0.58999999999999997},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.32700000000000001},"max":{"1min":0.69299999999999995,"5min":0.82699999999999996,"15min":1.284},"last":0.41399999999999998}]},{"osd":2,"last update":"Thu Oct 19 13:00:44 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.623,"15min":0.63},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.76900000000000002,"5min":0.82699999999999996,"15min":0.98599999999999999},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.61899999999999999,"15min":0.63},"min":{"1min":0.46200000000000002,"5min":0.41599999999999998,"15min":0.28399999999999997},"max":{"1min":0.78300000000000003,"5min":0.81200000000000006,"15min":1.2589999999999999},"last":0.42499999999999999}]},{"osd":3,"last update":"Thu Oct 19 13:00:44 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.58999999999999997,"15min":0.59999999999999998},"min":{"1min":0.39700000000000002,"5min":0.34999999999999998,"15min":0.34399999999999997},"max":{"1min":0.71599999999999997,"5min":0.81200000000000006,"15min":0.96899999999999997},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.621,"5min":0.65400000000000003,"15min":0.65600000000000003},"min":{"1min":0.45500000000000002,"5min":0.45300000000000001,"15min":0.39700000000000002},"max":{"1min":0.79800000000000004,"5min":0.92000000000000004,"15min":1.329},"last":0.38600000000000001}]},{"osd":4,"last update":"Thu Oct 19 13:00:44 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.47499999999999998,"15min":0.46999999999999997},"min":{"1min":0.315,"5min":0.27400000000000002,"15min":0.25600000000000001},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.80500000000000005},"last":0.247},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.47399999999999998,"15min":0.47199999999999998},"min":{"1min":0.30199999999999999,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.67100000000000004,"5min":0.75600000000000001,"15min":0.78900000000000003},"last":0.216}]},{"osd":5,"last update":"Thu Oct 19 13:00:44 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.48899999999999999,"15min":0.48499999999999999},"min":{"1min":0.32000000000000001,"5min":0.30499999999999999,"15min":0.29099999999999998},"max":{"1min":0.74099999999999999,"5min":0.75800000000000001,"15min":0.90400000000000003},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52000000000000002,"15min":0.51700000000000002},"min":{"1min":0.34000000000000002,"5min":0.314,"15min":0.308},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.92200000000000004},"last":0.374}]},{"osd":6,"last update":"Thu Oct 19 13:00:44 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.54000000000000004,"15min":0.54800000000000004},"min":{"1min":0.378,"5min":0.34100000000000003,"15min":0.308},"max":{"1min":0.749,"5min":0.81299999999999994,"15min":0.88900000000000001},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52000000000000002,"15min":0.52800000000000002},"min":{"1min":0.33400000000000002,"5min":0.30299999999999999,"15min":0.27300000000000002},"max":{"1min":0.68300000000000005,"5min":0.83999999999999997,"15min":0.90300000000000002},"last":0.36699999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575059,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32556,"kb_used_data":1368,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339284,"statfs":{"total":96636764160,"available":96603426816,"internally_reserved":0,"allocated":1400832,"data_stored":693588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 13:00:32 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.57199999999999995,"15min":0.58599999999999997},"min":{"1min":0.373,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.75800000000000001,"5min":0.76000000000000001,"15min":0.81999999999999995},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.57799999999999996,"15min":0.58499999999999996},"min":{"1min":0.36399999999999999,"5min":0.247,"15min":0.247},"max":{"1min":0.75600000000000001,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.46300000000000002}]},{"osd":1,"last update":"Thu Oct 19 13:00:32 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.35299999999999998,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.76000000000000001,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.59699999999999998,"15min":0.59899999999999998},"min":{"1min":0.42099999999999999,"5min":0.34300000000000003,"15min":0.28399999999999997},"max":{"1min":0.749,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.377}]},{"osd":2,"last update":"Thu Oct 19 13:00:32 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.57299999999999995,"15min":0.58199999999999996},"min":{"1min":0.36899999999999999,"5min":0.307,"15min":0.26600000000000001},"max":{"1min":0.755,"5min":0.84699999999999998,"15min":0.85699999999999998},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.61299999999999999,"15min":0.60999999999999999},"min":{"1min":0.38900000000000001,"5min":0.35999999999999999,"15min":0.28399999999999997},"max":{"1min":0.79400000000000004,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.36899999999999999}]},{"osd":3,"last update":"Thu Oct 19 13:00:32 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61899999999999999,"15min":0.624},"min":{"1min":0.38800000000000001,"5min":0.38500000000000001,"15min":0.32600000000000001},"max":{"1min":0.81299999999999994,"5min":0.871,"15min":0.871},"last":0.434},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.61899999999999999,"15min":0.624},"min":{"1min":0.40100000000000002,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.74099999999999999,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.48299999999999998}]},{"osd":4,"last update":"Thu Oct 19 13:00:32 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.45900000000000002,"15min":0.46600000000000003},"min":{"1min":0.29099999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.76700000000000002,"5min":0.77900000000000003,"15min":0.79300000000000004},"last":0.24099999999999999},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.47699999999999998,"15min":0.47999999999999998},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.247},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.77600000000000002},"last":0.29399999999999998}]},{"osd":5,"last update":"Thu Oct 19 13:00:32 2023","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.46100000000000002,"15min":0.47099999999999997},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.26600000000000001},"max":{"1min":0.55700000000000005,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.307},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.51400000000000001,"15min":0.51600000000000001},"min":{"1min":0.375,"5min":0.32100000000000001,"15min":0.312},"max":{"1min":0.747,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.27900000000000003}]},{"osd":7,"last update":"Thu Oct 19 13:00:32 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.52500000000000002,"15min":0.52100000000000002},"min":{"1min":0.32400000000000001,"5min":0.29299999999999998,"15min":0.28199999999999997},"max":{"1min":0.77600000000000002,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.53500000000000003},"min":{"1min":0.27900000000000003,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.66700000000000004,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.34799999999999998}]}]},{"osd":5,"up_from":13,"seq":55834575059,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50592,"kb_used_data":1360,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321248,"statfs":{"total":96636764160,"available":96584957952,"internally_reserved":0,"allocated":1392640,"data_stored":698020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 13:00:29 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58799999999999997,"15min":0.59099999999999997},"min":{"1min":0.307,"5min":0.28999999999999998,"15min":0.223},"max":{"1min":0.75900000000000001,"5min":0.79900000000000004,"15min":0.82699999999999996},"last":0.432},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58599999999999997,"15min":0.59799999999999998},"min":{"1min":0.41099999999999998,"5min":0.36099999999999999,"15min":0.27800000000000002},"max":{"1min":0.82999999999999996,"5min":0.84299999999999997,"15min":0.874},"last":0.498}]},{"osd":1,"last update":"Thu Oct 19 13:00:29 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.57799999999999996,"15min":0.59499999999999997},"min":{"1min":0.40400000000000003,"5min":0.34599999999999997,"15min":0.32800000000000001},"max":{"1min":0.71299999999999997,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.60199999999999998,"15min":0.61199999999999999},"min":{"1min":0.378,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.77200000000000002,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.39800000000000002}]},{"osd":2,"last update":"Thu Oct 19 13:00:29 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.56399999999999995,"15min":0.58499999999999996},"min":{"1min":0.41199999999999998,"5min":0.34100000000000003,"15min":0.307},"max":{"1min":0.71699999999999997,"5min":0.78600000000000003,"15min":0.80700000000000005},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.61199999999999999,"15min":0.624},"min":{"1min":0.47099999999999997,"5min":0.38400000000000001,"15min":0.36599999999999999},"max":{"1min":0.79300000000000004,"5min":0.88100000000000001,"15min":0.93600000000000005},"last":0.32300000000000001}]},{"osd":3,"last update":"Thu Oct 19 13:00:29 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.58899999999999997,"15min":0.60399999999999998},"min":{"1min":0.41699999999999998,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.752,"5min":0.79900000000000004,"15min":0.81299999999999994},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.61499999999999999,"15min":0.63400000000000001},"min":{"1min":0.501,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.73999999999999999,"5min":0.81000000000000005,"15min":0.89500000000000002},"last":0.45100000000000001}]},{"osd":4,"last update":"Thu Oct 19 13:00:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.47599999999999998,"15min":0.47499999999999998},"min":{"1min":0.35399999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.73599999999999999,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.376},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.46800000000000003,"15min":0.46999999999999997},"min":{"1min":0.33900000000000002,"5min":0.251,"15min":0.251},"max":{"1min":0.69199999999999995,"5min":0.75900000000000001,"15min":0.80500000000000005},"last":0.35699999999999998}]},{"osd":6,"last update":"Thu Oct 19 13:00:29 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.34100000000000003,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.51100000000000001,"15min":0.504},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.33000000000000002},"max":{"1min":0.67200000000000004,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.47099999999999997}]},{"osd":7,"last update":"Thu Oct 19 13:00:29 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.498,"15min":0.49399999999999999},"min":{"1min":0.39800000000000002,"5min":0.32000000000000001,"15min":0.315},"max":{"1min":0.72399999999999998,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.44},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.54900000000000004,"15min":0.53900000000000003},"min":{"1min":0.42599999999999999,"5min":0.311,"15min":0.311},"max":{"1min":0.74099999999999999,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.49099999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575059,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32536,"kb_used_data":1356,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339304,"statfs":{"total":96636764160,"available":96603447296,"internally_reserved":0,"allocated":1388544,"data_stored":691532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 13:00:43 2023","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.55200000000000005,"15min":0.56000000000000005},"min":{"1min":0.36099999999999999,"5min":0.29499999999999998,"15min":0.28899999999999998},"max":{"1min":0.72299999999999998,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.24199999999999999},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.54700000000000004,"15min":0.55700000000000005},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.27600000000000002},"max":{"1min":0.70099999999999996,"5min":0.77800000000000002,"15min":0.78200000000000003},"last":0.26200000000000001}]},{"osd":1,"last update":"Thu Oct 19 13:00:43 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.58099999999999996,"15min":0.60699999999999998},"min":{"1min":0.36599999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.72699999999999998,"5min":0.80600000000000005,"15min":0.86199999999999999},"last":0.498},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.60299999999999998,"15min":0.61499999999999999},"min":{"1min":0.433,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.73699999999999999,"5min":0.79900000000000004,"15min":0.873},"last":0.47399999999999998}]},{"osd":2,"last update":"Thu Oct 19 13:00:43 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.59399999999999997,"15min":0.61099999999999999},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.254},"max":{"1min":0.79300000000000004,"5min":0.85299999999999998,"15min":0.86599999999999999},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.59699999999999998,"15min":0.61299999999999999},"min":{"1min":0.38700000000000001,"5min":0.35799999999999998,"15min":0.34499999999999997},"max":{"1min":0.75800000000000001,"5min":0.82999999999999996,"15min":0.85999999999999999},"last":0.438}]},{"osd":3,"last update":"Thu Oct 19 13:00:43 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.57499999999999996,"15min":0.58499999999999996},"min":{"1min":0.36199999999999999,"5min":0.35199999999999998,"15min":0.32400000000000001},"max":{"1min":0.75,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.63,"15min":0.63700000000000001},"min":{"1min":0.44800000000000001,"5min":0.42599999999999999,"15min":0.37},"max":{"1min":0.76500000000000001,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.42099999999999999}]},{"osd":5,"last update":"Thu Oct 19 13:00:43 2023","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.45100000000000001,"15min":0.44800000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.249},"max":{"1min":0.63200000000000001,"5min":0.79600000000000004,"15min":0.81299999999999994},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.46899999999999997,"15min":0.45900000000000002},"min":{"1min":0.30099999999999999,"5min":0.26400000000000001,"15min":0.23499999999999999},"max":{"1min":0.69499999999999995,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.40400000000000003}]},{"osd":6,"last update":"Thu Oct 19 13:00:43 2023","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.45500000000000002,"15min":0.45400000000000001},"min":{"1min":0.30099999999999999,"5min":0.29499999999999998,"15min":0.26400000000000001},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.78600000000000003},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.48799999999999999,"15min":0.49399999999999999},"min":{"1min":0.308,"5min":0.308,"15min":0.27100000000000002},"max":{"1min":0.70699999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.32400000000000001}]},{"osd":7,"last update":"Thu Oct 19 13:00:43 2023","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.505,"15min":0.501},"min":{"1min":0.33200000000000002,"5min":0.307,"15min":0.27500000000000002},"max":{"1min":0.72799999999999998,"5min":0.79100000000000004,"15min":0.80200000000000005},"last":0.372},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54200000000000004,"15min":0.54400000000000004},"min":{"1min":0.371,"5min":0.35799999999999998,"15min":0.30199999999999999},"max":{"1min":0.70799999999999996,"5min":0.80500000000000005,"15min":0.83499999999999996},"last":0.52300000000000002}]}]},{"osd":3,"up_from":13,"seq":55834575058,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36692,"kb_used_data":1348,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335148,"statfs":{"total":96636764160,"available":96599191552,"internally_reserved":0,"allocated":1380352,"data_stored":691372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 13:00:27 2023","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.42199999999999999,"15min":0.40000000000000002},"min":{"1min":0.252,"5min":0.23599999999999999,"15min":0.20899999999999999},"max":{"1min":0.60399999999999998,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.41799999999999998,"15min":0.41599999999999998},"min":{"1min":0.28699999999999998,"5min":0.223,"15min":0.223},"max":{"1min":0.59899999999999998,"5min":0.75,"15min":0.75},"last":0.34599999999999997}]},{"osd":1,"last update":"Thu Oct 19 13:00:27 2023","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.41999999999999998,"15min":0.41199999999999998},"min":{"1min":0.26900000000000002,"5min":0.26100000000000001,"15min":0.246},"max":{"1min":0.628,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.307},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.48599999999999999,"15min":0.45900000000000002},"min":{"1min":0.30099999999999999,"5min":0.24099999999999999,"15min":0.215},"max":{"1min":0.64200000000000002,"5min":2.9399999999999999,"15min":2.9399999999999999},"last":0.221}]},{"osd":2,"last update":"Thu Oct 19 13:00:27 2023","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.47099999999999997,"15min":0.46800000000000003},"min":{"1min":0.317,"5min":0.314,"15min":0.26100000000000001},"max":{"1min":0.62,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.376},{"interface":"front","average":{"1min":0.496,"5min":0.48299999999999998,"15min":0.48699999999999999},"min":{"1min":0.34499999999999997,"5min":0.29999999999999999,"15min":0.247},"max":{"1min":0.66900000000000004,"5min":0.754,"15min":0.754},"last":0.33100000000000002}]},{"osd":4,"last update":"Thu Oct 19 13:00:27 2023","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.66000000000000003,"15min":0.66700000000000004},"min":{"1min":0.44700000000000001,"5min":0.42299999999999999,"15min":0.40300000000000002},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":1.0069999999999999},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.68799999999999994,"15min":0.69199999999999995},"min":{"1min":0.48899999999999999,"5min":0.46899999999999997,"15min":0.42099999999999999},"max":{"1min":0.91400000000000003,"5min":0.93000000000000005,"15min":0.99399999999999999},"last":0.40500000000000003}]},{"osd":5,"last update":"Thu Oct 19 13:00:27 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.68500000000000005,"15min":0.66000000000000003},"min":{"1min":0.47499999999999998,"5min":0.45500000000000002,"15min":0.41099999999999998},"max":{"1min":0.90600000000000003,"5min":2.9990000000000001,"15min":2.9990000000000001},"last":0.432},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.65000000000000002,"15min":0.65100000000000002},"min":{"1min":0.47699999999999998,"5min":0.40899999999999997,"15min":0.35899999999999999},"max":{"1min":0.873,"5min":1.774,"15min":1.774},"last":0.495}]},{"osd":6,"last update":"Thu Oct 19 13:00:27 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.66300000000000003,"15min":0.65400000000000003},"min":{"1min":0.503,"5min":0.433,"15min":0.41199999999999998},"max":{"1min":0.84499999999999997,"5min":0.93700000000000006,"15min":0.96799999999999997},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.68000000000000005,"15min":0.67600000000000005},"min":{"1min":0.501,"5min":0.377,"15min":0.377},"max":{"1min":0.94399999999999995,"5min":1.167,"15min":1.167},"last":0.36499999999999999}]},{"osd":7,"last update":"Thu Oct 19 13:00:27 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.65900000000000003,"15min":0.66500000000000004},"min":{"1min":0.47999999999999998,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.90700000000000003,"5min":1.042,"15min":1.042},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.69499999999999995,"15min":0.68100000000000005},"min":{"1min":0.51100000000000001,"5min":0.432,"15min":0.432},"max":{"1min":0.88300000000000001,"5min":2.8149999999999999,"15min":2.8149999999999999},"last":0.441}]}]},{"osd":2,"up_from":13,"seq":55834575060,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54744,"kb_used_data":1348,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317096,"statfs":{"total":96636764160,"available":96580706304,"internally_reserved":0,"allocated":1380352,"data_stored":691372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 13:00:37 2023","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.36899999999999999,"15min":0.378},"min":{"1min":0.215,"5min":0.20899999999999999,"15min":0.187},"max":{"1min":0.52900000000000003,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.374,"5min":0.39700000000000002,"15min":0.40699999999999997},"min":{"1min":0.28000000000000003,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.56000000000000005,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.44600000000000001}]},{"osd":1,"last update":"Thu Oct 19 13:00:37 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.38200000000000001,"15min":0.39600000000000002},"min":{"1min":0.26100000000000001,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.51100000000000001,"5min":0.63900000000000001,"15min":0.71099999999999997},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.41099999999999998,"15min":0.42799999999999999},"min":{"1min":0.26900000000000002,"5min":0.22,"15min":0.22},"max":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.68100000000000005},"last":0.24299999999999999}]},{"osd":3,"last update":"Thu Oct 19 13:00:37 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.45300000000000001,"15min":0.45500000000000002},"min":{"1min":0.33700000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.50700000000000001,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.439,"5min":0.46899999999999997,"15min":0.48199999999999998},"min":{"1min":0.34100000000000003,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.57499999999999996,"5min":0.77800000000000002,"15min":0.82999999999999996},"last":0.42499999999999999}]},{"osd":4,"last update":"Thu Oct 19 13:00:37 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.61899999999999999,"15min":0.64000000000000001},"min":{"1min":0.40100000000000002,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.68700000000000006,"5min":0.93999999999999995,"15min":0.98999999999999999},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.63400000000000001,"15min":0.65500000000000003},"min":{"1min":0.42999999999999999,"5min":0.35099999999999998,"15min":0.32000000000000001},"max":{"1min":0.78000000000000003,"5min":0.91200000000000003,"15min":1.046},"last":0.46200000000000002}]},{"osd":5,"last update":"Thu Oct 19 13:00:37 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.65000000000000002,"15min":0.66100000000000003},"min":{"1min":0.46899999999999997,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.73999999999999999,"5min":0.96499999999999997,"15min":0.99399999999999999},"last":0.496},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.63200000000000001,"15min":0.65600000000000003},"min":{"1min":0.39100000000000001,"5min":0.375,"15min":0.375},"max":{"1min":0.753,"5min":0.90200000000000002,"15min":0.95799999999999996},"last":0.47799999999999998}]},{"osd":6,"last update":"Thu Oct 19 13:00:37 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.63700000000000001,"15min":0.66300000000000003},"min":{"1min":0.48799999999999999,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.73599999999999999,"5min":1.008,"15min":1.008},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.65600000000000003,"15min":0.67400000000000004},"min":{"1min":0.46300000000000002,"5min":0.371,"15min":0.371},"max":{"1min":0.80500000000000005,"5min":0.995,"15min":0.995},"last":0.52300000000000002}]},{"osd":7,"last update":"Thu Oct 19 13:00:37 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.65400000000000003,"15min":0.68000000000000005},"min":{"1min":0.39800000000000002,"5min":0.39300000000000002,"15min":0.36399999999999999},"max":{"1min":0.71799999999999997,"5min":0.98999999999999999,"15min":1.0149999999999999},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.66800000000000004,"15min":0.68100000000000005},"min":{"1min":0.47199999999999998,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.80000000000000004,"5min":1.02,"15min":1.0329999999999999},"last":0.54100000000000004}]}]},{"osd":1,"up_from":13,"seq":55834575059,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32544,"kb_used_data":1364,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339296,"statfs":{"total":96636764160,"available":96603439104,"internally_reserved":0,"allocated":1396736,"data_stored":698090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 13:00:36 2023","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.39000000000000001,"15min":0.38600000000000001},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.187},"max":{"1min":0.57799999999999996,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.20399999999999999},{"interface":"front","average":{"1min":0.438,"5min":0.441,"15min":0.43099999999999999},"min":{"1min":0.217,"5min":0.217,"15min":0.21199999999999999},"max":{"1min":0.65800000000000003,"5min":2.5800000000000001,"15min":2.5800000000000001},"last":0.34599999999999997}]},{"osd":2,"last update":"Thu Oct 19 13:00:36 2023","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.41799999999999998,"15min":0.42199999999999999},"min":{"1min":0.24199999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.69999999999999996},"last":0.245},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.47599999999999998,"15min":0.45500000000000002},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.23699999999999999},"max":{"1min":0.75600000000000001,"5min":2.6230000000000002,"15min":2.6230000000000002},"last":0.27800000000000002}]},{"osd":3,"last update":"Thu Oct 19 13:00:36 2023","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.50700000000000001,"15min":0.48499999999999999},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.26200000000000001},"max":{"1min":0.69599999999999995,"5min":3.9740000000000002,"15min":3.9740000000000002},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.43099999999999999,"15min":0.42199999999999999},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27000000000000002},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.80500000000000005},"last":0.30199999999999999}]},{"osd":4,"last update":"Thu Oct 19 13:00:36 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.44900000000000001,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.86899999999999999,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":1.0409999999999999},{"interface":"front","average":{"1min":0.627,"5min":0.66100000000000003,"15min":0.64500000000000002},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.314},"max":{"1min":0.86899999999999999,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.55300000000000005}]},{"osd":5,"last update":"Thu Oct 19 13:00:36 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.65300000000000002,"15min":0.65300000000000002},"min":{"1min":0.55300000000000005,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.85299999999999998,"5min":1.1200000000000001,"15min":1.1200000000000001},"last":1.0549999999999999},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.69499999999999995,"15min":0.67300000000000004},"min":{"1min":0.436,"5min":0.42299999999999999,"15min":0.38700000000000001},"max":{"1min":0.84499999999999997,"5min":2.7120000000000002,"15min":2.7120000000000002},"last":1.1020000000000001}]},{"osd":6,"last update":"Thu Oct 19 13:00:36 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.64600000000000002,"15min":0.65400000000000003},"min":{"1min":0.46999999999999997,"5min":0.36799999999999999,"15min":0.36399999999999999},"max":{"1min":0.83799999999999997,"5min":0.92200000000000004,"15min":1.0069999999999999},"last":0.5},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.67800000000000005,"15min":0.67700000000000005},"min":{"1min":0.47299999999999998,"5min":0.376,"15min":0.376},"max":{"1min":1.129,"5min":1.129,"15min":1.129},"last":0.57699999999999996}]},{"osd":7,"last update":"Thu Oct 19 13:00:36 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.69199999999999995,"15min":0.67500000000000004},"min":{"1min":0.52900000000000003,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.80300000000000005,"5min":2.786,"15min":2.786},"last":1.1439999999999999},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.67400000000000004,"15min":0.67600000000000005},"min":{"1min":0.45200000000000001,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.79500000000000004,"5min":0.94199999999999995,"15min":1.034},"last":0.52600000000000002}]}]},{"osd":0,"up_from":13,"seq":55834575060,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28968,"kb_used_data":1948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342872,"statfs":{"total":96636764160,"available":96607100928,"internally_reserved":0,"allocated":1994752,"data_stored":1283439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Oct 19 13:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.38700000000000001,"15min":0.39000000000000001},"min":{"1min":0.20899999999999999,"5min":0.182,"15min":0.182},"max":{"1min":0.52800000000000002,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.372,"5min":0.42199999999999999,"15min":0.43099999999999999},"min":{"1min":0.248,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.58999999999999997,"5min":0.67600000000000005,"15min":0.746},"last":0.27700000000000002}]},{"osd":2,"last update":"Thu Oct 19 13:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.45100000000000001,"15min":0.45300000000000001},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.25800000000000001},"max":{"1min":0.64100000000000001,"5min":0.752,"15min":0.76300000000000001},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.42499999999999999,"15min":0.434},"min":{"1min":0.246,"5min":0.216,"15min":0.20699999999999999},"max":{"1min":0.623,"5min":0.72999999999999998,"15min":0.76800000000000002},"last":0.29999999999999999}]},{"osd":3,"last update":"Thu Oct 19 13:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.40999999999999998,"15min":0.40899999999999997},"min":{"1min":0.26600000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.56599999999999995,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.311},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.318,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.67500000000000004,"5min":1.4990000000000001,"15min":1.4990000000000001},"last":0.34699999999999998}]},{"osd":4,"last update":"Thu Oct 19 13:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.61399999999999999,"15min":0.63300000000000001},"min":{"1min":0.40300000000000002,"5min":0.40000000000000002,"15min":0.34999999999999998},"max":{"1min":0.71299999999999997,"5min":1.361,"15min":1.361},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.64500000000000002,"15min":0.64900000000000002},"min":{"1min":0.42899999999999999,"5min":0.41899999999999998,"15min":0.35999999999999999},"max":{"1min":0.80900000000000005,"5min":0.98999999999999999,"15min":0.998},"last":0.66500000000000004}]},{"osd":5,"last update":"Thu Oct 19 13:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.67900000000000005,"15min":0.68100000000000005},"min":{"1min":0.49199999999999999,"5min":0.378,"15min":0.378},"max":{"1min":0.90000000000000002,"5min":1.325,"15min":1.325},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.62,"5min":0.67900000000000005,"15min":0.68300000000000005},"min":{"1min":0.437,"5min":0.36299999999999999,"15min":0.318},"max":{"1min":0.88,"5min":1.024,"15min":1.024},"last":0.61499999999999999}]},{"osd":6,"last update":"Thu Oct 19 13:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.65600000000000003,"15min":0.66200000000000003},"min":{"1min":0.42499999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.79300000000000004,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.65200000000000002,"15min":0.66200000000000003},"min":{"1min":0.45300000000000001,"5min":0.41999999999999998,"15min":0.38900000000000001},"max":{"1min":0.73499999999999999,"5min":0.995,"15min":1.089},"last":0.70299999999999996}]},{"osd":7,"last update":"Thu Oct 19 13:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.66200000000000003,"15min":0.67900000000000005},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.82799999999999996,"5min":0.98799999999999999,"15min":1.0089999999999999},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.65900000000000003,"15min":0.66100000000000003},"min":{"1min":0.442,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.94499999999999995,"5min":1.389,"15min":1.389},"last":0.68100000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-19T13:01:30.150 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-19T13:01:30.150 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-19T13:01:30.242 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:30.239+0000 7fb508163640 -1 client.7141: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.7030:f55302c0-37aa-4235-8ccc-9775bc542785:v1:172.21.15.179:0/3304791412] 2023-10-19T13:01:30.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.297+0000 7f09b9cf2640 1 -- 172.21.15.57:0/582055460 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f09b406ac70 msgr2=0x7f09b406ca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:30.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.297+0000 7f09b9cf2640 1 --2- 172.21.15.57:0/582055460 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f09b406ac70 0x7f09b406ca90 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f09b40662b0 tx=0x7f09ac02c250 comp rx=0 tx=0).stop 2023-10-19T13:01:30.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.297+0000 7f09b9cf2640 1 -- 172.21.15.57:0/582055460 shutdown_connections 2023-10-19T13:01:30.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.297+0000 7f09b9cf2640 1 --2- 172.21.15.57:0/582055460 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f09b406ac70 0x7f09b406ca90 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:30.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.297+0000 7f09b9cf2640 1 --2- 172.21.15.57:0/582055460 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f09b415c1f0 0x7f09b406a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:30.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.297+0000 7f09b9cf2640 1 --2- 172.21.15.57:0/582055460 >> v2:172.21.15.179:3300/0 conn(0x7f09b41579f0 0x7f09b40b0e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:30.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.297+0000 7f09b9cf2640 1 -- 172.21.15.57:0/582055460 >> 172.21.15.57:0/582055460 conn(0x7f09b406fae0 msgr2=0x7f09b406fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:30.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.297+0000 7f09b9cf2640 1 -- 172.21.15.57:0/582055460 shutdown_connections 2023-10-19T13:01:30.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.297+0000 7f09b9cf2640 1 -- 172.21.15.57:0/582055460 wait complete. 2023-10-19T13:01:30.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.298+0000 7f09b9cf2640 1 Processor -- start 2023-10-19T13:01:30.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.298+0000 7f09b9cf2640 1 -- start start 2023-10-19T13:01:30.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.299+0000 7f09b9cf2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f09b406ac70 0x7f09b41e3d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:30.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.299+0000 7f09b9cf2640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f09b41579f0 0x7f09b40ae3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:30.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.299+0000 7f09b9cf2640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f09b415c1f0 0x7f09b4147910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:30.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.299+0000 7f09b9cf2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09b41e4240 con 0x7f09b406ac70 2023-10-19T13:01:30.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.299+0000 7f09b9cf2640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09b40ae910 con 0x7f09b415c1f0 2023-10-19T13:01:30.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.299+0000 7f09b9cf2640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09b40ad330 con 0x7f09b41579f0 2023-10-19T13:01:30.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.299+0000 7f09b37fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f09b406ac70 0x7f09b41e3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:30.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.299+0000 7f09b37fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f09b406ac70 0x7f09b41e3d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40142/0 (socket says 172.21.15.57:40142) 2023-10-19T13:01:30.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.299+0000 7f09b37fe640 1 -- 172.21.15.57:0/3362549414 learned_addr learned my addr 172.21.15.57:0/3362549414 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:01:30.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.299+0000 7f09b2ffd640 1 --2- 172.21.15.57:0/3362549414 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f09b41579f0 0x7f09b40ae3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:30.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.299+0000 7f09b3fff640 1 --2- 172.21.15.57:0/3362549414 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f09b415c1f0 0x7f09b4147910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:30.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.299+0000 7f09b37fe640 1 -- 172.21.15.57:0/3362549414 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f09b41579f0 msgr2=0x7f09b40ae3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:30.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.299+0000 7f09b37fe640 1 --2- 172.21.15.57:0/3362549414 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f09b41579f0 0x7f09b40ae3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:30.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.299+0000 7f09b37fe640 1 -- 172.21.15.57:0/3362549414 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f09b415c1f0 msgr2=0x7f09b4147910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:30.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.299+0000 7f09b37fe640 1 --2- 172.21.15.57:0/3362549414 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f09b415c1f0 0x7f09b4147910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:30.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.299+0000 7f09b37fe640 1 -- 172.21.15.57:0/3362549414 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09ac009900 con 0x7f09b406ac70 2023-10-19T13:01:30.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.300+0000 7f09b37fe640 1 --2- 172.21.15.57:0/3362549414 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f09b406ac70 0x7f09b41e3d00 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7f09a400c920 tx=0x7f09a400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:30.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.300+0000 7f099bfff640 1 -- 172.21.15.57:0/3362549414 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09a4007c70 con 0x7f09b406ac70 2023-10-19T13:01:30.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.300+0000 7f099bfff640 1 -- 172.21.15.57:0/3362549414 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f09a400f450 con 0x7f09b406ac70 2023-10-19T13:01:30.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.300+0000 7f09b9cf2640 1 -- 172.21.15.57:0/3362549414 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09b40b2330 con 0x7f09b406ac70 2023-10-19T13:01:30.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.300+0000 7f099bfff640 1 -- 172.21.15.57:0/3362549414 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09a400f860 con 0x7f09b406ac70 2023-10-19T13:01:30.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.300+0000 7f09b9cf2640 1 -- 172.21.15.57:0/3362549414 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09b40adbd0 con 0x7f09b406ac70 2023-10-19T13:01:30.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.301+0000 7f09b9cf2640 1 -- 172.21.15.57:0/3362549414 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09b40690e0 con 0x7f09b406ac70 2023-10-19T13:01:30.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.301+0000 7f099bfff640 1 -- 172.21.15.57:0/3362549414 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 47) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f09a400f9c0 con 0x7f09b406ac70 2023-10-19T13:01:30.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.302+0000 7f099bfff640 1 --2- 172.21.15.57:0/3362549414 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f09a006aed0 0x7f09a006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:30.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.302+0000 7f099bfff640 1 -- 172.21.15.57:0/3362549414 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f09a408a0d0 con 0x7f09b406ac70 2023-10-19T13:01:30.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.302+0000 7f09b2ffd640 1 --2- 172.21.15.57:0/3362549414 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f09a006aed0 0x7f09a006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:30.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.302+0000 7f09b2ffd640 1 --2- 172.21.15.57:0/3362549414 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f09a006aed0 0x7f09a006d390 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f09b41e44b0 tx=0x7f09a800a2d0 comp rx=0 tx=0).ready entity=mgr.7072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:30.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.306+0000 7f099bfff640 1 -- 172.21.15.57:0/3362549414 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f09a4054f40 con 0x7f09b406ac70 2023-10-19T13:01:30.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.561+0000 7f09b9cf2640 1 -- 172.21.15.57:0/3362549414 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f09b40b1d80 con 0x7f09b406ac70 2023-10-19T13:01:30.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.563+0000 7f099bfff640 1 -- 172.21.15.57:0/3362549414 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v26) v1 ==== 190+0+147 (secure 0 0 0) 0x7f09a4058560 con 0x7f09b406ac70 2023-10-19T13:01:30.566 INFO:teuthology.orchestra.run.smithi057.stdout:[client.0] 2023-10-19T13:01:30.566 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQBkJDFlg9yuExAATxiWDfS+p07G1F0pORbJEg== 2023-10-19T13:01:30.566 INFO:teuthology.orchestra.run.smithi057.stdout: caps mds = "allow" 2023-10-19T13:01:30.567 INFO:teuthology.orchestra.run.smithi057.stdout: caps mon = "allow r" 2023-10-19T13:01:30.567 INFO:teuthology.orchestra.run.smithi057.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-19T13:01:30.567 INFO:teuthology.orchestra.run.smithi057.stderr:updated caps for client.0 2023-10-19T13:01:30.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.565+0000 7f09b9cf2640 1 -- 172.21.15.57:0/3362549414 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f09a006aed0 msgr2=0x7f09a006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:30.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.565+0000 7f09b9cf2640 1 --2- 172.21.15.57:0/3362549414 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f09a006aed0 0x7f09a006d390 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f09b41e44b0 tx=0x7f09a800a2d0 comp rx=0 tx=0).stop 2023-10-19T13:01:30.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.565+0000 7f09b9cf2640 1 -- 172.21.15.57:0/3362549414 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f09b406ac70 msgr2=0x7f09b41e3d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:30.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.565+0000 7f09b9cf2640 1 --2- 172.21.15.57:0/3362549414 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f09b406ac70 0x7f09b41e3d00 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7f09a400c920 tx=0x7f09a400cde0 comp rx=0 tx=0).stop 2023-10-19T13:01:30.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.565+0000 7f09b9cf2640 1 -- 172.21.15.57:0/3362549414 shutdown_connections 2023-10-19T13:01:30.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.565+0000 7f09b9cf2640 1 --2- 172.21.15.57:0/3362549414 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f09a006aed0 0x7f09a006d390 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:30.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.565+0000 7f09b9cf2640 1 --2- 172.21.15.57:0/3362549414 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f09b415c1f0 0x7f09b4147910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:30.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.565+0000 7f09b9cf2640 1 --2- 172.21.15.57:0/3362549414 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f09b41579f0 0x7f09b40ae3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:30.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.565+0000 7f09b9cf2640 1 --2- 172.21.15.57:0/3362549414 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f09b406ac70 0x7f09b41e3d00 unknown :-1 s=CLOSED pgs=1294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:30.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.566+0000 7f09b9cf2640 1 -- 172.21.15.57:0/3362549414 >> 172.21.15.57:0/3362549414 conn(0x7f09b406fae0 msgr2=0x7f09b40cbc40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:30.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.566+0000 7f09b9cf2640 1 -- 172.21.15.57:0/3362549414 shutdown_connections 2023-10-19T13:01:30.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.566+0000 7f09b9cf2640 1 -- 172.21.15.57:0/3362549414 wait complete. 2023-10-19T13:01:30.579 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T13:01:30.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.688+0000 7fa64f17d640 1 -- 172.21.15.57:0/968006818 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa64815b2a0 msgr2=0x7fa6480cf720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:30.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.688+0000 7fa64f17d640 1 --2- 172.21.15.57:0/968006818 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa64815b2a0 0x7fa6480cf720 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7fa64808a810 tx=0x7fa63402bcf0 comp rx=0 tx=0).stop 2023-10-19T13:01:30.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.688+0000 7fa64f17d640 1 -- 172.21.15.57:0/968006818 shutdown_connections 2023-10-19T13:01:30.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.688+0000 7fa64f17d640 1 --2- 172.21.15.57:0/968006818 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa64815b2a0 0x7fa6480cf720 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:30.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.688+0000 7fa64f17d640 1 --2- 172.21.15.57:0/968006818 >> v2:172.21.15.179:3300/0 conn(0x7fa6480cf340 0x7fa6480d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:30.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.688+0000 7fa64f17d640 1 -- 172.21.15.57:0/968006818 >> 172.21.15.57:0/968006818 conn(0x7fa6480d0d60 msgr2=0x7fa6480d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:30.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.688+0000 7fa64f17d640 1 -- 172.21.15.57:0/968006818 shutdown_connections 2023-10-19T13:01:30.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.688+0000 7fa64f17d640 1 -- 172.21.15.57:0/968006818 wait complete. 2023-10-19T13:01:30.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.690+0000 7fa64f17d640 1 Processor -- start 2023-10-19T13:01:30.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.690+0000 7fa64f17d640 1 -- start start 2023-10-19T13:01:30.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.690+0000 7fa64f17d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa6480ceb40 0x7fa648156600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:30.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.690+0000 7fa64f17d640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa6480cf340 0x7fa6480b5f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:30.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.690+0000 7fa64f17d640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa64815b2a0 0x7fa6480ca940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:30.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.690+0000 7fa64f17d640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa648146e70 con 0x7fa6480ceb40 2023-10-19T13:01:30.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.690+0000 7fa64f17d640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6481479f0 con 0x7fa64815b2a0 2023-10-19T13:01:30.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.690+0000 7fa64f17d640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6481470c0 con 0x7fa6480cf340 2023-10-19T13:01:30.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.691+0000 7fa64cf19640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa6480ceb40 0x7fa648156600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:30.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.691+0000 7fa64cf19640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa6480ceb40 0x7fa648156600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40158/0 (socket says 172.21.15.57:40158) 2023-10-19T13:01:30.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.691+0000 7fa64cf19640 1 -- 172.21.15.57:0/719542150 learned_addr learned my addr 172.21.15.57:0/719542150 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:01:30.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.691+0000 7fa647fff640 1 --2- 172.21.15.57:0/719542150 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa6480cf340 0x7fa6480b5f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:30.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.691+0000 7fa64d71a640 1 --2- 172.21.15.57:0/719542150 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa64815b2a0 0x7fa6480ca940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:30.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.691+0000 7fa64cf19640 1 -- 172.21.15.57:0/719542150 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa6480cf340 msgr2=0x7fa6480b5f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:30.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.691+0000 7fa64cf19640 1 --2- 172.21.15.57:0/719542150 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa6480cf340 0x7fa6480b5f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:30.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.691+0000 7fa64cf19640 1 -- 172.21.15.57:0/719542150 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa64815b2a0 msgr2=0x7fa6480ca940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:30.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.691+0000 7fa64cf19640 1 --2- 172.21.15.57:0/719542150 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa64815b2a0 0x7fa6480ca940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:30.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.691+0000 7fa64cf19640 1 -- 172.21.15.57:0/719542150 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa634009490 con 0x7fa6480ceb40 2023-10-19T13:01:30.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.691+0000 7fa64cf19640 1 --2- 172.21.15.57:0/719542150 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa6480ceb40 0x7fa648156600 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7fa63c00ea20 tx=0x7fa63c00eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:30.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.691+0000 7fa6457fa640 1 -- 172.21.15.57:0/719542150 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa63c009700 con 0x7fa6480ceb40 2023-10-19T13:01:30.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.691+0000 7fa6457fa640 1 -- 172.21.15.57:0/719542150 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fa63c00ce70 con 0x7fa6480ceb40 2023-10-19T13:01:30.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.691+0000 7fa64f17d640 1 -- 172.21.15.57:0/719542150 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6481473a0 con 0x7fa6480ceb40 2023-10-19T13:01:30.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.691+0000 7fa6457fa640 1 -- 172.21.15.57:0/719542150 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa63c010690 con 0x7fa6480ceb40 2023-10-19T13:01:30.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.691+0000 7fa64f17d640 1 -- 172.21.15.57:0/719542150 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6480b6490 con 0x7fa6480ceb40 2023-10-19T13:01:30.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.692+0000 7fa64f17d640 1 -- 172.21.15.57:0/719542150 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6480690e0 con 0x7fa6480ceb40 2023-10-19T13:01:30.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.693+0000 7fa6457fa640 1 -- 172.21.15.57:0/719542150 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 47) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fa63c0107f0 con 0x7fa6480ceb40 2023-10-19T13:01:30.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.693+0000 7fa6457fa640 1 --2- 172.21.15.57:0/719542150 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7fa62c06aed0 0x7fa62c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:30.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.693+0000 7fa6457fa640 1 -- 172.21.15.57:0/719542150 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fa63c014070 con 0x7fa6480ceb40 2023-10-19T13:01:30.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.693+0000 7fa647fff640 1 --2- 172.21.15.57:0/719542150 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7fa62c06aed0 0x7fa62c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:30.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.694+0000 7fa647fff640 1 --2- 172.21.15.57:0/719542150 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7fa62c06aed0 0x7fa62c06d390 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa634036000 tx=0x7fa634004730 comp rx=0 tx=0).ready entity=mgr.7072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:30.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.697+0000 7fa6457fa640 1 -- 172.21.15.57:0/719542150 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa63c051ca0 con 0x7fa6480ceb40 2023-10-19T13:01:30.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.915+0000 7fa64f17d640 1 -- 172.21.15.57:0/719542150 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa6480cb980 con 0x7fa6480ceb40 2023-10-19T13:01:30.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.916+0000 7fa6457fa640 1 -- 172.21.15.57:0/719542150 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+4921 (secure 0 0 0) 0x7fa63c055320 con 0x7fa6480ceb40 2023-10-19T13:01:30.919 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:01:30.920 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":44,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6611,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/2708395824","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2708395824},{"type":"v1","addr":"172.21.15.179:6836","nonce":2708395824}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":41},{"gid":7270,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/3803632191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":3803632191},{"type":"v1","addr":"172.21.15.179:6837","nonce":3803632191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":41},{"gid":7282,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6833/2259979700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2259979700},{"type":"v1","addr":"172.21.15.57:6833","nonce":2259979700}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T13:01:26.118951+0000","modified":"2023-10-19T13:01:28.134083+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8046},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8046":{"gid":8046,"name":"c","rank":0,"incarnation":42,"state":"up:active","state_seq":3,"addr":"172.21.15.57:6835/2556427177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":2556427177},{"type":"v1","addr":"172.21.15.57:6835","nonce":2556427177}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-10-19T13:01:30.921 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 44 2023-10-19T13:01:30.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.918+0000 7fa64f17d640 1 -- 172.21.15.57:0/719542150 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7fa62c06aed0 msgr2=0x7fa62c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:30.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.918+0000 7fa64f17d640 1 --2- 172.21.15.57:0/719542150 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7fa62c06aed0 0x7fa62c06d390 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa634036000 tx=0x7fa634004730 comp rx=0 tx=0).stop 2023-10-19T13:01:30.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.918+0000 7fa64f17d640 1 -- 172.21.15.57:0/719542150 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa6480ceb40 msgr2=0x7fa648156600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:30.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.918+0000 7fa64f17d640 1 --2- 172.21.15.57:0/719542150 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa6480ceb40 0x7fa648156600 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7fa63c00ea20 tx=0x7fa63c00eee0 comp rx=0 tx=0).stop 2023-10-19T13:01:30.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.918+0000 7fa64f17d640 1 -- 172.21.15.57:0/719542150 shutdown_connections 2023-10-19T13:01:30.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.918+0000 7fa64f17d640 1 --2- 172.21.15.57:0/719542150 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7fa62c06aed0 0x7fa62c06d390 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:30.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.918+0000 7fa64f17d640 1 --2- 172.21.15.57:0/719542150 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa64815b2a0 0x7fa6480ca940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:30.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.918+0000 7fa64f17d640 1 --2- 172.21.15.57:0/719542150 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa6480cf340 0x7fa6480b5f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:30.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.918+0000 7fa64f17d640 1 --2- 172.21.15.57:0/719542150 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa6480ceb40 0x7fa648156600 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:30.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.918+0000 7fa64f17d640 1 -- 172.21.15.57:0/719542150 >> 172.21.15.57:0/719542150 conn(0x7fa6480d0d60 msgr2=0x7fa6480ad8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:30.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.918+0000 7fa64f17d640 1 -- 172.21.15.57:0/719542150 shutdown_connections 2023-10-19T13:01:30.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:30.918+0000 7fa64f17d640 1 -- 172.21.15.57:0/719542150 wait complete. 2023-10-19T13:01:30.989 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:30.986+0000 7fb508163640 -1 client.7141: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.7030:f55302c0-37aa-4235-8ccc-9775bc542785:v1:172.21.15.179:0/3304791412] 2023-10-19T13:01:31.334 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 44, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-19T13:01:26.118951+0000', 'modified': '2023-10-19T13:01:28.134083+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8046}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8046': {'gid': 8046, 'name': 'c', 'rank': 0, 'incarnation': 42, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.57:6835/2556427177', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 2556427177}, {'type': 'v1', 'addr': '172.21.15.57:6835', 'nonce': 2556427177}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [14], 'metadata_pool': 13, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-19T13:01:31.335 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-19T13:01:31.335 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-10-19T13:01:31.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.446+0000 7f0b01226640 1 -- 172.21.15.57:0/2461232197 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0afc06a730 msgr2=0x7f0afc06abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:31.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.446+0000 7f0b01226640 1 --2- 172.21.15.57:0/2461232197 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0afc06a730 0x7f0afc06abc0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f0aec009be0 tx=0x7f0aec02c0c0 comp rx=0 tx=0).stop 2023-10-19T13:01:31.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.446+0000 7f0b01226640 1 -- 172.21.15.57:0/2461232197 shutdown_connections 2023-10-19T13:01:31.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.446+0000 7f0b01226640 1 --2- 172.21.15.57:0/2461232197 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0afc06a730 0x7f0afc06abc0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:31.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.446+0000 7f0b01226640 1 --2- 172.21.15.57:0/2461232197 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0afc156790 0x7f0afc0cc990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:31.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.446+0000 7f0b01226640 1 -- 172.21.15.57:0/2461232197 >> 172.21.15.57:0/2461232197 conn(0x7f0afc06fae0 msgr2=0x7f0afc06fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:31.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.446+0000 7f0b01226640 1 -- 172.21.15.57:0/2461232197 shutdown_connections 2023-10-19T13:01:31.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.446+0000 7f0b01226640 1 -- 172.21.15.57:0/2461232197 wait complete. 2023-10-19T13:01:31.453 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.448+0000 7f0b01226640 1 Processor -- start 2023-10-19T13:01:31.453 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.448+0000 7f0b01226640 1 -- start start 2023-10-19T13:01:31.453 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.448+0000 7f0b01226640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0afc06a730 0x7f0afc0c23f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:31.453 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.448+0000 7f0b01226640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0afc0ac870 0x7f0afc0aab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:31.453 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.448+0000 7f0b01226640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0afc156790 0x7f0afc0b1430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:31.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.448+0000 7f0b01226640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0afc0b25a0 con 0x7f0afc156790 2023-10-19T13:01:31.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.448+0000 7f0b01226640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0afc0b06b0 con 0x7f0afc0ac870 2023-10-19T13:01:31.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.448+0000 7f0b01226640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0afc0b0950 con 0x7f0afc06a730 2023-10-19T13:01:31.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.448+0000 7f0afb59e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0afc156790 0x7f0afc0b1430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:31.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.449+0000 7f0afb59e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0afc156790 0x7f0afc0b1430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40170/0 (socket says 172.21.15.57:40170) 2023-10-19T13:01:31.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.449+0000 7f0afb59e640 1 -- 172.21.15.57:0/1347056663 learned_addr learned my addr 172.21.15.57:0/1347056663 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:01:31.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.449+0000 7f0afad9d640 1 --2- 172.21.15.57:0/1347056663 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0afc06a730 0x7f0afc0c23f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:31.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.449+0000 7f0afa59c640 1 --2- 172.21.15.57:0/1347056663 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0afc0ac870 0x7f0afc0aab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:31.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.449+0000 7f0afb59e640 1 -- 172.21.15.57:0/1347056663 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0afc06a730 msgr2=0x7f0afc0c23f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:31.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.449+0000 7f0afb59e640 1 --2- 172.21.15.57:0/1347056663 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0afc06a730 0x7f0afc0c23f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:31.456 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.449+0000 7f0afb59e640 1 -- 172.21.15.57:0/1347056663 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0afc0ac870 msgr2=0x7f0afc0aab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:31.456 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.449+0000 7f0afb59e640 1 --2- 172.21.15.57:0/1347056663 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0afc0ac870 0x7f0afc0aab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:31.456 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.449+0000 7f0afb59e640 1 -- 172.21.15.57:0/1347056663 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0af4009490 con 0x7f0afc156790 2023-10-19T13:01:31.456 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.449+0000 7f0afb59e640 1 --2- 172.21.15.57:0/1347056663 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0afc156790 0x7f0afc0b1430 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f0aec009be0 tx=0x7f0aec0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:31.456 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.449+0000 7f0af37fe640 1 -- 172.21.15.57:0/1347056663 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aec0046f0 con 0x7f0afc156790 2023-10-19T13:01:31.456 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.449+0000 7f0af37fe640 1 -- 172.21.15.57:0/1347056663 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f0aec0345a0 con 0x7f0afc156790 2023-10-19T13:01:31.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.449+0000 7f0b01226640 1 -- 172.21.15.57:0/1347056663 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0aec009840 con 0x7f0afc156790 2023-10-19T13:01:31.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.449+0000 7f0af37fe640 1 -- 172.21.15.57:0/1347056663 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aec0349b0 con 0x7f0afc156790 2023-10-19T13:01:31.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.449+0000 7f0b01226640 1 -- 172.21.15.57:0/1347056663 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0afc0b0f10 con 0x7f0afc156790 2023-10-19T13:01:31.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.451+0000 7f0af37fe640 1 -- 172.21.15.57:0/1347056663 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 47) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f0aec034b10 con 0x7f0afc156790 2023-10-19T13:01:31.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.451+0000 7f0af37fe640 1 --2- 172.21.15.57:0/1347056663 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f0adc06aed0 0x7f0adc06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:31.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.451+0000 7f0af37fe640 1 -- 172.21.15.57:0/1347056663 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f0aec0ac810 con 0x7f0afc156790 2023-10-19T13:01:31.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.451+0000 7f0afad9d640 1 --2- 172.21.15.57:0/1347056663 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f0adc06aed0 0x7f0adc06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:31.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.452+0000 7f0afad9d640 1 --2- 172.21.15.57:0/1347056663 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f0adc06aed0 0x7f0adc06d390 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0af4009d60 tx=0x7f0af4005f90 comp rx=0 tx=0).ready entity=mgr.7072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:31.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.453+0000 7f0b03dba740 1 Processor -- start 2023-10-19T13:01:31.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.453+0000 7f0b03dba740 1 -- start start 2023-10-19T13:01:31.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.454+0000 7f0b03dba740 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55f24afbeb40 0x55f24af91ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:31.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.454+0000 7f0b03dba740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55f24afbef20 0x55f24afc18d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:31.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.454+0000 7f0b03dba740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f24af92020 0x55f24afc1e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:31.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.454+0000 7f0b03dba740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f24aea3730 con 0x55f24afbef20 2023-10-19T13:01:31.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.454+0000 7f0b03dba740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f24af04780 con 0x55f24af92020 2023-10-19T13:01:31.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.454+0000 7f0b03dba740 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f24af01e70 con 0x55f24afbeb40 2023-10-19T13:01:31.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.454+0000 7f0afa59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55f24afbef20 0x55f24afc18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:31.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.454+0000 7f0afad9d640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55f24afbeb40 0x55f24af91ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:31.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.454+0000 7f0afa59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55f24afbef20 0x55f24afc18d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40172/0 (socket says 172.21.15.57:40172) 2023-10-19T13:01:31.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.454+0000 7f0afb59e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f24af92020 0x55f24afc1e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:31.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.454+0000 7f0afa59c640 1 -- 172.21.15.57:0/2074052789 learned_addr learned my addr 172.21.15.57:0/2074052789 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:01:31.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.454+0000 7f0afb59e640 1 -- 172.21.15.57:0/2074052789 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55f24afbeb40 msgr2=0x55f24af91ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T13:01:31.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.454+0000 7f0afb59e640 1 --2- 172.21.15.57:0/2074052789 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55f24afbeb40 0x55f24af91ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:31.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.454+0000 7f0afb59e640 1 -- 172.21.15.57:0/2074052789 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55f24afbef20 msgr2=0x55f24afc18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T13:01:31.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.454+0000 7f0afb59e640 1 --2- 172.21.15.57:0/2074052789 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55f24afbef20 0x55f24afc18d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:31.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.454+0000 7f0afb59e640 1 -- 172.21.15.57:0/2074052789 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f24af023d0 con 0x55f24af92020 2023-10-19T13:01:31.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.454+0000 7f0afb59e640 1 --2- 172.21.15.57:0/2074052789 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f24af92020 0x55f24afc1e10 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f0aec0aac60 tx=0x7f0aec07b0f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=92524c15e8bba1fe server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:31.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.454+0000 7f0af0ff9640 1 -- 172.21.15.57:0/2074052789 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aec07f590 con 0x55f24af92020 2023-10-19T13:01:31.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.454+0000 7f0af0ff9640 1 -- 172.21.15.57:0/2074052789 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f0aec079890 con 0x55f24af92020 2023-10-19T13:01:31.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.454+0000 7f0af0ff9640 1 -- 172.21.15.57:0/2074052789 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aec07f170 con 0x55f24af92020 2023-10-19T13:01:31.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.455+0000 7f0b03dba740 1 -- 172.21.15.57:0/2074052789 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f24af92020 msgr2=0x55f24afc1e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:31.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.455+0000 7f0b03dba740 1 --2- 172.21.15.57:0/2074052789 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f24af92020 0x55f24afc1e10 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f0aec0aac60 tx=0x7f0aec07b0f0 comp rx=0 tx=0).stop 2023-10-19T13:01:31.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.455+0000 7f0b03dba740 1 -- 172.21.15.57:0/2074052789 shutdown_connections 2023-10-19T13:01:31.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.455+0000 7f0b03dba740 1 --2- 172.21.15.57:0/2074052789 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f24af92020 0x55f24afc1e10 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:31.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.455+0000 7f0b03dba740 1 --2- 172.21.15.57:0/2074052789 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55f24afbef20 0x55f24afc18d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:31.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.455+0000 7f0b03dba740 1 --2- 172.21.15.57:0/2074052789 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55f24afbeb40 0x55f24af91ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:31.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.455+0000 7f0b03dba740 1 -- 172.21.15.57:0/2074052789 >> 172.21.15.57:0/2074052789 conn(0x55f24af93d50 msgr2=0x55f24afdc8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:31.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.455+0000 7f0b03dba740 1 -- 172.21.15.57:0/2074052789 shutdown_connections 2023-10-19T13:01:31.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.455+0000 7f0b03dba740 1 -- 172.21.15.57:0/2074052789 wait complete. 2023-10-19T13:01:31.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.456+0000 7f0b03dba740 1 -- start start 2023-10-19T13:01:31.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.456+0000 7f0b03dba740 1 Processor -- start 2023-10-19T13:01:31.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.457+0000 7f0b03dba740 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55f24afbeb40 0x55f24afac2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:31.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.457+0000 7f0b03dba740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f24afbef20 0x55f24afac7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:31.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.457+0000 7f0b03dba740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55f24afd6f00 0x55f24afaf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:31.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.457+0000 7f0b03dba740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f24aefd760 con 0x55f24afd6f00 2023-10-19T13:01:31.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.457+0000 7f0b03dba740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f24aeb88f0 con 0x55f24afbef20 2023-10-19T13:01:31.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.457+0000 7f0b03dba740 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f24aeb8590 con 0x55f24afbeb40 2023-10-19T13:01:31.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.457+0000 7f0b03dba740 10 client.0 _ll_register_callbacks cb 0x55f24aef4ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-19T13:01:31.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.457+0000 7f0afb59e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55f24afd6f00 0x55f24afaf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:31.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.457+0000 7f0afb59e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55f24afd6f00 0x55f24afaf3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40174/0 (socket says 172.21.15.57:40174) 2023-10-19T13:01:31.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.457+0000 7f0afb59e640 1 -- 172.21.15.57:0/394693315 learned_addr learned my addr 172.21.15.57:0/394693315 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:01:31.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.457+0000 7f0afad9d640 1 --2- 172.21.15.57:0/394693315 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55f24afbeb40 0x55f24afac2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:31.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.457+0000 7f0afa59c640 1 --2- 172.21.15.57:0/394693315 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f24afbef20 0x55f24afac7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:31.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.457+0000 7f0afb59e640 1 -- 172.21.15.57:0/394693315 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55f24afbeb40 msgr2=0x55f24afac2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T13:01:31.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.457+0000 7f0afb59e640 1 --2- 172.21.15.57:0/394693315 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55f24afbeb40 0x55f24afac2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:31.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.457+0000 7f0afb59e640 1 -- 172.21.15.57:0/394693315 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f24afbef20 msgr2=0x55f24afac7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T13:01:31.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.457+0000 7f0afb59e640 1 --2- 172.21.15.57:0/394693315 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f24afbef20 0x55f24afac7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:31.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.457+0000 7f0afb59e640 1 -- 172.21.15.57:0/394693315 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0afc0b0f10 con 0x55f24afd6f00 2023-10-19T13:01:31.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.457+0000 7f0afb59e640 1 --2- 172.21.15.57:0/394693315 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55f24afd6f00 0x55f24afaf3c0 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f0aec0779f0 tx=0x7f0aec02c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17bc018591201b16 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:31.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.457+0000 7f0ad9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2023-10-19T13:01:31.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.457+0000 7f0ad9ffb640 1 -- 172.21.15.57:0/394693315 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aec098070 con 0x55f24afd6f00 2023-10-19T13:01:31.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.457+0000 7f0ad9ffb640 1 -- 172.21.15.57:0/394693315 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(113..113 src has 1..113) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f0aec090020 con 0x55f24afd6f00 2023-10-19T13:01:31.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.457+0000 7f0b03dba740 1 -- 172.21.15.57:0/394693315 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f24aef6740 con 0x55f24afd6f00 2023-10-19T13:01:31.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.458+0000 7f0ad9ffb640 1 -- 172.21.15.57:0/394693315 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0aec07f370 con 0x55f24afd6f00 2023-10-19T13:01:31.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.458+0000 7f0b03dba740 10 client.8178 fetch_fsmap learned FSMap version 44 2023-10-19T13:01:31.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.458+0000 7f0b03dba740 1 -- 172.21.15.57:0/394693315 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=2+}) v3 -- 0x55f24aefbde0 con 0x55f24afd6f00 2023-10-19T13:01:31.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.458+0000 7f0ad9ffb640 1 -- 172.21.15.57:0/394693315 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x55f24aef6740 con 0x55f24afd6f00 2023-10-19T13:01:31.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.458+0000 7f0ad9ffb640 1 -- 172.21.15.57:0/394693315 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 44) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f0aec04f020 con 0x55f24afd6f00 2023-10-19T13:01:31.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.458+0000 7f0b03dba740 10 client.8178 fetch_fsmap finished waiting for FSMap version 44 2023-10-19T13:01:31.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.458+0000 7f0b03dba740 10 client.8178 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-19T13:01:31.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.458+0000 7f0b03dba740 20 client.8178 populate_metadata read hostname 'smithi057' 2023-10-19T13:01:31.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.458+0000 7f0b03dba740 1 --2- 172.21.15.57:0/394693315 >> [v2:172.21.15.57:6834/2556427177,v1:172.21.15.57:6835/2556427177] conn(0x55f24afc4210 0x55f24afade20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:31.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.458+0000 7f0b03dba740 4 client.8178 mds_command: new command op to 8046 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-19T13:01:31.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.458+0000 7f0b03dba740 1 -- 172.21.15.57:0/394693315 --> [v2:172.21.15.57:6834/2556427177,v1:172.21.15.57:6835/2556427177] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f24aef6460 con 0x55f24afc4210 2023-10-19T13:01:31.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.458+0000 7f0afad9d640 1 --2- 172.21.15.57:0/394693315 >> [v2:172.21.15.57:6834/2556427177,v1:172.21.15.57:6835/2556427177] conn(0x55f24afc4210 0x55f24afade20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:31.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.458+0000 7f0ad9ffb640 10 client.8178 ms_handle_connect on v2:172.21.15.57:6834/2556427177 2023-10-19T13:01:31.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.458+0000 7f0afad9d640 1 --2- 172.21.15.57:0/394693315 >> [v2:172.21.15.57:6834/2556427177,v1:172.21.15.57:6835/2556427177] conn(0x55f24afc4210 0x55f24afade20 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c4cf52567b136adf server_cookie=a41997c508126376 in_seq=0 out_seq=0 2023-10-19T13:01:31.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.459+0000 7f0ad9ffb640 1 -- 172.21.15.57:0/394693315 <== mds.0 v2:172.21.15.57:6834/2556427177 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55f24aef6460 con 0x55f24afc4210 2023-10-19T13:01:31.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.459+0000 7f0ad9ffb640 10 client.8178 handle_command_reply: tid=0 2023-10-19T13:01:31.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.459+0000 7f0b03dba740 1 client.8178 shutdown 2023-10-19T13:01:31.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.459+0000 7f0b03dba740 1 -- 172.21.15.57:0/394693315 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55f24afd6f00 msgr2=0x55f24afaf3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:31.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.459+0000 7f0b03dba740 1 --2- 172.21.15.57:0/394693315 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55f24afd6f00 0x55f24afaf3c0 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f0aec0779f0 tx=0x7f0aec02c760 comp rx=0 tx=0).stop 2023-10-19T13:01:31.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.459+0000 7f0b03dba740 1 -- 172.21.15.57:0/394693315 shutdown_connections 2023-10-19T13:01:31.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.459+0000 7f0b03dba740 1 --2- 172.21.15.57:0/394693315 >> [v2:172.21.15.57:6834/2556427177,v1:172.21.15.57:6835/2556427177] conn(0x55f24afc4210 0x55f24afade20 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:31.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.459+0000 7f0b03dba740 1 --2- 172.21.15.57:0/394693315 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55f24afd6f00 0x55f24afaf3c0 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:31.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.459+0000 7f0b03dba740 1 --2- 172.21.15.57:0/394693315 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f24afbef20 0x55f24afac7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:31.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.459+0000 7f0ad9ffb640 0 client.8178 ms_handle_reset on v2:172.21.15.57:6834/2556427177 2023-10-19T13:01:31.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.459+0000 7f0b03dba740 1 --2- 172.21.15.57:0/394693315 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55f24afbeb40 0x55f24afac2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:31.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.459+0000 7f0b03dba740 1 -- 172.21.15.57:0/394693315 >> 172.21.15.57:0/394693315 conn(0x55f24af92020 msgr2=0x55f24afc18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:31.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.459+0000 7f0b03dba740 1 -- 172.21.15.57:0/394693315 shutdown_connections 2023-10-19T13:01:31.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.459+0000 7f0b03dba740 1 -- 172.21.15.57:0/394693315 wait complete. 2023-10-19T13:01:31.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.459+0000 7f0b03dba740 20 client.8178 trim_cache size 0 max 16384 2023-10-19T13:01:31.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.475+0000 7f0b03dba740 1 Processor -- start 2023-10-19T13:01:31.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.475+0000 7f0b03dba740 1 -- start start 2023-10-19T13:01:31.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.475+0000 7f0b03dba740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55f24afbeb40 0x55f24afdf3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:31.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.475+0000 7f0b03dba740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f24afbef20 0x55f24afac070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:31.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.475+0000 7f0b03dba740 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55f24afd6f00 0x55f24afac5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:31.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.475+0000 7f0b03dba740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f24ae7e270 con 0x55f24afbeb40 2023-10-19T13:01:31.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.475+0000 7f0b03dba740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f24af95c00 con 0x55f24afbef20 2023-10-19T13:01:31.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.475+0000 7f0b03dba740 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f24ae83970 con 0x55f24afd6f00 2023-10-19T13:01:31.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.476+0000 7f0afad9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55f24afbeb40 0x55f24afdf3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:31.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.476+0000 7f0afad9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55f24afbeb40 0x55f24afdf3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40182/0 (socket says 172.21.15.57:40182) 2023-10-19T13:01:31.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.476+0000 7f0afad9d640 1 -- 172.21.15.57:0/1869011348 learned_addr learned my addr 172.21.15.57:0/1869011348 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:01:31.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.476+0000 7f0afb59e640 1 --2- 172.21.15.57:0/1869011348 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55f24afd6f00 0x55f24afac5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:31.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.476+0000 7f0afa59c640 1 --2- 172.21.15.57:0/1869011348 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f24afbef20 0x55f24afac070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:31.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.476+0000 7f0afad9d640 1 -- 172.21.15.57:0/1869011348 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55f24afd6f00 msgr2=0x55f24afac5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T13:01:31.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.476+0000 7f0afad9d640 1 --2- 172.21.15.57:0/1869011348 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55f24afd6f00 0x55f24afac5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:31.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.476+0000 7f0afad9d640 1 -- 172.21.15.57:0/1869011348 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f24afbef20 msgr2=0x55f24afac070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T13:01:31.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.476+0000 7f0afad9d640 1 --2- 172.21.15.57:0/1869011348 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f24afbef20 0x55f24afac070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:31.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.476+0000 7f0afad9d640 1 -- 172.21.15.57:0/1869011348 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f24ae9ad00 con 0x55f24afbeb40 2023-10-19T13:01:31.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.476+0000 7f0afa59c640 1 --2- 172.21.15.57:0/1869011348 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f24afbef20 0x55f24afac070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:01:31.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.476+0000 7f0afad9d640 1 --2- 172.21.15.57:0/1869011348 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55f24afbeb40 0x55f24afdf3b0 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f0af40045a0 tx=0x7f0af4014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a95726978bf3692f server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:31.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.476+0000 7f0af17fa640 1 -- 172.21.15.57:0/1869011348 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0af4021040 con 0x55f24afbeb40 2023-10-19T13:01:31.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.476+0000 7f0af17fa640 1 -- 172.21.15.57:0/1869011348 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f0af4022040 con 0x55f24afbeb40 2023-10-19T13:01:31.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.476+0000 7f0af17fa640 1 -- 172.21.15.57:0/1869011348 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0af4029490 con 0x55f24afbeb40 2023-10-19T13:01:31.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.477+0000 7f0b03dba740 1 -- 172.21.15.57:0/1869011348 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55f24afbeb40 msgr2=0x55f24afdf3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:31.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.477+0000 7f0b03dba740 1 --2- 172.21.15.57:0/1869011348 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55f24afbeb40 0x55f24afdf3b0 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f0af40045a0 tx=0x7f0af4014000 comp rx=0 tx=0).stop 2023-10-19T13:01:31.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.477+0000 7f0b03dba740 1 -- 172.21.15.57:0/1869011348 shutdown_connections 2023-10-19T13:01:31.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.477+0000 7f0b03dba740 1 --2- 172.21.15.57:0/1869011348 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55f24afd6f00 0x55f24afac5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:31.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.477+0000 7f0b03dba740 1 --2- 172.21.15.57:0/1869011348 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f24afbef20 0x55f24afac070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:31.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.477+0000 7f0b03dba740 1 --2- 172.21.15.57:0/1869011348 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55f24afbeb40 0x55f24afdf3b0 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:31.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.477+0000 7f0b03dba740 1 -- 172.21.15.57:0/1869011348 >> 172.21.15.57:0/1869011348 conn(0x55f24af92020 msgr2=0x55f24afdfde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:31.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.477+0000 7f0b03dba740 1 -- 172.21.15.57:0/1869011348 shutdown_connections 2023-10-19T13:01:31.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.477+0000 7f0b03dba740 1 -- 172.21.15.57:0/1869011348 wait complete. 2023-10-19T13:01:31.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.477+0000 7f0b03dba740 1 -- start start 2023-10-19T13:01:31.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.477+0000 7f0b03dba740 1 Processor -- start 2023-10-19T13:01:31.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.478+0000 7f0b03dba740 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55f24afbeb40 0x55f24b044a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:31.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.478+0000 7f0b03dba740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55f24afbef20 0x55f24b048630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:31.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.478+0000 7f0b03dba740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f24afd6f00 0x55f24afad9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:31.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.478+0000 7f0b03dba740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f24b0490b0 con 0x55f24afbef20 2023-10-19T13:01:31.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.478+0000 7f0b03dba740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f24b044fa0 con 0x55f24afd6f00 2023-10-19T13:01:31.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.478+0000 7f0b03dba740 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f24afc50a0 con 0x55f24afbeb40 2023-10-19T13:01:31.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.478+0000 7f0b03dba740 10 client.0 _ll_register_callbacks cb 0x7f0aec02ddd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-19T13:01:31.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.478+0000 7f0afa59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55f24afbef20 0x55f24b048630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:31.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.478+0000 7f0afa59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55f24afbef20 0x55f24b048630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40186/0 (socket says 172.21.15.57:40186) 2023-10-19T13:01:31.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.478+0000 7f0afad9d640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55f24afbeb40 0x55f24b044a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:31.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.478+0000 7f0afa59c640 1 -- 172.21.15.57:0/77541454 learned_addr learned my addr 172.21.15.57:0/77541454 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:01:31.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.478+0000 7f0afb59e640 1 --2- 172.21.15.57:0/77541454 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f24afd6f00 0x55f24afad9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:31.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.479+0000 7f0afa59c640 1 -- 172.21.15.57:0/77541454 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55f24afbeb40 msgr2=0x55f24b044a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T13:01:31.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.479+0000 7f0afa59c640 1 --2- 172.21.15.57:0/77541454 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55f24afbeb40 0x55f24b044a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:31.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.479+0000 7f0afa59c640 1 -- 172.21.15.57:0/77541454 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f24afd6f00 msgr2=0x55f24afad9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T13:01:31.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.479+0000 7f0afa59c640 1 --2- 172.21.15.57:0/77541454 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f24afd6f00 0x55f24afad9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:31.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.479+0000 7f0afa59c640 1 -- 172.21.15.57:0/77541454 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0af4017050 con 0x55f24afbef20 2023-10-19T13:01:31.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.479+0000 7f0afa59c640 1 --2- 172.21.15.57:0/77541454 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55f24afbef20 0x55f24b048630 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x55f24afbf880 tx=0x7f0ae4017af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a415d89a9c41b7b server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:31.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.479+0000 7f0ad9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2023-10-19T13:01:31.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.479+0000 7f0ad9ffb640 1 -- 172.21.15.57:0/77541454 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ae4005260 con 0x55f24afbef20 2023-10-19T13:01:31.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.479+0000 7f0ad9ffb640 1 -- 172.21.15.57:0/77541454 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(113..113 src has 1..113) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f0ae40053c0 con 0x55f24afbef20 2023-10-19T13:01:31.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.479+0000 7f0b03dba740 1 -- 172.21.15.57:0/77541454 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0aec079890 con 0x55f24afbef20 2023-10-19T13:01:31.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.479+0000 7f0ad9ffb640 1 -- 172.21.15.57:0/77541454 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0ae4005580 con 0x55f24afbef20 2023-10-19T13:01:31.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.479+0000 7f0b03dba740 10 client.8184 fetch_fsmap learned FSMap version 44 2023-10-19T13:01:31.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.479+0000 7f0b03dba740 1 -- 172.21.15.57:0/77541454 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=2+}) v3 -- 0x55f24aec04f0 con 0x55f24afbef20 2023-10-19T13:01:31.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.479+0000 7f0ad9ffb640 1 -- 172.21.15.57:0/77541454 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f0aec079890 con 0x55f24afbef20 2023-10-19T13:01:31.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.479+0000 7f0ad9ffb640 1 -- 172.21.15.57:0/77541454 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 44) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f0ae4012040 con 0x55f24afbef20 2023-10-19T13:01:31.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.479+0000 7f0b03dba740 10 client.8184 fetch_fsmap finished waiting for FSMap version 44 2023-10-19T13:01:31.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.479+0000 7f0b03dba740 10 client.8184 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-19T13:01:31.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.479+0000 7f0b03dba740 20 client.8184 populate_metadata read hostname 'smithi057' 2023-10-19T13:01:31.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.479+0000 7f0b03dba740 1 --2- 172.21.15.57:0/77541454 >> [v2:172.21.15.57:6834/2556427177,v1:172.21.15.57:6835/2556427177] conn(0x55f24afeade0 0x55f24afeb1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:31.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.479+0000 7f0b03dba740 4 client.8184 mds_command: new command op to 8046 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-19T13:01:31.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.479+0000 7f0b03dba740 1 -- 172.21.15.57:0/77541454 --> [v2:172.21.15.57:6834/2556427177,v1:172.21.15.57:6835/2556427177] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55f24abff6c0 con 0x55f24afeade0 2023-10-19T13:01:31.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.480+0000 7f0afad9d640 1 --2- 172.21.15.57:0/77541454 >> [v2:172.21.15.57:6834/2556427177,v1:172.21.15.57:6835/2556427177] conn(0x55f24afeade0 0x55f24afeb1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:31.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.480+0000 7f0afad9d640 1 --2- 172.21.15.57:0/77541454 >> [v2:172.21.15.57:6834/2556427177,v1:172.21.15.57:6835/2556427177] conn(0x55f24afeade0 0x55f24afeb1c0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cde2d6d7178582d3 server_cookie=685a6164e9f8897b in_seq=0 out_seq=0 2023-10-19T13:01:31.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.480+0000 7f0ad9ffb640 10 client.8184 ms_handle_connect on v2:172.21.15.57:6834/2556427177 2023-10-19T13:01:31.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.480+0000 7f0ad9ffb640 1 -- 172.21.15.57:0/77541454 <== mds.0 v2:172.21.15.57:6834/2556427177 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x55f24abff6c0 con 0x55f24afeade0 2023-10-19T13:01:31.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.480+0000 7f0ad9ffb640 10 client.8184 handle_command_reply: tid=0 2023-10-19T13:01:31.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.480+0000 7f0b03dba740 1 client.8184 shutdown 2023-10-19T13:01:31.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.480+0000 7f0b03dba740 1 -- 172.21.15.57:0/77541454 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55f24afbef20 msgr2=0x55f24b048630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:31.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.480+0000 7f0b03dba740 1 --2- 172.21.15.57:0/77541454 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55f24afbef20 0x55f24b048630 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x55f24afbf880 tx=0x7f0ae4017af0 comp rx=0 tx=0).stop 2023-10-19T13:01:31.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.480+0000 7f0b03dba740 1 -- 172.21.15.57:0/77541454 shutdown_connections 2023-10-19T13:01:31.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.480+0000 7f0b03dba740 1 --2- 172.21.15.57:0/77541454 >> [v2:172.21.15.57:6834/2556427177,v1:172.21.15.57:6835/2556427177] conn(0x55f24afeade0 0x55f24afeb1c0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:31.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.480+0000 7f0b03dba740 1 --2- 172.21.15.57:0/77541454 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f24afd6f00 0x55f24afad9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:31.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.480+0000 7f0b03dba740 1 --2- 172.21.15.57:0/77541454 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55f24afbef20 0x55f24b048630 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:31.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.480+0000 7f0ad9ffb640 0 client.8184 ms_handle_reset on v2:172.21.15.57:6834/2556427177 2023-10-19T13:01:31.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.480+0000 7f0b03dba740 1 --2- 172.21.15.57:0/77541454 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55f24afbeb40 0x55f24b044a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:31.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.480+0000 7f0b03dba740 1 -- 172.21.15.57:0/77541454 >> 172.21.15.57:0/77541454 conn(0x55f24af92020 msgr2=0x55f24afab850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:31.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.480+0000 7f0b03dba740 1 -- 172.21.15.57:0/77541454 shutdown_connections 2023-10-19T13:01:31.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.480+0000 7f0b03dba740 1 -- 172.21.15.57:0/77541454 wait complete. 2023-10-19T13:01:31.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.480+0000 7f0b03dba740 20 client.8184 trim_cache size 0 max 16384 2023-10-19T13:01:31.504 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2023-10-19T13:01:31.504 INFO:teuthology.orchestra.run.smithi057.stdout: "cluster_fsid": "1e9beb19-3260-4dd7-85ef-5b399ce774ac", 2023-10-19T13:01:31.504 INFO:teuthology.orchestra.run.smithi057.stdout: "whoami": 0, 2023-10-19T13:01:31.505 INFO:teuthology.orchestra.run.smithi057.stdout: "id": 8046, 2023-10-19T13:01:31.505 INFO:teuthology.orchestra.run.smithi057.stdout: "want_state": "up:active", 2023-10-19T13:01:31.505 INFO:teuthology.orchestra.run.smithi057.stdout: "state": "up:active", 2023-10-19T13:01:31.505 INFO:teuthology.orchestra.run.smithi057.stdout: "fs_name": "cephfs", 2023-10-19T13:01:31.505 INFO:teuthology.orchestra.run.smithi057.stdout: "rank_uptime": 4.7155599300000004, 2023-10-19T13:01:31.506 INFO:teuthology.orchestra.run.smithi057.stdout: "mdsmap_epoch": 44, 2023-10-19T13:01:31.506 INFO:teuthology.orchestra.run.smithi057.stdout: "osdmap_epoch": 113, 2023-10-19T13:01:31.506 INFO:teuthology.orchestra.run.smithi057.stdout: "osdmap_epoch_barrier": 113, 2023-10-19T13:01:31.506 INFO:teuthology.orchestra.run.smithi057.stdout: "uptime": 9.4291377789999995 2023-10-19T13:01:31.506 INFO:teuthology.orchestra.run.smithi057.stdout:} 2023-10-19T13:01:31.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.481+0000 7f0b01226640 1 -- 172.21.15.57:0/1347056663 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f0adc06aed0 msgr2=0x7f0adc06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:31.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.481+0000 7f0b01226640 1 --2- 172.21.15.57:0/1347056663 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f0adc06aed0 0x7f0adc06d390 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0af4009d60 tx=0x7f0af4005f90 comp rx=0 tx=0).stop 2023-10-19T13:01:31.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.481+0000 7f0b01226640 1 -- 172.21.15.57:0/1347056663 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0afc156790 msgr2=0x7f0afc0b1430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:31.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.481+0000 7f0b01226640 1 --2- 172.21.15.57:0/1347056663 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0afc156790 0x7f0afc0b1430 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f0aec009be0 tx=0x7f0aec0042f0 comp rx=0 tx=0).stop 2023-10-19T13:01:31.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.481+0000 7f0b01226640 1 -- 172.21.15.57:0/1347056663 shutdown_connections 2023-10-19T13:01:31.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.481+0000 7f0b01226640 1 --2- 172.21.15.57:0/1347056663 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f0adc06aed0 0x7f0adc06d390 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:31.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.481+0000 7f0b01226640 1 --2- 172.21.15.57:0/1347056663 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0afc156790 0x7f0afc0b1430 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:31.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.481+0000 7f0b01226640 1 --2- 172.21.15.57:0/1347056663 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0afc0ac870 0x7f0afc0aab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:31.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.481+0000 7f0b01226640 1 --2- 172.21.15.57:0/1347056663 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0afc06a730 0x7f0afc0c23f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:31.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.481+0000 7f0b01226640 1 -- 172.21.15.57:0/1347056663 >> 172.21.15.57:0/1347056663 conn(0x7f0afc06fae0 msgr2=0x7f0afc1e2500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:31.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.481+0000 7f0b01226640 1 -- 172.21.15.57:0/1347056663 shutdown_connections 2023-10-19T13:01:31.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.481+0000 7f0b01226640 1 -- 172.21.15.57:0/1347056663 wait complete. 2023-10-19T13:01:31.510 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-19T13:01:31.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.620+0000 7f081c3e7640 1 -- 172.21.15.57:0/2659007653 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f08140cbd20 msgr2=0x7f0814158300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:31.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.620+0000 7f081c3e7640 1 --2- 172.21.15.57:0/2659007653 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f08140cbd20 0x7f0814158300 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f08140cc490 tx=0x7f080802be40 comp rx=0 tx=0).stop 2023-10-19T13:01:31.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.621+0000 7f081c3e7640 1 -- 172.21.15.57:0/2659007653 shutdown_connections 2023-10-19T13:01:31.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.621+0000 7f081c3e7640 1 --2- 172.21.15.57:0/2659007653 >> v2:172.21.15.179:3300/0 conn(0x7f08140afa20 0x7f08140705a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:31.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.621+0000 7f081c3e7640 1 --2- 172.21.15.57:0/2659007653 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f08140cbd20 0x7f0814158300 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:31.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.621+0000 7f081c3e7640 1 -- 172.21.15.57:0/2659007653 >> 172.21.15.57:0/2659007653 conn(0x7f0814069930 msgr2=0x7f0814069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:31.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.621+0000 7f081c3e7640 1 -- 172.21.15.57:0/2659007653 shutdown_connections 2023-10-19T13:01:31.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.621+0000 7f081c3e7640 1 -- 172.21.15.57:0/2659007653 wait complete. 2023-10-19T13:01:31.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.622+0000 7f081c3e7640 1 Processor -- start 2023-10-19T13:01:31.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.622+0000 7f081c3e7640 1 -- start start 2023-10-19T13:01:31.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.622+0000 7f081c3e7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f08140afa20 0x7f08141424e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:31.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.622+0000 7f081c3e7640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f08140b0530 0x7f08141f46b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:31.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.622+0000 7f081c3e7640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f08140cbd20 0x7f08141f1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:31.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.623+0000 7f081c3e7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0814142a20 con 0x7f08140afa20 2023-10-19T13:01:31.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.623+0000 7f081c3e7640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08141f55e0 con 0x7f08140b0530 2023-10-19T13:01:31.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.623+0000 7f081c3e7640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08141f5020 con 0x7f08140cbd20 2023-10-19T13:01:31.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.623+0000 7f081a183640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f08140afa20 0x7f08141424e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:31.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.623+0000 7f081a183640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f08140afa20 0x7f08141424e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40204/0 (socket says 172.21.15.57:40204) 2023-10-19T13:01:31.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.623+0000 7f081a183640 1 -- 172.21.15.57:0/112668957 learned_addr learned my addr 172.21.15.57:0/112668957 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:01:31.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.623+0000 7f0819982640 1 --2- 172.21.15.57:0/112668957 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f08140b0530 0x7f08141f46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:31.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.623+0000 7f081a984640 1 --2- 172.21.15.57:0/112668957 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f08140cbd20 0x7f08141f1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:31.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.623+0000 7f081a183640 1 -- 172.21.15.57:0/112668957 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f08140cbd20 msgr2=0x7f08141f1ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:31.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.623+0000 7f081a183640 1 --2- 172.21.15.57:0/112668957 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f08140cbd20 0x7f08141f1ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:31.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.623+0000 7f081a183640 1 -- 172.21.15.57:0/112668957 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f08140b0530 msgr2=0x7f08141f46b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:31.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.623+0000 7f081a183640 1 --2- 172.21.15.57:0/112668957 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f08140b0530 0x7f08141f46b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:31.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.623+0000 7f081a183640 1 -- 172.21.15.57:0/112668957 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0808009560 con 0x7f08140afa20 2023-10-19T13:01:31.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.623+0000 7f081a183640 1 --2- 172.21.15.57:0/112668957 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f08140afa20 0x7f08141424e0 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f0808009850 tx=0x7f080802db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:31.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.623+0000 7f0806ffd640 1 -- 172.21.15.57:0/112668957 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0808003800 con 0x7f08140afa20 2023-10-19T13:01:31.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.623+0000 7f0806ffd640 1 -- 172.21.15.57:0/112668957 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f0808003a50 con 0x7f08140afa20 2023-10-19T13:01:31.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.623+0000 7f081c3e7640 1 -- 172.21.15.57:0/112668957 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08141f5300 con 0x7f08140afa20 2023-10-19T13:01:31.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.623+0000 7f0806ffd640 1 -- 172.21.15.57:0/112668957 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0808003e20 con 0x7f08140afa20 2023-10-19T13:01:31.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.623+0000 7f081c3e7640 1 -- 172.21.15.57:0/112668957 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08141f15a0 con 0x7f08140afa20 2023-10-19T13:01:31.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.624+0000 7f081c3e7640 1 -- 172.21.15.57:0/112668957 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08141f1120 con 0x7f08140afa20 2023-10-19T13:01:31.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.625+0000 7f0806ffd640 1 -- 172.21.15.57:0/112668957 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 47) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f080803e450 con 0x7f08140afa20 2023-10-19T13:01:31.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.625+0000 7f0806ffd640 1 --2- 172.21.15.57:0/112668957 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f07f806aed0 0x7f07f806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:31.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.625+0000 7f0806ffd640 1 -- 172.21.15.57:0/112668957 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f08080ac120 con 0x7f08140afa20 2023-10-19T13:01:31.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.626+0000 7f0819982640 1 --2- 172.21.15.57:0/112668957 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f07f806aed0 0x7f07f806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:31.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.626+0000 7f0819982640 1 --2- 172.21.15.57:0/112668957 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f07f806aed0 0x7f07f806d390 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f081414c300 tx=0x7f0810005f90 comp rx=0 tx=0).ready entity=mgr.7072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:31.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.630+0000 7f0806ffd640 1 -- 172.21.15.57:0/112668957 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f08141f1120 con 0x7f08140afa20 2023-10-19T13:01:31.732 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:31.730+0000 7fb508163640 -1 client.7141: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.7030:f55302c0-37aa-4235-8ccc-9775bc542785:v1:172.21.15.179:0/3304791412] 2023-10-19T13:01:31.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.847+0000 7f081c3e7640 1 -- 172.21.15.57:0/112668957 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f08141f1120 con 0x7f08140afa20 2023-10-19T13:01:31.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.848+0000 7f0806ffd640 1 -- 172.21.15.57:0/112668957 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v44) v1 ==== 53+0+73 (secure 0 0 0) 0x7f08141f1120 con 0x7f08140afa20 2023-10-19T13:01:31.851 INFO:teuthology.orchestra.run.smithi057.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-19T13:01:31.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.849+0000 7f081c3e7640 1 -- 172.21.15.57:0/112668957 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f07f806aed0 msgr2=0x7f07f806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:31.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.849+0000 7f081c3e7640 1 --2- 172.21.15.57:0/112668957 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f07f806aed0 0x7f07f806d390 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f081414c300 tx=0x7f0810005f90 comp rx=0 tx=0).stop 2023-10-19T13:01:31.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.849+0000 7f081c3e7640 1 -- 172.21.15.57:0/112668957 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f08140afa20 msgr2=0x7f08141424e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:31.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.849+0000 7f081c3e7640 1 --2- 172.21.15.57:0/112668957 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f08140afa20 0x7f08141424e0 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f0808009850 tx=0x7f080802db10 comp rx=0 tx=0).stop 2023-10-19T13:01:31.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.850+0000 7f081c3e7640 1 -- 172.21.15.57:0/112668957 shutdown_connections 2023-10-19T13:01:31.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.850+0000 7f081c3e7640 1 --2- 172.21.15.57:0/112668957 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f07f806aed0 0x7f07f806d390 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:31.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.850+0000 7f081c3e7640 1 --2- 172.21.15.57:0/112668957 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f08140cbd20 0x7f08141f1ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:31.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.850+0000 7f081c3e7640 1 --2- 172.21.15.57:0/112668957 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f08140b0530 0x7f08141f46b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:31.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.850+0000 7f081c3e7640 1 --2- 172.21.15.57:0/112668957 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f08140afa20 0x7f08141424e0 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:31.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.850+0000 7f081c3e7640 1 -- 172.21.15.57:0/112668957 >> 172.21.15.57:0/112668957 conn(0x7f0814069930 msgr2=0x7f0814157d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:31.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.850+0000 7f081c3e7640 1 -- 172.21.15.57:0/112668957 shutdown_connections 2023-10-19T13:01:31.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:31.850+0000 7f081c3e7640 1 -- 172.21.15.57:0/112668957 wait complete. 2023-10-19T13:01:31.864 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-10-19T13:01:31.864 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-19T13:01:31.864 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi057.front.sepia.ceph.com 2023-10-19T13:01:31.865 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-19T13:01:31.865 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-19T13:01:31.865 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-10-19T13:01:31.866 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-10-19T13:01:31.866 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-19T13:01:31.866 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-19T13:01:31.867 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:01:31.867 DEBUG:teuthology.orchestra.run.smithi057:> ip addr 2023-10-19T13:01:31.883 INFO:teuthology.orchestra.run.smithi057.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-19T13:01:31.883 INFO:teuthology.orchestra.run.smithi057.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-19T13:01:31.884 INFO:teuthology.orchestra.run.smithi057.stdout: inet 127.0.0.1/8 scope host lo 2023-10-19T13:01:31.884 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T13:01:31.884 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 ::1/128 scope host 2023-10-19T13:01:31.884 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T13:01:31.884 INFO:teuthology.orchestra.run.smithi057.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-19T13:01:31.884 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:6b:ff:86 brd ff:ff:ff:ff:ff:ff 2023-10-19T13:01:31.885 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp7s0f0 2023-10-19T13:01:31.885 INFO:teuthology.orchestra.run.smithi057.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-19T13:01:31.885 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:6b:ff:87 brd ff:ff:ff:ff:ff:ff 2023-10-19T13:01:31.885 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp7s0f1 2023-10-19T13:01:31.885 INFO:teuthology.orchestra.run.smithi057.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-19T13:01:31.886 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:bd:15:62 brd ff:ff:ff:ff:ff:ff 2023-10-19T13:01:31.886 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp1s0f0 2023-10-19T13:01:31.886 INFO:teuthology.orchestra.run.smithi057.stdout: inet 172.21.15.57/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-19T13:01:31.886 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft 41619sec preferred_lft 41619sec 2023-10-19T13:01:31.887 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 fe80::ec4:7aff:febd:1562/64 scope link noprefixroute 2023-10-19T13:01:31.887 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T13:01:31.887 INFO:teuthology.orchestra.run.smithi057.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-19T13:01:31.887 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:bd:15:63 brd ff:ff:ff:ff:ff:ff 2023-10-19T13:01:31.888 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp1s0f1 2023-10-19T13:01:31.888 INFO:teuthology.orchestra.run.smithi057.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-19T13:01:31.888 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0e:fe:f1:b5:41:59 brd ff:ff:ff:ff:ff:ff 2023-10-19T13:01:31.888 INFO:teuthology.orchestra.run.smithi057.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-19T13:01:31.889 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T13:01:31.889 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 fe80::64b1:a2ff:feb3:b08/64 scope link 2023-10-19T13:01:31.889 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T13:01:31.889 INFO:teuthology.orchestra.run.smithi057.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-19T13:01:31.889 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0e:fe:f1:b5:41:59 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T13:01:31.889 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 fe80::cfe:f1ff:feb5:4159/64 scope link 2023-10-19T13:01:31.890 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T13:01:31.890 INFO:teuthology.orchestra.run.smithi057.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-19T13:01:31.890 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 62:53:72:9d:52:71 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-19T13:01:31.890 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 fe80::6053:72ff:fe9d:5271/64 scope link 2023-10-19T13:01:31.891 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T13:01:31.891 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:01:31.891 DEBUG:teuthology.orchestra.run.smithi057:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-19T13:01:31.953 INFO:teuthology.orchestra.run.smithi057.stdout:1 2023-10-19T13:01:31.955 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:01:31.956 DEBUG:teuthology.orchestra.run.smithi057:> route 2023-10-19T13:01:32.011 INFO:teuthology.orchestra.run.smithi057.stdout:Kernel IP routing table 2023-10-19T13:01:32.012 INFO:teuthology.orchestra.run.smithi057.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-19T13:01:32.012 INFO:teuthology.orchestra.run.smithi057.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-19T13:01:32.012 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-19T13:01:32.012 INFO:teuthology.orchestra.run.smithi057.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-19T13:01:32.013 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:01:32.013 DEBUG:teuthology.orchestra.run.smithi057:> (cd / && exec bash -c ' 2023-10-19T13:01:32.014 DEBUG:teuthology.orchestra.run.smithi057:> set -e 2023-10-19T13:01:32.014 DEBUG:teuthology.orchestra.run.smithi057:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-19T13:01:32.014 DEBUG:teuthology.orchestra.run.smithi057:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-19T13:01:32.015 DEBUG:teuthology.orchestra.run.smithi057:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-19T13:01:32.015 DEBUG:teuthology.orchestra.run.smithi057:> ') 2023-10-19T13:01:32.119 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:01:32.120 DEBUG:teuthology.orchestra.run.smithi057:> ip netns list 2023-10-19T13:01:32.135 INFO:teuthology.orchestra.run.smithi057.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-19T13:01:32.135 INFO:teuthology.orchestra.run.smithi057.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-19T13:01:32.136 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-19T13:01:32.136 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:01:32.136 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip link set brx.0 up 2023-10-19T13:01:32.206 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-10-19T13:01:32.207 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-19T13:01:32.207 DEBUG:teuthology.orchestra.run.smithi057:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-19T13:01:32.261 INFO:teuthology.orchestra.run.smithi057.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-19T13:01:32.262 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-19T13:01:32.262 DEBUG:teuthology.orchestra.run.smithi057:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-19T13:01:32.316 DEBUG:teuthology.orchestra.run.smithi057:> sudo modprobe fuse 2023-10-19T13:01:32.388 DEBUG:teuthology.orchestra.run.smithi057:> cat /proc/self/mounts | awk '{print $2}' 2023-10-19T13:01:32.444 INFO:teuthology.orchestra.run.smithi057.stdout:/proc 2023-10-19T13:01:32.445 INFO:teuthology.orchestra.run.smithi057.stdout:/sys 2023-10-19T13:01:32.445 INFO:teuthology.orchestra.run.smithi057.stdout:/dev 2023-10-19T13:01:32.445 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/security 2023-10-19T13:01:32.445 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/shm 2023-10-19T13:01:32.446 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/pts 2023-10-19T13:01:32.446 INFO:teuthology.orchestra.run.smithi057.stdout:/run 2023-10-19T13:01:32.446 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/cgroup 2023-10-19T13:01:32.446 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/pstore 2023-10-19T13:01:32.447 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/bpf 2023-10-19T13:01:32.447 INFO:teuthology.orchestra.run.smithi057.stdout:/ 2023-10-19T13:01:32.447 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/selinux 2023-10-19T13:01:32.447 INFO:teuthology.orchestra.run.smithi057.stdout:/proc/sys/fs/binfmt_misc 2023-10-19T13:01:32.447 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/hugepages 2023-10-19T13:01:32.448 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/mqueue 2023-10-19T13:01:32.448 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/debug 2023-10-19T13:01:32.448 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/tracing 2023-10-19T13:01:32.448 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/fuse/connections 2023-10-19T13:01:32.448 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/config 2023-10-19T13:01:32.448 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-sysctl.service 2023-10-19T13:01:32.449 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-10-19T13:01:32.449 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-10-19T13:01:32.449 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/nfs/rpc_pipefs 2023-10-19T13:01:32.449 INFO:teuthology.orchestra.run.smithi057.stdout:/run/user/1000 2023-10-19T13:01:32.449 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph 2023-10-19T13:01:32.450 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-0 2023-10-19T13:01:32.450 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-1 2023-10-19T13:01:32.450 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-2 2023-10-19T13:01:32.450 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-3 2023-10-19T13:01:32.450 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns 2023-10-19T13:01:32.450 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T13:01:32.451 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T13:01:32.451 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-19T13:01:32.451 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-19T13:01:32.451 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:01:32.452 DEBUG:teuthology.orchestra.run.smithi057:> ls /sys/fs/fuse/connections 2023-10-19T13:01:32.471 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:32.468+0000 7fb508163640 -1 client.7141: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.7030:f55302c0-37aa-4235-8ccc-9775bc542785:v1:172.21.15.179:0/3304791412] 2023-10-19T13:01:32.499 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-10-19T13:01:32.500 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-10-19T13:01:32.542 DEBUG:teuthology.orchestra.run.smithi057:> sudo modprobe fuse 2023-10-19T13:01:32.574 DEBUG:teuthology.orchestra.run.smithi057:> cat /proc/self/mounts | awk '{print $2}' 2023-10-19T13:01:32.629 INFO:teuthology.orchestra.run.smithi057.stdout:/proc 2023-10-19T13:01:32.630 INFO:teuthology.orchestra.run.smithi057.stdout:/sys 2023-10-19T13:01:32.630 INFO:teuthology.orchestra.run.smithi057.stdout:/dev 2023-10-19T13:01:32.630 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/security 2023-10-19T13:01:32.631 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/shm 2023-10-19T13:01:32.631 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/pts 2023-10-19T13:01:32.631 INFO:teuthology.orchestra.run.smithi057.stdout:/run 2023-10-19T13:01:32.631 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/cgroup 2023-10-19T13:01:32.631 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/pstore 2023-10-19T13:01:32.632 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/bpf 2023-10-19T13:01:32.632 INFO:teuthology.orchestra.run.smithi057.stdout:/ 2023-10-19T13:01:32.632 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/selinux 2023-10-19T13:01:32.632 INFO:teuthology.orchestra.run.smithi057.stdout:/proc/sys/fs/binfmt_misc 2023-10-19T13:01:32.632 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/hugepages 2023-10-19T13:01:32.632 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/mqueue 2023-10-19T13:01:32.633 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/debug 2023-10-19T13:01:32.633 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/tracing 2023-10-19T13:01:32.633 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/fuse/connections 2023-10-19T13:01:32.633 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/config 2023-10-19T13:01:32.633 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-sysctl.service 2023-10-19T13:01:32.634 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-10-19T13:01:32.634 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-10-19T13:01:32.634 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/nfs/rpc_pipefs 2023-10-19T13:01:32.634 INFO:teuthology.orchestra.run.smithi057.stdout:/run/user/1000 2023-10-19T13:01:32.634 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph 2023-10-19T13:01:32.635 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-0 2023-10-19T13:01:32.635 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-1 2023-10-19T13:01:32.635 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-2 2023-10-19T13:01:32.635 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-3 2023-10-19T13:01:32.635 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns 2023-10-19T13:01:32.636 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T13:01:32.636 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T13:01:32.636 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-19T13:01:32.636 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-19T13:01:32.636 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:01:32.637 DEBUG:teuthology.orchestra.run.smithi057:> ls /sys/fs/fuse/connections 2023-10-19T13:01:32.638 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi057.stderr:2023-10-19T13:01:32.632+0000 7f8271841480 -1 init, newargv = 0x562e77d29b80 newargc=15 2023-10-19T13:01:32.638 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi057.stderr:ceph-fuse[125038]: starting ceph client 2023-10-19T13:01:32.639 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi057.stderr:ceph-fuse[125038]: starting fuse 2023-10-19T13:01:32.653 INFO:teuthology.orchestra.run.smithi057.stdout:43 2023-10-19T13:01:32.654 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2023-10-19T13:01:32.654 DEBUG:teuthology.orchestra.run.smithi057:> sudo stdin-killer -- python3 -c ' 2023-10-19T13:01:32.655 DEBUG:teuthology.orchestra.run.smithi057:> import glob 2023-10-19T13:01:32.655 DEBUG:teuthology.orchestra.run.smithi057:> import re 2023-10-19T13:01:32.655 DEBUG:teuthology.orchestra.run.smithi057:> import os 2023-10-19T13:01:32.656 DEBUG:teuthology.orchestra.run.smithi057:> import subprocess 2023-10-19T13:01:32.656 DEBUG:teuthology.orchestra.run.smithi057:> 2023-10-19T13:01:32.656 DEBUG:teuthology.orchestra.run.smithi057:> def _find_admin_socket(client_name): 2023-10-19T13:01:32.657 DEBUG:teuthology.orchestra.run.smithi057:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-10-19T13:01:32.657 DEBUG:teuthology.orchestra.run.smithi057:> files = glob.glob(asok_path) 2023-10-19T13:01:32.657 DEBUG:teuthology.orchestra.run.smithi057:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-10-19T13:01:32.658 DEBUG:teuthology.orchestra.run.smithi057:> 2023-10-19T13:01:32.658 DEBUG:teuthology.orchestra.run.smithi057:> # Given a non-glob path, it better be there 2023-10-19T13:01:32.659 DEBUG:teuthology.orchestra.run.smithi057:> if "*" not in asok_path: 2023-10-19T13:01:32.659 DEBUG:teuthology.orchestra.run.smithi057:> assert(len(files) == 1) 2023-10-19T13:01:32.659 DEBUG:teuthology.orchestra.run.smithi057:> return files[0] 2023-10-19T13:01:32.660 DEBUG:teuthology.orchestra.run.smithi057:> 2023-10-19T13:01:32.660 DEBUG:teuthology.orchestra.run.smithi057:> for f in files: 2023-10-19T13:01:32.660 DEBUG:teuthology.orchestra.run.smithi057:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-10-19T13:01:32.661 DEBUG:teuthology.orchestra.run.smithi057:> if os.path.exists("/proc/{0}".format(pid)): 2023-10-19T13:01:32.661 DEBUG:teuthology.orchestra.run.smithi057:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-10-19T13:01:32.662 DEBUG:teuthology.orchestra.run.smithi057:> contents = proc_f.read() 2023-10-19T13:01:32.662 DEBUG:teuthology.orchestra.run.smithi057:> if mountpoint in contents: 2023-10-19T13:01:32.663 DEBUG:teuthology.orchestra.run.smithi057:> return f 2023-10-19T13:01:32.663 DEBUG:teuthology.orchestra.run.smithi057:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-10-19T13:01:32.663 DEBUG:teuthology.orchestra.run.smithi057:> 2023-10-19T13:01:32.664 DEBUG:teuthology.orchestra.run.smithi057:> print(_find_admin_socket("client.0")) 2023-10-19T13:01:32.664 DEBUG:teuthology.orchestra.run.smithi057:> ' 2023-10-19T13:01:32.739 INFO:tasks.ceph.mds.c.smithi057.stderr:2023-10-19T13:01:32.736+0000 7f57a1c4a640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-19T13:01:32.779 INFO:teuthology.orchestra.run.smithi057.stdout:/var/run/ceph/ceph-client.0.125038.asok 2023-10-19T13:01:32.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:32 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2023-10-19T13:01:32.790 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.125038.asok 2023-10-19T13:01:32.790 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:01:32.791 DEBUG:teuthology.orchestra.run.smithi057:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.125038.asok status 2023-10-19T13:01:32.933 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2023-10-19T13:01:32.934 INFO:teuthology.orchestra.run.smithi057.stdout: "metadata": { 2023-10-19T13:01:32.934 INFO:teuthology.orchestra.run.smithi057.stdout: "ceph_sha1": "f7420d7f8e55ceea786dcbf536c338ce920c21eb", 2023-10-19T13:01:32.934 INFO:teuthology.orchestra.run.smithi057.stdout: "ceph_version": "ceph version 18.0.0-6786-gf7420d7f (f7420d7f8e55ceea786dcbf536c338ce920c21eb) reef (dev)", 2023-10-19T13:01:32.935 INFO:teuthology.orchestra.run.smithi057.stdout: "entity_id": "0", 2023-10-19T13:01:32.935 INFO:teuthology.orchestra.run.smithi057.stdout: "hostname": "smithi057", 2023-10-19T13:01:32.935 INFO:teuthology.orchestra.run.smithi057.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-10-19T13:01:32.935 INFO:teuthology.orchestra.run.smithi057.stdout: "pid": "125038", 2023-10-19T13:01:32.935 INFO:teuthology.orchestra.run.smithi057.stdout: "root": "/" 2023-10-19T13:01:32.935 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:01:32.936 INFO:teuthology.orchestra.run.smithi057.stdout: "dentry_count": 0, 2023-10-19T13:01:32.936 INFO:teuthology.orchestra.run.smithi057.stdout: "dentry_pinned_count": 0, 2023-10-19T13:01:32.936 INFO:teuthology.orchestra.run.smithi057.stdout: "id": 8199, 2023-10-19T13:01:32.936 INFO:teuthology.orchestra.run.smithi057.stdout: "inst": { 2023-10-19T13:01:32.936 INFO:teuthology.orchestra.run.smithi057.stdout: "name": { 2023-10-19T13:01:32.936 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "client", 2023-10-19T13:01:32.937 INFO:teuthology.orchestra.run.smithi057.stdout: "num": 8199 2023-10-19T13:01:32.937 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:01:32.937 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": { 2023-10-19T13:01:32.937 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "v1", 2023-10-19T13:01:32.937 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": "192.168.0.1:0", 2023-10-19T13:01:32.937 INFO:teuthology.orchestra.run.smithi057.stdout: "nonce": 3898856918 2023-10-19T13:01:32.938 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:01:32.938 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:01:32.938 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": { 2023-10-19T13:01:32.938 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "v1", 2023-10-19T13:01:32.938 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": "192.168.0.1:0", 2023-10-19T13:01:32.938 INFO:teuthology.orchestra.run.smithi057.stdout: "nonce": 3898856918 2023-10-19T13:01:32.939 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:01:32.939 INFO:teuthology.orchestra.run.smithi057.stdout: "inst_str": "client.8199 192.168.0.1:0/3898856918", 2023-10-19T13:01:32.939 INFO:teuthology.orchestra.run.smithi057.stdout: "addr_str": "192.168.0.1:0/3898856918", 2023-10-19T13:01:32.939 INFO:teuthology.orchestra.run.smithi057.stdout: "inode_count": 1, 2023-10-19T13:01:32.939 INFO:teuthology.orchestra.run.smithi057.stdout: "mds_epoch": 44, 2023-10-19T13:01:32.939 INFO:teuthology.orchestra.run.smithi057.stdout: "osd_epoch": 113, 2023-10-19T13:01:32.940 INFO:teuthology.orchestra.run.smithi057.stdout: "osd_epoch_barrier": 0, 2023-10-19T13:01:32.940 INFO:teuthology.orchestra.run.smithi057.stdout: "blocklisted": false, 2023-10-19T13:01:32.940 INFO:teuthology.orchestra.run.smithi057.stdout: "fs_name": "cephfs" 2023-10-19T13:01:32.940 INFO:teuthology.orchestra.run.smithi057.stdout:} 2023-10-19T13:01:32.942 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:01:32.942 DEBUG:teuthology.orchestra.run.smithi057:> stat --file-system '--printf=%T 2023-10-19T13:01:32.943 DEBUG:teuthology.orchestra.run.smithi057:> ' -- /home/ubuntu/cephtest/mnt.0 2023-10-19T13:01:32.958 INFO:teuthology.orchestra.run.smithi057.stdout:fuseblk 2023-10-19T13:01:32.959 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-10-19T13:01:32.959 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:01:32.960 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-19T13:01:33.030 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-19T13:01:33.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.178+0000 7f172b27c640 1 -- 172.21.15.57:0/2820726715 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f17100052a0 msgr2=0x7f171000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:33.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.178+0000 7f172b27c640 1 --2- 172.21.15.57:0/2820726715 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f17100052a0 0x7f171000b980 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7f171c00c690 tx=0x7f17100057f0 comp rx=0 tx=0).stop 2023-10-19T13:01:33.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.178+0000 7f172b27c640 1 -- 172.21.15.57:0/2820726715 shutdown_connections 2023-10-19T13:01:33.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.178+0000 7f172b27c640 1 --2- 172.21.15.57:0/2820726715 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f17100052a0 0x7f171000b980 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:33.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.178+0000 7f172b27c640 1 -- 172.21.15.57:0/2820726715 >> 172.21.15.57:0/2820726715 conn(0x7f172406fae0 msgr2=0x7f172406fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:33.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.178+0000 7f172b27c640 1 -- 172.21.15.57:0/2820726715 shutdown_connections 2023-10-19T13:01:33.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.178+0000 7f172b27c640 1 -- 172.21.15.57:0/2820726715 wait complete. 2023-10-19T13:01:33.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.179+0000 7f172b27c640 1 Processor -- start 2023-10-19T13:01:33.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.179+0000 7f172b27c640 1 -- start start 2023-10-19T13:01:33.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.180+0000 7f172b27c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f17100052a0 0x7f17241457d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:33.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.180+0000 7f172b27c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f17241e2f10 0x7f17240aa8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:33.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.180+0000 7f172b27c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f17241e3390 0x7f17240aae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:33.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.180+0000 7f172b27c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1724145d10 con 0x7f17241e2f10 2023-10-19T13:01:33.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.180+0000 7f172b27c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17241f4550 con 0x7f17100052a0 2023-10-19T13:01:33.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.180+0000 7f172b27c640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17241f4830 con 0x7f17241e3390 2023-10-19T13:01:33.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.180+0000 7f1728817640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f17241e2f10 0x7f17240aa8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:33.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.180+0000 7f1728817640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f17241e2f10 0x7f17240aa8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40212/0 (socket says 172.21.15.57:40212) 2023-10-19T13:01:33.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.180+0000 7f1728817640 1 -- 172.21.15.57:0/1670915656 learned_addr learned my addr 172.21.15.57:0/1670915656 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:01:33.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.180+0000 7f1729018640 1 --2- 172.21.15.57:0/1670915656 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f17100052a0 0x7f17241457d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:33.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.180+0000 7f1729819640 1 --2- 172.21.15.57:0/1670915656 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f17241e3390 0x7f17240aae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:33.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.180+0000 7f1728817640 1 -- 172.21.15.57:0/1670915656 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f17241e3390 msgr2=0x7f17240aae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:33.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.180+0000 7f1728817640 1 --2- 172.21.15.57:0/1670915656 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f17241e3390 0x7f17240aae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:33.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.180+0000 7f1728817640 1 -- 172.21.15.57:0/1670915656 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f17100052a0 msgr2=0x7f17241457d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:33.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.180+0000 7f1728817640 1 --2- 172.21.15.57:0/1670915656 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f17100052a0 0x7f17241457d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:33.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.180+0000 7f1728817640 1 -- 172.21.15.57:0/1670915656 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17241f4b10 con 0x7f17241e2f10 2023-10-19T13:01:33.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.181+0000 7f1728817640 1 --2- 172.21.15.57:0/1670915656 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f17241e2f10 0x7f17240aa8c0 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f171400c390 tx=0x7f171400c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:33.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.181+0000 7f1719ffb640 1 -- 172.21.15.57:0/1670915656 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17140035f0 con 0x7f17241e2f10 2023-10-19T13:01:33.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.181+0000 7f1719ffb640 1 -- 172.21.15.57:0/1670915656 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f1714003750 con 0x7f17241e2f10 2023-10-19T13:01:33.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.181+0000 7f172b27c640 1 -- 172.21.15.57:0/1670915656 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17241f4df0 con 0x7f17241e2f10 2023-10-19T13:01:33.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.181+0000 7f1719ffb640 1 -- 172.21.15.57:0/1670915656 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1714003b60 con 0x7f17241e2f10 2023-10-19T13:01:33.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.181+0000 7f172b27c640 1 -- 172.21.15.57:0/1670915656 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1724145fa0 con 0x7f17241e2f10 2023-10-19T13:01:33.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.182+0000 7f172b27c640 1 -- 172.21.15.57:0/1670915656 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17241f5310 con 0x7f17241e2f10 2023-10-19T13:01:33.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.183+0000 7f1719ffb640 1 -- 172.21.15.57:0/1670915656 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 47) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f1714003cc0 con 0x7f17241e2f10 2023-10-19T13:01:33.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.183+0000 7f1719ffb640 1 --2- 172.21.15.57:0/1670915656 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f170806aed0 0x7f170806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:33.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.183+0000 7f1719ffb640 1 -- 172.21.15.57:0/1670915656 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f1714088e80 con 0x7f17241e2f10 2023-10-19T13:01:33.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.184+0000 7f1729018640 1 --2- 172.21.15.57:0/1670915656 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f170806aed0 0x7f170806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:33.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.184+0000 7f1729018640 1 --2- 172.21.15.57:0/1670915656 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f170806aed0 0x7f170806d390 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1720004700 tx=0x7f1720005e30 comp rx=0 tx=0).ready entity=mgr.7072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:33.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.189+0000 7f1719ffb640 1 -- 172.21.15.57:0/1670915656 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f17241f5310 con 0x7f17241e2f10 2023-10-19T13:01:33.228 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:33.226+0000 7fb508163640 -1 client.7141: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.7030:f55302c0-37aa-4235-8ccc-9775bc542785:v1:172.21.15.179:0/3304791412] 2023-10-19T13:01:33.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.393+0000 7f172b27c640 1 -- 172.21.15.57:0/1670915656 --> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f172406abc0 con 0x7f170806aed0 2023-10-19T13:01:33.398 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.395+0000 7f1719ffb640 1 -- 172.21.15.57:0/1670915656 <== mgr.7072 v2:172.21.15.179:6832/3281457486 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f1720011b80 con 0x7f170806aed0 2023-10-19T13:01:33.398 INFO:teuthology.orchestra.run.smithi057.stdout:[ 2023-10-19T13:01:33.399 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:01:33.399 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "cephfs" 2023-10-19T13:01:33.399 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:01:33.399 INFO:teuthology.orchestra.run.smithi057.stdout:] 2023-10-19T13:01:33.400 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.397+0000 7f172b27c640 1 -- 172.21.15.57:0/1670915656 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f170806aed0 msgr2=0x7f170806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:33.401 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.397+0000 7f172b27c640 1 --2- 172.21.15.57:0/1670915656 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f170806aed0 0x7f170806d390 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1720004700 tx=0x7f1720005e30 comp rx=0 tx=0).stop 2023-10-19T13:01:33.401 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.397+0000 7f172b27c640 1 -- 172.21.15.57:0/1670915656 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f17241e2f10 msgr2=0x7f17240aa8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:33.401 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.397+0000 7f172b27c640 1 --2- 172.21.15.57:0/1670915656 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f17241e2f10 0x7f17240aa8c0 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f171400c390 tx=0x7f171400c850 comp rx=0 tx=0).stop 2023-10-19T13:01:33.401 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.397+0000 7f172b27c640 1 -- 172.21.15.57:0/1670915656 shutdown_connections 2023-10-19T13:01:33.401 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.397+0000 7f172b27c640 1 --2- 172.21.15.57:0/1670915656 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f170806aed0 0x7f170806d390 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:33.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.397+0000 7f172b27c640 1 --2- 172.21.15.57:0/1670915656 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f17241e3390 0x7f17240aae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:33.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.397+0000 7f172b27c640 1 --2- 172.21.15.57:0/1670915656 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f17241e2f10 0x7f17240aa8c0 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:33.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.397+0000 7f172b27c640 1 --2- 172.21.15.57:0/1670915656 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f17100052a0 0x7f17241457d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:33.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.397+0000 7f172b27c640 1 -- 172.21.15.57:0/1670915656 >> 172.21.15.57:0/1670915656 conn(0x7f172406fae0 msgr2=0x7f17240ae7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:33.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.397+0000 7f172b27c640 1 -- 172.21.15.57:0/1670915656 shutdown_connections 2023-10-19T13:01:33.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.398+0000 7f172b27c640 1 -- 172.21.15.57:0/1670915656 wait complete. 2023-10-19T13:01:33.411 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-19T13:01:33.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.522+0000 7febfc96e640 1 -- 172.21.15.57:0/1713091499 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7febe40039a0 msgr2=0x7febe4003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:33.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.522+0000 7febfc96e640 1 --2- 172.21.15.57:0/1713091499 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7febe40039a0 0x7febe4003dc0 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7febe800bb20 tx=0x7febe40057f0 comp rx=0 tx=0).stop 2023-10-19T13:01:33.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.522+0000 7febf6d9d640 1 -- 172.21.15.57:0/1713091499 reap_dead start 2023-10-19T13:01:33.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.522+0000 7febfc96e640 1 -- 172.21.15.57:0/1713091499 shutdown_connections 2023-10-19T13:01:33.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.522+0000 7febfc96e640 1 -- 172.21.15.57:0/1713091499 >> 172.21.15.57:0/1713091499 conn(0x7febf80d0d60 msgr2=0x7febf80d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:33.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.522+0000 7febfc96e640 1 -- 172.21.15.57:0/1713091499 shutdown_connections 2023-10-19T13:01:33.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.522+0000 7febfc96e640 1 -- 172.21.15.57:0/1713091499 wait complete. 2023-10-19T13:01:33.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.523+0000 7febfc96e640 1 Processor -- start 2023-10-19T13:01:33.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.523+0000 7febfc96e640 1 -- start start 2023-10-19T13:01:33.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.524+0000 7febfc96e640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7febe40039a0 0x7febf814f250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:33.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.524+0000 7febfc96e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7febf815d720 0x7febf815dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:33.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.524+0000 7febfc96e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7febf806a830 0x7febf806acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:33.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.524+0000 7febfc96e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febf806b1f0 con 0x7febf815d720 2023-10-19T13:01:33.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.524+0000 7febfc96e640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febf80b4330 con 0x7febf806a830 2023-10-19T13:01:33.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.524+0000 7febfc96e640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febf814f790 con 0x7febe40039a0 2023-10-19T13:01:33.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.524+0000 7febedd9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7febf815d720 0x7febf815dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:33.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.524+0000 7febedd9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7febf815d720 0x7febf815dba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40228/0 (socket says 172.21.15.57:40228) 2023-10-19T13:01:33.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.524+0000 7febedd9b640 1 -- 172.21.15.57:0/343379270 learned_addr learned my addr 172.21.15.57:0/343379270 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:01:33.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.524+0000 7febf6d9d640 1 --2- 172.21.15.57:0/343379270 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7febf806a830 0x7febf806acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:33.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.524+0000 7febf659c640 1 --2- 172.21.15.57:0/343379270 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7febe40039a0 0x7febf814f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:33.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.524+0000 7febedd9b640 1 -- 172.21.15.57:0/343379270 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7febe40039a0 msgr2=0x7febf814f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:33.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.524+0000 7febedd9b640 1 --2- 172.21.15.57:0/343379270 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7febe40039a0 0x7febf814f250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:33.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.524+0000 7febedd9b640 1 -- 172.21.15.57:0/343379270 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7febf806a830 msgr2=0x7febf806acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:33.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.524+0000 7febedd9b640 1 --2- 172.21.15.57:0/343379270 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7febf806a830 0x7febf806acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:33.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.524+0000 7febedd9b640 1 -- 172.21.15.57:0/343379270 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febf80c8d50 con 0x7febf815d720 2023-10-19T13:01:33.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.525+0000 7febedd9b640 1 --2- 172.21.15.57:0/343379270 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7febf815d720 0x7febf815dba0 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7febe0006600 tx=0x7febe00039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:33.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.525+0000 7febef7fe640 1 -- 172.21.15.57:0/343379270 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febe0011900 con 0x7febf815d720 2023-10-19T13:01:33.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.525+0000 7febef7fe640 1 -- 172.21.15.57:0/343379270 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7febe0011a60 con 0x7febf815d720 2023-10-19T13:01:33.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.525+0000 7febfc96e640 1 -- 172.21.15.57:0/343379270 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febf80c81d0 con 0x7febf815d720 2023-10-19T13:01:33.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.525+0000 7febef7fe640 1 -- 172.21.15.57:0/343379270 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febe0011e50 con 0x7febf815d720 2023-10-19T13:01:33.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.525+0000 7febfc96e640 1 -- 172.21.15.57:0/343379270 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febf80c8790 con 0x7febf815d720 2023-10-19T13:01:33.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.526+0000 7febfc96e640 1 -- 172.21.15.57:0/343379270 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febf80b44a0 con 0x7febf815d720 2023-10-19T13:01:33.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.526+0000 7febef7fe640 1 -- 172.21.15.57:0/343379270 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 47) v1 ==== 89275+0+0 (secure 0 0 0) 0x7febe0010420 con 0x7febf815d720 2023-10-19T13:01:33.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.527+0000 7febef7fe640 1 --2- 172.21.15.57:0/343379270 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7febd806aed0 0x7febd806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:33.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.527+0000 7febef7fe640 1 -- 172.21.15.57:0/343379270 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6057+0+0 (secure 0 0 0) 0x7febe008a550 con 0x7febf815d720 2023-10-19T13:01:33.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.527+0000 7febf659c640 1 --2- 172.21.15.57:0/343379270 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7febd806aed0 0x7febd806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:33.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.527+0000 7febf659c640 1 --2- 172.21.15.57:0/343379270 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7febd806aed0 0x7febd806d390 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7febe8037000 tx=0x7febe802e040 comp rx=0 tx=0).ready entity=mgr.7072 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:33.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.531+0000 7febef7fe640 1 -- 172.21.15.57:0/343379270 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7febe0055450 con 0x7febf815d720 2023-10-19T13:01:33.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:33.764+0000 7febfc96e640 1 -- 172.21.15.57:0/343379270 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7febf80690e0 con 0x7febf815d720 2023-10-19T13:01:33.983 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:33.980+0000 7fb508163640 -1 client.7141: SimpleRADOSStriper: lock: main.db: waiting for locks: lockers exclusive=1 tag= lockers=[client.7030:f55302c0-37aa-4235-8ccc-9775bc542785:v1:172.21.15.179:0/3304791412] 2023-10-19T13:01:34.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.188+0000 7febef7fe640 1 -- 172.21.15.57:0/343379270 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v48) v1 ==== 92+0+0 (secure 0 0 0) 0x7febe0058a70 con 0x7febf815d720 2023-10-19T13:01:34.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.191+0000 7febfc96e640 1 -- 172.21.15.57:0/343379270 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7febd806aed0 msgr2=0x7febd806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:34.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.191+0000 7febfc96e640 1 --2- 172.21.15.57:0/343379270 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7febd806aed0 0x7febd806d390 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7febe8037000 tx=0x7febe802e040 comp rx=0 tx=0).stop 2023-10-19T13:01:34.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.191+0000 7febfc96e640 1 -- 172.21.15.57:0/343379270 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7febf815d720 msgr2=0x7febf815dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:34.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.191+0000 7febfc96e640 1 --2- 172.21.15.57:0/343379270 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7febf815d720 0x7febf815dba0 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7febe0006600 tx=0x7febe00039c0 comp rx=0 tx=0).stop 2023-10-19T13:01:34.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.191+0000 7febfc96e640 1 -- 172.21.15.57:0/343379270 shutdown_connections 2023-10-19T13:01:34.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.191+0000 7febfc96e640 1 --2- 172.21.15.57:0/343379270 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7febd806aed0 0x7febd806d390 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:34.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.191+0000 7febfc96e640 1 --2- 172.21.15.57:0/343379270 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7febf806a830 0x7febf806acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:34.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.191+0000 7febfc96e640 1 --2- 172.21.15.57:0/343379270 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7febf815d720 0x7febf815dba0 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:34.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.191+0000 7febfc96e640 1 --2- 172.21.15.57:0/343379270 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7febe40039a0 0x7febf814f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:34.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.191+0000 7febfc96e640 1 -- 172.21.15.57:0/343379270 >> 172.21.15.57:0/343379270 conn(0x7febf80d0d60 msgr2=0x7febf806b770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:34.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.192+0000 7febfc96e640 1 -- 172.21.15.57:0/343379270 shutdown_connections 2023-10-19T13:01:34.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.192+0000 7febfc96e640 1 -- 172.21.15.57:0/343379270 wait complete. 2023-10-19T13:01:34.209 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-19T13:01:34.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.320+0000 7f0806bce640 1 -- 172.21.15.57:0/3581123798 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f080015b2a0 msgr2=0x7f08000d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:34.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.320+0000 7f0806bce640 1 --2- 172.21.15.57:0/3581123798 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f080015b2a0 0x7f08000d1910 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f07f0005fd0 tx=0x7f07f002c240 comp rx=0 tx=0).stop 2023-10-19T13:01:34.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.320+0000 7f0806bce640 1 -- 172.21.15.57:0/3581123798 shutdown_connections 2023-10-19T13:01:34.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.320+0000 7f0806bce640 1 --2- 172.21.15.57:0/3581123798 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f08000ceb40 0x7f08000b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:34.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.320+0000 7f0806bce640 1 --2- 172.21.15.57:0/3581123798 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f080015b2a0 0x7f08000d1910 unknown :-1 s=CLOSED pgs=1315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:34.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.320+0000 7f0806bce640 1 -- 172.21.15.57:0/3581123798 >> 172.21.15.57:0/3581123798 conn(0x7f08000d0d60 msgr2=0x7f08000d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:34.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.320+0000 7f0806bce640 1 -- 172.21.15.57:0/3581123798 shutdown_connections 2023-10-19T13:01:34.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.320+0000 7f0806bce640 1 -- 172.21.15.57:0/3581123798 wait complete. 2023-10-19T13:01:34.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.322+0000 7f0806bce640 1 Processor -- start 2023-10-19T13:01:34.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.322+0000 7f0806bce640 1 -- start start 2023-10-19T13:01:34.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.322+0000 7f0806bce640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f08000ceb40 0x7f080015cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:34.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.322+0000 7f0806bce640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f08000cf340 0x7f080015d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:34.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.322+0000 7f0806bce640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f080015b2a0 0x7f080014eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:34.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.322+0000 7f0806bce640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08000cade0 con 0x7f08000ceb40 2023-10-19T13:01:34.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.323+0000 7f0806bce640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f080015d310 con 0x7f08000cf340 2023-10-19T13:01:34.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.323+0000 7f0806bce640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08000b5250 con 0x7f080015b2a0 2023-10-19T13:01:34.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.323+0000 7f080496a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f08000ceb40 0x7f080015cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:34.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.323+0000 7f080496a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f08000ceb40 0x7f080015cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40270/0 (socket says 172.21.15.57:40270) 2023-10-19T13:01:34.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.323+0000 7f080496a640 1 -- 172.21.15.57:0/2401758460 learned_addr learned my addr 172.21.15.57:0/2401758460 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:01:34.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.323+0000 7f07fffff640 1 --2- 172.21.15.57:0/2401758460 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f08000cf340 0x7f080015d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:34.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.323+0000 7f080516b640 1 --2- 172.21.15.57:0/2401758460 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f080015b2a0 0x7f080014eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:34.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.323+0000 7f080496a640 1 -- 172.21.15.57:0/2401758460 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f080015b2a0 msgr2=0x7f080014eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:34.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.323+0000 7f080496a640 1 --2- 172.21.15.57:0/2401758460 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f080015b2a0 0x7f080014eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:34.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.323+0000 7f080496a640 1 -- 172.21.15.57:0/2401758460 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f08000cf340 msgr2=0x7f080015d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:34.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.323+0000 7f080496a640 1 --2- 172.21.15.57:0/2401758460 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f08000cf340 0x7f080015d720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:34.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.323+0000 7f080496a640 1 -- 172.21.15.57:0/2401758460 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07f0009900 con 0x7f08000ceb40 2023-10-19T13:01:34.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.323+0000 7f07fffff640 1 --2- 172.21.15.57:0/2401758460 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f08000cf340 0x7f080015d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T13:01:34.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.323+0000 7f080496a640 1 --2- 172.21.15.57:0/2401758460 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f08000ceb40 0x7f080015cdd0 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f07e800c480 tx=0x7f07e800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:34.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.323+0000 7f07fd7fa640 1 -- 172.21.15.57:0/2401758460 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07e80052d0 con 0x7f08000ceb40 2023-10-19T13:01:34.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.323+0000 7f07fd7fa640 1 -- 172.21.15.57:0/2401758460 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f07e8005430 con 0x7f08000ceb40 2023-10-19T13:01:34.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.323+0000 7f0806bce640 1 -- 172.21.15.57:0/2401758460 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08000cab00 con 0x7f08000ceb40 2023-10-19T13:01:34.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.323+0000 7f07fd7fa640 1 -- 172.21.15.57:0/2401758460 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07e8012890 con 0x7f08000ceb40 2023-10-19T13:01:34.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.323+0000 7f0806bce640 1 -- 172.21.15.57:0/2401758460 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08000b57d0 con 0x7f08000ceb40 2023-10-19T13:01:34.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.325+0000 7f0806bce640 1 -- 172.21.15.57:0/2401758460 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08000b50b0 con 0x7f08000ceb40 2023-10-19T13:01:34.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.325+0000 7f07fd7fa640 1 -- 172.21.15.57:0/2401758460 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 48) v1 ==== 89292+0+0 (secure 0 0 0) 0x7f07e80035f0 con 0x7f08000ceb40 2023-10-19T13:01:34.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.326+0000 7f07fd7fa640 1 --2- 172.21.15.57:0/2401758460 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f07e006af20 0x7f07e006d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:34.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.326+0000 7f07fffff640 1 -- 172.21.15.57:0/2401758460 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f07e006af20 msgr2=0x7f07e006d3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/3281457486 2023-10-19T13:01:34.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.326+0000 7f07fd7fa640 1 -- 172.21.15.57:0/2401758460 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f07e8089380 con 0x7f08000ceb40 2023-10-19T13:01:34.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.326+0000 7f07fffff640 1 --2- 172.21.15.57:0/2401758460 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f07e006af20 0x7f07e006d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T13:01:34.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.330+0000 7f07fd7fa640 1 -- 172.21.15.57:0/2401758460 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f08000b50b0 con 0x7f08000ceb40 2023-10-19T13:01:34.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.507+0000 7f0806bce640 1 -- 172.21.15.57:0/2401758460 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f08000b50b0 con 0x7f08000ceb40 2023-10-19T13:01:34.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.511+0000 7f07fd7fa640 1 -- 172.21.15.57:0/2401758460 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v48)=0 v48) v1 ==== 138+0+0 (secure 0 0 0) 0x7f08000b50b0 con 0x7f08000ceb40 2023-10-19T13:01:34.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.513+0000 7f0806bce640 1 -- 172.21.15.57:0/2401758460 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f07e006af20 msgr2=0x7f07e006d3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T13:01:34.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.513+0000 7f0806bce640 1 --2- 172.21.15.57:0/2401758460 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f07e006af20 0x7f07e006d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:34.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.513+0000 7f0806bce640 1 -- 172.21.15.57:0/2401758460 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f08000ceb40 msgr2=0x7f080015cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:34.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.513+0000 7f0806bce640 1 --2- 172.21.15.57:0/2401758460 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f08000ceb40 0x7f080015cdd0 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f07e800c480 tx=0x7f07e800c940 comp rx=0 tx=0).stop 2023-10-19T13:01:34.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.513+0000 7f0806bce640 1 -- 172.21.15.57:0/2401758460 shutdown_connections 2023-10-19T13:01:34.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.513+0000 7f0806bce640 1 --2- 172.21.15.57:0/2401758460 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f07e006af20 0x7f07e006d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:34.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.513+0000 7f0806bce640 1 --2- 172.21.15.57:0/2401758460 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f080015b2a0 0x7f080014eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:34.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.513+0000 7f0806bce640 1 --2- 172.21.15.57:0/2401758460 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f08000cf340 0x7f080015d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:34.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.513+0000 7f0806bce640 1 --2- 172.21.15.57:0/2401758460 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f08000ceb40 0x7f080015cdd0 unknown :-1 s=CLOSED pgs=1316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:34.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.513+0000 7f0806bce640 1 -- 172.21.15.57:0/2401758460 >> 172.21.15.57:0/2401758460 conn(0x7f08000d0d60 msgr2=0x7f0800151110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:34.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.514+0000 7f0806bce640 1 -- 172.21.15.57:0/2401758460 shutdown_connections 2023-10-19T13:01:34.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.514+0000 7f0806bce640 1 -- 172.21.15.57:0/2401758460 wait complete. 2023-10-19T13:01:34.527 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-19T13:01:34.565 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:34.563+0000 7ff9c302e040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-19T13:01:34.596 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:01:34.594+0000 7fd7fb856040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-19T13:01:34.609 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:34.606+0000 7ff9c302e040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-19T13:01:34.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.637+0000 7f7338a21640 1 -- 172.21.15.57:0/2879206275 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f733415b2a0 msgr2=0x7f73340d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:34.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.637+0000 7f7338a21640 1 --2- 172.21.15.57:0/2879206275 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f733415b2a0 0x7f73340d1910 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f731c005fd0 tx=0x7f731c02be20 comp rx=0 tx=0).stop 2023-10-19T13:01:34.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.637+0000 7f7338a21640 1 -- 172.21.15.57:0/2879206275 shutdown_connections 2023-10-19T13:01:34.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.638+0000 7f7338a21640 1 --2- 172.21.15.57:0/2879206275 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f733415b2a0 0x7f73340d1910 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:34.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.638+0000 7f7338a21640 1 -- 172.21.15.57:0/2879206275 >> 172.21.15.57:0/2879206275 conn(0x7f73340d0d60 msgr2=0x7f73340d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:34.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.638+0000 7f7338a21640 1 -- 172.21.15.57:0/2879206275 shutdown_connections 2023-10-19T13:01:34.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.638+0000 7f7338a21640 1 -- 172.21.15.57:0/2879206275 wait complete. 2023-10-19T13:01:34.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.639+0000 7f7338a21640 1 Processor -- start 2023-10-19T13:01:34.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.639+0000 7f7338a21640 1 -- start start 2023-10-19T13:01:34.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.640+0000 7f7338a21640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f73340ceb40 0x7f733415e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:34.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.640+0000 7f7338a21640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f73340cf340 0x7f73341f2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:34.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.640+0000 7f7338a21640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f733415b2a0 0x7f73340b5590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:34.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.640+0000 7f7338a21640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73341f58b0 con 0x7f733415b2a0 2023-10-19T13:01:34.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.640+0000 7f7338a21640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73340b5ad0 con 0x7f73340ceb40 2023-10-19T13:01:34.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.640+0000 7f7338a21640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73341f4cc0 con 0x7f73340cf340 2023-10-19T13:01:34.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.640+0000 7f7332d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f733415b2a0 0x7f73340b5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:34.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.640+0000 7f7332d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f733415b2a0 0x7f73340b5590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40294/0 (socket says 172.21.15.57:40294) 2023-10-19T13:01:34.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.640+0000 7f7332d9d640 1 -- 172.21.15.57:0/3384704391 learned_addr learned my addr 172.21.15.57:0/3384704391 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:01:34.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.640+0000 7f733259c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f73340ceb40 0x7f733415e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:34.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.640+0000 7f732bfff640 1 --2- 172.21.15.57:0/3384704391 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f73340cf340 0x7f73341f2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:34.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.640+0000 7f7332d9d640 1 -- 172.21.15.57:0/3384704391 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f73340cf340 msgr2=0x7f73341f2200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:34.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.640+0000 7f7332d9d640 1 --2- 172.21.15.57:0/3384704391 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f73340cf340 0x7f73341f2200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:34.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.640+0000 7f7332d9d640 1 -- 172.21.15.57:0/3384704391 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f73340ceb40 msgr2=0x7f733415e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:34.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.640+0000 7f7332d9d640 1 --2- 172.21.15.57:0/3384704391 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f73340ceb40 0x7f733415e100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:34.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.640+0000 7f7332d9d640 1 -- 172.21.15.57:0/3384704391 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f731c009560 con 0x7f733415b2a0 2023-10-19T13:01:34.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.640+0000 7f733259c640 1 --2- 172.21.15.57:0/3384704391 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f73340ceb40 0x7f733415e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T13:01:34.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.640+0000 7f7332d9d640 1 --2- 172.21.15.57:0/3384704391 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f733415b2a0 0x7f73340b5590 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f732c00c800 tx=0x7f732c00ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:34.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.641+0000 7f732affd640 1 -- 172.21.15.57:0/3384704391 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f732c003b70 con 0x7f733415b2a0 2023-10-19T13:01:34.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.641+0000 7f732affd640 1 -- 172.21.15.57:0/3384704391 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f732c003cf0 con 0x7f733415b2a0 2023-10-19T13:01:34.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.641+0000 7f7338a21640 1 -- 172.21.15.57:0/3384704391 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73341f4f10 con 0x7f733415b2a0 2023-10-19T13:01:34.650 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.641+0000 7f732affd640 1 -- 172.21.15.57:0/3384704391 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f732c003ea0 con 0x7f733415b2a0 2023-10-19T13:01:34.650 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.641+0000 7f7338a21640 1 -- 172.21.15.57:0/3384704391 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73341f54d0 con 0x7f733415b2a0 2023-10-19T13:01:34.650 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.642+0000 7f7338a21640 1 -- 172.21.15.57:0/3384704391 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73340690e0 con 0x7f733415b2a0 2023-10-19T13:01:34.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.642+0000 7f732affd640 1 -- 172.21.15.57:0/3384704391 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 48) v1 ==== 89292+0+0 (secure 0 0 0) 0x7f732c00aab0 con 0x7f733415b2a0 2023-10-19T13:01:34.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.643+0000 7f732affd640 1 --2- 172.21.15.57:0/3384704391 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f731406af20 0x7f731406d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:34.651 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:01:34.643+0000 7fd7fb856040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-19T13:01:34.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.643+0000 7f733259c640 1 -- 172.21.15.57:0/3384704391 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f731406af20 msgr2=0x7f731406d3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/3281457486 2023-10-19T13:01:34.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.643+0000 7f732affd640 1 -- 172.21.15.57:0/3384704391 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f732c089660 con 0x7f733415b2a0 2023-10-19T13:01:34.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.643+0000 7f733259c640 1 --2- 172.21.15.57:0/3384704391 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f731406af20 0x7f731406d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T13:01:34.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.647+0000 7f732affd640 1 -- 172.21.15.57:0/3384704391 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f732c054680 con 0x7f733415b2a0 2023-10-19T13:01:34.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.828+0000 7f7338a21640 1 -- 172.21.15.57:0/3384704391 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f73340c76c0 con 0x7f733415b2a0 2023-10-19T13:01:34.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.836+0000 7f732affd640 1 -- 172.21.15.57:0/3384704391 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v49)=0 v49) v1 ==== 133+0+0 (secure 0 0 0) 0x7f732c057ca0 con 0x7f733415b2a0 2023-10-19T13:01:34.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.838+0000 7f7338a21640 1 -- 172.21.15.57:0/3384704391 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f731406af20 msgr2=0x7f731406d3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T13:01:34.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.838+0000 7f7338a21640 1 --2- 172.21.15.57:0/3384704391 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f731406af20 0x7f731406d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:34.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.838+0000 7f7338a21640 1 -- 172.21.15.57:0/3384704391 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f733415b2a0 msgr2=0x7f73340b5590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:34.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.838+0000 7f7338a21640 1 --2- 172.21.15.57:0/3384704391 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f733415b2a0 0x7f73340b5590 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f732c00c800 tx=0x7f732c00ccc0 comp rx=0 tx=0).stop 2023-10-19T13:01:34.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.838+0000 7f7338a21640 1 -- 172.21.15.57:0/3384704391 shutdown_connections 2023-10-19T13:01:34.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.838+0000 7f7338a21640 1 --2- 172.21.15.57:0/3384704391 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f731406af20 0x7f731406d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:34.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.838+0000 7f7338a21640 1 --2- 172.21.15.57:0/3384704391 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f733415b2a0 0x7f73340b5590 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:34.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.838+0000 7f7338a21640 1 --2- 172.21.15.57:0/3384704391 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f73340cf340 0x7f73341f2200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:34.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.838+0000 7f7338a21640 1 --2- 172.21.15.57:0/3384704391 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f73340ceb40 0x7f733415e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:34.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.838+0000 7f7338a21640 1 -- 172.21.15.57:0/3384704391 >> 172.21.15.57:0/3384704391 conn(0x7f73340d0d60 msgr2=0x7f73340ae140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:34.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.838+0000 7f7338a21640 1 -- 172.21.15.57:0/3384704391 shutdown_connections 2023-10-19T13:01:34.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.838+0000 7f7338a21640 1 -- 172.21.15.57:0/3384704391 wait complete. 2023-10-19T13:01:34.852 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /bad-path 1m 2023-10-19T13:01:34.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.964+0000 7f581a3ce640 1 -- 172.21.15.57:0/3257338997 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f58140cf340 msgr2=0x7f58140d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:34.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.964+0000 7f581a3ce640 1 --2- 172.21.15.57:0/3257338997 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f58140cf340 0x7f58140d1910 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f5804005fd0 tx=0x7f580402bd90 comp rx=0 tx=0).stop 2023-10-19T13:01:34.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.964+0000 7f581a3ce640 1 -- 172.21.15.57:0/3257338997 shutdown_connections 2023-10-19T13:01:34.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.964+0000 7f581a3ce640 1 --2- 172.21.15.57:0/3257338997 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f58140ceb40 0x7f58140b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:34.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.964+0000 7f581a3ce640 1 --2- 172.21.15.57:0/3257338997 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f58140cf340 0x7f58140d1910 unknown :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:34.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.964+0000 7f581a3ce640 1 -- 172.21.15.57:0/3257338997 >> 172.21.15.57:0/3257338997 conn(0x7f58140d0d60 msgr2=0x7f58140d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:34.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.964+0000 7f581a3ce640 1 -- 172.21.15.57:0/3257338997 shutdown_connections 2023-10-19T13:01:34.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.964+0000 7f581a3ce640 1 -- 172.21.15.57:0/3257338997 wait complete. 2023-10-19T13:01:34.973 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.965+0000 7f581a3ce640 1 Processor -- start 2023-10-19T13:01:34.973 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.965+0000 7f581a3ce640 1 -- start start 2023-10-19T13:01:34.973 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.966+0000 7f581a3ce640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f58140ceb40 0x7f58140c9d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:34.973 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.966+0000 7f581a3ce640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f58140cf340 0x7f581415b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:34.974 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.966+0000 7f581a3ce640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f581415b2a0 0x7f58141576d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:34.974 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.966+0000 7f581a3ce640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f581415ce10 con 0x7f58140cf340 2023-10-19T13:01:34.974 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.966+0000 7f581a3ce640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f581415c190 con 0x7f58140ceb40 2023-10-19T13:01:34.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.966+0000 7f581a3ce640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f581415c470 con 0x7f581415b2a0 2023-10-19T13:01:34.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.966+0000 7f58137fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f58140cf340 0x7f581415b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:34.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.966+0000 7f58137fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f58140cf340 0x7f581415b6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40312/0 (socket says 172.21.15.57:40312) 2023-10-19T13:01:34.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.966+0000 7f58137fe640 1 -- 172.21.15.57:0/605136826 learned_addr learned my addr 172.21.15.57:0/605136826 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:01:34.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.966+0000 7f581896b640 1 --2- 172.21.15.57:0/605136826 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f581415b2a0 0x7f58141576d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:34.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.966+0000 7f5813fff640 1 --2- 172.21.15.57:0/605136826 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f58140ceb40 0x7f58140c9d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:34.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.966+0000 7f58137fe640 1 -- 172.21.15.57:0/605136826 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f581415b2a0 msgr2=0x7f58141576d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:34.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.966+0000 7f58137fe640 1 --2- 172.21.15.57:0/605136826 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f581415b2a0 0x7f58141576d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:34.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.966+0000 7f58137fe640 1 -- 172.21.15.57:0/605136826 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f58140ceb40 msgr2=0x7f58140c9d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:34.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.966+0000 7f58137fe640 1 --2- 172.21.15.57:0/605136826 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f58140ceb40 0x7f58140c9d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:34.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.966+0000 7f58137fe640 1 -- 172.21.15.57:0/605136826 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5804009560 con 0x7f58140cf340 2023-10-19T13:01:34.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.967+0000 7f58137fe640 1 --2- 172.21.15.57:0/605136826 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f58140cf340 0x7f581415b6d0 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f580800c400 tx=0x7f580800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:34.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.967+0000 7f5810ff9640 1 -- 172.21.15.57:0/605136826 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58080052d0 con 0x7f58140cf340 2023-10-19T13:01:34.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.967+0000 7f5810ff9640 1 -- 172.21.15.57:0/605136826 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f58080035f0 con 0x7f58140cf340 2023-10-19T13:01:34.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.967+0000 7f581a3ce640 1 -- 172.21.15.57:0/605136826 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58140b0d50 con 0x7f58140cf340 2023-10-19T13:01:34.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.967+0000 7f5810ff9640 1 -- 172.21.15.57:0/605136826 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5808003a00 con 0x7f58140cf340 2023-10-19T13:01:34.979 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.967+0000 7f581a3ce640 1 -- 172.21.15.57:0/605136826 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f581415ca30 con 0x7f58140cf340 2023-10-19T13:01:34.979 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.968+0000 7f581a3ce640 1 -- 172.21.15.57:0/605136826 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f581415cf50 con 0x7f58140cf340 2023-10-19T13:01:34.979 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.968+0000 7f5810ff9640 1 -- 172.21.15.57:0/605136826 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 48) v1 ==== 89292+0+0 (secure 0 0 0) 0x7f5808003b60 con 0x7f58140cf340 2023-10-19T13:01:34.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.969+0000 7f5810ff9640 1 --2- 172.21.15.57:0/605136826 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f580006af20 0x7f580006d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:34.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.969+0000 7f5813fff640 1 -- 172.21.15.57:0/605136826 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f580006af20 msgr2=0x7f580006d3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/3281457486 2023-10-19T13:01:34.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.969+0000 7f5810ff9640 1 -- 172.21.15.57:0/605136826 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f5808088ea0 con 0x7f58140cf340 2023-10-19T13:01:34.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.969+0000 7f5813fff640 1 --2- 172.21.15.57:0/605136826 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f580006af20 0x7f580006d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T13:01:34.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:34.973+0000 7f5810ff9640 1 -- 172.21.15.57:0/605136826 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f581415cf50 con 0x7f58140cf340 2023-10-19T13:01:34.998 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:34.995+0000 7ff9c302e040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-19T13:01:35.041 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:35.039+0000 7ff9c302e040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-19T13:01:35.072 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:01:35.069+0000 7fd7fb856040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-19T13:01:35.083 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:35.080+0000 7ff9c302e040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-19T13:01:35.119 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:01:35.116+0000 7fd7fb856040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-19T13:01:35.133 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:35.131+0000 7ff9c302e040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-19T13:01:35.164 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:01:35.161+0000 7fd7fb856040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-19T13:01:35.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:35.170+0000 7f5813fff640 1 -- 172.21.15.57:0/605136826 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f580006af20 msgr2=0x7f580006d3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/3281457486 2023-10-19T13:01:35.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:35.170+0000 7f5813fff640 1 --2- 172.21.15.57:0/605136826 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f580006af20 0x7f580006d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-19T13:01:35.174 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:35.171+0000 7ff9c302e040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-19T13:01:35.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:35.172+0000 7f581a3ce640 1 -- 172.21.15.57:0/605136826 --> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/bad-path", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f581415cf50 con 0x7f580006af20 2023-10-19T13:01:35.221 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:01:35.218+0000 7fd7fb856040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-19T13:01:35.264 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:01:35.262+0000 7fd7fb856040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-19T13:01:35.272 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:35.269+0000 7ff9c302e040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-19T13:01:35.324 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:35.321+0000 7ff9c302e040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-19T13:01:35.366 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:35.363+0000 7ff9c302e040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-19T13:01:35.368 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:01:35.365+0000 7fd7fb856040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-19T13:01:35.413 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:35.410+0000 7ff9c302e040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-19T13:01:35.424 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:01:35.421+0000 7fd7fb856040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-19T13:01:35.455 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:35.452+0000 7ff9c302e040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-19T13:01:35.468 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:01:35.465+0000 7fd7fb856040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-19T13:01:35.519 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:01:35.516+0000 7fd7fb856040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-19T13:01:35.563 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:01:35.561+0000 7fd7fb856040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-19T13:01:35.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:35.571+0000 7f5813fff640 1 -- 172.21.15.57:0/605136826 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f580006af20 msgr2=0x7f580006d3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/3281457486 2023-10-19T13:01:35.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:35.571+0000 7f5813fff640 1 --2- 172.21.15.57:0/605136826 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f580006af20 0x7f580006d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-19T13:01:35.766 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:35.763+0000 7ff9c302e040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-19T13:01:35.893 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:35.891+0000 7ff9c302e040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-19T13:01:35.898 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:01:35.896+0000 7fd7fb856040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-19T13:01:35.983 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:35.980+0000 7ff9c302e040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-19T13:01:36.024 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:36.022+0000 7ff9c302e040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-19T13:01:36.037 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:01:36.034+0000 7fd7fb856040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-19T13:01:36.117 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:36.114+0000 7ff9c302e040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-19T13:01:36.134 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:01:36.131+0000 7fd7fb856040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-19T13:01:36.178 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:01:36.176+0000 7fd7fb856040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-19T13:01:36.206 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:36.203+0000 7ff9c302e040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-19T13:01:36.279 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:01:36.276+0000 7fd7fb856040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-19T13:01:36.340 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:36.337+0000 7ff9c302e040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-19T13:01:36.374 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:01:36.372+0000 7fd7fb856040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-19T13:01:36.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:36.372+0000 7f5813fff640 1 -- 172.21.15.57:0/605136826 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f580006af20 msgr2=0x7f580006d3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/3281457486 2023-10-19T13:01:36.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:36.372+0000 7f5813fff640 1 --2- 172.21.15.57:0/605136826 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f580006af20 0x7f580006d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-19T13:01:36.380 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:36.378+0000 7ff9c302e040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-19T13:01:36.483 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:36.481+0000 7ff9c302e040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-19T13:01:36.518 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:01:36.515+0000 7fd7fb856040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-19T13:01:36.562 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:01:36.559+0000 7fd7fb856040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-19T13:01:36.672 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:01:36.670+0000 7fd7fb856040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-19T13:01:37.089 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:37.085+0000 7ff9c302e040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-19T13:01:37.320 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:01:37.318+0000 7fd7fb856040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-19T13:01:37.345 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:37.343+0000 7ff9c302e040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-19T13:01:37.394 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:37.391+0000 7ff9c302e040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-19T13:01:37.492 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:37.489+0000 7ff9c302e040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-19T13:01:37.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:37.514+0000 7f5810ff9640 1 -- 172.21.15.57:0/605136826 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mgrmap(e 49) v1 ==== 88957+0+0 (secure 0 0 0) 0x7f5808054310 con 0x7f58140cf340 2023-10-19T13:01:37.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:37.514+0000 7f5810ff9640 1 -- 172.21.15.57:0/605136826 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f580006af20 msgr2=0x7f580006d3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T13:01:37.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:37.514+0000 7f5810ff9640 1 --2- 172.21.15.57:0/605136826 >> [v2:172.21.15.179:6832/3281457486,v1:172.21.15.179:6833/3281457486] conn(0x7f580006af20 0x7f580006d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:37.605 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:01:37.603+0000 7fd7fb856040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-19T13:01:37.612 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:37.609+0000 7ff942464640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:01:37.612 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:37.609+0000 7ff942464640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:01:37.613 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:37.609+0000 7ff942464640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:01:37.613 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:37.609+0000 7ff942464640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:01:37.613 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:37.609+0000 7ff942464640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:01:37.614 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:37.610+0000 7ff93f41e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:01:37.614 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:37.610+0000 7ff93f41e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:01:37.614 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:37.610+0000 7ff93f41e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:01:37.614 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:37.610+0000 7ff93f41e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:01:37.615 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:01:37.610+0000 7ff93f41e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:01:37.658 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:01:37.656+0000 7fd7fb856040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-19T13:01:37.763 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:01:37.761+0000 7fd7fb856040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-19T13:01:38.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:38.516+0000 7f5810ff9640 1 -- 172.21.15.57:0/605136826 <== mon.0 v2:172.21.15.57:3300/0 8 ==== mgrmap(e 50) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f58080570f0 con 0x7f58140cf340 2023-10-19T13:01:38.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:38.517+0000 7f5810ff9640 1 --2- 172.21.15.57:0/605136826 >> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] conn(0x7f5800035b40 0x7f5800075350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:38.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:38.517+0000 7f5810ff9640 1 -- 172.21.15.57:0/605136826 --> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/bad-path", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f581415cf50 con 0x7f5800035b40 2023-10-19T13:01:38.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:38.517+0000 7f5813fff640 1 --2- 172.21.15.57:0/605136826 >> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] conn(0x7f5800035b40 0x7f5800075350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:38.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:38.521+0000 7f5813fff640 1 --2- 172.21.15.57:0/605136826 >> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] conn(0x7f5800035b40 0x7f5800075350 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f5804035000 tx=0x7f5804004730 comp rx=0 tx=0).ready entity=mgr.6725 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:38.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:38.577+0000 7f5810ff9640 1 -- 172.21.15.57:0/605136826 <== mgr.6725 v2:172.21.15.179:6832/2366702417 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f581415cf50 con 0x7f5800035b40 2023-10-19T13:01:38.580 INFO:teuthology.orchestra.run.smithi057.stdout:Schedule set for path /bad-path 2023-10-19T13:01:38.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:38.579+0000 7f581a3ce640 1 -- 172.21.15.57:0/605136826 >> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] conn(0x7f5800035b40 msgr2=0x7f5800075350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:38.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:38.579+0000 7f581a3ce640 1 --2- 172.21.15.57:0/605136826 >> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] conn(0x7f5800035b40 0x7f5800075350 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f5804035000 tx=0x7f5804004730 comp rx=0 tx=0).stop 2023-10-19T13:01:38.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:38.579+0000 7f581a3ce640 1 -- 172.21.15.57:0/605136826 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f58140cf340 msgr2=0x7f581415b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:38.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:38.579+0000 7f581a3ce640 1 --2- 172.21.15.57:0/605136826 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f58140cf340 0x7f581415b6d0 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f580800c400 tx=0x7f580800c8c0 comp rx=0 tx=0).stop 2023-10-19T13:01:38.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:38.579+0000 7f581896b640 1 -- 172.21.15.57:0/605136826 reap_dead start 2023-10-19T13:01:38.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:38.579+0000 7f581a3ce640 1 -- 172.21.15.57:0/605136826 shutdown_connections 2023-10-19T13:01:38.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:38.579+0000 7f581a3ce640 1 -- 172.21.15.57:0/605136826 >> 172.21.15.57:0/605136826 conn(0x7f58140d0d60 msgr2=0x7f58140cf720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:38.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:38.579+0000 7f581a3ce640 1 -- 172.21.15.57:0/605136826 shutdown_connections 2023-10-19T13:01:38.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:38.579+0000 7f581a3ce640 1 -- 172.21.15.57:0/605136826 wait complete. 2023-10-19T13:01:38.594 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /bad-path 2023-10-19T13:01:38.594 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /bad-path 2023-10-19T13:01:38.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:38.707+0000 7fcee260e640 1 -- 172.21.15.57:0/2488918192 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcec40047c0 msgr2=0x7fcec4004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:38.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:38.707+0000 7fcee260e640 1 --2- 172.21.15.57:0/2488918192 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcec40047c0 0x7fcec4004c40 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7fced0009b80 tx=0x7fcec40057f0 comp rx=0 tx=0).stop 2023-10-19T13:01:38.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:38.707+0000 7fcee0bab640 1 -- 172.21.15.57:0/2488918192 reap_dead start 2023-10-19T13:01:38.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:38.707+0000 7fcee260e640 1 -- 172.21.15.57:0/2488918192 shutdown_connections 2023-10-19T13:01:38.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:38.707+0000 7fcee260e640 1 -- 172.21.15.57:0/2488918192 >> 172.21.15.57:0/2488918192 conn(0x7fcedc0d0d60 msgr2=0x7fcedc0d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:38.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:38.707+0000 7fcee260e640 1 -- 172.21.15.57:0/2488918192 shutdown_connections 2023-10-19T13:01:38.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:38.707+0000 7fcee260e640 1 -- 172.21.15.57:0/2488918192 wait complete. 2023-10-19T13:01:38.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:38.708+0000 7fcee260e640 1 Processor -- start 2023-10-19T13:01:38.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:38.708+0000 7fcee260e640 1 -- start start 2023-10-19T13:01:38.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:38.709+0000 7fcee260e640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcec40047c0 0x7fcedc14f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:38.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:38.709+0000 7fcee260e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcedc0b4970 0x7fcedc15d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:38.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:38.709+0000 7fcee260e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcedc15dde0 0x7fcedc15c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:38.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:38.709+0000 7fcee260e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcedc15e260 con 0x7fcedc0b4970 2023-10-19T13:01:38.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:38.709+0000 7fcee260e640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcedc14fab0 con 0x7fcedc15dde0 2023-10-19T13:01:38.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:38.709+0000 7fcee260e640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcedc0c88e0 con 0x7fcec40047c0 2023-10-19T13:01:38.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:38.709+0000 7fcedb7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcedc0b4970 0x7fcedc15d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:38.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:38.709+0000 7fcedb7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcedc0b4970 0x7fcedc15d8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51642/0 (socket says 172.21.15.57:51642) 2023-10-19T13:01:38.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:38.709+0000 7fcedb7fe640 1 -- 172.21.15.57:0/2826036314 learned_addr learned my addr 172.21.15.57:0/2826036314 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:01:38.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:38.709+0000 7fcedbfff640 1 --2- 172.21.15.57:0/2826036314 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcec40047c0 0x7fcedc14f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:38.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:38.709+0000 7fcee0bab640 1 --2- 172.21.15.57:0/2826036314 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcedc15dde0 0x7fcedc15c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:38.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:38.709+0000 7fcedb7fe640 1 -- 172.21.15.57:0/2826036314 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcec40047c0 msgr2=0x7fcedc14f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:38.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:38.709+0000 7fcedb7fe640 1 --2- 172.21.15.57:0/2826036314 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcec40047c0 0x7fcedc14f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:38.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:38.709+0000 7fcedb7fe640 1 -- 172.21.15.57:0/2826036314 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcedc15dde0 msgr2=0x7fcedc15c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:38.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:38.709+0000 7fcedb7fe640 1 --2- 172.21.15.57:0/2826036314 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcedc15dde0 0x7fcedc15c0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:38.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:38.709+0000 7fcedb7fe640 1 -- 172.21.15.57:0/2826036314 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcedc0c7fb0 con 0x7fcedc0b4970 2023-10-19T13:01:38.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:38.710+0000 7fcedb7fe640 1 --2- 172.21.15.57:0/2826036314 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcedc0b4970 0x7fcedc15d8a0 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7fced0009ca0 tx=0x7fced0033c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:38.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:38.710+0000 7fced8ff9640 1 -- 172.21.15.57:0/2826036314 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fced003d820 con 0x7fcedc0b4970 2023-10-19T13:01:38.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:38.710+0000 7fced8ff9640 1 -- 172.21.15.57:0/2826036314 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fced003d980 con 0x7fcedc0b4970 2023-10-19T13:01:38.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:38.710+0000 7fcee260e640 1 -- 172.21.15.57:0/2826036314 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcedc0c8290 con 0x7fcedc0b4970 2023-10-19T13:01:38.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:38.710+0000 7fced8ff9640 1 -- 172.21.15.57:0/2826036314 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fced003dd70 con 0x7fcedc0b4970 2023-10-19T13:01:38.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:38.710+0000 7fcee260e640 1 -- 172.21.15.57:0/2826036314 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcedc0b4df0 con 0x7fcedc0b4970 2023-10-19T13:01:38.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:38.711+0000 7fcee260e640 1 -- 172.21.15.57:0/2826036314 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcedc15e1c0 con 0x7fcedc0b4970 2023-10-19T13:01:38.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:38.711+0000 7fced8ff9640 1 -- 172.21.15.57:0/2826036314 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 50) v1 ==== 89084+0+0 (secure 0 0 0) 0x7fced0045400 con 0x7fcedc0b4970 2023-10-19T13:01:38.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:38.712+0000 7fced8ff9640 1 --2- 172.21.15.57:0/2826036314 >> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] conn(0x7fcec806ad10 0x7fcec806d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:01:38.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:38.712+0000 7fced8ff9640 1 -- 172.21.15.57:0/2826036314 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 6358+0+0 (secure 0 0 0) 0x7fced0033e40 con 0x7fcedc0b4970 2023-10-19T13:01:38.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:38.712+0000 7fcedbfff640 1 --2- 172.21.15.57:0/2826036314 >> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] conn(0x7fcec806ad10 0x7fcec806d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:01:38.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:38.712+0000 7fcedbfff640 1 --2- 172.21.15.57:0/2826036314 >> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] conn(0x7fcec806ad10 0x7fcec806d1d0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fcecc009d70 tx=0x7fcecc005f90 comp rx=0 tx=0).ready entity=mgr.6725 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:01:38.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:38.716+0000 7fced8ff9640 1 -- 172.21.15.57:0/2826036314 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fced00774c0 con 0x7fcedc0b4970 2023-10-19T13:01:38.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:38.912+0000 7fcee260e640 1 -- 172.21.15.57:0/2826036314 --> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/bad-path", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fcedc0cf720 con 0x7fcec806ad10 2023-10-19T13:01:38.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:38.918+0000 7fced8ff9640 1 -- 172.21.15.57:0/2826036314 <== mgr.6725 v2:172.21.15.179:6832/2366702417 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+287 (secure 0 0 0) 0x7fcecc009da0 con 0x7fcec806ad10 2023-10-19T13:01:38.921 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:01:38.921 INFO:teuthology.orchestra.run.smithi057.stdout:[{"fs": "cephfs", "subvol": null, "path": "/bad-path", "rel_path": "/bad-path", "schedule": "1m", "retention": {}, "start": "2023-10-19T00:00:00", "created": "2023-10-19T13:01:38", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true}] 2023-10-19T13:01:38.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:38.920+0000 7fcee260e640 1 -- 172.21.15.57:0/2826036314 >> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] conn(0x7fcec806ad10 msgr2=0x7fcec806d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:38.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:38.920+0000 7fcee260e640 1 --2- 172.21.15.57:0/2826036314 >> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] conn(0x7fcec806ad10 0x7fcec806d1d0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fcecc009d70 tx=0x7fcecc005f90 comp rx=0 tx=0).stop 2023-10-19T13:01:38.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:38.920+0000 7fcee260e640 1 -- 172.21.15.57:0/2826036314 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcedc0b4970 msgr2=0x7fcedc15d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:01:38.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:38.920+0000 7fcee260e640 1 --2- 172.21.15.57:0/2826036314 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcedc0b4970 0x7fcedc15d8a0 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7fced0009ca0 tx=0x7fced0033c50 comp rx=0 tx=0).stop 2023-10-19T13:01:38.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:38.920+0000 7fcee260e640 1 -- 172.21.15.57:0/2826036314 shutdown_connections 2023-10-19T13:01:38.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:38.920+0000 7fcee260e640 1 --2- 172.21.15.57:0/2826036314 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcedc15dde0 0x7fcedc15c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:38.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:38.920+0000 7fcee260e640 1 --2- 172.21.15.57:0/2826036314 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcedc0b4970 0x7fcedc15d8a0 unknown :-1 s=CLOSED pgs=1324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:38.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:38.920+0000 7fcee260e640 1 --2- 172.21.15.57:0/2826036314 >> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] conn(0x7fcec806ad10 0x7fcec806d1d0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:38.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:38.920+0000 7fcee260e640 1 --2- 172.21.15.57:0/2826036314 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcec40047c0 0x7fcedc14f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:01:38.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:38.920+0000 7fcee260e640 1 -- 172.21.15.57:0/2826036314 >> 172.21.15.57:0/2826036314 conn(0x7fcedc0d0d60 msgr2=0x7fcedc06b250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:01:38.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:38.920+0000 7fcee260e640 1 -- 172.21.15.57:0/2826036314 shutdown_connections 2023-10-19T13:01:38.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:01:38.920+0000 7fcee260e640 1 -- 172.21.15.57:0/2826036314 wait complete. 2023-10-19T13:01:38.933 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/bad-path", "rel_path": "/bad-path", "schedule": "1m", "retention": {}, "start": "2023-10-19T00:00:00", "created": "2023-10-19T13:01:38", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true}] 2023-10-19T13:01:47.763 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:01:47.766 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:02:00.589 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:02:00.587+0000 7ff931c03640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:02:00.589 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:02:00.587+0000 7ff931c03640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:02:00.590 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:02:00.587+0000 7ff931c03640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:02:00.590 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:02:00.587+0000 7ff931c03640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:02:00.590 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:02:00.587+0000 7ff931c03640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:02:17.828 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:02:17.832 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:02:38.935 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /bad-path 2023-10-19T13:02:39.058 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.055+0000 7fc7dd78a640 1 -- 172.21.15.57:0/3141767057 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc7d815b2a0 msgr2=0x7fc7d80d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:39.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.055+0000 7fc7dd78a640 1 --2- 172.21.15.57:0/3141767057 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc7d815b2a0 0x7fc7d80d1910 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7fc7d815b680 tx=0x7fc7c002bd30 comp rx=0 tx=0).stop 2023-10-19T13:02:39.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.055+0000 7fc7dd78a640 1 -- 172.21.15.57:0/3141767057 shutdown_connections 2023-10-19T13:02:39.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.055+0000 7fc7dd78a640 1 --2- 172.21.15.57:0/3141767057 >> v2:172.21.15.179:3300/0 conn(0x7fc7d80ceb40 0x7fc7d80b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:39.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.055+0000 7fc7dd78a640 1 --2- 172.21.15.57:0/3141767057 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc7d815b2a0 0x7fc7d80d1910 unknown :-1 s=CLOSED pgs=1325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:39.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.055+0000 7fc7dd78a640 1 -- 172.21.15.57:0/3141767057 >> 172.21.15.57:0/3141767057 conn(0x7fc7d80d0d60 msgr2=0x7fc7d80d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:02:39.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.055+0000 7fc7dd78a640 1 -- 172.21.15.57:0/3141767057 shutdown_connections 2023-10-19T13:02:39.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.055+0000 7fc7dd78a640 1 -- 172.21.15.57:0/3141767057 wait complete. 2023-10-19T13:02:39.062 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.057+0000 7fc7dd78a640 1 Processor -- start 2023-10-19T13:02:39.062 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.057+0000 7fc7dd78a640 1 -- start start 2023-10-19T13:02:39.062 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.057+0000 7fc7dd78a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc7d80ceb40 0x7fc7d81e5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:39.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.057+0000 7fc7dd78a640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc7d80cf340 0x7fc7d81e6460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:39.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.057+0000 7fc7dd78a640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc7d815b2a0 0x7fc7d81e3a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:39.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.057+0000 7fc7dd78a640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d80c9eb0 con 0x7fc7d80ceb40 2023-10-19T13:02:39.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.057+0000 7fc7dd78a640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d80b4800 con 0x7fc7d80cf340 2023-10-19T13:02:39.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.057+0000 7fc7dd78a640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d81e69a0 con 0x7fc7d815b2a0 2023-10-19T13:02:39.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.057+0000 7fc7d6ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc7d80ceb40 0x7fc7d81e5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:39.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.058+0000 7fc7d6ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc7d80ceb40 0x7fc7d81e5cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43126/0 (socket says 172.21.15.57:43126) 2023-10-19T13:02:39.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.058+0000 7fc7d6ffd640 1 -- 172.21.15.57:0/117183417 learned_addr learned my addr 172.21.15.57:0/117183417 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:02:39.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.058+0000 7fc7d67fc640 1 --2- 172.21.15.57:0/117183417 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc7d80cf340 0x7fc7d81e6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:39.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.058+0000 7fc7d77fe640 1 --2- 172.21.15.57:0/117183417 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc7d815b2a0 0x7fc7d81e3a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:39.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.058+0000 7fc7d6ffd640 1 -- 172.21.15.57:0/117183417 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc7d815b2a0 msgr2=0x7fc7d81e3a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:39.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.058+0000 7fc7d6ffd640 1 --2- 172.21.15.57:0/117183417 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc7d815b2a0 0x7fc7d81e3a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:39.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.058+0000 7fc7d6ffd640 1 -- 172.21.15.57:0/117183417 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc7d80cf340 msgr2=0x7fc7d81e6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:39.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.058+0000 7fc7d6ffd640 1 --2- 172.21.15.57:0/117183417 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc7d80cf340 0x7fc7d81e6460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:39.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.058+0000 7fc7d6ffd640 1 -- 172.21.15.57:0/117183417 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7c0009560 con 0x7fc7d80ceb40 2023-10-19T13:02:39.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.058+0000 7fc7d6ffd640 1 --2- 172.21.15.57:0/117183417 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc7d80ceb40 0x7fc7d81e5cc0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7fc7d000c430 tx=0x7fc7d000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:02:39.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.058+0000 7fc7cf7fe640 1 -- 172.21.15.57:0/117183417 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7d0011790 con 0x7fc7d80ceb40 2023-10-19T13:02:39.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.058+0000 7fc7cf7fe640 1 -- 172.21.15.57:0/117183417 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fc7d00119e0 con 0x7fc7d80ceb40 2023-10-19T13:02:39.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.058+0000 7fc7dd78a640 1 -- 172.21.15.57:0/117183417 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7d80c98f0 con 0x7fc7d80ceb40 2023-10-19T13:02:39.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.058+0000 7fc7cf7fe640 1 -- 172.21.15.57:0/117183417 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7d0011e30 con 0x7fc7d80ceb40 2023-10-19T13:02:39.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.058+0000 7fc7dd78a640 1 -- 172.21.15.57:0/117183417 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7d80b4490 con 0x7fc7d80ceb40 2023-10-19T13:02:39.067 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.059+0000 7fc7dd78a640 1 -- 172.21.15.57:0/117183417 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7d80b42f0 con 0x7fc7d80ceb40 2023-10-19T13:02:39.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.060+0000 7fc7cf7fe640 1 -- 172.21.15.57:0/117183417 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 53) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fc7d001c460 con 0x7fc7d80ceb40 2023-10-19T13:02:39.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.060+0000 7fc7cf7fe640 1 --2- 172.21.15.57:0/117183417 >> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] conn(0x7fc7b806b0d0 0x7fc7b806d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:39.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.060+0000 7fc7cf7fe640 1 -- 172.21.15.57:0/117183417 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6517+0+0 (secure 0 0 0) 0x7fc7d0089ff0 con 0x7fc7d80ceb40 2023-10-19T13:02:39.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.060+0000 7fc7d67fc640 1 --2- 172.21.15.57:0/117183417 >> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] conn(0x7fc7b806b0d0 0x7fc7b806d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:39.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.061+0000 7fc7d67fc640 1 --2- 172.21.15.57:0/117183417 >> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] conn(0x7fc7b806b0d0 0x7fc7b806d590 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc7c0035000 tx=0x7fc7c0004730 comp rx=0 tx=0).ready entity=mgr.6725 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:02:39.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.065+0000 7fc7cf7fe640 1 -- 172.21.15.57:0/117183417 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc7d80b42f0 con 0x7fc7d80ceb40 2023-10-19T13:02:39.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.263+0000 7fc7dd78a640 1 -- 172.21.15.57:0/117183417 --> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/bad-path", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc7d80b0d30 con 0x7fc7b806b0d0 2023-10-19T13:02:39.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.271+0000 7fc7cf7fe640 1 -- 172.21.15.57:0/117183417 <== mgr.6725 v2:172.21.15.179:6832/2366702417 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+288 (secure 0 0 0) 0x7fc7c0004350 con 0x7fc7b806b0d0 2023-10-19T13:02:39.274 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:02:39.274 INFO:teuthology.orchestra.run.smithi057.stdout:[{"fs": "cephfs", "subvol": null, "path": "/bad-path", "rel_path": "/bad-path", "schedule": "1m", "retention": {}, "start": "2023-10-19T00:00:00", "created": "2023-10-19T13:01:38", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": false}] 2023-10-19T13:02:39.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.273+0000 7fc7dd78a640 1 -- 172.21.15.57:0/117183417 >> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] conn(0x7fc7b806b0d0 msgr2=0x7fc7b806d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:39.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.273+0000 7fc7dd78a640 1 --2- 172.21.15.57:0/117183417 >> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] conn(0x7fc7b806b0d0 0x7fc7b806d590 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc7c0035000 tx=0x7fc7c0004730 comp rx=0 tx=0).stop 2023-10-19T13:02:39.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.273+0000 7fc7dd78a640 1 -- 172.21.15.57:0/117183417 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc7d80ceb40 msgr2=0x7fc7d81e5cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:39.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.273+0000 7fc7dd78a640 1 --2- 172.21.15.57:0/117183417 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc7d80ceb40 0x7fc7d81e5cc0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7fc7d000c430 tx=0x7fc7d000c8f0 comp rx=0 tx=0).stop 2023-10-19T13:02:39.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.273+0000 7fc7dd78a640 1 -- 172.21.15.57:0/117183417 shutdown_connections 2023-10-19T13:02:39.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.273+0000 7fc7dd78a640 1 --2- 172.21.15.57:0/117183417 >> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] conn(0x7fc7b806b0d0 0x7fc7b806d590 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:39.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.273+0000 7fc7dd78a640 1 --2- 172.21.15.57:0/117183417 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc7d815b2a0 0x7fc7d81e3a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:39.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.273+0000 7fc7dd78a640 1 --2- 172.21.15.57:0/117183417 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc7d80cf340 0x7fc7d81e6460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:39.278 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.273+0000 7fc7dd78a640 1 --2- 172.21.15.57:0/117183417 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc7d80ceb40 0x7fc7d81e5cc0 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:39.278 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.273+0000 7fc7dd78a640 1 -- 172.21.15.57:0/117183417 >> 172.21.15.57:0/117183417 conn(0x7fc7d80d0d60 msgr2=0x7fc7d815a360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:02:39.278 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.273+0000 7fc7dd78a640 1 -- 172.21.15.57:0/117183417 shutdown_connections 2023-10-19T13:02:39.278 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.273+0000 7fc7dd78a640 1 -- 172.21.15.57:0/117183417 wait complete. 2023-10-19T13:02:39.286 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/bad-path", "rel_path": "/bad-path", "schedule": "1m", "retention": {}, "start": "2023-10-19T00:00:00", "created": "2023-10-19T13:01:38", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": false}] 2023-10-19T13:02:39.287 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /bad-path 2023-10-19T13:02:39.398 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.395+0000 7fbf706c9640 1 -- 172.21.15.57:0/3039052983 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbf680ceb40 msgr2=0x7fbf680b0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:39.399 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.395+0000 7fbf706c9640 1 --2- 172.21.15.57:0/3039052983 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbf680ceb40 0x7fbf680b0650 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7fbf60005fd0 tx=0x7fbf6002c240 comp rx=0 tx=0).stop 2023-10-19T13:02:39.399 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.395+0000 7fbf706c9640 1 -- 172.21.15.57:0/3039052983 shutdown_connections 2023-10-19T13:02:39.399 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.395+0000 7fbf706c9640 1 --2- 172.21.15.57:0/3039052983 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbf680ceb40 0x7fbf680b0650 unknown :-1 s=CLOSED pgs=1327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:39.399 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.395+0000 7fbf706c9640 1 --2- 172.21.15.57:0/3039052983 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbf6815b2a0 0x7fbf680d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:39.400 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.395+0000 7fbf706c9640 1 -- 172.21.15.57:0/3039052983 >> 172.21.15.57:0/3039052983 conn(0x7fbf680d0d60 msgr2=0x7fbf680d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:02:39.400 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.395+0000 7fbf706c9640 1 -- 172.21.15.57:0/3039052983 shutdown_connections 2023-10-19T13:02:39.400 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.395+0000 7fbf706c9640 1 -- 172.21.15.57:0/3039052983 wait complete. 2023-10-19T13:02:39.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.397+0000 7fbf706c9640 1 Processor -- start 2023-10-19T13:02:39.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.397+0000 7fbf706c9640 1 -- start start 2023-10-19T13:02:39.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.397+0000 7fbf706c9640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbf680ceb40 0x7fbf681511e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:39.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.397+0000 7fbf706c9640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbf680cf340 0x7fbf680c9a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:39.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.397+0000 7fbf706c9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbf6815b2a0 0x7fbf680c9f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:39.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.397+0000 7fbf706c9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf680b3b80 con 0x7fbf6815b2a0 2023-10-19T13:02:39.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.397+0000 7fbf706c9640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf680ca5b0 con 0x7fbf680cf340 2023-10-19T13:02:39.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.397+0000 7fbf706c9640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf680ca890 con 0x7fbf680ceb40 2023-10-19T13:02:39.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.397+0000 7fbf6ec66640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbf6815b2a0 0x7fbf680c9f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:39.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.397+0000 7fbf6ec66640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbf6815b2a0 0x7fbf680c9f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43150/0 (socket says 172.21.15.57:43150) 2023-10-19T13:02:39.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.397+0000 7fbf6ec66640 1 -- 172.21.15.57:0/162850769 learned_addr learned my addr 172.21.15.57:0/162850769 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:02:39.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.398+0000 7fbf6e465640 1 --2- 172.21.15.57:0/162850769 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbf680ceb40 0x7fbf681511e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:39.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.398+0000 7fbf6dc64640 1 --2- 172.21.15.57:0/162850769 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbf680cf340 0x7fbf680c9a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:39.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.398+0000 7fbf6ec66640 1 -- 172.21.15.57:0/162850769 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbf680ceb40 msgr2=0x7fbf681511e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:39.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.398+0000 7fbf6ec66640 1 --2- 172.21.15.57:0/162850769 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbf680ceb40 0x7fbf681511e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:39.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.398+0000 7fbf6ec66640 1 -- 172.21.15.57:0/162850769 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbf680cf340 msgr2=0x7fbf680c9a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:39.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.398+0000 7fbf6ec66640 1 --2- 172.21.15.57:0/162850769 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbf680cf340 0x7fbf680c9a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:39.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.398+0000 7fbf6ec66640 1 -- 172.21.15.57:0/162850769 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf60009900 con 0x7fbf6815b2a0 2023-10-19T13:02:39.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.398+0000 7fbf6ec66640 1 --2- 172.21.15.57:0/162850769 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbf6815b2a0 0x7fbf680c9f80 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7fbf60009bf0 tx=0x7fbf60035e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:02:39.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.398+0000 7fbf5effd640 1 -- 172.21.15.57:0/162850769 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf60034440 con 0x7fbf6815b2a0 2023-10-19T13:02:39.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.398+0000 7fbf5effd640 1 -- 172.21.15.57:0/162850769 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fbf60003e60 con 0x7fbf6815b2a0 2023-10-19T13:02:39.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.398+0000 7fbf5effd640 1 -- 172.21.15.57:0/162850769 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf600044b0 con 0x7fbf6815b2a0 2023-10-19T13:02:39.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.398+0000 7fbf706c9640 1 -- 172.21.15.57:0/162850769 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf680b0d50 con 0x7fbf6815b2a0 2023-10-19T13:02:39.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.398+0000 7fbf706c9640 1 -- 172.21.15.57:0/162850769 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf680cae50 con 0x7fbf6815b2a0 2023-10-19T13:02:39.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.399+0000 7fbf706c9640 1 -- 172.21.15.57:0/162850769 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf680b3cf0 con 0x7fbf6815b2a0 2023-10-19T13:02:39.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.400+0000 7fbf5effd640 1 -- 172.21.15.57:0/162850769 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 53) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fbf6000a040 con 0x7fbf6815b2a0 2023-10-19T13:02:39.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.400+0000 7fbf5effd640 1 --2- 172.21.15.57:0/162850769 >> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] conn(0x7fbf4c06b0d0 0x7fbf4c06d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:39.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.400+0000 7fbf5effd640 1 -- 172.21.15.57:0/162850769 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 6517+0+0 (secure 0 0 0) 0x7fbf600aca00 con 0x7fbf6815b2a0 2023-10-19T13:02:39.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.400+0000 7fbf6e465640 1 --2- 172.21.15.57:0/162850769 >> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] conn(0x7fbf4c06b0d0 0x7fbf4c06d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:39.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.401+0000 7fbf6e465640 1 --2- 172.21.15.57:0/162850769 >> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] conn(0x7fbf4c06b0d0 0x7fbf4c06d590 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fbf64001780 tx=0x7fbf64009040 comp rx=0 tx=0).ready entity=mgr.6725 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:02:39.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.404+0000 7fbf5effd640 1 -- 172.21.15.57:0/162850769 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fbf60077680 con 0x7fbf6815b2a0 2023-10-19T13:02:39.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.601+0000 7fbf706c9640 1 -- 172.21.15.57:0/162850769 --> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/bad-path", "target": ["mon-mgr", ""]}) v1 -- 0x7fbf680b0bd0 con 0x7fbf4c06b0d0 2023-10-19T13:02:39.662 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.659+0000 7fbf5effd640 1 -- 172.21.15.57:0/162850769 <== mgr.6725 v2:172.21.15.179:6832/2366702417 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+36 (secure 0 0 0) 0x7fbf64001d60 con 0x7fbf4c06b0d0 2023-10-19T13:02:39.662 INFO:teuthology.orchestra.run.smithi057.stdout:Schedule removed for path /bad-path 2023-10-19T13:02:39.664 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.661+0000 7fbf706c9640 1 -- 172.21.15.57:0/162850769 >> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] conn(0x7fbf4c06b0d0 msgr2=0x7fbf4c06d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:39.664 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.661+0000 7fbf706c9640 1 --2- 172.21.15.57:0/162850769 >> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] conn(0x7fbf4c06b0d0 0x7fbf4c06d590 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fbf64001780 tx=0x7fbf64009040 comp rx=0 tx=0).stop 2023-10-19T13:02:39.664 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.661+0000 7fbf706c9640 1 -- 172.21.15.57:0/162850769 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbf6815b2a0 msgr2=0x7fbf680c9f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:39.665 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.661+0000 7fbf706c9640 1 --2- 172.21.15.57:0/162850769 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbf6815b2a0 0x7fbf680c9f80 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7fbf60009bf0 tx=0x7fbf60035e30 comp rx=0 tx=0).stop 2023-10-19T13:02:39.665 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.661+0000 7fbf706c9640 1 -- 172.21.15.57:0/162850769 shutdown_connections 2023-10-19T13:02:39.665 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.661+0000 7fbf706c9640 1 --2- 172.21.15.57:0/162850769 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbf6815b2a0 0x7fbf680c9f80 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:39.665 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.661+0000 7fbf706c9640 1 --2- 172.21.15.57:0/162850769 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbf680cf340 0x7fbf680c9a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:39.665 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.661+0000 7fbf706c9640 1 --2- 172.21.15.57:0/162850769 >> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] conn(0x7fbf4c06b0d0 0x7fbf4c06d590 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:39.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.661+0000 7fbf706c9640 1 --2- 172.21.15.57:0/162850769 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbf680ceb40 0x7fbf681511e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:39.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.661+0000 7fbf706c9640 1 -- 172.21.15.57:0/162850769 >> 172.21.15.57:0/162850769 conn(0x7fbf680d0d60 msgr2=0x7fbf6806b730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:02:39.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.661+0000 7fbf706c9640 1 -- 172.21.15.57:0/162850769 shutdown_connections 2023-10-19T13:02:39.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.661+0000 7fbf706c9640 1 -- 172.21.15.57:0/162850769 wait complete. 2023-10-19T13:02:39.674 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /bad-path 2023-10-19T13:02:39.675 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-19T13:02:39.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.782+0000 7f14189c5640 1 -- 172.21.15.57:0/1962833001 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f14140abc10 msgr2=0x7f14140c9b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:39.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.782+0000 7f14189c5640 1 --2- 172.21.15.57:0/1962833001 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f14140abc10 0x7f14140c9b60 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f140c005fd0 tx=0x7f140c02c250 comp rx=0 tx=0).stop 2023-10-19T13:02:39.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.782+0000 7f14189c5640 1 -- 172.21.15.57:0/1962833001 shutdown_connections 2023-10-19T13:02:39.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.782+0000 7f14189c5640 1 --2- 172.21.15.57:0/1962833001 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f14140abc10 0x7f14140c9b60 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:39.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.782+0000 7f14189c5640 1 --2- 172.21.15.57:0/1962833001 >> v2:172.21.15.179:3300/0 conn(0x7f141406af00 0x7f141406b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:39.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.782+0000 7f14189c5640 1 -- 172.21.15.57:0/1962833001 >> 172.21.15.57:0/1962833001 conn(0x7f1414069930 msgr2=0x7f1414069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:02:39.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.782+0000 7f14189c5640 1 -- 172.21.15.57:0/1962833001 shutdown_connections 2023-10-19T13:02:39.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.782+0000 7f14189c5640 1 -- 172.21.15.57:0/1962833001 wait complete. 2023-10-19T13:02:39.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.784+0000 7f14189c5640 1 Processor -- start 2023-10-19T13:02:39.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.784+0000 7f14189c5640 1 -- start start 2023-10-19T13:02:39.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.785+0000 7f14189c5640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f141406af00 0x7f141413a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:39.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.785+0000 7f14189c5640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f14140abc10 0x7f14140a7200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:39.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.785+0000 7f14189c5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f14140c96b0 0x7f1414139290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:39.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.785+0000 7f14189c5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14140a6de0 con 0x7f14140c96b0 2023-10-19T13:02:39.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.785+0000 7f14189c5640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14140a8d80 con 0x7f141406af00 2023-10-19T13:02:39.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.785+0000 7f14189c5640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14140a9060 con 0x7f14140abc10 2023-10-19T13:02:39.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.785+0000 7f1412d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f14140c96b0 0x7f1414139290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:39.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.785+0000 7f1412d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f14140c96b0 0x7f1414139290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43162/0 (socket says 172.21.15.57:43162) 2023-10-19T13:02:39.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.785+0000 7f1412d9d640 1 -- 172.21.15.57:0/1340971109 learned_addr learned my addr 172.21.15.57:0/1340971109 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:02:39.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.785+0000 7f141259c640 1 --2- 172.21.15.57:0/1340971109 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f141406af00 0x7f141413a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:39.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.785+0000 7f1409d9b640 1 --2- 172.21.15.57:0/1340971109 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f14140abc10 0x7f14140a7200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:39.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.785+0000 7f1412d9d640 1 -- 172.21.15.57:0/1340971109 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f14140abc10 msgr2=0x7f14140a7200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:39.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.785+0000 7f1412d9d640 1 --2- 172.21.15.57:0/1340971109 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f14140abc10 0x7f14140a7200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:39.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.785+0000 7f1412d9d640 1 -- 172.21.15.57:0/1340971109 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f141406af00 msgr2=0x7f141413a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:39.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.785+0000 7f1412d9d640 1 --2- 172.21.15.57:0/1340971109 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f141406af00 0x7f141413a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:39.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.785+0000 7f1412d9d640 1 -- 172.21.15.57:0/1340971109 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f140c009900 con 0x7f14140c96b0 2023-10-19T13:02:39.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.785+0000 7f1412d9d640 1 --2- 172.21.15.57:0/1340971109 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f14140c96b0 0x7f1414139290 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7f140c037040 tx=0x7f140c035db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:02:39.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.785+0000 7f140b7fe640 1 -- 172.21.15.57:0/1340971109 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f140c034440 con 0x7f14140c96b0 2023-10-19T13:02:39.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.785+0000 7f140b7fe640 1 -- 172.21.15.57:0/1340971109 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f140c004250 con 0x7f14140c96b0 2023-10-19T13:02:39.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.785+0000 7f14189c5640 1 -- 172.21.15.57:0/1340971109 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14140a6a00 con 0x7f14140c96b0 2023-10-19T13:02:39.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.785+0000 7f140b7fe640 1 -- 172.21.15.57:0/1340971109 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f140c004600 con 0x7f14140c96b0 2023-10-19T13:02:39.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.785+0000 7f14189c5640 1 -- 172.21.15.57:0/1340971109 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14140a8630 con 0x7f14140c96b0 2023-10-19T13:02:39.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.786+0000 7f14189c5640 1 -- 172.21.15.57:0/1340971109 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14140a6f20 con 0x7f14140c96b0 2023-10-19T13:02:39.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.787+0000 7f140b7fe640 1 -- 172.21.15.57:0/1340971109 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 53) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f140c03b040 con 0x7f14140c96b0 2023-10-19T13:02:39.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.787+0000 7f140b7fe640 1 --2- 172.21.15.57:0/1340971109 >> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] conn(0x7f13f406b0d0 0x7f13f406d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:39.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.787+0000 7f140b7fe640 1 -- 172.21.15.57:0/1340971109 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f140c0ac900 con 0x7f14140c96b0 2023-10-19T13:02:39.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.788+0000 7f141259c640 1 --2- 172.21.15.57:0/1340971109 >> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] conn(0x7f13f406b0d0 0x7f13f406d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:39.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.788+0000 7f141259c640 1 --2- 172.21.15.57:0/1340971109 >> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] conn(0x7f13f406b0d0 0x7f13f406d590 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f14140c8bd0 tx=0x7f140400b390 comp rx=0 tx=0).ready entity=mgr.6725 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:02:39.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:39.792+0000 7f140b7fe640 1 -- 172.21.15.57:0/1340971109 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f14140a6f20 con 0x7f14140c96b0 2023-10-19T13:02:40.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.014+0000 7f14189c5640 1 -- 172.21.15.57:0/1340971109 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f14140a6f20 con 0x7f14140c96b0 2023-10-19T13:02:40.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.247+0000 7f140b7fe640 1 -- 172.21.15.57:0/1340971109 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v54) v1 ==== 93+0+0 (secure 0 0 0) 0x7f14140a6f20 con 0x7f14140c96b0 2023-10-19T13:02:40.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.249+0000 7f14189c5640 1 -- 172.21.15.57:0/1340971109 >> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] conn(0x7f13f406b0d0 msgr2=0x7f13f406d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:40.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.249+0000 7f14189c5640 1 --2- 172.21.15.57:0/1340971109 >> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] conn(0x7f13f406b0d0 0x7f13f406d590 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f14140c8bd0 tx=0x7f140400b390 comp rx=0 tx=0).stop 2023-10-19T13:02:40.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.249+0000 7f14189c5640 1 -- 172.21.15.57:0/1340971109 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f14140c96b0 msgr2=0x7f1414139290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:40.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.250+0000 7f140b7fe640 1 -- 172.21.15.57:0/1340971109 <== mon.0 v2:172.21.15.57:3300/0 8 ==== mgrmap(e 54) v1 ==== 89445+0+0 (secure 0 0 0) 0x7f140c00a040 con 0x7f14140c96b0 2023-10-19T13:02:40.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.250+0000 7f14189c5640 1 --2- 172.21.15.57:0/1340971109 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f14140c96b0 0x7f1414139290 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7f140c037040 tx=0x7f140c035db0 comp rx=0 tx=0).stop 2023-10-19T13:02:40.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.250+0000 7f14189c5640 1 -- 172.21.15.57:0/1340971109 shutdown_connections 2023-10-19T13:02:40.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.250+0000 7f14189c5640 1 --2- 172.21.15.57:0/1340971109 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f14140c96b0 0x7f1414139290 unknown :-1 s=CLOSED pgs=1331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:40.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.250+0000 7f14189c5640 1 --2- 172.21.15.57:0/1340971109 >> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] conn(0x7f13f406b0d0 0x7f13f406d590 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:40.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.250+0000 7f14189c5640 1 --2- 172.21.15.57:0/1340971109 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f14140abc10 0x7f14140a7200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:40.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.250+0000 7f14189c5640 1 --2- 172.21.15.57:0/1340971109 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f141406af00 0x7f141413a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:40.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.250+0000 7f14189c5640 1 -- 172.21.15.57:0/1340971109 >> 172.21.15.57:0/1340971109 conn(0x7f1414069930 msgr2=0x7f1414069d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:02:40.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.250+0000 7f14189c5640 1 -- 172.21.15.57:0/1340971109 shutdown_connections 2023-10-19T13:02:40.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.250+0000 7f14189c5640 1 -- 172.21.15.57:0/1340971109 wait complete. 2023-10-19T13:02:40.265 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-19T13:02:40.265 DEBUG:teuthology.orchestra.run.smithi057:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-19T13:02:40.308 DEBUG:teuthology.orchestra.run.smithi179:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-19T13:02:40.372 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-19T13:02:40.373 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T13:02:40.373 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T13:02:40.389 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T13:02:40.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.538+0000 7fb502d6c640 1 -- 172.21.15.57:0/528468349 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb4fc0cf340 msgr2=0x7fb4fc0d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:40.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.538+0000 7fb502d6c640 1 --2- 172.21.15.57:0/528468349 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb4fc0cf340 0x7fb4fc0d1910 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb4f0009830 tx=0x7fb4f002bcd0 comp rx=0 tx=0).stop 2023-10-19T13:02:40.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.538+0000 7fb502d6c640 1 -- 172.21.15.57:0/528468349 shutdown_connections 2023-10-19T13:02:40.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.538+0000 7fb502d6c640 1 --2- 172.21.15.57:0/528468349 >> v2:172.21.15.179:3300/0 conn(0x7fb4fc0ceb40 0x7fb4fc0b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:40.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.538+0000 7fb502d6c640 1 --2- 172.21.15.57:0/528468349 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb4fc0cf340 0x7fb4fc0d1910 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:40.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.538+0000 7fb502d6c640 1 -- 172.21.15.57:0/528468349 >> 172.21.15.57:0/528468349 conn(0x7fb4fc0d0d60 msgr2=0x7fb4fc0d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:02:40.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.538+0000 7fb502d6c640 1 -- 172.21.15.57:0/528468349 shutdown_connections 2023-10-19T13:02:40.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.538+0000 7fb502d6c640 1 -- 172.21.15.57:0/528468349 wait complete. 2023-10-19T13:02:40.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.539+0000 7fb502d6c640 1 Processor -- start 2023-10-19T13:02:40.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.539+0000 7fb502d6c640 1 -- start start 2023-10-19T13:02:40.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.540+0000 7fb502d6c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb4fc0ceb40 0x7fb4fc15e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:40.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.540+0000 7fb502d6c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4fc0cf340 0x7fb4fc0b1650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:40.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.540+0000 7fb502d6c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb4fc15b2a0 0x7fb4fc0b1b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:40.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.540+0000 7fb502d6c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4fc15be30 con 0x7fb4fc0cf340 2023-10-19T13:02:40.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.540+0000 7fb502d6c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4fc159a10 con 0x7fb4fc15b2a0 2023-10-19T13:02:40.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.540+0000 7fb502d6c640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4fc1f52f0 con 0x7fb4fc0ceb40 2023-10-19T13:02:40.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.540+0000 7fb4fbfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4fc0cf340 0x7fb4fc0b1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:40.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.540+0000 7fb500b08640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb4fc0ceb40 0x7fb4fc15e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:40.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.540+0000 7fb4fbfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4fc0cf340 0x7fb4fc0b1650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43184/0 (socket says 172.21.15.57:43184) 2023-10-19T13:02:40.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.540+0000 7fb4fbfff640 1 -- 172.21.15.57:0/1679208590 learned_addr learned my addr 172.21.15.57:0/1679208590 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:02:40.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.540+0000 7fb501309640 1 --2- 172.21.15.57:0/1679208590 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb4fc15b2a0 0x7fb4fc0b1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:40.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.540+0000 7fb4fbfff640 1 -- 172.21.15.57:0/1679208590 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb4fc0ceb40 msgr2=0x7fb4fc15e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:40.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.540+0000 7fb4fbfff640 1 --2- 172.21.15.57:0/1679208590 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb4fc0ceb40 0x7fb4fc15e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:40.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.540+0000 7fb4fbfff640 1 -- 172.21.15.57:0/1679208590 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb4fc15b2a0 msgr2=0x7fb4fc0b1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:40.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.540+0000 7fb4fbfff640 1 --2- 172.21.15.57:0/1679208590 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb4fc15b2a0 0x7fb4fc0b1b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:40.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.540+0000 7fb4fbfff640 1 -- 172.21.15.57:0/1679208590 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4f4009840 con 0x7fb4fc0cf340 2023-10-19T13:02:40.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.540+0000 7fb500b08640 1 --2- 172.21.15.57:0/1679208590 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb4fc0ceb40 0x7fb4fc15e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T13:02:40.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.540+0000 7fb501309640 1 --2- 172.21.15.57:0/1679208590 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb4fc15b2a0 0x7fb4fc0b1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T13:02:40.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.541+0000 7fb4fbfff640 1 --2- 172.21.15.57:0/1679208590 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4fc0cf340 0x7fb4fc0b1650 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7fb4e80046b0 tx=0x7fb4e800c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:02:40.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.541+0000 7fb4f97fa640 1 -- 172.21.15.57:0/1679208590 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4e8004230 con 0x7fb4fc0cf340 2023-10-19T13:02:40.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.541+0000 7fb4f97fa640 1 -- 172.21.15.57:0/1679208590 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fb4e8004480 con 0x7fb4fc0cf340 2023-10-19T13:02:40.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.541+0000 7fb4f97fa640 1 -- 172.21.15.57:0/1679208590 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4e8003850 con 0x7fb4fc0cf340 2023-10-19T13:02:40.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.541+0000 7fb502d6c640 1 -- 172.21.15.57:0/1679208590 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4f0009490 con 0x7fb4fc0cf340 2023-10-19T13:02:40.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.541+0000 7fb502d6c640 1 -- 172.21.15.57:0/1679208590 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4fc15b770 con 0x7fb4fc0cf340 2023-10-19T13:02:40.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.542+0000 7fb502d6c640 1 -- 172.21.15.57:0/1679208590 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4fc0690e0 con 0x7fb4fc0cf340 2023-10-19T13:02:40.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.542+0000 7fb4f97fa640 1 -- 172.21.15.57:0/1679208590 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 54) v1 ==== 89445+0+0 (secure 0 0 0) 0x7fb4e8003a70 con 0x7fb4fc0cf340 2023-10-19T13:02:40.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.543+0000 7fb4f97fa640 1 --2- 172.21.15.57:0/1679208590 >> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] conn(0x7fb4e006b080 0x7fb4e006d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:40.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.543+0000 7fb500b08640 1 -- 172.21.15.57:0/1679208590 >> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] conn(0x7fb4e006b080 msgr2=0x7fb4e006d540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/2366702417 2023-10-19T13:02:40.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.543+0000 7fb4f97fa640 1 -- 172.21.15.57:0/1679208590 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 6517+0+0 (secure 0 0 0) 0x7fb4e80892b0 con 0x7fb4fc0cf340 2023-10-19T13:02:40.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.543+0000 7fb500b08640 1 --2- 172.21.15.57:0/1679208590 >> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] conn(0x7fb4e006b080 0x7fb4e006d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T13:02:40.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.547+0000 7fb4f97fa640 1 -- 172.21.15.57:0/1679208590 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb4e8053e60 con 0x7fb4fc0cf340 2023-10-19T13:02:40.639 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:02:40.636+0000 7fc269232040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-19T13:02:40.651 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:02:40.649+0000 7f380220d040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-19T13:02:40.683 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:02:40.681+0000 7fc269232040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-19T13:02:40.699 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:02:40.697+0000 7f380220d040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-19T13:02:40.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.724+0000 7fb502d6c640 1 -- 172.21.15.57:0/1679208590 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb4fc15d8c0 con 0x7fb4fc0cf340 2023-10-19T13:02:40.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.725+0000 7fb4f97fa640 1 -- 172.21.15.57:0/1679208590 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v117) v1 ==== 74+0+17484 (secure 0 0 0) 0x7fb4e8057480 con 0x7fb4fc0cf340 2023-10-19T13:02:40.728 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:02:40.728 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":117,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T13:02:39.226727+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2023-10-19T13:01:24.443125+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2023-10-19T13:01:25.101276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"113","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.1","mappings":[{"from":1,"to":5}]},{"pgid":"14.12","mappings":[{"from":1,"to":7}]},{"pgid":"14.2d","mappings":[{"from":1,"to":7}]},{"pgid":"14.32","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.179:0/3304791412":"2023-10-19T14:01:39.552724+0000","172.21.15.179:0/3109348805":"2023-10-20T13:01:37.504869+0000","172.21.15.179:0/2984488835":"2023-10-20T13:01:37.504869+0000","172.21.15.179:0/205205608":"2023-10-20T13:01:37.504869+0000","172.21.15.179:0/4112649564":"2023-10-20T13:01:37.504869+0000","172.21.15.179:6833/3281457486":"2023-10-20T13:01:37.504869+0000","172.21.15.179:6832/3281457486":"2023-10-20T13:01:37.504869+0000","172.21.15.179:0/2509136709":"2023-10-20T13:01:37.504869+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T13:02:40.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.727+0000 7fb502d6c640 1 -- 172.21.15.57:0/1679208590 >> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] conn(0x7fb4e006b080 msgr2=0x7fb4e006d540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T13:02:40.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.727+0000 7fb502d6c640 1 --2- 172.21.15.57:0/1679208590 >> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] conn(0x7fb4e006b080 0x7fb4e006d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:40.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.727+0000 7fb502d6c640 1 -- 172.21.15.57:0/1679208590 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4fc0cf340 msgr2=0x7fb4fc0b1650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:40.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.727+0000 7fb502d6c640 1 --2- 172.21.15.57:0/1679208590 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4fc0cf340 0x7fb4fc0b1650 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7fb4e80046b0 tx=0x7fb4e800c760 comp rx=0 tx=0).stop 2023-10-19T13:02:40.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.727+0000 7fb502d6c640 1 -- 172.21.15.57:0/1679208590 shutdown_connections 2023-10-19T13:02:40.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.727+0000 7fb502d6c640 1 --2- 172.21.15.57:0/1679208590 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb4fc15b2a0 0x7fb4fc0b1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:40.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.727+0000 7fb502d6c640 1 --2- 172.21.15.57:0/1679208590 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4fc0cf340 0x7fb4fc0b1650 unknown :-1 s=CLOSED pgs=1334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:40.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.727+0000 7fb502d6c640 1 --2- 172.21.15.57:0/1679208590 >> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] conn(0x7fb4e006b080 0x7fb4e006d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:40.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.727+0000 7fb502d6c640 1 --2- 172.21.15.57:0/1679208590 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb4fc0ceb40 0x7fb4fc15e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:40.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.727+0000 7fb502d6c640 1 -- 172.21.15.57:0/1679208590 >> 172.21.15.57:0/1679208590 conn(0x7fb4fc0d0d60 msgr2=0x7fb4fc0adf30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:02:40.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.727+0000 7fb502d6c640 1 -- 172.21.15.57:0/1679208590 shutdown_connections 2023-10-19T13:02:40.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.727+0000 7fb502d6c640 1 -- 172.21.15.57:0/1679208590 wait complete. 2023-10-19T13:02:40.741 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-19T12:44:07.371186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-19T13:01:24.443125+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '112', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_data', 'create_time': '2023-10-19T13:01:25.101276+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '113', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-19T13:02:40.742 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-19T13:02:40.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.852+0000 7f7670efc640 1 -- 172.21.15.57:0/3248843967 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f766c15b2a0 msgr2=0x7f766c0d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:40.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.852+0000 7f7670efc640 1 --2- 172.21.15.57:0/3248843967 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f766c15b2a0 0x7f766c0d1910 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f7654009830 tx=0x7f765402bd60 comp rx=0 tx=0).stop 2023-10-19T13:02:40.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.852+0000 7f7670efc640 1 -- 172.21.15.57:0/3248843967 shutdown_connections 2023-10-19T13:02:40.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.852+0000 7f7670efc640 1 --2- 172.21.15.57:0/3248843967 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f766c15b2a0 0x7f766c0d1910 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:40.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.852+0000 7f7670efc640 1 -- 172.21.15.57:0/3248843967 >> 172.21.15.57:0/3248843967 conn(0x7f766c0d0d60 msgr2=0x7f766c0d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:02:40.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.853+0000 7f7670efc640 1 -- 172.21.15.57:0/3248843967 shutdown_connections 2023-10-19T13:02:40.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.853+0000 7f7670efc640 1 -- 172.21.15.57:0/3248843967 wait complete. 2023-10-19T13:02:40.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.854+0000 7f7670efc640 1 Processor -- start 2023-10-19T13:02:40.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.854+0000 7f7670efc640 1 -- start start 2023-10-19T13:02:40.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.855+0000 7f7670efc640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f766c0ceb40 0x7f766c1f0e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:40.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.855+0000 7f7670efc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f766c0cf340 0x7f766c0ca040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:40.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.855+0000 7f7670efc640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f766c15b2a0 0x7f766c0ca580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:40.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.855+0000 7f7670efc640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f766c0c8cd0 con 0x7f766c0cf340 2023-10-19T13:02:40.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.855+0000 7f7670efc640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f766c0c8fb0 con 0x7f766c0ceb40 2023-10-19T13:02:40.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.855+0000 7f7670efc640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f766c0c9290 con 0x7f766c15b2a0 2023-10-19T13:02:40.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.855+0000 7f7663fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f766c0cf340 0x7f766c0ca040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:40.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.855+0000 7f766a59c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f766c0ceb40 0x7f766c1f0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:40.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.855+0000 7f7663fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f766c0cf340 0x7f766c0ca040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43192/0 (socket says 172.21.15.57:43192) 2023-10-19T13:02:40.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.855+0000 7f7663fff640 1 -- 172.21.15.57:0/2008357317 learned_addr learned my addr 172.21.15.57:0/2008357317 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:02:40.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.855+0000 7f766ad9d640 1 --2- 172.21.15.57:0/2008357317 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f766c15b2a0 0x7f766c0ca580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:40.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.855+0000 7f7663fff640 1 -- 172.21.15.57:0/2008357317 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f766c15b2a0 msgr2=0x7f766c0ca580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:40.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.855+0000 7f7663fff640 1 --2- 172.21.15.57:0/2008357317 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f766c15b2a0 0x7f766c0ca580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:40.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.855+0000 7f7663fff640 1 -- 172.21.15.57:0/2008357317 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f766c0ceb40 msgr2=0x7f766c1f0e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:40.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.855+0000 7f7663fff640 1 --2- 172.21.15.57:0/2008357317 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f766c0ceb40 0x7f766c1f0e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:40.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.855+0000 7f7663fff640 1 -- 172.21.15.57:0/2008357317 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7654009490 con 0x7f766c0cf340 2023-10-19T13:02:40.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.855+0000 7f7663fff640 1 --2- 172.21.15.57:0/2008357317 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f766c0cf340 0x7f766c0ca040 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7f7654009950 tx=0x7f76540038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:02:40.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.855+0000 7f7662ffd640 1 -- 172.21.15.57:0/2008357317 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7654003ea0 con 0x7f766c0cf340 2023-10-19T13:02:40.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.855+0000 7f7662ffd640 1 -- 172.21.15.57:0/2008357317 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f7654005c90 con 0x7f766c0cf340 2023-10-19T13:02:40.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.855+0000 7f7670efc640 1 -- 172.21.15.57:0/2008357317 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f766c0c9570 con 0x7f766c0cf340 2023-10-19T13:02:40.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.855+0000 7f7662ffd640 1 -- 172.21.15.57:0/2008357317 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7654005280 con 0x7f766c0cf340 2023-10-19T13:02:40.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.856+0000 7f7670efc640 1 -- 172.21.15.57:0/2008357317 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f766c1f1360 con 0x7f766c0cf340 2023-10-19T13:02:40.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.857+0000 7f7670efc640 1 -- 172.21.15.57:0/2008357317 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f766c0690e0 con 0x7f766c0cf340 2023-10-19T13:02:40.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.857+0000 7f7662ffd640 1 -- 172.21.15.57:0/2008357317 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 54) v1 ==== 89445+0+0 (secure 0 0 0) 0x7f76540053e0 con 0x7f766c0cf340 2023-10-19T13:02:40.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.857+0000 7f7662ffd640 1 --2- 172.21.15.57:0/2008357317 >> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] conn(0x7f764c06b080 0x7f764c06d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:40.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.858+0000 7f766a59c640 1 -- 172.21.15.57:0/2008357317 >> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] conn(0x7f764c06b080 msgr2=0x7f764c06d540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/2366702417 2023-10-19T13:02:40.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.858+0000 7f7662ffd640 1 -- 172.21.15.57:0/2008357317 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 6517+0+0 (secure 0 0 0) 0x7f76540ac950 con 0x7f766c0cf340 2023-10-19T13:02:40.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.858+0000 7f766a59c640 1 --2- 172.21.15.57:0/2008357317 >> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] conn(0x7f764c06b080 0x7f764c06d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T13:02:40.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:40.862+0000 7f7662ffd640 1 -- 172.21.15.57:0/2008357317 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7654077390 con 0x7f766c0cf340 2023-10-19T13:02:41.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.039+0000 7f7670efc640 1 -- 172.21.15.57:0/2008357317 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f766c0d01d0 con 0x7f766c0cf340 2023-10-19T13:02:41.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.039+0000 7f7662ffd640 1 -- 172.21.15.57:0/2008357317 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v117) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7654042020 con 0x7f766c0cf340 2023-10-19T13:02:41.043 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-19T13:02:41.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.041+0000 7f7670efc640 1 -- 172.21.15.57:0/2008357317 >> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] conn(0x7f764c06b080 msgr2=0x7f764c06d540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T13:02:41.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.041+0000 7f7670efc640 1 --2- 172.21.15.57:0/2008357317 >> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] conn(0x7f764c06b080 0x7f764c06d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:41.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.041+0000 7f7670efc640 1 -- 172.21.15.57:0/2008357317 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f766c0cf340 msgr2=0x7f766c0ca040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:41.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.041+0000 7f7670efc640 1 --2- 172.21.15.57:0/2008357317 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f766c0cf340 0x7f766c0ca040 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7f7654009950 tx=0x7f76540038a0 comp rx=0 tx=0).stop 2023-10-19T13:02:41.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.042+0000 7f7670efc640 1 -- 172.21.15.57:0/2008357317 shutdown_connections 2023-10-19T13:02:41.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.042+0000 7f7670efc640 1 --2- 172.21.15.57:0/2008357317 >> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] conn(0x7f764c06b080 0x7f764c06d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:41.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.042+0000 7f7670efc640 1 --2- 172.21.15.57:0/2008357317 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f766c15b2a0 0x7f766c0ca580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:41.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.042+0000 7f7670efc640 1 --2- 172.21.15.57:0/2008357317 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f766c0cf340 0x7f766c0ca040 unknown :-1 s=CLOSED pgs=1336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:41.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.042+0000 7f7670efc640 1 --2- 172.21.15.57:0/2008357317 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f766c0ceb40 0x7f766c1f0e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:41.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.042+0000 7f7670efc640 1 -- 172.21.15.57:0/2008357317 >> 172.21.15.57:0/2008357317 conn(0x7f766c0d0d60 msgr2=0x7f766c06b590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:02:41.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.042+0000 7f7670efc640 1 -- 172.21.15.57:0/2008357317 shutdown_connections 2023-10-19T13:02:41.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.042+0000 7f7670efc640 1 -- 172.21.15.57:0/2008357317 wait complete. 2023-10-19T13:02:41.055 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-19T13:02:41.070 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:02:41.067+0000 7fc269232040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-19T13:02:41.113 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:02:41.110+0000 7fc269232040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-19T13:02:41.122 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:02:41.119+0000 7f380220d040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-19T13:02:41.154 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:02:41.152+0000 7fc269232040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-19T13:02:41.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.165+0000 7fb0a9f76640 1 -- 172.21.15.57:0/435425968 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb0a415b2a0 msgr2=0x7fb0a40d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:41.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.165+0000 7fb0a9f76640 1 --2- 172.21.15.57:0/435425968 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb0a415b2a0 0x7fb0a40d1910 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7fb0a415b680 tx=0x7fb09802bda0 comp rx=0 tx=0).stop 2023-10-19T13:02:41.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.166+0000 7fb0a9f76640 1 -- 172.21.15.57:0/435425968 shutdown_connections 2023-10-19T13:02:41.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.166+0000 7fb0a9f76640 1 --2- 172.21.15.57:0/435425968 >> v2:172.21.15.179:3300/0 conn(0x7fb0a40ceb40 0x7fb0a40b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:41.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.166+0000 7fb0a9f76640 1 --2- 172.21.15.57:0/435425968 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb0a415b2a0 0x7fb0a40d1910 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:41.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.166+0000 7fb0a9f76640 1 -- 172.21.15.57:0/435425968 >> 172.21.15.57:0/435425968 conn(0x7fb0a40d0d60 msgr2=0x7fb0a40d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:02:41.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.166+0000 7fb0a9f76640 1 -- 172.21.15.57:0/435425968 shutdown_connections 2023-10-19T13:02:41.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.166+0000 7fb0a9f76640 1 -- 172.21.15.57:0/435425968 wait complete. 2023-10-19T13:02:41.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.167+0000 7fb0a9f76640 1 Processor -- start 2023-10-19T13:02:41.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.167+0000 7fb0a9f76640 1 -- start start 2023-10-19T13:02:41.171 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:02:41.167+0000 7f380220d040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-19T13:02:41.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.168+0000 7fb0a9f76640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb0a40ceb40 0x7fb0a41f4560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:41.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.168+0000 7fb0a9f76640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb0a40cf340 0x7fb0a4145fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:41.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.168+0000 7fb0a9f76640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb0a415b2a0 0x7fb0a415dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:41.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.168+0000 7fb0a9f76640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a415d910 con 0x7fb0a40ceb40 2023-10-19T13:02:41.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.168+0000 7fb0a9f76640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a415cf70 con 0x7fb0a40cf340 2023-10-19T13:02:41.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.168+0000 7fb0a9f76640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a415d250 con 0x7fb0a415b2a0 2023-10-19T13:02:41.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.168+0000 7fb0a37fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb0a40ceb40 0x7fb0a41f4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:41.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.168+0000 7fb0a37fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb0a40ceb40 0x7fb0a41f4560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43206/0 (socket says 172.21.15.57:43206) 2023-10-19T13:02:41.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.168+0000 7fb0a37fe640 1 -- 172.21.15.57:0/615621307 learned_addr learned my addr 172.21.15.57:0/615621307 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:02:41.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.168+0000 7fb0a2ffd640 1 --2- 172.21.15.57:0/615621307 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb0a40cf340 0x7fb0a4145fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:41.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.168+0000 7fb0a3fff640 1 --2- 172.21.15.57:0/615621307 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb0a415b2a0 0x7fb0a415dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:41.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.168+0000 7fb0a37fe640 1 -- 172.21.15.57:0/615621307 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb0a415b2a0 msgr2=0x7fb0a415dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:41.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.168+0000 7fb0a37fe640 1 --2- 172.21.15.57:0/615621307 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb0a415b2a0 0x7fb0a415dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:41.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.168+0000 7fb0a37fe640 1 -- 172.21.15.57:0/615621307 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb0a40cf340 msgr2=0x7fb0a4145fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:41.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.168+0000 7fb0a37fe640 1 --2- 172.21.15.57:0/615621307 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb0a40cf340 0x7fb0a4145fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:41.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.168+0000 7fb0a37fe640 1 -- 172.21.15.57:0/615621307 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb098009560 con 0x7fb0a40ceb40 2023-10-19T13:02:41.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.168+0000 7fb0a37fe640 1 --2- 172.21.15.57:0/615621307 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb0a40ceb40 0x7fb0a41f4560 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7fb09400c450 tx=0x7fb09400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:02:41.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.168+0000 7fb08bfff640 1 -- 172.21.15.57:0/615621307 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0940052d0 con 0x7fb0a40ceb40 2023-10-19T13:02:41.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.168+0000 7fb08bfff640 1 -- 172.21.15.57:0/615621307 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fb094005430 con 0x7fb0a40ceb40 2023-10-19T13:02:41.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.168+0000 7fb08bfff640 1 -- 172.21.15.57:0/615621307 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb094012890 con 0x7fb0a40ceb40 2023-10-19T13:02:41.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.168+0000 7fb0a9f76640 1 -- 172.21.15.57:0/615621307 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0a415d530 con 0x7fb0a40ceb40 2023-10-19T13:02:41.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.169+0000 7fb0a9f76640 1 -- 172.21.15.57:0/615621307 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0a4146520 con 0x7fb0a40ceb40 2023-10-19T13:02:41.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.169+0000 7fb0a9f76640 1 -- 172.21.15.57:0/615621307 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0a40690e0 con 0x7fb0a40ceb40 2023-10-19T13:02:41.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.170+0000 7fb08bfff640 1 -- 172.21.15.57:0/615621307 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 54) v1 ==== 89445+0+0 (secure 0 0 0) 0x7fb0940035f0 con 0x7fb0a40ceb40 2023-10-19T13:02:41.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.170+0000 7fb08bfff640 1 --2- 172.21.15.57:0/615621307 >> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] conn(0x7fb09006b080 0x7fb09006d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:41.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.171+0000 7fb0a2ffd640 1 -- 172.21.15.57:0/615621307 >> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] conn(0x7fb09006b080 msgr2=0x7fb09006d540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/2366702417 2023-10-19T13:02:41.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.171+0000 7fb08bfff640 1 -- 172.21.15.57:0/615621307 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 6517+0+0 (secure 0 0 0) 0x7fb094089850 con 0x7fb0a40ceb40 2023-10-19T13:02:41.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.171+0000 7fb0a2ffd640 1 --2- 172.21.15.57:0/615621307 >> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] conn(0x7fb09006b080 0x7fb09006d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T13:02:41.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.175+0000 7fb08bfff640 1 -- 172.21.15.57:0/615621307 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb0940543b0 con 0x7fb0a40ceb40 2023-10-19T13:02:41.206 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:02:41.203+0000 7fc269232040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-19T13:02:41.216 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:02:41.214+0000 7f380220d040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-19T13:02:41.247 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:02:41.244+0000 7fc269232040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-19T13:02:41.272 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:02:41.269+0000 7f380220d040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-19T13:02:41.316 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:02:41.313+0000 7f380220d040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-19T13:02:41.345 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:02:41.342+0000 7fc269232040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-19T13:02:41.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.354+0000 7fb0a9f76640 1 -- 172.21.15.57:0/615621307 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb0a40b0d00 con 0x7fb0a40ceb40 2023-10-19T13:02:41.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.354+0000 7fb08bfff640 1 -- 172.21.15.57:0/615621307 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v117) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb0940579d0 con 0x7fb0a40ceb40 2023-10-19T13:02:41.357 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2023-10-19T13:02:41.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.356+0000 7fb0a9f76640 1 -- 172.21.15.57:0/615621307 >> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] conn(0x7fb09006b080 msgr2=0x7fb09006d540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T13:02:41.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.356+0000 7fb0a9f76640 1 --2- 172.21.15.57:0/615621307 >> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] conn(0x7fb09006b080 0x7fb09006d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:41.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.356+0000 7fb0a9f76640 1 -- 172.21.15.57:0/615621307 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb0a40ceb40 msgr2=0x7fb0a41f4560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:41.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.356+0000 7fb0a9f76640 1 --2- 172.21.15.57:0/615621307 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb0a40ceb40 0x7fb0a41f4560 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7fb09400c450 tx=0x7fb09400c910 comp rx=0 tx=0).stop 2023-10-19T13:02:41.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.356+0000 7fb0a9f76640 1 -- 172.21.15.57:0/615621307 shutdown_connections 2023-10-19T13:02:41.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.356+0000 7fb0a9f76640 1 --2- 172.21.15.57:0/615621307 >> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] conn(0x7fb09006b080 0x7fb09006d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:41.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.356+0000 7fb0a9f76640 1 --2- 172.21.15.57:0/615621307 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb0a415b2a0 0x7fb0a415dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:41.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.356+0000 7fb0a9f76640 1 --2- 172.21.15.57:0/615621307 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb0a40cf340 0x7fb0a4145fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:41.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.356+0000 7fb0a9f76640 1 --2- 172.21.15.57:0/615621307 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb0a40ceb40 0x7fb0a41f4560 unknown :-1 s=CLOSED pgs=1339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:41.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.356+0000 7fb0a9f76640 1 -- 172.21.15.57:0/615621307 >> 172.21.15.57:0/615621307 conn(0x7fb0a40d0d60 msgr2=0x7fb0a406b5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:02:41.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.357+0000 7fb0a9f76640 1 -- 172.21.15.57:0/615621307 shutdown_connections 2023-10-19T13:02:41.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.357+0000 7fb0a9f76640 1 -- 172.21.15.57:0/615621307 wait complete. 2023-10-19T13:02:41.370 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-19T13:02:41.397 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:02:41.395+0000 7fc269232040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-19T13:02:41.422 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:02:41.419+0000 7f380220d040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-19T13:02:41.438 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:02:41.436+0000 7fc269232040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-19T13:02:41.480 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:02:41.477+0000 7f380220d040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-19T13:02:41.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.481+0000 7fd5e58b4640 1 -- 172.21.15.57:0/2989934849 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd5e015b260 msgr2=0x7fd5e00cf310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:41.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.481+0000 7fd5e58b4640 1 --2- 172.21.15.57:0/2989934849 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd5e015b260 0x7fd5e00cf310 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fd5d0005fd0 tx=0x7fd5d002be10 comp rx=0 tx=0).stop 2023-10-19T13:02:41.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.481+0000 7fd5e58b4640 1 -- 172.21.15.57:0/2989934849 shutdown_connections 2023-10-19T13:02:41.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.481+0000 7fd5e58b4640 1 --2- 172.21.15.57:0/2989934849 >> v2:172.21.15.179:3300/0 conn(0x7fd5e015b640 0x7fd5e00cf850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:41.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.481+0000 7fd5e58b4640 1 --2- 172.21.15.57:0/2989934849 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd5e015b260 0x7fd5e00cf310 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:41.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.481+0000 7fd5e58b4640 1 -- 172.21.15.57:0/2989934849 >> 172.21.15.57:0/2989934849 conn(0x7fd5e009a030 msgr2=0x7fd5e00d0d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:02:41.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.481+0000 7fd5e58b4640 1 -- 172.21.15.57:0/2989934849 shutdown_connections 2023-10-19T13:02:41.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.481+0000 7fd5e58b4640 1 -- 172.21.15.57:0/2989934849 wait complete. 2023-10-19T13:02:41.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.482+0000 7fd5e58b4640 1 Processor -- start 2023-10-19T13:02:41.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.482+0000 7fd5e58b4640 1 -- start start 2023-10-19T13:02:41.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.483+0000 7fd5e58b4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5e00b0b10 0x7fd5e01580e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:41.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.483+0000 7fd5e58b4640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd5e015b260 0x7fd5e015c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:41.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.483+0000 7fd5e58b4640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd5e015b640 0x7fd5e0158d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:41.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.483+0000 7fd5e58b4640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5e00c4850 con 0x7fd5e00b0b10 2023-10-19T13:02:41.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.483+0000 7fd5e58b4640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5e015c970 con 0x7fd5e015b260 2023-10-19T13:02:41.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.483+0000 7fd5e58b4640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5e015d850 con 0x7fd5e015b640 2023-10-19T13:02:41.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.483+0000 7fd5deffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5e00b0b10 0x7fd5e01580e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:41.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.483+0000 7fd5deffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5e00b0b10 0x7fd5e01580e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43216/0 (socket says 172.21.15.57:43216) 2023-10-19T13:02:41.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.483+0000 7fd5deffd640 1 -- 172.21.15.57:0/1404426519 learned_addr learned my addr 172.21.15.57:0/1404426519 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:02:41.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.483+0000 7fd5de7fc640 1 --2- 172.21.15.57:0/1404426519 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd5e015b260 0x7fd5e015c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:41.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.483+0000 7fd5df7fe640 1 --2- 172.21.15.57:0/1404426519 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd5e015b640 0x7fd5e0158d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:41.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.483+0000 7fd5deffd640 1 -- 172.21.15.57:0/1404426519 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd5e015b640 msgr2=0x7fd5e0158d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:41.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.483+0000 7fd5deffd640 1 --2- 172.21.15.57:0/1404426519 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd5e015b640 0x7fd5e0158d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:41.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.483+0000 7fd5deffd640 1 -- 172.21.15.57:0/1404426519 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd5e015b260 msgr2=0x7fd5e015c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:41.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.483+0000 7fd5deffd640 1 --2- 172.21.15.57:0/1404426519 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd5e015b260 0x7fd5e015c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:41.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.483+0000 7fd5deffd640 1 -- 172.21.15.57:0/1404426519 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5d4009490 con 0x7fd5e00b0b10 2023-10-19T13:02:41.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.484+0000 7fd5deffd640 1 --2- 172.21.15.57:0/1404426519 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5e00b0b10 0x7fd5e01580e0 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7fd5d0009850 tx=0x7fd5d002da50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:02:41.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.484+0000 7fd5c7fff640 1 -- 172.21.15.57:0/1404426519 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5d002de50 con 0x7fd5e00b0b10 2023-10-19T13:02:41.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.484+0000 7fd5c7fff640 1 -- 172.21.15.57:0/1404426519 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fd5d00036e0 con 0x7fd5e00b0b10 2023-10-19T13:02:41.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.484+0000 7fd5c7fff640 1 -- 172.21.15.57:0/1404426519 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5d0003a90 con 0x7fd5e00b0b10 2023-10-19T13:02:41.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.484+0000 7fd5e58b4640 1 -- 172.21.15.57:0/1404426519 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5d0009560 con 0x7fd5e00b0b10 2023-10-19T13:02:41.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.484+0000 7fd5e58b4640 1 -- 172.21.15.57:0/1404426519 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5e00c4d10 con 0x7fd5e00b0b10 2023-10-19T13:02:41.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.485+0000 7fd5e58b4640 1 -- 172.21.15.57:0/1404426519 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5e00675f0 con 0x7fd5e00b0b10 2023-10-19T13:02:41.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.485+0000 7fd5c7fff640 1 -- 172.21.15.57:0/1404426519 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 54) v1 ==== 89445+0+0 (secure 0 0 0) 0x7fd5d0003bf0 con 0x7fd5e00b0b10 2023-10-19T13:02:41.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.486+0000 7fd5c7fff640 1 --2- 172.21.15.57:0/1404426519 >> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] conn(0x7fd5cc06b080 0x7fd5cc06d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:41.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.486+0000 7fd5c7fff640 1 -- 172.21.15.57:0/1404426519 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 6517+0+0 (secure 0 0 0) 0x7fd5d00ac740 con 0x7fd5e00b0b10 2023-10-19T13:02:41.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.486+0000 7fd5de7fc640 1 -- 172.21.15.57:0/1404426519 >> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] conn(0x7fd5cc06b080 msgr2=0x7fd5cc06d540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/2366702417 2023-10-19T13:02:41.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.486+0000 7fd5de7fc640 1 --2- 172.21.15.57:0/1404426519 >> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] conn(0x7fd5cc06b080 0x7fd5cc06d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T13:02:41.499 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:02:41.484+0000 7fc269232040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-19T13:02:41.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.490+0000 7fd5c7fff640 1 -- 172.21.15.57:0/1404426519 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd5d00772b0 con 0x7fd5e00b0b10 2023-10-19T13:02:41.526 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:02:41.523+0000 7f380220d040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-19T13:02:41.529 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:02:41.527+0000 7fc269232040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-19T13:02:41.577 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:02:41.574+0000 7f380220d040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-19T13:02:41.623 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:02:41.620+0000 7f380220d040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-19T13:02:41.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.667+0000 7fd5e58b4640 1 -- 172.21.15.57:0/1404426519 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd5e015e870 con 0x7fd5e00b0b10 2023-10-19T13:02:41.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.668+0000 7fd5c7fff640 1 -- 172.21.15.57:0/1404426519 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v117) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd5d007a8d0 con 0x7fd5e00b0b10 2023-10-19T13:02:41.671 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 64 2023-10-19T13:02:41.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.670+0000 7fd5e58b4640 1 -- 172.21.15.57:0/1404426519 >> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] conn(0x7fd5cc06b080 msgr2=0x7fd5cc06d540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T13:02:41.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.670+0000 7fd5e58b4640 1 --2- 172.21.15.57:0/1404426519 >> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] conn(0x7fd5cc06b080 0x7fd5cc06d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:41.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.670+0000 7fd5e58b4640 1 -- 172.21.15.57:0/1404426519 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5e00b0b10 msgr2=0x7fd5e01580e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:41.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.670+0000 7fd5e58b4640 1 --2- 172.21.15.57:0/1404426519 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5e00b0b10 0x7fd5e01580e0 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7fd5d0009850 tx=0x7fd5d002da50 comp rx=0 tx=0).stop 2023-10-19T13:02:41.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.670+0000 7fd5e58b4640 1 -- 172.21.15.57:0/1404426519 shutdown_connections 2023-10-19T13:02:41.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.670+0000 7fd5e58b4640 1 --2- 172.21.15.57:0/1404426519 >> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] conn(0x7fd5cc06b080 0x7fd5cc06d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:41.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.670+0000 7fd5e58b4640 1 --2- 172.21.15.57:0/1404426519 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd5e015b640 0x7fd5e0158d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:41.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.670+0000 7fd5e58b4640 1 --2- 172.21.15.57:0/1404426519 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd5e015b260 0x7fd5e015c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:41.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.670+0000 7fd5e58b4640 1 --2- 172.21.15.57:0/1404426519 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5e00b0b10 0x7fd5e01580e0 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:41.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.670+0000 7fd5e58b4640 1 -- 172.21.15.57:0/1404426519 >> 172.21.15.57:0/1404426519 conn(0x7fd5e009a030 msgr2=0x7fd5e00af0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:02:41.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.670+0000 7fd5e58b4640 1 -- 172.21.15.57:0/1404426519 shutdown_connections 2023-10-19T13:02:41.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.670+0000 7fd5e58b4640 1 -- 172.21.15.57:0/1404426519 wait complete. 2023-10-19T13:02:41.684 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-19T13:02:41.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.793+0000 7fa1a6087640 1 -- 172.21.15.57:0/3717148729 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa1a00cf340 msgr2=0x7fa1a00d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:41.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.793+0000 7fa1a6087640 1 --2- 172.21.15.57:0/3717148729 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa1a00cf340 0x7fa1a00d1910 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7fa1a006c850 tx=0x7fa19002bd10 comp rx=0 tx=0).stop 2023-10-19T13:02:41.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.794+0000 7fa1a6087640 1 -- 172.21.15.57:0/3717148729 shutdown_connections 2023-10-19T13:02:41.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.794+0000 7fa1a6087640 1 --2- 172.21.15.57:0/3717148729 >> v2:172.21.15.179:3300/0 conn(0x7fa1a00ceb40 0x7fa1a00b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:41.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.794+0000 7fa1a6087640 1 --2- 172.21.15.57:0/3717148729 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa1a015b2a0 0x7fa1a00cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:41.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.794+0000 7fa1a6087640 1 --2- 172.21.15.57:0/3717148729 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa1a00cf340 0x7fa1a00d1910 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:41.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.794+0000 7fa1a6087640 1 -- 172.21.15.57:0/3717148729 >> 172.21.15.57:0/3717148729 conn(0x7fa1a00d0d60 msgr2=0x7fa1a00d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:02:41.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.794+0000 7fa1a6087640 1 -- 172.21.15.57:0/3717148729 shutdown_connections 2023-10-19T13:02:41.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.794+0000 7fa1a6087640 1 -- 172.21.15.57:0/3717148729 wait complete. 2023-10-19T13:02:41.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.795+0000 7fa1a6087640 1 Processor -- start 2023-10-19T13:02:41.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.795+0000 7fa1a6087640 1 -- start start 2023-10-19T13:02:41.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.796+0000 7fa1a6087640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa1a00ceb40 0x7fa1a015df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:41.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.796+0000 7fa1a6087640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa1a00cf340 0x7fa1a0158370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:41.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.796+0000 7fa1a6087640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa1a015b2a0 0x7fa1a01e4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:41.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.796+0000 7fa1a6087640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1a01e3850 con 0x7fa1a00cf340 2023-10-19T13:02:41.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.796+0000 7fa1a6087640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1a01e46b0 con 0x7fa1a00ceb40 2023-10-19T13:02:41.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.796+0000 7fa1a6087640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1a01e3aa0 con 0x7fa1a015b2a0 2023-10-19T13:02:41.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.796+0000 7fa19effd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa1a00cf340 0x7fa1a0158370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:41.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.796+0000 7fa19effd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa1a00cf340 0x7fa1a0158370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43242/0 (socket says 172.21.15.57:43242) 2023-10-19T13:02:41.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.796+0000 7fa19effd640 1 -- 172.21.15.57:0/1623685084 learned_addr learned my addr 172.21.15.57:0/1623685084 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:02:41.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.796+0000 7fa19f7fe640 1 --2- 172.21.15.57:0/1623685084 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa1a00ceb40 0x7fa1a015df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:41.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.796+0000 7fa19ffff640 1 --2- 172.21.15.57:0/1623685084 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa1a015b2a0 0x7fa1a01e4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:41.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.796+0000 7fa19effd640 1 -- 172.21.15.57:0/1623685084 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa1a015b2a0 msgr2=0x7fa1a01e4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:41.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.796+0000 7fa19effd640 1 --2- 172.21.15.57:0/1623685084 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa1a015b2a0 0x7fa1a01e4e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:41.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.796+0000 7fa19effd640 1 -- 172.21.15.57:0/1623685084 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa1a00ceb40 msgr2=0x7fa1a015df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:41.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.796+0000 7fa19effd640 1 --2- 172.21.15.57:0/1623685084 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa1a00ceb40 0x7fa1a015df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:41.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.796+0000 7fa19effd640 1 -- 172.21.15.57:0/1623685084 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa190009560 con 0x7fa1a00cf340 2023-10-19T13:02:41.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.796+0000 7fa19effd640 1 --2- 172.21.15.57:0/1623685084 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa1a00cf340 0x7fa1a0158370 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7fa19400d770 tx=0x7fa19400db70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:02:41.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.797+0000 7fa187fff640 1 -- 172.21.15.57:0/1623685084 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa194009a20 con 0x7fa1a00cf340 2023-10-19T13:02:41.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.797+0000 7fa187fff640 1 -- 172.21.15.57:0/1623685084 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fa194009c70 con 0x7fa1a00cf340 2023-10-19T13:02:41.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.797+0000 7fa1a6087640 1 -- 172.21.15.57:0/1623685084 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1a01e3d80 con 0x7fa1a00cf340 2023-10-19T13:02:41.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.797+0000 7fa187fff640 1 -- 172.21.15.57:0/1623685084 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1940106c0 con 0x7fa1a00cf340 2023-10-19T13:02:41.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.797+0000 7fa1a6087640 1 -- 172.21.15.57:0/1623685084 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1a01e4340 con 0x7fa1a00cf340 2023-10-19T13:02:41.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.798+0000 7fa1a6087640 1 -- 172.21.15.57:0/1623685084 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1a00690e0 con 0x7fa1a00cf340 2023-10-19T13:02:41.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.798+0000 7fa187fff640 1 -- 172.21.15.57:0/1623685084 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 54) v1 ==== 89445+0+0 (secure 0 0 0) 0x7fa194010820 con 0x7fa1a00cf340 2023-10-19T13:02:41.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.799+0000 7fa187fff640 1 --2- 172.21.15.57:0/1623685084 >> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] conn(0x7fa18c06b0b0 0x7fa18c06d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:41.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.799+0000 7fa19f7fe640 1 -- 172.21.15.57:0/1623685084 >> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] conn(0x7fa18c06b0b0 msgr2=0x7fa18c06d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/2366702417 2023-10-19T13:02:41.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.799+0000 7fa187fff640 1 -- 172.21.15.57:0/1623685084 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 6517+0+0 (secure 0 0 0) 0x7fa19408a170 con 0x7fa1a00cf340 2023-10-19T13:02:41.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.799+0000 7fa19f7fe640 1 --2- 172.21.15.57:0/1623685084 >> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] conn(0x7fa18c06b0b0 0x7fa18c06d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T13:02:41.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.803+0000 7fa187fff640 1 -- 172.21.15.57:0/1623685084 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa194054bc0 con 0x7fa1a00cf340 2023-10-19T13:02:41.845 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:02:41.842+0000 7fc269232040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-19T13:02:41.969 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:02:41.967+0000 7f380220d040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-19T13:02:41.973 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:02:41.971+0000 7fc269232040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-19T13:02:41.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.993+0000 7fa1a6087640 1 -- 172.21.15.57:0/1623685084 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fa1a006ba10 con 0x7fa1a00cf340 2023-10-19T13:02:41.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.993+0000 7fa187fff640 1 -- 172.21.15.57:0/1623685084 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v117) v1 ==== 80+0+477 (secure 0 0 0) 0x7fa1940171f0 con 0x7fa1a00cf340 2023-10-19T13:02:41.997 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/3304791412 2023-10-19T14:01:39.552724+0000 2023-10-19T13:02:41.997 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/3109348805 2023-10-20T13:01:37.504869+0000 2023-10-19T13:02:41.998 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/2984488835 2023-10-20T13:01:37.504869+0000 2023-10-19T13:02:41.998 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/205205608 2023-10-20T13:01:37.504869+0000 2023-10-19T13:02:41.998 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/4112649564 2023-10-20T13:01:37.504869+0000 2023-10-19T13:02:41.999 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:6833/3281457486 2023-10-20T13:01:37.504869+0000 2023-10-19T13:02:41.999 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:6832/3281457486 2023-10-20T13:01:37.504869+0000 2023-10-19T13:02:42.000 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/2509136709 2023-10-20T13:01:37.504869+0000 2023-10-19T13:02:42.001 INFO:teuthology.orchestra.run.smithi057.stderr:listed 8 entries 2023-10-19T13:02:42.001 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.995+0000 7fa1a6087640 1 -- 172.21.15.57:0/1623685084 >> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] conn(0x7fa18c06b0b0 msgr2=0x7fa18c06d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T13:02:42.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.995+0000 7fa1a6087640 1 --2- 172.21.15.57:0/1623685084 >> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] conn(0x7fa18c06b0b0 0x7fa18c06d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:42.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.995+0000 7fa1a6087640 1 -- 172.21.15.57:0/1623685084 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa1a00cf340 msgr2=0x7fa1a0158370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:42.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.995+0000 7fa1a6087640 1 --2- 172.21.15.57:0/1623685084 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa1a00cf340 0x7fa1a0158370 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7fa19400d770 tx=0x7fa19400db70 comp rx=0 tx=0).stop 2023-10-19T13:02:42.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.995+0000 7fa1a6087640 1 -- 172.21.15.57:0/1623685084 shutdown_connections 2023-10-19T13:02:42.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.995+0000 7fa1a6087640 1 --2- 172.21.15.57:0/1623685084 >> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] conn(0x7fa18c06b0b0 0x7fa18c06d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:42.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.995+0000 7fa1a6087640 1 --2- 172.21.15.57:0/1623685084 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa1a015b2a0 0x7fa1a01e4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:42.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.995+0000 7fa1a6087640 1 --2- 172.21.15.57:0/1623685084 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa1a00cf340 0x7fa1a0158370 unknown :-1 s=CLOSED pgs=1342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:42.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.995+0000 7fa1a6087640 1 --2- 172.21.15.57:0/1623685084 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa1a00ceb40 0x7fa1a015df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:42.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.995+0000 7fa1a6087640 1 -- 172.21.15.57:0/1623685084 >> 172.21.15.57:0/1623685084 conn(0x7fa1a00d0d60 msgr2=0x7fa1a015a2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:02:42.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.996+0000 7fa1a6087640 1 -- 172.21.15.57:0/1623685084 shutdown_connections 2023-10-19T13:02:42.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:41.996+0000 7fa1a6087640 1 -- 172.21.15.57:0/1623685084 wait complete. 2023-10-19T13:02:42.009 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi057.front.sepia.ceph.com... 2023-10-19T13:02:42.009 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:02:42.010 DEBUG:teuthology.orchestra.run.smithi057:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-10-19T13:02:42.039 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi057.stderr:ceph-fuse[125038]: fuse finished with error 0 and tester_r 0 2023-10-19T13:02:42.041 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi057.front.sepia.ceph.com 2023-10-19T13:02:42.042 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:02:42.042 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-19T13:02:42.064 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:02:42.061+0000 7fc269232040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-19T13:02:42.106 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:02:42.103+0000 7fc269232040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-19T13:02:42.113 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:02:42.110+0000 7f380220d040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-19T13:02:42.122 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-19T13:02:42.122 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T13:02:42.137 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054833,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=284 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3247632k,nr_inodes=811908,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-19T13:02:42.138 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi179.front.sepia.ceph.com 2023-10-19T13:02:42.138 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:02:42.138 DEBUG:teuthology.orchestra.run.smithi179:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-10-19T13:02:42.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-19T13:02:42.192 INFO:teuthology.orchestra.run.smithi179.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2023-10-19T13:02:42.193 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T13:02:42.193 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T13:02:42.205 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:02:42.202+0000 7fc269232040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-19T13:02:42.208 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055614,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=358 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248260k,nr_inodes=812065,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-19T13:02:42.209 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi057.front.sepia.ceph.com 2023-10-19T13:02:42.209 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:02:42.210 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-10-19T13:02:42.211 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:02:42.208+0000 7f380220d040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-19T13:02:42.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-19T13:02:42.226 INFO:teuthology.orchestra.run.smithi057.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2023-10-19T13:02:42.227 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-19T13:02:42.227 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T13:02:42.242 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054833,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=284 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3247632k,nr_inodes=811908,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-19T13:02:42.243 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi179.front.sepia.ceph.com 2023-10-19T13:02:42.243 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:02:42.243 DEBUG:teuthology.orchestra.run.smithi179:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-10-19T13:02:42.256 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:02:42.254+0000 7f380220d040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-19T13:02:42.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-19T13:02:42.297 INFO:teuthology.orchestra.run.smithi179.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2023-10-19T13:02:42.298 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T13:02:42.301 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:02:42.299+0000 7fc269232040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-19T13:02:42.359 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:02:42.356+0000 7f380220d040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-19T13:02:42.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.405+0000 7fa2ecd25640 1 -- 172.21.15.57:0/1508276278 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa2e80cf340 msgr2=0x7fa2e80d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:42.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.405+0000 7fa2ecd25640 1 --2- 172.21.15.57:0/1508276278 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa2e80cf340 0x7fa2e80d1910 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7fa2e815b680 tx=0x7fa2d002bdb0 comp rx=0 tx=0).stop 2023-10-19T13:02:42.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.405+0000 7fa2ecd25640 1 -- 172.21.15.57:0/1508276278 shutdown_connections 2023-10-19T13:02:42.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.405+0000 7fa2ecd25640 1 --2- 172.21.15.57:0/1508276278 >> v2:172.21.15.179:3300/0 conn(0x7fa2e80ceb40 0x7fa2e80b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:42.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.405+0000 7fa2ecd25640 1 --2- 172.21.15.57:0/1508276278 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa2e80cf340 0x7fa2e80d1910 unknown :-1 s=CLOSED pgs=1343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:42.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.405+0000 7fa2ecd25640 1 -- 172.21.15.57:0/1508276278 >> 172.21.15.57:0/1508276278 conn(0x7fa2e80d0d60 msgr2=0x7fa2e80d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:02:42.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.405+0000 7fa2ecd25640 1 -- 172.21.15.57:0/1508276278 shutdown_connections 2023-10-19T13:02:42.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.406+0000 7fa2ecd25640 1 -- 172.21.15.57:0/1508276278 wait complete. 2023-10-19T13:02:42.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.407+0000 7fa2ecd25640 1 Processor -- start 2023-10-19T13:02:42.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.407+0000 7fa2ecd25640 1 -- start start 2023-10-19T13:02:42.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.408+0000 7fa2ecd25640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa2e80ceb40 0x7fa2e8156e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:42.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.408+0000 7fa2ecd25640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa2e80cf340 0x7fa2e81581b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:42.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.408+0000 7fa2ecd25640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa2e815b2a0 0x7fa2e80b5cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:42.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.408+0000 7fa2ecd25640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e815dd70 con 0x7fa2e815b2a0 2023-10-19T13:02:42.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.408+0000 7fa2ecd25640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e8157d90 con 0x7fa2e80cf340 2023-10-19T13:02:42.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.408+0000 7fa2ecd25640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e815da90 con 0x7fa2e80ceb40 2023-10-19T13:02:42.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.408+0000 7fa2e6d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa2e815b2a0 0x7fa2e80b5cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:42.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.408+0000 7fa2e6d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa2e815b2a0 0x7fa2e80b5cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43258/0 (socket says 172.21.15.57:43258) 2023-10-19T13:02:42.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.408+0000 7fa2e6d9d640 1 -- 172.21.15.57:0/947603541 learned_addr learned my addr 172.21.15.57:0/947603541 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:02:42.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.408+0000 7fa2e659c640 1 --2- 172.21.15.57:0/947603541 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa2e80ceb40 0x7fa2e8156e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:42.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.408+0000 7fa2e5d9b640 1 --2- 172.21.15.57:0/947603541 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa2e80cf340 0x7fa2e81581b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:42.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.408+0000 7fa2e6d9d640 1 -- 172.21.15.57:0/947603541 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa2e80ceb40 msgr2=0x7fa2e8156e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:42.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.408+0000 7fa2e6d9d640 1 --2- 172.21.15.57:0/947603541 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa2e80ceb40 0x7fa2e8156e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:42.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.408+0000 7fa2e6d9d640 1 -- 172.21.15.57:0/947603541 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa2e80cf340 msgr2=0x7fa2e81581b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:42.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.408+0000 7fa2e6d9d640 1 --2- 172.21.15.57:0/947603541 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa2e80cf340 0x7fa2e81581b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:42.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.408+0000 7fa2e6d9d640 1 -- 172.21.15.57:0/947603541 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2d0009560 con 0x7fa2e815b2a0 2023-10-19T13:02:42.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.408+0000 7fa2e6d9d640 1 --2- 172.21.15.57:0/947603541 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa2e815b2a0 0x7fa2e80b5cd0 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7fa2e000eb10 tx=0x7fa2e000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:02:42.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.408+0000 7fa2deffd640 1 -- 172.21.15.57:0/947603541 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2e0004b20 con 0x7fa2e815b2a0 2023-10-19T13:02:42.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.409+0000 7fa2deffd640 1 -- 172.21.15.57:0/947603541 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fa2e0004ca0 con 0x7fa2e815b2a0 2023-10-19T13:02:42.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.409+0000 7fa2ecd25640 1 -- 172.21.15.57:0/947603541 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2e81576d0 con 0x7fa2e815b2a0 2023-10-19T13:02:42.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.409+0000 7fa2deffd640 1 -- 172.21.15.57:0/947603541 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2e0005280 con 0x7fa2e815b2a0 2023-10-19T13:02:42.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.409+0000 7fa2ecd25640 1 -- 172.21.15.57:0/947603541 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2e8157350 con 0x7fa2e815b2a0 2023-10-19T13:02:42.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.410+0000 7fa2ecd25640 1 -- 172.21.15.57:0/947603541 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2e8157ed0 con 0x7fa2e815b2a0 2023-10-19T13:02:42.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.410+0000 7fa2deffd640 1 -- 172.21.15.57:0/947603541 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 54) v1 ==== 89445+0+0 (secure 0 0 0) 0x7fa2e00053e0 con 0x7fa2e815b2a0 2023-10-19T13:02:42.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.410+0000 7fa2deffd640 1 --2- 172.21.15.57:0/947603541 >> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] conn(0x7fa2c806b080 0x7fa2c806d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:42.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.411+0000 7fa2e659c640 1 -- 172.21.15.57:0/947603541 >> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] conn(0x7fa2c806b080 msgr2=0x7fa2c806d540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/2366702417 2023-10-19T13:02:42.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.411+0000 7fa2deffd640 1 -- 172.21.15.57:0/947603541 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 6517+0+0 (secure 0 0 0) 0x7fa2e0014070 con 0x7fa2e815b2a0 2023-10-19T13:02:42.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.411+0000 7fa2e659c640 1 --2- 172.21.15.57:0/947603541 >> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] conn(0x7fa2c806b080 0x7fa2c806d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T13:02:42.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.415+0000 7fa2deffd640 1 -- 172.21.15.57:0/947603541 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa2e8157ed0 con 0x7fa2e815b2a0 2023-10-19T13:02:42.439 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:02:42.436+0000 7fc269232040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-19T13:02:42.457 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:02:42.455+0000 7f380220d040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-19T13:02:42.481 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:02:42.478+0000 7fc269232040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-19T13:02:42.585 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:02:42.582+0000 7fc269232040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-19T13:02:42.603 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:02:42.600+0000 7f380220d040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-19T13:02:42.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.612+0000 7fa2e659c640 1 -- 172.21.15.57:0/947603541 >> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] conn(0x7fa2c806b080 msgr2=0x7fa2c806d540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/2366702417 2023-10-19T13:02:42.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.612+0000 7fa2e659c640 1 --2- 172.21.15.57:0/947603541 >> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] conn(0x7fa2c806b080 0x7fa2c806d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-19T13:02:42.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.632+0000 7fa2ecd25640 1 -- 172.21.15.57:0/947603541 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa2e8157ed0 con 0x7fa2e815b2a0 2023-10-19T13:02:42.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.632+0000 7fa2deffd640 1 -- 172.21.15.57:0/947603541 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+4921 (secure 0 0 0) 0x7fa2e8157ed0 con 0x7fa2e815b2a0 2023-10-19T13:02:42.635 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:02:42.635 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":44,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6611,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/2708395824","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2708395824},{"type":"v1","addr":"172.21.15.179:6836","nonce":2708395824}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":41},{"gid":7270,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/3803632191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":3803632191},{"type":"v1","addr":"172.21.15.179:6837","nonce":3803632191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":41},{"gid":7282,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6833/2259979700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2259979700},{"type":"v1","addr":"172.21.15.57:6833","nonce":2259979700}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T13:01:26.118951+0000","modified":"2023-10-19T13:01:28.134083+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8046},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8046":{"gid":8046,"name":"c","rank":0,"incarnation":42,"state":"up:active","state_seq":3,"addr":"172.21.15.57:6835/2556427177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":2556427177},{"type":"v1","addr":"172.21.15.57:6835","nonce":2556427177}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-10-19T13:02:42.636 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 44 2023-10-19T13:02:42.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.634+0000 7fa2ecd25640 1 -- 172.21.15.57:0/947603541 >> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] conn(0x7fa2c806b080 msgr2=0x7fa2c806d540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T13:02:42.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.634+0000 7fa2ecd25640 1 --2- 172.21.15.57:0/947603541 >> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] conn(0x7fa2c806b080 0x7fa2c806d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:42.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.634+0000 7fa2ecd25640 1 -- 172.21.15.57:0/947603541 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa2e815b2a0 msgr2=0x7fa2e80b5cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:42.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.634+0000 7fa2ecd25640 1 --2- 172.21.15.57:0/947603541 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa2e815b2a0 0x7fa2e80b5cd0 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7fa2e000eb10 tx=0x7fa2e000c2d0 comp rx=0 tx=0).stop 2023-10-19T13:02:42.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.634+0000 7fa2ecd25640 1 -- 172.21.15.57:0/947603541 shutdown_connections 2023-10-19T13:02:42.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.634+0000 7fa2ecd25640 1 --2- 172.21.15.57:0/947603541 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa2e815b2a0 0x7fa2e80b5cd0 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:42.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.634+0000 7fa2ecd25640 1 --2- 172.21.15.57:0/947603541 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa2e80cf340 0x7fa2e81581b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:42.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.634+0000 7fa2ecd25640 1 --2- 172.21.15.57:0/947603541 >> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] conn(0x7fa2c806b080 0x7fa2c806d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:42.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.634+0000 7fa2ecd25640 1 --2- 172.21.15.57:0/947603541 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa2e80ceb40 0x7fa2e8156e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:42.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.634+0000 7fa2ecd25640 1 -- 172.21.15.57:0/947603541 >> 172.21.15.57:0/947603541 conn(0x7fa2e80d0d60 msgr2=0x7fa2e806b150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:02:42.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.634+0000 7fa2ecd25640 1 -- 172.21.15.57:0/947603541 shutdown_connections 2023-10-19T13:02:42.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.634+0000 7fa2ecd25640 1 -- 172.21.15.57:0/947603541 wait complete. 2023-10-19T13:02:42.648 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T13:02:42.649 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:02:42.646+0000 7f380220d040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-19T13:02:42.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.755+0000 7fdb700b4640 1 -- 172.21.15.57:0/2857357172 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdb680cf340 msgr2=0x7fdb680d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:42.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.755+0000 7fdb700b4640 1 --2- 172.21.15.57:0/2857357172 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdb680cf340 0x7fdb680d1910 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fdb64009830 tx=0x7fdb6402bc50 comp rx=0 tx=0).stop 2023-10-19T13:02:42.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.755+0000 7fdb700b4640 1 -- 172.21.15.57:0/2857357172 shutdown_connections 2023-10-19T13:02:42.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.755+0000 7fdb700b4640 1 --2- 172.21.15.57:0/2857357172 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdb680cf340 0x7fdb680d1910 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:42.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.755+0000 7fdb700b4640 1 -- 172.21.15.57:0/2857357172 >> 172.21.15.57:0/2857357172 conn(0x7fdb680d0d60 msgr2=0x7fdb680d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:02:42.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.755+0000 7fdb700b4640 1 -- 172.21.15.57:0/2857357172 shutdown_connections 2023-10-19T13:02:42.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.755+0000 7fdb700b4640 1 -- 172.21.15.57:0/2857357172 wait complete. 2023-10-19T13:02:42.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.756+0000 7fdb700b4640 1 Processor -- start 2023-10-19T13:02:42.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.756+0000 7fdb700b4640 1 -- start start 2023-10-19T13:02:42.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.757+0000 7fdb700b4640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdb680ceb40 0x7fdb6815c050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:42.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.757+0000 7fdb700b4640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdb680cf340 0x7fdb680b1110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:42.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.757+0000 7fdb700b4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdb6815b2a0 0x7fdb681574d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:42.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.757+0000 7fdb700b4640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb680c9730 con 0x7fdb6815b2a0 2023-10-19T13:02:42.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.757+0000 7fdb700b4640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb681f1660 con 0x7fdb680ceb40 2023-10-19T13:02:42.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.757+0000 7fdb700b4640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb6815c950 con 0x7fdb680cf340 2023-10-19T13:02:42.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.757+0000 7fdb6e651640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdb6815b2a0 0x7fdb681574d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:42.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.757+0000 7fdb6e651640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdb6815b2a0 0x7fdb681574d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43262/0 (socket says 172.21.15.57:43262) 2023-10-19T13:02:42.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.757+0000 7fdb6e651640 1 -- 172.21.15.57:0/1187403169 learned_addr learned my addr 172.21.15.57:0/1187403169 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:02:42.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.757+0000 7fdb6de50640 1 --2- 172.21.15.57:0/1187403169 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdb680ceb40 0x7fdb6815c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:42.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.757+0000 7fdb6d64f640 1 --2- 172.21.15.57:0/1187403169 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdb680cf340 0x7fdb680b1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:42.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.758+0000 7fdb6e651640 1 -- 172.21.15.57:0/1187403169 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdb680cf340 msgr2=0x7fdb680b1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:42.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.758+0000 7fdb6e651640 1 --2- 172.21.15.57:0/1187403169 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdb680cf340 0x7fdb680b1110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:42.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.758+0000 7fdb6e651640 1 -- 172.21.15.57:0/1187403169 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdb680ceb40 msgr2=0x7fdb6815c050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:42.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.758+0000 7fdb6e651640 1 --2- 172.21.15.57:0/1187403169 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdb680ceb40 0x7fdb6815c050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:42.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.758+0000 7fdb6e651640 1 -- 172.21.15.57:0/1187403169 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb64009490 con 0x7fdb6815b2a0 2023-10-19T13:02:42.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.758+0000 7fdb6e651640 1 --2- 172.21.15.57:0/1187403169 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdb6815b2a0 0x7fdb681574d0 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7fdb60001870 tx=0x7fdb6000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:02:42.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.758+0000 7fdb5e7fc640 1 -- 172.21.15.57:0/1187403169 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb60012420 con 0x7fdb6815b2a0 2023-10-19T13:02:42.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.758+0000 7fdb5e7fc640 1 -- 172.21.15.57:0/1187403169 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fdb60012690 con 0x7fdb6815b2a0 2023-10-19T13:02:42.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.758+0000 7fdb5e7fc640 1 -- 172.21.15.57:0/1187403169 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb60012aa0 con 0x7fdb6815b2a0 2023-10-19T13:02:42.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.758+0000 7fdb700b4640 1 -- 172.21.15.57:0/1187403169 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb6815cc30 con 0x7fdb6815b2a0 2023-10-19T13:02:42.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.758+0000 7fdb700b4640 1 -- 172.21.15.57:0/1187403169 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb680c93c0 con 0x7fdb6815b2a0 2023-10-19T13:02:42.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.759+0000 7fdb700b4640 1 -- 172.21.15.57:0/1187403169 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb680c98e0 con 0x7fdb6815b2a0 2023-10-19T13:02:42.767 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:02:42.759+0000 7f380220d040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-19T13:02:42.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.759+0000 7fdb5e7fc640 1 -- 172.21.15.57:0/1187403169 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 54) v1 ==== 89445+0+0 (secure 0 0 0) 0x7fdb6001b400 con 0x7fdb6815b2a0 2023-10-19T13:02:42.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.760+0000 7fdb5e7fc640 1 --2- 172.21.15.57:0/1187403169 >> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] conn(0x7fdb4c06b080 0x7fdb4c06d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:42.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.760+0000 7fdb6de50640 1 -- 172.21.15.57:0/1187403169 >> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] conn(0x7fdb4c06b080 msgr2=0x7fdb4c06d540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/2366702417 2023-10-19T13:02:42.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.760+0000 7fdb5e7fc640 1 -- 172.21.15.57:0/1187403169 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 6517+0+0 (secure 0 0 0) 0x7fdb6008a8f0 con 0x7fdb6815b2a0 2023-10-19T13:02:42.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.760+0000 7fdb6de50640 1 --2- 172.21.15.57:0/1187403169 >> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] conn(0x7fdb4c06b080 0x7fdb4c06d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T13:02:42.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.764+0000 7fdb5e7fc640 1 -- 172.21.15.57:0/1187403169 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fdb680c98e0 con 0x7fdb6815b2a0 2023-10-19T13:02:42.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.941+0000 7fdb700b4640 1 -- 172.21.15.57:0/1187403169 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdb680690e0 con 0x7fdb6815b2a0 2023-10-19T13:02:42.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.943+0000 7fdb5e7fc640 1 -- 172.21.15.57:0/1187403169 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v117) v1 ==== 74+0+17484 (secure 0 0 0) 0x7fdb600553a0 con 0x7fdb6815b2a0 2023-10-19T13:02:42.946 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:02:42.946 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":117,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T13:02:39.226727+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2023-10-19T13:01:24.443125+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2023-10-19T13:01:25.101276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"113","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.1","mappings":[{"from":1,"to":5}]},{"pgid":"14.12","mappings":[{"from":1,"to":7}]},{"pgid":"14.2d","mappings":[{"from":1,"to":7}]},{"pgid":"14.32","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.179:0/3304791412":"2023-10-19T14:01:39.552724+0000","172.21.15.179:0/3109348805":"2023-10-20T13:01:37.504869+0000","172.21.15.179:0/2984488835":"2023-10-20T13:01:37.504869+0000","172.21.15.179:0/205205608":"2023-10-20T13:01:37.504869+0000","172.21.15.179:0/4112649564":"2023-10-20T13:01:37.504869+0000","172.21.15.179:6833/3281457486":"2023-10-20T13:01:37.504869+0000","172.21.15.179:6832/3281457486":"2023-10-20T13:01:37.504869+0000","172.21.15.179:0/2509136709":"2023-10-20T13:01:37.504869+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T13:02:42.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.945+0000 7fdb700b4640 1 -- 172.21.15.57:0/1187403169 >> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] conn(0x7fdb4c06b080 msgr2=0x7fdb4c06d540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T13:02:42.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.945+0000 7fdb700b4640 1 --2- 172.21.15.57:0/1187403169 >> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] conn(0x7fdb4c06b080 0x7fdb4c06d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:42.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.945+0000 7fdb700b4640 1 -- 172.21.15.57:0/1187403169 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdb6815b2a0 msgr2=0x7fdb681574d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:42.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.945+0000 7fdb700b4640 1 --2- 172.21.15.57:0/1187403169 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdb6815b2a0 0x7fdb681574d0 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7fdb60001870 tx=0x7fdb6000a430 comp rx=0 tx=0).stop 2023-10-19T13:02:42.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.945+0000 7fdb700b4640 1 -- 172.21.15.57:0/1187403169 shutdown_connections 2023-10-19T13:02:42.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.945+0000 7fdb700b4640 1 --2- 172.21.15.57:0/1187403169 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdb6815b2a0 0x7fdb681574d0 unknown :-1 s=CLOSED pgs=1346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:42.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.945+0000 7fdb700b4640 1 --2- 172.21.15.57:0/1187403169 >> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] conn(0x7fdb4c06b080 0x7fdb4c06d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:42.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.945+0000 7fdb700b4640 1 --2- 172.21.15.57:0/1187403169 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdb680cf340 0x7fdb680b1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:42.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.945+0000 7fdb700b4640 1 --2- 172.21.15.57:0/1187403169 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdb680ceb40 0x7fdb6815c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:42.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.945+0000 7fdb700b4640 1 -- 172.21.15.57:0/1187403169 >> 172.21.15.57:0/1187403169 conn(0x7fdb680d0d60 msgr2=0x7fdb6815a560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:02:42.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.945+0000 7fdb700b4640 1 -- 172.21.15.57:0/1187403169 shutdown_connections 2023-10-19T13:02:42.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:42.945+0000 7fdb700b4640 1 -- 172.21.15.57:0/1187403169 wait complete. 2023-10-19T13:02:42.958 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-19T12:44:07.371186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-19T13:01:24.443125+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '112', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_data', 'create_time': '2023-10-19T13:01:25.101276+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '113', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-19T13:02:42.959 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-19T13:02:43.072 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.069+0000 7fd79fe10640 1 -- 172.21.15.57:0/2299703858 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd7980d1910 msgr2=0x7fd7980b0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:43.072 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.069+0000 7fd79fe10640 1 --2- 172.21.15.57:0/2299703858 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd7980d1910 0x7fd7980b0650 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fd790009be0 tx=0x7fd79002bf90 comp rx=0 tx=0).stop 2023-10-19T13:02:43.072 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.069+0000 7fd79fe10640 1 -- 172.21.15.57:0/2299703858 shutdown_connections 2023-10-19T13:02:43.073 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.069+0000 7fd79fe10640 1 --2- 172.21.15.57:0/2299703858 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd7980d1910 0x7fd7980b0650 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:43.073 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.069+0000 7fd79fe10640 1 -- 172.21.15.57:0/2299703858 >> 172.21.15.57:0/2299703858 conn(0x7fd7980d0d60 msgr2=0x7fd7980d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:02:43.073 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.069+0000 7fd79fe10640 1 -- 172.21.15.57:0/2299703858 shutdown_connections 2023-10-19T13:02:43.073 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.069+0000 7fd79fe10640 1 -- 172.21.15.57:0/2299703858 wait complete. 2023-10-19T13:02:43.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.070+0000 7fd79fe10640 1 Processor -- start 2023-10-19T13:02:43.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.070+0000 7fd79fe10640 1 -- start start 2023-10-19T13:02:43.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.071+0000 7fd79fe10640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd7980cf340 0x7fd798157bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:43.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.071+0000 7fd79fe10640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd7980d1910 0x7fd7981f5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:43.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.071+0000 7fd79fe10640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd79815b2a0 0x7fd7980aee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:43.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.071+0000 7fd79fe10640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd798158130 con 0x7fd7980d1910 2023-10-19T13:02:43.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.071+0000 7fd79fe10640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7981f27d0 con 0x7fd7980cf340 2023-10-19T13:02:43.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.071+0000 7fd79fe10640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7981f5d50 con 0x7fd79815b2a0 2023-10-19T13:02:43.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.071+0000 7fd79d3ab640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd7980d1910 0x7fd7981f5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:43.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.071+0000 7fd79d3ab640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd7980d1910 0x7fd7981f5810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43276/0 (socket says 172.21.15.57:43276) 2023-10-19T13:02:43.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.071+0000 7fd79d3ab640 1 -- 172.21.15.57:0/646948640 learned_addr learned my addr 172.21.15.57:0/646948640 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:02:43.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.071+0000 7fd79dbac640 1 --2- 172.21.15.57:0/646948640 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd7980cf340 0x7fd798157bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:43.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.071+0000 7fd79e3ad640 1 --2- 172.21.15.57:0/646948640 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd79815b2a0 0x7fd7980aee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:43.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.071+0000 7fd79d3ab640 1 -- 172.21.15.57:0/646948640 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd79815b2a0 msgr2=0x7fd7980aee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:43.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.071+0000 7fd79d3ab640 1 --2- 172.21.15.57:0/646948640 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd79815b2a0 0x7fd7980aee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:43.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.071+0000 7fd79d3ab640 1 -- 172.21.15.57:0/646948640 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd7980cf340 msgr2=0x7fd798157bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:43.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.071+0000 7fd79d3ab640 1 --2- 172.21.15.57:0/646948640 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd7980cf340 0x7fd798157bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:43.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.071+0000 7fd79d3ab640 1 -- 172.21.15.57:0/646948640 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd790009840 con 0x7fd7980d1910 2023-10-19T13:02:43.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.072+0000 7fd79d3ab640 1 --2- 172.21.15.57:0/646948640 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd7980d1910 0x7fd7981f5810 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7fd78800c6d0 tx=0x7fd78800cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:02:43.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.072+0000 7fd78e7fc640 1 -- 172.21.15.57:0/646948640 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd78800a840 con 0x7fd7980d1910 2023-10-19T13:02:43.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.072+0000 7fd78e7fc640 1 -- 172.21.15.57:0/646948640 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fd78800aab0 con 0x7fd7980d1910 2023-10-19T13:02:43.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.072+0000 7fd79fe10640 1 -- 172.21.15.57:0/646948640 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7981f1c50 con 0x7fd7980d1910 2023-10-19T13:02:43.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.072+0000 7fd78e7fc640 1 -- 172.21.15.57:0/646948640 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd788012420 con 0x7fd7980d1910 2023-10-19T13:02:43.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.072+0000 7fd79fe10640 1 -- 172.21.15.57:0/646948640 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7981f2210 con 0x7fd7980d1910 2023-10-19T13:02:43.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.073+0000 7fd79fe10640 1 -- 172.21.15.57:0/646948640 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7980c7ec0 con 0x7fd7980d1910 2023-10-19T13:02:43.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.073+0000 7fd78e7fc640 1 -- 172.21.15.57:0/646948640 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 54) v1 ==== 89445+0+0 (secure 0 0 0) 0x7fd788012580 con 0x7fd7980d1910 2023-10-19T13:02:43.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.074+0000 7fd78e7fc640 1 --2- 172.21.15.57:0/646948640 >> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] conn(0x7fd77c06b080 0x7fd77c06d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:43.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.074+0000 7fd79dbac640 1 -- 172.21.15.57:0/646948640 >> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] conn(0x7fd77c06b080 msgr2=0x7fd77c06d540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/2366702417 2023-10-19T13:02:43.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.074+0000 7fd78e7fc640 1 -- 172.21.15.57:0/646948640 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 6517+0+0 (secure 0 0 0) 0x7fd788089da0 con 0x7fd7980d1910 2023-10-19T13:02:43.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.074+0000 7fd79dbac640 1 --2- 172.21.15.57:0/646948640 >> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] conn(0x7fd77c06b080 0x7fd77c06d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T13:02:43.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.078+0000 7fd78e7fc640 1 -- 172.21.15.57:0/646948640 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd7980c7ec0 con 0x7fd7980d1910 2023-10-19T13:02:43.200 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:02:43.197+0000 7fc269232040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-19T13:02:43.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.253+0000 7fd79fe10640 1 -- 172.21.15.57:0/646948640 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd7980c7ec0 con 0x7fd7980d1910 2023-10-19T13:02:43.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.253+0000 7fd78e7fc640 1 -- 172.21.15.57:0/646948640 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v117) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd7980c7ec0 con 0x7fd7980d1910 2023-10-19T13:02:43.257 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-19T13:02:43.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.255+0000 7fd79fe10640 1 -- 172.21.15.57:0/646948640 >> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] conn(0x7fd77c06b080 msgr2=0x7fd77c06d540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T13:02:43.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.255+0000 7fd79fe10640 1 --2- 172.21.15.57:0/646948640 >> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] conn(0x7fd77c06b080 0x7fd77c06d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:43.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.255+0000 7fd79fe10640 1 -- 172.21.15.57:0/646948640 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd7980d1910 msgr2=0x7fd7981f5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:43.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.255+0000 7fd79fe10640 1 --2- 172.21.15.57:0/646948640 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd7980d1910 0x7fd7981f5810 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7fd78800c6d0 tx=0x7fd78800cb90 comp rx=0 tx=0).stop 2023-10-19T13:02:43.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.256+0000 7fd79fe10640 1 -- 172.21.15.57:0/646948640 shutdown_connections 2023-10-19T13:02:43.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.256+0000 7fd79fe10640 1 --2- 172.21.15.57:0/646948640 >> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] conn(0x7fd77c06b080 0x7fd77c06d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:43.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.256+0000 7fd79fe10640 1 --2- 172.21.15.57:0/646948640 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd79815b2a0 0x7fd7980aee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:43.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.256+0000 7fd79fe10640 1 --2- 172.21.15.57:0/646948640 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd7980d1910 0x7fd7981f5810 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:43.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.256+0000 7fd79fe10640 1 --2- 172.21.15.57:0/646948640 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd7980cf340 0x7fd798157bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:43.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.256+0000 7fd79fe10640 1 -- 172.21.15.57:0/646948640 >> 172.21.15.57:0/646948640 conn(0x7fd7980d0d60 msgr2=0x7fd79806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:02:43.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.256+0000 7fd79fe10640 1 -- 172.21.15.57:0/646948640 shutdown_connections 2023-10-19T13:02:43.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.256+0000 7fd79fe10640 1 -- 172.21.15.57:0/646948640 wait complete. 2023-10-19T13:02:43.269 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-19T13:02:43.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.380+0000 7ffa16864640 1 -- 172.21.15.57:0/2682589418 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ffa100cf340 msgr2=0x7ffa100d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:43.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.380+0000 7ffa16864640 1 --2- 172.21.15.57:0/2682589418 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ffa100cf340 0x7ffa100d1910 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ffa04009830 tx=0x7ffa0402bcd0 comp rx=0 tx=0).stop 2023-10-19T13:02:43.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.380+0000 7ffa16864640 1 -- 172.21.15.57:0/2682589418 shutdown_connections 2023-10-19T13:02:43.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.380+0000 7ffa16864640 1 --2- 172.21.15.57:0/2682589418 >> v2:172.21.15.57:3300/0 conn(0x7ffa100ceb40 0x7ffa100b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:43.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.380+0000 7ffa16864640 1 --2- 172.21.15.57:0/2682589418 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ffa100cf340 0x7ffa100d1910 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:43.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.380+0000 7ffa16864640 1 -- 172.21.15.57:0/2682589418 >> 172.21.15.57:0/2682589418 conn(0x7ffa100d0d60 msgr2=0x7ffa100d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:02:43.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.380+0000 7ffa16864640 1 -- 172.21.15.57:0/2682589418 shutdown_connections 2023-10-19T13:02:43.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.380+0000 7ffa16864640 1 -- 172.21.15.57:0/2682589418 wait complete. 2023-10-19T13:02:43.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.381+0000 7ffa16864640 1 Processor -- start 2023-10-19T13:02:43.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.381+0000 7ffa16864640 1 -- start start 2023-10-19T13:02:43.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.382+0000 7ffa16864640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ffa100ceb40 0x7ffa1015c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:43.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.382+0000 7ffa16864640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffa100cf340 0x7ffa100c8040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:43.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.382+0000 7ffa16864640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffa1015b2a0 0x7ffa10150fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:43.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.382+0000 7ffa16864640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa10151510 con 0x7ffa100cf340 2023-10-19T13:02:43.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.382+0000 7ffa16864640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa1015df10 con 0x7ffa1015b2a0 2023-10-19T13:02:43.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.382+0000 7ffa16864640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa1015ce70 con 0x7ffa100ceb40 2023-10-19T13:02:43.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.382+0000 7ffa0f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffa100cf340 0x7ffa100c8040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:43.391 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.382+0000 7ffa0f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffa100cf340 0x7ffa100c8040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43298/0 (socket says 172.21.15.57:43298) 2023-10-19T13:02:43.391 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.382+0000 7ffa0f7fe640 1 -- 172.21.15.57:0/511231463 learned_addr learned my addr 172.21.15.57:0/511231463 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:02:43.391 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.382+0000 7ffa14e01640 1 --2- 172.21.15.57:0/511231463 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffa1015b2a0 0x7ffa10150fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:43.392 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.382+0000 7ffa0ffff640 1 --2- 172.21.15.57:0/511231463 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ffa100ceb40 0x7ffa1015c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:43.392 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.383+0000 7ffa0f7fe640 1 -- 172.21.15.57:0/511231463 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ffa100ceb40 msgr2=0x7ffa1015c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:43.392 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.383+0000 7ffa0f7fe640 1 --2- 172.21.15.57:0/511231463 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ffa100ceb40 0x7ffa1015c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:43.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.383+0000 7ffa0f7fe640 1 -- 172.21.15.57:0/511231463 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffa1015b2a0 msgr2=0x7ffa10150fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:43.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.383+0000 7ffa0f7fe640 1 --2- 172.21.15.57:0/511231463 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffa1015b2a0 0x7ffa10150fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:43.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.383+0000 7ffa0f7fe640 1 -- 172.21.15.57:0/511231463 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa04009490 con 0x7ffa100cf340 2023-10-19T13:02:43.394 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.383+0000 7ffa0f7fe640 1 --2- 172.21.15.57:0/511231463 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffa100cf340 0x7ffa100c8040 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7ffa000046b0 tx=0x7ffa0000c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:02:43.394 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.383+0000 7ffa0cff9640 1 -- 172.21.15.57:0/511231463 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa00004230 con 0x7ffa100cf340 2023-10-19T13:02:43.394 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.383+0000 7ffa16864640 1 -- 172.21.15.57:0/511231463 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa1015d150 con 0x7ffa100cf340 2023-10-19T13:02:43.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.383+0000 7ffa0cff9640 1 -- 172.21.15.57:0/511231463 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7ffa00004480 con 0x7ffa100cf340 2023-10-19T13:02:43.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.383+0000 7ffa0cff9640 1 -- 172.21.15.57:0/511231463 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa00003850 con 0x7ffa100cf340 2023-10-19T13:02:43.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.383+0000 7ffa16864640 1 -- 172.21.15.57:0/511231463 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa1015d710 con 0x7ffa100cf340 2023-10-19T13:02:43.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.384+0000 7ffa16864640 1 -- 172.21.15.57:0/511231463 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa100690e0 con 0x7ffa100cf340 2023-10-19T13:02:43.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.385+0000 7ffa0cff9640 1 -- 172.21.15.57:0/511231463 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 54) v1 ==== 89445+0+0 (secure 0 0 0) 0x7ffa00003a70 con 0x7ffa100cf340 2023-10-19T13:02:43.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.385+0000 7ffa0cff9640 1 --2- 172.21.15.57:0/511231463 >> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] conn(0x7ff9fc06b080 0x7ff9fc06d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:43.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.385+0000 7ffa0cff9640 1 -- 172.21.15.57:0/511231463 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 6517+0+0 (secure 0 0 0) 0x7ffa000894c0 con 0x7ffa100cf340 2023-10-19T13:02:43.397 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.385+0000 7ffa0ffff640 1 -- 172.21.15.57:0/511231463 >> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] conn(0x7ff9fc06b080 msgr2=0x7ff9fc06d540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/2366702417 2023-10-19T13:02:43.397 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.385+0000 7ffa0ffff640 1 --2- 172.21.15.57:0/511231463 >> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] conn(0x7ff9fc06b080 0x7ff9fc06d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T13:02:43.397 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.389+0000 7ffa0cff9640 1 -- 172.21.15.57:0/511231463 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ffa00053f50 con 0x7ffa100cf340 2023-10-19T13:02:43.435 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:02:43.433+0000 7f380220d040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-19T13:02:43.463 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:02:43.461+0000 7fc269232040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-19T13:02:43.511 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:02:43.508+0000 7fc269232040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-19T13:02:43.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.565+0000 7ffa16864640 1 -- 172.21.15.57:0/511231463 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ffa100d0b40 con 0x7ffa100cf340 2023-10-19T13:02:43.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.566+0000 7ffa0cff9640 1 -- 172.21.15.57:0/511231463 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v117) v1 ==== 104+0+11 (secure 0 0 0) 0x7ffa00057570 con 0x7ffa100cf340 2023-10-19T13:02:43.569 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2023-10-19T13:02:43.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.568+0000 7ffa16864640 1 -- 172.21.15.57:0/511231463 >> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] conn(0x7ff9fc06b080 msgr2=0x7ff9fc06d540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T13:02:43.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.568+0000 7ffa16864640 1 --2- 172.21.15.57:0/511231463 >> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] conn(0x7ff9fc06b080 0x7ff9fc06d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:43.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.568+0000 7ffa16864640 1 -- 172.21.15.57:0/511231463 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffa100cf340 msgr2=0x7ffa100c8040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:43.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.568+0000 7ffa16864640 1 --2- 172.21.15.57:0/511231463 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffa100cf340 0x7ffa100c8040 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7ffa000046b0 tx=0x7ffa0000c820 comp rx=0 tx=0).stop 2023-10-19T13:02:43.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.568+0000 7ffa16864640 1 -- 172.21.15.57:0/511231463 shutdown_connections 2023-10-19T13:02:43.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.569+0000 7ffa16864640 1 --2- 172.21.15.57:0/511231463 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffa1015b2a0 0x7ffa10150fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:43.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.569+0000 7ffa16864640 1 --2- 172.21.15.57:0/511231463 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffa100cf340 0x7ffa100c8040 unknown :-1 s=CLOSED pgs=1350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:43.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.569+0000 7ffa16864640 1 --2- 172.21.15.57:0/511231463 >> [v2:172.21.15.179:6832/2366702417,v1:172.21.15.179:6833/2366702417] conn(0x7ff9fc06b080 0x7ff9fc06d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:43.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.569+0000 7ffa16864640 1 --2- 172.21.15.57:0/511231463 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ffa100ceb40 0x7ffa1015c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:43.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.569+0000 7ffa16864640 1 -- 172.21.15.57:0/511231463 >> 172.21.15.57:0/511231463 conn(0x7ffa100d0d60 msgr2=0x7ffa10159fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:02:43.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.569+0000 7ffa16864640 1 -- 172.21.15.57:0/511231463 shutdown_connections 2023-10-19T13:02:43.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.569+0000 7ffa16864640 1 -- 172.21.15.57:0/511231463 wait complete. 2023-10-19T13:02:43.583 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-19T13:02:43.609 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:02:43.607+0000 7fc269232040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-19T13:02:43.692 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:02:43.689+0000 7fc1eae69640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:02:43.693 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:02:43.689+0000 7fc1eae69640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:02:43.693 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:02:43.689+0000 7fc1eae69640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:02:43.693 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:02:43.689+0000 7fc1eae69640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:02:43.693 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:02:43.689+0000 7fc1eae69640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:02:43.693 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:02:43.689+0000 7fc1ed66e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:02:43.694 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:02:43.689+0000 7fc1ed66e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:02:43.694 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:02:43.689+0000 7fc1ed66e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:02:43.694 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:02:43.689+0000 7fc1ed66e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:02:43.694 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:02:43.689+0000 7fc1ed66e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:02:43.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.700+0000 7f233faf2640 1 -- 172.21.15.57:0/1183655969 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f233815b260 msgr2=0x7f23380cf310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:43.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.700+0000 7f233faf2640 1 --2- 172.21.15.57:0/1183655969 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f233815b260 0x7f23380cf310 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f2334009830 tx=0x7f233402bd60 comp rx=0 tx=0).stop 2023-10-19T13:02:43.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.700+0000 7f233faf2640 1 -- 172.21.15.57:0/1183655969 shutdown_connections 2023-10-19T13:02:43.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.700+0000 7f233faf2640 1 --2- 172.21.15.57:0/1183655969 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f23380cf850 0x7f23380cc9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:43.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.700+0000 7f233faf2640 1 --2- 172.21.15.57:0/1183655969 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f233815b260 0x7f23380cf310 unknown :-1 s=CLOSED pgs=1353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:43.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.700+0000 7f233faf2640 1 -- 172.21.15.57:0/1183655969 >> 172.21.15.57:0/1183655969 conn(0x7f233809a030 msgr2=0x7f23380d0d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:02:43.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.700+0000 7f233faf2640 1 -- 172.21.15.57:0/1183655969 shutdown_connections 2023-10-19T13:02:43.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.700+0000 7f233faf2640 1 -- 172.21.15.57:0/1183655969 wait complete. 2023-10-19T13:02:43.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.701+0000 7f233faf2640 1 Processor -- start 2023-10-19T13:02:43.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.701+0000 7f233faf2640 1 -- start start 2023-10-19T13:02:43.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.702+0000 7f233faf2640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f23380cf850 0x7f233814eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:43.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.702+0000 7f233faf2640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f233815b260 0x7f23380ca690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:43.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.702+0000 7f233faf2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f233815b640 0x7f23380b4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:43.707 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.702+0000 7f233faf2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23380b4b80 con 0x7f233815b640 2023-10-19T13:02:43.707 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.702+0000 7f233faf2640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23380cabd0 con 0x7f23380cf850 2023-10-19T13:02:43.707 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.702+0000 7f233faf2640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23380b3f00 con 0x7f233815b260 2023-10-19T13:02:43.707 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.702+0000 7f233e08f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f233815b640 0x7f23380b4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:43.707 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.702+0000 7f233d88e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f23380cf850 0x7f233814eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:43.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.702+0000 7f233e08f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f233815b640 0x7f23380b4fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43316/0 (socket says 172.21.15.57:43316) 2023-10-19T13:02:43.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.702+0000 7f233e08f640 1 -- 172.21.15.57:0/1197229219 learned_addr learned my addr 172.21.15.57:0/1197229219 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:02:43.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.702+0000 7f233d08d640 1 --2- 172.21.15.57:0/1197229219 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f233815b260 0x7f23380ca690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:43.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.702+0000 7f233e08f640 1 -- 172.21.15.57:0/1197229219 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f233815b260 msgr2=0x7f23380ca690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:43.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.702+0000 7f233e08f640 1 --2- 172.21.15.57:0/1197229219 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f233815b260 0x7f23380ca690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:43.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.702+0000 7f233e08f640 1 -- 172.21.15.57:0/1197229219 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f23380cf850 msgr2=0x7f233814eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:43.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.702+0000 7f233e08f640 1 --2- 172.21.15.57:0/1197229219 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f23380cf850 0x7f233814eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:43.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.702+0000 7f233e08f640 1 -- 172.21.15.57:0/1197229219 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2334009490 con 0x7f233815b640 2023-10-19T13:02:43.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.702+0000 7f233d88e640 1 --2- 172.21.15.57:0/1197229219 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f23380cf850 0x7f233814eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T13:02:43.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.702+0000 7f233e08f640 1 --2- 172.21.15.57:0/1197229219 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f233815b640 0x7f23380b4fa0 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f233000eb10 tx=0x7f233000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:02:43.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.703+0000 7f232e7fc640 1 -- 172.21.15.57:0/1197229219 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2330004b20 con 0x7f233815b640 2023-10-19T13:02:43.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.703+0000 7f232e7fc640 1 -- 172.21.15.57:0/1197229219 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f2330004ca0 con 0x7f233815b640 2023-10-19T13:02:43.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.703+0000 7f233faf2640 1 -- 172.21.15.57:0/1197229219 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23380b41e0 con 0x7f233815b640 2023-10-19T13:02:43.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.703+0000 7f232e7fc640 1 -- 172.21.15.57:0/1197229219 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2330005280 con 0x7f233815b640 2023-10-19T13:02:43.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.703+0000 7f233faf2640 1 -- 172.21.15.57:0/1197229219 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23380b47a0 con 0x7f233815b640 2023-10-19T13:02:43.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.704+0000 7f233faf2640 1 -- 172.21.15.57:0/1197229219 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23380b4cc0 con 0x7f233815b640 2023-10-19T13:02:43.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.704+0000 7f232e7fc640 1 -- 172.21.15.57:0/1197229219 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 55) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f23300053e0 con 0x7f233815b640 2023-10-19T13:02:43.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.705+0000 7f232e7fc640 1 -- 172.21.15.57:0/1197229219 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6947+0+0 (secure 0 0 0) 0x7f2330014070 con 0x7f233815b640 2023-10-19T13:02:43.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.709+0000 7f232e7fc640 1 -- 172.21.15.57:0/1197229219 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f23380b4cc0 con 0x7f233815b640 2023-10-19T13:02:43.726 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:02:43.723+0000 7f380220d040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-19T13:02:43.778 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:02:43.776+0000 7f380220d040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-19T13:02:43.886 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:02:43.883+0000 7f380220d040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-19T13:02:43.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.885+0000 7f233faf2640 1 -- 172.21.15.57:0/1197229219 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f23380b4cc0 con 0x7f233815b640 2023-10-19T13:02:43.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.885+0000 7f232e7fc640 1 -- 172.21.15.57:0/1197229219 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v118) v1 ==== 100+0+11 (secure 0 0 0) 0x7f23380b4cc0 con 0x7f233815b640 2023-10-19T13:02:43.888 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 64 2023-10-19T13:02:43.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.887+0000 7f233faf2640 1 -- 172.21.15.57:0/1197229219 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f233815b640 msgr2=0x7f23380b4fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:43.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.887+0000 7f233faf2640 1 --2- 172.21.15.57:0/1197229219 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f233815b640 0x7f23380b4fa0 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f233000eb10 tx=0x7f233000c2d0 comp rx=0 tx=0).stop 2023-10-19T13:02:43.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.887+0000 7f233faf2640 1 -- 172.21.15.57:0/1197229219 shutdown_connections 2023-10-19T13:02:43.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.887+0000 7f233faf2640 1 --2- 172.21.15.57:0/1197229219 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f233815b640 0x7f23380b4fa0 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:43.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.887+0000 7f233faf2640 1 --2- 172.21.15.57:0/1197229219 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f233815b260 0x7f23380ca690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:43.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.887+0000 7f233faf2640 1 --2- 172.21.15.57:0/1197229219 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f23380cf850 0x7f233814eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:43.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.887+0000 7f233faf2640 1 -- 172.21.15.57:0/1197229219 >> 172.21.15.57:0/1197229219 conn(0x7f233809a030 msgr2=0x7f233815a020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:02:43.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.887+0000 7f233faf2640 1 -- 172.21.15.57:0/1197229219 shutdown_connections 2023-10-19T13:02:43.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:43.887+0000 7f233faf2640 1 -- 172.21.15.57:0/1197229219 wait complete. 2023-10-19T13:02:43.901 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T13:02:44.012 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.009+0000 7f798d2c9640 1 -- 172.21.15.57:0/3197181058 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f79880b0530 msgr2=0x7f79880b09a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:44.012 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.009+0000 7f798d2c9640 1 --2- 172.21.15.57:0/3197181058 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f79880b0530 0x7f79880b09a0 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f79880cc7e0 tx=0x7f798002c110 comp rx=0 tx=0).stop 2023-10-19T13:02:44.012 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.009+0000 7f798d2c9640 1 -- 172.21.15.57:0/3197181058 shutdown_connections 2023-10-19T13:02:44.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.009+0000 7f798d2c9640 1 --2- 172.21.15.57:0/3197181058 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f79880b0530 0x7f79880b09a0 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:44.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.009+0000 7f798d2c9640 1 --2- 172.21.15.57:0/3197181058 >> v2:172.21.15.179:3300/0 conn(0x7f79880cbd20 0x7f7988158300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:44.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.009+0000 7f798d2c9640 1 -- 172.21.15.57:0/3197181058 >> 172.21.15.57:0/3197181058 conn(0x7f7988069930 msgr2=0x7f7988069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:02:44.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.009+0000 7f798d2c9640 1 -- 172.21.15.57:0/3197181058 shutdown_connections 2023-10-19T13:02:44.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.009+0000 7f798d2c9640 1 -- 172.21.15.57:0/3197181058 wait complete. 2023-10-19T13:02:44.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.010+0000 7f798d2c9640 1 Processor -- start 2023-10-19T13:02:44.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.010+0000 7f798d2c9640 1 -- start start 2023-10-19T13:02:44.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.011+0000 7f798d2c9640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f79880afa20 0x7f798815a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:44.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.011+0000 7f798d2c9640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f79880b0530 0x7f7988141060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:44.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.011+0000 7f798d2c9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f79880cbd20 0x7f798814bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:44.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.011+0000 7f798d2c9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79881415a0 con 0x7f79880cbd20 2023-10-19T13:02:44.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.011+0000 7f798d2c9640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7988148230 con 0x7f79880b0530 2023-10-19T13:02:44.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.011+0000 7f798d2c9640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f798814c0a0 con 0x7f79880afa20 2023-10-19T13:02:44.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.011+0000 7f79877fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f79880cbd20 0x7f798814bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:44.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.011+0000 7f79877fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f79880cbd20 0x7f798814bb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43334/0 (socket says 172.21.15.57:43334) 2023-10-19T13:02:44.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.011+0000 7f79877fe640 1 -- 172.21.15.57:0/2484015735 learned_addr learned my addr 172.21.15.57:0/2484015735 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:02:44.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.011+0000 7f7986ffd640 1 --2- 172.21.15.57:0/2484015735 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f79880afa20 0x7f798815a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:44.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.011+0000 7f797e7fc640 1 --2- 172.21.15.57:0/2484015735 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f79880b0530 0x7f7988141060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:44.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.011+0000 7f79877fe640 1 -- 172.21.15.57:0/2484015735 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f79880afa20 msgr2=0x7f798815a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:44.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.011+0000 7f79877fe640 1 --2- 172.21.15.57:0/2484015735 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f79880afa20 0x7f798815a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:44.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.011+0000 7f79877fe640 1 -- 172.21.15.57:0/2484015735 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f79880b0530 msgr2=0x7f7988141060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:44.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.011+0000 7f79877fe640 1 --2- 172.21.15.57:0/2484015735 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f79880b0530 0x7f7988141060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:44.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.011+0000 7f79877fe640 1 -- 172.21.15.57:0/2484015735 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7980009840 con 0x7f79880cbd20 2023-10-19T13:02:44.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.011+0000 7f79877fe640 1 --2- 172.21.15.57:0/2484015735 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f79880cbd20 0x7f798814bb60 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7f7980009ca0 tx=0x7f7980004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:02:44.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.012+0000 7f797ffff640 1 -- 172.21.15.57:0/2484015735 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7980004760 con 0x7f79880cbd20 2023-10-19T13:02:44.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.012+0000 7f797ffff640 1 -- 172.21.15.57:0/2484015735 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f79800345a0 con 0x7f79880cbd20 2023-10-19T13:02:44.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.012+0000 7f798d2c9640 1 -- 172.21.15.57:0/2484015735 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7988147c70 con 0x7f79880cbd20 2023-10-19T13:02:44.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.012+0000 7f797ffff640 1 -- 172.21.15.57:0/2484015735 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79800349b0 con 0x7f79880cbd20 2023-10-19T13:02:44.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.012+0000 7f798d2c9640 1 -- 172.21.15.57:0/2484015735 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f798814caf0 con 0x7f79880cbd20 2023-10-19T13:02:44.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.013+0000 7f798d2c9640 1 -- 172.21.15.57:0/2484015735 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79880690e0 con 0x7f79880cbd20 2023-10-19T13:02:44.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.013+0000 7f797ffff640 1 -- 172.21.15.57:0/2484015735 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 55) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f7980034b10 con 0x7f79880cbd20 2023-10-19T13:02:44.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.014+0000 7f797ffff640 1 -- 172.21.15.57:0/2484015735 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6947+0+0 (secure 0 0 0) 0x7f79800aca60 con 0x7f79880cbd20 2023-10-19T13:02:44.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.018+0000 7f797ffff640 1 -- 172.21.15.57:0/2484015735 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7980077bd0 con 0x7f79880cbd20 2023-10-19T13:02:44.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.231+0000 7f798d2c9640 1 -- 172.21.15.57:0/2484015735 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f798806f700 con 0x7f79880cbd20 2023-10-19T13:02:44.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.231+0000 7f797ffff640 1 -- 172.21.15.57:0/2484015735 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+4921 (secure 0 0 0) 0x7f798007b1f0 con 0x7f79880cbd20 2023-10-19T13:02:44.234 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:02:44.235 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":44,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6611,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/2708395824","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2708395824},{"type":"v1","addr":"172.21.15.179:6836","nonce":2708395824}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":41},{"gid":7270,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/3803632191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":3803632191},{"type":"v1","addr":"172.21.15.179:6837","nonce":3803632191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":41},{"gid":7282,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6833/2259979700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2259979700},{"type":"v1","addr":"172.21.15.57:6833","nonce":2259979700}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T13:01:26.118951+0000","modified":"2023-10-19T13:01:28.134083+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8046},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8046":{"gid":8046,"name":"c","rank":0,"incarnation":42,"state":"up:active","state_seq":3,"addr":"172.21.15.57:6835/2556427177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":2556427177},{"type":"v1","addr":"172.21.15.57:6835","nonce":2556427177}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-10-19T13:02:44.235 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 44 2023-10-19T13:02:44.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.233+0000 7f798d2c9640 1 -- 172.21.15.57:0/2484015735 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f79880cbd20 msgr2=0x7f798814bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:44.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.233+0000 7f798d2c9640 1 --2- 172.21.15.57:0/2484015735 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f79880cbd20 0x7f798814bb60 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7f7980009ca0 tx=0x7f7980004360 comp rx=0 tx=0).stop 2023-10-19T13:02:44.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.233+0000 7f798d2c9640 1 -- 172.21.15.57:0/2484015735 shutdown_connections 2023-10-19T13:02:44.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.233+0000 7f798d2c9640 1 --2- 172.21.15.57:0/2484015735 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f79880cbd20 0x7f798814bb60 unknown :-1 s=CLOSED pgs=1357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:44.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.233+0000 7f798d2c9640 1 --2- 172.21.15.57:0/2484015735 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f79880b0530 0x7f7988141060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:44.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.233+0000 7f798d2c9640 1 --2- 172.21.15.57:0/2484015735 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f79880afa20 0x7f798815a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:44.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.233+0000 7f798d2c9640 1 -- 172.21.15.57:0/2484015735 >> 172.21.15.57:0/2484015735 conn(0x7f7988069930 msgr2=0x7f79880ca870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:02:44.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.233+0000 7f798d2c9640 1 -- 172.21.15.57:0/2484015735 shutdown_connections 2023-10-19T13:02:44.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.233+0000 7f798d2c9640 1 -- 172.21.15.57:0/2484015735 wait complete. 2023-10-19T13:02:44.247 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T13:02:44.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.353+0000 7f28c3e7c640 1 -- 172.21.15.57:0/3883224065 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f28bc13d360 msgr2=0x7f28bc15a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:44.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.353+0000 7f28c3e7c640 1 --2- 172.21.15.57:0/3883224065 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f28bc13d360 0x7f28bc15a9f0 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f28b8005fd0 tx=0x7f28b802c2e0 comp rx=0 tx=0).stop 2023-10-19T13:02:44.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.353+0000 7f28c3e7c640 1 -- 172.21.15.57:0/3883224065 shutdown_connections 2023-10-19T13:02:44.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.353+0000 7f28c3e7c640 1 --2- 172.21.15.57:0/3883224065 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f28bc0c9ed0 0x7f28bc06b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:44.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.353+0000 7f28c3e7c640 1 --2- 172.21.15.57:0/3883224065 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f28bc13d360 0x7f28bc15a9f0 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:44.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.353+0000 7f28c3e7c640 1 -- 172.21.15.57:0/3883224065 >> 172.21.15.57:0/3883224065 conn(0x7f28bc069930 msgr2=0x7f28bc069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:02:44.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.353+0000 7f28c3e7c640 1 -- 172.21.15.57:0/3883224065 shutdown_connections 2023-10-19T13:02:44.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.353+0000 7f28c3e7c640 1 -- 172.21.15.57:0/3883224065 wait complete. 2023-10-19T13:02:44.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.354+0000 7f28c3e7c640 1 Processor -- start 2023-10-19T13:02:44.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.354+0000 7f28c3e7c640 1 -- start start 2023-10-19T13:02:44.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.355+0000 7f28c3e7c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f28bc0c9ed0 0x7f28bc1f3280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:44.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.355+0000 7f28c3e7c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f28bc13d360 0x7f28bc133840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:44.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.355+0000 7f28c3e7c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f28bc140660 0x7f28bc132030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:44.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.355+0000 7f28c3e7c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28bc133d80 con 0x7f28bc0c9ed0 2023-10-19T13:02:44.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.355+0000 7f28c3e7c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28bc132570 con 0x7f28bc140660 2023-10-19T13:02:44.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.355+0000 7f28c3e7c640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28bc1327a0 con 0x7f28bc13d360 2023-10-19T13:02:44.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.355+0000 7f28c1c18640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f28bc0c9ed0 0x7f28bc1f3280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:44.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.355+0000 7f28c1c18640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f28bc0c9ed0 0x7f28bc1f3280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43350/0 (socket says 172.21.15.57:43350) 2023-10-19T13:02:44.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.355+0000 7f28c1c18640 1 -- 172.21.15.57:0/3126203239 learned_addr learned my addr 172.21.15.57:0/3126203239 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:02:44.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.356+0000 7f28c1417640 1 --2- 172.21.15.57:0/3126203239 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f28bc13d360 0x7f28bc133840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:44.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.356+0000 7f28c1c18640 1 -- 172.21.15.57:0/3126203239 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f28bc13d360 msgr2=0x7f28bc133840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:44.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.356+0000 7f28c2419640 1 --2- 172.21.15.57:0/3126203239 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f28bc140660 0x7f28bc132030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:44.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.356+0000 7f28c1c18640 1 --2- 172.21.15.57:0/3126203239 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f28bc13d360 0x7f28bc133840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:44.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.356+0000 7f28c1c18640 1 -- 172.21.15.57:0/3126203239 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f28bc140660 msgr2=0x7f28bc132030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:44.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.356+0000 7f28c1c18640 1 --2- 172.21.15.57:0/3126203239 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f28bc140660 0x7f28bc132030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:44.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.356+0000 7f28c1c18640 1 -- 172.21.15.57:0/3126203239 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28b8009900 con 0x7f28bc0c9ed0 2023-10-19T13:02:44.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.356+0000 7f28c1417640 1 --2- 172.21.15.57:0/3126203239 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f28bc13d360 0x7f28bc133840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T13:02:44.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.356+0000 7f28c2419640 1 --2- 172.21.15.57:0/3126203239 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f28bc140660 0x7f28bc132030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T13:02:44.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.356+0000 7f28c1c18640 1 --2- 172.21.15.57:0/3126203239 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f28bc0c9ed0 0x7f28bc1f3280 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7f28b8006010 tx=0x7f28b8035b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:02:44.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.356+0000 7f28b27fc640 1 -- 172.21.15.57:0/3126203239 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28b8004240 con 0x7f28bc0c9ed0 2023-10-19T13:02:44.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.356+0000 7f28b27fc640 1 -- 172.21.15.57:0/3126203239 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f28b8004490 con 0x7f28bc0c9ed0 2023-10-19T13:02:44.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.356+0000 7f28c3e7c640 1 -- 172.21.15.57:0/3126203239 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28bc132a80 con 0x7f28bc0c9ed0 2023-10-19T13:02:44.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.356+0000 7f28b27fc640 1 -- 172.21.15.57:0/3126203239 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28b8004860 con 0x7f28bc0c9ed0 2023-10-19T13:02:44.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.356+0000 7f28c3e7c640 1 -- 172.21.15.57:0/3126203239 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28bc133040 con 0x7f28bc0c9ed0 2023-10-19T13:02:44.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.357+0000 7f28c3e7c640 1 -- 172.21.15.57:0/3126203239 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28bc0690e0 con 0x7f28bc0c9ed0 2023-10-19T13:02:44.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.357+0000 7f28b27fc640 1 -- 172.21.15.57:0/3126203239 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 55) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f28b803b050 con 0x7f28bc0c9ed0 2023-10-19T13:02:44.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.358+0000 7f28b27fc640 1 -- 172.21.15.57:0/3126203239 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6947+0+0 (secure 0 0 0) 0x7f28b80ac3f0 con 0x7f28bc0c9ed0 2023-10-19T13:02:44.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.362+0000 7f28b27fc640 1 -- 172.21.15.57:0/3126203239 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f28b8077410 con 0x7f28bc0c9ed0 2023-10-19T13:02:44.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.538+0000 7f28c3e7c640 1 -- 172.21.15.57:0/3126203239 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f28bc159940 con 0x7f28bc0c9ed0 2023-10-19T13:02:44.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.539+0000 7f28b27fc640 1 -- 172.21.15.57:0/3126203239 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v118) v1 ==== 74+0+18117 (secure 0 0 0) 0x7f28b807aa30 con 0x7f28bc0c9ed0 2023-10-19T13:02:44.543 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:02:44.543 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":118,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T13:02:43.621627+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2023-10-19T13:01:24.443125+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2023-10-19T13:01:25.101276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"113","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.1","mappings":[{"from":1,"to":5}]},{"pgid":"14.12","mappings":[{"from":1,"to":7}]},{"pgid":"14.2d","mappings":[{"from":1,"to":7}]},{"pgid":"14.32","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.179:0/2599642592":"2023-10-20T13:02:43.621587+0000","172.21.15.179:0/2915428574":"2023-10-20T13:02:43.621587+0000","172.21.15.179:0/356140706":"2023-10-20T13:02:43.621587+0000","172.21.15.179:0/4203351642":"2023-10-20T13:02:43.621587+0000","172.21.15.179:0/842129505":"2023-10-20T13:02:43.621587+0000","172.21.15.179:0/1511457369":"2023-10-20T13:02:43.621587+0000","172.21.15.179:6833/2366702417":"2023-10-20T13:02:43.621587+0000","172.21.15.179:6832/2366702417":"2023-10-20T13:02:43.621587+0000","172.21.15.179:0/3304791412":"2023-10-19T14:01:39.552724+0000","172.21.15.179:0/3109348805":"2023-10-20T13:01:37.504869+0000","172.21.15.179:0/2984488835":"2023-10-20T13:01:37.504869+0000","172.21.15.179:0/205205608":"2023-10-20T13:01:37.504869+0000","172.21.15.179:0/4112649564":"2023-10-20T13:01:37.504869+0000","172.21.15.179:6833/3281457486":"2023-10-20T13:01:37.504869+0000","172.21.15.179:0/1633718057":"2023-10-20T13:02:43.621587+0000","172.21.15.179:6832/3281457486":"2023-10-20T13:01:37.504869+0000","172.21.15.179:0/318926584":"2023-10-20T13:02:43.621587+0000","172.21.15.179:0/2509136709":"2023-10-20T13:01:37.504869+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T13:02:44.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.541+0000 7f28c3e7c640 1 -- 172.21.15.57:0/3126203239 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f28bc0c9ed0 msgr2=0x7f28bc1f3280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:44.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.541+0000 7f28c3e7c640 1 --2- 172.21.15.57:0/3126203239 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f28bc0c9ed0 0x7f28bc1f3280 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7f28b8006010 tx=0x7f28b8035b50 comp rx=0 tx=0).stop 2023-10-19T13:02:44.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.541+0000 7f28c3e7c640 1 -- 172.21.15.57:0/3126203239 shutdown_connections 2023-10-19T13:02:44.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.541+0000 7f28c3e7c640 1 --2- 172.21.15.57:0/3126203239 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f28bc140660 0x7f28bc132030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:44.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.541+0000 7f28c3e7c640 1 --2- 172.21.15.57:0/3126203239 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f28bc13d360 0x7f28bc133840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:44.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.541+0000 7f28c3e7c640 1 --2- 172.21.15.57:0/3126203239 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f28bc0c9ed0 0x7f28bc1f3280 unknown :-1 s=CLOSED pgs=1360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:44.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.542+0000 7f28c3e7c640 1 -- 172.21.15.57:0/3126203239 >> 172.21.15.57:0/3126203239 conn(0x7f28bc069930 msgr2=0x7f28bc13f2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:02:44.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.542+0000 7f28c3e7c640 1 -- 172.21.15.57:0/3126203239 shutdown_connections 2023-10-19T13:02:44.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.542+0000 7f28c3e7c640 1 -- 172.21.15.57:0/3126203239 wait complete. 2023-10-19T13:02:44.555 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-19T13:02:44.556 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T13:02:44.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.667+0000 7fb213755640 1 -- 172.21.15.57:0/1603650005 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb20c06f0a0 msgr2=0x7fb20c06ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:44.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.667+0000 7fb213755640 1 --2- 172.21.15.57:0/1603650005 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb20c06f0a0 0x7fb20c06ddc0 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7fb20c0c9200 tx=0x7fb1fc02bd70 comp rx=0 tx=0).stop 2023-10-19T13:02:44.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.667+0000 7fb213755640 1 -- 172.21.15.57:0/1603650005 shutdown_connections 2023-10-19T13:02:44.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.667+0000 7fb213755640 1 --2- 172.21.15.57:0/1603650005 >> v2:172.21.15.179:3300/0 conn(0x7fb20c0ad370 0x7fb20c0ad800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:44.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.667+0000 7fb213755640 1 --2- 172.21.15.57:0/1603650005 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb20c06f0a0 0x7fb20c06ddc0 unknown :-1 s=CLOSED pgs=1362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:44.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.667+0000 7fb213755640 1 -- 172.21.15.57:0/1603650005 >> 172.21.15.57:0/1603650005 conn(0x7fb20c0b0670 msgr2=0x7fb20c0b0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:02:44.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.667+0000 7fb213755640 1 -- 172.21.15.57:0/1603650005 shutdown_connections 2023-10-19T13:02:44.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.667+0000 7fb213755640 1 -- 172.21.15.57:0/1603650005 wait complete. 2023-10-19T13:02:44.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.668+0000 7fb213755640 1 Processor -- start 2023-10-19T13:02:44.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.668+0000 7fb213755640 1 -- start start 2023-10-19T13:02:44.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.669+0000 7fb213755640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb20c06d2d0 0x7fb20c14e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:44.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.669+0000 7fb213755640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb20c06f0a0 0x7fb20c0cb370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:44.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.669+0000 7fb213755640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb20c0ad370 0x7fb20c0cb8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:44.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.669+0000 7fb213755640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb20c0b17d0 con 0x7fb20c06d2d0 2023-10-19T13:02:44.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.669+0000 7fb213755640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb20c0b2630 con 0x7fb20c0ad370 2023-10-19T13:02:44.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.669+0000 7fb213755640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb20c14e640 con 0x7fb20c06f0a0 2023-10-19T13:02:44.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.669+0000 7fb2114f1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb20c06d2d0 0x7fb20c14e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:44.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.669+0000 7fb2114f1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb20c06d2d0 0x7fb20c14e100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43368/0 (socket says 172.21.15.57:43368) 2023-10-19T13:02:44.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.669+0000 7fb2114f1640 1 -- 172.21.15.57:0/2539312910 learned_addr learned my addr 172.21.15.57:0/2539312910 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:02:44.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.669+0000 7fb210cf0640 1 --2- 172.21.15.57:0/2539312910 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb20c06f0a0 0x7fb20c0cb370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:44.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.669+0000 7fb211cf2640 1 --2- 172.21.15.57:0/2539312910 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb20c0ad370 0x7fb20c0cb8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:44.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.669+0000 7fb2114f1640 1 -- 172.21.15.57:0/2539312910 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb20c06f0a0 msgr2=0x7fb20c0cb370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:44.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.669+0000 7fb2114f1640 1 --2- 172.21.15.57:0/2539312910 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb20c06f0a0 0x7fb20c0cb370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:44.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.669+0000 7fb2114f1640 1 -- 172.21.15.57:0/2539312910 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb20c0ad370 msgr2=0x7fb20c0cb8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:44.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.669+0000 7fb2114f1640 1 --2- 172.21.15.57:0/2539312910 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb20c0ad370 0x7fb20c0cb8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:44.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.669+0000 7fb2114f1640 1 -- 172.21.15.57:0/2539312910 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1fc009490 con 0x7fb20c06d2d0 2023-10-19T13:02:44.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.669+0000 7fb210cf0640 1 --2- 172.21.15.57:0/2539312910 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb20c06f0a0 0x7fb20c0cb370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:02:44.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.669+0000 7fb211cf2640 1 --2- 172.21.15.57:0/2539312910 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb20c0ad370 0x7fb20c0cb8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:02:44.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.669+0000 7fb2114f1640 1 --2- 172.21.15.57:0/2539312910 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb20c06d2d0 0x7fb20c14e100 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7fb20800c4b0 tx=0x7fb20800c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:02:44.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.670+0000 7fb201ffb640 1 -- 172.21.15.57:0/2539312910 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb208005480 con 0x7fb20c06d2d0 2023-10-19T13:02:44.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.670+0000 7fb201ffb640 1 -- 172.21.15.57:0/2539312910 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fb2080036e0 con 0x7fb20c06d2d0 2023-10-19T13:02:44.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.670+0000 7fb213755640 1 -- 172.21.15.57:0/2539312910 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb20c0b1a20 con 0x7fb20c06d2d0 2023-10-19T13:02:44.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.670+0000 7fb201ffb640 1 -- 172.21.15.57:0/2539312910 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb208003b30 con 0x7fb20c06d2d0 2023-10-19T13:02:44.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.670+0000 7fb213755640 1 -- 172.21.15.57:0/2539312910 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb20c0b1fe0 con 0x7fb20c06d2d0 2023-10-19T13:02:44.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.671+0000 7fb213755640 1 -- 172.21.15.57:0/2539312910 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb20c0ce1d0 con 0x7fb20c06d2d0 2023-10-19T13:02:44.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.671+0000 7fb201ffb640 1 -- 172.21.15.57:0/2539312910 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 56) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fb208003c90 con 0x7fb20c06d2d0 2023-10-19T13:02:44.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.672+0000 7fb201ffb640 1 --2- 172.21.15.57:0/2539312910 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7fb1f006acc0 0x7fb1f006d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:44.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.672+0000 7fb201ffb640 1 -- 172.21.15.57:0/2539312910 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6947+0+0 (secure 0 0 0) 0x7fb208088f30 con 0x7fb20c06d2d0 2023-10-19T13:02:44.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.672+0000 7fb210cf0640 1 --2- 172.21.15.57:0/2539312910 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7fb1f006acc0 0x7fb1f006d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:44.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.672+0000 7fb210cf0640 1 --2- 172.21.15.57:0/2539312910 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7fb1f006acc0 0x7fb1f006d180 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fb1fc036000 tx=0x7fb1fc004730 comp rx=0 tx=0).ready entity=mgr.7540 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:02:44.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.676+0000 7fb201ffb640 1 -- 172.21.15.57:0/2539312910 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb208053e40 con 0x7fb20c06d2d0 2023-10-19T13:02:44.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.890+0000 7fb213755640 1 -- 172.21.15.57:0/2539312910 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb20c0690e0 con 0x7fb20c06d2d0 2023-10-19T13:02:44.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.891+0000 7fb201ffb640 1 -- 172.21.15.57:0/2539312910 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+4921 (secure 0 0 0) 0x7fb208017070 con 0x7fb20c06d2d0 2023-10-19T13:02:44.894 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:02:44.894 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":44,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6611,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/2708395824","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2708395824},{"type":"v1","addr":"172.21.15.179:6836","nonce":2708395824}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":41},{"gid":7270,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/3803632191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":3803632191},{"type":"v1","addr":"172.21.15.179:6837","nonce":3803632191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":41},{"gid":7282,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6833/2259979700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2259979700},{"type":"v1","addr":"172.21.15.57:6833","nonce":2259979700}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T13:01:26.118951+0000","modified":"2023-10-19T13:01:28.134083+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8046},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8046":{"gid":8046,"name":"c","rank":0,"incarnation":42,"state":"up:active","state_seq":3,"addr":"172.21.15.57:6835/2556427177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":2556427177},{"type":"v1","addr":"172.21.15.57:6835","nonce":2556427177}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-10-19T13:02:44.895 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 44 2023-10-19T13:02:44.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.893+0000 7fb213755640 1 -- 172.21.15.57:0/2539312910 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7fb1f006acc0 msgr2=0x7fb1f006d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:44.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.893+0000 7fb213755640 1 --2- 172.21.15.57:0/2539312910 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7fb1f006acc0 0x7fb1f006d180 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fb1fc036000 tx=0x7fb1fc004730 comp rx=0 tx=0).stop 2023-10-19T13:02:44.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.893+0000 7fb213755640 1 -- 172.21.15.57:0/2539312910 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb20c06d2d0 msgr2=0x7fb20c14e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:44.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.893+0000 7fb213755640 1 --2- 172.21.15.57:0/2539312910 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb20c06d2d0 0x7fb20c14e100 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7fb20800c4b0 tx=0x7fb20800c970 comp rx=0 tx=0).stop 2023-10-19T13:02:44.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.893+0000 7fb213755640 1 -- 172.21.15.57:0/2539312910 shutdown_connections 2023-10-19T13:02:44.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.893+0000 7fb213755640 1 --2- 172.21.15.57:0/2539312910 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7fb1f006acc0 0x7fb1f006d180 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:44.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.893+0000 7fb213755640 1 --2- 172.21.15.57:0/2539312910 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb20c0ad370 0x7fb20c0cb8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:44.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.893+0000 7fb213755640 1 --2- 172.21.15.57:0/2539312910 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb20c06f0a0 0x7fb20c0cb370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:44.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.893+0000 7fb213755640 1 --2- 172.21.15.57:0/2539312910 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb20c06d2d0 0x7fb20c14e100 unknown :-1 s=CLOSED pgs=1363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:44.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.893+0000 7fb213755640 1 -- 172.21.15.57:0/2539312910 >> 172.21.15.57:0/2539312910 conn(0x7fb20c0b0670 msgr2=0x7fb20c14fea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:02:44.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.893+0000 7fb213755640 1 -- 172.21.15.57:0/2539312910 shutdown_connections 2023-10-19T13:02:44.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:44.893+0000 7fb213755640 1 -- 172.21.15.57:0/2539312910 wait complete. 2023-10-19T13:02:44.906 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T13:02:45.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.013+0000 7f78feb39640 1 -- 172.21.15.57:0/2934094786 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f78f80cf340 msgr2=0x7f78f80d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:45.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.013+0000 7f78feb39640 1 --2- 172.21.15.57:0/2934094786 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f78f80cf340 0x7f78f80d1910 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f78f0005fd0 tx=0x7f78f002c050 comp rx=0 tx=0).stop 2023-10-19T13:02:45.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.013+0000 7f78feb39640 1 -- 172.21.15.57:0/2934094786 shutdown_connections 2023-10-19T13:02:45.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.013+0000 7f78feb39640 1 --2- 172.21.15.57:0/2934094786 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f78f815b2a0 0x7f78f80cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:45.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.013+0000 7f78feb39640 1 --2- 172.21.15.57:0/2934094786 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f78f80cf340 0x7f78f80d1910 unknown :-1 s=CLOSED pgs=1365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:45.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.013+0000 7f78feb39640 1 -- 172.21.15.57:0/2934094786 >> 172.21.15.57:0/2934094786 conn(0x7f78f80d0d60 msgr2=0x7f78f80d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:02:45.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.013+0000 7f78feb39640 1 -- 172.21.15.57:0/2934094786 shutdown_connections 2023-10-19T13:02:45.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.013+0000 7f78feb39640 1 -- 172.21.15.57:0/2934094786 wait complete. 2023-10-19T13:02:45.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.014+0000 7f78feb39640 1 Processor -- start 2023-10-19T13:02:45.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.014+0000 7f78feb39640 1 -- start start 2023-10-19T13:02:45.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.015+0000 7f78feb39640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f78f80ceb40 0x7f78f80c7000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:45.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.015+0000 7f78feb39640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f78f80cf340 0x7f78f80aec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:45.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.015+0000 7f78feb39640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f78f815b2a0 0x7f78f80af1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:45.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.015+0000 7f78feb39640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78f81e3870 con 0x7f78f80ceb40 2023-10-19T13:02:45.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.015+0000 7f78feb39640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78f81e4110 con 0x7f78f815b2a0 2023-10-19T13:02:45.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.015+0000 7f78feb39640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78f80c7540 con 0x7f78f80cf340 2023-10-19T13:02:45.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.015+0000 7f78fc8d5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f78f80ceb40 0x7f78f80c7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:45.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.015+0000 7f78fc8d5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f78f80ceb40 0x7f78f80c7000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43380/0 (socket says 172.21.15.57:43380) 2023-10-19T13:02:45.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.015+0000 7f78fc8d5640 1 -- 172.21.15.57:0/3323923642 learned_addr learned my addr 172.21.15.57:0/3323923642 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:02:45.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.015+0000 7f78fd0d6640 1 --2- 172.21.15.57:0/3323923642 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f78f815b2a0 0x7f78f80af1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:45.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.015+0000 7f78f7fff640 1 --2- 172.21.15.57:0/3323923642 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f78f80cf340 0x7f78f80aec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:45.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.016+0000 7f78fc8d5640 1 -- 172.21.15.57:0/3323923642 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f78f80cf340 msgr2=0x7f78f80aec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:45.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.016+0000 7f78fc8d5640 1 --2- 172.21.15.57:0/3323923642 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f78f80cf340 0x7f78f80aec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:45.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.016+0000 7f78fc8d5640 1 -- 172.21.15.57:0/3323923642 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f78f815b2a0 msgr2=0x7f78f80af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:45.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.016+0000 7f78fc8d5640 1 --2- 172.21.15.57:0/3323923642 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f78f815b2a0 0x7f78f80af1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:45.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.016+0000 7f78fc8d5640 1 -- 172.21.15.57:0/3323923642 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78f0009840 con 0x7f78f80ceb40 2023-10-19T13:02:45.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.016+0000 7f78fc8d5640 1 --2- 172.21.15.57:0/3323923642 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f78f80ceb40 0x7f78f80c7000 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f78f0009cd0 tx=0x7f78f00042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:02:45.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.016+0000 7f78f57fa640 1 -- 172.21.15.57:0/3323923642 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78f003e460 con 0x7f78f80ceb40 2023-10-19T13:02:45.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.016+0000 7f78f57fa640 1 -- 172.21.15.57:0/3323923642 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f78f003e6b0 con 0x7f78f80ceb40 2023-10-19T13:02:45.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.016+0000 7f78feb39640 1 -- 172.21.15.57:0/3323923642 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78f815e670 con 0x7f78f80ceb40 2023-10-19T13:02:45.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.016+0000 7f78f57fa640 1 -- 172.21.15.57:0/3323923642 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78f003eac0 con 0x7f78f80ceb40 2023-10-19T13:02:45.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.016+0000 7f78feb39640 1 -- 172.21.15.57:0/3323923642 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78f81e3da0 con 0x7f78f80ceb40 2023-10-19T13:02:45.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.017+0000 7f78feb39640 1 -- 172.21.15.57:0/3323923642 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78f80690e0 con 0x7f78f80ceb40 2023-10-19T13:02:45.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.017+0000 7f78f57fa640 1 -- 172.21.15.57:0/3323923642 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 56) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f78f003ec20 con 0x7f78f80ceb40 2023-10-19T13:02:45.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.018+0000 7f78f57fa640 1 --2- 172.21.15.57:0/3323923642 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f78e406ac90 0x7f78e406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:45.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.018+0000 7f78f57fa640 1 -- 172.21.15.57:0/3323923642 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6947+0+0 (secure 0 0 0) 0x7f78f00ac870 con 0x7f78f80ceb40 2023-10-19T13:02:45.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.018+0000 7f78f7fff640 1 --2- 172.21.15.57:0/3323923642 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f78e406ac90 0x7f78e406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:45.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.019+0000 7f78f7fff640 1 --2- 172.21.15.57:0/3323923642 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f78e406ac90 0x7f78e406d150 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f78ec005fd0 tx=0x7f78ec005850 comp rx=0 tx=0).ready entity=mgr.7540 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:02:45.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.022+0000 7f78f57fa640 1 -- 172.21.15.57:0/3323923642 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f78f0077700 con 0x7f78f80ceb40 2023-10-19T13:02:45.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.236+0000 7f78feb39640 1 -- 172.21.15.57:0/3323923642 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f78f80b08e0 con 0x7f78f80ceb40 2023-10-19T13:02:45.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.236+0000 7f78f57fa640 1 -- 172.21.15.57:0/3323923642 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+4921 (secure 0 0 0) 0x7f78f007ad20 con 0x7f78f80ceb40 2023-10-19T13:02:45.240 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:02:45.240 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":44,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6611,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/2708395824","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2708395824},{"type":"v1","addr":"172.21.15.179:6836","nonce":2708395824}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":41},{"gid":7270,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/3803632191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":3803632191},{"type":"v1","addr":"172.21.15.179:6837","nonce":3803632191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":41},{"gid":7282,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6833/2259979700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2259979700},{"type":"v1","addr":"172.21.15.57:6833","nonce":2259979700}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T13:01:26.118951+0000","modified":"2023-10-19T13:01:28.134083+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8046},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8046":{"gid":8046,"name":"c","rank":0,"incarnation":42,"state":"up:active","state_seq":3,"addr":"172.21.15.57:6835/2556427177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":2556427177},{"type":"v1","addr":"172.21.15.57:6835","nonce":2556427177}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-10-19T13:02:45.241 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 44 2023-10-19T13:02:45.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.238+0000 7f78feb39640 1 -- 172.21.15.57:0/3323923642 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f78e406ac90 msgr2=0x7f78e406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:45.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.238+0000 7f78feb39640 1 --2- 172.21.15.57:0/3323923642 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f78e406ac90 0x7f78e406d150 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f78ec005fd0 tx=0x7f78ec005850 comp rx=0 tx=0).stop 2023-10-19T13:02:45.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.238+0000 7f78feb39640 1 -- 172.21.15.57:0/3323923642 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f78f80ceb40 msgr2=0x7f78f80c7000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:45.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.238+0000 7f78feb39640 1 --2- 172.21.15.57:0/3323923642 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f78f80ceb40 0x7f78f80c7000 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f78f0009cd0 tx=0x7f78f00042f0 comp rx=0 tx=0).stop 2023-10-19T13:02:45.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.239+0000 7f78feb39640 1 -- 172.21.15.57:0/3323923642 shutdown_connections 2023-10-19T13:02:45.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.239+0000 7f78feb39640 1 --2- 172.21.15.57:0/3323923642 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f78e406ac90 0x7f78e406d150 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:45.243 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.239+0000 7f78feb39640 1 --2- 172.21.15.57:0/3323923642 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f78f815b2a0 0x7f78f80af1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:45.243 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.239+0000 7f78feb39640 1 --2- 172.21.15.57:0/3323923642 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f78f80cf340 0x7f78f80aec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:45.243 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.239+0000 7f78feb39640 1 --2- 172.21.15.57:0/3323923642 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f78f80ceb40 0x7f78f80c7000 unknown :-1 s=CLOSED pgs=1366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:45.243 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.239+0000 7f78feb39640 1 -- 172.21.15.57:0/3323923642 >> 172.21.15.57:0/3323923642 conn(0x7f78f80d0d60 msgr2=0x7f78f80ae280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:02:45.243 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.239+0000 7f78feb39640 1 -- 172.21.15.57:0/3323923642 shutdown_connections 2023-10-19T13:02:45.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.239+0000 7f78feb39640 1 -- 172.21.15.57:0/3323923642 wait complete. 2023-10-19T13:02:45.252 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T13:02:45.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.358+0000 7f08f32d7640 1 -- 172.21.15.57:0/3521695630 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f08ec156790 msgr2=0x7f08ec0cc990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:45.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.358+0000 7f08f32d7640 1 --2- 172.21.15.57:0/3521695630 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f08ec156790 0x7f08ec0cc990 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f08ec065ed0 tx=0x7f08e802bd90 comp rx=0 tx=0).stop 2023-10-19T13:02:45.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.358+0000 7f08f32d7640 1 -- 172.21.15.57:0/3521695630 shutdown_connections 2023-10-19T13:02:45.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.358+0000 7f08f32d7640 1 --2- 172.21.15.57:0/3521695630 >> v2:172.21.15.179:3300/0 conn(0x7f08ec06a730 0x7f08ec06abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:45.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.358+0000 7f08f32d7640 1 --2- 172.21.15.57:0/3521695630 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f08ec156790 0x7f08ec0cc990 unknown :-1 s=CLOSED pgs=1367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:45.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.358+0000 7f08f32d7640 1 -- 172.21.15.57:0/3521695630 >> 172.21.15.57:0/3521695630 conn(0x7f08ec06fae0 msgr2=0x7f08ec06fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:02:45.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.358+0000 7f08f32d7640 1 -- 172.21.15.57:0/3521695630 shutdown_connections 2023-10-19T13:02:45.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.358+0000 7f08f32d7640 1 -- 172.21.15.57:0/3521695630 wait complete. 2023-10-19T13:02:45.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.359+0000 7f08f32d7640 1 Processor -- start 2023-10-19T13:02:45.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.359+0000 7f08f32d7640 1 -- start start 2023-10-19T13:02:45.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.360+0000 7f08f32d7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f08ec06a730 0x7f08ec15b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:45.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.360+0000 7f08f32d7640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f08ec0ac870 0x7f08ec15a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:45.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.360+0000 7f08f32d7640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f08ec156790 0x7f08ec13d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:45.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.360+0000 7f08f32d7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08ec152d70 con 0x7f08ec06a730 2023-10-19T13:02:45.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.360+0000 7f08f32d7640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08ec153050 con 0x7f08ec0ac870 2023-10-19T13:02:45.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.360+0000 7f08f32d7640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08ec159d30 con 0x7f08ec156790 2023-10-19T13:02:45.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.360+0000 7f08f1073640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f08ec06a730 0x7f08ec15b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:45.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.360+0000 7f08f1073640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f08ec06a730 0x7f08ec15b410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43396/0 (socket says 172.21.15.57:43396) 2023-10-19T13:02:45.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.360+0000 7f08f1073640 1 -- 172.21.15.57:0/4058214253 learned_addr learned my addr 172.21.15.57:0/4058214253 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:02:45.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.360+0000 7f08f1874640 1 --2- 172.21.15.57:0/4058214253 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f08ec156790 0x7f08ec13d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:45.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.360+0000 7f08f0872640 1 --2- 172.21.15.57:0/4058214253 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f08ec0ac870 0x7f08ec15a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:45.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.361+0000 7f08f1073640 1 -- 172.21.15.57:0/4058214253 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f08ec156790 msgr2=0x7f08ec13d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:45.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.361+0000 7f08f1073640 1 --2- 172.21.15.57:0/4058214253 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f08ec156790 0x7f08ec13d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:45.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.361+0000 7f08f1073640 1 -- 172.21.15.57:0/4058214253 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f08ec0ac870 msgr2=0x7f08ec15a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:45.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.361+0000 7f08f1073640 1 --2- 172.21.15.57:0/4058214253 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f08ec0ac870 0x7f08ec15a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:45.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.361+0000 7f08f1073640 1 -- 172.21.15.57:0/4058214253 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08e8009560 con 0x7f08ec06a730 2023-10-19T13:02:45.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.361+0000 7f08f1874640 1 --2- 172.21.15.57:0/4058214253 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f08ec156790 0x7f08ec13d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:02:45.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.361+0000 7f08f1073640 1 --2- 172.21.15.57:0/4058214253 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f08ec06a730 0x7f08ec15b410 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f08e8036040 tx=0x7f08e802db30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:02:45.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.361+0000 7f08e1ffb640 1 -- 172.21.15.57:0/4058214253 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08e80035f0 con 0x7f08ec06a730 2023-10-19T13:02:45.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.361+0000 7f08e1ffb640 1 -- 172.21.15.57:0/4058214253 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f08e8003840 con 0x7f08ec06a730 2023-10-19T13:02:45.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.361+0000 7f08f32d7640 1 -- 172.21.15.57:0/4058214253 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08ec15a010 con 0x7f08ec06a730 2023-10-19T13:02:45.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.361+0000 7f08e1ffb640 1 -- 172.21.15.57:0/4058214253 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08e8003bf0 con 0x7f08ec06a730 2023-10-19T13:02:45.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.361+0000 7f08f32d7640 1 -- 172.21.15.57:0/4058214253 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08ec15b950 con 0x7f08ec06a730 2023-10-19T13:02:45.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.362+0000 7f08f32d7640 1 -- 172.21.15.57:0/4058214253 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08ec15a530 con 0x7f08ec06a730 2023-10-19T13:02:45.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.363+0000 7f08e1ffb640 1 -- 172.21.15.57:0/4058214253 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 56) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f08e8003d50 con 0x7f08ec06a730 2023-10-19T13:02:45.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.363+0000 7f08e1ffb640 1 --2- 172.21.15.57:0/4058214253 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f08d006ac90 0x7f08d006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:45.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.363+0000 7f08e1ffb640 1 -- 172.21.15.57:0/4058214253 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6947+0+0 (secure 0 0 0) 0x7f08e80ac370 con 0x7f08ec06a730 2023-10-19T13:02:45.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.363+0000 7f08f0872640 1 --2- 172.21.15.57:0/4058214253 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f08d006ac90 0x7f08d006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:45.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.364+0000 7f08f0872640 1 --2- 172.21.15.57:0/4058214253 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f08d006ac90 0x7f08d006d150 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f08ec06b090 tx=0x7f08dc009040 comp rx=0 tx=0).ready entity=mgr.7540 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:02:45.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.367+0000 7f08e1ffb640 1 -- 172.21.15.57:0/4058214253 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f08e80772b0 con 0x7f08ec06a730 2023-10-19T13:02:45.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.543+0000 7f08f32d7640 1 -- 172.21.15.57:0/4058214253 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f08ec0690e0 con 0x7f08ec06a730 2023-10-19T13:02:45.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.544+0000 7f08e1ffb640 1 -- 172.21.15.57:0/4058214253 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v118) v1 ==== 74+0+18117 (secure 0 0 0) 0x7f08e807a8d0 con 0x7f08ec06a730 2023-10-19T13:02:45.547 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:02:45.548 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":118,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T13:02:43.621627+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2023-10-19T13:01:24.443125+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2023-10-19T13:01:25.101276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"113","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.1","mappings":[{"from":1,"to":5}]},{"pgid":"14.12","mappings":[{"from":1,"to":7}]},{"pgid":"14.2d","mappings":[{"from":1,"to":7}]},{"pgid":"14.32","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.179:0/2599642592":"2023-10-20T13:02:43.621587+0000","172.21.15.179:0/2915428574":"2023-10-20T13:02:43.621587+0000","172.21.15.179:0/356140706":"2023-10-20T13:02:43.621587+0000","172.21.15.179:0/4203351642":"2023-10-20T13:02:43.621587+0000","172.21.15.179:0/842129505":"2023-10-20T13:02:43.621587+0000","172.21.15.179:0/1511457369":"2023-10-20T13:02:43.621587+0000","172.21.15.179:6833/2366702417":"2023-10-20T13:02:43.621587+0000","172.21.15.179:6832/2366702417":"2023-10-20T13:02:43.621587+0000","172.21.15.179:0/3304791412":"2023-10-19T14:01:39.552724+0000","172.21.15.179:0/3109348805":"2023-10-20T13:01:37.504869+0000","172.21.15.179:0/2984488835":"2023-10-20T13:01:37.504869+0000","172.21.15.179:0/205205608":"2023-10-20T13:01:37.504869+0000","172.21.15.179:0/4112649564":"2023-10-20T13:01:37.504869+0000","172.21.15.179:6833/3281457486":"2023-10-20T13:01:37.504869+0000","172.21.15.179:0/1633718057":"2023-10-20T13:02:43.621587+0000","172.21.15.179:6832/3281457486":"2023-10-20T13:01:37.504869+0000","172.21.15.179:0/318926584":"2023-10-20T13:02:43.621587+0000","172.21.15.179:0/2509136709":"2023-10-20T13:01:37.504869+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T13:02:45.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.546+0000 7f08f32d7640 1 -- 172.21.15.57:0/4058214253 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f08d006ac90 msgr2=0x7f08d006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:45.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.546+0000 7f08f32d7640 1 --2- 172.21.15.57:0/4058214253 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f08d006ac90 0x7f08d006d150 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f08ec06b090 tx=0x7f08dc009040 comp rx=0 tx=0).stop 2023-10-19T13:02:45.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.546+0000 7f08f32d7640 1 -- 172.21.15.57:0/4058214253 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f08ec06a730 msgr2=0x7f08ec15b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:45.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.546+0000 7f08f32d7640 1 --2- 172.21.15.57:0/4058214253 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f08ec06a730 0x7f08ec15b410 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f08e8036040 tx=0x7f08e802db30 comp rx=0 tx=0).stop 2023-10-19T13:02:45.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.546+0000 7f08f32d7640 1 -- 172.21.15.57:0/4058214253 shutdown_connections 2023-10-19T13:02:45.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.546+0000 7f08f32d7640 1 --2- 172.21.15.57:0/4058214253 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f08d006ac90 0x7f08d006d150 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:45.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.546+0000 7f08f32d7640 1 --2- 172.21.15.57:0/4058214253 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f08ec156790 0x7f08ec13d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:45.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.546+0000 7f08f32d7640 1 --2- 172.21.15.57:0/4058214253 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f08ec0ac870 0x7f08ec15a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:45.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.546+0000 7f08f32d7640 1 --2- 172.21.15.57:0/4058214253 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f08ec06a730 0x7f08ec15b410 unknown :-1 s=CLOSED pgs=1368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:45.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.546+0000 7f08f32d7640 1 -- 172.21.15.57:0/4058214253 >> 172.21.15.57:0/4058214253 conn(0x7f08ec06fae0 msgr2=0x7f08ec065490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:02:45.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.546+0000 7f08f32d7640 1 -- 172.21.15.57:0/4058214253 shutdown_connections 2023-10-19T13:02:45.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.546+0000 7f08f32d7640 1 -- 172.21.15.57:0/4058214253 wait complete. 2023-10-19T13:02:45.560 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-19T13:02:45.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.669+0000 7f1203983640 1 -- 172.21.15.57:0/3584566799 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f11fc158910 msgr2=0x7f11fc158cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:45.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.669+0000 7f1203983640 1 --2- 172.21.15.57:0/3584566799 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f11fc158910 0x7f11fc158cf0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f11f8009860 tx=0x7f11f802bde0 comp rx=0 tx=0).stop 2023-10-19T13:02:45.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.669+0000 7f1203983640 1 -- 172.21.15.57:0/3584566799 shutdown_connections 2023-10-19T13:02:45.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.669+0000 7f1203983640 1 --2- 172.21.15.57:0/3584566799 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f11fc158910 0x7f11fc158cf0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:45.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.669+0000 7f1203983640 1 -- 172.21.15.57:0/3584566799 >> 172.21.15.57:0/3584566799 conn(0x7f11fc0af7a0 msgr2=0x7f11fc0afb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:02:45.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.669+0000 7f1203983640 1 -- 172.21.15.57:0/3584566799 shutdown_connections 2023-10-19T13:02:45.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.669+0000 7f1203983640 1 -- 172.21.15.57:0/3584566799 wait complete. 2023-10-19T13:02:45.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.670+0000 7f1203983640 1 Processor -- start 2023-10-19T13:02:45.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.670+0000 7f1203983640 1 -- start start 2023-10-19T13:02:45.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.671+0000 7f1203983640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f11fc0cd1c0 0x7f11fc14a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:45.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.671+0000 7f1203983640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f11fc0cd9e0 0x7f11fc1e3100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:45.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.671+0000 7f1203983640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f11fc158910 0x7f11fc14b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:45.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.671+0000 7f1203983640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11fc1e3fc0 con 0x7f11fc158910 2023-10-19T13:02:45.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.671+0000 7f1203983640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11fc14ab10 con 0x7f11fc0cd9e0 2023-10-19T13:02:45.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.671+0000 7f1203983640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11fc1e3a00 con 0x7f11fc0cd1c0 2023-10-19T13:02:45.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.671+0000 7f1201f20640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f11fc158910 0x7f11fc14b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:45.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.671+0000 7f1201f20640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f11fc158910 0x7f11fc14b270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43412/0 (socket says 172.21.15.57:43412) 2023-10-19T13:02:45.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.671+0000 7f1201f20640 1 -- 172.21.15.57:0/3814694101 learned_addr learned my addr 172.21.15.57:0/3814694101 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:02:45.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.671+0000 7f120171f640 1 --2- 172.21.15.57:0/3814694101 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f11fc0cd1c0 0x7f11fc14a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:45.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.671+0000 7f1200f1e640 1 --2- 172.21.15.57:0/3814694101 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f11fc0cd9e0 0x7f11fc1e3100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:45.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.672+0000 7f1201f20640 1 -- 172.21.15.57:0/3814694101 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f11fc0cd1c0 msgr2=0x7f11fc14a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:45.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.672+0000 7f1201f20640 1 --2- 172.21.15.57:0/3814694101 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f11fc0cd1c0 0x7f11fc14a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:45.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.672+0000 7f1201f20640 1 -- 172.21.15.57:0/3814694101 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f11fc0cd9e0 msgr2=0x7f11fc1e3100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:45.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.672+0000 7f1201f20640 1 --2- 172.21.15.57:0/3814694101 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f11fc0cd9e0 0x7f11fc1e3100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:45.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.672+0000 7f1201f20640 1 -- 172.21.15.57:0/3814694101 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11f8009490 con 0x7f11fc158910 2023-10-19T13:02:45.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.672+0000 7f1201f20640 1 --2- 172.21.15.57:0/3814694101 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f11fc158910 0x7f11fc14b270 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7f11f400c790 tx=0x7f11f400cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:02:45.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.672+0000 7f11f1ffb640 1 -- 172.21.15.57:0/3814694101 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f400cea0 con 0x7f11fc158910 2023-10-19T13:02:45.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.672+0000 7f11f1ffb640 1 -- 172.21.15.57:0/3814694101 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f11f4003b70 con 0x7f11fc158910 2023-10-19T13:02:45.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.672+0000 7f1203983640 1 -- 172.21.15.57:0/3814694101 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11fc1e3ce0 con 0x7f11fc158910 2023-10-19T13:02:45.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.672+0000 7f11f1ffb640 1 -- 172.21.15.57:0/3814694101 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f400a850 con 0x7f11fc158910 2023-10-19T13:02:45.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.672+0000 7f1203983640 1 -- 172.21.15.57:0/3814694101 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11fc152d20 con 0x7f11fc158910 2023-10-19T13:02:45.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.673+0000 7f1203983640 1 -- 172.21.15.57:0/3814694101 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11fc152f40 con 0x7f11fc158910 2023-10-19T13:02:45.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.674+0000 7f11f1ffb640 1 -- 172.21.15.57:0/3814694101 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 56) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f11f400a9b0 con 0x7f11fc158910 2023-10-19T13:02:45.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.674+0000 7f11f1ffb640 1 --2- 172.21.15.57:0/3814694101 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f11e006ac90 0x7f11e006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:45.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.674+0000 7f11f1ffb640 1 -- 172.21.15.57:0/3814694101 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6947+0+0 (secure 0 0 0) 0x7f11f4089700 con 0x7f11fc158910 2023-10-19T13:02:45.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.674+0000 7f120171f640 1 --2- 172.21.15.57:0/3814694101 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f11e006ac90 0x7f11e006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:45.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.675+0000 7f120171f640 1 --2- 172.21.15.57:0/3814694101 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f11e006ac90 0x7f11e006d150 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f11f8036000 tx=0x7f11f8004730 comp rx=0 tx=0).ready entity=mgr.7540 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:02:45.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.678+0000 7f11f1ffb640 1 -- 172.21.15.57:0/3814694101 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f11f40545f0 con 0x7f11fc158910 2023-10-19T13:02:45.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.893+0000 7f1203983640 1 -- 172.21.15.57:0/3814694101 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f11fc0690e0 con 0x7f11fc158910 2023-10-19T13:02:45.897 INFO:tasks.ceph.mon.a.smithi057.stderr:2023-10-19T13:02:45.894+0000 7f1253ff3640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-19T13:02:45.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.899+0000 7f11f1ffb640 1 -- 172.21.15.57:0/3814694101 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v45) v1 ==== 161+0+0 (secure 0 0 0) 0x7f11f4057c10 con 0x7f11fc158910 2023-10-19T13:02:45.902 INFO:teuthology.orchestra.run.smithi057.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-19T13:02:45.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.902+0000 7f1203983640 1 -- 172.21.15.57:0/3814694101 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f11e006ac90 msgr2=0x7f11e006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:45.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.902+0000 7f1203983640 1 --2- 172.21.15.57:0/3814694101 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f11e006ac90 0x7f11e006d150 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f11f8036000 tx=0x7f11f8004730 comp rx=0 tx=0).stop 2023-10-19T13:02:45.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.902+0000 7f1203983640 1 -- 172.21.15.57:0/3814694101 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f11fc158910 msgr2=0x7f11fc14b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:45.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.902+0000 7f1203983640 1 --2- 172.21.15.57:0/3814694101 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f11fc158910 0x7f11fc14b270 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7f11f400c790 tx=0x7f11f400cc50 comp rx=0 tx=0).stop 2023-10-19T13:02:45.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.903+0000 7f1203983640 1 -- 172.21.15.57:0/3814694101 shutdown_connections 2023-10-19T13:02:45.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.903+0000 7f1203983640 1 --2- 172.21.15.57:0/3814694101 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f11e006ac90 0x7f11e006d150 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:45.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.903+0000 7f1203983640 1 --2- 172.21.15.57:0/3814694101 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f11fc158910 0x7f11fc14b270 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:45.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.903+0000 7f1203983640 1 --2- 172.21.15.57:0/3814694101 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f11fc0cd9e0 0x7f11fc1e3100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:45.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.903+0000 7f1203983640 1 --2- 172.21.15.57:0/3814694101 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f11fc0cd1c0 0x7f11fc14a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:45.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.903+0000 7f1203983640 1 -- 172.21.15.57:0/3814694101 >> 172.21.15.57:0/3814694101 conn(0x7f11fc0af7a0 msgr2=0x7f11fc14c370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:02:45.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.903+0000 7f1203983640 1 -- 172.21.15.57:0/3814694101 shutdown_connections 2023-10-19T13:02:45.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:45.903+0000 7f1203983640 1 -- 172.21.15.57:0/3814694101 wait complete. 2023-10-19T13:02:45.909 INFO:tasks.ceph.mds.c.smithi057.stderr: -5877> 2023-10-19T13:01:32.736+0000 7f57a1c4a640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-19T13:02:45.918 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-19T13:02:45.938 INFO:tasks.ceph.mds.c.smithi057.stdout:starting mds.c at 2023-10-19T13:02:46.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:46.028+0000 7f4c2845d640 1 -- 172.21.15.57:0/665060041 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4c20141690 msgr2=0x7f4c200703e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:46.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:46.028+0000 7f4c2845d640 1 --2- 172.21.15.57:0/665060041 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4c20141690 0x7f4c200703e0 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f4c200cbd00 tx=0x7f4c1402bd10 comp rx=0 tx=0).stop 2023-10-19T13:02:46.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:46.029+0000 7f4c2845d640 1 -- 172.21.15.57:0/665060041 shutdown_connections 2023-10-19T13:02:46.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:46.029+0000 7f4c2845d640 1 --2- 172.21.15.57:0/665060041 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4c2006c0a0 0x7f4c20070920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:46.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:46.029+0000 7f4c2845d640 1 --2- 172.21.15.57:0/665060041 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4c20141690 0x7f4c200703e0 unknown :-1 s=CLOSED pgs=1374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:46.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:46.029+0000 7f4c2845d640 1 --2- 172.21.15.57:0/665060041 >> v2:172.21.15.179:3300/0 conn(0x7f4c2013e520 0x7f4c2006bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:46.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:46.029+0000 7f4c2845d640 1 -- 172.21.15.57:0/665060041 >> 172.21.15.57:0/665060041 conn(0x7f4c20069930 msgr2=0x7f4c20069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:02:46.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:46.029+0000 7f4c2845d640 1 -- 172.21.15.57:0/665060041 shutdown_connections 2023-10-19T13:02:46.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:46.029+0000 7f4c2845d640 1 -- 172.21.15.57:0/665060041 wait complete. 2023-10-19T13:02:46.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:46.030+0000 7f4c2845d640 1 Processor -- start 2023-10-19T13:02:46.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:46.030+0000 7f4c2845d640 1 -- start start 2023-10-19T13:02:46.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:46.031+0000 7f4c2845d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4c2006c0a0 0x7f4c200c7ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:46.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:46.031+0000 7f4c2845d640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4c2013e520 0x7f4c200a7e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:46.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:46.031+0000 7f4c2845d640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4c20141690 0x7f4c200c6010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:46.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:46.031+0000 7f4c2845d640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c200c8ed0 con 0x7f4c2006c0a0 2023-10-19T13:02:46.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:46.031+0000 7f4c2845d640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c200c82c0 con 0x7f4c2013e520 2023-10-19T13:02:46.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:46.031+0000 7f4c2845d640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c200c85a0 con 0x7f4c20141690 2023-10-19T13:02:46.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:46.031+0000 7f4c261f9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4c2006c0a0 0x7f4c200c7ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:46.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:46.031+0000 7f4c261f9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4c2006c0a0 0x7f4c200c7ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43448/0 (socket says 172.21.15.57:43448) 2023-10-19T13:02:46.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:46.031+0000 7f4c261f9640 1 -- 172.21.15.57:0/2678316834 learned_addr learned my addr 172.21.15.57:0/2678316834 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:02:46.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:46.031+0000 7f4c269fa640 1 --2- 172.21.15.57:0/2678316834 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4c20141690 0x7f4c200c6010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:46.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:46.031+0000 7f4c259f8640 1 --2- 172.21.15.57:0/2678316834 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4c2013e520 0x7f4c200a7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:46.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:46.031+0000 7f4c261f9640 1 -- 172.21.15.57:0/2678316834 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4c20141690 msgr2=0x7f4c200c6010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:46.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:46.031+0000 7f4c261f9640 1 --2- 172.21.15.57:0/2678316834 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4c20141690 0x7f4c200c6010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:46.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:46.031+0000 7f4c261f9640 1 -- 172.21.15.57:0/2678316834 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4c2013e520 msgr2=0x7f4c200a7e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:46.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:46.031+0000 7f4c261f9640 1 --2- 172.21.15.57:0/2678316834 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4c2013e520 0x7f4c200a7e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:46.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:46.031+0000 7f4c261f9640 1 -- 172.21.15.57:0/2678316834 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c14009560 con 0x7f4c2006c0a0 2023-10-19T13:02:46.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:46.031+0000 7f4c259f8640 1 --2- 172.21.15.57:0/2678316834 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4c2013e520 0x7f4c200a7e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T13:02:46.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:46.031+0000 7f4c269fa640 1 --2- 172.21.15.57:0/2678316834 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4c20141690 0x7f4c200c6010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T13:02:46.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:46.031+0000 7f4c261f9640 1 --2- 172.21.15.57:0/2678316834 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4c2006c0a0 0x7f4c200c7ce0 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f4c1800d950 tx=0x7f4c1800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:02:46.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:46.031+0000 7f4c1effd640 1 -- 172.21.15.57:0/2678316834 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c18009c30 con 0x7f4c2006c0a0 2023-10-19T13:02:46.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:46.031+0000 7f4c1effd640 1 -- 172.21.15.57:0/2678316834 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f4c18009e80 con 0x7f4c2006c0a0 2023-10-19T13:02:46.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:46.031+0000 7f4c1effd640 1 -- 172.21.15.57:0/2678316834 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c18004280 con 0x7f4c2006c0a0 2023-10-19T13:02:46.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:46.031+0000 7f4c2845d640 1 -- 172.21.15.57:0/2678316834 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c200c8880 con 0x7f4c2006c0a0 2023-10-19T13:02:46.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:46.031+0000 7f4c2845d640 1 -- 172.21.15.57:0/2678316834 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c200a8340 con 0x7f4c2006c0a0 2023-10-19T13:02:46.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:46.032+0000 7f4c2845d640 1 -- 172.21.15.57:0/2678316834 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c200690e0 con 0x7f4c2006c0a0 2023-10-19T13:02:46.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:46.033+0000 7f4c1effd640 1 -- 172.21.15.57:0/2678316834 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 56) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f4c180035f0 con 0x7f4c2006c0a0 2023-10-19T13:02:46.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:46.033+0000 7f4c1effd640 1 --2- 172.21.15.57:0/2678316834 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f4c0c06acc0 0x7f4c0c06d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:46.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:46.033+0000 7f4c1effd640 1 -- 172.21.15.57:0/2678316834 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 7033+0+0 (secure 0 0 0) 0x7f4c180899c0 con 0x7f4c2006c0a0 2023-10-19T13:02:46.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:46.034+0000 7f4c259f8640 1 --2- 172.21.15.57:0/2678316834 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f4c0c06acc0 0x7f4c0c06d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:46.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:46.034+0000 7f4c259f8640 1 --2- 172.21.15.57:0/2678316834 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f4c0c06acc0 0x7f4c0c06d180 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4c14035000 tx=0x7f4c14004730 comp rx=0 tx=0).ready entity=mgr.7540 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:02:46.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:46.037+0000 7f4c1effd640 1 -- 172.21.15.57:0/2678316834 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4c180547f0 con 0x7f4c2006c0a0 2023-10-19T13:02:46.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:46.248+0000 7f4c2845d640 1 -- 172.21.15.57:0/2678316834 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f4c20072890 con 0x7f4c2006c0a0 2023-10-19T13:02:46.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:46.268+0000 7f4c1effd640 1 -- 172.21.15.57:0/2678316834 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mgrmap(e 57) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f4c18054bc0 con 0x7f4c2006c0a0 2023-10-19T13:02:46.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:46.905+0000 7f4c1effd640 1 -- 172.21.15.57:0/2678316834 <== mon.0 v2:172.21.15.57:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v46) v1 ==== 104+0+0 (secure 0 0 0) 0x7f4c180879f0 con 0x7f4c2006c0a0 2023-10-19T13:02:46.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:46.909+0000 7f4c2845d640 1 -- 172.21.15.57:0/2678316834 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f4c0c06acc0 msgr2=0x7f4c0c06d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:46.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:46.909+0000 7f4c2845d640 1 --2- 172.21.15.57:0/2678316834 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f4c0c06acc0 0x7f4c0c06d180 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4c14035000 tx=0x7f4c14004730 comp rx=0 tx=0).stop 2023-10-19T13:02:46.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:46.909+0000 7f4c2845d640 1 -- 172.21.15.57:0/2678316834 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4c2006c0a0 msgr2=0x7f4c200c7ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:46.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:46.909+0000 7f4c2845d640 1 --2- 172.21.15.57:0/2678316834 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4c2006c0a0 0x7f4c200c7ce0 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f4c1800d950 tx=0x7f4c1800de10 comp rx=0 tx=0).stop 2023-10-19T13:02:46.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:46.909+0000 7f4c2845d640 1 -- 172.21.15.57:0/2678316834 shutdown_connections 2023-10-19T13:02:46.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:46.909+0000 7f4c2845d640 1 --2- 172.21.15.57:0/2678316834 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f4c0c06acc0 0x7f4c0c06d180 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:46.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:46.909+0000 7f4c2845d640 1 --2- 172.21.15.57:0/2678316834 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4c20141690 0x7f4c200c6010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:46.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:46.909+0000 7f4c2845d640 1 --2- 172.21.15.57:0/2678316834 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4c2013e520 0x7f4c200a7e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:46.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:46.909+0000 7f4c2845d640 1 --2- 172.21.15.57:0/2678316834 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4c2006c0a0 0x7f4c200c7ce0 unknown :-1 s=CLOSED pgs=1375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:46.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:46.909+0000 7f4c2845d640 1 -- 172.21.15.57:0/2678316834 >> 172.21.15.57:0/2678316834 conn(0x7f4c20069930 msgr2=0x7f4c200adea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:02:46.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:46.909+0000 7f4c2845d640 1 -- 172.21.15.57:0/2678316834 shutdown_connections 2023-10-19T13:02:46.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:46.909+0000 7f4c2845d640 1 -- 172.21.15.57:0/2678316834 wait complete. 2023-10-19T13:02:46.925 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-19T13:02:47.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:47.083+0000 7fc95c7a5640 1 -- 172.21.15.57:0/1394579081 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc95415b2a0 msgr2=0x7fc9540cf720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:47.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:47.083+0000 7fc95c7a5640 1 --2- 172.21.15.57:0/1394579081 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc95415b2a0 0x7fc9540cf720 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7fc948009830 tx=0x7fc94802bcd0 comp rx=0 tx=0).stop 2023-10-19T13:02:47.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:47.083+0000 7fc95c7a5640 1 -- 172.21.15.57:0/1394579081 shutdown_connections 2023-10-19T13:02:47.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:47.083+0000 7fc95c7a5640 1 --2- 172.21.15.57:0/1394579081 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc95415b2a0 0x7fc9540cf720 unknown :-1 s=CLOSED pgs=1377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:47.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:47.083+0000 7fc95c7a5640 1 --2- 172.21.15.57:0/1394579081 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc9540cf340 0x7fc9540d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:47.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:47.083+0000 7fc95c7a5640 1 -- 172.21.15.57:0/1394579081 >> 172.21.15.57:0/1394579081 conn(0x7fc9540d0d60 msgr2=0x7fc9540d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:02:47.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:47.083+0000 7fc95c7a5640 1 -- 172.21.15.57:0/1394579081 shutdown_connections 2023-10-19T13:02:47.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:47.083+0000 7fc95c7a5640 1 -- 172.21.15.57:0/1394579081 wait complete. 2023-10-19T13:02:47.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:47.084+0000 7fc95c7a5640 1 Processor -- start 2023-10-19T13:02:47.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:47.085+0000 7fc95c7a5640 1 -- start start 2023-10-19T13:02:47.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:47.085+0000 7fc95c7a5640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc9540ceb40 0x7fc9541585a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:47.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:47.085+0000 7fc95c7a5640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc9540cf340 0x7fc9540c9560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:47.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:47.085+0000 7fc95c7a5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc95415b2a0 0x7fc9540c7e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:47.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:47.085+0000 7fc95c7a5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9541583a0 con 0x7fc95415b2a0 2023-10-19T13:02:47.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:47.085+0000 7fc95c7a5640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9540c84c0 con 0x7fc9540cf340 2023-10-19T13:02:47.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:47.085+0000 7fc95c7a5640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9540c87a0 con 0x7fc9540ceb40 2023-10-19T13:02:47.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:47.085+0000 7fc95ad42640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc95415b2a0 0x7fc9540c7e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:47.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:47.085+0000 7fc95ad42640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc95415b2a0 0x7fc9540c7e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43278/0 (socket says 172.21.15.57:43278) 2023-10-19T13:02:47.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:47.085+0000 7fc95ad42640 1 -- 172.21.15.57:0/2249342518 learned_addr learned my addr 172.21.15.57:0/2249342518 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:02:47.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:47.086+0000 7fc95a541640 1 --2- 172.21.15.57:0/2249342518 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc9540ceb40 0x7fc9541585a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:47.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:47.086+0000 7fc959d40640 1 --2- 172.21.15.57:0/2249342518 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc9540cf340 0x7fc9540c9560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:47.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:47.086+0000 7fc95ad42640 1 -- 172.21.15.57:0/2249342518 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc9540ceb40 msgr2=0x7fc9541585a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:47.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:47.086+0000 7fc95ad42640 1 --2- 172.21.15.57:0/2249342518 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc9540ceb40 0x7fc9541585a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:47.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:47.086+0000 7fc95ad42640 1 -- 172.21.15.57:0/2249342518 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc9540cf340 msgr2=0x7fc9540c9560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:47.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:47.086+0000 7fc95ad42640 1 --2- 172.21.15.57:0/2249342518 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc9540cf340 0x7fc9540c9560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:47.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:47.086+0000 7fc95ad42640 1 -- 172.21.15.57:0/2249342518 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc948009490 con 0x7fc95415b2a0 2023-10-19T13:02:47.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:47.086+0000 7fc95a541640 1 --2- 172.21.15.57:0/2249342518 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc9540ceb40 0x7fc9541585a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:02:47.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:47.086+0000 7fc95ad42640 1 --2- 172.21.15.57:0/2249342518 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc95415b2a0 0x7fc9540c7e30 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7fc94400cae0 tx=0x7fc94400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:02:47.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:47.086+0000 7fc952ffd640 1 -- 172.21.15.57:0/2249342518 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc94400ac40 con 0x7fc95415b2a0 2023-10-19T13:02:47.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:47.086+0000 7fc952ffd640 1 -- 172.21.15.57:0/2249342518 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fc94400ae90 con 0x7fc95415b2a0 2023-10-19T13:02:47.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:47.086+0000 7fc95c7a5640 1 -- 172.21.15.57:0/2249342518 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9540c8a80 con 0x7fc95415b2a0 2023-10-19T13:02:47.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:47.086+0000 7fc952ffd640 1 -- 172.21.15.57:0/2249342518 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc944003bd0 con 0x7fc95415b2a0 2023-10-19T13:02:47.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:47.086+0000 7fc95c7a5640 1 -- 172.21.15.57:0/2249342518 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9540c9040 con 0x7fc95415b2a0 2023-10-19T13:02:47.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:47.087+0000 7fc95c7a5640 1 -- 172.21.15.57:0/2249342518 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9540661f0 con 0x7fc95415b2a0 2023-10-19T13:02:47.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:47.088+0000 7fc952ffd640 1 -- 172.21.15.57:0/2249342518 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 57) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fc944003d30 con 0x7fc95415b2a0 2023-10-19T13:02:47.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:47.088+0000 7fc952ffd640 1 --2- 172.21.15.57:0/2249342518 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7fc94006aed0 0x7fc94006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:47.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:47.088+0000 7fc952ffd640 1 -- 172.21.15.57:0/2249342518 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 7033+0+0 (secure 0 0 0) 0x7fc94408a470 con 0x7fc95415b2a0 2023-10-19T13:02:47.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:47.088+0000 7fc95a541640 1 --2- 172.21.15.57:0/2249342518 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7fc94006aed0 0x7fc94006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:47.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:47.089+0000 7fc95a541640 1 --2- 172.21.15.57:0/2249342518 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7fc94006aed0 0x7fc94006d390 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc94c0058c0 tx=0x7fc94c005850 comp rx=0 tx=0).ready entity=mgr.7540 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:02:47.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:47.092+0000 7fc952ffd640 1 -- 172.21.15.57:0/2249342518 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc944055090 con 0x7fc95415b2a0 2023-10-19T13:02:47.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:47.274+0000 7fc95c7a5640 1 -- 172.21.15.57:0/2249342518 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc9540c9230 con 0x7fc95415b2a0 2023-10-19T13:02:47.993 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:02:47.996 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:02:48.278 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:48.275+0000 7fc952ffd640 1 -- 172.21.15.57:0/2249342518 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v120) v1 ==== 181+0+0 (secure 0 0 0) 0x7fc9440586b0 con 0x7fc95415b2a0 2023-10-19T13:02:48.278 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'cephfs_metadata' removed 2023-10-19T13:02:48.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:48.282+0000 7fc952ffd640 1 -- 172.21.15.57:0/2249342518 <== mon.0 v2:172.21.15.57:3300/0 8 ==== mgrmap(e 58) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc944055460 con 0x7fc95415b2a0 2023-10-19T13:02:48.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:48.283+0000 7fc95c7a5640 1 -- 172.21.15.57:0/2249342518 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7fc94006aed0 msgr2=0x7fc94006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:48.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:48.283+0000 7fc95c7a5640 1 --2- 172.21.15.57:0/2249342518 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7fc94006aed0 0x7fc94006d390 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc94c0058c0 tx=0x7fc94c005850 comp rx=0 tx=0).stop 2023-10-19T13:02:48.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:48.283+0000 7fc95c7a5640 1 -- 172.21.15.57:0/2249342518 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc95415b2a0 msgr2=0x7fc9540c7e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:48.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:48.283+0000 7fc95c7a5640 1 --2- 172.21.15.57:0/2249342518 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc95415b2a0 0x7fc9540c7e30 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7fc94400cae0 tx=0x7fc94400a060 comp rx=0 tx=0).stop 2023-10-19T13:02:48.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:48.283+0000 7fc95c7a5640 1 -- 172.21.15.57:0/2249342518 shutdown_connections 2023-10-19T13:02:48.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:48.283+0000 7fc95c7a5640 1 --2- 172.21.15.57:0/2249342518 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7fc94006aed0 0x7fc94006d390 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:48.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:48.283+0000 7fc95c7a5640 1 --2- 172.21.15.57:0/2249342518 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc95415b2a0 0x7fc9540c7e30 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:48.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:48.283+0000 7fc95c7a5640 1 --2- 172.21.15.57:0/2249342518 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc9540cf340 0x7fc9540c9560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:48.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:48.283+0000 7fc95c7a5640 1 --2- 172.21.15.57:0/2249342518 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc9540ceb40 0x7fc9541585a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:48.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:48.283+0000 7fc95c7a5640 1 -- 172.21.15.57:0/2249342518 >> 172.21.15.57:0/2249342518 conn(0x7fc9540d0d60 msgr2=0x7fc95406b760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:02:48.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:48.283+0000 7fc95c7a5640 1 -- 172.21.15.57:0/2249342518 shutdown_connections 2023-10-19T13:02:48.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:48.283+0000 7fc95c7a5640 1 -- 172.21.15.57:0/2249342518 wait complete. 2023-10-19T13:02:48.301 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-19T13:02:48.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:48.418+0000 7f46e7316640 1 -- 172.21.15.57:0/2388786079 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f46c8004ab0 msgr2=0x7f46c8004ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:48.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:48.418+0000 7f46e7316640 1 --2- 172.21.15.57:0/2388786079 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f46c8004ab0 0x7f46c8004ed0 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7f46d0009e20 tx=0x7f46c8006900 comp rx=0 tx=0).stop 2023-10-19T13:02:48.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:48.418+0000 7f46e58b3640 1 -- 172.21.15.57:0/2388786079 reap_dead start 2023-10-19T13:02:48.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:48.418+0000 7f46e7316640 1 -- 172.21.15.57:0/2388786079 shutdown_connections 2023-10-19T13:02:48.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:48.418+0000 7f46e7316640 1 -- 172.21.15.57:0/2388786079 >> 172.21.15.57:0/2388786079 conn(0x7f46e0069930 msgr2=0x7f46e0069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:02:48.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:48.418+0000 7f46e7316640 1 -- 172.21.15.57:0/2388786079 shutdown_connections 2023-10-19T13:02:48.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:48.418+0000 7f46e7316640 1 -- 172.21.15.57:0/2388786079 wait complete. 2023-10-19T13:02:48.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:48.419+0000 7f46e7316640 1 Processor -- start 2023-10-19T13:02:48.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:48.419+0000 7f46e7316640 1 -- start start 2023-10-19T13:02:48.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:48.420+0000 7f46e7316640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f46c8004ab0 0x7f46e01f13e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:48.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:48.420+0000 7f46e7316640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f46e00c2960 0x7f46e01f3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:48.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:48.420+0000 7f46e7316640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f46e00c2de0 0x7f46e01f4200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:48.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:48.420+0000 7f46e7316640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46e01f4740 con 0x7f46c8004ab0 2023-10-19T13:02:48.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:48.420+0000 7f46e7316640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46e00b1830 con 0x7f46e00c2de0 2023-10-19T13:02:48.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:48.420+0000 7f46e7316640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46e00aa960 con 0x7f46e00c2960 2023-10-19T13:02:48.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:48.420+0000 7f46e50b2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f46c8004ab0 0x7f46e01f13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:48.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:48.420+0000 7f46e50b2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f46c8004ab0 0x7f46e01f13e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43300/0 (socket says 172.21.15.57:43300) 2023-10-19T13:02:48.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:48.420+0000 7f46e50b2640 1 -- 172.21.15.57:0/1312089178 learned_addr learned my addr 172.21.15.57:0/1312089178 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:02:48.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:48.420+0000 7f46e58b3640 1 --2- 172.21.15.57:0/1312089178 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f46e00c2de0 0x7f46e01f4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:48.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:48.420+0000 7f46e48b1640 1 --2- 172.21.15.57:0/1312089178 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f46e00c2960 0x7f46e01f3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:48.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:48.420+0000 7f46e50b2640 1 -- 172.21.15.57:0/1312089178 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f46e00c2960 msgr2=0x7f46e01f3cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:48.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:48.420+0000 7f46e50b2640 1 --2- 172.21.15.57:0/1312089178 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f46e00c2960 0x7f46e01f3cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:48.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:48.420+0000 7f46e50b2640 1 -- 172.21.15.57:0/1312089178 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f46e00c2de0 msgr2=0x7f46e01f4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:48.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:48.420+0000 7f46e50b2640 1 --2- 172.21.15.57:0/1312089178 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f46e00c2de0 0x7f46e01f4200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:48.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:48.420+0000 7f46e50b2640 1 -- 172.21.15.57:0/1312089178 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46e00aac40 con 0x7f46c8004ab0 2023-10-19T13:02:48.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:48.420+0000 7f46e48b1640 1 --2- 172.21.15.57:0/1312089178 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f46e00c2960 0x7f46e01f3cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T13:02:48.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:48.420+0000 7f46e50b2640 1 --2- 172.21.15.57:0/1312089178 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f46c8004ab0 0x7f46e01f13e0 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f46d002ca50 tx=0x7f46d0033c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:02:48.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:48.420+0000 7f46ddffb640 1 -- 172.21.15.57:0/1312089178 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d0003b70 con 0x7f46c8004ab0 2023-10-19T13:02:48.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:48.420+0000 7f46ddffb640 1 -- 172.21.15.57:0/1312089178 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f46d0003cd0 con 0x7f46c8004ab0 2023-10-19T13:02:48.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:48.420+0000 7f46e7316640 1 -- 172.21.15.57:0/1312089178 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46e00aaf20 con 0x7f46c8004ab0 2023-10-19T13:02:48.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:48.420+0000 7f46ddffb640 1 -- 172.21.15.57:0/1312089178 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d0004270 con 0x7f46c8004ab0 2023-10-19T13:02:48.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:48.421+0000 7f46e7316640 1 -- 172.21.15.57:0/1312089178 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46e01f1920 con 0x7f46c8004ab0 2023-10-19T13:02:48.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:48.422+0000 7f46e7316640 1 -- 172.21.15.57:0/1312089178 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46e00690e0 con 0x7f46c8004ab0 2023-10-19T13:02:48.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:48.422+0000 7f46ddffb640 1 -- 172.21.15.57:0/1312089178 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 58) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f46d00043d0 con 0x7f46c8004ab0 2023-10-19T13:02:48.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:48.422+0000 7f46ddffb640 1 --2- 172.21.15.57:0/1312089178 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f46cc06af60 0x7f46cc06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:48.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:48.423+0000 7f46ddffb640 1 -- 172.21.15.57:0/1312089178 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6595+0+0 (secure 0 0 0) 0x7f46d00acb00 con 0x7f46c8004ab0 2023-10-19T13:02:48.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:48.423+0000 7f46e48b1640 1 --2- 172.21.15.57:0/1312089178 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f46cc06af60 0x7f46cc06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:48.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:48.423+0000 7f46e48b1640 1 --2- 172.21.15.57:0/1312089178 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f46cc06af60 0x7f46cc06d420 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f46d40065f0 tx=0x7f46d4006b30 comp rx=0 tx=0).ready entity=mgr.7540 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:02:48.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:48.427+0000 7f46ddffb640 1 -- 172.21.15.57:0/1312089178 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f46d0077730 con 0x7f46c8004ab0 2023-10-19T13:02:48.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:48.601+0000 7f46e7316640 1 -- 172.21.15.57:0/1312089178 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f46e0159c90 con 0x7f46c8004ab0 2023-10-19T13:02:49.279 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.277+0000 7f46ddffb640 1 -- 172.21.15.57:0/1312089178 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v121) v1 ==== 169+0+0 (secure 0 0 0) 0x7f46d0041020 con 0x7f46c8004ab0 2023-10-19T13:02:49.280 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'cephfs_data' removed 2023-10-19T13:02:49.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.279+0000 7f46e7316640 1 -- 172.21.15.57:0/1312089178 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f46cc06af60 msgr2=0x7f46cc06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:49.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.279+0000 7f46e7316640 1 --2- 172.21.15.57:0/1312089178 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f46cc06af60 0x7f46cc06d420 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f46d40065f0 tx=0x7f46d4006b30 comp rx=0 tx=0).stop 2023-10-19T13:02:49.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.279+0000 7f46e7316640 1 -- 172.21.15.57:0/1312089178 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f46c8004ab0 msgr2=0x7f46e01f13e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:49.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.279+0000 7f46e7316640 1 --2- 172.21.15.57:0/1312089178 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f46c8004ab0 0x7f46e01f13e0 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f46d002ca50 tx=0x7f46d0033c70 comp rx=0 tx=0).stop 2023-10-19T13:02:49.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.279+0000 7f46e7316640 1 -- 172.21.15.57:0/1312089178 shutdown_connections 2023-10-19T13:02:49.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.279+0000 7f46e7316640 1 --2- 172.21.15.57:0/1312089178 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f46cc06af60 0x7f46cc06d420 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:49.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.279+0000 7f46e7316640 1 --2- 172.21.15.57:0/1312089178 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f46e00c2de0 0x7f46e01f4200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:49.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.279+0000 7f46e7316640 1 --2- 172.21.15.57:0/1312089178 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f46e00c2960 0x7f46e01f3cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:49.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.279+0000 7f46e7316640 1 --2- 172.21.15.57:0/1312089178 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f46c8004ab0 0x7f46e01f13e0 unknown :-1 s=CLOSED pgs=1381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:49.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.279+0000 7f46e7316640 1 -- 172.21.15.57:0/1312089178 >> 172.21.15.57:0/1312089178 conn(0x7f46e0069930 msgr2=0x7f46e013b8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:02:49.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.279+0000 7f46e7316640 1 -- 172.21.15.57:0/1312089178 shutdown_connections 2023-10-19T13:02:49.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.279+0000 7f46e7316640 1 -- 172.21.15.57:0/1312089178 wait complete. 2023-10-19T13:02:49.299 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 30 2023-10-19T13:02:49.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.412+0000 7f7ca29a9640 1 -- 172.21.15.57:0/1344674603 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7c840039a0 msgr2=0x7f7c84003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:49.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.412+0000 7f7ca29a9640 1 --2- 172.21.15.57:0/1344674603 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7c840039a0 0x7f7c84003dc0 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f7c8c001680 tx=0x7f7c840057f0 comp rx=0 tx=0).stop 2023-10-19T13:02:49.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.412+0000 7f7ca0f46640 1 -- 172.21.15.57:0/1344674603 reap_dead start 2023-10-19T13:02:49.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.412+0000 7f7ca29a9640 1 -- 172.21.15.57:0/1344674603 shutdown_connections 2023-10-19T13:02:49.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.412+0000 7f7ca29a9640 1 -- 172.21.15.57:0/1344674603 >> 172.21.15.57:0/1344674603 conn(0x7f7c9c0d0d60 msgr2=0x7f7c9c0d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:02:49.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.412+0000 7f7ca29a9640 1 -- 172.21.15.57:0/1344674603 shutdown_connections 2023-10-19T13:02:49.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.412+0000 7f7ca29a9640 1 -- 172.21.15.57:0/1344674603 wait complete. 2023-10-19T13:02:49.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.413+0000 7f7ca29a9640 1 Processor -- start 2023-10-19T13:02:49.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.413+0000 7f7ca29a9640 1 -- start start 2023-10-19T13:02:49.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.414+0000 7f7ca29a9640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7c840039a0 0x7f7c9c157510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:49.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.414+0000 7f7ca29a9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7c9c15c7b0 0x7f7c9c0ca8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:49.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.414+0000 7f7ca29a9640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7c9c1f1290 0x7f7c9c1f1710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:49.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.414+0000 7f7ca29a9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c9c157a50 con 0x7f7c9c15c7b0 2023-10-19T13:02:49.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.414+0000 7f7ca29a9640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c9c15cc90 con 0x7f7c840039a0 2023-10-19T13:02:49.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.414+0000 7f7ca29a9640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c9c15cf70 con 0x7f7c9c1f1290 2023-10-19T13:02:49.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.414+0000 7f7c9b7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7c9c15c7b0 0x7f7c9c0ca8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:49.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.414+0000 7f7c9b7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7c9c15c7b0 0x7f7c9c0ca8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43318/0 (socket says 172.21.15.57:43318) 2023-10-19T13:02:49.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.414+0000 7f7c9b7fe640 1 -- 172.21.15.57:0/147787431 learned_addr learned my addr 172.21.15.57:0/147787431 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:02:49.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.414+0000 7f7c9bfff640 1 --2- 172.21.15.57:0/147787431 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7c840039a0 0x7f7c9c157510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:49.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.414+0000 7f7ca0f46640 1 --2- 172.21.15.57:0/147787431 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7c9c1f1290 0x7f7c9c1f1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:49.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.414+0000 7f7c9b7fe640 1 -- 172.21.15.57:0/147787431 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7c9c1f1290 msgr2=0x7f7c9c1f1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:49.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.414+0000 7f7c9b7fe640 1 --2- 172.21.15.57:0/147787431 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7c9c1f1290 0x7f7c9c1f1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:49.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.414+0000 7f7c9b7fe640 1 -- 172.21.15.57:0/147787431 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7c840039a0 msgr2=0x7f7c9c157510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:49.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.414+0000 7f7c9b7fe640 1 --2- 172.21.15.57:0/147787431 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7c840039a0 0x7f7c9c157510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:49.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.414+0000 7f7c9b7fe640 1 -- 172.21.15.57:0/147787431 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c9c15d250 con 0x7f7c9c15c7b0 2023-10-19T13:02:49.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.414+0000 7f7c9bfff640 1 --2- 172.21.15.57:0/147787431 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7c840039a0 0x7f7c9c157510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:02:49.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.415+0000 7f7c9b7fe640 1 --2- 172.21.15.57:0/147787431 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7c9c15c7b0 0x7f7c9c0ca8d0 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f7c90006600 tx=0x7f7c900039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:02:49.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.415+0000 7f7c98ff9640 1 -- 172.21.15.57:0/147787431 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c90011900 con 0x7f7c9c15c7b0 2023-10-19T13:02:49.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.415+0000 7f7c98ff9640 1 -- 172.21.15.57:0/147787431 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f7c90011a60 con 0x7f7c9c15c7b0 2023-10-19T13:02:49.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.415+0000 7f7ca29a9640 1 -- 172.21.15.57:0/147787431 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c9c15d530 con 0x7f7c9c15c7b0 2023-10-19T13:02:49.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.415+0000 7f7c98ff9640 1 -- 172.21.15.57:0/147787431 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c90011e50 con 0x7f7c9c15c7b0 2023-10-19T13:02:49.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.415+0000 7f7ca29a9640 1 -- 172.21.15.57:0/147787431 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c9c0cae10 con 0x7f7c9c15c7b0 2023-10-19T13:02:49.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.416+0000 7f7ca29a9640 1 -- 172.21.15.57:0/147787431 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c9c15da50 con 0x7f7c9c15c7b0 2023-10-19T13:02:49.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.416+0000 7f7c98ff9640 1 -- 172.21.15.57:0/147787431 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 58) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f7c90010420 con 0x7f7c9c15c7b0 2023-10-19T13:02:49.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.417+0000 7f7c98ff9640 1 --2- 172.21.15.57:0/147787431 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f7c8806af60 0x7f7c8806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:49.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.417+0000 7f7c98ff9640 1 -- 172.21.15.57:0/147787431 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6065+0+0 (secure 0 0 0) 0x7f7c9008a870 con 0x7f7c9c15c7b0 2023-10-19T13:02:49.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.417+0000 7f7c9bfff640 1 --2- 172.21.15.57:0/147787431 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f7c8806af60 0x7f7c8806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:49.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.417+0000 7f7c9bfff640 1 --2- 172.21.15.57:0/147787431 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f7c8806af60 0x7f7c8806d420 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7c8c034000 tx=0x7f7c8c02e040 comp rx=0 tx=0).ready entity=mgr.7540 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:02:49.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.421+0000 7f7c98ff9640 1 -- 172.21.15.57:0/147787431 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7c9c15da50 con 0x7f7c9c15c7b0 2023-10-19T13:02:49.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.595+0000 7f7ca29a9640 1 -- 172.21.15.57:0/147787431 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "config reset", "num": 30} v 0) v1 -- 0x7f7c9c15da50 con 0x7f7c9c15c7b0 2023-10-19T13:02:49.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.599+0000 7f7c98ff9640 1 -- 172.21.15.57:0/147787431 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 30}]=0 v50) v1 ==== 71+0+0 (secure 0 0 0) 0x7f7c9c15da50 con 0x7f7c9c15c7b0 2023-10-19T13:02:49.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.601+0000 7f7ca29a9640 1 -- 172.21.15.57:0/147787431 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f7c8806af60 msgr2=0x7f7c8806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:49.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.601+0000 7f7ca29a9640 1 --2- 172.21.15.57:0/147787431 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f7c8806af60 0x7f7c8806d420 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7c8c034000 tx=0x7f7c8c02e040 comp rx=0 tx=0).stop 2023-10-19T13:02:49.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.601+0000 7f7ca29a9640 1 -- 172.21.15.57:0/147787431 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7c9c15c7b0 msgr2=0x7f7c9c0ca8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:49.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.601+0000 7f7ca29a9640 1 --2- 172.21.15.57:0/147787431 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7c9c15c7b0 0x7f7c9c0ca8d0 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f7c90006600 tx=0x7f7c900039c0 comp rx=0 tx=0).stop 2023-10-19T13:02:49.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.601+0000 7f7ca29a9640 1 -- 172.21.15.57:0/147787431 shutdown_connections 2023-10-19T13:02:49.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.601+0000 7f7ca29a9640 1 --2- 172.21.15.57:0/147787431 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f7c8806af60 0x7f7c8806d420 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:49.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.601+0000 7f7ca29a9640 1 --2- 172.21.15.57:0/147787431 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7c9c1f1290 0x7f7c9c1f1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:49.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.601+0000 7f7ca29a9640 1 --2- 172.21.15.57:0/147787431 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7c9c15c7b0 0x7f7c9c0ca8d0 unknown :-1 s=CLOSED pgs=1384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:49.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.601+0000 7f7ca29a9640 1 --2- 172.21.15.57:0/147787431 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7c840039a0 0x7f7c9c157510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:49.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.601+0000 7f7ca29a9640 1 -- 172.21.15.57:0/147787431 >> 172.21.15.57:0/147787431 conn(0x7f7c9c0d0d60 msgr2=0x7f7c9c0cf720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:02:49.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.601+0000 7f7ca29a9640 1 -- 172.21.15.57:0/147787431 shutdown_connections 2023-10-19T13:02:49.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.601+0000 7f7ca29a9640 1 -- 172.21.15.57:0/147787431 wait complete. 2023-10-19T13:02:49.615 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path' 2023-10-19T13:02:49.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.721+0000 7fc49ad33640 1 -- 172.21.15.57:0/2046281388 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4940abc10 msgr2=0x7fc4940c9b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:49.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.721+0000 7fc49ad33640 1 --2- 172.21.15.57:0/2046281388 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4940abc10 0x7fc4940c9b30 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7fc48c009be0 tx=0x7fc48c02c0c0 comp rx=0 tx=0).stop 2023-10-19T13:02:49.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.721+0000 7fc49ad33640 1 -- 172.21.15.57:0/2046281388 shutdown_connections 2023-10-19T13:02:49.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.721+0000 7fc49ad33640 1 --2- 172.21.15.57:0/2046281388 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4940abc10 0x7fc4940c9b30 unknown :-1 s=CLOSED pgs=1386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:49.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.721+0000 7fc49ad33640 1 --2- 172.21.15.57:0/2046281388 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc49406af00 0x7fc49406b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:49.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.721+0000 7fc49ad33640 1 -- 172.21.15.57:0/2046281388 >> 172.21.15.57:0/2046281388 conn(0x7fc494069930 msgr2=0x7fc494069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:02:49.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.721+0000 7fc49ad33640 1 -- 172.21.15.57:0/2046281388 shutdown_connections 2023-10-19T13:02:49.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.721+0000 7fc49ad33640 1 -- 172.21.15.57:0/2046281388 wait complete. 2023-10-19T13:02:49.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.723+0000 7fc49ad33640 1 Processor -- start 2023-10-19T13:02:49.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.723+0000 7fc49ad33640 1 -- start start 2023-10-19T13:02:49.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.723+0000 7fc49ad33640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc49406af00 0x7fc49414f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:49.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.723+0000 7fc49ad33640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4940abc10 0x7fc4940a64b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:49.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.723+0000 7fc49ad33640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc4940c96b0 0x7fc4940a69f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:49.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.723+0000 7fc49ad33640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc494152690 con 0x7fc4940abc10 2023-10-19T13:02:49.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.723+0000 7fc49ad33640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4940a8430 con 0x7fc49406af00 2023-10-19T13:02:49.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.723+0000 7fc49ad33640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc49414fbf0 con 0x7fc4940c96b0 2023-10-19T13:02:49.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.724+0000 7fc493fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4940abc10 0x7fc4940a64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:49.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.724+0000 7fc493fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4940abc10 0x7fc4940a64b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43336/0 (socket says 172.21.15.57:43336) 2023-10-19T13:02:49.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.724+0000 7fc493fff640 1 -- 172.21.15.57:0/1275514744 learned_addr learned my addr 172.21.15.57:0/1275514744 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:02:49.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.724+0000 7fc4992d0640 1 --2- 172.21.15.57:0/1275514744 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc4940c96b0 0x7fc4940a69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:49.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.724+0000 7fc498acf640 1 --2- 172.21.15.57:0/1275514744 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc49406af00 0x7fc49414f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:49.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.724+0000 7fc493fff640 1 -- 172.21.15.57:0/1275514744 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc4940c96b0 msgr2=0x7fc4940a69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:49.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.724+0000 7fc493fff640 1 --2- 172.21.15.57:0/1275514744 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc4940c96b0 0x7fc4940a69f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:49.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.724+0000 7fc493fff640 1 -- 172.21.15.57:0/1275514744 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc49406af00 msgr2=0x7fc49414f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:49.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.724+0000 7fc493fff640 1 --2- 172.21.15.57:0/1275514744 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc49406af00 0x7fc49414f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:49.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.724+0000 7fc493fff640 1 -- 172.21.15.57:0/1275514744 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc48c009840 con 0x7fc4940abc10 2023-10-19T13:02:49.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.724+0000 7fc498acf640 1 --2- 172.21.15.57:0/1275514744 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc49406af00 0x7fc49414f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T13:02:49.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.724+0000 7fc493fff640 1 --2- 172.21.15.57:0/1275514744 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4940abc10 0x7fc4940a64b0 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7fc48000c440 tx=0x7fc48000c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:02:49.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.724+0000 7fc4917fa640 1 -- 172.21.15.57:0/1275514744 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc480005480 con 0x7fc4940abc10 2023-10-19T13:02:49.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.724+0000 7fc4917fa640 1 -- 172.21.15.57:0/1275514744 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fc4800035f0 con 0x7fc4940abc10 2023-10-19T13:02:49.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.724+0000 7fc49ad33640 1 -- 172.21.15.57:0/1275514744 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc49414fe20 con 0x7fc4940abc10 2023-10-19T13:02:49.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.724+0000 7fc4917fa640 1 -- 172.21.15.57:0/1275514744 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc480003a00 con 0x7fc4940abc10 2023-10-19T13:02:49.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.724+0000 7fc49ad33640 1 -- 172.21.15.57:0/1275514744 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4941503e0 con 0x7fc4940abc10 2023-10-19T13:02:49.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.725+0000 7fc49ad33640 1 -- 172.21.15.57:0/1275514744 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4940690e0 con 0x7fc4940abc10 2023-10-19T13:02:49.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.726+0000 7fc4917fa640 1 -- 172.21.15.57:0/1275514744 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 58) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc480003b60 con 0x7fc4940abc10 2023-10-19T13:02:49.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.726+0000 7fc4917fa640 1 --2- 172.21.15.57:0/1275514744 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7fc47806af90 0x7fc47806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:49.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.726+0000 7fc4917fa640 1 -- 172.21.15.57:0/1275514744 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6065+0+0 (secure 0 0 0) 0x7fc480088fd0 con 0x7fc4940abc10 2023-10-19T13:02:49.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.726+0000 7fc498acf640 1 --2- 172.21.15.57:0/1275514744 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7fc47806af90 0x7fc47806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:49.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.727+0000 7fc498acf640 1 --2- 172.21.15.57:0/1275514744 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7fc47806af90 0x7fc47806d450 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc488006fd0 tx=0x7fc488008040 comp rx=0 tx=0).ready entity=mgr.7540 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:02:49.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.730+0000 7fc4917fa640 1 -- 172.21.15.57:0/1275514744 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc480053e90 con 0x7fc4940abc10 2023-10-19T13:02:49.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:49.975+0000 7fc49ad33640 1 -- 172.21.15.57:0/1275514744 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path"]} v 0) v1 -- 0x7fc4940cc420 con 0x7fc4940abc10 2023-10-19T13:02:50.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.290+0000 7fc4917fa640 1 -- 172.21.15.57:0/1275514744 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path"]}]=0 v702) v1 ==== 183+0+0 (secure 0 0 0) 0x7fc480017020 con 0x7fc4940abc10 2023-10-19T13:02:50.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.292+0000 7fc49ad33640 1 -- 172.21.15.57:0/1275514744 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7fc47806af90 msgr2=0x7fc47806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:50.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.292+0000 7fc49ad33640 1 --2- 172.21.15.57:0/1275514744 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7fc47806af90 0x7fc47806d450 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc488006fd0 tx=0x7fc488008040 comp rx=0 tx=0).stop 2023-10-19T13:02:50.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.293+0000 7fc49ad33640 1 -- 172.21.15.57:0/1275514744 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4940abc10 msgr2=0x7fc4940a64b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:50.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.293+0000 7fc49ad33640 1 --2- 172.21.15.57:0/1275514744 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4940abc10 0x7fc4940a64b0 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7fc48000c440 tx=0x7fc48000c900 comp rx=0 tx=0).stop 2023-10-19T13:02:50.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.293+0000 7fc49ad33640 1 -- 172.21.15.57:0/1275514744 shutdown_connections 2023-10-19T13:02:50.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.293+0000 7fc49ad33640 1 --2- 172.21.15.57:0/1275514744 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7fc47806af90 0x7fc47806d450 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:50.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.293+0000 7fc49ad33640 1 --2- 172.21.15.57:0/1275514744 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc4940c96b0 0x7fc4940a69f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:50.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.293+0000 7fc49ad33640 1 --2- 172.21.15.57:0/1275514744 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4940abc10 0x7fc4940a64b0 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:50.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.293+0000 7fc49ad33640 1 --2- 172.21.15.57:0/1275514744 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc49406af00 0x7fc49414f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:50.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.293+0000 7fc49ad33640 1 -- 172.21.15.57:0/1275514744 >> 172.21.15.57:0/1275514744 conn(0x7fc494069930 msgr2=0x7fc4940cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:02:50.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.293+0000 7fc49ad33640 1 -- 172.21.15.57:0/1275514744 shutdown_connections 2023-10-19T13:02:50.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.293+0000 7fc49ad33640 1 -- 172.21.15.57:0/1275514744 wait complete. 2023-10-19T13:02:50.307 INFO:tasks.cephfs_test_runner:Test that a non-existent path leads to schedule deactivation after a few retries. ... ok 2023-10-19T13:02:50.307 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test existence of a scheduled snapshot 2023-10-19T13:02:50.307 INFO:tasks.cephfs_test_runner:test_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-10-19T13:02:50.308 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T13:02:50.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.416+0000 7f8be77d7640 1 -- 172.21.15.57:0/4164258758 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8be00c96b0 msgr2=0x7f8be00c9b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:50.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.416+0000 7f8be77d7640 1 --2- 172.21.15.57:0/4164258758 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8be00c96b0 0x7f8be00c9b60 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f8bd4005fd0 tx=0x7f8bd402bd10 comp rx=0 tx=0).stop 2023-10-19T13:02:50.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.416+0000 7f8be77d7640 1 -- 172.21.15.57:0/4164258758 shutdown_connections 2023-10-19T13:02:50.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.416+0000 7f8be77d7640 1 --2- 172.21.15.57:0/4164258758 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8be00c96b0 0x7f8be00c9b60 unknown :-1 s=CLOSED pgs=1388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:50.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.416+0000 7f8be77d7640 1 --2- 172.21.15.57:0/4164258758 >> v2:172.21.15.179:3300/0 conn(0x7f8be006af00 0x7f8be006b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:50.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.416+0000 7f8be77d7640 1 -- 172.21.15.57:0/4164258758 >> 172.21.15.57:0/4164258758 conn(0x7f8be0069930 msgr2=0x7f8be0069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:02:50.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.416+0000 7f8be77d7640 1 -- 172.21.15.57:0/4164258758 shutdown_connections 2023-10-19T13:02:50.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.416+0000 7f8be77d7640 1 -- 172.21.15.57:0/4164258758 wait complete. 2023-10-19T13:02:50.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.417+0000 7f8be77d7640 1 Processor -- start 2023-10-19T13:02:50.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.417+0000 7f8be77d7640 1 -- start start 2023-10-19T13:02:50.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.418+0000 7f8be77d7640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8be006af00 0x7f8be00a8ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:50.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.418+0000 7f8be77d7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8be00ab500 0x7f8be0158ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:50.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.418+0000 7f8be77d7640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8be00c96b0 0x7f8be013cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:50.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.418+0000 7f8be77d7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8be0159020 con 0x7f8be00ab500 2023-10-19T13:02:50.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.418+0000 7f8be77d7640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8be00adb70 con 0x7f8be00c96b0 2023-10-19T13:02:50.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.418+0000 7f8be77d7640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8be00ade50 con 0x7f8be006af00 2023-10-19T13:02:50.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.418+0000 7f8be4d72640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8be00ab500 0x7f8be0158ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:50.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.418+0000 7f8be4d72640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8be00ab500 0x7f8be0158ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43346/0 (socket says 172.21.15.57:43346) 2023-10-19T13:02:50.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.418+0000 7f8be4d72640 1 -- 172.21.15.57:0/3263439885 learned_addr learned my addr 172.21.15.57:0/3263439885 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:02:50.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.418+0000 7f8be5573640 1 --2- 172.21.15.57:0/3263439885 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8be006af00 0x7f8be00a8ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:50.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.418+0000 7f8be5d74640 1 --2- 172.21.15.57:0/3263439885 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8be00c96b0 0x7f8be013cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:50.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.418+0000 7f8be4d72640 1 -- 172.21.15.57:0/3263439885 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8be006af00 msgr2=0x7f8be00a8ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:50.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.418+0000 7f8be4d72640 1 --2- 172.21.15.57:0/3263439885 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8be006af00 0x7f8be00a8ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:50.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.418+0000 7f8be4d72640 1 -- 172.21.15.57:0/3263439885 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8be00c96b0 msgr2=0x7f8be013cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:50.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.418+0000 7f8be4d72640 1 --2- 172.21.15.57:0/3263439885 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8be00c96b0 0x7f8be013cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:50.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.418+0000 7f8be4d72640 1 -- 172.21.15.57:0/3263439885 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bd4009560 con 0x7f8be00ab500 2023-10-19T13:02:50.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.418+0000 7f8be4d72640 1 --2- 172.21.15.57:0/3263439885 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8be00ab500 0x7f8be0158ae0 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f8bd4009850 tx=0x7f8bd402da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:02:50.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.419+0000 7f8bddffb640 1 -- 172.21.15.57:0/3263439885 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bd402de90 con 0x7f8be00ab500 2023-10-19T13:02:50.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.419+0000 7f8bddffb640 1 -- 172.21.15.57:0/3263439885 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f8bd40036e0 con 0x7f8be00ab500 2023-10-19T13:02:50.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.419+0000 7f8be77d7640 1 -- 172.21.15.57:0/3263439885 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8be00ae130 con 0x7f8be00ab500 2023-10-19T13:02:50.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.419+0000 7f8bddffb640 1 -- 172.21.15.57:0/3263439885 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bd4003a90 con 0x7f8be00ab500 2023-10-19T13:02:50.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.419+0000 7f8be77d7640 1 -- 172.21.15.57:0/3263439885 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8be00cb300 con 0x7f8be00ab500 2023-10-19T13:02:50.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.420+0000 7f8be77d7640 1 -- 172.21.15.57:0/3263439885 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8be01f0450 con 0x7f8be00ab500 2023-10-19T13:02:50.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.420+0000 7f8bddffb640 1 -- 172.21.15.57:0/3263439885 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 58) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f8bd4003bf0 con 0x7f8be00ab500 2023-10-19T13:02:50.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.421+0000 7f8bddffb640 1 --2- 172.21.15.57:0/3263439885 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f8bcc06af60 0x7f8bcc06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:50.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.421+0000 7f8bddffb640 1 -- 172.21.15.57:0/3263439885 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6065+0+0 (secure 0 0 0) 0x7f8bd40ac670 con 0x7f8be00ab500 2023-10-19T13:02:50.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.421+0000 7f8be5573640 1 --2- 172.21.15.57:0/3263439885 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f8bcc06af60 0x7f8bcc06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:50.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.422+0000 7f8be5573640 1 --2- 172.21.15.57:0/3263439885 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f8bcc06af60 0x7f8bcc06d420 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f8bd8009890 tx=0x7f8bd8008040 comp rx=0 tx=0).ready entity=mgr.7540 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:02:50.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.425+0000 7f8bddffb640 1 -- 172.21.15.57:0/3263439885 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8be01f0450 con 0x7f8be00ab500 2023-10-19T13:02:50.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.602+0000 7f8be77d7640 1 -- 172.21.15.57:0/3263439885 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8be01f0830 con 0x7f8be00ab500 2023-10-19T13:02:50.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.603+0000 7f8bddffb640 1 -- 172.21.15.57:0/3263439885 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v121) v1 ==== 74+0+14398 (secure 0 0 0) 0x7f8be01f0830 con 0x7f8be00ab500 2023-10-19T13:02:50.606 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:02:50.606 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":121,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T13:02:49.276304+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.57:6835/2556427177":"2023-10-20T13:02:45.895250+0000","172.21.15.179:0/2599642592":"2023-10-20T13:02:43.621587+0000","172.21.15.179:0/2915428574":"2023-10-20T13:02:43.621587+0000","172.21.15.179:0/356140706":"2023-10-20T13:02:43.621587+0000","172.21.15.179:0/4203351642":"2023-10-20T13:02:43.621587+0000","172.21.15.179:0/842129505":"2023-10-20T13:02:43.621587+0000","172.21.15.179:0/1511457369":"2023-10-20T13:02:43.621587+0000","172.21.15.57:6834/2556427177":"2023-10-20T13:02:45.895250+0000","172.21.15.179:6833/2366702417":"2023-10-20T13:02:43.621587+0000","172.21.15.179:6832/2366702417":"2023-10-20T13:02:43.621587+0000","172.21.15.179:0/3304791412":"2023-10-19T14:01:39.552724+0000","172.21.15.179:0/3109348805":"2023-10-20T13:01:37.504869+0000","172.21.15.179:0/2984488835":"2023-10-20T13:01:37.504869+0000","172.21.15.179:0/205205608":"2023-10-20T13:01:37.504869+0000","172.21.15.179:0/4112649564":"2023-10-20T13:01:37.504869+0000","172.21.15.179:6833/3281457486":"2023-10-20T13:01:37.504869+0000","172.21.15.179:0/1633718057":"2023-10-20T13:02:43.621587+0000","172.21.15.179:6832/3281457486":"2023-10-20T13:01:37.504869+0000","172.21.15.179:0/318926584":"2023-10-20T13:02:43.621587+0000","172.21.15.179:0/2509136709":"2023-10-20T13:01:37.504869+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T13:02:50.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.605+0000 7f8be77d7640 1 -- 172.21.15.57:0/3263439885 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f8bcc06af60 msgr2=0x7f8bcc06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:50.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.605+0000 7f8be77d7640 1 --2- 172.21.15.57:0/3263439885 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f8bcc06af60 0x7f8bcc06d420 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f8bd8009890 tx=0x7f8bd8008040 comp rx=0 tx=0).stop 2023-10-19T13:02:50.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.605+0000 7f8be77d7640 1 -- 172.21.15.57:0/3263439885 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8be00ab500 msgr2=0x7f8be0158ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:50.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.605+0000 7f8be77d7640 1 --2- 172.21.15.57:0/3263439885 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8be00ab500 0x7f8be0158ae0 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f8bd4009850 tx=0x7f8bd402da90 comp rx=0 tx=0).stop 2023-10-19T13:02:50.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.605+0000 7f8be77d7640 1 -- 172.21.15.57:0/3263439885 shutdown_connections 2023-10-19T13:02:50.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.605+0000 7f8be77d7640 1 --2- 172.21.15.57:0/3263439885 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f8bcc06af60 0x7f8bcc06d420 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:50.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.605+0000 7f8be77d7640 1 --2- 172.21.15.57:0/3263439885 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8be00c96b0 0x7f8be013cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:50.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.605+0000 7f8be77d7640 1 --2- 172.21.15.57:0/3263439885 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8be00ab500 0x7f8be0158ae0 unknown :-1 s=CLOSED pgs=1389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:50.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.605+0000 7f8be77d7640 1 --2- 172.21.15.57:0/3263439885 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8be006af00 0x7f8be00a8ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:50.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.605+0000 7f8be77d7640 1 -- 172.21.15.57:0/3263439885 >> 172.21.15.57:0/3263439885 conn(0x7f8be0069930 msgr2=0x7f8be00ab950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:02:50.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.605+0000 7f8be77d7640 1 -- 172.21.15.57:0/3263439885 shutdown_connections 2023-10-19T13:02:50.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.605+0000 7f8be77d7640 1 -- 172.21.15.57:0/3263439885 wait complete. 2023-10-19T13:02:50.618 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-19T12:44:07.371186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-19T13:02:50.619 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-19T13:02:50.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.728+0000 7f0663a56640 1 -- 172.21.15.57:0/796144160 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f065c0d1910 msgr2=0x7f065c0b0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:50.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.728+0000 7f0663a56640 1 --2- 172.21.15.57:0/796144160 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f065c0d1910 0x7f065c0b0650 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f0654005fd0 tx=0x7f065402c0e0 comp rx=0 tx=0).stop 2023-10-19T13:02:50.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.729+0000 7f0663a56640 1 -- 172.21.15.57:0/796144160 shutdown_connections 2023-10-19T13:02:50.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.729+0000 7f0663a56640 1 --2- 172.21.15.57:0/796144160 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f065c0d1910 0x7f065c0b0650 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:50.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.729+0000 7f0663a56640 1 -- 172.21.15.57:0/796144160 >> 172.21.15.57:0/796144160 conn(0x7f065c0d0d60 msgr2=0x7f065c0d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:02:50.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.729+0000 7f0663a56640 1 -- 172.21.15.57:0/796144160 shutdown_connections 2023-10-19T13:02:50.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.729+0000 7f0663a56640 1 -- 172.21.15.57:0/796144160 wait complete. 2023-10-19T13:02:50.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.730+0000 7f0663a56640 1 Processor -- start 2023-10-19T13:02:50.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.730+0000 7f0663a56640 1 -- start start 2023-10-19T13:02:50.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.730+0000 7f0663a56640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f065c0cf340 0x7f065c1553d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:50.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.730+0000 7f0663a56640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f065c0d1910 0x7f065c155b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:50.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.730+0000 7f0663a56640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f065c15b2a0 0x7f065c157260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:50.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.730+0000 7f0663a56640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f065c155910 con 0x7f065c0cf340 2023-10-19T13:02:50.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.730+0000 7f0663a56640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f065c1563f0 con 0x7f065c15b2a0 2023-10-19T13:02:50.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.730+0000 7f0663a56640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f065c1566d0 con 0x7f065c0d1910 2023-10-19T13:02:50.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.731+0000 7f06617f2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f065c0cf340 0x7f065c1553d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:50.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.731+0000 7f06617f2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f065c0cf340 0x7f065c1553d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43352/0 (socket says 172.21.15.57:43352) 2023-10-19T13:02:50.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.731+0000 7f06617f2640 1 -- 172.21.15.57:0/41410887 learned_addr learned my addr 172.21.15.57:0/41410887 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:02:50.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.731+0000 7f0661ff3640 1 --2- 172.21.15.57:0/41410887 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f065c15b2a0 0x7f065c157260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:50.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.731+0000 7f0660ff1640 1 --2- 172.21.15.57:0/41410887 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f065c0d1910 0x7f065c155b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:50.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.731+0000 7f06617f2640 1 -- 172.21.15.57:0/41410887 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f065c0d1910 msgr2=0x7f065c155b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:50.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.731+0000 7f06617f2640 1 --2- 172.21.15.57:0/41410887 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f065c0d1910 0x7f065c155b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:50.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.731+0000 7f06617f2640 1 -- 172.21.15.57:0/41410887 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f065c15b2a0 msgr2=0x7f065c157260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:50.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.731+0000 7f06617f2640 1 --2- 172.21.15.57:0/41410887 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f065c15b2a0 0x7f065c157260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:50.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.731+0000 7f06617f2640 1 -- 172.21.15.57:0/41410887 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0654009840 con 0x7f065c0cf340 2023-10-19T13:02:50.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.731+0000 7f06617f2640 1 --2- 172.21.15.57:0/41410887 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f065c0cf340 0x7f065c1553d0 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f065800c490 tx=0x7f065800c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:02:50.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.731+0000 7f0651ffb640 1 -- 172.21.15.57:0/41410887 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0658012830 con 0x7f065c0cf340 2023-10-19T13:02:50.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.731+0000 7f0663a56640 1 -- 172.21.15.57:0/41410887 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f065c1569b0 con 0x7f065c0cf340 2023-10-19T13:02:50.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.731+0000 7f0651ffb640 1 -- 172.21.15.57:0/41410887 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f0658012a80 con 0x7f065c0cf340 2023-10-19T13:02:50.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.731+0000 7f0651ffb640 1 -- 172.21.15.57:0/41410887 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0658011380 con 0x7f065c0cf340 2023-10-19T13:02:50.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.732+0000 7f0663a56640 1 -- 172.21.15.57:0/41410887 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f065c156f70 con 0x7f065c0cf340 2023-10-19T13:02:50.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.732+0000 7f0663a56640 1 -- 172.21.15.57:0/41410887 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f065c0690e0 con 0x7f065c0cf340 2023-10-19T13:02:50.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.733+0000 7f0651ffb640 1 -- 172.21.15.57:0/41410887 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 58) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f06580114e0 con 0x7f065c0cf340 2023-10-19T13:02:50.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.733+0000 7f0651ffb640 1 --2- 172.21.15.57:0/41410887 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f064006af60 0x7f064006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:50.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.733+0000 7f0651ffb640 1 -- 172.21.15.57:0/41410887 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6065+0+0 (secure 0 0 0) 0x7f0658089740 con 0x7f065c0cf340 2023-10-19T13:02:50.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.734+0000 7f0660ff1640 1 --2- 172.21.15.57:0/41410887 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f064006af60 0x7f064006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:50.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.734+0000 7f0660ff1640 1 --2- 172.21.15.57:0/41410887 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f064006af60 0x7f064006d420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f065c0af720 tx=0x7f064c009040 comp rx=0 tx=0).ready entity=mgr.7540 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:02:50.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.738+0000 7f0651ffb640 1 -- 172.21.15.57:0/41410887 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0658058030 con 0x7f065c0cf340 2023-10-19T13:02:50.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.911+0000 7f0663a56640 1 -- 172.21.15.57:0/41410887 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f065c0d0b40 con 0x7f065c0cf340 2023-10-19T13:02:50.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.912+0000 7f0651ffb640 1 -- 172.21.15.57:0/41410887 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v121) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0658057bc0 con 0x7f065c0cf340 2023-10-19T13:02:50.915 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-19T13:02:50.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.913+0000 7f0663a56640 1 -- 172.21.15.57:0/41410887 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f064006af60 msgr2=0x7f064006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:50.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.913+0000 7f0663a56640 1 --2- 172.21.15.57:0/41410887 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f064006af60 0x7f064006d420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f065c0af720 tx=0x7f064c009040 comp rx=0 tx=0).stop 2023-10-19T13:02:50.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.914+0000 7f0663a56640 1 -- 172.21.15.57:0/41410887 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f065c0cf340 msgr2=0x7f065c1553d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:50.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.914+0000 7f0663a56640 1 --2- 172.21.15.57:0/41410887 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f065c0cf340 0x7f065c1553d0 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f065800c490 tx=0x7f065800c950 comp rx=0 tx=0).stop 2023-10-19T13:02:50.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.914+0000 7f0663a56640 1 -- 172.21.15.57:0/41410887 shutdown_connections 2023-10-19T13:02:50.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.914+0000 7f0663a56640 1 --2- 172.21.15.57:0/41410887 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f064006af60 0x7f064006d420 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:50.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.914+0000 7f0663a56640 1 --2- 172.21.15.57:0/41410887 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f065c15b2a0 0x7f065c157260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:50.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.914+0000 7f0663a56640 1 --2- 172.21.15.57:0/41410887 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f065c0d1910 0x7f065c155b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:50.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.914+0000 7f0663a56640 1 --2- 172.21.15.57:0/41410887 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f065c0cf340 0x7f065c1553d0 unknown :-1 s=CLOSED pgs=1391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:50.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.914+0000 7f0663a56640 1 -- 172.21.15.57:0/41410887 >> 172.21.15.57:0/41410887 conn(0x7f065c0d0d60 msgr2=0x7f065c0ad8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:02:50.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.914+0000 7f0663a56640 1 -- 172.21.15.57:0/41410887 shutdown_connections 2023-10-19T13:02:50.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:50.914+0000 7f0663a56640 1 -- 172.21.15.57:0/41410887 wait complete. 2023-10-19T13:02:50.927 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule' 2023-10-19T13:02:51.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.034+0000 7fb32b0f2640 1 -- 172.21.15.57:0/3171396150 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb314004360 msgr2=0x7fb314004840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:51.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.034+0000 7fb32b0f2640 1 --2- 172.21.15.57:0/3171396150 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb314004360 0x7fb314004840 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7fb31000b9d0 tx=0x7fb3140057f0 comp rx=0 tx=0).stop 2023-10-19T13:02:51.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.034+0000 7fb32968f640 1 -- 172.21.15.57:0/3171396150 reap_dead start 2023-10-19T13:02:51.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.034+0000 7fb32b0f2640 1 -- 172.21.15.57:0/3171396150 shutdown_connections 2023-10-19T13:02:51.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.034+0000 7fb32b0f2640 1 -- 172.21.15.57:0/3171396150 >> 172.21.15.57:0/3171396150 conn(0x7fb324069930 msgr2=0x7fb324069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:02:51.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.034+0000 7fb32b0f2640 1 -- 172.21.15.57:0/3171396150 shutdown_connections 2023-10-19T13:02:51.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.034+0000 7fb32b0f2640 1 -- 172.21.15.57:0/3171396150 wait complete. 2023-10-19T13:02:51.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.035+0000 7fb32b0f2640 1 Processor -- start 2023-10-19T13:02:51.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.035+0000 7fb32b0f2640 1 -- start start 2023-10-19T13:02:51.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.036+0000 7fb32b0f2640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb314004360 0x7fb324139bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:51.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.036+0000 7fb32b0f2640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb3240b1bc0 0x7fb32414f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:51.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.036+0000 7fb32b0f2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb32413a890 0x7fb32414fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:51.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.036+0000 7fb32b0f2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb32413a0f0 con 0x7fb32413a890 2023-10-19T13:02:51.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.036+0000 7fb32b0f2640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb324152b80 con 0x7fb3240b1bc0 2023-10-19T13:02:51.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.036+0000 7fb32b0f2640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb324152e60 con 0x7fb314004360 2023-10-19T13:02:51.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.036+0000 7fb32968f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb32413a890 0x7fb32414fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:51.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.036+0000 7fb32968f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb32413a890 0x7fb32414fc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43374/0 (socket says 172.21.15.57:43374) 2023-10-19T13:02:51.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.036+0000 7fb32968f640 1 -- 172.21.15.57:0/847736555 learned_addr learned my addr 172.21.15.57:0/847736555 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:02:51.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.037+0000 7fb32968f640 1 -- 172.21.15.57:0/847736555 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb314004360 msgr2=0x7fb324139bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:51.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.037+0000 7fb32968f640 1 --2- 172.21.15.57:0/847736555 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb314004360 0x7fb324139bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:51.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.037+0000 7fb32968f640 1 -- 172.21.15.57:0/847736555 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb3240b1bc0 msgr2=0x7fb32414f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:51.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.037+0000 7fb323fff640 1 --2- 172.21.15.57:0/847736555 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb3240b1bc0 0x7fb32414f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:51.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.037+0000 7fb32968f640 1 --2- 172.21.15.57:0/847736555 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb3240b1bc0 0x7fb32414f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:51.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.037+0000 7fb32968f640 1 -- 172.21.15.57:0/847736555 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb324159a20 con 0x7fb32413a890 2023-10-19T13:02:51.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.037+0000 7fb323fff640 1 --2- 172.21.15.57:0/847736555 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb3240b1bc0 0x7fb32414f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T13:02:51.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.037+0000 7fb32968f640 1 --2- 172.21.15.57:0/847736555 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb32413a890 0x7fb32414fc80 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7fb3240cc150 tx=0x7fb31c00f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:02:51.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.037+0000 7fb3217fa640 1 -- 172.21.15.57:0/847736555 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb31c003d80 con 0x7fb32413a890 2023-10-19T13:02:51.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.037+0000 7fb3217fa640 1 -- 172.21.15.57:0/847736555 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fb31c004b20 con 0x7fb32413a890 2023-10-19T13:02:51.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.037+0000 7fb3217fa640 1 -- 172.21.15.57:0/847736555 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb31c004030 con 0x7fb32413a890 2023-10-19T13:02:51.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.037+0000 7fb32b0f2640 1 -- 172.21.15.57:0/847736555 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb324153140 con 0x7fb32413a890 2023-10-19T13:02:51.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.037+0000 7fb32b0f2640 1 -- 172.21.15.57:0/847736555 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb32406f400 con 0x7fb32413a890 2023-10-19T13:02:51.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.038+0000 7fb32b0f2640 1 -- 172.21.15.57:0/847736555 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb32406f5f0 con 0x7fb32413a890 2023-10-19T13:02:51.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.038+0000 7fb3217fa640 1 -- 172.21.15.57:0/847736555 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 58) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb31c004190 con 0x7fb32413a890 2023-10-19T13:02:51.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.039+0000 7fb3217fa640 1 --2- 172.21.15.57:0/847736555 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7fb30806af90 0x7fb30806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:51.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.039+0000 7fb3217fa640 1 -- 172.21.15.57:0/847736555 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6065+0+0 (secure 0 0 0) 0x7fb31c08a940 con 0x7fb32413a890 2023-10-19T13:02:51.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.039+0000 7fb328e8e640 1 --2- 172.21.15.57:0/847736555 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7fb30806af90 0x7fb30806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:51.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.039+0000 7fb328e8e640 1 --2- 172.21.15.57:0/847736555 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7fb30806af90 0x7fb30806d450 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fb318000c00 tx=0x7fb318001010 comp rx=0 tx=0).ready entity=mgr.7540 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:02:51.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.043+0000 7fb3217fa640 1 -- 172.21.15.57:0/847736555 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb31c055670 con 0x7fb32413a890 2023-10-19T13:02:51.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.289+0000 7fb32b0f2640 1 -- 172.21.15.57:0/847736555 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule"]} v 0) v1 -- 0x7fb3240690e0 con 0x7fb32413a890 2023-10-19T13:02:51.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.292+0000 7fb3217fa640 1 -- 172.21.15.57:0/847736555 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule"]}]=0 v703) v1 ==== 151+0+0 (secure 0 0 0) 0x7fb31c01e0d0 con 0x7fb32413a890 2023-10-19T13:02:51.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.294+0000 7fb32b0f2640 1 -- 172.21.15.57:0/847736555 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7fb30806af90 msgr2=0x7fb30806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:51.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.294+0000 7fb32b0f2640 1 --2- 172.21.15.57:0/847736555 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7fb30806af90 0x7fb30806d450 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fb318000c00 tx=0x7fb318001010 comp rx=0 tx=0).stop 2023-10-19T13:02:51.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.294+0000 7fb32b0f2640 1 -- 172.21.15.57:0/847736555 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb32413a890 msgr2=0x7fb32414fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:51.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.294+0000 7fb32b0f2640 1 --2- 172.21.15.57:0/847736555 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb32413a890 0x7fb32414fc80 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7fb3240cc150 tx=0x7fb31c00f790 comp rx=0 tx=0).stop 2023-10-19T13:02:51.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.294+0000 7fb32b0f2640 1 -- 172.21.15.57:0/847736555 shutdown_connections 2023-10-19T13:02:51.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.294+0000 7fb32b0f2640 1 --2- 172.21.15.57:0/847736555 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7fb30806af90 0x7fb30806d450 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:51.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.294+0000 7fb32b0f2640 1 --2- 172.21.15.57:0/847736555 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb32413a890 0x7fb32414fc80 unknown :-1 s=CLOSED pgs=1394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:51.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.294+0000 7fb32b0f2640 1 --2- 172.21.15.57:0/847736555 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb3240b1bc0 0x7fb32414f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:51.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.294+0000 7fb32b0f2640 1 --2- 172.21.15.57:0/847736555 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb314004360 0x7fb324139bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:51.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.294+0000 7fb32b0f2640 1 -- 172.21.15.57:0/847736555 >> 172.21.15.57:0/847736555 conn(0x7fb324069930 msgr2=0x7fb32415acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:02:51.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.294+0000 7fb32b0f2640 1 -- 172.21.15.57:0/847736555 shutdown_connections 2023-10-19T13:02:51.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.294+0000 7fb32b0f2640 1 -- 172.21.15.57:0/847736555 wait complete. 2023-10-19T13:02:51.308 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2023-10-19T13:02:51.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.414+0000 7f814ec83640 1 -- 172.21.15.57:0/760825445 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8134004d00 msgr2=0x7f8134005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:51.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.414+0000 7f814ec83640 1 --2- 172.21.15.57:0/760825445 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8134004d00 0x7f8134005180 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7f813800c260 tx=0x7f81340057f0 comp rx=0 tx=0).stop 2023-10-19T13:02:51.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.414+0000 7f814d220640 1 -- 172.21.15.57:0/760825445 reap_dead start 2023-10-19T13:02:51.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.414+0000 7f814ec83640 1 -- 172.21.15.57:0/760825445 shutdown_connections 2023-10-19T13:02:51.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.414+0000 7f814ec83640 1 -- 172.21.15.57:0/760825445 >> 172.21.15.57:0/760825445 conn(0x7f814806fae0 msgr2=0x7f814806fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:02:51.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.414+0000 7f814ec83640 1 -- 172.21.15.57:0/760825445 shutdown_connections 2023-10-19T13:02:51.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.415+0000 7f814ec83640 1 -- 172.21.15.57:0/760825445 wait complete. 2023-10-19T13:02:51.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.416+0000 7f814ec83640 1 Processor -- start 2023-10-19T13:02:51.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.416+0000 7f814ec83640 1 -- start start 2023-10-19T13:02:51.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.417+0000 7f814ec83640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8134004d00 0x7f8148146920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:51.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.417+0000 7f814ec83640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8148146e60 0x7f81481f2900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:51.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.417+0000 7f814ec83640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f81481f2e40 0x7f814813ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:51.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.417+0000 7f814ec83640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8148149ce0 con 0x7f8134004d00 2023-10-19T13:02:51.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.417+0000 7f814ec83640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f814814d650 con 0x7f81481f2e40 2023-10-19T13:02:51.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.417+0000 7f814ec83640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8148149440 con 0x7f8148146e60 2023-10-19T13:02:51.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.417+0000 7f814ca1f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8134004d00 0x7f8148146920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:51.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.417+0000 7f814ca1f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8134004d00 0x7f8148146920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43392/0 (socket says 172.21.15.57:43392) 2023-10-19T13:02:51.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.417+0000 7f814ca1f640 1 -- 172.21.15.57:0/1557737226 learned_addr learned my addr 172.21.15.57:0/1557737226 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:02:51.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.417+0000 7f8147fff640 1 --2- 172.21.15.57:0/1557737226 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8148146e60 0x7f81481f2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:51.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.417+0000 7f814d220640 1 --2- 172.21.15.57:0/1557737226 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f81481f2e40 0x7f814813ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:51.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.417+0000 7f814ca1f640 1 -- 172.21.15.57:0/1557737226 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8148146e60 msgr2=0x7f81481f2900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:51.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.417+0000 7f814ca1f640 1 --2- 172.21.15.57:0/1557737226 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8148146e60 0x7f81481f2900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:51.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.417+0000 7f814ca1f640 1 -- 172.21.15.57:0/1557737226 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f81481f2e40 msgr2=0x7f814813ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:51.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.417+0000 7f814ca1f640 1 --2- 172.21.15.57:0/1557737226 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f81481f2e40 0x7f814813ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:51.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.417+0000 7f814ca1f640 1 -- 172.21.15.57:0/1557737226 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81480b2330 con 0x7f8134004d00 2023-10-19T13:02:51.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.417+0000 7f814ca1f640 1 --2- 172.21.15.57:0/1557737226 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8134004d00 0x7f8148146920 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7f8140001890 tx=0x7f8140001d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:02:51.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.417+0000 7f81457fa640 1 -- 172.21.15.57:0/1557737226 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f814000e8f0 con 0x7f8134004d00 2023-10-19T13:02:51.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.417+0000 7f81457fa640 1 -- 172.21.15.57:0/1557737226 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f814000eb40 con 0x7f8134004d00 2023-10-19T13:02:51.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.417+0000 7f814ec83640 1 -- 172.21.15.57:0/1557737226 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8148149720 con 0x7f8134004d00 2023-10-19T13:02:51.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.417+0000 7f81457fa640 1 -- 172.21.15.57:0/1557737226 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8140011460 con 0x7f8134004d00 2023-10-19T13:02:51.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.418+0000 7f814ec83640 1 -- 172.21.15.57:0/1557737226 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81481490d0 con 0x7f8134004d00 2023-10-19T13:02:51.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.418+0000 7f814ec83640 1 -- 172.21.15.57:0/1557737226 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81480690e0 con 0x7f8134004d00 2023-10-19T13:02:51.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.419+0000 7f81457fa640 1 -- 172.21.15.57:0/1557737226 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 58) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f81400115c0 con 0x7f8134004d00 2023-10-19T13:02:51.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.419+0000 7f81457fa640 1 --2- 172.21.15.57:0/1557737226 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f812806af90 0x7f812806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:51.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.419+0000 7f81457fa640 1 -- 172.21.15.57:0/1557737226 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6065+0+0 (secure 0 0 0) 0x7f814008a630 con 0x7f8134004d00 2023-10-19T13:02:51.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.419+0000 7f8147fff640 1 --2- 172.21.15.57:0/1557737226 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f812806af90 0x7f812806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:51.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.420+0000 7f8147fff640 1 --2- 172.21.15.57:0/1557737226 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f812806af90 0x7f812806d450 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f8148147240 tx=0x7f813000b260 comp rx=0 tx=0).ready entity=mgr.7540 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:02:51.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.424+0000 7f81457fa640 1 -- 172.21.15.57:0/1557737226 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8140055400 con 0x7f8134004d00 2023-10-19T13:02:51.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.597+0000 7f814ec83640 1 -- 172.21.15.57:0/1557737226 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "config dump"} v 0) v1 -- 0x7f814814d420 con 0x7f8134004d00 2023-10-19T13:02:51.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.598+0000 7f81457fa640 1 -- 172.21.15.57:0/1557737226 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v50) v1 ==== 59+0+2220 (secure 0 0 0) 0x7f81400158c0 con 0x7f8134004d00 2023-10-19T13:02:51.601 INFO:teuthology.orchestra.run.smithi057.stdout:WHO MASK LEVEL OPTION VALUE RO 2023-10-19T13:02:51.601 INFO:teuthology.orchestra.run.smithi057.stdout:mon advanced mon_op_complaint_time 120 2023-10-19T13:02:51.601 INFO:teuthology.orchestra.run.smithi057.stdout:osd advanced osd_op_complaint_time 180.000000 2023-10-19T13:02:51.602 INFO:teuthology.orchestra.run.smithi057.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 4775.995373 2023-10-19T13:02:51.602 INFO:teuthology.orchestra.run.smithi057.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 4208.118410 2023-10-19T13:02:51.602 INFO:teuthology.orchestra.run.smithi057.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 4541.123030 2023-10-19T13:02:51.602 INFO:teuthology.orchestra.run.smithi057.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 4877.592206 2023-10-19T13:02:51.602 INFO:teuthology.orchestra.run.smithi057.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 5019.748032 2023-10-19T13:02:51.603 INFO:teuthology.orchestra.run.smithi057.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 5000.074376 2023-10-19T13:02:51.603 INFO:teuthology.orchestra.run.smithi057.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 5153.479439 2023-10-19T13:02:51.603 INFO:teuthology.orchestra.run.smithi057.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 4977.363845 2023-10-19T13:02:51.603 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced debug_mds 20/20 2023-10-19T13:02:51.603 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced debug_mds_balancer 20/20 2023-10-19T13:02:51.604 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced debug_ms 1/1 2023-10-19T13:02:51.604 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced mds_bal_fragment_size_max 10000 2023-10-19T13:02:51.604 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced mds_bal_merge_size 5 2023-10-19T13:02:51.604 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced mds_bal_split_bits 3 2023-10-19T13:02:51.604 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced mds_bal_split_size 100 2023-10-19T13:02:51.604 INFO:teuthology.orchestra.run.smithi057.stdout:mds dev mds_debug_frag true 2023-10-19T13:02:51.605 INFO:teuthology.orchestra.run.smithi057.stdout:mds dev mds_debug_scatterstat true 2023-10-19T13:02:51.605 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced mds_op_complaint_time 180.000000 2023-10-19T13:02:51.605 INFO:teuthology.orchestra.run.smithi057.stdout:mds dev mds_verify_scatter true 2023-10-19T13:02:51.605 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced osd_op_complaint_time 180.000000 2023-10-19T13:02:51.605 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced rados_mon_op_timeout 900 2023-10-19T13:02:51.606 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced rados_osd_op_timeout 900 2023-10-19T13:02:51.606 INFO:teuthology.orchestra.run.smithi057.stdout:client advanced client_mount_timeout 600 2023-10-19T13:02:51.606 INFO:teuthology.orchestra.run.smithi057.stdout:client advanced debug_client 20/20 2023-10-19T13:02:51.606 INFO:teuthology.orchestra.run.smithi057.stdout:client advanced debug_ms 1/1 2023-10-19T13:02:51.606 INFO:teuthology.orchestra.run.smithi057.stdout:client advanced rados_mon_op_timeout 900 2023-10-19T13:02:51.607 INFO:teuthology.orchestra.run.smithi057.stdout:client advanced rados_osd_op_timeout 900 2023-10-19T13:02:51.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.599+0000 7f814ec83640 1 -- 172.21.15.57:0/1557737226 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f812806af90 msgr2=0x7f812806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:51.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.599+0000 7f814ec83640 1 --2- 172.21.15.57:0/1557737226 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f812806af90 0x7f812806d450 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f8148147240 tx=0x7f813000b260 comp rx=0 tx=0).stop 2023-10-19T13:02:51.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.599+0000 7f814ec83640 1 -- 172.21.15.57:0/1557737226 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8134004d00 msgr2=0x7f8148146920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:51.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.599+0000 7f814ec83640 1 --2- 172.21.15.57:0/1557737226 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8134004d00 0x7f8148146920 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7f8140001890 tx=0x7f8140001d50 comp rx=0 tx=0).stop 2023-10-19T13:02:51.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.600+0000 7f814ec83640 1 -- 172.21.15.57:0/1557737226 shutdown_connections 2023-10-19T13:02:51.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.600+0000 7f814ec83640 1 --2- 172.21.15.57:0/1557737226 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f812806af90 0x7f812806d450 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:51.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.600+0000 7f814ec83640 1 --2- 172.21.15.57:0/1557737226 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f81481f2e40 0x7f814813ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:51.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.600+0000 7f814ec83640 1 --2- 172.21.15.57:0/1557737226 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8148146e60 0x7f81481f2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:51.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.600+0000 7f814ec83640 1 --2- 172.21.15.57:0/1557737226 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8134004d00 0x7f8148146920 unknown :-1 s=CLOSED pgs=1397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:51.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.600+0000 7f814ec83640 1 -- 172.21.15.57:0/1557737226 >> 172.21.15.57:0/1557737226 conn(0x7f814806fae0 msgr2=0x7f814806a730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:02:51.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.600+0000 7f814ec83640 1 -- 172.21.15.57:0/1557737226 shutdown_connections 2023-10-19T13:02:51.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.600+0000 7f814ec83640 1 -- 172.21.15.57:0/1557737226 wait complete. 2023-10-19T13:02:51.614 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-19T13:02:51.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.723+0000 7fee951f5640 1 -- 172.21.15.57:0/2935703495 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fee900cf340 msgr2=0x7fee900d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:51.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.723+0000 7fee951f5640 1 --2- 172.21.15.57:0/2935703495 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fee900cf340 0x7fee900d1910 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7fee9008a120 tx=0x7fee8802bda0 comp rx=0 tx=0).stop 2023-10-19T13:02:51.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.723+0000 7fee951f5640 1 -- 172.21.15.57:0/2935703495 shutdown_connections 2023-10-19T13:02:51.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.723+0000 7fee951f5640 1 --2- 172.21.15.57:0/2935703495 >> v2:172.21.15.179:3300/0 conn(0x7fee9015b2a0 0x7fee900cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:51.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.723+0000 7fee951f5640 1 --2- 172.21.15.57:0/2935703495 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fee900cf340 0x7fee900d1910 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:51.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.723+0000 7fee951f5640 1 -- 172.21.15.57:0/2935703495 >> 172.21.15.57:0/2935703495 conn(0x7fee900d0d60 msgr2=0x7fee900d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:02:51.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.723+0000 7fee951f5640 1 -- 172.21.15.57:0/2935703495 shutdown_connections 2023-10-19T13:02:51.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.723+0000 7fee951f5640 1 -- 172.21.15.57:0/2935703495 wait complete. 2023-10-19T13:02:51.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.724+0000 7fee951f5640 1 Processor -- start 2023-10-19T13:02:51.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.724+0000 7fee951f5640 1 -- start start 2023-10-19T13:02:51.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.725+0000 7fee951f5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fee900ceb40 0x7fee900c7e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:51.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.725+0000 7fee951f5640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fee900cf340 0x7fee90158470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:51.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.725+0000 7fee951f5640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fee9015b2a0 0x7fee900c95c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:51.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.725+0000 7fee951f5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee900c8f20 con 0x7fee900ceb40 2023-10-19T13:02:51.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.725+0000 7fee951f5640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee900c8680 con 0x7fee9015b2a0 2023-10-19T13:02:51.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.725+0000 7fee951f5640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee900c8960 con 0x7fee900cf340 2023-10-19T13:02:51.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.725+0000 7fee8ed9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fee900ceb40 0x7fee900c7e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:51.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.725+0000 7fee8ed9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fee900ceb40 0x7fee900c7e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43408/0 (socket says 172.21.15.57:43408) 2023-10-19T13:02:51.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.725+0000 7fee8ed9d640 1 -- 172.21.15.57:0/3548731548 learned_addr learned my addr 172.21.15.57:0/3548731548 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:02:51.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.726+0000 7fee8ed9d640 1 -- 172.21.15.57:0/3548731548 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fee900cf340 msgr2=0x7fee90158470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:51.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.726+0000 7fee8f59e640 1 --2- 172.21.15.57:0/3548731548 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fee9015b2a0 0x7fee900c95c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:51.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.726+0000 7fee8e59c640 1 --2- 172.21.15.57:0/3548731548 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fee900cf340 0x7fee90158470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:51.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.726+0000 7fee8ed9d640 1 --2- 172.21.15.57:0/3548731548 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fee900cf340 0x7fee90158470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:51.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.726+0000 7fee8ed9d640 1 -- 172.21.15.57:0/3548731548 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fee9015b2a0 msgr2=0x7fee900c95c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:51.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.726+0000 7fee8ed9d640 1 --2- 172.21.15.57:0/3548731548 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fee9015b2a0 0x7fee900c95c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:51.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.726+0000 7fee8ed9d640 1 -- 172.21.15.57:0/3548731548 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee88009560 con 0x7fee900ceb40 2023-10-19T13:02:51.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.726+0000 7fee8f59e640 1 --2- 172.21.15.57:0/3548731548 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fee9015b2a0 0x7fee900c95c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T13:02:51.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.726+0000 7fee8e59c640 1 --2- 172.21.15.57:0/3548731548 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fee900cf340 0x7fee90158470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T13:02:51.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.726+0000 7fee8ed9d640 1 --2- 172.21.15.57:0/3548731548 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fee900ceb40 0x7fee900c7e30 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7fee88000c00 tx=0x7fee8802db40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:02:51.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.726+0000 7fee877fe640 1 -- 172.21.15.57:0/3548731548 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee88003800 con 0x7fee900ceb40 2023-10-19T13:02:51.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.726+0000 7fee877fe640 1 -- 172.21.15.57:0/3548731548 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fee88003a50 con 0x7fee900ceb40 2023-10-19T13:02:51.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.726+0000 7fee951f5640 1 -- 172.21.15.57:0/3548731548 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee900c8c40 con 0x7fee900ceb40 2023-10-19T13:02:51.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.726+0000 7fee877fe640 1 -- 172.21.15.57:0/3548731548 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee88003e60 con 0x7fee900ceb40 2023-10-19T13:02:51.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.726+0000 7fee951f5640 1 -- 172.21.15.57:0/3548731548 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee900c8370 con 0x7fee900ceb40 2023-10-19T13:02:51.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.727+0000 7fee951f5640 1 -- 172.21.15.57:0/3548731548 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee900b3950 con 0x7fee900ceb40 2023-10-19T13:02:51.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.728+0000 7fee877fe640 1 -- 172.21.15.57:0/3548731548 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 58) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fee8803e450 con 0x7fee900ceb40 2023-10-19T13:02:51.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.728+0000 7fee877fe640 1 --2- 172.21.15.57:0/3548731548 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7fee7006af60 0x7fee7006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:51.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.728+0000 7fee877fe640 1 -- 172.21.15.57:0/3548731548 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6065+0+0 (secure 0 0 0) 0x7fee880ac380 con 0x7fee900ceb40 2023-10-19T13:02:51.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.728+0000 7fee8e59c640 1 --2- 172.21.15.57:0/3548731548 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7fee7006af60 0x7fee7006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:51.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.729+0000 7fee8e59c640 1 --2- 172.21.15.57:0/3548731548 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7fee7006af60 0x7fee7006d420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fee9006bf10 tx=0x7fee78005db0 comp rx=0 tx=0).ready entity=mgr.7540 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:02:51.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.732+0000 7fee877fe640 1 -- 172.21.15.57:0/3548731548 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fee900b3950 con 0x7fee900ceb40 2023-10-19T13:02:51.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.912+0000 7fee951f5640 1 -- 172.21.15.57:0/3548731548 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fee900b3950 con 0x7fee900ceb40 2023-10-19T13:02:51.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.915+0000 7fee877fe640 1 -- 172.21.15.57:0/3548731548 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v51)=0 v51) v1 ==== 122+0+0 (secure 0 0 0) 0x7fee900b3950 con 0x7fee900ceb40 2023-10-19T13:02:51.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.917+0000 7fee951f5640 1 -- 172.21.15.57:0/3548731548 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7fee7006af60 msgr2=0x7fee7006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:51.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.917+0000 7fee951f5640 1 --2- 172.21.15.57:0/3548731548 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7fee7006af60 0x7fee7006d420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fee9006bf10 tx=0x7fee78005db0 comp rx=0 tx=0).stop 2023-10-19T13:02:51.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.917+0000 7fee951f5640 1 -- 172.21.15.57:0/3548731548 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fee900ceb40 msgr2=0x7fee900c7e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:51.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.917+0000 7fee951f5640 1 --2- 172.21.15.57:0/3548731548 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fee900ceb40 0x7fee900c7e30 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7fee88000c00 tx=0x7fee8802db40 comp rx=0 tx=0).stop 2023-10-19T13:02:51.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.918+0000 7fee951f5640 1 -- 172.21.15.57:0/3548731548 shutdown_connections 2023-10-19T13:02:51.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.918+0000 7fee951f5640 1 --2- 172.21.15.57:0/3548731548 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7fee7006af60 0x7fee7006d420 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:51.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.918+0000 7fee951f5640 1 --2- 172.21.15.57:0/3548731548 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fee9015b2a0 0x7fee900c95c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:51.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.918+0000 7fee951f5640 1 --2- 172.21.15.57:0/3548731548 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fee900cf340 0x7fee90158470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:51.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.918+0000 7fee951f5640 1 --2- 172.21.15.57:0/3548731548 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fee900ceb40 0x7fee900c7e30 unknown :-1 s=CLOSED pgs=1400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:51.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.918+0000 7fee951f5640 1 -- 172.21.15.57:0/3548731548 >> 172.21.15.57:0/3548731548 conn(0x7fee900d0d60 msgr2=0x7fee900cc800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:02:51.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.918+0000 7fee951f5640 1 -- 172.21.15.57:0/3548731548 shutdown_connections 2023-10-19T13:02:51.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:51.918+0000 7fee951f5640 1 -- 172.21.15.57:0/3548731548 wait complete. 2023-10-19T13:02:51.931 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-19T13:02:51.931 DEBUG:teuthology.orchestra.run.smithi057:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-19T13:02:51.972 DEBUG:teuthology.orchestra.run.smithi179:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-19T13:02:52.025 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-19T13:02:52.026 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T13:02:52.026 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T13:02:52.042 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055614,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=358 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248260k,nr_inodes=812065,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-19T13:02:52.043 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-19T13:02:52.043 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T13:02:52.061 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054833,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=284 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3247632k,nr_inodes=811908,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-19T13:02:52.061 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T13:02:52.061 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T13:02:52.099 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055614,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=358 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248260k,nr_inodes=812065,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-19T13:02:52.100 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-19T13:02:52.100 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T13:02:52.119 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054833,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=284 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3247632k,nr_inodes=811908,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-19T13:02:52.120 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-19T13:02:52.120 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T13:02:52.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.248+0000 7f002cb83640 1 -- 172.21.15.57:0/3605012472 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f00280afa20 msgr2=0x7f0028158300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:52.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.248+0000 7f002cb83640 1 --2- 172.21.15.57:0/3605012472 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f00280afa20 0x7f0028158300 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f0010009830 tx=0x7f001002bd60 comp rx=0 tx=0).stop 2023-10-19T13:02:52.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.248+0000 7f002cb83640 1 -- 172.21.15.57:0/3605012472 shutdown_connections 2023-10-19T13:02:52.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.248+0000 7f002cb83640 1 --2- 172.21.15.57:0/3605012472 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f00280afa20 0x7f0028158300 unknown :-1 s=CLOSED pgs=1402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:52.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.248+0000 7f002cb83640 1 -- 172.21.15.57:0/3605012472 >> 172.21.15.57:0/3605012472 conn(0x7f0028069930 msgr2=0x7f0028069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:02:52.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.248+0000 7f002cb83640 1 -- 172.21.15.57:0/3605012472 shutdown_connections 2023-10-19T13:02:52.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.248+0000 7f002cb83640 1 -- 172.21.15.57:0/3605012472 wait complete. 2023-10-19T13:02:52.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.250+0000 7f002cb83640 1 Processor -- start 2023-10-19T13:02:52.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.250+0000 7f002cb83640 1 -- start start 2023-10-19T13:02:52.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.251+0000 7f002cb83640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f00280afa20 0x7f0028145a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:52.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.251+0000 7f002cb83640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f00280b0530 0x7f002814a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:52.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.251+0000 7f002cb83640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f00280cbd20 0x7f002814cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:52.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.251+0000 7f002cb83640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0028145f40 con 0x7f00280b0530 2023-10-19T13:02:52.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.251+0000 7f002cb83640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00281f1f40 con 0x7f00280afa20 2023-10-19T13:02:52.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.251+0000 7f002cb83640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00281f2220 con 0x7f00280cbd20 2023-10-19T13:02:52.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.251+0000 7f001dd9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f00280b0530 0x7f002814a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:52.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.251+0000 7f001dd9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f00280b0530 0x7f002814a140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43436/0 (socket says 172.21.15.57:43436) 2023-10-19T13:02:52.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.251+0000 7f001dd9b640 1 -- 172.21.15.57:0/190252811 learned_addr learned my addr 172.21.15.57:0/190252811 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:02:52.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.251+0000 7f002659c640 1 --2- 172.21.15.57:0/190252811 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f00280afa20 0x7f0028145a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:52.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.251+0000 7f0026d9d640 1 --2- 172.21.15.57:0/190252811 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f00280cbd20 0x7f002814cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:52.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.251+0000 7f001dd9b640 1 -- 172.21.15.57:0/190252811 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f00280cbd20 msgr2=0x7f002814cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:52.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.251+0000 7f001dd9b640 1 --2- 172.21.15.57:0/190252811 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f00280cbd20 0x7f002814cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:52.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.251+0000 7f001dd9b640 1 -- 172.21.15.57:0/190252811 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f00280afa20 msgr2=0x7f0028145a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:52.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.251+0000 7f001dd9b640 1 --2- 172.21.15.57:0/190252811 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f00280afa20 0x7f0028145a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:52.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.251+0000 7f001dd9b640 1 -- 172.21.15.57:0/190252811 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0010009490 con 0x7f00280b0530 2023-10-19T13:02:52.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.251+0000 7f001dd9b640 1 --2- 172.21.15.57:0/190252811 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f00280b0530 0x7f002814a140 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f0010000c00 tx=0x7f00100038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:02:52.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.251+0000 7f001f7fe640 1 -- 172.21.15.57:0/190252811 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0010003ea0 con 0x7f00280b0530 2023-10-19T13:02:52.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.251+0000 7f001f7fe640 1 -- 172.21.15.57:0/190252811 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f0010034440 con 0x7f00280b0530 2023-10-19T13:02:52.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.251+0000 7f002cb83640 1 -- 172.21.15.57:0/190252811 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0028149660 con 0x7f00280b0530 2023-10-19T13:02:52.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.251+0000 7f001f7fe640 1 -- 172.21.15.57:0/190252811 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0010034850 con 0x7f00280b0530 2023-10-19T13:02:52.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.252+0000 7f002cb83640 1 -- 172.21.15.57:0/190252811 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0028149c20 con 0x7f00280b0530 2023-10-19T13:02:52.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.253+0000 7f002cb83640 1 -- 172.21.15.57:0/190252811 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00280690e0 con 0x7f00280b0530 2023-10-19T13:02:52.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.253+0000 7f001f7fe640 1 -- 172.21.15.57:0/190252811 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 58) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f00100349b0 con 0x7f00280b0530 2023-10-19T13:02:52.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.253+0000 7f001f7fe640 1 --2- 172.21.15.57:0/190252811 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f000806af90 0x7f000806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:52.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.254+0000 7f001f7fe640 1 -- 172.21.15.57:0/190252811 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6065+0+0 (secure 0 0 0) 0x7f00100abfe0 con 0x7f00280b0530 2023-10-19T13:02:52.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.254+0000 7f002659c640 1 --2- 172.21.15.57:0/190252811 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f000806af90 0x7f000806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:52.262 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.254+0000 7f002659c640 1 --2- 172.21.15.57:0/190252811 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f000806af90 0x7f000806d450 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0018001710 tx=0x7f0018007cc0 comp rx=0 tx=0).ready entity=mgr.7540 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:02:52.262 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.258+0000 7f001f7fe640 1 -- 172.21.15.57:0/190252811 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0010076ed0 con 0x7f00280b0530 2023-10-19T13:02:52.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.469+0000 7f002cb83640 1 -- 172.21.15.57:0/190252811 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f002806f700 con 0x7f00280b0530 2023-10-19T13:02:52.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.470+0000 7f001f7fe640 1 -- 172.21.15.57:0/190252811 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) v1 ==== 94+0+3705 (secure 0 0 0) 0x7f0010042090 con 0x7f00280b0530 2023-10-19T13:02:52.473 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:02:52.473 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":46,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6611,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/2708395824","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2708395824},{"type":"v1","addr":"172.21.15.179:6836","nonce":2708395824}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":41},{"gid":7270,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/3803632191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":3803632191},{"type":"v1","addr":"172.21.15.179:6837","nonce":3803632191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":41},{"gid":7282,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6833/2259979700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2259979700},{"type":"v1","addr":"172.21.15.57:6833","nonce":2259979700}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":41},{"gid":8478,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/1250380900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":1250380900},{"type":"v1","addr":"172.21.15.57:6835","nonce":1250380900}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":46}],"filesystems":[]} 2023-10-19T13:02:52.474 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 46 2023-10-19T13:02:52.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.471+0000 7f002cb83640 1 -- 172.21.15.57:0/190252811 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f000806af90 msgr2=0x7f000806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:52.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.471+0000 7f002cb83640 1 --2- 172.21.15.57:0/190252811 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f000806af90 0x7f000806d450 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0018001710 tx=0x7f0018007cc0 comp rx=0 tx=0).stop 2023-10-19T13:02:52.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.471+0000 7f002cb83640 1 -- 172.21.15.57:0/190252811 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f00280b0530 msgr2=0x7f002814a140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:52.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.471+0000 7f002cb83640 1 --2- 172.21.15.57:0/190252811 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f00280b0530 0x7f002814a140 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f0010000c00 tx=0x7f00100038a0 comp rx=0 tx=0).stop 2023-10-19T13:02:52.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.471+0000 7f002cb83640 1 -- 172.21.15.57:0/190252811 shutdown_connections 2023-10-19T13:02:52.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.471+0000 7f002cb83640 1 --2- 172.21.15.57:0/190252811 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f000806af90 0x7f000806d450 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:52.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.471+0000 7f002cb83640 1 --2- 172.21.15.57:0/190252811 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f00280cbd20 0x7f002814cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:52.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.471+0000 7f002cb83640 1 --2- 172.21.15.57:0/190252811 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f00280b0530 0x7f002814a140 unknown :-1 s=CLOSED pgs=1403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:52.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.471+0000 7f002cb83640 1 --2- 172.21.15.57:0/190252811 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f00280afa20 0x7f0028145a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:52.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.471+0000 7f002cb83640 1 -- 172.21.15.57:0/190252811 >> 172.21.15.57:0/190252811 conn(0x7f0028069930 msgr2=0x7f002806bfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:02:52.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.472+0000 7f002cb83640 1 -- 172.21.15.57:0/190252811 shutdown_connections 2023-10-19T13:02:52.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.472+0000 7f002cb83640 1 -- 172.21.15.57:0/190252811 wait complete. 2023-10-19T13:02:52.485 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-19T13:02:52.486 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-19T13:02:52.486 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-19T13:02:52.486 INFO:teuthology.orchestra.run:waiting for 300 2023-10-19T13:02:52.487 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-19T13:02:52.487 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-19T13:02:52.487 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-19T13:02:52.487 INFO:teuthology.orchestra.run:waiting for 300 2023-10-19T13:02:52.488 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-19T13:02:52.488 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-19T13:02:52.488 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-19T13:02:52.489 INFO:teuthology.orchestra.run:waiting for 300 2023-10-19T13:02:52.489 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-19T13:02:52.489 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-19T13:02:52.489 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-19T13:02:52.490 INFO:teuthology.orchestra.run:waiting for 300 2023-10-19T13:02:52.498 INFO:tasks.ceph.mds.a:Stopped 2023-10-19T13:02:52.498 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-19T13:02:52.499 INFO:tasks.ceph.mds.c:Stopped 2023-10-19T13:02:52.500 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-19T13:02:52.502 INFO:tasks.ceph.mds.b:Stopped 2023-10-19T13:02:52.503 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-19T13:02:52.504 INFO:tasks.ceph.mds.d:Stopped 2023-10-19T13:02:52.504 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-19T13:02:52.540 INFO:tasks.ceph.mds.a:Started 2023-10-19T13:02:52.541 DEBUG:teuthology.parallel:result is None 2023-10-19T13:02:52.542 INFO:tasks.ceph.mds.c:Started 2023-10-19T13:02:52.543 DEBUG:teuthology.parallel:result is None 2023-10-19T13:02:52.546 INFO:tasks.ceph.mds.b:Started 2023-10-19T13:02:52.546 DEBUG:teuthology.parallel:result is None 2023-10-19T13:02:52.547 INFO:tasks.ceph.mds.d:Started 2023-10-19T13:02:52.548 DEBUG:teuthology.parallel:result is None 2023-10-19T13:02:52.548 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-19T13:02:52.621 INFO:tasks.ceph.mds.a.smithi057.stdout:starting mds.a at 2023-10-19T13:02:52.621 INFO:tasks.ceph.mds.c.smithi057.stdout:starting mds.c at 2023-10-19T13:02:52.626 INFO:tasks.ceph.mds.b.smithi179.stdout:starting mds.b at 2023-10-19T13:02:52.627 INFO:tasks.ceph.mds.d.smithi179.stdout:starting mds.d at 2023-10-19T13:02:52.664 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.661+0000 7fbcdfdd7640 1 -- 172.21.15.57:0/654033444 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbcc40047c0 msgr2=0x7fbcc4004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:52.664 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.661+0000 7fbcdfdd7640 1 --2- 172.21.15.57:0/654033444 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbcc40047c0 0x7fbcc4004c40 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7fbcc8006fa0 tx=0x7fbcc40057f0 comp rx=0 tx=0).stop 2023-10-19T13:02:52.664 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.661+0000 7fbcdfdd7640 1 -- 172.21.15.57:0/654033444 shutdown_connections 2023-10-19T13:02:52.664 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.661+0000 7fbcdfdd7640 1 --2- 172.21.15.57:0/654033444 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbcc40047c0 0x7fbcc4004c40 unknown :-1 s=CLOSED pgs=1411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:52.665 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.661+0000 7fbcdfdd7640 1 -- 172.21.15.57:0/654033444 >> 172.21.15.57:0/654033444 conn(0x7fbcd809a010 msgr2=0x7fbcd80d0de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:02:52.665 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.661+0000 7fbcdfdd7640 1 -- 172.21.15.57:0/654033444 shutdown_connections 2023-10-19T13:02:52.665 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.661+0000 7fbcdfdd7640 1 -- 172.21.15.57:0/654033444 wait complete. 2023-10-19T13:02:52.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.662+0000 7fbcdfdd7640 1 Processor -- start 2023-10-19T13:02:52.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.662+0000 7fbcdfdd7640 1 -- start start 2023-10-19T13:02:52.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.663+0000 7fbcdfdd7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbcc40047c0 0x7fbcd81f09c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:52.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.663+0000 7fbcdfdd7640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbcd81f0f00 0x7fbcd80c7eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:52.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.663+0000 7fbcdfdd7640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbcd80c83f0 0x7fbcd815d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:52.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.663+0000 7fbcdfdd7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd81f1360 con 0x7fbcc40047c0 2023-10-19T13:02:52.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.663+0000 7fbcdfdd7640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd80c8870 con 0x7fbcd80c83f0 2023-10-19T13:02:52.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.663+0000 7fbcdfdd7640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd80c4f90 con 0x7fbcd81f0f00 2023-10-19T13:02:52.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.663+0000 7fbcddb73640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbcc40047c0 0x7fbcd81f09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:52.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.663+0000 7fbcddb73640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbcc40047c0 0x7fbcd81f09c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43458/0 (socket says 172.21.15.57:43458) 2023-10-19T13:02:52.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.663+0000 7fbcddb73640 1 -- 172.21.15.57:0/1387951518 learned_addr learned my addr 172.21.15.57:0/1387951518 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:02:52.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.663+0000 7fbcdd372640 1 --2- 172.21.15.57:0/1387951518 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbcd81f0f00 0x7fbcd80c7eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:52.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.663+0000 7fbcde374640 1 --2- 172.21.15.57:0/1387951518 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbcd80c83f0 0x7fbcd815d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:52.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.663+0000 7fbcddb73640 1 -- 172.21.15.57:0/1387951518 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbcd81f0f00 msgr2=0x7fbcd80c7eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:52.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.663+0000 7fbcddb73640 1 --2- 172.21.15.57:0/1387951518 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbcd81f0f00 0x7fbcd80c7eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:52.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.663+0000 7fbcddb73640 1 -- 172.21.15.57:0/1387951518 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbcd80c83f0 msgr2=0x7fbcd815d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:52.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.663+0000 7fbcddb73640 1 --2- 172.21.15.57:0/1387951518 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbcd80c83f0 0x7fbcd815d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:52.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.663+0000 7fbcddb73640 1 -- 172.21.15.57:0/1387951518 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcd80c5200 con 0x7fbcc40047c0 2023-10-19T13:02:52.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.663+0000 7fbcddb73640 1 --2- 172.21.15.57:0/1387951518 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbcc40047c0 0x7fbcd81f09c0 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7fbcd4003e40 tx=0x7fbcd4001630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:02:52.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.663+0000 7fbcce7fc640 1 -- 172.21.15.57:0/1387951518 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcd4011840 con 0x7fbcc40047c0 2023-10-19T13:02:52.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.664+0000 7fbcce7fc640 1 -- 172.21.15.57:0/1387951518 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fbcd40119a0 con 0x7fbcc40047c0 2023-10-19T13:02:52.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.664+0000 7fbcdfdd7640 1 -- 172.21.15.57:0/1387951518 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcd80c54e0 con 0x7fbcc40047c0 2023-10-19T13:02:52.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.664+0000 7fbcce7fc640 1 -- 172.21.15.57:0/1387951518 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcd4011d90 con 0x7fbcc40047c0 2023-10-19T13:02:52.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.664+0000 7fbcdfdd7640 1 -- 172.21.15.57:0/1387951518 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcd80c5aa0 con 0x7fbcc40047c0 2023-10-19T13:02:52.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.665+0000 7fbcdfdd7640 1 -- 172.21.15.57:0/1387951518 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcd80690e0 con 0x7fbcc40047c0 2023-10-19T13:02:52.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.665+0000 7fbcce7fc640 1 -- 172.21.15.57:0/1387951518 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 58) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fbcd40103c0 con 0x7fbcc40047c0 2023-10-19T13:02:52.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.665+0000 7fbcce7fc640 1 --2- 172.21.15.57:0/1387951518 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7fbcbc06af90 0x7fbcbc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:52.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.666+0000 7fbcce7fc640 1 -- 172.21.15.57:0/1387951518 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6065+0+0 (secure 0 0 0) 0x7fbcd408a8c0 con 0x7fbcc40047c0 2023-10-19T13:02:52.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.666+0000 7fbcdd372640 1 --2- 172.21.15.57:0/1387951518 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7fbcbc06af90 0x7fbcbc06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:52.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.666+0000 7fbcdd372640 1 --2- 172.21.15.57:0/1387951518 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7fbcbc06af90 0x7fbcbc06d450 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fbcc8034000 tx=0x7fbcc8001a00 comp rx=0 tx=0).ready entity=mgr.7540 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:02:52.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.670+0000 7fbcce7fc640 1 -- 172.21.15.57:0/1387951518 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fbcd4055560 con 0x7fbcc40047c0 2023-10-19T13:02:52.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:52.858+0000 7fbcdfdd7640 1 -- 172.21.15.57:0/1387951518 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fbcd80ada50 con 0x7fbcc40047c0 2023-10-19T13:02:54.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.230+0000 7fbcce7fc640 1 -- 172.21.15.57:0/1387951518 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v126) v1 ==== 97+0+0 (secure 0 0 0) 0x7fbcd401f030 con 0x7fbcc40047c0 2023-10-19T13:02:54.234 INFO:teuthology.orchestra.run.smithi057.stderr: removed all blocklist entries 2023-10-19T13:02:54.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.233+0000 7fbcdfdd7640 1 -- 172.21.15.57:0/1387951518 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7fbcbc06af90 msgr2=0x7fbcbc06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:54.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.233+0000 7fbcdfdd7640 1 --2- 172.21.15.57:0/1387951518 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7fbcbc06af90 0x7fbcbc06d450 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fbcc8034000 tx=0x7fbcc8001a00 comp rx=0 tx=0).stop 2023-10-19T13:02:54.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.233+0000 7fbcdfdd7640 1 -- 172.21.15.57:0/1387951518 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbcc40047c0 msgr2=0x7fbcd81f09c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:54.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.233+0000 7fbcdfdd7640 1 --2- 172.21.15.57:0/1387951518 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbcc40047c0 0x7fbcd81f09c0 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7fbcd4003e40 tx=0x7fbcd4001630 comp rx=0 tx=0).stop 2023-10-19T13:02:54.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.233+0000 7fbcdfdd7640 1 -- 172.21.15.57:0/1387951518 shutdown_connections 2023-10-19T13:02:54.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.233+0000 7fbcdfdd7640 1 --2- 172.21.15.57:0/1387951518 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7fbcbc06af90 0x7fbcbc06d450 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:54.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.233+0000 7fbcdfdd7640 1 --2- 172.21.15.57:0/1387951518 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbcd80c83f0 0x7fbcd815d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:54.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.233+0000 7fbcdfdd7640 1 --2- 172.21.15.57:0/1387951518 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbcd81f0f00 0x7fbcd80c7eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:54.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.233+0000 7fbcdfdd7640 1 --2- 172.21.15.57:0/1387951518 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbcc40047c0 0x7fbcd81f09c0 unknown :-1 s=CLOSED pgs=1412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:54.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.233+0000 7fbcdfdd7640 1 -- 172.21.15.57:0/1387951518 >> 172.21.15.57:0/1387951518 conn(0x7fbcd809a010 msgr2=0x7fbcd806b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:02:54.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.233+0000 7fbcdfdd7640 1 -- 172.21.15.57:0/1387951518 shutdown_connections 2023-10-19T13:02:54.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.233+0000 7fbcdfdd7640 1 -- 172.21.15.57:0/1387951518 wait complete. 2023-10-19T13:02:54.249 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-19T13:02:54.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.364+0000 7fef461e7640 1 -- 172.21.15.57:0/1953002449 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fef280047c0 msgr2=0x7fef28004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:54.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.364+0000 7fef461e7640 1 --2- 172.21.15.57:0/1953002449 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fef280047c0 0x7fef28004c40 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7fef34009b50 tx=0x7fef280057f0 comp rx=0 tx=0).stop 2023-10-19T13:02:54.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.364+0000 7fef3ffff640 1 -- 172.21.15.57:0/1953002449 reap_dead start 2023-10-19T13:02:54.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.364+0000 7fef461e7640 1 -- 172.21.15.57:0/1953002449 shutdown_connections 2023-10-19T13:02:54.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.364+0000 7fef461e7640 1 -- 172.21.15.57:0/1953002449 >> 172.21.15.57:0/1953002449 conn(0x7fef40069930 msgr2=0x7fef40069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:02:54.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.364+0000 7fef461e7640 1 -- 172.21.15.57:0/1953002449 shutdown_connections 2023-10-19T13:02:54.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.364+0000 7fef461e7640 1 -- 172.21.15.57:0/1953002449 wait complete. 2023-10-19T13:02:54.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.365+0000 7fef461e7640 1 Processor -- start 2023-10-19T13:02:54.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.366+0000 7fef461e7640 1 -- start start 2023-10-19T13:02:54.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.366+0000 7fef461e7640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fef280047c0 0x7fef401f4b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:54.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.366+0000 7fef461e7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fef400ad260 0x7fef4014d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:54.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.366+0000 7fef461e7640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fef40155f40 0x7fef401563c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:54.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.366+0000 7fef461e7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef401f5070 con 0x7fef400ad260 2023-10-19T13:02:54.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.366+0000 7fef461e7640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef401f5b00 con 0x7fef40155f40 2023-10-19T13:02:54.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.366+0000 7fef461e7640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef4015c780 con 0x7fef280047c0 2023-10-19T13:02:54.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.366+0000 7fef3effd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fef400ad260 0x7fef4014d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:54.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.366+0000 7fef3effd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fef400ad260 0x7fef4014d360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43478/0 (socket says 172.21.15.57:43478) 2023-10-19T13:02:54.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.366+0000 7fef3effd640 1 -- 172.21.15.57:0/2557013110 learned_addr learned my addr 172.21.15.57:0/2557013110 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:02:54.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.367+0000 7fef3f7fe640 1 --2- 172.21.15.57:0/2557013110 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fef280047c0 0x7fef401f4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:54.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.367+0000 7fef3ffff640 1 --2- 172.21.15.57:0/2557013110 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fef40155f40 0x7fef401563c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:54.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.367+0000 7fef3effd640 1 -- 172.21.15.57:0/2557013110 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fef280047c0 msgr2=0x7fef401f4b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:54.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.367+0000 7fef3effd640 1 --2- 172.21.15.57:0/2557013110 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fef280047c0 0x7fef401f4b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:54.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.367+0000 7fef3effd640 1 -- 172.21.15.57:0/2557013110 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fef40155f40 msgr2=0x7fef401563c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:54.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.367+0000 7fef3effd640 1 --2- 172.21.15.57:0/2557013110 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fef40155f40 0x7fef401563c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:54.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.367+0000 7fef3effd640 1 -- 172.21.15.57:0/2557013110 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef40070300 con 0x7fef400ad260 2023-10-19T13:02:54.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.367+0000 7fef3effd640 1 --2- 172.21.15.57:0/2557013110 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fef400ad260 0x7fef4014d360 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7fef34009c70 tx=0x7fef34033c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:02:54.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.367+0000 7fef27fff640 1 -- 172.21.15.57:0/2557013110 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef34004460 con 0x7fef400ad260 2023-10-19T13:02:54.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.367+0000 7fef27fff640 1 -- 172.21.15.57:0/2557013110 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fef34033e60 con 0x7fef400ad260 2023-10-19T13:02:54.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.367+0000 7fef27fff640 1 -- 172.21.15.57:0/2557013110 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef34003830 con 0x7fef400ad260 2023-10-19T13:02:54.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.367+0000 7fef461e7640 1 -- 172.21.15.57:0/2557013110 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef400ad6e0 con 0x7fef400ad260 2023-10-19T13:02:54.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.367+0000 7fef461e7640 1 -- 172.21.15.57:0/2557013110 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef401f5260 con 0x7fef400ad260 2023-10-19T13:02:54.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.368+0000 7fef461e7640 1 -- 172.21.15.57:0/2557013110 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef4015c8f0 con 0x7fef400ad260 2023-10-19T13:02:54.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.369+0000 7fef27fff640 1 -- 172.21.15.57:0/2557013110 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 58) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fef34003990 con 0x7fef400ad260 2023-10-19T13:02:54.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.369+0000 7fef27fff640 1 --2- 172.21.15.57:0/2557013110 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7fef2c06af60 0x7fef2c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:54.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.369+0000 7fef27fff640 1 -- 172.21.15.57:0/2557013110 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5205+0+0 (secure 0 0 0) 0x7fef340ac4b0 con 0x7fef400ad260 2023-10-19T13:02:54.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.369+0000 7fef3f7fe640 1 --2- 172.21.15.57:0/2557013110 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7fef2c06af60 0x7fef2c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:54.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.370+0000 7fef3f7fe640 1 --2- 172.21.15.57:0/2557013110 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7fef2c06af60 0x7fef2c06d420 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fef30006c60 tx=0x7fef30001e70 comp rx=0 tx=0).ready entity=mgr.7540 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:02:54.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.373+0000 7fef27fff640 1 -- 172.21.15.57:0/2557013110 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fef34077540 con 0x7fef400ad260 2023-10-19T13:02:54.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.619+0000 7fef461e7640 1 -- 172.21.15.57:0/2557013110 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fef4015c370 con 0x7fef400ad260 2023-10-19T13:02:54.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.620+0000 7fef27fff640 1 -- 172.21.15.57:0/2557013110 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v26) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fef3407ab60 con 0x7fef400ad260 2023-10-19T13:02:54.623 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:02:54.623 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2023-10-19T13:02:54.623 INFO:teuthology.orchestra.run.smithi057.stdout: "auth_dump": [ 2023-10-19T13:02:54.624 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:02:54.624 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "mds.a", 2023-10-19T13:02:54.624 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBjJDFlmbVANhAAP/ynHlU/OxaRq5lQoAa8Ng==", 2023-10-19T13:02:54.624 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:02:54.624 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T13:02:54.624 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow *", 2023-10-19T13:02:54.625 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow *", 2023-10-19T13:02:54.625 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:02:54.625 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:02:54.625 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:02:54.625 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:02:54.626 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "mds.b", 2023-10-19T13:02:54.626 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFl9l7RBxAAC0geI2tJ1SIZZr1++kS/BA==", 2023-10-19T13:02:54.626 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:02:54.626 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T13:02:54.626 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow *", 2023-10-19T13:02:54.627 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow *", 2023-10-19T13:02:54.627 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:02:54.627 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:02:54.627 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:02:54.627 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:02:54.627 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "mds.c", 2023-10-19T13:02:54.628 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFlFkaoAhAA/PBizkJ4bm958oP4CmPuIQ==", 2023-10-19T13:02:54.628 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:02:54.628 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T13:02:54.628 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow *", 2023-10-19T13:02:54.628 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow *", 2023-10-19T13:02:54.629 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:02:54.629 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:02:54.629 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:02:54.629 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:02:54.629 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "mds.d", 2023-10-19T13:02:54.630 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFlx8skDxAAAfCHTYtYJgJik0WHSWeQfQ==", 2023-10-19T13:02:54.630 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:02:54.630 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T13:02:54.630 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow *", 2023-10-19T13:02:54.630 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow *", 2023-10-19T13:02:54.630 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:02:54.631 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:02:54.631 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:02:54.631 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:02:54.631 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.0", 2023-10-19T13:02:54.631 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBrJDFlCo9sBxAAQN2cID4nCQgRwGaGFEV8LQ==", 2023-10-19T13:02:54.632 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:02:54.632 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T13:02:54.632 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T13:02:54.632 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:02:54.632 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:02:54.633 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:02:54.633 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:02:54.633 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.1", 2023-10-19T13:02:54.633 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBuJDFl4fsLCBAAp3Pi80yZtc5sju0GisEuVA==", 2023-10-19T13:02:54.633 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:02:54.633 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T13:02:54.634 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T13:02:54.634 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:02:54.634 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:02:54.634 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:02:54.634 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:02:54.635 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.2", 2023-10-19T13:02:54.635 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBxJDFl+KAKCBAAd4N9cluARxPVfBLU+7Hd7A==", 2023-10-19T13:02:54.635 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:02:54.635 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T13:02:54.635 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T13:02:54.636 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:02:54.636 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:02:54.636 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:02:54.636 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:02:54.636 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.3", 2023-10-19T13:02:54.636 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQB0JDFlU/toCBAAP0Ou1zcHKNx5SfbybJkh4Q==", 2023-10-19T13:02:54.637 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:02:54.637 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T13:02:54.637 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T13:02:54.637 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:02:54.637 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:02:54.638 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:02:54.638 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:02:54.638 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.4", 2023-10-19T13:02:54.638 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQB6JDFlc6stEBAALkgNyxoiCTzZnKpBSPxI6w==", 2023-10-19T13:02:54.638 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:02:54.638 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T13:02:54.639 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T13:02:54.639 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:02:54.639 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:02:54.639 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:02:54.639 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:02:54.640 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.5", 2023-10-19T13:02:54.640 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQB9JDFlI3cNDhAAuG3tHJ6RvPwAzPE76wTIVQ==", 2023-10-19T13:02:54.640 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:02:54.640 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T13:02:54.640 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T13:02:54.641 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:02:54.641 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:02:54.641 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:02:54.641 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:02:54.641 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.6", 2023-10-19T13:02:54.641 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQCAJDFlLPumCxAA/iy08t/2N9rXfn71KtbSQw==", 2023-10-19T13:02:54.642 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:02:54.642 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T13:02:54.642 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T13:02:54.642 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:02:54.642 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:02:54.643 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:02:54.643 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:02:54.643 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.7", 2023-10-19T13:02:54.643 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQCDJDFlHmrbCRAA+hpWygGIJydfU6cHWlfx/Q==", 2023-10-19T13:02:54.643 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:02:54.644 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T13:02:54.644 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T13:02:54.644 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:02:54.644 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:02:54.644 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:02:54.644 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:02:54.645 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "client.0", 2023-10-19T13:02:54.645 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFlg9yuExAATxiWDfS+p07G1F0pORbJEg==", 2023-10-19T13:02:54.645 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:02:54.645 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T13:02:54.645 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow r", 2023-10-19T13:02:54.646 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-19T13:02:54.646 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:02:54.646 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:02:54.646 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:02:54.646 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "client.1", 2023-10-19T13:02:54.647 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFl9IwYGxAAHPituiJSPkHDuD0QyG4Hww==", 2023-10-19T13:02:54.647 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:02:54.647 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T13:02:54.647 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow r", 2023-10-19T13:02:54.647 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow rw", 2023-10-19T13:02:54.648 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow rwx" 2023-10-19T13:02:54.648 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:02:54.648 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:02:54.648 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:02:54.648 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "client.2", 2023-10-19T13:02:54.648 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFlMd+AFxAAS/UTESDt76GmrhhN2o/Fdw==", 2023-10-19T13:02:54.649 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:02:54.649 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T13:02:54.649 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow r", 2023-10-19T13:02:54.649 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow rw", 2023-10-19T13:02:54.649 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow rwx" 2023-10-19T13:02:54.650 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:02:54.650 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:02:54.650 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:02:54.650 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "client.3", 2023-10-19T13:02:54.650 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFlHZ3fIBAA42B7n8opLS5smXIwJyVYzw==", 2023-10-19T13:02:54.650 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:02:54.651 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T13:02:54.651 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow r", 2023-10-19T13:02:54.651 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow rw", 2023-10-19T13:02:54.651 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow rwx" 2023-10-19T13:02:54.651 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:02:54.652 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:02:54.652 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:02:54.652 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "client.admin", 2023-10-19T13:02:54.652 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBjJDFlBgasFxAAMA24j8KszcXMEpYRQDXj4g==", 2023-10-19T13:02:54.652 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:02:54.653 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow *", 2023-10-19T13:02:54.653 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow *", 2023-10-19T13:02:54.653 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow *", 2023-10-19T13:02:54.653 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:02:54.653 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:02:54.653 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:02:54.654 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:02:54.654 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "mgr.x", 2023-10-19T13:02:54.654 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBjJDFllFBfMhAA5+KPXWmYZqcESmLObaUbBA==", 2023-10-19T13:02:54.654 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:02:54.654 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow *", 2023-10-19T13:02:54.655 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile mgr", 2023-10-19T13:02:54.655 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:02:54.655 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:02:54.655 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:02:54.655 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:02:54.656 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "mgr.y", 2023-10-19T13:02:54.656 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBjJDFlxRzJLRAAGjG58o7qXAHvIEIRYwl6QA==", 2023-10-19T13:02:54.656 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:02:54.656 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow *", 2023-10-19T13:02:54.656 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile mgr", 2023-10-19T13:02:54.656 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:02:54.657 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:02:54.657 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:02:54.657 INFO:teuthology.orchestra.run.smithi057.stdout: ] 2023-10-19T13:02:54.657 INFO:teuthology.orchestra.run.smithi057.stdout:} 2023-10-19T13:02:54.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.622+0000 7fef461e7640 1 -- 172.21.15.57:0/2557013110 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7fef2c06af60 msgr2=0x7fef2c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:54.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.622+0000 7fef461e7640 1 --2- 172.21.15.57:0/2557013110 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7fef2c06af60 0x7fef2c06d420 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fef30006c60 tx=0x7fef30001e70 comp rx=0 tx=0).stop 2023-10-19T13:02:54.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.622+0000 7fef461e7640 1 -- 172.21.15.57:0/2557013110 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fef400ad260 msgr2=0x7fef4014d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:54.660 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.622+0000 7fef461e7640 1 --2- 172.21.15.57:0/2557013110 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fef400ad260 0x7fef4014d360 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7fef34009c70 tx=0x7fef34033c30 comp rx=0 tx=0).stop 2023-10-19T13:02:54.660 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.622+0000 7fef461e7640 1 -- 172.21.15.57:0/2557013110 shutdown_connections 2023-10-19T13:02:54.660 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.622+0000 7fef461e7640 1 --2- 172.21.15.57:0/2557013110 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7fef2c06af60 0x7fef2c06d420 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:54.660 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.622+0000 7fef461e7640 1 --2- 172.21.15.57:0/2557013110 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fef40155f40 0x7fef401563c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:54.660 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.622+0000 7fef461e7640 1 --2- 172.21.15.57:0/2557013110 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fef400ad260 0x7fef4014d360 unknown :-1 s=CLOSED pgs=1415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:54.661 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.622+0000 7fef461e7640 1 --2- 172.21.15.57:0/2557013110 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fef280047c0 0x7fef401f4b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:54.661 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.622+0000 7fef461e7640 1 -- 172.21.15.57:0/2557013110 >> 172.21.15.57:0/2557013110 conn(0x7fef40069930 msgr2=0x7fef40158470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:02:54.661 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.622+0000 7fef461e7640 1 -- 172.21.15.57:0/2557013110 shutdown_connections 2023-10-19T13:02:54.661 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.622+0000 7fef461e7640 1 -- 172.21.15.57:0/2557013110 wait complete. 2023-10-19T13:02:54.662 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T13:02:54.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.769+0000 7f1149439640 1 -- 172.21.15.57:0/2881144 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f114406af00 msgr2=0x7f114406b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:54.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.769+0000 7f1149439640 1 --2- 172.21.15.57:0/2881144 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f114406af00 0x7f114406b2e0 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f1134005fd0 tx=0x7f113402bd10 comp rx=0 tx=0).stop 2023-10-19T13:02:54.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.769+0000 7f1149439640 1 -- 172.21.15.57:0/2881144 shutdown_connections 2023-10-19T13:02:54.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.769+0000 7f1149439640 1 --2- 172.21.15.57:0/2881144 >> v2:172.21.15.179:3300/0 conn(0x7f11440c96b0 0x7f11440c9b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:54.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.769+0000 7f1149439640 1 --2- 172.21.15.57:0/2881144 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f114406af00 0x7f114406b2e0 unknown :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:54.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.769+0000 7f1149439640 1 -- 172.21.15.57:0/2881144 >> 172.21.15.57:0/2881144 conn(0x7f1144069930 msgr2=0x7f1144069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:02:54.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.769+0000 7f1149439640 1 -- 172.21.15.57:0/2881144 shutdown_connections 2023-10-19T13:02:54.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.769+0000 7f1149439640 1 -- 172.21.15.57:0/2881144 wait complete. 2023-10-19T13:02:54.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.770+0000 7f1149439640 1 Processor -- start 2023-10-19T13:02:54.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.770+0000 7f1149439640 1 -- start start 2023-10-19T13:02:54.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.771+0000 7f1149439640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f114406af00 0x7f11441f05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:54.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.771+0000 7f1149439640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f11440ab500 0x7f11440ae700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:54.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.771+0000 7f1149439640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f11440c96b0 0x7f114415a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:54.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.771+0000 7f1149439640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11440b0040 con 0x7f114406af00 2023-10-19T13:02:54.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.771+0000 7f1149439640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11440af050 con 0x7f11440ab500 2023-10-19T13:02:54.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.771+0000 7f1149439640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11440af330 con 0x7f11440c96b0 2023-10-19T13:02:54.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.771+0000 7f1142ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f114406af00 0x7f11441f05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:54.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.771+0000 7f1142ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f114406af00 0x7f11441f05e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43486/0 (socket says 172.21.15.57:43486) 2023-10-19T13:02:54.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.771+0000 7f1142ffd640 1 -- 172.21.15.57:0/4214186865 learned_addr learned my addr 172.21.15.57:0/4214186865 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:02:54.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.771+0000 7f113a7fc640 1 --2- 172.21.15.57:0/4214186865 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f11440ab500 0x7f11440ae700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:54.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.771+0000 7f11437fe640 1 --2- 172.21.15.57:0/4214186865 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f11440c96b0 0x7f114415a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:54.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.771+0000 7f1142ffd640 1 -- 172.21.15.57:0/4214186865 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f11440c96b0 msgr2=0x7f114415a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:54.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.771+0000 7f1142ffd640 1 --2- 172.21.15.57:0/4214186865 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f11440c96b0 0x7f114415a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:54.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.771+0000 7f1142ffd640 1 -- 172.21.15.57:0/4214186865 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f11440ab500 msgr2=0x7f11440ae700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:54.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.771+0000 7f1142ffd640 1 --2- 172.21.15.57:0/4214186865 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f11440ab500 0x7f11440ae700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:54.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.771+0000 7f1142ffd640 1 -- 172.21.15.57:0/4214186865 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1134009560 con 0x7f114406af00 2023-10-19T13:02:54.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.771+0000 7f11437fe640 1 --2- 172.21.15.57:0/4214186865 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f11440c96b0 0x7f114415a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T13:02:54.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.772+0000 7f1142ffd640 1 --2- 172.21.15.57:0/4214186865 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f114406af00 0x7f11441f05e0 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7f1134009850 tx=0x7f113402d9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:02:54.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.772+0000 7f113bfff640 1 -- 172.21.15.57:0/4214186865 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f113402dde0 con 0x7f114406af00 2023-10-19T13:02:54.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.772+0000 7f113bfff640 1 -- 172.21.15.57:0/4214186865 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f11340036e0 con 0x7f114406af00 2023-10-19T13:02:54.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.772+0000 7f1149439640 1 -- 172.21.15.57:0/4214186865 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11440af610 con 0x7f114406af00 2023-10-19T13:02:54.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.772+0000 7f113bfff640 1 -- 172.21.15.57:0/4214186865 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1134003a90 con 0x7f114406af00 2023-10-19T13:02:54.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.772+0000 7f1149439640 1 -- 172.21.15.57:0/4214186865 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11440aec40 con 0x7f114406af00 2023-10-19T13:02:54.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.773+0000 7f1149439640 1 -- 172.21.15.57:0/4214186865 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11440ac320 con 0x7f114406af00 2023-10-19T13:02:54.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.773+0000 7f113bfff640 1 -- 172.21.15.57:0/4214186865 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 58) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f1134003bf0 con 0x7f114406af00 2023-10-19T13:02:54.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.774+0000 7f113bfff640 1 --2- 172.21.15.57:0/4214186865 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f112406af60 0x7f112406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:54.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.774+0000 7f113bfff640 1 -- 172.21.15.57:0/4214186865 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f11340ac120 con 0x7f114406af00 2023-10-19T13:02:54.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.774+0000 7f113a7fc640 1 --2- 172.21.15.57:0/4214186865 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f112406af60 0x7f112406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:54.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.774+0000 7f113a7fc640 1 --2- 172.21.15.57:0/4214186865 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f112406af60 0x7f112406d420 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f112c005fd0 tx=0x7f112c005850 comp rx=0 tx=0).ready entity=mgr.7540 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:02:54.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.778+0000 7f113bfff640 1 -- 172.21.15.57:0/4214186865 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f1134077330 con 0x7f114406af00 2023-10-19T13:02:54.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.954+0000 7f1149439640 1 -- 172.21.15.57:0/4214186865 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f11440690e0 con 0x7f114406af00 2023-10-19T13:02:54.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.955+0000 7f113bfff640 1 -- 172.21.15.57:0/4214186865 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v126) v1 ==== 74+0+13127 (secure 0 0 0) 0x7f113407a950 con 0x7f114406af00 2023-10-19T13:02:54.958 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:02:54.958 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":126,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T13:02:54.230108+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T13:02:54.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.957+0000 7f1149439640 1 -- 172.21.15.57:0/4214186865 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f112406af60 msgr2=0x7f112406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:54.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.957+0000 7f1149439640 1 --2- 172.21.15.57:0/4214186865 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f112406af60 0x7f112406d420 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f112c005fd0 tx=0x7f112c005850 comp rx=0 tx=0).stop 2023-10-19T13:02:54.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.957+0000 7f1149439640 1 -- 172.21.15.57:0/4214186865 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f114406af00 msgr2=0x7f11441f05e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:54.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.957+0000 7f1149439640 1 --2- 172.21.15.57:0/4214186865 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f114406af00 0x7f11441f05e0 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7f1134009850 tx=0x7f113402d9e0 comp rx=0 tx=0).stop 2023-10-19T13:02:54.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.957+0000 7f1149439640 1 -- 172.21.15.57:0/4214186865 shutdown_connections 2023-10-19T13:02:54.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.957+0000 7f1149439640 1 --2- 172.21.15.57:0/4214186865 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f112406af60 0x7f112406d420 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:54.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.957+0000 7f1149439640 1 --2- 172.21.15.57:0/4214186865 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f11440c96b0 0x7f114415a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:54.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.957+0000 7f1149439640 1 --2- 172.21.15.57:0/4214186865 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f11440ab500 0x7f11440ae700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:54.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.957+0000 7f1149439640 1 --2- 172.21.15.57:0/4214186865 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f114406af00 0x7f11441f05e0 unknown :-1 s=CLOSED pgs=1417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:54.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.957+0000 7f1149439640 1 -- 172.21.15.57:0/4214186865 >> 172.21.15.57:0/4214186865 conn(0x7f1144069930 msgr2=0x7f114413b1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:02:54.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.957+0000 7f1149439640 1 -- 172.21.15.57:0/4214186865 shutdown_connections 2023-10-19T13:02:54.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:54.957+0000 7f1149439640 1 -- 172.21.15.57:0/4214186865 wait complete. 2023-10-19T13:02:54.971 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-19T12:44:07.371186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-19T13:02:54.971 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-19T13:02:55.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.080+0000 7f97baa83640 1 -- 172.21.15.57:0/3129306393 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f97b40ad370 msgr2=0x7f97b40ad800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:55.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.080+0000 7f97baa83640 1 --2- 172.21.15.57:0/3129306393 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f97b40ad370 0x7f97b40ad800 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f97ac005fd0 tx=0x7f97ac02c0e0 comp rx=0 tx=0).stop 2023-10-19T13:02:55.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.081+0000 7f97baa83640 1 -- 172.21.15.57:0/3129306393 shutdown_connections 2023-10-19T13:02:55.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.081+0000 7f97baa83640 1 --2- 172.21.15.57:0/3129306393 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f97b406f0a0 0x7f97b406ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:55.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.081+0000 7f97baa83640 1 --2- 172.21.15.57:0/3129306393 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f97b40ad370 0x7f97b40ad800 unknown :-1 s=CLOSED pgs=1419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:55.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.081+0000 7f97baa83640 1 -- 172.21.15.57:0/3129306393 >> 172.21.15.57:0/3129306393 conn(0x7f97b40b0670 msgr2=0x7f97b40b0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:02:55.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.081+0000 7f97baa83640 1 -- 172.21.15.57:0/3129306393 shutdown_connections 2023-10-19T13:02:55.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.081+0000 7f97baa83640 1 -- 172.21.15.57:0/3129306393 wait complete. 2023-10-19T13:02:55.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.082+0000 7f97baa83640 1 Processor -- start 2023-10-19T13:02:55.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.082+0000 7f97baa83640 1 -- start start 2023-10-19T13:02:55.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.083+0000 7f97baa83640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f97b406d2d0 0x7f97b41526b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:55.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.083+0000 7f97baa83640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f97b406f0a0 0x7f97b40cbea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:55.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.083+0000 7f97baa83640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f97b40ad370 0x7f97b41f0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:55.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.083+0000 7f97baa83640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97b4145d30 con 0x7f97b406d2d0 2023-10-19T13:02:55.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.083+0000 7f97baa83640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97b4146010 con 0x7f97b406f0a0 2023-10-19T13:02:55.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.083+0000 7f97baa83640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97b41462f0 con 0x7f97b40ad370 2023-10-19T13:02:55.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.083+0000 7f97b881f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f97b406d2d0 0x7f97b41526b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:55.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.083+0000 7f97b881f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f97b406d2d0 0x7f97b41526b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43504/0 (socket says 172.21.15.57:43504) 2023-10-19T13:02:55.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.083+0000 7f97b881f640 1 -- 172.21.15.57:0/1661484591 learned_addr learned my addr 172.21.15.57:0/1661484591 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:02:55.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.083+0000 7f97b3fff640 1 --2- 172.21.15.57:0/1661484591 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f97b406f0a0 0x7f97b40cbea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:55.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.083+0000 7f97b9020640 1 --2- 172.21.15.57:0/1661484591 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f97b40ad370 0x7f97b41f0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:55.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.083+0000 7f97b881f640 1 -- 172.21.15.57:0/1661484591 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f97b40ad370 msgr2=0x7f97b41f0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:55.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.083+0000 7f97b881f640 1 --2- 172.21.15.57:0/1661484591 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f97b40ad370 0x7f97b41f0eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:55.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.083+0000 7f97b881f640 1 -- 172.21.15.57:0/1661484591 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f97b406f0a0 msgr2=0x7f97b40cbea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:55.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.083+0000 7f97b881f640 1 --2- 172.21.15.57:0/1661484591 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f97b406f0a0 0x7f97b40cbea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:55.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.083+0000 7f97b881f640 1 -- 172.21.15.57:0/1661484591 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97ac009840 con 0x7f97b406d2d0 2023-10-19T13:02:55.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.083+0000 7f97b3fff640 1 --2- 172.21.15.57:0/1661484591 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f97b406f0a0 0x7f97b40cbea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T13:02:55.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.083+0000 7f97b9020640 1 --2- 172.21.15.57:0/1661484591 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f97b40ad370 0x7f97b41f0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T13:02:55.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.083+0000 7f97b881f640 1 --2- 172.21.15.57:0/1661484591 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f97b406d2d0 0x7f97b41526b0 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f97a800c400 tx=0x7f97a800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:02:55.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.083+0000 7f97b17fa640 1 -- 172.21.15.57:0/1661484591 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97a8004230 con 0x7f97b406d2d0 2023-10-19T13:02:55.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.083+0000 7f97b17fa640 1 -- 172.21.15.57:0/1661484591 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f97a80044a0 con 0x7f97b406d2d0 2023-10-19T13:02:55.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.083+0000 7f97b17fa640 1 -- 172.21.15.57:0/1661484591 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97a8003850 con 0x7f97b406d2d0 2023-10-19T13:02:55.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.083+0000 7f97baa83640 1 -- 172.21.15.57:0/1661484591 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97b40ca670 con 0x7f97b406d2d0 2023-10-19T13:02:55.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.083+0000 7f97baa83640 1 -- 172.21.15.57:0/1661484591 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97b40cc3e0 con 0x7f97b406d2d0 2023-10-19T13:02:55.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.084+0000 7f97baa83640 1 -- 172.21.15.57:0/1661484591 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97b40ce1d0 con 0x7f97b406d2d0 2023-10-19T13:02:55.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.085+0000 7f97b17fa640 1 -- 172.21.15.57:0/1661484591 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 58) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f97a80039b0 con 0x7f97b406d2d0 2023-10-19T13:02:55.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.085+0000 7f97b17fa640 1 --2- 172.21.15.57:0/1661484591 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f979806af60 0x7f979806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:55.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.085+0000 7f97b17fa640 1 -- 172.21.15.57:0/1661484591 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f97a8088df0 con 0x7f97b406d2d0 2023-10-19T13:02:55.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.086+0000 7f97b3fff640 1 --2- 172.21.15.57:0/1661484591 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f979806af60 0x7f979806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:55.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.086+0000 7f97b3fff640 1 --2- 172.21.15.57:0/1661484591 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f979806af60 0x7f979806d420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f97b40c9100 tx=0x7f97a0009530 comp rx=0 tx=0).ready entity=mgr.7540 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:02:55.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.089+0000 7f97b17fa640 1 -- 172.21.15.57:0/1661484591 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f97a8053f80 con 0x7f97b406d2d0 2023-10-19T13:02:55.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.263+0000 7f97baa83640 1 -- 172.21.15.57:0/1661484591 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f97b40adb60 con 0x7f97b406d2d0 2023-10-19T13:02:55.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.263+0000 7f97b17fa640 1 -- 172.21.15.57:0/1661484591 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v126) v1 ==== 93+0+10 (secure 0 0 0) 0x7f97a8017070 con 0x7f97b406d2d0 2023-10-19T13:02:55.266 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-19T13:02:55.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.265+0000 7f97baa83640 1 -- 172.21.15.57:0/1661484591 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f979806af60 msgr2=0x7f979806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:55.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.265+0000 7f97baa83640 1 --2- 172.21.15.57:0/1661484591 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f979806af60 0x7f979806d420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f97b40c9100 tx=0x7f97a0009530 comp rx=0 tx=0).stop 2023-10-19T13:02:55.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.265+0000 7f97baa83640 1 -- 172.21.15.57:0/1661484591 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f97b406d2d0 msgr2=0x7f97b41526b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:55.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.265+0000 7f97baa83640 1 --2- 172.21.15.57:0/1661484591 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f97b406d2d0 0x7f97b41526b0 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f97a800c400 tx=0x7f97a800c8c0 comp rx=0 tx=0).stop 2023-10-19T13:02:55.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.265+0000 7f97baa83640 1 -- 172.21.15.57:0/1661484591 shutdown_connections 2023-10-19T13:02:55.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.265+0000 7f97baa83640 1 --2- 172.21.15.57:0/1661484591 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f979806af60 0x7f979806d420 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:55.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.265+0000 7f97baa83640 1 --2- 172.21.15.57:0/1661484591 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f97b40ad370 0x7f97b41f0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:55.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.265+0000 7f97baa83640 1 --2- 172.21.15.57:0/1661484591 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f97b406f0a0 0x7f97b40cbea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:55.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.265+0000 7f97baa83640 1 --2- 172.21.15.57:0/1661484591 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f97b406d2d0 0x7f97b41526b0 unknown :-1 s=CLOSED pgs=1420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:55.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.265+0000 7f97baa83640 1 -- 172.21.15.57:0/1661484591 >> 172.21.15.57:0/1661484591 conn(0x7f97b40b0670 msgr2=0x7f97b415ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:02:55.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.265+0000 7f97baa83640 1 -- 172.21.15.57:0/1661484591 shutdown_connections 2023-10-19T13:02:55.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.265+0000 7f97baa83640 1 -- 172.21.15.57:0/1661484591 wait complete. 2023-10-19T13:02:55.279 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-19T13:02:55.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.386+0000 7f41ee415640 1 -- 172.21.15.57:0/271515 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f41e80abc10 msgr2=0x7f41e80c9b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:55.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.386+0000 7f41ee415640 1 --2- 172.21.15.57:0/271515 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f41e80abc10 0x7f41e80c9b30 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7f41e0009be0 tx=0x7f41e002c0c0 comp rx=0 tx=0).stop 2023-10-19T13:02:55.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.386+0000 7f41ee415640 1 -- 172.21.15.57:0/271515 shutdown_connections 2023-10-19T13:02:55.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.386+0000 7f41ee415640 1 --2- 172.21.15.57:0/271515 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f41e80abc10 0x7f41e80c9b30 unknown :-1 s=CLOSED pgs=1422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:55.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.386+0000 7f41ee415640 1 --2- 172.21.15.57:0/271515 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f41e806af00 0x7f41e806b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:55.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.386+0000 7f41ee415640 1 -- 172.21.15.57:0/271515 >> 172.21.15.57:0/271515 conn(0x7f41e8069930 msgr2=0x7f41e8069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:02:55.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.386+0000 7f41ee415640 1 -- 172.21.15.57:0/271515 shutdown_connections 2023-10-19T13:02:55.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.386+0000 7f41ee415640 1 -- 172.21.15.57:0/271515 wait complete. 2023-10-19T13:02:55.392 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.387+0000 7f41ee415640 1 Processor -- start 2023-10-19T13:02:55.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.387+0000 7f41ee415640 1 -- start start 2023-10-19T13:02:55.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.388+0000 7f41ee415640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f41e806af00 0x7f41e80a9760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:55.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.388+0000 7f41ee415640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f41e80abc10 0x7f41e80a9ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:55.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.388+0000 7f41ee415640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f41e80c96b0 0x7f41e81ef1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:55.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.388+0000 7f41ee415640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41e80aad30 con 0x7f41e80abc10 2023-10-19T13:02:55.394 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.388+0000 7f41ee415640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41e80aa1e0 con 0x7f41e806af00 2023-10-19T13:02:55.394 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.388+0000 7f41ee415640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41e80aa490 con 0x7f41e80c96b0 2023-10-19T13:02:55.394 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.388+0000 7f41e77fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f41e80abc10 0x7f41e80a9ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:55.394 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.388+0000 7f41e77fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f41e80abc10 0x7f41e80a9ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43524/0 (socket says 172.21.15.57:43524) 2023-10-19T13:02:55.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.388+0000 7f41e77fe640 1 -- 172.21.15.57:0/1337887821 learned_addr learned my addr 172.21.15.57:0/1337887821 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:02:55.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.388+0000 7f41e7fff640 1 --2- 172.21.15.57:0/1337887821 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f41e806af00 0x7f41e80a9760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:55.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.388+0000 7f41ec9b2640 1 --2- 172.21.15.57:0/1337887821 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f41e80c96b0 0x7f41e81ef1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:55.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.388+0000 7f41e77fe640 1 -- 172.21.15.57:0/1337887821 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f41e80c96b0 msgr2=0x7f41e81ef1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:55.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.388+0000 7f41e77fe640 1 --2- 172.21.15.57:0/1337887821 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f41e80c96b0 0x7f41e81ef1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:55.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.388+0000 7f41e77fe640 1 -- 172.21.15.57:0/1337887821 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f41e806af00 msgr2=0x7f41e80a9760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:55.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.388+0000 7f41e77fe640 1 --2- 172.21.15.57:0/1337887821 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f41e806af00 0x7f41e80a9760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:55.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.388+0000 7f41e77fe640 1 -- 172.21.15.57:0/1337887821 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41e0009840 con 0x7f41e80abc10 2023-10-19T13:02:55.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.388+0000 7f41e77fe640 1 --2- 172.21.15.57:0/1337887821 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f41e80abc10 0x7f41e80a9ca0 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7f41d800c410 tx=0x7f41d800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:02:55.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.389+0000 7f41e4ff9640 1 -- 172.21.15.57:0/1337887821 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41d80035f0 con 0x7f41e80abc10 2023-10-19T13:02:55.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.389+0000 7f41e4ff9640 1 -- 172.21.15.57:0/1337887821 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f41d8003770 con 0x7f41e80abc10 2023-10-19T13:02:55.397 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.389+0000 7f41e4ff9640 1 -- 172.21.15.57:0/1337887821 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41d8003b80 con 0x7f41e80abc10 2023-10-19T13:02:55.397 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.389+0000 7f41ee415640 1 -- 172.21.15.57:0/1337887821 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41e80aa770 con 0x7f41e80abc10 2023-10-19T13:02:55.397 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.389+0000 7f41ee415640 1 -- 172.21.15.57:0/1337887821 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41e81507a0 con 0x7f41e80abc10 2023-10-19T13:02:55.397 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.390+0000 7f41ee415640 1 -- 172.21.15.57:0/1337887821 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41e80690e0 con 0x7f41e80abc10 2023-10-19T13:02:55.397 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.390+0000 7f41e4ff9640 1 -- 172.21.15.57:0/1337887821 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 58) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f41d8003ce0 con 0x7f41e80abc10 2023-10-19T13:02:55.398 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.390+0000 7f41e4ff9640 1 --2- 172.21.15.57:0/1337887821 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f41d406af90 0x7f41d406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:55.398 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.391+0000 7f41e4ff9640 1 -- 172.21.15.57:0/1337887821 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f41d8088d40 con 0x7f41e80abc10 2023-10-19T13:02:55.399 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.391+0000 7f41e7fff640 1 --2- 172.21.15.57:0/1337887821 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f41d406af90 0x7f41d406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:55.399 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.391+0000 7f41e7fff640 1 --2- 172.21.15.57:0/1337887821 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f41d406af90 0x7f41d406d450 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f41dc0058c0 tx=0x7f41dc005c80 comp rx=0 tx=0).ready entity=mgr.7540 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:02:55.399 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.395+0000 7f41e4ff9640 1 -- 172.21.15.57:0/1337887821 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f41d8053ec0 con 0x7f41e80abc10 2023-10-19T13:02:55.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.572+0000 7f41ee415640 1 -- 172.21.15.57:0/1337887821 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f41e80cc420 con 0x7f41e80abc10 2023-10-19T13:02:55.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.573+0000 7f41e4ff9640 1 -- 172.21.15.57:0/1337887821 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v126) v1 ==== 84+0+65 (secure 0 0 0) 0x7f41d8017070 con 0x7f41e80abc10 2023-10-19T13:02:55.576 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:02:55.576 INFO:teuthology.orchestra.run.smithi057.stdout:[ 2023-10-19T13:02:55.576 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:02:55.577 INFO:teuthology.orchestra.run.smithi057.stdout: "poolnum": 1, 2023-10-19T13:02:55.577 INFO:teuthology.orchestra.run.smithi057.stdout: "poolname": ".mgr" 2023-10-19T13:02:55.577 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:02:55.577 INFO:teuthology.orchestra.run.smithi057.stdout:] 2023-10-19T13:02:55.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.575+0000 7f41ee415640 1 -- 172.21.15.57:0/1337887821 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f41d406af90 msgr2=0x7f41d406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:55.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.575+0000 7f41ee415640 1 --2- 172.21.15.57:0/1337887821 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f41d406af90 0x7f41d406d450 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f41dc0058c0 tx=0x7f41dc005c80 comp rx=0 tx=0).stop 2023-10-19T13:02:55.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.575+0000 7f41ee415640 1 -- 172.21.15.57:0/1337887821 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f41e80abc10 msgr2=0x7f41e80a9ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:55.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.575+0000 7f41ee415640 1 --2- 172.21.15.57:0/1337887821 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f41e80abc10 0x7f41e80a9ca0 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7f41d800c410 tx=0x7f41d800c8d0 comp rx=0 tx=0).stop 2023-10-19T13:02:55.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.575+0000 7f41ee415640 1 -- 172.21.15.57:0/1337887821 shutdown_connections 2023-10-19T13:02:55.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.575+0000 7f41ee415640 1 --2- 172.21.15.57:0/1337887821 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f41d406af90 0x7f41d406d450 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:55.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.575+0000 7f41ee415640 1 --2- 172.21.15.57:0/1337887821 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f41e80c96b0 0x7f41e81ef1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:55.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.575+0000 7f41ee415640 1 --2- 172.21.15.57:0/1337887821 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f41e80abc10 0x7f41e80a9ca0 unknown :-1 s=CLOSED pgs=1423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:55.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.575+0000 7f41ee415640 1 --2- 172.21.15.57:0/1337887821 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f41e806af00 0x7f41e80a9760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:55.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.575+0000 7f41ee415640 1 -- 172.21.15.57:0/1337887821 >> 172.21.15.57:0/1337887821 conn(0x7f41e8069930 msgr2=0x7f41e8069d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:02:55.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.575+0000 7f41ee415640 1 -- 172.21.15.57:0/1337887821 shutdown_connections 2023-10-19T13:02:55.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.575+0000 7f41ee415640 1 -- 172.21.15.57:0/1337887821 wait complete. 2023-10-19T13:02:55.588 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-19T13:02:55.589 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-19T13:02:55.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.700+0000 7f7682d05640 1 -- 172.21.15.57:0/3718519327 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f76640039a0 msgr2=0x7f7664003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:55.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.700+0000 7f7682d05640 1 --2- 172.21.15.57:0/3718519327 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f76640039a0 0x7f7664003dc0 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7f7678009b50 tx=0x7f76640057f0 comp rx=0 tx=0).stop 2023-10-19T13:02:55.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.700+0000 7f76812a2640 1 -- 172.21.15.57:0/3718519327 reap_dead start 2023-10-19T13:02:55.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.700+0000 7f7682d05640 1 -- 172.21.15.57:0/3718519327 shutdown_connections 2023-10-19T13:02:55.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.700+0000 7f7682d05640 1 -- 172.21.15.57:0/3718519327 >> 172.21.15.57:0/3718519327 conn(0x7f767c0d0d60 msgr2=0x7f767c0d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:02:55.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.700+0000 7f7682d05640 1 -- 172.21.15.57:0/3718519327 shutdown_connections 2023-10-19T13:02:55.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.700+0000 7f7682d05640 1 -- 172.21.15.57:0/3718519327 wait complete. 2023-10-19T13:02:55.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.701+0000 7f7682d05640 1 Processor -- start 2023-10-19T13:02:55.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.701+0000 7f7682d05640 1 -- start start 2023-10-19T13:02:55.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.702+0000 7f7682d05640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f76640039a0 0x7f767c0b5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:55.707 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.702+0000 7f7682d05640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f767c0b2030 0x7f767c158620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:55.707 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.702+0000 7f7682d05640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f767c0b0ea0 0x7f767c0b1320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:55.707 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.702+0000 7f7682d05640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f767c0b6a10 con 0x7f76640039a0 2023-10-19T13:02:55.707 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.702+0000 7f7682d05640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f767c0b1860 con 0x7f767c0b2030 2023-10-19T13:02:55.707 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.702+0000 7f7682d05640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f767c0b6200 con 0x7f767c0b0ea0 2023-10-19T13:02:55.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.702+0000 7f7680aa1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f76640039a0 0x7f767c0b5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:55.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.702+0000 7f7680aa1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f76640039a0 0x7f767c0b5cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43532/0 (socket says 172.21.15.57:43532) 2023-10-19T13:02:55.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.702+0000 7f7680aa1640 1 -- 172.21.15.57:0/1883463135 learned_addr learned my addr 172.21.15.57:0/1883463135 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:02:55.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.702+0000 7f76812a2640 1 --2- 172.21.15.57:0/1883463135 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f767c0b0ea0 0x7f767c0b1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:55.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.702+0000 7f7673fff640 1 --2- 172.21.15.57:0/1883463135 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f767c0b2030 0x7f767c158620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:55.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.702+0000 7f7680aa1640 1 -- 172.21.15.57:0/1883463135 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f767c0b0ea0 msgr2=0x7f767c0b1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:55.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.702+0000 7f7680aa1640 1 --2- 172.21.15.57:0/1883463135 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f767c0b0ea0 0x7f767c0b1320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:55.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.702+0000 7f7680aa1640 1 -- 172.21.15.57:0/1883463135 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f767c0b2030 msgr2=0x7f767c158620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:55.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.702+0000 7f7680aa1640 1 --2- 172.21.15.57:0/1883463135 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f767c0b2030 0x7f767c158620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:55.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.702+0000 7f7680aa1640 1 -- 172.21.15.57:0/1883463135 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f767c158b60 con 0x7f76640039a0 2023-10-19T13:02:55.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.702+0000 7f7673fff640 1 --2- 172.21.15.57:0/1883463135 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f767c0b2030 0x7f767c158620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:02:55.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.702+0000 7f76812a2640 1 --2- 172.21.15.57:0/1883463135 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f767c0b0ea0 0x7f767c0b1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:02:55.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.702+0000 7f7680aa1640 1 --2- 172.21.15.57:0/1883463135 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f76640039a0 0x7f767c0b5cc0 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7f7678009c70 tx=0x7f7678033c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:02:55.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.702+0000 7f76717fa640 1 -- 172.21.15.57:0/1883463135 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7678004460 con 0x7f76640039a0 2023-10-19T13:02:55.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.702+0000 7f76717fa640 1 -- 172.21.15.57:0/1883463135 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f7678033e60 con 0x7f76640039a0 2023-10-19T13:02:55.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.702+0000 7f7682d05640 1 -- 172.21.15.57:0/1883463135 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f767c0b6450 con 0x7f76640039a0 2023-10-19T13:02:55.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.702+0000 7f76717fa640 1 -- 172.21.15.57:0/1883463135 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7678003830 con 0x7f76640039a0 2023-10-19T13:02:55.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.703+0000 7f7682d05640 1 -- 172.21.15.57:0/1883463135 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f767c0c7fd0 con 0x7f76640039a0 2023-10-19T13:02:55.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.704+0000 7f7682d05640 1 -- 172.21.15.57:0/1883463135 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f767c0c7e10 con 0x7f76640039a0 2023-10-19T13:02:55.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.704+0000 7f76717fa640 1 -- 172.21.15.57:0/1883463135 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 58) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f7678003990 con 0x7f76640039a0 2023-10-19T13:02:55.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.704+0000 7f76717fa640 1 --2- 172.21.15.57:0/1883463135 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f766806af60 0x7f766806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:55.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.704+0000 7f76717fa640 1 -- 172.21.15.57:0/1883463135 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f76780ac510 con 0x7f76640039a0 2023-10-19T13:02:55.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.705+0000 7f7673fff640 1 --2- 172.21.15.57:0/1883463135 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f766806af60 0x7f766806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:55.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.705+0000 7f7673fff640 1 --2- 172.21.15.57:0/1883463135 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f766806af60 0x7f766806d420 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f766c0017a0 tx=0x7f766c001e10 comp rx=0 tx=0).ready entity=mgr.7540 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:02:55.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.709+0000 7f76717fa640 1 -- 172.21.15.57:0/1883463135 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f76780776b0 con 0x7f76640039a0 2023-10-19T13:02:55.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:55.884+0000 7f7682d05640 1 -- 172.21.15.57:0/1883463135 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f767c0b6970 con 0x7f76640039a0 2023-10-19T13:02:56.321 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:56.319+0000 7f76717fa640 1 -- 172.21.15.57:0/1883463135 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v127) v1 ==== 138+0+0 (secure 0 0 0) 0x7f767807acd0 con 0x7f76640039a0 2023-10-19T13:02:56.322 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'cephfs_metadata' created 2023-10-19T13:02:56.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:56.321+0000 7f7682d05640 1 -- 172.21.15.57:0/1883463135 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f766806af60 msgr2=0x7f766806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:56.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:56.321+0000 7f7682d05640 1 --2- 172.21.15.57:0/1883463135 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f766806af60 0x7f766806d420 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f766c0017a0 tx=0x7f766c001e10 comp rx=0 tx=0).stop 2023-10-19T13:02:56.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:56.321+0000 7f7682d05640 1 -- 172.21.15.57:0/1883463135 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f76640039a0 msgr2=0x7f767c0b5cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:56.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:56.321+0000 7f7682d05640 1 --2- 172.21.15.57:0/1883463135 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f76640039a0 0x7f767c0b5cc0 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7f7678009c70 tx=0x7f7678033c30 comp rx=0 tx=0).stop 2023-10-19T13:02:56.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:56.321+0000 7f7682d05640 1 -- 172.21.15.57:0/1883463135 shutdown_connections 2023-10-19T13:02:56.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:56.321+0000 7f7682d05640 1 --2- 172.21.15.57:0/1883463135 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f766806af60 0x7f766806d420 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:56.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:56.321+0000 7f7682d05640 1 --2- 172.21.15.57:0/1883463135 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f767c0b0ea0 0x7f767c0b1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:56.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:56.321+0000 7f7682d05640 1 --2- 172.21.15.57:0/1883463135 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f767c0b2030 0x7f767c158620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:56.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:56.321+0000 7f7682d05640 1 --2- 172.21.15.57:0/1883463135 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f76640039a0 0x7f767c0b5cc0 unknown :-1 s=CLOSED pgs=1426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:56.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:56.321+0000 7f7682d05640 1 -- 172.21.15.57:0/1883463135 >> 172.21.15.57:0/1883463135 conn(0x7f767c0d0d60 msgr2=0x7f767c0ae050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:02:56.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:56.321+0000 7f7682d05640 1 -- 172.21.15.57:0/1883463135 shutdown_connections 2023-10-19T13:02:56.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:56.321+0000 7f7682d05640 1 -- 172.21.15.57:0/1883463135 wait complete. 2023-10-19T13:02:56.339 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-19T13:02:56.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:56.455+0000 7f360b1a4640 1 -- 172.21.15.57:0/635121549 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f35ec004d00 msgr2=0x7f35ec005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:56.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:56.455+0000 7f360b1a4640 1 --2- 172.21.15.57:0/635121549 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f35ec004d00 0x7f35ec005180 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f35f4009b50 tx=0x7f35ec0057f0 comp rx=0 tx=0).stop 2023-10-19T13:02:56.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:56.455+0000 7f3609741640 1 -- 172.21.15.57:0/635121549 reap_dead start 2023-10-19T13:02:56.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:56.455+0000 7f360b1a4640 1 -- 172.21.15.57:0/635121549 shutdown_connections 2023-10-19T13:02:56.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:56.455+0000 7f360b1a4640 1 -- 172.21.15.57:0/635121549 >> 172.21.15.57:0/635121549 conn(0x7f36040af7a0 msgr2=0x7f36040afb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:02:56.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:56.455+0000 7f360b1a4640 1 -- 172.21.15.57:0/635121549 shutdown_connections 2023-10-19T13:02:56.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:56.455+0000 7f360b1a4640 1 -- 172.21.15.57:0/635121549 wait complete. 2023-10-19T13:02:56.461 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:56.456+0000 7f360b1a4640 1 Processor -- start 2023-10-19T13:02:56.461 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:56.456+0000 7f360b1a4640 1 -- start start 2023-10-19T13:02:56.462 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:56.457+0000 7f360b1a4640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3604142390 0x7f3604153c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:56.462 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:56.457+0000 7f360b1a4640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3604148870 0x7f36041553d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:56.462 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:56.457+0000 7f360b1a4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3604155910 0x7f36041542e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:56.462 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:56.457+0000 7f360b1a4640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3604154140 con 0x7f3604155910 2023-10-19T13:02:56.462 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:56.457+0000 7f360b1a4640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3604154f40 con 0x7f3604148870 2023-10-19T13:02:56.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:56.457+0000 7f360b1a4640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36040b1af0 con 0x7f3604142390 2023-10-19T13:02:56.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:56.457+0000 7f3609741640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3604155910 0x7f36041542e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:56.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:56.457+0000 7f3609741640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3604155910 0x7f36041542e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43556/0 (socket says 172.21.15.57:43556) 2023-10-19T13:02:56.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:56.457+0000 7f3609741640 1 -- 172.21.15.57:0/4100723676 learned_addr learned my addr 172.21.15.57:0/4100723676 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:02:56.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:56.457+0000 7f3603fff640 1 --2- 172.21.15.57:0/4100723676 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3604148870 0x7f36041553d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:56.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:56.457+0000 7f3608f40640 1 --2- 172.21.15.57:0/4100723676 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3604142390 0x7f3604153c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:56.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:56.457+0000 7f3609741640 1 -- 172.21.15.57:0/4100723676 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3604142390 msgr2=0x7f3604153c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:56.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:56.457+0000 7f3609741640 1 --2- 172.21.15.57:0/4100723676 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3604142390 0x7f3604153c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:56.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:56.457+0000 7f3609741640 1 -- 172.21.15.57:0/4100723676 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3604148870 msgr2=0x7f36041553d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:56.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:56.457+0000 7f3609741640 1 --2- 172.21.15.57:0/4100723676 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3604148870 0x7f36041553d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:56.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:56.457+0000 7f3609741640 1 -- 172.21.15.57:0/4100723676 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36040b1dd0 con 0x7f3604155910 2023-10-19T13:02:56.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:56.458+0000 7f3609741640 1 --2- 172.21.15.57:0/4100723676 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3604155910 0x7f36041542e0 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f35f4009b50 tx=0x7f35f4034c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:02:56.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:56.458+0000 7f36017fa640 1 -- 172.21.15.57:0/4100723676 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35f4004230 con 0x7f3604155910 2023-10-19T13:02:56.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:56.458+0000 7f36017fa640 1 -- 172.21.15.57:0/4100723676 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f35f4004390 con 0x7f3604155910 2023-10-19T13:02:56.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:56.458+0000 7f360b1a4640 1 -- 172.21.15.57:0/4100723676 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36040b20b0 con 0x7f3604155910 2023-10-19T13:02:56.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:56.458+0000 7f36017fa640 1 -- 172.21.15.57:0/4100723676 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35f4003850 con 0x7f3604155910 2023-10-19T13:02:56.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:56.458+0000 7f360b1a4640 1 -- 172.21.15.57:0/4100723676 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3604142770 con 0x7f3604155910 2023-10-19T13:02:56.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:56.459+0000 7f360b1a4640 1 -- 172.21.15.57:0/4100723676 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36041550b0 con 0x7f3604155910 2023-10-19T13:02:56.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:56.459+0000 7f36017fa640 1 -- 172.21.15.57:0/4100723676 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 58) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f35f40039b0 con 0x7f3604155910 2023-10-19T13:02:56.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:56.460+0000 7f36017fa640 1 --2- 172.21.15.57:0/4100723676 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f35f006af90 0x7f35f006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:56.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:56.460+0000 7f36017fa640 1 -- 172.21.15.57:0/4100723676 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5575+0+0 (secure 0 0 0) 0x7f35f4000e40 con 0x7f3604155910 2023-10-19T13:02:56.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:56.460+0000 7f3608f40640 1 --2- 172.21.15.57:0/4100723676 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f35f006af90 0x7f35f006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:56.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:56.460+0000 7f3608f40640 1 --2- 172.21.15.57:0/4100723676 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f35f006af90 0x7f35f006d450 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f35fc006180 tx=0x7f35fc006eb0 comp rx=0 tx=0).ready entity=mgr.7540 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:02:56.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:56.464+0000 7f36017fa640 1 -- 172.21.15.57:0/4100723676 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f35f4077540 con 0x7f3604155910 2023-10-19T13:02:56.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:56.643+0000 7f360b1a4640 1 -- 172.21.15.57:0/4100723676 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f36040690e0 con 0x7f3604155910 2023-10-19T13:02:57.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:57.326+0000 7f36017fa640 1 -- 172.21.15.57:0/4100723676 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v128) v1 ==== 170+0+0 (secure 0 0 0) 0x7f35f4034e80 con 0x7f3604155910 2023-10-19T13:02:57.329 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'cephfs_data' created 2023-10-19T13:02:57.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:57.334+0000 7f360b1a4640 1 -- 172.21.15.57:0/4100723676 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f35f006af90 msgr2=0x7f35f006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:57.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:57.334+0000 7f360b1a4640 1 --2- 172.21.15.57:0/4100723676 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f35f006af90 0x7f35f006d450 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f35fc006180 tx=0x7f35fc006eb0 comp rx=0 tx=0).stop 2023-10-19T13:02:57.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:57.334+0000 7f360b1a4640 1 -- 172.21.15.57:0/4100723676 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3604155910 msgr2=0x7f36041542e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:57.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:57.334+0000 7f360b1a4640 1 --2- 172.21.15.57:0/4100723676 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3604155910 0x7f36041542e0 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f35f4009b50 tx=0x7f35f4034c00 comp rx=0 tx=0).stop 2023-10-19T13:02:57.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:57.335+0000 7f360b1a4640 1 -- 172.21.15.57:0/4100723676 shutdown_connections 2023-10-19T13:02:57.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:57.335+0000 7f360b1a4640 1 --2- 172.21.15.57:0/4100723676 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f35f006af90 0x7f35f006d450 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:57.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:57.335+0000 7f360b1a4640 1 --2- 172.21.15.57:0/4100723676 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3604155910 0x7f36041542e0 unknown :-1 s=CLOSED pgs=1429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:57.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:57.335+0000 7f360b1a4640 1 --2- 172.21.15.57:0/4100723676 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3604148870 0x7f36041553d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:57.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:57.335+0000 7f360b1a4640 1 --2- 172.21.15.57:0/4100723676 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3604142390 0x7f3604153c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:57.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:57.336+0000 7f360b1a4640 1 -- 172.21.15.57:0/4100723676 >> 172.21.15.57:0/4100723676 conn(0x7f36040af7a0 msgr2=0x7f36040cbe10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:02:57.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:57.337+0000 7f360b1a4640 1 -- 172.21.15.57:0/4100723676 shutdown_connections 2023-10-19T13:02:57.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:57.338+0000 7f360b1a4640 1 -- 172.21.15.57:0/4100723676 wait complete. 2023-10-19T13:02:57.361 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-19T13:02:57.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:57.475+0000 7ff7c8c21640 1 -- 172.21.15.57:0/2922043021 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff7c40cc990 msgr2=0x7ff7c40ac870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:57.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:57.475+0000 7ff7c8c21640 1 --2- 172.21.15.57:0/2922043021 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff7c40cc990 0x7ff7c40ac870 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7ff7c4066e70 tx=0x7ff7ac02bd70 comp rx=0 tx=0).stop 2023-10-19T13:02:57.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:57.475+0000 7ff7c8c21640 1 -- 172.21.15.57:0/2922043021 shutdown_connections 2023-10-19T13:02:57.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:57.475+0000 7ff7c8c21640 1 --2- 172.21.15.57:0/2922043021 >> v2:172.21.15.179:3300/0 conn(0x7ff7c406b980 0x7ff7c41596e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:57.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:57.475+0000 7ff7c8c21640 1 --2- 172.21.15.57:0/2922043021 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff7c40cc990 0x7ff7c40ac870 unknown :-1 s=CLOSED pgs=1431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:57.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:57.475+0000 7ff7c8c21640 1 -- 172.21.15.57:0/2922043021 >> 172.21.15.57:0/2922043021 conn(0x7ff7c406fae0 msgr2=0x7ff7c406fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:02:57.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:57.475+0000 7ff7c8c21640 1 -- 172.21.15.57:0/2922043021 shutdown_connections 2023-10-19T13:02:57.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:57.475+0000 7ff7c8c21640 1 -- 172.21.15.57:0/2922043021 wait complete. 2023-10-19T13:02:57.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:57.476+0000 7ff7c8c21640 1 Processor -- start 2023-10-19T13:02:57.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:57.476+0000 7ff7c8c21640 1 -- start start 2023-10-19T13:02:57.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:57.477+0000 7ff7c8c21640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff7c406a730 0x7ff7c414cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:57.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:57.477+0000 7ff7c8c21640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff7c406b980 0x7ff7c41e36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:57.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:57.477+0000 7ff7c8c21640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff7c40cc990 0x7ff7c41e3c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:57.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:57.477+0000 7ff7c8c21640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7c41e2bd0 con 0x7ff7c40cc990 2023-10-19T13:02:57.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:57.477+0000 7ff7c8c21640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7c41e1fc0 con 0x7ff7c406b980 2023-10-19T13:02:57.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:57.477+0000 7ff7c8c21640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7c41e22a0 con 0x7ff7c406a730 2023-10-19T13:02:57.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:57.477+0000 7ff7c2d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff7c40cc990 0x7ff7c41e3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:57.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:57.477+0000 7ff7c2d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff7c40cc990 0x7ff7c41e3c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:48504/0 (socket says 172.21.15.57:48504) 2023-10-19T13:02:57.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:57.477+0000 7ff7c2d9d640 1 -- 172.21.15.57:0/1705961059 learned_addr learned my addr 172.21.15.57:0/1705961059 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:02:57.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:57.478+0000 7ff7c1d9b640 1 --2- 172.21.15.57:0/1705961059 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff7c406b980 0x7ff7c41e36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:57.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:57.478+0000 7ff7c2d9d640 1 -- 172.21.15.57:0/1705961059 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff7c406a730 msgr2=0x7ff7c414cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:57.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:57.478+0000 7ff7c2d9d640 1 --2- 172.21.15.57:0/1705961059 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff7c406a730 0x7ff7c414cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:57.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:57.478+0000 7ff7c2d9d640 1 -- 172.21.15.57:0/1705961059 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff7c406b980 msgr2=0x7ff7c41e36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:57.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:57.478+0000 7ff7c2d9d640 1 --2- 172.21.15.57:0/1705961059 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff7c406b980 0x7ff7c41e36d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:57.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:57.478+0000 7ff7c2d9d640 1 -- 172.21.15.57:0/1705961059 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7ac009490 con 0x7ff7c40cc990 2023-10-19T13:02:57.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:57.478+0000 7ff7c2d9d640 1 --2- 172.21.15.57:0/1705961059 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff7c40cc990 0x7ff7c41e3c10 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7ff7b400eaa0 tx=0x7ff7b400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:02:57.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:57.478+0000 7ff7baffd640 1 -- 172.21.15.57:0/1705961059 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7b400cdb0 con 0x7ff7c40cc990 2023-10-19T13:02:57.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:57.478+0000 7ff7baffd640 1 -- 172.21.15.57:0/1705961059 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7ff7b4004b20 con 0x7ff7c40cc990 2023-10-19T13:02:57.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:57.478+0000 7ff7baffd640 1 -- 172.21.15.57:0/1705961059 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7b4005020 con 0x7ff7c40cc990 2023-10-19T13:02:57.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:57.478+0000 7ff7c8c21640 1 -- 172.21.15.57:0/1705961059 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7c41e2580 con 0x7ff7c40cc990 2023-10-19T13:02:57.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:57.478+0000 7ff7c8c21640 1 -- 172.21.15.57:0/1705961059 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7c414d510 con 0x7ff7c40cc990 2023-10-19T13:02:57.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:57.479+0000 7ff7c8c21640 1 -- 172.21.15.57:0/1705961059 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7c41e2d40 con 0x7ff7c40cc990 2023-10-19T13:02:57.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:57.480+0000 7ff7baffd640 1 -- 172.21.15.57:0/1705961059 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 58) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff7b4005180 con 0x7ff7c40cc990 2023-10-19T13:02:57.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:57.480+0000 7ff7baffd640 1 --2- 172.21.15.57:0/1705961059 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7ff7a406af60 0x7ff7a406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:57.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:57.480+0000 7ff7baffd640 1 -- 172.21.15.57:0/1705961059 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5957+0+0 (secure 0 0 0) 0x7ff7b408a860 con 0x7ff7c40cc990 2023-10-19T13:02:57.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:57.480+0000 7ff7c259c640 1 --2- 172.21.15.57:0/1705961059 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7ff7a406af60 0x7ff7a406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:57.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:57.481+0000 7ff7c259c640 1 --2- 172.21.15.57:0/1705961059 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7ff7a406af60 0x7ff7a406d420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff7bc001600 tx=0x7ff7bc007910 comp rx=0 tx=0).ready entity=mgr.7540 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:02:57.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:57.484+0000 7ff7baffd640 1 -- 172.21.15.57:0/1705961059 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff7b40555d0 con 0x7ff7c40cc990 2023-10-19T13:02:57.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:57.696+0000 7ff7c8c21640 1 -- 172.21.15.57:0/1705961059 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7ff7c40690e0 con 0x7ff7c40cc990 2023-10-19T13:02:58.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.334+0000 7ff7baffd640 1 -- 172.21.15.57:0/1705961059 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 15 and data pool 16 v48) v1 ==== 174+0+0 (secure 0 0 0) 0x7ff7b401f0c0 con 0x7ff7c40cc990 2023-10-19T13:02:58.337 INFO:teuthology.orchestra.run.smithi057.stderr:new fs with metadata pool 15 and data pool 16 2023-10-19T13:02:58.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.337+0000 7ff7c8c21640 1 -- 172.21.15.57:0/1705961059 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7ff7a406af60 msgr2=0x7ff7a406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:58.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.337+0000 7ff7c8c21640 1 --2- 172.21.15.57:0/1705961059 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7ff7a406af60 0x7ff7a406d420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff7bc001600 tx=0x7ff7bc007910 comp rx=0 tx=0).stop 2023-10-19T13:02:58.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.337+0000 7ff7c8c21640 1 -- 172.21.15.57:0/1705961059 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff7c40cc990 msgr2=0x7ff7c41e3c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:58.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.337+0000 7ff7c8c21640 1 --2- 172.21.15.57:0/1705961059 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff7c40cc990 0x7ff7c41e3c10 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7ff7b400eaa0 tx=0x7ff7b400ef60 comp rx=0 tx=0).stop 2023-10-19T13:02:58.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.337+0000 7ff7c8c21640 1 -- 172.21.15.57:0/1705961059 shutdown_connections 2023-10-19T13:02:58.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.337+0000 7ff7c8c21640 1 --2- 172.21.15.57:0/1705961059 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7ff7a406af60 0x7ff7a406d420 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:58.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.337+0000 7ff7c8c21640 1 --2- 172.21.15.57:0/1705961059 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff7c40cc990 0x7ff7c41e3c10 unknown :-1 s=CLOSED pgs=1432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:58.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.337+0000 7ff7c8c21640 1 --2- 172.21.15.57:0/1705961059 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff7c406b980 0x7ff7c41e36d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:58.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.337+0000 7ff7c8c21640 1 --2- 172.21.15.57:0/1705961059 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff7c406a730 0x7ff7c414cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:58.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.337+0000 7ff7c8c21640 1 -- 172.21.15.57:0/1705961059 >> 172.21.15.57:0/1705961059 conn(0x7ff7c406fae0 msgr2=0x7ff7c40ae500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:02:58.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.337+0000 7ff7c8c21640 1 -- 172.21.15.57:0/1705961059 shutdown_connections 2023-10-19T13:02:58.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.337+0000 7ff7c8c21640 1 -- 172.21.15.57:0/1705961059 wait complete. 2023-10-19T13:02:58.356 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T13:02:58.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.467+0000 7f63b9a10640 1 -- 172.21.15.57:0/3019957137 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f63b40cd9e0 msgr2=0x7f63b40cde70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:58.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.467+0000 7f63b9a10640 1 --2- 172.21.15.57:0/3019957137 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f63b40cd9e0 0x7f63b40cde70 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f63ac005fd0 tx=0x7f63ac02c2e0 comp rx=0 tx=0).stop 2023-10-19T13:02:58.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.467+0000 7f63b9a10640 1 -- 172.21.15.57:0/3019957137 shutdown_connections 2023-10-19T13:02:58.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.467+0000 7f63b9a10640 1 --2- 172.21.15.57:0/3019957137 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f63b40cd9e0 0x7f63b40cde70 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:58.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.467+0000 7f63b9a10640 1 --2- 172.21.15.57:0/3019957137 >> v2:172.21.15.179:3300/0 conn(0x7f63b4158cf0 0x7f63b415c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:58.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.467+0000 7f63b9a10640 1 -- 172.21.15.57:0/3019957137 >> 172.21.15.57:0/3019957137 conn(0x7f63b40af7a0 msgr2=0x7f63b40afb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:02:58.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.467+0000 7f63b9a10640 1 -- 172.21.15.57:0/3019957137 shutdown_connections 2023-10-19T13:02:58.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.467+0000 7f63b9a10640 1 -- 172.21.15.57:0/3019957137 wait complete. 2023-10-19T13:02:58.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.468+0000 7f63b9a10640 1 Processor -- start 2023-10-19T13:02:58.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.468+0000 7f63b9a10640 1 -- start start 2023-10-19T13:02:58.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.469+0000 7f63b9a10640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f63b40cd9e0 0x7f63b4153440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:58.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.469+0000 7f63b9a10640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f63b4158910 0x7f63b40b0890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:58.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.469+0000 7f63b9a10640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f63b4158cf0 0x7f63b41e2590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:58.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.469+0000 7f63b9a10640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63b4146060 con 0x7f63b4158cf0 2023-10-19T13:02:58.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.469+0000 7f63b9a10640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63b4151710 con 0x7f63b4158910 2023-10-19T13:02:58.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.469+0000 7f63b9a10640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63b4150570 con 0x7f63b40cd9e0 2023-10-19T13:02:58.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.469+0000 7f63b37fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f63b4158cf0 0x7f63b41e2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:58.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.469+0000 7f63b37fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f63b4158cf0 0x7f63b41e2590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:48508/0 (socket says 172.21.15.57:48508) 2023-10-19T13:02:58.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.469+0000 7f63b37fe640 1 -- 172.21.15.57:0/3554045656 learned_addr learned my addr 172.21.15.57:0/3554045656 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:02:58.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.469+0000 7f63b2ffd640 1 --2- 172.21.15.57:0/3554045656 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f63b40cd9e0 0x7f63b4153440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:58.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.469+0000 7f63aa7fc640 1 --2- 172.21.15.57:0/3554045656 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f63b4158910 0x7f63b40b0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:58.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.469+0000 7f63b37fe640 1 -- 172.21.15.57:0/3554045656 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f63b40cd9e0 msgr2=0x7f63b4153440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:58.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.469+0000 7f63b37fe640 1 --2- 172.21.15.57:0/3554045656 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f63b40cd9e0 0x7f63b4153440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:58.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.469+0000 7f63b37fe640 1 -- 172.21.15.57:0/3554045656 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f63b4158910 msgr2=0x7f63b40b0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:58.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.469+0000 7f63b37fe640 1 --2- 172.21.15.57:0/3554045656 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f63b4158910 0x7f63b40b0890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:58.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.469+0000 7f63b37fe640 1 -- 172.21.15.57:0/3554045656 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63ac009900 con 0x7f63b4158cf0 2023-10-19T13:02:58.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.469+0000 7f63aa7fc640 1 --2- 172.21.15.57:0/3554045656 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f63b4158910 0x7f63b40b0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:02:58.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.469+0000 7f63b2ffd640 1 --2- 172.21.15.57:0/3554045656 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f63b40cd9e0 0x7f63b4153440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:02:58.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.470+0000 7f63b37fe640 1 --2- 172.21.15.57:0/3554045656 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f63b4158cf0 0x7f63b41e2590 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f63ac009d90 tx=0x7f63ac009870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:02:58.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.470+0000 7f63b0ff9640 1 -- 172.21.15.57:0/3554045656 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63ac004040 con 0x7f63b4158cf0 2023-10-19T13:02:58.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.470+0000 7f63b0ff9640 1 -- 172.21.15.57:0/3554045656 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f63ac004290 con 0x7f63b4158cf0 2023-10-19T13:02:58.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.470+0000 7f63b9a10640 1 -- 172.21.15.57:0/3554045656 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63b4150850 con 0x7f63b4158cf0 2023-10-19T13:02:58.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.470+0000 7f63b0ff9640 1 -- 172.21.15.57:0/3554045656 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63ac004680 con 0x7f63b4158cf0 2023-10-19T13:02:58.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.470+0000 7f63b9a10640 1 -- 172.21.15.57:0/3554045656 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63b4149810 con 0x7f63b4158cf0 2023-10-19T13:02:58.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.471+0000 7f63b9a10640 1 -- 172.21.15.57:0/3554045656 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63b40690e0 con 0x7f63b4158cf0 2023-10-19T13:02:58.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.471+0000 7f63b0ff9640 1 -- 172.21.15.57:0/3554045656 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 58) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f63ac044070 con 0x7f63b4158cf0 2023-10-19T13:02:58.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.472+0000 7f63b0ff9640 1 --2- 172.21.15.57:0/3554045656 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f639406af60 0x7f639406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:58.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.472+0000 7f63b0ff9640 1 -- 172.21.15.57:0/3554045656 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f63ac0ae670 con 0x7f63b4158cf0 2023-10-19T13:02:58.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.472+0000 7f63b2ffd640 1 --2- 172.21.15.57:0/3554045656 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f639406af60 0x7f639406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:58.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.472+0000 7f63b2ffd640 1 --2- 172.21.15.57:0/3554045656 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f639406af60 0x7f639406d420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f63a4001700 tx=0x7f63a4009040 comp rx=0 tx=0).ready entity=mgr.7540 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:02:58.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.476+0000 7f63b0ff9640 1 -- 172.21.15.57:0/3554045656 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f63ac0794b0 con 0x7f63b4158cf0 2023-10-19T13:02:58.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.653+0000 7f63b9a10640 1 -- 172.21.15.57:0/3554045656 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f63b40c9cb0 con 0x7f63b4158cf0 2023-10-19T13:02:58.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.654+0000 7f63b0ff9640 1 -- 172.21.15.57:0/3554045656 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v129) v1 ==== 74+0+16784 (secure 0 0 0) 0x7f63ac07cad0 con 0x7f63b4158cf0 2023-10-19T13:02:58.657 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:02:58.657 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":129,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T13:02:57.697514+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2023-10-19T13:02:55.916158+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"129","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2023-10-19T13:02:56.675559+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"129","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T13:02:58.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.656+0000 7f63b9a10640 1 -- 172.21.15.57:0/3554045656 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f639406af60 msgr2=0x7f639406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:58.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.656+0000 7f63b9a10640 1 --2- 172.21.15.57:0/3554045656 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f639406af60 0x7f639406d420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f63a4001700 tx=0x7f63a4009040 comp rx=0 tx=0).stop 2023-10-19T13:02:58.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.656+0000 7f63b9a10640 1 -- 172.21.15.57:0/3554045656 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f63b4158cf0 msgr2=0x7f63b41e2590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:58.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.656+0000 7f63b9a10640 1 --2- 172.21.15.57:0/3554045656 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f63b4158cf0 0x7f63b41e2590 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f63ac009d90 tx=0x7f63ac009870 comp rx=0 tx=0).stop 2023-10-19T13:02:58.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.656+0000 7f63b9a10640 1 -- 172.21.15.57:0/3554045656 shutdown_connections 2023-10-19T13:02:58.660 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.656+0000 7f63b9a10640 1 --2- 172.21.15.57:0/3554045656 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f639406af60 0x7f639406d420 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:58.660 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.656+0000 7f63b9a10640 1 --2- 172.21.15.57:0/3554045656 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f63b4158cf0 0x7f63b41e2590 unknown :-1 s=CLOSED pgs=1433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:58.660 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.656+0000 7f63b9a10640 1 --2- 172.21.15.57:0/3554045656 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f63b4158910 0x7f63b40b0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:58.660 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.656+0000 7f63b9a10640 1 --2- 172.21.15.57:0/3554045656 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f63b40cd9e0 0x7f63b4153440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:58.660 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.656+0000 7f63b9a10640 1 -- 172.21.15.57:0/3554045656 >> 172.21.15.57:0/3554045656 conn(0x7f63b40af7a0 msgr2=0x7f63b4155550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:02:58.661 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.656+0000 7f63b9a10640 1 -- 172.21.15.57:0/3554045656 shutdown_connections 2023-10-19T13:02:58.661 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.656+0000 7f63b9a10640 1 -- 172.21.15.57:0/3554045656 wait complete. 2023-10-19T13:02:58.669 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T13:02:58.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.775+0000 7f545756c640 1 -- 172.21.15.57:0/603434146 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f545006ae40 msgr2=0x7f5450157dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:58.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.775+0000 7f545756c640 1 --2- 172.21.15.57:0/603434146 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f545006ae40 0x7f5450157dd0 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7f54500662b0 tx=0x7f544802c110 comp rx=0 tx=0).stop 2023-10-19T13:02:58.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.776+0000 7f545756c640 1 -- 172.21.15.57:0/603434146 shutdown_connections 2023-10-19T13:02:58.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.776+0000 7f545756c640 1 --2- 172.21.15.57:0/603434146 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f545006ae40 0x7f5450157dd0 unknown :-1 s=CLOSED pgs=1435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:58.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.776+0000 7f545756c640 1 --2- 172.21.15.57:0/603434146 >> v2:172.21.15.179:3300/0 conn(0x7f54501579f0 0x7f54500b0e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:58.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.776+0000 7f545756c640 1 -- 172.21.15.57:0/603434146 >> 172.21.15.57:0/603434146 conn(0x7f545006fae0 msgr2=0x7f545006fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:02:58.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.776+0000 7f545756c640 1 -- 172.21.15.57:0/603434146 shutdown_connections 2023-10-19T13:02:58.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.776+0000 7f545756c640 1 -- 172.21.15.57:0/603434146 wait complete. 2023-10-19T13:02:58.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.777+0000 7f545756c640 1 Processor -- start 2023-10-19T13:02:58.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.777+0000 7f545756c640 1 -- start start 2023-10-19T13:02:58.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.777+0000 7f545756c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f545006ae40 0x7f5450153eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:58.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.777+0000 7f545756c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f54501579f0 0x7f54501f1800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:58.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.777+0000 7f545756c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f545015c1f0 0x7f5450156c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:58.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.777+0000 7f545756c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54500ade40 con 0x7f54501579f0 2023-10-19T13:02:58.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.777+0000 7f545756c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54501f1d40 con 0x7f545015c1f0 2023-10-19T13:02:58.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.777+0000 7f545756c640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54501544e0 con 0x7f545006ae40 2023-10-19T13:02:58.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.778+0000 7f5454b07640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f54501579f0 0x7f54501f1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:58.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.778+0000 7f5454b07640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f54501579f0 0x7f54501f1800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:48526/0 (socket says 172.21.15.57:48526) 2023-10-19T13:02:58.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.778+0000 7f5454b07640 1 -- 172.21.15.57:0/3866767813 learned_addr learned my addr 172.21.15.57:0/3866767813 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:02:58.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.778+0000 7f5455b09640 1 --2- 172.21.15.57:0/3866767813 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f545015c1f0 0x7f5450156c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:58.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.778+0000 7f5455308640 1 --2- 172.21.15.57:0/3866767813 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f545006ae40 0x7f5450153eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:58.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.778+0000 7f5454b07640 1 -- 172.21.15.57:0/3866767813 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f545006ae40 msgr2=0x7f5450153eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:58.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.778+0000 7f5454b07640 1 --2- 172.21.15.57:0/3866767813 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f545006ae40 0x7f5450153eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:58.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.778+0000 7f5454b07640 1 -- 172.21.15.57:0/3866767813 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f545015c1f0 msgr2=0x7f5450156c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:58.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.778+0000 7f5454b07640 1 --2- 172.21.15.57:0/3866767813 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f545015c1f0 0x7f5450156c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:58.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.778+0000 7f5454b07640 1 -- 172.21.15.57:0/3866767813 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5448009840 con 0x7f54501579f0 2023-10-19T13:02:58.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.778+0000 7f5454b07640 1 --2- 172.21.15.57:0/3866767813 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f54501579f0 0x7f54501f1800 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f544000c410 tx=0x7f544000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:02:58.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.778+0000 7f5445ffb640 1 -- 172.21.15.57:0/3866767813 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54400035f0 con 0x7f54501579f0 2023-10-19T13:02:58.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.778+0000 7f5445ffb640 1 -- 172.21.15.57:0/3866767813 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f5440003770 con 0x7f54501579f0 2023-10-19T13:02:58.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.778+0000 7f545756c640 1 -- 172.21.15.57:0/3866767813 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54501f4520 con 0x7f54501579f0 2023-10-19T13:02:58.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.778+0000 7f5445ffb640 1 -- 172.21.15.57:0/3866767813 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5440003b80 con 0x7f54501579f0 2023-10-19T13:02:58.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.779+0000 7f545756c640 1 -- 172.21.15.57:0/3866767813 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54500adb60 con 0x7f54501579f0 2023-10-19T13:02:58.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.779+0000 7f545756c640 1 -- 172.21.15.57:0/3866767813 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54501f4380 con 0x7f54501579f0 2023-10-19T13:02:58.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.780+0000 7f5445ffb640 1 -- 172.21.15.57:0/3866767813 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 58) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f5440003ce0 con 0x7f54501579f0 2023-10-19T13:02:58.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.780+0000 7f5445ffb640 1 --2- 172.21.15.57:0/3866767813 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f543406af60 0x7f543406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:58.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.781+0000 7f5445ffb640 1 -- 172.21.15.57:0/3866767813 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f5440089120 con 0x7f54501579f0 2023-10-19T13:02:58.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.781+0000 7f5455308640 1 --2- 172.21.15.57:0/3866767813 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f543406af60 0x7f543406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:58.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.781+0000 7f5455308640 1 --2- 172.21.15.57:0/3866767813 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f543406af60 0x7f543406d420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f54500cac00 tx=0x7f544c008040 comp rx=0 tx=0).ready entity=mgr.7540 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:02:58.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.785+0000 7f5445ffb640 1 -- 172.21.15.57:0/3866767813 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f54501f4380 con 0x7f54501579f0 2023-10-19T13:02:58.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.962+0000 7f545756c640 1 -- 172.21.15.57:0/3866767813 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f54501f4380 con 0x7f54501579f0 2023-10-19T13:02:58.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.963+0000 7f5445ffb640 1 -- 172.21.15.57:0/3866767813 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v129) v1 ==== 74+0+16784 (secure 0 0 0) 0x7f54501f4380 con 0x7f54501579f0 2023-10-19T13:02:58.966 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:02:58.967 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":129,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T13:02:57.697514+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2023-10-19T13:02:55.916158+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"129","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2023-10-19T13:02:56.675559+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"129","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T13:02:58.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.965+0000 7f545756c640 1 -- 172.21.15.57:0/3866767813 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f543406af60 msgr2=0x7f543406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:58.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.965+0000 7f545756c640 1 --2- 172.21.15.57:0/3866767813 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f543406af60 0x7f543406d420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f54500cac00 tx=0x7f544c008040 comp rx=0 tx=0).stop 2023-10-19T13:02:58.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.965+0000 7f545756c640 1 -- 172.21.15.57:0/3866767813 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f54501579f0 msgr2=0x7f54501f1800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:58.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.965+0000 7f545756c640 1 --2- 172.21.15.57:0/3866767813 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f54501579f0 0x7f54501f1800 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f544000c410 tx=0x7f544000c8d0 comp rx=0 tx=0).stop 2023-10-19T13:02:58.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.965+0000 7f545756c640 1 -- 172.21.15.57:0/3866767813 shutdown_connections 2023-10-19T13:02:58.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.965+0000 7f545756c640 1 --2- 172.21.15.57:0/3866767813 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f543406af60 0x7f543406d420 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:58.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.965+0000 7f545756c640 1 --2- 172.21.15.57:0/3866767813 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f545015c1f0 0x7f5450156c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:58.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.965+0000 7f545756c640 1 --2- 172.21.15.57:0/3866767813 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f54501579f0 0x7f54501f1800 unknown :-1 s=CLOSED pgs=1436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:58.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.965+0000 7f545756c640 1 --2- 172.21.15.57:0/3866767813 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f545006ae40 0x7f5450153eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:58.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.965+0000 7f545756c640 1 -- 172.21.15.57:0/3866767813 >> 172.21.15.57:0/3866767813 conn(0x7f545006fae0 msgr2=0x7f54500cb8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:02:58.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.965+0000 7f545756c640 1 -- 172.21.15.57:0/3866767813 shutdown_connections 2023-10-19T13:02:58.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:58.965+0000 7f545756c640 1 -- 172.21.15.57:0/3866767813 wait complete. 2023-10-19T13:02:58.979 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-19T13:02:59.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.087+0000 7f6608d8e640 1 -- 172.21.15.57:0/2623972214 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f660413e520 msgr2=0x7f660406bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:59.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.087+0000 7f6608d8e640 1 --2- 172.21.15.57:0/2623972214 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f660413e520 0x7f660406bb60 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7f65f4009830 tx=0x7f65f402bcd0 comp rx=0 tx=0).stop 2023-10-19T13:02:59.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.087+0000 7f6608d8e640 1 -- 172.21.15.57:0/2623972214 shutdown_connections 2023-10-19T13:02:59.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.087+0000 7f6608d8e640 1 --2- 172.21.15.57:0/2623972214 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f660413e520 0x7f660406bb60 unknown :-1 s=CLOSED pgs=1438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:59.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.087+0000 7f6608d8e640 1 -- 172.21.15.57:0/2623972214 >> 172.21.15.57:0/2623972214 conn(0x7f6604069930 msgr2=0x7f6604069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:02:59.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.088+0000 7f6608d8e640 1 -- 172.21.15.57:0/2623972214 shutdown_connections 2023-10-19T13:02:59.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.088+0000 7f6608d8e640 1 -- 172.21.15.57:0/2623972214 wait complete. 2023-10-19T13:02:59.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.089+0000 7f6608d8e640 1 Processor -- start 2023-10-19T13:02:59.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.089+0000 7f6608d8e640 1 -- start start 2023-10-19T13:02:59.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.090+0000 7f6608d8e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6604070af0 0x7f6604158880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:59.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.090+0000 7f6608d8e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f660413e520 0x7f660413c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:59.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.090+0000 7f6608d8e640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6604141690 0x7f660413c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:59.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.090+0000 7f6608d8e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f660413ce60 con 0x7f6604070af0 2023-10-19T13:02:59.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.090+0000 7f6608d8e640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f660413d0d0 con 0x7f660413e520 2023-10-19T13:02:59.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.090+0000 7f6608d8e640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f660413d3b0 con 0x7f6604141690 2023-10-19T13:02:59.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.090+0000 7f660259c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6604070af0 0x7f6604158880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:59.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.090+0000 7f660259c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6604070af0 0x7f6604158880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:48550/0 (socket says 172.21.15.57:48550) 2023-10-19T13:02:59.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.090+0000 7f660259c640 1 -- 172.21.15.57:0/2234095910 learned_addr learned my addr 172.21.15.57:0/2234095910 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:02:59.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.090+0000 7f65f9d9b640 1 --2- 172.21.15.57:0/2234095910 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f660413e520 0x7f660413c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:59.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.090+0000 7f6602d9d640 1 --2- 172.21.15.57:0/2234095910 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6604141690 0x7f660413c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:59.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.090+0000 7f660259c640 1 -- 172.21.15.57:0/2234095910 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6604141690 msgr2=0x7f660413c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:59.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.090+0000 7f660259c640 1 --2- 172.21.15.57:0/2234095910 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6604141690 0x7f660413c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:59.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.090+0000 7f660259c640 1 -- 172.21.15.57:0/2234095910 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f660413e520 msgr2=0x7f660413c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:59.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.090+0000 7f660259c640 1 --2- 172.21.15.57:0/2234095910 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f660413e520 0x7f660413c3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:59.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.090+0000 7f660259c640 1 -- 172.21.15.57:0/2234095910 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65f4009490 con 0x7f6604070af0 2023-10-19T13:02:59.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.090+0000 7f6602d9d640 1 --2- 172.21.15.57:0/2234095910 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6604141690 0x7f660413c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T13:02:59.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.090+0000 7f660259c640 1 --2- 172.21.15.57:0/2234095910 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6604070af0 0x7f6604158880 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f65f4009780 tx=0x7f65f40038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:02:59.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.090+0000 7f65fb7fe640 1 -- 172.21.15.57:0/2234095910 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65f4003ea0 con 0x7f6604070af0 2023-10-19T13:02:59.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.090+0000 7f65fb7fe640 1 -- 172.21.15.57:0/2234095910 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f65f4034550 con 0x7f6604070af0 2023-10-19T13:02:59.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.090+0000 7f6608d8e640 1 -- 172.21.15.57:0/2234095910 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66041e1e50 con 0x7f6604070af0 2023-10-19T13:02:59.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.091+0000 7f65fb7fe640 1 -- 172.21.15.57:0/2234095910 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65f4034960 con 0x7f6604070af0 2023-10-19T13:02:59.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.091+0000 7f6608d8e640 1 -- 172.21.15.57:0/2234095910 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6604158dc0 con 0x7f6604070af0 2023-10-19T13:02:59.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.092+0000 7f6608d8e640 1 -- 172.21.15.57:0/2234095910 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66040690e0 con 0x7f6604070af0 2023-10-19T13:02:59.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.092+0000 7f65fb7fe640 1 -- 172.21.15.57:0/2234095910 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 58) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f65f403b040 con 0x7f6604070af0 2023-10-19T13:02:59.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.092+0000 7f65fb7fe640 1 --2- 172.21.15.57:0/2234095910 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f65e406af60 0x7f65e406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:59.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.093+0000 7f65fb7fe640 1 -- 172.21.15.57:0/2234095910 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f65f40abfe0 con 0x7f6604070af0 2023-10-19T13:02:59.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.093+0000 7f65f9d9b640 1 --2- 172.21.15.57:0/2234095910 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f65e406af60 0x7f65e406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:59.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.093+0000 7f65f9d9b640 1 --2- 172.21.15.57:0/2234095910 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f65e406af60 0x7f65e406d420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f65ec007fd0 tx=0x7f65ec009040 comp rx=0 tx=0).ready entity=mgr.7540 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:02:59.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.097+0000 7f65fb7fe640 1 -- 172.21.15.57:0/2234095910 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f65f4076da0 con 0x7f6604070af0 2023-10-19T13:02:59.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.308+0000 7f6608d8e640 1 -- 172.21.15.57:0/2234095910 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f660413dd40 con 0x7f6604070af0 2023-10-19T13:02:59.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.335+0000 7f65fb7fe640 1 -- 172.21.15.57:0/2234095910 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v49) v1 ==== 118+0+0 (secure 0 0 0) 0x7f65f407a3c0 con 0x7f6604070af0 2023-10-19T13:02:59.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.337+0000 7f6608d8e640 1 -- 172.21.15.57:0/2234095910 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f65e406af60 msgr2=0x7f65e406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:59.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.337+0000 7f6608d8e640 1 --2- 172.21.15.57:0/2234095910 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f65e406af60 0x7f65e406d420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f65ec007fd0 tx=0x7f65ec009040 comp rx=0 tx=0).stop 2023-10-19T13:02:59.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.337+0000 7f6608d8e640 1 -- 172.21.15.57:0/2234095910 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6604070af0 msgr2=0x7f6604158880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:59.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.337+0000 7f6608d8e640 1 --2- 172.21.15.57:0/2234095910 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6604070af0 0x7f6604158880 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f65f4009780 tx=0x7f65f40038a0 comp rx=0 tx=0).stop 2023-10-19T13:02:59.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.337+0000 7f6608d8e640 1 -- 172.21.15.57:0/2234095910 shutdown_connections 2023-10-19T13:02:59.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.337+0000 7f6608d8e640 1 --2- 172.21.15.57:0/2234095910 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f65e406af60 0x7f65e406d420 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:59.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.337+0000 7f6608d8e640 1 --2- 172.21.15.57:0/2234095910 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6604141690 0x7f660413c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:59.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.337+0000 7f6608d8e640 1 --2- 172.21.15.57:0/2234095910 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f660413e520 0x7f660413c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:59.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.337+0000 7f6608d8e640 1 --2- 172.21.15.57:0/2234095910 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6604070af0 0x7f6604158880 unknown :-1 s=CLOSED pgs=1439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:59.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.337+0000 7f6608d8e640 1 -- 172.21.15.57:0/2234095910 >> 172.21.15.57:0/2234095910 conn(0x7f6604069930 msgr2=0x7f6604069d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:02:59.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.337+0000 7f6608d8e640 1 -- 172.21.15.57:0/2234095910 shutdown_connections 2023-10-19T13:02:59.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.337+0000 7f6608d8e640 1 -- 172.21.15.57:0/2234095910 wait complete. 2023-10-19T13:02:59.351 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-19T13:02:59.351 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-19T13:02:59.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.461+0000 7f3d76b9e640 1 -- 172.21.15.57:0/4182130063 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3d7015b040 msgr2=0x7f3d700cd620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:59.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.461+0000 7f3d76b9e640 1 --2- 172.21.15.57:0/4182130063 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3d7015b040 0x7f3d700cd620 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f3d6c009830 tx=0x7f3d6c02bd60 comp rx=0 tx=0).stop 2023-10-19T13:02:59.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.461+0000 7f3d76b9e640 1 -- 172.21.15.57:0/4182130063 shutdown_connections 2023-10-19T13:02:59.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.461+0000 7f3d76b9e640 1 --2- 172.21.15.57:0/4182130063 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3d700cbe10 0x7f3d700b0bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:59.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.461+0000 7f3d76b9e640 1 --2- 172.21.15.57:0/4182130063 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3d7015b040 0x7f3d700cd620 unknown :-1 s=CLOSED pgs=1441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:59.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.461+0000 7f3d76b9e640 1 -- 172.21.15.57:0/4182130063 >> 172.21.15.57:0/4182130063 conn(0x7f3d70099eb0 msgr2=0x7f3d700d0c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:02:59.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.462+0000 7f3d76b9e640 1 -- 172.21.15.57:0/4182130063 shutdown_connections 2023-10-19T13:02:59.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.462+0000 7f3d76b9e640 1 -- 172.21.15.57:0/4182130063 wait complete. 2023-10-19T13:02:59.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.463+0000 7f3d76b9e640 1 Processor -- start 2023-10-19T13:02:59.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.463+0000 7f3d76b9e640 1 -- start start 2023-10-19T13:02:59.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.464+0000 7f3d76b9e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3d700cbe10 0x7f3d701e5d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:59.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.464+0000 7f3d76b9e640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3d700cdb60 0x7f3d700d0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:59.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.464+0000 7f3d76b9e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3d7015b040 0x7f3d700d10d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:59.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.464+0000 7f3d76b9e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d701e6250 con 0x7f3d700cbe10 2023-10-19T13:02:59.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.464+0000 7f3d76b9e640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d701e3840 con 0x7f3d7015b040 2023-10-19T13:02:59.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.464+0000 7f3d76b9e640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d701e5b40 con 0x7f3d700cdb60 2023-10-19T13:02:59.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.464+0000 7f3d75b9c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3d700cbe10 0x7f3d701e5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:59.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.464+0000 7f3d75b9c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3d700cbe10 0x7f3d701e5d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:48568/0 (socket says 172.21.15.57:48568) 2023-10-19T13:02:59.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.464+0000 7f3d75b9c640 1 -- 172.21.15.57:0/368041462 learned_addr learned my addr 172.21.15.57:0/368041462 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:02:59.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.464+0000 7f3d7639d640 1 --2- 172.21.15.57:0/368041462 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3d7015b040 0x7f3d700d10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:59.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.464+0000 7f3d7539b640 1 --2- 172.21.15.57:0/368041462 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3d700cdb60 0x7f3d700d0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:59.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.464+0000 7f3d75b9c640 1 -- 172.21.15.57:0/368041462 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3d700cdb60 msgr2=0x7f3d700d0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:59.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.464+0000 7f3d75b9c640 1 --2- 172.21.15.57:0/368041462 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3d700cdb60 0x7f3d700d0b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:59.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.464+0000 7f3d75b9c640 1 -- 172.21.15.57:0/368041462 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3d7015b040 msgr2=0x7f3d700d10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:02:59.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.464+0000 7f3d75b9c640 1 --2- 172.21.15.57:0/368041462 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3d7015b040 0x7f3d700d10d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:02:59.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.464+0000 7f3d75b9c640 1 -- 172.21.15.57:0/368041462 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d6c009490 con 0x7f3d700cbe10 2023-10-19T13:02:59.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.464+0000 7f3d75b9c640 1 --2- 172.21.15.57:0/368041462 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3d700cbe10 0x7f3d701e5d10 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f3d6c009950 tx=0x7f3d6c0038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:02:59.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.464+0000 7f3d667fc640 1 -- 172.21.15.57:0/368041462 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d6c003ea0 con 0x7f3d700cbe10 2023-10-19T13:02:59.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.465+0000 7f3d667fc640 1 -- 172.21.15.57:0/368041462 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f3d6c034530 con 0x7f3d700cbe10 2023-10-19T13:02:59.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.465+0000 7f3d76b9e640 1 -- 172.21.15.57:0/368041462 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d701e6460 con 0x7f3d700cbe10 2023-10-19T13:02:59.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.465+0000 7f3d667fc640 1 -- 172.21.15.57:0/368041462 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d6c034940 con 0x7f3d700cbe10 2023-10-19T13:02:59.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.465+0000 7f3d76b9e640 1 -- 172.21.15.57:0/368041462 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d700c6c00 con 0x7f3d700cbe10 2023-10-19T13:02:59.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.466+0000 7f3d76b9e640 1 -- 172.21.15.57:0/368041462 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d70066960 con 0x7f3d700cbe10 2023-10-19T13:02:59.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.466+0000 7f3d667fc640 1 -- 172.21.15.57:0/368041462 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 58) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f3d6c034aa0 con 0x7f3d700cbe10 2023-10-19T13:02:59.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.467+0000 7f3d667fc640 1 --2- 172.21.15.57:0/368041462 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f3d5406af60 0x7f3d5406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:02:59.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.467+0000 7f3d667fc640 1 -- 172.21.15.57:0/368041462 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f3d6c0ac220 con 0x7f3d700cbe10 2023-10-19T13:02:59.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.467+0000 7f3d7539b640 1 --2- 172.21.15.57:0/368041462 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f3d5406af60 0x7f3d5406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:02:59.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.467+0000 7f3d7539b640 1 --2- 172.21.15.57:0/368041462 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f3d5406af60 0x7f3d5406d420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3d7015deb0 tx=0x7f3d60009040 comp rx=0 tx=0).ready entity=mgr.7540 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:02:59.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.471+0000 7f3d667fc640 1 -- 172.21.15.57:0/368041462 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3d6c0770f0 con 0x7f3d700cbe10 2023-10-19T13:02:59.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:02:59.683+0000 7f3d76b9e640 1 -- 172.21.15.57:0/368041462 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f3d7004da70 con 0x7f3d700cbe10 2023-10-19T13:03:00.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.353+0000 7f3d667fc640 1 -- 172.21.15.57:0/368041462 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v50) v1 ==== 122+0+0 (secure 0 0 0) 0x7f3d6c07a710 con 0x7f3d700cbe10 2023-10-19T13:03:00.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.356+0000 7f3d76b9e640 1 -- 172.21.15.57:0/368041462 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f3d5406af60 msgr2=0x7f3d5406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:03:00.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.356+0000 7f3d76b9e640 1 --2- 172.21.15.57:0/368041462 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f3d5406af60 0x7f3d5406d420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3d7015deb0 tx=0x7f3d60009040 comp rx=0 tx=0).stop 2023-10-19T13:03:00.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.356+0000 7f3d76b9e640 1 -- 172.21.15.57:0/368041462 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3d700cbe10 msgr2=0x7f3d701e5d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:03:00.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.356+0000 7f3d76b9e640 1 --2- 172.21.15.57:0/368041462 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3d700cbe10 0x7f3d701e5d10 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f3d6c009950 tx=0x7f3d6c0038a0 comp rx=0 tx=0).stop 2023-10-19T13:03:00.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.356+0000 7f3d76b9e640 1 -- 172.21.15.57:0/368041462 shutdown_connections 2023-10-19T13:03:00.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.356+0000 7f3d76b9e640 1 --2- 172.21.15.57:0/368041462 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f3d5406af60 0x7f3d5406d420 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:00.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.356+0000 7f3d76b9e640 1 --2- 172.21.15.57:0/368041462 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3d7015b040 0x7f3d700d10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:00.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.356+0000 7f3d76b9e640 1 --2- 172.21.15.57:0/368041462 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3d700cdb60 0x7f3d700d0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:00.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.356+0000 7f3d76b9e640 1 --2- 172.21.15.57:0/368041462 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3d700cbe10 0x7f3d701e5d10 unknown :-1 s=CLOSED pgs=1442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:00.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.356+0000 7f3d76b9e640 1 -- 172.21.15.57:0/368041462 >> 172.21.15.57:0/368041462 conn(0x7f3d70099eb0 msgr2=0x7f3d70150dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:03:00.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.356+0000 7f3d76b9e640 1 -- 172.21.15.57:0/368041462 shutdown_connections 2023-10-19T13:03:00.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.356+0000 7f3d76b9e640 1 -- 172.21.15.57:0/368041462 wait complete. 2023-10-19T13:03:00.372 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-19T13:03:00.373 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T13:03:00.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.485+0000 7fdb04a64640 1 -- 172.21.15.57:0/68440966 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdb00070920 msgr2=0x7fdb000affe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:03:00.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.485+0000 7fdb04a64640 1 --2- 172.21.15.57:0/68440966 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdb00070920 0x7fdb000affe0 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7fdaf8005fd0 tx=0x7fdaf802c240 comp rx=0 tx=0).stop 2023-10-19T13:03:00.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.485+0000 7fdb04a64640 1 -- 172.21.15.57:0/68440966 shutdown_connections 2023-10-19T13:03:00.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.485+0000 7fdb04a64640 1 --2- 172.21.15.57:0/68440966 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdb00141690 0x7fdb000703e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:00.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.485+0000 7fdb04a64640 1 --2- 172.21.15.57:0/68440966 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdb00070920 0x7fdb000affe0 unknown :-1 s=CLOSED pgs=1444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:00.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.485+0000 7fdb04a64640 1 -- 172.21.15.57:0/68440966 >> 172.21.15.57:0/68440966 conn(0x7fdb00069930 msgr2=0x7fdb00069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:03:00.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.486+0000 7fdb04a64640 1 -- 172.21.15.57:0/68440966 shutdown_connections 2023-10-19T13:03:00.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.486+0000 7fdb04a64640 1 -- 172.21.15.57:0/68440966 wait complete. 2023-10-19T13:03:00.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.487+0000 7fdb04a64640 1 Processor -- start 2023-10-19T13:03:00.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.487+0000 7fdb04a64640 1 -- start start 2023-10-19T13:03:00.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.488+0000 7fdb04a64640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdb00070920 0x7fdb001d6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:03:00.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.488+0000 7fdb04a64640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdb0013e520 0x7fdb001361f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:03:00.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.488+0000 7fdb04a64640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdb00141690 0x7fdb001d8df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:03:00.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.488+0000 7fdb04a64640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb0013a040 con 0x7fdb00070920 2023-10-19T13:03:00.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.488+0000 7fdb04a64640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb001d67a0 con 0x7fdb0013e520 2023-10-19T13:03:00.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.488+0000 7fdb04a64640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb001dc400 con 0x7fdb00141690 2023-10-19T13:03:00.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.488+0000 7fdafe59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdb00070920 0x7fdb001d6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:03:00.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.488+0000 7fdafe59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdb00070920 0x7fdb001d6260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:48596/0 (socket says 172.21.15.57:48596) 2023-10-19T13:03:00.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.488+0000 7fdafe59c640 1 -- 172.21.15.57:0/3028304121 learned_addr learned my addr 172.21.15.57:0/3028304121 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:03:00.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.488+0000 7fdafed9d640 1 --2- 172.21.15.57:0/3028304121 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdb00141690 0x7fdb001d8df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:03:00.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.488+0000 7fdaf7fff640 1 --2- 172.21.15.57:0/3028304121 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdb0013e520 0x7fdb001361f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:03:00.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.488+0000 7fdafe59c640 1 -- 172.21.15.57:0/3028304121 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdb00141690 msgr2=0x7fdb001d8df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:03:00.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.488+0000 7fdafe59c640 1 --2- 172.21.15.57:0/3028304121 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdb00141690 0x7fdb001d8df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:00.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.488+0000 7fdafe59c640 1 -- 172.21.15.57:0/3028304121 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdb0013e520 msgr2=0x7fdb001361f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:03:00.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.488+0000 7fdafe59c640 1 --2- 172.21.15.57:0/3028304121 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdb0013e520 0x7fdb001361f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:00.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.488+0000 7fdafe59c640 1 -- 172.21.15.57:0/3028304121 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaf8009900 con 0x7fdb00070920 2023-10-19T13:03:00.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.488+0000 7fdaf7fff640 1 --2- 172.21.15.57:0/3028304121 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdb0013e520 0x7fdb001361f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T13:03:00.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.488+0000 7fdafe59c640 1 --2- 172.21.15.57:0/3028304121 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdb00070920 0x7fdb001d6260 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7fdaf000c4b0 tx=0x7fdaf000c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:03:00.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.488+0000 7fdaf6ffd640 1 -- 172.21.15.57:0/3028304121 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaf0005480 con 0x7fdb00070920 2023-10-19T13:03:00.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.489+0000 7fdaf6ffd640 1 -- 172.21.15.57:0/3028304121 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fdaf00036e0 con 0x7fdb00070920 2023-10-19T13:03:00.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.489+0000 7fdb04a64640 1 -- 172.21.15.57:0/3028304121 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb001dc6e0 con 0x7fdb00070920 2023-10-19T13:03:00.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.489+0000 7fdaf6ffd640 1 -- 172.21.15.57:0/3028304121 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaf0003b30 con 0x7fdb00070920 2023-10-19T13:03:00.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.489+0000 7fdb04a64640 1 -- 172.21.15.57:0/3028304121 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb0013a290 con 0x7fdb00070920 2023-10-19T13:03:00.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.490+0000 7fdb04a64640 1 -- 172.21.15.57:0/3028304121 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb000690e0 con 0x7fdb00070920 2023-10-19T13:03:00.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.490+0000 7fdaf6ffd640 1 -- 172.21.15.57:0/3028304121 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 58) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fdaf0003c90 con 0x7fdb00070920 2023-10-19T13:03:00.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.491+0000 7fdaf6ffd640 1 --2- 172.21.15.57:0/3028304121 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7fdae006af60 0x7fdae006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:03:00.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.491+0000 7fdaf6ffd640 1 -- 172.21.15.57:0/3028304121 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fdaf00891e0 con 0x7fdb00070920 2023-10-19T13:03:00.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.491+0000 7fdaf7fff640 1 --2- 172.21.15.57:0/3028304121 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7fdae006af60 0x7fdae006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:03:00.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.491+0000 7fdaf7fff640 1 --2- 172.21.15.57:0/3028304121 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7fdae006af60 0x7fdae006d420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fdb0006a2a0 tx=0x7fdae8005cf0 comp rx=0 tx=0).ready entity=mgr.7540 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:03:00.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.495+0000 7fdaf6ffd640 1 -- 172.21.15.57:0/3028304121 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fdaf0053f90 con 0x7fdb00070920 2023-10-19T13:03:00.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.706+0000 7fdb04a64640 1 -- 172.21.15.57:0/3028304121 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdb0006fce0 con 0x7fdb00070920 2023-10-19T13:03:00.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.706+0000 7fdaf6ffd640 1 -- 172.21.15.57:0/3028304121 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4912 (secure 0 0 0) 0x7fdaf00575b0 con 0x7fdb00070920 2023-10-19T13:03:00.709 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:03:00.710 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":50,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/2033977424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2033977424},{"type":"v1","addr":"172.21.15.179:6836","nonce":2033977424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":8568,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/981095684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":981095684},{"type":"v1","addr":"172.21.15.57:6834","nonce":981095684}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":8571,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/383147069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":383147069},{"type":"v1","addr":"172.21.15.57:6835","nonce":383147069}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T13:02:57.697820+0000","modified":"2023-10-19T13:02:59.684888+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8577},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8577":{"gid":8577,"name":"d","rank":0,"incarnation":48,"state":"up:active","state_seq":3,"addr":"172.21.15.179:6837/768231146","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":768231146},{"type":"v1","addr":"172.21.15.179:6837","nonce":768231146}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2023-10-19T13:03:00.710 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 50 2023-10-19T13:03:00.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.708+0000 7fdb04a64640 1 -- 172.21.15.57:0/3028304121 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7fdae006af60 msgr2=0x7fdae006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:03:00.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.708+0000 7fdb04a64640 1 --2- 172.21.15.57:0/3028304121 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7fdae006af60 0x7fdae006d420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fdb0006a2a0 tx=0x7fdae8005cf0 comp rx=0 tx=0).stop 2023-10-19T13:03:00.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.708+0000 7fdb04a64640 1 -- 172.21.15.57:0/3028304121 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdb00070920 msgr2=0x7fdb001d6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:03:00.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.708+0000 7fdb04a64640 1 --2- 172.21.15.57:0/3028304121 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdb00070920 0x7fdb001d6260 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7fdaf000c4b0 tx=0x7fdaf000c970 comp rx=0 tx=0).stop 2023-10-19T13:03:00.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.708+0000 7fdb04a64640 1 -- 172.21.15.57:0/3028304121 shutdown_connections 2023-10-19T13:03:00.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.708+0000 7fdb04a64640 1 --2- 172.21.15.57:0/3028304121 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7fdae006af60 0x7fdae006d420 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:00.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.708+0000 7fdb04a64640 1 --2- 172.21.15.57:0/3028304121 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdb00141690 0x7fdb001d8df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:00.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.708+0000 7fdb04a64640 1 --2- 172.21.15.57:0/3028304121 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdb0013e520 0x7fdb001361f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:00.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.708+0000 7fdb04a64640 1 --2- 172.21.15.57:0/3028304121 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdb00070920 0x7fdb001d6260 unknown :-1 s=CLOSED pgs=1445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:00.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.708+0000 7fdb04a64640 1 -- 172.21.15.57:0/3028304121 >> 172.21.15.57:0/3028304121 conn(0x7fdb00069930 msgr2=0x7fdb001e0830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:03:00.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.708+0000 7fdb04a64640 1 -- 172.21.15.57:0/3028304121 shutdown_connections 2023-10-19T13:03:00.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.708+0000 7fdb04a64640 1 -- 172.21.15.57:0/3028304121 wait complete. 2023-10-19T13:03:00.722 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T13:03:00.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.827+0000 7fbd67c44640 1 -- 172.21.15.57:0/3560188517 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd600b1920 msgr2=0x7fbd6006fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:03:00.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.827+0000 7fbd67c44640 1 --2- 172.21.15.57:0/3560188517 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd600b1920 0x7fbd6006fa70 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7fbd600abdc0 tx=0x7fbd5c02c250 comp rx=0 tx=0).stop 2023-10-19T13:03:00.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.828+0000 7fbd67c44640 1 -- 172.21.15.57:0/3560188517 shutdown_connections 2023-10-19T13:03:00.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.828+0000 7fbd67c44640 1 --2- 172.21.15.57:0/3560188517 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbd600c9bf0 0x7fbd60154af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:00.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.828+0000 7fbd67c44640 1 --2- 172.21.15.57:0/3560188517 >> v2:172.21.15.179:3300/0 conn(0x7fbd6006ffb0 0x7fbd600c96b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:00.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.828+0000 7fbd67c44640 1 --2- 172.21.15.57:0/3560188517 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd600b1920 0x7fbd6006fa70 unknown :-1 s=CLOSED pgs=1446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:00.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.828+0000 7fbd67c44640 1 -- 172.21.15.57:0/3560188517 >> 172.21.15.57:0/3560188517 conn(0x7fbd60069930 msgr2=0x7fbd60069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:03:00.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.828+0000 7fbd67c44640 1 -- 172.21.15.57:0/3560188517 shutdown_connections 2023-10-19T13:03:00.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.828+0000 7fbd67c44640 1 -- 172.21.15.57:0/3560188517 wait complete. 2023-10-19T13:03:00.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.829+0000 7fbd67c44640 1 Processor -- start 2023-10-19T13:03:00.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.829+0000 7fbd67c44640 1 -- start start 2023-10-19T13:03:00.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.830+0000 7fbd67c44640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbd6006ffb0 0x7fbd6015c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:03:00.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.830+0000 7fbd67c44640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbd600b1920 0x7fbd600ac460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:03:00.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.830+0000 7fbd67c44640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd600c9bf0 0x7fbd601f11b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:03:00.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.830+0000 7fbd67c44640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd600ad830 con 0x7fbd600c9bf0 2023-10-19T13:03:00.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.830+0000 7fbd67c44640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd600ac9a0 con 0x7fbd6006ffb0 2023-10-19T13:03:00.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.830+0000 7fbd67c44640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd600acc20 con 0x7fbd600b1920 2023-10-19T13:03:00.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.830+0000 7fbd661e1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd600c9bf0 0x7fbd601f11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:03:00.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.830+0000 7fbd661e1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd600c9bf0 0x7fbd601f11b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:48606/0 (socket says 172.21.15.57:48606) 2023-10-19T13:03:00.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.830+0000 7fbd661e1640 1 -- 172.21.15.57:0/580499070 learned_addr learned my addr 172.21.15.57:0/580499070 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:03:00.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.830+0000 7fbd659e0640 1 --2- 172.21.15.57:0/580499070 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbd6006ffb0 0x7fbd6015c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:03:00.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.830+0000 7fbd651df640 1 --2- 172.21.15.57:0/580499070 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbd600b1920 0x7fbd600ac460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:03:00.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.830+0000 7fbd661e1640 1 -- 172.21.15.57:0/580499070 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbd600b1920 msgr2=0x7fbd600ac460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:03:00.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.830+0000 7fbd661e1640 1 --2- 172.21.15.57:0/580499070 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbd600b1920 0x7fbd600ac460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:00.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.830+0000 7fbd661e1640 1 -- 172.21.15.57:0/580499070 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbd6006ffb0 msgr2=0x7fbd6015c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:03:00.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.830+0000 7fbd661e1640 1 --2- 172.21.15.57:0/580499070 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbd6006ffb0 0x7fbd6015c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:00.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.830+0000 7fbd661e1640 1 -- 172.21.15.57:0/580499070 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd5c009900 con 0x7fbd600c9bf0 2023-10-19T13:03:00.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.830+0000 7fbd659e0640 1 --2- 172.21.15.57:0/580499070 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbd6006ffb0 0x7fbd6015c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:03:00.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.830+0000 7fbd661e1640 1 --2- 172.21.15.57:0/580499070 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd600c9bf0 0x7fbd601f11b0 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7fbd5800b410 tx=0x7fbd5800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:03:00.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.830+0000 7fbd567fc640 1 -- 172.21.15.57:0/580499070 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd58011830 con 0x7fbd600c9bf0 2023-10-19T13:03:00.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.830+0000 7fbd567fc640 1 -- 172.21.15.57:0/580499070 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fbd58011aa0 con 0x7fbd600c9bf0 2023-10-19T13:03:00.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.830+0000 7fbd567fc640 1 -- 172.21.15.57:0/580499070 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd58010380 con 0x7fbd600c9bf0 2023-10-19T13:03:00.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.830+0000 7fbd67c44640 1 -- 172.21.15.57:0/580499070 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd600acf00 con 0x7fbd600c9bf0 2023-10-19T13:03:00.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.830+0000 7fbd67c44640 1 -- 172.21.15.57:0/580499070 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd600ad4c0 con 0x7fbd600c9bf0 2023-10-19T13:03:00.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.832+0000 7fbd67c44640 1 -- 172.21.15.57:0/580499070 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd600ad680 con 0x7fbd600c9bf0 2023-10-19T13:03:00.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.832+0000 7fbd567fc640 1 -- 172.21.15.57:0/580499070 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 58) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fbd580104e0 con 0x7fbd600c9bf0 2023-10-19T13:03:00.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.832+0000 7fbd567fc640 1 --2- 172.21.15.57:0/580499070 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7fbd4406af90 0x7fbd4406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:03:00.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.832+0000 7fbd567fc640 1 -- 172.21.15.57:0/580499070 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fbd58089d30 con 0x7fbd600c9bf0 2023-10-19T13:03:00.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.833+0000 7fbd659e0640 1 --2- 172.21.15.57:0/580499070 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7fbd4406af90 0x7fbd4406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:03:00.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.833+0000 7fbd659e0640 1 --2- 172.21.15.57:0/580499070 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7fbd4406af90 0x7fbd4406d450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fbd5c036000 tx=0x7fbd5c004ab0 comp rx=0 tx=0).ready entity=mgr.7540 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:03:00.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:00.837+0000 7fbd567fc640 1 -- 172.21.15.57:0/580499070 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fbd58054a40 con 0x7fbd600c9bf0 2023-10-19T13:03:01.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.013+0000 7fbd67c44640 1 -- 172.21.15.57:0/580499070 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbd600690e0 con 0x7fbd600c9bf0 2023-10-19T13:03:01.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.014+0000 7fbd567fc640 1 -- 172.21.15.57:0/580499070 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v130) v1 ==== 74+0+16771 (secure 0 0 0) 0x7fbd58015020 con 0x7fbd600c9bf0 2023-10-19T13:03:01.018 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:03:01.018 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":130,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T13:02:59.204190+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2023-10-19T13:02:55.916158+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"129","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2023-10-19T13:02:56.675559+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"130","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T13:03:01.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.016+0000 7fbd67c44640 1 -- 172.21.15.57:0/580499070 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7fbd4406af90 msgr2=0x7fbd4406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:03:01.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.016+0000 7fbd67c44640 1 --2- 172.21.15.57:0/580499070 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7fbd4406af90 0x7fbd4406d450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fbd5c036000 tx=0x7fbd5c004ab0 comp rx=0 tx=0).stop 2023-10-19T13:03:01.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.016+0000 7fbd67c44640 1 -- 172.21.15.57:0/580499070 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd600c9bf0 msgr2=0x7fbd601f11b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:03:01.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.016+0000 7fbd67c44640 1 --2- 172.21.15.57:0/580499070 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd600c9bf0 0x7fbd601f11b0 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7fbd5800b410 tx=0x7fbd5800b8d0 comp rx=0 tx=0).stop 2023-10-19T13:03:01.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.016+0000 7fbd67c44640 1 -- 172.21.15.57:0/580499070 shutdown_connections 2023-10-19T13:03:01.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.016+0000 7fbd67c44640 1 --2- 172.21.15.57:0/580499070 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7fbd4406af90 0x7fbd4406d450 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:01.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.016+0000 7fbd67c44640 1 --2- 172.21.15.57:0/580499070 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd600c9bf0 0x7fbd601f11b0 unknown :-1 s=CLOSED pgs=1447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:01.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.016+0000 7fbd67c44640 1 --2- 172.21.15.57:0/580499070 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbd600b1920 0x7fbd600ac460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:01.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.016+0000 7fbd67c44640 1 --2- 172.21.15.57:0/580499070 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbd6006ffb0 0x7fbd6015c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:01.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.016+0000 7fbd67c44640 1 -- 172.21.15.57:0/580499070 >> 172.21.15.57:0/580499070 conn(0x7fbd60069930 msgr2=0x7fbd600c8b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:03:01.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.016+0000 7fbd67c44640 1 -- 172.21.15.57:0/580499070 shutdown_connections 2023-10-19T13:03:01.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.017+0000 7fbd67c44640 1 -- 172.21.15.57:0/580499070 wait complete. 2023-10-19T13:03:01.030 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-19T13:03:01.030 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-19T13:03:01.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.136+0000 7f5a72e5d640 1 -- 172.21.15.57:0/1658252594 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a6c06bec0 msgr2=0x7f5a6c06a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:03:01.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.136+0000 7f5a72e5d640 1 --2- 172.21.15.57:0/1658252594 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a6c06bec0 0x7f5a6c06a730 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f5a6c0cc700 tx=0x7f5a5c02bd90 comp rx=0 tx=0).stop 2023-10-19T13:03:01.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.136+0000 7f5a72e5d640 1 -- 172.21.15.57:0/1658252594 shutdown_connections 2023-10-19T13:03:01.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.136+0000 7f5a72e5d640 1 --2- 172.21.15.57:0/1658252594 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a6c06bec0 0x7f5a6c06a730 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:01.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.136+0000 7f5a72e5d640 1 --2- 172.21.15.57:0/1658252594 >> v2:172.21.15.179:3300/0 conn(0x7f5a6c0ac870 0x7f5a6c06b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:01.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.137+0000 7f5a72e5d640 1 -- 172.21.15.57:0/1658252594 >> 172.21.15.57:0/1658252594 conn(0x7f5a6c06fae0 msgr2=0x7f5a6c06fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:03:01.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.137+0000 7f5a72e5d640 1 -- 172.21.15.57:0/1658252594 shutdown_connections 2023-10-19T13:03:01.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.137+0000 7f5a72e5d640 1 -- 172.21.15.57:0/1658252594 wait complete. 2023-10-19T13:03:01.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.138+0000 7f5a72e5d640 1 Processor -- start 2023-10-19T13:03:01.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.138+0000 7f5a72e5d640 1 -- start start 2023-10-19T13:03:01.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.139+0000 7f5a72e5d640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5a6c06bec0 0x7f5a6c146ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:03:01.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.139+0000 7f5a72e5d640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5a6c0ac870 0x7f5a6c1e2500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:03:01.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.139+0000 7f5a72e5d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a6c156790 0x7f5a6c1e2a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:03:01.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.139+0000 7f5a72e5d640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a6c1e3630 con 0x7f5a6c156790 2023-10-19T13:03:01.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.139+0000 7f5a72e5d640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a6c1e3b30 con 0x7f5a6c0ac870 2023-10-19T13:03:01.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.139+0000 7f5a72e5d640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a6c1e3070 con 0x7f5a6c06bec0 2023-10-19T13:03:01.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.139+0000 7f5a713fa640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a6c156790 0x7f5a6c1e2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:03:01.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.139+0000 7f5a713fa640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a6c156790 0x7f5a6c1e2a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:48622/0 (socket says 172.21.15.57:48622) 2023-10-19T13:03:01.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.139+0000 7f5a713fa640 1 -- 172.21.15.57:0/2382141526 learned_addr learned my addr 172.21.15.57:0/2382141526 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:03:01.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.139+0000 7f5a70bf9640 1 --2- 172.21.15.57:0/2382141526 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5a6c06bec0 0x7f5a6c146ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:03:01.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.139+0000 7f5a6bfff640 1 --2- 172.21.15.57:0/2382141526 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5a6c0ac870 0x7f5a6c1e2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:03:01.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.139+0000 7f5a713fa640 1 -- 172.21.15.57:0/2382141526 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5a6c06bec0 msgr2=0x7f5a6c146ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:03:01.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.139+0000 7f5a713fa640 1 --2- 172.21.15.57:0/2382141526 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5a6c06bec0 0x7f5a6c146ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:01.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.139+0000 7f5a713fa640 1 -- 172.21.15.57:0/2382141526 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5a6c0ac870 msgr2=0x7f5a6c1e2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:03:01.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.139+0000 7f5a713fa640 1 --2- 172.21.15.57:0/2382141526 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5a6c0ac870 0x7f5a6c1e2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:01.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.139+0000 7f5a713fa640 1 -- 172.21.15.57:0/2382141526 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a5c009560 con 0x7f5a6c156790 2023-10-19T13:03:01.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.139+0000 7f5a70bf9640 1 --2- 172.21.15.57:0/2382141526 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5a6c06bec0 0x7f5a6c146ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:03:01.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.139+0000 7f5a6bfff640 1 --2- 172.21.15.57:0/2382141526 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5a6c0ac870 0x7f5a6c1e2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:03:01.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.139+0000 7f5a713fa640 1 --2- 172.21.15.57:0/2382141526 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a6c156790 0x7f5a6c1e2a40 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f5a5c006010 tx=0x7f5a5c02dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:03:01.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.139+0000 7f5a697fa640 1 -- 172.21.15.57:0/2382141526 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a5c0035f0 con 0x7f5a6c156790 2023-10-19T13:03:01.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.139+0000 7f5a697fa640 1 -- 172.21.15.57:0/2382141526 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f5a5c003840 con 0x7f5a6c156790 2023-10-19T13:03:01.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.139+0000 7f5a72e5d640 1 -- 172.21.15.57:0/2382141526 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a6c1e3350 con 0x7f5a6c156790 2023-10-19T13:03:01.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.139+0000 7f5a697fa640 1 -- 172.21.15.57:0/2382141526 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a5c003bf0 con 0x7f5a6c156790 2023-10-19T13:03:01.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.139+0000 7f5a72e5d640 1 -- 172.21.15.57:0/2382141526 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a6c0b0990 con 0x7f5a6c156790 2023-10-19T13:03:01.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.141+0000 7f5a72e5d640 1 -- 172.21.15.57:0/2382141526 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a6c0b0b80 con 0x7f5a6c156790 2023-10-19T13:03:01.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.141+0000 7f5a697fa640 1 -- 172.21.15.57:0/2382141526 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 58) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f5a5c003d50 con 0x7f5a6c156790 2023-10-19T13:03:01.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.141+0000 7f5a697fa640 1 --2- 172.21.15.57:0/2382141526 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f5a5806af60 0x7f5a5806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:03:01.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.142+0000 7f5a697fa640 1 -- 172.21.15.57:0/2382141526 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f5a5c0ac400 con 0x7f5a6c156790 2023-10-19T13:03:01.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.142+0000 7f5a70bf9640 1 --2- 172.21.15.57:0/2382141526 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f5a5806af60 0x7f5a5806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:03:01.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.142+0000 7f5a70bf9640 1 --2- 172.21.15.57:0/2382141526 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f5a5806af60 0x7f5a5806d420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5a640045b0 tx=0x7f5a64009040 comp rx=0 tx=0).ready entity=mgr.7540 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:03:01.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.146+0000 7f5a697fa640 1 -- 172.21.15.57:0/2382141526 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f5a5c0773f0 con 0x7f5a6c156790 2023-10-19T13:03:01.320 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.317+0000 7f5a72e5d640 1 -- 172.21.15.57:0/2382141526 --> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5a6c1596e0 con 0x7f5a5806af60 2023-10-19T13:03:01.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.326+0000 7f5a697fa640 1 -- 172.21.15.57:0/2382141526 <== mgr.7540 v2:172.21.15.179:6832/4155586759 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+282613 (secure 0 0 0) 0x7f5a6400ed30 con 0x7f5a5806af60 2023-10-19T13:03:01.331 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:03:01.333 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2023-10-19T13:03:01.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.328+0000 7f5a72e5d640 1 -- 172.21.15.57:0/2382141526 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f5a5806af60 msgr2=0x7f5a5806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:03:01.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.328+0000 7f5a72e5d640 1 --2- 172.21.15.57:0/2382141526 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f5a5806af60 0x7f5a5806d420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5a640045b0 tx=0x7f5a64009040 comp rx=0 tx=0).stop 2023-10-19T13:03:01.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.328+0000 7f5a72e5d640 1 -- 172.21.15.57:0/2382141526 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a6c156790 msgr2=0x7f5a6c1e2a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:03:01.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.328+0000 7f5a72e5d640 1 --2- 172.21.15.57:0/2382141526 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a6c156790 0x7f5a6c1e2a40 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f5a5c006010 tx=0x7f5a5c02dab0 comp rx=0 tx=0).stop 2023-10-19T13:03:01.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.329+0000 7f5a72e5d640 1 -- 172.21.15.57:0/2382141526 shutdown_connections 2023-10-19T13:03:01.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.329+0000 7f5a72e5d640 1 --2- 172.21.15.57:0/2382141526 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f5a5806af60 0x7f5a5806d420 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:01.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.329+0000 7f5a72e5d640 1 --2- 172.21.15.57:0/2382141526 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a6c156790 0x7f5a6c1e2a40 unknown :-1 s=CLOSED pgs=1449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:01.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.329+0000 7f5a72e5d640 1 --2- 172.21.15.57:0/2382141526 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5a6c0ac870 0x7f5a6c1e2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:01.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.329+0000 7f5a72e5d640 1 --2- 172.21.15.57:0/2382141526 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5a6c06bec0 0x7f5a6c146ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:01.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.329+0000 7f5a72e5d640 1 -- 172.21.15.57:0/2382141526 >> 172.21.15.57:0/2382141526 conn(0x7f5a6c06fae0 msgr2=0x7f5a6c06d0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:03:01.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.329+0000 7f5a72e5d640 1 -- 172.21.15.57:0/2382141526 shutdown_connections 2023-10-19T13:03:01.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.329+0000 7f5a72e5d640 1 -- 172.21.15.57:0/2382141526 wait complete. 2023-10-19T13:03:01.342 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2023-10-19T13:02:59.644291+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592618,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":962,"num_read_kb":1946,"num_write":349,"num_write_kb":2428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3060,"ondisk_log_size":3060,"up":100,"acting":100,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":68,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":5,"kb":754974720,"kb_used":337944,"kb_used_data":13512,"kb_used_omap":12,"kb_used_meta":324275,"kb_avail":754636776,"statfs":{"total":773094113280,"available":772748058624,"internally_reserved":0,"allocated":13836288,"data_stored":7537572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":332058169},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2250,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000830"},"pg_stats":[{"pgid":"16.3f","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.408576+0000","last_change":"2023-10-19T13:02:58.218772+0000","last_active":"2023-10-19T13:02:59.408576+0000","last_peered":"2023-10-19T13:02:59.408576+0000","last_clean":"2023-10-19T13:02:59.408576+0000","last_became_active":"2023-10-19T13:02:58.218486+0000","last_became_peered":"2023-10-19T13:02:58.218486+0000","last_unstale":"2023-10-19T13:02:59.408576+0000","last_undegraded":"2023-10-19T13:02:59.408576+0000","last_fullsized":"2023-10-19T13:02:59.408576+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:16:09.032988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3e","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.409259+0000","last_change":"2023-10-19T13:02:58.225728+0000","last_active":"2023-10-19T13:02:59.409259+0000","last_peered":"2023-10-19T13:02:59.409259+0000","last_clean":"2023-10-19T13:02:59.409259+0000","last_became_active":"2023-10-19T13:02:58.225213+0000","last_became_peered":"2023-10-19T13:02:58.225213+0000","last_unstale":"2023-10-19T13:02:59.409259+0000","last_undegraded":"2023-10-19T13:02:59.409259+0000","last_fullsized":"2023-10-19T13:02:59.409259+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:03:30.317716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.3d","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.408296+0000","last_change":"2023-10-19T13:02:58.216453+0000","last_active":"2023-10-19T13:02:59.408296+0000","last_peered":"2023-10-19T13:02:59.408296+0000","last_clean":"2023-10-19T13:02:59.408296+0000","last_became_active":"2023-10-19T13:02:58.216192+0000","last_became_peered":"2023-10-19T13:02:58.216192+0000","last_unstale":"2023-10-19T13:02:59.408296+0000","last_undegraded":"2023-10-19T13:02:59.408296+0000","last_fullsized":"2023-10-19T13:02:59.408296+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:41:39.973763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3c","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.409059+0000","last_change":"2023-10-19T13:02:58.222548+0000","last_active":"2023-10-19T13:02:59.409059+0000","last_peered":"2023-10-19T13:02:59.409059+0000","last_clean":"2023-10-19T13:02:59.409059+0000","last_became_active":"2023-10-19T13:02:58.222097+0000","last_became_peered":"2023-10-19T13:02:58.222097+0000","last_unstale":"2023-10-19T13:02:59.409059+0000","last_undegraded":"2023-10-19T13:02:59.409059+0000","last_fullsized":"2023-10-19T13:02:59.409059+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:38:45.359597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.3b","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.210548+0000","last_change":"2023-10-19T13:02:58.223189+0000","last_active":"2023-10-19T13:02:59.210548+0000","last_peered":"2023-10-19T13:02:59.210548+0000","last_clean":"2023-10-19T13:02:59.210548+0000","last_became_active":"2023-10-19T13:02:58.222524+0000","last_became_peered":"2023-10-19T13:02:58.222524+0000","last_unstale":"2023-10-19T13:02:59.210548+0000","last_undegraded":"2023-10-19T13:02:59.210548+0000","last_fullsized":"2023-10-19T13:02:59.210548+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:11:08.241375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.3a","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.408536+0000","last_change":"2023-10-19T13:02:58.227127+0000","last_active":"2023-10-19T13:02:59.408536+0000","last_peered":"2023-10-19T13:02:59.408536+0000","last_clean":"2023-10-19T13:02:59.408536+0000","last_became_active":"2023-10-19T13:02:58.226839+0000","last_became_peered":"2023-10-19T13:02:58.226839+0000","last_unstale":"2023-10-19T13:02:59.408536+0000","last_undegraded":"2023-10-19T13:02:59.408536+0000","last_fullsized":"2023-10-19T13:02:59.408536+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:37:57.278166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.39","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.408396+0000","last_change":"2023-10-19T13:02:58.228791+0000","last_active":"2023-10-19T13:02:59.408396+0000","last_peered":"2023-10-19T13:02:59.408396+0000","last_clean":"2023-10-19T13:02:59.408396+0000","last_became_active":"2023-10-19T13:02:58.228570+0000","last_became_peered":"2023-10-19T13:02:58.228570+0000","last_unstale":"2023-10-19T13:02:59.408396+0000","last_undegraded":"2023-10-19T13:02:59.408396+0000","last_fullsized":"2023-10-19T13:02:59.408396+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:12:12.095113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:02:57.320575+0000","last_change":"2023-10-19T13:02:57.320575+0000","last_active":"2023-10-19T13:02:57.320575+0000","last_peered":"2023-10-19T13:02:57.320575+0000","last_clean":"2023-10-19T13:02:57.320575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:02:57.320575+0000","last_undegraded":"2023-10-19T13:02:57.320575+0000","last_fullsized":"2023-10-19T13:02:57.320575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.37","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.409390+0000","last_change":"2023-10-19T13:02:58.225273+0000","last_active":"2023-10-19T13:02:59.409390+0000","last_peered":"2023-10-19T13:02:59.409390+0000","last_clean":"2023-10-19T13:02:59.409390+0000","last_became_active":"2023-10-19T13:02:58.224972+0000","last_became_peered":"2023-10-19T13:02:58.224972+0000","last_unstale":"2023-10-19T13:02:59.409390+0000","last_undegraded":"2023-10-19T13:02:59.409390+0000","last_fullsized":"2023-10-19T13:02:59.409390+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:41:23.057082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.36","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.210783+0000","last_change":"2023-10-19T13:02:58.227439+0000","last_active":"2023-10-19T13:02:59.210783+0000","last_peered":"2023-10-19T13:02:59.210783+0000","last_clean":"2023-10-19T13:02:59.210783+0000","last_became_active":"2023-10-19T13:02:58.226745+0000","last_became_peered":"2023-10-19T13:02:58.226745+0000","last_unstale":"2023-10-19T13:02:59.210783+0000","last_undegraded":"2023-10-19T13:02:59.210783+0000","last_fullsized":"2023-10-19T13:02:59.210783+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:32:53.289930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:02:57.320575+0000","last_change":"2023-10-19T13:02:57.320575+0000","last_active":"2023-10-19T13:02:57.320575+0000","last_peered":"2023-10-19T13:02:57.320575+0000","last_clean":"2023-10-19T13:02:57.320575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:02:57.320575+0000","last_undegraded":"2023-10-19T13:02:57.320575+0000","last_fullsized":"2023-10-19T13:02:57.320575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:02:57.320575+0000","last_change":"2023-10-19T13:02:57.320575+0000","last_active":"2023-10-19T13:02:57.320575+0000","last_peered":"2023-10-19T13:02:57.320575+0000","last_clean":"2023-10-19T13:02:57.320575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:02:57.320575+0000","last_undegraded":"2023-10-19T13:02:57.320575+0000","last_fullsized":"2023-10-19T13:02:57.320575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.33","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.408415+0000","last_change":"2023-10-19T13:02:58.216355+0000","last_active":"2023-10-19T13:02:59.408415+0000","last_peered":"2023-10-19T13:02:59.408415+0000","last_clean":"2023-10-19T13:02:59.408415+0000","last_became_active":"2023-10-19T13:02:58.216086+0000","last_became_peered":"2023-10-19T13:02:58.216086+0000","last_unstale":"2023-10-19T13:02:59.408415+0000","last_undegraded":"2023-10-19T13:02:59.408415+0000","last_fullsized":"2023-10-19T13:02:59.408415+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:03:32.123711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:02:57.320575+0000","last_change":"2023-10-19T13:02:57.320575+0000","last_active":"2023-10-19T13:02:57.320575+0000","last_peered":"2023-10-19T13:02:57.320575+0000","last_clean":"2023-10-19T13:02:57.320575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:02:57.320575+0000","last_undegraded":"2023-10-19T13:02:57.320575+0000","last_fullsized":"2023-10-19T13:02:57.320575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:02:57.320575+0000","last_change":"2023-10-19T13:02:57.320575+0000","last_active":"2023-10-19T13:02:57.320575+0000","last_peered":"2023-10-19T13:02:57.320575+0000","last_clean":"2023-10-19T13:02:57.320575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:02:57.320575+0000","last_undegraded":"2023-10-19T13:02:57.320575+0000","last_fullsized":"2023-10-19T13:02:57.320575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:02:57.320575+0000","last_change":"2023-10-19T13:02:57.320575+0000","last_active":"2023-10-19T13:02:57.320575+0000","last_peered":"2023-10-19T13:02:57.320575+0000","last_clean":"2023-10-19T13:02:57.320575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:02:57.320575+0000","last_undegraded":"2023-10-19T13:02:57.320575+0000","last_fullsized":"2023-10-19T13:02:57.320575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.2f","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.408703+0000","last_change":"2023-10-19T13:02:58.213645+0000","last_active":"2023-10-19T13:02:59.408703+0000","last_peered":"2023-10-19T13:02:59.408703+0000","last_clean":"2023-10-19T13:02:59.408703+0000","last_became_active":"2023-10-19T13:02:58.213386+0000","last_became_peered":"2023-10-19T13:02:58.213386+0000","last_unstale":"2023-10-19T13:02:59.408703+0000","last_undegraded":"2023-10-19T13:02:59.408703+0000","last_fullsized":"2023-10-19T13:02:59.408703+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:50:56.716775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2e","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.409335+0000","last_change":"2023-10-19T13:02:58.228898+0000","last_active":"2023-10-19T13:02:59.409335+0000","last_peered":"2023-10-19T13:02:59.409335+0000","last_clean":"2023-10-19T13:02:59.409335+0000","last_became_active":"2023-10-19T13:02:58.228631+0000","last_became_peered":"2023-10-19T13:02:58.228631+0000","last_unstale":"2023-10-19T13:02:59.409335+0000","last_undegraded":"2023-10-19T13:02:59.409335+0000","last_fullsized":"2023-10-19T13:02:59.409335+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:32:53.289930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.2d","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.408419+0000","last_change":"2023-10-19T13:02:58.228433+0000","last_active":"2023-10-19T13:02:59.408419+0000","last_peered":"2023-10-19T13:02:59.408419+0000","last_clean":"2023-10-19T13:02:59.408419+0000","last_became_active":"2023-10-19T13:02:58.228160+0000","last_became_peered":"2023-10-19T13:02:58.228160+0000","last_unstale":"2023-10-19T13:02:59.408419+0000","last_undegraded":"2023-10-19T13:02:59.408419+0000","last_fullsized":"2023-10-19T13:02:59.408419+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:59:25.379540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:02:57.320575+0000","last_change":"2023-10-19T13:02:57.320575+0000","last_active":"2023-10-19T13:02:57.320575+0000","last_peered":"2023-10-19T13:02:57.320575+0000","last_clean":"2023-10-19T13:02:57.320575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:02:57.320575+0000","last_undegraded":"2023-10-19T13:02:57.320575+0000","last_fullsized":"2023-10-19T13:02:57.320575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:02:57.320575+0000","last_change":"2023-10-19T13:02:57.320575+0000","last_active":"2023-10-19T13:02:57.320575+0000","last_peered":"2023-10-19T13:02:57.320575+0000","last_clean":"2023-10-19T13:02:57.320575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:02:57.320575+0000","last_undegraded":"2023-10-19T13:02:57.320575+0000","last_fullsized":"2023-10-19T13:02:57.320575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.2a","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.210515+0000","last_change":"2023-10-19T13:02:58.221055+0000","last_active":"2023-10-19T13:02:59.210515+0000","last_peered":"2023-10-19T13:02:59.210515+0000","last_clean":"2023-10-19T13:02:59.210515+0000","last_became_active":"2023-10-19T13:02:58.220762+0000","last_became_peered":"2023-10-19T13:02:58.220762+0000","last_unstale":"2023-10-19T13:02:59.210515+0000","last_undegraded":"2023-10-19T13:02:59.210515+0000","last_fullsized":"2023-10-19T13:02:59.210515+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:00:45.825647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.29","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.408993+0000","last_change":"2023-10-19T13:02:58.212624+0000","last_active":"2023-10-19T13:02:59.408993+0000","last_peered":"2023-10-19T13:02:59.408993+0000","last_clean":"2023-10-19T13:02:59.408993+0000","last_became_active":"2023-10-19T13:02:58.212350+0000","last_became_peered":"2023-10-19T13:02:58.212350+0000","last_unstale":"2023-10-19T13:02:59.408993+0000","last_undegraded":"2023-10-19T13:02:59.408993+0000","last_fullsized":"2023-10-19T13:02:59.408993+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:55:35.630464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.28","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.210365+0000","last_change":"2023-10-19T13:02:58.216883+0000","last_active":"2023-10-19T13:02:59.210365+0000","last_peered":"2023-10-19T13:02:59.210365+0000","last_clean":"2023-10-19T13:02:59.210365+0000","last_became_active":"2023-10-19T13:02:58.216598+0000","last_became_peered":"2023-10-19T13:02:58.216598+0000","last_unstale":"2023-10-19T13:02:59.210365+0000","last_undegraded":"2023-10-19T13:02:59.210365+0000","last_fullsized":"2023-10-19T13:02:59.210365+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:02:01.165779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:02:57.320575+0000","last_change":"2023-10-19T13:02:57.320575+0000","last_active":"2023-10-19T13:02:57.320575+0000","last_peered":"2023-10-19T13:02:57.320575+0000","last_clean":"2023-10-19T13:02:57.320575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:02:57.320575+0000","last_undegraded":"2023-10-19T13:02:57.320575+0000","last_fullsized":"2023-10-19T13:02:57.320575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:02:57.320575+0000","last_change":"2023-10-19T13:02:57.320575+0000","last_active":"2023-10-19T13:02:57.320575+0000","last_peered":"2023-10-19T13:02:57.320575+0000","last_clean":"2023-10-19T13:02:57.320575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:02:57.320575+0000","last_undegraded":"2023-10-19T13:02:57.320575+0000","last_fullsized":"2023-10-19T13:02:57.320575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:02:57.320575+0000","last_change":"2023-10-19T13:02:57.320575+0000","last_active":"2023-10-19T13:02:57.320575+0000","last_peered":"2023-10-19T13:02:57.320575+0000","last_clean":"2023-10-19T13:02:57.320575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:02:57.320575+0000","last_undegraded":"2023-10-19T13:02:57.320575+0000","last_fullsized":"2023-10-19T13:02:57.320575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.24","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.409125+0000","last_change":"2023-10-19T13:02:58.227441+0000","last_active":"2023-10-19T13:02:59.409125+0000","last_peered":"2023-10-19T13:02:59.409125+0000","last_clean":"2023-10-19T13:02:59.409125+0000","last_became_active":"2023-10-19T13:02:58.227166+0000","last_became_peered":"2023-10-19T13:02:58.227166+0000","last_unstale":"2023-10-19T13:02:59.409125+0000","last_undegraded":"2023-10-19T13:02:59.409125+0000","last_fullsized":"2023-10-19T13:02:59.409125+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:11:08.241375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:02:57.320575+0000","last_change":"2023-10-19T13:02:57.320575+0000","last_active":"2023-10-19T13:02:57.320575+0000","last_peered":"2023-10-19T13:02:57.320575+0000","last_clean":"2023-10-19T13:02:57.320575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:02:57.320575+0000","last_undegraded":"2023-10-19T13:02:57.320575+0000","last_fullsized":"2023-10-19T13:02:57.320575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:02:57.320575+0000","last_change":"2023-10-19T13:02:57.320575+0000","last_active":"2023-10-19T13:02:57.320575+0000","last_peered":"2023-10-19T13:02:57.320575+0000","last_clean":"2023-10-19T13:02:57.320575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:02:57.320575+0000","last_undegraded":"2023-10-19T13:02:57.320575+0000","last_fullsized":"2023-10-19T13:02:57.320575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.21","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.408430+0000","last_change":"2023-10-19T13:02:58.229241+0000","last_active":"2023-10-19T13:02:59.408430+0000","last_peered":"2023-10-19T13:02:59.408430+0000","last_clean":"2023-10-19T13:02:59.408430+0000","last_became_active":"2023-10-19T13:02:58.229007+0000","last_became_peered":"2023-10-19T13:02:58.229007+0000","last_unstale":"2023-10-19T13:02:59.408430+0000","last_undegraded":"2023-10-19T13:02:59.408430+0000","last_fullsized":"2023-10-19T13:02:59.408430+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:02:01.165779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:02:57.320575+0000","last_change":"2023-10-19T13:02:57.320575+0000","last_active":"2023-10-19T13:02:57.320575+0000","last_peered":"2023-10-19T13:02:57.320575+0000","last_clean":"2023-10-19T13:02:57.320575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:02:57.320575+0000","last_undegraded":"2023-10-19T13:02:57.320575+0000","last_fullsized":"2023-10-19T13:02:57.320575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.409258+0000","last_change":"2023-10-19T13:02:58.224690+0000","last_active":"2023-10-19T13:02:59.409258+0000","last_peered":"2023-10-19T13:02:59.409258+0000","last_clean":"2023-10-19T13:02:59.409258+0000","last_became_active":"2023-10-19T13:02:58.224407+0000","last_became_peered":"2023-10-19T13:02:58.224407+0000","last_unstale":"2023-10-19T13:02:59.409258+0000","last_undegraded":"2023-10-19T13:02:59.409258+0000","last_fullsized":"2023-10-19T13:02:59.409258+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:09:55.205778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"129'4","reported_seq":16,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.409388+0000","last_change":"2023-10-19T13:02:57.341868+0000","last_active":"2023-10-19T13:02:59.409388+0000","last_peered":"2023-10-19T13:02:59.409388+0000","last_clean":"2023-10-19T13:02:59.409388+0000","last_became_active":"2023-10-19T13:02:57.341592+0000","last_became_peered":"2023-10-19T13:02:57.341592+0000","last_unstale":"2023-10-19T13:02:59.409388+0000","last_undegraded":"2023-10-19T13:02:59.409388+0000","last_fullsized":"2023-10-19T13:02:59.409388+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_clean_scrub_stamp":"2023-10-19T13:02:56.313383+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:50:36.546546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:02:57.320575+0000","last_change":"2023-10-19T13:02:57.320575+0000","last_active":"2023-10-19T13:02:57.320575+0000","last_peered":"2023-10-19T13:02:57.320575+0000","last_clean":"2023-10-19T13:02:57.320575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:02:57.320575+0000","last_undegraded":"2023-10-19T13:02:57.320575+0000","last_fullsized":"2023-10-19T13:02:57.320575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:02:56.313383+0000","last_change":"2023-10-19T13:02:56.313383+0000","last_active":"2023-10-19T13:02:56.313383+0000","last_peered":"2023-10-19T13:02:56.313383+0000","last_clean":"2023-10-19T13:02:56.313383+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:02:56.313383+0000","last_undegraded":"2023-10-19T13:02:56.313383+0000","last_fullsized":"2023-10-19T13:02:56.313383+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_clean_scrub_stamp":"2023-10-19T13:02:56.313383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:02:57.320575+0000","last_change":"2023-10-19T13:02:57.320575+0000","last_active":"2023-10-19T13:02:57.320575+0000","last_peered":"2023-10-19T13:02:57.320575+0000","last_clean":"2023-10-19T13:02:57.320575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:02:57.320575+0000","last_undegraded":"2023-10-19T13:02:57.320575+0000","last_fullsized":"2023-10-19T13:02:57.320575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:02:56.313383+0000","last_change":"2023-10-19T13:02:56.313383+0000","last_active":"2023-10-19T13:02:56.313383+0000","last_peered":"2023-10-19T13:02:56.313383+0000","last_clean":"2023-10-19T13:02:56.313383+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:02:56.313383+0000","last_undegraded":"2023-10-19T13:02:56.313383+0000","last_fullsized":"2023-10-19T13:02:56.313383+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_clean_scrub_stamp":"2023-10-19T13:02:56.313383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:02:57.320575+0000","last_change":"2023-10-19T13:02:57.320575+0000","last_active":"2023-10-19T13:02:57.320575+0000","last_peered":"2023-10-19T13:02:57.320575+0000","last_clean":"2023-10-19T13:02:57.320575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:02:57.320575+0000","last_undegraded":"2023-10-19T13:02:57.320575+0000","last_fullsized":"2023-10-19T13:02:57.320575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.1c","version":"129'2","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.210725+0000","last_change":"2023-10-19T13:02:57.347704+0000","last_active":"2023-10-19T13:02:59.210725+0000","last_peered":"2023-10-19T13:02:59.210725+0000","last_clean":"2023-10-19T13:02:59.210725+0000","last_became_active":"2023-10-19T13:02:57.347427+0000","last_became_peered":"2023-10-19T13:02:57.347427+0000","last_unstale":"2023-10-19T13:02:59.210725+0000","last_undegraded":"2023-10-19T13:02:59.210725+0000","last_fullsized":"2023-10-19T13:02:59.210725+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_clean_scrub_stamp":"2023-10-19T13:02:56.313383+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:53:29.668027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.409184+0000","last_change":"2023-10-19T13:02:58.226429+0000","last_active":"2023-10-19T13:02:59.409184+0000","last_peered":"2023-10-19T13:02:59.409184+0000","last_clean":"2023-10-19T13:02:59.409184+0000","last_became_active":"2023-10-19T13:02:58.226163+0000","last_became_peered":"2023-10-19T13:02:58.226163+0000","last_unstale":"2023-10-19T13:02:59.409184+0000","last_undegraded":"2023-10-19T13:02:59.409184+0000","last_fullsized":"2023-10-19T13:02:59.409184+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:33:31.616543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:02:56.313383+0000","last_change":"2023-10-19T13:02:56.313383+0000","last_active":"2023-10-19T13:02:56.313383+0000","last_peered":"2023-10-19T13:02:56.313383+0000","last_clean":"2023-10-19T13:02:56.313383+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:02:56.313383+0000","last_undegraded":"2023-10-19T13:02:56.313383+0000","last_fullsized":"2023-10-19T13:02:56.313383+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_clean_scrub_stamp":"2023-10-19T13:02:56.313383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.408663+0000","last_change":"2023-10-19T13:02:58.227324+0000","last_active":"2023-10-19T13:02:59.408663+0000","last_peered":"2023-10-19T13:02:59.408663+0000","last_clean":"2023-10-19T13:02:59.408663+0000","last_became_active":"2023-10-19T13:02:58.226965+0000","last_became_peered":"2023-10-19T13:02:58.226965+0000","last_unstale":"2023-10-19T13:02:59.408663+0000","last_undegraded":"2023-10-19T13:02:59.408663+0000","last_fullsized":"2023-10-19T13:02:59.408663+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:12:16.687425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:02:56.313383+0000","last_change":"2023-10-19T13:02:56.313383+0000","last_active":"2023-10-19T13:02:56.313383+0000","last_peered":"2023-10-19T13:02:56.313383+0000","last_clean":"2023-10-19T13:02:56.313383+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:02:56.313383+0000","last_undegraded":"2023-10-19T13:02:56.313383+0000","last_fullsized":"2023-10-19T13:02:56.313383+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_clean_scrub_stamp":"2023-10-19T13:02:56.313383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.409382+0000","last_change":"2023-10-19T13:02:58.225680+0000","last_active":"2023-10-19T13:02:59.409382+0000","last_peered":"2023-10-19T13:02:59.409382+0000","last_clean":"2023-10-19T13:02:59.409382+0000","last_became_active":"2023-10-19T13:02:58.225250+0000","last_became_peered":"2023-10-19T13:02:58.225250+0000","last_unstale":"2023-10-19T13:02:59.409382+0000","last_undegraded":"2023-10-19T13:02:59.409382+0000","last_fullsized":"2023-10-19T13:02:59.409382+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:28:08.887992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:02:56.313383+0000","last_change":"2023-10-19T13:02:56.313383+0000","last_active":"2023-10-19T13:02:56.313383+0000","last_peered":"2023-10-19T13:02:56.313383+0000","last_clean":"2023-10-19T13:02:56.313383+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:02:56.313383+0000","last_undegraded":"2023-10-19T13:02:56.313383+0000","last_fullsized":"2023-10-19T13:02:56.313383+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_clean_scrub_stamp":"2023-10-19T13:02:56.313383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.409399+0000","last_change":"2023-10-19T13:02:58.217721+0000","last_active":"2023-10-19T13:02:59.409399+0000","last_peered":"2023-10-19T13:02:59.409399+0000","last_clean":"2023-10-19T13:02:59.409399+0000","last_became_active":"2023-10-19T13:02:58.216651+0000","last_became_peered":"2023-10-19T13:02:58.216651+0000","last_unstale":"2023-10-19T13:02:59.409399+0000","last_undegraded":"2023-10-19T13:02:59.409399+0000","last_fullsized":"2023-10-19T13:02:59.409399+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:27:04.472644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.408912+0000","last_change":"2023-10-19T13:02:57.344873+0000","last_active":"2023-10-19T13:02:59.408912+0000","last_peered":"2023-10-19T13:02:59.408912+0000","last_clean":"2023-10-19T13:02:59.408912+0000","last_became_active":"2023-10-19T13:02:57.344610+0000","last_became_peered":"2023-10-19T13:02:57.344610+0000","last_unstale":"2023-10-19T13:02:59.408912+0000","last_undegraded":"2023-10-19T13:02:59.408912+0000","last_fullsized":"2023-10-19T13:02:59.408912+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_clean_scrub_stamp":"2023-10-19T13:02:56.313383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:46:15.256442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.409180+0000","last_change":"2023-10-19T13:02:58.225271+0000","last_active":"2023-10-19T13:02:59.409180+0000","last_peered":"2023-10-19T13:02:59.409180+0000","last_clean":"2023-10-19T13:02:59.409180+0000","last_became_active":"2023-10-19T13:02:58.225000+0000","last_became_peered":"2023-10-19T13:02:58.225000+0000","last_unstale":"2023-10-19T13:02:59.409180+0000","last_undegraded":"2023-10-19T13:02:59.409180+0000","last_fullsized":"2023-10-19T13:02:59.409180+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:49:07.743223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:02:56.313383+0000","last_change":"2023-10-19T13:02:56.313383+0000","last_active":"2023-10-19T13:02:56.313383+0000","last_peered":"2023-10-19T13:02:56.313383+0000","last_clean":"2023-10-19T13:02:56.313383+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:02:56.313383+0000","last_undegraded":"2023-10-19T13:02:56.313383+0000","last_fullsized":"2023-10-19T13:02:56.313383+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_clean_scrub_stamp":"2023-10-19T13:02:56.313383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.210483+0000","last_change":"2023-10-19T13:02:58.221059+0000","last_active":"2023-10-19T13:02:59.210483+0000","last_peered":"2023-10-19T13:02:59.210483+0000","last_clean":"2023-10-19T13:02:59.210483+0000","last_became_active":"2023-10-19T13:02:58.220775+0000","last_became_peered":"2023-10-19T13:02:58.220775+0000","last_unstale":"2023-10-19T13:02:59.210483+0000","last_undegraded":"2023-10-19T13:02:59.210483+0000","last_fullsized":"2023-10-19T13:02:59.210483+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:37:57.278166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.408871+0000","last_change":"2023-10-19T13:02:57.346719+0000","last_active":"2023-10-19T13:02:59.408871+0000","last_peered":"2023-10-19T13:02:59.408871+0000","last_clean":"2023-10-19T13:02:59.408871+0000","last_became_active":"2023-10-19T13:02:57.346389+0000","last_became_peered":"2023-10-19T13:02:57.346389+0000","last_unstale":"2023-10-19T13:02:59.408871+0000","last_undegraded":"2023-10-19T13:02:59.408871+0000","last_fullsized":"2023-10-19T13:02:59.408871+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_clean_scrub_stamp":"2023-10-19T13:02:56.313383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:16:13.963090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:02:57.320575+0000","last_change":"2023-10-19T13:02:57.320575+0000","last_active":"2023-10-19T13:02:57.320575+0000","last_peered":"2023-10-19T13:02:57.320575+0000","last_clean":"2023-10-19T13:02:57.320575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:02:57.320575+0000","last_undegraded":"2023-10-19T13:02:57.320575+0000","last_fullsized":"2023-10-19T13:02:57.320575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:02:56.313383+0000","last_change":"2023-10-19T13:02:56.313383+0000","last_active":"2023-10-19T13:02:56.313383+0000","last_peered":"2023-10-19T13:02:56.313383+0000","last_clean":"2023-10-19T13:02:56.313383+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:02:56.313383+0000","last_undegraded":"2023-10-19T13:02:56.313383+0000","last_fullsized":"2023-10-19T13:02:56.313383+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_clean_scrub_stamp":"2023-10-19T13:02:56.313383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.210684+0000","last_change":"2023-10-19T13:02:58.222283+0000","last_active":"2023-10-19T13:02:59.210684+0000","last_peered":"2023-10-19T13:02:59.210684+0000","last_clean":"2023-10-19T13:02:59.210684+0000","last_became_active":"2023-10-19T13:02:58.220921+0000","last_became_peered":"2023-10-19T13:02:58.220921+0000","last_unstale":"2023-10-19T13:02:59.210684+0000","last_undegraded":"2023-10-19T13:02:59.210684+0000","last_fullsized":"2023-10-19T13:02:59.210684+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:03:30.317716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.14","version":"129'6","reported_seq":18,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.409178+0000","last_change":"2023-10-19T13:02:57.350320+0000","last_active":"2023-10-19T13:02:59.409178+0000","last_peered":"2023-10-19T13:02:59.409178+0000","last_clean":"2023-10-19T13:02:59.409178+0000","last_became_active":"2023-10-19T13:02:57.350049+0000","last_became_peered":"2023-10-19T13:02:57.350049+0000","last_unstale":"2023-10-19T13:02:59.409178+0000","last_undegraded":"2023-10-19T13:02:59.409178+0000","last_fullsized":"2023-10-19T13:02:59.409178+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_clean_scrub_stamp":"2023-10-19T13:02:56.313383+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:20:18.570502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:02:57.320575+0000","last_change":"2023-10-19T13:02:57.320575+0000","last_active":"2023-10-19T13:02:57.320575+0000","last_peered":"2023-10-19T13:02:57.320575+0000","last_clean":"2023-10-19T13:02:57.320575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:02:57.320575+0000","last_undegraded":"2023-10-19T13:02:57.320575+0000","last_fullsized":"2023-10-19T13:02:57.320575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.13","version":"129'2","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.210740+0000","last_change":"2023-10-19T13:02:57.345527+0000","last_active":"2023-10-19T13:02:59.210740+0000","last_peered":"2023-10-19T13:02:59.210740+0000","last_clean":"2023-10-19T13:02:59.210740+0000","last_became_active":"2023-10-19T13:02:57.345022+0000","last_became_peered":"2023-10-19T13:02:57.345022+0000","last_unstale":"2023-10-19T13:02:59.210740+0000","last_undegraded":"2023-10-19T13:02:59.210740+0000","last_fullsized":"2023-10-19T13:02:59.210740+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_clean_scrub_stamp":"2023-10-19T13:02:56.313383+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:12:11.087920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.409209+0000","last_change":"2023-10-19T13:02:58.228358+0000","last_active":"2023-10-19T13:02:59.409209+0000","last_peered":"2023-10-19T13:02:59.409209+0000","last_clean":"2023-10-19T13:02:59.409209+0000","last_became_active":"2023-10-19T13:02:58.228101+0000","last_became_peered":"2023-10-19T13:02:58.228101+0000","last_unstale":"2023-10-19T13:02:59.409209+0000","last_undegraded":"2023-10-19T13:02:59.409209+0000","last_fullsized":"2023-10-19T13:02:59.409209+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:55:57.076264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.409218+0000","last_change":"2023-10-19T13:02:57.337039+0000","last_active":"2023-10-19T13:02:59.409218+0000","last_peered":"2023-10-19T13:02:59.409218+0000","last_clean":"2023-10-19T13:02:59.409218+0000","last_became_active":"2023-10-19T13:02:57.336770+0000","last_became_peered":"2023-10-19T13:02:57.336770+0000","last_unstale":"2023-10-19T13:02:59.409218+0000","last_undegraded":"2023-10-19T13:02:59.409218+0000","last_fullsized":"2023-10-19T13:02:59.409218+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_clean_scrub_stamp":"2023-10-19T13:02:56.313383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:17:15.888716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:02:57.320575+0000","last_change":"2023-10-19T13:02:57.320575+0000","last_active":"2023-10-19T13:02:57.320575+0000","last_peered":"2023-10-19T13:02:57.320575+0000","last_clean":"2023-10-19T13:02:57.320575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:02:57.320575+0000","last_undegraded":"2023-10-19T13:02:57.320575+0000","last_fullsized":"2023-10-19T13:02:57.320575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:02:56.313383+0000","last_change":"2023-10-19T13:02:56.313383+0000","last_active":"2023-10-19T13:02:56.313383+0000","last_peered":"2023-10-19T13:02:56.313383+0000","last_clean":"2023-10-19T13:02:56.313383+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:02:56.313383+0000","last_undegraded":"2023-10-19T13:02:56.313383+0000","last_fullsized":"2023-10-19T13:02:56.313383+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_clean_scrub_stamp":"2023-10-19T13:02:56.313383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.408829+0000","last_change":"2023-10-19T13:02:58.217109+0000","last_active":"2023-10-19T13:02:59.408829+0000","last_peered":"2023-10-19T13:02:59.408829+0000","last_clean":"2023-10-19T13:02:59.408829+0000","last_became_active":"2023-10-19T13:02:58.216846+0000","last_became_peered":"2023-10-19T13:02:58.216846+0000","last_unstale":"2023-10-19T13:02:59.408829+0000","last_undegraded":"2023-10-19T13:02:59.408829+0000","last_fullsized":"2023-10-19T13:02:59.408829+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:29:09.842513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:02:56.313383+0000","last_change":"2023-10-19T13:02:56.313383+0000","last_active":"2023-10-19T13:02:56.313383+0000","last_peered":"2023-10-19T13:02:56.313383+0000","last_clean":"2023-10-19T13:02:56.313383+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:02:56.313383+0000","last_undegraded":"2023-10-19T13:02:56.313383+0000","last_fullsized":"2023-10-19T13:02:56.313383+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_clean_scrub_stamp":"2023-10-19T13:02:56.313383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:02:57.320575+0000","last_change":"2023-10-19T13:02:57.320575+0000","last_active":"2023-10-19T13:02:57.320575+0000","last_peered":"2023-10-19T13:02:57.320575+0000","last_clean":"2023-10-19T13:02:57.320575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:02:57.320575+0000","last_undegraded":"2023-10-19T13:02:57.320575+0000","last_fullsized":"2023-10-19T13:02:57.320575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.f","version":"129'1","reported_seq":13,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.409335+0000","last_change":"2023-10-19T13:02:57.346451+0000","last_active":"2023-10-19T13:02:59.409335+0000","last_peered":"2023-10-19T13:02:59.409335+0000","last_clean":"2023-10-19T13:02:59.409335+0000","last_became_active":"2023-10-19T13:02:57.346192+0000","last_became_peered":"2023-10-19T13:02:57.346192+0000","last_unstale":"2023-10-19T13:02:59.409335+0000","last_undegraded":"2023-10-19T13:02:59.409335+0000","last_fullsized":"2023-10-19T13:02:59.409335+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_clean_scrub_stamp":"2023-10-19T13:02:56.313383+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:53:41.138926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.408776+0000","last_change":"2023-10-19T13:02:58.225451+0000","last_active":"2023-10-19T13:02:59.408776+0000","last_peered":"2023-10-19T13:02:59.408776+0000","last_clean":"2023-10-19T13:02:59.408776+0000","last_became_active":"2023-10-19T13:02:58.225176+0000","last_became_peered":"2023-10-19T13:02:58.225176+0000","last_unstale":"2023-10-19T13:02:59.408776+0000","last_undegraded":"2023-10-19T13:02:59.408776+0000","last_fullsized":"2023-10-19T13:02:59.408776+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:00:45.825647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:02:56.313383+0000","last_change":"2023-10-19T13:02:56.313383+0000","last_active":"2023-10-19T13:02:56.313383+0000","last_peered":"2023-10-19T13:02:56.313383+0000","last_clean":"2023-10-19T13:02:56.313383+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:02:56.313383+0000","last_undegraded":"2023-10-19T13:02:56.313383+0000","last_fullsized":"2023-10-19T13:02:56.313383+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_clean_scrub_stamp":"2023-10-19T13:02:56.313383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.210608+0000","last_change":"2023-10-19T13:02:58.222612+0000","last_active":"2023-10-19T13:02:59.210608+0000","last_peered":"2023-10-19T13:02:59.210608+0000","last_clean":"2023-10-19T13:02:59.210608+0000","last_became_active":"2023-10-19T13:02:58.222277+0000","last_became_peered":"2023-10-19T13:02:58.222277+0000","last_unstale":"2023-10-19T13:02:59.210608+0000","last_undegraded":"2023-10-19T13:02:59.210608+0000","last_fullsized":"2023-10-19T13:02:59.210608+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:55:57.076264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:02:56.313383+0000","last_change":"2023-10-19T13:02:56.313383+0000","last_active":"2023-10-19T13:02:56.313383+0000","last_peered":"2023-10-19T13:02:56.313383+0000","last_clean":"2023-10-19T13:02:56.313383+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:02:56.313383+0000","last_undegraded":"2023-10-19T13:02:56.313383+0000","last_fullsized":"2023-10-19T13:02:56.313383+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_clean_scrub_stamp":"2023-10-19T13:02:56.313383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.408353+0000","last_change":"2023-10-19T13:02:58.218268+0000","last_active":"2023-10-19T13:02:59.408353+0000","last_peered":"2023-10-19T13:02:59.408353+0000","last_clean":"2023-10-19T13:02:59.408353+0000","last_became_active":"2023-10-19T13:02:58.217998+0000","last_became_peered":"2023-10-19T13:02:58.217998+0000","last_unstale":"2023-10-19T13:02:59.408353+0000","last_undegraded":"2023-10-19T13:02:59.408353+0000","last_fullsized":"2023-10-19T13:02:59.408353+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:17:05.559019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:02:56.313383+0000","last_change":"2023-10-19T13:02:56.313383+0000","last_active":"2023-10-19T13:02:56.313383+0000","last_peered":"2023-10-19T13:02:56.313383+0000","last_clean":"2023-10-19T13:02:56.313383+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:02:56.313383+0000","last_undegraded":"2023-10-19T13:02:56.313383+0000","last_fullsized":"2023-10-19T13:02:56.313383+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_clean_scrub_stamp":"2023-10-19T13:02:56.313383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.408542+0000","last_change":"2023-10-19T13:02:58.222727+0000","last_active":"2023-10-19T13:02:59.408542+0000","last_peered":"2023-10-19T13:02:59.408542+0000","last_clean":"2023-10-19T13:02:59.408542+0000","last_became_active":"2023-10-19T13:02:58.222434+0000","last_became_peered":"2023-10-19T13:02:58.222434+0000","last_unstale":"2023-10-19T13:02:59.408542+0000","last_undegraded":"2023-10-19T13:02:59.408542+0000","last_fullsized":"2023-10-19T13:02:59.408542+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:58:57.535121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:02:56.313383+0000","last_change":"2023-10-19T13:02:56.313383+0000","last_active":"2023-10-19T13:02:56.313383+0000","last_peered":"2023-10-19T13:02:56.313383+0000","last_clean":"2023-10-19T13:02:56.313383+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:02:56.313383+0000","last_undegraded":"2023-10-19T13:02:56.313383+0000","last_fullsized":"2023-10-19T13:02:56.313383+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_clean_scrub_stamp":"2023-10-19T13:02:56.313383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.408762+0000","last_change":"2023-10-19T13:02:58.227358+0000","last_active":"2023-10-19T13:02:59.408762+0000","last_peered":"2023-10-19T13:02:59.408762+0000","last_clean":"2023-10-19T13:02:59.408762+0000","last_became_active":"2023-10-19T13:02:58.227073+0000","last_became_peered":"2023-10-19T13:02:58.227073+0000","last_unstale":"2023-10-19T13:02:59.408762+0000","last_undegraded":"2023-10-19T13:02:59.408762+0000","last_fullsized":"2023-10-19T13:02:59.408762+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:12:30.411021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.408686+0000","last_change":"2023-10-19T13:02:57.349912+0000","last_active":"2023-10-19T13:02:59.408686+0000","last_peered":"2023-10-19T13:02:59.408686+0000","last_clean":"2023-10-19T13:02:59.408686+0000","last_became_active":"2023-10-19T13:02:57.349674+0000","last_became_peered":"2023-10-19T13:02:57.349674+0000","last_unstale":"2023-10-19T13:02:59.408686+0000","last_undegraded":"2023-10-19T13:02:59.408686+0000","last_fullsized":"2023-10-19T13:02:59.408686+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_clean_scrub_stamp":"2023-10-19T13:02:56.313383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:51:32.174702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:02:57.320575+0000","last_change":"2023-10-19T13:02:57.320575+0000","last_active":"2023-10-19T13:02:57.320575+0000","last_peered":"2023-10-19T13:02:57.320575+0000","last_clean":"2023-10-19T13:02:57.320575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:02:57.320575+0000","last_undegraded":"2023-10-19T13:02:57.320575+0000","last_fullsized":"2023-10-19T13:02:57.320575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:02:56.313383+0000","last_change":"2023-10-19T13:02:56.313383+0000","last_active":"2023-10-19T13:02:56.313383+0000","last_peered":"2023-10-19T13:02:56.313383+0000","last_clean":"2023-10-19T13:02:56.313383+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:02:56.313383+0000","last_undegraded":"2023-10-19T13:02:56.313383+0000","last_fullsized":"2023-10-19T13:02:56.313383+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_clean_scrub_stamp":"2023-10-19T13:02:56.313383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.409130+0000","last_change":"2023-10-19T13:02:58.218497+0000","last_active":"2023-10-19T13:02:59.409130+0000","last_peered":"2023-10-19T13:02:59.409130+0000","last_clean":"2023-10-19T13:02:59.409130+0000","last_became_active":"2023-10-19T13:02:58.218231+0000","last_became_peered":"2023-10-19T13:02:58.218231+0000","last_unstale":"2023-10-19T13:02:59.409130+0000","last_undegraded":"2023-10-19T13:02:59.409130+0000","last_fullsized":"2023-10-19T13:02:59.409130+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:25:48.439527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.7","version":"129'2","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.210804+0000","last_change":"2023-10-19T13:02:57.345702+0000","last_active":"2023-10-19T13:02:59.210804+0000","last_peered":"2023-10-19T13:02:59.210804+0000","last_clean":"2023-10-19T13:02:59.210804+0000","last_became_active":"2023-10-19T13:02:57.345270+0000","last_became_peered":"2023-10-19T13:02:57.345270+0000","last_unstale":"2023-10-19T13:02:59.210804+0000","last_undegraded":"2023-10-19T13:02:59.210804+0000","last_fullsized":"2023-10-19T13:02:59.210804+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_clean_scrub_stamp":"2023-10-19T13:02:56.313383+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:20:18.570502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:02:57.320575+0000","last_change":"2023-10-19T13:02:57.320575+0000","last_active":"2023-10-19T13:02:57.320575+0000","last_peered":"2023-10-19T13:02:57.320575+0000","last_clean":"2023-10-19T13:02:57.320575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:02:57.320575+0000","last_undegraded":"2023-10-19T13:02:57.320575+0000","last_fullsized":"2023-10-19T13:02:57.320575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.408950+0000","last_change":"2023-10-19T13:02:57.339607+0000","last_active":"2023-10-19T13:02:59.408950+0000","last_peered":"2023-10-19T13:02:59.408950+0000","last_clean":"2023-10-19T13:02:59.408950+0000","last_became_active":"2023-10-19T13:02:57.339223+0000","last_became_peered":"2023-10-19T13:02:57.339223+0000","last_unstale":"2023-10-19T13:02:59.408950+0000","last_undegraded":"2023-10-19T13:02:59.408950+0000","last_fullsized":"2023-10-19T13:02:59.408950+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_clean_scrub_stamp":"2023-10-19T13:02:56.313383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:18:16.079990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.409228+0000","last_change":"2023-10-19T13:02:58.225583+0000","last_active":"2023-10-19T13:02:59.409228+0000","last_peered":"2023-10-19T13:02:59.409228+0000","last_clean":"2023-10-19T13:02:59.409228+0000","last_became_active":"2023-10-19T13:02:58.225294+0000","last_became_peered":"2023-10-19T13:02:58.225294+0000","last_unstale":"2023-10-19T13:02:59.409228+0000","last_undegraded":"2023-10-19T13:02:59.409228+0000","last_fullsized":"2023-10-19T13:02:59.409228+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:47:15.411344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.409189+0000","last_change":"2023-10-19T13:02:57.339438+0000","last_active":"2023-10-19T13:02:59.409189+0000","last_peered":"2023-10-19T13:02:59.409189+0000","last_clean":"2023-10-19T13:02:59.409189+0000","last_became_active":"2023-10-19T13:02:57.339138+0000","last_became_peered":"2023-10-19T13:02:57.339138+0000","last_unstale":"2023-10-19T13:02:59.409189+0000","last_undegraded":"2023-10-19T13:02:59.409189+0000","last_fullsized":"2023-10-19T13:02:59.409189+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_clean_scrub_stamp":"2023-10-19T13:02:56.313383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:53:22.719704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:02:57.320575+0000","last_change":"2023-10-19T13:02:57.320575+0000","last_active":"2023-10-19T13:02:57.320575+0000","last_peered":"2023-10-19T13:02:57.320575+0000","last_clean":"2023-10-19T13:02:57.320575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:02:57.320575+0000","last_undegraded":"2023-10-19T13:02:57.320575+0000","last_fullsized":"2023-10-19T13:02:57.320575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.210721+0000","last_change":"2023-10-19T13:02:57.345079+0000","last_active":"2023-10-19T13:02:59.210721+0000","last_peered":"2023-10-19T13:02:59.210721+0000","last_clean":"2023-10-19T13:02:59.210721+0000","last_became_active":"2023-10-19T13:02:57.344768+0000","last_became_peered":"2023-10-19T13:02:57.344768+0000","last_unstale":"2023-10-19T13:02:59.210721+0000","last_undegraded":"2023-10-19T13:02:59.210721+0000","last_fullsized":"2023-10-19T13:02:59.210721+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_clean_scrub_stamp":"2023-10-19T13:02:56.313383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:43:51.920104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:02:57.320575+0000","last_change":"2023-10-19T13:02:57.320575+0000","last_active":"2023-10-19T13:02:57.320575+0000","last_peered":"2023-10-19T13:02:57.320575+0000","last_clean":"2023-10-19T13:02:57.320575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:02:57.320575+0000","last_undegraded":"2023-10-19T13:02:57.320575+0000","last_fullsized":"2023-10-19T13:02:57.320575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:02:56.313383+0000","last_change":"2023-10-19T13:02:56.313383+0000","last_active":"2023-10-19T13:02:56.313383+0000","last_peered":"2023-10-19T13:02:56.313383+0000","last_clean":"2023-10-19T13:02:56.313383+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:02:56.313383+0000","last_undegraded":"2023-10-19T13:02:56.313383+0000","last_fullsized":"2023-10-19T13:02:56.313383+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_clean_scrub_stamp":"2023-10-19T13:02:56.313383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:02:57.320575+0000","last_change":"2023-10-19T13:02:57.320575+0000","last_active":"2023-10-19T13:02:57.320575+0000","last_peered":"2023-10-19T13:02:57.320575+0000","last_clean":"2023-10-19T13:02:57.320575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:02:57.320575+0000","last_undegraded":"2023-10-19T13:02:57.320575+0000","last_fullsized":"2023-10-19T13:02:57.320575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:02:56.313383+0000","last_change":"2023-10-19T13:02:56.313383+0000","last_active":"2023-10-19T13:02:56.313383+0000","last_peered":"2023-10-19T13:02:56.313383+0000","last_clean":"2023-10-19T13:02:56.313383+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:02:56.313383+0000","last_undegraded":"2023-10-19T13:02:56.313383+0000","last_fullsized":"2023-10-19T13:02:56.313383+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_clean_scrub_stamp":"2023-10-19T13:02:56.313383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:02:57.320575+0000","last_change":"2023-10-19T13:02:57.320575+0000","last_active":"2023-10-19T13:02:57.320575+0000","last_peered":"2023-10-19T13:02:57.320575+0000","last_clean":"2023-10-19T13:02:57.320575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:02:57.320575+0000","last_undegraded":"2023-10-19T13:02:57.320575+0000","last_fullsized":"2023-10-19T13:02:57.320575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.408527+0000","last_change":"2023-10-19T13:02:57.339476+0000","last_active":"2023-10-19T13:02:59.408527+0000","last_peered":"2023-10-19T13:02:59.408527+0000","last_clean":"2023-10-19T13:02:59.408527+0000","last_became_active":"2023-10-19T13:02:57.339036+0000","last_became_peered":"2023-10-19T13:02:57.339036+0000","last_unstale":"2023-10-19T13:02:59.408527+0000","last_undegraded":"2023-10-19T13:02:59.408527+0000","last_fullsized":"2023-10-19T13:02:59.408527+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_clean_scrub_stamp":"2023-10-19T13:02:56.313383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:43:51.920104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:02:57.320575+0000","last_change":"2023-10-19T13:02:57.320575+0000","last_active":"2023-10-19T13:02:57.320575+0000","last_peered":"2023-10-19T13:02:57.320575+0000","last_clean":"2023-10-19T13:02:57.320575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:02:57.320575+0000","last_undegraded":"2023-10-19T13:02:57.320575+0000","last_fullsized":"2023-10-19T13:02:57.320575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:02:56.313383+0000","last_change":"2023-10-19T13:02:56.313383+0000","last_active":"2023-10-19T13:02:56.313383+0000","last_peered":"2023-10-19T13:02:56.313383+0000","last_clean":"2023-10-19T13:02:56.313383+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:02:56.313383+0000","last_undegraded":"2023-10-19T13:02:56.313383+0000","last_fullsized":"2023-10-19T13:02:56.313383+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_clean_scrub_stamp":"2023-10-19T13:02:56.313383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:02:57.320575+0000","last_change":"2023-10-19T13:02:57.320575+0000","last_active":"2023-10-19T13:02:57.320575+0000","last_peered":"2023-10-19T13:02:57.320575+0000","last_clean":"2023-10-19T13:02:57.320575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:02:57.320575+0000","last_undegraded":"2023-10-19T13:02:57.320575+0000","last_fullsized":"2023-10-19T13:02:57.320575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T13:02:56.313383+0000","last_change":"2023-10-19T13:02:56.313383+0000","last_active":"2023-10-19T13:02:56.313383+0000","last_peered":"2023-10-19T13:02:56.313383+0000","last_clean":"2023-10-19T13:02:56.313383+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T13:02:56.313383+0000","last_undegraded":"2023-10-19T13:02:56.313383+0000","last_fullsized":"2023-10-19T13:02:56.313383+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_clean_scrub_stamp":"2023-10-19T13:02:56.313383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"118'3043","reported_seq":3456,"reported_epoch":124,"state":"active+clean","last_fresh":"2023-10-19T13:02:53.341007+0000","last_change":"2023-10-19T12:44:08.636613+0000","last_active":"2023-10-19T13:02:53.341007+0000","last_peered":"2023-10-19T13:02:53.341007+0000","last_clean":"2023-10-19T13:02:53.341007+0000","last_became_active":"2023-10-19T12:44:08.635697+0000","last_became_peered":"2023-10-19T12:44:08.635697+0000","last_unstale":"2023-10-19T13:02:53.341007+0000","last_undegraded":"2023-10-19T13:02:53.341007+0000","last_fullsized":"2023-10-19T13:02:53.341007+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_clean_scrub_stamp":"2023-10-19T12:44:07.399738+0000","objects_scrubbed":0,"log_size":3043,"log_dups_size":0,"ondisk_log_size":3043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:15:23.685485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":962,"num_read_kb":1946,"num_write":329,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":16,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":70,"acting":70,"num_store_stats":4},{"poolid":15,"num_pg":32,"stat_sum":{"num_bytes":2250,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":4534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":28,"acting":28,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":962,"num_read_kb":1946,"num_write":329,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3043,"ondisk_log_size":3043,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575076,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55484,"kb_used_data":2092,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316356,"statfs":{"total":96636764160,"available":96579948544,"internally_reserved":0,"allocated":2142208,"data_stored":1369881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54655435},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 13:02:49 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.53600000000000003,"15min":0.56299999999999994},"min":{"1min":0.44700000000000001,"5min":0.224,"15min":0.224},"max":{"1min":0.75,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.626,"5min":0.56699999999999995,"15min":0.58699999999999997},"min":{"1min":0.371,"5min":0.33200000000000002,"15min":0.29599999999999999},"max":{"1min":0.751,"5min":0.78200000000000003,"15min":0.93700000000000006},"last":0.56899999999999995}]},{"osd":1,"last update":"Thu Oct 19 13:02:49 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.55000000000000004,"15min":0.57399999999999995},"min":{"1min":0.432,"5min":0.33500000000000002,"15min":0.28199999999999997},"max":{"1min":0.69899999999999995,"5min":0.86299999999999999,"15min":1.3129999999999999},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.56899999999999995,"15min":0.58699999999999997},"min":{"1min":0.45100000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.755,"5min":0.85499999999999998,"15min":1.284},"last":0.67700000000000005}]},{"osd":2,"last update":"Thu Oct 19 13:02:49 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.60599999999999998,"15min":0.625},"min":{"1min":0.54100000000000004,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.76700000000000002,"5min":0.89200000000000002,"15min":0.98599999999999999},"last":0.874},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.60499999999999998,"15min":0.63200000000000001},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.28399999999999997},"max":{"1min":0.751,"5min":0.83699999999999997,"15min":1.2589999999999999},"last":0.74399999999999999}]},{"osd":3,"last update":"Thu Oct 19 13:02:49 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.57899999999999996,"15min":0.60199999999999998},"min":{"1min":0.497,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.73399999999999999,"5min":0.87,"15min":0.96899999999999997},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.64700000000000002,"15min":0.65300000000000002},"min":{"1min":0.56699999999999995,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.80600000000000005,"5min":0.92000000000000004,"15min":1.329},"last":0.83799999999999997}]},{"osd":4,"last update":"Thu Oct 19 13:02:49 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.46700000000000003},"min":{"1min":0.318,"5min":0.247,"15min":0.247},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.47799999999999998,"15min":0.46800000000000003},"min":{"1min":0.34200000000000003,"5min":0.216,"15min":0.216},"max":{"1min":0.73399999999999999,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.58099999999999996}]},{"osd":5,"last update":"Thu Oct 19 13:02:49 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.49099999999999999,"15min":0.48199999999999998},"min":{"1min":0.34999999999999998,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.80600000000000005},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.52500000000000002,"15min":0.51700000000000002},"min":{"1min":0.40300000000000002,"5min":0.314,"15min":0.314},"max":{"1min":0.72999999999999998,"5min":0.78900000000000003,"15min":0.81999999999999995},"last":0.80100000000000005}]},{"osd":6,"last update":"Thu Oct 19 13:02:49 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.55000000000000004,"15min":0.54600000000000004},"min":{"1min":0.42999999999999999,"5min":0.35299999999999998,"15min":0.308},"max":{"1min":0.78900000000000003,"5min":0.81299999999999994,"15min":0.88900000000000001},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.52500000000000002,"15min":0.52800000000000002},"min":{"1min":0.38600000000000001,"5min":0.318,"15min":0.27300000000000002},"max":{"1min":0.77000000000000002,"5min":0.83999999999999997,"15min":0.90300000000000002},"last":0.69499999999999995}]}]},{"osd":6,"up_from":13,"seq":55834575077,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32776,"kb_used_data":1592,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339064,"statfs":{"total":96636764160,"available":96603201536,"internally_reserved":0,"allocated":1630208,"data_stored":814393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 13:02:33 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.54600000000000004,"15min":0.58199999999999996},"min":{"1min":0.32400000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.76400000000000001,"5min":1.196,"15min":1.196},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.54300000000000004,"15min":0.57999999999999996},"min":{"1min":0.35299999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.71699999999999997,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.84699999999999998}]},{"osd":1,"last update":"Thu Oct 19 13:02:33 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.56200000000000006,"15min":0.59399999999999997},"min":{"1min":0.5,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.754,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.627,"5min":0.56699999999999995,"15min":0.59699999999999998},"min":{"1min":0.42999999999999999,"5min":0.307,"15min":0.307},"max":{"1min":0.748,"5min":0.78200000000000003,"15min":0.82099999999999995},"last":0.63900000000000001}]},{"osd":2,"last update":"Thu Oct 19 13:02:33 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.54400000000000004,"15min":0.57899999999999996},"min":{"1min":0.45200000000000001,"5min":0.307,"15min":0.307},"max":{"1min":0.76100000000000001,"5min":0.84699999999999998,"15min":0.85699999999999998},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.58699999999999997,"15min":0.60799999999999998},"min":{"1min":0.50800000000000001,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.71899999999999997,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.74199999999999999}]},{"osd":3,"last update":"Thu Oct 19 13:02:33 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.58699999999999997,"15min":0.624},"min":{"1min":0.47599999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.81000000000000005,"5min":0.871,"15min":0.871},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.59099999999999997,"15min":0.62},"min":{"1min":0.53800000000000003,"5min":0.375,"15min":0.36799999999999999},"max":{"1min":0.81699999999999995,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.82799999999999996}]},{"osd":4,"last update":"Thu Oct 19 13:02:33 2023","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.47499999999999998,"15min":0.46600000000000003},"min":{"1min":0.32600000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.72599999999999998,"5min":1.353,"15min":1.353},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.47599999999999998,"15min":0.47799999999999998},"min":{"1min":0.35199999999999998,"5min":0.254,"15min":0.254},"max":{"1min":0.73599999999999999,"5min":0.73799999999999999,"15min":0.77600000000000002},"last":0.81399999999999995}]},{"osd":5,"last update":"Thu Oct 19 13:02:33 2023","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.45600000000000002,"15min":0.46999999999999997},"min":{"1min":0.36899999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.58099999999999996,"5min":0.69099999999999995,"15min":0.77300000000000002},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52400000000000002,"15min":0.51400000000000001},"min":{"1min":0.373,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.75,"5min":0.75700000000000001,"15min":0.85699999999999998},"last":0.70899999999999996}]},{"osd":7,"last update":"Thu Oct 19 13:02:33 2023","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.52500000000000002,"15min":0.51900000000000002},"min":{"1min":0.39500000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.71999999999999997,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.39300000000000002,"5min":0.27900000000000003,"15min":0.26000000000000001},"max":{"1min":0.67700000000000005,"5min":1.127,"15min":1.127},"last":0.754}]}]},{"osd":5,"up_from":13,"seq":55834575076,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":50752,"kb_used_data":1512,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321088,"statfs":{"total":96636764160,"available":96584794112,"internally_reserved":0,"allocated":1548288,"data_stored":779513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 13:02:32 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.55900000000000005,"15min":0.59099999999999997},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.75800000000000001,"5min":0.79900000000000004,"15min":0.82699999999999996},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.57399999999999995,"15min":0.59499999999999997},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.27800000000000002},"max":{"1min":0.81499999999999995,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.77200000000000002}]},{"osd":1,"last update":"Thu Oct 19 13:02:32 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.57099999999999995,"15min":0.59599999999999997},"min":{"1min":0.41799999999999998,"5min":0.34599999999999997,"15min":0.32800000000000001},"max":{"1min":0.80300000000000005,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.59999999999999998,"15min":0.61199999999999999},"min":{"1min":0.39900000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.71499999999999997}]},{"osd":2,"last update":"Thu Oct 19 13:02:32 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.54900000000000004,"15min":0.58299999999999996},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.307},"max":{"1min":0.76900000000000002,"5min":0.77500000000000002,"15min":0.80700000000000005},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.60199999999999998,"15min":0.62},"min":{"1min":0.37,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.83999999999999997,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.74299999999999999}]},{"osd":3,"last update":"Thu Oct 19 13:02:32 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.57899999999999996,"15min":0.60399999999999998},"min":{"1min":0.38900000000000001,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.60399999999999998,"15min":0.63200000000000001},"min":{"1min":0.40899999999999997,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.73099999999999998}]},{"osd":4,"last update":"Thu Oct 19 13:02:32 2023","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.47199999999999998,"15min":0.47099999999999997},"min":{"1min":0.35299999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.67400000000000004,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.47199999999999998,"15min":0.47199999999999998},"min":{"1min":0.26100000000000001,"5min":0.251,"15min":0.251},"max":{"1min":0.69699999999999995,"5min":0.73299999999999998,"15min":0.79400000000000004},"last":0.69499999999999995}]},{"osd":6,"last update":"Thu Oct 19 13:02:32 2023","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.30199999999999999,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.50800000000000001,"15min":0.503},"min":{"1min":0.29599999999999999,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.75700000000000001,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.53100000000000003}]},{"osd":7,"last update":"Thu Oct 19 13:02:32 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.503,"15min":0.49199999999999999},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.78600000000000003,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.55300000000000005,"15min":0.53700000000000003},"min":{"1min":0.375,"5min":0.311,"15min":0.311},"max":{"1min":0.76800000000000002,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.60399999999999998}]}]},{"osd":4,"up_from":13,"seq":55834575076,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32696,"kb_used_data":1512,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339144,"statfs":{"total":96636764160,"available":96603283456,"internally_reserved":0,"allocated":1548288,"data_stored":779513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 13:02:45 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.52800000000000002,"15min":0.55300000000000005},"min":{"1min":0.40799999999999997,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.76100000000000001,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.53100000000000003,"15min":0.55500000000000005},"min":{"1min":0.44800000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.58599999999999997}]},{"osd":1,"last update":"Thu Oct 19 13:02:45 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.55700000000000005,"15min":0.60199999999999998},"min":{"1min":0.46999999999999997,"5min":0.317,"15min":0.317},"max":{"1min":0.80600000000000005,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.624,"5min":0.58299999999999996,"15min":0.61299999999999999},"min":{"1min":0.44700000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.79800000000000004,"5min":0.79900000000000004,"15min":0.86899999999999999},"last":0.44600000000000001}]},{"osd":2,"last update":"Thu Oct 19 13:02:45 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.57299999999999995,"15min":0.60699999999999998},"min":{"1min":0.39900000000000002,"5min":0.28399999999999997,"15min":0.254},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.86599999999999999},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.58299999999999996,"15min":0.61099999999999999},"min":{"1min":0.33300000000000002,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.82099999999999995,"5min":0.83799999999999997,"15min":0.85999999999999999},"last":0.54600000000000004}]},{"osd":3,"last update":"Thu Oct 19 13:02:45 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.55500000000000005,"15min":0.57799999999999996},"min":{"1min":0.36499999999999999,"5min":0.33200000000000002,"15min":0.32400000000000001},"max":{"1min":0.72599999999999998,"5min":0.75,"15min":0.79700000000000004},"last":0.503},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.60599999999999998,"15min":0.63700000000000001},"min":{"1min":0.44500000000000001,"5min":0.42099999999999999,"15min":0.41299999999999998},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.83899999999999997},"last":0.56699999999999995}]},{"osd":5,"last update":"Thu Oct 19 13:02:45 2023","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.44500000000000001,"15min":0.44400000000000001},"min":{"1min":0.30399999999999999,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.64200000000000002,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.46500000000000002,"15min":0.46100000000000002},"min":{"1min":0.34699999999999998,"5min":0.30099999999999999,"15min":0.23499999999999999},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.84999999999999998},"last":0.64200000000000002}]},{"osd":6,"last update":"Thu Oct 19 13:02:45 2023","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.45200000000000001,"15min":0.45500000000000002},"min":{"1min":0.32600000000000001,"5min":0.28499999999999998,"15min":0.26400000000000001},"max":{"1min":0.70299999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.48899999999999999,"15min":0.49199999999999999},"min":{"1min":0.35999999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.73999999999999999,"5min":0.78000000000000003,"15min":0.82499999999999996},"last":0.60299999999999998}]},{"osd":7,"last update":"Thu Oct 19 13:02:45 2023","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.5,"15min":0.5},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.27500000000000002},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.80200000000000005},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.41999999999999998,"5min":0.33100000000000002,"15min":0.30199999999999999},"max":{"1min":0.754,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.625}]}]},{"osd":3,"up_from":13,"seq":55834575076,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41072,"kb_used_data":1564,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330768,"statfs":{"total":96636764160,"available":96594706432,"internally_reserved":0,"allocated":1601536,"data_stored":806111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":40434120},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 13:02:29 2023","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.40799999999999997,"15min":0.40200000000000002},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.59199999999999997,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.39700000000000002,"15min":0.41499999999999998},"min":{"1min":0.23300000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.50700000000000001,"5min":0.75,"15min":0.75},"last":0.27600000000000002}]},{"osd":1,"last update":"Thu Oct 19 13:02:29 2023","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.40400000000000003,"15min":0.40799999999999997},"min":{"1min":0.27200000000000002,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.61699999999999999,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.46300000000000002,"15min":0.45800000000000002},"min":{"1min":0.32600000000000001,"5min":0.221,"15min":0.221},"max":{"1min":0.53600000000000003,"5min":2.9399999999999999,"15min":2.9399999999999999},"last":0.313}]},{"osd":2,"last update":"Thu Oct 19 13:02:29 2023","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.45300000000000001,"15min":0.46400000000000002},"min":{"1min":0.309,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.56499999999999995,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.47299999999999998,"15min":0.48699999999999999},"min":{"1min":0.36199999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.64400000000000002,"5min":0.754,"15min":0.754},"last":0.45600000000000002}]},{"osd":4,"last update":"Thu Oct 19 13:02:29 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.64000000000000001,"15min":0.66100000000000003},"min":{"1min":0.38100000000000001,"5min":0.371,"15min":0.371},"max":{"1min":0.88800000000000001,"5min":0.92500000000000004,"15min":1.0069999999999999},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.67000000000000004,"15min":0.68799999999999994},"min":{"1min":0.44900000000000001,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.99399999999999999},"last":0.65600000000000003}]},{"osd":5,"last update":"Thu Oct 19 13:02:29 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.67800000000000005,"15min":0.66000000000000003},"min":{"1min":0.48899999999999999,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.96699999999999997,"5min":2.9990000000000001,"15min":2.9990000000000001},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63300000000000001,"15min":0.64800000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.35899999999999999},"max":{"1min":0.83699999999999997,"5min":1.774,"15min":1.774},"last":0.47099999999999997}]},{"osd":6,"last update":"Thu Oct 19 13:02:29 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.64200000000000002,"15min":0.65100000000000002},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.92400000000000004,"5min":0.93700000000000006,"15min":0.96799999999999997},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.48499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.90400000000000003,"5min":1.167,"15min":1.167},"last":0.48899999999999999}]},{"osd":7,"last update":"Thu Oct 19 13:02:29 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64300000000000002,"15min":0.66100000000000003},"min":{"1min":0.42899999999999999,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.94199999999999995,"5min":1.042,"15min":1.042},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.68500000000000005,"15min":0.67900000000000005},"min":{"1min":0.44800000000000001,"5min":0.441,"15min":0.432},"max":{"1min":0.86099999999999999,"5min":2.8149999999999999,"15min":2.8149999999999999},"last":0.56599999999999995}]}]},{"osd":2,"up_from":13,"seq":55834575077,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":54924,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316916,"statfs":{"total":96636764160,"available":96580521984,"internally_reserved":0,"allocated":1560576,"data_stored":785618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 13:02:44 2023","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.36499999999999999,"15min":0.379},"min":{"1min":0.27700000000000002,"5min":0.20699999999999999,"15min":0.19700000000000001},"max":{"1min":0.496,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.316},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.39700000000000002,"15min":0.41199999999999998},"min":{"1min":0.25700000000000001,"5min":0.222,"15min":0.20999999999999999},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.74099999999999999},"last":0.38700000000000001}]},{"osd":1,"last update":"Thu Oct 19 13:02:44 2023","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.373,"15min":0.39700000000000002},"min":{"1min":0.27800000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.71099999999999997},"last":0.249},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.40400000000000003,"15min":0.42799999999999999},"min":{"1min":0.28000000000000003,"5min":0.22,"15min":0.22},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.68100000000000005},"last":0.36899999999999999}]},{"osd":3,"last update":"Thu Oct 19 13:02:44 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.44,"15min":0.45500000000000002},"min":{"1min":0.33400000000000002,"5min":0.29199999999999998,"15min":0.22800000000000001},"max":{"1min":0.69299999999999995,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.46200000000000002,"15min":0.48299999999999998},"min":{"1min":0.40400000000000003,"5min":0.32200000000000001,"15min":0.29099999999999998},"max":{"1min":0.72599999999999998,"5min":0.77800000000000002,"15min":0.82999999999999996},"last":0.34899999999999998}]},{"osd":4,"last update":"Thu Oct 19 13:02:44 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.60699999999999998,"15min":0.64000000000000001},"min":{"1min":0.501,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.81599999999999995,"5min":0.89500000000000002,"15min":0.98999999999999999},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.61099999999999999,"15min":0.65200000000000002},"min":{"1min":0.48899999999999999,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.78400000000000003,"5min":0.83399999999999996,"15min":1.046},"last":0.75900000000000001}]},{"osd":5,"last update":"Thu Oct 19 13:02:44 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.621,"15min":0.65700000000000003},"min":{"1min":0.46800000000000003,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.99399999999999999},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.61799999999999999,"15min":0.65400000000000003},"min":{"1min":0.51300000000000001,"5min":0.375,"15min":0.375},"max":{"1min":0.86299999999999999,"5min":0.94199999999999995,"15min":0.95799999999999996},"last":0.747}]},{"osd":6,"last update":"Thu Oct 19 13:02:44 2023","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.627,"15min":0.65900000000000003},"min":{"1min":0.45200000000000001,"5min":0.40400000000000003,"15min":0.40300000000000002},"max":{"1min":0.83699999999999997,"5min":0.89800000000000002,"15min":1.008},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.64000000000000001,"15min":0.67500000000000004},"min":{"1min":0.54000000000000004,"5min":0.38800000000000001,"15min":0.371},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.995},"last":0.73299999999999998}]},{"osd":7,"last update":"Thu Oct 19 13:02:44 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.64300000000000002,"15min":0.67700000000000005},"min":{"1min":0.49399999999999999,"5min":0.39800000000000002,"15min":0.36399999999999999},"max":{"1min":0.84699999999999998,"5min":0.96299999999999997,"15min":1.0149999999999999},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.65000000000000002,"15min":0.68000000000000005},"min":{"1min":0.438,"5min":0.438,"15min":0.38200000000000001},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":1.0329999999999999},"last":0.69799999999999995}]}]},{"osd":1,"up_from":13,"seq":55834575077,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36904,"kb_used_data":1564,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334936,"statfs":{"total":96636764160,"available":96598974464,"internally_reserved":0,"allocated":1601536,"data_stored":805563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 13:02:41 2023","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.376,"15min":0.38700000000000001},"min":{"1min":0.29399999999999998,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.56599999999999995,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.308},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.432,"15min":0.437},"min":{"1min":0.32000000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.76100000000000001,"5min":2.5800000000000001,"15min":2.5800000000000001},"last":0.23999999999999999}]},{"osd":2,"last update":"Thu Oct 19 13:02:41 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.40500000000000003,"15min":0.42599999999999999},"min":{"1min":0.315,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.64300000000000002,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.46800000000000003,"15min":0.46200000000000002},"min":{"1min":0.34599999999999997,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.79800000000000004,"5min":2.6230000000000002,"15min":2.6230000000000002},"last":0.29299999999999998}]},{"osd":3,"last update":"Thu Oct 19 13:02:41 2023","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.498,"15min":0.49199999999999999},"min":{"1min":0.375,"5min":0.246,"15min":0.246},"max":{"1min":0.71999999999999997,"5min":3.9740000000000002,"15min":3.9740000000000002},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.42099999999999999,"15min":0.42799999999999999},"min":{"1min":0.33100000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.624,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.32700000000000001}]},{"osd":4,"last update":"Thu Oct 19 13:02:41 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.61899999999999999,"15min":0.65000000000000002},"min":{"1min":0.46700000000000003,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.80200000000000005,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.61799999999999999,"15min":0.64000000000000001},"min":{"1min":0.434,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.81999999999999995,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.53500000000000003}]},{"osd":5,"last update":"Thu Oct 19 13:02:41 2023","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.63700000000000001,"15min":0.65200000000000002},"min":{"1min":0.50900000000000001,"5min":0.38700000000000001,"15min":0.38100000000000001},"max":{"1min":0.86299999999999999,"5min":1.1200000000000001,"15min":1.1200000000000001},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.66300000000000003,"15min":0.67400000000000004},"min":{"1min":0.50900000000000001,"5min":0.374,"15min":0.374},"max":{"1min":0.873,"5min":2.7120000000000002,"15min":2.7120000000000002},"last":0.66700000000000004}]},{"osd":6,"last update":"Thu Oct 19 13:02:41 2023","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.623,"15min":0.65300000000000002},"min":{"1min":0.43099999999999999,"5min":0.378,"15min":0.36799999999999999},"max":{"1min":0.83899999999999997,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.64600000000000002,"15min":0.67300000000000004},"min":{"1min":0.46700000000000003,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.84599999999999997,"5min":1.129,"15min":1.129},"last":0.68400000000000005}]},{"osd":7,"last update":"Thu Oct 19 13:02:41 2023","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.67100000000000004,"15min":0.68000000000000005},"min":{"1min":0.56799999999999995,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.91200000000000003,"5min":2.786,"15min":2.786},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.64100000000000001,"15min":0.67300000000000004},"min":{"1min":0.50900000000000001,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.88300000000000001,"5min":0.95399999999999996,"15min":1.034},"last":0.64800000000000002}]}]},{"osd":0,"up_from":13,"seq":55834575078,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33336,"kb_used_data":2152,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338504,"statfs":{"total":96636764160,"available":96602628096,"internally_reserved":0,"allocated":2203648,"data_stored":1396980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Oct 19 13:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.38200000000000001,"15min":0.39000000000000001},"min":{"1min":0.218,"5min":0.182,"15min":0.182},"max":{"1min":0.624,"5min":0.70199999999999996,"15min":0.81799999999999995},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.40500000000000003,"15min":0.42999999999999999},"min":{"1min":0.23300000000000001,"5min":0.22500000000000001,"15min":0.20200000000000001},"max":{"1min":0.64800000000000002,"5min":0.67600000000000005,"15min":0.746},"last":0.439}]},{"osd":2,"last update":"Thu Oct 19 13:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.42699999999999999,"15min":0.45200000000000001},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.26200000000000001},"max":{"1min":0.629,"5min":0.67900000000000005,"15min":0.76300000000000001},"last":0.498},{"interface":"front","average":{"1min":0.437,"5min":0.41499999999999998,"15min":0.434},"min":{"1min":0.254,"5min":0.246,"15min":0.20699999999999999},"max":{"1min":0.68500000000000005,"5min":0.72999999999999998,"15min":0.76800000000000002},"last":0.51000000000000001}]},{"osd":3,"last update":"Thu Oct 19 13:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.39700000000000002,"15min":0.40899999999999997},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.60399999999999998,"5min":0.65600000000000003,"15min":1.2829999999999999},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.45300000000000001,"15min":0.48899999999999999},"min":{"1min":0.29999999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":1.4990000000000001},"last":0.52400000000000002}]},{"osd":4,"last update":"Thu Oct 19 13:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.59899999999999998,"15min":0.63},"min":{"1min":0.47999999999999998,"5min":0.40000000000000002,"15min":0.34999999999999998},"max":{"1min":0.79400000000000004,"5min":0.98499999999999999,"15min":1.361},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.64200000000000002,"15min":0.65200000000000002},"min":{"1min":0.48199999999999998,"5min":0.36399999999999999,"15min":0.35999999999999999},"max":{"1min":0.89600000000000002,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.69999999999999996}]},{"osd":5,"last update":"Thu Oct 19 13:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.66900000000000004,"15min":0.68400000000000005},"min":{"1min":0.54800000000000004,"5min":0.378,"15min":0.378},"max":{"1min":0.879,"5min":1.004,"15min":1.325},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.66700000000000004,"15min":0.68400000000000005},"min":{"1min":0.53000000000000003,"5min":0.40100000000000002,"15min":0.318},"max":{"1min":0.90900000000000003,"5min":1.024,"15min":1.024},"last":0.65100000000000002}]},{"osd":6,"last update":"Thu Oct 19 13:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.63600000000000001,"15min":0.66100000000000003},"min":{"1min":0.51000000000000001,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.82099999999999995,"5min":0.88800000000000001,"15min":0.98099999999999998},"last":0.746},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.64100000000000001,"15min":0.66100000000000003},"min":{"1min":0.52400000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.88700000000000001,"5min":0.96599999999999997,"15min":0.995},"last":0.67500000000000004}]},{"osd":7,"last update":"Thu Oct 19 13:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.63300000000000001,"15min":0.67600000000000005},"min":{"1min":0.53400000000000003,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.83899999999999997,"5min":0.98799999999999999,"15min":1.0089999999999999},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.65000000000000002,"15min":0.66200000000000003},"min":{"1min":0.57399999999999995,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.91000000000000003,"5min":0.94499999999999995,"15min":1.389},"last":0.76100000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-19T13:03:01.349 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-19T13:03:01.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.490+0000 7f0989c61640 1 -- 172.21.15.57:0/3615604253 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f096c003970 msgr2=0x7f096c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:03:01.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.490+0000 7f0989c61640 1 --2- 172.21.15.57:0/3615604253 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f096c003970 0x7f096c003d90 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7f09840cb570 tx=0x7f096c0057f0 comp rx=0 tx=0).stop 2023-10-19T13:03:01.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.490+0000 7f0983fff640 1 -- 172.21.15.57:0/3615604253 reap_dead start 2023-10-19T13:03:01.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.490+0000 7f0989c61640 1 -- 172.21.15.57:0/3615604253 shutdown_connections 2023-10-19T13:03:01.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.490+0000 7f0989c61640 1 -- 172.21.15.57:0/3615604253 >> 172.21.15.57:0/3615604253 conn(0x7f09840af7a0 msgr2=0x7f09840afb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:03:01.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.490+0000 7f0989c61640 1 -- 172.21.15.57:0/3615604253 shutdown_connections 2023-10-19T13:03:01.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.490+0000 7f0989c61640 1 -- 172.21.15.57:0/3615604253 wait complete. 2023-10-19T13:03:01.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.491+0000 7f0989c61640 1 Processor -- start 2023-10-19T13:03:01.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.491+0000 7f0989c61640 1 -- start start 2023-10-19T13:03:01.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.492+0000 7f0989c61640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f096c003970 0x7f098413e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:03:01.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.492+0000 7f0989c61640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0984156df0 0x7f09841eec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:03:01.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.492+0000 7f0989c61640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0984155e70 0x7f09841562d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:03:01.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.492+0000 7f0989c61640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f098413f820 con 0x7f096c003970 2023-10-19T13:03:01.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.492+0000 7f0989c61640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0984156810 con 0x7f0984155e70 2023-10-19T13:03:01.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.492+0000 7f0989c61640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0984144070 con 0x7f0984156df0 2023-10-19T13:03:01.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.492+0000 7f09837fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f096c003970 0x7f098413e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:03:01.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.492+0000 7f09837fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f096c003970 0x7f098413e510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:48632/0 (socket says 172.21.15.57:48632) 2023-10-19T13:03:01.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.492+0000 7f09837fe640 1 -- 172.21.15.57:0/2017180622 learned_addr learned my addr 172.21.15.57:0/2017180622 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:03:01.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.492+0000 7f0982ffd640 1 --2- 172.21.15.57:0/2017180622 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0984156df0 0x7f09841eec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:03:01.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.492+0000 7f0983fff640 1 --2- 172.21.15.57:0/2017180622 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0984155e70 0x7f09841562d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:03:01.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.492+0000 7f09837fe640 1 -- 172.21.15.57:0/2017180622 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0984156df0 msgr2=0x7f09841eec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:03:01.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.492+0000 7f09837fe640 1 --2- 172.21.15.57:0/2017180622 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0984156df0 0x7f09841eec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:01.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.492+0000 7f09837fe640 1 -- 172.21.15.57:0/2017180622 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0984155e70 msgr2=0x7f09841562d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:03:01.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.492+0000 7f09837fe640 1 --2- 172.21.15.57:0/2017180622 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0984155e70 0x7f09841562d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:01.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.492+0000 7f09837fe640 1 -- 172.21.15.57:0/2017180622 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f098413ec10 con 0x7f096c003970 2023-10-19T13:03:01.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.492+0000 7f09837fe640 1 --2- 172.21.15.57:0/2017180622 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f096c003970 0x7f098413e510 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7f097400bac0 tx=0x7f09740366b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:03:01.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.492+0000 7f096bfff640 1 -- 172.21.15.57:0/2017180622 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0974036cb0 con 0x7f096c003970 2023-10-19T13:03:01.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.493+0000 7f096bfff640 1 -- 172.21.15.57:0/2017180622 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f0974005d30 con 0x7f096c003970 2023-10-19T13:03:01.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.493+0000 7f0989c61640 1 -- 172.21.15.57:0/2017180622 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f098413eef0 con 0x7f096c003970 2023-10-19T13:03:01.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.493+0000 7f096bfff640 1 -- 172.21.15.57:0/2017180622 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0974005220 con 0x7f096c003970 2023-10-19T13:03:01.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.493+0000 7f0989c61640 1 -- 172.21.15.57:0/2017180622 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f098413f4b0 con 0x7f096c003970 2023-10-19T13:03:01.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.494+0000 7f0989c61640 1 -- 172.21.15.57:0/2017180622 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f098413f820 con 0x7f096c003970 2023-10-19T13:03:01.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.494+0000 7f096bfff640 1 -- 172.21.15.57:0/2017180622 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 58) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f0974005380 con 0x7f096c003970 2023-10-19T13:03:01.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.494+0000 7f096bfff640 1 --2- 172.21.15.57:0/2017180622 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f097006af60 0x7f097006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:03:01.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.495+0000 7f096bfff640 1 -- 172.21.15.57:0/2017180622 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f09740abc10 con 0x7f096c003970 2023-10-19T13:03:01.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.495+0000 7f0982ffd640 1 --2- 172.21.15.57:0/2017180622 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f097006af60 0x7f097006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:03:01.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.495+0000 7f0982ffd640 1 --2- 172.21.15.57:0/2017180622 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f097006af60 0x7f097006d420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0984140c50 tx=0x7f09780068d0 comp rx=0 tx=0).ready entity=mgr.7540 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:03:01.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.499+0000 7f096bfff640 1 -- 172.21.15.57:0/2017180622 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0974076b00 con 0x7f096c003970 2023-10-19T13:03:01.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.672+0000 7f0989c61640 1 -- 172.21.15.57:0/2017180622 --> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f09840690e0 con 0x7f097006af60 2023-10-19T13:03:01.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.680+0000 7f096bfff640 1 -- 172.21.15.57:0/2017180622 <== mgr.7540 v2:172.21.15.179:6832/4155586759 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+290045 (secure 0 0 0) 0x7f09840690e0 con 0x7f097006af60 2023-10-19T13:03:01.685 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:03:01.687 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2023-10-19T13:03:01.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.683+0000 7f0989c61640 1 -- 172.21.15.57:0/2017180622 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f097006af60 msgr2=0x7f097006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:03:01.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.683+0000 7f0989c61640 1 --2- 172.21.15.57:0/2017180622 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f097006af60 0x7f097006d420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0984140c50 tx=0x7f09780068d0 comp rx=0 tx=0).stop 2023-10-19T13:03:01.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.683+0000 7f0989c61640 1 -- 172.21.15.57:0/2017180622 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f096c003970 msgr2=0x7f098413e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:03:01.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.683+0000 7f0989c61640 1 --2- 172.21.15.57:0/2017180622 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f096c003970 0x7f098413e510 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7f097400bac0 tx=0x7f09740366b0 comp rx=0 tx=0).stop 2023-10-19T13:03:01.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.683+0000 7f0989c61640 1 -- 172.21.15.57:0/2017180622 shutdown_connections 2023-10-19T13:03:01.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.683+0000 7f0989c61640 1 --2- 172.21.15.57:0/2017180622 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f097006af60 0x7f097006d420 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:01.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.683+0000 7f0989c61640 1 --2- 172.21.15.57:0/2017180622 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0984155e70 0x7f09841562d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:01.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.683+0000 7f0989c61640 1 --2- 172.21.15.57:0/2017180622 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0984156df0 0x7f09841eec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:01.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.683+0000 7f0989c61640 1 --2- 172.21.15.57:0/2017180622 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f096c003970 0x7f098413e510 unknown :-1 s=CLOSED pgs=1452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:01.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.683+0000 7f0989c61640 1 -- 172.21.15.57:0/2017180622 >> 172.21.15.57:0/2017180622 conn(0x7f09840af7a0 msgr2=0x7f09841593f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:03:01.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.683+0000 7f0989c61640 1 -- 172.21.15.57:0/2017180622 shutdown_connections 2023-10-19T13:03:01.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.683+0000 7f0989c61640 1 -- 172.21.15.57:0/2017180622 wait complete. 2023-10-19T13:03:01.696 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-10-19T13:03:01.644706+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":962,"num_read_kb":1946,"num_write":373,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3081,"ondisk_log_size":3081,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":84,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":338160,"kb_used_data":13748,"kb_used_omap":12,"kb_used_meta":324275,"kb_avail":754636560,"statfs":{"total":773094113280,"available":772747837440,"internally_reserved":0,"allocated":14077952,"data_stored":7655761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":332058169},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.319386"},"pg_stats":[{"pgid":"16.3f","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.408576+0000","last_change":"2023-10-19T13:02:58.218772+0000","last_active":"2023-10-19T13:02:59.408576+0000","last_peered":"2023-10-19T13:02:59.408576+0000","last_clean":"2023-10-19T13:02:59.408576+0000","last_became_active":"2023-10-19T13:02:58.218486+0000","last_became_peered":"2023-10-19T13:02:58.218486+0000","last_unstale":"2023-10-19T13:02:59.408576+0000","last_undegraded":"2023-10-19T13:02:59.408576+0000","last_fullsized":"2023-10-19T13:02:59.408576+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:16:09.032988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3e","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.409259+0000","last_change":"2023-10-19T13:02:58.225728+0000","last_active":"2023-10-19T13:02:59.409259+0000","last_peered":"2023-10-19T13:02:59.409259+0000","last_clean":"2023-10-19T13:02:59.409259+0000","last_became_active":"2023-10-19T13:02:58.225213+0000","last_became_peered":"2023-10-19T13:02:58.225213+0000","last_unstale":"2023-10-19T13:02:59.409259+0000","last_undegraded":"2023-10-19T13:02:59.409259+0000","last_fullsized":"2023-10-19T13:02:59.409259+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:03:30.317716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.3d","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.408296+0000","last_change":"2023-10-19T13:02:58.216453+0000","last_active":"2023-10-19T13:02:59.408296+0000","last_peered":"2023-10-19T13:02:59.408296+0000","last_clean":"2023-10-19T13:02:59.408296+0000","last_became_active":"2023-10-19T13:02:58.216192+0000","last_became_peered":"2023-10-19T13:02:58.216192+0000","last_unstale":"2023-10-19T13:02:59.408296+0000","last_undegraded":"2023-10-19T13:02:59.408296+0000","last_fullsized":"2023-10-19T13:02:59.408296+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:41:39.973763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3c","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.409059+0000","last_change":"2023-10-19T13:02:58.222548+0000","last_active":"2023-10-19T13:02:59.409059+0000","last_peered":"2023-10-19T13:02:59.409059+0000","last_clean":"2023-10-19T13:02:59.409059+0000","last_became_active":"2023-10-19T13:02:58.222097+0000","last_became_peered":"2023-10-19T13:02:58.222097+0000","last_unstale":"2023-10-19T13:02:59.409059+0000","last_undegraded":"2023-10-19T13:02:59.409059+0000","last_fullsized":"2023-10-19T13:02:59.409059+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:38:45.359597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.3b","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.210548+0000","last_change":"2023-10-19T13:02:58.223189+0000","last_active":"2023-10-19T13:02:59.210548+0000","last_peered":"2023-10-19T13:02:59.210548+0000","last_clean":"2023-10-19T13:02:59.210548+0000","last_became_active":"2023-10-19T13:02:58.222524+0000","last_became_peered":"2023-10-19T13:02:58.222524+0000","last_unstale":"2023-10-19T13:02:59.210548+0000","last_undegraded":"2023-10-19T13:02:59.210548+0000","last_fullsized":"2023-10-19T13:02:59.210548+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:11:08.241375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.3a","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.408536+0000","last_change":"2023-10-19T13:02:58.227127+0000","last_active":"2023-10-19T13:02:59.408536+0000","last_peered":"2023-10-19T13:02:59.408536+0000","last_clean":"2023-10-19T13:02:59.408536+0000","last_became_active":"2023-10-19T13:02:58.226839+0000","last_became_peered":"2023-10-19T13:02:58.226839+0000","last_unstale":"2023-10-19T13:02:59.408536+0000","last_undegraded":"2023-10-19T13:02:59.408536+0000","last_fullsized":"2023-10-19T13:02:59.408536+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:37:57.278166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.39","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.408396+0000","last_change":"2023-10-19T13:02:58.228791+0000","last_active":"2023-10-19T13:02:59.408396+0000","last_peered":"2023-10-19T13:02:59.408396+0000","last_clean":"2023-10-19T13:02:59.408396+0000","last_became_active":"2023-10-19T13:02:58.228570+0000","last_became_peered":"2023-10-19T13:02:58.228570+0000","last_unstale":"2023-10-19T13:02:59.408396+0000","last_undegraded":"2023-10-19T13:02:59.408396+0000","last_fullsized":"2023-10-19T13:02:59.408396+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:12:12.095113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.38","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.407925+0000","last_change":"2023-10-19T13:02:58.221216+0000","last_active":"2023-10-19T13:02:59.407925+0000","last_peered":"2023-10-19T13:02:59.407925+0000","last_clean":"2023-10-19T13:02:59.407925+0000","last_became_active":"2023-10-19T13:02:58.220946+0000","last_became_peered":"2023-10-19T13:02:58.220946+0000","last_unstale":"2023-10-19T13:02:59.407925+0000","last_undegraded":"2023-10-19T13:02:59.407925+0000","last_fullsized":"2023-10-19T13:02:59.407925+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:03:19.418162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.37","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.409390+0000","last_change":"2023-10-19T13:02:58.225273+0000","last_active":"2023-10-19T13:02:59.409390+0000","last_peered":"2023-10-19T13:02:59.409390+0000","last_clean":"2023-10-19T13:02:59.409390+0000","last_became_active":"2023-10-19T13:02:58.224972+0000","last_became_peered":"2023-10-19T13:02:58.224972+0000","last_unstale":"2023-10-19T13:02:59.409390+0000","last_undegraded":"2023-10-19T13:02:59.409390+0000","last_fullsized":"2023-10-19T13:02:59.409390+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:41:23.057082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.36","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.210783+0000","last_change":"2023-10-19T13:02:58.227439+0000","last_active":"2023-10-19T13:02:59.210783+0000","last_peered":"2023-10-19T13:02:59.210783+0000","last_clean":"2023-10-19T13:02:59.210783+0000","last_became_active":"2023-10-19T13:02:58.226745+0000","last_became_peered":"2023-10-19T13:02:58.226745+0000","last_unstale":"2023-10-19T13:02:59.210783+0000","last_undegraded":"2023-10-19T13:02:59.210783+0000","last_fullsized":"2023-10-19T13:02:59.210783+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:32:53.289930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.35","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.212789+0000","last_change":"2023-10-19T13:02:58.225564+0000","last_active":"2023-10-19T13:02:59.212789+0000","last_peered":"2023-10-19T13:02:59.212789+0000","last_clean":"2023-10-19T13:02:59.212789+0000","last_became_active":"2023-10-19T13:02:58.225245+0000","last_became_peered":"2023-10-19T13:02:58.225245+0000","last_unstale":"2023-10-19T13:02:59.212789+0000","last_undegraded":"2023-10-19T13:02:59.212789+0000","last_fullsized":"2023-10-19T13:02:59.212789+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:19:20.769178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.34","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.213968+0000","last_change":"2023-10-19T13:02:58.217750+0000","last_active":"2023-10-19T13:02:59.213968+0000","last_peered":"2023-10-19T13:02:59.213968+0000","last_clean":"2023-10-19T13:02:59.213968+0000","last_became_active":"2023-10-19T13:02:58.217470+0000","last_became_peered":"2023-10-19T13:02:58.217470+0000","last_unstale":"2023-10-19T13:02:59.213968+0000","last_undegraded":"2023-10-19T13:02:59.213968+0000","last_fullsized":"2023-10-19T13:02:59.213968+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:03:03.380116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.33","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.408415+0000","last_change":"2023-10-19T13:02:58.216355+0000","last_active":"2023-10-19T13:02:59.408415+0000","last_peered":"2023-10-19T13:02:59.408415+0000","last_clean":"2023-10-19T13:02:59.408415+0000","last_became_active":"2023-10-19T13:02:58.216086+0000","last_became_peered":"2023-10-19T13:02:58.216086+0000","last_unstale":"2023-10-19T13:02:59.408415+0000","last_undegraded":"2023-10-19T13:02:59.408415+0000","last_fullsized":"2023-10-19T13:02:59.408415+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:03:32.123711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.32","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.408597+0000","last_change":"2023-10-19T13:02:58.217487+0000","last_active":"2023-10-19T13:02:59.408597+0000","last_peered":"2023-10-19T13:02:59.408597+0000","last_clean":"2023-10-19T13:02:59.408597+0000","last_became_active":"2023-10-19T13:02:58.217236+0000","last_became_peered":"2023-10-19T13:02:58.217236+0000","last_unstale":"2023-10-19T13:02:59.408597+0000","last_undegraded":"2023-10-19T13:02:59.408597+0000","last_fullsized":"2023-10-19T13:02:59.408597+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:20:48.781729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.31","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.408599+0000","last_change":"2023-10-19T13:02:58.220950+0000","last_active":"2023-10-19T13:02:59.408599+0000","last_peered":"2023-10-19T13:02:59.408599+0000","last_clean":"2023-10-19T13:02:59.408599+0000","last_became_active":"2023-10-19T13:02:58.220665+0000","last_became_peered":"2023-10-19T13:02:58.220665+0000","last_unstale":"2023-10-19T13:02:59.408599+0000","last_undegraded":"2023-10-19T13:02:59.408599+0000","last_fullsized":"2023-10-19T13:02:59.408599+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:47:05.612852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.30","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.408465+0000","last_change":"2023-10-19T13:02:58.222450+0000","last_active":"2023-10-19T13:02:59.408465+0000","last_peered":"2023-10-19T13:02:59.408465+0000","last_clean":"2023-10-19T13:02:59.408465+0000","last_became_active":"2023-10-19T13:02:58.222170+0000","last_became_peered":"2023-10-19T13:02:58.222170+0000","last_unstale":"2023-10-19T13:02:59.408465+0000","last_undegraded":"2023-10-19T13:02:59.408465+0000","last_fullsized":"2023-10-19T13:02:59.408465+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:02:50.230615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.2f","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.408703+0000","last_change":"2023-10-19T13:02:58.213645+0000","last_active":"2023-10-19T13:02:59.408703+0000","last_peered":"2023-10-19T13:02:59.408703+0000","last_clean":"2023-10-19T13:02:59.408703+0000","last_became_active":"2023-10-19T13:02:58.213386+0000","last_became_peered":"2023-10-19T13:02:58.213386+0000","last_unstale":"2023-10-19T13:02:59.408703+0000","last_undegraded":"2023-10-19T13:02:59.408703+0000","last_fullsized":"2023-10-19T13:02:59.408703+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:50:56.716775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2e","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.409335+0000","last_change":"2023-10-19T13:02:58.228898+0000","last_active":"2023-10-19T13:02:59.409335+0000","last_peered":"2023-10-19T13:02:59.409335+0000","last_clean":"2023-10-19T13:02:59.409335+0000","last_became_active":"2023-10-19T13:02:58.228631+0000","last_became_peered":"2023-10-19T13:02:58.228631+0000","last_unstale":"2023-10-19T13:02:59.409335+0000","last_undegraded":"2023-10-19T13:02:59.409335+0000","last_fullsized":"2023-10-19T13:02:59.409335+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:32:53.289930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.2d","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.408419+0000","last_change":"2023-10-19T13:02:58.228433+0000","last_active":"2023-10-19T13:02:59.408419+0000","last_peered":"2023-10-19T13:02:59.408419+0000","last_clean":"2023-10-19T13:02:59.408419+0000","last_became_active":"2023-10-19T13:02:58.228160+0000","last_became_peered":"2023-10-19T13:02:58.228160+0000","last_unstale":"2023-10-19T13:02:59.408419+0000","last_undegraded":"2023-10-19T13:02:59.408419+0000","last_fullsized":"2023-10-19T13:02:59.408419+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:59:25.379540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2c","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.213988+0000","last_change":"2023-10-19T13:02:58.215712+0000","last_active":"2023-10-19T13:02:59.213988+0000","last_peered":"2023-10-19T13:02:59.213988+0000","last_clean":"2023-10-19T13:02:59.213988+0000","last_became_active":"2023-10-19T13:02:58.215382+0000","last_became_peered":"2023-10-19T13:02:58.215382+0000","last_unstale":"2023-10-19T13:02:59.213988+0000","last_undegraded":"2023-10-19T13:02:59.213988+0000","last_fullsized":"2023-10-19T13:02:59.213988+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:20:02.243435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2b","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.408552+0000","last_change":"2023-10-19T13:02:58.212208+0000","last_active":"2023-10-19T13:02:59.408552+0000","last_peered":"2023-10-19T13:02:59.408552+0000","last_clean":"2023-10-19T13:02:59.408552+0000","last_became_active":"2023-10-19T13:02:58.211906+0000","last_became_peered":"2023-10-19T13:02:58.211906+0000","last_unstale":"2023-10-19T13:02:59.408552+0000","last_undegraded":"2023-10-19T13:02:59.408552+0000","last_fullsized":"2023-10-19T13:02:59.408552+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:57:55.137521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2a","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.210515+0000","last_change":"2023-10-19T13:02:58.221055+0000","last_active":"2023-10-19T13:02:59.210515+0000","last_peered":"2023-10-19T13:02:59.210515+0000","last_clean":"2023-10-19T13:02:59.210515+0000","last_became_active":"2023-10-19T13:02:58.220762+0000","last_became_peered":"2023-10-19T13:02:58.220762+0000","last_unstale":"2023-10-19T13:02:59.210515+0000","last_undegraded":"2023-10-19T13:02:59.210515+0000","last_fullsized":"2023-10-19T13:02:59.210515+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:00:45.825647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.29","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.408993+0000","last_change":"2023-10-19T13:02:58.212624+0000","last_active":"2023-10-19T13:02:59.408993+0000","last_peered":"2023-10-19T13:02:59.408993+0000","last_clean":"2023-10-19T13:02:59.408993+0000","last_became_active":"2023-10-19T13:02:58.212350+0000","last_became_peered":"2023-10-19T13:02:58.212350+0000","last_unstale":"2023-10-19T13:02:59.408993+0000","last_undegraded":"2023-10-19T13:02:59.408993+0000","last_fullsized":"2023-10-19T13:02:59.408993+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:55:35.630464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.28","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.210365+0000","last_change":"2023-10-19T13:02:58.216883+0000","last_active":"2023-10-19T13:02:59.210365+0000","last_peered":"2023-10-19T13:02:59.210365+0000","last_clean":"2023-10-19T13:02:59.210365+0000","last_became_active":"2023-10-19T13:02:58.216598+0000","last_became_peered":"2023-10-19T13:02:58.216598+0000","last_unstale":"2023-10-19T13:02:59.210365+0000","last_undegraded":"2023-10-19T13:02:59.210365+0000","last_fullsized":"2023-10-19T13:02:59.210365+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:02:01.165779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.27","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.213842+0000","last_change":"2023-10-19T13:02:58.216291+0000","last_active":"2023-10-19T13:02:59.213842+0000","last_peered":"2023-10-19T13:02:59.213842+0000","last_clean":"2023-10-19T13:02:59.213842+0000","last_became_active":"2023-10-19T13:02:58.216012+0000","last_became_peered":"2023-10-19T13:02:58.216012+0000","last_unstale":"2023-10-19T13:02:59.213842+0000","last_undegraded":"2023-10-19T13:02:59.213842+0000","last_fullsized":"2023-10-19T13:02:59.213842+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:28:08.887992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.26","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.213852+0000","last_change":"2023-10-19T13:02:58.227110+0000","last_active":"2023-10-19T13:02:59.213852+0000","last_peered":"2023-10-19T13:02:59.213852+0000","last_clean":"2023-10-19T13:02:59.213852+0000","last_became_active":"2023-10-19T13:02:58.226832+0000","last_became_peered":"2023-10-19T13:02:58.226832+0000","last_unstale":"2023-10-19T13:02:59.213852+0000","last_undegraded":"2023-10-19T13:02:59.213852+0000","last_fullsized":"2023-10-19T13:02:59.213852+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:21:46.854887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.25","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.212980+0000","last_change":"2023-10-19T13:02:58.216250+0000","last_active":"2023-10-19T13:02:59.212980+0000","last_peered":"2023-10-19T13:02:59.212980+0000","last_clean":"2023-10-19T13:02:59.212980+0000","last_became_active":"2023-10-19T13:02:58.215987+0000","last_became_peered":"2023-10-19T13:02:58.215987+0000","last_unstale":"2023-10-19T13:02:59.212980+0000","last_undegraded":"2023-10-19T13:02:59.212980+0000","last_fullsized":"2023-10-19T13:02:59.212980+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:16:29.874475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.24","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.409125+0000","last_change":"2023-10-19T13:02:58.227441+0000","last_active":"2023-10-19T13:02:59.409125+0000","last_peered":"2023-10-19T13:02:59.409125+0000","last_clean":"2023-10-19T13:02:59.409125+0000","last_became_active":"2023-10-19T13:02:58.227166+0000","last_became_peered":"2023-10-19T13:02:58.227166+0000","last_unstale":"2023-10-19T13:02:59.409125+0000","last_undegraded":"2023-10-19T13:02:59.409125+0000","last_fullsized":"2023-10-19T13:02:59.409125+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:11:08.241375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.23","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.408673+0000","last_change":"2023-10-19T13:02:58.216138+0000","last_active":"2023-10-19T13:02:59.408673+0000","last_peered":"2023-10-19T13:02:59.408673+0000","last_clean":"2023-10-19T13:02:59.408673+0000","last_became_active":"2023-10-19T13:02:58.215870+0000","last_became_peered":"2023-10-19T13:02:58.215870+0000","last_unstale":"2023-10-19T13:02:59.408673+0000","last_undegraded":"2023-10-19T13:02:59.408673+0000","last_fullsized":"2023-10-19T13:02:59.408673+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:42:11.247167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.22","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.213608+0000","last_change":"2023-10-19T13:02:58.216990+0000","last_active":"2023-10-19T13:02:59.213608+0000","last_peered":"2023-10-19T13:02:59.213608+0000","last_clean":"2023-10-19T13:02:59.213608+0000","last_became_active":"2023-10-19T13:02:58.216449+0000","last_became_peered":"2023-10-19T13:02:58.216449+0000","last_unstale":"2023-10-19T13:02:59.213608+0000","last_undegraded":"2023-10-19T13:02:59.213608+0000","last_fullsized":"2023-10-19T13:02:59.213608+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:32:53.289930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.21","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.408430+0000","last_change":"2023-10-19T13:02:58.229241+0000","last_active":"2023-10-19T13:02:59.408430+0000","last_peered":"2023-10-19T13:02:59.408430+0000","last_clean":"2023-10-19T13:02:59.408430+0000","last_became_active":"2023-10-19T13:02:58.229007+0000","last_became_peered":"2023-10-19T13:02:58.229007+0000","last_unstale":"2023-10-19T13:02:59.408430+0000","last_undegraded":"2023-10-19T13:02:59.408430+0000","last_fullsized":"2023-10-19T13:02:59.408430+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:02:01.165779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.20","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.407984+0000","last_change":"2023-10-19T13:02:58.219917+0000","last_active":"2023-10-19T13:02:59.407984+0000","last_peered":"2023-10-19T13:02:59.407984+0000","last_clean":"2023-10-19T13:02:59.407984+0000","last_became_active":"2023-10-19T13:02:58.219258+0000","last_became_peered":"2023-10-19T13:02:58.219258+0000","last_unstale":"2023-10-19T13:02:59.407984+0000","last_undegraded":"2023-10-19T13:02:59.407984+0000","last_fullsized":"2023-10-19T13:02:59.407984+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:17:08.082110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.409258+0000","last_change":"2023-10-19T13:02:58.224690+0000","last_active":"2023-10-19T13:02:59.409258+0000","last_peered":"2023-10-19T13:02:59.409258+0000","last_clean":"2023-10-19T13:02:59.409258+0000","last_became_active":"2023-10-19T13:02:58.224407+0000","last_became_peered":"2023-10-19T13:02:58.224407+0000","last_unstale":"2023-10-19T13:02:59.409258+0000","last_undegraded":"2023-10-19T13:02:59.409258+0000","last_fullsized":"2023-10-19T13:02:59.409258+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:09:55.205778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"129'4","reported_seq":16,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.409388+0000","last_change":"2023-10-19T13:02:57.341868+0000","last_active":"2023-10-19T13:02:59.409388+0000","last_peered":"2023-10-19T13:02:59.409388+0000","last_clean":"2023-10-19T13:02:59.409388+0000","last_became_active":"2023-10-19T13:02:57.341592+0000","last_became_peered":"2023-10-19T13:02:57.341592+0000","last_unstale":"2023-10-19T13:02:59.409388+0000","last_undegraded":"2023-10-19T13:02:59.409388+0000","last_fullsized":"2023-10-19T13:02:59.409388+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_clean_scrub_stamp":"2023-10-19T13:02:56.313383+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:50:36.546546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.408470+0000","last_change":"2023-10-19T13:02:58.223460+0000","last_active":"2023-10-19T13:02:59.408470+0000","last_peered":"2023-10-19T13:02:59.408470+0000","last_clean":"2023-10-19T13:02:59.408470+0000","last_became_active":"2023-10-19T13:02:58.223203+0000","last_became_peered":"2023-10-19T13:02:58.223203+0000","last_unstale":"2023-10-19T13:02:59.408470+0000","last_undegraded":"2023-10-19T13:02:59.408470+0000","last_fullsized":"2023-10-19T13:02:59.408470+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:11:22.362227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.1e","version":"129'1","reported_seq":13,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.409422+0000","last_change":"2023-10-19T13:02:57.341613+0000","last_active":"2023-10-19T13:02:59.409422+0000","last_peered":"2023-10-19T13:02:59.409422+0000","last_clean":"2023-10-19T13:02:59.409422+0000","last_became_active":"2023-10-19T13:02:57.341004+0000","last_became_peered":"2023-10-19T13:02:57.341004+0000","last_unstale":"2023-10-19T13:02:59.409422+0000","last_undegraded":"2023-10-19T13:02:59.409422+0000","last_fullsized":"2023-10-19T13:02:59.409422+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_clean_scrub_stamp":"2023-10-19T13:02:56.313383+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:29:17.043080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.213646+0000","last_change":"2023-10-19T13:02:58.222847+0000","last_active":"2023-10-19T13:02:59.213646+0000","last_peered":"2023-10-19T13:02:59.213646+0000","last_clean":"2023-10-19T13:02:59.213646+0000","last_became_active":"2023-10-19T13:02:58.222574+0000","last_became_peered":"2023-10-19T13:02:58.222574+0000","last_unstale":"2023-10-19T13:02:59.213646+0000","last_undegraded":"2023-10-19T13:02:59.213646+0000","last_fullsized":"2023-10-19T13:02:59.213646+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:41:23.057082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.1d","version":"129'2","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.213912+0000","last_change":"2023-10-19T13:02:57.347098+0000","last_active":"2023-10-19T13:02:59.213912+0000","last_peered":"2023-10-19T13:02:59.213912+0000","last_clean":"2023-10-19T13:02:59.213912+0000","last_became_active":"2023-10-19T13:02:57.346133+0000","last_became_peered":"2023-10-19T13:02:57.346133+0000","last_unstale":"2023-10-19T13:02:59.213912+0000","last_undegraded":"2023-10-19T13:02:59.213912+0000","last_fullsized":"2023-10-19T13:02:59.213912+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_clean_scrub_stamp":"2023-10-19T13:02:56.313383+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:55:56.069072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.213094+0000","last_change":"2023-10-19T13:02:58.218317+0000","last_active":"2023-10-19T13:02:59.213094+0000","last_peered":"2023-10-19T13:02:59.213094+0000","last_clean":"2023-10-19T13:02:59.213094+0000","last_became_active":"2023-10-19T13:02:58.217907+0000","last_became_peered":"2023-10-19T13:02:58.217907+0000","last_unstale":"2023-10-19T13:02:59.213094+0000","last_undegraded":"2023-10-19T13:02:59.213094+0000","last_fullsized":"2023-10-19T13:02:59.213094+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:04:59.437455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.1c","version":"129'2","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.210725+0000","last_change":"2023-10-19T13:02:57.347704+0000","last_active":"2023-10-19T13:02:59.210725+0000","last_peered":"2023-10-19T13:02:59.210725+0000","last_clean":"2023-10-19T13:02:59.210725+0000","last_became_active":"2023-10-19T13:02:57.347427+0000","last_became_peered":"2023-10-19T13:02:57.347427+0000","last_unstale":"2023-10-19T13:02:59.210725+0000","last_undegraded":"2023-10-19T13:02:59.210725+0000","last_fullsized":"2023-10-19T13:02:59.210725+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_clean_scrub_stamp":"2023-10-19T13:02:56.313383+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:53:29.668027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.409184+0000","last_change":"2023-10-19T13:02:58.226429+0000","last_active":"2023-10-19T13:02:59.409184+0000","last_peered":"2023-10-19T13:02:59.409184+0000","last_clean":"2023-10-19T13:02:59.409184+0000","last_became_active":"2023-10-19T13:02:58.226163+0000","last_became_peered":"2023-10-19T13:02:58.226163+0000","last_unstale":"2023-10-19T13:02:59.409184+0000","last_undegraded":"2023-10-19T13:02:59.409184+0000","last_fullsized":"2023-10-19T13:02:59.409184+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:33:31.616543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1b","version":"129'1","reported_seq":13,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.408834+0000","last_change":"2023-10-19T13:02:57.344798+0000","last_active":"2023-10-19T13:02:59.408834+0000","last_peered":"2023-10-19T13:02:59.408834+0000","last_clean":"2023-10-19T13:02:59.408834+0000","last_became_active":"2023-10-19T13:02:57.344317+0000","last_became_peered":"2023-10-19T13:02:57.344317+0000","last_unstale":"2023-10-19T13:02:59.408834+0000","last_undegraded":"2023-10-19T13:02:59.408834+0000","last_fullsized":"2023-10-19T13:02:59.408834+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_clean_scrub_stamp":"2023-10-19T13:02:56.313383+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:04:58.430262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.408663+0000","last_change":"2023-10-19T13:02:58.227324+0000","last_active":"2023-10-19T13:02:59.408663+0000","last_peered":"2023-10-19T13:02:59.408663+0000","last_clean":"2023-10-19T13:02:59.408663+0000","last_became_active":"2023-10-19T13:02:58.226965+0000","last_became_peered":"2023-10-19T13:02:58.226965+0000","last_unstale":"2023-10-19T13:02:59.408663+0000","last_undegraded":"2023-10-19T13:02:59.408663+0000","last_fullsized":"2023-10-19T13:02:59.408663+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:12:16.687425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"129'1","reported_seq":13,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.212607+0000","last_change":"2023-10-19T13:02:57.347245+0000","last_active":"2023-10-19T13:02:59.212607+0000","last_peered":"2023-10-19T13:02:59.212607+0000","last_clean":"2023-10-19T13:02:59.212607+0000","last_became_active":"2023-10-19T13:02:57.346627+0000","last_became_peered":"2023-10-19T13:02:57.346627+0000","last_unstale":"2023-10-19T13:02:59.212607+0000","last_undegraded":"2023-10-19T13:02:59.212607+0000","last_fullsized":"2023-10-19T13:02:59.212607+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_clean_scrub_stamp":"2023-10-19T13:02:56.313383+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:33:56.945657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.409382+0000","last_change":"2023-10-19T13:02:58.225680+0000","last_active":"2023-10-19T13:02:59.409382+0000","last_peered":"2023-10-19T13:02:59.409382+0000","last_clean":"2023-10-19T13:02:59.409382+0000","last_became_active":"2023-10-19T13:02:58.225250+0000","last_became_peered":"2023-10-19T13:02:58.225250+0000","last_unstale":"2023-10-19T13:02:59.409382+0000","last_undegraded":"2023-10-19T13:02:59.409382+0000","last_fullsized":"2023-10-19T13:02:59.409382+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:28:08.887992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.213548+0000","last_change":"2023-10-19T13:02:57.342791+0000","last_active":"2023-10-19T13:02:59.213548+0000","last_peered":"2023-10-19T13:02:59.213548+0000","last_clean":"2023-10-19T13:02:59.213548+0000","last_became_active":"2023-10-19T13:02:57.342531+0000","last_became_peered":"2023-10-19T13:02:57.342531+0000","last_unstale":"2023-10-19T13:02:59.213548+0000","last_undegraded":"2023-10-19T13:02:59.213548+0000","last_fullsized":"2023-10-19T13:02:59.213548+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_clean_scrub_stamp":"2023-10-19T13:02:56.313383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:00:44.818454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.409399+0000","last_change":"2023-10-19T13:02:58.217721+0000","last_active":"2023-10-19T13:02:59.409399+0000","last_peered":"2023-10-19T13:02:59.409399+0000","last_clean":"2023-10-19T13:02:59.409399+0000","last_became_active":"2023-10-19T13:02:58.216651+0000","last_became_peered":"2023-10-19T13:02:58.216651+0000","last_unstale":"2023-10-19T13:02:59.409399+0000","last_undegraded":"2023-10-19T13:02:59.409399+0000","last_fullsized":"2023-10-19T13:02:59.409399+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:27:04.472644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.408912+0000","last_change":"2023-10-19T13:02:57.344873+0000","last_active":"2023-10-19T13:02:59.408912+0000","last_peered":"2023-10-19T13:02:59.408912+0000","last_clean":"2023-10-19T13:02:59.408912+0000","last_became_active":"2023-10-19T13:02:57.344610+0000","last_became_peered":"2023-10-19T13:02:57.344610+0000","last_unstale":"2023-10-19T13:02:59.408912+0000","last_undegraded":"2023-10-19T13:02:59.408912+0000","last_fullsized":"2023-10-19T13:02:59.408912+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_clean_scrub_stamp":"2023-10-19T13:02:56.313383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:46:15.256442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.409180+0000","last_change":"2023-10-19T13:02:58.225271+0000","last_active":"2023-10-19T13:02:59.409180+0000","last_peered":"2023-10-19T13:02:59.409180+0000","last_clean":"2023-10-19T13:02:59.409180+0000","last_became_active":"2023-10-19T13:02:58.225000+0000","last_became_peered":"2023-10-19T13:02:58.225000+0000","last_unstale":"2023-10-19T13:02:59.409180+0000","last_undegraded":"2023-10-19T13:02:59.409180+0000","last_fullsized":"2023-10-19T13:02:59.409180+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:49:07.743223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.213952+0000","last_change":"2023-10-19T13:02:57.346397+0000","last_active":"2023-10-19T13:02:59.213952+0000","last_peered":"2023-10-19T13:02:59.213952+0000","last_clean":"2023-10-19T13:02:59.213952+0000","last_became_active":"2023-10-19T13:02:57.345496+0000","last_became_peered":"2023-10-19T13:02:57.345496+0000","last_unstale":"2023-10-19T13:02:59.213952+0000","last_undegraded":"2023-10-19T13:02:59.213952+0000","last_fullsized":"2023-10-19T13:02:59.213952+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_clean_scrub_stamp":"2023-10-19T13:02:56.313383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:11:07.234183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.210483+0000","last_change":"2023-10-19T13:02:58.221059+0000","last_active":"2023-10-19T13:02:59.210483+0000","last_peered":"2023-10-19T13:02:59.210483+0000","last_clean":"2023-10-19T13:02:59.210483+0000","last_became_active":"2023-10-19T13:02:58.220775+0000","last_became_peered":"2023-10-19T13:02:58.220775+0000","last_unstale":"2023-10-19T13:02:59.210483+0000","last_undegraded":"2023-10-19T13:02:59.210483+0000","last_fullsized":"2023-10-19T13:02:59.210483+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:37:57.278166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.408871+0000","last_change":"2023-10-19T13:02:57.346719+0000","last_active":"2023-10-19T13:02:59.408871+0000","last_peered":"2023-10-19T13:02:59.408871+0000","last_clean":"2023-10-19T13:02:59.408871+0000","last_became_active":"2023-10-19T13:02:57.346389+0000","last_became_peered":"2023-10-19T13:02:57.346389+0000","last_unstale":"2023-10-19T13:02:59.408871+0000","last_undegraded":"2023-10-19T13:02:59.408871+0000","last_fullsized":"2023-10-19T13:02:59.408871+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_clean_scrub_stamp":"2023-10-19T13:02:56.313383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:16:13.963090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.408187+0000","last_change":"2023-10-19T13:02:58.224216+0000","last_active":"2023-10-19T13:02:59.408187+0000","last_peered":"2023-10-19T13:02:59.408187+0000","last_clean":"2023-10-19T13:02:59.408187+0000","last_became_active":"2023-10-19T13:02:58.223878+0000","last_became_peered":"2023-10-19T13:02:58.223878+0000","last_unstale":"2023-10-19T13:02:59.408187+0000","last_undegraded":"2023-10-19T13:02:59.408187+0000","last_fullsized":"2023-10-19T13:02:59.408187+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:30:23.503356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.15","version":"129'3","reported_seq":15,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.408882+0000","last_change":"2023-10-19T13:02:57.350562+0000","last_active":"2023-10-19T13:02:59.408882+0000","last_peered":"2023-10-19T13:02:59.408882+0000","last_clean":"2023-10-19T13:02:59.408882+0000","last_became_active":"2023-10-19T13:02:57.349996+0000","last_became_peered":"2023-10-19T13:02:57.349996+0000","last_unstale":"2023-10-19T13:02:59.408882+0000","last_undegraded":"2023-10-19T13:02:59.408882+0000","last_fullsized":"2023-10-19T13:02:59.408882+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_clean_scrub_stamp":"2023-10-19T13:02:56.313383+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:51:15.426449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.210684+0000","last_change":"2023-10-19T13:02:58.222283+0000","last_active":"2023-10-19T13:02:59.210684+0000","last_peered":"2023-10-19T13:02:59.210684+0000","last_clean":"2023-10-19T13:02:59.210684+0000","last_became_active":"2023-10-19T13:02:58.220921+0000","last_became_peered":"2023-10-19T13:02:58.220921+0000","last_unstale":"2023-10-19T13:02:59.210684+0000","last_undegraded":"2023-10-19T13:02:59.210684+0000","last_fullsized":"2023-10-19T13:02:59.210684+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:03:30.317716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.14","version":"129'6","reported_seq":18,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.409178+0000","last_change":"2023-10-19T13:02:57.350320+0000","last_active":"2023-10-19T13:02:59.409178+0000","last_peered":"2023-10-19T13:02:59.409178+0000","last_clean":"2023-10-19T13:02:59.409178+0000","last_became_active":"2023-10-19T13:02:57.350049+0000","last_became_peered":"2023-10-19T13:02:57.350049+0000","last_unstale":"2023-10-19T13:02:59.409178+0000","last_undegraded":"2023-10-19T13:02:59.409178+0000","last_fullsized":"2023-10-19T13:02:59.409178+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_clean_scrub_stamp":"2023-10-19T13:02:56.313383+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:20:18.570502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.212819+0000","last_change":"2023-10-19T13:02:58.216397+0000","last_active":"2023-10-19T13:02:59.212819+0000","last_peered":"2023-10-19T13:02:59.212819+0000","last_clean":"2023-10-19T13:02:59.212819+0000","last_became_active":"2023-10-19T13:02:58.215587+0000","last_became_peered":"2023-10-19T13:02:58.215587+0000","last_unstale":"2023-10-19T13:02:59.212819+0000","last_undegraded":"2023-10-19T13:02:59.212819+0000","last_fullsized":"2023-10-19T13:02:59.212819+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:45:40.674335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.13","version":"129'2","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.210740+0000","last_change":"2023-10-19T13:02:57.345527+0000","last_active":"2023-10-19T13:02:59.210740+0000","last_peered":"2023-10-19T13:02:59.210740+0000","last_clean":"2023-10-19T13:02:59.210740+0000","last_became_active":"2023-10-19T13:02:57.345022+0000","last_became_peered":"2023-10-19T13:02:57.345022+0000","last_unstale":"2023-10-19T13:02:59.210740+0000","last_undegraded":"2023-10-19T13:02:59.210740+0000","last_fullsized":"2023-10-19T13:02:59.210740+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_clean_scrub_stamp":"2023-10-19T13:02:56.313383+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:12:11.087920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.409209+0000","last_change":"2023-10-19T13:02:58.228358+0000","last_active":"2023-10-19T13:02:59.409209+0000","last_peered":"2023-10-19T13:02:59.409209+0000","last_clean":"2023-10-19T13:02:59.409209+0000","last_became_active":"2023-10-19T13:02:58.228101+0000","last_became_peered":"2023-10-19T13:02:58.228101+0000","last_unstale":"2023-10-19T13:02:59.409209+0000","last_undegraded":"2023-10-19T13:02:59.409209+0000","last_fullsized":"2023-10-19T13:02:59.409209+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:55:57.076264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.409218+0000","last_change":"2023-10-19T13:02:57.337039+0000","last_active":"2023-10-19T13:02:59.409218+0000","last_peered":"2023-10-19T13:02:59.409218+0000","last_clean":"2023-10-19T13:02:59.409218+0000","last_became_active":"2023-10-19T13:02:57.336770+0000","last_became_peered":"2023-10-19T13:02:57.336770+0000","last_unstale":"2023-10-19T13:02:59.409218+0000","last_undegraded":"2023-10-19T13:02:59.409218+0000","last_fullsized":"2023-10-19T13:02:59.409218+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_clean_scrub_stamp":"2023-10-19T13:02:56.313383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:17:15.888716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.409484+0000","last_change":"2023-10-19T13:02:58.215773+0000","last_active":"2023-10-19T13:02:59.409484+0000","last_peered":"2023-10-19T13:02:59.409484+0000","last_clean":"2023-10-19T13:02:59.409484+0000","last_became_active":"2023-10-19T13:02:58.215049+0000","last_became_peered":"2023-10-19T13:02:58.215049+0000","last_unstale":"2023-10-19T13:02:59.409484+0000","last_undegraded":"2023-10-19T13:02:59.409484+0000","last_fullsized":"2023-10-19T13:02:59.409484+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:34:16.626818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.408547+0000","last_change":"2023-10-19T13:02:57.346855+0000","last_active":"2023-10-19T13:02:59.408547+0000","last_peered":"2023-10-19T13:02:59.408547+0000","last_clean":"2023-10-19T13:02:59.408547+0000","last_became_active":"2023-10-19T13:02:57.346557+0000","last_became_peered":"2023-10-19T13:02:57.346557+0000","last_unstale":"2023-10-19T13:02:59.408547+0000","last_undegraded":"2023-10-19T13:02:59.408547+0000","last_fullsized":"2023-10-19T13:02:59.408547+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_clean_scrub_stamp":"2023-10-19T13:02:56.313383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:30:59.713074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.408829+0000","last_change":"2023-10-19T13:02:58.217109+0000","last_active":"2023-10-19T13:02:59.408829+0000","last_peered":"2023-10-19T13:02:59.408829+0000","last_clean":"2023-10-19T13:02:59.408829+0000","last_became_active":"2023-10-19T13:02:58.216846+0000","last_became_peered":"2023-10-19T13:02:58.216846+0000","last_unstale":"2023-10-19T13:02:59.408829+0000","last_undegraded":"2023-10-19T13:02:59.408829+0000","last_fullsized":"2023-10-19T13:02:59.408829+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:29:09.842513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.10","version":"129'8","reported_seq":20,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.213496+0000","last_change":"2023-10-19T13:02:57.346003+0000","last_active":"2023-10-19T13:02:59.213496+0000","last_peered":"2023-10-19T13:02:59.213496+0000","last_clean":"2023-10-19T13:02:59.213496+0000","last_became_active":"2023-10-19T13:02:57.345736+0000","last_became_peered":"2023-10-19T13:02:57.345736+0000","last_unstale":"2023-10-19T13:02:59.213496+0000","last_undegraded":"2023-10-19T13:02:59.213496+0000","last_fullsized":"2023-10-19T13:02:59.213496+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_clean_scrub_stamp":"2023-10-19T13:02:56.313383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:37:56.270973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.408061+0000","last_change":"2023-10-19T13:02:58.217845+0000","last_active":"2023-10-19T13:02:59.408061+0000","last_peered":"2023-10-19T13:02:59.408061+0000","last_clean":"2023-10-19T13:02:59.408061+0000","last_became_active":"2023-10-19T13:02:58.216987+0000","last_became_peered":"2023-10-19T13:02:58.216987+0000","last_unstale":"2023-10-19T13:02:59.408061+0000","last_undegraded":"2023-10-19T13:02:59.408061+0000","last_fullsized":"2023-10-19T13:02:59.408061+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:15:35.293533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.f","version":"129'1","reported_seq":13,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.409335+0000","last_change":"2023-10-19T13:02:57.346451+0000","last_active":"2023-10-19T13:02:59.409335+0000","last_peered":"2023-10-19T13:02:59.409335+0000","last_clean":"2023-10-19T13:02:59.409335+0000","last_became_active":"2023-10-19T13:02:57.346192+0000","last_became_peered":"2023-10-19T13:02:57.346192+0000","last_unstale":"2023-10-19T13:02:59.409335+0000","last_undegraded":"2023-10-19T13:02:59.409335+0000","last_fullsized":"2023-10-19T13:02:59.409335+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_clean_scrub_stamp":"2023-10-19T13:02:56.313383+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:53:41.138926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.408776+0000","last_change":"2023-10-19T13:02:58.225451+0000","last_active":"2023-10-19T13:02:59.408776+0000","last_peered":"2023-10-19T13:02:59.408776+0000","last_clean":"2023-10-19T13:02:59.408776+0000","last_became_active":"2023-10-19T13:02:58.225176+0000","last_became_peered":"2023-10-19T13:02:58.225176+0000","last_unstale":"2023-10-19T13:02:59.408776+0000","last_undegraded":"2023-10-19T13:02:59.408776+0000","last_fullsized":"2023-10-19T13:02:59.408776+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:00:45.825647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"129'1","reported_seq":13,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.408883+0000","last_change":"2023-10-19T13:02:57.346495+0000","last_active":"2023-10-19T13:02:59.408883+0000","last_peered":"2023-10-19T13:02:59.408883+0000","last_clean":"2023-10-19T13:02:59.408883+0000","last_became_active":"2023-10-19T13:02:57.346248+0000","last_became_peered":"2023-10-19T13:02:57.346248+0000","last_unstale":"2023-10-19T13:02:59.408883+0000","last_undegraded":"2023-10-19T13:02:59.408883+0000","last_fullsized":"2023-10-19T13:02:59.408883+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_clean_scrub_stamp":"2023-10-19T13:02:56.313383+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:45:39.667143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.210608+0000","last_change":"2023-10-19T13:02:58.222612+0000","last_active":"2023-10-19T13:02:59.210608+0000","last_peered":"2023-10-19T13:02:59.210608+0000","last_clean":"2023-10-19T13:02:59.210608+0000","last_became_active":"2023-10-19T13:02:58.222277+0000","last_became_peered":"2023-10-19T13:02:58.222277+0000","last_unstale":"2023-10-19T13:02:59.210608+0000","last_undegraded":"2023-10-19T13:02:59.210608+0000","last_fullsized":"2023-10-19T13:02:59.210608+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:55:57.076264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.212982+0000","last_change":"2023-10-19T13:02:57.342296+0000","last_active":"2023-10-19T13:02:59.212982+0000","last_peered":"2023-10-19T13:02:59.212982+0000","last_clean":"2023-10-19T13:02:59.212982+0000","last_became_active":"2023-10-19T13:02:57.342004+0000","last_became_peered":"2023-10-19T13:02:57.342004+0000","last_unstale":"2023-10-19T13:02:59.212982+0000","last_undegraded":"2023-10-19T13:02:59.212982+0000","last_fullsized":"2023-10-19T13:02:59.212982+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_clean_scrub_stamp":"2023-10-19T13:02:56.313383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:32:55.806215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.408353+0000","last_change":"2023-10-19T13:02:58.218268+0000","last_active":"2023-10-19T13:02:59.408353+0000","last_peered":"2023-10-19T13:02:59.408353+0000","last_clean":"2023-10-19T13:02:59.408353+0000","last_became_active":"2023-10-19T13:02:58.217998+0000","last_became_peered":"2023-10-19T13:02:58.217998+0000","last_unstale":"2023-10-19T13:02:59.408353+0000","last_undegraded":"2023-10-19T13:02:59.408353+0000","last_fullsized":"2023-10-19T13:02:59.408353+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:17:05.559019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"129'2","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.408757+0000","last_change":"2023-10-19T13:02:57.350309+0000","last_active":"2023-10-19T13:02:59.408757+0000","last_peered":"2023-10-19T13:02:59.408757+0000","last_clean":"2023-10-19T13:02:59.408757+0000","last_became_active":"2023-10-19T13:02:57.349972+0000","last_became_peered":"2023-10-19T13:02:57.349972+0000","last_unstale":"2023-10-19T13:02:59.408757+0000","last_undegraded":"2023-10-19T13:02:59.408757+0000","last_fullsized":"2023-10-19T13:02:59.408757+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_clean_scrub_stamp":"2023-10-19T13:02:56.313383+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:21:14.791478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.408542+0000","last_change":"2023-10-19T13:02:58.222727+0000","last_active":"2023-10-19T13:02:59.408542+0000","last_peered":"2023-10-19T13:02:59.408542+0000","last_clean":"2023-10-19T13:02:59.408542+0000","last_became_active":"2023-10-19T13:02:58.222434+0000","last_became_peered":"2023-10-19T13:02:58.222434+0000","last_unstale":"2023-10-19T13:02:59.408542+0000","last_undegraded":"2023-10-19T13:02:59.408542+0000","last_fullsized":"2023-10-19T13:02:59.408542+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:58:57.535121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.408050+0000","last_change":"2023-10-19T13:02:57.340349+0000","last_active":"2023-10-19T13:02:59.408050+0000","last_peered":"2023-10-19T13:02:59.408050+0000","last_clean":"2023-10-19T13:02:59.408050+0000","last_became_active":"2023-10-19T13:02:57.340078+0000","last_became_peered":"2023-10-19T13:02:57.340078+0000","last_unstale":"2023-10-19T13:02:59.408050+0000","last_undegraded":"2023-10-19T13:02:59.408050+0000","last_fullsized":"2023-10-19T13:02:59.408050+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_clean_scrub_stamp":"2023-10-19T13:02:56.313383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:35:12.202836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.408762+0000","last_change":"2023-10-19T13:02:58.227358+0000","last_active":"2023-10-19T13:02:59.408762+0000","last_peered":"2023-10-19T13:02:59.408762+0000","last_clean":"2023-10-19T13:02:59.408762+0000","last_became_active":"2023-10-19T13:02:58.227073+0000","last_became_peered":"2023-10-19T13:02:58.227073+0000","last_unstale":"2023-10-19T13:02:59.408762+0000","last_undegraded":"2023-10-19T13:02:59.408762+0000","last_fullsized":"2023-10-19T13:02:59.408762+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:12:30.411021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.408686+0000","last_change":"2023-10-19T13:02:57.349912+0000","last_active":"2023-10-19T13:02:59.408686+0000","last_peered":"2023-10-19T13:02:59.408686+0000","last_clean":"2023-10-19T13:02:59.408686+0000","last_became_active":"2023-10-19T13:02:57.349674+0000","last_became_peered":"2023-10-19T13:02:57.349674+0000","last_unstale":"2023-10-19T13:02:59.408686+0000","last_undegraded":"2023-10-19T13:02:59.408686+0000","last_fullsized":"2023-10-19T13:02:59.408686+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_clean_scrub_stamp":"2023-10-19T13:02:56.313383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:51:32.174702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.409500+0000","last_change":"2023-10-19T13:02:58.224881+0000","last_active":"2023-10-19T13:02:59.409500+0000","last_peered":"2023-10-19T13:02:59.409500+0000","last_clean":"2023-10-19T13:02:59.409500+0000","last_became_active":"2023-10-19T13:02:58.224574+0000","last_became_peered":"2023-10-19T13:02:58.224574+0000","last_unstale":"2023-10-19T13:02:59.409500+0000","last_undegraded":"2023-10-19T13:02:59.409500+0000","last_fullsized":"2023-10-19T13:02:59.409500+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:53:03.608451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.408072+0000","last_change":"2023-10-19T13:02:57.338251+0000","last_active":"2023-10-19T13:02:59.408072+0000","last_peered":"2023-10-19T13:02:59.408072+0000","last_clean":"2023-10-19T13:02:59.408072+0000","last_became_active":"2023-10-19T13:02:57.337986+0000","last_became_peered":"2023-10-19T13:02:57.337986+0000","last_unstale":"2023-10-19T13:02:59.408072+0000","last_undegraded":"2023-10-19T13:02:59.408072+0000","last_fullsized":"2023-10-19T13:02:59.408072+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_clean_scrub_stamp":"2023-10-19T13:02:56.313383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:44:07.363435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.409130+0000","last_change":"2023-10-19T13:02:58.218497+0000","last_active":"2023-10-19T13:02:59.409130+0000","last_peered":"2023-10-19T13:02:59.409130+0000","last_clean":"2023-10-19T13:02:59.409130+0000","last_became_active":"2023-10-19T13:02:58.218231+0000","last_became_peered":"2023-10-19T13:02:58.218231+0000","last_unstale":"2023-10-19T13:02:59.409130+0000","last_undegraded":"2023-10-19T13:02:59.409130+0000","last_fullsized":"2023-10-19T13:02:59.409130+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:25:48.439527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.7","version":"129'2","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.210804+0000","last_change":"2023-10-19T13:02:57.345702+0000","last_active":"2023-10-19T13:02:59.210804+0000","last_peered":"2023-10-19T13:02:59.210804+0000","last_clean":"2023-10-19T13:02:59.210804+0000","last_became_active":"2023-10-19T13:02:57.345270+0000","last_became_peered":"2023-10-19T13:02:57.345270+0000","last_unstale":"2023-10-19T13:02:59.210804+0000","last_undegraded":"2023-10-19T13:02:59.210804+0000","last_fullsized":"2023-10-19T13:02:59.210804+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_clean_scrub_stamp":"2023-10-19T13:02:56.313383+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:20:18.570502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.213903+0000","last_change":"2023-10-19T13:02:58.218877+0000","last_active":"2023-10-19T13:02:59.213903+0000","last_peered":"2023-10-19T13:02:59.213903+0000","last_clean":"2023-10-19T13:02:59.213903+0000","last_became_active":"2023-10-19T13:02:58.217823+0000","last_became_peered":"2023-10-19T13:02:58.217823+0000","last_unstale":"2023-10-19T13:02:59.213903+0000","last_undegraded":"2023-10-19T13:02:59.213903+0000","last_fullsized":"2023-10-19T13:02:59.213903+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:32:01.384881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.408950+0000","last_change":"2023-10-19T13:02:57.339607+0000","last_active":"2023-10-19T13:02:59.408950+0000","last_peered":"2023-10-19T13:02:59.408950+0000","last_clean":"2023-10-19T13:02:59.408950+0000","last_became_active":"2023-10-19T13:02:57.339223+0000","last_became_peered":"2023-10-19T13:02:57.339223+0000","last_unstale":"2023-10-19T13:02:59.408950+0000","last_undegraded":"2023-10-19T13:02:59.408950+0000","last_fullsized":"2023-10-19T13:02:59.408950+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_clean_scrub_stamp":"2023-10-19T13:02:56.313383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:18:16.079990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.409228+0000","last_change":"2023-10-19T13:02:58.225583+0000","last_active":"2023-10-19T13:02:59.409228+0000","last_peered":"2023-10-19T13:02:59.409228+0000","last_clean":"2023-10-19T13:02:59.409228+0000","last_became_active":"2023-10-19T13:02:58.225294+0000","last_became_peered":"2023-10-19T13:02:58.225294+0000","last_unstale":"2023-10-19T13:02:59.409228+0000","last_undegraded":"2023-10-19T13:02:59.409228+0000","last_fullsized":"2023-10-19T13:02:59.409228+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:47:15.411344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.409189+0000","last_change":"2023-10-19T13:02:57.339438+0000","last_active":"2023-10-19T13:02:59.409189+0000","last_peered":"2023-10-19T13:02:59.409189+0000","last_clean":"2023-10-19T13:02:59.409189+0000","last_became_active":"2023-10-19T13:02:57.339138+0000","last_became_peered":"2023-10-19T13:02:57.339138+0000","last_unstale":"2023-10-19T13:02:59.409189+0000","last_undegraded":"2023-10-19T13:02:59.409189+0000","last_fullsized":"2023-10-19T13:02:59.409189+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_clean_scrub_stamp":"2023-10-19T13:02:56.313383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:53:22.719704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.212780+0000","last_change":"2023-10-19T13:02:58.223848+0000","last_active":"2023-10-19T13:02:59.212780+0000","last_peered":"2023-10-19T13:02:59.212780+0000","last_clean":"2023-10-19T13:02:59.212780+0000","last_became_active":"2023-10-19T13:02:58.223560+0000","last_became_peered":"2023-10-19T13:02:58.223560+0000","last_unstale":"2023-10-19T13:02:59.212780+0000","last_undegraded":"2023-10-19T13:02:59.212780+0000","last_fullsized":"2023-10-19T13:02:59.212780+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:51:16.433642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.210721+0000","last_change":"2023-10-19T13:02:57.345079+0000","last_active":"2023-10-19T13:02:59.210721+0000","last_peered":"2023-10-19T13:02:59.210721+0000","last_clean":"2023-10-19T13:02:59.210721+0000","last_became_active":"2023-10-19T13:02:57.344768+0000","last_became_peered":"2023-10-19T13:02:57.344768+0000","last_unstale":"2023-10-19T13:02:59.210721+0000","last_undegraded":"2023-10-19T13:02:59.210721+0000","last_fullsized":"2023-10-19T13:02:59.210721+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_clean_scrub_stamp":"2023-10-19T13:02:56.313383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:43:51.920104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.408696+0000","last_change":"2023-10-19T13:02:58.222671+0000","last_active":"2023-10-19T13:02:59.408696+0000","last_peered":"2023-10-19T13:02:59.408696+0000","last_clean":"2023-10-19T13:02:59.408696+0000","last_became_active":"2023-10-19T13:02:58.222215+0000","last_became_peered":"2023-10-19T13:02:58.222215+0000","last_unstale":"2023-10-19T13:02:59.408696+0000","last_undegraded":"2023-10-19T13:02:59.408696+0000","last_fullsized":"2023-10-19T13:02:59.408696+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:49:46.093687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.214027+0000","last_change":"2023-10-19T13:02:57.343381+0000","last_active":"2023-10-19T13:02:59.214027+0000","last_peered":"2023-10-19T13:02:59.214027+0000","last_clean":"2023-10-19T13:02:59.214027+0000","last_became_active":"2023-10-19T13:02:57.342255+0000","last_became_peered":"2023-10-19T13:02:57.342255+0000","last_unstale":"2023-10-19T13:02:59.214027+0000","last_undegraded":"2023-10-19T13:02:59.214027+0000","last_fullsized":"2023-10-19T13:02:59.214027+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_clean_scrub_stamp":"2023-10-19T13:02:56.313383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:03:29.310523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.408284+0000","last_change":"2023-10-19T13:02:58.222029+0000","last_active":"2023-10-19T13:02:59.408284+0000","last_peered":"2023-10-19T13:02:59.408284+0000","last_clean":"2023-10-19T13:02:59.408284+0000","last_became_active":"2023-10-19T13:02:58.221767+0000","last_became_peered":"2023-10-19T13:02:58.221767+0000","last_unstale":"2023-10-19T13:02:59.408284+0000","last_undegraded":"2023-10-19T13:02:59.408284+0000","last_fullsized":"2023-10-19T13:02:59.408284+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:24:51.398803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.408264+0000","last_change":"2023-10-19T13:02:57.346182+0000","last_active":"2023-10-19T13:02:59.408264+0000","last_peered":"2023-10-19T13:02:59.408264+0000","last_clean":"2023-10-19T13:02:59.408264+0000","last_became_active":"2023-10-19T13:02:57.345888+0000","last_became_peered":"2023-10-19T13:02:57.345888+0000","last_unstale":"2023-10-19T13:02:59.408264+0000","last_undegraded":"2023-10-19T13:02:59.408264+0000","last_fullsized":"2023-10-19T13:02:59.408264+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_clean_scrub_stamp":"2023-10-19T13:02:56.313383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:31:39.380962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.408925+0000","last_change":"2023-10-19T13:02:58.220499+0000","last_active":"2023-10-19T13:02:59.408925+0000","last_peered":"2023-10-19T13:02:59.408925+0000","last_clean":"2023-10-19T13:02:59.408925+0000","last_became_active":"2023-10-19T13:02:58.220218+0000","last_became_peered":"2023-10-19T13:02:58.220218+0000","last_unstale":"2023-10-19T13:02:59.408925+0000","last_undegraded":"2023-10-19T13:02:59.408925+0000","last_fullsized":"2023-10-19T13:02:59.408925+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:59:25.738088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.408527+0000","last_change":"2023-10-19T13:02:57.339476+0000","last_active":"2023-10-19T13:02:59.408527+0000","last_peered":"2023-10-19T13:02:59.408527+0000","last_clean":"2023-10-19T13:02:59.408527+0000","last_became_active":"2023-10-19T13:02:57.339036+0000","last_became_peered":"2023-10-19T13:02:57.339036+0000","last_unstale":"2023-10-19T13:02:59.408527+0000","last_undegraded":"2023-10-19T13:02:59.408527+0000","last_fullsized":"2023-10-19T13:02:59.408527+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_clean_scrub_stamp":"2023-10-19T13:02:56.313383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:43:51.920104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.213191+0000","last_change":"2023-10-19T13:02:58.216921+0000","last_active":"2023-10-19T13:02:59.213191+0000","last_peered":"2023-10-19T13:02:59.213191+0000","last_clean":"2023-10-19T13:02:59.213191+0000","last_became_active":"2023-10-19T13:02:58.216622+0000","last_became_peered":"2023-10-19T13:02:58.216622+0000","last_unstale":"2023-10-19T13:02:59.213191+0000","last_undegraded":"2023-10-19T13:02:59.213191+0000","last_fullsized":"2023-10-19T13:02:59.213191+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:02:50.230615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.408148+0000","last_change":"2023-10-19T13:02:57.348114+0000","last_active":"2023-10-19T13:02:59.408148+0000","last_peered":"2023-10-19T13:02:59.408148+0000","last_clean":"2023-10-19T13:02:59.408148+0000","last_became_active":"2023-10-19T13:02:57.347896+0000","last_became_peered":"2023-10-19T13:02:57.347896+0000","last_unstale":"2023-10-19T13:02:59.408148+0000","last_undegraded":"2023-10-19T13:02:59.408148+0000","last_fullsized":"2023-10-19T13:02:59.408148+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_clean_scrub_stamp":"2023-10-19T13:02:56.313383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:33:00.094903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":11,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.212554+0000","last_change":"2023-10-19T13:02:58.215939+0000","last_active":"2023-10-19T13:02:59.212554+0000","last_peered":"2023-10-19T13:02:59.212554+0000","last_clean":"2023-10-19T13:02:59.212554+0000","last_became_active":"2023-10-19T13:02:58.215688+0000","last_became_peered":"2023-10-19T13:02:58.215688+0000","last_unstale":"2023-10-19T13:02:59.212554+0000","last_undegraded":"2023-10-19T13:02:59.212554+0000","last_fullsized":"2023-10-19T13:02:59.212554+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:57.320575+0000","last_clean_scrub_stamp":"2023-10-19T13:02:57.320575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:37:42.320918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.e","version":"129'2","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2023-10-19T13:02:59.408040+0000","last_change":"2023-10-19T13:02:57.341426+0000","last_active":"2023-10-19T13:02:59.408040+0000","last_peered":"2023-10-19T13:02:59.408040+0000","last_clean":"2023-10-19T13:02:59.408040+0000","last_became_active":"2023-10-19T13:02:57.340623+0000","last_became_peered":"2023-10-19T13:02:57.340623+0000","last_unstale":"2023-10-19T13:02:59.408040+0000","last_undegraded":"2023-10-19T13:02:59.408040+0000","last_fullsized":"2023-10-19T13:02:59.408040+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:02:56.313383+0000","last_clean_scrub_stamp":"2023-10-19T13:02:56.313383+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:54:18.810666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"118'3043","reported_seq":3461,"reported_epoch":129,"state":"active+clean","last_fresh":"2023-10-19T13:02:58.214127+0000","last_change":"2023-10-19T12:44:08.636613+0000","last_active":"2023-10-19T13:02:58.214127+0000","last_peered":"2023-10-19T13:02:58.214127+0000","last_clean":"2023-10-19T13:02:58.214127+0000","last_became_active":"2023-10-19T12:44:08.635697+0000","last_became_peered":"2023-10-19T12:44:08.635697+0000","last_unstale":"2023-10-19T13:02:58.214127+0000","last_undegraded":"2023-10-19T13:02:58.214127+0000","last_fullsized":"2023-10-19T13:02:58.214127+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_clean_scrub_stamp":"2023-10-19T12:44:07.399738+0000","objects_scrubbed":0,"log_size":3043,"log_dups_size":0,"ondisk_log_size":3043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:15:23.685485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":962,"num_read_kb":1946,"num_write":329,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":16,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":15,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":962,"num_read_kb":1946,"num_write":329,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3043,"ondisk_log_size":3043,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575077,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55548,"kb_used_data":2160,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316292,"statfs":{"total":96636764160,"available":96579883008,"internally_reserved":0,"allocated":2211840,"data_stored":1402705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54655435},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 13:02:49 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.53600000000000003,"15min":0.56299999999999994},"min":{"1min":0.44700000000000001,"5min":0.224,"15min":0.224},"max":{"1min":0.75,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.626,"5min":0.56699999999999995,"15min":0.58699999999999997},"min":{"1min":0.371,"5min":0.33200000000000002,"15min":0.29599999999999999},"max":{"1min":0.751,"5min":0.78200000000000003,"15min":0.93700000000000006},"last":0.45700000000000002}]},{"osd":1,"last update":"Thu Oct 19 13:02:49 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.55000000000000004,"15min":0.57399999999999995},"min":{"1min":0.432,"5min":0.33500000000000002,"15min":0.28199999999999997},"max":{"1min":0.69899999999999995,"5min":0.86299999999999999,"15min":1.3129999999999999},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.56899999999999995,"15min":0.58699999999999997},"min":{"1min":0.45100000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.755,"5min":0.85499999999999998,"15min":1.284},"last":0.60399999999999998}]},{"osd":2,"last update":"Thu Oct 19 13:02:49 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.60599999999999998,"15min":0.625},"min":{"1min":0.54100000000000004,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.76700000000000002,"5min":0.89200000000000002,"15min":0.98599999999999999},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.60499999999999998,"15min":0.63200000000000001},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.28399999999999997},"max":{"1min":0.751,"5min":0.83699999999999997,"15min":1.2589999999999999},"last":0.65000000000000002}]},{"osd":3,"last update":"Thu Oct 19 13:02:49 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.57899999999999996,"15min":0.60199999999999998},"min":{"1min":0.497,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.73399999999999999,"5min":0.87,"15min":0.96899999999999997},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.64700000000000002,"15min":0.65300000000000002},"min":{"1min":0.56699999999999995,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.80600000000000005,"5min":0.92000000000000004,"15min":1.329},"last":0.63600000000000001}]},{"osd":4,"last update":"Thu Oct 19 13:02:49 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.46700000000000003},"min":{"1min":0.318,"5min":0.247,"15min":0.247},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.47799999999999998,"15min":0.46800000000000003},"min":{"1min":0.34200000000000003,"5min":0.216,"15min":0.216},"max":{"1min":0.73399999999999999,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.52300000000000002}]},{"osd":5,"last update":"Thu Oct 19 13:02:49 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.49099999999999999,"15min":0.48199999999999998},"min":{"1min":0.34999999999999998,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.80600000000000005},"last":0.501},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.52500000000000002,"15min":0.51700000000000002},"min":{"1min":0.40300000000000002,"5min":0.314,"15min":0.314},"max":{"1min":0.72999999999999998,"5min":0.78900000000000003,"15min":0.81999999999999995},"last":0.61799999999999999}]},{"osd":6,"last update":"Thu Oct 19 13:02:49 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.55000000000000004,"15min":0.54600000000000004},"min":{"1min":0.42999999999999999,"5min":0.35299999999999998,"15min":0.308},"max":{"1min":0.78900000000000003,"5min":0.81299999999999994,"15min":0.88900000000000001},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.52500000000000002,"15min":0.52800000000000002},"min":{"1min":0.38600000000000001,"5min":0.318,"15min":0.27300000000000002},"max":{"1min":0.77000000000000002,"5min":0.83999999999999997,"15min":0.90300000000000002},"last":0.69499999999999995}]}]},{"osd":6,"up_from":13,"seq":55834575077,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32776,"kb_used_data":1592,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339064,"statfs":{"total":96636764160,"available":96603201536,"internally_reserved":0,"allocated":1630208,"data_stored":814393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 13:02:33 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.54600000000000004,"15min":0.58199999999999996},"min":{"1min":0.32400000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.76400000000000001,"5min":1.196,"15min":1.196},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.54300000000000004,"15min":0.57999999999999996},"min":{"1min":0.35299999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.71699999999999997,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.84699999999999998}]},{"osd":1,"last update":"Thu Oct 19 13:02:33 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.56200000000000006,"15min":0.59399999999999997},"min":{"1min":0.5,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.754,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.627,"5min":0.56699999999999995,"15min":0.59699999999999998},"min":{"1min":0.42999999999999999,"5min":0.307,"15min":0.307},"max":{"1min":0.748,"5min":0.78200000000000003,"15min":0.82099999999999995},"last":0.63900000000000001}]},{"osd":2,"last update":"Thu Oct 19 13:02:33 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.54400000000000004,"15min":0.57899999999999996},"min":{"1min":0.45200000000000001,"5min":0.307,"15min":0.307},"max":{"1min":0.76100000000000001,"5min":0.84699999999999998,"15min":0.85699999999999998},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.58699999999999997,"15min":0.60799999999999998},"min":{"1min":0.50800000000000001,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.71899999999999997,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.74199999999999999}]},{"osd":3,"last update":"Thu Oct 19 13:02:33 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.58699999999999997,"15min":0.624},"min":{"1min":0.47599999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.81000000000000005,"5min":0.871,"15min":0.871},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.59099999999999997,"15min":0.62},"min":{"1min":0.53800000000000003,"5min":0.375,"15min":0.36799999999999999},"max":{"1min":0.81699999999999995,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.82799999999999996}]},{"osd":4,"last update":"Thu Oct 19 13:02:33 2023","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.47499999999999998,"15min":0.46600000000000003},"min":{"1min":0.32600000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.72599999999999998,"5min":1.353,"15min":1.353},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.47599999999999998,"15min":0.47799999999999998},"min":{"1min":0.35199999999999998,"5min":0.254,"15min":0.254},"max":{"1min":0.73599999999999999,"5min":0.73799999999999999,"15min":0.77600000000000002},"last":0.81399999999999995}]},{"osd":5,"last update":"Thu Oct 19 13:02:33 2023","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.45600000000000002,"15min":0.46999999999999997},"min":{"1min":0.36899999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.58099999999999996,"5min":0.69099999999999995,"15min":0.77300000000000002},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52400000000000002,"15min":0.51400000000000001},"min":{"1min":0.373,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.75,"5min":0.75700000000000001,"15min":0.85699999999999998},"last":0.70899999999999996}]},{"osd":7,"last update":"Thu Oct 19 13:02:33 2023","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.52500000000000002,"15min":0.51900000000000002},"min":{"1min":0.39500000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.71999999999999997,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.39300000000000002,"5min":0.27900000000000003,"15min":0.26000000000000001},"max":{"1min":0.67700000000000005,"5min":1.127,"15min":1.127},"last":0.754}]}]},{"osd":5,"up_from":13,"seq":55834575077,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50792,"kb_used_data":1560,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321048,"statfs":{"total":96636764160,"available":96584753152,"internally_reserved":0,"allocated":1597440,"data_stored":805529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 13:02:32 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.55900000000000005,"15min":0.59099999999999997},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.75800000000000001,"5min":0.79900000000000004,"15min":0.82699999999999996},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.57399999999999995,"15min":0.59499999999999997},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.27800000000000002},"max":{"1min":0.81499999999999995,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.57699999999999996}]},{"osd":1,"last update":"Thu Oct 19 13:02:32 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.57099999999999995,"15min":0.59599999999999997},"min":{"1min":0.41799999999999998,"5min":0.34599999999999997,"15min":0.32800000000000001},"max":{"1min":0.80300000000000005,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.59999999999999998,"15min":0.61199999999999999},"min":{"1min":0.39900000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.497}]},{"osd":2,"last update":"Thu Oct 19 13:02:32 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.54900000000000004,"15min":0.58299999999999996},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.307},"max":{"1min":0.76900000000000002,"5min":0.77500000000000002,"15min":0.80700000000000005},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.60199999999999998,"15min":0.62},"min":{"1min":0.37,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.83999999999999997,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.68899999999999995}]},{"osd":3,"last update":"Thu Oct 19 13:02:32 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.57899999999999996,"15min":0.60399999999999998},"min":{"1min":0.38900000000000001,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.60399999999999998,"15min":0.63200000000000001},"min":{"1min":0.40899999999999997,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.70599999999999996}]},{"osd":4,"last update":"Thu Oct 19 13:02:32 2023","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.47199999999999998,"15min":0.47099999999999997},"min":{"1min":0.35299999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.67400000000000004,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.745},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.47199999999999998,"15min":0.47199999999999998},"min":{"1min":0.26100000000000001,"5min":0.251,"15min":0.251},"max":{"1min":0.69699999999999995,"5min":0.73299999999999998,"15min":0.79400000000000004},"last":0.63}]},{"osd":6,"last update":"Thu Oct 19 13:02:32 2023","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.30199999999999999,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.50800000000000001,"15min":0.503},"min":{"1min":0.29599999999999999,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.75700000000000001,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.80800000000000005}]},{"osd":7,"last update":"Thu Oct 19 13:02:32 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.503,"15min":0.49199999999999999},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.78600000000000003,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.55300000000000005,"15min":0.53700000000000003},"min":{"1min":0.375,"5min":0.311,"15min":0.311},"max":{"1min":0.76800000000000002,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.78000000000000003}]}]},{"osd":4,"up_from":13,"seq":55834575077,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32764,"kb_used_data":1580,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339076,"statfs":{"total":96636764160,"available":96603213824,"internally_reserved":0,"allocated":1617920,"data_stored":812269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 13:02:45 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.52800000000000002,"15min":0.55300000000000005},"min":{"1min":0.40799999999999997,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.76100000000000001,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.53100000000000003,"15min":0.55500000000000005},"min":{"1min":0.44800000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.502}]},{"osd":1,"last update":"Thu Oct 19 13:02:45 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.55700000000000005,"15min":0.60199999999999998},"min":{"1min":0.46999999999999997,"5min":0.317,"15min":0.317},"max":{"1min":0.80600000000000005,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.624,"5min":0.58299999999999996,"15min":0.61299999999999999},"min":{"1min":0.44700000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.79800000000000004,"5min":0.79900000000000004,"15min":0.86899999999999999},"last":0.441}]},{"osd":2,"last update":"Thu Oct 19 13:02:45 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.57299999999999995,"15min":0.60699999999999998},"min":{"1min":0.39900000000000002,"5min":0.28399999999999997,"15min":0.254},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.86599999999999999},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.58299999999999996,"15min":0.61099999999999999},"min":{"1min":0.33300000000000002,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.82099999999999995,"5min":0.83799999999999997,"15min":0.85999999999999999},"last":0.69599999999999995}]},{"osd":3,"last update":"Thu Oct 19 13:02:45 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.55500000000000005,"15min":0.57799999999999996},"min":{"1min":0.36499999999999999,"5min":0.33200000000000002,"15min":0.32400000000000001},"max":{"1min":0.72599999999999998,"5min":0.75,"15min":0.79700000000000004},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.60599999999999998,"15min":0.63700000000000001},"min":{"1min":0.44500000000000001,"5min":0.42099999999999999,"15min":0.41299999999999998},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.83899999999999997},"last":0.749}]},{"osd":5,"last update":"Thu Oct 19 13:02:45 2023","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.44500000000000001,"15min":0.44400000000000001},"min":{"1min":0.30399999999999999,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.64200000000000002,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.46500000000000002,"15min":0.46100000000000002},"min":{"1min":0.34699999999999998,"5min":0.30099999999999999,"15min":0.23499999999999999},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.84999999999999998},"last":0.86899999999999999}]},{"osd":6,"last update":"Thu Oct 19 13:02:45 2023","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.45200000000000001,"15min":0.45500000000000002},"min":{"1min":0.32600000000000001,"5min":0.28499999999999998,"15min":0.26400000000000001},"max":{"1min":0.70299999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.48899999999999999,"15min":0.49199999999999999},"min":{"1min":0.35999999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.73999999999999999,"5min":0.78000000000000003,"15min":0.82499999999999996},"last":0.84099999999999997}]},{"osd":7,"last update":"Thu Oct 19 13:02:45 2023","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.5,"15min":0.5},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.27500000000000002},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.80200000000000005},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.41999999999999998,"5min":0.33100000000000002,"15min":0.30199999999999999},"max":{"1min":0.754,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.85699999999999998}]}]},{"osd":3,"up_from":13,"seq":55834575076,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41072,"kb_used_data":1564,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330768,"statfs":{"total":96636764160,"available":96594706432,"internally_reserved":0,"allocated":1601536,"data_stored":806111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":40434120},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 13:02:29 2023","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.40799999999999997,"15min":0.40200000000000002},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.59199999999999997,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.39700000000000002,"15min":0.41499999999999998},"min":{"1min":0.23300000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.50700000000000001,"5min":0.75,"15min":0.75},"last":0.27600000000000002}]},{"osd":1,"last update":"Thu Oct 19 13:02:29 2023","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.40400000000000003,"15min":0.40799999999999997},"min":{"1min":0.27200000000000002,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.61699999999999999,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.46300000000000002,"15min":0.45800000000000002},"min":{"1min":0.32600000000000001,"5min":0.221,"15min":0.221},"max":{"1min":0.53600000000000003,"5min":2.9399999999999999,"15min":2.9399999999999999},"last":0.313}]},{"osd":2,"last update":"Thu Oct 19 13:02:29 2023","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.45300000000000001,"15min":0.46400000000000002},"min":{"1min":0.309,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.56499999999999995,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.47299999999999998,"15min":0.48699999999999999},"min":{"1min":0.36199999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.64400000000000002,"5min":0.754,"15min":0.754},"last":0.45600000000000002}]},{"osd":4,"last update":"Thu Oct 19 13:02:29 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.64000000000000001,"15min":0.66100000000000003},"min":{"1min":0.38100000000000001,"5min":0.371,"15min":0.371},"max":{"1min":0.88800000000000001,"5min":0.92500000000000004,"15min":1.0069999999999999},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.67000000000000004,"15min":0.68799999999999994},"min":{"1min":0.44900000000000001,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.99399999999999999},"last":0.65600000000000003}]},{"osd":5,"last update":"Thu Oct 19 13:02:29 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.67800000000000005,"15min":0.66000000000000003},"min":{"1min":0.48899999999999999,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.96699999999999997,"5min":2.9990000000000001,"15min":2.9990000000000001},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63300000000000001,"15min":0.64800000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.35899999999999999},"max":{"1min":0.83699999999999997,"5min":1.774,"15min":1.774},"last":0.47099999999999997}]},{"osd":6,"last update":"Thu Oct 19 13:02:29 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.64200000000000002,"15min":0.65100000000000002},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.92400000000000004,"5min":0.93700000000000006,"15min":0.96799999999999997},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.48499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.90400000000000003,"5min":1.167,"15min":1.167},"last":0.48899999999999999}]},{"osd":7,"last update":"Thu Oct 19 13:02:29 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64300000000000002,"15min":0.66100000000000003},"min":{"1min":0.42899999999999999,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.94199999999999995,"5min":1.042,"15min":1.042},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.68500000000000005,"15min":0.67900000000000005},"min":{"1min":0.44800000000000001,"5min":0.441,"15min":0.432},"max":{"1min":0.86099999999999999,"5min":2.8149999999999999,"15min":2.8149999999999999},"last":0.56599999999999995}]}]},{"osd":2,"up_from":13,"seq":55834575078,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54968,"kb_used_data":1576,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316872,"statfs":{"total":96636764160,"available":96580476928,"internally_reserved":0,"allocated":1613824,"data_stored":812211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 13:02:44 2023","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.36499999999999999,"15min":0.379},"min":{"1min":0.27700000000000002,"5min":0.20699999999999999,"15min":0.19700000000000001},"max":{"1min":0.496,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.39700000000000002,"15min":0.41199999999999998},"min":{"1min":0.25700000000000001,"5min":0.222,"15min":0.20999999999999999},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.74099999999999999},"last":0.373}]},{"osd":1,"last update":"Thu Oct 19 13:02:44 2023","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.373,"15min":0.39700000000000002},"min":{"1min":0.27800000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.71099999999999997},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.40400000000000003,"15min":0.42799999999999999},"min":{"1min":0.28000000000000003,"5min":0.22,"15min":0.22},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.68100000000000005},"last":0.34999999999999998}]},{"osd":3,"last update":"Thu Oct 19 13:02:44 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.44,"15min":0.45500000000000002},"min":{"1min":0.33400000000000002,"5min":0.29199999999999998,"15min":0.22800000000000001},"max":{"1min":0.69299999999999995,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.46200000000000002,"15min":0.48299999999999998},"min":{"1min":0.40400000000000003,"5min":0.32200000000000001,"15min":0.29099999999999998},"max":{"1min":0.72599999999999998,"5min":0.77800000000000002,"15min":0.82999999999999996},"last":0.42799999999999999}]},{"osd":4,"last update":"Thu Oct 19 13:02:44 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.60699999999999998,"15min":0.64000000000000001},"min":{"1min":0.501,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.81599999999999995,"5min":0.89500000000000002,"15min":0.98999999999999999},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.61099999999999999,"15min":0.65200000000000002},"min":{"1min":0.48899999999999999,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.78400000000000003,"5min":0.83399999999999996,"15min":1.046},"last":0.68500000000000005}]},{"osd":5,"last update":"Thu Oct 19 13:02:44 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.621,"15min":0.65700000000000003},"min":{"1min":0.46800000000000003,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.99399999999999999},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.61799999999999999,"15min":0.65400000000000003},"min":{"1min":0.51300000000000001,"5min":0.375,"15min":0.375},"max":{"1min":0.86299999999999999,"5min":0.94199999999999995,"15min":0.95799999999999996},"last":0.67100000000000004}]},{"osd":6,"last update":"Thu Oct 19 13:02:44 2023","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.627,"15min":0.65900000000000003},"min":{"1min":0.45200000000000001,"5min":0.40400000000000003,"15min":0.40300000000000002},"max":{"1min":0.83699999999999997,"5min":0.89800000000000002,"15min":1.008},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.64000000000000001,"15min":0.67500000000000004},"min":{"1min":0.54000000000000004,"5min":0.38800000000000001,"15min":0.371},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.995},"last":0.61899999999999999}]},{"osd":7,"last update":"Thu Oct 19 13:02:44 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.64300000000000002,"15min":0.67700000000000005},"min":{"1min":0.49399999999999999,"5min":0.39800000000000002,"15min":0.36399999999999999},"max":{"1min":0.84699999999999998,"5min":0.96299999999999997,"15min":1.0149999999999999},"last":0.629},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.65000000000000002,"15min":0.68000000000000005},"min":{"1min":0.438,"5min":0.438,"15min":0.38200000000000001},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":1.0329999999999999},"last":0.69299999999999995}]}]},{"osd":1,"up_from":13,"seq":55834575077,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36904,"kb_used_data":1564,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334936,"statfs":{"total":96636764160,"available":96598974464,"internally_reserved":0,"allocated":1601536,"data_stored":805563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 13:02:41 2023","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.376,"15min":0.38700000000000001},"min":{"1min":0.29399999999999998,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.56599999999999995,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.308},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.432,"15min":0.437},"min":{"1min":0.32000000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.76100000000000001,"5min":2.5800000000000001,"15min":2.5800000000000001},"last":0.23999999999999999}]},{"osd":2,"last update":"Thu Oct 19 13:02:41 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.40500000000000003,"15min":0.42599999999999999},"min":{"1min":0.315,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.64300000000000002,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.46800000000000003,"15min":0.46200000000000002},"min":{"1min":0.34599999999999997,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.79800000000000004,"5min":2.6230000000000002,"15min":2.6230000000000002},"last":0.29299999999999998}]},{"osd":3,"last update":"Thu Oct 19 13:02:41 2023","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.498,"15min":0.49199999999999999},"min":{"1min":0.375,"5min":0.246,"15min":0.246},"max":{"1min":0.71999999999999997,"5min":3.9740000000000002,"15min":3.9740000000000002},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.42099999999999999,"15min":0.42799999999999999},"min":{"1min":0.33100000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.624,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.32700000000000001}]},{"osd":4,"last update":"Thu Oct 19 13:02:41 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.61899999999999999,"15min":0.65000000000000002},"min":{"1min":0.46700000000000003,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.80200000000000005,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.61799999999999999,"15min":0.64000000000000001},"min":{"1min":0.434,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.81999999999999995,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.53500000000000003}]},{"osd":5,"last update":"Thu Oct 19 13:02:41 2023","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.63700000000000001,"15min":0.65200000000000002},"min":{"1min":0.50900000000000001,"5min":0.38700000000000001,"15min":0.38100000000000001},"max":{"1min":0.86299999999999999,"5min":1.1200000000000001,"15min":1.1200000000000001},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.66300000000000003,"15min":0.67400000000000004},"min":{"1min":0.50900000000000001,"5min":0.374,"15min":0.374},"max":{"1min":0.873,"5min":2.7120000000000002,"15min":2.7120000000000002},"last":0.66700000000000004}]},{"osd":6,"last update":"Thu Oct 19 13:02:41 2023","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.623,"15min":0.65300000000000002},"min":{"1min":0.43099999999999999,"5min":0.378,"15min":0.36799999999999999},"max":{"1min":0.83899999999999997,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.64600000000000002,"15min":0.67300000000000004},"min":{"1min":0.46700000000000003,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.84599999999999997,"5min":1.129,"15min":1.129},"last":0.68400000000000005}]},{"osd":7,"last update":"Thu Oct 19 13:02:41 2023","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.67100000000000004,"15min":0.68000000000000005},"min":{"1min":0.56799999999999995,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.91200000000000003,"5min":2.786,"15min":2.786},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.64100000000000001,"15min":0.67300000000000004},"min":{"1min":0.50900000000000001,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.88300000000000001,"5min":0.95399999999999996,"15min":1.034},"last":0.64800000000000002}]}]},{"osd":0,"up_from":13,"seq":55834575078,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33336,"kb_used_data":2152,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338504,"statfs":{"total":96636764160,"available":96602628096,"internally_reserved":0,"allocated":2203648,"data_stored":1396980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Oct 19 13:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.38200000000000001,"15min":0.39000000000000001},"min":{"1min":0.218,"5min":0.182,"15min":0.182},"max":{"1min":0.624,"5min":0.70199999999999996,"15min":0.81799999999999995},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.40500000000000003,"15min":0.42999999999999999},"min":{"1min":0.23300000000000001,"5min":0.22500000000000001,"15min":0.20200000000000001},"max":{"1min":0.64800000000000002,"5min":0.67600000000000005,"15min":0.746},"last":0.439}]},{"osd":2,"last update":"Thu Oct 19 13:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.42699999999999999,"15min":0.45200000000000001},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.26200000000000001},"max":{"1min":0.629,"5min":0.67900000000000005,"15min":0.76300000000000001},"last":0.498},{"interface":"front","average":{"1min":0.437,"5min":0.41499999999999998,"15min":0.434},"min":{"1min":0.254,"5min":0.246,"15min":0.20699999999999999},"max":{"1min":0.68500000000000005,"5min":0.72999999999999998,"15min":0.76800000000000002},"last":0.51000000000000001}]},{"osd":3,"last update":"Thu Oct 19 13:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.39700000000000002,"15min":0.40899999999999997},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.60399999999999998,"5min":0.65600000000000003,"15min":1.2829999999999999},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.45300000000000001,"15min":0.48899999999999999},"min":{"1min":0.29999999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":1.4990000000000001},"last":0.52400000000000002}]},{"osd":4,"last update":"Thu Oct 19 13:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.59899999999999998,"15min":0.63},"min":{"1min":0.47999999999999998,"5min":0.40000000000000002,"15min":0.34999999999999998},"max":{"1min":0.79400000000000004,"5min":0.98499999999999999,"15min":1.361},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.64200000000000002,"15min":0.65200000000000002},"min":{"1min":0.48199999999999998,"5min":0.36399999999999999,"15min":0.35999999999999999},"max":{"1min":0.89600000000000002,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.69999999999999996}]},{"osd":5,"last update":"Thu Oct 19 13:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.66900000000000004,"15min":0.68400000000000005},"min":{"1min":0.54800000000000004,"5min":0.378,"15min":0.378},"max":{"1min":0.879,"5min":1.004,"15min":1.325},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.66700000000000004,"15min":0.68400000000000005},"min":{"1min":0.53000000000000003,"5min":0.40100000000000002,"15min":0.318},"max":{"1min":0.90900000000000003,"5min":1.024,"15min":1.024},"last":0.65100000000000002}]},{"osd":6,"last update":"Thu Oct 19 13:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.63600000000000001,"15min":0.66100000000000003},"min":{"1min":0.51000000000000001,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.82099999999999995,"5min":0.88800000000000001,"15min":0.98099999999999998},"last":0.746},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.64100000000000001,"15min":0.66100000000000003},"min":{"1min":0.52400000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.88700000000000001,"5min":0.96599999999999997,"15min":0.995},"last":0.67500000000000004}]},{"osd":7,"last update":"Thu Oct 19 13:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.63300000000000001,"15min":0.67600000000000005},"min":{"1min":0.53400000000000003,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.83899999999999997,"5min":0.98799999999999999,"15min":1.0089999999999999},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.65000000000000002,"15min":0.66200000000000003},"min":{"1min":0.57399999999999995,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.91000000000000003,"5min":0.94499999999999995,"15min":1.389},"last":0.76100000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-19T13:03:01.701 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-19T13:03:01.702 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-19T13:03:01.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.849+0000 7f85f8c0e640 1 -- 172.21.15.57:0/3004261242 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85f40abc10 msgr2=0x7f85f40c9b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:03:01.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.849+0000 7f85f8c0e640 1 --2- 172.21.15.57:0/3004261242 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85f40abc10 0x7f85f40c9b30 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7f85ec009be0 tx=0x7f85ec02c0c0 comp rx=0 tx=0).stop 2023-10-19T13:03:01.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.849+0000 7f85f8c0e640 1 -- 172.21.15.57:0/3004261242 shutdown_connections 2023-10-19T13:03:01.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.849+0000 7f85f8c0e640 1 --2- 172.21.15.57:0/3004261242 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85f40abc10 0x7f85f40c9b30 unknown :-1 s=CLOSED pgs=1454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:01.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.849+0000 7f85f8c0e640 1 --2- 172.21.15.57:0/3004261242 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f85f406af00 0x7f85f406b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:01.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.849+0000 7f85f8c0e640 1 -- 172.21.15.57:0/3004261242 >> 172.21.15.57:0/3004261242 conn(0x7f85f4069930 msgr2=0x7f85f4069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:03:01.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.849+0000 7f85f8c0e640 1 -- 172.21.15.57:0/3004261242 shutdown_connections 2023-10-19T13:03:01.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.850+0000 7f85f8c0e640 1 -- 172.21.15.57:0/3004261242 wait complete. 2023-10-19T13:03:01.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.851+0000 7f85f8c0e640 1 Processor -- start 2023-10-19T13:03:01.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.851+0000 7f85f8c0e640 1 -- start start 2023-10-19T13:03:01.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.852+0000 7f85f8c0e640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f85f406af00 0x7f85f41f15a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:03:01.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.852+0000 7f85f8c0e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85f40abc10 0x7f85f413b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:03:01.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.852+0000 7f85f8c0e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f85f40c96b0 0x7f85f413ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:03:01.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.852+0000 7f85f8c0e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85f41f1ae0 con 0x7f85f40abc10 2023-10-19T13:03:01.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.852+0000 7f85f8c0e640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85f413b880 con 0x7f85f40c96b0 2023-10-19T13:03:01.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.852+0000 7f85f8c0e640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85f41f2570 con 0x7f85f406af00 2023-10-19T13:03:01.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.852+0000 7f85f1d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85f40abc10 0x7f85f413b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:03:01.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.852+0000 7f85f1d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85f40abc10 0x7f85f413b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:48648/0 (socket says 172.21.15.57:48648) 2023-10-19T13:03:01.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.852+0000 7f85f1d9b640 1 -- 172.21.15.57:0/3580927410 learned_addr learned my addr 172.21.15.57:0/3580927410 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:03:01.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.852+0000 7f85f259c640 1 --2- 172.21.15.57:0/3580927410 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f85f406af00 0x7f85f41f15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:03:01.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.852+0000 7f85f2d9d640 1 --2- 172.21.15.57:0/3580927410 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f85f40c96b0 0x7f85f413ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:03:01.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.852+0000 7f85f1d9b640 1 -- 172.21.15.57:0/3580927410 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f85f406af00 msgr2=0x7f85f41f15a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:03:01.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.852+0000 7f85f1d9b640 1 --2- 172.21.15.57:0/3580927410 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f85f406af00 0x7f85f41f15a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:01.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.852+0000 7f85f1d9b640 1 -- 172.21.15.57:0/3580927410 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f85f40c96b0 msgr2=0x7f85f413ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:03:01.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.852+0000 7f85f1d9b640 1 --2- 172.21.15.57:0/3580927410 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f85f40c96b0 0x7f85f413ca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:01.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.852+0000 7f85f1d9b640 1 -- 172.21.15.57:0/3580927410 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85ec009840 con 0x7f85f40abc10 2023-10-19T13:03:01.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.852+0000 7f85f1d9b640 1 --2- 172.21.15.57:0/3580927410 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85f40abc10 0x7f85f413b340 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f85e400c3c0 tx=0x7f85e400c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:03:01.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.852+0000 7f85eaffd640 1 -- 172.21.15.57:0/3580927410 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85e40035f0 con 0x7f85f40abc10 2023-10-19T13:03:01.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.852+0000 7f85eaffd640 1 -- 172.21.15.57:0/3580927410 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f85e4003770 con 0x7f85f40abc10 2023-10-19T13:03:01.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.852+0000 7f85eaffd640 1 -- 172.21.15.57:0/3580927410 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85e4003b80 con 0x7f85f40abc10 2023-10-19T13:03:01.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.852+0000 7f85f8c0e640 1 -- 172.21.15.57:0/3580927410 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85f41f1cd0 con 0x7f85f40abc10 2023-10-19T13:03:01.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.853+0000 7f85f8c0e640 1 -- 172.21.15.57:0/3580927410 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85f41f2290 con 0x7f85f40abc10 2023-10-19T13:03:01.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.854+0000 7f85f8c0e640 1 -- 172.21.15.57:0/3580927410 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85f40690e0 con 0x7f85f40abc10 2023-10-19T13:03:01.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.854+0000 7f85eaffd640 1 -- 172.21.15.57:0/3580927410 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 58) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f85e4003ce0 con 0x7f85f40abc10 2023-10-19T13:03:01.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.854+0000 7f85eaffd640 1 --2- 172.21.15.57:0/3580927410 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f85d406af60 0x7f85d406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:03:01.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.854+0000 7f85eaffd640 1 -- 172.21.15.57:0/3580927410 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f85e4089050 con 0x7f85f40abc10 2023-10-19T13:03:01.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.855+0000 7f85f259c640 1 --2- 172.21.15.57:0/3580927410 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f85d406af60 0x7f85d406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:03:01.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.855+0000 7f85f259c640 1 --2- 172.21.15.57:0/3580927410 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f85d406af60 0x7f85d406d420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f85dc009990 tx=0x7f85dc009110 comp rx=0 tx=0).ready entity=mgr.7540 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:03:01.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:01.858+0000 7f85eaffd640 1 -- 172.21.15.57:0/3580927410 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f85e40bbca0 con 0x7f85f40abc10 2023-10-19T13:03:02.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.107+0000 7f85f8c0e640 1 -- 172.21.15.57:0/3580927410 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f85f40cc420 con 0x7f85f40abc10 2023-10-19T13:03:02.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.109+0000 7f85eaffd640 1 -- 172.21.15.57:0/3580927410 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v27) v1 ==== 190+0+147 (secure 0 0 0) 0x7f85e40170e0 con 0x7f85f40abc10 2023-10-19T13:03:02.112 INFO:teuthology.orchestra.run.smithi057.stdout:[client.0] 2023-10-19T13:03:02.112 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQBkJDFlg9yuExAATxiWDfS+p07G1F0pORbJEg== 2023-10-19T13:03:02.113 INFO:teuthology.orchestra.run.smithi057.stdout: caps mds = "allow" 2023-10-19T13:03:02.113 INFO:teuthology.orchestra.run.smithi057.stdout: caps mon = "allow r" 2023-10-19T13:03:02.113 INFO:teuthology.orchestra.run.smithi057.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-19T13:03:02.113 INFO:teuthology.orchestra.run.smithi057.stderr:updated caps for client.0 2023-10-19T13:03:02.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.111+0000 7f85f8c0e640 1 -- 172.21.15.57:0/3580927410 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f85d406af60 msgr2=0x7f85d406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:03:02.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.111+0000 7f85f8c0e640 1 --2- 172.21.15.57:0/3580927410 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f85d406af60 0x7f85d406d420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f85dc009990 tx=0x7f85dc009110 comp rx=0 tx=0).stop 2023-10-19T13:03:02.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.111+0000 7f85f8c0e640 1 -- 172.21.15.57:0/3580927410 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85f40abc10 msgr2=0x7f85f413b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:03:02.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.111+0000 7f85f8c0e640 1 --2- 172.21.15.57:0/3580927410 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85f40abc10 0x7f85f413b340 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f85e400c3c0 tx=0x7f85e400c7c0 comp rx=0 tx=0).stop 2023-10-19T13:03:02.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.112+0000 7f85f8c0e640 1 -- 172.21.15.57:0/3580927410 shutdown_connections 2023-10-19T13:03:02.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.112+0000 7f85f8c0e640 1 --2- 172.21.15.57:0/3580927410 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f85d406af60 0x7f85d406d420 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:02.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.112+0000 7f85f8c0e640 1 --2- 172.21.15.57:0/3580927410 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f85f40c96b0 0x7f85f413ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:02.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.112+0000 7f85f8c0e640 1 --2- 172.21.15.57:0/3580927410 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85f40abc10 0x7f85f413b340 unknown :-1 s=CLOSED pgs=1455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:02.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.112+0000 7f85f8c0e640 1 --2- 172.21.15.57:0/3580927410 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f85f406af00 0x7f85f41f15a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:02.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.112+0000 7f85f8c0e640 1 -- 172.21.15.57:0/3580927410 >> 172.21.15.57:0/3580927410 conn(0x7f85f4069930 msgr2=0x7f85f4069d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:03:02.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.112+0000 7f85f8c0e640 1 -- 172.21.15.57:0/3580927410 shutdown_connections 2023-10-19T13:03:02.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.112+0000 7f85f8c0e640 1 -- 172.21.15.57:0/3580927410 wait complete. 2023-10-19T13:03:02.125 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T13:03:02.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.231+0000 7f7afe5e2640 1 -- 172.21.15.57:0/3232233824 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ae00052a0 msgr2=0x7f7ae000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:03:02.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.231+0000 7f7afe5e2640 1 --2- 172.21.15.57:0/3232233824 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ae00052a0 0x7f7ae000b980 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7f7af0001d50 tx=0x7f7ae00057f0 comp rx=0 tx=0).stop 2023-10-19T13:03:02.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.232+0000 7f7afe5e2640 1 -- 172.21.15.57:0/3232233824 shutdown_connections 2023-10-19T13:03:02.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.232+0000 7f7afe5e2640 1 --2- 172.21.15.57:0/3232233824 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ae00052a0 0x7f7ae000b980 unknown :-1 s=CLOSED pgs=1457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:02.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.232+0000 7f7afe5e2640 1 -- 172.21.15.57:0/3232233824 >> 172.21.15.57:0/3232233824 conn(0x7f7af80aec40 msgr2=0x7f7af80af050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:03:02.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.232+0000 7f7afe5e2640 1 -- 172.21.15.57:0/3232233824 shutdown_connections 2023-10-19T13:03:02.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.232+0000 7f7afe5e2640 1 -- 172.21.15.57:0/3232233824 wait complete. 2023-10-19T13:03:02.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.233+0000 7f7afe5e2640 1 Processor -- start 2023-10-19T13:03:02.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.233+0000 7f7afe5e2640 1 -- start start 2023-10-19T13:03:02.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.234+0000 7f7afe5e2640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7ae00052a0 0x7f7af815a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:03:02.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.234+0000 7f7afe5e2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7af8145060 0x7f7af81454e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:03:02.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.234+0000 7f7afe5e2640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7af81540c0 0x7f7af8154570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:03:02.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.234+0000 7f7afe5e2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7af8154ab0 con 0x7f7af8145060 2023-10-19T13:03:02.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.234+0000 7f7afe5e2640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7af81ecc40 con 0x7f7af81540c0 2023-10-19T13:03:02.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.234+0000 7f7afe5e2640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7af81eceb0 con 0x7f7ae00052a0 2023-10-19T13:03:02.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.234+0000 7f7af77fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7af8145060 0x7f7af81454e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:03:02.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.234+0000 7f7af77fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7af8145060 0x7f7af81454e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:48662/0 (socket says 172.21.15.57:48662) 2023-10-19T13:03:02.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.234+0000 7f7af77fe640 1 -- 172.21.15.57:0/3187617091 learned_addr learned my addr 172.21.15.57:0/3187617091 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:03:02.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.234+0000 7f7afcb7f640 1 --2- 172.21.15.57:0/3187617091 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7af81540c0 0x7f7af8154570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:03:02.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.234+0000 7f7af7fff640 1 --2- 172.21.15.57:0/3187617091 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7ae00052a0 0x7f7af815a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:03:02.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.234+0000 7f7af77fe640 1 -- 172.21.15.57:0/3187617091 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7ae00052a0 msgr2=0x7f7af815a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:03:02.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.234+0000 7f7af77fe640 1 --2- 172.21.15.57:0/3187617091 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7ae00052a0 0x7f7af815a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:02.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.234+0000 7f7af77fe640 1 -- 172.21.15.57:0/3187617091 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7af81540c0 msgr2=0x7f7af8154570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:03:02.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.234+0000 7f7af77fe640 1 --2- 172.21.15.57:0/3187617091 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7af81540c0 0x7f7af8154570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:02.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.234+0000 7f7af77fe640 1 -- 172.21.15.57:0/3187617091 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7af81edc30 con 0x7f7af8145060 2023-10-19T13:03:02.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.234+0000 7f7af7fff640 1 --2- 172.21.15.57:0/3187617091 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7ae00052a0 0x7f7af815a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:03:02.243 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.234+0000 7f7afcb7f640 1 --2- 172.21.15.57:0/3187617091 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7af81540c0 0x7f7af8154570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:03:02.243 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.234+0000 7f7af77fe640 1 --2- 172.21.15.57:0/3187617091 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7af8145060 0x7f7af81454e0 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7f7aec011800 tx=0x7f7aec011cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:03:02.243 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.234+0000 7f7af4ff9640 1 -- 172.21.15.57:0/3187617091 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aec004460 con 0x7f7af8145060 2023-10-19T13:03:02.243 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.234+0000 7f7af4ff9640 1 -- 172.21.15.57:0/3187617091 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f7aec00f940 con 0x7f7af8145060 2023-10-19T13:03:02.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.234+0000 7f7afe5e2640 1 -- 172.21.15.57:0/3187617091 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7af8145b60 con 0x7f7af8145060 2023-10-19T13:03:02.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.234+0000 7f7af4ff9640 1 -- 172.21.15.57:0/3187617091 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aec00fcd0 con 0x7f7af8145060 2023-10-19T13:03:02.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.234+0000 7f7afe5e2640 1 -- 172.21.15.57:0/3187617091 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7af81ed430 con 0x7f7af8145060 2023-10-19T13:03:02.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.235+0000 7f7afe5e2640 1 -- 172.21.15.57:0/3187617091 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7af80690e0 con 0x7f7af8145060 2023-10-19T13:03:02.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.236+0000 7f7af4ff9640 1 -- 172.21.15.57:0/3187617091 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 58) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f7aec018070 con 0x7f7af8145060 2023-10-19T13:03:02.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.236+0000 7f7af4ff9640 1 --2- 172.21.15.57:0/3187617091 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f7ae406af60 0x7f7ae406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:03:02.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.236+0000 7f7af4ff9640 1 -- 172.21.15.57:0/3187617091 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f7aec08a6a0 con 0x7f7af8145060 2023-10-19T13:03:02.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.237+0000 7f7af7fff640 1 --2- 172.21.15.57:0/3187617091 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f7ae406af60 0x7f7ae406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:03:02.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.237+0000 7f7af7fff640 1 --2- 172.21.15.57:0/3187617091 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f7ae406af60 0x7f7ae406d420 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7ae80017a0 tx=0x7f7ae80010b0 comp rx=0 tx=0).ready entity=mgr.7540 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:03:02.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.240+0000 7f7af4ff9640 1 -- 172.21.15.57:0/3187617091 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7aec0554d0 con 0x7f7af8145060 2023-10-19T13:03:02.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.452+0000 7f7afe5e2640 1 -- 172.21.15.57:0/3187617091 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7af80b0870 con 0x7f7af8145060 2023-10-19T13:03:02.456 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.453+0000 7f7af4ff9640 1 -- 172.21.15.57:0/3187617091 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4912 (secure 0 0 0) 0x7f7aec058af0 con 0x7f7af8145060 2023-10-19T13:03:02.456 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:03:02.457 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":50,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/2033977424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2033977424},{"type":"v1","addr":"172.21.15.179:6836","nonce":2033977424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":8568,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/981095684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":981095684},{"type":"v1","addr":"172.21.15.57:6834","nonce":981095684}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":8571,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/383147069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":383147069},{"type":"v1","addr":"172.21.15.57:6835","nonce":383147069}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T13:02:57.697820+0000","modified":"2023-10-19T13:02:59.684888+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8577},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8577":{"gid":8577,"name":"d","rank":0,"incarnation":48,"state":"up:active","state_seq":3,"addr":"172.21.15.179:6837/768231146","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":768231146},{"type":"v1","addr":"172.21.15.179:6837","nonce":768231146}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2023-10-19T13:03:02.458 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 50 2023-10-19T13:03:02.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.455+0000 7f7afe5e2640 1 -- 172.21.15.57:0/3187617091 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f7ae406af60 msgr2=0x7f7ae406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:03:02.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.455+0000 7f7afe5e2640 1 --2- 172.21.15.57:0/3187617091 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f7ae406af60 0x7f7ae406d420 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7ae80017a0 tx=0x7f7ae80010b0 comp rx=0 tx=0).stop 2023-10-19T13:03:02.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.455+0000 7f7afe5e2640 1 -- 172.21.15.57:0/3187617091 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7af8145060 msgr2=0x7f7af81454e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:03:02.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.455+0000 7f7afe5e2640 1 --2- 172.21.15.57:0/3187617091 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7af8145060 0x7f7af81454e0 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7f7aec011800 tx=0x7f7aec011cc0 comp rx=0 tx=0).stop 2023-10-19T13:03:02.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.455+0000 7f7afe5e2640 1 -- 172.21.15.57:0/3187617091 shutdown_connections 2023-10-19T13:03:02.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.455+0000 7f7afe5e2640 1 --2- 172.21.15.57:0/3187617091 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f7ae406af60 0x7f7ae406d420 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:02.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.455+0000 7f7afe5e2640 1 --2- 172.21.15.57:0/3187617091 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7af81540c0 0x7f7af8154570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:02.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.455+0000 7f7afe5e2640 1 --2- 172.21.15.57:0/3187617091 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7af8145060 0x7f7af81454e0 unknown :-1 s=CLOSED pgs=1458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:02.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.455+0000 7f7afe5e2640 1 --2- 172.21.15.57:0/3187617091 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7ae00052a0 0x7f7af815a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:02.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.455+0000 7f7afe5e2640 1 -- 172.21.15.57:0/3187617091 >> 172.21.15.57:0/3187617091 conn(0x7f7af80aec40 msgr2=0x7f7af80b0b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:03:02.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.455+0000 7f7afe5e2640 1 -- 172.21.15.57:0/3187617091 shutdown_connections 2023-10-19T13:03:02.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.455+0000 7f7afe5e2640 1 -- 172.21.15.57:0/3187617091 wait complete. 2023-10-19T13:03:02.870 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 50, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-19T13:02:57.697820+0000', 'modified': '2023-10-19T13:02:59.684888+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8577}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8577': {'gid': 8577, 'name': 'd', 'rank': 0, 'incarnation': 48, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.179:6837/768231146', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6835', 'nonce': 768231146}, {'type': 'v1', 'addr': '172.21.15.179:6837', 'nonce': 768231146}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [16], 'metadata_pool': 15, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-19T13:03:02.871 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-19T13:03:02.871 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2023-10-19T13:03:02.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.982+0000 7f5d34739640 1 -- 172.21.15.57:0/781942031 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5d2c070c30 msgr2=0x7f5d2c06bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:03:02.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.982+0000 7f5d34739640 1 --2- 172.21.15.57:0/781942031 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5d2c070c30 0x7f5d2c06bd90 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f5d2c072460 tx=0x7f5d2402c140 comp rx=0 tx=0).stop 2023-10-19T13:03:02.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.982+0000 7f5d34739640 1 -- 172.21.15.57:0/781942031 shutdown_connections 2023-10-19T13:03:02.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.982+0000 7f5d34739640 1 --2- 172.21.15.57:0/781942031 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5d2c070c30 0x7f5d2c06bd90 unknown :-1 s=CLOSED pgs=1459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:02.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.982+0000 7f5d34739640 1 --2- 172.21.15.57:0/781942031 >> v2:172.21.15.179:3300/0 conn(0x7f5d2c157c00 0x7f5d2c0c98b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:02.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.982+0000 7f5d34739640 1 -- 172.21.15.57:0/781942031 >> 172.21.15.57:0/781942031 conn(0x7f5d2c0aec40 msgr2=0x7f5d2c0af050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:03:02.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.982+0000 7f5d34739640 1 -- 172.21.15.57:0/781942031 shutdown_connections 2023-10-19T13:03:02.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.982+0000 7f5d34739640 1 -- 172.21.15.57:0/781942031 wait complete. 2023-10-19T13:03:02.989 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.983+0000 7f5d34739640 1 Processor -- start 2023-10-19T13:03:02.990 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.983+0000 7f5d34739640 1 -- start start 2023-10-19T13:03:02.990 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.984+0000 7f5d34739640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5d2c070c30 0x7f5d2c1483e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:03:02.990 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.984+0000 7f5d34739640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5d2c0cb2f0 0x7f5d2c1e1450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:03:02.990 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.984+0000 7f5d34739640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5d2c157c00 0x7f5d2c158190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:03:02.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.984+0000 7f5d34739640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d2c15afd0 con 0x7f5d2c070c30 2023-10-19T13:03:02.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.984+0000 7f5d34739640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d2c1e1ae0 con 0x7f5d2c157c00 2023-10-19T13:03:02.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.984+0000 7f5d34739640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d2c1e1dc0 con 0x7f5d2c0cb2f0 2023-10-19T13:03:02.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.984+0000 7f5d324d5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5d2c070c30 0x7f5d2c1483e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:03:02.992 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.984+0000 7f5d324d5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5d2c070c30 0x7f5d2c1483e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:48678/0 (socket says 172.21.15.57:48678) 2023-10-19T13:03:02.992 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.984+0000 7f5d324d5640 1 -- 172.21.15.57:0/963628474 learned_addr learned my addr 172.21.15.57:0/963628474 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:03:02.992 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.984+0000 7f5d32cd6640 1 --2- 172.21.15.57:0/963628474 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5d2c157c00 0x7f5d2c158190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:03:02.992 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.984+0000 7f5d31cd4640 1 --2- 172.21.15.57:0/963628474 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5d2c0cb2f0 0x7f5d2c1e1450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:03:02.992 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.984+0000 7f5d324d5640 1 -- 172.21.15.57:0/963628474 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5d2c0cb2f0 msgr2=0x7f5d2c1e1450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:03:02.993 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.984+0000 7f5d324d5640 1 --2- 172.21.15.57:0/963628474 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5d2c0cb2f0 0x7f5d2c1e1450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:02.993 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.984+0000 7f5d324d5640 1 -- 172.21.15.57:0/963628474 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5d2c157c00 msgr2=0x7f5d2c158190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:03:02.993 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.984+0000 7f5d324d5640 1 --2- 172.21.15.57:0/963628474 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5d2c157c00 0x7f5d2c158190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:02.993 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.984+0000 7f5d324d5640 1 -- 172.21.15.57:0/963628474 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d24009900 con 0x7f5d2c070c30 2023-10-19T13:03:02.993 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.984+0000 7f5d31cd4640 1 --2- 172.21.15.57:0/963628474 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5d2c0cb2f0 0x7f5d2c1e1450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T13:03:02.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.984+0000 7f5d324d5640 1 --2- 172.21.15.57:0/963628474 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5d2c070c30 0x7f5d2c1483e0 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f5d2800b410 tx=0x7f5d2800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:03:02.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.984+0000 7f5d22ffd640 1 -- 172.21.15.57:0/963628474 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d28011830 con 0x7f5d2c070c30 2023-10-19T13:03:02.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.984+0000 7f5d22ffd640 1 -- 172.21.15.57:0/963628474 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f5d28011aa0 con 0x7f5d2c070c30 2023-10-19T13:03:02.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.984+0000 7f5d34739640 1 -- 172.21.15.57:0/963628474 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d2c1e20a0 con 0x7f5d2c070c30 2023-10-19T13:03:02.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.984+0000 7f5d22ffd640 1 -- 172.21.15.57:0/963628474 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d28010380 con 0x7f5d2c070c30 2023-10-19T13:03:02.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.985+0000 7f5d34739640 1 -- 172.21.15.57:0/963628474 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d2c155290 con 0x7f5d2c070c30 2023-10-19T13:03:02.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.986+0000 7f5d22ffd640 1 -- 172.21.15.57:0/963628474 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 58) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f5d280104e0 con 0x7f5d2c070c30 2023-10-19T13:03:02.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.986+0000 7f5d22ffd640 1 --2- 172.21.15.57:0/963628474 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f5d1006af60 0x7f5d1006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:03:02.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.986+0000 7f5d22ffd640 1 -- 172.21.15.57:0/963628474 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f5d28089df0 con 0x7f5d2c070c30 2023-10-19T13:03:02.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.987+0000 7f5d31cd4640 1 --2- 172.21.15.57:0/963628474 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f5d1006af60 0x7f5d1006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:03:03.001 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.987+0000 7f5d31cd4640 1 --2- 172.21.15.57:0/963628474 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f5d1006af60 0x7f5d1006d420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c007fd0 tx=0x7f5d1c009040 comp rx=0 tx=0).ready entity=mgr.7540 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:03:03.001 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.987+0000 7f5d373a3740 1 Processor -- start 2023-10-19T13:03:03.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.988+0000 7f5d373a3740 1 -- start start 2023-10-19T13:03:03.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.988+0000 7f5d373a3740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557184f4fb40 0x557184f22ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:03:03.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.988+0000 7f5d373a3740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557184f4ff20 0x557184f528d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:03:03.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.988+0000 7f5d373a3740 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x557184f23020 0x557184f52e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:03:03.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.988+0000 7f5d373a3740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x557184e34730 con 0x557184f4ff20 2023-10-19T13:03:03.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.988+0000 7f5d373a3740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x557184e95780 con 0x557184f4fb40 2023-10-19T13:03:03.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.988+0000 7f5d373a3740 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x557184e92e70 con 0x557184f23020 2023-10-19T13:03:03.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.988+0000 7f5d31cd4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557184f4ff20 0x557184f528d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:03:03.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.988+0000 7f5d31cd4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557184f4ff20 0x557184f528d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:48686/0 (socket says 172.21.15.57:48686) 2023-10-19T13:03:03.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.988+0000 7f5d31cd4640 1 -- 172.21.15.57:0/2538902890 learned_addr learned my addr 172.21.15.57:0/2538902890 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:03:03.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.988+0000 7f5d324d5640 1 --2- 172.21.15.57:0/2538902890 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557184f4fb40 0x557184f22ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:03:03.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.989+0000 7f5d32cd6640 1 --2- 172.21.15.57:0/2538902890 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x557184f23020 0x557184f52e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:03:03.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.989+0000 7f5d31cd4640 1 -- 172.21.15.57:0/2538902890 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x557184f23020 msgr2=0x557184f52e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T13:03:03.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.989+0000 7f5d31cd4640 1 --2- 172.21.15.57:0/2538902890 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x557184f23020 0x557184f52e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:03.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.989+0000 7f5d31cd4640 1 -- 172.21.15.57:0/2538902890 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557184f4fb40 msgr2=0x557184f22ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T13:03:03.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.989+0000 7f5d31cd4640 1 --2- 172.21.15.57:0/2538902890 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557184f4fb40 0x557184f22ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:03.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.989+0000 7f5d31cd4640 1 -- 172.21.15.57:0/2538902890 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557184e933d0 con 0x557184f4ff20 2023-10-19T13:03:03.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.989+0000 7f5d31cd4640 1 --2- 172.21.15.57:0/2538902890 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557184f4ff20 0x557184f528d0 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c005390 tx=0x7f5d1c010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb854be3ea61700a server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:03:03.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.989+0000 7f5d0ffff640 1 -- 172.21.15.57:0/2538902890 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d1c003a00 con 0x557184f4ff20 2023-10-19T13:03:03.006 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.989+0000 7f5d0ffff640 1 -- 172.21.15.57:0/2538902890 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f5d1c003b60 con 0x557184f4ff20 2023-10-19T13:03:03.006 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.989+0000 7f5d0ffff640 1 -- 172.21.15.57:0/2538902890 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d1c020450 con 0x557184f4ff20 2023-10-19T13:03:03.006 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.990+0000 7f5d373a3740 1 -- 172.21.15.57:0/2538902890 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557184f4ff20 msgr2=0x557184f528d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:03:03.006 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.990+0000 7f5d373a3740 1 --2- 172.21.15.57:0/2538902890 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557184f4ff20 0x557184f528d0 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c005390 tx=0x7f5d1c010000 comp rx=0 tx=0).stop 2023-10-19T13:03:03.006 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.990+0000 7f5d373a3740 1 -- 172.21.15.57:0/2538902890 shutdown_connections 2023-10-19T13:03:03.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.990+0000 7f5d373a3740 1 --2- 172.21.15.57:0/2538902890 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x557184f23020 0x557184f52e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:03.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.990+0000 7f5d373a3740 1 --2- 172.21.15.57:0/2538902890 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557184f4ff20 0x557184f528d0 unknown :-1 s=CLOSED pgs=1461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:03.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.990+0000 7f5d373a3740 1 --2- 172.21.15.57:0/2538902890 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557184f4fb40 0x557184f22ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:03.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.990+0000 7f5d373a3740 1 -- 172.21.15.57:0/2538902890 >> 172.21.15.57:0/2538902890 conn(0x557184f24d50 msgr2=0x557184f6d8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:03:03.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.990+0000 7f5d373a3740 1 -- 172.21.15.57:0/2538902890 shutdown_connections 2023-10-19T13:03:03.008 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.990+0000 7f5d373a3740 1 -- 172.21.15.57:0/2538902890 wait complete. 2023-10-19T13:03:03.008 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.990+0000 7f5d373a3740 1 -- start start 2023-10-19T13:03:03.008 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.990+0000 7f5d373a3740 1 Processor -- start 2023-10-19T13:03:03.008 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.991+0000 7f5d373a3740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557184f4fb40 0x557184f55320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:03:03.008 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.991+0000 7f5d373a3740 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x557184f4ff20 0x557184f68710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:03:03.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.991+0000 7f5d373a3740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557184f6e4f0 0x557184f68c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:03:03.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.991+0000 7f5d373a3740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x557184e8e760 con 0x557184f6e4f0 2023-10-19T13:03:03.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.991+0000 7f5d373a3740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x557184e498f0 con 0x557184f4fb40 2023-10-19T13:03:03.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.991+0000 7f5d373a3740 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x557184e49590 con 0x557184f4ff20 2023-10-19T13:03:03.010 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.991+0000 7f5d373a3740 10 client.0 _ll_register_callbacks cb 0x557184e85ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-19T13:03:03.010 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.991+0000 7f5d32cd6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557184f6e4f0 0x557184f68c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:03:03.010 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.991+0000 7f5d32cd6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557184f6e4f0 0x557184f68c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:48692/0 (socket says 172.21.15.57:48692) 2023-10-19T13:03:03.010 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.991+0000 7f5d32cd6640 1 -- 172.21.15.57:0/1426568631 learned_addr learned my addr 172.21.15.57:0/1426568631 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:03:03.010 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.992+0000 7f5d31cd4640 1 --2- 172.21.15.57:0/1426568631 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x557184f4ff20 0x557184f68710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:03:03.011 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.992+0000 7f5d324d5640 1 --2- 172.21.15.57:0/1426568631 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557184f4fb40 0x557184f55320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:03:03.011 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.992+0000 7f5d32cd6640 1 -- 172.21.15.57:0/1426568631 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x557184f4ff20 msgr2=0x557184f68710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T13:03:03.011 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.992+0000 7f5d32cd6640 1 --2- 172.21.15.57:0/1426568631 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x557184f4ff20 0x557184f68710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:03.011 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.992+0000 7f5d32cd6640 1 -- 172.21.15.57:0/1426568631 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557184f4fb40 msgr2=0x557184f55320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T13:03:03.012 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.992+0000 7f5d32cd6640 1 --2- 172.21.15.57:0/1426568631 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557184f4fb40 0x557184f55320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:03.012 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.992+0000 7f5d32cd6640 1 -- 172.21.15.57:0/1426568631 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d1c0050d0 con 0x557184f6e4f0 2023-10-19T13:03:03.012 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.992+0000 7f5d32cd6640 1 --2- 172.21.15.57:0/1426568631 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557184f6e4f0 0x557184f68c50 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7f5d24009bf0 tx=0x7f5d24034a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a21ab30f055b22ce server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:03:03.012 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.992+0000 7f5d0d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2023-10-19T13:03:03.012 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.992+0000 7f5d0d7fa640 1 -- 172.21.15.57:0/1426568631 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d24034ba0 con 0x557184f6e4f0 2023-10-19T13:03:03.012 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.992+0000 7f5d0d7fa640 1 -- 172.21.15.57:0/1426568631 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(130..130 src has 1..130) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f5d24003b80 con 0x557184f6e4f0 2023-10-19T13:03:03.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.992+0000 7f5d373a3740 1 -- 172.21.15.57:0/1426568631 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557184e87740 con 0x557184f6e4f0 2023-10-19T13:03:03.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.992+0000 7f5d0d7fa640 1 -- 172.21.15.57:0/1426568631 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5d24003d40 con 0x557184f6e4f0 2023-10-19T13:03:03.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.992+0000 7f5d373a3740 10 client.8718 fetch_fsmap learned FSMap version 50 2023-10-19T13:03:03.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.992+0000 7f5d373a3740 1 -- 172.21.15.57:0/1426568631 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=2+}) v3 -- 0x557184e8cde0 con 0x557184f6e4f0 2023-10-19T13:03:03.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.992+0000 7f5d0d7fa640 1 -- 172.21.15.57:0/1426568631 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x557184e87740 con 0x557184f6e4f0 2023-10-19T13:03:03.014 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.993+0000 7f5d0d7fa640 1 -- 172.21.15.57:0/1426568631 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 50) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f5d24004040 con 0x557184f6e4f0 2023-10-19T13:03:03.014 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.993+0000 7f5d373a3740 10 client.8718 fetch_fsmap finished waiting for FSMap version 50 2023-10-19T13:03:03.014 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.993+0000 7f5d373a3740 10 client.8718 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-19T13:03:03.014 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.993+0000 7f5d373a3740 20 client.8718 populate_metadata read hostname 'smithi057' 2023-10-19T13:03:03.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.993+0000 7f5d373a3740 1 --2- 172.21.15.57:0/1426568631 >> [v2:172.21.15.179:6835/768231146,v1:172.21.15.179:6837/768231146] conn(0x557184f65dc0 0x557184f661a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:03:03.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.993+0000 7f5d373a3740 4 client.8718 mds_command: new command op to 8577 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-19T13:03:03.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.993+0000 7f5d373a3740 1 -- 172.21.15.57:0/1426568631 --> [v2:172.21.15.179:6835/768231146,v1:172.21.15.179:6837/768231146] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557184e87460 con 0x557184f65dc0 2023-10-19T13:03:03.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.993+0000 7f5d324d5640 1 --2- 172.21.15.57:0/1426568631 >> [v2:172.21.15.179:6835/768231146,v1:172.21.15.179:6837/768231146] conn(0x557184f65dc0 0x557184f661a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:03:03.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.993+0000 7f5d0d7fa640 10 client.8718 ms_handle_connect on v2:172.21.15.179:6835/768231146 2023-10-19T13:03:03.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.993+0000 7f5d324d5640 1 --2- 172.21.15.57:0/1426568631 >> [v2:172.21.15.179:6835/768231146,v1:172.21.15.179:6837/768231146] conn(0x557184f65dc0 0x557184f661a0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b876b56f912b4b22 server_cookie=1a49dd77e5aec94e in_seq=0 out_seq=0 2023-10-19T13:03:03.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.994+0000 7f5d0d7fa640 1 -- 172.21.15.57:0/1426568631 <== mds.0 v2:172.21.15.179:6835/768231146 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x557184e87460 con 0x557184f65dc0 2023-10-19T13:03:03.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.994+0000 7f5d0d7fa640 10 client.8718 handle_command_reply: tid=0 2023-10-19T13:03:03.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.994+0000 7f5d373a3740 1 client.8718 shutdown 2023-10-19T13:03:03.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.994+0000 7f5d373a3740 1 -- 172.21.15.57:0/1426568631 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557184f6e4f0 msgr2=0x557184f68c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:03:03.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.994+0000 7f5d373a3740 1 --2- 172.21.15.57:0/1426568631 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557184f6e4f0 0x557184f68c50 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7f5d24009bf0 tx=0x7f5d24034a30 comp rx=0 tx=0).stop 2023-10-19T13:03:03.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.994+0000 7f5d373a3740 1 -- 172.21.15.57:0/1426568631 shutdown_connections 2023-10-19T13:03:03.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.994+0000 7f5d373a3740 1 --2- 172.21.15.57:0/1426568631 >> [v2:172.21.15.179:6835/768231146,v1:172.21.15.179:6837/768231146] conn(0x557184f65dc0 0x557184f661a0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:03.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.994+0000 7f5d373a3740 1 --2- 172.21.15.57:0/1426568631 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557184f6e4f0 0x557184f68c50 unknown :-1 s=CLOSED pgs=1462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:03.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.994+0000 7f5d373a3740 1 --2- 172.21.15.57:0/1426568631 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x557184f4ff20 0x557184f68710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:03.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.994+0000 7f5d0d7fa640 0 client.8718 ms_handle_reset on v2:172.21.15.179:6835/768231146 2023-10-19T13:03:03.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.994+0000 7f5d373a3740 1 --2- 172.21.15.57:0/1426568631 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557184f4fb40 0x557184f55320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:03.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.994+0000 7f5d373a3740 1 -- 172.21.15.57:0/1426568631 >> 172.21.15.57:0/1426568631 conn(0x557184f23020 msgr2=0x557184f528c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:03:03.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.994+0000 7f5d373a3740 1 -- 172.21.15.57:0/1426568631 shutdown_connections 2023-10-19T13:03:03.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.994+0000 7f5d373a3740 1 -- 172.21.15.57:0/1426568631 wait complete. 2023-10-19T13:03:03.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:02.994+0000 7f5d373a3740 20 client.8718 trim_cache size 0 max 16384 2023-10-19T13:03:03.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.009+0000 7f5d373a3740 1 Processor -- start 2023-10-19T13:03:03.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.010+0000 7f5d373a3740 1 -- start start 2023-10-19T13:03:03.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.010+0000 7f5d373a3740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557184f4fb40 0x557184f70350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:03:03.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.010+0000 7f5d373a3740 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x557184f4ff20 0x557184f277e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:03:03.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.010+0000 7f5d373a3740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557184f6e4f0 0x557184f27d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:03:03.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.010+0000 7f5d373a3740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x557184e0f270 con 0x557184f4fb40 2023-10-19T13:03:03.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.010+0000 7f5d373a3740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x557184f561e0 con 0x557184f6e4f0 2023-10-19T13:03:03.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.010+0000 7f5d373a3740 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x557184f71d90 con 0x557184f4ff20 2023-10-19T13:03:03.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.010+0000 7f5d324d5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557184f4fb40 0x557184f70350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:03:03.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.010+0000 7f5d324d5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557184f4fb40 0x557184f70350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:48694/0 (socket says 172.21.15.57:48694) 2023-10-19T13:03:03.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.010+0000 7f5d324d5640 1 -- 172.21.15.57:0/3802861147 learned_addr learned my addr 172.21.15.57:0/3802861147 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:03:03.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.011+0000 7f5d324d5640 1 -- 172.21.15.57:0/3802861147 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x557184f4ff20 msgr2=0x557184f277e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T13:03:03.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.011+0000 7f5d324d5640 1 --2- 172.21.15.57:0/3802861147 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x557184f4ff20 0x557184f277e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:03.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.011+0000 7f5d324d5640 1 -- 172.21.15.57:0/3802861147 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557184f6e4f0 msgr2=0x557184f27d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T13:03:03.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.011+0000 7f5d324d5640 1 --2- 172.21.15.57:0/3802861147 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557184f6e4f0 0x557184f27d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:03.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.011+0000 7f5d324d5640 1 -- 172.21.15.57:0/3802861147 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557184e2bd00 con 0x557184f4fb40 2023-10-19T13:03:03.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.011+0000 7f5d324d5640 1 --2- 172.21.15.57:0/3802861147 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557184f4fb40 0x557184f70350 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7f5d2800bdd0 tx=0x7f5d2801c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e5af5c49210a3cd server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:03:03.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.011+0000 7f5d20ff9640 1 -- 172.21.15.57:0/3802861147 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d28056a50 con 0x557184f4fb40 2023-10-19T13:03:03.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.011+0000 7f5d20ff9640 1 -- 172.21.15.57:0/3802861147 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f5d28055050 con 0x557184f4fb40 2023-10-19T13:03:03.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.011+0000 7f5d20ff9640 1 -- 172.21.15.57:0/3802861147 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d28057d40 con 0x557184f4fb40 2023-10-19T13:03:03.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.011+0000 7f5d373a3740 1 -- 172.21.15.57:0/3802861147 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557184f4fb40 msgr2=0x557184f70350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:03:03.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.011+0000 7f5d373a3740 1 --2- 172.21.15.57:0/3802861147 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557184f4fb40 0x557184f70350 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7f5d2800bdd0 tx=0x7f5d2801c040 comp rx=0 tx=0).stop 2023-10-19T13:03:03.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.011+0000 7f5d373a3740 1 -- 172.21.15.57:0/3802861147 shutdown_connections 2023-10-19T13:03:03.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.011+0000 7f5d373a3740 1 --2- 172.21.15.57:0/3802861147 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557184f6e4f0 0x557184f27d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:03.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.011+0000 7f5d373a3740 1 --2- 172.21.15.57:0/3802861147 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x557184f4ff20 0x557184f277e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:03.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.011+0000 7f5d373a3740 1 --2- 172.21.15.57:0/3802861147 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557184f4fb40 0x557184f70350 unknown :-1 s=CLOSED pgs=1463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:03.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.011+0000 7f5d373a3740 1 -- 172.21.15.57:0/3802861147 >> 172.21.15.57:0/3802861147 conn(0x557184f23020 msgr2=0x557184f1e2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:03:03.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.011+0000 7f5d373a3740 1 -- 172.21.15.57:0/3802861147 shutdown_connections 2023-10-19T13:03:03.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.011+0000 7f5d373a3740 1 -- 172.21.15.57:0/3802861147 wait complete. 2023-10-19T13:03:03.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.012+0000 7f5d373a3740 1 -- start start 2023-10-19T13:03:03.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.012+0000 7f5d373a3740 1 Processor -- start 2023-10-19T13:03:03.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.013+0000 7f5d373a3740 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x557184f4fb40 0x557184fd6150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:03:03.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.013+0000 7f5d373a3740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557184f4ff20 0x557184f1e200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:03:03.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.013+0000 7f5d373a3740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557184f6e4f0 0x557184f1e740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:03:03.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.013+0000 7f5d373a3740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x557184f6f8b0 con 0x557184f6e4f0 2023-10-19T13:03:03.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.013+0000 7f5d373a3740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x557184fd9db0 con 0x557184f4ff20 2023-10-19T13:03:03.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.013+0000 7f5d373a3740 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x557184f68f40 con 0x557184f4fb40 2023-10-19T13:03:03.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.013+0000 7f5d373a3740 10 client.0 _ll_register_callbacks cb 0x7f5d1c0013a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-19T13:03:03.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.013+0000 7f5d32cd6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557184f6e4f0 0x557184f1e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:03:03.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.013+0000 7f5d32cd6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557184f6e4f0 0x557184f1e740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:48702/0 (socket says 172.21.15.57:48702) 2023-10-19T13:03:03.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.013+0000 7f5d32cd6640 1 -- 172.21.15.57:0/53659951 learned_addr learned my addr 172.21.15.57:0/53659951 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:03:03.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.013+0000 7f5d324d5640 1 --2- 172.21.15.57:0/53659951 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x557184f4fb40 0x557184fd6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:03:03.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.013+0000 7f5d31cd4640 1 --2- 172.21.15.57:0/53659951 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557184f4ff20 0x557184f1e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:03:03.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.013+0000 7f5d32cd6640 1 -- 172.21.15.57:0/53659951 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x557184f4fb40 msgr2=0x557184fd6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T13:03:03.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.013+0000 7f5d32cd6640 1 --2- 172.21.15.57:0/53659951 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x557184f4fb40 0x557184fd6150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:03.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.013+0000 7f5d32cd6640 1 -- 172.21.15.57:0/53659951 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557184f4ff20 msgr2=0x557184f1e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T13:03:03.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.013+0000 7f5d32cd6640 1 --2- 172.21.15.57:0/53659951 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557184f4ff20 0x557184f1e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:03.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.013+0000 7f5d32cd6640 1 -- 172.21.15.57:0/53659951 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d2c155290 con 0x557184f6e4f0 2023-10-19T13:03:03.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.014+0000 7f5d32cd6640 1 --2- 172.21.15.57:0/53659951 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557184f6e4f0 0x557184f1e740 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7f5d2402f020 tx=0x7f5d24009870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b4a9a54f08b6f66 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:03:03.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.014+0000 7f5d0d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2023-10-19T13:03:03.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.014+0000 7f5d0d7fa640 1 -- 172.21.15.57:0/53659951 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d240047e0 con 0x557184f6e4f0 2023-10-19T13:03:03.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.014+0000 7f5d0d7fa640 1 -- 172.21.15.57:0/53659951 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(130..130 src has 1..130) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f5d24054c00 con 0x557184f6e4f0 2023-10-19T13:03:03.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.014+0000 7f5d373a3740 1 -- 172.21.15.57:0/53659951 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5d1c003b60 con 0x557184f6e4f0 2023-10-19T13:03:03.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.014+0000 7f5d0d7fa640 1 -- 172.21.15.57:0/53659951 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5d24054dc0 con 0x557184f6e4f0 2023-10-19T13:03:03.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.014+0000 7f5d373a3740 10 client.8724 fetch_fsmap learned FSMap version 50 2023-10-19T13:03:03.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.014+0000 7f5d373a3740 1 -- 172.21.15.57:0/53659951 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=2+}) v3 -- 0x557184e514f0 con 0x557184f6e4f0 2023-10-19T13:03:03.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.014+0000 7f5d0d7fa640 1 -- 172.21.15.57:0/53659951 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f5d1c003b60 con 0x557184f6e4f0 2023-10-19T13:03:03.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.014+0000 7f5d0d7fa640 1 -- 172.21.15.57:0/53659951 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 50) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f5d24003d90 con 0x557184f6e4f0 2023-10-19T13:03:03.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.014+0000 7f5d373a3740 10 client.8724 fetch_fsmap finished waiting for FSMap version 50 2023-10-19T13:03:03.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.014+0000 7f5d373a3740 10 client.8724 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-19T13:03:03.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.014+0000 7f5d373a3740 20 client.8724 populate_metadata read hostname 'smithi057' 2023-10-19T13:03:03.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.014+0000 7f5d373a3740 1 --2- 172.21.15.57:0/53659951 >> [v2:172.21.15.179:6835/768231146,v1:172.21.15.179:6837/768231146] conn(0x557184f7fbe0 0x557184f7ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:03:03.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.014+0000 7f5d373a3740 4 client.8724 mds_command: new command op to 8577 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-19T13:03:03.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.014+0000 7f5d373a3740 1 -- 172.21.15.57:0/53659951 --> [v2:172.21.15.179:6835/768231146,v1:172.21.15.179:6837/768231146] -- command(tid 0: {"prefix": "status"}) v1 -- 0x557184b906c0 con 0x557184f7fbe0 2023-10-19T13:03:03.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.014+0000 7f5d324d5640 1 --2- 172.21.15.57:0/53659951 >> [v2:172.21.15.179:6835/768231146,v1:172.21.15.179:6837/768231146] conn(0x557184f7fbe0 0x557184f7ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:03:03.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.015+0000 7f5d324d5640 1 --2- 172.21.15.57:0/53659951 >> [v2:172.21.15.179:6835/768231146,v1:172.21.15.179:6837/768231146] conn(0x557184f7fbe0 0x557184f7ffc0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4720952a3bc0ebb2 server_cookie=fe6a5051f1d904ce in_seq=0 out_seq=0 2023-10-19T13:03:03.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.015+0000 7f5d0d7fa640 10 client.8724 ms_handle_connect on v2:172.21.15.179:6835/768231146 2023-10-19T13:03:03.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.015+0000 7f5d0d7fa640 1 -- 172.21.15.57:0/53659951 <== mds.0 v2:172.21.15.179:6835/768231146 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+320 (crc 0 0 0) 0x557184b906c0 con 0x557184f7fbe0 2023-10-19T13:03:03.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.015+0000 7f5d0d7fa640 10 client.8724 handle_command_reply: tid=0 2023-10-19T13:03:03.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.015+0000 7f5d373a3740 1 client.8724 shutdown 2023-10-19T13:03:03.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.015+0000 7f5d373a3740 1 -- 172.21.15.57:0/53659951 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557184f6e4f0 msgr2=0x557184f1e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:03:03.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.015+0000 7f5d373a3740 1 --2- 172.21.15.57:0/53659951 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557184f6e4f0 0x557184f1e740 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7f5d2402f020 tx=0x7f5d24009870 comp rx=0 tx=0).stop 2023-10-19T13:03:03.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.015+0000 7f5d373a3740 1 -- 172.21.15.57:0/53659951 shutdown_connections 2023-10-19T13:03:03.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.015+0000 7f5d373a3740 1 --2- 172.21.15.57:0/53659951 >> [v2:172.21.15.179:6835/768231146,v1:172.21.15.179:6837/768231146] conn(0x557184f7fbe0 0x557184f7ffc0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:03.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.015+0000 7f5d373a3740 1 --2- 172.21.15.57:0/53659951 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557184f6e4f0 0x557184f1e740 unknown :-1 s=CLOSED pgs=1464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:03.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.015+0000 7f5d0d7fa640 0 client.8724 ms_handle_reset on v2:172.21.15.179:6835/768231146 2023-10-19T13:03:03.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.015+0000 7f5d373a3740 1 --2- 172.21.15.57:0/53659951 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557184f4ff20 0x557184f1e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:03.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.015+0000 7f5d373a3740 1 --2- 172.21.15.57:0/53659951 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x557184f4fb40 0x557184fd6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:03.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.015+0000 7f5d373a3740 1 -- 172.21.15.57:0/53659951 >> 172.21.15.57:0/53659951 conn(0x557184f23020 msgr2=0x557184f651a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:03:03.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.015+0000 7f5d373a3740 1 -- 172.21.15.57:0/53659951 shutdown_connections 2023-10-19T13:03:03.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.015+0000 7f5d373a3740 1 -- 172.21.15.57:0/53659951 wait complete. 2023-10-19T13:03:03.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.015+0000 7f5d373a3740 20 client.8724 trim_cache size 0 max 16384 2023-10-19T13:03:03.040 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2023-10-19T13:03:03.040 INFO:teuthology.orchestra.run.smithi057.stdout: "cluster_fsid": "1e9beb19-3260-4dd7-85ef-5b399ce774ac", 2023-10-19T13:03:03.040 INFO:teuthology.orchestra.run.smithi057.stdout: "whoami": 0, 2023-10-19T13:03:03.040 INFO:teuthology.orchestra.run.smithi057.stdout: "id": 8577, 2023-10-19T13:03:03.041 INFO:teuthology.orchestra.run.smithi057.stdout: "want_state": "up:active", 2023-10-19T13:03:03.041 INFO:teuthology.orchestra.run.smithi057.stdout: "state": "up:active", 2023-10-19T13:03:03.041 INFO:teuthology.orchestra.run.smithi057.stdout: "fs_name": "cephfs", 2023-10-19T13:03:03.041 INFO:teuthology.orchestra.run.smithi057.stdout: "rank_uptime": 4.680673208, 2023-10-19T13:03:03.041 INFO:teuthology.orchestra.run.smithi057.stdout: "mdsmap_epoch": 50, 2023-10-19T13:03:03.042 INFO:teuthology.orchestra.run.smithi057.stdout: "osdmap_epoch": 130, 2023-10-19T13:03:03.042 INFO:teuthology.orchestra.run.smithi057.stdout: "osdmap_epoch_barrier": 130, 2023-10-19T13:03:03.042 INFO:teuthology.orchestra.run.smithi057.stdout: "uptime": 10.389576561 2023-10-19T13:03:03.042 INFO:teuthology.orchestra.run.smithi057.stdout:} 2023-10-19T13:03:03.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.016+0000 7f5d34739640 1 -- 172.21.15.57:0/963628474 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f5d1006af60 msgr2=0x7f5d1006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:03:03.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.016+0000 7f5d34739640 1 --2- 172.21.15.57:0/963628474 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f5d1006af60 0x7f5d1006d420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c007fd0 tx=0x7f5d1c009040 comp rx=0 tx=0).stop 2023-10-19T13:03:03.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.016+0000 7f5d34739640 1 -- 172.21.15.57:0/963628474 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5d2c070c30 msgr2=0x7f5d2c1483e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:03:03.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.016+0000 7f5d34739640 1 --2- 172.21.15.57:0/963628474 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5d2c070c30 0x7f5d2c1483e0 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f5d2800b410 tx=0x7f5d2800b8d0 comp rx=0 tx=0).stop 2023-10-19T13:03:03.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.016+0000 7f5d34739640 1 -- 172.21.15.57:0/963628474 shutdown_connections 2023-10-19T13:03:03.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.016+0000 7f5d34739640 1 --2- 172.21.15.57:0/963628474 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f5d1006af60 0x7f5d1006d420 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:03.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.016+0000 7f5d34739640 1 --2- 172.21.15.57:0/963628474 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5d2c157c00 0x7f5d2c158190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:03.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.016+0000 7f5d34739640 1 --2- 172.21.15.57:0/963628474 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5d2c0cb2f0 0x7f5d2c1e1450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:03.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.016+0000 7f5d34739640 1 --2- 172.21.15.57:0/963628474 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5d2c070c30 0x7f5d2c1483e0 unknown :-1 s=CLOSED pgs=1460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:03.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.016+0000 7f5d34739640 1 -- 172.21.15.57:0/963628474 >> 172.21.15.57:0/963628474 conn(0x7f5d2c0aec40 msgr2=0x7f5d2c06fae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:03:03.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.016+0000 7f5d34739640 1 -- 172.21.15.57:0/963628474 shutdown_connections 2023-10-19T13:03:03.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.016+0000 7f5d34739640 1 -- 172.21.15.57:0/963628474 wait complete. 2023-10-19T13:03:03.046 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-19T13:03:03.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.153+0000 7fec3ac0e640 1 -- 172.21.15.57:0/1708338823 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec1c005180 msgr2=0x7fec1c00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:03:03.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.153+0000 7fec3ac0e640 1 --2- 172.21.15.57:0/1708338823 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec1c005180 0x7fec1c00b980 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7fec2c00c560 tx=0x7fec1c0057f0 comp rx=0 tx=0).stop 2023-10-19T13:03:03.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.153+0000 7fec391ab640 1 -- 172.21.15.57:0/1708338823 reap_dead start 2023-10-19T13:03:03.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.153+0000 7fec3ac0e640 1 -- 172.21.15.57:0/1708338823 shutdown_connections 2023-10-19T13:03:03.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.153+0000 7fec3ac0e640 1 -- 172.21.15.57:0/1708338823 >> 172.21.15.57:0/1708338823 conn(0x7fec34069930 msgr2=0x7fec34069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:03:03.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.153+0000 7fec3ac0e640 1 -- 172.21.15.57:0/1708338823 shutdown_connections 2023-10-19T13:03:03.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.153+0000 7fec3ac0e640 1 -- 172.21.15.57:0/1708338823 wait complete. 2023-10-19T13:03:03.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.154+0000 7fec3ac0e640 1 Processor -- start 2023-10-19T13:03:03.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.154+0000 7fec3ac0e640 1 -- start start 2023-10-19T13:03:03.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.155+0000 7fec3ac0e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fec340a6720 0x7fec34141a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:03:03.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.155+0000 7fec3ac0e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec340a6b00 0x7fec34152890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:03:03.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.155+0000 7fec3ac0e640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fec3413e2b0 0x7fec3413e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:03:03.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.155+0000 7fec3ac0e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec34152dd0 con 0x7fec340a6b00 2023-10-19T13:03:03.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.155+0000 7fec3ac0e640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec3413ef70 con 0x7fec340a6720 2023-10-19T13:03:03.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.155+0000 7fec3ac0e640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec3413ec70 con 0x7fec3413e2b0 2023-10-19T13:03:03.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.155+0000 7fec33fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec340a6b00 0x7fec34152890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:03:03.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.155+0000 7fec33fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec340a6b00 0x7fec34152890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:48728/0 (socket says 172.21.15.57:48728) 2023-10-19T13:03:03.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.155+0000 7fec33fff640 1 -- 172.21.15.57:0/3640768666 learned_addr learned my addr 172.21.15.57:0/3640768666 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:03:03.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.155+0000 7fec389aa640 1 --2- 172.21.15.57:0/3640768666 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fec340a6720 0x7fec34141a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:03:03.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.155+0000 7fec391ab640 1 --2- 172.21.15.57:0/3640768666 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fec3413e2b0 0x7fec3413e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:03:03.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.155+0000 7fec33fff640 1 -- 172.21.15.57:0/3640768666 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fec3413e2b0 msgr2=0x7fec3413e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:03:03.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.155+0000 7fec33fff640 1 --2- 172.21.15.57:0/3640768666 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fec3413e2b0 0x7fec3413e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:03.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.155+0000 7fec33fff640 1 -- 172.21.15.57:0/3640768666 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fec340a6720 msgr2=0x7fec34141a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:03:03.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.155+0000 7fec33fff640 1 --2- 172.21.15.57:0/3640768666 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fec340a6720 0x7fec34141a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:03.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.155+0000 7fec33fff640 1 -- 172.21.15.57:0/3640768666 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec34152fc0 con 0x7fec340a6b00 2023-10-19T13:03:03.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.156+0000 7fec33fff640 1 --2- 172.21.15.57:0/3640768666 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec340a6b00 0x7fec34152890 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7fec28011fb0 tx=0x7fec2800d7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:03:03.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.156+0000 7fec317fa640 1 -- 172.21.15.57:0/3640768666 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec28016070 con 0x7fec340a6b00 2023-10-19T13:03:03.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.156+0000 7fec317fa640 1 -- 172.21.15.57:0/3640768666 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fec2800c5e0 con 0x7fec340a6b00 2023-10-19T13:03:03.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.156+0000 7fec3ac0e640 1 -- 172.21.15.57:0/3640768666 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec341532a0 con 0x7fec340a6b00 2023-10-19T13:03:03.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.156+0000 7fec317fa640 1 -- 172.21.15.57:0/3640768666 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec2800c9b0 con 0x7fec340a6b00 2023-10-19T13:03:03.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.156+0000 7fec3ac0e640 1 -- 172.21.15.57:0/3640768666 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec34153860 con 0x7fec340a6b00 2023-10-19T13:03:03.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.157+0000 7fec3ac0e640 1 -- 172.21.15.57:0/3640768666 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec340690e0 con 0x7fec340a6b00 2023-10-19T13:03:03.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.157+0000 7fec317fa640 1 -- 172.21.15.57:0/3640768666 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 58) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fec2800cb10 con 0x7fec340a6b00 2023-10-19T13:03:03.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.158+0000 7fec317fa640 1 --2- 172.21.15.57:0/3640768666 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7fec2006af90 0x7fec2006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:03:03.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.158+0000 7fec317fa640 1 -- 172.21.15.57:0/3640768666 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fec2808c510 con 0x7fec340a6b00 2023-10-19T13:03:03.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.158+0000 7fec389aa640 1 --2- 172.21.15.57:0/3640768666 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7fec2006af90 0x7fec2006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:03:03.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.158+0000 7fec389aa640 1 --2- 172.21.15.57:0/3640768666 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7fec2006af90 0x7fec2006d450 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fec24006fd0 tx=0x7fec24006f60 comp rx=0 tx=0).ready entity=mgr.7540 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:03:03.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.162+0000 7fec317fa640 1 -- 172.21.15.57:0/3640768666 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fec280572e0 con 0x7fec340a6b00 2023-10-19T13:03:03.311 INFO:tasks.ceph.mds.d.smithi179.stderr:2023-10-19T13:03:03.309+0000 7f9557120640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-19T13:03:03.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.376+0000 7fec3ac0e640 1 -- 172.21.15.57:0/3640768666 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fec340af510 con 0x7fec340a6b00 2023-10-19T13:03:03.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.376+0000 7fec317fa640 1 -- 172.21.15.57:0/3640768666 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v50) v1 ==== 53+0+73 (secure 0 0 0) 0x7fec2805a900 con 0x7fec340a6b00 2023-10-19T13:03:03.379 INFO:teuthology.orchestra.run.smithi057.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-19T13:03:03.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.378+0000 7fec3ac0e640 1 -- 172.21.15.57:0/3640768666 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7fec2006af90 msgr2=0x7fec2006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:03:03.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.378+0000 7fec3ac0e640 1 --2- 172.21.15.57:0/3640768666 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7fec2006af90 0x7fec2006d450 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fec24006fd0 tx=0x7fec24006f60 comp rx=0 tx=0).stop 2023-10-19T13:03:03.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.378+0000 7fec3ac0e640 1 -- 172.21.15.57:0/3640768666 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec340a6b00 msgr2=0x7fec34152890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:03:03.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.378+0000 7fec3ac0e640 1 --2- 172.21.15.57:0/3640768666 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec340a6b00 0x7fec34152890 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7fec28011fb0 tx=0x7fec2800d7b0 comp rx=0 tx=0).stop 2023-10-19T13:03:03.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.378+0000 7fec3ac0e640 1 -- 172.21.15.57:0/3640768666 shutdown_connections 2023-10-19T13:03:03.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.378+0000 7fec3ac0e640 1 --2- 172.21.15.57:0/3640768666 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7fec2006af90 0x7fec2006d450 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:03.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.378+0000 7fec3ac0e640 1 --2- 172.21.15.57:0/3640768666 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fec3413e2b0 0x7fec3413e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:03.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.378+0000 7fec3ac0e640 1 --2- 172.21.15.57:0/3640768666 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec340a6b00 0x7fec34152890 unknown :-1 s=CLOSED pgs=1467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:03.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.379+0000 7fec3ac0e640 1 --2- 172.21.15.57:0/3640768666 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fec340a6720 0x7fec34141a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:03.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.379+0000 7fec3ac0e640 1 -- 172.21.15.57:0/3640768666 >> 172.21.15.57:0/3640768666 conn(0x7fec34069930 msgr2=0x7fec3406b200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:03:03.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.379+0000 7fec3ac0e640 1 -- 172.21.15.57:0/3640768666 shutdown_connections 2023-10-19T13:03:03.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:03.379+0000 7fec3ac0e640 1 -- 172.21.15.57:0/3640768666 wait complete. 2023-10-19T13:03:03.392 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-10-19T13:03:03.392 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-19T13:03:03.392 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi057.front.sepia.ceph.com 2023-10-19T13:03:03.393 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-19T13:03:03.393 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-19T13:03:03.393 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-10-19T13:03:03.394 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-10-19T13:03:03.394 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-19T13:03:03.394 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-19T13:03:03.395 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:03:03.395 DEBUG:teuthology.orchestra.run.smithi057:> ip addr 2023-10-19T13:03:03.411 INFO:teuthology.orchestra.run.smithi057.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-19T13:03:03.411 INFO:teuthology.orchestra.run.smithi057.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-19T13:03:03.411 INFO:teuthology.orchestra.run.smithi057.stdout: inet 127.0.0.1/8 scope host lo 2023-10-19T13:03:03.411 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T13:03:03.412 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 ::1/128 scope host 2023-10-19T13:03:03.412 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T13:03:03.412 INFO:teuthology.orchestra.run.smithi057.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-19T13:03:03.412 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:6b:ff:86 brd ff:ff:ff:ff:ff:ff 2023-10-19T13:03:03.412 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp7s0f0 2023-10-19T13:03:03.413 INFO:teuthology.orchestra.run.smithi057.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-19T13:03:03.413 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:6b:ff:87 brd ff:ff:ff:ff:ff:ff 2023-10-19T13:03:03.413 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp7s0f1 2023-10-19T13:03:03.413 INFO:teuthology.orchestra.run.smithi057.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-19T13:03:03.413 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:bd:15:62 brd ff:ff:ff:ff:ff:ff 2023-10-19T13:03:03.413 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp1s0f0 2023-10-19T13:03:03.414 INFO:teuthology.orchestra.run.smithi057.stdout: inet 172.21.15.57/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-19T13:03:03.414 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft 41527sec preferred_lft 41527sec 2023-10-19T13:03:03.414 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 fe80::ec4:7aff:febd:1562/64 scope link noprefixroute 2023-10-19T13:03:03.414 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T13:03:03.414 INFO:teuthology.orchestra.run.smithi057.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-19T13:03:03.415 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:bd:15:63 brd ff:ff:ff:ff:ff:ff 2023-10-19T13:03:03.415 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp1s0f1 2023-10-19T13:03:03.415 INFO:teuthology.orchestra.run.smithi057.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-19T13:03:03.415 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0e:fe:f1:b5:41:59 brd ff:ff:ff:ff:ff:ff 2023-10-19T13:03:03.415 INFO:teuthology.orchestra.run.smithi057.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-19T13:03:03.416 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T13:03:03.416 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 fe80::64b1:a2ff:feb3:b08/64 scope link 2023-10-19T13:03:03.416 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T13:03:03.416 INFO:teuthology.orchestra.run.smithi057.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-19T13:03:03.416 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0e:fe:f1:b5:41:59 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T13:03:03.417 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 fe80::cfe:f1ff:feb5:4159/64 scope link 2023-10-19T13:03:03.417 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T13:03:03.417 INFO:teuthology.orchestra.run.smithi057.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-19T13:03:03.417 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 62:53:72:9d:52:71 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-19T13:03:03.418 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 fe80::6053:72ff:fe9d:5271/64 scope link 2023-10-19T13:03:03.418 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T13:03:03.418 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:03:03.418 DEBUG:teuthology.orchestra.run.smithi057:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-19T13:03:03.480 INFO:teuthology.orchestra.run.smithi057.stdout:1 2023-10-19T13:03:03.482 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:03:03.482 DEBUG:teuthology.orchestra.run.smithi057:> route 2023-10-19T13:03:03.538 INFO:teuthology.orchestra.run.smithi057.stdout:Kernel IP routing table 2023-10-19T13:03:03.538 INFO:teuthology.orchestra.run.smithi057.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-19T13:03:03.539 INFO:teuthology.orchestra.run.smithi057.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-19T13:03:03.539 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-19T13:03:03.539 INFO:teuthology.orchestra.run.smithi057.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-19T13:03:03.539 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:03:03.540 DEBUG:teuthology.orchestra.run.smithi057:> (cd / && exec bash -c ' 2023-10-19T13:03:03.540 DEBUG:teuthology.orchestra.run.smithi057:> set -e 2023-10-19T13:03:03.541 DEBUG:teuthology.orchestra.run.smithi057:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-19T13:03:03.541 DEBUG:teuthology.orchestra.run.smithi057:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-19T13:03:03.541 DEBUG:teuthology.orchestra.run.smithi057:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-19T13:03:03.542 DEBUG:teuthology.orchestra.run.smithi057:> ') 2023-10-19T13:03:03.644 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:03:03.644 DEBUG:teuthology.orchestra.run.smithi057:> ip netns list 2023-10-19T13:03:03.659 INFO:teuthology.orchestra.run.smithi057.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-19T13:03:03.659 INFO:teuthology.orchestra.run.smithi057.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-19T13:03:03.660 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-19T13:03:03.660 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:03:03.661 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip link set brx.0 up 2023-10-19T13:03:03.731 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-10-19T13:03:03.731 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-19T13:03:03.732 DEBUG:teuthology.orchestra.run.smithi057:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-19T13:03:03.788 INFO:teuthology.orchestra.run.smithi057.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-19T13:03:03.789 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-19T13:03:03.789 DEBUG:teuthology.orchestra.run.smithi057:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-19T13:03:03.843 DEBUG:teuthology.orchestra.run.smithi057:> sudo modprobe fuse 2023-10-19T13:03:03.914 DEBUG:teuthology.orchestra.run.smithi057:> cat /proc/self/mounts | awk '{print $2}' 2023-10-19T13:03:03.970 INFO:teuthology.orchestra.run.smithi057.stdout:/proc 2023-10-19T13:03:03.970 INFO:teuthology.orchestra.run.smithi057.stdout:/sys 2023-10-19T13:03:03.970 INFO:teuthology.orchestra.run.smithi057.stdout:/dev 2023-10-19T13:03:03.971 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/security 2023-10-19T13:03:03.971 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/shm 2023-10-19T13:03:03.971 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/pts 2023-10-19T13:03:03.971 INFO:teuthology.orchestra.run.smithi057.stdout:/run 2023-10-19T13:03:03.971 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/cgroup 2023-10-19T13:03:03.972 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/pstore 2023-10-19T13:03:03.972 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/bpf 2023-10-19T13:03:03.972 INFO:teuthology.orchestra.run.smithi057.stdout:/ 2023-10-19T13:03:03.972 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/selinux 2023-10-19T13:03:03.972 INFO:teuthology.orchestra.run.smithi057.stdout:/proc/sys/fs/binfmt_misc 2023-10-19T13:03:03.972 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/hugepages 2023-10-19T13:03:03.973 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/mqueue 2023-10-19T13:03:03.973 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/debug 2023-10-19T13:03:03.973 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/tracing 2023-10-19T13:03:03.973 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/fuse/connections 2023-10-19T13:03:03.973 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/config 2023-10-19T13:03:03.973 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-sysctl.service 2023-10-19T13:03:03.974 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-10-19T13:03:03.974 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-10-19T13:03:03.974 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/nfs/rpc_pipefs 2023-10-19T13:03:03.974 INFO:teuthology.orchestra.run.smithi057.stdout:/run/user/1000 2023-10-19T13:03:03.974 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph 2023-10-19T13:03:03.974 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-0 2023-10-19T13:03:03.975 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-1 2023-10-19T13:03:03.975 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-2 2023-10-19T13:03:03.975 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-3 2023-10-19T13:03:03.975 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns 2023-10-19T13:03:03.975 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T13:03:03.976 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T13:03:03.976 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-19T13:03:03.976 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-19T13:03:03.977 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:03:03.977 DEBUG:teuthology.orchestra.run.smithi057:> ls /sys/fs/fuse/connections 2023-10-19T13:03:04.032 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-10-19T13:03:04.032 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-10-19T13:03:04.074 DEBUG:teuthology.orchestra.run.smithi057:> sudo modprobe fuse 2023-10-19T13:03:04.105 DEBUG:teuthology.orchestra.run.smithi057:> cat /proc/self/mounts | awk '{print $2}' 2023-10-19T13:03:04.160 INFO:teuthology.orchestra.run.smithi057.stdout:/proc 2023-10-19T13:03:04.160 INFO:teuthology.orchestra.run.smithi057.stdout:/sys 2023-10-19T13:03:04.161 INFO:teuthology.orchestra.run.smithi057.stdout:/dev 2023-10-19T13:03:04.161 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/security 2023-10-19T13:03:04.161 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/shm 2023-10-19T13:03:04.161 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/pts 2023-10-19T13:03:04.161 INFO:teuthology.orchestra.run.smithi057.stdout:/run 2023-10-19T13:03:04.162 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/cgroup 2023-10-19T13:03:04.162 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/pstore 2023-10-19T13:03:04.162 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/bpf 2023-10-19T13:03:04.162 INFO:teuthology.orchestra.run.smithi057.stdout:/ 2023-10-19T13:03:04.162 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/selinux 2023-10-19T13:03:04.163 INFO:teuthology.orchestra.run.smithi057.stdout:/proc/sys/fs/binfmt_misc 2023-10-19T13:03:04.163 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/hugepages 2023-10-19T13:03:04.163 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/mqueue 2023-10-19T13:03:04.163 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/debug 2023-10-19T13:03:04.163 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/tracing 2023-10-19T13:03:04.163 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/fuse/connections 2023-10-19T13:03:04.164 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/config 2023-10-19T13:03:04.164 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-sysctl.service 2023-10-19T13:03:04.164 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-10-19T13:03:04.164 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-10-19T13:03:04.164 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/nfs/rpc_pipefs 2023-10-19T13:03:04.164 INFO:teuthology.orchestra.run.smithi057.stdout:/run/user/1000 2023-10-19T13:03:04.165 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph 2023-10-19T13:03:04.165 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-0 2023-10-19T13:03:04.165 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-1 2023-10-19T13:03:04.165 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-2 2023-10-19T13:03:04.165 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-3 2023-10-19T13:03:04.165 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns 2023-10-19T13:03:04.166 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T13:03:04.166 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T13:03:04.166 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-19T13:03:04.166 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-19T13:03:04.167 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:03:04.167 DEBUG:teuthology.orchestra.run.smithi057:> ls /sys/fs/fuse/connections 2023-10-19T13:03:04.168 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi057.stderr:2023-10-19T13:03:04.164+0000 7f40abeb5480 -1 init, newargv = 0x555939925db0 newargc=15 2023-10-19T13:03:04.168 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi057.stderr:ceph-fuse[128173]: starting ceph client 2023-10-19T13:03:04.171 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi057.stderr:ceph-fuse[128173]: starting fuse 2023-10-19T13:03:04.183 INFO:teuthology.orchestra.run.smithi057.stdout:43 2023-10-19T13:03:04.184 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2023-10-19T13:03:04.184 DEBUG:teuthology.orchestra.run.smithi057:> sudo stdin-killer -- python3 -c ' 2023-10-19T13:03:04.184 DEBUG:teuthology.orchestra.run.smithi057:> import glob 2023-10-19T13:03:04.185 DEBUG:teuthology.orchestra.run.smithi057:> import re 2023-10-19T13:03:04.185 DEBUG:teuthology.orchestra.run.smithi057:> import os 2023-10-19T13:03:04.185 DEBUG:teuthology.orchestra.run.smithi057:> import subprocess 2023-10-19T13:03:04.186 DEBUG:teuthology.orchestra.run.smithi057:> 2023-10-19T13:03:04.186 DEBUG:teuthology.orchestra.run.smithi057:> def _find_admin_socket(client_name): 2023-10-19T13:03:04.186 DEBUG:teuthology.orchestra.run.smithi057:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-10-19T13:03:04.187 DEBUG:teuthology.orchestra.run.smithi057:> files = glob.glob(asok_path) 2023-10-19T13:03:04.187 DEBUG:teuthology.orchestra.run.smithi057:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-10-19T13:03:04.187 DEBUG:teuthology.orchestra.run.smithi057:> 2023-10-19T13:03:04.188 DEBUG:teuthology.orchestra.run.smithi057:> # Given a non-glob path, it better be there 2023-10-19T13:03:04.188 DEBUG:teuthology.orchestra.run.smithi057:> if "*" not in asok_path: 2023-10-19T13:03:04.188 DEBUG:teuthology.orchestra.run.smithi057:> assert(len(files) == 1) 2023-10-19T13:03:04.189 DEBUG:teuthology.orchestra.run.smithi057:> return files[0] 2023-10-19T13:03:04.189 DEBUG:teuthology.orchestra.run.smithi057:> 2023-10-19T13:03:04.189 DEBUG:teuthology.orchestra.run.smithi057:> for f in files: 2023-10-19T13:03:04.190 DEBUG:teuthology.orchestra.run.smithi057:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-10-19T13:03:04.190 DEBUG:teuthology.orchestra.run.smithi057:> if os.path.exists("/proc/{0}".format(pid)): 2023-10-19T13:03:04.191 DEBUG:teuthology.orchestra.run.smithi057:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-10-19T13:03:04.191 DEBUG:teuthology.orchestra.run.smithi057:> contents = proc_f.read() 2023-10-19T13:03:04.191 DEBUG:teuthology.orchestra.run.smithi057:> if mountpoint in contents: 2023-10-19T13:03:04.192 DEBUG:teuthology.orchestra.run.smithi057:> return f 2023-10-19T13:03:04.192 DEBUG:teuthology.orchestra.run.smithi057:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-10-19T13:03:04.192 DEBUG:teuthology.orchestra.run.smithi057:> 2023-10-19T13:03:04.193 DEBUG:teuthology.orchestra.run.smithi057:> print(_find_admin_socket("client.0")) 2023-10-19T13:03:04.193 DEBUG:teuthology.orchestra.run.smithi057:> ' 2023-10-19T13:03:04.308 INFO:teuthology.orchestra.run.smithi057.stdout:/var/run/ceph/ceph-client.0.128173.asok 2023-10-19T13:03:04.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2023-10-19T13:03:04.318 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.128173.asok 2023-10-19T13:03:04.318 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:03:04.319 DEBUG:teuthology.orchestra.run.smithi057:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.128173.asok status 2023-10-19T13:03:04.459 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2023-10-19T13:03:04.460 INFO:teuthology.orchestra.run.smithi057.stdout: "metadata": { 2023-10-19T13:03:04.460 INFO:teuthology.orchestra.run.smithi057.stdout: "ceph_sha1": "f7420d7f8e55ceea786dcbf536c338ce920c21eb", 2023-10-19T13:03:04.460 INFO:teuthology.orchestra.run.smithi057.stdout: "ceph_version": "ceph version 18.0.0-6786-gf7420d7f (f7420d7f8e55ceea786dcbf536c338ce920c21eb) reef (dev)", 2023-10-19T13:03:04.460 INFO:teuthology.orchestra.run.smithi057.stdout: "entity_id": "0", 2023-10-19T13:03:04.461 INFO:teuthology.orchestra.run.smithi057.stdout: "hostname": "smithi057", 2023-10-19T13:03:04.461 INFO:teuthology.orchestra.run.smithi057.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-10-19T13:03:04.461 INFO:teuthology.orchestra.run.smithi057.stdout: "pid": "128173", 2023-10-19T13:03:04.461 INFO:teuthology.orchestra.run.smithi057.stdout: "root": "/" 2023-10-19T13:03:04.462 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:03:04.462 INFO:teuthology.orchestra.run.smithi057.stdout: "dentry_count": 0, 2023-10-19T13:03:04.462 INFO:teuthology.orchestra.run.smithi057.stdout: "dentry_pinned_count": 0, 2023-10-19T13:03:04.462 INFO:teuthology.orchestra.run.smithi057.stdout: "id": 8736, 2023-10-19T13:03:04.462 INFO:teuthology.orchestra.run.smithi057.stdout: "inst": { 2023-10-19T13:03:04.463 INFO:teuthology.orchestra.run.smithi057.stdout: "name": { 2023-10-19T13:03:04.463 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "client", 2023-10-19T13:03:04.463 INFO:teuthology.orchestra.run.smithi057.stdout: "num": 8736 2023-10-19T13:03:04.463 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:03:04.464 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": { 2023-10-19T13:03:04.464 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "v1", 2023-10-19T13:03:04.464 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": "192.168.0.1:0", 2023-10-19T13:03:04.464 INFO:teuthology.orchestra.run.smithi057.stdout: "nonce": 1544610110 2023-10-19T13:03:04.464 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:03:04.464 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:03:04.465 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": { 2023-10-19T13:03:04.465 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "v1", 2023-10-19T13:03:04.465 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": "192.168.0.1:0", 2023-10-19T13:03:04.465 INFO:teuthology.orchestra.run.smithi057.stdout: "nonce": 1544610110 2023-10-19T13:03:04.465 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:03:04.466 INFO:teuthology.orchestra.run.smithi057.stdout: "inst_str": "client.8736 192.168.0.1:0/1544610110", 2023-10-19T13:03:04.466 INFO:teuthology.orchestra.run.smithi057.stdout: "addr_str": "192.168.0.1:0/1544610110", 2023-10-19T13:03:04.466 INFO:teuthology.orchestra.run.smithi057.stdout: "inode_count": 1, 2023-10-19T13:03:04.466 INFO:teuthology.orchestra.run.smithi057.stdout: "mds_epoch": 50, 2023-10-19T13:03:04.466 INFO:teuthology.orchestra.run.smithi057.stdout: "osd_epoch": 130, 2023-10-19T13:03:04.467 INFO:teuthology.orchestra.run.smithi057.stdout: "osd_epoch_barrier": 0, 2023-10-19T13:03:04.467 INFO:teuthology.orchestra.run.smithi057.stdout: "blocklisted": false, 2023-10-19T13:03:04.467 INFO:teuthology.orchestra.run.smithi057.stdout: "fs_name": "cephfs" 2023-10-19T13:03:04.467 INFO:teuthology.orchestra.run.smithi057.stdout:} 2023-10-19T13:03:04.468 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:03:04.469 DEBUG:teuthology.orchestra.run.smithi057:> stat --file-system '--printf=%T 2023-10-19T13:03:04.469 DEBUG:teuthology.orchestra.run.smithi057:> ' -- /home/ubuntu/cephtest/mnt.0 2023-10-19T13:03:04.485 INFO:teuthology.orchestra.run.smithi057.stdout:fuseblk 2023-10-19T13:03:04.485 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-10-19T13:03:04.485 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:03:04.486 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-19T13:03:04.558 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-19T13:03:04.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:04.705+0000 7ff165efc640 1 -- 172.21.15.57:0/3521912835 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff1600afa20 msgr2=0x7ff1600705a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:03:04.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:04.705+0000 7ff165efc640 1 --2- 172.21.15.57:0/3521912835 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff1600afa20 0x7ff1600705a0 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7ff1600b1450 tx=0x7ff15402bda0 comp rx=0 tx=0).stop 2023-10-19T13:03:04.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:04.706+0000 7ff165efc640 1 -- 172.21.15.57:0/3521912835 shutdown_connections 2023-10-19T13:03:04.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:04.706+0000 7ff165efc640 1 --2- 172.21.15.57:0/3521912835 >> v2:172.21.15.179:3300/0 conn(0x7ff1600b0530 0x7ff1600b09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:04.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:04.706+0000 7ff165efc640 1 --2- 172.21.15.57:0/3521912835 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff1600afa20 0x7ff1600705a0 unknown :-1 s=CLOSED pgs=1471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:04.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:04.706+0000 7ff165efc640 1 --2- 172.21.15.57:0/3521912835 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff1600cbd20 0x7ff160158300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:04.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:04.706+0000 7ff165efc640 1 -- 172.21.15.57:0/3521912835 >> 172.21.15.57:0/3521912835 conn(0x7ff160069930 msgr2=0x7ff160069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:03:04.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:04.706+0000 7ff165efc640 1 -- 172.21.15.57:0/3521912835 shutdown_connections 2023-10-19T13:03:04.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:04.706+0000 7ff165efc640 1 -- 172.21.15.57:0/3521912835 wait complete. 2023-10-19T13:03:04.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:04.707+0000 7ff165efc640 1 Processor -- start 2023-10-19T13:03:04.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:04.707+0000 7ff165efc640 1 -- start start 2023-10-19T13:03:04.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:04.708+0000 7ff165efc640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff1600afa20 0x7ff160147220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:03:04.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:04.708+0000 7ff165efc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff1600b0530 0x7ff160145a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:03:04.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:04.708+0000 7ff165efc640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff1600cbd20 0x7ff160155750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:03:04.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:04.708+0000 7ff165efc640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1601562b0 con 0x7ff1600b0530 2023-10-19T13:03:04.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:04.708+0000 7ff165efc640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff16012f340 con 0x7ff1600afa20 2023-10-19T13:03:04.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:04.708+0000 7ff165efc640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff160155fd0 con 0x7ff1600cbd20 2023-10-19T13:03:04.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:04.708+0000 7ff15effd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff1600b0530 0x7ff160145a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:03:04.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:04.708+0000 7ff15effd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff1600b0530 0x7ff160145a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:48746/0 (socket says 172.21.15.57:48746) 2023-10-19T13:03:04.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:04.708+0000 7ff15effd640 1 -- 172.21.15.57:0/4078857349 learned_addr learned my addr 172.21.15.57:0/4078857349 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:03:04.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:04.708+0000 7ff15effd640 1 -- 172.21.15.57:0/4078857349 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff1600cbd20 msgr2=0x7ff160155750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:03:04.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:04.708+0000 7ff15effd640 1 --2- 172.21.15.57:0/4078857349 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff1600cbd20 0x7ff160155750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:04.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:04.708+0000 7ff15effd640 1 -- 172.21.15.57:0/4078857349 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff1600afa20 msgr2=0x7ff160147220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:03:04.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:04.708+0000 7ff15f7fe640 1 --2- 172.21.15.57:0/4078857349 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff1600afa20 0x7ff160147220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:03:04.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:04.708+0000 7ff15effd640 1 --2- 172.21.15.57:0/4078857349 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff1600afa20 0x7ff160147220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:04.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:04.708+0000 7ff15effd640 1 -- 172.21.15.57:0/4078857349 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff154009560 con 0x7ff1600b0530 2023-10-19T13:03:04.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:04.708+0000 7ff15effd640 1 --2- 172.21.15.57:0/4078857349 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff1600b0530 0x7ff160145a90 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7ff154036040 tx=0x7ff15402dad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:03:04.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:04.708+0000 7ff147fff640 1 -- 172.21.15.57:0/4078857349 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff154003800 con 0x7ff1600b0530 2023-10-19T13:03:04.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:04.708+0000 7ff147fff640 1 -- 172.21.15.57:0/4078857349 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7ff154003a50 con 0x7ff1600b0530 2023-10-19T13:03:04.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:04.708+0000 7ff165efc640 1 -- 172.21.15.57:0/4078857349 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff16012ecf0 con 0x7ff1600b0530 2023-10-19T13:03:04.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:04.708+0000 7ff147fff640 1 -- 172.21.15.57:0/4078857349 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff154003e20 con 0x7ff1600b0530 2023-10-19T13:03:04.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:04.708+0000 7ff165efc640 1 -- 172.21.15.57:0/4078857349 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff160145fd0 con 0x7ff1600b0530 2023-10-19T13:03:04.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:04.709+0000 7ff165efc640 1 -- 172.21.15.57:0/4078857349 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1600c96b0 con 0x7ff1600b0530 2023-10-19T13:03:04.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:04.710+0000 7ff147fff640 1 -- 172.21.15.57:0/4078857349 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 58) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff15403e450 con 0x7ff1600b0530 2023-10-19T13:03:04.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:04.710+0000 7ff147fff640 1 --2- 172.21.15.57:0/4078857349 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7ff14c06af60 0x7ff14c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:03:04.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:04.710+0000 7ff147fff640 1 -- 172.21.15.57:0/4078857349 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6057+0+0 (secure 0 0 0) 0x7ff1540ac390 con 0x7ff1600b0530 2023-10-19T13:03:04.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:04.711+0000 7ff15f7fe640 1 --2- 172.21.15.57:0/4078857349 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7ff14c06af60 0x7ff14c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:03:04.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:04.711+0000 7ff15f7fe640 1 --2- 172.21.15.57:0/4078857349 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7ff14c06af60 0x7ff14c06d420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff1500058c0 tx=0x7ff150005850 comp rx=0 tx=0).ready entity=mgr.7540 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:03:04.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:04.715+0000 7ff147fff640 1 -- 172.21.15.57:0/4078857349 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff154077380 con 0x7ff1600b0530 2023-10-19T13:03:04.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:04.914+0000 7ff165efc640 1 -- 172.21.15.57:0/4078857349 --> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7ff1600690e0 con 0x7ff14c06af60 2023-10-19T13:03:04.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:04.916+0000 7ff147fff640 1 -- 172.21.15.57:0/4078857349 <== mgr.7540 v2:172.21.15.179:6832/4155586759 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7ff1600690e0 con 0x7ff14c06af60 2023-10-19T13:03:04.918 INFO:teuthology.orchestra.run.smithi057.stdout:[ 2023-10-19T13:03:04.918 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:03:04.919 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "cephfs" 2023-10-19T13:03:04.919 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:03:04.919 INFO:teuthology.orchestra.run.smithi057.stdout:] 2023-10-19T13:03:04.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:04.917+0000 7ff165efc640 1 -- 172.21.15.57:0/4078857349 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7ff14c06af60 msgr2=0x7ff14c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:03:04.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:04.917+0000 7ff165efc640 1 --2- 172.21.15.57:0/4078857349 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7ff14c06af60 0x7ff14c06d420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff1500058c0 tx=0x7ff150005850 comp rx=0 tx=0).stop 2023-10-19T13:03:04.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:04.917+0000 7ff165efc640 1 -- 172.21.15.57:0/4078857349 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff1600b0530 msgr2=0x7ff160145a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:03:04.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:04.917+0000 7ff165efc640 1 --2- 172.21.15.57:0/4078857349 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff1600b0530 0x7ff160145a90 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7ff154036040 tx=0x7ff15402dad0 comp rx=0 tx=0).stop 2023-10-19T13:03:04.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:04.917+0000 7ff165efc640 1 -- 172.21.15.57:0/4078857349 shutdown_connections 2023-10-19T13:03:04.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:04.917+0000 7ff165efc640 1 --2- 172.21.15.57:0/4078857349 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7ff14c06af60 0x7ff14c06d420 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:04.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:04.917+0000 7ff165efc640 1 --2- 172.21.15.57:0/4078857349 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff1600cbd20 0x7ff160155750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:04.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:04.917+0000 7ff165efc640 1 --2- 172.21.15.57:0/4078857349 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff1600b0530 0x7ff160145a90 unknown :-1 s=CLOSED pgs=1472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:04.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:04.917+0000 7ff165efc640 1 --2- 172.21.15.57:0/4078857349 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff1600afa20 0x7ff160147220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:04.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:04.917+0000 7ff165efc640 1 -- 172.21.15.57:0/4078857349 >> 172.21.15.57:0/4078857349 conn(0x7ff160069930 msgr2=0x7ff160069d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:03:04.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:04.917+0000 7ff165efc640 1 -- 172.21.15.57:0/4078857349 shutdown_connections 2023-10-19T13:03:04.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:04.917+0000 7ff165efc640 1 -- 172.21.15.57:0/4078857349 wait complete. 2023-10-19T13:03:04.931 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-19T13:03:05.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.039+0000 7f0cb778e640 1 -- 172.21.15.57:0/1603066506 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0cb015b2a0 msgr2=0x7f0cb00cf720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:03:05.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.039+0000 7f0cb778e640 1 --2- 172.21.15.57:0/1603066506 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0cb015b2a0 0x7f0cb00cf720 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7f0ca0005fd0 tx=0x7f0ca002bd90 comp rx=0 tx=0).stop 2023-10-19T13:03:05.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.039+0000 7f0cb778e640 1 -- 172.21.15.57:0/1603066506 shutdown_connections 2023-10-19T13:03:05.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.039+0000 7f0cb778e640 1 --2- 172.21.15.57:0/1603066506 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0cb015b2a0 0x7f0cb00cf720 unknown :-1 s=CLOSED pgs=1473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:05.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.039+0000 7f0cb778e640 1 --2- 172.21.15.57:0/1603066506 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0cb00cf340 0x7f0cb00d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:05.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.039+0000 7f0cb778e640 1 -- 172.21.15.57:0/1603066506 >> 172.21.15.57:0/1603066506 conn(0x7f0cb00d0d60 msgr2=0x7f0cb00d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:03:05.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.039+0000 7f0cb778e640 1 -- 172.21.15.57:0/1603066506 shutdown_connections 2023-10-19T13:03:05.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.039+0000 7f0cb778e640 1 -- 172.21.15.57:0/1603066506 wait complete. 2023-10-19T13:03:05.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.041+0000 7f0cb778e640 1 Processor -- start 2023-10-19T13:03:05.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.041+0000 7f0cb778e640 1 -- start start 2023-10-19T13:03:05.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.041+0000 7f0cb778e640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0cb00ceb40 0x7f0cb00aef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:03:05.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.041+0000 7f0cb778e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0cb00cf340 0x7f0cb00b6050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:03:05.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.041+0000 7f0cb778e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0cb015b2a0 0x7f0cb006b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:03:05.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.041+0000 7f0cb778e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb015c820 con 0x7f0cb015b2a0 2023-10-19T13:03:05.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.041+0000 7f0cb778e640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb015d680 con 0x7f0cb00cf340 2023-10-19T13:03:05.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.041+0000 7f0cb778e640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb015ca70 con 0x7f0cb00ceb40 2023-10-19T13:03:05.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.041+0000 7f0cb5d2b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0cb015b2a0 0x7f0cb006b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:03:05.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.041+0000 7f0cb5d2b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0cb015b2a0 0x7f0cb006b3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:48754/0 (socket says 172.21.15.57:48754) 2023-10-19T13:03:05.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.041+0000 7f0cb5d2b640 1 -- 172.21.15.57:0/22743678 learned_addr learned my addr 172.21.15.57:0/22743678 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:03:05.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.042+0000 7f0cb4d29640 1 --2- 172.21.15.57:0/22743678 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0cb00cf340 0x7f0cb00b6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:03:05.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.042+0000 7f0cb552a640 1 --2- 172.21.15.57:0/22743678 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0cb00ceb40 0x7f0cb00aef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:03:05.049 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.042+0000 7f0cb5d2b640 1 -- 172.21.15.57:0/22743678 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0cb00ceb40 msgr2=0x7f0cb00aef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:03:05.049 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.042+0000 7f0cb5d2b640 1 --2- 172.21.15.57:0/22743678 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0cb00ceb40 0x7f0cb00aef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:05.049 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.042+0000 7f0cb5d2b640 1 -- 172.21.15.57:0/22743678 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0cb00cf340 msgr2=0x7f0cb00b6050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:03:05.049 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.042+0000 7f0cb5d2b640 1 --2- 172.21.15.57:0/22743678 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0cb00cf340 0x7f0cb00b6050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:05.049 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.042+0000 7f0cb5d2b640 1 -- 172.21.15.57:0/22743678 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ca0009560 con 0x7f0cb015b2a0 2023-10-19T13:03:05.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.042+0000 7f0cb5d2b640 1 --2- 172.21.15.57:0/22743678 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0cb015b2a0 0x7f0cb006b3c0 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7f0cac00c7d0 tx=0x7f0cac00cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:03:05.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.042+0000 7f0ca5ffb640 1 -- 172.21.15.57:0/22743678 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cac003b70 con 0x7f0cb015b2a0 2023-10-19T13:03:05.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.042+0000 7f0ca5ffb640 1 -- 172.21.15.57:0/22743678 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f0cac003cf0 con 0x7f0cb015b2a0 2023-10-19T13:03:05.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.042+0000 7f0cb778e640 1 -- 172.21.15.57:0/22743678 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cb015cd50 con 0x7f0cb015b2a0 2023-10-19T13:03:05.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.042+0000 7f0ca5ffb640 1 -- 172.21.15.57:0/22743678 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cac003ea0 con 0x7f0cb015b2a0 2023-10-19T13:03:05.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.042+0000 7f0cb778e640 1 -- 172.21.15.57:0/22743678 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cb015d310 con 0x7f0cb015b2a0 2023-10-19T13:03:05.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.043+0000 7f0cb778e640 1 -- 172.21.15.57:0/22743678 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cb015d810 con 0x7f0cb015b2a0 2023-10-19T13:03:05.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.044+0000 7f0ca5ffb640 1 -- 172.21.15.57:0/22743678 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 58) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f0cac00aab0 con 0x7f0cb015b2a0 2023-10-19T13:03:05.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.044+0000 7f0ca5ffb640 1 --2- 172.21.15.57:0/22743678 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f0c9406af60 0x7f0c9406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:03:05.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.044+0000 7f0ca5ffb640 1 -- 172.21.15.57:0/22743678 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f0cac0898d0 con 0x7f0cb015b2a0 2023-10-19T13:03:05.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.044+0000 7f0cb552a640 1 --2- 172.21.15.57:0/22743678 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f0c9406af60 0x7f0c9406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:03:05.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.045+0000 7f0cb552a640 1 --2- 172.21.15.57:0/22743678 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f0c9406af60 0x7f0c9406d420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f0ca8005fd0 tx=0x7f0ca8005850 comp rx=0 tx=0).ready entity=mgr.7540 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:03:05.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.049+0000 7f0ca5ffb640 1 -- 172.21.15.57:0/22743678 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0cac054710 con 0x7f0cb015b2a0 2023-10-19T13:03:05.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.270+0000 7f0cb778e640 1 -- 172.21.15.57:0/22743678 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f0cb00677b0 con 0x7f0cb015b2a0 2023-10-19T13:03:05.398 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.395+0000 7f0ca5ffb640 1 -- 172.21.15.57:0/22743678 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v59) v1 ==== 92+0+0 (secure 0 0 0) 0x7f0cac057d30 con 0x7f0cb015b2a0 2023-10-19T13:03:05.401 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.397+0000 7f0cb778e640 1 -- 172.21.15.57:0/22743678 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f0c9406af60 msgr2=0x7f0c9406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:03:05.401 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.397+0000 7f0cb778e640 1 --2- 172.21.15.57:0/22743678 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f0c9406af60 0x7f0c9406d420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f0ca8005fd0 tx=0x7f0ca8005850 comp rx=0 tx=0).stop 2023-10-19T13:03:05.401 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.397+0000 7f0cb778e640 1 -- 172.21.15.57:0/22743678 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0cb015b2a0 msgr2=0x7f0cb006b3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:03:05.401 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.397+0000 7f0cb778e640 1 --2- 172.21.15.57:0/22743678 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0cb015b2a0 0x7f0cb006b3c0 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7f0cac00c7d0 tx=0x7f0cac00cc90 comp rx=0 tx=0).stop 2023-10-19T13:03:05.401 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.398+0000 7f0cb778e640 1 -- 172.21.15.57:0/22743678 shutdown_connections 2023-10-19T13:03:05.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.398+0000 7f0cb778e640 1 --2- 172.21.15.57:0/22743678 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f0c9406af60 0x7f0c9406d420 secure :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f0ca8005fd0 tx=0x7f0ca8005850 comp rx=0 tx=0).stop 2023-10-19T13:03:05.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.398+0000 7f0cb778e640 1 --2- 172.21.15.57:0/22743678 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0cb015b2a0 0x7f0cb006b3c0 unknown :-1 s=CLOSED pgs=1474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:05.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.398+0000 7f0cb778e640 1 --2- 172.21.15.57:0/22743678 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0cb00cf340 0x7f0cb00b6050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:05.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.398+0000 7f0cb778e640 1 --2- 172.21.15.57:0/22743678 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0cb00ceb40 0x7f0cb00aef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:05.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.398+0000 7f0cb778e640 1 -- 172.21.15.57:0/22743678 >> 172.21.15.57:0/22743678 conn(0x7f0cb00d0d60 msgr2=0x7f0cb00ae180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:03:05.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.398+0000 7f0cb778e640 1 -- 172.21.15.57:0/22743678 shutdown_connections 2023-10-19T13:03:05.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.398+0000 7f0cb778e640 1 -- 172.21.15.57:0/22743678 wait complete. 2023-10-19T13:03:05.413 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-19T13:03:05.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.525+0000 7feb10e64640 1 -- 172.21.15.57:0/1757010956 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feaf4003ef0 msgr2=0x7feaf40047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:03:05.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.525+0000 7feb10e64640 1 --2- 172.21.15.57:0/1757010956 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feaf4003ef0 0x7feaf40047c0 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7feafc009b50 tx=0x7feaf40057f0 comp rx=0 tx=0).stop 2023-10-19T13:03:05.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.525+0000 7feb0ad9d640 1 -- 172.21.15.57:0/1757010956 reap_dead start 2023-10-19T13:03:05.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.525+0000 7feb10e64640 1 -- 172.21.15.57:0/1757010956 shutdown_connections 2023-10-19T13:03:05.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.525+0000 7feb10e64640 1 -- 172.21.15.57:0/1757010956 >> 172.21.15.57:0/1757010956 conn(0x7feb0c09a030 msgr2=0x7feb0c0d0d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:03:05.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.525+0000 7feb10e64640 1 -- 172.21.15.57:0/1757010956 shutdown_connections 2023-10-19T13:03:05.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.525+0000 7feb10e64640 1 -- 172.21.15.57:0/1757010956 wait complete. 2023-10-19T13:03:05.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.526+0000 7feb10e64640 1 Processor -- start 2023-10-19T13:03:05.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.526+0000 7feb10e64640 1 -- start start 2023-10-19T13:03:05.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.527+0000 7feb10e64640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feaf4003ef0 0x7feb0c09aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:03:05.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.527+0000 7feb10e64640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feb0c09afe0 0x7feb0c15e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:03:05.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.527+0000 7feb10e64640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7feb0c15e630 0x7feb0c14a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:03:05.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.527+0000 7feb10e64640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb0c09e1f0 con 0x7feaf4003ef0 2023-10-19T13:03:05.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.527+0000 7feb10e64640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb0c09d5e0 con 0x7feb0c09afe0 2023-10-19T13:03:05.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.527+0000 7feb10e64640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb0c09d8c0 con 0x7feb0c15e630 2023-10-19T13:03:05.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.527+0000 7feb0a59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feaf4003ef0 0x7feb0c09aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:03:05.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.527+0000 7feb0a59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feaf4003ef0 0x7feb0c09aaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:48788/0 (socket says 172.21.15.57:48788) 2023-10-19T13:03:05.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.527+0000 7feb0a59c640 1 -- 172.21.15.57:0/422892803 learned_addr learned my addr 172.21.15.57:0/422892803 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:03:05.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.527+0000 7feb09d9b640 1 --2- 172.21.15.57:0/422892803 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feb0c09afe0 0x7feb0c15e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:03:05.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.527+0000 7feb0ad9d640 1 --2- 172.21.15.57:0/422892803 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7feb0c15e630 0x7feb0c14a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:03:05.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.528+0000 7feb0a59c640 1 -- 172.21.15.57:0/422892803 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7feb0c15e630 msgr2=0x7feb0c14a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:03:05.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.528+0000 7feb0a59c640 1 --2- 172.21.15.57:0/422892803 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7feb0c15e630 0x7feb0c14a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:05.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.528+0000 7feb0a59c640 1 -- 172.21.15.57:0/422892803 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feb0c09afe0 msgr2=0x7feb0c15e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:03:05.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.528+0000 7feb0a59c640 1 --2- 172.21.15.57:0/422892803 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feb0c09afe0 0x7feb0c15e0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:05.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.528+0000 7feb0a59c640 1 -- 172.21.15.57:0/422892803 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb0c09dba0 con 0x7feaf4003ef0 2023-10-19T13:03:05.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.528+0000 7feb0a59c640 1 --2- 172.21.15.57:0/422892803 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feaf4003ef0 0x7feb0c09aaa0 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7feb00006680 tx=0x7feb000066b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:03:05.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.528+0000 7feafaffd640 1 -- 172.21.15.57:0/422892803 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb0000f960 con 0x7feaf4003ef0 2023-10-19T13:03:05.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.528+0000 7feafaffd640 1 -- 172.21.15.57:0/422892803 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7feb0000fac0 con 0x7feaf4003ef0 2023-10-19T13:03:05.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.528+0000 7feafaffd640 1 -- 172.21.15.57:0/422892803 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb0000fed0 con 0x7feaf4003ef0 2023-10-19T13:03:05.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.528+0000 7feb10e64640 1 -- 172.21.15.57:0/422892803 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb0c09de80 con 0x7feaf4003ef0 2023-10-19T13:03:05.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.528+0000 7feb10e64640 1 -- 172.21.15.57:0/422892803 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb0c09d390 con 0x7feaf4003ef0 2023-10-19T13:03:05.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.529+0000 7feb10e64640 1 -- 172.21.15.57:0/422892803 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb0c0675f0 con 0x7feaf4003ef0 2023-10-19T13:03:05.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.530+0000 7feafaffd640 1 -- 172.21.15.57:0/422892803 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 59) v1 ==== 89348+0+0 (secure 0 0 0) 0x7feb00011430 con 0x7feaf4003ef0 2023-10-19T13:03:05.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.530+0000 7feafaffd640 1 --2- 172.21.15.57:0/422892803 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7feaec06afb0 0x7feaec06d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:03:05.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.530+0000 7feafaffd640 1 -- 172.21.15.57:0/422892803 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6057+0+0 (secure 0 0 0) 0x7feb0008a890 con 0x7feaf4003ef0 2023-10-19T13:03:05.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.530+0000 7feb09d9b640 1 -- 172.21.15.57:0/422892803 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7feaec06afb0 msgr2=0x7feaec06d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/4155586759 2023-10-19T13:03:05.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.530+0000 7feb09d9b640 1 --2- 172.21.15.57:0/422892803 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7feaec06afb0 0x7feaec06d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T13:03:05.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.534+0000 7feafaffd640 1 -- 172.21.15.57:0/422892803 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7feb00055670 con 0x7feaf4003ef0 2023-10-19T13:03:05.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.712+0000 7feb10e64640 1 -- 172.21.15.57:0/422892803 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7feb0c06ba90 con 0x7feaf4003ef0 2023-10-19T13:03:05.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.718+0000 7feafaffd640 1 -- 172.21.15.57:0/422892803 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v52)=0 v52) v1 ==== 138+0+0 (secure 0 0 0) 0x7feb00020020 con 0x7feaf4003ef0 2023-10-19T13:03:05.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.720+0000 7feb10e64640 1 -- 172.21.15.57:0/422892803 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7feaec06afb0 msgr2=0x7feaec06d470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T13:03:05.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.720+0000 7feb10e64640 1 --2- 172.21.15.57:0/422892803 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7feaec06afb0 0x7feaec06d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:05.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.720+0000 7feb10e64640 1 -- 172.21.15.57:0/422892803 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feaf4003ef0 msgr2=0x7feb0c09aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:03:05.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.720+0000 7feb10e64640 1 --2- 172.21.15.57:0/422892803 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feaf4003ef0 0x7feb0c09aaa0 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7feb00006680 tx=0x7feb000066b0 comp rx=0 tx=0).stop 2023-10-19T13:03:05.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.720+0000 7feb10e64640 1 -- 172.21.15.57:0/422892803 shutdown_connections 2023-10-19T13:03:05.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.720+0000 7feb10e64640 1 --2- 172.21.15.57:0/422892803 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7feaec06afb0 0x7feaec06d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:05.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.720+0000 7feb10e64640 1 --2- 172.21.15.57:0/422892803 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7feb0c15e630 0x7feb0c14a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:05.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.720+0000 7feb10e64640 1 --2- 172.21.15.57:0/422892803 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feb0c09afe0 0x7feb0c15e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:05.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.720+0000 7feb10e64640 1 --2- 172.21.15.57:0/422892803 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feaf4003ef0 0x7feb0c09aaa0 unknown :-1 s=CLOSED pgs=1478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:05.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.720+0000 7feb10e64640 1 -- 172.21.15.57:0/422892803 >> 172.21.15.57:0/422892803 conn(0x7feb0c09a030 msgr2=0x7feb0c06a7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:03:05.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.721+0000 7feb10e64640 1 -- 172.21.15.57:0/422892803 shutdown_connections 2023-10-19T13:03:05.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.721+0000 7feb10e64640 1 -- 172.21.15.57:0/422892803 wait complete. 2023-10-19T13:03:05.735 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-19T13:03:05.781 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:03:05.776+0000 7f4acb652040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-19T13:03:05.800 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:03:05.797+0000 7f2383c2e040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-19T13:03:05.823 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:03:05.821+0000 7f4acb652040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-19T13:03:05.848 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:03:05.846+0000 7f2383c2e040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-19T13:03:05.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.846+0000 7f81795a9640 1 -- 172.21.15.57:0/3647177986 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f81740b1ca0 msgr2=0x7f81740b42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:03:05.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.846+0000 7f81795a9640 1 --2- 172.21.15.57:0/3647177986 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f81740b1ca0 0x7f81740b42d0 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7f816c005fd0 tx=0x7f816c02c2e0 comp rx=0 tx=0).stop 2023-10-19T13:03:05.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.846+0000 7f81795a9640 1 -- 172.21.15.57:0/3647177986 shutdown_connections 2023-10-19T13:03:05.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.846+0000 7f81795a9640 1 --2- 172.21.15.57:0/3647177986 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f81740b1ca0 0x7f81740b42d0 unknown :-1 s=CLOSED pgs=1479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:05.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.846+0000 7f81795a9640 1 --2- 172.21.15.57:0/3647177986 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f81740cf8b0 0x7f81740ce9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:05.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.846+0000 7f81795a9640 1 -- 172.21.15.57:0/3647177986 >> 172.21.15.57:0/3647177986 conn(0x7f81740d0c50 msgr2=0x7f81740d1060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:03:05.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.846+0000 7f81795a9640 1 -- 172.21.15.57:0/3647177986 shutdown_connections 2023-10-19T13:03:05.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.846+0000 7f81795a9640 1 -- 172.21.15.57:0/3647177986 wait complete. 2023-10-19T13:03:05.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.847+0000 7f81795a9640 1 Processor -- start 2023-10-19T13:03:05.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.847+0000 7f81795a9640 1 -- start start 2023-10-19T13:03:05.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.848+0000 7f81795a9640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f81740b1ca0 0x7f8174146bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:03:05.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.848+0000 7f81795a9640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f81740cf8b0 0x7f8174152f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:03:05.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.848+0000 7f81795a9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f817415e5e0 0x7f8174145ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:03:05.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.848+0000 7f81795a9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8174155630 con 0x7f817415e5e0 2023-10-19T13:03:05.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.848+0000 7f81795a9640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8174155880 con 0x7f81740cf8b0 2023-10-19T13:03:05.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.848+0000 7f81795a9640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8174155b60 con 0x7f81740b1ca0 2023-10-19T13:03:05.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.848+0000 7f81737fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f817415e5e0 0x7f8174145ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:03:05.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.848+0000 7f81737fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f817415e5e0 0x7f8174145ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:48808/0 (socket says 172.21.15.57:48808) 2023-10-19T13:03:05.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.848+0000 7f81737fe640 1 -- 172.21.15.57:0/1938112250 learned_addr learned my addr 172.21.15.57:0/1938112250 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:03:05.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.848+0000 7f8172ffd640 1 --2- 172.21.15.57:0/1938112250 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f81740b1ca0 0x7f8174146bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:03:05.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.848+0000 7f81727fc640 1 --2- 172.21.15.57:0/1938112250 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f81740cf8b0 0x7f8174152f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:03:05.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.849+0000 7f81737fe640 1 -- 172.21.15.57:0/1938112250 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f81740b1ca0 msgr2=0x7f8174146bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:03:05.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.849+0000 7f81737fe640 1 --2- 172.21.15.57:0/1938112250 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f81740b1ca0 0x7f8174146bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:05.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.849+0000 7f81737fe640 1 -- 172.21.15.57:0/1938112250 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f81740cf8b0 msgr2=0x7f8174152f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:03:05.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.849+0000 7f81737fe640 1 --2- 172.21.15.57:0/1938112250 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f81740cf8b0 0x7f8174152f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:05.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.849+0000 7f81737fe640 1 -- 172.21.15.57:0/1938112250 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f816c009900 con 0x7f817415e5e0 2023-10-19T13:03:05.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.849+0000 7f81737fe640 1 --2- 172.21.15.57:0/1938112250 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f817415e5e0 0x7f8174145ff0 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7f816c037040 tx=0x7f816c035ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:03:05.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.849+0000 7f815b7fe640 1 -- 172.21.15.57:0/1938112250 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f816c004240 con 0x7f817415e5e0 2023-10-19T13:03:05.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.849+0000 7f815b7fe640 1 -- 172.21.15.57:0/1938112250 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f816c004490 con 0x7f817415e5e0 2023-10-19T13:03:05.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.849+0000 7f81795a9640 1 -- 172.21.15.57:0/1938112250 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8174155e40 con 0x7f817415e5e0 2023-10-19T13:03:05.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.849+0000 7f815b7fe640 1 -- 172.21.15.57:0/1938112250 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f816c004820 con 0x7f817415e5e0 2023-10-19T13:03:05.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.849+0000 7f81795a9640 1 -- 172.21.15.57:0/1938112250 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8174153460 con 0x7f817415e5e0 2023-10-19T13:03:05.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.850+0000 7f815b7fe640 1 -- 172.21.15.57:0/1938112250 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 59) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f816c03b040 con 0x7f817415e5e0 2023-10-19T13:03:05.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.850+0000 7f81795a9640 1 -- 172.21.15.57:0/1938112250 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81740c8ce0 con 0x7f817415e5e0 2023-10-19T13:03:05.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.851+0000 7f815b7fe640 1 --2- 172.21.15.57:0/1938112250 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f816006afb0 0x7f816006d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:03:05.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.851+0000 7f8172ffd640 1 -- 172.21.15.57:0/1938112250 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f816006afb0 msgr2=0x7f816006d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/4155586759 2023-10-19T13:03:05.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.851+0000 7f815b7fe640 1 -- 172.21.15.57:0/1938112250 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f816c0ac360 con 0x7f817415e5e0 2023-10-19T13:03:05.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.851+0000 7f8172ffd640 1 --2- 172.21.15.57:0/1938112250 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f816006afb0 0x7f816006d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T13:03:05.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:05.855+0000 7f815b7fe640 1 -- 172.21.15.57:0/1938112250 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f816c077240 con 0x7f817415e5e0 2023-10-19T13:03:06.055 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:06.052+0000 7f8172ffd640 1 -- 172.21.15.57:0/1938112250 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f816006afb0 msgr2=0x7f816006d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/4155586759 2023-10-19T13:03:06.055 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:06.052+0000 7f8172ffd640 1 --2- 172.21.15.57:0/1938112250 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f816006afb0 0x7f816006d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-19T13:03:06.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:06.074+0000 7f81795a9640 1 -- 172.21.15.57:0/1938112250 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f81740690e0 con 0x7f817415e5e0 2023-10-19T13:03:06.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:06.077+0000 7f815b7fe640 1 -- 172.21.15.57:0/1938112250 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v53)=0 v53) v1 ==== 133+0+0 (secure 0 0 0) 0x7f816c07a860 con 0x7f817415e5e0 2023-10-19T13:03:06.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:06.079+0000 7f81795a9640 1 -- 172.21.15.57:0/1938112250 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f816006afb0 msgr2=0x7f816006d470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T13:03:06.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:06.079+0000 7f81795a9640 1 --2- 172.21.15.57:0/1938112250 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f816006afb0 0x7f816006d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:06.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:06.079+0000 7f81795a9640 1 -- 172.21.15.57:0/1938112250 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f817415e5e0 msgr2=0x7f8174145ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:03:06.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:06.079+0000 7f81795a9640 1 --2- 172.21.15.57:0/1938112250 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f817415e5e0 0x7f8174145ff0 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7f816c037040 tx=0x7f816c035ad0 comp rx=0 tx=0).stop 2023-10-19T13:03:06.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:06.079+0000 7f81795a9640 1 -- 172.21.15.57:0/1938112250 shutdown_connections 2023-10-19T13:03:06.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:06.079+0000 7f81795a9640 1 --2- 172.21.15.57:0/1938112250 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7f816006afb0 0x7f816006d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:06.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:06.079+0000 7f81795a9640 1 --2- 172.21.15.57:0/1938112250 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f817415e5e0 0x7f8174145ff0 unknown :-1 s=CLOSED pgs=1480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:06.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:06.079+0000 7f81795a9640 1 --2- 172.21.15.57:0/1938112250 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f81740cf8b0 0x7f8174152f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:06.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:06.079+0000 7f81795a9640 1 --2- 172.21.15.57:0/1938112250 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f81740b1ca0 0x7f8174146bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:06.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:06.079+0000 7f81795a9640 1 -- 172.21.15.57:0/1938112250 >> 172.21.15.57:0/1938112250 conn(0x7f81740d0c50 msgr2=0x7f81740d1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:03:06.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:06.079+0000 7f81795a9640 1 -- 172.21.15.57:0/1938112250 shutdown_connections 2023-10-19T13:03:06.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:06.079+0000 7f81795a9640 1 -- 172.21.15.57:0/1938112250 wait complete. 2023-10-19T13:03:06.093 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:03:06.093 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-10-19T13:03:06.113 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 1m 2023-10-19T13:03:06.213 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:03:06.211+0000 7f4acb652040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-19T13:03:06.260 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:03:06.257+0000 7f4acb652040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-19T13:03:06.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:06.266+0000 7fdf9253c640 1 -- 172.21.15.57:0/331847849 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdf8c0cde90 msgr2=0x7fdf8c0b11b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:03:06.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:06.266+0000 7fdf9253c640 1 --2- 172.21.15.57:0/331847849 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdf8c0cde90 0x7fdf8c0b11b0 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7fdf84005fd0 tx=0x7fdf8402c0e0 comp rx=0 tx=0).stop 2023-10-19T13:03:06.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:06.266+0000 7fdf9253c640 1 -- 172.21.15.57:0/331847849 shutdown_connections 2023-10-19T13:03:06.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:06.266+0000 7fdf9253c640 1 --2- 172.21.15.57:0/331847849 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdf8c0cde90 0x7fdf8c0b11b0 unknown :-1 s=CLOSED pgs=1482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:06.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:06.266+0000 7fdf9253c640 1 --2- 172.21.15.57:0/331847849 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdf8c15af80 0x7fdf8c0d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:06.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:06.266+0000 7fdf9253c640 1 -- 172.21.15.57:0/331847849 >> 172.21.15.57:0/331847849 conn(0x7fdf8c0d0d90 msgr2=0x7fdf8c0d11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:03:06.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:06.266+0000 7fdf9253c640 1 -- 172.21.15.57:0/331847849 shutdown_connections 2023-10-19T13:03:06.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:06.266+0000 7fdf9253c640 1 -- 172.21.15.57:0/331847849 wait complete. 2023-10-19T13:03:06.272 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:06.267+0000 7fdf9253c640 1 Processor -- start 2023-10-19T13:03:06.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:06.267+0000 7fdf9253c640 1 -- start start 2023-10-19T13:03:06.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:06.268+0000 7fdf9253c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdf8c0cde90 0x7fdf8c158280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:03:06.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:06.268+0000 7fdf9253c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdf8c0cf320 0x7fdf8c1e54e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:03:06.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:06.268+0000 7fdf9253c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdf8c15af80 0x7fdf8c1e5a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:03:06.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:06.268+0000 7fdf9253c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf8c0c96d0 con 0x7fdf8c0cde90 2023-10-19T13:03:06.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:06.268+0000 7fdf9253c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf8c1e3e60 con 0x7fdf8c0cf320 2023-10-19T13:03:06.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:06.268+0000 7fdf9253c640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf8c1e4140 con 0x7fdf8c15af80 2023-10-19T13:03:06.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:06.268+0000 7fdf8bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdf8c0cde90 0x7fdf8c158280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:03:06.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:06.268+0000 7fdf8bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdf8c0cde90 0x7fdf8c158280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:48830/0 (socket says 172.21.15.57:48830) 2023-10-19T13:03:06.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:06.268+0000 7fdf8bfff640 1 -- 172.21.15.57:0/2282615863 learned_addr learned my addr 172.21.15.57:0/2282615863 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:03:06.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:06.268+0000 7fdf90ad9640 1 --2- 172.21.15.57:0/2282615863 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdf8c15af80 0x7fdf8c1e5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:03:06.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:06.268+0000 7fdf8b7fe640 1 --2- 172.21.15.57:0/2282615863 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdf8c0cf320 0x7fdf8c1e54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:03:06.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:06.268+0000 7fdf8bfff640 1 -- 172.21.15.57:0/2282615863 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdf8c15af80 msgr2=0x7fdf8c1e5a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:03:06.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:06.268+0000 7fdf8bfff640 1 --2- 172.21.15.57:0/2282615863 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdf8c15af80 0x7fdf8c1e5a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:06.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:06.268+0000 7fdf8bfff640 1 -- 172.21.15.57:0/2282615863 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdf8c0cf320 msgr2=0x7fdf8c1e54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:03:06.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:06.268+0000 7fdf8bfff640 1 --2- 172.21.15.57:0/2282615863 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdf8c0cf320 0x7fdf8c1e54e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:06.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:06.269+0000 7fdf8bfff640 1 -- 172.21.15.57:0/2282615863 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf84009840 con 0x7fdf8c0cde90 2023-10-19T13:03:06.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:06.269+0000 7fdf8bfff640 1 --2- 172.21.15.57:0/2282615863 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdf8c0cde90 0x7fdf8c158280 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7fdf8000e790 tx=0x7fdf8000ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:03:06.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:06.269+0000 7fdf88ff9640 1 -- 172.21.15.57:0/2282615863 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf80009580 con 0x7fdf8c0cde90 2023-10-19T13:03:06.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:06.269+0000 7fdf88ff9640 1 -- 172.21.15.57:0/2282615863 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fdf8000cdb0 con 0x7fdf8c0cde90 2023-10-19T13:03:06.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:06.269+0000 7fdf9253c640 1 -- 172.21.15.57:0/2282615863 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf8c1e4420 con 0x7fdf8c0cde90 2023-10-19T13:03:06.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:06.269+0000 7fdf88ff9640 1 -- 172.21.15.57:0/2282615863 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf80010690 con 0x7fdf8c0cde90 2023-10-19T13:03:06.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:06.269+0000 7fdf9253c640 1 -- 172.21.15.57:0/2282615863 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf8c1e49e0 con 0x7fdf8c0cde90 2023-10-19T13:03:06.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:06.270+0000 7fdf9253c640 1 -- 172.21.15.57:0/2282615863 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf8c066960 con 0x7fdf8c0cde90 2023-10-19T13:03:06.278 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:06.271+0000 7fdf88ff9640 1 -- 172.21.15.57:0/2282615863 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 59) v1 ==== 89348+0+0 (secure 0 0 0) 0x7fdf800107f0 con 0x7fdf8c0cde90 2023-10-19T13:03:06.278 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:06.271+0000 7fdf88ff9640 1 --2- 172.21.15.57:0/2282615863 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7fdf7806afb0 0x7fdf7806d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:03:06.278 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:06.271+0000 7fdf88ff9640 1 -- 172.21.15.57:0/2282615863 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fdf80014070 con 0x7fdf8c0cde90 2023-10-19T13:03:06.279 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:06.271+0000 7fdf8b7fe640 1 -- 172.21.15.57:0/2282615863 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7fdf7806afb0 msgr2=0x7fdf7806d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/4155586759 2023-10-19T13:03:06.279 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:06.271+0000 7fdf8b7fe640 1 --2- 172.21.15.57:0/2282615863 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7fdf7806afb0 0x7fdf7806d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T13:03:06.279 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:06.275+0000 7fdf88ff9640 1 -- 172.21.15.57:0/2282615863 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fdf80051ca0 con 0x7fdf8c0cde90 2023-10-19T13:03:06.301 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:03:06.299+0000 7f2383c2e040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-19T13:03:06.304 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:03:06.301+0000 7f4acb652040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-19T13:03:06.348 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:03:06.345+0000 7f2383c2e040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-19T13:03:06.355 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:03:06.352+0000 7f4acb652040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-19T13:03:06.393 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:03:06.391+0000 7f2383c2e040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-19T13:03:06.396 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:03:06.393+0000 7f4acb652040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-19T13:03:06.449 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:03:06.446+0000 7f2383c2e040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-19T13:03:06.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:06.472+0000 7fdf8b7fe640 1 -- 172.21.15.57:0/2282615863 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7fdf7806afb0 msgr2=0x7fdf7806d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/4155586759 2023-10-19T13:03:06.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:06.472+0000 7fdf8b7fe640 1 --2- 172.21.15.57:0/2282615863 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7fdf7806afb0 0x7fdf7806d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-19T13:03:06.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:06.473+0000 7fdf9253c640 1 -- 172.21.15.57:0/2282615863 --> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fdf8c0b42e0 con 0x7fdf7806afb0 2023-10-19T13:03:06.493 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:03:06.490+0000 7f2383c2e040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-19T13:03:06.495 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:03:06.493+0000 7f4acb652040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-19T13:03:06.547 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:03:06.545+0000 7f4acb652040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-19T13:03:06.589 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:03:06.586+0000 7f4acb652040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-19T13:03:06.596 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:03:06.593+0000 7f2383c2e040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-19T13:03:06.636 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:03:06.633+0000 7f4acb652040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-19T13:03:06.651 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:03:06.649+0000 7f2383c2e040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-19T13:03:06.677 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:03:06.675+0000 7f4acb652040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-19T13:03:06.695 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:03:06.693+0000 7f2383c2e040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-19T13:03:06.746 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:03:06.743+0000 7f2383c2e040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-19T13:03:06.790 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:03:06.788+0000 7f2383c2e040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-19T13:03:06.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:06.873+0000 7fdf8b7fe640 1 -- 172.21.15.57:0/2282615863 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7fdf7806afb0 msgr2=0x7fdf7806d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/4155586759 2023-10-19T13:03:06.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:06.873+0000 7fdf8b7fe640 1 --2- 172.21.15.57:0/2282615863 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7fdf7806afb0 0x7fdf7806d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-19T13:03:06.992 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:03:06.989+0000 7f4acb652040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-19T13:03:07.121 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:03:07.118+0000 7f4acb652040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-19T13:03:07.122 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:03:07.119+0000 7f2383c2e040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-19T13:03:07.212 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:03:07.209+0000 7f4acb652040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-19T13:03:07.254 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:03:07.251+0000 7f4acb652040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-19T13:03:07.260 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:03:07.257+0000 7f2383c2e040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-19T13:03:07.348 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:03:07.346+0000 7f4acb652040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-19T13:03:07.356 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:03:07.354+0000 7f2383c2e040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-19T13:03:07.401 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:03:07.398+0000 7f2383c2e040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-19T13:03:07.439 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:03:07.436+0000 7f4acb652040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-19T13:03:07.500 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:03:07.498+0000 7f2383c2e040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-19T13:03:07.573 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:03:07.570+0000 7f4acb652040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-19T13:03:07.595 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:03:07.593+0000 7f2383c2e040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-19T13:03:07.619 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:03:07.612+0000 7f4acb652040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-19T13:03:07.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:07.674+0000 7fdf8b7fe640 1 -- 172.21.15.57:0/2282615863 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7fdf7806afb0 msgr2=0x7fdf7806d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/4155586759 2023-10-19T13:03:07.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:07.674+0000 7fdf8b7fe640 1 --2- 172.21.15.57:0/2282615863 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7fdf7806afb0 0x7fdf7806d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-19T13:03:07.719 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:03:07.716+0000 7f4acb652040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-19T13:03:07.739 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:03:07.736+0000 7f2383c2e040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-19T13:03:07.782 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:03:07.780+0000 7f2383c2e040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-19T13:03:07.892 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:03:07.890+0000 7f2383c2e040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-19T13:03:08.341 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:03:08.338+0000 7f4acb652040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-19T13:03:08.542 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:03:08.539+0000 7f2383c2e040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-19T13:03:08.602 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:03:08.599+0000 7f4acb652040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-19T13:03:08.650 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:03:08.648+0000 7f4acb652040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-19T13:03:08.749 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:03:08.747+0000 7f4acb652040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-19T13:03:08.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:08.770+0000 7fdf88ff9640 1 -- 172.21.15.57:0/2282615863 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mgrmap(e 60) v1 ==== 88957+0+0 (secure 0 0 0) 0x7fdf80088820 con 0x7fdf8c0cde90 2023-10-19T13:03:08.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:08.770+0000 7fdf88ff9640 1 -- 172.21.15.57:0/2282615863 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7fdf7806afb0 msgr2=0x7fdf7806d470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T13:03:08.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:08.770+0000 7fdf88ff9640 1 --2- 172.21.15.57:0/2282615863 >> [v2:172.21.15.179:6832/4155586759,v1:172.21.15.179:6833/4155586759] conn(0x7fdf7806afb0 0x7fdf7806d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:08.821 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:03:08.818+0000 7f2383c2e040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-19T13:03:08.869 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:03:08.866+0000 7f4a48166640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:03:08.869 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:03:08.866+0000 7f4a48166640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:03:08.870 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:03:08.866+0000 7f4a48166640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:03:08.870 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:03:08.866+0000 7f4a48166640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:03:08.870 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:03:08.866+0000 7f4a48166640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:03:08.870 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:03:08.866+0000 7f4a4b1ac640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:03:08.870 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:03:08.866+0000 7f4a4b1ac640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:03:08.871 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:03:08.866+0000 7f4a4b1ac640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:03:08.871 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:03:08.866+0000 7f4a4b1ac640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:03:08.871 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:03:08.866+0000 7f4a4b1ac640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:03:08.875 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:03:08.873+0000 7f2383c2e040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-19T13:03:08.982 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:03:08.979+0000 7f2383c2e040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-19T13:03:09.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:09.777+0000 7fdf88ff9640 1 -- 172.21.15.57:0/2282615863 <== mon.0 v2:172.21.15.57:3300/0 8 ==== mgrmap(e 61) v1 ==== 89084+0+0 (secure 0 0 0) 0x7fdf80070970 con 0x7fdf8c0cde90 2023-10-19T13:03:09.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:09.777+0000 7fdf88ff9640 1 --2- 172.21.15.57:0/2282615863 >> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] conn(0x7fdf78035850 0x7fdf780753e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:03:09.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:09.777+0000 7fdf88ff9640 1 -- 172.21.15.57:0/2282615863 --> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fdf8c0b42e0 con 0x7fdf78035850 2023-10-19T13:03:09.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:09.779+0000 7fdf8b7fe640 1 --2- 172.21.15.57:0/2282615863 >> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] conn(0x7fdf78035850 0x7fdf780753e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:03:09.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:09.783+0000 7fdf8b7fe640 1 --2- 172.21.15.57:0/2282615863 >> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] conn(0x7fdf78035850 0x7fdf780753e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fdf8c1e31e0 tx=0x7fdf7c007d30 comp rx=0 tx=0).ready entity=mgr.7091 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:03:09.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:09.844+0000 7fdf88ff9640 1 -- 172.21.15.57:0/2282615863 <== mgr.7091 v2:172.21.15.179:6832/1210673354 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fdf8c0b42e0 con 0x7fdf78035850 2023-10-19T13:03:09.847 INFO:teuthology.orchestra.run.smithi057.stdout:Schedule set for path snap_test_dir1 2023-10-19T13:03:09.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:09.846+0000 7fdf9253c640 1 -- 172.21.15.57:0/2282615863 >> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] conn(0x7fdf78035850 msgr2=0x7fdf780753e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:03:09.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:09.846+0000 7fdf9253c640 1 --2- 172.21.15.57:0/2282615863 >> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] conn(0x7fdf78035850 0x7fdf780753e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fdf8c1e31e0 tx=0x7fdf7c007d30 comp rx=0 tx=0).stop 2023-10-19T13:03:09.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:09.846+0000 7fdf9253c640 1 -- 172.21.15.57:0/2282615863 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdf8c0cde90 msgr2=0x7fdf8c158280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:03:09.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:09.846+0000 7fdf9253c640 1 --2- 172.21.15.57:0/2282615863 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdf8c0cde90 0x7fdf8c158280 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7fdf8000e790 tx=0x7fdf8000ec50 comp rx=0 tx=0).stop 2023-10-19T13:03:09.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:09.846+0000 7fdf90ad9640 1 -- 172.21.15.57:0/2282615863 reap_dead start 2023-10-19T13:03:09.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:09.846+0000 7fdf9253c640 1 -- 172.21.15.57:0/2282615863 shutdown_connections 2023-10-19T13:03:09.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:09.846+0000 7fdf9253c640 1 -- 172.21.15.57:0/2282615863 >> 172.21.15.57:0/2282615863 conn(0x7fdf8c0d0d90 msgr2=0x7fdf8c15a0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:03:09.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:09.846+0000 7fdf9253c640 1 -- 172.21.15.57:0/2282615863 shutdown_connections 2023-10-19T13:03:09.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:09.846+0000 7fdf9253c640 1 -- 172.21.15.57:0/2282615863 wait complete. 2023-10-19T13:03:09.861 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2023-10-19T13:03:09.862 DEBUG:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2023-10-19-13_04_00 in ~51s... 2023-10-19T13:03:09.862 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1m'], expected_retention: [] 2023-10-19T13:03:09.863 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --format json snap_test_dir1 2023-10-19T13:03:09.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:09.975+0000 7f0e0c496640 1 -- 172.21.15.57:0/946068691 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0dec0047c0 msgr2=0x7f0dec004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:03:09.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:09.975+0000 7f0e0c496640 1 --2- 172.21.15.57:0/946068691 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0dec0047c0 0x7f0dec004c40 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7f0df8009b50 tx=0x7f0dec0057f0 comp rx=0 tx=0).stop 2023-10-19T13:03:09.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:09.975+0000 7f0e0c496640 1 -- 172.21.15.57:0/946068691 shutdown_connections 2023-10-19T13:03:09.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:09.975+0000 7f0e0c496640 1 --2- 172.21.15.57:0/946068691 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0dec0047c0 0x7f0dec004c40 unknown :-1 s=CLOSED pgs=1485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:09.979 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:09.975+0000 7f0e0c496640 1 -- 172.21.15.57:0/946068691 >> 172.21.15.57:0/946068691 conn(0x7f0e04069930 msgr2=0x7f0e04069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:03:09.979 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:09.975+0000 7f0e0c496640 1 -- 172.21.15.57:0/946068691 shutdown_connections 2023-10-19T13:03:09.979 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:09.975+0000 7f0e0c496640 1 -- 172.21.15.57:0/946068691 wait complete. 2023-10-19T13:03:09.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:09.976+0000 7f0e0c496640 1 Processor -- start 2023-10-19T13:03:09.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:09.976+0000 7f0e0c496640 1 -- start start 2023-10-19T13:03:09.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:09.977+0000 7f0e0c496640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0dec0047c0 0x7f0e041391f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:03:09.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:09.977+0000 7f0e0c496640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0e040ad480 0x7f0e040aa8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:03:09.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:09.977+0000 7f0e0c496640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0e040aae00 0x7f0e0406f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:03:09.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:09.977+0000 7f0e0c496640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e040ada10 con 0x7f0e040aae00 2023-10-19T13:03:09.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:09.977+0000 7f0e0c496640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e040ab280 con 0x7f0dec0047c0 2023-10-19T13:03:09.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:09.977+0000 7f0e0c496640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e040703d0 con 0x7f0e040ad480 2023-10-19T13:03:09.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:09.977+0000 7f0e0aa33640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0e040aae00 0x7f0e0406f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:03:09.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:09.977+0000 7f0e0aa33640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0e040aae00 0x7f0e0406f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:54686/0 (socket says 172.21.15.57:54686) 2023-10-19T13:03:09.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:09.977+0000 7f0e0aa33640 1 -- 172.21.15.57:0/3801110040 learned_addr learned my addr 172.21.15.57:0/3801110040 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:03:09.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:09.977+0000 7f0e0a232640 1 --2- 172.21.15.57:0/3801110040 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0dec0047c0 0x7f0e041391f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:03:09.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:09.977+0000 7f0e09a31640 1 --2- 172.21.15.57:0/3801110040 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0e040ad480 0x7f0e040aa8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:03:09.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:09.977+0000 7f0e0aa33640 1 -- 172.21.15.57:0/3801110040 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0e040ad480 msgr2=0x7f0e040aa8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:03:09.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:09.977+0000 7f0e0aa33640 1 --2- 172.21.15.57:0/3801110040 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0e040ad480 0x7f0e040aa8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:09.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:09.977+0000 7f0e0aa33640 1 -- 172.21.15.57:0/3801110040 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0dec0047c0 msgr2=0x7f0e041391f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:03:09.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:09.977+0000 7f0e0aa33640 1 --2- 172.21.15.57:0/3801110040 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0dec0047c0 0x7f0e041391f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:09.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:09.977+0000 7f0e0aa33640 1 -- 172.21.15.57:0/3801110040 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e04152380 con 0x7f0e040aae00 2023-10-19T13:03:09.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:09.977+0000 7f0e0a232640 1 --2- 172.21.15.57:0/3801110040 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0dec0047c0 0x7f0e041391f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T13:03:09.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:09.977+0000 7f0e0aa33640 1 --2- 172.21.15.57:0/3801110040 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0e040aae00 0x7f0e0406f080 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7f0e040ca3f0 tx=0x7f0df40064f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:03:09.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:09.977+0000 7f0e02ffd640 1 -- 172.21.15.57:0/3801110040 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0df4011900 con 0x7f0e040aae00 2023-10-19T13:03:09.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:09.977+0000 7f0e02ffd640 1 -- 172.21.15.57:0/3801110040 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f0df4011aa0 con 0x7f0e040aae00 2023-10-19T13:03:09.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:09.977+0000 7f0e02ffd640 1 -- 172.21.15.57:0/3801110040 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0df400f430 con 0x7f0e040aae00 2023-10-19T13:03:09.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:09.977+0000 7f0e0c496640 1 -- 172.21.15.57:0/3801110040 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e040ae8a0 con 0x7f0e040aae00 2023-10-19T13:03:09.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:09.977+0000 7f0e0c496640 1 -- 172.21.15.57:0/3801110040 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e04070ed0 con 0x7f0e040aae00 2023-10-19T13:03:09.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:09.978+0000 7f0e0c496640 1 -- 172.21.15.57:0/3801110040 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e04152930 con 0x7f0e040aae00 2023-10-19T13:03:09.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:09.979+0000 7f0e02ffd640 1 -- 172.21.15.57:0/3801110040 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 61) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f0df400f590 con 0x7f0e040aae00 2023-10-19T13:03:09.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:09.979+0000 7f0e02ffd640 1 --2- 172.21.15.57:0/3801110040 >> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] conn(0x7f0df006ace0 0x7f0df006d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:03:09.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:09.979+0000 7f0e02ffd640 1 -- 172.21.15.57:0/3801110040 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f0df408a260 con 0x7f0e040aae00 2023-10-19T13:03:09.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:09.979+0000 7f0e0a232640 1 --2- 172.21.15.57:0/3801110040 >> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] conn(0x7f0df006ace0 0x7f0df006d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:03:09.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:09.980+0000 7f0e0a232640 1 --2- 172.21.15.57:0/3801110040 >> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] conn(0x7f0df006ace0 0x7f0df006d1a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc006fd0 tx=0x7f0dfc006b30 comp rx=0 tx=0).ready entity=mgr.7091 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:03:09.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:09.983+0000 7f0e02ffd640 1 -- 172.21.15.57:0/3801110040 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0df4055440 con 0x7f0e040aae00 2023-10-19T13:03:10.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:10.178+0000 7f0e0c496640 1 -- 172.21.15.57:0/3801110040 --> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0e04139730 con 0x7f0df006ace0 2023-10-19T13:03:10.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:10.185+0000 7f0e02ffd640 1 -- 172.21.15.57:0/3801110040 <== mgr.7091 v2:172.21.15.179:6832/1210673354 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+66 (secure 0 0 0) 0x7f0e04139730 con 0x7f0df006ace0 2023-10-19T13:03:10.188 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:03:10.188 INFO:teuthology.orchestra.run.smithi057.stdout:{"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{}]} 2023-10-19T13:03:10.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:10.187+0000 7f0e0c496640 1 -- 172.21.15.57:0/3801110040 >> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] conn(0x7f0df006ace0 msgr2=0x7f0df006d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:03:10.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:10.187+0000 7f0e0c496640 1 --2- 172.21.15.57:0/3801110040 >> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] conn(0x7f0df006ace0 0x7f0df006d1a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc006fd0 tx=0x7f0dfc006b30 comp rx=0 tx=0).stop 2023-10-19T13:03:10.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:10.187+0000 7f0e0c496640 1 -- 172.21.15.57:0/3801110040 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0e040aae00 msgr2=0x7f0e0406f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:03:10.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:10.187+0000 7f0e0c496640 1 --2- 172.21.15.57:0/3801110040 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0e040aae00 0x7f0e0406f080 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7f0e040ca3f0 tx=0x7f0df40064f0 comp rx=0 tx=0).stop 2023-10-19T13:03:10.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:10.187+0000 7f0e0c496640 1 -- 172.21.15.57:0/3801110040 shutdown_connections 2023-10-19T13:03:10.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:10.187+0000 7f0e0c496640 1 --2- 172.21.15.57:0/3801110040 >> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] conn(0x7f0df006ace0 0x7f0df006d1a0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:10.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:10.187+0000 7f0e0c496640 1 --2- 172.21.15.57:0/3801110040 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0e040aae00 0x7f0e0406f080 unknown :-1 s=CLOSED pgs=1486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:10.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:10.187+0000 7f0e0c496640 1 --2- 172.21.15.57:0/3801110040 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0e040ad480 0x7f0e040aa8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:10.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:10.187+0000 7f0e0c496640 1 --2- 172.21.15.57:0/3801110040 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0dec0047c0 0x7f0e041391f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:03:10.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:10.187+0000 7f0e0c496640 1 -- 172.21.15.57:0/3801110040 >> 172.21.15.57:0/3801110040 conn(0x7f0e04069930 msgr2=0x7f0e04069d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:03:10.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:10.187+0000 7f0e0c496640 1 -- 172.21.15.57:0/3801110040 shutdown_connections 2023-10-19T13:03:10.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:03:10.187+0000 7f0e0c496640 1 -- 172.21.15.57:0/3801110040 wait complete. 2023-10-19T13:03:10.201 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{}]} 2023-10-19T13:03:10.201 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': 'snap_test_dir1', 'schedule': ['1m'], 'retention': [{}]} 2023-10-19T13:03:10.201 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:03:10.202 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T13:03:10.222 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T13:03:10.223 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T13:03:10.223 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T13:03:11.224 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:03:11.224 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T13:03:11.249 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T13:03:11.249 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T13:03:11.250 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T13:03:12.251 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:03:12.252 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T13:03:12.276 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T13:03:12.277 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T13:03:12.277 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T13:03:13.278 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:03:13.279 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T13:03:13.303 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T13:03:13.304 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T13:03:13.304 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T13:03:14.306 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:03:14.306 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T13:03:14.331 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T13:03:14.331 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T13:03:14.331 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T13:03:15.332 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:03:15.333 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T13:03:15.357 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T13:03:15.358 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T13:03:15.358 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T13:03:16.359 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:03:16.359 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T13:03:16.384 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T13:03:16.384 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T13:03:16.385 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T13:03:17.385 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:03:17.386 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T13:03:17.411 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T13:03:17.411 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T13:03:17.411 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T13:03:18.140 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:03:18.143 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:03:18.412 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:03:18.413 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T13:03:18.437 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T13:03:18.438 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T13:03:18.438 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T13:03:19.439 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:03:19.440 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T13:03:19.465 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T13:03:19.465 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T13:03:19.465 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T13:03:20.466 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:03:20.467 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T13:03:20.491 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T13:03:20.491 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T13:03:20.492 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T13:03:21.492 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:03:21.493 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T13:03:21.517 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T13:03:21.517 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T13:03:21.517 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T13:03:22.518 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:03:22.519 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T13:03:22.543 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T13:03:22.543 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T13:03:22.543 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T13:03:23.544 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:03:23.545 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T13:03:23.570 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T13:03:23.571 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T13:03:23.571 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T13:03:24.571 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:03:24.572 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T13:03:24.597 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T13:03:24.597 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T13:03:24.597 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T13:03:25.598 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:03:25.599 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T13:03:25.624 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T13:03:25.624 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T13:03:25.625 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T13:03:26.626 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:03:26.627 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T13:03:26.650 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T13:03:26.651 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T13:03:26.651 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T13:03:27.652 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:03:27.653 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T13:03:27.678 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T13:03:27.678 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T13:03:27.678 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T13:03:28.680 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:03:28.680 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T13:03:28.706 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T13:03:28.706 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T13:03:28.707 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T13:03:29.708 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:03:29.708 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T13:03:29.733 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T13:03:29.733 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T13:03:29.733 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T13:03:30.734 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:03:30.735 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T13:03:30.762 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T13:03:30.762 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T13:03:30.762 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T13:03:31.764 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:03:31.764 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T13:03:31.789 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T13:03:31.790 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T13:03:31.790 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T13:03:32.791 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:03:32.792 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T13:03:32.818 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T13:03:32.818 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T13:03:32.819 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T13:03:33.819 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:03:33.820 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T13:03:33.845 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T13:03:33.845 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T13:03:33.846 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T13:03:34.847 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:03:34.848 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T13:03:34.872 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T13:03:34.872 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T13:03:34.873 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T13:03:35.874 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:03:35.875 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T13:03:35.899 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T13:03:35.900 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T13:03:35.900 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T13:03:36.901 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:03:36.902 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T13:03:36.928 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T13:03:36.929 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T13:03:36.929 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T13:03:37.930 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:03:37.931 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T13:03:37.955 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T13:03:37.956 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T13:03:37.956 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T13:03:38.957 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:03:38.958 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T13:03:38.983 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T13:03:38.983 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T13:03:38.983 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T13:03:39.985 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:03:39.985 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T13:03:40.010 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T13:03:40.010 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T13:03:40.011 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T13:03:41.012 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:03:41.013 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T13:03:41.038 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T13:03:41.038 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T13:03:41.039 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T13:03:42.040 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:03:42.040 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T13:03:42.065 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T13:03:42.066 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T13:03:42.066 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T13:03:43.068 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:03:43.068 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T13:03:43.093 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T13:03:43.093 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T13:03:43.094 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T13:03:44.094 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:03:44.095 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T13:03:44.120 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T13:03:44.121 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T13:03:44.121 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T13:03:45.122 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:03:45.122 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T13:03:45.147 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T13:03:45.147 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T13:03:45.148 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T13:03:46.149 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:03:46.150 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T13:03:46.175 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T13:03:46.175 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T13:03:46.176 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T13:03:47.177 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:03:47.177 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T13:03:47.203 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T13:03:47.203 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T13:03:47.203 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T13:03:48.205 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:03:48.205 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T13:03:48.230 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T13:03:48.230 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T13:03:48.231 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T13:03:48.246 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:03:48.273 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:03:49.232 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:03:49.233 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T13:03:49.258 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T13:03:49.258 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T13:03:49.258 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T13:03:50.259 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:03:50.260 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T13:03:50.284 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T13:03:50.284 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T13:03:50.285 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T13:03:51.286 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:03:51.286 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T13:03:51.310 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T13:03:51.310 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T13:03:51.311 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T13:03:52.312 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:03:52.312 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T13:03:52.336 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T13:03:52.337 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T13:03:52.337 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T13:03:53.338 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:03:53.339 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T13:03:53.364 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T13:03:53.364 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T13:03:53.364 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T13:03:54.365 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:03:54.366 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T13:03:54.390 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T13:03:54.390 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T13:03:54.391 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T13:03:55.392 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:03:55.392 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T13:03:55.416 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T13:03:55.416 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T13:03:55.417 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T13:03:56.417 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:03:56.418 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T13:03:56.442 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T13:03:56.443 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T13:03:56.443 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T13:03:57.444 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:03:57.445 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T13:03:57.469 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T13:03:57.469 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T13:03:57.470 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T13:03:58.471 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:03:58.472 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T13:03:58.496 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T13:03:58.496 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T13:03:58.497 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T13:03:59.498 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:03:59.499 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T13:03:59.523 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T13:03:59.523 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T13:03:59.523 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T13:04:00.524 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:04:00.525 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T13:04:00.550 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T13:04:00.550 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T13:04:00.550 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T13:04:00.857 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:00.854+0000 7f4a3a94b640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:04:00.857 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:00.854+0000 7f4a3a94b640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:04:00.857 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:00.854+0000 7f4a3a94b640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:04:00.858 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:00.854+0000 7f4a3a94b640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:04:00.858 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:00.854+0000 7f4a3a94b640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:04:01.551 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:04:01.552 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T13:04:01.576 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_04_00_UTC 2023-10-19T13:04:01.576 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-13_04_00_UTC'} 2023-10-19T13:04:01.577 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2023-10-19-13_04_00_UTC'} 2023-10-19T13:04:01.577 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T13:04:01.577 DEBUG:tasks.cephfs.test_snap_schedules:snapshots added=['scheduled-2023-10-19-13_04_00_UTC'] 2023-10-19T13:04:01.578 DEBUG:tasks.cephfs.test_snap_schedules:exec=1697720589.8623085, now = 1697720641.5781865, timo = 51 2023-10-19T13:04:02.578 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove snap_test_dir1 2023-10-19T13:04:02.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:02.701+0000 7f6d6cda9640 1 -- 172.21.15.57:0/3767838446 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6d6806af00 msgr2=0x7f6d6806b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:02.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:02.701+0000 7f6d6cda9640 1 --2- 172.21.15.57:0/3767838446 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6d6806af00 0x7f6d6806b2e0 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7f6d58005fd0 tx=0x7f6d5802bd90 comp rx=0 tx=0).stop 2023-10-19T13:04:02.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:02.701+0000 7f6d6cda9640 1 -- 172.21.15.57:0/3767838446 shutdown_connections 2023-10-19T13:04:02.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:02.701+0000 7f6d6cda9640 1 --2- 172.21.15.57:0/3767838446 >> v2:172.21.15.179:3300/0 conn(0x7f6d680c96b0 0x7f6d680c9b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:02.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:02.701+0000 7f6d6cda9640 1 --2- 172.21.15.57:0/3767838446 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6d6806af00 0x7f6d6806b2e0 unknown :-1 s=CLOSED pgs=1487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:02.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:02.701+0000 7f6d6cda9640 1 -- 172.21.15.57:0/3767838446 >> 172.21.15.57:0/3767838446 conn(0x7f6d68069930 msgr2=0x7f6d68069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:02.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:02.701+0000 7f6d6cda9640 1 -- 172.21.15.57:0/3767838446 shutdown_connections 2023-10-19T13:04:02.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:02.701+0000 7f6d6cda9640 1 -- 172.21.15.57:0/3767838446 wait complete. 2023-10-19T13:04:02.707 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:02.702+0000 7f6d6cda9640 1 Processor -- start 2023-10-19T13:04:02.707 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:02.702+0000 7f6d6cda9640 1 -- start start 2023-10-19T13:04:02.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:02.703+0000 7f6d6cda9640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6d6806af00 0x7f6d68151de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:02.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:02.703+0000 7f6d6cda9640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6d680ab500 0x7f6d6815c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:02.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:02.703+0000 7f6d6cda9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6d680c96b0 0x7f6d6815cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:02.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:02.703+0000 7f6d6cda9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d68152320 con 0x7f6d680c96b0 2023-10-19T13:04:02.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:02.703+0000 7f6d6cda9640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d680ab2e0 con 0x7f6d680ab500 2023-10-19T13:04:02.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:02.703+0000 7f6d6cda9640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d680aaa40 con 0x7f6d6806af00 2023-10-19T13:04:02.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:02.703+0000 7f6d66d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6d680c96b0 0x7f6d6815cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:02.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:02.703+0000 7f6d66d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6d680c96b0 0x7f6d6815cb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34610/0 (socket says 172.21.15.57:34610) 2023-10-19T13:04:02.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:02.703+0000 7f6d66d9d640 1 -- 172.21.15.57:0/3510241284 learned_addr learned my addr 172.21.15.57:0/3510241284 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:02.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:02.703+0000 7f6d5ffff640 1 --2- 172.21.15.57:0/3510241284 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6d680ab500 0x7f6d6815c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:02.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:02.703+0000 7f6d6659c640 1 --2- 172.21.15.57:0/3510241284 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6d6806af00 0x7f6d68151de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:02.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:02.703+0000 7f6d66d9d640 1 -- 172.21.15.57:0/3510241284 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6d6806af00 msgr2=0x7f6d68151de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:02.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:02.703+0000 7f6d66d9d640 1 --2- 172.21.15.57:0/3510241284 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6d6806af00 0x7f6d68151de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:02.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:02.703+0000 7f6d66d9d640 1 -- 172.21.15.57:0/3510241284 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6d680ab500 msgr2=0x7f6d6815c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:02.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:02.703+0000 7f6d66d9d640 1 --2- 172.21.15.57:0/3510241284 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6d680ab500 0x7f6d6815c620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:02.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:02.703+0000 7f6d66d9d640 1 -- 172.21.15.57:0/3510241284 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d58009560 con 0x7f6d680c96b0 2023-10-19T13:04:02.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:02.703+0000 7f6d5ffff640 1 --2- 172.21.15.57:0/3510241284 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6d680ab500 0x7f6d6815c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:04:02.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:02.704+0000 7f6d66d9d640 1 --2- 172.21.15.57:0/3510241284 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6d680c96b0 0x7f6d6815cb60 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f6d6000c7a0 tx=0x7f6d6000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:02.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:02.704+0000 7f6d5effd640 1 -- 172.21.15.57:0/3510241284 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d60003b70 con 0x7f6d680c96b0 2023-10-19T13:04:02.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:02.704+0000 7f6d5effd640 1 -- 172.21.15.57:0/3510241284 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f6d60003cf0 con 0x7f6d680c96b0 2023-10-19T13:04:02.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:02.704+0000 7f6d6cda9640 1 -- 172.21.15.57:0/3510241284 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d680aad20 con 0x7f6d680c96b0 2023-10-19T13:04:02.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:02.704+0000 7f6d5effd640 1 -- 172.21.15.57:0/3510241284 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d6000aab0 con 0x7f6d680c96b0 2023-10-19T13:04:02.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:02.704+0000 7f6d6cda9640 1 -- 172.21.15.57:0/3510241284 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d680a6550 con 0x7f6d680c96b0 2023-10-19T13:04:02.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:02.705+0000 7f6d6cda9640 1 -- 172.21.15.57:0/3510241284 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d680690e0 con 0x7f6d680c96b0 2023-10-19T13:04:02.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:02.705+0000 7f6d5effd640 1 -- 172.21.15.57:0/3510241284 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 64) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f6d6000ac10 con 0x7f6d680c96b0 2023-10-19T13:04:02.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:02.706+0000 7f6d5effd640 1 --2- 172.21.15.57:0/3510241284 >> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] conn(0x7f6d4806b0d0 0x7f6d4806d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:02.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:02.706+0000 7f6d5effd640 1 -- 172.21.15.57:0/3510241284 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f6d60089d00 con 0x7f6d680c96b0 2023-10-19T13:04:02.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:02.706+0000 7f6d6659c640 1 --2- 172.21.15.57:0/3510241284 >> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] conn(0x7f6d4806b0d0 0x7f6d4806d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:02.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:02.706+0000 7f6d6659c640 1 --2- 172.21.15.57:0/3510241284 >> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] conn(0x7f6d4806b0d0 0x7f6d4806d590 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f6d58035000 tx=0x7f6d58004730 comp rx=0 tx=0).ready entity=mgr.7091 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:02.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:02.710+0000 7f6d5effd640 1 -- 172.21.15.57:0/3510241284 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6d60054820 con 0x7f6d680c96b0 2023-10-19T13:04:02.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:02.909+0000 7f6d6cda9640 1 -- 172.21.15.57:0/3510241284 --> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f6d68159c50 con 0x7f6d4806b0d0 2023-10-19T13:04:02.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:02.969+0000 7f6d5effd640 1 -- 172.21.15.57:0/3510241284 <== mgr.7091 v2:172.21.15.179:6832/1210673354 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f6d58004210 con 0x7f6d4806b0d0 2023-10-19T13:04:02.972 INFO:teuthology.orchestra.run.smithi057.stdout:Schedule removed for path snap_test_dir1 2023-10-19T13:04:02.974 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:02.971+0000 7f6d6cda9640 1 -- 172.21.15.57:0/3510241284 >> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] conn(0x7f6d4806b0d0 msgr2=0x7f6d4806d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:02.974 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:02.971+0000 7f6d6cda9640 1 --2- 172.21.15.57:0/3510241284 >> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] conn(0x7f6d4806b0d0 0x7f6d4806d590 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f6d58035000 tx=0x7f6d58004730 comp rx=0 tx=0).stop 2023-10-19T13:04:02.974 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:02.971+0000 7f6d6cda9640 1 -- 172.21.15.57:0/3510241284 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6d680c96b0 msgr2=0x7f6d6815cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:02.974 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:02.971+0000 7f6d6cda9640 1 --2- 172.21.15.57:0/3510241284 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6d680c96b0 0x7f6d6815cb60 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f6d6000c7a0 tx=0x7f6d6000cc60 comp rx=0 tx=0).stop 2023-10-19T13:04:02.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:02.971+0000 7f6d6cda9640 1 -- 172.21.15.57:0/3510241284 shutdown_connections 2023-10-19T13:04:02.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:02.971+0000 7f6d6cda9640 1 --2- 172.21.15.57:0/3510241284 >> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] conn(0x7f6d4806b0d0 0x7f6d4806d590 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:02.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:02.971+0000 7f6d6cda9640 1 --2- 172.21.15.57:0/3510241284 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6d680c96b0 0x7f6d6815cb60 unknown :-1 s=CLOSED pgs=1488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:02.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:02.971+0000 7f6d6cda9640 1 --2- 172.21.15.57:0/3510241284 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6d680ab500 0x7f6d6815c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:02.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:02.971+0000 7f6d6cda9640 1 --2- 172.21.15.57:0/3510241284 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6d6806af00 0x7f6d68151de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:02.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:02.971+0000 7f6d6cda9640 1 -- 172.21.15.57:0/3510241284 >> 172.21.15.57:0/3510241284 conn(0x7f6d68069930 msgr2=0x7f6d68069d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:02.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:02.971+0000 7f6d6cda9640 1 -- 172.21.15.57:0/3510241284 shutdown_connections 2023-10-19T13:04:02.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:02.971+0000 7f6d6cda9640 1 -- 172.21.15.57:0/3510241284 wait complete. 2023-10-19T13:04:02.984 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2023-10-19T13:04:02.984 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:04:02.985 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T13:04:03.004 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_04_00_UTC 2023-10-19T13:04:03.004 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-19-13_04_00_UTC 2023-10-19T13:04:03.005 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:04:03.005 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-19-13_04_00_UTC) 2023-10-19T13:04:03.068 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:04:03.068 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2023-10-19T13:04:03.127 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-19T13:04:03.279 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:03.276+0000 7fe14c090640 1 -- 172.21.15.57:0/3596027920 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1300047c0 msgr2=0x7fe130004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:03.280 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:03.276+0000 7fe14c090640 1 --2- 172.21.15.57:0/3596027920 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1300047c0 0x7fe130004c40 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7fe134009b50 tx=0x7fe1300057f0 comp rx=0 tx=0).stop 2023-10-19T13:04:03.280 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:03.276+0000 7fe14a62d640 1 -- 172.21.15.57:0/3596027920 reap_dead start 2023-10-19T13:04:03.280 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:03.276+0000 7fe14c090640 1 -- 172.21.15.57:0/3596027920 shutdown_connections 2023-10-19T13:04:03.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:03.276+0000 7fe14c090640 1 -- 172.21.15.57:0/3596027920 >> 172.21.15.57:0/3596027920 conn(0x7fe144069930 msgr2=0x7fe144069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:03.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:03.277+0000 7fe14c090640 1 -- 172.21.15.57:0/3596027920 shutdown_connections 2023-10-19T13:04:03.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:03.277+0000 7fe14c090640 1 -- 172.21.15.57:0/3596027920 wait complete. 2023-10-19T13:04:03.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:03.278+0000 7fe14c090640 1 Processor -- start 2023-10-19T13:04:03.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:03.278+0000 7fe14c090640 1 -- start start 2023-10-19T13:04:03.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:03.278+0000 7fe14c090640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1300047c0 0x7fe144156690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:03.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:03.278+0000 7fe14c090640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe1441e0c90 0x7fe14406bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:03.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:03.278+0000 7fe14c090640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe14413bc90 0x7fe144159f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:03.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:03.278+0000 7fe14c090640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe144159af0 con 0x7fe1300047c0 2023-10-19T13:04:03.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:03.278+0000 7fe14c090640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1441e1190 con 0x7fe14413bc90 2023-10-19T13:04:03.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:03.278+0000 7fe14c090640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe144159430 con 0x7fe1441e0c90 2023-10-19T13:04:03.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:03.279+0000 7fe149e2c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1300047c0 0x7fe144156690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:03.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:03.279+0000 7fe149e2c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1300047c0 0x7fe144156690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34618/0 (socket says 172.21.15.57:34618) 2023-10-19T13:04:03.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:03.279+0000 7fe149e2c640 1 -- 172.21.15.57:0/2199560173 learned_addr learned my addr 172.21.15.57:0/2199560173 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:03.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:03.279+0000 7fe14962b640 1 --2- 172.21.15.57:0/2199560173 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe1441e0c90 0x7fe14406bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:03.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:03.279+0000 7fe14a62d640 1 --2- 172.21.15.57:0/2199560173 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe14413bc90 0x7fe144159f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:03.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:03.279+0000 7fe149e2c640 1 -- 172.21.15.57:0/2199560173 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe1441e0c90 msgr2=0x7fe14406bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:03.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:03.279+0000 7fe149e2c640 1 --2- 172.21.15.57:0/2199560173 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe1441e0c90 0x7fe14406bcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:03.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:03.279+0000 7fe149e2c640 1 -- 172.21.15.57:0/2199560173 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe14413bc90 msgr2=0x7fe144159f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:03.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:03.279+0000 7fe149e2c640 1 --2- 172.21.15.57:0/2199560173 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe14413bc90 0x7fe144159f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:03.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:03.279+0000 7fe149e2c640 1 -- 172.21.15.57:0/2199560173 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe144159710 con 0x7fe1300047c0 2023-10-19T13:04:03.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:03.279+0000 7fe149e2c640 1 --2- 172.21.15.57:0/2199560173 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1300047c0 0x7fe144156690 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7fe140006680 tx=0x7fe1400039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:03.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:03.279+0000 7fe13a7fc640 1 -- 172.21.15.57:0/2199560173 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe140011900 con 0x7fe1300047c0 2023-10-19T13:04:03.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:03.279+0000 7fe13a7fc640 1 -- 172.21.15.57:0/2199560173 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fe140011a60 con 0x7fe1300047c0 2023-10-19T13:04:03.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:03.279+0000 7fe14c090640 1 -- 172.21.15.57:0/2199560173 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1441e1470 con 0x7fe1300047c0 2023-10-19T13:04:03.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:03.279+0000 7fe13a7fc640 1 -- 172.21.15.57:0/2199560173 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe140011e70 con 0x7fe1300047c0 2023-10-19T13:04:03.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:03.279+0000 7fe14c090640 1 -- 172.21.15.57:0/2199560173 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe14406c200 con 0x7fe1300047c0 2023-10-19T13:04:03.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:03.280+0000 7fe14c090640 1 -- 172.21.15.57:0/2199560173 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe14406c3f0 con 0x7fe1300047c0 2023-10-19T13:04:03.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:03.281+0000 7fe13a7fc640 1 -- 172.21.15.57:0/2199560173 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 64) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fe140010420 con 0x7fe1300047c0 2023-10-19T13:04:03.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:03.281+0000 7fe13a7fc640 1 --2- 172.21.15.57:0/2199560173 >> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] conn(0x7fe12806b0d0 0x7fe12806d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:03.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:03.281+0000 7fe13a7fc640 1 -- 172.21.15.57:0/2199560173 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6401+0+0 (secure 0 0 0) 0x7fe14008abf0 con 0x7fe1300047c0 2023-10-19T13:04:03.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:03.281+0000 7fe14962b640 1 --2- 172.21.15.57:0/2199560173 >> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] conn(0x7fe12806b0d0 0x7fe12806d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:03.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:03.282+0000 7fe14962b640 1 --2- 172.21.15.57:0/2199560173 >> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] conn(0x7fe12806b0d0 0x7fe12806d590 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe134036000 tx=0x7fe1340046c0 comp rx=0 tx=0).ready entity=mgr.7091 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:03.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:03.285+0000 7fe13a7fc640 1 -- 172.21.15.57:0/2199560173 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe14406c3f0 con 0x7fe1300047c0 2023-10-19T13:04:03.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:03.510+0000 7fe14c090640 1 -- 172.21.15.57:0/2199560173 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7fe144159c30 con 0x7fe1300047c0 2023-10-19T13:04:04.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.334+0000 7fe13a7fc640 1 -- 172.21.15.57:0/2199560173 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v65) v1 ==== 93+0+0 (secure 0 0 0) 0x7fe144159c30 con 0x7fe1300047c0 2023-10-19T13:04:04.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.337+0000 7fe14c090640 1 -- 172.21.15.57:0/2199560173 >> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] conn(0x7fe12806b0d0 msgr2=0x7fe12806d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:04.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.337+0000 7fe14c090640 1 --2- 172.21.15.57:0/2199560173 >> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] conn(0x7fe12806b0d0 0x7fe12806d590 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe134036000 tx=0x7fe1340046c0 comp rx=0 tx=0).stop 2023-10-19T13:04:04.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.337+0000 7fe14c090640 1 -- 172.21.15.57:0/2199560173 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1300047c0 msgr2=0x7fe144156690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:04.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.337+0000 7fe14c090640 1 --2- 172.21.15.57:0/2199560173 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1300047c0 0x7fe144156690 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7fe140006680 tx=0x7fe1400039c0 comp rx=0 tx=0).stop 2023-10-19T13:04:04.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.337+0000 7fe14c090640 1 -- 172.21.15.57:0/2199560173 shutdown_connections 2023-10-19T13:04:04.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.337+0000 7fe14c090640 1 --2- 172.21.15.57:0/2199560173 >> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] conn(0x7fe12806b0d0 0x7fe12806d590 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:04.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.337+0000 7fe14c090640 1 --2- 172.21.15.57:0/2199560173 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe14413bc90 0x7fe144159f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:04.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.337+0000 7fe14c090640 1 --2- 172.21.15.57:0/2199560173 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe1441e0c90 0x7fe14406bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:04.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.337+0000 7fe14c090640 1 --2- 172.21.15.57:0/2199560173 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1300047c0 0x7fe144156690 unknown :-1 s=CLOSED pgs=1491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:04.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.337+0000 7fe14c090640 1 -- 172.21.15.57:0/2199560173 >> 172.21.15.57:0/2199560173 conn(0x7fe144069930 msgr2=0x7fe144069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:04.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.338+0000 7fe14c090640 1 -- 172.21.15.57:0/2199560173 shutdown_connections 2023-10-19T13:04:04.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.338+0000 7fe14c090640 1 -- 172.21.15.57:0/2199560173 wait complete. 2023-10-19T13:04:04.355 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-19T13:04:04.355 DEBUG:teuthology.orchestra.run.smithi057:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-19T13:04:04.404 DEBUG:teuthology.orchestra.run.smithi179:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-19T13:04:04.448 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-19T13:04:04.449 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T13:04:04.449 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T13:04:04.466 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T13:04:04.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.613+0000 7f3ec858e640 1 -- 172.21.15.57:0/2302175427 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3eac003b30 msgr2=0x7f3eac003f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:04.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.613+0000 7f3ec858e640 1 --2- 172.21.15.57:0/2302175427 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3eac003b30 0x7f3eac003f30 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7f3ebc009b50 tx=0x7f3eac0057f0 comp rx=0 tx=0).stop 2023-10-19T13:04:04.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.613+0000 7f3ec6b2b640 1 -- 172.21.15.57:0/2302175427 reap_dead start 2023-10-19T13:04:04.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.613+0000 7f3ec858e640 1 -- 172.21.15.57:0/2302175427 shutdown_connections 2023-10-19T13:04:04.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.613+0000 7f3ec858e640 1 -- 172.21.15.57:0/2302175427 >> 172.21.15.57:0/2302175427 conn(0x7f3ec00d0d60 msgr2=0x7f3ec00d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:04.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.613+0000 7f3ec858e640 1 -- 172.21.15.57:0/2302175427 shutdown_connections 2023-10-19T13:04:04.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.613+0000 7f3ec858e640 1 -- 172.21.15.57:0/2302175427 wait complete. 2023-10-19T13:04:04.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.615+0000 7f3ec858e640 1 Processor -- start 2023-10-19T13:04:04.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.615+0000 7f3ec858e640 1 -- start start 2023-10-19T13:04:04.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.615+0000 7f3ec858e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3eac003b30 0x7f3ec00b5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:04.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.615+0000 7f3ec858e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3ec0158730 0x7f3ec00b3950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:04.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.615+0000 7f3ec858e640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3ec0158b90 0x7f3ec00cc3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:04.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.615+0000 7f3ec858e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec00b43a0 con 0x7f3ec0158730 2023-10-19T13:04:04.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.615+0000 7f3ec858e640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec00b4680 con 0x7f3eac003b30 2023-10-19T13:04:04.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.615+0000 7f3ec858e640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec00b4960 con 0x7f3ec0158b90 2023-10-19T13:04:04.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.615+0000 7f3ec5b29640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3ec0158730 0x7f3ec00b3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:04.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.615+0000 7f3ec5b29640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3ec0158730 0x7f3ec00b3950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34650/0 (socket says 172.21.15.57:34650) 2023-10-19T13:04:04.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.615+0000 7f3ec5b29640 1 -- 172.21.15.57:0/4248980958 learned_addr learned my addr 172.21.15.57:0/4248980958 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:04.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.615+0000 7f3ec632a640 1 --2- 172.21.15.57:0/4248980958 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3eac003b30 0x7f3ec00b5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:04.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.616+0000 7f3ec6b2b640 1 --2- 172.21.15.57:0/4248980958 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3ec0158b90 0x7f3ec00cc3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:04.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.616+0000 7f3ec5b29640 1 -- 172.21.15.57:0/4248980958 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3ec0158b90 msgr2=0x7f3ec00cc3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:04.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.616+0000 7f3ec5b29640 1 --2- 172.21.15.57:0/4248980958 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3ec0158b90 0x7f3ec00cc3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:04.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.616+0000 7f3ec5b29640 1 -- 172.21.15.57:0/4248980958 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3eac003b30 msgr2=0x7f3ec00b5440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:04.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.616+0000 7f3ec5b29640 1 --2- 172.21.15.57:0/4248980958 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3eac003b30 0x7f3ec00b5440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:04.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.616+0000 7f3ec5b29640 1 -- 172.21.15.57:0/4248980958 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ec00b4c40 con 0x7f3ec0158730 2023-10-19T13:04:04.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.616+0000 7f3ec5b29640 1 --2- 172.21.15.57:0/4248980958 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3ec0158730 0x7f3ec00b3950 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7f3eb0003e10 tx=0x7f3eb0001770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:04.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.616+0000 7f3eb6ffd640 1 -- 172.21.15.57:0/4248980958 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eb0011840 con 0x7f3ec0158730 2023-10-19T13:04:04.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.616+0000 7f3eb6ffd640 1 -- 172.21.15.57:0/4248980958 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f3eb00119a0 con 0x7f3ec0158730 2023-10-19T13:04:04.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.616+0000 7f3ec858e640 1 -- 172.21.15.57:0/4248980958 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ec00b4f20 con 0x7f3ec0158730 2023-10-19T13:04:04.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.616+0000 7f3eb6ffd640 1 -- 172.21.15.57:0/4248980958 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eb0011db0 con 0x7f3ec0158730 2023-10-19T13:04:04.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.616+0000 7f3ec858e640 1 -- 172.21.15.57:0/4248980958 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ec00b3e90 con 0x7f3ec0158730 2023-10-19T13:04:04.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.617+0000 7f3ec858e640 1 -- 172.21.15.57:0/4248980958 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ec00b5160 con 0x7f3ec0158730 2023-10-19T13:04:04.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.618+0000 7f3eb6ffd640 1 -- 172.21.15.57:0/4248980958 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 65) v1 ==== 89445+0+0 (secure 0 0 0) 0x7f3eb00103c0 con 0x7f3ec0158730 2023-10-19T13:04:04.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.618+0000 7f3eb6ffd640 1 --2- 172.21.15.57:0/4248980958 >> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] conn(0x7f3ea406b0b0 0x7f3ea406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:04.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.618+0000 7f3ec632a640 1 -- 172.21.15.57:0/4248980958 >> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] conn(0x7f3ea406b0b0 msgr2=0x7f3ea406d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/1210673354 2023-10-19T13:04:04.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.618+0000 7f3eb6ffd640 1 -- 172.21.15.57:0/4248980958 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f3eb008adb0 con 0x7f3ec0158730 2023-10-19T13:04:04.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.619+0000 7f3ec632a640 1 --2- 172.21.15.57:0/4248980958 >> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] conn(0x7f3ea406b0b0 0x7f3ea406d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T13:04:04.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.622+0000 7f3eb6ffd640 1 -- 172.21.15.57:0/4248980958 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3ec00b5160 con 0x7f3ec0158730 2023-10-19T13:04:04.728 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:04.726+0000 7f4bbda14040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-19T13:04:04.746 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:04:04.744+0000 7fdc13e3e040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-19T13:04:04.773 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:04.770+0000 7f4bbda14040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-19T13:04:04.794 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:04:04.791+0000 7fdc13e3e040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-19T13:04:04.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.799+0000 7f3ec858e640 1 -- 172.21.15.57:0/4248980958 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3ec00b4110 con 0x7f3ec0158730 2023-10-19T13:04:04.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.801+0000 7f3eb6ffd640 1 -- 172.21.15.57:0/4248980958 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v131) v1 ==== 74+0+17277 (secure 0 0 0) 0x7f3eb0015070 con 0x7f3ec0158730 2023-10-19T13:04:04.804 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:04:04.804 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":131,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T13:03:08.761584+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2023-10-19T13:02:55.916158+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"129","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2023-10-19T13:02:56.675559+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"130","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.179:6833/4155586759":"2023-10-20T13:03:08.761549+0000","172.21.15.179:6832/4155586759":"2023-10-20T13:03:08.761549+0000","172.21.15.179:0/1341670560":"2023-10-20T13:03:08.761549+0000","172.21.15.179:0/33839263":"2023-10-20T13:03:08.761549+0000","172.21.15.179:0/3689030544":"2023-10-20T13:03:08.761549+0000","172.21.15.179:0/3037001824":"2023-10-20T13:03:08.761549+0000","172.21.15.179:0/1069074991":"2023-10-20T13:03:08.761549+0000","172.21.15.179:0/831719196":"2023-10-20T13:03:08.761549+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T13:04:04.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.803+0000 7f3ec858e640 1 -- 172.21.15.57:0/4248980958 >> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] conn(0x7f3ea406b0b0 msgr2=0x7f3ea406d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T13:04:04.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.803+0000 7f3ec858e640 1 --2- 172.21.15.57:0/4248980958 >> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] conn(0x7f3ea406b0b0 0x7f3ea406d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:04.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.803+0000 7f3ec858e640 1 -- 172.21.15.57:0/4248980958 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3ec0158730 msgr2=0x7f3ec00b3950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:04.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.803+0000 7f3ec858e640 1 --2- 172.21.15.57:0/4248980958 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3ec0158730 0x7f3ec00b3950 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7f3eb0003e10 tx=0x7f3eb0001770 comp rx=0 tx=0).stop 2023-10-19T13:04:04.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.803+0000 7f3ec858e640 1 -- 172.21.15.57:0/4248980958 shutdown_connections 2023-10-19T13:04:04.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.803+0000 7f3ec858e640 1 --2- 172.21.15.57:0/4248980958 >> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] conn(0x7f3ea406b0b0 0x7f3ea406d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:04.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.803+0000 7f3ec858e640 1 --2- 172.21.15.57:0/4248980958 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3ec0158b90 0x7f3ec00cc3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:04.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.803+0000 7f3ec858e640 1 --2- 172.21.15.57:0/4248980958 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3ec0158730 0x7f3ec00b3950 unknown :-1 s=CLOSED pgs=1496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:04.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.803+0000 7f3ec858e640 1 --2- 172.21.15.57:0/4248980958 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3eac003b30 0x7f3ec00b5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:04.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.803+0000 7f3ec858e640 1 -- 172.21.15.57:0/4248980958 >> 172.21.15.57:0/4248980958 conn(0x7f3ec00d0d60 msgr2=0x7f3ec006b650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:04.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.803+0000 7f3ec858e640 1 -- 172.21.15.57:0/4248980958 shutdown_connections 2023-10-19T13:04:04.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.803+0000 7f3ec858e640 1 -- 172.21.15.57:0/4248980958 wait complete. 2023-10-19T13:04:04.817 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-19T12:44:07.371186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-19T13:02:55.916158+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '129', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_data', 'create_time': '2023-10-19T13:02:56.675559+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '130', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-19T13:04:04.817 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-19T13:04:04.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.927+0000 7f3af2839640 1 -- 172.21.15.57:0/380403243 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3aec0ceb40 msgr2=0x7f3aec0b0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:04.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.927+0000 7f3af2839640 1 --2- 172.21.15.57:0/380403243 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3aec0ceb40 0x7f3aec0b0650 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7f3ae4005fd0 tx=0x7f3ae402c240 comp rx=0 tx=0).stop 2023-10-19T13:04:04.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.927+0000 7f3af2839640 1 -- 172.21.15.57:0/380403243 shutdown_connections 2023-10-19T13:04:04.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.927+0000 7f3af2839640 1 --2- 172.21.15.57:0/380403243 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3aec0ceb40 0x7f3aec0b0650 unknown :-1 s=CLOSED pgs=1497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:04.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.927+0000 7f3af2839640 1 --2- 172.21.15.57:0/380403243 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3aec0cf340 0x7f3aec0d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:04.932 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.927+0000 7f3af2839640 1 -- 172.21.15.57:0/380403243 >> 172.21.15.57:0/380403243 conn(0x7f3aec0d0d60 msgr2=0x7f3aec0d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:04.932 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.927+0000 7f3af2839640 1 -- 172.21.15.57:0/380403243 shutdown_connections 2023-10-19T13:04:04.932 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.927+0000 7f3af2839640 1 -- 172.21.15.57:0/380403243 wait complete. 2023-10-19T13:04:04.934 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.929+0000 7f3af2839640 1 Processor -- start 2023-10-19T13:04:04.935 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.929+0000 7f3af2839640 1 -- start start 2023-10-19T13:04:04.935 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.929+0000 7f3af2839640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3aec0ceb40 0x7f3aec06a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:04.935 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.929+0000 7f3af2839640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3aec0cf340 0x7f3aec15e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:04.935 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.929+0000 7f3af2839640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3aec15b2a0 0x7f3aec0c9c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:04.935 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.929+0000 7f3af2839640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aec15d660 con 0x7f3aec15b2a0 2023-10-19T13:04:04.936 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.929+0000 7f3af2839640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aec15cae0 con 0x7f3aec0cf340 2023-10-19T13:04:04.936 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.929+0000 7f3af2839640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3aec15cdc0 con 0x7f3aec0ceb40 2023-10-19T13:04:04.936 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.930+0000 7f3af0dd6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3aec15b2a0 0x7f3aec0c9c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:04.936 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.930+0000 7f3af0dd6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3aec15b2a0 0x7f3aec0c9c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34666/0 (socket says 172.21.15.57:34666) 2023-10-19T13:04:04.937 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.930+0000 7f3af0dd6640 1 -- 172.21.15.57:0/2556799668 learned_addr learned my addr 172.21.15.57:0/2556799668 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:04.937 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.930+0000 7f3aebfff640 1 --2- 172.21.15.57:0/2556799668 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3aec0ceb40 0x7f3aec06a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:04.937 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.930+0000 7f3aeb7fe640 1 --2- 172.21.15.57:0/2556799668 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3aec0cf340 0x7f3aec15e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:04.937 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.930+0000 7f3af0dd6640 1 -- 172.21.15.57:0/2556799668 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3aec0ceb40 msgr2=0x7f3aec06a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:04.937 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.930+0000 7f3af0dd6640 1 --2- 172.21.15.57:0/2556799668 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3aec0ceb40 0x7f3aec06a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:04.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.930+0000 7f3af0dd6640 1 -- 172.21.15.57:0/2556799668 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3aec0cf340 msgr2=0x7f3aec15e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:04.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.930+0000 7f3af0dd6640 1 --2- 172.21.15.57:0/2556799668 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3aec0cf340 0x7f3aec15e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:04.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.930+0000 7f3af0dd6640 1 -- 172.21.15.57:0/2556799668 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ae4009900 con 0x7f3aec15b2a0 2023-10-19T13:04:04.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.930+0000 7f3af0dd6640 1 --2- 172.21.15.57:0/2556799668 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3aec15b2a0 0x7f3aec0c9c00 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f3ae4037040 tx=0x7f3ae4035a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:04.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.930+0000 7f3ae8ff9640 1 -- 172.21.15.57:0/2556799668 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ae4034440 con 0x7f3aec15b2a0 2023-10-19T13:04:04.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.930+0000 7f3ae8ff9640 1 -- 172.21.15.57:0/2556799668 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f3ae4004330 con 0x7f3aec15b2a0 2023-10-19T13:04:04.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.930+0000 7f3af2839640 1 -- 172.21.15.57:0/2556799668 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3aec15d0a0 con 0x7f3aec15b2a0 2023-10-19T13:04:04.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.930+0000 7f3ae8ff9640 1 -- 172.21.15.57:0/2556799668 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ae40046c0 con 0x7f3aec15b2a0 2023-10-19T13:04:04.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.931+0000 7f3af2839640 1 -- 172.21.15.57:0/2556799668 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3aec15c720 con 0x7f3aec15b2a0 2023-10-19T13:04:04.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.931+0000 7f3af2839640 1 -- 172.21.15.57:0/2556799668 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3aec15c8b0 con 0x7f3aec15b2a0 2023-10-19T13:04:04.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.932+0000 7f3ae8ff9640 1 -- 172.21.15.57:0/2556799668 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 65) v1 ==== 89445+0+0 (secure 0 0 0) 0x7f3ae403b040 con 0x7f3aec15b2a0 2023-10-19T13:04:04.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.933+0000 7f3ae8ff9640 1 --2- 172.21.15.57:0/2556799668 >> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] conn(0x7f3acc06b080 0x7f3acc06d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:04.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.933+0000 7f3aebfff640 1 -- 172.21.15.57:0/2556799668 >> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] conn(0x7f3acc06b080 msgr2=0x7f3acc06d540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/1210673354 2023-10-19T13:04:04.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.933+0000 7f3ae8ff9640 1 -- 172.21.15.57:0/2556799668 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f3ae40ac740 con 0x7f3aec15b2a0 2023-10-19T13:04:04.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.933+0000 7f3aebfff640 1 --2- 172.21.15.57:0/2556799668 >> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] conn(0x7f3acc06b080 0x7f3acc06d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T13:04:04.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:04.937+0000 7f3ae8ff9640 1 -- 172.21.15.57:0/2556799668 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3ae40771e0 con 0x7f3aec15b2a0 2023-10-19T13:04:05.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.112+0000 7f3af2839640 1 -- 172.21.15.57:0/2556799668 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3aec0cba00 con 0x7f3aec15b2a0 2023-10-19T13:04:05.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.113+0000 7f3ae8ff9640 1 -- 172.21.15.57:0/2556799668 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v131) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3ae407a800 con 0x7f3aec15b2a0 2023-10-19T13:04:05.116 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-19T13:04:05.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.114+0000 7f3af2839640 1 -- 172.21.15.57:0/2556799668 >> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] conn(0x7f3acc06b080 msgr2=0x7f3acc06d540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T13:04:05.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.115+0000 7f3af2839640 1 --2- 172.21.15.57:0/2556799668 >> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] conn(0x7f3acc06b080 0x7f3acc06d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:05.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.115+0000 7f3af2839640 1 -- 172.21.15.57:0/2556799668 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3aec15b2a0 msgr2=0x7f3aec0c9c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:05.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.115+0000 7f3af2839640 1 --2- 172.21.15.57:0/2556799668 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3aec15b2a0 0x7f3aec0c9c00 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f3ae4037040 tx=0x7f3ae4035a20 comp rx=0 tx=0).stop 2023-10-19T13:04:05.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.115+0000 7f3af2839640 1 -- 172.21.15.57:0/2556799668 shutdown_connections 2023-10-19T13:04:05.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.115+0000 7f3af2839640 1 --2- 172.21.15.57:0/2556799668 >> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] conn(0x7f3acc06b080 0x7f3acc06d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:05.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.115+0000 7f3af2839640 1 --2- 172.21.15.57:0/2556799668 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3aec15b2a0 0x7f3aec0c9c00 unknown :-1 s=CLOSED pgs=1498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:05.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.115+0000 7f3af2839640 1 --2- 172.21.15.57:0/2556799668 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3aec0cf340 0x7f3aec15e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:05.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.115+0000 7f3af2839640 1 --2- 172.21.15.57:0/2556799668 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3aec0ceb40 0x7f3aec06a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:05.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.115+0000 7f3af2839640 1 -- 172.21.15.57:0/2556799668 >> 172.21.15.57:0/2556799668 conn(0x7f3aec0d0d60 msgr2=0x7f3aec0d1140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:05.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.115+0000 7f3af2839640 1 -- 172.21.15.57:0/2556799668 shutdown_connections 2023-10-19T13:04:05.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.115+0000 7f3af2839640 1 -- 172.21.15.57:0/2556799668 wait complete. 2023-10-19T13:04:05.128 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-19T13:04:05.162 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:05.159+0000 7f4bbda14040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-19T13:04:05.205 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:05.202+0000 7f4bbda14040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-19T13:04:05.214 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:04:05.212+0000 7fdc13e3e040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-19T13:04:05.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.241+0000 7fd607421640 1 -- 172.21.15.57:0/1630010353 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5ec003b00 msgr2=0x7fd5ec003f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:05.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.241+0000 7fd607421640 1 --2- 172.21.15.57:0/1630010353 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5ec003b00 0x7fd5ec003f00 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc009d00 tx=0x7fd5ec0057f0 comp rx=0 tx=0).stop 2023-10-19T13:04:05.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.241+0000 7fd607421640 1 -- 172.21.15.57:0/1630010353 shutdown_connections 2023-10-19T13:04:05.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.241+0000 7fd607421640 1 --2- 172.21.15.57:0/1630010353 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5ec003b00 0x7fd5ec003f00 unknown :-1 s=CLOSED pgs=1500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:05.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.241+0000 7fd607421640 1 -- 172.21.15.57:0/1630010353 >> 172.21.15.57:0/1630010353 conn(0x7fd6000d0d60 msgr2=0x7fd6000d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:05.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.241+0000 7fd607421640 1 -- 172.21.15.57:0/1630010353 shutdown_connections 2023-10-19T13:04:05.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.241+0000 7fd607421640 1 -- 172.21.15.57:0/1630010353 wait complete. 2023-10-19T13:04:05.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.242+0000 7fd607421640 1 Processor -- start 2023-10-19T13:04:05.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.242+0000 7fd607421640 1 -- start start 2023-10-19T13:04:05.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.243+0000 7fd607421640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd5ec003b00 0x7fd60014cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:05.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.243+0000 7fd607421640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd6000b16b0 0x7fd60015c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:05.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.243+0000 7fd607421640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd6000b1b30 0x7fd60015c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:05.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.243+0000 7fd607421640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd60015cce0 con 0x7fd6000b1b30 2023-10-19T13:04:05.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.243+0000 7fd607421640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6000c91c0 con 0x7fd5ec003b00 2023-10-19T13:04:05.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.243+0000 7fd607421640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6000c9480 con 0x7fd6000b16b0 2023-10-19T13:04:05.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.243+0000 7fd6059be640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd6000b1b30 0x7fd60015c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:05.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.243+0000 7fd6059be640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd6000b1b30 0x7fd60015c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34694/0 (socket says 172.21.15.57:34694) 2023-10-19T13:04:05.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.243+0000 7fd6059be640 1 -- 172.21.15.57:0/1440032947 learned_addr learned my addr 172.21.15.57:0/1440032947 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:05.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.243+0000 7fd6051bd640 1 --2- 172.21.15.57:0/1440032947 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd5ec003b00 0x7fd60014cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:05.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.243+0000 7fd6049bc640 1 --2- 172.21.15.57:0/1440032947 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd6000b16b0 0x7fd60015c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:05.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.244+0000 7fd6059be640 1 -- 172.21.15.57:0/1440032947 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd6000b16b0 msgr2=0x7fd60015c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:05.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.244+0000 7fd6059be640 1 --2- 172.21.15.57:0/1440032947 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd6000b16b0 0x7fd60015c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:05.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.244+0000 7fd6059be640 1 -- 172.21.15.57:0/1440032947 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd5ec003b00 msgr2=0x7fd60014cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:05.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.244+0000 7fd6059be640 1 --2- 172.21.15.57:0/1440032947 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd5ec003b00 0x7fd60014cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:05.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.244+0000 7fd6059be640 1 -- 172.21.15.57:0/1440032947 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6000c9760 con 0x7fd6000b1b30 2023-10-19T13:04:05.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.244+0000 7fd6059be640 1 --2- 172.21.15.57:0/1440032947 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd6000b1b30 0x7fd60015c7a0 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7fd5f8001330 tx=0x7fd5f8006430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:05.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.244+0000 7fd5f5ffb640 1 -- 172.21.15.57:0/1440032947 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5f80119c0 con 0x7fd6000b1b30 2023-10-19T13:04:05.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.244+0000 7fd5f5ffb640 1 -- 172.21.15.57:0/1440032947 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fd5f8011b20 con 0x7fd6000b1b30 2023-10-19T13:04:05.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.244+0000 7fd607421640 1 -- 172.21.15.57:0/1440032947 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6000c9a40 con 0x7fd6000b1b30 2023-10-19T13:04:05.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.244+0000 7fd5f5ffb640 1 -- 172.21.15.57:0/1440032947 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5f80103c0 con 0x7fd6000b1b30 2023-10-19T13:04:05.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.244+0000 7fd607421640 1 -- 172.21.15.57:0/1440032947 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6000ca240 con 0x7fd6000b1b30 2023-10-19T13:04:05.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.245+0000 7fd607421640 1 -- 172.21.15.57:0/1440032947 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6000c9f60 con 0x7fd6000b1b30 2023-10-19T13:04:05.254 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:05.246+0000 7f4bbda14040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-19T13:04:05.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.246+0000 7fd5f5ffb640 1 -- 172.21.15.57:0/1440032947 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 65) v1 ==== 89445+0+0 (secure 0 0 0) 0x7fd5f8010520 con 0x7fd6000b1b30 2023-10-19T13:04:05.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.246+0000 7fd5f5ffb640 1 --2- 172.21.15.57:0/1440032947 >> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] conn(0x7fd5e406b080 0x7fd5e406d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:05.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.246+0000 7fd6051bd640 1 -- 172.21.15.57:0/1440032947 >> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] conn(0x7fd5e406b080 msgr2=0x7fd5e406d540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/1210673354 2023-10-19T13:04:05.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.246+0000 7fd5f5ffb640 1 -- 172.21.15.57:0/1440032947 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6401+0+0 (secure 0 0 0) 0x7fd5f808a900 con 0x7fd6000b1b30 2023-10-19T13:04:05.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.246+0000 7fd6051bd640 1 --2- 172.21.15.57:0/1440032947 >> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] conn(0x7fd5e406b080 0x7fd5e406d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T13:04:05.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.251+0000 7fd5f5ffb640 1 -- 172.21.15.57:0/1440032947 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd6000c9f60 con 0x7fd6000b1b30 2023-10-19T13:04:05.264 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:04:05.261+0000 7fdc13e3e040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-19T13:04:05.299 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:05.297+0000 7f4bbda14040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-19T13:04:05.309 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:04:05.306+0000 7fdc13e3e040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-19T13:04:05.340 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:05.338+0000 7f4bbda14040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-19T13:04:05.364 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:04:05.362+0000 7fdc13e3e040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-19T13:04:05.408 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:04:05.406+0000 7fdc13e3e040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-19T13:04:05.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.429+0000 7fd607421640 1 -- 172.21.15.57:0/1440032947 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd6000c9f60 con 0x7fd6000b1b30 2023-10-19T13:04:05.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.429+0000 7fd5f5ffb640 1 -- 172.21.15.57:0/1440032947 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v131) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd6000c9f60 con 0x7fd6000b1b30 2023-10-19T13:04:05.432 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2023-10-19T13:04:05.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.431+0000 7fd607421640 1 -- 172.21.15.57:0/1440032947 >> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] conn(0x7fd5e406b080 msgr2=0x7fd5e406d540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T13:04:05.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.431+0000 7fd607421640 1 --2- 172.21.15.57:0/1440032947 >> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] conn(0x7fd5e406b080 0x7fd5e406d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:05.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.431+0000 7fd607421640 1 -- 172.21.15.57:0/1440032947 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd6000b1b30 msgr2=0x7fd60015c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:05.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.431+0000 7fd607421640 1 --2- 172.21.15.57:0/1440032947 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd6000b1b30 0x7fd60015c7a0 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7fd5f8001330 tx=0x7fd5f8006430 comp rx=0 tx=0).stop 2023-10-19T13:04:05.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.431+0000 7fd607421640 1 -- 172.21.15.57:0/1440032947 shutdown_connections 2023-10-19T13:04:05.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.431+0000 7fd607421640 1 --2- 172.21.15.57:0/1440032947 >> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] conn(0x7fd5e406b080 0x7fd5e406d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:05.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.431+0000 7fd607421640 1 --2- 172.21.15.57:0/1440032947 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd6000b1b30 0x7fd60015c7a0 unknown :-1 s=CLOSED pgs=1501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:05.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.431+0000 7fd607421640 1 --2- 172.21.15.57:0/1440032947 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd6000b16b0 0x7fd60015c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:05.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.431+0000 7fd607421640 1 --2- 172.21.15.57:0/1440032947 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd5ec003b00 0x7fd60014cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:05.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.431+0000 7fd607421640 1 -- 172.21.15.57:0/1440032947 >> 172.21.15.57:0/1440032947 conn(0x7fd6000d0d60 msgr2=0x7fd6000ad8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:05.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.431+0000 7fd607421640 1 -- 172.21.15.57:0/1440032947 shutdown_connections 2023-10-19T13:04:05.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.431+0000 7fd607421640 1 -- 172.21.15.57:0/1440032947 wait complete. 2023-10-19T13:04:05.439 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:05.436+0000 7f4bbda14040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-19T13:04:05.445 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-19T13:04:05.491 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:05.489+0000 7f4bbda14040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-19T13:04:05.514 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:04:05.512+0000 7fdc13e3e040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-19T13:04:05.533 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:05.530+0000 7f4bbda14040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-19T13:04:05.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.555+0000 7f8cfb032640 1 -- 172.21.15.57:0/1973241150 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8cf415b260 msgr2=0x7f8cf40cf310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:05.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.555+0000 7f8cfb032640 1 --2- 172.21.15.57:0/1973241150 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8cf415b260 0x7f8cf40cf310 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7f8cf40895b0 tx=0x7f8ce802bda0 comp rx=0 tx=0).stop 2023-10-19T13:04:05.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.555+0000 7f8cfb032640 1 -- 172.21.15.57:0/1973241150 shutdown_connections 2023-10-19T13:04:05.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.555+0000 7f8cfb032640 1 --2- 172.21.15.57:0/1973241150 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8cf40b0b10 0x7f8cf40b0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:05.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.555+0000 7f8cfb032640 1 --2- 172.21.15.57:0/1973241150 >> v2:172.21.15.179:3300/0 conn(0x7f8cf415b640 0x7f8cf40cf850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:05.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.555+0000 7f8cfb032640 1 --2- 172.21.15.57:0/1973241150 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8cf415b260 0x7f8cf40cf310 unknown :-1 s=CLOSED pgs=1502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:05.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.555+0000 7f8cfb032640 1 -- 172.21.15.57:0/1973241150 >> 172.21.15.57:0/1973241150 conn(0x7f8cf409a030 msgr2=0x7f8cf40d0d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:05.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.555+0000 7f8cfb032640 1 -- 172.21.15.57:0/1973241150 shutdown_connections 2023-10-19T13:04:05.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.555+0000 7f8cfb032640 1 -- 172.21.15.57:0/1973241150 wait complete. 2023-10-19T13:04:05.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.556+0000 7f8cfb032640 1 Processor -- start 2023-10-19T13:04:05.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.557+0000 7f8cfb032640 1 -- start start 2023-10-19T13:04:05.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.557+0000 7f8cfb032640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8cf40b0b10 0x7f8cf40b5ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:05.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.557+0000 7f8cfb032640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8cf415b260 0x7f8cf40b3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:05.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.557+0000 7f8cfb032640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8cf415b640 0x7f8cf409b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:05.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.557+0000 7f8cfb032640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cf415c690 con 0x7f8cf40b0b10 2023-10-19T13:04:05.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.557+0000 7f8cfb032640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cf415bac0 con 0x7f8cf415b640 2023-10-19T13:04:05.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.557+0000 7f8cfb032640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cf415bd60 con 0x7f8cf415b260 2023-10-19T13:04:05.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.558+0000 7f8cf8dce640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8cf40b0b10 0x7f8cf40b5ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:05.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.558+0000 7f8cf8dce640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8cf40b0b10 0x7f8cf40b5ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34712/0 (socket says 172.21.15.57:34712) 2023-10-19T13:04:05.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.558+0000 7f8cf8dce640 1 -- 172.21.15.57:0/2916700841 learned_addr learned my addr 172.21.15.57:0/2916700841 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:05.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.558+0000 7f8cf3fff640 1 --2- 172.21.15.57:0/2916700841 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8cf415b260 0x7f8cf40b3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:05.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.558+0000 7f8cf95cf640 1 --2- 172.21.15.57:0/2916700841 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8cf415b640 0x7f8cf409b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:05.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.558+0000 7f8cf8dce640 1 -- 172.21.15.57:0/2916700841 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8cf415b260 msgr2=0x7f8cf40b3bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:05.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.558+0000 7f8cf8dce640 1 --2- 172.21.15.57:0/2916700841 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8cf415b260 0x7f8cf40b3bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:05.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.558+0000 7f8cf8dce640 1 -- 172.21.15.57:0/2916700841 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8cf415b640 msgr2=0x7f8cf409b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:05.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.558+0000 7f8cf8dce640 1 --2- 172.21.15.57:0/2916700841 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8cf415b640 0x7f8cf409b890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:05.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.558+0000 7f8cf8dce640 1 -- 172.21.15.57:0/2916700841 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ce8009560 con 0x7f8cf40b0b10 2023-10-19T13:04:05.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.558+0000 7f8cf8dce640 1 --2- 172.21.15.57:0/2916700841 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8cf40b0b10 0x7f8cf40b5ce0 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7f8ce8009850 tx=0x7f8ce802da70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:05.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.558+0000 7f8cf17fa640 1 -- 172.21.15.57:0/2916700841 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ce8003800 con 0x7f8cf40b0b10 2023-10-19T13:04:05.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.558+0000 7f8cf17fa640 1 -- 172.21.15.57:0/2916700841 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f8ce8003a50 con 0x7f8cf40b0b10 2023-10-19T13:04:05.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.558+0000 7f8cfb032640 1 -- 172.21.15.57:0/2916700841 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cf415c040 con 0x7f8cf40b0b10 2023-10-19T13:04:05.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.558+0000 7f8cf17fa640 1 -- 172.21.15.57:0/2916700841 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ce8003e20 con 0x7f8cf40b0b10 2023-10-19T13:04:05.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.558+0000 7f8cfb032640 1 -- 172.21.15.57:0/2916700841 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cf40b6220 con 0x7f8cf40b0b10 2023-10-19T13:04:05.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.559+0000 7f8cfb032640 1 -- 172.21.15.57:0/2916700841 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cf40675f0 con 0x7f8cf40b0b10 2023-10-19T13:04:05.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.560+0000 7f8cf17fa640 1 -- 172.21.15.57:0/2916700841 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 65) v1 ==== 89445+0+0 (secure 0 0 0) 0x7f8ce803e450 con 0x7f8cf40b0b10 2023-10-19T13:04:05.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.560+0000 7f8cf17fa640 1 --2- 172.21.15.57:0/2916700841 >> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] conn(0x7f8cd806b080 0x7f8cd806d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:05.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.560+0000 7f8cf3fff640 1 -- 172.21.15.57:0/2916700841 >> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] conn(0x7f8cd806b080 msgr2=0x7f8cd806d540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/1210673354 2023-10-19T13:04:05.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.560+0000 7f8cf17fa640 1 -- 172.21.15.57:0/2916700841 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f8ce80ac680 con 0x7f8cf40b0b10 2023-10-19T13:04:05.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.560+0000 7f8cf3fff640 1 --2- 172.21.15.57:0/2916700841 >> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] conn(0x7f8cd806b080 0x7f8cd806d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T13:04:05.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.564+0000 7f8cf17fa640 1 -- 172.21.15.57:0/2916700841 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8ce8077230 con 0x7f8cf40b0b10 2023-10-19T13:04:05.573 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:04:05.570+0000 7fdc13e3e040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-19T13:04:05.581 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:05.579+0000 7f4bbda14040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-19T13:04:05.617 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:04:05.615+0000 7fdc13e3e040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-19T13:04:05.624 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:05.621+0000 7f4bbda14040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-19T13:04:05.668 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:04:05.666+0000 7fdc13e3e040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-19T13:04:05.714 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:04:05.711+0000 7fdc13e3e040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-19T13:04:05.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.740+0000 7f8cfb032640 1 -- 172.21.15.57:0/2916700841 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f8cf40cb030 con 0x7f8cf40b0b10 2023-10-19T13:04:05.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.740+0000 7f8cf17fa640 1 -- 172.21.15.57:0/2916700841 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v131) v1 ==== 100+0+11 (secure 0 0 0) 0x7f8ce807a850 con 0x7f8cf40b0b10 2023-10-19T13:04:05.744 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 64 2023-10-19T13:04:05.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.742+0000 7f8cfb032640 1 -- 172.21.15.57:0/2916700841 >> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] conn(0x7f8cd806b080 msgr2=0x7f8cd806d540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T13:04:05.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.742+0000 7f8cfb032640 1 --2- 172.21.15.57:0/2916700841 >> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] conn(0x7f8cd806b080 0x7f8cd806d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:05.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.742+0000 7f8cfb032640 1 -- 172.21.15.57:0/2916700841 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8cf40b0b10 msgr2=0x7f8cf40b5ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:05.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.742+0000 7f8cfb032640 1 --2- 172.21.15.57:0/2916700841 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8cf40b0b10 0x7f8cf40b5ce0 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7f8ce8009850 tx=0x7f8ce802da70 comp rx=0 tx=0).stop 2023-10-19T13:04:05.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.742+0000 7f8cfb032640 1 -- 172.21.15.57:0/2916700841 shutdown_connections 2023-10-19T13:04:05.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.742+0000 7f8cfb032640 1 --2- 172.21.15.57:0/2916700841 >> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] conn(0x7f8cd806b080 0x7f8cd806d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:05.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.742+0000 7f8cfb032640 1 --2- 172.21.15.57:0/2916700841 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8cf415b640 0x7f8cf409b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:05.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.742+0000 7f8cfb032640 1 --2- 172.21.15.57:0/2916700841 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8cf415b260 0x7f8cf40b3bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:05.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.742+0000 7f8cfb032640 1 --2- 172.21.15.57:0/2916700841 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8cf40b0b10 0x7f8cf40b5ce0 unknown :-1 s=CLOSED pgs=1503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:05.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.742+0000 7f8cfb032640 1 -- 172.21.15.57:0/2916700841 >> 172.21.15.57:0/2916700841 conn(0x7f8cf409a030 msgr2=0x7f8cf415a200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:05.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.742+0000 7f8cfb032640 1 -- 172.21.15.57:0/2916700841 shutdown_connections 2023-10-19T13:04:05.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.742+0000 7f8cfb032640 1 -- 172.21.15.57:0/2916700841 wait complete. 2023-10-19T13:04:05.757 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-19T13:04:05.868 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.865+0000 7f4cab5dc640 1 -- 172.21.15.57:0/2060539304 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4ca40ceb40 msgr2=0x7f4ca40b0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:05.868 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.865+0000 7f4cab5dc640 1 --2- 172.21.15.57:0/2060539304 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4ca40ceb40 0x7f4ca40b0650 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7f4ca408a120 tx=0x7f4c9c02c1d0 comp rx=0 tx=0).stop 2023-10-19T13:04:05.869 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.865+0000 7f4cab5dc640 1 -- 172.21.15.57:0/2060539304 shutdown_connections 2023-10-19T13:04:05.869 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.865+0000 7f4cab5dc640 1 --2- 172.21.15.57:0/2060539304 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4ca40ceb40 0x7f4ca40b0650 unknown :-1 s=CLOSED pgs=1504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:05.869 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.865+0000 7f4cab5dc640 1 --2- 172.21.15.57:0/2060539304 >> v2:172.21.15.179:3300/0 conn(0x7f4ca415b2a0 0x7f4ca40d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:05.869 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.865+0000 7f4cab5dc640 1 -- 172.21.15.57:0/2060539304 >> 172.21.15.57:0/2060539304 conn(0x7f4ca40d0d60 msgr2=0x7f4ca40d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:05.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.865+0000 7f4cab5dc640 1 -- 172.21.15.57:0/2060539304 shutdown_connections 2023-10-19T13:04:05.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.865+0000 7f4cab5dc640 1 -- 172.21.15.57:0/2060539304 wait complete. 2023-10-19T13:04:05.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.866+0000 7f4cab5dc640 1 Processor -- start 2023-10-19T13:04:05.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.866+0000 7f4cab5dc640 1 -- start start 2023-10-19T13:04:05.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.867+0000 7f4cab5dc640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4ca40ceb40 0x7f4ca41580d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:05.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.867+0000 7f4cab5dc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4ca40cf340 0x7f4ca415d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:05.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.867+0000 7f4cab5dc640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4ca415b2a0 0x7f4ca40c9e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:05.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.867+0000 7f4cab5dc640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca415cef0 con 0x7f4ca40cf340 2023-10-19T13:04:05.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.867+0000 7f4cab5dc640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca415c040 con 0x7f4ca415b2a0 2023-10-19T13:04:05.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.867+0000 7f4cab5dc640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca415c2e0 con 0x7f4ca40ceb40 2023-10-19T13:04:05.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.867+0000 7f4ca8b77640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4ca40cf340 0x7f4ca415d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:05.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.867+0000 7f4ca8b77640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4ca40cf340 0x7f4ca415d380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34732/0 (socket says 172.21.15.57:34732) 2023-10-19T13:04:05.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.867+0000 7f4ca8b77640 1 -- 172.21.15.57:0/3017670366 learned_addr learned my addr 172.21.15.57:0/3017670366 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:05.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.867+0000 7f4ca9b79640 1 --2- 172.21.15.57:0/3017670366 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4ca415b2a0 0x7f4ca40c9e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:05.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.867+0000 7f4ca9378640 1 --2- 172.21.15.57:0/3017670366 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4ca40ceb40 0x7f4ca41580d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:05.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.867+0000 7f4ca8b77640 1 -- 172.21.15.57:0/3017670366 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4ca40ceb40 msgr2=0x7f4ca41580d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:05.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.867+0000 7f4ca8b77640 1 --2- 172.21.15.57:0/3017670366 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4ca40ceb40 0x7f4ca41580d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:05.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.867+0000 7f4ca8b77640 1 -- 172.21.15.57:0/3017670366 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4ca415b2a0 msgr2=0x7f4ca40c9e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:05.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.867+0000 7f4ca8b77640 1 --2- 172.21.15.57:0/3017670366 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4ca415b2a0 0x7f4ca40c9e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:05.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.867+0000 7f4ca8b77640 1 -- 172.21.15.57:0/3017670366 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c9c009900 con 0x7f4ca40cf340 2023-10-19T13:04:05.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.868+0000 7f4ca8b77640 1 --2- 172.21.15.57:0/3017670366 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4ca40cf340 0x7f4ca415d380 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f4ca000d6d0 tx=0x7f4ca000db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:05.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.868+0000 7f4c99ffb640 1 -- 172.21.15.57:0/3017670366 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ca0005220 con 0x7f4ca40cf340 2023-10-19T13:04:05.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.868+0000 7f4c99ffb640 1 -- 172.21.15.57:0/3017670366 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f4ca0005490 con 0x7f4ca40cf340 2023-10-19T13:04:05.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.868+0000 7f4cab5dc640 1 -- 172.21.15.57:0/3017670366 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ca415c5c0 con 0x7f4ca40cf340 2023-10-19T13:04:05.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.868+0000 7f4c99ffb640 1 -- 172.21.15.57:0/3017670366 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ca0009a80 con 0x7f4ca40cf340 2023-10-19T13:04:05.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.868+0000 7f4cab5dc640 1 -- 172.21.15.57:0/3017670366 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ca415cb80 con 0x7f4ca40cf340 2023-10-19T13:04:05.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.869+0000 7f4cab5dc640 1 -- 172.21.15.57:0/3017670366 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ca40690e0 con 0x7f4ca40cf340 2023-10-19T13:04:05.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.869+0000 7f4c99ffb640 1 -- 172.21.15.57:0/3017670366 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 65) v1 ==== 89445+0+0 (secure 0 0 0) 0x7f4ca00047a0 con 0x7f4ca40cf340 2023-10-19T13:04:05.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.870+0000 7f4c99ffb640 1 --2- 172.21.15.57:0/3017670366 >> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] conn(0x7f4c8806b080 0x7f4c8806d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:05.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.870+0000 7f4ca9378640 1 -- 172.21.15.57:0/3017670366 >> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] conn(0x7f4c8806b080 msgr2=0x7f4c8806d540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/1210673354 2023-10-19T13:04:05.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.870+0000 7f4c99ffb640 1 -- 172.21.15.57:0/3017670366 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f4ca008a190 con 0x7f4ca40cf340 2023-10-19T13:04:05.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.870+0000 7f4ca9378640 1 --2- 172.21.15.57:0/3017670366 >> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] conn(0x7f4c8806b080 0x7f4c8806d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T13:04:05.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:05.878+0000 7f4c99ffb640 1 -- 172.21.15.57:0/3017670366 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4ca0054b40 con 0x7f4ca40cf340 2023-10-19T13:04:05.938 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:05.935+0000 7f4bbda14040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-19T13:04:06.057 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:04:06.055+0000 7fdc13e3e040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-19T13:04:06.066 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:06.064+0000 7f4bbda14040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-19T13:04:06.072 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.069+0000 7f4cab5dc640 1 -- 172.21.15.57:0/3017670366 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f4ca40d0b40 con 0x7f4ca40cf340 2023-10-19T13:04:06.072 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.070+0000 7f4c99ffb640 1 -- 172.21.15.57:0/3017670366 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v131) v1 ==== 80+0+475 (secure 0 0 0) 0x7f4ca0058160 con 0x7f4ca40cf340 2023-10-19T13:04:06.073 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:6833/4155586759 2023-10-20T13:03:08.761549+0000 2023-10-19T13:04:06.073 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:6832/4155586759 2023-10-20T13:03:08.761549+0000 2023-10-19T13:04:06.073 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/1341670560 2023-10-20T13:03:08.761549+0000 2023-10-19T13:04:06.074 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/33839263 2023-10-20T13:03:08.761549+0000 2023-10-19T13:04:06.074 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/3689030544 2023-10-20T13:03:08.761549+0000 2023-10-19T13:04:06.074 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/3037001824 2023-10-20T13:03:08.761549+0000 2023-10-19T13:04:06.074 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/1069074991 2023-10-20T13:03:08.761549+0000 2023-10-19T13:04:06.074 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/831719196 2023-10-20T13:03:08.761549+0000 2023-10-19T13:04:06.076 INFO:teuthology.orchestra.run.smithi057.stderr:listed 8 entries 2023-10-19T13:04:06.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.071+0000 7f4ca9378640 1 -- 172.21.15.57:0/3017670366 >> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] conn(0x7f4c8806b080 msgr2=0x7f4c8806d540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/1210673354 2023-10-19T13:04:06.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.071+0000 7f4ca9378640 1 --2- 172.21.15.57:0/3017670366 >> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] conn(0x7f4c8806b080 0x7f4c8806d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-19T13:04:06.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.073+0000 7f4cab5dc640 1 -- 172.21.15.57:0/3017670366 >> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] conn(0x7f4c8806b080 msgr2=0x7f4c8806d540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T13:04:06.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.073+0000 7f4cab5dc640 1 --2- 172.21.15.57:0/3017670366 >> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] conn(0x7f4c8806b080 0x7f4c8806d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:06.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.073+0000 7f4cab5dc640 1 -- 172.21.15.57:0/3017670366 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4ca40cf340 msgr2=0x7f4ca415d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:06.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.073+0000 7f4cab5dc640 1 --2- 172.21.15.57:0/3017670366 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4ca40cf340 0x7f4ca415d380 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f4ca000d6d0 tx=0x7f4ca000db90 comp rx=0 tx=0).stop 2023-10-19T13:04:06.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.073+0000 7f4cab5dc640 1 -- 172.21.15.57:0/3017670366 shutdown_connections 2023-10-19T13:04:06.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.073+0000 7f4cab5dc640 1 --2- 172.21.15.57:0/3017670366 >> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] conn(0x7f4c8806b080 0x7f4c8806d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:06.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.073+0000 7f4cab5dc640 1 --2- 172.21.15.57:0/3017670366 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4ca415b2a0 0x7f4ca40c9e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:06.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.073+0000 7f4cab5dc640 1 --2- 172.21.15.57:0/3017670366 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4ca40cf340 0x7f4ca415d380 unknown :-1 s=CLOSED pgs=1505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:06.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.073+0000 7f4cab5dc640 1 --2- 172.21.15.57:0/3017670366 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4ca40ceb40 0x7f4ca41580d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:06.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.073+0000 7f4cab5dc640 1 -- 172.21.15.57:0/3017670366 >> 172.21.15.57:0/3017670366 conn(0x7f4ca40d0d60 msgr2=0x7f4ca40cc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:06.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.073+0000 7f4cab5dc640 1 -- 172.21.15.57:0/3017670366 shutdown_connections 2023-10-19T13:04:06.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.073+0000 7f4cab5dc640 1 -- 172.21.15.57:0/3017670366 wait complete. 2023-10-19T13:04:06.086 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi057.front.sepia.ceph.com... 2023-10-19T13:04:06.087 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:04:06.087 DEBUG:teuthology.orchestra.run.smithi057:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-10-19T13:04:06.116 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi057.stderr:ceph-fuse[128173]: fuse finished with error 0 and tester_r 0 2023-10-19T13:04:06.118 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi057.front.sepia.ceph.com 2023-10-19T13:04:06.119 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:04:06.119 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-19T13:04:06.157 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:06.155+0000 7f4bbda14040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-19T13:04:06.200 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:04:06.197+0000 7fdc13e3e040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-19T13:04:06.200 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:06.198+0000 7f4bbda14040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-19T13:04:06.244 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-19T13:04:06.244 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T13:04:06.260 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054833,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=284 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3247632k,nr_inodes=811908,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-19T13:04:06.260 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi179.front.sepia.ceph.com 2023-10-19T13:04:06.260 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:04:06.261 DEBUG:teuthology.orchestra.run.smithi179:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-10-19T13:04:06.297 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:04:06.295+0000 7fdc13e3e040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-19T13:04:06.298 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:06.295+0000 7f4bbda14040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-19T13:04:06.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-19T13:04:06.313 INFO:teuthology.orchestra.run.smithi179.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2023-10-19T13:04:06.313 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T13:04:06.313 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T13:04:06.329 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055614,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=358 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248260k,nr_inodes=812065,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-19T13:04:06.329 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi057.front.sepia.ceph.com 2023-10-19T13:04:06.330 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:04:06.330 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-10-19T13:04:06.343 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:04:06.340+0000 7fdc13e3e040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-19T13:04:06.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-19T13:04:06.358 INFO:teuthology.orchestra.run.smithi057.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2023-10-19T13:04:06.358 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-19T13:04:06.359 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T13:04:06.375 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054833,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=284 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3247632k,nr_inodes=811908,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-19T13:04:06.376 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi179.front.sepia.ceph.com 2023-10-19T13:04:06.376 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:04:06.376 DEBUG:teuthology.orchestra.run.smithi179:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-10-19T13:04:06.396 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:06.393+0000 7f4bbda14040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-19T13:04:06.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-19T13:04:06.410 INFO:teuthology.orchestra.run.smithi179.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2023-10-19T13:04:06.411 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T13:04:06.444 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:04:06.442+0000 7fdc13e3e040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-19T13:04:06.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.518+0000 7f9df53b6640 1 -- 172.21.15.57:0/1929828271 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9df015b640 msgr2=0x7f9df00cbf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:06.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.518+0000 7f9df53b6640 1 --2- 172.21.15.57:0/1929828271 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9df015b640 0x7f9df00cbf20 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7f9df00895b0 tx=0x7f9de802c110 comp rx=0 tx=0).stop 2023-10-19T13:04:06.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.518+0000 7f9df53b6640 1 -- 172.21.15.57:0/1929828271 shutdown_connections 2023-10-19T13:04:06.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.518+0000 7f9df53b6640 1 --2- 172.21.15.57:0/1929828271 >> v2:172.21.15.179:3300/0 conn(0x7f9df00cfa20 0x7f9df00cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:06.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.518+0000 7f9df53b6640 1 --2- 172.21.15.57:0/1929828271 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9df015b640 0x7f9df00cbf20 unknown :-1 s=CLOSED pgs=1507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:06.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.518+0000 7f9df53b6640 1 -- 172.21.15.57:0/1929828271 >> 172.21.15.57:0/1929828271 conn(0x7f9df009a030 msgr2=0x7f9df00d0d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:06.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.518+0000 7f9df53b6640 1 -- 172.21.15.57:0/1929828271 shutdown_connections 2023-10-19T13:04:06.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.518+0000 7f9df53b6640 1 -- 172.21.15.57:0/1929828271 wait complete. 2023-10-19T13:04:06.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.520+0000 7f9df53b6640 1 Processor -- start 2023-10-19T13:04:06.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.520+0000 7f9df53b6640 1 -- start start 2023-10-19T13:04:06.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.522+0000 7f9df53b6640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9df00cfa20 0x7f9df0158450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:06.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.522+0000 7f9df53b6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9df015b260 0x7f9df00b5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:06.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.522+0000 7f9df53b6640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9df015b640 0x7f9df00b3940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:06.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.522+0000 7f9df53b6640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df00b4fa0 con 0x7f9df015b260 2023-10-19T13:04:06.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.522+0000 7f9df53b6640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df00b4390 con 0x7f9df015b640 2023-10-19T13:04:06.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.522+0000 7f9df53b6640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9df00b4670 con 0x7f9df00cfa20 2023-10-19T13:04:06.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.522+0000 7f9de7fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9df015b260 0x7f9df00b5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:06.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.522+0000 7f9de7fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9df015b260 0x7f9df00b5430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34750/0 (socket says 172.21.15.57:34750) 2023-10-19T13:04:06.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.522+0000 7f9de7fff640 1 -- 172.21.15.57:0/2530552546 learned_addr learned my addr 172.21.15.57:0/2530552546 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:06.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.522+0000 7f9deeffd640 1 --2- 172.21.15.57:0/2530552546 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9df00cfa20 0x7f9df0158450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:06.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.522+0000 7f9def7fe640 1 --2- 172.21.15.57:0/2530552546 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9df015b640 0x7f9df00b3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:06.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.522+0000 7f9de7fff640 1 -- 172.21.15.57:0/2530552546 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9df00cfa20 msgr2=0x7f9df0158450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:06.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.522+0000 7f9de7fff640 1 --2- 172.21.15.57:0/2530552546 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9df00cfa20 0x7f9df0158450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:06.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.522+0000 7f9de7fff640 1 -- 172.21.15.57:0/2530552546 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9df015b640 msgr2=0x7f9df00b3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:06.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.522+0000 7f9de7fff640 1 --2- 172.21.15.57:0/2530552546 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9df015b640 0x7f9df00b3940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:06.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.522+0000 7f9de7fff640 1 -- 172.21.15.57:0/2530552546 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9de8009840 con 0x7f9df015b260 2023-10-19T13:04:06.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.522+0000 7f9de7fff640 1 --2- 172.21.15.57:0/2530552546 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9df015b260 0x7f9df00b5430 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7f9dd800e9f0 tx=0x7f9dd800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:06.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.522+0000 7f9de77fe640 1 -- 172.21.15.57:0/2530552546 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dd8009700 con 0x7f9df015b260 2023-10-19T13:04:06.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.522+0000 7f9de77fe640 1 -- 172.21.15.57:0/2530552546 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f9dd800ce70 con 0x7f9df015b260 2023-10-19T13:04:06.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.522+0000 7f9df53b6640 1 -- 172.21.15.57:0/2530552546 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9df00b4950 con 0x7f9df015b260 2023-10-19T13:04:06.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.522+0000 7f9de77fe640 1 -- 172.21.15.57:0/2530552546 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dd8010670 con 0x7f9df015b260 2023-10-19T13:04:06.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.522+0000 7f9df53b6640 1 -- 172.21.15.57:0/2530552546 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9df00b5970 con 0x7f9df015b260 2023-10-19T13:04:06.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.523+0000 7f9df53b6640 1 -- 172.21.15.57:0/2530552546 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9df00675f0 con 0x7f9df015b260 2023-10-19T13:04:06.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.524+0000 7f9de77fe640 1 -- 172.21.15.57:0/2530552546 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 65) v1 ==== 89445+0+0 (secure 0 0 0) 0x7f9dd80107d0 con 0x7f9df015b260 2023-10-19T13:04:06.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.525+0000 7f9de77fe640 1 --2- 172.21.15.57:0/2530552546 >> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] conn(0x7f9dd006b080 0x7f9dd006d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:06.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.525+0000 7f9deeffd640 1 -- 172.21.15.57:0/2530552546 >> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] conn(0x7f9dd006b080 msgr2=0x7f9dd006d540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/1210673354 2023-10-19T13:04:06.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.525+0000 7f9de77fe640 1 -- 172.21.15.57:0/2530552546 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f9dd8014070 con 0x7f9df015b260 2023-10-19T13:04:06.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.525+0000 7f9deeffd640 1 --2- 172.21.15.57:0/2530552546 >> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] conn(0x7f9dd006b080 0x7f9dd006d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T13:04:06.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.529+0000 7f9de77fe640 1 -- 172.21.15.57:0/2530552546 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9dd8051ca0 con 0x7f9df015b260 2023-10-19T13:04:06.535 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:06.533+0000 7f4bbda14040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-19T13:04:06.545 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:04:06.542+0000 7fdc13e3e040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-19T13:04:06.577 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:06.574+0000 7f4bbda14040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-19T13:04:06.681 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:06.678+0000 7f4bbda14040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-19T13:04:06.690 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:04:06.687+0000 7fdc13e3e040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-19T13:04:06.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.726+0000 7f9deeffd640 1 -- 172.21.15.57:0/2530552546 >> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] conn(0x7f9dd006b080 msgr2=0x7f9dd006d540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/1210673354 2023-10-19T13:04:06.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.726+0000 7f9deeffd640 1 --2- 172.21.15.57:0/2530552546 >> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] conn(0x7f9dd006b080 0x7f9dd006d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-19T13:04:06.734 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:04:06.731+0000 7fdc13e3e040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-19T13:04:06.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.741+0000 7f9df53b6640 1 -- 172.21.15.57:0/2530552546 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9df006abf0 con 0x7f9df015b260 2023-10-19T13:04:06.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.742+0000 7f9de77fe640 1 -- 172.21.15.57:0/2530552546 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4912 (secure 0 0 0) 0x7f9dd8055490 con 0x7f9df015b260 2023-10-19T13:04:06.745 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:04:06.745 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":50,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/2033977424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2033977424},{"type":"v1","addr":"172.21.15.179:6836","nonce":2033977424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":8568,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/981095684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":981095684},{"type":"v1","addr":"172.21.15.57:6834","nonce":981095684}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":8571,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/383147069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":383147069},{"type":"v1","addr":"172.21.15.57:6835","nonce":383147069}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T13:02:57.697820+0000","modified":"2023-10-19T13:02:59.684888+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8577},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8577":{"gid":8577,"name":"d","rank":0,"incarnation":48,"state":"up:active","state_seq":3,"addr":"172.21.15.179:6837/768231146","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":768231146},{"type":"v1","addr":"172.21.15.179:6837","nonce":768231146}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2023-10-19T13:04:06.745 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 50 2023-10-19T13:04:06.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.744+0000 7f9df53b6640 1 -- 172.21.15.57:0/2530552546 >> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] conn(0x7f9dd006b080 msgr2=0x7f9dd006d540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T13:04:06.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.744+0000 7f9df53b6640 1 --2- 172.21.15.57:0/2530552546 >> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] conn(0x7f9dd006b080 0x7f9dd006d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:06.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.744+0000 7f9df53b6640 1 -- 172.21.15.57:0/2530552546 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9df015b260 msgr2=0x7f9df00b5430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:06.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.744+0000 7f9df53b6640 1 --2- 172.21.15.57:0/2530552546 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9df015b260 0x7f9df00b5430 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7f9dd800e9f0 tx=0x7f9dd800eeb0 comp rx=0 tx=0).stop 2023-10-19T13:04:06.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.744+0000 7f9df53b6640 1 -- 172.21.15.57:0/2530552546 shutdown_connections 2023-10-19T13:04:06.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.744+0000 7f9df53b6640 1 --2- 172.21.15.57:0/2530552546 >> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] conn(0x7f9dd006b080 0x7f9dd006d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:06.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.744+0000 7f9df53b6640 1 --2- 172.21.15.57:0/2530552546 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9df015b640 0x7f9df00b3940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:06.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.744+0000 7f9df53b6640 1 --2- 172.21.15.57:0/2530552546 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9df015b260 0x7f9df00b5430 unknown :-1 s=CLOSED pgs=1508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:06.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.744+0000 7f9df53b6640 1 --2- 172.21.15.57:0/2530552546 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9df00cfa20 0x7f9df0158450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:06.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.744+0000 7f9df53b6640 1 -- 172.21.15.57:0/2530552546 >> 172.21.15.57:0/2530552546 conn(0x7f9df009a030 msgr2=0x7f9df00cee60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:06.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.744+0000 7f9df53b6640 1 -- 172.21.15.57:0/2530552546 shutdown_connections 2023-10-19T13:04:06.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.744+0000 7f9df53b6640 1 -- 172.21.15.57:0/2530552546 wait complete. 2023-10-19T13:04:06.758 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T13:04:06.847 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:04:06.844+0000 7fdc13e3e040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-19T13:04:06.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.904+0000 7fbaeb305640 1 -- 172.21.15.57:0/1381138964 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbae40cf340 msgr2=0x7fbae40d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:06.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.904+0000 7fbaeb305640 1 --2- 172.21.15.57:0/1381138964 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbae40cf340 0x7fbae40d1910 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7fbae0009830 tx=0x7fbae002bd60 comp rx=0 tx=0).stop 2023-10-19T13:04:06.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.904+0000 7fbaeb305640 1 -- 172.21.15.57:0/1381138964 shutdown_connections 2023-10-19T13:04:06.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.904+0000 7fbaeb305640 1 --2- 172.21.15.57:0/1381138964 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbae40cf340 0x7fbae40d1910 unknown :-1 s=CLOSED pgs=1510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:06.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.904+0000 7fbaeb305640 1 -- 172.21.15.57:0/1381138964 >> 172.21.15.57:0/1381138964 conn(0x7fbae40d0d60 msgr2=0x7fbae40d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:06.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.904+0000 7fbaeb305640 1 -- 172.21.15.57:0/1381138964 shutdown_connections 2023-10-19T13:04:06.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.904+0000 7fbaeb305640 1 -- 172.21.15.57:0/1381138964 wait complete. 2023-10-19T13:04:06.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.905+0000 7fbaeb305640 1 Processor -- start 2023-10-19T13:04:06.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.905+0000 7fbaeb305640 1 -- start start 2023-10-19T13:04:06.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.906+0000 7fbaeb305640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbae40ceb40 0x7fbae40b5940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:06.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.906+0000 7fbaeb305640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbae40cf340 0x7fbae414dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:06.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.906+0000 7fbaeb305640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbae415b2a0 0x7fbae40b38e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:06.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.906+0000 7fbaeb305640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbae414e6a0 con 0x7fbae40cf340 2023-10-19T13:04:06.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.906+0000 7fbaeb305640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbae414e980 con 0x7fbae415b2a0 2023-10-19T13:04:06.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.906+0000 7fbaeb305640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbae414ec60 con 0x7fbae40ceb40 2023-10-19T13:04:06.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.906+0000 7fbae88a0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbae40cf340 0x7fbae414dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:06.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.906+0000 7fbae88a0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbae40cf340 0x7fbae414dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33624/0 (socket says 172.21.15.57:33624) 2023-10-19T13:04:06.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.906+0000 7fbae88a0640 1 -- 172.21.15.57:0/4024049608 learned_addr learned my addr 172.21.15.57:0/4024049608 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:06.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.906+0000 7fbae98a2640 1 --2- 172.21.15.57:0/4024049608 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbae415b2a0 0x7fbae40b38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:06.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.906+0000 7fbae90a1640 1 --2- 172.21.15.57:0/4024049608 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbae40ceb40 0x7fbae40b5940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:06.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.907+0000 7fbae88a0640 1 -- 172.21.15.57:0/4024049608 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbae40ceb40 msgr2=0x7fbae40b5940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:06.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.907+0000 7fbae88a0640 1 --2- 172.21.15.57:0/4024049608 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbae40ceb40 0x7fbae40b5940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:06.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.907+0000 7fbae88a0640 1 -- 172.21.15.57:0/4024049608 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbae415b2a0 msgr2=0x7fbae40b38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:06.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.907+0000 7fbae88a0640 1 --2- 172.21.15.57:0/4024049608 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbae415b2a0 0x7fbae40b38e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:06.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.907+0000 7fbae88a0640 1 -- 172.21.15.57:0/4024049608 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbae0009490 con 0x7fbae40cf340 2023-10-19T13:04:06.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.907+0000 7fbae88a0640 1 --2- 172.21.15.57:0/4024049608 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbae40cf340 0x7fbae414dda0 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7fbad400c430 tx=0x7fbad400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:06.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.907+0000 7fbad9ffb640 1 -- 172.21.15.57:0/4024049608 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbad4011790 con 0x7fbae40cf340 2023-10-19T13:04:06.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.907+0000 7fbad9ffb640 1 -- 172.21.15.57:0/4024049608 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fbad4011a00 con 0x7fbae40cf340 2023-10-19T13:04:06.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.907+0000 7fbaeb305640 1 -- 172.21.15.57:0/4024049608 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbae414ef40 con 0x7fbae40cf340 2023-10-19T13:04:06.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.907+0000 7fbad9ffb640 1 -- 172.21.15.57:0/4024049608 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbad4011e10 con 0x7fbae40cf340 2023-10-19T13:04:06.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.907+0000 7fbaeb305640 1 -- 172.21.15.57:0/4024049608 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbae40b5e80 con 0x7fbae40cf340 2023-10-19T13:04:06.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.908+0000 7fbad9ffb640 1 -- 172.21.15.57:0/4024049608 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 65) v1 ==== 89445+0+0 (secure 0 0 0) 0x7fbad401c460 con 0x7fbae40cf340 2023-10-19T13:04:06.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.909+0000 7fbaeb305640 1 -- 172.21.15.57:0/4024049608 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbae414f460 con 0x7fbae40cf340 2023-10-19T13:04:06.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.909+0000 7fbad9ffb640 1 --2- 172.21.15.57:0/4024049608 >> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] conn(0x7fbac806b0b0 0x7fbac806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:06.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.909+0000 7fbad9ffb640 1 -- 172.21.15.57:0/4024049608 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6401+0+0 (secure 0 0 0) 0x7fbad4089c00 con 0x7fbae40cf340 2023-10-19T13:04:06.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.909+0000 7fbae90a1640 1 -- 172.21.15.57:0/4024049608 >> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] conn(0x7fbac806b0b0 msgr2=0x7fbac806d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/1210673354 2023-10-19T13:04:06.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.909+0000 7fbae90a1640 1 --2- 172.21.15.57:0/4024049608 >> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] conn(0x7fbac806b0b0 0x7fbac806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T13:04:06.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:06.913+0000 7fbad9ffb640 1 -- 172.21.15.57:0/4024049608 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fbae414f460 con 0x7fbae40cf340 2023-10-19T13:04:07.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.110+0000 7fbae90a1640 1 -- 172.21.15.57:0/4024049608 >> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] conn(0x7fbac806b0b0 msgr2=0x7fbac806d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/1210673354 2023-10-19T13:04:07.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.110+0000 7fbae90a1640 1 --2- 172.21.15.57:0/4024049608 >> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] conn(0x7fbac806b0b0 0x7fbac806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-19T13:04:07.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.161+0000 7fbaeb305640 1 -- 172.21.15.57:0/4024049608 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbae414f460 con 0x7fbae40cf340 2023-10-19T13:04:07.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.163+0000 7fbad9ffb640 1 -- 172.21.15.57:0/4024049608 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v131) v1 ==== 74+0+17277 (secure 0 0 0) 0x7fbae414f460 con 0x7fbae40cf340 2023-10-19T13:04:07.166 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:04:07.166 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":131,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T13:03:08.761584+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2023-10-19T13:02:55.916158+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"129","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2023-10-19T13:02:56.675559+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"130","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.179:6833/4155586759":"2023-10-20T13:03:08.761549+0000","172.21.15.179:6832/4155586759":"2023-10-20T13:03:08.761549+0000","172.21.15.179:0/1341670560":"2023-10-20T13:03:08.761549+0000","172.21.15.179:0/33839263":"2023-10-20T13:03:08.761549+0000","172.21.15.179:0/3689030544":"2023-10-20T13:03:08.761549+0000","172.21.15.179:0/3037001824":"2023-10-20T13:03:08.761549+0000","172.21.15.179:0/1069074991":"2023-10-20T13:03:08.761549+0000","172.21.15.179:0/831719196":"2023-10-20T13:03:08.761549+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T13:04:07.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.165+0000 7fbaeb305640 1 -- 172.21.15.57:0/4024049608 >> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] conn(0x7fbac806b0b0 msgr2=0x7fbac806d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T13:04:07.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.165+0000 7fbaeb305640 1 --2- 172.21.15.57:0/4024049608 >> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] conn(0x7fbac806b0b0 0x7fbac806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:07.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.165+0000 7fbaeb305640 1 -- 172.21.15.57:0/4024049608 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbae40cf340 msgr2=0x7fbae414dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:07.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.165+0000 7fbaeb305640 1 --2- 172.21.15.57:0/4024049608 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbae40cf340 0x7fbae414dda0 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7fbad400c430 tx=0x7fbad400c8f0 comp rx=0 tx=0).stop 2023-10-19T13:04:07.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.165+0000 7fbaeb305640 1 -- 172.21.15.57:0/4024049608 shutdown_connections 2023-10-19T13:04:07.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.165+0000 7fbaeb305640 1 --2- 172.21.15.57:0/4024049608 >> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] conn(0x7fbac806b0b0 0x7fbac806d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:07.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.165+0000 7fbaeb305640 1 --2- 172.21.15.57:0/4024049608 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbae415b2a0 0x7fbae40b38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:07.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.165+0000 7fbaeb305640 1 --2- 172.21.15.57:0/4024049608 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbae40cf340 0x7fbae414dda0 unknown :-1 s=CLOSED pgs=1511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:07.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.165+0000 7fbaeb305640 1 --2- 172.21.15.57:0/4024049608 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbae40ceb40 0x7fbae40b5940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:07.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.165+0000 7fbaeb305640 1 -- 172.21.15.57:0/4024049608 >> 172.21.15.57:0/4024049608 conn(0x7fbae40d0d60 msgr2=0x7fbae406b620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:07.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.165+0000 7fbaeb305640 1 -- 172.21.15.57:0/4024049608 shutdown_connections 2023-10-19T13:04:07.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.165+0000 7fbaeb305640 1 -- 172.21.15.57:0/4024049608 wait complete. 2023-10-19T13:04:07.179 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-19T12:44:07.371186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-19T13:02:55.916158+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '129', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_data', 'create_time': '2023-10-19T13:02:56.675559+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '130', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-19T13:04:07.179 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-19T13:04:07.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.290+0000 7fbcf3258640 1 -- 172.21.15.57:0/814493120 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbcec0cf340 msgr2=0x7fbcec0d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:07.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.290+0000 7fbcf3258640 1 --2- 172.21.15.57:0/814493120 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbcec0cf340 0x7fbcec0d1910 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fbce0009830 tx=0x7fbce002bcd0 comp rx=0 tx=0).stop 2023-10-19T13:04:07.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.290+0000 7fbcf3258640 1 -- 172.21.15.57:0/814493120 shutdown_connections 2023-10-19T13:04:07.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.290+0000 7fbcf3258640 1 --2- 172.21.15.57:0/814493120 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbcec15b2a0 0x7fbcec0cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:07.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.290+0000 7fbcf3258640 1 --2- 172.21.15.57:0/814493120 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbcec0cf340 0x7fbcec0d1910 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:07.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.290+0000 7fbcf3258640 1 -- 172.21.15.57:0/814493120 >> 172.21.15.57:0/814493120 conn(0x7fbcec0d0d60 msgr2=0x7fbcec0d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:07.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.290+0000 7fbcf3258640 1 -- 172.21.15.57:0/814493120 shutdown_connections 2023-10-19T13:04:07.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.290+0000 7fbcf3258640 1 -- 172.21.15.57:0/814493120 wait complete. 2023-10-19T13:04:07.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.291+0000 7fbcf3258640 1 Processor -- start 2023-10-19T13:04:07.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.291+0000 7fbcf3258640 1 -- start start 2023-10-19T13:04:07.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.292+0000 7fbcf3258640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbcec0ceb40 0x7fbcec0ca230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:07.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.292+0000 7fbcf3258640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbcec0cf340 0x7fbcec1e3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:07.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.292+0000 7fbcf3258640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbcec15b2a0 0x7fbcec1e40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:07.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.292+0000 7fbcf3258640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcec15ce40 con 0x7fbcec0cf340 2023-10-19T13:04:07.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.292+0000 7fbcf3258640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcec15c7c0 con 0x7fbcec15b2a0 2023-10-19T13:04:07.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.292+0000 7fbcf3258640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcec0cacb0 con 0x7fbcec0ceb40 2023-10-19T13:04:07.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.292+0000 7fbcebfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbcec0cf340 0x7fbcec1e3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:07.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.292+0000 7fbcebfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbcec0cf340 0x7fbcec1e3ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33630/0 (socket says 172.21.15.57:33630) 2023-10-19T13:04:07.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.292+0000 7fbcebfff640 1 -- 172.21.15.57:0/1304880951 learned_addr learned my addr 172.21.15.57:0/1304880951 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:07.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.292+0000 7fbcf17f5640 1 --2- 172.21.15.57:0/1304880951 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbcec15b2a0 0x7fbcec1e40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:07.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.292+0000 7fbcf0ff4640 1 --2- 172.21.15.57:0/1304880951 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbcec0ceb40 0x7fbcec0ca230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:07.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.293+0000 7fbcebfff640 1 -- 172.21.15.57:0/1304880951 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbcec0ceb40 msgr2=0x7fbcec0ca230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:07.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.293+0000 7fbcebfff640 1 --2- 172.21.15.57:0/1304880951 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbcec0ceb40 0x7fbcec0ca230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:07.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.293+0000 7fbcebfff640 1 -- 172.21.15.57:0/1304880951 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbcec15b2a0 msgr2=0x7fbcec1e40e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:07.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.293+0000 7fbcebfff640 1 --2- 172.21.15.57:0/1304880951 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbcec15b2a0 0x7fbcec1e40e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:07.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.293+0000 7fbcebfff640 1 -- 172.21.15.57:0/1304880951 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbce0009490 con 0x7fbcec0cf340 2023-10-19T13:04:07.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.293+0000 7fbcf17f5640 1 --2- 172.21.15.57:0/1304880951 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbcec15b2a0 0x7fbcec1e40e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T13:04:07.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.293+0000 7fbcf0ff4640 1 --2- 172.21.15.57:0/1304880951 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbcec0ceb40 0x7fbcec0ca230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T13:04:07.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.293+0000 7fbcebfff640 1 --2- 172.21.15.57:0/1304880951 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbcec0cf340 0x7fbcec1e3ba0 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7fbcd800e790 tx=0x7fbcd800ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:07.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.293+0000 7fbce97fa640 1 -- 172.21.15.57:0/1304880951 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcd8009580 con 0x7fbcec0cf340 2023-10-19T13:04:07.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.293+0000 7fbce97fa640 1 -- 172.21.15.57:0/1304880951 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fbcd800cdb0 con 0x7fbcec0cf340 2023-10-19T13:04:07.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.293+0000 7fbcf3258640 1 -- 172.21.15.57:0/1304880951 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcec15ca60 con 0x7fbcec0cf340 2023-10-19T13:04:07.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.293+0000 7fbce97fa640 1 -- 172.21.15.57:0/1304880951 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcd8010690 con 0x7fbcec0cf340 2023-10-19T13:04:07.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.293+0000 7fbcf3258640 1 -- 172.21.15.57:0/1304880951 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcec0c8050 con 0x7fbcec0cf340 2023-10-19T13:04:07.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.294+0000 7fbcf3258640 1 -- 172.21.15.57:0/1304880951 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcec0690e0 con 0x7fbcec0cf340 2023-10-19T13:04:07.304 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:07.295+0000 7f4bbda14040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-19T13:04:07.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.295+0000 7fbce97fa640 1 -- 172.21.15.57:0/1304880951 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 65) v1 ==== 89445+0+0 (secure 0 0 0) 0x7fbcd80107f0 con 0x7fbcec0cf340 2023-10-19T13:04:07.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.295+0000 7fbce97fa640 1 --2- 172.21.15.57:0/1304880951 >> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] conn(0x7fbcd006b080 0x7fbcd006d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:07.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.295+0000 7fbcf0ff4640 1 -- 172.21.15.57:0/1304880951 >> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] conn(0x7fbcd006b080 msgr2=0x7fbcd006d540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/1210673354 2023-10-19T13:04:07.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.295+0000 7fbce97fa640 1 -- 172.21.15.57:0/1304880951 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6401+0+0 (secure 0 0 0) 0x7fbcd8014070 con 0x7fbcec0cf340 2023-10-19T13:04:07.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.295+0000 7fbcf0ff4640 1 --2- 172.21.15.57:0/1304880951 >> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] conn(0x7fbcd006b080 0x7fbcd006d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T13:04:07.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.300+0000 7fbce97fa640 1 -- 172.21.15.57:0/1304880951 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fbcd8051ca0 con 0x7fbcec0cf340 2023-10-19T13:04:07.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.487+0000 7fbcf3258640 1 -- 172.21.15.57:0/1304880951 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fbcec156950 con 0x7fbcec0cf340 2023-10-19T13:04:07.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.487+0000 7fbce97fa640 1 -- 172.21.15.57:0/1304880951 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v131) v1 ==== 93+0+10 (secure 0 0 0) 0x7fbcd8055250 con 0x7fbcec0cf340 2023-10-19T13:04:07.490 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-19T13:04:07.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.489+0000 7fbcf3258640 1 -- 172.21.15.57:0/1304880951 >> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] conn(0x7fbcd006b080 msgr2=0x7fbcd006d540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T13:04:07.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.489+0000 7fbcf3258640 1 --2- 172.21.15.57:0/1304880951 >> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] conn(0x7fbcd006b080 0x7fbcd006d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:07.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.489+0000 7fbcf3258640 1 -- 172.21.15.57:0/1304880951 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbcec0cf340 msgr2=0x7fbcec1e3ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:07.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.489+0000 7fbcf3258640 1 --2- 172.21.15.57:0/1304880951 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbcec0cf340 0x7fbcec1e3ba0 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7fbcd800e790 tx=0x7fbcd800ec50 comp rx=0 tx=0).stop 2023-10-19T13:04:07.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.489+0000 7fbcf3258640 1 -- 172.21.15.57:0/1304880951 shutdown_connections 2023-10-19T13:04:07.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.489+0000 7fbcf3258640 1 --2- 172.21.15.57:0/1304880951 >> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] conn(0x7fbcd006b080 0x7fbcd006d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:07.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.489+0000 7fbcf3258640 1 --2- 172.21.15.57:0/1304880951 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbcec15b2a0 0x7fbcec1e40e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:07.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.489+0000 7fbcf3258640 1 --2- 172.21.15.57:0/1304880951 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbcec0cf340 0x7fbcec1e3ba0 unknown :-1 s=CLOSED pgs=1512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:07.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.489+0000 7fbcf3258640 1 --2- 172.21.15.57:0/1304880951 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbcec0ceb40 0x7fbcec0ca230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:07.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.489+0000 7fbcf3258640 1 -- 172.21.15.57:0/1304880951 >> 172.21.15.57:0/1304880951 conn(0x7fbcec0d0d60 msgr2=0x7fbcec0af1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:07.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.489+0000 7fbcf3258640 1 -- 172.21.15.57:0/1304880951 shutdown_connections 2023-10-19T13:04:07.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.489+0000 7fbcf3258640 1 -- 172.21.15.57:0/1304880951 wait complete. 2023-10-19T13:04:07.503 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-19T13:04:07.561 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:07.559+0000 7f4bbda14040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-19T13:04:07.582 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:04:07.579+0000 7fdc13e3e040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-19T13:04:07.609 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:07.607+0000 7f4bbda14040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-19T13:04:07.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.613+0000 7fab9461a640 1 -- 172.21.15.57:0/3851168703 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab8c0cf340 msgr2=0x7fab8c0d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:07.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.613+0000 7fab9461a640 1 --2- 172.21.15.57:0/3851168703 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab8c0cf340 0x7fab8c0d1910 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7fab7c005fd0 tx=0x7fab7c02bd90 comp rx=0 tx=0).stop 2023-10-19T13:04:07.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.613+0000 7fab9461a640 1 -- 172.21.15.57:0/3851168703 shutdown_connections 2023-10-19T13:04:07.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.613+0000 7fab9461a640 1 --2- 172.21.15.57:0/3851168703 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fab8c0ceb40 0x7fab8c0b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:07.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.613+0000 7fab9461a640 1 --2- 172.21.15.57:0/3851168703 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab8c0cf340 0x7fab8c0d1910 unknown :-1 s=CLOSED pgs=1513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:07.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.613+0000 7fab9461a640 1 -- 172.21.15.57:0/3851168703 >> 172.21.15.57:0/3851168703 conn(0x7fab8c0d0d60 msgr2=0x7fab8c0d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:07.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.613+0000 7fab9461a640 1 -- 172.21.15.57:0/3851168703 shutdown_connections 2023-10-19T13:04:07.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.613+0000 7fab9461a640 1 -- 172.21.15.57:0/3851168703 wait complete. 2023-10-19T13:04:07.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.614+0000 7fab9461a640 1 Processor -- start 2023-10-19T13:04:07.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.614+0000 7fab9461a640 1 -- start start 2023-10-19T13:04:07.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.615+0000 7fab9461a640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fab8c0ceb40 0x7fab8c0c8840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:07.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.615+0000 7fab9461a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab8c0cf340 0x7fab8c15c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:07.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.615+0000 7fab9461a640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fab8c15b2a0 0x7fab8c15cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:07.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.615+0000 7fab9461a640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab8c15b6d0 con 0x7fab8c0cf340 2023-10-19T13:04:07.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.615+0000 7fab9461a640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab8c15b920 con 0x7fab8c0ceb40 2023-10-19T13:04:07.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.615+0000 7fab9461a640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab8c15bc00 con 0x7fab8c15b2a0 2023-10-19T13:04:07.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.615+0000 7fab91bb5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab8c0cf340 0x7fab8c15c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:07.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.615+0000 7fab91bb5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab8c0cf340 0x7fab8c15c6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33648/0 (socket says 172.21.15.57:33648) 2023-10-19T13:04:07.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.615+0000 7fab923b6640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fab8c0ceb40 0x7fab8c0c8840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:07.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.615+0000 7fab91bb5640 1 -- 172.21.15.57:0/3415528425 learned_addr learned my addr 172.21.15.57:0/3415528425 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:07.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.615+0000 7fab92bb7640 1 --2- 172.21.15.57:0/3415528425 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fab8c15b2a0 0x7fab8c15cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:07.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.616+0000 7fab923b6640 1 -- 172.21.15.57:0/3415528425 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fab8c15b2a0 msgr2=0x7fab8c15cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:07.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.616+0000 7fab923b6640 1 --2- 172.21.15.57:0/3415528425 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fab8c15b2a0 0x7fab8c15cc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:07.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.616+0000 7fab923b6640 1 -- 172.21.15.57:0/3415528425 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab8c0cf340 msgr2=0x7fab8c15c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:07.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.616+0000 7fab923b6640 1 --2- 172.21.15.57:0/3415528425 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab8c0cf340 0x7fab8c15c6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:07.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.616+0000 7fab923b6640 1 -- 172.21.15.57:0/3415528425 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab7c009560 con 0x7fab8c0ceb40 2023-10-19T13:04:07.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.616+0000 7fab91bb5640 1 --2- 172.21.15.57:0/3415528425 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab8c0cf340 0x7fab8c15c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T13:04:07.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.616+0000 7fab923b6640 1 --2- 172.21.15.57:0/3415528425 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fab8c0ceb40 0x7fab8c0c8840 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fab7c033000 tx=0x7fab7c02d9d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:07.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.616+0000 7fab82ffd640 1 -- 172.21.15.57:0/3415528425 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab7c02ddd0 con 0x7fab8c0ceb40 2023-10-19T13:04:07.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.616+0000 7fab82ffd640 1 -- 172.21.15.57:0/3415528425 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fab7c0036e0 con 0x7fab8c0ceb40 2023-10-19T13:04:07.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.616+0000 7fab9461a640 1 -- 172.21.15.57:0/3415528425 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab8c0b0d50 con 0x7fab8c0ceb40 2023-10-19T13:04:07.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.616+0000 7fab82ffd640 1 -- 172.21.15.57:0/3415528425 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab7c003ad0 con 0x7fab8c0ceb40 2023-10-19T13:04:07.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.616+0000 7fab9461a640 1 -- 172.21.15.57:0/3415528425 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab8c15c1c0 con 0x7fab8c0ceb40 2023-10-19T13:04:07.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.617+0000 7fab9461a640 1 -- 172.21.15.57:0/3415528425 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab8c15bd40 con 0x7fab8c0ceb40 2023-10-19T13:04:07.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.618+0000 7fab82ffd640 1 -- 172.21.15.57:0/3415528425 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 65) v1 ==== 89445+0+0 (secure 0 0 0) 0x7fab7c003c30 con 0x7fab8c0ceb40 2023-10-19T13:04:07.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.618+0000 7fab82ffd640 1 --2- 172.21.15.57:0/3415528425 >> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] conn(0x7fab7006b080 0x7fab7006d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:07.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.619+0000 7fab91bb5640 1 -- 172.21.15.57:0/3415528425 >> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] conn(0x7fab7006b080 msgr2=0x7fab7006d540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/1210673354 2023-10-19T13:04:07.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.619+0000 7fab82ffd640 1 -- 172.21.15.57:0/3415528425 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6401+0+0 (secure 0 0 0) 0x7fab7c0ac700 con 0x7fab8c0ceb40 2023-10-19T13:04:07.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.619+0000 7fab91bb5640 1 --2- 172.21.15.57:0/3415528425 >> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] conn(0x7fab7006b080 0x7fab7006d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T13:04:07.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.623+0000 7fab82ffd640 1 -- 172.21.15.57:0/3415528425 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fab8c15bd40 con 0x7fab8c0ceb40 2023-10-19T13:04:07.711 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:07.708+0000 7f4bbda14040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-19T13:04:07.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.737+0000 7fab82ffd640 1 -- 172.21.15.57:0/3415528425 <== mon.1 v2:172.21.15.179:3300/0 7 ==== mgrmap(e 66) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fab7c0776a0 con 0x7fab8c0ceb40 2023-10-19T13:04:07.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.738+0000 7fab82ffd640 1 -- 172.21.15.57:0/3415528425 >> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] conn(0x7fab7006b080 msgr2=0x7fab7006d540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T13:04:07.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.738+0000 7fab82ffd640 1 --2- 172.21.15.57:0/3415528425 >> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] conn(0x7fab7006b080 0x7fab7006d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:07.791 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:07.788+0000 7f4b42a6f640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:04:07.791 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:07.788+0000 7f4b42a6f640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:04:07.791 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:07.788+0000 7f4b42a6f640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:04:07.791 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:07.788+0000 7f4b42a6f640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:04:07.791 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:07.788+0000 7f4b42a6f640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:04:07.792 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:07.788+0000 7f4b3fa69640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:04:07.792 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:07.788+0000 7f4b3fa69640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:04:07.792 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:07.788+0000 7f4b3fa69640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:04:07.792 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:07.788+0000 7f4b3fa69640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:04:07.792 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:07.788+0000 7f4b3fa69640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:04:07.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.809+0000 7fab9461a640 1 -- 172.21.15.57:0/3415528425 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fab8c15bd40 con 0x7fab8c0ceb40 2023-10-19T13:04:07.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.810+0000 7fab82ffd640 1 -- 172.21.15.57:0/3415528425 <== mon.1 v2:172.21.15.179:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v132) v1 ==== 104+0+11 (secure 0 0 0) 0x7fab8c15bd40 con 0x7fab8c0ceb40 2023-10-19T13:04:07.813 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2023-10-19T13:04:07.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.812+0000 7fab9461a640 1 -- 172.21.15.57:0/3415528425 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fab8c0ceb40 msgr2=0x7fab8c0c8840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:07.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.812+0000 7fab9461a640 1 --2- 172.21.15.57:0/3415528425 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fab8c0ceb40 0x7fab8c0c8840 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fab7c033000 tx=0x7fab7c02d9d0 comp rx=0 tx=0).stop 2023-10-19T13:04:07.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.812+0000 7fab9461a640 1 -- 172.21.15.57:0/3415528425 shutdown_connections 2023-10-19T13:04:07.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.812+0000 7fab9461a640 1 --2- 172.21.15.57:0/3415528425 >> [v2:172.21.15.179:6832/1210673354,v1:172.21.15.179:6833/1210673354] conn(0x7fab7006b080 0x7fab7006d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:07.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.812+0000 7fab9461a640 1 --2- 172.21.15.57:0/3415528425 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fab8c15b2a0 0x7fab8c15cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:07.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.812+0000 7fab9461a640 1 --2- 172.21.15.57:0/3415528425 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab8c0cf340 0x7fab8c15c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:07.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.812+0000 7fab9461a640 1 --2- 172.21.15.57:0/3415528425 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fab8c0ceb40 0x7fab8c0c8840 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:07.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.812+0000 7fab9461a640 1 -- 172.21.15.57:0/3415528425 >> 172.21.15.57:0/3415528425 conn(0x7fab8c0d0d60 msgr2=0x7fab8c0cf720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:07.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.812+0000 7fab9461a640 1 -- 172.21.15.57:0/3415528425 shutdown_connections 2023-10-19T13:04:07.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.812+0000 7fab9461a640 1 -- 172.21.15.57:0/3415528425 wait complete. 2023-10-19T13:04:07.826 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-19T13:04:07.871 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:04:07.869+0000 7fdc13e3e040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-19T13:04:07.923 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:04:07.921+0000 7fdc13e3e040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-19T13:04:07.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.935+0000 7f156b79e640 1 -- 172.21.15.57:0/45017507 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f156415b640 msgr2=0x7f15640cf850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:07.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.935+0000 7f156b79e640 1 --2- 172.21.15.57:0/45017507 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f156415b640 0x7f15640cf850 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f15640894b0 tx=0x7f155402bd90 comp rx=0 tx=0).stop 2023-10-19T13:04:07.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.935+0000 7f156b79e640 1 -- 172.21.15.57:0/45017507 shutdown_connections 2023-10-19T13:04:07.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.935+0000 7f156b79e640 1 --2- 172.21.15.57:0/45017507 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f15640b0b10 0x7f15640b0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:07.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.935+0000 7f156b79e640 1 --2- 172.21.15.57:0/45017507 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f156415b640 0x7f15640cf850 unknown :-1 s=CLOSED pgs=1514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:07.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.935+0000 7f156b79e640 1 --2- 172.21.15.57:0/45017507 >> v2:172.21.15.179:3300/0 conn(0x7f156415b260 0x7f15640cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:07.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.935+0000 7f156b79e640 1 -- 172.21.15.57:0/45017507 >> 172.21.15.57:0/45017507 conn(0x7f156409a030 msgr2=0x7f15640d0d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:07.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.935+0000 7f156b79e640 1 -- 172.21.15.57:0/45017507 shutdown_connections 2023-10-19T13:04:07.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.936+0000 7f156b79e640 1 -- 172.21.15.57:0/45017507 wait complete. 2023-10-19T13:04:07.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.937+0000 7f156b79e640 1 Processor -- start 2023-10-19T13:04:07.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.937+0000 7f156b79e640 1 -- start start 2023-10-19T13:04:07.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.937+0000 7f156b79e640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f15640b0b10 0x7f156415c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:07.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.937+0000 7f156b79e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f156415b260 0x7f156409aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:07.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.937+0000 7f156b79e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f156415b640 0x7f156409afe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:07.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.937+0000 7f156b79e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f156415cd20 con 0x7f156415b260 2023-10-19T13:04:07.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.937+0000 7f156b79e640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f156415d000 con 0x7f156415b640 2023-10-19T13:04:07.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.937+0000 7f156b79e640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f156415d2e0 con 0x7f15640b0b10 2023-10-19T13:04:07.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.937+0000 7f1568d39640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f156415b260 0x7f156409aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:07.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.937+0000 7f1568d39640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f156415b260 0x7f156409aaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33664/0 (socket says 172.21.15.57:33664) 2023-10-19T13:04:07.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.937+0000 7f1568d39640 1 -- 172.21.15.57:0/3610184835 learned_addr learned my addr 172.21.15.57:0/3610184835 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:07.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.938+0000 7f1569d3b640 1 --2- 172.21.15.57:0/3610184835 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f156415b640 0x7f156409afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:07.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.938+0000 7f156953a640 1 --2- 172.21.15.57:0/3610184835 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f15640b0b10 0x7f156415c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:07.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.938+0000 7f1568d39640 1 -- 172.21.15.57:0/3610184835 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f15640b0b10 msgr2=0x7f156415c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:07.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.938+0000 7f1568d39640 1 --2- 172.21.15.57:0/3610184835 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f15640b0b10 0x7f156415c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:07.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.938+0000 7f1568d39640 1 -- 172.21.15.57:0/3610184835 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f156415b640 msgr2=0x7f156409afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:07.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.938+0000 7f1568d39640 1 --2- 172.21.15.57:0/3610184835 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f156415b640 0x7f156409afe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:07.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.938+0000 7f1568d39640 1 -- 172.21.15.57:0/3610184835 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1554009560 con 0x7f156415b260 2023-10-19T13:04:07.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.938+0000 7f1568d39640 1 --2- 172.21.15.57:0/3610184835 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f156415b260 0x7f156409aaa0 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7f1554009850 tx=0x7f155402dd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:07.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.938+0000 7f1559ffb640 1 -- 172.21.15.57:0/3610184835 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1554005650 con 0x7f156415b260 2023-10-19T13:04:07.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.938+0000 7f1559ffb640 1 -- 172.21.15.57:0/3610184835 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f15540038f0 con 0x7f156415b260 2023-10-19T13:04:07.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.938+0000 7f1559ffb640 1 -- 172.21.15.57:0/3610184835 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1554003ce0 con 0x7f156415b260 2023-10-19T13:04:07.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.938+0000 7f156b79e640 1 -- 172.21.15.57:0/3610184835 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f156415d5c0 con 0x7f156415b260 2023-10-19T13:04:07.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.938+0000 7f156b79e640 1 -- 172.21.15.57:0/3610184835 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f156415ddc0 con 0x7f156415b260 2023-10-19T13:04:07.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.939+0000 7f156b79e640 1 -- 172.21.15.57:0/3610184835 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15640d0610 con 0x7f156415b260 2023-10-19T13:04:07.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.940+0000 7f1559ffb640 1 -- 172.21.15.57:0/3610184835 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 66) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f155403e450 con 0x7f156415b260 2023-10-19T13:04:07.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.940+0000 7f1559ffb640 1 -- 172.21.15.57:0/3610184835 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 6887+0+0 (secure 0 0 0) 0x7f15540abec0 con 0x7f156415b260 2023-10-19T13:04:07.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:07.945+0000 7f1559ffb640 1 -- 172.21.15.57:0/3610184835 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f1554077040 con 0x7f156415b260 2023-10-19T13:04:08.031 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:04:08.028+0000 7fdc13e3e040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-19T13:04:08.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.120+0000 7f156b79e640 1 -- 172.21.15.57:0/3610184835 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f156415da80 con 0x7f156415b260 2023-10-19T13:04:08.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.121+0000 7f1559ffb640 1 -- 172.21.15.57:0/3610184835 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v132) v1 ==== 100+0+11 (secure 0 0 0) 0x7f15540420a0 con 0x7f156415b260 2023-10-19T13:04:08.124 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 64 2023-10-19T13:04:08.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.123+0000 7f156b79e640 1 -- 172.21.15.57:0/3610184835 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f156415b260 msgr2=0x7f156409aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:08.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.123+0000 7f156b79e640 1 --2- 172.21.15.57:0/3610184835 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f156415b260 0x7f156409aaa0 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7f1554009850 tx=0x7f155402dd60 comp rx=0 tx=0).stop 2023-10-19T13:04:08.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.123+0000 7f156b79e640 1 -- 172.21.15.57:0/3610184835 shutdown_connections 2023-10-19T13:04:08.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.123+0000 7f156b79e640 1 --2- 172.21.15.57:0/3610184835 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f156415b640 0x7f156409afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:08.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.123+0000 7f156b79e640 1 --2- 172.21.15.57:0/3610184835 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f156415b260 0x7f156409aaa0 unknown :-1 s=CLOSED pgs=1515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:08.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.123+0000 7f156b79e640 1 --2- 172.21.15.57:0/3610184835 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f15640b0b10 0x7f156415c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:08.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.123+0000 7f156b79e640 1 -- 172.21.15.57:0/3610184835 >> 172.21.15.57:0/3610184835 conn(0x7f156409a030 msgr2=0x7f15640aecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:08.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.123+0000 7f156b79e640 1 -- 172.21.15.57:0/3610184835 shutdown_connections 2023-10-19T13:04:08.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.123+0000 7f156b79e640 1 -- 172.21.15.57:0/3610184835 wait complete. 2023-10-19T13:04:08.136 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T13:04:08.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.243+0000 7f39ce91d640 1 -- 172.21.15.57:0/2527517853 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f39c815ccb0 msgr2=0x7f39c80c9750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:08.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.243+0000 7f39ce91d640 1 --2- 172.21.15.57:0/2527517853 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f39c815ccb0 0x7f39c80c9750 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7f39c80cc8a0 tx=0x7f39c002c260 comp rx=0 tx=0).stop 2023-10-19T13:04:08.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.243+0000 7f39ce91d640 1 -- 172.21.15.57:0/2527517853 shutdown_connections 2023-10-19T13:04:08.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.243+0000 7f39ce91d640 1 --2- 172.21.15.57:0/2527517853 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f39c815ccb0 0x7f39c80c9750 unknown :-1 s=CLOSED pgs=1516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:08.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.243+0000 7f39ce91d640 1 --2- 172.21.15.57:0/2527517853 >> v2:172.21.15.179:3300/0 conn(0x7f39c806b410 0x7f39c815c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:08.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.243+0000 7f39ce91d640 1 -- 172.21.15.57:0/2527517853 >> 172.21.15.57:0/2527517853 conn(0x7f39c8069930 msgr2=0x7f39c8069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:08.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.243+0000 7f39ce91d640 1 -- 172.21.15.57:0/2527517853 shutdown_connections 2023-10-19T13:04:08.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.243+0000 7f39ce91d640 1 -- 172.21.15.57:0/2527517853 wait complete. 2023-10-19T13:04:08.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.244+0000 7f39ce91d640 1 Processor -- start 2023-10-19T13:04:08.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.244+0000 7f39ce91d640 1 -- start start 2023-10-19T13:04:08.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.245+0000 7f39ce91d640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f39c806b030 0x7f39c8138dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:08.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.245+0000 7f39ce91d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f39c806b410 0x7f39c8150c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:08.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.245+0000 7f39ce91d640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f39c815ccb0 0x7f39c815b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:08.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.245+0000 7f39ce91d640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39c81ef1f0 con 0x7f39c806b410 2023-10-19T13:04:08.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.245+0000 7f39ce91d640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39c8151670 con 0x7f39c815ccb0 2023-10-19T13:04:08.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.245+0000 7f39ce91d640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39c8152490 con 0x7f39c806b030 2023-10-19T13:04:08.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.245+0000 7f39c77fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f39c806b410 0x7f39c8150c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:08.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.245+0000 7f39c77fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f39c806b410 0x7f39c8150c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33684/0 (socket says 172.21.15.57:33684) 2023-10-19T13:04:08.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.245+0000 7f39c77fe640 1 -- 172.21.15.57:0/2916508473 learned_addr learned my addr 172.21.15.57:0/2916508473 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:08.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.245+0000 7f39cceba640 1 --2- 172.21.15.57:0/2916508473 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f39c815ccb0 0x7f39c815b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:08.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.245+0000 7f39c7fff640 1 --2- 172.21.15.57:0/2916508473 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f39c806b030 0x7f39c8138dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:08.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.246+0000 7f39c77fe640 1 -- 172.21.15.57:0/2916508473 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f39c806b030 msgr2=0x7f39c8138dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:08.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.246+0000 7f39c77fe640 1 --2- 172.21.15.57:0/2916508473 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f39c806b030 0x7f39c8138dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:08.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.246+0000 7f39c77fe640 1 -- 172.21.15.57:0/2916508473 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f39c815ccb0 msgr2=0x7f39c815b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:08.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.246+0000 7f39c77fe640 1 --2- 172.21.15.57:0/2916508473 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f39c815ccb0 0x7f39c815b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:08.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.246+0000 7f39c77fe640 1 -- 172.21.15.57:0/2916508473 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39c0009900 con 0x7f39c806b410 2023-10-19T13:04:08.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.246+0000 7f39c77fe640 1 --2- 172.21.15.57:0/2916508473 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f39c806b410 0x7f39c8150c70 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f39bc00d980 tx=0x7f39bc00de40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:08.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.246+0000 7f39c4ff9640 1 -- 172.21.15.57:0/2916508473 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39bc009c30 con 0x7f39c806b410 2023-10-19T13:04:08.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.246+0000 7f39c4ff9640 1 -- 172.21.15.57:0/2916508473 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f39bc009e80 con 0x7f39c806b410 2023-10-19T13:04:08.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.246+0000 7f39ce91d640 1 -- 172.21.15.57:0/2916508473 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39c8139310 con 0x7f39c806b410 2023-10-19T13:04:08.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.246+0000 7f39c4ff9640 1 -- 172.21.15.57:0/2916508473 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39bc0106a0 con 0x7f39c806b410 2023-10-19T13:04:08.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.246+0000 7f39ce91d640 1 -- 172.21.15.57:0/2916508473 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39c8151b60 con 0x7f39c806b410 2023-10-19T13:04:08.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.247+0000 7f39ce91d640 1 -- 172.21.15.57:0/2916508473 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39c80690e0 con 0x7f39c806b410 2023-10-19T13:04:08.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.247+0000 7f39c4ff9640 1 -- 172.21.15.57:0/2916508473 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 66) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f39bc010800 con 0x7f39c806b410 2023-10-19T13:04:08.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.248+0000 7f39c4ff9640 1 -- 172.21.15.57:0/2916508473 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 6887+0+0 (secure 0 0 0) 0x7f39bc089940 con 0x7f39c806b410 2023-10-19T13:04:08.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.252+0000 7f39c4ff9640 1 -- 172.21.15.57:0/2916508473 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f39bc054a50 con 0x7f39c806b410 2023-10-19T13:04:08.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.470+0000 7f39ce91d640 1 -- 172.21.15.57:0/2916508473 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f39c80cc460 con 0x7f39c806b410 2023-10-19T13:04:08.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.471+0000 7f39c4ff9640 1 -- 172.21.15.57:0/2916508473 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4912 (secure 0 0 0) 0x7f39bc020090 con 0x7f39c806b410 2023-10-19T13:04:08.474 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:04:08.474 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":50,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/2033977424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2033977424},{"type":"v1","addr":"172.21.15.179:6836","nonce":2033977424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":8568,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/981095684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":981095684},{"type":"v1","addr":"172.21.15.57:6834","nonce":981095684}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":8571,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/383147069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":383147069},{"type":"v1","addr":"172.21.15.57:6835","nonce":383147069}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T13:02:57.697820+0000","modified":"2023-10-19T13:02:59.684888+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8577},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8577":{"gid":8577,"name":"d","rank":0,"incarnation":48,"state":"up:active","state_seq":3,"addr":"172.21.15.179:6837/768231146","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":768231146},{"type":"v1","addr":"172.21.15.179:6837","nonce":768231146}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2023-10-19T13:04:08.475 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 50 2023-10-19T13:04:08.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.472+0000 7f39ce91d640 1 -- 172.21.15.57:0/2916508473 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f39c806b410 msgr2=0x7f39c8150c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:08.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.472+0000 7f39ce91d640 1 --2- 172.21.15.57:0/2916508473 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f39c806b410 0x7f39c8150c70 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f39bc00d980 tx=0x7f39bc00de40 comp rx=0 tx=0).stop 2023-10-19T13:04:08.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.472+0000 7f39ce91d640 1 -- 172.21.15.57:0/2916508473 shutdown_connections 2023-10-19T13:04:08.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.472+0000 7f39ce91d640 1 --2- 172.21.15.57:0/2916508473 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f39c815ccb0 0x7f39c815b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:08.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.472+0000 7f39ce91d640 1 --2- 172.21.15.57:0/2916508473 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f39c806b410 0x7f39c8150c70 unknown :-1 s=CLOSED pgs=1517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:08.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.472+0000 7f39ce91d640 1 --2- 172.21.15.57:0/2916508473 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f39c806b030 0x7f39c8138dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:08.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.472+0000 7f39ce91d640 1 -- 172.21.15.57:0/2916508473 >> 172.21.15.57:0/2916508473 conn(0x7f39c8069930 msgr2=0x7f39c8069d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:08.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.473+0000 7f39ce91d640 1 -- 172.21.15.57:0/2916508473 shutdown_connections 2023-10-19T13:04:08.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.473+0000 7f39ce91d640 1 -- 172.21.15.57:0/2916508473 wait complete. 2023-10-19T13:04:08.486 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T13:04:08.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.592+0000 7f324dfa5640 1 -- 172.21.15.57:0/1575506770 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f32480abc10 msgr2=0x7f32480c9b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:08.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.592+0000 7f324dfa5640 1 --2- 172.21.15.57:0/1575506770 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f32480abc10 0x7f32480c9b30 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f3240009ca0 tx=0x7f324002c2e0 comp rx=0 tx=0).stop 2023-10-19T13:04:08.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.592+0000 7f324dfa5640 1 -- 172.21.15.57:0/1575506770 shutdown_connections 2023-10-19T13:04:08.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.592+0000 7f324dfa5640 1 --2- 172.21.15.57:0/1575506770 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f32480abc10 0x7f32480c9b30 unknown :-1 s=CLOSED pgs=1519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:08.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.592+0000 7f324dfa5640 1 --2- 172.21.15.57:0/1575506770 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f324806af00 0x7f324806b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:08.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.592+0000 7f324dfa5640 1 -- 172.21.15.57:0/1575506770 >> 172.21.15.57:0/1575506770 conn(0x7f3248069930 msgr2=0x7f3248069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:08.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.592+0000 7f324dfa5640 1 -- 172.21.15.57:0/1575506770 shutdown_connections 2023-10-19T13:04:08.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.592+0000 7f324dfa5640 1 -- 172.21.15.57:0/1575506770 wait complete. 2023-10-19T13:04:08.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.593+0000 7f324dfa5640 1 Processor -- start 2023-10-19T13:04:08.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.593+0000 7f324dfa5640 1 -- start start 2023-10-19T13:04:08.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.594+0000 7f324dfa5640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f324806af00 0x7f3248139440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:08.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.594+0000 7f324dfa5640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f32480abc10 0x7f3248070a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:08.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.594+0000 7f324dfa5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f32480c96b0 0x7f3248139c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:08.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.594+0000 7f324dfa5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32480705c0 con 0x7f32480c96b0 2023-10-19T13:04:08.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.594+0000 7f324dfa5640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3248070130 con 0x7f324806af00 2023-10-19T13:04:08.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.594+0000 7f324dfa5640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f324806f520 con 0x7f32480abc10 2023-10-19T13:04:08.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.594+0000 7f3247fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f32480c96b0 0x7f3248139c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:08.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.594+0000 7f3247fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f32480c96b0 0x7f3248139c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33706/0 (socket says 172.21.15.57:33706) 2023-10-19T13:04:08.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.594+0000 7f3247fff640 1 -- 172.21.15.57:0/1692817335 learned_addr learned my addr 172.21.15.57:0/1692817335 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:08.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.594+0000 7f32477fe640 1 --2- 172.21.15.57:0/1692817335 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f324806af00 0x7f3248139440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:08.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.594+0000 7f3246ffd640 1 --2- 172.21.15.57:0/1692817335 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f32480abc10 0x7f3248070a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:08.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.594+0000 7f3247fff640 1 -- 172.21.15.57:0/1692817335 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f32480abc10 msgr2=0x7f3248070a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:08.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.594+0000 7f3247fff640 1 --2- 172.21.15.57:0/1692817335 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f32480abc10 0x7f3248070a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:08.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.594+0000 7f3247fff640 1 -- 172.21.15.57:0/1692817335 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f324806af00 msgr2=0x7f3248139440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:08.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.594+0000 7f3247fff640 1 --2- 172.21.15.57:0/1692817335 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f324806af00 0x7f3248139440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:08.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.594+0000 7f3247fff640 1 -- 172.21.15.57:0/1692817335 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3240009900 con 0x7f32480c96b0 2023-10-19T13:04:08.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.594+0000 7f3246ffd640 1 --2- 172.21.15.57:0/1692817335 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f32480abc10 0x7f3248070a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T13:04:08.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.595+0000 7f3247fff640 1 --2- 172.21.15.57:0/1692817335 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f32480c96b0 0x7f3248139c60 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f3240006010 tx=0x7f3240035ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:08.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.595+0000 7f322ffff640 1 -- 172.21.15.57:0/1692817335 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3240004240 con 0x7f32480c96b0 2023-10-19T13:04:08.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.595+0000 7f322ffff640 1 -- 172.21.15.57:0/1692817335 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f3240004490 con 0x7f32480c96b0 2023-10-19T13:04:08.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.595+0000 7f324dfa5640 1 -- 172.21.15.57:0/1692817335 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f324806f800 con 0x7f32480c96b0 2023-10-19T13:04:08.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.595+0000 7f322ffff640 1 -- 172.21.15.57:0/1692817335 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3240004840 con 0x7f32480c96b0 2023-10-19T13:04:08.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.595+0000 7f324dfa5640 1 -- 172.21.15.57:0/1692817335 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f324806fdc0 con 0x7f32480c96b0 2023-10-19T13:04:08.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.596+0000 7f324dfa5640 1 -- 172.21.15.57:0/1692817335 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32480702a0 con 0x7f32480c96b0 2023-10-19T13:04:08.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.596+0000 7f322ffff640 1 -- 172.21.15.57:0/1692817335 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 66) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f324003b040 con 0x7f32480c96b0 2023-10-19T13:04:08.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.597+0000 7f322ffff640 1 -- 172.21.15.57:0/1692817335 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 6887+0+0 (secure 0 0 0) 0x7f32400ac3c0 con 0x7f32480c96b0 2023-10-19T13:04:08.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.601+0000 7f322ffff640 1 -- 172.21.15.57:0/1692817335 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3240077330 con 0x7f32480c96b0 2023-10-19T13:04:08.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.735+0000 7f322ffff640 1 -- 172.21.15.57:0/1692817335 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mgrmap(e 67) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f324000a040 con 0x7f32480c96b0 2023-10-19T13:04:08.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.735+0000 7f322ffff640 1 --2- 172.21.15.57:0/1692817335 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f3234070cf0 0x7f32340730e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:08.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.737+0000 7f32477fe640 1 --2- 172.21.15.57:0/1692817335 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f3234070cf0 0x7f32340730e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:08.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.740+0000 7f32477fe640 1 --2- 172.21.15.57:0/1692817335 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f3234070cf0 0x7f32340730e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3238005e10 tx=0x7f3238005da0 comp rx=0 tx=0).ready entity=mgr.7960 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:08.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.782+0000 7f324dfa5640 1 -- 172.21.15.57:0/1692817335 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f32480690e0 con 0x7f32480c96b0 2023-10-19T13:04:08.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.783+0000 7f322ffff640 1 -- 172.21.15.57:0/1692817335 <== mon.0 v2:172.21.15.57:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v132) v1 ==== 74+0+18024 (secure 0 0 0) 0x7f32400a5330 con 0x7f32480c96b0 2023-10-19T13:04:08.786 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:04:08.786 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":132,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T13:04:07.723992+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2023-10-19T13:02:55.916158+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"129","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2023-10-19T13:02:56.675559+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"132","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":132,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.179:0/865284344":"2023-10-20T13:04:07.723901+0000","172.21.15.179:6832/1210673354":"2023-10-20T13:04:07.723901+0000","172.21.15.179:0/4226732685":"2023-10-20T13:04:07.723901+0000","172.21.15.179:0/3662182511":"2023-10-20T13:04:07.723901+0000","172.21.15.179:0/2769057100":"2023-10-20T13:04:07.723901+0000","172.21.15.179:6833/1210673354":"2023-10-20T13:04:07.723901+0000","172.21.15.179:6833/4155586759":"2023-10-20T13:03:08.761549+0000","172.21.15.179:0/2669494882":"2023-10-20T13:04:07.723901+0000","172.21.15.179:0/1859890742":"2023-10-20T13:04:07.723901+0000","172.21.15.179:6832/4155586759":"2023-10-20T13:03:08.761549+0000","172.21.15.179:0/1341670560":"2023-10-20T13:03:08.761549+0000","172.21.15.179:0/33839263":"2023-10-20T13:03:08.761549+0000","172.21.15.179:0/2362202135":"2023-10-20T13:04:07.723901+0000","172.21.15.179:0/3689030544":"2023-10-20T13:03:08.761549+0000","172.21.15.179:0/3560759535":"2023-10-20T13:04:07.723901+0000","172.21.15.179:0/3037001824":"2023-10-20T13:03:08.761549+0000","172.21.15.179:0/1069074991":"2023-10-20T13:03:08.761549+0000","172.21.15.179:0/831719196":"2023-10-20T13:03:08.761549+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T13:04:08.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.785+0000 7f324dfa5640 1 -- 172.21.15.57:0/1692817335 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f3234070cf0 msgr2=0x7f32340730e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:08.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.785+0000 7f324dfa5640 1 --2- 172.21.15.57:0/1692817335 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f3234070cf0 0x7f32340730e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3238005e10 tx=0x7f3238005da0 comp rx=0 tx=0).stop 2023-10-19T13:04:08.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.785+0000 7f324dfa5640 1 -- 172.21.15.57:0/1692817335 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f32480c96b0 msgr2=0x7f3248139c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:08.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.785+0000 7f324dfa5640 1 --2- 172.21.15.57:0/1692817335 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f32480c96b0 0x7f3248139c60 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f3240006010 tx=0x7f3240035ad0 comp rx=0 tx=0).stop 2023-10-19T13:04:08.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.785+0000 7f324dfa5640 1 -- 172.21.15.57:0/1692817335 shutdown_connections 2023-10-19T13:04:08.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.785+0000 7f324dfa5640 1 --2- 172.21.15.57:0/1692817335 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f3234070cf0 0x7f32340730e0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:08.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.785+0000 7f324dfa5640 1 --2- 172.21.15.57:0/1692817335 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f32480c96b0 0x7f3248139c60 unknown :-1 s=CLOSED pgs=1520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:08.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.785+0000 7f324dfa5640 1 --2- 172.21.15.57:0/1692817335 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f32480abc10 0x7f3248070a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:08.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.785+0000 7f324dfa5640 1 --2- 172.21.15.57:0/1692817335 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f324806af00 0x7f3248139440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:08.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.785+0000 7f324dfa5640 1 -- 172.21.15.57:0/1692817335 >> 172.21.15.57:0/1692817335 conn(0x7f3248069930 msgr2=0x7f324815bae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:08.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.785+0000 7f324dfa5640 1 -- 172.21.15.57:0/1692817335 shutdown_connections 2023-10-19T13:04:08.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.785+0000 7f324dfa5640 1 -- 172.21.15.57:0/1692817335 wait complete. 2023-10-19T13:04:08.798 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-19T13:04:08.799 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T13:04:08.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.906+0000 7faea9a10640 1 -- 172.21.15.57:0/3420999378 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fae900047c0 msgr2=0x7fae90004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:08.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.906+0000 7faea9a10640 1 --2- 172.21.15.57:0/3420999378 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fae900047c0 0x7fae90004c40 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7fae8c009b80 tx=0x7fae900057f0 comp rx=0 tx=0).stop 2023-10-19T13:04:08.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.906+0000 7faea37fe640 1 -- 172.21.15.57:0/3420999378 reap_dead start 2023-10-19T13:04:08.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.906+0000 7faea9a10640 1 -- 172.21.15.57:0/3420999378 shutdown_connections 2023-10-19T13:04:08.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.906+0000 7faea9a10640 1 -- 172.21.15.57:0/3420999378 >> 172.21.15.57:0/3420999378 conn(0x7faea409a010 msgr2=0x7faea40d0de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:08.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.906+0000 7faea9a10640 1 -- 172.21.15.57:0/3420999378 shutdown_connections 2023-10-19T13:04:08.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.906+0000 7faea9a10640 1 -- 172.21.15.57:0/3420999378 wait complete. 2023-10-19T13:04:08.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.907+0000 7faea9a10640 1 Processor -- start 2023-10-19T13:04:08.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.907+0000 7faea9a10640 1 -- start start 2023-10-19T13:04:08.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.908+0000 7faea9a10640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fae900047c0 0x7faea40c7240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:08.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.908+0000 7faea9a10640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7faea40ae410 0x7faea41e4850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:08.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.908+0000 7faea9a10640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faea41e4d90 0x7faea4157420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:08.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.908+0000 7faea9a10640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faea41e3a90 con 0x7faea41e4d90 2023-10-19T13:04:08.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.908+0000 7faea9a10640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faea41e3d70 con 0x7fae900047c0 2023-10-19T13:04:08.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.908+0000 7faea9a10640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faea41e4050 con 0x7faea40ae410 2023-10-19T13:04:08.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.908+0000 7faea37fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faea41e4d90 0x7faea4157420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:08.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.908+0000 7faea37fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faea41e4d90 0x7faea4157420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33730/0 (socket says 172.21.15.57:33730) 2023-10-19T13:04:08.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.908+0000 7faea37fe640 1 -- 172.21.15.57:0/4243174935 learned_addr learned my addr 172.21.15.57:0/4243174935 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:08.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.908+0000 7faea2ffd640 1 --2- 172.21.15.57:0/4243174935 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fae900047c0 0x7faea40c7240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:08.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.908+0000 7faea27fc640 1 --2- 172.21.15.57:0/4243174935 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7faea40ae410 0x7faea41e4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:08.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.908+0000 7faea37fe640 1 -- 172.21.15.57:0/4243174935 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7faea40ae410 msgr2=0x7faea41e4850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:08.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.908+0000 7faea37fe640 1 --2- 172.21.15.57:0/4243174935 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7faea40ae410 0x7faea41e4850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:08.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.908+0000 7faea37fe640 1 -- 172.21.15.57:0/4243174935 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fae900047c0 msgr2=0x7faea40c7240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:08.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.908+0000 7faea37fe640 1 --2- 172.21.15.57:0/4243174935 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fae900047c0 0x7faea40c7240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:08.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.908+0000 7faea37fe640 1 -- 172.21.15.57:0/4243174935 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faea41e4330 con 0x7faea41e4d90 2023-10-19T13:04:08.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.908+0000 7faea37fe640 1 --2- 172.21.15.57:0/4243174935 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faea41e4d90 0x7faea4157420 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7faea40cb3c0 tx=0x7fae9400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:08.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.908+0000 7fae9bfff640 1 -- 172.21.15.57:0/4243174935 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae94003d80 con 0x7faea41e4d90 2023-10-19T13:04:08.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.909+0000 7fae9bfff640 1 -- 172.21.15.57:0/4243174935 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fae94004b20 con 0x7faea41e4d90 2023-10-19T13:04:08.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.909+0000 7faea9a10640 1 -- 172.21.15.57:0/4243174935 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faea40c7780 con 0x7faea41e4d90 2023-10-19T13:04:08.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.909+0000 7fae9bfff640 1 -- 172.21.15.57:0/4243174935 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae94004030 con 0x7faea41e4d90 2023-10-19T13:04:08.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.909+0000 7faea9a10640 1 -- 172.21.15.57:0/4243174935 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faea40aea70 con 0x7faea41e4d90 2023-10-19T13:04:08.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.909+0000 7faea9a10640 1 -- 172.21.15.57:0/4243174935 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faea40690e0 con 0x7faea41e4d90 2023-10-19T13:04:08.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.910+0000 7fae9bfff640 1 -- 172.21.15.57:0/4243174935 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 67) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fae94004190 con 0x7faea41e4d90 2023-10-19T13:04:08.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.910+0000 7fae9bfff640 1 --2- 172.21.15.57:0/4243174935 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7fae8406ac90 0x7fae8406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:08.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.911+0000 7fae9bfff640 1 -- 172.21.15.57:0/4243174935 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 6887+0+0 (secure 0 0 0) 0x7fae9408a680 con 0x7faea41e4d90 2023-10-19T13:04:08.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.911+0000 7faea2ffd640 1 --2- 172.21.15.57:0/4243174935 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7fae8406ac90 0x7fae8406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:08.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.911+0000 7faea2ffd640 1 --2- 172.21.15.57:0/4243174935 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7fae8406ac90 0x7fae8406d150 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fae9c000fb0 tx=0x7fae9c0067a0 comp rx=0 tx=0).ready entity=mgr.7960 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:08.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:08.915+0000 7fae9bfff640 1 -- 172.21.15.57:0/4243174935 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fae94055550 con 0x7faea41e4d90 2023-10-19T13:04:09.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.127+0000 7faea9a10640 1 -- 172.21.15.57:0/4243174935 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faea41e3bd0 con 0x7faea41e4d90 2023-10-19T13:04:09.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.127+0000 7fae9bfff640 1 -- 172.21.15.57:0/4243174935 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4912 (secure 0 0 0) 0x7faea41e3bd0 con 0x7faea41e4d90 2023-10-19T13:04:09.131 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:04:09.131 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":50,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/2033977424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2033977424},{"type":"v1","addr":"172.21.15.179:6836","nonce":2033977424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":8568,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/981095684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":981095684},{"type":"v1","addr":"172.21.15.57:6834","nonce":981095684}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":8571,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/383147069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":383147069},{"type":"v1","addr":"172.21.15.57:6835","nonce":383147069}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T13:02:57.697820+0000","modified":"2023-10-19T13:02:59.684888+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8577},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8577":{"gid":8577,"name":"d","rank":0,"incarnation":48,"state":"up:active","state_seq":3,"addr":"172.21.15.179:6837/768231146","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":768231146},{"type":"v1","addr":"172.21.15.179:6837","nonce":768231146}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2023-10-19T13:04:09.131 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 50 2023-10-19T13:04:09.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.129+0000 7faea9a10640 1 -- 172.21.15.57:0/4243174935 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7fae8406ac90 msgr2=0x7fae8406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:09.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.129+0000 7faea9a10640 1 --2- 172.21.15.57:0/4243174935 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7fae8406ac90 0x7fae8406d150 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fae9c000fb0 tx=0x7fae9c0067a0 comp rx=0 tx=0).stop 2023-10-19T13:04:09.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.129+0000 7faea9a10640 1 -- 172.21.15.57:0/4243174935 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faea41e4d90 msgr2=0x7faea4157420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:09.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.129+0000 7faea9a10640 1 --2- 172.21.15.57:0/4243174935 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faea41e4d90 0x7faea4157420 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7faea40cb3c0 tx=0x7fae9400f790 comp rx=0 tx=0).stop 2023-10-19T13:04:09.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.129+0000 7faea9a10640 1 -- 172.21.15.57:0/4243174935 shutdown_connections 2023-10-19T13:04:09.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.129+0000 7faea9a10640 1 --2- 172.21.15.57:0/4243174935 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7fae8406ac90 0x7fae8406d150 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:09.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.129+0000 7faea9a10640 1 --2- 172.21.15.57:0/4243174935 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faea41e4d90 0x7faea4157420 unknown :-1 s=CLOSED pgs=1523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:09.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.129+0000 7faea9a10640 1 --2- 172.21.15.57:0/4243174935 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7faea40ae410 0x7faea41e4850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:09.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.129+0000 7faea9a10640 1 --2- 172.21.15.57:0/4243174935 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fae900047c0 0x7faea40c7240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:09.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.129+0000 7faea9a10640 1 -- 172.21.15.57:0/4243174935 >> 172.21.15.57:0/4243174935 conn(0x7faea409a010 msgr2=0x7faea4159de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:09.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.129+0000 7faea9a10640 1 -- 172.21.15.57:0/4243174935 shutdown_connections 2023-10-19T13:04:09.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.129+0000 7faea9a10640 1 -- 172.21.15.57:0/4243174935 wait complete. 2023-10-19T13:04:09.143 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T13:04:09.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.249+0000 7f000ede2640 1 -- 172.21.15.57:0/1677770013 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f00080b3070 msgr2=0x7f000806da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:09.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.249+0000 7f000ede2640 1 --2- 172.21.15.57:0/1677770013 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f00080b3070 0x7f000806da20 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7efff8009ca0 tx=0x7efff802c240 comp rx=0 tx=0).stop 2023-10-19T13:04:09.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.249+0000 7f000ede2640 1 -- 172.21.15.57:0/1677770013 shutdown_connections 2023-10-19T13:04:09.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.249+0000 7f000ede2640 1 --2- 172.21.15.57:0/1677770013 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f00080b3070 0x7f000806da20 unknown :-1 s=CLOSED pgs=1525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:09.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.249+0000 7f000ede2640 1 -- 172.21.15.57:0/1677770013 >> 172.21.15.57:0/1677770013 conn(0x7f00080b3f00 msgr2=0x7f00080b4310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:09.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.249+0000 7f000ede2640 1 -- 172.21.15.57:0/1677770013 shutdown_connections 2023-10-19T13:04:09.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.249+0000 7f000ede2640 1 -- 172.21.15.57:0/1677770013 wait complete. 2023-10-19T13:04:09.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.250+0000 7f000ede2640 1 Processor -- start 2023-10-19T13:04:09.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.250+0000 7f000ede2640 1 -- start start 2023-10-19T13:04:09.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.251+0000 7f000ede2640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f00080acd40 0x7f00080afbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:09.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.251+0000 7f000ede2640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f00080b24d0 0x7f00081f0220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:09.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.251+0000 7f000ede2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f00080b3070 0x7f000814d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:09.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.251+0000 7f000ede2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00080b0ee0 con 0x7f00080b3070 2023-10-19T13:04:09.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.251+0000 7f000ede2640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00080b0260 con 0x7f00080acd40 2023-10-19T13:04:09.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.251+0000 7f000ede2640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00080b0540 con 0x7f00080b24d0 2023-10-19T13:04:09.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.251+0000 7f000d37f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f00080b3070 0x7f000814d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:09.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.251+0000 7f000d37f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f00080b3070 0x7f000814d500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33740/0 (socket says 172.21.15.57:33740) 2023-10-19T13:04:09.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.251+0000 7f000d37f640 1 -- 172.21.15.57:0/1569422807 learned_addr learned my addr 172.21.15.57:0/1569422807 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:09.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.251+0000 7f000cb7e640 1 --2- 172.21.15.57:0/1569422807 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f00080acd40 0x7f00080afbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:09.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.252+0000 7f000d37f640 1 -- 172.21.15.57:0/1569422807 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f00080b24d0 msgr2=0x7f00081f0220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:09.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.252+0000 7f0007fff640 1 --2- 172.21.15.57:0/1569422807 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f00080b24d0 0x7f00081f0220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:09.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.252+0000 7f000d37f640 1 --2- 172.21.15.57:0/1569422807 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f00080b24d0 0x7f00081f0220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:09.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.252+0000 7f000d37f640 1 -- 172.21.15.57:0/1569422807 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f00080acd40 msgr2=0x7f00080afbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:09.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.252+0000 7f000d37f640 1 --2- 172.21.15.57:0/1569422807 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f00080acd40 0x7f00080afbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:09.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.252+0000 7f000d37f640 1 -- 172.21.15.57:0/1569422807 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efff8009900 con 0x7f00080b3070 2023-10-19T13:04:09.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.252+0000 7f000cb7e640 1 --2- 172.21.15.57:0/1569422807 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f00080acd40 0x7f00080afbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T13:04:09.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.252+0000 7f000d37f640 1 --2- 172.21.15.57:0/1569422807 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f00080b3070 0x7f000814d500 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7efff0001eb0 tx=0x7efff000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:09.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.252+0000 7f00057fa640 1 -- 172.21.15.57:0/1569422807 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efff00052d0 con 0x7f00080b3070 2023-10-19T13:04:09.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.252+0000 7f00057fa640 1 -- 172.21.15.57:0/1569422807 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7efff00035f0 con 0x7f00080b3070 2023-10-19T13:04:09.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.252+0000 7f000ede2640 1 -- 172.21.15.57:0/1569422807 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00080b0820 con 0x7f00080b3070 2023-10-19T13:04:09.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.252+0000 7f00057fa640 1 -- 172.21.15.57:0/1569422807 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efff00039e0 con 0x7f00080b3070 2023-10-19T13:04:09.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.252+0000 7f000ede2640 1 -- 172.21.15.57:0/1569422807 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00081f0760 con 0x7f00080b3070 2023-10-19T13:04:09.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.253+0000 7f000ede2640 1 -- 172.21.15.57:0/1569422807 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00080b1020 con 0x7f00080b3070 2023-10-19T13:04:09.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.253+0000 7f00057fa640 1 -- 172.21.15.57:0/1569422807 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 67) v1 ==== 89067+0+0 (secure 0 0 0) 0x7efff0003b40 con 0x7f00080b3070 2023-10-19T13:04:09.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.254+0000 7f00057fa640 1 --2- 172.21.15.57:0/1569422807 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7effe806ac90 0x7effe806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:09.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.254+0000 7f00057fa640 1 -- 172.21.15.57:0/1569422807 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 6887+0+0 (secure 0 0 0) 0x7efff0088ed0 con 0x7f00080b3070 2023-10-19T13:04:09.262 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.254+0000 7f000cb7e640 1 --2- 172.21.15.57:0/1569422807 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7effe806ac90 0x7effe806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:09.262 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.255+0000 7f000cb7e640 1 --2- 172.21.15.57:0/1569422807 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7effe806ac90 0x7effe806d150 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0008155ee0 tx=0x7f0000009040 comp rx=0 tx=0).ready entity=mgr.7960 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:09.262 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.258+0000 7f00057fa640 1 -- 172.21.15.57:0/1569422807 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f00080b1020 con 0x7f00080b3070 2023-10-19T13:04:09.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.471+0000 7f000ede2640 1 -- 172.21.15.57:0/1569422807 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f00080b1020 con 0x7f00080b3070 2023-10-19T13:04:09.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.472+0000 7f00057fa640 1 -- 172.21.15.57:0/1569422807 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4912 (secure 0 0 0) 0x7f00080b1020 con 0x7f00080b3070 2023-10-19T13:04:09.475 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:04:09.475 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":50,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/2033977424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2033977424},{"type":"v1","addr":"172.21.15.179:6836","nonce":2033977424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":8568,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/981095684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":981095684},{"type":"v1","addr":"172.21.15.57:6834","nonce":981095684}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":8571,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/383147069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":383147069},{"type":"v1","addr":"172.21.15.57:6835","nonce":383147069}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T13:02:57.697820+0000","modified":"2023-10-19T13:02:59.684888+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8577},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8577":{"gid":8577,"name":"d","rank":0,"incarnation":48,"state":"up:active","state_seq":3,"addr":"172.21.15.179:6837/768231146","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":768231146},{"type":"v1","addr":"172.21.15.179:6837","nonce":768231146}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2023-10-19T13:04:09.475 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 50 2023-10-19T13:04:09.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.473+0000 7f000ede2640 1 -- 172.21.15.57:0/1569422807 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7effe806ac90 msgr2=0x7effe806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:09.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.473+0000 7f000ede2640 1 --2- 172.21.15.57:0/1569422807 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7effe806ac90 0x7effe806d150 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0008155ee0 tx=0x7f0000009040 comp rx=0 tx=0).stop 2023-10-19T13:04:09.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.474+0000 7f000ede2640 1 -- 172.21.15.57:0/1569422807 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f00080b3070 msgr2=0x7f000814d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:09.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.474+0000 7f000ede2640 1 --2- 172.21.15.57:0/1569422807 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f00080b3070 0x7f000814d500 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7efff0001eb0 tx=0x7efff000c890 comp rx=0 tx=0).stop 2023-10-19T13:04:09.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.474+0000 7f000ede2640 1 -- 172.21.15.57:0/1569422807 shutdown_connections 2023-10-19T13:04:09.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.474+0000 7f000ede2640 1 --2- 172.21.15.57:0/1569422807 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7effe806ac90 0x7effe806d150 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:09.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.474+0000 7f000ede2640 1 --2- 172.21.15.57:0/1569422807 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f00080b3070 0x7f000814d500 unknown :-1 s=CLOSED pgs=1526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:09.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.474+0000 7f000ede2640 1 --2- 172.21.15.57:0/1569422807 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f00080b24d0 0x7f00081f0220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:09.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.474+0000 7f000ede2640 1 --2- 172.21.15.57:0/1569422807 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f00080acd40 0x7f00080afbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:09.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.474+0000 7f000ede2640 1 -- 172.21.15.57:0/1569422807 >> 172.21.15.57:0/1569422807 conn(0x7f00080b3f00 msgr2=0x7f000815b1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:09.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.474+0000 7f000ede2640 1 -- 172.21.15.57:0/1569422807 shutdown_connections 2023-10-19T13:04:09.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.474+0000 7f000ede2640 1 -- 172.21.15.57:0/1569422807 wait complete. 2023-10-19T13:04:09.487 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T13:04:09.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.593+0000 7f0b32cd7640 1 -- 172.21.15.57:0/3238723155 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0b2c13e520 msgr2=0x7f0b2c06bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:09.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.593+0000 7f0b32cd7640 1 --2- 172.21.15.57:0/3238723155 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0b2c13e520 0x7f0b2c06bb60 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7f0b24005fd0 tx=0x7f0b2402c1b0 comp rx=0 tx=0).stop 2023-10-19T13:04:09.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.593+0000 7f0b32cd7640 1 -- 172.21.15.57:0/3238723155 shutdown_connections 2023-10-19T13:04:09.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.593+0000 7f0b32cd7640 1 --2- 172.21.15.57:0/3238723155 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0b2c141690 0x7f0b2c0703e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:09.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.593+0000 7f0b32cd7640 1 --2- 172.21.15.57:0/3238723155 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0b2c13e520 0x7f0b2c06bb60 unknown :-1 s=CLOSED pgs=1528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:09.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.593+0000 7f0b32cd7640 1 -- 172.21.15.57:0/3238723155 >> 172.21.15.57:0/3238723155 conn(0x7f0b2c069930 msgr2=0x7f0b2c069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:09.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.593+0000 7f0b32cd7640 1 -- 172.21.15.57:0/3238723155 shutdown_connections 2023-10-19T13:04:09.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.593+0000 7f0b32cd7640 1 -- 172.21.15.57:0/3238723155 wait complete. 2023-10-19T13:04:09.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.594+0000 7f0b32cd7640 1 Processor -- start 2023-10-19T13:04:09.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.594+0000 7f0b32cd7640 1 -- start start 2023-10-19T13:04:09.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.595+0000 7f0b32cd7640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0b2c06c0a0 0x7f0b2c1cf0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:09.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.595+0000 7f0b32cd7640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0b2c13e520 0x7f0b2c0abe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:09.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.595+0000 7f0b32cd7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0b2c141690 0x7f0b2c1eed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:09.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.595+0000 7f0b32cd7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b2c0ac370 con 0x7f0b2c141690 2023-10-19T13:04:09.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.595+0000 7f0b32cd7640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b2c1cf740 con 0x7f0b2c06c0a0 2023-10-19T13:04:09.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.595+0000 7f0b32cd7640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b2c1cfa20 con 0x7f0b2c13e520 2023-10-19T13:04:09.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.595+0000 7f0b31274640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0b2c141690 0x7f0b2c1eed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:09.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.595+0000 7f0b31274640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0b2c141690 0x7f0b2c1eed20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33748/0 (socket says 172.21.15.57:33748) 2023-10-19T13:04:09.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.595+0000 7f0b31274640 1 -- 172.21.15.57:0/3126338972 learned_addr learned my addr 172.21.15.57:0/3126338972 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:09.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.595+0000 7f0b2bfff640 1 --2- 172.21.15.57:0/3126338972 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0b2c13e520 0x7f0b2c0abe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:09.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.595+0000 7f0b31274640 1 -- 172.21.15.57:0/3126338972 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0b2c13e520 msgr2=0x7f0b2c0abe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:09.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.595+0000 7f0b31274640 1 --2- 172.21.15.57:0/3126338972 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0b2c13e520 0x7f0b2c0abe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:09.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.595+0000 7f0b31274640 1 -- 172.21.15.57:0/3126338972 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0b2c06c0a0 msgr2=0x7f0b2c1cf0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:09.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.595+0000 7f0b30a73640 1 --2- 172.21.15.57:0/3126338972 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0b2c06c0a0 0x7f0b2c1cf0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:09.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.595+0000 7f0b31274640 1 --2- 172.21.15.57:0/3126338972 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0b2c06c0a0 0x7f0b2c1cf0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:09.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.595+0000 7f0b31274640 1 -- 172.21.15.57:0/3126338972 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b24009900 con 0x7f0b2c141690 2023-10-19T13:04:09.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.596+0000 7f0b31274640 1 --2- 172.21.15.57:0/3126338972 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0b2c141690 0x7f0b2c1eed20 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7f0b1400c3c0 tx=0x7f0b1400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:09.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.596+0000 7f0b297fa640 1 -- 172.21.15.57:0/3126338972 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b140052d0 con 0x7f0b2c141690 2023-10-19T13:04:09.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.596+0000 7f0b297fa640 1 -- 172.21.15.57:0/3126338972 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f0b14007020 con 0x7f0b2c141690 2023-10-19T13:04:09.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.596+0000 7f0b32cd7640 1 -- 172.21.15.57:0/3126338972 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b2c1cfd00 con 0x7f0b2c141690 2023-10-19T13:04:09.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.596+0000 7f0b297fa640 1 -- 172.21.15.57:0/3126338972 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b14007430 con 0x7f0b2c141690 2023-10-19T13:04:09.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.596+0000 7f0b32cd7640 1 -- 172.21.15.57:0/3126338972 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b2c1cd100 con 0x7f0b2c141690 2023-10-19T13:04:09.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.597+0000 7f0b32cd7640 1 -- 172.21.15.57:0/3126338972 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b2c0690e0 con 0x7f0b2c141690 2023-10-19T13:04:09.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.597+0000 7f0b297fa640 1 -- 172.21.15.57:0/3126338972 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 67) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f0b140067d0 con 0x7f0b2c141690 2023-10-19T13:04:09.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.598+0000 7f0b297fa640 1 --2- 172.21.15.57:0/3126338972 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f0b0c06ac90 0x7f0b0c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:09.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.598+0000 7f0b297fa640 1 -- 172.21.15.57:0/3126338972 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 6887+0+0 (secure 0 0 0) 0x7f0b14088ca0 con 0x7f0b2c141690 2023-10-19T13:04:09.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.598+0000 7f0b30a73640 1 --2- 172.21.15.57:0/3126338972 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f0b0c06ac90 0x7f0b0c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:09.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.598+0000 7f0b30a73640 1 --2- 172.21.15.57:0/3126338972 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f0b0c06ac90 0x7f0b0c06d150 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f0b24036000 tx=0x7f0b24004ab0 comp rx=0 tx=0).ready entity=mgr.7960 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:09.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.602+0000 7f0b297fa640 1 -- 172.21.15.57:0/3126338972 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0b14053bf0 con 0x7f0b2c141690 2023-10-19T13:04:09.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.779+0000 7f0b32cd7640 1 -- 172.21.15.57:0/3126338972 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0b2c0ae0e0 con 0x7f0b2c141690 2023-10-19T13:04:09.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.780+0000 7f0b297fa640 1 -- 172.21.15.57:0/3126338972 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v132) v1 ==== 74+0+18024 (secure 0 0 0) 0x7f0b14057210 con 0x7f0b2c141690 2023-10-19T13:04:09.783 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:04:09.784 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":132,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T13:04:07.723992+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2023-10-19T13:02:55.916158+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"129","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2023-10-19T13:02:56.675559+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"132","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":132,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.179:0/865284344":"2023-10-20T13:04:07.723901+0000","172.21.15.179:6832/1210673354":"2023-10-20T13:04:07.723901+0000","172.21.15.179:0/4226732685":"2023-10-20T13:04:07.723901+0000","172.21.15.179:0/3662182511":"2023-10-20T13:04:07.723901+0000","172.21.15.179:0/2769057100":"2023-10-20T13:04:07.723901+0000","172.21.15.179:6833/1210673354":"2023-10-20T13:04:07.723901+0000","172.21.15.179:6833/4155586759":"2023-10-20T13:03:08.761549+0000","172.21.15.179:0/2669494882":"2023-10-20T13:04:07.723901+0000","172.21.15.179:0/1859890742":"2023-10-20T13:04:07.723901+0000","172.21.15.179:6832/4155586759":"2023-10-20T13:03:08.761549+0000","172.21.15.179:0/1341670560":"2023-10-20T13:03:08.761549+0000","172.21.15.179:0/33839263":"2023-10-20T13:03:08.761549+0000","172.21.15.179:0/2362202135":"2023-10-20T13:04:07.723901+0000","172.21.15.179:0/3689030544":"2023-10-20T13:03:08.761549+0000","172.21.15.179:0/3560759535":"2023-10-20T13:04:07.723901+0000","172.21.15.179:0/3037001824":"2023-10-20T13:03:08.761549+0000","172.21.15.179:0/1069074991":"2023-10-20T13:03:08.761549+0000","172.21.15.179:0/831719196":"2023-10-20T13:03:08.761549+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T13:04:09.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.783+0000 7f0b32cd7640 1 -- 172.21.15.57:0/3126338972 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f0b0c06ac90 msgr2=0x7f0b0c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:09.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.783+0000 7f0b32cd7640 1 --2- 172.21.15.57:0/3126338972 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f0b0c06ac90 0x7f0b0c06d150 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f0b24036000 tx=0x7f0b24004ab0 comp rx=0 tx=0).stop 2023-10-19T13:04:09.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.783+0000 7f0b32cd7640 1 -- 172.21.15.57:0/3126338972 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0b2c141690 msgr2=0x7f0b2c1eed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:09.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.783+0000 7f0b32cd7640 1 --2- 172.21.15.57:0/3126338972 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0b2c141690 0x7f0b2c1eed20 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7f0b1400c3c0 tx=0x7f0b1400c880 comp rx=0 tx=0).stop 2023-10-19T13:04:09.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.783+0000 7f0b32cd7640 1 -- 172.21.15.57:0/3126338972 shutdown_connections 2023-10-19T13:04:09.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.783+0000 7f0b32cd7640 1 --2- 172.21.15.57:0/3126338972 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f0b0c06ac90 0x7f0b0c06d150 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:09.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.783+0000 7f0b32cd7640 1 --2- 172.21.15.57:0/3126338972 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0b2c141690 0x7f0b2c1eed20 unknown :-1 s=CLOSED pgs=1529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:09.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.783+0000 7f0b32cd7640 1 --2- 172.21.15.57:0/3126338972 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0b2c13e520 0x7f0b2c0abe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:09.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.783+0000 7f0b32cd7640 1 --2- 172.21.15.57:0/3126338972 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0b2c06c0a0 0x7f0b2c1cf0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:09.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.783+0000 7f0b32cd7640 1 -- 172.21.15.57:0/3126338972 >> 172.21.15.57:0/3126338972 conn(0x7f0b2c069930 msgr2=0x7f0b2c0c9ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:09.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.783+0000 7f0b32cd7640 1 -- 172.21.15.57:0/3126338972 shutdown_connections 2023-10-19T13:04:09.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.783+0000 7f0b32cd7640 1 -- 172.21.15.57:0/3126338972 wait complete. 2023-10-19T13:04:09.797 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-19T13:04:09.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.904+0000 7f30804d4640 1 -- 172.21.15.57:0/2029843306 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f30640047c0 msgr2=0x7f3064004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:09.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.904+0000 7f30804d4640 1 --2- 172.21.15.57:0/2029843306 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f30640047c0 0x7f3064004c40 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7f3068006fc0 tx=0x7f30640057f0 comp rx=0 tx=0).stop 2023-10-19T13:04:09.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.904+0000 7f30804d4640 1 -- 172.21.15.57:0/2029843306 shutdown_connections 2023-10-19T13:04:09.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.904+0000 7f30804d4640 1 --2- 172.21.15.57:0/2029843306 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f30640047c0 0x7f3064004c40 unknown :-1 s=CLOSED pgs=1531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:09.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.904+0000 7f30804d4640 1 -- 172.21.15.57:0/2029843306 >> 172.21.15.57:0/2029843306 conn(0x7f3078069930 msgr2=0x7f3078069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:09.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.904+0000 7f30804d4640 1 -- 172.21.15.57:0/2029843306 shutdown_connections 2023-10-19T13:04:09.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.904+0000 7f30804d4640 1 -- 172.21.15.57:0/2029843306 wait complete. 2023-10-19T13:04:09.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.905+0000 7f30804d4640 1 Processor -- start 2023-10-19T13:04:09.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.906+0000 7f30804d4640 1 -- start start 2023-10-19T13:04:09.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.906+0000 7f30804d4640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f30640047c0 0x7f30781ef9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:09.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.906+0000 7f30804d4640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f30781deeb0 0x7f30780b0c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:09.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.906+0000 7f30804d4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f307806f080 0x7f307806f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:09.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.906+0000 7f30804d4640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30781e00a0 con 0x7f307806f080 2023-10-19T13:04:09.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.906+0000 7f30804d4640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f307806fa50 con 0x7f30781deeb0 2023-10-19T13:04:09.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.906+0000 7f30804d4640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30781df420 con 0x7f30640047c0 2023-10-19T13:04:09.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.906+0000 7f307ea71640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f307806f080 0x7f307806f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:09.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.906+0000 7f307ea71640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f307806f080 0x7f307806f510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33768/0 (socket says 172.21.15.57:33768) 2023-10-19T13:04:09.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.906+0000 7f307ea71640 1 -- 172.21.15.57:0/3558567052 learned_addr learned my addr 172.21.15.57:0/3558567052 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:09.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.907+0000 7f307da6f640 1 --2- 172.21.15.57:0/3558567052 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f30781deeb0 0x7f30780b0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:09.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.907+0000 7f307e270640 1 --2- 172.21.15.57:0/3558567052 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f30640047c0 0x7f30781ef9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:09.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.907+0000 7f307ea71640 1 -- 172.21.15.57:0/3558567052 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f30640047c0 msgr2=0x7f30781ef9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:09.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.907+0000 7f307ea71640 1 --2- 172.21.15.57:0/3558567052 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f30640047c0 0x7f30781ef9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:09.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.907+0000 7f307ea71640 1 -- 172.21.15.57:0/3558567052 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f30781deeb0 msgr2=0x7f30780b0c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:09.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.907+0000 7f307ea71640 1 --2- 172.21.15.57:0/3558567052 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f30781deeb0 0x7f30780b0c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:09.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.907+0000 7f307ea71640 1 -- 172.21.15.57:0/3558567052 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30781df700 con 0x7f307806f080 2023-10-19T13:04:09.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.907+0000 7f307ea71640 1 --2- 172.21.15.57:0/3558567052 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f307806f080 0x7f307806f510 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7f30780ac640 tx=0x7f307000d9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:09.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.907+0000 7f306effd640 1 -- 172.21.15.57:0/3558567052 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f307000f880 con 0x7f307806f080 2023-10-19T13:04:09.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.907+0000 7f306effd640 1 -- 172.21.15.57:0/3558567052 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f307000fad0 con 0x7f307806f080 2023-10-19T13:04:09.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.907+0000 7f30804d4640 1 -- 172.21.15.57:0/3558567052 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30781df9e0 con 0x7f307806f080 2023-10-19T13:04:09.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.907+0000 7f306effd640 1 -- 172.21.15.57:0/3558567052 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3070010450 con 0x7f307806f080 2023-10-19T13:04:09.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.907+0000 7f30804d4640 1 -- 172.21.15.57:0/3558567052 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30780b1150 con 0x7f307806f080 2023-10-19T13:04:09.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.908+0000 7f30804d4640 1 -- 172.21.15.57:0/3558567052 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30781ecaa0 con 0x7f307806f080 2023-10-19T13:04:09.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.908+0000 7f306effd640 1 -- 172.21.15.57:0/3558567052 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 67) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f30700105b0 con 0x7f307806f080 2023-10-19T13:04:09.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.909+0000 7f306effd640 1 --2- 172.21.15.57:0/3558567052 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f305c06ac90 0x7f305c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:09.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.909+0000 7f306effd640 1 -- 172.21.15.57:0/3558567052 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 6887+0+0 (secure 0 0 0) 0x7f3070089cd0 con 0x7f307806f080 2023-10-19T13:04:09.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.909+0000 7f307e270640 1 --2- 172.21.15.57:0/3558567052 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f305c06ac90 0x7f305c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:09.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.910+0000 7f307e270640 1 --2- 172.21.15.57:0/3558567052 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f305c06ac90 0x7f305c06d150 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3078069da0 tx=0x7f307400b260 comp rx=0 tx=0).ready entity=mgr.7960 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:09.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:09.913+0000 7f306effd640 1 -- 172.21.15.57:0/3558567052 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3070054ba0 con 0x7f307806f080 2023-10-19T13:04:10.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:10.127+0000 7f30804d4640 1 -- 172.21.15.57:0/3558567052 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f30780690e0 con 0x7f307806f080 2023-10-19T13:04:10.130 INFO:tasks.ceph.mon.a.smithi057.stderr:2023-10-19T13:04:10.128+0000 7f1253ff3640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-19T13:04:10.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:10.129+0000 7f306effd640 1 -- 172.21.15.57:0/3558567052 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v51) v1 ==== 161+0+0 (secure 0 0 0) 0x7f30700581c0 con 0x7f307806f080 2023-10-19T13:04:10.132 INFO:teuthology.orchestra.run.smithi057.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-19T13:04:10.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:10.132+0000 7f30804d4640 1 -- 172.21.15.57:0/3558567052 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f305c06ac90 msgr2=0x7f305c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:10.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:10.132+0000 7f30804d4640 1 --2- 172.21.15.57:0/3558567052 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f305c06ac90 0x7f305c06d150 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3078069da0 tx=0x7f307400b260 comp rx=0 tx=0).stop 2023-10-19T13:04:10.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:10.132+0000 7f30804d4640 1 -- 172.21.15.57:0/3558567052 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f307806f080 msgr2=0x7f307806f510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:10.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:10.132+0000 7f30804d4640 1 --2- 172.21.15.57:0/3558567052 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f307806f080 0x7f307806f510 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7f30780ac640 tx=0x7f307000d9f0 comp rx=0 tx=0).stop 2023-10-19T13:04:10.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:10.132+0000 7f30804d4640 1 -- 172.21.15.57:0/3558567052 shutdown_connections 2023-10-19T13:04:10.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:10.132+0000 7f30804d4640 1 --2- 172.21.15.57:0/3558567052 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f305c06ac90 0x7f305c06d150 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:10.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:10.132+0000 7f30804d4640 1 --2- 172.21.15.57:0/3558567052 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f307806f080 0x7f307806f510 unknown :-1 s=CLOSED pgs=1532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:10.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:10.132+0000 7f30804d4640 1 --2- 172.21.15.57:0/3558567052 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f30781deeb0 0x7f30780b0c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:10.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:10.132+0000 7f30804d4640 1 --2- 172.21.15.57:0/3558567052 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f30640047c0 0x7f30781ef9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:10.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:10.132+0000 7f30804d4640 1 -- 172.21.15.57:0/3558567052 >> 172.21.15.57:0/3558567052 conn(0x7f3078069930 msgr2=0x7f30780ac9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:10.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:10.132+0000 7f30804d4640 1 -- 172.21.15.57:0/3558567052 shutdown_connections 2023-10-19T13:04:10.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:10.132+0000 7f30804d4640 1 -- 172.21.15.57:0/3558567052 wait complete. 2023-10-19T13:04:10.145 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-19T13:04:10.172 INFO:tasks.ceph.mds.d.smithi179.stdout:starting mds.d at 2023-10-19T13:04:10.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:10.257+0000 7fe19d4d7640 1 -- 172.21.15.57:0/399208685 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe1980cf340 msgr2=0x7fe1980d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:10.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:10.257+0000 7fe19d4d7640 1 --2- 172.21.15.57:0/399208685 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe1980cf340 0x7fe1980d1910 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fe190005fd0 tx=0x7fe19002bd90 comp rx=0 tx=0).stop 2023-10-19T13:04:10.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:10.257+0000 7fe19d4d7640 1 -- 172.21.15.57:0/399208685 shutdown_connections 2023-10-19T13:04:10.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:10.257+0000 7fe19d4d7640 1 --2- 172.21.15.57:0/399208685 >> v2:172.21.15.179:3300/0 conn(0x7fe1980ceb40 0x7fe1980b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:10.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:10.257+0000 7fe19d4d7640 1 --2- 172.21.15.57:0/399208685 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe1980cf340 0x7fe1980d1910 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:10.262 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:10.257+0000 7fe19d4d7640 1 -- 172.21.15.57:0/399208685 >> 172.21.15.57:0/399208685 conn(0x7fe1980d0d60 msgr2=0x7fe1980d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:10.262 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:10.258+0000 7fe19d4d7640 1 -- 172.21.15.57:0/399208685 shutdown_connections 2023-10-19T13:04:10.262 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:10.258+0000 7fe19d4d7640 1 -- 172.21.15.57:0/399208685 wait complete. 2023-10-19T13:04:10.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:10.259+0000 7fe19d4d7640 1 Processor -- start 2023-10-19T13:04:10.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:10.259+0000 7fe19d4d7640 1 -- start start 2023-10-19T13:04:10.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:10.260+0000 7fe19d4d7640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe1980ceb40 0x7fe1980c6b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:10.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:10.260+0000 7fe19d4d7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1980cf340 0x7fe1980b6020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:10.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:10.260+0000 7fe19d4d7640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe19815b2a0 0x7fe198069f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:10.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:10.260+0000 7fe19d4d7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1980b45a0 con 0x7fe1980cf340 2023-10-19T13:04:10.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:10.260+0000 7fe19d4d7640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1980b4880 con 0x7fe19815b2a0 2023-10-19T13:04:10.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:10.260+0000 7fe19d4d7640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1980b5540 con 0x7fe1980ceb40 2023-10-19T13:04:10.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:10.260+0000 7fe1967fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1980cf340 0x7fe1980b6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:10.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:10.260+0000 7fe196ffd640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe1980ceb40 0x7fe1980c6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:10.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:10.260+0000 7fe196ffd640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe1980ceb40 0x7fe1980c6b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3301/0 says I am v2:172.21.15.57:45292/0 (socket says 172.21.15.57:45292) 2023-10-19T13:04:10.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:10.260+0000 7fe196ffd640 1 -- 172.21.15.57:0/3165403829 learned_addr learned my addr 172.21.15.57:0/3165403829 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:10.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:10.260+0000 7fe1977fe640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe19815b2a0 0x7fe198069f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:10.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:10.260+0000 7fe1967fc640 1 --2- 172.21.15.57:0/3165403829 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1980cf340 0x7fe1980b6020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33784/0 (socket says 172.21.15.57:33784) 2023-10-19T13:04:10.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:10.260+0000 7fe196ffd640 1 -- 172.21.15.57:0/3165403829 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe19815b2a0 msgr2=0x7fe198069f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:10.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:10.260+0000 7fe196ffd640 1 --2- 172.21.15.57:0/3165403829 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe19815b2a0 0x7fe198069f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:10.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:10.260+0000 7fe196ffd640 1 -- 172.21.15.57:0/3165403829 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1980cf340 msgr2=0x7fe1980b6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:10.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:10.260+0000 7fe196ffd640 1 --2- 172.21.15.57:0/3165403829 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1980cf340 0x7fe1980b6020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:10.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:10.260+0000 7fe196ffd640 1 -- 172.21.15.57:0/3165403829 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe190009560 con 0x7fe1980ceb40 2023-10-19T13:04:10.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:10.260+0000 7fe1967fc640 1 --2- 172.21.15.57:0/3165403829 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1980cf340 0x7fe1980b6020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T13:04:10.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:10.260+0000 7fe196ffd640 1 --2- 172.21.15.57:0/3165403829 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe1980ceb40 0x7fe1980c6b90 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fe190036040 tx=0x7fe19002d970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:10.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:10.260+0000 7fe18f7fe640 1 -- 172.21.15.57:0/3165403829 <== mon.2 v2:172.21.15.179:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe19002dd70 con 0x7fe1980ceb40 2023-10-19T13:04:10.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:10.260+0000 7fe18f7fe640 1 -- 172.21.15.57:0/3165403829 <== mon.2 v2:172.21.15.179:3301/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fe1900035f0 con 0x7fe1980ceb40 2023-10-19T13:04:10.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:10.260+0000 7fe19d4d7640 1 -- 172.21.15.57:0/3165403829 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1980b5820 con 0x7fe1980ceb40 2023-10-19T13:04:10.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:10.260+0000 7fe18f7fe640 1 -- 172.21.15.57:0/3165403829 <== mon.2 v2:172.21.15.179:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1900039e0 con 0x7fe1980ceb40 2023-10-19T13:04:10.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:10.261+0000 7fe19d4d7640 1 -- 172.21.15.57:0/3165403829 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1980c70d0 con 0x7fe1980ceb40 2023-10-19T13:04:10.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:10.261+0000 7fe19d4d7640 1 -- 172.21.15.57:0/3165403829 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1980c7260 con 0x7fe1980ceb40 2023-10-19T13:04:10.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:10.262+0000 7fe18f7fe640 1 -- 172.21.15.57:0/3165403829 <== mon.2 v2:172.21.15.179:3301/0 4 ==== mgrmap(e 67) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fe190003b40 con 0x7fe1980ceb40 2023-10-19T13:04:10.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:10.262+0000 7fe18f7fe640 1 --2- 172.21.15.57:0/3165403829 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7fe17806ac90 0x7fe17806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:10.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:10.262+0000 7fe18f7fe640 1 -- 172.21.15.57:0/3165403829 <== mon.2 v2:172.21.15.179:3301/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 6945+0+0 (secure 0 0 0) 0x7fe1900ac250 con 0x7fe1980ceb40 2023-10-19T13:04:10.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:10.263+0000 7fe1967fc640 1 --2- 172.21.15.57:0/3165403829 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7fe17806ac90 0x7fe17806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:10.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:10.263+0000 7fe1967fc640 1 --2- 172.21.15.57:0/3165403829 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7fe17806ac90 0x7fe17806d150 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe19809ec30 tx=0x7fe180009040 comp rx=0 tx=0).ready entity=mgr.7960 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:10.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:10.266+0000 7fe18f7fe640 1 -- 172.21.15.57:0/3165403829 <== mon.2 v2:172.21.15.179:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe1900773f0 con 0x7fe1980ceb40 2023-10-19T13:04:10.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:10.479+0000 7fe19d4d7640 1 -- 172.21.15.57:0/3165403829 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fe1980cb9b0 con 0x7fe1980ceb40 2023-10-19T13:04:10.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:10.751+0000 7fe18f7fe640 1 -- 172.21.15.57:0/3165403829 <== mon.2 v2:172.21.15.179:3301/0 7 ==== mgrmap(e 68) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fe1900777c0 con 0x7fe1980ceb40 2023-10-19T13:04:11.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:11.136+0000 7fe18f7fe640 1 -- 172.21.15.57:0/3165403829 <== mon.2 v2:172.21.15.179:3301/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v52) v1 ==== 104+0+0 (secure 0 0 0) 0x7fe1900925e0 con 0x7fe1980ceb40 2023-10-19T13:04:11.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:11.138+0000 7fe19d4d7640 1 -- 172.21.15.57:0/3165403829 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7fe17806ac90 msgr2=0x7fe17806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:11.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:11.138+0000 7fe19d4d7640 1 --2- 172.21.15.57:0/3165403829 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7fe17806ac90 0x7fe17806d150 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe19809ec30 tx=0x7fe180009040 comp rx=0 tx=0).stop 2023-10-19T13:04:11.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:11.138+0000 7fe19d4d7640 1 -- 172.21.15.57:0/3165403829 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe1980ceb40 msgr2=0x7fe1980c6b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:11.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:11.138+0000 7fe19d4d7640 1 --2- 172.21.15.57:0/3165403829 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe1980ceb40 0x7fe1980c6b90 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fe190036040 tx=0x7fe19002d970 comp rx=0 tx=0).stop 2023-10-19T13:04:11.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:11.138+0000 7fe19d4d7640 1 -- 172.21.15.57:0/3165403829 shutdown_connections 2023-10-19T13:04:11.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:11.138+0000 7fe19d4d7640 1 --2- 172.21.15.57:0/3165403829 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7fe17806ac90 0x7fe17806d150 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:11.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:11.138+0000 7fe19d4d7640 1 --2- 172.21.15.57:0/3165403829 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe19815b2a0 0x7fe198069f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:11.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:11.138+0000 7fe19d4d7640 1 --2- 172.21.15.57:0/3165403829 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1980cf340 0x7fe1980b6020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:11.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:11.138+0000 7fe19d4d7640 1 --2- 172.21.15.57:0/3165403829 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe1980ceb40 0x7fe1980c6b90 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:11.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:11.138+0000 7fe19d4d7640 1 -- 172.21.15.57:0/3165403829 >> 172.21.15.57:0/3165403829 conn(0x7fe1980d0d60 msgr2=0x7fe19806b490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:11.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:11.139+0000 7fe19d4d7640 1 -- 172.21.15.57:0/3165403829 shutdown_connections 2023-10-19T13:04:11.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:11.139+0000 7fe19d4d7640 1 -- 172.21.15.57:0/3165403829 wait complete. 2023-10-19T13:04:11.154 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-19T13:04:11.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:11.272+0000 7f5dc40b4640 1 -- 172.21.15.57:0/1460270912 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5dbc0cd9e0 msgr2=0x7f5dbc0cde50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:11.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:11.272+0000 7f5dc40b4640 1 --2- 172.21.15.57:0/1460270912 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5dbc0cd9e0 0x7f5dbc0cde50 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f5dbc072450 tx=0x7f5db402c260 comp rx=0 tx=0).stop 2023-10-19T13:04:11.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:11.272+0000 7f5dc40b4640 1 -- 172.21.15.57:0/1460270912 shutdown_connections 2023-10-19T13:04:11.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:11.272+0000 7f5dc40b4640 1 --2- 172.21.15.57:0/1460270912 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5dbc0cd9e0 0x7f5dbc0cde50 unknown :-1 s=CLOSED pgs=1534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:11.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:11.272+0000 7f5dc40b4640 1 --2- 172.21.15.57:0/1460270912 >> v2:172.21.15.179:3300/0 conn(0x7f5dbc158cf0 0x7f5dbc15c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:11.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:11.272+0000 7f5dc40b4640 1 -- 172.21.15.57:0/1460270912 >> 172.21.15.57:0/1460270912 conn(0x7f5dbc0af7a0 msgr2=0x7f5dbc0afb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:11.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:11.272+0000 7f5dc40b4640 1 -- 172.21.15.57:0/1460270912 shutdown_connections 2023-10-19T13:04:11.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:11.272+0000 7f5dc40b4640 1 -- 172.21.15.57:0/1460270912 wait complete. 2023-10-19T13:04:11.279 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:11.273+0000 7f5dc40b4640 1 Processor -- start 2023-10-19T13:04:11.280 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:11.273+0000 7f5dc40b4640 1 -- start start 2023-10-19T13:04:11.280 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:11.274+0000 7f5dc40b4640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5dbc0cd9e0 0x7f5dbc1f1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:11.280 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:11.274+0000 7f5dc40b4640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5dbc158910 0x7f5dbc14c110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:11.280 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:11.274+0000 7f5dc40b4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5dbc158cf0 0x7f5dbc1f4b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:11.280 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:11.274+0000 7f5dc40b4640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dbc1f2510 con 0x7f5dbc158cf0 2023-10-19T13:04:11.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:11.274+0000 7f5dc40b4640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dbc1f1b40 con 0x7f5dbc0cd9e0 2023-10-19T13:04:11.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:11.274+0000 7f5dc40b4640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dbc1f1500 con 0x7f5dbc158910 2023-10-19T13:04:11.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:11.274+0000 7f5dc2651640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5dbc158cf0 0x7f5dbc1f4b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:11.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:11.274+0000 7f5dc2651640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5dbc158cf0 0x7f5dbc1f4b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33812/0 (socket says 172.21.15.57:33812) 2023-10-19T13:04:11.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:11.274+0000 7f5dc2651640 1 -- 172.21.15.57:0/2476794070 learned_addr learned my addr 172.21.15.57:0/2476794070 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:11.282 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:11.274+0000 7f5dc1e50640 1 --2- 172.21.15.57:0/2476794070 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5dbc0cd9e0 0x7f5dbc1f1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:11.282 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:11.274+0000 7f5dc164f640 1 --2- 172.21.15.57:0/2476794070 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5dbc158910 0x7f5dbc14c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:11.282 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:11.274+0000 7f5dc2651640 1 -- 172.21.15.57:0/2476794070 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5dbc158910 msgr2=0x7f5dbc14c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:11.282 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:11.274+0000 7f5dc2651640 1 --2- 172.21.15.57:0/2476794070 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5dbc158910 0x7f5dbc14c110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:11.282 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:11.274+0000 7f5dc2651640 1 -- 172.21.15.57:0/2476794070 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5dbc0cd9e0 msgr2=0x7f5dbc1f1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:11.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:11.274+0000 7f5dc2651640 1 --2- 172.21.15.57:0/2476794070 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5dbc0cd9e0 0x7f5dbc1f1fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:11.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:11.274+0000 7f5dc2651640 1 -- 172.21.15.57:0/2476794070 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5db4009900 con 0x7f5dbc158cf0 2023-10-19T13:04:11.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:11.274+0000 7f5dc164f640 1 --2- 172.21.15.57:0/2476794070 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5dbc158910 0x7f5dbc14c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T13:04:11.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:11.274+0000 7f5dc2651640 1 --2- 172.21.15.57:0/2476794070 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5dbc158cf0 0x7f5dbc1f4b40 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7f5db4037040 tx=0x7f5db4003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:11.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:11.275+0000 7f5db27fc640 1 -- 172.21.15.57:0/2476794070 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5db4004250 con 0x7f5dbc158cf0 2023-10-19T13:04:11.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:11.275+0000 7f5db27fc640 1 -- 172.21.15.57:0/2476794070 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f5db40044a0 con 0x7f5dbc158cf0 2023-10-19T13:04:11.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:11.275+0000 7f5dc40b4640 1 -- 172.21.15.57:0/2476794070 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dbc14c650 con 0x7f5dbc158cf0 2023-10-19T13:04:11.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:11.275+0000 7f5db27fc640 1 -- 172.21.15.57:0/2476794070 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5db4004890 con 0x7f5dbc158cf0 2023-10-19T13:04:11.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:11.275+0000 7f5dc40b4640 1 -- 172.21.15.57:0/2476794070 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dbc1f58e0 con 0x7f5dbc158cf0 2023-10-19T13:04:11.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:11.275+0000 7f5dc40b4640 1 -- 172.21.15.57:0/2476794070 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dbc0690e0 con 0x7f5dbc158cf0 2023-10-19T13:04:11.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:11.276+0000 7f5db27fc640 1 -- 172.21.15.57:0/2476794070 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 68) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f5db403b040 con 0x7f5dbc158cf0 2023-10-19T13:04:11.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:11.277+0000 7f5db27fc640 1 --2- 172.21.15.57:0/2476794070 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f5da006af00 0x7f5da006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:11.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:11.277+0000 7f5db27fc640 1 -- 172.21.15.57:0/2476794070 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 6945+0+0 (secure 0 0 0) 0x7f5db40ad260 con 0x7f5dbc158cf0 2023-10-19T13:04:11.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:11.277+0000 7f5dc1e50640 1 --2- 172.21.15.57:0/2476794070 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f5da006af00 0x7f5da006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:11.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:11.277+0000 7f5dc1e50640 1 --2- 172.21.15.57:0/2476794070 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f5da006af00 0x7f5da006d3c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f5db8007fd0 tx=0x7f5db8009040 comp rx=0 tx=0).ready entity=mgr.7960 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:11.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:11.280+0000 7f5db27fc640 1 -- 172.21.15.57:0/2476794070 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f5db4078010 con 0x7f5dbc158cf0 2023-10-19T13:04:11.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:11.457+0000 7f5dc40b4640 1 -- 172.21.15.57:0/2476794070 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f5dbc1f1c80 con 0x7f5dbc158cf0 2023-10-19T13:04:12.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:12.361+0000 7f5db27fc640 1 -- 172.21.15.57:0/2476794070 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v134) v1 ==== 181+0+0 (secure 0 0 0) 0x7f5db407b630 con 0x7f5dbc158cf0 2023-10-19T13:04:12.364 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'cephfs_metadata' removed 2023-10-19T13:04:12.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:12.363+0000 7f5dc40b4640 1 -- 172.21.15.57:0/2476794070 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f5da006af00 msgr2=0x7f5da006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:12.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:12.363+0000 7f5dc40b4640 1 --2- 172.21.15.57:0/2476794070 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f5da006af00 0x7f5da006d3c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f5db8007fd0 tx=0x7f5db8009040 comp rx=0 tx=0).stop 2023-10-19T13:04:12.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:12.363+0000 7f5dc40b4640 1 -- 172.21.15.57:0/2476794070 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5dbc158cf0 msgr2=0x7f5dbc1f4b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:12.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:12.363+0000 7f5dc40b4640 1 --2- 172.21.15.57:0/2476794070 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5dbc158cf0 0x7f5dbc1f4b40 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7f5db4037040 tx=0x7f5db4003c10 comp rx=0 tx=0).stop 2023-10-19T13:04:12.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:12.363+0000 7f5dc40b4640 1 -- 172.21.15.57:0/2476794070 shutdown_connections 2023-10-19T13:04:12.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:12.363+0000 7f5dc40b4640 1 --2- 172.21.15.57:0/2476794070 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f5da006af00 0x7f5da006d3c0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:12.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:12.363+0000 7f5dc40b4640 1 --2- 172.21.15.57:0/2476794070 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5dbc158cf0 0x7f5dbc1f4b40 unknown :-1 s=CLOSED pgs=1535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:12.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:12.363+0000 7f5dc40b4640 1 --2- 172.21.15.57:0/2476794070 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5dbc158910 0x7f5dbc14c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:12.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:12.363+0000 7f5dc40b4640 1 --2- 172.21.15.57:0/2476794070 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5dbc0cd9e0 0x7f5dbc1f1fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:12.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:12.364+0000 7f5dc40b4640 1 -- 172.21.15.57:0/2476794070 >> 172.21.15.57:0/2476794070 conn(0x7f5dbc0af7a0 msgr2=0x7f5dbc15ab90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:12.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:12.364+0000 7f5dc40b4640 1 -- 172.21.15.57:0/2476794070 shutdown_connections 2023-10-19T13:04:12.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:12.364+0000 7f5dc40b4640 1 -- 172.21.15.57:0/2476794070 wait complete. 2023-10-19T13:04:12.384 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-19T13:04:12.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:12.500+0000 7fbd2afd7640 1 -- 172.21.15.57:0/783118169 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd2413e520 msgr2=0x7fbd2406bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:12.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:12.500+0000 7fbd2afd7640 1 --2- 172.21.15.57:0/783118169 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd2413e520 0x7fbd2406bb60 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7fbd1c005fd0 tx=0x7fbd1c02c1b0 comp rx=0 tx=0).stop 2023-10-19T13:04:12.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:12.500+0000 7fbd2afd7640 1 -- 172.21.15.57:0/783118169 shutdown_connections 2023-10-19T13:04:12.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:12.500+0000 7fbd2afd7640 1 --2- 172.21.15.57:0/783118169 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbd24070af0 0x7fbd240affe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:12.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:12.500+0000 7fbd2afd7640 1 --2- 172.21.15.57:0/783118169 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd2413e520 0x7fbd2406bb60 unknown :-1 s=CLOSED pgs=1536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:12.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:12.500+0000 7fbd2afd7640 1 -- 172.21.15.57:0/783118169 >> 172.21.15.57:0/783118169 conn(0x7fbd24069930 msgr2=0x7fbd24069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:12.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:12.500+0000 7fbd2afd7640 1 -- 172.21.15.57:0/783118169 shutdown_connections 2023-10-19T13:04:12.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:12.500+0000 7fbd2afd7640 1 -- 172.21.15.57:0/783118169 wait complete. 2023-10-19T13:04:12.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:12.501+0000 7fbd2afd7640 1 Processor -- start 2023-10-19T13:04:12.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:12.501+0000 7fbd2afd7640 1 -- start start 2023-10-19T13:04:12.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:12.502+0000 7fbd2afd7640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbd24070af0 0x7fbd240ad9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:12.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:12.502+0000 7fbd2afd7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd2413e520 0x7fbd2413adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:12.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:12.502+0000 7fbd2afd7640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbd24141690 0x7fbd240ca870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:12.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:12.502+0000 7fbd2afd7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd240adef0 con 0x7fbd2413e520 2023-10-19T13:04:12.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:12.502+0000 7fbd2afd7640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd240a90f0 con 0x7fbd24070af0 2023-10-19T13:04:12.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:12.502+0000 7fbd2afd7640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd240a9390 con 0x7fbd24141690 2023-10-19T13:04:12.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:12.502+0000 7fbd23fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd2413e520 0x7fbd2413adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:12.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:12.502+0000 7fbd23fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd2413e520 0x7fbd2413adf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33830/0 (socket says 172.21.15.57:33830) 2023-10-19T13:04:12.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:12.502+0000 7fbd23fff640 1 -- 172.21.15.57:0/1917313868 learned_addr learned my addr 172.21.15.57:0/1917313868 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:12.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:12.502+0000 7fbd29574640 1 --2- 172.21.15.57:0/1917313868 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbd24141690 0x7fbd240ca870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:12.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:12.502+0000 7fbd28d73640 1 --2- 172.21.15.57:0/1917313868 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbd24070af0 0x7fbd240ad9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:12.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:12.503+0000 7fbd23fff640 1 -- 172.21.15.57:0/1917313868 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbd24141690 msgr2=0x7fbd240ca870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:12.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:12.503+0000 7fbd23fff640 1 --2- 172.21.15.57:0/1917313868 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbd24141690 0x7fbd240ca870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:12.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:12.503+0000 7fbd23fff640 1 -- 172.21.15.57:0/1917313868 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbd24070af0 msgr2=0x7fbd240ad9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:12.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:12.503+0000 7fbd23fff640 1 --2- 172.21.15.57:0/1917313868 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbd24070af0 0x7fbd240ad9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:12.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:12.503+0000 7fbd23fff640 1 -- 172.21.15.57:0/1917313868 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd1c009900 con 0x7fbd2413e520 2023-10-19T13:04:12.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:12.503+0000 7fbd28d73640 1 --2- 172.21.15.57:0/1917313868 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbd24070af0 0x7fbd240ad9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T13:04:12.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:12.503+0000 7fbd23fff640 1 --2- 172.21.15.57:0/1917313868 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd2413e520 0x7fbd2413adf0 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7fbd0c00b3f0 tx=0x7fbd0c00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:12.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:12.503+0000 7fbd217fa640 1 -- 172.21.15.57:0/1917313868 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd0c010a90 con 0x7fbd2413e520 2023-10-19T13:04:12.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:12.503+0000 7fbd217fa640 1 -- 172.21.15.57:0/1917313868 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fbd0c010ce0 con 0x7fbd2413e520 2023-10-19T13:04:12.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:12.503+0000 7fbd2afd7640 1 -- 172.21.15.57:0/1917313868 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd240a9670 con 0x7fbd2413e520 2023-10-19T13:04:12.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:12.503+0000 7fbd217fa640 1 -- 172.21.15.57:0/1917313868 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd0c0186a0 con 0x7fbd2413e520 2023-10-19T13:04:12.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:12.503+0000 7fbd2afd7640 1 -- 172.21.15.57:0/1917313868 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd240a64b0 con 0x7fbd2413e520 2023-10-19T13:04:12.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:12.504+0000 7fbd2afd7640 1 -- 172.21.15.57:0/1917313868 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd240a66d0 con 0x7fbd2413e520 2023-10-19T13:04:12.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:12.504+0000 7fbd217fa640 1 -- 172.21.15.57:0/1917313868 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 69) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fbd0c0213b0 con 0x7fbd2413e520 2023-10-19T13:04:12.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:12.505+0000 7fbd217fa640 1 --2- 172.21.15.57:0/1917313868 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7fbd0406af60 0x7fbd0406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:12.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:12.505+0000 7fbd217fa640 1 -- 172.21.15.57:0/1917313868 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 6507+0+0 (secure 0 0 0) 0x7fbd0c08d890 con 0x7fbd2413e520 2023-10-19T13:04:12.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:12.505+0000 7fbd28d73640 1 --2- 172.21.15.57:0/1917313868 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7fbd0406af60 0x7fbd0406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:12.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:12.506+0000 7fbd28d73640 1 --2- 172.21.15.57:0/1917313868 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7fbd0406af60 0x7fbd0406d420 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fbd1c036000 tx=0x7fbd1c004ab0 comp rx=0 tx=0).ready entity=mgr.7960 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:12.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:12.509+0000 7fbd217fa640 1 -- 172.21.15.57:0/1917313868 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fbd0c058480 con 0x7fbd2413e520 2023-10-19T13:04:12.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:12.686+0000 7fbd2afd7640 1 -- 172.21.15.57:0/1917313868 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fbd240690e0 con 0x7fbd2413e520 2023-10-19T13:04:13.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.223+0000 7fbd217fa640 1 -- 172.21.15.57:0/1917313868 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v135) v1 ==== 169+0+0 (secure 0 0 0) 0x7fbd0c01d0d0 con 0x7fbd2413e520 2023-10-19T13:04:13.226 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'cephfs_data' removed 2023-10-19T13:04:13.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.227+0000 7fbd2afd7640 1 -- 172.21.15.57:0/1917313868 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7fbd0406af60 msgr2=0x7fbd0406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:13.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.227+0000 7fbd2afd7640 1 --2- 172.21.15.57:0/1917313868 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7fbd0406af60 0x7fbd0406d420 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fbd1c036000 tx=0x7fbd1c004ab0 comp rx=0 tx=0).stop 2023-10-19T13:04:13.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.227+0000 7fbd2afd7640 1 -- 172.21.15.57:0/1917313868 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd2413e520 msgr2=0x7fbd2413adf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:13.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.227+0000 7fbd2afd7640 1 --2- 172.21.15.57:0/1917313868 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd2413e520 0x7fbd2413adf0 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7fbd0c00b3f0 tx=0x7fbd0c00b8b0 comp rx=0 tx=0).stop 2023-10-19T13:04:13.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.230+0000 7fbd2afd7640 1 -- 172.21.15.57:0/1917313868 shutdown_connections 2023-10-19T13:04:13.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.230+0000 7fbd2afd7640 1 --2- 172.21.15.57:0/1917313868 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7fbd0406af60 0x7fbd0406d420 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:13.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.230+0000 7fbd2afd7640 1 --2- 172.21.15.57:0/1917313868 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbd24141690 0x7fbd240ca870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:13.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.230+0000 7fbd2afd7640 1 --2- 172.21.15.57:0/1917313868 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd2413e520 0x7fbd2413adf0 unknown :-1 s=CLOSED pgs=1537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:13.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.230+0000 7fbd2afd7640 1 --2- 172.21.15.57:0/1917313868 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbd24070af0 0x7fbd240ad9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:13.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.230+0000 7fbd2afd7640 1 -- 172.21.15.57:0/1917313868 >> 172.21.15.57:0/1917313868 conn(0x7fbd24069930 msgr2=0x7fbd24140380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:13.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.230+0000 7fbd2afd7640 1 -- 172.21.15.57:0/1917313868 shutdown_connections 2023-10-19T13:04:13.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.230+0000 7fbd2afd7640 1 -- 172.21.15.57:0/1917313868 wait complete. 2023-10-19T13:04:13.252 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 30 2023-10-19T13:04:13.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.370+0000 7f108bed7640 1 -- 172.21.15.57:0/2005598007 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f108415b2a0 msgr2=0x7f10840d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:13.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.370+0000 7f108bed7640 1 --2- 172.21.15.57:0/2005598007 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f108415b2a0 0x7f10840d1910 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7f1078005fd0 tx=0x7f107802bd90 comp rx=0 tx=0).stop 2023-10-19T13:04:13.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.370+0000 7f108bed7640 1 -- 172.21.15.57:0/2005598007 shutdown_connections 2023-10-19T13:04:13.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.370+0000 7f108bed7640 1 --2- 172.21.15.57:0/2005598007 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f108415b2a0 0x7f10840d1910 unknown :-1 s=CLOSED pgs=1539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:13.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.370+0000 7f108bed7640 1 -- 172.21.15.57:0/2005598007 >> 172.21.15.57:0/2005598007 conn(0x7f10840d0d60 msgr2=0x7f10840d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:13.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.370+0000 7f108bed7640 1 -- 172.21.15.57:0/2005598007 shutdown_connections 2023-10-19T13:04:13.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.370+0000 7f108bed7640 1 -- 172.21.15.57:0/2005598007 wait complete. 2023-10-19T13:04:13.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.371+0000 7f108bed7640 1 Processor -- start 2023-10-19T13:04:13.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.371+0000 7f108bed7640 1 -- start start 2023-10-19T13:04:13.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.372+0000 7f108bed7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f10840ceb40 0x7f1084155b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:13.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.372+0000 7f108bed7640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f10840cf340 0x7f10840b6510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:13.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.372+0000 7f108bed7640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f108415b2a0 0x7f108406afe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:13.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.372+0000 7f108bed7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1084156480 con 0x7f10840ceb40 2023-10-19T13:04:13.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.372+0000 7f108bed7640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1084156760 con 0x7f10840cf340 2023-10-19T13:04:13.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.372+0000 7f108bed7640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1084156a40 con 0x7f108415b2a0 2023-10-19T13:04:13.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.372+0000 7f1089c73640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f10840ceb40 0x7f1084155b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:13.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.372+0000 7f1089c73640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f10840ceb40 0x7f1084155b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33838/0 (socket says 172.21.15.57:33838) 2023-10-19T13:04:13.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.372+0000 7f1089c73640 1 -- 172.21.15.57:0/1196650626 learned_addr learned my addr 172.21.15.57:0/1196650626 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:13.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.372+0000 7f1089472640 1 --2- 172.21.15.57:0/1196650626 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f10840cf340 0x7f10840b6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:13.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.372+0000 7f108a474640 1 --2- 172.21.15.57:0/1196650626 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f108415b2a0 0x7f108406afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:13.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.372+0000 7f1089c73640 1 -- 172.21.15.57:0/1196650626 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f108415b2a0 msgr2=0x7f108406afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:13.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.372+0000 7f1089c73640 1 --2- 172.21.15.57:0/1196650626 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f108415b2a0 0x7f108406afe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:13.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.372+0000 7f1089c73640 1 -- 172.21.15.57:0/1196650626 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f10840cf340 msgr2=0x7f10840b6510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:13.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.372+0000 7f1089c73640 1 --2- 172.21.15.57:0/1196650626 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f10840cf340 0x7f10840b6510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:13.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.372+0000 7f1089c73640 1 -- 172.21.15.57:0/1196650626 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1078009560 con 0x7f10840ceb40 2023-10-19T13:04:13.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.372+0000 7f108a474640 1 --2- 172.21.15.57:0/1196650626 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f108415b2a0 0x7f108406afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:04:13.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.373+0000 7f1089c73640 1 --2- 172.21.15.57:0/1196650626 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f10840ceb40 0x7f1084155b90 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7f107400c7c0 tx=0x7f107400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:13.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.373+0000 7f10827fc640 1 -- 172.21.15.57:0/1196650626 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f107400aa50 con 0x7f10840ceb40 2023-10-19T13:04:13.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.373+0000 7f10827fc640 1 -- 172.21.15.57:0/1196650626 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f107400aca0 con 0x7f10840ceb40 2023-10-19T13:04:13.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.373+0000 7f108bed7640 1 -- 172.21.15.57:0/1196650626 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1084156d20 con 0x7f10840ceb40 2023-10-19T13:04:13.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.373+0000 7f10827fc640 1 -- 172.21.15.57:0/1196650626 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1074012680 con 0x7f10840ceb40 2023-10-19T13:04:13.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.373+0000 7f108bed7640 1 -- 172.21.15.57:0/1196650626 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10841560d0 con 0x7f10840ceb40 2023-10-19T13:04:13.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.374+0000 7f108bed7640 1 -- 172.21.15.57:0/1196650626 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10840690e0 con 0x7f10840ceb40 2023-10-19T13:04:13.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.374+0000 7f10827fc640 1 -- 172.21.15.57:0/1196650626 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 69) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f107401a3b0 con 0x7f10840ceb40 2023-10-19T13:04:13.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.375+0000 7f10827fc640 1 --2- 172.21.15.57:0/1196650626 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f107006af90 0x7f107006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:13.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.375+0000 7f10827fc640 1 -- 172.21.15.57:0/1196650626 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 6093+0+0 (secure 0 0 0) 0x7f1074089dc0 con 0x7f10840ceb40 2023-10-19T13:04:13.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.375+0000 7f1089472640 1 --2- 172.21.15.57:0/1196650626 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f107006af90 0x7f107006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:13.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.375+0000 7f1089472640 1 --2- 172.21.15.57:0/1196650626 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f107006af90 0x7f107006d450 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f1078035000 tx=0x7f1078004730 comp rx=0 tx=0).ready entity=mgr.7960 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:13.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.379+0000 7f10827fc640 1 -- 172.21.15.57:0/1196650626 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f1074054b60 con 0x7f10840ceb40 2023-10-19T13:04:13.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.558+0000 7f108bed7640 1 -- 172.21.15.57:0/1196650626 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "config reset", "num": 30} v 0) v1 -- 0x7f10840d0b40 con 0x7f10840ceb40 2023-10-19T13:04:13.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.566+0000 7f10827fc640 1 -- 172.21.15.57:0/1196650626 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 30}]=0 v54) v1 ==== 71+0+0 (secure 0 0 0) 0x7f1074058180 con 0x7f10840ceb40 2023-10-19T13:04:13.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.568+0000 7f108bed7640 1 -- 172.21.15.57:0/1196650626 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f107006af90 msgr2=0x7f107006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:13.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.568+0000 7f108bed7640 1 --2- 172.21.15.57:0/1196650626 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f107006af90 0x7f107006d450 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f1078035000 tx=0x7f1078004730 comp rx=0 tx=0).stop 2023-10-19T13:04:13.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.568+0000 7f108bed7640 1 -- 172.21.15.57:0/1196650626 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f10840ceb40 msgr2=0x7f1084155b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:13.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.568+0000 7f108bed7640 1 --2- 172.21.15.57:0/1196650626 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f10840ceb40 0x7f1084155b90 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7f107400c7c0 tx=0x7f107400cc80 comp rx=0 tx=0).stop 2023-10-19T13:04:13.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.568+0000 7f108bed7640 1 -- 172.21.15.57:0/1196650626 shutdown_connections 2023-10-19T13:04:13.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.568+0000 7f108bed7640 1 --2- 172.21.15.57:0/1196650626 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f107006af90 0x7f107006d450 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:13.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.568+0000 7f108bed7640 1 --2- 172.21.15.57:0/1196650626 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f108415b2a0 0x7f108406afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:13.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.568+0000 7f108bed7640 1 --2- 172.21.15.57:0/1196650626 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f10840cf340 0x7f10840b6510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:13.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.568+0000 7f108bed7640 1 --2- 172.21.15.57:0/1196650626 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f10840ceb40 0x7f1084155b90 unknown :-1 s=CLOSED pgs=1540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:13.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.568+0000 7f108bed7640 1 -- 172.21.15.57:0/1196650626 >> 172.21.15.57:0/1196650626 conn(0x7f10840d0d60 msgr2=0x7f10840b0ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:13.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.569+0000 7f108bed7640 1 -- 172.21.15.57:0/1196650626 shutdown_connections 2023-10-19T13:04:13.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.569+0000 7f108bed7640 1 -- 172.21.15.57:0/1196650626 wait complete. 2023-10-19T13:04:13.582 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule' 2023-10-19T13:04:13.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.694+0000 7f87b7294640 1 -- 172.21.15.57:0/3181916592 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f87b00b1920 msgr2=0x7f87b006fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:13.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.694+0000 7f87b7294640 1 --2- 172.21.15.57:0/3181916592 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f87b00b1920 0x7f87b006fa70 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f87a8009830 tx=0x7f87a802bd60 comp rx=0 tx=0).stop 2023-10-19T13:04:13.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.694+0000 7f87b7294640 1 -- 172.21.15.57:0/3181916592 shutdown_connections 2023-10-19T13:04:13.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.694+0000 7f87b7294640 1 --2- 172.21.15.57:0/3181916592 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f87b00c9dc0 0x7f87b0154af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:13.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.694+0000 7f87b7294640 1 --2- 172.21.15.57:0/3181916592 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f87b00b1920 0x7f87b006fa70 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:13.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.694+0000 7f87b7294640 1 -- 172.21.15.57:0/3181916592 >> 172.21.15.57:0/3181916592 conn(0x7f87b0069930 msgr2=0x7f87b0069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:13.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.694+0000 7f87b7294640 1 -- 172.21.15.57:0/3181916592 shutdown_connections 2023-10-19T13:04:13.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.694+0000 7f87b7294640 1 -- 172.21.15.57:0/3181916592 wait complete. 2023-10-19T13:04:13.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.696+0000 7f87b7294640 1 Processor -- start 2023-10-19T13:04:13.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.696+0000 7f87b7294640 1 -- start start 2023-10-19T13:04:13.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.696+0000 7f87b7294640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f87b006ffb0 0x7f87b01f57d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:13.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.696+0000 7f87b7294640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f87b00b1920 0x7f87b00ad680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:13.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.696+0000 7f87b7294640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f87b00c9dc0 0x7f87b00b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:13.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.696+0000 7f87b7294640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87b01f5db0 con 0x7f87b00b1920 2023-10-19T13:04:13.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.696+0000 7f87b7294640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87b0156060 con 0x7f87b00c9dc0 2023-10-19T13:04:13.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.696+0000 7f87b7294640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87b01556c0 con 0x7f87b006ffb0 2023-10-19T13:04:13.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.696+0000 7f87b482f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f87b00b1920 0x7f87b00ad680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:13.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.696+0000 7f87b482f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f87b00b1920 0x7f87b00ad680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33840/0 (socket says 172.21.15.57:33840) 2023-10-19T13:04:13.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.696+0000 7f87b482f640 1 -- 172.21.15.57:0/2544062562 learned_addr learned my addr 172.21.15.57:0/2544062562 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:13.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.696+0000 7f87b5831640 1 --2- 172.21.15.57:0/2544062562 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f87b00c9dc0 0x7f87b00b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:13.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.697+0000 7f87b5030640 1 --2- 172.21.15.57:0/2544062562 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f87b006ffb0 0x7f87b01f57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:13.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.697+0000 7f87b482f640 1 -- 172.21.15.57:0/2544062562 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f87b006ffb0 msgr2=0x7f87b01f57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:13.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.697+0000 7f87b482f640 1 --2- 172.21.15.57:0/2544062562 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f87b006ffb0 0x7f87b01f57d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:13.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.697+0000 7f87b482f640 1 -- 172.21.15.57:0/2544062562 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f87b00c9dc0 msgr2=0x7f87b00b03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:13.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.697+0000 7f87b482f640 1 --2- 172.21.15.57:0/2544062562 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f87b00c9dc0 0x7f87b00b03e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:13.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.697+0000 7f87b482f640 1 -- 172.21.15.57:0/2544062562 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87a0009840 con 0x7f87b00b1920 2023-10-19T13:04:13.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.697+0000 7f87b5831640 1 --2- 172.21.15.57:0/2544062562 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f87b00c9dc0 0x7f87b00b03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T13:04:13.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.697+0000 7f87b482f640 1 --2- 172.21.15.57:0/2544062562 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f87b00b1920 0x7f87b00ad680 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7f87a400c440 tx=0x7f87a4001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:13.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.697+0000 7f87adffb640 1 -- 172.21.15.57:0/2544062562 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87a4004230 con 0x7f87b00b1920 2023-10-19T13:04:13.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.697+0000 7f87b7294640 1 -- 172.21.15.57:0/2544062562 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87a8009490 con 0x7f87b00b1920 2023-10-19T13:04:13.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.697+0000 7f87adffb640 1 -- 172.21.15.57:0/2544062562 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f87a4004480 con 0x7f87b00b1920 2023-10-19T13:04:13.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.697+0000 7f87adffb640 1 -- 172.21.15.57:0/2544062562 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87a4003850 con 0x7f87b00b1920 2023-10-19T13:04:13.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.697+0000 7f87b7294640 1 -- 172.21.15.57:0/2544062562 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87b0155c80 con 0x7f87b00b1920 2023-10-19T13:04:13.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.698+0000 7f87b7294640 1 -- 172.21.15.57:0/2544062562 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87b01561a0 con 0x7f87b00b1920 2023-10-19T13:04:13.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.699+0000 7f87adffb640 1 -- 172.21.15.57:0/2544062562 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 69) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f87a40039b0 con 0x7f87b00b1920 2023-10-19T13:04:13.707 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.699+0000 7f87adffb640 1 --2- 172.21.15.57:0/2544062562 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f879c06af60 0x7f879c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:13.707 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.699+0000 7f87adffb640 1 -- 172.21.15.57:0/2544062562 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 6093+0+0 (secure 0 0 0) 0x7f87a4088fe0 con 0x7f87b00b1920 2023-10-19T13:04:13.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.699+0000 7f87b5030640 1 --2- 172.21.15.57:0/2544062562 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f879c06af60 0x7f879c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:13.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.700+0000 7f87b5030640 1 --2- 172.21.15.57:0/2544062562 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f879c06af60 0x7f879c06d420 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f87a8036000 tx=0x7f87a8004730 comp rx=0 tx=0).ready entity=mgr.7960 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:13.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.703+0000 7f87adffb640 1 -- 172.21.15.57:0/2544062562 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f87b01561a0 con 0x7f87b00b1920 2023-10-19T13:04:13.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:13.947+0000 7f87b7294640 1 -- 172.21.15.57:0/2544062562 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule"]} v 0) v1 -- 0x7f87b01561a0 con 0x7f87b00b1920 2023-10-19T13:04:14.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.364+0000 7f87adffb640 1 -- 172.21.15.57:0/2544062562 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule"]}]=0 v757) v1 ==== 148+0+0 (secure 0 0 0) 0x7f87b01561a0 con 0x7f87b00b1920 2023-10-19T13:04:14.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.367+0000 7f87b7294640 1 -- 172.21.15.57:0/2544062562 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f879c06af60 msgr2=0x7f879c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:14.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.367+0000 7f87b7294640 1 --2- 172.21.15.57:0/2544062562 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f879c06af60 0x7f879c06d420 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f87a8036000 tx=0x7f87a8004730 comp rx=0 tx=0).stop 2023-10-19T13:04:14.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.367+0000 7f87b7294640 1 -- 172.21.15.57:0/2544062562 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f87b00b1920 msgr2=0x7f87b00ad680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:14.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.367+0000 7f87b7294640 1 --2- 172.21.15.57:0/2544062562 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f87b00b1920 0x7f87b00ad680 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7f87a400c440 tx=0x7f87a4001e70 comp rx=0 tx=0).stop 2023-10-19T13:04:14.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.367+0000 7f87b7294640 1 -- 172.21.15.57:0/2544062562 shutdown_connections 2023-10-19T13:04:14.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.367+0000 7f87b7294640 1 --2- 172.21.15.57:0/2544062562 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f879c06af60 0x7f879c06d420 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:14.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.367+0000 7f87b7294640 1 --2- 172.21.15.57:0/2544062562 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f87b00c9dc0 0x7f87b00b03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:14.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.367+0000 7f87b7294640 1 --2- 172.21.15.57:0/2544062562 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f87b00b1920 0x7f87b00ad680 unknown :-1 s=CLOSED pgs=1541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:14.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.367+0000 7f87b7294640 1 --2- 172.21.15.57:0/2544062562 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f87b006ffb0 0x7f87b01f57d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:14.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.367+0000 7f87b7294640 1 -- 172.21.15.57:0/2544062562 >> 172.21.15.57:0/2544062562 conn(0x7f87b0069930 msgr2=0x7f87b00aef70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:14.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.367+0000 7f87b7294640 1 -- 172.21.15.57:0/2544062562 shutdown_connections 2023-10-19T13:04:14.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.367+0000 7f87b7294640 1 -- 172.21.15.57:0/2544062562 wait complete. 2023-10-19T13:04:14.382 INFO:tasks.cephfs_test_runner:Test existence of a scheduled snapshot ... ok 2023-10-19T13:04:14.382 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_all_periods (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-10-19T13:04:14.383 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T13:04:14.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.493+0000 7f3a60444640 1 -- 172.21.15.57:0/1580966639 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3a440039a0 msgr2=0x7f3a44003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:14.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.493+0000 7f3a60444640 1 --2- 172.21.15.57:0/1580966639 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3a440039a0 0x7f3a44003dc0 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7f3a54009b80 tx=0x7f3a440057f0 comp rx=0 tx=0).stop 2023-10-19T13:04:14.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.493+0000 7f3a5e9e1640 1 -- 172.21.15.57:0/1580966639 reap_dead start 2023-10-19T13:04:14.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.494+0000 7f3a60444640 1 -- 172.21.15.57:0/1580966639 shutdown_connections 2023-10-19T13:04:14.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.494+0000 7f3a60444640 1 -- 172.21.15.57:0/1580966639 >> 172.21.15.57:0/1580966639 conn(0x7f3a58069930 msgr2=0x7f3a58069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:14.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.494+0000 7f3a60444640 1 -- 172.21.15.57:0/1580966639 shutdown_connections 2023-10-19T13:04:14.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.494+0000 7f3a60444640 1 -- 172.21.15.57:0/1580966639 wait complete. 2023-10-19T13:04:14.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.495+0000 7f3a60444640 1 Processor -- start 2023-10-19T13:04:14.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.495+0000 7f3a60444640 1 -- start start 2023-10-19T13:04:14.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.495+0000 7f3a60444640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3a440039a0 0x7f3a5813aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:14.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.495+0000 7f3a60444640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3a581e3010 0x7f3a580ad0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:14.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.495+0000 7f3a60444640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3a581e3470 0x7f3a580ad5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:14.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.495+0000 7f3a60444640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a581f1120 con 0x7f3a581e3470 2023-10-19T13:04:14.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.495+0000 7f3a60444640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a5813b530 con 0x7f3a581e3010 2023-10-19T13:04:14.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.495+0000 7f3a60444640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a58070590 con 0x7f3a440039a0 2023-10-19T13:04:14.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.496+0000 7f3a5e9e1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3a581e3470 0x7f3a580ad5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:14.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.496+0000 7f3a5e9e1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3a581e3470 0x7f3a580ad5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33870/0 (socket says 172.21.15.57:33870) 2023-10-19T13:04:14.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.496+0000 7f3a5e9e1640 1 -- 172.21.15.57:0/2378470973 learned_addr learned my addr 172.21.15.57:0/2378470973 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:14.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.496+0000 7f3a5e1e0640 1 --2- 172.21.15.57:0/2378470973 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3a440039a0 0x7f3a5813aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:14.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.496+0000 7f3a5d9df640 1 --2- 172.21.15.57:0/2378470973 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3a581e3010 0x7f3a580ad0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:14.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.496+0000 7f3a5e9e1640 1 -- 172.21.15.57:0/2378470973 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3a440039a0 msgr2=0x7f3a5813aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:14.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.496+0000 7f3a5e9e1640 1 --2- 172.21.15.57:0/2378470973 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3a440039a0 0x7f3a5813aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:14.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.496+0000 7f3a5e9e1640 1 -- 172.21.15.57:0/2378470973 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3a581e3010 msgr2=0x7f3a580ad0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:14.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.496+0000 7f3a5e9e1640 1 --2- 172.21.15.57:0/2378470973 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3a581e3010 0x7f3a580ad0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:14.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.496+0000 7f3a5e9e1640 1 -- 172.21.15.57:0/2378470973 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a58070870 con 0x7f3a581e3470 2023-10-19T13:04:14.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.496+0000 7f3a5e9e1640 1 --2- 172.21.15.57:0/2378470973 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3a581e3470 0x7f3a580ad5e0 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7f3a5806d560 tx=0x7f3a5000f3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:14.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.496+0000 7f3a4effd640 1 -- 172.21.15.57:0/2378470973 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a50004b20 con 0x7f3a581e3470 2023-10-19T13:04:14.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.496+0000 7f3a4effd640 1 -- 172.21.15.57:0/2378470973 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f3a50004c80 con 0x7f3a581e3470 2023-10-19T13:04:14.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.496+0000 7f3a4effd640 1 -- 172.21.15.57:0/2378470973 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a50003bd0 con 0x7f3a581e3470 2023-10-19T13:04:14.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.496+0000 7f3a60444640 1 -- 172.21.15.57:0/2378470973 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a58070b50 con 0x7f3a581e3470 2023-10-19T13:04:14.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.496+0000 7f3a60444640 1 -- 172.21.15.57:0/2378470973 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a5806f080 con 0x7f3a581e3470 2023-10-19T13:04:14.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.497+0000 7f3a60444640 1 -- 172.21.15.57:0/2378470973 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a58071070 con 0x7f3a581e3470 2023-10-19T13:04:14.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.498+0000 7f3a4effd640 1 -- 172.21.15.57:0/2378470973 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 69) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f3a50003d30 con 0x7f3a581e3470 2023-10-19T13:04:14.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.498+0000 7f3a4effd640 1 --2- 172.21.15.57:0/2378470973 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f3a3c06af60 0x7f3a3c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:14.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.498+0000 7f3a4effd640 1 -- 172.21.15.57:0/2378470973 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 6093+0+0 (secure 0 0 0) 0x7f3a5008a160 con 0x7f3a581e3470 2023-10-19T13:04:14.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.499+0000 7f3a5e1e0640 1 --2- 172.21.15.57:0/2378470973 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f3a3c06af60 0x7f3a3c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:14.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.499+0000 7f3a5e1e0640 1 --2- 172.21.15.57:0/2378470973 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f3a3c06af60 0x7f3a3c06d420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f3a54036000 tx=0x7f3a540046c0 comp rx=0 tx=0).ready entity=mgr.7960 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:14.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.503+0000 7f3a4effd640 1 -- 172.21.15.57:0/2378470973 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3a58071070 con 0x7f3a581e3470 2023-10-19T13:04:14.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.678+0000 7f3a60444640 1 -- 172.21.15.57:0/2378470973 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3a581f12e0 con 0x7f3a581e3470 2023-10-19T13:04:14.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.679+0000 7f3a4effd640 1 -- 172.21.15.57:0/2378470973 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v135) v1 ==== 74+0+14442 (secure 0 0 0) 0x7f3a581f12e0 con 0x7f3a581e3470 2023-10-19T13:04:14.682 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:04:14.682 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":135,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T13:04:13.219676+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.179:6837/768231146":"2023-10-20T13:04:10.128840+0000","172.21.15.179:6835/768231146":"2023-10-20T13:04:10.128840+0000","172.21.15.179:0/865284344":"2023-10-20T13:04:07.723901+0000","172.21.15.179:6832/1210673354":"2023-10-20T13:04:07.723901+0000","172.21.15.179:0/4226732685":"2023-10-20T13:04:07.723901+0000","172.21.15.179:0/3662182511":"2023-10-20T13:04:07.723901+0000","172.21.15.179:0/2769057100":"2023-10-20T13:04:07.723901+0000","172.21.15.179:6833/1210673354":"2023-10-20T13:04:07.723901+0000","172.21.15.179:6833/4155586759":"2023-10-20T13:03:08.761549+0000","172.21.15.179:0/2669494882":"2023-10-20T13:04:07.723901+0000","172.21.15.179:0/1859890742":"2023-10-20T13:04:07.723901+0000","172.21.15.179:6832/4155586759":"2023-10-20T13:03:08.761549+0000","172.21.15.179:0/1341670560":"2023-10-20T13:03:08.761549+0000","172.21.15.179:0/33839263":"2023-10-20T13:03:08.761549+0000","172.21.15.179:0/2362202135":"2023-10-20T13:04:07.723901+0000","172.21.15.179:0/3689030544":"2023-10-20T13:03:08.761549+0000","172.21.15.179:0/3560759535":"2023-10-20T13:04:07.723901+0000","172.21.15.179:0/3037001824":"2023-10-20T13:03:08.761549+0000","172.21.15.179:0/1069074991":"2023-10-20T13:03:08.761549+0000","172.21.15.179:0/831719196":"2023-10-20T13:03:08.761549+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T13:04:14.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.681+0000 7f3a60444640 1 -- 172.21.15.57:0/2378470973 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f3a3c06af60 msgr2=0x7f3a3c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:14.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.681+0000 7f3a60444640 1 --2- 172.21.15.57:0/2378470973 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f3a3c06af60 0x7f3a3c06d420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f3a54036000 tx=0x7f3a540046c0 comp rx=0 tx=0).stop 2023-10-19T13:04:14.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.681+0000 7f3a60444640 1 -- 172.21.15.57:0/2378470973 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3a581e3470 msgr2=0x7f3a580ad5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:14.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.681+0000 7f3a60444640 1 --2- 172.21.15.57:0/2378470973 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3a581e3470 0x7f3a580ad5e0 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7f3a5806d560 tx=0x7f3a5000f3c0 comp rx=0 tx=0).stop 2023-10-19T13:04:14.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.681+0000 7f3a60444640 1 -- 172.21.15.57:0/2378470973 shutdown_connections 2023-10-19T13:04:14.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.681+0000 7f3a60444640 1 --2- 172.21.15.57:0/2378470973 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f3a3c06af60 0x7f3a3c06d420 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:14.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.681+0000 7f3a60444640 1 --2- 172.21.15.57:0/2378470973 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3a581e3470 0x7f3a580ad5e0 unknown :-1 s=CLOSED pgs=1544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:14.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.681+0000 7f3a60444640 1 --2- 172.21.15.57:0/2378470973 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3a581e3010 0x7f3a580ad0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:14.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.681+0000 7f3a60444640 1 --2- 172.21.15.57:0/2378470973 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3a440039a0 0x7f3a5813aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:14.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.681+0000 7f3a60444640 1 -- 172.21.15.57:0/2378470973 >> 172.21.15.57:0/2378470973 conn(0x7f3a58069930 msgr2=0x7f3a5815cf50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:14.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.681+0000 7f3a60444640 1 -- 172.21.15.57:0/2378470973 shutdown_connections 2023-10-19T13:04:14.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.681+0000 7f3a60444640 1 -- 172.21.15.57:0/2378470973 wait complete. 2023-10-19T13:04:14.694 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-19T12:44:07.371186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-19T13:04:14.695 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-19T13:04:14.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.805+0000 7fa573dd0640 1 -- 172.21.15.57:0/2309375087 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa56c070920 msgr2=0x7fa56c0affe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:14.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.805+0000 7fa573dd0640 1 --2- 172.21.15.57:0/2309375087 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa56c070920 0x7fa56c0affe0 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7fa564005fd0 tx=0x7fa56402c240 comp rx=0 tx=0).stop 2023-10-19T13:04:14.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.805+0000 7fa573dd0640 1 -- 172.21.15.57:0/2309375087 shutdown_connections 2023-10-19T13:04:14.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.805+0000 7fa573dd0640 1 --2- 172.21.15.57:0/2309375087 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa56c141690 0x7fa56c0703e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:14.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.805+0000 7fa573dd0640 1 --2- 172.21.15.57:0/2309375087 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa56c070920 0x7fa56c0affe0 unknown :-1 s=CLOSED pgs=1546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:14.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.805+0000 7fa573dd0640 1 -- 172.21.15.57:0/2309375087 >> 172.21.15.57:0/2309375087 conn(0x7fa56c069930 msgr2=0x7fa56c069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:14.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.805+0000 7fa573dd0640 1 -- 172.21.15.57:0/2309375087 shutdown_connections 2023-10-19T13:04:14.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.805+0000 7fa573dd0640 1 -- 172.21.15.57:0/2309375087 wait complete. 2023-10-19T13:04:14.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.806+0000 7fa573dd0640 1 Processor -- start 2023-10-19T13:04:14.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.806+0000 7fa573dd0640 1 -- start start 2023-10-19T13:04:14.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.807+0000 7fa573dd0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa56c070920 0x7fa56c13b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:14.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.807+0000 7fa573dd0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa56c13e520 0x7fa56c0c78c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:14.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.807+0000 7fa573dd0640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa56c141690 0x7fa56c0c5da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:14.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.807+0000 7fa573dd0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa56c0c6820 con 0x7fa56c13e520 2023-10-19T13:04:14.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.807+0000 7fa573dd0640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa56c0c6b00 con 0x7fa56c070920 2023-10-19T13:04:14.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.807+0000 7fa573dd0640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa56c0c6de0 con 0x7fa56c141690 2023-10-19T13:04:14.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.807+0000 7fa57136b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa56c13e520 0x7fa56c0c78c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:14.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.807+0000 7fa57136b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa56c13e520 0x7fa56c0c78c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33898/0 (socket says 172.21.15.57:33898) 2023-10-19T13:04:14.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.807+0000 7fa57136b640 1 -- 172.21.15.57:0/891209859 learned_addr learned my addr 172.21.15.57:0/891209859 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:14.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.807+0000 7fa571b6c640 1 --2- 172.21.15.57:0/891209859 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa56c070920 0x7fa56c13b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:14.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.807+0000 7fa57236d640 1 --2- 172.21.15.57:0/891209859 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa56c141690 0x7fa56c0c5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:14.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.807+0000 7fa57136b640 1 -- 172.21.15.57:0/891209859 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa56c141690 msgr2=0x7fa56c0c5da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:14.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.807+0000 7fa57136b640 1 --2- 172.21.15.57:0/891209859 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa56c141690 0x7fa56c0c5da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:14.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.807+0000 7fa57136b640 1 -- 172.21.15.57:0/891209859 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa56c070920 msgr2=0x7fa56c13b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:14.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.807+0000 7fa57136b640 1 --2- 172.21.15.57:0/891209859 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa56c070920 0x7fa56c13b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:14.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.807+0000 7fa57136b640 1 -- 172.21.15.57:0/891209859 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa564009900 con 0x7fa56c13e520 2023-10-19T13:04:14.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.807+0000 7fa571b6c640 1 --2- 172.21.15.57:0/891209859 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa56c070920 0x7fa56c13b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:04:14.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.807+0000 7fa57136b640 1 --2- 172.21.15.57:0/891209859 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa56c13e520 0x7fa56c0c78c0 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7fa55c00d740 tx=0x7fa55c00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:14.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.808+0000 7fa5627fc640 1 -- 172.21.15.57:0/891209859 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa55c009c30 con 0x7fa56c13e520 2023-10-19T13:04:14.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.808+0000 7fa5627fc640 1 -- 172.21.15.57:0/891209859 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fa55c009e80 con 0x7fa56c13e520 2023-10-19T13:04:14.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.808+0000 7fa573dd0640 1 -- 172.21.15.57:0/891209859 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa56c0c70c0 con 0x7fa56c13e520 2023-10-19T13:04:14.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.808+0000 7fa5627fc640 1 -- 172.21.15.57:0/891209859 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa55c0106c0 con 0x7fa56c13e520 2023-10-19T13:04:14.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.808+0000 7fa573dd0640 1 -- 172.21.15.57:0/891209859 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa56c0c7e00 con 0x7fa56c13e520 2023-10-19T13:04:14.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.809+0000 7fa573dd0640 1 -- 172.21.15.57:0/891209859 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa56c0b1bd0 con 0x7fa56c13e520 2023-10-19T13:04:14.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.809+0000 7fa5627fc640 1 -- 172.21.15.57:0/891209859 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 69) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fa55c010820 con 0x7fa56c13e520 2023-10-19T13:04:14.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.809+0000 7fa5627fc640 1 --2- 172.21.15.57:0/891209859 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7fa55006af60 0x7fa55006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:14.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.810+0000 7fa5627fc640 1 -- 172.21.15.57:0/891209859 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 6093+0+0 (secure 0 0 0) 0x7fa55c089fc0 con 0x7fa56c13e520 2023-10-19T13:04:14.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.810+0000 7fa571b6c640 1 --2- 172.21.15.57:0/891209859 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7fa55006af60 0x7fa55006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:14.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.810+0000 7fa571b6c640 1 --2- 172.21.15.57:0/891209859 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7fa55006af60 0x7fa55006d420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fa568007fd0 tx=0x7fa568009040 comp rx=0 tx=0).ready entity=mgr.7960 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:14.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.814+0000 7fa5627fc640 1 -- 172.21.15.57:0/891209859 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa55c0bcca0 con 0x7fa56c13e520 2023-10-19T13:04:14.992 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.990+0000 7fa573dd0640 1 -- 172.21.15.57:0/891209859 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa56c067930 con 0x7fa56c13e520 2023-10-19T13:04:14.993 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.990+0000 7fa5627fc640 1 -- 172.21.15.57:0/891209859 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v135) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa55c0586e0 con 0x7fa56c13e520 2023-10-19T13:04:14.993 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-19T13:04:14.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.992+0000 7fa573dd0640 1 -- 172.21.15.57:0/891209859 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7fa55006af60 msgr2=0x7fa55006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:14.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.992+0000 7fa573dd0640 1 --2- 172.21.15.57:0/891209859 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7fa55006af60 0x7fa55006d420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fa568007fd0 tx=0x7fa568009040 comp rx=0 tx=0).stop 2023-10-19T13:04:14.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.992+0000 7fa573dd0640 1 -- 172.21.15.57:0/891209859 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa56c13e520 msgr2=0x7fa56c0c78c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:14.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.992+0000 7fa573dd0640 1 --2- 172.21.15.57:0/891209859 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa56c13e520 0x7fa56c0c78c0 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7fa55c00d740 tx=0x7fa55c00dc00 comp rx=0 tx=0).stop 2023-10-19T13:04:14.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.992+0000 7fa573dd0640 1 -- 172.21.15.57:0/891209859 shutdown_connections 2023-10-19T13:04:14.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.992+0000 7fa573dd0640 1 --2- 172.21.15.57:0/891209859 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7fa55006af60 0x7fa55006d420 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:14.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.992+0000 7fa573dd0640 1 --2- 172.21.15.57:0/891209859 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa56c141690 0x7fa56c0c5da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:14.997 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.992+0000 7fa573dd0640 1 --2- 172.21.15.57:0/891209859 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa56c13e520 0x7fa56c0c78c0 unknown :-1 s=CLOSED pgs=1547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:14.997 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.992+0000 7fa573dd0640 1 --2- 172.21.15.57:0/891209859 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa56c070920 0x7fa56c13b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:14.997 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.992+0000 7fa573dd0640 1 -- 172.21.15.57:0/891209859 >> 172.21.15.57:0/891209859 conn(0x7fa56c069930 msgr2=0x7fa56c0ae670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:14.997 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.992+0000 7fa573dd0640 1 -- 172.21.15.57:0/891209859 shutdown_connections 2023-10-19T13:04:14.997 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:14.992+0000 7fa573dd0640 1 -- 172.21.15.57:0/891209859 wait complete. 2023-10-19T13:04:15.005 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods' 2023-10-19T13:04:15.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:15.112+0000 7f0ace021640 1 -- 172.21.15.57:0/1449928601 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ac8070350 msgr2=0x7f0ac8070780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:15.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:15.112+0000 7f0ace021640 1 --2- 172.21.15.57:0/1449928601 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ac8070350 0x7f0ac8070780 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7f0ac80cc560 tx=0x7f0ac002c260 comp rx=0 tx=0).stop 2023-10-19T13:04:15.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:15.112+0000 7f0ace021640 1 -- 172.21.15.57:0/1449928601 shutdown_connections 2023-10-19T13:04:15.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:15.112+0000 7f0ace021640 1 --2- 172.21.15.57:0/1449928601 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ac8070350 0x7f0ac8070780 unknown :-1 s=CLOSED pgs=1548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:15.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:15.112+0000 7f0ace021640 1 --2- 172.21.15.57:0/1449928601 >> v2:172.21.15.179:3300/0 conn(0x7f0ac80cb980 0x7f0ac813db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:15.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:15.112+0000 7f0ace021640 1 -- 172.21.15.57:0/1449928601 >> 172.21.15.57:0/1449928601 conn(0x7f0ac8069930 msgr2=0x7f0ac8069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:15.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:15.112+0000 7f0ace021640 1 -- 172.21.15.57:0/1449928601 shutdown_connections 2023-10-19T13:04:15.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:15.112+0000 7f0ace021640 1 -- 172.21.15.57:0/1449928601 wait complete. 2023-10-19T13:04:15.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:15.113+0000 7f0ace021640 1 Processor -- start 2023-10-19T13:04:15.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:15.114+0000 7f0ace021640 1 -- start start 2023-10-19T13:04:15.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:15.114+0000 7f0ace021640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0ac8070350 0x7f0ac81f10d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:15.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:15.114+0000 7f0ace021640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ac80cb980 0x7f0ac80ad5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:15.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:15.114+0000 7f0ace021640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0ac8133b90 0x7f0ac80adb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:15.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:15.114+0000 7f0ace021640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ac80aabc0 con 0x7f0ac80cb980 2023-10-19T13:04:15.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:15.114+0000 7f0ace021640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ac813c9f0 con 0x7f0ac8133b90 2023-10-19T13:04:15.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:15.114+0000 7f0ace021640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ac813cc40 con 0x7f0ac8070350 2023-10-19T13:04:15.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:15.114+0000 7f0ac6ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ac80cb980 0x7f0ac80ad5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:15.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:15.114+0000 7f0ac6ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ac80cb980 0x7f0ac80ad5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33922/0 (socket says 172.21.15.57:33922) 2023-10-19T13:04:15.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:15.114+0000 7f0ac6ffd640 1 -- 172.21.15.57:0/2496118084 learned_addr learned my addr 172.21.15.57:0/2496118084 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:15.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:15.115+0000 7f0ac77fe640 1 --2- 172.21.15.57:0/2496118084 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0ac8070350 0x7f0ac81f10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:15.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:15.115+0000 7f0ac7fff640 1 --2- 172.21.15.57:0/2496118084 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0ac8133b90 0x7f0ac80adb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:15.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:15.115+0000 7f0ac6ffd640 1 -- 172.21.15.57:0/2496118084 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0ac8070350 msgr2=0x7f0ac81f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:15.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:15.115+0000 7f0ac6ffd640 1 --2- 172.21.15.57:0/2496118084 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0ac8070350 0x7f0ac81f10d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:15.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:15.115+0000 7f0ac6ffd640 1 -- 172.21.15.57:0/2496118084 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0ac8133b90 msgr2=0x7f0ac80adb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:15.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:15.115+0000 7f0ac6ffd640 1 --2- 172.21.15.57:0/2496118084 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0ac8133b90 0x7f0ac80adb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:15.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:15.115+0000 7f0ac6ffd640 1 -- 172.21.15.57:0/2496118084 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ac0009900 con 0x7f0ac80cb980 2023-10-19T13:04:15.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:15.115+0000 7f0ac77fe640 1 --2- 172.21.15.57:0/2496118084 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0ac8070350 0x7f0ac81f10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:04:15.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:15.115+0000 7f0ac7fff640 1 --2- 172.21.15.57:0/2496118084 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0ac8133b90 0x7f0ac80adb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:04:15.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:15.115+0000 7f0ac6ffd640 1 --2- 172.21.15.57:0/2496118084 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ac80cb980 0x7f0ac80ad5e0 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7f0abc00b3e0 tx=0x7f0abc00b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:15.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:15.115+0000 7f0aaffff640 1 -- 172.21.15.57:0/2496118084 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0abc011830 con 0x7f0ac80cb980 2023-10-19T13:04:15.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:15.115+0000 7f0aaffff640 1 -- 172.21.15.57:0/2496118084 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f0abc011aa0 con 0x7f0ac80cb980 2023-10-19T13:04:15.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:15.115+0000 7f0ace021640 1 -- 172.21.15.57:0/2496118084 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ac81f5960 con 0x7f0ac80cb980 2023-10-19T13:04:15.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:15.115+0000 7f0aaffff640 1 -- 172.21.15.57:0/2496118084 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0abc010380 con 0x7f0ac80cb980 2023-10-19T13:04:15.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:15.115+0000 7f0ace021640 1 -- 172.21.15.57:0/2496118084 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ac81f1610 con 0x7f0ac80cb980 2023-10-19T13:04:15.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:15.116+0000 7f0ace021640 1 -- 172.21.15.57:0/2496118084 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ac8139020 con 0x7f0ac80cb980 2023-10-19T13:04:15.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:15.117+0000 7f0aaffff640 1 -- 172.21.15.57:0/2496118084 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 69) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f0abc0104e0 con 0x7f0ac80cb980 2023-10-19T13:04:15.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:15.117+0000 7f0aaffff640 1 --2- 172.21.15.57:0/2496118084 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f0ab406af60 0x7f0ab406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:15.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:15.117+0000 7f0aaffff640 1 -- 172.21.15.57:0/2496118084 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 6093+0+0 (secure 0 0 0) 0x7f0abc089d40 con 0x7f0ac80cb980 2023-10-19T13:04:15.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:15.117+0000 7f0ac77fe640 1 --2- 172.21.15.57:0/2496118084 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f0ab406af60 0x7f0ab406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:15.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:15.118+0000 7f0ac77fe640 1 --2- 172.21.15.57:0/2496118084 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f0ab406af60 0x7f0ab406d420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0ab8005fd0 tx=0x7f0ab8005ea0 comp rx=0 tx=0).ready entity=mgr.7960 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:15.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:15.122+0000 7f0aaffff640 1 -- 172.21.15.57:0/2496118084 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0ac8139020 con 0x7f0ac80cb980 2023-10-19T13:04:15.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:15.364+0000 7f0ace021640 1 -- 172.21.15.57:0/2496118084 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods"]} v 0) v1 -- 0x7f0ac8139020 con 0x7f0ac80cb980 2023-10-19T13:04:16.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.376+0000 7f0aaffff640 1 -- 172.21.15.57:0/2496118084 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods"]}]=0 v759) v1 ==== 163+0+0 (secure 0 0 0) 0x7f0ac8139020 con 0x7f0ac80cb980 2023-10-19T13:04:16.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.379+0000 7f0ace021640 1 -- 172.21.15.57:0/2496118084 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f0ab406af60 msgr2=0x7f0ab406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:16.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.379+0000 7f0ace021640 1 --2- 172.21.15.57:0/2496118084 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f0ab406af60 0x7f0ab406d420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0ab8005fd0 tx=0x7f0ab8005ea0 comp rx=0 tx=0).stop 2023-10-19T13:04:16.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.379+0000 7f0ace021640 1 -- 172.21.15.57:0/2496118084 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ac80cb980 msgr2=0x7f0ac80ad5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:16.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.379+0000 7f0ace021640 1 --2- 172.21.15.57:0/2496118084 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ac80cb980 0x7f0ac80ad5e0 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7f0abc00b3e0 tx=0x7f0abc00b8a0 comp rx=0 tx=0).stop 2023-10-19T13:04:16.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.379+0000 7f0ace021640 1 -- 172.21.15.57:0/2496118084 shutdown_connections 2023-10-19T13:04:16.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.379+0000 7f0ace021640 1 --2- 172.21.15.57:0/2496118084 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f0ab406af60 0x7f0ab406d420 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:16.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.379+0000 7f0ace021640 1 --2- 172.21.15.57:0/2496118084 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0ac8133b90 0x7f0ac80adb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:16.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.379+0000 7f0ace021640 1 --2- 172.21.15.57:0/2496118084 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ac80cb980 0x7f0ac80ad5e0 unknown :-1 s=CLOSED pgs=1549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:16.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.379+0000 7f0ace021640 1 --2- 172.21.15.57:0/2496118084 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0ac8070350 0x7f0ac81f10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:16.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.379+0000 7f0ace021640 1 -- 172.21.15.57:0/2496118084 >> 172.21.15.57:0/2496118084 conn(0x7f0ac8069930 msgr2=0x7f0ac806be60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:16.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.379+0000 7f0ace021640 1 -- 172.21.15.57:0/2496118084 shutdown_connections 2023-10-19T13:04:16.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.379+0000 7f0ace021640 1 -- 172.21.15.57:0/2496118084 wait complete. 2023-10-19T13:04:16.395 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2023-10-19T13:04:16.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.507+0000 7f9a4fd7d640 1 -- 172.21.15.57:0/3867948728 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a4813d350 msgr2=0x7f9a480cbd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:16.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.507+0000 7f9a4fd7d640 1 --2- 172.21.15.57:0/3867948728 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a4813d350 0x7f9a480cbd60 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f9a480cc560 tx=0x7f9a3802bdb0 comp rx=0 tx=0).stop 2023-10-19T13:04:16.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.507+0000 7f9a4fd7d640 1 -- 172.21.15.57:0/3867948728 shutdown_connections 2023-10-19T13:04:16.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.507+0000 7f9a4fd7d640 1 --2- 172.21.15.57:0/3867948728 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a4813d350 0x7f9a480cbd60 unknown :-1 s=CLOSED pgs=1550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:16.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.507+0000 7f9a4fd7d640 1 --2- 172.21.15.57:0/3867948728 >> v2:172.21.15.179:3300/0 conn(0x7f9a480cb980 0x7f9a4813db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:16.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.507+0000 7f9a4fd7d640 1 -- 172.21.15.57:0/3867948728 >> 172.21.15.57:0/3867948728 conn(0x7f9a48069930 msgr2=0x7f9a48069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:16.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.508+0000 7f9a4fd7d640 1 -- 172.21.15.57:0/3867948728 shutdown_connections 2023-10-19T13:04:16.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.508+0000 7f9a4fd7d640 1 -- 172.21.15.57:0/3867948728 wait complete. 2023-10-19T13:04:16.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.509+0000 7f9a4fd7d640 1 Processor -- start 2023-10-19T13:04:16.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.509+0000 7f9a4fd7d640 1 -- start start 2023-10-19T13:04:16.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.509+0000 7f9a4fd7d640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9a480c96b0 0x7f9a48136750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:16.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.510+0000 7f9a4fd7d640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9a480cb980 0x7f9a480ab8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:16.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.510+0000 7f9a4fd7d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a4813bab0 0x7f9a480ca870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:16.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.510+0000 7f9a4fd7d640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a4815aa60 con 0x7f9a4813bab0 2023-10-19T13:04:16.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.510+0000 7f9a4fd7d640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a480abe20 con 0x7f9a480cb980 2023-10-19T13:04:16.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.510+0000 7f9a4fd7d640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a480ab400 con 0x7f9a480c96b0 2023-10-19T13:04:16.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.510+0000 7f9a4e31a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a4813bab0 0x7f9a480ca870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:16.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.510+0000 7f9a4e31a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a4813bab0 0x7f9a480ca870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33942/0 (socket says 172.21.15.57:33942) 2023-10-19T13:04:16.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.510+0000 7f9a4e31a640 1 -- 172.21.15.57:0/1117183508 learned_addr learned my addr 172.21.15.57:0/1117183508 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:16.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.510+0000 7f9a4db19640 1 --2- 172.21.15.57:0/1117183508 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9a480c96b0 0x7f9a48136750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:16.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.510+0000 7f9a4d318640 1 --2- 172.21.15.57:0/1117183508 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9a480cb980 0x7f9a480ab8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:16.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.510+0000 7f9a4e31a640 1 -- 172.21.15.57:0/1117183508 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9a480c96b0 msgr2=0x7f9a48136750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:16.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.510+0000 7f9a4e31a640 1 --2- 172.21.15.57:0/1117183508 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9a480c96b0 0x7f9a48136750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:16.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.510+0000 7f9a4e31a640 1 -- 172.21.15.57:0/1117183508 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9a480cb980 msgr2=0x7f9a480ab8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:16.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.510+0000 7f9a4e31a640 1 --2- 172.21.15.57:0/1117183508 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9a480cb980 0x7f9a480ab8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:16.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.510+0000 7f9a4e31a640 1 -- 172.21.15.57:0/1117183508 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a38009560 con 0x7f9a4813bab0 2023-10-19T13:04:16.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.510+0000 7f9a4e31a640 1 --2- 172.21.15.57:0/1117183508 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a4813bab0 0x7f9a480ca870 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7f9a4000c780 tx=0x7f9a4000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:16.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.510+0000 7f9a3e7fc640 1 -- 172.21.15.57:0/1117183508 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a40004b60 con 0x7f9a4813bab0 2023-10-19T13:04:16.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.510+0000 7f9a4fd7d640 1 -- 172.21.15.57:0/1117183508 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a481364b0 con 0x7f9a4813bab0 2023-10-19T13:04:16.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.510+0000 7f9a3e7fc640 1 -- 172.21.15.57:0/1117183508 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f9a4000ce90 con 0x7f9a4813bab0 2023-10-19T13:04:16.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.510+0000 7f9a3e7fc640 1 -- 172.21.15.57:0/1117183508 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a4000aab0 con 0x7f9a4813bab0 2023-10-19T13:04:16.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.511+0000 7f9a4fd7d640 1 -- 172.21.15.57:0/1117183508 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a4815a780 con 0x7f9a4813bab0 2023-10-19T13:04:16.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.511+0000 7f9a4fd7d640 1 -- 172.21.15.57:0/1117183508 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a480690e0 con 0x7f9a4813bab0 2023-10-19T13:04:16.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.512+0000 7f9a3e7fc640 1 -- 172.21.15.57:0/1117183508 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 69) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f9a4000ac10 con 0x7f9a4813bab0 2023-10-19T13:04:16.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.512+0000 7f9a3e7fc640 1 --2- 172.21.15.57:0/1117183508 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f9a2c06af60 0x7f9a2c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:16.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.513+0000 7f9a3e7fc640 1 -- 172.21.15.57:0/1117183508 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 6093+0+0 (secure 0 0 0) 0x7f9a40089bc0 con 0x7f9a4813bab0 2023-10-19T13:04:16.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.513+0000 7f9a4db19640 1 --2- 172.21.15.57:0/1117183508 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f9a2c06af60 0x7f9a2c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:16.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.513+0000 7f9a4db19640 1 --2- 172.21.15.57:0/1117183508 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f9a2c06af60 0x7f9a2c06d420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9a4400bbe0 tx=0x7f9a4400b390 comp rx=0 tx=0).ready entity=mgr.7960 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:16.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.517+0000 7f9a3e7fc640 1 -- 172.21.15.57:0/1117183508 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9a40054aa0 con 0x7f9a4813bab0 2023-10-19T13:04:16.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.694+0000 7f9a4fd7d640 1 -- 172.21.15.57:0/1117183508 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "config dump"} v 0) v1 -- 0x7f9a4806ff80 con 0x7f9a4813bab0 2023-10-19T13:04:16.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.694+0000 7f9a3e7fc640 1 -- 172.21.15.57:0/1117183508 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v54) v1 ==== 59+0+2220 (secure 0 0 0) 0x7f9a400580c0 con 0x7f9a4813bab0 2023-10-19T13:04:16.698 INFO:teuthology.orchestra.run.smithi057.stdout:WHO MASK LEVEL OPTION VALUE RO 2023-10-19T13:04:16.698 INFO:teuthology.orchestra.run.smithi057.stdout:mon advanced mon_op_complaint_time 120 2023-10-19T13:04:16.698 INFO:teuthology.orchestra.run.smithi057.stdout:osd advanced osd_op_complaint_time 180.000000 2023-10-19T13:04:16.698 INFO:teuthology.orchestra.run.smithi057.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 4775.995373 2023-10-19T13:04:16.699 INFO:teuthology.orchestra.run.smithi057.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 4208.118410 2023-10-19T13:04:16.699 INFO:teuthology.orchestra.run.smithi057.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 4541.123030 2023-10-19T13:04:16.699 INFO:teuthology.orchestra.run.smithi057.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 4877.592206 2023-10-19T13:04:16.699 INFO:teuthology.orchestra.run.smithi057.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 5019.748032 2023-10-19T13:04:16.699 INFO:teuthology.orchestra.run.smithi057.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 5000.074376 2023-10-19T13:04:16.699 INFO:teuthology.orchestra.run.smithi057.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 5153.479439 2023-10-19T13:04:16.700 INFO:teuthology.orchestra.run.smithi057.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 4977.363845 2023-10-19T13:04:16.700 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced debug_mds 20/20 2023-10-19T13:04:16.700 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced debug_mds_balancer 20/20 2023-10-19T13:04:16.700 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced debug_ms 1/1 2023-10-19T13:04:16.700 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced mds_bal_fragment_size_max 10000 2023-10-19T13:04:16.701 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced mds_bal_merge_size 5 2023-10-19T13:04:16.701 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced mds_bal_split_bits 3 2023-10-19T13:04:16.701 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced mds_bal_split_size 100 2023-10-19T13:04:16.701 INFO:teuthology.orchestra.run.smithi057.stdout:mds dev mds_debug_frag true 2023-10-19T13:04:16.701 INFO:teuthology.orchestra.run.smithi057.stdout:mds dev mds_debug_scatterstat true 2023-10-19T13:04:16.702 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced mds_op_complaint_time 180.000000 2023-10-19T13:04:16.702 INFO:teuthology.orchestra.run.smithi057.stdout:mds dev mds_verify_scatter true 2023-10-19T13:04:16.702 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced osd_op_complaint_time 180.000000 2023-10-19T13:04:16.702 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced rados_mon_op_timeout 900 2023-10-19T13:04:16.703 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced rados_osd_op_timeout 900 2023-10-19T13:04:16.703 INFO:teuthology.orchestra.run.smithi057.stdout:client advanced client_mount_timeout 600 2023-10-19T13:04:16.703 INFO:teuthology.orchestra.run.smithi057.stdout:client advanced debug_client 20/20 2023-10-19T13:04:16.703 INFO:teuthology.orchestra.run.smithi057.stdout:client advanced debug_ms 1/1 2023-10-19T13:04:16.703 INFO:teuthology.orchestra.run.smithi057.stdout:client advanced rados_mon_op_timeout 900 2023-10-19T13:04:16.704 INFO:teuthology.orchestra.run.smithi057.stdout:client advanced rados_osd_op_timeout 900 2023-10-19T13:04:16.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.696+0000 7f9a4fd7d640 1 -- 172.21.15.57:0/1117183508 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f9a2c06af60 msgr2=0x7f9a2c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:16.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.696+0000 7f9a4fd7d640 1 --2- 172.21.15.57:0/1117183508 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f9a2c06af60 0x7f9a2c06d420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9a4400bbe0 tx=0x7f9a4400b390 comp rx=0 tx=0).stop 2023-10-19T13:04:16.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.696+0000 7f9a4fd7d640 1 -- 172.21.15.57:0/1117183508 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a4813bab0 msgr2=0x7f9a480ca870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:16.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.696+0000 7f9a4fd7d640 1 --2- 172.21.15.57:0/1117183508 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a4813bab0 0x7f9a480ca870 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7f9a4000c780 tx=0x7f9a4000cc40 comp rx=0 tx=0).stop 2023-10-19T13:04:16.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.696+0000 7f9a4fd7d640 1 -- 172.21.15.57:0/1117183508 shutdown_connections 2023-10-19T13:04:16.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.696+0000 7f9a4fd7d640 1 --2- 172.21.15.57:0/1117183508 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f9a2c06af60 0x7f9a2c06d420 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:16.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.696+0000 7f9a4fd7d640 1 --2- 172.21.15.57:0/1117183508 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a4813bab0 0x7f9a480ca870 unknown :-1 s=CLOSED pgs=1551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:16.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.696+0000 7f9a4fd7d640 1 --2- 172.21.15.57:0/1117183508 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9a480cb980 0x7f9a480ab8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:16.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.696+0000 7f9a4fd7d640 1 --2- 172.21.15.57:0/1117183508 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9a480c96b0 0x7f9a48136750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:16.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.696+0000 7f9a4fd7d640 1 -- 172.21.15.57:0/1117183508 >> 172.21.15.57:0/1117183508 conn(0x7f9a48069930 msgr2=0x7f9a48158a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:16.707 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.697+0000 7f9a4fd7d640 1 -- 172.21.15.57:0/1117183508 shutdown_connections 2023-10-19T13:04:16.707 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.697+0000 7f9a4fd7d640 1 -- 172.21.15.57:0/1117183508 wait complete. 2023-10-19T13:04:16.710 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-19T13:04:16.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.818+0000 7f9462db6640 1 -- 172.21.15.57:0/3979676401 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f945c0cbd20 msgr2=0x7f945c158300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:16.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.818+0000 7f9462db6640 1 --2- 172.21.15.57:0/3979676401 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f945c0cbd20 0x7f945c158300 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f9450005fd0 tx=0x7f945002be20 comp rx=0 tx=0).stop 2023-10-19T13:04:16.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.818+0000 7f9462db6640 1 -- 172.21.15.57:0/3979676401 shutdown_connections 2023-10-19T13:04:16.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.818+0000 7f9462db6640 1 --2- 172.21.15.57:0/3979676401 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f945c0cbd20 0x7f945c158300 unknown :-1 s=CLOSED pgs=1553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:16.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.818+0000 7f9462db6640 1 -- 172.21.15.57:0/3979676401 >> 172.21.15.57:0/3979676401 conn(0x7f945c069930 msgr2=0x7f945c069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:16.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.818+0000 7f9462db6640 1 -- 172.21.15.57:0/3979676401 shutdown_connections 2023-10-19T13:04:16.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.818+0000 7f9462db6640 1 -- 172.21.15.57:0/3979676401 wait complete. 2023-10-19T13:04:16.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.819+0000 7f9462db6640 1 Processor -- start 2023-10-19T13:04:16.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.820+0000 7f9462db6640 1 -- start start 2023-10-19T13:04:16.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.820+0000 7f9462db6640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f945c0afa20 0x7f945c155bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:16.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.820+0000 7f9462db6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f945c0b0530 0x7f945c14de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:16.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.820+0000 7f9462db6640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f945c0cbd20 0x7f945c1530b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:16.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.820+0000 7f9462db6640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f945c1e1ce0 con 0x7f945c0b0530 2023-10-19T13:04:16.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.820+0000 7f9462db6640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f945c156400 con 0x7f945c0afa20 2023-10-19T13:04:16.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.820+0000 7f9462db6640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f945c1566e0 con 0x7f945c0cbd20 2023-10-19T13:04:16.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.820+0000 7f945bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f945c0b0530 0x7f945c14de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:16.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.821+0000 7f945bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f945c0b0530 0x7f945c14de90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35356/0 (socket says 172.21.15.57:35356) 2023-10-19T13:04:16.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.821+0000 7f945bfff640 1 -- 172.21.15.57:0/3502762180 learned_addr learned my addr 172.21.15.57:0/3502762180 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:16.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.821+0000 7f9460b52640 1 --2- 172.21.15.57:0/3502762180 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f945c0afa20 0x7f945c155bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:16.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.821+0000 7f9461353640 1 --2- 172.21.15.57:0/3502762180 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f945c0cbd20 0x7f945c1530b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:16.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.821+0000 7f945bfff640 1 -- 172.21.15.57:0/3502762180 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f945c0cbd20 msgr2=0x7f945c1530b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:16.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.821+0000 7f945bfff640 1 --2- 172.21.15.57:0/3502762180 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f945c0cbd20 0x7f945c1530b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:16.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.821+0000 7f945bfff640 1 -- 172.21.15.57:0/3502762180 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f945c0afa20 msgr2=0x7f945c155bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:16.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.821+0000 7f945bfff640 1 --2- 172.21.15.57:0/3502762180 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f945c0afa20 0x7f945c155bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:16.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.821+0000 7f945bfff640 1 -- 172.21.15.57:0/3502762180 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9450009560 con 0x7f945c0b0530 2023-10-19T13:04:16.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.821+0000 7f9460b52640 1 --2- 172.21.15.57:0/3502762180 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f945c0afa20 0x7f945c155bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T13:04:16.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.821+0000 7f945bfff640 1 --2- 172.21.15.57:0/3502762180 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f945c0b0530 0x7f945c14de90 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7f944800c430 tx=0x7f944800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:16.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.821+0000 7f94597fa640 1 -- 172.21.15.57:0/3502762180 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94480108a0 con 0x7f945c0b0530 2023-10-19T13:04:16.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.821+0000 7f94597fa640 1 -- 172.21.15.57:0/3502762180 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f9448010af0 con 0x7f945c0b0530 2023-10-19T13:04:16.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.821+0000 7f9462db6640 1 -- 172.21.15.57:0/3502762180 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f945c1569c0 con 0x7f945c0b0530 2023-10-19T13:04:16.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.821+0000 7f94597fa640 1 -- 172.21.15.57:0/3502762180 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9448011470 con 0x7f945c0b0530 2023-10-19T13:04:16.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.821+0000 7f9462db6640 1 -- 172.21.15.57:0/3502762180 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f945c1560f0 con 0x7f945c0b0530 2023-10-19T13:04:16.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.822+0000 7f9462db6640 1 -- 172.21.15.57:0/3502762180 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f945c1e1e50 con 0x7f945c0b0530 2023-10-19T13:04:16.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.823+0000 7f94597fa640 1 -- 172.21.15.57:0/3502762180 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 69) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f94480115d0 con 0x7f945c0b0530 2023-10-19T13:04:16.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.823+0000 7f94597fa640 1 --2- 172.21.15.57:0/3502762180 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f944006af60 0x7f944006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:16.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.823+0000 7f94597fa640 1 -- 172.21.15.57:0/3502762180 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 6093+0+0 (secure 0 0 0) 0x7f944808b970 con 0x7f945c0b0530 2023-10-19T13:04:16.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.823+0000 7f9460b52640 1 --2- 172.21.15.57:0/3502762180 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f944006af60 0x7f944006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:16.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.824+0000 7f9460b52640 1 --2- 172.21.15.57:0/3502762180 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f944006af60 0x7f944006d420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9450035000 tx=0x7f9450004730 comp rx=0 tx=0).ready entity=mgr.7960 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:16.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:16.827+0000 7f94597fa640 1 -- 172.21.15.57:0/3502762180 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9448056790 con 0x7f945c0b0530 2023-10-19T13:04:17.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.005+0000 7f9462db6640 1 -- 172.21.15.57:0/3502762180 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f945c0690e0 con 0x7f945c0b0530 2023-10-19T13:04:17.011 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.008+0000 7f94597fa640 1 -- 172.21.15.57:0/3502762180 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v55)=0 v55) v1 ==== 122+0+0 (secure 0 0 0) 0x7f94480200c0 con 0x7f945c0b0530 2023-10-19T13:04:17.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.010+0000 7f9462db6640 1 -- 172.21.15.57:0/3502762180 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f944006af60 msgr2=0x7f944006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:17.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.010+0000 7f9462db6640 1 --2- 172.21.15.57:0/3502762180 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f944006af60 0x7f944006d420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9450035000 tx=0x7f9450004730 comp rx=0 tx=0).stop 2023-10-19T13:04:17.014 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.010+0000 7f9462db6640 1 -- 172.21.15.57:0/3502762180 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f945c0b0530 msgr2=0x7f945c14de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:17.014 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.010+0000 7f9462db6640 1 --2- 172.21.15.57:0/3502762180 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f945c0b0530 0x7f945c14de90 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7f944800c430 tx=0x7f944800c8f0 comp rx=0 tx=0).stop 2023-10-19T13:04:17.014 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.010+0000 7f9462db6640 1 -- 172.21.15.57:0/3502762180 shutdown_connections 2023-10-19T13:04:17.014 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.010+0000 7f9462db6640 1 --2- 172.21.15.57:0/3502762180 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f944006af60 0x7f944006d420 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:17.014 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.010+0000 7f9462db6640 1 --2- 172.21.15.57:0/3502762180 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f945c0cbd20 0x7f945c1530b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:17.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.010+0000 7f9462db6640 1 --2- 172.21.15.57:0/3502762180 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f945c0b0530 0x7f945c14de90 unknown :-1 s=CLOSED pgs=1554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:17.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.010+0000 7f9462db6640 1 --2- 172.21.15.57:0/3502762180 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f945c0afa20 0x7f945c155bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:17.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.010+0000 7f9462db6640 1 -- 172.21.15.57:0/3502762180 >> 172.21.15.57:0/3502762180 conn(0x7f945c069930 msgr2=0x7f945c0ac590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:17.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.010+0000 7f9462db6640 1 -- 172.21.15.57:0/3502762180 shutdown_connections 2023-10-19T13:04:17.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.011+0000 7f9462db6640 1 -- 172.21.15.57:0/3502762180 wait complete. 2023-10-19T13:04:17.023 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-19T13:04:17.024 DEBUG:teuthology.orchestra.run.smithi057:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-19T13:04:17.070 DEBUG:teuthology.orchestra.run.smithi179:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-19T13:04:17.113 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-19T13:04:17.114 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T13:04:17.114 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T13:04:17.129 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055614,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=358 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248260k,nr_inodes=812065,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-19T13:04:17.130 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-19T13:04:17.130 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T13:04:17.170 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054833,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=284 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3247632k,nr_inodes=811908,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-19T13:04:17.171 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T13:04:17.171 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T13:04:17.186 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055614,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=358 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248260k,nr_inodes=812065,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-19T13:04:17.187 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-19T13:04:17.187 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T13:04:17.227 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054833,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=284 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3247632k,nr_inodes=811908,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-19T13:04:17.227 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-19T13:04:17.228 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T13:04:17.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.334+0000 7f1c458bb640 1 -- 172.21.15.57:0/2679816101 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1c4006a730 msgr2=0x7f1c4006abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:17.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.334+0000 7f1c458bb640 1 --2- 172.21.15.57:0/2679816101 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1c4006a730 0x7f1c4006abc0 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7f1c38005fd0 tx=0x7f1c3802c2e0 comp rx=0 tx=0).stop 2023-10-19T13:04:17.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.334+0000 7f1c458bb640 1 -- 172.21.15.57:0/2679816101 shutdown_connections 2023-10-19T13:04:17.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.334+0000 7f1c458bb640 1 --2- 172.21.15.57:0/2679816101 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1c4006a730 0x7f1c4006abc0 unknown :-1 s=CLOSED pgs=1556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:17.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.334+0000 7f1c458bb640 1 -- 172.21.15.57:0/2679816101 >> 172.21.15.57:0/2679816101 conn(0x7f1c4006fae0 msgr2=0x7f1c4006fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:17.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.334+0000 7f1c458bb640 1 -- 172.21.15.57:0/2679816101 shutdown_connections 2023-10-19T13:04:17.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.334+0000 7f1c458bb640 1 -- 172.21.15.57:0/2679816101 wait complete. 2023-10-19T13:04:17.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.335+0000 7f1c458bb640 1 Processor -- start 2023-10-19T13:04:17.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.335+0000 7f1c458bb640 1 -- start start 2023-10-19T13:04:17.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.336+0000 7f1c458bb640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1c4006a730 0x7f1c4014a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:17.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.336+0000 7f1c458bb640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1c400ac870 0x7f1c400b0750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:17.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.336+0000 7f1c458bb640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1c40156790 0x7f1c4014c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:17.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.336+0000 7f1c458bb640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c400b15b0 con 0x7f1c4006a730 2023-10-19T13:04:17.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.336+0000 7f1c458bb640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c40152a50 con 0x7f1c400ac870 2023-10-19T13:04:17.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.336+0000 7f1c458bb640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c40152420 con 0x7f1c40156790 2023-10-19T13:04:17.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.336+0000 7f1c3effd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1c4006a730 0x7f1c4014a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:17.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.336+0000 7f1c3effd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1c4006a730 0x7f1c4014a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35372/0 (socket says 172.21.15.57:35372) 2023-10-19T13:04:17.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.336+0000 7f1c3effd640 1 -- 172.21.15.57:0/1806180205 learned_addr learned my addr 172.21.15.57:0/1806180205 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:17.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.336+0000 7f1c3f7fe640 1 --2- 172.21.15.57:0/1806180205 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1c40156790 0x7f1c4014c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:17.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.336+0000 7f1c37fff640 1 --2- 172.21.15.57:0/1806180205 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1c400ac870 0x7f1c400b0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:17.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.336+0000 7f1c3effd640 1 -- 172.21.15.57:0/1806180205 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1c40156790 msgr2=0x7f1c4014c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:17.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.336+0000 7f1c3effd640 1 --2- 172.21.15.57:0/1806180205 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1c40156790 0x7f1c4014c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:17.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.336+0000 7f1c3effd640 1 -- 172.21.15.57:0/1806180205 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1c400ac870 msgr2=0x7f1c400b0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:17.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.336+0000 7f1c3effd640 1 --2- 172.21.15.57:0/1806180205 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1c400ac870 0x7f1c400b0750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:17.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.336+0000 7f1c3effd640 1 -- 172.21.15.57:0/1806180205 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c38009900 con 0x7f1c4006a730 2023-10-19T13:04:17.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.336+0000 7f1c3effd640 1 --2- 172.21.15.57:0/1806180205 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1c4006a730 0x7f1c4014a8d0 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7f1c3000c430 tx=0x7f1c3000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:17.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.336+0000 7f1c3cff9640 1 -- 172.21.15.57:0/1806180205 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c30011790 con 0x7f1c4006a730 2023-10-19T13:04:17.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.336+0000 7f1c3cff9640 1 -- 172.21.15.57:0/1806180205 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f1c300119e0 con 0x7f1c4006a730 2023-10-19T13:04:17.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.336+0000 7f1c3cff9640 1 -- 172.21.15.57:0/1806180205 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c30011e30 con 0x7f1c4006a730 2023-10-19T13:04:17.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.337+0000 7f1c458bb640 1 -- 172.21.15.57:0/1806180205 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c4014ae10 con 0x7f1c4006a730 2023-10-19T13:04:17.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.337+0000 7f1c458bb640 1 -- 172.21.15.57:0/1806180205 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c40151e60 con 0x7f1c4006a730 2023-10-19T13:04:17.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.338+0000 7f1c458bb640 1 -- 172.21.15.57:0/1806180205 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c400690e0 con 0x7f1c4006a730 2023-10-19T13:04:17.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.338+0000 7f1c3cff9640 1 -- 172.21.15.57:0/1806180205 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 69) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f1c3001c460 con 0x7f1c4006a730 2023-10-19T13:04:17.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.338+0000 7f1c3cff9640 1 --2- 172.21.15.57:0/1806180205 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f1c2006af60 0x7f1c2006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:17.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.339+0000 7f1c3cff9640 1 -- 172.21.15.57:0/1806180205 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 6093+0+0 (secure 0 0 0) 0x7f1c30089b60 con 0x7f1c4006a730 2023-10-19T13:04:17.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.339+0000 7f1c37fff640 1 --2- 172.21.15.57:0/1806180205 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f1c2006af60 0x7f1c2006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:17.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.339+0000 7f1c37fff640 1 --2- 172.21.15.57:0/1806180205 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f1c2006af60 0x7f1c2006d420 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f1c4015d110 tx=0x7f1c28009040 comp rx=0 tx=0).ready entity=mgr.7960 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:17.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.343+0000 7f1c3cff9640 1 -- 172.21.15.57:0/1806180205 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f1c30054980 con 0x7f1c4006a730 2023-10-19T13:04:17.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.558+0000 7f1c458bb640 1 -- 172.21.15.57:0/1806180205 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1c4006ff60 con 0x7f1c4006a730 2023-10-19T13:04:17.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.558+0000 7f1c3cff9640 1 -- 172.21.15.57:0/1806180205 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+3699 (secure 0 0 0) 0x7f1c30057fa0 con 0x7f1c4006a730 2023-10-19T13:04:17.561 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:04:17.562 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":52,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/2033977424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2033977424},{"type":"v1","addr":"172.21.15.179:6836","nonce":2033977424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":8089,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/3377205310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":3377205310},{"type":"v1","addr":"172.21.15.179:6837","nonce":3377205310}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":52},{"gid":8568,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/981095684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":981095684},{"type":"v1","addr":"172.21.15.57:6834","nonce":981095684}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":8571,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/383147069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":383147069},{"type":"v1","addr":"172.21.15.57:6835","nonce":383147069}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47}],"filesystems":[]} 2023-10-19T13:04:17.562 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 52 2023-10-19T13:04:17.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.560+0000 7f1c458bb640 1 -- 172.21.15.57:0/1806180205 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f1c2006af60 msgr2=0x7f1c2006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:17.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.560+0000 7f1c458bb640 1 --2- 172.21.15.57:0/1806180205 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f1c2006af60 0x7f1c2006d420 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f1c4015d110 tx=0x7f1c28009040 comp rx=0 tx=0).stop 2023-10-19T13:04:17.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.560+0000 7f1c458bb640 1 -- 172.21.15.57:0/1806180205 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1c4006a730 msgr2=0x7f1c4014a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:17.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.560+0000 7f1c458bb640 1 --2- 172.21.15.57:0/1806180205 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1c4006a730 0x7f1c4014a8d0 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7f1c3000c430 tx=0x7f1c3000c8f0 comp rx=0 tx=0).stop 2023-10-19T13:04:17.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.560+0000 7f1c458bb640 1 -- 172.21.15.57:0/1806180205 shutdown_connections 2023-10-19T13:04:17.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.560+0000 7f1c458bb640 1 --2- 172.21.15.57:0/1806180205 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f1c2006af60 0x7f1c2006d420 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:17.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.560+0000 7f1c458bb640 1 --2- 172.21.15.57:0/1806180205 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1c40156790 0x7f1c4014c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:17.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.560+0000 7f1c458bb640 1 --2- 172.21.15.57:0/1806180205 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1c400ac870 0x7f1c400b0750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:17.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.560+0000 7f1c458bb640 1 --2- 172.21.15.57:0/1806180205 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1c4006a730 0x7f1c4014a8d0 unknown :-1 s=CLOSED pgs=1557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:17.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.560+0000 7f1c458bb640 1 -- 172.21.15.57:0/1806180205 >> 172.21.15.57:0/1806180205 conn(0x7f1c4006fae0 msgr2=0x7f1c400aea00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:17.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.560+0000 7f1c458bb640 1 -- 172.21.15.57:0/1806180205 shutdown_connections 2023-10-19T13:04:17.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.560+0000 7f1c458bb640 1 -- 172.21.15.57:0/1806180205 wait complete. 2023-10-19T13:04:17.574 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-19T13:04:17.574 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-19T13:04:17.574 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-19T13:04:17.575 INFO:teuthology.orchestra.run:waiting for 300 2023-10-19T13:04:17.575 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-19T13:04:17.575 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-19T13:04:17.576 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-19T13:04:17.576 INFO:teuthology.orchestra.run:waiting for 300 2023-10-19T13:04:17.576 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-19T13:04:17.576 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-19T13:04:17.577 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-19T13:04:17.577 INFO:teuthology.orchestra.run:waiting for 300 2023-10-19T13:04:17.577 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-19T13:04:17.577 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-19T13:04:17.578 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-19T13:04:17.578 INFO:teuthology.orchestra.run:waiting for 300 2023-10-19T13:04:17.584 INFO:tasks.ceph.mds.a:Stopped 2023-10-19T13:04:17.584 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-19T13:04:17.585 INFO:tasks.ceph.mds.c:Stopped 2023-10-19T13:04:17.585 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-19T13:04:17.590 INFO:tasks.ceph.mds.b:Stopped 2023-10-19T13:04:17.590 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-19T13:04:17.592 INFO:tasks.ceph.mds.d:Stopped 2023-10-19T13:04:17.592 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-19T13:04:17.626 INFO:tasks.ceph.mds.a:Started 2023-10-19T13:04:17.626 DEBUG:teuthology.parallel:result is None 2023-10-19T13:04:17.627 INFO:tasks.ceph.mds.c:Started 2023-10-19T13:04:17.627 DEBUG:teuthology.parallel:result is None 2023-10-19T13:04:17.633 INFO:tasks.ceph.mds.b:Started 2023-10-19T13:04:17.634 DEBUG:teuthology.parallel:result is None 2023-10-19T13:04:17.635 INFO:tasks.ceph.mds.d:Started 2023-10-19T13:04:17.635 DEBUG:teuthology.parallel:result is None 2023-10-19T13:04:17.636 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-19T13:04:17.706 INFO:tasks.ceph.mds.c.smithi057.stdout:starting mds.c at 2023-10-19T13:04:17.706 INFO:tasks.ceph.mds.a.smithi057.stdout:starting mds.a at 2023-10-19T13:04:17.716 INFO:tasks.ceph.mds.b.smithi179.stdout:starting mds.b at 2023-10-19T13:04:17.716 INFO:tasks.ceph.mds.d.smithi179.stdout:starting mds.d at 2023-10-19T13:04:17.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.747+0000 7f44e495e640 1 -- 172.21.15.57:0/3883416459 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f44c80047c0 msgr2=0x7f44c8004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:17.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.747+0000 7f44e495e640 1 --2- 172.21.15.57:0/3883416459 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f44c80047c0 0x7f44c8004c40 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7f44d4006f20 tx=0x7f44c80057f0 comp rx=0 tx=0).stop 2023-10-19T13:04:17.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.747+0000 7f44ded9d640 1 -- 172.21.15.57:0/3883416459 reap_dead start 2023-10-19T13:04:17.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.747+0000 7f44e495e640 1 -- 172.21.15.57:0/3883416459 shutdown_connections 2023-10-19T13:04:17.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.747+0000 7f44e495e640 1 -- 172.21.15.57:0/3883416459 >> 172.21.15.57:0/3883416459 conn(0x7f44e00d0d60 msgr2=0x7f44e00d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:17.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.747+0000 7f44e495e640 1 -- 172.21.15.57:0/3883416459 shutdown_connections 2023-10-19T13:04:17.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.747+0000 7f44e495e640 1 -- 172.21.15.57:0/3883416459 wait complete. 2023-10-19T13:04:17.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.749+0000 7f44e495e640 1 Processor -- start 2023-10-19T13:04:17.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.749+0000 7f44e495e640 1 -- start start 2023-10-19T13:04:17.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.749+0000 7f44e495e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f44c80047c0 0x7f44e00c9d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:17.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.749+0000 7f44e495e640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f44e014f020 0x7f44e00b5680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:17.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.749+0000 7f44e495e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f44e00ae8e0 0x7f44e00aed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:17.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.749+0000 7f44e495e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44e00af2a0 con 0x7f44c80047c0 2023-10-19T13:04:17.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.749+0000 7f44e495e640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44e00c7e30 con 0x7f44e00ae8e0 2023-10-19T13:04:17.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.749+0000 7f44e495e640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44e00c8080 con 0x7f44e014f020 2023-10-19T13:04:17.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.749+0000 7f44de59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f44c80047c0 0x7f44e00c9d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:17.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.749+0000 7f44de59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f44c80047c0 0x7f44e00c9d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35396/0 (socket says 172.21.15.57:35396) 2023-10-19T13:04:17.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.750+0000 7f44de59c640 1 -- 172.21.15.57:0/3501482590 learned_addr learned my addr 172.21.15.57:0/3501482590 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:17.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.750+0000 7f44ddd9b640 1 --2- 172.21.15.57:0/3501482590 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f44e014f020 0x7f44e00b5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:17.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.750+0000 7f44ded9d640 1 --2- 172.21.15.57:0/3501482590 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f44e00ae8e0 0x7f44e00aed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:17.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.750+0000 7f44de59c640 1 -- 172.21.15.57:0/3501482590 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f44e014f020 msgr2=0x7f44e00b5680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:17.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.750+0000 7f44de59c640 1 --2- 172.21.15.57:0/3501482590 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f44e014f020 0x7f44e00b5680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:17.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.750+0000 7f44de59c640 1 -- 172.21.15.57:0/3501482590 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f44e00ae8e0 msgr2=0x7f44e00aed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:17.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.750+0000 7f44de59c640 1 --2- 172.21.15.57:0/3501482590 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f44e00ae8e0 0x7f44e00aed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:17.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.750+0000 7f44de59c640 1 -- 172.21.15.57:0/3501482590 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44e00c8360 con 0x7f44c80047c0 2023-10-19T13:04:17.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.750+0000 7f44ded9d640 1 --2- 172.21.15.57:0/3501482590 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f44e00ae8e0 0x7f44e00aed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T13:04:17.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.750+0000 7f44de59c640 1 --2- 172.21.15.57:0/3501482590 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f44c80047c0 0x7f44e00c9d10 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7f44d0001330 tx=0x7f44d00017f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:17.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.750+0000 7f44ceffd640 1 -- 172.21.15.57:0/3501482590 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44d00062a0 con 0x7f44c80047c0 2023-10-19T13:04:17.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.750+0000 7f44ceffd640 1 -- 172.21.15.57:0/3501482590 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f44d0011900 con 0x7f44c80047c0 2023-10-19T13:04:17.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.750+0000 7f44ceffd640 1 -- 172.21.15.57:0/3501482590 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44d0011d10 con 0x7f44c80047c0 2023-10-19T13:04:17.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.750+0000 7f44e495e640 1 -- 172.21.15.57:0/3501482590 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44e00c8640 con 0x7f44c80047c0 2023-10-19T13:04:17.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.750+0000 7f44e495e640 1 -- 172.21.15.57:0/3501482590 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44e00b5bc0 con 0x7f44c80047c0 2023-10-19T13:04:17.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.751+0000 7f44e495e640 1 -- 172.21.15.57:0/3501482590 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44e00c8b60 con 0x7f44c80047c0 2023-10-19T13:04:17.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.752+0000 7f44ceffd640 1 -- 172.21.15.57:0/3501482590 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 69) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f44d0010420 con 0x7f44c80047c0 2023-10-19T13:04:17.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.752+0000 7f44ceffd640 1 --2- 172.21.15.57:0/3501482590 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f44c006af90 0x7f44c006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:17.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.752+0000 7f44ceffd640 1 -- 172.21.15.57:0/3501482590 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 6093+0+0 (secure 0 0 0) 0x7f44d008a7a0 con 0x7f44c80047c0 2023-10-19T13:04:17.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.752+0000 7f44ddd9b640 1 --2- 172.21.15.57:0/3501482590 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f44c006af90 0x7f44c006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:17.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.753+0000 7f44ddd9b640 1 --2- 172.21.15.57:0/3501482590 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f44c006af90 0x7f44c006d450 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f44d4034000 tx=0x7f44d4039040 comp rx=0 tx=0).ready entity=mgr.7960 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:17.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.756+0000 7f44ceffd640 1 -- 172.21.15.57:0/3501482590 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f44e00c8b60 con 0x7f44c80047c0 2023-10-19T13:04:17.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:17.942+0000 7f44e495e640 1 -- 172.21.15.57:0/3501482590 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f44e00c8b60 con 0x7f44c80047c0 2023-10-19T13:04:18.343 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:04:18.346 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:04:19.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.252+0000 7f44ceffd640 1 -- 172.21.15.57:0/3501482590 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v140) v1 ==== 97+0+0 (secure 0 0 0) 0x7f44e00c8b60 con 0x7f44c80047c0 2023-10-19T13:04:19.255 INFO:teuthology.orchestra.run.smithi057.stderr: removed all blocklist entries 2023-10-19T13:04:19.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.255+0000 7f44e495e640 1 -- 172.21.15.57:0/3501482590 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f44c006af90 msgr2=0x7f44c006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:19.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.255+0000 7f44e495e640 1 --2- 172.21.15.57:0/3501482590 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f44c006af90 0x7f44c006d450 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f44d4034000 tx=0x7f44d4039040 comp rx=0 tx=0).stop 2023-10-19T13:04:19.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.255+0000 7f44e495e640 1 -- 172.21.15.57:0/3501482590 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f44c80047c0 msgr2=0x7f44e00c9d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:19.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.255+0000 7f44e495e640 1 --2- 172.21.15.57:0/3501482590 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f44c80047c0 0x7f44e00c9d10 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7f44d0001330 tx=0x7f44d00017f0 comp rx=0 tx=0).stop 2023-10-19T13:04:19.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.255+0000 7f44e495e640 1 -- 172.21.15.57:0/3501482590 shutdown_connections 2023-10-19T13:04:19.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.255+0000 7f44e495e640 1 --2- 172.21.15.57:0/3501482590 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f44c006af90 0x7f44c006d450 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:19.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.255+0000 7f44e495e640 1 --2- 172.21.15.57:0/3501482590 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f44e00ae8e0 0x7f44e00aed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:19.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.255+0000 7f44e495e640 1 --2- 172.21.15.57:0/3501482590 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f44e014f020 0x7f44e00b5680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:19.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.255+0000 7f44e495e640 1 --2- 172.21.15.57:0/3501482590 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f44c80047c0 0x7f44e00c9d10 unknown :-1 s=CLOSED pgs=1565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:19.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.255+0000 7f44e495e640 1 -- 172.21.15.57:0/3501482590 >> 172.21.15.57:0/3501482590 conn(0x7f44e00d0d60 msgr2=0x7f44e015a2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:19.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.255+0000 7f44e495e640 1 -- 172.21.15.57:0/3501482590 shutdown_connections 2023-10-19T13:04:19.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.255+0000 7f44e495e640 1 -- 172.21.15.57:0/3501482590 wait complete. 2023-10-19T13:04:19.271 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-19T13:04:19.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.383+0000 7f07d7a40640 1 -- 172.21.15.57:0/1893233584 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f07d015b2a0 msgr2=0x7f07d00cf720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:19.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.383+0000 7f07d7a40640 1 --2- 172.21.15.57:0/1893233584 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f07d015b2a0 0x7f07d00cf720 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7f07d0089ae0 tx=0x7f07c402bd10 comp rx=0 tx=0).stop 2023-10-19T13:04:19.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.383+0000 7f07d7a40640 1 -- 172.21.15.57:0/1893233584 shutdown_connections 2023-10-19T13:04:19.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.383+0000 7f07d7a40640 1 --2- 172.21.15.57:0/1893233584 >> v2:172.21.15.179:3300/0 conn(0x7f07d00ceb40 0x7f07d00b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:19.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.383+0000 7f07d7a40640 1 --2- 172.21.15.57:0/1893233584 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f07d015b2a0 0x7f07d00cf720 unknown :-1 s=CLOSED pgs=1566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:19.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.383+0000 7f07d7a40640 1 --2- 172.21.15.57:0/1893233584 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f07d00cf340 0x7f07d00d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:19.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.383+0000 7f07d7a40640 1 -- 172.21.15.57:0/1893233584 >> 172.21.15.57:0/1893233584 conn(0x7f07d00d0d60 msgr2=0x7f07d00d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:19.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.383+0000 7f07d7a40640 1 -- 172.21.15.57:0/1893233584 shutdown_connections 2023-10-19T13:04:19.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.383+0000 7f07d7a40640 1 -- 172.21.15.57:0/1893233584 wait complete. 2023-10-19T13:04:19.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.384+0000 7f07d7a40640 1 Processor -- start 2023-10-19T13:04:19.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.384+0000 7f07d7a40640 1 -- start start 2023-10-19T13:04:19.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.385+0000 7f07d7a40640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f07d00ceb40 0x7f07d014a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:19.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.385+0000 7f07d7a40640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f07d00cf340 0x7f07d006b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:19.391 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.385+0000 7f07d7a40640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f07d015b2a0 0x7f07d006b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:19.391 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.385+0000 7f07d7a40640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07d014a8f0 con 0x7f07d00ceb40 2023-10-19T13:04:19.391 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.385+0000 7f07d7a40640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07d00ca1a0 con 0x7f07d015b2a0 2023-10-19T13:04:19.391 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.385+0000 7f07d7a40640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07d00ca480 con 0x7f07d00cf340 2023-10-19T13:04:19.391 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.385+0000 7f07d57dc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f07d00ceb40 0x7f07d014a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:19.391 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.385+0000 7f07d57dc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f07d00ceb40 0x7f07d014a3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35424/0 (socket says 172.21.15.57:35424) 2023-10-19T13:04:19.392 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.385+0000 7f07d57dc640 1 -- 172.21.15.57:0/3592148048 learned_addr learned my addr 172.21.15.57:0/3592148048 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:19.392 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.386+0000 7f07d57dc640 1 -- 172.21.15.57:0/3592148048 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f07d00cf340 msgr2=0x7f07d006b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:19.392 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.386+0000 7f07d5fdd640 1 --2- 172.21.15.57:0/3592148048 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f07d015b2a0 0x7f07d006b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:19.392 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.386+0000 7f07d4fdb640 1 --2- 172.21.15.57:0/3592148048 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f07d00cf340 0x7f07d006b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:19.392 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.386+0000 7f07d57dc640 1 --2- 172.21.15.57:0/3592148048 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f07d00cf340 0x7f07d006b090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:19.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.386+0000 7f07d57dc640 1 -- 172.21.15.57:0/3592148048 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f07d015b2a0 msgr2=0x7f07d006b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:19.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.386+0000 7f07d57dc640 1 --2- 172.21.15.57:0/3592148048 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f07d015b2a0 0x7f07d006b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:19.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.386+0000 7f07d57dc640 1 -- 172.21.15.57:0/3592148048 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07c4009560 con 0x7f07d00ceb40 2023-10-19T13:04:19.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.386+0000 7f07d57dc640 1 --2- 172.21.15.57:0/3592148048 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f07d00ceb40 0x7f07d014a3b0 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f07c800d770 tx=0x7f07c800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:19.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.386+0000 7f07cdffb640 1 -- 172.21.15.57:0/3592148048 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07c8009c30 con 0x7f07d00ceb40 2023-10-19T13:04:19.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.386+0000 7f07cdffb640 1 -- 172.21.15.57:0/3592148048 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f07c800de80 con 0x7f07d00ceb40 2023-10-19T13:04:19.394 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.386+0000 7f07cdffb640 1 -- 172.21.15.57:0/3592148048 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07c8005280 con 0x7f07d00ceb40 2023-10-19T13:04:19.394 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.386+0000 7f07d7a40640 1 -- 172.21.15.57:0/3592148048 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07d00ca760 con 0x7f07d00ceb40 2023-10-19T13:04:19.394 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.386+0000 7f07d7a40640 1 -- 172.21.15.57:0/3592148048 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07d00cad20 con 0x7f07d00ceb40 2023-10-19T13:04:19.394 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.387+0000 7f07d7a40640 1 -- 172.21.15.57:0/3592148048 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07d00690e0 con 0x7f07d00ceb40 2023-10-19T13:04:19.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.388+0000 7f07cdffb640 1 -- 172.21.15.57:0/3592148048 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 69) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f07c80053e0 con 0x7f07d00ceb40 2023-10-19T13:04:19.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.388+0000 7f07cdffb640 1 --2- 172.21.15.57:0/3592148048 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f07bc06af90 0x7f07bc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:19.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.388+0000 7f07cdffb640 1 -- 172.21.15.57:0/3592148048 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f07c80898c0 con 0x7f07d00ceb40 2023-10-19T13:04:19.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.388+0000 7f07d4fdb640 1 --2- 172.21.15.57:0/3592148048 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f07bc06af90 0x7f07bc06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:19.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.389+0000 7f07d4fdb640 1 --2- 172.21.15.57:0/3592148048 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f07bc06af90 0x7f07bc06d450 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f07c4035000 tx=0x7f07c4004730 comp rx=0 tx=0).ready entity=mgr.7960 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:19.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.392+0000 7f07cdffb640 1 -- 172.21.15.57:0/3592148048 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f07c8054940 con 0x7f07d00ceb40 2023-10-19T13:04:19.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.634+0000 7f07d7a40640 1 -- 172.21.15.57:0/3592148048 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f07d00d0b40 con 0x7f07d00ceb40 2023-10-19T13:04:19.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.635+0000 7f07cdffb640 1 -- 172.21.15.57:0/3592148048 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v28) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f07c8017020 con 0x7f07d00ceb40 2023-10-19T13:04:19.638 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:04:19.638 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2023-10-19T13:04:19.638 INFO:teuthology.orchestra.run.smithi057.stdout: "auth_dump": [ 2023-10-19T13:04:19.639 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:04:19.639 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "mds.a", 2023-10-19T13:04:19.639 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBjJDFlmbVANhAAP/ynHlU/OxaRq5lQoAa8Ng==", 2023-10-19T13:04:19.639 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:04:19.639 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T13:04:19.640 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow *", 2023-10-19T13:04:19.640 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow *", 2023-10-19T13:04:19.640 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:04:19.640 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:04:19.640 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:04:19.641 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:04:19.641 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "mds.b", 2023-10-19T13:04:19.641 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFl9l7RBxAAC0geI2tJ1SIZZr1++kS/BA==", 2023-10-19T13:04:19.641 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:04:19.641 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T13:04:19.641 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow *", 2023-10-19T13:04:19.642 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow *", 2023-10-19T13:04:19.642 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:04:19.642 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:04:19.642 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:04:19.642 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:04:19.643 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "mds.c", 2023-10-19T13:04:19.643 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFlFkaoAhAA/PBizkJ4bm958oP4CmPuIQ==", 2023-10-19T13:04:19.643 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:04:19.643 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T13:04:19.643 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow *", 2023-10-19T13:04:19.643 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow *", 2023-10-19T13:04:19.644 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:04:19.644 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:04:19.644 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:04:19.644 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:04:19.644 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "mds.d", 2023-10-19T13:04:19.644 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFlx8skDxAAAfCHTYtYJgJik0WHSWeQfQ==", 2023-10-19T13:04:19.645 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:04:19.645 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T13:04:19.645 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow *", 2023-10-19T13:04:19.645 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow *", 2023-10-19T13:04:19.645 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:04:19.646 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:04:19.646 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:04:19.646 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:04:19.646 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.0", 2023-10-19T13:04:19.646 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBrJDFlCo9sBxAAQN2cID4nCQgRwGaGFEV8LQ==", 2023-10-19T13:04:19.647 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:04:19.647 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T13:04:19.647 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T13:04:19.647 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:04:19.647 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:04:19.648 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:04:19.648 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:04:19.648 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.1", 2023-10-19T13:04:19.648 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBuJDFl4fsLCBAAp3Pi80yZtc5sju0GisEuVA==", 2023-10-19T13:04:19.648 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:04:19.649 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T13:04:19.649 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T13:04:19.649 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:04:19.649 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:04:19.649 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:04:19.650 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:04:19.650 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.2", 2023-10-19T13:04:19.650 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBxJDFl+KAKCBAAd4N9cluARxPVfBLU+7Hd7A==", 2023-10-19T13:04:19.650 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:04:19.650 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T13:04:19.650 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T13:04:19.651 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:04:19.651 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:04:19.651 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:04:19.651 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:04:19.651 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.3", 2023-10-19T13:04:19.651 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQB0JDFlU/toCBAAP0Ou1zcHKNx5SfbybJkh4Q==", 2023-10-19T13:04:19.652 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:04:19.652 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T13:04:19.652 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T13:04:19.652 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:04:19.653 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:04:19.653 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:04:19.653 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:04:19.653 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.4", 2023-10-19T13:04:19.653 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQB6JDFlc6stEBAALkgNyxoiCTzZnKpBSPxI6w==", 2023-10-19T13:04:19.654 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:04:19.654 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T13:04:19.654 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T13:04:19.654 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:04:19.654 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:04:19.654 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:04:19.655 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:04:19.655 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.5", 2023-10-19T13:04:19.655 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQB9JDFlI3cNDhAAuG3tHJ6RvPwAzPE76wTIVQ==", 2023-10-19T13:04:19.655 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:04:19.655 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T13:04:19.656 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T13:04:19.656 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:04:19.656 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:04:19.656 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:04:19.656 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:04:19.656 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.6", 2023-10-19T13:04:19.657 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQCAJDFlLPumCxAA/iy08t/2N9rXfn71KtbSQw==", 2023-10-19T13:04:19.657 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:04:19.657 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T13:04:19.657 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T13:04:19.657 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:04:19.658 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:04:19.658 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:04:19.658 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:04:19.658 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.7", 2023-10-19T13:04:19.659 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQCDJDFlHmrbCRAA+hpWygGIJydfU6cHWlfx/Q==", 2023-10-19T13:04:19.659 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:04:19.659 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T13:04:19.659 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T13:04:19.659 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:04:19.659 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:04:19.660 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:04:19.660 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:04:19.660 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "client.0", 2023-10-19T13:04:19.660 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFlg9yuExAATxiWDfS+p07G1F0pORbJEg==", 2023-10-19T13:04:19.660 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:04:19.661 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T13:04:19.661 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow r", 2023-10-19T13:04:19.661 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-19T13:04:19.661 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:04:19.661 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:04:19.662 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:04:19.662 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "client.1", 2023-10-19T13:04:19.662 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFl9IwYGxAAHPituiJSPkHDuD0QyG4Hww==", 2023-10-19T13:04:19.662 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:04:19.662 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T13:04:19.663 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow r", 2023-10-19T13:04:19.663 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow rw", 2023-10-19T13:04:19.663 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow rwx" 2023-10-19T13:04:19.663 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:04:19.663 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:04:19.664 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:04:19.664 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "client.2", 2023-10-19T13:04:19.664 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFlMd+AFxAAS/UTESDt76GmrhhN2o/Fdw==", 2023-10-19T13:04:19.664 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:04:19.664 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T13:04:19.665 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow r", 2023-10-19T13:04:19.665 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow rw", 2023-10-19T13:04:19.665 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow rwx" 2023-10-19T13:04:19.665 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:04:19.665 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:04:19.666 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:04:19.666 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "client.3", 2023-10-19T13:04:19.666 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFlHZ3fIBAA42B7n8opLS5smXIwJyVYzw==", 2023-10-19T13:04:19.666 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:04:19.666 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T13:04:19.666 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow r", 2023-10-19T13:04:19.667 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow rw", 2023-10-19T13:04:19.667 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow rwx" 2023-10-19T13:04:19.667 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:04:19.667 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:04:19.668 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:04:19.668 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "client.admin", 2023-10-19T13:04:19.668 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBjJDFlBgasFxAAMA24j8KszcXMEpYRQDXj4g==", 2023-10-19T13:04:19.668 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:04:19.668 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow *", 2023-10-19T13:04:19.669 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow *", 2023-10-19T13:04:19.669 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow *", 2023-10-19T13:04:19.669 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:04:19.669 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:04:19.669 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:04:19.670 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:04:19.670 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "mgr.x", 2023-10-19T13:04:19.670 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBjJDFllFBfMhAA5+KPXWmYZqcESmLObaUbBA==", 2023-10-19T13:04:19.670 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:04:19.670 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow *", 2023-10-19T13:04:19.671 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile mgr", 2023-10-19T13:04:19.671 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:04:19.671 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:04:19.671 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:04:19.671 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:04:19.672 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "mgr.y", 2023-10-19T13:04:19.672 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBjJDFlxRzJLRAAGjG58o7qXAHvIEIRYwl6QA==", 2023-10-19T13:04:19.672 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:04:19.672 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow *", 2023-10-19T13:04:19.672 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile mgr", 2023-10-19T13:04:19.672 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:04:19.673 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:04:19.673 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:04:19.673 INFO:teuthology.orchestra.run.smithi057.stdout: ] 2023-10-19T13:04:19.673 INFO:teuthology.orchestra.run.smithi057.stdout:} 2023-10-19T13:04:19.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.636+0000 7f07d7a40640 1 -- 172.21.15.57:0/3592148048 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f07bc06af90 msgr2=0x7f07bc06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:19.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.636+0000 7f07d7a40640 1 --2- 172.21.15.57:0/3592148048 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f07bc06af90 0x7f07bc06d450 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f07c4035000 tx=0x7f07c4004730 comp rx=0 tx=0).stop 2023-10-19T13:04:19.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.636+0000 7f07d7a40640 1 -- 172.21.15.57:0/3592148048 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f07d00ceb40 msgr2=0x7f07d014a3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:19.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.636+0000 7f07d7a40640 1 --2- 172.21.15.57:0/3592148048 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f07d00ceb40 0x7f07d014a3b0 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f07c800d770 tx=0x7f07c800dc30 comp rx=0 tx=0).stop 2023-10-19T13:04:19.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.636+0000 7f07d7a40640 1 -- 172.21.15.57:0/3592148048 shutdown_connections 2023-10-19T13:04:19.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.636+0000 7f07d7a40640 1 --2- 172.21.15.57:0/3592148048 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f07bc06af90 0x7f07bc06d450 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:19.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.636+0000 7f07d7a40640 1 --2- 172.21.15.57:0/3592148048 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f07d015b2a0 0x7f07d006b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:19.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.636+0000 7f07d7a40640 1 --2- 172.21.15.57:0/3592148048 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f07d00cf340 0x7f07d006b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:19.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.636+0000 7f07d7a40640 1 --2- 172.21.15.57:0/3592148048 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f07d00ceb40 0x7f07d014a3b0 unknown :-1 s=CLOSED pgs=1567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:19.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.636+0000 7f07d7a40640 1 -- 172.21.15.57:0/3592148048 >> 172.21.15.57:0/3592148048 conn(0x7f07d00d0d60 msgr2=0x7f07d00ae050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:19.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.636+0000 7f07d7a40640 1 -- 172.21.15.57:0/3592148048 shutdown_connections 2023-10-19T13:04:19.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.637+0000 7f07d7a40640 1 -- 172.21.15.57:0/3592148048 wait complete. 2023-10-19T13:04:19.678 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T13:04:19.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.785+0000 7f3782f9e640 1 -- 172.21.15.57:0/2192619841 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f377c158b40 msgr2=0x7f377c0ae550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:19.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.785+0000 7f3782f9e640 1 --2- 172.21.15.57:0/2192619841 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f377c158b40 0x7f377c0ae550 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f3778009900 tx=0x7f377802be10 comp rx=0 tx=0).stop 2023-10-19T13:04:19.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.785+0000 7f3782f9e640 1 -- 172.21.15.57:0/2192619841 shutdown_connections 2023-10-19T13:04:19.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.785+0000 7f3782f9e640 1 --2- 172.21.15.57:0/2192619841 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f377c158b40 0x7f377c0ae550 unknown :-1 s=CLOSED pgs=1569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:19.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.785+0000 7f3782f9e640 1 -- 172.21.15.57:0/2192619841 >> 172.21.15.57:0/2192619841 conn(0x7f377c071650 msgr2=0x7f377c097a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:19.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.785+0000 7f3782f9e640 1 -- 172.21.15.57:0/2192619841 shutdown_connections 2023-10-19T13:04:19.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.785+0000 7f3782f9e640 1 -- 172.21.15.57:0/2192619841 wait complete. 2023-10-19T13:04:19.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.786+0000 7f3782f9e640 1 Processor -- start 2023-10-19T13:04:19.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.786+0000 7f3782f9e640 1 -- start start 2023-10-19T13:04:19.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.787+0000 7f3782f9e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f377c0b01b0 0x7f377c150630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:19.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.787+0000 7f3782f9e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f377c0b2be0 0x7f377c140f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:19.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.787+0000 7f3782f9e640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f377c158b40 0x7f377c141490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:19.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.787+0000 7f3782f9e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f377c1419d0 con 0x7f377c0b01b0 2023-10-19T13:04:19.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.787+0000 7f3782f9e640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f377c151290 con 0x7f377c0b2be0 2023-10-19T13:04:19.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.787+0000 7f3782f9e640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f377c155e60 con 0x7f377c158b40 2023-10-19T13:04:19.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.787+0000 7f3781f9c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f377c0b01b0 0x7f377c150630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:19.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.787+0000 7f3781f9c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f377c0b01b0 0x7f377c150630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35450/0 (socket says 172.21.15.57:35450) 2023-10-19T13:04:19.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.787+0000 7f3781f9c640 1 -- 172.21.15.57:0/211245781 learned_addr learned my addr 172.21.15.57:0/211245781 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:19.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.787+0000 7f378279d640 1 --2- 172.21.15.57:0/211245781 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f377c158b40 0x7f377c141490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:19.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.787+0000 7f378179b640 1 --2- 172.21.15.57:0/211245781 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f377c0b2be0 0x7f377c140f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:19.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.787+0000 7f3781f9c640 1 -- 172.21.15.57:0/211245781 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f377c158b40 msgr2=0x7f377c141490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:19.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.787+0000 7f3781f9c640 1 --2- 172.21.15.57:0/211245781 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f377c158b40 0x7f377c141490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:19.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.787+0000 7f3781f9c640 1 -- 172.21.15.57:0/211245781 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f377c0b2be0 msgr2=0x7f377c140f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:19.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.787+0000 7f3781f9c640 1 --2- 172.21.15.57:0/211245781 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f377c0b2be0 0x7f377c140f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:19.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.787+0000 7f3781f9c640 1 -- 172.21.15.57:0/211245781 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3778009560 con 0x7f377c0b01b0 2023-10-19T13:04:19.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.787+0000 7f3781f9c640 1 --2- 172.21.15.57:0/211245781 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f377c0b01b0 0x7f377c150630 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7f3778000c00 tx=0x7f377802ddc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:19.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.787+0000 7f37727fc640 1 -- 172.21.15.57:0/211245781 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3778005690 con 0x7f377c0b01b0 2023-10-19T13:04:19.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.788+0000 7f37727fc640 1 -- 172.21.15.57:0/211245781 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f37780038f0 con 0x7f377c0b01b0 2023-10-19T13:04:19.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.788+0000 7f3782f9e640 1 -- 172.21.15.57:0/211245781 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f377c156140 con 0x7f377c0b01b0 2023-10-19T13:04:19.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.788+0000 7f37727fc640 1 -- 172.21.15.57:0/211245781 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3778003ac0 con 0x7f377c0b01b0 2023-10-19T13:04:19.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.788+0000 7f3782f9e640 1 -- 172.21.15.57:0/211245781 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f377c156700 con 0x7f377c0b01b0 2023-10-19T13:04:19.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.789+0000 7f3782f9e640 1 -- 172.21.15.57:0/211245781 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f377c04da70 con 0x7f377c0b01b0 2023-10-19T13:04:19.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.789+0000 7f37727fc640 1 -- 172.21.15.57:0/211245781 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 69) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f3778003c20 con 0x7f377c0b01b0 2023-10-19T13:04:19.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.789+0000 7f37727fc640 1 --2- 172.21.15.57:0/211245781 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f376006af60 0x7f376006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:19.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.789+0000 7f37727fc640 1 -- 172.21.15.57:0/211245781 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f37780abed0 con 0x7f377c0b01b0 2023-10-19T13:04:19.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.790+0000 7f378179b640 1 --2- 172.21.15.57:0/211245781 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f376006af60 0x7f376006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:19.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.790+0000 7f378179b640 1 --2- 172.21.15.57:0/211245781 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f376006af60 0x7f376006d420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f377c14ac70 tx=0x7f376c009040 comp rx=0 tx=0).ready entity=mgr.7960 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:19.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.793+0000 7f37727fc640 1 -- 172.21.15.57:0/211245781 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f37780770f0 con 0x7f377c0b01b0 2023-10-19T13:04:19.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.969+0000 7f3782f9e640 1 -- 172.21.15.57:0/211245781 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f377c158240 con 0x7f377c0b01b0 2023-10-19T13:04:19.973 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.970+0000 7f37727fc640 1 -- 172.21.15.57:0/211245781 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v140) v1 ==== 74+0+13127 (secure 0 0 0) 0x7f377807a710 con 0x7f377c0b01b0 2023-10-19T13:04:19.974 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:04:19.974 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":140,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T13:04:19.251379+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T13:04:19.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.972+0000 7f3782f9e640 1 -- 172.21.15.57:0/211245781 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f376006af60 msgr2=0x7f376006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:19.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.972+0000 7f3782f9e640 1 --2- 172.21.15.57:0/211245781 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f376006af60 0x7f376006d420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f377c14ac70 tx=0x7f376c009040 comp rx=0 tx=0).stop 2023-10-19T13:04:19.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.972+0000 7f3782f9e640 1 -- 172.21.15.57:0/211245781 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f377c0b01b0 msgr2=0x7f377c150630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:19.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.972+0000 7f3782f9e640 1 --2- 172.21.15.57:0/211245781 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f377c0b01b0 0x7f377c150630 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7f3778000c00 tx=0x7f377802ddc0 comp rx=0 tx=0).stop 2023-10-19T13:04:19.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.972+0000 7f3782f9e640 1 -- 172.21.15.57:0/211245781 shutdown_connections 2023-10-19T13:04:19.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.972+0000 7f3782f9e640 1 --2- 172.21.15.57:0/211245781 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f376006af60 0x7f376006d420 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:19.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.972+0000 7f3782f9e640 1 --2- 172.21.15.57:0/211245781 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f377c158b40 0x7f377c141490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:19.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.972+0000 7f3782f9e640 1 --2- 172.21.15.57:0/211245781 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f377c0b2be0 0x7f377c140f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:19.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.972+0000 7f3782f9e640 1 --2- 172.21.15.57:0/211245781 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f377c0b01b0 0x7f377c150630 unknown :-1 s=CLOSED pgs=1570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:19.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.972+0000 7f3782f9e640 1 -- 172.21.15.57:0/211245781 >> 172.21.15.57:0/211245781 conn(0x7f377c071650 msgr2=0x7f377c15b9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:19.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.972+0000 7f3782f9e640 1 -- 172.21.15.57:0/211245781 shutdown_connections 2023-10-19T13:04:19.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:19.972+0000 7f3782f9e640 1 -- 172.21.15.57:0/211245781 wait complete. 2023-10-19T13:04:19.985 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-19T12:44:07.371186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-19T13:04:19.986 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-19T13:04:20.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.095+0000 7fc44c3b0640 1 -- 172.21.15.57:0/3305630855 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4441579f0 msgr2=0x7fc4440b0e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:20.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.095+0000 7fc44c3b0640 1 --2- 172.21.15.57:0/3305630855 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4441579f0 0x7fc4440b0e90 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7fc44406b930 tx=0x7fc44002bda0 comp rx=0 tx=0).stop 2023-10-19T13:04:20.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.096+0000 7fc44c3b0640 1 -- 172.21.15.57:0/3305630855 shutdown_connections 2023-10-19T13:04:20.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.096+0000 7fc44c3b0640 1 --2- 172.21.15.57:0/3305630855 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc44406ac70 0x7fc44406ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:20.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.096+0000 7fc44c3b0640 1 --2- 172.21.15.57:0/3305630855 >> v2:172.21.15.179:3300/0 conn(0x7fc44415c1f0 0x7fc44406a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:20.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.096+0000 7fc44c3b0640 1 --2- 172.21.15.57:0/3305630855 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4441579f0 0x7fc4440b0e90 unknown :-1 s=CLOSED pgs=1571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:20.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.096+0000 7fc44c3b0640 1 -- 172.21.15.57:0/3305630855 >> 172.21.15.57:0/3305630855 conn(0x7fc44406fae0 msgr2=0x7fc44406fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:20.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.096+0000 7fc44c3b0640 1 -- 172.21.15.57:0/3305630855 shutdown_connections 2023-10-19T13:04:20.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.096+0000 7fc44c3b0640 1 -- 172.21.15.57:0/3305630855 wait complete. 2023-10-19T13:04:20.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.097+0000 7fc44c3b0640 1 Processor -- start 2023-10-19T13:04:20.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.097+0000 7fc44c3b0640 1 -- start start 2023-10-19T13:04:20.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.097+0000 7fc44c3b0640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc44406ac70 0x7fc4441448e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:20.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.097+0000 7fc44c3b0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc4441579f0 0x7fc44414ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:20.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.097+0000 7fc44c3b0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc44415c1f0 0x7fc444149a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:20.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.097+0000 7fc44c3b0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4440ad8c0 con 0x7fc44415c1f0 2023-10-19T13:04:20.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.097+0000 7fc44c3b0640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4440adba0 con 0x7fc4441579f0 2023-10-19T13:04:20.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.097+0000 7fc44c3b0640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4440ade80 con 0x7fc44406ac70 2023-10-19T13:04:20.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.098+0000 7fc44a94d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc44415c1f0 0x7fc444149a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:20.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.098+0000 7fc44a94d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc44415c1f0 0x7fc444149a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35458/0 (socket says 172.21.15.57:35458) 2023-10-19T13:04:20.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.098+0000 7fc44a94d640 1 -- 172.21.15.57:0/2111625399 learned_addr learned my addr 172.21.15.57:0/2111625399 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:20.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.098+0000 7fc44a14c640 1 --2- 172.21.15.57:0/2111625399 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc44406ac70 0x7fc4441448e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:20.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.098+0000 7fc44994b640 1 --2- 172.21.15.57:0/2111625399 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc4441579f0 0x7fc44414ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:20.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.098+0000 7fc44a94d640 1 -- 172.21.15.57:0/2111625399 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc44406ac70 msgr2=0x7fc4441448e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:20.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.098+0000 7fc44a94d640 1 --2- 172.21.15.57:0/2111625399 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc44406ac70 0x7fc4441448e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:20.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.098+0000 7fc44a94d640 1 -- 172.21.15.57:0/2111625399 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc4441579f0 msgr2=0x7fc44414ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:20.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.098+0000 7fc44a94d640 1 --2- 172.21.15.57:0/2111625399 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc4441579f0 0x7fc44414ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:20.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.098+0000 7fc44a94d640 1 -- 172.21.15.57:0/2111625399 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc440009560 con 0x7fc44415c1f0 2023-10-19T13:04:20.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.098+0000 7fc44994b640 1 --2- 172.21.15.57:0/2111625399 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc4441579f0 0x7fc44414ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:04:20.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.098+0000 7fc44a94d640 1 --2- 172.21.15.57:0/2111625399 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc44415c1f0 0x7fc444149a10 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7fc43c00b970 tx=0x7fc43c00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:20.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.098+0000 7fc43affd640 1 -- 172.21.15.57:0/2111625399 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc43c00cb40 con 0x7fc44415c1f0 2023-10-19T13:04:20.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.098+0000 7fc43affd640 1 -- 172.21.15.57:0/2111625399 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fc43c00cdb0 con 0x7fc44415c1f0 2023-10-19T13:04:20.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.098+0000 7fc44c3b0640 1 -- 172.21.15.57:0/2111625399 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4440ae160 con 0x7fc44415c1f0 2023-10-19T13:04:20.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.098+0000 7fc43affd640 1 -- 172.21.15.57:0/2111625399 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc43c0146a0 con 0x7fc44415c1f0 2023-10-19T13:04:20.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.098+0000 7fc44c3b0640 1 -- 172.21.15.57:0/2111625399 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc444144e20 con 0x7fc44415c1f0 2023-10-19T13:04:20.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.099+0000 7fc44c3b0640 1 -- 172.21.15.57:0/2111625399 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4440ae680 con 0x7fc44415c1f0 2023-10-19T13:04:20.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.100+0000 7fc43affd640 1 -- 172.21.15.57:0/2111625399 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 69) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc43c01d3b0 con 0x7fc44415c1f0 2023-10-19T13:04:20.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.100+0000 7fc43affd640 1 --2- 172.21.15.57:0/2111625399 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7fc42806af90 0x7fc42806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:20.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.100+0000 7fc43affd640 1 -- 172.21.15.57:0/2111625399 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5205+0+0 (secure 0 0 0) 0x7fc43c089600 con 0x7fc44415c1f0 2023-10-19T13:04:20.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.100+0000 7fc44a14c640 1 --2- 172.21.15.57:0/2111625399 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7fc42806af90 0x7fc42806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:20.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.101+0000 7fc44a14c640 1 --2- 172.21.15.57:0/2111625399 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7fc42806af90 0x7fc42806d450 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc440035000 tx=0x7fc440004730 comp rx=0 tx=0).ready entity=mgr.7960 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:20.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.104+0000 7fc43affd640 1 -- 172.21.15.57:0/2111625399 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc4440ae680 con 0x7fc44415c1f0 2023-10-19T13:04:20.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.278+0000 7fc44c3b0640 1 -- 172.21.15.57:0/2111625399 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc4440ae680 con 0x7fc44415c1f0 2023-10-19T13:04:20.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.278+0000 7fc43affd640 1 -- 172.21.15.57:0/2111625399 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v140) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc4440ae680 con 0x7fc44415c1f0 2023-10-19T13:04:20.282 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-19T13:04:20.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.280+0000 7fc44c3b0640 1 -- 172.21.15.57:0/2111625399 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7fc42806af90 msgr2=0x7fc42806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:20.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.280+0000 7fc44c3b0640 1 --2- 172.21.15.57:0/2111625399 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7fc42806af90 0x7fc42806d450 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc440035000 tx=0x7fc440004730 comp rx=0 tx=0).stop 2023-10-19T13:04:20.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.280+0000 7fc44c3b0640 1 -- 172.21.15.57:0/2111625399 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc44415c1f0 msgr2=0x7fc444149a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:20.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.280+0000 7fc44c3b0640 1 --2- 172.21.15.57:0/2111625399 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc44415c1f0 0x7fc444149a10 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7fc43c00b970 tx=0x7fc43c00be30 comp rx=0 tx=0).stop 2023-10-19T13:04:20.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.280+0000 7fc44c3b0640 1 -- 172.21.15.57:0/2111625399 shutdown_connections 2023-10-19T13:04:20.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.280+0000 7fc44c3b0640 1 --2- 172.21.15.57:0/2111625399 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7fc42806af90 0x7fc42806d450 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:20.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.280+0000 7fc44c3b0640 1 --2- 172.21.15.57:0/2111625399 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc44415c1f0 0x7fc444149a10 unknown :-1 s=CLOSED pgs=1572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:20.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.280+0000 7fc44c3b0640 1 --2- 172.21.15.57:0/2111625399 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc4441579f0 0x7fc44414ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:20.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.280+0000 7fc44c3b0640 1 --2- 172.21.15.57:0/2111625399 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc44406ac70 0x7fc4441448e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:20.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.280+0000 7fc44c3b0640 1 -- 172.21.15.57:0/2111625399 >> 172.21.15.57:0/2111625399 conn(0x7fc44406fae0 msgr2=0x7fc44414ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:20.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.280+0000 7fc44c3b0640 1 -- 172.21.15.57:0/2111625399 shutdown_connections 2023-10-19T13:04:20.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.280+0000 7fc44c3b0640 1 -- 172.21.15.57:0/2111625399 wait complete. 2023-10-19T13:04:20.294 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-19T13:04:20.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.404+0000 7ff75f3f5640 1 -- 172.21.15.57:0/2793800491 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff740004d00 msgr2=0x7ff740005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:20.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.404+0000 7ff75f3f5640 1 --2- 172.21.15.57:0/2793800491 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff740004d00 0x7ff740005180 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7ff74800c640 tx=0x7ff7400057f0 comp rx=0 tx=0).stop 2023-10-19T13:04:20.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.404+0000 7ff75d992640 1 -- 172.21.15.57:0/2793800491 reap_dead start 2023-10-19T13:04:20.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.404+0000 7ff75f3f5640 1 -- 172.21.15.57:0/2793800491 shutdown_connections 2023-10-19T13:04:20.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.404+0000 7ff75f3f5640 1 -- 172.21.15.57:0/2793800491 >> 172.21.15.57:0/2793800491 conn(0x7ff7580af830 msgr2=0x7ff7580afc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:20.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.404+0000 7ff75f3f5640 1 -- 172.21.15.57:0/2793800491 shutdown_connections 2023-10-19T13:04:20.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.404+0000 7ff75f3f5640 1 -- 172.21.15.57:0/2793800491 wait complete. 2023-10-19T13:04:20.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.405+0000 7ff75f3f5640 1 Processor -- start 2023-10-19T13:04:20.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.405+0000 7ff75f3f5640 1 -- start start 2023-10-19T13:04:20.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.406+0000 7ff75f3f5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff758141390 0x7ff7581ddf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:20.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.406+0000 7ff75f3f5640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff758141770 0x7ff7581f4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:20.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.406+0000 7ff75f3f5640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff758156cd0 0x7ff758157150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:20.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.406+0000 7ff75f3f5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff758157690 con 0x7ff758141390 2023-10-19T13:04:20.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.406+0000 7ff75f3f5640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7581f52c0 con 0x7ff758156cd0 2023-10-19T13:04:20.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.406+0000 7ff75f3f5640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7581f55a0 con 0x7ff758141770 2023-10-19T13:04:20.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.406+0000 7ff75d191640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff758141390 0x7ff7581ddf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:20.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.406+0000 7ff75d191640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff758141390 0x7ff7581ddf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35480/0 (socket says 172.21.15.57:35480) 2023-10-19T13:04:20.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.406+0000 7ff75d191640 1 -- 172.21.15.57:0/1373286564 learned_addr learned my addr 172.21.15.57:0/1373286564 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:20.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.406+0000 7ff75d992640 1 --2- 172.21.15.57:0/1373286564 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff758156cd0 0x7ff758157150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:20.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.406+0000 7ff75c990640 1 --2- 172.21.15.57:0/1373286564 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff758141770 0x7ff7581f4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:20.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.406+0000 7ff75d191640 1 -- 172.21.15.57:0/1373286564 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff758141770 msgr2=0x7ff7581f4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:20.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.406+0000 7ff75d191640 1 --2- 172.21.15.57:0/1373286564 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff758141770 0x7ff7581f4bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:20.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.406+0000 7ff75d191640 1 -- 172.21.15.57:0/1373286564 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff758156cd0 msgr2=0x7ff758157150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:20.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.406+0000 7ff75d191640 1 --2- 172.21.15.57:0/1373286564 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff758156cd0 0x7ff758157150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:20.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.406+0000 7ff75d191640 1 -- 172.21.15.57:0/1373286564 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7581f5880 con 0x7ff758141390 2023-10-19T13:04:20.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.406+0000 7ff75d191640 1 --2- 172.21.15.57:0/1373286564 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff758141390 0x7ff7581ddf50 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7ff750011ab0 tx=0x7ff750011eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:20.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.407+0000 7ff755ffb640 1 -- 172.21.15.57:0/1373286564 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff750004460 con 0x7ff758141390 2023-10-19T13:04:20.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.407+0000 7ff755ffb640 1 -- 172.21.15.57:0/1373286564 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7ff75000f9a0 con 0x7ff758141390 2023-10-19T13:04:20.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.407+0000 7ff75f3f5640 1 -- 172.21.15.57:0/1373286564 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7580c2cf0 con 0x7ff758141390 2023-10-19T13:04:20.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.407+0000 7ff755ffb640 1 -- 172.21.15.57:0/1373286564 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff75000fd90 con 0x7ff758141390 2023-10-19T13:04:20.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.407+0000 7ff75f3f5640 1 -- 172.21.15.57:0/1373286564 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff758069900 con 0x7ff758141390 2023-10-19T13:04:20.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.408+0000 7ff75f3f5640 1 -- 172.21.15.57:0/1373286564 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7580c2b50 con 0x7ff758141390 2023-10-19T13:04:20.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.408+0000 7ff755ffb640 1 -- 172.21.15.57:0/1373286564 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 69) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff7500035f0 con 0x7ff758141390 2023-10-19T13:04:20.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.409+0000 7ff755ffb640 1 --2- 172.21.15.57:0/1373286564 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7ff74406af60 0x7ff74406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:20.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.409+0000 7ff755ffb640 1 -- 172.21.15.57:0/1373286564 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5205+0+0 (secure 0 0 0) 0x7ff75008a370 con 0x7ff758141390 2023-10-19T13:04:20.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.409+0000 7ff75c990640 1 --2- 172.21.15.57:0/1373286564 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7ff74406af60 0x7ff74406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:20.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.410+0000 7ff75c990640 1 --2- 172.21.15.57:0/1373286564 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7ff74406af60 0x7ff74406d420 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7ff7580c9290 tx=0x7ff74c005f90 comp rx=0 tx=0).ready entity=mgr.7960 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:20.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.413+0000 7ff755ffb640 1 -- 172.21.15.57:0/1373286564 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff7580c2b50 con 0x7ff758141390 2023-10-19T13:04:20.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.590+0000 7ff75f3f5640 1 -- 172.21.15.57:0/1373286564 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7ff7580c2b50 con 0x7ff758141390 2023-10-19T13:04:20.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.590+0000 7ff755ffb640 1 -- 172.21.15.57:0/1373286564 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v140) v1 ==== 84+0+65 (secure 0 0 0) 0x7ff7580c2b50 con 0x7ff758141390 2023-10-19T13:04:20.593 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:04:20.594 INFO:teuthology.orchestra.run.smithi057.stdout:[ 2023-10-19T13:04:20.594 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:04:20.594 INFO:teuthology.orchestra.run.smithi057.stdout: "poolnum": 1, 2023-10-19T13:04:20.594 INFO:teuthology.orchestra.run.smithi057.stdout: "poolname": ".mgr" 2023-10-19T13:04:20.594 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:04:20.595 INFO:teuthology.orchestra.run.smithi057.stdout:] 2023-10-19T13:04:20.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.592+0000 7ff75f3f5640 1 -- 172.21.15.57:0/1373286564 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7ff74406af60 msgr2=0x7ff74406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:20.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.592+0000 7ff75f3f5640 1 --2- 172.21.15.57:0/1373286564 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7ff74406af60 0x7ff74406d420 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7ff7580c9290 tx=0x7ff74c005f90 comp rx=0 tx=0).stop 2023-10-19T13:04:20.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.592+0000 7ff75f3f5640 1 -- 172.21.15.57:0/1373286564 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff758141390 msgr2=0x7ff7581ddf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:20.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.592+0000 7ff75f3f5640 1 --2- 172.21.15.57:0/1373286564 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff758141390 0x7ff7581ddf50 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7ff750011ab0 tx=0x7ff750011eb0 comp rx=0 tx=0).stop 2023-10-19T13:04:20.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.592+0000 7ff75f3f5640 1 -- 172.21.15.57:0/1373286564 shutdown_connections 2023-10-19T13:04:20.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.592+0000 7ff75f3f5640 1 --2- 172.21.15.57:0/1373286564 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7ff74406af60 0x7ff74406d420 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:20.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.592+0000 7ff75f3f5640 1 --2- 172.21.15.57:0/1373286564 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff758156cd0 0x7ff758157150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:20.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.592+0000 7ff75f3f5640 1 --2- 172.21.15.57:0/1373286564 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff758141770 0x7ff7581f4bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:20.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.592+0000 7ff75f3f5640 1 --2- 172.21.15.57:0/1373286564 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff758141390 0x7ff7581ddf50 unknown :-1 s=CLOSED pgs=1575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:20.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.593+0000 7ff75f3f5640 1 -- 172.21.15.57:0/1373286564 >> 172.21.15.57:0/1373286564 conn(0x7ff7580af830 msgr2=0x7ff7580cb4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:20.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.593+0000 7ff75f3f5640 1 -- 172.21.15.57:0/1373286564 shutdown_connections 2023-10-19T13:04:20.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.593+0000 7ff75f3f5640 1 -- 172.21.15.57:0/1373286564 wait complete. 2023-10-19T13:04:20.606 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-19T13:04:20.606 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-19T13:04:20.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.717+0000 7f0ec82c9640 1 -- 172.21.15.57:0/4181354911 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0eac004d00 msgr2=0x7f0eac005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:20.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.717+0000 7f0ec82c9640 1 --2- 172.21.15.57:0/4181354911 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0eac004d00 0x7f0eac005180 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f0eb8005fd0 tx=0x7f0eac0057f0 comp rx=0 tx=0).stop 2023-10-19T13:04:20.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.717+0000 7f0ec6866640 1 -- 172.21.15.57:0/4181354911 reap_dead start 2023-10-19T13:04:20.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.717+0000 7f0ec82c9640 1 -- 172.21.15.57:0/4181354911 shutdown_connections 2023-10-19T13:04:20.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.717+0000 7f0ec82c9640 1 -- 172.21.15.57:0/4181354911 >> 172.21.15.57:0/4181354911 conn(0x7f0ec0069930 msgr2=0x7f0ec0069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:20.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.717+0000 7f0ec82c9640 1 -- 172.21.15.57:0/4181354911 shutdown_connections 2023-10-19T13:04:20.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.717+0000 7f0ec82c9640 1 -- 172.21.15.57:0/4181354911 wait complete. 2023-10-19T13:04:20.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.718+0000 7f0ec82c9640 1 Processor -- start 2023-10-19T13:04:20.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.718+0000 7f0ec82c9640 1 -- start start 2023-10-19T13:04:20.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.719+0000 7f0ec82c9640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0eac004d00 0x7f0ec0137dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:20.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.719+0000 7f0ec82c9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ec0158b40 0x7f0ec00ad390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:20.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.719+0000 7f0ec82c9640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0ec013aa70 0x7f0ec013aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:20.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.719+0000 7f0ec82c9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ec013b430 con 0x7f0ec0158b40 2023-10-19T13:04:20.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.719+0000 7f0ec82c9640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ec00ae130 con 0x7f0ec013aa70 2023-10-19T13:04:20.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.719+0000 7f0ec82c9640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ec00aef90 con 0x7f0eac004d00 2023-10-19T13:04:20.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.719+0000 7f0ec5864640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ec0158b40 0x7f0ec00ad390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:20.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.719+0000 7f0ec5864640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ec0158b40 0x7f0ec00ad390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35492/0 (socket says 172.21.15.57:35492) 2023-10-19T13:04:20.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.719+0000 7f0ec5864640 1 -- 172.21.15.57:0/3111064352 learned_addr learned my addr 172.21.15.57:0/3111064352 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:20.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.719+0000 7f0ec6866640 1 --2- 172.21.15.57:0/3111064352 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0ec013aa70 0x7f0ec013aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:20.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.719+0000 7f0ec6065640 1 --2- 172.21.15.57:0/3111064352 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0eac004d00 0x7f0ec0137dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:20.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.720+0000 7f0ec5864640 1 -- 172.21.15.57:0/3111064352 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0eac004d00 msgr2=0x7f0ec0137dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:20.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.720+0000 7f0ec5864640 1 --2- 172.21.15.57:0/3111064352 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0eac004d00 0x7f0ec0137dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:20.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.720+0000 7f0ec5864640 1 -- 172.21.15.57:0/3111064352 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0ec013aa70 msgr2=0x7f0ec013aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:20.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.720+0000 7f0ec5864640 1 --2- 172.21.15.57:0/3111064352 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0ec013aa70 0x7f0ec013aef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:20.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.720+0000 7f0ec5864640 1 -- 172.21.15.57:0/3111064352 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ec00ae380 con 0x7f0ec0158b40 2023-10-19T13:04:20.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.720+0000 7f0ec5864640 1 --2- 172.21.15.57:0/3111064352 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ec0158b40 0x7f0ec00ad390 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f0eb0000ff0 tx=0x7f0eb000ea90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:20.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.720+0000 7f0eb6ffd640 1 -- 172.21.15.57:0/3111064352 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eb000c8f0 con 0x7f0ec0158b40 2023-10-19T13:04:20.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.720+0000 7f0eb6ffd640 1 -- 172.21.15.57:0/3111064352 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f0eb000cb40 con 0x7f0ec0158b40 2023-10-19T13:04:20.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.720+0000 7f0ec82c9640 1 -- 172.21.15.57:0/3111064352 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ec00ae660 con 0x7f0ec0158b40 2023-10-19T13:04:20.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.720+0000 7f0eb6ffd640 1 -- 172.21.15.57:0/3111064352 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eb0011460 con 0x7f0ec0158b40 2023-10-19T13:04:20.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.720+0000 7f0ec82c9640 1 -- 172.21.15.57:0/3111064352 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ec00aec20 con 0x7f0ec0158b40 2023-10-19T13:04:20.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.721+0000 7f0ec82c9640 1 -- 172.21.15.57:0/3111064352 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ec00690e0 con 0x7f0ec0158b40 2023-10-19T13:04:20.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.722+0000 7f0eb6ffd640 1 -- 172.21.15.57:0/3111064352 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 69) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f0eb00115c0 con 0x7f0ec0158b40 2023-10-19T13:04:20.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.722+0000 7f0eb6ffd640 1 --2- 172.21.15.57:0/3111064352 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f0ea406af90 0x7f0ea406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:20.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.722+0000 7f0eb6ffd640 1 -- 172.21.15.57:0/3111064352 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f0eb0089e40 con 0x7f0ec0158b40 2023-10-19T13:04:20.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.722+0000 7f0ec6065640 1 --2- 172.21.15.57:0/3111064352 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f0ea406af90 0x7f0ea406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:20.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.723+0000 7f0ec6065640 1 --2- 172.21.15.57:0/3111064352 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f0ea406af90 0x7f0ea406d450 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc0064b0 tx=0x7f0ebc006c00 comp rx=0 tx=0).ready entity=mgr.7960 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:20.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.726+0000 7f0eb6ffd640 1 -- 172.21.15.57:0/3111064352 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0eb0054ee0 con 0x7f0ec0158b40 2023-10-19T13:04:20.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:20.900+0000 7f0ec82c9640 1 -- 172.21.15.57:0/3111064352 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f0ec00c9fb0 con 0x7f0ec0158b40 2023-10-19T13:04:21.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:21.409+0000 7f0eb6ffd640 1 -- 172.21.15.57:0/3111064352 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v141) v1 ==== 138+0+0 (secure 0 0 0) 0x7f0eb001f070 con 0x7f0ec0158b40 2023-10-19T13:04:21.413 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'cephfs_metadata' created 2023-10-19T13:04:21.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:21.413+0000 7f0ec82c9640 1 -- 172.21.15.57:0/3111064352 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f0ea406af90 msgr2=0x7f0ea406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:21.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:21.413+0000 7f0ec82c9640 1 --2- 172.21.15.57:0/3111064352 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f0ea406af90 0x7f0ea406d450 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc0064b0 tx=0x7f0ebc006c00 comp rx=0 tx=0).stop 2023-10-19T13:04:21.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:21.413+0000 7f0ec82c9640 1 -- 172.21.15.57:0/3111064352 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ec0158b40 msgr2=0x7f0ec00ad390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:21.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:21.413+0000 7f0ec82c9640 1 --2- 172.21.15.57:0/3111064352 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ec0158b40 0x7f0ec00ad390 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f0eb0000ff0 tx=0x7f0eb000ea90 comp rx=0 tx=0).stop 2023-10-19T13:04:21.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:21.413+0000 7f0ec82c9640 1 -- 172.21.15.57:0/3111064352 shutdown_connections 2023-10-19T13:04:21.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:21.413+0000 7f0ec82c9640 1 --2- 172.21.15.57:0/3111064352 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f0ea406af90 0x7f0ea406d450 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:21.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:21.413+0000 7f0ec82c9640 1 --2- 172.21.15.57:0/3111064352 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0ec013aa70 0x7f0ec013aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:21.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:21.413+0000 7f0ec82c9640 1 --2- 172.21.15.57:0/3111064352 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ec0158b40 0x7f0ec00ad390 unknown :-1 s=CLOSED pgs=1578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:21.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:21.413+0000 7f0ec82c9640 1 --2- 172.21.15.57:0/3111064352 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0eac004d00 0x7f0ec0137dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:21.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:21.413+0000 7f0ec82c9640 1 -- 172.21.15.57:0/3111064352 >> 172.21.15.57:0/3111064352 conn(0x7f0ec0069930 msgr2=0x7f0ec0159c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:21.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:21.414+0000 7f0ec82c9640 1 -- 172.21.15.57:0/3111064352 shutdown_connections 2023-10-19T13:04:21.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:21.414+0000 7f0ec82c9640 1 -- 172.21.15.57:0/3111064352 wait complete. 2023-10-19T13:04:21.431 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-19T13:04:21.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:21.546+0000 7f444e33c640 1 -- 172.21.15.57:0/3188776958 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f44480c96b0 msgr2=0x7f44480c9b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:21.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:21.546+0000 7f444e33c640 1 --2- 172.21.15.57:0/3188776958 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f44480c96b0 0x7f44480c9b30 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f443c005fd0 tx=0x7f443c02be10 comp rx=0 tx=0).stop 2023-10-19T13:04:21.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:21.546+0000 7f444e33c640 1 -- 172.21.15.57:0/3188776958 shutdown_connections 2023-10-19T13:04:21.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:21.546+0000 7f444e33c640 1 --2- 172.21.15.57:0/3188776958 >> v2:172.21.15.179:3300/0 conn(0x7f4448154af0 0x7f4448154f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:21.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:21.546+0000 7f444e33c640 1 --2- 172.21.15.57:0/3188776958 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f44480c96b0 0x7f44480c9b30 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:21.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:21.546+0000 7f444e33c640 1 -- 172.21.15.57:0/3188776958 >> 172.21.15.57:0/3188776958 conn(0x7f4448069930 msgr2=0x7f4448069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:21.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:21.546+0000 7f444e33c640 1 -- 172.21.15.57:0/3188776958 shutdown_connections 2023-10-19T13:04:21.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:21.546+0000 7f444e33c640 1 -- 172.21.15.57:0/3188776958 wait complete. 2023-10-19T13:04:21.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:21.547+0000 7f444e33c640 1 Processor -- start 2023-10-19T13:04:21.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:21.547+0000 7f444e33c640 1 -- start start 2023-10-19T13:04:21.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:21.548+0000 7f444e33c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f44480b1920 0x7f44480ad830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:21.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:21.548+0000 7f444e33c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f44480c96b0 0x7f44480b0170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:21.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:21.548+0000 7f444e33c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4448154af0 0x7f44480b06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:21.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:21.548+0000 7f444e33c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44480ae410 con 0x7f44480c96b0 2023-10-19T13:04:21.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:21.548+0000 7f444e33c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44481f49c0 con 0x7f4448154af0 2023-10-19T13:04:21.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:21.548+0000 7f444e33c640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f444815d940 con 0x7f44480b1920 2023-10-19T13:04:21.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:21.548+0000 7f44477fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f44480c96b0 0x7f44480b0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:21.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:21.548+0000 7f44477fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f44480c96b0 0x7f44480b0170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35506/0 (socket says 172.21.15.57:35506) 2023-10-19T13:04:21.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:21.548+0000 7f44477fe640 1 -- 172.21.15.57:0/1005990187 learned_addr learned my addr 172.21.15.57:0/1005990187 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:21.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:21.548+0000 7f4447fff640 1 --2- 172.21.15.57:0/1005990187 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f44480b1920 0x7f44480ad830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:21.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:21.548+0000 7f444c8d9640 1 --2- 172.21.15.57:0/1005990187 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4448154af0 0x7f44480b06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:21.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:21.548+0000 7f44477fe640 1 -- 172.21.15.57:0/1005990187 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f44480b1920 msgr2=0x7f44480ad830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:21.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:21.548+0000 7f44477fe640 1 --2- 172.21.15.57:0/1005990187 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f44480b1920 0x7f44480ad830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:21.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:21.548+0000 7f44477fe640 1 -- 172.21.15.57:0/1005990187 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4448154af0 msgr2=0x7f44480b06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:21.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:21.548+0000 7f44477fe640 1 --2- 172.21.15.57:0/1005990187 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4448154af0 0x7f44480b06b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:21.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:21.548+0000 7f44477fe640 1 -- 172.21.15.57:0/1005990187 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4440009840 con 0x7f44480c96b0 2023-10-19T13:04:21.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:21.549+0000 7f44477fe640 1 --2- 172.21.15.57:0/1005990187 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f44480c96b0 0x7f44480b0170 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7f443c006010 tx=0x7f443c02da70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:21.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:21.549+0000 7f4444ff9640 1 -- 172.21.15.57:0/1005990187 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f443c02de70 con 0x7f44480c96b0 2023-10-19T13:04:21.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:21.549+0000 7f4444ff9640 1 -- 172.21.15.57:0/1005990187 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f443c0036e0 con 0x7f44480c96b0 2023-10-19T13:04:21.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:21.549+0000 7f444e33c640 1 -- 172.21.15.57:0/1005990187 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f443c009560 con 0x7f44480c96b0 2023-10-19T13:04:21.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:21.549+0000 7f4444ff9640 1 -- 172.21.15.57:0/1005990187 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f443c003af0 con 0x7f44480c96b0 2023-10-19T13:04:21.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:21.549+0000 7f444e33c640 1 -- 172.21.15.57:0/1005990187 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4448070300 con 0x7f44480c96b0 2023-10-19T13:04:21.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:21.550+0000 7f444e33c640 1 -- 172.21.15.57:0/1005990187 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f444815e6d0 con 0x7f44480c96b0 2023-10-19T13:04:21.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:21.551+0000 7f4444ff9640 1 -- 172.21.15.57:0/1005990187 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 69) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f443c003c50 con 0x7f44480c96b0 2023-10-19T13:04:21.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:21.551+0000 7f4444ff9640 1 --2- 172.21.15.57:0/1005990187 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f443406af60 0x7f443406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:21.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:21.551+0000 7f4444ff9640 1 -- 172.21.15.57:0/1005990187 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5575+0+0 (secure 0 0 0) 0x7f443c0ac3e0 con 0x7f44480c96b0 2023-10-19T13:04:21.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:21.551+0000 7f4447fff640 1 --2- 172.21.15.57:0/1005990187 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f443406af60 0x7f443406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:21.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:21.552+0000 7f4447fff640 1 --2- 172.21.15.57:0/1005990187 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f443406af60 0x7f443406d420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4438001780 tx=0x7f4438009040 comp rx=0 tx=0).ready entity=mgr.7960 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:21.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:21.555+0000 7f4444ff9640 1 -- 172.21.15.57:0/1005990187 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f443c077570 con 0x7f44480c96b0 2023-10-19T13:04:21.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:21.730+0000 7f444e33c640 1 -- 172.21.15.57:0/1005990187 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f44480690e0 con 0x7f44480c96b0 2023-10-19T13:04:22.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:22.418+0000 7f4444ff9640 1 -- 172.21.15.57:0/1005990187 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v142) v1 ==== 170+0+0 (secure 0 0 0) 0x7f443c07ab90 con 0x7f44480c96b0 2023-10-19T13:04:22.421 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'cephfs_data' created 2023-10-19T13:04:22.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:22.421+0000 7f444e33c640 1 -- 172.21.15.57:0/1005990187 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f443406af60 msgr2=0x7f443406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:22.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:22.421+0000 7f444e33c640 1 --2- 172.21.15.57:0/1005990187 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f443406af60 0x7f443406d420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4438001780 tx=0x7f4438009040 comp rx=0 tx=0).stop 2023-10-19T13:04:22.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:22.421+0000 7f444e33c640 1 -- 172.21.15.57:0/1005990187 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f44480c96b0 msgr2=0x7f44480b0170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:22.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:22.421+0000 7f444e33c640 1 --2- 172.21.15.57:0/1005990187 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f44480c96b0 0x7f44480b0170 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7f443c006010 tx=0x7f443c02da70 comp rx=0 tx=0).stop 2023-10-19T13:04:22.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:22.421+0000 7f444e33c640 1 -- 172.21.15.57:0/1005990187 shutdown_connections 2023-10-19T13:04:22.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:22.421+0000 7f444e33c640 1 --2- 172.21.15.57:0/1005990187 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f443406af60 0x7f443406d420 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:22.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:22.421+0000 7f444e33c640 1 --2- 172.21.15.57:0/1005990187 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4448154af0 0x7f44480b06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:22.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:22.421+0000 7f444e33c640 1 --2- 172.21.15.57:0/1005990187 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f44480c96b0 0x7f44480b0170 unknown :-1 s=CLOSED pgs=1579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:22.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:22.421+0000 7f444e33c640 1 --2- 172.21.15.57:0/1005990187 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f44480b1920 0x7f44480ad830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:22.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:22.421+0000 7f444e33c640 1 -- 172.21.15.57:0/1005990187 >> 172.21.15.57:0/1005990187 conn(0x7f4448069930 msgr2=0x7f44481586c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:22.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:22.421+0000 7f444e33c640 1 -- 172.21.15.57:0/1005990187 shutdown_connections 2023-10-19T13:04:22.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:22.421+0000 7f444e33c640 1 -- 172.21.15.57:0/1005990187 wait complete. 2023-10-19T13:04:22.450 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-19T13:04:22.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:22.564+0000 7f9dfef16640 1 -- 172.21.15.57:0/3890586543 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9df8157bb0 msgr2=0x7f9df8070520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:22.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:22.564+0000 7f9dfef16640 1 --2- 172.21.15.57:0/3890586543 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9df8157bb0 0x7f9df8070520 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7f9df80afd00 tx=0x7f9dec02bd10 comp rx=0 tx=0).stop 2023-10-19T13:04:22.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:22.564+0000 7f9dfef16640 1 -- 172.21.15.57:0/3890586543 shutdown_connections 2023-10-19T13:04:22.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:22.564+0000 7f9dfef16640 1 --2- 172.21.15.57:0/3890586543 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9df815b7a0 0x7f9df815bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:22.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:22.564+0000 7f9dfef16640 1 --2- 172.21.15.57:0/3890586543 >> v2:172.21.15.179:3300/0 conn(0x7f9df806c960 0x7f9df8070a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:22.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:22.564+0000 7f9dfef16640 1 --2- 172.21.15.57:0/3890586543 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9df8157bb0 0x7f9df8070520 unknown :-1 s=CLOSED pgs=1580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:22.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:22.564+0000 7f9dfef16640 1 -- 172.21.15.57:0/3890586543 >> 172.21.15.57:0/3890586543 conn(0x7f9df80aec40 msgr2=0x7f9df80af050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:22.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:22.564+0000 7f9dfef16640 1 -- 172.21.15.57:0/3890586543 shutdown_connections 2023-10-19T13:04:22.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:22.564+0000 7f9dfef16640 1 -- 172.21.15.57:0/3890586543 wait complete. 2023-10-19T13:04:22.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:22.565+0000 7f9dfef16640 1 Processor -- start 2023-10-19T13:04:22.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:22.565+0000 7f9dfef16640 1 -- start start 2023-10-19T13:04:22.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:22.567+0000 7f9dfef16640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9df806c960 0x7f9df81f3420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:22.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:22.567+0000 7f9dfef16640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9df8157bb0 0x7f9df8153bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:22.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:22.567+0000 7f9dfef16640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9df815b7a0 0x7f9df806f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:22.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:22.567+0000 7f9dfef16640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df8139ed0 con 0x7f9df815b7a0 2023-10-19T13:04:22.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:22.567+0000 7f9dfef16640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df8154490 con 0x7f9df8157bb0 2023-10-19T13:04:22.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:22.567+0000 7f9dfef16640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9df8154770 con 0x7f9df806c960 2023-10-19T13:04:22.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:22.567+0000 7f9dfd4b3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9df815b7a0 0x7f9df806f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:22.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:22.567+0000 7f9dfd4b3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9df815b7a0 0x7f9df806f780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35524/0 (socket says 172.21.15.57:35524) 2023-10-19T13:04:22.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:22.567+0000 7f9dfd4b3640 1 -- 172.21.15.57:0/2853477936 learned_addr learned my addr 172.21.15.57:0/2853477936 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:22.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:22.567+0000 7f9dfccb2640 1 --2- 172.21.15.57:0/2853477936 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9df806c960 0x7f9df81f3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:22.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:22.567+0000 7f9df7fff640 1 --2- 172.21.15.57:0/2853477936 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9df8157bb0 0x7f9df8153bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:22.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:22.567+0000 7f9dfd4b3640 1 -- 172.21.15.57:0/2853477936 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9df806c960 msgr2=0x7f9df81f3420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:22.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:22.567+0000 7f9dfd4b3640 1 --2- 172.21.15.57:0/2853477936 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9df806c960 0x7f9df81f3420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:22.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:22.567+0000 7f9dfd4b3640 1 -- 172.21.15.57:0/2853477936 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9df8157bb0 msgr2=0x7f9df8153bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:22.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:22.567+0000 7f9dfd4b3640 1 --2- 172.21.15.57:0/2853477936 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9df8157bb0 0x7f9df8153bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:22.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:22.567+0000 7f9dfd4b3640 1 -- 172.21.15.57:0/2853477936 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dec009560 con 0x7f9df815b7a0 2023-10-19T13:04:22.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:22.567+0000 7f9dfd4b3640 1 --2- 172.21.15.57:0/2853477936 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9df815b7a0 0x7f9df806f780 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7f9df000e960 tx=0x7f9df000ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:22.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:22.567+0000 7f9df57fa640 1 -- 172.21.15.57:0/2853477936 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9df0004d20 con 0x7f9df815b7a0 2023-10-19T13:04:22.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:22.567+0000 7f9df57fa640 1 -- 172.21.15.57:0/2853477936 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f9df000cb20 con 0x7f9df815b7a0 2023-10-19T13:04:22.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:22.567+0000 7f9dfef16640 1 -- 172.21.15.57:0/2853477936 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9df8154a50 con 0x7f9df815b7a0 2023-10-19T13:04:22.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:22.567+0000 7f9df57fa640 1 -- 172.21.15.57:0/2853477936 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9df0005020 con 0x7f9df815b7a0 2023-10-19T13:04:22.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:22.567+0000 7f9dfef16640 1 -- 172.21.15.57:0/2853477936 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9df81540f0 con 0x7f9df815b7a0 2023-10-19T13:04:22.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:22.568+0000 7f9dfef16640 1 -- 172.21.15.57:0/2853477936 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9df813a360 con 0x7f9df815b7a0 2023-10-19T13:04:22.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:22.568+0000 7f9df57fa640 1 -- 172.21.15.57:0/2853477936 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 69) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f9df0005180 con 0x7f9df815b7a0 2023-10-19T13:04:22.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:22.569+0000 7f9df57fa640 1 --2- 172.21.15.57:0/2853477936 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f9ddc06af90 0x7f9ddc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:22.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:22.569+0000 7f9df57fa640 1 -- 172.21.15.57:0/2853477936 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 5957+0+0 (secure 0 0 0) 0x7f9df008a250 con 0x7f9df815b7a0 2023-10-19T13:04:22.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:22.569+0000 7f9dfccb2640 1 --2- 172.21.15.57:0/2853477936 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f9ddc06af90 0x7f9ddc06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:22.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:22.570+0000 7f9dfccb2640 1 --2- 172.21.15.57:0/2853477936 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f9ddc06af90 0x7f9ddc06d450 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9dec035000 tx=0x7f9dec004730 comp rx=0 tx=0).ready entity=mgr.7960 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:22.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:22.573+0000 7f9df57fa640 1 -- 172.21.15.57:0/2853477936 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9df813a360 con 0x7f9df815b7a0 2023-10-19T13:04:22.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:22.783+0000 7f9dfef16640 1 -- 172.21.15.57:0/2853477936 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f9df813a360 con 0x7f9df815b7a0 2023-10-19T13:04:23.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.420+0000 7f9df57fa640 1 -- 172.21.15.57:0/2853477936 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 17 and data pool 18 v54) v1 ==== 174+0+0 (secure 0 0 0) 0x7f9df813a360 con 0x7f9df815b7a0 2023-10-19T13:04:23.422 INFO:teuthology.orchestra.run.smithi057.stderr:new fs with metadata pool 17 and data pool 18 2023-10-19T13:04:23.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.423+0000 7f9dfef16640 1 -- 172.21.15.57:0/2853477936 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f9ddc06af90 msgr2=0x7f9ddc06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:23.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.423+0000 7f9dfef16640 1 --2- 172.21.15.57:0/2853477936 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f9ddc06af90 0x7f9ddc06d450 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9dec035000 tx=0x7f9dec004730 comp rx=0 tx=0).stop 2023-10-19T13:04:23.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.423+0000 7f9dfef16640 1 -- 172.21.15.57:0/2853477936 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9df815b7a0 msgr2=0x7f9df806f780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:23.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.423+0000 7f9dfef16640 1 --2- 172.21.15.57:0/2853477936 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9df815b7a0 0x7f9df806f780 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7f9df000e960 tx=0x7f9df000ee20 comp rx=0 tx=0).stop 2023-10-19T13:04:23.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.423+0000 7f9dfef16640 1 -- 172.21.15.57:0/2853477936 shutdown_connections 2023-10-19T13:04:23.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.423+0000 7f9dfef16640 1 --2- 172.21.15.57:0/2853477936 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f9ddc06af90 0x7f9ddc06d450 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:23.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.423+0000 7f9dfef16640 1 --2- 172.21.15.57:0/2853477936 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9df815b7a0 0x7f9df806f780 unknown :-1 s=CLOSED pgs=1581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:23.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.423+0000 7f9dfef16640 1 --2- 172.21.15.57:0/2853477936 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9df8157bb0 0x7f9df8153bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:23.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.423+0000 7f9dfef16640 1 --2- 172.21.15.57:0/2853477936 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9df806c960 0x7f9df81f3420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:23.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.423+0000 7f9dfef16640 1 -- 172.21.15.57:0/2853477936 >> 172.21.15.57:0/2853477936 conn(0x7f9df80aec40 msgr2=0x7f9df814da30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:23.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.423+0000 7f9dfef16640 1 -- 172.21.15.57:0/2853477936 shutdown_connections 2023-10-19T13:04:23.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.423+0000 7f9dfef16640 1 -- 172.21.15.57:0/2853477936 wait complete. 2023-10-19T13:04:23.443 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T13:04:23.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.551+0000 7fc7d3e61640 1 -- 172.21.15.57:0/1789256552 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc7cc157d20 msgr2=0x7fc7cc15b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:23.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.551+0000 7fc7d3e61640 1 --2- 172.21.15.57:0/1789256552 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc7cc157d20 0x7fc7cc15b6d0 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc0b27c0 tx=0x7fc7c402bd90 comp rx=0 tx=0).stop 2023-10-19T13:04:23.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.551+0000 7fc7d3e61640 1 -- 172.21.15.57:0/1789256552 shutdown_connections 2023-10-19T13:04:23.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.551+0000 7fc7d3e61640 1 --2- 172.21.15.57:0/1789256552 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc7cc157d20 0x7fc7cc15b6d0 unknown :-1 s=CLOSED pgs=1582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:23.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.551+0000 7fc7d3e61640 1 --2- 172.21.15.57:0/1789256552 >> v2:172.21.15.179:3300/0 conn(0x7fc7cc06fee0 0x7fc7cc0ad550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:23.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.551+0000 7fc7d3e61640 1 --2- 172.21.15.57:0/1789256552 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc7cc0ad170 0x7fc7cc0af770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:23.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.551+0000 7fc7d3e61640 1 -- 172.21.15.57:0/1789256552 >> 172.21.15.57:0/1789256552 conn(0x7fc7cc0b0cd0 msgr2=0x7fc7cc0b1f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:23.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.551+0000 7fc7d3e61640 1 -- 172.21.15.57:0/1789256552 shutdown_connections 2023-10-19T13:04:23.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.551+0000 7fc7d3e61640 1 -- 172.21.15.57:0/1789256552 wait complete. 2023-10-19T13:04:23.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.552+0000 7fc7d3e61640 1 Processor -- start 2023-10-19T13:04:23.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.552+0000 7fc7d3e61640 1 -- start start 2023-10-19T13:04:23.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.553+0000 7fc7d3e61640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc7cc06fee0 0x7fc7cc06c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:23.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.553+0000 7fc7d3e61640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc7cc0ad170 0x7fc7cc06cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:23.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.553+0000 7fc7d3e61640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc7cc157d20 0x7fc7cc152f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:23.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.553+0000 7fc7d3e61640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7cc1627e0 con 0x7fc7cc0ad170 2023-10-19T13:04:23.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.553+0000 7fc7d3e61640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7cc162ac0 con 0x7fc7cc157d20 2023-10-19T13:04:23.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.553+0000 7fc7d3e61640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7cc162da0 con 0x7fc7cc06fee0 2023-10-19T13:04:23.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.553+0000 7fc7d13fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc7cc0ad170 0x7fc7cc06cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:23.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.553+0000 7fc7d13fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc7cc0ad170 0x7fc7cc06cd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35542/0 (socket says 172.21.15.57:35542) 2023-10-19T13:04:23.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.553+0000 7fc7d13fc640 1 -- 172.21.15.57:0/3523301337 learned_addr learned my addr 172.21.15.57:0/3523301337 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:23.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.553+0000 7fc7d1bfd640 1 --2- 172.21.15.57:0/3523301337 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc7cc06fee0 0x7fc7cc06c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:23.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.553+0000 7fc7d23fe640 1 --2- 172.21.15.57:0/3523301337 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc7cc157d20 0x7fc7cc152f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:23.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.554+0000 7fc7d13fc640 1 -- 172.21.15.57:0/3523301337 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc7cc06fee0 msgr2=0x7fc7cc06c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:23.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.554+0000 7fc7d13fc640 1 --2- 172.21.15.57:0/3523301337 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc7cc06fee0 0x7fc7cc06c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:23.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.554+0000 7fc7d13fc640 1 -- 172.21.15.57:0/3523301337 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc7cc157d20 msgr2=0x7fc7cc152f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:23.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.554+0000 7fc7d13fc640 1 --2- 172.21.15.57:0/3523301337 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc7cc157d20 0x7fc7cc152f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:23.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.554+0000 7fc7d13fc640 1 -- 172.21.15.57:0/3523301337 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7c4009560 con 0x7fc7cc0ad170 2023-10-19T13:04:23.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.554+0000 7fc7d13fc640 1 --2- 172.21.15.57:0/3523301337 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc7cc0ad170 0x7fc7cc06cd20 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7fc7c0005940 tx=0x7fc7c000cd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:23.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.554+0000 7fc7ca7fc640 1 -- 172.21.15.57:0/3523301337 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7c0007bb0 con 0x7fc7cc0ad170 2023-10-19T13:04:23.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.554+0000 7fc7ca7fc640 1 -- 172.21.15.57:0/3523301337 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fc7c0007e00 con 0x7fc7cc0ad170 2023-10-19T13:04:23.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.554+0000 7fc7d3e61640 1 -- 172.21.15.57:0/3523301337 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7cc163080 con 0x7fc7cc0ad170 2023-10-19T13:04:23.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.554+0000 7fc7ca7fc640 1 -- 172.21.15.57:0/3523301337 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7c000f6b0 con 0x7fc7cc0ad170 2023-10-19T13:04:23.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.554+0000 7fc7d3e61640 1 -- 172.21.15.57:0/3523301337 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7cc156eb0 con 0x7fc7cc0ad170 2023-10-19T13:04:23.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.555+0000 7fc7d3e61640 1 -- 172.21.15.57:0/3523301337 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7cc1635a0 con 0x7fc7cc0ad170 2023-10-19T13:04:23.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.555+0000 7fc7ca7fc640 1 -- 172.21.15.57:0/3523301337 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 69) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc7c000f810 con 0x7fc7cc0ad170 2023-10-19T13:04:23.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.556+0000 7fc7ca7fc640 1 --2- 172.21.15.57:0/3523301337 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7fc7b806af60 0x7fc7b806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:23.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.556+0000 7fc7ca7fc640 1 -- 172.21.15.57:0/3523301337 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fc7c008a2d0 con 0x7fc7cc0ad170 2023-10-19T13:04:23.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.556+0000 7fc7d1bfd640 1 --2- 172.21.15.57:0/3523301337 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7fc7b806af60 0x7fc7b806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:23.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.557+0000 7fc7d1bfd640 1 --2- 172.21.15.57:0/3523301337 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7fc7b806af60 0x7fc7b806d420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc7bc005c40 tx=0x7fc7bc00a390 comp rx=0 tx=0).ready entity=mgr.7960 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:23.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.560+0000 7fc7ca7fc640 1 -- 172.21.15.57:0/3523301337 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc7cc1635a0 con 0x7fc7cc0ad170 2023-10-19T13:04:23.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.736+0000 7fc7d3e61640 1 -- 172.21.15.57:0/3523301337 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc7cc1635a0 con 0x7fc7cc0ad170 2023-10-19T13:04:23.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.737+0000 7fc7ca7fc640 1 -- 172.21.15.57:0/3523301337 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v143) v1 ==== 74+0+16852 (secure 0 0 0) 0x7fc7cc1635a0 con 0x7fc7cc0ad170 2023-10-19T13:04:23.739 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:04:23.740 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":143,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T13:04:22.785121+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2023-10-19T13:04:20.932329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"143","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2023-10-19T13:04:21.762157+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"143","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T13:04:23.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.739+0000 7fc7d3e61640 1 -- 172.21.15.57:0/3523301337 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7fc7b806af60 msgr2=0x7fc7b806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:23.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.739+0000 7fc7d3e61640 1 --2- 172.21.15.57:0/3523301337 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7fc7b806af60 0x7fc7b806d420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc7bc005c40 tx=0x7fc7bc00a390 comp rx=0 tx=0).stop 2023-10-19T13:04:23.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.739+0000 7fc7d3e61640 1 -- 172.21.15.57:0/3523301337 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc7cc0ad170 msgr2=0x7fc7cc06cd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:23.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.739+0000 7fc7d3e61640 1 --2- 172.21.15.57:0/3523301337 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc7cc0ad170 0x7fc7cc06cd20 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7fc7c0005940 tx=0x7fc7c000cd60 comp rx=0 tx=0).stop 2023-10-19T13:04:23.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.739+0000 7fc7d3e61640 1 -- 172.21.15.57:0/3523301337 shutdown_connections 2023-10-19T13:04:23.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.739+0000 7fc7d3e61640 1 --2- 172.21.15.57:0/3523301337 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7fc7b806af60 0x7fc7b806d420 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:23.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.739+0000 7fc7d3e61640 1 --2- 172.21.15.57:0/3523301337 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc7cc157d20 0x7fc7cc152f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:23.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.739+0000 7fc7d3e61640 1 --2- 172.21.15.57:0/3523301337 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc7cc0ad170 0x7fc7cc06cd20 unknown :-1 s=CLOSED pgs=1583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:23.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.739+0000 7fc7d3e61640 1 --2- 172.21.15.57:0/3523301337 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc7cc06fee0 0x7fc7cc06c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:23.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.739+0000 7fc7d3e61640 1 -- 172.21.15.57:0/3523301337 >> 172.21.15.57:0/3523301337 conn(0x7fc7cc0b0cd0 msgr2=0x7fc7cc0ac710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:23.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.739+0000 7fc7d3e61640 1 -- 172.21.15.57:0/3523301337 shutdown_connections 2023-10-19T13:04:23.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.739+0000 7fc7d3e61640 1 -- 172.21.15.57:0/3523301337 wait complete. 2023-10-19T13:04:23.752 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T13:04:23.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.860+0000 7fd677dfc640 1 -- 172.21.15.57:0/1146872600 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd6700c9dc0 msgr2=0x7fd670154af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:23.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.860+0000 7fd677dfc640 1 --2- 172.21.15.57:0/1146872600 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd6700c9dc0 0x7fd670154af0 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7fd668005fd0 tx=0x7fd66802c0e0 comp rx=0 tx=0).stop 2023-10-19T13:04:23.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.860+0000 7fd677dfc640 1 -- 172.21.15.57:0/1146872600 shutdown_connections 2023-10-19T13:04:23.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.860+0000 7fd677dfc640 1 --2- 172.21.15.57:0/1146872600 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd6700c9dc0 0x7fd670154af0 unknown :-1 s=CLOSED pgs=1585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:23.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.860+0000 7fd677dfc640 1 --2- 172.21.15.57:0/1146872600 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd6700b1920 0x7fd67006fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:23.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.860+0000 7fd677dfc640 1 -- 172.21.15.57:0/1146872600 >> 172.21.15.57:0/1146872600 conn(0x7fd670069930 msgr2=0x7fd670069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:23.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.860+0000 7fd677dfc640 1 -- 172.21.15.57:0/1146872600 shutdown_connections 2023-10-19T13:04:23.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.860+0000 7fd677dfc640 1 -- 172.21.15.57:0/1146872600 wait complete. 2023-10-19T13:04:23.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.861+0000 7fd677dfc640 1 Processor -- start 2023-10-19T13:04:23.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.861+0000 7fd677dfc640 1 -- start start 2023-10-19T13:04:23.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.862+0000 7fd677dfc640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd67006ffb0 0x7fd67006c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:23.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.862+0000 7fd677dfc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd6700b1920 0x7fd670149540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:23.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.862+0000 7fd677dfc640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd6700c9dc0 0x7fd6700cb240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:23.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.862+0000 7fd677dfc640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd670155bf0 con 0x7fd6700b1920 2023-10-19T13:04:23.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.862+0000 7fd677dfc640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6701555c0 con 0x7fd6700c9dc0 2023-10-19T13:04:23.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.862+0000 7fd677dfc640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd67006c8e0 con 0x7fd67006ffb0 2023-10-19T13:04:23.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.862+0000 7fd675397640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd6700b1920 0x7fd670149540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:23.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.862+0000 7fd675397640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd6700b1920 0x7fd670149540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35564/0 (socket says 172.21.15.57:35564) 2023-10-19T13:04:23.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.862+0000 7fd675397640 1 -- 172.21.15.57:0/2470504301 learned_addr learned my addr 172.21.15.57:0/2470504301 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:23.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.862+0000 7fd676399640 1 --2- 172.21.15.57:0/2470504301 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd6700c9dc0 0x7fd6700cb240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:23.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.862+0000 7fd675b98640 1 --2- 172.21.15.57:0/2470504301 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd67006ffb0 0x7fd67006c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:23.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.862+0000 7fd675397640 1 -- 172.21.15.57:0/2470504301 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd67006ffb0 msgr2=0x7fd67006c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:23.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.862+0000 7fd675397640 1 --2- 172.21.15.57:0/2470504301 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd67006ffb0 0x7fd67006c020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:23.868 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.862+0000 7fd675397640 1 -- 172.21.15.57:0/2470504301 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd6700c9dc0 msgr2=0x7fd6700cb240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:23.868 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.862+0000 7fd675397640 1 --2- 172.21.15.57:0/2470504301 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd6700c9dc0 0x7fd6700cb240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:23.868 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.862+0000 7fd675397640 1 -- 172.21.15.57:0/2470504301 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd668009840 con 0x7fd6700b1920 2023-10-19T13:04:23.868 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.862+0000 7fd675b98640 1 --2- 172.21.15.57:0/2470504301 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd67006ffb0 0x7fd67006c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:04:23.868 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.862+0000 7fd675397640 1 --2- 172.21.15.57:0/2470504301 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd6700b1920 0x7fd670149540 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7fd66000c400 tx=0x7fd66000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:23.868 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.863+0000 7fd6667fc640 1 -- 172.21.15.57:0/2470504301 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd660004230 con 0x7fd6700b1920 2023-10-19T13:04:23.869 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.863+0000 7fd6667fc640 1 -- 172.21.15.57:0/2470504301 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fd6600044a0 con 0x7fd6700b1920 2023-10-19T13:04:23.869 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.863+0000 7fd677dfc640 1 -- 172.21.15.57:0/2470504301 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd67006cbc0 con 0x7fd6700b1920 2023-10-19T13:04:23.869 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.863+0000 7fd6667fc640 1 -- 172.21.15.57:0/2470504301 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd660003850 con 0x7fd6700b1920 2023-10-19T13:04:23.869 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.863+0000 7fd677dfc640 1 -- 172.21.15.57:0/2470504301 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd67006c560 con 0x7fd6700b1920 2023-10-19T13:04:23.869 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.864+0000 7fd677dfc640 1 -- 172.21.15.57:0/2470504301 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd670070ae0 con 0x7fd6700b1920 2023-10-19T13:04:23.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.864+0000 7fd6667fc640 1 -- 172.21.15.57:0/2470504301 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 69) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd6600039b0 con 0x7fd6700b1920 2023-10-19T13:04:23.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.864+0000 7fd6667fc640 1 --2- 172.21.15.57:0/2470504301 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7fd65406af60 0x7fd65406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:23.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.865+0000 7fd6667fc640 1 -- 172.21.15.57:0/2470504301 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fd660089040 con 0x7fd6700b1920 2023-10-19T13:04:23.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.865+0000 7fd675b98640 1 --2- 172.21.15.57:0/2470504301 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7fd65406af60 0x7fd65406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:23.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.865+0000 7fd675b98640 1 --2- 172.21.15.57:0/2470504301 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7fd65406af60 0x7fd65406d420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd66c0058c0 tx=0x7fd66c005850 comp rx=0 tx=0).ready entity=mgr.7960 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:23.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:23.869+0000 7fd6667fc640 1 -- 172.21.15.57:0/2470504301 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd660053f30 con 0x7fd6700b1920 2023-10-19T13:04:24.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.044+0000 7fd677dfc640 1 -- 172.21.15.57:0/2470504301 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd6700690e0 con 0x7fd6700b1920 2023-10-19T13:04:24.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.045+0000 7fd6667fc640 1 -- 172.21.15.57:0/2470504301 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v143) v1 ==== 74+0+16852 (secure 0 0 0) 0x7fd660017020 con 0x7fd6700b1920 2023-10-19T13:04:24.047 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:04:24.048 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":143,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T13:04:22.785121+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2023-10-19T13:04:20.932329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"143","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2023-10-19T13:04:21.762157+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"143","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T13:04:24.049 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.047+0000 7fd677dfc640 1 -- 172.21.15.57:0/2470504301 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7fd65406af60 msgr2=0x7fd65406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:24.049 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.047+0000 7fd677dfc640 1 --2- 172.21.15.57:0/2470504301 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7fd65406af60 0x7fd65406d420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd66c0058c0 tx=0x7fd66c005850 comp rx=0 tx=0).stop 2023-10-19T13:04:24.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.047+0000 7fd677dfc640 1 -- 172.21.15.57:0/2470504301 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd6700b1920 msgr2=0x7fd670149540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:24.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.047+0000 7fd677dfc640 1 --2- 172.21.15.57:0/2470504301 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd6700b1920 0x7fd670149540 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7fd66000c400 tx=0x7fd66000c8c0 comp rx=0 tx=0).stop 2023-10-19T13:04:24.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.047+0000 7fd677dfc640 1 -- 172.21.15.57:0/2470504301 shutdown_connections 2023-10-19T13:04:24.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.047+0000 7fd677dfc640 1 --2- 172.21.15.57:0/2470504301 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7fd65406af60 0x7fd65406d420 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:24.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.047+0000 7fd677dfc640 1 --2- 172.21.15.57:0/2470504301 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd6700c9dc0 0x7fd6700cb240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:24.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.047+0000 7fd677dfc640 1 --2- 172.21.15.57:0/2470504301 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd6700b1920 0x7fd670149540 unknown :-1 s=CLOSED pgs=1586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:24.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.047+0000 7fd677dfc640 1 --2- 172.21.15.57:0/2470504301 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd67006ffb0 0x7fd67006c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:24.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.047+0000 7fd677dfc640 1 -- 172.21.15.57:0/2470504301 >> 172.21.15.57:0/2470504301 conn(0x7fd670069930 msgr2=0x7fd670154890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:24.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.047+0000 7fd677dfc640 1 -- 172.21.15.57:0/2470504301 shutdown_connections 2023-10-19T13:04:24.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.047+0000 7fd677dfc640 1 -- 172.21.15.57:0/2470504301 wait complete. 2023-10-19T13:04:24.060 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-19T13:04:24.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.171+0000 7f57af675640 1 -- 172.21.15.57:0/251555509 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f57a80cb980 msgr2=0x7f57a813db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:24.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.171+0000 7f57af675640 1 --2- 172.21.15.57:0/251555509 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f57a80cb980 0x7f57a813db80 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7f57a806f4d0 tx=0x7f57a402bda0 comp rx=0 tx=0).stop 2023-10-19T13:04:24.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.171+0000 7f57af675640 1 -- 172.21.15.57:0/251555509 shutdown_connections 2023-10-19T13:04:24.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.171+0000 7f57af675640 1 --2- 172.21.15.57:0/251555509 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f57a80c96b0 0x7f57a80c9b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:24.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.171+0000 7f57af675640 1 --2- 172.21.15.57:0/251555509 >> v2:172.21.15.179:3300/0 conn(0x7f57a813d350 0x7f57a80cbd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:24.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.171+0000 7f57af675640 1 --2- 172.21.15.57:0/251555509 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f57a80cb980 0x7f57a813db80 unknown :-1 s=CLOSED pgs=1587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:24.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.171+0000 7f57af675640 1 -- 172.21.15.57:0/251555509 >> 172.21.15.57:0/251555509 conn(0x7f57a8069930 msgr2=0x7f57a8069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:24.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.172+0000 7f57af675640 1 -- 172.21.15.57:0/251555509 shutdown_connections 2023-10-19T13:04:24.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.172+0000 7f57af675640 1 -- 172.21.15.57:0/251555509 wait complete. 2023-10-19T13:04:24.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.173+0000 7f57af675640 1 Processor -- start 2023-10-19T13:04:24.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.173+0000 7f57af675640 1 -- start start 2023-10-19T13:04:24.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.173+0000 7f57af675640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f57a80c96b0 0x7f57a813ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:24.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.173+0000 7f57af675640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f57a80cb980 0x7f57a813f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:24.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.173+0000 7f57af675640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f57a813ff10 0x7f57a814fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:24.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.173+0000 7f57af675640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57a81403c0 con 0x7f57a80cb980 2023-10-19T13:04:24.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.173+0000 7f57af675640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57a8152a70 con 0x7f57a813ff10 2023-10-19T13:04:24.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.173+0000 7f57af675640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57a81367d0 con 0x7f57a80c96b0 2023-10-19T13:04:24.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.173+0000 7f57acc10640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f57a80cb980 0x7f57a813f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:24.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.174+0000 7f57acc10640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f57a80cb980 0x7f57a813f9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35580/0 (socket says 172.21.15.57:35580) 2023-10-19T13:04:24.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.174+0000 7f57acc10640 1 -- 172.21.15.57:0/1920384357 learned_addr learned my addr 172.21.15.57:0/1920384357 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:24.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.174+0000 7f57adc12640 1 --2- 172.21.15.57:0/1920384357 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f57a813ff10 0x7f57a814fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:24.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.174+0000 7f57ad411640 1 --2- 172.21.15.57:0/1920384357 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f57a80c96b0 0x7f57a813ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:24.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.174+0000 7f57acc10640 1 -- 172.21.15.57:0/1920384357 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f57a80c96b0 msgr2=0x7f57a813ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:24.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.174+0000 7f57acc10640 1 --2- 172.21.15.57:0/1920384357 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f57a80c96b0 0x7f57a813ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:24.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.174+0000 7f57acc10640 1 -- 172.21.15.57:0/1920384357 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f57a813ff10 msgr2=0x7f57a814fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:24.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.174+0000 7f57acc10640 1 --2- 172.21.15.57:0/1920384357 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f57a813ff10 0x7f57a814fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:24.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.174+0000 7f57acc10640 1 -- 172.21.15.57:0/1920384357 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57a4009560 con 0x7f57a80cb980 2023-10-19T13:04:24.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.174+0000 7f57ad411640 1 --2- 172.21.15.57:0/1920384357 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f57a80c96b0 0x7f57a813ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:04:24.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.174+0000 7f57acc10640 1 --2- 172.21.15.57:0/1920384357 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f57a80cb980 0x7f57a813f9d0 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7f579800c970 tx=0x7f579800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:24.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.174+0000 7f579dffb640 1 -- 172.21.15.57:0/1920384357 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5798007bb0 con 0x7f57a80cb980 2023-10-19T13:04:24.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.174+0000 7f579dffb640 1 -- 172.21.15.57:0/1920384357 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f5798007e00 con 0x7f57a80cb980 2023-10-19T13:04:24.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.174+0000 7f579dffb640 1 -- 172.21.15.57:0/1920384357 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5798005280 con 0x7f57a80cb980 2023-10-19T13:04:24.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.174+0000 7f57af675640 1 -- 172.21.15.57:0/1920384357 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57a814f350 con 0x7f57a80cb980 2023-10-19T13:04:24.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.174+0000 7f57af675640 1 -- 172.21.15.57:0/1920384357 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57a8151940 con 0x7f57a80cb980 2023-10-19T13:04:24.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.176+0000 7f57af675640 1 -- 172.21.15.57:0/1920384357 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57a80690e0 con 0x7f57a80cb980 2023-10-19T13:04:24.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.176+0000 7f579dffb640 1 -- 172.21.15.57:0/1920384357 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 69) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f57980053e0 con 0x7f57a80cb980 2023-10-19T13:04:24.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.176+0000 7f579dffb640 1 --2- 172.21.15.57:0/1920384357 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f578c06af60 0x7f578c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:24.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.176+0000 7f579dffb640 1 -- 172.21.15.57:0/1920384357 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f5798089de0 con 0x7f57a80cb980 2023-10-19T13:04:24.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.176+0000 7f57ad411640 1 --2- 172.21.15.57:0/1920384357 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f578c06af60 0x7f578c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:24.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.177+0000 7f57ad411640 1 --2- 172.21.15.57:0/1920384357 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f578c06af60 0x7f578c06d420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f57a4035000 tx=0x7f57a4004730 comp rx=0 tx=0).ready entity=mgr.7960 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:24.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.181+0000 7f579dffb640 1 -- 172.21.15.57:0/1920384357 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f5798054d10 con 0x7f57a80cb980 2023-10-19T13:04:24.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.394+0000 7f57af675640 1 -- 172.21.15.57:0/1920384357 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f57a8155a60 con 0x7f57a80cb980 2023-10-19T13:04:24.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.431+0000 7f579dffb640 1 -- 172.21.15.57:0/1920384357 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v55) v1 ==== 118+0+0 (secure 0 0 0) 0x7f579801d070 con 0x7f57a80cb980 2023-10-19T13:04:24.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.433+0000 7f57af675640 1 -- 172.21.15.57:0/1920384357 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f578c06af60 msgr2=0x7f578c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:24.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.433+0000 7f57af675640 1 --2- 172.21.15.57:0/1920384357 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f578c06af60 0x7f578c06d420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f57a4035000 tx=0x7f57a4004730 comp rx=0 tx=0).stop 2023-10-19T13:04:24.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.433+0000 7f57af675640 1 -- 172.21.15.57:0/1920384357 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f57a80cb980 msgr2=0x7f57a813f9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:24.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.433+0000 7f57af675640 1 --2- 172.21.15.57:0/1920384357 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f57a80cb980 0x7f57a813f9d0 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7f579800c970 tx=0x7f579800ce30 comp rx=0 tx=0).stop 2023-10-19T13:04:24.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.433+0000 7f57af675640 1 -- 172.21.15.57:0/1920384357 shutdown_connections 2023-10-19T13:04:24.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.433+0000 7f57af675640 1 --2- 172.21.15.57:0/1920384357 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f578c06af60 0x7f578c06d420 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:24.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.433+0000 7f57af675640 1 --2- 172.21.15.57:0/1920384357 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f57a813ff10 0x7f57a814fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:24.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.433+0000 7f57af675640 1 --2- 172.21.15.57:0/1920384357 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f57a80cb980 0x7f57a813f9d0 unknown :-1 s=CLOSED pgs=1588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:24.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.433+0000 7f57af675640 1 --2- 172.21.15.57:0/1920384357 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f57a80c96b0 0x7f57a813ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:24.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.433+0000 7f57af675640 1 -- 172.21.15.57:0/1920384357 >> 172.21.15.57:0/1920384357 conn(0x7f57a8069930 msgr2=0x7f57a80c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:24.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.433+0000 7f57af675640 1 -- 172.21.15.57:0/1920384357 shutdown_connections 2023-10-19T13:04:24.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.433+0000 7f57af675640 1 -- 172.21.15.57:0/1920384357 wait complete. 2023-10-19T13:04:24.446 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-19T13:04:24.446 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-19T13:04:24.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.556+0000 7ff7cc426640 1 -- 172.21.15.57:0/2408554022 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff7c40cf340 msgr2=0x7ff7c40d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:24.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.556+0000 7ff7cc426640 1 --2- 172.21.15.57:0/2408554022 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff7c40cf340 0x7ff7c40d1910 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7ff7c0009830 tx=0x7ff7c002bcd0 comp rx=0 tx=0).stop 2023-10-19T13:04:24.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.556+0000 7ff7cc426640 1 -- 172.21.15.57:0/2408554022 shutdown_connections 2023-10-19T13:04:24.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.556+0000 7ff7cc426640 1 --2- 172.21.15.57:0/2408554022 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff7c40cf340 0x7ff7c40d1910 unknown :-1 s=CLOSED pgs=1590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:24.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.556+0000 7ff7cc426640 1 -- 172.21.15.57:0/2408554022 >> 172.21.15.57:0/2408554022 conn(0x7ff7c40d0d60 msgr2=0x7ff7c40d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:24.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.556+0000 7ff7cc426640 1 -- 172.21.15.57:0/2408554022 shutdown_connections 2023-10-19T13:04:24.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.556+0000 7ff7cc426640 1 -- 172.21.15.57:0/2408554022 wait complete. 2023-10-19T13:04:24.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.558+0000 7ff7cc426640 1 Processor -- start 2023-10-19T13:04:24.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.558+0000 7ff7cc426640 1 -- start start 2023-10-19T13:04:24.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.558+0000 7ff7cc426640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff7c40ceb40 0x7ff7c415d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:24.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.558+0000 7ff7cc426640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff7c40cf340 0x7ff7c40b5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:24.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.558+0000 7ff7cc426640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff7c415b2a0 0x7ff7c40b6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:24.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.558+0000 7ff7cc426640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7c41f2670 con 0x7ff7c415b2a0 2023-10-19T13:04:24.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.558+0000 7ff7cc426640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7c41f4e40 con 0x7ff7c40cf340 2023-10-19T13:04:24.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.558+0000 7ff7cc426640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7c41f5120 con 0x7ff7c40ceb40 2023-10-19T13:04:24.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.558+0000 7ff7ca9c3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff7c415b2a0 0x7ff7c40b6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:24.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.558+0000 7ff7ca9c3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff7c415b2a0 0x7ff7c40b6420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35594/0 (socket says 172.21.15.57:35594) 2023-10-19T13:04:24.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.559+0000 7ff7ca9c3640 1 -- 172.21.15.57:0/1848140191 learned_addr learned my addr 172.21.15.57:0/1848140191 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:24.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.559+0000 7ff7c99c1640 1 --2- 172.21.15.57:0/1848140191 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff7c40cf340 0x7ff7c40b5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:24.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.559+0000 7ff7ca1c2640 1 --2- 172.21.15.57:0/1848140191 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff7c40ceb40 0x7ff7c415d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:24.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.559+0000 7ff7ca9c3640 1 -- 172.21.15.57:0/1848140191 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff7c40ceb40 msgr2=0x7ff7c415d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:24.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.559+0000 7ff7ca9c3640 1 --2- 172.21.15.57:0/1848140191 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff7c40ceb40 0x7ff7c415d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:24.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.559+0000 7ff7ca9c3640 1 -- 172.21.15.57:0/1848140191 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff7c40cf340 msgr2=0x7ff7c40b5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:24.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.559+0000 7ff7ca9c3640 1 --2- 172.21.15.57:0/1848140191 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff7c40cf340 0x7ff7c40b5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:24.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.559+0000 7ff7ca9c3640 1 -- 172.21.15.57:0/1848140191 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7c0009490 con 0x7ff7c415b2a0 2023-10-19T13:04:24.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.559+0000 7ff7ca9c3640 1 --2- 172.21.15.57:0/1848140191 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff7c415b2a0 0x7ff7c40b6420 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7ff7bc00c780 tx=0x7ff7bc00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:24.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.559+0000 7ff7baffd640 1 -- 172.21.15.57:0/1848140191 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7bc003d80 con 0x7ff7c415b2a0 2023-10-19T13:04:24.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.559+0000 7ff7baffd640 1 -- 172.21.15.57:0/1848140191 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7ff7bc00ce90 con 0x7ff7c415b2a0 2023-10-19T13:04:24.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.559+0000 7ff7cc426640 1 -- 172.21.15.57:0/1848140191 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7c41f5400 con 0x7ff7c415b2a0 2023-10-19T13:04:24.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.559+0000 7ff7baffd640 1 -- 172.21.15.57:0/1848140191 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7bc0052d0 con 0x7ff7c415b2a0 2023-10-19T13:04:24.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.559+0000 7ff7cc426640 1 -- 172.21.15.57:0/1848140191 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7c415dbc0 con 0x7ff7c415b2a0 2023-10-19T13:04:24.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.560+0000 7ff7cc426640 1 -- 172.21.15.57:0/1848140191 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7c40b6960 con 0x7ff7c415b2a0 2023-10-19T13:04:24.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.561+0000 7ff7baffd640 1 -- 172.21.15.57:0/1848140191 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 69) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff7bc005430 con 0x7ff7c415b2a0 2023-10-19T13:04:24.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.561+0000 7ff7baffd640 1 --2- 172.21.15.57:0/1848140191 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7ff7a806af60 0x7ff7a806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:24.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.561+0000 7ff7baffd640 1 -- 172.21.15.57:0/1848140191 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6057+0+0 (secure 0 0 0) 0x7ff7bc089130 con 0x7ff7c415b2a0 2023-10-19T13:04:24.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.561+0000 7ff7ca1c2640 1 --2- 172.21.15.57:0/1848140191 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7ff7a806af60 0x7ff7a806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:24.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.562+0000 7ff7ca1c2640 1 --2- 172.21.15.57:0/1848140191 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7ff7a806af60 0x7ff7a806d420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff7c0036000 tx=0x7ff7c0004730 comp rx=0 tx=0).ready entity=mgr.7960 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:24.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.565+0000 7ff7baffd640 1 -- 172.21.15.57:0/1848140191 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff7bc053ef0 con 0x7ff7c415b2a0 2023-10-19T13:04:24.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:24.778+0000 7ff7cc426640 1 -- 172.21.15.57:0/1848140191 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7ff7c40690e0 con 0x7ff7c415b2a0 2023-10-19T13:04:25.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.440+0000 7ff7baffd640 1 -- 172.21.15.57:0/1848140191 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v56) v1 ==== 122+0+0 (secure 0 0 0) 0x7ff7bc057510 con 0x7ff7c415b2a0 2023-10-19T13:04:25.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.442+0000 7ff7cc426640 1 -- 172.21.15.57:0/1848140191 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7ff7a806af60 msgr2=0x7ff7a806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:25.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.442+0000 7ff7cc426640 1 --2- 172.21.15.57:0/1848140191 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7ff7a806af60 0x7ff7a806d420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff7c0036000 tx=0x7ff7c0004730 comp rx=0 tx=0).stop 2023-10-19T13:04:25.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.442+0000 7ff7cc426640 1 -- 172.21.15.57:0/1848140191 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff7c415b2a0 msgr2=0x7ff7c40b6420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:25.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.442+0000 7ff7cc426640 1 --2- 172.21.15.57:0/1848140191 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff7c415b2a0 0x7ff7c40b6420 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7ff7bc00c780 tx=0x7ff7bc00cc40 comp rx=0 tx=0).stop 2023-10-19T13:04:25.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.442+0000 7ff7cc426640 1 -- 172.21.15.57:0/1848140191 shutdown_connections 2023-10-19T13:04:25.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.442+0000 7ff7cc426640 1 --2- 172.21.15.57:0/1848140191 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7ff7a806af60 0x7ff7a806d420 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:25.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.442+0000 7ff7cc426640 1 --2- 172.21.15.57:0/1848140191 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff7c415b2a0 0x7ff7c40b6420 unknown :-1 s=CLOSED pgs=1591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:25.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.442+0000 7ff7cc426640 1 --2- 172.21.15.57:0/1848140191 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff7c40cf340 0x7ff7c40b5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:25.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.442+0000 7ff7cc426640 1 --2- 172.21.15.57:0/1848140191 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff7c40ceb40 0x7ff7c415d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:25.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.442+0000 7ff7cc426640 1 -- 172.21.15.57:0/1848140191 >> 172.21.15.57:0/1848140191 conn(0x7ff7c40d0d60 msgr2=0x7ff7c40c9e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:25.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.443+0000 7ff7cc426640 1 -- 172.21.15.57:0/1848140191 shutdown_connections 2023-10-19T13:04:25.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.443+0000 7ff7cc426640 1 -- 172.21.15.57:0/1848140191 wait complete. 2023-10-19T13:04:25.457 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-19T13:04:25.458 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T13:04:25.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.571+0000 7ff82da1a640 1 -- 172.21.15.57:0/2835543355 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff8281579f0 msgr2=0x7ff8280b0e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:25.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.571+0000 7ff82da1a640 1 --2- 172.21.15.57:0/2835543355 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff8281579f0 0x7ff8280b0e90 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7ff820009830 tx=0x7ff82002bd60 comp rx=0 tx=0).stop 2023-10-19T13:04:25.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.571+0000 7ff82da1a640 1 -- 172.21.15.57:0/2835543355 shutdown_connections 2023-10-19T13:04:25.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.571+0000 7ff82da1a640 1 --2- 172.21.15.57:0/2835543355 >> v2:172.21.15.179:3300/0 conn(0x7ff82806ae40 0x7ff828157dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:25.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.571+0000 7ff82da1a640 1 --2- 172.21.15.57:0/2835543355 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff8281579f0 0x7ff8280b0e90 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:25.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.571+0000 7ff82da1a640 1 -- 172.21.15.57:0/2835543355 >> 172.21.15.57:0/2835543355 conn(0x7ff82806fae0 msgr2=0x7ff82806fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:25.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.571+0000 7ff82da1a640 1 -- 172.21.15.57:0/2835543355 shutdown_connections 2023-10-19T13:04:25.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.571+0000 7ff82da1a640 1 -- 172.21.15.57:0/2835543355 wait complete. 2023-10-19T13:04:25.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.571+0000 7ff82da1a640 1 Processor -- start 2023-10-19T13:04:25.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.571+0000 7ff82da1a640 1 -- start start 2023-10-19T13:04:25.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.572+0000 7ff82da1a640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff82806ae40 0x7ff8280cc680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:25.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.572+0000 7ff82da1a640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff8281579f0 0x7ff8280ccbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:25.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.572+0000 7ff82da1a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff82815c1f0 0x7ff828140810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:25.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.572+0000 7ff82da1a640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8281f39f0 con 0x7ff82815c1f0 2023-10-19T13:04:25.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.572+0000 7ff82da1a640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8280a6750 con 0x7ff82806ae40 2023-10-19T13:04:25.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.572+0000 7ff82da1a640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8281f2e70 con 0x7ff8281579f0 2023-10-19T13:04:25.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.572+0000 7ff8277fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff82815c1f0 0x7ff828140810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:25.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.572+0000 7ff8277fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff82815c1f0 0x7ff828140810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35606/0 (socket says 172.21.15.57:35606) 2023-10-19T13:04:25.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.572+0000 7ff8277fe640 1 -- 172.21.15.57:0/1807214516 learned_addr learned my addr 172.21.15.57:0/1807214516 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:25.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.573+0000 7ff826ffd640 1 --2- 172.21.15.57:0/1807214516 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff82806ae40 0x7ff8280cc680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:25.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.573+0000 7ff8267fc640 1 --2- 172.21.15.57:0/1807214516 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff8281579f0 0x7ff8280ccbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:25.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.573+0000 7ff8277fe640 1 -- 172.21.15.57:0/1807214516 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff8281579f0 msgr2=0x7ff8280ccbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:25.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.573+0000 7ff8277fe640 1 --2- 172.21.15.57:0/1807214516 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff8281579f0 0x7ff8280ccbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:25.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.573+0000 7ff8277fe640 1 -- 172.21.15.57:0/1807214516 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff82806ae40 msgr2=0x7ff8280cc680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:25.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.573+0000 7ff8277fe640 1 --2- 172.21.15.57:0/1807214516 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff82806ae40 0x7ff8280cc680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:25.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.573+0000 7ff8277fe640 1 -- 172.21.15.57:0/1807214516 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff818009900 con 0x7ff82815c1f0 2023-10-19T13:04:25.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.573+0000 7ff826ffd640 1 --2- 172.21.15.57:0/1807214516 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff82806ae40 0x7ff8280cc680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T13:04:25.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.573+0000 7ff8267fc640 1 --2- 172.21.15.57:0/1807214516 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff8281579f0 0x7ff8280ccbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T13:04:25.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.573+0000 7ff8277fe640 1 --2- 172.21.15.57:0/1807214516 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff82815c1f0 0x7ff828140810 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7ff81800c3a0 tx=0x7ff81800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:25.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.573+0000 7ff80ffff640 1 -- 172.21.15.57:0/1807214516 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff818003b80 con 0x7ff82815c1f0 2023-10-19T13:04:25.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.573+0000 7ff80ffff640 1 -- 172.21.15.57:0/1807214516 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7ff818003df0 con 0x7ff82815c1f0 2023-10-19T13:04:25.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.573+0000 7ff80ffff640 1 -- 172.21.15.57:0/1807214516 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff818004290 con 0x7ff82815c1f0 2023-10-19T13:04:25.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.573+0000 7ff82da1a640 1 -- 172.21.15.57:0/1807214516 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff820009490 con 0x7ff82815c1f0 2023-10-19T13:04:25.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.573+0000 7ff82da1a640 1 -- 172.21.15.57:0/1807214516 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8280b2330 con 0x7ff82815c1f0 2023-10-19T13:04:25.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.574+0000 7ff82da1a640 1 -- 172.21.15.57:0/1807214516 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff828154a70 con 0x7ff82815c1f0 2023-10-19T13:04:25.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.574+0000 7ff80ffff640 1 -- 172.21.15.57:0/1807214516 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 69) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff81800a040 con 0x7ff82815c1f0 2023-10-19T13:04:25.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.575+0000 7ff80ffff640 1 --2- 172.21.15.57:0/1807214516 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7ff81406af60 0x7ff81406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:25.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.575+0000 7ff80ffff640 1 -- 172.21.15.57:0/1807214516 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6057+0+0 (secure 0 0 0) 0x7ff81808a1d0 con 0x7ff82815c1f0 2023-10-19T13:04:25.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.575+0000 7ff826ffd640 1 --2- 172.21.15.57:0/1807214516 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7ff81406af60 0x7ff81406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:25.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.576+0000 7ff826ffd640 1 --2- 172.21.15.57:0/1807214516 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7ff81406af60 0x7ff81406d420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff820036000 tx=0x7ff820004730 comp rx=0 tx=0).ready entity=mgr.7960 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:25.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.579+0000 7ff80ffff640 1 -- 172.21.15.57:0/1807214516 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff818054f90 con 0x7ff82815c1f0 2023-10-19T13:04:25.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.790+0000 7ff82da1a640 1 -- 172.21.15.57:0/1807214516 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff8280b19e0 con 0x7ff82815c1f0 2023-10-19T13:04:25.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.790+0000 7ff80ffff640 1 -- 172.21.15.57:0/1807214516 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+4921 (secure 0 0 0) 0x7ff8180585b0 con 0x7ff82815c1f0 2023-10-19T13:04:25.794 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:04:25.794 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":56,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7325,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/3318110028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":3318110028},{"type":"v1","addr":"172.21.15.179:6837","nonce":3318110028}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":8143,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/3625556517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":3625556517},{"type":"v1","addr":"172.21.15.57:6835","nonce":3625556517}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":8158,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/4167860955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":4167860955},{"type":"v1","addr":"172.21.15.179:6836","nonce":4167860955}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T13:04:22.785432+0000","modified":"2023-10-19T13:04:24.778587+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9066},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9066":{"gid":9066,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":3,"addr":"172.21.15.57:6834/3474229632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":3474229632},{"type":"v1","addr":"172.21.15.57:6834","nonce":3474229632}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2023-10-19T13:04:25.795 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 56 2023-10-19T13:04:25.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.792+0000 7ff82da1a640 1 -- 172.21.15.57:0/1807214516 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7ff81406af60 msgr2=0x7ff81406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:25.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.792+0000 7ff82da1a640 1 --2- 172.21.15.57:0/1807214516 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7ff81406af60 0x7ff81406d420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff820036000 tx=0x7ff820004730 comp rx=0 tx=0).stop 2023-10-19T13:04:25.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.792+0000 7ff82da1a640 1 -- 172.21.15.57:0/1807214516 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff82815c1f0 msgr2=0x7ff828140810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:25.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.792+0000 7ff82da1a640 1 --2- 172.21.15.57:0/1807214516 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff82815c1f0 0x7ff828140810 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7ff81800c3a0 tx=0x7ff81800c860 comp rx=0 tx=0).stop 2023-10-19T13:04:25.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.792+0000 7ff82da1a640 1 -- 172.21.15.57:0/1807214516 shutdown_connections 2023-10-19T13:04:25.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.792+0000 7ff82da1a640 1 --2- 172.21.15.57:0/1807214516 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7ff81406af60 0x7ff81406d420 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:25.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.792+0000 7ff82da1a640 1 --2- 172.21.15.57:0/1807214516 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff82815c1f0 0x7ff828140810 unknown :-1 s=CLOSED pgs=1592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:25.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.792+0000 7ff82da1a640 1 --2- 172.21.15.57:0/1807214516 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff8281579f0 0x7ff8280ccbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:25.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.792+0000 7ff82da1a640 1 --2- 172.21.15.57:0/1807214516 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff82806ae40 0x7ff8280cc680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:25.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.792+0000 7ff82da1a640 1 -- 172.21.15.57:0/1807214516 >> 172.21.15.57:0/1807214516 conn(0x7ff82806fae0 msgr2=0x7ff828159780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:25.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.792+0000 7ff82da1a640 1 -- 172.21.15.57:0/1807214516 shutdown_connections 2023-10-19T13:04:25.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.792+0000 7ff82da1a640 1 -- 172.21.15.57:0/1807214516 wait complete. 2023-10-19T13:04:25.806 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T13:04:25.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.913+0000 7fa0a01b0640 1 -- 172.21.15.57:0/85486562 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0980cbd20 msgr2=0x7fa098158300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:25.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.913+0000 7fa0a01b0640 1 --2- 172.21.15.57:0/85486562 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0980cbd20 0x7fa098158300 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7fa0980b10f0 tx=0x7fa09402bdb0 comp rx=0 tx=0).stop 2023-10-19T13:04:25.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.913+0000 7fa0a01b0640 1 -- 172.21.15.57:0/85486562 shutdown_connections 2023-10-19T13:04:25.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.913+0000 7fa0a01b0640 1 --2- 172.21.15.57:0/85486562 >> v2:172.21.15.179:3300/0 conn(0x7fa0980b0530 0x7fa0980b09a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:25.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.913+0000 7fa0a01b0640 1 --2- 172.21.15.57:0/85486562 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0980cbd20 0x7fa098158300 unknown :-1 s=CLOSED pgs=1593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:25.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.913+0000 7fa0a01b0640 1 -- 172.21.15.57:0/85486562 >> 172.21.15.57:0/85486562 conn(0x7fa098069930 msgr2=0x7fa098069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:25.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.913+0000 7fa0a01b0640 1 -- 172.21.15.57:0/85486562 shutdown_connections 2023-10-19T13:04:25.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.913+0000 7fa0a01b0640 1 -- 172.21.15.57:0/85486562 wait complete. 2023-10-19T13:04:25.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.914+0000 7fa0a01b0640 1 Processor -- start 2023-10-19T13:04:25.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.914+0000 7fa0a01b0640 1 -- start start 2023-10-19T13:04:25.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.915+0000 7fa0a01b0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa0980afa20 0x7fa098148bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:25.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.915+0000 7fa0a01b0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0980b0530 0x7fa09814bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:25.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.915+0000 7fa0a01b0640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa0980cbd20 0x7fa09806bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:25.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.915+0000 7fa0a01b0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0981e2690 con 0x7fa0980b0530 2023-10-19T13:04:25.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.915+0000 7fa0a01b0640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0981e1a10 con 0x7fa0980afa20 2023-10-19T13:04:25.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.915+0000 7fa0a01b0640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0981e1cf0 con 0x7fa0980cbd20 2023-10-19T13:04:25.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.915+0000 7fa09d74b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0980b0530 0x7fa09814bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:25.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.915+0000 7fa09d74b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0980b0530 0x7fa09814bf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35630/0 (socket says 172.21.15.57:35630) 2023-10-19T13:04:25.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.915+0000 7fa09d74b640 1 -- 172.21.15.57:0/2054153404 learned_addr learned my addr 172.21.15.57:0/2054153404 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:25.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.915+0000 7fa09df4c640 1 --2- 172.21.15.57:0/2054153404 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa0980afa20 0x7fa098148bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:25.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.915+0000 7fa09e74d640 1 --2- 172.21.15.57:0/2054153404 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa0980cbd20 0x7fa09806bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:25.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.915+0000 7fa09d74b640 1 -- 172.21.15.57:0/2054153404 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa0980cbd20 msgr2=0x7fa09806bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:25.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.915+0000 7fa09d74b640 1 --2- 172.21.15.57:0/2054153404 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa0980cbd20 0x7fa09806bec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:25.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.915+0000 7fa09d74b640 1 -- 172.21.15.57:0/2054153404 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa0980afa20 msgr2=0x7fa098148bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:25.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.915+0000 7fa09d74b640 1 --2- 172.21.15.57:0/2054153404 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa0980afa20 0x7fa098148bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:25.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.915+0000 7fa09d74b640 1 -- 172.21.15.57:0/2054153404 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa094009560 con 0x7fa0980b0530 2023-10-19T13:04:25.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.915+0000 7fa09e74d640 1 --2- 172.21.15.57:0/2054153404 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa0980cbd20 0x7fa09806bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:04:25.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.915+0000 7fa09d74b640 1 --2- 172.21.15.57:0/2054153404 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0980b0530 0x7fa09814bf30 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7fa08800c430 tx=0x7fa08800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:25.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.915+0000 7fa08e7fc640 1 -- 172.21.15.57:0/2054153404 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa088011790 con 0x7fa0980b0530 2023-10-19T13:04:25.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.915+0000 7fa08e7fc640 1 -- 172.21.15.57:0/2054153404 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fa0880119e0 con 0x7fa0980b0530 2023-10-19T13:04:25.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.915+0000 7fa0a01b0640 1 -- 172.21.15.57:0/2054153404 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0981e1fd0 con 0x7fa0980b0530 2023-10-19T13:04:25.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.915+0000 7fa08e7fc640 1 -- 172.21.15.57:0/2054153404 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa088011e30 con 0x7fa0980b0530 2023-10-19T13:04:25.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.916+0000 7fa0a01b0640 1 -- 172.21.15.57:0/2054153404 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0981e1650 con 0x7fa0980b0530 2023-10-19T13:04:25.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.916+0000 7fa0a01b0640 1 -- 172.21.15.57:0/2054153404 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0981e27d0 con 0x7fa0980b0530 2023-10-19T13:04:25.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.917+0000 7fa08e7fc640 1 -- 172.21.15.57:0/2054153404 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 69) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fa08801c460 con 0x7fa0980b0530 2023-10-19T13:04:25.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.917+0000 7fa08e7fc640 1 --2- 172.21.15.57:0/2054153404 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7fa07c06af60 0x7fa07c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:25.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.917+0000 7fa08e7fc640 1 -- 172.21.15.57:0/2054153404 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fa0880899f0 con 0x7fa0980b0530 2023-10-19T13:04:25.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.918+0000 7fa09df4c640 1 --2- 172.21.15.57:0/2054153404 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7fa07c06af60 0x7fa07c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:25.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.918+0000 7fa09df4c640 1 --2- 172.21.15.57:0/2054153404 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7fa07c06af60 0x7fa07c06d420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa094035000 tx=0x7fa094004730 comp rx=0 tx=0).ready entity=mgr.7960 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:25.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:25.922+0000 7fa08e7fc640 1 -- 172.21.15.57:0/2054153404 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa0981e27d0 con 0x7fa0980b0530 2023-10-19T13:04:26.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.097+0000 7fa0a01b0640 1 -- 172.21.15.57:0/2054153404 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa0981e27d0 con 0x7fa0980b0530 2023-10-19T13:04:26.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.098+0000 7fa08e7fc640 1 -- 172.21.15.57:0/2054153404 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v144) v1 ==== 74+0+16839 (secure 0 0 0) 0x7fa0981e27d0 con 0x7fa0980b0530 2023-10-19T13:04:26.101 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:04:26.101 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":144,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T13:04:24.227191+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2023-10-19T13:04:20.932329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"143","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2023-10-19T13:04:21.762157+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"144","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T13:04:26.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.100+0000 7fa0a01b0640 1 -- 172.21.15.57:0/2054153404 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7fa07c06af60 msgr2=0x7fa07c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:26.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.100+0000 7fa0a01b0640 1 --2- 172.21.15.57:0/2054153404 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7fa07c06af60 0x7fa07c06d420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa094035000 tx=0x7fa094004730 comp rx=0 tx=0).stop 2023-10-19T13:04:26.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.100+0000 7fa0a01b0640 1 -- 172.21.15.57:0/2054153404 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0980b0530 msgr2=0x7fa09814bf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:26.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.100+0000 7fa0a01b0640 1 --2- 172.21.15.57:0/2054153404 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0980b0530 0x7fa09814bf30 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7fa08800c430 tx=0x7fa08800c8f0 comp rx=0 tx=0).stop 2023-10-19T13:04:26.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.100+0000 7fa0a01b0640 1 -- 172.21.15.57:0/2054153404 shutdown_connections 2023-10-19T13:04:26.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.100+0000 7fa0a01b0640 1 --2- 172.21.15.57:0/2054153404 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7fa07c06af60 0x7fa07c06d420 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:26.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.100+0000 7fa0a01b0640 1 --2- 172.21.15.57:0/2054153404 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa0980cbd20 0x7fa09806bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:26.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.100+0000 7fa0a01b0640 1 --2- 172.21.15.57:0/2054153404 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0980b0530 0x7fa09814bf30 unknown :-1 s=CLOSED pgs=1594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:26.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.100+0000 7fa0a01b0640 1 --2- 172.21.15.57:0/2054153404 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa0980afa20 0x7fa098148bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:26.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.100+0000 7fa0a01b0640 1 -- 172.21.15.57:0/2054153404 >> 172.21.15.57:0/2054153404 conn(0x7fa098069930 msgr2=0x7fa098069d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:26.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.100+0000 7fa0a01b0640 1 -- 172.21.15.57:0/2054153404 shutdown_connections 2023-10-19T13:04:26.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.100+0000 7fa0a01b0640 1 -- 172.21.15.57:0/2054153404 wait complete. 2023-10-19T13:04:26.113 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-19T13:04:26.114 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-19T13:04:26.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.220+0000 7fef2daec640 1 -- 172.21.15.57:0/1490847761 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fef280afa20 msgr2=0x7fef28158300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:26.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.220+0000 7fef2daec640 1 --2- 172.21.15.57:0/1490847761 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fef280afa20 0x7fef28158300 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fef1c009830 tx=0x7fef1c02bdf0 comp rx=0 tx=0).stop 2023-10-19T13:04:26.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.220+0000 7fef2daec640 1 -- 172.21.15.57:0/1490847761 shutdown_connections 2023-10-19T13:04:26.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.220+0000 7fef2daec640 1 --2- 172.21.15.57:0/1490847761 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fef280afa20 0x7fef28158300 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:26.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.220+0000 7fef2daec640 1 -- 172.21.15.57:0/1490847761 >> 172.21.15.57:0/1490847761 conn(0x7fef28069930 msgr2=0x7fef28069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:26.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.220+0000 7fef2daec640 1 -- 172.21.15.57:0/1490847761 shutdown_connections 2023-10-19T13:04:26.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.220+0000 7fef2daec640 1 -- 172.21.15.57:0/1490847761 wait complete. 2023-10-19T13:04:26.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.221+0000 7fef2daec640 1 Processor -- start 2023-10-19T13:04:26.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.221+0000 7fef2daec640 1 -- start start 2023-10-19T13:04:26.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.222+0000 7fef2daec640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fef280afa20 0x7fef2814a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:26.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.222+0000 7fef2daec640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fef280b0530 0x7fef281338e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:26.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.222+0000 7fef2daec640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fef280cbd20 0x7fef2814c110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:26.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.222+0000 7fef2daec640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef28134c50 con 0x7fef280cbd20 2023-10-19T13:04:26.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.222+0000 7fef2daec640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef28147640 con 0x7fef280b0530 2023-10-19T13:04:26.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.222+0000 7fef2daec640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef2814ae50 con 0x7fef280afa20 2023-10-19T13:04:26.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.222+0000 7fef27fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fef280cbd20 0x7fef2814c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:26.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.222+0000 7fef27fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fef280cbd20 0x7fef2814c110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35644/0 (socket says 172.21.15.57:35644) 2023-10-19T13:04:26.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.222+0000 7fef27fff640 1 -- 172.21.15.57:0/2230255344 learned_addr learned my addr 172.21.15.57:0/2230255344 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:26.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.223+0000 7fef26ffd640 1 --2- 172.21.15.57:0/2230255344 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fef280b0530 0x7fef281338e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:26.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.223+0000 7fef277fe640 1 --2- 172.21.15.57:0/2230255344 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fef280afa20 0x7fef2814a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:26.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.223+0000 7fef27fff640 1 -- 172.21.15.57:0/2230255344 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fef280afa20 msgr2=0x7fef2814a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:26.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.223+0000 7fef27fff640 1 --2- 172.21.15.57:0/2230255344 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fef280afa20 0x7fef2814a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:26.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.223+0000 7fef27fff640 1 -- 172.21.15.57:0/2230255344 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fef280b0530 msgr2=0x7fef281338e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:26.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.223+0000 7fef27fff640 1 --2- 172.21.15.57:0/2230255344 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fef280b0530 0x7fef281338e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:26.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.223+0000 7fef27fff640 1 -- 172.21.15.57:0/2230255344 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef1c009490 con 0x7fef280cbd20 2023-10-19T13:04:26.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.223+0000 7fef27fff640 1 --2- 172.21.15.57:0/2230255344 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fef280cbd20 0x7fef2814c110 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7fef2000c7a0 tx=0x7fef2000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:26.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.223+0000 7fef0ffff640 1 -- 172.21.15.57:0/2230255344 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef20003b70 con 0x7fef280cbd20 2023-10-19T13:04:26.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.223+0000 7fef0ffff640 1 -- 172.21.15.57:0/2230255344 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fef20003cf0 con 0x7fef280cbd20 2023-10-19T13:04:26.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.223+0000 7fef0ffff640 1 -- 172.21.15.57:0/2230255344 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef2000aab0 con 0x7fef280cbd20 2023-10-19T13:04:26.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.223+0000 7fef2daec640 1 -- 172.21.15.57:0/2230255344 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef281342b0 con 0x7fef280cbd20 2023-10-19T13:04:26.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.223+0000 7fef2daec640 1 -- 172.21.15.57:0/2230255344 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef28134870 con 0x7fef280cbd20 2023-10-19T13:04:26.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.224+0000 7fef2daec640 1 -- 172.21.15.57:0/2230255344 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef28134d90 con 0x7fef280cbd20 2023-10-19T13:04:26.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.225+0000 7fef0ffff640 1 -- 172.21.15.57:0/2230255344 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 69) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fef2000ac10 con 0x7fef280cbd20 2023-10-19T13:04:26.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.225+0000 7fef0ffff640 1 --2- 172.21.15.57:0/2230255344 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7fef1406af60 0x7fef1406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:26.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.225+0000 7fef0ffff640 1 -- 172.21.15.57:0/2230255344 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fef20089880 con 0x7fef280cbd20 2023-10-19T13:04:26.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.225+0000 7fef277fe640 1 --2- 172.21.15.57:0/2230255344 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7fef1406af60 0x7fef1406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:26.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.226+0000 7fef277fe640 1 --2- 172.21.15.57:0/2230255344 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7fef1406af60 0x7fef1406d420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fef180011e0 tx=0x7fef18009040 comp rx=0 tx=0).ready entity=mgr.7960 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:26.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.229+0000 7fef0ffff640 1 -- 172.21.15.57:0/2230255344 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fef28134d90 con 0x7fef280cbd20 2023-10-19T13:04:26.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.403+0000 7fef2daec640 1 -- 172.21.15.57:0/2230255344 --> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fef28134d90 con 0x7fef1406af60 2023-10-19T13:04:26.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.411+0000 7fef0ffff640 1 -- 172.21.15.57:0/2230255344 <== mgr.7960 v2:172.21.15.179:6832/2346394725 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+290081 (secure 0 0 0) 0x7fef28134d90 con 0x7fef1406af60 2023-10-19T13:04:26.417 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:04:26.418 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2023-10-19T13:04:26.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.414+0000 7fef2daec640 1 -- 172.21.15.57:0/2230255344 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7fef1406af60 msgr2=0x7fef1406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:26.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.414+0000 7fef2daec640 1 --2- 172.21.15.57:0/2230255344 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7fef1406af60 0x7fef1406d420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fef180011e0 tx=0x7fef18009040 comp rx=0 tx=0).stop 2023-10-19T13:04:26.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.414+0000 7fef2daec640 1 -- 172.21.15.57:0/2230255344 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fef280cbd20 msgr2=0x7fef2814c110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:26.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.414+0000 7fef2daec640 1 --2- 172.21.15.57:0/2230255344 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fef280cbd20 0x7fef2814c110 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7fef2000c7a0 tx=0x7fef2000cc60 comp rx=0 tx=0).stop 2023-10-19T13:04:26.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.414+0000 7fef2daec640 1 -- 172.21.15.57:0/2230255344 shutdown_connections 2023-10-19T13:04:26.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.414+0000 7fef2daec640 1 --2- 172.21.15.57:0/2230255344 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7fef1406af60 0x7fef1406d420 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:26.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.414+0000 7fef2daec640 1 --2- 172.21.15.57:0/2230255344 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fef280cbd20 0x7fef2814c110 unknown :-1 s=CLOSED pgs=1596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:26.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.414+0000 7fef2daec640 1 --2- 172.21.15.57:0/2230255344 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fef280b0530 0x7fef281338e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:26.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.414+0000 7fef2daec640 1 --2- 172.21.15.57:0/2230255344 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fef280afa20 0x7fef2814a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:26.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.414+0000 7fef2daec640 1 -- 172.21.15.57:0/2230255344 >> 172.21.15.57:0/2230255344 conn(0x7fef28069930 msgr2=0x7fef281593c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:26.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.415+0000 7fef2daec640 1 -- 172.21.15.57:0/2230255344 shutdown_connections 2023-10-19T13:04:26.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.415+0000 7fef2daec640 1 -- 172.21.15.57:0/2230255344 wait complete. 2023-10-19T13:04:26.430 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-10-19T13:04:25.745126+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1154,"num_read_kb":2358,"num_write":413,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3103,"ondisk_log_size":3103,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":6,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":339428,"kb_used_data":15016,"kb_used_omap":12,"kb_used_meta":324275,"kb_avail":754635292,"statfs":{"total":773094113280,"available":772746539008,"internally_reserved":0,"allocated":15376384,"data_stored":8347381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":332058169},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.330833"},"pg_stats":[{"pgid":"18.3f","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.254263+0000","last_change":"2023-10-19T13:04:23.249380+0000","last_active":"2023-10-19T13:04:23.254263+0000","last_peered":"2023-10-19T13:04:23.254263+0000","last_clean":"2023-10-19T13:04:23.254263+0000","last_became_active":"2023-10-19T13:04:23.249130+0000","last_became_peered":"2023-10-19T13:04:23.249130+0000","last_unstale":"2023-10-19T13:04:23.254263+0000","last_undegraded":"2023-10-19T13:04:23.254263+0000","last_fullsized":"2023-10-19T13:04:23.254263+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:20:45.859782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3e","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.254311+0000","last_change":"2023-10-19T13:04:23.245224+0000","last_active":"2023-10-19T13:04:23.254311+0000","last_peered":"2023-10-19T13:04:23.254311+0000","last_clean":"2023-10-19T13:04:23.254311+0000","last_became_active":"2023-10-19T13:04:23.244923+0000","last_became_peered":"2023-10-19T13:04:23.244923+0000","last_unstale":"2023-10-19T13:04:23.254311+0000","last_undegraded":"2023-10-19T13:04:23.254311+0000","last_fullsized":"2023-10-19T13:04:23.254311+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:28:43.173352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3d","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.254683+0000","last_change":"2023-10-19T13:04:23.247264+0000","last_active":"2023-10-19T13:04:23.254683+0000","last_peered":"2023-10-19T13:04:23.254683+0000","last_clean":"2023-10-19T13:04:23.254683+0000","last_became_active":"2023-10-19T13:04:23.246996+0000","last_became_peered":"2023-10-19T13:04:23.246996+0000","last_unstale":"2023-10-19T13:04:23.254683+0000","last_undegraded":"2023-10-19T13:04:23.254683+0000","last_fullsized":"2023-10-19T13:04:23.254683+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:36:11.644511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3c","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.252462+0000","last_change":"2023-10-19T13:04:23.250157+0000","last_active":"2023-10-19T13:04:23.252462+0000","last_peered":"2023-10-19T13:04:23.252462+0000","last_clean":"2023-10-19T13:04:23.252462+0000","last_became_active":"2023-10-19T13:04:23.249875+0000","last_became_peered":"2023-10-19T13:04:23.249875+0000","last_unstale":"2023-10-19T13:04:23.252462+0000","last_undegraded":"2023-10-19T13:04:23.252462+0000","last_fullsized":"2023-10-19T13:04:23.252462+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:17:34.123593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3b","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.246485+0000","last_change":"2023-10-19T13:04:23.244884+0000","last_active":"2023-10-19T13:04:23.246485+0000","last_peered":"2023-10-19T13:04:23.246485+0000","last_clean":"2023-10-19T13:04:23.246485+0000","last_became_active":"2023-10-19T13:04:23.244620+0000","last_became_peered":"2023-10-19T13:04:23.244620+0000","last_unstale":"2023-10-19T13:04:23.246485+0000","last_undegraded":"2023-10-19T13:04:23.246485+0000","last_fullsized":"2023-10-19T13:04:23.246485+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:48:30.703457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3a","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.258871+0000","last_change":"2023-10-19T13:04:23.254891+0000","last_active":"2023-10-19T13:04:23.258871+0000","last_peered":"2023-10-19T13:04:23.258871+0000","last_clean":"2023-10-19T13:04:23.258871+0000","last_became_active":"2023-10-19T13:04:23.254633+0000","last_became_peered":"2023-10-19T13:04:23.254633+0000","last_unstale":"2023-10-19T13:04:23.258871+0000","last_undegraded":"2023-10-19T13:04:23.258871+0000","last_fullsized":"2023-10-19T13:04:23.258871+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:42:10.908893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.39","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.258540+0000","last_change":"2023-10-19T13:04:23.248971+0000","last_active":"2023-10-19T13:04:23.258540+0000","last_peered":"2023-10-19T13:04:23.258540+0000","last_clean":"2023-10-19T13:04:23.258540+0000","last_became_active":"2023-10-19T13:04:23.248698+0000","last_became_peered":"2023-10-19T13:04:23.248698+0000","last_unstale":"2023-10-19T13:04:23.258540+0000","last_undegraded":"2023-10-19T13:04:23.258540+0000","last_fullsized":"2023-10-19T13:04:23.258540+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:45:28.119851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.38","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.254513+0000","last_change":"2023-10-19T13:04:23.245992+0000","last_active":"2023-10-19T13:04:23.254513+0000","last_peered":"2023-10-19T13:04:23.254513+0000","last_clean":"2023-10-19T13:04:23.254513+0000","last_became_active":"2023-10-19T13:04:23.245478+0000","last_became_peered":"2023-10-19T13:04:23.245478+0000","last_unstale":"2023-10-19T13:04:23.254513+0000","last_undegraded":"2023-10-19T13:04:23.254513+0000","last_fullsized":"2023-10-19T13:04:23.254513+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:26:46.252277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.37","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.255483+0000","last_change":"2023-10-19T13:04:23.255026+0000","last_active":"2023-10-19T13:04:23.255483+0000","last_peered":"2023-10-19T13:04:23.255483+0000","last_clean":"2023-10-19T13:04:23.255483+0000","last_became_active":"2023-10-19T13:04:23.254753+0000","last_became_peered":"2023-10-19T13:04:23.254753+0000","last_unstale":"2023-10-19T13:04:23.255483+0000","last_undegraded":"2023-10-19T13:04:23.255483+0000","last_fullsized":"2023-10-19T13:04:23.255483+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:00:22.625725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.36","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.258733+0000","last_change":"2023-10-19T13:04:23.250614+0000","last_active":"2023-10-19T13:04:23.258733+0000","last_peered":"2023-10-19T13:04:23.258733+0000","last_clean":"2023-10-19T13:04:23.258733+0000","last_became_active":"2023-10-19T13:04:23.250322+0000","last_became_peered":"2023-10-19T13:04:23.250322+0000","last_unstale":"2023-10-19T13:04:23.258733+0000","last_undegraded":"2023-10-19T13:04:23.258733+0000","last_fullsized":"2023-10-19T13:04:23.258733+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:29:38.675177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.35","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.251604+0000","last_change":"2023-10-19T13:04:23.248749+0000","last_active":"2023-10-19T13:04:23.251604+0000","last_peered":"2023-10-19T13:04:23.251604+0000","last_clean":"2023-10-19T13:04:23.251604+0000","last_became_active":"2023-10-19T13:04:23.248463+0000","last_became_peered":"2023-10-19T13:04:23.248463+0000","last_unstale":"2023-10-19T13:04:23.251604+0000","last_undegraded":"2023-10-19T13:04:23.251604+0000","last_fullsized":"2023-10-19T13:04:23.251604+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:06:24.528059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.34","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.258665+0000","last_change":"2023-10-19T13:04:23.249174+0000","last_active":"2023-10-19T13:04:23.258665+0000","last_peered":"2023-10-19T13:04:23.258665+0000","last_clean":"2023-10-19T13:04:23.258665+0000","last_became_active":"2023-10-19T13:04:23.248875+0000","last_became_peered":"2023-10-19T13:04:23.248875+0000","last_unstale":"2023-10-19T13:04:23.258665+0000","last_undegraded":"2023-10-19T13:04:23.258665+0000","last_fullsized":"2023-10-19T13:04:23.258665+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:17:58.642825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.33","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.253552+0000","last_change":"2023-10-19T13:04:23.243990+0000","last_active":"2023-10-19T13:04:23.253552+0000","last_peered":"2023-10-19T13:04:23.253552+0000","last_clean":"2023-10-19T13:04:23.253552+0000","last_became_active":"2023-10-19T13:04:23.243730+0000","last_became_peered":"2023-10-19T13:04:23.243730+0000","last_unstale":"2023-10-19T13:04:23.253552+0000","last_undegraded":"2023-10-19T13:04:23.253552+0000","last_fullsized":"2023-10-19T13:04:23.253552+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:48:30.703457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.32","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.254450+0000","last_change":"2023-10-19T13:04:23.246493+0000","last_active":"2023-10-19T13:04:23.254450+0000","last_peered":"2023-10-19T13:04:23.254450+0000","last_clean":"2023-10-19T13:04:23.254450+0000","last_became_active":"2023-10-19T13:04:23.246135+0000","last_became_peered":"2023-10-19T13:04:23.246135+0000","last_unstale":"2023-10-19T13:04:23.254450+0000","last_undegraded":"2023-10-19T13:04:23.254450+0000","last_fullsized":"2023-10-19T13:04:23.254450+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:28:49.954421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.31","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.246471+0000","last_change":"2023-10-19T13:04:23.244804+0000","last_active":"2023-10-19T13:04:23.246471+0000","last_peered":"2023-10-19T13:04:23.246471+0000","last_clean":"2023-10-19T13:04:23.246471+0000","last_became_active":"2023-10-19T13:04:23.244536+0000","last_became_peered":"2023-10-19T13:04:23.244536+0000","last_unstale":"2023-10-19T13:04:23.246471+0000","last_undegraded":"2023-10-19T13:04:23.246471+0000","last_fullsized":"2023-10-19T13:04:23.246471+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:17:54.965079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.30","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.258700+0000","last_change":"2023-10-19T13:04:23.249193+0000","last_active":"2023-10-19T13:04:23.258700+0000","last_peered":"2023-10-19T13:04:23.258700+0000","last_clean":"2023-10-19T13:04:23.258700+0000","last_became_active":"2023-10-19T13:04:23.248650+0000","last_became_peered":"2023-10-19T13:04:23.248650+0000","last_unstale":"2023-10-19T13:04:23.258700+0000","last_undegraded":"2023-10-19T13:04:23.258700+0000","last_fullsized":"2023-10-19T13:04:23.258700+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:49:40.332015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.2f","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.248433+0000","last_change":"2023-10-19T13:04:23.240965+0000","last_active":"2023-10-19T13:04:23.248433+0000","last_peered":"2023-10-19T13:04:23.248433+0000","last_clean":"2023-10-19T13:04:23.248433+0000","last_became_active":"2023-10-19T13:04:23.240698+0000","last_became_peered":"2023-10-19T13:04:23.240698+0000","last_unstale":"2023-10-19T13:04:23.248433+0000","last_undegraded":"2023-10-19T13:04:23.248433+0000","last_fullsized":"2023-10-19T13:04:23.248433+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:39:07.411522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.2e","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.254030+0000","last_change":"2023-10-19T13:04:23.244595+0000","last_active":"2023-10-19T13:04:23.254030+0000","last_peered":"2023-10-19T13:04:23.254030+0000","last_clean":"2023-10-19T13:04:23.254030+0000","last_became_active":"2023-10-19T13:04:23.244297+0000","last_became_peered":"2023-10-19T13:04:23.244297+0000","last_unstale":"2023-10-19T13:04:23.254030+0000","last_undegraded":"2023-10-19T13:04:23.254030+0000","last_fullsized":"2023-10-19T13:04:23.254030+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:33:36.981974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.2d","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.257006+0000","last_change":"2023-10-19T13:04:23.253311+0000","last_active":"2023-10-19T13:04:23.257006+0000","last_peered":"2023-10-19T13:04:23.257006+0000","last_clean":"2023-10-19T13:04:23.257006+0000","last_became_active":"2023-10-19T13:04:23.253046+0000","last_became_peered":"2023-10-19T13:04:23.253046+0000","last_unstale":"2023-10-19T13:04:23.257006+0000","last_undegraded":"2023-10-19T13:04:23.257006+0000","last_fullsized":"2023-10-19T13:04:23.257006+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:17:54.965079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.2c","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.258646+0000","last_change":"2023-10-19T13:04:23.250054+0000","last_active":"2023-10-19T13:04:23.258646+0000","last_peered":"2023-10-19T13:04:23.258646+0000","last_clean":"2023-10-19T13:04:23.258646+0000","last_became_active":"2023-10-19T13:04:23.249773+0000","last_became_peered":"2023-10-19T13:04:23.249773+0000","last_unstale":"2023-10-19T13:04:23.258646+0000","last_undegraded":"2023-10-19T13:04:23.258646+0000","last_fullsized":"2023-10-19T13:04:23.258646+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:13:37.185717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.2b","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.253515+0000","last_change":"2023-10-19T13:04:23.252029+0000","last_active":"2023-10-19T13:04:23.253515+0000","last_peered":"2023-10-19T13:04:23.253515+0000","last_clean":"2023-10-19T13:04:23.253515+0000","last_became_active":"2023-10-19T13:04:23.251603+0000","last_became_peered":"2023-10-19T13:04:23.251603+0000","last_unstale":"2023-10-19T13:04:23.253515+0000","last_undegraded":"2023-10-19T13:04:23.253515+0000","last_fullsized":"2023-10-19T13:04:23.253515+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:11:20.296382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.2a","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.259595+0000","last_change":"2023-10-19T13:04:23.254291+0000","last_active":"2023-10-19T13:04:23.259595+0000","last_peered":"2023-10-19T13:04:23.259595+0000","last_clean":"2023-10-19T13:04:23.259595+0000","last_became_active":"2023-10-19T13:04:23.254000+0000","last_became_peered":"2023-10-19T13:04:23.254000+0000","last_unstale":"2023-10-19T13:04:23.259595+0000","last_undegraded":"2023-10-19T13:04:23.259595+0000","last_fullsized":"2023-10-19T13:04:23.259595+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:04:15.321219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.29","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.255308+0000","last_change":"2023-10-19T13:04:23.254920+0000","last_active":"2023-10-19T13:04:23.255308+0000","last_peered":"2023-10-19T13:04:23.255308+0000","last_clean":"2023-10-19T13:04:23.255308+0000","last_became_active":"2023-10-19T13:04:23.254661+0000","last_became_peered":"2023-10-19T13:04:23.254661+0000","last_unstale":"2023-10-19T13:04:23.255308+0000","last_undegraded":"2023-10-19T13:04:23.255308+0000","last_fullsized":"2023-10-19T13:04:23.255308+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:43:05.064367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.28","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.247313+0000","last_change":"2023-10-19T13:04:23.246405+0000","last_active":"2023-10-19T13:04:23.247313+0000","last_peered":"2023-10-19T13:04:23.247313+0000","last_clean":"2023-10-19T13:04:23.247313+0000","last_became_active":"2023-10-19T13:04:23.246135+0000","last_became_peered":"2023-10-19T13:04:23.246135+0000","last_unstale":"2023-10-19T13:04:23.247313+0000","last_undegraded":"2023-10-19T13:04:23.247313+0000","last_fullsized":"2023-10-19T13:04:23.247313+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:17:34.123593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.27","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.248444+0000","last_change":"2023-10-19T13:04:23.247245+0000","last_active":"2023-10-19T13:04:23.248444+0000","last_peered":"2023-10-19T13:04:23.248444+0000","last_clean":"2023-10-19T13:04:23.248444+0000","last_became_active":"2023-10-19T13:04:23.246943+0000","last_became_peered":"2023-10-19T13:04:23.246943+0000","last_unstale":"2023-10-19T13:04:23.248444+0000","last_undegraded":"2023-10-19T13:04:23.248444+0000","last_fullsized":"2023-10-19T13:04:23.248444+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:30:34.933118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.26","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.258587+0000","last_change":"2023-10-19T13:04:23.250036+0000","last_active":"2023-10-19T13:04:23.258587+0000","last_peered":"2023-10-19T13:04:23.258587+0000","last_clean":"2023-10-19T13:04:23.258587+0000","last_became_active":"2023-10-19T13:04:23.249602+0000","last_became_peered":"2023-10-19T13:04:23.249602+0000","last_unstale":"2023-10-19T13:04:23.258587+0000","last_undegraded":"2023-10-19T13:04:23.258587+0000","last_fullsized":"2023-10-19T13:04:23.258587+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:03:26.256383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.25","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.257017+0000","last_change":"2023-10-19T13:04:23.253529+0000","last_active":"2023-10-19T13:04:23.257017+0000","last_peered":"2023-10-19T13:04:23.257017+0000","last_clean":"2023-10-19T13:04:23.257017+0000","last_became_active":"2023-10-19T13:04:23.253265+0000","last_became_peered":"2023-10-19T13:04:23.253265+0000","last_unstale":"2023-10-19T13:04:23.257017+0000","last_undegraded":"2023-10-19T13:04:23.257017+0000","last_fullsized":"2023-10-19T13:04:23.257017+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:20:45.859782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.24","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.258512+0000","last_change":"2023-10-19T13:04:23.249000+0000","last_active":"2023-10-19T13:04:23.258512+0000","last_peered":"2023-10-19T13:04:23.258512+0000","last_clean":"2023-10-19T13:04:23.258512+0000","last_became_active":"2023-10-19T13:04:23.248580+0000","last_became_peered":"2023-10-19T13:04:23.248580+0000","last_unstale":"2023-10-19T13:04:23.258512+0000","last_undegraded":"2023-10-19T13:04:23.258512+0000","last_fullsized":"2023-10-19T13:04:23.258512+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:23:04.278774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.23","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.258771+0000","last_change":"2023-10-19T13:04:23.254509+0000","last_active":"2023-10-19T13:04:23.258771+0000","last_peered":"2023-10-19T13:04:23.258771+0000","last_clean":"2023-10-19T13:04:23.258771+0000","last_became_active":"2023-10-19T13:04:23.254180+0000","last_became_peered":"2023-10-19T13:04:23.254180+0000","last_unstale":"2023-10-19T13:04:23.258771+0000","last_undegraded":"2023-10-19T13:04:23.258771+0000","last_fullsized":"2023-10-19T13:04:23.258771+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:54:48.817501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.22","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.254396+0000","last_change":"2023-10-19T13:04:23.245440+0000","last_active":"2023-10-19T13:04:23.254396+0000","last_peered":"2023-10-19T13:04:23.254396+0000","last_clean":"2023-10-19T13:04:23.254396+0000","last_became_active":"2023-10-19T13:04:23.244956+0000","last_became_peered":"2023-10-19T13:04:23.244956+0000","last_unstale":"2023-10-19T13:04:23.254396+0000","last_undegraded":"2023-10-19T13:04:23.254396+0000","last_fullsized":"2023-10-19T13:04:23.254396+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:42:32.750724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.21","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.243656+0000","last_change":"2023-10-19T13:04:23.242180+0000","last_active":"2023-10-19T13:04:23.243656+0000","last_peered":"2023-10-19T13:04:23.243656+0000","last_clean":"2023-10-19T13:04:23.243656+0000","last_became_active":"2023-10-19T13:04:23.241890+0000","last_became_peered":"2023-10-19T13:04:23.241890+0000","last_unstale":"2023-10-19T13:04:23.243656+0000","last_undegraded":"2023-10-19T13:04:23.243656+0000","last_fullsized":"2023-10-19T13:04:23.243656+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:13:41.778030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.20","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.245859+0000","last_change":"2023-10-19T13:04:23.245272+0000","last_active":"2023-10-19T13:04:23.245859+0000","last_peered":"2023-10-19T13:04:23.245859+0000","last_clean":"2023-10-19T13:04:23.245859+0000","last_became_active":"2023-10-19T13:04:23.245017+0000","last_became_peered":"2023-10-19T13:04:23.245017+0000","last_unstale":"2023-10-19T13:04:23.245859+0000","last_undegraded":"2023-10-19T13:04:23.245859+0000","last_fullsized":"2023-10-19T13:04:23.245859+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:20:38.292322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1c","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.258810+0000","last_change":"2023-10-19T13:04:23.250759+0000","last_active":"2023-10-19T13:04:23.258810+0000","last_peered":"2023-10-19T13:04:23.258810+0000","last_clean":"2023-10-19T13:04:23.258810+0000","last_became_active":"2023-10-19T13:04:23.250487+0000","last_became_peered":"2023-10-19T13:04:23.250487+0000","last_unstale":"2023-10-19T13:04:23.258810+0000","last_undegraded":"2023-10-19T13:04:23.258810+0000","last_fullsized":"2023-10-19T13:04:23.258810+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:20:45.859782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1f","version":"143'4","reported_seq":15,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.430345+0000","last_change":"2023-10-19T13:04:22.440574+0000","last_active":"2023-10-19T13:04:23.430345+0000","last_peered":"2023-10-19T13:04:23.430345+0000","last_clean":"2023-10-19T13:04:23.430345+0000","last_became_active":"2023-10-19T13:04:22.440310+0000","last_became_peered":"2023-10-19T13:04:22.440310+0000","last_unstale":"2023-10-19T13:04:23.430345+0000","last_undegraded":"2023-10-19T13:04:23.430345+0000","last_fullsized":"2023-10-19T13:04:23.430345+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_clean_scrub_stamp":"2023-10-19T13:04:21.401563+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:22:12.862717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1d","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.258872+0000","last_change":"2023-10-19T13:04:23.255051+0000","last_active":"2023-10-19T13:04:23.258872+0000","last_peered":"2023-10-19T13:04:23.258872+0000","last_clean":"2023-10-19T13:04:23.258872+0000","last_became_active":"2023-10-19T13:04:23.254731+0000","last_became_peered":"2023-10-19T13:04:23.254731+0000","last_unstale":"2023-10-19T13:04:23.258872+0000","last_undegraded":"2023-10-19T13:04:23.258872+0000","last_fullsized":"2023-10-19T13:04:23.258872+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:34:18.198764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1e","version":"143'1","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.429585+0000","last_change":"2023-10-19T13:04:22.439332+0000","last_active":"2023-10-19T13:04:23.429585+0000","last_peered":"2023-10-19T13:04:23.429585+0000","last_clean":"2023-10-19T13:04:23.429585+0000","last_became_active":"2023-10-19T13:04:22.437985+0000","last_became_peered":"2023-10-19T13:04:22.437985+0000","last_unstale":"2023-10-19T13:04:23.429585+0000","last_undegraded":"2023-10-19T13:04:23.429585+0000","last_fullsized":"2023-10-19T13:04:23.429585+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_clean_scrub_stamp":"2023-10-19T13:04:21.401563+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:52:40.514630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.1e","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.256790+0000","last_change":"2023-10-19T13:04:23.254221+0000","last_active":"2023-10-19T13:04:23.256790+0000","last_peered":"2023-10-19T13:04:23.256790+0000","last_clean":"2023-10-19T13:04:23.256790+0000","last_became_active":"2023-10-19T13:04:23.253941+0000","last_became_peered":"2023-10-19T13:04:23.253941+0000","last_unstale":"2023-10-19T13:04:23.256790+0000","last_undegraded":"2023-10-19T13:04:23.256790+0000","last_fullsized":"2023-10-19T13:04:23.256790+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:58:24.553560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1d","version":"143'2","reported_seq":13,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.429667+0000","last_change":"2023-10-19T13:04:22.441364+0000","last_active":"2023-10-19T13:04:23.429667+0000","last_peered":"2023-10-19T13:04:23.429667+0000","last_clean":"2023-10-19T13:04:23.429667+0000","last_became_active":"2023-10-19T13:04:22.440928+0000","last_became_peered":"2023-10-19T13:04:22.440928+0000","last_unstale":"2023-10-19T13:04:23.429667+0000","last_undegraded":"2023-10-19T13:04:23.429667+0000","last_fullsized":"2023-10-19T13:04:23.429667+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_clean_scrub_stamp":"2023-10-19T13:04:21.401563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:30:33.923501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1f","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.252997+0000","last_change":"2023-10-19T13:04:23.250194+0000","last_active":"2023-10-19T13:04:23.252997+0000","last_peered":"2023-10-19T13:04:23.252997+0000","last_clean":"2023-10-19T13:04:23.252997+0000","last_became_active":"2023-10-19T13:04:23.249919+0000","last_became_peered":"2023-10-19T13:04:23.249919+0000","last_unstale":"2023-10-19T13:04:23.252997+0000","last_undegraded":"2023-10-19T13:04:23.252997+0000","last_fullsized":"2023-10-19T13:04:23.252997+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:52:21.807380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"143'2","reported_seq":13,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.432710+0000","last_change":"2023-10-19T13:04:22.443417+0000","last_active":"2023-10-19T13:04:23.432710+0000","last_peered":"2023-10-19T13:04:23.432710+0000","last_clean":"2023-10-19T13:04:23.432710+0000","last_became_active":"2023-10-19T13:04:22.443116+0000","last_became_peered":"2023-10-19T13:04:22.443116+0000","last_unstale":"2023-10-19T13:04:23.432710+0000","last_undegraded":"2023-10-19T13:04:23.432710+0000","last_fullsized":"2023-10-19T13:04:23.432710+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_clean_scrub_stamp":"2023-10-19T13:04:21.401563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:06:23.518443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.253449+0000","last_change":"2023-10-19T13:04:23.244883+0000","last_active":"2023-10-19T13:04:23.253449+0000","last_peered":"2023-10-19T13:04:23.253449+0000","last_clean":"2023-10-19T13:04:23.253449+0000","last_became_active":"2023-10-19T13:04:23.244621+0000","last_became_peered":"2023-10-19T13:04:23.244621+0000","last_unstale":"2023-10-19T13:04:23.253449+0000","last_undegraded":"2023-10-19T13:04:23.253449+0000","last_fullsized":"2023-10-19T13:04:23.253449+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:34:21.904012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1b","version":"143'1","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.432410+0000","last_change":"2023-10-19T13:04:22.442894+0000","last_active":"2023-10-19T13:04:23.432410+0000","last_peered":"2023-10-19T13:04:23.432410+0000","last_clean":"2023-10-19T13:04:23.432410+0000","last_became_active":"2023-10-19T13:04:22.442627+0000","last_became_peered":"2023-10-19T13:04:22.442627+0000","last_unstale":"2023-10-19T13:04:23.432410+0000","last_undegraded":"2023-10-19T13:04:23.432410+0000","last_fullsized":"2023-10-19T13:04:23.432410+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_clean_scrub_stamp":"2023-10-19T13:04:21.401563+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:00:49.819076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.247693+0000","last_change":"2023-10-19T13:04:23.246321+0000","last_active":"2023-10-19T13:04:23.247693+0000","last_peered":"2023-10-19T13:04:23.247693+0000","last_clean":"2023-10-19T13:04:23.247693+0000","last_became_active":"2023-10-19T13:04:23.246028+0000","last_became_peered":"2023-10-19T13:04:23.246028+0000","last_unstale":"2023-10-19T13:04:23.247693+0000","last_undegraded":"2023-10-19T13:04:23.247693+0000","last_fullsized":"2023-10-19T13:04:23.247693+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:52:41.524246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1a","version":"143'1","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.432691+0000","last_change":"2023-10-19T13:04:22.444403+0000","last_active":"2023-10-19T13:04:23.432691+0000","last_peered":"2023-10-19T13:04:23.432691+0000","last_clean":"2023-10-19T13:04:23.432691+0000","last_became_active":"2023-10-19T13:04:22.444072+0000","last_became_peered":"2023-10-19T13:04:22.444072+0000","last_unstale":"2023-10-19T13:04:23.432691+0000","last_undegraded":"2023-10-19T13:04:23.432691+0000","last_fullsized":"2023-10-19T13:04:23.432691+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_clean_scrub_stamp":"2023-10-19T13:04:21.401563+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:03:51.165898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.1a","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.253737+0000","last_change":"2023-10-19T13:04:23.242570+0000","last_active":"2023-10-19T13:04:23.253737+0000","last_peered":"2023-10-19T13:04:23.253737+0000","last_clean":"2023-10-19T13:04:23.253737+0000","last_became_active":"2023-10-19T13:04:23.242330+0000","last_became_peered":"2023-10-19T13:04:23.242330+0000","last_unstale":"2023-10-19T13:04:23.253737+0000","last_undegraded":"2023-10-19T13:04:23.253737+0000","last_fullsized":"2023-10-19T13:04:23.253737+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:26:52.277682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.231246+0000","last_change":"2023-10-19T13:04:22.439373+0000","last_active":"2023-10-19T13:04:23.231246+0000","last_peered":"2023-10-19T13:04:23.231246+0000","last_clean":"2023-10-19T13:04:23.231246+0000","last_became_active":"2023-10-19T13:04:22.438988+0000","last_became_peered":"2023-10-19T13:04:22.438988+0000","last_unstale":"2023-10-19T13:04:23.231246+0000","last_undegraded":"2023-10-19T13:04:23.231246+0000","last_fullsized":"2023-10-19T13:04:23.231246+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_clean_scrub_stamp":"2023-10-19T13:04:21.401563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:56:59.711452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1b","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.253533+0000","last_change":"2023-10-19T13:04:23.243508+0000","last_active":"2023-10-19T13:04:23.253533+0000","last_peered":"2023-10-19T13:04:23.253533+0000","last_clean":"2023-10-19T13:04:23.253533+0000","last_became_active":"2023-10-19T13:04:23.243199+0000","last_became_peered":"2023-10-19T13:04:23.243199+0000","last_unstale":"2023-10-19T13:04:23.253533+0000","last_undegraded":"2023-10-19T13:04:23.253533+0000","last_fullsized":"2023-10-19T13:04:23.253533+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:04:15.321219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.242695+0000","last_change":"2023-10-19T13:04:22.436803+0000","last_active":"2023-10-19T13:04:23.242695+0000","last_peered":"2023-10-19T13:04:23.242695+0000","last_clean":"2023-10-19T13:04:23.242695+0000","last_became_active":"2023-10-19T13:04:22.436550+0000","last_became_peered":"2023-10-19T13:04:22.436550+0000","last_unstale":"2023-10-19T13:04:23.242695+0000","last_undegraded":"2023-10-19T13:04:23.242695+0000","last_fullsized":"2023-10-19T13:04:23.242695+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_clean_scrub_stamp":"2023-10-19T13:04:21.401563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:23:11.451513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.14","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.256870+0000","last_change":"2023-10-19T13:04:23.251227+0000","last_active":"2023-10-19T13:04:23.256870+0000","last_peered":"2023-10-19T13:04:23.256870+0000","last_clean":"2023-10-19T13:04:23.256870+0000","last_became_active":"2023-10-19T13:04:23.250906+0000","last_became_peered":"2023-10-19T13:04:23.250906+0000","last_unstale":"2023-10-19T13:04:23.256870+0000","last_undegraded":"2023-10-19T13:04:23.256870+0000","last_fullsized":"2023-10-19T13:04:23.256870+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:06:24.528059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.241047+0000","last_change":"2023-10-19T13:04:22.448426+0000","last_active":"2023-10-19T13:04:23.241047+0000","last_peered":"2023-10-19T13:04:23.241047+0000","last_clean":"2023-10-19T13:04:23.241047+0000","last_became_active":"2023-10-19T13:04:22.448121+0000","last_became_peered":"2023-10-19T13:04:22.448121+0000","last_unstale":"2023-10-19T13:04:23.241047+0000","last_undegraded":"2023-10-19T13:04:23.241047+0000","last_fullsized":"2023-10-19T13:04:23.241047+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_clean_scrub_stamp":"2023-10-19T13:04:21.401563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:48:29.693840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.15","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.245588+0000","last_change":"2023-10-19T13:04:23.241676+0000","last_active":"2023-10-19T13:04:23.245588+0000","last_peered":"2023-10-19T13:04:23.245588+0000","last_clean":"2023-10-19T13:04:23.245588+0000","last_became_active":"2023-10-19T13:04:23.241372+0000","last_became_peered":"2023-10-19T13:04:23.241372+0000","last_unstale":"2023-10-19T13:04:23.245588+0000","last_undegraded":"2023-10-19T13:04:23.245588+0000","last_fullsized":"2023-10-19T13:04:23.245588+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:52:21.807380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.230451+0000","last_change":"2023-10-19T13:04:22.440265+0000","last_active":"2023-10-19T13:04:23.230451+0000","last_peered":"2023-10-19T13:04:23.230451+0000","last_clean":"2023-10-19T13:04:23.230451+0000","last_became_active":"2023-10-19T13:04:22.439995+0000","last_became_peered":"2023-10-19T13:04:22.439995+0000","last_unstale":"2023-10-19T13:04:23.230451+0000","last_undegraded":"2023-10-19T13:04:23.230451+0000","last_fullsized":"2023-10-19T13:04:23.230451+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_clean_scrub_stamp":"2023-10-19T13:04:21.401563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:00:49.460528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.254415+0000","last_change":"2023-10-19T13:04:23.246009+0000","last_active":"2023-10-19T13:04:23.254415+0000","last_peered":"2023-10-19T13:04:23.254415+0000","last_clean":"2023-10-19T13:04:23.254415+0000","last_became_active":"2023-10-19T13:04:23.245738+0000","last_became_peered":"2023-10-19T13:04:23.245738+0000","last_unstale":"2023-10-19T13:04:23.254415+0000","last_undegraded":"2023-10-19T13:04:23.254415+0000","last_fullsized":"2023-10-19T13:04:23.254415+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:10:45.863043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.15","version":"143'3","reported_seq":14,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.430874+0000","last_change":"2023-10-19T13:04:22.439823+0000","last_active":"2023-10-19T13:04:23.430874+0000","last_peered":"2023-10-19T13:04:23.430874+0000","last_clean":"2023-10-19T13:04:23.430874+0000","last_became_active":"2023-10-19T13:04:22.439492+0000","last_became_peered":"2023-10-19T13:04:22.439492+0000","last_unstale":"2023-10-19T13:04:23.430874+0000","last_undegraded":"2023-10-19T13:04:23.430874+0000","last_fullsized":"2023-10-19T13:04:23.430874+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_clean_scrub_stamp":"2023-10-19T13:04:21.401563+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:27:12.520515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.245895+0000","last_change":"2023-10-19T13:04:23.244129+0000","last_active":"2023-10-19T13:04:23.245895+0000","last_peered":"2023-10-19T13:04:23.245895+0000","last_clean":"2023-10-19T13:04:23.245895+0000","last_became_active":"2023-10-19T13:04:23.243865+0000","last_became_peered":"2023-10-19T13:04:23.243865+0000","last_unstale":"2023-10-19T13:04:23.245895+0000","last_undegraded":"2023-10-19T13:04:23.245895+0000","last_fullsized":"2023-10-19T13:04:23.245895+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:18:30.649624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.14","version":"143'6","reported_seq":17,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.434547+0000","last_change":"2023-10-19T13:04:22.442680+0000","last_active":"2023-10-19T13:04:23.434547+0000","last_peered":"2023-10-19T13:04:23.434547+0000","last_clean":"2023-10-19T13:04:23.434547+0000","last_became_active":"2023-10-19T13:04:22.442413+0000","last_became_peered":"2023-10-19T13:04:22.442413+0000","last_unstale":"2023-10-19T13:04:23.434547+0000","last_undegraded":"2023-10-19T13:04:23.434547+0000","last_fullsized":"2023-10-19T13:04:23.434547+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_clean_scrub_stamp":"2023-10-19T13:04:21.401563+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:20:13.314826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.10","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.244301+0000","last_change":"2023-10-19T13:04:23.243720+0000","last_active":"2023-10-19T13:04:23.244301+0000","last_peered":"2023-10-19T13:04:23.244301+0000","last_clean":"2023-10-19T13:04:23.244301+0000","last_became_active":"2023-10-19T13:04:23.243308+0000","last_became_peered":"2023-10-19T13:04:23.243308+0000","last_unstale":"2023-10-19T13:04:23.244301+0000","last_undegraded":"2023-10-19T13:04:23.244301+0000","last_fullsized":"2023-10-19T13:04:23.244301+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:47:05.764939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.13","version":"143'2","reported_seq":13,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.432465+0000","last_change":"2023-10-19T13:04:22.442994+0000","last_active":"2023-10-19T13:04:23.432465+0000","last_peered":"2023-10-19T13:04:23.432465+0000","last_clean":"2023-10-19T13:04:23.432465+0000","last_became_active":"2023-10-19T13:04:22.442649+0000","last_became_peered":"2023-10-19T13:04:22.442649+0000","last_unstale":"2023-10-19T13:04:23.432465+0000","last_undegraded":"2023-10-19T13:04:23.432465+0000","last_fullsized":"2023-10-19T13:04:23.432465+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_clean_scrub_stamp":"2023-10-19T13:04:21.401563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:21:17.556322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.253693+0000","last_change":"2023-10-19T13:04:23.246917+0000","last_active":"2023-10-19T13:04:23.253693+0000","last_peered":"2023-10-19T13:04:23.253693+0000","last_clean":"2023-10-19T13:04:23.253693+0000","last_became_active":"2023-10-19T13:04:23.246485+0000","last_became_peered":"2023-10-19T13:04:23.246485+0000","last_unstale":"2023-10-19T13:04:23.253693+0000","last_undegraded":"2023-10-19T13:04:23.253693+0000","last_fullsized":"2023-10-19T13:04:23.253693+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:47:05.764939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.235097+0000","last_change":"2023-10-19T13:04:22.443482+0000","last_active":"2023-10-19T13:04:23.235097+0000","last_peered":"2023-10-19T13:04:23.235097+0000","last_clean":"2023-10-19T13:04:23.235097+0000","last_became_active":"2023-10-19T13:04:22.443193+0000","last_became_peered":"2023-10-19T13:04:22.443193+0000","last_unstale":"2023-10-19T13:04:23.235097+0000","last_undegraded":"2023-10-19T13:04:23.235097+0000","last_fullsized":"2023-10-19T13:04:23.235097+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_clean_scrub_stamp":"2023-10-19T13:04:21.401563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:55:13.479682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.253981+0000","last_change":"2023-10-19T13:04:23.248278+0000","last_active":"2023-10-19T13:04:23.253981+0000","last_peered":"2023-10-19T13:04:23.253981+0000","last_clean":"2023-10-19T13:04:23.253981+0000","last_became_active":"2023-10-19T13:04:23.247991+0000","last_became_peered":"2023-10-19T13:04:23.247991+0000","last_unstale":"2023-10-19T13:04:23.253981+0000","last_undegraded":"2023-10-19T13:04:23.253981+0000","last_fullsized":"2023-10-19T13:04:23.253981+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:55:07.236723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.234786+0000","last_change":"2023-10-19T13:04:22.442674+0000","last_active":"2023-10-19T13:04:23.234786+0000","last_peered":"2023-10-19T13:04:23.234786+0000","last_clean":"2023-10-19T13:04:23.234786+0000","last_became_active":"2023-10-19T13:04:22.442236+0000","last_became_peered":"2023-10-19T13:04:22.442236+0000","last_unstale":"2023-10-19T13:04:23.234786+0000","last_undegraded":"2023-10-19T13:04:23.234786+0000","last_fullsized":"2023-10-19T13:04:23.234786+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_clean_scrub_stamp":"2023-10-19T13:04:21.401563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:44:07.324452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.254283+0000","last_change":"2023-10-19T13:04:23.251683+0000","last_active":"2023-10-19T13:04:23.254283+0000","last_peered":"2023-10-19T13:04:23.254283+0000","last_clean":"2023-10-19T13:04:23.254283+0000","last_became_active":"2023-10-19T13:04:23.251427+0000","last_became_peered":"2023-10-19T13:04:23.251427+0000","last_unstale":"2023-10-19T13:04:23.254283+0000","last_undegraded":"2023-10-19T13:04:23.254283+0000","last_fullsized":"2023-10-19T13:04:23.254283+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:01:25.278083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.f","version":"143'1","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.431966+0000","last_change":"2023-10-19T13:04:22.438449+0000","last_active":"2023-10-19T13:04:23.431966+0000","last_peered":"2023-10-19T13:04:23.431966+0000","last_clean":"2023-10-19T13:04:23.431966+0000","last_became_active":"2023-10-19T13:04:22.438168+0000","last_became_peered":"2023-10-19T13:04:22.438168+0000","last_unstale":"2023-10-19T13:04:23.431966+0000","last_undegraded":"2023-10-19T13:04:23.431966+0000","last_fullsized":"2023-10-19T13:04:23.431966+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_clean_scrub_stamp":"2023-10-19T13:04:21.401563+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:17:53.955463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.d","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.256734+0000","last_change":"2023-10-19T13:04:23.251516+0000","last_active":"2023-10-19T13:04:23.256734+0000","last_peered":"2023-10-19T13:04:23.256734+0000","last_clean":"2023-10-19T13:04:23.256734+0000","last_became_active":"2023-10-19T13:04:23.251247+0000","last_became_peered":"2023-10-19T13:04:23.251247+0000","last_unstale":"2023-10-19T13:04:23.256734+0000","last_undegraded":"2023-10-19T13:04:23.256734+0000","last_fullsized":"2023-10-19T13:04:23.256734+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:47:05.764939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.e","version":"143'2","reported_seq":13,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.432208+0000","last_change":"2023-10-19T13:04:22.435568+0000","last_active":"2023-10-19T13:04:23.432208+0000","last_peered":"2023-10-19T13:04:23.432208+0000","last_clean":"2023-10-19T13:04:23.432208+0000","last_became_active":"2023-10-19T13:04:22.435294+0000","last_became_peered":"2023-10-19T13:04:22.435294+0000","last_unstale":"2023-10-19T13:04:23.432208+0000","last_undegraded":"2023-10-19T13:04:23.432208+0000","last_fullsized":"2023-10-19T13:04:23.432208+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_clean_scrub_stamp":"2023-10-19T13:04:21.401563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:39:21.359154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.e","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.258757+0000","last_change":"2023-10-19T13:04:23.251902+0000","last_active":"2023-10-19T13:04:23.258757+0000","last_peered":"2023-10-19T13:04:23.258757+0000","last_clean":"2023-10-19T13:04:23.258757+0000","last_became_active":"2023-10-19T13:04:23.251679+0000","last_became_peered":"2023-10-19T13:04:23.251679+0000","last_unstale":"2023-10-19T13:04:23.258757+0000","last_undegraded":"2023-10-19T13:04:23.258757+0000","last_fullsized":"2023-10-19T13:04:23.258757+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:45:02.121585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.d","version":"143'1","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.430308+0000","last_change":"2023-10-19T13:04:22.441163+0000","last_active":"2023-10-19T13:04:23.430308+0000","last_peered":"2023-10-19T13:04:23.430308+0000","last_clean":"2023-10-19T13:04:23.430308+0000","last_became_active":"2023-10-19T13:04:22.440541+0000","last_became_peered":"2023-10-19T13:04:22.440541+0000","last_unstale":"2023-10-19T13:04:23.430308+0000","last_undegraded":"2023-10-19T13:04:23.430308+0000","last_fullsized":"2023-10-19T13:04:23.430308+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_clean_scrub_stamp":"2023-10-19T13:04:21.401563+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:27:12.520515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.f","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.249063+0000","last_change":"2023-10-19T13:04:23.248187+0000","last_active":"2023-10-19T13:04:23.249063+0000","last_peered":"2023-10-19T13:04:23.249063+0000","last_clean":"2023-10-19T13:04:23.249063+0000","last_became_active":"2023-10-19T13:04:23.247299+0000","last_became_peered":"2023-10-19T13:04:23.247299+0000","last_unstale":"2023-10-19T13:04:23.249063+0000","last_undegraded":"2023-10-19T13:04:23.249063+0000","last_fullsized":"2023-10-19T13:04:23.249063+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:43:05.064367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.239762+0000","last_change":"2023-10-19T13:04:22.448640+0000","last_active":"2023-10-19T13:04:23.239762+0000","last_peered":"2023-10-19T13:04:23.239762+0000","last_clean":"2023-10-19T13:04:23.239762+0000","last_became_active":"2023-10-19T13:04:22.448367+0000","last_became_peered":"2023-10-19T13:04:22.448367+0000","last_unstale":"2023-10-19T13:04:23.239762+0000","last_undegraded":"2023-10-19T13:04:23.239762+0000","last_fullsized":"2023-10-19T13:04:23.239762+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_clean_scrub_stamp":"2023-10-19T13:04:21.401563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:13:40.768413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.253397+0000","last_change":"2023-10-19T13:04:23.245918+0000","last_active":"2023-10-19T13:04:23.253397+0000","last_peered":"2023-10-19T13:04:23.253397+0000","last_clean":"2023-10-19T13:04:23.253397+0000","last_became_active":"2023-10-19T13:04:23.245649+0000","last_became_peered":"2023-10-19T13:04:23.245649+0000","last_unstale":"2023-10-19T13:04:23.253397+0000","last_undegraded":"2023-10-19T13:04:23.253397+0000","last_fullsized":"2023-10-19T13:04:23.253397+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:39:23.090865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.b","version":"143'2","reported_seq":13,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.431626+0000","last_change":"2023-10-19T13:04:22.433508+0000","last_active":"2023-10-19T13:04:23.431626+0000","last_peered":"2023-10-19T13:04:23.431626+0000","last_clean":"2023-10-19T13:04:23.431626+0000","last_became_active":"2023-10-19T13:04:22.433228+0000","last_became_peered":"2023-10-19T13:04:22.433228+0000","last_unstale":"2023-10-19T13:04:23.431626+0000","last_undegraded":"2023-10-19T13:04:23.431626+0000","last_fullsized":"2023-10-19T13:04:23.431626+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_clean_scrub_stamp":"2023-10-19T13:04:21.401563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:33:03.294362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.250161+0000","last_change":"2023-10-19T13:04:23.239176+0000","last_active":"2023-10-19T13:04:23.250161+0000","last_peered":"2023-10-19T13:04:23.250161+0000","last_clean":"2023-10-19T13:04:23.250161+0000","last_became_active":"2023-10-19T13:04:23.238874+0000","last_became_peered":"2023-10-19T13:04:23.238874+0000","last_unstale":"2023-10-19T13:04:23.250161+0000","last_undegraded":"2023-10-19T13:04:23.250161+0000","last_fullsized":"2023-10-19T13:04:23.250161+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:03:08.516373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.239422+0000","last_change":"2023-10-19T13:04:22.438982+0000","last_active":"2023-10-19T13:04:23.239422+0000","last_peered":"2023-10-19T13:04:23.239422+0000","last_clean":"2023-10-19T13:04:23.239422+0000","last_became_active":"2023-10-19T13:04:22.438694+0000","last_became_peered":"2023-10-19T13:04:22.438694+0000","last_unstale":"2023-10-19T13:04:23.239422+0000","last_undegraded":"2023-10-19T13:04:23.239422+0000","last_fullsized":"2023-10-19T13:04:23.239422+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_clean_scrub_stamp":"2023-10-19T13:04:21.401563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:00:49.460528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.254376+0000","last_change":"2023-10-19T13:04:23.245254+0000","last_active":"2023-10-19T13:04:23.254376+0000","last_peered":"2023-10-19T13:04:23.254376+0000","last_clean":"2023-10-19T13:04:23.254376+0000","last_became_active":"2023-10-19T13:04:23.244623+0000","last_became_peered":"2023-10-19T13:04:23.244623+0000","last_unstale":"2023-10-19T13:04:23.254376+0000","last_undegraded":"2023-10-19T13:04:23.254376+0000","last_fullsized":"2023-10-19T13:04:23.254376+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:22:46.466822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.234928+0000","last_change":"2023-10-19T13:04:22.448172+0000","last_active":"2023-10-19T13:04:23.234928+0000","last_peered":"2023-10-19T13:04:23.234928+0000","last_clean":"2023-10-19T13:04:23.234928+0000","last_became_active":"2023-10-19T13:04:22.447872+0000","last_became_peered":"2023-10-19T13:04:22.447872+0000","last_unstale":"2023-10-19T13:04:23.234928+0000","last_undegraded":"2023-10-19T13:04:23.234928+0000","last_fullsized":"2023-10-19T13:04:23.234928+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_clean_scrub_stamp":"2023-10-19T13:04:21.401563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:53:16.490214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.b","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.246801+0000","last_change":"2023-10-19T13:04:23.245772+0000","last_active":"2023-10-19T13:04:23.246801+0000","last_peered":"2023-10-19T13:04:23.246801+0000","last_clean":"2023-10-19T13:04:23.246801+0000","last_became_active":"2023-10-19T13:04:23.245523+0000","last_became_peered":"2023-10-19T13:04:23.245523+0000","last_unstale":"2023-10-19T13:04:23.246801+0000","last_undegraded":"2023-10-19T13:04:23.246801+0000","last_fullsized":"2023-10-19T13:04:23.246801+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:00:22.625725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.240768+0000","last_change":"2023-10-19T13:04:22.439768+0000","last_active":"2023-10-19T13:04:23.240768+0000","last_peered":"2023-10-19T13:04:23.240768+0000","last_clean":"2023-10-19T13:04:23.240768+0000","last_became_active":"2023-10-19T13:04:22.438538+0000","last_became_peered":"2023-10-19T13:04:22.438538+0000","last_unstale":"2023-10-19T13:04:23.240768+0000","last_undegraded":"2023-10-19T13:04:23.240768+0000","last_fullsized":"2023-10-19T13:04:23.240768+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_clean_scrub_stamp":"2023-10-19T13:04:21.401563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:14:52.753658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.258684+0000","last_change":"2023-10-19T13:04:23.249836+0000","last_active":"2023-10-19T13:04:23.258684+0000","last_peered":"2023-10-19T13:04:23.258684+0000","last_clean":"2023-10-19T13:04:23.258684+0000","last_became_active":"2023-10-19T13:04:23.249592+0000","last_became_peered":"2023-10-19T13:04:23.249592+0000","last_unstale":"2023-10-19T13:04:23.258684+0000","last_undegraded":"2023-10-19T13:04:23.258684+0000","last_fullsized":"2023-10-19T13:04:23.258684+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:51:56.835268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.7","version":"143'2","reported_seq":13,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.431713+0000","last_change":"2023-10-19T13:04:22.435611+0000","last_active":"2023-10-19T13:04:23.431713+0000","last_peered":"2023-10-19T13:04:23.431713+0000","last_clean":"2023-10-19T13:04:23.431713+0000","last_became_active":"2023-10-19T13:04:22.435339+0000","last_became_peered":"2023-10-19T13:04:22.435339+0000","last_unstale":"2023-10-19T13:04:23.431713+0000","last_undegraded":"2023-10-19T13:04:23.431713+0000","last_fullsized":"2023-10-19T13:04:23.431713+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_clean_scrub_stamp":"2023-10-19T13:04:21.401563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:10:14.617741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.253496+0000","last_change":"2023-10-19T13:04:23.244187+0000","last_active":"2023-10-19T13:04:23.253496+0000","last_peered":"2023-10-19T13:04:23.253496+0000","last_clean":"2023-10-19T13:04:23.253496+0000","last_became_active":"2023-10-19T13:04:23.243871+0000","last_became_peered":"2023-10-19T13:04:23.243871+0000","last_unstale":"2023-10-19T13:04:23.253496+0000","last_undegraded":"2023-10-19T13:04:23.253496+0000","last_fullsized":"2023-10-19T13:04:23.253496+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:08:34.773183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.241843+0000","last_change":"2023-10-19T13:04:22.447531+0000","last_active":"2023-10-19T13:04:23.241843+0000","last_peered":"2023-10-19T13:04:23.241843+0000","last_clean":"2023-10-19T13:04:23.241843+0000","last_became_active":"2023-10-19T13:04:22.447291+0000","last_became_peered":"2023-10-19T13:04:22.447291+0000","last_unstale":"2023-10-19T13:04:23.241843+0000","last_undegraded":"2023-10-19T13:04:23.241843+0000","last_fullsized":"2023-10-19T13:04:23.241843+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_clean_scrub_stamp":"2023-10-19T13:04:21.401563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:00:49.819076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.252447+0000","last_change":"2023-10-19T13:04:23.249549+0000","last_active":"2023-10-19T13:04:23.252447+0000","last_peered":"2023-10-19T13:04:23.252447+0000","last_clean":"2023-10-19T13:04:23.252447+0000","last_became_active":"2023-10-19T13:04:23.249189+0000","last_became_peered":"2023-10-19T13:04:23.249189+0000","last_unstale":"2023-10-19T13:04:23.252447+0000","last_undegraded":"2023-10-19T13:04:23.252447+0000","last_fullsized":"2023-10-19T13:04:23.252447+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:57:00.721069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.239566+0000","last_change":"2023-10-19T13:04:22.437924+0000","last_active":"2023-10-19T13:04:23.239566+0000","last_peered":"2023-10-19T13:04:23.239566+0000","last_clean":"2023-10-19T13:04:23.239566+0000","last_became_active":"2023-10-19T13:04:22.437639+0000","last_became_peered":"2023-10-19T13:04:22.437639+0000","last_unstale":"2023-10-19T13:04:23.239566+0000","last_undegraded":"2023-10-19T13:04:23.239566+0000","last_fullsized":"2023-10-19T13:04:23.239566+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_clean_scrub_stamp":"2023-10-19T13:04:21.401563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:04:14.311603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.7","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.249152+0000","last_change":"2023-10-19T13:04:23.247801+0000","last_active":"2023-10-19T13:04:23.249152+0000","last_peered":"2023-10-19T13:04:23.249152+0000","last_clean":"2023-10-19T13:04:23.249152+0000","last_became_active":"2023-10-19T13:04:23.247543+0000","last_became_peered":"2023-10-19T13:04:23.247543+0000","last_unstale":"2023-10-19T13:04:23.249152+0000","last_undegraded":"2023-10-19T13:04:23.249152+0000","last_fullsized":"2023-10-19T13:04:23.249152+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:21:44.668299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.242489+0000","last_change":"2023-10-19T13:04:22.435346+0000","last_active":"2023-10-19T13:04:23.242489+0000","last_peered":"2023-10-19T13:04:23.242489+0000","last_clean":"2023-10-19T13:04:23.242489+0000","last_became_active":"2023-10-19T13:04:22.434909+0000","last_became_peered":"2023-10-19T13:04:22.434909+0000","last_unstale":"2023-10-19T13:04:23.242489+0000","last_undegraded":"2023-10-19T13:04:23.242489+0000","last_fullsized":"2023-10-19T13:04:23.242489+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_clean_scrub_stamp":"2023-10-19T13:04:21.401563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:12:16.733002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.253416+0000","last_change":"2023-10-19T13:04:23.247396+0000","last_active":"2023-10-19T13:04:23.253416+0000","last_peered":"2023-10-19T13:04:23.253416+0000","last_clean":"2023-10-19T13:04:23.253416+0000","last_became_active":"2023-10-19T13:04:23.247133+0000","last_became_peered":"2023-10-19T13:04:23.247133+0000","last_unstale":"2023-10-19T13:04:23.253416+0000","last_undegraded":"2023-10-19T13:04:23.253416+0000","last_fullsized":"2023-10-19T13:04:23.253416+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:59:04.607606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.240021+0000","last_change":"2023-10-19T13:04:22.439734+0000","last_active":"2023-10-19T13:04:23.240021+0000","last_peered":"2023-10-19T13:04:23.240021+0000","last_clean":"2023-10-19T13:04:23.240021+0000","last_became_active":"2023-10-19T13:04:22.439471+0000","last_became_peered":"2023-10-19T13:04:22.439471+0000","last_unstale":"2023-10-19T13:04:23.240021+0000","last_undegraded":"2023-10-19T13:04:23.240021+0000","last_fullsized":"2023-10-19T13:04:23.240021+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_clean_scrub_stamp":"2023-10-19T13:04:21.401563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:02:52.376821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.254450+0000","last_change":"2023-10-19T13:04:23.246875+0000","last_active":"2023-10-19T13:04:23.254450+0000","last_peered":"2023-10-19T13:04:23.254450+0000","last_clean":"2023-10-19T13:04:23.254450+0000","last_became_active":"2023-10-19T13:04:23.246355+0000","last_became_peered":"2023-10-19T13:04:23.246355+0000","last_unstale":"2023-10-19T13:04:23.254450+0000","last_undegraded":"2023-10-19T13:04:23.254450+0000","last_fullsized":"2023-10-19T13:04:23.254450+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:16:20.228806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.239404+0000","last_change":"2023-10-19T13:04:22.435164+0000","last_active":"2023-10-19T13:04:23.239404+0000","last_peered":"2023-10-19T13:04:23.239404+0000","last_clean":"2023-10-19T13:04:23.239404+0000","last_became_active":"2023-10-19T13:04:22.434903+0000","last_became_peered":"2023-10-19T13:04:22.434903+0000","last_unstale":"2023-10-19T13:04:23.239404+0000","last_undegraded":"2023-10-19T13:04:23.239404+0000","last_fullsized":"2023-10-19T13:04:23.239404+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_clean_scrub_stamp":"2023-10-19T13:04:21.401563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:22:12.862717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.253041+0000","last_change":"2023-10-19T13:04:23.250788+0000","last_active":"2023-10-19T13:04:23.253041+0000","last_peered":"2023-10-19T13:04:23.253041+0000","last_clean":"2023-10-19T13:04:23.253041+0000","last_became_active":"2023-10-19T13:04:23.250521+0000","last_became_peered":"2023-10-19T13:04:23.250521+0000","last_unstale":"2023-10-19T13:04:23.253041+0000","last_undegraded":"2023-10-19T13:04:23.253041+0000","last_fullsized":"2023-10-19T13:04:23.253041+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:18:30.649624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.241756+0000","last_change":"2023-10-19T13:04:22.437643+0000","last_active":"2023-10-19T13:04:23.241756+0000","last_peered":"2023-10-19T13:04:23.241756+0000","last_clean":"2023-10-19T13:04:23.241756+0000","last_became_active":"2023-10-19T13:04:22.437351+0000","last_became_peered":"2023-10-19T13:04:22.437351+0000","last_unstale":"2023-10-19T13:04:23.241756+0000","last_undegraded":"2023-10-19T13:04:23.241756+0000","last_fullsized":"2023-10-19T13:04:23.241756+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_clean_scrub_stamp":"2023-10-19T13:04:21.401563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:40:27.499691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.253572+0000","last_change":"2023-10-19T13:04:23.244134+0000","last_active":"2023-10-19T13:04:23.253572+0000","last_peered":"2023-10-19T13:04:23.253572+0000","last_clean":"2023-10-19T13:04:23.253572+0000","last_became_active":"2023-10-19T13:04:23.243815+0000","last_became_peered":"2023-10-19T13:04:23.243815+0000","last_unstale":"2023-10-19T13:04:23.253572+0000","last_undegraded":"2023-10-19T13:04:23.253572+0000","last_fullsized":"2023-10-19T13:04:23.253572+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:22:13.872334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.240907+0000","last_change":"2023-10-19T13:04:22.434454+0000","last_active":"2023-10-19T13:04:23.240907+0000","last_peered":"2023-10-19T13:04:23.240907+0000","last_clean":"2023-10-19T13:04:23.240907+0000","last_became_active":"2023-10-19T13:04:22.434174+0000","last_became_peered":"2023-10-19T13:04:22.434174+0000","last_unstale":"2023-10-19T13:04:23.240907+0000","last_undegraded":"2023-10-19T13:04:23.240907+0000","last_fullsized":"2023-10-19T13:04:23.240907+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_clean_scrub_stamp":"2023-10-19T13:04:21.401563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:55:52.552633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.253496+0000","last_change":"2023-10-19T13:04:23.251854+0000","last_active":"2023-10-19T13:04:23.253496+0000","last_peered":"2023-10-19T13:04:23.253496+0000","last_clean":"2023-10-19T13:04:23.253496+0000","last_became_active":"2023-10-19T13:04:23.250574+0000","last_became_peered":"2023-10-19T13:04:23.250574+0000","last_unstale":"2023-10-19T13:04:23.253496+0000","last_undegraded":"2023-10-19T13:04:23.253496+0000","last_fullsized":"2023-10-19T13:04:23.253496+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:50:32.833828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.10","version":"143'8","reported_seq":19,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.432869+0000","last_change":"2023-10-19T13:04:22.442776+0000","last_active":"2023-10-19T13:04:23.432869+0000","last_peered":"2023-10-19T13:04:23.432869+0000","last_clean":"2023-10-19T13:04:23.432869+0000","last_became_active":"2023-10-19T13:04:22.442359+0000","last_became_peered":"2023-10-19T13:04:22.442359+0000","last_unstale":"2023-10-19T13:04:23.432869+0000","last_undegraded":"2023-10-19T13:04:23.432869+0000","last_fullsized":"2023-10-19T13:04:23.432869+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_clean_scrub_stamp":"2023-10-19T13:04:21.401563+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:35:46.829987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"132'3065","reported_seq":3551,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-19T13:04:22.429456+0000","last_change":"2023-10-19T12:44:08.636613+0000","last_active":"2023-10-19T13:04:22.429456+0000","last_peered":"2023-10-19T13:04:22.429456+0000","last_clean":"2023-10-19T13:04:22.429456+0000","last_became_active":"2023-10-19T12:44:08.635697+0000","last_became_peered":"2023-10-19T12:44:08.635697+0000","last_unstale":"2023-10-19T13:04:22.429456+0000","last_undegraded":"2023-10-19T13:04:22.429456+0000","last_fullsized":"2023-10-19T13:04:22.429456+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_clean_scrub_stamp":"2023-10-19T12:44:07.399738+0000","objects_scrubbed":0,"log_size":3065,"log_dups_size":0,"ondisk_log_size":3065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:15:23.685485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1154,"num_read_kb":2358,"num_write":369,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":18,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":17,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1154,"num_read_kb":2358,"num_write":369,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3065,"ondisk_log_size":3065,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575093,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55696,"kb_used_data":2304,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316144,"statfs":{"total":96636764160,"available":96579731456,"internally_reserved":0,"allocated":2359296,"data_stored":1482506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54655435},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 13:03:51 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53200000000000003,"15min":0.55900000000000005},"min":{"1min":0.36199999999999999,"5min":0.224,"15min":0.224},"max":{"1min":0.69399999999999995,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56499999999999995,"15min":0.58299999999999996},"min":{"1min":0.45200000000000001,"5min":0.33200000000000002,"15min":0.29599999999999999},"max":{"1min":0.71099999999999997,"5min":0.78200000000000003,"15min":0.93700000000000006},"last":0.503}]},{"osd":1,"last update":"Thu Oct 19 13:03:51 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.55000000000000004,"15min":0.57299999999999995},"min":{"1min":0.45700000000000002,"5min":0.33500000000000002,"15min":0.28199999999999997},"max":{"1min":0.82399999999999995,"5min":0.84299999999999997,"15min":1.3129999999999999},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.56100000000000005,"15min":0.58099999999999996},"min":{"1min":0.39100000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.72999999999999998,"5min":0.85499999999999998,"15min":1.284},"last":0.42599999999999999}]},{"osd":2,"last update":"Thu Oct 19 13:03:51 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.60799999999999998,"15min":0.623},"min":{"1min":0.53200000000000003,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.874,"5min":0.89200000000000002,"15min":0.98599999999999999},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.61099999999999999,"15min":0.63},"min":{"1min":0.52100000000000002,"5min":0.40300000000000002,"15min":0.28399999999999997},"max":{"1min":0.749,"5min":0.83699999999999997,"15min":1.2589999999999999},"last":0.59499999999999997}]},{"osd":3,"last update":"Thu Oct 19 13:03:51 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.57499999999999996,"15min":0.59899999999999998},"min":{"1min":0.373,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.76500000000000001,"5min":0.87,"15min":0.96899999999999997},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.64800000000000002,"15min":0.65300000000000002},"min":{"1min":0.56200000000000006,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.83799999999999997,"5min":0.92000000000000004,"15min":1.329},"last":0.61899999999999999}]},{"osd":4,"last update":"Thu Oct 19 13:03:51 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.46700000000000003,"15min":0.46600000000000003},"min":{"1min":0.31900000000000001,"5min":0.247,"15min":0.247},"max":{"1min":0.59399999999999997,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.47199999999999998,"15min":0.46899999999999997},"min":{"1min":0.36099999999999999,"5min":0.216,"15min":0.216},"max":{"1min":0.60799999999999998,"5min":0.73399999999999999,"15min":0.75600000000000001},"last":0.63600000000000001}]},{"osd":5,"last update":"Thu Oct 19 13:03:51 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.49299999999999999,"15min":0.48499999999999999},"min":{"1min":0.40799999999999997,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.78000000000000003,"5min":0.78800000000000003,"15min":0.80600000000000005},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.52300000000000002,"15min":0.51900000000000002},"min":{"1min":0.42099999999999999,"5min":0.32200000000000001,"15min":0.314},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.81999999999999995},"last":0.72999999999999998}]},{"osd":6,"last update":"Thu Oct 19 13:03:51 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.54700000000000004,"15min":0.54600000000000004},"min":{"1min":0.39200000000000002,"5min":0.35999999999999999,"15min":0.308},"max":{"1min":0.71899999999999997,"5min":0.78900000000000003,"15min":0.88900000000000001},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.52600000000000002,"15min":0.53100000000000003},"min":{"1min":0.439,"5min":0.318,"15min":0.27300000000000002},"max":{"1min":0.70399999999999996,"5min":0.77000000000000002,"15min":0.90300000000000002},"last":0.68500000000000005}]}]},{"osd":6,"up_from":13,"seq":55834575093,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32920,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338920,"statfs":{"total":96636764160,"available":96603054080,"internally_reserved":0,"allocated":1777664,"data_stored":899261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 13:03:37 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.56899999999999995,"15min":0.58099999999999996},"min":{"1min":0.38800000000000001,"5min":0.23999999999999999,"15min":0.19500000000000001},"max":{"1min":0.77300000000000002,"5min":1.196,"15min":1.196},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.55900000000000005,"15min":0.57899999999999996},"min":{"1min":0.33300000000000002,"5min":0.28799999999999998,"15min":0.247},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.59999999999999998}]},{"osd":1,"last update":"Thu Oct 19 13:03:37 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.57699999999999996,"15min":0.59399999999999997},"min":{"1min":0.45900000000000002,"5min":0.29099999999999998,"15min":0.27400000000000002},"max":{"1min":0.80100000000000005,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.57999999999999996,"15min":0.59599999999999997},"min":{"1min":0.34300000000000003,"5min":0.307,"15min":0.307},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.82099999999999995},"last":0.67500000000000004}]},{"osd":2,"last update":"Thu Oct 19 13:03:37 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.55700000000000005,"15min":0.57899999999999996},"min":{"1min":0.34699999999999998,"5min":0.34599999999999997,"15min":0.307},"max":{"1min":0.79400000000000004,"5min":0.84699999999999998,"15min":0.85699999999999998},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.621,"5min":0.59699999999999998,"15min":0.60699999999999998},"min":{"1min":0.35699999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.80800000000000005,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.61599999999999999}]},{"osd":3,"last update":"Thu Oct 19 13:03:37 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.60099999999999998,"15min":0.623},"min":{"1min":0.379,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.79600000000000004,"5min":0.871,"15min":0.871},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.60699999999999998,"15min":0.62},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.36799999999999999},"max":{"1min":0.82799999999999996,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.78200000000000003}]},{"osd":4,"last update":"Thu Oct 19 13:03:37 2023","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.47799999999999998,"15min":0.46700000000000003},"min":{"1min":0.29799999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.71499999999999997,"5min":1.353,"15min":1.353},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.47799999999999998,"15min":0.48099999999999998},"min":{"1min":0.32700000000000001,"5min":0.254,"15min":0.254},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.41899999999999998}]},{"osd":5,"last update":"Thu Oct 19 13:03:37 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.46400000000000002,"15min":0.47399999999999998},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.52900000000000003,"15min":0.51900000000000002},"min":{"1min":0.31900000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.85699999999999998},"last":0.48299999999999998}]},{"osd":7,"last update":"Thu Oct 19 13:03:37 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.53600000000000003,"15min":0.52400000000000002},"min":{"1min":0.28000000000000003,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.82499999999999996,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.54100000000000004,"15min":0.53500000000000003},"min":{"1min":0.314,"5min":0.27900000000000003,"15min":0.26000000000000001},"max":{"1min":0.77900000000000003,"5min":1.127,"15min":1.127},"last":0.49399999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575093,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50956,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320884,"statfs":{"total":96636764160,"available":96584585216,"internally_reserved":0,"allocated":1765376,"data_stored":892138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 13:03:33 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57299999999999995,"15min":0.58699999999999997},"min":{"1min":0.39100000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.77100000000000002,"5min":0.79900000000000004,"15min":0.82699999999999996},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.59499999999999997,"15min":0.59699999999999998},"min":{"1min":0.41999999999999998,"5min":0.33900000000000002,"15min":0.27800000000000002},"max":{"1min":0.78200000000000003,"5min":0.82999999999999996,"15min":0.84299999999999997},"last":0.71999999999999997}]},{"osd":1,"last update":"Thu Oct 19 13:03:33 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.58899999999999997,"15min":0.59699999999999998},"min":{"1min":0.45600000000000002,"5min":0.34799999999999998,"15min":0.32800000000000001},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.91200000000000003},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.622,"5min":0.61699999999999999,"15min":0.61099999999999999},"min":{"1min":0.41199999999999998,"5min":0.378,"15min":0.34000000000000002},"max":{"1min":0.77200000000000002,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.78700000000000003}]},{"osd":2,"last update":"Thu Oct 19 13:03:33 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.56499999999999995,"15min":0.57999999999999996},"min":{"1min":0.373,"5min":0.32700000000000001,"15min":0.307},"max":{"1min":0.75700000000000001,"5min":0.77500000000000002,"15min":0.80700000000000005},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.621,"15min":0.61799999999999999},"min":{"1min":0.47599999999999998,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.82499999999999996,"5min":0.83999999999999997,"15min":0.88100000000000001},"last":0.63500000000000001}]},{"osd":3,"last update":"Thu Oct 19 13:03:33 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.59199999999999997,"15min":0.60199999999999998},"min":{"1min":0.37,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.79900000000000004,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.61899999999999999,"15min":0.63},"min":{"1min":0.48899999999999999,"5min":0.40899999999999997,"15min":0.32100000000000001},"max":{"1min":0.75600000000000001,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.52500000000000002}]},{"osd":4,"last update":"Thu Oct 19 13:03:33 2023","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.47099999999999997,"15min":0.46999999999999997},"min":{"1min":0.32100000000000001,"5min":0.307,"15min":0.29099999999999998},"max":{"1min":0.745,"5min":0.745,"15min":0.98099999999999998},"last":0.501},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.48599999999999999,"15min":0.47299999999999998},"min":{"1min":0.372,"5min":0.26100000000000001,"15min":0.251},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.79400000000000004},"last":0.48399999999999999}]},{"osd":6,"last update":"Thu Oct 19 13:03:33 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.48999999999999999,"15min":0.47799999999999998},"min":{"1min":0.34599999999999997,"5min":0.27800000000000002,"15min":0.27200000000000002},"max":{"1min":0.80200000000000005,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.51500000000000001,"15min":0.50700000000000001},"min":{"1min":0.40699999999999997,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.82499999999999996},"last":0.67900000000000005}]},{"osd":7,"last update":"Thu Oct 19 13:03:33 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.503,"15min":0.49299999999999999},"min":{"1min":0.35999999999999999,"5min":0.311,"15min":0.311},"max":{"1min":0.66800000000000004,"5min":0.78600000000000003,"15min":1.0589999999999999},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.55500000000000005,"15min":0.53800000000000003},"min":{"1min":0.34699999999999998,"5min":0.34599999999999997,"15min":0.311},"max":{"1min":0.78000000000000003,"5min":0.78700000000000003,"15min":0.94899999999999995},"last":0.74399999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575093,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32924,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338916,"statfs":{"total":96636764160,"available":96603049984,"internally_reserved":0,"allocated":1777664,"data_stored":899261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 13:03:47 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.52700000000000002,"15min":0.55300000000000005},"min":{"1min":0.37,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.78800000000000003,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.54800000000000004,"15min":0.55500000000000005},"min":{"1min":0.46300000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.68899999999999995,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.46500000000000002}]},{"osd":1,"last update":"Thu Oct 19 13:03:47 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56200000000000006,"15min":0.59699999999999998},"min":{"1min":0.37,"5min":0.32500000000000001,"15min":0.317},"max":{"1min":0.73999999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.58699999999999997,"15min":0.61099999999999999},"min":{"1min":0.441,"5min":0.38200000000000001,"15min":0.32600000000000001},"max":{"1min":0.76400000000000001,"5min":0.79800000000000004,"15min":0.86899999999999999},"last":0.59699999999999998}]},{"osd":2,"last update":"Thu Oct 19 13:03:47 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.57999999999999996,"15min":0.60599999999999998},"min":{"1min":0.371,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.77600000000000002,"5min":0.80100000000000005,"15min":0.86599999999999999},"last":0.498},{"interface":"front","average":{"1min":0.622,"5min":0.59299999999999997,"15min":0.60999999999999999},"min":{"1min":0.47999999999999998,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.78600000000000003,"5min":0.83799999999999997,"15min":0.85999999999999999},"last":0.48399999999999999}]},{"osd":3,"last update":"Thu Oct 19 13:03:47 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.55700000000000005,"15min":0.57599999999999996},"min":{"1min":0.40799999999999997,"5min":0.33200000000000002,"15min":0.32400000000000001},"max":{"1min":0.71999999999999997,"5min":0.75,"15min":0.79700000000000004},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.61199999999999999,"15min":0.63600000000000001},"min":{"1min":0.51600000000000001,"5min":0.42099999999999999,"15min":0.41299999999999998},"max":{"1min":0.80700000000000005,"5min":0.82599999999999996,"15min":0.83899999999999997},"last":0.55600000000000005}]},{"osd":5,"last update":"Thu Oct 19 13:03:47 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.44500000000000001,"15min":0.44600000000000001},"min":{"1min":0.34699999999999998,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.79600000000000004},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.46400000000000002,"15min":0.46100000000000002},"min":{"1min":0.314,"5min":0.30099999999999999,"15min":0.23499999999999999},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.36699999999999999}]},{"osd":6,"last update":"Thu Oct 19 13:03:47 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.45500000000000002,"15min":0.45700000000000002},"min":{"1min":0.34100000000000003,"5min":0.28499999999999998,"15min":0.26400000000000001},"max":{"1min":0.81299999999999994,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.497,"15min":0.49299999999999999},"min":{"1min":0.33300000000000002,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.53900000000000003}]},{"osd":7,"last update":"Thu Oct 19 13:03:47 2023","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.499,"15min":0.501},"min":{"1min":0.34999999999999998,"5min":0.31900000000000001,"15min":0.27500000000000002},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.80200000000000005},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.55000000000000004,"15min":0.54500000000000004},"min":{"1min":0.39000000000000001,"5min":0.33100000000000002,"15min":0.30199999999999999},"max":{"1min":0.85699999999999998,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.44600000000000001}]}]},{"osd":3,"up_from":13,"seq":55834575092,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41252,"kb_used_data":1752,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330588,"statfs":{"total":96636764160,"available":96594522112,"internally_reserved":0,"allocated":1794048,"data_stored":900907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":40434120},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 13:03:30 2023","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.40500000000000003,"15min":0.40300000000000002},"min":{"1min":0.25800000000000001,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.59399999999999997,"5min":0.69099999999999995,"15min":1.1279999999999999},"last":0.373},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.40200000000000002,"15min":0.41399999999999998},"min":{"1min":0.249,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.56299999999999994,"5min":0.75,"15min":0.75},"last":0.30099999999999999}]},{"osd":1,"last update":"Thu Oct 19 13:03:30 2023","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.41499999999999998,"15min":0.40999999999999998},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.626,"5min":0.63500000000000001,"15min":0.72099999999999997},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.438,"5min":0.438,"15min":0.45900000000000002},"min":{"1min":0.28499999999999998,"5min":0.221,"15min":0.221},"max":{"1min":0.59699999999999998,"5min":0.66700000000000004,"15min":2.9399999999999999},"last":0.45100000000000001}]},{"osd":2,"last update":"Thu Oct 19 13:03:30 2023","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.45700000000000002,"15min":0.46300000000000002},"min":{"1min":0.29699999999999999,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.97699999999999998},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.48799999999999999,"15min":0.48899999999999999},"min":{"1min":0.33200000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.42299999999999999}]},{"osd":4,"last update":"Thu Oct 19 13:03:30 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.64800000000000002,"15min":0.66400000000000003},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":1.0069999999999999},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.67300000000000004,"15min":0.68999999999999995},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.66100000000000003}]},{"osd":5,"last update":"Thu Oct 19 13:03:30 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.66400000000000003,"15min":0.66300000000000003},"min":{"1min":0.40699999999999997,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.93700000000000006,"5min":0.96699999999999997,"15min":2.9990000000000001},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.627,"15min":0.64900000000000002},"min":{"1min":0.46700000000000003,"5min":0.38600000000000001,"15min":0.35899999999999999},"max":{"1min":0.86599999999999999,"5min":0.873,"15min":1.774},"last":0.68400000000000005}]},{"osd":6,"last update":"Thu Oct 19 13:03:30 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.64500000000000002,"15min":0.65200000000000002},"min":{"1min":0.45300000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.67600000000000005,"15min":0.67900000000000005},"min":{"1min":0.42899999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.167},"last":0.622}]},{"osd":7,"last update":"Thu Oct 19 13:03:30 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.64600000000000002,"15min":0.66200000000000003},"min":{"1min":0.438,"5min":0.42099999999999999,"15min":0.39400000000000002},"max":{"1min":0.86799999999999999,"5min":0.94299999999999995,"15min":1.042},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.67100000000000004,"15min":0.68000000000000005},"min":{"1min":0.48499999999999999,"5min":0.441,"15min":0.432},"max":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":2.8149999999999999},"last":0.70099999999999996}]}]},{"osd":2,"up_from":13,"seq":55834575094,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55112,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316728,"statfs":{"total":96636764160,"available":96580329472,"internally_reserved":0,"allocated":1765376,"data_stored":892138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 13:03:48 2023","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.38500000000000001,"15min":0.379},"min":{"1min":0.253,"5min":0.20699999999999999,"15min":0.19700000000000001},"max":{"1min":0.59099999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.433,"5min":0.41199999999999998,"15min":0.40999999999999998},"min":{"1min":0.27700000000000002,"5min":0.222,"15min":0.20999999999999999},"max":{"1min":0.621,"5min":0.68000000000000005,"15min":0.74099999999999999},"last":0.371}]},{"osd":1,"last update":"Thu Oct 19 13:03:48 2023","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.39100000000000001,"15min":0.39600000000000002},"min":{"1min":0.249,"5min":0.249,"15min":0.24299999999999999},"max":{"1min":0.60799999999999998,"5min":0.64100000000000001,"15min":0.71099999999999997},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.42999999999999999,"15min":0.42699999999999999},"min":{"1min":0.30099999999999999,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.29299999999999998}]},{"osd":3,"last update":"Thu Oct 19 13:03:48 2023","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.45900000000000002,"15min":0.45600000000000002},"min":{"1min":0.318,"5min":0.29199999999999998,"15min":0.22800000000000001},"max":{"1min":0.66500000000000004,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.48099999999999998,"15min":0.48399999999999999},"min":{"1min":0.34899999999999998,"5min":0.32200000000000001,"15min":0.29099999999999998},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.82999999999999996},"last":0.39000000000000001}]},{"osd":4,"last update":"Thu Oct 19 13:03:48 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.625,"15min":0.64000000000000001},"min":{"1min":0.435,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.874,"5min":0.89500000000000002,"15min":0.98999999999999999},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.63700000000000001,"15min":0.65100000000000002},"min":{"1min":0.498,"5min":0.41199999999999998,"15min":0.35099999999999998},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":1.046},"last":0.77000000000000002}]},{"osd":5,"last update":"Thu Oct 19 13:03:48 2023","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.64200000000000002,"15min":0.65800000000000003},"min":{"1min":0.42399999999999999,"5min":0.40799999999999997,"15min":0.31900000000000001},"max":{"1min":0.94099999999999995,"5min":0.96199999999999997,"15min":0.99399999999999999},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.64000000000000001,"15min":0.65500000000000003},"min":{"1min":0.48399999999999999,"5min":0.39100000000000001,"15min":0.375},"max":{"1min":0.91500000000000004,"5min":0.94199999999999995,"15min":0.95799999999999996},"last":0.748}]},{"osd":6,"last update":"Thu Oct 19 13:03:48 2023","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.65500000000000003,"15min":0.66000000000000003},"min":{"1min":0.51300000000000001,"5min":0.40400000000000003,"15min":0.40300000000000002},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":1.008},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.66200000000000003,"15min":0.67600000000000005},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.371},"max":{"1min":0.90300000000000002,"5min":0.90400000000000003,"15min":0.995},"last":0.67600000000000005}]},{"osd":7,"last update":"Thu Oct 19 13:03:48 2023","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.66200000000000003,"15min":0.67500000000000004},"min":{"1min":0.47399999999999998,"5min":0.39800000000000002,"15min":0.36399999999999999},"max":{"1min":0.89500000000000002,"5min":0.96299999999999997,"15min":1.0149999999999999},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.67500000000000004,"15min":0.68300000000000005},"min":{"1min":0.46700000000000003,"5min":0.438,"15min":0.38200000000000001},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":1.0329999999999999},"last":0.58899999999999997}]}]},{"osd":1,"up_from":13,"seq":55834575093,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37064,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334776,"statfs":{"total":96636764160,"available":96598810624,"internally_reserved":0,"allocated":1765376,"data_stored":892138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 13:03:43 2023","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.371,"15min":0.38300000000000001},"min":{"1min":0.28699999999999998,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":1.3879999999999999},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.41599999999999998,"15min":0.434},"min":{"1min":0.22800000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.65000000000000002,"5min":0.76100000000000001,"15min":2.5800000000000001},"last":0.28899999999999998}]},{"osd":2,"last update":"Thu Oct 19 13:03:43 2023","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.40999999999999998,"15min":0.42199999999999999},"min":{"1min":0.27400000000000002,"5min":0.24199999999999999,"15min":0.21299999999999999},"max":{"1min":0.69399999999999995,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.437,"5min":0.44800000000000001,"15min":0.45700000000000002},"min":{"1min":0.27100000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.58499999999999996,"5min":0.79800000000000004,"15min":2.6230000000000002},"last":0.35499999999999998}]},{"osd":3,"last update":"Thu Oct 19 13:03:43 2023","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.46800000000000003,"15min":0.48899999999999999},"min":{"1min":0.33500000000000002,"5min":0.246,"15min":0.246},"max":{"1min":0.72899999999999998,"5min":1.008,"15min":3.9740000000000002},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.42299999999999999,"15min":0.42599999999999999},"min":{"1min":0.309,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.56299999999999994,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.377}]},{"osd":4,"last update":"Thu Oct 19 13:03:43 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.629,"15min":0.64800000000000002},"min":{"1min":0.40899999999999997,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.76300000000000001,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.63800000000000001},"min":{"1min":0.434,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.755,"5min":0.86899999999999999,"15min":1.0860000000000001},"last":0.60599999999999998}]},{"osd":5,"last update":"Thu Oct 19 13:03:43 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.63900000000000001,"15min":0.65000000000000002},"min":{"1min":0.439,"5min":0.38700000000000001,"15min":0.38100000000000001},"max":{"1min":0.79200000000000004,"5min":1.0549999999999999,"15min":1.1200000000000001},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.65500000000000003,"15min":0.67400000000000004},"min":{"1min":0.5,"5min":0.374,"15min":0.374},"max":{"1min":0.80600000000000005,"5min":1.1020000000000001,"15min":2.7120000000000002},"last":0.55400000000000005}]},{"osd":6,"last update":"Thu Oct 19 13:03:43 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.63200000000000001,"15min":0.64900000000000002},"min":{"1min":0.45000000000000001,"5min":0.41499999999999998,"15min":0.36799999999999999},"max":{"1min":0.79300000000000004,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.67000000000000004},"min":{"1min":0.46100000000000002,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.81799999999999995,"5min":1.129,"15min":1.129},"last":0.56899999999999995}]},{"osd":7,"last update":"Thu Oct 19 13:03:43 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.66500000000000004,"15min":0.67800000000000005},"min":{"1min":0.52500000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.81000000000000005,"5min":1.1439999999999999,"15min":2.786},"last":0.499},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64300000000000002,"15min":0.67000000000000004},"min":{"1min":0.48099999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.746,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.58799999999999997}]}]},{"osd":0,"up_from":13,"seq":55834575094,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33504,"kb_used_data":2316,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338336,"statfs":{"total":96636764160,"available":96602456064,"internally_reserved":0,"allocated":2371584,"data_stored":1489032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Oct 19 13:03:47 2023","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.38500000000000001,"15min":0.39000000000000001},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.182},"max":{"1min":0.57599999999999996,"5min":0.624,"15min":0.81799999999999995},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.40799999999999997,"15min":0.42699999999999999},"min":{"1min":0.318,"5min":0.22500000000000001,"15min":0.20200000000000001},"max":{"1min":0.56999999999999995,"5min":0.67600000000000005,"15min":0.746},"last":0.48999999999999999}]},{"osd":2,"last update":"Thu Oct 19 13:03:47 2023","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.42099999999999999,"15min":0.44900000000000001},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.26200000000000001},"max":{"1min":0.59899999999999998,"5min":0.64100000000000001,"15min":0.76300000000000001},"last":0.309},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.41599999999999998,"15min":0.432},"min":{"1min":0.30199999999999999,"5min":0.246,"15min":0.20699999999999999},"max":{"1min":0.56499999999999995,"5min":0.72999999999999998,"15min":0.76800000000000002},"last":0.39400000000000002}]},{"osd":3,"last update":"Thu Oct 19 13:03:47 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.38900000000000001,"15min":0.40600000000000003},"min":{"1min":0.218,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.48099999999999998,"5min":0.60399999999999998,"15min":1.2829999999999999},"last":0.376},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.45400000000000001,"15min":0.48599999999999999},"min":{"1min":0.316,"5min":0.29299999999999998,"15min":0.29099999999999998},"max":{"1min":0.63200000000000001,"5min":0.71199999999999997,"15min":1.4990000000000001},"last":0.45300000000000001}]},{"osd":4,"last update":"Thu Oct 19 13:03:47 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.59699999999999998,"15min":0.627},"min":{"1min":0.45800000000000002,"5min":0.40300000000000002,"15min":0.34999999999999998},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":1.361},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.63900000000000001,"15min":0.65100000000000002},"min":{"1min":0.44,"5min":0.36399999999999999,"15min":0.35999999999999999},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.98999999999999999},"last":0.53600000000000003}]},{"osd":5,"last update":"Thu Oct 19 13:03:47 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.66800000000000004,"15min":0.68200000000000005},"min":{"1min":0.46000000000000002,"5min":0.45100000000000001,"15min":0.378},"max":{"1min":0.92000000000000004,"5min":0.94599999999999995,"15min":1.325},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.66400000000000003,"15min":0.68300000000000005},"min":{"1min":0.44800000000000001,"5min":0.40100000000000002,"15min":0.318},"max":{"1min":0.94099999999999995,"5min":1.02,"15min":1.024},"last":0.56799999999999995}]},{"osd":6,"last update":"Thu Oct 19 13:03:47 2023","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.64200000000000002,"15min":0.65900000000000003},"min":{"1min":0.56699999999999995,"5min":0.36899999999999999,"15min":0.35899999999999999},"max":{"1min":0.879,"5min":0.879,"15min":0.98099999999999998},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.64100000000000001,"15min":0.65900000000000003},"min":{"1min":0.46200000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.86699999999999999,"5min":0.92000000000000004,"15min":0.995},"last":0.64800000000000002}]},{"osd":7,"last update":"Thu Oct 19 13:03:47 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.63300000000000001,"15min":0.67200000000000004},"min":{"1min":0.53200000000000003,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.83599999999999997,"5min":0.89900000000000002,"15min":1.0089999999999999},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.65000000000000002,"15min":0.65900000000000003},"min":{"1min":0.46899999999999997,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.86099999999999999,"5min":0.94499999999999995,"15min":1.389},"last":0.51500000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-19T13:04:26.437 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-19T13:04:26.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.577+0000 7f6da2bc0640 1 -- 172.21.15.57:0/2450121272 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6d9c0b0c40 msgr2=0x7f6d9c158300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:26.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.577+0000 7f6da2bc0640 1 --2- 172.21.15.57:0/2450121272 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6d9c0b0c40 0x7f6d9c158300 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7f6d94005fd0 tx=0x7f6d9402d1e0 comp rx=0 tx=0).stop 2023-10-19T13:04:26.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.577+0000 7f6da2bc0640 1 -- 172.21.15.57:0/2450121272 shutdown_connections 2023-10-19T13:04:26.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.577+0000 7f6da2bc0640 1 --2- 172.21.15.57:0/2450121272 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6d9c0b0c40 0x7f6d9c158300 unknown :-1 s=CLOSED pgs=1598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:26.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.577+0000 7f6da2bc0640 1 -- 172.21.15.57:0/2450121272 >> 172.21.15.57:0/2450121272 conn(0x7f6d9c069930 msgr2=0x7f6d9c069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:26.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.577+0000 7f6da2bc0640 1 -- 172.21.15.57:0/2450121272 shutdown_connections 2023-10-19T13:04:26.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.577+0000 7f6da2bc0640 1 -- 172.21.15.57:0/2450121272 wait complete. 2023-10-19T13:04:26.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.578+0000 7f6da2bc0640 1 Processor -- start 2023-10-19T13:04:26.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.578+0000 7f6da2bc0640 1 -- start start 2023-10-19T13:04:26.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.579+0000 7f6da2bc0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6d9c0afa20 0x7f6d9c1392d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:26.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.579+0000 7f6da2bc0640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6d9c0b0c40 0x7f6d9c138a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:26.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.579+0000 7f6da2bc0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6d9c0cbd20 0x7f6d9c13aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:26.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.579+0000 7f6da2bc0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d9c14cbd0 con 0x7f6d9c0cbd20 2023-10-19T13:04:26.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.579+0000 7f6da2bc0640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d9c139990 con 0x7f6d9c0afa20 2023-10-19T13:04:26.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.579+0000 7f6da2bc0640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d9c139c70 con 0x7f6d9c0b0c40 2023-10-19T13:04:26.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.579+0000 7f6da115d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6d9c0cbd20 0x7f6d9c13aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:26.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.579+0000 7f6da115d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6d9c0cbd20 0x7f6d9c13aa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35664/0 (socket says 172.21.15.57:35664) 2023-10-19T13:04:26.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.579+0000 7f6da115d640 1 -- 172.21.15.57:0/3892066753 learned_addr learned my addr 172.21.15.57:0/3892066753 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:26.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.579+0000 7f6da095c640 1 --2- 172.21.15.57:0/3892066753 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6d9c0afa20 0x7f6d9c1392d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:26.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.579+0000 7f6d9bfff640 1 --2- 172.21.15.57:0/3892066753 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6d9c0b0c40 0x7f6d9c138a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:26.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.579+0000 7f6da115d640 1 -- 172.21.15.57:0/3892066753 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6d9c0b0c40 msgr2=0x7f6d9c138a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:26.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.579+0000 7f6da115d640 1 --2- 172.21.15.57:0/3892066753 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6d9c0b0c40 0x7f6d9c138a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:26.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.579+0000 7f6da115d640 1 -- 172.21.15.57:0/3892066753 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6d9c0afa20 msgr2=0x7f6d9c1392d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:26.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.579+0000 7f6da115d640 1 --2- 172.21.15.57:0/3892066753 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6d9c0afa20 0x7f6d9c1392d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:26.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.579+0000 7f6da115d640 1 -- 172.21.15.57:0/3892066753 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d9400aa10 con 0x7f6d9c0cbd20 2023-10-19T13:04:26.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.579+0000 7f6da115d640 1 --2- 172.21.15.57:0/3892066753 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6d9c0cbd20 0x7f6d9c13aa30 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7f6d94038040 tx=0x7f6d94036db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:26.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.580+0000 7f6d997fa640 1 -- 172.21.15.57:0/3892066753 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d94035440 con 0x7f6d9c0cbd20 2023-10-19T13:04:26.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.580+0000 7f6d997fa640 1 -- 172.21.15.57:0/3892066753 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f6d94004540 con 0x7f6d9c0cbd20 2023-10-19T13:04:26.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.580+0000 7f6d997fa640 1 -- 172.21.15.57:0/3892066753 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d94040450 con 0x7f6d9c0cbd20 2023-10-19T13:04:26.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.580+0000 7f6da2bc0640 1 -- 172.21.15.57:0/3892066753 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d9c139f50 con 0x7f6d9c0cbd20 2023-10-19T13:04:26.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.580+0000 7f6da2bc0640 1 -- 172.21.15.57:0/3892066753 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d9c13a510 con 0x7f6d9c0cbd20 2023-10-19T13:04:26.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.581+0000 7f6da2bc0640 1 -- 172.21.15.57:0/3892066753 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d9c13a700 con 0x7f6d9c0cbd20 2023-10-19T13:04:26.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.581+0000 7f6d997fa640 1 -- 172.21.15.57:0/3892066753 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 69) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6d9403c070 con 0x7f6d9c0cbd20 2023-10-19T13:04:26.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.581+0000 7f6d997fa640 1 --2- 172.21.15.57:0/3892066753 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f6d8006af60 0x7f6d8006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:26.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.581+0000 7f6d997fa640 1 -- 172.21.15.57:0/3892066753 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f6d940accb0 con 0x7f6d9c0cbd20 2023-10-19T13:04:26.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.582+0000 7f6da095c640 1 --2- 172.21.15.57:0/3892066753 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f6d8006af60 0x7f6d8006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:26.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.582+0000 7f6da095c640 1 --2- 172.21.15.57:0/3892066753 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f6d8006af60 0x7f6d8006d420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6d90000f60 tx=0x7f6d90009040 comp rx=0 tx=0).ready entity=mgr.7960 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:26.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.585+0000 7f6d997fa640 1 -- 172.21.15.57:0/3892066753 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6d94077cc0 con 0x7f6d9c0cbd20 2023-10-19T13:04:26.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.758+0000 7f6da2bc0640 1 -- 172.21.15.57:0/3892066753 --> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6d9c0690e0 con 0x7f6d8006af60 2023-10-19T13:04:26.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.766+0000 7f6d997fa640 1 -- 172.21.15.57:0/3892066753 <== mgr.7960 v2:172.21.15.179:6832/2346394725 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+290081 (secure 0 0 0) 0x7f6d9c0690e0 con 0x7f6d8006af60 2023-10-19T13:04:26.774 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:04:26.776 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2023-10-19T13:04:26.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.769+0000 7f6da2bc0640 1 -- 172.21.15.57:0/3892066753 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f6d8006af60 msgr2=0x7f6d8006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:26.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.769+0000 7f6da2bc0640 1 --2- 172.21.15.57:0/3892066753 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f6d8006af60 0x7f6d8006d420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6d90000f60 tx=0x7f6d90009040 comp rx=0 tx=0).stop 2023-10-19T13:04:26.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.769+0000 7f6da2bc0640 1 -- 172.21.15.57:0/3892066753 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6d9c0cbd20 msgr2=0x7f6d9c13aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:26.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.769+0000 7f6da2bc0640 1 --2- 172.21.15.57:0/3892066753 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6d9c0cbd20 0x7f6d9c13aa30 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7f6d94038040 tx=0x7f6d94036db0 comp rx=0 tx=0).stop 2023-10-19T13:04:26.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.769+0000 7f6da2bc0640 1 -- 172.21.15.57:0/3892066753 shutdown_connections 2023-10-19T13:04:26.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.769+0000 7f6da2bc0640 1 --2- 172.21.15.57:0/3892066753 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f6d8006af60 0x7f6d8006d420 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:26.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.769+0000 7f6da2bc0640 1 --2- 172.21.15.57:0/3892066753 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6d9c0cbd20 0x7f6d9c13aa30 unknown :-1 s=CLOSED pgs=1599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:26.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.769+0000 7f6da2bc0640 1 --2- 172.21.15.57:0/3892066753 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6d9c0b0c40 0x7f6d9c138a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:26.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.769+0000 7f6da2bc0640 1 --2- 172.21.15.57:0/3892066753 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6d9c0afa20 0x7f6d9c1392d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:26.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.769+0000 7f6da2bc0640 1 -- 172.21.15.57:0/3892066753 >> 172.21.15.57:0/3892066753 conn(0x7f6d9c069930 msgr2=0x7f6d9c14f450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:26.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.769+0000 7f6da2bc0640 1 -- 172.21.15.57:0/3892066753 shutdown_connections 2023-10-19T13:04:26.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.769+0000 7f6da2bc0640 1 -- 172.21.15.57:0/3892066753 wait complete. 2023-10-19T13:04:26.784 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-10-19T13:04:25.745126+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1154,"num_read_kb":2358,"num_write":413,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3103,"ondisk_log_size":3103,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":6,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":339428,"kb_used_data":15016,"kb_used_omap":12,"kb_used_meta":324275,"kb_avail":754635292,"statfs":{"total":773094113280,"available":772746539008,"internally_reserved":0,"allocated":15376384,"data_stored":8347381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":332058169},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.330833"},"pg_stats":[{"pgid":"18.3f","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.254263+0000","last_change":"2023-10-19T13:04:23.249380+0000","last_active":"2023-10-19T13:04:23.254263+0000","last_peered":"2023-10-19T13:04:23.254263+0000","last_clean":"2023-10-19T13:04:23.254263+0000","last_became_active":"2023-10-19T13:04:23.249130+0000","last_became_peered":"2023-10-19T13:04:23.249130+0000","last_unstale":"2023-10-19T13:04:23.254263+0000","last_undegraded":"2023-10-19T13:04:23.254263+0000","last_fullsized":"2023-10-19T13:04:23.254263+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:20:45.859782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3e","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.254311+0000","last_change":"2023-10-19T13:04:23.245224+0000","last_active":"2023-10-19T13:04:23.254311+0000","last_peered":"2023-10-19T13:04:23.254311+0000","last_clean":"2023-10-19T13:04:23.254311+0000","last_became_active":"2023-10-19T13:04:23.244923+0000","last_became_peered":"2023-10-19T13:04:23.244923+0000","last_unstale":"2023-10-19T13:04:23.254311+0000","last_undegraded":"2023-10-19T13:04:23.254311+0000","last_fullsized":"2023-10-19T13:04:23.254311+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:28:43.173352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3d","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.254683+0000","last_change":"2023-10-19T13:04:23.247264+0000","last_active":"2023-10-19T13:04:23.254683+0000","last_peered":"2023-10-19T13:04:23.254683+0000","last_clean":"2023-10-19T13:04:23.254683+0000","last_became_active":"2023-10-19T13:04:23.246996+0000","last_became_peered":"2023-10-19T13:04:23.246996+0000","last_unstale":"2023-10-19T13:04:23.254683+0000","last_undegraded":"2023-10-19T13:04:23.254683+0000","last_fullsized":"2023-10-19T13:04:23.254683+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:36:11.644511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3c","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.252462+0000","last_change":"2023-10-19T13:04:23.250157+0000","last_active":"2023-10-19T13:04:23.252462+0000","last_peered":"2023-10-19T13:04:23.252462+0000","last_clean":"2023-10-19T13:04:23.252462+0000","last_became_active":"2023-10-19T13:04:23.249875+0000","last_became_peered":"2023-10-19T13:04:23.249875+0000","last_unstale":"2023-10-19T13:04:23.252462+0000","last_undegraded":"2023-10-19T13:04:23.252462+0000","last_fullsized":"2023-10-19T13:04:23.252462+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:17:34.123593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3b","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.246485+0000","last_change":"2023-10-19T13:04:23.244884+0000","last_active":"2023-10-19T13:04:23.246485+0000","last_peered":"2023-10-19T13:04:23.246485+0000","last_clean":"2023-10-19T13:04:23.246485+0000","last_became_active":"2023-10-19T13:04:23.244620+0000","last_became_peered":"2023-10-19T13:04:23.244620+0000","last_unstale":"2023-10-19T13:04:23.246485+0000","last_undegraded":"2023-10-19T13:04:23.246485+0000","last_fullsized":"2023-10-19T13:04:23.246485+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:48:30.703457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3a","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.258871+0000","last_change":"2023-10-19T13:04:23.254891+0000","last_active":"2023-10-19T13:04:23.258871+0000","last_peered":"2023-10-19T13:04:23.258871+0000","last_clean":"2023-10-19T13:04:23.258871+0000","last_became_active":"2023-10-19T13:04:23.254633+0000","last_became_peered":"2023-10-19T13:04:23.254633+0000","last_unstale":"2023-10-19T13:04:23.258871+0000","last_undegraded":"2023-10-19T13:04:23.258871+0000","last_fullsized":"2023-10-19T13:04:23.258871+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:42:10.908893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.39","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.258540+0000","last_change":"2023-10-19T13:04:23.248971+0000","last_active":"2023-10-19T13:04:23.258540+0000","last_peered":"2023-10-19T13:04:23.258540+0000","last_clean":"2023-10-19T13:04:23.258540+0000","last_became_active":"2023-10-19T13:04:23.248698+0000","last_became_peered":"2023-10-19T13:04:23.248698+0000","last_unstale":"2023-10-19T13:04:23.258540+0000","last_undegraded":"2023-10-19T13:04:23.258540+0000","last_fullsized":"2023-10-19T13:04:23.258540+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:45:28.119851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.38","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.254513+0000","last_change":"2023-10-19T13:04:23.245992+0000","last_active":"2023-10-19T13:04:23.254513+0000","last_peered":"2023-10-19T13:04:23.254513+0000","last_clean":"2023-10-19T13:04:23.254513+0000","last_became_active":"2023-10-19T13:04:23.245478+0000","last_became_peered":"2023-10-19T13:04:23.245478+0000","last_unstale":"2023-10-19T13:04:23.254513+0000","last_undegraded":"2023-10-19T13:04:23.254513+0000","last_fullsized":"2023-10-19T13:04:23.254513+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:26:46.252277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.37","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.255483+0000","last_change":"2023-10-19T13:04:23.255026+0000","last_active":"2023-10-19T13:04:23.255483+0000","last_peered":"2023-10-19T13:04:23.255483+0000","last_clean":"2023-10-19T13:04:23.255483+0000","last_became_active":"2023-10-19T13:04:23.254753+0000","last_became_peered":"2023-10-19T13:04:23.254753+0000","last_unstale":"2023-10-19T13:04:23.255483+0000","last_undegraded":"2023-10-19T13:04:23.255483+0000","last_fullsized":"2023-10-19T13:04:23.255483+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:00:22.625725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.36","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.258733+0000","last_change":"2023-10-19T13:04:23.250614+0000","last_active":"2023-10-19T13:04:23.258733+0000","last_peered":"2023-10-19T13:04:23.258733+0000","last_clean":"2023-10-19T13:04:23.258733+0000","last_became_active":"2023-10-19T13:04:23.250322+0000","last_became_peered":"2023-10-19T13:04:23.250322+0000","last_unstale":"2023-10-19T13:04:23.258733+0000","last_undegraded":"2023-10-19T13:04:23.258733+0000","last_fullsized":"2023-10-19T13:04:23.258733+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:29:38.675177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.35","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.251604+0000","last_change":"2023-10-19T13:04:23.248749+0000","last_active":"2023-10-19T13:04:23.251604+0000","last_peered":"2023-10-19T13:04:23.251604+0000","last_clean":"2023-10-19T13:04:23.251604+0000","last_became_active":"2023-10-19T13:04:23.248463+0000","last_became_peered":"2023-10-19T13:04:23.248463+0000","last_unstale":"2023-10-19T13:04:23.251604+0000","last_undegraded":"2023-10-19T13:04:23.251604+0000","last_fullsized":"2023-10-19T13:04:23.251604+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:06:24.528059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.34","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.258665+0000","last_change":"2023-10-19T13:04:23.249174+0000","last_active":"2023-10-19T13:04:23.258665+0000","last_peered":"2023-10-19T13:04:23.258665+0000","last_clean":"2023-10-19T13:04:23.258665+0000","last_became_active":"2023-10-19T13:04:23.248875+0000","last_became_peered":"2023-10-19T13:04:23.248875+0000","last_unstale":"2023-10-19T13:04:23.258665+0000","last_undegraded":"2023-10-19T13:04:23.258665+0000","last_fullsized":"2023-10-19T13:04:23.258665+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:17:58.642825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.33","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.253552+0000","last_change":"2023-10-19T13:04:23.243990+0000","last_active":"2023-10-19T13:04:23.253552+0000","last_peered":"2023-10-19T13:04:23.253552+0000","last_clean":"2023-10-19T13:04:23.253552+0000","last_became_active":"2023-10-19T13:04:23.243730+0000","last_became_peered":"2023-10-19T13:04:23.243730+0000","last_unstale":"2023-10-19T13:04:23.253552+0000","last_undegraded":"2023-10-19T13:04:23.253552+0000","last_fullsized":"2023-10-19T13:04:23.253552+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:48:30.703457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.32","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.254450+0000","last_change":"2023-10-19T13:04:23.246493+0000","last_active":"2023-10-19T13:04:23.254450+0000","last_peered":"2023-10-19T13:04:23.254450+0000","last_clean":"2023-10-19T13:04:23.254450+0000","last_became_active":"2023-10-19T13:04:23.246135+0000","last_became_peered":"2023-10-19T13:04:23.246135+0000","last_unstale":"2023-10-19T13:04:23.254450+0000","last_undegraded":"2023-10-19T13:04:23.254450+0000","last_fullsized":"2023-10-19T13:04:23.254450+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:28:49.954421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.31","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.246471+0000","last_change":"2023-10-19T13:04:23.244804+0000","last_active":"2023-10-19T13:04:23.246471+0000","last_peered":"2023-10-19T13:04:23.246471+0000","last_clean":"2023-10-19T13:04:23.246471+0000","last_became_active":"2023-10-19T13:04:23.244536+0000","last_became_peered":"2023-10-19T13:04:23.244536+0000","last_unstale":"2023-10-19T13:04:23.246471+0000","last_undegraded":"2023-10-19T13:04:23.246471+0000","last_fullsized":"2023-10-19T13:04:23.246471+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:17:54.965079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.30","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.258700+0000","last_change":"2023-10-19T13:04:23.249193+0000","last_active":"2023-10-19T13:04:23.258700+0000","last_peered":"2023-10-19T13:04:23.258700+0000","last_clean":"2023-10-19T13:04:23.258700+0000","last_became_active":"2023-10-19T13:04:23.248650+0000","last_became_peered":"2023-10-19T13:04:23.248650+0000","last_unstale":"2023-10-19T13:04:23.258700+0000","last_undegraded":"2023-10-19T13:04:23.258700+0000","last_fullsized":"2023-10-19T13:04:23.258700+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:49:40.332015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.2f","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.248433+0000","last_change":"2023-10-19T13:04:23.240965+0000","last_active":"2023-10-19T13:04:23.248433+0000","last_peered":"2023-10-19T13:04:23.248433+0000","last_clean":"2023-10-19T13:04:23.248433+0000","last_became_active":"2023-10-19T13:04:23.240698+0000","last_became_peered":"2023-10-19T13:04:23.240698+0000","last_unstale":"2023-10-19T13:04:23.248433+0000","last_undegraded":"2023-10-19T13:04:23.248433+0000","last_fullsized":"2023-10-19T13:04:23.248433+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:39:07.411522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.2e","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.254030+0000","last_change":"2023-10-19T13:04:23.244595+0000","last_active":"2023-10-19T13:04:23.254030+0000","last_peered":"2023-10-19T13:04:23.254030+0000","last_clean":"2023-10-19T13:04:23.254030+0000","last_became_active":"2023-10-19T13:04:23.244297+0000","last_became_peered":"2023-10-19T13:04:23.244297+0000","last_unstale":"2023-10-19T13:04:23.254030+0000","last_undegraded":"2023-10-19T13:04:23.254030+0000","last_fullsized":"2023-10-19T13:04:23.254030+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:33:36.981974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.2d","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.257006+0000","last_change":"2023-10-19T13:04:23.253311+0000","last_active":"2023-10-19T13:04:23.257006+0000","last_peered":"2023-10-19T13:04:23.257006+0000","last_clean":"2023-10-19T13:04:23.257006+0000","last_became_active":"2023-10-19T13:04:23.253046+0000","last_became_peered":"2023-10-19T13:04:23.253046+0000","last_unstale":"2023-10-19T13:04:23.257006+0000","last_undegraded":"2023-10-19T13:04:23.257006+0000","last_fullsized":"2023-10-19T13:04:23.257006+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:17:54.965079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.2c","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.258646+0000","last_change":"2023-10-19T13:04:23.250054+0000","last_active":"2023-10-19T13:04:23.258646+0000","last_peered":"2023-10-19T13:04:23.258646+0000","last_clean":"2023-10-19T13:04:23.258646+0000","last_became_active":"2023-10-19T13:04:23.249773+0000","last_became_peered":"2023-10-19T13:04:23.249773+0000","last_unstale":"2023-10-19T13:04:23.258646+0000","last_undegraded":"2023-10-19T13:04:23.258646+0000","last_fullsized":"2023-10-19T13:04:23.258646+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:13:37.185717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.2b","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.253515+0000","last_change":"2023-10-19T13:04:23.252029+0000","last_active":"2023-10-19T13:04:23.253515+0000","last_peered":"2023-10-19T13:04:23.253515+0000","last_clean":"2023-10-19T13:04:23.253515+0000","last_became_active":"2023-10-19T13:04:23.251603+0000","last_became_peered":"2023-10-19T13:04:23.251603+0000","last_unstale":"2023-10-19T13:04:23.253515+0000","last_undegraded":"2023-10-19T13:04:23.253515+0000","last_fullsized":"2023-10-19T13:04:23.253515+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:11:20.296382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.2a","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.259595+0000","last_change":"2023-10-19T13:04:23.254291+0000","last_active":"2023-10-19T13:04:23.259595+0000","last_peered":"2023-10-19T13:04:23.259595+0000","last_clean":"2023-10-19T13:04:23.259595+0000","last_became_active":"2023-10-19T13:04:23.254000+0000","last_became_peered":"2023-10-19T13:04:23.254000+0000","last_unstale":"2023-10-19T13:04:23.259595+0000","last_undegraded":"2023-10-19T13:04:23.259595+0000","last_fullsized":"2023-10-19T13:04:23.259595+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:04:15.321219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.29","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.255308+0000","last_change":"2023-10-19T13:04:23.254920+0000","last_active":"2023-10-19T13:04:23.255308+0000","last_peered":"2023-10-19T13:04:23.255308+0000","last_clean":"2023-10-19T13:04:23.255308+0000","last_became_active":"2023-10-19T13:04:23.254661+0000","last_became_peered":"2023-10-19T13:04:23.254661+0000","last_unstale":"2023-10-19T13:04:23.255308+0000","last_undegraded":"2023-10-19T13:04:23.255308+0000","last_fullsized":"2023-10-19T13:04:23.255308+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:43:05.064367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.28","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.247313+0000","last_change":"2023-10-19T13:04:23.246405+0000","last_active":"2023-10-19T13:04:23.247313+0000","last_peered":"2023-10-19T13:04:23.247313+0000","last_clean":"2023-10-19T13:04:23.247313+0000","last_became_active":"2023-10-19T13:04:23.246135+0000","last_became_peered":"2023-10-19T13:04:23.246135+0000","last_unstale":"2023-10-19T13:04:23.247313+0000","last_undegraded":"2023-10-19T13:04:23.247313+0000","last_fullsized":"2023-10-19T13:04:23.247313+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:17:34.123593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.27","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.248444+0000","last_change":"2023-10-19T13:04:23.247245+0000","last_active":"2023-10-19T13:04:23.248444+0000","last_peered":"2023-10-19T13:04:23.248444+0000","last_clean":"2023-10-19T13:04:23.248444+0000","last_became_active":"2023-10-19T13:04:23.246943+0000","last_became_peered":"2023-10-19T13:04:23.246943+0000","last_unstale":"2023-10-19T13:04:23.248444+0000","last_undegraded":"2023-10-19T13:04:23.248444+0000","last_fullsized":"2023-10-19T13:04:23.248444+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:30:34.933118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.26","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.258587+0000","last_change":"2023-10-19T13:04:23.250036+0000","last_active":"2023-10-19T13:04:23.258587+0000","last_peered":"2023-10-19T13:04:23.258587+0000","last_clean":"2023-10-19T13:04:23.258587+0000","last_became_active":"2023-10-19T13:04:23.249602+0000","last_became_peered":"2023-10-19T13:04:23.249602+0000","last_unstale":"2023-10-19T13:04:23.258587+0000","last_undegraded":"2023-10-19T13:04:23.258587+0000","last_fullsized":"2023-10-19T13:04:23.258587+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:03:26.256383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.25","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.257017+0000","last_change":"2023-10-19T13:04:23.253529+0000","last_active":"2023-10-19T13:04:23.257017+0000","last_peered":"2023-10-19T13:04:23.257017+0000","last_clean":"2023-10-19T13:04:23.257017+0000","last_became_active":"2023-10-19T13:04:23.253265+0000","last_became_peered":"2023-10-19T13:04:23.253265+0000","last_unstale":"2023-10-19T13:04:23.257017+0000","last_undegraded":"2023-10-19T13:04:23.257017+0000","last_fullsized":"2023-10-19T13:04:23.257017+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:20:45.859782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.24","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.258512+0000","last_change":"2023-10-19T13:04:23.249000+0000","last_active":"2023-10-19T13:04:23.258512+0000","last_peered":"2023-10-19T13:04:23.258512+0000","last_clean":"2023-10-19T13:04:23.258512+0000","last_became_active":"2023-10-19T13:04:23.248580+0000","last_became_peered":"2023-10-19T13:04:23.248580+0000","last_unstale":"2023-10-19T13:04:23.258512+0000","last_undegraded":"2023-10-19T13:04:23.258512+0000","last_fullsized":"2023-10-19T13:04:23.258512+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:23:04.278774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.23","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.258771+0000","last_change":"2023-10-19T13:04:23.254509+0000","last_active":"2023-10-19T13:04:23.258771+0000","last_peered":"2023-10-19T13:04:23.258771+0000","last_clean":"2023-10-19T13:04:23.258771+0000","last_became_active":"2023-10-19T13:04:23.254180+0000","last_became_peered":"2023-10-19T13:04:23.254180+0000","last_unstale":"2023-10-19T13:04:23.258771+0000","last_undegraded":"2023-10-19T13:04:23.258771+0000","last_fullsized":"2023-10-19T13:04:23.258771+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:54:48.817501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.22","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.254396+0000","last_change":"2023-10-19T13:04:23.245440+0000","last_active":"2023-10-19T13:04:23.254396+0000","last_peered":"2023-10-19T13:04:23.254396+0000","last_clean":"2023-10-19T13:04:23.254396+0000","last_became_active":"2023-10-19T13:04:23.244956+0000","last_became_peered":"2023-10-19T13:04:23.244956+0000","last_unstale":"2023-10-19T13:04:23.254396+0000","last_undegraded":"2023-10-19T13:04:23.254396+0000","last_fullsized":"2023-10-19T13:04:23.254396+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:42:32.750724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.21","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.243656+0000","last_change":"2023-10-19T13:04:23.242180+0000","last_active":"2023-10-19T13:04:23.243656+0000","last_peered":"2023-10-19T13:04:23.243656+0000","last_clean":"2023-10-19T13:04:23.243656+0000","last_became_active":"2023-10-19T13:04:23.241890+0000","last_became_peered":"2023-10-19T13:04:23.241890+0000","last_unstale":"2023-10-19T13:04:23.243656+0000","last_undegraded":"2023-10-19T13:04:23.243656+0000","last_fullsized":"2023-10-19T13:04:23.243656+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:13:41.778030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.20","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.245859+0000","last_change":"2023-10-19T13:04:23.245272+0000","last_active":"2023-10-19T13:04:23.245859+0000","last_peered":"2023-10-19T13:04:23.245859+0000","last_clean":"2023-10-19T13:04:23.245859+0000","last_became_active":"2023-10-19T13:04:23.245017+0000","last_became_peered":"2023-10-19T13:04:23.245017+0000","last_unstale":"2023-10-19T13:04:23.245859+0000","last_undegraded":"2023-10-19T13:04:23.245859+0000","last_fullsized":"2023-10-19T13:04:23.245859+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:20:38.292322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1c","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.258810+0000","last_change":"2023-10-19T13:04:23.250759+0000","last_active":"2023-10-19T13:04:23.258810+0000","last_peered":"2023-10-19T13:04:23.258810+0000","last_clean":"2023-10-19T13:04:23.258810+0000","last_became_active":"2023-10-19T13:04:23.250487+0000","last_became_peered":"2023-10-19T13:04:23.250487+0000","last_unstale":"2023-10-19T13:04:23.258810+0000","last_undegraded":"2023-10-19T13:04:23.258810+0000","last_fullsized":"2023-10-19T13:04:23.258810+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:20:45.859782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1f","version":"143'4","reported_seq":15,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.430345+0000","last_change":"2023-10-19T13:04:22.440574+0000","last_active":"2023-10-19T13:04:23.430345+0000","last_peered":"2023-10-19T13:04:23.430345+0000","last_clean":"2023-10-19T13:04:23.430345+0000","last_became_active":"2023-10-19T13:04:22.440310+0000","last_became_peered":"2023-10-19T13:04:22.440310+0000","last_unstale":"2023-10-19T13:04:23.430345+0000","last_undegraded":"2023-10-19T13:04:23.430345+0000","last_fullsized":"2023-10-19T13:04:23.430345+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_clean_scrub_stamp":"2023-10-19T13:04:21.401563+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:22:12.862717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1d","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.258872+0000","last_change":"2023-10-19T13:04:23.255051+0000","last_active":"2023-10-19T13:04:23.258872+0000","last_peered":"2023-10-19T13:04:23.258872+0000","last_clean":"2023-10-19T13:04:23.258872+0000","last_became_active":"2023-10-19T13:04:23.254731+0000","last_became_peered":"2023-10-19T13:04:23.254731+0000","last_unstale":"2023-10-19T13:04:23.258872+0000","last_undegraded":"2023-10-19T13:04:23.258872+0000","last_fullsized":"2023-10-19T13:04:23.258872+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:34:18.198764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1e","version":"143'1","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.429585+0000","last_change":"2023-10-19T13:04:22.439332+0000","last_active":"2023-10-19T13:04:23.429585+0000","last_peered":"2023-10-19T13:04:23.429585+0000","last_clean":"2023-10-19T13:04:23.429585+0000","last_became_active":"2023-10-19T13:04:22.437985+0000","last_became_peered":"2023-10-19T13:04:22.437985+0000","last_unstale":"2023-10-19T13:04:23.429585+0000","last_undegraded":"2023-10-19T13:04:23.429585+0000","last_fullsized":"2023-10-19T13:04:23.429585+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_clean_scrub_stamp":"2023-10-19T13:04:21.401563+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:52:40.514630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.1e","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.256790+0000","last_change":"2023-10-19T13:04:23.254221+0000","last_active":"2023-10-19T13:04:23.256790+0000","last_peered":"2023-10-19T13:04:23.256790+0000","last_clean":"2023-10-19T13:04:23.256790+0000","last_became_active":"2023-10-19T13:04:23.253941+0000","last_became_peered":"2023-10-19T13:04:23.253941+0000","last_unstale":"2023-10-19T13:04:23.256790+0000","last_undegraded":"2023-10-19T13:04:23.256790+0000","last_fullsized":"2023-10-19T13:04:23.256790+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:58:24.553560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1d","version":"143'2","reported_seq":13,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.429667+0000","last_change":"2023-10-19T13:04:22.441364+0000","last_active":"2023-10-19T13:04:23.429667+0000","last_peered":"2023-10-19T13:04:23.429667+0000","last_clean":"2023-10-19T13:04:23.429667+0000","last_became_active":"2023-10-19T13:04:22.440928+0000","last_became_peered":"2023-10-19T13:04:22.440928+0000","last_unstale":"2023-10-19T13:04:23.429667+0000","last_undegraded":"2023-10-19T13:04:23.429667+0000","last_fullsized":"2023-10-19T13:04:23.429667+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_clean_scrub_stamp":"2023-10-19T13:04:21.401563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:30:33.923501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1f","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.252997+0000","last_change":"2023-10-19T13:04:23.250194+0000","last_active":"2023-10-19T13:04:23.252997+0000","last_peered":"2023-10-19T13:04:23.252997+0000","last_clean":"2023-10-19T13:04:23.252997+0000","last_became_active":"2023-10-19T13:04:23.249919+0000","last_became_peered":"2023-10-19T13:04:23.249919+0000","last_unstale":"2023-10-19T13:04:23.252997+0000","last_undegraded":"2023-10-19T13:04:23.252997+0000","last_fullsized":"2023-10-19T13:04:23.252997+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:52:21.807380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"143'2","reported_seq":13,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.432710+0000","last_change":"2023-10-19T13:04:22.443417+0000","last_active":"2023-10-19T13:04:23.432710+0000","last_peered":"2023-10-19T13:04:23.432710+0000","last_clean":"2023-10-19T13:04:23.432710+0000","last_became_active":"2023-10-19T13:04:22.443116+0000","last_became_peered":"2023-10-19T13:04:22.443116+0000","last_unstale":"2023-10-19T13:04:23.432710+0000","last_undegraded":"2023-10-19T13:04:23.432710+0000","last_fullsized":"2023-10-19T13:04:23.432710+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_clean_scrub_stamp":"2023-10-19T13:04:21.401563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:06:23.518443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.253449+0000","last_change":"2023-10-19T13:04:23.244883+0000","last_active":"2023-10-19T13:04:23.253449+0000","last_peered":"2023-10-19T13:04:23.253449+0000","last_clean":"2023-10-19T13:04:23.253449+0000","last_became_active":"2023-10-19T13:04:23.244621+0000","last_became_peered":"2023-10-19T13:04:23.244621+0000","last_unstale":"2023-10-19T13:04:23.253449+0000","last_undegraded":"2023-10-19T13:04:23.253449+0000","last_fullsized":"2023-10-19T13:04:23.253449+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:34:21.904012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1b","version":"143'1","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.432410+0000","last_change":"2023-10-19T13:04:22.442894+0000","last_active":"2023-10-19T13:04:23.432410+0000","last_peered":"2023-10-19T13:04:23.432410+0000","last_clean":"2023-10-19T13:04:23.432410+0000","last_became_active":"2023-10-19T13:04:22.442627+0000","last_became_peered":"2023-10-19T13:04:22.442627+0000","last_unstale":"2023-10-19T13:04:23.432410+0000","last_undegraded":"2023-10-19T13:04:23.432410+0000","last_fullsized":"2023-10-19T13:04:23.432410+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_clean_scrub_stamp":"2023-10-19T13:04:21.401563+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:00:49.819076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.247693+0000","last_change":"2023-10-19T13:04:23.246321+0000","last_active":"2023-10-19T13:04:23.247693+0000","last_peered":"2023-10-19T13:04:23.247693+0000","last_clean":"2023-10-19T13:04:23.247693+0000","last_became_active":"2023-10-19T13:04:23.246028+0000","last_became_peered":"2023-10-19T13:04:23.246028+0000","last_unstale":"2023-10-19T13:04:23.247693+0000","last_undegraded":"2023-10-19T13:04:23.247693+0000","last_fullsized":"2023-10-19T13:04:23.247693+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:52:41.524246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1a","version":"143'1","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.432691+0000","last_change":"2023-10-19T13:04:22.444403+0000","last_active":"2023-10-19T13:04:23.432691+0000","last_peered":"2023-10-19T13:04:23.432691+0000","last_clean":"2023-10-19T13:04:23.432691+0000","last_became_active":"2023-10-19T13:04:22.444072+0000","last_became_peered":"2023-10-19T13:04:22.444072+0000","last_unstale":"2023-10-19T13:04:23.432691+0000","last_undegraded":"2023-10-19T13:04:23.432691+0000","last_fullsized":"2023-10-19T13:04:23.432691+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_clean_scrub_stamp":"2023-10-19T13:04:21.401563+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:03:51.165898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.1a","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.253737+0000","last_change":"2023-10-19T13:04:23.242570+0000","last_active":"2023-10-19T13:04:23.253737+0000","last_peered":"2023-10-19T13:04:23.253737+0000","last_clean":"2023-10-19T13:04:23.253737+0000","last_became_active":"2023-10-19T13:04:23.242330+0000","last_became_peered":"2023-10-19T13:04:23.242330+0000","last_unstale":"2023-10-19T13:04:23.253737+0000","last_undegraded":"2023-10-19T13:04:23.253737+0000","last_fullsized":"2023-10-19T13:04:23.253737+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:26:52.277682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.231246+0000","last_change":"2023-10-19T13:04:22.439373+0000","last_active":"2023-10-19T13:04:23.231246+0000","last_peered":"2023-10-19T13:04:23.231246+0000","last_clean":"2023-10-19T13:04:23.231246+0000","last_became_active":"2023-10-19T13:04:22.438988+0000","last_became_peered":"2023-10-19T13:04:22.438988+0000","last_unstale":"2023-10-19T13:04:23.231246+0000","last_undegraded":"2023-10-19T13:04:23.231246+0000","last_fullsized":"2023-10-19T13:04:23.231246+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_clean_scrub_stamp":"2023-10-19T13:04:21.401563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:56:59.711452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1b","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.253533+0000","last_change":"2023-10-19T13:04:23.243508+0000","last_active":"2023-10-19T13:04:23.253533+0000","last_peered":"2023-10-19T13:04:23.253533+0000","last_clean":"2023-10-19T13:04:23.253533+0000","last_became_active":"2023-10-19T13:04:23.243199+0000","last_became_peered":"2023-10-19T13:04:23.243199+0000","last_unstale":"2023-10-19T13:04:23.253533+0000","last_undegraded":"2023-10-19T13:04:23.253533+0000","last_fullsized":"2023-10-19T13:04:23.253533+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:04:15.321219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.242695+0000","last_change":"2023-10-19T13:04:22.436803+0000","last_active":"2023-10-19T13:04:23.242695+0000","last_peered":"2023-10-19T13:04:23.242695+0000","last_clean":"2023-10-19T13:04:23.242695+0000","last_became_active":"2023-10-19T13:04:22.436550+0000","last_became_peered":"2023-10-19T13:04:22.436550+0000","last_unstale":"2023-10-19T13:04:23.242695+0000","last_undegraded":"2023-10-19T13:04:23.242695+0000","last_fullsized":"2023-10-19T13:04:23.242695+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_clean_scrub_stamp":"2023-10-19T13:04:21.401563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:23:11.451513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.14","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.256870+0000","last_change":"2023-10-19T13:04:23.251227+0000","last_active":"2023-10-19T13:04:23.256870+0000","last_peered":"2023-10-19T13:04:23.256870+0000","last_clean":"2023-10-19T13:04:23.256870+0000","last_became_active":"2023-10-19T13:04:23.250906+0000","last_became_peered":"2023-10-19T13:04:23.250906+0000","last_unstale":"2023-10-19T13:04:23.256870+0000","last_undegraded":"2023-10-19T13:04:23.256870+0000","last_fullsized":"2023-10-19T13:04:23.256870+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:06:24.528059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.241047+0000","last_change":"2023-10-19T13:04:22.448426+0000","last_active":"2023-10-19T13:04:23.241047+0000","last_peered":"2023-10-19T13:04:23.241047+0000","last_clean":"2023-10-19T13:04:23.241047+0000","last_became_active":"2023-10-19T13:04:22.448121+0000","last_became_peered":"2023-10-19T13:04:22.448121+0000","last_unstale":"2023-10-19T13:04:23.241047+0000","last_undegraded":"2023-10-19T13:04:23.241047+0000","last_fullsized":"2023-10-19T13:04:23.241047+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_clean_scrub_stamp":"2023-10-19T13:04:21.401563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:48:29.693840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.15","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.245588+0000","last_change":"2023-10-19T13:04:23.241676+0000","last_active":"2023-10-19T13:04:23.245588+0000","last_peered":"2023-10-19T13:04:23.245588+0000","last_clean":"2023-10-19T13:04:23.245588+0000","last_became_active":"2023-10-19T13:04:23.241372+0000","last_became_peered":"2023-10-19T13:04:23.241372+0000","last_unstale":"2023-10-19T13:04:23.245588+0000","last_undegraded":"2023-10-19T13:04:23.245588+0000","last_fullsized":"2023-10-19T13:04:23.245588+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:52:21.807380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.230451+0000","last_change":"2023-10-19T13:04:22.440265+0000","last_active":"2023-10-19T13:04:23.230451+0000","last_peered":"2023-10-19T13:04:23.230451+0000","last_clean":"2023-10-19T13:04:23.230451+0000","last_became_active":"2023-10-19T13:04:22.439995+0000","last_became_peered":"2023-10-19T13:04:22.439995+0000","last_unstale":"2023-10-19T13:04:23.230451+0000","last_undegraded":"2023-10-19T13:04:23.230451+0000","last_fullsized":"2023-10-19T13:04:23.230451+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_clean_scrub_stamp":"2023-10-19T13:04:21.401563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:00:49.460528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.254415+0000","last_change":"2023-10-19T13:04:23.246009+0000","last_active":"2023-10-19T13:04:23.254415+0000","last_peered":"2023-10-19T13:04:23.254415+0000","last_clean":"2023-10-19T13:04:23.254415+0000","last_became_active":"2023-10-19T13:04:23.245738+0000","last_became_peered":"2023-10-19T13:04:23.245738+0000","last_unstale":"2023-10-19T13:04:23.254415+0000","last_undegraded":"2023-10-19T13:04:23.254415+0000","last_fullsized":"2023-10-19T13:04:23.254415+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:10:45.863043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.15","version":"143'3","reported_seq":14,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.430874+0000","last_change":"2023-10-19T13:04:22.439823+0000","last_active":"2023-10-19T13:04:23.430874+0000","last_peered":"2023-10-19T13:04:23.430874+0000","last_clean":"2023-10-19T13:04:23.430874+0000","last_became_active":"2023-10-19T13:04:22.439492+0000","last_became_peered":"2023-10-19T13:04:22.439492+0000","last_unstale":"2023-10-19T13:04:23.430874+0000","last_undegraded":"2023-10-19T13:04:23.430874+0000","last_fullsized":"2023-10-19T13:04:23.430874+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_clean_scrub_stamp":"2023-10-19T13:04:21.401563+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:27:12.520515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.245895+0000","last_change":"2023-10-19T13:04:23.244129+0000","last_active":"2023-10-19T13:04:23.245895+0000","last_peered":"2023-10-19T13:04:23.245895+0000","last_clean":"2023-10-19T13:04:23.245895+0000","last_became_active":"2023-10-19T13:04:23.243865+0000","last_became_peered":"2023-10-19T13:04:23.243865+0000","last_unstale":"2023-10-19T13:04:23.245895+0000","last_undegraded":"2023-10-19T13:04:23.245895+0000","last_fullsized":"2023-10-19T13:04:23.245895+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:18:30.649624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.14","version":"143'6","reported_seq":17,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.434547+0000","last_change":"2023-10-19T13:04:22.442680+0000","last_active":"2023-10-19T13:04:23.434547+0000","last_peered":"2023-10-19T13:04:23.434547+0000","last_clean":"2023-10-19T13:04:23.434547+0000","last_became_active":"2023-10-19T13:04:22.442413+0000","last_became_peered":"2023-10-19T13:04:22.442413+0000","last_unstale":"2023-10-19T13:04:23.434547+0000","last_undegraded":"2023-10-19T13:04:23.434547+0000","last_fullsized":"2023-10-19T13:04:23.434547+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_clean_scrub_stamp":"2023-10-19T13:04:21.401563+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:20:13.314826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.10","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.244301+0000","last_change":"2023-10-19T13:04:23.243720+0000","last_active":"2023-10-19T13:04:23.244301+0000","last_peered":"2023-10-19T13:04:23.244301+0000","last_clean":"2023-10-19T13:04:23.244301+0000","last_became_active":"2023-10-19T13:04:23.243308+0000","last_became_peered":"2023-10-19T13:04:23.243308+0000","last_unstale":"2023-10-19T13:04:23.244301+0000","last_undegraded":"2023-10-19T13:04:23.244301+0000","last_fullsized":"2023-10-19T13:04:23.244301+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:47:05.764939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.13","version":"143'2","reported_seq":13,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.432465+0000","last_change":"2023-10-19T13:04:22.442994+0000","last_active":"2023-10-19T13:04:23.432465+0000","last_peered":"2023-10-19T13:04:23.432465+0000","last_clean":"2023-10-19T13:04:23.432465+0000","last_became_active":"2023-10-19T13:04:22.442649+0000","last_became_peered":"2023-10-19T13:04:22.442649+0000","last_unstale":"2023-10-19T13:04:23.432465+0000","last_undegraded":"2023-10-19T13:04:23.432465+0000","last_fullsized":"2023-10-19T13:04:23.432465+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_clean_scrub_stamp":"2023-10-19T13:04:21.401563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:21:17.556322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.253693+0000","last_change":"2023-10-19T13:04:23.246917+0000","last_active":"2023-10-19T13:04:23.253693+0000","last_peered":"2023-10-19T13:04:23.253693+0000","last_clean":"2023-10-19T13:04:23.253693+0000","last_became_active":"2023-10-19T13:04:23.246485+0000","last_became_peered":"2023-10-19T13:04:23.246485+0000","last_unstale":"2023-10-19T13:04:23.253693+0000","last_undegraded":"2023-10-19T13:04:23.253693+0000","last_fullsized":"2023-10-19T13:04:23.253693+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:47:05.764939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.235097+0000","last_change":"2023-10-19T13:04:22.443482+0000","last_active":"2023-10-19T13:04:23.235097+0000","last_peered":"2023-10-19T13:04:23.235097+0000","last_clean":"2023-10-19T13:04:23.235097+0000","last_became_active":"2023-10-19T13:04:22.443193+0000","last_became_peered":"2023-10-19T13:04:22.443193+0000","last_unstale":"2023-10-19T13:04:23.235097+0000","last_undegraded":"2023-10-19T13:04:23.235097+0000","last_fullsized":"2023-10-19T13:04:23.235097+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_clean_scrub_stamp":"2023-10-19T13:04:21.401563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:55:13.479682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.253981+0000","last_change":"2023-10-19T13:04:23.248278+0000","last_active":"2023-10-19T13:04:23.253981+0000","last_peered":"2023-10-19T13:04:23.253981+0000","last_clean":"2023-10-19T13:04:23.253981+0000","last_became_active":"2023-10-19T13:04:23.247991+0000","last_became_peered":"2023-10-19T13:04:23.247991+0000","last_unstale":"2023-10-19T13:04:23.253981+0000","last_undegraded":"2023-10-19T13:04:23.253981+0000","last_fullsized":"2023-10-19T13:04:23.253981+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:55:07.236723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.234786+0000","last_change":"2023-10-19T13:04:22.442674+0000","last_active":"2023-10-19T13:04:23.234786+0000","last_peered":"2023-10-19T13:04:23.234786+0000","last_clean":"2023-10-19T13:04:23.234786+0000","last_became_active":"2023-10-19T13:04:22.442236+0000","last_became_peered":"2023-10-19T13:04:22.442236+0000","last_unstale":"2023-10-19T13:04:23.234786+0000","last_undegraded":"2023-10-19T13:04:23.234786+0000","last_fullsized":"2023-10-19T13:04:23.234786+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_clean_scrub_stamp":"2023-10-19T13:04:21.401563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:44:07.324452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.254283+0000","last_change":"2023-10-19T13:04:23.251683+0000","last_active":"2023-10-19T13:04:23.254283+0000","last_peered":"2023-10-19T13:04:23.254283+0000","last_clean":"2023-10-19T13:04:23.254283+0000","last_became_active":"2023-10-19T13:04:23.251427+0000","last_became_peered":"2023-10-19T13:04:23.251427+0000","last_unstale":"2023-10-19T13:04:23.254283+0000","last_undegraded":"2023-10-19T13:04:23.254283+0000","last_fullsized":"2023-10-19T13:04:23.254283+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:01:25.278083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.f","version":"143'1","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.431966+0000","last_change":"2023-10-19T13:04:22.438449+0000","last_active":"2023-10-19T13:04:23.431966+0000","last_peered":"2023-10-19T13:04:23.431966+0000","last_clean":"2023-10-19T13:04:23.431966+0000","last_became_active":"2023-10-19T13:04:22.438168+0000","last_became_peered":"2023-10-19T13:04:22.438168+0000","last_unstale":"2023-10-19T13:04:23.431966+0000","last_undegraded":"2023-10-19T13:04:23.431966+0000","last_fullsized":"2023-10-19T13:04:23.431966+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_clean_scrub_stamp":"2023-10-19T13:04:21.401563+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:17:53.955463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.d","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.256734+0000","last_change":"2023-10-19T13:04:23.251516+0000","last_active":"2023-10-19T13:04:23.256734+0000","last_peered":"2023-10-19T13:04:23.256734+0000","last_clean":"2023-10-19T13:04:23.256734+0000","last_became_active":"2023-10-19T13:04:23.251247+0000","last_became_peered":"2023-10-19T13:04:23.251247+0000","last_unstale":"2023-10-19T13:04:23.256734+0000","last_undegraded":"2023-10-19T13:04:23.256734+0000","last_fullsized":"2023-10-19T13:04:23.256734+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:47:05.764939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.e","version":"143'2","reported_seq":13,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.432208+0000","last_change":"2023-10-19T13:04:22.435568+0000","last_active":"2023-10-19T13:04:23.432208+0000","last_peered":"2023-10-19T13:04:23.432208+0000","last_clean":"2023-10-19T13:04:23.432208+0000","last_became_active":"2023-10-19T13:04:22.435294+0000","last_became_peered":"2023-10-19T13:04:22.435294+0000","last_unstale":"2023-10-19T13:04:23.432208+0000","last_undegraded":"2023-10-19T13:04:23.432208+0000","last_fullsized":"2023-10-19T13:04:23.432208+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_clean_scrub_stamp":"2023-10-19T13:04:21.401563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:39:21.359154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.e","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.258757+0000","last_change":"2023-10-19T13:04:23.251902+0000","last_active":"2023-10-19T13:04:23.258757+0000","last_peered":"2023-10-19T13:04:23.258757+0000","last_clean":"2023-10-19T13:04:23.258757+0000","last_became_active":"2023-10-19T13:04:23.251679+0000","last_became_peered":"2023-10-19T13:04:23.251679+0000","last_unstale":"2023-10-19T13:04:23.258757+0000","last_undegraded":"2023-10-19T13:04:23.258757+0000","last_fullsized":"2023-10-19T13:04:23.258757+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:45:02.121585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.d","version":"143'1","reported_seq":12,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.430308+0000","last_change":"2023-10-19T13:04:22.441163+0000","last_active":"2023-10-19T13:04:23.430308+0000","last_peered":"2023-10-19T13:04:23.430308+0000","last_clean":"2023-10-19T13:04:23.430308+0000","last_became_active":"2023-10-19T13:04:22.440541+0000","last_became_peered":"2023-10-19T13:04:22.440541+0000","last_unstale":"2023-10-19T13:04:23.430308+0000","last_undegraded":"2023-10-19T13:04:23.430308+0000","last_fullsized":"2023-10-19T13:04:23.430308+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_clean_scrub_stamp":"2023-10-19T13:04:21.401563+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:27:12.520515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.f","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.249063+0000","last_change":"2023-10-19T13:04:23.248187+0000","last_active":"2023-10-19T13:04:23.249063+0000","last_peered":"2023-10-19T13:04:23.249063+0000","last_clean":"2023-10-19T13:04:23.249063+0000","last_became_active":"2023-10-19T13:04:23.247299+0000","last_became_peered":"2023-10-19T13:04:23.247299+0000","last_unstale":"2023-10-19T13:04:23.249063+0000","last_undegraded":"2023-10-19T13:04:23.249063+0000","last_fullsized":"2023-10-19T13:04:23.249063+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:43:05.064367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.239762+0000","last_change":"2023-10-19T13:04:22.448640+0000","last_active":"2023-10-19T13:04:23.239762+0000","last_peered":"2023-10-19T13:04:23.239762+0000","last_clean":"2023-10-19T13:04:23.239762+0000","last_became_active":"2023-10-19T13:04:22.448367+0000","last_became_peered":"2023-10-19T13:04:22.448367+0000","last_unstale":"2023-10-19T13:04:23.239762+0000","last_undegraded":"2023-10-19T13:04:23.239762+0000","last_fullsized":"2023-10-19T13:04:23.239762+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_clean_scrub_stamp":"2023-10-19T13:04:21.401563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:13:40.768413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.253397+0000","last_change":"2023-10-19T13:04:23.245918+0000","last_active":"2023-10-19T13:04:23.253397+0000","last_peered":"2023-10-19T13:04:23.253397+0000","last_clean":"2023-10-19T13:04:23.253397+0000","last_became_active":"2023-10-19T13:04:23.245649+0000","last_became_peered":"2023-10-19T13:04:23.245649+0000","last_unstale":"2023-10-19T13:04:23.253397+0000","last_undegraded":"2023-10-19T13:04:23.253397+0000","last_fullsized":"2023-10-19T13:04:23.253397+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:39:23.090865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.b","version":"143'2","reported_seq":13,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.431626+0000","last_change":"2023-10-19T13:04:22.433508+0000","last_active":"2023-10-19T13:04:23.431626+0000","last_peered":"2023-10-19T13:04:23.431626+0000","last_clean":"2023-10-19T13:04:23.431626+0000","last_became_active":"2023-10-19T13:04:22.433228+0000","last_became_peered":"2023-10-19T13:04:22.433228+0000","last_unstale":"2023-10-19T13:04:23.431626+0000","last_undegraded":"2023-10-19T13:04:23.431626+0000","last_fullsized":"2023-10-19T13:04:23.431626+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_clean_scrub_stamp":"2023-10-19T13:04:21.401563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:33:03.294362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.250161+0000","last_change":"2023-10-19T13:04:23.239176+0000","last_active":"2023-10-19T13:04:23.250161+0000","last_peered":"2023-10-19T13:04:23.250161+0000","last_clean":"2023-10-19T13:04:23.250161+0000","last_became_active":"2023-10-19T13:04:23.238874+0000","last_became_peered":"2023-10-19T13:04:23.238874+0000","last_unstale":"2023-10-19T13:04:23.250161+0000","last_undegraded":"2023-10-19T13:04:23.250161+0000","last_fullsized":"2023-10-19T13:04:23.250161+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:03:08.516373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.239422+0000","last_change":"2023-10-19T13:04:22.438982+0000","last_active":"2023-10-19T13:04:23.239422+0000","last_peered":"2023-10-19T13:04:23.239422+0000","last_clean":"2023-10-19T13:04:23.239422+0000","last_became_active":"2023-10-19T13:04:22.438694+0000","last_became_peered":"2023-10-19T13:04:22.438694+0000","last_unstale":"2023-10-19T13:04:23.239422+0000","last_undegraded":"2023-10-19T13:04:23.239422+0000","last_fullsized":"2023-10-19T13:04:23.239422+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_clean_scrub_stamp":"2023-10-19T13:04:21.401563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:00:49.460528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.254376+0000","last_change":"2023-10-19T13:04:23.245254+0000","last_active":"2023-10-19T13:04:23.254376+0000","last_peered":"2023-10-19T13:04:23.254376+0000","last_clean":"2023-10-19T13:04:23.254376+0000","last_became_active":"2023-10-19T13:04:23.244623+0000","last_became_peered":"2023-10-19T13:04:23.244623+0000","last_unstale":"2023-10-19T13:04:23.254376+0000","last_undegraded":"2023-10-19T13:04:23.254376+0000","last_fullsized":"2023-10-19T13:04:23.254376+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:22:46.466822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.234928+0000","last_change":"2023-10-19T13:04:22.448172+0000","last_active":"2023-10-19T13:04:23.234928+0000","last_peered":"2023-10-19T13:04:23.234928+0000","last_clean":"2023-10-19T13:04:23.234928+0000","last_became_active":"2023-10-19T13:04:22.447872+0000","last_became_peered":"2023-10-19T13:04:22.447872+0000","last_unstale":"2023-10-19T13:04:23.234928+0000","last_undegraded":"2023-10-19T13:04:23.234928+0000","last_fullsized":"2023-10-19T13:04:23.234928+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_clean_scrub_stamp":"2023-10-19T13:04:21.401563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:53:16.490214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.b","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.246801+0000","last_change":"2023-10-19T13:04:23.245772+0000","last_active":"2023-10-19T13:04:23.246801+0000","last_peered":"2023-10-19T13:04:23.246801+0000","last_clean":"2023-10-19T13:04:23.246801+0000","last_became_active":"2023-10-19T13:04:23.245523+0000","last_became_peered":"2023-10-19T13:04:23.245523+0000","last_unstale":"2023-10-19T13:04:23.246801+0000","last_undegraded":"2023-10-19T13:04:23.246801+0000","last_fullsized":"2023-10-19T13:04:23.246801+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:00:22.625725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.240768+0000","last_change":"2023-10-19T13:04:22.439768+0000","last_active":"2023-10-19T13:04:23.240768+0000","last_peered":"2023-10-19T13:04:23.240768+0000","last_clean":"2023-10-19T13:04:23.240768+0000","last_became_active":"2023-10-19T13:04:22.438538+0000","last_became_peered":"2023-10-19T13:04:22.438538+0000","last_unstale":"2023-10-19T13:04:23.240768+0000","last_undegraded":"2023-10-19T13:04:23.240768+0000","last_fullsized":"2023-10-19T13:04:23.240768+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_clean_scrub_stamp":"2023-10-19T13:04:21.401563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:14:52.753658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.258684+0000","last_change":"2023-10-19T13:04:23.249836+0000","last_active":"2023-10-19T13:04:23.258684+0000","last_peered":"2023-10-19T13:04:23.258684+0000","last_clean":"2023-10-19T13:04:23.258684+0000","last_became_active":"2023-10-19T13:04:23.249592+0000","last_became_peered":"2023-10-19T13:04:23.249592+0000","last_unstale":"2023-10-19T13:04:23.258684+0000","last_undegraded":"2023-10-19T13:04:23.258684+0000","last_fullsized":"2023-10-19T13:04:23.258684+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:51:56.835268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.7","version":"143'2","reported_seq":13,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.431713+0000","last_change":"2023-10-19T13:04:22.435611+0000","last_active":"2023-10-19T13:04:23.431713+0000","last_peered":"2023-10-19T13:04:23.431713+0000","last_clean":"2023-10-19T13:04:23.431713+0000","last_became_active":"2023-10-19T13:04:22.435339+0000","last_became_peered":"2023-10-19T13:04:22.435339+0000","last_unstale":"2023-10-19T13:04:23.431713+0000","last_undegraded":"2023-10-19T13:04:23.431713+0000","last_fullsized":"2023-10-19T13:04:23.431713+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_clean_scrub_stamp":"2023-10-19T13:04:21.401563+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:10:14.617741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.253496+0000","last_change":"2023-10-19T13:04:23.244187+0000","last_active":"2023-10-19T13:04:23.253496+0000","last_peered":"2023-10-19T13:04:23.253496+0000","last_clean":"2023-10-19T13:04:23.253496+0000","last_became_active":"2023-10-19T13:04:23.243871+0000","last_became_peered":"2023-10-19T13:04:23.243871+0000","last_unstale":"2023-10-19T13:04:23.253496+0000","last_undegraded":"2023-10-19T13:04:23.253496+0000","last_fullsized":"2023-10-19T13:04:23.253496+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:08:34.773183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.241843+0000","last_change":"2023-10-19T13:04:22.447531+0000","last_active":"2023-10-19T13:04:23.241843+0000","last_peered":"2023-10-19T13:04:23.241843+0000","last_clean":"2023-10-19T13:04:23.241843+0000","last_became_active":"2023-10-19T13:04:22.447291+0000","last_became_peered":"2023-10-19T13:04:22.447291+0000","last_unstale":"2023-10-19T13:04:23.241843+0000","last_undegraded":"2023-10-19T13:04:23.241843+0000","last_fullsized":"2023-10-19T13:04:23.241843+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_clean_scrub_stamp":"2023-10-19T13:04:21.401563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:00:49.819076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.252447+0000","last_change":"2023-10-19T13:04:23.249549+0000","last_active":"2023-10-19T13:04:23.252447+0000","last_peered":"2023-10-19T13:04:23.252447+0000","last_clean":"2023-10-19T13:04:23.252447+0000","last_became_active":"2023-10-19T13:04:23.249189+0000","last_became_peered":"2023-10-19T13:04:23.249189+0000","last_unstale":"2023-10-19T13:04:23.252447+0000","last_undegraded":"2023-10-19T13:04:23.252447+0000","last_fullsized":"2023-10-19T13:04:23.252447+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:57:00.721069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.239566+0000","last_change":"2023-10-19T13:04:22.437924+0000","last_active":"2023-10-19T13:04:23.239566+0000","last_peered":"2023-10-19T13:04:23.239566+0000","last_clean":"2023-10-19T13:04:23.239566+0000","last_became_active":"2023-10-19T13:04:22.437639+0000","last_became_peered":"2023-10-19T13:04:22.437639+0000","last_unstale":"2023-10-19T13:04:23.239566+0000","last_undegraded":"2023-10-19T13:04:23.239566+0000","last_fullsized":"2023-10-19T13:04:23.239566+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_clean_scrub_stamp":"2023-10-19T13:04:21.401563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:04:14.311603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.7","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.249152+0000","last_change":"2023-10-19T13:04:23.247801+0000","last_active":"2023-10-19T13:04:23.249152+0000","last_peered":"2023-10-19T13:04:23.249152+0000","last_clean":"2023-10-19T13:04:23.249152+0000","last_became_active":"2023-10-19T13:04:23.247543+0000","last_became_peered":"2023-10-19T13:04:23.247543+0000","last_unstale":"2023-10-19T13:04:23.249152+0000","last_undegraded":"2023-10-19T13:04:23.249152+0000","last_fullsized":"2023-10-19T13:04:23.249152+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:21:44.668299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.242489+0000","last_change":"2023-10-19T13:04:22.435346+0000","last_active":"2023-10-19T13:04:23.242489+0000","last_peered":"2023-10-19T13:04:23.242489+0000","last_clean":"2023-10-19T13:04:23.242489+0000","last_became_active":"2023-10-19T13:04:22.434909+0000","last_became_peered":"2023-10-19T13:04:22.434909+0000","last_unstale":"2023-10-19T13:04:23.242489+0000","last_undegraded":"2023-10-19T13:04:23.242489+0000","last_fullsized":"2023-10-19T13:04:23.242489+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_clean_scrub_stamp":"2023-10-19T13:04:21.401563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:12:16.733002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.253416+0000","last_change":"2023-10-19T13:04:23.247396+0000","last_active":"2023-10-19T13:04:23.253416+0000","last_peered":"2023-10-19T13:04:23.253416+0000","last_clean":"2023-10-19T13:04:23.253416+0000","last_became_active":"2023-10-19T13:04:23.247133+0000","last_became_peered":"2023-10-19T13:04:23.247133+0000","last_unstale":"2023-10-19T13:04:23.253416+0000","last_undegraded":"2023-10-19T13:04:23.253416+0000","last_fullsized":"2023-10-19T13:04:23.253416+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:59:04.607606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.240021+0000","last_change":"2023-10-19T13:04:22.439734+0000","last_active":"2023-10-19T13:04:23.240021+0000","last_peered":"2023-10-19T13:04:23.240021+0000","last_clean":"2023-10-19T13:04:23.240021+0000","last_became_active":"2023-10-19T13:04:22.439471+0000","last_became_peered":"2023-10-19T13:04:22.439471+0000","last_unstale":"2023-10-19T13:04:23.240021+0000","last_undegraded":"2023-10-19T13:04:23.240021+0000","last_fullsized":"2023-10-19T13:04:23.240021+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_clean_scrub_stamp":"2023-10-19T13:04:21.401563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:02:52.376821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.254450+0000","last_change":"2023-10-19T13:04:23.246875+0000","last_active":"2023-10-19T13:04:23.254450+0000","last_peered":"2023-10-19T13:04:23.254450+0000","last_clean":"2023-10-19T13:04:23.254450+0000","last_became_active":"2023-10-19T13:04:23.246355+0000","last_became_peered":"2023-10-19T13:04:23.246355+0000","last_unstale":"2023-10-19T13:04:23.254450+0000","last_undegraded":"2023-10-19T13:04:23.254450+0000","last_fullsized":"2023-10-19T13:04:23.254450+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:16:20.228806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.239404+0000","last_change":"2023-10-19T13:04:22.435164+0000","last_active":"2023-10-19T13:04:23.239404+0000","last_peered":"2023-10-19T13:04:23.239404+0000","last_clean":"2023-10-19T13:04:23.239404+0000","last_became_active":"2023-10-19T13:04:22.434903+0000","last_became_peered":"2023-10-19T13:04:22.434903+0000","last_unstale":"2023-10-19T13:04:23.239404+0000","last_undegraded":"2023-10-19T13:04:23.239404+0000","last_fullsized":"2023-10-19T13:04:23.239404+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_clean_scrub_stamp":"2023-10-19T13:04:21.401563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:22:12.862717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.253041+0000","last_change":"2023-10-19T13:04:23.250788+0000","last_active":"2023-10-19T13:04:23.253041+0000","last_peered":"2023-10-19T13:04:23.253041+0000","last_clean":"2023-10-19T13:04:23.253041+0000","last_became_active":"2023-10-19T13:04:23.250521+0000","last_became_peered":"2023-10-19T13:04:23.250521+0000","last_unstale":"2023-10-19T13:04:23.253041+0000","last_undegraded":"2023-10-19T13:04:23.253041+0000","last_fullsized":"2023-10-19T13:04:23.253041+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:18:30.649624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.241756+0000","last_change":"2023-10-19T13:04:22.437643+0000","last_active":"2023-10-19T13:04:23.241756+0000","last_peered":"2023-10-19T13:04:23.241756+0000","last_clean":"2023-10-19T13:04:23.241756+0000","last_became_active":"2023-10-19T13:04:22.437351+0000","last_became_peered":"2023-10-19T13:04:22.437351+0000","last_unstale":"2023-10-19T13:04:23.241756+0000","last_undegraded":"2023-10-19T13:04:23.241756+0000","last_fullsized":"2023-10-19T13:04:23.241756+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_clean_scrub_stamp":"2023-10-19T13:04:21.401563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:40:27.499691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.253572+0000","last_change":"2023-10-19T13:04:23.244134+0000","last_active":"2023-10-19T13:04:23.253572+0000","last_peered":"2023-10-19T13:04:23.253572+0000","last_clean":"2023-10-19T13:04:23.253572+0000","last_became_active":"2023-10-19T13:04:23.243815+0000","last_became_peered":"2023-10-19T13:04:23.243815+0000","last_unstale":"2023-10-19T13:04:23.253572+0000","last_undegraded":"2023-10-19T13:04:23.253572+0000","last_fullsized":"2023-10-19T13:04:23.253572+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:22:13.872334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":11,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.240907+0000","last_change":"2023-10-19T13:04:22.434454+0000","last_active":"2023-10-19T13:04:23.240907+0000","last_peered":"2023-10-19T13:04:23.240907+0000","last_clean":"2023-10-19T13:04:23.240907+0000","last_became_active":"2023-10-19T13:04:22.434174+0000","last_became_peered":"2023-10-19T13:04:22.434174+0000","last_unstale":"2023-10-19T13:04:23.240907+0000","last_undegraded":"2023-10-19T13:04:23.240907+0000","last_fullsized":"2023-10-19T13:04:23.240907+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_clean_scrub_stamp":"2023-10-19T13:04:21.401563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:55:52.552633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"0'0","reported_seq":9,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.253496+0000","last_change":"2023-10-19T13:04:23.251854+0000","last_active":"2023-10-19T13:04:23.253496+0000","last_peered":"2023-10-19T13:04:23.253496+0000","last_clean":"2023-10-19T13:04:23.253496+0000","last_became_active":"2023-10-19T13:04:23.250574+0000","last_became_peered":"2023-10-19T13:04:23.250574+0000","last_unstale":"2023-10-19T13:04:23.253496+0000","last_undegraded":"2023-10-19T13:04:23.253496+0000","last_fullsized":"2023-10-19T13:04:23.253496+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:22.411180+0000","last_clean_scrub_stamp":"2023-10-19T13:04:22.411180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:50:32.833828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.10","version":"143'8","reported_seq":19,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-19T13:04:23.432869+0000","last_change":"2023-10-19T13:04:22.442776+0000","last_active":"2023-10-19T13:04:23.432869+0000","last_peered":"2023-10-19T13:04:23.432869+0000","last_clean":"2023-10-19T13:04:23.432869+0000","last_became_active":"2023-10-19T13:04:22.442359+0000","last_became_peered":"2023-10-19T13:04:22.442359+0000","last_unstale":"2023-10-19T13:04:23.432869+0000","last_undegraded":"2023-10-19T13:04:23.432869+0000","last_fullsized":"2023-10-19T13:04:23.432869+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:21.401563+0000","last_clean_scrub_stamp":"2023-10-19T13:04:21.401563+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:35:46.829987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"132'3065","reported_seq":3551,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-19T13:04:22.429456+0000","last_change":"2023-10-19T12:44:08.636613+0000","last_active":"2023-10-19T13:04:22.429456+0000","last_peered":"2023-10-19T13:04:22.429456+0000","last_clean":"2023-10-19T13:04:22.429456+0000","last_became_active":"2023-10-19T12:44:08.635697+0000","last_became_peered":"2023-10-19T12:44:08.635697+0000","last_unstale":"2023-10-19T13:04:22.429456+0000","last_undegraded":"2023-10-19T13:04:22.429456+0000","last_fullsized":"2023-10-19T13:04:22.429456+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_clean_scrub_stamp":"2023-10-19T12:44:07.399738+0000","objects_scrubbed":0,"log_size":3065,"log_dups_size":0,"ondisk_log_size":3065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:15:23.685485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1154,"num_read_kb":2358,"num_write":369,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":18,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":17,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1154,"num_read_kb":2358,"num_write":369,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3065,"ondisk_log_size":3065,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575093,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55696,"kb_used_data":2304,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316144,"statfs":{"total":96636764160,"available":96579731456,"internally_reserved":0,"allocated":2359296,"data_stored":1482506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54655435},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 13:03:51 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53200000000000003,"15min":0.55900000000000005},"min":{"1min":0.36199999999999999,"5min":0.224,"15min":0.224},"max":{"1min":0.69399999999999995,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56499999999999995,"15min":0.58299999999999996},"min":{"1min":0.45200000000000001,"5min":0.33200000000000002,"15min":0.29599999999999999},"max":{"1min":0.71099999999999997,"5min":0.78200000000000003,"15min":0.93700000000000006},"last":0.503}]},{"osd":1,"last update":"Thu Oct 19 13:03:51 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.55000000000000004,"15min":0.57299999999999995},"min":{"1min":0.45700000000000002,"5min":0.33500000000000002,"15min":0.28199999999999997},"max":{"1min":0.82399999999999995,"5min":0.84299999999999997,"15min":1.3129999999999999},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.56100000000000005,"15min":0.58099999999999996},"min":{"1min":0.39100000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.72999999999999998,"5min":0.85499999999999998,"15min":1.284},"last":0.42599999999999999}]},{"osd":2,"last update":"Thu Oct 19 13:03:51 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.60799999999999998,"15min":0.623},"min":{"1min":0.53200000000000003,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.874,"5min":0.89200000000000002,"15min":0.98599999999999999},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.61099999999999999,"15min":0.63},"min":{"1min":0.52100000000000002,"5min":0.40300000000000002,"15min":0.28399999999999997},"max":{"1min":0.749,"5min":0.83699999999999997,"15min":1.2589999999999999},"last":0.59499999999999997}]},{"osd":3,"last update":"Thu Oct 19 13:03:51 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.57499999999999996,"15min":0.59899999999999998},"min":{"1min":0.373,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.76500000000000001,"5min":0.87,"15min":0.96899999999999997},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.64800000000000002,"15min":0.65300000000000002},"min":{"1min":0.56200000000000006,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.83799999999999997,"5min":0.92000000000000004,"15min":1.329},"last":0.61899999999999999}]},{"osd":4,"last update":"Thu Oct 19 13:03:51 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.46700000000000003,"15min":0.46600000000000003},"min":{"1min":0.31900000000000001,"5min":0.247,"15min":0.247},"max":{"1min":0.59399999999999997,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.47199999999999998,"15min":0.46899999999999997},"min":{"1min":0.36099999999999999,"5min":0.216,"15min":0.216},"max":{"1min":0.60799999999999998,"5min":0.73399999999999999,"15min":0.75600000000000001},"last":0.63600000000000001}]},{"osd":5,"last update":"Thu Oct 19 13:03:51 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.49299999999999999,"15min":0.48499999999999999},"min":{"1min":0.40799999999999997,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.78000000000000003,"5min":0.78800000000000003,"15min":0.80600000000000005},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.52300000000000002,"15min":0.51900000000000002},"min":{"1min":0.42099999999999999,"5min":0.32200000000000001,"15min":0.314},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.81999999999999995},"last":0.72999999999999998}]},{"osd":6,"last update":"Thu Oct 19 13:03:51 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.54700000000000004,"15min":0.54600000000000004},"min":{"1min":0.39200000000000002,"5min":0.35999999999999999,"15min":0.308},"max":{"1min":0.71899999999999997,"5min":0.78900000000000003,"15min":0.88900000000000001},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.52600000000000002,"15min":0.53100000000000003},"min":{"1min":0.439,"5min":0.318,"15min":0.27300000000000002},"max":{"1min":0.70399999999999996,"5min":0.77000000000000002,"15min":0.90300000000000002},"last":0.68500000000000005}]}]},{"osd":6,"up_from":13,"seq":55834575093,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32920,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338920,"statfs":{"total":96636764160,"available":96603054080,"internally_reserved":0,"allocated":1777664,"data_stored":899261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 13:03:37 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.56899999999999995,"15min":0.58099999999999996},"min":{"1min":0.38800000000000001,"5min":0.23999999999999999,"15min":0.19500000000000001},"max":{"1min":0.77300000000000002,"5min":1.196,"15min":1.196},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.55900000000000005,"15min":0.57899999999999996},"min":{"1min":0.33300000000000002,"5min":0.28799999999999998,"15min":0.247},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.59999999999999998}]},{"osd":1,"last update":"Thu Oct 19 13:03:37 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.57699999999999996,"15min":0.59399999999999997},"min":{"1min":0.45900000000000002,"5min":0.29099999999999998,"15min":0.27400000000000002},"max":{"1min":0.80100000000000005,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.57999999999999996,"15min":0.59599999999999997},"min":{"1min":0.34300000000000003,"5min":0.307,"15min":0.307},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.82099999999999995},"last":0.67500000000000004}]},{"osd":2,"last update":"Thu Oct 19 13:03:37 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.55700000000000005,"15min":0.57899999999999996},"min":{"1min":0.34699999999999998,"5min":0.34599999999999997,"15min":0.307},"max":{"1min":0.79400000000000004,"5min":0.84699999999999998,"15min":0.85699999999999998},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.621,"5min":0.59699999999999998,"15min":0.60699999999999998},"min":{"1min":0.35699999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.80800000000000005,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.61599999999999999}]},{"osd":3,"last update":"Thu Oct 19 13:03:37 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.60099999999999998,"15min":0.623},"min":{"1min":0.379,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.79600000000000004,"5min":0.871,"15min":0.871},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.60699999999999998,"15min":0.62},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.36799999999999999},"max":{"1min":0.82799999999999996,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.78200000000000003}]},{"osd":4,"last update":"Thu Oct 19 13:03:37 2023","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.47799999999999998,"15min":0.46700000000000003},"min":{"1min":0.29799999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.71499999999999997,"5min":1.353,"15min":1.353},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.47799999999999998,"15min":0.48099999999999998},"min":{"1min":0.32700000000000001,"5min":0.254,"15min":0.254},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.41899999999999998}]},{"osd":5,"last update":"Thu Oct 19 13:03:37 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.46400000000000002,"15min":0.47399999999999998},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.52900000000000003,"15min":0.51900000000000002},"min":{"1min":0.31900000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.85699999999999998},"last":0.48299999999999998}]},{"osd":7,"last update":"Thu Oct 19 13:03:37 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.53600000000000003,"15min":0.52400000000000002},"min":{"1min":0.28000000000000003,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.82499999999999996,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.54100000000000004,"15min":0.53500000000000003},"min":{"1min":0.314,"5min":0.27900000000000003,"15min":0.26000000000000001},"max":{"1min":0.77900000000000003,"5min":1.127,"15min":1.127},"last":0.49399999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575093,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50956,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320884,"statfs":{"total":96636764160,"available":96584585216,"internally_reserved":0,"allocated":1765376,"data_stored":892138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 13:03:33 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57299999999999995,"15min":0.58699999999999997},"min":{"1min":0.39100000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.77100000000000002,"5min":0.79900000000000004,"15min":0.82699999999999996},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.59499999999999997,"15min":0.59699999999999998},"min":{"1min":0.41999999999999998,"5min":0.33900000000000002,"15min":0.27800000000000002},"max":{"1min":0.78200000000000003,"5min":0.82999999999999996,"15min":0.84299999999999997},"last":0.71999999999999997}]},{"osd":1,"last update":"Thu Oct 19 13:03:33 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.58899999999999997,"15min":0.59699999999999998},"min":{"1min":0.45600000000000002,"5min":0.34799999999999998,"15min":0.32800000000000001},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.91200000000000003},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.622,"5min":0.61699999999999999,"15min":0.61099999999999999},"min":{"1min":0.41199999999999998,"5min":0.378,"15min":0.34000000000000002},"max":{"1min":0.77200000000000002,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.78700000000000003}]},{"osd":2,"last update":"Thu Oct 19 13:03:33 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.56499999999999995,"15min":0.57999999999999996},"min":{"1min":0.373,"5min":0.32700000000000001,"15min":0.307},"max":{"1min":0.75700000000000001,"5min":0.77500000000000002,"15min":0.80700000000000005},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.621,"15min":0.61799999999999999},"min":{"1min":0.47599999999999998,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.82499999999999996,"5min":0.83999999999999997,"15min":0.88100000000000001},"last":0.63500000000000001}]},{"osd":3,"last update":"Thu Oct 19 13:03:33 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.59199999999999997,"15min":0.60199999999999998},"min":{"1min":0.37,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.79900000000000004,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.61899999999999999,"15min":0.63},"min":{"1min":0.48899999999999999,"5min":0.40899999999999997,"15min":0.32100000000000001},"max":{"1min":0.75600000000000001,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.52500000000000002}]},{"osd":4,"last update":"Thu Oct 19 13:03:33 2023","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.47099999999999997,"15min":0.46999999999999997},"min":{"1min":0.32100000000000001,"5min":0.307,"15min":0.29099999999999998},"max":{"1min":0.745,"5min":0.745,"15min":0.98099999999999998},"last":0.501},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.48599999999999999,"15min":0.47299999999999998},"min":{"1min":0.372,"5min":0.26100000000000001,"15min":0.251},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.79400000000000004},"last":0.48399999999999999}]},{"osd":6,"last update":"Thu Oct 19 13:03:33 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.48999999999999999,"15min":0.47799999999999998},"min":{"1min":0.34599999999999997,"5min":0.27800000000000002,"15min":0.27200000000000002},"max":{"1min":0.80200000000000005,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.51500000000000001,"15min":0.50700000000000001},"min":{"1min":0.40699999999999997,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.82499999999999996},"last":0.67900000000000005}]},{"osd":7,"last update":"Thu Oct 19 13:03:33 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.503,"15min":0.49299999999999999},"min":{"1min":0.35999999999999999,"5min":0.311,"15min":0.311},"max":{"1min":0.66800000000000004,"5min":0.78600000000000003,"15min":1.0589999999999999},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.55500000000000005,"15min":0.53800000000000003},"min":{"1min":0.34699999999999998,"5min":0.34599999999999997,"15min":0.311},"max":{"1min":0.78000000000000003,"5min":0.78700000000000003,"15min":0.94899999999999995},"last":0.74399999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575093,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32924,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338916,"statfs":{"total":96636764160,"available":96603049984,"internally_reserved":0,"allocated":1777664,"data_stored":899261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 13:03:47 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.52700000000000002,"15min":0.55300000000000005},"min":{"1min":0.37,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.78800000000000003,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.54800000000000004,"15min":0.55500000000000005},"min":{"1min":0.46300000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.68899999999999995,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.46500000000000002}]},{"osd":1,"last update":"Thu Oct 19 13:03:47 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56200000000000006,"15min":0.59699999999999998},"min":{"1min":0.37,"5min":0.32500000000000001,"15min":0.317},"max":{"1min":0.73999999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.58699999999999997,"15min":0.61099999999999999},"min":{"1min":0.441,"5min":0.38200000000000001,"15min":0.32600000000000001},"max":{"1min":0.76400000000000001,"5min":0.79800000000000004,"15min":0.86899999999999999},"last":0.59699999999999998}]},{"osd":2,"last update":"Thu Oct 19 13:03:47 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.57999999999999996,"15min":0.60599999999999998},"min":{"1min":0.371,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.77600000000000002,"5min":0.80100000000000005,"15min":0.86599999999999999},"last":0.498},{"interface":"front","average":{"1min":0.622,"5min":0.59299999999999997,"15min":0.60999999999999999},"min":{"1min":0.47999999999999998,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.78600000000000003,"5min":0.83799999999999997,"15min":0.85999999999999999},"last":0.48399999999999999}]},{"osd":3,"last update":"Thu Oct 19 13:03:47 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.55700000000000005,"15min":0.57599999999999996},"min":{"1min":0.40799999999999997,"5min":0.33200000000000002,"15min":0.32400000000000001},"max":{"1min":0.71999999999999997,"5min":0.75,"15min":0.79700000000000004},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.61199999999999999,"15min":0.63600000000000001},"min":{"1min":0.51600000000000001,"5min":0.42099999999999999,"15min":0.41299999999999998},"max":{"1min":0.80700000000000005,"5min":0.82599999999999996,"15min":0.83899999999999997},"last":0.55600000000000005}]},{"osd":5,"last update":"Thu Oct 19 13:03:47 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.44500000000000001,"15min":0.44600000000000001},"min":{"1min":0.34699999999999998,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.79600000000000004},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.46400000000000002,"15min":0.46100000000000002},"min":{"1min":0.314,"5min":0.30099999999999999,"15min":0.23499999999999999},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.36699999999999999}]},{"osd":6,"last update":"Thu Oct 19 13:03:47 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.45500000000000002,"15min":0.45700000000000002},"min":{"1min":0.34100000000000003,"5min":0.28499999999999998,"15min":0.26400000000000001},"max":{"1min":0.81299999999999994,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.497,"15min":0.49299999999999999},"min":{"1min":0.33300000000000002,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.53900000000000003}]},{"osd":7,"last update":"Thu Oct 19 13:03:47 2023","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.499,"15min":0.501},"min":{"1min":0.34999999999999998,"5min":0.31900000000000001,"15min":0.27500000000000002},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.80200000000000005},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.55000000000000004,"15min":0.54500000000000004},"min":{"1min":0.39000000000000001,"5min":0.33100000000000002,"15min":0.30199999999999999},"max":{"1min":0.85699999999999998,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.44600000000000001}]}]},{"osd":3,"up_from":13,"seq":55834575092,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41252,"kb_used_data":1752,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330588,"statfs":{"total":96636764160,"available":96594522112,"internally_reserved":0,"allocated":1794048,"data_stored":900907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":40434120},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 13:03:30 2023","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.40500000000000003,"15min":0.40300000000000002},"min":{"1min":0.25800000000000001,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.59399999999999997,"5min":0.69099999999999995,"15min":1.1279999999999999},"last":0.373},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.40200000000000002,"15min":0.41399999999999998},"min":{"1min":0.249,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.56299999999999994,"5min":0.75,"15min":0.75},"last":0.30099999999999999}]},{"osd":1,"last update":"Thu Oct 19 13:03:30 2023","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.41499999999999998,"15min":0.40999999999999998},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.626,"5min":0.63500000000000001,"15min":0.72099999999999997},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.438,"5min":0.438,"15min":0.45900000000000002},"min":{"1min":0.28499999999999998,"5min":0.221,"15min":0.221},"max":{"1min":0.59699999999999998,"5min":0.66700000000000004,"15min":2.9399999999999999},"last":0.45100000000000001}]},{"osd":2,"last update":"Thu Oct 19 13:03:30 2023","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.45700000000000002,"15min":0.46300000000000002},"min":{"1min":0.29699999999999999,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.97699999999999998},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.48799999999999999,"15min":0.48899999999999999},"min":{"1min":0.33200000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.42299999999999999}]},{"osd":4,"last update":"Thu Oct 19 13:03:30 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.64800000000000002,"15min":0.66400000000000003},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":1.0069999999999999},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.67300000000000004,"15min":0.68999999999999995},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.66100000000000003}]},{"osd":5,"last update":"Thu Oct 19 13:03:30 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.66400000000000003,"15min":0.66300000000000003},"min":{"1min":0.40699999999999997,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.93700000000000006,"5min":0.96699999999999997,"15min":2.9990000000000001},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.627,"15min":0.64900000000000002},"min":{"1min":0.46700000000000003,"5min":0.38600000000000001,"15min":0.35899999999999999},"max":{"1min":0.86599999999999999,"5min":0.873,"15min":1.774},"last":0.68400000000000005}]},{"osd":6,"last update":"Thu Oct 19 13:03:30 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.64500000000000002,"15min":0.65200000000000002},"min":{"1min":0.45300000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.67600000000000005,"15min":0.67900000000000005},"min":{"1min":0.42899999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.167},"last":0.622}]},{"osd":7,"last update":"Thu Oct 19 13:03:30 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.64600000000000002,"15min":0.66200000000000003},"min":{"1min":0.438,"5min":0.42099999999999999,"15min":0.39400000000000002},"max":{"1min":0.86799999999999999,"5min":0.94299999999999995,"15min":1.042},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.67100000000000004,"15min":0.68000000000000005},"min":{"1min":0.48499999999999999,"5min":0.441,"15min":0.432},"max":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":2.8149999999999999},"last":0.70099999999999996}]}]},{"osd":2,"up_from":13,"seq":55834575094,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55112,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316728,"statfs":{"total":96636764160,"available":96580329472,"internally_reserved":0,"allocated":1765376,"data_stored":892138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 13:03:48 2023","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.38500000000000001,"15min":0.379},"min":{"1min":0.253,"5min":0.20699999999999999,"15min":0.19700000000000001},"max":{"1min":0.59099999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.433,"5min":0.41199999999999998,"15min":0.40999999999999998},"min":{"1min":0.27700000000000002,"5min":0.222,"15min":0.20999999999999999},"max":{"1min":0.621,"5min":0.68000000000000005,"15min":0.74099999999999999},"last":0.371}]},{"osd":1,"last update":"Thu Oct 19 13:03:48 2023","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.39100000000000001,"15min":0.39600000000000002},"min":{"1min":0.249,"5min":0.249,"15min":0.24299999999999999},"max":{"1min":0.60799999999999998,"5min":0.64100000000000001,"15min":0.71099999999999997},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.42999999999999999,"15min":0.42699999999999999},"min":{"1min":0.30099999999999999,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.29299999999999998}]},{"osd":3,"last update":"Thu Oct 19 13:03:48 2023","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.45900000000000002,"15min":0.45600000000000002},"min":{"1min":0.318,"5min":0.29199999999999998,"15min":0.22800000000000001},"max":{"1min":0.66500000000000004,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.48099999999999998,"15min":0.48399999999999999},"min":{"1min":0.34899999999999998,"5min":0.32200000000000001,"15min":0.29099999999999998},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.82999999999999996},"last":0.39000000000000001}]},{"osd":4,"last update":"Thu Oct 19 13:03:48 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.625,"15min":0.64000000000000001},"min":{"1min":0.435,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.874,"5min":0.89500000000000002,"15min":0.98999999999999999},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.63700000000000001,"15min":0.65100000000000002},"min":{"1min":0.498,"5min":0.41199999999999998,"15min":0.35099999999999998},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":1.046},"last":0.77000000000000002}]},{"osd":5,"last update":"Thu Oct 19 13:03:48 2023","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.64200000000000002,"15min":0.65800000000000003},"min":{"1min":0.42399999999999999,"5min":0.40799999999999997,"15min":0.31900000000000001},"max":{"1min":0.94099999999999995,"5min":0.96199999999999997,"15min":0.99399999999999999},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.64000000000000001,"15min":0.65500000000000003},"min":{"1min":0.48399999999999999,"5min":0.39100000000000001,"15min":0.375},"max":{"1min":0.91500000000000004,"5min":0.94199999999999995,"15min":0.95799999999999996},"last":0.748}]},{"osd":6,"last update":"Thu Oct 19 13:03:48 2023","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.65500000000000003,"15min":0.66000000000000003},"min":{"1min":0.51300000000000001,"5min":0.40400000000000003,"15min":0.40300000000000002},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":1.008},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.66200000000000003,"15min":0.67600000000000005},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.371},"max":{"1min":0.90300000000000002,"5min":0.90400000000000003,"15min":0.995},"last":0.67600000000000005}]},{"osd":7,"last update":"Thu Oct 19 13:03:48 2023","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.66200000000000003,"15min":0.67500000000000004},"min":{"1min":0.47399999999999998,"5min":0.39800000000000002,"15min":0.36399999999999999},"max":{"1min":0.89500000000000002,"5min":0.96299999999999997,"15min":1.0149999999999999},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.67500000000000004,"15min":0.68300000000000005},"min":{"1min":0.46700000000000003,"5min":0.438,"15min":0.38200000000000001},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":1.0329999999999999},"last":0.58899999999999997}]}]},{"osd":1,"up_from":13,"seq":55834575093,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37064,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334776,"statfs":{"total":96636764160,"available":96598810624,"internally_reserved":0,"allocated":1765376,"data_stored":892138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 13:03:43 2023","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.371,"15min":0.38300000000000001},"min":{"1min":0.28699999999999998,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":1.3879999999999999},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.41599999999999998,"15min":0.434},"min":{"1min":0.22800000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.65000000000000002,"5min":0.76100000000000001,"15min":2.5800000000000001},"last":0.28899999999999998}]},{"osd":2,"last update":"Thu Oct 19 13:03:43 2023","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.40999999999999998,"15min":0.42199999999999999},"min":{"1min":0.27400000000000002,"5min":0.24199999999999999,"15min":0.21299999999999999},"max":{"1min":0.69399999999999995,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.437,"5min":0.44800000000000001,"15min":0.45700000000000002},"min":{"1min":0.27100000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.58499999999999996,"5min":0.79800000000000004,"15min":2.6230000000000002},"last":0.35499999999999998}]},{"osd":3,"last update":"Thu Oct 19 13:03:43 2023","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.46800000000000003,"15min":0.48899999999999999},"min":{"1min":0.33500000000000002,"5min":0.246,"15min":0.246},"max":{"1min":0.72899999999999998,"5min":1.008,"15min":3.9740000000000002},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.42299999999999999,"15min":0.42599999999999999},"min":{"1min":0.309,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.56299999999999994,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.377}]},{"osd":4,"last update":"Thu Oct 19 13:03:43 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.629,"15min":0.64800000000000002},"min":{"1min":0.40899999999999997,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.76300000000000001,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.63800000000000001},"min":{"1min":0.434,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.755,"5min":0.86899999999999999,"15min":1.0860000000000001},"last":0.60599999999999998}]},{"osd":5,"last update":"Thu Oct 19 13:03:43 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.63900000000000001,"15min":0.65000000000000002},"min":{"1min":0.439,"5min":0.38700000000000001,"15min":0.38100000000000001},"max":{"1min":0.79200000000000004,"5min":1.0549999999999999,"15min":1.1200000000000001},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.65500000000000003,"15min":0.67400000000000004},"min":{"1min":0.5,"5min":0.374,"15min":0.374},"max":{"1min":0.80600000000000005,"5min":1.1020000000000001,"15min":2.7120000000000002},"last":0.55400000000000005}]},{"osd":6,"last update":"Thu Oct 19 13:03:43 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.63200000000000001,"15min":0.64900000000000002},"min":{"1min":0.45000000000000001,"5min":0.41499999999999998,"15min":0.36799999999999999},"max":{"1min":0.79300000000000004,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.67000000000000004},"min":{"1min":0.46100000000000002,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.81799999999999995,"5min":1.129,"15min":1.129},"last":0.56899999999999995}]},{"osd":7,"last update":"Thu Oct 19 13:03:43 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.66500000000000004,"15min":0.67800000000000005},"min":{"1min":0.52500000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.81000000000000005,"5min":1.1439999999999999,"15min":2.786},"last":0.499},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64300000000000002,"15min":0.67000000000000004},"min":{"1min":0.48099999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.746,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.58799999999999997}]}]},{"osd":0,"up_from":13,"seq":55834575094,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33504,"kb_used_data":2316,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338336,"statfs":{"total":96636764160,"available":96602456064,"internally_reserved":0,"allocated":2371584,"data_stored":1489032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Oct 19 13:03:47 2023","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.38500000000000001,"15min":0.39000000000000001},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.182},"max":{"1min":0.57599999999999996,"5min":0.624,"15min":0.81799999999999995},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.40799999999999997,"15min":0.42699999999999999},"min":{"1min":0.318,"5min":0.22500000000000001,"15min":0.20200000000000001},"max":{"1min":0.56999999999999995,"5min":0.67600000000000005,"15min":0.746},"last":0.48999999999999999}]},{"osd":2,"last update":"Thu Oct 19 13:03:47 2023","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.42099999999999999,"15min":0.44900000000000001},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.26200000000000001},"max":{"1min":0.59899999999999998,"5min":0.64100000000000001,"15min":0.76300000000000001},"last":0.309},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.41599999999999998,"15min":0.432},"min":{"1min":0.30199999999999999,"5min":0.246,"15min":0.20699999999999999},"max":{"1min":0.56499999999999995,"5min":0.72999999999999998,"15min":0.76800000000000002},"last":0.39400000000000002}]},{"osd":3,"last update":"Thu Oct 19 13:03:47 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.38900000000000001,"15min":0.40600000000000003},"min":{"1min":0.218,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.48099999999999998,"5min":0.60399999999999998,"15min":1.2829999999999999},"last":0.376},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.45400000000000001,"15min":0.48599999999999999},"min":{"1min":0.316,"5min":0.29299999999999998,"15min":0.29099999999999998},"max":{"1min":0.63200000000000001,"5min":0.71199999999999997,"15min":1.4990000000000001},"last":0.45300000000000001}]},{"osd":4,"last update":"Thu Oct 19 13:03:47 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.59699999999999998,"15min":0.627},"min":{"1min":0.45800000000000002,"5min":0.40300000000000002,"15min":0.34999999999999998},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":1.361},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.63900000000000001,"15min":0.65100000000000002},"min":{"1min":0.44,"5min":0.36399999999999999,"15min":0.35999999999999999},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.98999999999999999},"last":0.53600000000000003}]},{"osd":5,"last update":"Thu Oct 19 13:03:47 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.66800000000000004,"15min":0.68200000000000005},"min":{"1min":0.46000000000000002,"5min":0.45100000000000001,"15min":0.378},"max":{"1min":0.92000000000000004,"5min":0.94599999999999995,"15min":1.325},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.66400000000000003,"15min":0.68300000000000005},"min":{"1min":0.44800000000000001,"5min":0.40100000000000002,"15min":0.318},"max":{"1min":0.94099999999999995,"5min":1.02,"15min":1.024},"last":0.56799999999999995}]},{"osd":6,"last update":"Thu Oct 19 13:03:47 2023","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.64200000000000002,"15min":0.65900000000000003},"min":{"1min":0.56699999999999995,"5min":0.36899999999999999,"15min":0.35899999999999999},"max":{"1min":0.879,"5min":0.879,"15min":0.98099999999999998},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.64100000000000001,"15min":0.65900000000000003},"min":{"1min":0.46200000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.86699999999999999,"5min":0.92000000000000004,"15min":0.995},"last":0.64800000000000002}]},{"osd":7,"last update":"Thu Oct 19 13:03:47 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.63300000000000001,"15min":0.67200000000000004},"min":{"1min":0.53200000000000003,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.83599999999999997,"5min":0.89900000000000002,"15min":1.0089999999999999},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.65000000000000002,"15min":0.65900000000000003},"min":{"1min":0.46899999999999997,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.86099999999999999,"5min":0.94499999999999995,"15min":1.389},"last":0.51500000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-19T13:04:26.790 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-19T13:04:26.791 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-19T13:04:26.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.938+0000 7efdc8e7c640 1 -- 172.21.15.57:0/2743629301 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efdb00039a0 msgr2=0x7efdb0003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:26.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.938+0000 7efdc8e7c640 1 --2- 172.21.15.57:0/2743629301 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efdb00039a0 0x7efdb0003dc0 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7efdb4006fd0 tx=0x7efdb00057f0 comp rx=0 tx=0).stop 2023-10-19T13:04:26.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.938+0000 7efdc2d9d640 1 -- 172.21.15.57:0/2743629301 reap_dead start 2023-10-19T13:04:26.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.938+0000 7efdc8e7c640 1 -- 172.21.15.57:0/2743629301 shutdown_connections 2023-10-19T13:04:26.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.938+0000 7efdc8e7c640 1 -- 172.21.15.57:0/2743629301 >> 172.21.15.57:0/2743629301 conn(0x7efdc40aec40 msgr2=0x7efdc40af050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:26.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.938+0000 7efdc8e7c640 1 -- 172.21.15.57:0/2743629301 shutdown_connections 2023-10-19T13:04:26.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.938+0000 7efdc8e7c640 1 -- 172.21.15.57:0/2743629301 wait complete. 2023-10-19T13:04:26.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.940+0000 7efdc8e7c640 1 Processor -- start 2023-10-19T13:04:26.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.940+0000 7efdc8e7c640 1 -- start start 2023-10-19T13:04:26.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.940+0000 7efdc8e7c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efdb00039a0 0x7efdc41f0e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:26.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.940+0000 7efdc8e7c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efdc4154fd0 0x7efdc4155410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:26.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.940+0000 7efdc8e7c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7efdc40ad8c0 0x7efdc40add40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:26.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.940+0000 7efdc8e7c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdc40ae280 con 0x7efdb00039a0 2023-10-19T13:04:26.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.940+0000 7efdc8e7c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdc4156730 con 0x7efdc4154fd0 2023-10-19T13:04:26.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.940+0000 7efdc8e7c640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdc4155b20 con 0x7efdc40ad8c0 2023-10-19T13:04:26.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.940+0000 7efdc259c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efdb00039a0 0x7efdc41f0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:26.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.940+0000 7efdc259c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efdb00039a0 0x7efdc41f0e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46102/0 (socket says 172.21.15.57:46102) 2023-10-19T13:04:26.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.941+0000 7efdc259c640 1 -- 172.21.15.57:0/4029009273 learned_addr learned my addr 172.21.15.57:0/4029009273 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:26.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.941+0000 7efdc2d9d640 1 --2- 172.21.15.57:0/4029009273 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7efdc40ad8c0 0x7efdc40add40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:26.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.941+0000 7efdb9d9b640 1 --2- 172.21.15.57:0/4029009273 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efdc4154fd0 0x7efdc4155410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:26.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.941+0000 7efdc259c640 1 -- 172.21.15.57:0/4029009273 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7efdc40ad8c0 msgr2=0x7efdc40add40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:26.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.941+0000 7efdc259c640 1 --2- 172.21.15.57:0/4029009273 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7efdc40ad8c0 0x7efdc40add40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:26.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.941+0000 7efdc259c640 1 -- 172.21.15.57:0/4029009273 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efdc4154fd0 msgr2=0x7efdc4155410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:26.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.941+0000 7efdc259c640 1 --2- 172.21.15.57:0/4029009273 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efdc4154fd0 0x7efdc4155410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:26.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.941+0000 7efdc259c640 1 -- 172.21.15.57:0/4029009273 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdc4155e00 con 0x7efdb00039a0 2023-10-19T13:04:26.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.941+0000 7efdc2d9d640 1 --2- 172.21.15.57:0/4029009273 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7efdc40ad8c0 0x7efdc40add40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:04:26.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.941+0000 7efdc259c640 1 --2- 172.21.15.57:0/4029009273 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efdb00039a0 0x7efdc41f0e50 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7efdb402c900 tx=0x7efdb40010b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:26.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.941+0000 7efdbb7fe640 1 -- 172.21.15.57:0/4029009273 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb40060d0 con 0x7efdb00039a0 2023-10-19T13:04:26.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.941+0000 7efdbb7fe640 1 -- 172.21.15.57:0/4029009273 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7efdb4006230 con 0x7efdb00039a0 2023-10-19T13:04:26.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.941+0000 7efdc8e7c640 1 -- 172.21.15.57:0/4029009273 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdc41560e0 con 0x7efdb00039a0 2023-10-19T13:04:26.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.941+0000 7efdbb7fe640 1 -- 172.21.15.57:0/4029009273 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb40035f0 con 0x7efdb00039a0 2023-10-19T13:04:26.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.941+0000 7efdc8e7c640 1 -- 172.21.15.57:0/4029009273 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdc41f1390 con 0x7efdb00039a0 2023-10-19T13:04:26.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.942+0000 7efdc8e7c640 1 -- 172.21.15.57:0/4029009273 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdc40690e0 con 0x7efdb00039a0 2023-10-19T13:04:26.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.943+0000 7efdbb7fe640 1 -- 172.21.15.57:0/4029009273 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 69) v1 ==== 89331+0+0 (secure 0 0 0) 0x7efdb4031030 con 0x7efdb00039a0 2023-10-19T13:04:26.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.943+0000 7efdbb7fe640 1 --2- 172.21.15.57:0/4029009273 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7efda406af90 0x7efda406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:26.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.943+0000 7efdbb7fe640 1 -- 172.21.15.57:0/4029009273 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6057+0+0 (secure 0 0 0) 0x7efdb40ab080 con 0x7efdb00039a0 2023-10-19T13:04:26.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.944+0000 7efdb9d9b640 1 --2- 172.21.15.57:0/4029009273 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7efda406af90 0x7efda406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:26.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.944+0000 7efdb9d9b640 1 --2- 172.21.15.57:0/4029009273 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7efda406af90 0x7efda406d450 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7efdc415b1b0 tx=0x7efdac006ac0 comp rx=0 tx=0).ready entity=mgr.7960 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:26.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:26.948+0000 7efdbb7fe640 1 -- 172.21.15.57:0/4029009273 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7efdb4075e40 con 0x7efdb00039a0 2023-10-19T13:04:27.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:27.195+0000 7efdc8e7c640 1 -- 172.21.15.57:0/4029009273 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7efdc40690e0 con 0x7efdb00039a0 2023-10-19T13:04:27.201 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:27.199+0000 7efdbb7fe640 1 -- 172.21.15.57:0/4029009273 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v29) v1 ==== 190+0+147 (secure 0 0 0) 0x7efdb4079460 con 0x7efdb00039a0 2023-10-19T13:04:27.202 INFO:teuthology.orchestra.run.smithi057.stdout:[client.0] 2023-10-19T13:04:27.202 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQBkJDFlg9yuExAATxiWDfS+p07G1F0pORbJEg== 2023-10-19T13:04:27.202 INFO:teuthology.orchestra.run.smithi057.stdout: caps mds = "allow" 2023-10-19T13:04:27.202 INFO:teuthology.orchestra.run.smithi057.stdout: caps mon = "allow r" 2023-10-19T13:04:27.203 INFO:teuthology.orchestra.run.smithi057.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-19T13:04:27.203 INFO:teuthology.orchestra.run.smithi057.stderr:updated caps for client.0 2023-10-19T13:04:27.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:27.201+0000 7efdc8e7c640 1 -- 172.21.15.57:0/4029009273 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7efda406af90 msgr2=0x7efda406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:27.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:27.201+0000 7efdc8e7c640 1 --2- 172.21.15.57:0/4029009273 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7efda406af90 0x7efda406d450 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7efdc415b1b0 tx=0x7efdac006ac0 comp rx=0 tx=0).stop 2023-10-19T13:04:27.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:27.201+0000 7efdc8e7c640 1 -- 172.21.15.57:0/4029009273 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efdb00039a0 msgr2=0x7efdc41f0e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:27.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:27.201+0000 7efdc8e7c640 1 --2- 172.21.15.57:0/4029009273 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efdb00039a0 0x7efdc41f0e50 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7efdb402c900 tx=0x7efdb40010b0 comp rx=0 tx=0).stop 2023-10-19T13:04:27.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:27.201+0000 7efdc8e7c640 1 -- 172.21.15.57:0/4029009273 shutdown_connections 2023-10-19T13:04:27.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:27.201+0000 7efdc8e7c640 1 --2- 172.21.15.57:0/4029009273 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7efda406af90 0x7efda406d450 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:27.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:27.201+0000 7efdc8e7c640 1 --2- 172.21.15.57:0/4029009273 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7efdc40ad8c0 0x7efdc40add40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:27.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:27.201+0000 7efdc8e7c640 1 --2- 172.21.15.57:0/4029009273 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efdc4154fd0 0x7efdc4155410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:27.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:27.201+0000 7efdc8e7c640 1 --2- 172.21.15.57:0/4029009273 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efdb00039a0 0x7efdc41f0e50 unknown :-1 s=CLOSED pgs=1602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:27.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:27.201+0000 7efdc8e7c640 1 -- 172.21.15.57:0/4029009273 >> 172.21.15.57:0/4029009273 conn(0x7efdc40aec40 msgr2=0x7efdc40cb960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:27.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:27.201+0000 7efdc8e7c640 1 -- 172.21.15.57:0/4029009273 shutdown_connections 2023-10-19T13:04:27.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:27.201+0000 7efdc8e7c640 1 -- 172.21.15.57:0/4029009273 wait complete. 2023-10-19T13:04:27.214 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T13:04:27.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:27.321+0000 7f6773b05640 1 -- 172.21.15.57:0/903353861 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f676c15c1f0 msgr2=0x7f676c06a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:27.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:27.321+0000 7f6773b05640 1 --2- 172.21.15.57:0/903353861 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f676c15c1f0 0x7f676c06a730 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f6760005fd0 tx=0x7f676002be10 comp rx=0 tx=0).stop 2023-10-19T13:04:27.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:27.321+0000 7f6773b05640 1 -- 172.21.15.57:0/903353861 shutdown_connections 2023-10-19T13:04:27.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:27.321+0000 7f6773b05640 1 --2- 172.21.15.57:0/903353861 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f676c15c1f0 0x7f676c06a730 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:27.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:27.321+0000 7f6773b05640 1 --2- 172.21.15.57:0/903353861 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f676c1579f0 0x7f676c0b0e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:27.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:27.321+0000 7f6773b05640 1 -- 172.21.15.57:0/903353861 >> 172.21.15.57:0/903353861 conn(0x7f676c06fae0 msgr2=0x7f676c06fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:27.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:27.321+0000 7f6773b05640 1 -- 172.21.15.57:0/903353861 shutdown_connections 2023-10-19T13:04:27.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:27.321+0000 7f6773b05640 1 -- 172.21.15.57:0/903353861 wait complete. 2023-10-19T13:04:27.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:27.322+0000 7f6773b05640 1 Processor -- start 2023-10-19T13:04:27.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:27.322+0000 7f6773b05640 1 -- start start 2023-10-19T13:04:27.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:27.323+0000 7f6773b05640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f676c06ac70 0x7f676c0b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:27.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:27.323+0000 7f6773b05640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f676c1579f0 0x7f676c156200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:27.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:27.323+0000 7f6773b05640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f676c15c1f0 0x7f676c156740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:27.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:27.323+0000 7f6773b05640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f676c154c50 con 0x7f676c06ac70 2023-10-19T13:04:27.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:27.323+0000 7f6773b05640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f676c154f30 con 0x7f676c1579f0 2023-10-19T13:04:27.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:27.323+0000 7f6773b05640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f676c155210 con 0x7f676c15c1f0 2023-10-19T13:04:27.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:27.323+0000 7f67718a1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f676c06ac70 0x7f676c0b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:27.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:27.323+0000 7f67718a1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f676c06ac70 0x7f676c0b04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46104/0 (socket says 172.21.15.57:46104) 2023-10-19T13:04:27.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:27.323+0000 7f67718a1640 1 -- 172.21.15.57:0/851658440 learned_addr learned my addr 172.21.15.57:0/851658440 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:27.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:27.324+0000 7f67710a0640 1 --2- 172.21.15.57:0/851658440 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f676c1579f0 0x7f676c156200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:27.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:27.324+0000 7f67720a2640 1 --2- 172.21.15.57:0/851658440 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f676c15c1f0 0x7f676c156740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:27.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:27.324+0000 7f67718a1640 1 -- 172.21.15.57:0/851658440 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f676c15c1f0 msgr2=0x7f676c156740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:27.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:27.324+0000 7f67718a1640 1 --2- 172.21.15.57:0/851658440 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f676c15c1f0 0x7f676c156740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:27.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:27.324+0000 7f67718a1640 1 -- 172.21.15.57:0/851658440 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f676c1579f0 msgr2=0x7f676c156200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:27.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:27.324+0000 7f67718a1640 1 --2- 172.21.15.57:0/851658440 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f676c1579f0 0x7f676c156200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:27.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:27.324+0000 7f67718a1640 1 -- 172.21.15.57:0/851658440 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6760009560 con 0x7f676c06ac70 2023-10-19T13:04:27.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:27.324+0000 7f67718a1640 1 --2- 172.21.15.57:0/851658440 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f676c06ac70 0x7f676c0b04c0 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7f676400e9f0 tx=0x7f676400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:27.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:27.324+0000 7f676a7fc640 1 -- 172.21.15.57:0/851658440 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6764009700 con 0x7f676c06ac70 2023-10-19T13:04:27.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:27.324+0000 7f676a7fc640 1 -- 172.21.15.57:0/851658440 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f676400ce70 con 0x7f676c06ac70 2023-10-19T13:04:27.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:27.324+0000 7f6773b05640 1 -- 172.21.15.57:0/851658440 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f676c1554f0 con 0x7f676c06ac70 2023-10-19T13:04:27.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:27.324+0000 7f676a7fc640 1 -- 172.21.15.57:0/851658440 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6764010670 con 0x7f676c06ac70 2023-10-19T13:04:27.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:27.324+0000 7f6773b05640 1 -- 172.21.15.57:0/851658440 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f676c0b0a00 con 0x7f676c06ac70 2023-10-19T13:04:27.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:27.325+0000 7f6773b05640 1 -- 172.21.15.57:0/851658440 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f676c0b0bf0 con 0x7f676c06ac70 2023-10-19T13:04:27.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:27.326+0000 7f676a7fc640 1 -- 172.21.15.57:0/851658440 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 69) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f67640107d0 con 0x7f676c06ac70 2023-10-19T13:04:27.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:27.326+0000 7f676a7fc640 1 --2- 172.21.15.57:0/851658440 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f675806af90 0x7f675806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:27.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:27.326+0000 7f676a7fc640 1 -- 172.21.15.57:0/851658440 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f6764014070 con 0x7f676c06ac70 2023-10-19T13:04:27.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:27.326+0000 7f67710a0640 1 --2- 172.21.15.57:0/851658440 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f675806af90 0x7f675806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:27.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:27.327+0000 7f67710a0640 1 --2- 172.21.15.57:0/851658440 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f675806af90 0x7f675806d450 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f6760035000 tx=0x7f6760004730 comp rx=0 tx=0).ready entity=mgr.7960 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:27.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:27.330+0000 7f676a7fc640 1 -- 172.21.15.57:0/851658440 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6764051ca0 con 0x7f676c06ac70 2023-10-19T13:04:27.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:27.546+0000 7f6773b05640 1 -- 172.21.15.57:0/851658440 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f676c0690e0 con 0x7f676c06ac70 2023-10-19T13:04:27.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:27.546+0000 7f676a7fc640 1 -- 172.21.15.57:0/851658440 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+4921 (secure 0 0 0) 0x7f6764058e40 con 0x7f676c06ac70 2023-10-19T13:04:27.549 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:04:27.550 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":56,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7325,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/3318110028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":3318110028},{"type":"v1","addr":"172.21.15.179:6837","nonce":3318110028}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":8143,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/3625556517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":3625556517},{"type":"v1","addr":"172.21.15.57:6835","nonce":3625556517}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":8158,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/4167860955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":4167860955},{"type":"v1","addr":"172.21.15.179:6836","nonce":4167860955}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T13:04:22.785432+0000","modified":"2023-10-19T13:04:24.778587+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9066},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9066":{"gid":9066,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":3,"addr":"172.21.15.57:6834/3474229632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":3474229632},{"type":"v1","addr":"172.21.15.57:6834","nonce":3474229632}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2023-10-19T13:04:27.550 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 56 2023-10-19T13:04:27.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:27.548+0000 7f6773b05640 1 -- 172.21.15.57:0/851658440 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f675806af90 msgr2=0x7f675806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:27.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:27.548+0000 7f6773b05640 1 --2- 172.21.15.57:0/851658440 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f675806af90 0x7f675806d450 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f6760035000 tx=0x7f6760004730 comp rx=0 tx=0).stop 2023-10-19T13:04:27.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:27.548+0000 7f6773b05640 1 -- 172.21.15.57:0/851658440 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f676c06ac70 msgr2=0x7f676c0b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:27.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:27.548+0000 7f6773b05640 1 --2- 172.21.15.57:0/851658440 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f676c06ac70 0x7f676c0b04c0 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7f676400e9f0 tx=0x7f676400eeb0 comp rx=0 tx=0).stop 2023-10-19T13:04:27.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:27.548+0000 7f6773b05640 1 -- 172.21.15.57:0/851658440 shutdown_connections 2023-10-19T13:04:27.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:27.548+0000 7f6773b05640 1 --2- 172.21.15.57:0/851658440 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f675806af90 0x7f675806d450 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:27.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:27.548+0000 7f6773b05640 1 --2- 172.21.15.57:0/851658440 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f676c15c1f0 0x7f676c156740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:27.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:27.548+0000 7f6773b05640 1 --2- 172.21.15.57:0/851658440 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f676c1579f0 0x7f676c156200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:27.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:27.548+0000 7f6773b05640 1 --2- 172.21.15.57:0/851658440 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f676c06ac70 0x7f676c0b04c0 unknown :-1 s=CLOSED pgs=1603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:27.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:27.548+0000 7f6773b05640 1 -- 172.21.15.57:0/851658440 >> 172.21.15.57:0/851658440 conn(0x7f676c06fae0 msgr2=0x7f676c14fb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:27.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:27.548+0000 7f6773b05640 1 -- 172.21.15.57:0/851658440 shutdown_connections 2023-10-19T13:04:27.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:27.548+0000 7f6773b05640 1 -- 172.21.15.57:0/851658440 wait complete. 2023-10-19T13:04:27.964 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 56, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-19T13:04:22.785432+0000', 'modified': '2023-10-19T13:04:24.778587+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9066}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9066': {'gid': 9066, 'name': 'c', 'rank': 0, 'incarnation': 54, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.57:6834/3474229632', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 3474229632}, {'type': 'v1', 'addr': '172.21.15.57:6834', 'nonce': 3474229632}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [18], 'metadata_pool': 17, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-19T13:04:27.964 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-19T13:04:27.965 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-10-19T13:04:28.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.076+0000 7f1248f05640 1 -- 172.21.15.57:0/16289168 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f12440cf340 msgr2=0x7f12440d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:28.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.076+0000 7f1248f05640 1 --2- 172.21.15.57:0/16289168 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f12440cf340 0x7f12440d1910 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7f124408a120 tx=0x7f123c02bcf0 comp rx=0 tx=0).stop 2023-10-19T13:04:28.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.076+0000 7f1248f05640 1 -- 172.21.15.57:0/16289168 shutdown_connections 2023-10-19T13:04:28.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.076+0000 7f1248f05640 1 --2- 172.21.15.57:0/16289168 >> v2:172.21.15.179:3300/0 conn(0x7f124415b2a0 0x7f12440cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:28.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.076+0000 7f1248f05640 1 --2- 172.21.15.57:0/16289168 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f12440cf340 0x7f12440d1910 unknown :-1 s=CLOSED pgs=1605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:28.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.076+0000 7f1248f05640 1 -- 172.21.15.57:0/16289168 >> 172.21.15.57:0/16289168 conn(0x7f12440d0d60 msgr2=0x7f12440d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:28.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.076+0000 7f1248f05640 1 -- 172.21.15.57:0/16289168 shutdown_connections 2023-10-19T13:04:28.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.076+0000 7f1248f05640 1 -- 172.21.15.57:0/16289168 wait complete. 2023-10-19T13:04:28.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.078+0000 7f1248f05640 1 Processor -- start 2023-10-19T13:04:28.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.078+0000 7f1248f05640 1 -- start start 2023-10-19T13:04:28.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.078+0000 7f1248f05640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f12440ceb40 0x7f12440b4310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:28.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.078+0000 7f1248f05640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f12440cf340 0x7f124415d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:28.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.078+0000 7f1248f05640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f124415b2a0 0x7f12441f0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:28.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.078+0000 7f1248f05640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12440b4850 con 0x7f124415b2a0 2023-10-19T13:04:28.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.078+0000 7f1248f05640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f124415dd80 con 0x7f12440ceb40 2023-10-19T13:04:28.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.078+0000 7f1248f05640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12440b4a40 con 0x7f12440cf340 2023-10-19T13:04:28.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.078+0000 7f1242d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f124415b2a0 0x7f12441f0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:28.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.079+0000 7f1242d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f124415b2a0 0x7f12441f0e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46118/0 (socket says 172.21.15.57:46118) 2023-10-19T13:04:28.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.079+0000 7f1242d9d640 1 -- 172.21.15.57:0/3127712258 learned_addr learned my addr 172.21.15.57:0/3127712258 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:28.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.079+0000 7f1241d9b640 1 --2- 172.21.15.57:0/3127712258 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f12440cf340 0x7f124415d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:28.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.079+0000 7f124259c640 1 --2- 172.21.15.57:0/3127712258 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f12440ceb40 0x7f12440b4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:28.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.079+0000 7f1242d9d640 1 -- 172.21.15.57:0/3127712258 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f12440cf340 msgr2=0x7f124415d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:28.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.079+0000 7f1242d9d640 1 --2- 172.21.15.57:0/3127712258 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f12440cf340 0x7f124415d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:28.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.079+0000 7f1242d9d640 1 -- 172.21.15.57:0/3127712258 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f12440ceb40 msgr2=0x7f12440b4310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:28.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.079+0000 7f1242d9d640 1 --2- 172.21.15.57:0/3127712258 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f12440ceb40 0x7f12440b4310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:28.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.079+0000 7f1242d9d640 1 -- 172.21.15.57:0/3127712258 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f123c009490 con 0x7f124415b2a0 2023-10-19T13:04:28.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.079+0000 7f1242d9d640 1 --2- 172.21.15.57:0/3127712258 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f124415b2a0 0x7f12441f0e60 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7f123400cc10 tx=0x7f123400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:28.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.079+0000 7f123affd640 1 -- 172.21.15.57:0/3127712258 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f123400ad40 con 0x7f124415b2a0 2023-10-19T13:04:28.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.079+0000 7f123affd640 1 -- 172.21.15.57:0/3127712258 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f1234012420 con 0x7f124415b2a0 2023-10-19T13:04:28.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.079+0000 7f1248f05640 1 -- 172.21.15.57:0/3127712258 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12440b4d20 con 0x7f124415b2a0 2023-10-19T13:04:28.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.079+0000 7f123affd640 1 -- 172.21.15.57:0/3127712258 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1234012830 con 0x7f124415b2a0 2023-10-19T13:04:28.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.079+0000 7f1248f05640 1 -- 172.21.15.57:0/3127712258 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12440b52e0 con 0x7f124415b2a0 2023-10-19T13:04:28.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.081+0000 7f123affd640 1 -- 172.21.15.57:0/3127712258 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 69) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f123401a3b0 con 0x7f124415b2a0 2023-10-19T13:04:28.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.081+0000 7f123affd640 1 --2- 172.21.15.57:0/3127712258 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f122406af60 0x7f122406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:28.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.081+0000 7f123affd640 1 -- 172.21.15.57:0/3127712258 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f123408a110 con 0x7f124415b2a0 2023-10-19T13:04:28.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.081+0000 7f124259c640 1 --2- 172.21.15.57:0/3127712258 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f122406af60 0x7f122406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:28.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.082+0000 7f124259c640 1 --2- 172.21.15.57:0/3127712258 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f122406af60 0x7f122406d420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f123c036000 tx=0x7f123c004730 comp rx=0 tx=0).ready entity=mgr.7960 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:28.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.082+0000 7f124bb72740 1 Processor -- start 2023-10-19T13:04:28.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.082+0000 7f124bb72740 1 -- start start 2023-10-19T13:04:28.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.083+0000 7f124bb72740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fd157bdb40 0x55fd15790ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:28.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.083+0000 7f124bb72740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fd157bdf20 0x55fd157c08d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:28.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.083+0000 7f124bb72740 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55fd15791020 0x55fd157c0e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:28.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.083+0000 7f124bb72740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fd156a2730 con 0x55fd157bdf20 2023-10-19T13:04:28.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.083+0000 7f124bb72740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fd15703780 con 0x55fd157bdb40 2023-10-19T13:04:28.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.083+0000 7f124bb72740 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x55fd15700e70 con 0x55fd15791020 2023-10-19T13:04:28.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.083+0000 7f1241d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fd157bdf20 0x55fd157c08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:28.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.083+0000 7f1241d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fd157bdf20 0x55fd157c08d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46120/0 (socket says 172.21.15.57:46120) 2023-10-19T13:04:28.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.083+0000 7f1241d9b640 1 -- 172.21.15.57:0/3368938130 learned_addr learned my addr 172.21.15.57:0/3368938130 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:28.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.083+0000 7f124259c640 1 --2- 172.21.15.57:0/3368938130 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fd157bdb40 0x55fd15790ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:28.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.083+0000 7f1242d9d640 1 --2- 172.21.15.57:0/3368938130 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55fd15791020 0x55fd157c0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:28.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.083+0000 7f1241d9b640 1 -- 172.21.15.57:0/3368938130 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55fd15791020 msgr2=0x55fd157c0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T13:04:28.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.083+0000 7f1241d9b640 1 --2- 172.21.15.57:0/3368938130 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55fd15791020 0x55fd157c0e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:28.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.083+0000 7f1241d9b640 1 -- 172.21.15.57:0/3368938130 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fd157bdb40 msgr2=0x55fd15790ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T13:04:28.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.083+0000 7f1241d9b640 1 --2- 172.21.15.57:0/3368938130 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fd157bdb40 0x55fd15790ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:28.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.083+0000 7f1241d9b640 1 -- 172.21.15.57:0/3368938130 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fd157013d0 con 0x55fd157bdf20 2023-10-19T13:04:28.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.083+0000 7f1241d9b640 1 --2- 172.21.15.57:0/3368938130 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fd157bdf20 0x55fd157c08d0 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7f122c00dbc0 tx=0x7f122c00f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a7b1ea7284b4bd server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:28.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.084+0000 7f1223fff640 1 -- 172.21.15.57:0/3368938130 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f122c00deb0 con 0x55fd157bdf20 2023-10-19T13:04:28.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.084+0000 7f1223fff640 1 -- 172.21.15.57:0/3368938130 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f122c004030 con 0x55fd157bdf20 2023-10-19T13:04:28.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.084+0000 7f1223fff640 1 -- 172.21.15.57:0/3368938130 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f122c004420 con 0x55fd157bdf20 2023-10-19T13:04:28.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.084+0000 7f124bb72740 1 -- 172.21.15.57:0/3368938130 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fd157bdf20 msgr2=0x55fd157c08d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:28.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.084+0000 7f124bb72740 1 --2- 172.21.15.57:0/3368938130 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fd157bdf20 0x55fd157c08d0 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7f122c00dbc0 tx=0x7f122c00f730 comp rx=0 tx=0).stop 2023-10-19T13:04:28.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.084+0000 7f124bb72740 1 -- 172.21.15.57:0/3368938130 shutdown_connections 2023-10-19T13:04:28.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.084+0000 7f124bb72740 1 --2- 172.21.15.57:0/3368938130 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55fd15791020 0x55fd157c0e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:28.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.084+0000 7f124bb72740 1 --2- 172.21.15.57:0/3368938130 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fd157bdf20 0x55fd157c08d0 unknown :-1 s=CLOSED pgs=1607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:28.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.084+0000 7f124bb72740 1 --2- 172.21.15.57:0/3368938130 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fd157bdb40 0x55fd15790ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:28.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.084+0000 7f124bb72740 1 -- 172.21.15.57:0/3368938130 >> 172.21.15.57:0/3368938130 conn(0x55fd15792d50 msgr2=0x55fd157db8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:28.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.084+0000 7f124bb72740 1 -- 172.21.15.57:0/3368938130 shutdown_connections 2023-10-19T13:04:28.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.084+0000 7f124bb72740 1 -- 172.21.15.57:0/3368938130 wait complete. 2023-10-19T13:04:28.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.085+0000 7f124bb72740 1 -- start start 2023-10-19T13:04:28.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.085+0000 7f124bb72740 1 Processor -- start 2023-10-19T13:04:28.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.086+0000 7f124bb72740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fd157bdb40 0x55fd157b1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:28.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.086+0000 7f124bb72740 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55fd157bdf20 0x55fd157c32d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:28.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.086+0000 7f124bb72740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fd157c3810 0x55fd157d5f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:28.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.086+0000 7f124bb72740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fd156fc760 con 0x55fd157bdb40 2023-10-19T13:04:28.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.086+0000 7f124bb72740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fd156b78f0 con 0x55fd157c3810 2023-10-19T13:04:28.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.086+0000 7f124bb72740 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x55fd156b7590 con 0x55fd157bdf20 2023-10-19T13:04:28.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.086+0000 7f124bb72740 10 client.0 _ll_register_callbacks cb 0x55fd156f3ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-19T13:04:28.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.086+0000 7f124259c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fd157bdb40 0x55fd157b1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:28.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.086+0000 7f124259c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fd157bdb40 0x55fd157b1fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46122/0 (socket says 172.21.15.57:46122) 2023-10-19T13:04:28.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.086+0000 7f124259c640 1 -- 172.21.15.57:0/1947696316 learned_addr learned my addr 172.21.15.57:0/1947696316 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:28.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.086+0000 7f1241d9b640 1 --2- 172.21.15.57:0/1947696316 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55fd157bdf20 0x55fd157c32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:28.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.086+0000 7f1242d9d640 1 --2- 172.21.15.57:0/1947696316 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fd157c3810 0x55fd157d5f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:28.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.086+0000 7f124259c640 1 -- 172.21.15.57:0/1947696316 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55fd157bdf20 msgr2=0x55fd157c32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T13:04:28.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.086+0000 7f124259c640 1 --2- 172.21.15.57:0/1947696316 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55fd157bdf20 0x55fd157c32d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:28.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.086+0000 7f124259c640 1 -- 172.21.15.57:0/1947696316 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fd157c3810 msgr2=0x55fd157d5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T13:04:28.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.086+0000 7f124259c640 1 --2- 172.21.15.57:0/1947696316 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fd157c3810 0x55fd157d5f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:28.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.086+0000 7f124259c640 1 -- 172.21.15.57:0/1947696316 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f122c00d820 con 0x55fd157bdb40 2023-10-19T13:04:28.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.086+0000 7f1241d9b640 1 --2- 172.21.15.57:0/1947696316 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55fd157bdf20 0x55fd157c32d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:04:28.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.086+0000 7f124259c640 1 --2- 172.21.15.57:0/1947696316 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fd157bdb40 0x55fd157b1fa0 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f123c009920 tx=0x7f123c005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51ecd8cc6168ab3f server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:28.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.086+0000 7f12217fa640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2023-10-19T13:04:28.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.087+0000 7f12217fa640 1 -- 172.21.15.57:0/1947696316 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f123c034650 con 0x55fd157bdb40 2023-10-19T13:04:28.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.087+0000 7f124bb72740 1 -- 172.21.15.57:0/1947696316 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55fd156f5740 con 0x55fd157bdb40 2023-10-19T13:04:28.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.087+0000 7f12217fa640 1 -- 172.21.15.57:0/1947696316 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(144..144 src has 1..144) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f123c046040 con 0x55fd157bdb40 2023-10-19T13:04:28.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.087+0000 7f12217fa640 1 -- 172.21.15.57:0/1947696316 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=56) v2 ==== 24+0+0 (secure 0 0 0) 0x7f123c005020 con 0x55fd157bdb40 2023-10-19T13:04:28.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.087+0000 7f124bb72740 10 client.9207 fetch_fsmap learned FSMap version 56 2023-10-19T13:04:28.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.087+0000 7f124bb72740 1 -- 172.21.15.57:0/1947696316 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=56,monmap=2+}) v3 -- 0x55fd156fade0 con 0x55fd157bdb40 2023-10-19T13:04:28.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.087+0000 7f12217fa640 1 -- 172.21.15.57:0/1947696316 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x55fd156f5740 con 0x55fd157bdb40 2023-10-19T13:04:28.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.087+0000 7f12217fa640 1 -- 172.21.15.57:0/1947696316 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 56) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f123c005270 con 0x55fd157bdb40 2023-10-19T13:04:28.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.087+0000 7f124bb72740 10 client.9207 fetch_fsmap finished waiting for FSMap version 56 2023-10-19T13:04:28.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.087+0000 7f124bb72740 10 client.9207 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-19T13:04:28.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.087+0000 7f124bb72740 20 client.9207 populate_metadata read hostname 'smithi057' 2023-10-19T13:04:28.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.087+0000 7f124bb72740 1 --2- 172.21.15.57:0/1947696316 >> [v2:172.21.15.57:6832/3474229632,v1:172.21.15.57:6834/3474229632] conn(0x55fd157f7120 0x55fd157f9510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:28.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.087+0000 7f124bb72740 4 client.9207 mds_command: new command op to 9066 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-19T13:04:28.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.087+0000 7f124bb72740 1 -- 172.21.15.57:0/1947696316 --> [v2:172.21.15.57:6832/3474229632,v1:172.21.15.57:6834/3474229632] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fd156f5460 con 0x55fd157f7120 2023-10-19T13:04:28.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.087+0000 7f1241d9b640 1 --2- 172.21.15.57:0/1947696316 >> [v2:172.21.15.57:6832/3474229632,v1:172.21.15.57:6834/3474229632] conn(0x55fd157f7120 0x55fd157f9510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:28.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.088+0000 7f12217fa640 10 client.9207 ms_handle_connect on v2:172.21.15.57:6832/3474229632 2023-10-19T13:04:28.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.088+0000 7f1241d9b640 1 --2- 172.21.15.57:0/1947696316 >> [v2:172.21.15.57:6832/3474229632,v1:172.21.15.57:6834/3474229632] conn(0x55fd157f7120 0x55fd157f9510 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7c559f24c131b77a server_cookie=cdf59dc838c05790 in_seq=0 out_seq=0 2023-10-19T13:04:28.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.088+0000 7f12217fa640 1 -- 172.21.15.57:0/1947696316 <== mds.0 v2:172.21.15.57:6832/3474229632 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55fd156f5460 con 0x55fd157f7120 2023-10-19T13:04:28.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.088+0000 7f12217fa640 10 client.9207 handle_command_reply: tid=0 2023-10-19T13:04:28.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.088+0000 7f124bb72740 1 client.9207 shutdown 2023-10-19T13:04:28.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.088+0000 7f124bb72740 1 -- 172.21.15.57:0/1947696316 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fd157bdb40 msgr2=0x55fd157b1fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:28.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.088+0000 7f124bb72740 1 --2- 172.21.15.57:0/1947696316 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fd157bdb40 0x55fd157b1fa0 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f123c009920 tx=0x7f123c005f90 comp rx=0 tx=0).stop 2023-10-19T13:04:28.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.088+0000 7f124bb72740 1 -- 172.21.15.57:0/1947696316 shutdown_connections 2023-10-19T13:04:28.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.088+0000 7f124bb72740 1 --2- 172.21.15.57:0/1947696316 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fd157c3810 0x55fd157d5f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:28.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.088+0000 7f124bb72740 1 --2- 172.21.15.57:0/1947696316 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55fd157bdf20 0x55fd157c32d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:28.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.088+0000 7f124bb72740 1 --2- 172.21.15.57:0/1947696316 >> [v2:172.21.15.57:6832/3474229632,v1:172.21.15.57:6834/3474229632] conn(0x55fd157f7120 0x55fd157f9510 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:28.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.088+0000 7f124bb72740 1 --2- 172.21.15.57:0/1947696316 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fd157bdb40 0x55fd157b1fa0 unknown :-1 s=CLOSED pgs=1608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:28.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.088+0000 7f124bb72740 1 -- 172.21.15.57:0/1947696316 >> 172.21.15.57:0/1947696316 conn(0x55fd15791020 msgr2=0x55fd157949c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:28.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.088+0000 7f12217fa640 0 client.9207 ms_handle_reset on v2:172.21.15.57:6832/3474229632 2023-10-19T13:04:28.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.089+0000 7f124bb72740 1 -- 172.21.15.57:0/1947696316 shutdown_connections 2023-10-19T13:04:28.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.089+0000 7f124bb72740 1 -- 172.21.15.57:0/1947696316 wait complete. 2023-10-19T13:04:28.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.089+0000 7f124bb72740 20 client.9207 trim_cache size 0 max 16384 2023-10-19T13:04:28.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.104+0000 7f124bb72740 1 Processor -- start 2023-10-19T13:04:28.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.104+0000 7f124bb72740 1 -- start start 2023-10-19T13:04:28.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.105+0000 7f124bb72740 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55fd157bdb40 0x55fd157bcdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:28.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.105+0000 7f124bb72740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fd157bdf20 0x55fd157defd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:28.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.105+0000 7f124bb72740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fd157c3810 0x55fd157c2b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:28.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.105+0000 7f124bb72740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fd1567d270 con 0x55fd157bdf20 2023-10-19T13:04:28.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.105+0000 7f124bb72740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fd15792540 con 0x55fd157c3810 2023-10-19T13:04:28.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.105+0000 7f124bb72740 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x55fd157aecd0 con 0x55fd157bdb40 2023-10-19T13:04:28.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.105+0000 7f1241d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fd157bdf20 0x55fd157defd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:28.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.105+0000 7f1241d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fd157bdf20 0x55fd157defd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46132/0 (socket says 172.21.15.57:46132) 2023-10-19T13:04:28.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.105+0000 7f1241d9b640 1 -- 172.21.15.57:0/166646248 learned_addr learned my addr 172.21.15.57:0/166646248 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:28.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.105+0000 7f124259c640 1 --2- 172.21.15.57:0/166646248 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55fd157bdb40 0x55fd157bcdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:28.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.105+0000 7f1242d9d640 1 --2- 172.21.15.57:0/166646248 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fd157c3810 0x55fd157c2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:28.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.105+0000 7f1241d9b640 1 -- 172.21.15.57:0/166646248 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55fd157bdb40 msgr2=0x55fd157bcdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T13:04:28.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.105+0000 7f1241d9b640 1 --2- 172.21.15.57:0/166646248 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55fd157bdb40 0x55fd157bcdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:28.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.105+0000 7f1241d9b640 1 -- 172.21.15.57:0/166646248 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fd157c3810 msgr2=0x55fd157c2b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T13:04:28.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.105+0000 7f1241d9b640 1 --2- 172.21.15.57:0/166646248 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fd157c3810 0x55fd157c2b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:28.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.105+0000 7f1241d9b640 1 -- 172.21.15.57:0/166646248 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fd15699d00 con 0x55fd157bdf20 2023-10-19T13:04:28.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.105+0000 7f124259c640 1 --2- 172.21.15.57:0/166646248 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55fd157bdb40 0x55fd157bcdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T13:04:28.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.105+0000 7f1242d9d640 1 --2- 172.21.15.57:0/166646248 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fd157c3810 0x55fd157c2b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T13:04:28.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.105+0000 7f1241d9b640 1 --2- 172.21.15.57:0/166646248 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fd157bdf20 0x55fd157defd0 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7f122c00d7f0 tx=0x7f122c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74925d311800c2fb server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:28.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.105+0000 7f1238ff9640 1 -- 172.21.15.57:0/166646248 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f122c0044a0 con 0x55fd157bdf20 2023-10-19T13:04:28.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.106+0000 7f1238ff9640 1 -- 172.21.15.57:0/166646248 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f122c021040 con 0x55fd157bdf20 2023-10-19T13:04:28.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.106+0000 7f1238ff9640 1 -- 172.21.15.57:0/166646248 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f122c028c10 con 0x55fd157bdf20 2023-10-19T13:04:28.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.106+0000 7f124bb72740 1 -- 172.21.15.57:0/166646248 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fd157bdf20 msgr2=0x55fd157defd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:28.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.106+0000 7f124bb72740 1 --2- 172.21.15.57:0/166646248 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fd157bdf20 0x55fd157defd0 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7f122c00d7f0 tx=0x7f122c004730 comp rx=0 tx=0).stop 2023-10-19T13:04:28.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.106+0000 7f124bb72740 1 -- 172.21.15.57:0/166646248 shutdown_connections 2023-10-19T13:04:28.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.106+0000 7f124bb72740 1 --2- 172.21.15.57:0/166646248 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fd157c3810 0x55fd157c2b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:28.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.106+0000 7f124bb72740 1 --2- 172.21.15.57:0/166646248 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fd157bdf20 0x55fd157defd0 unknown :-1 s=CLOSED pgs=1609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:28.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.106+0000 7f124bb72740 1 --2- 172.21.15.57:0/166646248 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55fd157bdb40 0x55fd157bcdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:28.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.106+0000 7f124bb72740 1 -- 172.21.15.57:0/166646248 >> 172.21.15.57:0/166646248 conn(0x55fd15791020 msgr2=0x55fd1578c440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:28.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.106+0000 7f124bb72740 1 -- 172.21.15.57:0/166646248 shutdown_connections 2023-10-19T13:04:28.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.106+0000 7f124bb72740 1 -- 172.21.15.57:0/166646248 wait complete. 2023-10-19T13:04:28.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.107+0000 7f124bb72740 1 -- start start 2023-10-19T13:04:28.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.107+0000 7f124bb72740 1 Processor -- start 2023-10-19T13:04:28.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.108+0000 7f124bb72740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fd157bdb40 0x55fd158437b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:28.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.108+0000 7f124bb72740 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55fd157bdf20 0x55fd157bcdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:28.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.108+0000 7f124bb72740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fd157c3810 0x55fd157bd310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:28.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.108+0000 7f124bb72740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fd157d9580 con 0x55fd157c3810 2023-10-19T13:04:28.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.108+0000 7f124bb72740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fd157d24d0 con 0x55fd157bdb40 2023-10-19T13:04:28.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.108+0000 7f124bb72740 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x55fd157defd0 con 0x55fd157bdf20 2023-10-19T13:04:28.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.108+0000 7f124bb72740 10 client.0 _ll_register_callbacks cb 0x7f122c00d360 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-19T13:04:28.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.108+0000 7f1242d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fd157c3810 0x55fd157bd310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:28.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.108+0000 7f1242d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fd157c3810 0x55fd157bd310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46140/0 (socket says 172.21.15.57:46140) 2023-10-19T13:04:28.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.108+0000 7f1242d9d640 1 -- 172.21.15.57:0/3723091246 learned_addr learned my addr 172.21.15.57:0/3723091246 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:28.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.108+0000 7f124259c640 1 --2- 172.21.15.57:0/3723091246 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fd157bdb40 0x55fd158437b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:28.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.108+0000 7f1241d9b640 1 --2- 172.21.15.57:0/3723091246 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55fd157bdf20 0x55fd157bcdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:28.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.108+0000 7f1242d9d640 1 -- 172.21.15.57:0/3723091246 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55fd157bdf20 msgr2=0x55fd157bcdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T13:04:28.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.108+0000 7f1242d9d640 1 --2- 172.21.15.57:0/3723091246 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55fd157bdf20 0x55fd157bcdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:28.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.108+0000 7f1242d9d640 1 -- 172.21.15.57:0/3723091246 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fd157bdb40 msgr2=0x55fd158437b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T13:04:28.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.108+0000 7f1242d9d640 1 --2- 172.21.15.57:0/3723091246 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fd157bdb40 0x55fd158437b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:28.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.108+0000 7f1242d9d640 1 -- 172.21.15.57:0/3723091246 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fd157013d0 con 0x55fd157c3810 2023-10-19T13:04:28.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.108+0000 7f124259c640 1 --2- 172.21.15.57:0/3723091246 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fd157bdb40 0x55fd158437b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T13:04:28.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.108+0000 7f1242d9d640 1 --2- 172.21.15.57:0/3723091246 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fd157c3810 0x55fd157bd310 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7f1234055120 tx=0x7f1234058d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76077d097327c28f server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:28.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.108+0000 7f12217fa640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2023-10-19T13:04:28.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.108+0000 7f12217fa640 1 -- 172.21.15.57:0/3723091246 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1234074070 con 0x55fd157c3810 2023-10-19T13:04:28.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.108+0000 7f12217fa640 1 -- 172.21.15.57:0/3723091246 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(144..144 src has 1..144) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f1234021e20 con 0x55fd157c3810 2023-10-19T13:04:28.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.108+0000 7f124bb72740 1 -- 172.21.15.57:0/3723091246 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f122c004030 con 0x55fd157c3810 2023-10-19T13:04:28.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.108+0000 7f12217fa640 1 -- 172.21.15.57:0/3723091246 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=56) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1234055330 con 0x55fd157c3810 2023-10-19T13:04:28.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.108+0000 7f124bb72740 10 client.9213 fetch_fsmap learned FSMap version 56 2023-10-19T13:04:28.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.109+0000 7f124bb72740 1 -- 172.21.15.57:0/3723091246 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=56,monmap=2+}) v3 -- 0x55fd156bf4f0 con 0x55fd157c3810 2023-10-19T13:04:28.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.109+0000 7f12217fa640 1 -- 172.21.15.57:0/3723091246 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f122c004030 con 0x55fd157c3810 2023-10-19T13:04:28.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.109+0000 7f12217fa640 1 -- 172.21.15.57:0/3723091246 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 56) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f1234022020 con 0x55fd157c3810 2023-10-19T13:04:28.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.109+0000 7f124bb72740 10 client.9213 fetch_fsmap finished waiting for FSMap version 56 2023-10-19T13:04:28.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.109+0000 7f124bb72740 10 client.9213 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-19T13:04:28.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.109+0000 7f124bb72740 20 client.9213 populate_metadata read hostname 'smithi057' 2023-10-19T13:04:28.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.109+0000 7f124bb72740 1 --2- 172.21.15.57:0/3723091246 >> [v2:172.21.15.57:6832/3474229632,v1:172.21.15.57:6834/3474229632] conn(0x55fd157f54c0 0x55fd157f58a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:28.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.109+0000 7f124bb72740 4 client.9213 mds_command: new command op to 9066 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-19T13:04:28.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.109+0000 7f124bb72740 1 -- 172.21.15.57:0/3723091246 --> [v2:172.21.15.57:6832/3474229632,v1:172.21.15.57:6834/3474229632] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55fd153fe6c0 con 0x55fd157f54c0 2023-10-19T13:04:28.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.109+0000 7f124259c640 1 --2- 172.21.15.57:0/3723091246 >> [v2:172.21.15.57:6832/3474229632,v1:172.21.15.57:6834/3474229632] conn(0x55fd157f54c0 0x55fd157f58a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:28.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.109+0000 7f12217fa640 10 client.9213 ms_handle_connect on v2:172.21.15.57:6832/3474229632 2023-10-19T13:04:28.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.109+0000 7f124259c640 1 --2- 172.21.15.57:0/3723091246 >> [v2:172.21.15.57:6832/3474229632,v1:172.21.15.57:6834/3474229632] conn(0x55fd157f54c0 0x55fd157f58a0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2e6ddba8a3296b21 server_cookie=a4634061cea5367e in_seq=0 out_seq=0 2023-10-19T13:04:28.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.109+0000 7f12217fa640 1 -- 172.21.15.57:0/3723091246 <== mds.0 v2:172.21.15.57:6832/3474229632 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x55fd153fe6c0 con 0x55fd157f54c0 2023-10-19T13:04:28.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.109+0000 7f12217fa640 10 client.9213 handle_command_reply: tid=0 2023-10-19T13:04:28.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.109+0000 7f124bb72740 1 client.9213 shutdown 2023-10-19T13:04:28.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.110+0000 7f124bb72740 1 -- 172.21.15.57:0/3723091246 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fd157c3810 msgr2=0x55fd157bd310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:28.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.110+0000 7f124bb72740 1 --2- 172.21.15.57:0/3723091246 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fd157c3810 0x55fd157bd310 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7f1234055120 tx=0x7f1234058d50 comp rx=0 tx=0).stop 2023-10-19T13:04:28.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.110+0000 7f124bb72740 1 -- 172.21.15.57:0/3723091246 shutdown_connections 2023-10-19T13:04:28.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.110+0000 7f124bb72740 1 --2- 172.21.15.57:0/3723091246 >> [v2:172.21.15.57:6832/3474229632,v1:172.21.15.57:6834/3474229632] conn(0x55fd157f54c0 0x55fd157f58a0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:28.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.110+0000 7f124bb72740 1 --2- 172.21.15.57:0/3723091246 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fd157c3810 0x55fd157bd310 unknown :-1 s=CLOSED pgs=1610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:28.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.110+0000 7f124bb72740 1 --2- 172.21.15.57:0/3723091246 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55fd157bdf20 0x55fd157bcdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:28.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.110+0000 7f12217fa640 0 client.9213 ms_handle_reset on v2:172.21.15.57:6832/3474229632 2023-10-19T13:04:28.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.110+0000 7f124bb72740 1 --2- 172.21.15.57:0/3723091246 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fd157bdb40 0x55fd158437b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:28.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.110+0000 7f124bb72740 1 -- 172.21.15.57:0/3723091246 >> 172.21.15.57:0/3723091246 conn(0x55fd15791020 msgr2=0x55fd157b1db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:28.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.110+0000 7f124bb72740 1 -- 172.21.15.57:0/3723091246 shutdown_connections 2023-10-19T13:04:28.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.110+0000 7f124bb72740 1 -- 172.21.15.57:0/3723091246 wait complete. 2023-10-19T13:04:28.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.110+0000 7f124bb72740 20 client.9213 trim_cache size 0 max 16384 2023-10-19T13:04:28.132 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2023-10-19T13:04:28.132 INFO:teuthology.orchestra.run.smithi057.stdout: "cluster_fsid": "1e9beb19-3260-4dd7-85ef-5b399ce774ac", 2023-10-19T13:04:28.132 INFO:teuthology.orchestra.run.smithi057.stdout: "whoami": 0, 2023-10-19T13:04:28.132 INFO:teuthology.orchestra.run.smithi057.stdout: "id": 9066, 2023-10-19T13:04:28.132 INFO:teuthology.orchestra.run.smithi057.stdout: "want_state": "up:active", 2023-10-19T13:04:28.133 INFO:teuthology.orchestra.run.smithi057.stdout: "state": "up:active", 2023-10-19T13:04:28.133 INFO:teuthology.orchestra.run.smithi057.stdout: "fs_name": "cephfs", 2023-10-19T13:04:28.133 INFO:teuthology.orchestra.run.smithi057.stdout: "rank_uptime": 4.6900241869999997, 2023-10-19T13:04:28.133 INFO:teuthology.orchestra.run.smithi057.stdout: "mdsmap_epoch": 56, 2023-10-19T13:04:28.133 INFO:teuthology.orchestra.run.smithi057.stdout: "osdmap_epoch": 144, 2023-10-19T13:04:28.133 INFO:teuthology.orchestra.run.smithi057.stdout: "osdmap_epoch_barrier": 144, 2023-10-19T13:04:28.134 INFO:teuthology.orchestra.run.smithi057.stdout: "uptime": 10.404219998 2023-10-19T13:04:28.134 INFO:teuthology.orchestra.run.smithi057.stdout:} 2023-10-19T13:04:28.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.110+0000 7f1248f05640 1 -- 172.21.15.57:0/3127712258 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f122406af60 msgr2=0x7f122406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:28.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.110+0000 7f1248f05640 1 --2- 172.21.15.57:0/3127712258 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f122406af60 0x7f122406d420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f123c036000 tx=0x7f123c004730 comp rx=0 tx=0).stop 2023-10-19T13:04:28.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.110+0000 7f1248f05640 1 -- 172.21.15.57:0/3127712258 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f124415b2a0 msgr2=0x7f12441f0e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:28.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.110+0000 7f1248f05640 1 --2- 172.21.15.57:0/3127712258 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f124415b2a0 0x7f12441f0e60 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7f123400cc10 tx=0x7f123400a430 comp rx=0 tx=0).stop 2023-10-19T13:04:28.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.110+0000 7f1248f05640 1 -- 172.21.15.57:0/3127712258 shutdown_connections 2023-10-19T13:04:28.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.110+0000 7f1248f05640 1 --2- 172.21.15.57:0/3127712258 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f122406af60 0x7f122406d420 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:28.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.110+0000 7f1248f05640 1 --2- 172.21.15.57:0/3127712258 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f124415b2a0 0x7f12441f0e60 unknown :-1 s=CLOSED pgs=1606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:28.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.110+0000 7f1248f05640 1 --2- 172.21.15.57:0/3127712258 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f12440cf340 0x7f124415d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:28.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.110+0000 7f1248f05640 1 --2- 172.21.15.57:0/3127712258 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f12440ceb40 0x7f12440b4310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:28.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.110+0000 7f1248f05640 1 -- 172.21.15.57:0/3127712258 >> 172.21.15.57:0/3127712258 conn(0x7f12440d0d60 msgr2=0x7f124406a470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:28.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.110+0000 7f1248f05640 1 -- 172.21.15.57:0/3127712258 shutdown_connections 2023-10-19T13:04:28.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.110+0000 7f1248f05640 1 -- 172.21.15.57:0/3127712258 wait complete. 2023-10-19T13:04:28.138 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-19T13:04:28.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.244+0000 7f69a4b4c640 1 -- 172.21.15.57:0/3648275169 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f69a00cf340 msgr2=0x7f69a00d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:28.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.244+0000 7f69a4b4c640 1 --2- 172.21.15.57:0/3648275169 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f69a00cf340 0x7f69a00d1910 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f6990005fd0 tx=0x7f699002bd90 comp rx=0 tx=0).stop 2023-10-19T13:04:28.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.245+0000 7f69a4b4c640 1 -- 172.21.15.57:0/3648275169 shutdown_connections 2023-10-19T13:04:28.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.245+0000 7f69a4b4c640 1 --2- 172.21.15.57:0/3648275169 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f69a00ceb40 0x7f69a00b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:28.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.245+0000 7f69a4b4c640 1 --2- 172.21.15.57:0/3648275169 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f69a00cf340 0x7f69a00d1910 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:28.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.245+0000 7f69a4b4c640 1 -- 172.21.15.57:0/3648275169 >> 172.21.15.57:0/3648275169 conn(0x7f69a00d0d60 msgr2=0x7f69a00d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:28.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.245+0000 7f69a4b4c640 1 -- 172.21.15.57:0/3648275169 shutdown_connections 2023-10-19T13:04:28.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.245+0000 7f69a4b4c640 1 -- 172.21.15.57:0/3648275169 wait complete. 2023-10-19T13:04:28.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.246+0000 7f69a4b4c640 1 Processor -- start 2023-10-19T13:04:28.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.246+0000 7f69a4b4c640 1 -- start start 2023-10-19T13:04:28.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.247+0000 7f69a4b4c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f69a00ceb40 0x7f69a009e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:28.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.247+0000 7f69a4b4c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f69a00cf340 0x7f69a01573f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:28.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.247+0000 7f69a4b4c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f69a015b2a0 0x7f69a01f1f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:28.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.247+0000 7f69a4b4c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a015cb70 con 0x7f69a00ceb40 2023-10-19T13:04:28.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.247+0000 7f69a4b4c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a009e9b0 con 0x7f69a015b2a0 2023-10-19T13:04:28.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.247+0000 7f69a4b4c640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69a0157930 con 0x7f69a00cf340 2023-10-19T13:04:28.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.247+0000 7f699e59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f69a00ceb40 0x7f69a009e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:28.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.247+0000 7f699e59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f69a00ceb40 0x7f69a009e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46142/0 (socket says 172.21.15.57:46142) 2023-10-19T13:04:28.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.247+0000 7f699e59c640 1 -- 172.21.15.57:0/513564377 learned_addr learned my addr 172.21.15.57:0/513564377 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:28.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.247+0000 7f699ed9d640 1 --2- 172.21.15.57:0/513564377 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f69a015b2a0 0x7f69a01f1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:28.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.247+0000 7f6997fff640 1 --2- 172.21.15.57:0/513564377 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f69a00cf340 0x7f69a01573f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:28.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.247+0000 7f699e59c640 1 -- 172.21.15.57:0/513564377 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f69a00cf340 msgr2=0x7f69a01573f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:28.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.247+0000 7f699e59c640 1 --2- 172.21.15.57:0/513564377 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f69a00cf340 0x7f69a01573f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:28.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.247+0000 7f699e59c640 1 -- 172.21.15.57:0/513564377 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f69a015b2a0 msgr2=0x7f69a01f1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:28.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.247+0000 7f699e59c640 1 --2- 172.21.15.57:0/513564377 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f69a015b2a0 0x7f69a01f1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:28.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.247+0000 7f699e59c640 1 -- 172.21.15.57:0/513564377 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6990009560 con 0x7f69a00ceb40 2023-10-19T13:04:28.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.247+0000 7f699e59c640 1 --2- 172.21.15.57:0/513564377 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f69a00ceb40 0x7f69a009e470 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7f6990009850 tx=0x7f699002d9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:28.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.248+0000 7f6996ffd640 1 -- 172.21.15.57:0/513564377 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f699002ddd0 con 0x7f69a00ceb40 2023-10-19T13:04:28.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.248+0000 7f6996ffd640 1 -- 172.21.15.57:0/513564377 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f69900036e0 con 0x7f69a00ceb40 2023-10-19T13:04:28.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.248+0000 7f69a4b4c640 1 -- 172.21.15.57:0/513564377 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69a00b0d50 con 0x7f69a00ceb40 2023-10-19T13:04:28.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.248+0000 7f6996ffd640 1 -- 172.21.15.57:0/513564377 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6990003a90 con 0x7f69a00ceb40 2023-10-19T13:04:28.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.248+0000 7f69a4b4c640 1 -- 172.21.15.57:0/513564377 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69a00c86d0 con 0x7f69a00ceb40 2023-10-19T13:04:28.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.249+0000 7f69a4b4c640 1 -- 172.21.15.57:0/513564377 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69a00d1ac0 con 0x7f69a00ceb40 2023-10-19T13:04:28.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.249+0000 7f6996ffd640 1 -- 172.21.15.57:0/513564377 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 69) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6990003bf0 con 0x7f69a00ceb40 2023-10-19T13:04:28.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.249+0000 7f6996ffd640 1 --2- 172.21.15.57:0/513564377 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f698006af60 0x7f698006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:28.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.250+0000 7f6996ffd640 1 -- 172.21.15.57:0/513564377 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f69900ac2d0 con 0x7f69a00ceb40 2023-10-19T13:04:28.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.250+0000 7f6997fff640 1 --2- 172.21.15.57:0/513564377 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f698006af60 0x7f698006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:28.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.250+0000 7f6997fff640 1 --2- 172.21.15.57:0/513564377 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f698006af60 0x7f698006d420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f69a0159820 tx=0x7f6988009040 comp rx=0 tx=0).ready entity=mgr.7960 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:28.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.254+0000 7f6996ffd640 1 -- 172.21.15.57:0/513564377 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6990077140 con 0x7f69a00ceb40 2023-10-19T13:04:28.393 INFO:tasks.ceph.mds.c.smithi057.stderr:2023-10-19T13:04:28.391+0000 7f02b96dc640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-19T13:04:28.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.467+0000 7f69a4b4c640 1 -- 172.21.15.57:0/513564377 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f69a00690e0 con 0x7f69a00ceb40 2023-10-19T13:04:28.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.467+0000 7f6996ffd640 1 -- 172.21.15.57:0/513564377 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v56) v1 ==== 53+0+73 (secure 0 0 0) 0x7f699007a760 con 0x7f69a00ceb40 2023-10-19T13:04:28.470 INFO:teuthology.orchestra.run.smithi057.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-19T13:04:28.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.469+0000 7f69a4b4c640 1 -- 172.21.15.57:0/513564377 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f698006af60 msgr2=0x7f698006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:28.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.469+0000 7f69a4b4c640 1 --2- 172.21.15.57:0/513564377 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f698006af60 0x7f698006d420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f69a0159820 tx=0x7f6988009040 comp rx=0 tx=0).stop 2023-10-19T13:04:28.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.469+0000 7f69a4b4c640 1 -- 172.21.15.57:0/513564377 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f69a00ceb40 msgr2=0x7f69a009e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:28.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.469+0000 7f69a4b4c640 1 --2- 172.21.15.57:0/513564377 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f69a00ceb40 0x7f69a009e470 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7f6990009850 tx=0x7f699002d9d0 comp rx=0 tx=0).stop 2023-10-19T13:04:28.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.469+0000 7f69a4b4c640 1 -- 172.21.15.57:0/513564377 shutdown_connections 2023-10-19T13:04:28.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.469+0000 7f69a4b4c640 1 --2- 172.21.15.57:0/513564377 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f698006af60 0x7f698006d420 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:28.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.469+0000 7f69a4b4c640 1 --2- 172.21.15.57:0/513564377 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f69a015b2a0 0x7f69a01f1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:28.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.469+0000 7f69a4b4c640 1 --2- 172.21.15.57:0/513564377 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f69a00cf340 0x7f69a01573f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:28.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.469+0000 7f69a4b4c640 1 --2- 172.21.15.57:0/513564377 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f69a00ceb40 0x7f69a009e470 unknown :-1 s=CLOSED pgs=1611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:28.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.469+0000 7f69a4b4c640 1 -- 172.21.15.57:0/513564377 >> 172.21.15.57:0/513564377 conn(0x7f69a00d0d60 msgr2=0x7f69a00ad8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:28.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.469+0000 7f69a4b4c640 1 -- 172.21.15.57:0/513564377 shutdown_connections 2023-10-19T13:04:28.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:28.469+0000 7f69a4b4c640 1 -- 172.21.15.57:0/513564377 wait complete. 2023-10-19T13:04:28.482 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-10-19T13:04:28.483 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-19T13:04:28.483 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi057.front.sepia.ceph.com 2023-10-19T13:04:28.483 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-19T13:04:28.483 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-19T13:04:28.484 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-10-19T13:04:28.484 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-10-19T13:04:28.484 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-19T13:04:28.485 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-19T13:04:28.485 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:04:28.486 DEBUG:teuthology.orchestra.run.smithi057:> ip addr 2023-10-19T13:04:28.501 INFO:teuthology.orchestra.run.smithi057.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-19T13:04:28.501 INFO:teuthology.orchestra.run.smithi057.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-19T13:04:28.502 INFO:teuthology.orchestra.run.smithi057.stdout: inet 127.0.0.1/8 scope host lo 2023-10-19T13:04:28.502 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T13:04:28.502 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 ::1/128 scope host 2023-10-19T13:04:28.502 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T13:04:28.502 INFO:teuthology.orchestra.run.smithi057.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-19T13:04:28.503 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:6b:ff:86 brd ff:ff:ff:ff:ff:ff 2023-10-19T13:04:28.503 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp7s0f0 2023-10-19T13:04:28.503 INFO:teuthology.orchestra.run.smithi057.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-19T13:04:28.503 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:6b:ff:87 brd ff:ff:ff:ff:ff:ff 2023-10-19T13:04:28.503 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp7s0f1 2023-10-19T13:04:28.504 INFO:teuthology.orchestra.run.smithi057.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-19T13:04:28.504 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:bd:15:62 brd ff:ff:ff:ff:ff:ff 2023-10-19T13:04:28.504 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp1s0f0 2023-10-19T13:04:28.504 INFO:teuthology.orchestra.run.smithi057.stdout: inet 172.21.15.57/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-19T13:04:28.504 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft 41442sec preferred_lft 41442sec 2023-10-19T13:04:28.505 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 fe80::ec4:7aff:febd:1562/64 scope link noprefixroute 2023-10-19T13:04:28.505 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T13:04:28.505 INFO:teuthology.orchestra.run.smithi057.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-19T13:04:28.505 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:bd:15:63 brd ff:ff:ff:ff:ff:ff 2023-10-19T13:04:28.505 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp1s0f1 2023-10-19T13:04:28.506 INFO:teuthology.orchestra.run.smithi057.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-19T13:04:28.506 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0e:fe:f1:b5:41:59 brd ff:ff:ff:ff:ff:ff 2023-10-19T13:04:28.506 INFO:teuthology.orchestra.run.smithi057.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-19T13:04:28.506 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T13:04:28.506 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 fe80::64b1:a2ff:feb3:b08/64 scope link 2023-10-19T13:04:28.507 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T13:04:28.507 INFO:teuthology.orchestra.run.smithi057.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-19T13:04:28.507 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0e:fe:f1:b5:41:59 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T13:04:28.507 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 fe80::cfe:f1ff:feb5:4159/64 scope link 2023-10-19T13:04:28.507 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T13:04:28.508 INFO:teuthology.orchestra.run.smithi057.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-19T13:04:28.508 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 62:53:72:9d:52:71 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-19T13:04:28.508 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 fe80::6053:72ff:fe9d:5271/64 scope link 2023-10-19T13:04:28.508 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T13:04:28.508 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:04:28.509 DEBUG:teuthology.orchestra.run.smithi057:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-19T13:04:28.570 INFO:teuthology.orchestra.run.smithi057.stdout:1 2023-10-19T13:04:28.572 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:04:28.572 DEBUG:teuthology.orchestra.run.smithi057:> route 2023-10-19T13:04:28.628 INFO:teuthology.orchestra.run.smithi057.stdout:Kernel IP routing table 2023-10-19T13:04:28.628 INFO:teuthology.orchestra.run.smithi057.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-19T13:04:28.629 INFO:teuthology.orchestra.run.smithi057.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-19T13:04:28.629 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-19T13:04:28.629 INFO:teuthology.orchestra.run.smithi057.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-19T13:04:28.629 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:04:28.630 DEBUG:teuthology.orchestra.run.smithi057:> (cd / && exec bash -c ' 2023-10-19T13:04:28.630 DEBUG:teuthology.orchestra.run.smithi057:> set -e 2023-10-19T13:04:28.631 DEBUG:teuthology.orchestra.run.smithi057:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-19T13:04:28.631 DEBUG:teuthology.orchestra.run.smithi057:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-19T13:04:28.631 DEBUG:teuthology.orchestra.run.smithi057:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-19T13:04:28.632 DEBUG:teuthology.orchestra.run.smithi057:> ') 2023-10-19T13:04:28.734 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:04:28.734 DEBUG:teuthology.orchestra.run.smithi057:> ip netns list 2023-10-19T13:04:28.749 INFO:teuthology.orchestra.run.smithi057.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-19T13:04:28.750 INFO:teuthology.orchestra.run.smithi057.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-19T13:04:28.750 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-19T13:04:28.750 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:04:28.751 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip link set brx.0 up 2023-10-19T13:04:28.820 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-10-19T13:04:28.820 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-19T13:04:28.821 DEBUG:teuthology.orchestra.run.smithi057:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-19T13:04:28.875 INFO:teuthology.orchestra.run.smithi057.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-19T13:04:28.875 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-19T13:04:28.876 DEBUG:teuthology.orchestra.run.smithi057:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-19T13:04:28.930 DEBUG:teuthology.orchestra.run.smithi057:> sudo modprobe fuse 2023-10-19T13:04:29.001 DEBUG:teuthology.orchestra.run.smithi057:> cat /proc/self/mounts | awk '{print $2}' 2023-10-19T13:04:29.056 INFO:teuthology.orchestra.run.smithi057.stdout:/proc 2023-10-19T13:04:29.057 INFO:teuthology.orchestra.run.smithi057.stdout:/sys 2023-10-19T13:04:29.057 INFO:teuthology.orchestra.run.smithi057.stdout:/dev 2023-10-19T13:04:29.057 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/security 2023-10-19T13:04:29.057 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/shm 2023-10-19T13:04:29.057 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/pts 2023-10-19T13:04:29.058 INFO:teuthology.orchestra.run.smithi057.stdout:/run 2023-10-19T13:04:29.058 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/cgroup 2023-10-19T13:04:29.058 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/pstore 2023-10-19T13:04:29.058 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/bpf 2023-10-19T13:04:29.058 INFO:teuthology.orchestra.run.smithi057.stdout:/ 2023-10-19T13:04:29.059 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/selinux 2023-10-19T13:04:29.059 INFO:teuthology.orchestra.run.smithi057.stdout:/proc/sys/fs/binfmt_misc 2023-10-19T13:04:29.059 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/hugepages 2023-10-19T13:04:29.059 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/mqueue 2023-10-19T13:04:29.059 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/debug 2023-10-19T13:04:29.059 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/tracing 2023-10-19T13:04:29.060 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/fuse/connections 2023-10-19T13:04:29.060 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/config 2023-10-19T13:04:29.060 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-sysctl.service 2023-10-19T13:04:29.060 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-10-19T13:04:29.060 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-10-19T13:04:29.060 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/nfs/rpc_pipefs 2023-10-19T13:04:29.061 INFO:teuthology.orchestra.run.smithi057.stdout:/run/user/1000 2023-10-19T13:04:29.061 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph 2023-10-19T13:04:29.061 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-0 2023-10-19T13:04:29.061 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-1 2023-10-19T13:04:29.061 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-2 2023-10-19T13:04:29.062 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-3 2023-10-19T13:04:29.062 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns 2023-10-19T13:04:29.062 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T13:04:29.062 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T13:04:29.062 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-19T13:04:29.062 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-19T13:04:29.063 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:04:29.064 DEBUG:teuthology.orchestra.run.smithi057:> ls /sys/fs/fuse/connections 2023-10-19T13:04:29.118 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-10-19T13:04:29.119 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-10-19T13:04:29.160 DEBUG:teuthology.orchestra.run.smithi057:> sudo modprobe fuse 2023-10-19T13:04:29.191 DEBUG:teuthology.orchestra.run.smithi057:> cat /proc/self/mounts | awk '{print $2}' 2023-10-19T13:04:29.247 INFO:teuthology.orchestra.run.smithi057.stdout:/proc 2023-10-19T13:04:29.247 INFO:teuthology.orchestra.run.smithi057.stdout:/sys 2023-10-19T13:04:29.247 INFO:teuthology.orchestra.run.smithi057.stdout:/dev 2023-10-19T13:04:29.248 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/security 2023-10-19T13:04:29.248 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/shm 2023-10-19T13:04:29.248 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/pts 2023-10-19T13:04:29.248 INFO:teuthology.orchestra.run.smithi057.stdout:/run 2023-10-19T13:04:29.248 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/cgroup 2023-10-19T13:04:29.249 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/pstore 2023-10-19T13:04:29.249 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/bpf 2023-10-19T13:04:29.249 INFO:teuthology.orchestra.run.smithi057.stdout:/ 2023-10-19T13:04:29.249 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/selinux 2023-10-19T13:04:29.249 INFO:teuthology.orchestra.run.smithi057.stdout:/proc/sys/fs/binfmt_misc 2023-10-19T13:04:29.250 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/hugepages 2023-10-19T13:04:29.250 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/mqueue 2023-10-19T13:04:29.250 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/debug 2023-10-19T13:04:29.250 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/tracing 2023-10-19T13:04:29.250 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/fuse/connections 2023-10-19T13:04:29.251 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/config 2023-10-19T13:04:29.251 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-sysctl.service 2023-10-19T13:04:29.251 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-10-19T13:04:29.251 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-10-19T13:04:29.251 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/nfs/rpc_pipefs 2023-10-19T13:04:29.252 INFO:teuthology.orchestra.run.smithi057.stdout:/run/user/1000 2023-10-19T13:04:29.252 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph 2023-10-19T13:04:29.252 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-0 2023-10-19T13:04:29.252 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-1 2023-10-19T13:04:29.252 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-2 2023-10-19T13:04:29.253 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-3 2023-10-19T13:04:29.253 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns 2023-10-19T13:04:29.253 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T13:04:29.253 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T13:04:29.253 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-19T13:04:29.253 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-19T13:04:29.254 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:04:29.254 DEBUG:teuthology.orchestra.run.smithi057:> ls /sys/fs/fuse/connections 2023-10-19T13:04:29.255 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi057.stderr:2023-10-19T13:04:29.250+0000 7f0641e3e480 -1 init, newargv = 0x5621ec4b6b60 newargc=15 2023-10-19T13:04:29.255 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi057.stderr:ceph-fuse[132192]: starting ceph client 2023-10-19T13:04:29.257 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi057.stderr:ceph-fuse[132192]: starting fuse 2023-10-19T13:04:29.270 INFO:teuthology.orchestra.run.smithi057.stdout:43 2023-10-19T13:04:29.271 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2023-10-19T13:04:29.271 DEBUG:teuthology.orchestra.run.smithi057:> sudo stdin-killer -- python3 -c ' 2023-10-19T13:04:29.272 DEBUG:teuthology.orchestra.run.smithi057:> import glob 2023-10-19T13:04:29.272 DEBUG:teuthology.orchestra.run.smithi057:> import re 2023-10-19T13:04:29.272 DEBUG:teuthology.orchestra.run.smithi057:> import os 2023-10-19T13:04:29.273 DEBUG:teuthology.orchestra.run.smithi057:> import subprocess 2023-10-19T13:04:29.273 DEBUG:teuthology.orchestra.run.smithi057:> 2023-10-19T13:04:29.273 DEBUG:teuthology.orchestra.run.smithi057:> def _find_admin_socket(client_name): 2023-10-19T13:04:29.274 DEBUG:teuthology.orchestra.run.smithi057:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-10-19T13:04:29.274 DEBUG:teuthology.orchestra.run.smithi057:> files = glob.glob(asok_path) 2023-10-19T13:04:29.275 DEBUG:teuthology.orchestra.run.smithi057:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-10-19T13:04:29.275 DEBUG:teuthology.orchestra.run.smithi057:> 2023-10-19T13:04:29.275 DEBUG:teuthology.orchestra.run.smithi057:> # Given a non-glob path, it better be there 2023-10-19T13:04:29.276 DEBUG:teuthology.orchestra.run.smithi057:> if "*" not in asok_path: 2023-10-19T13:04:29.276 DEBUG:teuthology.orchestra.run.smithi057:> assert(len(files) == 1) 2023-10-19T13:04:29.277 DEBUG:teuthology.orchestra.run.smithi057:> return files[0] 2023-10-19T13:04:29.277 DEBUG:teuthology.orchestra.run.smithi057:> 2023-10-19T13:04:29.277 DEBUG:teuthology.orchestra.run.smithi057:> for f in files: 2023-10-19T13:04:29.278 DEBUG:teuthology.orchestra.run.smithi057:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-10-19T13:04:29.278 DEBUG:teuthology.orchestra.run.smithi057:> if os.path.exists("/proc/{0}".format(pid)): 2023-10-19T13:04:29.278 DEBUG:teuthology.orchestra.run.smithi057:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-10-19T13:04:29.279 DEBUG:teuthology.orchestra.run.smithi057:> contents = proc_f.read() 2023-10-19T13:04:29.279 DEBUG:teuthology.orchestra.run.smithi057:> if mountpoint in contents: 2023-10-19T13:04:29.280 DEBUG:teuthology.orchestra.run.smithi057:> return f 2023-10-19T13:04:29.280 DEBUG:teuthology.orchestra.run.smithi057:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-10-19T13:04:29.280 DEBUG:teuthology.orchestra.run.smithi057:> 2023-10-19T13:04:29.281 DEBUG:teuthology.orchestra.run.smithi057:> print(_find_admin_socket("client.0")) 2023-10-19T13:04:29.281 DEBUG:teuthology.orchestra.run.smithi057:> ' 2023-10-19T13:04:29.395 INFO:teuthology.orchestra.run.smithi057.stdout:/var/run/ceph/ceph-client.0.132192.asok 2023-10-19T13:04:29.398 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2023-10-19T13:04:29.405 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.132192.asok 2023-10-19T13:04:29.406 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:04:29.406 DEBUG:teuthology.orchestra.run.smithi057:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.132192.asok status 2023-10-19T13:04:29.546 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2023-10-19T13:04:29.546 INFO:teuthology.orchestra.run.smithi057.stdout: "metadata": { 2023-10-19T13:04:29.547 INFO:teuthology.orchestra.run.smithi057.stdout: "ceph_sha1": "f7420d7f8e55ceea786dcbf536c338ce920c21eb", 2023-10-19T13:04:29.547 INFO:teuthology.orchestra.run.smithi057.stdout: "ceph_version": "ceph version 18.0.0-6786-gf7420d7f (f7420d7f8e55ceea786dcbf536c338ce920c21eb) reef (dev)", 2023-10-19T13:04:29.547 INFO:teuthology.orchestra.run.smithi057.stdout: "entity_id": "0", 2023-10-19T13:04:29.547 INFO:teuthology.orchestra.run.smithi057.stdout: "hostname": "smithi057", 2023-10-19T13:04:29.548 INFO:teuthology.orchestra.run.smithi057.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-10-19T13:04:29.548 INFO:teuthology.orchestra.run.smithi057.stdout: "pid": "132192", 2023-10-19T13:04:29.548 INFO:teuthology.orchestra.run.smithi057.stdout: "root": "/" 2023-10-19T13:04:29.548 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:04:29.548 INFO:teuthology.orchestra.run.smithi057.stdout: "dentry_count": 0, 2023-10-19T13:04:29.548 INFO:teuthology.orchestra.run.smithi057.stdout: "dentry_pinned_count": 0, 2023-10-19T13:04:29.549 INFO:teuthology.orchestra.run.smithi057.stdout: "id": 9222, 2023-10-19T13:04:29.549 INFO:teuthology.orchestra.run.smithi057.stdout: "inst": { 2023-10-19T13:04:29.549 INFO:teuthology.orchestra.run.smithi057.stdout: "name": { 2023-10-19T13:04:29.549 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "client", 2023-10-19T13:04:29.549 INFO:teuthology.orchestra.run.smithi057.stdout: "num": 9222 2023-10-19T13:04:29.550 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:04:29.550 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": { 2023-10-19T13:04:29.550 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "v1", 2023-10-19T13:04:29.550 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": "192.168.0.1:0", 2023-10-19T13:04:29.550 INFO:teuthology.orchestra.run.smithi057.stdout: "nonce": 61972619 2023-10-19T13:04:29.551 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:04:29.551 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:04:29.551 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": { 2023-10-19T13:04:29.551 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "v1", 2023-10-19T13:04:29.551 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": "192.168.0.1:0", 2023-10-19T13:04:29.552 INFO:teuthology.orchestra.run.smithi057.stdout: "nonce": 61972619 2023-10-19T13:04:29.552 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:04:29.552 INFO:teuthology.orchestra.run.smithi057.stdout: "inst_str": "client.9222 192.168.0.1:0/61972619", 2023-10-19T13:04:29.552 INFO:teuthology.orchestra.run.smithi057.stdout: "addr_str": "192.168.0.1:0/61972619", 2023-10-19T13:04:29.552 INFO:teuthology.orchestra.run.smithi057.stdout: "inode_count": 1, 2023-10-19T13:04:29.552 INFO:teuthology.orchestra.run.smithi057.stdout: "mds_epoch": 56, 2023-10-19T13:04:29.553 INFO:teuthology.orchestra.run.smithi057.stdout: "osd_epoch": 144, 2023-10-19T13:04:29.553 INFO:teuthology.orchestra.run.smithi057.stdout: "osd_epoch_barrier": 0, 2023-10-19T13:04:29.553 INFO:teuthology.orchestra.run.smithi057.stdout: "blocklisted": false, 2023-10-19T13:04:29.553 INFO:teuthology.orchestra.run.smithi057.stdout: "fs_name": "cephfs" 2023-10-19T13:04:29.553 INFO:teuthology.orchestra.run.smithi057.stdout:} 2023-10-19T13:04:29.555 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:04:29.555 DEBUG:teuthology.orchestra.run.smithi057:> stat --file-system '--printf=%T 2023-10-19T13:04:29.555 DEBUG:teuthology.orchestra.run.smithi057:> ' -- /home/ubuntu/cephtest/mnt.0 2023-10-19T13:04:29.571 INFO:teuthology.orchestra.run.smithi057.stdout:fuseblk 2023-10-19T13:04:29.571 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-10-19T13:04:29.571 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:04:29.572 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-19T13:04:29.643 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-19T13:04:29.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:29.790+0000 7f9602739640 1 -- 172.21.15.57:0/2533074087 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f95fc0ab500 msgr2=0x7f95fc0ab990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:29.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:29.790+0000 7f9602739640 1 --2- 172.21.15.57:0/2533074087 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f95fc0ab500 0x7f95fc0ab990 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7f95fc0cbac0 tx=0x7f95f402c250 comp rx=0 tx=0).stop 2023-10-19T13:04:29.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:29.790+0000 7f9602739640 1 -- 172.21.15.57:0/2533074087 shutdown_connections 2023-10-19T13:04:29.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:29.790+0000 7f9602739640 1 --2- 172.21.15.57:0/2533074087 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f95fc0ab500 0x7f95fc0ab990 unknown :-1 s=CLOSED pgs=1614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:29.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:29.790+0000 7f9602739640 1 --2- 172.21.15.57:0/2533074087 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f95fc0c96b0 0x7f95fc0c9b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:29.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:29.790+0000 7f9602739640 1 --2- 172.21.15.57:0/2533074087 >> v2:172.21.15.179:3300/0 conn(0x7f95fc06af00 0x7f95fc06b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:29.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:29.790+0000 7f9602739640 1 -- 172.21.15.57:0/2533074087 >> 172.21.15.57:0/2533074087 conn(0x7f95fc069930 msgr2=0x7f95fc069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:29.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:29.790+0000 7f9602739640 1 -- 172.21.15.57:0/2533074087 shutdown_connections 2023-10-19T13:04:29.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:29.790+0000 7f9602739640 1 -- 172.21.15.57:0/2533074087 wait complete. 2023-10-19T13:04:29.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:29.791+0000 7f9602739640 1 Processor -- start 2023-10-19T13:04:29.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:29.791+0000 7f9602739640 1 -- start start 2023-10-19T13:04:29.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:29.792+0000 7f9602739640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f95fc06af00 0x7f95fc1ee730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:29.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:29.792+0000 7f9602739640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f95fc0ab500 0x7f95fc1eec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:29.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:29.792+0000 7f9602739640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f95fc0c96b0 0x7f95fc12df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:29.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:29.792+0000 7f9602739640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95fc0ab310 con 0x7f95fc0c96b0 2023-10-19T13:04:29.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:29.792+0000 7f9602739640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95fc12f1a0 con 0x7f95fc0ab500 2023-10-19T13:04:29.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:29.792+0000 7f9602739640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95fc1ef1b0 con 0x7f95fc06af00 2023-10-19T13:04:29.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:29.792+0000 7f9600cd6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f95fc0c96b0 0x7f95fc12df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:29.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:29.792+0000 7f9600cd6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f95fc0c96b0 0x7f95fc12df50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46156/0 (socket says 172.21.15.57:46156) 2023-10-19T13:04:29.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:29.792+0000 7f9600cd6640 1 -- 172.21.15.57:0/1917791124 learned_addr learned my addr 172.21.15.57:0/1917791124 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:29.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:29.792+0000 7f95fb7fe640 1 --2- 172.21.15.57:0/1917791124 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f95fc0ab500 0x7f95fc1eec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:29.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:29.792+0000 7f95fbfff640 1 --2- 172.21.15.57:0/1917791124 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f95fc06af00 0x7f95fc1ee730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:29.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:29.792+0000 7f9600cd6640 1 -- 172.21.15.57:0/1917791124 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f95fc06af00 msgr2=0x7f95fc1ee730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:29.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:29.792+0000 7f9600cd6640 1 --2- 172.21.15.57:0/1917791124 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f95fc06af00 0x7f95fc1ee730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:29.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:29.792+0000 7f9600cd6640 1 -- 172.21.15.57:0/1917791124 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f95fc0ab500 msgr2=0x7f95fc1eec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:29.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:29.792+0000 7f9600cd6640 1 --2- 172.21.15.57:0/1917791124 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f95fc0ab500 0x7f95fc1eec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:29.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:29.792+0000 7f9600cd6640 1 -- 172.21.15.57:0/1917791124 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95f4009900 con 0x7f95fc0c96b0 2023-10-19T13:04:29.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:29.793+0000 7f9600cd6640 1 --2- 172.21.15.57:0/1917791124 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f95fc0c96b0 0x7f95fc12df50 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7f95f4009d60 tx=0x7f95f4035e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:29.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:29.793+0000 7f95f8ff9640 1 -- 172.21.15.57:0/1917791124 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95f4034440 con 0x7f95fc0c96b0 2023-10-19T13:04:29.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:29.793+0000 7f95f8ff9640 1 -- 172.21.15.57:0/1917791124 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f95f4004450 con 0x7f95fc0c96b0 2023-10-19T13:04:29.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:29.793+0000 7f9602739640 1 -- 172.21.15.57:0/1917791124 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95fc0ab030 con 0x7f95fc0c96b0 2023-10-19T13:04:29.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:29.793+0000 7f95f8ff9640 1 -- 172.21.15.57:0/1917791124 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95f4004860 con 0x7f95fc0c96b0 2023-10-19T13:04:29.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:29.793+0000 7f9602739640 1 -- 172.21.15.57:0/1917791124 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95fc12c130 con 0x7f95fc0c96b0 2023-10-19T13:04:29.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:29.794+0000 7f9602739640 1 -- 172.21.15.57:0/1917791124 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95fc0690e0 con 0x7f95fc0c96b0 2023-10-19T13:04:29.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:29.794+0000 7f95f8ff9640 1 -- 172.21.15.57:0/1917791124 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 69) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f95f400a040 con 0x7f95fc0c96b0 2023-10-19T13:04:29.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:29.795+0000 7f95f8ff9640 1 --2- 172.21.15.57:0/1917791124 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f95dc06af60 0x7f95dc06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:29.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:29.795+0000 7f95f8ff9640 1 -- 172.21.15.57:0/1917791124 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f95f40ac930 con 0x7f95fc0c96b0 2023-10-19T13:04:29.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:29.795+0000 7f95fbfff640 1 --2- 172.21.15.57:0/1917791124 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f95dc06af60 0x7f95dc06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:29.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:29.795+0000 7f95fbfff640 1 --2- 172.21.15.57:0/1917791124 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f95dc06af60 0x7f95dc06d420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f95e4005d00 tx=0x7f95e400b410 comp rx=0 tx=0).ready entity=mgr.7960 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:29.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:29.799+0000 7f95f8ff9640 1 -- 172.21.15.57:0/1917791124 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f95f40778c0 con 0x7f95fc0c96b0 2023-10-19T13:04:29.999 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:29.996+0000 7f9602739640 1 -- 172.21.15.57:0/1917791124 --> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f95fc0abed0 con 0x7f95dc06af60 2023-10-19T13:04:30.001 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:29.998+0000 7f95f8ff9640 1 -- 172.21.15.57:0/1917791124 <== mgr.7960 v2:172.21.15.179:6832/2346394725 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f95e4015070 con 0x7f95dc06af60 2023-10-19T13:04:30.001 INFO:teuthology.orchestra.run.smithi057.stdout:[ 2023-10-19T13:04:30.001 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:04:30.001 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "cephfs" 2023-10-19T13:04:30.002 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:04:30.002 INFO:teuthology.orchestra.run.smithi057.stdout:] 2023-10-19T13:04:30.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.000+0000 7f9602739640 1 -- 172.21.15.57:0/1917791124 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f95dc06af60 msgr2=0x7f95dc06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:30.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.000+0000 7f9602739640 1 --2- 172.21.15.57:0/1917791124 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f95dc06af60 0x7f95dc06d420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f95e4005d00 tx=0x7f95e400b410 comp rx=0 tx=0).stop 2023-10-19T13:04:30.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.000+0000 7f9602739640 1 -- 172.21.15.57:0/1917791124 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f95fc0c96b0 msgr2=0x7f95fc12df50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:30.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.000+0000 7f9602739640 1 --2- 172.21.15.57:0/1917791124 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f95fc0c96b0 0x7f95fc12df50 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7f95f4009d60 tx=0x7f95f4035e30 comp rx=0 tx=0).stop 2023-10-19T13:04:30.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.000+0000 7f9602739640 1 -- 172.21.15.57:0/1917791124 shutdown_connections 2023-10-19T13:04:30.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.000+0000 7f9602739640 1 --2- 172.21.15.57:0/1917791124 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f95dc06af60 0x7f95dc06d420 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:30.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.000+0000 7f9602739640 1 --2- 172.21.15.57:0/1917791124 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f95fc0c96b0 0x7f95fc12df50 unknown :-1 s=CLOSED pgs=1615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:30.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.000+0000 7f9602739640 1 --2- 172.21.15.57:0/1917791124 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f95fc0ab500 0x7f95fc1eec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:30.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.000+0000 7f9602739640 1 --2- 172.21.15.57:0/1917791124 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f95fc06af00 0x7f95fc1ee730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:30.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.000+0000 7f9602739640 1 -- 172.21.15.57:0/1917791124 >> 172.21.15.57:0/1917791124 conn(0x7f95fc069930 msgr2=0x7f95fc069d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:30.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.000+0000 7f9602739640 1 -- 172.21.15.57:0/1917791124 shutdown_connections 2023-10-19T13:04:30.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.000+0000 7f9602739640 1 -- 172.21.15.57:0/1917791124 wait complete. 2023-10-19T13:04:30.013 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-19T13:04:30.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.122+0000 7f03dd026640 1 -- 172.21.15.57:0/3304118783 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f03c4003970 msgr2=0x7f03c4003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:30.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.122+0000 7f03dd026640 1 --2- 172.21.15.57:0/3304118783 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f03c4003970 0x7f03c4003d90 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7f03c8009b50 tx=0x7f03c40057f0 comp rx=0 tx=0).stop 2023-10-19T13:04:30.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.122+0000 7f03d759e640 1 -- 172.21.15.57:0/3304118783 reap_dead start 2023-10-19T13:04:30.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.122+0000 7f03dd026640 1 -- 172.21.15.57:0/3304118783 shutdown_connections 2023-10-19T13:04:30.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.122+0000 7f03dd026640 1 -- 172.21.15.57:0/3304118783 >> 172.21.15.57:0/3304118783 conn(0x7f03d806fae0 msgr2=0x7f03d806fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:30.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.122+0000 7f03dd026640 1 -- 172.21.15.57:0/3304118783 shutdown_connections 2023-10-19T13:04:30.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.122+0000 7f03dd026640 1 -- 172.21.15.57:0/3304118783 wait complete. 2023-10-19T13:04:30.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.124+0000 7f03dd026640 1 Processor -- start 2023-10-19T13:04:30.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.124+0000 7f03dd026640 1 -- start start 2023-10-19T13:04:30.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.124+0000 7f03dd026640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f03c4003970 0x7f03d814c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:30.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.124+0000 7f03dd026640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f03d81541c0 0x7f03d81ee8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:30.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.124+0000 7f03dd026640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f03d813cc20 0x7f03d813d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:30.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.124+0000 7f03dd026640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03d8155e50 con 0x7f03d813cc20 2023-10-19T13:04:30.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.124+0000 7f03dd026640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03d813d5e0 con 0x7f03d81541c0 2023-10-19T13:04:30.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.124+0000 7f03dd026640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03d814d5b0 con 0x7f03c4003970 2023-10-19T13:04:30.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.125+0000 7f03d759e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f03d813cc20 0x7f03d813d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:30.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.125+0000 7f03d759e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f03d813cc20 0x7f03d813d0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46178/0 (socket says 172.21.15.57:46178) 2023-10-19T13:04:30.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.125+0000 7f03d759e640 1 -- 172.21.15.57:0/1078610017 learned_addr learned my addr 172.21.15.57:0/1078610017 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:30.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.125+0000 7f03d6d9d640 1 --2- 172.21.15.57:0/1078610017 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f03c4003970 0x7f03d814c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:30.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.125+0000 7f03cffff640 1 --2- 172.21.15.57:0/1078610017 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f03d81541c0 0x7f03d81ee8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:30.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.125+0000 7f03d759e640 1 -- 172.21.15.57:0/1078610017 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f03c4003970 msgr2=0x7f03d814c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:30.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.125+0000 7f03d759e640 1 --2- 172.21.15.57:0/1078610017 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f03c4003970 0x7f03d814c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:30.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.125+0000 7f03d759e640 1 -- 172.21.15.57:0/1078610017 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f03d81541c0 msgr2=0x7f03d81ee8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:30.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.125+0000 7f03d759e640 1 --2- 172.21.15.57:0/1078610017 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f03d81541c0 0x7f03d81ee8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:30.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.125+0000 7f03d759e640 1 -- 172.21.15.57:0/1078610017 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03d8155b20 con 0x7f03d813cc20 2023-10-19T13:04:30.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.125+0000 7f03d6d9d640 1 --2- 172.21.15.57:0/1078610017 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f03c4003970 0x7f03d814c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:04:30.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.125+0000 7f03cffff640 1 --2- 172.21.15.57:0/1078610017 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f03d81541c0 0x7f03d81ee8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:04:30.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.125+0000 7f03d759e640 1 --2- 172.21.15.57:0/1078610017 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f03d813cc20 0x7f03d813d0a0 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7f03d0000f80 tx=0x7f03d000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:30.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.125+0000 7f03cf7fe640 1 -- 172.21.15.57:0/1078610017 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03d000c940 con 0x7f03d813cc20 2023-10-19T13:04:30.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.125+0000 7f03cf7fe640 1 -- 172.21.15.57:0/1078610017 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f03d000cb90 con 0x7f03d813cc20 2023-10-19T13:04:30.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.125+0000 7f03dd026640 1 -- 172.21.15.57:0/1078610017 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03d814cd10 con 0x7f03d813cc20 2023-10-19T13:04:30.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.125+0000 7f03cf7fe640 1 -- 172.21.15.57:0/1078610017 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03d0011460 con 0x7f03d813cc20 2023-10-19T13:04:30.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.125+0000 7f03dd026640 1 -- 172.21.15.57:0/1078610017 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03d814d2d0 con 0x7f03d813cc20 2023-10-19T13:04:30.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.126+0000 7f03dd026640 1 -- 172.21.15.57:0/1078610017 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03d8155fc0 con 0x7f03d813cc20 2023-10-19T13:04:30.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.127+0000 7f03cf7fe640 1 -- 172.21.15.57:0/1078610017 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 69) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f03d00115c0 con 0x7f03d813cc20 2023-10-19T13:04:30.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.127+0000 7f03cf7fe640 1 --2- 172.21.15.57:0/1078610017 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f03b806af90 0x7f03b806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:30.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.127+0000 7f03cf7fe640 1 -- 172.21.15.57:0/1078610017 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f03d0089ee0 con 0x7f03d813cc20 2023-10-19T13:04:30.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.127+0000 7f03d6d9d640 1 --2- 172.21.15.57:0/1078610017 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f03b806af90 0x7f03b806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:30.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.128+0000 7f03d6d9d640 1 --2- 172.21.15.57:0/1078610017 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f03b806af90 0x7f03b806d450 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f03c8036000 tx=0x7f03c80046c0 comp rx=0 tx=0).ready entity=mgr.7960 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:30.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.131+0000 7f03cf7fe640 1 -- 172.21.15.57:0/1078610017 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f03d8155fc0 con 0x7f03d813cc20 2023-10-19T13:04:30.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.355+0000 7f03dd026640 1 -- 172.21.15.57:0/1078610017 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f03d8155f90 con 0x7f03d813cc20 2023-10-19T13:04:30.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.467+0000 7f03cf7fe640 1 -- 172.21.15.57:0/1078610017 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v70) v1 ==== 92+0+0 (secure 0 0 0) 0x7f03d00586a0 con 0x7f03d813cc20 2023-10-19T13:04:30.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.471+0000 7f03dd026640 1 -- 172.21.15.57:0/1078610017 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f03b806af90 msgr2=0x7f03b806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:30.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.471+0000 7f03dd026640 1 --2- 172.21.15.57:0/1078610017 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f03b806af90 0x7f03b806d450 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f03c8036000 tx=0x7f03c80046c0 comp rx=0 tx=0).stop 2023-10-19T13:04:30.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.471+0000 7f03dd026640 1 -- 172.21.15.57:0/1078610017 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f03d813cc20 msgr2=0x7f03d813d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:30.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.471+0000 7f03cf7fe640 1 -- 172.21.15.57:0/1078610017 <== mon.0 v2:172.21.15.57:3300/0 8 ==== mgrmap(e 70) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f03d008b070 con 0x7f03d813cc20 2023-10-19T13:04:30.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.471+0000 7f03dd026640 1 --2- 172.21.15.57:0/1078610017 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f03d813cc20 0x7f03d813d0a0 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7f03d0000f80 tx=0x7f03d000ed30 comp rx=0 tx=0).stop 2023-10-19T13:04:30.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.471+0000 7f03dd026640 1 -- 172.21.15.57:0/1078610017 shutdown_connections 2023-10-19T13:04:30.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.471+0000 7f03dd026640 1 --2- 172.21.15.57:0/1078610017 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f03b806af90 0x7f03b806d450 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:30.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.471+0000 7f03dd026640 1 --2- 172.21.15.57:0/1078610017 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f03d813cc20 0x7f03d813d0a0 unknown :-1 s=CLOSED pgs=1618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:30.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.471+0000 7f03dd026640 1 --2- 172.21.15.57:0/1078610017 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f03d81541c0 0x7f03d81ee8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:30.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.471+0000 7f03dd026640 1 --2- 172.21.15.57:0/1078610017 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f03c4003970 0x7f03d814c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:30.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.471+0000 7f03dd026640 1 -- 172.21.15.57:0/1078610017 >> 172.21.15.57:0/1078610017 conn(0x7f03d806fae0 msgr2=0x7f03d815a370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:30.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.472+0000 7f03dd026640 1 -- 172.21.15.57:0/1078610017 shutdown_connections 2023-10-19T13:04:30.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.472+0000 7f03dd026640 1 -- 172.21.15.57:0/1078610017 wait complete. 2023-10-19T13:04:30.486 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-19T13:04:30.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.599+0000 7fe478355640 1 -- 172.21.15.57:0/2593573226 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe47015b2a0 msgr2=0x7fe4700d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:30.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.599+0000 7fe478355640 1 --2- 172.21.15.57:0/2593573226 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe47015b2a0 0x7fe4700d1910 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7fe464005fd0 tx=0x7fe46402bd90 comp rx=0 tx=0).stop 2023-10-19T13:04:30.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.599+0000 7fe478355640 1 -- 172.21.15.57:0/2593573226 shutdown_connections 2023-10-19T13:04:30.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.599+0000 7fe478355640 1 --2- 172.21.15.57:0/2593573226 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe4700ceb40 0x7fe4700b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:30.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.599+0000 7fe478355640 1 --2- 172.21.15.57:0/2593573226 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe47015b2a0 0x7fe4700d1910 unknown :-1 s=CLOSED pgs=1623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:30.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.599+0000 7fe478355640 1 -- 172.21.15.57:0/2593573226 >> 172.21.15.57:0/2593573226 conn(0x7fe4700d0d60 msgr2=0x7fe4700d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:30.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.599+0000 7fe478355640 1 -- 172.21.15.57:0/2593573226 shutdown_connections 2023-10-19T13:04:30.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.599+0000 7fe478355640 1 -- 172.21.15.57:0/2593573226 wait complete. 2023-10-19T13:04:30.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.601+0000 7fe478355640 1 Processor -- start 2023-10-19T13:04:30.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.601+0000 7fe478355640 1 -- start start 2023-10-19T13:04:30.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.601+0000 7fe478355640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe4700ceb40 0x7fe4701582e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:30.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.601+0000 7fe478355640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe4700cf340 0x7fe47015b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:30.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.601+0000 7fe478355640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe47015b2a0 0x7fe4701e5db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:30.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.601+0000 7fe478355640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4701e5990 con 0x7fe47015b2a0 2023-10-19T13:04:30.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.601+0000 7fe478355640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4701e4610 con 0x7fe4700ceb40 2023-10-19T13:04:30.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.601+0000 7fe478355640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4701e52d0 con 0x7fe4700cf340 2023-10-19T13:04:30.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.602+0000 7fe4768f2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe47015b2a0 0x7fe4701e5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:30.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.602+0000 7fe4768f2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe47015b2a0 0x7fe4701e5db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46202/0 (socket says 172.21.15.57:46202) 2023-10-19T13:04:30.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.602+0000 7fe4768f2640 1 -- 172.21.15.57:0/3772018921 learned_addr learned my addr 172.21.15.57:0/3772018921 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:30.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.602+0000 7fe4758f0640 1 --2- 172.21.15.57:0/3772018921 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe4700cf340 0x7fe47015b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:30.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.602+0000 7fe4760f1640 1 --2- 172.21.15.57:0/3772018921 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe4700ceb40 0x7fe4701582e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:30.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.602+0000 7fe4768f2640 1 -- 172.21.15.57:0/3772018921 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe4700cf340 msgr2=0x7fe47015b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:30.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.602+0000 7fe4768f2640 1 --2- 172.21.15.57:0/3772018921 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe4700cf340 0x7fe47015b6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:30.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.602+0000 7fe4768f2640 1 -- 172.21.15.57:0/3772018921 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe4700ceb40 msgr2=0x7fe4701582e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:30.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.602+0000 7fe4768f2640 1 --2- 172.21.15.57:0/3772018921 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe4700ceb40 0x7fe4701582e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:30.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.602+0000 7fe4768f2640 1 -- 172.21.15.57:0/3772018921 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe464009560 con 0x7fe47015b2a0 2023-10-19T13:04:30.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.602+0000 7fe4768f2640 1 --2- 172.21.15.57:0/3772018921 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe47015b2a0 0x7fe4701e5db0 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7fe46800eb10 tx=0x7fe46800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:30.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.602+0000 7fe462ffd640 1 -- 172.21.15.57:0/3772018921 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe468004b20 con 0x7fe47015b2a0 2023-10-19T13:04:30.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.602+0000 7fe462ffd640 1 -- 172.21.15.57:0/3772018921 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fe468004ca0 con 0x7fe47015b2a0 2023-10-19T13:04:30.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.602+0000 7fe478355640 1 -- 172.21.15.57:0/3772018921 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4700b0d50 con 0x7fe47015b2a0 2023-10-19T13:04:30.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.602+0000 7fe462ffd640 1 -- 172.21.15.57:0/3772018921 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe468005280 con 0x7fe47015b2a0 2023-10-19T13:04:30.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.602+0000 7fe478355640 1 -- 172.21.15.57:0/3772018921 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe470158820 con 0x7fe47015b2a0 2023-10-19T13:04:30.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.603+0000 7fe478355640 1 -- 172.21.15.57:0/3772018921 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4701e5130 con 0x7fe47015b2a0 2023-10-19T13:04:30.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.604+0000 7fe462ffd640 1 -- 172.21.15.57:0/3772018921 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 70) v1 ==== 89348+0+0 (secure 0 0 0) 0x7fe4680053e0 con 0x7fe47015b2a0 2023-10-19T13:04:30.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.604+0000 7fe462ffd640 1 --2- 172.21.15.57:0/3772018921 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7fe45406afb0 0x7fe45406d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:30.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.605+0000 7fe4760f1640 1 -- 172.21.15.57:0/3772018921 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7fe45406afb0 msgr2=0x7fe45406d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/2346394725 2023-10-19T13:04:30.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.605+0000 7fe462ffd640 1 -- 172.21.15.57:0/3772018921 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fe468014070 con 0x7fe47015b2a0 2023-10-19T13:04:30.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.605+0000 7fe4760f1640 1 --2- 172.21.15.57:0/3772018921 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7fe45406afb0 0x7fe45406d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T13:04:30.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.609+0000 7fe462ffd640 1 -- 172.21.15.57:0/3772018921 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe4701e5130 con 0x7fe47015b2a0 2023-10-19T13:04:30.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.787+0000 7fe478355640 1 -- 172.21.15.57:0/3772018921 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7fe4701e5ad0 con 0x7fe47015b2a0 2023-10-19T13:04:30.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.790+0000 7fe462ffd640 1 -- 172.21.15.57:0/3772018921 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v56)=0 v56) v1 ==== 138+0+0 (secure 0 0 0) 0x7fe4701e5ad0 con 0x7fe47015b2a0 2023-10-19T13:04:30.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.793+0000 7fe478355640 1 -- 172.21.15.57:0/3772018921 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7fe45406afb0 msgr2=0x7fe45406d470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T13:04:30.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.793+0000 7fe478355640 1 --2- 172.21.15.57:0/3772018921 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7fe45406afb0 0x7fe45406d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:30.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.793+0000 7fe478355640 1 -- 172.21.15.57:0/3772018921 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe47015b2a0 msgr2=0x7fe4701e5db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:30.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.793+0000 7fe478355640 1 --2- 172.21.15.57:0/3772018921 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe47015b2a0 0x7fe4701e5db0 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7fe46800eb10 tx=0x7fe46800c2d0 comp rx=0 tx=0).stop 2023-10-19T13:04:30.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.793+0000 7fe478355640 1 -- 172.21.15.57:0/3772018921 shutdown_connections 2023-10-19T13:04:30.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.793+0000 7fe478355640 1 --2- 172.21.15.57:0/3772018921 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7fe45406afb0 0x7fe45406d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:30.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.793+0000 7fe478355640 1 --2- 172.21.15.57:0/3772018921 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe47015b2a0 0x7fe4701e5db0 unknown :-1 s=CLOSED pgs=1624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:30.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.793+0000 7fe478355640 1 --2- 172.21.15.57:0/3772018921 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe4700cf340 0x7fe47015b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:30.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.793+0000 7fe478355640 1 --2- 172.21.15.57:0/3772018921 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe4700ceb40 0x7fe4701582e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:30.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.793+0000 7fe478355640 1 -- 172.21.15.57:0/3772018921 >> 172.21.15.57:0/3772018921 conn(0x7fe4700d0d60 msgr2=0x7fe47006b7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:30.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.793+0000 7fe478355640 1 -- 172.21.15.57:0/3772018921 shutdown_connections 2023-10-19T13:04:30.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.793+0000 7fe478355640 1 -- 172.21.15.57:0/3772018921 wait complete. 2023-10-19T13:04:30.810 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-19T13:04:30.854 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:30.852+0000 7f3f66c0f040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-19T13:04:30.877 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:04:30.875+0000 7f6453a2f040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-19T13:04:30.899 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:30.897+0000 7f3f66c0f040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-19T13:04:30.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.921+0000 7f696f5b6640 1 -- 172.21.15.57:0/4283407910 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f696815b2a0 msgr2=0x7f69680cf720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:30.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.921+0000 7f696f5b6640 1 --2- 172.21.15.57:0/4283407910 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f696815b2a0 0x7f69680cf720 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f6964009830 tx=0x7f696402bcd0 comp rx=0 tx=0).stop 2023-10-19T13:04:30.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.921+0000 7f696f5b6640 1 -- 172.21.15.57:0/4283407910 shutdown_connections 2023-10-19T13:04:30.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.921+0000 7f696f5b6640 1 --2- 172.21.15.57:0/4283407910 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f696815b2a0 0x7f69680cf720 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:30.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.921+0000 7f696f5b6640 1 --2- 172.21.15.57:0/4283407910 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f69680cf340 0x7f69680d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:30.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.921+0000 7f696f5b6640 1 -- 172.21.15.57:0/4283407910 >> 172.21.15.57:0/4283407910 conn(0x7f69680d0d60 msgr2=0x7f69680d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:30.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.921+0000 7f696f5b6640 1 -- 172.21.15.57:0/4283407910 shutdown_connections 2023-10-19T13:04:30.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.921+0000 7f696f5b6640 1 -- 172.21.15.57:0/4283407910 wait complete. 2023-10-19T13:04:30.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.922+0000 7f696f5b6640 1 Processor -- start 2023-10-19T13:04:30.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.922+0000 7f696f5b6640 1 -- start start 2023-10-19T13:04:30.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.923+0000 7f696f5b6640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f69680ceb40 0x7f696815e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:30.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.923+0000 7f696f5b6640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f69680cf340 0x7f6968157eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:30.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.923+0000 7f696f5b6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f696815b2a0 0x7f69680b3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:30.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.923+0000 7f696f5b6640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69681583f0 con 0x7f696815b2a0 2023-10-19T13:04:30.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.923+0000 7f696f5b6640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f696815d180 con 0x7f69680cf340 2023-10-19T13:04:30.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.923+0000 7f696f5b6640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f696815d460 con 0x7f69680ceb40 2023-10-19T13:04:30.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.923+0000 7f696db53640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f696815b2a0 0x7f69680b3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:30.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.923+0000 7f696d352640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f69680ceb40 0x7f696815e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:30.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.923+0000 7f696db53640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f696815b2a0 0x7f69680b3eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46216/0 (socket says 172.21.15.57:46216) 2023-10-19T13:04:30.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.923+0000 7f696db53640 1 -- 172.21.15.57:0/3948403673 learned_addr learned my addr 172.21.15.57:0/3948403673 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:30.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.923+0000 7f696cb51640 1 --2- 172.21.15.57:0/3948403673 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f69680cf340 0x7f6968157eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:30.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.923+0000 7f696db53640 1 -- 172.21.15.57:0/3948403673 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f69680ceb40 msgr2=0x7f696815e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:30.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.923+0000 7f696db53640 1 --2- 172.21.15.57:0/3948403673 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f69680ceb40 0x7f696815e220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:30.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.923+0000 7f696db53640 1 -- 172.21.15.57:0/3948403673 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f69680cf340 msgr2=0x7f6968157eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:30.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.923+0000 7f696db53640 1 --2- 172.21.15.57:0/3948403673 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f69680cf340 0x7f6968157eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:30.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.923+0000 7f696db53640 1 -- 172.21.15.57:0/3948403673 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6964009490 con 0x7f696815b2a0 2023-10-19T13:04:30.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.923+0000 7f696cb51640 1 --2- 172.21.15.57:0/3948403673 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f69680cf340 0x7f6968157eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T13:04:30.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.923+0000 7f696d352640 1 --2- 172.21.15.57:0/3948403673 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f69680ceb40 0x7f696815e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T13:04:30.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.923+0000 7f696db53640 1 --2- 172.21.15.57:0/3948403673 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f696815b2a0 0x7f69680b3eb0 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7f696000cb40 tx=0x7f696000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:30.931 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:04:30.924+0000 7f6453a2f040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-19T13:04:30.932 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.924+0000 7f695dffb640 1 -- 172.21.15.57:0/3948403673 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f696000aca0 con 0x7f696815b2a0 2023-10-19T13:04:30.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.924+0000 7f695dffb640 1 -- 172.21.15.57:0/3948403673 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f6960012420 con 0x7f696815b2a0 2023-10-19T13:04:30.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.924+0000 7f696f5b6640 1 -- 172.21.15.57:0/3948403673 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f696815d740 con 0x7f696815b2a0 2023-10-19T13:04:30.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.924+0000 7f695dffb640 1 -- 172.21.15.57:0/3948403673 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6960012850 con 0x7f696815b2a0 2023-10-19T13:04:30.934 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.924+0000 7f696f5b6640 1 -- 172.21.15.57:0/3948403673 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f696815dd00 con 0x7f696815b2a0 2023-10-19T13:04:30.934 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.925+0000 7f695dffb640 1 -- 172.21.15.57:0/3948403673 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 70) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f69600129b0 con 0x7f696815b2a0 2023-10-19T13:04:30.934 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.925+0000 7f696f5b6640 1 -- 172.21.15.57:0/3948403673 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f696815def0 con 0x7f696815b2a0 2023-10-19T13:04:30.934 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.926+0000 7f695dffb640 1 --2- 172.21.15.57:0/3948403673 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f694c06afb0 0x7f694c06d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:30.934 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.926+0000 7f696d352640 1 -- 172.21.15.57:0/3948403673 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f694c06afb0 msgr2=0x7f694c06d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/2346394725 2023-10-19T13:04:30.935 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.926+0000 7f695dffb640 1 -- 172.21.15.57:0/3948403673 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f6960089fa0 con 0x7f696815b2a0 2023-10-19T13:04:30.935 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.926+0000 7f696d352640 1 --2- 172.21.15.57:0/3948403673 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f694c06afb0 0x7f694c06d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T13:04:30.935 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:30.930+0000 7f695dffb640 1 -- 172.21.15.57:0/3948403673 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6960054e80 con 0x7f696815b2a0 2023-10-19T13:04:31.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:31.127+0000 7f696d352640 1 -- 172.21.15.57:0/3948403673 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f694c06afb0 msgr2=0x7f694c06d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/2346394725 2023-10-19T13:04:31.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:31.127+0000 7f696d352640 1 --2- 172.21.15.57:0/3948403673 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f694c06afb0 0x7f694c06d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-19T13:04:31.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:31.143+0000 7f696f5b6640 1 -- 172.21.15.57:0/3948403673 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f69680690e0 con 0x7f696815b2a0 2023-10-19T13:04:31.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:31.148+0000 7f695dffb640 1 -- 172.21.15.57:0/3948403673 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v57)=0 v57) v1 ==== 133+0+0 (secure 0 0 0) 0x7f69600584a0 con 0x7f696815b2a0 2023-10-19T13:04:31.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:31.150+0000 7f696f5b6640 1 -- 172.21.15.57:0/3948403673 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f694c06afb0 msgr2=0x7f694c06d470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T13:04:31.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:31.150+0000 7f696f5b6640 1 --2- 172.21.15.57:0/3948403673 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f694c06afb0 0x7f694c06d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:31.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:31.150+0000 7f696f5b6640 1 -- 172.21.15.57:0/3948403673 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f696815b2a0 msgr2=0x7f69680b3eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:31.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:31.150+0000 7f696f5b6640 1 --2- 172.21.15.57:0/3948403673 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f696815b2a0 0x7f69680b3eb0 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7f696000cb40 tx=0x7f696000a060 comp rx=0 tx=0).stop 2023-10-19T13:04:31.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:31.150+0000 7f696f5b6640 1 -- 172.21.15.57:0/3948403673 shutdown_connections 2023-10-19T13:04:31.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:31.150+0000 7f696f5b6640 1 --2- 172.21.15.57:0/3948403673 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f694c06afb0 0x7f694c06d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:31.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:31.150+0000 7f696f5b6640 1 --2- 172.21.15.57:0/3948403673 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f696815b2a0 0x7f69680b3eb0 unknown :-1 s=CLOSED pgs=1625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:31.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:31.150+0000 7f696f5b6640 1 --2- 172.21.15.57:0/3948403673 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f69680cf340 0x7f6968157eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:31.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:31.150+0000 7f696f5b6640 1 --2- 172.21.15.57:0/3948403673 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f69680ceb40 0x7f696815e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:31.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:31.150+0000 7f696f5b6640 1 -- 172.21.15.57:0/3948403673 >> 172.21.15.57:0/3948403673 conn(0x7f69680d0d60 msgr2=0x7f696815a110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:31.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:31.150+0000 7f696f5b6640 1 -- 172.21.15.57:0/3948403673 shutdown_connections 2023-10-19T13:04:31.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:31.150+0000 7f696f5b6640 1 -- 172.21.15.57:0/3948403673 wait complete. 2023-10-19T13:04:31.164 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:04:31.164 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/minutes) 2023-10-19T13:04:31.186 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/minutes 1m 2023-10-19T13:04:31.289 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:31.287+0000 7f3f66c0f040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-19T13:04:31.333 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:31.330+0000 7f3f66c0f040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-19T13:04:31.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:31.339+0000 7f32ff9fc640 1 -- 172.21.15.57:0/1145914789 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f32f80cf340 msgr2=0x7f32f80d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:31.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:31.339+0000 7f32ff9fc640 1 --2- 172.21.15.57:0/1145914789 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f32f80cf340 0x7f32f80d1910 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7f32f4009830 tx=0x7f32f402bd60 comp rx=0 tx=0).stop 2023-10-19T13:04:31.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:31.339+0000 7f32ff9fc640 1 -- 172.21.15.57:0/1145914789 shutdown_connections 2023-10-19T13:04:31.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:31.339+0000 7f32ff9fc640 1 --2- 172.21.15.57:0/1145914789 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f32f80cf340 0x7f32f80d1910 unknown :-1 s=CLOSED pgs=1627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:31.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:31.339+0000 7f32ff9fc640 1 -- 172.21.15.57:0/1145914789 >> 172.21.15.57:0/1145914789 conn(0x7f32f80d0d60 msgr2=0x7f32f80d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:31.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:31.340+0000 7f32ff9fc640 1 -- 172.21.15.57:0/1145914789 shutdown_connections 2023-10-19T13:04:31.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:31.340+0000 7f32ff9fc640 1 -- 172.21.15.57:0/1145914789 wait complete. 2023-10-19T13:04:31.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:31.341+0000 7f32ff9fc640 1 Processor -- start 2023-10-19T13:04:31.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:31.341+0000 7f32ff9fc640 1 -- start start 2023-10-19T13:04:31.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:31.341+0000 7f32ff9fc640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f32f80ceb40 0x7f32f8159610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:31.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:31.341+0000 7f32ff9fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f32f80cf340 0x7f32f81e6020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:31.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:31.341+0000 7f32ff9fc640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f32f815b2a0 0x7f32f80b2850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:31.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:31.341+0000 7f32ff9fc640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32f80c9fc0 con 0x7f32f80cf340 2023-10-19T13:04:31.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:31.341+0000 7f32ff9fc640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32f80c8f20 con 0x7f32f815b2a0 2023-10-19T13:04:31.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:31.341+0000 7f32ff9fc640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32f80c9200 con 0x7f32f80ceb40 2023-10-19T13:04:31.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:31.342+0000 7f32fcf97640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f32f80cf340 0x7f32f81e6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:31.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:31.342+0000 7f32fcf97640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f32f80cf340 0x7f32f81e6020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46238/0 (socket says 172.21.15.57:46238) 2023-10-19T13:04:31.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:31.342+0000 7f32fd798640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f32f80ceb40 0x7f32f8159610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:31.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:31.342+0000 7f32fcf97640 1 -- 172.21.15.57:0/1657694187 learned_addr learned my addr 172.21.15.57:0/1657694187 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:31.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:31.342+0000 7f32fdf99640 1 --2- 172.21.15.57:0/1657694187 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f32f815b2a0 0x7f32f80b2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:31.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:31.342+0000 7f32fcf97640 1 -- 172.21.15.57:0/1657694187 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f32f80ceb40 msgr2=0x7f32f8159610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:31.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:31.342+0000 7f32fcf97640 1 --2- 172.21.15.57:0/1657694187 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f32f80ceb40 0x7f32f8159610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:31.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:31.342+0000 7f32fcf97640 1 -- 172.21.15.57:0/1657694187 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f32f815b2a0 msgr2=0x7f32f80b2850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:31.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:31.342+0000 7f32fcf97640 1 --2- 172.21.15.57:0/1657694187 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f32f815b2a0 0x7f32f80b2850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:31.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:31.342+0000 7f32fcf97640 1 -- 172.21.15.57:0/1657694187 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32f4009490 con 0x7f32f80cf340 2023-10-19T13:04:31.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:31.342+0000 7f32fcf97640 1 --2- 172.21.15.57:0/1657694187 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f32f80cf340 0x7f32f81e6020 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f32e800c400 tx=0x7f32e800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:31.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:31.342+0000 7f32edffb640 1 -- 172.21.15.57:0/1657694187 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32e80052d0 con 0x7f32f80cf340 2023-10-19T13:04:31.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:31.342+0000 7f32edffb640 1 -- 172.21.15.57:0/1657694187 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f32e80035f0 con 0x7f32f80cf340 2023-10-19T13:04:31.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:31.342+0000 7f32ff9fc640 1 -- 172.21.15.57:0/1657694187 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32f80c94e0 con 0x7f32f80cf340 2023-10-19T13:04:31.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:31.342+0000 7f32edffb640 1 -- 172.21.15.57:0/1657694187 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32e8003a00 con 0x7f32f80cf340 2023-10-19T13:04:31.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:31.343+0000 7f32ff9fc640 1 -- 172.21.15.57:0/1657694187 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32f80c9aa0 con 0x7f32f80cf340 2023-10-19T13:04:31.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:31.343+0000 7f32ff9fc640 1 -- 172.21.15.57:0/1657694187 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32f80690e0 con 0x7f32f80cf340 2023-10-19T13:04:31.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:31.344+0000 7f32edffb640 1 -- 172.21.15.57:0/1657694187 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 70) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f32e8003b60 con 0x7f32f80cf340 2023-10-19T13:04:31.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:31.345+0000 7f32edffb640 1 --2- 172.21.15.57:0/1657694187 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f32dc06afb0 0x7f32dc06d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:31.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:31.345+0000 7f32fd798640 1 -- 172.21.15.57:0/1657694187 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f32dc06afb0 msgr2=0x7f32dc06d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/2346394725 2023-10-19T13:04:31.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:31.345+0000 7f32edffb640 1 -- 172.21.15.57:0/1657694187 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f32e80890a0 con 0x7f32f80cf340 2023-10-19T13:04:31.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:31.345+0000 7f32fd798640 1 --2- 172.21.15.57:0/1657694187 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f32dc06afb0 0x7f32dc06d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T13:04:31.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:31.349+0000 7f32edffb640 1 -- 172.21.15.57:0/1657694187 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f32e8053f90 con 0x7f32f80cf340 2023-10-19T13:04:31.373 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:04:31.370+0000 7f6453a2f040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-19T13:04:31.376 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:31.373+0000 7f3f66c0f040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-19T13:04:31.420 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:04:31.417+0000 7f6453a2f040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-19T13:04:31.427 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:31.424+0000 7f3f66c0f040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-19T13:04:31.465 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:04:31.462+0000 7f6453a2f040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-19T13:04:31.468 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:31.465+0000 7f3f66c0f040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-19T13:04:31.520 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:04:31.517+0000 7f6453a2f040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-19T13:04:31.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:31.546+0000 7f32fd798640 1 -- 172.21.15.57:0/1657694187 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f32dc06afb0 msgr2=0x7f32dc06d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/2346394725 2023-10-19T13:04:31.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:31.546+0000 7f32fd798640 1 --2- 172.21.15.57:0/1657694187 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f32dc06afb0 0x7f32dc06d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-19T13:04:31.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:31.550+0000 7f32ff9fc640 1 -- 172.21.15.57:0/1657694187 --> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/minutes", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f32f80cf820 con 0x7f32dc06afb0 2023-10-19T13:04:31.563 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:04:31.561+0000 7f6453a2f040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-19T13:04:31.566 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:31.563+0000 7f3f66c0f040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-19T13:04:31.618 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:31.615+0000 7f3f66c0f040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-19T13:04:31.659 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:31.657+0000 7f3f66c0f040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-19T13:04:31.667 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:04:31.664+0000 7f6453a2f040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-19T13:04:31.707 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:31.704+0000 7f3f66c0f040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-19T13:04:31.722 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:04:31.719+0000 7f6453a2f040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-19T13:04:31.749 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:31.746+0000 7f3f66c0f040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-19T13:04:31.766 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:04:31.763+0000 7f6453a2f040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-19T13:04:31.816 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:04:31.813+0000 7f6453a2f040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-19T13:04:31.860 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:04:31.857+0000 7f6453a2f040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-19T13:04:31.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:31.947+0000 7f32fd798640 1 -- 172.21.15.57:0/1657694187 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f32dc06afb0 msgr2=0x7f32dc06d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/2346394725 2023-10-19T13:04:31.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:31.947+0000 7f32fd798640 1 --2- 172.21.15.57:0/1657694187 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f32dc06afb0 0x7f32dc06d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-19T13:04:32.061 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:32.058+0000 7f3f66c0f040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-19T13:04:32.189 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:32.186+0000 7f3f66c0f040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-19T13:04:32.191 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:04:32.188+0000 7f6453a2f040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-19T13:04:32.280 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:32.277+0000 7f3f66c0f040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-19T13:04:32.321 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:32.319+0000 7f3f66c0f040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-19T13:04:32.329 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:04:32.327+0000 7f6453a2f040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-19T13:04:32.416 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:32.413+0000 7f3f66c0f040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-19T13:04:32.425 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:04:32.423+0000 7f6453a2f040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-19T13:04:32.469 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:04:32.467+0000 7f6453a2f040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-19T13:04:32.504 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:32.502+0000 7f3f66c0f040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-19T13:04:32.569 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:04:32.566+0000 7f6453a2f040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-19T13:04:32.639 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:32.636+0000 7f3f66c0f040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-19T13:04:32.663 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:04:32.661+0000 7f6453a2f040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-19T13:04:32.680 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:32.677+0000 7f3f66c0f040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-19T13:04:32.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:32.748+0000 7f32fd798640 1 -- 172.21.15.57:0/1657694187 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f32dc06afb0 msgr2=0x7f32dc06d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/2346394725 2023-10-19T13:04:32.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:32.748+0000 7f32fd798640 1 --2- 172.21.15.57:0/1657694187 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f32dc06afb0 0x7f32dc06d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-19T13:04:32.784 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:32.781+0000 7f3f66c0f040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-19T13:04:32.806 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:04:32.804+0000 7f6453a2f040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-19T13:04:32.850 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:04:32.847+0000 7f6453a2f040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-19T13:04:32.959 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:04:32.956+0000 7f6453a2f040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-19T13:04:33.397 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:33.395+0000 7f3f66c0f040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-19T13:04:33.604 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:04:33.602+0000 7f6453a2f040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-19T13:04:33.657 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:33.654+0000 7f3f66c0f040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-19T13:04:33.705 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:33.702+0000 7f3f66c0f040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-19T13:04:33.805 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:33.802+0000 7f3f66c0f040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-19T13:04:33.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:33.824+0000 7f32edffb640 1 -- 172.21.15.57:0/1657694187 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mgrmap(e 71) v1 ==== 88957+0+0 (secure 0 0 0) 0x7f32e8087460 con 0x7f32f80cf340 2023-10-19T13:04:33.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:33.824+0000 7f32edffb640 1 -- 172.21.15.57:0/1657694187 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f32dc06afb0 msgr2=0x7f32dc06d470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T13:04:33.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:33.824+0000 7f32edffb640 1 --2- 172.21.15.57:0/1657694187 >> [v2:172.21.15.179:6832/2346394725,v1:172.21.15.179:6833/2346394725] conn(0x7f32dc06afb0 0x7f32dc06d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:33.884 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:04:33.882+0000 7f6453a2f040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-19T13:04:33.922 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:33.919+0000 7f3ee329b640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:04:33.922 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:33.919+0000 7f3ee329b640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:04:33.922 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:33.919+0000 7f3ee329b640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:04:33.922 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:33.919+0000 7f3ee329b640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:04:33.923 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:33.919+0000 7f3ee329b640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:04:33.923 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:33.919+0000 7f3ee62e1640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:04:33.923 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:33.919+0000 7f3ee62e1640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:04:33.923 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:33.919+0000 7f3ee62e1640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:04:33.923 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:33.919+0000 7f3ee62e1640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:04:33.924 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:33.919+0000 7f3ee62e1640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:04:33.938 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:04:33.935+0000 7f6453a2f040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-19T13:04:34.042 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:04:34.040+0000 7f6453a2f040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-19T13:04:34.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:34.830+0000 7f32edffb640 1 -- 172.21.15.57:0/1657694187 <== mon.0 v2:172.21.15.57:3300/0 8 ==== mgrmap(e 72) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f32e8054360 con 0x7f32f80cf340 2023-10-19T13:04:34.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:34.830+0000 7f32edffb640 1 --2- 172.21.15.57:0/1657694187 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7f32dc035850 0x7f32dc0753e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:34.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:34.830+0000 7f32edffb640 1 -- 172.21.15.57:0/1657694187 --> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/minutes", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f32f80cf820 con 0x7f32dc035850 2023-10-19T13:04:34.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:34.832+0000 7f32fd798640 1 --2- 172.21.15.57:0/1657694187 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7f32dc035850 0x7f32dc0753e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:34.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:34.835+0000 7f32fd798640 1 --2- 172.21.15.57:0/1657694187 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7f32dc035850 0x7f32dc0753e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f32f4036000 tx=0x7f32f4004730 comp rx=0 tx=0).ready entity=mgr.8299 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:34.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:34.889+0000 7f32edffb640 1 -- 172.21.15.57:0/1657694187 <== mgr.8299 v2:172.21.15.179:6832/3526956956 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+45 (secure 0 0 0) 0x7f32f80cf820 con 0x7f32dc035850 2023-10-19T13:04:34.892 INFO:teuthology.orchestra.run.smithi057.stdout:Schedule set for path snap_test_dir1/minutes 2023-10-19T13:04:34.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:34.891+0000 7f32ff9fc640 1 -- 172.21.15.57:0/1657694187 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7f32dc035850 msgr2=0x7f32dc0753e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:34.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:34.891+0000 7f32ff9fc640 1 --2- 172.21.15.57:0/1657694187 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7f32dc035850 0x7f32dc0753e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f32f4036000 tx=0x7f32f4004730 comp rx=0 tx=0).stop 2023-10-19T13:04:34.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:34.891+0000 7f32ff9fc640 1 -- 172.21.15.57:0/1657694187 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f32f80cf340 msgr2=0x7f32f81e6020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:34.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:34.891+0000 7f32ff9fc640 1 --2- 172.21.15.57:0/1657694187 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f32f80cf340 0x7f32f81e6020 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f32e800c400 tx=0x7f32e800c8c0 comp rx=0 tx=0).stop 2023-10-19T13:04:34.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:34.891+0000 7f32fdf99640 1 -- 172.21.15.57:0/1657694187 reap_dead start 2023-10-19T13:04:34.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:34.891+0000 7f32ff9fc640 1 -- 172.21.15.57:0/1657694187 shutdown_connections 2023-10-19T13:04:34.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:34.891+0000 7f32ff9fc640 1 -- 172.21.15.57:0/1657694187 >> 172.21.15.57:0/1657694187 conn(0x7f32f80d0d60 msgr2=0x7f32f815d6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:34.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:34.891+0000 7f32ff9fc640 1 -- 172.21.15.57:0/1657694187 shutdown_connections 2023-10-19T13:04:34.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:34.891+0000 7f32ff9fc640 1 -- 172.21.15.57:0/1657694187 wait complete. 2023-10-19T13:04:34.906 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/minutes 2023-10-19T13:04:34.907 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:04:34.907 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/hourly) 2023-10-19T13:04:34.926 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/hourly 1h 2023-10-19T13:04:35.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.077+0000 7f77231a0640 1 -- 172.21.15.57:0/1176040872 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f771c0ce2f0 msgr2=0x7f771c0c5550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:35.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.077+0000 7f77231a0640 1 --2- 172.21.15.57:0/1176040872 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f771c0ce2f0 0x7f771c0c5550 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7f770c005fd0 tx=0x7f770c02c2e0 comp rx=0 tx=0).stop 2023-10-19T13:04:35.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.077+0000 7f77231a0640 1 -- 172.21.15.57:0/1176040872 shutdown_connections 2023-10-19T13:04:35.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.077+0000 7f77231a0640 1 --2- 172.21.15.57:0/1176040872 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f771c0ce2f0 0x7f771c0c5550 unknown :-1 s=CLOSED pgs=1629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:35.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.077+0000 7f77231a0640 1 --2- 172.21.15.57:0/1176040872 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f771c0cda60 0x7f771c0c5010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:35.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.077+0000 7f77231a0640 1 -- 172.21.15.57:0/1176040872 >> 172.21.15.57:0/1176040872 conn(0x7f771c0d0c20 msgr2=0x7f771c0d1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:35.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.077+0000 7f77231a0640 1 -- 172.21.15.57:0/1176040872 shutdown_connections 2023-10-19T13:04:35.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.077+0000 7f77231a0640 1 -- 172.21.15.57:0/1176040872 wait complete. 2023-10-19T13:04:35.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.079+0000 7f77231a0640 1 Processor -- start 2023-10-19T13:04:35.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.079+0000 7f77231a0640 1 -- start start 2023-10-19T13:04:35.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.079+0000 7f77231a0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f771c0cda60 0x7f771c0b6570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:35.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.079+0000 7f77231a0640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f771c0ce2f0 0x7f771c0b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:35.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.079+0000 7f77231a0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f771c15b2f0 0x7f771c155d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:35.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.079+0000 7f77231a0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f771c0b4940 con 0x7f771c0cda60 2023-10-19T13:04:35.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.079+0000 7f77231a0640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f771c15cbe0 con 0x7f771c15b2f0 2023-10-19T13:04:35.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.079+0000 7f77231a0640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f771c0b42f0 con 0x7f771c0ce2f0 2023-10-19T13:04:35.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.079+0000 7f772219e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f771c0cda60 0x7f771c0b6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:35.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.079+0000 7f772219e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f771c0cda60 0x7f771c0b6570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46254/0 (socket says 172.21.15.57:46254) 2023-10-19T13:04:35.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.079+0000 7f772219e640 1 -- 172.21.15.57:0/1004442430 learned_addr learned my addr 172.21.15.57:0/1004442430 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:35.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.080+0000 7f772299f640 1 --2- 172.21.15.57:0/1004442430 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f771c15b2f0 0x7f771c155d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:35.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.080+0000 7f772199d640 1 --2- 172.21.15.57:0/1004442430 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f771c0ce2f0 0x7f771c0b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:35.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.080+0000 7f772219e640 1 -- 172.21.15.57:0/1004442430 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f771c0ce2f0 msgr2=0x7f771c0b4dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:35.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.080+0000 7f772219e640 1 --2- 172.21.15.57:0/1004442430 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f771c0ce2f0 0x7f771c0b4dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:35.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.080+0000 7f772219e640 1 -- 172.21.15.57:0/1004442430 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f771c15b2f0 msgr2=0x7f771c155d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:35.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.080+0000 7f772219e640 1 --2- 172.21.15.57:0/1004442430 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f771c15b2f0 0x7f771c155d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:35.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.080+0000 7f772219e640 1 -- 172.21.15.57:0/1004442430 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f770c009900 con 0x7f771c0cda60 2023-10-19T13:04:35.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.080+0000 7f772299f640 1 --2- 172.21.15.57:0/1004442430 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f771c15b2f0 0x7f771c155d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:04:35.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.080+0000 7f772199d640 1 --2- 172.21.15.57:0/1004442430 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f771c0ce2f0 0x7f771c0b4dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:04:35.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.080+0000 7f772219e640 1 --2- 172.21.15.57:0/1004442430 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f771c0cda60 0x7f771c0b6570 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7f771400c410 tx=0x7f771400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:35.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.080+0000 7f771affd640 1 -- 172.21.15.57:0/1004442430 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77140035f0 con 0x7f771c0cda60 2023-10-19T13:04:35.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.080+0000 7f771affd640 1 -- 172.21.15.57:0/1004442430 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f7714003770 con 0x7f771c0cda60 2023-10-19T13:04:35.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.080+0000 7f77231a0640 1 -- 172.21.15.57:0/1004442430 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f771c0b45d0 con 0x7f771c0cda60 2023-10-19T13:04:35.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.080+0000 7f771affd640 1 -- 172.21.15.57:0/1004442430 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7714003b80 con 0x7f771c0cda60 2023-10-19T13:04:35.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.080+0000 7f77231a0640 1 -- 172.21.15.57:0/1004442430 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f771c156290 con 0x7f771c0cda60 2023-10-19T13:04:35.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.081+0000 7f77231a0640 1 -- 172.21.15.57:0/1004442430 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f771c066960 con 0x7f771c0cda60 2023-10-19T13:04:35.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.082+0000 7f771affd640 1 -- 172.21.15.57:0/1004442430 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 72) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f7714003ce0 con 0x7f771c0cda60 2023-10-19T13:04:35.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.082+0000 7f771affd640 1 --2- 172.21.15.57:0/1004442430 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7f770806ad10 0x7f770806d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:35.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.082+0000 7f771affd640 1 -- 172.21.15.57:0/1004442430 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f7714088e10 con 0x7f771c0cda60 2023-10-19T13:04:35.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.082+0000 7f772199d640 1 --2- 172.21.15.57:0/1004442430 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7f770806ad10 0x7f770806d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:35.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.083+0000 7f772199d640 1 --2- 172.21.15.57:0/1004442430 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7f770806ad10 0x7f770806d1d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f771c0b6500 tx=0x7f7710005f90 comp rx=0 tx=0).ready entity=mgr.8299 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:35.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.086+0000 7f771affd640 1 -- 172.21.15.57:0/1004442430 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7714053e30 con 0x7f771c0cda60 2023-10-19T13:04:35.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.284+0000 7f77231a0640 1 -- 172.21.15.57:0/1004442430 --> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/hourly", "snap_schedule": "1h", "target": ["mon-mgr", ""]}) v1 -- 0x7f771c0d1740 con 0x7f770806ad10 2023-10-19T13:04:35.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.332+0000 7f771affd640 1 -- 172.21.15.57:0/1004442430 <== mgr.8299 v2:172.21.15.179:6832/3526956956 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+44 (secure 0 0 0) 0x7f7710016070 con 0x7f770806ad10 2023-10-19T13:04:35.335 INFO:teuthology.orchestra.run.smithi057.stdout:Schedule set for path snap_test_dir1/hourly 2023-10-19T13:04:35.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.334+0000 7f77231a0640 1 -- 172.21.15.57:0/1004442430 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7f770806ad10 msgr2=0x7f770806d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:35.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.334+0000 7f77231a0640 1 --2- 172.21.15.57:0/1004442430 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7f770806ad10 0x7f770806d1d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f771c0b6500 tx=0x7f7710005f90 comp rx=0 tx=0).stop 2023-10-19T13:04:35.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.334+0000 7f77231a0640 1 -- 172.21.15.57:0/1004442430 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f771c0cda60 msgr2=0x7f771c0b6570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:35.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.334+0000 7f77231a0640 1 --2- 172.21.15.57:0/1004442430 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f771c0cda60 0x7f771c0b6570 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7f771400c410 tx=0x7f771400c8d0 comp rx=0 tx=0).stop 2023-10-19T13:04:35.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.334+0000 7f77231a0640 1 -- 172.21.15.57:0/1004442430 shutdown_connections 2023-10-19T13:04:35.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.334+0000 7f77231a0640 1 --2- 172.21.15.57:0/1004442430 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7f770806ad10 0x7f770806d1d0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:35.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.334+0000 7f77231a0640 1 --2- 172.21.15.57:0/1004442430 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f771c15b2f0 0x7f771c155d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:35.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.334+0000 7f77231a0640 1 --2- 172.21.15.57:0/1004442430 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f771c0ce2f0 0x7f771c0b4dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:35.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.334+0000 7f77231a0640 1 --2- 172.21.15.57:0/1004442430 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f771c0cda60 0x7f771c0b6570 unknown :-1 s=CLOSED pgs=1630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:35.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.334+0000 7f77231a0640 1 -- 172.21.15.57:0/1004442430 >> 172.21.15.57:0/1004442430 conn(0x7f771c0d0c20 msgr2=0x7f771c0ceb20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:35.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.334+0000 7f77231a0640 1 -- 172.21.15.57:0/1004442430 shutdown_connections 2023-10-19T13:04:35.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.334+0000 7f77231a0640 1 -- 172.21.15.57:0/1004442430 wait complete. 2023-10-19T13:04:35.348 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/hourly 2023-10-19T13:04:35.348 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:04:35.348 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/daily) 2023-10-19T13:04:35.365 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/daily 1d 2023-10-19T13:04:35.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.515+0000 7f23f3be7640 1 -- 172.21.15.57:0/3824827308 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f23d8004a80 msgr2=0x7f23d8004ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:35.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.515+0000 7f23f3be7640 1 --2- 172.21.15.57:0/3824827308 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f23d8004a80 0x7f23d8004ea0 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7f23e800c260 tx=0x7f23d8006900 comp rx=0 tx=0).stop 2023-10-19T13:04:35.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.515+0000 7f23f2184640 1 -- 172.21.15.57:0/3824827308 reap_dead start 2023-10-19T13:04:35.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.515+0000 7f23f3be7640 1 -- 172.21.15.57:0/3824827308 shutdown_connections 2023-10-19T13:04:35.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.515+0000 7f23f3be7640 1 -- 172.21.15.57:0/3824827308 >> 172.21.15.57:0/3824827308 conn(0x7f23ec069930 msgr2=0x7f23ec069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:35.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.515+0000 7f23f3be7640 1 -- 172.21.15.57:0/3824827308 shutdown_connections 2023-10-19T13:04:35.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.515+0000 7f23f3be7640 1 -- 172.21.15.57:0/3824827308 wait complete. 2023-10-19T13:04:35.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.516+0000 7f23f3be7640 1 Processor -- start 2023-10-19T13:04:35.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.516+0000 7f23f3be7640 1 -- start start 2023-10-19T13:04:35.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.517+0000 7f23f3be7640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f23d8004a80 0x7f23ec138ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:35.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.517+0000 7f23f3be7640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f23ec13bde0 0x7f23ec0ae760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:35.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.517+0000 7f23f3be7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f23ec0aeca0 0x7f23ec14d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:35.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.517+0000 7f23f3be7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23ec0af120 con 0x7f23ec0aeca0 2023-10-19T13:04:35.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.517+0000 7f23f3be7640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23ec139a70 con 0x7f23ec13bde0 2023-10-19T13:04:35.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.517+0000 7f23f3be7640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23ec139490 con 0x7f23d8004a80 2023-10-19T13:04:35.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.517+0000 7f23f2184640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f23ec0aeca0 0x7f23ec14d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:35.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.517+0000 7f23f2184640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f23ec0aeca0 0x7f23ec14d700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46280/0 (socket says 172.21.15.57:46280) 2023-10-19T13:04:35.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.517+0000 7f23f2184640 1 -- 172.21.15.57:0/2576220575 learned_addr learned my addr 172.21.15.57:0/2576220575 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:35.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.517+0000 7f23f1983640 1 --2- 172.21.15.57:0/2576220575 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f23d8004a80 0x7f23ec138ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:35.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.517+0000 7f23f1182640 1 --2- 172.21.15.57:0/2576220575 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f23ec13bde0 0x7f23ec0ae760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:35.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.518+0000 7f23f2184640 1 -- 172.21.15.57:0/2576220575 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f23d8004a80 msgr2=0x7f23ec138ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:35.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.518+0000 7f23f2184640 1 --2- 172.21.15.57:0/2576220575 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f23d8004a80 0x7f23ec138ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:35.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.518+0000 7f23f2184640 1 -- 172.21.15.57:0/2576220575 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f23ec13bde0 msgr2=0x7f23ec0ae760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:35.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.518+0000 7f23f2184640 1 --2- 172.21.15.57:0/2576220575 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f23ec13bde0 0x7f23ec0ae760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:35.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.518+0000 7f23f2184640 1 -- 172.21.15.57:0/2576220575 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23ec1390e0 con 0x7f23ec0aeca0 2023-10-19T13:04:35.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.518+0000 7f23f2184640 1 --2- 172.21.15.57:0/2576220575 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f23ec0aeca0 0x7f23ec14d700 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7f23e4005fd0 tx=0x7f23e400f3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:35.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.518+0000 7f23e27fc640 1 -- 172.21.15.57:0/2576220575 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e4004b20 con 0x7f23ec0aeca0 2023-10-19T13:04:35.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.518+0000 7f23e27fc640 1 -- 172.21.15.57:0/2576220575 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f23e4004c80 con 0x7f23ec0aeca0 2023-10-19T13:04:35.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.518+0000 7f23f3be7640 1 -- 172.21.15.57:0/2576220575 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23ec14af40 con 0x7f23ec0aeca0 2023-10-19T13:04:35.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.518+0000 7f23e27fc640 1 -- 172.21.15.57:0/2576220575 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e4003bd0 con 0x7f23ec0aeca0 2023-10-19T13:04:35.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.518+0000 7f23f3be7640 1 -- 172.21.15.57:0/2576220575 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23ec14ac40 con 0x7f23ec0aeca0 2023-10-19T13:04:35.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.519+0000 7f23f3be7640 1 -- 172.21.15.57:0/2576220575 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23ec0690e0 con 0x7f23ec0aeca0 2023-10-19T13:04:35.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.520+0000 7f23e27fc640 1 -- 172.21.15.57:0/2576220575 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 72) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f23e4003d30 con 0x7f23ec0aeca0 2023-10-19T13:04:35.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.520+0000 7f23e27fc640 1 --2- 172.21.15.57:0/2576220575 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7f23d006ace0 0x7f23d006d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:35.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.520+0000 7f23e27fc640 1 -- 172.21.15.57:0/2576220575 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f23e40898f0 con 0x7f23ec0aeca0 2023-10-19T13:04:35.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.520+0000 7f23f1983640 1 --2- 172.21.15.57:0/2576220575 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7f23d006ace0 0x7f23d006d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:35.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.521+0000 7f23f1983640 1 --2- 172.21.15.57:0/2576220575 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7f23d006ace0 0x7f23d006d1a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f23e8034000 tx=0x7f23e8001a00 comp rx=0 tx=0).ready entity=mgr.8299 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:35.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.524+0000 7f23e27fc640 1 -- 172.21.15.57:0/2576220575 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f23e4054a40 con 0x7f23ec0aeca0 2023-10-19T13:04:35.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.719+0000 7f23f3be7640 1 -- 172.21.15.57:0/2576220575 --> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/daily", "snap_schedule": "1d", "target": ["mon-mgr", ""]}) v1 -- 0x7f23ec0cc100 con 0x7f23d006ace0 2023-10-19T13:04:35.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.772+0000 7f23e27fc640 1 -- 172.21.15.57:0/2576220575 <== mgr.8299 v2:172.21.15.179:6832/3526956956 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f23e8006050 con 0x7f23d006ace0 2023-10-19T13:04:35.775 INFO:teuthology.orchestra.run.smithi057.stdout:Schedule set for path snap_test_dir1/daily 2023-10-19T13:04:35.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.774+0000 7f23f3be7640 1 -- 172.21.15.57:0/2576220575 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7f23d006ace0 msgr2=0x7f23d006d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:35.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.774+0000 7f23f3be7640 1 --2- 172.21.15.57:0/2576220575 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7f23d006ace0 0x7f23d006d1a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f23e8034000 tx=0x7f23e8001a00 comp rx=0 tx=0).stop 2023-10-19T13:04:35.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.774+0000 7f23f3be7640 1 -- 172.21.15.57:0/2576220575 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f23ec0aeca0 msgr2=0x7f23ec14d700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:35.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.774+0000 7f23f3be7640 1 --2- 172.21.15.57:0/2576220575 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f23ec0aeca0 0x7f23ec14d700 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7f23e4005fd0 tx=0x7f23e400f3c0 comp rx=0 tx=0).stop 2023-10-19T13:04:35.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.774+0000 7f23f3be7640 1 -- 172.21.15.57:0/2576220575 shutdown_connections 2023-10-19T13:04:35.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.774+0000 7f23f3be7640 1 --2- 172.21.15.57:0/2576220575 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7f23d006ace0 0x7f23d006d1a0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:35.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.774+0000 7f23f3be7640 1 --2- 172.21.15.57:0/2576220575 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f23ec0aeca0 0x7f23ec14d700 unknown :-1 s=CLOSED pgs=1633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:35.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.774+0000 7f23f3be7640 1 --2- 172.21.15.57:0/2576220575 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f23ec13bde0 0x7f23ec0ae760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:35.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.774+0000 7f23f3be7640 1 --2- 172.21.15.57:0/2576220575 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f23d8004a80 0x7f23ec138ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:35.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.774+0000 7f23f3be7640 1 -- 172.21.15.57:0/2576220575 >> 172.21.15.57:0/2576220575 conn(0x7f23ec069930 msgr2=0x7f23ec159040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:35.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.774+0000 7f23f3be7640 1 -- 172.21.15.57:0/2576220575 shutdown_connections 2023-10-19T13:04:35.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.774+0000 7f23f3be7640 1 -- 172.21.15.57:0/2576220575 wait complete. 2023-10-19T13:04:35.787 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/daily 2023-10-19T13:04:35.788 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:04:35.788 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/weekly) 2023-10-19T13:04:35.805 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/weekly 1w 2023-10-19T13:04:35.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.955+0000 7fd702f4c640 1 -- 172.21.15.57:0/1799301268 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd6fc0ceb40 msgr2=0x7fd6fc0b0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:35.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.955+0000 7fd702f4c640 1 --2- 172.21.15.57:0/1799301268 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd6fc0ceb40 0x7fd6fc0b0650 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7fd6e4005fd0 tx=0x7fd6e402bcf0 comp rx=0 tx=0).stop 2023-10-19T13:04:35.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.955+0000 7fd702f4c640 1 -- 172.21.15.57:0/1799301268 shutdown_connections 2023-10-19T13:04:35.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.955+0000 7fd702f4c640 1 --2- 172.21.15.57:0/1799301268 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd6fc0ceb40 0x7fd6fc0b0650 unknown :-1 s=CLOSED pgs=1634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:35.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.955+0000 7fd702f4c640 1 --2- 172.21.15.57:0/1799301268 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd6fc15b2a0 0x7fd6fc0d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:35.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.955+0000 7fd702f4c640 1 -- 172.21.15.57:0/1799301268 >> 172.21.15.57:0/1799301268 conn(0x7fd6fc0d0d60 msgr2=0x7fd6fc0d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:35.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.955+0000 7fd702f4c640 1 -- 172.21.15.57:0/1799301268 shutdown_connections 2023-10-19T13:04:35.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.955+0000 7fd702f4c640 1 -- 172.21.15.57:0/1799301268 wait complete. 2023-10-19T13:04:35.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.956+0000 7fd702f4c640 1 Processor -- start 2023-10-19T13:04:35.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.957+0000 7fd702f4c640 1 -- start start 2023-10-19T13:04:35.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.957+0000 7fd702f4c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd6fc0ceb40 0x7fd6fc151290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:35.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.957+0000 7fd702f4c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd6fc0cf340 0x7fd6fc0c80d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:35.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.957+0000 7fd702f4c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd6fc15b2a0 0x7fd6fc158740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:35.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.957+0000 7fd702f4c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6fc0c96d0 con 0x7fd6fc0cf340 2023-10-19T13:04:35.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.957+0000 7fd702f4c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6fc0c8b50 con 0x7fd6fc0ceb40 2023-10-19T13:04:35.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.957+0000 7fd702f4c640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6fc0c8e30 con 0x7fd6fc15b2a0 2023-10-19T13:04:35.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.958+0000 7fd6fbfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd6fc0cf340 0x7fd6fc0c80d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:35.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.958+0000 7fd6fbfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd6fc0cf340 0x7fd6fc0c80d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46302/0 (socket says 172.21.15.57:46302) 2023-10-19T13:04:35.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.958+0000 7fd6fbfff640 1 -- 172.21.15.57:0/4292802634 learned_addr learned my addr 172.21.15.57:0/4292802634 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:35.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.958+0000 7fd7014e9640 1 --2- 172.21.15.57:0/4292802634 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd6fc15b2a0 0x7fd6fc158740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:35.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.958+0000 7fd700ce8640 1 --2- 172.21.15.57:0/4292802634 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd6fc0ceb40 0x7fd6fc151290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:35.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.958+0000 7fd6fbfff640 1 -- 172.21.15.57:0/4292802634 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd6fc15b2a0 msgr2=0x7fd6fc158740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:35.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.958+0000 7fd6fbfff640 1 --2- 172.21.15.57:0/4292802634 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd6fc15b2a0 0x7fd6fc158740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:35.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.958+0000 7fd6fbfff640 1 -- 172.21.15.57:0/4292802634 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd6fc0ceb40 msgr2=0x7fd6fc151290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:35.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.958+0000 7fd6fbfff640 1 --2- 172.21.15.57:0/4292802634 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd6fc0ceb40 0x7fd6fc151290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:35.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.958+0000 7fd6fbfff640 1 -- 172.21.15.57:0/4292802634 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6e4009560 con 0x7fd6fc0cf340 2023-10-19T13:04:35.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.958+0000 7fd6fbfff640 1 --2- 172.21.15.57:0/4292802634 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd6fc0cf340 0x7fd6fc0c80d0 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7fd6ec00c970 tx=0x7fd6ec00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:35.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.958+0000 7fd6f97fa640 1 -- 172.21.15.57:0/4292802634 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6ec007bb0 con 0x7fd6fc0cf340 2023-10-19T13:04:35.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.958+0000 7fd6f97fa640 1 -- 172.21.15.57:0/4292802634 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fd6ec007e00 con 0x7fd6fc0cf340 2023-10-19T13:04:35.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.958+0000 7fd702f4c640 1 -- 172.21.15.57:0/4292802634 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6fc0b0d50 con 0x7fd6fc0cf340 2023-10-19T13:04:35.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.958+0000 7fd6f97fa640 1 -- 172.21.15.57:0/4292802634 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6ec00f6b0 con 0x7fd6fc0cf340 2023-10-19T13:04:35.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.958+0000 7fd702f4c640 1 -- 172.21.15.57:0/4292802634 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6fc0c93f0 con 0x7fd6fc0cf340 2023-10-19T13:04:35.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.959+0000 7fd702f4c640 1 -- 172.21.15.57:0/4292802634 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6fc0690e0 con 0x7fd6fc0cf340 2023-10-19T13:04:35.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.960+0000 7fd6f97fa640 1 -- 172.21.15.57:0/4292802634 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 72) v1 ==== 89084+0+0 (secure 0 0 0) 0x7fd6ec00f810 con 0x7fd6fc0cf340 2023-10-19T13:04:35.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.960+0000 7fd6f97fa640 1 --2- 172.21.15.57:0/4292802634 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7fd6dc06ace0 0x7fd6dc06d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:35.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.960+0000 7fd6f97fa640 1 -- 172.21.15.57:0/4292802634 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6401+0+0 (secure 0 0 0) 0x7fd6ec089c90 con 0x7fd6fc0cf340 2023-10-19T13:04:35.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.960+0000 7fd700ce8640 1 --2- 172.21.15.57:0/4292802634 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7fd6dc06ace0 0x7fd6dc06d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:35.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.961+0000 7fd700ce8640 1 --2- 172.21.15.57:0/4292802634 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7fd6dc06ace0 0x7fd6dc06d1a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd6fc15ea00 tx=0x7fd6f4003d90 comp rx=0 tx=0).ready entity=mgr.8299 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:35.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:35.964+0000 7fd6f97fa640 1 -- 172.21.15.57:0/4292802634 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd6ec054df0 con 0x7fd6fc0cf340 2023-10-19T13:04:36.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.163+0000 7fd702f4c640 1 -- 172.21.15.57:0/4292802634 --> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/weekly", "snap_schedule": "1w", "target": ["mon-mgr", ""]}) v1 -- 0x7fd6fc0b0bd0 con 0x7fd6dc06ace0 2023-10-19T13:04:36.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.212+0000 7fd6f97fa640 1 -- 172.21.15.57:0/4292802634 <== mgr.8299 v2:172.21.15.179:6832/3526956956 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+44 (secure 0 0 0) 0x7fd6f40042a0 con 0x7fd6dc06ace0 2023-10-19T13:04:36.215 INFO:teuthology.orchestra.run.smithi057.stdout:Schedule set for path snap_test_dir1/weekly 2023-10-19T13:04:36.217 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.214+0000 7fd702f4c640 1 -- 172.21.15.57:0/4292802634 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7fd6dc06ace0 msgr2=0x7fd6dc06d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:36.217 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.214+0000 7fd702f4c640 1 --2- 172.21.15.57:0/4292802634 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7fd6dc06ace0 0x7fd6dc06d1a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd6fc15ea00 tx=0x7fd6f4003d90 comp rx=0 tx=0).stop 2023-10-19T13:04:36.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.214+0000 7fd702f4c640 1 -- 172.21.15.57:0/4292802634 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd6fc0cf340 msgr2=0x7fd6fc0c80d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:36.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.214+0000 7fd702f4c640 1 --2- 172.21.15.57:0/4292802634 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd6fc0cf340 0x7fd6fc0c80d0 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7fd6ec00c970 tx=0x7fd6ec00ce30 comp rx=0 tx=0).stop 2023-10-19T13:04:36.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.215+0000 7fd702f4c640 1 -- 172.21.15.57:0/4292802634 shutdown_connections 2023-10-19T13:04:36.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.215+0000 7fd702f4c640 1 --2- 172.21.15.57:0/4292802634 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7fd6dc06ace0 0x7fd6dc06d1a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:36.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.215+0000 7fd702f4c640 1 --2- 172.21.15.57:0/4292802634 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd6fc15b2a0 0x7fd6fc158740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:36.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.215+0000 7fd702f4c640 1 --2- 172.21.15.57:0/4292802634 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd6fc0cf340 0x7fd6fc0c80d0 unknown :-1 s=CLOSED pgs=1635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:36.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.215+0000 7fd702f4c640 1 --2- 172.21.15.57:0/4292802634 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd6fc0ceb40 0x7fd6fc151290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:36.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.215+0000 7fd702f4c640 1 -- 172.21.15.57:0/4292802634 >> 172.21.15.57:0/4292802634 conn(0x7fd6fc0d0d60 msgr2=0x7fd6fc15a260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:36.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.215+0000 7fd702f4c640 1 -- 172.21.15.57:0/4292802634 shutdown_connections 2023-10-19T13:04:36.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.215+0000 7fd702f4c640 1 -- 172.21.15.57:0/4292802634 wait complete. 2023-10-19T13:04:36.228 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/weekly 2023-10-19T13:04:36.228 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:04:36.229 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/monthly) 2023-10-19T13:04:36.246 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/monthly 1M 2023-10-19T13:04:36.398 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.395+0000 7f8c8b14c640 1 -- 172.21.15.57:0/1105054758 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8c8406ac70 msgr2=0x7f8c8406ca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:36.398 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.395+0000 7f8c8b14c640 1 --2- 172.21.15.57:0/1105054758 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8c8406ac70 0x7f8c8406ca90 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7f8c84071f00 tx=0x7f8c6c02bda0 comp rx=0 tx=0).stop 2023-10-19T13:04:36.398 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.395+0000 7f8c8b14c640 1 -- 172.21.15.57:0/1105054758 shutdown_connections 2023-10-19T13:04:36.398 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.395+0000 7f8c8b14c640 1 --2- 172.21.15.57:0/1105054758 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8c8406ac70 0x7f8c8406ca90 unknown :-1 s=CLOSED pgs=1636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:36.399 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.395+0000 7f8c8b14c640 1 --2- 172.21.15.57:0/1105054758 >> v2:172.21.15.179:3300/0 conn(0x7f8c8415c1f0 0x7f8c8406a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:36.399 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.395+0000 7f8c8b14c640 1 --2- 172.21.15.57:0/1105054758 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8c841579f0 0x7f8c840b0e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:36.399 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.395+0000 7f8c8b14c640 1 -- 172.21.15.57:0/1105054758 >> 172.21.15.57:0/1105054758 conn(0x7f8c8406fae0 msgr2=0x7f8c8406fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:36.399 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.395+0000 7f8c8b14c640 1 -- 172.21.15.57:0/1105054758 shutdown_connections 2023-10-19T13:04:36.399 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.395+0000 7f8c8b14c640 1 -- 172.21.15.57:0/1105054758 wait complete. 2023-10-19T13:04:36.401 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.396+0000 7f8c8b14c640 1 Processor -- start 2023-10-19T13:04:36.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.396+0000 7f8c8b14c640 1 -- start start 2023-10-19T13:04:36.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.397+0000 7f8c8b14c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8c8406ac70 0x7f8c8413cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:36.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.397+0000 7f8c8b14c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8c841579f0 0x7f8c84152410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:36.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.397+0000 7f8c8b14c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8c8415c1f0 0x7f8c84152950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:36.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.397+0000 7f8c8b14c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c84155120 con 0x7f8c8415c1f0 2023-10-19T13:04:36.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.397+0000 7f8c8b14c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c84163ff0 con 0x7f8c841579f0 2023-10-19T13:04:36.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.397+0000 7f8c8b14c640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c841642d0 con 0x7f8c8406ac70 2023-10-19T13:04:36.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.397+0000 7f8c896e9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8c8415c1f0 0x7f8c84152950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:36.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.397+0000 7f8c896e9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8c8415c1f0 0x7f8c84152950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46328/0 (socket says 172.21.15.57:46328) 2023-10-19T13:04:36.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.397+0000 7f8c896e9640 1 -- 172.21.15.57:0/2469790737 learned_addr learned my addr 172.21.15.57:0/2469790737 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:36.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.397+0000 7f8c88ee8640 1 --2- 172.21.15.57:0/2469790737 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8c8406ac70 0x7f8c8413cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:36.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.397+0000 7f8c83fff640 1 --2- 172.21.15.57:0/2469790737 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8c841579f0 0x7f8c84152410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:36.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.397+0000 7f8c896e9640 1 -- 172.21.15.57:0/2469790737 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8c8406ac70 msgr2=0x7f8c8413cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:36.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.397+0000 7f8c896e9640 1 --2- 172.21.15.57:0/2469790737 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8c8406ac70 0x7f8c8413cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:36.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.397+0000 7f8c896e9640 1 -- 172.21.15.57:0/2469790737 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8c841579f0 msgr2=0x7f8c84152410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:36.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.397+0000 7f8c896e9640 1 --2- 172.21.15.57:0/2469790737 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8c841579f0 0x7f8c84152410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:36.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.397+0000 7f8c896e9640 1 -- 172.21.15.57:0/2469790737 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c6c009560 con 0x7f8c8415c1f0 2023-10-19T13:04:36.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.397+0000 7f8c88ee8640 1 --2- 172.21.15.57:0/2469790737 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8c8406ac70 0x7f8c8413cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:04:36.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.398+0000 7f8c896e9640 1 --2- 172.21.15.57:0/2469790737 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8c8415c1f0 0x7f8c84152950 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7f8c6c036040 tx=0x7f8c6c02dac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:36.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.398+0000 7f8c817fa640 1 -- 172.21.15.57:0/2469790737 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c6c003800 con 0x7f8c8415c1f0 2023-10-19T13:04:36.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.398+0000 7f8c817fa640 1 -- 172.21.15.57:0/2469790737 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f8c6c003a50 con 0x7f8c8415c1f0 2023-10-19T13:04:36.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.398+0000 7f8c8b14c640 1 -- 172.21.15.57:0/2469790737 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c840b2330 con 0x7f8c8415c1f0 2023-10-19T13:04:36.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.398+0000 7f8c817fa640 1 -- 172.21.15.57:0/2469790737 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c6c003e60 con 0x7f8c8415c1f0 2023-10-19T13:04:36.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.398+0000 7f8c8b14c640 1 -- 172.21.15.57:0/2469790737 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c841553c0 con 0x7f8c8415c1f0 2023-10-19T13:04:36.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.399+0000 7f8c8b14c640 1 -- 172.21.15.57:0/2469790737 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c841555b0 con 0x7f8c8415c1f0 2023-10-19T13:04:36.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.399+0000 7f8c817fa640 1 -- 172.21.15.57:0/2469790737 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 72) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f8c6c03e450 con 0x7f8c8415c1f0 2023-10-19T13:04:36.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.400+0000 7f8c817fa640 1 --2- 172.21.15.57:0/2469790737 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7f8c6406ace0 0x7f8c6406d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:36.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.400+0000 7f8c817fa640 1 -- 172.21.15.57:0/2469790737 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f8c6c0ac0b0 con 0x7f8c8415c1f0 2023-10-19T13:04:36.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.400+0000 7f8c88ee8640 1 --2- 172.21.15.57:0/2469790737 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7f8c6406ace0 0x7f8c6406d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:36.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.400+0000 7f8c88ee8640 1 --2- 172.21.15.57:0/2469790737 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7f8c6406ace0 0x7f8c6406d1a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c005fd0 tx=0x7f8c7c009500 comp rx=0 tx=0).ready entity=mgr.8299 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:36.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.404+0000 7f8c817fa640 1 -- 172.21.15.57:0/2469790737 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8c6c077280 con 0x7f8c8415c1f0 2023-10-19T13:04:36.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.599+0000 7f8c8b14c640 1 -- 172.21.15.57:0/2469790737 --> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/monthly", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f8c840b1d80 con 0x7f8c6406ace0 2023-10-19T13:04:36.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.645+0000 7f8c817fa640 1 -- 172.21.15.57:0/2469790737 <== mgr.8299 v2:172.21.15.179:6832/3526956956 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+45 (secure 0 0 0) 0x7f8c840b1d80 con 0x7f8c6406ace0 2023-10-19T13:04:36.648 INFO:teuthology.orchestra.run.smithi057.stdout:Schedule set for path snap_test_dir1/monthly 2023-10-19T13:04:36.650 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.647+0000 7f8c8b14c640 1 -- 172.21.15.57:0/2469790737 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7f8c6406ace0 msgr2=0x7f8c6406d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:36.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.647+0000 7f8c8b14c640 1 --2- 172.21.15.57:0/2469790737 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7f8c6406ace0 0x7f8c6406d1a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c005fd0 tx=0x7f8c7c009500 comp rx=0 tx=0).stop 2023-10-19T13:04:36.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.647+0000 7f8c8b14c640 1 -- 172.21.15.57:0/2469790737 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8c8415c1f0 msgr2=0x7f8c84152950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:36.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.647+0000 7f8c8b14c640 1 --2- 172.21.15.57:0/2469790737 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8c8415c1f0 0x7f8c84152950 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7f8c6c036040 tx=0x7f8c6c02dac0 comp rx=0 tx=0).stop 2023-10-19T13:04:36.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.647+0000 7f8c8b14c640 1 -- 172.21.15.57:0/2469790737 shutdown_connections 2023-10-19T13:04:36.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.647+0000 7f8c8b14c640 1 --2- 172.21.15.57:0/2469790737 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7f8c6406ace0 0x7f8c6406d1a0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:36.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.647+0000 7f8c8b14c640 1 --2- 172.21.15.57:0/2469790737 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8c8415c1f0 0x7f8c84152950 unknown :-1 s=CLOSED pgs=1637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:36.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.647+0000 7f8c8b14c640 1 --2- 172.21.15.57:0/2469790737 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8c841579f0 0x7f8c84152410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:36.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.647+0000 7f8c8b14c640 1 --2- 172.21.15.57:0/2469790737 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8c8406ac70 0x7f8c8413cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:36.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.647+0000 7f8c8b14c640 1 -- 172.21.15.57:0/2469790737 >> 172.21.15.57:0/2469790737 conn(0x7f8c8406fae0 msgr2=0x7f8c84157dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:36.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.648+0000 7f8c8b14c640 1 -- 172.21.15.57:0/2469790737 shutdown_connections 2023-10-19T13:04:36.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.648+0000 7f8c8b14c640 1 -- 172.21.15.57:0/2469790737 wait complete. 2023-10-19T13:04:36.661 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/monthly 2023-10-19T13:04:36.661 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:04:36.662 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/yearly) 2023-10-19T13:04:36.679 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/yearly 1Y 2023-10-19T13:04:36.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.829+0000 7fae71675640 1 -- 172.21.15.57:0/847689523 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fae6c15c1f0 msgr2=0x7fae6c06a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:36.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.829+0000 7fae71675640 1 --2- 172.21.15.57:0/847689523 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fae6c15c1f0 0x7fae6c06a730 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7fae60009830 tx=0x7fae6002bd60 comp rx=0 tx=0).stop 2023-10-19T13:04:36.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.829+0000 7fae71675640 1 -- 172.21.15.57:0/847689523 shutdown_connections 2023-10-19T13:04:36.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.829+0000 7fae71675640 1 --2- 172.21.15.57:0/847689523 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fae6c15c1f0 0x7fae6c06a730 unknown :-1 s=CLOSED pgs=1639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:36.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.829+0000 7fae71675640 1 --2- 172.21.15.57:0/847689523 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fae6c1579f0 0x7fae6c0b0e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:36.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.829+0000 7fae71675640 1 -- 172.21.15.57:0/847689523 >> 172.21.15.57:0/847689523 conn(0x7fae6c06fae0 msgr2=0x7fae6c06fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:36.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.829+0000 7fae71675640 1 -- 172.21.15.57:0/847689523 shutdown_connections 2023-10-19T13:04:36.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.829+0000 7fae71675640 1 -- 172.21.15.57:0/847689523 wait complete. 2023-10-19T13:04:36.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.831+0000 7fae71675640 1 Processor -- start 2023-10-19T13:04:36.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.831+0000 7fae71675640 1 -- start start 2023-10-19T13:04:36.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.832+0000 7fae71675640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fae6c06ac70 0x7fae6c13fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:36.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.832+0000 7fae71675640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fae6c1579f0 0x7fae6c0ae1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:36.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.832+0000 7fae71675640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fae6c15c1f0 0x7fae6c155980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:36.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.832+0000 7fae71675640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae6c140290 con 0x7fae6c1579f0 2023-10-19T13:04:36.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.832+0000 7fae71675640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae6c154f30 con 0x7fae6c06ac70 2023-10-19T13:04:36.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.832+0000 7fae71675640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae6c1553c0 con 0x7fae6c15c1f0 2023-10-19T13:04:36.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.833+0000 7fae6a7fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fae6c1579f0 0x7fae6c0ae1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:36.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.833+0000 7fae6a7fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fae6c1579f0 0x7fae6c0ae1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57238/0 (socket says 172.21.15.57:57238) 2023-10-19T13:04:36.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.833+0000 7fae6a7fc640 1 -- 172.21.15.57:0/2845200749 learned_addr learned my addr 172.21.15.57:0/2845200749 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:36.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.833+0000 7fae6a7fc640 1 -- 172.21.15.57:0/2845200749 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fae6c15c1f0 msgr2=0x7fae6c155980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:36.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.833+0000 7fae6a7fc640 1 --2- 172.21.15.57:0/2845200749 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fae6c15c1f0 0x7fae6c155980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:36.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.833+0000 7fae6a7fc640 1 -- 172.21.15.57:0/2845200749 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fae6c06ac70 msgr2=0x7fae6c13fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:36.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.833+0000 7fae6a7fc640 1 --2- 172.21.15.57:0/2845200749 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fae6c06ac70 0x7fae6c13fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:36.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.833+0000 7fae6a7fc640 1 -- 172.21.15.57:0/2845200749 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae60009490 con 0x7fae6c1579f0 2023-10-19T13:04:36.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.833+0000 7fae6a7fc640 1 --2- 172.21.15.57:0/2845200749 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fae6c1579f0 0x7fae6c0ae1f0 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7fae60009780 tx=0x7fae600038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:36.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.834+0000 7fae537fe640 1 -- 172.21.15.57:0/2845200749 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae60003ea0 con 0x7fae6c1579f0 2023-10-19T13:04:36.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.834+0000 7fae537fe640 1 -- 172.21.15.57:0/2845200749 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fae60034440 con 0x7fae6c1579f0 2023-10-19T13:04:36.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.834+0000 7fae71675640 1 -- 172.21.15.57:0/2845200749 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae6c154320 con 0x7fae6c1579f0 2023-10-19T13:04:36.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.834+0000 7fae537fe640 1 -- 172.21.15.57:0/2845200749 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae60034850 con 0x7fae6c1579f0 2023-10-19T13:04:36.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.834+0000 7fae71675640 1 -- 172.21.15.57:0/2845200749 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae6c1548e0 con 0x7fae6c1579f0 2023-10-19T13:04:36.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.835+0000 7fae71675640 1 -- 172.21.15.57:0/2845200749 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae6c0690e0 con 0x7fae6c1579f0 2023-10-19T13:04:36.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.836+0000 7fae537fe640 1 -- 172.21.15.57:0/2845200749 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 73) v1 ==== 89349+0+0 (secure 0 0 0) 0x7fae6003b040 con 0x7fae6c1579f0 2023-10-19T13:04:36.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.836+0000 7fae537fe640 1 --2- 172.21.15.57:0/2845200749 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7fae5806afb0 0x7fae5806d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:36.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.836+0000 7fae537fe640 1 -- 172.21.15.57:0/2845200749 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6401+0+0 (secure 0 0 0) 0x7fae600ac000 con 0x7fae6c1579f0 2023-10-19T13:04:36.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.836+0000 7fae6affd640 1 --2- 172.21.15.57:0/2845200749 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7fae5806afb0 0x7fae5806d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:36.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.837+0000 7fae6affd640 1 --2- 172.21.15.57:0/2845200749 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7fae5806afb0 0x7fae5806d470 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fae640058c0 tx=0x7fae64005850 comp rx=0 tx=0).ready entity=mgr.8299 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:36.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:36.843+0000 7fae537fe640 1 -- 172.21.15.57:0/2845200749 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fae60076c50 con 0x7fae6c1579f0 2023-10-19T13:04:37.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.044+0000 7fae71675640 1 -- 172.21.15.57:0/2845200749 --> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/yearly", "snap_schedule": "1Y", "target": ["mon-mgr", ""]}) v1 -- 0x7fae6c1550a0 con 0x7fae5806afb0 2023-10-19T13:04:37.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.098+0000 7fae537fe640 1 -- 172.21.15.57:0/2845200749 <== mgr.8299 v2:172.21.15.179:6832/3526956956 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+44 (secure 0 0 0) 0x7fae6c1550a0 con 0x7fae5806afb0 2023-10-19T13:04:37.101 INFO:teuthology.orchestra.run.smithi057.stdout:Schedule set for path snap_test_dir1/yearly 2023-10-19T13:04:37.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.100+0000 7fae71675640 1 -- 172.21.15.57:0/2845200749 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7fae5806afb0 msgr2=0x7fae5806d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:37.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.100+0000 7fae71675640 1 --2- 172.21.15.57:0/2845200749 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7fae5806afb0 0x7fae5806d470 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fae640058c0 tx=0x7fae64005850 comp rx=0 tx=0).stop 2023-10-19T13:04:37.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.100+0000 7fae71675640 1 -- 172.21.15.57:0/2845200749 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fae6c1579f0 msgr2=0x7fae6c0ae1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:37.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.100+0000 7fae71675640 1 --2- 172.21.15.57:0/2845200749 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fae6c1579f0 0x7fae6c0ae1f0 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7fae60009780 tx=0x7fae600038a0 comp rx=0 tx=0).stop 2023-10-19T13:04:37.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.100+0000 7fae71675640 1 -- 172.21.15.57:0/2845200749 shutdown_connections 2023-10-19T13:04:37.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.100+0000 7fae71675640 1 --2- 172.21.15.57:0/2845200749 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7fae5806afb0 0x7fae5806d470 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:37.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.100+0000 7fae71675640 1 --2- 172.21.15.57:0/2845200749 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fae6c15c1f0 0x7fae6c155980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:37.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.100+0000 7fae71675640 1 --2- 172.21.15.57:0/2845200749 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fae6c1579f0 0x7fae6c0ae1f0 unknown :-1 s=CLOSED pgs=1640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:37.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.100+0000 7fae71675640 1 --2- 172.21.15.57:0/2845200749 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fae6c06ac70 0x7fae6c13fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:37.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.100+0000 7fae71675640 1 -- 172.21.15.57:0/2845200749 >> 172.21.15.57:0/2845200749 conn(0x7fae6c06fae0 msgr2=0x7fae6c15a4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:37.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.100+0000 7fae71675640 1 -- 172.21.15.57:0/2845200749 shutdown_connections 2023-10-19T13:04:37.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.100+0000 7fae71675640 1 -- 172.21.15.57:0/2845200749 wait complete. 2023-10-19T13:04:37.113 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/yearly 2023-10-19T13:04:37.114 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:04:37.114 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/bad_period_spec) 2023-10-19T13:04:37.131 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec 1X 2023-10-19T13:04:37.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.280+0000 7f332c900640 1 -- 172.21.15.57:0/799491962 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f332806af00 msgr2=0x7f332806b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:37.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.280+0000 7f332c900640 1 --2- 172.21.15.57:0/799491962 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f332806af00 0x7f332806b2e0 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7f3318009900 tx=0x7f331802be00 comp rx=0 tx=0).stop 2023-10-19T13:04:37.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.280+0000 7f332c900640 1 -- 172.21.15.57:0/799491962 shutdown_connections 2023-10-19T13:04:37.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.280+0000 7f332c900640 1 --2- 172.21.15.57:0/799491962 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f33280c96b0 0x7f33280c9b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:37.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.280+0000 7f332c900640 1 --2- 172.21.15.57:0/799491962 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f332806af00 0x7f332806b2e0 unknown :-1 s=CLOSED pgs=1642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:37.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.280+0000 7f332c900640 1 -- 172.21.15.57:0/799491962 >> 172.21.15.57:0/799491962 conn(0x7f3328069930 msgr2=0x7f3328069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:37.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.280+0000 7f332c900640 1 -- 172.21.15.57:0/799491962 shutdown_connections 2023-10-19T13:04:37.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.280+0000 7f332c900640 1 -- 172.21.15.57:0/799491962 wait complete. 2023-10-19T13:04:37.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.282+0000 7f332c900640 1 Processor -- start 2023-10-19T13:04:37.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.282+0000 7f332c900640 1 -- start start 2023-10-19T13:04:37.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.283+0000 7f332c900640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f332806af00 0x7f332812ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:37.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.283+0000 7f332c900640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f33280ab500 0x7f33281f3550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:37.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.283+0000 7f332c900640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f33280c96b0 0x7f332812c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:37.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.283+0000 7f332c900640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f332812c070 con 0x7f332806af00 2023-10-19T13:04:37.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.283+0000 7f332c900640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33281f3a90 con 0x7f33280ab500 2023-10-19T13:04:37.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.283+0000 7f332c900640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f332812ec10 con 0x7f33280c96b0 2023-10-19T13:04:37.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.283+0000 7f332659c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f332806af00 0x7f332812ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:37.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.283+0000 7f332659c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f332806af00 0x7f332812ad30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57266/0 (socket says 172.21.15.57:57266) 2023-10-19T13:04:37.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.283+0000 7f332659c640 1 -- 172.21.15.57:0/2159761526 learned_addr learned my addr 172.21.15.57:0/2159761526 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:37.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.283+0000 7f3326d9d640 1 --2- 172.21.15.57:0/2159761526 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f33280c96b0 0x7f332812c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:37.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.283+0000 7f331dd9b640 1 --2- 172.21.15.57:0/2159761526 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f33280ab500 0x7f33281f3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:37.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.283+0000 7f332659c640 1 -- 172.21.15.57:0/2159761526 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f33280c96b0 msgr2=0x7f332812c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:37.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.283+0000 7f332659c640 1 --2- 172.21.15.57:0/2159761526 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f33280c96b0 0x7f332812c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:37.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.283+0000 7f332659c640 1 -- 172.21.15.57:0/2159761526 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f33280ab500 msgr2=0x7f33281f3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:37.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.283+0000 7f332659c640 1 --2- 172.21.15.57:0/2159761526 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f33280ab500 0x7f33281f3550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:37.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.283+0000 7f332659c640 1 -- 172.21.15.57:0/2159761526 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3318009560 con 0x7f332806af00 2023-10-19T13:04:37.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.283+0000 7f3326d9d640 1 --2- 172.21.15.57:0/2159761526 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f33280c96b0 0x7f332812c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T13:04:37.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.283+0000 7f331dd9b640 1 --2- 172.21.15.57:0/2159761526 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f33280ab500 0x7f33281f3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T13:04:37.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.283+0000 7f332659c640 1 --2- 172.21.15.57:0/2159761526 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f332806af00 0x7f332812ad30 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7f3318036040 tx=0x7f331802db30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:37.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.283+0000 7f331f7fe640 1 -- 172.21.15.57:0/2159761526 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3318003800 con 0x7f332806af00 2023-10-19T13:04:37.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.284+0000 7f331f7fe640 1 -- 172.21.15.57:0/2159761526 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f3318003a50 con 0x7f332806af00 2023-10-19T13:04:37.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.284+0000 7f332c900640 1 -- 172.21.15.57:0/2159761526 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f332812b3f0 con 0x7f332806af00 2023-10-19T13:04:37.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.284+0000 7f331f7fe640 1 -- 172.21.15.57:0/2159761526 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3318003e20 con 0x7f332806af00 2023-10-19T13:04:37.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.284+0000 7f332c900640 1 -- 172.21.15.57:0/2159761526 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f332812b9b0 con 0x7f332806af00 2023-10-19T13:04:37.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.284+0000 7f332c900640 1 -- 172.21.15.57:0/2159761526 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33280690e0 con 0x7f332806af00 2023-10-19T13:04:37.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.285+0000 7f331f7fe640 1 -- 172.21.15.57:0/2159761526 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 73) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f331803e450 con 0x7f332806af00 2023-10-19T13:04:37.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.285+0000 7f331f7fe640 1 --2- 172.21.15.57:0/2159761526 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7f330806afb0 0x7f330806d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:37.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.286+0000 7f331f7fe640 1 -- 172.21.15.57:0/2159761526 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f33180ac6a0 con 0x7f332806af00 2023-10-19T13:04:37.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.286+0000 7f331dd9b640 1 --2- 172.21.15.57:0/2159761526 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7f330806afb0 0x7f330806d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:37.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.286+0000 7f331dd9b640 1 --2- 172.21.15.57:0/2159761526 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7f330806afb0 0x7f330806d470 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f332813e2b0 tx=0x7f3310005dd0 comp rx=0 tx=0).ready entity=mgr.8299 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:37.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.290+0000 7f331f7fe640 1 -- 172.21.15.57:0/2159761526 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f33180772f0 con 0x7f332806af00 2023-10-19T13:04:37.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.487+0000 7f332c900640 1 -- 172.21.15.57:0/2159761526 --> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "1X", "target": ["mon-mgr", ""]}) v1 -- 0x7f3328159c50 con 0x7f330806afb0 2023-10-19T13:04:37.490 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:37.488+0000 7f3eed2ef640 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - multiplier should be one of h,d,w,M,Y 2023-10-19T13:04:37.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.488+0000 7f331f7fe640 1 -- 172.21.15.57:0/2159761526 <== mgr.8299 v2:172.21.15.179:6832/3526956956 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - multiplier should be one of h,d,w,M,Y) v1 ==== 74+0+0 (secure 0 0 0) 0x7f331000ea00 con 0x7f330806afb0 2023-10-19T13:04:37.491 INFO:teuthology.orchestra.run.smithi057.stderr:Error ENOENT: invalid schedule specified - multiplier should be one of h,d,w,M,Y 2023-10-19T13:04:37.495 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-10-19T13:04:37.496 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec 1MM 2023-10-19T13:04:37.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.606+0000 7f5831210640 1 -- 172.21.15.57:0/3080691975 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f582c0afa20 msgr2=0x7f582c158300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:37.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.606+0000 7f5831210640 1 --2- 172.21.15.57:0/3080691975 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f582c0afa20 0x7f582c158300 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f5814009860 tx=0x7f581402bde0 comp rx=0 tx=0).stop 2023-10-19T13:04:37.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.606+0000 7f5831210640 1 -- 172.21.15.57:0/3080691975 shutdown_connections 2023-10-19T13:04:37.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.606+0000 7f5831210640 1 --2- 172.21.15.57:0/3080691975 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f582c0afa20 0x7f582c158300 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:37.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.606+0000 7f5831210640 1 -- 172.21.15.57:0/3080691975 >> 172.21.15.57:0/3080691975 conn(0x7f582c069930 msgr2=0x7f582c069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:37.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.606+0000 7f5831210640 1 -- 172.21.15.57:0/3080691975 shutdown_connections 2023-10-19T13:04:37.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.606+0000 7f5831210640 1 -- 172.21.15.57:0/3080691975 wait complete. 2023-10-19T13:04:37.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.607+0000 7f5831210640 1 Processor -- start 2023-10-19T13:04:37.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.607+0000 7f5831210640 1 -- start start 2023-10-19T13:04:37.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.608+0000 7f5831210640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f582c0afa20 0x7f582c1e4450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:37.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.608+0000 7f5831210640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f582c0b0530 0x7f582c141550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:37.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.608+0000 7f5831210640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f582c0cbd20 0x7f582c1444d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:37.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.608+0000 7f5831210640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f582c1e4990 con 0x7f582c0cbd20 2023-10-19T13:04:37.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.608+0000 7f5831210640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f582c148260 con 0x7f582c0b0530 2023-10-19T13:04:37.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.608+0000 7f5831210640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f582c144da0 con 0x7f582c0afa20 2023-10-19T13:04:37.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.608+0000 7f582b59e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f582c0cbd20 0x7f582c1444d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:37.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.608+0000 7f582b59e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f582c0cbd20 0x7f582c1444d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57268/0 (socket says 172.21.15.57:57268) 2023-10-19T13:04:37.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.608+0000 7f582b59e640 1 -- 172.21.15.57:0/186213212 learned_addr learned my addr 172.21.15.57:0/186213212 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:37.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.608+0000 7f582259c640 1 --2- 172.21.15.57:0/186213212 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f582c0b0530 0x7f582c141550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:37.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.608+0000 7f582ad9d640 1 --2- 172.21.15.57:0/186213212 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f582c0afa20 0x7f582c1e4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:37.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.608+0000 7f582b59e640 1 -- 172.21.15.57:0/186213212 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f582c0afa20 msgr2=0x7f582c1e4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:37.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.608+0000 7f582b59e640 1 --2- 172.21.15.57:0/186213212 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f582c0afa20 0x7f582c1e4450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:37.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.608+0000 7f582b59e640 1 -- 172.21.15.57:0/186213212 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f582c0b0530 msgr2=0x7f582c141550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:37.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.608+0000 7f582b59e640 1 --2- 172.21.15.57:0/186213212 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f582c0b0530 0x7f582c141550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:37.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.608+0000 7f582b59e640 1 -- 172.21.15.57:0/186213212 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5814009490 con 0x7f582c0cbd20 2023-10-19T13:04:37.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.608+0000 7f582ad9d640 1 --2- 172.21.15.57:0/186213212 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f582c0afa20 0x7f582c1e4450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:04:37.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.609+0000 7f582b59e640 1 --2- 172.21.15.57:0/186213212 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f582c0cbd20 0x7f582c1444d0 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7f582400c790 tx=0x7f582400cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:37.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.609+0000 7f5823fff640 1 -- 172.21.15.57:0/186213212 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f582400cea0 con 0x7f582c0cbd20 2023-10-19T13:04:37.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.609+0000 7f5823fff640 1 -- 172.21.15.57:0/186213212 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f5824003b70 con 0x7f582c0cbd20 2023-10-19T13:04:37.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.609+0000 7f5831210640 1 -- 172.21.15.57:0/186213212 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f582c1e1ad0 con 0x7f582c0cbd20 2023-10-19T13:04:37.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.609+0000 7f5823fff640 1 -- 172.21.15.57:0/186213212 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f582400a850 con 0x7f582c0cbd20 2023-10-19T13:04:37.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.609+0000 7f5831210640 1 -- 172.21.15.57:0/186213212 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f582c141fa0 con 0x7f582c0cbd20 2023-10-19T13:04:37.618 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.610+0000 7f5831210640 1 -- 172.21.15.57:0/186213212 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f582c0690e0 con 0x7f582c0cbd20 2023-10-19T13:04:37.618 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.610+0000 7f5823fff640 1 -- 172.21.15.57:0/186213212 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 73) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f582400a9b0 con 0x7f582c0cbd20 2023-10-19T13:04:37.618 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.611+0000 7f5823fff640 1 --2- 172.21.15.57:0/186213212 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7f580c06afb0 0x7f580c06d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:37.618 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.611+0000 7f5823fff640 1 -- 172.21.15.57:0/186213212 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f5824089ae0 con 0x7f582c0cbd20 2023-10-19T13:04:37.618 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.611+0000 7f582ad9d640 1 --2- 172.21.15.57:0/186213212 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7f580c06afb0 0x7f580c06d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:37.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.611+0000 7f582ad9d640 1 --2- 172.21.15.57:0/186213212 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7f580c06afb0 0x7f580c06d470 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f581c0011e0 tx=0x7f581c009040 comp rx=0 tx=0).ready entity=mgr.8299 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:37.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.615+0000 7f5823fff640 1 -- 172.21.15.57:0/186213212 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f5824054740 con 0x7f582c0cbd20 2023-10-19T13:04:37.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.814+0000 7f5831210640 1 -- 172.21.15.57:0/186213212 --> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "1MM", "target": ["mon-mgr", ""]}) v1 -- 0x7f582c142190 con 0x7f580c06afb0 2023-10-19T13:04:37.817 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:37.815+0000 7f3eed2ef640 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2023-10-19T13:04:37.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.815+0000 7f5823fff640 1 -- 172.21.15.57:0/186213212 <== mgr.8299 v2:172.21.15.179:6832/3526956956 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc.) v1 ==== 138+0+0 (secure 0 0 0) 0x7f582c142190 con 0x7f580c06afb0 2023-10-19T13:04:37.818 INFO:teuthology.orchestra.run.smithi057.stderr:Error ENOENT: invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2023-10-19T13:04:37.823 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-10-19T13:04:37.823 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec 1 2023-10-19T13:04:37.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.935+0000 7f3dbaa07640 1 -- 172.21.15.57:0/2148274577 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3db40afa20 msgr2=0x7f3db40705a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:37.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.935+0000 7f3dbaa07640 1 --2- 172.21.15.57:0/2148274577 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3db40afa20 0x7f3db40705a0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f3da4009830 tx=0x7f3da402bd60 comp rx=0 tx=0).stop 2023-10-19T13:04:37.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.935+0000 7f3dbaa07640 1 -- 172.21.15.57:0/2148274577 shutdown_connections 2023-10-19T13:04:37.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.935+0000 7f3dbaa07640 1 --2- 172.21.15.57:0/2148274577 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3db40afa20 0x7f3db40705a0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:37.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.935+0000 7f3dbaa07640 1 --2- 172.21.15.57:0/2148274577 >> v2:172.21.15.179:3300/0 conn(0x7f3db40cbd20 0x7f3db4158300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:37.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.935+0000 7f3dbaa07640 1 -- 172.21.15.57:0/2148274577 >> 172.21.15.57:0/2148274577 conn(0x7f3db4069930 msgr2=0x7f3db4069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:37.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.935+0000 7f3dbaa07640 1 -- 172.21.15.57:0/2148274577 shutdown_connections 2023-10-19T13:04:37.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.935+0000 7f3dbaa07640 1 -- 172.21.15.57:0/2148274577 wait complete. 2023-10-19T13:04:37.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.936+0000 7f3dbaa07640 1 Processor -- start 2023-10-19T13:04:37.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.937+0000 7f3dbaa07640 1 -- start start 2023-10-19T13:04:37.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.937+0000 7f3dbaa07640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3db40afa20 0x7f3db4140b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:37.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.937+0000 7f3dbaa07640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3db40b0530 0x7f3db4146200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:37.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.937+0000 7f3dbaa07640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3db40cbd20 0x7f3db4141830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:37.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.937+0000 7f3dbaa07640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3db41437f0 con 0x7f3db40afa20 2023-10-19T13:04:37.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.937+0000 7f3dbaa07640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3db4144470 con 0x7f3db40b0530 2023-10-19T13:04:37.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.937+0000 7f3dbaa07640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3db41410d0 con 0x7f3db40cbd20 2023-10-19T13:04:37.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.937+0000 7f3db3fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3db40afa20 0x7f3db4140b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:37.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.937+0000 7f3db3fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3db40afa20 0x7f3db4140b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57278/0 (socket says 172.21.15.57:57278) 2023-10-19T13:04:37.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.937+0000 7f3db3fff640 1 -- 172.21.15.57:0/3544782014 learned_addr learned my addr 172.21.15.57:0/3544782014 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:37.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.938+0000 7f3db37fe640 1 --2- 172.21.15.57:0/3544782014 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3db40b0530 0x7f3db4146200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:37.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.938+0000 7f3db8fa4640 1 --2- 172.21.15.57:0/3544782014 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3db40cbd20 0x7f3db4141830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:37.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.938+0000 7f3db3fff640 1 -- 172.21.15.57:0/3544782014 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3db40cbd20 msgr2=0x7f3db4141830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:37.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.938+0000 7f3db3fff640 1 --2- 172.21.15.57:0/3544782014 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3db40cbd20 0x7f3db4141830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:37.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.938+0000 7f3db3fff640 1 -- 172.21.15.57:0/3544782014 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3db40b0530 msgr2=0x7f3db4146200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:37.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.938+0000 7f3db3fff640 1 --2- 172.21.15.57:0/3544782014 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3db40b0530 0x7f3db4146200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:37.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.938+0000 7f3db3fff640 1 -- 172.21.15.57:0/3544782014 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d9c009560 con 0x7f3db40afa20 2023-10-19T13:04:37.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.938+0000 7f3db3fff640 1 --2- 172.21.15.57:0/3544782014 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3db40afa20 0x7f3db4140b90 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c00e9c0 tx=0x7f3d9c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:37.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.938+0000 7f3db0ff9640 1 -- 172.21.15.57:0/3544782014 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d9c00cd80 con 0x7f3db40afa20 2023-10-19T13:04:37.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.938+0000 7f3db0ff9640 1 -- 172.21.15.57:0/3544782014 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f3d9c004020 con 0x7f3db40afa20 2023-10-19T13:04:37.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.938+0000 7f3dbaa07640 1 -- 172.21.15.57:0/3544782014 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3da4009490 con 0x7f3db40afa20 2023-10-19T13:04:37.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.938+0000 7f3db0ff9640 1 -- 172.21.15.57:0/3544782014 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d9c004410 con 0x7f3db40afa20 2023-10-19T13:04:37.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.938+0000 7f3dbaa07640 1 -- 172.21.15.57:0/3544782014 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3db4070cf0 con 0x7f3db40afa20 2023-10-19T13:04:37.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.939+0000 7f3dbaa07640 1 -- 172.21.15.57:0/3544782014 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3db40690e0 con 0x7f3db40afa20 2023-10-19T13:04:37.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.940+0000 7f3db0ff9640 1 -- 172.21.15.57:0/3544782014 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 74) v1 ==== 89405+0+0 (secure 0 0 0) 0x7f3d9c0035f0 con 0x7f3db40afa20 2023-10-19T13:04:37.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.940+0000 7f3db0ff9640 1 --2- 172.21.15.57:0/3544782014 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7f3d9406b040 0x7f3d9406d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:37.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.940+0000 7f3db0ff9640 1 -- 172.21.15.57:0/3544782014 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f3d9c014070 con 0x7f3db40afa20 2023-10-19T13:04:37.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.940+0000 7f3db37fe640 1 --2- 172.21.15.57:0/3544782014 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7f3d9406b040 0x7f3d9406d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:37.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.941+0000 7f3db37fe640 1 --2- 172.21.15.57:0/3544782014 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7f3d9406b040 0x7f3d9406d500 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f3da4036000 tx=0x7f3da4004730 comp rx=0 tx=0).ready entity=mgr.8299 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:37.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:37.944+0000 7f3db0ff9640 1 -- 172.21.15.57:0/3544782014 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3d9c055280 con 0x7f3db40afa20 2023-10-19T13:04:38.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.141+0000 7f3dbaa07640 1 -- 172.21.15.57:0/3544782014 --> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f3db406f700 con 0x7f3d9406b040 2023-10-19T13:04:38.145 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:38.142+0000 7f3eed2ef640 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2023-10-19T13:04:38.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.142+0000 7f3db0ff9640 1 -- 172.21.15.57:0/3544782014 <== mgr.8299 v2:172.21.15.179:6832/3526956956 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc.) v1 ==== 138+0+0 (secure 0 0 0) 0x7f3db406f700 con 0x7f3d9406b040 2023-10-19T13:04:38.145 INFO:teuthology.orchestra.run.smithi057.stderr:Error ENOENT: invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2023-10-19T13:04:38.150 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-10-19T13:04:38.151 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec M 2023-10-19T13:04:38.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.261+0000 7f70268a9640 1 -- 172.21.15.57:0/2920177525 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f702006af00 msgr2=0x7f702006b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:38.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.261+0000 7f70268a9640 1 --2- 172.21.15.57:0/2920177525 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f702006af00 0x7f702006b2e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f7008009860 tx=0x7f700802bca0 comp rx=0 tx=0).stop 2023-10-19T13:04:38.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.261+0000 7f70268a9640 1 -- 172.21.15.57:0/2920177525 shutdown_connections 2023-10-19T13:04:38.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.261+0000 7f70268a9640 1 --2- 172.21.15.57:0/2920177525 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f702006af00 0x7f702006b2e0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:38.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.261+0000 7f70268a9640 1 -- 172.21.15.57:0/2920177525 >> 172.21.15.57:0/2920177525 conn(0x7f7020069930 msgr2=0x7f7020069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:38.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.261+0000 7f70268a9640 1 -- 172.21.15.57:0/2920177525 shutdown_connections 2023-10-19T13:04:38.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.261+0000 7f70268a9640 1 -- 172.21.15.57:0/2920177525 wait complete. 2023-10-19T13:04:38.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.262+0000 7f70268a9640 1 Processor -- start 2023-10-19T13:04:38.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.263+0000 7f70268a9640 1 -- start start 2023-10-19T13:04:38.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.263+0000 7f70268a9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f702006af00 0x7f702006fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:38.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.263+0000 7f70268a9640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f70200abc10 0x7f7020070820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:38.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.263+0000 7f70268a9640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f70200c96b0 0x7f7020151540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:38.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.263+0000 7f70268a9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70200afe00 con 0x7f702006af00 2023-10-19T13:04:38.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.263+0000 7f70268a9640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7020070d60 con 0x7f70200abc10 2023-10-19T13:04:38.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.263+0000 7f70268a9640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70200b0050 con 0x7f70200c96b0 2023-10-19T13:04:38.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.263+0000 7f701ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f702006af00 0x7f702006fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:38.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.263+0000 7f701ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f702006af00 0x7f702006fdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57292/0 (socket says 172.21.15.57:57292) 2023-10-19T13:04:38.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.263+0000 7f701ffff640 1 -- 172.21.15.57:0/810154468 learned_addr learned my addr 172.21.15.57:0/810154468 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:38.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.264+0000 7f7024e46640 1 --2- 172.21.15.57:0/810154468 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f70200c96b0 0x7f7020151540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:38.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.264+0000 7f701ffff640 1 -- 172.21.15.57:0/810154468 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f70200c96b0 msgr2=0x7f7020151540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:38.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.264+0000 7f701f7fe640 1 --2- 172.21.15.57:0/810154468 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f70200abc10 0x7f7020070820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:38.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.264+0000 7f701ffff640 1 --2- 172.21.15.57:0/810154468 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f70200c96b0 0x7f7020151540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:38.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.264+0000 7f701ffff640 1 -- 172.21.15.57:0/810154468 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f70200abc10 msgr2=0x7f7020070820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:38.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.264+0000 7f701ffff640 1 --2- 172.21.15.57:0/810154468 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f70200abc10 0x7f7020070820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:38.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.264+0000 7f701ffff640 1 -- 172.21.15.57:0/810154468 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7008009490 con 0x7f702006af00 2023-10-19T13:04:38.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.264+0000 7f701ffff640 1 --2- 172.21.15.57:0/810154468 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f702006af00 0x7f702006fdc0 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7f7008009980 tx=0x7f700802cd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:38.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.264+0000 7f701cff9640 1 -- 172.21.15.57:0/810154468 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7008005220 con 0x7f702006af00 2023-10-19T13:04:38.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.264+0000 7f70268a9640 1 -- 172.21.15.57:0/810154468 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70200b0330 con 0x7f702006af00 2023-10-19T13:04:38.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.264+0000 7f701cff9640 1 -- 172.21.15.57:0/810154468 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f7008005470 con 0x7f702006af00 2023-10-19T13:04:38.272 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.264+0000 7f701cff9640 1 -- 172.21.15.57:0/810154468 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7008005c90 con 0x7f702006af00 2023-10-19T13:04:38.272 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.264+0000 7f70268a9640 1 -- 172.21.15.57:0/810154468 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70200b08f0 con 0x7f702006af00 2023-10-19T13:04:38.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.265+0000 7f70268a9640 1 -- 172.21.15.57:0/810154468 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70200690e0 con 0x7f702006af00 2023-10-19T13:04:38.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.265+0000 7f701cff9640 1 -- 172.21.15.57:0/810154468 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 74) v1 ==== 89405+0+0 (secure 0 0 0) 0x7f700803b040 con 0x7f702006af00 2023-10-19T13:04:38.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.266+0000 7f701cff9640 1 --2- 172.21.15.57:0/810154468 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7f700006b040 0x7f700006d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:38.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.266+0000 7f701cff9640 1 -- 172.21.15.57:0/810154468 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f70080ab950 con 0x7f702006af00 2023-10-19T13:04:38.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.266+0000 7f701f7fe640 1 --2- 172.21.15.57:0/810154468 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7f700006b040 0x7f700006d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:38.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.267+0000 7f701f7fe640 1 --2- 172.21.15.57:0/810154468 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7f700006b040 0x7f700006d500 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f70100017a0 tx=0x7f7010009040 comp rx=0 tx=0).ready entity=mgr.8299 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:38.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.270+0000 7f701cff9640 1 -- 172.21.15.57:0/810154468 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f70080764d0 con 0x7f702006af00 2023-10-19T13:04:38.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.467+0000 7f70268a9640 1 -- 172.21.15.57:0/810154468 --> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "M", "target": ["mon-mgr", ""]}) v1 -- 0x7f70200cc420 con 0x7f700006b040 2023-10-19T13:04:38.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.469+0000 7f701cff9640 1 -- 172.21.15.57:0/810154468 <== mgr.8299 v2:172.21.15.179:6832/3526956956 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc.) v1 ==== 138+0+0 (secure 0 0 0) 0x7f70200cc420 con 0x7f700006b040 2023-10-19T13:04:38.474 INFO:teuthology.orchestra.run.smithi057.stderr:Error ENOENT: invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2023-10-19T13:04:38.474 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:38.469+0000 7f3eed2ef640 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2023-10-19T13:04:38.476 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-10-19T13:04:38.477 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec -1m 2023-10-19T13:04:38.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.587+0000 7f734154c640 1 -- 172.21.15.57:0/2697531408 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f733c06f0a0 msgr2=0x7f733c06dbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:38.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.587+0000 7f734154c640 1 --2- 172.21.15.57:0/2697531408 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f733c06f0a0 0x7f733c06dbf0 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7f7330005fd0 tx=0x7f733002be10 comp rx=0 tx=0).stop 2023-10-19T13:04:38.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.587+0000 7f734154c640 1 -- 172.21.15.57:0/2697531408 shutdown_connections 2023-10-19T13:04:38.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.587+0000 7f734154c640 1 --2- 172.21.15.57:0/2697531408 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f733c06f0a0 0x7f733c06dbf0 unknown :-1 s=CLOSED pgs=1649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:38.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.587+0000 7f734154c640 1 --2- 172.21.15.57:0/2697531408 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f733c06d2d0 0x7f733c06d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:38.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.587+0000 7f734154c640 1 -- 172.21.15.57:0/2697531408 >> 172.21.15.57:0/2697531408 conn(0x7f733c0b0670 msgr2=0x7f733c0b0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:38.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.587+0000 7f734154c640 1 -- 172.21.15.57:0/2697531408 shutdown_connections 2023-10-19T13:04:38.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.587+0000 7f734154c640 1 -- 172.21.15.57:0/2697531408 wait complete. 2023-10-19T13:04:38.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.588+0000 7f734154c640 1 Processor -- start 2023-10-19T13:04:38.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.588+0000 7f734154c640 1 -- start start 2023-10-19T13:04:38.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.589+0000 7f734154c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f733c06d2d0 0x7f733c150de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:38.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.589+0000 7f734154c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f733c06f0a0 0x7f733c154890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:38.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.589+0000 7f734154c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f733c0ab950 0x7f733c156310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:38.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.589+0000 7f734154c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f733c0cc420 con 0x7f733c06f0a0 2023-10-19T13:04:38.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.589+0000 7f734154c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f733c151320 con 0x7f733c0ab950 2023-10-19T13:04:38.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.589+0000 7f734154c640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f733c153130 con 0x7f733c06d2d0 2023-10-19T13:04:38.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.589+0000 7f733a7fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f733c06f0a0 0x7f733c154890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:38.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.589+0000 7f733a7fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f733c06f0a0 0x7f733c154890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57320/0 (socket says 172.21.15.57:57320) 2023-10-19T13:04:38.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.589+0000 7f733a7fc640 1 -- 172.21.15.57:0/3166287273 learned_addr learned my addr 172.21.15.57:0/3166287273 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:38.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.589+0000 7f733affd640 1 --2- 172.21.15.57:0/3166287273 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f733c06d2d0 0x7f733c150de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:38.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.589+0000 7f733b7fe640 1 --2- 172.21.15.57:0/3166287273 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f733c0ab950 0x7f733c156310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:38.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.589+0000 7f733a7fc640 1 -- 172.21.15.57:0/3166287273 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f733c06d2d0 msgr2=0x7f733c150de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:38.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.589+0000 7f733a7fc640 1 --2- 172.21.15.57:0/3166287273 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f733c06d2d0 0x7f733c150de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:38.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.589+0000 7f733a7fc640 1 -- 172.21.15.57:0/3166287273 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f733c0ab950 msgr2=0x7f733c156310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:38.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.589+0000 7f733a7fc640 1 --2- 172.21.15.57:0/3166287273 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f733c0ab950 0x7f733c156310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:38.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.589+0000 7f733a7fc640 1 -- 172.21.15.57:0/3166287273 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7330009560 con 0x7f733c06f0a0 2023-10-19T13:04:38.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.589+0000 7f733a7fc640 1 --2- 172.21.15.57:0/3166287273 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f733c06f0a0 0x7f733c154890 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7f7330006010 tx=0x7f733002dbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:38.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.590+0000 7f73237fe640 1 -- 172.21.15.57:0/3166287273 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73300035f0 con 0x7f733c06f0a0 2023-10-19T13:04:38.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.590+0000 7f73237fe640 1 -- 172.21.15.57:0/3166287273 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f7330003840 con 0x7f733c06f0a0 2023-10-19T13:04:38.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.590+0000 7f734154c640 1 -- 172.21.15.57:0/3166287273 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f733c153410 con 0x7f733c06f0a0 2023-10-19T13:04:38.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.590+0000 7f73237fe640 1 -- 172.21.15.57:0/3166287273 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7330003c50 con 0x7f733c06f0a0 2023-10-19T13:04:38.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.590+0000 7f734154c640 1 -- 172.21.15.57:0/3166287273 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f733c1539d0 con 0x7f733c06f0a0 2023-10-19T13:04:38.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.591+0000 7f734154c640 1 -- 172.21.15.57:0/3166287273 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f733c0ce1d0 con 0x7f733c06f0a0 2023-10-19T13:04:38.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.591+0000 7f73237fe640 1 -- 172.21.15.57:0/3166287273 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 74) v1 ==== 89405+0+0 (secure 0 0 0) 0x7f733003e450 con 0x7f733c06f0a0 2023-10-19T13:04:38.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.592+0000 7f73237fe640 1 --2- 172.21.15.57:0/3166287273 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7f732806b040 0x7f732806d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:38.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.592+0000 7f73237fe640 1 -- 172.21.15.57:0/3166287273 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f73300ac720 con 0x7f733c06f0a0 2023-10-19T13:04:38.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.592+0000 7f733affd640 1 --2- 172.21.15.57:0/3166287273 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7f732806b040 0x7f732806d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:38.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.592+0000 7f733affd640 1 --2- 172.21.15.57:0/3166287273 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7f732806b040 0x7f732806d500 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f732c00bbe0 tx=0x7f732c00b390 comp rx=0 tx=0).ready entity=mgr.8299 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:38.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.596+0000 7f73237fe640 1 -- 172.21.15.57:0/3166287273 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7330077460 con 0x7f733c06f0a0 2023-10-19T13:04:38.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.795+0000 7f734154c640 1 -- 172.21.15.57:0/3166287273 --> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "-1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f733c0ae380 con 0x7f732806b040 2023-10-19T13:04:38.798 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:38.796+0000 7f3eed2ef640 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - period must be a non-zero positive value e.g. 1h or 4d etc. 2023-10-19T13:04:38.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.796+0000 7f73237fe640 1 -- 172.21.15.57:0/3166287273 <== mgr.8299 v2:172.21.15.179:6832/3526956956 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - period must be a non-zero positive value e.g. 1h or 4d etc.) v1 ==== 96+0+0 (secure 0 0 0) 0x7f732c015070 con 0x7f732806b040 2023-10-19T13:04:38.799 INFO:teuthology.orchestra.run.smithi057.stderr:Error ENOENT: invalid schedule specified - period must be a non-zero positive value e.g. 1h or 4d etc. 2023-10-19T13:04:38.803 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-10-19T13:04:38.804 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec '' 2023-10-19T13:04:38.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.913+0000 7f1ac699e640 1 -- 172.21.15.57:0/1396841504 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1ac00b0c40 msgr2=0x7f1ac0158300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:38.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.913+0000 7f1ac699e640 1 --2- 172.21.15.57:0/1396841504 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1ac00b0c40 0x7f1ac0158300 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7f1ab8005fd0 tx=0x7f1ab802c170 comp rx=0 tx=0).stop 2023-10-19T13:04:38.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.913+0000 7f1ac699e640 1 -- 172.21.15.57:0/1396841504 shutdown_connections 2023-10-19T13:04:38.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.914+0000 7f1ac699e640 1 --2- 172.21.15.57:0/1396841504 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1ac00b0c40 0x7f1ac0158300 unknown :-1 s=CLOSED pgs=1652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:38.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.914+0000 7f1ac699e640 1 -- 172.21.15.57:0/1396841504 >> 172.21.15.57:0/1396841504 conn(0x7f1ac0069930 msgr2=0x7f1ac0069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:38.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.914+0000 7f1ac699e640 1 -- 172.21.15.57:0/1396841504 shutdown_connections 2023-10-19T13:04:38.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.914+0000 7f1ac699e640 1 -- 172.21.15.57:0/1396841504 wait complete. 2023-10-19T13:04:38.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.915+0000 7f1ac699e640 1 Processor -- start 2023-10-19T13:04:38.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.915+0000 7f1ac699e640 1 -- start start 2023-10-19T13:04:38.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.916+0000 7f1ac699e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1ac00afa20 0x7f1ac0152bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:38.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.916+0000 7f1ac699e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1ac00b0c40 0x7f1ac01ee000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:38.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.916+0000 7f1ac699e640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1ac00cbd20 0x7f1ac014ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:38.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.916+0000 7f1ac699e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac0156da0 con 0x7f1ac00afa20 2023-10-19T13:04:38.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.916+0000 7f1ac699e640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac014e320 con 0x7f1ac00b0c40 2023-10-19T13:04:38.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.916+0000 7f1ac699e640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac0152610 con 0x7f1ac00cbd20 2023-10-19T13:04:38.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.916+0000 7f1abffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1ac00afa20 0x7f1ac0152bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:38.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.916+0000 7f1abffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1ac00afa20 0x7f1ac0152bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57334/0 (socket says 172.21.15.57:57334) 2023-10-19T13:04:38.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.916+0000 7f1abffff640 1 -- 172.21.15.57:0/2521369465 learned_addr learned my addr 172.21.15.57:0/2521369465 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:38.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.916+0000 7f1ac4f3b640 1 --2- 172.21.15.57:0/2521369465 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1ac00cbd20 0x7f1ac014ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:38.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.916+0000 7f1abf7fe640 1 --2- 172.21.15.57:0/2521369465 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1ac00b0c40 0x7f1ac01ee000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:38.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.916+0000 7f1abffff640 1 -- 172.21.15.57:0/2521369465 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1ac00cbd20 msgr2=0x7f1ac014ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:38.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.916+0000 7f1abffff640 1 --2- 172.21.15.57:0/2521369465 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1ac00cbd20 0x7f1ac014ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:38.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.916+0000 7f1abffff640 1 -- 172.21.15.57:0/2521369465 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1ac00b0c40 msgr2=0x7f1ac01ee000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:38.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.916+0000 7f1abffff640 1 --2- 172.21.15.57:0/2521369465 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1ac00b0c40 0x7f1ac01ee000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:38.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.916+0000 7f1abffff640 1 -- 172.21.15.57:0/2521369465 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ab8009840 con 0x7f1ac00afa20 2023-10-19T13:04:38.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.916+0000 7f1abffff640 1 --2- 172.21.15.57:0/2521369465 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1ac00afa20 0x7f1ac0152bc0 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7f1ab000c430 tx=0x7f1ab000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:38.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.916+0000 7f1abcff9640 1 -- 172.21.15.57:0/2521369465 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab00052d0 con 0x7f1ac00afa20 2023-10-19T13:04:38.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.916+0000 7f1abcff9640 1 -- 172.21.15.57:0/2521369465 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f1ab00035f0 con 0x7f1ac00afa20 2023-10-19T13:04:38.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.916+0000 7f1ac699e640 1 -- 172.21.15.57:0/2521369465 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ac0152050 con 0x7f1ac00afa20 2023-10-19T13:04:38.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.916+0000 7f1abcff9640 1 -- 172.21.15.57:0/2521369465 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab0003a00 con 0x7f1ac00afa20 2023-10-19T13:04:38.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.917+0000 7f1ac699e640 1 -- 172.21.15.57:0/2521369465 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ac01e17b0 con 0x7f1ac00afa20 2023-10-19T13:04:38.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.918+0000 7f1ac699e640 1 -- 172.21.15.57:0/2521369465 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ac00690e0 con 0x7f1ac00afa20 2023-10-19T13:04:38.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.918+0000 7f1abcff9640 1 -- 172.21.15.57:0/2521369465 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 74) v1 ==== 89405+0+0 (secure 0 0 0) 0x7f1ab0003b60 con 0x7f1ac00afa20 2023-10-19T13:04:38.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.918+0000 7f1abcff9640 1 --2- 172.21.15.57:0/2521369465 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7f1aac06b040 0x7f1aac06d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:38.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.918+0000 7f1abcff9640 1 -- 172.21.15.57:0/2521369465 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f1ab0089350 con 0x7f1ac00afa20 2023-10-19T13:04:38.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.918+0000 7f1abf7fe640 1 --2- 172.21.15.57:0/2521369465 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7f1aac06b040 0x7f1aac06d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:38.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.919+0000 7f1abf7fe640 1 --2- 172.21.15.57:0/2521369465 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7f1aac06b040 0x7f1aac06d500 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1ab4000f60 tx=0x7f1ab4009040 comp rx=0 tx=0).ready entity=mgr.8299 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:38.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:38.922+0000 7f1abcff9640 1 -- 172.21.15.57:0/2521369465 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f1ab0053f50 con 0x7f1ac00afa20 2023-10-19T13:04:39.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:39.121+0000 7f1ac699e640 1 -- 172.21.15.57:0/2521369465 --> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "", "target": ["mon-mgr", ""]}) v1 -- 0x7f1ac006f700 con 0x7f1aac06b040 2023-10-19T13:04:39.125 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:39.123+0000 7f3eed2ef640 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2023-10-19T13:04:39.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:39.123+0000 7f1abcff9640 1 -- 172.21.15.57:0/2521369465 <== mgr.8299 v2:172.21.15.179:6832/3526956956 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc.) v1 ==== 138+0+0 (secure 0 0 0) 0x7f1ac006f700 con 0x7f1aac06b040 2023-10-19T13:04:39.126 INFO:teuthology.orchestra.run.smithi057.stderr:Error ENOENT: invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2023-10-19T13:04:39.131 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-10-19T13:04:39.132 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:04:39.132 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/minutes) 2023-10-19T13:04:39.149 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:04:39.150 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/hourly) 2023-10-19T13:04:39.207 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:04:39.208 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/daily) 2023-10-19T13:04:39.265 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:04:39.266 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/weekly) 2023-10-19T13:04:39.323 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:04:39.324 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/monthly) 2023-10-19T13:04:39.381 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:04:39.381 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/yearly) 2023-10-19T13:04:39.438 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:04:39.438 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/bad_period_spec) 2023-10-19T13:04:39.497 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-19T13:04:39.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:39.643+0000 7f39db59e640 1 -- 172.21.15.57:0/4020283540 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f39dc0ce2f0 msgr2=0x7f39dc0c5550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:39.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:39.643+0000 7f39db59e640 1 --2- 172.21.15.57:0/4020283540 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f39dc0ce2f0 0x7f39dc0c5550 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7f39d4005fd0 tx=0x7f39d402c0e0 comp rx=0 tx=0).stop 2023-10-19T13:04:39.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:39.643+0000 7f39db59e640 1 -- 172.21.15.57:0/4020283540 shutdown_connections 2023-10-19T13:04:39.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:39.643+0000 7f39db59e640 1 --2- 172.21.15.57:0/4020283540 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f39dc0cda60 0x7f39dc0c5010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:39.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:39.643+0000 7f39db59e640 1 --2- 172.21.15.57:0/4020283540 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f39dc0ce2f0 0x7f39dc0c5550 unknown :-1 s=CLOSED pgs=1655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:39.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:39.643+0000 7f39db59e640 1 -- 172.21.15.57:0/4020283540 >> 172.21.15.57:0/4020283540 conn(0x7f39dc0d0c20 msgr2=0x7f39dc0d1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:39.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:39.643+0000 7f39db59e640 1 -- 172.21.15.57:0/4020283540 shutdown_connections 2023-10-19T13:04:39.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:39.643+0000 7f39db59e640 1 -- 172.21.15.57:0/4020283540 wait complete. 2023-10-19T13:04:39.650 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:39.645+0000 7f39db59e640 1 Processor -- start 2023-10-19T13:04:39.650 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:39.645+0000 7f39db59e640 1 -- start start 2023-10-19T13:04:39.650 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:39.645+0000 7f39db59e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f39dc0cda60 0x7f39dc14e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:39.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:39.645+0000 7f39db59e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f39dc0ce2f0 0x7f39dc0c7cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:39.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:39.645+0000 7f39db59e640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f39dc15b2f0 0x7f39dc0c8230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:39.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:39.645+0000 7f39db59e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39dc15b990 con 0x7f39dc0ce2f0 2023-10-19T13:04:39.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:39.645+0000 7f39db59e640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39dc15bc70 con 0x7f39dc0cda60 2023-10-19T13:04:39.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:39.645+0000 7f39db59e640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39dc15bf50 con 0x7f39dc15b2f0 2023-10-19T13:04:39.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:39.645+0000 7f39d3fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f39dc0ce2f0 0x7f39dc0c7cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:39.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:39.645+0000 7f39d3fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f39dc0ce2f0 0x7f39dc0c7cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57348/0 (socket says 172.21.15.57:57348) 2023-10-19T13:04:39.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:39.645+0000 7f39d3fff640 1 -- 172.21.15.57:0/3295673298 learned_addr learned my addr 172.21.15.57:0/3295673298 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:39.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:39.646+0000 7f39dad9d640 1 --2- 172.21.15.57:0/3295673298 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f39dc15b2f0 0x7f39dc0c8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:39.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:39.646+0000 7f39da59c640 1 --2- 172.21.15.57:0/3295673298 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f39dc0cda60 0x7f39dc14e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:39.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:39.646+0000 7f39d3fff640 1 -- 172.21.15.57:0/3295673298 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f39dc15b2f0 msgr2=0x7f39dc0c8230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:39.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:39.646+0000 7f39d3fff640 1 --2- 172.21.15.57:0/3295673298 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f39dc15b2f0 0x7f39dc0c8230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:39.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:39.646+0000 7f39d3fff640 1 -- 172.21.15.57:0/3295673298 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f39dc0cda60 msgr2=0x7f39dc14e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:39.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:39.646+0000 7f39d3fff640 1 --2- 172.21.15.57:0/3295673298 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f39dc0cda60 0x7f39dc14e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:39.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:39.646+0000 7f39d3fff640 1 -- 172.21.15.57:0/3295673298 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39d4009840 con 0x7f39dc0ce2f0 2023-10-19T13:04:39.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:39.646+0000 7f39d3fff640 1 --2- 172.21.15.57:0/3295673298 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f39dc0ce2f0 0x7f39dc0c7cf0 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7f39c400b410 tx=0x7f39c400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:39.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:39.646+0000 7f39d2ffd640 1 -- 172.21.15.57:0/3295673298 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39c4011830 con 0x7f39dc0ce2f0 2023-10-19T13:04:39.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:39.646+0000 7f39d2ffd640 1 -- 172.21.15.57:0/3295673298 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f39c4011aa0 con 0x7f39dc0ce2f0 2023-10-19T13:04:39.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:39.646+0000 7f39db59e640 1 -- 172.21.15.57:0/3295673298 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39dc15c230 con 0x7f39dc0ce2f0 2023-10-19T13:04:39.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:39.646+0000 7f39d2ffd640 1 -- 172.21.15.57:0/3295673298 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39c4010380 con 0x7f39dc0ce2f0 2023-10-19T13:04:39.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:39.646+0000 7f39db59e640 1 -- 172.21.15.57:0/3295673298 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39dc0ca7c0 con 0x7f39dc0ce2f0 2023-10-19T13:04:39.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:39.647+0000 7f39db59e640 1 -- 172.21.15.57:0/3295673298 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39dc066960 con 0x7f39dc0ce2f0 2023-10-19T13:04:39.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:39.648+0000 7f39d2ffd640 1 -- 172.21.15.57:0/3295673298 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 74) v1 ==== 89405+0+0 (secure 0 0 0) 0x7f39c40104e0 con 0x7f39dc0ce2f0 2023-10-19T13:04:39.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:39.648+0000 7f39d2ffd640 1 --2- 172.21.15.57:0/3295673298 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7f39bc06b040 0x7f39bc06d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:39.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:39.648+0000 7f39d2ffd640 1 -- 172.21.15.57:0/3295673298 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f39c408a030 con 0x7f39dc0ce2f0 2023-10-19T13:04:39.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:39.648+0000 7f39da59c640 1 --2- 172.21.15.57:0/3295673298 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7f39bc06b040 0x7f39bc06d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:39.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:39.649+0000 7f39da59c640 1 --2- 172.21.15.57:0/3295673298 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7f39bc06b040 0x7f39bc06d500 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f39cc0046b0 tx=0x7f39cc009040 comp rx=0 tx=0).ready entity=mgr.8299 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:39.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:39.652+0000 7f39d2ffd640 1 -- 172.21.15.57:0/3295673298 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f39c4054ba0 con 0x7f39dc0ce2f0 2023-10-19T13:04:39.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:39.880+0000 7f39db59e640 1 -- 172.21.15.57:0/3295673298 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f39dc0c7750 con 0x7f39dc0ce2f0 2023-10-19T13:04:40.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:40.880+0000 7f39d2ffd640 1 -- 172.21.15.57:0/3295673298 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v75) v1 ==== 93+0+0 (secure 0 0 0) 0x7f39c40581c0 con 0x7f39dc0ce2f0 2023-10-19T13:04:40.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:40.885+0000 7f39db59e640 1 -- 172.21.15.57:0/3295673298 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7f39bc06b040 msgr2=0x7f39bc06d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:40.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:40.885+0000 7f39db59e640 1 --2- 172.21.15.57:0/3295673298 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7f39bc06b040 0x7f39bc06d500 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f39cc0046b0 tx=0x7f39cc009040 comp rx=0 tx=0).stop 2023-10-19T13:04:40.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:40.885+0000 7f39db59e640 1 -- 172.21.15.57:0/3295673298 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f39dc0ce2f0 msgr2=0x7f39dc0c7cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:40.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:40.885+0000 7f39d2ffd640 1 -- 172.21.15.57:0/3295673298 <== mon.0 v2:172.21.15.57:3300/0 8 ==== mgrmap(e 75) v1 ==== 89388+0+0 (secure 0 0 0) 0x7f39c4054f70 con 0x7f39dc0ce2f0 2023-10-19T13:04:40.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:40.885+0000 7f39db59e640 1 --2- 172.21.15.57:0/3295673298 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f39dc0ce2f0 0x7f39dc0c7cf0 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7f39c400b410 tx=0x7f39c400b8d0 comp rx=0 tx=0).stop 2023-10-19T13:04:40.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:40.885+0000 7f39db59e640 1 -- 172.21.15.57:0/3295673298 shutdown_connections 2023-10-19T13:04:40.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:40.885+0000 7f39db59e640 1 --2- 172.21.15.57:0/3295673298 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7f39bc06b040 0x7f39bc06d500 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:40.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:40.885+0000 7f39db59e640 1 --2- 172.21.15.57:0/3295673298 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f39dc15b2f0 0x7f39dc0c8230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:40.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:40.885+0000 7f39db59e640 1 --2- 172.21.15.57:0/3295673298 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f39dc0ce2f0 0x7f39dc0c7cf0 unknown :-1 s=CLOSED pgs=1656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:40.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:40.885+0000 7f39db59e640 1 --2- 172.21.15.57:0/3295673298 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f39dc0cda60 0x7f39dc14e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:40.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:40.885+0000 7f39db59e640 1 -- 172.21.15.57:0/3295673298 >> 172.21.15.57:0/3295673298 conn(0x7f39dc0d0c20 msgr2=0x7f39dc159010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:40.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:40.885+0000 7f39d2ffd640 1 --2- 172.21.15.57:0/3295673298 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7f39bc033cd0 0x7f39bc0360c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:40.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:40.885+0000 7f39db59e640 1 -- 172.21.15.57:0/3295673298 shutdown_connections 2023-10-19T13:04:40.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:40.885+0000 7f39db59e640 1 --2- 172.21.15.57:0/3295673298 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7f39bc033cd0 0x7f39bc0360c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:40.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:40.886+0000 7f39db59e640 1 -- 172.21.15.57:0/3295673298 wait complete. 2023-10-19T13:04:40.902 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-19T13:04:40.903 DEBUG:teuthology.orchestra.run.smithi057:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-19T13:04:40.948 DEBUG:teuthology.orchestra.run.smithi179:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-19T13:04:41.008 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-19T13:04:41.009 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T13:04:41.009 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T13:04:41.026 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T13:04:41.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.173+0000 7feec259e640 1 -- 172.21.15.57:0/355103355 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7feebc159060 msgr2=0x7feebc06d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:41.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.173+0000 7feec259e640 1 --2- 172.21.15.57:0/355103355 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7feebc159060 0x7feebc06d260 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7feeac005fd0 tx=0x7feeac02bd90 comp rx=0 tx=0).stop 2023-10-19T13:04:41.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.173+0000 7feec259e640 1 -- 172.21.15.57:0/355103355 shutdown_connections 2023-10-19T13:04:41.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.173+0000 7feec259e640 1 --2- 172.21.15.57:0/355103355 >> v2:172.21.15.179:3300/0 conn(0x7feebc0cdb30 0x7feebc0cdfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:41.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.173+0000 7feec259e640 1 --2- 172.21.15.57:0/355103355 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7feebc159060 0x7feebc06d260 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:41.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.173+0000 7feec259e640 1 -- 172.21.15.57:0/355103355 >> 172.21.15.57:0/355103355 conn(0x7feebc0b0670 msgr2=0x7feebc0b0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:41.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.173+0000 7feec259e640 1 -- 172.21.15.57:0/355103355 shutdown_connections 2023-10-19T13:04:41.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.173+0000 7feec259e640 1 -- 172.21.15.57:0/355103355 wait complete. 2023-10-19T13:04:41.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.174+0000 7feec259e640 1 Processor -- start 2023-10-19T13:04:41.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.174+0000 7feec259e640 1 -- start start 2023-10-19T13:04:41.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.175+0000 7feec259e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feebc0cdb30 0x7feebc159f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:41.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.175+0000 7feec259e640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7feebc159060 0x7feebc15c040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:41.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.175+0000 7feec259e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feebc15ccf0 0x7feebc0ae5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:41.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.175+0000 7feec259e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feebc15be20 con 0x7feebc0cdb30 2023-10-19T13:04:41.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.175+0000 7feec259e640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feebc1f2cd0 con 0x7feebc15ccf0 2023-10-19T13:04:41.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.175+0000 7feec259e640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feebc1f2f70 con 0x7feebc159060 2023-10-19T13:04:41.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.175+0000 7feebbfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feebc0cdb30 0x7feebc159f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:41.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.175+0000 7feebbfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feebc0cdb30 0x7feebc159f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57364/0 (socket says 172.21.15.57:57364) 2023-10-19T13:04:41.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.175+0000 7feebbfff640 1 -- 172.21.15.57:0/979470770 learned_addr learned my addr 172.21.15.57:0/979470770 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:41.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.175+0000 7feebb7fe640 1 --2- 172.21.15.57:0/979470770 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7feebc159060 0x7feebc15c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:41.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.175+0000 7feec0b3b640 1 --2- 172.21.15.57:0/979470770 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feebc15ccf0 0x7feebc0ae5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:41.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.175+0000 7feebbfff640 1 -- 172.21.15.57:0/979470770 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7feebc159060 msgr2=0x7feebc15c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:41.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.175+0000 7feebbfff640 1 --2- 172.21.15.57:0/979470770 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7feebc159060 0x7feebc15c040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:41.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.175+0000 7feebbfff640 1 -- 172.21.15.57:0/979470770 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feebc15ccf0 msgr2=0x7feebc0ae5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:41.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.175+0000 7feebbfff640 1 --2- 172.21.15.57:0/979470770 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feebc15ccf0 0x7feebc0ae5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:41.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.175+0000 7feebbfff640 1 -- 172.21.15.57:0/979470770 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feeac009560 con 0x7feebc0cdb30 2023-10-19T13:04:41.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.175+0000 7feebbfff640 1 --2- 172.21.15.57:0/979470770 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feebc0cdb30 0x7feebc159f90 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7feeac009850 tx=0x7feeac02d9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:41.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.176+0000 7feeb8ff9640 1 -- 172.21.15.57:0/979470770 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeac02ddd0 con 0x7feebc0cdb30 2023-10-19T13:04:41.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.176+0000 7feeb8ff9640 1 -- 172.21.15.57:0/979470770 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7feeac0036e0 con 0x7feebc0cdb30 2023-10-19T13:04:41.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.176+0000 7feec259e640 1 -- 172.21.15.57:0/979470770 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feebc1f3250 con 0x7feebc0cdb30 2023-10-19T13:04:41.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.176+0000 7feeb8ff9640 1 -- 172.21.15.57:0/979470770 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeac003a90 con 0x7feebc0cdb30 2023-10-19T13:04:41.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.176+0000 7feec259e640 1 -- 172.21.15.57:0/979470770 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feebc1f3810 con 0x7feebc0cdb30 2023-10-19T13:04:41.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.177+0000 7feec259e640 1 -- 172.21.15.57:0/979470770 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feebc0c6550 con 0x7feebc0cdb30 2023-10-19T13:04:41.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.177+0000 7feeb8ff9640 1 -- 172.21.15.57:0/979470770 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 75) v1 ==== 89388+0+0 (secure 0 0 0) 0x7feeac003bf0 con 0x7feebc0cdb30 2023-10-19T13:04:41.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.178+0000 7feeb8ff9640 1 --2- 172.21.15.57:0/979470770 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7feea806aff0 0x7feea806d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:41.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.178+0000 7feebb7fe640 1 -- 172.21.15.57:0/979470770 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7feea806aff0 msgr2=0x7feea806d4b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/3526956956 2023-10-19T13:04:41.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.178+0000 7feeb8ff9640 1 -- 172.21.15.57:0/979470770 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6401+0+0 (secure 0 0 0) 0x7feeac0ac670 con 0x7feebc0cdb30 2023-10-19T13:04:41.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.178+0000 7feebb7fe640 1 --2- 172.21.15.57:0/979470770 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7feea806aff0 0x7feea806d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T13:04:41.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.182+0000 7feeb8ff9640 1 -- 172.21.15.57:0/979470770 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7feeac077170 con 0x7feebc0cdb30 2023-10-19T13:04:41.273 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:41.270+0000 7f088fe45040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-19T13:04:41.291 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:04:41.289+0000 7fed52a3a040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-19T13:04:41.317 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:41.315+0000 7f088fe45040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-19T13:04:41.339 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:04:41.336+0000 7fed52a3a040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-19T13:04:41.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.364+0000 7feec259e640 1 -- 172.21.15.57:0/979470770 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feebc0cc4d0 con 0x7feebc0cdb30 2023-10-19T13:04:41.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.365+0000 7feeb8ff9640 1 -- 172.21.15.57:0/979470770 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v145) v1 ==== 74+0+17348 (secure 0 0 0) 0x7feeac07a790 con 0x7feebc0cdb30 2023-10-19T13:04:41.368 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:04:41.369 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":145,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T13:04:33.816818+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2023-10-19T13:04:20.932329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"143","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2023-10-19T13:04:21.762157+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"144","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.179:0/1700025841":"2023-10-20T13:04:33.816783+0000","172.21.15.179:0/3160483813":"2023-10-20T13:04:33.816783+0000","172.21.15.179:0/4039784875":"2023-10-20T13:04:33.816783+0000","172.21.15.179:0/1023851911":"2023-10-20T13:04:33.816783+0000","172.21.15.179:0/2968654981":"2023-10-20T13:04:33.816783+0000","172.21.15.179:0/3605212267":"2023-10-20T13:04:33.816783+0000","172.21.15.179:6833/2346394725":"2023-10-20T13:04:33.816783+0000","172.21.15.179:6832/2346394725":"2023-10-20T13:04:33.816783+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T13:04:41.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.367+0000 7feec259e640 1 -- 172.21.15.57:0/979470770 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7feea806aff0 msgr2=0x7feea806d4b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T13:04:41.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.367+0000 7feec259e640 1 --2- 172.21.15.57:0/979470770 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7feea806aff0 0x7feea806d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:41.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.367+0000 7feec259e640 1 -- 172.21.15.57:0/979470770 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feebc0cdb30 msgr2=0x7feebc159f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:41.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.367+0000 7feec259e640 1 --2- 172.21.15.57:0/979470770 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feebc0cdb30 0x7feebc159f90 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7feeac009850 tx=0x7feeac02d9d0 comp rx=0 tx=0).stop 2023-10-19T13:04:41.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.368+0000 7feec259e640 1 -- 172.21.15.57:0/979470770 shutdown_connections 2023-10-19T13:04:41.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.368+0000 7feec259e640 1 --2- 172.21.15.57:0/979470770 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7feea806aff0 0x7feea806d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:41.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.368+0000 7feec259e640 1 --2- 172.21.15.57:0/979470770 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feebc15ccf0 0x7feebc0ae5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:41.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.368+0000 7feec259e640 1 --2- 172.21.15.57:0/979470770 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7feebc159060 0x7feebc15c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:41.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.368+0000 7feec259e640 1 --2- 172.21.15.57:0/979470770 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feebc0cdb30 0x7feebc159f90 unknown :-1 s=CLOSED pgs=1658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:41.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.368+0000 7feec259e640 1 -- 172.21.15.57:0/979470770 >> 172.21.15.57:0/979470770 conn(0x7feebc0b0670 msgr2=0x7feebc0cae60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:41.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.368+0000 7feec259e640 1 -- 172.21.15.57:0/979470770 shutdown_connections 2023-10-19T13:04:41.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.368+0000 7feec259e640 1 -- 172.21.15.57:0/979470770 wait complete. 2023-10-19T13:04:41.382 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-19T12:44:07.371186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-19T13:04:20.932329+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '143', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_data', 'create_time': '2023-10-19T13:04:21.762157+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '144', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-19T13:04:41.383 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-19T13:04:41.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.494+0000 7f2593ae5640 1 -- 172.21.15.57:0/3637455476 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2578003970 msgr2=0x7f2578003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:41.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.494+0000 7f2593ae5640 1 --2- 172.21.15.57:0/3637455476 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2578003970 0x7f2578003d90 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7f257c009b80 tx=0x7f25780057f0 comp rx=0 tx=0).stop 2023-10-19T13:04:41.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.494+0000 7f2592082640 1 -- 172.21.15.57:0/3637455476 reap_dead start 2023-10-19T13:04:41.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.494+0000 7f2593ae5640 1 -- 172.21.15.57:0/3637455476 shutdown_connections 2023-10-19T13:04:41.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.494+0000 7f2593ae5640 1 -- 172.21.15.57:0/3637455476 >> 172.21.15.57:0/3637455476 conn(0x7f258c09a030 msgr2=0x7f258c0d0d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:41.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.495+0000 7f2593ae5640 1 -- 172.21.15.57:0/3637455476 shutdown_connections 2023-10-19T13:04:41.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.495+0000 7f2593ae5640 1 -- 172.21.15.57:0/3637455476 wait complete. 2023-10-19T13:04:41.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.496+0000 7f2593ae5640 1 Processor -- start 2023-10-19T13:04:41.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.496+0000 7f2593ae5640 1 -- start start 2023-10-19T13:04:41.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.498+0000 7f2593ae5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2578003970 0x7f258c0ae1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:41.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.498+0000 7f2593ae5640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f258c0b3a70 0x7f258c151580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:41.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.498+0000 7f2593ae5640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f258c0b3ef0 0x7f258c0ca560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:41.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.498+0000 7f2593ae5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f258c141ba0 con 0x7f2578003970 2023-10-19T13:04:41.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.498+0000 7f2593ae5640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f258c0ae8d0 con 0x7f258c0b3ef0 2023-10-19T13:04:41.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.498+0000 7f2593ae5640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f258c0aebb0 con 0x7f258c0b3a70 2023-10-19T13:04:41.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.498+0000 7f2591881640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2578003970 0x7f258c0ae1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:41.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.498+0000 7f2591881640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2578003970 0x7f258c0ae1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57390/0 (socket says 172.21.15.57:57390) 2023-10-19T13:04:41.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.498+0000 7f2591881640 1 -- 172.21.15.57:0/3068574477 learned_addr learned my addr 172.21.15.57:0/3068574477 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:41.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.498+0000 7f2591080640 1 --2- 172.21.15.57:0/3068574477 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f258c0b3a70 0x7f258c151580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:41.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.498+0000 7f2592082640 1 --2- 172.21.15.57:0/3068574477 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f258c0b3ef0 0x7f258c0ca560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:41.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.498+0000 7f2591881640 1 -- 172.21.15.57:0/3068574477 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f258c0b3a70 msgr2=0x7f258c151580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:41.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.498+0000 7f2591881640 1 --2- 172.21.15.57:0/3068574477 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f258c0b3a70 0x7f258c151580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:41.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.498+0000 7f2591881640 1 -- 172.21.15.57:0/3068574477 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f258c0b3ef0 msgr2=0x7f258c0ca560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:41.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.498+0000 7f2591881640 1 --2- 172.21.15.57:0/3068574477 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f258c0b3ef0 0x7f258c0ca560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:41.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.498+0000 7f2591881640 1 -- 172.21.15.57:0/3068574477 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f258c0aee90 con 0x7f2578003970 2023-10-19T13:04:41.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.498+0000 7f2591881640 1 --2- 172.21.15.57:0/3068574477 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2578003970 0x7f258c0ae1f0 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7f257c009ca0 tx=0x7f257c033bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:41.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.498+0000 7f25827fc640 1 -- 172.21.15.57:0/3068574477 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f257c03d8e0 con 0x7f2578003970 2023-10-19T13:04:41.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.498+0000 7f25827fc640 1 -- 172.21.15.57:0/3068574477 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f257c03da40 con 0x7f2578003970 2023-10-19T13:04:41.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.498+0000 7f2593ae5640 1 -- 172.21.15.57:0/3068574477 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f258c0af170 con 0x7f2578003970 2023-10-19T13:04:41.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.498+0000 7f25827fc640 1 -- 172.21.15.57:0/3068574477 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f257c03de50 con 0x7f2578003970 2023-10-19T13:04:41.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.499+0000 7f2593ae5640 1 -- 172.21.15.57:0/3068574477 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f258c151ca0 con 0x7f2578003970 2023-10-19T13:04:41.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.500+0000 7f2593ae5640 1 -- 172.21.15.57:0/3068574477 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f258c0675f0 con 0x7f2578003970 2023-10-19T13:04:41.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.500+0000 7f25827fc640 1 -- 172.21.15.57:0/3068574477 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 75) v1 ==== 89388+0+0 (secure 0 0 0) 0x7f257c045400 con 0x7f2578003970 2023-10-19T13:04:41.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.500+0000 7f25827fc640 1 --2- 172.21.15.57:0/3068574477 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7f257006b020 0x7f257006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:41.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.500+0000 7f2591080640 1 -- 172.21.15.57:0/3068574477 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7f257006b020 msgr2=0x7f257006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/3526956956 2023-10-19T13:04:41.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.500+0000 7f25827fc640 1 -- 172.21.15.57:0/3068574477 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f257c0acae0 con 0x7f2578003970 2023-10-19T13:04:41.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.500+0000 7f2591080640 1 --2- 172.21.15.57:0/3068574477 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7f257006b020 0x7f257006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T13:04:41.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.505+0000 7f25827fc640 1 -- 172.21.15.57:0/3068574477 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f257c0775f0 con 0x7f2578003970 2023-10-19T13:04:41.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.682+0000 7f2593ae5640 1 -- 172.21.15.57:0/3068574477 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f258c0b1650 con 0x7f2578003970 2023-10-19T13:04:41.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.682+0000 7f25827fc640 1 -- 172.21.15.57:0/3068574477 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v145) v1 ==== 93+0+10 (secure 0 0 0) 0x7f257c0ac030 con 0x7f2578003970 2023-10-19T13:04:41.686 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-19T13:04:41.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.685+0000 7f2593ae5640 1 -- 172.21.15.57:0/3068574477 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7f257006b020 msgr2=0x7f257006d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T13:04:41.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.685+0000 7f2593ae5640 1 --2- 172.21.15.57:0/3068574477 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7f257006b020 0x7f257006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:41.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.685+0000 7f2593ae5640 1 -- 172.21.15.57:0/3068574477 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2578003970 msgr2=0x7f258c0ae1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:41.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.685+0000 7f2593ae5640 1 --2- 172.21.15.57:0/3068574477 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2578003970 0x7f258c0ae1f0 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7f257c009ca0 tx=0x7f257c033bd0 comp rx=0 tx=0).stop 2023-10-19T13:04:41.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.685+0000 7f2593ae5640 1 -- 172.21.15.57:0/3068574477 shutdown_connections 2023-10-19T13:04:41.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.685+0000 7f2593ae5640 1 --2- 172.21.15.57:0/3068574477 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7f257006b020 0x7f257006d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:41.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.685+0000 7f2593ae5640 1 --2- 172.21.15.57:0/3068574477 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f258c0b3ef0 0x7f258c0ca560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:41.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.685+0000 7f2593ae5640 1 --2- 172.21.15.57:0/3068574477 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f258c0b3a70 0x7f258c151580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:41.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.685+0000 7f2593ae5640 1 --2- 172.21.15.57:0/3068574477 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2578003970 0x7f258c0ae1f0 unknown :-1 s=CLOSED pgs=1661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:41.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.685+0000 7f2593ae5640 1 -- 172.21.15.57:0/3068574477 >> 172.21.15.57:0/3068574477 conn(0x7f258c09a030 msgr2=0x7f258c0b0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:41.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.685+0000 7f2593ae5640 1 -- 172.21.15.57:0/3068574477 shutdown_connections 2023-10-19T13:04:41.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.685+0000 7f2593ae5640 1 -- 172.21.15.57:0/3068574477 wait complete. 2023-10-19T13:04:41.701 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-19T13:04:41.706 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:41.704+0000 7f088fe45040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-19T13:04:41.750 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:41.748+0000 7f088fe45040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-19T13:04:41.763 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:04:41.761+0000 7fed52a3a040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-19T13:04:41.792 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:41.789+0000 7f088fe45040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-19T13:04:41.810 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:04:41.807+0000 7fed52a3a040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-19T13:04:41.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.811+0000 7f5789087640 1 -- 172.21.15.57:0/1869503602 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f578415b2a0 msgr2=0x7f57840cf720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:41.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.811+0000 7f5789087640 1 --2- 172.21.15.57:0/1869503602 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f578415b2a0 0x7f57840cf720 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f5774009830 tx=0x7f577402bcd0 comp rx=0 tx=0).stop 2023-10-19T13:04:41.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.811+0000 7f5789087640 1 -- 172.21.15.57:0/1869503602 shutdown_connections 2023-10-19T13:04:41.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.811+0000 7f5789087640 1 --2- 172.21.15.57:0/1869503602 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f578415b2a0 0x7f57840cf720 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:41.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.811+0000 7f5789087640 1 --2- 172.21.15.57:0/1869503602 >> v2:172.21.15.179:3300/0 conn(0x7f57840cf340 0x7f57840d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:41.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.811+0000 7f5789087640 1 -- 172.21.15.57:0/1869503602 >> 172.21.15.57:0/1869503602 conn(0x7f57840d0d60 msgr2=0x7f57840d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:41.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.811+0000 7f5789087640 1 -- 172.21.15.57:0/1869503602 shutdown_connections 2023-10-19T13:04:41.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.811+0000 7f5789087640 1 -- 172.21.15.57:0/1869503602 wait complete. 2023-10-19T13:04:41.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.812+0000 7f5789087640 1 Processor -- start 2023-10-19T13:04:41.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.813+0000 7f5789087640 1 -- start start 2023-10-19T13:04:41.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.813+0000 7f5789087640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f57840ceb40 0x7f57841f4510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:41.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.813+0000 7f5789087640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f57840cf340 0x7f5784158ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:41.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.813+0000 7f5789087640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f578415b2a0 0x7f57840b6310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:41.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.813+0000 7f5789087640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f578415b800 con 0x7f578415b2a0 2023-10-19T13:04:41.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.813+0000 7f5789087640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57840b6850 con 0x7f57840ceb40 2023-10-19T13:04:41.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.813+0000 7f5789087640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57841f4af0 con 0x7f57840cf340 2023-10-19T13:04:41.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.813+0000 7f578359e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f578415b2a0 0x7f57840b6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:41.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.813+0000 7f578359e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f578415b2a0 0x7f57840b6310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57402/0 (socket says 172.21.15.57:57402) 2023-10-19T13:04:41.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.813+0000 7f578359e640 1 -- 172.21.15.57:0/2959513704 learned_addr learned my addr 172.21.15.57:0/2959513704 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:41.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.813+0000 7f5782d9d640 1 --2- 172.21.15.57:0/2959513704 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f57840ceb40 0x7f57841f4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:41.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.813+0000 7f578259c640 1 --2- 172.21.15.57:0/2959513704 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f57840cf340 0x7f5784158ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:41.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.814+0000 7f578359e640 1 -- 172.21.15.57:0/2959513704 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f57840cf340 msgr2=0x7f5784158ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:41.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.814+0000 7f578359e640 1 --2- 172.21.15.57:0/2959513704 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f57840cf340 0x7f5784158ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:41.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.814+0000 7f578359e640 1 -- 172.21.15.57:0/2959513704 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f57840ceb40 msgr2=0x7f57841f4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:41.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.814+0000 7f578359e640 1 --2- 172.21.15.57:0/2959513704 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f57840ceb40 0x7f57841f4510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:41.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.814+0000 7f578359e640 1 -- 172.21.15.57:0/2959513704 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5774009490 con 0x7f578415b2a0 2023-10-19T13:04:41.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.814+0000 7f578259c640 1 --2- 172.21.15.57:0/2959513704 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f57840cf340 0x7f5784158ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T13:04:41.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.814+0000 7f578359e640 1 --2- 172.21.15.57:0/2959513704 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f578415b2a0 0x7f57840b6310 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7f577c00cbc0 tx=0x7f577c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:41.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.814+0000 7f576b7fe640 1 -- 172.21.15.57:0/2959513704 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f577c00ad40 con 0x7f578415b2a0 2023-10-19T13:04:41.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.814+0000 7f576b7fe640 1 -- 172.21.15.57:0/2959513704 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f577c012420 con 0x7f578415b2a0 2023-10-19T13:04:41.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.814+0000 7f5789087640 1 -- 172.21.15.57:0/2959513704 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57841f4dd0 con 0x7f578415b2a0 2023-10-19T13:04:41.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.814+0000 7f576b7fe640 1 -- 172.21.15.57:0/2959513704 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f577c012850 con 0x7f578415b2a0 2023-10-19T13:04:41.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.814+0000 7f5789087640 1 -- 172.21.15.57:0/2959513704 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57841f5390 con 0x7f578415b2a0 2023-10-19T13:04:41.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.815+0000 7f5789087640 1 -- 172.21.15.57:0/2959513704 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f578415b940 con 0x7f578415b2a0 2023-10-19T13:04:41.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.816+0000 7f576b7fe640 1 -- 172.21.15.57:0/2959513704 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 75) v1 ==== 89388+0+0 (secure 0 0 0) 0x7f577c01a3b0 con 0x7f578415b2a0 2023-10-19T13:04:41.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.816+0000 7f576b7fe640 1 --2- 172.21.15.57:0/2959513704 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7f577006aff0 0x7f577006d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:41.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.816+0000 7f5782d9d640 1 -- 172.21.15.57:0/2959513704 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7f577006aff0 msgr2=0x7f577006d4b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/3526956956 2023-10-19T13:04:41.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.816+0000 7f576b7fe640 1 -- 172.21.15.57:0/2959513704 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f577c08a380 con 0x7f578415b2a0 2023-10-19T13:04:41.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.816+0000 7f5782d9d640 1 --2- 172.21.15.57:0/2959513704 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7f577006aff0 0x7f577006d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T13:04:41.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.820+0000 7f576b7fe640 1 -- 172.21.15.57:0/2959513704 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f578415b940 con 0x7f578415b2a0 2023-10-19T13:04:41.844 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:41.842+0000 7f088fe45040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-19T13:04:41.856 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:04:41.854+0000 7fed52a3a040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-19T13:04:41.885 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:41.882+0000 7f088fe45040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-19T13:04:41.911 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:04:41.909+0000 7fed52a3a040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-19T13:04:41.955 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:04:41.952+0000 7fed52a3a040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-19T13:04:41.983 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:41.981+0000 7f088fe45040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-19T13:04:42.000 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.997+0000 7f5789087640 1 -- 172.21.15.57:0/2959513704 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f578415b940 con 0x7f578415b2a0 2023-10-19T13:04:42.001 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:41.998+0000 7f576b7fe640 1 -- 172.21.15.57:0/2959513704 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v145) v1 ==== 104+0+11 (secure 0 0 0) 0x7f578415b940 con 0x7f578415b2a0 2023-10-19T13:04:42.001 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2023-10-19T13:04:42.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.000+0000 7f5789087640 1 -- 172.21.15.57:0/2959513704 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7f577006aff0 msgr2=0x7f577006d4b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T13:04:42.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.000+0000 7f5789087640 1 --2- 172.21.15.57:0/2959513704 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7f577006aff0 0x7f577006d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:42.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.000+0000 7f5789087640 1 -- 172.21.15.57:0/2959513704 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f578415b2a0 msgr2=0x7f57840b6310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:42.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.000+0000 7f5789087640 1 --2- 172.21.15.57:0/2959513704 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f578415b2a0 0x7f57840b6310 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7f577c00cbc0 tx=0x7f577c00a430 comp rx=0 tx=0).stop 2023-10-19T13:04:42.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.000+0000 7f5789087640 1 -- 172.21.15.57:0/2959513704 shutdown_connections 2023-10-19T13:04:42.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.000+0000 7f5789087640 1 --2- 172.21.15.57:0/2959513704 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7f577006aff0 0x7f577006d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:42.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.000+0000 7f5789087640 1 --2- 172.21.15.57:0/2959513704 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f578415b2a0 0x7f57840b6310 unknown :-1 s=CLOSED pgs=1663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:42.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.000+0000 7f5789087640 1 --2- 172.21.15.57:0/2959513704 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f57840cf340 0x7f5784158ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:42.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.000+0000 7f5789087640 1 --2- 172.21.15.57:0/2959513704 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f57840ceb40 0x7f57841f4510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:42.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.000+0000 7f5789087640 1 -- 172.21.15.57:0/2959513704 >> 172.21.15.57:0/2959513704 conn(0x7f57840d0d60 msgr2=0x7f578406a9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:42.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.000+0000 7f5789087640 1 -- 172.21.15.57:0/2959513704 shutdown_connections 2023-10-19T13:04:42.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.000+0000 7f5789087640 1 -- 172.21.15.57:0/2959513704 wait complete. 2023-10-19T13:04:42.014 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-19T13:04:42.036 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:42.033+0000 7f088fe45040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-19T13:04:42.060 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:04:42.058+0000 7fed52a3a040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-19T13:04:42.077 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:42.075+0000 7f088fe45040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-19T13:04:42.117 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:04:42.115+0000 7fed52a3a040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-19T13:04:42.125 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:42.122+0000 7f088fe45040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-19T13:04:42.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.125+0000 7fab21d6e640 1 -- 172.21.15.57:0/3288193159 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fab1c0ceb40 msgr2=0x7fab1c0b0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:42.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.125+0000 7fab21d6e640 1 --2- 172.21.15.57:0/3288193159 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fab1c0ceb40 0x7fab1c0b0650 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fab14005fd0 tx=0x7fab1402c050 comp rx=0 tx=0).stop 2023-10-19T13:04:42.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.126+0000 7fab21d6e640 1 -- 172.21.15.57:0/3288193159 shutdown_connections 2023-10-19T13:04:42.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.126+0000 7fab21d6e640 1 --2- 172.21.15.57:0/3288193159 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fab1c0ceb40 0x7fab1c0b0650 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:42.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.126+0000 7fab21d6e640 1 --2- 172.21.15.57:0/3288193159 >> v2:172.21.15.179:3300/0 conn(0x7fab1c15b2a0 0x7fab1c0d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:42.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.126+0000 7fab21d6e640 1 -- 172.21.15.57:0/3288193159 >> 172.21.15.57:0/3288193159 conn(0x7fab1c0d0d60 msgr2=0x7fab1c0d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:42.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.126+0000 7fab21d6e640 1 -- 172.21.15.57:0/3288193159 shutdown_connections 2023-10-19T13:04:42.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.126+0000 7fab21d6e640 1 -- 172.21.15.57:0/3288193159 wait complete. 2023-10-19T13:04:42.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.127+0000 7fab21d6e640 1 Processor -- start 2023-10-19T13:04:42.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.127+0000 7fab21d6e640 1 -- start start 2023-10-19T13:04:42.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.128+0000 7fab21d6e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fab1c0ceb40 0x7fab1c1f1540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:42.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.128+0000 7fab21d6e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab1c0cf340 0x7fab1c0c9e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:42.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.128+0000 7fab21d6e640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fab1c15b2a0 0x7fab1c0ca380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:42.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.128+0000 7fab21d6e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab1c1f5b30 con 0x7fab1c0cf340 2023-10-19T13:04:42.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.128+0000 7fab21d6e640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab1c0c8600 con 0x7fab1c0ceb40 2023-10-19T13:04:42.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.128+0000 7fab21d6e640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab1c1f5570 con 0x7fab1c15b2a0 2023-10-19T13:04:42.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.128+0000 7fab1affd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab1c0cf340 0x7fab1c0c9e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:42.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.128+0000 7fab1affd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab1c0cf340 0x7fab1c0c9e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57414/0 (socket says 172.21.15.57:57414) 2023-10-19T13:04:42.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.128+0000 7fab1affd640 1 -- 172.21.15.57:0/646084509 learned_addr learned my addr 172.21.15.57:0/646084509 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:42.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.128+0000 7fab1bfff640 1 --2- 172.21.15.57:0/646084509 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fab1c15b2a0 0x7fab1c0ca380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:42.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.128+0000 7fab1b7fe640 1 --2- 172.21.15.57:0/646084509 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fab1c0ceb40 0x7fab1c1f1540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:42.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.128+0000 7fab1affd640 1 -- 172.21.15.57:0/646084509 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fab1c15b2a0 msgr2=0x7fab1c0ca380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:42.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.128+0000 7fab1affd640 1 --2- 172.21.15.57:0/646084509 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fab1c15b2a0 0x7fab1c0ca380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:42.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.128+0000 7fab1affd640 1 -- 172.21.15.57:0/646084509 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fab1c0ceb40 msgr2=0x7fab1c1f1540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:42.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.128+0000 7fab1affd640 1 --2- 172.21.15.57:0/646084509 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fab1c0ceb40 0x7fab1c1f1540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:42.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.128+0000 7fab1affd640 1 -- 172.21.15.57:0/646084509 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab14009840 con 0x7fab1c0cf340 2023-10-19T13:04:42.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.128+0000 7fab1affd640 1 --2- 172.21.15.57:0/646084509 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab1c0cf340 0x7fab1c0c9e40 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7fab0400e790 tx=0x7fab0400ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:42.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.128+0000 7fab13fff640 1 -- 172.21.15.57:0/646084509 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab04009580 con 0x7fab1c0cf340 2023-10-19T13:04:42.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.129+0000 7fab21d6e640 1 -- 172.21.15.57:0/646084509 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab1c0b0d50 con 0x7fab1c0cf340 2023-10-19T13:04:42.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.129+0000 7fab13fff640 1 -- 172.21.15.57:0/646084509 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fab0400cdb0 con 0x7fab1c0cf340 2023-10-19T13:04:42.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.129+0000 7fab13fff640 1 -- 172.21.15.57:0/646084509 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab04010690 con 0x7fab1c0cf340 2023-10-19T13:04:42.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.129+0000 7fab21d6e640 1 -- 172.21.15.57:0/646084509 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab1c0c7fb0 con 0x7fab1c0cf340 2023-10-19T13:04:42.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.129+0000 7fab21d6e640 1 -- 172.21.15.57:0/646084509 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab1c0c7e10 con 0x7fab1c0cf340 2023-10-19T13:04:42.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.130+0000 7fab13fff640 1 -- 172.21.15.57:0/646084509 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 75) v1 ==== 89388+0+0 (secure 0 0 0) 0x7fab040107f0 con 0x7fab1c0cf340 2023-10-19T13:04:42.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.130+0000 7fab13fff640 1 --2- 172.21.15.57:0/646084509 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7faafc06aff0 0x7faafc06d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:42.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.131+0000 7fab1b7fe640 1 -- 172.21.15.57:0/646084509 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7faafc06aff0 msgr2=0x7faafc06d4b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/3526956956 2023-10-19T13:04:42.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.131+0000 7fab13fff640 1 -- 172.21.15.57:0/646084509 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6401+0+0 (secure 0 0 0) 0x7fab04014070 con 0x7fab1c0cf340 2023-10-19T13:04:42.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.131+0000 7fab1b7fe640 1 --2- 172.21.15.57:0/646084509 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7faafc06aff0 0x7faafc06d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T13:04:42.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.135+0000 7fab13fff640 1 -- 172.21.15.57:0/646084509 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fab04051ca0 con 0x7fab1c0cf340 2023-10-19T13:04:42.163 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:04:42.160+0000 7fed52a3a040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-19T13:04:42.168 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:42.165+0000 7f088fe45040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-19T13:04:42.214 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:04:42.211+0000 7fed52a3a040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-19T13:04:42.258 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:04:42.256+0000 7fed52a3a040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-19T13:04:42.315 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.313+0000 7fab21d6e640 1 -- 172.21.15.57:0/646084509 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fab1c0d07d0 con 0x7fab1c0cf340 2023-10-19T13:04:42.316 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.313+0000 7fab13fff640 1 -- 172.21.15.57:0/646084509 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v145) v1 ==== 100+0+11 (secure 0 0 0) 0x7fab1c0c7e10 con 0x7fab1c0cf340 2023-10-19T13:04:42.316 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 64 2023-10-19T13:04:42.318 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.315+0000 7fab21d6e640 1 -- 172.21.15.57:0/646084509 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7faafc06aff0 msgr2=0x7faafc06d4b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T13:04:42.318 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.315+0000 7fab21d6e640 1 --2- 172.21.15.57:0/646084509 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7faafc06aff0 0x7faafc06d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:42.318 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.315+0000 7fab21d6e640 1 -- 172.21.15.57:0/646084509 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab1c0cf340 msgr2=0x7fab1c0c9e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:42.318 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.315+0000 7fab21d6e640 1 --2- 172.21.15.57:0/646084509 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab1c0cf340 0x7fab1c0c9e40 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7fab0400e790 tx=0x7fab0400ec50 comp rx=0 tx=0).stop 2023-10-19T13:04:42.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.315+0000 7fab21d6e640 1 -- 172.21.15.57:0/646084509 shutdown_connections 2023-10-19T13:04:42.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.315+0000 7fab21d6e640 1 --2- 172.21.15.57:0/646084509 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7faafc06aff0 0x7faafc06d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:42.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.315+0000 7fab21d6e640 1 --2- 172.21.15.57:0/646084509 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fab1c15b2a0 0x7fab1c0ca380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:42.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.315+0000 7fab21d6e640 1 --2- 172.21.15.57:0/646084509 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab1c0cf340 0x7fab1c0c9e40 unknown :-1 s=CLOSED pgs=1665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:42.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.315+0000 7fab21d6e640 1 --2- 172.21.15.57:0/646084509 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fab1c0ceb40 0x7fab1c1f1540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:42.320 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.315+0000 7fab21d6e640 1 -- 172.21.15.57:0/646084509 >> 172.21.15.57:0/646084509 conn(0x7fab1c0d0d60 msgr2=0x7fab1c0cca00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:42.320 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.315+0000 7fab21d6e640 1 -- 172.21.15.57:0/646084509 shutdown_connections 2023-10-19T13:04:42.320 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.315+0000 7fab21d6e640 1 -- 172.21.15.57:0/646084509 wait complete. 2023-10-19T13:04:42.329 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-19T13:04:42.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.438+0000 7f90dfb16640 1 -- 172.21.15.57:0/1433611993 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f90d80cf340 msgr2=0x7f90d80d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:42.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.438+0000 7f90dfb16640 1 --2- 172.21.15.57:0/1433611993 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f90d80cf340 0x7f90d80d1910 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7f90d815b680 tx=0x7f90d402bdb0 comp rx=0 tx=0).stop 2023-10-19T13:04:42.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.438+0000 7f90dfb16640 1 -- 172.21.15.57:0/1433611993 shutdown_connections 2023-10-19T13:04:42.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.438+0000 7f90dfb16640 1 --2- 172.21.15.57:0/1433611993 >> v2:172.21.15.179:3300/0 conn(0x7f90d80ceb40 0x7f90d80b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:42.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.438+0000 7f90dfb16640 1 --2- 172.21.15.57:0/1433611993 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f90d80cf340 0x7f90d80d1910 unknown :-1 s=CLOSED pgs=1666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:42.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.438+0000 7f90dfb16640 1 -- 172.21.15.57:0/1433611993 >> 172.21.15.57:0/1433611993 conn(0x7f90d80d0d60 msgr2=0x7f90d80d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:42.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.439+0000 7f90dfb16640 1 -- 172.21.15.57:0/1433611993 shutdown_connections 2023-10-19T13:04:42.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.439+0000 7f90dfb16640 1 -- 172.21.15.57:0/1433611993 wait complete. 2023-10-19T13:04:42.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.440+0000 7f90dfb16640 1 Processor -- start 2023-10-19T13:04:42.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.440+0000 7f90dfb16640 1 -- start start 2023-10-19T13:04:42.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.440+0000 7f90dfb16640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f90d80ceb40 0x7f90d81e5d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:42.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.440+0000 7f90dfb16640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f90d80cf340 0x7f90d8159410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:42.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.440+0000 7f90dfb16640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f90d815b2a0 0x7f90d80ca950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:42.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.440+0000 7f90dfb16640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90d80c9c20 con 0x7f90d80cf340 2023-10-19T13:04:42.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.440+0000 7f90dfb16640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90d80cae90 con 0x7f90d80ceb40 2023-10-19T13:04:42.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.440+0000 7f90dfb16640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90d80cb0c0 con 0x7f90d815b2a0 2023-10-19T13:04:42.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.441+0000 7f90dd0b1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f90d80cf340 0x7f90d8159410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:42.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.441+0000 7f90dd0b1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f90d80cf340 0x7f90d8159410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57422/0 (socket says 172.21.15.57:57422) 2023-10-19T13:04:42.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.441+0000 7f90dd0b1640 1 -- 172.21.15.57:0/2189824139 learned_addr learned my addr 172.21.15.57:0/2189824139 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:42.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.441+0000 7f90dd8b2640 1 --2- 172.21.15.57:0/2189824139 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f90d80ceb40 0x7f90d81e5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:42.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.441+0000 7f90de0b3640 1 --2- 172.21.15.57:0/2189824139 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f90d815b2a0 0x7f90d80ca950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:42.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.441+0000 7f90dd0b1640 1 -- 172.21.15.57:0/2189824139 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f90d815b2a0 msgr2=0x7f90d80ca950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:42.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.441+0000 7f90dd0b1640 1 --2- 172.21.15.57:0/2189824139 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f90d815b2a0 0x7f90d80ca950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:42.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.441+0000 7f90dd0b1640 1 -- 172.21.15.57:0/2189824139 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f90d80ceb40 msgr2=0x7f90d81e5d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:42.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.441+0000 7f90dd0b1640 1 --2- 172.21.15.57:0/2189824139 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f90d80ceb40 0x7f90d81e5d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:42.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.441+0000 7f90dd0b1640 1 -- 172.21.15.57:0/2189824139 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90d4009560 con 0x7f90d80cf340 2023-10-19T13:04:42.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.441+0000 7f90de0b3640 1 --2- 172.21.15.57:0/2189824139 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f90d815b2a0 0x7f90d80ca950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T13:04:42.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.441+0000 7f90dd0b1640 1 --2- 172.21.15.57:0/2189824139 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f90d80cf340 0x7f90d8159410 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7f90c800c430 tx=0x7f90c800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:42.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.441+0000 7f90ce7fc640 1 -- 172.21.15.57:0/2189824139 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90c80052d0 con 0x7f90d80cf340 2023-10-19T13:04:42.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.441+0000 7f90ce7fc640 1 -- 172.21.15.57:0/2189824139 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f90c80035f0 con 0x7f90d80cf340 2023-10-19T13:04:42.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.441+0000 7f90dfb16640 1 -- 172.21.15.57:0/2189824139 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90d80c9280 con 0x7f90d80cf340 2023-10-19T13:04:42.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.441+0000 7f90ce7fc640 1 -- 172.21.15.57:0/2189824139 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90c8003a00 con 0x7f90d80cf340 2023-10-19T13:04:42.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.441+0000 7f90dfb16640 1 -- 172.21.15.57:0/2189824139 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90d80c9840 con 0x7f90d80cf340 2023-10-19T13:04:42.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.442+0000 7f90dfb16640 1 -- 172.21.15.57:0/2189824139 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90d80690e0 con 0x7f90d80cf340 2023-10-19T13:04:42.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.443+0000 7f90ce7fc640 1 -- 172.21.15.57:0/2189824139 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 75) v1 ==== 89388+0+0 (secure 0 0 0) 0x7f90c8003b60 con 0x7f90d80cf340 2023-10-19T13:04:42.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.443+0000 7f90ce7fc640 1 --2- 172.21.15.57:0/2189824139 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7f90bc06aff0 0x7f90bc06d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:42.452 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.443+0000 7f90dd8b2640 1 -- 172.21.15.57:0/2189824139 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7f90bc06aff0 msgr2=0x7f90bc06d4b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/3526956956 2023-10-19T13:04:42.452 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.443+0000 7f90ce7fc640 1 -- 172.21.15.57:0/2189824139 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f90c8089290 con 0x7f90d80cf340 2023-10-19T13:04:42.452 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.444+0000 7f90dd8b2640 1 --2- 172.21.15.57:0/2189824139 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7f90bc06aff0 0x7f90bc06d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T13:04:42.452 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.448+0000 7f90ce7fc640 1 -- 172.21.15.57:0/2189824139 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f90c8053f60 con 0x7f90d80cf340 2023-10-19T13:04:42.486 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:42.483+0000 7f088fe45040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-19T13:04:42.598 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:04:42.595+0000 7fed52a3a040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-19T13:04:42.614 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:42.611+0000 7f088fe45040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-19T13:04:42.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.636+0000 7f90dfb16640 1 -- 172.21.15.57:0/2189824139 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f90d80b0d00 con 0x7f90d80cf340 2023-10-19T13:04:42.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.636+0000 7f90ce7fc640 1 -- 172.21.15.57:0/2189824139 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v145) v1 ==== 80+0+478 (secure 0 0 0) 0x7f90c80171f0 con 0x7f90d80cf340 2023-10-19T13:04:42.639 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/1700025841 2023-10-20T13:04:33.816783+0000 2023-10-19T13:04:42.640 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/3160483813 2023-10-20T13:04:33.816783+0000 2023-10-19T13:04:42.640 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/4039784875 2023-10-20T13:04:33.816783+0000 2023-10-19T13:04:42.640 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/1023851911 2023-10-20T13:04:33.816783+0000 2023-10-19T13:04:42.640 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/2968654981 2023-10-20T13:04:33.816783+0000 2023-10-19T13:04:42.640 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/3605212267 2023-10-20T13:04:33.816783+0000 2023-10-19T13:04:42.640 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:6833/2346394725 2023-10-20T13:04:33.816783+0000 2023-10-19T13:04:42.641 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:6832/2346394725 2023-10-20T13:04:33.816783+0000 2023-10-19T13:04:42.642 INFO:teuthology.orchestra.run.smithi057.stderr:listed 8 entries 2023-10-19T13:04:42.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.638+0000 7f90dfb16640 1 -- 172.21.15.57:0/2189824139 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7f90bc06aff0 msgr2=0x7f90bc06d4b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T13:04:42.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.638+0000 7f90dfb16640 1 --2- 172.21.15.57:0/2189824139 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7f90bc06aff0 0x7f90bc06d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:42.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.638+0000 7f90dfb16640 1 -- 172.21.15.57:0/2189824139 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f90d80cf340 msgr2=0x7f90d8159410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:42.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.638+0000 7f90dfb16640 1 --2- 172.21.15.57:0/2189824139 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f90d80cf340 0x7f90d8159410 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7f90c800c430 tx=0x7f90c800c8f0 comp rx=0 tx=0).stop 2023-10-19T13:04:42.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.638+0000 7f90dfb16640 1 -- 172.21.15.57:0/2189824139 shutdown_connections 2023-10-19T13:04:42.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.638+0000 7f90dfb16640 1 --2- 172.21.15.57:0/2189824139 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7f90bc06aff0 0x7f90bc06d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:42.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.638+0000 7f90dfb16640 1 --2- 172.21.15.57:0/2189824139 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f90d815b2a0 0x7f90d80ca950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:42.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.638+0000 7f90dfb16640 1 --2- 172.21.15.57:0/2189824139 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f90d80cf340 0x7f90d8159410 unknown :-1 s=CLOSED pgs=1667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:42.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.638+0000 7f90dfb16640 1 --2- 172.21.15.57:0/2189824139 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f90d80ceb40 0x7f90d81e5d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:42.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.638+0000 7f90dfb16640 1 -- 172.21.15.57:0/2189824139 >> 172.21.15.57:0/2189824139 conn(0x7f90d80d0d60 msgr2=0x7f90d815d6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:42.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.638+0000 7f90dfb16640 1 -- 172.21.15.57:0/2189824139 shutdown_connections 2023-10-19T13:04:42.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:42.638+0000 7f90dfb16640 1 -- 172.21.15.57:0/2189824139 wait complete. 2023-10-19T13:04:42.652 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi057.front.sepia.ceph.com... 2023-10-19T13:04:42.652 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:04:42.652 DEBUG:teuthology.orchestra.run.smithi057:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-10-19T13:04:42.681 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi057.stderr:ceph-fuse[132192]: fuse finished with error 0 and tester_r 0 2023-10-19T13:04:42.683 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi057.front.sepia.ceph.com 2023-10-19T13:04:42.684 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:04:42.684 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-19T13:04:42.705 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:42.703+0000 7f088fe45040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-19T13:04:42.740 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:04:42.737+0000 7fed52a3a040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-19T13:04:42.747 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:42.745+0000 7f088fe45040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-19T13:04:42.836 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:04:42.833+0000 7fed52a3a040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-19T13:04:42.842 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:42.839+0000 7f088fe45040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-19T13:04:42.848 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-19T13:04:42.848 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T13:04:42.864 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054833,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=284 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3247632k,nr_inodes=811908,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-19T13:04:42.864 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi179.front.sepia.ceph.com 2023-10-19T13:04:42.864 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:04:42.865 DEBUG:teuthology.orchestra.run.smithi179:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-10-19T13:04:42.880 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:04:42.878+0000 7fed52a3a040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-19T13:04:42.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-19T13:04:42.918 INFO:teuthology.orchestra.run.smithi179.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2023-10-19T13:04:42.919 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T13:04:42.919 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T13:04:42.935 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055614,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=358 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248260k,nr_inodes=812065,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-19T13:04:42.935 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi057.front.sepia.ceph.com 2023-10-19T13:04:42.935 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:04:42.936 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-10-19T13:04:42.938 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:42.936+0000 7f088fe45040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-19T13:04:42.980 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:04:42.978+0000 7fed52a3a040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-19T13:04:42.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-19T13:04:42.991 INFO:teuthology.orchestra.run.smithi057.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2023-10-19T13:04:42.991 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-19T13:04:42.991 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T13:04:43.007 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054833,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=284 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3247632k,nr_inodes=811908,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-19T13:04:43.007 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi179.front.sepia.ceph.com 2023-10-19T13:04:43.008 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:04:43.008 DEBUG:teuthology.orchestra.run.smithi179:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-10-19T13:04:43.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-19T13:04:43.062 INFO:teuthology.orchestra.run.smithi179.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2023-10-19T13:04:43.063 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T13:04:43.076 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:04:43.074+0000 7fed52a3a040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-19T13:04:43.079 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:43.077+0000 7f088fe45040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-19T13:04:43.121 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:43.118+0000 7f088fe45040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-19T13:04:43.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.171+0000 7fca56461640 1 -- 172.21.15.57:0/788671739 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fca500abc10 msgr2=0x7fca5006b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:43.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.171+0000 7fca56461640 1 --2- 172.21.15.57:0/788671739 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fca500abc10 0x7fca5006b2e0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fca48009be0 tx=0x7fca4802c020 comp rx=0 tx=0).stop 2023-10-19T13:04:43.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.171+0000 7fca56461640 1 -- 172.21.15.57:0/788671739 shutdown_connections 2023-10-19T13:04:43.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.171+0000 7fca56461640 1 --2- 172.21.15.57:0/788671739 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fca500abc10 0x7fca5006b2e0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:43.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.171+0000 7fca56461640 1 -- 172.21.15.57:0/788671739 >> 172.21.15.57:0/788671739 conn(0x7fca50069930 msgr2=0x7fca50069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:43.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.171+0000 7fca56461640 1 -- 172.21.15.57:0/788671739 shutdown_connections 2023-10-19T13:04:43.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.171+0000 7fca56461640 1 -- 172.21.15.57:0/788671739 wait complete. 2023-10-19T13:04:43.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.172+0000 7fca56461640 1 Processor -- start 2023-10-19T13:04:43.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.172+0000 7fca56461640 1 -- start start 2023-10-19T13:04:43.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.173+0000 7fca56461640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fca5006af00 0x7fca5013a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:43.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.173+0000 7fca56461640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fca500abc10 0x7fca500ad6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:43.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.173+0000 7fca56461640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fca500c9dc0 0x7fca500adc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:43.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.173+0000 7fca56461640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca500aa4b0 con 0x7fca500abc10 2023-10-19T13:04:43.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.173+0000 7fca56461640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca500a9c10 con 0x7fca5006af00 2023-10-19T13:04:43.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.173+0000 7fca56461640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca500a9ef0 con 0x7fca500c9dc0 2023-10-19T13:04:43.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.173+0000 7fca4f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fca500abc10 0x7fca500ad6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:43.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.173+0000 7fca4f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fca500abc10 0x7fca500ad6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57436/0 (socket says 172.21.15.57:57436) 2023-10-19T13:04:43.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.173+0000 7fca4f7fe640 1 -- 172.21.15.57:0/1878484791 learned_addr learned my addr 172.21.15.57:0/1878484791 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:43.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.173+0000 7fca549fe640 1 --2- 172.21.15.57:0/1878484791 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fca500c9dc0 0x7fca500adc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:43.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.173+0000 7fca4ffff640 1 --2- 172.21.15.57:0/1878484791 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fca5006af00 0x7fca5013a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:43.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.173+0000 7fca549fe640 1 -- 172.21.15.57:0/1878484791 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fca5006af00 msgr2=0x7fca5013a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:43.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.173+0000 7fca549fe640 1 --2- 172.21.15.57:0/1878484791 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fca5006af00 0x7fca5013a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:43.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.173+0000 7fca549fe640 1 -- 172.21.15.57:0/1878484791 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fca500abc10 msgr2=0x7fca500ad6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:43.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.173+0000 7fca549fe640 1 --2- 172.21.15.57:0/1878484791 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fca500abc10 0x7fca500ad6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:43.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.173+0000 7fca549fe640 1 -- 172.21.15.57:0/1878484791 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca48009840 con 0x7fca500c9dc0 2023-10-19T13:04:43.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.173+0000 7fca4f7fe640 1 --2- 172.21.15.57:0/1878484791 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fca500abc10 0x7fca500ad6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T13:04:43.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.174+0000 7fca549fe640 1 --2- 172.21.15.57:0/1878484791 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fca500c9dc0 0x7fca500adc20 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fca48009000 tx=0x7fca480042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:43.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.174+0000 7fca4cff9640 1 -- 172.21.15.57:0/1878484791 <== mon.2 v2:172.21.15.179:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca48003e20 con 0x7fca500c9dc0 2023-10-19T13:04:43.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.174+0000 7fca4cff9640 1 -- 172.21.15.57:0/1878484791 <== mon.2 v2:172.21.15.179:3301/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fca48004800 con 0x7fca500c9dc0 2023-10-19T13:04:43.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.174+0000 7fca56461640 1 -- 172.21.15.57:0/1878484791 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca500aa1d0 con 0x7fca500c9dc0 2023-10-19T13:04:43.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.174+0000 7fca4cff9640 1 -- 172.21.15.57:0/1878484791 <== mon.2 v2:172.21.15.179:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca4803e6b0 con 0x7fca500c9dc0 2023-10-19T13:04:43.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.174+0000 7fca56461640 1 -- 172.21.15.57:0/1878484791 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca5013ac50 con 0x7fca500c9dc0 2023-10-19T13:04:43.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.175+0000 7fca56461640 1 -- 172.21.15.57:0/1878484791 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca500aea70 con 0x7fca500c9dc0 2023-10-19T13:04:43.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.176+0000 7fca4cff9640 1 -- 172.21.15.57:0/1878484791 <== mon.2 v2:172.21.15.179:3301/0 4 ==== mgrmap(e 75) v1 ==== 89388+0+0 (secure 0 0 0) 0x7fca4803e810 con 0x7fca500c9dc0 2023-10-19T13:04:43.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.176+0000 7fca4cff9640 1 --2- 172.21.15.57:0/1878484791 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7fca3c06aff0 0x7fca3c06d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:43.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.176+0000 7fca4ffff640 1 -- 172.21.15.57:0/1878484791 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7fca3c06aff0 msgr2=0x7fca3c06d4b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/3526956956 2023-10-19T13:04:43.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.176+0000 7fca4cff9640 1 -- 172.21.15.57:0/1878484791 <== mon.2 v2:172.21.15.179:3301/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6401+0+0 (secure 0 0 0) 0x7fca480acd50 con 0x7fca500c9dc0 2023-10-19T13:04:43.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.176+0000 7fca4ffff640 1 --2- 172.21.15.57:0/1878484791 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7fca3c06aff0 0x7fca3c06d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T13:04:43.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.184+0000 7fca4cff9640 1 -- 172.21.15.57:0/1878484791 <== mon.2 v2:172.21.15.179:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fca48077b00 con 0x7fca500c9dc0 2023-10-19T13:04:43.223 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:04:43.220+0000 7fed52a3a040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-19T13:04:43.229 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:43.227+0000 7f088fe45040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-19T13:04:43.267 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:04:43.265+0000 7fed52a3a040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-19T13:04:43.378 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:04:43.375+0000 7fed52a3a040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-19T13:04:43.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.377+0000 7fca4ffff640 1 -- 172.21.15.57:0/1878484791 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7fca3c06aff0 msgr2=0x7fca3c06d4b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/3526956956 2023-10-19T13:04:43.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.377+0000 7fca4ffff640 1 --2- 172.21.15.57:0/1878484791 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7fca3c06aff0 0x7fca3c06d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-19T13:04:43.401 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.398+0000 7fca56461640 1 -- 172.21.15.57:0/1878484791 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fca500aea70 con 0x7fca500c9dc0 2023-10-19T13:04:43.401 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.399+0000 7fca4cff9640 1 -- 172.21.15.57:0/1878484791 <== mon.2 v2:172.21.15.179:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+4921 (secure 0 0 0) 0x7fca500aea70 con 0x7fca500c9dc0 2023-10-19T13:04:43.402 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:04:43.402 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":56,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7325,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/3318110028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":3318110028},{"type":"v1","addr":"172.21.15.179:6837","nonce":3318110028}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":8143,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/3625556517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":3625556517},{"type":"v1","addr":"172.21.15.57:6835","nonce":3625556517}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":8158,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/4167860955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":4167860955},{"type":"v1","addr":"172.21.15.179:6836","nonce":4167860955}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T13:04:22.785432+0000","modified":"2023-10-19T13:04:24.778587+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9066},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9066":{"gid":9066,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":3,"addr":"172.21.15.57:6834/3474229632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":3474229632},{"type":"v1","addr":"172.21.15.57:6834","nonce":3474229632}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2023-10-19T13:04:43.403 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 56 2023-10-19T13:04:43.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.401+0000 7fca56461640 1 -- 172.21.15.57:0/1878484791 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7fca3c06aff0 msgr2=0x7fca3c06d4b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T13:04:43.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.401+0000 7fca56461640 1 --2- 172.21.15.57:0/1878484791 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7fca3c06aff0 0x7fca3c06d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:43.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.401+0000 7fca56461640 1 -- 172.21.15.57:0/1878484791 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fca500c9dc0 msgr2=0x7fca500adc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:43.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.401+0000 7fca56461640 1 --2- 172.21.15.57:0/1878484791 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fca500c9dc0 0x7fca500adc20 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fca48009000 tx=0x7fca480042f0 comp rx=0 tx=0).stop 2023-10-19T13:04:43.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.401+0000 7fca56461640 1 -- 172.21.15.57:0/1878484791 shutdown_connections 2023-10-19T13:04:43.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.401+0000 7fca56461640 1 --2- 172.21.15.57:0/1878484791 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7fca3c06aff0 0x7fca3c06d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:43.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.401+0000 7fca56461640 1 --2- 172.21.15.57:0/1878484791 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fca500c9dc0 0x7fca500adc20 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:43.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.401+0000 7fca56461640 1 --2- 172.21.15.57:0/1878484791 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fca500abc10 0x7fca500ad6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:43.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.401+0000 7fca56461640 1 --2- 172.21.15.57:0/1878484791 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fca5006af00 0x7fca5013a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:43.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.401+0000 7fca56461640 1 -- 172.21.15.57:0/1878484791 >> 172.21.15.57:0/1878484791 conn(0x7fca50069930 msgr2=0x7fca50069d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:43.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.401+0000 7fca56461640 1 -- 172.21.15.57:0/1878484791 shutdown_connections 2023-10-19T13:04:43.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.401+0000 7fca56461640 1 -- 172.21.15.57:0/1878484791 wait complete. 2023-10-19T13:04:43.415 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T13:04:43.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.561+0000 7fb9daac2640 1 -- 172.21.15.57:0/936614137 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb9d40d1910 msgr2=0x7fb9d40b0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:43.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.561+0000 7fb9daac2640 1 --2- 172.21.15.57:0/936614137 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb9d40d1910 0x7fb9d40b0650 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fb9cc009be0 tx=0x7fb9cc02c0c0 comp rx=0 tx=0).stop 2023-10-19T13:04:43.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.561+0000 7fb9daac2640 1 -- 172.21.15.57:0/936614137 shutdown_connections 2023-10-19T13:04:43.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.561+0000 7fb9daac2640 1 --2- 172.21.15.57:0/936614137 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb9d40d1910 0x7fb9d40b0650 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:43.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.561+0000 7fb9daac2640 1 -- 172.21.15.57:0/936614137 >> 172.21.15.57:0/936614137 conn(0x7fb9d40d0d60 msgr2=0x7fb9d40d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:43.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.561+0000 7fb9daac2640 1 -- 172.21.15.57:0/936614137 shutdown_connections 2023-10-19T13:04:43.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.561+0000 7fb9daac2640 1 -- 172.21.15.57:0/936614137 wait complete. 2023-10-19T13:04:43.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.562+0000 7fb9daac2640 1 Processor -- start 2023-10-19T13:04:43.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.563+0000 7fb9daac2640 1 -- start start 2023-10-19T13:04:43.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.563+0000 7fb9daac2640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb9d40cf340 0x7fb9d40af310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:43.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.563+0000 7fb9daac2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb9d40d1910 0x7fb9d40c8420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:43.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.563+0000 7fb9daac2640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb9d415b2a0 0x7fb9d40c8960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:43.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.563+0000 7fb9daac2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9d4152ac0 con 0x7fb9d40d1910 2023-10-19T13:04:43.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.563+0000 7fb9daac2640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9d4157580 con 0x7fb9d415b2a0 2023-10-19T13:04:43.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.563+0000 7fb9daac2640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9d40ca870 con 0x7fb9d40cf340 2023-10-19T13:04:43.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.564+0000 7fb9d3fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb9d40d1910 0x7fb9d40c8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:43.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.564+0000 7fb9d3fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb9d40d1910 0x7fb9d40c8420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57452/0 (socket says 172.21.15.57:57452) 2023-10-19T13:04:43.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.564+0000 7fb9d3fff640 1 -- 172.21.15.57:0/534932485 learned_addr learned my addr 172.21.15.57:0/534932485 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:43.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.564+0000 7fb9d905f640 1 --2- 172.21.15.57:0/534932485 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb9d415b2a0 0x7fb9d40c8960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:43.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.564+0000 7fb9d885e640 1 --2- 172.21.15.57:0/534932485 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb9d40cf340 0x7fb9d40af310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:43.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.564+0000 7fb9d3fff640 1 -- 172.21.15.57:0/534932485 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb9d40cf340 msgr2=0x7fb9d40af310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:43.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.564+0000 7fb9d3fff640 1 --2- 172.21.15.57:0/534932485 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb9d40cf340 0x7fb9d40af310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:43.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.564+0000 7fb9d3fff640 1 -- 172.21.15.57:0/534932485 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb9d415b2a0 msgr2=0x7fb9d40c8960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:43.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.564+0000 7fb9d3fff640 1 --2- 172.21.15.57:0/534932485 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb9d415b2a0 0x7fb9d40c8960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:43.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.564+0000 7fb9d3fff640 1 -- 172.21.15.57:0/534932485 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9cc009840 con 0x7fb9d40d1910 2023-10-19T13:04:43.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.564+0000 7fb9d885e640 1 --2- 172.21.15.57:0/534932485 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb9d40cf340 0x7fb9d40af310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T13:04:43.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.564+0000 7fb9d905f640 1 --2- 172.21.15.57:0/534932485 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb9d415b2a0 0x7fb9d40c8960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T13:04:43.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.564+0000 7fb9d3fff640 1 --2- 172.21.15.57:0/534932485 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb9d40d1910 0x7fb9d40c8420 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7fb9c000c500 tx=0x7fb9c000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:43.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.564+0000 7fb9d17fa640 1 -- 172.21.15.57:0/534932485 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9c0012830 con 0x7fb9d40d1910 2023-10-19T13:04:43.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.564+0000 7fb9d17fa640 1 -- 172.21.15.57:0/534932485 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fb9c0012a80 con 0x7fb9d40d1910 2023-10-19T13:04:43.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.564+0000 7fb9daac2640 1 -- 172.21.15.57:0/534932485 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9d40cab50 con 0x7fb9d40d1910 2023-10-19T13:04:43.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.564+0000 7fb9d17fa640 1 -- 172.21.15.57:0/534932485 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9c0012e90 con 0x7fb9d40d1910 2023-10-19T13:04:43.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.564+0000 7fb9daac2640 1 -- 172.21.15.57:0/534932485 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9d4157060 con 0x7fb9d40d1910 2023-10-19T13:04:43.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.565+0000 7fb9daac2640 1 -- 172.21.15.57:0/534932485 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9d4156be0 con 0x7fb9d40d1910 2023-10-19T13:04:43.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.566+0000 7fb9d17fa640 1 -- 172.21.15.57:0/534932485 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 75) v1 ==== 89388+0+0 (secure 0 0 0) 0x7fb9c001a3b0 con 0x7fb9d40d1910 2023-10-19T13:04:43.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.566+0000 7fb9d17fa640 1 --2- 172.21.15.57:0/534932485 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7fb9b806aff0 0x7fb9b806d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:43.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.567+0000 7fb9d885e640 1 -- 172.21.15.57:0/534932485 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7fb9b806aff0 msgr2=0x7fb9b806d4b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/3526956956 2023-10-19T13:04:43.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.567+0000 7fb9d17fa640 1 -- 172.21.15.57:0/534932485 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6401+0+0 (secure 0 0 0) 0x7fb9c0089ca0 con 0x7fb9d40d1910 2023-10-19T13:04:43.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.567+0000 7fb9d885e640 1 --2- 172.21.15.57:0/534932485 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7fb9b806aff0 0x7fb9b806d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T13:04:43.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.571+0000 7fb9d17fa640 1 -- 172.21.15.57:0/534932485 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb9d4156be0 con 0x7fb9d40d1910 2023-10-19T13:04:43.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.749+0000 7fb9daac2640 1 -- 172.21.15.57:0/534932485 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb9d4156be0 con 0x7fb9d40d1910 2023-10-19T13:04:43.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.750+0000 7fb9d17fa640 1 -- 172.21.15.57:0/534932485 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v145) v1 ==== 74+0+17348 (secure 0 0 0) 0x7fb9d4156be0 con 0x7fb9d40d1910 2023-10-19T13:04:43.753 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:04:43.754 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":145,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T13:04:33.816818+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2023-10-19T13:04:20.932329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"143","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2023-10-19T13:04:21.762157+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"144","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.179:0/1700025841":"2023-10-20T13:04:33.816783+0000","172.21.15.179:0/3160483813":"2023-10-20T13:04:33.816783+0000","172.21.15.179:0/4039784875":"2023-10-20T13:04:33.816783+0000","172.21.15.179:0/1023851911":"2023-10-20T13:04:33.816783+0000","172.21.15.179:0/2968654981":"2023-10-20T13:04:33.816783+0000","172.21.15.179:0/3605212267":"2023-10-20T13:04:33.816783+0000","172.21.15.179:6833/2346394725":"2023-10-20T13:04:33.816783+0000","172.21.15.179:6832/2346394725":"2023-10-20T13:04:33.816783+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T13:04:43.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.752+0000 7fb9daac2640 1 -- 172.21.15.57:0/534932485 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7fb9b806aff0 msgr2=0x7fb9b806d4b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T13:04:43.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.752+0000 7fb9daac2640 1 --2- 172.21.15.57:0/534932485 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7fb9b806aff0 0x7fb9b806d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:43.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.752+0000 7fb9daac2640 1 -- 172.21.15.57:0/534932485 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb9d40d1910 msgr2=0x7fb9d40c8420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:43.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.752+0000 7fb9daac2640 1 --2- 172.21.15.57:0/534932485 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb9d40d1910 0x7fb9d40c8420 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7fb9c000c500 tx=0x7fb9c000c9c0 comp rx=0 tx=0).stop 2023-10-19T13:04:43.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.752+0000 7fb9daac2640 1 -- 172.21.15.57:0/534932485 shutdown_connections 2023-10-19T13:04:43.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.752+0000 7fb9daac2640 1 --2- 172.21.15.57:0/534932485 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7fb9b806aff0 0x7fb9b806d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:43.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.752+0000 7fb9daac2640 1 --2- 172.21.15.57:0/534932485 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb9d415b2a0 0x7fb9d40c8960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:43.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.752+0000 7fb9daac2640 1 --2- 172.21.15.57:0/534932485 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb9d40d1910 0x7fb9d40c8420 unknown :-1 s=CLOSED pgs=1670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:43.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.752+0000 7fb9daac2640 1 --2- 172.21.15.57:0/534932485 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb9d40cf340 0x7fb9d40af310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:43.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.752+0000 7fb9daac2640 1 -- 172.21.15.57:0/534932485 >> 172.21.15.57:0/534932485 conn(0x7fb9d40d0d60 msgr2=0x7fb9d40cea30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:43.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.753+0000 7fb9daac2640 1 -- 172.21.15.57:0/534932485 shutdown_connections 2023-10-19T13:04:43.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.753+0000 7fb9daac2640 1 -- 172.21.15.57:0/534932485 wait complete. 2023-10-19T13:04:43.766 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-19T12:44:07.371186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-19T13:04:20.932329+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '143', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_data', 'create_time': '2023-10-19T13:04:21.762157+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '144', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-19T13:04:43.767 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-19T13:04:43.853 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:43.850+0000 7f088fe45040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-19T13:04:43.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.878+0000 7fc5f38e5640 1 -- 172.21.15.57:0/3369707799 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc5ec0cf340 msgr2=0x7fc5ec0d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:43.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.878+0000 7fc5f38e5640 1 --2- 172.21.15.57:0/3369707799 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc5ec0cf340 0x7fc5ec0d1910 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fc5dc009830 tx=0x7fc5dc02bd60 comp rx=0 tx=0).stop 2023-10-19T13:04:43.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.878+0000 7fc5f38e5640 1 -- 172.21.15.57:0/3369707799 shutdown_connections 2023-10-19T13:04:43.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.878+0000 7fc5f38e5640 1 --2- 172.21.15.57:0/3369707799 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc5ec0cf340 0x7fc5ec0d1910 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:43.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.878+0000 7fc5f38e5640 1 -- 172.21.15.57:0/3369707799 >> 172.21.15.57:0/3369707799 conn(0x7fc5ec0d0d60 msgr2=0x7fc5ec0d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:43.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.878+0000 7fc5f38e5640 1 -- 172.21.15.57:0/3369707799 shutdown_connections 2023-10-19T13:04:43.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.878+0000 7fc5f38e5640 1 -- 172.21.15.57:0/3369707799 wait complete. 2023-10-19T13:04:43.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.880+0000 7fc5f38e5640 1 Processor -- start 2023-10-19T13:04:43.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.880+0000 7fc5f38e5640 1 -- start start 2023-10-19T13:04:43.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.880+0000 7fc5f38e5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5ec0ceb40 0x7fc5ec158140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:43.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.880+0000 7fc5f38e5640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc5ec0cf340 0x7fc5ec0b38e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:43.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.880+0000 7fc5f38e5640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc5ec15b2a0 0x7fc5ec0b5400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:43.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.880+0000 7fc5f38e5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5ec15bb50 con 0x7fc5ec0ceb40 2023-10-19T13:04:43.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.880+0000 7fc5f38e5640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5ec158680 con 0x7fc5ec15b2a0 2023-10-19T13:04:43.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.880+0000 7fc5f38e5640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5ec0b41c0 con 0x7fc5ec0cf340 2023-10-19T13:04:43.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.880+0000 7fc5f1681640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5ec0ceb40 0x7fc5ec158140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:43.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.880+0000 7fc5f1681640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5ec0ceb40 0x7fc5ec158140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57464/0 (socket says 172.21.15.57:57464) 2023-10-19T13:04:43.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.880+0000 7fc5f1681640 1 -- 172.21.15.57:0/3983265888 learned_addr learned my addr 172.21.15.57:0/3983265888 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:43.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.881+0000 7fc5f1e82640 1 --2- 172.21.15.57:0/3983265888 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc5ec15b2a0 0x7fc5ec0b5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:43.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.881+0000 7fc5f0e80640 1 --2- 172.21.15.57:0/3983265888 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc5ec0cf340 0x7fc5ec0b38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:43.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.881+0000 7fc5f1681640 1 -- 172.21.15.57:0/3983265888 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc5ec0cf340 msgr2=0x7fc5ec0b38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:43.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.881+0000 7fc5f1681640 1 --2- 172.21.15.57:0/3983265888 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc5ec0cf340 0x7fc5ec0b38e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:43.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.881+0000 7fc5f1681640 1 -- 172.21.15.57:0/3983265888 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc5ec15b2a0 msgr2=0x7fc5ec0b5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:43.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.881+0000 7fc5f1681640 1 --2- 172.21.15.57:0/3983265888 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc5ec15b2a0 0x7fc5ec0b5400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:43.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.881+0000 7fc5f1681640 1 -- 172.21.15.57:0/3983265888 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5dc009490 con 0x7fc5ec0ceb40 2023-10-19T13:04:43.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.881+0000 7fc5f0e80640 1 --2- 172.21.15.57:0/3983265888 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc5ec0cf340 0x7fc5ec0b38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T13:04:43.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.881+0000 7fc5f1681640 1 --2- 172.21.15.57:0/3983265888 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5ec0ceb40 0x7fc5ec158140 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7fc5dc000c00 tx=0x7fc5dc003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:43.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.881+0000 7fc5e1ffb640 1 -- 172.21.15.57:0/3983265888 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5dc003e20 con 0x7fc5ec0ceb40 2023-10-19T13:04:43.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.881+0000 7fc5e1ffb640 1 -- 172.21.15.57:0/3983265888 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fc5dc034440 con 0x7fc5ec0ceb40 2023-10-19T13:04:43.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.881+0000 7fc5f38e5640 1 -- 172.21.15.57:0/3983265888 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5ec0b44a0 con 0x7fc5ec0ceb40 2023-10-19T13:04:43.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.881+0000 7fc5e1ffb640 1 -- 172.21.15.57:0/3983265888 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5dc034850 con 0x7fc5ec0ceb40 2023-10-19T13:04:43.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.882+0000 7fc5f38e5640 1 -- 172.21.15.57:0/3983265888 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5ec15b770 con 0x7fc5ec0ceb40 2023-10-19T13:04:43.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.883+0000 7fc5f38e5640 1 -- 172.21.15.57:0/3983265888 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5ec0690e0 con 0x7fc5ec0ceb40 2023-10-19T13:04:43.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.883+0000 7fc5e1ffb640 1 -- 172.21.15.57:0/3983265888 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 75) v1 ==== 89388+0+0 (secure 0 0 0) 0x7fc5dc0349b0 con 0x7fc5ec0ceb40 2023-10-19T13:04:43.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.884+0000 7fc5e1ffb640 1 --2- 172.21.15.57:0/3983265888 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7fc5d006aff0 0x7fc5d006d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:43.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.884+0000 7fc5e1ffb640 1 -- 172.21.15.57:0/3983265888 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6401+0+0 (secure 0 0 0) 0x7fc5dc0ac1b0 con 0x7fc5ec0ceb40 2023-10-19T13:04:43.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.884+0000 7fc5f0e80640 1 -- 172.21.15.57:0/3983265888 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7fc5d006aff0 msgr2=0x7fc5d006d4b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/3526956956 2023-10-19T13:04:43.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.884+0000 7fc5f0e80640 1 --2- 172.21.15.57:0/3983265888 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7fc5d006aff0 0x7fc5d006d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T13:04:43.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:43.888+0000 7fc5e1ffb640 1 -- 172.21.15.57:0/3983265888 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc5dc076e60 con 0x7fc5ec0ceb40 2023-10-19T13:04:44.046 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:04:44.043+0000 7fed52a3a040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-19T13:04:44.067 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.065+0000 7fc5f38e5640 1 -- 172.21.15.57:0/3983265888 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc5ec0d0b40 con 0x7fc5ec0ceb40 2023-10-19T13:04:44.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.065+0000 7fc5e1ffb640 1 -- 172.21.15.57:0/3983265888 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v145) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc5dc07a480 con 0x7fc5ec0ceb40 2023-10-19T13:04:44.068 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-19T13:04:44.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.067+0000 7fc5f38e5640 1 -- 172.21.15.57:0/3983265888 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7fc5d006aff0 msgr2=0x7fc5d006d4b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T13:04:44.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.067+0000 7fc5f38e5640 1 --2- 172.21.15.57:0/3983265888 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7fc5d006aff0 0x7fc5d006d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:44.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.067+0000 7fc5f38e5640 1 -- 172.21.15.57:0/3983265888 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5ec0ceb40 msgr2=0x7fc5ec158140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:44.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.067+0000 7fc5f38e5640 1 --2- 172.21.15.57:0/3983265888 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5ec0ceb40 0x7fc5ec158140 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7fc5dc000c00 tx=0x7fc5dc003820 comp rx=0 tx=0).stop 2023-10-19T13:04:44.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.068+0000 7fc5f38e5640 1 -- 172.21.15.57:0/3983265888 shutdown_connections 2023-10-19T13:04:44.072 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.068+0000 7fc5f38e5640 1 --2- 172.21.15.57:0/3983265888 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7fc5d006aff0 0x7fc5d006d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:44.072 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.068+0000 7fc5f38e5640 1 --2- 172.21.15.57:0/3983265888 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc5ec15b2a0 0x7fc5ec0b5400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:44.072 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.068+0000 7fc5f38e5640 1 --2- 172.21.15.57:0/3983265888 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc5ec0cf340 0x7fc5ec0b38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:44.072 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.068+0000 7fc5f38e5640 1 --2- 172.21.15.57:0/3983265888 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5ec0ceb40 0x7fc5ec158140 unknown :-1 s=CLOSED pgs=1672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:44.072 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.068+0000 7fc5f38e5640 1 -- 172.21.15.57:0/3983265888 >> 172.21.15.57:0/3983265888 conn(0x7fc5ec0d0d60 msgr2=0x7fc5ec06b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:44.073 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.068+0000 7fc5f38e5640 1 -- 172.21.15.57:0/3983265888 shutdown_connections 2023-10-19T13:04:44.073 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.068+0000 7fc5f38e5640 1 -- 172.21.15.57:0/3983265888 wait complete. 2023-10-19T13:04:44.081 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-19T13:04:44.115 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:44.112+0000 7f088fe45040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-19T13:04:44.163 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:44.160+0000 7f088fe45040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-19T13:04:44.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.192+0000 7fa8eb01a640 1 -- 172.21.15.57:0/1113799341 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa8e40d1910 msgr2=0x7fa8e40b0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:44.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.192+0000 7fa8eb01a640 1 --2- 172.21.15.57:0/1113799341 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa8e40d1910 0x7fa8e40b0650 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fa8dc009be0 tx=0x7fa8dc02c0c0 comp rx=0 tx=0).stop 2023-10-19T13:04:44.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.192+0000 7fa8eb01a640 1 -- 172.21.15.57:0/1113799341 shutdown_connections 2023-10-19T13:04:44.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.192+0000 7fa8eb01a640 1 --2- 172.21.15.57:0/1113799341 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa8e40d1910 0x7fa8e40b0650 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:44.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.192+0000 7fa8eb01a640 1 -- 172.21.15.57:0/1113799341 >> 172.21.15.57:0/1113799341 conn(0x7fa8e40d0d60 msgr2=0x7fa8e40d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:44.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.192+0000 7fa8eb01a640 1 -- 172.21.15.57:0/1113799341 shutdown_connections 2023-10-19T13:04:44.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.192+0000 7fa8eb01a640 1 -- 172.21.15.57:0/1113799341 wait complete. 2023-10-19T13:04:44.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.193+0000 7fa8eb01a640 1 Processor -- start 2023-10-19T13:04:44.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.193+0000 7fa8eb01a640 1 -- start start 2023-10-19T13:04:44.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.194+0000 7fa8eb01a640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa8e40cf340 0x7fa8e406a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:44.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.194+0000 7fa8eb01a640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa8e40d1910 0x7fa8e415bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:44.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.194+0000 7fa8eb01a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa8e415b2a0 0x7fa8e4157110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:44.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.194+0000 7fa8eb01a640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e406b2f0 con 0x7fa8e415b2a0 2023-10-19T13:04:44.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.194+0000 7fa8eb01a640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e406b5d0 con 0x7fa8e40cf340 2023-10-19T13:04:44.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.194+0000 7fa8eb01a640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e40b4890 con 0x7fa8e40d1910 2023-10-19T13:04:44.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.194+0000 7fa8e95b7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa8e415b2a0 0x7fa8e4157110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:44.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.194+0000 7fa8e95b7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa8e415b2a0 0x7fa8e4157110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57466/0 (socket says 172.21.15.57:57466) 2023-10-19T13:04:44.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.194+0000 7fa8e95b7640 1 -- 172.21.15.57:0/2397607057 learned_addr learned my addr 172.21.15.57:0/2397607057 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:44.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.194+0000 7fa8e8db6640 1 --2- 172.21.15.57:0/2397607057 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa8e40cf340 0x7fa8e406a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:44.201 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.194+0000 7fa8e3fff640 1 --2- 172.21.15.57:0/2397607057 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa8e40d1910 0x7fa8e415bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:44.201 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.194+0000 7fa8e95b7640 1 -- 172.21.15.57:0/2397607057 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa8e40d1910 msgr2=0x7fa8e415bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:44.201 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.194+0000 7fa8e95b7640 1 --2- 172.21.15.57:0/2397607057 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa8e40d1910 0x7fa8e415bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:44.201 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.194+0000 7fa8e95b7640 1 -- 172.21.15.57:0/2397607057 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa8e40cf340 msgr2=0x7fa8e406a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:44.201 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.194+0000 7fa8e95b7640 1 --2- 172.21.15.57:0/2397607057 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa8e40cf340 0x7fa8e406a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:44.202 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.194+0000 7fa8e95b7640 1 -- 172.21.15.57:0/2397607057 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8dc009840 con 0x7fa8e415b2a0 2023-10-19T13:04:44.202 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.195+0000 7fa8e95b7640 1 --2- 172.21.15.57:0/2397607057 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa8e415b2a0 0x7fa8e4157110 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7fa8dc036040 tx=0x7fa8dc004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:44.202 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.195+0000 7fa8e17fa640 1 -- 172.21.15.57:0/2397607057 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8dc0045f0 con 0x7fa8e415b2a0 2023-10-19T13:04:44.202 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.195+0000 7fa8e17fa640 1 -- 172.21.15.57:0/2397607057 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fa8dc004860 con 0x7fa8e415b2a0 2023-10-19T13:04:44.202 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.195+0000 7fa8e17fa640 1 -- 172.21.15.57:0/2397607057 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8dc0347e0 con 0x7fa8e415b2a0 2023-10-19T13:04:44.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.195+0000 7fa8eb01a640 1 -- 172.21.15.57:0/2397607057 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8e40b4b70 con 0x7fa8e415b2a0 2023-10-19T13:04:44.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.195+0000 7fa8eb01a640 1 -- 172.21.15.57:0/2397607057 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8e406aee0 con 0x7fa8e415b2a0 2023-10-19T13:04:44.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.196+0000 7fa8eb01a640 1 -- 172.21.15.57:0/2397607057 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8e40d01d0 con 0x7fa8e415b2a0 2023-10-19T13:04:44.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.197+0000 7fa8e17fa640 1 -- 172.21.15.57:0/2397607057 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 75) v1 ==== 89388+0+0 (secure 0 0 0) 0x7fa8dc034940 con 0x7fa8e415b2a0 2023-10-19T13:04:44.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.197+0000 7fa8e17fa640 1 --2- 172.21.15.57:0/2397607057 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7fa8c806aff0 0x7fa8c806d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:44.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.197+0000 7fa8e8db6640 1 -- 172.21.15.57:0/2397607057 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7fa8c806aff0 msgr2=0x7fa8c806d4b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/3526956956 2023-10-19T13:04:44.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.197+0000 7fa8e17fa640 1 -- 172.21.15.57:0/2397607057 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6401+0+0 (secure 0 0 0) 0x7fa8dc0acd70 con 0x7fa8e415b2a0 2023-10-19T13:04:44.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.197+0000 7fa8e8db6640 1 --2- 172.21.15.57:0/2397607057 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7fa8c806aff0 0x7fa8c806d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T13:04:44.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.201+0000 7fa8e17fa640 1 -- 172.21.15.57:0/2397607057 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa8dc077990 con 0x7fa8e415b2a0 2023-10-19T13:04:44.265 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:44.263+0000 7f088fe45040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-19T13:04:44.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.283+0000 7fa8e17fa640 1 -- 172.21.15.57:0/2397607057 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mgrmap(e 76) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fa8dc077d60 con 0x7fa8e415b2a0 2023-10-19T13:04:44.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.283+0000 7fa8e17fa640 1 -- 172.21.15.57:0/2397607057 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7fa8c806aff0 msgr2=0x7fa8c806d4b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T13:04:44.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.283+0000 7fa8e17fa640 1 --2- 172.21.15.57:0/2397607057 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7fa8c806aff0 0x7fa8c806d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:44.332 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:04:44.329+0000 7fed52a3a040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-19T13:04:44.344 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:44.341+0000 7f081497f640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:04:44.344 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:44.341+0000 7f081497f640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:04:44.344 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:44.341+0000 7f081497f640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:04:44.344 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:44.341+0000 7f081497f640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:04:44.344 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:44.341+0000 7f081497f640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:04:44.345 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:44.342+0000 7f081217a640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:04:44.345 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:44.342+0000 7f081217a640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:04:44.345 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:44.342+0000 7f081217a640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:04:44.346 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:44.342+0000 7f081217a640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:04:44.346 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:04:44.342+0000 7f081217a640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:04:44.385 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:04:44.382+0000 7fed52a3a040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-19T13:04:44.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.386+0000 7fa8eb01a640 1 -- 172.21.15.57:0/2397607057 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa8e40b49d0 con 0x7fa8e415b2a0 2023-10-19T13:04:44.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.386+0000 7fa8e17fa640 1 -- 172.21.15.57:0/2397607057 <== mon.0 v2:172.21.15.57:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v146) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa8e40b49d0 con 0x7fa8e415b2a0 2023-10-19T13:04:44.389 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2023-10-19T13:04:44.391 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.388+0000 7fa8eb01a640 1 -- 172.21.15.57:0/2397607057 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa8e415b2a0 msgr2=0x7fa8e4157110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:44.392 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.388+0000 7fa8eb01a640 1 --2- 172.21.15.57:0/2397607057 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa8e415b2a0 0x7fa8e4157110 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7fa8dc036040 tx=0x7fa8dc004270 comp rx=0 tx=0).stop 2023-10-19T13:04:44.392 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.388+0000 7fa8eb01a640 1 -- 172.21.15.57:0/2397607057 shutdown_connections 2023-10-19T13:04:44.392 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.388+0000 7fa8eb01a640 1 --2- 172.21.15.57:0/2397607057 >> [v2:172.21.15.179:6832/3526956956,v1:172.21.15.179:6833/3526956956] conn(0x7fa8c806aff0 0x7fa8c806d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:44.392 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.388+0000 7fa8eb01a640 1 --2- 172.21.15.57:0/2397607057 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa8e415b2a0 0x7fa8e4157110 unknown :-1 s=CLOSED pgs=1674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:44.392 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.388+0000 7fa8eb01a640 1 --2- 172.21.15.57:0/2397607057 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa8e40d1910 0x7fa8e415bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:44.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.388+0000 7fa8eb01a640 1 --2- 172.21.15.57:0/2397607057 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa8e40cf340 0x7fa8e406a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:44.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.388+0000 7fa8eb01a640 1 -- 172.21.15.57:0/2397607057 >> 172.21.15.57:0/2397607057 conn(0x7fa8e40d0d60 msgr2=0x7fa8e415a4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:44.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.389+0000 7fa8eb01a640 1 -- 172.21.15.57:0/2397607057 shutdown_connections 2023-10-19T13:04:44.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.389+0000 7fa8eb01a640 1 -- 172.21.15.57:0/2397607057 wait complete. 2023-10-19T13:04:44.402 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-19T13:04:44.493 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:04:44.490+0000 7fed52a3a040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-19T13:04:44.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.512+0000 7f98a80b4640 1 -- 172.21.15.57:0/2659242210 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f98a00cd6d0 msgr2=0x7f98a00cdb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:44.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.512+0000 7f98a80b4640 1 --2- 172.21.15.57:0/2659242210 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f98a00cd6d0 0x7f98a00cdb30 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f9890005fd0 tx=0x7f989002bd90 comp rx=0 tx=0).stop 2023-10-19T13:04:44.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.512+0000 7f98a80b4640 1 -- 172.21.15.57:0/2659242210 shutdown_connections 2023-10-19T13:04:44.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.512+0000 7f98a80b4640 1 --2- 172.21.15.57:0/2659242210 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f98a00cd6d0 0x7f98a00cdb30 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:44.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.512+0000 7f98a80b4640 1 --2- 172.21.15.57:0/2659242210 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f98a0142850 0x7f98a0159440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:44.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.512+0000 7f98a80b4640 1 -- 172.21.15.57:0/2659242210 >> 172.21.15.57:0/2659242210 conn(0x7f98a00af7a0 msgr2=0x7f98a00afb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:44.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.513+0000 7f98a80b4640 1 -- 172.21.15.57:0/2659242210 shutdown_connections 2023-10-19T13:04:44.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.513+0000 7f98a80b4640 1 -- 172.21.15.57:0/2659242210 wait complete. 2023-10-19T13:04:44.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.514+0000 7f98a80b4640 1 Processor -- start 2023-10-19T13:04:44.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.514+0000 7f98a80b4640 1 -- start start 2023-10-19T13:04:44.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.514+0000 7f98a80b4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f98a0069720 0x7f98a00c75b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:44.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.514+0000 7f98a80b4640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f98a00cd6d0 0x7f98a00c7af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:44.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.514+0000 7f98a80b4640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f98a0142850 0x7f98a01510c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:44.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.514+0000 7f98a80b4640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98a00a6c00 con 0x7f98a0069720 2023-10-19T13:04:44.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.514+0000 7f98a80b4640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98a00a6920 con 0x7f98a0142850 2023-10-19T13:04:44.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.514+0000 7f98a80b4640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98a006ca60 con 0x7f98a00cd6d0 2023-10-19T13:04:44.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.515+0000 7f98a5e50640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f98a0069720 0x7f98a00c75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:44.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.515+0000 7f98a5e50640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f98a0069720 0x7f98a00c75b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57478/0 (socket says 172.21.15.57:57478) 2023-10-19T13:04:44.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.515+0000 7f98a5e50640 1 -- 172.21.15.57:0/3848284382 learned_addr learned my addr 172.21.15.57:0/3848284382 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:44.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.515+0000 7f98a564f640 1 --2- 172.21.15.57:0/3848284382 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f98a00cd6d0 0x7f98a00c7af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:44.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.515+0000 7f98a6651640 1 --2- 172.21.15.57:0/3848284382 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f98a0142850 0x7f98a01510c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:44.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.515+0000 7f98a6651640 1 -- 172.21.15.57:0/3848284382 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f98a00cd6d0 msgr2=0x7f98a00c7af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:44.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.515+0000 7f98a6651640 1 --2- 172.21.15.57:0/3848284382 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f98a00cd6d0 0x7f98a00c7af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:44.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.515+0000 7f98a6651640 1 -- 172.21.15.57:0/3848284382 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f98a0069720 msgr2=0x7f98a00c75b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:44.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.515+0000 7f98a6651640 1 --2- 172.21.15.57:0/3848284382 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f98a0069720 0x7f98a00c75b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:44.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.515+0000 7f98a6651640 1 -- 172.21.15.57:0/3848284382 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9890009560 con 0x7f98a0142850 2023-10-19T13:04:44.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.515+0000 7f98a5e50640 1 --2- 172.21.15.57:0/3848284382 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f98a0069720 0x7f98a00c75b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T13:04:44.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.515+0000 7f98a6651640 1 --2- 172.21.15.57:0/3848284382 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f98a0142850 0x7f98a01510c0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f989800c690 tx=0x7f989800cb50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:44.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.515+0000 7f98967fc640 1 -- 172.21.15.57:0/3848284382 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9898003b70 con 0x7f98a0142850 2023-10-19T13:04:44.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.515+0000 7f98967fc640 1 -- 172.21.15.57:0/3848284382 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f9898003cd0 con 0x7f98a0142850 2023-10-19T13:04:44.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.515+0000 7f98967fc640 1 -- 172.21.15.57:0/3848284382 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9898005330 con 0x7f98a0142850 2023-10-19T13:04:44.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.515+0000 7f98a80b4640 1 -- 172.21.15.57:0/3848284382 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98a006cd40 con 0x7f98a0142850 2023-10-19T13:04:44.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.515+0000 7f98a80b4640 1 -- 172.21.15.57:0/3848284382 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98a0155480 con 0x7f98a0142850 2023-10-19T13:04:44.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.516+0000 7f98a80b4640 1 -- 172.21.15.57:0/3848284382 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98a006c8c0 con 0x7f98a0142850 2023-10-19T13:04:44.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.517+0000 7f98967fc640 1 -- 172.21.15.57:0/3848284382 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 76) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f9898005490 con 0x7f98a0142850 2023-10-19T13:04:44.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.517+0000 7f98967fc640 1 -- 172.21.15.57:0/3848284382 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 6788+0+0 (secure 0 0 0) 0x7f9898089290 con 0x7f98a0142850 2023-10-19T13:04:44.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.522+0000 7f98967fc640 1 -- 172.21.15.57:0/3848284382 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f98a006c8c0 con 0x7f98a0142850 2023-10-19T13:04:44.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.696+0000 7f98a80b4640 1 -- 172.21.15.57:0/3848284382 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f98a006d260 con 0x7f98a0142850 2023-10-19T13:04:44.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.696+0000 7f98967fc640 1 -- 172.21.15.57:0/3848284382 <== mon.1 v2:172.21.15.179:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v146) v1 ==== 100+0+11 (secure 0 0 0) 0x7f98a006d260 con 0x7f98a0142850 2023-10-19T13:04:44.700 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 64 2023-10-19T13:04:44.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.698+0000 7f98a80b4640 1 -- 172.21.15.57:0/3848284382 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f98a0142850 msgr2=0x7f98a01510c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:44.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.698+0000 7f98a80b4640 1 --2- 172.21.15.57:0/3848284382 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f98a0142850 0x7f98a01510c0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f989800c690 tx=0x7f989800cb50 comp rx=0 tx=0).stop 2023-10-19T13:04:44.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.698+0000 7f98a80b4640 1 -- 172.21.15.57:0/3848284382 shutdown_connections 2023-10-19T13:04:44.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.698+0000 7f98a80b4640 1 --2- 172.21.15.57:0/3848284382 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f98a0142850 0x7f98a01510c0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:44.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.698+0000 7f98a80b4640 1 --2- 172.21.15.57:0/3848284382 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f98a00cd6d0 0x7f98a00c7af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:44.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.698+0000 7f98a80b4640 1 --2- 172.21.15.57:0/3848284382 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f98a0069720 0x7f98a00c75b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:44.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.698+0000 7f98a80b4640 1 -- 172.21.15.57:0/3848284382 >> 172.21.15.57:0/3848284382 conn(0x7f98a00af7a0 msgr2=0x7f98a00c9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:44.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.698+0000 7f98a80b4640 1 -- 172.21.15.57:0/3848284382 shutdown_connections 2023-10-19T13:04:44.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.698+0000 7f98a80b4640 1 -- 172.21.15.57:0/3848284382 wait complete. 2023-10-19T13:04:44.712 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T13:04:44.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.817+0000 7fa04af83640 1 -- 172.21.15.57:0/3002174242 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0440c96b0 msgr2=0x7fa0440c9b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:44.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.817+0000 7fa04af83640 1 --2- 172.21.15.57:0/3002174242 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0440c96b0 0x7fa0440c9b30 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7fa04406d590 tx=0x7fa02c02bda0 comp rx=0 tx=0).stop 2023-10-19T13:04:44.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.817+0000 7fa04af83640 1 -- 172.21.15.57:0/3002174242 shutdown_connections 2023-10-19T13:04:44.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.817+0000 7fa04af83640 1 --2- 172.21.15.57:0/3002174242 >> v2:172.21.15.179:3300/0 conn(0x7fa0440ab500 0x7fa0440ab990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:44.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.817+0000 7fa04af83640 1 --2- 172.21.15.57:0/3002174242 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0440c96b0 0x7fa0440c9b30 unknown :-1 s=CLOSED pgs=1676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:44.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.817+0000 7fa04af83640 1 --2- 172.21.15.57:0/3002174242 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa04406af00 0x7fa04406b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:44.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.817+0000 7fa04af83640 1 -- 172.21.15.57:0/3002174242 >> 172.21.15.57:0/3002174242 conn(0x7fa044069930 msgr2=0x7fa044069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:44.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.817+0000 7fa04af83640 1 -- 172.21.15.57:0/3002174242 shutdown_connections 2023-10-19T13:04:44.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.818+0000 7fa04af83640 1 -- 172.21.15.57:0/3002174242 wait complete. 2023-10-19T13:04:44.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.819+0000 7fa04af83640 1 Processor -- start 2023-10-19T13:04:44.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.819+0000 7fa04af83640 1 -- start start 2023-10-19T13:04:44.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.819+0000 7fa04af83640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa04406af00 0x7fa0440aab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:44.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.819+0000 7fa04af83640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa0440ab500 0x7fa044151a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:44.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.819+0000 7fa04af83640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa0440c96b0 0x7fa044151fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:44.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.819+0000 7fa04af83640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0441f5ba0 con 0x7fa04406af00 2023-10-19T13:04:44.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.820+0000 7fa04af83640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa044070a00 con 0x7fa0440ab500 2023-10-19T13:04:44.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.820+0000 7fa04af83640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0441f58c0 con 0x7fa0440c96b0 2023-10-19T13:04:44.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.820+0000 7fa048d1f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa04406af00 0x7fa0440aab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:44.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.820+0000 7fa048d1f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa04406af00 0x7fa0440aab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57496/0 (socket says 172.21.15.57:57496) 2023-10-19T13:04:44.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.820+0000 7fa048d1f640 1 -- 172.21.15.57:0/319555481 learned_addr learned my addr 172.21.15.57:0/319555481 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:44.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.820+0000 7fa049520640 1 --2- 172.21.15.57:0/319555481 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa0440c96b0 0x7fa044151fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:44.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.820+0000 7fa043fff640 1 --2- 172.21.15.57:0/319555481 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa0440ab500 0x7fa044151a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:44.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.820+0000 7fa048d1f640 1 -- 172.21.15.57:0/319555481 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa0440c96b0 msgr2=0x7fa044151fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:44.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.820+0000 7fa048d1f640 1 --2- 172.21.15.57:0/319555481 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa0440c96b0 0x7fa044151fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:44.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.820+0000 7fa048d1f640 1 -- 172.21.15.57:0/319555481 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa0440ab500 msgr2=0x7fa044151a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:44.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.820+0000 7fa048d1f640 1 --2- 172.21.15.57:0/319555481 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa0440ab500 0x7fa044151a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:44.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.820+0000 7fa048d1f640 1 -- 172.21.15.57:0/319555481 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa02c009560 con 0x7fa04406af00 2023-10-19T13:04:44.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.820+0000 7fa048d1f640 1 --2- 172.21.15.57:0/319555481 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa04406af00 0x7fa0440aab30 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7fa03c00eae0 tx=0x7fa03c00c270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:44.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.820+0000 7fa0417fa640 1 -- 172.21.15.57:0/319555481 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa03c00ce50 con 0x7fa04406af00 2023-10-19T13:04:44.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.820+0000 7fa0417fa640 1 -- 172.21.15.57:0/319555481 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fa03c011560 con 0x7fa04406af00 2023-10-19T13:04:44.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.820+0000 7fa04af83640 1 -- 172.21.15.57:0/319555481 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa04406f220 con 0x7fa04406af00 2023-10-19T13:04:44.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.820+0000 7fa0417fa640 1 -- 172.21.15.57:0/319555481 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa03c011950 con 0x7fa04406af00 2023-10-19T13:04:44.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.820+0000 7fa04af83640 1 -- 172.21.15.57:0/319555481 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa04406f7e0 con 0x7fa04406af00 2023-10-19T13:04:44.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.821+0000 7fa04af83640 1 -- 172.21.15.57:0/319555481 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa04406f060 con 0x7fa04406af00 2023-10-19T13:04:44.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.822+0000 7fa0417fa640 1 -- 172.21.15.57:0/319555481 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 76) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fa03c011ab0 con 0x7fa04406af00 2023-10-19T13:04:44.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.823+0000 7fa0417fa640 1 -- 172.21.15.57:0/319555481 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 6788+0+0 (secure 0 0 0) 0x7fa03c08a1d0 con 0x7fa04406af00 2023-10-19T13:04:44.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:44.827+0000 7fa0417fa640 1 -- 172.21.15.57:0/319555481 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa03c055250 con 0x7fa04406af00 2023-10-19T13:04:45.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.041+0000 7fa04af83640 1 -- 172.21.15.57:0/319555481 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa0440b1ea0 con 0x7fa04406af00 2023-10-19T13:04:45.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.041+0000 7fa0417fa640 1 -- 172.21.15.57:0/319555481 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+4921 (secure 0 0 0) 0x7fa03c058870 con 0x7fa04406af00 2023-10-19T13:04:45.045 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:04:45.045 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":56,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7325,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/3318110028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":3318110028},{"type":"v1","addr":"172.21.15.179:6837","nonce":3318110028}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":8143,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/3625556517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":3625556517},{"type":"v1","addr":"172.21.15.57:6835","nonce":3625556517}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":8158,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/4167860955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":4167860955},{"type":"v1","addr":"172.21.15.179:6836","nonce":4167860955}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T13:04:22.785432+0000","modified":"2023-10-19T13:04:24.778587+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9066},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9066":{"gid":9066,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":3,"addr":"172.21.15.57:6834/3474229632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":3474229632},{"type":"v1","addr":"172.21.15.57:6834","nonce":3474229632}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2023-10-19T13:04:45.045 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 56 2023-10-19T13:04:45.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.043+0000 7fa04af83640 1 -- 172.21.15.57:0/319555481 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa04406af00 msgr2=0x7fa0440aab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:45.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.043+0000 7fa04af83640 1 --2- 172.21.15.57:0/319555481 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa04406af00 0x7fa0440aab30 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7fa03c00eae0 tx=0x7fa03c00c270 comp rx=0 tx=0).stop 2023-10-19T13:04:45.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.043+0000 7fa04af83640 1 -- 172.21.15.57:0/319555481 shutdown_connections 2023-10-19T13:04:45.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.044+0000 7fa04af83640 1 --2- 172.21.15.57:0/319555481 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa0440c96b0 0x7fa044151fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:45.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.044+0000 7fa04af83640 1 --2- 172.21.15.57:0/319555481 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa0440ab500 0x7fa044151a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:45.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.044+0000 7fa04af83640 1 --2- 172.21.15.57:0/319555481 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa04406af00 0x7fa0440aab30 unknown :-1 s=CLOSED pgs=1677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:45.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.044+0000 7fa04af83640 1 -- 172.21.15.57:0/319555481 >> 172.21.15.57:0/319555481 conn(0x7fa044069930 msgr2=0x7fa04413ea60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:45.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.044+0000 7fa04af83640 1 -- 172.21.15.57:0/319555481 shutdown_connections 2023-10-19T13:04:45.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.044+0000 7fa04af83640 1 -- 172.21.15.57:0/319555481 wait complete. 2023-10-19T13:04:45.057 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T13:04:45.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.164+0000 7fbaf579e640 1 -- 172.21.15.57:0/3337460314 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbaf015c1f0 msgr2=0x7fbaf00b0e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:45.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.164+0000 7fbaf579e640 1 --2- 172.21.15.57:0/3337460314 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbaf015c1f0 0x7fbaf00b0e90 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7fbae0005fd0 tx=0x7fbae002c0e0 comp rx=0 tx=0).stop 2023-10-19T13:04:45.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.164+0000 7fbaf579e640 1 -- 172.21.15.57:0/3337460314 shutdown_connections 2023-10-19T13:04:45.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.164+0000 7fbaf579e640 1 --2- 172.21.15.57:0/3337460314 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbaf015c1f0 0x7fbaf00b0e90 unknown :-1 s=CLOSED pgs=1679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:45.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.164+0000 7fbaf579e640 1 -- 172.21.15.57:0/3337460314 >> 172.21.15.57:0/3337460314 conn(0x7fbaf006fae0 msgr2=0x7fbaf006fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:45.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.164+0000 7fbaf579e640 1 -- 172.21.15.57:0/3337460314 shutdown_connections 2023-10-19T13:04:45.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.164+0000 7fbaf579e640 1 -- 172.21.15.57:0/3337460314 wait complete. 2023-10-19T13:04:45.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.165+0000 7fbaf579e640 1 Processor -- start 2023-10-19T13:04:45.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.165+0000 7fbaf579e640 1 -- start start 2023-10-19T13:04:45.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.166+0000 7fbaf579e640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbaf006ae40 0x7fbaf0159120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:45.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.166+0000 7fbaf579e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbaf01579f0 0x7fbaf00c4b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:45.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.166+0000 7fbaf579e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbaf015c1f0 0x7fbaf014cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:45.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.166+0000 7fbaf579e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf00c50c0 con 0x7fbaf015c1f0 2023-10-19T13:04:45.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.166+0000 7fbaf579e640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf01597b0 con 0x7fbaf01579f0 2023-10-19T13:04:45.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.166+0000 7fbaf579e640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf0159a90 con 0x7fbaf006ae40 2023-10-19T13:04:45.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.166+0000 7fbaef7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbaf015c1f0 0x7fbaf014cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:45.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.166+0000 7fbaef7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbaf015c1f0 0x7fbaf014cdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57508/0 (socket says 172.21.15.57:57508) 2023-10-19T13:04:45.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.166+0000 7fbaef7fe640 1 -- 172.21.15.57:0/680867301 learned_addr learned my addr 172.21.15.57:0/680867301 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:45.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.166+0000 7fbaeeffd640 1 --2- 172.21.15.57:0/680867301 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbaf006ae40 0x7fbaf0159120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:45.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.166+0000 7fbaee7fc640 1 --2- 172.21.15.57:0/680867301 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbaf01579f0 0x7fbaf00c4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:45.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.166+0000 7fbaef7fe640 1 -- 172.21.15.57:0/680867301 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbaf006ae40 msgr2=0x7fbaf0159120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:45.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.166+0000 7fbaef7fe640 1 --2- 172.21.15.57:0/680867301 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbaf006ae40 0x7fbaf0159120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:45.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.166+0000 7fbaef7fe640 1 -- 172.21.15.57:0/680867301 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbaf01579f0 msgr2=0x7fbaf00c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:45.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.166+0000 7fbaef7fe640 1 --2- 172.21.15.57:0/680867301 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbaf01579f0 0x7fbaf00c4b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:45.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.166+0000 7fbaef7fe640 1 -- 172.21.15.57:0/680867301 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbae0009840 con 0x7fbaf015c1f0 2023-10-19T13:04:45.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.166+0000 7fbaef7fe640 1 --2- 172.21.15.57:0/680867301 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbaf015c1f0 0x7fbaf014cdb0 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7fbae0009cd0 tx=0x7fbae0004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:45.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.167+0000 7fbae77fe640 1 -- 172.21.15.57:0/680867301 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbae0004760 con 0x7fbaf015c1f0 2023-10-19T13:04:45.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.167+0000 7fbae77fe640 1 -- 172.21.15.57:0/680867301 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fbae00345a0 con 0x7fbaf015c1f0 2023-10-19T13:04:45.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.167+0000 7fbae77fe640 1 -- 172.21.15.57:0/680867301 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbae0034990 con 0x7fbaf015c1f0 2023-10-19T13:04:45.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.167+0000 7fbaf579e640 1 -- 172.21.15.57:0/680867301 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbaf0159d70 con 0x7fbaf015c1f0 2023-10-19T13:04:45.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.167+0000 7fbaf579e640 1 -- 172.21.15.57:0/680867301 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbaf014d2f0 con 0x7fbaf015c1f0 2023-10-19T13:04:45.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.168+0000 7fbaf579e640 1 -- 172.21.15.57:0/680867301 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbaf014d480 con 0x7fbaf015c1f0 2023-10-19T13:04:45.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.168+0000 7fbae77fe640 1 -- 172.21.15.57:0/680867301 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 76) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fbae0034af0 con 0x7fbaf015c1f0 2023-10-19T13:04:45.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.169+0000 7fbae77fe640 1 -- 172.21.15.57:0/680867301 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 6788+0+0 (secure 0 0 0) 0x7fbae00ac780 con 0x7fbaf015c1f0 2023-10-19T13:04:45.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.173+0000 7fbae77fe640 1 -- 172.21.15.57:0/680867301 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fbaf014d480 con 0x7fbaf015c1f0 2023-10-19T13:04:45.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.284+0000 7fbae77fe640 1 -- 172.21.15.57:0/680867301 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mgrmap(e 77) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fbae00772b0 con 0x7fbaf015c1f0 2023-10-19T13:04:45.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.285+0000 7fbae77fe640 1 --2- 172.21.15.57:0/680867301 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fbad0070bd0 0x7fbad0072fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:45.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.286+0000 7fbaeeffd640 1 --2- 172.21.15.57:0/680867301 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fbad0070bd0 0x7fbad0072fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:45.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.290+0000 7fbaeeffd640 1 --2- 172.21.15.57:0/680867301 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fbad0070bd0 0x7fbad0072fc0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fbae8004f60 tx=0x7fbae8007e30 comp rx=0 tx=0).ready entity=mgr.8419 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:45.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.354+0000 7fbaf579e640 1 -- 172.21.15.57:0/680867301 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbaf014d480 con 0x7fbaf015c1f0 2023-10-19T13:04:45.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.355+0000 7fbae77fe640 1 -- 172.21.15.57:0/680867301 <== mon.0 v2:172.21.15.57:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v146) v1 ==== 74+0+17921 (secure 0 0 0) 0x7fbaf014d480 con 0x7fbaf015c1f0 2023-10-19T13:04:45.358 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:04:45.358 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":146,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T13:04:44.277597+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2023-10-19T13:04:20.932329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"143","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2023-10-19T13:04:21.762157+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"144","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.179:0/2820959197":"2023-10-20T13:04:44.277539+0000","172.21.15.179:6833/3526956956":"2023-10-20T13:04:44.277539+0000","172.21.15.179:0/2214725838":"2023-10-20T13:04:44.277539+0000","172.21.15.179:6832/3526956956":"2023-10-20T13:04:44.277539+0000","172.21.15.179:0/2775321950":"2023-10-20T13:04:44.277539+0000","172.21.15.179:0/3591554987":"2023-10-20T13:04:44.277539+0000","172.21.15.179:0/1299954463":"2023-10-20T13:04:44.277539+0000","172.21.15.179:0/1024578486":"2023-10-20T13:04:44.277539+0000","172.21.15.179:0/1700025841":"2023-10-20T13:04:33.816783+0000","172.21.15.179:0/3160483813":"2023-10-20T13:04:33.816783+0000","172.21.15.179:0/4039784875":"2023-10-20T13:04:33.816783+0000","172.21.15.179:0/1023851911":"2023-10-20T13:04:33.816783+0000","172.21.15.179:0/2968654981":"2023-10-20T13:04:33.816783+0000","172.21.15.179:0/1247110426":"2023-10-20T13:04:44.277539+0000","172.21.15.179:0/3605212267":"2023-10-20T13:04:33.816783+0000","172.21.15.179:6833/2346394725":"2023-10-20T13:04:33.816783+0000","172.21.15.179:6832/2346394725":"2023-10-20T13:04:33.816783+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T13:04:45.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.357+0000 7fbaf579e640 1 -- 172.21.15.57:0/680867301 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fbad0070bd0 msgr2=0x7fbad0072fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:45.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.357+0000 7fbaf579e640 1 --2- 172.21.15.57:0/680867301 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fbad0070bd0 0x7fbad0072fc0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fbae8004f60 tx=0x7fbae8007e30 comp rx=0 tx=0).stop 2023-10-19T13:04:45.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.357+0000 7fbaf579e640 1 -- 172.21.15.57:0/680867301 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbaf015c1f0 msgr2=0x7fbaf014cdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:45.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.357+0000 7fbaf579e640 1 --2- 172.21.15.57:0/680867301 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbaf015c1f0 0x7fbaf014cdb0 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7fbae0009cd0 tx=0x7fbae0004360 comp rx=0 tx=0).stop 2023-10-19T13:04:45.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.357+0000 7fbaf579e640 1 -- 172.21.15.57:0/680867301 shutdown_connections 2023-10-19T13:04:45.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.357+0000 7fbaf579e640 1 --2- 172.21.15.57:0/680867301 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fbad0070bd0 0x7fbad0072fc0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:45.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.357+0000 7fbaf579e640 1 --2- 172.21.15.57:0/680867301 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbaf015c1f0 0x7fbaf014cdb0 unknown :-1 s=CLOSED pgs=1680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:45.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.357+0000 7fbaf579e640 1 --2- 172.21.15.57:0/680867301 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbaf01579f0 0x7fbaf00c4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:45.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.357+0000 7fbaf579e640 1 --2- 172.21.15.57:0/680867301 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbaf006ae40 0x7fbaf0159120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:45.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.357+0000 7fbaf579e640 1 -- 172.21.15.57:0/680867301 >> 172.21.15.57:0/680867301 conn(0x7fbaf006fae0 msgr2=0x7fbaf006b980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:45.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.357+0000 7fbaf579e640 1 -- 172.21.15.57:0/680867301 shutdown_connections 2023-10-19T13:04:45.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.357+0000 7fbaf579e640 1 -- 172.21.15.57:0/680867301 wait complete. 2023-10-19T13:04:45.370 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-19T13:04:45.371 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T13:04:45.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.478+0000 7feb7c9a0640 1 -- 172.21.15.57:0/3205343942 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feb60003970 msgr2=0x7feb60003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:45.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.478+0000 7feb7c9a0640 1 --2- 172.21.15.57:0/3205343942 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feb60003970 0x7feb60003d90 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7feb68001680 tx=0x7feb600057f0 comp rx=0 tx=0).stop 2023-10-19T13:04:45.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.478+0000 7feb7c9a0640 1 -- 172.21.15.57:0/3205343942 shutdown_connections 2023-10-19T13:04:45.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.478+0000 7feb7c9a0640 1 --2- 172.21.15.57:0/3205343942 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feb60003970 0x7feb60003d90 unknown :-1 s=CLOSED pgs=1682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:45.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.478+0000 7feb7c9a0640 1 -- 172.21.15.57:0/3205343942 >> 172.21.15.57:0/3205343942 conn(0x7feb780ae5d0 msgr2=0x7feb780ae9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:45.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.478+0000 7feb7c9a0640 1 -- 172.21.15.57:0/3205343942 shutdown_connections 2023-10-19T13:04:45.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.478+0000 7feb7c9a0640 1 -- 172.21.15.57:0/3205343942 wait complete. 2023-10-19T13:04:45.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.479+0000 7feb7c9a0640 1 Processor -- start 2023-10-19T13:04:45.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.480+0000 7feb7c9a0640 1 -- start start 2023-10-19T13:04:45.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.480+0000 7feb7c9a0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feb60003970 0x7feb78147c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:45.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.480+0000 7feb7c9a0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feb781547b0 0x7feb78154c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:45.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.480+0000 7feb7c9a0640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7feb7806f1e0 0x7feb7806f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:45.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.480+0000 7feb7c9a0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb78155150 con 0x7feb781547b0 2023-10-19T13:04:45.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.480+0000 7feb7c9a0640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb7806fba0 con 0x7feb60003970 2023-10-19T13:04:45.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.480+0000 7feb7c9a0640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb78153940 con 0x7feb7806f1e0 2023-10-19T13:04:45.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.480+0000 7feb76ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feb781547b0 0x7feb78154c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:45.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.481+0000 7feb76ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feb781547b0 0x7feb78154c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57530/0 (socket says 172.21.15.57:57530) 2023-10-19T13:04:45.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.481+0000 7feb76ffd640 1 -- 172.21.15.57:0/3809618671 learned_addr learned my addr 172.21.15.57:0/3809618671 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:45.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.481+0000 7feb777fe640 1 --2- 172.21.15.57:0/3809618671 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feb60003970 0x7feb78147c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:45.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.481+0000 7feb77fff640 1 --2- 172.21.15.57:0/3809618671 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7feb7806f1e0 0x7feb7806f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:45.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.481+0000 7feb76ffd640 1 -- 172.21.15.57:0/3809618671 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7feb7806f1e0 msgr2=0x7feb7806f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:45.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.481+0000 7feb76ffd640 1 --2- 172.21.15.57:0/3809618671 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7feb7806f1e0 0x7feb7806f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:45.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.481+0000 7feb76ffd640 1 -- 172.21.15.57:0/3809618671 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feb60003970 msgr2=0x7feb78147c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:45.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.481+0000 7feb76ffd640 1 --2- 172.21.15.57:0/3809618671 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feb60003970 0x7feb78147c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:45.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.481+0000 7feb76ffd640 1 -- 172.21.15.57:0/3809618671 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb781525c0 con 0x7feb781547b0 2023-10-19T13:04:45.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.481+0000 7feb76ffd640 1 --2- 172.21.15.57:0/3809618671 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feb781547b0 0x7feb78154c10 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7feb6c00ea00 tx=0x7feb6c00eec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:45.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.481+0000 7feb5ffff640 1 -- 172.21.15.57:0/3809618671 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb6c014070 con 0x7feb781547b0 2023-10-19T13:04:45.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.481+0000 7feb5ffff640 1 -- 172.21.15.57:0/3809618671 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7feb6c00dcf0 con 0x7feb781547b0 2023-10-19T13:04:45.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.481+0000 7feb5ffff640 1 -- 172.21.15.57:0/3809618671 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb6c004280 con 0x7feb781547b0 2023-10-19T13:04:45.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.481+0000 7feb7c9a0640 1 -- 172.21.15.57:0/3809618671 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb781528a0 con 0x7feb781547b0 2023-10-19T13:04:45.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.481+0000 7feb7c9a0640 1 -- 172.21.15.57:0/3809618671 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb78152e60 con 0x7feb781547b0 2023-10-19T13:04:45.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.482+0000 7feb7c9a0640 1 -- 172.21.15.57:0/3809618671 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb7804da70 con 0x7feb781547b0 2023-10-19T13:04:45.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.483+0000 7feb5ffff640 1 -- 172.21.15.57:0/3809618671 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 77) v1 ==== 89067+0+0 (secure 0 0 0) 0x7feb6c0035f0 con 0x7feb781547b0 2023-10-19T13:04:45.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.483+0000 7feb5ffff640 1 --2- 172.21.15.57:0/3809618671 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7feb6406acc0 0x7feb6406d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:45.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.483+0000 7feb5ffff640 1 -- 172.21.15.57:0/3809618671 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 6788+0+0 (secure 0 0 0) 0x7feb6c089ce0 con 0x7feb781547b0 2023-10-19T13:04:45.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.483+0000 7feb777fe640 1 --2- 172.21.15.57:0/3809618671 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7feb6406acc0 0x7feb6406d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:45.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.484+0000 7feb777fe640 1 --2- 172.21.15.57:0/3809618671 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7feb6406acc0 0x7feb6406d180 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7feb68034000 tx=0x7feb68001190 comp rx=0 tx=0).ready entity=mgr.8419 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:45.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.487+0000 7feb5ffff640 1 -- 172.21.15.57:0/3809618671 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7feb6c08e070 con 0x7feb781547b0 2023-10-19T13:04:45.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.697+0000 7feb7c9a0640 1 -- 172.21.15.57:0/3809618671 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feb780cb250 con 0x7feb781547b0 2023-10-19T13:04:45.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.698+0000 7feb5ffff640 1 -- 172.21.15.57:0/3809618671 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+4921 (secure 0 0 0) 0x7feb6c054cb0 con 0x7feb781547b0 2023-10-19T13:04:45.701 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:04:45.701 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":56,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7325,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/3318110028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":3318110028},{"type":"v1","addr":"172.21.15.179:6837","nonce":3318110028}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":8143,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/3625556517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":3625556517},{"type":"v1","addr":"172.21.15.57:6835","nonce":3625556517}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":8158,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/4167860955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":4167860955},{"type":"v1","addr":"172.21.15.179:6836","nonce":4167860955}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T13:04:22.785432+0000","modified":"2023-10-19T13:04:24.778587+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9066},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9066":{"gid":9066,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":3,"addr":"172.21.15.57:6834/3474229632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":3474229632},{"type":"v1","addr":"172.21.15.57:6834","nonce":3474229632}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2023-10-19T13:04:45.701 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 56 2023-10-19T13:04:45.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.699+0000 7feb7c9a0640 1 -- 172.21.15.57:0/3809618671 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7feb6406acc0 msgr2=0x7feb6406d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:45.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.699+0000 7feb7c9a0640 1 --2- 172.21.15.57:0/3809618671 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7feb6406acc0 0x7feb6406d180 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7feb68034000 tx=0x7feb68001190 comp rx=0 tx=0).stop 2023-10-19T13:04:45.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.699+0000 7feb7c9a0640 1 -- 172.21.15.57:0/3809618671 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feb781547b0 msgr2=0x7feb78154c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:45.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.699+0000 7feb7c9a0640 1 --2- 172.21.15.57:0/3809618671 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feb781547b0 0x7feb78154c10 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7feb6c00ea00 tx=0x7feb6c00eec0 comp rx=0 tx=0).stop 2023-10-19T13:04:45.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.700+0000 7feb7c9a0640 1 -- 172.21.15.57:0/3809618671 shutdown_connections 2023-10-19T13:04:45.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.700+0000 7feb7c9a0640 1 --2- 172.21.15.57:0/3809618671 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7feb6406acc0 0x7feb6406d180 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:45.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.700+0000 7feb7c9a0640 1 --2- 172.21.15.57:0/3809618671 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7feb7806f1e0 0x7feb7806f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:45.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.700+0000 7feb7c9a0640 1 --2- 172.21.15.57:0/3809618671 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feb781547b0 0x7feb78154c10 unknown :-1 s=CLOSED pgs=1683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:45.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.700+0000 7feb7c9a0640 1 --2- 172.21.15.57:0/3809618671 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feb60003970 0x7feb78147c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:45.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.700+0000 7feb7c9a0640 1 -- 172.21.15.57:0/3809618671 >> 172.21.15.57:0/3809618671 conn(0x7feb780ae5d0 msgr2=0x7feb780adbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:45.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.700+0000 7feb7c9a0640 1 -- 172.21.15.57:0/3809618671 shutdown_connections 2023-10-19T13:04:45.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.700+0000 7feb7c9a0640 1 -- 172.21.15.57:0/3809618671 wait complete. 2023-10-19T13:04:45.713 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T13:04:45.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.820+0000 7f7b78ca9640 1 -- 172.21.15.57:0/181048593 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7b740abc10 msgr2=0x7f7b740c9b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:45.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.820+0000 7f7b78ca9640 1 --2- 172.21.15.57:0/181048593 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7b740abc10 0x7f7b740c9b60 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f7b6c009be0 tx=0x7f7b6c02c0c0 comp rx=0 tx=0).stop 2023-10-19T13:04:45.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.820+0000 7f7b78ca9640 1 -- 172.21.15.57:0/181048593 shutdown_connections 2023-10-19T13:04:45.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.820+0000 7f7b78ca9640 1 --2- 172.21.15.57:0/181048593 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7b740abc10 0x7f7b740c9b60 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:45.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.820+0000 7f7b78ca9640 1 --2- 172.21.15.57:0/181048593 >> v2:172.21.15.179:3300/0 conn(0x7f7b7406af00 0x7f7b7406b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:45.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.820+0000 7f7b78ca9640 1 -- 172.21.15.57:0/181048593 >> 172.21.15.57:0/181048593 conn(0x7f7b74069930 msgr2=0x7f7b74069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:45.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.820+0000 7f7b78ca9640 1 -- 172.21.15.57:0/181048593 shutdown_connections 2023-10-19T13:04:45.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.820+0000 7f7b78ca9640 1 -- 172.21.15.57:0/181048593 wait complete. 2023-10-19T13:04:45.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.821+0000 7f7b78ca9640 1 Processor -- start 2023-10-19T13:04:45.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.821+0000 7f7b78ca9640 1 -- start start 2023-10-19T13:04:45.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.822+0000 7f7b78ca9640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7b7406af00 0x7f7b74137d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:45.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.822+0000 7f7b78ca9640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7b740abc10 0x7f7b740ac320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:45.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.822+0000 7f7b78ca9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7b740c96b0 0x7f7b740ac860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:45.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.822+0000 7f7b78ca9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b740af440 con 0x7f7b740c96b0 2023-10-19T13:04:45.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.822+0000 7f7b78ca9640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b740ae830 con 0x7f7b7406af00 2023-10-19T13:04:45.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.822+0000 7f7b78ca9640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b740aeb10 con 0x7f7b740abc10 2023-10-19T13:04:45.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.822+0000 7f7b72d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7b740c96b0 0x7f7b740ac860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:45.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.822+0000 7f7b72d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7b740c96b0 0x7f7b740ac860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57542/0 (socket says 172.21.15.57:57542) 2023-10-19T13:04:45.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.822+0000 7f7b72d9d640 1 -- 172.21.15.57:0/3487198312 learned_addr learned my addr 172.21.15.57:0/3487198312 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:45.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.822+0000 7f7b7259c640 1 --2- 172.21.15.57:0/3487198312 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7b7406af00 0x7f7b74137d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:45.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.822+0000 7f7b6bfff640 1 --2- 172.21.15.57:0/3487198312 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7b740abc10 0x7f7b740ac320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:45.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.822+0000 7f7b72d9d640 1 -- 172.21.15.57:0/3487198312 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7b740abc10 msgr2=0x7f7b740ac320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:45.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.822+0000 7f7b72d9d640 1 --2- 172.21.15.57:0/3487198312 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7b740abc10 0x7f7b740ac320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:45.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.822+0000 7f7b72d9d640 1 -- 172.21.15.57:0/3487198312 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7b7406af00 msgr2=0x7f7b74137d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:45.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.822+0000 7f7b72d9d640 1 --2- 172.21.15.57:0/3487198312 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7b7406af00 0x7f7b74137d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:45.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.822+0000 7f7b72d9d640 1 -- 172.21.15.57:0/3487198312 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b64009560 con 0x7f7b740c96b0 2023-10-19T13:04:45.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.822+0000 7f7b6bfff640 1 --2- 172.21.15.57:0/3487198312 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7b740abc10 0x7f7b740ac320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T13:04:45.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.823+0000 7f7b72d9d640 1 --2- 172.21.15.57:0/3487198312 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7b740c96b0 0x7f7b740ac860 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7f7b6c009be0 tx=0x7f7b6c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:45.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.823+0000 7f7b6affd640 1 -- 172.21.15.57:0/3487198312 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b6c0046f0 con 0x7f7b740c96b0 2023-10-19T13:04:45.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.823+0000 7f7b6affd640 1 -- 172.21.15.57:0/3487198312 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f7b6c0345a0 con 0x7f7b740c96b0 2023-10-19T13:04:45.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.823+0000 7f7b6affd640 1 -- 172.21.15.57:0/3487198312 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b6c0349b0 con 0x7f7b740c96b0 2023-10-19T13:04:45.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.823+0000 7f7b78ca9640 1 -- 172.21.15.57:0/3487198312 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b6c009840 con 0x7f7b740c96b0 2023-10-19T13:04:45.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.823+0000 7f7b78ca9640 1 -- 172.21.15.57:0/3487198312 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b740af0d0 con 0x7f7b740c96b0 2023-10-19T13:04:45.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.824+0000 7f7b78ca9640 1 -- 172.21.15.57:0/3487198312 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b740af5f0 con 0x7f7b740c96b0 2023-10-19T13:04:45.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.824+0000 7f7b6affd640 1 -- 172.21.15.57:0/3487198312 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 77) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f7b6c034b10 con 0x7f7b740c96b0 2023-10-19T13:04:45.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.825+0000 7f7b6affd640 1 --2- 172.21.15.57:0/3487198312 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f7b5406ac90 0x7f7b5406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:45.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.825+0000 7f7b6affd640 1 -- 172.21.15.57:0/3487198312 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 6788+0+0 (secure 0 0 0) 0x7f7b6c0ac830 con 0x7f7b740c96b0 2023-10-19T13:04:45.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.825+0000 7f7b7259c640 1 --2- 172.21.15.57:0/3487198312 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f7b5406ac90 0x7f7b5406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:45.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.825+0000 7f7b7259c640 1 --2- 172.21.15.57:0/3487198312 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f7b5406ac90 0x7f7b5406d150 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f7b74136250 tx=0x7f7b64009260 comp rx=0 tx=0).ready entity=mgr.8419 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:45.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:45.829+0000 7f7b6affd640 1 -- 172.21.15.57:0/3487198312 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7b740af5f0 con 0x7f7b740c96b0 2023-10-19T13:04:46.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.044+0000 7f7b78ca9640 1 -- 172.21.15.57:0/3487198312 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7b740690e0 con 0x7f7b740c96b0 2023-10-19T13:04:46.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.044+0000 7f7b6affd640 1 -- 172.21.15.57:0/3487198312 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+4921 (secure 0 0 0) 0x7f7b6c0777e0 con 0x7f7b740c96b0 2023-10-19T13:04:46.047 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:04:46.048 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":56,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7325,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/3318110028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":3318110028},{"type":"v1","addr":"172.21.15.179:6837","nonce":3318110028}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":8143,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/3625556517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":3625556517},{"type":"v1","addr":"172.21.15.57:6835","nonce":3625556517}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":8158,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/4167860955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":4167860955},{"type":"v1","addr":"172.21.15.179:6836","nonce":4167860955}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T13:04:22.785432+0000","modified":"2023-10-19T13:04:24.778587+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9066},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9066":{"gid":9066,"name":"c","rank":0,"incarnation":54,"state":"up:active","state_seq":3,"addr":"172.21.15.57:6834/3474229632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":3474229632},{"type":"v1","addr":"172.21.15.57:6834","nonce":3474229632}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2023-10-19T13:04:46.048 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 56 2023-10-19T13:04:46.049 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.046+0000 7f7b78ca9640 1 -- 172.21.15.57:0/3487198312 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f7b5406ac90 msgr2=0x7f7b5406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:46.049 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.046+0000 7f7b78ca9640 1 --2- 172.21.15.57:0/3487198312 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f7b5406ac90 0x7f7b5406d150 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f7b74136250 tx=0x7f7b64009260 comp rx=0 tx=0).stop 2023-10-19T13:04:46.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.046+0000 7f7b78ca9640 1 -- 172.21.15.57:0/3487198312 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7b740c96b0 msgr2=0x7f7b740ac860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:46.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.046+0000 7f7b78ca9640 1 --2- 172.21.15.57:0/3487198312 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7b740c96b0 0x7f7b740ac860 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7f7b6c009be0 tx=0x7f7b6c0042f0 comp rx=0 tx=0).stop 2023-10-19T13:04:46.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.046+0000 7f7b78ca9640 1 -- 172.21.15.57:0/3487198312 shutdown_connections 2023-10-19T13:04:46.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.046+0000 7f7b78ca9640 1 --2- 172.21.15.57:0/3487198312 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f7b5406ac90 0x7f7b5406d150 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:46.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.046+0000 7f7b78ca9640 1 --2- 172.21.15.57:0/3487198312 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7b740c96b0 0x7f7b740ac860 unknown :-1 s=CLOSED pgs=1684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:46.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.046+0000 7f7b78ca9640 1 --2- 172.21.15.57:0/3487198312 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7b740abc10 0x7f7b740ac320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:46.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.046+0000 7f7b78ca9640 1 --2- 172.21.15.57:0/3487198312 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7b7406af00 0x7f7b74137d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:46.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.046+0000 7f7b78ca9640 1 -- 172.21.15.57:0/3487198312 >> 172.21.15.57:0/3487198312 conn(0x7f7b74069930 msgr2=0x7f7b74069d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:46.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.046+0000 7f7b78ca9640 1 -- 172.21.15.57:0/3487198312 shutdown_connections 2023-10-19T13:04:46.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.046+0000 7f7b78ca9640 1 -- 172.21.15.57:0/3487198312 wait complete. 2023-10-19T13:04:46.060 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T13:04:46.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.166+0000 7f78c3fd0640 1 -- 172.21.15.57:0/2915617667 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f78bc0cf340 msgr2=0x7f78bc0d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:46.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.166+0000 7f78c3fd0640 1 --2- 172.21.15.57:0/2915617667 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f78bc0cf340 0x7f78bc0d1910 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7f78ac005fd0 tx=0x7f78ac02c250 comp rx=0 tx=0).stop 2023-10-19T13:04:46.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.166+0000 7f78c3fd0640 1 -- 172.21.15.57:0/2915617667 shutdown_connections 2023-10-19T13:04:46.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.166+0000 7f78c3fd0640 1 --2- 172.21.15.57:0/2915617667 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f78bc0cf340 0x7f78bc0d1910 unknown :-1 s=CLOSED pgs=1686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:46.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.166+0000 7f78c3fd0640 1 -- 172.21.15.57:0/2915617667 >> 172.21.15.57:0/2915617667 conn(0x7f78bc0d0d60 msgr2=0x7f78bc0d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:46.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.166+0000 7f78c3fd0640 1 -- 172.21.15.57:0/2915617667 shutdown_connections 2023-10-19T13:04:46.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.166+0000 7f78c3fd0640 1 -- 172.21.15.57:0/2915617667 wait complete. 2023-10-19T13:04:46.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.167+0000 7f78c3fd0640 1 Processor -- start 2023-10-19T13:04:46.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.167+0000 7f78c3fd0640 1 -- start start 2023-10-19T13:04:46.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.168+0000 7f78c3fd0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f78bc0ceb40 0x7f78bc1f2520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:46.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.168+0000 7f78c3fd0640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f78bc0cf340 0x7f78bc1f2c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:46.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.168+0000 7f78c3fd0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f78bc15b2a0 0x7f78bc1e5e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:46.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.168+0000 7f78c3fd0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78bc1e59a0 con 0x7f78bc15b2a0 2023-10-19T13:04:46.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.168+0000 7f78c3fd0640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78bc1f2a60 con 0x7f78bc0ceb40 2023-10-19T13:04:46.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.168+0000 7f78c3fd0640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78bc1e4d90 con 0x7f78bc0cf340 2023-10-19T13:04:46.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.168+0000 7f78c256d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f78bc15b2a0 0x7f78bc1e5e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:46.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.168+0000 7f78c256d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f78bc15b2a0 0x7f78bc1e5e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57562/0 (socket says 172.21.15.57:57562) 2023-10-19T13:04:46.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.168+0000 7f78c256d640 1 -- 172.21.15.57:0/868120559 learned_addr learned my addr 172.21.15.57:0/868120559 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:46.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.168+0000 7f78c256d640 1 -- 172.21.15.57:0/868120559 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f78bc0cf340 msgr2=0x7f78bc1f2c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:46.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.168+0000 7f78c256d640 1 --2- 172.21.15.57:0/868120559 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f78bc0cf340 0x7f78bc1f2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:46.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.168+0000 7f78c256d640 1 -- 172.21.15.57:0/868120559 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f78bc0ceb40 msgr2=0x7f78bc1f2520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:46.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.168+0000 7f78c256d640 1 --2- 172.21.15.57:0/868120559 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f78bc0ceb40 0x7f78bc1f2520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:46.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.169+0000 7f78c256d640 1 -- 172.21.15.57:0/868120559 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78ac009900 con 0x7f78bc15b2a0 2023-10-19T13:04:46.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.169+0000 7f78c256d640 1 --2- 172.21.15.57:0/868120559 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f78bc15b2a0 0x7f78bc1e5e30 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7f78b400c3c0 tx=0x7f78b400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:46.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.169+0000 7f78ba7fc640 1 -- 172.21.15.57:0/868120559 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78b40035f0 con 0x7f78bc15b2a0 2023-10-19T13:04:46.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.169+0000 7f78ba7fc640 1 -- 172.21.15.57:0/868120559 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f78b4003770 con 0x7f78bc15b2a0 2023-10-19T13:04:46.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.169+0000 7f78c3fd0640 1 -- 172.21.15.57:0/868120559 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78bc1e5070 con 0x7f78bc15b2a0 2023-10-19T13:04:46.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.169+0000 7f78ba7fc640 1 -- 172.21.15.57:0/868120559 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78b4003b80 con 0x7f78bc15b2a0 2023-10-19T13:04:46.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.169+0000 7f78c3fd0640 1 -- 172.21.15.57:0/868120559 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78bc1e5630 con 0x7f78bc15b2a0 2023-10-19T13:04:46.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.170+0000 7f78c3fd0640 1 -- 172.21.15.57:0/868120559 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78bc15ab00 con 0x7f78bc15b2a0 2023-10-19T13:04:46.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.170+0000 7f78ba7fc640 1 -- 172.21.15.57:0/868120559 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 77) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f78b4003ce0 con 0x7f78bc15b2a0 2023-10-19T13:04:46.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.171+0000 7f78ba7fc640 1 --2- 172.21.15.57:0/868120559 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f78a806ac90 0x7f78a806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:46.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.171+0000 7f78ba7fc640 1 -- 172.21.15.57:0/868120559 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 6788+0+0 (secure 0 0 0) 0x7f78b4088a10 con 0x7f78bc15b2a0 2023-10-19T13:04:46.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.171+0000 7f78c1d6c640 1 --2- 172.21.15.57:0/868120559 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f78a806ac90 0x7f78a806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:46.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.172+0000 7f78c1d6c640 1 --2- 172.21.15.57:0/868120559 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f78a806ac90 0x7f78a806d150 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f78ac036000 tx=0x7f78ac004a40 comp rx=0 tx=0).ready entity=mgr.8419 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:46.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.175+0000 7f78ba7fc640 1 -- 172.21.15.57:0/868120559 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f78b40538f0 con 0x7f78bc15b2a0 2023-10-19T13:04:46.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.354+0000 7f78c3fd0640 1 -- 172.21.15.57:0/868120559 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f78bc1e5b50 con 0x7f78bc15b2a0 2023-10-19T13:04:46.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.355+0000 7f78ba7fc640 1 -- 172.21.15.57:0/868120559 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v146) v1 ==== 74+0+17921 (secure 0 0 0) 0x7f78bc1e5b50 con 0x7f78bc15b2a0 2023-10-19T13:04:46.358 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:04:46.359 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":146,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T13:04:44.277597+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2023-10-19T13:04:20.932329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"143","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2023-10-19T13:04:21.762157+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"144","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.179:0/2820959197":"2023-10-20T13:04:44.277539+0000","172.21.15.179:6833/3526956956":"2023-10-20T13:04:44.277539+0000","172.21.15.179:0/2214725838":"2023-10-20T13:04:44.277539+0000","172.21.15.179:6832/3526956956":"2023-10-20T13:04:44.277539+0000","172.21.15.179:0/2775321950":"2023-10-20T13:04:44.277539+0000","172.21.15.179:0/3591554987":"2023-10-20T13:04:44.277539+0000","172.21.15.179:0/1299954463":"2023-10-20T13:04:44.277539+0000","172.21.15.179:0/1024578486":"2023-10-20T13:04:44.277539+0000","172.21.15.179:0/1700025841":"2023-10-20T13:04:33.816783+0000","172.21.15.179:0/3160483813":"2023-10-20T13:04:33.816783+0000","172.21.15.179:0/4039784875":"2023-10-20T13:04:33.816783+0000","172.21.15.179:0/1023851911":"2023-10-20T13:04:33.816783+0000","172.21.15.179:0/2968654981":"2023-10-20T13:04:33.816783+0000","172.21.15.179:0/1247110426":"2023-10-20T13:04:44.277539+0000","172.21.15.179:0/3605212267":"2023-10-20T13:04:33.816783+0000","172.21.15.179:6833/2346394725":"2023-10-20T13:04:33.816783+0000","172.21.15.179:6832/2346394725":"2023-10-20T13:04:33.816783+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T13:04:46.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.357+0000 7f78c3fd0640 1 -- 172.21.15.57:0/868120559 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f78a806ac90 msgr2=0x7f78a806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:46.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.357+0000 7f78c3fd0640 1 --2- 172.21.15.57:0/868120559 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f78a806ac90 0x7f78a806d150 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f78ac036000 tx=0x7f78ac004a40 comp rx=0 tx=0).stop 2023-10-19T13:04:46.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.357+0000 7f78c3fd0640 1 -- 172.21.15.57:0/868120559 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f78bc15b2a0 msgr2=0x7f78bc1e5e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:46.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.357+0000 7f78c3fd0640 1 --2- 172.21.15.57:0/868120559 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f78bc15b2a0 0x7f78bc1e5e30 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7f78b400c3c0 tx=0x7f78b400c880 comp rx=0 tx=0).stop 2023-10-19T13:04:46.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.357+0000 7f78c3fd0640 1 -- 172.21.15.57:0/868120559 shutdown_connections 2023-10-19T13:04:46.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.357+0000 7f78c3fd0640 1 --2- 172.21.15.57:0/868120559 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f78a806ac90 0x7f78a806d150 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:46.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.357+0000 7f78c3fd0640 1 --2- 172.21.15.57:0/868120559 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f78bc15b2a0 0x7f78bc1e5e30 unknown :-1 s=CLOSED pgs=1687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:46.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.357+0000 7f78c3fd0640 1 --2- 172.21.15.57:0/868120559 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f78bc0cf340 0x7f78bc1f2c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:46.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.357+0000 7f78c3fd0640 1 --2- 172.21.15.57:0/868120559 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f78bc0ceb40 0x7f78bc1f2520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:46.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.357+0000 7f78c3fd0640 1 -- 172.21.15.57:0/868120559 >> 172.21.15.57:0/868120559 conn(0x7f78bc0d0d60 msgr2=0x7f78bc0b58f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:46.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.357+0000 7f78c3fd0640 1 -- 172.21.15.57:0/868120559 shutdown_connections 2023-10-19T13:04:46.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.357+0000 7f78c3fd0640 1 -- 172.21.15.57:0/868120559 wait complete. 2023-10-19T13:04:46.371 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-19T13:04:46.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.480+0000 7f1ee6244640 1 -- 172.21.15.57:0/148099162 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1ec80047c0 msgr2=0x7f1ec8004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:46.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.480+0000 7f1ee6244640 1 --2- 172.21.15.57:0/148099162 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1ec80047c0 0x7f1ec8004c40 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7f1ed4001800 tx=0x7f1ec80057f0 comp rx=0 tx=0).stop 2023-10-19T13:04:46.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.480+0000 7f1edffff640 1 -- 172.21.15.57:0/148099162 reap_dead start 2023-10-19T13:04:46.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.480+0000 7f1ee6244640 1 -- 172.21.15.57:0/148099162 shutdown_connections 2023-10-19T13:04:46.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.480+0000 7f1ee6244640 1 -- 172.21.15.57:0/148099162 >> 172.21.15.57:0/148099162 conn(0x7f1ee00aec40 msgr2=0x7f1ee00af050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:46.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.480+0000 7f1ee6244640 1 -- 172.21.15.57:0/148099162 shutdown_connections 2023-10-19T13:04:46.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.480+0000 7f1ee6244640 1 -- 172.21.15.57:0/148099162 wait complete. 2023-10-19T13:04:46.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.482+0000 7f1ee6244640 1 Processor -- start 2023-10-19T13:04:46.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.482+0000 7f1ee6244640 1 -- start start 2023-10-19T13:04:46.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.482+0000 7f1ee6244640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1ec80047c0 0x7f1ee0152f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:46.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.482+0000 7f1ee6244640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1ee0151010 0x7f1ee014e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:46.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.482+0000 7f1ee6244640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1ee0151490 0x7f1ee00ae020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:46.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.482+0000 7f1ee6244640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee0140230 con 0x7f1ec80047c0 2023-10-19T13:04:46.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.482+0000 7f1ee6244640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee0155830 con 0x7f1ee0151010 2023-10-19T13:04:46.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.482+0000 7f1ee6244640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee0155b10 con 0x7f1ee0151490 2023-10-19T13:04:46.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.482+0000 7f1edf7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1ec80047c0 0x7f1ee0152f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:46.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.482+0000 7f1edf7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1ec80047c0 0x7f1ee0152f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57578/0 (socket says 172.21.15.57:57578) 2023-10-19T13:04:46.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.482+0000 7f1edf7fe640 1 -- 172.21.15.57:0/2666740502 learned_addr learned my addr 172.21.15.57:0/2666740502 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:46.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.483+0000 7f1edeffd640 1 --2- 172.21.15.57:0/2666740502 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1ee0151010 0x7f1ee014e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:46.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.483+0000 7f1edffff640 1 --2- 172.21.15.57:0/2666740502 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1ee0151490 0x7f1ee00ae020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:46.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.483+0000 7f1edf7fe640 1 -- 172.21.15.57:0/2666740502 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1ee0151490 msgr2=0x7f1ee00ae020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:46.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.483+0000 7f1edf7fe640 1 --2- 172.21.15.57:0/2666740502 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1ee0151490 0x7f1ee00ae020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:46.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.483+0000 7f1edf7fe640 1 -- 172.21.15.57:0/2666740502 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1ee0151010 msgr2=0x7f1ee014e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:46.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.483+0000 7f1edf7fe640 1 --2- 172.21.15.57:0/2666740502 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1ee0151010 0x7f1ee014e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:46.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.483+0000 7f1edf7fe640 1 -- 172.21.15.57:0/2666740502 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ee0155df0 con 0x7f1ec80047c0 2023-10-19T13:04:46.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.483+0000 7f1edffff640 1 --2- 172.21.15.57:0/2666740502 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1ee0151490 0x7f1ee00ae020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:04:46.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.483+0000 7f1edf7fe640 1 --2- 172.21.15.57:0/2666740502 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1ec80047c0 0x7f1ee0152f20 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7f1ed0006600 tx=0x7f1ed00039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:46.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.483+0000 7f1ec7fff640 1 -- 172.21.15.57:0/2666740502 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ed0011900 con 0x7f1ec80047c0 2023-10-19T13:04:46.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.483+0000 7f1ec7fff640 1 -- 172.21.15.57:0/2666740502 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f1ed0011a60 con 0x7f1ec80047c0 2023-10-19T13:04:46.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.483+0000 7f1ec7fff640 1 -- 172.21.15.57:0/2666740502 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ed0011e70 con 0x7f1ec80047c0 2023-10-19T13:04:46.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.483+0000 7f1ee6244640 1 -- 172.21.15.57:0/2666740502 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ee01560d0 con 0x7f1ec80047c0 2023-10-19T13:04:46.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.483+0000 7f1ee6244640 1 -- 172.21.15.57:0/2666740502 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ee0153460 con 0x7f1ec80047c0 2023-10-19T13:04:46.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.484+0000 7f1ee6244640 1 -- 172.21.15.57:0/2666740502 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ee00690e0 con 0x7f1ec80047c0 2023-10-19T13:04:46.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.485+0000 7f1ec7fff640 1 -- 172.21.15.57:0/2666740502 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 77) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f1ed00103c0 con 0x7f1ec80047c0 2023-10-19T13:04:46.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.485+0000 7f1ec7fff640 1 --2- 172.21.15.57:0/2666740502 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f1ecc06ac90 0x7f1ecc06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:46.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.485+0000 7f1ec7fff640 1 -- 172.21.15.57:0/2666740502 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 6788+0+0 (secure 0 0 0) 0x7f1ed008a6f0 con 0x7f1ec80047c0 2023-10-19T13:04:46.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.485+0000 7f1edeffd640 1 --2- 172.21.15.57:0/2666740502 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f1ecc06ac90 0x7f1ecc06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:46.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.486+0000 7f1edeffd640 1 --2- 172.21.15.57:0/2666740502 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f1ecc06ac90 0x7f1ecc06d150 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1ed4034000 tx=0x7f1ed4039040 comp rx=0 tx=0).ready entity=mgr.8419 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:46.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.489+0000 7f1ec7fff640 1 -- 172.21.15.57:0/2666740502 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f1ed0055750 con 0x7f1ec80047c0 2023-10-19T13:04:46.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.702+0000 7f1ee6244640 1 -- 172.21.15.57:0/2666740502 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f1ee00ac510 con 0x7f1ec80047c0 2023-10-19T13:04:46.705 INFO:tasks.ceph.mon.a.smithi057.stderr:2023-10-19T13:04:46.703+0000 7f1253ff3640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-19T13:04:46.707 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.704+0000 7f1ec7fff640 1 -- 172.21.15.57:0/2666740502 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v57) v1 ==== 161+0+0 (secure 0 0 0) 0x7f1ed0015020 con 0x7f1ec80047c0 2023-10-19T13:04:46.707 INFO:teuthology.orchestra.run.smithi057.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-19T13:04:46.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.707+0000 7f1ee6244640 1 -- 172.21.15.57:0/2666740502 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f1ecc06ac90 msgr2=0x7f1ecc06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:46.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.708+0000 7f1ee6244640 1 --2- 172.21.15.57:0/2666740502 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f1ecc06ac90 0x7f1ecc06d150 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1ed4034000 tx=0x7f1ed4039040 comp rx=0 tx=0).stop 2023-10-19T13:04:46.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.708+0000 7f1ee6244640 1 -- 172.21.15.57:0/2666740502 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1ec80047c0 msgr2=0x7f1ee0152f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:46.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.708+0000 7f1ee6244640 1 --2- 172.21.15.57:0/2666740502 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1ec80047c0 0x7f1ee0152f20 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7f1ed0006600 tx=0x7f1ed00039c0 comp rx=0 tx=0).stop 2023-10-19T13:04:46.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.708+0000 7f1ee6244640 1 -- 172.21.15.57:0/2666740502 shutdown_connections 2023-10-19T13:04:46.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.708+0000 7f1ee6244640 1 --2- 172.21.15.57:0/2666740502 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f1ecc06ac90 0x7f1ecc06d150 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:46.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.708+0000 7f1ee6244640 1 --2- 172.21.15.57:0/2666740502 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1ee0151490 0x7f1ee00ae020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:46.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.708+0000 7f1ee6244640 1 --2- 172.21.15.57:0/2666740502 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1ee0151010 0x7f1ee014e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:46.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.708+0000 7f1ee6244640 1 --2- 172.21.15.57:0/2666740502 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1ec80047c0 0x7f1ee0152f20 unknown :-1 s=CLOSED pgs=1690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:46.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.708+0000 7f1ee6244640 1 -- 172.21.15.57:0/2666740502 >> 172.21.15.57:0/2666740502 conn(0x7f1ee00aec40 msgr2=0x7f1ee014b910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:46.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.708+0000 7f1ee6244640 1 -- 172.21.15.57:0/2666740502 shutdown_connections 2023-10-19T13:04:46.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.708+0000 7f1ee6244640 1 -- 172.21.15.57:0/2666740502 wait complete. 2023-10-19T13:04:46.715 INFO:tasks.ceph.mds.c.smithi057.stderr: -8581> 2023-10-19T13:04:28.391+0000 7f02b96dc640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-19T13:04:46.725 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-19T13:04:46.750 INFO:tasks.ceph.mds.c.smithi057.stdout:starting mds.c at 2023-10-19T13:04:46.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.835+0000 7f9e1362b640 1 -- 172.21.15.57:0/775579700 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9e0c15c2e0 msgr2=0x7f9e0c0cf5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:46.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.835+0000 7f9e1362b640 1 --2- 172.21.15.57:0/775579700 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9e0c15c2e0 0x7f9e0c0cf5e0 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7f9e0c066730 tx=0x7f9e0802bdb0 comp rx=0 tx=0).stop 2023-10-19T13:04:46.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.835+0000 7f9e1362b640 1 -- 172.21.15.57:0/775579700 shutdown_connections 2023-10-19T13:04:46.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.835+0000 7f9e1362b640 1 --2- 172.21.15.57:0/775579700 >> v2:172.21.15.179:3300/0 conn(0x7f9e0c15c6c0 0x7f9e0c0cb4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:46.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.835+0000 7f9e1362b640 1 --2- 172.21.15.57:0/775579700 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9e0c15c2e0 0x7f9e0c0cf5e0 unknown :-1 s=CLOSED pgs=1693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:46.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.835+0000 7f9e1362b640 1 -- 172.21.15.57:0/775579700 >> 172.21.15.57:0/775579700 conn(0x7f9e0c0ceac0 msgr2=0x7f9e0c0ceed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:46.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.835+0000 7f9e1362b640 1 -- 172.21.15.57:0/775579700 shutdown_connections 2023-10-19T13:04:46.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.835+0000 7f9e1362b640 1 -- 172.21.15.57:0/775579700 wait complete. 2023-10-19T13:04:46.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.836+0000 7f9e1362b640 1 Processor -- start 2023-10-19T13:04:46.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.836+0000 7f9e1362b640 1 -- start start 2023-10-19T13:04:46.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.837+0000 7f9e1362b640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9e0c0afbf0 0x7f9e0c14fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:46.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.837+0000 7f9e1362b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9e0c15c2e0 0x7f9e0c0c6fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:46.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.837+0000 7f9e1362b640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9e0c15c6c0 0x7f9e0c0c7b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:46.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.837+0000 7f9e1362b640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e0c0c6b20 con 0x7f9e0c15c2e0 2023-10-19T13:04:46.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.837+0000 7f9e1362b640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e0c0c5f10 con 0x7f9e0c0afbf0 2023-10-19T13:04:46.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.837+0000 7f9e1362b640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e0c0c61f0 con 0x7f9e0c15c6c0 2023-10-19T13:04:46.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.837+0000 7f9e10bc6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9e0c15c2e0 0x7f9e0c0c6fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:46.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.837+0000 7f9e10bc6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9e0c15c2e0 0x7f9e0c0c6fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38350/0 (socket says 172.21.15.57:38350) 2023-10-19T13:04:46.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.837+0000 7f9e10bc6640 1 -- 172.21.15.57:0/507143419 learned_addr learned my addr 172.21.15.57:0/507143419 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:46.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.837+0000 7f9e113c7640 1 --2- 172.21.15.57:0/507143419 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9e0c0afbf0 0x7f9e0c14fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:46.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.837+0000 7f9e11bc8640 1 --2- 172.21.15.57:0/507143419 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9e0c15c6c0 0x7f9e0c0c7b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:46.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.837+0000 7f9e10bc6640 1 -- 172.21.15.57:0/507143419 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9e0c15c6c0 msgr2=0x7f9e0c0c7b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:46.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.837+0000 7f9e10bc6640 1 --2- 172.21.15.57:0/507143419 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9e0c15c6c0 0x7f9e0c0c7b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:46.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.837+0000 7f9e10bc6640 1 -- 172.21.15.57:0/507143419 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9e0c0afbf0 msgr2=0x7f9e0c14fee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:46.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.837+0000 7f9e10bc6640 1 --2- 172.21.15.57:0/507143419 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9e0c0afbf0 0x7f9e0c14fee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:46.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.837+0000 7f9e10bc6640 1 -- 172.21.15.57:0/507143419 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e08009560 con 0x7f9e0c15c2e0 2023-10-19T13:04:46.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.837+0000 7f9e10bc6640 1 --2- 172.21.15.57:0/507143419 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9e0c15c2e0 0x7f9e0c0c6fb0 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7f9dfc00c400 tx=0x7f9dfc00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:46.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.837+0000 7f9e01ffb640 1 -- 172.21.15.57:0/507143419 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dfc011790 con 0x7f9e0c15c2e0 2023-10-19T13:04:46.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.837+0000 7f9e01ffb640 1 -- 172.21.15.57:0/507143419 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f9dfc011a20 con 0x7f9e0c15c2e0 2023-10-19T13:04:46.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.837+0000 7f9e1362b640 1 -- 172.21.15.57:0/507143419 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e0c0c64d0 con 0x7f9e0c15c2e0 2023-10-19T13:04:46.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.837+0000 7f9e01ffb640 1 -- 172.21.15.57:0/507143419 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dfc011e30 con 0x7f9e0c15c2e0 2023-10-19T13:04:46.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.838+0000 7f9e1362b640 1 -- 172.21.15.57:0/507143419 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e0c150650 con 0x7f9e0c15c2e0 2023-10-19T13:04:46.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.838+0000 7f9e1362b640 1 -- 172.21.15.57:0/507143419 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e0c04dae0 con 0x7f9e0c15c2e0 2023-10-19T13:04:46.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.839+0000 7f9e01ffb640 1 -- 172.21.15.57:0/507143419 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 77) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f9dfc01c460 con 0x7f9e0c15c2e0 2023-10-19T13:04:46.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.839+0000 7f9e01ffb640 1 --2- 172.21.15.57:0/507143419 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f9df006ac90 0x7f9df006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:46.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.839+0000 7f9e01ffb640 1 -- 172.21.15.57:0/507143419 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f9dfc089850 con 0x7f9e0c15c2e0 2023-10-19T13:04:46.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.840+0000 7f9e113c7640 1 --2- 172.21.15.57:0/507143419 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f9df006ac90 0x7f9df006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:46.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.840+0000 7f9e113c7640 1 --2- 172.21.15.57:0/507143419 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f9df006ac90 0x7f9df006d150 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f9e08035000 tx=0x7f9e08004730 comp rx=0 tx=0).ready entity=mgr.8419 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:46.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:46.844+0000 7f9e01ffb640 1 -- 172.21.15.57:0/507143419 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9dfc054790 con 0x7f9e0c15c2e0 2023-10-19T13:04:47.062 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:47.059+0000 7f9e1362b640 1 -- 172.21.15.57:0/507143419 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f9e0c153cf0 con 0x7f9e0c15c2e0 2023-10-19T13:04:47.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:47.300+0000 7f9e01ffb640 1 -- 172.21.15.57:0/507143419 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mgrmap(e 78) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f9dfc054b60 con 0x7f9e0c15c2e0 2023-10-19T13:04:47.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:47.706+0000 7f9e01ffb640 1 -- 172.21.15.57:0/507143419 <== mon.0 v2:172.21.15.57:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v58) v1 ==== 104+0+0 (secure 0 0 0) 0x7f9dfc0719a0 con 0x7f9e0c15c2e0 2023-10-19T13:04:47.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:47.709+0000 7f9e1362b640 1 -- 172.21.15.57:0/507143419 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f9df006ac90 msgr2=0x7f9df006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:47.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:47.709+0000 7f9e1362b640 1 --2- 172.21.15.57:0/507143419 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f9df006ac90 0x7f9df006d150 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f9e08035000 tx=0x7f9e08004730 comp rx=0 tx=0).stop 2023-10-19T13:04:47.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:47.709+0000 7f9e1362b640 1 -- 172.21.15.57:0/507143419 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9e0c15c2e0 msgr2=0x7f9e0c0c6fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:47.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:47.709+0000 7f9e1362b640 1 --2- 172.21.15.57:0/507143419 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9e0c15c2e0 0x7f9e0c0c6fb0 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7f9dfc00c400 tx=0x7f9dfc00c8c0 comp rx=0 tx=0).stop 2023-10-19T13:04:47.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:47.709+0000 7f9e1362b640 1 -- 172.21.15.57:0/507143419 shutdown_connections 2023-10-19T13:04:47.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:47.709+0000 7f9e1362b640 1 --2- 172.21.15.57:0/507143419 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f9df006ac90 0x7f9df006d150 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:47.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:47.709+0000 7f9e1362b640 1 --2- 172.21.15.57:0/507143419 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9e0c15c6c0 0x7f9e0c0c7b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:47.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:47.709+0000 7f9e1362b640 1 --2- 172.21.15.57:0/507143419 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9e0c15c2e0 0x7f9e0c0c6fb0 unknown :-1 s=CLOSED pgs=1694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:47.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:47.709+0000 7f9e1362b640 1 --2- 172.21.15.57:0/507143419 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9e0c0afbf0 0x7f9e0c14fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:47.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:47.709+0000 7f9e1362b640 1 -- 172.21.15.57:0/507143419 >> 172.21.15.57:0/507143419 conn(0x7f9e0c0ceac0 msgr2=0x7f9e0c0cc670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:47.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:47.709+0000 7f9e1362b640 1 -- 172.21.15.57:0/507143419 shutdown_connections 2023-10-19T13:04:47.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:47.709+0000 7f9e1362b640 1 -- 172.21.15.57:0/507143419 wait complete. 2023-10-19T13:04:47.726 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-19T13:04:47.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:47.884+0000 7fbb24383640 1 -- 172.21.15.57:0/1605091538 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbb08004d00 msgr2=0x7fbb08005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:47.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:47.884+0000 7fbb24383640 1 --2- 172.21.15.57:0/1605091538 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbb08004d00 0x7fbb08005180 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7fbb1400c4a0 tx=0x7fbb080057f0 comp rx=0 tx=0).stop 2023-10-19T13:04:47.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:47.884+0000 7fbb22920640 1 -- 172.21.15.57:0/1605091538 reap_dead start 2023-10-19T13:04:47.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:47.884+0000 7fbb24383640 1 -- 172.21.15.57:0/1605091538 shutdown_connections 2023-10-19T13:04:47.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:47.884+0000 7fbb24383640 1 -- 172.21.15.57:0/1605091538 >> 172.21.15.57:0/1605091538 conn(0x7fbb1c069930 msgr2=0x7fbb1c069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:47.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:47.884+0000 7fbb24383640 1 -- 172.21.15.57:0/1605091538 shutdown_connections 2023-10-19T13:04:47.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:47.884+0000 7fbb24383640 1 -- 172.21.15.57:0/1605091538 wait complete. 2023-10-19T13:04:47.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:47.885+0000 7fbb24383640 1 Processor -- start 2023-10-19T13:04:47.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:47.886+0000 7fbb24383640 1 -- start start 2023-10-19T13:04:47.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:47.886+0000 7fbb24383640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbb08004d00 0x7fbb1c13ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:47.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:47.886+0000 7fbb24383640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbb1c158530 0x7fbb1c156160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:47.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:47.886+0000 7fbb24383640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbb1c1566a0 0x7fbb1c153e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:47.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:47.886+0000 7fbb24383640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb1c1589b0 con 0x7fbb08004d00 2023-10-19T13:04:47.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:47.886+0000 7fbb24383640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb1c0ad9f0 con 0x7fbb1c1566a0 2023-10-19T13:04:47.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:47.886+0000 7fbb24383640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb1c0adcd0 con 0x7fbb1c158530 2023-10-19T13:04:47.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:47.886+0000 7fbb2211f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbb08004d00 0x7fbb1c13ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:47.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:47.886+0000 7fbb2211f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbb08004d00 0x7fbb1c13ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38360/0 (socket says 172.21.15.57:38360) 2023-10-19T13:04:47.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:47.886+0000 7fbb2211f640 1 -- 172.21.15.57:0/2554173007 learned_addr learned my addr 172.21.15.57:0/2554173007 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:47.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:47.887+0000 7fbb2191e640 1 --2- 172.21.15.57:0/2554173007 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbb1c158530 0x7fbb1c156160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:47.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:47.887+0000 7fbb22920640 1 --2- 172.21.15.57:0/2554173007 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbb1c1566a0 0x7fbb1c153e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:47.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:47.887+0000 7fbb2211f640 1 -- 172.21.15.57:0/2554173007 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbb1c158530 msgr2=0x7fbb1c156160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:47.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:47.887+0000 7fbb2211f640 1 --2- 172.21.15.57:0/2554173007 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbb1c158530 0x7fbb1c156160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:47.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:47.887+0000 7fbb2211f640 1 -- 172.21.15.57:0/2554173007 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbb1c1566a0 msgr2=0x7fbb1c153e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:47.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:47.887+0000 7fbb2211f640 1 --2- 172.21.15.57:0/2554173007 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbb1c1566a0 0x7fbb1c153e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:47.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:47.887+0000 7fbb2211f640 1 -- 172.21.15.57:0/2554173007 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb1c070300 con 0x7fbb08004d00 2023-10-19T13:04:47.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:47.887+0000 7fbb2211f640 1 --2- 172.21.15.57:0/2554173007 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbb08004d00 0x7fbb1c13ca70 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7fbb18003e40 tx=0x7fbb18001770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:47.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:47.887+0000 7fbb12ffd640 1 -- 172.21.15.57:0/2554173007 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb18011840 con 0x7fbb08004d00 2023-10-19T13:04:47.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:47.887+0000 7fbb12ffd640 1 -- 172.21.15.57:0/2554173007 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fbb180119a0 con 0x7fbb08004d00 2023-10-19T13:04:47.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:47.887+0000 7fbb24383640 1 -- 172.21.15.57:0/2554173007 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb1c0adfb0 con 0x7fbb08004d00 2023-10-19T13:04:47.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:47.887+0000 7fbb12ffd640 1 -- 172.21.15.57:0/2554173007 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb18011db0 con 0x7fbb08004d00 2023-10-19T13:04:47.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:47.887+0000 7fbb24383640 1 -- 172.21.15.57:0/2554173007 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb1c0ad6f0 con 0x7fbb08004d00 2023-10-19T13:04:47.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:47.888+0000 7fbb24383640 1 -- 172.21.15.57:0/2554173007 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb1c0ae4d0 con 0x7fbb08004d00 2023-10-19T13:04:47.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:47.888+0000 7fbb12ffd640 1 -- 172.21.15.57:0/2554173007 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 78) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fbb180103c0 con 0x7fbb08004d00 2023-10-19T13:04:47.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:47.889+0000 7fbb12ffd640 1 --2- 172.21.15.57:0/2554173007 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fbb0006af00 0x7fbb0006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:47.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:47.889+0000 7fbb12ffd640 1 -- 172.21.15.57:0/2554173007 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6874+0+0 (secure 0 0 0) 0x7fbb1808ac70 con 0x7fbb08004d00 2023-10-19T13:04:47.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:47.889+0000 7fbb2191e640 1 --2- 172.21.15.57:0/2554173007 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fbb0006af00 0x7fbb0006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:47.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:47.890+0000 7fbb2191e640 1 --2- 172.21.15.57:0/2554173007 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fbb0006af00 0x7fbb0006d3c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fbb1c1f47e0 tx=0x7fbb0c005f90 comp rx=0 tx=0).ready entity=mgr.8419 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:47.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:47.893+0000 7fbb12ffd640 1 -- 172.21.15.57:0/2554173007 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fbb1c0ae4d0 con 0x7fbb08004d00 2023-10-19T13:04:48.072 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:48.069+0000 7fbb24383640 1 -- 172.21.15.57:0/2554173007 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fbb1c0690e0 con 0x7fbb08004d00 2023-10-19T13:04:48.405 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:04:48.408 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:04:48.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:48.711+0000 7fbb12ffd640 1 -- 172.21.15.57:0/2554173007 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v148) v1 ==== 181+0+0 (secure 0 0 0) 0x7fbb18015020 con 0x7fbb08004d00 2023-10-19T13:04:48.715 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'cephfs_metadata' removed 2023-10-19T13:04:48.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:48.715+0000 7fbb24383640 1 -- 172.21.15.57:0/2554173007 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fbb0006af00 msgr2=0x7fbb0006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:48.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:48.715+0000 7fbb24383640 1 --2- 172.21.15.57:0/2554173007 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fbb0006af00 0x7fbb0006d3c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fbb1c1f47e0 tx=0x7fbb0c005f90 comp rx=0 tx=0).stop 2023-10-19T13:04:48.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:48.715+0000 7fbb24383640 1 -- 172.21.15.57:0/2554173007 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbb08004d00 msgr2=0x7fbb1c13ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:48.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:48.715+0000 7fbb24383640 1 --2- 172.21.15.57:0/2554173007 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbb08004d00 0x7fbb1c13ca70 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7fbb18003e40 tx=0x7fbb18001770 comp rx=0 tx=0).stop 2023-10-19T13:04:48.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:48.715+0000 7fbb24383640 1 -- 172.21.15.57:0/2554173007 shutdown_connections 2023-10-19T13:04:48.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:48.715+0000 7fbb24383640 1 --2- 172.21.15.57:0/2554173007 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fbb0006af00 0x7fbb0006d3c0 secure :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fbb1c1f47e0 tx=0x7fbb0c005f90 comp rx=0 tx=0).stop 2023-10-19T13:04:48.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:48.715+0000 7fbb24383640 1 --2- 172.21.15.57:0/2554173007 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbb1c1566a0 0x7fbb1c153e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:48.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:48.715+0000 7fbb24383640 1 --2- 172.21.15.57:0/2554173007 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbb1c158530 0x7fbb1c156160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:48.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:48.715+0000 7fbb24383640 1 --2- 172.21.15.57:0/2554173007 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbb08004d00 0x7fbb1c13ca70 unknown :-1 s=CLOSED pgs=1697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:48.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:48.715+0000 7fbb24383640 1 -- 172.21.15.57:0/2554173007 >> 172.21.15.57:0/2554173007 conn(0x7fbb1c069930 msgr2=0x7fbb1c06f310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:48.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:48.716+0000 7fbb24383640 1 -- 172.21.15.57:0/2554173007 shutdown_connections 2023-10-19T13:04:48.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:48.716+0000 7fbb24383640 1 -- 172.21.15.57:0/2554173007 wait complete. 2023-10-19T13:04:48.733 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-19T13:04:48.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:48.848+0000 7fbfa8fb6640 1 -- 172.21.15.57:0/2292283434 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbf90003970 msgr2=0x7fbf90003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:48.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:48.848+0000 7fbfa8fb6640 1 --2- 172.21.15.57:0/2292283434 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbf90003970 0x7fbf90003d90 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7fbf94009b50 tx=0x7fbf900057f0 comp rx=0 tx=0).stop 2023-10-19T13:04:48.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:48.848+0000 7fbfa2d9d640 1 -- 172.21.15.57:0/2292283434 reap_dead start 2023-10-19T13:04:48.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:48.848+0000 7fbfa8fb6640 1 -- 172.21.15.57:0/2292283434 shutdown_connections 2023-10-19T13:04:48.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:48.848+0000 7fbfa8fb6640 1 -- 172.21.15.57:0/2292283434 >> 172.21.15.57:0/2292283434 conn(0x7fbfa4069930 msgr2=0x7fbfa4069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:48.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:48.848+0000 7fbfa8fb6640 1 -- 172.21.15.57:0/2292283434 shutdown_connections 2023-10-19T13:04:48.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:48.848+0000 7fbfa8fb6640 1 -- 172.21.15.57:0/2292283434 wait complete. 2023-10-19T13:04:48.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:48.850+0000 7fbfa8fb6640 1 Processor -- start 2023-10-19T13:04:48.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:48.850+0000 7fbfa8fb6640 1 -- start start 2023-10-19T13:04:48.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:48.850+0000 7fbfa8fb6640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbf90003970 0x7fbfa4152750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:48.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:48.850+0000 7fbfa8fb6640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbfa4140720 0x7fbfa41eedd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:48.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:48.850+0000 7fbfa8fb6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbfa40702c0 0x7fbfa4070770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:48.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:48.850+0000 7fbfa8fb6640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfa4070cb0 con 0x7fbfa40702c0 2023-10-19T13:04:48.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:48.850+0000 7fbfa8fb6640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfa4152c90 con 0x7fbfa4140720 2023-10-19T13:04:48.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:48.850+0000 7fbfa8fb6640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfa4141150 con 0x7fbf90003970 2023-10-19T13:04:48.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:48.850+0000 7fbfa2d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbfa40702c0 0x7fbfa4070770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:48.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:48.850+0000 7fbfa2d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbfa40702c0 0x7fbfa4070770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38370/0 (socket says 172.21.15.57:38370) 2023-10-19T13:04:48.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:48.850+0000 7fbfa2d9d640 1 -- 172.21.15.57:0/2526031867 learned_addr learned my addr 172.21.15.57:0/2526031867 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:48.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:48.851+0000 7fbfa259c640 1 --2- 172.21.15.57:0/2526031867 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbf90003970 0x7fbfa4152750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:48.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:48.851+0000 7fbf99d9b640 1 --2- 172.21.15.57:0/2526031867 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbfa4140720 0x7fbfa41eedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:48.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:48.851+0000 7fbfa2d9d640 1 -- 172.21.15.57:0/2526031867 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbf90003970 msgr2=0x7fbfa4152750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:48.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:48.851+0000 7fbfa2d9d640 1 --2- 172.21.15.57:0/2526031867 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbf90003970 0x7fbfa4152750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:48.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:48.851+0000 7fbfa2d9d640 1 -- 172.21.15.57:0/2526031867 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbfa4140720 msgr2=0x7fbfa41eedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:48.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:48.851+0000 7fbfa2d9d640 1 --2- 172.21.15.57:0/2526031867 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbfa4140720 0x7fbfa41eedd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:48.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:48.851+0000 7fbfa2d9d640 1 -- 172.21.15.57:0/2526031867 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfa4153740 con 0x7fbfa40702c0 2023-10-19T13:04:48.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:48.851+0000 7fbfa2d9d640 1 --2- 172.21.15.57:0/2526031867 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbfa40702c0 0x7fbfa4070770 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7fbf9c000f80 tx=0x7fbf9c00ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:48.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:48.851+0000 7fbf9b7fe640 1 -- 172.21.15.57:0/2526031867 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf9c00c940 con 0x7fbfa40702c0 2023-10-19T13:04:48.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:48.851+0000 7fbf9b7fe640 1 -- 172.21.15.57:0/2526031867 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fbf9c00cbd0 con 0x7fbfa40702c0 2023-10-19T13:04:48.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:48.851+0000 7fbfa8fb6640 1 -- 172.21.15.57:0/2526031867 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfa4152ea0 con 0x7fbfa40702c0 2023-10-19T13:04:48.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:48.851+0000 7fbf9b7fe640 1 -- 172.21.15.57:0/2526031867 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf9c011460 con 0x7fbfa40702c0 2023-10-19T13:04:48.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:48.851+0000 7fbfa8fb6640 1 -- 172.21.15.57:0/2526031867 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfa4153460 con 0x7fbfa40702c0 2023-10-19T13:04:48.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:48.852+0000 7fbfa8fb6640 1 -- 172.21.15.57:0/2526031867 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfa406fd70 con 0x7fbfa40702c0 2023-10-19T13:04:48.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:48.853+0000 7fbf9b7fe640 1 -- 172.21.15.57:0/2526031867 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 78) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fbf9c0115c0 con 0x7fbfa40702c0 2023-10-19T13:04:48.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:48.853+0000 7fbf9b7fe640 1 --2- 172.21.15.57:0/2526031867 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fbf8406af00 0x7fbf8406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:48.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:48.853+0000 7fbf9b7fe640 1 -- 172.21.15.57:0/2526031867 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6436+0+0 (secure 0 0 0) 0x7fbf9c08a140 con 0x7fbfa40702c0 2023-10-19T13:04:48.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:48.853+0000 7fbfa259c640 1 --2- 172.21.15.57:0/2526031867 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fbf8406af00 0x7fbf8406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:48.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:48.854+0000 7fbfa259c640 1 --2- 172.21.15.57:0/2526031867 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fbf8406af00 0x7fbf8406d3c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fbf94036000 tx=0x7fbf940046c0 comp rx=0 tx=0).ready entity=mgr.8419 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:48.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:48.857+0000 7fbf9b7fe640 1 -- 172.21.15.57:0/2526031867 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fbf9c054d10 con 0x7fbfa40702c0 2023-10-19T13:04:49.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:49.035+0000 7fbfa8fb6640 1 -- 172.21.15.57:0/2526031867 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fbfa40690e0 con 0x7fbfa40702c0 2023-10-19T13:04:49.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:49.301+0000 7fbf9b7fe640 1 -- 172.21.15.57:0/2526031867 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mgrmap(e 79) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fbf9c0550e0 con 0x7fbfa40702c0 2023-10-19T13:04:49.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:49.725+0000 7fbf9b7fe640 1 -- 172.21.15.57:0/2526031867 <== mon.0 v2:172.21.15.57:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v149) v1 ==== 169+0+0 (secure 0 0 0) 0x7fbf9c073d40 con 0x7fbfa40702c0 2023-10-19T13:04:49.728 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'cephfs_data' removed 2023-10-19T13:04:49.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:49.727+0000 7fbfa8fb6640 1 -- 172.21.15.57:0/2526031867 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fbf8406af00 msgr2=0x7fbf8406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:49.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:49.727+0000 7fbfa8fb6640 1 --2- 172.21.15.57:0/2526031867 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fbf8406af00 0x7fbf8406d3c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fbf94036000 tx=0x7fbf940046c0 comp rx=0 tx=0).stop 2023-10-19T13:04:49.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:49.727+0000 7fbfa8fb6640 1 -- 172.21.15.57:0/2526031867 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbfa40702c0 msgr2=0x7fbfa4070770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:49.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:49.727+0000 7fbfa8fb6640 1 --2- 172.21.15.57:0/2526031867 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbfa40702c0 0x7fbfa4070770 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7fbf9c000f80 tx=0x7fbf9c00ed10 comp rx=0 tx=0).stop 2023-10-19T13:04:49.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:49.727+0000 7fbfa8fb6640 1 -- 172.21.15.57:0/2526031867 shutdown_connections 2023-10-19T13:04:49.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:49.727+0000 7fbfa8fb6640 1 --2- 172.21.15.57:0/2526031867 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fbf8406af00 0x7fbf8406d3c0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:49.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:49.727+0000 7fbfa8fb6640 1 --2- 172.21.15.57:0/2526031867 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbfa40702c0 0x7fbfa4070770 unknown :-1 s=CLOSED pgs=1700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:49.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:49.727+0000 7fbfa8fb6640 1 --2- 172.21.15.57:0/2526031867 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbfa4140720 0x7fbfa41eedd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:49.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:49.727+0000 7fbfa8fb6640 1 --2- 172.21.15.57:0/2526031867 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbf90003970 0x7fbfa4152750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:49.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:49.727+0000 7fbfa8fb6640 1 -- 172.21.15.57:0/2526031867 >> 172.21.15.57:0/2526031867 conn(0x7fbfa4069930 msgr2=0x7fbfa413b660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:49.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:49.727+0000 7fbfa8fb6640 1 -- 172.21.15.57:0/2526031867 shutdown_connections 2023-10-19T13:04:49.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:49.727+0000 7fbfa8fb6640 1 -- 172.21.15.57:0/2526031867 wait complete. 2023-10-19T13:04:49.746 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 30 2023-10-19T13:04:49.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:49.860+0000 7f69f770f640 1 -- 172.21.15.57:0/4103757419 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f69f0158cf0 msgr2=0x7f69f015c0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:49.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:49.860+0000 7f69f770f640 1 --2- 172.21.15.57:0/4103757419 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f69f0158cf0 0x7f69f015c0e0 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7f69e0005fd0 tx=0x7f69e002bed0 comp rx=0 tx=0).stop 2023-10-19T13:04:49.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:49.860+0000 7f69f770f640 1 -- 172.21.15.57:0/4103757419 shutdown_connections 2023-10-19T13:04:49.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:49.861+0000 7f69f770f640 1 --2- 172.21.15.57:0/4103757419 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f69f0158cf0 0x7f69f015c0e0 unknown :-1 s=CLOSED pgs=1702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:49.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:49.861+0000 7f69f770f640 1 -- 172.21.15.57:0/4103757419 >> 172.21.15.57:0/4103757419 conn(0x7f69f00af7a0 msgr2=0x7f69f00afb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:49.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:49.861+0000 7f69f770f640 1 -- 172.21.15.57:0/4103757419 shutdown_connections 2023-10-19T13:04:49.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:49.861+0000 7f69f770f640 1 -- 172.21.15.57:0/4103757419 wait complete. 2023-10-19T13:04:49.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:49.862+0000 7f69f770f640 1 Processor -- start 2023-10-19T13:04:49.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:49.862+0000 7f69f770f640 1 -- start start 2023-10-19T13:04:49.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:49.862+0000 7f69f770f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f69f00cd9e0 0x7f69f00b0890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:49.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:49.862+0000 7f69f770f640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f69f0158910 0x7f69f00b24e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:49.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:49.862+0000 7f69f770f640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f69f0158cf0 0x7f69f00ad5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:49.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:49.862+0000 7f69f770f640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69f0154a40 con 0x7f69f00cd9e0 2023-10-19T13:04:49.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:49.862+0000 7f69f770f640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69f0154710 con 0x7f69f0158910 2023-10-19T13:04:49.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:49.862+0000 7f69f770f640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69f00ade00 con 0x7f69f0158cf0 2023-10-19T13:04:49.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:49.863+0000 7f69f54ab640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f69f00cd9e0 0x7f69f00b0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:49.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:49.863+0000 7f69f54ab640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f69f00cd9e0 0x7f69f00b0890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38388/0 (socket says 172.21.15.57:38388) 2023-10-19T13:04:49.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:49.863+0000 7f69f54ab640 1 -- 172.21.15.57:0/987140553 learned_addr learned my addr 172.21.15.57:0/987140553 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:49.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:49.863+0000 7f69f4caa640 1 --2- 172.21.15.57:0/987140553 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f69f0158910 0x7f69f00b24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:49.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:49.863+0000 7f69f5cac640 1 --2- 172.21.15.57:0/987140553 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f69f0158cf0 0x7f69f00ad5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:49.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:49.863+0000 7f69f54ab640 1 -- 172.21.15.57:0/987140553 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f69f0158cf0 msgr2=0x7f69f00ad5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:49.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:49.863+0000 7f69f54ab640 1 --2- 172.21.15.57:0/987140553 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f69f0158cf0 0x7f69f00ad5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:49.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:49.863+0000 7f69f54ab640 1 -- 172.21.15.57:0/987140553 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f69f0158910 msgr2=0x7f69f00b24e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:49.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:49.863+0000 7f69f54ab640 1 --2- 172.21.15.57:0/987140553 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f69f0158910 0x7f69f00b24e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:49.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:49.863+0000 7f69f54ab640 1 -- 172.21.15.57:0/987140553 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69e0009560 con 0x7f69f00cd9e0 2023-10-19T13:04:49.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:49.863+0000 7f69f5cac640 1 --2- 172.21.15.57:0/987140553 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f69f0158cf0 0x7f69f00ad5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T13:04:49.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:49.863+0000 7f69f54ab640 1 --2- 172.21.15.57:0/987140553 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f69f00cd9e0 0x7f69f00b0890 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7f69ec00c7e0 tx=0x7f69ec00cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:49.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:49.863+0000 7f69e5ffb640 1 -- 172.21.15.57:0/987140553 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69ec003da0 con 0x7f69f00cd9e0 2023-10-19T13:04:49.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:49.863+0000 7f69e5ffb640 1 -- 172.21.15.57:0/987140553 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f69ec005070 con 0x7f69f00cd9e0 2023-10-19T13:04:49.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:49.863+0000 7f69f770f640 1 -- 172.21.15.57:0/987140553 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69f00ae0e0 con 0x7f69f00cd9e0 2023-10-19T13:04:49.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:49.863+0000 7f69e5ffb640 1 -- 172.21.15.57:0/987140553 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69ec005480 con 0x7f69f00cd9e0 2023-10-19T13:04:49.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:49.863+0000 7f69f770f640 1 -- 172.21.15.57:0/987140553 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69f00adaf0 con 0x7f69f00cd9e0 2023-10-19T13:04:49.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:49.865+0000 7f69f770f640 1 -- 172.21.15.57:0/987140553 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69f00ae600 con 0x7f69f00cd9e0 2023-10-19T13:04:49.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:49.865+0000 7f69e5ffb640 1 -- 172.21.15.57:0/987140553 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 79) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f69ec0055e0 con 0x7f69f00cd9e0 2023-10-19T13:04:49.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:49.865+0000 7f69e5ffb640 1 --2- 172.21.15.57:0/987140553 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f69d406af60 0x7f69d406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:49.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:49.866+0000 7f69e5ffb640 1 -- 172.21.15.57:0/987140553 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6022+0+0 (secure 0 0 0) 0x7f69ec08baa0 con 0x7f69f00cd9e0 2023-10-19T13:04:49.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:49.866+0000 7f69f4caa640 1 --2- 172.21.15.57:0/987140553 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f69d406af60 0x7f69d406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:49.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:49.866+0000 7f69f4caa640 1 --2- 172.21.15.57:0/987140553 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f69d406af60 0x7f69d406d420 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f69e0035000 tx=0x7f69e002e040 comp rx=0 tx=0).ready entity=mgr.8419 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:49.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:49.870+0000 7f69e5ffb640 1 -- 172.21.15.57:0/987140553 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f69f00ae600 con 0x7f69f00cd9e0 2023-10-19T13:04:50.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.045+0000 7f69f770f640 1 -- 172.21.15.57:0/987140553 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "config reset", "num": 30} v 0) v1 -- 0x7f69f00ae600 con 0x7f69f00cd9e0 2023-10-19T13:04:50.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.048+0000 7f69e5ffb640 1 -- 172.21.15.57:0/987140553 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 30}]=0 v58) v1 ==== 71+0+0 (secure 0 0 0) 0x7f69f00ae600 con 0x7f69f00cd9e0 2023-10-19T13:04:50.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.050+0000 7f69f770f640 1 -- 172.21.15.57:0/987140553 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f69d406af60 msgr2=0x7f69d406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:50.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.050+0000 7f69f770f640 1 --2- 172.21.15.57:0/987140553 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f69d406af60 0x7f69d406d420 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f69e0035000 tx=0x7f69e002e040 comp rx=0 tx=0).stop 2023-10-19T13:04:50.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.050+0000 7f69f770f640 1 -- 172.21.15.57:0/987140553 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f69f00cd9e0 msgr2=0x7f69f00b0890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:50.054 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.050+0000 7f69f770f640 1 --2- 172.21.15.57:0/987140553 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f69f00cd9e0 0x7f69f00b0890 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7f69ec00c7e0 tx=0x7f69ec00cca0 comp rx=0 tx=0).stop 2023-10-19T13:04:50.054 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.050+0000 7f69f770f640 1 -- 172.21.15.57:0/987140553 shutdown_connections 2023-10-19T13:04:50.054 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.050+0000 7f69f770f640 1 --2- 172.21.15.57:0/987140553 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f69d406af60 0x7f69d406d420 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:50.054 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.050+0000 7f69f770f640 1 --2- 172.21.15.57:0/987140553 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f69f0158cf0 0x7f69f00ad5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:50.055 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.050+0000 7f69f770f640 1 --2- 172.21.15.57:0/987140553 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f69f0158910 0x7f69f00b24e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:50.055 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.050+0000 7f69f770f640 1 --2- 172.21.15.57:0/987140553 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f69f00cd9e0 0x7f69f00b0890 unknown :-1 s=CLOSED pgs=1703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:50.055 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.050+0000 7f69f770f640 1 -- 172.21.15.57:0/987140553 >> 172.21.15.57:0/987140553 conn(0x7f69f00af7a0 msgr2=0x7f69f0157830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:50.055 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.050+0000 7f69f770f640 1 -- 172.21.15.57:0/987140553 shutdown_connections 2023-10-19T13:04:50.055 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.050+0000 7f69f770f640 1 -- 172.21.15.57:0/987140553 wait complete. 2023-10-19T13:04:50.064 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods' 2023-10-19T13:04:50.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.171+0000 7f1e7ed55640 1 -- 172.21.15.57:0/3413767204 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1e780afa20 msgr2=0x7f1e780705a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:50.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.171+0000 7f1e7ed55640 1 --2- 172.21.15.57:0/3413767204 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1e780afa20 0x7f1e780705a0 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7f1e780b1450 tx=0x7f1e6c02d0d0 comp rx=0 tx=0).stop 2023-10-19T13:04:50.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.171+0000 7f1e7ed55640 1 -- 172.21.15.57:0/3413767204 shutdown_connections 2023-10-19T13:04:50.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.171+0000 7f1e7ed55640 1 --2- 172.21.15.57:0/3413767204 >> v2:172.21.15.179:3300/0 conn(0x7f1e780b0530 0x7f1e780b09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:50.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.171+0000 7f1e7ed55640 1 --2- 172.21.15.57:0/3413767204 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1e780afa20 0x7f1e780705a0 unknown :-1 s=CLOSED pgs=1704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:50.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.171+0000 7f1e7ed55640 1 --2- 172.21.15.57:0/3413767204 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1e780cbd20 0x7f1e78158300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:50.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.171+0000 7f1e7ed55640 1 -- 172.21.15.57:0/3413767204 >> 172.21.15.57:0/3413767204 conn(0x7f1e78069930 msgr2=0x7f1e78069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:50.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.171+0000 7f1e7ed55640 1 -- 172.21.15.57:0/3413767204 shutdown_connections 2023-10-19T13:04:50.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.171+0000 7f1e7ed55640 1 -- 172.21.15.57:0/3413767204 wait complete. 2023-10-19T13:04:50.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.172+0000 7f1e7ed55640 1 Processor -- start 2023-10-19T13:04:50.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.172+0000 7f1e7ed55640 1 -- start start 2023-10-19T13:04:50.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.173+0000 7f1e7ed55640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1e780afa20 0x7f1e78137ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:50.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.173+0000 7f1e7ed55640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1e780b0530 0x7f1e7813b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:50.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.173+0000 7f1e7ed55640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1e780cbd20 0x7f1e781e0890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:50.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.173+0000 7f1e7ed55640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e78138420 con 0x7f1e780afa20 2023-10-19T13:04:50.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.173+0000 7f1e7ed55640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e78138a90 con 0x7f1e780b0530 2023-10-19T13:04:50.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.173+0000 7f1e7ed55640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e78138d70 con 0x7f1e780cbd20 2023-10-19T13:04:50.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.173+0000 7f1e7caf1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1e780afa20 0x7f1e78137ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:50.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.173+0000 7f1e7caf1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1e780afa20 0x7f1e78137ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38400/0 (socket says 172.21.15.57:38400) 2023-10-19T13:04:50.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.173+0000 7f1e7caf1640 1 -- 172.21.15.57:0/4268279602 learned_addr learned my addr 172.21.15.57:0/4268279602 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:50.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.173+0000 7f1e77fff640 1 --2- 172.21.15.57:0/4268279602 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1e780b0530 0x7f1e7813b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:50.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.173+0000 7f1e7d2f2640 1 --2- 172.21.15.57:0/4268279602 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1e780cbd20 0x7f1e781e0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:50.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.173+0000 7f1e7caf1640 1 -- 172.21.15.57:0/4268279602 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1e780cbd20 msgr2=0x7f1e781e0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:50.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.173+0000 7f1e7caf1640 1 --2- 172.21.15.57:0/4268279602 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1e780cbd20 0x7f1e781e0890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:50.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.173+0000 7f1e7caf1640 1 -- 172.21.15.57:0/4268279602 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1e780b0530 msgr2=0x7f1e7813b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:50.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.173+0000 7f1e7caf1640 1 --2- 172.21.15.57:0/4268279602 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1e780b0530 0x7f1e7813b130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:50.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.173+0000 7f1e7caf1640 1 -- 172.21.15.57:0/4268279602 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e6c00a670 con 0x7f1e780afa20 2023-10-19T13:04:50.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.173+0000 7f1e7d2f2640 1 --2- 172.21.15.57:0/4268279602 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1e780cbd20 0x7f1e781e0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:04:50.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.174+0000 7f1e7caf1640 1 --2- 172.21.15.57:0/4268279602 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1e780afa20 0x7f1e78137ee0 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7f1e6800c970 tx=0x7f1e6800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:50.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.174+0000 7f1e757fa640 1 -- 172.21.15.57:0/4268279602 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e68007bb0 con 0x7f1e780afa20 2023-10-19T13:04:50.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.174+0000 7f1e757fa640 1 -- 172.21.15.57:0/4268279602 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f1e68007e00 con 0x7f1e780afa20 2023-10-19T13:04:50.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.174+0000 7f1e757fa640 1 -- 172.21.15.57:0/4268279602 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e68005280 con 0x7f1e780afa20 2023-10-19T13:04:50.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.174+0000 7f1e7ed55640 1 -- 172.21.15.57:0/4268279602 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e78139050 con 0x7f1e780afa20 2023-10-19T13:04:50.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.174+0000 7f1e7ed55640 1 -- 172.21.15.57:0/4268279602 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e7813b670 con 0x7f1e780afa20 2023-10-19T13:04:50.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.175+0000 7f1e7ed55640 1 -- 172.21.15.57:0/4268279602 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e78139850 con 0x7f1e780afa20 2023-10-19T13:04:50.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.175+0000 7f1e757fa640 1 -- 172.21.15.57:0/4268279602 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 79) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f1e680053e0 con 0x7f1e780afa20 2023-10-19T13:04:50.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.176+0000 7f1e757fa640 1 --2- 172.21.15.57:0/4268279602 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f1e6406af60 0x7f1e6406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:50.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.176+0000 7f1e757fa640 1 -- 172.21.15.57:0/4268279602 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6022+0+0 (secure 0 0 0) 0x7f1e68089d60 con 0x7f1e780afa20 2023-10-19T13:04:50.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.176+0000 7f1e77fff640 1 --2- 172.21.15.57:0/4268279602 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f1e6406af60 0x7f1e6406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:50.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.177+0000 7f1e77fff640 1 --2- 172.21.15.57:0/4268279602 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f1e6406af60 0x7f1e6406d420 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f1e6c036000 tx=0x7f1e6c004730 comp rx=0 tx=0).ready entity=mgr.8419 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:50.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.180+0000 7f1e757fa640 1 -- 172.21.15.57:0/4268279602 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f1e78139850 con 0x7f1e780afa20 2023-10-19T13:04:50.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.428+0000 7f1e7ed55640 1 -- 172.21.15.57:0/4268279602 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods"]} v 0) v1 -- 0x7f1e78139850 con 0x7f1e780afa20 2023-10-19T13:04:50.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.728+0000 7f1e757fa640 1 -- 172.21.15.57:0/4268279602 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods"]}]=0 v790) v1 ==== 160+0+0 (secure 0 0 0) 0x7f1e78139850 con 0x7f1e780afa20 2023-10-19T13:04:50.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.730+0000 7f1e7ed55640 1 -- 172.21.15.57:0/4268279602 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f1e6406af60 msgr2=0x7f1e6406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:50.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.730+0000 7f1e7ed55640 1 --2- 172.21.15.57:0/4268279602 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f1e6406af60 0x7f1e6406d420 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f1e6c036000 tx=0x7f1e6c004730 comp rx=0 tx=0).stop 2023-10-19T13:04:50.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.730+0000 7f1e7ed55640 1 -- 172.21.15.57:0/4268279602 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1e780afa20 msgr2=0x7f1e78137ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:50.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.730+0000 7f1e7ed55640 1 --2- 172.21.15.57:0/4268279602 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1e780afa20 0x7f1e78137ee0 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7f1e6800c970 tx=0x7f1e6800ce30 comp rx=0 tx=0).stop 2023-10-19T13:04:50.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.730+0000 7f1e7ed55640 1 -- 172.21.15.57:0/4268279602 shutdown_connections 2023-10-19T13:04:50.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.730+0000 7f1e7ed55640 1 --2- 172.21.15.57:0/4268279602 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f1e6406af60 0x7f1e6406d420 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:50.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.730+0000 7f1e7ed55640 1 --2- 172.21.15.57:0/4268279602 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1e780cbd20 0x7f1e781e0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:50.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.730+0000 7f1e7ed55640 1 --2- 172.21.15.57:0/4268279602 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1e780b0530 0x7f1e7813b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:50.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.730+0000 7f1e7ed55640 1 --2- 172.21.15.57:0/4268279602 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1e780afa20 0x7f1e78137ee0 unknown :-1 s=CLOSED pgs=1705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:50.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.730+0000 7f1e7ed55640 1 -- 172.21.15.57:0/4268279602 >> 172.21.15.57:0/4268279602 conn(0x7f1e78069930 msgr2=0x7f1e78069d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:50.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.730+0000 7f1e7ed55640 1 -- 172.21.15.57:0/4268279602 shutdown_connections 2023-10-19T13:04:50.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.731+0000 7f1e7ed55640 1 -- 172.21.15.57:0/4268279602 wait complete. 2023-10-19T13:04:50.745 INFO:tasks.cephfs_test_runner:test_snap_schedule_all_periods (tasks.cephfs.test_snap_schedules.TestSnapSchedules) ... ok 2023-10-19T13:04:50.746 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_for_number_of_snaps_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test that number of snaps retained are as per user spec. 2023-10-19T13:04:50.746 INFO:tasks.cephfs_test_runner:test_snap_schedule_for_number_of_snaps_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-10-19T13:04:50.747 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T13:04:50.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.858+0000 7fbef21e7640 1 -- 172.21.15.57:0/1056127657 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbed40039a0 msgr2=0x7fbed4003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:50.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.858+0000 7fbef21e7640 1 --2- 172.21.15.57:0/1056127657 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbed40039a0 0x7fbed4003dc0 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7fbedc001650 tx=0x7fbed40057f0 comp rx=0 tx=0).stop 2023-10-19T13:04:50.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.858+0000 7fbeebfff640 1 -- 172.21.15.57:0/1056127657 reap_dead start 2023-10-19T13:04:50.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.858+0000 7fbef21e7640 1 -- 172.21.15.57:0/1056127657 shutdown_connections 2023-10-19T13:04:50.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.858+0000 7fbef21e7640 1 -- 172.21.15.57:0/1056127657 >> 172.21.15.57:0/1056127657 conn(0x7fbeec09a030 msgr2=0x7fbeec0d0d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:50.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.858+0000 7fbef21e7640 1 -- 172.21.15.57:0/1056127657 shutdown_connections 2023-10-19T13:04:50.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.858+0000 7fbef21e7640 1 -- 172.21.15.57:0/1056127657 wait complete. 2023-10-19T13:04:50.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.859+0000 7fbef21e7640 1 Processor -- start 2023-10-19T13:04:50.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.859+0000 7fbef21e7640 1 -- start start 2023-10-19T13:04:50.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.860+0000 7fbef21e7640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbed40039a0 0x7fbeec0b2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:50.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.860+0000 7fbef21e7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbeec0c82d0 0x7fbeec0ca740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:50.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.860+0000 7fbef21e7640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbeec0cac80 0x7fbeec0b5db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:50.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.860+0000 7fbef21e7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbeec0cb1e0 con 0x7fbeec0c82d0 2023-10-19T13:04:50.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.860+0000 7fbef21e7640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbeec1f1db0 con 0x7fbed40039a0 2023-10-19T13:04:50.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.860+0000 7fbef21e7640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbeec1f2090 con 0x7fbeec0cac80 2023-10-19T13:04:50.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.860+0000 7fbeeaffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbeec0c82d0 0x7fbeec0ca740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:50.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.860+0000 7fbeeaffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbeec0c82d0 0x7fbeec0ca740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38418/0 (socket says 172.21.15.57:38418) 2023-10-19T13:04:50.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.860+0000 7fbeeaffd640 1 -- 172.21.15.57:0/3103342292 learned_addr learned my addr 172.21.15.57:0/3103342292 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:50.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.860+0000 7fbeeb7fe640 1 --2- 172.21.15.57:0/3103342292 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbed40039a0 0x7fbeec0b2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:50.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.860+0000 7fbeebfff640 1 --2- 172.21.15.57:0/3103342292 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbeec0cac80 0x7fbeec0b5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:50.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.860+0000 7fbeeaffd640 1 -- 172.21.15.57:0/3103342292 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbeec0cac80 msgr2=0x7fbeec0b5db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:50.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.860+0000 7fbeeaffd640 1 --2- 172.21.15.57:0/3103342292 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbeec0cac80 0x7fbeec0b5db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:50.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.860+0000 7fbeeaffd640 1 -- 172.21.15.57:0/3103342292 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbed40039a0 msgr2=0x7fbeec0b2410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:50.868 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.860+0000 7fbeeaffd640 1 --2- 172.21.15.57:0/3103342292 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbed40039a0 0x7fbeec0b2410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:50.868 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.860+0000 7fbeeaffd640 1 -- 172.21.15.57:0/3103342292 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbeec1f2370 con 0x7fbeec0c82d0 2023-10-19T13:04:50.868 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.860+0000 7fbeeb7fe640 1 --2- 172.21.15.57:0/3103342292 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbed40039a0 0x7fbeec0b2410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:04:50.868 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.860+0000 7fbeeaffd640 1 --2- 172.21.15.57:0/3103342292 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbeec0c82d0 0x7fbeec0ca740 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7fbee0011a50 tx=0x7fbee0011f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:50.868 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.860+0000 7fbed3fff640 1 -- 172.21.15.57:0/3103342292 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbee000fcd0 con 0x7fbeec0c82d0 2023-10-19T13:04:50.868 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.861+0000 7fbed3fff640 1 -- 172.21.15.57:0/3103342292 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fbee0012450 con 0x7fbeec0c82d0 2023-10-19T13:04:50.869 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.861+0000 7fbef21e7640 1 -- 172.21.15.57:0/3103342292 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbeec1f1aa0 con 0x7fbeec0c82d0 2023-10-19T13:04:50.869 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.861+0000 7fbed3fff640 1 -- 172.21.15.57:0/3103342292 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbee0012840 con 0x7fbeec0c82d0 2023-10-19T13:04:50.869 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.861+0000 7fbef21e7640 1 -- 172.21.15.57:0/3103342292 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbeec0b2b30 con 0x7fbeec0c82d0 2023-10-19T13:04:50.869 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.862+0000 7fbef21e7640 1 -- 172.21.15.57:0/3103342292 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbeec0675f0 con 0x7fbeec0c82d0 2023-10-19T13:04:50.869 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.862+0000 7fbed3fff640 1 -- 172.21.15.57:0/3103342292 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 79) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fbee00129a0 con 0x7fbeec0c82d0 2023-10-19T13:04:50.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.862+0000 7fbed3fff640 1 --2- 172.21.15.57:0/3103342292 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fbed806af60 0x7fbed806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:50.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.863+0000 7fbed3fff640 1 -- 172.21.15.57:0/3103342292 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6022+0+0 (secure 0 0 0) 0x7fbee008a550 con 0x7fbeec0c82d0 2023-10-19T13:04:50.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.863+0000 7fbeeb7fe640 1 --2- 172.21.15.57:0/3103342292 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fbed806af60 0x7fbed806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:50.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.863+0000 7fbeeb7fe640 1 --2- 172.21.15.57:0/3103342292 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fbed806af60 0x7fbed806d420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fbedc034000 tx=0x7fbedc02f040 comp rx=0 tx=0).ready entity=mgr.8419 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:50.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:50.867+0000 7fbed3fff640 1 -- 172.21.15.57:0/3103342292 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fbee0055460 con 0x7fbeec0c82d0 2023-10-19T13:04:51.049 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.047+0000 7fbef21e7640 1 -- 172.21.15.57:0/3103342292 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbeec0b2b30 con 0x7fbeec0c82d0 2023-10-19T13:04:51.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.048+0000 7fbed3fff640 1 -- 172.21.15.57:0/3103342292 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v149) v1 ==== 74+0+14339 (secure 0 0 0) 0x7fbeec0b2b30 con 0x7fbeec0c82d0 2023-10-19T13:04:51.051 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:04:51.051 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":149,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T13:04:49.712969+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.57:6834/3474229632":"2023-10-20T13:04:46.703793+0000","172.21.15.179:0/2820959197":"2023-10-20T13:04:44.277539+0000","172.21.15.179:6833/3526956956":"2023-10-20T13:04:44.277539+0000","172.21.15.179:0/2214725838":"2023-10-20T13:04:44.277539+0000","172.21.15.179:6832/3526956956":"2023-10-20T13:04:44.277539+0000","172.21.15.179:0/2775321950":"2023-10-20T13:04:44.277539+0000","172.21.15.179:0/3591554987":"2023-10-20T13:04:44.277539+0000","172.21.15.179:0/1299954463":"2023-10-20T13:04:44.277539+0000","172.21.15.179:0/1024578486":"2023-10-20T13:04:44.277539+0000","172.21.15.179:0/1700025841":"2023-10-20T13:04:33.816783+0000","172.21.15.179:0/3160483813":"2023-10-20T13:04:33.816783+0000","172.21.15.179:0/4039784875":"2023-10-20T13:04:33.816783+0000","172.21.15.179:0/1023851911":"2023-10-20T13:04:33.816783+0000","172.21.15.179:0/2968654981":"2023-10-20T13:04:33.816783+0000","172.21.15.179:0/1247110426":"2023-10-20T13:04:44.277539+0000","172.21.15.179:0/3605212267":"2023-10-20T13:04:33.816783+0000","172.21.15.179:6833/2346394725":"2023-10-20T13:04:33.816783+0000","172.21.15.57:6832/3474229632":"2023-10-20T13:04:46.703793+0000","172.21.15.179:6832/2346394725":"2023-10-20T13:04:33.816783+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T13:04:51.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.050+0000 7fbef21e7640 1 -- 172.21.15.57:0/3103342292 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fbed806af60 msgr2=0x7fbed806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:51.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.050+0000 7fbef21e7640 1 --2- 172.21.15.57:0/3103342292 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fbed806af60 0x7fbed806d420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fbedc034000 tx=0x7fbedc02f040 comp rx=0 tx=0).stop 2023-10-19T13:04:51.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.050+0000 7fbef21e7640 1 -- 172.21.15.57:0/3103342292 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbeec0c82d0 msgr2=0x7fbeec0ca740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:51.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.050+0000 7fbef21e7640 1 --2- 172.21.15.57:0/3103342292 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbeec0c82d0 0x7fbeec0ca740 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7fbee0011a50 tx=0x7fbee0011f10 comp rx=0 tx=0).stop 2023-10-19T13:04:51.054 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.050+0000 7fbef21e7640 1 -- 172.21.15.57:0/3103342292 shutdown_connections 2023-10-19T13:04:51.054 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.050+0000 7fbef21e7640 1 --2- 172.21.15.57:0/3103342292 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fbed806af60 0x7fbed806d420 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:51.054 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.050+0000 7fbef21e7640 1 --2- 172.21.15.57:0/3103342292 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbeec0cac80 0x7fbeec0b5db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:51.054 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.050+0000 7fbef21e7640 1 --2- 172.21.15.57:0/3103342292 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbeec0c82d0 0x7fbeec0ca740 unknown :-1 s=CLOSED pgs=1708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:51.054 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.050+0000 7fbef21e7640 1 --2- 172.21.15.57:0/3103342292 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbed40039a0 0x7fbeec0b2410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:51.055 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.050+0000 7fbef21e7640 1 -- 172.21.15.57:0/3103342292 >> 172.21.15.57:0/3103342292 conn(0x7fbeec09a030 msgr2=0x7fbeec15a400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:51.055 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.050+0000 7fbef21e7640 1 -- 172.21.15.57:0/3103342292 shutdown_connections 2023-10-19T13:04:51.055 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.050+0000 7fbef21e7640 1 -- 172.21.15.57:0/3103342292 wait complete. 2023-10-19T13:04:51.063 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-19T12:44:07.371186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-19T13:04:51.064 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-19T13:04:51.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.174+0000 7fc5ee3f5640 1 -- 172.21.15.57:0/3887710684 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc5e806f0a0 msgr2=0x7fc5e806dbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:51.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.174+0000 7fc5ee3f5640 1 --2- 172.21.15.57:0/3887710684 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc5e806f0a0 0x7fc5e806dbf0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fc5d0009830 tx=0x7fc5d002bd60 comp rx=0 tx=0).stop 2023-10-19T13:04:51.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.174+0000 7fc5ee3f5640 1 -- 172.21.15.57:0/3887710684 shutdown_connections 2023-10-19T13:04:51.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.174+0000 7fc5ee3f5640 1 --2- 172.21.15.57:0/3887710684 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc5e806f0a0 0x7fc5e806dbf0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:51.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.174+0000 7fc5ee3f5640 1 --2- 172.21.15.57:0/3887710684 >> v2:172.21.15.179:3300/0 conn(0x7fc5e806d2d0 0x7fc5e806d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:51.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.174+0000 7fc5ee3f5640 1 -- 172.21.15.57:0/3887710684 >> 172.21.15.57:0/3887710684 conn(0x7fc5e80b0670 msgr2=0x7fc5e80b0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:51.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.174+0000 7fc5ee3f5640 1 -- 172.21.15.57:0/3887710684 shutdown_connections 2023-10-19T13:04:51.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.174+0000 7fc5ee3f5640 1 -- 172.21.15.57:0/3887710684 wait complete. 2023-10-19T13:04:51.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.175+0000 7fc5ee3f5640 1 Processor -- start 2023-10-19T13:04:51.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.175+0000 7fc5ee3f5640 1 -- start start 2023-10-19T13:04:51.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.176+0000 7fc5ee3f5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5e806d2d0 0x7fc5e814fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:51.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.176+0000 7fc5ee3f5640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc5e806f0a0 0x7fc5e80ca150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:51.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.176+0000 7fc5ee3f5640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc5e80ab950 0x7fc5e80cbf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:51.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.176+0000 7fc5ee3f5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e80ca690 con 0x7fc5e806d2d0 2023-10-19T13:04:51.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.176+0000 7fc5ee3f5640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e80cc4b0 con 0x7fc5e806f0a0 2023-10-19T13:04:51.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.176+0000 7fc5ee3f5640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e80c2a00 con 0x7fc5e80ab950 2023-10-19T13:04:51.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.176+0000 7fc5e7fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5e806d2d0 0x7fc5e814fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:51.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.176+0000 7fc5e7fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5e806d2d0 0x7fc5e814fea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38434/0 (socket says 172.21.15.57:38434) 2023-10-19T13:04:51.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.176+0000 7fc5e7fff640 1 -- 172.21.15.57:0/3641393017 learned_addr learned my addr 172.21.15.57:0/3641393017 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:51.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.176+0000 7fc5e77fe640 1 --2- 172.21.15.57:0/3641393017 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc5e806f0a0 0x7fc5e80ca150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:51.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.176+0000 7fc5ec992640 1 --2- 172.21.15.57:0/3641393017 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc5e80ab950 0x7fc5e80cbf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:51.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.176+0000 7fc5e7fff640 1 -- 172.21.15.57:0/3641393017 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc5e80ab950 msgr2=0x7fc5e80cbf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:51.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.176+0000 7fc5e7fff640 1 --2- 172.21.15.57:0/3641393017 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc5e80ab950 0x7fc5e80cbf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:51.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.176+0000 7fc5e7fff640 1 -- 172.21.15.57:0/3641393017 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc5e806f0a0 msgr2=0x7fc5e80ca150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:51.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.176+0000 7fc5e7fff640 1 --2- 172.21.15.57:0/3641393017 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc5e806f0a0 0x7fc5e80ca150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:51.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.176+0000 7fc5e7fff640 1 -- 172.21.15.57:0/3641393017 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5d8009490 con 0x7fc5e806d2d0 2023-10-19T13:04:51.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.176+0000 7fc5e7fff640 1 --2- 172.21.15.57:0/3641393017 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5e806d2d0 0x7fc5e814fea0 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7fc5d800ea60 tx=0x7fc5d800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:51.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.177+0000 7fc5e4ff9640 1 -- 172.21.15.57:0/3641393017 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5d800cde0 con 0x7fc5e806d2d0 2023-10-19T13:04:51.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.177+0000 7fc5e4ff9640 1 -- 172.21.15.57:0/3641393017 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fc5d8004020 con 0x7fc5e806d2d0 2023-10-19T13:04:51.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.177+0000 7fc5ee3f5640 1 -- 172.21.15.57:0/3641393017 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5d0009490 con 0x7fc5e806d2d0 2023-10-19T13:04:51.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.177+0000 7fc5e4ff9640 1 -- 172.21.15.57:0/3641393017 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5d8004410 con 0x7fc5e806d2d0 2023-10-19T13:04:51.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.177+0000 7fc5ee3f5640 1 -- 172.21.15.57:0/3641393017 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5e80c2fc0 con 0x7fc5e806d2d0 2023-10-19T13:04:51.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.178+0000 7fc5ee3f5640 1 -- 172.21.15.57:0/3641393017 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5e80ce1d0 con 0x7fc5e806d2d0 2023-10-19T13:04:51.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.178+0000 7fc5e4ff9640 1 -- 172.21.15.57:0/3641393017 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 79) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc5d80035f0 con 0x7fc5e806d2d0 2023-10-19T13:04:51.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.178+0000 7fc5e4ff9640 1 --2- 172.21.15.57:0/3641393017 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fc5c806af60 0x7fc5c806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:51.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.178+0000 7fc5e4ff9640 1 -- 172.21.15.57:0/3641393017 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6022+0+0 (secure 0 0 0) 0x7fc5d8014070 con 0x7fc5e806d2d0 2023-10-19T13:04:51.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.179+0000 7fc5e77fe640 1 --2- 172.21.15.57:0/3641393017 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fc5c806af60 0x7fc5c806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:51.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.179+0000 7fc5e77fe640 1 --2- 172.21.15.57:0/3641393017 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fc5c806af60 0x7fc5c806d420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fc5d0036000 tx=0x7fc5d0004730 comp rx=0 tx=0).ready entity=mgr.8419 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:51.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.182+0000 7fc5e4ff9640 1 -- 172.21.15.57:0/3641393017 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc5d8055280 con 0x7fc5e806d2d0 2023-10-19T13:04:51.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.358+0000 7fc5ee3f5640 1 -- 172.21.15.57:0/3641393017 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc5e80690e0 con 0x7fc5e806d2d0 2023-10-19T13:04:51.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.358+0000 7fc5e4ff9640 1 -- 172.21.15.57:0/3641393017 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v149) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc5d80588a0 con 0x7fc5e806d2d0 2023-10-19T13:04:51.361 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-19T13:04:51.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.360+0000 7fc5ee3f5640 1 -- 172.21.15.57:0/3641393017 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fc5c806af60 msgr2=0x7fc5c806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:51.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.360+0000 7fc5ee3f5640 1 --2- 172.21.15.57:0/3641393017 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fc5c806af60 0x7fc5c806d420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fc5d0036000 tx=0x7fc5d0004730 comp rx=0 tx=0).stop 2023-10-19T13:04:51.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.360+0000 7fc5ee3f5640 1 -- 172.21.15.57:0/3641393017 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5e806d2d0 msgr2=0x7fc5e814fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:51.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.360+0000 7fc5ee3f5640 1 --2- 172.21.15.57:0/3641393017 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5e806d2d0 0x7fc5e814fea0 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7fc5d800ea60 tx=0x7fc5d800ef20 comp rx=0 tx=0).stop 2023-10-19T13:04:51.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.360+0000 7fc5ee3f5640 1 -- 172.21.15.57:0/3641393017 shutdown_connections 2023-10-19T13:04:51.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.360+0000 7fc5ee3f5640 1 --2- 172.21.15.57:0/3641393017 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fc5c806af60 0x7fc5c806d420 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:51.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.360+0000 7fc5ee3f5640 1 --2- 172.21.15.57:0/3641393017 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc5e80ab950 0x7fc5e80cbf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:51.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.360+0000 7fc5ee3f5640 1 --2- 172.21.15.57:0/3641393017 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc5e806f0a0 0x7fc5e80ca150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:51.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.360+0000 7fc5ee3f5640 1 --2- 172.21.15.57:0/3641393017 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5e806d2d0 0x7fc5e814fea0 unknown :-1 s=CLOSED pgs=1709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:51.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.360+0000 7fc5ee3f5640 1 -- 172.21.15.57:0/3641393017 >> 172.21.15.57:0/3641393017 conn(0x7fc5e80b0670 msgr2=0x7fc5e8156ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:51.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.360+0000 7fc5ee3f5640 1 -- 172.21.15.57:0/3641393017 shutdown_connections 2023-10-19T13:04:51.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.360+0000 7fc5ee3f5640 1 -- 172.21.15.57:0/3641393017 wait complete. 2023-10-19T13:04:51.373 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention' 2023-10-19T13:04:51.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.480+0000 7faa2d94c640 1 -- 172.21.15.57:0/235284661 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa2815b2a0 msgr2=0x7faa280cf720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:51.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.480+0000 7faa2d94c640 1 --2- 172.21.15.57:0/235284661 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa2815b2a0 0x7faa280cf720 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7faa10009830 tx=0x7faa1002bcd0 comp rx=0 tx=0).stop 2023-10-19T13:04:51.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.480+0000 7faa2d94c640 1 -- 172.21.15.57:0/235284661 shutdown_connections 2023-10-19T13:04:51.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.480+0000 7faa2d94c640 1 --2- 172.21.15.57:0/235284661 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa2815b2a0 0x7faa280cf720 unknown :-1 s=CLOSED pgs=1711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:51.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.480+0000 7faa2d94c640 1 --2- 172.21.15.57:0/235284661 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7faa280cf340 0x7faa280d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:51.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.480+0000 7faa2d94c640 1 -- 172.21.15.57:0/235284661 >> 172.21.15.57:0/235284661 conn(0x7faa280d0d60 msgr2=0x7faa280d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:51.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.480+0000 7faa2d94c640 1 -- 172.21.15.57:0/235284661 shutdown_connections 2023-10-19T13:04:51.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.480+0000 7faa2d94c640 1 -- 172.21.15.57:0/235284661 wait complete. 2023-10-19T13:04:51.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.481+0000 7faa2d94c640 1 Processor -- start 2023-10-19T13:04:51.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.481+0000 7faa2d94c640 1 -- start start 2023-10-19T13:04:51.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.482+0000 7faa2d94c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faa280ceb40 0x7faa280c7e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:51.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.482+0000 7faa2d94c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7faa280cf340 0x7faa28157bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:51.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.482+0000 7faa2d94c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa2815b2a0 0x7faa280ca840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:51.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.482+0000 7faa2d94c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa28158130 con 0x7faa2815b2a0 2023-10-19T13:04:51.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.482+0000 7faa2d94c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa280ca320 con 0x7faa280ceb40 2023-10-19T13:04:51.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.482+0000 7faa2d94c640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa280cad80 con 0x7faa280cf340 2023-10-19T13:04:51.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.482+0000 7faa277fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa2815b2a0 0x7faa280ca840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:51.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.482+0000 7faa277fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa2815b2a0 0x7faa280ca840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38446/0 (socket says 172.21.15.57:38446) 2023-10-19T13:04:51.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.482+0000 7faa277fe640 1 -- 172.21.15.57:0/3326070431 learned_addr learned my addr 172.21.15.57:0/3326070431 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:51.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.482+0000 7faa26ffd640 1 --2- 172.21.15.57:0/3326070431 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faa280ceb40 0x7faa280c7e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:51.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.482+0000 7faa267fc640 1 --2- 172.21.15.57:0/3326070431 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7faa280cf340 0x7faa28157bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:51.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.483+0000 7faa277fe640 1 -- 172.21.15.57:0/3326070431 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7faa280cf340 msgr2=0x7faa28157bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:51.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.483+0000 7faa277fe640 1 --2- 172.21.15.57:0/3326070431 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7faa280cf340 0x7faa28157bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:51.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.483+0000 7faa277fe640 1 -- 172.21.15.57:0/3326070431 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faa280ceb40 msgr2=0x7faa280c7e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:51.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.483+0000 7faa277fe640 1 --2- 172.21.15.57:0/3326070431 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faa280ceb40 0x7faa280c7e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:51.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.483+0000 7faa277fe640 1 -- 172.21.15.57:0/3326070431 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa10009490 con 0x7faa2815b2a0 2023-10-19T13:04:51.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.483+0000 7faa267fc640 1 --2- 172.21.15.57:0/3326070431 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7faa280cf340 0x7faa28157bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T13:04:51.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.483+0000 7faa277fe640 1 --2- 172.21.15.57:0/3326070431 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa2815b2a0 0x7faa280ca840 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7faa2000c430 tx=0x7faa2000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:51.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.483+0000 7faa1ffff640 1 -- 172.21.15.57:0/3326070431 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa20011790 con 0x7faa2815b2a0 2023-10-19T13:04:51.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.483+0000 7faa1ffff640 1 -- 172.21.15.57:0/3326070431 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7faa200119e0 con 0x7faa2815b2a0 2023-10-19T13:04:51.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.483+0000 7faa2d94c640 1 -- 172.21.15.57:0/3326070431 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa280c8d60 con 0x7faa2815b2a0 2023-10-19T13:04:51.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.483+0000 7faa1ffff640 1 -- 172.21.15.57:0/3326070431 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa20011e10 con 0x7faa2815b2a0 2023-10-19T13:04:51.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.483+0000 7faa2d94c640 1 -- 172.21.15.57:0/3326070431 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa280c87a0 con 0x7faa2815b2a0 2023-10-19T13:04:51.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.484+0000 7faa2d94c640 1 -- 172.21.15.57:0/3326070431 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa280690e0 con 0x7faa2815b2a0 2023-10-19T13:04:51.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.484+0000 7faa1ffff640 1 -- 172.21.15.57:0/3326070431 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 79) v1 ==== 89331+0+0 (secure 0 0 0) 0x7faa2001c460 con 0x7faa2815b2a0 2023-10-19T13:04:51.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.485+0000 7faa1ffff640 1 --2- 172.21.15.57:0/3326070431 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7faa0806af90 0x7faa0806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:51.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.485+0000 7faa1ffff640 1 -- 172.21.15.57:0/3326070431 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6022+0+0 (secure 0 0 0) 0x7faa200899d0 con 0x7faa2815b2a0 2023-10-19T13:04:51.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.485+0000 7faa26ffd640 1 --2- 172.21.15.57:0/3326070431 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7faa0806af90 0x7faa0806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:51.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.485+0000 7faa26ffd640 1 --2- 172.21.15.57:0/3326070431 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7faa0806af90 0x7faa0806d450 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7faa18009f20 tx=0x7faa180094f0 comp rx=0 tx=0).ready entity=mgr.8419 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:51.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.489+0000 7faa1ffff640 1 -- 172.21.15.57:0/3326070431 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7faa20054830 con 0x7faa2815b2a0 2023-10-19T13:04:51.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:51.730+0000 7faa2d94c640 1 -- 172.21.15.57:0/3326070431 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention"]} v 0) v1 -- 0x7faa280d0b40 con 0x7faa2815b2a0 2023-10-19T13:04:52.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:52.737+0000 7faa1ffff640 1 -- 172.21.15.57:0/3326070431 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention"]}]=0 v792) v1 ==== 181+0+0 (secure 0 0 0) 0x7faa20057e50 con 0x7faa2815b2a0 2023-10-19T13:04:52.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:52.740+0000 7faa2d94c640 1 -- 172.21.15.57:0/3326070431 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7faa0806af90 msgr2=0x7faa0806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:52.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:52.740+0000 7faa2d94c640 1 --2- 172.21.15.57:0/3326070431 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7faa0806af90 0x7faa0806d450 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7faa18009f20 tx=0x7faa180094f0 comp rx=0 tx=0).stop 2023-10-19T13:04:52.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:52.740+0000 7faa2d94c640 1 -- 172.21.15.57:0/3326070431 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa2815b2a0 msgr2=0x7faa280ca840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:52.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:52.740+0000 7faa2d94c640 1 --2- 172.21.15.57:0/3326070431 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa2815b2a0 0x7faa280ca840 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7faa2000c430 tx=0x7faa2000c8f0 comp rx=0 tx=0).stop 2023-10-19T13:04:52.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:52.740+0000 7faa2d94c640 1 -- 172.21.15.57:0/3326070431 shutdown_connections 2023-10-19T13:04:52.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:52.740+0000 7faa2d94c640 1 --2- 172.21.15.57:0/3326070431 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7faa0806af90 0x7faa0806d450 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:52.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:52.740+0000 7faa2d94c640 1 --2- 172.21.15.57:0/3326070431 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa2815b2a0 0x7faa280ca840 unknown :-1 s=CLOSED pgs=1712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:52.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:52.740+0000 7faa2d94c640 1 --2- 172.21.15.57:0/3326070431 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7faa280cf340 0x7faa28157bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:52.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:52.740+0000 7faa2d94c640 1 --2- 172.21.15.57:0/3326070431 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faa280ceb40 0x7faa280c7e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:52.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:52.740+0000 7faa2d94c640 1 -- 172.21.15.57:0/3326070431 >> 172.21.15.57:0/3326070431 conn(0x7faa280d0d60 msgr2=0x7faa280b0630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:52.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:52.740+0000 7faa2d94c640 1 -- 172.21.15.57:0/3326070431 shutdown_connections 2023-10-19T13:04:52.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:52.740+0000 7faa2d94c640 1 -- 172.21.15.57:0/3326070431 wait complete. 2023-10-19T13:04:52.756 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2023-10-19T13:04:52.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:52.869+0000 7fcb2553e640 1 -- 172.21.15.57:0/34335748 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcb0c0047c0 msgr2=0x7fcb0c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:52.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:52.869+0000 7fcb2553e640 1 --2- 172.21.15.57:0/34335748 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcb0c0047c0 0x7fcb0c004c40 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7fcb08009b50 tx=0x7fcb0c0057f0 comp rx=0 tx=0).stop 2023-10-19T13:04:52.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:52.869+0000 7fcb2553e640 1 -- 172.21.15.57:0/34335748 shutdown_connections 2023-10-19T13:04:52.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:52.869+0000 7fcb2553e640 1 --2- 172.21.15.57:0/34335748 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcb0c0047c0 0x7fcb0c004c40 unknown :-1 s=CLOSED pgs=1714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:52.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:52.869+0000 7fcb2553e640 1 -- 172.21.15.57:0/34335748 >> 172.21.15.57:0/34335748 conn(0x7fcb2006fae0 msgr2=0x7fcb2006fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:52.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:52.869+0000 7fcb2553e640 1 -- 172.21.15.57:0/34335748 shutdown_connections 2023-10-19T13:04:52.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:52.869+0000 7fcb2553e640 1 -- 172.21.15.57:0/34335748 wait complete. 2023-10-19T13:04:52.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:52.870+0000 7fcb2553e640 1 Processor -- start 2023-10-19T13:04:52.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:52.870+0000 7fcb2553e640 1 -- start start 2023-10-19T13:04:52.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:52.871+0000 7fcb2553e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcb0c0047c0 0x7fcb2014b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:52.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:52.871+0000 7fcb2553e640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcb2014b8f0 0x7fcb200cb8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:52.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:52.871+0000 7fcb2553e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcb2013bf30 0x7fcb200cbe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:52.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:52.871+0000 7fcb2553e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb2014c9a0 con 0x7fcb2013bf30 2023-10-19T13:04:52.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:52.871+0000 7fcb2553e640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb2014bd90 con 0x7fcb0c0047c0 2023-10-19T13:04:52.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:52.871+0000 7fcb2553e640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb2014c070 con 0x7fcb2014b8f0 2023-10-19T13:04:52.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:52.871+0000 7fcb1f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcb2013bf30 0x7fcb200cbe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:52.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:52.871+0000 7fcb1f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcb2013bf30 0x7fcb200cbe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38456/0 (socket says 172.21.15.57:38456) 2023-10-19T13:04:52.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:52.871+0000 7fcb1f7fe640 1 -- 172.21.15.57:0/755641426 learned_addr learned my addr 172.21.15.57:0/755641426 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:52.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:52.871+0000 7fcb1e7fc640 1 --2- 172.21.15.57:0/755641426 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcb2014b8f0 0x7fcb200cb8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:52.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:52.871+0000 7fcb1effd640 1 --2- 172.21.15.57:0/755641426 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcb0c0047c0 0x7fcb2014b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:52.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:52.872+0000 7fcb1f7fe640 1 -- 172.21.15.57:0/755641426 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcb2014b8f0 msgr2=0x7fcb200cb8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:52.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:52.872+0000 7fcb1f7fe640 1 --2- 172.21.15.57:0/755641426 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcb2014b8f0 0x7fcb200cb8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:52.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:52.872+0000 7fcb1f7fe640 1 -- 172.21.15.57:0/755641426 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcb0c0047c0 msgr2=0x7fcb2014b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:52.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:52.872+0000 7fcb1f7fe640 1 --2- 172.21.15.57:0/755641426 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcb0c0047c0 0x7fcb2014b3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:52.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:52.872+0000 7fcb1f7fe640 1 -- 172.21.15.57:0/755641426 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb200b2330 con 0x7fcb2013bf30 2023-10-19T13:04:52.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:52.872+0000 7fcb1e7fc640 1 --2- 172.21.15.57:0/755641426 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcb2014b8f0 0x7fcb200cb8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:04:52.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:52.872+0000 7fcb1effd640 1 --2- 172.21.15.57:0/755641426 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcb0c0047c0 0x7fcb2014b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:04:52.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:52.872+0000 7fcb1f7fe640 1 --2- 172.21.15.57:0/755641426 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcb2013bf30 0x7fcb200cbe00 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7fcb10002230 tx=0x7fcb1000eb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:52.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:52.872+0000 7fcb177fe640 1 -- 172.21.15.57:0/755641426 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb10001f30 con 0x7fcb2013bf30 2023-10-19T13:04:52.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:52.872+0000 7fcb177fe640 1 -- 172.21.15.57:0/755641426 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fcb1000c9e0 con 0x7fcb2013bf30 2023-10-19T13:04:52.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:52.872+0000 7fcb2553e640 1 -- 172.21.15.57:0/755641426 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb2014c350 con 0x7fcb2013bf30 2023-10-19T13:04:52.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:52.872+0000 7fcb177fe640 1 -- 172.21.15.57:0/755641426 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb1000cdf0 con 0x7fcb2013bf30 2023-10-19T13:04:52.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:52.872+0000 7fcb2553e640 1 -- 172.21.15.57:0/755641426 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb20154000 con 0x7fcb2013bf30 2023-10-19T13:04:52.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:52.873+0000 7fcb2553e640 1 -- 172.21.15.57:0/755641426 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb20070d80 con 0x7fcb2013bf30 2023-10-19T13:04:52.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:52.873+0000 7fcb177fe640 1 -- 172.21.15.57:0/755641426 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 79) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fcb10011460 con 0x7fcb2013bf30 2023-10-19T13:04:52.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:52.874+0000 7fcb177fe640 1 --2- 172.21.15.57:0/755641426 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fcb0006af90 0x7fcb0006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:52.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:52.874+0000 7fcb177fe640 1 -- 172.21.15.57:0/755641426 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6022+0+0 (secure 0 0 0) 0x7fcb1008a130 con 0x7fcb2013bf30 2023-10-19T13:04:52.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:52.874+0000 7fcb1effd640 1 --2- 172.21.15.57:0/755641426 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fcb0006af90 0x7fcb0006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:52.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:52.875+0000 7fcb1effd640 1 --2- 172.21.15.57:0/755641426 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fcb0006af90 0x7fcb0006d450 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fcb18000c00 tx=0x7fcb18006ae0 comp rx=0 tx=0).ready entity=mgr.8419 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:52.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:52.878+0000 7fcb177fe640 1 -- 172.21.15.57:0/755641426 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fcb10054e80 con 0x7fcb2013bf30 2023-10-19T13:04:53.057 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.054+0000 7fcb2553e640 1 -- 172.21.15.57:0/755641426 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "config dump"} v 0) v1 -- 0x7fcb200690e0 con 0x7fcb2013bf30 2023-10-19T13:04:53.057 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.054+0000 7fcb177fe640 1 -- 172.21.15.57:0/755641426 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v58) v1 ==== 59+0+2220 (secure 0 0 0) 0x7fcb1001f8c0 con 0x7fcb2013bf30 2023-10-19T13:04:53.058 INFO:teuthology.orchestra.run.smithi057.stdout:WHO MASK LEVEL OPTION VALUE RO 2023-10-19T13:04:53.058 INFO:teuthology.orchestra.run.smithi057.stdout:mon advanced mon_op_complaint_time 120 2023-10-19T13:04:53.058 INFO:teuthology.orchestra.run.smithi057.stdout:osd advanced osd_op_complaint_time 180.000000 2023-10-19T13:04:53.059 INFO:teuthology.orchestra.run.smithi057.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 4775.995373 2023-10-19T13:04:53.059 INFO:teuthology.orchestra.run.smithi057.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 4208.118410 2023-10-19T13:04:53.059 INFO:teuthology.orchestra.run.smithi057.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 4541.123030 2023-10-19T13:04:53.059 INFO:teuthology.orchestra.run.smithi057.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 4877.592206 2023-10-19T13:04:53.060 INFO:teuthology.orchestra.run.smithi057.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 5019.748032 2023-10-19T13:04:53.060 INFO:teuthology.orchestra.run.smithi057.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 5000.074376 2023-10-19T13:04:53.060 INFO:teuthology.orchestra.run.smithi057.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 5153.479439 2023-10-19T13:04:53.060 INFO:teuthology.orchestra.run.smithi057.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 4977.363845 2023-10-19T13:04:53.060 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced debug_mds 20/20 2023-10-19T13:04:53.061 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced debug_mds_balancer 20/20 2023-10-19T13:04:53.061 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced debug_ms 1/1 2023-10-19T13:04:53.061 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced mds_bal_fragment_size_max 10000 2023-10-19T13:04:53.061 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced mds_bal_merge_size 5 2023-10-19T13:04:53.062 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced mds_bal_split_bits 3 2023-10-19T13:04:53.062 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced mds_bal_split_size 100 2023-10-19T13:04:53.062 INFO:teuthology.orchestra.run.smithi057.stdout:mds dev mds_debug_frag true 2023-10-19T13:04:53.062 INFO:teuthology.orchestra.run.smithi057.stdout:mds dev mds_debug_scatterstat true 2023-10-19T13:04:53.062 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced mds_op_complaint_time 180.000000 2023-10-19T13:04:53.063 INFO:teuthology.orchestra.run.smithi057.stdout:mds dev mds_verify_scatter true 2023-10-19T13:04:53.063 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced osd_op_complaint_time 180.000000 2023-10-19T13:04:53.063 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced rados_mon_op_timeout 900 2023-10-19T13:04:53.063 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced rados_osd_op_timeout 900 2023-10-19T13:04:53.064 INFO:teuthology.orchestra.run.smithi057.stdout:client advanced client_mount_timeout 600 2023-10-19T13:04:53.064 INFO:teuthology.orchestra.run.smithi057.stdout:client advanced debug_client 20/20 2023-10-19T13:04:53.064 INFO:teuthology.orchestra.run.smithi057.stdout:client advanced debug_ms 1/1 2023-10-19T13:04:53.064 INFO:teuthology.orchestra.run.smithi057.stdout:client advanced rados_mon_op_timeout 900 2023-10-19T13:04:53.064 INFO:teuthology.orchestra.run.smithi057.stdout:client advanced rados_osd_op_timeout 900 2023-10-19T13:04:53.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.056+0000 7fcb2553e640 1 -- 172.21.15.57:0/755641426 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fcb0006af90 msgr2=0x7fcb0006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:53.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.056+0000 7fcb2553e640 1 --2- 172.21.15.57:0/755641426 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fcb0006af90 0x7fcb0006d450 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fcb18000c00 tx=0x7fcb18006ae0 comp rx=0 tx=0).stop 2023-10-19T13:04:53.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.056+0000 7fcb2553e640 1 -- 172.21.15.57:0/755641426 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcb2013bf30 msgr2=0x7fcb200cbe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:53.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.056+0000 7fcb2553e640 1 --2- 172.21.15.57:0/755641426 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcb2013bf30 0x7fcb200cbe00 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7fcb10002230 tx=0x7fcb1000eb00 comp rx=0 tx=0).stop 2023-10-19T13:04:53.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.056+0000 7fcb2553e640 1 -- 172.21.15.57:0/755641426 shutdown_connections 2023-10-19T13:04:53.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.056+0000 7fcb2553e640 1 --2- 172.21.15.57:0/755641426 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fcb0006af90 0x7fcb0006d450 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:53.067 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.056+0000 7fcb2553e640 1 --2- 172.21.15.57:0/755641426 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcb2013bf30 0x7fcb200cbe00 unknown :-1 s=CLOSED pgs=1715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:53.067 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.056+0000 7fcb2553e640 1 --2- 172.21.15.57:0/755641426 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcb2014b8f0 0x7fcb200cb8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:53.067 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.056+0000 7fcb2553e640 1 --2- 172.21.15.57:0/755641426 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcb0c0047c0 0x7fcb2014b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:53.067 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.056+0000 7fcb2553e640 1 -- 172.21.15.57:0/755641426 >> 172.21.15.57:0/755641426 conn(0x7fcb2006fae0 msgr2=0x7fcb20158620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:53.067 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.056+0000 7fcb2553e640 1 -- 172.21.15.57:0/755641426 shutdown_connections 2023-10-19T13:04:53.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.056+0000 7fcb2553e640 1 -- 172.21.15.57:0/755641426 wait complete. 2023-10-19T13:04:53.070 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-19T13:04:53.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.180+0000 7f0e2f44b640 1 -- 172.21.15.57:0/1033046077 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0e280cd1c0 msgr2=0x7f0e2815c0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:53.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.180+0000 7f0e2f44b640 1 --2- 172.21.15.57:0/1033046077 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0e280cd1c0 0x7f0e2815c0e0 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7f0e28073460 tx=0x7f0e1802be40 comp rx=0 tx=0).stop 2023-10-19T13:04:53.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.180+0000 7f0e2f44b640 1 -- 172.21.15.57:0/1033046077 shutdown_connections 2023-10-19T13:04:53.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.180+0000 7f0e2f44b640 1 --2- 172.21.15.57:0/1033046077 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0e280cd1c0 0x7f0e2815c0e0 unknown :-1 s=CLOSED pgs=1717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:53.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.180+0000 7f0e2f44b640 1 --2- 172.21.15.57:0/1033046077 >> v2:172.21.15.179:3300/0 conn(0x7f0e28158910 0x7f0e28158cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:53.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.180+0000 7f0e2f44b640 1 -- 172.21.15.57:0/1033046077 >> 172.21.15.57:0/1033046077 conn(0x7f0e280af7a0 msgr2=0x7f0e280afb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:53.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.180+0000 7f0e2f44b640 1 -- 172.21.15.57:0/1033046077 shutdown_connections 2023-10-19T13:04:53.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.180+0000 7f0e2f44b640 1 -- 172.21.15.57:0/1033046077 wait complete. 2023-10-19T13:04:53.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.181+0000 7f0e2f44b640 1 Processor -- start 2023-10-19T13:04:53.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.181+0000 7f0e2f44b640 1 -- start start 2023-10-19T13:04:53.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.182+0000 7f0e2f44b640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0e280cd1c0 0x7f0e281e3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:53.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.182+0000 7f0e2f44b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0e280cd9e0 0x7f0e2813ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:53.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.182+0000 7f0e2f44b640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0e28158910 0x7f0e2813b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:53.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.182+0000 7f0e2f44b640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e2813dcf0 con 0x7f0e280cd9e0 2023-10-19T13:04:53.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.182+0000 7f0e2f44b640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e281e30e0 con 0x7f0e280cd1c0 2023-10-19T13:04:53.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.182+0000 7f0e2f44b640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e281e33c0 con 0x7f0e28158910 2023-10-19T13:04:53.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.182+0000 7f0e2c9e6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0e280cd9e0 0x7f0e2813ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:53.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.182+0000 7f0e2c9e6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0e280cd9e0 0x7f0e2813ae90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38466/0 (socket says 172.21.15.57:38466) 2023-10-19T13:04:53.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.182+0000 7f0e2c9e6640 1 -- 172.21.15.57:0/639621322 learned_addr learned my addr 172.21.15.57:0/639621322 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:53.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.182+0000 7f0e2d1e7640 1 --2- 172.21.15.57:0/639621322 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0e280cd1c0 0x7f0e281e3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:53.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.182+0000 7f0e2d9e8640 1 --2- 172.21.15.57:0/639621322 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0e28158910 0x7f0e2813b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:53.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.182+0000 7f0e2c9e6640 1 -- 172.21.15.57:0/639621322 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0e28158910 msgr2=0x7f0e2813b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:53.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.182+0000 7f0e2c9e6640 1 --2- 172.21.15.57:0/639621322 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0e28158910 0x7f0e2813b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:53.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.182+0000 7f0e2c9e6640 1 -- 172.21.15.57:0/639621322 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0e280cd1c0 msgr2=0x7f0e281e3bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:53.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.182+0000 7f0e2c9e6640 1 --2- 172.21.15.57:0/639621322 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0e280cd1c0 0x7f0e281e3bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:53.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.182+0000 7f0e2c9e6640 1 -- 172.21.15.57:0/639621322 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e18009560 con 0x7f0e280cd9e0 2023-10-19T13:04:53.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.182+0000 7f0e2d9e8640 1 --2- 172.21.15.57:0/639621322 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0e28158910 0x7f0e2813b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:04:53.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.182+0000 7f0e2c9e6640 1 --2- 172.21.15.57:0/639621322 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0e280cd9e0 0x7f0e2813ae90 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7f0e18009850 tx=0x7f0e1802db70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:53.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.182+0000 7f0e1dffb640 1 -- 172.21.15.57:0/639621322 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e18003800 con 0x7f0e280cd9e0 2023-10-19T13:04:53.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.182+0000 7f0e1dffb640 1 -- 172.21.15.57:0/639621322 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f0e18003a50 con 0x7f0e280cd9e0 2023-10-19T13:04:53.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.182+0000 7f0e1dffb640 1 -- 172.21.15.57:0/639621322 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e18003e20 con 0x7f0e280cd9e0 2023-10-19T13:04:53.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.182+0000 7f0e2f44b640 1 -- 172.21.15.57:0/639621322 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e281e36a0 con 0x7f0e280cd9e0 2023-10-19T13:04:53.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.183+0000 7f0e2f44b640 1 -- 172.21.15.57:0/639621322 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e28140fa0 con 0x7f0e280cd9e0 2023-10-19T13:04:53.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.184+0000 7f0e2f44b640 1 -- 172.21.15.57:0/639621322 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e28141280 con 0x7f0e280cd9e0 2023-10-19T13:04:53.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.184+0000 7f0e1dffb640 1 -- 172.21.15.57:0/639621322 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 79) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f0e1803e450 con 0x7f0e280cd9e0 2023-10-19T13:04:53.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.185+0000 7f0e1dffb640 1 --2- 172.21.15.57:0/639621322 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f0e0c06af60 0x7f0e0c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:53.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.185+0000 7f0e1dffb640 1 -- 172.21.15.57:0/639621322 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6022+0+0 (secure 0 0 0) 0x7f0e180ac3b0 con 0x7f0e280cd9e0 2023-10-19T13:04:53.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.185+0000 7f0e2d1e7640 1 --2- 172.21.15.57:0/639621322 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f0e0c06af60 0x7f0e0c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:53.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.185+0000 7f0e2d1e7640 1 --2- 172.21.15.57:0/639621322 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f0e0c06af60 0x7f0e0c06d420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0e24005fd0 tx=0x7f0e2400b2e0 comp rx=0 tx=0).ready entity=mgr.8419 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:53.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.189+0000 7f0e1dffb640 1 -- 172.21.15.57:0/639621322 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0e18077210 con 0x7f0e280cd9e0 2023-10-19T13:04:53.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.365+0000 7f0e2f44b640 1 -- 172.21.15.57:0/639621322 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f0e281e38e0 con 0x7f0e280cd9e0 2023-10-19T13:04:53.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.369+0000 7f0e1dffb640 1 -- 172.21.15.57:0/639621322 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v59)=0 v59) v1 ==== 122+0+0 (secure 0 0 0) 0x7f0e281e38e0 con 0x7f0e280cd9e0 2023-10-19T13:04:53.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.371+0000 7f0e2f44b640 1 -- 172.21.15.57:0/639621322 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f0e0c06af60 msgr2=0x7f0e0c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:53.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.371+0000 7f0e2f44b640 1 --2- 172.21.15.57:0/639621322 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f0e0c06af60 0x7f0e0c06d420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0e24005fd0 tx=0x7f0e2400b2e0 comp rx=0 tx=0).stop 2023-10-19T13:04:53.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.371+0000 7f0e2f44b640 1 -- 172.21.15.57:0/639621322 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0e280cd9e0 msgr2=0x7f0e2813ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:53.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.371+0000 7f0e2f44b640 1 --2- 172.21.15.57:0/639621322 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0e280cd9e0 0x7f0e2813ae90 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7f0e18009850 tx=0x7f0e1802db70 comp rx=0 tx=0).stop 2023-10-19T13:04:53.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.371+0000 7f0e2f44b640 1 -- 172.21.15.57:0/639621322 shutdown_connections 2023-10-19T13:04:53.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.371+0000 7f0e2f44b640 1 --2- 172.21.15.57:0/639621322 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f0e0c06af60 0x7f0e0c06d420 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:53.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.371+0000 7f0e2f44b640 1 --2- 172.21.15.57:0/639621322 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0e28158910 0x7f0e2813b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:53.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.371+0000 7f0e2f44b640 1 --2- 172.21.15.57:0/639621322 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0e280cd9e0 0x7f0e2813ae90 unknown :-1 s=CLOSED pgs=1718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:53.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.371+0000 7f0e2f44b640 1 --2- 172.21.15.57:0/639621322 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0e280cd1c0 0x7f0e281e3bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:53.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.371+0000 7f0e2f44b640 1 -- 172.21.15.57:0/639621322 >> 172.21.15.57:0/639621322 conn(0x7f0e280af7a0 msgr2=0x7f0e2806db10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:53.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.371+0000 7f0e2f44b640 1 -- 172.21.15.57:0/639621322 shutdown_connections 2023-10-19T13:04:53.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.371+0000 7f0e2f44b640 1 -- 172.21.15.57:0/639621322 wait complete. 2023-10-19T13:04:53.385 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-19T13:04:53.385 DEBUG:teuthology.orchestra.run.smithi057:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-19T13:04:53.435 DEBUG:teuthology.orchestra.run.smithi179:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-19T13:04:53.484 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-19T13:04:53.485 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T13:04:53.485 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T13:04:53.500 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055614,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=358 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248260k,nr_inodes=812065,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-19T13:04:53.501 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-19T13:04:53.501 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T13:04:53.518 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054833,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=284 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3247632k,nr_inodes=811908,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-19T13:04:53.519 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T13:04:53.519 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T13:04:53.555 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055614,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=358 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248260k,nr_inodes=812065,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-19T13:04:53.556 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-19T13:04:53.556 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T13:04:53.574 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054833,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=284 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3247632k,nr_inodes=811908,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-19T13:04:53.575 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-19T13:04:53.575 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T13:04:53.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.701+0000 7fb040464640 1 -- 172.21.15.57:0/453229956 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb0380ceb40 msgr2=0x7fb0380b0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:53.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.701+0000 7fb040464640 1 --2- 172.21.15.57:0/453229956 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb0380ceb40 0x7fb0380b0650 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7fb03808a810 tx=0x7fb03002bcf0 comp rx=0 tx=0).stop 2023-10-19T13:04:53.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.702+0000 7fb040464640 1 -- 172.21.15.57:0/453229956 shutdown_connections 2023-10-19T13:04:53.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.702+0000 7fb040464640 1 --2- 172.21.15.57:0/453229956 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb0380ceb40 0x7fb0380b0650 unknown :-1 s=CLOSED pgs=1720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:53.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.702+0000 7fb040464640 1 --2- 172.21.15.57:0/453229956 >> v2:172.21.15.179:3300/0 conn(0x7fb0380cf340 0x7fb0380d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:53.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.702+0000 7fb040464640 1 -- 172.21.15.57:0/453229956 >> 172.21.15.57:0/453229956 conn(0x7fb0380d0d60 msgr2=0x7fb0380d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:53.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.702+0000 7fb040464640 1 -- 172.21.15.57:0/453229956 shutdown_connections 2023-10-19T13:04:53.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.702+0000 7fb040464640 1 -- 172.21.15.57:0/453229956 wait complete. 2023-10-19T13:04:53.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.703+0000 7fb040464640 1 Processor -- start 2023-10-19T13:04:53.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.703+0000 7fb040464640 1 -- start start 2023-10-19T13:04:53.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.703+0000 7fb040464640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb0380ceb40 0x7fb03815dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:53.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.703+0000 7fb040464640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb0380cf340 0x7fb0381e52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:53.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.703+0000 7fb040464640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb03815b2a0 0x7fb0381e5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:53.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.703+0000 7fb040464640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb038158440 con 0x7fb0380ceb40 2023-10-19T13:04:53.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.703+0000 7fb040464640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb038158720 con 0x7fb0380cf340 2023-10-19T13:04:53.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.703+0000 7fb040464640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0380c7ed0 con 0x7fb03815b2a0 2023-10-19T13:04:53.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.704+0000 7fb03e200640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb0380ceb40 0x7fb03815dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:53.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.704+0000 7fb03e200640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb0380ceb40 0x7fb03815dc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38486/0 (socket says 172.21.15.57:38486) 2023-10-19T13:04:53.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.704+0000 7fb03e200640 1 -- 172.21.15.57:0/758805236 learned_addr learned my addr 172.21.15.57:0/758805236 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:53.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.704+0000 7fb03d9ff640 1 --2- 172.21.15.57:0/758805236 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb0380cf340 0x7fb0381e52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:53.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.704+0000 7fb03ea01640 1 --2- 172.21.15.57:0/758805236 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb03815b2a0 0x7fb0381e5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:53.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.704+0000 7fb03e200640 1 -- 172.21.15.57:0/758805236 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb03815b2a0 msgr2=0x7fb0381e5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:53.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.704+0000 7fb03e200640 1 --2- 172.21.15.57:0/758805236 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb03815b2a0 0x7fb0381e5810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:53.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.704+0000 7fb03e200640 1 -- 172.21.15.57:0/758805236 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb0380cf340 msgr2=0x7fb0381e52d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:53.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.704+0000 7fb03e200640 1 --2- 172.21.15.57:0/758805236 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb0380cf340 0x7fb0381e52d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:53.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.704+0000 7fb03e200640 1 -- 172.21.15.57:0/758805236 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb030009490 con 0x7fb0380ceb40 2023-10-19T13:04:53.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.704+0000 7fb03d9ff640 1 --2- 172.21.15.57:0/758805236 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb0380cf340 0x7fb0381e52d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:04:53.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.704+0000 7fb03e200640 1 --2- 172.21.15.57:0/758805236 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb0380ceb40 0x7fb03815dc20 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7fb02800db00 tx=0x7fb02800b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:53.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.704+0000 7fb036ffd640 1 -- 172.21.15.57:0/758805236 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb02800bea0 con 0x7fb0380ceb40 2023-10-19T13:04:53.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.704+0000 7fb036ffd640 1 -- 172.21.15.57:0/758805236 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fb028003a60 con 0x7fb0380ceb40 2023-10-19T13:04:53.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.704+0000 7fb040464640 1 -- 172.21.15.57:0/758805236 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0380c81b0 con 0x7fb0380ceb40 2023-10-19T13:04:53.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.704+0000 7fb036ffd640 1 -- 172.21.15.57:0/758805236 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb028003e70 con 0x7fb0380ceb40 2023-10-19T13:04:53.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.704+0000 7fb040464640 1 -- 172.21.15.57:0/758805236 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb03815e160 con 0x7fb0380ceb40 2023-10-19T13:04:53.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.705+0000 7fb040464640 1 -- 172.21.15.57:0/758805236 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb03815e3e0 con 0x7fb0380ceb40 2023-10-19T13:04:53.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.706+0000 7fb036ffd640 1 -- 172.21.15.57:0/758805236 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 79) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb028004020 con 0x7fb0380ceb40 2023-10-19T13:04:53.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.706+0000 7fb036ffd640 1 --2- 172.21.15.57:0/758805236 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fb02406af60 0x7fb02406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:53.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.706+0000 7fb036ffd640 1 -- 172.21.15.57:0/758805236 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6022+0+0 (secure 0 0 0) 0x7fb02808a200 con 0x7fb0380ceb40 2023-10-19T13:04:53.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.707+0000 7fb03d9ff640 1 --2- 172.21.15.57:0/758805236 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fb02406af60 0x7fb02406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:53.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.707+0000 7fb03d9ff640 1 --2- 172.21.15.57:0/758805236 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fb02406af60 0x7fb02406d420 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb0380ce980 tx=0x7fb02c009040 comp rx=0 tx=0).ready entity=mgr.8419 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:53.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.711+0000 7fb036ffd640 1 -- 172.21.15.57:0/758805236 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb028054f40 con 0x7fb0380ceb40 2023-10-19T13:04:53.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.921+0000 7fb040464640 1 -- 172.21.15.57:0/758805236 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb0380b0d00 con 0x7fb0380ceb40 2023-10-19T13:04:53.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.922+0000 7fb036ffd640 1 -- 172.21.15.57:0/758805236 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+3705 (secure 0 0 0) 0x7fb028017e90 con 0x7fb0380ceb40 2023-10-19T13:04:53.925 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:04:53.925 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":58,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7325,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/3318110028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":3318110028},{"type":"v1","addr":"172.21.15.179:6837","nonce":3318110028}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":8143,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/3625556517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":3625556517},{"type":"v1","addr":"172.21.15.57:6835","nonce":3625556517}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":8158,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/4167860955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":4167860955},{"type":"v1","addr":"172.21.15.179:6836","nonce":4167860955}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":53},{"gid":9531,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/1165579027","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":1165579027},{"type":"v1","addr":"172.21.15.57:6834","nonce":1165579027}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":58}],"filesystems":[]} 2023-10-19T13:04:53.925 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 58 2023-10-19T13:04:53.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.923+0000 7fb040464640 1 -- 172.21.15.57:0/758805236 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fb02406af60 msgr2=0x7fb02406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:53.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.923+0000 7fb040464640 1 --2- 172.21.15.57:0/758805236 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fb02406af60 0x7fb02406d420 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb0380ce980 tx=0x7fb02c009040 comp rx=0 tx=0).stop 2023-10-19T13:04:53.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.923+0000 7fb040464640 1 -- 172.21.15.57:0/758805236 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb0380ceb40 msgr2=0x7fb03815dc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:53.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.923+0000 7fb040464640 1 --2- 172.21.15.57:0/758805236 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb0380ceb40 0x7fb03815dc20 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7fb02800db00 tx=0x7fb02800b260 comp rx=0 tx=0).stop 2023-10-19T13:04:53.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.923+0000 7fb040464640 1 -- 172.21.15.57:0/758805236 shutdown_connections 2023-10-19T13:04:53.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.923+0000 7fb040464640 1 --2- 172.21.15.57:0/758805236 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fb02406af60 0x7fb02406d420 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:53.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.923+0000 7fb040464640 1 --2- 172.21.15.57:0/758805236 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb03815b2a0 0x7fb0381e5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:53.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.923+0000 7fb040464640 1 --2- 172.21.15.57:0/758805236 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb0380cf340 0x7fb0381e52d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:53.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.923+0000 7fb040464640 1 --2- 172.21.15.57:0/758805236 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb0380ceb40 0x7fb03815dc20 unknown :-1 s=CLOSED pgs=1721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:53.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.923+0000 7fb040464640 1 -- 172.21.15.57:0/758805236 >> 172.21.15.57:0/758805236 conn(0x7fb0380d0d60 msgr2=0x7fb03815a2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:53.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.924+0000 7fb040464640 1 -- 172.21.15.57:0/758805236 shutdown_connections 2023-10-19T13:04:53.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:53.924+0000 7fb040464640 1 -- 172.21.15.57:0/758805236 wait complete. 2023-10-19T13:04:53.937 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-19T13:04:53.937 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-19T13:04:53.938 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-19T13:04:53.938 INFO:teuthology.orchestra.run:waiting for 300 2023-10-19T13:04:53.938 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-19T13:04:53.939 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-19T13:04:53.939 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-19T13:04:53.939 INFO:teuthology.orchestra.run:waiting for 300 2023-10-19T13:04:53.939 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-19T13:04:53.940 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-19T13:04:53.940 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-19T13:04:53.940 INFO:teuthology.orchestra.run:waiting for 300 2023-10-19T13:04:53.940 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-19T13:04:53.941 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-19T13:04:53.941 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-19T13:04:53.941 INFO:teuthology.orchestra.run:waiting for 300 2023-10-19T13:04:53.946 INFO:tasks.ceph.mds.a:Stopped 2023-10-19T13:04:53.947 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-19T13:04:53.948 INFO:tasks.ceph.mds.c:Stopped 2023-10-19T13:04:53.948 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-19T13:04:53.952 INFO:tasks.ceph.mds.b:Stopped 2023-10-19T13:04:53.952 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-19T13:04:53.953 INFO:tasks.ceph.mds.d:Stopped 2023-10-19T13:04:53.953 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-19T13:04:53.990 INFO:tasks.ceph.mds.a:Started 2023-10-19T13:04:53.991 DEBUG:teuthology.parallel:result is None 2023-10-19T13:04:53.991 INFO:tasks.ceph.mds.c:Started 2023-10-19T13:04:53.992 DEBUG:teuthology.parallel:result is None 2023-10-19T13:04:53.994 INFO:tasks.ceph.mds.b:Started 2023-10-19T13:04:53.995 DEBUG:teuthology.parallel:result is None 2023-10-19T13:04:53.995 INFO:tasks.ceph.mds.d:Started 2023-10-19T13:04:53.996 DEBUG:teuthology.parallel:result is None 2023-10-19T13:04:53.996 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-19T13:04:54.071 INFO:tasks.ceph.mds.c.smithi057.stdout:starting mds.c at 2023-10-19T13:04:54.072 INFO:tasks.ceph.mds.a.smithi057.stdout:starting mds.a at 2023-10-19T13:04:54.079 INFO:tasks.ceph.mds.b.smithi179.stdout:starting mds.b at 2023-10-19T13:04:54.079 INFO:tasks.ceph.mds.d.smithi179.stdout:starting mds.d at 2023-10-19T13:04:54.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.109+0000 7f3a77e94640 1 -- 172.21.15.57:0/484347467 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3a700d1910 msgr2=0x7f3a700b0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:54.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.109+0000 7f3a77e94640 1 --2- 172.21.15.57:0/484347467 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3a700d1910 0x7f3a700b0650 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f3a68009830 tx=0x7f3a6802bd60 comp rx=0 tx=0).stop 2023-10-19T13:04:54.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.109+0000 7f3a77e94640 1 -- 172.21.15.57:0/484347467 shutdown_connections 2023-10-19T13:04:54.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.109+0000 7f3a77e94640 1 --2- 172.21.15.57:0/484347467 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3a700d1910 0x7f3a700b0650 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:54.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.109+0000 7f3a77e94640 1 -- 172.21.15.57:0/484347467 >> 172.21.15.57:0/484347467 conn(0x7f3a700d0d60 msgr2=0x7f3a700d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:54.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.109+0000 7f3a77e94640 1 -- 172.21.15.57:0/484347467 shutdown_connections 2023-10-19T13:04:54.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.109+0000 7f3a77e94640 1 -- 172.21.15.57:0/484347467 wait complete. 2023-10-19T13:04:54.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.110+0000 7f3a77e94640 1 Processor -- start 2023-10-19T13:04:54.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.110+0000 7f3a77e94640 1 -- start start 2023-10-19T13:04:54.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.111+0000 7f3a77e94640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3a700cf340 0x7f3a700b4270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:54.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.111+0000 7f3a77e94640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3a700d1910 0x7f3a7014f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:54.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.111+0000 7f3a77e94640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3a7015b2a0 0x7f3a700ae280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:54.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.111+0000 7f3a77e94640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a700b3f40 con 0x7f3a700cf340 2023-10-19T13:04:54.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.111+0000 7f3a77e94640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a700b3980 con 0x7f3a700d1910 2023-10-19T13:04:54.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.111+0000 7f3a77e94640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a700b3c60 con 0x7f3a7015b2a0 2023-10-19T13:04:54.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.111+0000 7f3a75c30640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3a700cf340 0x7f3a700b4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:54.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.111+0000 7f3a75c30640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3a700cf340 0x7f3a700b4270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38520/0 (socket says 172.21.15.57:38520) 2023-10-19T13:04:54.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.111+0000 7f3a75c30640 1 -- 172.21.15.57:0/3366494084 learned_addr learned my addr 172.21.15.57:0/3366494084 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:54.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.111+0000 7f3a7542f640 1 --2- 172.21.15.57:0/3366494084 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3a700d1910 0x7f3a7014f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:54.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.111+0000 7f3a76431640 1 --2- 172.21.15.57:0/3366494084 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3a7015b2a0 0x7f3a700ae280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:54.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.111+0000 7f3a75c30640 1 -- 172.21.15.57:0/3366494084 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3a7015b2a0 msgr2=0x7f3a700ae280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:54.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.111+0000 7f3a75c30640 1 --2- 172.21.15.57:0/3366494084 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3a7015b2a0 0x7f3a700ae280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:54.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.111+0000 7f3a75c30640 1 -- 172.21.15.57:0/3366494084 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3a700d1910 msgr2=0x7f3a7014f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:54.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.111+0000 7f3a75c30640 1 --2- 172.21.15.57:0/3366494084 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3a700d1910 0x7f3a7014f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:54.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.111+0000 7f3a75c30640 1 -- 172.21.15.57:0/3366494084 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a68009490 con 0x7f3a700cf340 2023-10-19T13:04:54.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.112+0000 7f3a75c30640 1 --2- 172.21.15.57:0/3366494084 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3a700cf340 0x7f3a700b4270 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7f3a6000cbc0 tx=0x7f3a6000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:54.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.112+0000 7f3a6e7fc640 1 -- 172.21.15.57:0/3366494084 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a6000ad40 con 0x7f3a700cf340 2023-10-19T13:04:54.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.112+0000 7f3a6e7fc640 1 -- 172.21.15.57:0/3366494084 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f3a60012420 con 0x7f3a700cf340 2023-10-19T13:04:54.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.112+0000 7f3a77e94640 1 -- 172.21.15.57:0/3366494084 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a701f29e0 con 0x7f3a700cf340 2023-10-19T13:04:54.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.112+0000 7f3a6e7fc640 1 -- 172.21.15.57:0/3366494084 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a60012810 con 0x7f3a700cf340 2023-10-19T13:04:54.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.112+0000 7f3a77e94640 1 -- 172.21.15.57:0/3366494084 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a700b47b0 con 0x7f3a700cf340 2023-10-19T13:04:54.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.113+0000 7f3a77e94640 1 -- 172.21.15.57:0/3366494084 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a700690e0 con 0x7f3a700cf340 2023-10-19T13:04:54.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.113+0000 7f3a6e7fc640 1 -- 172.21.15.57:0/3366494084 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 79) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f3a6001b400 con 0x7f3a700cf340 2023-10-19T13:04:54.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.114+0000 7f3a6e7fc640 1 --2- 172.21.15.57:0/3366494084 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f3a5c06af60 0x7f3a5c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:54.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.114+0000 7f3a6e7fc640 1 -- 172.21.15.57:0/3366494084 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6022+0+0 (secure 0 0 0) 0x7f3a6008a4d0 con 0x7f3a700cf340 2023-10-19T13:04:54.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.114+0000 7f3a7542f640 1 --2- 172.21.15.57:0/3366494084 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f3a5c06af60 0x7f3a5c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:54.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.115+0000 7f3a7542f640 1 --2- 172.21.15.57:0/3366494084 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f3a5c06af60 0x7f3a5c06d420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f3a7015e7d0 tx=0x7f3a64009040 comp rx=0 tx=0).ready entity=mgr.8419 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:54.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.118+0000 7f3a6e7fc640 1 -- 172.21.15.57:0/3366494084 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3a600552d0 con 0x7f3a700cf340 2023-10-19T13:04:54.313 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.310+0000 7f3a77e94640 1 -- 172.21.15.57:0/3366494084 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f3a700d01d0 con 0x7f3a700cf340 2023-10-19T13:04:54.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.752+0000 7f3a6e7fc640 1 -- 172.21.15.57:0/3366494084 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v151) v1 ==== 97+0+0 (secure 0 0 0) 0x7f3a600588f0 con 0x7f3a700cf340 2023-10-19T13:04:54.755 INFO:teuthology.orchestra.run.smithi057.stderr: removed all blocklist entries 2023-10-19T13:04:54.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.754+0000 7f3a77e94640 1 -- 172.21.15.57:0/3366494084 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f3a5c06af60 msgr2=0x7f3a5c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:54.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.754+0000 7f3a77e94640 1 --2- 172.21.15.57:0/3366494084 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f3a5c06af60 0x7f3a5c06d420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f3a7015e7d0 tx=0x7f3a64009040 comp rx=0 tx=0).stop 2023-10-19T13:04:54.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.754+0000 7f3a77e94640 1 -- 172.21.15.57:0/3366494084 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3a700cf340 msgr2=0x7f3a700b4270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:54.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.754+0000 7f3a77e94640 1 --2- 172.21.15.57:0/3366494084 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3a700cf340 0x7f3a700b4270 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7f3a6000cbc0 tx=0x7f3a6000a430 comp rx=0 tx=0).stop 2023-10-19T13:04:54.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.754+0000 7f3a77e94640 1 -- 172.21.15.57:0/3366494084 shutdown_connections 2023-10-19T13:04:54.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.754+0000 7f3a77e94640 1 --2- 172.21.15.57:0/3366494084 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f3a5c06af60 0x7f3a5c06d420 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:54.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.754+0000 7f3a77e94640 1 --2- 172.21.15.57:0/3366494084 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3a7015b2a0 0x7f3a700ae280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:54.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.754+0000 7f3a77e94640 1 --2- 172.21.15.57:0/3366494084 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3a700d1910 0x7f3a7014f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:54.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.754+0000 7f3a77e94640 1 --2- 172.21.15.57:0/3366494084 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3a700cf340 0x7f3a700b4270 unknown :-1 s=CLOSED pgs=1729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:54.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.754+0000 7f3a77e94640 1 -- 172.21.15.57:0/3366494084 >> 172.21.15.57:0/3366494084 conn(0x7f3a700d0d60 msgr2=0x7f3a700cc9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:54.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.754+0000 7f3a77e94640 1 -- 172.21.15.57:0/3366494084 shutdown_connections 2023-10-19T13:04:54.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.754+0000 7f3a77e94640 1 -- 172.21.15.57:0/3366494084 wait complete. 2023-10-19T13:04:54.771 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-19T13:04:54.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.880+0000 7f560ffc5640 1 -- 172.21.15.57:0/1681276588 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5608070350 msgr2=0x7f5608070780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:54.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.880+0000 7f560ffc5640 1 --2- 172.21.15.57:0/1681276588 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5608070350 0x7f5608070780 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7f56080cc560 tx=0x7f560002c260 comp rx=0 tx=0).stop 2023-10-19T13:04:54.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.880+0000 7f560ffc5640 1 -- 172.21.15.57:0/1681276588 shutdown_connections 2023-10-19T13:04:54.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.880+0000 7f560ffc5640 1 --2- 172.21.15.57:0/1681276588 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5608070350 0x7f5608070780 unknown :-1 s=CLOSED pgs=1730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:54.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.880+0000 7f560ffc5640 1 --2- 172.21.15.57:0/1681276588 >> v2:172.21.15.179:3300/0 conn(0x7f56080cb980 0x7f560813db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:54.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.880+0000 7f560ffc5640 1 -- 172.21.15.57:0/1681276588 >> 172.21.15.57:0/1681276588 conn(0x7f5608069930 msgr2=0x7f5608069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:54.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.881+0000 7f560ffc5640 1 -- 172.21.15.57:0/1681276588 shutdown_connections 2023-10-19T13:04:54.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.881+0000 7f560ffc5640 1 -- 172.21.15.57:0/1681276588 wait complete. 2023-10-19T13:04:54.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.882+0000 7f560ffc5640 1 Processor -- start 2023-10-19T13:04:54.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.882+0000 7f560ffc5640 1 -- start start 2023-10-19T13:04:54.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.883+0000 7f560ffc5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5608070350 0x7f5608139680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:54.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.883+0000 7f560ffc5640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f56080cb980 0x7f5608135730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:54.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.883+0000 7f560ffc5640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5608134460 0x7f5608135c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:54.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.883+0000 7f560ffc5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f560813a500 con 0x7f5608070350 2023-10-19T13:04:54.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.883+0000 7f560ffc5640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56080aaa80 con 0x7f5608134460 2023-10-19T13:04:54.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.883+0000 7f560ffc5640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5608139c60 con 0x7f56080cb980 2023-10-19T13:04:54.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.883+0000 7f560dd61640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5608070350 0x7f5608139680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:54.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.883+0000 7f560dd61640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5608070350 0x7f5608139680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38538/0 (socket says 172.21.15.57:38538) 2023-10-19T13:04:54.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.883+0000 7f560dd61640 1 -- 172.21.15.57:0/545628797 learned_addr learned my addr 172.21.15.57:0/545628797 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:54.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.883+0000 7f560e562640 1 --2- 172.21.15.57:0/545628797 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5608134460 0x7f5608135c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:54.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.883+0000 7f560d560640 1 --2- 172.21.15.57:0/545628797 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f56080cb980 0x7f5608135730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:54.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.883+0000 7f560dd61640 1 -- 172.21.15.57:0/545628797 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f56080cb980 msgr2=0x7f5608135730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:54.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.883+0000 7f560dd61640 1 --2- 172.21.15.57:0/545628797 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f56080cb980 0x7f5608135730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:54.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.883+0000 7f560dd61640 1 -- 172.21.15.57:0/545628797 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5608134460 msgr2=0x7f5608135c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:54.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.883+0000 7f560dd61640 1 --2- 172.21.15.57:0/545628797 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5608134460 0x7f5608135c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:54.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.883+0000 7f560dd61640 1 -- 172.21.15.57:0/545628797 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5600009900 con 0x7f5608070350 2023-10-19T13:04:54.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.883+0000 7f560d560640 1 --2- 172.21.15.57:0/545628797 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f56080cb980 0x7f5608135730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T13:04:54.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.883+0000 7f560e562640 1 --2- 172.21.15.57:0/545628797 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5608134460 0x7f5608135c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:04:54.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.883+0000 7f560dd61640 1 --2- 172.21.15.57:0/545628797 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5608070350 0x7f5608139680 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7f55f800d7a0 tx=0x7f55f800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:54.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.883+0000 7f55fe7fc640 1 -- 172.21.15.57:0/545628797 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55f8009bd0 con 0x7f5608070350 2023-10-19T13:04:54.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.883+0000 7f55fe7fc640 1 -- 172.21.15.57:0/545628797 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f55f8009e20 con 0x7f5608070350 2023-10-19T13:04:54.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.883+0000 7f55fe7fc640 1 -- 172.21.15.57:0/545628797 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55f8005280 con 0x7f5608070350 2023-10-19T13:04:54.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.883+0000 7f560ffc5640 1 -- 172.21.15.57:0/545628797 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5608139f40 con 0x7f5608070350 2023-10-19T13:04:54.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.883+0000 7f560ffc5640 1 -- 172.21.15.57:0/545628797 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5608136420 con 0x7f5608070350 2023-10-19T13:04:54.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.884+0000 7f560ffc5640 1 -- 172.21.15.57:0/545628797 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5608136640 con 0x7f5608070350 2023-10-19T13:04:54.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.885+0000 7f55fe7fc640 1 -- 172.21.15.57:0/545628797 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 79) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f55f80053e0 con 0x7f5608070350 2023-10-19T13:04:54.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.885+0000 7f55fe7fc640 1 --2- 172.21.15.57:0/545628797 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f55ec06af90 0x7f55ec06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:54.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.885+0000 7f55fe7fc640 1 -- 172.21.15.57:0/545628797 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f55f80897d0 con 0x7f5608070350 2023-10-19T13:04:54.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.886+0000 7f560d560640 1 --2- 172.21.15.57:0/545628797 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f55ec06af90 0x7f55ec06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:54.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.886+0000 7f560d560640 1 --2- 172.21.15.57:0/545628797 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f55ec06af90 0x7f55ec06d450 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f560806d0d0 tx=0x7f5604008040 comp rx=0 tx=0).ready entity=mgr.8419 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:54.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:54.890+0000 7f55fe7fc640 1 -- 172.21.15.57:0/545628797 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f55f80548e0 con 0x7f5608070350 2023-10-19T13:04:55.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.130+0000 7f560ffc5640 1 -- 172.21.15.57:0/545628797 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f560806b460 con 0x7f5608070350 2023-10-19T13:04:55.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.131+0000 7f55fe7fc640 1 -- 172.21.15.57:0/545628797 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v29) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f55f8017020 con 0x7f5608070350 2023-10-19T13:04:55.134 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:04:55.134 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2023-10-19T13:04:55.135 INFO:teuthology.orchestra.run.smithi057.stdout: "auth_dump": [ 2023-10-19T13:04:55.135 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:04:55.135 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "mds.a", 2023-10-19T13:04:55.135 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBjJDFlmbVANhAAP/ynHlU/OxaRq5lQoAa8Ng==", 2023-10-19T13:04:55.135 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:04:55.136 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T13:04:55.136 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow *", 2023-10-19T13:04:55.136 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow *", 2023-10-19T13:04:55.136 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:04:55.136 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:04:55.137 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:04:55.137 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:04:55.137 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "mds.b", 2023-10-19T13:04:55.137 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFl9l7RBxAAC0geI2tJ1SIZZr1++kS/BA==", 2023-10-19T13:04:55.137 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:04:55.138 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T13:04:55.138 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow *", 2023-10-19T13:04:55.138 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow *", 2023-10-19T13:04:55.138 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:04:55.138 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:04:55.139 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:04:55.139 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:04:55.139 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "mds.c", 2023-10-19T13:04:55.139 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFlFkaoAhAA/PBizkJ4bm958oP4CmPuIQ==", 2023-10-19T13:04:55.139 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:04:55.140 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T13:04:55.140 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow *", 2023-10-19T13:04:55.140 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow *", 2023-10-19T13:04:55.140 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:04:55.141 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:04:55.141 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:04:55.141 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:04:55.141 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "mds.d", 2023-10-19T13:04:55.141 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFlx8skDxAAAfCHTYtYJgJik0WHSWeQfQ==", 2023-10-19T13:04:55.142 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:04:55.142 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T13:04:55.142 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow *", 2023-10-19T13:04:55.142 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow *", 2023-10-19T13:04:55.142 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:04:55.143 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:04:55.143 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:04:55.143 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:04:55.143 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.0", 2023-10-19T13:04:55.143 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBrJDFlCo9sBxAAQN2cID4nCQgRwGaGFEV8LQ==", 2023-10-19T13:04:55.143 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:04:55.144 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T13:04:55.144 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T13:04:55.144 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:04:55.144 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:04:55.144 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:04:55.145 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:04:55.145 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.1", 2023-10-19T13:04:55.145 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBuJDFl4fsLCBAAp3Pi80yZtc5sju0GisEuVA==", 2023-10-19T13:04:55.145 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:04:55.145 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T13:04:55.146 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T13:04:55.146 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:04:55.146 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:04:55.146 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:04:55.146 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:04:55.147 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.2", 2023-10-19T13:04:55.147 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBxJDFl+KAKCBAAd4N9cluARxPVfBLU+7Hd7A==", 2023-10-19T13:04:55.147 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:04:55.147 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T13:04:55.147 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T13:04:55.148 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:04:55.148 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:04:55.148 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:04:55.148 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:04:55.148 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.3", 2023-10-19T13:04:55.149 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQB0JDFlU/toCBAAP0Ou1zcHKNx5SfbybJkh4Q==", 2023-10-19T13:04:55.149 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:04:55.149 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T13:04:55.149 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T13:04:55.149 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:04:55.150 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:04:55.150 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:04:55.150 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:04:55.150 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.4", 2023-10-19T13:04:55.150 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQB6JDFlc6stEBAALkgNyxoiCTzZnKpBSPxI6w==", 2023-10-19T13:04:55.151 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:04:55.151 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T13:04:55.151 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T13:04:55.151 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:04:55.151 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:04:55.152 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:04:55.152 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:04:55.152 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.5", 2023-10-19T13:04:55.152 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQB9JDFlI3cNDhAAuG3tHJ6RvPwAzPE76wTIVQ==", 2023-10-19T13:04:55.152 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:04:55.153 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T13:04:55.153 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T13:04:55.153 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:04:55.153 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:04:55.153 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:04:55.153 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:04:55.154 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.6", 2023-10-19T13:04:55.154 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQCAJDFlLPumCxAA/iy08t/2N9rXfn71KtbSQw==", 2023-10-19T13:04:55.154 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:04:55.154 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T13:04:55.155 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T13:04:55.155 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:04:55.155 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:04:55.155 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:04:55.155 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:04:55.156 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.7", 2023-10-19T13:04:55.156 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQCDJDFlHmrbCRAA+hpWygGIJydfU6cHWlfx/Q==", 2023-10-19T13:04:55.156 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:04:55.156 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T13:04:55.156 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T13:04:55.156 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:04:55.157 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:04:55.157 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:04:55.157 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:04:55.157 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "client.0", 2023-10-19T13:04:55.157 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFlg9yuExAATxiWDfS+p07G1F0pORbJEg==", 2023-10-19T13:04:55.158 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:04:55.158 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T13:04:55.158 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow r", 2023-10-19T13:04:55.158 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-19T13:04:55.158 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:04:55.159 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:04:55.159 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:04:55.159 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "client.1", 2023-10-19T13:04:55.159 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFl9IwYGxAAHPituiJSPkHDuD0QyG4Hww==", 2023-10-19T13:04:55.159 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:04:55.160 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T13:04:55.160 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow r", 2023-10-19T13:04:55.160 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow rw", 2023-10-19T13:04:55.160 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow rwx" 2023-10-19T13:04:55.160 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:04:55.161 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:04:55.161 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:04:55.161 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "client.2", 2023-10-19T13:04:55.161 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFlMd+AFxAAS/UTESDt76GmrhhN2o/Fdw==", 2023-10-19T13:04:55.161 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:04:55.162 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T13:04:55.162 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow r", 2023-10-19T13:04:55.162 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow rw", 2023-10-19T13:04:55.162 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow rwx" 2023-10-19T13:04:55.162 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:04:55.163 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:04:55.163 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:04:55.163 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "client.3", 2023-10-19T13:04:55.163 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFlHZ3fIBAA42B7n8opLS5smXIwJyVYzw==", 2023-10-19T13:04:55.163 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:04:55.164 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T13:04:55.164 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow r", 2023-10-19T13:04:55.164 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow rw", 2023-10-19T13:04:55.164 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow rwx" 2023-10-19T13:04:55.164 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:04:55.164 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:04:55.165 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:04:55.165 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "client.admin", 2023-10-19T13:04:55.165 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBjJDFlBgasFxAAMA24j8KszcXMEpYRQDXj4g==", 2023-10-19T13:04:55.165 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:04:55.165 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow *", 2023-10-19T13:04:55.166 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow *", 2023-10-19T13:04:55.166 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow *", 2023-10-19T13:04:55.166 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:04:55.166 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:04:55.166 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:04:55.167 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:04:55.167 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "mgr.x", 2023-10-19T13:04:55.167 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBjJDFllFBfMhAA5+KPXWmYZqcESmLObaUbBA==", 2023-10-19T13:04:55.167 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:04:55.167 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow *", 2023-10-19T13:04:55.167 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile mgr", 2023-10-19T13:04:55.168 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:04:55.168 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:04:55.168 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:04:55.168 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:04:55.168 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "mgr.y", 2023-10-19T13:04:55.169 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBjJDFlxRzJLRAAGjG58o7qXAHvIEIRYwl6QA==", 2023-10-19T13:04:55.169 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T13:04:55.169 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow *", 2023-10-19T13:04:55.169 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile mgr", 2023-10-19T13:04:55.169 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T13:04:55.170 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:04:55.170 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:04:55.170 INFO:teuthology.orchestra.run.smithi057.stdout: ] 2023-10-19T13:04:55.170 INFO:teuthology.orchestra.run.smithi057.stdout:} 2023-10-19T13:04:55.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.133+0000 7f560ffc5640 1 -- 172.21.15.57:0/545628797 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f55ec06af90 msgr2=0x7f55ec06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:55.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.133+0000 7f560ffc5640 1 --2- 172.21.15.57:0/545628797 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f55ec06af90 0x7f55ec06d450 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f560806d0d0 tx=0x7f5604008040 comp rx=0 tx=0).stop 2023-10-19T13:04:55.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.133+0000 7f560ffc5640 1 -- 172.21.15.57:0/545628797 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5608070350 msgr2=0x7f5608139680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:55.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.133+0000 7f560ffc5640 1 --2- 172.21.15.57:0/545628797 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5608070350 0x7f5608139680 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7f55f800d7a0 tx=0x7f55f800dc60 comp rx=0 tx=0).stop 2023-10-19T13:04:55.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.133+0000 7f560ffc5640 1 -- 172.21.15.57:0/545628797 shutdown_connections 2023-10-19T13:04:55.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.133+0000 7f560ffc5640 1 --2- 172.21.15.57:0/545628797 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f55ec06af90 0x7f55ec06d450 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:55.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.133+0000 7f560ffc5640 1 --2- 172.21.15.57:0/545628797 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5608134460 0x7f5608135c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:55.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.133+0000 7f560ffc5640 1 --2- 172.21.15.57:0/545628797 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f56080cb980 0x7f5608135730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:55.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.133+0000 7f560ffc5640 1 --2- 172.21.15.57:0/545628797 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5608070350 0x7f5608139680 unknown :-1 s=CLOSED pgs=1731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:55.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.133+0000 7f560ffc5640 1 -- 172.21.15.57:0/545628797 >> 172.21.15.57:0/545628797 conn(0x7f5608069930 msgr2=0x7f56081571f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:55.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.133+0000 7f560ffc5640 1 -- 172.21.15.57:0/545628797 shutdown_connections 2023-10-19T13:04:55.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.133+0000 7f560ffc5640 1 -- 172.21.15.57:0/545628797 wait complete. 2023-10-19T13:04:55.175 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T13:04:55.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.282+0000 7fa83f87c640 1 -- 172.21.15.57:0/3504872830 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa838156790 msgr2=0x7fa8380cc990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:55.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.282+0000 7fa83f87c640 1 --2- 172.21.15.57:0/3504872830 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa838156790 0x7fa8380cc990 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7fa838065ed0 tx=0x7fa83402bda0 comp rx=0 tx=0).stop 2023-10-19T13:04:55.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.283+0000 7fa83f87c640 1 -- 172.21.15.57:0/3504872830 shutdown_connections 2023-10-19T13:04:55.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.283+0000 7fa83f87c640 1 --2- 172.21.15.57:0/3504872830 >> v2:172.21.15.179:3300/0 conn(0x7fa83806bec0 0x7fa83806a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:55.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.283+0000 7fa83f87c640 1 --2- 172.21.15.57:0/3504872830 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa8380ac870 0x7fa83806b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:55.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.283+0000 7fa83f87c640 1 --2- 172.21.15.57:0/3504872830 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa838156790 0x7fa8380cc990 unknown :-1 s=CLOSED pgs=1732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:55.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.283+0000 7fa83f87c640 1 -- 172.21.15.57:0/3504872830 >> 172.21.15.57:0/3504872830 conn(0x7fa83806fae0 msgr2=0x7fa83806fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:55.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.283+0000 7fa83f87c640 1 -- 172.21.15.57:0/3504872830 shutdown_connections 2023-10-19T13:04:55.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.283+0000 7fa83f87c640 1 -- 172.21.15.57:0/3504872830 wait complete. 2023-10-19T13:04:55.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.284+0000 7fa83f87c640 1 Processor -- start 2023-10-19T13:04:55.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.284+0000 7fa83f87c640 1 -- start start 2023-10-19T13:04:55.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.285+0000 7fa83f87c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa83806bec0 0x7fa838143c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:55.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.285+0000 7fa83f87c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa8380ac870 0x7fa8380aa8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:55.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.285+0000 7fa83f87c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa838156790 0x7fa8380aae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:55.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.285+0000 7fa83f87c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8380aedd0 con 0x7fa8380ac870 2023-10-19T13:04:55.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.285+0000 7fa83f87c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8380b06b0 con 0x7fa838156790 2023-10-19T13:04:55.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.285+0000 7fa83f87c640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8380ae810 con 0x7fa83806bec0 2023-10-19T13:04:55.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.285+0000 7fa83ce17640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa8380ac870 0x7fa8380aa8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:55.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.285+0000 7fa83ce17640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa8380ac870 0x7fa8380aa8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38548/0 (socket says 172.21.15.57:38548) 2023-10-19T13:04:55.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.285+0000 7fa83ce17640 1 -- 172.21.15.57:0/1354392352 learned_addr learned my addr 172.21.15.57:0/1354392352 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:55.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.285+0000 7fa83de19640 1 --2- 172.21.15.57:0/1354392352 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa838156790 0x7fa8380aae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:55.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.285+0000 7fa83d618640 1 --2- 172.21.15.57:0/1354392352 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa83806bec0 0x7fa838143c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:55.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.285+0000 7fa83ce17640 1 -- 172.21.15.57:0/1354392352 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa83806bec0 msgr2=0x7fa838143c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:55.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.285+0000 7fa83ce17640 1 --2- 172.21.15.57:0/1354392352 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa83806bec0 0x7fa838143c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:55.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.285+0000 7fa83ce17640 1 -- 172.21.15.57:0/1354392352 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa838156790 msgr2=0x7fa8380aae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:55.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.285+0000 7fa83ce17640 1 --2- 172.21.15.57:0/1354392352 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa838156790 0x7fa8380aae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:55.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.285+0000 7fa83ce17640 1 -- 172.21.15.57:0/1354392352 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa834009560 con 0x7fa8380ac870 2023-10-19T13:04:55.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.285+0000 7fa83ce17640 1 --2- 172.21.15.57:0/1354392352 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa8380ac870 0x7fa8380aa8c0 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7fa82800e9f0 tx=0x7fa82800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:55.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.285+0000 7fa82dffb640 1 -- 172.21.15.57:0/1354392352 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa828009720 con 0x7fa8380ac870 2023-10-19T13:04:55.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.285+0000 7fa82dffb640 1 -- 172.21.15.57:0/1354392352 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fa82800ccc0 con 0x7fa8380ac870 2023-10-19T13:04:55.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.285+0000 7fa83f87c640 1 -- 172.21.15.57:0/1354392352 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8380aeaf0 con 0x7fa8380ac870 2023-10-19T13:04:55.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.285+0000 7fa82dffb640 1 -- 172.21.15.57:0/1354392352 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa828005280 con 0x7fa8380ac870 2023-10-19T13:04:55.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.286+0000 7fa83f87c640 1 -- 172.21.15.57:0/1354392352 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8380ada90 con 0x7fa8380ac870 2023-10-19T13:04:55.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.287+0000 7fa83f87c640 1 -- 172.21.15.57:0/1354392352 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8380690e0 con 0x7fa8380ac870 2023-10-19T13:04:55.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.287+0000 7fa82dffb640 1 -- 172.21.15.57:0/1354392352 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 79) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fa8280053e0 con 0x7fa8380ac870 2023-10-19T13:04:55.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.288+0000 7fa82dffb640 1 --2- 172.21.15.57:0/1354392352 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fa81c06af60 0x7fa81c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:55.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.288+0000 7fa82dffb640 1 -- 172.21.15.57:0/1354392352 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 5205+0+0 (secure 0 0 0) 0x7fa828014070 con 0x7fa8380ac870 2023-10-19T13:04:55.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.288+0000 7fa83d618640 1 --2- 172.21.15.57:0/1354392352 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fa81c06af60 0x7fa81c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:55.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.288+0000 7fa83d618640 1 --2- 172.21.15.57:0/1354392352 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fa81c06af60 0x7fa81c06d420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa834035000 tx=0x7fa834004730 comp rx=0 tx=0).ready entity=mgr.8419 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:55.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.292+0000 7fa82dffb640 1 -- 172.21.15.57:0/1354392352 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa828055300 con 0x7fa8380ac870 2023-10-19T13:04:55.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.467+0000 7fa83f87c640 1 -- 172.21.15.57:0/1354392352 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa83806a730 con 0x7fa8380ac870 2023-10-19T13:04:55.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.468+0000 7fa82dffb640 1 -- 172.21.15.57:0/1354392352 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v153) v1 ==== 74+0+13127 (secure 0 0 0) 0x7fa828058920 con 0x7fa8380ac870 2023-10-19T13:04:55.471 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:04:55.471 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":153,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T13:04:54.759773+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T13:04:55.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.470+0000 7fa83f87c640 1 -- 172.21.15.57:0/1354392352 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fa81c06af60 msgr2=0x7fa81c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:55.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.470+0000 7fa83f87c640 1 --2- 172.21.15.57:0/1354392352 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fa81c06af60 0x7fa81c06d420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa834035000 tx=0x7fa834004730 comp rx=0 tx=0).stop 2023-10-19T13:04:55.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.470+0000 7fa83f87c640 1 -- 172.21.15.57:0/1354392352 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa8380ac870 msgr2=0x7fa8380aa8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:55.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.470+0000 7fa83f87c640 1 --2- 172.21.15.57:0/1354392352 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa8380ac870 0x7fa8380aa8c0 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7fa82800e9f0 tx=0x7fa82800edf0 comp rx=0 tx=0).stop 2023-10-19T13:04:55.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.470+0000 7fa83f87c640 1 -- 172.21.15.57:0/1354392352 shutdown_connections 2023-10-19T13:04:55.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.470+0000 7fa83f87c640 1 --2- 172.21.15.57:0/1354392352 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fa81c06af60 0x7fa81c06d420 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:55.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.470+0000 7fa83f87c640 1 --2- 172.21.15.57:0/1354392352 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa838156790 0x7fa8380aae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:55.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.470+0000 7fa83f87c640 1 --2- 172.21.15.57:0/1354392352 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa8380ac870 0x7fa8380aa8c0 unknown :-1 s=CLOSED pgs=1733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:55.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.470+0000 7fa83f87c640 1 --2- 172.21.15.57:0/1354392352 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa83806bec0 0x7fa838143c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:55.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.470+0000 7fa83f87c640 1 -- 172.21.15.57:0/1354392352 >> 172.21.15.57:0/1354392352 conn(0x7fa83806fae0 msgr2=0x7fa838155d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:55.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.470+0000 7fa83f87c640 1 -- 172.21.15.57:0/1354392352 shutdown_connections 2023-10-19T13:04:55.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.470+0000 7fa83f87c640 1 -- 172.21.15.57:0/1354392352 wait complete. 2023-10-19T13:04:55.483 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-19T12:44:07.371186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-19T13:04:55.484 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-19T13:04:55.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.594+0000 7f05eb410640 1 -- 172.21.15.57:0/3400529188 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f05d0003fd0 msgr2=0x7f05d00047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:55.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.594+0000 7f05eb410640 1 --2- 172.21.15.57:0/3400529188 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f05d0003fd0 0x7f05d00047c0 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7f05d4001eb0 tx=0x7f05d00057f0 comp rx=0 tx=0).stop 2023-10-19T13:04:55.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.594+0000 7f05e99ad640 1 -- 172.21.15.57:0/3400529188 reap_dead start 2023-10-19T13:04:55.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.594+0000 7f05eb410640 1 -- 172.21.15.57:0/3400529188 shutdown_connections 2023-10-19T13:04:55.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.594+0000 7f05eb410640 1 -- 172.21.15.57:0/3400529188 >> 172.21.15.57:0/3400529188 conn(0x7f05e4069930 msgr2=0x7f05e4069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:55.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.594+0000 7f05eb410640 1 -- 172.21.15.57:0/3400529188 shutdown_connections 2023-10-19T13:04:55.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.594+0000 7f05eb410640 1 -- 172.21.15.57:0/3400529188 wait complete. 2023-10-19T13:04:55.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.595+0000 7f05eb410640 1 Processor -- start 2023-10-19T13:04:55.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.595+0000 7f05eb410640 1 -- start start 2023-10-19T13:04:55.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.596+0000 7f05eb410640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f05d0003fd0 0x7f05e41350c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:55.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.596+0000 7f05eb410640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f05e4162f30 0x7f05e4163390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:55.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.596+0000 7f05eb410640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f05e414aac0 0x7f05e414af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:55.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.596+0000 7f05eb410640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05e406b880 con 0x7f05e4162f30 2023-10-19T13:04:55.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.596+0000 7f05eb410640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05e414b480 con 0x7f05e414aac0 2023-10-19T13:04:55.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.596+0000 7f05eb410640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05e41638d0 con 0x7f05d0003fd0 2023-10-19T13:04:55.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.596+0000 7f05e89ab640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f05e4162f30 0x7f05e4163390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:55.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.596+0000 7f05e89ab640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f05e4162f30 0x7f05e4163390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38554/0 (socket says 172.21.15.57:38554) 2023-10-19T13:04:55.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.596+0000 7f05e89ab640 1 -- 172.21.15.57:0/999736870 learned_addr learned my addr 172.21.15.57:0/999736870 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:55.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.596+0000 7f05e99ad640 1 --2- 172.21.15.57:0/999736870 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f05e414aac0 0x7f05e414af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:55.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.596+0000 7f05e91ac640 1 --2- 172.21.15.57:0/999736870 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f05d0003fd0 0x7f05e41350c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:55.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.596+0000 7f05e89ab640 1 -- 172.21.15.57:0/999736870 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f05d0003fd0 msgr2=0x7f05e41350c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:55.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.596+0000 7f05e89ab640 1 --2- 172.21.15.57:0/999736870 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f05d0003fd0 0x7f05e41350c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:55.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.596+0000 7f05e89ab640 1 -- 172.21.15.57:0/999736870 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f05e414aac0 msgr2=0x7f05e414af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:55.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.596+0000 7f05e89ab640 1 --2- 172.21.15.57:0/999736870 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f05e414aac0 0x7f05e414af40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:55.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.596+0000 7f05e89ab640 1 -- 172.21.15.57:0/999736870 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05e40acfc0 con 0x7f05e4162f30 2023-10-19T13:04:55.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.596+0000 7f05e91ac640 1 --2- 172.21.15.57:0/999736870 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f05d0003fd0 0x7f05e41350c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T13:04:55.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.596+0000 7f05e89ab640 1 --2- 172.21.15.57:0/999736870 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f05e4162f30 0x7f05e4163390 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7f05d402c950 tx=0x7f05d402dfa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:55.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.596+0000 7f05d9ffb640 1 -- 172.21.15.57:0/999736870 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05d403c040 con 0x7f05e4162f30 2023-10-19T13:04:55.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.596+0000 7f05d9ffb640 1 -- 172.21.15.57:0/999736870 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f05d40035f0 con 0x7f05e4162f30 2023-10-19T13:04:55.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.596+0000 7f05eb410640 1 -- 172.21.15.57:0/999736870 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05e41579e0 con 0x7f05e4162f30 2023-10-19T13:04:55.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.596+0000 7f05d9ffb640 1 -- 172.21.15.57:0/999736870 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05d40039e0 con 0x7f05e4162f30 2023-10-19T13:04:55.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.597+0000 7f05eb410640 1 -- 172.21.15.57:0/999736870 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05e414b6a0 con 0x7f05e4162f30 2023-10-19T13:04:55.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.597+0000 7f05eb410640 1 -- 172.21.15.57:0/999736870 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05e40690e0 con 0x7f05e4162f30 2023-10-19T13:04:55.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.598+0000 7f05d9ffb640 1 -- 172.21.15.57:0/999736870 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 79) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f05d4032070 con 0x7f05e4162f30 2023-10-19T13:04:55.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.598+0000 7f05d9ffb640 1 --2- 172.21.15.57:0/999736870 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f05c806af60 0x7f05c806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:55.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.599+0000 7f05d9ffb640 1 -- 172.21.15.57:0/999736870 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f05d40aae30 con 0x7f05e4162f30 2023-10-19T13:04:55.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.599+0000 7f05e91ac640 1 --2- 172.21.15.57:0/999736870 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f05c806af60 0x7f05c806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:55.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.599+0000 7f05e91ac640 1 --2- 172.21.15.57:0/999736870 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f05c806af60 0x7f05c806d420 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f05dc005fd0 tx=0x7f05dc009530 comp rx=0 tx=0).ready entity=mgr.8419 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:55.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.603+0000 7f05d9ffb640 1 -- 172.21.15.57:0/999736870 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f05d4075f30 con 0x7f05e4162f30 2023-10-19T13:04:55.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.781+0000 7f05eb410640 1 -- 172.21.15.57:0/999736870 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f05e4069ff0 con 0x7f05e4162f30 2023-10-19T13:04:55.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.781+0000 7f05d9ffb640 1 -- 172.21.15.57:0/999736870 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v153) v1 ==== 93+0+10 (secure 0 0 0) 0x7f05d4079550 con 0x7f05e4162f30 2023-10-19T13:04:55.784 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-19T13:04:55.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.783+0000 7f05eb410640 1 -- 172.21.15.57:0/999736870 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f05c806af60 msgr2=0x7f05c806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:55.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.783+0000 7f05eb410640 1 --2- 172.21.15.57:0/999736870 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f05c806af60 0x7f05c806d420 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f05dc005fd0 tx=0x7f05dc009530 comp rx=0 tx=0).stop 2023-10-19T13:04:55.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.783+0000 7f05eb410640 1 -- 172.21.15.57:0/999736870 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f05e4162f30 msgr2=0x7f05e4163390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:55.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.783+0000 7f05eb410640 1 --2- 172.21.15.57:0/999736870 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f05e4162f30 0x7f05e4163390 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7f05d402c950 tx=0x7f05d402dfa0 comp rx=0 tx=0).stop 2023-10-19T13:04:55.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.783+0000 7f05eb410640 1 -- 172.21.15.57:0/999736870 shutdown_connections 2023-10-19T13:04:55.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.783+0000 7f05eb410640 1 --2- 172.21.15.57:0/999736870 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f05c806af60 0x7f05c806d420 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:55.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.783+0000 7f05eb410640 1 --2- 172.21.15.57:0/999736870 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f05e414aac0 0x7f05e414af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:55.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.783+0000 7f05eb410640 1 --2- 172.21.15.57:0/999736870 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f05e4162f30 0x7f05e4163390 unknown :-1 s=CLOSED pgs=1736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:55.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.783+0000 7f05eb410640 1 --2- 172.21.15.57:0/999736870 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f05d0003fd0 0x7f05e41350c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:55.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.783+0000 7f05eb410640 1 -- 172.21.15.57:0/999736870 >> 172.21.15.57:0/999736870 conn(0x7f05e4069930 msgr2=0x7f05e414e200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:55.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.783+0000 7f05eb410640 1 -- 172.21.15.57:0/999736870 shutdown_connections 2023-10-19T13:04:55.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.783+0000 7f05eb410640 1 -- 172.21.15.57:0/999736870 wait complete. 2023-10-19T13:04:55.796 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-19T13:04:55.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.903+0000 7fcf61e21640 1 -- 172.21.15.57:0/1908323768 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcf44003970 msgr2=0x7fcf44003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:55.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.903+0000 7fcf61e21640 1 --2- 172.21.15.57:0/1908323768 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcf44003970 0x7fcf44003d90 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7fcf4c006f10 tx=0x7fcf440057f0 comp rx=0 tx=0).stop 2023-10-19T13:04:55.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.903+0000 7fcf5bfff640 1 -- 172.21.15.57:0/1908323768 reap_dead start 2023-10-19T13:04:55.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.903+0000 7fcf61e21640 1 -- 172.21.15.57:0/1908323768 shutdown_connections 2023-10-19T13:04:55.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.903+0000 7fcf61e21640 1 -- 172.21.15.57:0/1908323768 >> 172.21.15.57:0/1908323768 conn(0x7fcf5c069930 msgr2=0x7fcf5c069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:55.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.903+0000 7fcf61e21640 1 -- 172.21.15.57:0/1908323768 shutdown_connections 2023-10-19T13:04:55.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.903+0000 7fcf61e21640 1 -- 172.21.15.57:0/1908323768 wait complete. 2023-10-19T13:04:55.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.904+0000 7fcf61e21640 1 Processor -- start 2023-10-19T13:04:55.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.905+0000 7fcf61e21640 1 -- start start 2023-10-19T13:04:55.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.905+0000 7fcf61e21640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcf44003970 0x7fcf5c13a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:55.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.905+0000 7fcf61e21640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcf5c12e990 0x7fcf5c12edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:55.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.905+0000 7fcf61e21640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcf5c0a8420 0x7fcf5c0a88a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:55.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.905+0000 7fcf61e21640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf5c12f330 con 0x7fcf5c0a8420 2023-10-19T13:04:55.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.905+0000 7fcf61e21640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf5c0aa5b0 con 0x7fcf44003970 2023-10-19T13:04:55.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.905+0000 7fcf61e21640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf5c0aa890 con 0x7fcf5c12e990 2023-10-19T13:04:55.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.905+0000 7fcf5bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcf5c0a8420 0x7fcf5c0a88a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:55.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.905+0000 7fcf5bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcf5c0a8420 0x7fcf5c0a88a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38558/0 (socket says 172.21.15.57:38558) 2023-10-19T13:04:55.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.905+0000 7fcf5bfff640 1 -- 172.21.15.57:0/2133751200 learned_addr learned my addr 172.21.15.57:0/2133751200 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:55.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.906+0000 7fcf5b7fe640 1 --2- 172.21.15.57:0/2133751200 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcf44003970 0x7fcf5c13a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:55.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.906+0000 7fcf5affd640 1 --2- 172.21.15.57:0/2133751200 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcf5c12e990 0x7fcf5c12edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:55.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.906+0000 7fcf5bfff640 1 -- 172.21.15.57:0/2133751200 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcf5c12e990 msgr2=0x7fcf5c12edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:55.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.906+0000 7fcf5bfff640 1 --2- 172.21.15.57:0/2133751200 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcf5c12e990 0x7fcf5c12edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:55.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.906+0000 7fcf5bfff640 1 -- 172.21.15.57:0/2133751200 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcf44003970 msgr2=0x7fcf5c13a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:55.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.906+0000 7fcf5bfff640 1 --2- 172.21.15.57:0/2133751200 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcf44003970 0x7fcf5c13a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:55.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.906+0000 7fcf5bfff640 1 -- 172.21.15.57:0/2133751200 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf5c0aab70 con 0x7fcf5c0a8420 2023-10-19T13:04:55.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.906+0000 7fcf5bfff640 1 --2- 172.21.15.57:0/2133751200 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcf5c0a8420 0x7fcf5c0a88a0 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7fcf5c0cc150 tx=0x7fcf54009a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:55.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.906+0000 7fcf43fff640 1 -- 172.21.15.57:0/2133751200 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf54018070 con 0x7fcf5c0a8420 2023-10-19T13:04:55.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.906+0000 7fcf43fff640 1 -- 172.21.15.57:0/2133751200 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fcf54004c30 con 0x7fcf5c0a8420 2023-10-19T13:04:55.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.906+0000 7fcf61e21640 1 -- 172.21.15.57:0/2133751200 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf5c0aae50 con 0x7fcf5c0a8420 2023-10-19T13:04:55.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.906+0000 7fcf43fff640 1 -- 172.21.15.57:0/2133751200 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf54011710 con 0x7fcf5c0a8420 2023-10-19T13:04:55.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.906+0000 7fcf61e21640 1 -- 172.21.15.57:0/2133751200 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf5c13a800 con 0x7fcf5c0a8420 2023-10-19T13:04:55.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.907+0000 7fcf61e21640 1 -- 172.21.15.57:0/2133751200 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf5c13a990 con 0x7fcf5c0a8420 2023-10-19T13:04:55.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.908+0000 7fcf43fff640 1 -- 172.21.15.57:0/2133751200 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 79) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fcf54011870 con 0x7fcf5c0a8420 2023-10-19T13:04:55.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.908+0000 7fcf43fff640 1 --2- 172.21.15.57:0/2133751200 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fcf4806af60 0x7fcf4806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:55.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.908+0000 7fcf43fff640 1 -- 172.21.15.57:0/2133751200 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 5205+0+0 (secure 0 0 0) 0x7fcf54089ba0 con 0x7fcf5c0a8420 2023-10-19T13:04:55.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.908+0000 7fcf5b7fe640 1 --2- 172.21.15.57:0/2133751200 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fcf4806af60 0x7fcf4806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:55.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.909+0000 7fcf5b7fe640 1 --2- 172.21.15.57:0/2133751200 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fcf4806af60 0x7fcf4806d420 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fcf4c034000 tx=0x7fcf4c0019a0 comp rx=0 tx=0).ready entity=mgr.8419 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:55.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:55.912+0000 7fcf43fff640 1 -- 172.21.15.57:0/2133751200 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fcf54054b90 con 0x7fcf5c0a8420 2023-10-19T13:04:56.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.090+0000 7fcf61e21640 1 -- 172.21.15.57:0/2133751200 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fcf5c0690e0 con 0x7fcf5c0a8420 2023-10-19T13:04:56.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.090+0000 7fcf43fff640 1 -- 172.21.15.57:0/2133751200 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v153) v1 ==== 84+0+65 (secure 0 0 0) 0x7fcf540581b0 con 0x7fcf5c0a8420 2023-10-19T13:04:56.094 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:04:56.094 INFO:teuthology.orchestra.run.smithi057.stdout:[ 2023-10-19T13:04:56.094 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:04:56.094 INFO:teuthology.orchestra.run.smithi057.stdout: "poolnum": 1, 2023-10-19T13:04:56.094 INFO:teuthology.orchestra.run.smithi057.stdout: "poolname": ".mgr" 2023-10-19T13:04:56.095 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:04:56.095 INFO:teuthology.orchestra.run.smithi057.stdout:] 2023-10-19T13:04:56.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.092+0000 7fcf61e21640 1 -- 172.21.15.57:0/2133751200 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fcf4806af60 msgr2=0x7fcf4806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:56.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.092+0000 7fcf61e21640 1 --2- 172.21.15.57:0/2133751200 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fcf4806af60 0x7fcf4806d420 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fcf4c034000 tx=0x7fcf4c0019a0 comp rx=0 tx=0).stop 2023-10-19T13:04:56.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.092+0000 7fcf61e21640 1 -- 172.21.15.57:0/2133751200 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcf5c0a8420 msgr2=0x7fcf5c0a88a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:56.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.092+0000 7fcf61e21640 1 --2- 172.21.15.57:0/2133751200 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcf5c0a8420 0x7fcf5c0a88a0 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7fcf5c0cc150 tx=0x7fcf54009a80 comp rx=0 tx=0).stop 2023-10-19T13:04:56.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.092+0000 7fcf61e21640 1 -- 172.21.15.57:0/2133751200 shutdown_connections 2023-10-19T13:04:56.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.092+0000 7fcf61e21640 1 --2- 172.21.15.57:0/2133751200 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fcf4806af60 0x7fcf4806d420 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:56.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.092+0000 7fcf61e21640 1 --2- 172.21.15.57:0/2133751200 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcf5c0a8420 0x7fcf5c0a88a0 unknown :-1 s=CLOSED pgs=1739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:56.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.092+0000 7fcf61e21640 1 --2- 172.21.15.57:0/2133751200 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcf5c12e990 0x7fcf5c12edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:56.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.092+0000 7fcf61e21640 1 --2- 172.21.15.57:0/2133751200 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcf44003970 0x7fcf5c13a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:56.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.092+0000 7fcf61e21640 1 -- 172.21.15.57:0/2133751200 >> 172.21.15.57:0/2133751200 conn(0x7fcf5c069930 msgr2=0x7fcf5c15c620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:56.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.092+0000 7fcf61e21640 1 -- 172.21.15.57:0/2133751200 shutdown_connections 2023-10-19T13:04:56.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.092+0000 7fcf61e21640 1 -- 172.21.15.57:0/2133751200 wait complete. 2023-10-19T13:04:56.106 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-19T13:04:56.106 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-19T13:04:56.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.216+0000 7f4beefe7640 1 -- 172.21.15.57:0/2177491746 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4be80cf340 msgr2=0x7f4be80d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:56.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.216+0000 7f4beefe7640 1 --2- 172.21.15.57:0/2177491746 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4be80cf340 0x7f4be80d1910 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7f4be808a120 tx=0x7f4bd802bcf0 comp rx=0 tx=0).stop 2023-10-19T13:04:56.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.216+0000 7f4beefe7640 1 -- 172.21.15.57:0/2177491746 shutdown_connections 2023-10-19T13:04:56.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.216+0000 7f4beefe7640 1 --2- 172.21.15.57:0/2177491746 >> v2:172.21.15.179:3300/0 conn(0x7f4be815b2a0 0x7f4be80cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:56.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.216+0000 7f4beefe7640 1 --2- 172.21.15.57:0/2177491746 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4be80cf340 0x7f4be80d1910 unknown :-1 s=CLOSED pgs=1741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:56.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.216+0000 7f4beefe7640 1 -- 172.21.15.57:0/2177491746 >> 172.21.15.57:0/2177491746 conn(0x7f4be80d0d60 msgr2=0x7f4be80d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:56.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.216+0000 7f4beefe7640 1 -- 172.21.15.57:0/2177491746 shutdown_connections 2023-10-19T13:04:56.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.216+0000 7f4beefe7640 1 -- 172.21.15.57:0/2177491746 wait complete. 2023-10-19T13:04:56.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.217+0000 7f4beefe7640 1 Processor -- start 2023-10-19T13:04:56.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.217+0000 7f4beefe7640 1 -- start start 2023-10-19T13:04:56.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.218+0000 7f4beefe7640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4be80ceb40 0x7f4be8158350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:56.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.218+0000 7f4beefe7640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4be80cf340 0x7f4be80c8a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:56.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.218+0000 7f4beefe7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4be815b2a0 0x7f4be815b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:56.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.218+0000 7f4beefe7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4be80c9540 con 0x7f4be815b2a0 2023-10-19T13:04:56.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.218+0000 7f4beefe7640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4be80c9820 con 0x7f4be80ceb40 2023-10-19T13:04:56.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.218+0000 7f4beefe7640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4be80c9b00 con 0x7f4be80cf340 2023-10-19T13:04:56.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.218+0000 7f4bed584640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4be815b2a0 0x7f4be815b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:56.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.218+0000 7f4bed584640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4be815b2a0 0x7f4be815b6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38568/0 (socket says 172.21.15.57:38568) 2023-10-19T13:04:56.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.218+0000 7f4bed584640 1 -- 172.21.15.57:0/3019188420 learned_addr learned my addr 172.21.15.57:0/3019188420 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:56.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.218+0000 7f4be7fff640 1 --2- 172.21.15.57:0/3019188420 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4be80cf340 0x7f4be80c8a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:56.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.219+0000 7f4becd83640 1 --2- 172.21.15.57:0/3019188420 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4be80ceb40 0x7f4be8158350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:56.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.219+0000 7f4bed584640 1 -- 172.21.15.57:0/3019188420 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4be80cf340 msgr2=0x7f4be80c8a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:56.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.219+0000 7f4bed584640 1 --2- 172.21.15.57:0/3019188420 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4be80cf340 0x7f4be80c8a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:56.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.219+0000 7f4bed584640 1 -- 172.21.15.57:0/3019188420 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4be80ceb40 msgr2=0x7f4be8158350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:56.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.219+0000 7f4bed584640 1 --2- 172.21.15.57:0/3019188420 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4be80ceb40 0x7f4be8158350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:56.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.219+0000 7f4bed584640 1 -- 172.21.15.57:0/3019188420 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bd8009490 con 0x7f4be815b2a0 2023-10-19T13:04:56.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.219+0000 7f4becd83640 1 --2- 172.21.15.57:0/3019188420 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4be80ceb40 0x7f4be8158350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:04:56.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.219+0000 7f4bed584640 1 --2- 172.21.15.57:0/3019188420 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4be815b2a0 0x7f4be815b6d0 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7f4be000cb10 tx=0x7f4be000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:56.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.219+0000 7f4be57fa640 1 -- 172.21.15.57:0/3019188420 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4be000ac70 con 0x7f4be815b2a0 2023-10-19T13:04:56.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.219+0000 7f4be57fa640 1 -- 172.21.15.57:0/3019188420 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f4be0012420 con 0x7f4be815b2a0 2023-10-19T13:04:56.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.219+0000 7f4beefe7640 1 -- 172.21.15.57:0/3019188420 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4be80c9de0 con 0x7f4be815b2a0 2023-10-19T13:04:56.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.219+0000 7f4be57fa640 1 -- 172.21.15.57:0/3019188420 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4be0012830 con 0x7f4be815b2a0 2023-10-19T13:04:56.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.219+0000 7f4beefe7640 1 -- 172.21.15.57:0/3019188420 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4be80cab10 con 0x7f4be815b2a0 2023-10-19T13:04:56.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.220+0000 7f4beefe7640 1 -- 172.21.15.57:0/3019188420 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4be80ca300 con 0x7f4be815b2a0 2023-10-19T13:04:56.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.221+0000 7f4be57fa640 1 -- 172.21.15.57:0/3019188420 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 79) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4be001b400 con 0x7f4be815b2a0 2023-10-19T13:04:56.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.221+0000 7f4be57fa640 1 --2- 172.21.15.57:0/3019188420 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f4bd406af60 0x7f4bd406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:56.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.221+0000 7f4be57fa640 1 -- 172.21.15.57:0/3019188420 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f4be0089ff0 con 0x7f4be815b2a0 2023-10-19T13:04:56.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.221+0000 7f4becd83640 1 --2- 172.21.15.57:0/3019188420 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f4bd406af60 0x7f4bd406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:56.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.222+0000 7f4becd83640 1 --2- 172.21.15.57:0/3019188420 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f4bd406af60 0x7f4bd406d420 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4bd8036000 tx=0x7f4bd8004730 comp rx=0 tx=0).ready entity=mgr.8419 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:56.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.225+0000 7f4be57fa640 1 -- 172.21.15.57:0/3019188420 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4be0055180 con 0x7f4be815b2a0 2023-10-19T13:04:56.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.401+0000 7f4beefe7640 1 -- 172.21.15.57:0/3019188420 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f4be80cb980 con 0x7f4be815b2a0 2023-10-19T13:04:56.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.758+0000 7f4be57fa640 1 -- 172.21.15.57:0/3019188420 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v154) v1 ==== 138+0+0 (secure 0 0 0) 0x7f4be0001b80 con 0x7f4be815b2a0 2023-10-19T13:04:56.762 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'cephfs_metadata' created 2023-10-19T13:04:56.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.761+0000 7f4beefe7640 1 -- 172.21.15.57:0/3019188420 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f4bd406af60 msgr2=0x7f4bd406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:56.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.761+0000 7f4beefe7640 1 --2- 172.21.15.57:0/3019188420 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f4bd406af60 0x7f4bd406d420 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4bd8036000 tx=0x7f4bd8004730 comp rx=0 tx=0).stop 2023-10-19T13:04:56.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.761+0000 7f4beefe7640 1 -- 172.21.15.57:0/3019188420 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4be815b2a0 msgr2=0x7f4be815b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:56.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.761+0000 7f4beefe7640 1 --2- 172.21.15.57:0/3019188420 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4be815b2a0 0x7f4be815b6d0 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7f4be000cb10 tx=0x7f4be000a060 comp rx=0 tx=0).stop 2023-10-19T13:04:56.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.761+0000 7f4beefe7640 1 -- 172.21.15.57:0/3019188420 shutdown_connections 2023-10-19T13:04:56.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.761+0000 7f4beefe7640 1 --2- 172.21.15.57:0/3019188420 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f4bd406af60 0x7f4bd406d420 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:56.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.761+0000 7f4beefe7640 1 --2- 172.21.15.57:0/3019188420 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4be815b2a0 0x7f4be815b6d0 unknown :-1 s=CLOSED pgs=1742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:56.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.761+0000 7f4beefe7640 1 --2- 172.21.15.57:0/3019188420 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4be80cf340 0x7f4be80c8a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:56.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.761+0000 7f4beefe7640 1 --2- 172.21.15.57:0/3019188420 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4be80ceb40 0x7f4be8158350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:56.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.761+0000 7f4beefe7640 1 -- 172.21.15.57:0/3019188420 >> 172.21.15.57:0/3019188420 conn(0x7f4be80d0d60 msgr2=0x7f4be80b1d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:56.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.761+0000 7f4beefe7640 1 -- 172.21.15.57:0/3019188420 shutdown_connections 2023-10-19T13:04:56.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.761+0000 7f4beefe7640 1 -- 172.21.15.57:0/3019188420 wait complete. 2023-10-19T13:04:56.778 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-19T13:04:56.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.895+0000 7f4f4f6e7640 1 -- 172.21.15.57:0/2812956395 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4f480ab500 msgr2=0x7f4f480ab9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:56.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.895+0000 7f4f4f6e7640 1 --2- 172.21.15.57:0/2812956395 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4f480ab500 0x7f4f480ab9b0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f4f40005fd0 tx=0x7f4f4002c0e0 comp rx=0 tx=0).stop 2023-10-19T13:04:56.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.895+0000 7f4f4f6e7640 1 -- 172.21.15.57:0/2812956395 shutdown_connections 2023-10-19T13:04:56.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.895+0000 7f4f4f6e7640 1 --2- 172.21.15.57:0/2812956395 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4f480ab500 0x7f4f480ab9b0 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:56.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.895+0000 7f4f4f6e7640 1 --2- 172.21.15.57:0/2812956395 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4f480c9dc0 0x7f4f4806b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:56.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.895+0000 7f4f4f6e7640 1 -- 172.21.15.57:0/2812956395 >> 172.21.15.57:0/2812956395 conn(0x7f4f48069930 msgr2=0x7f4f48069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:56.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.895+0000 7f4f4f6e7640 1 -- 172.21.15.57:0/2812956395 shutdown_connections 2023-10-19T13:04:56.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.895+0000 7f4f4f6e7640 1 -- 172.21.15.57:0/2812956395 wait complete. 2023-10-19T13:04:56.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.896+0000 7f4f4f6e7640 1 Processor -- start 2023-10-19T13:04:56.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.896+0000 7f4f4f6e7640 1 -- start start 2023-10-19T13:04:56.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.897+0000 7f4f4f6e7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4f4806af00 0x7f4f480a66e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:56.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.897+0000 7f4f4f6e7640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4f480ab500 0x7f4f480a6c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:56.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.897+0000 7f4f4f6e7640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4f480c9dc0 0x7f4f481ef730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:56.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.897+0000 7f4f4f6e7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f4813d2a0 con 0x7f4f4806af00 2023-10-19T13:04:56.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.897+0000 7f4f4f6e7640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f4813d580 con 0x7f4f480ab500 2023-10-19T13:04:56.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.897+0000 7f4f4f6e7640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f4813d860 con 0x7f4f480c9dc0 2023-10-19T13:04:56.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.897+0000 7f4f4d483640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4f4806af00 0x7f4f480a66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:56.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.897+0000 7f4f4d483640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4f4806af00 0x7f4f480a66e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42710/0 (socket says 172.21.15.57:42710) 2023-10-19T13:04:56.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.897+0000 7f4f4d483640 1 -- 172.21.15.57:0/3930913681 learned_addr learned my addr 172.21.15.57:0/3930913681 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:56.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.897+0000 7f4f4cc82640 1 --2- 172.21.15.57:0/3930913681 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4f480ab500 0x7f4f480a6c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:56.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.897+0000 7f4f4dc84640 1 --2- 172.21.15.57:0/3930913681 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4f480c9dc0 0x7f4f481ef730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:56.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.897+0000 7f4f4d483640 1 -- 172.21.15.57:0/3930913681 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4f480c9dc0 msgr2=0x7f4f481ef730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:56.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.897+0000 7f4f4d483640 1 --2- 172.21.15.57:0/3930913681 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4f480c9dc0 0x7f4f481ef730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:56.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.897+0000 7f4f4d483640 1 -- 172.21.15.57:0/3930913681 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4f480ab500 msgr2=0x7f4f480a6c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:56.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.897+0000 7f4f4d483640 1 --2- 172.21.15.57:0/3930913681 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4f480ab500 0x7f4f480a6c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:56.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.898+0000 7f4f4d483640 1 -- 172.21.15.57:0/3930913681 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f40009840 con 0x7f4f4806af00 2023-10-19T13:04:56.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.898+0000 7f4f4cc82640 1 --2- 172.21.15.57:0/3930913681 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4f480ab500 0x7f4f480a6c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:04:56.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.898+0000 7f4f4d483640 1 --2- 172.21.15.57:0/3930913681 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4f4806af00 0x7f4f480a66e0 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7f4f3800c400 tx=0x7f4f3800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:56.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.898+0000 7f4f3dffb640 1 -- 172.21.15.57:0/3930913681 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f38004230 con 0x7f4f4806af00 2023-10-19T13:04:56.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.898+0000 7f4f3dffb640 1 -- 172.21.15.57:0/3930913681 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f4f380044a0 con 0x7f4f4806af00 2023-10-19T13:04:56.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.898+0000 7f4f4f6e7640 1 -- 172.21.15.57:0/3930913681 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f4813db40 con 0x7f4f4806af00 2023-10-19T13:04:56.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.898+0000 7f4f3dffb640 1 -- 172.21.15.57:0/3930913681 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f38003850 con 0x7f4f4806af00 2023-10-19T13:04:56.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.898+0000 7f4f4f6e7640 1 -- 172.21.15.57:0/3930913681 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f481efe50 con 0x7f4f4806af00 2023-10-19T13:04:56.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.899+0000 7f4f4f6e7640 1 -- 172.21.15.57:0/3930913681 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f4813dd80 con 0x7f4f4806af00 2023-10-19T13:04:56.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.900+0000 7f4f3dffb640 1 -- 172.21.15.57:0/3930913681 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 79) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4f380039b0 con 0x7f4f4806af00 2023-10-19T13:04:56.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.900+0000 7f4f3dffb640 1 --2- 172.21.15.57:0/3930913681 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f4f2c06af60 0x7f4f2c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:56.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.900+0000 7f4f3dffb640 1 -- 172.21.15.57:0/3930913681 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 5575+0+0 (secure 0 0 0) 0x7f4f38088ff0 con 0x7f4f4806af00 2023-10-19T13:04:56.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.900+0000 7f4f4cc82640 1 --2- 172.21.15.57:0/3930913681 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f4f2c06af60 0x7f4f2c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:56.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.901+0000 7f4f4cc82640 1 --2- 172.21.15.57:0/3930913681 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f4f2c06af60 0x7f4f2c06d420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4f4806bc40 tx=0x7f4f44005f90 comp rx=0 tx=0).ready entity=mgr.8419 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:56.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:56.904+0000 7f4f3dffb640 1 -- 172.21.15.57:0/3930913681 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4f38053f50 con 0x7f4f4806af00 2023-10-19T13:04:57.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:57.080+0000 7f4f4f6e7640 1 -- 172.21.15.57:0/3930913681 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f4f4806c500 con 0x7f4f4806af00 2023-10-19T13:04:57.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:57.761+0000 7f4f3dffb640 1 -- 172.21.15.57:0/3930913681 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v155) v1 ==== 170+0+0 (secure 0 0 0) 0x7f4f380170d0 con 0x7f4f4806af00 2023-10-19T13:04:57.764 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'cephfs_data' created 2023-10-19T13:04:57.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:57.764+0000 7f4f4f6e7640 1 -- 172.21.15.57:0/3930913681 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f4f2c06af60 msgr2=0x7f4f2c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:57.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:57.764+0000 7f4f4f6e7640 1 --2- 172.21.15.57:0/3930913681 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f4f2c06af60 0x7f4f2c06d420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4f4806bc40 tx=0x7f4f44005f90 comp rx=0 tx=0).stop 2023-10-19T13:04:57.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:57.764+0000 7f4f4f6e7640 1 -- 172.21.15.57:0/3930913681 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4f4806af00 msgr2=0x7f4f480a66e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:57.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:57.764+0000 7f4f4f6e7640 1 --2- 172.21.15.57:0/3930913681 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4f4806af00 0x7f4f480a66e0 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7f4f3800c400 tx=0x7f4f3800c8c0 comp rx=0 tx=0).stop 2023-10-19T13:04:57.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:57.764+0000 7f4f4f6e7640 1 -- 172.21.15.57:0/3930913681 shutdown_connections 2023-10-19T13:04:57.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:57.764+0000 7f4f4f6e7640 1 --2- 172.21.15.57:0/3930913681 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f4f2c06af60 0x7f4f2c06d420 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:57.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:57.764+0000 7f4f4f6e7640 1 --2- 172.21.15.57:0/3930913681 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4f480c9dc0 0x7f4f481ef730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:57.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:57.765+0000 7f4f4f6e7640 1 --2- 172.21.15.57:0/3930913681 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4f480ab500 0x7f4f480a6c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:57.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:57.765+0000 7f4f4f6e7640 1 --2- 172.21.15.57:0/3930913681 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4f4806af00 0x7f4f480a66e0 unknown :-1 s=CLOSED pgs=1743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:57.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:57.765+0000 7f4f4f6e7640 1 -- 172.21.15.57:0/3930913681 >> 172.21.15.57:0/3930913681 conn(0x7f4f48069930 msgr2=0x7f4f4815b8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:57.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:57.765+0000 7f4f4f6e7640 1 -- 172.21.15.57:0/3930913681 shutdown_connections 2023-10-19T13:04:57.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:57.765+0000 7f4f4f6e7640 1 -- 172.21.15.57:0/3930913681 wait complete. 2023-10-19T13:04:57.792 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-19T13:04:57.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:57.907+0000 7f23e88f2640 1 -- 172.21.15.57:0/345298969 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f23e4140620 msgr2=0x7f23e40703e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:57.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:57.907+0000 7f23e88f2640 1 --2- 172.21.15.57:0/345298969 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f23e4140620 0x7f23e40703e0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f23dc009830 tx=0x7f23dc02bd60 comp rx=0 tx=0).stop 2023-10-19T13:04:57.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:57.907+0000 7f23e88f2640 1 -- 172.21.15.57:0/345298969 shutdown_connections 2023-10-19T13:04:57.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:57.907+0000 7f23e88f2640 1 --2- 172.21.15.57:0/345298969 >> v2:172.21.15.179:3300/0 conn(0x7f23e406fab0 0x7f23e40c97a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:57.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:57.907+0000 7f23e88f2640 1 --2- 172.21.15.57:0/345298969 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f23e4140620 0x7f23e40703e0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:57.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:57.907+0000 7f23e88f2640 1 -- 172.21.15.57:0/345298969 >> 172.21.15.57:0/345298969 conn(0x7f23e4069930 msgr2=0x7f23e4069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:57.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:57.907+0000 7f23e88f2640 1 -- 172.21.15.57:0/345298969 shutdown_connections 2023-10-19T13:04:57.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:57.908+0000 7f23e88f2640 1 -- 172.21.15.57:0/345298969 wait complete. 2023-10-19T13:04:57.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:57.909+0000 7f23e88f2640 1 Processor -- start 2023-10-19T13:04:57.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:57.909+0000 7f23e88f2640 1 -- start start 2023-10-19T13:04:57.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:57.909+0000 7f23e88f2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f23e406fab0 0x7f23e4157600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:57.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:57.909+0000 7f23e88f2640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f23e4140620 0x7f23e40ca870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:57.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:57.910+0000 7f23e88f2640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f23e4140a00 0x7f23e40cadb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:57.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:57.910+0000 7f23e88f2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23e40aba60 con 0x7f23e406fab0 2023-10-19T13:04:57.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:57.910+0000 7f23e88f2640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23e413a720 con 0x7f23e4140a00 2023-10-19T13:04:57.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:57.910+0000 7f23e88f2640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23e413d520 con 0x7f23e4140620 2023-10-19T13:04:57.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:57.910+0000 7f23e259c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f23e406fab0 0x7f23e4157600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:57.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:57.910+0000 7f23e259c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f23e406fab0 0x7f23e4157600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42712/0 (socket says 172.21.15.57:42712) 2023-10-19T13:04:57.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:57.910+0000 7f23e259c640 1 -- 172.21.15.57:0/1937005779 learned_addr learned my addr 172.21.15.57:0/1937005779 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:57.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:57.910+0000 7f23e2d9d640 1 --2- 172.21.15.57:0/1937005779 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f23e4140a00 0x7f23e40cadb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:57.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:57.910+0000 7f23e1d9b640 1 --2- 172.21.15.57:0/1937005779 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f23e4140620 0x7f23e40ca870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:57.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:57.910+0000 7f23e259c640 1 -- 172.21.15.57:0/1937005779 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f23e4140620 msgr2=0x7f23e40ca870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:57.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:57.910+0000 7f23e259c640 1 --2- 172.21.15.57:0/1937005779 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f23e4140620 0x7f23e40ca870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:57.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:57.910+0000 7f23e259c640 1 -- 172.21.15.57:0/1937005779 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f23e4140a00 msgr2=0x7f23e40cadb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:57.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:57.910+0000 7f23e259c640 1 --2- 172.21.15.57:0/1937005779 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f23e4140a00 0x7f23e40cadb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:57.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:57.910+0000 7f23e259c640 1 -- 172.21.15.57:0/1937005779 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23dc009490 con 0x7f23e406fab0 2023-10-19T13:04:57.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:57.910+0000 7f23e2d9d640 1 --2- 172.21.15.57:0/1937005779 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f23e4140a00 0x7f23e40cadb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T13:04:57.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:57.910+0000 7f23e259c640 1 --2- 172.21.15.57:0/1937005779 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f23e406fab0 0x7f23e4157600 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7f23dc000c00 tx=0x7f23dc0038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:57.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:57.910+0000 7f23daffd640 1 -- 172.21.15.57:0/1937005779 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23dc003ea0 con 0x7f23e406fab0 2023-10-19T13:04:57.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:57.910+0000 7f23e88f2640 1 -- 172.21.15.57:0/1937005779 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23e40acaa0 con 0x7f23e406fab0 2023-10-19T13:04:57.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:57.910+0000 7f23daffd640 1 -- 172.21.15.57:0/1937005779 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f23dc034440 con 0x7f23e406fab0 2023-10-19T13:04:57.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:57.910+0000 7f23daffd640 1 -- 172.21.15.57:0/1937005779 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23dc034850 con 0x7f23e406fab0 2023-10-19T13:04:57.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:57.911+0000 7f23e88f2640 1 -- 172.21.15.57:0/1937005779 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23e40ab780 con 0x7f23e406fab0 2023-10-19T13:04:57.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:57.912+0000 7f23e88f2640 1 -- 172.21.15.57:0/1937005779 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23e40ac900 con 0x7f23e406fab0 2023-10-19T13:04:57.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:57.912+0000 7f23daffd640 1 -- 172.21.15.57:0/1937005779 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 79) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f23dc0349b0 con 0x7f23e406fab0 2023-10-19T13:04:57.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:57.913+0000 7f23daffd640 1 --2- 172.21.15.57:0/1937005779 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f23c406af60 0x7f23c406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:57.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:57.913+0000 7f23daffd640 1 -- 172.21.15.57:0/1937005779 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5957+0+0 (secure 0 0 0) 0x7f23dc0abf60 con 0x7f23e406fab0 2023-10-19T13:04:57.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:57.913+0000 7f23e1d9b640 1 --2- 172.21.15.57:0/1937005779 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f23c406af60 0x7f23c406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:57.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:57.914+0000 7f23e1d9b640 1 --2- 172.21.15.57:0/1937005779 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f23c406af60 0x7f23c406d420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f23e4158a50 tx=0x7f23cc007eb0 comp rx=0 tx=0).ready entity=mgr.8419 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:57.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:57.918+0000 7f23daffd640 1 -- 172.21.15.57:0/1937005779 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f23e40ac900 con 0x7f23e406fab0 2023-10-19T13:04:58.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:58.132+0000 7f23e88f2640 1 -- 172.21.15.57:0/1937005779 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f23e40ac900 con 0x7f23e406fab0 2023-10-19T13:04:58.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:58.774+0000 7f23daffd640 1 -- 172.21.15.57:0/1937005779 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 19 and data pool 20 v61) v1 ==== 174+0+0 (secure 0 0 0) 0x7f23e40ac900 con 0x7f23e406fab0 2023-10-19T13:04:58.777 INFO:teuthology.orchestra.run.smithi057.stderr:new fs with metadata pool 19 and data pool 20 2023-10-19T13:04:58.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:58.777+0000 7f23e88f2640 1 -- 172.21.15.57:0/1937005779 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f23c406af60 msgr2=0x7f23c406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:58.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:58.777+0000 7f23e88f2640 1 --2- 172.21.15.57:0/1937005779 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f23c406af60 0x7f23c406d420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f23e4158a50 tx=0x7f23cc007eb0 comp rx=0 tx=0).stop 2023-10-19T13:04:58.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:58.778+0000 7f23e88f2640 1 -- 172.21.15.57:0/1937005779 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f23e406fab0 msgr2=0x7f23e4157600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:58.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:58.778+0000 7f23e88f2640 1 --2- 172.21.15.57:0/1937005779 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f23e406fab0 0x7f23e4157600 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7f23dc000c00 tx=0x7f23dc0038a0 comp rx=0 tx=0).stop 2023-10-19T13:04:58.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:58.778+0000 7f23e88f2640 1 -- 172.21.15.57:0/1937005779 shutdown_connections 2023-10-19T13:04:58.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:58.778+0000 7f23e88f2640 1 --2- 172.21.15.57:0/1937005779 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f23c406af60 0x7f23c406d420 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:58.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:58.778+0000 7f23e88f2640 1 --2- 172.21.15.57:0/1937005779 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f23e4140a00 0x7f23e40cadb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:58.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:58.778+0000 7f23e88f2640 1 --2- 172.21.15.57:0/1937005779 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f23e4140620 0x7f23e40ca870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:58.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:58.778+0000 7f23e88f2640 1 --2- 172.21.15.57:0/1937005779 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f23e406fab0 0x7f23e4157600 unknown :-1 s=CLOSED pgs=1744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:58.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:58.778+0000 7f23e88f2640 1 -- 172.21.15.57:0/1937005779 >> 172.21.15.57:0/1937005779 conn(0x7f23e4069930 msgr2=0x7f23e40b11e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:58.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:58.778+0000 7f23e88f2640 1 -- 172.21.15.57:0/1937005779 shutdown_connections 2023-10-19T13:04:58.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:58.778+0000 7f23e88f2640 1 -- 172.21.15.57:0/1937005779 wait complete. 2023-10-19T13:04:58.797 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T13:04:58.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:58.907+0000 7f6e1afb6640 1 -- 172.21.15.57:0/3696136336 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6dfc003ef0 msgr2=0x7f6dfc0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:58.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:58.907+0000 7f6e1afb6640 1 --2- 172.21.15.57:0/3696136336 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6dfc003ef0 0x7f6dfc0047c0 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7f6e0800c260 tx=0x7f6dfc0057f0 comp rx=0 tx=0).stop 2023-10-19T13:04:58.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:58.907+0000 7f6e19553640 1 -- 172.21.15.57:0/3696136336 reap_dead start 2023-10-19T13:04:58.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:58.907+0000 7f6e1afb6640 1 -- 172.21.15.57:0/3696136336 shutdown_connections 2023-10-19T13:04:58.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:58.907+0000 7f6e1afb6640 1 -- 172.21.15.57:0/3696136336 >> 172.21.15.57:0/3696136336 conn(0x7f6e140d0d60 msgr2=0x7f6e140d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:58.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:58.907+0000 7f6e1afb6640 1 -- 172.21.15.57:0/3696136336 shutdown_connections 2023-10-19T13:04:58.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:58.907+0000 7f6e1afb6640 1 -- 172.21.15.57:0/3696136336 wait complete. 2023-10-19T13:04:58.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:58.908+0000 7f6e1afb6640 1 Processor -- start 2023-10-19T13:04:58.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:58.908+0000 7f6e1afb6640 1 -- start start 2023-10-19T13:04:58.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:58.909+0000 7f6e1afb6640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6dfc003ef0 0x7f6e1414db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:58.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:58.909+0000 7f6e1afb6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6e1415b6a0 0x7f6e1415bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:58.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:58.909+0000 7f6e1afb6640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6e1414ed60 0x7f6e1414f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:58.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:58.909+0000 7f6e1afb6640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e141e60f0 con 0x7f6e1415b6a0 2023-10-19T13:04:58.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:58.909+0000 7f6e1afb6640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e1414e310 con 0x7f6e1414ed60 2023-10-19T13:04:58.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:58.909+0000 7f6e1afb6640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e141e5b30 con 0x7f6dfc003ef0 2023-10-19T13:04:58.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:58.909+0000 7f6e13fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6e1415b6a0 0x7f6e1415bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:58.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:58.909+0000 7f6e13fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6e1415b6a0 0x7f6e1415bb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42740/0 (socket says 172.21.15.57:42740) 2023-10-19T13:04:58.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:58.909+0000 7f6e13fff640 1 -- 172.21.15.57:0/2306879821 learned_addr learned my addr 172.21.15.57:0/2306879821 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:58.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:58.909+0000 7f6e18d52640 1 --2- 172.21.15.57:0/2306879821 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6dfc003ef0 0x7f6e1414db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:58.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:58.909+0000 7f6e19553640 1 --2- 172.21.15.57:0/2306879821 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6e1414ed60 0x7f6e1414f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:58.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:58.909+0000 7f6e13fff640 1 -- 172.21.15.57:0/2306879821 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6dfc003ef0 msgr2=0x7f6e1414db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:58.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:58.909+0000 7f6e13fff640 1 --2- 172.21.15.57:0/2306879821 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6dfc003ef0 0x7f6e1414db50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:58.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:58.909+0000 7f6e13fff640 1 -- 172.21.15.57:0/2306879821 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6e1414ed60 msgr2=0x7f6e1414f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:58.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:58.909+0000 7f6e13fff640 1 --2- 172.21.15.57:0/2306879821 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6e1414ed60 0x7f6e1414f1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:58.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:58.909+0000 7f6e13fff640 1 -- 172.21.15.57:0/2306879821 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e141e5e10 con 0x7f6e1415b6a0 2023-10-19T13:04:58.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:58.909+0000 7f6e18d52640 1 --2- 172.21.15.57:0/2306879821 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6dfc003ef0 0x7f6e1414db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:04:58.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:58.909+0000 7f6e19553640 1 --2- 172.21.15.57:0/2306879821 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6e1414ed60 0x7f6e1414f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:04:58.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:58.909+0000 7f6e13fff640 1 --2- 172.21.15.57:0/2306879821 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6e1415b6a0 0x7f6e1415bb20 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7f6e0802c8e0 tx=0x7f6e080015f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:58.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:58.910+0000 7f6e117fa640 1 -- 172.21.15.57:0/2306879821 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e08006310 con 0x7f6e1415b6a0 2023-10-19T13:04:58.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:58.910+0000 7f6e117fa640 1 -- 172.21.15.57:0/2306879821 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f6e08006470 con 0x7f6e1415b6a0 2023-10-19T13:04:58.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:58.910+0000 7f6e1afb6640 1 -- 172.21.15.57:0/2306879821 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e1414e560 con 0x7f6e1415b6a0 2023-10-19T13:04:58.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:58.910+0000 7f6e117fa640 1 -- 172.21.15.57:0/2306879821 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e08003850 con 0x7f6e1415b6a0 2023-10-19T13:04:58.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:58.910+0000 7f6e1afb6640 1 -- 172.21.15.57:0/2306879821 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e1414e090 con 0x7f6e1415b6a0 2023-10-19T13:04:58.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:58.911+0000 7f6e1afb6640 1 -- 172.21.15.57:0/2306879821 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e140d07d0 con 0x7f6e1415b6a0 2023-10-19T13:04:58.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:58.911+0000 7f6e117fa640 1 -- 172.21.15.57:0/2306879821 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 79) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6e08031030 con 0x7f6e1415b6a0 2023-10-19T13:04:58.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:58.912+0000 7f6e117fa640 1 --2- 172.21.15.57:0/2306879821 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f6e0006af60 0x7f6e0006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:58.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:58.912+0000 7f6e117fa640 1 -- 172.21.15.57:0/2306879821 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(156..156 src has 1..156) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f6e080aa820 con 0x7f6e1415b6a0 2023-10-19T13:04:58.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:58.912+0000 7f6e18d52640 1 --2- 172.21.15.57:0/2306879821 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f6e0006af60 0x7f6e0006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:58.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:58.912+0000 7f6e18d52640 1 --2- 172.21.15.57:0/2306879821 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f6e0006af60 0x7f6e0006d420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6e0c005fd0 tx=0x7f6e0c009500 comp rx=0 tx=0).ready entity=mgr.8419 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:58.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:58.916+0000 7f6e117fa640 1 -- 172.21.15.57:0/2306879821 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6e08078fd0 con 0x7f6e1415b6a0 2023-10-19T13:04:59.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.094+0000 7f6e1afb6640 1 -- 172.21.15.57:0/2306879821 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6e140677b0 con 0x7f6e1415b6a0 2023-10-19T13:04:59.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.096+0000 7f6e117fa640 1 -- 172.21.15.57:0/2306879821 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v156) v1 ==== 74+0+16800 (secure 0 0 0) 0x7f6e08078b60 con 0x7f6e1415b6a0 2023-10-19T13:04:59.099 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:04:59.099 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":156,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T13:04:58.134268+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2023-10-19T13:04:56.433332+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"156","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2023-10-19T13:04:57.112749+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"156","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T13:04:59.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.097+0000 7f6e1afb6640 1 -- 172.21.15.57:0/2306879821 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f6e0006af60 msgr2=0x7f6e0006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:59.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.097+0000 7f6e1afb6640 1 --2- 172.21.15.57:0/2306879821 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f6e0006af60 0x7f6e0006d420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6e0c005fd0 tx=0x7f6e0c009500 comp rx=0 tx=0).stop 2023-10-19T13:04:59.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.097+0000 7f6e1afb6640 1 -- 172.21.15.57:0/2306879821 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6e1415b6a0 msgr2=0x7f6e1415bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:59.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.097+0000 7f6e1afb6640 1 --2- 172.21.15.57:0/2306879821 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6e1415b6a0 0x7f6e1415bb20 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7f6e0802c8e0 tx=0x7f6e080015f0 comp rx=0 tx=0).stop 2023-10-19T13:04:59.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.098+0000 7f6e1afb6640 1 -- 172.21.15.57:0/2306879821 shutdown_connections 2023-10-19T13:04:59.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.098+0000 7f6e1afb6640 1 --2- 172.21.15.57:0/2306879821 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f6e0006af60 0x7f6e0006d420 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:59.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.098+0000 7f6e1afb6640 1 --2- 172.21.15.57:0/2306879821 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6e1414ed60 0x7f6e1414f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:59.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.098+0000 7f6e1afb6640 1 --2- 172.21.15.57:0/2306879821 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6e1415b6a0 0x7f6e1415bb20 unknown :-1 s=CLOSED pgs=1747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:59.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.098+0000 7f6e1afb6640 1 --2- 172.21.15.57:0/2306879821 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6dfc003ef0 0x7f6e1414db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:59.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.098+0000 7f6e1afb6640 1 -- 172.21.15.57:0/2306879821 >> 172.21.15.57:0/2306879821 conn(0x7f6e140d0d60 msgr2=0x7f6e14069f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:59.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.098+0000 7f6e1afb6640 1 -- 172.21.15.57:0/2306879821 shutdown_connections 2023-10-19T13:04:59.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.098+0000 7f6e1afb6640 1 -- 172.21.15.57:0/2306879821 wait complete. 2023-10-19T13:04:59.111 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T13:04:59.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.218+0000 7fb3eb733640 1 -- 172.21.15.57:0/3873881386 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3e40afa20 msgr2=0x7fb3e4158300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:59.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.218+0000 7fb3eb733640 1 --2- 172.21.15.57:0/3873881386 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3e40afa20 0x7fb3e4158300 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7fb3d4005fd0 tx=0x7fb3d402d160 comp rx=0 tx=0).stop 2023-10-19T13:04:59.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.218+0000 7fb3eb733640 1 -- 172.21.15.57:0/3873881386 shutdown_connections 2023-10-19T13:04:59.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.218+0000 7fb3eb733640 1 --2- 172.21.15.57:0/3873881386 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb3e40b0530 0x7fb3e40b09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:59.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.218+0000 7fb3eb733640 1 --2- 172.21.15.57:0/3873881386 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3e40afa20 0x7fb3e4158300 unknown :-1 s=CLOSED pgs=1749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:59.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.218+0000 7fb3eb733640 1 -- 172.21.15.57:0/3873881386 >> 172.21.15.57:0/3873881386 conn(0x7fb3e4069930 msgr2=0x7fb3e4069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:59.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.218+0000 7fb3eb733640 1 -- 172.21.15.57:0/3873881386 shutdown_connections 2023-10-19T13:04:59.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.218+0000 7fb3eb733640 1 -- 172.21.15.57:0/3873881386 wait complete. 2023-10-19T13:04:59.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.219+0000 7fb3eb733640 1 Processor -- start 2023-10-19T13:04:59.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.219+0000 7fb3eb733640 1 -- start start 2023-10-19T13:04:59.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.220+0000 7fb3eb733640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3e40afa20 0x7fb3e40ae970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:59.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.220+0000 7fb3eb733640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb3e40b0530 0x7fb3e4137e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:59.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.220+0000 7fb3eb733640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb3e40cbd20 0x7fb3e4138340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:59.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.220+0000 7fb3eb733640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3e4147ce0 con 0x7fb3e40afa20 2023-10-19T13:04:59.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.220+0000 7fb3eb733640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3e41470d0 con 0x7fb3e40cbd20 2023-10-19T13:04:59.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.220+0000 7fb3eb733640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3e41473b0 con 0x7fb3e40b0530 2023-10-19T13:04:59.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.220+0000 7fb3e94cf640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3e40afa20 0x7fb3e40ae970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:59.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.220+0000 7fb3e94cf640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3e40afa20 0x7fb3e40ae970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42748/0 (socket says 172.21.15.57:42748) 2023-10-19T13:04:59.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.220+0000 7fb3e94cf640 1 -- 172.21.15.57:0/2797443709 learned_addr learned my addr 172.21.15.57:0/2797443709 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:59.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.220+0000 7fb3e8cce640 1 --2- 172.21.15.57:0/2797443709 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb3e40b0530 0x7fb3e4137e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:59.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.220+0000 7fb3e9cd0640 1 --2- 172.21.15.57:0/2797443709 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb3e40cbd20 0x7fb3e4138340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:59.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.220+0000 7fb3e94cf640 1 -- 172.21.15.57:0/2797443709 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb3e40b0530 msgr2=0x7fb3e4137e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:59.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.220+0000 7fb3e94cf640 1 --2- 172.21.15.57:0/2797443709 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb3e40b0530 0x7fb3e4137e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:59.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.220+0000 7fb3e94cf640 1 -- 172.21.15.57:0/2797443709 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb3e40cbd20 msgr2=0x7fb3e4138340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:59.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.220+0000 7fb3e94cf640 1 --2- 172.21.15.57:0/2797443709 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb3e40cbd20 0x7fb3e4138340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:59.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.220+0000 7fb3e94cf640 1 -- 172.21.15.57:0/2797443709 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3d400a670 con 0x7fb3e40afa20 2023-10-19T13:04:59.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.220+0000 7fb3e8cce640 1 --2- 172.21.15.57:0/2797443709 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb3e40b0530 0x7fb3e4137e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:04:59.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.220+0000 7fb3e94cf640 1 --2- 172.21.15.57:0/2797443709 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3e40afa20 0x7fb3e40ae970 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7fb3e000c3c0 tx=0x7fb3e000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:59.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.220+0000 7fb3d9ffb640 1 -- 172.21.15.57:0/2797443709 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3e00052d0 con 0x7fb3e40afa20 2023-10-19T13:04:59.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.221+0000 7fb3d9ffb640 1 -- 172.21.15.57:0/2797443709 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fb3e0007020 con 0x7fb3e40afa20 2023-10-19T13:04:59.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.221+0000 7fb3d9ffb640 1 -- 172.21.15.57:0/2797443709 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3e0007470 con 0x7fb3e40afa20 2023-10-19T13:04:59.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.221+0000 7fb3eb733640 1 -- 172.21.15.57:0/2797443709 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3e4147690 con 0x7fb3e40afa20 2023-10-19T13:04:59.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.221+0000 7fb3eb733640 1 -- 172.21.15.57:0/2797443709 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3e40aeeb0 con 0x7fb3e40afa20 2023-10-19T13:04:59.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.222+0000 7fb3eb733640 1 -- 172.21.15.57:0/2797443709 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3e40af040 con 0x7fb3e40afa20 2023-10-19T13:04:59.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.222+0000 7fb3d9ffb640 1 -- 172.21.15.57:0/2797443709 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 79) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb3e00067d0 con 0x7fb3e40afa20 2023-10-19T13:04:59.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.223+0000 7fb3d9ffb640 1 --2- 172.21.15.57:0/2797443709 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fb3c806af60 0x7fb3c806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:59.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.223+0000 7fb3d9ffb640 1 -- 172.21.15.57:0/2797443709 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(156..156 src has 1..156) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fb3e0088e60 con 0x7fb3e40afa20 2023-10-19T13:04:59.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.223+0000 7fb3e8cce640 1 --2- 172.21.15.57:0/2797443709 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fb3c806af60 0x7fb3c806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:59.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.223+0000 7fb3e8cce640 1 --2- 172.21.15.57:0/2797443709 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fb3c806af60 0x7fb3c806d420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb3d4036000 tx=0x7fb3d4004730 comp rx=0 tx=0).ready entity=mgr.8419 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:59.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.227+0000 7fb3d9ffb640 1 -- 172.21.15.57:0/2797443709 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb3e0053d50 con 0x7fb3e40afa20 2023-10-19T13:04:59.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.403+0000 7fb3eb733640 1 -- 172.21.15.57:0/2797443709 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb3e40690e0 con 0x7fb3e40afa20 2023-10-19T13:04:59.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.404+0000 7fb3d9ffb640 1 -- 172.21.15.57:0/2797443709 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v157) v1 ==== 74+0+16787 (secure 0 0 0) 0x7fb3e0057370 con 0x7fb3e40afa20 2023-10-19T13:04:59.408 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:04:59.408 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":157,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T13:04:59.234270+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2023-10-19T13:04:56.433332+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"156","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2023-10-19T13:04:57.112749+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"157","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T13:04:59.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.406+0000 7fb3eb733640 1 -- 172.21.15.57:0/2797443709 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fb3c806af60 msgr2=0x7fb3c806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:59.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.406+0000 7fb3eb733640 1 --2- 172.21.15.57:0/2797443709 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fb3c806af60 0x7fb3c806d420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb3d4036000 tx=0x7fb3d4004730 comp rx=0 tx=0).stop 2023-10-19T13:04:59.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.406+0000 7fb3eb733640 1 -- 172.21.15.57:0/2797443709 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3e40afa20 msgr2=0x7fb3e40ae970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:59.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.406+0000 7fb3eb733640 1 --2- 172.21.15.57:0/2797443709 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3e40afa20 0x7fb3e40ae970 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7fb3e000c3c0 tx=0x7fb3e000c880 comp rx=0 tx=0).stop 2023-10-19T13:04:59.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.406+0000 7fb3eb733640 1 -- 172.21.15.57:0/2797443709 shutdown_connections 2023-10-19T13:04:59.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.406+0000 7fb3eb733640 1 --2- 172.21.15.57:0/2797443709 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fb3c806af60 0x7fb3c806d420 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:59.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.406+0000 7fb3eb733640 1 --2- 172.21.15.57:0/2797443709 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb3e40cbd20 0x7fb3e4138340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:59.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.406+0000 7fb3eb733640 1 --2- 172.21.15.57:0/2797443709 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb3e40b0530 0x7fb3e4137e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:59.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.406+0000 7fb3eb733640 1 --2- 172.21.15.57:0/2797443709 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3e40afa20 0x7fb3e40ae970 unknown :-1 s=CLOSED pgs=1750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:59.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.406+0000 7fb3eb733640 1 -- 172.21.15.57:0/2797443709 >> 172.21.15.57:0/2797443709 conn(0x7fb3e4069930 msgr2=0x7fb3e4150980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:59.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.407+0000 7fb3eb733640 1 -- 172.21.15.57:0/2797443709 shutdown_connections 2023-10-19T13:04:59.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.407+0000 7fb3eb733640 1 -- 172.21.15.57:0/2797443709 wait complete. 2023-10-19T13:04:59.420 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-19T13:04:59.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.529+0000 7f0fd29a0640 1 -- 172.21.15.57:0/2722371498 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0fcc15b2f0 msgr2=0x7f0fcc0cd350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:59.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.529+0000 7f0fd29a0640 1 --2- 172.21.15.57:0/2722371498 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0fcc15b2f0 0x7f0fcc0cd350 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc005fd0 tx=0x7f0fbc02c0e0 comp rx=0 tx=0).stop 2023-10-19T13:04:59.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.529+0000 7f0fd29a0640 1 -- 172.21.15.57:0/2722371498 shutdown_connections 2023-10-19T13:04:59.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.529+0000 7f0fd29a0640 1 --2- 172.21.15.57:0/2722371498 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0fcc15b2f0 0x7f0fcc0cd350 unknown :-1 s=CLOSED pgs=1752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:59.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.529+0000 7f0fd29a0640 1 -- 172.21.15.57:0/2722371498 >> 172.21.15.57:0/2722371498 conn(0x7f0fcc0d0c20 msgr2=0x7f0fcc0d1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:59.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.529+0000 7f0fd29a0640 1 -- 172.21.15.57:0/2722371498 shutdown_connections 2023-10-19T13:04:59.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.529+0000 7f0fd29a0640 1 -- 172.21.15.57:0/2722371498 wait complete. 2023-10-19T13:04:59.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.530+0000 7f0fd29a0640 1 Processor -- start 2023-10-19T13:04:59.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.530+0000 7f0fd29a0640 1 -- start start 2023-10-19T13:04:59.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.531+0000 7f0fd29a0640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0fcc0ce2f0 0x7f0fcc152620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:59.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.531+0000 7f0fd29a0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0fcc0d1740 0x7f0fcc0b5850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:59.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.531+0000 7f0fd29a0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0fcc15b2f0 0x7f0fcc0b5d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:59.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.531+0000 7f0fd29a0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fcc15d240 con 0x7f0fcc15b2f0 2023-10-19T13:04:59.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.531+0000 7f0fd29a0640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fcc152b60 con 0x7f0fcc0d1740 2023-10-19T13:04:59.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.531+0000 7f0fd29a0640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fcc15c630 con 0x7f0fcc0ce2f0 2023-10-19T13:04:59.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.531+0000 7f0fd219f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0fcc15b2f0 0x7f0fcc0b5d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:59.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.531+0000 7f0fd219f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0fcc15b2f0 0x7f0fcc0b5d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42770/0 (socket says 172.21.15.57:42770) 2023-10-19T13:04:59.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.531+0000 7f0fd219f640 1 -- 172.21.15.57:0/3369845753 learned_addr learned my addr 172.21.15.57:0/3369845753 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:59.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.532+0000 7f0fd119d640 1 --2- 172.21.15.57:0/3369845753 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0fcc0d1740 0x7f0fcc0b5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:59.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.532+0000 7f0fd199e640 1 --2- 172.21.15.57:0/3369845753 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0fcc0ce2f0 0x7f0fcc152620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:59.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.532+0000 7f0fd219f640 1 -- 172.21.15.57:0/3369845753 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0fcc0ce2f0 msgr2=0x7f0fcc152620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:59.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.532+0000 7f0fd219f640 1 --2- 172.21.15.57:0/3369845753 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0fcc0ce2f0 0x7f0fcc152620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:59.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.532+0000 7f0fd219f640 1 -- 172.21.15.57:0/3369845753 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0fcc0d1740 msgr2=0x7f0fcc0b5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:59.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.532+0000 7f0fd219f640 1 --2- 172.21.15.57:0/3369845753 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0fcc0d1740 0x7f0fcc0b5850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:59.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.532+0000 7f0fd219f640 1 -- 172.21.15.57:0/3369845753 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fbc009840 con 0x7f0fcc15b2f0 2023-10-19T13:04:59.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.532+0000 7f0fd219f640 1 --2- 172.21.15.57:0/3369845753 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0fcc15b2f0 0x7f0fcc0b5d90 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7f0fc400c430 tx=0x7f0fc400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:59.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.532+0000 7f0fca7fc640 1 -- 172.21.15.57:0/3369845753 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc4011790 con 0x7f0fcc15b2f0 2023-10-19T13:04:59.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.532+0000 7f0fca7fc640 1 -- 172.21.15.57:0/3369845753 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f0fc40119e0 con 0x7f0fcc15b2f0 2023-10-19T13:04:59.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.532+0000 7f0fca7fc640 1 -- 172.21.15.57:0/3369845753 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc4011e10 con 0x7f0fcc15b2f0 2023-10-19T13:04:59.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.532+0000 7f0fd29a0640 1 -- 172.21.15.57:0/3369845753 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fcc15c910 con 0x7f0fcc15b2f0 2023-10-19T13:04:59.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.532+0000 7f0fd29a0640 1 -- 172.21.15.57:0/3369845753 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fcc15ced0 con 0x7f0fcc15b2f0 2023-10-19T13:04:59.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.533+0000 7f0fd29a0640 1 -- 172.21.15.57:0/3369845753 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fcc066960 con 0x7f0fcc15b2f0 2023-10-19T13:04:59.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.534+0000 7f0fca7fc640 1 -- 172.21.15.57:0/3369845753 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 79) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f0fc401c460 con 0x7f0fcc15b2f0 2023-10-19T13:04:59.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.534+0000 7f0fca7fc640 1 --2- 172.21.15.57:0/3369845753 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f0fb806af90 0x7f0fb806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:59.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.534+0000 7f0fca7fc640 1 -- 172.21.15.57:0/3369845753 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f0fc4089950 con 0x7f0fcc15b2f0 2023-10-19T13:04:59.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.534+0000 7f0fd199e640 1 --2- 172.21.15.57:0/3369845753 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f0fb806af90 0x7f0fb806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:59.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.535+0000 7f0fd199e640 1 --2- 172.21.15.57:0/3369845753 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f0fb806af90 0x7f0fb806d450 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc035000 tx=0x7f0fbc004ab0 comp rx=0 tx=0).ready entity=mgr.8419 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:59.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.538+0000 7f0fca7fc640 1 -- 172.21.15.57:0/3369845753 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0fc4054820 con 0x7f0fcc15b2f0 2023-10-19T13:04:59.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.748+0000 7f0fd29a0640 1 -- 172.21.15.57:0/3369845753 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f0fcc04da70 con 0x7f0fcc15b2f0 2023-10-19T13:04:59.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.791+0000 7f0fca7fc640 1 -- 172.21.15.57:0/3369845753 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v62) v1 ==== 118+0+0 (secure 0 0 0) 0x7f0fc4057e40 con 0x7f0fcc15b2f0 2023-10-19T13:04:59.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.793+0000 7f0fd29a0640 1 -- 172.21.15.57:0/3369845753 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f0fb806af90 msgr2=0x7f0fb806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:59.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.793+0000 7f0fd29a0640 1 --2- 172.21.15.57:0/3369845753 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f0fb806af90 0x7f0fb806d450 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc035000 tx=0x7f0fbc004ab0 comp rx=0 tx=0).stop 2023-10-19T13:04:59.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.793+0000 7f0fd29a0640 1 -- 172.21.15.57:0/3369845753 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0fcc15b2f0 msgr2=0x7f0fcc0b5d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:59.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.793+0000 7f0fd29a0640 1 --2- 172.21.15.57:0/3369845753 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0fcc15b2f0 0x7f0fcc0b5d90 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7f0fc400c430 tx=0x7f0fc400c8f0 comp rx=0 tx=0).stop 2023-10-19T13:04:59.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.793+0000 7f0fd29a0640 1 -- 172.21.15.57:0/3369845753 shutdown_connections 2023-10-19T13:04:59.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.793+0000 7f0fd29a0640 1 --2- 172.21.15.57:0/3369845753 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f0fb806af90 0x7f0fb806d450 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:59.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.793+0000 7f0fd29a0640 1 --2- 172.21.15.57:0/3369845753 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0fcc15b2f0 0x7f0fcc0b5d90 unknown :-1 s=CLOSED pgs=1753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:59.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.793+0000 7f0fd29a0640 1 --2- 172.21.15.57:0/3369845753 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0fcc0d1740 0x7f0fcc0b5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:59.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.793+0000 7f0fd29a0640 1 --2- 172.21.15.57:0/3369845753 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0fcc0ce2f0 0x7f0fcc152620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:59.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.793+0000 7f0fd29a0640 1 -- 172.21.15.57:0/3369845753 >> 172.21.15.57:0/3369845753 conn(0x7f0fcc0d0c20 msgr2=0x7f0fcc0cceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:59.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.793+0000 7f0fd29a0640 1 -- 172.21.15.57:0/3369845753 shutdown_connections 2023-10-19T13:04:59.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.793+0000 7f0fd29a0640 1 -- 172.21.15.57:0/3369845753 wait complete. 2023-10-19T13:04:59.806 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-19T13:04:59.807 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-19T13:04:59.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.916+0000 7f362198a640 1 -- 172.21.15.57:0/3622843115 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3608003970 msgr2=0x7f3608003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:59.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.916+0000 7f362198a640 1 --2- 172.21.15.57:0/3622843115 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3608003970 0x7f3608003d90 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7f360c00c260 tx=0x7f36080057f0 comp rx=0 tx=0).stop 2023-10-19T13:04:59.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.916+0000 7f361b7fe640 1 -- 172.21.15.57:0/3622843115 reap_dead start 2023-10-19T13:04:59.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.916+0000 7f362198a640 1 -- 172.21.15.57:0/3622843115 shutdown_connections 2023-10-19T13:04:59.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.916+0000 7f362198a640 1 -- 172.21.15.57:0/3622843115 >> 172.21.15.57:0/3622843115 conn(0x7f361c069930 msgr2=0x7f361c069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:04:59.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.916+0000 7f362198a640 1 -- 172.21.15.57:0/3622843115 shutdown_connections 2023-10-19T13:04:59.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.916+0000 7f362198a640 1 -- 172.21.15.57:0/3622843115 wait complete. 2023-10-19T13:04:59.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.918+0000 7f362198a640 1 Processor -- start 2023-10-19T13:04:59.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.918+0000 7f362198a640 1 -- start start 2023-10-19T13:04:59.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.918+0000 7f362198a640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3608003970 0x7f361c13b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:59.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.918+0000 7f362198a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f361c0ab790 0x7f361c13f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:59.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.918+0000 7f362198a640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f361c1dbd30 0x7f361c140030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:59.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.918+0000 7f362198a640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f361c13b780 con 0x7f361c0ab790 2023-10-19T13:04:59.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.918+0000 7f362198a640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f361c1dc680 con 0x7f3608003970 2023-10-19T13:04:59.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.918+0000 7f362198a640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f361c1dc960 con 0x7f361c1dbd30 2023-10-19T13:04:59.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.918+0000 7f3613fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f361c0ab790 0x7f361c13f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:59.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.918+0000 7f3613fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f361c0ab790 0x7f361c13f510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42778/0 (socket says 172.21.15.57:42778) 2023-10-19T13:04:59.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.918+0000 7f3613fff640 1 -- 172.21.15.57:0/4090563046 learned_addr learned my addr 172.21.15.57:0/4090563046 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:04:59.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.919+0000 7f3613fff640 1 -- 172.21.15.57:0/4090563046 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f361c1dbd30 msgr2=0x7f361c140030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:59.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.919+0000 7f361affd640 1 --2- 172.21.15.57:0/4090563046 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3608003970 0x7f361c13b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:59.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.919+0000 7f3613fff640 1 --2- 172.21.15.57:0/4090563046 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f361c1dbd30 0x7f361c140030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:59.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.919+0000 7f3613fff640 1 -- 172.21.15.57:0/4090563046 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3608003970 msgr2=0x7f361c13b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:04:59.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.919+0000 7f3613fff640 1 --2- 172.21.15.57:0/4090563046 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3608003970 0x7f361c13b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:04:59.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.919+0000 7f3613fff640 1 -- 172.21.15.57:0/4090563046 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f361c13ea30 con 0x7f361c0ab790 2023-10-19T13:04:59.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.919+0000 7f361affd640 1 --2- 172.21.15.57:0/4090563046 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3608003970 0x7f361c13b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T13:04:59.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.919+0000 7f3613fff640 1 --2- 172.21.15.57:0/4090563046 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f361c0ab790 0x7f361c13f510 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7f3604005f60 tx=0x7f3604011c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:59.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.919+0000 7f3618ff9640 1 -- 172.21.15.57:0/4090563046 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f360400fd50 con 0x7f361c0ab790 2023-10-19T13:04:59.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.919+0000 7f3618ff9640 1 -- 172.21.15.57:0/4090563046 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f3604011e80 con 0x7f361c0ab790 2023-10-19T13:04:59.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.919+0000 7f362198a640 1 -- 172.21.15.57:0/4090563046 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f361c13ed10 con 0x7f361c0ab790 2023-10-19T13:04:59.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.919+0000 7f3618ff9640 1 -- 172.21.15.57:0/4090563046 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36040126b0 con 0x7f361c0ab790 2023-10-19T13:04:59.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.919+0000 7f362198a640 1 -- 172.21.15.57:0/4090563046 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f361c13fa50 con 0x7f361c0ab790 2023-10-19T13:04:59.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.920+0000 7f362198a640 1 -- 172.21.15.57:0/4090563046 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f361c0690e0 con 0x7f361c0ab790 2023-10-19T13:04:59.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.921+0000 7f3618ff9640 1 -- 172.21.15.57:0/4090563046 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 79) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f3604012810 con 0x7f361c0ab790 2023-10-19T13:04:59.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.921+0000 7f3618ff9640 1 --2- 172.21.15.57:0/4090563046 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f35fc06af60 0x7f35fc06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:04:59.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.921+0000 7f3618ff9640 1 -- 172.21.15.57:0/4090563046 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f360408a650 con 0x7f361c0ab790 2023-10-19T13:04:59.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.921+0000 7f361affd640 1 --2- 172.21.15.57:0/4090563046 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f35fc06af60 0x7f35fc06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:04:59.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.922+0000 7f361affd640 1 --2- 172.21.15.57:0/4090563046 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f35fc06af60 0x7f35fc06d420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f360c034000 tx=0x7f360c001a00 comp rx=0 tx=0).ready entity=mgr.8419 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:04:59.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:04:59.925+0000 7f3618ff9640 1 -- 172.21.15.57:0/4090563046 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f36040555d0 con 0x7f361c0ab790 2023-10-19T13:05:00.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:00.137+0000 7f362198a640 1 -- 172.21.15.57:0/4090563046 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f361c0cc450 con 0x7f361c0ab790 2023-10-19T13:05:00.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:00.802+0000 7f3618ff9640 1 -- 172.21.15.57:0/4090563046 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v63) v1 ==== 122+0+0 (secure 0 0 0) 0x7f3604058bf0 con 0x7f361c0ab790 2023-10-19T13:05:00.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:00.805+0000 7f362198a640 1 -- 172.21.15.57:0/4090563046 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f35fc06af60 msgr2=0x7f35fc06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:05:00.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:00.805+0000 7f362198a640 1 --2- 172.21.15.57:0/4090563046 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f35fc06af60 0x7f35fc06d420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f360c034000 tx=0x7f360c001a00 comp rx=0 tx=0).stop 2023-10-19T13:05:00.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:00.805+0000 7f362198a640 1 -- 172.21.15.57:0/4090563046 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f361c0ab790 msgr2=0x7f361c13f510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:05:00.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:00.805+0000 7f362198a640 1 --2- 172.21.15.57:0/4090563046 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f361c0ab790 0x7f361c13f510 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7f3604005f60 tx=0x7f3604011c30 comp rx=0 tx=0).stop 2023-10-19T13:05:00.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:00.805+0000 7f362198a640 1 -- 172.21.15.57:0/4090563046 shutdown_connections 2023-10-19T13:05:00.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:00.805+0000 7f362198a640 1 --2- 172.21.15.57:0/4090563046 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f35fc06af60 0x7f35fc06d420 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:00.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:00.805+0000 7f362198a640 1 --2- 172.21.15.57:0/4090563046 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f361c1dbd30 0x7f361c140030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:00.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:00.805+0000 7f362198a640 1 --2- 172.21.15.57:0/4090563046 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f361c0ab790 0x7f361c13f510 unknown :-1 s=CLOSED pgs=1756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:00.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:00.805+0000 7f362198a640 1 --2- 172.21.15.57:0/4090563046 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3608003970 0x7f361c13b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:00.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:00.805+0000 7f362198a640 1 -- 172.21.15.57:0/4090563046 >> 172.21.15.57:0/4090563046 conn(0x7f361c069930 msgr2=0x7f361c1df280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:05:00.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:00.805+0000 7f362198a640 1 -- 172.21.15.57:0/4090563046 shutdown_connections 2023-10-19T13:05:00.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:00.805+0000 7f362198a640 1 -- 172.21.15.57:0/4090563046 wait complete. 2023-10-19T13:05:00.821 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-19T13:05:00.821 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T13:05:00.937 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:00.934+0000 7f4351c61640 1 -- 172.21.15.57:0/1436053692 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f434c0cd6d0 msgr2=0x7f434c0cdb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:05:00.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:00.934+0000 7f4351c61640 1 --2- 172.21.15.57:0/1436053692 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f434c0cd6d0 0x7f434c0cdb30 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f433c009860 tx=0x7f433c02bc20 comp rx=0 tx=0).stop 2023-10-19T13:05:00.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:00.935+0000 7f4351c61640 1 -- 172.21.15.57:0/1436053692 shutdown_connections 2023-10-19T13:05:00.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:00.935+0000 7f4351c61640 1 --2- 172.21.15.57:0/1436053692 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f434c0cd6d0 0x7f434c0cdb30 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:00.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:00.935+0000 7f4351c61640 1 -- 172.21.15.57:0/1436053692 >> 172.21.15.57:0/1436053692 conn(0x7f434c0af7a0 msgr2=0x7f434c0afb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:05:00.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:00.935+0000 7f4351c61640 1 -- 172.21.15.57:0/1436053692 shutdown_connections 2023-10-19T13:05:00.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:00.935+0000 7f4351c61640 1 -- 172.21.15.57:0/1436053692 wait complete. 2023-10-19T13:05:00.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:00.936+0000 7f4351c61640 1 Processor -- start 2023-10-19T13:05:00.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:00.936+0000 7f4351c61640 1 -- start start 2023-10-19T13:05:00.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:00.937+0000 7f4351c61640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f434c069720 0x7f434c137830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:05:00.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:00.937+0000 7f4351c61640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f434c0cd6d0 0x7f434c13d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:05:00.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:00.937+0000 7f4351c61640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f434c142850 0x7f434c149f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:05:00.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:00.937+0000 7f4351c61640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f434c139a70 con 0x7f434c0cd6d0 2023-10-19T13:05:00.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:00.937+0000 7f4351c61640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f434c139d50 con 0x7f434c142850 2023-10-19T13:05:00.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:00.937+0000 7f4351c61640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f434c13a030 con 0x7f434c069720 2023-10-19T13:05:00.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:00.937+0000 7f434affd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f434c0cd6d0 0x7f434c13d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:05:00.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:00.937+0000 7f434affd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f434c0cd6d0 0x7f434c13d060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42794/0 (socket says 172.21.15.57:42794) 2023-10-19T13:05:00.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:00.937+0000 7f434affd640 1 -- 172.21.15.57:0/213449309 learned_addr learned my addr 172.21.15.57:0/213449309 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:05:00.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:00.937+0000 7f434bfff640 1 --2- 172.21.15.57:0/213449309 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f434c142850 0x7f434c149f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:05:00.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:00.937+0000 7f434b7fe640 1 --2- 172.21.15.57:0/213449309 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f434c069720 0x7f434c137830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:05:00.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:00.937+0000 7f434affd640 1 -- 172.21.15.57:0/213449309 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f434c069720 msgr2=0x7f434c137830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:05:00.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:00.937+0000 7f434affd640 1 --2- 172.21.15.57:0/213449309 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f434c069720 0x7f434c137830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:00.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:00.937+0000 7f434affd640 1 -- 172.21.15.57:0/213449309 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f434c142850 msgr2=0x7f434c149f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:05:00.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:00.937+0000 7f434affd640 1 --2- 172.21.15.57:0/213449309 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f434c142850 0x7f434c149f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:00.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:00.937+0000 7f434affd640 1 -- 172.21.15.57:0/213449309 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f433c009490 con 0x7f434c0cd6d0 2023-10-19T13:05:00.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:00.937+0000 7f434affd640 1 --2- 172.21.15.57:0/213449309 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f434c0cd6d0 0x7f434c13d060 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7f433c009980 tx=0x7f433c02dd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:05:00.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:00.937+0000 7f4333fff640 1 -- 172.21.15.57:0/213449309 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f433c005c90 con 0x7f434c0cd6d0 2023-10-19T13:05:00.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:00.938+0000 7f4333fff640 1 -- 172.21.15.57:0/213449309 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f433c005020 con 0x7f434c0cd6d0 2023-10-19T13:05:00.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:00.938+0000 7f4351c61640 1 -- 172.21.15.57:0/213449309 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f434c13a310 con 0x7f434c0cd6d0 2023-10-19T13:05:00.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:00.938+0000 7f4333fff640 1 -- 172.21.15.57:0/213449309 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f433c005430 con 0x7f434c0cd6d0 2023-10-19T13:05:00.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:00.938+0000 7f4351c61640 1 -- 172.21.15.57:0/213449309 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f434c137d70 con 0x7f434c0cd6d0 2023-10-19T13:05:00.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:00.939+0000 7f4351c61640 1 -- 172.21.15.57:0/213449309 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f434c13a830 con 0x7f434c0cd6d0 2023-10-19T13:05:00.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:00.939+0000 7f4333fff640 1 -- 172.21.15.57:0/213449309 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 79) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f433c003800 con 0x7f434c0cd6d0 2023-10-19T13:05:00.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:00.940+0000 7f4333fff640 1 --2- 172.21.15.57:0/213449309 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f433806af60 0x7f433806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:05:00.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:00.940+0000 7f4333fff640 1 -- 172.21.15.57:0/213449309 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f433c0abb00 con 0x7f434c0cd6d0 2023-10-19T13:05:00.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:00.940+0000 7f434b7fe640 1 --2- 172.21.15.57:0/213449309 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f433806af60 0x7f433806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:05:00.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:00.940+0000 7f434b7fe640 1 --2- 172.21.15.57:0/213449309 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f433806af60 0x7f433806d420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f43400011e0 tx=0x7f4340009040 comp rx=0 tx=0).ready entity=mgr.8419 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:05:00.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:00.944+0000 7f4333fff640 1 -- 172.21.15.57:0/213449309 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f434c13a830 con 0x7f434c0cd6d0 2023-10-19T13:05:01.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.155+0000 7f4351c61640 1 -- 172.21.15.57:0/213449309 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f434c13a830 con 0x7f434c0cd6d0 2023-10-19T13:05:01.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.155+0000 7f4333fff640 1 -- 172.21.15.57:0/213449309 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+4918 (secure 0 0 0) 0x7f434c13a830 con 0x7f434c0cd6d0 2023-10-19T13:05:01.158 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:05:01.159 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":63,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7733,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/1818414654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":1818414654},{"type":"v1","addr":"172.21.15.179:6836","nonce":1818414654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":8647,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/490712721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":490712721},{"type":"v1","addr":"172.21.15.57:6835","nonce":490712721}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":8656,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/2005811112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":2005811112},{"type":"v1","addr":"172.21.15.179:6837","nonce":2005811112}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T13:04:58.134572+0000","modified":"2023-10-19T13:05:00.138505+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9618},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9618":{"gid":9618,"name":"c","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.15.57:6833/1460919655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":1460919655},{"type":"v1","addr":"172.21.15.57:6833","nonce":1460919655}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2023-10-19T13:05:01.159 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 63 2023-10-19T13:05:01.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.157+0000 7f4351c61640 1 -- 172.21.15.57:0/213449309 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f433806af60 msgr2=0x7f433806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:05:01.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.157+0000 7f4351c61640 1 --2- 172.21.15.57:0/213449309 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f433806af60 0x7f433806d420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f43400011e0 tx=0x7f4340009040 comp rx=0 tx=0).stop 2023-10-19T13:05:01.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.157+0000 7f4351c61640 1 -- 172.21.15.57:0/213449309 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f434c0cd6d0 msgr2=0x7f434c13d060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:05:01.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.157+0000 7f4351c61640 1 --2- 172.21.15.57:0/213449309 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f434c0cd6d0 0x7f434c13d060 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7f433c009980 tx=0x7f433c02dd20 comp rx=0 tx=0).stop 2023-10-19T13:05:01.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.157+0000 7f4351c61640 1 -- 172.21.15.57:0/213449309 shutdown_connections 2023-10-19T13:05:01.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.157+0000 7f4351c61640 1 --2- 172.21.15.57:0/213449309 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f433806af60 0x7f433806d420 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:01.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.157+0000 7f4351c61640 1 --2- 172.21.15.57:0/213449309 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f434c142850 0x7f434c149f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:01.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.157+0000 7f4351c61640 1 --2- 172.21.15.57:0/213449309 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f434c0cd6d0 0x7f434c13d060 unknown :-1 s=CLOSED pgs=1758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:01.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.157+0000 7f4351c61640 1 --2- 172.21.15.57:0/213449309 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f434c069720 0x7f434c137830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:01.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.157+0000 7f4351c61640 1 -- 172.21.15.57:0/213449309 >> 172.21.15.57:0/213449309 conn(0x7f434c0af7a0 msgr2=0x7f434c140e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:05:01.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.157+0000 7f4351c61640 1 -- 172.21.15.57:0/213449309 shutdown_connections 2023-10-19T13:05:01.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.157+0000 7f4351c61640 1 -- 172.21.15.57:0/213449309 wait complete. 2023-10-19T13:05:01.170 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T13:05:01.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.278+0000 7f08f1ee5640 1 -- 172.21.15.57:0/1920315643 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f08ec0b0530 msgr2=0x7f08ec0b09c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:05:01.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.278+0000 7f08f1ee5640 1 --2- 172.21.15.57:0/1920315643 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f08ec0b0530 0x7f08ec0b09c0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f08e4005fd0 tx=0x7f08e402bfa0 comp rx=0 tx=0).stop 2023-10-19T13:05:01.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.278+0000 7f08f1ee5640 1 -- 172.21.15.57:0/1920315643 shutdown_connections 2023-10-19T13:05:01.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.278+0000 7f08f1ee5640 1 --2- 172.21.15.57:0/1920315643 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f08ec0b0530 0x7f08ec0b09c0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:01.282 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.278+0000 7f08f1ee5640 1 --2- 172.21.15.57:0/1920315643 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f08ec0cbd20 0x7f08ec158300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:01.282 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.278+0000 7f08f1ee5640 1 -- 172.21.15.57:0/1920315643 >> 172.21.15.57:0/1920315643 conn(0x7f08ec069930 msgr2=0x7f08ec069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:05:01.282 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.278+0000 7f08f1ee5640 1 -- 172.21.15.57:0/1920315643 shutdown_connections 2023-10-19T13:05:01.282 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.278+0000 7f08f1ee5640 1 -- 172.21.15.57:0/1920315643 wait complete. 2023-10-19T13:05:01.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.279+0000 7f08f1ee5640 1 Processor -- start 2023-10-19T13:05:01.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.279+0000 7f08f1ee5640 1 -- start start 2023-10-19T13:05:01.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.280+0000 7f08f1ee5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f08ec0afa20 0x7f08ec13a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:05:01.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.280+0000 7f08f1ee5640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f08ec0b0530 0x7f08ec13d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:05:01.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.280+0000 7f08f1ee5640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f08ec0cbd20 0x7f08ec14d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:05:01.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.280+0000 7f08f1ee5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08ec13db10 con 0x7f08ec0afa20 2023-10-19T13:05:01.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.280+0000 7f08f1ee5640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08ec13c530 con 0x7f08ec0b0530 2023-10-19T13:05:01.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.280+0000 7f08f1ee5640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08ec13c810 con 0x7f08ec0cbd20 2023-10-19T13:05:01.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.280+0000 7f08eb7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f08ec0afa20 0x7f08ec13a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:05:01.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.280+0000 7f08eb7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f08ec0afa20 0x7f08ec13a330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42806/0 (socket says 172.21.15.57:42806) 2023-10-19T13:05:01.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.280+0000 7f08eb7fe640 1 -- 172.21.15.57:0/3889302426 learned_addr learned my addr 172.21.15.57:0/3889302426 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:05:01.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.280+0000 7f08ebfff640 1 --2- 172.21.15.57:0/3889302426 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f08ec0cbd20 0x7f08ec14d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:05:01.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.280+0000 7f08eaffd640 1 --2- 172.21.15.57:0/3889302426 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f08ec0b0530 0x7f08ec13d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:05:01.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.280+0000 7f08eb7fe640 1 -- 172.21.15.57:0/3889302426 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f08ec0cbd20 msgr2=0x7f08ec14d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:05:01.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.280+0000 7f08eb7fe640 1 --2- 172.21.15.57:0/3889302426 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f08ec0cbd20 0x7f08ec14d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:01.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.280+0000 7f08eb7fe640 1 -- 172.21.15.57:0/3889302426 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f08ec0b0530 msgr2=0x7f08ec13d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:05:01.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.280+0000 7f08eb7fe640 1 --2- 172.21.15.57:0/3889302426 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f08ec0b0530 0x7f08ec13d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:01.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.280+0000 7f08eb7fe640 1 -- 172.21.15.57:0/3889302426 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08e4009840 con 0x7f08ec0afa20 2023-10-19T13:05:01.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.280+0000 7f08eaffd640 1 --2- 172.21.15.57:0/3889302426 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f08ec0b0530 0x7f08ec13d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:05:01.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.281+0000 7f08eb7fe640 1 --2- 172.21.15.57:0/3889302426 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f08ec0afa20 0x7f08ec13a330 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7f08dc00ea90 tx=0x7f08dc00ef50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:05:01.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.281+0000 7f08d3fff640 1 -- 172.21.15.57:0/3889302426 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08dc00cc80 con 0x7f08ec0afa20 2023-10-19T13:05:01.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.281+0000 7f08d3fff640 1 -- 172.21.15.57:0/3889302426 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f08dc016410 con 0x7f08ec0afa20 2023-10-19T13:05:01.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.281+0000 7f08f1ee5640 1 -- 172.21.15.57:0/3889302426 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08ec13caf0 con 0x7f08ec0afa20 2023-10-19T13:05:01.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.281+0000 7f08d3fff640 1 -- 172.21.15.57:0/3889302426 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08dc016820 con 0x7f08ec0afa20 2023-10-19T13:05:01.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.281+0000 7f08f1ee5640 1 -- 172.21.15.57:0/3889302426 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08ec13d0b0 con 0x7f08ec0afa20 2023-10-19T13:05:01.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.282+0000 7f08f1ee5640 1 -- 172.21.15.57:0/3889302426 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08ec13d270 con 0x7f08ec0afa20 2023-10-19T13:05:01.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.282+0000 7f08d3fff640 1 -- 172.21.15.57:0/3889302426 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 79) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f08dc016980 con 0x7f08ec0afa20 2023-10-19T13:05:01.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.283+0000 7f08d3fff640 1 --2- 172.21.15.57:0/3889302426 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f08d806af60 0x7f08d806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:05:01.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.283+0000 7f08d3fff640 1 -- 172.21.15.57:0/3889302426 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f08dc08a020 con 0x7f08ec0afa20 2023-10-19T13:05:01.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.283+0000 7f08eaffd640 1 --2- 172.21.15.57:0/3889302426 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f08d806af60 0x7f08d806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:05:01.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.283+0000 7f08eaffd640 1 --2- 172.21.15.57:0/3889302426 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f08d806af60 0x7f08d806d420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f08ec0ac310 tx=0x7f08e0009040 comp rx=0 tx=0).ready entity=mgr.8419 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:05:01.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.287+0000 7f08d3fff640 1 -- 172.21.15.57:0/3889302426 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f08dc054d70 con 0x7f08ec0afa20 2023-10-19T13:05:01.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.463+0000 7f08f1ee5640 1 -- 172.21.15.57:0/3889302426 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f08ec0690e0 con 0x7f08ec0afa20 2023-10-19T13:05:01.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.464+0000 7f08d3fff640 1 -- 172.21.15.57:0/3889302426 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v157) v1 ==== 74+0+16787 (secure 0 0 0) 0x7f08dc01b020 con 0x7f08ec0afa20 2023-10-19T13:05:01.467 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:05:01.467 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":157,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T13:04:59.234270+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2023-10-19T13:04:56.433332+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"156","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2023-10-19T13:04:57.112749+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"157","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T13:05:01.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.466+0000 7f08f1ee5640 1 -- 172.21.15.57:0/3889302426 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f08d806af60 msgr2=0x7f08d806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:05:01.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.466+0000 7f08f1ee5640 1 --2- 172.21.15.57:0/3889302426 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f08d806af60 0x7f08d806d420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f08ec0ac310 tx=0x7f08e0009040 comp rx=0 tx=0).stop 2023-10-19T13:05:01.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.466+0000 7f08f1ee5640 1 -- 172.21.15.57:0/3889302426 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f08ec0afa20 msgr2=0x7f08ec13a330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:05:01.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.466+0000 7f08f1ee5640 1 --2- 172.21.15.57:0/3889302426 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f08ec0afa20 0x7f08ec13a330 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7f08dc00ea90 tx=0x7f08dc00ef50 comp rx=0 tx=0).stop 2023-10-19T13:05:01.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.466+0000 7f08f1ee5640 1 -- 172.21.15.57:0/3889302426 shutdown_connections 2023-10-19T13:05:01.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.466+0000 7f08f1ee5640 1 --2- 172.21.15.57:0/3889302426 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f08d806af60 0x7f08d806d420 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:01.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.466+0000 7f08f1ee5640 1 --2- 172.21.15.57:0/3889302426 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f08ec0cbd20 0x7f08ec14d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:01.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.466+0000 7f08f1ee5640 1 --2- 172.21.15.57:0/3889302426 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f08ec0b0530 0x7f08ec13d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:01.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.466+0000 7f08f1ee5640 1 --2- 172.21.15.57:0/3889302426 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f08ec0afa20 0x7f08ec13a330 unknown :-1 s=CLOSED pgs=1760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:01.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.466+0000 7f08f1ee5640 1 -- 172.21.15.57:0/3889302426 >> 172.21.15.57:0/3889302426 conn(0x7f08ec069930 msgr2=0x7f08ec06c060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:05:01.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.466+0000 7f08f1ee5640 1 -- 172.21.15.57:0/3889302426 shutdown_connections 2023-10-19T13:05:01.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.466+0000 7f08f1ee5640 1 -- 172.21.15.57:0/3889302426 wait complete. 2023-10-19T13:05:01.479 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-19T13:05:01.480 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-19T13:05:01.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.586+0000 7f40f9cde640 1 -- 172.21.15.57:0/3097867210 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f40f40cbd20 msgr2=0x7f40f4158300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:05:01.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.586+0000 7f40f9cde640 1 --2- 172.21.15.57:0/3097867210 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f40f40cbd20 0x7f40f4158300 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7f40f40b10f0 tx=0x7f40e402d0a0 comp rx=0 tx=0).stop 2023-10-19T13:05:01.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.586+0000 7f40f9cde640 1 -- 172.21.15.57:0/3097867210 shutdown_connections 2023-10-19T13:05:01.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.586+0000 7f40f9cde640 1 --2- 172.21.15.57:0/3097867210 >> v2:172.21.15.179:3300/0 conn(0x7f40f40b0530 0x7f40f40b09a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:01.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.586+0000 7f40f9cde640 1 --2- 172.21.15.57:0/3097867210 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f40f40cbd20 0x7f40f4158300 unknown :-1 s=CLOSED pgs=1762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:01.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.586+0000 7f40f9cde640 1 -- 172.21.15.57:0/3097867210 >> 172.21.15.57:0/3097867210 conn(0x7f40f4069930 msgr2=0x7f40f4069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:05:01.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.586+0000 7f40f9cde640 1 -- 172.21.15.57:0/3097867210 shutdown_connections 2023-10-19T13:05:01.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.586+0000 7f40f9cde640 1 -- 172.21.15.57:0/3097867210 wait complete. 2023-10-19T13:05:01.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.587+0000 7f40f9cde640 1 Processor -- start 2023-10-19T13:05:01.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.587+0000 7f40f9cde640 1 -- start start 2023-10-19T13:05:01.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.588+0000 7f40f9cde640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f40f40afa20 0x7f40f4147ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:05:01.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.588+0000 7f40f9cde640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f40f40b0530 0x7f40f414dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:05:01.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.588+0000 7f40f9cde640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f40f40cbd20 0x7f40f40ad390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:05:01.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.588+0000 7f40f9cde640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40f414bc40 con 0x7f40f40afa20 2023-10-19T13:05:01.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.588+0000 7f40f9cde640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40f41e2d80 con 0x7f40f40cbd20 2023-10-19T13:05:01.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.588+0000 7f40f9cde640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40f4149e20 con 0x7f40f40b0530 2023-10-19T13:05:01.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.588+0000 7f40f37fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f40f40afa20 0x7f40f4147ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:05:01.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.588+0000 7f40f37fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f40f40afa20 0x7f40f4147ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42834/0 (socket says 172.21.15.57:42834) 2023-10-19T13:05:01.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.588+0000 7f40f37fe640 1 -- 172.21.15.57:0/514952644 learned_addr learned my addr 172.21.15.57:0/514952644 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:05:01.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.588+0000 7f40f3fff640 1 --2- 172.21.15.57:0/514952644 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f40f40cbd20 0x7f40f40ad390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:05:01.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.589+0000 7f40f2ffd640 1 --2- 172.21.15.57:0/514952644 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f40f40b0530 0x7f40f414dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:05:01.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.589+0000 7f40f37fe640 1 -- 172.21.15.57:0/514952644 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f40f40b0530 msgr2=0x7f40f414dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:05:01.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.589+0000 7f40f37fe640 1 --2- 172.21.15.57:0/514952644 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f40f40b0530 0x7f40f414dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:01.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.589+0000 7f40f37fe640 1 -- 172.21.15.57:0/514952644 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f40f40cbd20 msgr2=0x7f40f40ad390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:05:01.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.589+0000 7f40f37fe640 1 --2- 172.21.15.57:0/514952644 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f40f40cbd20 0x7f40f40ad390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:01.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.589+0000 7f40f37fe640 1 -- 172.21.15.57:0/514952644 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40e400a5a0 con 0x7f40f40afa20 2023-10-19T13:05:01.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.589+0000 7f40f2ffd640 1 --2- 172.21.15.57:0/514952644 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f40f40b0530 0x7f40f414dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T13:05:01.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.589+0000 7f40f37fe640 1 --2- 172.21.15.57:0/514952644 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f40f40afa20 0x7f40f4147ec0 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7f40e4000c00 tx=0x7f40e4036cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:05:01.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.589+0000 7f40dbfff640 1 -- 172.21.15.57:0/514952644 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40e4003a00 con 0x7f40f40afa20 2023-10-19T13:05:01.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.589+0000 7f40dbfff640 1 -- 172.21.15.57:0/514952644 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f40e4003c70 con 0x7f40f40afa20 2023-10-19T13:05:01.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.589+0000 7f40f9cde640 1 -- 172.21.15.57:0/514952644 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40f41e2170 con 0x7f40f40afa20 2023-10-19T13:05:01.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.589+0000 7f40dbfff640 1 -- 172.21.15.57:0/514952644 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40e4005c90 con 0x7f40f40afa20 2023-10-19T13:05:01.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.589+0000 7f40f9cde640 1 -- 172.21.15.57:0/514952644 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40f41e2730 con 0x7f40f40afa20 2023-10-19T13:05:01.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.590+0000 7f40f9cde640 1 -- 172.21.15.57:0/514952644 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40f40690e0 con 0x7f40f40afa20 2023-10-19T13:05:01.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.590+0000 7f40dbfff640 1 -- 172.21.15.57:0/514952644 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 79) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f40e403c070 con 0x7f40f40afa20 2023-10-19T13:05:01.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.591+0000 7f40dbfff640 1 --2- 172.21.15.57:0/514952644 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f40e006af60 0x7f40e006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:05:01.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.591+0000 7f40dbfff640 1 -- 172.21.15.57:0/514952644 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f40e40ac9e0 con 0x7f40f40afa20 2023-10-19T13:05:01.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.591+0000 7f40f2ffd640 1 --2- 172.21.15.57:0/514952644 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f40e006af60 0x7f40e006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:05:01.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.592+0000 7f40f2ffd640 1 --2- 172.21.15.57:0/514952644 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f40e006af60 0x7f40e006d420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f40f4157540 tx=0x7f40e8009040 comp rx=0 tx=0).ready entity=mgr.8419 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:05:01.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.595+0000 7f40dbfff640 1 -- 172.21.15.57:0/514952644 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f40e4077940 con 0x7f40f40afa20 2023-10-19T13:05:01.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.768+0000 7f40f9cde640 1 -- 172.21.15.57:0/514952644 --> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f40f40cc100 con 0x7f40e006af60 2023-10-19T13:05:01.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.777+0000 7f40dbfff640 1 -- 172.21.15.57:0/514952644 <== mgr.8419 v2:172.21.15.179:6832/2581690007 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289976 (secure 0 0 0) 0x7f40e8001ca0 con 0x7f40e006af60 2023-10-19T13:05:01.782 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:05:01.784 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2023-10-19T13:05:01.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.780+0000 7f40f9cde640 1 -- 172.21.15.57:0/514952644 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f40e006af60 msgr2=0x7f40e006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:05:01.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.780+0000 7f40f9cde640 1 --2- 172.21.15.57:0/514952644 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f40e006af60 0x7f40e006d420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f40f4157540 tx=0x7f40e8009040 comp rx=0 tx=0).stop 2023-10-19T13:05:01.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.780+0000 7f40f9cde640 1 -- 172.21.15.57:0/514952644 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f40f40afa20 msgr2=0x7f40f4147ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:05:01.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.780+0000 7f40f9cde640 1 --2- 172.21.15.57:0/514952644 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f40f40afa20 0x7f40f4147ec0 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7f40e4000c00 tx=0x7f40e4036cd0 comp rx=0 tx=0).stop 2023-10-19T13:05:01.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.780+0000 7f40f9cde640 1 -- 172.21.15.57:0/514952644 shutdown_connections 2023-10-19T13:05:01.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.780+0000 7f40f9cde640 1 --2- 172.21.15.57:0/514952644 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f40e006af60 0x7f40e006d420 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:01.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.780+0000 7f40f9cde640 1 --2- 172.21.15.57:0/514952644 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f40f40cbd20 0x7f40f40ad390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:01.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.780+0000 7f40f9cde640 1 --2- 172.21.15.57:0/514952644 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f40f40b0530 0x7f40f414dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:01.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.780+0000 7f40f9cde640 1 --2- 172.21.15.57:0/514952644 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f40f40afa20 0x7f40f4147ec0 unknown :-1 s=CLOSED pgs=1763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:01.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.780+0000 7f40f9cde640 1 -- 172.21.15.57:0/514952644 >> 172.21.15.57:0/514952644 conn(0x7f40f4069930 msgr2=0x7f40f40ae700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:05:01.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.780+0000 7f40f9cde640 1 -- 172.21.15.57:0/514952644 shutdown_connections 2023-10-19T13:05:01.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.780+0000 7f40f9cde640 1 -- 172.21.15.57:0/514952644 wait complete. 2023-10-19T13:05:01.793 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2023-10-19T13:05:00.298652+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1346,"num_read_kb":2770,"num_write":453,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3125,"ondisk_log_size":3125,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":142,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":345032,"kb_used_data":16468,"kb_used_omap":12,"kb_used_meta":328435,"kb_avail":754629688,"statfs":{"total":773094113280,"available":772740800512,"internally_reserved":0,"allocated":16863232,"data_stored":9084489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":336318009},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001037"},"pg_stats":[{"pgid":"20.3f","version":"0'0","reported_seq":11,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.242336+0000","last_change":"2023-10-19T13:04:58.242023+0000","last_active":"2023-10-19T13:04:59.242336+0000","last_peered":"2023-10-19T13:04:59.242336+0000","last_clean":"2023-10-19T13:04:59.242336+0000","last_became_active":"2023-10-19T13:04:58.241755+0000","last_became_peered":"2023-10-19T13:04:58.241755+0000","last_unstale":"2023-10-19T13:04:59.242336+0000","last_undegraded":"2023-10-19T13:04:59.242336+0000","last_fullsized":"2023-10-19T13:04:59.242336+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:09:10.121831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.3e","version":"0'0","reported_seq":11,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.245756+0000","last_change":"2023-10-19T13:04:58.250174+0000","last_active":"2023-10-19T13:04:59.245756+0000","last_peered":"2023-10-19T13:04:59.245756+0000","last_clean":"2023-10-19T13:04:59.245756+0000","last_became_active":"2023-10-19T13:04:58.249870+0000","last_became_peered":"2023-10-19T13:04:58.249870+0000","last_unstale":"2023-10-19T13:04:59.245756+0000","last_undegraded":"2023-10-19T13:04:59.245756+0000","last_fullsized":"2023-10-19T13:04:59.245756+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:04:35.399006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.3d","version":"0'0","reported_seq":9,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.260326+0000","last_change":"2023-10-19T13:04:58.256603+0000","last_active":"2023-10-19T13:04:58.260326+0000","last_peered":"2023-10-19T13:04:58.260326+0000","last_clean":"2023-10-19T13:04:58.260326+0000","last_became_active":"2023-10-19T13:04:58.256283+0000","last_became_peered":"2023-10-19T13:04:58.256283+0000","last_unstale":"2023-10-19T13:04:58.260326+0000","last_undegraded":"2023-10-19T13:04:58.260326+0000","last_fullsized":"2023-10-19T13:04:58.260326+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:42:37.465815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.3c","version":"0'0","reported_seq":11,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.245677+0000","last_change":"2023-10-19T13:04:58.252328+0000","last_active":"2023-10-19T13:04:59.245677+0000","last_peered":"2023-10-19T13:04:59.245677+0000","last_clean":"2023-10-19T13:04:59.245677+0000","last_became_active":"2023-10-19T13:04:58.252039+0000","last_became_peered":"2023-10-19T13:04:58.252039+0000","last_unstale":"2023-10-19T13:04:59.245677+0000","last_undegraded":"2023-10-19T13:04:59.245677+0000","last_fullsized":"2023-10-19T13:04:59.245677+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:07:16.246850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.3b","version":"0'0","reported_seq":11,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.684863+0000","last_change":"2023-10-19T13:04:58.246606+0000","last_active":"2023-10-19T13:04:59.684863+0000","last_peered":"2023-10-19T13:04:59.684863+0000","last_clean":"2023-10-19T13:04:59.684863+0000","last_became_active":"2023-10-19T13:04:58.246413+0000","last_became_peered":"2023-10-19T13:04:58.246413+0000","last_unstale":"2023-10-19T13:04:59.684863+0000","last_undegraded":"2023-10-19T13:04:59.684863+0000","last_fullsized":"2023-10-19T13:04:59.684863+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:18:09.376247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.3a","version":"0'0","reported_seq":9,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.260314+0000","last_change":"2023-10-19T13:04:58.252857+0000","last_active":"2023-10-19T13:04:58.260314+0000","last_peered":"2023-10-19T13:04:58.260314+0000","last_clean":"2023-10-19T13:04:58.260314+0000","last_became_active":"2023-10-19T13:04:58.252581+0000","last_became_peered":"2023-10-19T13:04:58.252581+0000","last_unstale":"2023-10-19T13:04:58.260314+0000","last_undegraded":"2023-10-19T13:04:58.260314+0000","last_fullsized":"2023-10-19T13:04:58.260314+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:01:25.818792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.39","version":"0'0","reported_seq":9,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.256171+0000","last_change":"2023-10-19T13:04:58.253657+0000","last_active":"2023-10-19T13:04:58.256171+0000","last_peered":"2023-10-19T13:04:58.256171+0000","last_clean":"2023-10-19T13:04:58.256171+0000","last_became_active":"2023-10-19T13:04:58.253085+0000","last_became_peered":"2023-10-19T13:04:58.253085+0000","last_unstale":"2023-10-19T13:04:58.256171+0000","last_undegraded":"2023-10-19T13:04:58.256171+0000","last_fullsized":"2023-10-19T13:04:58.256171+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:29:18.521999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.38","version":"0'0","reported_seq":11,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.246141+0000","last_change":"2023-10-19T13:04:58.246792+0000","last_active":"2023-10-19T13:04:59.246141+0000","last_peered":"2023-10-19T13:04:59.246141+0000","last_clean":"2023-10-19T13:04:59.246141+0000","last_became_active":"2023-10-19T13:04:58.246438+0000","last_became_peered":"2023-10-19T13:04:58.246438+0000","last_unstale":"2023-10-19T13:04:59.246141+0000","last_undegraded":"2023-10-19T13:04:59.246141+0000","last_fullsized":"2023-10-19T13:04:59.246141+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:43:40.413015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.37","version":"0'0","reported_seq":9,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.260105+0000","last_change":"2023-10-19T13:04:58.251146+0000","last_active":"2023-10-19T13:04:58.260105+0000","last_peered":"2023-10-19T13:04:58.260105+0000","last_clean":"2023-10-19T13:04:58.260105+0000","last_became_active":"2023-10-19T13:04:58.250871+0000","last_became_peered":"2023-10-19T13:04:58.250871+0000","last_unstale":"2023-10-19T13:04:58.260105+0000","last_undegraded":"2023-10-19T13:04:58.260105+0000","last_fullsized":"2023-10-19T13:04:58.260105+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:00:57.974372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.36","version":"0'0","reported_seq":11,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:05:00.280520+0000","last_change":"2023-10-19T13:04:58.247772+0000","last_active":"2023-10-19T13:05:00.280520+0000","last_peered":"2023-10-19T13:05:00.280520+0000","last_clean":"2023-10-19T13:05:00.280520+0000","last_became_active":"2023-10-19T13:04:58.247449+0000","last_became_peered":"2023-10-19T13:04:58.247449+0000","last_unstale":"2023-10-19T13:05:00.280520+0000","last_undegraded":"2023-10-19T13:05:00.280520+0000","last_fullsized":"2023-10-19T13:05:00.280520+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:47:36.006848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.35","version":"0'0","reported_seq":11,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.245615+0000","last_change":"2023-10-19T13:04:58.261170+0000","last_active":"2023-10-19T13:04:59.245615+0000","last_peered":"2023-10-19T13:04:59.245615+0000","last_clean":"2023-10-19T13:04:59.245615+0000","last_became_active":"2023-10-19T13:04:58.260860+0000","last_became_peered":"2023-10-19T13:04:58.260860+0000","last_unstale":"2023-10-19T13:04:59.245615+0000","last_undegraded":"2023-10-19T13:04:59.245615+0000","last_fullsized":"2023-10-19T13:04:59.245615+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:37:30.289123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.34","version":"0'0","reported_seq":11,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.242514+0000","last_change":"2023-10-19T13:04:58.251636+0000","last_active":"2023-10-19T13:04:59.242514+0000","last_peered":"2023-10-19T13:04:59.242514+0000","last_clean":"2023-10-19T13:04:59.242514+0000","last_became_active":"2023-10-19T13:04:58.251380+0000","last_became_peered":"2023-10-19T13:04:58.251380+0000","last_unstale":"2023-10-19T13:04:59.242514+0000","last_undegraded":"2023-10-19T13:04:59.242514+0000","last_fullsized":"2023-10-19T13:04:59.242514+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:34:57.252659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.33","version":"0'0","reported_seq":9,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.260369+0000","last_change":"2023-10-19T13:04:58.253732+0000","last_active":"2023-10-19T13:04:58.260369+0000","last_peered":"2023-10-19T13:04:58.260369+0000","last_clean":"2023-10-19T13:04:58.260369+0000","last_became_active":"2023-10-19T13:04:58.253503+0000","last_became_peered":"2023-10-19T13:04:58.253503+0000","last_unstale":"2023-10-19T13:04:58.260369+0000","last_undegraded":"2023-10-19T13:04:58.260369+0000","last_fullsized":"2023-10-19T13:04:58.260369+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:05:39.299198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.32","version":"0'0","reported_seq":11,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.242430+0000","last_change":"2023-10-19T13:04:58.243157+0000","last_active":"2023-10-19T13:04:59.242430+0000","last_peered":"2023-10-19T13:04:59.242430+0000","last_clean":"2023-10-19T13:04:59.242430+0000","last_became_active":"2023-10-19T13:04:58.242897+0000","last_became_peered":"2023-10-19T13:04:58.242897+0000","last_unstale":"2023-10-19T13:04:59.242430+0000","last_undegraded":"2023-10-19T13:04:59.242430+0000","last_fullsized":"2023-10-19T13:04:59.242430+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:30:14.023824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.31","version":"0'0","reported_seq":11,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.242133+0000","last_change":"2023-10-19T13:04:58.242545+0000","last_active":"2023-10-19T13:04:59.242133+0000","last_peered":"2023-10-19T13:04:59.242133+0000","last_clean":"2023-10-19T13:04:59.242133+0000","last_became_active":"2023-10-19T13:04:58.242284+0000","last_became_peered":"2023-10-19T13:04:58.242284+0000","last_unstale":"2023-10-19T13:04:59.242133+0000","last_undegraded":"2023-10-19T13:04:59.242133+0000","last_fullsized":"2023-10-19T13:04:59.242133+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:17:46.270258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.30","version":"0'0","reported_seq":9,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.259905+0000","last_change":"2023-10-19T13:04:58.250297+0000","last_active":"2023-10-19T13:04:58.259905+0000","last_peered":"2023-10-19T13:04:58.259905+0000","last_clean":"2023-10-19T13:04:58.259905+0000","last_became_active":"2023-10-19T13:04:58.249934+0000","last_became_peered":"2023-10-19T13:04:58.249934+0000","last_unstale":"2023-10-19T13:04:58.259905+0000","last_undegraded":"2023-10-19T13:04:58.259905+0000","last_fullsized":"2023-10-19T13:04:58.259905+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:20:17.526434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.2f","version":"0'0","reported_seq":9,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.260089+0000","last_change":"2023-10-19T13:04:58.251059+0000","last_active":"2023-10-19T13:04:58.260089+0000","last_peered":"2023-10-19T13:04:58.260089+0000","last_clean":"2023-10-19T13:04:58.260089+0000","last_became_active":"2023-10-19T13:04:58.250807+0000","last_became_peered":"2023-10-19T13:04:58.250807+0000","last_unstale":"2023-10-19T13:04:58.260089+0000","last_undegraded":"2023-10-19T13:04:58.260089+0000","last_fullsized":"2023-10-19T13:04:58.260089+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:52:57.156027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.2e","version":"0'0","reported_seq":11,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.684870+0000","last_change":"2023-10-19T13:04:58.250958+0000","last_active":"2023-10-19T13:04:59.684870+0000","last_peered":"2023-10-19T13:04:59.684870+0000","last_clean":"2023-10-19T13:04:59.684870+0000","last_became_active":"2023-10-19T13:04:58.250144+0000","last_became_peered":"2023-10-19T13:04:58.250144+0000","last_unstale":"2023-10-19T13:04:59.684870+0000","last_undegraded":"2023-10-19T13:04:59.684870+0000","last_fullsized":"2023-10-19T13:04:59.684870+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:25:51.958820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.2d","version":"0'0","reported_seq":9,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.254463+0000","last_change":"2023-10-19T13:04:58.247906+0000","last_active":"2023-10-19T13:04:58.254463+0000","last_peered":"2023-10-19T13:04:58.254463+0000","last_clean":"2023-10-19T13:04:58.254463+0000","last_became_active":"2023-10-19T13:04:58.247629+0000","last_became_peered":"2023-10-19T13:04:58.247629+0000","last_unstale":"2023-10-19T13:04:58.254463+0000","last_undegraded":"2023-10-19T13:04:58.254463+0000","last_fullsized":"2023-10-19T13:04:58.254463+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:34:12.330622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.2c","version":"0'0","reported_seq":11,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.684881+0000","last_change":"2023-10-19T13:04:58.255611+0000","last_active":"2023-10-19T13:04:59.684881+0000","last_peered":"2023-10-19T13:04:59.684881+0000","last_clean":"2023-10-19T13:04:59.684881+0000","last_became_active":"2023-10-19T13:04:58.255324+0000","last_became_peered":"2023-10-19T13:04:58.255324+0000","last_unstale":"2023-10-19T13:04:59.684881+0000","last_undegraded":"2023-10-19T13:04:59.684881+0000","last_fullsized":"2023-10-19T13:04:59.684881+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:45:55.505917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.2b","version":"0'0","reported_seq":11,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.245583+0000","last_change":"2023-10-19T13:04:58.258147+0000","last_active":"2023-10-19T13:04:59.245583+0000","last_peered":"2023-10-19T13:04:59.245583+0000","last_clean":"2023-10-19T13:04:59.245583+0000","last_became_active":"2023-10-19T13:04:58.257822+0000","last_became_peered":"2023-10-19T13:04:58.257822+0000","last_unstale":"2023-10-19T13:04:59.245583+0000","last_undegraded":"2023-10-19T13:04:59.245583+0000","last_fullsized":"2023-10-19T13:04:59.245583+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:47:10.547688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.2a","version":"0'0","reported_seq":11,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.242265+0000","last_change":"2023-10-19T13:04:58.248348+0000","last_active":"2023-10-19T13:04:59.242265+0000","last_peered":"2023-10-19T13:04:59.242265+0000","last_clean":"2023-10-19T13:04:59.242265+0000","last_became_active":"2023-10-19T13:04:58.248093+0000","last_became_peered":"2023-10-19T13:04:58.248093+0000","last_unstale":"2023-10-19T13:04:59.242265+0000","last_undegraded":"2023-10-19T13:04:59.242265+0000","last_fullsized":"2023-10-19T13:04:59.242265+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:01:09.959518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.29","version":"0'0","reported_seq":11,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.245433+0000","last_change":"2023-10-19T13:04:58.257763+0000","last_active":"2023-10-19T13:04:59.245433+0000","last_peered":"2023-10-19T13:04:59.245433+0000","last_clean":"2023-10-19T13:04:59.245433+0000","last_became_active":"2023-10-19T13:04:58.257164+0000","last_became_peered":"2023-10-19T13:04:58.257164+0000","last_unstale":"2023-10-19T13:04:59.245433+0000","last_undegraded":"2023-10-19T13:04:59.245433+0000","last_fullsized":"2023-10-19T13:04:59.245433+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:55:49.837946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.28","version":"0'0","reported_seq":9,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.255484+0000","last_change":"2023-10-19T13:04:58.248215+0000","last_active":"2023-10-19T13:04:58.255484+0000","last_peered":"2023-10-19T13:04:58.255484+0000","last_clean":"2023-10-19T13:04:58.255484+0000","last_became_active":"2023-10-19T13:04:58.247945+0000","last_became_peered":"2023-10-19T13:04:58.247945+0000","last_unstale":"2023-10-19T13:04:58.255484+0000","last_undegraded":"2023-10-19T13:04:58.255484+0000","last_fullsized":"2023-10-19T13:04:58.255484+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:42:37.465815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.27","version":"0'0","reported_seq":9,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.259776+0000","last_change":"2023-10-19T13:04:58.254641+0000","last_active":"2023-10-19T13:04:58.259776+0000","last_peered":"2023-10-19T13:04:58.259776+0000","last_clean":"2023-10-19T13:04:58.259776+0000","last_became_active":"2023-10-19T13:04:58.254378+0000","last_became_peered":"2023-10-19T13:04:58.254378+0000","last_unstale":"2023-10-19T13:04:58.259776+0000","last_undegraded":"2023-10-19T13:04:58.259776+0000","last_fullsized":"2023-10-19T13:04:58.259776+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:35:32.055794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.26","version":"0'0","reported_seq":9,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.263757+0000","last_change":"2023-10-19T13:04:58.262809+0000","last_active":"2023-10-19T13:04:58.263757+0000","last_peered":"2023-10-19T13:04:58.263757+0000","last_clean":"2023-10-19T13:04:58.263757+0000","last_became_active":"2023-10-19T13:04:58.262487+0000","last_became_peered":"2023-10-19T13:04:58.262487+0000","last_unstale":"2023-10-19T13:04:58.263757+0000","last_undegraded":"2023-10-19T13:04:58.263757+0000","last_fullsized":"2023-10-19T13:04:58.263757+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:02:00.626730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.25","version":"0'0","reported_seq":9,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.259638+0000","last_change":"2023-10-19T13:04:58.258577+0000","last_active":"2023-10-19T13:04:58.259638+0000","last_peered":"2023-10-19T13:04:58.259638+0000","last_clean":"2023-10-19T13:04:58.259638+0000","last_became_active":"2023-10-19T13:04:58.252021+0000","last_became_peered":"2023-10-19T13:04:58.252021+0000","last_unstale":"2023-10-19T13:04:58.259638+0000","last_undegraded":"2023-10-19T13:04:58.259638+0000","last_fullsized":"2023-10-19T13:04:58.259638+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:59:39.956253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.24","version":"0'0","reported_seq":11,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:05:00.280496+0000","last_change":"2023-10-19T13:04:58.247785+0000","last_active":"2023-10-19T13:05:00.280496+0000","last_peered":"2023-10-19T13:05:00.280496+0000","last_clean":"2023-10-19T13:05:00.280496+0000","last_became_active":"2023-10-19T13:04:58.247504+0000","last_became_peered":"2023-10-19T13:04:58.247504+0000","last_unstale":"2023-10-19T13:05:00.280496+0000","last_undegraded":"2023-10-19T13:05:00.280496+0000","last_fullsized":"2023-10-19T13:05:00.280496+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:59:52.674088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.23","version":"0'0","reported_seq":9,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.260025+0000","last_change":"2023-10-19T13:04:58.256809+0000","last_active":"2023-10-19T13:04:58.260025+0000","last_peered":"2023-10-19T13:04:58.260025+0000","last_clean":"2023-10-19T13:04:58.260025+0000","last_became_active":"2023-10-19T13:04:58.256215+0000","last_became_peered":"2023-10-19T13:04:58.256215+0000","last_unstale":"2023-10-19T13:04:58.260025+0000","last_undegraded":"2023-10-19T13:04:58.260025+0000","last_fullsized":"2023-10-19T13:04:58.260025+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:20:17.526434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.22","version":"0'0","reported_seq":9,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.259795+0000","last_change":"2023-10-19T13:04:58.255029+0000","last_active":"2023-10-19T13:04:58.259795+0000","last_peered":"2023-10-19T13:04:58.259795+0000","last_clean":"2023-10-19T13:04:58.259795+0000","last_became_active":"2023-10-19T13:04:58.254768+0000","last_became_peered":"2023-10-19T13:04:58.254768+0000","last_unstale":"2023-10-19T13:04:58.259795+0000","last_undegraded":"2023-10-19T13:04:58.259795+0000","last_fullsized":"2023-10-19T13:04:58.259795+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:40:45.798849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.21","version":"0'0","reported_seq":11,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.246195+0000","last_change":"2023-10-19T13:04:58.256475+0000","last_active":"2023-10-19T13:04:59.246195+0000","last_peered":"2023-10-19T13:04:59.246195+0000","last_clean":"2023-10-19T13:04:59.246195+0000","last_became_active":"2023-10-19T13:04:58.256206+0000","last_became_peered":"2023-10-19T13:04:58.256206+0000","last_unstale":"2023-10-19T13:04:59.246195+0000","last_undegraded":"2023-10-19T13:04:59.246195+0000","last_fullsized":"2023-10-19T13:04:59.246195+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:19:05.998271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.20","version":"0'0","reported_seq":11,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:05:00.280346+0000","last_change":"2023-10-19T13:04:58.252022+0000","last_active":"2023-10-19T13:05:00.280346+0000","last_peered":"2023-10-19T13:05:00.280346+0000","last_clean":"2023-10-19T13:05:00.280346+0000","last_became_active":"2023-10-19T13:04:58.251734+0000","last_became_peered":"2023-10-19T13:04:58.251734+0000","last_unstale":"2023-10-19T13:05:00.280346+0000","last_undegraded":"2023-10-19T13:05:00.280346+0000","last_fullsized":"2023-10-19T13:05:00.280346+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:09:58.468043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":11,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.246135+0000","last_change":"2023-10-19T13:04:58.253969+0000","last_active":"2023-10-19T13:04:59.246135+0000","last_peered":"2023-10-19T13:04:59.246135+0000","last_clean":"2023-10-19T13:04:59.246135+0000","last_became_active":"2023-10-19T13:04:58.253684+0000","last_became_peered":"2023-10-19T13:04:58.253684+0000","last_unstale":"2023-10-19T13:04:59.246135+0000","last_undegraded":"2023-10-19T13:04:59.246135+0000","last_fullsized":"2023-10-19T13:04:59.246135+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:21:13.640969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1f","version":"156'4","reported_seq":15,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.787196+0000","last_change":"2023-10-19T13:04:57.781301+0000","last_active":"2023-10-19T13:04:58.787196+0000","last_peered":"2023-10-19T13:04:58.787196+0000","last_clean":"2023-10-19T13:04:58.787196+0000","last_became_active":"2023-10-19T13:04:57.781027+0000","last_became_peered":"2023-10-19T13:04:57.781027+0000","last_unstale":"2023-10-19T13:04:58.787196+0000","last_undegraded":"2023-10-19T13:04:58.787196+0000","last_fullsized":"2023-10-19T13:04:58.787196+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_clean_scrub_stamp":"2023-10-19T13:04:56.754267+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:49:14.845036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":11,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.242190+0000","last_change":"2023-10-19T13:04:58.247912+0000","last_active":"2023-10-19T13:04:59.242190+0000","last_peered":"2023-10-19T13:04:59.242190+0000","last_clean":"2023-10-19T13:04:59.242190+0000","last_became_active":"2023-10-19T13:04:58.247657+0000","last_became_peered":"2023-10-19T13:04:58.247657+0000","last_unstale":"2023-10-19T13:04:59.242190+0000","last_undegraded":"2023-10-19T13:04:59.242190+0000","last_fullsized":"2023-10-19T13:04:59.242190+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:50:20.587539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1e","version":"156'1","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.685096+0000","last_change":"2023-10-19T13:04:57.782578+0000","last_active":"2023-10-19T13:04:59.685096+0000","last_peered":"2023-10-19T13:04:59.685096+0000","last_clean":"2023-10-19T13:04:59.685096+0000","last_became_active":"2023-10-19T13:04:57.782329+0000","last_became_peered":"2023-10-19T13:04:57.782329+0000","last_unstale":"2023-10-19T13:04:59.685096+0000","last_undegraded":"2023-10-19T13:04:59.685096+0000","last_fullsized":"2023-10-19T13:04:59.685096+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_clean_scrub_stamp":"2023-10-19T13:04:56.754267+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:05:19.027331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1a","version":"0'0","reported_seq":11,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.245582+0000","last_change":"2023-10-19T13:04:58.257796+0000","last_active":"2023-10-19T13:04:59.245582+0000","last_peered":"2023-10-19T13:04:59.245582+0000","last_clean":"2023-10-19T13:04:59.245582+0000","last_became_active":"2023-10-19T13:04:58.257214+0000","last_became_peered":"2023-10-19T13:04:58.257214+0000","last_unstale":"2023-10-19T13:04:59.245582+0000","last_undegraded":"2023-10-19T13:04:59.245582+0000","last_fullsized":"2023-10-19T13:04:59.245582+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:21:53.914585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1d","version":"156'2","reported_seq":13,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.787220+0000","last_change":"2023-10-19T13:04:57.789046+0000","last_active":"2023-10-19T13:04:58.787220+0000","last_peered":"2023-10-19T13:04:58.787220+0000","last_clean":"2023-10-19T13:04:58.787220+0000","last_became_active":"2023-10-19T13:04:57.788769+0000","last_became_peered":"2023-10-19T13:04:57.788769+0000","last_unstale":"2023-10-19T13:04:58.787220+0000","last_undegraded":"2023-10-19T13:04:58.787220+0000","last_fullsized":"2023-10-19T13:04:58.787220+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_clean_scrub_stamp":"2023-10-19T13:04:56.754267+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:21:52.909025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1b","version":"0'0","reported_seq":11,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.242321+0000","last_change":"2023-10-19T13:04:58.243725+0000","last_active":"2023-10-19T13:04:59.242321+0000","last_peered":"2023-10-19T13:04:59.242321+0000","last_clean":"2023-10-19T13:04:59.242321+0000","last_became_active":"2023-10-19T13:04:58.243487+0000","last_became_peered":"2023-10-19T13:04:58.243487+0000","last_unstale":"2023-10-19T13:04:59.242321+0000","last_undegraded":"2023-10-19T13:04:59.242321+0000","last_fullsized":"2023-10-19T13:04:59.242321+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:17:52.329819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1c","version":"156'2","reported_seq":13,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.786704+0000","last_change":"2023-10-19T13:04:57.786044+0000","last_active":"2023-10-19T13:04:58.786704+0000","last_peered":"2023-10-19T13:04:58.786704+0000","last_clean":"2023-10-19T13:04:58.786704+0000","last_became_active":"2023-10-19T13:04:57.785572+0000","last_became_peered":"2023-10-19T13:04:57.785572+0000","last_unstale":"2023-10-19T13:04:58.786704+0000","last_undegraded":"2023-10-19T13:04:58.786704+0000","last_fullsized":"2023-10-19T13:04:58.786704+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_clean_scrub_stamp":"2023-10-19T13:04:56.754267+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:44:42.677156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1c","version":"0'0","reported_seq":11,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.246588+0000","last_change":"2023-10-19T13:04:58.256747+0000","last_active":"2023-10-19T13:04:59.246588+0000","last_peered":"2023-10-19T13:04:59.246588+0000","last_clean":"2023-10-19T13:04:59.246588+0000","last_became_active":"2023-10-19T13:04:58.256446+0000","last_became_peered":"2023-10-19T13:04:58.256446+0000","last_unstale":"2023-10-19T13:04:59.246588+0000","last_undegraded":"2023-10-19T13:04:59.246588+0000","last_fullsized":"2023-10-19T13:04:59.246588+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:01:25.818792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1b","version":"156'1","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.246453+0000","last_change":"2023-10-19T13:04:57.782263+0000","last_active":"2023-10-19T13:04:59.246453+0000","last_peered":"2023-10-19T13:04:59.246453+0000","last_clean":"2023-10-19T13:04:59.246453+0000","last_became_active":"2023-10-19T13:04:57.781924+0000","last_became_peered":"2023-10-19T13:04:57.781924+0000","last_unstale":"2023-10-19T13:04:59.246453+0000","last_undegraded":"2023-10-19T13:04:59.246453+0000","last_fullsized":"2023-10-19T13:04:59.246453+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_clean_scrub_stamp":"2023-10-19T13:04:56.754267+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:05:31.203658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.1d","version":"0'0","reported_seq":11,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.246636+0000","last_change":"2023-10-19T13:04:58.253745+0000","last_active":"2023-10-19T13:04:59.246636+0000","last_peered":"2023-10-19T13:04:59.246636+0000","last_clean":"2023-10-19T13:04:59.246636+0000","last_became_active":"2023-10-19T13:04:58.253469+0000","last_became_peered":"2023-10-19T13:04:58.253469+0000","last_unstale":"2023-10-19T13:04:59.246636+0000","last_undegraded":"2023-10-19T13:04:59.246636+0000","last_fullsized":"2023-10-19T13:04:59.246636+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:00:57.974372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1a","version":"156'1","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.684688+0000","last_change":"2023-10-19T13:04:57.785484+0000","last_active":"2023-10-19T13:04:59.684688+0000","last_peered":"2023-10-19T13:04:59.684688+0000","last_clean":"2023-10-19T13:04:59.684688+0000","last_became_active":"2023-10-19T13:04:57.785122+0000","last_became_peered":"2023-10-19T13:04:57.785122+0000","last_unstale":"2023-10-19T13:04:59.684688+0000","last_undegraded":"2023-10-19T13:04:59.684688+0000","last_fullsized":"2023-10-19T13:04:59.684688+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_clean_scrub_stamp":"2023-10-19T13:04:56.754267+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:46:37.159756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1e","version":"0'0","reported_seq":11,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.684994+0000","last_change":"2023-10-19T13:04:58.256786+0000","last_active":"2023-10-19T13:04:59.684994+0000","last_peered":"2023-10-19T13:04:59.684994+0000","last_clean":"2023-10-19T13:04:59.684994+0000","last_became_active":"2023-10-19T13:04:58.256522+0000","last_became_peered":"2023-10-19T13:04:58.256522+0000","last_unstale":"2023-10-19T13:04:59.684994+0000","last_undegraded":"2023-10-19T13:04:59.684994+0000","last_fullsized":"2023-10-19T13:04:59.684994+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:33:47.290259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.240562+0000","last_change":"2023-10-19T13:04:57.782709+0000","last_active":"2023-10-19T13:04:58.240562+0000","last_peered":"2023-10-19T13:04:58.240562+0000","last_clean":"2023-10-19T13:04:58.240562+0000","last_became_active":"2023-10-19T13:04:57.782415+0000","last_became_peered":"2023-10-19T13:04:57.782415+0000","last_unstale":"2023-10-19T13:04:58.240562+0000","last_undegraded":"2023-10-19T13:04:58.240562+0000","last_fullsized":"2023-10-19T13:04:58.240562+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_clean_scrub_stamp":"2023-10-19T13:04:56.754267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:04:24.813151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1f","version":"0'0","reported_seq":9,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.259107+0000","last_change":"2023-10-19T13:04:58.258308+0000","last_active":"2023-10-19T13:04:58.259107+0000","last_peered":"2023-10-19T13:04:58.259107+0000","last_clean":"2023-10-19T13:04:58.259107+0000","last_became_active":"2023-10-19T13:04:58.257299+0000","last_became_peered":"2023-10-19T13:04:58.257299+0000","last_unstale":"2023-10-19T13:04:58.259107+0000","last_undegraded":"2023-10-19T13:04:58.259107+0000","last_fullsized":"2023-10-19T13:04:58.259107+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:36:23.188250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.254159+0000","last_change":"2023-10-19T13:04:57.785330+0000","last_active":"2023-10-19T13:04:58.254159+0000","last_peered":"2023-10-19T13:04:58.254159+0000","last_clean":"2023-10-19T13:04:58.254159+0000","last_became_active":"2023-10-19T13:04:57.785054+0000","last_became_peered":"2023-10-19T13:04:57.785054+0000","last_unstale":"2023-10-19T13:04:58.254159+0000","last_undegraded":"2023-10-19T13:04:58.254159+0000","last_fullsized":"2023-10-19T13:04:58.254159+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_clean_scrub_stamp":"2023-10-19T13:04:56.754267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:04:24.813151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.10","version":"0'0","reported_seq":11,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.242126+0000","last_change":"2023-10-19T13:04:58.260661+0000","last_active":"2023-10-19T13:04:59.242126+0000","last_peered":"2023-10-19T13:04:59.242126+0000","last_clean":"2023-10-19T13:04:59.242126+0000","last_became_active":"2023-10-19T13:04:58.260370+0000","last_became_peered":"2023-10-19T13:04:58.260370+0000","last_unstale":"2023-10-19T13:04:59.242126+0000","last_undegraded":"2023-10-19T13:04:59.242126+0000","last_fullsized":"2023-10-19T13:04:59.242126+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:35:58.392101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":12,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.245736+0000","last_change":"2023-10-19T13:04:57.782445+0000","last_active":"2023-10-19T13:04:59.245736+0000","last_peered":"2023-10-19T13:04:59.245736+0000","last_clean":"2023-10-19T13:04:59.245736+0000","last_became_active":"2023-10-19T13:04:57.782166+0000","last_became_peered":"2023-10-19T13:04:57.782166+0000","last_unstale":"2023-10-19T13:04:59.245736+0000","last_undegraded":"2023-10-19T13:04:59.245736+0000","last_fullsized":"2023-10-19T13:04:59.245736+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_clean_scrub_stamp":"2023-10-19T13:04:56.754267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:44:42.677156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":9,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.260101+0000","last_change":"2023-10-19T13:04:58.250760+0000","last_active":"2023-10-19T13:04:58.260101+0000","last_peered":"2023-10-19T13:04:58.260101+0000","last_clean":"2023-10-19T13:04:58.260101+0000","last_became_active":"2023-10-19T13:04:58.250474+0000","last_became_peered":"2023-10-19T13:04:58.250474+0000","last_unstale":"2023-10-19T13:04:58.260101+0000","last_undegraded":"2023-10-19T13:04:58.260101+0000","last_fullsized":"2023-10-19T13:04:58.260101+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:00:28.926969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":12,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.242419+0000","last_change":"2023-10-19T13:04:57.783533+0000","last_active":"2023-10-19T13:04:59.242419+0000","last_peered":"2023-10-19T13:04:59.242419+0000","last_clean":"2023-10-19T13:04:59.242419+0000","last_became_active":"2023-10-19T13:04:57.783271+0000","last_became_peered":"2023-10-19T13:04:57.783271+0000","last_unstale":"2023-10-19T13:04:59.242419+0000","last_undegraded":"2023-10-19T13:04:59.242419+0000","last_fullsized":"2023-10-19T13:04:59.242419+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_clean_scrub_stamp":"2023-10-19T13:04:56.754267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:18:54.089803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":9,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.255266+0000","last_change":"2023-10-19T13:04:58.252270+0000","last_active":"2023-10-19T13:04:58.255266+0000","last_peered":"2023-10-19T13:04:58.255266+0000","last_clean":"2023-10-19T13:04:58.255266+0000","last_became_active":"2023-10-19T13:04:58.250864+0000","last_became_peered":"2023-10-19T13:04:58.250864+0000","last_unstale":"2023-10-19T13:04:58.255266+0000","last_undegraded":"2023-10-19T13:04:58.255266+0000","last_fullsized":"2023-10-19T13:04:58.255266+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:14:30.850273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.15","version":"156'3","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.787642+0000","last_change":"2023-10-19T13:04:57.787963+0000","last_active":"2023-10-19T13:04:58.787642+0000","last_peered":"2023-10-19T13:04:58.787642+0000","last_clean":"2023-10-19T13:04:58.787642+0000","last_became_active":"2023-10-19T13:04:57.787643+0000","last_became_peered":"2023-10-19T13:04:57.787643+0000","last_unstale":"2023-10-19T13:04:58.787642+0000","last_undegraded":"2023-10-19T13:04:58.787642+0000","last_fullsized":"2023-10-19T13:04:58.787642+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_clean_scrub_stamp":"2023-10-19T13:04:56.754267+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:07:15.241290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.13","version":"0'0","reported_seq":9,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.260373+0000","last_change":"2023-10-19T13:04:58.259272+0000","last_active":"2023-10-19T13:04:58.260373+0000","last_peered":"2023-10-19T13:04:58.260373+0000","last_clean":"2023-10-19T13:04:58.260373+0000","last_became_active":"2023-10-19T13:04:58.258949+0000","last_became_peered":"2023-10-19T13:04:58.258949+0000","last_unstale":"2023-10-19T13:04:58.260373+0000","last_undegraded":"2023-10-19T13:04:58.260373+0000","last_fullsized":"2023-10-19T13:04:58.260373+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:18:15.409534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.14","version":"156'6","reported_seq":17,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.789400+0000","last_change":"2023-10-19T13:04:57.786159+0000","last_active":"2023-10-19T13:04:58.789400+0000","last_peered":"2023-10-19T13:04:58.789400+0000","last_clean":"2023-10-19T13:04:58.789400+0000","last_became_active":"2023-10-19T13:04:57.785254+0000","last_became_peered":"2023-10-19T13:04:57.785254+0000","last_unstale":"2023-10-19T13:04:58.789400+0000","last_undegraded":"2023-10-19T13:04:58.789400+0000","last_fullsized":"2023-10-19T13:04:58.789400+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_clean_scrub_stamp":"2023-10-19T13:04:56.754267+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:53:51.842918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.14","version":"0'0","reported_seq":11,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:05:00.280618+0000","last_change":"2023-10-19T13:04:58.241710+0000","last_active":"2023-10-19T13:05:00.280618+0000","last_peered":"2023-10-19T13:05:00.280618+0000","last_clean":"2023-10-19T13:05:00.280618+0000","last_became_active":"2023-10-19T13:04:58.241418+0000","last_became_peered":"2023-10-19T13:04:58.241418+0000","last_unstale":"2023-10-19T13:05:00.280618+0000","last_undegraded":"2023-10-19T13:05:00.280618+0000","last_fullsized":"2023-10-19T13:05:00.280618+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:15:51.554211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.13","version":"156'2","reported_seq":14,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.242413+0000","last_change":"2023-10-19T13:04:57.784220+0000","last_active":"2023-10-19T13:04:59.242413+0000","last_peered":"2023-10-19T13:04:59.242413+0000","last_clean":"2023-10-19T13:04:59.242413+0000","last_became_active":"2023-10-19T13:04:57.783948+0000","last_became_peered":"2023-10-19T13:04:57.783948+0000","last_unstale":"2023-10-19T13:04:59.242413+0000","last_undegraded":"2023-10-19T13:04:59.242413+0000","last_fullsized":"2023-10-19T13:04:59.242413+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_clean_scrub_stamp":"2023-10-19T13:04:56.754267+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:39:38.374564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":9,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.256477+0000","last_change":"2023-10-19T13:04:58.255428+0000","last_active":"2023-10-19T13:04:58.256477+0000","last_peered":"2023-10-19T13:04:58.256477+0000","last_clean":"2023-10-19T13:04:58.256477+0000","last_became_active":"2023-10-19T13:04:58.255173+0000","last_became_peered":"2023-10-19T13:04:58.255173+0000","last_unstale":"2023-10-19T13:04:58.256477+0000","last_undegraded":"2023-10-19T13:04:58.256477+0000","last_fullsized":"2023-10-19T13:04:58.256477+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:37:30.289123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.249501+0000","last_change":"2023-10-19T13:04:57.778139+0000","last_active":"2023-10-19T13:04:58.249501+0000","last_peered":"2023-10-19T13:04:58.249501+0000","last_clean":"2023-10-19T13:04:58.249501+0000","last_became_active":"2023-10-19T13:04:57.777871+0000","last_became_peered":"2023-10-19T13:04:57.777871+0000","last_unstale":"2023-10-19T13:04:58.249501+0000","last_undegraded":"2023-10-19T13:04:58.249501+0000","last_fullsized":"2023-10-19T13:04:58.249501+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_clean_scrub_stamp":"2023-10-19T13:04:56.754267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:00:27.921409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":11,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.684971+0000","last_change":"2023-10-19T13:04:58.256004+0000","last_active":"2023-10-19T13:04:59.684971+0000","last_peered":"2023-10-19T13:04:59.684971+0000","last_clean":"2023-10-19T13:04:59.684971+0000","last_became_active":"2023-10-19T13:04:58.255728+0000","last_became_peered":"2023-10-19T13:04:58.255728+0000","last_unstale":"2023-10-19T13:04:59.684971+0000","last_undegraded":"2023-10-19T13:04:59.684971+0000","last_fullsized":"2023-10-19T13:04:59.684971+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:58:49.086652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.10","version":"156'8","reported_seq":19,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.787960+0000","last_change":"2023-10-19T13:04:57.785842+0000","last_active":"2023-10-19T13:04:58.787960+0000","last_peered":"2023-10-19T13:04:58.787960+0000","last_clean":"2023-10-19T13:04:58.787960+0000","last_became_active":"2023-10-19T13:04:57.785580+0000","last_became_peered":"2023-10-19T13:04:57.785580+0000","last_unstale":"2023-10-19T13:04:58.787960+0000","last_undegraded":"2023-10-19T13:04:58.787960+0000","last_fullsized":"2023-10-19T13:04:58.787960+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_clean_scrub_stamp":"2023-10-19T13:04:56.754267+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:58:58.896647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":9,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.262064+0000","last_change":"2023-10-19T13:04:58.259820+0000","last_active":"2023-10-19T13:04:58.262064+0000","last_peered":"2023-10-19T13:04:58.262064+0000","last_clean":"2023-10-19T13:04:58.262064+0000","last_became_active":"2023-10-19T13:04:58.259534+0000","last_became_peered":"2023-10-19T13:04:58.259534+0000","last_unstale":"2023-10-19T13:04:58.262064+0000","last_undegraded":"2023-10-19T13:04:58.262064+0000","last_fullsized":"2023-10-19T13:04:58.262064+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:05:32.209217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.f","version":"156'1","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.245824+0000","last_change":"2023-10-19T13:04:57.778607+0000","last_active":"2023-10-19T13:04:59.245824+0000","last_peered":"2023-10-19T13:04:59.245824+0000","last_clean":"2023-10-19T13:04:59.245824+0000","last_became_active":"2023-10-19T13:04:57.778349+0000","last_became_peered":"2023-10-19T13:04:57.778349+0000","last_unstale":"2023-10-19T13:04:59.245824+0000","last_undegraded":"2023-10-19T13:04:59.245824+0000","last_fullsized":"2023-10-19T13:04:59.245824+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_clean_scrub_stamp":"2023-10-19T13:04:56.754267+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:20:48.667529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":9,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.254467+0000","last_change":"2023-10-19T13:04:58.251317+0000","last_active":"2023-10-19T13:04:58.254467+0000","last_peered":"2023-10-19T13:04:58.254467+0000","last_clean":"2023-10-19T13:04:58.254467+0000","last_became_active":"2023-10-19T13:04:58.250877+0000","last_became_peered":"2023-10-19T13:04:58.250877+0000","last_unstale":"2023-10-19T13:04:58.254467+0000","last_undegraded":"2023-10-19T13:04:58.254467+0000","last_fullsized":"2023-10-19T13:04:58.254467+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:47:10.547688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.e","version":"156'2","reported_seq":13,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.786567+0000","last_change":"2023-10-19T13:04:57.781567+0000","last_active":"2023-10-19T13:04:58.786567+0000","last_peered":"2023-10-19T13:04:58.786567+0000","last_clean":"2023-10-19T13:04:58.786567+0000","last_became_active":"2023-10-19T13:04:57.781069+0000","last_became_peered":"2023-10-19T13:04:57.781069+0000","last_unstale":"2023-10-19T13:04:58.786567+0000","last_undegraded":"2023-10-19T13:04:58.786567+0000","last_fullsized":"2023-10-19T13:04:58.786567+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_clean_scrub_stamp":"2023-10-19T13:04:56.754267+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:58:58.896647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":9,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.254615+0000","last_change":"2023-10-19T13:04:58.252783+0000","last_active":"2023-10-19T13:04:58.254615+0000","last_peered":"2023-10-19T13:04:58.254615+0000","last_clean":"2023-10-19T13:04:58.254615+0000","last_became_active":"2023-10-19T13:04:58.252311+0000","last_became_peered":"2023-10-19T13:04:58.252311+0000","last_unstale":"2023-10-19T13:04:58.254615+0000","last_undegraded":"2023-10-19T13:04:58.254615+0000","last_fullsized":"2023-10-19T13:04:58.254615+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:55:49.837946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.d","version":"156'1","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.785450+0000","last_change":"2023-10-19T13:04:57.785216+0000","last_active":"2023-10-19T13:04:58.785450+0000","last_peered":"2023-10-19T13:04:58.785450+0000","last_clean":"2023-10-19T13:04:58.785450+0000","last_became_active":"2023-10-19T13:04:57.784960+0000","last_became_peered":"2023-10-19T13:04:57.784960+0000","last_unstale":"2023-10-19T13:04:58.785450+0000","last_undegraded":"2023-10-19T13:04:58.785450+0000","last_fullsized":"2023-10-19T13:04:58.785450+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_clean_scrub_stamp":"2023-10-19T13:04:56.754267+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:14:16.121117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.b","version":"0'0","reported_seq":11,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:05:00.280507+0000","last_change":"2023-10-19T13:04:58.253006+0000","last_active":"2023-10-19T13:05:00.280507+0000","last_peered":"2023-10-19T13:05:00.280507+0000","last_clean":"2023-10-19T13:05:00.280507+0000","last_became_active":"2023-10-19T13:04:58.252754+0000","last_became_peered":"2023-10-19T13:04:58.252754+0000","last_unstale":"2023-10-19T13:05:00.280507+0000","last_undegraded":"2023-10-19T13:05:00.280507+0000","last_fullsized":"2023-10-19T13:05:00.280507+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:02:41.859778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":12,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.685046+0000","last_change":"2023-10-19T13:04:57.786506+0000","last_active":"2023-10-19T13:04:59.685046+0000","last_peered":"2023-10-19T13:04:59.685046+0000","last_clean":"2023-10-19T13:04:59.685046+0000","last_became_active":"2023-10-19T13:04:57.786259+0000","last_became_peered":"2023-10-19T13:04:57.786259+0000","last_unstale":"2023-10-19T13:04:59.685046+0000","last_undegraded":"2023-10-19T13:04:59.685046+0000","last_fullsized":"2023-10-19T13:04:59.685046+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_clean_scrub_stamp":"2023-10-19T13:04:56.754267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:36:08.623527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":11,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.685022+0000","last_change":"2023-10-19T13:04:58.245985+0000","last_active":"2023-10-19T13:04:59.685022+0000","last_peered":"2023-10-19T13:04:59.685022+0000","last_clean":"2023-10-19T13:04:59.685022+0000","last_became_active":"2023-10-19T13:04:58.245751+0000","last_became_peered":"2023-10-19T13:04:58.245751+0000","last_unstale":"2023-10-19T13:04:59.685022+0000","last_undegraded":"2023-10-19T13:04:59.685022+0000","last_fullsized":"2023-10-19T13:04:59.685022+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:17:39.136602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.b","version":"156'2","reported_seq":14,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.246465+0000","last_change":"2023-10-19T13:04:57.780196+0000","last_active":"2023-10-19T13:04:59.246465+0000","last_peered":"2023-10-19T13:04:59.246465+0000","last_clean":"2023-10-19T13:04:59.246465+0000","last_became_active":"2023-10-19T13:04:57.779939+0000","last_became_peered":"2023-10-19T13:04:57.779939+0000","last_unstale":"2023-10-19T13:04:59.246465+0000","last_undegraded":"2023-10-19T13:04:59.246465+0000","last_fullsized":"2023-10-19T13:04:59.246465+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_clean_scrub_stamp":"2023-10-19T13:04:56.754267+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:00:27.921409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.d","version":"0'0","reported_seq":11,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.242538+0000","last_change":"2023-10-19T13:04:58.243583+0000","last_active":"2023-10-19T13:04:59.242538+0000","last_peered":"2023-10-19T13:04:59.242538+0000","last_clean":"2023-10-19T13:04:59.242538+0000","last_became_active":"2023-10-19T13:04:58.242868+0000","last_became_peered":"2023-10-19T13:04:58.242868+0000","last_unstale":"2023-10-19T13:04:59.242538+0000","last_undegraded":"2023-10-19T13:04:59.242538+0000","last_fullsized":"2023-10-19T13:04:59.242538+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:39:42.760170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":12,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.242404+0000","last_change":"2023-10-19T13:04:57.774119+0000","last_active":"2023-10-19T13:04:59.242404+0000","last_peered":"2023-10-19T13:04:59.242404+0000","last_clean":"2023-10-19T13:04:59.242404+0000","last_became_active":"2023-10-19T13:04:57.773825+0000","last_became_peered":"2023-10-19T13:04:57.773825+0000","last_unstale":"2023-10-19T13:04:59.242404+0000","last_undegraded":"2023-10-19T13:04:59.242404+0000","last_fullsized":"2023-10-19T13:04:59.242404+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_clean_scrub_stamp":"2023-10-19T13:04:56.754267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:47:49.295344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.e","version":"0'0","reported_seq":11,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.685012+0000","last_change":"2023-10-19T13:04:58.261839+0000","last_active":"2023-10-19T13:04:59.685012+0000","last_peered":"2023-10-19T13:04:59.685012+0000","last_clean":"2023-10-19T13:04:59.685012+0000","last_became_active":"2023-10-19T13:04:58.261672+0000","last_became_peered":"2023-10-19T13:04:58.261672+0000","last_unstale":"2023-10-19T13:04:59.685012+0000","last_undegraded":"2023-10-19T13:04:59.685012+0000","last_fullsized":"2023-10-19T13:04:59.685012+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:51:48.722094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.242033+0000","last_change":"2023-10-19T13:04:57.788324+0000","last_active":"2023-10-19T13:04:58.242033+0000","last_peered":"2023-10-19T13:04:58.242033+0000","last_clean":"2023-10-19T13:04:58.242033+0000","last_became_active":"2023-10-19T13:04:57.787998+0000","last_became_peered":"2023-10-19T13:04:57.787998+0000","last_unstale":"2023-10-19T13:04:58.242033+0000","last_undegraded":"2023-10-19T13:04:58.242033+0000","last_fullsized":"2023-10-19T13:04:58.242033+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_clean_scrub_stamp":"2023-10-19T13:04:56.754267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:32:27.384468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.f","version":"0'0","reported_seq":11,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.245784+0000","last_change":"2023-10-19T13:04:58.254773+0000","last_active":"2023-10-19T13:04:59.245784+0000","last_peered":"2023-10-19T13:04:59.245784+0000","last_clean":"2023-10-19T13:04:59.245784+0000","last_became_active":"2023-10-19T13:04:58.254512+0000","last_became_peered":"2023-10-19T13:04:58.254512+0000","last_unstale":"2023-10-19T13:04:59.245784+0000","last_undegraded":"2023-10-19T13:04:59.245784+0000","last_fullsized":"2023-10-19T13:04:59.245784+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:32:28.390028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.240877+0000","last_change":"2023-10-19T13:04:57.779244+0000","last_active":"2023-10-19T13:04:58.240877+0000","last_peered":"2023-10-19T13:04:58.240877+0000","last_clean":"2023-10-19T13:04:58.240877+0000","last_became_active":"2023-10-19T13:04:57.778889+0000","last_became_peered":"2023-10-19T13:04:57.778889+0000","last_unstale":"2023-10-19T13:04:58.240877+0000","last_undegraded":"2023-10-19T13:04:58.240877+0000","last_fullsized":"2023-10-19T13:04:58.240877+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_clean_scrub_stamp":"2023-10-19T13:04:56.754267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:43:39.407455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":11,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.684819+0000","last_change":"2023-10-19T13:04:58.243809+0000","last_active":"2023-10-19T13:04:59.684819+0000","last_peered":"2023-10-19T13:04:59.684819+0000","last_clean":"2023-10-19T13:04:59.684819+0000","last_became_active":"2023-10-19T13:04:58.243536+0000","last_became_peered":"2023-10-19T13:04:58.243536+0000","last_unstale":"2023-10-19T13:04:59.684819+0000","last_undegraded":"2023-10-19T13:04:59.684819+0000","last_fullsized":"2023-10-19T13:04:59.684819+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:46:25.741582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.7","version":"156'2","reported_seq":14,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.242660+0000","last_change":"2023-10-19T13:04:57.781780+0000","last_active":"2023-10-19T13:04:59.242660+0000","last_peered":"2023-10-19T13:04:59.242660+0000","last_clean":"2023-10-19T13:04:59.242660+0000","last_became_active":"2023-10-19T13:04:57.781487+0000","last_became_peered":"2023-10-19T13:04:57.781487+0000","last_unstale":"2023-10-19T13:04:59.242660+0000","last_undegraded":"2023-10-19T13:04:59.242660+0000","last_fullsized":"2023-10-19T13:04:59.242660+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_clean_scrub_stamp":"2023-10-19T13:04:56.754267+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:35:24.389421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":9,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.259834+0000","last_change":"2023-10-19T13:04:58.255630+0000","last_active":"2023-10-19T13:04:58.259834+0000","last_peered":"2023-10-19T13:04:58.259834+0000","last_clean":"2023-10-19T13:04:58.259834+0000","last_became_active":"2023-10-19T13:04:58.255377+0000","last_became_peered":"2023-10-19T13:04:58.255377+0000","last_unstale":"2023-10-19T13:04:58.259834+0000","last_undegraded":"2023-10-19T13:04:58.259834+0000","last_fullsized":"2023-10-19T13:04:58.259834+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:29:04.911896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":12,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.246758+0000","last_change":"2023-10-19T13:04:57.775919+0000","last_active":"2023-10-19T13:04:59.246758+0000","last_peered":"2023-10-19T13:04:59.246758+0000","last_clean":"2023-10-19T13:04:59.246758+0000","last_became_active":"2023-10-19T13:04:57.775602+0000","last_became_peered":"2023-10-19T13:04:57.775602+0000","last_unstale":"2023-10-19T13:04:59.246758+0000","last_undegraded":"2023-10-19T13:04:59.246758+0000","last_fullsized":"2023-10-19T13:04:59.246758+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_clean_scrub_stamp":"2023-10-19T13:04:56.754267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:04:24.813151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":9,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.254600+0000","last_change":"2023-10-19T13:04:58.252594+0000","last_active":"2023-10-19T13:04:58.254600+0000","last_peered":"2023-10-19T13:04:58.254600+0000","last_clean":"2023-10-19T13:04:58.254600+0000","last_became_active":"2023-10-19T13:04:58.251889+0000","last_became_peered":"2023-10-19T13:04:58.251889+0000","last_unstale":"2023-10-19T13:04:58.254600+0000","last_undegraded":"2023-10-19T13:04:58.254600+0000","last_fullsized":"2023-10-19T13:04:58.254600+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:29:25.303068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":12,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.246718+0000","last_change":"2023-10-19T13:04:57.778790+0000","last_active":"2023-10-19T13:04:59.246718+0000","last_peered":"2023-10-19T13:04:59.246718+0000","last_clean":"2023-10-19T13:04:59.246718+0000","last_became_active":"2023-10-19T13:04:57.778481+0000","last_became_peered":"2023-10-19T13:04:57.778481+0000","last_unstale":"2023-10-19T13:04:59.246718+0000","last_undegraded":"2023-10-19T13:04:59.246718+0000","last_fullsized":"2023-10-19T13:04:59.246718+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_clean_scrub_stamp":"2023-10-19T13:04:56.754267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:01:59.621170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":11,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:05:00.280491+0000","last_change":"2023-10-19T13:04:58.247632+0000","last_active":"2023-10-19T13:05:00.280491+0000","last_peered":"2023-10-19T13:05:00.280491+0000","last_clean":"2023-10-19T13:05:00.280491+0000","last_became_active":"2023-10-19T13:04:58.247338+0000","last_became_peered":"2023-10-19T13:04:58.247338+0000","last_unstale":"2023-10-19T13:05:00.280491+0000","last_undegraded":"2023-10-19T13:05:00.280491+0000","last_fullsized":"2023-10-19T13:05:00.280491+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:41:50.218888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.241079+0000","last_change":"2023-10-19T13:04:57.783219+0000","last_active":"2023-10-19T13:04:58.241079+0000","last_peered":"2023-10-19T13:04:58.241079+0000","last_clean":"2023-10-19T13:04:58.241079+0000","last_became_active":"2023-10-19T13:04:57.782922+0000","last_became_peered":"2023-10-19T13:04:57.782922+0000","last_unstale":"2023-10-19T13:04:58.241079+0000","last_undegraded":"2023-10-19T13:04:58.241079+0000","last_fullsized":"2023-10-19T13:04:58.241079+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_clean_scrub_stamp":"2023-10-19T13:04:56.754267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:19:04.992711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":9,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.260030+0000","last_change":"2023-10-19T13:04:58.250837+0000","last_active":"2023-10-19T13:04:58.260030+0000","last_peered":"2023-10-19T13:04:58.260030+0000","last_clean":"2023-10-19T13:04:58.260030+0000","last_became_active":"2023-10-19T13:04:58.250559+0000","last_became_peered":"2023-10-19T13:04:58.250559+0000","last_unstale":"2023-10-19T13:04:58.260030+0000","last_undegraded":"2023-10-19T13:04:58.260030+0000","last_fullsized":"2023-10-19T13:04:58.260030+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:21:13.640969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":12,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.245534+0000","last_change":"2023-10-19T13:04:57.771866+0000","last_active":"2023-10-19T13:04:59.245534+0000","last_peered":"2023-10-19T13:04:59.245534+0000","last_clean":"2023-10-19T13:04:59.245534+0000","last_became_active":"2023-10-19T13:04:57.771607+0000","last_became_peered":"2023-10-19T13:04:57.771607+0000","last_unstale":"2023-10-19T13:04:59.245534+0000","last_undegraded":"2023-10-19T13:04:59.245534+0000","last_fullsized":"2023-10-19T13:04:59.245534+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_clean_scrub_stamp":"2023-10-19T13:04:56.754267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:34:11.325062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":9,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.259958+0000","last_change":"2023-10-19T13:04:58.250627+0000","last_active":"2023-10-19T13:04:58.259958+0000","last_peered":"2023-10-19T13:04:58.259958+0000","last_clean":"2023-10-19T13:04:58.259958+0000","last_became_active":"2023-10-19T13:04:58.250314+0000","last_became_peered":"2023-10-19T13:04:58.250314+0000","last_unstale":"2023-10-19T13:04:58.259958+0000","last_undegraded":"2023-10-19T13:04:58.259958+0000","last_fullsized":"2023-10-19T13:04:58.259958+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:18:09.472240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.242113+0000","last_change":"2023-10-19T13:04:57.785600+0000","last_active":"2023-10-19T13:04:58.242113+0000","last_peered":"2023-10-19T13:04:58.242113+0000","last_clean":"2023-10-19T13:04:58.242113+0000","last_became_active":"2023-10-19T13:04:57.785327+0000","last_became_peered":"2023-10-19T13:04:57.785327+0000","last_unstale":"2023-10-19T13:04:58.242113+0000","last_undegraded":"2023-10-19T13:04:58.242113+0000","last_fullsized":"2023-10-19T13:04:58.242113+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_clean_scrub_stamp":"2023-10-19T13:04:56.754267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:53:19.112926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.6","version":"0'0","reported_seq":9,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.265150+0000","last_change":"2023-10-19T13:04:58.259056+0000","last_active":"2023-10-19T13:04:58.265150+0000","last_peered":"2023-10-19T13:04:58.265150+0000","last_clean":"2023-10-19T13:04:58.265150+0000","last_became_active":"2023-10-19T13:04:58.258843+0000","last_became_peered":"2023-10-19T13:04:58.258843+0000","last_unstale":"2023-10-19T13:04:58.265150+0000","last_undegraded":"2023-10-19T13:04:58.265150+0000","last_fullsized":"2023-10-19T13:04:58.265150+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:05:32.209217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.240583+0000","last_change":"2023-10-19T13:04:57.783271+0000","last_active":"2023-10-19T13:04:58.240583+0000","last_peered":"2023-10-19T13:04:58.240583+0000","last_clean":"2023-10-19T13:04:58.240583+0000","last_became_active":"2023-10-19T13:04:57.782989+0000","last_became_peered":"2023-10-19T13:04:57.782989+0000","last_unstale":"2023-10-19T13:04:58.240583+0000","last_undegraded":"2023-10-19T13:04:58.240583+0000","last_fullsized":"2023-10-19T13:04:58.240583+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_clean_scrub_stamp":"2023-10-19T13:04:56.754267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:01:59.621170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.7","version":"0'0","reported_seq":9,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.256230+0000","last_change":"2023-10-19T13:04:58.253636+0000","last_active":"2023-10-19T13:04:58.256230+0000","last_peered":"2023-10-19T13:04:58.256230+0000","last_clean":"2023-10-19T13:04:58.256230+0000","last_became_active":"2023-10-19T13:04:58.251824+0000","last_became_peered":"2023-10-19T13:04:58.251824+0000","last_unstale":"2023-10-19T13:04:58.256230+0000","last_undegraded":"2023-10-19T13:04:58.256230+0000","last_fullsized":"2023-10-19T13:04:58.256230+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:04:35.399006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":12,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.684822+0000","last_change":"2023-10-19T13:04:57.788217+0000","last_active":"2023-10-19T13:04:59.684822+0000","last_peered":"2023-10-19T13:04:59.684822+0000","last_clean":"2023-10-19T13:04:59.684822+0000","last_became_active":"2023-10-19T13:04:57.787983+0000","last_became_peered":"2023-10-19T13:04:57.787983+0000","last_unstale":"2023-10-19T13:04:59.684822+0000","last_undegraded":"2023-10-19T13:04:59.684822+0000","last_fullsized":"2023-10-19T13:04:59.684822+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_clean_scrub_stamp":"2023-10-19T13:04:56.754267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:23:14.121433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.15","version":"0'0","reported_seq":9,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.260160+0000","last_change":"2023-10-19T13:04:58.256033+0000","last_active":"2023-10-19T13:04:58.260160+0000","last_peered":"2023-10-19T13:04:58.260160+0000","last_clean":"2023-10-19T13:04:58.260160+0000","last_became_active":"2023-10-19T13:04:58.255772+0000","last_became_peered":"2023-10-19T13:04:58.255772+0000","last_unstale":"2023-10-19T13:04:58.260160+0000","last_undegraded":"2023-10-19T13:04:58.260160+0000","last_fullsized":"2023-10-19T13:04:58.260160+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:05:39.299198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.241353+0000","last_change":"2023-10-19T13:04:57.788792+0000","last_active":"2023-10-19T13:04:58.241353+0000","last_peered":"2023-10-19T13:04:58.241353+0000","last_clean":"2023-10-19T13:04:58.241353+0000","last_became_active":"2023-10-19T13:04:57.788511+0000","last_became_peered":"2023-10-19T13:04:57.788511+0000","last_unstale":"2023-10-19T13:04:58.241353+0000","last_undegraded":"2023-10-19T13:04:58.241353+0000","last_fullsized":"2023-10-19T13:04:58.241353+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_clean_scrub_stamp":"2023-10-19T13:04:56.754267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:20:48.667529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"146'3087","reported_seq":3639,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-19T13:04:57.780323+0000","last_change":"2023-10-19T12:44:08.636613+0000","last_active":"2023-10-19T13:04:57.780323+0000","last_peered":"2023-10-19T13:04:57.780323+0000","last_clean":"2023-10-19T13:04:57.780323+0000","last_became_active":"2023-10-19T12:44:08.635697+0000","last_became_peered":"2023-10-19T12:44:08.635697+0000","last_unstale":"2023-10-19T13:04:57.780323+0000","last_undegraded":"2023-10-19T13:04:57.780323+0000","last_fullsized":"2023-10-19T13:04:57.780323+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_clean_scrub_stamp":"2023-10-19T12:44:07.399738+0000","objects_scrubbed":0,"log_size":3087,"log_dups_size":0,"ondisk_log_size":3087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:15:23.685485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1346,"num_read_kb":2770,"num_write":409,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":20,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":19,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1346,"num_read_kb":2770,"num_write":409,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3087,"ondisk_log_size":3087,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575100,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55884,"kb_used_data":2496,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315956,"statfs":{"total":96636764160,"available":96579538944,"internally_reserved":0,"allocated":2555904,"data_stored":1576190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54655435},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 13:04:51 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51800000000000002,"15min":0.55300000000000005},"min":{"1min":0.38900000000000001,"5min":0.224,"15min":0.224},"max":{"1min":0.77300000000000002,"5min":0.80100000000000005,"15min":0.89400000000000002},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.55100000000000005,"15min":0.58199999999999996},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.29599999999999999},"max":{"1min":0.73799999999999999,"5min":0.75600000000000001,"15min":0.93700000000000006},"last":0.63900000000000001}]},{"osd":1,"last update":"Thu Oct 19 13:04:51 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.54200000000000004,"15min":0.57099999999999995},"min":{"1min":0.42299999999999999,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.73999999999999999,"5min":0.82399999999999995,"15min":1.3129999999999999},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.54300000000000004,"15min":0.57599999999999996},"min":{"1min":0.31900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.79700000000000004,"5min":0.85499999999999998,"15min":1.284},"last":0.60199999999999998}]},{"osd":2,"last update":"Thu Oct 19 13:04:51 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.60099999999999998,"15min":0.622},"min":{"1min":0.37,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.80100000000000005,"5min":0.89200000000000002,"15min":0.98599999999999999},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.60199999999999998,"15min":0.627},"min":{"1min":0.42699999999999999,"5min":0.40300000000000002,"15min":0.28399999999999997},"max":{"1min":0.755,"5min":0.83699999999999997,"15min":1.2589999999999999},"last":0.69199999999999995}]},{"osd":3,"last update":"Thu Oct 19 13:04:51 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.56499999999999995,"15min":0.59599999999999997},"min":{"1min":0.35799999999999998,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.71999999999999997,"5min":0.87,"15min":0.96899999999999997},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.62,"5min":0.63500000000000001,"15min":0.65100000000000002},"min":{"1min":0.435,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.75900000000000001,"5min":0.91600000000000004,"15min":1.329},"last":0.67800000000000005}]},{"osd":4,"last update":"Thu Oct 19 13:04:51 2023","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.46899999999999997,"15min":0.46800000000000003},"min":{"1min":0.28000000000000003,"5min":0.247,"15min":0.247},"max":{"1min":0.65400000000000003,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.502,"5min":0.48199999999999998,"15min":0.46999999999999997},"min":{"1min":0.32000000000000001,"5min":0.216,"15min":0.216},"max":{"1min":0.68000000000000005,"5min":0.73399999999999999,"15min":0.75600000000000001},"last":0.45500000000000002}]},{"osd":5,"last update":"Thu Oct 19 13:04:51 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.5,"15min":0.48499999999999999},"min":{"1min":0.33500000000000002,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.77900000000000003,"5min":0.78800000000000003,"15min":0.80600000000000005},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.53000000000000003,"15min":0.51900000000000002},"min":{"1min":0.35799999999999998,"5min":0.32200000000000001,"15min":0.314},"max":{"1min":0.73199999999999998,"5min":0.80100000000000005,"15min":0.81999999999999995},"last":0.47799999999999998}]},{"osd":6,"last update":"Thu Oct 19 13:04:51 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55500000000000005,"15min":0.54600000000000004},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.308},"max":{"1min":0.78000000000000003,"5min":0.78900000000000003,"15min":0.88900000000000001},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53600000000000003,"15min":0.52800000000000002},"min":{"1min":0.38500000000000001,"5min":0.318,"15min":0.27300000000000002},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.90300000000000002},"last":0.503}]}]},{"osd":6,"up_from":13,"seq":55834575100,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33088,"kb_used_data":1908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338752,"statfs":{"total":96636764160,"available":96602882048,"internally_reserved":0,"allocated":1953792,"data_stored":984792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 13:04:40 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.56000000000000005,"15min":0.58099999999999996},"min":{"1min":0.29099999999999998,"5min":0.23999999999999999,"15min":0.19500000000000001},"max":{"1min":0.76700000000000002,"5min":1.196,"15min":1.196},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54500000000000004,"15min":0.57699999999999996},"min":{"1min":0.41899999999999998,"5min":0.28799999999999998,"15min":0.247},"max":{"1min":0.71899999999999997,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.55900000000000005}]},{"osd":1,"last update":"Thu Oct 19 13:04:40 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.56899999999999995,"15min":0.59199999999999997},"min":{"1min":0.34799999999999998,"5min":0.29099999999999998,"15min":0.27400000000000002},"max":{"1min":0.69499999999999995,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.57599999999999996,"15min":0.59699999999999998},"min":{"1min":0.39800000000000002,"5min":0.307,"15min":0.307},"max":{"1min":0.70399999999999996,"5min":0.80600000000000005,"15min":0.82099999999999995},"last":0.64300000000000002}]},{"osd":2,"last update":"Thu Oct 19 13:04:40 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.54900000000000004,"15min":0.57899999999999996},"min":{"1min":0.38400000000000001,"5min":0.34599999999999997,"15min":0.307},"max":{"1min":0.73899999999999999,"5min":0.79400000000000004,"15min":0.85699999999999998},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.58599999999999997,"15min":0.60499999999999998},"min":{"1min":0.436,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.77300000000000002,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.41599999999999998}]},{"osd":3,"last update":"Thu Oct 19 13:04:40 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.59999999999999998,"15min":0.622},"min":{"1min":0.47099999999999997,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.749,"5min":0.81299999999999994,"15min":0.871},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.625,"5min":0.60599999999999998,"15min":0.621},"min":{"1min":0.40799999999999997,"5min":0.39200000000000002,"15min":0.36799999999999999},"max":{"1min":0.78200000000000003,"5min":0.82799999999999996,"15min":0.92800000000000005},"last":0.59499999999999997}]},{"osd":4,"last update":"Thu Oct 19 13:04:40 2023","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.47399999999999998,"15min":0.46700000000000003},"min":{"1min":0.32800000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.60899999999999999,"5min":1.353,"15min":1.353},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.48099999999999998,"15min":0.47799999999999998},"min":{"1min":0.308,"5min":0.254,"15min":0.254},"max":{"1min":0.65600000000000003,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.439}]},{"osd":5,"last update":"Thu Oct 19 13:04:40 2023","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.46899999999999997,"15min":0.47299999999999998},"min":{"1min":0.34000000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.60599999999999998,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.502,"5min":0.53100000000000003,"15min":0.51600000000000001},"min":{"1min":0.38900000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.72899999999999998,"5min":0.77400000000000002,"15min":0.85699999999999998},"last":0.50600000000000001}]},{"osd":7,"last update":"Thu Oct 19 13:04:40 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.53500000000000003,"15min":0.52300000000000002},"min":{"1min":0.312,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.78900000000000003,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.54100000000000004,"15min":0.53400000000000003},"min":{"1min":0.41699999999999998,"5min":0.27900000000000003,"15min":0.26000000000000001},"max":{"1min":0.70199999999999996,"5min":1.127,"15min":1.127},"last":0.66400000000000003}]}]},{"osd":5,"up_from":13,"seq":55834575100,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51148,"kb_used_data":1916,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320692,"statfs":{"total":96636764160,"available":96584388608,"internally_reserved":0,"allocated":1961984,"data_stored":990790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 13:04:36 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55500000000000005,"15min":0.58199999999999996},"min":{"1min":0.36799999999999999,"5min":0.223,"15min":0.223},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.58499999999999996,"15min":0.59399999999999997},"min":{"1min":0.379,"5min":0.33900000000000002,"15min":0.27800000000000002},"max":{"1min":0.748,"5min":0.82999999999999996,"15min":0.84299999999999997},"last":0.63600000000000001}]},{"osd":1,"last update":"Thu Oct 19 13:04:36 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.57599999999999996,"15min":0.58899999999999997},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.32800000000000001},"max":{"1min":0.79700000000000004,"5min":0.81399999999999995,"15min":0.91200000000000003},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.60499999999999998,"15min":0.60799999999999998},"min":{"1min":0.48099999999999998,"5min":0.378,"15min":0.34000000000000002},"max":{"1min":0.78700000000000003,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.56499999999999995}]},{"osd":2,"last update":"Thu Oct 19 13:04:36 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.54900000000000004,"15min":0.57399999999999995},"min":{"1min":0.36799999999999999,"5min":0.32700000000000001,"15min":0.307},"max":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.622,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.42799999999999999,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.60999999999999999}]},{"osd":3,"last update":"Thu Oct 19 13:04:36 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.58099999999999996,"15min":0.59699999999999998},"min":{"1min":0.373,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.60399999999999998,"15min":0.625},"min":{"1min":0.437,"5min":0.40899999999999997,"15min":0.32100000000000001},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.58099999999999996}]},{"osd":4,"last update":"Thu Oct 19 13:04:36 2023","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.46700000000000003,"15min":0.46899999999999997},"min":{"1min":0.36899999999999999,"5min":0.307,"15min":0.29099999999999998},"max":{"1min":0.65500000000000003,"5min":0.745,"15min":0.98099999999999998},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.48699999999999999,"15min":0.47399999999999998},"min":{"1min":0.33800000000000002,"5min":0.26100000000000001,"15min":0.251},"max":{"1min":0.68200000000000005,"5min":0.78700000000000003,"15min":0.79400000000000004},"last":0.45600000000000002}]},{"osd":6,"last update":"Thu Oct 19 13:04:36 2023","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.497,"15min":0.47799999999999998},"min":{"1min":0.29899999999999999,"5min":0.27800000000000002,"15min":0.27200000000000002},"max":{"1min":0.75800000000000001,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52100000000000002,"15min":0.50900000000000001},"min":{"1min":0.41699999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.68600000000000005,"5min":0.80800000000000005,"15min":0.82499999999999996},"last":0.623}]},{"osd":7,"last update":"Thu Oct 19 13:04:36 2023","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.502,"15min":0.49299999999999999},"min":{"1min":0.33200000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.70099999999999996,"5min":0.78600000000000003,"15min":1.0589999999999999},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.55700000000000005,"15min":0.54100000000000004},"min":{"1min":0.318,"5min":0.318,"15min":0.311},"max":{"1min":0.74399999999999999,"5min":0.78700000000000003,"15min":0.94899999999999995},"last":0.54400000000000004}]}]},{"osd":4,"up_from":13,"seq":55834575100,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33100,"kb_used_data":1912,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338740,"statfs":{"total":96636764160,"available":96602869760,"internally_reserved":0,"allocated":1957888,"data_stored":990768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 13:04:49 2023","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.51300000000000001,"15min":0.54800000000000004},"min":{"1min":0.30399999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.623,"5min":0.78800000000000003,"15min":0.86699999999999999},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.5,"5min":0.53100000000000003,"15min":0.54900000000000004},"min":{"1min":0.33700000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.68700000000000006,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.443}]},{"osd":1,"last update":"Thu Oct 19 13:04:49 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.53800000000000003,"15min":0.58599999999999997},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.70199999999999996,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.309},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.57299999999999995,"15min":0.60399999999999998},"min":{"1min":0.374,"5min":0.374,"15min":0.32600000000000001},"max":{"1min":0.67100000000000004,"5min":0.79800000000000004,"15min":0.86899999999999999},"last":0.41899999999999998}]},{"osd":2,"last update":"Thu Oct 19 13:04:49 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.56399999999999995,"15min":0.59899999999999998},"min":{"1min":0.30399999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.66000000000000003,"5min":0.80100000000000005,"15min":0.86599999999999999},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.57599999999999996,"15min":0.60399999999999998},"min":{"1min":0.34999999999999998,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.64700000000000002,"5min":0.83799999999999997,"15min":0.85999999999999999},"last":0.46999999999999997}]},{"osd":3,"last update":"Thu Oct 19 13:04:49 2023","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.53800000000000003,"15min":0.56799999999999995},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.66800000000000004,"5min":0.75,"15min":0.79700000000000004},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.59599999999999997,"15min":0.628},"min":{"1min":0.42399999999999999,"5min":0.42099999999999999,"15min":0.41299999999999998},"max":{"1min":0.73199999999999998,"5min":0.82599999999999996,"15min":0.83899999999999997},"last":0.39700000000000002}]},{"osd":5,"last update":"Thu Oct 19 13:04:49 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.44900000000000001,"15min":0.44700000000000001},"min":{"1min":0.32200000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.63900000000000001,"5min":0.77100000000000002,"15min":0.79600000000000004},"last":0.379},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.46400000000000002,"15min":0.46000000000000002},"min":{"1min":0.33700000000000002,"5min":0.30099999999999999,"15min":0.23499999999999999},"max":{"1min":0.625,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.54000000000000004}]},{"osd":6,"last update":"Thu Oct 19 13:04:49 2023","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.46500000000000002,"15min":0.45800000000000002},"min":{"1min":0.34300000000000003,"5min":0.28499999999999998,"15min":0.26400000000000001},"max":{"1min":0.65300000000000002,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.502,"15min":0.49099999999999999},"min":{"1min":0.36599999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.67300000000000004,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.505}]},{"osd":7,"last update":"Thu Oct 19 13:04:49 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.501},"min":{"1min":0.34999999999999998,"5min":0.31900000000000001,"15min":0.27500000000000002},"max":{"1min":0.65300000000000002,"5min":0.78700000000000003,"15min":0.79100000000000004},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56200000000000006,"15min":0.54700000000000004},"min":{"1min":0.39200000000000002,"5min":0.33100000000000002,"15min":0.30199999999999999},"max":{"1min":0.68600000000000005,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.57999999999999996}]}]},{"osd":3,"up_from":13,"seq":55834575099,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41428,"kb_used_data":1928,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330412,"statfs":{"total":96636764160,"available":96594341888,"internally_reserved":0,"allocated":1974272,"data_stored":992450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":40434120},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 13:04:33 2023","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.39600000000000002,"15min":0.40300000000000002},"min":{"1min":0.26400000000000001,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":1.1279999999999999},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.38400000000000001,"15min":0.41099999999999998},"min":{"1min":0.25900000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.56399999999999995,"5min":0.59899999999999998,"15min":0.75},"last":0.46200000000000002}]},{"osd":1,"last update":"Thu Oct 19 13:04:33 2023","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.40200000000000002,"15min":0.40799999999999997},"min":{"1min":0.249,"5min":0.22,"15min":0.22},"max":{"1min":0.59499999999999997,"5min":0.628,"15min":0.72099999999999997},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.42799999999999999,"15min":0.45800000000000002},"min":{"1min":0.28199999999999997,"5min":0.221,"15min":0.221},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":2.9399999999999999},"last":0.33900000000000002}]},{"osd":2,"last update":"Thu Oct 19 13:04:33 2023","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.439,"15min":0.45900000000000002},"min":{"1min":0.30599999999999999,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.65700000000000003,"5min":0.70799999999999996,"15min":0.97699999999999998},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.499,"5min":0.48099999999999998,"15min":0.48799999999999999},"min":{"1min":0.34100000000000003,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.73099999999999998,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.41799999999999998}]},{"osd":4,"last update":"Thu Oct 19 13:04:33 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.63500000000000001,"15min":0.66400000000000003},"min":{"1min":0.46999999999999997,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.88400000000000001,"5min":0.96299999999999997,"15min":1.0069999999999999},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.66000000000000003,"15min":0.68999999999999995},"min":{"1min":0.44800000000000001,"5min":0.311,"15min":0.311},"max":{"1min":0.90400000000000003,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.752}]},{"osd":5,"last update":"Thu Oct 19 13:04:33 2023","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.64400000000000002,"15min":0.66500000000000004},"min":{"1min":0.44900000000000001,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.96299999999999997,"5min":0.96699999999999997,"15min":2.9990000000000001},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.623,"15min":0.64900000000000002},"min":{"1min":0.435,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":1.774},"last":0.69999999999999996}]},{"osd":6,"last update":"Thu Oct 19 13:04:33 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.63800000000000001,"15min":0.65300000000000002},"min":{"1min":0.498,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.97599999999999998,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.66600000000000004,"15min":0.68000000000000005},"min":{"1min":0.41699999999999998,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.94899999999999995,"5min":1.0429999999999999,"15min":1.167},"last":0.53600000000000003}]},{"osd":7,"last update":"Thu Oct 19 13:04:33 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.64000000000000001,"15min":0.66200000000000003},"min":{"1min":0.433,"5min":0.42099999999999999,"15min":0.39400000000000002},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":1.042},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.66600000000000004,"15min":0.68100000000000005},"min":{"1min":0.46999999999999997,"5min":0.441,"15min":0.432},"max":{"1min":1.0109999999999999,"5min":1.1160000000000001,"15min":2.8149999999999999},"last":0.66700000000000004}]}]},{"osd":2,"up_from":13,"seq":55834575101,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59472,"kb_used_data":1920,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312368,"statfs":{"total":96636764160,"available":96575864832,"internally_reserved":0,"allocated":1966080,"data_stored":991384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 13:04:49 2023","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.376,"15min":0.377},"min":{"1min":0.23200000000000001,"5min":0.20699999999999999,"15min":0.19700000000000001},"max":{"1min":0.53000000000000003,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.20499999999999999},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.39600000000000002,"15min":0.40600000000000003},"min":{"1min":0.25800000000000001,"5min":0.222,"15min":0.20999999999999999},"max":{"1min":0.497,"5min":0.68000000000000005,"15min":0.74099999999999999},"last":0.22600000000000001}]},{"osd":1,"last update":"Thu Oct 19 13:04:49 2023","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.38300000000000001,"15min":0.39100000000000001},"min":{"1min":0.27200000000000002,"5min":0.249,"15min":0.24299999999999999},"max":{"1min":0.59099999999999997,"5min":0.64100000000000001,"15min":0.71099999999999997},"last":0.248},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.41199999999999998,"15min":0.42199999999999999},"min":{"1min":0.29299999999999998,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":0.64800000000000002,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.27400000000000002}]},{"osd":3,"last update":"Thu Oct 19 13:04:49 2023","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.434,"15min":0.45000000000000001},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.22800000000000001},"max":{"1min":0.53000000000000003,"5min":0.69299999999999995,"15min":0.86599999999999999},"last":0.318},{"interface":"front","average":{"1min":0.439,"5min":0.46600000000000003,"15min":0.47899999999999998},"min":{"1min":0.35999999999999999,"5min":0.32200000000000001,"15min":0.29099999999999998},"max":{"1min":0.56200000000000006,"5min":0.72999999999999998,"15min":0.82999999999999996},"last":0.30199999999999999}]},{"osd":4,"last update":"Thu Oct 19 13:04:49 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.60199999999999998,"15min":0.63300000000000001},"min":{"1min":0.42999999999999999,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.66900000000000004,"5min":0.874,"15min":0.98999999999999999},"last":0.496},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.61499999999999999,"15min":0.64500000000000002},"min":{"1min":0.45800000000000002,"5min":0.41199999999999998,"15min":0.35099999999999998},"max":{"1min":0.77000000000000002,"5min":0.97699999999999998,"15min":1.046},"last":0.53400000000000003}]},{"osd":5,"last update":"Thu Oct 19 13:04:49 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.621,"15min":0.65100000000000002},"min":{"1min":0.44400000000000001,"5min":0.40799999999999997,"15min":0.31900000000000001},"max":{"1min":0.73899999999999999,"5min":0.96199999999999997,"15min":0.99399999999999999},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.62,"15min":0.65000000000000002},"min":{"1min":0.39600000000000002,"5min":0.39100000000000001,"15min":0.375},"max":{"1min":0.748,"5min":0.94199999999999995,"15min":0.95799999999999996},"last":0.46300000000000002}]},{"osd":6,"last update":"Thu Oct 19 13:04:49 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.63700000000000001,"15min":0.65200000000000002},"min":{"1min":0.45900000000000002,"5min":0.40400000000000003,"15min":0.40300000000000002},"max":{"1min":0.77400000000000002,"5min":0.93000000000000005,"15min":1.008},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.64100000000000001,"15min":0.66800000000000004},"min":{"1min":0.46700000000000003,"5min":0.44600000000000001,"15min":0.371},"max":{"1min":0.76300000000000001,"5min":0.90400000000000003,"15min":0.995},"last":0.56699999999999995}]},{"osd":7,"last update":"Thu Oct 19 13:04:49 2023","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.64700000000000002,"15min":0.67200000000000004},"min":{"1min":0.49099999999999999,"5min":0.39800000000000002,"15min":0.36399999999999999},"max":{"1min":1.163,"5min":1.163,"15min":1.163},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.65200000000000002,"15min":0.67300000000000004},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.38200000000000001},"max":{"1min":0.79700000000000004,"5min":0.95199999999999996,"15min":1.0329999999999999},"last":0.58999999999999997}]}]},{"osd":1,"up_from":13,"seq":55834575100,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37244,"kb_used_data":1904,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334596,"statfs":{"total":96636764160,"available":96598626304,"internally_reserved":0,"allocated":1949696,"data_stored":984190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 13:04:45 2023","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.36099999999999999,"15min":0.38},"min":{"1min":0.23100000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.50600000000000001,"5min":0.59299999999999997,"15min":1.3879999999999999},"last":0.20200000000000001},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.41199999999999998,"15min":0.433},"min":{"1min":0.252,"5min":0.217,"15min":0.217},"max":{"1min":0.63900000000000001,"5min":0.76100000000000001,"15min":2.5800000000000001},"last":0.308}]},{"osd":2,"last update":"Thu Oct 19 13:04:45 2023","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.40799999999999997,"15min":0.42099999999999999},"min":{"1min":0.30099999999999999,"5min":0.24199999999999999,"15min":0.21299999999999999},"max":{"1min":0.61199999999999999,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.437,"15min":0.45500000000000002},"min":{"1min":0.27500000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.56899999999999995,"5min":0.79800000000000004,"15min":2.6230000000000002},"last":0.34499999999999997}]},{"osd":3,"last update":"Thu Oct 19 13:04:45 2023","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.45600000000000002,"15min":0.48499999999999999},"min":{"1min":0.29199999999999998,"5min":0.246,"15min":0.246},"max":{"1min":0.64700000000000002,"5min":1.008,"15min":3.9740000000000002},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.41299999999999998,"15min":0.42399999999999999},"min":{"1min":0.30199999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.67000000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.32700000000000001}]},{"osd":4,"last update":"Thu Oct 19 13:04:45 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.61499999999999999,"15min":0.64200000000000002},"min":{"1min":0.40400000000000003,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.78300000000000003,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.622,"5min":0.60499999999999998,"15min":0.63600000000000001},"min":{"1min":0.41399999999999998,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":1.0860000000000001},"last":0.54500000000000004}]},{"osd":5,"last update":"Thu Oct 19 13:04:45 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.63600000000000001,"15min":0.64900000000000002},"min":{"1min":0.45200000000000001,"5min":0.38700000000000001,"15min":0.38100000000000001},"max":{"1min":0.86599999999999999,"5min":1.0549999999999999,"15min":1.1200000000000001},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.64300000000000002,"15min":0.67000000000000004},"min":{"1min":0.46500000000000002,"5min":0.374,"15min":0.374},"max":{"1min":0.90300000000000002,"5min":1.1020000000000001,"15min":2.7120000000000002},"last":0.56100000000000005}]},{"osd":6,"last update":"Thu Oct 19 13:04:45 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.60699999999999998,"15min":0.64300000000000002},"min":{"1min":0.435,"5min":0.41499999999999998,"15min":0.36799999999999999},"max":{"1min":0.81599999999999995,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64400000000000002,"15min":0.66700000000000004},"min":{"1min":0.503,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.89200000000000002,"5min":1.129,"15min":1.129},"last":0.58899999999999997}]},{"osd":7,"last update":"Thu Oct 19 13:04:45 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.64800000000000002,"15min":0.67300000000000004},"min":{"1min":0.46800000000000003,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.76400000000000001,"5min":1.1439999999999999,"15min":2.786},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.63600000000000001,"15min":0.66600000000000004},"min":{"1min":0.47699999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.88700000000000001,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.53100000000000003}]}]},{"osd":0,"up_from":13,"seq":55834575101,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33668,"kb_used_data":2484,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338172,"statfs":{"total":96636764160,"available":96602288128,"internally_reserved":0,"allocated":2543616,"data_stored":1573925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Oct 19 13:04:48 2023","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.374,"15min":0.39000000000000001},"min":{"1min":0.24299999999999999,"5min":0.20799999999999999,"15min":0.182},"max":{"1min":0.54900000000000004,"5min":0.624,"15min":0.81799999999999995},"last":0.252},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.39100000000000001,"15min":0.42299999999999999},"min":{"1min":0.26000000000000001,"5min":0.22500000000000001,"15min":0.20200000000000001},"max":{"1min":0.628,"5min":0.64800000000000002,"15min":0.69999999999999996},"last":0.38300000000000001}]},{"osd":2,"last update":"Thu Oct 19 13:04:48 2023","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.41799999999999998,"15min":0.44900000000000001},"min":{"1min":0.30599999999999999,"5min":0.27100000000000002,"15min":0.26200000000000001},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.76300000000000001},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.40400000000000003,"15min":0.42999999999999999},"min":{"1min":0.253,"5min":0.246,"15min":0.20699999999999999},"max":{"1min":0.60299999999999998,"5min":0.68500000000000005,"15min":0.72999999999999998},"last":0.36299999999999999}]},{"osd":3,"last update":"Thu Oct 19 13:04:48 2023","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.376,"15min":0.40400000000000003},"min":{"1min":0.27500000000000002,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.50900000000000001,"5min":0.60399999999999998,"15min":1.2829999999999999},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.44,"5min":0.441,"15min":0.48399999999999999},"min":{"1min":0.33600000000000002,"5min":0.29299999999999998,"15min":0.29099999999999998},"max":{"1min":0.58199999999999996,"5min":0.71199999999999997,"15min":1.4990000000000001},"last":0.41199999999999998}]},{"osd":4,"last update":"Thu Oct 19 13:04:48 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.58999999999999997,"15min":0.625},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.34999999999999998},"max":{"1min":1.175,"5min":1.175,"15min":1.361},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.61199999999999999,"15min":0.64700000000000002},"min":{"1min":0.38600000000000001,"5min":0.36399999999999999,"15min":0.35999999999999999},"max":{"1min":0.77500000000000002,"5min":0.93799999999999994,"15min":0.98999999999999999},"last":0.63500000000000001}]},{"osd":5,"last update":"Thu Oct 19 13:04:48 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.64400000000000002,"15min":0.67600000000000005},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.378},"max":{"1min":1.137,"5min":1.137,"15min":1.325},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.63900000000000001,"15min":0.67900000000000005},"min":{"1min":0.41599999999999998,"5min":0.40100000000000002,"15min":0.318},"max":{"1min":0.82399999999999995,"5min":0.94099999999999995,"15min":1.024},"last":0.73799999999999999}]},{"osd":6,"last update":"Thu Oct 19 13:04:48 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.60999999999999999,"15min":0.65400000000000003},"min":{"1min":0.46700000000000003,"5min":0.36899999999999999,"15min":0.35899999999999999},"max":{"1min":0.79500000000000004,"5min":0.879,"15min":0.98099999999999998},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.61199999999999999,"15min":0.65600000000000003},"min":{"1min":0.41799999999999998,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.78000000000000003,"5min":0.88700000000000001,"15min":0.995},"last":0.70499999999999996}]},{"osd":7,"last update":"Thu Oct 19 13:04:48 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.61499999999999999,"15min":0.66700000000000004},"min":{"1min":0.42799999999999999,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.872,"5min":0.89100000000000001,"15min":1.0049999999999999},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.626,"15min":0.65600000000000003},"min":{"1min":0.41799999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.204,"5min":1.204,"15min":1.389},"last":0.60899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-19T13:05:01.800 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-19T13:05:01.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.939+0000 7ffa0d8e1640 1 -- 172.21.15.57:0/2480912473 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffa08070920 msgr2=0x7ffa080affe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:05:01.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.939+0000 7ffa0d8e1640 1 --2- 172.21.15.57:0/2480912473 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffa08070920 0x7ffa080affe0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7ffa00005fd0 tx=0x7ffa0002c050 comp rx=0 tx=0).stop 2023-10-19T13:05:01.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.939+0000 7ffa0d8e1640 1 -- 172.21.15.57:0/2480912473 shutdown_connections 2023-10-19T13:05:01.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.939+0000 7ffa0d8e1640 1 --2- 172.21.15.57:0/2480912473 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffa08070920 0x7ffa080affe0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:01.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.939+0000 7ffa0d8e1640 1 --2- 172.21.15.57:0/2480912473 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ffa08141690 0x7ffa080703e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:01.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.939+0000 7ffa0d8e1640 1 -- 172.21.15.57:0/2480912473 >> 172.21.15.57:0/2480912473 conn(0x7ffa08069930 msgr2=0x7ffa08069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:05:01.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.940+0000 7ffa0d8e1640 1 -- 172.21.15.57:0/2480912473 shutdown_connections 2023-10-19T13:05:01.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.940+0000 7ffa0d8e1640 1 -- 172.21.15.57:0/2480912473 wait complete. 2023-10-19T13:05:01.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.941+0000 7ffa0d8e1640 1 Processor -- start 2023-10-19T13:05:01.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.941+0000 7ffa0d8e1640 1 -- start start 2023-10-19T13:05:01.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.942+0000 7ffa0d8e1640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ffa08070920 0x7ffa0813b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:05:01.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.942+0000 7ffa0d8e1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffa0813e520 0x7ffa081e3220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:05:01.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.942+0000 7ffa0d8e1640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffa08141690 0x7ffa080aee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:05:01.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.942+0000 7ffa0d8e1640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa0813b780 con 0x7ffa0813e520 2023-10-19T13:05:01.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.942+0000 7ffa0d8e1640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa081e3ce0 con 0x7ffa08141690 2023-10-19T13:05:01.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.942+0000 7ffa0d8e1640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa081e3fc0 con 0x7ffa08070920 2023-10-19T13:05:01.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.942+0000 7ffa067fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffa0813e520 0x7ffa081e3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:05:01.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.942+0000 7ffa067fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffa0813e520 0x7ffa081e3220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42846/0 (socket says 172.21.15.57:42846) 2023-10-19T13:05:01.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.942+0000 7ffa067fc640 1 -- 172.21.15.57:0/4220255789 learned_addr learned my addr 172.21.15.57:0/4220255789 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:05:01.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.942+0000 7ffa077fe640 1 --2- 172.21.15.57:0/4220255789 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffa08141690 0x7ffa080aee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:05:01.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.942+0000 7ffa06ffd640 1 --2- 172.21.15.57:0/4220255789 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ffa08070920 0x7ffa0813b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:05:01.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.942+0000 7ffa067fc640 1 -- 172.21.15.57:0/4220255789 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ffa08070920 msgr2=0x7ffa0813b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:05:01.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.942+0000 7ffa067fc640 1 --2- 172.21.15.57:0/4220255789 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ffa08070920 0x7ffa0813b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:01.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.942+0000 7ffa067fc640 1 -- 172.21.15.57:0/4220255789 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffa08141690 msgr2=0x7ffa080aee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:05:01.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.942+0000 7ffa067fc640 1 --2- 172.21.15.57:0/4220255789 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffa08141690 0x7ffa080aee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:01.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.942+0000 7ffa067fc640 1 -- 172.21.15.57:0/4220255789 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa00009840 con 0x7ffa0813e520 2023-10-19T13:05:01.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.942+0000 7ffa067fc640 1 --2- 172.21.15.57:0/4220255789 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffa0813e520 0x7ffa081e3220 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7ff9fc00e790 tx=0x7ff9fc00ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:05:01.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.942+0000 7ff9effff640 1 -- 172.21.15.57:0/4220255789 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9fc009580 con 0x7ffa0813e520 2023-10-19T13:05:01.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.942+0000 7ff9effff640 1 -- 172.21.15.57:0/4220255789 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7ff9fc00cdb0 con 0x7ffa0813e520 2023-10-19T13:05:01.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.942+0000 7ffa0d8e1640 1 -- 172.21.15.57:0/4220255789 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa081e42a0 con 0x7ffa0813e520 2023-10-19T13:05:01.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.943+0000 7ff9effff640 1 -- 172.21.15.57:0/4220255789 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9fc010690 con 0x7ffa0813e520 2023-10-19T13:05:01.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.943+0000 7ffa0d8e1640 1 -- 172.21.15.57:0/4220255789 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa081e4860 con 0x7ffa0813e520 2023-10-19T13:05:01.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.943+0000 7ffa0d8e1640 1 -- 172.21.15.57:0/4220255789 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa080690e0 con 0x7ffa0813e520 2023-10-19T13:05:01.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.944+0000 7ff9effff640 1 -- 172.21.15.57:0/4220255789 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 79) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff9fc0107f0 con 0x7ffa0813e520 2023-10-19T13:05:01.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.944+0000 7ff9effff640 1 --2- 172.21.15.57:0/4220255789 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7ff9f406af60 0x7ff9f406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:05:01.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.944+0000 7ff9effff640 1 -- 172.21.15.57:0/4220255789 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 6057+0+0 (secure 0 0 0) 0x7ff9fc014070 con 0x7ffa0813e520 2023-10-19T13:05:01.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.945+0000 7ffa06ffd640 1 --2- 172.21.15.57:0/4220255789 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7ff9f406af60 0x7ff9f406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:05:01.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.945+0000 7ffa06ffd640 1 --2- 172.21.15.57:0/4220255789 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7ff9f406af60 0x7ff9f406d420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff9f80046b0 tx=0x7ff9f8007eb0 comp rx=0 tx=0).ready entity=mgr.8419 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:05:01.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:01.948+0000 7ff9effff640 1 -- 172.21.15.57:0/4220255789 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff9fc051ca0 con 0x7ffa0813e520 2023-10-19T13:05:02.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.120+0000 7ffa0d8e1640 1 -- 172.21.15.57:0/4220255789 --> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ffa0806b140 con 0x7ff9f406af60 2023-10-19T13:05:02.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.128+0000 7ff9effff640 1 -- 172.21.15.57:0/4220255789 <== mgr.8419 v2:172.21.15.179:6832/2581690007 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289976 (secure 0 0 0) 0x7ffa0806b140 con 0x7ff9f406af60 2023-10-19T13:05:02.133 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:05:02.134 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2023-10-19T13:05:02.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.131+0000 7ffa0d8e1640 1 -- 172.21.15.57:0/4220255789 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7ff9f406af60 msgr2=0x7ff9f406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:05:02.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.131+0000 7ffa0d8e1640 1 --2- 172.21.15.57:0/4220255789 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7ff9f406af60 0x7ff9f406d420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff9f80046b0 tx=0x7ff9f8007eb0 comp rx=0 tx=0).stop 2023-10-19T13:05:02.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.131+0000 7ffa0d8e1640 1 -- 172.21.15.57:0/4220255789 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffa0813e520 msgr2=0x7ffa081e3220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:05:02.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.131+0000 7ffa0d8e1640 1 --2- 172.21.15.57:0/4220255789 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffa0813e520 0x7ffa081e3220 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7ff9fc00e790 tx=0x7ff9fc00ec50 comp rx=0 tx=0).stop 2023-10-19T13:05:02.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.131+0000 7ffa0d8e1640 1 -- 172.21.15.57:0/4220255789 shutdown_connections 2023-10-19T13:05:02.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.131+0000 7ffa0d8e1640 1 --2- 172.21.15.57:0/4220255789 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7ff9f406af60 0x7ff9f406d420 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:02.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.131+0000 7ffa0d8e1640 1 --2- 172.21.15.57:0/4220255789 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffa08141690 0x7ffa080aee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:02.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.131+0000 7ffa0d8e1640 1 --2- 172.21.15.57:0/4220255789 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffa0813e520 0x7ffa081e3220 unknown :-1 s=CLOSED pgs=1764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:02.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.131+0000 7ffa0d8e1640 1 --2- 172.21.15.57:0/4220255789 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ffa08070920 0x7ffa0813b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:02.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.131+0000 7ffa0d8e1640 1 -- 172.21.15.57:0/4220255789 >> 172.21.15.57:0/4220255789 conn(0x7ffa08069930 msgr2=0x7ffa080b1050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:05:02.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.131+0000 7ffa0d8e1640 1 -- 172.21.15.57:0/4220255789 shutdown_connections 2023-10-19T13:05:02.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.131+0000 7ffa0d8e1640 1 -- 172.21.15.57:0/4220255789 wait complete. 2023-10-19T13:05:02.144 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2023-10-19T13:05:00.298652+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1346,"num_read_kb":2770,"num_write":453,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3125,"ondisk_log_size":3125,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":142,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":345032,"kb_used_data":16468,"kb_used_omap":12,"kb_used_meta":328435,"kb_avail":754629688,"statfs":{"total":773094113280,"available":772740800512,"internally_reserved":0,"allocated":16863232,"data_stored":9084489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":336318009},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001037"},"pg_stats":[{"pgid":"20.3f","version":"0'0","reported_seq":11,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.242336+0000","last_change":"2023-10-19T13:04:58.242023+0000","last_active":"2023-10-19T13:04:59.242336+0000","last_peered":"2023-10-19T13:04:59.242336+0000","last_clean":"2023-10-19T13:04:59.242336+0000","last_became_active":"2023-10-19T13:04:58.241755+0000","last_became_peered":"2023-10-19T13:04:58.241755+0000","last_unstale":"2023-10-19T13:04:59.242336+0000","last_undegraded":"2023-10-19T13:04:59.242336+0000","last_fullsized":"2023-10-19T13:04:59.242336+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:09:10.121831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.3e","version":"0'0","reported_seq":11,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.245756+0000","last_change":"2023-10-19T13:04:58.250174+0000","last_active":"2023-10-19T13:04:59.245756+0000","last_peered":"2023-10-19T13:04:59.245756+0000","last_clean":"2023-10-19T13:04:59.245756+0000","last_became_active":"2023-10-19T13:04:58.249870+0000","last_became_peered":"2023-10-19T13:04:58.249870+0000","last_unstale":"2023-10-19T13:04:59.245756+0000","last_undegraded":"2023-10-19T13:04:59.245756+0000","last_fullsized":"2023-10-19T13:04:59.245756+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:04:35.399006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.3d","version":"0'0","reported_seq":9,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.260326+0000","last_change":"2023-10-19T13:04:58.256603+0000","last_active":"2023-10-19T13:04:58.260326+0000","last_peered":"2023-10-19T13:04:58.260326+0000","last_clean":"2023-10-19T13:04:58.260326+0000","last_became_active":"2023-10-19T13:04:58.256283+0000","last_became_peered":"2023-10-19T13:04:58.256283+0000","last_unstale":"2023-10-19T13:04:58.260326+0000","last_undegraded":"2023-10-19T13:04:58.260326+0000","last_fullsized":"2023-10-19T13:04:58.260326+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:42:37.465815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.3c","version":"0'0","reported_seq":11,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.245677+0000","last_change":"2023-10-19T13:04:58.252328+0000","last_active":"2023-10-19T13:04:59.245677+0000","last_peered":"2023-10-19T13:04:59.245677+0000","last_clean":"2023-10-19T13:04:59.245677+0000","last_became_active":"2023-10-19T13:04:58.252039+0000","last_became_peered":"2023-10-19T13:04:58.252039+0000","last_unstale":"2023-10-19T13:04:59.245677+0000","last_undegraded":"2023-10-19T13:04:59.245677+0000","last_fullsized":"2023-10-19T13:04:59.245677+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:07:16.246850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.3b","version":"0'0","reported_seq":11,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.684863+0000","last_change":"2023-10-19T13:04:58.246606+0000","last_active":"2023-10-19T13:04:59.684863+0000","last_peered":"2023-10-19T13:04:59.684863+0000","last_clean":"2023-10-19T13:04:59.684863+0000","last_became_active":"2023-10-19T13:04:58.246413+0000","last_became_peered":"2023-10-19T13:04:58.246413+0000","last_unstale":"2023-10-19T13:04:59.684863+0000","last_undegraded":"2023-10-19T13:04:59.684863+0000","last_fullsized":"2023-10-19T13:04:59.684863+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:18:09.376247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.3a","version":"0'0","reported_seq":9,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.260314+0000","last_change":"2023-10-19T13:04:58.252857+0000","last_active":"2023-10-19T13:04:58.260314+0000","last_peered":"2023-10-19T13:04:58.260314+0000","last_clean":"2023-10-19T13:04:58.260314+0000","last_became_active":"2023-10-19T13:04:58.252581+0000","last_became_peered":"2023-10-19T13:04:58.252581+0000","last_unstale":"2023-10-19T13:04:58.260314+0000","last_undegraded":"2023-10-19T13:04:58.260314+0000","last_fullsized":"2023-10-19T13:04:58.260314+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:01:25.818792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.39","version":"0'0","reported_seq":9,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.256171+0000","last_change":"2023-10-19T13:04:58.253657+0000","last_active":"2023-10-19T13:04:58.256171+0000","last_peered":"2023-10-19T13:04:58.256171+0000","last_clean":"2023-10-19T13:04:58.256171+0000","last_became_active":"2023-10-19T13:04:58.253085+0000","last_became_peered":"2023-10-19T13:04:58.253085+0000","last_unstale":"2023-10-19T13:04:58.256171+0000","last_undegraded":"2023-10-19T13:04:58.256171+0000","last_fullsized":"2023-10-19T13:04:58.256171+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:29:18.521999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.38","version":"0'0","reported_seq":11,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.246141+0000","last_change":"2023-10-19T13:04:58.246792+0000","last_active":"2023-10-19T13:04:59.246141+0000","last_peered":"2023-10-19T13:04:59.246141+0000","last_clean":"2023-10-19T13:04:59.246141+0000","last_became_active":"2023-10-19T13:04:58.246438+0000","last_became_peered":"2023-10-19T13:04:58.246438+0000","last_unstale":"2023-10-19T13:04:59.246141+0000","last_undegraded":"2023-10-19T13:04:59.246141+0000","last_fullsized":"2023-10-19T13:04:59.246141+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:43:40.413015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.37","version":"0'0","reported_seq":9,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.260105+0000","last_change":"2023-10-19T13:04:58.251146+0000","last_active":"2023-10-19T13:04:58.260105+0000","last_peered":"2023-10-19T13:04:58.260105+0000","last_clean":"2023-10-19T13:04:58.260105+0000","last_became_active":"2023-10-19T13:04:58.250871+0000","last_became_peered":"2023-10-19T13:04:58.250871+0000","last_unstale":"2023-10-19T13:04:58.260105+0000","last_undegraded":"2023-10-19T13:04:58.260105+0000","last_fullsized":"2023-10-19T13:04:58.260105+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:00:57.974372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.36","version":"0'0","reported_seq":11,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:05:00.280520+0000","last_change":"2023-10-19T13:04:58.247772+0000","last_active":"2023-10-19T13:05:00.280520+0000","last_peered":"2023-10-19T13:05:00.280520+0000","last_clean":"2023-10-19T13:05:00.280520+0000","last_became_active":"2023-10-19T13:04:58.247449+0000","last_became_peered":"2023-10-19T13:04:58.247449+0000","last_unstale":"2023-10-19T13:05:00.280520+0000","last_undegraded":"2023-10-19T13:05:00.280520+0000","last_fullsized":"2023-10-19T13:05:00.280520+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:47:36.006848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.35","version":"0'0","reported_seq":11,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.245615+0000","last_change":"2023-10-19T13:04:58.261170+0000","last_active":"2023-10-19T13:04:59.245615+0000","last_peered":"2023-10-19T13:04:59.245615+0000","last_clean":"2023-10-19T13:04:59.245615+0000","last_became_active":"2023-10-19T13:04:58.260860+0000","last_became_peered":"2023-10-19T13:04:58.260860+0000","last_unstale":"2023-10-19T13:04:59.245615+0000","last_undegraded":"2023-10-19T13:04:59.245615+0000","last_fullsized":"2023-10-19T13:04:59.245615+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:37:30.289123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.34","version":"0'0","reported_seq":11,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.242514+0000","last_change":"2023-10-19T13:04:58.251636+0000","last_active":"2023-10-19T13:04:59.242514+0000","last_peered":"2023-10-19T13:04:59.242514+0000","last_clean":"2023-10-19T13:04:59.242514+0000","last_became_active":"2023-10-19T13:04:58.251380+0000","last_became_peered":"2023-10-19T13:04:58.251380+0000","last_unstale":"2023-10-19T13:04:59.242514+0000","last_undegraded":"2023-10-19T13:04:59.242514+0000","last_fullsized":"2023-10-19T13:04:59.242514+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:34:57.252659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.33","version":"0'0","reported_seq":9,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.260369+0000","last_change":"2023-10-19T13:04:58.253732+0000","last_active":"2023-10-19T13:04:58.260369+0000","last_peered":"2023-10-19T13:04:58.260369+0000","last_clean":"2023-10-19T13:04:58.260369+0000","last_became_active":"2023-10-19T13:04:58.253503+0000","last_became_peered":"2023-10-19T13:04:58.253503+0000","last_unstale":"2023-10-19T13:04:58.260369+0000","last_undegraded":"2023-10-19T13:04:58.260369+0000","last_fullsized":"2023-10-19T13:04:58.260369+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:05:39.299198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.32","version":"0'0","reported_seq":11,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.242430+0000","last_change":"2023-10-19T13:04:58.243157+0000","last_active":"2023-10-19T13:04:59.242430+0000","last_peered":"2023-10-19T13:04:59.242430+0000","last_clean":"2023-10-19T13:04:59.242430+0000","last_became_active":"2023-10-19T13:04:58.242897+0000","last_became_peered":"2023-10-19T13:04:58.242897+0000","last_unstale":"2023-10-19T13:04:59.242430+0000","last_undegraded":"2023-10-19T13:04:59.242430+0000","last_fullsized":"2023-10-19T13:04:59.242430+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:30:14.023824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.31","version":"0'0","reported_seq":11,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.242133+0000","last_change":"2023-10-19T13:04:58.242545+0000","last_active":"2023-10-19T13:04:59.242133+0000","last_peered":"2023-10-19T13:04:59.242133+0000","last_clean":"2023-10-19T13:04:59.242133+0000","last_became_active":"2023-10-19T13:04:58.242284+0000","last_became_peered":"2023-10-19T13:04:58.242284+0000","last_unstale":"2023-10-19T13:04:59.242133+0000","last_undegraded":"2023-10-19T13:04:59.242133+0000","last_fullsized":"2023-10-19T13:04:59.242133+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:17:46.270258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.30","version":"0'0","reported_seq":9,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.259905+0000","last_change":"2023-10-19T13:04:58.250297+0000","last_active":"2023-10-19T13:04:58.259905+0000","last_peered":"2023-10-19T13:04:58.259905+0000","last_clean":"2023-10-19T13:04:58.259905+0000","last_became_active":"2023-10-19T13:04:58.249934+0000","last_became_peered":"2023-10-19T13:04:58.249934+0000","last_unstale":"2023-10-19T13:04:58.259905+0000","last_undegraded":"2023-10-19T13:04:58.259905+0000","last_fullsized":"2023-10-19T13:04:58.259905+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:20:17.526434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.2f","version":"0'0","reported_seq":9,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.260089+0000","last_change":"2023-10-19T13:04:58.251059+0000","last_active":"2023-10-19T13:04:58.260089+0000","last_peered":"2023-10-19T13:04:58.260089+0000","last_clean":"2023-10-19T13:04:58.260089+0000","last_became_active":"2023-10-19T13:04:58.250807+0000","last_became_peered":"2023-10-19T13:04:58.250807+0000","last_unstale":"2023-10-19T13:04:58.260089+0000","last_undegraded":"2023-10-19T13:04:58.260089+0000","last_fullsized":"2023-10-19T13:04:58.260089+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:52:57.156027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.2e","version":"0'0","reported_seq":11,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.684870+0000","last_change":"2023-10-19T13:04:58.250958+0000","last_active":"2023-10-19T13:04:59.684870+0000","last_peered":"2023-10-19T13:04:59.684870+0000","last_clean":"2023-10-19T13:04:59.684870+0000","last_became_active":"2023-10-19T13:04:58.250144+0000","last_became_peered":"2023-10-19T13:04:58.250144+0000","last_unstale":"2023-10-19T13:04:59.684870+0000","last_undegraded":"2023-10-19T13:04:59.684870+0000","last_fullsized":"2023-10-19T13:04:59.684870+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:25:51.958820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.2d","version":"0'0","reported_seq":9,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.254463+0000","last_change":"2023-10-19T13:04:58.247906+0000","last_active":"2023-10-19T13:04:58.254463+0000","last_peered":"2023-10-19T13:04:58.254463+0000","last_clean":"2023-10-19T13:04:58.254463+0000","last_became_active":"2023-10-19T13:04:58.247629+0000","last_became_peered":"2023-10-19T13:04:58.247629+0000","last_unstale":"2023-10-19T13:04:58.254463+0000","last_undegraded":"2023-10-19T13:04:58.254463+0000","last_fullsized":"2023-10-19T13:04:58.254463+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:34:12.330622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.2c","version":"0'0","reported_seq":11,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.684881+0000","last_change":"2023-10-19T13:04:58.255611+0000","last_active":"2023-10-19T13:04:59.684881+0000","last_peered":"2023-10-19T13:04:59.684881+0000","last_clean":"2023-10-19T13:04:59.684881+0000","last_became_active":"2023-10-19T13:04:58.255324+0000","last_became_peered":"2023-10-19T13:04:58.255324+0000","last_unstale":"2023-10-19T13:04:59.684881+0000","last_undegraded":"2023-10-19T13:04:59.684881+0000","last_fullsized":"2023-10-19T13:04:59.684881+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:45:55.505917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.2b","version":"0'0","reported_seq":11,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.245583+0000","last_change":"2023-10-19T13:04:58.258147+0000","last_active":"2023-10-19T13:04:59.245583+0000","last_peered":"2023-10-19T13:04:59.245583+0000","last_clean":"2023-10-19T13:04:59.245583+0000","last_became_active":"2023-10-19T13:04:58.257822+0000","last_became_peered":"2023-10-19T13:04:58.257822+0000","last_unstale":"2023-10-19T13:04:59.245583+0000","last_undegraded":"2023-10-19T13:04:59.245583+0000","last_fullsized":"2023-10-19T13:04:59.245583+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:47:10.547688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.2a","version":"0'0","reported_seq":11,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.242265+0000","last_change":"2023-10-19T13:04:58.248348+0000","last_active":"2023-10-19T13:04:59.242265+0000","last_peered":"2023-10-19T13:04:59.242265+0000","last_clean":"2023-10-19T13:04:59.242265+0000","last_became_active":"2023-10-19T13:04:58.248093+0000","last_became_peered":"2023-10-19T13:04:58.248093+0000","last_unstale":"2023-10-19T13:04:59.242265+0000","last_undegraded":"2023-10-19T13:04:59.242265+0000","last_fullsized":"2023-10-19T13:04:59.242265+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:01:09.959518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.29","version":"0'0","reported_seq":11,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.245433+0000","last_change":"2023-10-19T13:04:58.257763+0000","last_active":"2023-10-19T13:04:59.245433+0000","last_peered":"2023-10-19T13:04:59.245433+0000","last_clean":"2023-10-19T13:04:59.245433+0000","last_became_active":"2023-10-19T13:04:58.257164+0000","last_became_peered":"2023-10-19T13:04:58.257164+0000","last_unstale":"2023-10-19T13:04:59.245433+0000","last_undegraded":"2023-10-19T13:04:59.245433+0000","last_fullsized":"2023-10-19T13:04:59.245433+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:55:49.837946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.28","version":"0'0","reported_seq":9,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.255484+0000","last_change":"2023-10-19T13:04:58.248215+0000","last_active":"2023-10-19T13:04:58.255484+0000","last_peered":"2023-10-19T13:04:58.255484+0000","last_clean":"2023-10-19T13:04:58.255484+0000","last_became_active":"2023-10-19T13:04:58.247945+0000","last_became_peered":"2023-10-19T13:04:58.247945+0000","last_unstale":"2023-10-19T13:04:58.255484+0000","last_undegraded":"2023-10-19T13:04:58.255484+0000","last_fullsized":"2023-10-19T13:04:58.255484+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:42:37.465815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.27","version":"0'0","reported_seq":9,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.259776+0000","last_change":"2023-10-19T13:04:58.254641+0000","last_active":"2023-10-19T13:04:58.259776+0000","last_peered":"2023-10-19T13:04:58.259776+0000","last_clean":"2023-10-19T13:04:58.259776+0000","last_became_active":"2023-10-19T13:04:58.254378+0000","last_became_peered":"2023-10-19T13:04:58.254378+0000","last_unstale":"2023-10-19T13:04:58.259776+0000","last_undegraded":"2023-10-19T13:04:58.259776+0000","last_fullsized":"2023-10-19T13:04:58.259776+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:35:32.055794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.26","version":"0'0","reported_seq":9,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.263757+0000","last_change":"2023-10-19T13:04:58.262809+0000","last_active":"2023-10-19T13:04:58.263757+0000","last_peered":"2023-10-19T13:04:58.263757+0000","last_clean":"2023-10-19T13:04:58.263757+0000","last_became_active":"2023-10-19T13:04:58.262487+0000","last_became_peered":"2023-10-19T13:04:58.262487+0000","last_unstale":"2023-10-19T13:04:58.263757+0000","last_undegraded":"2023-10-19T13:04:58.263757+0000","last_fullsized":"2023-10-19T13:04:58.263757+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:02:00.626730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.25","version":"0'0","reported_seq":9,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.259638+0000","last_change":"2023-10-19T13:04:58.258577+0000","last_active":"2023-10-19T13:04:58.259638+0000","last_peered":"2023-10-19T13:04:58.259638+0000","last_clean":"2023-10-19T13:04:58.259638+0000","last_became_active":"2023-10-19T13:04:58.252021+0000","last_became_peered":"2023-10-19T13:04:58.252021+0000","last_unstale":"2023-10-19T13:04:58.259638+0000","last_undegraded":"2023-10-19T13:04:58.259638+0000","last_fullsized":"2023-10-19T13:04:58.259638+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:59:39.956253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.24","version":"0'0","reported_seq":11,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:05:00.280496+0000","last_change":"2023-10-19T13:04:58.247785+0000","last_active":"2023-10-19T13:05:00.280496+0000","last_peered":"2023-10-19T13:05:00.280496+0000","last_clean":"2023-10-19T13:05:00.280496+0000","last_became_active":"2023-10-19T13:04:58.247504+0000","last_became_peered":"2023-10-19T13:04:58.247504+0000","last_unstale":"2023-10-19T13:05:00.280496+0000","last_undegraded":"2023-10-19T13:05:00.280496+0000","last_fullsized":"2023-10-19T13:05:00.280496+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:59:52.674088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.23","version":"0'0","reported_seq":9,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.260025+0000","last_change":"2023-10-19T13:04:58.256809+0000","last_active":"2023-10-19T13:04:58.260025+0000","last_peered":"2023-10-19T13:04:58.260025+0000","last_clean":"2023-10-19T13:04:58.260025+0000","last_became_active":"2023-10-19T13:04:58.256215+0000","last_became_peered":"2023-10-19T13:04:58.256215+0000","last_unstale":"2023-10-19T13:04:58.260025+0000","last_undegraded":"2023-10-19T13:04:58.260025+0000","last_fullsized":"2023-10-19T13:04:58.260025+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:20:17.526434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.22","version":"0'0","reported_seq":9,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.259795+0000","last_change":"2023-10-19T13:04:58.255029+0000","last_active":"2023-10-19T13:04:58.259795+0000","last_peered":"2023-10-19T13:04:58.259795+0000","last_clean":"2023-10-19T13:04:58.259795+0000","last_became_active":"2023-10-19T13:04:58.254768+0000","last_became_peered":"2023-10-19T13:04:58.254768+0000","last_unstale":"2023-10-19T13:04:58.259795+0000","last_undegraded":"2023-10-19T13:04:58.259795+0000","last_fullsized":"2023-10-19T13:04:58.259795+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:40:45.798849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.21","version":"0'0","reported_seq":11,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.246195+0000","last_change":"2023-10-19T13:04:58.256475+0000","last_active":"2023-10-19T13:04:59.246195+0000","last_peered":"2023-10-19T13:04:59.246195+0000","last_clean":"2023-10-19T13:04:59.246195+0000","last_became_active":"2023-10-19T13:04:58.256206+0000","last_became_peered":"2023-10-19T13:04:58.256206+0000","last_unstale":"2023-10-19T13:04:59.246195+0000","last_undegraded":"2023-10-19T13:04:59.246195+0000","last_fullsized":"2023-10-19T13:04:59.246195+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:19:05.998271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.20","version":"0'0","reported_seq":11,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:05:00.280346+0000","last_change":"2023-10-19T13:04:58.252022+0000","last_active":"2023-10-19T13:05:00.280346+0000","last_peered":"2023-10-19T13:05:00.280346+0000","last_clean":"2023-10-19T13:05:00.280346+0000","last_became_active":"2023-10-19T13:04:58.251734+0000","last_became_peered":"2023-10-19T13:04:58.251734+0000","last_unstale":"2023-10-19T13:05:00.280346+0000","last_undegraded":"2023-10-19T13:05:00.280346+0000","last_fullsized":"2023-10-19T13:05:00.280346+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:09:58.468043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":11,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.246135+0000","last_change":"2023-10-19T13:04:58.253969+0000","last_active":"2023-10-19T13:04:59.246135+0000","last_peered":"2023-10-19T13:04:59.246135+0000","last_clean":"2023-10-19T13:04:59.246135+0000","last_became_active":"2023-10-19T13:04:58.253684+0000","last_became_peered":"2023-10-19T13:04:58.253684+0000","last_unstale":"2023-10-19T13:04:59.246135+0000","last_undegraded":"2023-10-19T13:04:59.246135+0000","last_fullsized":"2023-10-19T13:04:59.246135+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:21:13.640969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1f","version":"156'4","reported_seq":15,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.787196+0000","last_change":"2023-10-19T13:04:57.781301+0000","last_active":"2023-10-19T13:04:58.787196+0000","last_peered":"2023-10-19T13:04:58.787196+0000","last_clean":"2023-10-19T13:04:58.787196+0000","last_became_active":"2023-10-19T13:04:57.781027+0000","last_became_peered":"2023-10-19T13:04:57.781027+0000","last_unstale":"2023-10-19T13:04:58.787196+0000","last_undegraded":"2023-10-19T13:04:58.787196+0000","last_fullsized":"2023-10-19T13:04:58.787196+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_clean_scrub_stamp":"2023-10-19T13:04:56.754267+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:49:14.845036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":11,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.242190+0000","last_change":"2023-10-19T13:04:58.247912+0000","last_active":"2023-10-19T13:04:59.242190+0000","last_peered":"2023-10-19T13:04:59.242190+0000","last_clean":"2023-10-19T13:04:59.242190+0000","last_became_active":"2023-10-19T13:04:58.247657+0000","last_became_peered":"2023-10-19T13:04:58.247657+0000","last_unstale":"2023-10-19T13:04:59.242190+0000","last_undegraded":"2023-10-19T13:04:59.242190+0000","last_fullsized":"2023-10-19T13:04:59.242190+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:50:20.587539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1e","version":"156'1","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.685096+0000","last_change":"2023-10-19T13:04:57.782578+0000","last_active":"2023-10-19T13:04:59.685096+0000","last_peered":"2023-10-19T13:04:59.685096+0000","last_clean":"2023-10-19T13:04:59.685096+0000","last_became_active":"2023-10-19T13:04:57.782329+0000","last_became_peered":"2023-10-19T13:04:57.782329+0000","last_unstale":"2023-10-19T13:04:59.685096+0000","last_undegraded":"2023-10-19T13:04:59.685096+0000","last_fullsized":"2023-10-19T13:04:59.685096+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_clean_scrub_stamp":"2023-10-19T13:04:56.754267+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:05:19.027331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1a","version":"0'0","reported_seq":11,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.245582+0000","last_change":"2023-10-19T13:04:58.257796+0000","last_active":"2023-10-19T13:04:59.245582+0000","last_peered":"2023-10-19T13:04:59.245582+0000","last_clean":"2023-10-19T13:04:59.245582+0000","last_became_active":"2023-10-19T13:04:58.257214+0000","last_became_peered":"2023-10-19T13:04:58.257214+0000","last_unstale":"2023-10-19T13:04:59.245582+0000","last_undegraded":"2023-10-19T13:04:59.245582+0000","last_fullsized":"2023-10-19T13:04:59.245582+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:21:53.914585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1d","version":"156'2","reported_seq":13,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.787220+0000","last_change":"2023-10-19T13:04:57.789046+0000","last_active":"2023-10-19T13:04:58.787220+0000","last_peered":"2023-10-19T13:04:58.787220+0000","last_clean":"2023-10-19T13:04:58.787220+0000","last_became_active":"2023-10-19T13:04:57.788769+0000","last_became_peered":"2023-10-19T13:04:57.788769+0000","last_unstale":"2023-10-19T13:04:58.787220+0000","last_undegraded":"2023-10-19T13:04:58.787220+0000","last_fullsized":"2023-10-19T13:04:58.787220+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_clean_scrub_stamp":"2023-10-19T13:04:56.754267+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:21:52.909025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1b","version":"0'0","reported_seq":11,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.242321+0000","last_change":"2023-10-19T13:04:58.243725+0000","last_active":"2023-10-19T13:04:59.242321+0000","last_peered":"2023-10-19T13:04:59.242321+0000","last_clean":"2023-10-19T13:04:59.242321+0000","last_became_active":"2023-10-19T13:04:58.243487+0000","last_became_peered":"2023-10-19T13:04:58.243487+0000","last_unstale":"2023-10-19T13:04:59.242321+0000","last_undegraded":"2023-10-19T13:04:59.242321+0000","last_fullsized":"2023-10-19T13:04:59.242321+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:17:52.329819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1c","version":"156'2","reported_seq":13,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.786704+0000","last_change":"2023-10-19T13:04:57.786044+0000","last_active":"2023-10-19T13:04:58.786704+0000","last_peered":"2023-10-19T13:04:58.786704+0000","last_clean":"2023-10-19T13:04:58.786704+0000","last_became_active":"2023-10-19T13:04:57.785572+0000","last_became_peered":"2023-10-19T13:04:57.785572+0000","last_unstale":"2023-10-19T13:04:58.786704+0000","last_undegraded":"2023-10-19T13:04:58.786704+0000","last_fullsized":"2023-10-19T13:04:58.786704+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_clean_scrub_stamp":"2023-10-19T13:04:56.754267+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:44:42.677156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1c","version":"0'0","reported_seq":11,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.246588+0000","last_change":"2023-10-19T13:04:58.256747+0000","last_active":"2023-10-19T13:04:59.246588+0000","last_peered":"2023-10-19T13:04:59.246588+0000","last_clean":"2023-10-19T13:04:59.246588+0000","last_became_active":"2023-10-19T13:04:58.256446+0000","last_became_peered":"2023-10-19T13:04:58.256446+0000","last_unstale":"2023-10-19T13:04:59.246588+0000","last_undegraded":"2023-10-19T13:04:59.246588+0000","last_fullsized":"2023-10-19T13:04:59.246588+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:01:25.818792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1b","version":"156'1","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.246453+0000","last_change":"2023-10-19T13:04:57.782263+0000","last_active":"2023-10-19T13:04:59.246453+0000","last_peered":"2023-10-19T13:04:59.246453+0000","last_clean":"2023-10-19T13:04:59.246453+0000","last_became_active":"2023-10-19T13:04:57.781924+0000","last_became_peered":"2023-10-19T13:04:57.781924+0000","last_unstale":"2023-10-19T13:04:59.246453+0000","last_undegraded":"2023-10-19T13:04:59.246453+0000","last_fullsized":"2023-10-19T13:04:59.246453+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_clean_scrub_stamp":"2023-10-19T13:04:56.754267+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:05:31.203658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.1d","version":"0'0","reported_seq":11,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.246636+0000","last_change":"2023-10-19T13:04:58.253745+0000","last_active":"2023-10-19T13:04:59.246636+0000","last_peered":"2023-10-19T13:04:59.246636+0000","last_clean":"2023-10-19T13:04:59.246636+0000","last_became_active":"2023-10-19T13:04:58.253469+0000","last_became_peered":"2023-10-19T13:04:58.253469+0000","last_unstale":"2023-10-19T13:04:59.246636+0000","last_undegraded":"2023-10-19T13:04:59.246636+0000","last_fullsized":"2023-10-19T13:04:59.246636+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:00:57.974372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1a","version":"156'1","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.684688+0000","last_change":"2023-10-19T13:04:57.785484+0000","last_active":"2023-10-19T13:04:59.684688+0000","last_peered":"2023-10-19T13:04:59.684688+0000","last_clean":"2023-10-19T13:04:59.684688+0000","last_became_active":"2023-10-19T13:04:57.785122+0000","last_became_peered":"2023-10-19T13:04:57.785122+0000","last_unstale":"2023-10-19T13:04:59.684688+0000","last_undegraded":"2023-10-19T13:04:59.684688+0000","last_fullsized":"2023-10-19T13:04:59.684688+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_clean_scrub_stamp":"2023-10-19T13:04:56.754267+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:46:37.159756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1e","version":"0'0","reported_seq":11,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.684994+0000","last_change":"2023-10-19T13:04:58.256786+0000","last_active":"2023-10-19T13:04:59.684994+0000","last_peered":"2023-10-19T13:04:59.684994+0000","last_clean":"2023-10-19T13:04:59.684994+0000","last_became_active":"2023-10-19T13:04:58.256522+0000","last_became_peered":"2023-10-19T13:04:58.256522+0000","last_unstale":"2023-10-19T13:04:59.684994+0000","last_undegraded":"2023-10-19T13:04:59.684994+0000","last_fullsized":"2023-10-19T13:04:59.684994+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:33:47.290259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.240562+0000","last_change":"2023-10-19T13:04:57.782709+0000","last_active":"2023-10-19T13:04:58.240562+0000","last_peered":"2023-10-19T13:04:58.240562+0000","last_clean":"2023-10-19T13:04:58.240562+0000","last_became_active":"2023-10-19T13:04:57.782415+0000","last_became_peered":"2023-10-19T13:04:57.782415+0000","last_unstale":"2023-10-19T13:04:58.240562+0000","last_undegraded":"2023-10-19T13:04:58.240562+0000","last_fullsized":"2023-10-19T13:04:58.240562+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_clean_scrub_stamp":"2023-10-19T13:04:56.754267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:04:24.813151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1f","version":"0'0","reported_seq":9,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.259107+0000","last_change":"2023-10-19T13:04:58.258308+0000","last_active":"2023-10-19T13:04:58.259107+0000","last_peered":"2023-10-19T13:04:58.259107+0000","last_clean":"2023-10-19T13:04:58.259107+0000","last_became_active":"2023-10-19T13:04:58.257299+0000","last_became_peered":"2023-10-19T13:04:58.257299+0000","last_unstale":"2023-10-19T13:04:58.259107+0000","last_undegraded":"2023-10-19T13:04:58.259107+0000","last_fullsized":"2023-10-19T13:04:58.259107+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:36:23.188250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.254159+0000","last_change":"2023-10-19T13:04:57.785330+0000","last_active":"2023-10-19T13:04:58.254159+0000","last_peered":"2023-10-19T13:04:58.254159+0000","last_clean":"2023-10-19T13:04:58.254159+0000","last_became_active":"2023-10-19T13:04:57.785054+0000","last_became_peered":"2023-10-19T13:04:57.785054+0000","last_unstale":"2023-10-19T13:04:58.254159+0000","last_undegraded":"2023-10-19T13:04:58.254159+0000","last_fullsized":"2023-10-19T13:04:58.254159+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_clean_scrub_stamp":"2023-10-19T13:04:56.754267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:04:24.813151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.10","version":"0'0","reported_seq":11,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.242126+0000","last_change":"2023-10-19T13:04:58.260661+0000","last_active":"2023-10-19T13:04:59.242126+0000","last_peered":"2023-10-19T13:04:59.242126+0000","last_clean":"2023-10-19T13:04:59.242126+0000","last_became_active":"2023-10-19T13:04:58.260370+0000","last_became_peered":"2023-10-19T13:04:58.260370+0000","last_unstale":"2023-10-19T13:04:59.242126+0000","last_undegraded":"2023-10-19T13:04:59.242126+0000","last_fullsized":"2023-10-19T13:04:59.242126+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:35:58.392101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":12,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.245736+0000","last_change":"2023-10-19T13:04:57.782445+0000","last_active":"2023-10-19T13:04:59.245736+0000","last_peered":"2023-10-19T13:04:59.245736+0000","last_clean":"2023-10-19T13:04:59.245736+0000","last_became_active":"2023-10-19T13:04:57.782166+0000","last_became_peered":"2023-10-19T13:04:57.782166+0000","last_unstale":"2023-10-19T13:04:59.245736+0000","last_undegraded":"2023-10-19T13:04:59.245736+0000","last_fullsized":"2023-10-19T13:04:59.245736+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_clean_scrub_stamp":"2023-10-19T13:04:56.754267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:44:42.677156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":9,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.260101+0000","last_change":"2023-10-19T13:04:58.250760+0000","last_active":"2023-10-19T13:04:58.260101+0000","last_peered":"2023-10-19T13:04:58.260101+0000","last_clean":"2023-10-19T13:04:58.260101+0000","last_became_active":"2023-10-19T13:04:58.250474+0000","last_became_peered":"2023-10-19T13:04:58.250474+0000","last_unstale":"2023-10-19T13:04:58.260101+0000","last_undegraded":"2023-10-19T13:04:58.260101+0000","last_fullsized":"2023-10-19T13:04:58.260101+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:00:28.926969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":12,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.242419+0000","last_change":"2023-10-19T13:04:57.783533+0000","last_active":"2023-10-19T13:04:59.242419+0000","last_peered":"2023-10-19T13:04:59.242419+0000","last_clean":"2023-10-19T13:04:59.242419+0000","last_became_active":"2023-10-19T13:04:57.783271+0000","last_became_peered":"2023-10-19T13:04:57.783271+0000","last_unstale":"2023-10-19T13:04:59.242419+0000","last_undegraded":"2023-10-19T13:04:59.242419+0000","last_fullsized":"2023-10-19T13:04:59.242419+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_clean_scrub_stamp":"2023-10-19T13:04:56.754267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:18:54.089803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":9,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.255266+0000","last_change":"2023-10-19T13:04:58.252270+0000","last_active":"2023-10-19T13:04:58.255266+0000","last_peered":"2023-10-19T13:04:58.255266+0000","last_clean":"2023-10-19T13:04:58.255266+0000","last_became_active":"2023-10-19T13:04:58.250864+0000","last_became_peered":"2023-10-19T13:04:58.250864+0000","last_unstale":"2023-10-19T13:04:58.255266+0000","last_undegraded":"2023-10-19T13:04:58.255266+0000","last_fullsized":"2023-10-19T13:04:58.255266+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:14:30.850273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.15","version":"156'3","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.787642+0000","last_change":"2023-10-19T13:04:57.787963+0000","last_active":"2023-10-19T13:04:58.787642+0000","last_peered":"2023-10-19T13:04:58.787642+0000","last_clean":"2023-10-19T13:04:58.787642+0000","last_became_active":"2023-10-19T13:04:57.787643+0000","last_became_peered":"2023-10-19T13:04:57.787643+0000","last_unstale":"2023-10-19T13:04:58.787642+0000","last_undegraded":"2023-10-19T13:04:58.787642+0000","last_fullsized":"2023-10-19T13:04:58.787642+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_clean_scrub_stamp":"2023-10-19T13:04:56.754267+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:07:15.241290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.13","version":"0'0","reported_seq":9,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.260373+0000","last_change":"2023-10-19T13:04:58.259272+0000","last_active":"2023-10-19T13:04:58.260373+0000","last_peered":"2023-10-19T13:04:58.260373+0000","last_clean":"2023-10-19T13:04:58.260373+0000","last_became_active":"2023-10-19T13:04:58.258949+0000","last_became_peered":"2023-10-19T13:04:58.258949+0000","last_unstale":"2023-10-19T13:04:58.260373+0000","last_undegraded":"2023-10-19T13:04:58.260373+0000","last_fullsized":"2023-10-19T13:04:58.260373+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T13:18:15.409534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.14","version":"156'6","reported_seq":17,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.789400+0000","last_change":"2023-10-19T13:04:57.786159+0000","last_active":"2023-10-19T13:04:58.789400+0000","last_peered":"2023-10-19T13:04:58.789400+0000","last_clean":"2023-10-19T13:04:58.789400+0000","last_became_active":"2023-10-19T13:04:57.785254+0000","last_became_peered":"2023-10-19T13:04:57.785254+0000","last_unstale":"2023-10-19T13:04:58.789400+0000","last_undegraded":"2023-10-19T13:04:58.789400+0000","last_fullsized":"2023-10-19T13:04:58.789400+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_clean_scrub_stamp":"2023-10-19T13:04:56.754267+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:53:51.842918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.14","version":"0'0","reported_seq":11,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:05:00.280618+0000","last_change":"2023-10-19T13:04:58.241710+0000","last_active":"2023-10-19T13:05:00.280618+0000","last_peered":"2023-10-19T13:05:00.280618+0000","last_clean":"2023-10-19T13:05:00.280618+0000","last_became_active":"2023-10-19T13:04:58.241418+0000","last_became_peered":"2023-10-19T13:04:58.241418+0000","last_unstale":"2023-10-19T13:05:00.280618+0000","last_undegraded":"2023-10-19T13:05:00.280618+0000","last_fullsized":"2023-10-19T13:05:00.280618+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:15:51.554211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.13","version":"156'2","reported_seq":14,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.242413+0000","last_change":"2023-10-19T13:04:57.784220+0000","last_active":"2023-10-19T13:04:59.242413+0000","last_peered":"2023-10-19T13:04:59.242413+0000","last_clean":"2023-10-19T13:04:59.242413+0000","last_became_active":"2023-10-19T13:04:57.783948+0000","last_became_peered":"2023-10-19T13:04:57.783948+0000","last_unstale":"2023-10-19T13:04:59.242413+0000","last_undegraded":"2023-10-19T13:04:59.242413+0000","last_fullsized":"2023-10-19T13:04:59.242413+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_clean_scrub_stamp":"2023-10-19T13:04:56.754267+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:39:38.374564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":9,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.256477+0000","last_change":"2023-10-19T13:04:58.255428+0000","last_active":"2023-10-19T13:04:58.256477+0000","last_peered":"2023-10-19T13:04:58.256477+0000","last_clean":"2023-10-19T13:04:58.256477+0000","last_became_active":"2023-10-19T13:04:58.255173+0000","last_became_peered":"2023-10-19T13:04:58.255173+0000","last_unstale":"2023-10-19T13:04:58.256477+0000","last_undegraded":"2023-10-19T13:04:58.256477+0000","last_fullsized":"2023-10-19T13:04:58.256477+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:37:30.289123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.249501+0000","last_change":"2023-10-19T13:04:57.778139+0000","last_active":"2023-10-19T13:04:58.249501+0000","last_peered":"2023-10-19T13:04:58.249501+0000","last_clean":"2023-10-19T13:04:58.249501+0000","last_became_active":"2023-10-19T13:04:57.777871+0000","last_became_peered":"2023-10-19T13:04:57.777871+0000","last_unstale":"2023-10-19T13:04:58.249501+0000","last_undegraded":"2023-10-19T13:04:58.249501+0000","last_fullsized":"2023-10-19T13:04:58.249501+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_clean_scrub_stamp":"2023-10-19T13:04:56.754267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:00:27.921409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":11,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.684971+0000","last_change":"2023-10-19T13:04:58.256004+0000","last_active":"2023-10-19T13:04:59.684971+0000","last_peered":"2023-10-19T13:04:59.684971+0000","last_clean":"2023-10-19T13:04:59.684971+0000","last_became_active":"2023-10-19T13:04:58.255728+0000","last_became_peered":"2023-10-19T13:04:58.255728+0000","last_unstale":"2023-10-19T13:04:59.684971+0000","last_undegraded":"2023-10-19T13:04:59.684971+0000","last_fullsized":"2023-10-19T13:04:59.684971+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:58:49.086652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.10","version":"156'8","reported_seq":19,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.787960+0000","last_change":"2023-10-19T13:04:57.785842+0000","last_active":"2023-10-19T13:04:58.787960+0000","last_peered":"2023-10-19T13:04:58.787960+0000","last_clean":"2023-10-19T13:04:58.787960+0000","last_became_active":"2023-10-19T13:04:57.785580+0000","last_became_peered":"2023-10-19T13:04:57.785580+0000","last_unstale":"2023-10-19T13:04:58.787960+0000","last_undegraded":"2023-10-19T13:04:58.787960+0000","last_fullsized":"2023-10-19T13:04:58.787960+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_clean_scrub_stamp":"2023-10-19T13:04:56.754267+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:58:58.896647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":9,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.262064+0000","last_change":"2023-10-19T13:04:58.259820+0000","last_active":"2023-10-19T13:04:58.262064+0000","last_peered":"2023-10-19T13:04:58.262064+0000","last_clean":"2023-10-19T13:04:58.262064+0000","last_became_active":"2023-10-19T13:04:58.259534+0000","last_became_peered":"2023-10-19T13:04:58.259534+0000","last_unstale":"2023-10-19T13:04:58.262064+0000","last_undegraded":"2023-10-19T13:04:58.262064+0000","last_fullsized":"2023-10-19T13:04:58.262064+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:05:32.209217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.f","version":"156'1","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.245824+0000","last_change":"2023-10-19T13:04:57.778607+0000","last_active":"2023-10-19T13:04:59.245824+0000","last_peered":"2023-10-19T13:04:59.245824+0000","last_clean":"2023-10-19T13:04:59.245824+0000","last_became_active":"2023-10-19T13:04:57.778349+0000","last_became_peered":"2023-10-19T13:04:57.778349+0000","last_unstale":"2023-10-19T13:04:59.245824+0000","last_undegraded":"2023-10-19T13:04:59.245824+0000","last_fullsized":"2023-10-19T13:04:59.245824+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_clean_scrub_stamp":"2023-10-19T13:04:56.754267+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:20:48.667529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":9,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.254467+0000","last_change":"2023-10-19T13:04:58.251317+0000","last_active":"2023-10-19T13:04:58.254467+0000","last_peered":"2023-10-19T13:04:58.254467+0000","last_clean":"2023-10-19T13:04:58.254467+0000","last_became_active":"2023-10-19T13:04:58.250877+0000","last_became_peered":"2023-10-19T13:04:58.250877+0000","last_unstale":"2023-10-19T13:04:58.254467+0000","last_undegraded":"2023-10-19T13:04:58.254467+0000","last_fullsized":"2023-10-19T13:04:58.254467+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:47:10.547688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.e","version":"156'2","reported_seq":13,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.786567+0000","last_change":"2023-10-19T13:04:57.781567+0000","last_active":"2023-10-19T13:04:58.786567+0000","last_peered":"2023-10-19T13:04:58.786567+0000","last_clean":"2023-10-19T13:04:58.786567+0000","last_became_active":"2023-10-19T13:04:57.781069+0000","last_became_peered":"2023-10-19T13:04:57.781069+0000","last_unstale":"2023-10-19T13:04:58.786567+0000","last_undegraded":"2023-10-19T13:04:58.786567+0000","last_fullsized":"2023-10-19T13:04:58.786567+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_clean_scrub_stamp":"2023-10-19T13:04:56.754267+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:58:58.896647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":9,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.254615+0000","last_change":"2023-10-19T13:04:58.252783+0000","last_active":"2023-10-19T13:04:58.254615+0000","last_peered":"2023-10-19T13:04:58.254615+0000","last_clean":"2023-10-19T13:04:58.254615+0000","last_became_active":"2023-10-19T13:04:58.252311+0000","last_became_peered":"2023-10-19T13:04:58.252311+0000","last_unstale":"2023-10-19T13:04:58.254615+0000","last_undegraded":"2023-10-19T13:04:58.254615+0000","last_fullsized":"2023-10-19T13:04:58.254615+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:55:49.837946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.d","version":"156'1","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.785450+0000","last_change":"2023-10-19T13:04:57.785216+0000","last_active":"2023-10-19T13:04:58.785450+0000","last_peered":"2023-10-19T13:04:58.785450+0000","last_clean":"2023-10-19T13:04:58.785450+0000","last_became_active":"2023-10-19T13:04:57.784960+0000","last_became_peered":"2023-10-19T13:04:57.784960+0000","last_unstale":"2023-10-19T13:04:58.785450+0000","last_undegraded":"2023-10-19T13:04:58.785450+0000","last_fullsized":"2023-10-19T13:04:58.785450+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_clean_scrub_stamp":"2023-10-19T13:04:56.754267+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:14:16.121117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.b","version":"0'0","reported_seq":11,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:05:00.280507+0000","last_change":"2023-10-19T13:04:58.253006+0000","last_active":"2023-10-19T13:05:00.280507+0000","last_peered":"2023-10-19T13:05:00.280507+0000","last_clean":"2023-10-19T13:05:00.280507+0000","last_became_active":"2023-10-19T13:04:58.252754+0000","last_became_peered":"2023-10-19T13:04:58.252754+0000","last_unstale":"2023-10-19T13:05:00.280507+0000","last_undegraded":"2023-10-19T13:05:00.280507+0000","last_fullsized":"2023-10-19T13:05:00.280507+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:02:41.859778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":12,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.685046+0000","last_change":"2023-10-19T13:04:57.786506+0000","last_active":"2023-10-19T13:04:59.685046+0000","last_peered":"2023-10-19T13:04:59.685046+0000","last_clean":"2023-10-19T13:04:59.685046+0000","last_became_active":"2023-10-19T13:04:57.786259+0000","last_became_peered":"2023-10-19T13:04:57.786259+0000","last_unstale":"2023-10-19T13:04:59.685046+0000","last_undegraded":"2023-10-19T13:04:59.685046+0000","last_fullsized":"2023-10-19T13:04:59.685046+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_clean_scrub_stamp":"2023-10-19T13:04:56.754267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:36:08.623527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":11,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.685022+0000","last_change":"2023-10-19T13:04:58.245985+0000","last_active":"2023-10-19T13:04:59.685022+0000","last_peered":"2023-10-19T13:04:59.685022+0000","last_clean":"2023-10-19T13:04:59.685022+0000","last_became_active":"2023-10-19T13:04:58.245751+0000","last_became_peered":"2023-10-19T13:04:58.245751+0000","last_unstale":"2023-10-19T13:04:59.685022+0000","last_undegraded":"2023-10-19T13:04:59.685022+0000","last_fullsized":"2023-10-19T13:04:59.685022+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:17:39.136602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.b","version":"156'2","reported_seq":14,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.246465+0000","last_change":"2023-10-19T13:04:57.780196+0000","last_active":"2023-10-19T13:04:59.246465+0000","last_peered":"2023-10-19T13:04:59.246465+0000","last_clean":"2023-10-19T13:04:59.246465+0000","last_became_active":"2023-10-19T13:04:57.779939+0000","last_became_peered":"2023-10-19T13:04:57.779939+0000","last_unstale":"2023-10-19T13:04:59.246465+0000","last_undegraded":"2023-10-19T13:04:59.246465+0000","last_fullsized":"2023-10-19T13:04:59.246465+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_clean_scrub_stamp":"2023-10-19T13:04:56.754267+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:00:27.921409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.d","version":"0'0","reported_seq":11,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.242538+0000","last_change":"2023-10-19T13:04:58.243583+0000","last_active":"2023-10-19T13:04:59.242538+0000","last_peered":"2023-10-19T13:04:59.242538+0000","last_clean":"2023-10-19T13:04:59.242538+0000","last_became_active":"2023-10-19T13:04:58.242868+0000","last_became_peered":"2023-10-19T13:04:58.242868+0000","last_unstale":"2023-10-19T13:04:59.242538+0000","last_undegraded":"2023-10-19T13:04:59.242538+0000","last_fullsized":"2023-10-19T13:04:59.242538+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:39:42.760170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":12,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.242404+0000","last_change":"2023-10-19T13:04:57.774119+0000","last_active":"2023-10-19T13:04:59.242404+0000","last_peered":"2023-10-19T13:04:59.242404+0000","last_clean":"2023-10-19T13:04:59.242404+0000","last_became_active":"2023-10-19T13:04:57.773825+0000","last_became_peered":"2023-10-19T13:04:57.773825+0000","last_unstale":"2023-10-19T13:04:59.242404+0000","last_undegraded":"2023-10-19T13:04:59.242404+0000","last_fullsized":"2023-10-19T13:04:59.242404+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_clean_scrub_stamp":"2023-10-19T13:04:56.754267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:47:49.295344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.e","version":"0'0","reported_seq":11,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.685012+0000","last_change":"2023-10-19T13:04:58.261839+0000","last_active":"2023-10-19T13:04:59.685012+0000","last_peered":"2023-10-19T13:04:59.685012+0000","last_clean":"2023-10-19T13:04:59.685012+0000","last_became_active":"2023-10-19T13:04:58.261672+0000","last_became_peered":"2023-10-19T13:04:58.261672+0000","last_unstale":"2023-10-19T13:04:59.685012+0000","last_undegraded":"2023-10-19T13:04:59.685012+0000","last_fullsized":"2023-10-19T13:04:59.685012+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:51:48.722094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.242033+0000","last_change":"2023-10-19T13:04:57.788324+0000","last_active":"2023-10-19T13:04:58.242033+0000","last_peered":"2023-10-19T13:04:58.242033+0000","last_clean":"2023-10-19T13:04:58.242033+0000","last_became_active":"2023-10-19T13:04:57.787998+0000","last_became_peered":"2023-10-19T13:04:57.787998+0000","last_unstale":"2023-10-19T13:04:58.242033+0000","last_undegraded":"2023-10-19T13:04:58.242033+0000","last_fullsized":"2023-10-19T13:04:58.242033+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_clean_scrub_stamp":"2023-10-19T13:04:56.754267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:32:27.384468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.f","version":"0'0","reported_seq":11,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.245784+0000","last_change":"2023-10-19T13:04:58.254773+0000","last_active":"2023-10-19T13:04:59.245784+0000","last_peered":"2023-10-19T13:04:59.245784+0000","last_clean":"2023-10-19T13:04:59.245784+0000","last_became_active":"2023-10-19T13:04:58.254512+0000","last_became_peered":"2023-10-19T13:04:58.254512+0000","last_unstale":"2023-10-19T13:04:59.245784+0000","last_undegraded":"2023-10-19T13:04:59.245784+0000","last_fullsized":"2023-10-19T13:04:59.245784+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:32:28.390028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.240877+0000","last_change":"2023-10-19T13:04:57.779244+0000","last_active":"2023-10-19T13:04:58.240877+0000","last_peered":"2023-10-19T13:04:58.240877+0000","last_clean":"2023-10-19T13:04:58.240877+0000","last_became_active":"2023-10-19T13:04:57.778889+0000","last_became_peered":"2023-10-19T13:04:57.778889+0000","last_unstale":"2023-10-19T13:04:58.240877+0000","last_undegraded":"2023-10-19T13:04:58.240877+0000","last_fullsized":"2023-10-19T13:04:58.240877+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_clean_scrub_stamp":"2023-10-19T13:04:56.754267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:43:39.407455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":11,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.684819+0000","last_change":"2023-10-19T13:04:58.243809+0000","last_active":"2023-10-19T13:04:59.684819+0000","last_peered":"2023-10-19T13:04:59.684819+0000","last_clean":"2023-10-19T13:04:59.684819+0000","last_became_active":"2023-10-19T13:04:58.243536+0000","last_became_peered":"2023-10-19T13:04:58.243536+0000","last_unstale":"2023-10-19T13:04:59.684819+0000","last_undegraded":"2023-10-19T13:04:59.684819+0000","last_fullsized":"2023-10-19T13:04:59.684819+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:46:25.741582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.7","version":"156'2","reported_seq":14,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.242660+0000","last_change":"2023-10-19T13:04:57.781780+0000","last_active":"2023-10-19T13:04:59.242660+0000","last_peered":"2023-10-19T13:04:59.242660+0000","last_clean":"2023-10-19T13:04:59.242660+0000","last_became_active":"2023-10-19T13:04:57.781487+0000","last_became_peered":"2023-10-19T13:04:57.781487+0000","last_unstale":"2023-10-19T13:04:59.242660+0000","last_undegraded":"2023-10-19T13:04:59.242660+0000","last_fullsized":"2023-10-19T13:04:59.242660+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_clean_scrub_stamp":"2023-10-19T13:04:56.754267+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:35:24.389421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":9,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.259834+0000","last_change":"2023-10-19T13:04:58.255630+0000","last_active":"2023-10-19T13:04:58.259834+0000","last_peered":"2023-10-19T13:04:58.259834+0000","last_clean":"2023-10-19T13:04:58.259834+0000","last_became_active":"2023-10-19T13:04:58.255377+0000","last_became_peered":"2023-10-19T13:04:58.255377+0000","last_unstale":"2023-10-19T13:04:58.259834+0000","last_undegraded":"2023-10-19T13:04:58.259834+0000","last_fullsized":"2023-10-19T13:04:58.259834+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:29:04.911896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":12,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.246758+0000","last_change":"2023-10-19T13:04:57.775919+0000","last_active":"2023-10-19T13:04:59.246758+0000","last_peered":"2023-10-19T13:04:59.246758+0000","last_clean":"2023-10-19T13:04:59.246758+0000","last_became_active":"2023-10-19T13:04:57.775602+0000","last_became_peered":"2023-10-19T13:04:57.775602+0000","last_unstale":"2023-10-19T13:04:59.246758+0000","last_undegraded":"2023-10-19T13:04:59.246758+0000","last_fullsized":"2023-10-19T13:04:59.246758+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_clean_scrub_stamp":"2023-10-19T13:04:56.754267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:04:24.813151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":9,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.254600+0000","last_change":"2023-10-19T13:04:58.252594+0000","last_active":"2023-10-19T13:04:58.254600+0000","last_peered":"2023-10-19T13:04:58.254600+0000","last_clean":"2023-10-19T13:04:58.254600+0000","last_became_active":"2023-10-19T13:04:58.251889+0000","last_became_peered":"2023-10-19T13:04:58.251889+0000","last_unstale":"2023-10-19T13:04:58.254600+0000","last_undegraded":"2023-10-19T13:04:58.254600+0000","last_fullsized":"2023-10-19T13:04:58.254600+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:29:25.303068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":12,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.246718+0000","last_change":"2023-10-19T13:04:57.778790+0000","last_active":"2023-10-19T13:04:59.246718+0000","last_peered":"2023-10-19T13:04:59.246718+0000","last_clean":"2023-10-19T13:04:59.246718+0000","last_became_active":"2023-10-19T13:04:57.778481+0000","last_became_peered":"2023-10-19T13:04:57.778481+0000","last_unstale":"2023-10-19T13:04:59.246718+0000","last_undegraded":"2023-10-19T13:04:59.246718+0000","last_fullsized":"2023-10-19T13:04:59.246718+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_clean_scrub_stamp":"2023-10-19T13:04:56.754267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:01:59.621170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":11,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:05:00.280491+0000","last_change":"2023-10-19T13:04:58.247632+0000","last_active":"2023-10-19T13:05:00.280491+0000","last_peered":"2023-10-19T13:05:00.280491+0000","last_clean":"2023-10-19T13:05:00.280491+0000","last_became_active":"2023-10-19T13:04:58.247338+0000","last_became_peered":"2023-10-19T13:04:58.247338+0000","last_unstale":"2023-10-19T13:05:00.280491+0000","last_undegraded":"2023-10-19T13:05:00.280491+0000","last_fullsized":"2023-10-19T13:05:00.280491+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:41:50.218888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.241079+0000","last_change":"2023-10-19T13:04:57.783219+0000","last_active":"2023-10-19T13:04:58.241079+0000","last_peered":"2023-10-19T13:04:58.241079+0000","last_clean":"2023-10-19T13:04:58.241079+0000","last_became_active":"2023-10-19T13:04:57.782922+0000","last_became_peered":"2023-10-19T13:04:57.782922+0000","last_unstale":"2023-10-19T13:04:58.241079+0000","last_undegraded":"2023-10-19T13:04:58.241079+0000","last_fullsized":"2023-10-19T13:04:58.241079+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_clean_scrub_stamp":"2023-10-19T13:04:56.754267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:19:04.992711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":9,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.260030+0000","last_change":"2023-10-19T13:04:58.250837+0000","last_active":"2023-10-19T13:04:58.260030+0000","last_peered":"2023-10-19T13:04:58.260030+0000","last_clean":"2023-10-19T13:04:58.260030+0000","last_became_active":"2023-10-19T13:04:58.250559+0000","last_became_peered":"2023-10-19T13:04:58.250559+0000","last_unstale":"2023-10-19T13:04:58.260030+0000","last_undegraded":"2023-10-19T13:04:58.260030+0000","last_fullsized":"2023-10-19T13:04:58.260030+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:21:13.640969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":12,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.245534+0000","last_change":"2023-10-19T13:04:57.771866+0000","last_active":"2023-10-19T13:04:59.245534+0000","last_peered":"2023-10-19T13:04:59.245534+0000","last_clean":"2023-10-19T13:04:59.245534+0000","last_became_active":"2023-10-19T13:04:57.771607+0000","last_became_peered":"2023-10-19T13:04:57.771607+0000","last_unstale":"2023-10-19T13:04:59.245534+0000","last_undegraded":"2023-10-19T13:04:59.245534+0000","last_fullsized":"2023-10-19T13:04:59.245534+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_clean_scrub_stamp":"2023-10-19T13:04:56.754267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:34:11.325062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":9,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.259958+0000","last_change":"2023-10-19T13:04:58.250627+0000","last_active":"2023-10-19T13:04:58.259958+0000","last_peered":"2023-10-19T13:04:58.259958+0000","last_clean":"2023-10-19T13:04:58.259958+0000","last_became_active":"2023-10-19T13:04:58.250314+0000","last_became_peered":"2023-10-19T13:04:58.250314+0000","last_unstale":"2023-10-19T13:04:58.259958+0000","last_undegraded":"2023-10-19T13:04:58.259958+0000","last_fullsized":"2023-10-19T13:04:58.259958+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:18:09.472240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.242113+0000","last_change":"2023-10-19T13:04:57.785600+0000","last_active":"2023-10-19T13:04:58.242113+0000","last_peered":"2023-10-19T13:04:58.242113+0000","last_clean":"2023-10-19T13:04:58.242113+0000","last_became_active":"2023-10-19T13:04:57.785327+0000","last_became_peered":"2023-10-19T13:04:57.785327+0000","last_unstale":"2023-10-19T13:04:58.242113+0000","last_undegraded":"2023-10-19T13:04:58.242113+0000","last_fullsized":"2023-10-19T13:04:58.242113+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_clean_scrub_stamp":"2023-10-19T13:04:56.754267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:53:19.112926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.6","version":"0'0","reported_seq":9,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.265150+0000","last_change":"2023-10-19T13:04:58.259056+0000","last_active":"2023-10-19T13:04:58.265150+0000","last_peered":"2023-10-19T13:04:58.265150+0000","last_clean":"2023-10-19T13:04:58.265150+0000","last_became_active":"2023-10-19T13:04:58.258843+0000","last_became_peered":"2023-10-19T13:04:58.258843+0000","last_unstale":"2023-10-19T13:04:58.265150+0000","last_undegraded":"2023-10-19T13:04:58.265150+0000","last_fullsized":"2023-10-19T13:04:58.265150+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:05:32.209217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.240583+0000","last_change":"2023-10-19T13:04:57.783271+0000","last_active":"2023-10-19T13:04:58.240583+0000","last_peered":"2023-10-19T13:04:58.240583+0000","last_clean":"2023-10-19T13:04:58.240583+0000","last_became_active":"2023-10-19T13:04:57.782989+0000","last_became_peered":"2023-10-19T13:04:57.782989+0000","last_unstale":"2023-10-19T13:04:58.240583+0000","last_undegraded":"2023-10-19T13:04:58.240583+0000","last_fullsized":"2023-10-19T13:04:58.240583+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_clean_scrub_stamp":"2023-10-19T13:04:56.754267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:01:59.621170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.7","version":"0'0","reported_seq":9,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.256230+0000","last_change":"2023-10-19T13:04:58.253636+0000","last_active":"2023-10-19T13:04:58.256230+0000","last_peered":"2023-10-19T13:04:58.256230+0000","last_clean":"2023-10-19T13:04:58.256230+0000","last_became_active":"2023-10-19T13:04:58.251824+0000","last_became_peered":"2023-10-19T13:04:58.251824+0000","last_unstale":"2023-10-19T13:04:58.256230+0000","last_undegraded":"2023-10-19T13:04:58.256230+0000","last_fullsized":"2023-10-19T13:04:58.256230+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:04:35.399006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":12,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-19T13:04:59.684822+0000","last_change":"2023-10-19T13:04:57.788217+0000","last_active":"2023-10-19T13:04:59.684822+0000","last_peered":"2023-10-19T13:04:59.684822+0000","last_clean":"2023-10-19T13:04:59.684822+0000","last_became_active":"2023-10-19T13:04:57.787983+0000","last_became_peered":"2023-10-19T13:04:57.787983+0000","last_unstale":"2023-10-19T13:04:59.684822+0000","last_undegraded":"2023-10-19T13:04:59.684822+0000","last_fullsized":"2023-10-19T13:04:59.684822+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_clean_scrub_stamp":"2023-10-19T13:04:56.754267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:23:14.121433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.15","version":"0'0","reported_seq":9,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.260160+0000","last_change":"2023-10-19T13:04:58.256033+0000","last_active":"2023-10-19T13:04:58.260160+0000","last_peered":"2023-10-19T13:04:58.260160+0000","last_clean":"2023-10-19T13:04:58.260160+0000","last_became_active":"2023-10-19T13:04:58.255772+0000","last_became_peered":"2023-10-19T13:04:58.255772+0000","last_unstale":"2023-10-19T13:04:58.260160+0000","last_undegraded":"2023-10-19T13:04:58.260160+0000","last_fullsized":"2023-10-19T13:04:58.260160+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:57.759827+0000","last_clean_scrub_stamp":"2023-10-19T13:04:57.759827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:05:39.299198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":11,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-19T13:04:58.241353+0000","last_change":"2023-10-19T13:04:57.788792+0000","last_active":"2023-10-19T13:04:58.241353+0000","last_peered":"2023-10-19T13:04:58.241353+0000","last_clean":"2023-10-19T13:04:58.241353+0000","last_became_active":"2023-10-19T13:04:57.788511+0000","last_became_peered":"2023-10-19T13:04:57.788511+0000","last_unstale":"2023-10-19T13:04:58.241353+0000","last_undegraded":"2023-10-19T13:04:58.241353+0000","last_fullsized":"2023-10-19T13:04:58.241353+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T13:04:56.754267+0000","last_clean_scrub_stamp":"2023-10-19T13:04:56.754267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:20:48.667529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"146'3087","reported_seq":3639,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-19T13:04:57.780323+0000","last_change":"2023-10-19T12:44:08.636613+0000","last_active":"2023-10-19T13:04:57.780323+0000","last_peered":"2023-10-19T13:04:57.780323+0000","last_clean":"2023-10-19T13:04:57.780323+0000","last_became_active":"2023-10-19T12:44:08.635697+0000","last_became_peered":"2023-10-19T12:44:08.635697+0000","last_unstale":"2023-10-19T13:04:57.780323+0000","last_undegraded":"2023-10-19T13:04:57.780323+0000","last_fullsized":"2023-10-19T13:04:57.780323+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_clean_scrub_stamp":"2023-10-19T12:44:07.399738+0000","objects_scrubbed":0,"log_size":3087,"log_dups_size":0,"ondisk_log_size":3087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:15:23.685485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1346,"num_read_kb":2770,"num_write":409,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":20,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":19,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1346,"num_read_kb":2770,"num_write":409,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3087,"ondisk_log_size":3087,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575100,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55884,"kb_used_data":2496,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315956,"statfs":{"total":96636764160,"available":96579538944,"internally_reserved":0,"allocated":2555904,"data_stored":1576190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54655435},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 13:04:51 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51800000000000002,"15min":0.55300000000000005},"min":{"1min":0.38900000000000001,"5min":0.224,"15min":0.224},"max":{"1min":0.77300000000000002,"5min":0.80100000000000005,"15min":0.89400000000000002},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.55100000000000005,"15min":0.58199999999999996},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.29599999999999999},"max":{"1min":0.73799999999999999,"5min":0.75600000000000001,"15min":0.93700000000000006},"last":0.63900000000000001}]},{"osd":1,"last update":"Thu Oct 19 13:04:51 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.54200000000000004,"15min":0.57099999999999995},"min":{"1min":0.42299999999999999,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.73999999999999999,"5min":0.82399999999999995,"15min":1.3129999999999999},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.54300000000000004,"15min":0.57599999999999996},"min":{"1min":0.31900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.79700000000000004,"5min":0.85499999999999998,"15min":1.284},"last":0.60199999999999998}]},{"osd":2,"last update":"Thu Oct 19 13:04:51 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.60099999999999998,"15min":0.622},"min":{"1min":0.37,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.80100000000000005,"5min":0.89200000000000002,"15min":0.98599999999999999},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.60199999999999998,"15min":0.627},"min":{"1min":0.42699999999999999,"5min":0.40300000000000002,"15min":0.28399999999999997},"max":{"1min":0.755,"5min":0.83699999999999997,"15min":1.2589999999999999},"last":0.69199999999999995}]},{"osd":3,"last update":"Thu Oct 19 13:04:51 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.56499999999999995,"15min":0.59599999999999997},"min":{"1min":0.35799999999999998,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.71999999999999997,"5min":0.87,"15min":0.96899999999999997},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.62,"5min":0.63500000000000001,"15min":0.65100000000000002},"min":{"1min":0.435,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.75900000000000001,"5min":0.91600000000000004,"15min":1.329},"last":0.67800000000000005}]},{"osd":4,"last update":"Thu Oct 19 13:04:51 2023","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.46899999999999997,"15min":0.46800000000000003},"min":{"1min":0.28000000000000003,"5min":0.247,"15min":0.247},"max":{"1min":0.65400000000000003,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.502,"5min":0.48199999999999998,"15min":0.46999999999999997},"min":{"1min":0.32000000000000001,"5min":0.216,"15min":0.216},"max":{"1min":0.68000000000000005,"5min":0.73399999999999999,"15min":0.75600000000000001},"last":0.45500000000000002}]},{"osd":5,"last update":"Thu Oct 19 13:04:51 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.5,"15min":0.48499999999999999},"min":{"1min":0.33500000000000002,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.77900000000000003,"5min":0.78800000000000003,"15min":0.80600000000000005},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.53000000000000003,"15min":0.51900000000000002},"min":{"1min":0.35799999999999998,"5min":0.32200000000000001,"15min":0.314},"max":{"1min":0.73199999999999998,"5min":0.80100000000000005,"15min":0.81999999999999995},"last":0.47799999999999998}]},{"osd":6,"last update":"Thu Oct 19 13:04:51 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55500000000000005,"15min":0.54600000000000004},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.308},"max":{"1min":0.78000000000000003,"5min":0.78900000000000003,"15min":0.88900000000000001},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53600000000000003,"15min":0.52800000000000002},"min":{"1min":0.38500000000000001,"5min":0.318,"15min":0.27300000000000002},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.90300000000000002},"last":0.503}]}]},{"osd":6,"up_from":13,"seq":55834575100,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33088,"kb_used_data":1908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338752,"statfs":{"total":96636764160,"available":96602882048,"internally_reserved":0,"allocated":1953792,"data_stored":984792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 13:04:40 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.56000000000000005,"15min":0.58099999999999996},"min":{"1min":0.29099999999999998,"5min":0.23999999999999999,"15min":0.19500000000000001},"max":{"1min":0.76700000000000002,"5min":1.196,"15min":1.196},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54500000000000004,"15min":0.57699999999999996},"min":{"1min":0.41899999999999998,"5min":0.28799999999999998,"15min":0.247},"max":{"1min":0.71899999999999997,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.55900000000000005}]},{"osd":1,"last update":"Thu Oct 19 13:04:40 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.56899999999999995,"15min":0.59199999999999997},"min":{"1min":0.34799999999999998,"5min":0.29099999999999998,"15min":0.27400000000000002},"max":{"1min":0.69499999999999995,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.57599999999999996,"15min":0.59699999999999998},"min":{"1min":0.39800000000000002,"5min":0.307,"15min":0.307},"max":{"1min":0.70399999999999996,"5min":0.80600000000000005,"15min":0.82099999999999995},"last":0.64300000000000002}]},{"osd":2,"last update":"Thu Oct 19 13:04:40 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.54900000000000004,"15min":0.57899999999999996},"min":{"1min":0.38400000000000001,"5min":0.34599999999999997,"15min":0.307},"max":{"1min":0.73899999999999999,"5min":0.79400000000000004,"15min":0.85699999999999998},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.58599999999999997,"15min":0.60499999999999998},"min":{"1min":0.436,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.77300000000000002,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.41599999999999998}]},{"osd":3,"last update":"Thu Oct 19 13:04:40 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.59999999999999998,"15min":0.622},"min":{"1min":0.47099999999999997,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.749,"5min":0.81299999999999994,"15min":0.871},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.625,"5min":0.60599999999999998,"15min":0.621},"min":{"1min":0.40799999999999997,"5min":0.39200000000000002,"15min":0.36799999999999999},"max":{"1min":0.78200000000000003,"5min":0.82799999999999996,"15min":0.92800000000000005},"last":0.59499999999999997}]},{"osd":4,"last update":"Thu Oct 19 13:04:40 2023","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.47399999999999998,"15min":0.46700000000000003},"min":{"1min":0.32800000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.60899999999999999,"5min":1.353,"15min":1.353},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.48099999999999998,"15min":0.47799999999999998},"min":{"1min":0.308,"5min":0.254,"15min":0.254},"max":{"1min":0.65600000000000003,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.439}]},{"osd":5,"last update":"Thu Oct 19 13:04:40 2023","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.46899999999999997,"15min":0.47299999999999998},"min":{"1min":0.34000000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.60599999999999998,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.502,"5min":0.53100000000000003,"15min":0.51600000000000001},"min":{"1min":0.38900000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.72899999999999998,"5min":0.77400000000000002,"15min":0.85699999999999998},"last":0.50600000000000001}]},{"osd":7,"last update":"Thu Oct 19 13:04:40 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.53500000000000003,"15min":0.52300000000000002},"min":{"1min":0.312,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.78900000000000003,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.54100000000000004,"15min":0.53400000000000003},"min":{"1min":0.41699999999999998,"5min":0.27900000000000003,"15min":0.26000000000000001},"max":{"1min":0.70199999999999996,"5min":1.127,"15min":1.127},"last":0.66400000000000003}]}]},{"osd":5,"up_from":13,"seq":55834575100,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51148,"kb_used_data":1916,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320692,"statfs":{"total":96636764160,"available":96584388608,"internally_reserved":0,"allocated":1961984,"data_stored":990790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 13:04:36 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55500000000000005,"15min":0.58199999999999996},"min":{"1min":0.36799999999999999,"5min":0.223,"15min":0.223},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.58499999999999996,"15min":0.59399999999999997},"min":{"1min":0.379,"5min":0.33900000000000002,"15min":0.27800000000000002},"max":{"1min":0.748,"5min":0.82999999999999996,"15min":0.84299999999999997},"last":0.63600000000000001}]},{"osd":1,"last update":"Thu Oct 19 13:04:36 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.57599999999999996,"15min":0.58899999999999997},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.32800000000000001},"max":{"1min":0.79700000000000004,"5min":0.81399999999999995,"15min":0.91200000000000003},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.60499999999999998,"15min":0.60799999999999998},"min":{"1min":0.48099999999999998,"5min":0.378,"15min":0.34000000000000002},"max":{"1min":0.78700000000000003,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.56499999999999995}]},{"osd":2,"last update":"Thu Oct 19 13:04:36 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.54900000000000004,"15min":0.57399999999999995},"min":{"1min":0.36799999999999999,"5min":0.32700000000000001,"15min":0.307},"max":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.622,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.42799999999999999,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.60999999999999999}]},{"osd":3,"last update":"Thu Oct 19 13:04:36 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.58099999999999996,"15min":0.59699999999999998},"min":{"1min":0.373,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.60399999999999998,"15min":0.625},"min":{"1min":0.437,"5min":0.40899999999999997,"15min":0.32100000000000001},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.58099999999999996}]},{"osd":4,"last update":"Thu Oct 19 13:04:36 2023","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.46700000000000003,"15min":0.46899999999999997},"min":{"1min":0.36899999999999999,"5min":0.307,"15min":0.29099999999999998},"max":{"1min":0.65500000000000003,"5min":0.745,"15min":0.98099999999999998},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.48699999999999999,"15min":0.47399999999999998},"min":{"1min":0.33800000000000002,"5min":0.26100000000000001,"15min":0.251},"max":{"1min":0.68200000000000005,"5min":0.78700000000000003,"15min":0.79400000000000004},"last":0.45600000000000002}]},{"osd":6,"last update":"Thu Oct 19 13:04:36 2023","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.497,"15min":0.47799999999999998},"min":{"1min":0.29899999999999999,"5min":0.27800000000000002,"15min":0.27200000000000002},"max":{"1min":0.75800000000000001,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52100000000000002,"15min":0.50900000000000001},"min":{"1min":0.41699999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.68600000000000005,"5min":0.80800000000000005,"15min":0.82499999999999996},"last":0.623}]},{"osd":7,"last update":"Thu Oct 19 13:04:36 2023","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.502,"15min":0.49299999999999999},"min":{"1min":0.33200000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.70099999999999996,"5min":0.78600000000000003,"15min":1.0589999999999999},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.55700000000000005,"15min":0.54100000000000004},"min":{"1min":0.318,"5min":0.318,"15min":0.311},"max":{"1min":0.74399999999999999,"5min":0.78700000000000003,"15min":0.94899999999999995},"last":0.54400000000000004}]}]},{"osd":4,"up_from":13,"seq":55834575100,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33100,"kb_used_data":1912,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338740,"statfs":{"total":96636764160,"available":96602869760,"internally_reserved":0,"allocated":1957888,"data_stored":990768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 13:04:49 2023","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.51300000000000001,"15min":0.54800000000000004},"min":{"1min":0.30399999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.623,"5min":0.78800000000000003,"15min":0.86699999999999999},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.5,"5min":0.53100000000000003,"15min":0.54900000000000004},"min":{"1min":0.33700000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.68700000000000006,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.443}]},{"osd":1,"last update":"Thu Oct 19 13:04:49 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.53800000000000003,"15min":0.58599999999999997},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.70199999999999996,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.309},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.57299999999999995,"15min":0.60399999999999998},"min":{"1min":0.374,"5min":0.374,"15min":0.32600000000000001},"max":{"1min":0.67100000000000004,"5min":0.79800000000000004,"15min":0.86899999999999999},"last":0.41899999999999998}]},{"osd":2,"last update":"Thu Oct 19 13:04:49 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.56399999999999995,"15min":0.59899999999999998},"min":{"1min":0.30399999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.66000000000000003,"5min":0.80100000000000005,"15min":0.86599999999999999},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.57599999999999996,"15min":0.60399999999999998},"min":{"1min":0.34999999999999998,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.64700000000000002,"5min":0.83799999999999997,"15min":0.85999999999999999},"last":0.46999999999999997}]},{"osd":3,"last update":"Thu Oct 19 13:04:49 2023","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.53800000000000003,"15min":0.56799999999999995},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.66800000000000004,"5min":0.75,"15min":0.79700000000000004},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.59599999999999997,"15min":0.628},"min":{"1min":0.42399999999999999,"5min":0.42099999999999999,"15min":0.41299999999999998},"max":{"1min":0.73199999999999998,"5min":0.82599999999999996,"15min":0.83899999999999997},"last":0.39700000000000002}]},{"osd":5,"last update":"Thu Oct 19 13:04:49 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.44900000000000001,"15min":0.44700000000000001},"min":{"1min":0.32200000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.63900000000000001,"5min":0.77100000000000002,"15min":0.79600000000000004},"last":0.379},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.46400000000000002,"15min":0.46000000000000002},"min":{"1min":0.33700000000000002,"5min":0.30099999999999999,"15min":0.23499999999999999},"max":{"1min":0.625,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.54000000000000004}]},{"osd":6,"last update":"Thu Oct 19 13:04:49 2023","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.46500000000000002,"15min":0.45800000000000002},"min":{"1min":0.34300000000000003,"5min":0.28499999999999998,"15min":0.26400000000000001},"max":{"1min":0.65300000000000002,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.502,"15min":0.49099999999999999},"min":{"1min":0.36599999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.67300000000000004,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.505}]},{"osd":7,"last update":"Thu Oct 19 13:04:49 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.501},"min":{"1min":0.34999999999999998,"5min":0.31900000000000001,"15min":0.27500000000000002},"max":{"1min":0.65300000000000002,"5min":0.78700000000000003,"15min":0.79100000000000004},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56200000000000006,"15min":0.54700000000000004},"min":{"1min":0.39200000000000002,"5min":0.33100000000000002,"15min":0.30199999999999999},"max":{"1min":0.68600000000000005,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.57999999999999996}]}]},{"osd":3,"up_from":13,"seq":55834575099,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41428,"kb_used_data":1928,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330412,"statfs":{"total":96636764160,"available":96594341888,"internally_reserved":0,"allocated":1974272,"data_stored":992450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":40434120},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 13:04:33 2023","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.39600000000000002,"15min":0.40300000000000002},"min":{"1min":0.26400000000000001,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":1.1279999999999999},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.38400000000000001,"15min":0.41099999999999998},"min":{"1min":0.25900000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.56399999999999995,"5min":0.59899999999999998,"15min":0.75},"last":0.46200000000000002}]},{"osd":1,"last update":"Thu Oct 19 13:04:33 2023","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.40200000000000002,"15min":0.40799999999999997},"min":{"1min":0.249,"5min":0.22,"15min":0.22},"max":{"1min":0.59499999999999997,"5min":0.628,"15min":0.72099999999999997},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.42799999999999999,"15min":0.45800000000000002},"min":{"1min":0.28199999999999997,"5min":0.221,"15min":0.221},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":2.9399999999999999},"last":0.33900000000000002}]},{"osd":2,"last update":"Thu Oct 19 13:04:33 2023","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.439,"15min":0.45900000000000002},"min":{"1min":0.30599999999999999,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.65700000000000003,"5min":0.70799999999999996,"15min":0.97699999999999998},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.499,"5min":0.48099999999999998,"15min":0.48799999999999999},"min":{"1min":0.34100000000000003,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.73099999999999998,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.41799999999999998}]},{"osd":4,"last update":"Thu Oct 19 13:04:33 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.63500000000000001,"15min":0.66400000000000003},"min":{"1min":0.46999999999999997,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.88400000000000001,"5min":0.96299999999999997,"15min":1.0069999999999999},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.66000000000000003,"15min":0.68999999999999995},"min":{"1min":0.44800000000000001,"5min":0.311,"15min":0.311},"max":{"1min":0.90400000000000003,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.752}]},{"osd":5,"last update":"Thu Oct 19 13:04:33 2023","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.64400000000000002,"15min":0.66500000000000004},"min":{"1min":0.44900000000000001,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.96299999999999997,"5min":0.96699999999999997,"15min":2.9990000000000001},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.623,"15min":0.64900000000000002},"min":{"1min":0.435,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":1.774},"last":0.69999999999999996}]},{"osd":6,"last update":"Thu Oct 19 13:04:33 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.63800000000000001,"15min":0.65300000000000002},"min":{"1min":0.498,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.97599999999999998,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.66600000000000004,"15min":0.68000000000000005},"min":{"1min":0.41699999999999998,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.94899999999999995,"5min":1.0429999999999999,"15min":1.167},"last":0.53600000000000003}]},{"osd":7,"last update":"Thu Oct 19 13:04:33 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.64000000000000001,"15min":0.66200000000000003},"min":{"1min":0.433,"5min":0.42099999999999999,"15min":0.39400000000000002},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":1.042},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.66600000000000004,"15min":0.68100000000000005},"min":{"1min":0.46999999999999997,"5min":0.441,"15min":0.432},"max":{"1min":1.0109999999999999,"5min":1.1160000000000001,"15min":2.8149999999999999},"last":0.66700000000000004}]}]},{"osd":2,"up_from":13,"seq":55834575101,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59472,"kb_used_data":1920,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312368,"statfs":{"total":96636764160,"available":96575864832,"internally_reserved":0,"allocated":1966080,"data_stored":991384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 13:04:49 2023","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.376,"15min":0.377},"min":{"1min":0.23200000000000001,"5min":0.20699999999999999,"15min":0.19700000000000001},"max":{"1min":0.53000000000000003,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.20499999999999999},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.39600000000000002,"15min":0.40600000000000003},"min":{"1min":0.25800000000000001,"5min":0.222,"15min":0.20999999999999999},"max":{"1min":0.497,"5min":0.68000000000000005,"15min":0.74099999999999999},"last":0.22600000000000001}]},{"osd":1,"last update":"Thu Oct 19 13:04:49 2023","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.38300000000000001,"15min":0.39100000000000001},"min":{"1min":0.27200000000000002,"5min":0.249,"15min":0.24299999999999999},"max":{"1min":0.59099999999999997,"5min":0.64100000000000001,"15min":0.71099999999999997},"last":0.248},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.41199999999999998,"15min":0.42199999999999999},"min":{"1min":0.29299999999999998,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":0.64800000000000002,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.27400000000000002}]},{"osd":3,"last update":"Thu Oct 19 13:04:49 2023","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.434,"15min":0.45000000000000001},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.22800000000000001},"max":{"1min":0.53000000000000003,"5min":0.69299999999999995,"15min":0.86599999999999999},"last":0.318},{"interface":"front","average":{"1min":0.439,"5min":0.46600000000000003,"15min":0.47899999999999998},"min":{"1min":0.35999999999999999,"5min":0.32200000000000001,"15min":0.29099999999999998},"max":{"1min":0.56200000000000006,"5min":0.72999999999999998,"15min":0.82999999999999996},"last":0.30199999999999999}]},{"osd":4,"last update":"Thu Oct 19 13:04:49 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.60199999999999998,"15min":0.63300000000000001},"min":{"1min":0.42999999999999999,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.66900000000000004,"5min":0.874,"15min":0.98999999999999999},"last":0.496},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.61499999999999999,"15min":0.64500000000000002},"min":{"1min":0.45800000000000002,"5min":0.41199999999999998,"15min":0.35099999999999998},"max":{"1min":0.77000000000000002,"5min":0.97699999999999998,"15min":1.046},"last":0.53400000000000003}]},{"osd":5,"last update":"Thu Oct 19 13:04:49 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.621,"15min":0.65100000000000002},"min":{"1min":0.44400000000000001,"5min":0.40799999999999997,"15min":0.31900000000000001},"max":{"1min":0.73899999999999999,"5min":0.96199999999999997,"15min":0.99399999999999999},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.62,"15min":0.65000000000000002},"min":{"1min":0.39600000000000002,"5min":0.39100000000000001,"15min":0.375},"max":{"1min":0.748,"5min":0.94199999999999995,"15min":0.95799999999999996},"last":0.46300000000000002}]},{"osd":6,"last update":"Thu Oct 19 13:04:49 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.63700000000000001,"15min":0.65200000000000002},"min":{"1min":0.45900000000000002,"5min":0.40400000000000003,"15min":0.40300000000000002},"max":{"1min":0.77400000000000002,"5min":0.93000000000000005,"15min":1.008},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.64100000000000001,"15min":0.66800000000000004},"min":{"1min":0.46700000000000003,"5min":0.44600000000000001,"15min":0.371},"max":{"1min":0.76300000000000001,"5min":0.90400000000000003,"15min":0.995},"last":0.56699999999999995}]},{"osd":7,"last update":"Thu Oct 19 13:04:49 2023","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.64700000000000002,"15min":0.67200000000000004},"min":{"1min":0.49099999999999999,"5min":0.39800000000000002,"15min":0.36399999999999999},"max":{"1min":1.163,"5min":1.163,"15min":1.163},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.65200000000000002,"15min":0.67300000000000004},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.38200000000000001},"max":{"1min":0.79700000000000004,"5min":0.95199999999999996,"15min":1.0329999999999999},"last":0.58999999999999997}]}]},{"osd":1,"up_from":13,"seq":55834575100,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37244,"kb_used_data":1904,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334596,"statfs":{"total":96636764160,"available":96598626304,"internally_reserved":0,"allocated":1949696,"data_stored":984190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 13:04:45 2023","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.36099999999999999,"15min":0.38},"min":{"1min":0.23100000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.50600000000000001,"5min":0.59299999999999997,"15min":1.3879999999999999},"last":0.20200000000000001},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.41199999999999998,"15min":0.433},"min":{"1min":0.252,"5min":0.217,"15min":0.217},"max":{"1min":0.63900000000000001,"5min":0.76100000000000001,"15min":2.5800000000000001},"last":0.308}]},{"osd":2,"last update":"Thu Oct 19 13:04:45 2023","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.40799999999999997,"15min":0.42099999999999999},"min":{"1min":0.30099999999999999,"5min":0.24199999999999999,"15min":0.21299999999999999},"max":{"1min":0.61199999999999999,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.437,"15min":0.45500000000000002},"min":{"1min":0.27500000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.56899999999999995,"5min":0.79800000000000004,"15min":2.6230000000000002},"last":0.34499999999999997}]},{"osd":3,"last update":"Thu Oct 19 13:04:45 2023","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.45600000000000002,"15min":0.48499999999999999},"min":{"1min":0.29199999999999998,"5min":0.246,"15min":0.246},"max":{"1min":0.64700000000000002,"5min":1.008,"15min":3.9740000000000002},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.41299999999999998,"15min":0.42399999999999999},"min":{"1min":0.30199999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.67000000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.32700000000000001}]},{"osd":4,"last update":"Thu Oct 19 13:04:45 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.61499999999999999,"15min":0.64200000000000002},"min":{"1min":0.40400000000000003,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.78300000000000003,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.622,"5min":0.60499999999999998,"15min":0.63600000000000001},"min":{"1min":0.41399999999999998,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":1.0860000000000001},"last":0.54500000000000004}]},{"osd":5,"last update":"Thu Oct 19 13:04:45 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.63600000000000001,"15min":0.64900000000000002},"min":{"1min":0.45200000000000001,"5min":0.38700000000000001,"15min":0.38100000000000001},"max":{"1min":0.86599999999999999,"5min":1.0549999999999999,"15min":1.1200000000000001},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.64300000000000002,"15min":0.67000000000000004},"min":{"1min":0.46500000000000002,"5min":0.374,"15min":0.374},"max":{"1min":0.90300000000000002,"5min":1.1020000000000001,"15min":2.7120000000000002},"last":0.56100000000000005}]},{"osd":6,"last update":"Thu Oct 19 13:04:45 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.60699999999999998,"15min":0.64300000000000002},"min":{"1min":0.435,"5min":0.41499999999999998,"15min":0.36799999999999999},"max":{"1min":0.81599999999999995,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64400000000000002,"15min":0.66700000000000004},"min":{"1min":0.503,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.89200000000000002,"5min":1.129,"15min":1.129},"last":0.58899999999999997}]},{"osd":7,"last update":"Thu Oct 19 13:04:45 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.64800000000000002,"15min":0.67300000000000004},"min":{"1min":0.46800000000000003,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.76400000000000001,"5min":1.1439999999999999,"15min":2.786},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.63600000000000001,"15min":0.66600000000000004},"min":{"1min":0.47699999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.88700000000000001,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.53100000000000003}]}]},{"osd":0,"up_from":13,"seq":55834575101,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33668,"kb_used_data":2484,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338172,"statfs":{"total":96636764160,"available":96602288128,"internally_reserved":0,"allocated":2543616,"data_stored":1573925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Oct 19 13:04:48 2023","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.374,"15min":0.39000000000000001},"min":{"1min":0.24299999999999999,"5min":0.20799999999999999,"15min":0.182},"max":{"1min":0.54900000000000004,"5min":0.624,"15min":0.81799999999999995},"last":0.252},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.39100000000000001,"15min":0.42299999999999999},"min":{"1min":0.26000000000000001,"5min":0.22500000000000001,"15min":0.20200000000000001},"max":{"1min":0.628,"5min":0.64800000000000002,"15min":0.69999999999999996},"last":0.38300000000000001}]},{"osd":2,"last update":"Thu Oct 19 13:04:48 2023","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.41799999999999998,"15min":0.44900000000000001},"min":{"1min":0.30599999999999999,"5min":0.27100000000000002,"15min":0.26200000000000001},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.76300000000000001},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.40400000000000003,"15min":0.42999999999999999},"min":{"1min":0.253,"5min":0.246,"15min":0.20699999999999999},"max":{"1min":0.60299999999999998,"5min":0.68500000000000005,"15min":0.72999999999999998},"last":0.36299999999999999}]},{"osd":3,"last update":"Thu Oct 19 13:04:48 2023","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.376,"15min":0.40400000000000003},"min":{"1min":0.27500000000000002,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.50900000000000001,"5min":0.60399999999999998,"15min":1.2829999999999999},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.44,"5min":0.441,"15min":0.48399999999999999},"min":{"1min":0.33600000000000002,"5min":0.29299999999999998,"15min":0.29099999999999998},"max":{"1min":0.58199999999999996,"5min":0.71199999999999997,"15min":1.4990000000000001},"last":0.41199999999999998}]},{"osd":4,"last update":"Thu Oct 19 13:04:48 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.58999999999999997,"15min":0.625},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.34999999999999998},"max":{"1min":1.175,"5min":1.175,"15min":1.361},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.61199999999999999,"15min":0.64700000000000002},"min":{"1min":0.38600000000000001,"5min":0.36399999999999999,"15min":0.35999999999999999},"max":{"1min":0.77500000000000002,"5min":0.93799999999999994,"15min":0.98999999999999999},"last":0.63500000000000001}]},{"osd":5,"last update":"Thu Oct 19 13:04:48 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.64400000000000002,"15min":0.67600000000000005},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.378},"max":{"1min":1.137,"5min":1.137,"15min":1.325},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.63900000000000001,"15min":0.67900000000000005},"min":{"1min":0.41599999999999998,"5min":0.40100000000000002,"15min":0.318},"max":{"1min":0.82399999999999995,"5min":0.94099999999999995,"15min":1.024},"last":0.73799999999999999}]},{"osd":6,"last update":"Thu Oct 19 13:04:48 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.60999999999999999,"15min":0.65400000000000003},"min":{"1min":0.46700000000000003,"5min":0.36899999999999999,"15min":0.35899999999999999},"max":{"1min":0.79500000000000004,"5min":0.879,"15min":0.98099999999999998},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.61199999999999999,"15min":0.65600000000000003},"min":{"1min":0.41799999999999998,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.78000000000000003,"5min":0.88700000000000001,"15min":0.995},"last":0.70499999999999996}]},{"osd":7,"last update":"Thu Oct 19 13:04:48 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.61499999999999999,"15min":0.66700000000000004},"min":{"1min":0.42799999999999999,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.872,"5min":0.89100000000000001,"15min":1.0049999999999999},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.626,"15min":0.65600000000000003},"min":{"1min":0.41799999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.204,"5min":1.204,"15min":1.389},"last":0.60899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-19T13:05:02.150 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-19T13:05:02.150 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-19T13:05:02.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.297+0000 7ff7d6ce0640 1 -- 172.21.15.57:0/1598361306 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff7d006d2d0 msgr2=0x7ff7d006d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:05:02.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.297+0000 7ff7d6ce0640 1 --2- 172.21.15.57:0/1598361306 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff7d006d2d0 0x7ff7d006d6b0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7ff7c0009830 tx=0x7ff7c002bd60 comp rx=0 tx=0).stop 2023-10-19T13:05:02.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.297+0000 7ff7d6ce0640 1 -- 172.21.15.57:0/1598361306 shutdown_connections 2023-10-19T13:05:02.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.297+0000 7ff7d6ce0640 1 --2- 172.21.15.57:0/1598361306 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff7d00ad370 0x7ff7d00ad800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:02.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.297+0000 7ff7d6ce0640 1 --2- 172.21.15.57:0/1598361306 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff7d006d2d0 0x7ff7d006d6b0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:02.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.297+0000 7ff7d6ce0640 1 -- 172.21.15.57:0/1598361306 >> 172.21.15.57:0/1598361306 conn(0x7ff7d00b0670 msgr2=0x7ff7d00b0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:05:02.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.297+0000 7ff7d6ce0640 1 -- 172.21.15.57:0/1598361306 shutdown_connections 2023-10-19T13:05:02.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.297+0000 7ff7d6ce0640 1 -- 172.21.15.57:0/1598361306 wait complete. 2023-10-19T13:05:02.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.298+0000 7ff7d6ce0640 1 Processor -- start 2023-10-19T13:05:02.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.299+0000 7ff7d6ce0640 1 -- start start 2023-10-19T13:05:02.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.299+0000 7ff7d6ce0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff7d006d2d0 0x7ff7d00c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:05:02.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.299+0000 7ff7d6ce0640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff7d006f0a0 0x7ff7d0150120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:05:02.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.299+0000 7ff7d6ce0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff7d00ad370 0x7ff7d00c4500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:05:02.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.299+0000 7ff7d6ce0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d00cbc70 con 0x7ff7d00ad370 2023-10-19T13:05:02.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.299+0000 7ff7d6ce0640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d00ca930 con 0x7ff7d006d2d0 2023-10-19T13:05:02.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.299+0000 7ff7d6ce0640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d00cabd0 con 0x7ff7d006f0a0 2023-10-19T13:05:02.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.300+0000 7ff7d527d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff7d00ad370 0x7ff7d00c4500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:05:02.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.300+0000 7ff7d527d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff7d00ad370 0x7ff7d00c4500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42848/0 (socket says 172.21.15.57:42848) 2023-10-19T13:05:02.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.300+0000 7ff7d527d640 1 -- 172.21.15.57:0/2250886871 learned_addr learned my addr 172.21.15.57:0/2250886871 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:05:02.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.300+0000 7ff7d4a7c640 1 --2- 172.21.15.57:0/2250886871 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff7d006d2d0 0x7ff7d00c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:05:02.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.300+0000 7ff7cffff640 1 --2- 172.21.15.57:0/2250886871 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff7d006f0a0 0x7ff7d0150120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:05:02.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.300+0000 7ff7d527d640 1 -- 172.21.15.57:0/2250886871 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff7d006f0a0 msgr2=0x7ff7d0150120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:05:02.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.300+0000 7ff7d527d640 1 --2- 172.21.15.57:0/2250886871 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff7d006f0a0 0x7ff7d0150120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:02.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.300+0000 7ff7d527d640 1 -- 172.21.15.57:0/2250886871 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff7d006d2d0 msgr2=0x7ff7d00c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:05:02.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.300+0000 7ff7d527d640 1 --2- 172.21.15.57:0/2250886871 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff7d006d2d0 0x7ff7d00c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:02.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.300+0000 7ff7d527d640 1 -- 172.21.15.57:0/2250886871 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7c8009900 con 0x7ff7d00ad370 2023-10-19T13:05:02.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.300+0000 7ff7d527d640 1 --2- 172.21.15.57:0/2250886871 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff7d00ad370 0x7ff7d00c4500 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7ff7c8009fd0 tx=0x7ff7c800c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:05:02.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.300+0000 7ff7cd7fa640 1 -- 172.21.15.57:0/2250886871 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7c8003970 con 0x7ff7d00ad370 2023-10-19T13:05:02.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.300+0000 7ff7cd7fa640 1 -- 172.21.15.57:0/2250886871 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7ff7c8003be0 con 0x7ff7d00ad370 2023-10-19T13:05:02.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.300+0000 7ff7d6ce0640 1 -- 172.21.15.57:0/2250886871 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7c0009490 con 0x7ff7d00ad370 2023-10-19T13:05:02.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.300+0000 7ff7cd7fa640 1 -- 172.21.15.57:0/2250886871 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7c8004020 con 0x7ff7d00ad370 2023-10-19T13:05:02.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.300+0000 7ff7d6ce0640 1 -- 172.21.15.57:0/2250886871 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7d00cb470 con 0x7ff7d00ad370 2023-10-19T13:05:02.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.301+0000 7ff7d6ce0640 1 -- 172.21.15.57:0/2250886871 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7d00ce1d0 con 0x7ff7d00ad370 2023-10-19T13:05:02.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.302+0000 7ff7cd7fa640 1 -- 172.21.15.57:0/2250886871 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 79) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff7c800a040 con 0x7ff7d00ad370 2023-10-19T13:05:02.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.302+0000 7ff7cd7fa640 1 --2- 172.21.15.57:0/2250886871 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7ff7bc06af60 0x7ff7bc06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:05:02.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.302+0000 7ff7cd7fa640 1 -- 172.21.15.57:0/2250886871 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 6057+0+0 (secure 0 0 0) 0x7ff7c808a060 con 0x7ff7d00ad370 2023-10-19T13:05:02.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.303+0000 7ff7d4a7c640 1 --2- 172.21.15.57:0/2250886871 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7ff7bc06af60 0x7ff7bc06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:05:02.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.303+0000 7ff7d4a7c640 1 --2- 172.21.15.57:0/2250886871 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7ff7bc06af60 0x7ff7bc06d420 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff7c0036000 tx=0x7ff7c0004730 comp rx=0 tx=0).ready entity=mgr.8419 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:05:02.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.307+0000 7ff7cd7fa640 1 -- 172.21.15.57:0/2250886871 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff7c8054e40 con 0x7ff7d00ad370 2023-10-19T13:05:02.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.549+0000 7ff7d6ce0640 1 -- 172.21.15.57:0/2250886871 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7ff7d00cafc0 con 0x7ff7d00ad370 2023-10-19T13:05:02.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.555+0000 7ff7cd7fa640 1 -- 172.21.15.57:0/2250886871 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v30) v1 ==== 190+0+147 (secure 0 0 0) 0x7ff7c8058460 con 0x7ff7d00ad370 2023-10-19T13:05:02.558 INFO:teuthology.orchestra.run.smithi057.stdout:[client.0] 2023-10-19T13:05:02.558 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQBkJDFlg9yuExAATxiWDfS+p07G1F0pORbJEg== 2023-10-19T13:05:02.558 INFO:teuthology.orchestra.run.smithi057.stdout: caps mds = "allow" 2023-10-19T13:05:02.559 INFO:teuthology.orchestra.run.smithi057.stdout: caps mon = "allow r" 2023-10-19T13:05:02.559 INFO:teuthology.orchestra.run.smithi057.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-19T13:05:02.560 INFO:teuthology.orchestra.run.smithi057.stderr:updated caps for client.0 2023-10-19T13:05:02.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.557+0000 7ff7d6ce0640 1 -- 172.21.15.57:0/2250886871 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7ff7bc06af60 msgr2=0x7ff7bc06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:05:02.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.557+0000 7ff7d6ce0640 1 --2- 172.21.15.57:0/2250886871 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7ff7bc06af60 0x7ff7bc06d420 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff7c0036000 tx=0x7ff7c0004730 comp rx=0 tx=0).stop 2023-10-19T13:05:02.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.557+0000 7ff7d6ce0640 1 -- 172.21.15.57:0/2250886871 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff7d00ad370 msgr2=0x7ff7d00c4500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:05:02.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.557+0000 7ff7d6ce0640 1 --2- 172.21.15.57:0/2250886871 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff7d00ad370 0x7ff7d00c4500 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7ff7c8009fd0 tx=0x7ff7c800c740 comp rx=0 tx=0).stop 2023-10-19T13:05:02.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.557+0000 7ff7d6ce0640 1 -- 172.21.15.57:0/2250886871 shutdown_connections 2023-10-19T13:05:02.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.557+0000 7ff7d6ce0640 1 --2- 172.21.15.57:0/2250886871 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7ff7bc06af60 0x7ff7bc06d420 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:02.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.557+0000 7ff7d6ce0640 1 --2- 172.21.15.57:0/2250886871 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff7d00ad370 0x7ff7d00c4500 unknown :-1 s=CLOSED pgs=1765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:02.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.557+0000 7ff7d6ce0640 1 --2- 172.21.15.57:0/2250886871 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff7d006f0a0 0x7ff7d0150120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:02.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.557+0000 7ff7d6ce0640 1 --2- 172.21.15.57:0/2250886871 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff7d006d2d0 0x7ff7d00c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:02.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.557+0000 7ff7d6ce0640 1 -- 172.21.15.57:0/2250886871 >> 172.21.15.57:0/2250886871 conn(0x7ff7d00b0670 msgr2=0x7ff7d00ac8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:05:02.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.557+0000 7ff7d6ce0640 1 -- 172.21.15.57:0/2250886871 shutdown_connections 2023-10-19T13:05:02.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.557+0000 7ff7d6ce0640 1 -- 172.21.15.57:0/2250886871 wait complete. 2023-10-19T13:05:02.570 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T13:05:02.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.678+0000 7fa0f8bc5640 1 -- 172.21.15.57:0/3891855339 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0e0004d00 msgr2=0x7fa0e0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:05:02.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.678+0000 7fa0f8bc5640 1 --2- 172.21.15.57:0/3891855339 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0e0004d00 0x7fa0e0005180 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7fa0ec001680 tx=0x7fa0e00057f0 comp rx=0 tx=0).stop 2023-10-19T13:05:02.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.678+0000 7fa0f2d9d640 1 -- 172.21.15.57:0/3891855339 reap_dead start 2023-10-19T13:05:02.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.678+0000 7fa0f8bc5640 1 -- 172.21.15.57:0/3891855339 shutdown_connections 2023-10-19T13:05:02.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.678+0000 7fa0f8bc5640 1 -- 172.21.15.57:0/3891855339 >> 172.21.15.57:0/3891855339 conn(0x7fa0f40d0d60 msgr2=0x7fa0f40d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:05:02.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.678+0000 7fa0f8bc5640 1 -- 172.21.15.57:0/3891855339 shutdown_connections 2023-10-19T13:05:02.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.678+0000 7fa0f8bc5640 1 -- 172.21.15.57:0/3891855339 wait complete. 2023-10-19T13:05:02.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.679+0000 7fa0f8bc5640 1 Processor -- start 2023-10-19T13:05:02.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.679+0000 7fa0f8bc5640 1 -- start start 2023-10-19T13:05:02.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.680+0000 7fa0f8bc5640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa0e0004d00 0x7fa0f40cab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:05:02.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.680+0000 7fa0f8bc5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0f40b5400 0x7fa0f4157c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:05:02.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.680+0000 7fa0f8bc5640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa0f40c80d0 0x7fa0f40c8580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:05:02.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.680+0000 7fa0f8bc5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f4158540 con 0x7fa0f40b5400 2023-10-19T13:05:02.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.680+0000 7fa0f8bc5640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f40b5910 con 0x7fa0e0004d00 2023-10-19T13:05:02.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.680+0000 7fa0f8bc5640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f40b5bf0 con 0x7fa0f40c80d0 2023-10-19T13:05:02.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.680+0000 7fa0f1d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0f40b5400 0x7fa0f4157c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:05:02.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.680+0000 7fa0f1d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0f40b5400 0x7fa0f4157c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42864/0 (socket says 172.21.15.57:42864) 2023-10-19T13:05:02.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.680+0000 7fa0f1d9b640 1 -- 172.21.15.57:0/2254636041 learned_addr learned my addr 172.21.15.57:0/2254636041 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:05:02.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.680+0000 7fa0f259c640 1 --2- 172.21.15.57:0/2254636041 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa0e0004d00 0x7fa0f40cab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:05:02.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.680+0000 7fa0f2d9d640 1 --2- 172.21.15.57:0/2254636041 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa0f40c80d0 0x7fa0f40c8580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:05:02.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.680+0000 7fa0f1d9b640 1 -- 172.21.15.57:0/2254636041 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa0f40c80d0 msgr2=0x7fa0f40c8580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:05:02.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.680+0000 7fa0f1d9b640 1 --2- 172.21.15.57:0/2254636041 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa0f40c80d0 0x7fa0f40c8580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:02.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.680+0000 7fa0f1d9b640 1 -- 172.21.15.57:0/2254636041 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa0e0004d00 msgr2=0x7fa0f40cab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:05:02.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.680+0000 7fa0f1d9b640 1 --2- 172.21.15.57:0/2254636041 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa0e0004d00 0x7fa0f40cab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:02.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.680+0000 7fa0f1d9b640 1 -- 172.21.15.57:0/2254636041 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0f4158800 con 0x7fa0f40b5400 2023-10-19T13:05:02.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.681+0000 7fa0f1d9b640 1 --2- 172.21.15.57:0/2254636041 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0f40b5400 0x7fa0f4157c90 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7fa0dc0017a0 tx=0x7fa0dc003a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:05:02.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.681+0000 7fa0eaffd640 1 -- 172.21.15.57:0/2254636041 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0dc00f980 con 0x7fa0f40b5400 2023-10-19T13:05:02.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.681+0000 7fa0eaffd640 1 -- 172.21.15.57:0/2254636041 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fa0dc00fb00 con 0x7fa0f40b5400 2023-10-19T13:05:02.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.681+0000 7fa0f8bc5640 1 -- 172.21.15.57:0/2254636041 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0f4158ae0 con 0x7fa0f40b5400 2023-10-19T13:05:02.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.681+0000 7fa0eaffd640 1 -- 172.21.15.57:0/2254636041 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0dc011430 con 0x7fa0f40b5400 2023-10-19T13:05:02.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.681+0000 7fa0f8bc5640 1 -- 172.21.15.57:0/2254636041 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0f41581d0 con 0x7fa0f40b5400 2023-10-19T13:05:02.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.682+0000 7fa0f8bc5640 1 -- 172.21.15.57:0/2254636041 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0f4158d00 con 0x7fa0f40b5400 2023-10-19T13:05:02.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.682+0000 7fa0eaffd640 1 -- 172.21.15.57:0/2254636041 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 79) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fa0dc011590 con 0x7fa0f40b5400 2023-10-19T13:05:02.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.683+0000 7fa0eaffd640 1 --2- 172.21.15.57:0/2254636041 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fa0d406af90 0x7fa0d406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:05:02.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.683+0000 7fa0eaffd640 1 -- 172.21.15.57:0/2254636041 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fa0dc08a700 con 0x7fa0f40b5400 2023-10-19T13:05:02.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.683+0000 7fa0f259c640 1 --2- 172.21.15.57:0/2254636041 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fa0d406af90 0x7fa0d406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:05:02.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.684+0000 7fa0f259c640 1 --2- 172.21.15.57:0/2254636041 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fa0d406af90 0x7fa0d406d450 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa0e4005fd0 tx=0x7fa0e4009500 comp rx=0 tx=0).ready entity=mgr.8419 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:05:02.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.687+0000 7fa0eaffd640 1 -- 172.21.15.57:0/2254636041 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa0dc055570 con 0x7fa0f40b5400 2023-10-19T13:05:02.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.900+0000 7fa0f8bc5640 1 -- 172.21.15.57:0/2254636041 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa0f40690e0 con 0x7fa0f40b5400 2023-10-19T13:05:02.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.900+0000 7fa0eaffd640 1 -- 172.21.15.57:0/2254636041 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+4918 (secure 0 0 0) 0x7fa0dc015090 con 0x7fa0f40b5400 2023-10-19T13:05:02.904 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:05:02.904 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":63,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7733,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/1818414654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":1818414654},{"type":"v1","addr":"172.21.15.179:6836","nonce":1818414654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":8647,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/490712721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":490712721},{"type":"v1","addr":"172.21.15.57:6835","nonce":490712721}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":8656,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/2005811112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":2005811112},{"type":"v1","addr":"172.21.15.179:6837","nonce":2005811112}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T13:04:58.134572+0000","modified":"2023-10-19T13:05:00.138505+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9618},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9618":{"gid":9618,"name":"c","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.15.57:6833/1460919655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":1460919655},{"type":"v1","addr":"172.21.15.57:6833","nonce":1460919655}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2023-10-19T13:05:02.904 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 63 2023-10-19T13:05:02.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.902+0000 7fa0f8bc5640 1 -- 172.21.15.57:0/2254636041 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fa0d406af90 msgr2=0x7fa0d406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:05:02.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.902+0000 7fa0f8bc5640 1 --2- 172.21.15.57:0/2254636041 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fa0d406af90 0x7fa0d406d450 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa0e4005fd0 tx=0x7fa0e4009500 comp rx=0 tx=0).stop 2023-10-19T13:05:02.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.902+0000 7fa0f8bc5640 1 -- 172.21.15.57:0/2254636041 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0f40b5400 msgr2=0x7fa0f4157c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:05:02.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.902+0000 7fa0f8bc5640 1 --2- 172.21.15.57:0/2254636041 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0f40b5400 0x7fa0f4157c90 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7fa0dc0017a0 tx=0x7fa0dc003a80 comp rx=0 tx=0).stop 2023-10-19T13:05:02.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.902+0000 7fa0f8bc5640 1 -- 172.21.15.57:0/2254636041 shutdown_connections 2023-10-19T13:05:02.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.902+0000 7fa0f8bc5640 1 --2- 172.21.15.57:0/2254636041 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fa0d406af90 0x7fa0d406d450 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:02.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.902+0000 7fa0f8bc5640 1 --2- 172.21.15.57:0/2254636041 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa0f40c80d0 0x7fa0f40c8580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:02.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.902+0000 7fa0f8bc5640 1 --2- 172.21.15.57:0/2254636041 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0f40b5400 0x7fa0f4157c90 unknown :-1 s=CLOSED pgs=1768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:02.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.902+0000 7fa0f8bc5640 1 --2- 172.21.15.57:0/2254636041 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa0e0004d00 0x7fa0f40cab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:02.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.902+0000 7fa0f8bc5640 1 -- 172.21.15.57:0/2254636041 >> 172.21.15.57:0/2254636041 conn(0x7fa0f40d0d60 msgr2=0x7fa0f406ae90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:05:02.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.902+0000 7fa0f8bc5640 1 -- 172.21.15.57:0/2254636041 shutdown_connections 2023-10-19T13:05:02.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:02.902+0000 7fa0f8bc5640 1 -- 172.21.15.57:0/2254636041 wait complete. 2023-10-19T13:05:03.318 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 63, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-19T13:04:58.134572+0000', 'modified': '2023-10-19T13:05:00.138505+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9618}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9618': {'gid': 9618, 'name': 'c', 'rank': 0, 'incarnation': 61, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.57:6833/1460919655', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 1460919655}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 1460919655}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [20], 'metadata_pool': 19, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-19T13:05:03.318 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-19T13:05:03.319 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-10-19T13:05:03.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.430+0000 7f9367f33640 1 -- 172.21.15.57:0/214321283 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f93480039a0 msgr2=0x7f9348003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:05:03.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.430+0000 7f9367f33640 1 --2- 172.21.15.57:0/214321283 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f93480039a0 0x7f9348003dc0 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7f9358001d80 tx=0x7f93480057f0 comp rx=0 tx=0).stop 2023-10-19T13:05:03.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.430+0000 7f93664d0640 1 -- 172.21.15.57:0/214321283 reap_dead start 2023-10-19T13:05:03.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.431+0000 7f9367f33640 1 -- 172.21.15.57:0/214321283 shutdown_connections 2023-10-19T13:05:03.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.431+0000 7f9367f33640 1 -- 172.21.15.57:0/214321283 >> 172.21.15.57:0/214321283 conn(0x7f9360069930 msgr2=0x7f9360069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:05:03.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.431+0000 7f9367f33640 1 -- 172.21.15.57:0/214321283 shutdown_connections 2023-10-19T13:05:03.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.431+0000 7f9367f33640 1 -- 172.21.15.57:0/214321283 wait complete. 2023-10-19T13:05:03.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.432+0000 7f9367f33640 1 Processor -- start 2023-10-19T13:05:03.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.432+0000 7f9367f33640 1 -- start start 2023-10-19T13:05:03.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.433+0000 7f9367f33640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f93480039a0 0x7f9360136180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:05:03.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.433+0000 7f9367f33640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9360156be0 0x7f9360152750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:05:03.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.433+0000 7f9367f33640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f93600abdc0 0x7f9360155cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:05:03.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.433+0000 7f9367f33640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93601366c0 con 0x7f93480039a0 2023-10-19T13:05:03.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.433+0000 7f9367f33640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93600b1400 con 0x7f9360156be0 2023-10-19T13:05:03.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.433+0000 7f9367f33640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93600ad040 con 0x7f93600abdc0 2023-10-19T13:05:03.439 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.433+0000 7f9365ccf640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f93480039a0 0x7f9360136180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:05:03.439 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.433+0000 7f9365ccf640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f93480039a0 0x7f9360136180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42886/0 (socket says 172.21.15.57:42886) 2023-10-19T13:05:03.439 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.433+0000 7f9365ccf640 1 -- 172.21.15.57:0/2875185489 learned_addr learned my addr 172.21.15.57:0/2875185489 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:05:03.439 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.433+0000 7f93654ce640 1 --2- 172.21.15.57:0/2875185489 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9360156be0 0x7f9360152750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:05:03.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.433+0000 7f93664d0640 1 --2- 172.21.15.57:0/2875185489 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f93600abdc0 0x7f9360155cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:05:03.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.433+0000 7f9365ccf640 1 -- 172.21.15.57:0/2875185489 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f93600abdc0 msgr2=0x7f9360155cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:05:03.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.433+0000 7f9365ccf640 1 --2- 172.21.15.57:0/2875185489 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f93600abdc0 0x7f9360155cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:03.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.433+0000 7f9365ccf640 1 -- 172.21.15.57:0/2875185489 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9360156be0 msgr2=0x7f9360152750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:05:03.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.433+0000 7f9365ccf640 1 --2- 172.21.15.57:0/2875185489 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9360156be0 0x7f9360152750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:03.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.433+0000 7f9365ccf640 1 -- 172.21.15.57:0/2875185489 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93600ad320 con 0x7f93480039a0 2023-10-19T13:05:03.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.433+0000 7f9365ccf640 1 --2- 172.21.15.57:0/2875185489 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f93480039a0 0x7f9360136180 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7f9358001ea0 tx=0x7f935802c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:05:03.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.433+0000 7f935e7fc640 1 -- 172.21.15.57:0/2875185489 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9358006050 con 0x7f93480039a0 2023-10-19T13:05:03.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.433+0000 7f935e7fc640 1 -- 172.21.15.57:0/2875185489 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f93580062a0 con 0x7f93480039a0 2023-10-19T13:05:03.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.433+0000 7f9367f33640 1 -- 172.21.15.57:0/2875185489 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93600ad600 con 0x7f93480039a0 2023-10-19T13:05:03.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.433+0000 7f935e7fc640 1 -- 172.21.15.57:0/2875185489 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9358006690 con 0x7f93480039a0 2023-10-19T13:05:03.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.433+0000 7f9367f33640 1 -- 172.21.15.57:0/2875185489 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9360152c90 con 0x7f93480039a0 2023-10-19T13:05:03.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.435+0000 7f935e7fc640 1 -- 172.21.15.57:0/2875185489 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 79) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f9358035420 con 0x7f93480039a0 2023-10-19T13:05:03.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.436+0000 7f935e7fc640 1 --2- 172.21.15.57:0/2875185489 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f934c06af60 0x7f934c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:05:03.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.436+0000 7f935e7fc640 1 -- 172.21.15.57:0/2875185489 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f93580ab2d0 con 0x7f93480039a0 2023-10-19T13:05:03.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.436+0000 7f93654ce640 1 --2- 172.21.15.57:0/2875185489 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f934c06af60 0x7f934c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:05:03.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.436+0000 7f93654ce640 1 --2- 172.21.15.57:0/2875185489 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f934c06af60 0x7f934c06d420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f93601556f0 tx=0x7f9354005f90 comp rx=0 tx=0).ready entity=mgr.8419 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:05:03.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.437+0000 7f936ab9f740 1 Processor -- start 2023-10-19T13:05:03.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.437+0000 7f936ab9f740 1 -- start start 2023-10-19T13:05:03.452 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.437+0000 7f936ab9f740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557de5721b40 0x557de56f4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:05:03.452 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.437+0000 7f936ab9f740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557de5721f20 0x557de57248d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:05:03.452 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.437+0000 7f936ab9f740 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x557de56f5020 0x557de5724e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:05:03.452 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.437+0000 7f936ab9f740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x557de5606730 con 0x557de5721b40 2023-10-19T13:05:03.452 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.437+0000 7f936ab9f740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x557de5667780 con 0x557de5721f20 2023-10-19T13:05:03.453 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.437+0000 7f936ab9f740 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x557de5664e70 con 0x557de56f5020 2023-10-19T13:05:03.453 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.438+0000 7f9365ccf640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557de5721b40 0x557de56f4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:05:03.453 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.438+0000 7f9365ccf640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557de5721b40 0x557de56f4ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42890/0 (socket says 172.21.15.57:42890) 2023-10-19T13:05:03.453 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.438+0000 7f9365ccf640 1 -- 172.21.15.57:0/1468673829 learned_addr learned my addr 172.21.15.57:0/1468673829 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:05:03.453 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.438+0000 7f93664d0640 1 --2- 172.21.15.57:0/1468673829 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x557de56f5020 0x557de5724e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:05:03.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.438+0000 7f93654ce640 1 --2- 172.21.15.57:0/1468673829 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557de5721f20 0x557de57248d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:05:03.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.438+0000 7f9365ccf640 1 -- 172.21.15.57:0/1468673829 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x557de56f5020 msgr2=0x557de5724e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T13:05:03.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.438+0000 7f9365ccf640 1 --2- 172.21.15.57:0/1468673829 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x557de56f5020 0x557de5724e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:03.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.438+0000 7f9365ccf640 1 -- 172.21.15.57:0/1468673829 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557de5721f20 msgr2=0x557de57248d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T13:05:03.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.438+0000 7f9365ccf640 1 --2- 172.21.15.57:0/1468673829 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557de5721f20 0x557de57248d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:03.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.438+0000 7f9365ccf640 1 -- 172.21.15.57:0/1468673829 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557de56653d0 con 0x557de5721b40 2023-10-19T13:05:03.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.438+0000 7f9365ccf640 1 --2- 172.21.15.57:0/1468673829 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557de5721b40 0x557de56f4ae0 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7f9358079d70 tx=0x7f9358079a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae9fface579bbdb server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:05:03.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.438+0000 7f93477fe640 1 -- 172.21.15.57:0/1468673829 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93580ab0d0 con 0x557de5721b40 2023-10-19T13:05:03.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.438+0000 7f93477fe640 1 -- 172.21.15.57:0/1468673829 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f9358082030 con 0x557de5721b40 2023-10-19T13:05:03.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.438+0000 7f93477fe640 1 -- 172.21.15.57:0/1468673829 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93580019a0 con 0x557de5721b40 2023-10-19T13:05:03.456 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.439+0000 7f936ab9f740 1 -- 172.21.15.57:0/1468673829 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557de5721b40 msgr2=0x557de56f4ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:05:03.456 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.439+0000 7f936ab9f740 1 --2- 172.21.15.57:0/1468673829 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557de5721b40 0x557de56f4ae0 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7f9358079d70 tx=0x7f9358079a50 comp rx=0 tx=0).stop 2023-10-19T13:05:03.456 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.439+0000 7f936ab9f740 1 -- 172.21.15.57:0/1468673829 shutdown_connections 2023-10-19T13:05:03.456 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.439+0000 7f936ab9f740 1 --2- 172.21.15.57:0/1468673829 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x557de56f5020 0x557de5724e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:03.456 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.439+0000 7f936ab9f740 1 --2- 172.21.15.57:0/1468673829 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557de5721f20 0x557de57248d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:03.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.439+0000 7f936ab9f740 1 --2- 172.21.15.57:0/1468673829 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557de5721b40 0x557de56f4ae0 unknown :-1 s=CLOSED pgs=1772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:03.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.439+0000 7f936ab9f740 1 -- 172.21.15.57:0/1468673829 >> 172.21.15.57:0/1468673829 conn(0x557de56f6d50 msgr2=0x557de573f8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:05:03.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.439+0000 7f936ab9f740 1 -- 172.21.15.57:0/1468673829 shutdown_connections 2023-10-19T13:05:03.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.439+0000 7f936ab9f740 1 -- 172.21.15.57:0/1468673829 wait complete. 2023-10-19T13:05:03.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.439+0000 7f936ab9f740 1 -- start start 2023-10-19T13:05:03.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.440+0000 7f936ab9f740 1 Processor -- start 2023-10-19T13:05:03.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.440+0000 7f936ab9f740 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x557de5721b40 0x557de570f640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:05:03.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.440+0000 7f936ab9f740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557de5721f20 0x557de5726b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:05:03.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.440+0000 7f936ab9f740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557de57270d0 0x557de5739f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:05:03.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.440+0000 7f936ab9f740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x557de5660760 con 0x557de5721f20 2023-10-19T13:05:03.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.440+0000 7f936ab9f740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x557de561b8f0 con 0x557de57270d0 2023-10-19T13:05:03.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.440+0000 7f936ab9f740 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x557de561b590 con 0x557de5721b40 2023-10-19T13:05:03.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.440+0000 7f936ab9f740 10 client.0 _ll_register_callbacks cb 0x557de5657ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-19T13:05:03.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.441+0000 7f93654ce640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557de5721f20 0x557de5726b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:05:03.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.441+0000 7f93654ce640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557de5721f20 0x557de5726b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42900/0 (socket says 172.21.15.57:42900) 2023-10-19T13:05:03.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.441+0000 7f93654ce640 1 -- 172.21.15.57:0/3738720692 learned_addr learned my addr 172.21.15.57:0/3738720692 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:05:03.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.441+0000 7f9365ccf640 1 --2- 172.21.15.57:0/3738720692 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x557de5721b40 0x557de570f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:05:03.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.441+0000 7f93664d0640 1 --2- 172.21.15.57:0/3738720692 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557de57270d0 0x557de5739f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:05:03.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.441+0000 7f93654ce640 1 -- 172.21.15.57:0/3738720692 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x557de5721b40 msgr2=0x557de570f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T13:05:03.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.441+0000 7f93654ce640 1 --2- 172.21.15.57:0/3738720692 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x557de5721b40 0x557de570f640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:03.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.441+0000 7f93654ce640 1 -- 172.21.15.57:0/3738720692 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557de57270d0 msgr2=0x557de5739f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T13:05:03.461 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.441+0000 7f93654ce640 1 --2- 172.21.15.57:0/3738720692 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557de57270d0 0x557de5739f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:03.461 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.441+0000 7f93654ce640 1 -- 172.21.15.57:0/3738720692 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9360152c90 con 0x557de5721f20 2023-10-19T13:05:03.461 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.441+0000 7f9365ccf640 1 --2- 172.21.15.57:0/3738720692 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x557de5721b40 0x557de570f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T13:05:03.461 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.441+0000 7f93664d0640 1 --2- 172.21.15.57:0/3738720692 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557de57270d0 0x557de5739f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T13:05:03.461 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.441+0000 7f93654ce640 1 --2- 172.21.15.57:0/3738720692 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557de5721f20 0x557de5726b90 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7f935401f670 tx=0x7f93540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52d5d927dff19921 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:05:03.462 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.441+0000 7f9344ff9640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2023-10-19T13:05:03.462 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.441+0000 7f9344ff9640 1 -- 172.21.15.57:0/3738720692 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9354025040 con 0x557de5721f20 2023-10-19T13:05:03.462 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.441+0000 7f936ab9f740 1 -- 172.21.15.57:0/3738720692 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557de5659740 con 0x557de5721f20 2023-10-19T13:05:03.462 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.441+0000 7f9344ff9640 1 -- 172.21.15.57:0/3738720692 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(157..157 src has 1..157) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f935402bc40 con 0x557de5721f20 2023-10-19T13:05:03.462 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.441+0000 7f9344ff9640 1 -- 172.21.15.57:0/3738720692 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=63) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9354017070 con 0x557de5721f20 2023-10-19T13:05:03.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.441+0000 7f936ab9f740 10 client.9753 fetch_fsmap learned FSMap version 63 2023-10-19T13:05:03.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.441+0000 7f936ab9f740 1 -- 172.21.15.57:0/3738720692 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=63,monmap=2+}) v3 -- 0x557de565ede0 con 0x557de5721f20 2023-10-19T13:05:03.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.442+0000 7f9344ff9640 1 -- 172.21.15.57:0/3738720692 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x557de5659740 con 0x557de5721f20 2023-10-19T13:05:03.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.442+0000 7f9344ff9640 1 -- 172.21.15.57:0/3738720692 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 63) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f9354011810 con 0x557de5721f20 2023-10-19T13:05:03.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.442+0000 7f936ab9f740 10 client.9753 fetch_fsmap finished waiting for FSMap version 63 2023-10-19T13:05:03.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.442+0000 7f936ab9f740 10 client.9753 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-19T13:05:03.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.442+0000 7f936ab9f740 20 client.9753 populate_metadata read hostname 'smithi057' 2023-10-19T13:05:03.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.442+0000 7f936ab9f740 1 --2- 172.21.15.57:0/3738720692 >> [v2:172.21.15.57:6832/1460919655,v1:172.21.15.57:6833/1460919655] conn(0x557de57529b0 0x557de5752d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:05:03.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.442+0000 7f936ab9f740 4 client.9753 mds_command: new command op to 9618 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-19T13:05:03.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.442+0000 7f936ab9f740 1 -- 172.21.15.57:0/3738720692 --> [v2:172.21.15.57:6832/1460919655,v1:172.21.15.57:6833/1460919655] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557de5659460 con 0x557de57529b0 2023-10-19T13:05:03.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.442+0000 7f9365ccf640 1 --2- 172.21.15.57:0/3738720692 >> [v2:172.21.15.57:6832/1460919655,v1:172.21.15.57:6833/1460919655] conn(0x557de57529b0 0x557de5752d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:05:03.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.442+0000 7f9365ccf640 1 --2- 172.21.15.57:0/3738720692 >> [v2:172.21.15.57:6832/1460919655,v1:172.21.15.57:6833/1460919655] conn(0x557de57529b0 0x557de5752d90 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9567f517627a4417 server_cookie=358a072e9b939928 in_seq=0 out_seq=0 2023-10-19T13:05:03.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.442+0000 7f9344ff9640 10 client.9753 ms_handle_connect on v2:172.21.15.57:6832/1460919655 2023-10-19T13:05:03.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.443+0000 7f9344ff9640 1 -- 172.21.15.57:0/3738720692 <== mds.0 v2:172.21.15.57:6832/1460919655 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x557de5659460 con 0x557de57529b0 2023-10-19T13:05:03.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.443+0000 7f9344ff9640 10 client.9753 handle_command_reply: tid=0 2023-10-19T13:05:03.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.443+0000 7f936ab9f740 1 client.9753 shutdown 2023-10-19T13:05:03.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.443+0000 7f936ab9f740 1 -- 172.21.15.57:0/3738720692 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557de5721f20 msgr2=0x557de5726b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:05:03.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.443+0000 7f936ab9f740 1 --2- 172.21.15.57:0/3738720692 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557de5721f20 0x557de5726b90 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7f935401f670 tx=0x7f93540046c0 comp rx=0 tx=0).stop 2023-10-19T13:05:03.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.443+0000 7f936ab9f740 1 -- 172.21.15.57:0/3738720692 shutdown_connections 2023-10-19T13:05:03.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.443+0000 7f936ab9f740 1 --2- 172.21.15.57:0/3738720692 >> [v2:172.21.15.57:6832/1460919655,v1:172.21.15.57:6833/1460919655] conn(0x557de57529b0 0x557de5752d90 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:03.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.443+0000 7f936ab9f740 1 --2- 172.21.15.57:0/3738720692 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557de57270d0 0x557de5739f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:03.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.443+0000 7f9344ff9640 0 client.9753 ms_handle_reset on v2:172.21.15.57:6832/1460919655 2023-10-19T13:05:03.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.443+0000 7f936ab9f740 1 --2- 172.21.15.57:0/3738720692 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557de5721f20 0x557de5726b90 unknown :-1 s=CLOSED pgs=1773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:03.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.443+0000 7f936ab9f740 1 --2- 172.21.15.57:0/3738720692 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x557de5721b40 0x557de570f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:03.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.443+0000 7f936ab9f740 1 -- 172.21.15.57:0/3738720692 >> 172.21.15.57:0/3738720692 conn(0x557de56f5020 msgr2=0x557de5741ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:05:03.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.443+0000 7f936ab9f740 1 -- 172.21.15.57:0/3738720692 shutdown_connections 2023-10-19T13:05:03.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.443+0000 7f936ab9f740 1 -- 172.21.15.57:0/3738720692 wait complete. 2023-10-19T13:05:03.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.443+0000 7f936ab9f740 20 client.9753 trim_cache size 0 max 16384 2023-10-19T13:05:03.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.458+0000 7f936ab9f740 1 Processor -- start 2023-10-19T13:05:03.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.458+0000 7f936ab9f740 1 -- start start 2023-10-19T13:05:03.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.459+0000 7f936ab9f740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557de5721b40 0x557de5740310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:05:03.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.459+0000 7f936ab9f740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557de5721f20 0x557de5742de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:05:03.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.459+0000 7f936ab9f740 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x557de57270d0 0x557de5743320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:05:03.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.459+0000 7f936ab9f740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x557de55e1270 con 0x557de5721f20 2023-10-19T13:05:03.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.459+0000 7f936ab9f740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x557de56f6870 con 0x557de5721b40 2023-10-19T13:05:03.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.459+0000 7f936ab9f740 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x557de56f8a70 con 0x557de57270d0 2023-10-19T13:05:03.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.459+0000 7f93654ce640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557de5721f20 0x557de5742de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:05:03.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.459+0000 7f93654ce640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557de5721f20 0x557de5742de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42908/0 (socket says 172.21.15.57:42908) 2023-10-19T13:05:03.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.459+0000 7f93654ce640 1 -- 172.21.15.57:0/4265358726 learned_addr learned my addr 172.21.15.57:0/4265358726 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:05:03.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.459+0000 7f93664d0640 1 --2- 172.21.15.57:0/4265358726 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x557de57270d0 0x557de5743320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:05:03.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.459+0000 7f9365ccf640 1 --2- 172.21.15.57:0/4265358726 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557de5721b40 0x557de5740310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:05:03.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.459+0000 7f93654ce640 1 -- 172.21.15.57:0/4265358726 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x557de57270d0 msgr2=0x557de5743320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T13:05:03.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.459+0000 7f93654ce640 1 --2- 172.21.15.57:0/4265358726 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x557de57270d0 0x557de5743320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:03.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.459+0000 7f93654ce640 1 -- 172.21.15.57:0/4265358726 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557de5721b40 msgr2=0x557de5740310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T13:05:03.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.459+0000 7f93654ce640 1 --2- 172.21.15.57:0/4265358726 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557de5721b40 0x557de5740310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:03.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.459+0000 7f93654ce640 1 -- 172.21.15.57:0/4265358726 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557de55fdd00 con 0x557de5721f20 2023-10-19T13:05:03.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.459+0000 7f9365ccf640 1 --2- 172.21.15.57:0/4265358726 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557de5721b40 0x557de5740310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T13:05:03.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.459+0000 7f93664d0640 1 --2- 172.21.15.57:0/4265358726 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x557de57270d0 0x557de5743320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T13:05:03.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.460+0000 7f93654ce640 1 --2- 172.21.15.57:0/4265358726 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557de5721f20 0x557de5742de0 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7f9354012870 tx=0x7f9354004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=531ea3ea34bdbd25 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:05:03.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.460+0000 7f9347fff640 1 -- 172.21.15.57:0/4265358726 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9354025070 con 0x557de5721f20 2023-10-19T13:05:03.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.460+0000 7f9347fff640 1 -- 172.21.15.57:0/4265358726 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f935401f4e0 con 0x557de5721f20 2023-10-19T13:05:03.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.460+0000 7f9347fff640 1 -- 172.21.15.57:0/4265358726 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9354004040 con 0x557de5721f20 2023-10-19T13:05:03.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.460+0000 7f936ab9f740 1 -- 172.21.15.57:0/4265358726 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557de5721f20 msgr2=0x557de5742de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:05:03.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.460+0000 7f936ab9f740 1 --2- 172.21.15.57:0/4265358726 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557de5721f20 0x557de5742de0 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7f9354012870 tx=0x7f9354004730 comp rx=0 tx=0).stop 2023-10-19T13:05:03.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.460+0000 7f936ab9f740 1 -- 172.21.15.57:0/4265358726 shutdown_connections 2023-10-19T13:05:03.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.460+0000 7f936ab9f740 1 --2- 172.21.15.57:0/4265358726 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x557de57270d0 0x557de5743320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:03.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.460+0000 7f936ab9f740 1 --2- 172.21.15.57:0/4265358726 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557de5721f20 0x557de5742de0 unknown :-1 s=CLOSED pgs=1774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:03.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.460+0000 7f936ab9f740 1 --2- 172.21.15.57:0/4265358726 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557de5721b40 0x557de5740310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:03.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.460+0000 7f936ab9f740 1 -- 172.21.15.57:0/4265358726 >> 172.21.15.57:0/4265358726 conn(0x557de56f5020 msgr2=0x557de57288b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:05:03.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.460+0000 7f936ab9f740 1 -- 172.21.15.57:0/4265358726 shutdown_connections 2023-10-19T13:05:03.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.460+0000 7f936ab9f740 1 -- 172.21.15.57:0/4265358726 wait complete. 2023-10-19T13:05:03.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.461+0000 7f936ab9f740 1 -- start start 2023-10-19T13:05:03.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.461+0000 7f936ab9f740 1 Processor -- start 2023-10-19T13:05:03.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.462+0000 7f936ab9f740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557de5721b40 0x557de57a17b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:05:03.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.462+0000 7f936ab9f740 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x557de5721f20 0x557de56f0520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:05:03.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.462+0000 7f936ab9f740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557de57270d0 0x557de5742de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:05:03.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.462+0000 7f936ab9f740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x557de5726640 con 0x557de5721b40 2023-10-19T13:05:03.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.462+0000 7f936ab9f740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x557de573fea0 con 0x557de57270d0 2023-10-19T13:05:03.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.462+0000 7f936ab9f740 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x557de5739b70 con 0x557de5721f20 2023-10-19T13:05:03.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.462+0000 7f936ab9f740 10 client.0 _ll_register_callbacks cb 0x7f935802ceb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-19T13:05:03.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.462+0000 7f9365ccf640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557de5721b40 0x557de57a17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:05:03.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.462+0000 7f9365ccf640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557de5721b40 0x557de57a17b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42912/0 (socket says 172.21.15.57:42912) 2023-10-19T13:05:03.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.462+0000 7f9365ccf640 1 -- 172.21.15.57:0/2415850195 learned_addr learned my addr 172.21.15.57:0/2415850195 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:05:03.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.462+0000 7f93664d0640 1 --2- 172.21.15.57:0/2415850195 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557de57270d0 0x557de5742de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:05:03.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.462+0000 7f93654ce640 1 --2- 172.21.15.57:0/2415850195 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x557de5721f20 0x557de56f0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:05:03.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.462+0000 7f9365ccf640 1 -- 172.21.15.57:0/2415850195 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x557de5721f20 msgr2=0x557de56f0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T13:05:03.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.462+0000 7f9365ccf640 1 --2- 172.21.15.57:0/2415850195 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x557de5721f20 0x557de56f0520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:03.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.462+0000 7f9365ccf640 1 -- 172.21.15.57:0/2415850195 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557de57270d0 msgr2=0x557de5742de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T13:05:03.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.462+0000 7f9365ccf640 1 --2- 172.21.15.57:0/2415850195 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557de57270d0 0x557de5742de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:03.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.462+0000 7f9365ccf640 1 -- 172.21.15.57:0/2415850195 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557de565ede0 con 0x557de5721b40 2023-10-19T13:05:03.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.462+0000 7f93664d0640 1 --2- 172.21.15.57:0/2415850195 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557de57270d0 0x557de5742de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:05:03.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.462+0000 7f93654ce640 1 --2- 172.21.15.57:0/2415850195 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x557de5721f20 0x557de56f0520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:05:03.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.462+0000 7f9365ccf640 1 --2- 172.21.15.57:0/2415850195 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557de5721b40 0x557de57a17b0 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7f9358006cf0 tx=0x7f9358079de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=534938f18b4f6656 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:05:03.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.462+0000 7f9344ff9640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2023-10-19T13:05:03.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.463+0000 7f9344ff9640 1 -- 172.21.15.57:0/2415850195 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93580ab150 con 0x557de5721b40 2023-10-19T13:05:03.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.463+0000 7f9344ff9640 1 -- 172.21.15.57:0/2415850195 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(157..157 src has 1..157) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f93580019a0 con 0x557de5721b40 2023-10-19T13:05:03.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.463+0000 7f936ab9f740 1 -- 172.21.15.57:0/2415850195 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9358082030 con 0x557de5721b40 2023-10-19T13:05:03.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.463+0000 7f9344ff9640 1 -- 172.21.15.57:0/2415850195 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=63) v2 ==== 24+0+0 (secure 0 0 0) 0x7f93580a9120 con 0x557de5721b40 2023-10-19T13:05:03.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.463+0000 7f936ab9f740 10 client.9759 fetch_fsmap learned FSMap version 63 2023-10-19T13:05:03.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.463+0000 7f936ab9f740 1 -- 172.21.15.57:0/2415850195 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=63,monmap=2+}) v3 -- 0x557de56234f0 con 0x557de5721b40 2023-10-19T13:05:03.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.463+0000 7f9344ff9640 1 -- 172.21.15.57:0/2415850195 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f935809c080 con 0x557de5721b40 2023-10-19T13:05:03.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.463+0000 7f9344ff9640 1 -- 172.21.15.57:0/2415850195 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 63) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f9358046060 con 0x557de5721b40 2023-10-19T13:05:03.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.463+0000 7f936ab9f740 10 client.9759 fetch_fsmap finished waiting for FSMap version 63 2023-10-19T13:05:03.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.463+0000 7f936ab9f740 10 client.9759 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-19T13:05:03.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.463+0000 7f936ab9f740 20 client.9759 populate_metadata read hostname 'smithi057' 2023-10-19T13:05:03.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.463+0000 7f936ab9f740 1 --2- 172.21.15.57:0/2415850195 >> [v2:172.21.15.57:6832/1460919655,v1:172.21.15.57:6833/1460919655] conn(0x557de57256b0 0x557de5725a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:05:03.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.463+0000 7f936ab9f740 4 client.9759 mds_command: new command op to 9618 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-19T13:05:03.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.463+0000 7f936ab9f740 1 -- 172.21.15.57:0/2415850195 --> [v2:172.21.15.57:6832/1460919655,v1:172.21.15.57:6833/1460919655] -- command(tid 0: {"prefix": "status"}) v1 -- 0x557de53626c0 con 0x557de57256b0 2023-10-19T13:05:03.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.463+0000 7f93654ce640 1 --2- 172.21.15.57:0/2415850195 >> [v2:172.21.15.57:6832/1460919655,v1:172.21.15.57:6833/1460919655] conn(0x557de57256b0 0x557de5725a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:05:03.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.463+0000 7f9344ff9640 10 client.9759 ms_handle_connect on v2:172.21.15.57:6832/1460919655 2023-10-19T13:05:03.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.463+0000 7f93654ce640 1 --2- 172.21.15.57:0/2415850195 >> [v2:172.21.15.57:6832/1460919655,v1:172.21.15.57:6833/1460919655] conn(0x557de57256b0 0x557de5725a90 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=27f99a75c83172c4 server_cookie=fa3cda42794af18a in_seq=0 out_seq=0 2023-10-19T13:05:03.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.464+0000 7f9344ff9640 1 -- 172.21.15.57:0/2415850195 <== mds.0 v2:172.21.15.57:6832/1460919655 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+326 (crc 0 0 0) 0x557de53626c0 con 0x557de57256b0 2023-10-19T13:05:03.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.464+0000 7f9344ff9640 10 client.9759 handle_command_reply: tid=0 2023-10-19T13:05:03.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.464+0000 7f936ab9f740 1 client.9759 shutdown 2023-10-19T13:05:03.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.464+0000 7f936ab9f740 1 -- 172.21.15.57:0/2415850195 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557de5721b40 msgr2=0x557de57a17b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:05:03.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.464+0000 7f936ab9f740 1 --2- 172.21.15.57:0/2415850195 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557de5721b40 0x557de57a17b0 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7f9358006cf0 tx=0x7f9358079de0 comp rx=0 tx=0).stop 2023-10-19T13:05:03.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.464+0000 7f936ab9f740 1 -- 172.21.15.57:0/2415850195 shutdown_connections 2023-10-19T13:05:03.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.464+0000 7f936ab9f740 1 --2- 172.21.15.57:0/2415850195 >> [v2:172.21.15.57:6832/1460919655,v1:172.21.15.57:6833/1460919655] conn(0x557de57256b0 0x557de5725a90 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:03.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.464+0000 7f936ab9f740 1 --2- 172.21.15.57:0/2415850195 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557de57270d0 0x557de5742de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:03.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.464+0000 7f936ab9f740 1 --2- 172.21.15.57:0/2415850195 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x557de5721f20 0x557de56f0520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:03.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.464+0000 7f9344ff9640 0 client.9759 ms_handle_reset on v2:172.21.15.57:6832/1460919655 2023-10-19T13:05:03.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.464+0000 7f936ab9f740 1 --2- 172.21.15.57:0/2415850195 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557de5721b40 0x557de57a17b0 unknown :-1 s=CLOSED pgs=1775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:03.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.464+0000 7f936ab9f740 1 -- 172.21.15.57:0/2415850195 >> 172.21.15.57:0/2415850195 conn(0x557de56f5020 msgr2=0x557de5728840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:05:03.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.464+0000 7f936ab9f740 1 -- 172.21.15.57:0/2415850195 shutdown_connections 2023-10-19T13:05:03.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.464+0000 7f936ab9f740 1 -- 172.21.15.57:0/2415850195 wait complete. 2023-10-19T13:05:03.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.464+0000 7f936ab9f740 20 client.9759 trim_cache size 0 max 16384 2023-10-19T13:05:03.490 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2023-10-19T13:05:03.490 INFO:teuthology.orchestra.run.smithi057.stdout: "cluster_fsid": "1e9beb19-3260-4dd7-85ef-5b399ce774ac", 2023-10-19T13:05:03.491 INFO:teuthology.orchestra.run.smithi057.stdout: "whoami": 0, 2023-10-19T13:05:03.491 INFO:teuthology.orchestra.run.smithi057.stdout: "id": 9618, 2023-10-19T13:05:03.491 INFO:teuthology.orchestra.run.smithi057.stdout: "want_state": "up:active", 2023-10-19T13:05:03.491 INFO:teuthology.orchestra.run.smithi057.stdout: "state": "up:active", 2023-10-19T13:05:03.491 INFO:teuthology.orchestra.run.smithi057.stdout: "fs_name": "cephfs", 2023-10-19T13:05:03.492 INFO:teuthology.orchestra.run.smithi057.stdout: "rank_uptime": 4.688985111, 2023-10-19T13:05:03.492 INFO:teuthology.orchestra.run.smithi057.stdout: "mdsmap_epoch": 63, 2023-10-19T13:05:03.492 INFO:teuthology.orchestra.run.smithi057.stdout: "osdmap_epoch": 157, 2023-10-19T13:05:03.492 INFO:teuthology.orchestra.run.smithi057.stdout: "osdmap_epoch_barrier": 157, 2023-10-19T13:05:03.492 INFO:teuthology.orchestra.run.smithi057.stdout: "uptime": 9.3932050839999999 2023-10-19T13:05:03.492 INFO:teuthology.orchestra.run.smithi057.stdout:} 2023-10-19T13:05:03.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.464+0000 7f9367f33640 1 -- 172.21.15.57:0/2875185489 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f934c06af60 msgr2=0x7f934c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:05:03.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.464+0000 7f9367f33640 1 --2- 172.21.15.57:0/2875185489 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f934c06af60 0x7f934c06d420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f93601556f0 tx=0x7f9354005f90 comp rx=0 tx=0).stop 2023-10-19T13:05:03.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.464+0000 7f9367f33640 1 -- 172.21.15.57:0/2875185489 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f93480039a0 msgr2=0x7f9360136180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:05:03.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.464+0000 7f9367f33640 1 --2- 172.21.15.57:0/2875185489 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f93480039a0 0x7f9360136180 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7f9358001ea0 tx=0x7f935802c8b0 comp rx=0 tx=0).stop 2023-10-19T13:05:03.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.464+0000 7f9367f33640 1 -- 172.21.15.57:0/2875185489 shutdown_connections 2023-10-19T13:05:03.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.464+0000 7f9367f33640 1 --2- 172.21.15.57:0/2875185489 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f934c06af60 0x7f934c06d420 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:03.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.464+0000 7f9367f33640 1 --2- 172.21.15.57:0/2875185489 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f93600abdc0 0x7f9360155cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:03.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.464+0000 7f9367f33640 1 --2- 172.21.15.57:0/2875185489 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9360156be0 0x7f9360152750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:03.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.464+0000 7f9367f33640 1 --2- 172.21.15.57:0/2875185489 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f93480039a0 0x7f9360136180 unknown :-1 s=CLOSED pgs=1771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:03.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.464+0000 7f9367f33640 1 -- 172.21.15.57:0/2875185489 >> 172.21.15.57:0/2875185489 conn(0x7f9360069930 msgr2=0x7f93601593b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:05:03.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.465+0000 7f9367f33640 1 -- 172.21.15.57:0/2875185489 shutdown_connections 2023-10-19T13:05:03.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.465+0000 7f9367f33640 1 -- 172.21.15.57:0/2875185489 wait complete. 2023-10-19T13:05:03.496 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-19T13:05:03.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.602+0000 7fab8eb33640 1 -- 172.21.15.57:0/1402411937 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fab880afa20 msgr2=0x7fab880705a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:05:03.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.602+0000 7fab8eb33640 1 --2- 172.21.15.57:0/1402411937 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fab880afa20 0x7fab880705a0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fab74005fd0 tx=0x7fab7402d160 comp rx=0 tx=0).stop 2023-10-19T13:05:03.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.602+0000 7fab8eb33640 1 -- 172.21.15.57:0/1402411937 shutdown_connections 2023-10-19T13:05:03.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.602+0000 7fab8eb33640 1 --2- 172.21.15.57:0/1402411937 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fab880afa20 0x7fab880705a0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:03.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.602+0000 7fab8eb33640 1 --2- 172.21.15.57:0/1402411937 >> v2:172.21.15.179:3300/0 conn(0x7fab880cbd20 0x7fab88158300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:03.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.602+0000 7fab8eb33640 1 -- 172.21.15.57:0/1402411937 >> 172.21.15.57:0/1402411937 conn(0x7fab88069930 msgr2=0x7fab88069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:05:03.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.602+0000 7fab8eb33640 1 -- 172.21.15.57:0/1402411937 shutdown_connections 2023-10-19T13:05:03.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.602+0000 7fab8eb33640 1 -- 172.21.15.57:0/1402411937 wait complete. 2023-10-19T13:05:03.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.604+0000 7fab8eb33640 1 Processor -- start 2023-10-19T13:05:03.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.604+0000 7fab8eb33640 1 -- start start 2023-10-19T13:05:03.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.604+0000 7fab8eb33640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fab880afa20 0x7fab880add30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:05:03.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.604+0000 7fab8eb33640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fab880b0530 0x7fab881593c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:05:03.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.604+0000 7fab8eb33640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab880cbd20 0x7fab881f3da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:05:03.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.605+0000 7fab8eb33640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab88159d30 con 0x7fab880cbd20 2023-10-19T13:05:03.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.605+0000 7fab8eb33640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab8815a010 con 0x7fab880afa20 2023-10-19T13:05:03.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.605+0000 7fab8eb33640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab8815a2f0 con 0x7fab880b0530 2023-10-19T13:05:03.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.605+0000 7fab87fff640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fab880b0530 0x7fab881593c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:05:03.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.605+0000 7fab8d0d0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab880cbd20 0x7fab881f3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:05:03.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.605+0000 7fab8c8cf640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fab880afa20 0x7fab880add30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:05:03.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.605+0000 7fab87fff640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fab880b0530 0x7fab881593c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3301/0 says I am v2:172.21.15.57:38798/0 (socket says 172.21.15.57:38798) 2023-10-19T13:05:03.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.605+0000 7fab87fff640 1 -- 172.21.15.57:0/371441244 learned_addr learned my addr 172.21.15.57:0/371441244 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:05:03.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.605+0000 7fab87fff640 1 -- 172.21.15.57:0/371441244 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fab880afa20 msgr2=0x7fab880add30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:05:03.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.605+0000 7fab87fff640 1 --2- 172.21.15.57:0/371441244 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fab880afa20 0x7fab880add30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:03.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.605+0000 7fab87fff640 1 -- 172.21.15.57:0/371441244 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab880cbd20 msgr2=0x7fab881f3da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:05:03.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.605+0000 7fab87fff640 1 --2- 172.21.15.57:0/371441244 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab880cbd20 0x7fab881f3da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:03.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.605+0000 7fab87fff640 1 -- 172.21.15.57:0/371441244 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab7400a670 con 0x7fab880b0530 2023-10-19T13:05:03.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.605+0000 7fab87fff640 1 --2- 172.21.15.57:0/371441244 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fab880b0530 0x7fab881593c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fab7400a960 tx=0x7fab740045e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:05:03.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.605+0000 7fab857fa640 1 -- 172.21.15.57:0/371441244 <== mon.2 v2:172.21.15.179:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab74004bb0 con 0x7fab880b0530 2023-10-19T13:05:03.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.605+0000 7fab857fa640 1 -- 172.21.15.57:0/371441244 <== mon.2 v2:172.21.15.179:3301/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fab74004e00 con 0x7fab880b0530 2023-10-19T13:05:03.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.605+0000 7fab8eb33640 1 -- 172.21.15.57:0/371441244 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab88070cf0 con 0x7fab880b0530 2023-10-19T13:05:03.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.605+0000 7fab857fa640 1 -- 172.21.15.57:0/371441244 <== mon.2 v2:172.21.15.179:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab740035f0 con 0x7fab880b0530 2023-10-19T13:05:03.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.606+0000 7fab8eb33640 1 -- 172.21.15.57:0/371441244 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab8815a8b0 con 0x7fab880b0530 2023-10-19T13:05:03.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.606+0000 7fab8eb33640 1 -- 172.21.15.57:0/371441244 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab881f1580 con 0x7fab880b0530 2023-10-19T13:05:03.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.607+0000 7fab857fa640 1 -- 172.21.15.57:0/371441244 <== mon.2 v2:172.21.15.179:3301/0 4 ==== mgrmap(e 79) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fab74003750 con 0x7fab880b0530 2023-10-19T13:05:03.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.607+0000 7fab857fa640 1 --2- 172.21.15.57:0/371441244 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fab6c06af60 0x7fab6c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:05:03.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.608+0000 7fab857fa640 1 -- 172.21.15.57:0/371441244 <== mon.2 v2:172.21.15.179:3301/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fab740ac6f0 con 0x7fab880b0530 2023-10-19T13:05:03.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.608+0000 7fab8c8cf640 1 --2- 172.21.15.57:0/371441244 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fab6c06af60 0x7fab6c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:05:03.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.608+0000 7fab8c8cf640 1 --2- 172.21.15.57:0/371441244 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fab6c06af60 0x7fab6c06d420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fab8814dde0 tx=0x7fab7c005f90 comp rx=0 tx=0).ready entity=mgr.8419 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:05:03.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.612+0000 7fab857fa640 1 -- 172.21.15.57:0/371441244 <== mon.2 v2:172.21.15.179:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fab740775e0 con 0x7fab880b0530 2023-10-19T13:05:03.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.825+0000 7fab8eb33640 1 -- 172.21.15.57:0/371441244 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fab880cc1e0 con 0x7fab880b0530 2023-10-19T13:05:03.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.825+0000 7fab857fa640 1 -- 172.21.15.57:0/371441244 <== mon.2 v2:172.21.15.179:3301/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v63) v1 ==== 53+0+73 (secure 0 0 0) 0x7fab7407ac00 con 0x7fab880b0530 2023-10-19T13:05:03.828 INFO:teuthology.orchestra.run.smithi057.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-19T13:05:03.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.827+0000 7fab8eb33640 1 -- 172.21.15.57:0/371441244 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fab6c06af60 msgr2=0x7fab6c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:05:03.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.827+0000 7fab8eb33640 1 --2- 172.21.15.57:0/371441244 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fab6c06af60 0x7fab6c06d420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fab8814dde0 tx=0x7fab7c005f90 comp rx=0 tx=0).stop 2023-10-19T13:05:03.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.827+0000 7fab8eb33640 1 -- 172.21.15.57:0/371441244 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fab880b0530 msgr2=0x7fab881593c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:05:03.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.827+0000 7fab8eb33640 1 --2- 172.21.15.57:0/371441244 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fab880b0530 0x7fab881593c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fab7400a960 tx=0x7fab740045e0 comp rx=0 tx=0).stop 2023-10-19T13:05:03.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.827+0000 7fab8eb33640 1 -- 172.21.15.57:0/371441244 shutdown_connections 2023-10-19T13:05:03.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.827+0000 7fab8eb33640 1 --2- 172.21.15.57:0/371441244 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fab6c06af60 0x7fab6c06d420 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:03.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.827+0000 7fab8eb33640 1 --2- 172.21.15.57:0/371441244 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab880cbd20 0x7fab881f3da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:03.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.827+0000 7fab8eb33640 1 --2- 172.21.15.57:0/371441244 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fab880b0530 0x7fab881593c0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:03.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.827+0000 7fab8eb33640 1 --2- 172.21.15.57:0/371441244 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fab880afa20 0x7fab880add30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:03.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.827+0000 7fab8eb33640 1 -- 172.21.15.57:0/371441244 >> 172.21.15.57:0/371441244 conn(0x7fab88069930 msgr2=0x7fab880ae910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:05:03.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.827+0000 7fab8eb33640 1 -- 172.21.15.57:0/371441244 shutdown_connections 2023-10-19T13:05:03.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:03.827+0000 7fab8eb33640 1 -- 172.21.15.57:0/371441244 wait complete. 2023-10-19T13:05:03.840 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-10-19T13:05:03.840 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-19T13:05:03.841 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi057.front.sepia.ceph.com 2023-10-19T13:05:03.841 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-19T13:05:03.841 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-19T13:05:03.842 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-10-19T13:05:03.842 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-10-19T13:05:03.843 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-19T13:05:03.843 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-19T13:05:03.843 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:05:03.844 DEBUG:teuthology.orchestra.run.smithi057:> ip addr 2023-10-19T13:05:03.860 INFO:teuthology.orchestra.run.smithi057.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-19T13:05:03.860 INFO:teuthology.orchestra.run.smithi057.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-19T13:05:03.860 INFO:teuthology.orchestra.run.smithi057.stdout: inet 127.0.0.1/8 scope host lo 2023-10-19T13:05:03.860 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T13:05:03.860 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 ::1/128 scope host 2023-10-19T13:05:03.861 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T13:05:03.861 INFO:teuthology.orchestra.run.smithi057.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-19T13:05:03.861 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:6b:ff:86 brd ff:ff:ff:ff:ff:ff 2023-10-19T13:05:03.861 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp7s0f0 2023-10-19T13:05:03.861 INFO:teuthology.orchestra.run.smithi057.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-19T13:05:03.862 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:6b:ff:87 brd ff:ff:ff:ff:ff:ff 2023-10-19T13:05:03.862 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp7s0f1 2023-10-19T13:05:03.862 INFO:teuthology.orchestra.run.smithi057.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-19T13:05:03.862 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:bd:15:62 brd ff:ff:ff:ff:ff:ff 2023-10-19T13:05:03.862 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp1s0f0 2023-10-19T13:05:03.863 INFO:teuthology.orchestra.run.smithi057.stdout: inet 172.21.15.57/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-19T13:05:03.863 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft 41407sec preferred_lft 41407sec 2023-10-19T13:05:03.863 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 fe80::ec4:7aff:febd:1562/64 scope link noprefixroute 2023-10-19T13:05:03.863 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T13:05:03.863 INFO:teuthology.orchestra.run.smithi057.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-19T13:05:03.864 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:bd:15:63 brd ff:ff:ff:ff:ff:ff 2023-10-19T13:05:03.864 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp1s0f1 2023-10-19T13:05:03.864 INFO:teuthology.orchestra.run.smithi057.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-19T13:05:03.864 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0e:fe:f1:b5:41:59 brd ff:ff:ff:ff:ff:ff 2023-10-19T13:05:03.864 INFO:teuthology.orchestra.run.smithi057.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-19T13:05:03.864 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T13:05:03.865 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 fe80::64b1:a2ff:feb3:b08/64 scope link 2023-10-19T13:05:03.865 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T13:05:03.865 INFO:teuthology.orchestra.run.smithi057.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-19T13:05:03.865 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0e:fe:f1:b5:41:59 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T13:05:03.865 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 fe80::cfe:f1ff:feb5:4159/64 scope link 2023-10-19T13:05:03.866 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T13:05:03.866 INFO:teuthology.orchestra.run.smithi057.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-19T13:05:03.866 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 62:53:72:9d:52:71 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-19T13:05:03.866 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 fe80::6053:72ff:fe9d:5271/64 scope link 2023-10-19T13:05:03.866 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T13:05:03.867 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:05:03.867 DEBUG:teuthology.orchestra.run.smithi057:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-19T13:05:03.929 INFO:teuthology.orchestra.run.smithi057.stdout:1 2023-10-19T13:05:03.931 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:05:03.931 DEBUG:teuthology.orchestra.run.smithi057:> route 2023-10-19T13:05:03.986 INFO:teuthology.orchestra.run.smithi057.stdout:Kernel IP routing table 2023-10-19T13:05:03.986 INFO:teuthology.orchestra.run.smithi057.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-19T13:05:03.986 INFO:teuthology.orchestra.run.smithi057.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-19T13:05:03.986 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-19T13:05:03.987 INFO:teuthology.orchestra.run.smithi057.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-19T13:05:03.987 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:05:03.987 DEBUG:teuthology.orchestra.run.smithi057:> (cd / && exec bash -c ' 2023-10-19T13:05:03.988 DEBUG:teuthology.orchestra.run.smithi057:> set -e 2023-10-19T13:05:03.988 DEBUG:teuthology.orchestra.run.smithi057:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-19T13:05:03.989 DEBUG:teuthology.orchestra.run.smithi057:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-19T13:05:03.989 DEBUG:teuthology.orchestra.run.smithi057:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-19T13:05:03.989 DEBUG:teuthology.orchestra.run.smithi057:> ') 2023-10-19T13:05:04.092 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:05:04.092 DEBUG:teuthology.orchestra.run.smithi057:> ip netns list 2023-10-19T13:05:04.107 INFO:teuthology.orchestra.run.smithi057.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-19T13:05:04.108 INFO:teuthology.orchestra.run.smithi057.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-19T13:05:04.108 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-19T13:05:04.108 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:05:04.109 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip link set brx.0 up 2023-10-19T13:05:04.178 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-10-19T13:05:04.178 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-19T13:05:04.179 DEBUG:teuthology.orchestra.run.smithi057:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-19T13:05:04.233 INFO:teuthology.orchestra.run.smithi057.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-19T13:05:04.234 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-19T13:05:04.234 DEBUG:teuthology.orchestra.run.smithi057:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-19T13:05:04.289 DEBUG:teuthology.orchestra.run.smithi057:> sudo modprobe fuse 2023-10-19T13:05:04.359 DEBUG:teuthology.orchestra.run.smithi057:> cat /proc/self/mounts | awk '{print $2}' 2023-10-19T13:05:04.415 INFO:teuthology.orchestra.run.smithi057.stdout:/proc 2023-10-19T13:05:04.416 INFO:teuthology.orchestra.run.smithi057.stdout:/sys 2023-10-19T13:05:04.416 INFO:teuthology.orchestra.run.smithi057.stdout:/dev 2023-10-19T13:05:04.416 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/security 2023-10-19T13:05:04.417 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/shm 2023-10-19T13:05:04.417 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/pts 2023-10-19T13:05:04.417 INFO:teuthology.orchestra.run.smithi057.stdout:/run 2023-10-19T13:05:04.417 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/cgroup 2023-10-19T13:05:04.417 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/pstore 2023-10-19T13:05:04.418 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/bpf 2023-10-19T13:05:04.418 INFO:teuthology.orchestra.run.smithi057.stdout:/ 2023-10-19T13:05:04.418 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/selinux 2023-10-19T13:05:04.418 INFO:teuthology.orchestra.run.smithi057.stdout:/proc/sys/fs/binfmt_misc 2023-10-19T13:05:04.418 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/hugepages 2023-10-19T13:05:04.419 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/mqueue 2023-10-19T13:05:04.419 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/debug 2023-10-19T13:05:04.419 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/tracing 2023-10-19T13:05:04.419 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/fuse/connections 2023-10-19T13:05:04.419 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/config 2023-10-19T13:05:04.420 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-sysctl.service 2023-10-19T13:05:04.420 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-10-19T13:05:04.420 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-10-19T13:05:04.420 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/nfs/rpc_pipefs 2023-10-19T13:05:04.420 INFO:teuthology.orchestra.run.smithi057.stdout:/run/user/1000 2023-10-19T13:05:04.420 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph 2023-10-19T13:05:04.421 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-0 2023-10-19T13:05:04.421 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-1 2023-10-19T13:05:04.421 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-2 2023-10-19T13:05:04.421 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-3 2023-10-19T13:05:04.421 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns 2023-10-19T13:05:04.422 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T13:05:04.422 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T13:05:04.422 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-19T13:05:04.422 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-19T13:05:04.422 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:05:04.423 DEBUG:teuthology.orchestra.run.smithi057:> ls /sys/fs/fuse/connections 2023-10-19T13:05:04.470 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-10-19T13:05:04.471 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-10-19T13:05:04.513 DEBUG:teuthology.orchestra.run.smithi057:> sudo modprobe fuse 2023-10-19T13:05:04.543 DEBUG:teuthology.orchestra.run.smithi057:> cat /proc/self/mounts | awk '{print $2}' 2023-10-19T13:05:04.598 INFO:teuthology.orchestra.run.smithi057.stdout:/proc 2023-10-19T13:05:04.599 INFO:teuthology.orchestra.run.smithi057.stdout:/sys 2023-10-19T13:05:04.599 INFO:teuthology.orchestra.run.smithi057.stdout:/dev 2023-10-19T13:05:04.599 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/security 2023-10-19T13:05:04.599 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/shm 2023-10-19T13:05:04.599 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/pts 2023-10-19T13:05:04.600 INFO:teuthology.orchestra.run.smithi057.stdout:/run 2023-10-19T13:05:04.600 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/cgroup 2023-10-19T13:05:04.600 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/pstore 2023-10-19T13:05:04.600 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/bpf 2023-10-19T13:05:04.600 INFO:teuthology.orchestra.run.smithi057.stdout:/ 2023-10-19T13:05:04.601 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/selinux 2023-10-19T13:05:04.601 INFO:teuthology.orchestra.run.smithi057.stdout:/proc/sys/fs/binfmt_misc 2023-10-19T13:05:04.601 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/hugepages 2023-10-19T13:05:04.601 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/mqueue 2023-10-19T13:05:04.601 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/debug 2023-10-19T13:05:04.601 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/tracing 2023-10-19T13:05:04.602 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/fuse/connections 2023-10-19T13:05:04.602 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/config 2023-10-19T13:05:04.602 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-sysctl.service 2023-10-19T13:05:04.602 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-10-19T13:05:04.602 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-10-19T13:05:04.603 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/nfs/rpc_pipefs 2023-10-19T13:05:04.603 INFO:teuthology.orchestra.run.smithi057.stdout:/run/user/1000 2023-10-19T13:05:04.603 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph 2023-10-19T13:05:04.603 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-0 2023-10-19T13:05:04.603 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-1 2023-10-19T13:05:04.604 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-2 2023-10-19T13:05:04.604 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-3 2023-10-19T13:05:04.604 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns 2023-10-19T13:05:04.604 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T13:05:04.604 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T13:05:04.604 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-19T13:05:04.605 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-19T13:05:04.606 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:05:04.606 DEBUG:teuthology.orchestra.run.smithi057:> ls /sys/fs/fuse/connections 2023-10-19T13:05:04.607 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi057.stderr:2023-10-19T13:05:04.602+0000 7f4fa083c480 -1 init, newargv = 0x5594fc096b60 newargc=15 2023-10-19T13:05:04.607 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi057.stderr:ceph-fuse[135869]: starting ceph client 2023-10-19T13:05:04.609 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi057.stderr:ceph-fuse[135869]: starting fuse 2023-10-19T13:05:04.624 INFO:teuthology.orchestra.run.smithi057.stdout:43 2023-10-19T13:05:04.624 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2023-10-19T13:05:04.625 DEBUG:teuthology.orchestra.run.smithi057:> sudo stdin-killer -- python3 -c ' 2023-10-19T13:05:04.625 DEBUG:teuthology.orchestra.run.smithi057:> import glob 2023-10-19T13:05:04.625 DEBUG:teuthology.orchestra.run.smithi057:> import re 2023-10-19T13:05:04.626 DEBUG:teuthology.orchestra.run.smithi057:> import os 2023-10-19T13:05:04.626 DEBUG:teuthology.orchestra.run.smithi057:> import subprocess 2023-10-19T13:05:04.627 DEBUG:teuthology.orchestra.run.smithi057:> 2023-10-19T13:05:04.627 DEBUG:teuthology.orchestra.run.smithi057:> def _find_admin_socket(client_name): 2023-10-19T13:05:04.627 DEBUG:teuthology.orchestra.run.smithi057:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-10-19T13:05:04.628 DEBUG:teuthology.orchestra.run.smithi057:> files = glob.glob(asok_path) 2023-10-19T13:05:04.628 DEBUG:teuthology.orchestra.run.smithi057:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-10-19T13:05:04.628 DEBUG:teuthology.orchestra.run.smithi057:> 2023-10-19T13:05:04.629 DEBUG:teuthology.orchestra.run.smithi057:> # Given a non-glob path, it better be there 2023-10-19T13:05:04.629 DEBUG:teuthology.orchestra.run.smithi057:> if "*" not in asok_path: 2023-10-19T13:05:04.630 DEBUG:teuthology.orchestra.run.smithi057:> assert(len(files) == 1) 2023-10-19T13:05:04.630 DEBUG:teuthology.orchestra.run.smithi057:> return files[0] 2023-10-19T13:05:04.630 DEBUG:teuthology.orchestra.run.smithi057:> 2023-10-19T13:05:04.631 DEBUG:teuthology.orchestra.run.smithi057:> for f in files: 2023-10-19T13:05:04.631 DEBUG:teuthology.orchestra.run.smithi057:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-10-19T13:05:04.631 DEBUG:teuthology.orchestra.run.smithi057:> if os.path.exists("/proc/{0}".format(pid)): 2023-10-19T13:05:04.632 DEBUG:teuthology.orchestra.run.smithi057:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-10-19T13:05:04.632 DEBUG:teuthology.orchestra.run.smithi057:> contents = proc_f.read() 2023-10-19T13:05:04.632 DEBUG:teuthology.orchestra.run.smithi057:> if mountpoint in contents: 2023-10-19T13:05:04.633 DEBUG:teuthology.orchestra.run.smithi057:> return f 2023-10-19T13:05:04.633 DEBUG:teuthology.orchestra.run.smithi057:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-10-19T13:05:04.634 DEBUG:teuthology.orchestra.run.smithi057:> 2023-10-19T13:05:04.634 DEBUG:teuthology.orchestra.run.smithi057:> print(_find_admin_socket("client.0")) 2023-10-19T13:05:04.634 DEBUG:teuthology.orchestra.run.smithi057:> ' 2023-10-19T13:05:04.749 INFO:teuthology.orchestra.run.smithi057.stdout:/var/run/ceph/ceph-client.0.135869.asok 2023-10-19T13:05:04.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2023-10-19T13:05:04.756 INFO:tasks.ceph.mds.c.smithi057.stderr:2023-10-19T13:05:04.753+0000 7fdf0665d640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-19T13:05:04.760 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.135869.asok 2023-10-19T13:05:04.760 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:05:04.761 DEBUG:teuthology.orchestra.run.smithi057:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.135869.asok status 2023-10-19T13:05:04.902 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2023-10-19T13:05:04.903 INFO:teuthology.orchestra.run.smithi057.stdout: "metadata": { 2023-10-19T13:05:04.903 INFO:teuthology.orchestra.run.smithi057.stdout: "ceph_sha1": "f7420d7f8e55ceea786dcbf536c338ce920c21eb", 2023-10-19T13:05:04.903 INFO:teuthology.orchestra.run.smithi057.stdout: "ceph_version": "ceph version 18.0.0-6786-gf7420d7f (f7420d7f8e55ceea786dcbf536c338ce920c21eb) reef (dev)", 2023-10-19T13:05:04.903 INFO:teuthology.orchestra.run.smithi057.stdout: "entity_id": "0", 2023-10-19T13:05:04.903 INFO:teuthology.orchestra.run.smithi057.stdout: "hostname": "smithi057", 2023-10-19T13:05:04.904 INFO:teuthology.orchestra.run.smithi057.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-10-19T13:05:04.904 INFO:teuthology.orchestra.run.smithi057.stdout: "pid": "135869", 2023-10-19T13:05:04.904 INFO:teuthology.orchestra.run.smithi057.stdout: "root": "/" 2023-10-19T13:05:04.904 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:05:04.904 INFO:teuthology.orchestra.run.smithi057.stdout: "dentry_count": 0, 2023-10-19T13:05:04.904 INFO:teuthology.orchestra.run.smithi057.stdout: "dentry_pinned_count": 0, 2023-10-19T13:05:04.905 INFO:teuthology.orchestra.run.smithi057.stdout: "id": 9771, 2023-10-19T13:05:04.905 INFO:teuthology.orchestra.run.smithi057.stdout: "inst": { 2023-10-19T13:05:04.905 INFO:teuthology.orchestra.run.smithi057.stdout: "name": { 2023-10-19T13:05:04.905 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "client", 2023-10-19T13:05:04.905 INFO:teuthology.orchestra.run.smithi057.stdout: "num": 9771 2023-10-19T13:05:04.905 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:05:04.906 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": { 2023-10-19T13:05:04.906 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "v1", 2023-10-19T13:05:04.906 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": "192.168.0.1:0", 2023-10-19T13:05:04.906 INFO:teuthology.orchestra.run.smithi057.stdout: "nonce": 3991390606 2023-10-19T13:05:04.906 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:05:04.907 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:05:04.907 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": { 2023-10-19T13:05:04.907 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "v1", 2023-10-19T13:05:04.907 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": "192.168.0.1:0", 2023-10-19T13:05:04.907 INFO:teuthology.orchestra.run.smithi057.stdout: "nonce": 3991390606 2023-10-19T13:05:04.907 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T13:05:04.908 INFO:teuthology.orchestra.run.smithi057.stdout: "inst_str": "client.9771 192.168.0.1:0/3991390606", 2023-10-19T13:05:04.908 INFO:teuthology.orchestra.run.smithi057.stdout: "addr_str": "192.168.0.1:0/3991390606", 2023-10-19T13:05:04.908 INFO:teuthology.orchestra.run.smithi057.stdout: "inode_count": 1, 2023-10-19T13:05:04.908 INFO:teuthology.orchestra.run.smithi057.stdout: "mds_epoch": 63, 2023-10-19T13:05:04.908 INFO:teuthology.orchestra.run.smithi057.stdout: "osd_epoch": 157, 2023-10-19T13:05:04.909 INFO:teuthology.orchestra.run.smithi057.stdout: "osd_epoch_barrier": 0, 2023-10-19T13:05:04.909 INFO:teuthology.orchestra.run.smithi057.stdout: "blocklisted": false, 2023-10-19T13:05:04.909 INFO:teuthology.orchestra.run.smithi057.stdout: "fs_name": "cephfs" 2023-10-19T13:05:04.909 INFO:teuthology.orchestra.run.smithi057.stdout:} 2023-10-19T13:05:04.911 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:05:04.911 DEBUG:teuthology.orchestra.run.smithi057:> stat --file-system '--printf=%T 2023-10-19T13:05:04.911 DEBUG:teuthology.orchestra.run.smithi057:> ' -- /home/ubuntu/cephtest/mnt.0 2023-10-19T13:05:04.927 INFO:teuthology.orchestra.run.smithi057.stdout:fuseblk 2023-10-19T13:05:04.928 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-10-19T13:05:04.928 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:05:04.928 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-19T13:05:04.998 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-19T13:05:05.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.145+0000 7fd7549b0640 1 -- 172.21.15.57:0/3684680700 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd73c0047c0 msgr2=0x7fd73c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:05:05.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.145+0000 7fd7549b0640 1 --2- 172.21.15.57:0/3684680700 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd73c0047c0 0x7fd73c004c40 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7fd73800bb20 tx=0x7fd73c0057f0 comp rx=0 tx=0).stop 2023-10-19T13:05:05.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.145+0000 7fd74ed9d640 1 -- 172.21.15.57:0/3684680700 reap_dead start 2023-10-19T13:05:05.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.146+0000 7fd7549b0640 1 -- 172.21.15.57:0/3684680700 shutdown_connections 2023-10-19T13:05:05.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.146+0000 7fd7549b0640 1 -- 172.21.15.57:0/3684680700 >> 172.21.15.57:0/3684680700 conn(0x7fd75006fae0 msgr2=0x7fd75006fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:05:05.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.146+0000 7fd7549b0640 1 -- 172.21.15.57:0/3684680700 shutdown_connections 2023-10-19T13:05:05.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.146+0000 7fd7549b0640 1 -- 172.21.15.57:0/3684680700 wait complete. 2023-10-19T13:05:05.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.147+0000 7fd7549b0640 1 Processor -- start 2023-10-19T13:05:05.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.147+0000 7fd7549b0640 1 -- start start 2023-10-19T13:05:05.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.148+0000 7fd7549b0640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd73c0047c0 0x7fd7500a6750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:05:05.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.148+0000 7fd7549b0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd750156a90 0x7fd7500a6e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:05:05.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.148+0000 7fd7549b0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd7500cc820 0x7fd750150470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:05:05.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.148+0000 7fd7549b0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7500a75a0 con 0x7fd750156a90 2023-10-19T13:05:05.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.148+0000 7fd7549b0640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd750155cd0 con 0x7fd7500cc820 2023-10-19T13:05:05.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.148+0000 7fd7549b0640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd750155fb0 con 0x7fd73c0047c0 2023-10-19T13:05:05.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.148+0000 7fd74dd9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd750156a90 0x7fd7500a6e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:05:05.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.148+0000 7fd74dd9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd750156a90 0x7fd7500a6e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42932/0 (socket says 172.21.15.57:42932) 2023-10-19T13:05:05.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.148+0000 7fd74dd9b640 1 -- 172.21.15.57:0/3048643844 learned_addr learned my addr 172.21.15.57:0/3048643844 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:05:05.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.148+0000 7fd74e59c640 1 --2- 172.21.15.57:0/3048643844 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd73c0047c0 0x7fd7500a6750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:05:05.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.148+0000 7fd74ed9d640 1 --2- 172.21.15.57:0/3048643844 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd7500cc820 0x7fd750150470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:05:05.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.148+0000 7fd74dd9b640 1 -- 172.21.15.57:0/3048643844 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd73c0047c0 msgr2=0x7fd7500a6750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:05:05.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.148+0000 7fd74dd9b640 1 --2- 172.21.15.57:0/3048643844 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd73c0047c0 0x7fd7500a6750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:05.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.148+0000 7fd74dd9b640 1 -- 172.21.15.57:0/3048643844 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd7500cc820 msgr2=0x7fd750150470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:05:05.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.148+0000 7fd74dd9b640 1 --2- 172.21.15.57:0/3048643844 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd7500cc820 0x7fd750150470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:05.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.148+0000 7fd74dd9b640 1 -- 172.21.15.57:0/3048643844 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7500b2330 con 0x7fd750156a90 2023-10-19T13:05:05.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.148+0000 7fd74e59c640 1 --2- 172.21.15.57:0/3048643844 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd73c0047c0 0x7fd7500a6750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:05:05.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.148+0000 7fd74dd9b640 1 --2- 172.21.15.57:0/3048643844 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd750156a90 0x7fd7500a6e90 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7fd738038040 tx=0x7fd738036720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:05:05.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.148+0000 7fd746ffd640 1 -- 172.21.15.57:0/3048643844 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd738036d20 con 0x7fd750156a90 2023-10-19T13:05:05.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.148+0000 7fd746ffd640 1 -- 172.21.15.57:0/3048643844 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fd738004030 con 0x7fd750156a90 2023-10-19T13:05:05.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.148+0000 7fd7549b0640 1 -- 172.21.15.57:0/3048643844 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd750156290 con 0x7fd750156a90 2023-10-19T13:05:05.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.149+0000 7fd746ffd640 1 -- 172.21.15.57:0/3048643844 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd738004420 con 0x7fd750156a90 2023-10-19T13:05:05.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.149+0000 7fd7549b0640 1 -- 172.21.15.57:0/3048643844 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7500cccd0 con 0x7fd750156a90 2023-10-19T13:05:05.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.149+0000 7fd7549b0640 1 -- 172.21.15.57:0/3048643844 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7500690e0 con 0x7fd750156a90 2023-10-19T13:05:05.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.150+0000 7fd746ffd640 1 -- 172.21.15.57:0/3048643844 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 79) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd7380035f0 con 0x7fd750156a90 2023-10-19T13:05:05.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.151+0000 7fd746ffd640 1 --2- 172.21.15.57:0/3048643844 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fd73006af60 0x7fd73006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:05:05.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.151+0000 7fd746ffd640 1 -- 172.21.15.57:0/3048643844 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fd7380ab9b0 con 0x7fd750156a90 2023-10-19T13:05:05.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.151+0000 7fd74e59c640 1 --2- 172.21.15.57:0/3048643844 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fd73006af60 0x7fd73006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:05:05.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.151+0000 7fd74e59c640 1 --2- 172.21.15.57:0/3048643844 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fd73006af60 0x7fd73006d420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd7480017a0 tx=0x7fd748001470 comp rx=0 tx=0).ready entity=mgr.8419 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:05:05.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.155+0000 7fd746ffd640 1 -- 172.21.15.57:0/3048643844 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd738076880 con 0x7fd750156a90 2023-10-19T13:05:05.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.353+0000 7fd7549b0640 1 -- 172.21.15.57:0/3048643844 --> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fd750159d10 con 0x7fd73006af60 2023-10-19T13:05:05.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.355+0000 7fd746ffd640 1 -- 172.21.15.57:0/3048643844 <== mgr.8419 v2:172.21.15.179:6832/2581690007 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fd750159d10 con 0x7fd73006af60 2023-10-19T13:05:05.358 INFO:teuthology.orchestra.run.smithi057.stdout:[ 2023-10-19T13:05:05.359 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T13:05:05.359 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "cephfs" 2023-10-19T13:05:05.359 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T13:05:05.359 INFO:teuthology.orchestra.run.smithi057.stdout:] 2023-10-19T13:05:05.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.357+0000 7fd7549b0640 1 -- 172.21.15.57:0/3048643844 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fd73006af60 msgr2=0x7fd73006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:05:05.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.357+0000 7fd7549b0640 1 --2- 172.21.15.57:0/3048643844 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fd73006af60 0x7fd73006d420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd7480017a0 tx=0x7fd748001470 comp rx=0 tx=0).stop 2023-10-19T13:05:05.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.357+0000 7fd7549b0640 1 -- 172.21.15.57:0/3048643844 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd750156a90 msgr2=0x7fd7500a6e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:05:05.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.357+0000 7fd7549b0640 1 --2- 172.21.15.57:0/3048643844 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd750156a90 0x7fd7500a6e90 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7fd738038040 tx=0x7fd738036720 comp rx=0 tx=0).stop 2023-10-19T13:05:05.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.357+0000 7fd7549b0640 1 -- 172.21.15.57:0/3048643844 shutdown_connections 2023-10-19T13:05:05.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.357+0000 7fd7549b0640 1 --2- 172.21.15.57:0/3048643844 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fd73006af60 0x7fd73006d420 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:05.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.357+0000 7fd7549b0640 1 --2- 172.21.15.57:0/3048643844 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd7500cc820 0x7fd750150470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:05.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.357+0000 7fd7549b0640 1 --2- 172.21.15.57:0/3048643844 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd750156a90 0x7fd7500a6e90 unknown :-1 s=CLOSED pgs=1781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:05.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.357+0000 7fd7549b0640 1 --2- 172.21.15.57:0/3048643844 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd73c0047c0 0x7fd7500a6750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:05.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.357+0000 7fd7549b0640 1 -- 172.21.15.57:0/3048643844 >> 172.21.15.57:0/3048643844 conn(0x7fd75006fae0 msgr2=0x7fd750151230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:05:05.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.357+0000 7fd7549b0640 1 -- 172.21.15.57:0/3048643844 shutdown_connections 2023-10-19T13:05:05.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.357+0000 7fd7549b0640 1 -- 172.21.15.57:0/3048643844 wait complete. 2023-10-19T13:05:05.370 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-19T13:05:05.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.479+0000 7fe13adee640 1 -- 172.21.15.57:0/1203527408 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1340ad370 msgr2=0x7fe1340ad800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:05:05.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.479+0000 7fe13adee640 1 --2- 172.21.15.57:0/1203527408 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1340ad370 0x7fe1340ad800 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7fe124009830 tx=0x7fe12402bd60 comp rx=0 tx=0).stop 2023-10-19T13:05:05.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.479+0000 7fe13adee640 1 -- 172.21.15.57:0/1203527408 shutdown_connections 2023-10-19T13:05:05.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.479+0000 7fe13adee640 1 --2- 172.21.15.57:0/1203527408 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe13406f0a0 0x7fe13406ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:05.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.479+0000 7fe13adee640 1 --2- 172.21.15.57:0/1203527408 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1340ad370 0x7fe1340ad800 unknown :-1 s=CLOSED pgs=1783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:05.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.479+0000 7fe13adee640 1 -- 172.21.15.57:0/1203527408 >> 172.21.15.57:0/1203527408 conn(0x7fe1340b0670 msgr2=0x7fe1340b0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:05:05.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.479+0000 7fe13adee640 1 -- 172.21.15.57:0/1203527408 shutdown_connections 2023-10-19T13:05:05.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.479+0000 7fe13adee640 1 -- 172.21.15.57:0/1203527408 wait complete. 2023-10-19T13:05:05.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.480+0000 7fe13adee640 1 Processor -- start 2023-10-19T13:05:05.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.480+0000 7fe13adee640 1 -- start start 2023-10-19T13:05:05.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.481+0000 7fe13adee640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe13406d2d0 0x7fe1340ca450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:05:05.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.481+0000 7fe13adee640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe13406f0a0 0x7fe1341f24d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:05:05.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.481+0000 7fe13adee640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe1340ad370 0x7fe1341f2a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:05:05.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.481+0000 7fe13adee640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1340af540 con 0x7fe13406d2d0 2023-10-19T13:05:05.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.481+0000 7fe13adee640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1340cad10 con 0x7fe1340ad370 2023-10-19T13:05:05.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.481+0000 7fe13adee640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1340caff0 con 0x7fe13406f0a0 2023-10-19T13:05:05.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.481+0000 7fe138b8a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe13406d2d0 0x7fe1340ca450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:05:05.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.481+0000 7fe138b8a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe13406d2d0 0x7fe1340ca450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42956/0 (socket says 172.21.15.57:42956) 2023-10-19T13:05:05.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.481+0000 7fe138b8a640 1 -- 172.21.15.57:0/3490900560 learned_addr learned my addr 172.21.15.57:0/3490900560 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:05:05.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.482+0000 7fe13938b640 1 --2- 172.21.15.57:0/3490900560 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe1340ad370 0x7fe1341f2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:05:05.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.482+0000 7fe133fff640 1 --2- 172.21.15.57:0/3490900560 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe13406f0a0 0x7fe1341f24d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:05:05.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.482+0000 7fe138b8a640 1 -- 172.21.15.57:0/3490900560 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe13406f0a0 msgr2=0x7fe1341f24d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:05:05.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.482+0000 7fe138b8a640 1 --2- 172.21.15.57:0/3490900560 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe13406f0a0 0x7fe1341f24d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:05.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.482+0000 7fe138b8a640 1 -- 172.21.15.57:0/3490900560 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe1340ad370 msgr2=0x7fe1341f2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:05:05.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.482+0000 7fe138b8a640 1 --2- 172.21.15.57:0/3490900560 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe1340ad370 0x7fe1341f2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:05.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.482+0000 7fe138b8a640 1 -- 172.21.15.57:0/3490900560 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe124009490 con 0x7fe13406d2d0 2023-10-19T13:05:05.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.482+0000 7fe133fff640 1 --2- 172.21.15.57:0/3490900560 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe13406f0a0 0x7fe1341f24d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:05:05.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.482+0000 7fe13938b640 1 --2- 172.21.15.57:0/3490900560 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe1340ad370 0x7fe1341f2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T13:05:05.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.482+0000 7fe138b8a640 1 --2- 172.21.15.57:0/3490900560 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe13406d2d0 0x7fe1340ca450 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7fe12c00c890 tx=0x7fe12c00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:05:05.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.482+0000 7fe1317fa640 1 -- 172.21.15.57:0/3490900560 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe12c003da0 con 0x7fe13406d2d0 2023-10-19T13:05:05.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.482+0000 7fe1317fa640 1 -- 172.21.15.57:0/3490900560 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fe12c005070 con 0x7fe13406d2d0 2023-10-19T13:05:05.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.482+0000 7fe13adee640 1 -- 172.21.15.57:0/3490900560 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1340cb2d0 con 0x7fe13406d2d0 2023-10-19T13:05:05.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.482+0000 7fe1317fa640 1 -- 172.21.15.57:0/3490900560 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe12c005460 con 0x7fe13406d2d0 2023-10-19T13:05:05.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.482+0000 7fe13adee640 1 -- 172.21.15.57:0/3490900560 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1340ca990 con 0x7fe13406d2d0 2023-10-19T13:05:05.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.483+0000 7fe13adee640 1 -- 172.21.15.57:0/3490900560 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1340ce1d0 con 0x7fe13406d2d0 2023-10-19T13:05:05.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.484+0000 7fe1317fa640 1 -- 172.21.15.57:0/3490900560 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 79) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe12c0055c0 con 0x7fe13406d2d0 2023-10-19T13:05:05.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.484+0000 7fe1317fa640 1 --2- 172.21.15.57:0/3490900560 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fe12006af60 0x7fe12006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:05:05.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.484+0000 7fe1317fa640 1 -- 172.21.15.57:0/3490900560 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fe12c0894f0 con 0x7fe13406d2d0 2023-10-19T13:05:05.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.484+0000 7fe133fff640 1 --2- 172.21.15.57:0/3490900560 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fe12006af60 0x7fe12006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:05:05.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.485+0000 7fe133fff640 1 --2- 172.21.15.57:0/3490900560 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fe12006af60 0x7fe12006d420 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe1280058c0 tx=0x7fe128005850 comp rx=0 tx=0).ready entity=mgr.8419 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:05:05.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.488+0000 7fe1317fa640 1 -- 172.21.15.57:0/3490900560 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe12c054330 con 0x7fe13406d2d0 2023-10-19T13:05:05.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.710+0000 7fe13adee640 1 -- 172.21.15.57:0/3490900560 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7fe1340690e0 con 0x7fe13406d2d0 2023-10-19T13:05:05.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.833+0000 7fe1317fa640 1 -- 172.21.15.57:0/3490900560 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v80) v1 ==== 92+0+0 (secure 0 0 0) 0x7fe12c057950 con 0x7fe13406d2d0 2023-10-19T13:05:05.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.836+0000 7fe13adee640 1 -- 172.21.15.57:0/3490900560 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fe12006af60 msgr2=0x7fe12006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:05:05.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.836+0000 7fe13adee640 1 --2- 172.21.15.57:0/3490900560 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fe12006af60 0x7fe12006d420 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe1280058c0 tx=0x7fe128005850 comp rx=0 tx=0).stop 2023-10-19T13:05:05.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.836+0000 7fe13adee640 1 -- 172.21.15.57:0/3490900560 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe13406d2d0 msgr2=0x7fe1340ca450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:05:05.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.836+0000 7fe13adee640 1 --2- 172.21.15.57:0/3490900560 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe13406d2d0 0x7fe1340ca450 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7fe12c00c890 tx=0x7fe12c00cd50 comp rx=0 tx=0).stop 2023-10-19T13:05:05.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.836+0000 7fe13adee640 1 -- 172.21.15.57:0/3490900560 shutdown_connections 2023-10-19T13:05:05.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.836+0000 7fe13adee640 1 --2- 172.21.15.57:0/3490900560 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fe12006af60 0x7fe12006d420 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:05.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.836+0000 7fe13adee640 1 --2- 172.21.15.57:0/3490900560 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe1340ad370 0x7fe1341f2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:05.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.836+0000 7fe13adee640 1 --2- 172.21.15.57:0/3490900560 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe13406f0a0 0x7fe1341f24d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:05.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.836+0000 7fe13adee640 1 --2- 172.21.15.57:0/3490900560 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe13406d2d0 0x7fe1340ca450 unknown :-1 s=CLOSED pgs=1784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:05.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.836+0000 7fe13adee640 1 -- 172.21.15.57:0/3490900560 >> 172.21.15.57:0/3490900560 conn(0x7fe1340b0670 msgr2=0x7fe1340ac750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:05:05.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.837+0000 7fe13adee640 1 -- 172.21.15.57:0/3490900560 shutdown_connections 2023-10-19T13:05:05.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.837+0000 7fe13adee640 1 -- 172.21.15.57:0/3490900560 wait complete. 2023-10-19T13:05:05.851 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-19T13:05:05.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.963+0000 7f84cf632640 1 -- 172.21.15.57:0/1841195969 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f84c815b260 msgr2=0x7f84c80cf310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:05:05.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.963+0000 7f84cf632640 1 --2- 172.21.15.57:0/1841195969 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f84c815b260 0x7f84c80cf310 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f84c4009830 tx=0x7f84c402bd60 comp rx=0 tx=0).stop 2023-10-19T13:05:05.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.963+0000 7f84cf632640 1 -- 172.21.15.57:0/1841195969 shutdown_connections 2023-10-19T13:05:05.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.963+0000 7f84cf632640 1 --2- 172.21.15.57:0/1841195969 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f84c80cf850 0x7f84c80cc9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:05.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.963+0000 7f84cf632640 1 --2- 172.21.15.57:0/1841195969 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f84c815b260 0x7f84c80cf310 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:05.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.963+0000 7f84cf632640 1 -- 172.21.15.57:0/1841195969 >> 172.21.15.57:0/1841195969 conn(0x7f84c809a030 msgr2=0x7f84c80d0d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:05:05.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.963+0000 7f84cf632640 1 -- 172.21.15.57:0/1841195969 shutdown_connections 2023-10-19T13:05:05.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.963+0000 7f84cf632640 1 -- 172.21.15.57:0/1841195969 wait complete. 2023-10-19T13:05:05.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.964+0000 7f84cf632640 1 Processor -- start 2023-10-19T13:05:05.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.964+0000 7f84cf632640 1 -- start start 2023-10-19T13:05:05.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.965+0000 7f84cf632640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f84c80cf850 0x7f84c815a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:05:05.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.965+0000 7f84cf632640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f84c815b260 0x7f84c80b5c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:05:05.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.965+0000 7f84cf632640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f84c815b640 0x7f84c809b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:05:05.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.965+0000 7f84cf632640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84c809ac60 con 0x7f84c815b640 2023-10-19T13:05:05.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.965+0000 7f84cf632640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84c809af40 con 0x7f84c815b260 2023-10-19T13:05:05.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.965+0000 7f84cf632640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84c809b220 con 0x7f84c80cf850 2023-10-19T13:05:05.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.965+0000 7f84cdbcf640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f84c815b640 0x7f84c809b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:05:05.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.965+0000 7f84cdbcf640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f84c815b640 0x7f84c809b480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42980/0 (socket says 172.21.15.57:42980) 2023-10-19T13:05:05.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.965+0000 7f84cdbcf640 1 -- 172.21.15.57:0/4014142512 learned_addr learned my addr 172.21.15.57:0/4014142512 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:05:05.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.965+0000 7f84cd3ce640 1 --2- 172.21.15.57:0/4014142512 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f84c80cf850 0x7f84c815a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:05:05.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.965+0000 7f84ccbcd640 1 --2- 172.21.15.57:0/4014142512 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f84c815b260 0x7f84c80b5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:05:05.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.965+0000 7f84cdbcf640 1 -- 172.21.15.57:0/4014142512 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f84c80cf850 msgr2=0x7f84c815a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:05:05.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.965+0000 7f84cdbcf640 1 --2- 172.21.15.57:0/4014142512 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f84c80cf850 0x7f84c815a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:05.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.965+0000 7f84cdbcf640 1 -- 172.21.15.57:0/4014142512 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f84c815b260 msgr2=0x7f84c80b5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:05:05.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.965+0000 7f84cdbcf640 1 --2- 172.21.15.57:0/4014142512 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f84c815b260 0x7f84c80b5c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:05.973 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.965+0000 7f84cdbcf640 1 -- 172.21.15.57:0/4014142512 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84c0009840 con 0x7f84c815b640 2023-10-19T13:05:05.973 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.965+0000 7f84cdbcf640 1 --2- 172.21.15.57:0/4014142512 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f84c815b640 0x7f84c809b480 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7f84c000efd0 tx=0x7f84c000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:05:05.973 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.966+0000 7f84bdffb640 1 -- 172.21.15.57:0/4014142512 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84c0009b40 con 0x7f84c815b640 2023-10-19T13:05:05.973 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.966+0000 7f84bdffb640 1 -- 172.21.15.57:0/4014142512 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f84c0003c60 con 0x7f84c815b640 2023-10-19T13:05:05.973 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.966+0000 7f84bdffb640 1 -- 172.21.15.57:0/4014142512 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84c0004020 con 0x7f84c815b640 2023-10-19T13:05:05.974 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.966+0000 7f84cf632640 1 -- 172.21.15.57:0/4014142512 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84c4009490 con 0x7f84c815b640 2023-10-19T13:05:05.974 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.966+0000 7f84cf632640 1 -- 172.21.15.57:0/4014142512 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84c80b5720 con 0x7f84c815b640 2023-10-19T13:05:05.974 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.967+0000 7f84cf632640 1 -- 172.21.15.57:0/4014142512 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84c80675f0 con 0x7f84c815b640 2023-10-19T13:05:05.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.967+0000 7f84bdffb640 1 -- 172.21.15.57:0/4014142512 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 80) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f84c00041a0 con 0x7f84c815b640 2023-10-19T13:05:05.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.968+0000 7f84bdffb640 1 --2- 172.21.15.57:0/4014142512 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f84ac06afb0 0x7f84ac06d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:05:05.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.968+0000 7f84cd3ce640 1 -- 172.21.15.57:0/4014142512 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f84ac06afb0 msgr2=0x7f84ac06d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/2581690007 2023-10-19T13:05:05.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.968+0000 7f84bdffb640 1 -- 172.21.15.57:0/4014142512 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f84c0014070 con 0x7f84c815b640 2023-10-19T13:05:05.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.968+0000 7f84cd3ce640 1 --2- 172.21.15.57:0/4014142512 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f84ac06afb0 0x7f84ac06d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T13:05:05.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:05.972+0000 7f84bdffb640 1 -- 172.21.15.57:0/4014142512 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f84c0055850 con 0x7f84c815b640 2023-10-19T13:05:06.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.150+0000 7f84cf632640 1 -- 172.21.15.57:0/4014142512 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f84c80cd870 con 0x7f84c815b640 2023-10-19T13:05:06.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.156+0000 7f84bdffb640 1 -- 172.21.15.57:0/4014142512 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v60)=0 v60) v1 ==== 138+0+0 (secure 0 0 0) 0x7f84c001d020 con 0x7f84c815b640 2023-10-19T13:05:06.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.158+0000 7f84cf632640 1 -- 172.21.15.57:0/4014142512 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f84ac06afb0 msgr2=0x7f84ac06d470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T13:05:06.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.158+0000 7f84cf632640 1 --2- 172.21.15.57:0/4014142512 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f84ac06afb0 0x7f84ac06d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:06.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.158+0000 7f84cf632640 1 -- 172.21.15.57:0/4014142512 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f84c815b640 msgr2=0x7f84c809b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:05:06.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.158+0000 7f84cf632640 1 --2- 172.21.15.57:0/4014142512 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f84c815b640 0x7f84c809b480 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7f84c000efd0 tx=0x7f84c000c6a0 comp rx=0 tx=0).stop 2023-10-19T13:05:06.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.158+0000 7f84cf632640 1 -- 172.21.15.57:0/4014142512 shutdown_connections 2023-10-19T13:05:06.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.158+0000 7f84cf632640 1 --2- 172.21.15.57:0/4014142512 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f84ac06afb0 0x7f84ac06d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:06.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.158+0000 7f84cf632640 1 --2- 172.21.15.57:0/4014142512 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f84c815b640 0x7f84c809b480 unknown :-1 s=CLOSED pgs=1788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:06.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.158+0000 7f84cf632640 1 --2- 172.21.15.57:0/4014142512 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f84c815b260 0x7f84c80b5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:06.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.158+0000 7f84cf632640 1 --2- 172.21.15.57:0/4014142512 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f84c80cf850 0x7f84c815a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:06.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.158+0000 7f84cf632640 1 -- 172.21.15.57:0/4014142512 >> 172.21.15.57:0/4014142512 conn(0x7f84c809a030 msgr2=0x7f84c80c9e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:05:06.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.158+0000 7f84cf632640 1 -- 172.21.15.57:0/4014142512 shutdown_connections 2023-10-19T13:05:06.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.158+0000 7f84cf632640 1 -- 172.21.15.57:0/4014142512 wait complete. 2023-10-19T13:05:06.172 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-19T13:05:06.225 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:05:06.222+0000 7fed9f832040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-19T13:05:06.238 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:05:06.235+0000 7ff401c32040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-19T13:05:06.273 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:05:06.271+0000 7fed9f832040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-19T13:05:06.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.285+0000 7f507ae64640 1 -- 172.21.15.57:0/1026747868 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f50740ceb40 msgr2=0x7f50740b0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:05:06.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.285+0000 7f507ae64640 1 --2- 172.21.15.57:0/1026747868 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f50740ceb40 0x7f50740b0650 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7f507408a810 tx=0x7f506c02c080 comp rx=0 tx=0).stop 2023-10-19T13:05:06.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.285+0000 7f507ae64640 1 -- 172.21.15.57:0/1026747868 shutdown_connections 2023-10-19T13:05:06.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.285+0000 7f507ae64640 1 --2- 172.21.15.57:0/1026747868 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f50740ceb40 0x7f50740b0650 unknown :-1 s=CLOSED pgs=1789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:06.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.285+0000 7f507ae64640 1 --2- 172.21.15.57:0/1026747868 >> v2:172.21.15.179:3300/0 conn(0x7f50740cf340 0x7f50740d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:06.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.285+0000 7f507ae64640 1 -- 172.21.15.57:0/1026747868 >> 172.21.15.57:0/1026747868 conn(0x7f50740d0d60 msgr2=0x7f50740d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:05:06.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.285+0000 7f507ae64640 1 -- 172.21.15.57:0/1026747868 shutdown_connections 2023-10-19T13:05:06.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.285+0000 7f507ae64640 1 -- 172.21.15.57:0/1026747868 wait complete. 2023-10-19T13:05:06.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.286+0000 7f507ae64640 1 Processor -- start 2023-10-19T13:05:06.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.286+0000 7f507ae64640 1 -- start start 2023-10-19T13:05:06.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.287+0000 7f507ae64640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f50740ceb40 0x7f50740caad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:05:06.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.287+0000 7f507ae64640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f50740cf340 0x7f5074157480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:05:06.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.287+0000 7f507ae64640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f507415b2a0 0x7f507415b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:05:06.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.287+0000 7f507ae64640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f507415cf10 con 0x7f507415b2a0 2023-10-19T13:05:06.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.287+0000 7f507ae64640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50741f1440 con 0x7f50740ceb40 2023-10-19T13:05:06.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.287+0000 7f507ae64640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50741579c0 con 0x7f50740cf340 2023-10-19T13:05:06.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.287+0000 7f5079401640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f507415b2a0 0x7f507415b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:05:06.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.287+0000 7f5079401640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f507415b2a0 0x7f507415b680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43000/0 (socket says 172.21.15.57:43000) 2023-10-19T13:05:06.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.287+0000 7f5079401640 1 -- 172.21.15.57:0/695712100 learned_addr learned my addr 172.21.15.57:0/695712100 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:05:06.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.287+0000 7f5078c00640 1 --2- 172.21.15.57:0/695712100 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f50740ceb40 0x7f50740caad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:05:06.293 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:05:06.287+0000 7ff401c32040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-19T13:05:06.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.287+0000 7f5073fff640 1 --2- 172.21.15.57:0/695712100 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f50740cf340 0x7f5074157480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:05:06.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.287+0000 7f5079401640 1 -- 172.21.15.57:0/695712100 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f50740cf340 msgr2=0x7f5074157480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:05:06.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.287+0000 7f5079401640 1 --2- 172.21.15.57:0/695712100 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f50740cf340 0x7f5074157480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:06.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.287+0000 7f5079401640 1 -- 172.21.15.57:0/695712100 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f50740ceb40 msgr2=0x7f50740caad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:05:06.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.287+0000 7f5079401640 1 --2- 172.21.15.57:0/695712100 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f50740ceb40 0x7f50740caad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:06.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.287+0000 7f5079401640 1 -- 172.21.15.57:0/695712100 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f506c009840 con 0x7f507415b2a0 2023-10-19T13:05:06.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.288+0000 7f5079401640 1 --2- 172.21.15.57:0/695712100 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f507415b2a0 0x7f507415b680 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f506c02c690 tx=0x7f506c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:05:06.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.288+0000 7f50717fa640 1 -- 172.21.15.57:0/695712100 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f506c03d430 con 0x7f507415b2a0 2023-10-19T13:05:06.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.288+0000 7f50717fa640 1 -- 172.21.15.57:0/695712100 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f506c03d680 con 0x7f507415b2a0 2023-10-19T13:05:06.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.288+0000 7f507ae64640 1 -- 172.21.15.57:0/695712100 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f507415bbc0 con 0x7f507415b2a0 2023-10-19T13:05:06.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.288+0000 7f50717fa640 1 -- 172.21.15.57:0/695712100 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f506c03da90 con 0x7f507415b2a0 2023-10-19T13:05:06.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.288+0000 7f507ae64640 1 -- 172.21.15.57:0/695712100 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f507415c140 con 0x7f507415b2a0 2023-10-19T13:05:06.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.289+0000 7f507ae64640 1 -- 172.21.15.57:0/695712100 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f507415d050 con 0x7f507415b2a0 2023-10-19T13:05:06.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.289+0000 7f50717fa640 1 -- 172.21.15.57:0/695712100 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 80) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f506c03dbf0 con 0x7f507415b2a0 2023-10-19T13:05:06.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.290+0000 7f50717fa640 1 --2- 172.21.15.57:0/695712100 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f505806afb0 0x7f505806d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:05:06.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.290+0000 7f50717fa640 1 -- 172.21.15.57:0/695712100 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f506c0aca10 con 0x7f507415b2a0 2023-10-19T13:05:06.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.290+0000 7f5078c00640 1 -- 172.21.15.57:0/695712100 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f505806afb0 msgr2=0x7f505806d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/2581690007 2023-10-19T13:05:06.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.290+0000 7f5078c00640 1 --2- 172.21.15.57:0/695712100 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f505806afb0 0x7f505806d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T13:05:06.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.298+0000 7f50717fa640 1 -- 172.21.15.57:0/695712100 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f506c077990 con 0x7f507415b2a0 2023-10-19T13:05:06.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.476+0000 7f507ae64640 1 -- 172.21.15.57:0/695712100 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f50740b0d00 con 0x7f507415b2a0 2023-10-19T13:05:06.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.480+0000 7f50717fa640 1 -- 172.21.15.57:0/695712100 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v61)=0 v61) v1 ==== 133+0+0 (secure 0 0 0) 0x7f506c0410c0 con 0x7f507415b2a0 2023-10-19T13:05:06.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.482+0000 7f507ae64640 1 -- 172.21.15.57:0/695712100 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f505806afb0 msgr2=0x7f505806d470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T13:05:06.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.482+0000 7f507ae64640 1 --2- 172.21.15.57:0/695712100 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f505806afb0 0x7f505806d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:06.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.482+0000 7f507ae64640 1 -- 172.21.15.57:0/695712100 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f507415b2a0 msgr2=0x7f507415b680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:05:06.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.482+0000 7f507ae64640 1 --2- 172.21.15.57:0/695712100 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f507415b2a0 0x7f507415b680 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f506c02c690 tx=0x7f506c0042f0 comp rx=0 tx=0).stop 2023-10-19T13:05:06.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.482+0000 7f507ae64640 1 -- 172.21.15.57:0/695712100 shutdown_connections 2023-10-19T13:05:06.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.482+0000 7f507ae64640 1 --2- 172.21.15.57:0/695712100 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7f505806afb0 0x7f505806d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:06.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.482+0000 7f507ae64640 1 --2- 172.21.15.57:0/695712100 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f507415b2a0 0x7f507415b680 unknown :-1 s=CLOSED pgs=1790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:06.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.482+0000 7f507ae64640 1 --2- 172.21.15.57:0/695712100 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f50740cf340 0x7f5074157480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:06.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.482+0000 7f507ae64640 1 --2- 172.21.15.57:0/695712100 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f50740ceb40 0x7f50740caad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:06.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.482+0000 7f507ae64640 1 -- 172.21.15.57:0/695712100 >> 172.21.15.57:0/695712100 conn(0x7f50740d0d60 msgr2=0x7f50740ad8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:05:06.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.482+0000 7f507ae64640 1 -- 172.21.15.57:0/695712100 shutdown_connections 2023-10-19T13:05:06.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.482+0000 7f507ae64640 1 -- 172.21.15.57:0/695712100 wait complete. 2023-10-19T13:05:06.496 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T13:05:06.497 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-10-19T13:05:06.516 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1 1m 2023-10-19T13:05:06.664 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:05:06.661+0000 7fed9f832040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-19T13:05:06.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.667+0000 7fdda067c640 1 -- 172.21.15.57:0/3742963997 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd980cddc0 msgr2=0x7fdd980ae820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:05:06.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.667+0000 7fdda067c640 1 --2- 172.21.15.57:0/3742963997 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd980cddc0 0x7fdd980ae820 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7fdd90005fd0 tx=0x7fdd9002c360 comp rx=0 tx=0).stop 2023-10-19T13:05:06.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.668+0000 7fdda067c640 1 -- 172.21.15.57:0/3742963997 shutdown_connections 2023-10-19T13:05:06.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.668+0000 7fdda067c640 1 --2- 172.21.15.57:0/3742963997 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd980cddc0 0x7fdd980ae820 unknown :-1 s=CLOSED pgs=1792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:06.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.668+0000 7fdda067c640 1 --2- 172.21.15.57:0/3742963997 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdd9815b180 0x7fdd980cf350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:06.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.668+0000 7fdda067c640 1 -- 172.21.15.57:0/3742963997 >> 172.21.15.57:0/3742963997 conn(0x7fdd98099ff0 msgr2=0x7fdd980d0d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:05:06.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.668+0000 7fdda067c640 1 -- 172.21.15.57:0/3742963997 shutdown_connections 2023-10-19T13:05:06.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.668+0000 7fdda067c640 1 -- 172.21.15.57:0/3742963997 wait complete. 2023-10-19T13:05:06.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.669+0000 7fdda067c640 1 Processor -- start 2023-10-19T13:05:06.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.669+0000 7fdda067c640 1 -- start start 2023-10-19T13:05:06.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.669+0000 7fdda067c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdd980cddc0 0x7fdd980b61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:05:06.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.669+0000 7fdda067c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdd980ce420 0x7fdd98159500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:05:06.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.670+0000 7fdda067c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd9815b180 0x7fdd98159a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:05:06.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.670+0000 7fdda067c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd980b0ae0 con 0x7fdd9815b180 2023-10-19T13:05:06.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.670+0000 7fdda067c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd98158760 con 0x7fdd980cddc0 2023-10-19T13:05:06.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.670+0000 7fdda067c640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd980b6710 con 0x7fdd980ce420 2023-10-19T13:05:06.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.670+0000 7fdd9ec19640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd9815b180 0x7fdd98159a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:05:06.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.670+0000 7fdd9ec19640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd9815b180 0x7fdd98159a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35184/0 (socket says 172.21.15.57:35184) 2023-10-19T13:05:06.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.670+0000 7fdd9ec19640 1 -- 172.21.15.57:0/3020186089 learned_addr learned my addr 172.21.15.57:0/3020186089 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:05:06.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.670+0000 7fdd9e418640 1 --2- 172.21.15.57:0/3020186089 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdd980cddc0 0x7fdd980b61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:05:06.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.670+0000 7fdd9dc17640 1 --2- 172.21.15.57:0/3020186089 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdd980ce420 0x7fdd98159500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:05:06.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.670+0000 7fdd9ec19640 1 -- 172.21.15.57:0/3020186089 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdd980ce420 msgr2=0x7fdd98159500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:05:06.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.670+0000 7fdd9ec19640 1 --2- 172.21.15.57:0/3020186089 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdd980ce420 0x7fdd98159500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:06.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.670+0000 7fdd9ec19640 1 -- 172.21.15.57:0/3020186089 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdd980cddc0 msgr2=0x7fdd980b61d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:05:06.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.670+0000 7fdd9ec19640 1 --2- 172.21.15.57:0/3020186089 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdd980cddc0 0x7fdd980b61d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:06.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.670+0000 7fdd9ec19640 1 -- 172.21.15.57:0/3020186089 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd90009900 con 0x7fdd9815b180 2023-10-19T13:05:06.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.670+0000 7fdd9ec19640 1 --2- 172.21.15.57:0/3020186089 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd9815b180 0x7fdd98159a40 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7fdd90009bf0 tx=0x7fdd90035bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:05:06.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.670+0000 7fdd8effd640 1 -- 172.21.15.57:0/3020186089 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd90004240 con 0x7fdd9815b180 2023-10-19T13:05:06.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.670+0000 7fdd8effd640 1 -- 172.21.15.57:0/3020186089 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fdd90004490 con 0x7fdd9815b180 2023-10-19T13:05:06.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.670+0000 7fdd8effd640 1 -- 172.21.15.57:0/3020186089 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd90004860 con 0x7fdd9815b180 2023-10-19T13:05:06.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.670+0000 7fdda067c640 1 -- 172.21.15.57:0/3020186089 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd980b5d40 con 0x7fdd9815b180 2023-10-19T13:05:06.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.670+0000 7fdda067c640 1 -- 172.21.15.57:0/3020186089 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd980b0420 con 0x7fdd9815b180 2023-10-19T13:05:06.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.671+0000 7fdda067c640 1 -- 172.21.15.57:0/3020186089 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd98066960 con 0x7fdd9815b180 2023-10-19T13:05:06.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.672+0000 7fdd8effd640 1 -- 172.21.15.57:0/3020186089 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 80) v1 ==== 89348+0+0 (secure 0 0 0) 0x7fdd9003b050 con 0x7fdd9815b180 2023-10-19T13:05:06.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.672+0000 7fdd8effd640 1 --2- 172.21.15.57:0/3020186089 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fdd7c06afb0 0x7fdd7c06d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:05:06.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.672+0000 7fdd8effd640 1 -- 172.21.15.57:0/3020186089 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fdd900ac6e0 con 0x7fdd9815b180 2023-10-19T13:05:06.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.672+0000 7fdd9e418640 1 -- 172.21.15.57:0/3020186089 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fdd7c06afb0 msgr2=0x7fdd7c06d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/2581690007 2023-10-19T13:05:06.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.672+0000 7fdd9e418640 1 --2- 172.21.15.57:0/3020186089 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fdd7c06afb0 0x7fdd7c06d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T13:05:06.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.676+0000 7fdd8effd640 1 -- 172.21.15.57:0/3020186089 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fdd900775c0 con 0x7fdd9815b180 2023-10-19T13:05:06.709 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:05:06.706+0000 7fed9f832040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-19T13:05:06.712 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:05:06.709+0000 7ff401c32040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-19T13:05:06.751 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:05:06.749+0000 7fed9f832040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-19T13:05:06.759 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:05:06.756+0000 7ff401c32040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-19T13:05:06.802 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:05:06.800+0000 7fed9f832040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-19T13:05:06.804 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:05:06.802+0000 7ff401c32040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-19T13:05:06.844 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:05:06.841+0000 7fed9f832040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-19T13:05:06.860 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:05:06.857+0000 7ff401c32040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-19T13:05:06.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.873+0000 7fdd9e418640 1 -- 172.21.15.57:0/3020186089 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fdd7c06afb0 msgr2=0x7fdd7c06d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/2581690007 2023-10-19T13:05:06.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.873+0000 7fdd9e418640 1 --2- 172.21.15.57:0/3020186089 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fdd7c06afb0 0x7fdd7c06d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-19T13:05:06.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:06.877+0000 7fdda067c640 1 -- 172.21.15.57:0/3020186089 --> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fdd980b0c20 con 0x7fdd7c06afb0 2023-10-19T13:05:06.903 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:05:06.901+0000 7ff401c32040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-19T13:05:06.941 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:05:06.939+0000 7fed9f832040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-19T13:05:06.994 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:05:06.991+0000 7fed9f832040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-19T13:05:07.006 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:05:07.004+0000 7ff401c32040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-19T13:05:07.035 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:05:07.032+0000 7fed9f832040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-19T13:05:07.062 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:05:07.059+0000 7ff401c32040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-19T13:05:07.084 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:05:07.081+0000 7fed9f832040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-19T13:05:07.106 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:05:07.103+0000 7ff401c32040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-19T13:05:07.127 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:05:07.124+0000 7fed9f832040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-19T13:05:07.156 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:05:07.154+0000 7ff401c32040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-19T13:05:07.201 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:05:07.199+0000 7ff401c32040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-19T13:05:07.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:07.274+0000 7fdd9e418640 1 -- 172.21.15.57:0/3020186089 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fdd7c06afb0 msgr2=0x7fdd7c06d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/2581690007 2023-10-19T13:05:07.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:07.274+0000 7fdd9e418640 1 --2- 172.21.15.57:0/3020186089 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fdd7c06afb0 0x7fdd7c06d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-19T13:05:07.444 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:05:07.441+0000 7fed9f832040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-19T13:05:07.535 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:05:07.532+0000 7ff401c32040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-19T13:05:07.574 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:05:07.571+0000 7fed9f832040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-19T13:05:07.665 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:05:07.662+0000 7fed9f832040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-19T13:05:07.674 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:05:07.671+0000 7ff401c32040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-19T13:05:07.707 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:05:07.704+0000 7fed9f832040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-19T13:05:07.771 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:05:07.768+0000 7ff401c32040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-19T13:05:07.802 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:05:07.800+0000 7fed9f832040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-19T13:05:07.815 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:05:07.813+0000 7ff401c32040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-19T13:05:07.893 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:05:07.890+0000 7fed9f832040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-19T13:05:07.916 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:05:07.913+0000 7ff401c32040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-19T13:05:08.011 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:05:08.009+0000 7ff401c32040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-19T13:05:08.027 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:05:08.025+0000 7fed9f832040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-19T13:05:08.069 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:05:08.067+0000 7fed9f832040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-19T13:05:08.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:08.075+0000 7fdd9e418640 1 -- 172.21.15.57:0/3020186089 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fdd7c06afb0 msgr2=0x7fdd7c06d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/2581690007 2023-10-19T13:05:08.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:08.075+0000 7fdd9e418640 1 --2- 172.21.15.57:0/3020186089 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fdd7c06afb0 0x7fdd7c06d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-19T13:05:08.155 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:05:08.153+0000 7ff401c32040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-19T13:05:08.173 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:05:08.170+0000 7fed9f832040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-19T13:05:08.200 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:05:08.197+0000 7ff401c32040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-19T13:05:08.310 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:05:08.308+0000 7ff401c32040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-19T13:05:08.791 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:05:08.788+0000 7fed9f832040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-19T13:05:08.962 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:05:08.959+0000 7ff401c32040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-19T13:05:09.054 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:05:09.051+0000 7fed9f832040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-19T13:05:09.103 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:05:09.100+0000 7fed9f832040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-19T13:05:09.205 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:05:09.202+0000 7fed9f832040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-19T13:05:09.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:09.223+0000 7fdd8effd640 1 -- 172.21.15.57:0/3020186089 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mgrmap(e 81) v1 ==== 88957+0+0 (secure 0 0 0) 0x7fdd9000a040 con 0x7fdd9815b180 2023-10-19T13:05:09.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:09.224+0000 7fdd8effd640 1 -- 172.21.15.57:0/3020186089 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fdd7c06afb0 msgr2=0x7fdd7c06d470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T13:05:09.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:09.224+0000 7fdd8effd640 1 --2- 172.21.15.57:0/3020186089 >> [v2:172.21.15.179:6832/2581690007,v1:172.21.15.179:6833/2581690007] conn(0x7fdd7c06afb0 0x7fdd7c06d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:09.241 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:05:09.239+0000 7ff401c32040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-19T13:05:09.295 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:05:09.293+0000 7ff401c32040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-19T13:05:09.338 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:05:09.335+0000 7fed1bbf7640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:05:09.338 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:05:09.335+0000 7fed1bbf7640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:05:09.338 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:05:09.335+0000 7fed1bbf7640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:05:09.339 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:05:09.335+0000 7fed1bbf7640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:05:09.339 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:05:09.335+0000 7fed1bbf7640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:05:09.339 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:05:09.335+0000 7fed1ec3d640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:05:09.339 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:05:09.335+0000 7fed1ec3d640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:05:09.339 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:05:09.335+0000 7fed1ec3d640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:05:09.340 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:05:09.335+0000 7fed1ec3d640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:05:09.340 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:05:09.335+0000 7fed1ec3d640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:05:09.405 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T13:05:09.402+0000 7ff401c32040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-19T13:05:10.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.247+0000 7fdd8effd640 1 -- 172.21.15.57:0/3020186089 <== mon.0 v2:172.21.15.57:3300/0 8 ==== mgrmap(e 82) v1 ==== 89084+0+0 (secure 0 0 0) 0x7fdd9008efb0 con 0x7fdd9815b180 2023-10-19T13:05:10.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.248+0000 7fdd8effd640 1 --2- 172.21.15.57:0/3020186089 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7fdd7c0358b0 0x7fdd7c0753e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:05:10.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.248+0000 7fdd8effd640 1 -- 172.21.15.57:0/3020186089 --> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fdd980b0c20 con 0x7fdd7c0358b0 2023-10-19T13:05:10.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.248+0000 7fdd9e418640 1 --2- 172.21.15.57:0/3020186089 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7fdd7c0358b0 0x7fdd7c0753e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:05:10.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.251+0000 7fdd9e418640 1 --2- 172.21.15.57:0/3020186089 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7fdd7c0358b0 0x7fdd7c0753e0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fdd88009620 tx=0x7fdd88005f90 comp rx=0 tx=0).ready entity=mgr.8818 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:05:10.312 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.309+0000 7fdd8effd640 1 -- 172.21.15.57:0/3020186089 <== mgr.8818 v2:172.21.15.179:6832/4251935070 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7fdd980b0c20 con 0x7fdd7c0358b0 2023-10-19T13:05:10.313 INFO:teuthology.orchestra.run.smithi057.stdout:Schedule set for path /snap_test_dir1 2023-10-19T13:05:10.315 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.312+0000 7fdda067c640 1 -- 172.21.15.57:0/3020186089 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7fdd7c0358b0 msgr2=0x7fdd7c0753e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:05:10.315 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.312+0000 7fdda067c640 1 --2- 172.21.15.57:0/3020186089 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7fdd7c0358b0 0x7fdd7c0753e0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fdd88009620 tx=0x7fdd88005f90 comp rx=0 tx=0).stop 2023-10-19T13:05:10.315 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.312+0000 7fdda067c640 1 -- 172.21.15.57:0/3020186089 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd9815b180 msgr2=0x7fdd98159a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:05:10.315 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.312+0000 7fdda067c640 1 --2- 172.21.15.57:0/3020186089 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd9815b180 0x7fdd98159a40 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7fdd90009bf0 tx=0x7fdd90035bd0 comp rx=0 tx=0).stop 2023-10-19T13:05:10.316 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.312+0000 7fdd9ec19640 1 -- 172.21.15.57:0/3020186089 reap_dead start 2023-10-19T13:05:10.316 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.312+0000 7fdda067c640 1 -- 172.21.15.57:0/3020186089 shutdown_connections 2023-10-19T13:05:10.316 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.312+0000 7fdda067c640 1 -- 172.21.15.57:0/3020186089 >> 172.21.15.57:0/3020186089 conn(0x7fdd98099ff0 msgr2=0x7fdd9815a470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:05:10.316 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.312+0000 7fdda067c640 1 -- 172.21.15.57:0/3020186089 shutdown_connections 2023-10-19T13:05:10.316 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.312+0000 7fdda067c640 1 -- 172.21.15.57:0/3020186089 wait complete. 2023-10-19T13:05:10.327 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1 2023-10-19T13:05:10.328 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention add /snap_test_dir1 55n 2023-10-19T13:05:10.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.442+0000 7f60f2a26640 1 -- 172.21.15.57:0/1421370418 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f60ec15b2a0 msgr2=0x7f60ec0cf720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:05:10.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.442+0000 7f60f2a26640 1 --2- 172.21.15.57:0/1421370418 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f60ec15b2a0 0x7f60ec0cf720 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f60e0009830 tx=0x7f60e002bcd0 comp rx=0 tx=0).stop 2023-10-19T13:05:10.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.442+0000 7f60f2a26640 1 -- 172.21.15.57:0/1421370418 shutdown_connections 2023-10-19T13:05:10.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.442+0000 7f60f2a26640 1 --2- 172.21.15.57:0/1421370418 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f60ec15b2a0 0x7f60ec0cf720 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:10.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.442+0000 7f60f2a26640 1 --2- 172.21.15.57:0/1421370418 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f60ec0cf340 0x7f60ec0d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:10.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.442+0000 7f60f2a26640 1 -- 172.21.15.57:0/1421370418 >> 172.21.15.57:0/1421370418 conn(0x7f60ec0d0d60 msgr2=0x7f60ec0d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:05:10.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.442+0000 7f60f2a26640 1 -- 172.21.15.57:0/1421370418 shutdown_connections 2023-10-19T13:05:10.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.442+0000 7f60f2a26640 1 -- 172.21.15.57:0/1421370418 wait complete. 2023-10-19T13:05:10.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.443+0000 7f60f2a26640 1 Processor -- start 2023-10-19T13:05:10.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.444+0000 7f60f2a26640 1 -- start start 2023-10-19T13:05:10.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.444+0000 7f60f2a26640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f60ec0ceb40 0x7f60ec1f28a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:05:10.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.444+0000 7f60f2a26640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f60ec0cf340 0x7f60ec0c8840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:05:10.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.444+0000 7f60f2a26640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f60ec15b2a0 0x7f60ec0c9d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:05:10.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.444+0000 7f60f2a26640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60ec0c9290 con 0x7f60ec0cf340 2023-10-19T13:05:10.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.444+0000 7f60f2a26640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60ec0b3a20 con 0x7f60ec0ceb40 2023-10-19T13:05:10.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.444+0000 7f60f2a26640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60ec0b3d00 con 0x7f60ec15b2a0 2023-10-19T13:05:10.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.444+0000 7f60eb7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f60ec0cf340 0x7f60ec0c8840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:05:10.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.444+0000 7f60eb7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f60ec0cf340 0x7f60ec0c8840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35200/0 (socket says 172.21.15.57:35200) 2023-10-19T13:05:10.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.444+0000 7f60eb7fe640 1 -- 172.21.15.57:0/2973326608 learned_addr learned my addr 172.21.15.57:0/2973326608 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:05:10.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.445+0000 7f60f0fc3640 1 --2- 172.21.15.57:0/2973326608 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f60ec15b2a0 0x7f60ec0c9d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:05:10.452 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.445+0000 7f60ebfff640 1 --2- 172.21.15.57:0/2973326608 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f60ec0ceb40 0x7f60ec1f28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:05:10.452 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.445+0000 7f60eb7fe640 1 -- 172.21.15.57:0/2973326608 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f60ec15b2a0 msgr2=0x7f60ec0c9d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:05:10.452 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.445+0000 7f60eb7fe640 1 --2- 172.21.15.57:0/2973326608 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f60ec15b2a0 0x7f60ec0c9d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:10.452 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.445+0000 7f60eb7fe640 1 -- 172.21.15.57:0/2973326608 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f60ec0ceb40 msgr2=0x7f60ec1f28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:05:10.452 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.445+0000 7f60eb7fe640 1 --2- 172.21.15.57:0/2973326608 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f60ec0ceb40 0x7f60ec1f28a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:10.453 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.445+0000 7f60eb7fe640 1 -- 172.21.15.57:0/2973326608 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60e0009490 con 0x7f60ec0cf340 2023-10-19T13:05:10.453 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.445+0000 7f60eb7fe640 1 --2- 172.21.15.57:0/2973326608 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f60ec0cf340 0x7f60ec0c8840 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7f60e0009780 tx=0x7f60e00038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:05:10.453 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.445+0000 7f60e8ff9640 1 -- 172.21.15.57:0/2973326608 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60e0003ea0 con 0x7f60ec0cf340 2023-10-19T13:05:10.453 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.445+0000 7f60e8ff9640 1 -- 172.21.15.57:0/2973326608 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f60e0034440 con 0x7f60ec0cf340 2023-10-19T13:05:10.453 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.445+0000 7f60f2a26640 1 -- 172.21.15.57:0/2973326608 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60ec0b3fe0 con 0x7f60ec0cf340 2023-10-19T13:05:10.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.445+0000 7f60e8ff9640 1 -- 172.21.15.57:0/2973326608 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60e0034850 con 0x7f60ec0cf340 2023-10-19T13:05:10.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.445+0000 7f60f2a26640 1 -- 172.21.15.57:0/2973326608 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60ec1f2de0 con 0x7f60ec0cf340 2023-10-19T13:05:10.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.446+0000 7f60f2a26640 1 -- 172.21.15.57:0/2973326608 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60ec0b4500 con 0x7f60ec0cf340 2023-10-19T13:05:10.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.446+0000 7f60e8ff9640 1 -- 172.21.15.57:0/2973326608 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 82) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f60e003b040 con 0x7f60ec0cf340 2023-10-19T13:05:10.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.447+0000 7f60e8ff9640 1 --2- 172.21.15.57:0/2973326608 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7f60d806ace0 0x7f60d806d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:05:10.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.447+0000 7f60e8ff9640 1 -- 172.21.15.57:0/2973326608 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f60e00ab9b0 con 0x7f60ec0cf340 2023-10-19T13:05:10.456 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.447+0000 7f60ebfff640 1 --2- 172.21.15.57:0/2973326608 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7f60d806ace0 0x7f60d806d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:05:10.456 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.448+0000 7f60ebfff640 1 --2- 172.21.15.57:0/2973326608 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7f60d806ace0 0x7f60d806d1a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f60ec0b63a0 tx=0x7f60dc005f90 comp rx=0 tx=0).ready entity=mgr.8818 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:05:10.456 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.451+0000 7f60e8ff9640 1 -- 172.21.15.57:0/2973326608 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f60ec0b4500 con 0x7f60ec0cf340 2023-10-19T13:05:10.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.648+0000 7f60f2a26640 1 -- 172.21.15.57:0/2973326608 --> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "path": "/snap_test_dir1", "retention_spec_or_period": "55n", "target": ["mon-mgr", ""]}) v1 -- 0x7f60ec0cf0f0 con 0x7f60d806ace0 2023-10-19T13:05:10.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.708+0000 7f60e8ff9640 1 -- 172.21.15.57:0/2973326608 <== mgr.8818 v2:172.21.15.179:6832/4251935070 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+40 (secure 0 0 0) 0x7f60dc00e4a0 con 0x7f60d806ace0 2023-10-19T13:05:10.710 INFO:teuthology.orchestra.run.smithi057.stdout:Retention added to path /snap_test_dir1 2023-10-19T13:05:10.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.709+0000 7f60f2a26640 1 -- 172.21.15.57:0/2973326608 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7f60d806ace0 msgr2=0x7f60d806d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:05:10.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.709+0000 7f60f2a26640 1 --2- 172.21.15.57:0/2973326608 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7f60d806ace0 0x7f60d806d1a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f60ec0b63a0 tx=0x7f60dc005f90 comp rx=0 tx=0).stop 2023-10-19T13:05:10.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.709+0000 7f60f2a26640 1 -- 172.21.15.57:0/2973326608 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f60ec0cf340 msgr2=0x7f60ec0c8840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:05:10.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.709+0000 7f60f2a26640 1 --2- 172.21.15.57:0/2973326608 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f60ec0cf340 0x7f60ec0c8840 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7f60e0009780 tx=0x7f60e00038a0 comp rx=0 tx=0).stop 2023-10-19T13:05:10.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.709+0000 7f60f2a26640 1 -- 172.21.15.57:0/2973326608 shutdown_connections 2023-10-19T13:05:10.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.709+0000 7f60f2a26640 1 --2- 172.21.15.57:0/2973326608 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7f60d806ace0 0x7f60d806d1a0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:10.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.709+0000 7f60f2a26640 1 --2- 172.21.15.57:0/2973326608 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f60ec15b2a0 0x7f60ec0c9d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:10.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.709+0000 7f60f2a26640 1 --2- 172.21.15.57:0/2973326608 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f60ec0cf340 0x7f60ec0c8840 unknown :-1 s=CLOSED pgs=1794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:10.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.709+0000 7f60f2a26640 1 --2- 172.21.15.57:0/2973326608 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f60ec0ceb40 0x7f60ec1f28a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:10.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.709+0000 7f60f2a26640 1 -- 172.21.15.57:0/2973326608 >> 172.21.15.57:0/2973326608 conn(0x7f60ec0d0d60 msgr2=0x7f60ec06b880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:05:10.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.709+0000 7f60f2a26640 1 -- 172.21.15.57:0/2973326608 shutdown_connections 2023-10-19T13:05:10.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.709+0000 7f60f2a26640 1 -- 172.21.15.57:0/2973326608 wait complete. 2023-10-19T13:05:10.723 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention added to path /snap_test_dir1 2023-10-19T13:05:10.723 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1m'], expected_retention: [] 2023-10-19T13:05:10.723 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --format json /snap_test_dir1 2023-10-19T13:05:10.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.832+0000 7fc5f8bd0640 1 -- 172.21.15.57:0/377932014 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5e00039a0 msgr2=0x7fc5e0003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:05:10.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.832+0000 7fc5f8bd0640 1 --2- 172.21.15.57:0/377932014 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5e00039a0 0x7fc5e0003dc0 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7fc5e4006fa0 tx=0x7fc5e00057f0 comp rx=0 tx=0).stop 2023-10-19T13:05:10.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.832+0000 7fc5f2d9d640 1 -- 172.21.15.57:0/377932014 reap_dead start 2023-10-19T13:05:10.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.832+0000 7fc5f8bd0640 1 -- 172.21.15.57:0/377932014 shutdown_connections 2023-10-19T13:05:10.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.832+0000 7fc5f8bd0640 1 -- 172.21.15.57:0/377932014 >> 172.21.15.57:0/377932014 conn(0x7fc5f40aec40 msgr2=0x7fc5f40af050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:05:10.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.832+0000 7fc5f8bd0640 1 -- 172.21.15.57:0/377932014 shutdown_connections 2023-10-19T13:05:10.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.832+0000 7fc5f8bd0640 1 -- 172.21.15.57:0/377932014 wait complete. 2023-10-19T13:05:10.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.833+0000 7fc5f8bd0640 1 Processor -- start 2023-10-19T13:05:10.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.834+0000 7fc5f8bd0640 1 -- start start 2023-10-19T13:05:10.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.834+0000 7fc5f8bd0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc5e00039a0 0x7fc5f413c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:05:10.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.834+0000 7fc5f8bd0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5f4139910 0x7fc5f414a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:05:10.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.834+0000 7fc5f8bd0640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc5f4154250 0x7fc5f41546e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:05:10.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.834+0000 7fc5f8bd0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5f4139660 con 0x7fc5f4139910 2023-10-19T13:05:10.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.834+0000 7fc5f8bd0640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5f413c8f0 con 0x7fc5e00039a0 2023-10-19T13:05:10.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.834+0000 7fc5f8bd0640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5f4156f90 con 0x7fc5f4154250 2023-10-19T13:05:10.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.834+0000 7fc5e9d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5f4139910 0x7fc5f414a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:05:10.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.834+0000 7fc5e9d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5f4139910 0x7fc5f414a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35216/0 (socket says 172.21.15.57:35216) 2023-10-19T13:05:10.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.834+0000 7fc5e9d9b640 1 -- 172.21.15.57:0/3067167571 learned_addr learned my addr 172.21.15.57:0/3067167571 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T13:05:10.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.835+0000 7fc5f2d9d640 1 --2- 172.21.15.57:0/3067167571 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc5f4154250 0x7fc5f41546e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:05:10.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.835+0000 7fc5f259c640 1 --2- 172.21.15.57:0/3067167571 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc5e00039a0 0x7fc5f413c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:05:10.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.835+0000 7fc5e9d9b640 1 -- 172.21.15.57:0/3067167571 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc5f4154250 msgr2=0x7fc5f41546e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:05:10.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.835+0000 7fc5e9d9b640 1 --2- 172.21.15.57:0/3067167571 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc5f4154250 0x7fc5f41546e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:10.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.835+0000 7fc5e9d9b640 1 -- 172.21.15.57:0/3067167571 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc5e00039a0 msgr2=0x7fc5f413c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:05:10.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.835+0000 7fc5e9d9b640 1 --2- 172.21.15.57:0/3067167571 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc5e00039a0 0x7fc5f413c3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:10.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.835+0000 7fc5e9d9b640 1 -- 172.21.15.57:0/3067167571 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5f4139dd0 con 0x7fc5f4139910 2023-10-19T13:05:10.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.835+0000 7fc5f2d9d640 1 --2- 172.21.15.57:0/3067167571 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc5f4154250 0x7fc5f41546e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T13:05:10.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.835+0000 7fc5f259c640 1 --2- 172.21.15.57:0/3067167571 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc5e00039a0 0x7fc5f413c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T13:05:10.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.835+0000 7fc5e9d9b640 1 --2- 172.21.15.57:0/3067167571 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5f4139910 0x7fc5f414a790 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7fc5dc004e50 tx=0x7fc5dc001630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:05:10.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.835+0000 7fc5eb7fe640 1 -- 172.21.15.57:0/3067167571 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5dc0107d0 con 0x7fc5f4139910 2023-10-19T13:05:10.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.835+0000 7fc5eb7fe640 1 -- 172.21.15.57:0/3067167571 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fc5dc010a40 con 0x7fc5f4139910 2023-10-19T13:05:10.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.835+0000 7fc5f8bd0640 1 -- 172.21.15.57:0/3067167571 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5f413a0b0 con 0x7fc5f4139910 2023-10-19T13:05:10.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.835+0000 7fc5eb7fe640 1 -- 172.21.15.57:0/3067167571 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5dc010e50 con 0x7fc5f4139910 2023-10-19T13:05:10.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.835+0000 7fc5f8bd0640 1 -- 172.21.15.57:0/3067167571 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5f413a670 con 0x7fc5f4139910 2023-10-19T13:05:10.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.836+0000 7fc5f8bd0640 1 -- 172.21.15.57:0/3067167571 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5f40690e0 con 0x7fc5f4139910 2023-10-19T13:05:10.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.837+0000 7fc5eb7fe640 1 -- 172.21.15.57:0/3067167571 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 82) v1 ==== 89084+0+0 (secure 0 0 0) 0x7fc5dc01b460 con 0x7fc5f4139910 2023-10-19T13:05:10.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.837+0000 7fc5eb7fe640 1 --2- 172.21.15.57:0/3067167571 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7fc5d406ace0 0x7fc5d406d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T13:05:10.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.837+0000 7fc5eb7fe640 1 -- 172.21.15.57:0/3067167571 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 6401+0+0 (secure 0 0 0) 0x7fc5dc08a1e0 con 0x7fc5f4139910 2023-10-19T13:05:10.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.837+0000 7fc5f259c640 1 --2- 172.21.15.57:0/3067167571 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7fc5d406ace0 0x7fc5d406d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T13:05:10.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.838+0000 7fc5f259c640 1 --2- 172.21.15.57:0/3067167571 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7fc5d406ace0 0x7fc5d406d1a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fc5e4034000 tx=0x7fc5e4001a00 comp rx=0 tx=0).ready entity=mgr.8818 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T13:05:10.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:10.841+0000 7fc5eb7fe640 1 -- 172.21.15.57:0/3067167571 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc5dc055130 con 0x7fc5f4139910 2023-10-19T13:05:11.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:11.038+0000 7fc5f8bd0640 1 -- 172.21.15.57:0/3067167571 --> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "/snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc5f40b1360 con 0x7fc5d406ace0 2023-10-19T13:05:11.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:11.045+0000 7fc5eb7fe640 1 -- 172.21.15.57:0/3067167571 <== mgr.8818 v2:172.21.15.179:6832/4251935070 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+74 (secure 0 0 0) 0x7fc5e4001770 con 0x7fc5d406ace0 2023-10-19T13:05:11.048 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T13:05:11.048 INFO:teuthology.orchestra.run.smithi057.stdout:{"path": "/snap_test_dir1", "schedule": ["1m"], "retention": [{"n": 55}]} 2023-10-19T13:05:11.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:11.047+0000 7fc5f8bd0640 1 -- 172.21.15.57:0/3067167571 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7fc5d406ace0 msgr2=0x7fc5d406d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:05:11.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:11.047+0000 7fc5f8bd0640 1 --2- 172.21.15.57:0/3067167571 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7fc5d406ace0 0x7fc5d406d1a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fc5e4034000 tx=0x7fc5e4001a00 comp rx=0 tx=0).stop 2023-10-19T13:05:11.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:11.047+0000 7fc5f8bd0640 1 -- 172.21.15.57:0/3067167571 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5f4139910 msgr2=0x7fc5f414a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T13:05:11.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:11.047+0000 7fc5f8bd0640 1 --2- 172.21.15.57:0/3067167571 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5f4139910 0x7fc5f414a790 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7fc5dc004e50 tx=0x7fc5dc001630 comp rx=0 tx=0).stop 2023-10-19T13:05:11.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:11.047+0000 7fc5f8bd0640 1 -- 172.21.15.57:0/3067167571 shutdown_connections 2023-10-19T13:05:11.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:11.047+0000 7fc5f8bd0640 1 --2- 172.21.15.57:0/3067167571 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7fc5d406ace0 0x7fc5d406d1a0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:11.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:11.047+0000 7fc5f8bd0640 1 --2- 172.21.15.57:0/3067167571 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc5f4154250 0x7fc5f41546e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:11.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:11.047+0000 7fc5f8bd0640 1 --2- 172.21.15.57:0/3067167571 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5f4139910 0x7fc5f414a790 unknown :-1 s=CLOSED pgs=1797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:11.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:11.047+0000 7fc5f8bd0640 1 --2- 172.21.15.57:0/3067167571 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc5e00039a0 0x7fc5f413c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T13:05:11.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:11.047+0000 7fc5f8bd0640 1 -- 172.21.15.57:0/3067167571 >> 172.21.15.57:0/3067167571 conn(0x7fc5f40aec40 msgr2=0x7fc5f415bf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T13:05:11.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:11.047+0000 7fc5f8bd0640 1 -- 172.21.15.57:0/3067167571 shutdown_connections 2023-10-19T13:05:11.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T13:05:11.047+0000 7fc5f8bd0640 1 -- 172.21.15.57:0/3067167571 wait complete. 2023-10-19T13:05:11.061 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "/snap_test_dir1", "schedule": ["1m"], "retention": [{"n": 55}]} 2023-10-19T13:05:11.061 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': '/snap_test_dir1', 'schedule': ['1m'], 'retention': [{'n': 55}]} 2023-10-19T13:05:18.548 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:05:18.551 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:05:48.679 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:05:48.683 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:06:00.719 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:06:00.716+0000 7fed0dbdb640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:06:00.719 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:06:00.716+0000 7fed0dbdb640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:06:00.720 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:06:00.716+0000 7fed0dbdb640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:06:00.720 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:06:00.716+0000 7fed0dbdb640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:06:00.720 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T13:06:00.716+0000 7fed0dbdb640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T13:06:18.756 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:06:18.759 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:06:48.895 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:06:48.898 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:07:18.958 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:07:18.962 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:07:49.086 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:07:49.089 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:08:19.158 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:08:19.162 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:08:49.367 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:08:49.370 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:09:19.439 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:09:19.443 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:09:49.559 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:09:49.562 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:10:19.656 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:10:19.659 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:10:49.812 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:10:49.816 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:11:19.934 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:11:19.938 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:11:50.034 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:11:50.038 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:12:20.229 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:12:20.233 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:12:50.352 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:12:50.356 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:13:20.491 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:13:20.494 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:13:50.555 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:13:50.559 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:14:20.711 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:14:20.714 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:14:50.800 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:14:50.803 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:15:20.942 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:15:20.946 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:15:51.004 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:15:51.007 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:16:21.077 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:16:21.081 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:16:51.308 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:16:51.311 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:17:21.388 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:17:21.391 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:17:51.511 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:17:51.514 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:18:21.633 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:18:21.637 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:18:51.781 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:18:51.784 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:19:21.920 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:19:21.923 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:19:52.042 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:19:52.046 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:20:22.115 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:20:22.120 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:20:52.180 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:20:52.183 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:21:22.247 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:21:22.250 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:21:52.377 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:21:52.380 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:22:22.433 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:22:22.436 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:22:52.552 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:22:52.556 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:23:22.625 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:23:22.629 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:23:52.756 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:23:52.760 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:24:22.903 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:24:22.908 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:24:53.051 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:24:53.055 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:25:23.157 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:25:23.161 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:25:53.279 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:25:53.282 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:26:23.385 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:26:23.388 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:26:53.508 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:26:53.511 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:27:23.622 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:27:23.625 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:27:53.696 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:27:53.700 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:28:23.858 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:28:23.862 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:28:53.956 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:28:53.959 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:29:24.112 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:29:24.115 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:29:54.193 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:29:54.196 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:30:24.341 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:30:24.344 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:30:54.530 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:30:54.534 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:31:24.668 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:31:24.672 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:31:54.791 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:31:54.794 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:32:24.906 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:32:24.909 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:32:54.986 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:32:54.989 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:33:25.118 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:33:25.121 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:33:55.191 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:33:55.195 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:34:25.263 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:34:25.266 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:34:55.412 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:34:55.416 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:35:25.475 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:35:25.480 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:35:55.648 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:35:55.651 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:36:25.738 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:36:25.741 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:36:56.027 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:36:56.030 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:37:26.091 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:37:26.094 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:37:56.239 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:37:56.242 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:38:26.370 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:38:26.374 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:38:56.501 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:38:56.505 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:39:26.648 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:39:26.652 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:39:56.726 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:39:56.730 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:40:26.868 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:40:26.872 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:40:56.932 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:40:56.936 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:41:27.002 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:41:27.006 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:41:57.069 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:41:57.073 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:42:27.217 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:42:27.220 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:42:57.304 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:42:57.307 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:43:27.445 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:43:27.448 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:43:57.532 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:43:57.536 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:44:27.698 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:44:27.702 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:44:57.855 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:44:57.858 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:45:27.994 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:45:27.997 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:45:58.117 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:45:58.120 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:46:28.214 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:46:28.217 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:46:58.361 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:46:58.364 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:47:28.531 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:47:28.535 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:47:58.640 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:47:58.694 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:48:28.783 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:48:28.786 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:48:58.928 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:48:58.931 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:49:28.995 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:49:28.998 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:49:59.357 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:49:59.360 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:50:29.431 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:50:29.435 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:50:59.593 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:50:59.596 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:51:29.766 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:51:29.769 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:51:59.897 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:51:59.901 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:52:30.053 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:52:30.056 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:53:00.209 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:53:00.212 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:53:30.274 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:53:30.277 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:54:00.338 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:54:00.342 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:54:30.434 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:54:30.438 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:55:00.500 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:55:00.503 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:55:30.645 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:55:30.649 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:56:00.711 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:56:00.714 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:56:30.865 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:56:30.869 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:57:00.930 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:57:00.933 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:57:31.061 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:57:31.065 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:58:01.201 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:58:01.204 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:58:31.322 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:58:31.326 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:59:01.404 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:59:01.407 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:59:31.635 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T13:59:31.638 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:00:01.757 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:00:01.761 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:00:31.888 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:00:31.892 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:01:02.027 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:01:02.030 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:01:16.063 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:01:16.063 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T14:01:16.091 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_07_00_UTC 2023-10-19T14:01:16.092 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_08_00_UTC 2023-10-19T14:01:16.092 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_09_00_UTC 2023-10-19T14:01:16.092 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_10_00_UTC 2023-10-19T14:01:16.093 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_11_00_UTC 2023-10-19T14:01:16.093 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_12_00_UTC 2023-10-19T14:01:16.093 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_13_00_UTC 2023-10-19T14:01:16.093 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_14_00_UTC 2023-10-19T14:01:16.093 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_15_00_UTC 2023-10-19T14:01:16.094 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_16_00_UTC 2023-10-19T14:01:16.094 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_17_00_UTC 2023-10-19T14:01:16.094 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_18_00_UTC 2023-10-19T14:01:16.094 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_19_00_UTC 2023-10-19T14:01:16.094 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_20_00_UTC 2023-10-19T14:01:16.095 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_21_00_UTC 2023-10-19T14:01:16.095 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_22_00_UTC 2023-10-19T14:01:16.095 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_23_00_UTC 2023-10-19T14:01:16.095 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_24_00_UTC 2023-10-19T14:01:16.095 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_25_00_UTC 2023-10-19T14:01:16.095 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_26_00_UTC 2023-10-19T14:01:16.096 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_27_00_UTC 2023-10-19T14:01:16.096 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_28_00_UTC 2023-10-19T14:01:16.096 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_29_00_UTC 2023-10-19T14:01:16.096 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_30_00_UTC 2023-10-19T14:01:16.096 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_31_00_UTC 2023-10-19T14:01:16.097 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_32_00_UTC 2023-10-19T14:01:16.097 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_33_00_UTC 2023-10-19T14:01:16.097 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_34_00_UTC 2023-10-19T14:01:16.097 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_35_00_UTC 2023-10-19T14:01:16.097 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_36_00_UTC 2023-10-19T14:01:16.098 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_37_00_UTC 2023-10-19T14:01:16.098 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_38_00_UTC 2023-10-19T14:01:16.098 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_39_00_UTC 2023-10-19T14:01:16.098 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_40_00_UTC 2023-10-19T14:01:16.098 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_41_00_UTC 2023-10-19T14:01:16.099 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_42_00_UTC 2023-10-19T14:01:16.099 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_43_00_UTC 2023-10-19T14:01:16.099 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_44_00_UTC 2023-10-19T14:01:16.099 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_45_00_UTC 2023-10-19T14:01:16.099 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_46_00_UTC 2023-10-19T14:01:16.100 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_47_00_UTC 2023-10-19T14:01:16.100 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_48_00_UTC 2023-10-19T14:01:16.100 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_49_00_UTC 2023-10-19T14:01:16.100 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_50_00_UTC 2023-10-19T14:01:16.100 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_51_00_UTC 2023-10-19T14:01:16.101 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_52_00_UTC 2023-10-19T14:01:16.101 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_53_00_UTC 2023-10-19T14:01:16.101 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_54_00_UTC 2023-10-19T14:01:16.101 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_55_00_UTC 2023-10-19T14:01:16.101 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_56_00_UTC 2023-10-19T14:01:16.102 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_57_00_UTC 2023-10-19T14:01:16.102 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_58_00_UTC 2023-10-19T14:01:16.102 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_59_00_UTC 2023-10-19T14:01:16.102 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_00_00_UTC 2023-10-19T14:01:16.102 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_01_00_UTC 2023-10-19T14:01:16.103 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-19-13_07_00_UTC', 'scheduled-2023-10-19-13_08_00_UTC', 'scheduled-2023-10-19-13_09_00_UTC', 'scheduled-2023-10-19-13_10_00_UTC', 'scheduled-2023-10-19-13_11_00_UTC', 'scheduled-2023-10-19-13_12_00_UTC', 'scheduled-2023-10-19-13_13_00_UTC', 'scheduled-2023-10-19-13_14_00_UTC', 'scheduled-2023-10-19-13_15_00_UTC', 'scheduled-2023-10-19-13_16_00_UTC', 'scheduled-2023-10-19-13_17_00_UTC', 'scheduled-2023-10-19-13_18_00_UTC', 'scheduled-2023-10-19-13_19_00_UTC', 'scheduled-2023-10-19-13_20_00_UTC', 'scheduled-2023-10-19-13_21_00_UTC', 'scheduled-2023-10-19-13_22_00_UTC', 'scheduled-2023-10-19-13_23_00_UTC', 'scheduled-2023-10-19-13_24_00_UTC', 'scheduled-2023-10-19-13_25_00_UTC', 'scheduled-2023-10-19-13_26_00_UTC', 'scheduled-2023-10-19-13_27_00_UTC', 'scheduled-2023-10-19-13_28_00_UTC', 'scheduled-2023-10-19-13_29_00_UTC', 'scheduled-2023-10-19-13_30_00_UTC', 'scheduled-2023-10-19-13_31_00_UTC', 'scheduled-2023-10-19-13_32_00_UTC', 'scheduled-2023-10-19-13_33_00_UTC', 'scheduled-2023-10-19-13_34_00_UTC', 'scheduled-2023-10-19-13_35_00_UTC', 'scheduled-2023-10-19-13_36_00_UTC', 'scheduled-2023-10-19-13_37_00_UTC', 'scheduled-2023-10-19-13_38_00_UTC', 'scheduled-2023-10-19-13_39_00_UTC', 'scheduled-2023-10-19-13_40_00_UTC', 'scheduled-2023-10-19-13_41_00_UTC', 'scheduled-2023-10-19-13_42_00_UTC', 'scheduled-2023-10-19-13_43_00_UTC', 'scheduled-2023-10-19-13_44_00_UTC', 'scheduled-2023-10-19-13_45_00_UTC', 'scheduled-2023-10-19-13_46_00_UTC', 'scheduled-2023-10-19-13_47_00_UTC', 'scheduled-2023-10-19-13_48_00_UTC', 'scheduled-2023-10-19-13_49_00_UTC', 'scheduled-2023-10-19-13_50_00_UTC', 'scheduled-2023-10-19-13_51_00_UTC', 'scheduled-2023-10-19-13_52_00_UTC', 'scheduled-2023-10-19-13_53_00_UTC', 'scheduled-2023-10-19-13_54_00_UTC', 'scheduled-2023-10-19-13_55_00_UTC', 'scheduled-2023-10-19-13_56_00_UTC', 'scheduled-2023-10-19-13_57_00_UTC', 'scheduled-2023-10-19-13_58_00_UTC', 'scheduled-2023-10-19-13_59_00_UTC', 'scheduled-2023-10-19-14_00_00_UTC', 'scheduled-2023-10-19-14_01_00_UTC'] 2023-10-19T14:01:16.104 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1 2023-10-19T14:01:16.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.252+0000 7fcbebe39640 1 -- 172.21.15.57:0/440807033 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcbd0004d00 msgr2=0x7fcbd0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:16.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.252+0000 7fcbebe39640 1 --2- 172.21.15.57:0/440807033 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcbd0004d00 0x7fcbd0005180 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7fcbe0001790 tx=0x7fcbd00057f0 comp rx=0 tx=0).stop 2023-10-19T14:01:16.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.252+0000 7fcbea3d6640 1 -- 172.21.15.57:0/440807033 reap_dead start 2023-10-19T14:01:16.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.252+0000 7fcbebe39640 1 -- 172.21.15.57:0/440807033 shutdown_connections 2023-10-19T14:01:16.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.252+0000 7fcbebe39640 1 -- 172.21.15.57:0/440807033 >> 172.21.15.57:0/440807033 conn(0x7fcbe4069930 msgr2=0x7fcbe4069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:16.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.252+0000 7fcbebe39640 1 -- 172.21.15.57:0/440807033 shutdown_connections 2023-10-19T14:01:16.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.252+0000 7fcbebe39640 1 -- 172.21.15.57:0/440807033 wait complete. 2023-10-19T14:01:16.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.253+0000 7fcbebe39640 1 Processor -- start 2023-10-19T14:01:16.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.253+0000 7fcbebe39640 1 -- start start 2023-10-19T14:01:16.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.254+0000 7fcbebe39640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcbe40ad4d0 0x7fcbe40ad8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:16.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.254+0000 7fcbebe39640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcbe4132e20 0x7fcbe4152180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:16.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.254+0000 7fcbebe39640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcbe40addf0 0x7fcbe41526c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:16.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.254+0000 7fcbebe39640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe4133830 con 0x7fcbe40addf0 2023-10-19T14:01:16.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.254+0000 7fcbebe39640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe413f660 con 0x7fcbe4132e20 2023-10-19T14:01:16.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.254+0000 7fcbebe39640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe4134090 con 0x7fcbe40ad4d0 2023-10-19T14:01:16.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.254+0000 7fcbea3d6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcbe40addf0 0x7fcbe41526c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:16.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.254+0000 7fcbea3d6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcbe40addf0 0x7fcbe41526c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:49360/0 (socket says 172.21.15.57:49360) 2023-10-19T14:01:16.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.254+0000 7fcbea3d6640 1 -- 172.21.15.57:0/3587366455 learned_addr learned my addr 172.21.15.57:0/3587366455 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:01:16.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.254+0000 7fcbe9bd5640 1 --2- 172.21.15.57:0/3587366455 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcbe40ad4d0 0x7fcbe40ad8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:16.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.254+0000 7fcbea3d6640 1 -- 172.21.15.57:0/3587366455 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcbe40ad4d0 msgr2=0x7fcbe40ad8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:16.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.254+0000 7fcbe93d4640 1 --2- 172.21.15.57:0/3587366455 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcbe4132e20 0x7fcbe4152180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:16.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.254+0000 7fcbea3d6640 1 --2- 172.21.15.57:0/3587366455 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcbe40ad4d0 0x7fcbe40ad8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:16.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.254+0000 7fcbea3d6640 1 -- 172.21.15.57:0/3587366455 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcbe4132e20 msgr2=0x7fcbe4152180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:16.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.254+0000 7fcbea3d6640 1 --2- 172.21.15.57:0/3587366455 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcbe4132e20 0x7fcbe4152180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:16.262 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.254+0000 7fcbea3d6640 1 -- 172.21.15.57:0/3587366455 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbe413ece0 con 0x7fcbe40addf0 2023-10-19T14:01:16.262 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.254+0000 7fcbe9bd5640 1 --2- 172.21.15.57:0/3587366455 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcbe40ad4d0 0x7fcbe40ad8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:01:16.262 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.254+0000 7fcbe93d4640 1 --2- 172.21.15.57:0/3587366455 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcbe4132e20 0x7fcbe4152180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:01:16.262 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.254+0000 7fcbea3d6640 1 --2- 172.21.15.57:0/3587366455 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcbe40addf0 0x7fcbe41526c0 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7fcbe002c1c0 tx=0x7fcbe0004dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:16.262 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.255+0000 7fcbda7fc640 1 -- 172.21.15.57:0/3587366455 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbe002c820 con 0x7fcbe40addf0 2023-10-19T14:01:16.263 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.255+0000 7fcbda7fc640 1 -- 172.21.15.57:0/3587366455 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fcbe002ca70 con 0x7fcbe40addf0 2023-10-19T14:01:16.263 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.255+0000 7fcbebe39640 1 -- 172.21.15.57:0/3587366455 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbe4133390 con 0x7fcbe40addf0 2023-10-19T14:01:16.263 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.255+0000 7fcbda7fc640 1 -- 172.21.15.57:0/3587366455 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbe002ce60 con 0x7fcbe40addf0 2023-10-19T14:01:16.263 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.255+0000 7fcbebe39640 1 -- 172.21.15.57:0/3587366455 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbe4133d20 con 0x7fcbe40addf0 2023-10-19T14:01:16.263 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.256+0000 7fcbebe39640 1 -- 172.21.15.57:0/3587366455 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbe4133670 con 0x7fcbe40addf0 2023-10-19T14:01:16.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.256+0000 7fcbda7fc640 1 -- 172.21.15.57:0/3587366455 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 86) v1 ==== 89360+0+0 (secure 0 0 0) 0x7fcbe0032070 con 0x7fcbe40addf0 2023-10-19T14:01:16.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.257+0000 7fcbda7fc640 1 --2- 172.21.15.57:0/3587366455 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7fcbc806afb0 0x7fcbc806d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:16.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.257+0000 7fcbda7fc640 1 -- 172.21.15.57:0/3587366455 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 6489+0+0 (secure 0 0 0) 0x7fcbe003c070 con 0x7fcbe40addf0 2023-10-19T14:01:16.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.257+0000 7fcbe9bd5640 1 --2- 172.21.15.57:0/3587366455 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7fcbc806afb0 0x7fcbc806d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:16.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.257+0000 7fcbe9bd5640 1 --2- 172.21.15.57:0/3587366455 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7fcbc806afb0 0x7fcbc806d470 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fcbe4153670 tx=0x7fcbdc009500 comp rx=0 tx=0).ready entity=mgr.8818 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:16.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.261+0000 7fcbda7fc640 1 -- 172.21.15.57:0/3587366455 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fcbe0075690 con 0x7fcbe40addf0 2023-10-19T14:01:16.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.456+0000 7fcbebe39640 1 -- 172.21.15.57:0/3587366455 --> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fcbe40690e0 con 0x7fcbc806afb0 2023-10-19T14:01:16.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.465+0000 7fcbda7fc640 1 -- 172.21.15.57:0/3587366455 <== mgr.8818 v2:172.21.15.179:6832/4251935070 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7fcbe40690e0 con 0x7fcbc806afb0 2023-10-19T14:01:16.467 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:01:16.467 INFO:teuthology.orchestra.run.smithi057.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1", "rel_path": "/snap_test_dir1", "schedule": "1m", "retention": {"n": 55}, "start": "2023-10-19T00:00:00", "created": "2023-10-19T13:05:10", "first": "2023-10-19T13:06:00", "last": "2023-10-19T14:01:00", "last_pruned": "2023-10-19T14:01:00", "created_count": 56, "pruned_count": 1, "active": true}] 2023-10-19T14:01:16.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.466+0000 7fcbebe39640 1 -- 172.21.15.57:0/3587366455 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7fcbc806afb0 msgr2=0x7fcbc806d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:16.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.466+0000 7fcbebe39640 1 --2- 172.21.15.57:0/3587366455 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7fcbc806afb0 0x7fcbc806d470 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fcbe4153670 tx=0x7fcbdc009500 comp rx=0 tx=0).stop 2023-10-19T14:01:16.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.466+0000 7fcbebe39640 1 -- 172.21.15.57:0/3587366455 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcbe40addf0 msgr2=0x7fcbe41526c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:16.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.466+0000 7fcbebe39640 1 --2- 172.21.15.57:0/3587366455 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcbe40addf0 0x7fcbe41526c0 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7fcbe002c1c0 tx=0x7fcbe0004dd0 comp rx=0 tx=0).stop 2023-10-19T14:01:16.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.466+0000 7fcbebe39640 1 -- 172.21.15.57:0/3587366455 shutdown_connections 2023-10-19T14:01:16.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.466+0000 7fcbebe39640 1 --2- 172.21.15.57:0/3587366455 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcbe40addf0 0x7fcbe41526c0 unknown :-1 s=CLOSED pgs=1800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:16.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.466+0000 7fcbebe39640 1 --2- 172.21.15.57:0/3587366455 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcbe4132e20 0x7fcbe4152180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:16.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.466+0000 7fcbebe39640 1 --2- 172.21.15.57:0/3587366455 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7fcbc806afb0 0x7fcbc806d470 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:16.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.466+0000 7fcbebe39640 1 --2- 172.21.15.57:0/3587366455 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcbe40ad4d0 0x7fcbe40ad8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:16.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.466+0000 7fcbebe39640 1 -- 172.21.15.57:0/3587366455 >> 172.21.15.57:0/3587366455 conn(0x7fcbe4069930 msgr2=0x7fcbe415aef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:16.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.466+0000 7fcbebe39640 1 -- 172.21.15.57:0/3587366455 shutdown_connections 2023-10-19T14:01:16.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.466+0000 7fcbebe39640 1 -- 172.21.15.57:0/3587366455 wait complete. 2023-10-19T14:01:16.479 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1", "rel_path": "/snap_test_dir1", "schedule": "1m", "retention": {"n": 55}, "start": "2023-10-19T00:00:00", "created": "2023-10-19T13:05:10", "first": "2023-10-19T13:06:00", "last": "2023-10-19T14:01:00", "last_pruned": "2023-10-19T14:01:00", "created_count": 56, "pruned_count": 1, "active": true}] 2023-10-19T14:01:16.480 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1', 'rel_path': '/snap_test_dir1', 'schedule': '1m', 'retention': {'n': 55}, 'start': '2023-10-19T00:00:00', 'created': '2023-10-19T13:05:10', 'first': '2023-10-19T13:06:00', 'last': '2023-10-19T14:01:00', 'last_pruned': '2023-10-19T14:01:00', 'created_count': 56, 'pruned_count': 1, 'active': True} 2023-10-19T14:01:16.480 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 55 2023-10-19T14:01:16.480 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 56 2023-10-19T14:01:16.481 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1 2023-10-19T14:01:16.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.590+0000 7f9929500640 1 -- 172.21.15.57:0/961181508 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f992406af00 msgr2=0x7f992406b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:16.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.590+0000 7f9929500640 1 --2- 172.21.15.57:0/961181508 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f992406af00 0x7f992406b2e0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f9914009830 tx=0x7f991402bd60 comp rx=0 tx=0).stop 2023-10-19T14:01:16.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.590+0000 7f9929500640 1 -- 172.21.15.57:0/961181508 shutdown_connections 2023-10-19T14:01:16.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.590+0000 7f9929500640 1 --2- 172.21.15.57:0/961181508 >> v2:172.21.15.179:3300/0 conn(0x7f99240abc10 0x7f99240c9b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:16.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.590+0000 7f9929500640 1 --2- 172.21.15.57:0/961181508 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f992406af00 0x7f992406b2e0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:16.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.590+0000 7f9929500640 1 -- 172.21.15.57:0/961181508 >> 172.21.15.57:0/961181508 conn(0x7f9924069930 msgr2=0x7f9924069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:16.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.590+0000 7f9929500640 1 -- 172.21.15.57:0/961181508 shutdown_connections 2023-10-19T14:01:16.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.590+0000 7f9929500640 1 -- 172.21.15.57:0/961181508 wait complete. 2023-10-19T14:01:16.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.591+0000 7f9929500640 1 Processor -- start 2023-10-19T14:01:16.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.591+0000 7f9929500640 1 -- start start 2023-10-19T14:01:16.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.592+0000 7f9929500640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f992406af00 0x7f99241f41f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:16.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.592+0000 7f9929500640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f99240abc10 0x7f99240aa8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:16.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.592+0000 7f9929500640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f99240c96b0 0x7f99240aae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:16.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.592+0000 7f9929500640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9924070a10 con 0x7f99240abc10 2023-10-19T14:01:16.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.592+0000 7f9929500640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f992406fd90 con 0x7f99240c96b0 2023-10-19T14:01:16.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.592+0000 7f9929500640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9924070070 con 0x7f992406af00 2023-10-19T14:01:16.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.592+0000 7f991a7fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f99240abc10 0x7f99240aa8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:16.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.592+0000 7f991a7fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f99240abc10 0x7f99240aa8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:49372/0 (socket says 172.21.15.57:49372) 2023-10-19T14:01:16.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.592+0000 7f991a7fc640 1 -- 172.21.15.57:0/3663968043 learned_addr learned my addr 172.21.15.57:0/3663968043 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:01:16.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.592+0000 7f9922ffd640 1 --2- 172.21.15.57:0/3663968043 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f992406af00 0x7f99241f41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:16.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.592+0000 7f99237fe640 1 --2- 172.21.15.57:0/3663968043 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f99240c96b0 0x7f99240aae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:16.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.593+0000 7f991a7fc640 1 -- 172.21.15.57:0/3663968043 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f992406af00 msgr2=0x7f99241f41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:16.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.593+0000 7f991a7fc640 1 --2- 172.21.15.57:0/3663968043 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f992406af00 0x7f99241f41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:16.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.593+0000 7f991a7fc640 1 -- 172.21.15.57:0/3663968043 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f99240c96b0 msgr2=0x7f99240aae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:16.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.593+0000 7f991a7fc640 1 --2- 172.21.15.57:0/3663968043 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f99240c96b0 0x7f99240aae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:16.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.593+0000 7f991a7fc640 1 -- 172.21.15.57:0/3663968043 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9914009490 con 0x7f99240abc10 2023-10-19T14:01:16.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.593+0000 7f9922ffd640 1 --2- 172.21.15.57:0/3663968043 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f992406af00 0x7f99241f41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:01:16.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.593+0000 7f991a7fc640 1 --2- 172.21.15.57:0/3663968043 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f99240abc10 0x7f99240aa8c0 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7f990c00c430 tx=0x7f990c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:16.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.593+0000 7f991bfff640 1 -- 172.21.15.57:0/3663968043 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f990c0052d0 con 0x7f99240abc10 2023-10-19T14:01:16.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.593+0000 7f991bfff640 1 -- 172.21.15.57:0/3663968043 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f990c0035f0 con 0x7f99240abc10 2023-10-19T14:01:16.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.593+0000 7f9929500640 1 -- 172.21.15.57:0/3663968043 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9924070350 con 0x7f99240abc10 2023-10-19T14:01:16.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.593+0000 7f991bfff640 1 -- 172.21.15.57:0/3663968043 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f990c003a00 con 0x7f99240abc10 2023-10-19T14:01:16.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.593+0000 7f9929500640 1 -- 172.21.15.57:0/3663968043 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9924070e30 con 0x7f99240abc10 2023-10-19T14:01:16.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.594+0000 7f9929500640 1 -- 172.21.15.57:0/3663968043 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99240690e0 con 0x7f99240abc10 2023-10-19T14:01:16.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.594+0000 7f991bfff640 1 -- 172.21.15.57:0/3663968043 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 86) v1 ==== 89360+0+0 (secure 0 0 0) 0x7f990c003b60 con 0x7f99240abc10 2023-10-19T14:01:16.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.595+0000 7f991bfff640 1 --2- 172.21.15.57:0/3663968043 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7f990406afb0 0x7f990406d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:16.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.595+0000 7f991bfff640 1 -- 172.21.15.57:0/3663968043 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 6489+0+0 (secure 0 0 0) 0x7f990c0892b0 con 0x7f99240abc10 2023-10-19T14:01:16.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.595+0000 7f9922ffd640 1 --2- 172.21.15.57:0/3663968043 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7f990406afb0 0x7f990406d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:16.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.595+0000 7f9922ffd640 1 --2- 172.21.15.57:0/3663968043 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7f990406afb0 0x7f990406d470 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9914009830 tx=0x7f9914004730 comp rx=0 tx=0).ready entity=mgr.8818 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:16.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.599+0000 7f991bfff640 1 -- 172.21.15.57:0/3663968043 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f990c053fd0 con 0x7f99240abc10 2023-10-19T14:01:16.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.797+0000 7f9929500640 1 -- 172.21.15.57:0/3663968043 --> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f9924159c50 con 0x7f990406afb0 2023-10-19T14:01:16.868 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.866+0000 7f991bfff640 1 -- 172.21.15.57:0/3663968043 <== mgr.8818 v2:172.21.15.179:6832/4251935070 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+42 (secure 0 0 0) 0x7f991402c800 con 0x7f990406afb0 2023-10-19T14:01:16.869 INFO:teuthology.orchestra.run.smithi057.stdout:Schedule removed for path /snap_test_dir1 2023-10-19T14:01:16.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.868+0000 7f9929500640 1 -- 172.21.15.57:0/3663968043 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7f990406afb0 msgr2=0x7f990406d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:16.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.868+0000 7f9929500640 1 --2- 172.21.15.57:0/3663968043 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7f990406afb0 0x7f990406d470 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9914009830 tx=0x7f9914004730 comp rx=0 tx=0).stop 2023-10-19T14:01:16.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.868+0000 7f9929500640 1 -- 172.21.15.57:0/3663968043 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f99240abc10 msgr2=0x7f99240aa8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:16.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.868+0000 7f9929500640 1 --2- 172.21.15.57:0/3663968043 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f99240abc10 0x7f99240aa8c0 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7f990c00c430 tx=0x7f990c00c8f0 comp rx=0 tx=0).stop 2023-10-19T14:01:16.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.868+0000 7f9929500640 1 -- 172.21.15.57:0/3663968043 shutdown_connections 2023-10-19T14:01:16.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.868+0000 7f9929500640 1 --2- 172.21.15.57:0/3663968043 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f99240c96b0 0x7f99240aae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:16.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.868+0000 7f9929500640 1 --2- 172.21.15.57:0/3663968043 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f99240abc10 0x7f99240aa8c0 unknown :-1 s=CLOSED pgs=1801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:16.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.868+0000 7f9929500640 1 --2- 172.21.15.57:0/3663968043 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7f990406afb0 0x7f990406d470 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:16.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.868+0000 7f9929500640 1 --2- 172.21.15.57:0/3663968043 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f992406af00 0x7f99241f41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:16.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.868+0000 7f9929500640 1 -- 172.21.15.57:0/3663968043 >> 172.21.15.57:0/3663968043 conn(0x7f9924069930 msgr2=0x7f9924069d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:16.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.868+0000 7f9929500640 1 -- 172.21.15.57:0/3663968043 shutdown_connections 2023-10-19T14:01:16.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:16.868+0000 7f9929500640 1 -- 172.21.15.57:0/3663968043 wait complete. 2023-10-19T14:01:16.881 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1 2023-10-19T14:01:16.881 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:01:16.882 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T14:01:16.903 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_07_00_UTC 2023-10-19T14:01:16.903 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_08_00_UTC 2023-10-19T14:01:16.903 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_09_00_UTC 2023-10-19T14:01:16.903 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_10_00_UTC 2023-10-19T14:01:16.904 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_11_00_UTC 2023-10-19T14:01:16.904 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_12_00_UTC 2023-10-19T14:01:16.904 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_13_00_UTC 2023-10-19T14:01:16.904 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_14_00_UTC 2023-10-19T14:01:16.904 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_15_00_UTC 2023-10-19T14:01:16.904 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_16_00_UTC 2023-10-19T14:01:16.905 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_17_00_UTC 2023-10-19T14:01:16.905 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_18_00_UTC 2023-10-19T14:01:16.905 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_19_00_UTC 2023-10-19T14:01:16.905 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_20_00_UTC 2023-10-19T14:01:16.906 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_21_00_UTC 2023-10-19T14:01:16.906 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_22_00_UTC 2023-10-19T14:01:16.906 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_23_00_UTC 2023-10-19T14:01:16.906 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_24_00_UTC 2023-10-19T14:01:16.906 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_25_00_UTC 2023-10-19T14:01:16.907 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_26_00_UTC 2023-10-19T14:01:16.907 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_27_00_UTC 2023-10-19T14:01:16.907 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_28_00_UTC 2023-10-19T14:01:16.907 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_29_00_UTC 2023-10-19T14:01:16.907 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_30_00_UTC 2023-10-19T14:01:16.908 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_31_00_UTC 2023-10-19T14:01:16.908 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_32_00_UTC 2023-10-19T14:01:16.908 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_33_00_UTC 2023-10-19T14:01:16.908 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_34_00_UTC 2023-10-19T14:01:16.908 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_35_00_UTC 2023-10-19T14:01:16.909 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_36_00_UTC 2023-10-19T14:01:16.909 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_37_00_UTC 2023-10-19T14:01:16.909 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_38_00_UTC 2023-10-19T14:01:16.909 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_39_00_UTC 2023-10-19T14:01:16.909 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_40_00_UTC 2023-10-19T14:01:16.909 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_41_00_UTC 2023-10-19T14:01:16.910 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_42_00_UTC 2023-10-19T14:01:16.910 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_43_00_UTC 2023-10-19T14:01:16.910 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_44_00_UTC 2023-10-19T14:01:16.910 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_45_00_UTC 2023-10-19T14:01:16.910 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_46_00_UTC 2023-10-19T14:01:16.911 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_47_00_UTC 2023-10-19T14:01:16.911 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_48_00_UTC 2023-10-19T14:01:16.911 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_49_00_UTC 2023-10-19T14:01:16.911 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_50_00_UTC 2023-10-19T14:01:16.911 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_51_00_UTC 2023-10-19T14:01:16.911 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_52_00_UTC 2023-10-19T14:01:16.912 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_53_00_UTC 2023-10-19T14:01:16.912 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_54_00_UTC 2023-10-19T14:01:16.912 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_55_00_UTC 2023-10-19T14:01:16.912 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_56_00_UTC 2023-10-19T14:01:16.912 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_57_00_UTC 2023-10-19T14:01:16.912 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_58_00_UTC 2023-10-19T14:01:16.913 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-13_59_00_UTC 2023-10-19T14:01:16.913 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_00_00_UTC 2023-10-19T14:01:16.913 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_01_00_UTC 2023-10-19T14:01:16.913 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-19-13_07_00_UTC 2023-10-19T14:01:16.914 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:01:16.914 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-19-13_07_00_UTC) 2023-10-19T14:01:16.965 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-19-13_08_00_UTC 2023-10-19T14:01:16.966 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:01:16.966 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-19-13_08_00_UTC) 2023-10-19T14:01:17.028 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-19-13_09_00_UTC 2023-10-19T14:01:17.028 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:01:17.029 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-19-13_09_00_UTC) 2023-10-19T14:01:17.090 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-19-13_10_00_UTC 2023-10-19T14:01:17.091 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:01:17.091 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-19-13_10_00_UTC) 2023-10-19T14:01:17.152 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-19-13_11_00_UTC 2023-10-19T14:01:17.153 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:01:17.153 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-19-13_11_00_UTC) 2023-10-19T14:01:17.214 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-19-13_12_00_UTC 2023-10-19T14:01:17.214 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:01:17.215 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-19-13_12_00_UTC) 2023-10-19T14:01:17.276 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-19-13_13_00_UTC 2023-10-19T14:01:17.277 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:01:17.277 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-19-13_13_00_UTC) 2023-10-19T14:01:17.339 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-19-13_14_00_UTC 2023-10-19T14:01:17.340 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:01:17.340 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-19-13_14_00_UTC) 2023-10-19T14:01:17.402 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-19-13_15_00_UTC 2023-10-19T14:01:17.402 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:01:17.403 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-19-13_15_00_UTC) 2023-10-19T14:01:17.465 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-19-13_16_00_UTC 2023-10-19T14:01:17.465 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:01:17.465 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-19-13_16_00_UTC) 2023-10-19T14:01:17.527 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-19-13_17_00_UTC 2023-10-19T14:01:17.528 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:01:17.528 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-19-13_17_00_UTC) 2023-10-19T14:01:17.589 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-19-13_18_00_UTC 2023-10-19T14:01:17.590 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:01:17.591 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-19-13_18_00_UTC) 2023-10-19T14:01:17.652 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-19-13_19_00_UTC 2023-10-19T14:01:17.652 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:01:17.653 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-19-13_19_00_UTC) 2023-10-19T14:01:17.714 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-19-13_20_00_UTC 2023-10-19T14:01:17.714 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:01:17.715 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-19-13_20_00_UTC) 2023-10-19T14:01:17.776 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-19-13_21_00_UTC 2023-10-19T14:01:17.776 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:01:17.777 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-19-13_21_00_UTC) 2023-10-19T14:01:17.838 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-19-13_22_00_UTC 2023-10-19T14:01:17.838 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:01:17.839 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-19-13_22_00_UTC) 2023-10-19T14:01:17.900 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-19-13_23_00_UTC 2023-10-19T14:01:17.900 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:01:17.901 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-19-13_23_00_UTC) 2023-10-19T14:01:17.962 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-19-13_24_00_UTC 2023-10-19T14:01:17.963 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:01:17.963 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-19-13_24_00_UTC) 2023-10-19T14:01:18.024 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-19-13_25_00_UTC 2023-10-19T14:01:18.024 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:01:18.025 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-19-13_25_00_UTC) 2023-10-19T14:01:18.086 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-19-13_26_00_UTC 2023-10-19T14:01:18.087 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:01:18.087 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-19-13_26_00_UTC) 2023-10-19T14:01:18.148 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-19-13_27_00_UTC 2023-10-19T14:01:18.149 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:01:18.149 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-19-13_27_00_UTC) 2023-10-19T14:01:18.209 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-19-13_28_00_UTC 2023-10-19T14:01:18.210 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:01:18.210 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-19-13_28_00_UTC) 2023-10-19T14:01:18.272 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-19-13_29_00_UTC 2023-10-19T14:01:18.272 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:01:18.273 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-19-13_29_00_UTC) 2023-10-19T14:01:18.334 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-19-13_30_00_UTC 2023-10-19T14:01:18.335 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:01:18.335 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-19-13_30_00_UTC) 2023-10-19T14:01:18.396 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-19-13_31_00_UTC 2023-10-19T14:01:18.396 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:01:18.397 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-19-13_31_00_UTC) 2023-10-19T14:01:18.458 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-19-13_32_00_UTC 2023-10-19T14:01:18.459 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:01:18.459 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-19-13_32_00_UTC) 2023-10-19T14:01:18.520 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-19-13_33_00_UTC 2023-10-19T14:01:18.520 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:01:18.521 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-19-13_33_00_UTC) 2023-10-19T14:01:18.582 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-19-13_34_00_UTC 2023-10-19T14:01:18.583 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:01:18.583 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-19-13_34_00_UTC) 2023-10-19T14:01:18.644 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-19-13_35_00_UTC 2023-10-19T14:01:18.644 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:01:18.645 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-19-13_35_00_UTC) 2023-10-19T14:01:18.706 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-19-13_36_00_UTC 2023-10-19T14:01:18.706 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:01:18.707 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-19-13_36_00_UTC) 2023-10-19T14:01:18.768 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-19-13_37_00_UTC 2023-10-19T14:01:18.768 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:01:18.769 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-19-13_37_00_UTC) 2023-10-19T14:01:18.830 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-19-13_38_00_UTC 2023-10-19T14:01:18.831 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:01:18.831 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-19-13_38_00_UTC) 2023-10-19T14:01:18.892 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-19-13_39_00_UTC 2023-10-19T14:01:18.893 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:01:18.893 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-19-13_39_00_UTC) 2023-10-19T14:01:18.954 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-19-13_40_00_UTC 2023-10-19T14:01:18.955 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:01:18.955 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-19-13_40_00_UTC) 2023-10-19T14:01:19.016 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-19-13_41_00_UTC 2023-10-19T14:01:19.016 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:01:19.017 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-19-13_41_00_UTC) 2023-10-19T14:01:19.082 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-19-13_42_00_UTC 2023-10-19T14:01:19.083 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:01:19.083 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-19-13_42_00_UTC) 2023-10-19T14:01:19.144 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-19-13_43_00_UTC 2023-10-19T14:01:19.144 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:01:19.145 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-19-13_43_00_UTC) 2023-10-19T14:01:19.206 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-19-13_44_00_UTC 2023-10-19T14:01:19.206 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:01:19.207 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-19-13_44_00_UTC) 2023-10-19T14:01:19.269 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-19-13_45_00_UTC 2023-10-19T14:01:19.270 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:01:19.270 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-19-13_45_00_UTC) 2023-10-19T14:01:19.332 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-19-13_46_00_UTC 2023-10-19T14:01:19.333 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:01:19.334 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-19-13_46_00_UTC) 2023-10-19T14:01:19.394 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-19-13_47_00_UTC 2023-10-19T14:01:19.395 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:01:19.395 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-19-13_47_00_UTC) 2023-10-19T14:01:19.458 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-19-13_48_00_UTC 2023-10-19T14:01:19.459 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:01:19.459 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-19-13_48_00_UTC) 2023-10-19T14:01:19.521 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-19-13_49_00_UTC 2023-10-19T14:01:19.522 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:01:19.522 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-19-13_49_00_UTC) 2023-10-19T14:01:19.588 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-19-13_50_00_UTC 2023-10-19T14:01:19.588 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:01:19.589 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-19-13_50_00_UTC) 2023-10-19T14:01:19.650 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-19-13_51_00_UTC 2023-10-19T14:01:19.651 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:01:19.651 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-19-13_51_00_UTC) 2023-10-19T14:01:19.713 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-19-13_52_00_UTC 2023-10-19T14:01:19.713 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:01:19.714 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-19-13_52_00_UTC) 2023-10-19T14:01:19.775 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-19-13_53_00_UTC 2023-10-19T14:01:19.775 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:01:19.776 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-19-13_53_00_UTC) 2023-10-19T14:01:19.837 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-19-13_54_00_UTC 2023-10-19T14:01:19.838 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:01:19.838 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-19-13_54_00_UTC) 2023-10-19T14:01:19.899 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-19-13_55_00_UTC 2023-10-19T14:01:19.899 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:01:19.900 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-19-13_55_00_UTC) 2023-10-19T14:01:19.962 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-19-13_56_00_UTC 2023-10-19T14:01:19.962 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:01:19.963 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-19-13_56_00_UTC) 2023-10-19T14:01:20.025 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-19-13_57_00_UTC 2023-10-19T14:01:20.026 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:01:20.026 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-19-13_57_00_UTC) 2023-10-19T14:01:20.119 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-19-13_58_00_UTC 2023-10-19T14:01:20.120 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:01:20.120 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-19-13_58_00_UTC) 2023-10-19T14:01:20.171 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-19-13_59_00_UTC 2023-10-19T14:01:20.171 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:01:20.172 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-19-13_59_00_UTC) 2023-10-19T14:01:20.211 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-19-14_00_00_UTC 2023-10-19T14:01:20.212 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:01:20.212 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-19-14_00_00_UTC) 2023-10-19T14:01:20.275 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-19-14_01_00_UTC 2023-10-19T14:01:20.275 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:01:20.275 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-19-14_01_00_UTC) 2023-10-19T14:01:20.337 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:01:20.338 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2023-10-19T14:01:20.396 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-19T14:01:20.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:20.551+0000 7f22b3173640 1 -- 172.21.15.57:0/489608957 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f22ac06f5b0 msgr2=0x7f22ac070390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:20.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:20.551+0000 7f22b3173640 1 --2- 172.21.15.57:0/489608957 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f22ac06f5b0 0x7f22ac070390 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7f22ac0af650 tx=0x7f22a402c250 comp rx=0 tx=0).stop 2023-10-19T14:01:20.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:20.551+0000 7f22b3173640 1 -- 172.21.15.57:0/489608957 shutdown_connections 2023-10-19T14:01:20.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:20.552+0000 7f22b3173640 1 --2- 172.21.15.57:0/489608957 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f22ac06f5b0 0x7f22ac070390 unknown :-1 s=CLOSED pgs=1802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:20.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:20.552+0000 7f22b3173640 1 --2- 172.21.15.57:0/489608957 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f22ac15f720 0x7f22ac06fe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:20.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:20.552+0000 7f22b3173640 1 --2- 172.21.15.57:0/489608957 >> v2:172.21.15.179:3300/0 conn(0x7f22ac0abf90 0x7f22ac15bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:20.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:20.552+0000 7f22b3173640 1 -- 172.21.15.57:0/489608957 >> 172.21.15.57:0/489608957 conn(0x7f22ac0ae720 msgr2=0x7f22ac0aeb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:20.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:20.552+0000 7f22b3173640 1 -- 172.21.15.57:0/489608957 shutdown_connections 2023-10-19T14:01:20.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:20.552+0000 7f22b3173640 1 -- 172.21.15.57:0/489608957 wait complete. 2023-10-19T14:01:20.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:20.553+0000 7f22b3173640 1 Processor -- start 2023-10-19T14:01:20.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:20.554+0000 7f22b3173640 1 -- start start 2023-10-19T14:01:20.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:20.555+0000 7f22b3173640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f22ac06f5b0 0x7f22ac06bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:20.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:20.555+0000 7f22b3173640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f22ac0abf90 0x7f22ac1f1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:20.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:20.555+0000 7f22b3173640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f22ac15f720 0x7f22ac06b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:20.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:20.555+0000 7f22b3173640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22ac1f16b0 con 0x7f22ac15f720 2023-10-19T14:01:20.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:20.555+0000 7f22b3173640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22ac148bc0 con 0x7f22ac06f5b0 2023-10-19T14:01:20.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:20.555+0000 7f22b3173640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22ac06bd70 con 0x7f22ac0abf90 2023-10-19T14:01:20.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:20.555+0000 7f22b1710640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f22ac15f720 0x7f22ac06b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:20.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:20.555+0000 7f22b1710640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f22ac15f720 0x7f22ac06b830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40632/0 (socket says 172.21.15.57:40632) 2023-10-19T14:01:20.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:20.555+0000 7f22b1710640 1 -- 172.21.15.57:0/268617263 learned_addr learned my addr 172.21.15.57:0/268617263 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:01:20.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:20.555+0000 7f22a3fff640 1 --2- 172.21.15.57:0/268617263 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f22ac0abf90 0x7f22ac1f1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:20.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:20.556+0000 7f22b0f0f640 1 --2- 172.21.15.57:0/268617263 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f22ac06f5b0 0x7f22ac06bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:20.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:20.556+0000 7f22b1710640 1 -- 172.21.15.57:0/268617263 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f22ac0abf90 msgr2=0x7f22ac1f1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:20.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:20.556+0000 7f22b1710640 1 --2- 172.21.15.57:0/268617263 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f22ac0abf90 0x7f22ac1f1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:20.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:20.556+0000 7f22b1710640 1 -- 172.21.15.57:0/268617263 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f22ac06f5b0 msgr2=0x7f22ac06bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:20.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:20.556+0000 7f22b1710640 1 --2- 172.21.15.57:0/268617263 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f22ac06f5b0 0x7f22ac06bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:20.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:20.556+0000 7f22b1710640 1 -- 172.21.15.57:0/268617263 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22a4009900 con 0x7f22ac15f720 2023-10-19T14:01:20.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:20.556+0000 7f22b1710640 1 --2- 172.21.15.57:0/268617263 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f22ac15f720 0x7f22ac06b830 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7f22a4037040 tx=0x7f22a4035db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:20.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:20.557+0000 7f22a17fa640 1 -- 172.21.15.57:0/268617263 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22a4034440 con 0x7f22ac15f720 2023-10-19T14:01:20.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:20.557+0000 7f22a17fa640 1 -- 172.21.15.57:0/268617263 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f22a4003e60 con 0x7f22ac15f720 2023-10-19T14:01:20.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:20.557+0000 7f22a17fa640 1 -- 172.21.15.57:0/268617263 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22a4004490 con 0x7f22ac15f720 2023-10-19T14:01:20.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:20.557+0000 7f22b3173640 1 -- 172.21.15.57:0/268617263 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22ac06c5f0 con 0x7f22ac15f720 2023-10-19T14:01:20.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:20.557+0000 7f22b3173640 1 -- 172.21.15.57:0/268617263 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22ac06cbb0 con 0x7f22ac15f720 2023-10-19T14:01:20.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:20.558+0000 7f22b3173640 1 -- 172.21.15.57:0/268617263 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22ac0cb4a0 con 0x7f22ac15f720 2023-10-19T14:01:20.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:20.560+0000 7f22a17fa640 1 -- 172.21.15.57:0/268617263 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 86) v1 ==== 89360+0+0 (secure 0 0 0) 0x7f22a403b040 con 0x7f22ac15f720 2023-10-19T14:01:20.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:20.560+0000 7f22a17fa640 1 --2- 172.21.15.57:0/268617263 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7f229806b080 0x7f229806d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:20.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:20.560+0000 7f22a17fa640 1 -- 172.21.15.57:0/268617263 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6489+0+0 (secure 0 0 0) 0x7f22a40ac5d0 con 0x7f22ac15f720 2023-10-19T14:01:20.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:20.561+0000 7f22b0f0f640 1 --2- 172.21.15.57:0/268617263 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7f229806b080 0x7f229806d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:20.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:20.562+0000 7f22b0f0f640 1 --2- 172.21.15.57:0/268617263 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7f229806b080 0x7f229806d540 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f22a8005e00 tx=0x7f22a8005c50 comp rx=0 tx=0).ready entity=mgr.8818 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:20.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:20.567+0000 7f22a17fa640 1 -- 172.21.15.57:0/268617263 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f22a4077240 con 0x7f22ac15f720 2023-10-19T14:01:20.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:20.840+0000 7f22b3173640 1 -- 172.21.15.57:0/268617263 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f22ac0690e0 con 0x7f22ac15f720 2023-10-19T14:01:21.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.057+0000 7f22a17fa640 1 -- 172.21.15.57:0/268617263 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v87) v1 ==== 93+0+0 (secure 0 0 0) 0x7f22a407a860 con 0x7f22ac15f720 2023-10-19T14:01:21.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.060+0000 7f22b3173640 1 -- 172.21.15.57:0/268617263 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7f229806b080 msgr2=0x7f229806d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:21.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.060+0000 7f22b3173640 1 --2- 172.21.15.57:0/268617263 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7f229806b080 0x7f229806d540 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f22a8005e00 tx=0x7f22a8005c50 comp rx=0 tx=0).stop 2023-10-19T14:01:21.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.060+0000 7f22b3173640 1 -- 172.21.15.57:0/268617263 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f22ac15f720 msgr2=0x7f22ac06b830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:21.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.060+0000 7f22b3173640 1 --2- 172.21.15.57:0/268617263 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f22ac15f720 0x7f22ac06b830 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7f22a4037040 tx=0x7f22a4035db0 comp rx=0 tx=0).stop 2023-10-19T14:01:21.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.061+0000 7f22b3173640 1 -- 172.21.15.57:0/268617263 shutdown_connections 2023-10-19T14:01:21.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.061+0000 7f22b3173640 1 --2- 172.21.15.57:0/268617263 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f22ac15f720 0x7f22ac06b830 unknown :-1 s=CLOSED pgs=1803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:21.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.061+0000 7f22b3173640 1 --2- 172.21.15.57:0/268617263 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7f229806b080 0x7f229806d540 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:21.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.061+0000 7f22b3173640 1 --2- 172.21.15.57:0/268617263 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f22ac0abf90 0x7f22ac1f1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:21.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.061+0000 7f22b3173640 1 --2- 172.21.15.57:0/268617263 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f22ac06f5b0 0x7f22ac06bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:21.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.061+0000 7f22b3173640 1 -- 172.21.15.57:0/268617263 >> 172.21.15.57:0/268617263 conn(0x7f22ac0ae720 msgr2=0x7f22ac15dd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:21.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.063+0000 7f22b3173640 1 -- 172.21.15.57:0/268617263 shutdown_connections 2023-10-19T14:01:21.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.063+0000 7f22b3173640 1 -- 172.21.15.57:0/268617263 wait complete. 2023-10-19T14:01:21.084 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-19T14:01:21.084 DEBUG:teuthology.orchestra.run.smithi057:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-19T14:01:21.166 DEBUG:teuthology.orchestra.run.smithi179:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-19T14:01:21.210 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-19T14:01:21.210 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T14:01:21.211 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T14:01:21.239 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T14:01:21.392 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.389+0000 7f3ef8a44640 1 -- 172.21.15.57:0/4123621070 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3ef40cf340 msgr2=0x7f3ef40d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:21.392 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.389+0000 7f3ef8a44640 1 --2- 172.21.15.57:0/4123621070 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3ef40cf340 0x7f3ef40d1910 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7f3ee8005fd0 tx=0x7f3ee802bd90 comp rx=0 tx=0).stop 2023-10-19T14:01:21.392 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.389+0000 7f3ef8a44640 1 -- 172.21.15.57:0/4123621070 shutdown_connections 2023-10-19T14:01:21.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.389+0000 7f3ef8a44640 1 --2- 172.21.15.57:0/4123621070 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3ef415b2a0 0x7f3ef40cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:21.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.389+0000 7f3ef8a44640 1 --2- 172.21.15.57:0/4123621070 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3ef40cf340 0x7f3ef40d1910 unknown :-1 s=CLOSED pgs=1808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:21.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.389+0000 7f3ef8a44640 1 -- 172.21.15.57:0/4123621070 >> 172.21.15.57:0/4123621070 conn(0x7f3ef40d0d60 msgr2=0x7f3ef40d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:21.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.389+0000 7f3ef8a44640 1 -- 172.21.15.57:0/4123621070 shutdown_connections 2023-10-19T14:01:21.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.389+0000 7f3ef8a44640 1 -- 172.21.15.57:0/4123621070 wait complete. 2023-10-19T14:01:21.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.390+0000 7f3ef8a44640 1 Processor -- start 2023-10-19T14:01:21.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.391+0000 7f3ef8a44640 1 -- start start 2023-10-19T14:01:21.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.391+0000 7f3ef8a44640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3ef40ceb40 0x7f3ef40b5510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:21.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.391+0000 7f3ef8a44640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3ef40cf340 0x7f3ef4158d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:21.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.391+0000 7f3ef8a44640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3ef415b2a0 0x7f3ef4157f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:21.397 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.391+0000 7f3ef8a44640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ef415c3b0 con 0x7f3ef415b2a0 2023-10-19T14:01:21.397 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.391+0000 7f3ef8a44640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ef415b770 con 0x7f3ef40cf340 2023-10-19T14:01:21.397 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.391+0000 7f3ef8a44640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ef415ba10 con 0x7f3ef40ceb40 2023-10-19T14:01:21.397 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.391+0000 7f3ef2d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3ef415b2a0 0x7f3ef4157f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:21.397 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.391+0000 7f3ef2d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3ef415b2a0 0x7f3ef4157f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40668/0 (socket says 172.21.15.57:40668) 2023-10-19T14:01:21.398 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.391+0000 7f3ef2d9d640 1 -- 172.21.15.57:0/4167754013 learned_addr learned my addr 172.21.15.57:0/4167754013 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:01:21.398 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.391+0000 7f3ef259c640 1 --2- 172.21.15.57:0/4167754013 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3ef40ceb40 0x7f3ef40b5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:21.398 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.392+0000 7f3ef1d9b640 1 --2- 172.21.15.57:0/4167754013 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3ef40cf340 0x7f3ef4158d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:21.398 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.392+0000 7f3ef2d9d640 1 -- 172.21.15.57:0/4167754013 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3ef40ceb40 msgr2=0x7f3ef40b5510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:21.398 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.392+0000 7f3ef2d9d640 1 --2- 172.21.15.57:0/4167754013 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3ef40ceb40 0x7f3ef40b5510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:21.399 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.392+0000 7f3ef2d9d640 1 -- 172.21.15.57:0/4167754013 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3ef40cf340 msgr2=0x7f3ef4158d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:21.399 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.392+0000 7f3ef2d9d640 1 --2- 172.21.15.57:0/4167754013 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3ef40cf340 0x7f3ef4158d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:21.399 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.392+0000 7f3ef2d9d640 1 -- 172.21.15.57:0/4167754013 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ee8009560 con 0x7f3ef415b2a0 2023-10-19T14:01:21.399 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.392+0000 7f3ef259c640 1 --2- 172.21.15.57:0/4167754013 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3ef40ceb40 0x7f3ef40b5510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T14:01:21.399 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.392+0000 7f3ef2d9d640 1 --2- 172.21.15.57:0/4167754013 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3ef415b2a0 0x7f3ef4157f70 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7f3eec00c710 tx=0x7f3eec00cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:21.400 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.392+0000 7f3ee2ffd640 1 -- 172.21.15.57:0/4167754013 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eec006d50 con 0x7f3ef415b2a0 2023-10-19T14:01:21.400 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.392+0000 7f3ee2ffd640 1 -- 172.21.15.57:0/4167754013 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f3eec00ce20 con 0x7f3ef415b2a0 2023-10-19T14:01:21.400 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.392+0000 7f3ef8a44640 1 -- 172.21.15.57:0/4167754013 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ef415bcf0 con 0x7f3ef415b2a0 2023-10-19T14:01:21.400 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.392+0000 7f3ee2ffd640 1 -- 172.21.15.57:0/4167754013 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eec003bd0 con 0x7f3ef415b2a0 2023-10-19T14:01:21.400 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.392+0000 7f3ef8a44640 1 -- 172.21.15.57:0/4167754013 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ef40b5a50 con 0x7f3ef415b2a0 2023-10-19T14:01:21.401 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.393+0000 7f3ef8a44640 1 -- 172.21.15.57:0/4167754013 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ef415c4f0 con 0x7f3ef415b2a0 2023-10-19T14:01:21.401 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.394+0000 7f3ee2ffd640 1 -- 172.21.15.57:0/4167754013 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 87) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f3eec003d30 con 0x7f3ef415b2a0 2023-10-19T14:01:21.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.394+0000 7f3ee2ffd640 1 --2- 172.21.15.57:0/4167754013 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7f3ed406af60 0x7f3ed406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:21.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.394+0000 7f3ef259c640 1 -- 172.21.15.57:0/4167754013 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7f3ed406af60 msgr2=0x7f3ed406d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/4251935070 2023-10-19T14:01:21.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.394+0000 7f3ee2ffd640 1 -- 172.21.15.57:0/4167754013 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 6489+0+0 (secure 0 0 0) 0x7f3eec0897a0 con 0x7f3ef415b2a0 2023-10-19T14:01:21.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.394+0000 7f3ef259c640 1 --2- 172.21.15.57:0/4167754013 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7f3ed406af60 0x7f3ed406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T14:01:21.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.399+0000 7f3ee2ffd640 1 -- 172.21.15.57:0/4167754013 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3eec0544f0 con 0x7f3ef415b2a0 2023-10-19T14:01:21.496 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:21.494+0000 7f4e31232040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-19T14:01:21.519 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:01:21.517+0000 7f4d21f7a040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-19T14:01:21.540 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:21.538+0000 7f4e31232040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-19T14:01:21.567 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:01:21.564+0000 7f4d21f7a040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-19T14:01:21.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.576+0000 7f3ef8a44640 1 -- 172.21.15.57:0/4167754013 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3ef40690e0 con 0x7f3ef415b2a0 2023-10-19T14:01:21.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.577+0000 7f3ee2ffd640 1 -- 172.21.15.57:0/4167754013 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v162) v1 ==== 74+0+17458 (secure 0 0 0) 0x7f3eec057b10 con 0x7f3ef415b2a0 2023-10-19T14:01:21.580 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:01:21.580 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":162,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T14:01:21.052505+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2023-10-19T13:04:56.433332+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"156","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2023-10-19T13:04:57.112749+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"162","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":107,"snap_epoch":162,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.179:0/744328422":"2023-10-20T13:05:09.216573+0000","172.21.15.179:0/1909872102":"2023-10-20T13:05:09.216573+0000","172.21.15.179:0/1880188133":"2023-10-20T13:05:09.216573+0000","172.21.15.179:0/40128169":"2023-10-20T13:05:09.216573+0000","172.21.15.179:6833/2581690007":"2023-10-20T13:05:09.216573+0000","172.21.15.179:0/3857987280":"2023-10-20T13:05:09.216573+0000","172.21.15.179:6832/2581690007":"2023-10-20T13:05:09.216573+0000","172.21.15.179:0/57720407":"2023-10-20T13:05:09.216573+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":20,"snaps":[{"begin":3,"length":49},{"begin":59,"length":49}]}],"new_removed_snaps":[{"pool":20,"snaps":[{"begin":38,"length":14},{"begin":94,"length":14}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T14:01:21.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.579+0000 7f3ef8a44640 1 -- 172.21.15.57:0/4167754013 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7f3ed406af60 msgr2=0x7f3ed406d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T14:01:21.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.579+0000 7f3ef8a44640 1 --2- 172.21.15.57:0/4167754013 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7f3ed406af60 0x7f3ed406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:21.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.579+0000 7f3ef8a44640 1 -- 172.21.15.57:0/4167754013 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3ef415b2a0 msgr2=0x7f3ef4157f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:21.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.579+0000 7f3ef8a44640 1 --2- 172.21.15.57:0/4167754013 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3ef415b2a0 0x7f3ef4157f70 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7f3eec00c710 tx=0x7f3eec00cbd0 comp rx=0 tx=0).stop 2023-10-19T14:01:21.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.579+0000 7f3ef8a44640 1 -- 172.21.15.57:0/4167754013 shutdown_connections 2023-10-19T14:01:21.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.579+0000 7f3ef8a44640 1 --2- 172.21.15.57:0/4167754013 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3ef415b2a0 0x7f3ef4157f70 unknown :-1 s=CLOSED pgs=1809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:21.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.579+0000 7f3ef8a44640 1 --2- 172.21.15.57:0/4167754013 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3ef40cf340 0x7f3ef4158d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:21.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.579+0000 7f3ef8a44640 1 --2- 172.21.15.57:0/4167754013 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7f3ed406af60 0x7f3ed406d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:21.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.579+0000 7f3ef8a44640 1 --2- 172.21.15.57:0/4167754013 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3ef40ceb40 0x7f3ef40b5510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:21.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.579+0000 7f3ef8a44640 1 -- 172.21.15.57:0/4167754013 >> 172.21.15.57:0/4167754013 conn(0x7f3ef40d0d60 msgr2=0x7f3ef415a2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:21.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.579+0000 7f3ef8a44640 1 -- 172.21.15.57:0/4167754013 shutdown_connections 2023-10-19T14:01:21.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.579+0000 7f3ef8a44640 1 -- 172.21.15.57:0/4167754013 wait complete. 2023-10-19T14:01:21.593 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-19T12:44:07.371186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 19, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-19T13:04:56.433332+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '156', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 20, 'pool_name': 'cephfs_data', 'create_time': '2023-10-19T13:04:57.112749+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '162', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 107, 'snap_epoch': 162, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-19T14:01:21.594 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-19T14:01:21.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.705+0000 7fba6418e640 1 -- 172.21.15.57:0/1969281505 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fba5c15ccf0 msgr2=0x7fba5c15d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:21.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.705+0000 7fba6418e640 1 --2- 172.21.15.57:0/1969281505 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fba5c15ccf0 0x7fba5c15d160 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7fba54005fd0 tx=0x7fba5402c240 comp rx=0 tx=0).stop 2023-10-19T14:01:21.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.705+0000 7fba6418e640 1 -- 172.21.15.57:0/1969281505 shutdown_connections 2023-10-19T14:01:21.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.705+0000 7fba6418e640 1 --2- 172.21.15.57:0/1969281505 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fba5c15ccf0 0x7fba5c15d160 unknown :-1 s=CLOSED pgs=1810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:21.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.705+0000 7fba6418e640 1 --2- 172.21.15.57:0/1969281505 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fba5c0ce240 0x7fba5c06d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:21.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.705+0000 7fba6418e640 1 -- 172.21.15.57:0/1969281505 >> 172.21.15.57:0/1969281505 conn(0x7fba5c0b0670 msgr2=0x7fba5c0b0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:21.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.705+0000 7fba6418e640 1 -- 172.21.15.57:0/1969281505 shutdown_connections 2023-10-19T14:01:21.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.705+0000 7fba6418e640 1 -- 172.21.15.57:0/1969281505 wait complete. 2023-10-19T14:01:21.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.706+0000 7fba6418e640 1 Processor -- start 2023-10-19T14:01:21.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.706+0000 7fba6418e640 1 -- start start 2023-10-19T14:01:21.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.707+0000 7fba6418e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fba5c0ce240 0x7fba5c1f35d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:21.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.707+0000 7fba6418e640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fba5c159060 0x7fba5c150fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:21.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.707+0000 7fba6418e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fba5c15ccf0 0x7fba5c0b1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:21.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.707+0000 7fba6418e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba5c15a1c0 con 0x7fba5c15ccf0 2023-10-19T14:01:21.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.707+0000 7fba6418e640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba5c0c4d80 con 0x7fba5c0ce240 2023-10-19T14:01:21.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.707+0000 7fba6418e640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba5c0c47c0 con 0x7fba5c159060 2023-10-19T14:01:21.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.707+0000 7fba6272b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fba5c15ccf0 0x7fba5c0b1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:21.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.707+0000 7fba6272b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fba5c15ccf0 0x7fba5c0b1760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40682/0 (socket says 172.21.15.57:40682) 2023-10-19T14:01:21.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.707+0000 7fba6272b640 1 -- 172.21.15.57:0/3793715405 learned_addr learned my addr 172.21.15.57:0/3793715405 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:01:21.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.707+0000 7fba61f2a640 1 --2- 172.21.15.57:0/3793715405 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fba5c0ce240 0x7fba5c1f35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:21.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.707+0000 7fba61729640 1 --2- 172.21.15.57:0/3793715405 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fba5c159060 0x7fba5c150fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:21.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.707+0000 7fba6272b640 1 -- 172.21.15.57:0/3793715405 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fba5c159060 msgr2=0x7fba5c150fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:21.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.707+0000 7fba6272b640 1 --2- 172.21.15.57:0/3793715405 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fba5c159060 0x7fba5c150fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:21.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.707+0000 7fba6272b640 1 -- 172.21.15.57:0/3793715405 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fba5c0ce240 msgr2=0x7fba5c1f35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:21.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.707+0000 7fba6272b640 1 --2- 172.21.15.57:0/3793715405 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fba5c0ce240 0x7fba5c1f35d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:21.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.707+0000 7fba6272b640 1 -- 172.21.15.57:0/3793715405 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba54009900 con 0x7fba5c15ccf0 2023-10-19T14:01:21.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.707+0000 7fba61729640 1 --2- 172.21.15.57:0/3793715405 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fba5c159060 0x7fba5c150fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:01:21.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.708+0000 7fba6272b640 1 --2- 172.21.15.57:0/3793715405 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fba5c15ccf0 0x7fba5c0b1760 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7fba54009bf0 tx=0x7fba54035e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:21.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.708+0000 7fba527fc640 1 -- 172.21.15.57:0/3793715405 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba54034440 con 0x7fba5c15ccf0 2023-10-19T14:01:21.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.708+0000 7fba527fc640 1 -- 172.21.15.57:0/3793715405 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fba54003e60 con 0x7fba5c15ccf0 2023-10-19T14:01:21.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.708+0000 7fba6418e640 1 -- 172.21.15.57:0/3793715405 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba5c0c4aa0 con 0x7fba5c15ccf0 2023-10-19T14:01:21.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.708+0000 7fba527fc640 1 -- 172.21.15.57:0/3793715405 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba540044b0 con 0x7fba5c15ccf0 2023-10-19T14:01:21.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.708+0000 7fba6418e640 1 -- 172.21.15.57:0/3793715405 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba5c0c2990 con 0x7fba5c15ccf0 2023-10-19T14:01:21.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.709+0000 7fba6418e640 1 -- 172.21.15.57:0/3793715405 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba5c0c4ef0 con 0x7fba5c15ccf0 2023-10-19T14:01:21.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.709+0000 7fba527fc640 1 -- 172.21.15.57:0/3793715405 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 87) v1 ==== 89343+0+0 (secure 0 0 0) 0x7fba5400a040 con 0x7fba5c15ccf0 2023-10-19T14:01:21.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.710+0000 7fba527fc640 1 --2- 172.21.15.57:0/3793715405 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7fba4006af60 0x7fba4006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:21.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.710+0000 7fba61f2a640 1 -- 172.21.15.57:0/3793715405 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7fba4006af60 msgr2=0x7fba4006d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/4251935070 2023-10-19T14:01:21.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.710+0000 7fba527fc640 1 -- 172.21.15.57:0/3793715405 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 6489+0+0 (secure 0 0 0) 0x7fba540ac940 con 0x7fba5c15ccf0 2023-10-19T14:01:21.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.710+0000 7fba61f2a640 1 --2- 172.21.15.57:0/3793715405 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7fba4006af60 0x7fba4006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T14:01:21.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.714+0000 7fba527fc640 1 -- 172.21.15.57:0/3793715405 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fba540776f0 con 0x7fba5c15ccf0 2023-10-19T14:01:21.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.891+0000 7fba6418e640 1 -- 172.21.15.57:0/3793715405 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fba5c0c2b80 con 0x7fba5c15ccf0 2023-10-19T14:01:21.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.891+0000 7fba527fc640 1 -- 172.21.15.57:0/3793715405 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v162) v1 ==== 93+0+10 (secure 0 0 0) 0x7fba54044020 con 0x7fba5c15ccf0 2023-10-19T14:01:21.894 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-19T14:01:21.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.893+0000 7fba6418e640 1 -- 172.21.15.57:0/3793715405 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7fba4006af60 msgr2=0x7fba4006d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T14:01:21.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.893+0000 7fba6418e640 1 --2- 172.21.15.57:0/3793715405 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7fba4006af60 0x7fba4006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:21.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.893+0000 7fba6418e640 1 -- 172.21.15.57:0/3793715405 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fba5c15ccf0 msgr2=0x7fba5c0b1760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:21.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.893+0000 7fba6418e640 1 --2- 172.21.15.57:0/3793715405 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fba5c15ccf0 0x7fba5c0b1760 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7fba54009bf0 tx=0x7fba54035e30 comp rx=0 tx=0).stop 2023-10-19T14:01:21.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.893+0000 7fba6418e640 1 -- 172.21.15.57:0/3793715405 shutdown_connections 2023-10-19T14:01:21.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.893+0000 7fba6418e640 1 --2- 172.21.15.57:0/3793715405 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fba5c15ccf0 0x7fba5c0b1760 unknown :-1 s=CLOSED pgs=1811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:21.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.893+0000 7fba6418e640 1 --2- 172.21.15.57:0/3793715405 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7fba4006af60 0x7fba4006d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:21.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.893+0000 7fba6418e640 1 --2- 172.21.15.57:0/3793715405 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fba5c159060 0x7fba5c150fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:21.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.893+0000 7fba6418e640 1 --2- 172.21.15.57:0/3793715405 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fba5c0ce240 0x7fba5c1f35d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:21.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.893+0000 7fba6418e640 1 -- 172.21.15.57:0/3793715405 >> 172.21.15.57:0/3793715405 conn(0x7fba5c0b0670 msgr2=0x7fba5c156b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:21.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.893+0000 7fba6418e640 1 -- 172.21.15.57:0/3793715405 shutdown_connections 2023-10-19T14:01:21.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:21.893+0000 7fba6418e640 1 -- 172.21.15.57:0/3793715405 wait complete. 2023-10-19T14:01:21.906 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-19T14:01:21.928 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:21.926+0000 7f4e31232040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-19T14:01:21.972 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:21.969+0000 7f4e31232040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-19T14:01:21.985 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:01:21.983+0000 7f4d21f7a040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-19T14:01:22.013 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:22.011+0000 7f4e31232040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-19T14:01:22.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.017+0000 7fcd914d7640 1 -- 172.21.15.57:0/3186011280 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcd8c15b640 msgr2=0x7fcd8c165fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:22.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.017+0000 7fcd914d7640 1 --2- 172.21.15.57:0/3186011280 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcd8c15b640 0x7fcd8c165fa0 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7fcd84005fd0 tx=0x7fcd8402c370 comp rx=0 tx=0).stop 2023-10-19T14:01:22.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.017+0000 7fcd914d7640 1 -- 172.21.15.57:0/3186011280 shutdown_connections 2023-10-19T14:01:22.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.017+0000 7fcd914d7640 1 --2- 172.21.15.57:0/3186011280 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcd8c15b640 0x7fcd8c165fa0 unknown :-1 s=CLOSED pgs=1813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:22.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.018+0000 7fcd914d7640 1 -- 172.21.15.57:0/3186011280 >> 172.21.15.57:0/3186011280 conn(0x7fcd8c09a030 msgr2=0x7fcd8c0d0d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:22.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.018+0000 7fcd914d7640 1 -- 172.21.15.57:0/3186011280 shutdown_connections 2023-10-19T14:01:22.021 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.018+0000 7fcd914d7640 1 -- 172.21.15.57:0/3186011280 wait complete. 2023-10-19T14:01:22.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.019+0000 7fcd914d7640 1 Processor -- start 2023-10-19T14:01:22.022 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.019+0000 7fcd914d7640 1 -- start start 2023-10-19T14:01:22.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.021+0000 7fcd914d7640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcd8c0cfa20 0x7fcd8c15e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:22.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.021+0000 7fcd914d7640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcd8c15b260 0x7fcd8c154c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:22.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.021+0000 7fcd914d7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcd8c15b640 0x7fcd8c1551b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:22.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.021+0000 7fcd914d7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd8c15e770 con 0x7fcd8c15b640 2023-10-19T14:01:22.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.021+0000 7fcd914d7640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd8c152280 con 0x7fcd8c0cfa20 2023-10-19T14:01:22.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.021+0000 7fcd914d7640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd8c152560 con 0x7fcd8c15b260 2023-10-19T14:01:22.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.021+0000 7fcd8b7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcd8c15b640 0x7fcd8c1551b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:22.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.021+0000 7fcd8b7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcd8c15b640 0x7fcd8c1551b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40694/0 (socket says 172.21.15.57:40694) 2023-10-19T14:01:22.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.021+0000 7fcd8b7fe640 1 -- 172.21.15.57:0/3547052581 learned_addr learned my addr 172.21.15.57:0/3547052581 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:01:22.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.021+0000 7fcd83fff640 1 --2- 172.21.15.57:0/3547052581 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcd8c15b260 0x7fcd8c154c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:22.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.021+0000 7fcd8affd640 1 --2- 172.21.15.57:0/3547052581 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcd8c0cfa20 0x7fcd8c15e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:22.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.021+0000 7fcd8b7fe640 1 -- 172.21.15.57:0/3547052581 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcd8c15b260 msgr2=0x7fcd8c154c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:22.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.021+0000 7fcd8b7fe640 1 --2- 172.21.15.57:0/3547052581 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcd8c15b260 0x7fcd8c154c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:22.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.021+0000 7fcd8b7fe640 1 -- 172.21.15.57:0/3547052581 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcd8c0cfa20 msgr2=0x7fcd8c15e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:22.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.021+0000 7fcd8b7fe640 1 --2- 172.21.15.57:0/3547052581 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcd8c0cfa20 0x7fcd8c15e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:22.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.021+0000 7fcd8b7fe640 1 -- 172.21.15.57:0/3547052581 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd84009900 con 0x7fcd8c15b640 2023-10-19T14:01:22.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.021+0000 7fcd8affd640 1 --2- 172.21.15.57:0/3547052581 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcd8c0cfa20 0x7fcd8c15e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:01:22.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.021+0000 7fcd8b7fe640 1 --2- 172.21.15.57:0/3547052581 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcd8c15b640 0x7fcd8c1551b0 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7fcd84006010 tx=0x7fcd840045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:22.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.022+0000 7fcd837fe640 1 -- 172.21.15.57:0/3547052581 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd84031710 con 0x7fcd8c15b640 2023-10-19T14:01:22.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.022+0000 7fcd914d7640 1 -- 172.21.15.57:0/3547052581 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd8c152840 con 0x7fcd8c15b640 2023-10-19T14:01:22.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.022+0000 7fcd837fe640 1 -- 172.21.15.57:0/3547052581 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fcd84031960 con 0x7fcd8c15b640 2023-10-19T14:01:22.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.022+0000 7fcd837fe640 1 -- 172.21.15.57:0/3547052581 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd84031d30 con 0x7fcd8c15b640 2023-10-19T14:01:22.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.022+0000 7fcd914d7640 1 -- 172.21.15.57:0/3547052581 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd8c152e00 con 0x7fcd8c15b640 2023-10-19T14:01:22.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.023+0000 7fcd914d7640 1 -- 172.21.15.57:0/3547052581 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd8c0675f0 con 0x7fcd8c15b640 2023-10-19T14:01:22.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.023+0000 7fcd837fe640 1 -- 172.21.15.57:0/3547052581 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 87) v1 ==== 89343+0+0 (secure 0 0 0) 0x7fcd8400a040 con 0x7fcd8c15b640 2023-10-19T14:01:22.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.023+0000 7fcd837fe640 1 --2- 172.21.15.57:0/3547052581 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7fcd6c06af60 0x7fcd6c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:22.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.024+0000 7fcd837fe640 1 -- 172.21.15.57:0/3547052581 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 6489+0+0 (secure 0 0 0) 0x7fcd840ad030 con 0x7fcd8c15b640 2023-10-19T14:01:22.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.024+0000 7fcd8affd640 1 -- 172.21.15.57:0/3547052581 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7fcd6c06af60 msgr2=0x7fcd6c06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/4251935070 2023-10-19T14:01:22.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.024+0000 7fcd8affd640 1 --2- 172.21.15.57:0/3547052581 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7fcd6c06af60 0x7fcd6c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T14:01:22.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.032+0000 7fcd837fe640 1 -- 172.21.15.57:0/3547052581 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fcd84077ce0 con 0x7fcd8c15b640 2023-10-19T14:01:22.038 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:01:22.036+0000 7f4d21f7a040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-19T14:01:22.068 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:22.066+0000 7f4e31232040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-19T14:01:22.089 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:01:22.087+0000 7f4d21f7a040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-19T14:01:22.114 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:22.112+0000 7f4e31232040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-19T14:01:22.144 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:01:22.141+0000 7f4d21f7a040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-19T14:01:22.187 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:01:22.185+0000 7f4d21f7a040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-19T14:01:22.211 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:22.208+0000 7f4e31232040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-19T14:01:22.217 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.215+0000 7fcd914d7640 1 -- 172.21.15.57:0/3547052581 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fcd8c165d10 con 0x7fcd8c15b640 2023-10-19T14:01:22.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.215+0000 7fcd837fe640 1 -- 172.21.15.57:0/3547052581 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v163) v1 ==== 104+0+11 (secure 0 0 0) 0x7fcd8407b300 con 0x7fcd8c15b640 2023-10-19T14:01:22.219 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2023-10-19T14:01:22.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.217+0000 7fcd914d7640 1 -- 172.21.15.57:0/3547052581 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7fcd6c06af60 msgr2=0x7fcd6c06d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T14:01:22.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.217+0000 7fcd914d7640 1 --2- 172.21.15.57:0/3547052581 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7fcd6c06af60 0x7fcd6c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:22.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.217+0000 7fcd914d7640 1 -- 172.21.15.57:0/3547052581 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcd8c15b640 msgr2=0x7fcd8c1551b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:22.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.217+0000 7fcd914d7640 1 --2- 172.21.15.57:0/3547052581 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcd8c15b640 0x7fcd8c1551b0 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7fcd84006010 tx=0x7fcd840045e0 comp rx=0 tx=0).stop 2023-10-19T14:01:22.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.217+0000 7fcd914d7640 1 -- 172.21.15.57:0/3547052581 shutdown_connections 2023-10-19T14:01:22.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.217+0000 7fcd914d7640 1 --2- 172.21.15.57:0/3547052581 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcd8c15b640 0x7fcd8c1551b0 unknown :-1 s=CLOSED pgs=1814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:22.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.217+0000 7fcd914d7640 1 --2- 172.21.15.57:0/3547052581 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7fcd6c06af60 0x7fcd6c06d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:22.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.217+0000 7fcd914d7640 1 --2- 172.21.15.57:0/3547052581 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcd8c15b260 0x7fcd8c154c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:22.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.217+0000 7fcd914d7640 1 --2- 172.21.15.57:0/3547052581 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcd8c0cfa20 0x7fcd8c15e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:22.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.217+0000 7fcd914d7640 1 -- 172.21.15.57:0/3547052581 >> 172.21.15.57:0/3547052581 conn(0x7fcd8c09a030 msgr2=0x7fcd8c0af280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:22.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.218+0000 7fcd914d7640 1 -- 172.21.15.57:0/3547052581 shutdown_connections 2023-10-19T14:01:22.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.218+0000 7fcd914d7640 1 -- 172.21.15.57:0/3547052581 wait complete. 2023-10-19T14:01:22.234 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-19T14:01:22.263 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:22.261+0000 7f4e31232040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-19T14:01:22.292 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:01:22.290+0000 7f4d21f7a040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-19T14:01:22.305 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:22.302+0000 7f4e31232040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-19T14:01:22.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.344+0000 7fee9b456640 1 -- 172.21.15.57:0/1776636986 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fee940ceb40 msgr2=0x7fee940b0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:22.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.344+0000 7fee9b456640 1 --2- 172.21.15.57:0/1776636986 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fee940ceb40 0x7fee940b0650 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7fee8c005fd0 tx=0x7fee8c02c050 comp rx=0 tx=0).stop 2023-10-19T14:01:22.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.344+0000 7fee9b456640 1 -- 172.21.15.57:0/1776636986 shutdown_connections 2023-10-19T14:01:22.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.344+0000 7fee9b456640 1 --2- 172.21.15.57:0/1776636986 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fee940ceb40 0x7fee940b0650 unknown :-1 s=CLOSED pgs=1816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:22.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.344+0000 7fee9b456640 1 --2- 172.21.15.57:0/1776636986 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fee940cf340 0x7fee940d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:22.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.344+0000 7fee9b456640 1 -- 172.21.15.57:0/1776636986 >> 172.21.15.57:0/1776636986 conn(0x7fee940d0d60 msgr2=0x7fee940d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:22.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.344+0000 7fee9b456640 1 -- 172.21.15.57:0/1776636986 shutdown_connections 2023-10-19T14:01:22.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.344+0000 7fee9b456640 1 -- 172.21.15.57:0/1776636986 wait complete. 2023-10-19T14:01:22.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.345+0000 7fee9b456640 1 Processor -- start 2023-10-19T14:01:22.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.345+0000 7fee9b456640 1 -- start start 2023-10-19T14:01:22.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.346+0000 7fee9b456640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fee940ceb40 0x7fee941f5590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:22.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.346+0000 7fee9b456640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fee940cf340 0x7fee9406b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:22.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.346+0000 7fee9b456640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fee9415b2a0 0x7fee941f2750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:22.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.346+0000 7fee9b456640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee941f5ad0 con 0x7fee940ceb40 2023-10-19T14:01:22.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.346+0000 7fee9b456640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee941e6030 con 0x7fee940cf340 2023-10-19T14:01:22.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.346+0000 7fee9b456640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee941e5d50 con 0x7fee9415b2a0 2023-10-19T14:01:22.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.346+0000 7fee991f2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fee940ceb40 0x7fee941f5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:22.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.346+0000 7fee991f2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fee940ceb40 0x7fee941f5590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40716/0 (socket says 172.21.15.57:40716) 2023-10-19T14:01:22.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.346+0000 7fee991f2640 1 -- 172.21.15.57:0/3070246293 learned_addr learned my addr 172.21.15.57:0/3070246293 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:01:22.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.346+0000 7fee999f3640 1 --2- 172.21.15.57:0/3070246293 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fee9415b2a0 0x7fee941f2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:22.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.346+0000 7fee989f1640 1 --2- 172.21.15.57:0/3070246293 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fee940cf340 0x7fee9406b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:22.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.347+0000 7fee991f2640 1 -- 172.21.15.57:0/3070246293 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fee9415b2a0 msgr2=0x7fee941f2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:22.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.347+0000 7fee991f2640 1 --2- 172.21.15.57:0/3070246293 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fee9415b2a0 0x7fee941f2750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:22.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.347+0000 7fee991f2640 1 -- 172.21.15.57:0/3070246293 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fee940cf340 msgr2=0x7fee9406b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:22.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.347+0000 7fee991f2640 1 --2- 172.21.15.57:0/3070246293 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fee940cf340 0x7fee9406b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:22.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.347+0000 7fee991f2640 1 -- 172.21.15.57:0/3070246293 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee8c009840 con 0x7fee940ceb40 2023-10-19T14:01:22.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.347+0000 7fee991f2640 1 --2- 172.21.15.57:0/3070246293 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fee940ceb40 0x7fee941f5590 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7fee9000d950 tx=0x7fee9000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:22.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.347+0000 7fee89ffb640 1 -- 172.21.15.57:0/3070246293 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee90009c30 con 0x7fee940ceb40 2023-10-19T14:01:22.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.347+0000 7fee89ffb640 1 -- 172.21.15.57:0/3070246293 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fee90009e80 con 0x7fee940ceb40 2023-10-19T14:01:22.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.347+0000 7fee9b456640 1 -- 172.21.15.57:0/3070246293 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee940b4da0 con 0x7fee940ceb40 2023-10-19T14:01:22.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.347+0000 7fee89ffb640 1 -- 172.21.15.57:0/3070246293 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee90004280 con 0x7fee940ceb40 2023-10-19T14:01:22.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.347+0000 7fee9b456640 1 -- 172.21.15.57:0/3070246293 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee940b5360 con 0x7fee940ceb40 2023-10-19T14:01:22.357 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:01:22.348+0000 7f4d21f7a040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-19T14:01:22.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.348+0000 7fee9b456640 1 -- 172.21.15.57:0/3070246293 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee940b4c00 con 0x7fee940ceb40 2023-10-19T14:01:22.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.349+0000 7fee89ffb640 1 -- 172.21.15.57:0/3070246293 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 87) v1 ==== 89343+0+0 (secure 0 0 0) 0x7fee900035f0 con 0x7fee940ceb40 2023-10-19T14:01:22.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.349+0000 7fee89ffb640 1 --2- 172.21.15.57:0/3070246293 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7fee7806af60 0x7fee7806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:22.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.349+0000 7fee989f1640 1 -- 172.21.15.57:0/3070246293 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7fee7806af60 msgr2=0x7fee7806d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/4251935070 2023-10-19T14:01:22.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.349+0000 7fee89ffb640 1 -- 172.21.15.57:0/3070246293 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(163..163 src has 1..163) v4 ==== 6489+0+0 (secure 0 0 0) 0x7fee90089ca0 con 0x7fee940ceb40 2023-10-19T14:01:22.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.349+0000 7fee989f1640 1 --2- 172.21.15.57:0/3070246293 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7fee7806af60 0x7fee7806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T14:01:22.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.353+0000 7fee89ffb640 1 -- 172.21.15.57:0/3070246293 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fee940b4c00 con 0x7fee940ceb40 2023-10-19T14:01:22.360 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:22.351+0000 7f4e31232040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-19T14:01:22.397 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:01:22.394+0000 7f4d21f7a040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-19T14:01:22.398 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:22.395+0000 7f4e31232040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-19T14:01:22.451 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:01:22.449+0000 7f4d21f7a040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-19T14:01:22.498 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:01:22.496+0000 7f4d21f7a040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-19T14:01:22.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.542+0000 7fee9b456640 1 -- 172.21.15.57:0/3070246293 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fee940b4c00 con 0x7fee940ceb40 2023-10-19T14:01:22.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.542+0000 7fee89ffb640 1 -- 172.21.15.57:0/3070246293 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v163) v1 ==== 100+0+11 (secure 0 0 0) 0x7fee940b4c00 con 0x7fee940ceb40 2023-10-19T14:01:22.546 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 64 2023-10-19T14:01:22.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.544+0000 7fee9b456640 1 -- 172.21.15.57:0/3070246293 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7fee7806af60 msgr2=0x7fee7806d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T14:01:22.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.544+0000 7fee9b456640 1 --2- 172.21.15.57:0/3070246293 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7fee7806af60 0x7fee7806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:22.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.544+0000 7fee9b456640 1 -- 172.21.15.57:0/3070246293 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fee940ceb40 msgr2=0x7fee941f5590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:22.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.544+0000 7fee9b456640 1 --2- 172.21.15.57:0/3070246293 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fee940ceb40 0x7fee941f5590 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7fee9000d950 tx=0x7fee9000de10 comp rx=0 tx=0).stop 2023-10-19T14:01:22.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.544+0000 7fee9b456640 1 -- 172.21.15.57:0/3070246293 shutdown_connections 2023-10-19T14:01:22.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.545+0000 7fee9b456640 1 --2- 172.21.15.57:0/3070246293 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7fee7806af60 0x7fee7806d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:22.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.545+0000 7fee9b456640 1 --2- 172.21.15.57:0/3070246293 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fee9415b2a0 0x7fee941f2750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:22.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.545+0000 7fee9b456640 1 --2- 172.21.15.57:0/3070246293 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fee940cf340 0x7fee9406b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:22.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.545+0000 7fee9b456640 1 --2- 172.21.15.57:0/3070246293 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fee940ceb40 0x7fee941f5590 unknown :-1 s=CLOSED pgs=1817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:22.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.545+0000 7fee9b456640 1 -- 172.21.15.57:0/3070246293 >> 172.21.15.57:0/3070246293 conn(0x7fee940d0d60 msgr2=0x7fee940ae180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:22.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.545+0000 7fee9b456640 1 -- 172.21.15.57:0/3070246293 shutdown_connections 2023-10-19T14:01:22.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.545+0000 7fee9b456640 1 -- 172.21.15.57:0/3070246293 wait complete. 2023-10-19T14:01:22.558 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-19T14:01:22.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.682+0000 7fd099d4c640 1 -- 172.21.15.57:0/3922235969 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd09406afd0 msgr2=0x7fd0940ae330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:22.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.682+0000 7fd099d4c640 1 --2- 172.21.15.57:0/3922235969 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd09406afd0 0x7fd0940ae330 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7fd09406ba00 tx=0x7fd08c02bd70 comp rx=0 tx=0).stop 2023-10-19T14:01:22.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.682+0000 7fd099d4c640 1 -- 172.21.15.57:0/3922235969 shutdown_connections 2023-10-19T14:01:22.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.682+0000 7fd099d4c640 1 --2- 172.21.15.57:0/3922235969 >> v2:172.21.15.179:3300/0 conn(0x7fd0940b00b0 0x7fd0940ae870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:22.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.682+0000 7fd099d4c640 1 --2- 172.21.15.57:0/3922235969 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd09406afd0 0x7fd0940ae330 unknown :-1 s=CLOSED pgs=1818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:22.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.682+0000 7fd099d4c640 1 -- 172.21.15.57:0/3922235969 >> 172.21.15.57:0/3922235969 conn(0x7fd094099ee0 msgr2=0x7fd0940d0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:22.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.682+0000 7fd099d4c640 1 -- 172.21.15.57:0/3922235969 shutdown_connections 2023-10-19T14:01:22.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.682+0000 7fd099d4c640 1 -- 172.21.15.57:0/3922235969 wait complete. 2023-10-19T14:01:22.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.684+0000 7fd099d4c640 1 Processor -- start 2023-10-19T14:01:22.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.684+0000 7fd099d4c640 1 -- start start 2023-10-19T14:01:22.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.685+0000 7fd099d4c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd09406afd0 0x7fd0941f0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:22.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.685+0000 7fd099d4c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd0940b00b0 0x7fd0940c99b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:22.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.685+0000 7fd099d4c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd09415e500 0x7fd0940c9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:22.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.685+0000 7fd099d4c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0940c5790 con 0x7fd0940b00b0 2023-10-19T14:01:22.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.685+0000 7fd099d4c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0940c4da0 con 0x7fd09406afd0 2023-10-19T14:01:22.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.685+0000 7fd099d4c640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0940c5160 con 0x7fd09415e500 2023-10-19T14:01:22.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.685+0000 7fd092ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd0940b00b0 0x7fd0940c99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:22.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.685+0000 7fd0937fe640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd09406afd0 0x7fd0941f0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:22.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.685+0000 7fd092ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd0940b00b0 0x7fd0940c99b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40744/0 (socket says 172.21.15.57:40744) 2023-10-19T14:01:22.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.685+0000 7fd092ffd640 1 -- 172.21.15.57:0/1044040350 learned_addr learned my addr 172.21.15.57:0/1044040350 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:01:22.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.685+0000 7fd093fff640 1 --2- 172.21.15.57:0/1044040350 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd09415e500 0x7fd0940c9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:22.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.686+0000 7fd092ffd640 1 -- 172.21.15.57:0/1044040350 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd09415e500 msgr2=0x7fd0940c9ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:22.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.686+0000 7fd092ffd640 1 --2- 172.21.15.57:0/1044040350 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd09415e500 0x7fd0940c9ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:22.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.686+0000 7fd092ffd640 1 -- 172.21.15.57:0/1044040350 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd09406afd0 msgr2=0x7fd0941f0fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:22.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.686+0000 7fd092ffd640 1 --2- 172.21.15.57:0/1044040350 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd09406afd0 0x7fd0941f0fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:22.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.686+0000 7fd092ffd640 1 -- 172.21.15.57:0/1044040350 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd08c009490 con 0x7fd0940b00b0 2023-10-19T14:01:22.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.686+0000 7fd092ffd640 1 --2- 172.21.15.57:0/1044040350 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd0940b00b0 0x7fd0940c99b0 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7fd08c000c00 tx=0x7fd08c0038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:22.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.686+0000 7fd08bfff640 1 -- 172.21.15.57:0/1044040350 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd08c003ea0 con 0x7fd0940b00b0 2023-10-19T14:01:22.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.686+0000 7fd08bfff640 1 -- 172.21.15.57:0/1044040350 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fd08c005020 con 0x7fd0940b00b0 2023-10-19T14:01:22.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.686+0000 7fd08bfff640 1 -- 172.21.15.57:0/1044040350 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd08c005450 con 0x7fd0940b00b0 2023-10-19T14:01:22.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.686+0000 7fd099d4c640 1 -- 172.21.15.57:0/1044040350 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0940cce60 con 0x7fd0940b00b0 2023-10-19T14:01:22.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.686+0000 7fd099d4c640 1 -- 172.21.15.57:0/1044040350 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0940c53b0 con 0x7fd0940b00b0 2023-10-19T14:01:22.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.687+0000 7fd099d4c640 1 -- 172.21.15.57:0/1044040350 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0940cd380 con 0x7fd0940b00b0 2023-10-19T14:01:22.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.688+0000 7fd08bfff640 1 -- 172.21.15.57:0/1044040350 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 87) v1 ==== 89343+0+0 (secure 0 0 0) 0x7fd08c0047a0 con 0x7fd0940b00b0 2023-10-19T14:01:22.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.688+0000 7fd08bfff640 1 --2- 172.21.15.57:0/1044040350 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7fd07406af60 0x7fd07406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:22.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.689+0000 7fd0937fe640 1 -- 172.21.15.57:0/1044040350 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7fd07406af60 msgr2=0x7fd07406d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/4251935070 2023-10-19T14:01:22.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.689+0000 7fd08bfff640 1 -- 172.21.15.57:0/1044040350 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(163..163 src has 1..163) v4 ==== 6489+0+0 (secure 0 0 0) 0x7fd08c0ac280 con 0x7fd0940b00b0 2023-10-19T14:01:22.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.689+0000 7fd0937fe640 1 --2- 172.21.15.57:0/1044040350 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7fd07406af60 0x7fd07406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T14:01:22.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.693+0000 7fd08bfff640 1 -- 172.21.15.57:0/1044040350 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd08c076f20 con 0x7fd0940b00b0 2023-10-19T14:01:22.736 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:22.734+0000 7f4e31232040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-19T14:01:22.853 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:01:22.850+0000 7f4d21f7a040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-19T14:01:22.868 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:22.866+0000 7f4e31232040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-19T14:01:22.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.880+0000 7fd099d4c640 1 -- 172.21.15.57:0/1044040350 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fd094066960 con 0x7fd0940b00b0 2023-10-19T14:01:22.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.881+0000 7fd08bfff640 1 -- 172.21.15.57:0/1044040350 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v163) v1 ==== 80+0+473 (secure 0 0 0) 0x7fd08c07a540 con 0x7fd0940b00b0 2023-10-19T14:01:22.884 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/744328422 2023-10-20T13:05:09.216573+0000 2023-10-19T14:01:22.884 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/1909872102 2023-10-20T13:05:09.216573+0000 2023-10-19T14:01:22.884 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/1880188133 2023-10-20T13:05:09.216573+0000 2023-10-19T14:01:22.885 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/40128169 2023-10-20T13:05:09.216573+0000 2023-10-19T14:01:22.885 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:6833/2581690007 2023-10-20T13:05:09.216573+0000 2023-10-19T14:01:22.885 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/3857987280 2023-10-20T13:05:09.216573+0000 2023-10-19T14:01:22.885 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:6832/2581690007 2023-10-20T13:05:09.216573+0000 2023-10-19T14:01:22.885 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/57720407 2023-10-20T13:05:09.216573+0000 2023-10-19T14:01:22.886 INFO:teuthology.orchestra.run.smithi057.stderr:listed 8 entries 2023-10-19T14:01:22.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.883+0000 7fd099d4c640 1 -- 172.21.15.57:0/1044040350 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7fd07406af60 msgr2=0x7fd07406d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T14:01:22.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.883+0000 7fd099d4c640 1 --2- 172.21.15.57:0/1044040350 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7fd07406af60 0x7fd07406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:22.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.883+0000 7fd099d4c640 1 -- 172.21.15.57:0/1044040350 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd0940b00b0 msgr2=0x7fd0940c99b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:22.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.883+0000 7fd099d4c640 1 --2- 172.21.15.57:0/1044040350 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd0940b00b0 0x7fd0940c99b0 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7fd08c000c00 tx=0x7fd08c0038a0 comp rx=0 tx=0).stop 2023-10-19T14:01:22.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.883+0000 7fd099d4c640 1 -- 172.21.15.57:0/1044040350 shutdown_connections 2023-10-19T14:01:22.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.883+0000 7fd099d4c640 1 --2- 172.21.15.57:0/1044040350 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7fd07406af60 0x7fd07406d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:22.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.883+0000 7fd099d4c640 1 --2- 172.21.15.57:0/1044040350 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd09415e500 0x7fd0940c9ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:22.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.883+0000 7fd099d4c640 1 --2- 172.21.15.57:0/1044040350 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd0940b00b0 0x7fd0940c99b0 unknown :-1 s=CLOSED pgs=1819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:22.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.883+0000 7fd099d4c640 1 --2- 172.21.15.57:0/1044040350 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd09406afd0 0x7fd0941f0fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:22.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.883+0000 7fd099d4c640 1 -- 172.21.15.57:0/1044040350 >> 172.21.15.57:0/1044040350 conn(0x7fd094099ee0 msgr2=0x7fd0940b0e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:22.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.883+0000 7fd099d4c640 1 -- 172.21.15.57:0/1044040350 shutdown_connections 2023-10-19T14:01:22.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:22.883+0000 7fd099d4c640 1 -- 172.21.15.57:0/1044040350 wait complete. 2023-10-19T14:01:22.896 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi057.front.sepia.ceph.com... 2023-10-19T14:01:22.897 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:01:22.897 DEBUG:teuthology.orchestra.run.smithi057:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-10-19T14:01:22.926 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi057.stderr:ceph-fuse[135869]: fuse finished with error 0 and tester_r 0 2023-10-19T14:01:22.929 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi057.front.sepia.ceph.com 2023-10-19T14:01:22.929 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:01:22.930 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-19T14:01:22.960 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:22.957+0000 7f4e31232040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-19T14:01:22.994 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:01:22.992+0000 7f4d21f7a040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-19T14:01:23.001 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:22.999+0000 7f4e31232040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-19T14:01:23.091 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:01:23.089+0000 7f4d21f7a040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-19T14:01:23.097 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:23.095+0000 7f4e31232040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-19T14:01:23.098 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-19T14:01:23.099 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T14:01:23.114 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054833,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=284 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3247632k,nr_inodes=811908,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-19T14:01:23.115 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi179.front.sepia.ceph.com 2023-10-19T14:01:23.115 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:01:23.115 DEBUG:teuthology.orchestra.run.smithi179:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-10-19T14:01:23.136 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:01:23.134+0000 7f4d21f7a040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-19T14:01:23.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-19T14:01:23.169 INFO:teuthology.orchestra.run.smithi179.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2023-10-19T14:01:23.169 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T14:01:23.170 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T14:01:23.186 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055614,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=358 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248260k,nr_inodes=812065,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-19T14:01:23.186 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi057.front.sepia.ceph.com 2023-10-19T14:01:23.187 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:01:23.187 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-10-19T14:01:23.192 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:23.189+0000 7f4e31232040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-19T14:01:23.237 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:01:23.234+0000 7f4d21f7a040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-19T14:01:23.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-19T14:01:23.241 INFO:teuthology.orchestra.run.smithi057.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2023-10-19T14:01:23.242 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-19T14:01:23.242 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T14:01:23.257 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054833,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=284 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3247632k,nr_inodes=811908,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-19T14:01:23.257 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi179.front.sepia.ceph.com 2023-10-19T14:01:23.258 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:01:23.258 DEBUG:teuthology.orchestra.run.smithi179:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-10-19T14:01:23.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-19T14:01:23.312 INFO:teuthology.orchestra.run.smithi179.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2023-10-19T14:01:23.313 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T14:01:23.333 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:23.330+0000 7f4e31232040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-19T14:01:23.333 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:01:23.330+0000 7f4d21f7a040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-19T14:01:23.377 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:23.373+0000 7f4e31232040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-19T14:01:23.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.420+0000 7f84d45e9640 1 -- 172.21.15.57:0/799090656 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f84cc0cf340 msgr2=0x7f84cc0d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:23.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.420+0000 7f84d45e9640 1 --2- 172.21.15.57:0/799090656 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f84cc0cf340 0x7f84cc0d1910 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7f84c8009830 tx=0x7f84c802bd60 comp rx=0 tx=0).stop 2023-10-19T14:01:23.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.421+0000 7f84d45e9640 1 -- 172.21.15.57:0/799090656 shutdown_connections 2023-10-19T14:01:23.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.421+0000 7f84d45e9640 1 --2- 172.21.15.57:0/799090656 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f84cc0cf340 0x7f84cc0d1910 unknown :-1 s=CLOSED pgs=1821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:23.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.421+0000 7f84d45e9640 1 -- 172.21.15.57:0/799090656 >> 172.21.15.57:0/799090656 conn(0x7f84cc0d0d60 msgr2=0x7f84cc0d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:23.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.421+0000 7f84d45e9640 1 -- 172.21.15.57:0/799090656 shutdown_connections 2023-10-19T14:01:23.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.421+0000 7f84d45e9640 1 -- 172.21.15.57:0/799090656 wait complete. 2023-10-19T14:01:23.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.422+0000 7f84d45e9640 1 Processor -- start 2023-10-19T14:01:23.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.422+0000 7f84d45e9640 1 -- start start 2023-10-19T14:01:23.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.423+0000 7f84d45e9640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f84cc0ceb40 0x7f84cc0c9620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:23.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.423+0000 7f84d45e9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f84cc0cf340 0x7f84cc15e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:23.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.423+0000 7f84d45e9640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f84cc15b2a0 0x7f84cc0c7e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:23.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.423+0000 7f84d45e9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84cc0c8370 con 0x7f84cc0cf340 2023-10-19T14:01:23.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.423+0000 7f84d45e9640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84cc0c8580 con 0x7f84cc0ceb40 2023-10-19T14:01:23.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.423+0000 7f84d45e9640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84cc0c8860 con 0x7f84cc15b2a0 2023-10-19T14:01:23.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.423+0000 7f84d1b84640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f84cc0cf340 0x7f84cc15e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:23.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.423+0000 7f84d1b84640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f84cc0cf340 0x7f84cc15e1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40752/0 (socket says 172.21.15.57:40752) 2023-10-19T14:01:23.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.423+0000 7f84d1b84640 1 -- 172.21.15.57:0/2268362352 learned_addr learned my addr 172.21.15.57:0/2268362352 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:01:23.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.423+0000 7f84d2385640 1 --2- 172.21.15.57:0/2268362352 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f84cc0ceb40 0x7f84cc0c9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:23.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.423+0000 7f84d2b86640 1 --2- 172.21.15.57:0/2268362352 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f84cc15b2a0 0x7f84cc0c7e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:23.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.423+0000 7f84d1b84640 1 -- 172.21.15.57:0/2268362352 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f84cc15b2a0 msgr2=0x7f84cc0c7e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:23.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.423+0000 7f84d1b84640 1 --2- 172.21.15.57:0/2268362352 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f84cc15b2a0 0x7f84cc0c7e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:23.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.423+0000 7f84d1b84640 1 -- 172.21.15.57:0/2268362352 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f84cc0ceb40 msgr2=0x7f84cc0c9620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:23.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.423+0000 7f84d1b84640 1 --2- 172.21.15.57:0/2268362352 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f84cc0ceb40 0x7f84cc0c9620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:23.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.423+0000 7f84d1b84640 1 -- 172.21.15.57:0/2268362352 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84c8009490 con 0x7f84cc0cf340 2023-10-19T14:01:23.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.423+0000 7f84d1b84640 1 --2- 172.21.15.57:0/2268362352 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f84cc0cf340 0x7f84cc15e1a0 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7f84bc00c400 tx=0x7f84bc00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:23.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.423+0000 7f84c2ffd640 1 -- 172.21.15.57:0/2268362352 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84bc0052d0 con 0x7f84cc0cf340 2023-10-19T14:01:23.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.424+0000 7f84c2ffd640 1 -- 172.21.15.57:0/2268362352 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f84bc0035f0 con 0x7f84cc0cf340 2023-10-19T14:01:23.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.424+0000 7f84c2ffd640 1 -- 172.21.15.57:0/2268362352 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84bc003a00 con 0x7f84cc0cf340 2023-10-19T14:01:23.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.424+0000 7f84d45e9640 1 -- 172.21.15.57:0/2268362352 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84cc0c8b40 con 0x7f84cc0cf340 2023-10-19T14:01:23.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.424+0000 7f84d45e9640 1 -- 172.21.15.57:0/2268362352 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84cc0c9100 con 0x7f84cc0cf340 2023-10-19T14:01:23.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.425+0000 7f84d45e9640 1 -- 172.21.15.57:0/2268362352 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84cc0cc940 con 0x7f84cc0cf340 2023-10-19T14:01:23.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.425+0000 7f84c2ffd640 1 -- 172.21.15.57:0/2268362352 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 87) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f84bc003b60 con 0x7f84cc0cf340 2023-10-19T14:01:23.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.425+0000 7f84c2ffd640 1 --2- 172.21.15.57:0/2268362352 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7f84b006af60 0x7f84b006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:23.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.425+0000 7f84d2385640 1 -- 172.21.15.57:0/2268362352 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7f84b006af60 msgr2=0x7f84b006d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/4251935070 2023-10-19T14:01:23.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.425+0000 7f84c2ffd640 1 -- 172.21.15.57:0/2268362352 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(163..163 src has 1..163) v4 ==== 6489+0+0 (secure 0 0 0) 0x7f84bc089270 con 0x7f84cc0cf340 2023-10-19T14:01:23.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.426+0000 7f84d2385640 1 --2- 172.21.15.57:0/2268362352 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7f84b006af60 0x7f84b006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T14:01:23.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.430+0000 7f84c2ffd640 1 -- 172.21.15.57:0/2268362352 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f84bc053f20 con 0x7f84cc0cf340 2023-10-19T14:01:23.479 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:01:23.477+0000 7f4d21f7a040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-19T14:01:23.480 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:23.477+0000 7f4e31232040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-19T14:01:23.524 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:01:23.521+0000 7f4d21f7a040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-19T14:01:23.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.627+0000 7f84d2385640 1 -- 172.21.15.57:0/2268362352 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7f84b006af60 msgr2=0x7f84b006d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/4251935070 2023-10-19T14:01:23.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.627+0000 7f84d2385640 1 --2- 172.21.15.57:0/2268362352 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7f84b006af60 0x7f84b006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-19T14:01:23.633 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:01:23.631+0000 7f4d21f7a040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-19T14:01:23.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.641+0000 7f84d45e9640 1 -- 172.21.15.57:0/2268362352 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f84cc0690e0 con 0x7f84cc0cf340 2023-10-19T14:01:23.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.642+0000 7f84c2ffd640 1 -- 172.21.15.57:0/2268362352 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+4918 (secure 0 0 0) 0x7f84bc057540 con 0x7f84cc0cf340 2023-10-19T14:01:23.645 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:01:23.645 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":63,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7733,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/1818414654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":1818414654},{"type":"v1","addr":"172.21.15.179:6836","nonce":1818414654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":8647,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/490712721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":490712721},{"type":"v1","addr":"172.21.15.57:6835","nonce":490712721}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":8656,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/2005811112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":2005811112},{"type":"v1","addr":"172.21.15.179:6837","nonce":2005811112}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T13:04:58.134572+0000","modified":"2023-10-19T13:05:00.138505+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9618},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9618":{"gid":9618,"name":"c","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.15.57:6833/1460919655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":1460919655},{"type":"v1","addr":"172.21.15.57:6833","nonce":1460919655}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2023-10-19T14:01:23.645 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 63 2023-10-19T14:01:23.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.644+0000 7f84d45e9640 1 -- 172.21.15.57:0/2268362352 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7f84b006af60 msgr2=0x7f84b006d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T14:01:23.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.644+0000 7f84d45e9640 1 --2- 172.21.15.57:0/2268362352 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7f84b006af60 0x7f84b006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:23.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.644+0000 7f84d45e9640 1 -- 172.21.15.57:0/2268362352 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f84cc0cf340 msgr2=0x7f84cc15e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:23.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.644+0000 7f84d45e9640 1 --2- 172.21.15.57:0/2268362352 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f84cc0cf340 0x7f84cc15e1a0 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7f84bc00c400 tx=0x7f84bc00c8c0 comp rx=0 tx=0).stop 2023-10-19T14:01:23.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.644+0000 7f84d45e9640 1 -- 172.21.15.57:0/2268362352 shutdown_connections 2023-10-19T14:01:23.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.644+0000 7f84d45e9640 1 --2- 172.21.15.57:0/2268362352 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7f84b006af60 0x7f84b006d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:23.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.644+0000 7f84d45e9640 1 --2- 172.21.15.57:0/2268362352 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f84cc15b2a0 0x7f84cc0c7e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:23.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.644+0000 7f84d45e9640 1 --2- 172.21.15.57:0/2268362352 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f84cc0cf340 0x7f84cc15e1a0 unknown :-1 s=CLOSED pgs=1822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:23.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.644+0000 7f84d45e9640 1 --2- 172.21.15.57:0/2268362352 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f84cc0ceb40 0x7f84cc0c9620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:23.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.644+0000 7f84d45e9640 1 -- 172.21.15.57:0/2268362352 >> 172.21.15.57:0/2268362352 conn(0x7f84cc0d0d60 msgr2=0x7f84cc0b58f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:23.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.644+0000 7f84d45e9640 1 -- 172.21.15.57:0/2268362352 shutdown_connections 2023-10-19T14:01:23.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.644+0000 7f84d45e9640 1 -- 172.21.15.57:0/2268362352 wait complete. 2023-10-19T14:01:23.657 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T14:01:23.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.804+0000 7fc7bb1f5640 1 -- 172.21.15.57:0/2087894482 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc7b40cf340 msgr2=0x7fc7b40d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:23.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.804+0000 7fc7bb1f5640 1 --2- 172.21.15.57:0/2087894482 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc7b40cf340 0x7fc7b40d1910 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7fc7a8005fd0 tx=0x7fc7a802bcf0 comp rx=0 tx=0).stop 2023-10-19T14:01:23.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.804+0000 7fc7bb1f5640 1 -- 172.21.15.57:0/2087894482 shutdown_connections 2023-10-19T14:01:23.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.804+0000 7fc7bb1f5640 1 --2- 172.21.15.57:0/2087894482 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc7b415b2a0 0x7fc7b40cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:23.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.804+0000 7fc7bb1f5640 1 --2- 172.21.15.57:0/2087894482 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc7b40cf340 0x7fc7b40d1910 unknown :-1 s=CLOSED pgs=1823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:23.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.804+0000 7fc7bb1f5640 1 -- 172.21.15.57:0/2087894482 >> 172.21.15.57:0/2087894482 conn(0x7fc7b40d0d60 msgr2=0x7fc7b40d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:23.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.804+0000 7fc7bb1f5640 1 -- 172.21.15.57:0/2087894482 shutdown_connections 2023-10-19T14:01:23.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.804+0000 7fc7bb1f5640 1 -- 172.21.15.57:0/2087894482 wait complete. 2023-10-19T14:01:23.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.805+0000 7fc7bb1f5640 1 Processor -- start 2023-10-19T14:01:23.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.805+0000 7fc7bb1f5640 1 -- start start 2023-10-19T14:01:23.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.806+0000 7fc7bb1f5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc7b40ceb40 0x7fc7b40b5d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:23.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.806+0000 7fc7bb1f5640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc7b40cf340 0x7fc7b40ca8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:23.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.806+0000 7fc7bb1f5640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc7b415b2a0 0x7fc7b4155960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:23.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.806+0000 7fc7bb1f5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b40c83d0 con 0x7fc7b40ceb40 2023-10-19T14:01:23.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.806+0000 7fc7bb1f5640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b40c86b0 con 0x7fc7b40cf340 2023-10-19T14:01:23.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.806+0000 7fc7bb1f5640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b40c8990 con 0x7fc7b415b2a0 2023-10-19T14:01:23.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.806+0000 7fc7b8f91640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc7b40ceb40 0x7fc7b40b5d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:23.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.806+0000 7fc7b8f91640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc7b40ceb40 0x7fc7b40b5d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40762/0 (socket says 172.21.15.57:40762) 2023-10-19T14:01:23.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.806+0000 7fc7b8f91640 1 -- 172.21.15.57:0/2118811857 learned_addr learned my addr 172.21.15.57:0/2118811857 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:01:23.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.806+0000 7fc7b3fff640 1 --2- 172.21.15.57:0/2118811857 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc7b40cf340 0x7fc7b40ca8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:23.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.807+0000 7fc7b9792640 1 --2- 172.21.15.57:0/2118811857 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc7b415b2a0 0x7fc7b4155960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:23.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.807+0000 7fc7b8f91640 1 -- 172.21.15.57:0/2118811857 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc7b415b2a0 msgr2=0x7fc7b4155960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:23.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.807+0000 7fc7b8f91640 1 --2- 172.21.15.57:0/2118811857 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc7b415b2a0 0x7fc7b4155960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:23.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.807+0000 7fc7b8f91640 1 -- 172.21.15.57:0/2118811857 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc7b40cf340 msgr2=0x7fc7b40ca8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:23.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.807+0000 7fc7b8f91640 1 --2- 172.21.15.57:0/2118811857 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc7b40cf340 0x7fc7b40ca8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:23.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.807+0000 7fc7b8f91640 1 -- 172.21.15.57:0/2118811857 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7a8009560 con 0x7fc7b40ceb40 2023-10-19T14:01:23.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.807+0000 7fc7b3fff640 1 --2- 172.21.15.57:0/2118811857 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc7b40cf340 0x7fc7b40ca8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:01:23.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.807+0000 7fc7b8f91640 1 --2- 172.21.15.57:0/2118811857 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc7b40ceb40 0x7fc7b40b5d20 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7fc7a8009850 tx=0x7fc7a802d9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:23.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.807+0000 7fc7b17fa640 1 -- 172.21.15.57:0/2118811857 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7a802ddc0 con 0x7fc7b40ceb40 2023-10-19T14:01:23.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.807+0000 7fc7b17fa640 1 -- 172.21.15.57:0/2118811857 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fc7a80036e0 con 0x7fc7b40ceb40 2023-10-19T14:01:23.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.807+0000 7fc7b17fa640 1 -- 172.21.15.57:0/2118811857 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7a8003a90 con 0x7fc7b40ceb40 2023-10-19T14:01:23.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.807+0000 7fc7bb1f5640 1 -- 172.21.15.57:0/2118811857 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7b40c8c70 con 0x7fc7b40ceb40 2023-10-19T14:01:23.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.807+0000 7fc7bb1f5640 1 -- 172.21.15.57:0/2118811857 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7b40b6260 con 0x7fc7b40ceb40 2023-10-19T14:01:23.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.808+0000 7fc7bb1f5640 1 -- 172.21.15.57:0/2118811857 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7b40c9130 con 0x7fc7b40ceb40 2023-10-19T14:01:23.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.809+0000 7fc7b17fa640 1 -- 172.21.15.57:0/2118811857 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 87) v1 ==== 89343+0+0 (secure 0 0 0) 0x7fc7a8003bf0 con 0x7fc7b40ceb40 2023-10-19T14:01:23.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.809+0000 7fc7b17fa640 1 --2- 172.21.15.57:0/2118811857 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7fc79806af60 0x7fc79806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:23.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.809+0000 7fc7b3fff640 1 -- 172.21.15.57:0/2118811857 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7fc79806af60 msgr2=0x7fc79806d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/4251935070 2023-10-19T14:01:23.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.809+0000 7fc7b17fa640 1 -- 172.21.15.57:0/2118811857 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(163..163 src has 1..163) v4 ==== 6489+0+0 (secure 0 0 0) 0x7fc7a80ac520 con 0x7fc7b40ceb40 2023-10-19T14:01:23.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.809+0000 7fc7b3fff640 1 --2- 172.21.15.57:0/2118811857 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7fc79806af60 0x7fc79806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T14:01:23.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.817+0000 7fc7b17fa640 1 -- 172.21.15.57:0/2118811857 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc7a8077190 con 0x7fc7b40ceb40 2023-10-19T14:01:23.998 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.995+0000 7fc7bb1f5640 1 -- 172.21.15.57:0/2118811857 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc7b40690e0 con 0x7fc7b40ceb40 2023-10-19T14:01:23.999 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.997+0000 7fc7b17fa640 1 -- 172.21.15.57:0/2118811857 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v163) v1 ==== 74+0+17457 (secure 0 0 0) 0x7fc7a807a7b0 con 0x7fc7b40ceb40 2023-10-19T14:01:24.000 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:01:24.000 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":163,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T14:01:22.057521+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2023-10-19T13:04:56.433332+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"156","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2023-10-19T13:04:57.112749+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"163","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":112,"snap_epoch":163,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.179:0/744328422":"2023-10-20T13:05:09.216573+0000","172.21.15.179:0/1909872102":"2023-10-20T13:05:09.216573+0000","172.21.15.179:0/1880188133":"2023-10-20T13:05:09.216573+0000","172.21.15.179:0/40128169":"2023-10-20T13:05:09.216573+0000","172.21.15.179:6833/2581690007":"2023-10-20T13:05:09.216573+0000","172.21.15.179:0/3857987280":"2023-10-20T13:05:09.216573+0000","172.21.15.179:6832/2581690007":"2023-10-20T13:05:09.216573+0000","172.21.15.179:0/57720407":"2023-10-20T13:05:09.216573+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":20,"snaps":[{"begin":3,"length":54},{"begin":59,"length":54}]}],"new_removed_snaps":[{"pool":20,"snaps":[{"begin":52,"length":5},{"begin":108,"length":5}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T14:01:24.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.999+0000 7fc7bb1f5640 1 -- 172.21.15.57:0/2118811857 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7fc79806af60 msgr2=0x7fc79806d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T14:01:24.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.999+0000 7fc7bb1f5640 1 --2- 172.21.15.57:0/2118811857 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7fc79806af60 0x7fc79806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:24.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.999+0000 7fc7bb1f5640 1 -- 172.21.15.57:0/2118811857 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc7b40ceb40 msgr2=0x7fc7b40b5d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:24.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.999+0000 7fc7bb1f5640 1 --2- 172.21.15.57:0/2118811857 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc7b40ceb40 0x7fc7b40b5d20 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7fc7a8009850 tx=0x7fc7a802d9c0 comp rx=0 tx=0).stop 2023-10-19T14:01:24.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.999+0000 7fc7bb1f5640 1 -- 172.21.15.57:0/2118811857 shutdown_connections 2023-10-19T14:01:24.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.999+0000 7fc7bb1f5640 1 --2- 172.21.15.57:0/2118811857 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7fc79806af60 0x7fc79806d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:24.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.999+0000 7fc7bb1f5640 1 --2- 172.21.15.57:0/2118811857 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc7b415b2a0 0x7fc7b4155960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:24.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.999+0000 7fc7bb1f5640 1 --2- 172.21.15.57:0/2118811857 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc7b40cf340 0x7fc7b40ca8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:24.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.999+0000 7fc7bb1f5640 1 --2- 172.21.15.57:0/2118811857 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc7b40ceb40 0x7fc7b40b5d20 unknown :-1 s=CLOSED pgs=1824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:24.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.999+0000 7fc7bb1f5640 1 -- 172.21.15.57:0/2118811857 >> 172.21.15.57:0/2118811857 conn(0x7fc7b40d0d60 msgr2=0x7fc7b40cc660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:24.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.999+0000 7fc7bb1f5640 1 -- 172.21.15.57:0/2118811857 shutdown_connections 2023-10-19T14:01:24.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:23.999+0000 7fc7bb1f5640 1 -- 172.21.15.57:0/2118811857 wait complete. 2023-10-19T14:01:24.013 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-19T12:44:07.371186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 19, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-19T13:04:56.433332+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '156', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 20, 'pool_name': 'cephfs_data', 'create_time': '2023-10-19T13:04:57.112749+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '163', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 112, 'snap_epoch': 163, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-19T14:01:24.014 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-19T14:01:24.097 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:24.094+0000 7f4e31232040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-19T14:01:24.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.130+0000 7f9a0199e640 1 -- 172.21.15.57:0/779256410 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f99fc06d2a0 msgr2=0x7f99fc0caf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:24.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.130+0000 7f9a0199e640 1 --2- 172.21.15.57:0/779256410 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f99fc06d2a0 0x7f99fc0caf80 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f99e8005fd0 tx=0x7f99e802be10 comp rx=0 tx=0).stop 2023-10-19T14:01:24.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.130+0000 7f9a0199e640 1 -- 172.21.15.57:0/779256410 shutdown_connections 2023-10-19T14:01:24.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.130+0000 7f9a0199e640 1 --2- 172.21.15.57:0/779256410 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f99fc06d2a0 0x7f99fc0caf80 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:24.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.130+0000 7f9a0199e640 1 --2- 172.21.15.57:0/779256410 >> v2:172.21.15.179:3300/0 conn(0x7f99fc159040 0x7f99fc0cf400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:24.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.130+0000 7f9a0199e640 1 -- 172.21.15.57:0/779256410 >> 172.21.15.57:0/779256410 conn(0x7f99fc0ce8e0 msgr2=0x7f99fc0cecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:24.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.130+0000 7f9a0199e640 1 -- 172.21.15.57:0/779256410 shutdown_connections 2023-10-19T14:01:24.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.130+0000 7f9a0199e640 1 -- 172.21.15.57:0/779256410 wait complete. 2023-10-19T14:01:24.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.131+0000 7f9a0199e640 1 Processor -- start 2023-10-19T14:01:24.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.131+0000 7f9a0199e640 1 -- start start 2023-10-19T14:01:24.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.132+0000 7f9a0199e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f99fc06d2a0 0x7f99fc1e41f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:24.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.132+0000 7f9a0199e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f99fc0cb4c0 0x7f99fc1594d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:24.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.132+0000 7f9a0199e640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f99fc159040 0x7f99fc15ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:24.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.132+0000 7f9a0199e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99fc1e1890 con 0x7f99fc0cb4c0 2023-10-19T14:01:24.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.132+0000 7f9a0199e640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99fc1e1b70 con 0x7f99fc06d2a0 2023-10-19T14:01:24.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.132+0000 7f9a0199e640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99fc1e1e50 con 0x7f99fc159040 2023-10-19T14:01:24.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.132+0000 7f99fbfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f99fc0cb4c0 0x7f99fc1594d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:24.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.132+0000 7f99fbfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f99fc0cb4c0 0x7f99fc1594d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40770/0 (socket says 172.21.15.57:40770) 2023-10-19T14:01:24.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.132+0000 7f99fbfff640 1 -- 172.21.15.57:0/514073806 learned_addr learned my addr 172.21.15.57:0/514073806 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:01:24.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.132+0000 7f9a0099c640 1 --2- 172.21.15.57:0/514073806 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f99fc06d2a0 0x7f99fc1e41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:24.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.132+0000 7f9a0119d640 1 --2- 172.21.15.57:0/514073806 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f99fc159040 0x7f99fc15ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:24.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.133+0000 7f99fbfff640 1 -- 172.21.15.57:0/514073806 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f99fc159040 msgr2=0x7f99fc15ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:24.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.133+0000 7f99fbfff640 1 --2- 172.21.15.57:0/514073806 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f99fc159040 0x7f99fc15ae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:24.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.133+0000 7f99fbfff640 1 -- 172.21.15.57:0/514073806 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f99fc06d2a0 msgr2=0x7f99fc1e41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:24.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.133+0000 7f99fbfff640 1 --2- 172.21.15.57:0/514073806 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f99fc06d2a0 0x7f99fc1e41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:24.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.133+0000 7f99fbfff640 1 -- 172.21.15.57:0/514073806 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99f0009490 con 0x7f99fc0cb4c0 2023-10-19T14:01:24.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.133+0000 7f9a0099c640 1 --2- 172.21.15.57:0/514073806 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f99fc06d2a0 0x7f99fc1e41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T14:01:24.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.133+0000 7f9a0119d640 1 --2- 172.21.15.57:0/514073806 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f99fc159040 0x7f99fc15ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T14:01:24.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.133+0000 7f99fbfff640 1 --2- 172.21.15.57:0/514073806 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f99fc0cb4c0 0x7f99fc1594d0 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7f99e8006010 tx=0x7f99e802da70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:24.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.133+0000 7f99f97fa640 1 -- 172.21.15.57:0/514073806 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99e802de70 con 0x7f99fc0cb4c0 2023-10-19T14:01:24.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.133+0000 7f99f97fa640 1 -- 172.21.15.57:0/514073806 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f99e80036e0 con 0x7f99fc0cb4c0 2023-10-19T14:01:24.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.133+0000 7f9a0199e640 1 -- 172.21.15.57:0/514073806 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99e8009560 con 0x7f99fc0cb4c0 2023-10-19T14:01:24.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.133+0000 7f99f97fa640 1 -- 172.21.15.57:0/514073806 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99e8003af0 con 0x7f99fc0cb4c0 2023-10-19T14:01:24.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.133+0000 7f9a0199e640 1 -- 172.21.15.57:0/514073806 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99fc15c2d0 con 0x7f99fc0cb4c0 2023-10-19T14:01:24.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.134+0000 7f9a0199e640 1 -- 172.21.15.57:0/514073806 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99fc15c4c0 con 0x7f99fc0cb4c0 2023-10-19T14:01:24.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.135+0000 7f99f97fa640 1 -- 172.21.15.57:0/514073806 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 87) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f99e8003c50 con 0x7f99fc0cb4c0 2023-10-19T14:01:24.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.135+0000 7f99f97fa640 1 --2- 172.21.15.57:0/514073806 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7f99e006af60 0x7f99e006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:24.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.135+0000 7f99f97fa640 1 -- 172.21.15.57:0/514073806 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(163..163 src has 1..163) v4 ==== 6489+0+0 (secure 0 0 0) 0x7f99e80ac4e0 con 0x7f99fc0cb4c0 2023-10-19T14:01:24.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.135+0000 7f9a0099c640 1 -- 172.21.15.57:0/514073806 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7f99e006af60 msgr2=0x7f99e006d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/4251935070 2023-10-19T14:01:24.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.135+0000 7f9a0099c640 1 --2- 172.21.15.57:0/514073806 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7f99e006af60 0x7f99e006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T14:01:24.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.144+0000 7f99f97fa640 1 -- 172.21.15.57:0/514073806 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f99e8077230 con 0x7f99fc0cb4c0 2023-10-19T14:01:24.314 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:01:24.312+0000 7f4d21f7a040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-19T14:01:24.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.323+0000 7f9a0199e640 1 -- 172.21.15.57:0/514073806 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f99fc04da70 con 0x7f99fc0cb4c0 2023-10-19T14:01:24.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.323+0000 7f99f97fa640 1 -- 172.21.15.57:0/514073806 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v163) v1 ==== 93+0+10 (secure 0 0 0) 0x7f99e807a850 con 0x7f99fc0cb4c0 2023-10-19T14:01:24.326 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-19T14:01:24.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.325+0000 7f9a0199e640 1 -- 172.21.15.57:0/514073806 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7f99e006af60 msgr2=0x7f99e006d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T14:01:24.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.325+0000 7f9a0199e640 1 --2- 172.21.15.57:0/514073806 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7f99e006af60 0x7f99e006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:24.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.325+0000 7f9a0199e640 1 -- 172.21.15.57:0/514073806 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f99fc0cb4c0 msgr2=0x7f99fc1594d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:24.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.325+0000 7f9a0199e640 1 --2- 172.21.15.57:0/514073806 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f99fc0cb4c0 0x7f99fc1594d0 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7f99e8006010 tx=0x7f99e802da70 comp rx=0 tx=0).stop 2023-10-19T14:01:24.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.326+0000 7f9a0199e640 1 -- 172.21.15.57:0/514073806 shutdown_connections 2023-10-19T14:01:24.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.326+0000 7f9a0199e640 1 --2- 172.21.15.57:0/514073806 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7f99e006af60 0x7f99e006d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:24.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.326+0000 7f9a0199e640 1 --2- 172.21.15.57:0/514073806 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f99fc159040 0x7f99fc15ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:24.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.326+0000 7f9a0199e640 1 --2- 172.21.15.57:0/514073806 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f99fc0cb4c0 0x7f99fc1594d0 unknown :-1 s=CLOSED pgs=1825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:24.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.326+0000 7f9a0199e640 1 --2- 172.21.15.57:0/514073806 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f99fc06d2a0 0x7f99fc1e41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:24.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.326+0000 7f9a0199e640 1 -- 172.21.15.57:0/514073806 >> 172.21.15.57:0/514073806 conn(0x7f99fc0ce8e0 msgr2=0x7f99fc158130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:24.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.326+0000 7f9a0199e640 1 -- 172.21.15.57:0/514073806 shutdown_connections 2023-10-19T14:01:24.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.326+0000 7f9a0199e640 1 -- 172.21.15.57:0/514073806 wait complete. 2023-10-19T14:01:24.339 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-19T14:01:24.359 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:24.357+0000 7f4e31232040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-19T14:01:24.407 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:24.405+0000 7f4e31232040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-19T14:01:24.453 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.451+0000 7f376a27c640 1 -- 172.21.15.57:0/1997338485 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f37640cf340 msgr2=0x7f37640d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:24.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.451+0000 7f376a27c640 1 --2- 172.21.15.57:0/1997338485 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f37640cf340 0x7f37640d1910 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7f376408a120 tx=0x7f375402bdb0 comp rx=0 tx=0).stop 2023-10-19T14:01:24.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.451+0000 7f376a27c640 1 -- 172.21.15.57:0/1997338485 shutdown_connections 2023-10-19T14:01:24.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.451+0000 7f376a27c640 1 --2- 172.21.15.57:0/1997338485 >> v2:172.21.15.179:3300/0 conn(0x7f376415b2a0 0x7f37640cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:24.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.451+0000 7f376a27c640 1 --2- 172.21.15.57:0/1997338485 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f37640cf340 0x7f37640d1910 unknown :-1 s=CLOSED pgs=1827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:24.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.451+0000 7f376a27c640 1 -- 172.21.15.57:0/1997338485 >> 172.21.15.57:0/1997338485 conn(0x7f37640d0d60 msgr2=0x7f37640d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:24.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.451+0000 7f376a27c640 1 -- 172.21.15.57:0/1997338485 shutdown_connections 2023-10-19T14:01:24.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.451+0000 7f376a27c640 1 -- 172.21.15.57:0/1997338485 wait complete. 2023-10-19T14:01:24.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.452+0000 7f376a27c640 1 Processor -- start 2023-10-19T14:01:24.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.452+0000 7f376a27c640 1 -- start start 2023-10-19T14:01:24.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.453+0000 7f376a27c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f37640ceb40 0x7f37640c7e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:24.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.453+0000 7f376a27c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f37640cf340 0x7f3764158470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:24.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.453+0000 7f376a27c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f376415b2a0 0x7f37640c95c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:24.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.453+0000 7f376a27c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37640c8f20 con 0x7f37640cf340 2023-10-19T14:01:24.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.453+0000 7f376a27c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37640c8680 con 0x7f37640ceb40 2023-10-19T14:01:24.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.453+0000 7f376a27c640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37640c8960 con 0x7f376415b2a0 2023-10-19T14:01:24.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.453+0000 7f37637fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f37640cf340 0x7f3764158470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:24.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.453+0000 7f37637fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f37640cf340 0x7f3764158470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40800/0 (socket says 172.21.15.57:40800) 2023-10-19T14:01:24.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.453+0000 7f37637fe640 1 -- 172.21.15.57:0/2104005477 learned_addr learned my addr 172.21.15.57:0/2104005477 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:01:24.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.453+0000 7f3763fff640 1 --2- 172.21.15.57:0/2104005477 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f37640ceb40 0x7f37640c7e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:24.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.453+0000 7f3768819640 1 --2- 172.21.15.57:0/2104005477 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f376415b2a0 0x7f37640c95c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:24.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.453+0000 7f37637fe640 1 -- 172.21.15.57:0/2104005477 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f376415b2a0 msgr2=0x7f37640c95c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:24.461 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.453+0000 7f37637fe640 1 --2- 172.21.15.57:0/2104005477 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f376415b2a0 0x7f37640c95c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:24.461 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.453+0000 7f37637fe640 1 -- 172.21.15.57:0/2104005477 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f37640ceb40 msgr2=0x7f37640c7e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:24.461 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.453+0000 7f37637fe640 1 --2- 172.21.15.57:0/2104005477 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f37640ceb40 0x7f37640c7e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:24.461 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.453+0000 7f37637fe640 1 -- 172.21.15.57:0/2104005477 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3754009560 con 0x7f37640cf340 2023-10-19T14:01:24.461 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.453+0000 7f3763fff640 1 --2- 172.21.15.57:0/2104005477 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f37640ceb40 0x7f37640c7e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T14:01:24.462 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.453+0000 7f37637fe640 1 --2- 172.21.15.57:0/2104005477 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f37640cf340 0x7f3764158470 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7f375800e790 tx=0x7f375800ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:24.462 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.454+0000 7f3760ff9640 1 -- 172.21.15.57:0/2104005477 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3758009580 con 0x7f37640cf340 2023-10-19T14:01:24.462 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.454+0000 7f3760ff9640 1 -- 172.21.15.57:0/2104005477 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f375800cdb0 con 0x7f37640cf340 2023-10-19T14:01:24.462 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.454+0000 7f376a27c640 1 -- 172.21.15.57:0/2104005477 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37640c8c40 con 0x7f37640cf340 2023-10-19T14:01:24.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.454+0000 7f3760ff9640 1 -- 172.21.15.57:0/2104005477 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3758010690 con 0x7f37640cf340 2023-10-19T14:01:24.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.454+0000 7f376a27c640 1 -- 172.21.15.57:0/2104005477 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37640c8370 con 0x7f37640cf340 2023-10-19T14:01:24.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.455+0000 7f376a27c640 1 -- 172.21.15.57:0/2104005477 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37640b3950 con 0x7f37640cf340 2023-10-19T14:01:24.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.455+0000 7f3760ff9640 1 -- 172.21.15.57:0/2104005477 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 87) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f37580107f0 con 0x7f37640cf340 2023-10-19T14:01:24.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.456+0000 7f3760ff9640 1 --2- 172.21.15.57:0/2104005477 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7f375006af60 0x7f375006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:24.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.456+0000 7f3760ff9640 1 -- 172.21.15.57:0/2104005477 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(163..163 src has 1..163) v4 ==== 6489+0+0 (secure 0 0 0) 0x7f3758014070 con 0x7f37640cf340 2023-10-19T14:01:24.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.456+0000 7f3763fff640 1 -- 172.21.15.57:0/2104005477 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7f375006af60 msgr2=0x7f375006d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/4251935070 2023-10-19T14:01:24.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.456+0000 7f3763fff640 1 --2- 172.21.15.57:0/2104005477 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7f375006af60 0x7f375006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T14:01:24.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.460+0000 7f3760ff9640 1 -- 172.21.15.57:0/2104005477 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3758051ca0 con 0x7f37640cf340 2023-10-19T14:01:24.509 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:24.507+0000 7f4e31232040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-19T14:01:24.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.530+0000 7f3760ff9640 1 -- 172.21.15.57:0/2104005477 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mgrmap(e 88) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f3758088700 con 0x7f37640cf340 2023-10-19T14:01:24.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.531+0000 7f3760ff9640 1 -- 172.21.15.57:0/2104005477 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7f375006af60 msgr2=0x7f375006d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T14:01:24.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.531+0000 7f3760ff9640 1 --2- 172.21.15.57:0/2104005477 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7f375006af60 0x7f375006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:24.592 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:24.590+0000 7f4db5df7640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:01:24.593 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:24.590+0000 7f4db5df7640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:01:24.593 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:24.590+0000 7f4db5df7640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:01:24.593 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:24.590+0000 7f4db5df7640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:01:24.593 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:24.590+0000 7f4db5df7640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:01:24.594 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:24.591+0000 7f4db2df1640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:01:24.594 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:24.591+0000 7f4db2df1640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:01:24.595 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:24.591+0000 7f4db2df1640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:01:24.595 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:24.591+0000 7f4db2df1640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:01:24.595 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:24.591+0000 7f4db2df1640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:01:24.604 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:01:24.602+0000 7f4d21f7a040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-19T14:01:24.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.644+0000 7f376a27c640 1 -- 172.21.15.57:0/2104005477 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f37640b3950 con 0x7f37640cf340 2023-10-19T14:01:24.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.644+0000 7f3760ff9640 1 -- 172.21.15.57:0/2104005477 <== mon.0 v2:172.21.15.57:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v164) v1 ==== 104+0+11 (secure 0 0 0) 0x7f37640b3950 con 0x7f37640cf340 2023-10-19T14:01:24.647 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2023-10-19T14:01:24.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.646+0000 7f376a27c640 1 -- 172.21.15.57:0/2104005477 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f37640cf340 msgr2=0x7f3764158470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:24.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.646+0000 7f376a27c640 1 --2- 172.21.15.57:0/2104005477 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f37640cf340 0x7f3764158470 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7f375800e790 tx=0x7f375800ec50 comp rx=0 tx=0).stop 2023-10-19T14:01:24.650 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.646+0000 7f376a27c640 1 -- 172.21.15.57:0/2104005477 shutdown_connections 2023-10-19T14:01:24.650 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.646+0000 7f376a27c640 1 --2- 172.21.15.57:0/2104005477 >> [v2:172.21.15.179:6832/4251935070,v1:172.21.15.179:6833/4251935070] conn(0x7f375006af60 0x7f375006d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:24.650 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.646+0000 7f376a27c640 1 --2- 172.21.15.57:0/2104005477 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f376415b2a0 0x7f37640c95c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:24.650 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.646+0000 7f376a27c640 1 --2- 172.21.15.57:0/2104005477 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f37640cf340 0x7f3764158470 unknown :-1 s=CLOSED pgs=1828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:24.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.646+0000 7f376a27c640 1 --2- 172.21.15.57:0/2104005477 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f37640ceb40 0x7f37640c7e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:24.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.646+0000 7f376a27c640 1 -- 172.21.15.57:0/2104005477 >> 172.21.15.57:0/2104005477 conn(0x7f37640d0d60 msgr2=0x7f37640cc800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:24.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.647+0000 7f376a27c640 1 -- 172.21.15.57:0/2104005477 shutdown_connections 2023-10-19T14:01:24.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.647+0000 7f376a27c640 1 -- 172.21.15.57:0/2104005477 wait complete. 2023-10-19T14:01:24.656 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:01:24.654+0000 7f4d21f7a040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-19T14:01:24.660 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-19T14:01:24.764 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:01:24.762+0000 7f4d21f7a040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-19T14:01:24.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.770+0000 7fcea8ad4640 1 -- 172.21.15.57:0/1093404296 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fce8c0047c0 msgr2=0x7fce8c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:24.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.770+0000 7fcea8ad4640 1 --2- 172.21.15.57:0/1093404296 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fce8c0047c0 0x7fce8c004c40 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7fce98001800 tx=0x7fce8c0057f0 comp rx=0 tx=0).stop 2023-10-19T14:01:24.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.770+0000 7fcea2d9d640 1 -- 172.21.15.57:0/1093404296 reap_dead start 2023-10-19T14:01:24.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.770+0000 7fcea8ad4640 1 -- 172.21.15.57:0/1093404296 shutdown_connections 2023-10-19T14:01:24.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.770+0000 7fcea8ad4640 1 -- 172.21.15.57:0/1093404296 >> 172.21.15.57:0/1093404296 conn(0x7fcea40d0d60 msgr2=0x7fcea40d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:24.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.770+0000 7fcea8ad4640 1 -- 172.21.15.57:0/1093404296 shutdown_connections 2023-10-19T14:01:24.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.770+0000 7fcea8ad4640 1 -- 172.21.15.57:0/1093404296 wait complete. 2023-10-19T14:01:24.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.772+0000 7fcea8ad4640 1 Processor -- start 2023-10-19T14:01:24.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.772+0000 7fcea8ad4640 1 -- start start 2023-10-19T14:01:24.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.772+0000 7fcea8ad4640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fce8c0047c0 0x7fcea40ca1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:24.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.772+0000 7fcea8ad4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcea415e250 0x7fcea40c80a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:24.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.772+0000 7fcea8ad4640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcea40b4540 0x7fcea415c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:24.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.772+0000 7fcea8ad4640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea415b680 con 0x7fcea415e250 2023-10-19T14:01:24.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.772+0000 7fcea8ad4640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea415b8d0 con 0x7fcea40b4540 2023-10-19T14:01:24.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.772+0000 7fcea8ad4640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcea415bbb0 con 0x7fce8c0047c0 2023-10-19T14:01:24.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.773+0000 7fcea1d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcea415e250 0x7fcea40c80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:24.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.773+0000 7fcea1d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcea415e250 0x7fcea40c80a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40810/0 (socket says 172.21.15.57:40810) 2023-10-19T14:01:24.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.773+0000 7fcea1d9b640 1 -- 172.21.15.57:0/73945705 learned_addr learned my addr 172.21.15.57:0/73945705 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:01:24.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.773+0000 7fcea259c640 1 --2- 172.21.15.57:0/73945705 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fce8c0047c0 0x7fcea40ca1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:24.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.773+0000 7fcea2d9d640 1 --2- 172.21.15.57:0/73945705 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcea40b4540 0x7fcea415c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:24.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.773+0000 7fcea1d9b640 1 -- 172.21.15.57:0/73945705 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fce8c0047c0 msgr2=0x7fcea40ca1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:24.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.773+0000 7fcea1d9b640 1 --2- 172.21.15.57:0/73945705 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fce8c0047c0 0x7fcea40ca1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:24.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.773+0000 7fcea1d9b640 1 -- 172.21.15.57:0/73945705 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcea40b4540 msgr2=0x7fcea415c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:24.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.773+0000 7fcea1d9b640 1 --2- 172.21.15.57:0/73945705 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcea40b4540 0x7fcea415c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:24.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.773+0000 7fcea1d9b640 1 -- 172.21.15.57:0/73945705 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcea415be90 con 0x7fcea415e250 2023-10-19T14:01:24.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.773+0000 7fcea259c640 1 --2- 172.21.15.57:0/73945705 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fce8c0047c0 0x7fcea40ca1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:01:24.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.773+0000 7fcea2d9d640 1 --2- 172.21.15.57:0/73945705 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcea40b4540 0x7fcea415c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:01:24.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.773+0000 7fcea1d9b640 1 --2- 172.21.15.57:0/73945705 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcea415e250 0x7fcea40c80a0 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7fce98001920 tx=0x7fce98006300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:24.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.773+0000 7fce92ffd640 1 -- 172.21.15.57:0/73945705 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce9803c040 con 0x7fcea415e250 2023-10-19T14:01:24.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.773+0000 7fce92ffd640 1 -- 172.21.15.57:0/73945705 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fce9802dbb0 con 0x7fcea415e250 2023-10-19T14:01:24.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.773+0000 7fcea8ad4640 1 -- 172.21.15.57:0/73945705 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcea415c170 con 0x7fcea415e250 2023-10-19T14:01:24.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.773+0000 7fce92ffd640 1 -- 172.21.15.57:0/73945705 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce980035f0 con 0x7fcea415e250 2023-10-19T14:01:24.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.773+0000 7fcea8ad4640 1 -- 172.21.15.57:0/73945705 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcea40c85e0 con 0x7fcea415e250 2023-10-19T14:01:24.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.774+0000 7fcea8ad4640 1 -- 172.21.15.57:0/73945705 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcea40c87d0 con 0x7fcea415e250 2023-10-19T14:01:24.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.775+0000 7fce92ffd640 1 -- 172.21.15.57:0/73945705 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 88) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fce98003750 con 0x7fcea415e250 2023-10-19T14:01:24.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.775+0000 7fce92ffd640 1 -- 172.21.15.57:0/73945705 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 6833+0+0 (secure 0 0 0) 0x7fce980aac50 con 0x7fcea415e250 2023-10-19T14:01:24.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.780+0000 7fce92ffd640 1 -- 172.21.15.57:0/73945705 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fce98075b70 con 0x7fcea415e250 2023-10-19T14:01:24.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.955+0000 7fcea8ad4640 1 -- 172.21.15.57:0/73945705 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fcea415c300 con 0x7fcea415e250 2023-10-19T14:01:24.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.955+0000 7fce92ffd640 1 -- 172.21.15.57:0/73945705 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v164) v1 ==== 100+0+11 (secure 0 0 0) 0x7fce98079190 con 0x7fcea415e250 2023-10-19T14:01:24.958 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 64 2023-10-19T14:01:24.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.957+0000 7fcea8ad4640 1 -- 172.21.15.57:0/73945705 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcea415e250 msgr2=0x7fcea40c80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:24.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.957+0000 7fcea8ad4640 1 --2- 172.21.15.57:0/73945705 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcea415e250 0x7fcea40c80a0 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7fce98001920 tx=0x7fce98006300 comp rx=0 tx=0).stop 2023-10-19T14:01:24.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.957+0000 7fcea8ad4640 1 -- 172.21.15.57:0/73945705 shutdown_connections 2023-10-19T14:01:24.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.957+0000 7fcea8ad4640 1 --2- 172.21.15.57:0/73945705 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcea40b4540 0x7fcea415c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:24.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.957+0000 7fcea8ad4640 1 --2- 172.21.15.57:0/73945705 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcea415e250 0x7fcea40c80a0 unknown :-1 s=CLOSED pgs=1832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:24.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.957+0000 7fcea8ad4640 1 --2- 172.21.15.57:0/73945705 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fce8c0047c0 0x7fcea40ca1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:24.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.957+0000 7fcea8ad4640 1 -- 172.21.15.57:0/73945705 >> 172.21.15.57:0/73945705 conn(0x7fcea40d0d60 msgr2=0x7fcea415a1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:24.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.957+0000 7fcea8ad4640 1 -- 172.21.15.57:0/73945705 shutdown_connections 2023-10-19T14:01:24.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:24.957+0000 7fcea8ad4640 1 -- 172.21.15.57:0/73945705 wait complete. 2023-10-19T14:01:24.970 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T14:01:25.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.079+0000 7f55569a9640 1 -- 172.21.15.57:0/514315768 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f55500cf340 msgr2=0x7f55500d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:25.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.079+0000 7f55569a9640 1 --2- 172.21.15.57:0/514315768 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f55500cf340 0x7f55500d1910 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f5540005fd0 tx=0x7f554002bd90 comp rx=0 tx=0).stop 2023-10-19T14:01:25.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.079+0000 7f55569a9640 1 -- 172.21.15.57:0/514315768 shutdown_connections 2023-10-19T14:01:25.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.079+0000 7f55569a9640 1 --2- 172.21.15.57:0/514315768 >> v2:172.21.15.179:3300/0 conn(0x7f55500ceb40 0x7f55500b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:25.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.079+0000 7f55569a9640 1 --2- 172.21.15.57:0/514315768 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f55500cf340 0x7f55500d1910 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:25.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.079+0000 7f55569a9640 1 -- 172.21.15.57:0/514315768 >> 172.21.15.57:0/514315768 conn(0x7f55500d0d60 msgr2=0x7f55500d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:25.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.079+0000 7f55569a9640 1 -- 172.21.15.57:0/514315768 shutdown_connections 2023-10-19T14:01:25.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.079+0000 7f55569a9640 1 -- 172.21.15.57:0/514315768 wait complete. 2023-10-19T14:01:25.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.081+0000 7f55569a9640 1 Processor -- start 2023-10-19T14:01:25.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.081+0000 7f55569a9640 1 -- start start 2023-10-19T14:01:25.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.081+0000 7f55569a9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f55500ceb40 0x7f55500c7e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:25.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.081+0000 7f55569a9640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f55500cf340 0x7f55500b6710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:25.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.081+0000 7f55569a9640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f555015b2a0 0x7f55501510d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:25.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.081+0000 7f55569a9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55500c9440 con 0x7f55500ceb40 2023-10-19T14:01:25.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.081+0000 7f55569a9640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55500b6c50 con 0x7f55500cf340 2023-10-19T14:01:25.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.081+0000 7f55569a9640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55500c8530 con 0x7f555015b2a0 2023-10-19T14:01:25.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.082+0000 7f554ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f55500ceb40 0x7f55500c7e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:25.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.082+0000 7f5554f46640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f555015b2a0 0x7f55501510d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:25.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.082+0000 7f554ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f55500ceb40 0x7f55500c7e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40822/0 (socket says 172.21.15.57:40822) 2023-10-19T14:01:25.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.082+0000 7f554ffff640 1 -- 172.21.15.57:0/734677585 learned_addr learned my addr 172.21.15.57:0/734677585 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:01:25.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.082+0000 7f554f7fe640 1 --2- 172.21.15.57:0/734677585 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f55500cf340 0x7f55500b6710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:25.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.082+0000 7f554ffff640 1 -- 172.21.15.57:0/734677585 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f555015b2a0 msgr2=0x7f55501510d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:25.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.082+0000 7f554ffff640 1 --2- 172.21.15.57:0/734677585 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f555015b2a0 0x7f55501510d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:25.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.082+0000 7f554ffff640 1 -- 172.21.15.57:0/734677585 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f55500cf340 msgr2=0x7f55500b6710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:25.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.082+0000 7f554ffff640 1 --2- 172.21.15.57:0/734677585 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f55500cf340 0x7f55500b6710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:25.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.082+0000 7f554ffff640 1 -- 172.21.15.57:0/734677585 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5548009840 con 0x7f55500ceb40 2023-10-19T14:01:25.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.082+0000 7f5554f46640 1 --2- 172.21.15.57:0/734677585 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f555015b2a0 0x7f55501510d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T14:01:25.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.082+0000 7f554f7fe640 1 --2- 172.21.15.57:0/734677585 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f55500cf340 0x7f55500b6710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T14:01:25.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.082+0000 7f554ffff640 1 --2- 172.21.15.57:0/734677585 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f55500ceb40 0x7f55500c7e30 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7f5540009850 tx=0x7f554002d9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:25.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.082+0000 7f554cff9640 1 -- 172.21.15.57:0/734677585 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f554002ddf0 con 0x7f55500ceb40 2023-10-19T14:01:25.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.082+0000 7f55569a9640 1 -- 172.21.15.57:0/734677585 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5540009560 con 0x7f55500ceb40 2023-10-19T14:01:25.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.082+0000 7f554cff9640 1 -- 172.21.15.57:0/734677585 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f55400036e0 con 0x7f55500ceb40 2023-10-19T14:01:25.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.082+0000 7f554cff9640 1 -- 172.21.15.57:0/734677585 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5540003ad0 con 0x7f55500ceb40 2023-10-19T14:01:25.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.082+0000 7f55569a9640 1 -- 172.21.15.57:0/734677585 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55500c8af0 con 0x7f55500ceb40 2023-10-19T14:01:25.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.083+0000 7f55569a9640 1 -- 172.21.15.57:0/734677585 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55500690e0 con 0x7f55500ceb40 2023-10-19T14:01:25.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.084+0000 7f554cff9640 1 -- 172.21.15.57:0/734677585 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 88) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f5540003c30 con 0x7f55500ceb40 2023-10-19T14:01:25.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.084+0000 7f554cff9640 1 -- 172.21.15.57:0/734677585 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 6833+0+0 (secure 0 0 0) 0x7f55400abd10 con 0x7f55500ceb40 2023-10-19T14:01:25.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.088+0000 7f554cff9640 1 -- 172.21.15.57:0/734677585 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f5540076e40 con 0x7f55500ceb40 2023-10-19T14:01:25.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.298+0000 7f55569a9640 1 -- 172.21.15.57:0/734677585 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f55500cb9b0 con 0x7f55500ceb40 2023-10-19T14:01:25.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.298+0000 7f554cff9640 1 -- 172.21.15.57:0/734677585 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+4918 (secure 0 0 0) 0x7f5540042020 con 0x7f55500ceb40 2023-10-19T14:01:25.301 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:01:25.301 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":63,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7733,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/1818414654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":1818414654},{"type":"v1","addr":"172.21.15.179:6836","nonce":1818414654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":8647,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/490712721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":490712721},{"type":"v1","addr":"172.21.15.57:6835","nonce":490712721}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":8656,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/2005811112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":2005811112},{"type":"v1","addr":"172.21.15.179:6837","nonce":2005811112}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T13:04:58.134572+0000","modified":"2023-10-19T13:05:00.138505+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9618},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9618":{"gid":9618,"name":"c","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.15.57:6833/1460919655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":1460919655},{"type":"v1","addr":"172.21.15.57:6833","nonce":1460919655}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2023-10-19T14:01:25.302 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 63 2023-10-19T14:01:25.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.300+0000 7f55569a9640 1 -- 172.21.15.57:0/734677585 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f55500ceb40 msgr2=0x7f55500c7e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:25.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.300+0000 7f55569a9640 1 --2- 172.21.15.57:0/734677585 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f55500ceb40 0x7f55500c7e30 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7f5540009850 tx=0x7f554002d9f0 comp rx=0 tx=0).stop 2023-10-19T14:01:25.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.300+0000 7f55569a9640 1 -- 172.21.15.57:0/734677585 shutdown_connections 2023-10-19T14:01:25.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.300+0000 7f55569a9640 1 --2- 172.21.15.57:0/734677585 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f555015b2a0 0x7f55501510d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:25.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.300+0000 7f55569a9640 1 --2- 172.21.15.57:0/734677585 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f55500cf340 0x7f55500b6710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:25.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.300+0000 7f55569a9640 1 --2- 172.21.15.57:0/734677585 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f55500ceb40 0x7f55500c7e30 unknown :-1 s=CLOSED pgs=1833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:25.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.300+0000 7f55569a9640 1 -- 172.21.15.57:0/734677585 >> 172.21.15.57:0/734677585 conn(0x7f55500d0d60 msgr2=0x7f5550159ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:25.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.300+0000 7f55569a9640 1 -- 172.21.15.57:0/734677585 shutdown_connections 2023-10-19T14:01:25.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.300+0000 7f55569a9640 1 -- 172.21.15.57:0/734677585 wait complete. 2023-10-19T14:01:25.313 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T14:01:25.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.419+0000 7f296444b640 1 -- 172.21.15.57:0/301654578 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f295c0cf340 msgr2=0x7f295c0d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:25.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.419+0000 7f296444b640 1 --2- 172.21.15.57:0/301654578 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f295c0cf340 0x7f295c0d1910 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x7f294c005fd0 tx=0x7f294c02c250 comp rx=0 tx=0).stop 2023-10-19T14:01:25.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.419+0000 7f296444b640 1 -- 172.21.15.57:0/301654578 shutdown_connections 2023-10-19T14:01:25.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.419+0000 7f296444b640 1 --2- 172.21.15.57:0/301654578 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f295c0cf340 0x7f295c0d1910 unknown :-1 s=CLOSED pgs=1835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:25.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.419+0000 7f296444b640 1 -- 172.21.15.57:0/301654578 >> 172.21.15.57:0/301654578 conn(0x7f295c0d0d60 msgr2=0x7f295c0d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:25.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.419+0000 7f296444b640 1 -- 172.21.15.57:0/301654578 shutdown_connections 2023-10-19T14:01:25.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.419+0000 7f296444b640 1 -- 172.21.15.57:0/301654578 wait complete. 2023-10-19T14:01:25.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.420+0000 7f296444b640 1 Processor -- start 2023-10-19T14:01:25.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.420+0000 7f296444b640 1 -- start start 2023-10-19T14:01:25.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.421+0000 7f296444b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f295c0ceb40 0x7f295c158c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:25.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.421+0000 7f296444b640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f295c0cf340 0x7f295c156ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:25.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.421+0000 7f296444b640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f295c15b2a0 0x7f295c15d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:25.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.421+0000 7f296444b640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f295c0b6750 con 0x7f295c0ceb40 2023-10-19T14:01:25.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.421+0000 7f296444b640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f295c1591c0 con 0x7f295c15b2a0 2023-10-19T14:01:25.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.421+0000 7f296444b640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f295c15ccf0 con 0x7f295c0cf340 2023-10-19T14:01:25.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.421+0000 7f29621e7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f295c0ceb40 0x7f295c158c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:25.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.421+0000 7f29621e7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f295c0ceb40 0x7f295c158c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40850/0 (socket says 172.21.15.57:40850) 2023-10-19T14:01:25.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.421+0000 7f29621e7640 1 -- 172.21.15.57:0/1719809884 learned_addr learned my addr 172.21.15.57:0/1719809884 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:01:25.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.421+0000 7f29629e8640 1 --2- 172.21.15.57:0/1719809884 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f295c15b2a0 0x7f295c15d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:25.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.421+0000 7f29619e6640 1 --2- 172.21.15.57:0/1719809884 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f295c0cf340 0x7f295c156ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:25.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.422+0000 7f29621e7640 1 -- 172.21.15.57:0/1719809884 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f295c0cf340 msgr2=0x7f295c156ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:25.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.422+0000 7f29621e7640 1 --2- 172.21.15.57:0/1719809884 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f295c0cf340 0x7f295c156ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:25.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.422+0000 7f29621e7640 1 -- 172.21.15.57:0/1719809884 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f295c15b2a0 msgr2=0x7f295c15d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:25.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.422+0000 7f29621e7640 1 --2- 172.21.15.57:0/1719809884 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f295c15b2a0 0x7f295c15d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:25.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.422+0000 7f29621e7640 1 -- 172.21.15.57:0/1719809884 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f294c009900 con 0x7f295c0ceb40 2023-10-19T14:01:25.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.422+0000 7f29619e6640 1 --2- 172.21.15.57:0/1719809884 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f295c0cf340 0x7f295c156ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:01:25.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.422+0000 7f29621e7640 1 --2- 172.21.15.57:0/1719809884 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f295c0ceb40 0x7f295c158c80 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7f294c006010 tx=0x7f294c035e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:25.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.422+0000 7f295affd640 1 -- 172.21.15.57:0/1719809884 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f294c034440 con 0x7f295c0ceb40 2023-10-19T14:01:25.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.422+0000 7f295affd640 1 -- 172.21.15.57:0/1719809884 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f294c004250 con 0x7f295c0ceb40 2023-10-19T14:01:25.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.422+0000 7f296444b640 1 -- 172.21.15.57:0/1719809884 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f295c0b3980 con 0x7f295c0ceb40 2023-10-19T14:01:25.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.422+0000 7f295affd640 1 -- 172.21.15.57:0/1719809884 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f294c004620 con 0x7f295c0ceb40 2023-10-19T14:01:25.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.422+0000 7f296444b640 1 -- 172.21.15.57:0/1719809884 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f295c0b3f40 con 0x7f295c0ceb40 2023-10-19T14:01:25.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.423+0000 7f296444b640 1 -- 172.21.15.57:0/1719809884 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f295c0690e0 con 0x7f295c0ceb40 2023-10-19T14:01:25.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.423+0000 7f295affd640 1 -- 172.21.15.57:0/1719809884 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 88) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f294c03b040 con 0x7f295c0ceb40 2023-10-19T14:01:25.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.424+0000 7f295affd640 1 -- 172.21.15.57:0/1719809884 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 6833+0+0 (secure 0 0 0) 0x7f294c0ac320 con 0x7f295c0ceb40 2023-10-19T14:01:25.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.428+0000 7f295affd640 1 -- 172.21.15.57:0/1719809884 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f294c0772c0 con 0x7f295c0ceb40 2023-10-19T14:01:25.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.530+0000 7f295affd640 1 -- 172.21.15.57:0/1719809884 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mgrmap(e 89) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f294c00a040 con 0x7f295c0ceb40 2023-10-19T14:01:25.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.531+0000 7f295affd640 1 --2- 172.21.15.57:0/1719809884 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f2948070c70 0x7f2948073060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:25.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.535+0000 7f29619e6640 1 --2- 172.21.15.57:0/1719809884 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f2948070c70 0x7f2948073060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:25.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.537+0000 7f29619e6640 1 --2- 172.21.15.57:0/1719809884 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f2948070c70 0x7f2948073060 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f295c156930 tx=0x7f2950009040 comp rx=0 tx=0).ready entity=mgr.8923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:25.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.612+0000 7f296444b640 1 -- 172.21.15.57:0/1719809884 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f295c0d0b40 con 0x7f295c0ceb40 2023-10-19T14:01:25.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.613+0000 7f295affd640 1 -- 172.21.15.57:0/1719809884 <== mon.0 v2:172.21.15.57:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v164) v1 ==== 74+0+17966 (secure 0 0 0) 0x7f294c0901f0 con 0x7f295c0ceb40 2023-10-19T14:01:25.616 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:01:25.616 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":164,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T14:01:24.521727+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2023-10-19T13:04:56.433332+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"156","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2023-10-19T13:04:57.112749+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"164","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":113,"snap_epoch":164,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.179:0/3898405031":"2023-10-20T14:01:24.521670+0000","172.21.15.179:6833/4251935070":"2023-10-20T14:01:24.521670+0000","172.21.15.179:6832/4251935070":"2023-10-20T14:01:24.521670+0000","172.21.15.179:0/4276889975":"2023-10-20T14:01:24.521670+0000","172.21.15.179:0/1193476158":"2023-10-20T14:01:24.521670+0000","172.21.15.179:0/273107227":"2023-10-20T14:01:24.521670+0000","172.21.15.179:0/3350224887":"2023-10-20T14:01:24.521670+0000","172.21.15.179:0/744328422":"2023-10-20T13:05:09.216573+0000","172.21.15.179:0/1909872102":"2023-10-20T13:05:09.216573+0000","172.21.15.179:0/1800709778":"2023-10-20T14:01:24.521670+0000","172.21.15.179:0/1880188133":"2023-10-20T13:05:09.216573+0000","172.21.15.179:0/40128169":"2023-10-20T13:05:09.216573+0000","172.21.15.179:6833/2581690007":"2023-10-20T13:05:09.216573+0000","172.21.15.179:0/3857987280":"2023-10-20T13:05:09.216573+0000","172.21.15.179:6832/2581690007":"2023-10-20T13:05:09.216573+0000","172.21.15.179:0/57720407":"2023-10-20T13:05:09.216573+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[{"pool":20,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T14:01:25.618 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.615+0000 7f296444b640 1 -- 172.21.15.57:0/1719809884 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f2948070c70 msgr2=0x7f2948073060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:25.618 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.615+0000 7f296444b640 1 --2- 172.21.15.57:0/1719809884 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f2948070c70 0x7f2948073060 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f295c156930 tx=0x7f2950009040 comp rx=0 tx=0).stop 2023-10-19T14:01:25.618 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.615+0000 7f296444b640 1 -- 172.21.15.57:0/1719809884 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f295c0ceb40 msgr2=0x7f295c158c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:25.618 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.615+0000 7f296444b640 1 --2- 172.21.15.57:0/1719809884 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f295c0ceb40 0x7f295c158c80 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7f294c006010 tx=0x7f294c035e30 comp rx=0 tx=0).stop 2023-10-19T14:01:25.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.615+0000 7f296444b640 1 -- 172.21.15.57:0/1719809884 shutdown_connections 2023-10-19T14:01:25.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.615+0000 7f296444b640 1 --2- 172.21.15.57:0/1719809884 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f2948070c70 0x7f2948073060 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:25.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.615+0000 7f296444b640 1 --2- 172.21.15.57:0/1719809884 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f295c15b2a0 0x7f295c15d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:25.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.615+0000 7f296444b640 1 --2- 172.21.15.57:0/1719809884 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f295c0cf340 0x7f295c156ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:25.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.615+0000 7f296444b640 1 --2- 172.21.15.57:0/1719809884 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f295c0ceb40 0x7f295c158c80 unknown :-1 s=CLOSED pgs=1836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:25.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.615+0000 7f296444b640 1 -- 172.21.15.57:0/1719809884 >> 172.21.15.57:0/1719809884 conn(0x7f295c0d0d60 msgr2=0x7f295c0ae330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:25.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.615+0000 7f296444b640 1 -- 172.21.15.57:0/1719809884 shutdown_connections 2023-10-19T14:01:25.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.615+0000 7f296444b640 1 -- 172.21.15.57:0/1719809884 wait complete. 2023-10-19T14:01:25.628 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-19T14:01:25.629 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T14:01:25.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.735+0000 7fdeea539640 1 -- 172.21.15.57:0/2205141096 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdee415ccb0 msgr2=0x7fdee40c9750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:25.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.735+0000 7fdeea539640 1 --2- 172.21.15.57:0/2205141096 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdee415ccb0 0x7fdee40c9750 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7fdee415d3c0 tx=0x7fdedc02c1b0 comp rx=0 tx=0).stop 2023-10-19T14:01:25.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.735+0000 7fdeea539640 1 -- 172.21.15.57:0/2205141096 shutdown_connections 2023-10-19T14:01:25.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.735+0000 7fdeea539640 1 --2- 172.21.15.57:0/2205141096 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdee415ccb0 0x7fdee40c9750 unknown :-1 s=CLOSED pgs=1837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:25.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.735+0000 7fdeea539640 1 --2- 172.21.15.57:0/2205141096 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdee406b410 0x7fdee415c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:25.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.735+0000 7fdeea539640 1 -- 172.21.15.57:0/2205141096 >> 172.21.15.57:0/2205141096 conn(0x7fdee4069930 msgr2=0x7fdee4069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:25.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.735+0000 7fdeea539640 1 -- 172.21.15.57:0/2205141096 shutdown_connections 2023-10-19T14:01:25.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.735+0000 7fdeea539640 1 -- 172.21.15.57:0/2205141096 wait complete. 2023-10-19T14:01:25.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.737+0000 7fdeea539640 1 Processor -- start 2023-10-19T14:01:25.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.737+0000 7fdeea539640 1 -- start start 2023-10-19T14:01:25.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.737+0000 7fdeea539640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdee406b030 0x7fdee413d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:25.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.737+0000 7fdeea539640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdee406b410 0x7fdee4152bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:25.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.737+0000 7fdeea539640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdee415ccb0 0x7fdee40ca870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:25.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.737+0000 7fdeea539640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdee41f6dd0 con 0x7fdee406b030 2023-10-19T14:01:25.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.737+0000 7fdeea539640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdee413e210 con 0x7fdee415ccb0 2023-10-19T14:01:25.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.737+0000 7fdeea539640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdee413d970 con 0x7fdee406b410 2023-10-19T14:01:25.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.738+0000 7fdee3fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdee406b030 0x7fdee413d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:25.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.738+0000 7fdee3fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdee406b030 0x7fdee413d120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40868/0 (socket says 172.21.15.57:40868) 2023-10-19T14:01:25.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.738+0000 7fdee3fff640 1 -- 172.21.15.57:0/3902789659 learned_addr learned my addr 172.21.15.57:0/3902789659 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:01:25.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.738+0000 7fdee8ad6640 1 --2- 172.21.15.57:0/3902789659 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdee415ccb0 0x7fdee40ca870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:25.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.738+0000 7fdee37fe640 1 --2- 172.21.15.57:0/3902789659 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdee406b410 0x7fdee4152bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:25.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.738+0000 7fdee3fff640 1 -- 172.21.15.57:0/3902789659 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdee406b410 msgr2=0x7fdee4152bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:25.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.738+0000 7fdee3fff640 1 --2- 172.21.15.57:0/3902789659 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdee406b410 0x7fdee4152bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:25.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.738+0000 7fdee3fff640 1 -- 172.21.15.57:0/3902789659 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdee415ccb0 msgr2=0x7fdee40ca870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:25.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.738+0000 7fdee3fff640 1 --2- 172.21.15.57:0/3902789659 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdee415ccb0 0x7fdee40ca870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:25.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.738+0000 7fdee3fff640 1 -- 172.21.15.57:0/3902789659 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdedc009900 con 0x7fdee406b030 2023-10-19T14:01:25.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.738+0000 7fdee3fff640 1 --2- 172.21.15.57:0/3902789659 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdee406b030 0x7fdee413d120 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7fded400c410 tx=0x7fded400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:25.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.738+0000 7fdee0ff9640 1 -- 172.21.15.57:0/3902789659 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fded40035f0 con 0x7fdee406b030 2023-10-19T14:01:25.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.738+0000 7fdee0ff9640 1 -- 172.21.15.57:0/3902789659 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fded4003750 con 0x7fdee406b030 2023-10-19T14:01:25.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.738+0000 7fdeea539640 1 -- 172.21.15.57:0/3902789659 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdee413dc50 con 0x7fdee406b030 2023-10-19T14:01:25.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.738+0000 7fdee0ff9640 1 -- 172.21.15.57:0/3902789659 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fded4003ba0 con 0x7fdee406b030 2023-10-19T14:01:25.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.738+0000 7fdeea539640 1 -- 172.21.15.57:0/3902789659 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdee413d660 con 0x7fdee406b030 2023-10-19T14:01:25.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.739+0000 7fdeea539640 1 -- 172.21.15.57:0/3902789659 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdee40690e0 con 0x7fdee406b030 2023-10-19T14:01:25.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.740+0000 7fdee0ff9640 1 -- 172.21.15.57:0/3902789659 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 89) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fded4003d00 con 0x7fdee406b030 2023-10-19T14:01:25.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.740+0000 7fdee0ff9640 1 --2- 172.21.15.57:0/3902789659 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7fded006acc0 0x7fded006d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:25.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.740+0000 7fdee0ff9640 1 -- 172.21.15.57:0/3902789659 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 6833+0+0 (secure 0 0 0) 0x7fded4088e60 con 0x7fdee406b030 2023-10-19T14:01:25.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.740+0000 7fdee37fe640 1 --2- 172.21.15.57:0/3902789659 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7fded006acc0 0x7fded006d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:25.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.741+0000 7fdee37fe640 1 --2- 172.21.15.57:0/3902789659 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7fded006acc0 0x7fded006d180 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fdee40aad60 tx=0x7fded8008040 comp rx=0 tx=0).ready entity=mgr.8923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:25.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.744+0000 7fdee0ff9640 1 -- 172.21.15.57:0/3902789659 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fded4053de0 con 0x7fdee406b030 2023-10-19T14:01:25.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.956+0000 7fdeea539640 1 -- 172.21.15.57:0/3902789659 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdee40cc460 con 0x7fdee406b030 2023-10-19T14:01:25.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.957+0000 7fdee0ff9640 1 -- 172.21.15.57:0/3902789659 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+4918 (secure 0 0 0) 0x7fded4017070 con 0x7fdee406b030 2023-10-19T14:01:25.960 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:01:25.960 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":63,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7733,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/1818414654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":1818414654},{"type":"v1","addr":"172.21.15.179:6836","nonce":1818414654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":8647,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/490712721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":490712721},{"type":"v1","addr":"172.21.15.57:6835","nonce":490712721}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":8656,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/2005811112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":2005811112},{"type":"v1","addr":"172.21.15.179:6837","nonce":2005811112}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T13:04:58.134572+0000","modified":"2023-10-19T13:05:00.138505+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9618},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9618":{"gid":9618,"name":"c","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.15.57:6833/1460919655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":1460919655},{"type":"v1","addr":"172.21.15.57:6833","nonce":1460919655}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2023-10-19T14:01:25.961 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 63 2023-10-19T14:01:25.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.959+0000 7fdeea539640 1 -- 172.21.15.57:0/3902789659 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7fded006acc0 msgr2=0x7fded006d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:25.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.959+0000 7fdeea539640 1 --2- 172.21.15.57:0/3902789659 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7fded006acc0 0x7fded006d180 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fdee40aad60 tx=0x7fded8008040 comp rx=0 tx=0).stop 2023-10-19T14:01:25.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.959+0000 7fdeea539640 1 -- 172.21.15.57:0/3902789659 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdee406b030 msgr2=0x7fdee413d120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:25.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.959+0000 7fdeea539640 1 --2- 172.21.15.57:0/3902789659 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdee406b030 0x7fdee413d120 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7fded400c410 tx=0x7fded400c8d0 comp rx=0 tx=0).stop 2023-10-19T14:01:25.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.959+0000 7fdeea539640 1 -- 172.21.15.57:0/3902789659 shutdown_connections 2023-10-19T14:01:25.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.959+0000 7fdeea539640 1 --2- 172.21.15.57:0/3902789659 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7fded006acc0 0x7fded006d180 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:25.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.959+0000 7fdeea539640 1 --2- 172.21.15.57:0/3902789659 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdee415ccb0 0x7fdee40ca870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:25.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.959+0000 7fdeea539640 1 --2- 172.21.15.57:0/3902789659 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdee406b410 0x7fdee4152bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:25.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.959+0000 7fdeea539640 1 --2- 172.21.15.57:0/3902789659 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdee406b030 0x7fdee413d120 unknown :-1 s=CLOSED pgs=1838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:25.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.959+0000 7fdeea539640 1 -- 172.21.15.57:0/3902789659 >> 172.21.15.57:0/3902789659 conn(0x7fdee4069930 msgr2=0x7fdee415a490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:25.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.959+0000 7fdeea539640 1 -- 172.21.15.57:0/3902789659 shutdown_connections 2023-10-19T14:01:25.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:25.959+0000 7fdeea539640 1 -- 172.21.15.57:0/3902789659 wait complete. 2023-10-19T14:01:25.972 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T14:01:26.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.078+0000 7fc5eaaf7640 1 -- 172.21.15.57:0/750985117 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc5e415d400 msgr2=0x7fc5e40cdfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:26.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.078+0000 7fc5eaaf7640 1 --2- 172.21.15.57:0/750985117 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc5e415d400 0x7fc5e40cdfb0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fc5dc005fd0 tx=0x7fc5dc02c050 comp rx=0 tx=0).stop 2023-10-19T14:01:26.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.079+0000 7fc5eaaf7640 1 -- 172.21.15.57:0/750985117 shutdown_connections 2023-10-19T14:01:26.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.079+0000 7fc5eaaf7640 1 --2- 172.21.15.57:0/750985117 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc5e415d400 0x7fc5e40cdfb0 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:26.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.079+0000 7fc5eaaf7640 1 --2- 172.21.15.57:0/750985117 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc5e4159060 0x7fc5e406d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:26.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.079+0000 7fc5eaaf7640 1 -- 172.21.15.57:0/750985117 >> 172.21.15.57:0/750985117 conn(0x7fc5e40b0670 msgr2=0x7fc5e40b0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:26.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.079+0000 7fc5eaaf7640 1 -- 172.21.15.57:0/750985117 shutdown_connections 2023-10-19T14:01:26.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.079+0000 7fc5eaaf7640 1 -- 172.21.15.57:0/750985117 wait complete. 2023-10-19T14:01:26.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.080+0000 7fc5eaaf7640 1 Processor -- start 2023-10-19T14:01:26.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.080+0000 7fc5eaaf7640 1 -- start start 2023-10-19T14:01:26.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.081+0000 7fc5eaaf7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5e40cdb30 0x7fc5e40c7cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:26.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.081+0000 7fc5eaaf7640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc5e4159060 0x7fc5e40c3560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:26.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.081+0000 7fc5eaaf7640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc5e415d400 0x7fc5e4150b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:26.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.081+0000 7fc5eaaf7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e40c3160 con 0x7fc5e40cdb30 2023-10-19T14:01:26.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.081+0000 7fc5eaaf7640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e40c81f0 con 0x7fc5e415d400 2023-10-19T14:01:26.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.081+0000 7fc5eaaf7640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e40c24e0 con 0x7fc5e4159060 2023-10-19T14:01:26.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.081+0000 7fc5e8893640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5e40cdb30 0x7fc5e40c7cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:26.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.081+0000 7fc5e8893640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5e40cdb30 0x7fc5e40c7cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40870/0 (socket says 172.21.15.57:40870) 2023-10-19T14:01:26.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.081+0000 7fc5e8893640 1 -- 172.21.15.57:0/3208365965 learned_addr learned my addr 172.21.15.57:0/3208365965 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:01:26.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.081+0000 7fc5e9094640 1 --2- 172.21.15.57:0/3208365965 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc5e415d400 0x7fc5e4150b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:26.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.081+0000 7fc5e3fff640 1 --2- 172.21.15.57:0/3208365965 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc5e4159060 0x7fc5e40c3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:26.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.081+0000 7fc5e8893640 1 -- 172.21.15.57:0/3208365965 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc5e4159060 msgr2=0x7fc5e40c3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:26.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.081+0000 7fc5e8893640 1 --2- 172.21.15.57:0/3208365965 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc5e4159060 0x7fc5e40c3560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:26.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.081+0000 7fc5e8893640 1 -- 172.21.15.57:0/3208365965 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc5e415d400 msgr2=0x7fc5e4150b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:26.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.081+0000 7fc5e8893640 1 --2- 172.21.15.57:0/3208365965 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc5e415d400 0x7fc5e4150b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:26.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.081+0000 7fc5e8893640 1 -- 172.21.15.57:0/3208365965 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5d8009490 con 0x7fc5e40cdb30 2023-10-19T14:01:26.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.081+0000 7fc5e3fff640 1 --2- 172.21.15.57:0/3208365965 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc5e4159060 0x7fc5e40c3560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:01:26.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.081+0000 7fc5e9094640 1 --2- 172.21.15.57:0/3208365965 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc5e415d400 0x7fc5e4150b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:01:26.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.081+0000 7fc5e8893640 1 --2- 172.21.15.57:0/3208365965 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5e40cdb30 0x7fc5e40c7cb0 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x7fc5d800ea90 tx=0x7fc5d800ef50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:26.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.081+0000 7fc5e17fa640 1 -- 172.21.15.57:0/3208365965 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5d800ce40 con 0x7fc5e40cdb30 2023-10-19T14:01:26.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.081+0000 7fc5e17fa640 1 -- 172.21.15.57:0/3208365965 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fc5d8004020 con 0x7fc5e40cdb30 2023-10-19T14:01:26.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.081+0000 7fc5eaaf7640 1 -- 172.21.15.57:0/3208365965 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5dc009840 con 0x7fc5e40cdb30 2023-10-19T14:01:26.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.081+0000 7fc5e17fa640 1 -- 172.21.15.57:0/3208365965 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5d8004430 con 0x7fc5e40cdb30 2023-10-19T14:01:26.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.082+0000 7fc5eaaf7640 1 -- 172.21.15.57:0/3208365965 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5e40c2d80 con 0x7fc5e40cdb30 2023-10-19T14:01:26.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.082+0000 7fc5eaaf7640 1 -- 172.21.15.57:0/3208365965 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5e40c28d0 con 0x7fc5e40cdb30 2023-10-19T14:01:26.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.083+0000 7fc5e17fa640 1 -- 172.21.15.57:0/3208365965 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 89) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fc5d80035f0 con 0x7fc5e40cdb30 2023-10-19T14:01:26.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.083+0000 7fc5e17fa640 1 --2- 172.21.15.57:0/3208365965 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7fc5c806ac90 0x7fc5c806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:26.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.083+0000 7fc5e17fa640 1 -- 172.21.15.57:0/3208365965 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 6833+0+0 (secure 0 0 0) 0x7fc5d8014070 con 0x7fc5e40cdb30 2023-10-19T14:01:26.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.083+0000 7fc5e3fff640 1 --2- 172.21.15.57:0/3208365965 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7fc5c806ac90 0x7fc5c806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:26.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.084+0000 7fc5e3fff640 1 --2- 172.21.15.57:0/3208365965 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7fc5c806ac90 0x7fc5c806d150 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc5e40c4210 tx=0x7fc5d0009040 comp rx=0 tx=0).ready entity=mgr.8923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:26.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.087+0000 7fc5e17fa640 1 -- 172.21.15.57:0/3208365965 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc5d80552f0 con 0x7fc5e40cdb30 2023-10-19T14:01:26.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.299+0000 7fc5eaaf7640 1 -- 172.21.15.57:0/3208365965 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc5e40cc4d0 con 0x7fc5e40cdb30 2023-10-19T14:01:26.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.299+0000 7fc5e17fa640 1 -- 172.21.15.57:0/3208365965 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+4918 (secure 0 0 0) 0x7fc5d8058910 con 0x7fc5e40cdb30 2023-10-19T14:01:26.302 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:01:26.303 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":63,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7733,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/1818414654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":1818414654},{"type":"v1","addr":"172.21.15.179:6836","nonce":1818414654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":8647,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/490712721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":490712721},{"type":"v1","addr":"172.21.15.57:6835","nonce":490712721}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":8656,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/2005811112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":2005811112},{"type":"v1","addr":"172.21.15.179:6837","nonce":2005811112}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T13:04:58.134572+0000","modified":"2023-10-19T13:05:00.138505+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9618},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9618":{"gid":9618,"name":"c","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.15.57:6833/1460919655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":1460919655},{"type":"v1","addr":"172.21.15.57:6833","nonce":1460919655}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2023-10-19T14:01:26.304 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 63 2023-10-19T14:01:26.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.301+0000 7fc5eaaf7640 1 -- 172.21.15.57:0/3208365965 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7fc5c806ac90 msgr2=0x7fc5c806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:26.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.301+0000 7fc5eaaf7640 1 --2- 172.21.15.57:0/3208365965 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7fc5c806ac90 0x7fc5c806d150 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc5e40c4210 tx=0x7fc5d0009040 comp rx=0 tx=0).stop 2023-10-19T14:01:26.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.301+0000 7fc5eaaf7640 1 -- 172.21.15.57:0/3208365965 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5e40cdb30 msgr2=0x7fc5e40c7cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:26.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.301+0000 7fc5eaaf7640 1 --2- 172.21.15.57:0/3208365965 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5e40cdb30 0x7fc5e40c7cb0 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x7fc5d800ea90 tx=0x7fc5d800ef50 comp rx=0 tx=0).stop 2023-10-19T14:01:26.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.301+0000 7fc5eaaf7640 1 -- 172.21.15.57:0/3208365965 shutdown_connections 2023-10-19T14:01:26.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.301+0000 7fc5eaaf7640 1 --2- 172.21.15.57:0/3208365965 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7fc5c806ac90 0x7fc5c806d150 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:26.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.301+0000 7fc5eaaf7640 1 --2- 172.21.15.57:0/3208365965 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc5e415d400 0x7fc5e4150b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:26.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.301+0000 7fc5eaaf7640 1 --2- 172.21.15.57:0/3208365965 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc5e4159060 0x7fc5e40c3560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:26.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.301+0000 7fc5eaaf7640 1 --2- 172.21.15.57:0/3208365965 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5e40cdb30 0x7fc5e40c7cb0 unknown :-1 s=CLOSED pgs=1839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:26.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.301+0000 7fc5eaaf7640 1 -- 172.21.15.57:0/3208365965 >> 172.21.15.57:0/3208365965 conn(0x7fc5e40b0670 msgr2=0x7fc5e40aec00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:26.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.301+0000 7fc5eaaf7640 1 -- 172.21.15.57:0/3208365965 shutdown_connections 2023-10-19T14:01:26.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.301+0000 7fc5eaaf7640 1 -- 172.21.15.57:0/3208365965 wait complete. 2023-10-19T14:01:26.315 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T14:01:26.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.422+0000 7f05ed9a5640 1 -- 172.21.15.57:0/176119990 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f05e813d350 msgr2=0x7f05e80cbd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:26.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.422+0000 7f05ed9a5640 1 --2- 172.21.15.57:0/176119990 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f05e813d350 0x7f05e80cbd60 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f05d0005fd0 tx=0x7f05d002be10 comp rx=0 tx=0).stop 2023-10-19T14:01:26.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.422+0000 7f05ed9a5640 1 -- 172.21.15.57:0/176119990 shutdown_connections 2023-10-19T14:01:26.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.422+0000 7f05ed9a5640 1 --2- 172.21.15.57:0/176119990 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f05e8070350 0x7f05e80707c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:26.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.422+0000 7f05ed9a5640 1 --2- 172.21.15.57:0/176119990 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f05e813d350 0x7f05e80cbd60 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:26.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.422+0000 7f05ed9a5640 1 -- 172.21.15.57:0/176119990 >> 172.21.15.57:0/176119990 conn(0x7f05e8069930 msgr2=0x7f05e8069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:26.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.422+0000 7f05ed9a5640 1 -- 172.21.15.57:0/176119990 shutdown_connections 2023-10-19T14:01:26.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.422+0000 7f05ed9a5640 1 -- 172.21.15.57:0/176119990 wait complete. 2023-10-19T14:01:26.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.423+0000 7f05ed9a5640 1 Processor -- start 2023-10-19T14:01:26.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.423+0000 7f05ed9a5640 1 -- start start 2023-10-19T14:01:26.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.424+0000 7f05ed9a5640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f05e8070350 0x7f05e8151c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:26.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.424+0000 7f05ed9a5640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f05e80cb980 0x7f05e81298b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:26.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.424+0000 7f05ed9a5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f05e812a4e0 0x7f05e8129df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:26.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.424+0000 7f05ed9a5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05e8154300 con 0x7f05e812a4e0 2023-10-19T14:01:26.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.424+0000 7f05ed9a5640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05e8154e10 con 0x7f05e8070350 2023-10-19T14:01:26.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.424+0000 7f05ed9a5640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05e81552a0 con 0x7f05e80cb980 2023-10-19T14:01:26.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.424+0000 7f05e77fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f05e812a4e0 0x7f05e8129df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:26.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.424+0000 7f05e77fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f05e812a4e0 0x7f05e8129df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40884/0 (socket says 172.21.15.57:40884) 2023-10-19T14:01:26.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.424+0000 7f05e77fe640 1 -- 172.21.15.57:0/596410734 learned_addr learned my addr 172.21.15.57:0/596410734 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:01:26.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.425+0000 7f05e67fc640 1 --2- 172.21.15.57:0/596410734 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f05e80cb980 0x7f05e81298b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:26.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.425+0000 7f05e6ffd640 1 --2- 172.21.15.57:0/596410734 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f05e8070350 0x7f05e8151c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:26.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.425+0000 7f05e77fe640 1 -- 172.21.15.57:0/596410734 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f05e80cb980 msgr2=0x7f05e81298b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:26.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.425+0000 7f05e77fe640 1 --2- 172.21.15.57:0/596410734 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f05e80cb980 0x7f05e81298b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:26.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.425+0000 7f05e77fe640 1 -- 172.21.15.57:0/596410734 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f05e8070350 msgr2=0x7f05e8151c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:26.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.425+0000 7f05e77fe640 1 --2- 172.21.15.57:0/596410734 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f05e8070350 0x7f05e8151c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:26.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.425+0000 7f05e77fe640 1 -- 172.21.15.57:0/596410734 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05d0009560 con 0x7f05e812a4e0 2023-10-19T14:01:26.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.425+0000 7f05e6ffd640 1 --2- 172.21.15.57:0/596410734 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f05e8070350 0x7f05e8151c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:01:26.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.425+0000 7f05e77fe640 1 --2- 172.21.15.57:0/596410734 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f05e812a4e0 0x7f05e8129df0 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7f05d800eb40 tx=0x7f05d800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:26.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.425+0000 7f05dffff640 1 -- 172.21.15.57:0/596410734 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05d8004b20 con 0x7f05e812a4e0 2023-10-19T14:01:26.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.425+0000 7f05dffff640 1 -- 172.21.15.57:0/596410734 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f05d8004ca0 con 0x7f05e812a4e0 2023-10-19T14:01:26.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.425+0000 7f05ed9a5640 1 -- 172.21.15.57:0/596410734 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05e81544e0 con 0x7f05e812a4e0 2023-10-19T14:01:26.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.425+0000 7f05dffff640 1 -- 172.21.15.57:0/596410734 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05d8005280 con 0x7f05e812a4e0 2023-10-19T14:01:26.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.425+0000 7f05ed9a5640 1 -- 172.21.15.57:0/596410734 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05e8154aa0 con 0x7f05e812a4e0 2023-10-19T14:01:26.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.426+0000 7f05ed9a5640 1 -- 172.21.15.57:0/596410734 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05e80690e0 con 0x7f05e812a4e0 2023-10-19T14:01:26.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.427+0000 7f05dffff640 1 -- 172.21.15.57:0/596410734 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 89) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f05d80053e0 con 0x7f05e812a4e0 2023-10-19T14:01:26.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.427+0000 7f05dffff640 1 --2- 172.21.15.57:0/596410734 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f05c806acc0 0x7f05c806d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:26.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.427+0000 7f05dffff640 1 -- 172.21.15.57:0/596410734 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 6833+0+0 (secure 0 0 0) 0x7f05d8014070 con 0x7f05e812a4e0 2023-10-19T14:01:26.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.427+0000 7f05e6ffd640 1 --2- 172.21.15.57:0/596410734 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f05c806acc0 0x7f05c806d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:26.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.428+0000 7f05e6ffd640 1 --2- 172.21.15.57:0/596410734 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f05c806acc0 0x7f05c806d180 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f05e0007fd0 tx=0x7f05e0009040 comp rx=0 tx=0).ready entity=mgr.8923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:26.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.431+0000 7f05dffff640 1 -- 172.21.15.57:0/596410734 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f05d8055370 con 0x7f05e812a4e0 2023-10-19T14:01:26.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.613+0000 7f05ed9a5640 1 -- 172.21.15.57:0/596410734 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f05e8154f50 con 0x7f05e812a4e0 2023-10-19T14:01:26.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.614+0000 7f05dffff640 1 -- 172.21.15.57:0/596410734 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v164) v1 ==== 74+0+17966 (secure 0 0 0) 0x7f05d8058990 con 0x7f05e812a4e0 2023-10-19T14:01:26.617 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:01:26.617 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":164,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T14:01:24.521727+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2023-10-19T13:04:56.433332+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"156","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2023-10-19T13:04:57.112749+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"164","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":113,"snap_epoch":164,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.179:0/3898405031":"2023-10-20T14:01:24.521670+0000","172.21.15.179:6833/4251935070":"2023-10-20T14:01:24.521670+0000","172.21.15.179:6832/4251935070":"2023-10-20T14:01:24.521670+0000","172.21.15.179:0/4276889975":"2023-10-20T14:01:24.521670+0000","172.21.15.179:0/1193476158":"2023-10-20T14:01:24.521670+0000","172.21.15.179:0/273107227":"2023-10-20T14:01:24.521670+0000","172.21.15.179:0/3350224887":"2023-10-20T14:01:24.521670+0000","172.21.15.179:0/744328422":"2023-10-20T13:05:09.216573+0000","172.21.15.179:0/1909872102":"2023-10-20T13:05:09.216573+0000","172.21.15.179:0/1800709778":"2023-10-20T14:01:24.521670+0000","172.21.15.179:0/1880188133":"2023-10-20T13:05:09.216573+0000","172.21.15.179:0/40128169":"2023-10-20T13:05:09.216573+0000","172.21.15.179:6833/2581690007":"2023-10-20T13:05:09.216573+0000","172.21.15.179:0/3857987280":"2023-10-20T13:05:09.216573+0000","172.21.15.179:6832/2581690007":"2023-10-20T13:05:09.216573+0000","172.21.15.179:0/57720407":"2023-10-20T13:05:09.216573+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[{"pool":20,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T14:01:26.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.616+0000 7f05ed9a5640 1 -- 172.21.15.57:0/596410734 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f05c806acc0 msgr2=0x7f05c806d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:26.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.616+0000 7f05ed9a5640 1 --2- 172.21.15.57:0/596410734 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f05c806acc0 0x7f05c806d180 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f05e0007fd0 tx=0x7f05e0009040 comp rx=0 tx=0).stop 2023-10-19T14:01:26.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.616+0000 7f05ed9a5640 1 -- 172.21.15.57:0/596410734 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f05e812a4e0 msgr2=0x7f05e8129df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:26.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.616+0000 7f05ed9a5640 1 --2- 172.21.15.57:0/596410734 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f05e812a4e0 0x7f05e8129df0 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7f05d800eb40 tx=0x7f05d800c2d0 comp rx=0 tx=0).stop 2023-10-19T14:01:26.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.616+0000 7f05ed9a5640 1 -- 172.21.15.57:0/596410734 shutdown_connections 2023-10-19T14:01:26.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.616+0000 7f05ed9a5640 1 --2- 172.21.15.57:0/596410734 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f05c806acc0 0x7f05c806d180 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:26.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.616+0000 7f05ed9a5640 1 --2- 172.21.15.57:0/596410734 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f05e812a4e0 0x7f05e8129df0 unknown :-1 s=CLOSED pgs=1840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:26.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.616+0000 7f05ed9a5640 1 --2- 172.21.15.57:0/596410734 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f05e80cb980 0x7f05e81298b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:26.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.616+0000 7f05ed9a5640 1 --2- 172.21.15.57:0/596410734 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f05e8070350 0x7f05e8151c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:26.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.616+0000 7f05ed9a5640 1 -- 172.21.15.57:0/596410734 >> 172.21.15.57:0/596410734 conn(0x7f05e8069930 msgr2=0x7f05e81369a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:26.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.617+0000 7f05ed9a5640 1 -- 172.21.15.57:0/596410734 shutdown_connections 2023-10-19T14:01:26.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.617+0000 7f05ed9a5640 1 -- 172.21.15.57:0/596410734 wait complete. 2023-10-19T14:01:26.630 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-19T14:01:26.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.737+0000 7fde08c32640 1 -- 172.21.15.57:0/4204235067 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde040b0eb0 msgr2=0x7fde040b1320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:26.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.737+0000 7fde08c32640 1 --2- 172.21.15.57:0/4204235067 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde040b0eb0 0x7fde040b1320 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7fddfc009830 tx=0x7fddfc02bcd0 comp rx=0 tx=0).stop 2023-10-19T14:01:26.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.737+0000 7fde08c32640 1 -- 172.21.15.57:0/4204235067 shutdown_connections 2023-10-19T14:01:26.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.737+0000 7fde08c32640 1 --2- 172.21.15.57:0/4204235067 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fde040cd740 0x7fde040cdba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:26.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.737+0000 7fde08c32640 1 --2- 172.21.15.57:0/4204235067 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde040b0eb0 0x7fde040b1320 unknown :-1 s=CLOSED pgs=1842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:26.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.737+0000 7fde08c32640 1 -- 172.21.15.57:0/4204235067 >> 172.21.15.57:0/4204235067 conn(0x7fde0409a010 msgr2=0x7fde040d0de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:26.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.737+0000 7fde08c32640 1 -- 172.21.15.57:0/4204235067 shutdown_connections 2023-10-19T14:01:26.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.737+0000 7fde08c32640 1 -- 172.21.15.57:0/4204235067 wait complete. 2023-10-19T14:01:26.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.739+0000 7fde08c32640 1 Processor -- start 2023-10-19T14:01:26.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.739+0000 7fde08c32640 1 -- start start 2023-10-19T14:01:26.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.739+0000 7fde08c32640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde040b0eb0 0x7fde040c5950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:26.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.739+0000 7fde08c32640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fde040cd740 0x7fde04154340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:26.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.739+0000 7fde08c32640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fde0415aff0 0x7fde040ada50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:26.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.739+0000 7fde08c32640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde040c61a0 con 0x7fde040b0eb0 2023-10-19T14:01:26.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.739+0000 7fde08c32640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde040c6480 con 0x7fde0415aff0 2023-10-19T14:01:26.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.739+0000 7fde08c32640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde040c6760 con 0x7fde040cd740 2023-10-19T14:01:26.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.740+0000 7fde0259c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde040b0eb0 0x7fde040c5950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:26.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.740+0000 7fde0259c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde040b0eb0 0x7fde040c5950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:53690/0 (socket says 172.21.15.57:53690) 2023-10-19T14:01:26.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.740+0000 7fde0259c640 1 -- 172.21.15.57:0/2876231218 learned_addr learned my addr 172.21.15.57:0/2876231218 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:01:26.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.740+0000 7fde01d9b640 1 --2- 172.21.15.57:0/2876231218 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fde040cd740 0x7fde04154340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:26.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.740+0000 7fde02d9d640 1 --2- 172.21.15.57:0/2876231218 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fde0415aff0 0x7fde040ada50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:26.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.740+0000 7fde0259c640 1 -- 172.21.15.57:0/2876231218 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fde040cd740 msgr2=0x7fde04154340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:26.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.740+0000 7fde0259c640 1 --2- 172.21.15.57:0/2876231218 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fde040cd740 0x7fde04154340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:26.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.740+0000 7fde0259c640 1 -- 172.21.15.57:0/2876231218 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fde0415aff0 msgr2=0x7fde040ada50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:26.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.740+0000 7fde0259c640 1 --2- 172.21.15.57:0/2876231218 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fde0415aff0 0x7fde040ada50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:26.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.740+0000 7fde0259c640 1 -- 172.21.15.57:0/2876231218 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddfc009490 con 0x7fde040b0eb0 2023-10-19T14:01:26.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.740+0000 7fde02d9d640 1 --2- 172.21.15.57:0/2876231218 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fde0415aff0 0x7fde040ada50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:01:26.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.740+0000 7fde0259c640 1 --2- 172.21.15.57:0/2876231218 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde040b0eb0 0x7fde040c5950 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7fddf400c7f0 tx=0x7fddf400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:26.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.740+0000 7fddfaffd640 1 -- 172.21.15.57:0/2876231218 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf4003da0 con 0x7fde040b0eb0 2023-10-19T14:01:26.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.740+0000 7fddfaffd640 1 -- 172.21.15.57:0/2876231218 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fddf4005070 con 0x7fde040b0eb0 2023-10-19T14:01:26.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.740+0000 7fddfaffd640 1 -- 172.21.15.57:0/2876231218 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf4005460 con 0x7fde040b0eb0 2023-10-19T14:01:26.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.740+0000 7fde08c32640 1 -- 172.21.15.57:0/2876231218 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde040c6a40 con 0x7fde040b0eb0 2023-10-19T14:01:26.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.740+0000 7fde08c32640 1 -- 172.21.15.57:0/2876231218 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde040c5e90 con 0x7fde040b0eb0 2023-10-19T14:01:26.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.741+0000 7fde08c32640 1 -- 172.21.15.57:0/2876231218 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde040690e0 con 0x7fde040b0eb0 2023-10-19T14:01:26.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.742+0000 7fddfaffd640 1 -- 172.21.15.57:0/2876231218 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 89) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fddf40055c0 con 0x7fde040b0eb0 2023-10-19T14:01:26.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.742+0000 7fddfaffd640 1 --2- 172.21.15.57:0/2876231218 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7fdde406ac90 0x7fdde406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:26.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.742+0000 7fddfaffd640 1 -- 172.21.15.57:0/2876231218 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 6833+0+0 (secure 0 0 0) 0x7fddf4089190 con 0x7fde040b0eb0 2023-10-19T14:01:26.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.743+0000 7fde01d9b640 1 --2- 172.21.15.57:0/2876231218 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7fdde406ac90 0x7fdde406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:26.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.743+0000 7fde01d9b640 1 --2- 172.21.15.57:0/2876231218 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7fdde406ac90 0x7fdde406d150 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fde040ae620 tx=0x7fddec00b390 comp rx=0 tx=0).ready entity=mgr.8923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:26.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.747+0000 7fddfaffd640 1 -- 172.21.15.57:0/2876231218 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fddf4054140 con 0x7fde040b0eb0 2023-10-19T14:01:26.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.962+0000 7fde08c32640 1 -- 172.21.15.57:0/2876231218 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fde040b0010 con 0x7fde040b0eb0 2023-10-19T14:01:26.965 INFO:tasks.ceph.mon.a.smithi057.stderr:2023-10-19T14:01:26.963+0000 7f1253ff3640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-19T14:01:26.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.964+0000 7fddfaffd640 1 -- 172.21.15.57:0/2876231218 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v64) v1 ==== 161+0+0 (secure 0 0 0) 0x7fddf4057760 con 0x7fde040b0eb0 2023-10-19T14:01:26.967 INFO:teuthology.orchestra.run.smithi057.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-19T14:01:26.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.967+0000 7fde08c32640 1 -- 172.21.15.57:0/2876231218 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7fdde406ac90 msgr2=0x7fdde406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:26.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.967+0000 7fde08c32640 1 --2- 172.21.15.57:0/2876231218 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7fdde406ac90 0x7fdde406d150 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fde040ae620 tx=0x7fddec00b390 comp rx=0 tx=0).stop 2023-10-19T14:01:26.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.967+0000 7fde08c32640 1 -- 172.21.15.57:0/2876231218 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde040b0eb0 msgr2=0x7fde040c5950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:26.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.967+0000 7fde08c32640 1 --2- 172.21.15.57:0/2876231218 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde040b0eb0 0x7fde040c5950 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7fddf400c7f0 tx=0x7fddf400ccb0 comp rx=0 tx=0).stop 2023-10-19T14:01:26.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.968+0000 7fde08c32640 1 -- 172.21.15.57:0/2876231218 shutdown_connections 2023-10-19T14:01:26.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.968+0000 7fde08c32640 1 --2- 172.21.15.57:0/2876231218 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7fdde406ac90 0x7fdde406d150 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:26.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.968+0000 7fde08c32640 1 --2- 172.21.15.57:0/2876231218 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fde0415aff0 0x7fde040ada50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:26.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.968+0000 7fde08c32640 1 --2- 172.21.15.57:0/2876231218 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fde040cd740 0x7fde04154340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:26.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.968+0000 7fde08c32640 1 --2- 172.21.15.57:0/2876231218 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde040b0eb0 0x7fde040c5950 unknown :-1 s=CLOSED pgs=1843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:26.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.968+0000 7fde08c32640 1 -- 172.21.15.57:0/2876231218 >> 172.21.15.57:0/2876231218 conn(0x7fde0409a010 msgr2=0x7fde0415ddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:26.973 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.968+0000 7fde08c32640 1 -- 172.21.15.57:0/2876231218 shutdown_connections 2023-10-19T14:01:26.973 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:26.968+0000 7fde08c32640 1 -- 172.21.15.57:0/2876231218 wait complete. 2023-10-19T14:01:26.981 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-19T14:01:27.005 INFO:tasks.ceph.mds.c.smithi057.stdout:starting mds.c at 2023-10-19T14:01:27.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:27.093+0000 7f6e20768640 1 -- 172.21.15.57:0/382732479 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6e1806ae40 msgr2=0x7f6e180b0e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:27.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:27.093+0000 7f6e20768640 1 --2- 172.21.15.57:0/382732479 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6e1806ae40 0x7f6e180b0e90 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f6e08005fd0 tx=0x7f6e0802be10 comp rx=0 tx=0).stop 2023-10-19T14:01:27.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:27.093+0000 7f6e20768640 1 -- 172.21.15.57:0/382732479 shutdown_connections 2023-10-19T14:01:27.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:27.093+0000 7f6e20768640 1 --2- 172.21.15.57:0/382732479 >> v2:172.21.15.179:3300/0 conn(0x7f6e1815c1f0 0x7f6e18157dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:27.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:27.093+0000 7f6e20768640 1 --2- 172.21.15.57:0/382732479 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6e1806ae40 0x7f6e180b0e90 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:27.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:27.093+0000 7f6e20768640 1 -- 172.21.15.57:0/382732479 >> 172.21.15.57:0/382732479 conn(0x7f6e1806fae0 msgr2=0x7f6e1806fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:27.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:27.093+0000 7f6e20768640 1 -- 172.21.15.57:0/382732479 shutdown_connections 2023-10-19T14:01:27.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:27.093+0000 7f6e20768640 1 -- 172.21.15.57:0/382732479 wait complete. 2023-10-19T14:01:27.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:27.094+0000 7f6e20768640 1 Processor -- start 2023-10-19T14:01:27.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:27.094+0000 7f6e20768640 1 -- start start 2023-10-19T14:01:27.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:27.095+0000 7f6e20768640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6e1806ae40 0x7f6e18149e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:27.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:27.095+0000 7f6e20768640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6e181579f0 0x7f6e180ac420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:27.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:27.095+0000 7f6e20768640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6e1815c1f0 0x7f6e180b0250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:27.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:27.095+0000 7f6e20768640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e180ada30 con 0x7f6e1806ae40 2023-10-19T14:01:27.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:27.095+0000 7f6e20768640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e180ace20 con 0x7f6e1815c1f0 2023-10-19T14:01:27.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:27.095+0000 7f6e20768640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e180ad100 con 0x7f6e181579f0 2023-10-19T14:01:27.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:27.095+0000 7f6e1e504640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6e1806ae40 0x7f6e18149e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:27.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:27.095+0000 7f6e1e504640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6e1806ae40 0x7f6e18149e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:53706/0 (socket says 172.21.15.57:53706) 2023-10-19T14:01:27.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:27.095+0000 7f6e1e504640 1 -- 172.21.15.57:0/2802694179 learned_addr learned my addr 172.21.15.57:0/2802694179 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:01:27.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:27.095+0000 7f6e1ed05640 1 --2- 172.21.15.57:0/2802694179 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6e1815c1f0 0x7f6e180b0250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:27.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:27.095+0000 7f6e1dd03640 1 --2- 172.21.15.57:0/2802694179 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6e181579f0 0x7f6e180ac420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:27.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:27.095+0000 7f6e1e504640 1 -- 172.21.15.57:0/2802694179 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6e181579f0 msgr2=0x7f6e180ac420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:27.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:27.095+0000 7f6e1e504640 1 --2- 172.21.15.57:0/2802694179 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6e181579f0 0x7f6e180ac420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:27.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:27.096+0000 7f6e1e504640 1 -- 172.21.15.57:0/2802694179 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6e1815c1f0 msgr2=0x7f6e180b0250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:27.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:27.096+0000 7f6e1e504640 1 --2- 172.21.15.57:0/2802694179 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6e1815c1f0 0x7f6e180b0250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:27.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:27.096+0000 7f6e1e504640 1 -- 172.21.15.57:0/2802694179 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e10009840 con 0x7f6e1806ae40 2023-10-19T14:01:27.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:27.096+0000 7f6e1e504640 1 --2- 172.21.15.57:0/2802694179 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6e1806ae40 0x7f6e18149e60 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7f6e1400c450 tx=0x7f6e1400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:27.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:27.096+0000 7f6e0effd640 1 -- 172.21.15.57:0/2802694179 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e140052d0 con 0x7f6e1806ae40 2023-10-19T14:01:27.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:27.096+0000 7f6e0effd640 1 -- 172.21.15.57:0/2802694179 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f6e14005430 con 0x7f6e1806ae40 2023-10-19T14:01:27.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:27.096+0000 7f6e20768640 1 -- 172.21.15.57:0/2802694179 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e08009560 con 0x7f6e1806ae40 2023-10-19T14:01:27.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:27.096+0000 7f6e0effd640 1 -- 172.21.15.57:0/2802694179 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e14012890 con 0x7f6e1806ae40 2023-10-19T14:01:27.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:27.096+0000 7f6e20768640 1 -- 172.21.15.57:0/2802694179 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e180b2330 con 0x7f6e1806ae40 2023-10-19T14:01:27.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:27.098+0000 7f6e20768640 1 -- 172.21.15.57:0/2802694179 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e180690e0 con 0x7f6e1806ae40 2023-10-19T14:01:27.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:27.098+0000 7f6e0effd640 1 -- 172.21.15.57:0/2802694179 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 89) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f6e140035f0 con 0x7f6e1806ae40 2023-10-19T14:01:27.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:27.099+0000 7f6e0effd640 1 --2- 172.21.15.57:0/2802694179 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f6dfc06acc0 0x7f6dfc06d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:27.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:27.099+0000 7f6e0effd640 1 -- 172.21.15.57:0/2802694179 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 6875+0+0 (secure 0 0 0) 0x7f6e14089410 con 0x7f6e1806ae40 2023-10-19T14:01:27.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:27.099+0000 7f6e1dd03640 1 --2- 172.21.15.57:0/2802694179 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f6dfc06acc0 0x7f6dfc06d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:27.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:27.099+0000 7f6e1dd03640 1 --2- 172.21.15.57:0/2802694179 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f6dfc06acc0 0x7f6dfc06d180 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6e08035000 tx=0x7f6e08004730 comp rx=0 tx=0).ready entity=mgr.8923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:27.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:27.103+0000 7f6e0effd640 1 -- 172.21.15.57:0/2802694179 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6e14054330 con 0x7f6e1806ae40 2023-10-19T14:01:27.320 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:27.318+0000 7f6e20768640 1 -- 172.21.15.57:0/2802694179 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f6e181597e0 con 0x7f6e1806ae40 2023-10-19T14:01:27.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:27.540+0000 7f6e0effd640 1 -- 172.21.15.57:0/2802694179 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mgrmap(e 90) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f6e14054700 con 0x7f6e1806ae40 2023-10-19T14:01:27.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:27.970+0000 7f6e0effd640 1 -- 172.21.15.57:0/2802694179 <== mon.0 v2:172.21.15.57:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v65) v1 ==== 104+0+0 (secure 0 0 0) 0x7f6e14071e30 con 0x7f6e1806ae40 2023-10-19T14:01:27.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:27.973+0000 7f6e20768640 1 -- 172.21.15.57:0/2802694179 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f6dfc06acc0 msgr2=0x7f6dfc06d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:27.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:27.973+0000 7f6e20768640 1 --2- 172.21.15.57:0/2802694179 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f6dfc06acc0 0x7f6dfc06d180 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6e08035000 tx=0x7f6e08004730 comp rx=0 tx=0).stop 2023-10-19T14:01:27.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:27.973+0000 7f6e20768640 1 -- 172.21.15.57:0/2802694179 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6e1806ae40 msgr2=0x7f6e18149e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:27.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:27.973+0000 7f6e20768640 1 --2- 172.21.15.57:0/2802694179 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6e1806ae40 0x7f6e18149e60 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7f6e1400c450 tx=0x7f6e1400c910 comp rx=0 tx=0).stop 2023-10-19T14:01:27.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:27.973+0000 7f6e20768640 1 -- 172.21.15.57:0/2802694179 shutdown_connections 2023-10-19T14:01:27.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:27.973+0000 7f6e20768640 1 --2- 172.21.15.57:0/2802694179 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f6dfc06acc0 0x7f6dfc06d180 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:27.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:27.973+0000 7f6e20768640 1 --2- 172.21.15.57:0/2802694179 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6e1815c1f0 0x7f6e180b0250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:27.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:27.973+0000 7f6e20768640 1 --2- 172.21.15.57:0/2802694179 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6e181579f0 0x7f6e180ac420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:27.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:27.973+0000 7f6e20768640 1 --2- 172.21.15.57:0/2802694179 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6e1806ae40 0x7f6e18149e60 unknown :-1 s=CLOSED pgs=1847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:27.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:27.973+0000 7f6e20768640 1 -- 172.21.15.57:0/2802694179 >> 172.21.15.57:0/2802694179 conn(0x7f6e1806fae0 msgr2=0x7f6e18159d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:27.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:27.973+0000 7f6e20768640 1 -- 172.21.15.57:0/2802694179 shutdown_connections 2023-10-19T14:01:27.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:27.973+0000 7f6e20768640 1 -- 172.21.15.57:0/2802694179 wait complete. 2023-10-19T14:01:27.989 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-19T14:01:28.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:28.152+0000 7f0cf1644640 1 -- 172.21.15.57:0/942720865 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0cec070920 msgr2=0x7f0cec0affe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:28.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:28.152+0000 7f0cf1644640 1 --2- 172.21.15.57:0/942720865 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0cec070920 0x7f0cec0affe0 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7f0cec0cbdc0 tx=0x7f0ce402c1d0 comp rx=0 tx=0).stop 2023-10-19T14:01:28.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:28.152+0000 7f0cf1644640 1 -- 172.21.15.57:0/942720865 shutdown_connections 2023-10-19T14:01:28.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:28.152+0000 7f0cf1644640 1 --2- 172.21.15.57:0/942720865 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0cec070920 0x7f0cec0affe0 unknown :-1 s=CLOSED pgs=1848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:28.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:28.152+0000 7f0cf1644640 1 --2- 172.21.15.57:0/942720865 >> v2:172.21.15.179:3300/0 conn(0x7f0cec141690 0x7f0cec0703e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:28.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:28.152+0000 7f0cf1644640 1 -- 172.21.15.57:0/942720865 >> 172.21.15.57:0/942720865 conn(0x7f0cec069930 msgr2=0x7f0cec069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:28.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:28.152+0000 7f0cf1644640 1 -- 172.21.15.57:0/942720865 shutdown_connections 2023-10-19T14:01:28.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:28.152+0000 7f0cf1644640 1 -- 172.21.15.57:0/942720865 wait complete. 2023-10-19T14:01:28.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:28.153+0000 7f0cf1644640 1 Processor -- start 2023-10-19T14:01:28.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:28.153+0000 7f0cf1644640 1 -- start start 2023-10-19T14:01:28.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:28.154+0000 7f0cf1644640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0cec070920 0x7f0cec159990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:28.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:28.154+0000 7f0cf1644640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0cec13e520 0x7f0cec1f29b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:28.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:28.154+0000 7f0cf1644640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0cec141690 0x7f0cec1f2ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:28.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:28.154+0000 7f0cf1644640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cec0c1ce0 con 0x7f0cec141690 2023-10-19T14:01:28.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:28.154+0000 7f0cf1644640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cec0c1990 con 0x7f0cec13e520 2023-10-19T14:01:28.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:28.154+0000 7f0cf1644640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cec0ad890 con 0x7f0cec070920 2023-10-19T14:01:28.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:28.154+0000 7f0ceb7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0cec141690 0x7f0cec1f2ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:28.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:28.154+0000 7f0ceb7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0cec141690 0x7f0cec1f2ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:53714/0 (socket says 172.21.15.57:53714) 2023-10-19T14:01:28.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:28.154+0000 7f0ceb7fe640 1 -- 172.21.15.57:0/772731004 learned_addr learned my addr 172.21.15.57:0/772731004 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:01:28.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:28.154+0000 7f0cea7fc640 1 --2- 172.21.15.57:0/772731004 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0cec13e520 0x7f0cec1f29b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:28.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:28.154+0000 7f0ceaffd640 1 --2- 172.21.15.57:0/772731004 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0cec070920 0x7f0cec159990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:28.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:28.154+0000 7f0ceb7fe640 1 -- 172.21.15.57:0/772731004 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0cec070920 msgr2=0x7f0cec159990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:28.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:28.154+0000 7f0ceb7fe640 1 --2- 172.21.15.57:0/772731004 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0cec070920 0x7f0cec159990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:28.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:28.154+0000 7f0ceb7fe640 1 -- 172.21.15.57:0/772731004 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0cec13e520 msgr2=0x7f0cec1f29b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:28.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:28.154+0000 7f0ceb7fe640 1 --2- 172.21.15.57:0/772731004 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0cec13e520 0x7f0cec1f29b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:28.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:28.154+0000 7f0ceb7fe640 1 -- 172.21.15.57:0/772731004 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ce4009900 con 0x7f0cec141690 2023-10-19T14:01:28.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:28.154+0000 7f0ceb7fe640 1 --2- 172.21.15.57:0/772731004 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0cec141690 0x7f0cec1f2ef0 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7f0ce4037040 tx=0x7f0ce4035db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:28.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:28.155+0000 7f0cd37fe640 1 -- 172.21.15.57:0/772731004 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce4034440 con 0x7f0cec141690 2023-10-19T14:01:28.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:28.155+0000 7f0cd37fe640 1 -- 172.21.15.57:0/772731004 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f0ce4004450 con 0x7f0cec141690 2023-10-19T14:01:28.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:28.155+0000 7f0cf1644640 1 -- 172.21.15.57:0/772731004 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cec0adb70 con 0x7f0cec141690 2023-10-19T14:01:28.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:28.155+0000 7f0cd37fe640 1 -- 172.21.15.57:0/772731004 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce4004840 con 0x7f0cec141690 2023-10-19T14:01:28.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:28.155+0000 7f0cf1644640 1 -- 172.21.15.57:0/772731004 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cec1f3430 con 0x7f0cec141690 2023-10-19T14:01:28.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:28.156+0000 7f0cf1644640 1 -- 172.21.15.57:0/772731004 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cec0c1e20 con 0x7f0cec141690 2023-10-19T14:01:28.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:28.156+0000 7f0cd37fe640 1 -- 172.21.15.57:0/772731004 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 90) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f0ce403b040 con 0x7f0cec141690 2023-10-19T14:01:28.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:28.157+0000 7f0cd37fe640 1 --2- 172.21.15.57:0/772731004 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f0cd806aed0 0x7f0cd806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:28.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:28.157+0000 7f0cd37fe640 1 -- 172.21.15.57:0/772731004 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 6875+0+0 (secure 0 0 0) 0x7f0ce40ac9b0 con 0x7f0cec141690 2023-10-19T14:01:28.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:28.157+0000 7f0ceaffd640 1 --2- 172.21.15.57:0/772731004 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f0cd806aed0 0x7f0cd806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:28.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:28.158+0000 7f0ceaffd640 1 --2- 172.21.15.57:0/772731004 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f0cd806aed0 0x7f0cd806d390 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0cdc007fd0 tx=0x7f0cdc009040 comp rx=0 tx=0).ready entity=mgr.8923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:28.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:28.161+0000 7f0cd37fe640 1 -- 172.21.15.57:0/772731004 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0cec0c1e20 con 0x7f0cec141690 2023-10-19T14:01:28.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:28.338+0000 7f0cf1644640 1 -- 172.21.15.57:0/772731004 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f0cec0c1e20 con 0x7f0cec141690 2023-10-19T14:01:29.054 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:29.051+0000 7f0cd37fe640 1 -- 172.21.15.57:0/772731004 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v166) v1 ==== 181+0+0 (secure 0 0 0) 0x7f0cec0c1e20 con 0x7f0cec141690 2023-10-19T14:01:29.054 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'cephfs_metadata' removed 2023-10-19T14:01:29.057 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:29.054+0000 7f0cf1644640 1 -- 172.21.15.57:0/772731004 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f0cd806aed0 msgr2=0x7f0cd806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:29.057 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:29.054+0000 7f0cf1644640 1 --2- 172.21.15.57:0/772731004 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f0cd806aed0 0x7f0cd806d390 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0cdc007fd0 tx=0x7f0cdc009040 comp rx=0 tx=0).stop 2023-10-19T14:01:29.057 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:29.054+0000 7f0cf1644640 1 -- 172.21.15.57:0/772731004 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0cec141690 msgr2=0x7f0cec1f2ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:29.057 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:29.054+0000 7f0cf1644640 1 --2- 172.21.15.57:0/772731004 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0cec141690 0x7f0cec1f2ef0 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7f0ce4037040 tx=0x7f0ce4035db0 comp rx=0 tx=0).stop 2023-10-19T14:01:29.058 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:29.054+0000 7f0cf1644640 1 -- 172.21.15.57:0/772731004 shutdown_connections 2023-10-19T14:01:29.058 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:29.054+0000 7f0cf1644640 1 --2- 172.21.15.57:0/772731004 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f0cd806aed0 0x7f0cd806d390 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:29.058 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:29.054+0000 7f0cf1644640 1 --2- 172.21.15.57:0/772731004 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0cec141690 0x7f0cec1f2ef0 unknown :-1 s=CLOSED pgs=1849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:29.058 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:29.054+0000 7f0cf1644640 1 --2- 172.21.15.57:0/772731004 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0cec13e520 0x7f0cec1f29b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:29.058 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:29.054+0000 7f0cf1644640 1 --2- 172.21.15.57:0/772731004 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0cec070920 0x7f0cec159990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:29.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:29.054+0000 7f0cf1644640 1 -- 172.21.15.57:0/772731004 >> 172.21.15.57:0/772731004 conn(0x7f0cec069930 msgr2=0x7f0cec13ce00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:29.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:29.054+0000 7f0cf1644640 1 -- 172.21.15.57:0/772731004 shutdown_connections 2023-10-19T14:01:29.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:29.054+0000 7f0cf1644640 1 -- 172.21.15.57:0/772731004 wait complete. 2023-10-19T14:01:29.075 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-19T14:01:29.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:29.192+0000 7f0c310a9640 1 -- 172.21.15.57:0/1980480583 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0c140047c0 msgr2=0x7f0c14004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:29.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:29.192+0000 7f0c310a9640 1 --2- 172.21.15.57:0/1980480583 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0c140047c0 0x7f0c14004c40 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7f0c20009b50 tx=0x7f0c140057f0 comp rx=0 tx=0).stop 2023-10-19T14:01:29.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:29.192+0000 7f0c2b59e640 1 -- 172.21.15.57:0/1980480583 reap_dead start 2023-10-19T14:01:29.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:29.192+0000 7f0c310a9640 1 -- 172.21.15.57:0/1980480583 shutdown_connections 2023-10-19T14:01:29.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:29.192+0000 7f0c310a9640 1 -- 172.21.15.57:0/1980480583 >> 172.21.15.57:0/1980480583 conn(0x7f0c2c069930 msgr2=0x7f0c2c069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:29.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:29.192+0000 7f0c310a9640 1 -- 172.21.15.57:0/1980480583 shutdown_connections 2023-10-19T14:01:29.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:29.192+0000 7f0c310a9640 1 -- 172.21.15.57:0/1980480583 wait complete. 2023-10-19T14:01:29.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:29.193+0000 7f0c310a9640 1 Processor -- start 2023-10-19T14:01:29.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:29.193+0000 7f0c310a9640 1 -- start start 2023-10-19T14:01:29.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:29.194+0000 7f0c310a9640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0c140047c0 0x7f0c2c12e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:29.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:29.194+0000 7f0c310a9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0c2c1314d0 0x7f0c2c1522c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:29.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:29.194+0000 7f0c310a9640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0c2c0a9c80 0x7f0c2c0aa100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:29.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:29.194+0000 7f0c310a9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c2c0aa640 con 0x7f0c2c1314d0 2023-10-19T14:01:29.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:29.194+0000 7f0c310a9640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c2c12eed0 con 0x7f0c2c0a9c80 2023-10-19T14:01:29.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:29.194+0000 7f0c310a9640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c2c152800 con 0x7f0c140047c0 2023-10-19T14:01:29.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:29.194+0000 7f0c2a59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0c2c1314d0 0x7f0c2c1522c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:29.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:29.194+0000 7f0c2a59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0c2c1314d0 0x7f0c2c1522c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:53726/0 (socket says 172.21.15.57:53726) 2023-10-19T14:01:29.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:29.194+0000 7f0c2a59c640 1 -- 172.21.15.57:0/4054057940 learned_addr learned my addr 172.21.15.57:0/4054057940 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:01:29.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:29.194+0000 7f0c2b59e640 1 --2- 172.21.15.57:0/4054057940 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0c2c0a9c80 0x7f0c2c0aa100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:29.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:29.194+0000 7f0c2ad9d640 1 --2- 172.21.15.57:0/4054057940 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0c140047c0 0x7f0c2c12e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:29.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:29.195+0000 7f0c2a59c640 1 -- 172.21.15.57:0/4054057940 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0c140047c0 msgr2=0x7f0c2c12e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:29.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:29.195+0000 7f0c2a59c640 1 --2- 172.21.15.57:0/4054057940 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0c140047c0 0x7f0c2c12e990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:29.201 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:29.195+0000 7f0c2a59c640 1 -- 172.21.15.57:0/4054057940 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0c2c0a9c80 msgr2=0x7f0c2c0aa100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:29.201 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:29.195+0000 7f0c2a59c640 1 --2- 172.21.15.57:0/4054057940 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0c2c0a9c80 0x7f0c2c0aa100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:29.201 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:29.195+0000 7f0c2a59c640 1 -- 172.21.15.57:0/4054057940 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c2c0a64b0 con 0x7f0c2c1314d0 2023-10-19T14:01:29.201 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:29.195+0000 7f0c2a59c640 1 --2- 172.21.15.57:0/4054057940 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0c2c1314d0 0x7f0c2c1522c0 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7f0c20009b50 tx=0x7f0c20033d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:29.202 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:29.195+0000 7f0c137fe640 1 -- 172.21.15.57:0/4054057940 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c2003e830 con 0x7f0c2c1314d0 2023-10-19T14:01:29.202 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:29.195+0000 7f0c137fe640 1 -- 172.21.15.57:0/4054057940 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f0c2003e990 con 0x7f0c2c1314d0 2023-10-19T14:01:29.202 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:29.195+0000 7f0c310a9640 1 -- 172.21.15.57:0/4054057940 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c2c12f100 con 0x7f0c2c1314d0 2023-10-19T14:01:29.202 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:29.195+0000 7f0c137fe640 1 -- 172.21.15.57:0/4054057940 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c2003ed40 con 0x7f0c2c1314d0 2023-10-19T14:01:29.202 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:29.195+0000 7f0c310a9640 1 -- 172.21.15.57:0/4054057940 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c2c12f6c0 con 0x7f0c2c1314d0 2023-10-19T14:01:29.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:29.196+0000 7f0c310a9640 1 -- 172.21.15.57:0/4054057940 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c2c0690e0 con 0x7f0c2c1314d0 2023-10-19T14:01:29.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:29.196+0000 7f0c137fe640 1 -- 172.21.15.57:0/4054057940 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 91) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f0c20046400 con 0x7f0c2c1314d0 2023-10-19T14:01:29.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:29.197+0000 7f0c137fe640 1 --2- 172.21.15.57:0/4054057940 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f0c1806af60 0x7f0c1806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:29.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:29.197+0000 7f0c137fe640 1 -- 172.21.15.57:0/4054057940 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(166..166 src has 1..166) v4 ==== 6437+0+0 (secure 0 0 0) 0x7f0c200accd0 con 0x7f0c2c1314d0 2023-10-19T14:01:29.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:29.197+0000 7f0c2ad9d640 1 --2- 172.21.15.57:0/4054057940 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f0c1806af60 0x7f0c1806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:29.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:29.198+0000 7f0c2ad9d640 1 --2- 172.21.15.57:0/4054057940 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f0c1806af60 0x7f0c1806d420 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0c24006c60 tx=0x7f0c24001e70 comp rx=0 tx=0).ready entity=mgr.8923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:29.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:29.201+0000 7f0c137fe640 1 -- 172.21.15.57:0/4054057940 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0c20077ae0 con 0x7f0c2c1314d0 2023-10-19T14:01:29.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:29.381+0000 7f0c310a9640 1 -- 172.21.15.57:0/4054057940 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f0c2c0cc420 con 0x7f0c2c1314d0 2023-10-19T14:01:30.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.087+0000 7f0c137fe640 1 -- 172.21.15.57:0/4054057940 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v167) v1 ==== 169+0+0 (secure 0 0 0) 0x7f0c2004b070 con 0x7f0c2c1314d0 2023-10-19T14:01:30.090 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'cephfs_data' removed 2023-10-19T14:01:30.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.092+0000 7f0c310a9640 1 -- 172.21.15.57:0/4054057940 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f0c1806af60 msgr2=0x7f0c1806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:30.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.092+0000 7f0c310a9640 1 --2- 172.21.15.57:0/4054057940 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f0c1806af60 0x7f0c1806d420 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0c24006c60 tx=0x7f0c24001e70 comp rx=0 tx=0).stop 2023-10-19T14:01:30.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.092+0000 7f0c310a9640 1 -- 172.21.15.57:0/4054057940 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0c2c1314d0 msgr2=0x7f0c2c1522c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:30.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.092+0000 7f0c310a9640 1 --2- 172.21.15.57:0/4054057940 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0c2c1314d0 0x7f0c2c1522c0 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7f0c20009b50 tx=0x7f0c20033d50 comp rx=0 tx=0).stop 2023-10-19T14:01:30.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.095+0000 7f0c310a9640 1 -- 172.21.15.57:0/4054057940 shutdown_connections 2023-10-19T14:01:30.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.095+0000 7f0c310a9640 1 --2- 172.21.15.57:0/4054057940 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f0c1806af60 0x7f0c1806d420 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:30.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.095+0000 7f0c310a9640 1 --2- 172.21.15.57:0/4054057940 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0c2c0a9c80 0x7f0c2c0aa100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:30.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.095+0000 7f0c310a9640 1 --2- 172.21.15.57:0/4054057940 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0c2c1314d0 0x7f0c2c1522c0 unknown :-1 s=CLOSED pgs=1852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:30.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.095+0000 7f0c310a9640 1 --2- 172.21.15.57:0/4054057940 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0c140047c0 0x7f0c2c12e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:30.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.095+0000 7f0c310a9640 1 -- 172.21.15.57:0/4054057940 >> 172.21.15.57:0/4054057940 conn(0x7f0c2c069930 msgr2=0x7f0c2c152ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:30.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.096+0000 7f0c310a9640 1 -- 172.21.15.57:0/4054057940 shutdown_connections 2023-10-19T14:01:30.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.096+0000 7f0c310a9640 1 -- 172.21.15.57:0/4054057940 wait complete. 2023-10-19T14:01:30.115 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 30 2023-10-19T14:01:30.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.228+0000 7f565917d640 1 -- 172.21.15.57:0/2154078591 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5654157c00 msgr2=0x7f56540c98b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:30.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.228+0000 7f565917d640 1 --2- 172.21.15.57:0/2154078591 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5654157c00 0x7f56540c98b0 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7f5644009830 tx=0x7f564402bc30 comp rx=0 tx=0).stop 2023-10-19T14:01:30.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.229+0000 7f565917d640 1 -- 172.21.15.57:0/2154078591 shutdown_connections 2023-10-19T14:01:30.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.229+0000 7f565917d640 1 --2- 172.21.15.57:0/2154078591 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5654070c30 0x7f565406bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:30.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.229+0000 7f565917d640 1 --2- 172.21.15.57:0/2154078591 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5654157c00 0x7f56540c98b0 unknown :-1 s=CLOSED pgs=1854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:30.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.229+0000 7f565917d640 1 -- 172.21.15.57:0/2154078591 >> 172.21.15.57:0/2154078591 conn(0x7f56540aec40 msgr2=0x7f56540af050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:30.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.229+0000 7f565917d640 1 -- 172.21.15.57:0/2154078591 shutdown_connections 2023-10-19T14:01:30.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.229+0000 7f565917d640 1 -- 172.21.15.57:0/2154078591 wait complete. 2023-10-19T14:01:30.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.230+0000 7f565917d640 1 Processor -- start 2023-10-19T14:01:30.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.230+0000 7f565917d640 1 -- start start 2023-10-19T14:01:30.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.231+0000 7f565917d640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5654070c30 0x7f56541569e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:30.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.231+0000 7f565917d640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f56540cb2f0 0x7f565413d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:30.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.231+0000 7f565917d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5654157c00 0x7f5654147a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:30.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.231+0000 7f565917d640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f565413dd30 con 0x7f5654157c00 2023-10-19T14:01:30.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.231+0000 7f565917d640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5654155570 con 0x7f5654070c30 2023-10-19T14:01:30.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.231+0000 7f565917d640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5654147f90 con 0x7f56540cb2f0 2023-10-19T14:01:30.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.231+0000 7f565359e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5654157c00 0x7f5654147a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:30.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.231+0000 7f565359e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5654157c00 0x7f5654147a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:53748/0 (socket says 172.21.15.57:53748) 2023-10-19T14:01:30.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.231+0000 7f565359e640 1 -- 172.21.15.57:0/970444366 learned_addr learned my addr 172.21.15.57:0/970444366 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:01:30.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.231+0000 7f564bfff640 1 --2- 172.21.15.57:0/970444366 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f56540cb2f0 0x7f565413d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:30.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.231+0000 7f565359e640 1 -- 172.21.15.57:0/970444366 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f56540cb2f0 msgr2=0x7f565413d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:30.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.231+0000 7f5652d9d640 1 --2- 172.21.15.57:0/970444366 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5654070c30 0x7f56541569e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:30.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.231+0000 7f565359e640 1 --2- 172.21.15.57:0/970444366 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f56540cb2f0 0x7f565413d7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:30.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.231+0000 7f565359e640 1 -- 172.21.15.57:0/970444366 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5654070c30 msgr2=0x7f56541569e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:30.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.231+0000 7f565359e640 1 --2- 172.21.15.57:0/970444366 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5654070c30 0x7f56541569e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:30.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.231+0000 7f565359e640 1 -- 172.21.15.57:0/970444366 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5644009490 con 0x7f5654157c00 2023-10-19T14:01:30.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.231+0000 7f565359e640 1 --2- 172.21.15.57:0/970444366 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5654157c00 0x7f5654147a50 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7f564c00eaa0 tx=0x7f564c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:30.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.231+0000 7f564b7fe640 1 -- 172.21.15.57:0/970444366 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f564c00cdb0 con 0x7f5654157c00 2023-10-19T14:01:30.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.232+0000 7f564b7fe640 1 -- 172.21.15.57:0/970444366 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f564c004b20 con 0x7f5654157c00 2023-10-19T14:01:30.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.232+0000 7f565917d640 1 -- 172.21.15.57:0/970444366 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5654148230 con 0x7f5654157c00 2023-10-19T14:01:30.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.232+0000 7f564b7fe640 1 -- 172.21.15.57:0/970444366 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f564c005020 con 0x7f5654157c00 2023-10-19T14:01:30.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.232+0000 7f565917d640 1 -- 172.21.15.57:0/970444366 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56541487f0 con 0x7f5654157c00 2023-10-19T14:01:30.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.233+0000 7f565917d640 1 -- 172.21.15.57:0/970444366 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56540690e0 con 0x7f5654157c00 2023-10-19T14:01:30.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.233+0000 7f564b7fe640 1 -- 172.21.15.57:0/970444366 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 91) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f564c005180 con 0x7f5654157c00 2023-10-19T14:01:30.243 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.234+0000 7f564b7fe640 1 --2- 172.21.15.57:0/970444366 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f563406af60 0x7f563406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:30.243 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.234+0000 7f564b7fe640 1 -- 172.21.15.57:0/970444366 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 5979+0+0 (secure 0 0 0) 0x7f564c08a410 con 0x7f5654157c00 2023-10-19T14:01:30.243 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.234+0000 7f5652d9d640 1 --2- 172.21.15.57:0/970444366 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f563406af60 0x7f563406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:30.243 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.235+0000 7f5652d9d640 1 --2- 172.21.15.57:0/970444366 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f563406af60 0x7f563406d420 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f5644036000 tx=0x7f5644004730 comp rx=0 tx=0).ready entity=mgr.8923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:30.243 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.238+0000 7f564b7fe640 1 -- 172.21.15.57:0/970444366 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f564c055110 con 0x7f5654157c00 2023-10-19T14:01:30.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.416+0000 7f565917d640 1 -- 172.21.15.57:0/970444366 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "config reset", "num": 30} v 0) v1 -- 0x7f5654071440 con 0x7f5654157c00 2023-10-19T14:01:30.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.420+0000 7f564b7fe640 1 -- 172.21.15.57:0/970444366 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 30}]=0 v62) v1 ==== 71+0+0 (secure 0 0 0) 0x7f564c058730 con 0x7f5654157c00 2023-10-19T14:01:30.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.422+0000 7f565917d640 1 -- 172.21.15.57:0/970444366 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f563406af60 msgr2=0x7f563406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:30.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.422+0000 7f565917d640 1 --2- 172.21.15.57:0/970444366 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f563406af60 0x7f563406d420 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f5644036000 tx=0x7f5644004730 comp rx=0 tx=0).stop 2023-10-19T14:01:30.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.422+0000 7f565917d640 1 -- 172.21.15.57:0/970444366 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5654157c00 msgr2=0x7f5654147a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:30.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.422+0000 7f565917d640 1 --2- 172.21.15.57:0/970444366 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5654157c00 0x7f5654147a50 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7f564c00eaa0 tx=0x7f564c00ef60 comp rx=0 tx=0).stop 2023-10-19T14:01:30.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.422+0000 7f565917d640 1 -- 172.21.15.57:0/970444366 shutdown_connections 2023-10-19T14:01:30.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.422+0000 7f565917d640 1 --2- 172.21.15.57:0/970444366 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f563406af60 0x7f563406d420 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:30.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.422+0000 7f565917d640 1 --2- 172.21.15.57:0/970444366 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5654157c00 0x7f5654147a50 unknown :-1 s=CLOSED pgs=1855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:30.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.422+0000 7f565917d640 1 --2- 172.21.15.57:0/970444366 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f56540cb2f0 0x7f565413d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:30.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.422+0000 7f565917d640 1 --2- 172.21.15.57:0/970444366 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5654070c30 0x7f56541569e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:30.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.422+0000 7f565917d640 1 -- 172.21.15.57:0/970444366 >> 172.21.15.57:0/970444366 conn(0x7f56540aec40 msgr2=0x7f565406d3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:30.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.422+0000 7f565917d640 1 -- 172.21.15.57:0/970444366 shutdown_connections 2023-10-19T14:01:30.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.422+0000 7f565917d640 1 -- 172.21.15.57:0/970444366 wait complete. 2023-10-19T14:01:30.435 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention' 2023-10-19T14:01:30.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.542+0000 7f71f69a4640 1 -- 172.21.15.57:0/2648156499 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f71f015b2a0 msgr2=0x7f71f00d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:30.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.542+0000 7f71f69a4640 1 --2- 172.21.15.57:0/2648156499 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f71f015b2a0 0x7f71f00d1910 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7f71e0009830 tx=0x7f71e002bd60 comp rx=0 tx=0).stop 2023-10-19T14:01:30.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.542+0000 7f71f69a4640 1 -- 172.21.15.57:0/2648156499 shutdown_connections 2023-10-19T14:01:30.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.542+0000 7f71f69a4640 1 --2- 172.21.15.57:0/2648156499 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f71f015b2a0 0x7f71f00d1910 unknown :-1 s=CLOSED pgs=1857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:30.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.542+0000 7f71f69a4640 1 -- 172.21.15.57:0/2648156499 >> 172.21.15.57:0/2648156499 conn(0x7f71f00d0d60 msgr2=0x7f71f00d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:30.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.542+0000 7f71f69a4640 1 -- 172.21.15.57:0/2648156499 shutdown_connections 2023-10-19T14:01:30.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.542+0000 7f71f69a4640 1 -- 172.21.15.57:0/2648156499 wait complete. 2023-10-19T14:01:30.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.543+0000 7f71f69a4640 1 Processor -- start 2023-10-19T14:01:30.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.544+0000 7f71f69a4640 1 -- start start 2023-10-19T14:01:30.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.544+0000 7f71f69a4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f71f00ceb40 0x7f71f01e5b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:30.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.544+0000 7f71f69a4640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f71f00cf340 0x7f71f01e60d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:30.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.544+0000 7f71f69a4640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f71f015b2a0 0x7f71f01e4c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:30.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.544+0000 7f71f69a4640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71f01e5a00 con 0x7f71f00ceb40 2023-10-19T14:01:30.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.544+0000 7f71f69a4640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71f0150b30 con 0x7f71f015b2a0 2023-10-19T14:01:30.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.544+0000 7f71f69a4640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71f0157960 con 0x7f71f00cf340 2023-10-19T14:01:30.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.545+0000 7f71effff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f71f00ceb40 0x7f71f01e5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:30.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.545+0000 7f71effff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f71f00ceb40 0x7f71f01e5b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:53764/0 (socket says 172.21.15.57:53764) 2023-10-19T14:01:30.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.545+0000 7f71effff640 1 -- 172.21.15.57:0/2969862996 learned_addr learned my addr 172.21.15.57:0/2969862996 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:01:30.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.545+0000 7f71f4f41640 1 --2- 172.21.15.57:0/2969862996 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f71f015b2a0 0x7f71f01e4c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:30.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.545+0000 7f71ef7fe640 1 --2- 172.21.15.57:0/2969862996 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f71f00cf340 0x7f71f01e60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:30.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.545+0000 7f71effff640 1 -- 172.21.15.57:0/2969862996 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f71f00cf340 msgr2=0x7f71f01e60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:30.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.545+0000 7f71effff640 1 --2- 172.21.15.57:0/2969862996 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f71f00cf340 0x7f71f01e60d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:30.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.545+0000 7f71effff640 1 -- 172.21.15.57:0/2969862996 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f71f015b2a0 msgr2=0x7f71f01e4c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:30.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.545+0000 7f71effff640 1 --2- 172.21.15.57:0/2969862996 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f71f015b2a0 0x7f71f01e4c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:30.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.545+0000 7f71effff640 1 -- 172.21.15.57:0/2969862996 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71e0009490 con 0x7f71f00ceb40 2023-10-19T14:01:30.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.545+0000 7f71ef7fe640 1 --2- 172.21.15.57:0/2969862996 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f71f00cf340 0x7f71f01e60d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:01:30.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.545+0000 7f71f4f41640 1 --2- 172.21.15.57:0/2969862996 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f71f015b2a0 0x7f71f01e4c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:01:30.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.545+0000 7f71effff640 1 --2- 172.21.15.57:0/2969862996 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f71f00ceb40 0x7f71f01e5b90 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7f71d800c400 tx=0x7f71d800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:30.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.545+0000 7f71ecff9640 1 -- 172.21.15.57:0/2969862996 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71d80052d0 con 0x7f71f00ceb40 2023-10-19T14:01:30.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.545+0000 7f71ecff9640 1 -- 172.21.15.57:0/2969862996 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f71d8007020 con 0x7f71f00ceb40 2023-10-19T14:01:30.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.545+0000 7f71f69a4640 1 -- 172.21.15.57:0/2969862996 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71f0157c40 con 0x7f71f00ceb40 2023-10-19T14:01:30.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.545+0000 7f71ecff9640 1 -- 172.21.15.57:0/2969862996 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71d8007470 con 0x7f71f00ceb40 2023-10-19T14:01:30.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.545+0000 7f71f69a4640 1 -- 172.21.15.57:0/2969862996 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71f0158200 con 0x7f71f00ceb40 2023-10-19T14:01:30.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.546+0000 7f71f69a4640 1 -- 172.21.15.57:0/2969862996 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71f0157aa0 con 0x7f71f00ceb40 2023-10-19T14:01:30.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.547+0000 7f71ecff9640 1 -- 172.21.15.57:0/2969862996 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 91) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f71d80067d0 con 0x7f71f00ceb40 2023-10-19T14:01:30.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.547+0000 7f71ecff9640 1 --2- 172.21.15.57:0/2969862996 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f71d006af60 0x7f71d006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:30.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.547+0000 7f71ecff9640 1 -- 172.21.15.57:0/2969862996 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 5979+0+0 (secure 0 0 0) 0x7f71d8088f50 con 0x7f71f00ceb40 2023-10-19T14:01:30.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.547+0000 7f71ef7fe640 1 --2- 172.21.15.57:0/2969862996 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f71d006af60 0x7f71d006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:30.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.548+0000 7f71ef7fe640 1 --2- 172.21.15.57:0/2969862996 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f71d006af60 0x7f71d006d420 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f71e0036000 tx=0x7f71e0004730 comp rx=0 tx=0).ready entity=mgr.8923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:30.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.551+0000 7f71ecff9640 1 -- 172.21.15.57:0/2969862996 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f71f0157aa0 con 0x7f71f00ceb40 2023-10-19T14:01:30.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:30.794+0000 7f71f69a4640 1 -- 172.21.15.57:0/2969862996 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention"]} v 0) v1 -- 0x7f71f0157aa0 con 0x7f71f00ceb40 2023-10-19T14:01:31.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.096+0000 7f71ecff9640 1 -- 172.21.15.57:0/2969862996 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention"]}]=0 v2508) v1 ==== 178+0+0 (secure 0 0 0) 0x7f71f0157aa0 con 0x7f71f00ceb40 2023-10-19T14:01:31.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.098+0000 7f71f69a4640 1 -- 172.21.15.57:0/2969862996 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f71d006af60 msgr2=0x7f71d006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:31.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.098+0000 7f71f69a4640 1 --2- 172.21.15.57:0/2969862996 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f71d006af60 0x7f71d006d420 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f71e0036000 tx=0x7f71e0004730 comp rx=0 tx=0).stop 2023-10-19T14:01:31.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.098+0000 7f71f69a4640 1 -- 172.21.15.57:0/2969862996 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f71f00ceb40 msgr2=0x7f71f01e5b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:31.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.098+0000 7f71f69a4640 1 --2- 172.21.15.57:0/2969862996 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f71f00ceb40 0x7f71f01e5b90 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7f71d800c400 tx=0x7f71d800c8c0 comp rx=0 tx=0).stop 2023-10-19T14:01:31.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.098+0000 7f71f69a4640 1 -- 172.21.15.57:0/2969862996 shutdown_connections 2023-10-19T14:01:31.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.098+0000 7f71f69a4640 1 --2- 172.21.15.57:0/2969862996 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f71d006af60 0x7f71d006d420 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:31.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.098+0000 7f71f69a4640 1 --2- 172.21.15.57:0/2969862996 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f71f015b2a0 0x7f71f01e4c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:31.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.098+0000 7f71f69a4640 1 --2- 172.21.15.57:0/2969862996 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f71f00cf340 0x7f71f01e60d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:31.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.098+0000 7f71f69a4640 1 --2- 172.21.15.57:0/2969862996 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f71f00ceb40 0x7f71f01e5b90 unknown :-1 s=CLOSED pgs=1858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:31.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.098+0000 7f71f69a4640 1 -- 172.21.15.57:0/2969862996 >> 172.21.15.57:0/2969862996 conn(0x7f71f00d0d60 msgr2=0x7f71f015a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:31.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.098+0000 7f71f69a4640 1 -- 172.21.15.57:0/2969862996 shutdown_connections 2023-10-19T14:01:31.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.098+0000 7f71f69a4640 1 -- 172.21.15.57:0/2969862996 wait complete. 2023-10-19T14:01:31.112 INFO:tasks.cephfs_test_runner:Test that number of snaps retained are as per user spec. ... ok 2023-10-19T14:01:31.112 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_list_post_schedule_remove (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test listing snap schedules post removal of a schedule 2023-10-19T14:01:31.113 INFO:tasks.cephfs_test_runner:test_snap_schedule_list_post_schedule_remove (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-10-19T14:01:31.113 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T14:01:31.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.223+0000 7fb019e7c640 1 -- 172.21.15.57:0/979566919 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faffc005140 msgr2=0x7faffc00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:31.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.223+0000 7fb019e7c640 1 --2- 172.21.15.57:0/979566919 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faffc005140 0x7faffc00b980 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7fb00c00c610 tx=0x7faffc0057f0 comp rx=0 tx=0).stop 2023-10-19T14:01:31.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.223+0000 7fb013fff640 1 -- 172.21.15.57:0/979566919 reap_dead start 2023-10-19T14:01:31.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.223+0000 7fb019e7c640 1 -- 172.21.15.57:0/979566919 shutdown_connections 2023-10-19T14:01:31.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.223+0000 7fb019e7c640 1 -- 172.21.15.57:0/979566919 >> 172.21.15.57:0/979566919 conn(0x7fb014069930 msgr2=0x7fb014069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:31.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.223+0000 7fb019e7c640 1 -- 172.21.15.57:0/979566919 shutdown_connections 2023-10-19T14:01:31.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.223+0000 7fb019e7c640 1 -- 172.21.15.57:0/979566919 wait complete. 2023-10-19T14:01:31.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.224+0000 7fb019e7c640 1 Processor -- start 2023-10-19T14:01:31.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.225+0000 7fb019e7c640 1 -- start start 2023-10-19T14:01:31.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.225+0000 7fb019e7c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faffc005140 0x7fb01414de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:31.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.225+0000 7fb019e7c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb0140cad50 0x7fb0140afc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:31.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.225+0000 7fb019e7c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb0140cb1d0 0x7fb0140b0180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:31.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.225+0000 7fb019e7c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb014158f30 con 0x7faffc005140 2023-10-19T14:01:31.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.225+0000 7fb019e7c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb014159210 con 0x7fb0140cad50 2023-10-19T14:01:31.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.225+0000 7fb019e7c640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0141594f0 con 0x7fb0140cb1d0 2023-10-19T14:01:31.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.225+0000 7fb0137fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faffc005140 0x7fb01414de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:31.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.225+0000 7fb0137fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faffc005140 0x7fb01414de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:53772/0 (socket says 172.21.15.57:53772) 2023-10-19T14:01:31.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.225+0000 7fb0137fe640 1 -- 172.21.15.57:0/3574624841 learned_addr learned my addr 172.21.15.57:0/3574624841 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:01:31.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.226+0000 7fb012ffd640 1 --2- 172.21.15.57:0/3574624841 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb0140cad50 0x7fb0140afc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:31.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.226+0000 7fb013fff640 1 --2- 172.21.15.57:0/3574624841 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb0140cb1d0 0x7fb0140b0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:31.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.226+0000 7fb0137fe640 1 -- 172.21.15.57:0/3574624841 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb0140cb1d0 msgr2=0x7fb0140b0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:31.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.226+0000 7fb0137fe640 1 --2- 172.21.15.57:0/3574624841 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb0140cb1d0 0x7fb0140b0180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:31.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.226+0000 7fb0137fe640 1 -- 172.21.15.57:0/3574624841 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb0140cad50 msgr2=0x7fb0140afc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:31.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.226+0000 7fb0137fe640 1 --2- 172.21.15.57:0/3574624841 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb0140cad50 0x7fb0140afc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:31.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.226+0000 7fb0137fe640 1 -- 172.21.15.57:0/3574624841 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0141597d0 con 0x7faffc005140 2023-10-19T14:01:31.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.226+0000 7fb012ffd640 1 --2- 172.21.15.57:0/3574624841 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb0140cad50 0x7fb0140afc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:01:31.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.226+0000 7fb0137fe640 1 --2- 172.21.15.57:0/3574624841 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faffc005140 0x7fb01414de20 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7fb004004ed0 tx=0x7fb004001630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:31.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.226+0000 7faffbfff640 1 -- 172.21.15.57:0/3574624841 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb004011870 con 0x7faffc005140 2023-10-19T14:01:31.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.226+0000 7faffbfff640 1 -- 172.21.15.57:0/3574624841 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fb004011ac0 con 0x7faffc005140 2023-10-19T14:01:31.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.226+0000 7fb019e7c640 1 -- 172.21.15.57:0/3574624841 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb014159ab0 con 0x7faffc005140 2023-10-19T14:01:31.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.226+0000 7faffbfff640 1 -- 172.21.15.57:0/3574624841 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb004011e70 con 0x7faffc005140 2023-10-19T14:01:31.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.226+0000 7fb019e7c640 1 -- 172.21.15.57:0/3574624841 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb01414e360 con 0x7faffc005140 2023-10-19T14:01:31.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.227+0000 7fb019e7c640 1 -- 172.21.15.57:0/3574624841 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0140690e0 con 0x7faffc005140 2023-10-19T14:01:31.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.227+0000 7faffbfff640 1 -- 172.21.15.57:0/3574624841 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 91) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb0040103c0 con 0x7faffc005140 2023-10-19T14:01:31.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.228+0000 7faffbfff640 1 --2- 172.21.15.57:0/3574624841 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7fb00006af60 0x7fb00006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:31.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.228+0000 7faffbfff640 1 -- 172.21.15.57:0/3574624841 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 5979+0+0 (secure 0 0 0) 0x7fb00408a4a0 con 0x7faffc005140 2023-10-19T14:01:31.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.228+0000 7fb012ffd640 1 --2- 172.21.15.57:0/3574624841 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7fb00006af60 0x7fb00006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:31.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.229+0000 7fb012ffd640 1 --2- 172.21.15.57:0/3574624841 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7fb00006af60 0x7fb00006d420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb0141572a0 tx=0x7fb008005f90 comp rx=0 tx=0).ready entity=mgr.8923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:31.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.232+0000 7faffbfff640 1 -- 172.21.15.57:0/3574624841 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb0040552c0 con 0x7faffc005140 2023-10-19T14:01:31.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.408+0000 7fb019e7c640 1 -- 172.21.15.57:0/3574624841 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb01406ff80 con 0x7faffc005140 2023-10-19T14:01:31.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.409+0000 7faffbfff640 1 -- 172.21.15.57:0/3574624841 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v167) v1 ==== 74+0+14270 (secure 0 0 0) 0x7fb0040588e0 con 0x7faffc005140 2023-10-19T14:01:31.412 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:01:31.412 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":167,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T14:01:30.086810+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.57:6833/1460919655":"2023-10-20T14:01:26.963927+0000","172.21.15.179:0/3898405031":"2023-10-20T14:01:24.521670+0000","172.21.15.57:6832/1460919655":"2023-10-20T14:01:26.963927+0000","172.21.15.179:6833/4251935070":"2023-10-20T14:01:24.521670+0000","172.21.15.179:6832/4251935070":"2023-10-20T14:01:24.521670+0000","172.21.15.179:0/4276889975":"2023-10-20T14:01:24.521670+0000","172.21.15.179:0/1193476158":"2023-10-20T14:01:24.521670+0000","172.21.15.179:0/273107227":"2023-10-20T14:01:24.521670+0000","172.21.15.179:0/3350224887":"2023-10-20T14:01:24.521670+0000","172.21.15.179:0/744328422":"2023-10-20T13:05:09.216573+0000","172.21.15.179:0/1909872102":"2023-10-20T13:05:09.216573+0000","172.21.15.179:0/1800709778":"2023-10-20T14:01:24.521670+0000","172.21.15.179:0/1880188133":"2023-10-20T13:05:09.216573+0000","172.21.15.179:0/40128169":"2023-10-20T13:05:09.216573+0000","172.21.15.179:6833/2581690007":"2023-10-20T13:05:09.216573+0000","172.21.15.179:0/3857987280":"2023-10-20T13:05:09.216573+0000","172.21.15.179:6832/2581690007":"2023-10-20T13:05:09.216573+0000","172.21.15.179:0/57720407":"2023-10-20T13:05:09.216573+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T14:01:31.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.411+0000 7fb019e7c640 1 -- 172.21.15.57:0/3574624841 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7fb00006af60 msgr2=0x7fb00006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:31.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.411+0000 7fb019e7c640 1 --2- 172.21.15.57:0/3574624841 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7fb00006af60 0x7fb00006d420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb0141572a0 tx=0x7fb008005f90 comp rx=0 tx=0).stop 2023-10-19T14:01:31.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.411+0000 7fb019e7c640 1 -- 172.21.15.57:0/3574624841 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faffc005140 msgr2=0x7fb01414de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:31.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.411+0000 7fb019e7c640 1 --2- 172.21.15.57:0/3574624841 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faffc005140 0x7fb01414de20 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7fb004004ed0 tx=0x7fb004001630 comp rx=0 tx=0).stop 2023-10-19T14:01:31.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.411+0000 7fb019e7c640 1 -- 172.21.15.57:0/3574624841 shutdown_connections 2023-10-19T14:01:31.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.411+0000 7fb019e7c640 1 --2- 172.21.15.57:0/3574624841 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7fb00006af60 0x7fb00006d420 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:31.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.411+0000 7fb019e7c640 1 --2- 172.21.15.57:0/3574624841 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb0140cb1d0 0x7fb0140b0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:31.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.411+0000 7fb019e7c640 1 --2- 172.21.15.57:0/3574624841 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb0140cad50 0x7fb0140afc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:31.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.411+0000 7fb019e7c640 1 --2- 172.21.15.57:0/3574624841 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faffc005140 0x7fb01414de20 unknown :-1 s=CLOSED pgs=1861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:31.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.411+0000 7fb019e7c640 1 -- 172.21.15.57:0/3574624841 >> 172.21.15.57:0/3574624841 conn(0x7fb014069930 msgr2=0x7fb0141354b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:31.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.411+0000 7fb019e7c640 1 -- 172.21.15.57:0/3574624841 shutdown_connections 2023-10-19T14:01:31.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.411+0000 7fb019e7c640 1 -- 172.21.15.57:0/3574624841 wait complete. 2023-10-19T14:01:31.425 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-19T12:44:07.371186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-19T14:01:31.425 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-19T14:01:31.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.535+0000 7fcb58339640 1 -- 172.21.15.57:0/2317872378 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcb500ceb40 msgr2=0x7fcb500b0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:31.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.535+0000 7fcb58339640 1 --2- 172.21.15.57:0/2317872378 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcb500ceb40 0x7fcb500b0650 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7fcb5008a810 tx=0x7fcb4802c1d0 comp rx=0 tx=0).stop 2023-10-19T14:01:31.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.535+0000 7fcb58339640 1 -- 172.21.15.57:0/2317872378 shutdown_connections 2023-10-19T14:01:31.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.535+0000 7fcb58339640 1 --2- 172.21.15.57:0/2317872378 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcb500ceb40 0x7fcb500b0650 unknown :-1 s=CLOSED pgs=1862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:31.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.535+0000 7fcb58339640 1 --2- 172.21.15.57:0/2317872378 >> v2:172.21.15.179:3300/0 conn(0x7fcb500cf340 0x7fcb500d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:31.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.535+0000 7fcb58339640 1 -- 172.21.15.57:0/2317872378 >> 172.21.15.57:0/2317872378 conn(0x7fcb500d0d60 msgr2=0x7fcb500d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:31.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.535+0000 7fcb58339640 1 -- 172.21.15.57:0/2317872378 shutdown_connections 2023-10-19T14:01:31.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.535+0000 7fcb58339640 1 -- 172.21.15.57:0/2317872378 wait complete. 2023-10-19T14:01:31.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.537+0000 7fcb58339640 1 Processor -- start 2023-10-19T14:01:31.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.537+0000 7fcb58339640 1 -- start start 2023-10-19T14:01:31.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.537+0000 7fcb58339640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcb500ceb40 0x7fcb500c80a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:31.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.537+0000 7fcb58339640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcb500cf340 0x7fcb501e5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:31.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.537+0000 7fcb58339640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcb5015b2a0 0x7fcb50155eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:31.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.537+0000 7fcb58339640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb5015b680 con 0x7fcb5015b2a0 2023-10-19T14:01:31.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.537+0000 7fcb58339640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb501e5ce0 con 0x7fcb500cf340 2023-10-19T14:01:31.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.537+0000 7fcb58339640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb501e5fc0 con 0x7fcb500ceb40 2023-10-19T14:01:31.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.537+0000 7fcb568d6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcb5015b2a0 0x7fcb50155eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:31.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.537+0000 7fcb568d6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcb5015b2a0 0x7fcb50155eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:53778/0 (socket says 172.21.15.57:53778) 2023-10-19T14:01:31.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.538+0000 7fcb568d6640 1 -- 172.21.15.57:0/3546736789 learned_addr learned my addr 172.21.15.57:0/3546736789 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:01:31.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.538+0000 7fcb558d4640 1 --2- 172.21.15.57:0/3546736789 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcb500cf340 0x7fcb501e5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:31.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.538+0000 7fcb560d5640 1 --2- 172.21.15.57:0/3546736789 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcb500ceb40 0x7fcb500c80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:31.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.538+0000 7fcb568d6640 1 -- 172.21.15.57:0/3546736789 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcb500ceb40 msgr2=0x7fcb500c80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:31.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.538+0000 7fcb568d6640 1 --2- 172.21.15.57:0/3546736789 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcb500ceb40 0x7fcb500c80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:31.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.538+0000 7fcb568d6640 1 -- 172.21.15.57:0/3546736789 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcb500cf340 msgr2=0x7fcb501e5300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:31.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.538+0000 7fcb568d6640 1 --2- 172.21.15.57:0/3546736789 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcb500cf340 0x7fcb501e5300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:31.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.538+0000 7fcb568d6640 1 -- 172.21.15.57:0/3546736789 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb48009900 con 0x7fcb5015b2a0 2023-10-19T14:01:31.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.538+0000 7fcb560d5640 1 --2- 172.21.15.57:0/3546736789 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcb500ceb40 0x7fcb500c80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:01:31.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.538+0000 7fcb568d6640 1 --2- 172.21.15.57:0/3546736789 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcb5015b2a0 0x7fcb50155eb0 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7fcb48037040 tx=0x7fcb48035db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:31.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.538+0000 7fcb46ffd640 1 -- 172.21.15.57:0/3546736789 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb48034440 con 0x7fcb5015b2a0 2023-10-19T14:01:31.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.538+0000 7fcb46ffd640 1 -- 172.21.15.57:0/3546736789 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fcb48004450 con 0x7fcb5015b2a0 2023-10-19T14:01:31.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.538+0000 7fcb46ffd640 1 -- 172.21.15.57:0/3546736789 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb48004840 con 0x7fcb5015b2a0 2023-10-19T14:01:31.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.538+0000 7fcb58339640 1 -- 172.21.15.57:0/3546736789 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb5015b8d0 con 0x7fcb5015b2a0 2023-10-19T14:01:31.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.538+0000 7fcb58339640 1 -- 172.21.15.57:0/3546736789 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb500c8b20 con 0x7fcb5015b2a0 2023-10-19T14:01:31.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.539+0000 7fcb58339640 1 -- 172.21.15.57:0/3546736789 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb5015bdf0 con 0x7fcb5015b2a0 2023-10-19T14:01:31.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.540+0000 7fcb46ffd640 1 -- 172.21.15.57:0/3546736789 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 91) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fcb4803b040 con 0x7fcb5015b2a0 2023-10-19T14:01:31.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.540+0000 7fcb46ffd640 1 --2- 172.21.15.57:0/3546736789 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7fcb3406af60 0x7fcb3406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:31.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.540+0000 7fcb46ffd640 1 -- 172.21.15.57:0/3546736789 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 5979+0+0 (secure 0 0 0) 0x7fcb480ac650 con 0x7fcb5015b2a0 2023-10-19T14:01:31.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.540+0000 7fcb560d5640 1 --2- 172.21.15.57:0/3546736789 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7fcb3406af60 0x7fcb3406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:31.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.541+0000 7fcb560d5640 1 --2- 172.21.15.57:0/3546736789 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7fcb3406af60 0x7fcb3406d420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fcb4c005fd0 tx=0x7fcb4c005ea0 comp rx=0 tx=0).ready entity=mgr.8923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:31.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.544+0000 7fcb46ffd640 1 -- 172.21.15.57:0/3546736789 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fcb5015bdf0 con 0x7fcb5015b2a0 2023-10-19T14:01:31.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.719+0000 7fcb58339640 1 -- 172.21.15.57:0/3546736789 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fcb5015bdf0 con 0x7fcb5015b2a0 2023-10-19T14:01:31.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.719+0000 7fcb46ffd640 1 -- 172.21.15.57:0/3546736789 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v167) v1 ==== 93+0+10 (secure 0 0 0) 0x7fcb5015bdf0 con 0x7fcb5015b2a0 2023-10-19T14:01:31.722 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-19T14:01:31.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.721+0000 7fcb58339640 1 -- 172.21.15.57:0/3546736789 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7fcb3406af60 msgr2=0x7fcb3406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:31.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.721+0000 7fcb58339640 1 --2- 172.21.15.57:0/3546736789 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7fcb3406af60 0x7fcb3406d420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fcb4c005fd0 tx=0x7fcb4c005ea0 comp rx=0 tx=0).stop 2023-10-19T14:01:31.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.721+0000 7fcb58339640 1 -- 172.21.15.57:0/3546736789 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcb5015b2a0 msgr2=0x7fcb50155eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:31.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.721+0000 7fcb58339640 1 --2- 172.21.15.57:0/3546736789 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcb5015b2a0 0x7fcb50155eb0 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7fcb48037040 tx=0x7fcb48035db0 comp rx=0 tx=0).stop 2023-10-19T14:01:31.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.721+0000 7fcb58339640 1 -- 172.21.15.57:0/3546736789 shutdown_connections 2023-10-19T14:01:31.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.721+0000 7fcb58339640 1 --2- 172.21.15.57:0/3546736789 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7fcb3406af60 0x7fcb3406d420 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:31.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.721+0000 7fcb58339640 1 --2- 172.21.15.57:0/3546736789 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcb5015b2a0 0x7fcb50155eb0 unknown :-1 s=CLOSED pgs=1863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:31.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.721+0000 7fcb58339640 1 --2- 172.21.15.57:0/3546736789 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcb500cf340 0x7fcb501e5300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:31.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.721+0000 7fcb58339640 1 --2- 172.21.15.57:0/3546736789 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcb500ceb40 0x7fcb500c80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:31.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.721+0000 7fcb58339640 1 -- 172.21.15.57:0/3546736789 >> 172.21.15.57:0/3546736789 conn(0x7fcb500d0d60 msgr2=0x7fcb5015a4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:31.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.721+0000 7fcb58339640 1 -- 172.21.15.57:0/3546736789 shutdown_connections 2023-10-19T14:01:31.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.721+0000 7fcb58339640 1 -- 172.21.15.57:0/3546736789 wait complete. 2023-10-19T14:01:31.735 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove' 2023-10-19T14:01:31.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.842+0000 7f08e8858640 1 -- 172.21.15.57:0/2157607126 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f08d0004ed0 msgr2=0x7f08d000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:31.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.842+0000 7f08e8858640 1 --2- 172.21.15.57:0/2157607126 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f08d0004ed0 0x7f08d000b980 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7f08d400c640 tx=0x7f08d00057f0 comp rx=0 tx=0).stop 2023-10-19T14:01:31.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.842+0000 7f08e2d9d640 1 -- 172.21.15.57:0/2157607126 reap_dead start 2023-10-19T14:01:31.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.842+0000 7f08e8858640 1 -- 172.21.15.57:0/2157607126 shutdown_connections 2023-10-19T14:01:31.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.842+0000 7f08e8858640 1 -- 172.21.15.57:0/2157607126 >> 172.21.15.57:0/2157607126 conn(0x7f08e4069930 msgr2=0x7f08e4069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:31.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.842+0000 7f08e8858640 1 -- 172.21.15.57:0/2157607126 shutdown_connections 2023-10-19T14:01:31.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.842+0000 7f08e8858640 1 -- 172.21.15.57:0/2157607126 wait complete. 2023-10-19T14:01:31.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.843+0000 7f08e8858640 1 Processor -- start 2023-10-19T14:01:31.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.843+0000 7f08e8858640 1 -- start start 2023-10-19T14:01:31.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.844+0000 7f08e8858640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f08d0004ed0 0x7f08e40aadd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:31.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.844+0000 7f08e8858640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f08e40aa070 0x7f08e415af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:31.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.844+0000 7f08e8858640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f08e415b470 0x7f08e413e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:31.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.844+0000 7f08e8858640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08e41f2f80 con 0x7f08e415b470 2023-10-19T14:01:31.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.844+0000 7f08e8858640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08e41f23d0 con 0x7f08e40aa070 2023-10-19T14:01:31.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.844+0000 7f08e8858640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08e415b8f0 con 0x7f08d0004ed0 2023-10-19T14:01:31.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.844+0000 7f08e2d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f08e415b470 0x7f08e413e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:31.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.844+0000 7f08e2d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f08e415b470 0x7f08e413e310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:53796/0 (socket says 172.21.15.57:53796) 2023-10-19T14:01:31.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.844+0000 7f08e2d9d640 1 -- 172.21.15.57:0/740267014 learned_addr learned my addr 172.21.15.57:0/740267014 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:01:31.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.844+0000 7f08e259c640 1 --2- 172.21.15.57:0/740267014 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f08d0004ed0 0x7f08e40aadd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:31.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.844+0000 7f08e1d9b640 1 --2- 172.21.15.57:0/740267014 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f08e40aa070 0x7f08e415af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:31.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.845+0000 7f08e2d9d640 1 -- 172.21.15.57:0/740267014 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f08d0004ed0 msgr2=0x7f08e40aadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:31.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.845+0000 7f08e2d9d640 1 --2- 172.21.15.57:0/740267014 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f08d0004ed0 0x7f08e40aadd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:31.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.845+0000 7f08e2d9d640 1 -- 172.21.15.57:0/740267014 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f08e40aa070 msgr2=0x7f08e415af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:31.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.845+0000 7f08e2d9d640 1 --2- 172.21.15.57:0/740267014 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f08e40aa070 0x7f08e415af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:31.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.845+0000 7f08e2d9d640 1 -- 172.21.15.57:0/740267014 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08e40ab310 con 0x7f08e415b470 2023-10-19T14:01:31.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.845+0000 7f08e1d9b640 1 --2- 172.21.15.57:0/740267014 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f08e40aa070 0x7f08e415af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:01:31.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.845+0000 7f08e2d9d640 1 --2- 172.21.15.57:0/740267014 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f08e415b470 0x7f08e413e310 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7f08d402cd20 tx=0x7f08d4001300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:31.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.845+0000 7f08daffd640 1 -- 172.21.15.57:0/740267014 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08d40018f0 con 0x7f08e415b470 2023-10-19T14:01:31.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.845+0000 7f08daffd640 1 -- 172.21.15.57:0/740267014 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f08d4001a50 con 0x7f08e415b470 2023-10-19T14:01:31.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.845+0000 7f08e8858640 1 -- 172.21.15.57:0/740267014 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08e41f25e0 con 0x7f08e415b470 2023-10-19T14:01:31.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.845+0000 7f08daffd640 1 -- 172.21.15.57:0/740267014 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08d4003bd0 con 0x7f08e415b470 2023-10-19T14:01:31.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.845+0000 7f08e8858640 1 -- 172.21.15.57:0/740267014 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08e41f2ba0 con 0x7f08e415b470 2023-10-19T14:01:31.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.846+0000 7f08e8858640 1 -- 172.21.15.57:0/740267014 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08e41f30c0 con 0x7f08e415b470 2023-10-19T14:01:31.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.846+0000 7f08daffd640 1 -- 172.21.15.57:0/740267014 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 91) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f08d4031030 con 0x7f08e415b470 2023-10-19T14:01:31.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.847+0000 7f08daffd640 1 --2- 172.21.15.57:0/740267014 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f08c406af90 0x7f08c406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:31.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.847+0000 7f08daffd640 1 -- 172.21.15.57:0/740267014 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 5979+0+0 (secure 0 0 0) 0x7f08d40aaaf0 con 0x7f08e415b470 2023-10-19T14:01:31.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.847+0000 7f08e259c640 1 --2- 172.21.15.57:0/740267014 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f08c406af90 0x7f08c406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:31.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.848+0000 7f08e259c640 1 --2- 172.21.15.57:0/740267014 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f08c406af90 0x7f08c406d450 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f08dc009cf0 tx=0x7f08dc005740 comp rx=0 tx=0).ready entity=mgr.8923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:31.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:31.851+0000 7f08daffd640 1 -- 172.21.15.57:0/740267014 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f08e41f30c0 con 0x7f08e415b470 2023-10-19T14:01:32.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:32.098+0000 7f08e8858640 1 -- 172.21.15.57:0/740267014 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove"]} v 0) v1 -- 0x7f08e41f30c0 con 0x7f08e415b470 2023-10-19T14:01:32.101 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:01:32.103 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:01:33.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.105+0000 7f08daffd640 1 -- 172.21.15.57:0/740267014 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove"]}]=0 v2510) v1 ==== 177+0+0 (secure 0 0 0) 0x7f08e41f30c0 con 0x7f08e415b470 2023-10-19T14:01:33.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.108+0000 7f08e8858640 1 -- 172.21.15.57:0/740267014 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f08c406af90 msgr2=0x7f08c406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:33.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.108+0000 7f08e8858640 1 --2- 172.21.15.57:0/740267014 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f08c406af90 0x7f08c406d450 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f08dc009cf0 tx=0x7f08dc005740 comp rx=0 tx=0).stop 2023-10-19T14:01:33.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.108+0000 7f08e8858640 1 -- 172.21.15.57:0/740267014 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f08e415b470 msgr2=0x7f08e413e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:33.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.108+0000 7f08e8858640 1 --2- 172.21.15.57:0/740267014 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f08e415b470 0x7f08e413e310 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7f08d402cd20 tx=0x7f08d4001300 comp rx=0 tx=0).stop 2023-10-19T14:01:33.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.108+0000 7f08e8858640 1 -- 172.21.15.57:0/740267014 shutdown_connections 2023-10-19T14:01:33.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.108+0000 7f08e8858640 1 --2- 172.21.15.57:0/740267014 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f08c406af90 0x7f08c406d450 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:33.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.108+0000 7f08e8858640 1 --2- 172.21.15.57:0/740267014 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f08e415b470 0x7f08e413e310 unknown :-1 s=CLOSED pgs=1866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:33.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.108+0000 7f08e8858640 1 --2- 172.21.15.57:0/740267014 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f08e40aa070 0x7f08e415af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:33.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.108+0000 7f08e8858640 1 --2- 172.21.15.57:0/740267014 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f08d0004ed0 0x7f08e40aadd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:33.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.108+0000 7f08e8858640 1 -- 172.21.15.57:0/740267014 >> 172.21.15.57:0/740267014 conn(0x7f08e4069930 msgr2=0x7f08e415a3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:33.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.108+0000 7f08e8858640 1 -- 172.21.15.57:0/740267014 shutdown_connections 2023-10-19T14:01:33.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.108+0000 7f08e8858640 1 -- 172.21.15.57:0/740267014 wait complete. 2023-10-19T14:01:33.124 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2023-10-19T14:01:33.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.237+0000 7fb70239e640 1 -- 172.21.15.57:0/2539265480 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb6fc0afd20 msgr2=0x7fb6fc0b0120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:33.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.237+0000 7fb70239e640 1 --2- 172.21.15.57:0/2539265480 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb6fc0afd20 0x7fb6fc0b0120 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fb6ec009830 tx=0x7fb6ec02bd60 comp rx=0 tx=0).stop 2023-10-19T14:01:33.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.237+0000 7fb70239e640 1 -- 172.21.15.57:0/2539265480 shutdown_connections 2023-10-19T14:01:33.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.237+0000 7fb70239e640 1 --2- 172.21.15.57:0/2539265480 >> v2:172.21.15.179:3300/0 conn(0x7fb6fc15b250 0x7fb6fc0cb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:33.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.237+0000 7fb70239e640 1 --2- 172.21.15.57:0/2539265480 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb6fc0afd20 0x7fb6fc0b0120 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:33.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.237+0000 7fb70239e640 1 -- 172.21.15.57:0/2539265480 >> 172.21.15.57:0/2539265480 conn(0x7fb6fc0d0cc0 msgr2=0x7fb6fc0d10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:33.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.238+0000 7fb70239e640 1 -- 172.21.15.57:0/2539265480 shutdown_connections 2023-10-19T14:01:33.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.238+0000 7fb70239e640 1 -- 172.21.15.57:0/2539265480 wait complete. 2023-10-19T14:01:33.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.239+0000 7fb70239e640 1 Processor -- start 2023-10-19T14:01:33.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.239+0000 7fb70239e640 1 -- start start 2023-10-19T14:01:33.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.240+0000 7fb70239e640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb6fc0afd20 0x7fb6fc1585b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:33.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.240+0000 7fb70239e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb6fc0cf1c0 0x7fb6fc157b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:33.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.240+0000 7fb70239e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb6fc15b250 0x7fb6fc0c7d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:33.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.240+0000 7fb70239e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6fc0b3b20 con 0x7fb6fc0cf1c0 2023-10-19T14:01:33.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.240+0000 7fb70239e640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6fc0c8e80 con 0x7fb6fc15b250 2023-10-19T14:01:33.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.240+0000 7fb70239e640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6fc0c85e0 con 0x7fb6fc0afd20 2023-10-19T14:01:33.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.240+0000 7fb700b9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb6fc0cf1c0 0x7fb6fc157b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:33.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.240+0000 7fb700b9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb6fc0cf1c0 0x7fb6fc157b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:53804/0 (socket says 172.21.15.57:53804) 2023-10-19T14:01:33.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.240+0000 7fb700b9b640 1 -- 172.21.15.57:0/1857333808 learned_addr learned my addr 172.21.15.57:0/1857333808 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:01:33.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.240+0000 7fb70139c640 1 --2- 172.21.15.57:0/1857333808 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb6fc0afd20 0x7fb6fc1585b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:33.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.240+0000 7fb701b9d640 1 --2- 172.21.15.57:0/1857333808 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb6fc15b250 0x7fb6fc0c7d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:33.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.240+0000 7fb700b9b640 1 -- 172.21.15.57:0/1857333808 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb6fc0afd20 msgr2=0x7fb6fc1585b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:33.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.240+0000 7fb700b9b640 1 --2- 172.21.15.57:0/1857333808 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb6fc0afd20 0x7fb6fc1585b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:33.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.240+0000 7fb700b9b640 1 -- 172.21.15.57:0/1857333808 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb6fc15b250 msgr2=0x7fb6fc0c7d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:33.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.240+0000 7fb700b9b640 1 --2- 172.21.15.57:0/1857333808 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb6fc15b250 0x7fb6fc0c7d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:33.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.240+0000 7fb700b9b640 1 -- 172.21.15.57:0/1857333808 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6ec009490 con 0x7fb6fc0cf1c0 2023-10-19T14:01:33.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.240+0000 7fb700b9b640 1 --2- 172.21.15.57:0/1857333808 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb6fc0cf1c0 0x7fb6fc157b50 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7fb6ec000c00 tx=0x7fb6ec003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:33.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.240+0000 7fb6f1ffb640 1 -- 172.21.15.57:0/1857333808 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6ec003e20 con 0x7fb6fc0cf1c0 2023-10-19T14:01:33.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.241+0000 7fb6f1ffb640 1 -- 172.21.15.57:0/1857333808 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fb6ec034440 con 0x7fb6fc0cf1c0 2023-10-19T14:01:33.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.241+0000 7fb70239e640 1 -- 172.21.15.57:0/1857333808 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6fc0c88c0 con 0x7fb6fc0cf1c0 2023-10-19T14:01:33.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.241+0000 7fb6f1ffb640 1 -- 172.21.15.57:0/1857333808 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6ec034850 con 0x7fb6fc0cf1c0 2023-10-19T14:01:33.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.241+0000 7fb70239e640 1 -- 172.21.15.57:0/1857333808 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6fc0c82d0 con 0x7fb6fc0cf1c0 2023-10-19T14:01:33.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.242+0000 7fb70239e640 1 -- 172.21.15.57:0/1857333808 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6fc066960 con 0x7fb6fc0cf1c0 2023-10-19T14:01:33.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.242+0000 7fb6f1ffb640 1 -- 172.21.15.57:0/1857333808 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 91) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb6ec0349b0 con 0x7fb6fc0cf1c0 2023-10-19T14:01:33.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.243+0000 7fb6f1ffb640 1 --2- 172.21.15.57:0/1857333808 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7fb6e006af90 0x7fb6e006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:33.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.243+0000 7fb6f1ffb640 1 -- 172.21.15.57:0/1857333808 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 5979+0+0 (secure 0 0 0) 0x7fb6ec0abed0 con 0x7fb6fc0cf1c0 2023-10-19T14:01:33.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.243+0000 7fb70139c640 1 --2- 172.21.15.57:0/1857333808 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7fb6e006af90 0x7fb6e006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:33.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.243+0000 7fb70139c640 1 --2- 172.21.15.57:0/1857333808 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7fb6e006af90 0x7fb6e006d450 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb6f8007fd0 tx=0x7fb6f8007cc0 comp rx=0 tx=0).ready entity=mgr.8923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:33.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.247+0000 7fb6f1ffb640 1 -- 172.21.15.57:0/1857333808 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb6ec076e00 con 0x7fb6fc0cf1c0 2023-10-19T14:01:33.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.423+0000 7fb70239e640 1 -- 172.21.15.57:0/1857333808 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "config dump"} v 0) v1 -- 0x7fb6fc0b3880 con 0x7fb6fc0cf1c0 2023-10-19T14:01:33.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.423+0000 7fb6f1ffb640 1 -- 172.21.15.57:0/1857333808 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v62) v1 ==== 59+0+2220 (secure 0 0 0) 0x7fb6ec0428c0 con 0x7fb6fc0cf1c0 2023-10-19T14:01:33.426 INFO:teuthology.orchestra.run.smithi057.stdout:WHO MASK LEVEL OPTION VALUE RO 2023-10-19T14:01:33.426 INFO:teuthology.orchestra.run.smithi057.stdout:mon advanced mon_op_complaint_time 120 2023-10-19T14:01:33.427 INFO:teuthology.orchestra.run.smithi057.stdout:osd advanced osd_op_complaint_time 180.000000 2023-10-19T14:01:33.427 INFO:teuthology.orchestra.run.smithi057.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 4775.995373 2023-10-19T14:01:33.427 INFO:teuthology.orchestra.run.smithi057.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 4208.118410 2023-10-19T14:01:33.427 INFO:teuthology.orchestra.run.smithi057.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 4541.123030 2023-10-19T14:01:33.427 INFO:teuthology.orchestra.run.smithi057.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 4877.592206 2023-10-19T14:01:33.427 INFO:teuthology.orchestra.run.smithi057.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 5019.748032 2023-10-19T14:01:33.428 INFO:teuthology.orchestra.run.smithi057.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 5000.074376 2023-10-19T14:01:33.428 INFO:teuthology.orchestra.run.smithi057.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 5153.479439 2023-10-19T14:01:33.428 INFO:teuthology.orchestra.run.smithi057.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 4977.363845 2023-10-19T14:01:33.428 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced debug_mds 20/20 2023-10-19T14:01:33.428 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced debug_mds_balancer 20/20 2023-10-19T14:01:33.429 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced debug_ms 1/1 2023-10-19T14:01:33.429 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced mds_bal_fragment_size_max 10000 2023-10-19T14:01:33.429 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced mds_bal_merge_size 5 2023-10-19T14:01:33.429 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced mds_bal_split_bits 3 2023-10-19T14:01:33.429 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced mds_bal_split_size 100 2023-10-19T14:01:33.429 INFO:teuthology.orchestra.run.smithi057.stdout:mds dev mds_debug_frag true 2023-10-19T14:01:33.430 INFO:teuthology.orchestra.run.smithi057.stdout:mds dev mds_debug_scatterstat true 2023-10-19T14:01:33.430 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced mds_op_complaint_time 180.000000 2023-10-19T14:01:33.430 INFO:teuthology.orchestra.run.smithi057.stdout:mds dev mds_verify_scatter true 2023-10-19T14:01:33.430 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced osd_op_complaint_time 180.000000 2023-10-19T14:01:33.430 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced rados_mon_op_timeout 900 2023-10-19T14:01:33.431 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced rados_osd_op_timeout 900 2023-10-19T14:01:33.431 INFO:teuthology.orchestra.run.smithi057.stdout:client advanced client_mount_timeout 600 2023-10-19T14:01:33.431 INFO:teuthology.orchestra.run.smithi057.stdout:client advanced debug_client 20/20 2023-10-19T14:01:33.431 INFO:teuthology.orchestra.run.smithi057.stdout:client advanced debug_ms 1/1 2023-10-19T14:01:33.431 INFO:teuthology.orchestra.run.smithi057.stdout:client advanced rados_mon_op_timeout 900 2023-10-19T14:01:33.432 INFO:teuthology.orchestra.run.smithi057.stdout:client advanced rados_osd_op_timeout 900 2023-10-19T14:01:33.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.425+0000 7fb70239e640 1 -- 172.21.15.57:0/1857333808 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7fb6e006af90 msgr2=0x7fb6e006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:33.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.425+0000 7fb70239e640 1 --2- 172.21.15.57:0/1857333808 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7fb6e006af90 0x7fb6e006d450 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb6f8007fd0 tx=0x7fb6f8007cc0 comp rx=0 tx=0).stop 2023-10-19T14:01:33.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.425+0000 7fb70239e640 1 -- 172.21.15.57:0/1857333808 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb6fc0cf1c0 msgr2=0x7fb6fc157b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:33.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.425+0000 7fb70239e640 1 --2- 172.21.15.57:0/1857333808 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb6fc0cf1c0 0x7fb6fc157b50 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7fb6ec000c00 tx=0x7fb6ec003820 comp rx=0 tx=0).stop 2023-10-19T14:01:33.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.425+0000 7fb70239e640 1 -- 172.21.15.57:0/1857333808 shutdown_connections 2023-10-19T14:01:33.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.425+0000 7fb70239e640 1 --2- 172.21.15.57:0/1857333808 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7fb6e006af90 0x7fb6e006d450 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:33.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.425+0000 7fb70239e640 1 --2- 172.21.15.57:0/1857333808 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb6fc15b250 0x7fb6fc0c7d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:33.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.425+0000 7fb70239e640 1 --2- 172.21.15.57:0/1857333808 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb6fc0cf1c0 0x7fb6fc157b50 unknown :-1 s=CLOSED pgs=1867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:33.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.425+0000 7fb70239e640 1 --2- 172.21.15.57:0/1857333808 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb6fc0afd20 0x7fb6fc1585b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:33.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.425+0000 7fb70239e640 1 -- 172.21.15.57:0/1857333808 >> 172.21.15.57:0/1857333808 conn(0x7fb6fc0d0cc0 msgr2=0x7fb6fc159160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:33.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.425+0000 7fb70239e640 1 -- 172.21.15.57:0/1857333808 shutdown_connections 2023-10-19T14:01:33.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.425+0000 7fb70239e640 1 -- 172.21.15.57:0/1857333808 wait complete. 2023-10-19T14:01:33.438 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-19T14:01:33.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.547+0000 7f689ad8e640 1 -- 172.21.15.57:0/2957411512 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f68940cf340 msgr2=0x7f68940d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:33.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.547+0000 7f689ad8e640 1 --2- 172.21.15.57:0/2957411512 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f68940cf340 0x7f68940d1910 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7f688c005fd0 tx=0x7f688c02c050 comp rx=0 tx=0).stop 2023-10-19T14:01:33.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.547+0000 7f689ad8e640 1 -- 172.21.15.57:0/2957411512 shutdown_connections 2023-10-19T14:01:33.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.547+0000 7f689ad8e640 1 --2- 172.21.15.57:0/2957411512 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f689415b2a0 0x7f68940cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:33.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.547+0000 7f689ad8e640 1 --2- 172.21.15.57:0/2957411512 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f68940cf340 0x7f68940d1910 unknown :-1 s=CLOSED pgs=1869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:33.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.547+0000 7f689ad8e640 1 -- 172.21.15.57:0/2957411512 >> 172.21.15.57:0/2957411512 conn(0x7f68940d0d60 msgr2=0x7f68940d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:33.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.547+0000 7f689ad8e640 1 -- 172.21.15.57:0/2957411512 shutdown_connections 2023-10-19T14:01:33.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.547+0000 7f689ad8e640 1 -- 172.21.15.57:0/2957411512 wait complete. 2023-10-19T14:01:33.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.548+0000 7f689ad8e640 1 Processor -- start 2023-10-19T14:01:33.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.548+0000 7f689ad8e640 1 -- start start 2023-10-19T14:01:33.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.549+0000 7f689ad8e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f68940ceb40 0x7f68941580d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:33.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.549+0000 7f689ad8e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f68940cf340 0x7f68940c9880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:33.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.549+0000 7f689ad8e640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f689415b2a0 0x7f689406b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:33.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.549+0000 7f689ad8e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f689409aad0 con 0x7f68940ceb40 2023-10-19T14:01:33.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.549+0000 7f689ad8e640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68940b5670 con 0x7f68940cf340 2023-10-19T14:01:33.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.549+0000 7f689ad8e640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68940b42f0 con 0x7f689415b2a0 2023-10-19T14:01:33.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.549+0000 7f6898b2a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f68940ceb40 0x7f68941580d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:33.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.549+0000 7f6898b2a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f68940ceb40 0x7f68941580d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:53822/0 (socket says 172.21.15.57:53822) 2023-10-19T14:01:33.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.549+0000 7f6898b2a640 1 -- 172.21.15.57:0/1738572419 learned_addr learned my addr 172.21.15.57:0/1738572419 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:01:33.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.549+0000 7f6898b2a640 1 -- 172.21.15.57:0/1738572419 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f689415b2a0 msgr2=0x7f689406b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:33.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.549+0000 7f689932b640 1 --2- 172.21.15.57:0/1738572419 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f689415b2a0 0x7f689406b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:33.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.550+0000 7f6898b2a640 1 --2- 172.21.15.57:0/1738572419 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f689415b2a0 0x7f689406b000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:33.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.550+0000 7f6898b2a640 1 -- 172.21.15.57:0/1738572419 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f68940cf340 msgr2=0x7f68940c9880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:33.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.550+0000 7f6898b2a640 1 --2- 172.21.15.57:0/1738572419 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f68940cf340 0x7f68940c9880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:33.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.550+0000 7f6898b2a640 1 -- 172.21.15.57:0/1738572419 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f688c009840 con 0x7f68940ceb40 2023-10-19T14:01:33.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.550+0000 7f689932b640 1 --2- 172.21.15.57:0/1738572419 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f689415b2a0 0x7f689406b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:01:33.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.550+0000 7f6898b2a640 1 --2- 172.21.15.57:0/1738572419 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f68940ceb40 0x7f68941580d0 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7f688c036040 tx=0x7f688c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:33.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.550+0000 7f68917fa640 1 -- 172.21.15.57:0/1738572419 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f688c03e460 con 0x7f68940ceb40 2023-10-19T14:01:33.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.550+0000 7f68917fa640 1 -- 172.21.15.57:0/1738572419 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f688c03e6d0 con 0x7f68940ceb40 2023-10-19T14:01:33.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.550+0000 7f689ad8e640 1 -- 172.21.15.57:0/1738572419 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68940b45d0 con 0x7f68940ceb40 2023-10-19T14:01:33.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.550+0000 7f68917fa640 1 -- 172.21.15.57:0/1738572419 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f688c03eae0 con 0x7f68940ceb40 2023-10-19T14:01:33.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.550+0000 7f689ad8e640 1 -- 172.21.15.57:0/1738572419 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68940b4b90 con 0x7f68940ceb40 2023-10-19T14:01:33.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.551+0000 7f689ad8e640 1 -- 172.21.15.57:0/1738572419 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68940690e0 con 0x7f68940ceb40 2023-10-19T14:01:33.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.551+0000 7f68917fa640 1 -- 172.21.15.57:0/1738572419 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 91) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f688c03ec40 con 0x7f68940ceb40 2023-10-19T14:01:33.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.552+0000 7f68917fa640 1 --2- 172.21.15.57:0/1738572419 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f688006af60 0x7f688006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:33.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.552+0000 7f68917fa640 1 -- 172.21.15.57:0/1738572419 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 5979+0+0 (secure 0 0 0) 0x7f688c0ac9e0 con 0x7f68940ceb40 2023-10-19T14:01:33.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.552+0000 7f6893fff640 1 --2- 172.21.15.57:0/1738572419 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f688006af60 0x7f688006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:33.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.553+0000 7f6893fff640 1 --2- 172.21.15.57:0/1738572419 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f688006af60 0x7f688006d420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f68880058c0 tx=0x7f6888005850 comp rx=0 tx=0).ready entity=mgr.8923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:33.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.557+0000 7f68917fa640 1 -- 172.21.15.57:0/1738572419 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f688c077870 con 0x7f68940ceb40 2023-10-19T14:01:33.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.735+0000 7f689ad8e640 1 -- 172.21.15.57:0/1738572419 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f68940d0b40 con 0x7f68940ceb40 2023-10-19T14:01:33.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.738+0000 7f68917fa640 1 -- 172.21.15.57:0/1738572419 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v63)=0 v63) v1 ==== 122+0+0 (secure 0 0 0) 0x7f688c042020 con 0x7f68940ceb40 2023-10-19T14:01:33.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.740+0000 7f689ad8e640 1 -- 172.21.15.57:0/1738572419 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f688006af60 msgr2=0x7f688006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:33.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.740+0000 7f689ad8e640 1 --2- 172.21.15.57:0/1738572419 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f688006af60 0x7f688006d420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f68880058c0 tx=0x7f6888005850 comp rx=0 tx=0).stop 2023-10-19T14:01:33.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.740+0000 7f689ad8e640 1 -- 172.21.15.57:0/1738572419 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f68940ceb40 msgr2=0x7f68941580d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:33.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.740+0000 7f689ad8e640 1 --2- 172.21.15.57:0/1738572419 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f68940ceb40 0x7f68941580d0 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7f688c036040 tx=0x7f688c0042f0 comp rx=0 tx=0).stop 2023-10-19T14:01:33.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.740+0000 7f689ad8e640 1 -- 172.21.15.57:0/1738572419 shutdown_connections 2023-10-19T14:01:33.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.740+0000 7f689ad8e640 1 --2- 172.21.15.57:0/1738572419 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f688006af60 0x7f688006d420 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:33.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.740+0000 7f689ad8e640 1 --2- 172.21.15.57:0/1738572419 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f689415b2a0 0x7f689406b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:33.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.740+0000 7f689ad8e640 1 --2- 172.21.15.57:0/1738572419 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f68940cf340 0x7f68940c9880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:33.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.740+0000 7f689ad8e640 1 --2- 172.21.15.57:0/1738572419 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f68940ceb40 0x7f68941580d0 unknown :-1 s=CLOSED pgs=1870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:33.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.740+0000 7f689ad8e640 1 -- 172.21.15.57:0/1738572419 >> 172.21.15.57:0/1738572419 conn(0x7f68940d0d60 msgr2=0x7f68940ae280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:33.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.740+0000 7f689ad8e640 1 -- 172.21.15.57:0/1738572419 shutdown_connections 2023-10-19T14:01:33.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:33.740+0000 7f689ad8e640 1 -- 172.21.15.57:0/1738572419 wait complete. 2023-10-19T14:01:33.752 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-19T14:01:33.753 DEBUG:teuthology.orchestra.run.smithi057:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-19T14:01:33.802 DEBUG:teuthology.orchestra.run.smithi179:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-19T14:01:33.844 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-19T14:01:33.845 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T14:01:33.845 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T14:01:33.861 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055614,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=358 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248260k,nr_inodes=812065,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-19T14:01:33.861 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-19T14:01:33.862 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T14:01:33.902 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054833,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=284 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3247632k,nr_inodes=811908,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-19T14:01:33.903 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T14:01:33.903 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T14:01:33.919 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055614,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=358 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248260k,nr_inodes=812065,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-19T14:01:33.920 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-19T14:01:33.920 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T14:01:33.960 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054833,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=284 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3247632k,nr_inodes=811908,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-19T14:01:33.961 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-19T14:01:33.961 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T14:01:34.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.073+0000 7f5272dd7640 1 -- 172.21.15.57:0/3813220406 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f526c0cdb30 msgr2=0x7f526c0cdfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:34.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.073+0000 7f5272dd7640 1 --2- 172.21.15.57:0/3813220406 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f526c0cdb30 0x7f526c0cdfe0 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7f5258005fd0 tx=0x7f525802bd90 comp rx=0 tx=0).stop 2023-10-19T14:01:34.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.073+0000 7f5272dd7640 1 -- 172.21.15.57:0/3813220406 shutdown_connections 2023-10-19T14:01:34.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.073+0000 7f5272dd7640 1 --2- 172.21.15.57:0/3813220406 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f526c0cdb30 0x7f526c0cdfe0 unknown :-1 s=CLOSED pgs=1871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:34.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.073+0000 7f5272dd7640 1 --2- 172.21.15.57:0/3813220406 >> v2:172.21.15.179:3300/0 conn(0x7f526c159060 0x7f526c06d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:34.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.073+0000 7f5272dd7640 1 -- 172.21.15.57:0/3813220406 >> 172.21.15.57:0/3813220406 conn(0x7f526c0b0670 msgr2=0x7f526c0b0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:34.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.074+0000 7f5272dd7640 1 -- 172.21.15.57:0/3813220406 shutdown_connections 2023-10-19T14:01:34.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.074+0000 7f5272dd7640 1 -- 172.21.15.57:0/3813220406 wait complete. 2023-10-19T14:01:34.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.075+0000 7f5272dd7640 1 Processor -- start 2023-10-19T14:01:34.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.075+0000 7f5272dd7640 1 -- start start 2023-10-19T14:01:34.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.076+0000 7f5272dd7640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f526c0cdb30 0x7f526c06afa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:34.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.076+0000 7f5272dd7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f526c159060 0x7f526c154390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:34.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.076+0000 7f5272dd7640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f526c15ccf0 0x7f526c1f4c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:34.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.076+0000 7f5272dd7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f526c1536a0 con 0x7f526c159060 2023-10-19T14:01:34.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.076+0000 7f5272dd7640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f526c1f5a90 con 0x7f526c15ccf0 2023-10-19T14:01:34.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.076+0000 7f5272dd7640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f526c153f00 con 0x7f526c0cdb30 2023-10-19T14:01:34.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.076+0000 7f526bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f526c159060 0x7f526c154390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:34.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.076+0000 7f526bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f526c159060 0x7f526c154390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:53838/0 (socket says 172.21.15.57:53838) 2023-10-19T14:01:34.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.076+0000 7f526bfff640 1 -- 172.21.15.57:0/2597358824 learned_addr learned my addr 172.21.15.57:0/2597358824 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:01:34.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.076+0000 7f5270b73640 1 --2- 172.21.15.57:0/2597358824 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f526c0cdb30 0x7f526c06afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:34.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.076+0000 7f5271374640 1 --2- 172.21.15.57:0/2597358824 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f526c15ccf0 0x7f526c1f4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:34.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.076+0000 7f526bfff640 1 -- 172.21.15.57:0/2597358824 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f526c0cdb30 msgr2=0x7f526c06afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:34.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.076+0000 7f526bfff640 1 --2- 172.21.15.57:0/2597358824 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f526c0cdb30 0x7f526c06afa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:34.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.076+0000 7f526bfff640 1 -- 172.21.15.57:0/2597358824 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f526c15ccf0 msgr2=0x7f526c1f4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:34.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.076+0000 7f526bfff640 1 --2- 172.21.15.57:0/2597358824 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f526c15ccf0 0x7f526c1f4c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:34.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.076+0000 7f526bfff640 1 -- 172.21.15.57:0/2597358824 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5258009560 con 0x7f526c159060 2023-10-19T14:01:34.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.077+0000 7f526bfff640 1 --2- 172.21.15.57:0/2597358824 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f526c159060 0x7f526c154390 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7f5258036040 tx=0x7f525802d9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:34.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.077+0000 7f52697fa640 1 -- 172.21.15.57:0/2597358824 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f525802ddd0 con 0x7f526c159060 2023-10-19T14:01:34.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.077+0000 7f52697fa640 1 -- 172.21.15.57:0/2597358824 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f52580036e0 con 0x7f526c159060 2023-10-19T14:01:34.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.077+0000 7f5272dd7640 1 -- 172.21.15.57:0/2597358824 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f526c06c690 con 0x7f526c159060 2023-10-19T14:01:34.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.077+0000 7f52697fa640 1 -- 172.21.15.57:0/2597358824 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5258003a90 con 0x7f526c159060 2023-10-19T14:01:34.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.077+0000 7f5272dd7640 1 -- 172.21.15.57:0/2597358824 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f526c153b90 con 0x7f526c159060 2023-10-19T14:01:34.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.078+0000 7f5272dd7640 1 -- 172.21.15.57:0/2597358824 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f526c0675f0 con 0x7f526c159060 2023-10-19T14:01:34.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.078+0000 7f52697fa640 1 -- 172.21.15.57:0/2597358824 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 91) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f5258003bf0 con 0x7f526c159060 2023-10-19T14:01:34.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.079+0000 7f52697fa640 1 --2- 172.21.15.57:0/2597358824 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f525006af60 0x7f525006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:34.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.079+0000 7f52697fa640 1 -- 172.21.15.57:0/2597358824 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 5979+0+0 (secure 0 0 0) 0x7f52580abe40 con 0x7f526c159060 2023-10-19T14:01:34.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.079+0000 7f5270b73640 1 --2- 172.21.15.57:0/2597358824 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f525006af60 0x7f525006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:34.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.080+0000 7f5270b73640 1 --2- 172.21.15.57:0/2597358824 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f525006af60 0x7f525006d420 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f526c0afbc0 tx=0x7f5260005f90 comp rx=0 tx=0).ready entity=mgr.8923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:34.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.083+0000 7f52697fa640 1 -- 172.21.15.57:0/2597358824 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f5258076f30 con 0x7f526c159060 2023-10-19T14:01:34.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.296+0000 7f5272dd7640 1 -- 172.21.15.57:0/2597358824 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f526c0b4aa0 con 0x7f526c159060 2023-10-19T14:01:34.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.297+0000 7f52697fa640 1 -- 172.21.15.57:0/2597358824 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+3700 (secure 0 0 0) 0x7f5258042070 con 0x7f526c159060 2023-10-19T14:01:34.299 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:01:34.299 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":65,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7733,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/1818414654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":1818414654},{"type":"v1","addr":"172.21.15.179:6836","nonce":1818414654}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":8647,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/490712721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":490712721},{"type":"v1","addr":"172.21.15.57:6835","nonce":490712721}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":8656,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/2005811112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":2005811112},{"type":"v1","addr":"172.21.15.179:6837","nonce":2005811112}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":10047,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6833/104040435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":104040435},{"type":"v1","addr":"172.21.15.57:6833","nonce":104040435}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65}],"filesystems":[]} 2023-10-19T14:01:34.299 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 65 2023-10-19T14:01:34.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.299+0000 7f5272dd7640 1 -- 172.21.15.57:0/2597358824 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f525006af60 msgr2=0x7f525006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:34.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.299+0000 7f5272dd7640 1 --2- 172.21.15.57:0/2597358824 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f525006af60 0x7f525006d420 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f526c0afbc0 tx=0x7f5260005f90 comp rx=0 tx=0).stop 2023-10-19T14:01:34.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.299+0000 7f5272dd7640 1 -- 172.21.15.57:0/2597358824 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f526c159060 msgr2=0x7f526c154390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:34.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.299+0000 7f5272dd7640 1 --2- 172.21.15.57:0/2597358824 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f526c159060 0x7f526c154390 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7f5258036040 tx=0x7f525802d9d0 comp rx=0 tx=0).stop 2023-10-19T14:01:34.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.299+0000 7f5272dd7640 1 -- 172.21.15.57:0/2597358824 shutdown_connections 2023-10-19T14:01:34.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.299+0000 7f5272dd7640 1 --2- 172.21.15.57:0/2597358824 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f525006af60 0x7f525006d420 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:34.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.299+0000 7f5272dd7640 1 --2- 172.21.15.57:0/2597358824 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f526c15ccf0 0x7f526c1f4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:34.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.299+0000 7f5272dd7640 1 --2- 172.21.15.57:0/2597358824 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f526c159060 0x7f526c154390 unknown :-1 s=CLOSED pgs=1872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:34.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.299+0000 7f5272dd7640 1 --2- 172.21.15.57:0/2597358824 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f526c0cdb30 0x7f526c06afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:34.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.299+0000 7f5272dd7640 1 -- 172.21.15.57:0/2597358824 >> 172.21.15.57:0/2597358824 conn(0x7f526c0b0670 msgr2=0x7f526c0ca330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:34.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.299+0000 7f5272dd7640 1 -- 172.21.15.57:0/2597358824 shutdown_connections 2023-10-19T14:01:34.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.299+0000 7f5272dd7640 1 -- 172.21.15.57:0/2597358824 wait complete. 2023-10-19T14:01:34.311 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-19T14:01:34.311 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-19T14:01:34.312 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-19T14:01:34.312 INFO:teuthology.orchestra.run:waiting for 300 2023-10-19T14:01:34.312 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-19T14:01:34.312 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-19T14:01:34.313 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-19T14:01:34.313 INFO:teuthology.orchestra.run:waiting for 300 2023-10-19T14:01:34.313 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-19T14:01:34.314 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-19T14:01:34.314 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-19T14:01:34.314 INFO:teuthology.orchestra.run:waiting for 300 2023-10-19T14:01:34.315 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-19T14:01:34.315 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-19T14:01:34.315 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-19T14:01:34.315 INFO:teuthology.orchestra.run:waiting for 300 2023-10-19T14:01:34.322 INFO:tasks.ceph.mds.a:Stopped 2023-10-19T14:01:34.322 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-19T14:01:34.323 INFO:tasks.ceph.mds.c:Stopped 2023-10-19T14:01:34.323 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-19T14:01:34.326 INFO:tasks.ceph.mds.b:Stopped 2023-10-19T14:01:34.326 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-19T14:01:34.327 INFO:tasks.ceph.mds.d:Stopped 2023-10-19T14:01:34.327 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-19T14:01:34.365 INFO:tasks.ceph.mds.a:Started 2023-10-19T14:01:34.367 INFO:tasks.ceph.mds.c:Started 2023-10-19T14:01:34.367 DEBUG:teuthology.parallel:result is None 2023-10-19T14:01:34.368 DEBUG:teuthology.parallel:result is None 2023-10-19T14:01:34.371 INFO:tasks.ceph.mds.b:Started 2023-10-19T14:01:34.371 DEBUG:teuthology.parallel:result is None 2023-10-19T14:01:34.372 INFO:tasks.ceph.mds.d:Started 2023-10-19T14:01:34.373 DEBUG:teuthology.parallel:result is None 2023-10-19T14:01:34.373 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-19T14:01:34.445 INFO:tasks.ceph.mds.a.smithi057.stdout:starting mds.a at 2023-10-19T14:01:34.446 INFO:tasks.ceph.mds.c.smithi057.stdout:starting mds.c at 2023-10-19T14:01:34.451 INFO:tasks.ceph.mds.b.smithi179.stdout:starting mds.b at 2023-10-19T14:01:34.451 INFO:tasks.ceph.mds.d.smithi179.stdout:starting mds.d at 2023-10-19T14:01:34.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.487+0000 7f00f2fb0640 1 -- 172.21.15.57:0/4008067328 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f00ec0ceb40 msgr2=0x7f00ec0b0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:34.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.487+0000 7f00f2fb0640 1 --2- 172.21.15.57:0/4008067328 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f00ec0ceb40 0x7f00ec0b0650 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7f00e4005fd0 tx=0x7f00e402bd90 comp rx=0 tx=0).stop 2023-10-19T14:01:34.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.487+0000 7f00f2fb0640 1 -- 172.21.15.57:0/4008067328 shutdown_connections 2023-10-19T14:01:34.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.487+0000 7f00f2fb0640 1 --2- 172.21.15.57:0/4008067328 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f00ec0ceb40 0x7f00ec0b0650 unknown :-1 s=CLOSED pgs=1877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:34.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.487+0000 7f00f2fb0640 1 --2- 172.21.15.57:0/4008067328 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f00ec0cf340 0x7f00ec0d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:34.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.487+0000 7f00f2fb0640 1 -- 172.21.15.57:0/4008067328 >> 172.21.15.57:0/4008067328 conn(0x7f00ec0d0d60 msgr2=0x7f00ec0d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:34.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.487+0000 7f00f2fb0640 1 -- 172.21.15.57:0/4008067328 shutdown_connections 2023-10-19T14:01:34.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.487+0000 7f00f2fb0640 1 -- 172.21.15.57:0/4008067328 wait complete. 2023-10-19T14:01:34.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.488+0000 7f00f2fb0640 1 Processor -- start 2023-10-19T14:01:34.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.488+0000 7f00f2fb0640 1 -- start start 2023-10-19T14:01:34.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.489+0000 7f00f2fb0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f00ec0ceb40 0x7f00ec157bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:34.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.489+0000 7f00f2fb0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f00ec0cf340 0x7f00ec1584c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:34.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.489+0000 7f00f2fb0640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f00ec15b2a0 0x7f00ec15cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:34.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.489+0000 7f00f2fb0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00ec0b5a40 con 0x7f00ec0cf340 2023-10-19T14:01:34.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.489+0000 7f00f2fb0640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00ec158130 con 0x7f00ec0ceb40 2023-10-19T14:01:34.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.489+0000 7f00f2fb0640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00ec09e5d0 con 0x7f00ec15b2a0 2023-10-19T14:01:34.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.489+0000 7f00ebfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f00ec0cf340 0x7f00ec1584c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:34.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.489+0000 7f00ebfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f00ec0cf340 0x7f00ec1584c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:53864/0 (socket says 172.21.15.57:53864) 2023-10-19T14:01:34.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.489+0000 7f00ebfff640 1 -- 172.21.15.57:0/1522622093 learned_addr learned my addr 172.21.15.57:0/1522622093 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:01:34.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.489+0000 7f00f0d4c640 1 --2- 172.21.15.57:0/1522622093 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f00ec0ceb40 0x7f00ec157bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:34.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.489+0000 7f00ebfff640 1 -- 172.21.15.57:0/1522622093 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f00ec15b2a0 msgr2=0x7f00ec15cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:34.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.489+0000 7f00ebfff640 1 --2- 172.21.15.57:0/1522622093 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f00ec15b2a0 0x7f00ec15cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:34.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.489+0000 7f00ebfff640 1 -- 172.21.15.57:0/1522622093 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f00ec0ceb40 msgr2=0x7f00ec157bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:34.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.489+0000 7f00ebfff640 1 --2- 172.21.15.57:0/1522622093 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f00ec0ceb40 0x7f00ec157bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:34.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.489+0000 7f00ebfff640 1 -- 172.21.15.57:0/1522622093 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00e4009560 con 0x7f00ec0cf340 2023-10-19T14:01:34.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.489+0000 7f00ebfff640 1 --2- 172.21.15.57:0/1522622093 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f00ec0cf340 0x7f00ec1584c0 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7f00dc00c730 tx=0x7f00dc00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:34.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.490+0000 7f00e97fa640 1 -- 172.21.15.57:0/1522622093 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00dc003b70 con 0x7f00ec0cf340 2023-10-19T14:01:34.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.490+0000 7f00e97fa640 1 -- 172.21.15.57:0/1522622093 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f00dc003cd0 con 0x7f00ec0cf340 2023-10-19T14:01:34.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.490+0000 7f00f2fb0640 1 -- 172.21.15.57:0/1522622093 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00ec158a00 con 0x7f00ec0cf340 2023-10-19T14:01:34.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.490+0000 7f00e97fa640 1 -- 172.21.15.57:0/1522622093 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00dc00aab0 con 0x7f00ec0cf340 2023-10-19T14:01:34.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.490+0000 7f00f2fb0640 1 -- 172.21.15.57:0/1522622093 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00ec15c280 con 0x7f00ec0cf340 2023-10-19T14:01:34.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.491+0000 7f00f2fb0640 1 -- 172.21.15.57:0/1522622093 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00ec0690e0 con 0x7f00ec0cf340 2023-10-19T14:01:34.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.491+0000 7f00e97fa640 1 -- 172.21.15.57:0/1522622093 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 91) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f00dc00ac10 con 0x7f00ec0cf340 2023-10-19T14:01:34.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.492+0000 7f00e97fa640 1 --2- 172.21.15.57:0/1522622093 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f00cc06af60 0x7f00cc06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:34.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.492+0000 7f00e97fa640 1 -- 172.21.15.57:0/1522622093 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 5979+0+0 (secure 0 0 0) 0x7f00dc0897b0 con 0x7f00ec0cf340 2023-10-19T14:01:34.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.492+0000 7f00f0d4c640 1 --2- 172.21.15.57:0/1522622093 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f00cc06af60 0x7f00cc06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:34.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.492+0000 7f00f0d4c640 1 --2- 172.21.15.57:0/1522622093 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f00cc06af60 0x7f00cc06d420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f00d4009a60 tx=0x7f00d4005ed0 comp rx=0 tx=0).ready entity=mgr.8923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:34.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.496+0000 7f00e97fa640 1 -- 172.21.15.57:0/1522622093 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f00dc0546d0 con 0x7f00ec0cf340 2023-10-19T14:01:34.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:34.685+0000 7f00f2fb0640 1 -- 172.21.15.57:0/1522622093 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f00ec0cb9b0 con 0x7f00ec0cf340 2023-10-19T14:01:35.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.121+0000 7f00e97fa640 1 -- 172.21.15.57:0/1522622093 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v169) v1 ==== 97+0+0 (secure 0 0 0) 0x7f00dc057cf0 con 0x7f00ec0cf340 2023-10-19T14:01:35.123 INFO:teuthology.orchestra.run.smithi057.stderr: removed all blocklist entries 2023-10-19T14:01:35.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.123+0000 7f00f2fb0640 1 -- 172.21.15.57:0/1522622093 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f00cc06af60 msgr2=0x7f00cc06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:35.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.123+0000 7f00f2fb0640 1 --2- 172.21.15.57:0/1522622093 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f00cc06af60 0x7f00cc06d420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f00d4009a60 tx=0x7f00d4005ed0 comp rx=0 tx=0).stop 2023-10-19T14:01:35.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.123+0000 7f00f2fb0640 1 -- 172.21.15.57:0/1522622093 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f00ec0cf340 msgr2=0x7f00ec1584c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:35.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.123+0000 7f00f2fb0640 1 --2- 172.21.15.57:0/1522622093 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f00ec0cf340 0x7f00ec1584c0 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7f00dc00c730 tx=0x7f00dc00cbf0 comp rx=0 tx=0).stop 2023-10-19T14:01:35.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.123+0000 7f00f2fb0640 1 -- 172.21.15.57:0/1522622093 shutdown_connections 2023-10-19T14:01:35.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.123+0000 7f00f2fb0640 1 --2- 172.21.15.57:0/1522622093 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f00cc06af60 0x7f00cc06d420 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:35.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.123+0000 7f00f2fb0640 1 --2- 172.21.15.57:0/1522622093 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f00ec15b2a0 0x7f00ec15cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:35.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.123+0000 7f00f2fb0640 1 --2- 172.21.15.57:0/1522622093 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f00ec0cf340 0x7f00ec1584c0 unknown :-1 s=CLOSED pgs=1878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:35.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.123+0000 7f00f2fb0640 1 --2- 172.21.15.57:0/1522622093 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f00ec0ceb40 0x7f00ec157bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:35.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.123+0000 7f00f2fb0640 1 -- 172.21.15.57:0/1522622093 >> 172.21.15.57:0/1522622093 conn(0x7f00ec0d0d60 msgr2=0x7f00ec06b450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:35.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.123+0000 7f00f2fb0640 1 -- 172.21.15.57:0/1522622093 shutdown_connections 2023-10-19T14:01:35.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.123+0000 7f00f2fb0640 1 -- 172.21.15.57:0/1522622093 wait complete. 2023-10-19T14:01:35.138 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-19T14:01:35.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.248+0000 7f0380b1d640 1 -- 172.21.15.57:0/739893254 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0368003970 msgr2=0x7f0368003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:35.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.248+0000 7f0380b1d640 1 --2- 172.21.15.57:0/739893254 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0368003970 0x7f0368003d90 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7f0374001680 tx=0x7f03680057f0 comp rx=0 tx=0).stop 2023-10-19T14:01:35.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.248+0000 7f0380b1d640 1 -- 172.21.15.57:0/739893254 shutdown_connections 2023-10-19T14:01:35.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.248+0000 7f0380b1d640 1 --2- 172.21.15.57:0/739893254 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0368003970 0x7f0368003d90 unknown :-1 s=CLOSED pgs=1880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:35.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.248+0000 7f0380b1d640 1 -- 172.21.15.57:0/739893254 >> 172.21.15.57:0/739893254 conn(0x7f037c069930 msgr2=0x7f037c069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:35.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.248+0000 7f0380b1d640 1 -- 172.21.15.57:0/739893254 shutdown_connections 2023-10-19T14:01:35.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.248+0000 7f0380b1d640 1 -- 172.21.15.57:0/739893254 wait complete. 2023-10-19T14:01:35.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.249+0000 7f0380b1d640 1 Processor -- start 2023-10-19T14:01:35.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.249+0000 7f0380b1d640 1 -- start start 2023-10-19T14:01:35.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.250+0000 7f0380b1d640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0368003970 0x7f037c152f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:35.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.250+0000 7f0380b1d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f037c134290 0x7f037c138e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:35.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.250+0000 7f0380b1d640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f037c1ef150 0x7f037c1ef600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:35.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.250+0000 7f0380b1d640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f037c1efb40 con 0x7f037c134290 2023-10-19T14:01:35.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.250+0000 7f0380b1d640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f037c1358c0 con 0x7f037c1ef150 2023-10-19T14:01:35.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.250+0000 7f0380b1d640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f037c134820 con 0x7f0368003970 2023-10-19T14:01:35.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.250+0000 7f0379d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f037c134290 0x7f037c138e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:35.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.250+0000 7f0379d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f037c134290 0x7f037c138e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:53890/0 (socket says 172.21.15.57:53890) 2023-10-19T14:01:35.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.250+0000 7f0379d9b640 1 -- 172.21.15.57:0/2147756492 learned_addr learned my addr 172.21.15.57:0/2147756492 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:01:35.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.250+0000 7f037ad9d640 1 --2- 172.21.15.57:0/2147756492 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f037c1ef150 0x7f037c1ef600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:35.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.250+0000 7f037a59c640 1 --2- 172.21.15.57:0/2147756492 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0368003970 0x7f037c152f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:35.262 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.250+0000 7f0379d9b640 1 -- 172.21.15.57:0/2147756492 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0368003970 msgr2=0x7f037c152f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:35.262 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.251+0000 7f0379d9b640 1 --2- 172.21.15.57:0/2147756492 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0368003970 0x7f037c152f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:35.262 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.251+0000 7f0379d9b640 1 -- 172.21.15.57:0/2147756492 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f037c1ef150 msgr2=0x7f037c1ef600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:35.263 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.251+0000 7f0379d9b640 1 --2- 172.21.15.57:0/2147756492 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f037c1ef150 0x7f037c1ef600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:35.263 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.251+0000 7f0379d9b640 1 -- 172.21.15.57:0/2147756492 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f037c134b00 con 0x7f037c134290 2023-10-19T14:01:35.263 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.251+0000 7f0379d9b640 1 --2- 172.21.15.57:0/2147756492 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f037c134290 0x7f037c138e70 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7f0364006550 tx=0x7f03640039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:35.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.251+0000 7f0372ffd640 1 -- 172.21.15.57:0/2147756492 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03640011c0 con 0x7f037c134290 2023-10-19T14:01:35.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.251+0000 7f0372ffd640 1 -- 172.21.15.57:0/2147756492 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f0364011940 con 0x7f037c134290 2023-10-19T14:01:35.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.251+0000 7f0380b1d640 1 -- 172.21.15.57:0/2147756492 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f037c134de0 con 0x7f037c134290 2023-10-19T14:01:35.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.251+0000 7f0372ffd640 1 -- 172.21.15.57:0/2147756492 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0364011d50 con 0x7f037c134290 2023-10-19T14:01:35.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.251+0000 7f0380b1d640 1 -- 172.21.15.57:0/2147756492 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f037c1353a0 con 0x7f037c134290 2023-10-19T14:01:35.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.252+0000 7f0380b1d640 1 -- 172.21.15.57:0/2147756492 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f037c1355c0 con 0x7f037c134290 2023-10-19T14:01:35.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.252+0000 7f0372ffd640 1 -- 172.21.15.57:0/2147756492 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 91) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f03640103c0 con 0x7f037c134290 2023-10-19T14:01:35.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.253+0000 7f0372ffd640 1 --2- 172.21.15.57:0/2147756492 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f035c06af60 0x7f035c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:35.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.253+0000 7f0372ffd640 1 -- 172.21.15.57:0/2147756492 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f036408a660 con 0x7f037c134290 2023-10-19T14:01:35.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.253+0000 7f037a59c640 1 --2- 172.21.15.57:0/2147756492 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f035c06af60 0x7f035c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:35.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.253+0000 7f037a59c640 1 --2- 172.21.15.57:0/2147756492 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f035c06af60 0x7f035c06d420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0374034000 tx=0x7f0374001190 comp rx=0 tx=0).ready entity=mgr.8923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:35.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.257+0000 7f0372ffd640 1 -- 172.21.15.57:0/2147756492 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0364055670 con 0x7f037c134290 2023-10-19T14:01:35.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.497+0000 7f0380b1d640 1 -- 172.21.15.57:0/2147756492 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f037c0ca0e0 con 0x7f037c134290 2023-10-19T14:01:35.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.498+0000 7f0372ffd640 1 -- 172.21.15.57:0/2147756492 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v50) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f037c0ca0e0 con 0x7f037c134290 2023-10-19T14:01:35.500 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:01:35.500 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2023-10-19T14:01:35.500 INFO:teuthology.orchestra.run.smithi057.stdout: "auth_dump": [ 2023-10-19T14:01:35.500 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:01:35.501 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "mds.a", 2023-10-19T14:01:35.501 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBjJDFlmbVANhAAP/ynHlU/OxaRq5lQoAa8Ng==", 2023-10-19T14:01:35.501 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:01:35.501 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T14:01:35.501 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow *", 2023-10-19T14:01:35.502 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow *", 2023-10-19T14:01:35.502 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:01:35.502 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:01:35.502 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:01:35.503 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:01:35.503 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "mds.b", 2023-10-19T14:01:35.503 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFl9l7RBxAAC0geI2tJ1SIZZr1++kS/BA==", 2023-10-19T14:01:35.503 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:01:35.503 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T14:01:35.504 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow *", 2023-10-19T14:01:35.504 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow *", 2023-10-19T14:01:35.504 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:01:35.504 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:01:35.504 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:01:35.504 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:01:35.505 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "mds.c", 2023-10-19T14:01:35.505 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFlFkaoAhAA/PBizkJ4bm958oP4CmPuIQ==", 2023-10-19T14:01:35.505 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:01:35.505 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T14:01:35.506 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow *", 2023-10-19T14:01:35.506 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow *", 2023-10-19T14:01:35.506 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:01:35.506 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:01:35.507 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:01:35.507 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:01:35.507 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "mds.d", 2023-10-19T14:01:35.507 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFlx8skDxAAAfCHTYtYJgJik0WHSWeQfQ==", 2023-10-19T14:01:35.507 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:01:35.508 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T14:01:35.508 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow *", 2023-10-19T14:01:35.508 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow *", 2023-10-19T14:01:35.508 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:01:35.508 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:01:35.508 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:01:35.509 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:01:35.509 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.0", 2023-10-19T14:01:35.509 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBrJDFlCo9sBxAAQN2cID4nCQgRwGaGFEV8LQ==", 2023-10-19T14:01:35.509 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:01:35.509 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T14:01:35.510 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T14:01:35.510 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:01:35.510 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:01:35.510 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:01:35.510 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:01:35.511 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.1", 2023-10-19T14:01:35.511 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBuJDFl4fsLCBAAp3Pi80yZtc5sju0GisEuVA==", 2023-10-19T14:01:35.511 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:01:35.511 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T14:01:35.511 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T14:01:35.512 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:01:35.512 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:01:35.512 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:01:35.512 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:01:35.512 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.2", 2023-10-19T14:01:35.513 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBxJDFl+KAKCBAAd4N9cluARxPVfBLU+7Hd7A==", 2023-10-19T14:01:35.513 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:01:35.513 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T14:01:35.513 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T14:01:35.513 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:01:35.514 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:01:35.514 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:01:35.514 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:01:35.514 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.3", 2023-10-19T14:01:35.514 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQB0JDFlU/toCBAAP0Ou1zcHKNx5SfbybJkh4Q==", 2023-10-19T14:01:35.515 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:01:35.515 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T14:01:35.515 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T14:01:35.516 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:01:35.517 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:01:35.517 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:01:35.517 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:01:35.517 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.4", 2023-10-19T14:01:35.517 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQB6JDFlc6stEBAALkgNyxoiCTzZnKpBSPxI6w==", 2023-10-19T14:01:35.518 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:01:35.518 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T14:01:35.518 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T14:01:35.518 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:01:35.518 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:01:35.519 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:01:35.519 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:01:35.519 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.5", 2023-10-19T14:01:35.519 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQB9JDFlI3cNDhAAuG3tHJ6RvPwAzPE76wTIVQ==", 2023-10-19T14:01:35.520 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:01:35.520 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T14:01:35.520 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T14:01:35.520 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:01:35.520 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:01:35.521 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:01:35.521 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:01:35.521 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.6", 2023-10-19T14:01:35.521 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQCAJDFlLPumCxAA/iy08t/2N9rXfn71KtbSQw==", 2023-10-19T14:01:35.521 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:01:35.521 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T14:01:35.522 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T14:01:35.522 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:01:35.522 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:01:35.522 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:01:35.523 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:01:35.523 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.7", 2023-10-19T14:01:35.523 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQCDJDFlHmrbCRAA+hpWygGIJydfU6cHWlfx/Q==", 2023-10-19T14:01:35.523 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:01:35.523 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T14:01:35.524 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T14:01:35.524 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:01:35.524 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:01:35.524 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:01:35.524 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:01:35.525 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "client.0", 2023-10-19T14:01:35.525 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFlg9yuExAATxiWDfS+p07G1F0pORbJEg==", 2023-10-19T14:01:35.525 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:01:35.525 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T14:01:35.525 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow r", 2023-10-19T14:01:35.525 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-19T14:01:35.526 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:01:35.526 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:01:35.526 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:01:35.526 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "client.1", 2023-10-19T14:01:35.526 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFl9IwYGxAAHPituiJSPkHDuD0QyG4Hww==", 2023-10-19T14:01:35.527 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:01:35.527 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T14:01:35.527 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow r", 2023-10-19T14:01:35.527 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow rw", 2023-10-19T14:01:35.528 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow rwx" 2023-10-19T14:01:35.528 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:01:35.528 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:01:35.528 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:01:35.528 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "client.2", 2023-10-19T14:01:35.529 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFlMd+AFxAAS/UTESDt76GmrhhN2o/Fdw==", 2023-10-19T14:01:35.529 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:01:35.529 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T14:01:35.529 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow r", 2023-10-19T14:01:35.529 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow rw", 2023-10-19T14:01:35.530 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow rwx" 2023-10-19T14:01:35.530 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:01:35.530 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:01:35.530 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:01:35.530 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "client.3", 2023-10-19T14:01:35.531 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFlHZ3fIBAA42B7n8opLS5smXIwJyVYzw==", 2023-10-19T14:01:35.531 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:01:35.531 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T14:01:35.531 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow r", 2023-10-19T14:01:35.531 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow rw", 2023-10-19T14:01:35.532 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow rwx" 2023-10-19T14:01:35.532 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:01:35.532 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:01:35.532 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:01:35.532 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "client.admin", 2023-10-19T14:01:35.532 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBjJDFlBgasFxAAMA24j8KszcXMEpYRQDXj4g==", 2023-10-19T14:01:35.533 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:01:35.533 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow *", 2023-10-19T14:01:35.533 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow *", 2023-10-19T14:01:35.533 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow *", 2023-10-19T14:01:35.533 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:01:35.534 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:01:35.534 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:01:35.534 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:01:35.534 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "mgr.x", 2023-10-19T14:01:35.534 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBjJDFllFBfMhAA5+KPXWmYZqcESmLObaUbBA==", 2023-10-19T14:01:35.535 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:01:35.535 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow *", 2023-10-19T14:01:35.535 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile mgr", 2023-10-19T14:01:35.535 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:01:35.535 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:01:35.536 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:01:35.536 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:01:35.536 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "mgr.y", 2023-10-19T14:01:35.536 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBjJDFlxRzJLRAAGjG58o7qXAHvIEIRYwl6QA==", 2023-10-19T14:01:35.536 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:01:35.536 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow *", 2023-10-19T14:01:35.537 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile mgr", 2023-10-19T14:01:35.537 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:01:35.537 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:01:35.537 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:01:35.537 INFO:teuthology.orchestra.run.smithi057.stdout: ] 2023-10-19T14:01:35.538 INFO:teuthology.orchestra.run.smithi057.stdout:} 2023-10-19T14:01:35.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.500+0000 7f0380b1d640 1 -- 172.21.15.57:0/2147756492 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f035c06af60 msgr2=0x7f035c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:35.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.500+0000 7f0380b1d640 1 --2- 172.21.15.57:0/2147756492 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f035c06af60 0x7f035c06d420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0374034000 tx=0x7f0374001190 comp rx=0 tx=0).stop 2023-10-19T14:01:35.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.500+0000 7f0380b1d640 1 -- 172.21.15.57:0/2147756492 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f037c134290 msgr2=0x7f037c138e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:35.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.500+0000 7f0380b1d640 1 --2- 172.21.15.57:0/2147756492 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f037c134290 0x7f037c138e70 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7f0364006550 tx=0x7f03640039c0 comp rx=0 tx=0).stop 2023-10-19T14:01:35.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.500+0000 7f0380b1d640 1 -- 172.21.15.57:0/2147756492 shutdown_connections 2023-10-19T14:01:35.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.500+0000 7f0380b1d640 1 --2- 172.21.15.57:0/2147756492 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f035c06af60 0x7f035c06d420 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:35.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.500+0000 7f0380b1d640 1 --2- 172.21.15.57:0/2147756492 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f037c1ef150 0x7f037c1ef600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:35.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.500+0000 7f0380b1d640 1 --2- 172.21.15.57:0/2147756492 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f037c134290 0x7f037c138e70 unknown :-1 s=CLOSED pgs=1881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:35.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.500+0000 7f0380b1d640 1 --2- 172.21.15.57:0/2147756492 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0368003970 0x7f037c152f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:35.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.500+0000 7f0380b1d640 1 -- 172.21.15.57:0/2147756492 >> 172.21.15.57:0/2147756492 conn(0x7f037c069930 msgr2=0x7f037c0ab950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:35.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.500+0000 7f0380b1d640 1 -- 172.21.15.57:0/2147756492 shutdown_connections 2023-10-19T14:01:35.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.500+0000 7f0380b1d640 1 -- 172.21.15.57:0/2147756492 wait complete. 2023-10-19T14:01:35.543 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T14:01:35.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.650+0000 7f478b7a5640 1 -- 172.21.15.57:0/1921988453 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f478415b2a0 msgr2=0x7f47840cf720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:35.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.650+0000 7f478b7a5640 1 --2- 172.21.15.57:0/1921988453 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f478415b2a0 0x7f47840cf720 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7f478408a810 tx=0x7f477402bd10 comp rx=0 tx=0).stop 2023-10-19T14:01:35.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.650+0000 7f478b7a5640 1 -- 172.21.15.57:0/1921988453 shutdown_connections 2023-10-19T14:01:35.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.650+0000 7f478b7a5640 1 --2- 172.21.15.57:0/1921988453 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f47840ceb40 0x7f47840b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:35.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.650+0000 7f478b7a5640 1 --2- 172.21.15.57:0/1921988453 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f478415b2a0 0x7f47840cf720 unknown :-1 s=CLOSED pgs=1882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:35.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.650+0000 7f478b7a5640 1 --2- 172.21.15.57:0/1921988453 >> v2:172.21.15.179:3300/0 conn(0x7f47840cf340 0x7f47840d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:35.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.650+0000 7f478b7a5640 1 -- 172.21.15.57:0/1921988453 >> 172.21.15.57:0/1921988453 conn(0x7f47840d0d60 msgr2=0x7f47840d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:35.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.650+0000 7f478b7a5640 1 -- 172.21.15.57:0/1921988453 shutdown_connections 2023-10-19T14:01:35.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.650+0000 7f478b7a5640 1 -- 172.21.15.57:0/1921988453 wait complete. 2023-10-19T14:01:35.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.651+0000 7f478b7a5640 1 Processor -- start 2023-10-19T14:01:35.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.652+0000 7f478b7a5640 1 -- start start 2023-10-19T14:01:35.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.652+0000 7f478b7a5640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f47840ceb40 0x7f478406b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:35.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.652+0000 7f478b7a5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f47840cf340 0x7f4784146870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:35.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.652+0000 7f478b7a5640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f478415b2a0 0x7f4784146db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:35.658 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.652+0000 7f478b7a5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47841472f0 con 0x7f47840cf340 2023-10-19T14:01:35.658 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.652+0000 7f478b7a5640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4784148020 con 0x7f47840ceb40 2023-10-19T14:01:35.658 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.652+0000 7f478b7a5640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47841476f0 con 0x7f478415b2a0 2023-10-19T14:01:35.658 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.652+0000 7f4788d40640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f47840cf340 0x7f4784146870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:35.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.652+0000 7f4788d40640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f47840cf340 0x7f4784146870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:53916/0 (socket says 172.21.15.57:53916) 2023-10-19T14:01:35.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.652+0000 7f4788d40640 1 -- 172.21.15.57:0/1515530690 learned_addr learned my addr 172.21.15.57:0/1515530690 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:01:35.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.652+0000 7f4789541640 1 --2- 172.21.15.57:0/1515530690 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f47840ceb40 0x7f478406b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:35.660 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.652+0000 7f4789d42640 1 --2- 172.21.15.57:0/1515530690 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f478415b2a0 0x7f4784146db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:35.660 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.653+0000 7f4788d40640 1 -- 172.21.15.57:0/1515530690 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f478415b2a0 msgr2=0x7f4784146db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:35.660 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.653+0000 7f4788d40640 1 --2- 172.21.15.57:0/1515530690 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f478415b2a0 0x7f4784146db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:35.661 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.653+0000 7f4788d40640 1 -- 172.21.15.57:0/1515530690 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f47840ceb40 msgr2=0x7f478406b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:35.661 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.653+0000 7f4788d40640 1 --2- 172.21.15.57:0/1515530690 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f47840ceb40 0x7f478406b270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:35.661 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.653+0000 7f4788d40640 1 -- 172.21.15.57:0/1515530690 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4774009560 con 0x7f47840cf340 2023-10-19T14:01:35.662 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.653+0000 7f4788d40640 1 --2- 172.21.15.57:0/1515530690 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f47840cf340 0x7f4784146870 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7f47740099c0 tx=0x7f477402dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:35.662 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.653+0000 7f4779ffb640 1 -- 172.21.15.57:0/1515530690 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4774003800 con 0x7f47840cf340 2023-10-19T14:01:35.662 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.653+0000 7f4779ffb640 1 -- 172.21.15.57:0/1515530690 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f4774003a50 con 0x7f47840cf340 2023-10-19T14:01:35.663 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.653+0000 7f4779ffb640 1 -- 172.21.15.57:0/1515530690 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4774003e60 con 0x7f47840cf340 2023-10-19T14:01:35.663 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.653+0000 7f478b7a5640 1 -- 172.21.15.57:0/1515530690 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47841479d0 con 0x7f47840cf340 2023-10-19T14:01:35.663 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.653+0000 7f478b7a5640 1 -- 172.21.15.57:0/1515530690 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f478415e7b0 con 0x7f47840cf340 2023-10-19T14:01:35.664 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.655+0000 7f478b7a5640 1 -- 172.21.15.57:0/1515530690 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47840690e0 con 0x7f47840cf340 2023-10-19T14:01:35.664 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.655+0000 7f4779ffb640 1 -- 172.21.15.57:0/1515530690 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 91) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f477403e450 con 0x7f47840cf340 2023-10-19T14:01:35.664 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.655+0000 7f4779ffb640 1 --2- 172.21.15.57:0/1515530690 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f476806af60 0x7f476806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:35.664 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.655+0000 7f4779ffb640 1 -- 172.21.15.57:0/1515530690 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f47740abf30 con 0x7f47840cf340 2023-10-19T14:01:35.665 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.656+0000 7f4789541640 1 --2- 172.21.15.57:0/1515530690 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f476806af60 0x7f476806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:35.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.656+0000 7f4789541640 1 --2- 172.21.15.57:0/1515530690 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f476806af60 0x7f476806d420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f478000bbe0 tx=0x7f478000b390 comp rx=0 tx=0).ready entity=mgr.8923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:35.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.659+0000 7f4779ffb640 1 -- 172.21.15.57:0/1515530690 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4774077290 con 0x7f47840cf340 2023-10-19T14:01:35.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.835+0000 7f478b7a5640 1 -- 172.21.15.57:0/1515530690 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f47840d0b40 con 0x7f47840cf340 2023-10-19T14:01:35.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.836+0000 7f4779ffb640 1 -- 172.21.15.57:0/1515530690 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v171) v1 ==== 74+0+13127 (secure 0 0 0) 0x7f4774042090 con 0x7f47840cf340 2023-10-19T14:01:35.837 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:01:35.838 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":171,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T14:01:35.129074+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T14:01:35.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.838+0000 7f478b7a5640 1 -- 172.21.15.57:0/1515530690 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f476806af60 msgr2=0x7f476806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:35.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.838+0000 7f478b7a5640 1 --2- 172.21.15.57:0/1515530690 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f476806af60 0x7f476806d420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f478000bbe0 tx=0x7f478000b390 comp rx=0 tx=0).stop 2023-10-19T14:01:35.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.838+0000 7f478b7a5640 1 -- 172.21.15.57:0/1515530690 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f47840cf340 msgr2=0x7f4784146870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:35.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.838+0000 7f478b7a5640 1 --2- 172.21.15.57:0/1515530690 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f47840cf340 0x7f4784146870 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7f47740099c0 tx=0x7f477402dab0 comp rx=0 tx=0).stop 2023-10-19T14:01:35.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.838+0000 7f478b7a5640 1 -- 172.21.15.57:0/1515530690 shutdown_connections 2023-10-19T14:01:35.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.838+0000 7f478b7a5640 1 --2- 172.21.15.57:0/1515530690 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f476806af60 0x7f476806d420 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:35.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.838+0000 7f478b7a5640 1 --2- 172.21.15.57:0/1515530690 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f478415b2a0 0x7f4784146db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:35.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.838+0000 7f478b7a5640 1 --2- 172.21.15.57:0/1515530690 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f47840cf340 0x7f4784146870 unknown :-1 s=CLOSED pgs=1883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:35.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.838+0000 7f478b7a5640 1 --2- 172.21.15.57:0/1515530690 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f47840ceb40 0x7f478406b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:35.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.838+0000 7f478b7a5640 1 -- 172.21.15.57:0/1515530690 >> 172.21.15.57:0/1515530690 conn(0x7f47840d0d60 msgr2=0x7f47840aed40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:35.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.838+0000 7f478b7a5640 1 -- 172.21.15.57:0/1515530690 shutdown_connections 2023-10-19T14:01:35.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.838+0000 7f478b7a5640 1 -- 172.21.15.57:0/1515530690 wait complete. 2023-10-19T14:01:35.850 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-19T12:44:07.371186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-19T14:01:35.851 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-19T14:01:35.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.961+0000 7f1863a58640 1 -- 172.21.15.57:0/1219810970 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f185c0b0530 msgr2=0x7f185c0b09a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:35.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.961+0000 7f1863a58640 1 --2- 172.21.15.57:0/1219810970 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f185c0b0530 0x7f185c0b09a0 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7f185c0cc490 tx=0x7f185402c260 comp rx=0 tx=0).stop 2023-10-19T14:01:35.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.961+0000 7f1863a58640 1 -- 172.21.15.57:0/1219810970 shutdown_connections 2023-10-19T14:01:35.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.961+0000 7f1863a58640 1 --2- 172.21.15.57:0/1219810970 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f185c0b0530 0x7f185c0b09a0 unknown :-1 s=CLOSED pgs=1884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:35.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.961+0000 7f1863a58640 1 --2- 172.21.15.57:0/1219810970 >> v2:172.21.15.179:3300/0 conn(0x7f185c0afa20 0x7f185c158300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:35.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.961+0000 7f1863a58640 1 -- 172.21.15.57:0/1219810970 >> 172.21.15.57:0/1219810970 conn(0x7f185c069930 msgr2=0x7f185c069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:35.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.961+0000 7f1863a58640 1 -- 172.21.15.57:0/1219810970 shutdown_connections 2023-10-19T14:01:35.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.961+0000 7f1863a58640 1 -- 172.21.15.57:0/1219810970 wait complete. 2023-10-19T14:01:35.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.962+0000 7f1863a58640 1 Processor -- start 2023-10-19T14:01:35.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.962+0000 7f1863a58640 1 -- start start 2023-10-19T14:01:35.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.963+0000 7f1863a58640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f185c0afa20 0x7f185c12cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:35.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.963+0000 7f1863a58640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f185c0b0530 0x7f185c1434a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:35.973 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.963+0000 7f1863a58640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f185c0cbd20 0x7f185c1439e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:35.973 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.963+0000 7f1863a58640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f185c146760 con 0x7f185c0cbd20 2023-10-19T14:01:35.973 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.963+0000 7f1863a58640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f185c146a40 con 0x7f185c0b0530 2023-10-19T14:01:35.974 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.963+0000 7f1863a58640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f185c147f10 con 0x7f185c0afa20 2023-10-19T14:01:35.974 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.963+0000 7f1861ff5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f185c0cbd20 0x7f185c1439e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:35.974 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.963+0000 7f1861ff5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f185c0cbd20 0x7f185c1439e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:53932/0 (socket says 172.21.15.57:53932) 2023-10-19T14:01:35.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.963+0000 7f1861ff5640 1 -- 172.21.15.57:0/3205495949 learned_addr learned my addr 172.21.15.57:0/3205495949 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:01:35.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.963+0000 7f1860ff3640 1 --2- 172.21.15.57:0/3205495949 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f185c0b0530 0x7f185c1434a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:35.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.963+0000 7f18617f4640 1 --2- 172.21.15.57:0/3205495949 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f185c0afa20 0x7f185c12cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:35.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.963+0000 7f1861ff5640 1 -- 172.21.15.57:0/3205495949 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f185c0afa20 msgr2=0x7f185c12cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:35.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.963+0000 7f1861ff5640 1 --2- 172.21.15.57:0/3205495949 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f185c0afa20 0x7f185c12cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:35.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.963+0000 7f1861ff5640 1 -- 172.21.15.57:0/3205495949 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f185c0b0530 msgr2=0x7f185c1434a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:35.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.963+0000 7f1861ff5640 1 --2- 172.21.15.57:0/3205495949 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f185c0b0530 0x7f185c1434a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:35.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.963+0000 7f1861ff5640 1 -- 172.21.15.57:0/3205495949 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1854009900 con 0x7f185c0cbd20 2023-10-19T14:01:35.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.964+0000 7f1861ff5640 1 --2- 172.21.15.57:0/3205495949 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f185c0cbd20 0x7f185c1439e0 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 crypto rx=0x7f1854009d60 tx=0x7f1854003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:35.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.964+0000 7f1851ffb640 1 -- 172.21.15.57:0/3205495949 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1854004450 con 0x7f185c0cbd20 2023-10-19T14:01:35.979 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.964+0000 7f1851ffb640 1 -- 172.21.15.57:0/3205495949 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f18540046a0 con 0x7f185c0cbd20 2023-10-19T14:01:35.979 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.964+0000 7f1851ffb640 1 -- 172.21.15.57:0/3205495949 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18540406b0 con 0x7f185c0cbd20 2023-10-19T14:01:35.979 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.964+0000 7f1863a58640 1 -- 172.21.15.57:0/3205495949 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f185c1481f0 con 0x7f185c0cbd20 2023-10-19T14:01:35.979 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.964+0000 7f1863a58640 1 -- 172.21.15.57:0/3205495949 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f185c12d4a0 con 0x7f185c0cbd20 2023-10-19T14:01:35.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.965+0000 7f1863a58640 1 -- 172.21.15.57:0/3205495949 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f185c0cb8a0 con 0x7f185c0cbd20 2023-10-19T14:01:35.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.965+0000 7f1851ffb640 1 -- 172.21.15.57:0/3205495949 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 91) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f185400a040 con 0x7f185c0cbd20 2023-10-19T14:01:35.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.966+0000 7f1851ffb640 1 --2- 172.21.15.57:0/3205495949 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f184006af90 0x7f184006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:35.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.966+0000 7f1851ffb640 1 -- 172.21.15.57:0/3205495949 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f18540ac510 con 0x7f185c0cbd20 2023-10-19T14:01:35.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.966+0000 7f18617f4640 1 --2- 172.21.15.57:0/3205495949 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f184006af90 0x7f184006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:35.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.967+0000 7f18617f4640 1 --2- 172.21.15.57:0/3205495949 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f184006af90 0x7f184006d450 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f1858006fd0 tx=0x7f1858009060 comp rx=0 tx=0).ready entity=mgr.8923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:35.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:35.970+0000 7f1851ffb640 1 -- 172.21.15.57:0/3205495949 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f1854077750 con 0x7f185c0cbd20 2023-10-19T14:01:36.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.145+0000 7f1863a58640 1 -- 172.21.15.57:0/3205495949 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f185c1486b0 con 0x7f185c0cbd20 2023-10-19T14:01:36.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.145+0000 7f1851ffb640 1 -- 172.21.15.57:0/3205495949 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v171) v1 ==== 93+0+10 (secure 0 0 0) 0x7f18540390a0 con 0x7f185c0cbd20 2023-10-19T14:01:36.147 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-19T14:01:36.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.147+0000 7f1863a58640 1 -- 172.21.15.57:0/3205495949 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f184006af90 msgr2=0x7f184006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:36.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.147+0000 7f1863a58640 1 --2- 172.21.15.57:0/3205495949 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f184006af90 0x7f184006d450 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f1858006fd0 tx=0x7f1858009060 comp rx=0 tx=0).stop 2023-10-19T14:01:36.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.147+0000 7f1863a58640 1 -- 172.21.15.57:0/3205495949 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f185c0cbd20 msgr2=0x7f185c1439e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:36.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.147+0000 7f1863a58640 1 --2- 172.21.15.57:0/3205495949 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f185c0cbd20 0x7f185c1439e0 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 crypto rx=0x7f1854009d60 tx=0x7f1854003c20 comp rx=0 tx=0).stop 2023-10-19T14:01:36.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.147+0000 7f1863a58640 1 -- 172.21.15.57:0/3205495949 shutdown_connections 2023-10-19T14:01:36.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.147+0000 7f1863a58640 1 --2- 172.21.15.57:0/3205495949 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f184006af90 0x7f184006d450 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:36.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.147+0000 7f1863a58640 1 --2- 172.21.15.57:0/3205495949 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f185c0cbd20 0x7f185c1439e0 unknown :-1 s=CLOSED pgs=1885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:36.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.147+0000 7f1863a58640 1 --2- 172.21.15.57:0/3205495949 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f185c0b0530 0x7f185c1434a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:36.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.147+0000 7f1863a58640 1 --2- 172.21.15.57:0/3205495949 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f185c0afa20 0x7f185c12cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:36.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.147+0000 7f1863a58640 1 -- 172.21.15.57:0/3205495949 >> 172.21.15.57:0/3205495949 conn(0x7f185c069930 msgr2=0x7f185c0aef90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:36.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.147+0000 7f1863a58640 1 -- 172.21.15.57:0/3205495949 shutdown_connections 2023-10-19T14:01:36.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.147+0000 7f1863a58640 1 -- 172.21.15.57:0/3205495949 wait complete. 2023-10-19T14:01:36.159 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-19T14:01:36.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.267+0000 7f7259ff5640 1 -- 172.21.15.57:0/101237503 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f723c0047c0 msgr2=0x7f723c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:36.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.267+0000 7f7259ff5640 1 --2- 172.21.15.57:0/101237503 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f723c0047c0 0x7f723c004c40 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7f7248009b50 tx=0x7f723c0057f0 comp rx=0 tx=0).stop 2023-10-19T14:01:36.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.268+0000 7f7253fff640 1 -- 172.21.15.57:0/101237503 reap_dead start 2023-10-19T14:01:36.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.268+0000 7f7259ff5640 1 -- 172.21.15.57:0/101237503 shutdown_connections 2023-10-19T14:01:36.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.268+0000 7f7259ff5640 1 -- 172.21.15.57:0/101237503 >> 172.21.15.57:0/101237503 conn(0x7f72540af7a0 msgr2=0x7f72540afb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:36.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.268+0000 7f7259ff5640 1 -- 172.21.15.57:0/101237503 shutdown_connections 2023-10-19T14:01:36.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.268+0000 7f7259ff5640 1 -- 172.21.15.57:0/101237503 wait complete. 2023-10-19T14:01:36.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.269+0000 7f7259ff5640 1 Processor -- start 2023-10-19T14:01:36.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.269+0000 7f7259ff5640 1 -- start start 2023-10-19T14:01:36.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.270+0000 7f7259ff5640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f723c0047c0 0x7f725414d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:36.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.270+0000 7f7259ff5640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f72541e3fa0 0x7f72541e1650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:36.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.270+0000 7f7259ff5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7254154760 0x7f7254154be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:36.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.270+0000 7f7259ff5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72540c2a80 con 0x7f7254154760 2023-10-19T14:01:36.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.270+0000 7f7259ff5640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7254155120 con 0x7f723c0047c0 2023-10-19T14:01:36.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.270+0000 7f7259ff5640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72541e2740 con 0x7f72541e3fa0 2023-10-19T14:01:36.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.270+0000 7f7253fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7254154760 0x7f7254154be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:36.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.270+0000 7f7253fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7254154760 0x7f7254154be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:53948/0 (socket says 172.21.15.57:53948) 2023-10-19T14:01:36.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.270+0000 7f7253fff640 1 -- 172.21.15.57:0/2683638281 learned_addr learned my addr 172.21.15.57:0/2683638281 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:01:36.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.270+0000 7f7252ffd640 1 --2- 172.21.15.57:0/2683638281 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f72541e3fa0 0x7f72541e1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:36.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.270+0000 7f72537fe640 1 --2- 172.21.15.57:0/2683638281 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f723c0047c0 0x7f725414d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:36.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.270+0000 7f7253fff640 1 -- 172.21.15.57:0/2683638281 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f72541e3fa0 msgr2=0x7f72541e1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:36.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.270+0000 7f7253fff640 1 --2- 172.21.15.57:0/2683638281 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f72541e3fa0 0x7f72541e1650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:36.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.270+0000 7f7253fff640 1 -- 172.21.15.57:0/2683638281 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f723c0047c0 msgr2=0x7f725414d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:36.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.270+0000 7f7253fff640 1 --2- 172.21.15.57:0/2683638281 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f723c0047c0 0x7f725414d670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:36.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.270+0000 7f7253fff640 1 -- 172.21.15.57:0/2683638281 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f725414dbb0 con 0x7f7254154760 2023-10-19T14:01:36.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.270+0000 7f7253fff640 1 --2- 172.21.15.57:0/2683638281 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7254154760 0x7f7254154be0 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 crypto rx=0x7f725406bb30 tx=0x7f724c006720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:36.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.270+0000 7f723bfff640 1 -- 172.21.15.57:0/2683638281 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f724c00fc30 con 0x7f7254154760 2023-10-19T14:01:36.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.270+0000 7f723bfff640 1 -- 172.21.15.57:0/2683638281 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f724c00fdb0 con 0x7f7254154760 2023-10-19T14:01:36.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.270+0000 7f7259ff5640 1 -- 172.21.15.57:0/2683638281 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72541e1ea0 con 0x7f7254154760 2023-10-19T14:01:36.278 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.271+0000 7f723bfff640 1 -- 172.21.15.57:0/2683638281 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f724c011690 con 0x7f7254154760 2023-10-19T14:01:36.278 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.271+0000 7f7259ff5640 1 -- 172.21.15.57:0/2683638281 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72541e2460 con 0x7f7254154760 2023-10-19T14:01:36.278 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.272+0000 7f7259ff5640 1 -- 172.21.15.57:0/2683638281 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72540690e0 con 0x7f7254154760 2023-10-19T14:01:36.278 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.272+0000 7f723bfff640 1 -- 172.21.15.57:0/2683638281 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 91) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f724c0117f0 con 0x7f7254154760 2023-10-19T14:01:36.279 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.272+0000 7f723bfff640 1 --2- 172.21.15.57:0/2683638281 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f724006af60 0x7f724006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:36.279 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.273+0000 7f723bfff640 1 -- 172.21.15.57:0/2683638281 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f724c08a0f0 con 0x7f7254154760 2023-10-19T14:01:36.280 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.273+0000 7f72537fe640 1 --2- 172.21.15.57:0/2683638281 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f724006af60 0x7f724006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:36.280 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.273+0000 7f72537fe640 1 --2- 172.21.15.57:0/2683638281 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f724006af60 0x7f724006d420 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7244009930 tx=0x7f7244005f90 comp rx=0 tx=0).ready entity=mgr.8923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:36.280 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.277+0000 7f723bfff640 1 -- 172.21.15.57:0/2683638281 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f724c055170 con 0x7f7254154760 2023-10-19T14:01:36.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.452+0000 7f7259ff5640 1 -- 172.21.15.57:0/2683638281 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f72540ce5e0 con 0x7f7254154760 2023-10-19T14:01:36.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.453+0000 7f723bfff640 1 -- 172.21.15.57:0/2683638281 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v171) v1 ==== 84+0+65 (secure 0 0 0) 0x7f724c058790 con 0x7f7254154760 2023-10-19T14:01:36.456 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:01:36.456 INFO:teuthology.orchestra.run.smithi057.stdout:[ 2023-10-19T14:01:36.456 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:01:36.456 INFO:teuthology.orchestra.run.smithi057.stdout: "poolnum": 1, 2023-10-19T14:01:36.457 INFO:teuthology.orchestra.run.smithi057.stdout: "poolname": ".mgr" 2023-10-19T14:01:36.457 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:01:36.457 INFO:teuthology.orchestra.run.smithi057.stdout:] 2023-10-19T14:01:36.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.455+0000 7f7259ff5640 1 -- 172.21.15.57:0/2683638281 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f724006af60 msgr2=0x7f724006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:36.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.455+0000 7f7259ff5640 1 --2- 172.21.15.57:0/2683638281 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f724006af60 0x7f724006d420 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7244009930 tx=0x7f7244005f90 comp rx=0 tx=0).stop 2023-10-19T14:01:36.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.455+0000 7f7259ff5640 1 -- 172.21.15.57:0/2683638281 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7254154760 msgr2=0x7f7254154be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:36.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.455+0000 7f7259ff5640 1 --2- 172.21.15.57:0/2683638281 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7254154760 0x7f7254154be0 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 crypto rx=0x7f725406bb30 tx=0x7f724c006720 comp rx=0 tx=0).stop 2023-10-19T14:01:36.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.455+0000 7f7259ff5640 1 -- 172.21.15.57:0/2683638281 shutdown_connections 2023-10-19T14:01:36.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.455+0000 7f7259ff5640 1 --2- 172.21.15.57:0/2683638281 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f724006af60 0x7f724006d420 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:36.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.455+0000 7f7259ff5640 1 --2- 172.21.15.57:0/2683638281 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7254154760 0x7f7254154be0 unknown :-1 s=CLOSED pgs=1888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:36.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.455+0000 7f7259ff5640 1 --2- 172.21.15.57:0/2683638281 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f72541e3fa0 0x7f72541e1650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:36.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.455+0000 7f7259ff5640 1 --2- 172.21.15.57:0/2683638281 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f723c0047c0 0x7f725414d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:36.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.455+0000 7f7259ff5640 1 -- 172.21.15.57:0/2683638281 >> 172.21.15.57:0/2683638281 conn(0x7f72540af7a0 msgr2=0x7f72541599e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:36.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.455+0000 7f7259ff5640 1 -- 172.21.15.57:0/2683638281 shutdown_connections 2023-10-19T14:01:36.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.455+0000 7f7259ff5640 1 -- 172.21.15.57:0/2683638281 wait complete. 2023-10-19T14:01:36.468 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-19T14:01:36.469 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-19T14:01:36.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.580+0000 7ff6e099e640 1 -- 172.21.15.57:0/3121318404 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff6dc0cf280 msgr2=0x7ff6dc0b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:36.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.580+0000 7ff6e099e640 1 --2- 172.21.15.57:0/3121318404 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff6dc0cf280 0x7ff6dc0b04d0 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc08a000 tx=0x7ff6d402c260 comp rx=0 tx=0).stop 2023-10-19T14:01:36.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.580+0000 7ff6e099e640 1 -- 172.21.15.57:0/3121318404 shutdown_connections 2023-10-19T14:01:36.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.580+0000 7ff6e099e640 1 --2- 172.21.15.57:0/3121318404 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff6dc0cf280 0x7ff6dc0b04d0 unknown :-1 s=CLOSED pgs=1889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:36.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.580+0000 7ff6e099e640 1 --2- 172.21.15.57:0/3121318404 >> v2:172.21.15.179:3300/0 conn(0x7ff6dc15b0e0 0x7ff6dc0d17c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:36.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.580+0000 7ff6e099e640 1 -- 172.21.15.57:0/3121318404 >> 172.21.15.57:0/3121318404 conn(0x7ff6dc0d0ca0 msgr2=0x7ff6dc0d10b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:36.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.580+0000 7ff6e099e640 1 -- 172.21.15.57:0/3121318404 shutdown_connections 2023-10-19T14:01:36.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.580+0000 7ff6e099e640 1 -- 172.21.15.57:0/3121318404 wait complete. 2023-10-19T14:01:36.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.581+0000 7ff6e099e640 1 Processor -- start 2023-10-19T14:01:36.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.582+0000 7ff6e099e640 1 -- start start 2023-10-19T14:01:36.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.582+0000 7ff6e099e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff6dc0cf280 0x7ff6dc1536c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:36.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.582+0000 7ff6e099e640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff6dc15ad00 0x7ff6dc159700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:36.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.582+0000 7ff6e099e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff6dc15b0e0 0x7ff6dc159c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:36.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.582+0000 7ff6e099e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6dc0c70b0 con 0x7ff6dc15b0e0 2023-10-19T14:01:36.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.582+0000 7ff6e099e640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6dc151a80 con 0x7ff6dc0cf280 2023-10-19T14:01:36.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.582+0000 7ff6e099e640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6dc152a90 con 0x7ff6dc15ad00 2023-10-19T14:01:36.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.583+0000 7ff6dbfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff6dc15b0e0 0x7ff6dc159c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:36.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.583+0000 7ff6dbfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff6dc15b0e0 0x7ff6dc159c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:53958/0 (socket says 172.21.15.57:53958) 2023-10-19T14:01:36.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.583+0000 7ff6dbfff640 1 -- 172.21.15.57:0/607446494 learned_addr learned my addr 172.21.15.57:0/607446494 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:01:36.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.583+0000 7ff6db7fe640 1 --2- 172.21.15.57:0/607446494 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff6dc0cf280 0x7ff6dc1536c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:36.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.583+0000 7ff6daffd640 1 --2- 172.21.15.57:0/607446494 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff6dc15ad00 0x7ff6dc159700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:36.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.583+0000 7ff6dbfff640 1 -- 172.21.15.57:0/607446494 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff6dc15ad00 msgr2=0x7ff6dc159700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:36.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.583+0000 7ff6dbfff640 1 --2- 172.21.15.57:0/607446494 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff6dc15ad00 0x7ff6dc159700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:36.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.583+0000 7ff6dbfff640 1 -- 172.21.15.57:0/607446494 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff6dc0cf280 msgr2=0x7ff6dc1536c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:36.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.583+0000 7ff6dbfff640 1 --2- 172.21.15.57:0/607446494 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff6dc0cf280 0x7ff6dc1536c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:36.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.583+0000 7ff6dbfff640 1 -- 172.21.15.57:0/607446494 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6d4009900 con 0x7ff6dc15b0e0 2023-10-19T14:01:36.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.583+0000 7ff6dbfff640 1 --2- 172.21.15.57:0/607446494 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff6dc15b0e0 0x7ff6dc159c40 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7ff6d4009d60 tx=0x7ff6d4003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:36.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.583+0000 7ff6c3fff640 1 -- 172.21.15.57:0/607446494 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d4004450 con 0x7ff6dc15b0e0 2023-10-19T14:01:36.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.583+0000 7ff6c3fff640 1 -- 172.21.15.57:0/607446494 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7ff6d40046a0 con 0x7ff6dc15b0e0 2023-10-19T14:01:36.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.583+0000 7ff6e099e640 1 -- 172.21.15.57:0/607446494 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6dc151cd0 con 0x7ff6dc15b0e0 2023-10-19T14:01:36.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.583+0000 7ff6c3fff640 1 -- 172.21.15.57:0/607446494 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d40406b0 con 0x7ff6dc15b0e0 2023-10-19T14:01:36.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.583+0000 7ff6e099e640 1 -- 172.21.15.57:0/607446494 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6dc152290 con 0x7ff6dc15b0e0 2023-10-19T14:01:36.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.584+0000 7ff6e099e640 1 -- 172.21.15.57:0/607446494 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6dc066960 con 0x7ff6dc15b0e0 2023-10-19T14:01:36.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.585+0000 7ff6c3fff640 1 -- 172.21.15.57:0/607446494 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 91) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff6d400a040 con 0x7ff6dc15b0e0 2023-10-19T14:01:36.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.585+0000 7ff6c3fff640 1 --2- 172.21.15.57:0/607446494 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7ff6c806af60 0x7ff6c806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:36.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.585+0000 7ff6c3fff640 1 -- 172.21.15.57:0/607446494 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 5205+0+0 (secure 0 0 0) 0x7ff6d40ac870 con 0x7ff6dc15b0e0 2023-10-19T14:01:36.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.585+0000 7ff6db7fe640 1 --2- 172.21.15.57:0/607446494 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7ff6c806af60 0x7ff6c806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:36.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.586+0000 7ff6db7fe640 1 --2- 172.21.15.57:0/607446494 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7ff6c806af60 0x7ff6c806d420 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc004f60 tx=0x7ff6cc007e30 comp rx=0 tx=0).ready entity=mgr.8923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:36.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.589+0000 7ff6c3fff640 1 -- 172.21.15.57:0/607446494 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff6d4077aa0 con 0x7ff6dc15b0e0 2023-10-19T14:01:36.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:36.765+0000 7ff6e099e640 1 -- 172.21.15.57:0/607446494 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7ff6dc04da70 con 0x7ff6dc15b0e0 2023-10-19T14:01:37.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:37.129+0000 7ff6c3fff640 1 -- 172.21.15.57:0/607446494 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v172) v1 ==== 138+0+0 (secure 0 0 0) 0x7ff6d40390b0 con 0x7ff6dc15b0e0 2023-10-19T14:01:37.132 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'cephfs_metadata' created 2023-10-19T14:01:37.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:37.131+0000 7ff6e099e640 1 -- 172.21.15.57:0/607446494 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7ff6c806af60 msgr2=0x7ff6c806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:37.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:37.131+0000 7ff6e099e640 1 --2- 172.21.15.57:0/607446494 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7ff6c806af60 0x7ff6c806d420 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc004f60 tx=0x7ff6cc007e30 comp rx=0 tx=0).stop 2023-10-19T14:01:37.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:37.131+0000 7ff6e099e640 1 -- 172.21.15.57:0/607446494 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff6dc15b0e0 msgr2=0x7ff6dc159c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:37.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:37.131+0000 7ff6e099e640 1 --2- 172.21.15.57:0/607446494 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff6dc15b0e0 0x7ff6dc159c40 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7ff6d4009d60 tx=0x7ff6d4003c20 comp rx=0 tx=0).stop 2023-10-19T14:01:37.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:37.131+0000 7ff6e099e640 1 -- 172.21.15.57:0/607446494 shutdown_connections 2023-10-19T14:01:37.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:37.131+0000 7ff6e099e640 1 --2- 172.21.15.57:0/607446494 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7ff6c806af60 0x7ff6c806d420 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:37.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:37.131+0000 7ff6e099e640 1 --2- 172.21.15.57:0/607446494 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff6dc15b0e0 0x7ff6dc159c40 unknown :-1 s=CLOSED pgs=1890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:37.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:37.131+0000 7ff6e099e640 1 --2- 172.21.15.57:0/607446494 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff6dc15ad00 0x7ff6dc159700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:37.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:37.131+0000 7ff6e099e640 1 --2- 172.21.15.57:0/607446494 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff6dc0cf280 0x7ff6dc1536c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:37.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:37.131+0000 7ff6e099e640 1 -- 172.21.15.57:0/607446494 >> 172.21.15.57:0/607446494 conn(0x7ff6dc0d0ca0 msgr2=0x7ff6dc06a310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:37.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:37.131+0000 7ff6e099e640 1 -- 172.21.15.57:0/607446494 shutdown_connections 2023-10-19T14:01:37.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:37.131+0000 7ff6e099e640 1 -- 172.21.15.57:0/607446494 wait complete. 2023-10-19T14:01:37.148 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-19T14:01:37.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:37.263+0000 7f4a1dbe2640 1 -- 172.21.15.57:0/3894025863 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a180afa20 msgr2=0x7f4a180705a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:37.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:37.263+0000 7f4a1dbe2640 1 --2- 172.21.15.57:0/3894025863 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a180afa20 0x7f4a180705a0 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7f4a180cc7e0 tx=0x7f4a0c02bd90 comp rx=0 tx=0).stop 2023-10-19T14:01:37.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:37.263+0000 7f4a1dbe2640 1 -- 172.21.15.57:0/3894025863 shutdown_connections 2023-10-19T14:01:37.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:37.263+0000 7f4a1dbe2640 1 --2- 172.21.15.57:0/3894025863 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4a180b0530 0x7f4a180b09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:37.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:37.263+0000 7f4a1dbe2640 1 --2- 172.21.15.57:0/3894025863 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a180afa20 0x7f4a180705a0 unknown :-1 s=CLOSED pgs=1891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:37.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:37.263+0000 7f4a1dbe2640 1 --2- 172.21.15.57:0/3894025863 >> v2:172.21.15.179:3300/0 conn(0x7f4a180cbd20 0x7f4a18158300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:37.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:37.263+0000 7f4a1dbe2640 1 -- 172.21.15.57:0/3894025863 >> 172.21.15.57:0/3894025863 conn(0x7f4a18069930 msgr2=0x7f4a18069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:37.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:37.263+0000 7f4a1dbe2640 1 -- 172.21.15.57:0/3894025863 shutdown_connections 2023-10-19T14:01:37.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:37.263+0000 7f4a1dbe2640 1 -- 172.21.15.57:0/3894025863 wait complete. 2023-10-19T14:01:37.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:37.264+0000 7f4a1dbe2640 1 Processor -- start 2023-10-19T14:01:37.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:37.264+0000 7f4a1dbe2640 1 -- start start 2023-10-19T14:01:37.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:37.265+0000 7f4a1dbe2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a180afa20 0x7f4a18153cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:37.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:37.265+0000 7f4a1dbe2640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4a180b0530 0x7f4a18156a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:37.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:37.265+0000 7f4a1dbe2640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4a180cbd20 0x7f4a181e2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:37.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:37.265+0000 7f4a1dbe2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a18154350 con 0x7f4a180afa20 2023-10-19T14:01:37.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:37.265+0000 7f4a1dbe2640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a18154630 con 0x7f4a180b0530 2023-10-19T14:01:37.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:37.265+0000 7f4a1dbe2640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a18154910 con 0x7f4a180cbd20 2023-10-19T14:01:37.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:37.265+0000 7f4a177fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a180afa20 0x7f4a18153cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:37.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:37.265+0000 7f4a177fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a180afa20 0x7f4a18153cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35144/0 (socket says 172.21.15.57:35144) 2023-10-19T14:01:37.272 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:37.265+0000 7f4a177fe640 1 -- 172.21.15.57:0/1103436126 learned_addr learned my addr 172.21.15.57:0/1103436126 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:01:37.272 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:37.265+0000 7f4a16ffd640 1 --2- 172.21.15.57:0/1103436126 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4a180b0530 0x7f4a18156a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:37.272 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:37.265+0000 7f4a17fff640 1 --2- 172.21.15.57:0/1103436126 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4a180cbd20 0x7f4a181e2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:37.272 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:37.266+0000 7f4a177fe640 1 -- 172.21.15.57:0/1103436126 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4a180cbd20 msgr2=0x7f4a181e2200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:37.272 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:37.266+0000 7f4a177fe640 1 --2- 172.21.15.57:0/1103436126 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4a180cbd20 0x7f4a181e2200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:37.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:37.266+0000 7f4a177fe640 1 -- 172.21.15.57:0/1103436126 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4a180b0530 msgr2=0x7f4a18156a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:37.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:37.266+0000 7f4a177fe640 1 --2- 172.21.15.57:0/1103436126 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4a180b0530 0x7f4a18156a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:37.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:37.266+0000 7f4a177fe640 1 -- 172.21.15.57:0/1103436126 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a0c009560 con 0x7f4a180afa20 2023-10-19T14:01:37.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:37.266+0000 7f4a17fff640 1 --2- 172.21.15.57:0/1103436126 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4a180cbd20 0x7f4a181e2200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:01:37.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:37.266+0000 7f4a177fe640 1 --2- 172.21.15.57:0/1103436126 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a180afa20 0x7f4a18153cc0 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7f4a0800c970 tx=0x7f4a0800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:37.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:37.266+0000 7f49fffff640 1 -- 172.21.15.57:0/1103436126 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a08007bb0 con 0x7f4a180afa20 2023-10-19T14:01:37.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:37.266+0000 7f49fffff640 1 -- 172.21.15.57:0/1103436126 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f4a08007e00 con 0x7f4a180afa20 2023-10-19T14:01:37.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:37.266+0000 7f4a1dbe2640 1 -- 172.21.15.57:0/1103436126 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a18154bf0 con 0x7f4a180afa20 2023-10-19T14:01:37.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:37.266+0000 7f49fffff640 1 -- 172.21.15.57:0/1103436126 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a0800f6b0 con 0x7f4a180afa20 2023-10-19T14:01:37.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:37.266+0000 7f4a1dbe2640 1 -- 172.21.15.57:0/1103436126 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a181e2740 con 0x7f4a180afa20 2023-10-19T14:01:37.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:37.267+0000 7f4a1dbe2640 1 -- 172.21.15.57:0/1103436126 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a18155110 con 0x7f4a180afa20 2023-10-19T14:01:37.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:37.268+0000 7f49fffff640 1 -- 172.21.15.57:0/1103436126 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 91) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4a08018400 con 0x7f4a180afa20 2023-10-19T14:01:37.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:37.268+0000 7f49fffff640 1 --2- 172.21.15.57:0/1103436126 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f4a0406af60 0x7f4a0406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:37.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:37.268+0000 7f49fffff640 1 -- 172.21.15.57:0/1103436126 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5575+0+0 (secure 0 0 0) 0x7f4a0808a370 con 0x7f4a180afa20 2023-10-19T14:01:37.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:37.268+0000 7f4a16ffd640 1 --2- 172.21.15.57:0/1103436126 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f4a0406af60 0x7f4a0406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:37.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:37.269+0000 7f4a16ffd640 1 --2- 172.21.15.57:0/1103436126 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f4a0406af60 0x7f4a0406d420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4a0c035000 tx=0x7f4a0c004730 comp rx=0 tx=0).ready entity=mgr.8923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:37.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:37.272+0000 7f49fffff640 1 -- 172.21.15.57:0/1103436126 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4a080553f0 con 0x7f4a180afa20 2023-10-19T14:01:37.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:37.447+0000 7f4a1dbe2640 1 -- 172.21.15.57:0/1103436126 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f4a18155110 con 0x7f4a180afa20 2023-10-19T14:01:38.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:38.135+0000 7f49fffff640 1 -- 172.21.15.57:0/1103436126 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v173) v1 ==== 170+0+0 (secure 0 0 0) 0x7f4a18155110 con 0x7f4a180afa20 2023-10-19T14:01:38.137 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'cephfs_data' created 2023-10-19T14:01:38.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:38.138+0000 7f4a1dbe2640 1 -- 172.21.15.57:0/1103436126 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f4a0406af60 msgr2=0x7f4a0406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:38.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:38.138+0000 7f4a1dbe2640 1 --2- 172.21.15.57:0/1103436126 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f4a0406af60 0x7f4a0406d420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4a0c035000 tx=0x7f4a0c004730 comp rx=0 tx=0).stop 2023-10-19T14:01:38.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:38.138+0000 7f4a1dbe2640 1 -- 172.21.15.57:0/1103436126 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a180afa20 msgr2=0x7f4a18153cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:38.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:38.138+0000 7f4a1dbe2640 1 --2- 172.21.15.57:0/1103436126 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a180afa20 0x7f4a18153cc0 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7f4a0800c970 tx=0x7f4a0800ce30 comp rx=0 tx=0).stop 2023-10-19T14:01:38.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:38.139+0000 7f4a1dbe2640 1 -- 172.21.15.57:0/1103436126 shutdown_connections 2023-10-19T14:01:38.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:38.139+0000 7f4a1dbe2640 1 --2- 172.21.15.57:0/1103436126 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f4a0406af60 0x7f4a0406d420 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:38.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:38.139+0000 7f4a1dbe2640 1 --2- 172.21.15.57:0/1103436126 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4a180cbd20 0x7f4a181e2200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:38.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:38.139+0000 7f4a1dbe2640 1 --2- 172.21.15.57:0/1103436126 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4a180b0530 0x7f4a18156a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:38.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:38.139+0000 7f4a1dbe2640 1 --2- 172.21.15.57:0/1103436126 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a180afa20 0x7f4a18153cc0 unknown :-1 s=CLOSED pgs=1892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:38.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:38.139+0000 7f4a1dbe2640 1 -- 172.21.15.57:0/1103436126 >> 172.21.15.57:0/1103436126 conn(0x7f4a18069930 msgr2=0x7f4a180ad630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:38.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:38.143+0000 7f4a1dbe2640 1 -- 172.21.15.57:0/1103436126 shutdown_connections 2023-10-19T14:01:38.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:38.144+0000 7f4a1dbe2640 1 -- 172.21.15.57:0/1103436126 wait complete. 2023-10-19T14:01:38.167 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-19T14:01:38.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:38.280+0000 7fc6db10c640 1 -- 172.21.15.57:0/3743697595 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc6d406af00 msgr2=0x7fc6d406b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:38.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:38.280+0000 7fc6db10c640 1 --2- 172.21.15.57:0/3743697595 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc6d406af00 0x7fc6d406b2e0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc005fd0 tx=0x7fc6cc02c0e0 comp rx=0 tx=0).stop 2023-10-19T14:01:38.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:38.280+0000 7fc6db10c640 1 -- 172.21.15.57:0/3743697595 shutdown_connections 2023-10-19T14:01:38.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:38.280+0000 7fc6db10c640 1 --2- 172.21.15.57:0/3743697595 >> v2:172.21.15.179:3300/0 conn(0x7fc6d40c96b0 0x7fc6d40c9af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:38.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:38.280+0000 7fc6db10c640 1 --2- 172.21.15.57:0/3743697595 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc6d406af00 0x7fc6d406b2e0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:38.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:38.280+0000 7fc6db10c640 1 -- 172.21.15.57:0/3743697595 >> 172.21.15.57:0/3743697595 conn(0x7fc6d4069930 msgr2=0x7fc6d4069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:38.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:38.280+0000 7fc6db10c640 1 -- 172.21.15.57:0/3743697595 shutdown_connections 2023-10-19T14:01:38.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:38.280+0000 7fc6db10c640 1 -- 172.21.15.57:0/3743697595 wait complete. 2023-10-19T14:01:38.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:38.281+0000 7fc6db10c640 1 Processor -- start 2023-10-19T14:01:38.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:38.281+0000 7fc6db10c640 1 -- start start 2023-10-19T14:01:38.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:38.282+0000 7fc6db10c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc6d406af00 0x7fc6d4157b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:38.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:38.282+0000 7fc6db10c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc6d40ab500 0x7fc6d4151b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:38.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:38.282+0000 7fc6db10c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc6d40c96b0 0x7fc6d41538b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:38.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:38.282+0000 7fc6db10c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d4154710 con 0x7fc6d406af00 2023-10-19T14:01:38.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:38.282+0000 7fc6db10c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d4158330 con 0x7fc6d40ab500 2023-10-19T14:01:38.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:38.282+0000 7fc6db10c640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d4158610 con 0x7fc6d40c96b0 2023-10-19T14:01:38.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:38.282+0000 7fc6d8ea8640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc6d406af00 0x7fc6d4157b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:38.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:38.282+0000 7fc6d8ea8640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc6d406af00 0x7fc6d4157b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35148/0 (socket says 172.21.15.57:35148) 2023-10-19T14:01:38.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:38.282+0000 7fc6d8ea8640 1 -- 172.21.15.57:0/4208070173 learned_addr learned my addr 172.21.15.57:0/4208070173 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:01:38.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:38.282+0000 7fc6d96a9640 1 --2- 172.21.15.57:0/4208070173 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc6d40c96b0 0x7fc6d41538b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:38.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:38.283+0000 7fc6d3fff640 1 --2- 172.21.15.57:0/4208070173 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc6d40ab500 0x7fc6d4151b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:38.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:38.283+0000 7fc6d8ea8640 1 -- 172.21.15.57:0/4208070173 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc6d40c96b0 msgr2=0x7fc6d41538b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:38.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:38.283+0000 7fc6d8ea8640 1 --2- 172.21.15.57:0/4208070173 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc6d40c96b0 0x7fc6d41538b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:38.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:38.283+0000 7fc6d8ea8640 1 -- 172.21.15.57:0/4208070173 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc6d40ab500 msgr2=0x7fc6d4151b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:38.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:38.283+0000 7fc6d8ea8640 1 --2- 172.21.15.57:0/4208070173 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc6d40ab500 0x7fc6d4151b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:38.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:38.283+0000 7fc6d8ea8640 1 -- 172.21.15.57:0/4208070173 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6c8009490 con 0x7fc6d406af00 2023-10-19T14:01:38.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:38.283+0000 7fc6d3fff640 1 --2- 172.21.15.57:0/4208070173 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc6d40ab500 0x7fc6d4151b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T14:01:38.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:38.283+0000 7fc6d96a9640 1 --2- 172.21.15.57:0/4208070173 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc6d40c96b0 0x7fc6d41538b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T14:01:38.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:38.283+0000 7fc6d8ea8640 1 --2- 172.21.15.57:0/4208070173 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc6d406af00 0x7fc6d4157b00 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc006010 tx=0x7fc6cc0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:38.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:38.283+0000 7fc6d17fa640 1 -- 172.21.15.57:0/4208070173 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6cc0046f0 con 0x7fc6d406af00 2023-10-19T14:01:38.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:38.283+0000 7fc6d17fa640 1 -- 172.21.15.57:0/4208070173 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fc6cc0345a0 con 0x7fc6d406af00 2023-10-19T14:01:38.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:38.283+0000 7fc6db10c640 1 -- 172.21.15.57:0/4208070173 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6cc009840 con 0x7fc6d406af00 2023-10-19T14:01:38.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:38.283+0000 7fc6d17fa640 1 -- 172.21.15.57:0/4208070173 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6cc0349b0 con 0x7fc6d406af00 2023-10-19T14:01:38.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:38.283+0000 7fc6db10c640 1 -- 172.21.15.57:0/4208070173 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6d4159cc0 con 0x7fc6d406af00 2023-10-19T14:01:38.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:38.284+0000 7fc6db10c640 1 -- 172.21.15.57:0/4208070173 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6d406c500 con 0x7fc6d406af00 2023-10-19T14:01:38.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:38.285+0000 7fc6d17fa640 1 -- 172.21.15.57:0/4208070173 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 91) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc6cc034b10 con 0x7fc6d406af00 2023-10-19T14:01:38.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:38.285+0000 7fc6d17fa640 1 --2- 172.21.15.57:0/4208070173 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7fc6c006af90 0x7fc6c006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:38.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:38.285+0000 7fc6d17fa640 1 -- 172.21.15.57:0/4208070173 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(173..173 src has 1..173) v4 ==== 5957+0+0 (secure 0 0 0) 0x7fc6cc0ad1f0 con 0x7fc6d406af00 2023-10-19T14:01:38.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:38.286+0000 7fc6d3fff640 1 --2- 172.21.15.57:0/4208070173 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7fc6c006af90 0x7fc6c006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:38.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:38.286+0000 7fc6d3fff640 1 --2- 172.21.15.57:0/4208070173 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7fc6c006af90 0x7fc6c006d450 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc6d41f2c30 tx=0x7fc6c8005f90 comp rx=0 tx=0).ready entity=mgr.8923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:38.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:38.289+0000 7fc6d17fa640 1 -- 172.21.15.57:0/4208070173 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc6cc078220 con 0x7fc6d406af00 2023-10-19T14:01:38.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:38.501+0000 7fc6db10c640 1 -- 172.21.15.57:0/4208070173 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fc6d415d2a0 con 0x7fc6d406af00 2023-10-19T14:01:39.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.139+0000 7fc6d17fa640 1 -- 172.21.15.57:0/4208070173 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 21 and data pool 22 v68) v1 ==== 174+0+0 (secure 0 0 0) 0x7fc6cc0430c0 con 0x7fc6d406af00 2023-10-19T14:01:39.142 INFO:teuthology.orchestra.run.smithi057.stderr:new fs with metadata pool 21 and data pool 22 2023-10-19T14:01:39.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.141+0000 7fc6db10c640 1 -- 172.21.15.57:0/4208070173 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7fc6c006af90 msgr2=0x7fc6c006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:39.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.141+0000 7fc6db10c640 1 --2- 172.21.15.57:0/4208070173 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7fc6c006af90 0x7fc6c006d450 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc6d41f2c30 tx=0x7fc6c8005f90 comp rx=0 tx=0).stop 2023-10-19T14:01:39.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.141+0000 7fc6db10c640 1 -- 172.21.15.57:0/4208070173 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc6d406af00 msgr2=0x7fc6d4157b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:39.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.141+0000 7fc6db10c640 1 --2- 172.21.15.57:0/4208070173 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc6d406af00 0x7fc6d4157b00 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc006010 tx=0x7fc6cc0042f0 comp rx=0 tx=0).stop 2023-10-19T14:01:39.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.141+0000 7fc6db10c640 1 -- 172.21.15.57:0/4208070173 shutdown_connections 2023-10-19T14:01:39.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.141+0000 7fc6db10c640 1 --2- 172.21.15.57:0/4208070173 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7fc6c006af90 0x7fc6c006d450 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:39.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.141+0000 7fc6db10c640 1 --2- 172.21.15.57:0/4208070173 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc6d40c96b0 0x7fc6d41538b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:39.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.141+0000 7fc6db10c640 1 --2- 172.21.15.57:0/4208070173 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc6d40ab500 0x7fc6d4151b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:39.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.141+0000 7fc6db10c640 1 --2- 172.21.15.57:0/4208070173 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc6d406af00 0x7fc6d4157b00 unknown :-1 s=CLOSED pgs=1893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:39.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.141+0000 7fc6db10c640 1 -- 172.21.15.57:0/4208070173 >> 172.21.15.57:0/4208070173 conn(0x7fc6d4069930 msgr2=0x7fc6d40b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:39.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.141+0000 7fc6db10c640 1 -- 172.21.15.57:0/4208070173 shutdown_connections 2023-10-19T14:01:39.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.141+0000 7fc6db10c640 1 -- 172.21.15.57:0/4208070173 wait complete. 2023-10-19T14:01:39.160 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T14:01:39.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.270+0000 7fa15d1ce640 1 -- 172.21.15.57:0/3497321728 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa1580cf340 msgr2=0x7fa1580d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:39.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.270+0000 7fa15d1ce640 1 --2- 172.21.15.57:0/3497321728 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa1580cf340 0x7fa1580d1910 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7fa15808a120 tx=0x7fa15002bdb0 comp rx=0 tx=0).stop 2023-10-19T14:01:39.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.270+0000 7fa15d1ce640 1 -- 172.21.15.57:0/3497321728 shutdown_connections 2023-10-19T14:01:39.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.270+0000 7fa15d1ce640 1 --2- 172.21.15.57:0/3497321728 >> v2:172.21.15.179:3300/0 conn(0x7fa15815b2a0 0x7fa1580cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:39.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.270+0000 7fa15d1ce640 1 --2- 172.21.15.57:0/3497321728 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa1580cf340 0x7fa1580d1910 unknown :-1 s=CLOSED pgs=1895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:39.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.270+0000 7fa15d1ce640 1 -- 172.21.15.57:0/3497321728 >> 172.21.15.57:0/3497321728 conn(0x7fa1580d0d60 msgr2=0x7fa1580d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:39.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.270+0000 7fa15d1ce640 1 -- 172.21.15.57:0/3497321728 shutdown_connections 2023-10-19T14:01:39.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.270+0000 7fa15d1ce640 1 -- 172.21.15.57:0/3497321728 wait complete. 2023-10-19T14:01:39.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.272+0000 7fa15d1ce640 1 Processor -- start 2023-10-19T14:01:39.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.272+0000 7fa15d1ce640 1 -- start start 2023-10-19T14:01:39.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.272+0000 7fa15d1ce640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa1580ceb40 0x7fa1580b4a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:39.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.272+0000 7fa15d1ce640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa1580cf340 0x7fa15815b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:39.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.272+0000 7fa15d1ce640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa15815b2a0 0x7fa15806b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:39.278 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.272+0000 7fa15d1ce640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa15815dc60 con 0x7fa1580cf340 2023-10-19T14:01:39.278 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.272+0000 7fa15d1ce640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1581f1280 con 0x7fa1580ceb40 2023-10-19T14:01:39.278 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.272+0000 7fa15d1ce640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1581f2080 con 0x7fa15815b2a0 2023-10-19T14:01:39.278 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.273+0000 7fa15659c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa1580cf340 0x7fa15815b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:39.278 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.273+0000 7fa15659c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa1580cf340 0x7fa15815b6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35168/0 (socket says 172.21.15.57:35168) 2023-10-19T14:01:39.279 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.273+0000 7fa15659c640 1 -- 172.21.15.57:0/3851451405 learned_addr learned my addr 172.21.15.57:0/3851451405 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:01:39.279 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.273+0000 7fa156d9d640 1 --2- 172.21.15.57:0/3851451405 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa1580ceb40 0x7fa1580b4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:39.279 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.273+0000 7fa15759e640 1 --2- 172.21.15.57:0/3851451405 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa15815b2a0 0x7fa15806b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:39.279 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.273+0000 7fa15659c640 1 -- 172.21.15.57:0/3851451405 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa15815b2a0 msgr2=0x7fa15806b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:39.279 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.273+0000 7fa15659c640 1 --2- 172.21.15.57:0/3851451405 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa15815b2a0 0x7fa15806b330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:39.280 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.273+0000 7fa15659c640 1 -- 172.21.15.57:0/3851451405 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa1580ceb40 msgr2=0x7fa1580b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:39.280 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.273+0000 7fa15659c640 1 --2- 172.21.15.57:0/3851451405 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa1580ceb40 0x7fa1580b4a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:39.280 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.273+0000 7fa15659c640 1 -- 172.21.15.57:0/3851451405 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa150009560 con 0x7fa1580cf340 2023-10-19T14:01:39.280 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.273+0000 7fa15659c640 1 --2- 172.21.15.57:0/3851451405 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa1580cf340 0x7fa15815b6b0 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7fa14000d740 tx=0x7fa14000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:39.280 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.273+0000 7fa14f7fe640 1 -- 172.21.15.57:0/3851451405 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa140009c30 con 0x7fa1580cf340 2023-10-19T14:01:39.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.273+0000 7fa14f7fe640 1 -- 172.21.15.57:0/3851451405 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fa140009e80 con 0x7fa1580cf340 2023-10-19T14:01:39.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.273+0000 7fa15d1ce640 1 -- 172.21.15.57:0/3851451405 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa158151ab0 con 0x7fa1580cf340 2023-10-19T14:01:39.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.273+0000 7fa14f7fe640 1 -- 172.21.15.57:0/3851451405 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1400106c0 con 0x7fa1580cf340 2023-10-19T14:01:39.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.273+0000 7fa15d1ce640 1 -- 172.21.15.57:0/3851451405 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1581f1750 con 0x7fa1580cf340 2023-10-19T14:01:39.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.274+0000 7fa15d1ce640 1 -- 172.21.15.57:0/3851451405 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1580690e0 con 0x7fa1580cf340 2023-10-19T14:01:39.282 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.275+0000 7fa14f7fe640 1 -- 172.21.15.57:0/3851451405 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 91) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fa140010820 con 0x7fa1580cf340 2023-10-19T14:01:39.282 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.275+0000 7fa14f7fe640 1 --2- 172.21.15.57:0/3851451405 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7fa13806af90 0x7fa13806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:39.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.275+0000 7fa14f7fe640 1 -- 172.21.15.57:0/3851451405 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fa140089ee0 con 0x7fa1580cf340 2023-10-19T14:01:39.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.276+0000 7fa156d9d640 1 --2- 172.21.15.57:0/3851451405 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7fa13806af90 0x7fa13806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:39.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.276+0000 7fa156d9d640 1 --2- 172.21.15.57:0/3851451405 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7fa13806af90 0x7fa13806d450 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa150035000 tx=0x7fa150004730 comp rx=0 tx=0).ready entity=mgr.8923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:39.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.280+0000 7fa14f7fe640 1 -- 172.21.15.57:0/3851451405 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa140054c20 con 0x7fa1580cf340 2023-10-19T14:01:39.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.455+0000 7fa15d1ce640 1 -- 172.21.15.57:0/3851451405 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa1580cb9d0 con 0x7fa1580cf340 2023-10-19T14:01:39.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.456+0000 7fa14f7fe640 1 -- 172.21.15.57:0/3851451405 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v174) v1 ==== 74+0+16792 (secure 0 0 0) 0x7fa140021090 con 0x7fa1580cf340 2023-10-19T14:01:39.459 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:01:39.460 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":174,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T14:01:38.502706+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2023-10-19T14:01:36.799358+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"174","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_data","create_time":"2023-10-19T14:01:37.481213+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"174","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T14:01:39.461 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.458+0000 7fa15d1ce640 1 -- 172.21.15.57:0/3851451405 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7fa13806af90 msgr2=0x7fa13806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:39.461 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.458+0000 7fa15d1ce640 1 --2- 172.21.15.57:0/3851451405 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7fa13806af90 0x7fa13806d450 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa150035000 tx=0x7fa150004730 comp rx=0 tx=0).stop 2023-10-19T14:01:39.461 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.458+0000 7fa15d1ce640 1 -- 172.21.15.57:0/3851451405 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa1580cf340 msgr2=0x7fa15815b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:39.461 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.458+0000 7fa15d1ce640 1 --2- 172.21.15.57:0/3851451405 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa1580cf340 0x7fa15815b6b0 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7fa14000d740 tx=0x7fa14000dc00 comp rx=0 tx=0).stop 2023-10-19T14:01:39.462 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.458+0000 7fa15d1ce640 1 -- 172.21.15.57:0/3851451405 shutdown_connections 2023-10-19T14:01:39.462 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.458+0000 7fa15d1ce640 1 --2- 172.21.15.57:0/3851451405 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7fa13806af90 0x7fa13806d450 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:39.462 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.458+0000 7fa15d1ce640 1 --2- 172.21.15.57:0/3851451405 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa15815b2a0 0x7fa15806b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:39.462 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.458+0000 7fa15d1ce640 1 --2- 172.21.15.57:0/3851451405 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa1580cf340 0x7fa15815b6b0 unknown :-1 s=CLOSED pgs=1896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:39.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.458+0000 7fa15d1ce640 1 --2- 172.21.15.57:0/3851451405 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa1580ceb40 0x7fa1580b4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:39.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.458+0000 7fa15d1ce640 1 -- 172.21.15.57:0/3851451405 >> 172.21.15.57:0/3851451405 conn(0x7fa1580d0d60 msgr2=0x7fa1580b1370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:39.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.458+0000 7fa15d1ce640 1 -- 172.21.15.57:0/3851451405 shutdown_connections 2023-10-19T14:01:39.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.458+0000 7fa15d1ce640 1 -- 172.21.15.57:0/3851451405 wait complete. 2023-10-19T14:01:39.472 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T14:01:39.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.579+0000 7f0dd5d8a640 1 -- 172.21.15.57:0/472011172 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0dd00cbd20 msgr2=0x7f0dd0158300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:39.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.579+0000 7f0dd5d8a640 1 --2- 172.21.15.57:0/472011172 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0dd00cbd20 0x7f0dd0158300 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7f0dc0009830 tx=0x7f0dc002bd60 comp rx=0 tx=0).stop 2023-10-19T14:01:39.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.579+0000 7f0dd5d8a640 1 -- 172.21.15.57:0/472011172 shutdown_connections 2023-10-19T14:01:39.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.579+0000 7f0dd5d8a640 1 --2- 172.21.15.57:0/472011172 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0dd00afa20 0x7f0dd00705a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:39.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.579+0000 7f0dd5d8a640 1 --2- 172.21.15.57:0/472011172 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0dd00cbd20 0x7f0dd0158300 unknown :-1 s=CLOSED pgs=1898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:39.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.579+0000 7f0dd5d8a640 1 -- 172.21.15.57:0/472011172 >> 172.21.15.57:0/472011172 conn(0x7f0dd0069930 msgr2=0x7f0dd0069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:39.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.579+0000 7f0dd5d8a640 1 -- 172.21.15.57:0/472011172 shutdown_connections 2023-10-19T14:01:39.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.579+0000 7f0dd5d8a640 1 -- 172.21.15.57:0/472011172 wait complete. 2023-10-19T14:01:39.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.580+0000 7f0dd5d8a640 1 Processor -- start 2023-10-19T14:01:39.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.580+0000 7f0dd5d8a640 1 -- start start 2023-10-19T14:01:39.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.581+0000 7f0dd5d8a640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0dd00afa20 0x7f0dd01481d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:39.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.581+0000 7f0dd5d8a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0dd00b0530 0x7f0dd0145f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:39.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.581+0000 7f0dd5d8a640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0dd00cbd20 0x7f0dd0143de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:39.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.581+0000 7f0dd5d8a640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd0147130 con 0x7f0dd00b0530 2023-10-19T14:01:39.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.581+0000 7f0dd5d8a640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd0147410 con 0x7f0dd00cbd20 2023-10-19T14:01:39.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.581+0000 7f0dd5d8a640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd01476f0 con 0x7f0dd00afa20 2023-10-19T14:01:39.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.581+0000 7f0dceffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0dd00b0530 0x7f0dd0145f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:39.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.581+0000 7f0dceffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0dd00b0530 0x7f0dd0145f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35196/0 (socket says 172.21.15.57:35196) 2023-10-19T14:01:39.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.581+0000 7f0dceffd640 1 -- 172.21.15.57:0/1880865097 learned_addr learned my addr 172.21.15.57:0/1880865097 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:01:39.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.581+0000 7f0dcf7fe640 1 --2- 172.21.15.57:0/1880865097 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0dd00afa20 0x7f0dd01481d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:39.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.581+0000 7f0dcffff640 1 --2- 172.21.15.57:0/1880865097 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0dd00cbd20 0x7f0dd0143de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:39.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.581+0000 7f0dceffd640 1 -- 172.21.15.57:0/1880865097 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0dd00afa20 msgr2=0x7f0dd01481d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:39.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.581+0000 7f0dceffd640 1 --2- 172.21.15.57:0/1880865097 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0dd00afa20 0x7f0dd01481d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:39.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.581+0000 7f0dceffd640 1 -- 172.21.15.57:0/1880865097 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0dd00cbd20 msgr2=0x7f0dd0143de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:39.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.581+0000 7f0dceffd640 1 --2- 172.21.15.57:0/1880865097 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0dd00cbd20 0x7f0dd0143de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:39.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.581+0000 7f0dceffd640 1 -- 172.21.15.57:0/1880865097 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dc0009490 con 0x7f0dd00b0530 2023-10-19T14:01:39.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.581+0000 7f0dceffd640 1 --2- 172.21.15.57:0/1880865097 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0dd00b0530 0x7f0dd0145f30 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7f0dc400c970 tx=0x7f0dc400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:39.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.581+0000 7f0db7fff640 1 -- 172.21.15.57:0/1880865097 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dc4007bb0 con 0x7f0dd00b0530 2023-10-19T14:01:39.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.581+0000 7f0db7fff640 1 -- 172.21.15.57:0/1880865097 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f0dc4007e20 con 0x7f0dd00b0530 2023-10-19T14:01:39.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.582+0000 7f0db7fff640 1 -- 172.21.15.57:0/1880865097 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dc4005280 con 0x7f0dd00b0530 2023-10-19T14:01:39.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.582+0000 7f0dd5d8a640 1 -- 172.21.15.57:0/1880865097 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dd0070cf0 con 0x7f0dd00b0530 2023-10-19T14:01:39.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.582+0000 7f0dd5d8a640 1 -- 172.21.15.57:0/1880865097 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dd0147cb0 con 0x7f0dd00b0530 2023-10-19T14:01:39.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.583+0000 7f0dd5d8a640 1 -- 172.21.15.57:0/1880865097 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dd0147ed0 con 0x7f0dd00b0530 2023-10-19T14:01:39.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.583+0000 7f0db7fff640 1 -- 172.21.15.57:0/1880865097 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 91) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f0dc40053e0 con 0x7f0dd00b0530 2023-10-19T14:01:39.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.583+0000 7f0db7fff640 1 --2- 172.21.15.57:0/1880865097 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f0dbc06af60 0x7f0dbc06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:39.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.583+0000 7f0db7fff640 1 -- 172.21.15.57:0/1880865097 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f0dc4089ee0 con 0x7f0dd00b0530 2023-10-19T14:01:39.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.584+0000 7f0dcf7fe640 1 --2- 172.21.15.57:0/1880865097 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f0dbc06af60 0x7f0dbc06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:39.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.584+0000 7f0dcf7fe640 1 --2- 172.21.15.57:0/1880865097 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f0dbc06af60 0x7f0dbc06d420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0dc0036000 tx=0x7f0dc0004730 comp rx=0 tx=0).ready entity=mgr.8923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:39.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.588+0000 7f0db7fff640 1 -- 172.21.15.57:0/1880865097 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0dc4054c40 con 0x7f0dd00b0530 2023-10-19T14:01:39.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.764+0000 7f0dd5d8a640 1 -- 172.21.15.57:0/1880865097 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0dd00690e0 con 0x7f0dd00b0530 2023-10-19T14:01:39.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.766+0000 7f0db7fff640 1 -- 172.21.15.57:0/1880865097 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v174) v1 ==== 74+0+16792 (secure 0 0 0) 0x7f0dc4058260 con 0x7f0dd00b0530 2023-10-19T14:01:39.769 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:01:39.769 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":174,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T14:01:38.502706+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2023-10-19T14:01:36.799358+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"174","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_data","create_time":"2023-10-19T14:01:37.481213+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"174","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T14:01:39.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.768+0000 7f0dd5d8a640 1 -- 172.21.15.57:0/1880865097 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f0dbc06af60 msgr2=0x7f0dbc06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:39.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.768+0000 7f0dd5d8a640 1 --2- 172.21.15.57:0/1880865097 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f0dbc06af60 0x7f0dbc06d420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0dc0036000 tx=0x7f0dc0004730 comp rx=0 tx=0).stop 2023-10-19T14:01:39.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.768+0000 7f0dd5d8a640 1 -- 172.21.15.57:0/1880865097 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0dd00b0530 msgr2=0x7f0dd0145f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:39.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.768+0000 7f0dd5d8a640 1 --2- 172.21.15.57:0/1880865097 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0dd00b0530 0x7f0dd0145f30 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7f0dc400c970 tx=0x7f0dc400ce30 comp rx=0 tx=0).stop 2023-10-19T14:01:39.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.768+0000 7f0dd5d8a640 1 -- 172.21.15.57:0/1880865097 shutdown_connections 2023-10-19T14:01:39.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.768+0000 7f0dd5d8a640 1 --2- 172.21.15.57:0/1880865097 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f0dbc06af60 0x7f0dbc06d420 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:39.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.768+0000 7f0dd5d8a640 1 --2- 172.21.15.57:0/1880865097 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0dd00cbd20 0x7f0dd0143de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:39.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.768+0000 7f0dd5d8a640 1 --2- 172.21.15.57:0/1880865097 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0dd00b0530 0x7f0dd0145f30 unknown :-1 s=CLOSED pgs=1899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:39.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.768+0000 7f0dd5d8a640 1 --2- 172.21.15.57:0/1880865097 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0dd00afa20 0x7f0dd01481d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:39.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.768+0000 7f0dd5d8a640 1 -- 172.21.15.57:0/1880865097 >> 172.21.15.57:0/1880865097 conn(0x7f0dd0069930 msgr2=0x7f0dd00c9730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:39.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.768+0000 7f0dd5d8a640 1 -- 172.21.15.57:0/1880865097 shutdown_connections 2023-10-19T14:01:39.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.768+0000 7f0dd5d8a640 1 -- 172.21.15.57:0/1880865097 wait complete. 2023-10-19T14:01:39.782 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-19T14:01:39.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.891+0000 7f9d463a5640 1 -- 172.21.15.57:0/220894055 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9d400cf340 msgr2=0x7f9d400d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:39.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.891+0000 7f9d463a5640 1 --2- 172.21.15.57:0/220894055 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9d400cf340 0x7f9d400d1910 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x7f9d30005fd0 tx=0x7f9d3002bd90 comp rx=0 tx=0).stop 2023-10-19T14:01:39.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.891+0000 7f9d463a5640 1 -- 172.21.15.57:0/220894055 shutdown_connections 2023-10-19T14:01:39.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.891+0000 7f9d463a5640 1 --2- 172.21.15.57:0/220894055 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9d400cf340 0x7f9d400d1910 unknown :-1 s=CLOSED pgs=1901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:39.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.891+0000 7f9d463a5640 1 -- 172.21.15.57:0/220894055 >> 172.21.15.57:0/220894055 conn(0x7f9d400d0d60 msgr2=0x7f9d400d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:39.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.891+0000 7f9d463a5640 1 -- 172.21.15.57:0/220894055 shutdown_connections 2023-10-19T14:01:39.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.891+0000 7f9d463a5640 1 -- 172.21.15.57:0/220894055 wait complete. 2023-10-19T14:01:39.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.892+0000 7f9d463a5640 1 Processor -- start 2023-10-19T14:01:39.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.893+0000 7f9d463a5640 1 -- start start 2023-10-19T14:01:39.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.893+0000 7f9d463a5640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9d400ceb40 0x7f9d40158990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:39.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.893+0000 7f9d463a5640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9d400cf340 0x7f9d400c9e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:39.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.893+0000 7f9d463a5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9d4015b2a0 0x7f9d400ca380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:39.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.893+0000 7f9d463a5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d400b38e0 con 0x7f9d4015b2a0 2023-10-19T14:01:39.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.893+0000 7f9d463a5640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d401f1550 con 0x7f9d400cf340 2023-10-19T14:01:39.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.893+0000 7f9d463a5640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d401f1810 con 0x7f9d400ceb40 2023-10-19T14:01:39.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.893+0000 7f9d44942640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9d4015b2a0 0x7f9d400ca380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:39.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.893+0000 7f9d44942640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9d4015b2a0 0x7f9d400ca380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35206/0 (socket says 172.21.15.57:35206) 2023-10-19T14:01:39.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.893+0000 7f9d44942640 1 -- 172.21.15.57:0/92748034 learned_addr learned my addr 172.21.15.57:0/92748034 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:01:39.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.894+0000 7f9d3ffff640 1 --2- 172.21.15.57:0/92748034 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9d400ceb40 0x7f9d40158990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:39.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.894+0000 7f9d3f7fe640 1 --2- 172.21.15.57:0/92748034 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9d400cf340 0x7f9d400c9e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:39.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.894+0000 7f9d44942640 1 -- 172.21.15.57:0/92748034 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9d400ceb40 msgr2=0x7f9d40158990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:39.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.894+0000 7f9d44942640 1 --2- 172.21.15.57:0/92748034 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9d400ceb40 0x7f9d40158990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:39.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.894+0000 7f9d44942640 1 -- 172.21.15.57:0/92748034 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9d400cf340 msgr2=0x7f9d400c9e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:39.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.894+0000 7f9d44942640 1 --2- 172.21.15.57:0/92748034 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9d400cf340 0x7f9d400c9e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:39.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.894+0000 7f9d44942640 1 -- 172.21.15.57:0/92748034 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d30009560 con 0x7f9d4015b2a0 2023-10-19T14:01:39.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.894+0000 7f9d44942640 1 --2- 172.21.15.57:0/92748034 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9d4015b2a0 0x7f9d400ca380 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7f9d3800c7a0 tx=0x7f9d3800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:39.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.894+0000 7f9d3cff9640 1 -- 172.21.15.57:0/92748034 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d38003b70 con 0x7f9d4015b2a0 2023-10-19T14:01:39.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.894+0000 7f9d3cff9640 1 -- 172.21.15.57:0/92748034 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f9d38003cf0 con 0x7f9d4015b2a0 2023-10-19T14:01:39.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.894+0000 7f9d463a5640 1 -- 172.21.15.57:0/92748034 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d401f1af0 con 0x7f9d4015b2a0 2023-10-19T14:01:39.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.894+0000 7f9d3cff9640 1 -- 172.21.15.57:0/92748034 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d3800aab0 con 0x7f9d4015b2a0 2023-10-19T14:01:39.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.894+0000 7f9d463a5640 1 -- 172.21.15.57:0/92748034 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d401f20b0 con 0x7f9d4015b2a0 2023-10-19T14:01:39.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.895+0000 7f9d463a5640 1 -- 172.21.15.57:0/92748034 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d400690e0 con 0x7f9d4015b2a0 2023-10-19T14:01:39.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.896+0000 7f9d3cff9640 1 -- 172.21.15.57:0/92748034 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 91) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f9d3800ac10 con 0x7f9d4015b2a0 2023-10-19T14:01:39.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.896+0000 7f9d3cff9640 1 --2- 172.21.15.57:0/92748034 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f9d2c06af60 0x7f9d2c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:39.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.896+0000 7f9d3cff9640 1 -- 172.21.15.57:0/92748034 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f9d38089790 con 0x7f9d4015b2a0 2023-10-19T14:01:39.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.896+0000 7f9d3ffff640 1 --2- 172.21.15.57:0/92748034 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f9d2c06af60 0x7f9d2c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:39.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.897+0000 7f9d3ffff640 1 --2- 172.21.15.57:0/92748034 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f9d2c06af60 0x7f9d2c06d420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9d30035000 tx=0x7f9d30004730 comp rx=0 tx=0).ready entity=mgr.8923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:39.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:39.900+0000 7f9d3cff9640 1 -- 172.21.15.57:0/92748034 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9d38054680 con 0x7f9d4015b2a0 2023-10-19T14:01:40.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:40.114+0000 7f9d463a5640 1 -- 172.21.15.57:0/92748034 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f9d400d0b40 con 0x7f9d4015b2a0 2023-10-19T14:01:40.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:40.145+0000 7f9d3cff9640 1 -- 172.21.15.57:0/92748034 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v69) v1 ==== 118+0+0 (secure 0 0 0) 0x7f9d38057ca0 con 0x7f9d4015b2a0 2023-10-19T14:01:40.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:40.148+0000 7f9d463a5640 1 -- 172.21.15.57:0/92748034 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f9d2c06af60 msgr2=0x7f9d2c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:40.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:40.148+0000 7f9d463a5640 1 --2- 172.21.15.57:0/92748034 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f9d2c06af60 0x7f9d2c06d420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9d30035000 tx=0x7f9d30004730 comp rx=0 tx=0).stop 2023-10-19T14:01:40.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:40.148+0000 7f9d463a5640 1 -- 172.21.15.57:0/92748034 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9d4015b2a0 msgr2=0x7f9d400ca380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:40.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:40.148+0000 7f9d463a5640 1 --2- 172.21.15.57:0/92748034 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9d4015b2a0 0x7f9d400ca380 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7f9d3800c7a0 tx=0x7f9d3800cc60 comp rx=0 tx=0).stop 2023-10-19T14:01:40.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:40.148+0000 7f9d463a5640 1 -- 172.21.15.57:0/92748034 shutdown_connections 2023-10-19T14:01:40.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:40.148+0000 7f9d463a5640 1 --2- 172.21.15.57:0/92748034 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f9d2c06af60 0x7f9d2c06d420 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:40.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:40.148+0000 7f9d463a5640 1 --2- 172.21.15.57:0/92748034 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9d4015b2a0 0x7f9d400ca380 unknown :-1 s=CLOSED pgs=1902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:40.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:40.148+0000 7f9d463a5640 1 --2- 172.21.15.57:0/92748034 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9d400cf340 0x7f9d400c9e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:40.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:40.148+0000 7f9d463a5640 1 --2- 172.21.15.57:0/92748034 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9d400ceb40 0x7f9d40158990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:40.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:40.148+0000 7f9d463a5640 1 -- 172.21.15.57:0/92748034 >> 172.21.15.57:0/92748034 conn(0x7f9d400d0d60 msgr2=0x7f9d4006b6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:40.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:40.148+0000 7f9d463a5640 1 -- 172.21.15.57:0/92748034 shutdown_connections 2023-10-19T14:01:40.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:40.148+0000 7f9d463a5640 1 -- 172.21.15.57:0/92748034 wait complete. 2023-10-19T14:01:40.161 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-19T14:01:40.162 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-19T14:01:40.280 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:40.273+0000 7fb39ed00640 1 -- 172.21.15.57:0/1491151410 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb39806ffb0 msgr2=0x7fb3980c96b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:40.280 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:40.273+0000 7fb39ed00640 1 --2- 172.21.15.57:0/1491151410 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb39806ffb0 0x7fb3980c96b0 secure :-1 s=READY pgs=1904 cs=0 l=1 rev1=1 crypto rx=0x7fb38c009830 tx=0x7fb38c02bd60 comp rx=0 tx=0).stop 2023-10-19T14:01:40.280 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:40.273+0000 7fb39ed00640 1 -- 172.21.15.57:0/1491151410 shutdown_connections 2023-10-19T14:01:40.280 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:40.273+0000 7fb39ed00640 1 --2- 172.21.15.57:0/1491151410 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb39806ffb0 0x7fb3980c96b0 unknown :-1 s=CLOSED pgs=1904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:40.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:40.273+0000 7fb39ed00640 1 --2- 172.21.15.57:0/1491151410 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb3980b1920 0x7fb39806fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:40.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:40.273+0000 7fb39ed00640 1 -- 172.21.15.57:0/1491151410 >> 172.21.15.57:0/1491151410 conn(0x7fb398069930 msgr2=0x7fb398069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:40.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:40.273+0000 7fb39ed00640 1 -- 172.21.15.57:0/1491151410 shutdown_connections 2023-10-19T14:01:40.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:40.273+0000 7fb39ed00640 1 -- 172.21.15.57:0/1491151410 wait complete. 2023-10-19T14:01:40.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:40.274+0000 7fb39ed00640 1 Processor -- start 2023-10-19T14:01:40.282 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:40.274+0000 7fb39ed00640 1 -- start start 2023-10-19T14:01:40.282 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:40.275+0000 7fb39ed00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb39806ffb0 0x7fb3981554a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:40.282 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:40.275+0000 7fb39ed00640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb3980b1920 0x7fb398152030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:40.282 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:40.275+0000 7fb39ed00640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3980c9bf0 0x7fb3981f0f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:40.282 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:40.275+0000 7fb39ed00640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb39815cf90 con 0x7fb3980c9bf0 2023-10-19T14:01:40.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:40.275+0000 7fb39ed00640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb398155ef0 con 0x7fb39806ffb0 2023-10-19T14:01:40.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:40.275+0000 7fb39ed00640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb398156160 con 0x7fb3980b1920 2023-10-19T14:01:40.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:40.275+0000 7fb39d29d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3980c9bf0 0x7fb3981f0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:40.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:40.275+0000 7fb39d29d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3980c9bf0 0x7fb3981f0f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35228/0 (socket says 172.21.15.57:35228) 2023-10-19T14:01:40.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:40.275+0000 7fb39d29d640 1 -- 172.21.15.57:0/2230322556 learned_addr learned my addr 172.21.15.57:0/2230322556 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:01:40.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:40.275+0000 7fb397fff640 1 --2- 172.21.15.57:0/2230322556 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb3980b1920 0x7fb398152030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:40.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:40.275+0000 7fb39ca9c640 1 --2- 172.21.15.57:0/2230322556 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb39806ffb0 0x7fb3981554a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:40.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:40.275+0000 7fb39d29d640 1 -- 172.21.15.57:0/2230322556 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb3980b1920 msgr2=0x7fb398152030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:40.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:40.275+0000 7fb39d29d640 1 --2- 172.21.15.57:0/2230322556 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb3980b1920 0x7fb398152030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:40.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:40.275+0000 7fb39d29d640 1 -- 172.21.15.57:0/2230322556 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb39806ffb0 msgr2=0x7fb3981554a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:40.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:40.275+0000 7fb39d29d640 1 --2- 172.21.15.57:0/2230322556 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb39806ffb0 0x7fb3981554a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:40.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:40.275+0000 7fb39d29d640 1 -- 172.21.15.57:0/2230322556 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb38c009490 con 0x7fb3980c9bf0 2023-10-19T14:01:40.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:40.276+0000 7fb39ca9c640 1 --2- 172.21.15.57:0/2230322556 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb39806ffb0 0x7fb3981554a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:01:40.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:40.276+0000 7fb39d29d640 1 --2- 172.21.15.57:0/2230322556 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3980c9bf0 0x7fb3981f0f30 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7fb38800c370 tx=0x7fb38800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:40.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:40.276+0000 7fb3957fa640 1 -- 172.21.15.57:0/2230322556 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb388011790 con 0x7fb3980c9bf0 2023-10-19T14:01:40.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:40.276+0000 7fb3957fa640 1 -- 172.21.15.57:0/2230322556 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fb3880119e0 con 0x7fb3980c9bf0 2023-10-19T14:01:40.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:40.276+0000 7fb39ed00640 1 -- 172.21.15.57:0/2230322556 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb398156440 con 0x7fb3980c9bf0 2023-10-19T14:01:40.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:40.276+0000 7fb3957fa640 1 -- 172.21.15.57:0/2230322556 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb388011dd0 con 0x7fb3980c9bf0 2023-10-19T14:01:40.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:40.276+0000 7fb39ed00640 1 -- 172.21.15.57:0/2230322556 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb398155aa0 con 0x7fb3980c9bf0 2023-10-19T14:01:40.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:40.277+0000 7fb39ed00640 1 -- 172.21.15.57:0/2230322556 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb398156680 con 0x7fb3980c9bf0 2023-10-19T14:01:40.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:40.277+0000 7fb3957fa640 1 -- 172.21.15.57:0/2230322556 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 91) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb38801c460 con 0x7fb3980c9bf0 2023-10-19T14:01:40.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:40.278+0000 7fb3957fa640 1 --2- 172.21.15.57:0/2230322556 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7fb38406af60 0x7fb38406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:40.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:40.278+0000 7fb3957fa640 1 -- 172.21.15.57:0/2230322556 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fb388089840 con 0x7fb3980c9bf0 2023-10-19T14:01:40.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:40.278+0000 7fb39ca9c640 1 --2- 172.21.15.57:0/2230322556 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7fb38406af60 0x7fb38406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:40.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:40.279+0000 7fb39ca9c640 1 --2- 172.21.15.57:0/2230322556 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7fb38406af60 0x7fb38406d420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb390005c40 tx=0x7fb3900082f0 comp rx=0 tx=0).ready entity=mgr.8923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:40.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:40.282+0000 7fb3957fa640 1 -- 172.21.15.57:0/2230322556 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb398156680 con 0x7fb3980c9bf0 2023-10-19T14:01:40.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:40.494+0000 7fb39ed00640 1 -- 172.21.15.57:0/2230322556 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fb398156680 con 0x7fb3980c9bf0 2023-10-19T14:01:41.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.157+0000 7fb3957fa640 1 -- 172.21.15.57:0/2230322556 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v70) v1 ==== 122+0+0 (secure 0 0 0) 0x7fb398156680 con 0x7fb3980c9bf0 2023-10-19T14:01:41.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.159+0000 7fb39ed00640 1 -- 172.21.15.57:0/2230322556 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7fb38406af60 msgr2=0x7fb38406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:41.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.159+0000 7fb39ed00640 1 --2- 172.21.15.57:0/2230322556 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7fb38406af60 0x7fb38406d420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb390005c40 tx=0x7fb3900082f0 comp rx=0 tx=0).stop 2023-10-19T14:01:41.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.159+0000 7fb39ed00640 1 -- 172.21.15.57:0/2230322556 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3980c9bf0 msgr2=0x7fb3981f0f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:41.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.159+0000 7fb39ed00640 1 --2- 172.21.15.57:0/2230322556 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3980c9bf0 0x7fb3981f0f30 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7fb38800c370 tx=0x7fb38800c830 comp rx=0 tx=0).stop 2023-10-19T14:01:41.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.159+0000 7fb39ed00640 1 -- 172.21.15.57:0/2230322556 shutdown_connections 2023-10-19T14:01:41.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.159+0000 7fb39ed00640 1 --2- 172.21.15.57:0/2230322556 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7fb38406af60 0x7fb38406d420 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:41.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.159+0000 7fb39ed00640 1 --2- 172.21.15.57:0/2230322556 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3980c9bf0 0x7fb3981f0f30 unknown :-1 s=CLOSED pgs=1905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:41.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.159+0000 7fb39ed00640 1 --2- 172.21.15.57:0/2230322556 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb3980b1920 0x7fb398152030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:41.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.159+0000 7fb39ed00640 1 --2- 172.21.15.57:0/2230322556 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb39806ffb0 0x7fb3981554a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:41.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.159+0000 7fb39ed00640 1 -- 172.21.15.57:0/2230322556 >> 172.21.15.57:0/2230322556 conn(0x7fb398069930 msgr2=0x7fb3980af340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:41.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.159+0000 7fb39ed00640 1 -- 172.21.15.57:0/2230322556 shutdown_connections 2023-10-19T14:01:41.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.159+0000 7fb39ed00640 1 -- 172.21.15.57:0/2230322556 wait complete. 2023-10-19T14:01:41.173 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-19T14:01:41.174 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T14:01:41.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.284+0000 7f42dc6e5640 1 -- 172.21.15.57:0/1218696290 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f42d40ceb40 msgr2=0x7f42d40b0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:41.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.284+0000 7f42dc6e5640 1 --2- 172.21.15.57:0/1218696290 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f42d40ceb40 0x7f42d40b0650 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f42cc005fd0 tx=0x7f42cc02c090 comp rx=0 tx=0).stop 2023-10-19T14:01:41.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.284+0000 7f42dc6e5640 1 -- 172.21.15.57:0/1218696290 shutdown_connections 2023-10-19T14:01:41.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.284+0000 7f42dc6e5640 1 --2- 172.21.15.57:0/1218696290 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f42d40ceb40 0x7f42d40b0650 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:41.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.284+0000 7f42dc6e5640 1 --2- 172.21.15.57:0/1218696290 >> v2:172.21.15.179:3300/0 conn(0x7f42d40cf340 0x7f42d40d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:41.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.284+0000 7f42dc6e5640 1 -- 172.21.15.57:0/1218696290 >> 172.21.15.57:0/1218696290 conn(0x7f42d40d0d60 msgr2=0x7f42d40d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:41.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.284+0000 7f42dc6e5640 1 -- 172.21.15.57:0/1218696290 shutdown_connections 2023-10-19T14:01:41.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.284+0000 7f42dc6e5640 1 -- 172.21.15.57:0/1218696290 wait complete. 2023-10-19T14:01:41.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.285+0000 7f42dc6e5640 1 Processor -- start 2023-10-19T14:01:41.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.285+0000 7f42dc6e5640 1 -- start start 2023-10-19T14:01:41.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.286+0000 7f42dc6e5640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f42d40ceb40 0x7f42d41e5710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:41.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.286+0000 7f42dc6e5640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f42d40cf340 0x7f42d415cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:41.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.286+0000 7f42dc6e5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f42d415b2a0 0x7f42d415d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:41.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.286+0000 7f42dc6e5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42d41e5c50 con 0x7f42d415b2a0 2023-10-19T14:01:41.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.286+0000 7f42dc6e5640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42d415daf0 con 0x7f42d40ceb40 2023-10-19T14:01:41.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.286+0000 7f42dc6e5640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42d415ddd0 con 0x7f42d40cf340 2023-10-19T14:01:41.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.286+0000 7f42dac82640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f42d415b2a0 0x7f42d415d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:41.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.286+0000 7f42dac82640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f42d415b2a0 0x7f42d415d2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35238/0 (socket says 172.21.15.57:35238) 2023-10-19T14:01:41.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.286+0000 7f42dac82640 1 -- 172.21.15.57:0/611747529 learned_addr learned my addr 172.21.15.57:0/611747529 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:01:41.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.286+0000 7f42d9c80640 1 --2- 172.21.15.57:0/611747529 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f42d40cf340 0x7f42d415cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:41.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.286+0000 7f42da481640 1 --2- 172.21.15.57:0/611747529 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f42d40ceb40 0x7f42d41e5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:41.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.287+0000 7f42dac82640 1 -- 172.21.15.57:0/611747529 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f42d40cf340 msgr2=0x7f42d415cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:41.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.287+0000 7f42dac82640 1 --2- 172.21.15.57:0/611747529 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f42d40cf340 0x7f42d415cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:41.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.287+0000 7f42dac82640 1 -- 172.21.15.57:0/611747529 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f42d40ceb40 msgr2=0x7f42d41e5710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:41.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.287+0000 7f42dac82640 1 --2- 172.21.15.57:0/611747529 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f42d40ceb40 0x7f42d41e5710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:41.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.287+0000 7f42dac82640 1 -- 172.21.15.57:0/611747529 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42cc009900 con 0x7f42d415b2a0 2023-10-19T14:01:41.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.287+0000 7f42da481640 1 --2- 172.21.15.57:0/611747529 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f42d40ceb40 0x7f42d41e5710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:01:41.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.287+0000 7f42dac82640 1 --2- 172.21.15.57:0/611747529 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f42d415b2a0 0x7f42d415d2c0 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7f42cc009bf0 tx=0x7f42cc003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:41.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.287+0000 7f42caffd640 1 -- 172.21.15.57:0/611747529 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42cc0044d0 con 0x7f42d415b2a0 2023-10-19T14:01:41.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.287+0000 7f42caffd640 1 -- 172.21.15.57:0/611747529 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f42cc004720 con 0x7f42d415b2a0 2023-10-19T14:01:41.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.287+0000 7f42dc6e5640 1 -- 172.21.15.57:0/611747529 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42d415e0b0 con 0x7f42d415b2a0 2023-10-19T14:01:41.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.287+0000 7f42caffd640 1 -- 172.21.15.57:0/611747529 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42cc034810 con 0x7f42d415b2a0 2023-10-19T14:01:41.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.287+0000 7f42dc6e5640 1 -- 172.21.15.57:0/611747529 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42d415d800 con 0x7f42d415b2a0 2023-10-19T14:01:41.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.288+0000 7f42dc6e5640 1 -- 172.21.15.57:0/611747529 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42d40690e0 con 0x7f42d415b2a0 2023-10-19T14:01:41.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.288+0000 7f42caffd640 1 -- 172.21.15.57:0/611747529 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 91) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f42cc00a040 con 0x7f42d415b2a0 2023-10-19T14:01:41.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.289+0000 7f42caffd640 1 --2- 172.21.15.57:0/611747529 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f42b806af60 0x7f42b806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:41.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.289+0000 7f42caffd640 1 -- 172.21.15.57:0/611747529 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f42cc0ac2c0 con 0x7f42d415b2a0 2023-10-19T14:01:41.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.289+0000 7f42da481640 1 --2- 172.21.15.57:0/611747529 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f42b806af60 0x7f42b806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:41.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.290+0000 7f42da481640 1 --2- 172.21.15.57:0/611747529 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f42b806af60 0x7f42b806d420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f42d41e68b0 tx=0x7f42d0005f90 comp rx=0 tx=0).ready entity=mgr.8923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:41.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.293+0000 7f42caffd640 1 -- 172.21.15.57:0/611747529 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f42cc077100 con 0x7f42d415b2a0 2023-10-19T14:01:41.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.508+0000 7f42dc6e5640 1 -- 172.21.15.57:0/611747529 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f42d40b0cd0 con 0x7f42d415b2a0 2023-10-19T14:01:41.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.508+0000 7f42caffd640 1 -- 172.21.15.57:0/611747529 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+4920 (secure 0 0 0) 0x7f42cc07a720 con 0x7f42d415b2a0 2023-10-19T14:01:41.511 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:01:41.512 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":70,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8126,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/1759316409","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1759316409},{"type":"v1","addr":"172.21.15.57:6835","nonce":1759316409}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67},{"gid":8129,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/2462599644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2462599644},{"type":"v1","addr":"172.21.15.57:6834","nonce":2462599644}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67},{"gid":9136,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/373220657","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":373220657},{"type":"v1","addr":"172.21.15.179:6836","nonce":373220657}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T14:01:38.503105+0000","modified":"2023-10-19T14:01:40.495529+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9139},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9139":{"gid":9139,"name":"d","rank":0,"incarnation":68,"state":"up:active","state_seq":3,"addr":"172.21.15.179:6837/1102326590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":1102326590},{"type":"v1","addr":"172.21.15.179:6837","nonce":1102326590}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2023-10-19T14:01:41.512 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 70 2023-10-19T14:01:41.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.510+0000 7f42dc6e5640 1 -- 172.21.15.57:0/611747529 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f42b806af60 msgr2=0x7f42b806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:41.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.510+0000 7f42dc6e5640 1 --2- 172.21.15.57:0/611747529 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f42b806af60 0x7f42b806d420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f42d41e68b0 tx=0x7f42d0005f90 comp rx=0 tx=0).stop 2023-10-19T14:01:41.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.510+0000 7f42dc6e5640 1 -- 172.21.15.57:0/611747529 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f42d415b2a0 msgr2=0x7f42d415d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:41.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.510+0000 7f42dc6e5640 1 --2- 172.21.15.57:0/611747529 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f42d415b2a0 0x7f42d415d2c0 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7f42cc009bf0 tx=0x7f42cc003f90 comp rx=0 tx=0).stop 2023-10-19T14:01:41.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.510+0000 7f42dc6e5640 1 -- 172.21.15.57:0/611747529 shutdown_connections 2023-10-19T14:01:41.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.510+0000 7f42dc6e5640 1 --2- 172.21.15.57:0/611747529 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f42b806af60 0x7f42b806d420 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:41.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.510+0000 7f42dc6e5640 1 --2- 172.21.15.57:0/611747529 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f42d415b2a0 0x7f42d415d2c0 unknown :-1 s=CLOSED pgs=1906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:41.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.510+0000 7f42dc6e5640 1 --2- 172.21.15.57:0/611747529 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f42d40cf340 0x7f42d415cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:41.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.510+0000 7f42dc6e5640 1 --2- 172.21.15.57:0/611747529 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f42d40ceb40 0x7f42d41e5710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:41.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.510+0000 7f42dc6e5640 1 -- 172.21.15.57:0/611747529 >> 172.21.15.57:0/611747529 conn(0x7f42d40d0d60 msgr2=0x7f42d406b6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:41.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.510+0000 7f42dc6e5640 1 -- 172.21.15.57:0/611747529 shutdown_connections 2023-10-19T14:01:41.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.510+0000 7f42dc6e5640 1 -- 172.21.15.57:0/611747529 wait complete. 2023-10-19T14:01:41.524 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T14:01:41.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.630+0000 7fba54eb4640 1 -- 172.21.15.57:0/3213587580 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fba500cf340 msgr2=0x7fba500d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:41.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.630+0000 7fba54eb4640 1 --2- 172.21.15.57:0/3213587580 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fba500cf340 0x7fba500d1910 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7fba5015b680 tx=0x7fba4002bcf0 comp rx=0 tx=0).stop 2023-10-19T14:01:41.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.631+0000 7fba54eb4640 1 -- 172.21.15.57:0/3213587580 shutdown_connections 2023-10-19T14:01:41.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.631+0000 7fba54eb4640 1 --2- 172.21.15.57:0/3213587580 >> v2:172.21.15.179:3300/0 conn(0x7fba500ceb40 0x7fba500b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:41.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.631+0000 7fba54eb4640 1 --2- 172.21.15.57:0/3213587580 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fba500cf340 0x7fba500d1910 unknown :-1 s=CLOSED pgs=1908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:41.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.631+0000 7fba54eb4640 1 -- 172.21.15.57:0/3213587580 >> 172.21.15.57:0/3213587580 conn(0x7fba500d0d60 msgr2=0x7fba500d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:41.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.631+0000 7fba54eb4640 1 -- 172.21.15.57:0/3213587580 shutdown_connections 2023-10-19T14:01:41.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.631+0000 7fba54eb4640 1 -- 172.21.15.57:0/3213587580 wait complete. 2023-10-19T14:01:41.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.632+0000 7fba54eb4640 1 Processor -- start 2023-10-19T14:01:41.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.632+0000 7fba54eb4640 1 -- start start 2023-10-19T14:01:41.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.633+0000 7fba54eb4640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fba500ceb40 0x7fba500b50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:41.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.633+0000 7fba54eb4640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fba500cf340 0x7fba501f20b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:41.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.633+0000 7fba54eb4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fba5015b2a0 0x7fba50158c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:41.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.633+0000 7fba54eb4640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba501f25f0 con 0x7fba5015b2a0 2023-10-19T14:01:41.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.633+0000 7fba54eb4640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba501f4c20 con 0x7fba500cf340 2023-10-19T14:01:41.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.633+0000 7fba54eb4640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba501f4e70 con 0x7fba500ceb40 2023-10-19T14:01:41.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.633+0000 7fba4ed9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fba5015b2a0 0x7fba50158c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:41.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.633+0000 7fba4ed9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fba5015b2a0 0x7fba50158c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35262/0 (socket says 172.21.15.57:35262) 2023-10-19T14:01:41.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.633+0000 7fba4ed9d640 1 -- 172.21.15.57:0/3266411006 learned_addr learned my addr 172.21.15.57:0/3266411006 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:01:41.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.633+0000 7fba4e59c640 1 --2- 172.21.15.57:0/3266411006 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fba500ceb40 0x7fba500b50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:41.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.633+0000 7fba4dd9b640 1 --2- 172.21.15.57:0/3266411006 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fba500cf340 0x7fba501f20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:41.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.633+0000 7fba4ed9d640 1 -- 172.21.15.57:0/3266411006 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fba500ceb40 msgr2=0x7fba500b50a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:41.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.633+0000 7fba4ed9d640 1 --2- 172.21.15.57:0/3266411006 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fba500ceb40 0x7fba500b50a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:41.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.633+0000 7fba4ed9d640 1 -- 172.21.15.57:0/3266411006 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fba500cf340 msgr2=0x7fba501f20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:41.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.633+0000 7fba4ed9d640 1 --2- 172.21.15.57:0/3266411006 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fba500cf340 0x7fba501f20b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:41.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.633+0000 7fba4ed9d640 1 -- 172.21.15.57:0/3266411006 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba40009490 con 0x7fba5015b2a0 2023-10-19T14:01:41.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.633+0000 7fba4dd9b640 1 --2- 172.21.15.57:0/3266411006 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fba500cf340 0x7fba501f20b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:01:41.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.633+0000 7fba4ed9d640 1 --2- 172.21.15.57:0/3266411006 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fba5015b2a0 0x7fba50158c70 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7fba4800eb10 tx=0x7fba4800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:41.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.634+0000 7fba3effd640 1 -- 172.21.15.57:0/3266411006 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba48004b20 con 0x7fba5015b2a0 2023-10-19T14:01:41.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.634+0000 7fba3effd640 1 -- 172.21.15.57:0/3266411006 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fba48004ca0 con 0x7fba5015b2a0 2023-10-19T14:01:41.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.634+0000 7fba54eb4640 1 -- 172.21.15.57:0/3266411006 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba501f5150 con 0x7fba5015b2a0 2023-10-19T14:01:41.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.634+0000 7fba3effd640 1 -- 172.21.15.57:0/3266411006 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba48005280 con 0x7fba5015b2a0 2023-10-19T14:01:41.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.634+0000 7fba54eb4640 1 -- 172.21.15.57:0/3266411006 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba501f5710 con 0x7fba5015b2a0 2023-10-19T14:01:41.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.635+0000 7fba54eb4640 1 -- 172.21.15.57:0/3266411006 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba500690e0 con 0x7fba5015b2a0 2023-10-19T14:01:41.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.635+0000 7fba3effd640 1 -- 172.21.15.57:0/3266411006 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 91) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fba480053e0 con 0x7fba5015b2a0 2023-10-19T14:01:41.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.635+0000 7fba3effd640 1 --2- 172.21.15.57:0/3266411006 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7fba3006af90 0x7fba3006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:41.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.636+0000 7fba3effd640 1 -- 172.21.15.57:0/3266411006 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fba48014070 con 0x7fba5015b2a0 2023-10-19T14:01:41.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.636+0000 7fba4e59c640 1 --2- 172.21.15.57:0/3266411006 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7fba3006af90 0x7fba3006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:41.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.636+0000 7fba4e59c640 1 --2- 172.21.15.57:0/3266411006 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7fba3006af90 0x7fba3006d450 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fba40036000 tx=0x7fba40004730 comp rx=0 tx=0).ready entity=mgr.8923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:41.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.640+0000 7fba3effd640 1 -- 172.21.15.57:0/3266411006 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fba48055490 con 0x7fba5015b2a0 2023-10-19T14:01:41.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.815+0000 7fba54eb4640 1 -- 172.21.15.57:0/3266411006 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fba5006b250 con 0x7fba5015b2a0 2023-10-19T14:01:41.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.817+0000 7fba3effd640 1 -- 172.21.15.57:0/3266411006 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v175) v1 ==== 74+0+16779 (secure 0 0 0) 0x7fba48058ab0 con 0x7fba5015b2a0 2023-10-19T14:01:41.819 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:01:41.820 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":175,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T14:01:40.056651+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2023-10-19T14:01:36.799358+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"174","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_data","create_time":"2023-10-19T14:01:37.481213+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"175","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T14:01:41.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.818+0000 7fba54eb4640 1 -- 172.21.15.57:0/3266411006 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7fba3006af90 msgr2=0x7fba3006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:41.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.818+0000 7fba54eb4640 1 --2- 172.21.15.57:0/3266411006 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7fba3006af90 0x7fba3006d450 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fba40036000 tx=0x7fba40004730 comp rx=0 tx=0).stop 2023-10-19T14:01:41.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.818+0000 7fba54eb4640 1 -- 172.21.15.57:0/3266411006 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fba5015b2a0 msgr2=0x7fba50158c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:41.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.818+0000 7fba54eb4640 1 --2- 172.21.15.57:0/3266411006 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fba5015b2a0 0x7fba50158c70 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7fba4800eb10 tx=0x7fba4800c2d0 comp rx=0 tx=0).stop 2023-10-19T14:01:41.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.819+0000 7fba54eb4640 1 -- 172.21.15.57:0/3266411006 shutdown_connections 2023-10-19T14:01:41.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.819+0000 7fba54eb4640 1 --2- 172.21.15.57:0/3266411006 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7fba3006af90 0x7fba3006d450 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:41.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.819+0000 7fba54eb4640 1 --2- 172.21.15.57:0/3266411006 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fba5015b2a0 0x7fba50158c70 unknown :-1 s=CLOSED pgs=1909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:41.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.819+0000 7fba54eb4640 1 --2- 172.21.15.57:0/3266411006 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fba500cf340 0x7fba501f20b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:41.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.819+0000 7fba54eb4640 1 --2- 172.21.15.57:0/3266411006 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fba500ceb40 0x7fba500b50a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:41.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.819+0000 7fba54eb4640 1 -- 172.21.15.57:0/3266411006 >> 172.21.15.57:0/3266411006 conn(0x7fba500d0d60 msgr2=0x7fba500af350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:41.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.819+0000 7fba54eb4640 1 -- 172.21.15.57:0/3266411006 shutdown_connections 2023-10-19T14:01:41.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.819+0000 7fba54eb4640 1 -- 172.21.15.57:0/3266411006 wait complete. 2023-10-19T14:01:41.832 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-19T14:01:41.833 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-19T14:01:41.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.939+0000 7f7d56fc0640 1 -- 172.21.15.57:0/4058975628 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7d5015b2a0 msgr2=0x7f7d500cf720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:41.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.939+0000 7f7d56fc0640 1 --2- 172.21.15.57:0/4058975628 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7d5015b2a0 0x7f7d500cf720 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 crypto rx=0x7f7d5008a810 tx=0x7f7d4002bd10 comp rx=0 tx=0).stop 2023-10-19T14:01:41.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.940+0000 7f7d56fc0640 1 -- 172.21.15.57:0/4058975628 shutdown_connections 2023-10-19T14:01:41.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.940+0000 7f7d56fc0640 1 --2- 172.21.15.57:0/4058975628 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7d500ceb40 0x7f7d500b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:41.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.940+0000 7f7d56fc0640 1 --2- 172.21.15.57:0/4058975628 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7d5015b2a0 0x7f7d500cf720 unknown :-1 s=CLOSED pgs=1910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:41.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.940+0000 7f7d56fc0640 1 --2- 172.21.15.57:0/4058975628 >> v2:172.21.15.179:3300/0 conn(0x7f7d500cf340 0x7f7d500d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:41.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.940+0000 7f7d56fc0640 1 -- 172.21.15.57:0/4058975628 >> 172.21.15.57:0/4058975628 conn(0x7f7d500d0d60 msgr2=0x7f7d500d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:41.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.940+0000 7f7d56fc0640 1 -- 172.21.15.57:0/4058975628 shutdown_connections 2023-10-19T14:01:41.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.940+0000 7f7d56fc0640 1 -- 172.21.15.57:0/4058975628 wait complete. 2023-10-19T14:01:41.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.941+0000 7f7d56fc0640 1 Processor -- start 2023-10-19T14:01:41.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.941+0000 7f7d56fc0640 1 -- start start 2023-10-19T14:01:41.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.942+0000 7f7d56fc0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7d500ceb40 0x7f7d500ccb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:41.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.942+0000 7f7d56fc0640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7d500cf340 0x7f7d501501c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:41.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.942+0000 7f7d56fc0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7d5015b2a0 0x7f7d50150700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:41.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.942+0000 7f7d56fc0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d50150c40 con 0x7f7d5015b2a0 2023-10-19T14:01:41.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.942+0000 7f7d56fc0640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d5009d6a0 con 0x7f7d500ceb40 2023-10-19T14:01:41.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.942+0000 7f7d56fc0640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d5009d980 con 0x7f7d500cf340 2023-10-19T14:01:41.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.942+0000 7f7d5555d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7d5015b2a0 0x7f7d50150700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:41.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.942+0000 7f7d5555d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7d5015b2a0 0x7f7d50150700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35282/0 (socket says 172.21.15.57:35282) 2023-10-19T14:01:41.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.942+0000 7f7d5555d640 1 -- 172.21.15.57:0/2430704007 learned_addr learned my addr 172.21.15.57:0/2430704007 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:01:41.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.942+0000 7f7d54d5c640 1 --2- 172.21.15.57:0/2430704007 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7d500ceb40 0x7f7d500ccb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:41.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.942+0000 7f7d4ffff640 1 --2- 172.21.15.57:0/2430704007 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7d500cf340 0x7f7d501501c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:41.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.942+0000 7f7d5555d640 1 -- 172.21.15.57:0/2430704007 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7d500cf340 msgr2=0x7f7d501501c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:41.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.942+0000 7f7d5555d640 1 --2- 172.21.15.57:0/2430704007 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7d500cf340 0x7f7d501501c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:41.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.942+0000 7f7d5555d640 1 -- 172.21.15.57:0/2430704007 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7d500ceb40 msgr2=0x7f7d500ccb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:41.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.942+0000 7f7d5555d640 1 --2- 172.21.15.57:0/2430704007 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7d500ceb40 0x7f7d500ccb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:41.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.942+0000 7f7d5555d640 1 -- 172.21.15.57:0/2430704007 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d40009560 con 0x7f7d5015b2a0 2023-10-19T14:01:41.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.942+0000 7f7d5555d640 1 --2- 172.21.15.57:0/2430704007 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7d5015b2a0 0x7f7d50150700 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7f7d3800d650 tx=0x7f7d3800db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:41.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.942+0000 7f7d4d7fa640 1 -- 172.21.15.57:0/2430704007 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d38005220 con 0x7f7d5015b2a0 2023-10-19T14:01:41.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.942+0000 7f7d4d7fa640 1 -- 172.21.15.57:0/2430704007 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f7d38005490 con 0x7f7d5015b2a0 2023-10-19T14:01:41.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.942+0000 7f7d56fc0640 1 -- 172.21.15.57:0/2430704007 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d5009dc60 con 0x7f7d5015b2a0 2023-10-19T14:01:41.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.942+0000 7f7d4d7fa640 1 -- 172.21.15.57:0/2430704007 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d38009a80 con 0x7f7d5015b2a0 2023-10-19T14:01:41.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.942+0000 7f7d56fc0640 1 -- 172.21.15.57:0/2430704007 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d5009e220 con 0x7f7d5015b2a0 2023-10-19T14:01:41.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.943+0000 7f7d56fc0640 1 -- 172.21.15.57:0/2430704007 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d5009e410 con 0x7f7d5015b2a0 2023-10-19T14:01:41.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.944+0000 7f7d4d7fa640 1 -- 172.21.15.57:0/2430704007 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 91) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f7d38009be0 con 0x7f7d5015b2a0 2023-10-19T14:01:41.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.944+0000 7f7d4d7fa640 1 --2- 172.21.15.57:0/2430704007 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f7d3006af90 0x7f7d3006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:41.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.944+0000 7f7d4d7fa640 1 -- 172.21.15.57:0/2430704007 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f7d38089aa0 con 0x7f7d5015b2a0 2023-10-19T14:01:41.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.945+0000 7f7d54d5c640 1 --2- 172.21.15.57:0/2430704007 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f7d3006af90 0x7f7d3006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:41.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.945+0000 7f7d54d5c640 1 --2- 172.21.15.57:0/2430704007 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f7d3006af90 0x7f7d3006d450 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7d48005fd0 tx=0x7f7d48009680 comp rx=0 tx=0).ready entity=mgr.8923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:41.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:41.948+0000 7f7d4d7fa640 1 -- 172.21.15.57:0/2430704007 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7d38054800 con 0x7f7d5015b2a0 2023-10-19T14:01:42.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.120+0000 7f7d56fc0640 1 -- 172.21.15.57:0/2430704007 --> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7d500cc170 con 0x7f7d3006af90 2023-10-19T14:01:42.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.129+0000 7f7d4d7fa640 1 -- 172.21.15.57:0/2430704007 <== mgr.8923 v2:172.21.15.179:6832/2037224568 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+290172 (secure 0 0 0) 0x7f7d48009c20 con 0x7f7d3006af90 2023-10-19T14:01:42.134 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:01:42.136 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2023-10-19T14:01:42.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.132+0000 7f7d56fc0640 1 -- 172.21.15.57:0/2430704007 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f7d3006af90 msgr2=0x7f7d3006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:42.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.132+0000 7f7d56fc0640 1 --2- 172.21.15.57:0/2430704007 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f7d3006af90 0x7f7d3006d450 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7d48005fd0 tx=0x7f7d48009680 comp rx=0 tx=0).stop 2023-10-19T14:01:42.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.132+0000 7f7d56fc0640 1 -- 172.21.15.57:0/2430704007 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7d5015b2a0 msgr2=0x7f7d50150700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:42.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.132+0000 7f7d56fc0640 1 --2- 172.21.15.57:0/2430704007 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7d5015b2a0 0x7f7d50150700 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7f7d3800d650 tx=0x7f7d3800db10 comp rx=0 tx=0).stop 2023-10-19T14:01:42.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.132+0000 7f7d56fc0640 1 -- 172.21.15.57:0/2430704007 shutdown_connections 2023-10-19T14:01:42.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.132+0000 7f7d56fc0640 1 --2- 172.21.15.57:0/2430704007 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f7d3006af90 0x7f7d3006d450 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:42.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.132+0000 7f7d56fc0640 1 --2- 172.21.15.57:0/2430704007 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7d5015b2a0 0x7f7d50150700 unknown :-1 s=CLOSED pgs=1911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:42.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.132+0000 7f7d56fc0640 1 --2- 172.21.15.57:0/2430704007 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7d500cf340 0x7f7d501501c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:42.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.132+0000 7f7d56fc0640 1 --2- 172.21.15.57:0/2430704007 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7d500ceb40 0x7f7d500ccb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:42.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.132+0000 7f7d56fc0640 1 -- 172.21.15.57:0/2430704007 >> 172.21.15.57:0/2430704007 conn(0x7f7d500d0d60 msgr2=0x7f7d500aeca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:42.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.132+0000 7f7d56fc0640 1 -- 172.21.15.57:0/2430704007 shutdown_connections 2023-10-19T14:01:42.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.132+0000 7f7d56fc0640 1 -- 172.21.15.57:0/2430704007 wait complete. 2023-10-19T14:01:42.145 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2023-10-19T14:01:40.545197+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1538,"num_read_kb":3182,"num_write":493,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3147,"ondisk_log_size":3147,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":148,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":385704,"kb_used_data":18272,"kb_used_omap":12,"kb_used_meta":367283,"kb_avail":754589016,"statfs":{"total":773094113280,"available":772699152384,"internally_reserved":0,"allocated":18710528,"data_stored":10054449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":376098361},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000806"},"pg_stats":[{"pgid":"22.3f","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.061645+0000","last_change":"2023-10-19T14:01:39.071873+0000","last_active":"2023-10-19T14:01:40.061645+0000","last_peered":"2023-10-19T14:01:40.061645+0000","last_clean":"2023-10-19T14:01:40.061645+0000","last_became_active":"2023-10-19T14:01:39.071596+0000","last_became_peered":"2023-10-19T14:01:39.071596+0000","last_unstale":"2023-10-19T14:01:40.061645+0000","last_undegraded":"2023-10-19T14:01:40.061645+0000","last_fullsized":"2023-10-19T14:01:40.061645+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:03:21.766515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.3e","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.241296+0000","last_change":"2023-10-19T14:01:39.080355+0000","last_active":"2023-10-19T14:01:40.241296+0000","last_peered":"2023-10-19T14:01:40.241296+0000","last_clean":"2023-10-19T14:01:40.241296+0000","last_became_active":"2023-10-19T14:01:39.080067+0000","last_became_peered":"2023-10-19T14:01:39.080067+0000","last_unstale":"2023-10-19T14:01:40.241296+0000","last_undegraded":"2023-10-19T14:01:40.241296+0000","last_fullsized":"2023-10-19T14:01:40.241296+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:08:23.201667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.3d","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.241076+0000","last_change":"2023-10-19T14:01:39.075158+0000","last_active":"2023-10-19T14:01:40.241076+0000","last_peered":"2023-10-19T14:01:40.241076+0000","last_clean":"2023-10-19T14:01:40.241076+0000","last_became_active":"2023-10-19T14:01:39.074871+0000","last_became_peered":"2023-10-19T14:01:39.074871+0000","last_unstale":"2023-10-19T14:01:40.241076+0000","last_undegraded":"2023-10-19T14:01:40.241076+0000","last_fullsized":"2023-10-19T14:01:40.241076+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:40:04.433729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.3c","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.242382+0000","last_change":"2023-10-19T14:01:39.077385+0000","last_active":"2023-10-19T14:01:40.242382+0000","last_peered":"2023-10-19T14:01:40.242382+0000","last_clean":"2023-10-19T14:01:40.242382+0000","last_became_active":"2023-10-19T14:01:39.076971+0000","last_became_peered":"2023-10-19T14:01:39.076971+0000","last_unstale":"2023-10-19T14:01:40.242382+0000","last_undegraded":"2023-10-19T14:01:40.242382+0000","last_fullsized":"2023-10-19T14:01:40.242382+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:28:53.177652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.3b","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.061832+0000","last_change":"2023-10-19T14:01:39.079807+0000","last_active":"2023-10-19T14:01:40.061832+0000","last_peered":"2023-10-19T14:01:40.061832+0000","last_clean":"2023-10-19T14:01:40.061832+0000","last_became_active":"2023-10-19T14:01:39.079401+0000","last_became_peered":"2023-10-19T14:01:39.079401+0000","last_unstale":"2023-10-19T14:01:40.061832+0000","last_undegraded":"2023-10-19T14:01:40.061832+0000","last_fullsized":"2023-10-19T14:01:40.061832+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:17:25.334467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.3a","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.061615+0000","last_change":"2023-10-19T14:01:39.080122+0000","last_active":"2023-10-19T14:01:40.061615+0000","last_peered":"2023-10-19T14:01:40.061615+0000","last_clean":"2023-10-19T14:01:40.061615+0000","last_became_active":"2023-10-19T14:01:39.079614+0000","last_became_peered":"2023-10-19T14:01:39.079614+0000","last_unstale":"2023-10-19T14:01:40.061615+0000","last_undegraded":"2023-10-19T14:01:40.061615+0000","last_fullsized":"2023-10-19T14:01:40.061615+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:21:59.248909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.39","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.061330+0000","last_change":"2023-10-19T14:01:39.075475+0000","last_active":"2023-10-19T14:01:40.061330+0000","last_peered":"2023-10-19T14:01:40.061330+0000","last_clean":"2023-10-19T14:01:40.061330+0000","last_became_active":"2023-10-19T14:01:39.074883+0000","last_became_peered":"2023-10-19T14:01:39.074883+0000","last_unstale":"2023-10-19T14:01:40.061330+0000","last_undegraded":"2023-10-19T14:01:40.061330+0000","last_fullsized":"2023-10-19T14:01:40.061330+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:20:28.271547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.38","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.241930+0000","last_change":"2023-10-19T14:01:39.074913+0000","last_active":"2023-10-19T14:01:40.241930+0000","last_peered":"2023-10-19T14:01:40.241930+0000","last_clean":"2023-10-19T14:01:40.241930+0000","last_became_active":"2023-10-19T14:01:39.073378+0000","last_became_peered":"2023-10-19T14:01:39.073378+0000","last_unstale":"2023-10-19T14:01:40.241930+0000","last_undegraded":"2023-10-19T14:01:40.241930+0000","last_fullsized":"2023-10-19T14:01:40.241930+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:40:12.362136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.37","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.061543+0000","last_change":"2023-10-19T14:01:39.075601+0000","last_active":"2023-10-19T14:01:40.061543+0000","last_peered":"2023-10-19T14:01:40.061543+0000","last_clean":"2023-10-19T14:01:40.061543+0000","last_became_active":"2023-10-19T14:01:39.074903+0000","last_became_peered":"2023-10-19T14:01:39.074903+0000","last_unstale":"2023-10-19T14:01:40.061543+0000","last_undegraded":"2023-10-19T14:01:40.061543+0000","last_fullsized":"2023-10-19T14:01:40.061543+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:54:44.077594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.36","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.061464+0000","last_change":"2023-10-19T14:01:39.077407+0000","last_active":"2023-10-19T14:01:40.061464+0000","last_peered":"2023-10-19T14:01:40.061464+0000","last_clean":"2023-10-19T14:01:40.061464+0000","last_became_active":"2023-10-19T14:01:39.077030+0000","last_became_peered":"2023-10-19T14:01:39.077030+0000","last_unstale":"2023-10-19T14:01:40.061464+0000","last_undegraded":"2023-10-19T14:01:40.061464+0000","last_fullsized":"2023-10-19T14:01:40.061464+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:14:13.643726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.35","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.062379+0000","last_change":"2023-10-19T14:01:39.075978+0000","last_active":"2023-10-19T14:01:40.062379+0000","last_peered":"2023-10-19T14:01:40.062379+0000","last_clean":"2023-10-19T14:01:40.062379+0000","last_became_active":"2023-10-19T14:01:39.075623+0000","last_became_peered":"2023-10-19T14:01:39.075623+0000","last_unstale":"2023-10-19T14:01:40.062379+0000","last_undegraded":"2023-10-19T14:01:40.062379+0000","last_fullsized":"2023-10-19T14:01:40.062379+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:23:46.731396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.34","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.242603+0000","last_change":"2023-10-19T14:01:39.077218+0000","last_active":"2023-10-19T14:01:40.242603+0000","last_peered":"2023-10-19T14:01:40.242603+0000","last_clean":"2023-10-19T14:01:40.242603+0000","last_became_active":"2023-10-19T14:01:39.076680+0000","last_became_peered":"2023-10-19T14:01:39.076680+0000","last_unstale":"2023-10-19T14:01:40.242603+0000","last_undegraded":"2023-10-19T14:01:40.242603+0000","last_fullsized":"2023-10-19T14:01:40.242603+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:52:50.282449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.33","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.061857+0000","last_change":"2023-10-19T14:01:39.077328+0000","last_active":"2023-10-19T14:01:40.061857+0000","last_peered":"2023-10-19T14:01:40.061857+0000","last_clean":"2023-10-19T14:01:40.061857+0000","last_became_active":"2023-10-19T14:01:39.075953+0000","last_became_peered":"2023-10-19T14:01:39.075953+0000","last_unstale":"2023-10-19T14:01:40.061857+0000","last_undegraded":"2023-10-19T14:01:40.061857+0000","last_fullsized":"2023-10-19T14:01:40.061857+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:48:15.769444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.32","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.243278+0000","last_change":"2023-10-19T14:01:39.077498+0000","last_active":"2023-10-19T14:01:40.243278+0000","last_peered":"2023-10-19T14:01:40.243278+0000","last_clean":"2023-10-19T14:01:40.243278+0000","last_became_active":"2023-10-19T14:01:39.077146+0000","last_became_peered":"2023-10-19T14:01:39.077146+0000","last_unstale":"2023-10-19T14:01:40.243278+0000","last_undegraded":"2023-10-19T14:01:40.243278+0000","last_fullsized":"2023-10-19T14:01:40.243278+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:34:11.454508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.31","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.240956+0000","last_change":"2023-10-19T14:01:39.081418+0000","last_active":"2023-10-19T14:01:40.240956+0000","last_peered":"2023-10-19T14:01:40.240956+0000","last_clean":"2023-10-19T14:01:40.240956+0000","last_became_active":"2023-10-19T14:01:39.081051+0000","last_became_peered":"2023-10-19T14:01:39.081051+0000","last_unstale":"2023-10-19T14:01:40.240956+0000","last_undegraded":"2023-10-19T14:01:40.240956+0000","last_fullsized":"2023-10-19T14:01:40.240956+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:43:05.627474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.30","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.061249+0000","last_change":"2023-10-19T14:01:39.075843+0000","last_active":"2023-10-19T14:01:40.061249+0000","last_peered":"2023-10-19T14:01:40.061249+0000","last_clean":"2023-10-19T14:01:40.061249+0000","last_became_active":"2023-10-19T14:01:39.075386+0000","last_became_peered":"2023-10-19T14:01:39.075386+0000","last_unstale":"2023-10-19T14:01:40.061249+0000","last_undegraded":"2023-10-19T14:01:40.061249+0000","last_fullsized":"2023-10-19T14:01:40.061249+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:47:13.388172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.2f","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.243015+0000","last_change":"2023-10-19T14:01:39.076284+0000","last_active":"2023-10-19T14:01:40.243015+0000","last_peered":"2023-10-19T14:01:40.243015+0000","last_clean":"2023-10-19T14:01:40.243015+0000","last_became_active":"2023-10-19T14:01:39.075965+0000","last_became_peered":"2023-10-19T14:01:39.075965+0000","last_unstale":"2023-10-19T14:01:40.243015+0000","last_undegraded":"2023-10-19T14:01:40.243015+0000","last_fullsized":"2023-10-19T14:01:40.243015+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:07:41.757569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.2e","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.062463+0000","last_change":"2023-10-19T14:01:39.067100+0000","last_active":"2023-10-19T14:01:40.062463+0000","last_peered":"2023-10-19T14:01:40.062463+0000","last_clean":"2023-10-19T14:01:40.062463+0000","last_became_active":"2023-10-19T14:01:39.066796+0000","last_became_peered":"2023-10-19T14:01:39.066796+0000","last_unstale":"2023-10-19T14:01:40.062463+0000","last_undegraded":"2023-10-19T14:01:40.062463+0000","last_fullsized":"2023-10-19T14:01:40.062463+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:25:36.367891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.2d","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.243360+0000","last_change":"2023-10-19T14:01:39.082921+0000","last_active":"2023-10-19T14:01:40.243360+0000","last_peered":"2023-10-19T14:01:40.243360+0000","last_clean":"2023-10-19T14:01:40.243360+0000","last_became_active":"2023-10-19T14:01:39.082089+0000","last_became_peered":"2023-10-19T14:01:39.082089+0000","last_unstale":"2023-10-19T14:01:40.243360+0000","last_undegraded":"2023-10-19T14:01:40.243360+0000","last_fullsized":"2023-10-19T14:01:40.243360+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:54:37.937188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.2c","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.193612+0000","last_change":"2023-10-19T14:01:39.066830+0000","last_active":"2023-10-19T14:01:40.193612+0000","last_peered":"2023-10-19T14:01:40.193612+0000","last_clean":"2023-10-19T14:01:40.193612+0000","last_became_active":"2023-10-19T14:01:39.066556+0000","last_became_peered":"2023-10-19T14:01:39.066556+0000","last_unstale":"2023-10-19T14:01:40.193612+0000","last_undegraded":"2023-10-19T14:01:40.193612+0000","last_fullsized":"2023-10-19T14:01:40.193612+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:33:15.193395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.2b","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.061698+0000","last_change":"2023-10-19T14:01:39.077198+0000","last_active":"2023-10-19T14:01:40.061698+0000","last_peered":"2023-10-19T14:01:40.061698+0000","last_clean":"2023-10-19T14:01:40.061698+0000","last_became_active":"2023-10-19T14:01:39.076809+0000","last_became_peered":"2023-10-19T14:01:39.076809+0000","last_unstale":"2023-10-19T14:01:40.061698+0000","last_undegraded":"2023-10-19T14:01:40.061698+0000","last_fullsized":"2023-10-19T14:01:40.061698+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:53:06.193155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.2a","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.061339+0000","last_change":"2023-10-19T14:01:39.074528+0000","last_active":"2023-10-19T14:01:40.061339+0000","last_peered":"2023-10-19T14:01:40.061339+0000","last_clean":"2023-10-19T14:01:40.061339+0000","last_became_active":"2023-10-19T14:01:39.074278+0000","last_became_peered":"2023-10-19T14:01:39.074278+0000","last_unstale":"2023-10-19T14:01:40.061339+0000","last_undegraded":"2023-10-19T14:01:40.061339+0000","last_fullsized":"2023-10-19T14:01:40.061339+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:11:47.227977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.29","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.242695+0000","last_change":"2023-10-19T14:01:39.069413+0000","last_active":"2023-10-19T14:01:40.242695+0000","last_peered":"2023-10-19T14:01:40.242695+0000","last_clean":"2023-10-19T14:01:40.242695+0000","last_became_active":"2023-10-19T14:01:39.068967+0000","last_became_peered":"2023-10-19T14:01:39.068967+0000","last_unstale":"2023-10-19T14:01:40.242695+0000","last_undegraded":"2023-10-19T14:01:40.242695+0000","last_fullsized":"2023-10-19T14:01:40.242695+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:46:49.160506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.28","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.062627+0000","last_change":"2023-10-19T14:01:39.075350+0000","last_active":"2023-10-19T14:01:40.062627+0000","last_peered":"2023-10-19T14:01:40.062627+0000","last_clean":"2023-10-19T14:01:40.062627+0000","last_became_active":"2023-10-19T14:01:39.074936+0000","last_became_peered":"2023-10-19T14:01:39.074936+0000","last_unstale":"2023-10-19T14:01:40.062627+0000","last_undegraded":"2023-10-19T14:01:40.062627+0000","last_fullsized":"2023-10-19T14:01:40.062627+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:32:59.224807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.27","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.061670+0000","last_change":"2023-10-19T14:01:39.069828+0000","last_active":"2023-10-19T14:01:40.061670+0000","last_peered":"2023-10-19T14:01:40.061670+0000","last_clean":"2023-10-19T14:01:40.061670+0000","last_became_active":"2023-10-19T14:01:39.069531+0000","last_became_peered":"2023-10-19T14:01:39.069531+0000","last_unstale":"2023-10-19T14:01:40.061670+0000","last_undegraded":"2023-10-19T14:01:40.061670+0000","last_fullsized":"2023-10-19T14:01:40.061670+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:33:57.927060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.26","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.061556+0000","last_change":"2023-10-19T14:01:39.069124+0000","last_active":"2023-10-19T14:01:40.061556+0000","last_peered":"2023-10-19T14:01:40.061556+0000","last_clean":"2023-10-19T14:01:40.061556+0000","last_became_active":"2023-10-19T14:01:39.067572+0000","last_became_peered":"2023-10-19T14:01:39.067572+0000","last_unstale":"2023-10-19T14:01:40.061556+0000","last_undegraded":"2023-10-19T14:01:40.061556+0000","last_fullsized":"2023-10-19T14:01:40.061556+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:27:12.855540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.25","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.061502+0000","last_change":"2023-10-19T14:01:39.075348+0000","last_active":"2023-10-19T14:01:40.061502+0000","last_peered":"2023-10-19T14:01:40.061502+0000","last_clean":"2023-10-19T14:01:40.061502+0000","last_became_active":"2023-10-19T14:01:39.075053+0000","last_became_peered":"2023-10-19T14:01:39.075053+0000","last_unstale":"2023-10-19T14:01:40.061502+0000","last_undegraded":"2023-10-19T14:01:40.061502+0000","last_fullsized":"2023-10-19T14:01:40.061502+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:32:12.211178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.24","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.062677+0000","last_change":"2023-10-19T14:01:39.064768+0000","last_active":"2023-10-19T14:01:40.062677+0000","last_peered":"2023-10-19T14:01:40.062677+0000","last_clean":"2023-10-19T14:01:40.062677+0000","last_became_active":"2023-10-19T14:01:39.064331+0000","last_became_peered":"2023-10-19T14:01:39.064331+0000","last_unstale":"2023-10-19T14:01:40.062677+0000","last_undegraded":"2023-10-19T14:01:40.062677+0000","last_fullsized":"2023-10-19T14:01:40.062677+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:21:33.111873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.23","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.242991+0000","last_change":"2023-10-19T14:01:39.075985+0000","last_active":"2023-10-19T14:01:40.242991+0000","last_peered":"2023-10-19T14:01:40.242991+0000","last_clean":"2023-10-19T14:01:40.242991+0000","last_became_active":"2023-10-19T14:01:39.075344+0000","last_became_peered":"2023-10-19T14:01:39.075344+0000","last_unstale":"2023-10-19T14:01:40.242991+0000","last_undegraded":"2023-10-19T14:01:40.242991+0000","last_fullsized":"2023-10-19T14:01:40.242991+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:00:24.365723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.22","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.062576+0000","last_change":"2023-10-19T14:01:39.078224+0000","last_active":"2023-10-19T14:01:40.062576+0000","last_peered":"2023-10-19T14:01:40.062576+0000","last_clean":"2023-10-19T14:01:40.062576+0000","last_became_active":"2023-10-19T14:01:39.077926+0000","last_became_peered":"2023-10-19T14:01:39.077926+0000","last_unstale":"2023-10-19T14:01:40.062576+0000","last_undegraded":"2023-10-19T14:01:40.062576+0000","last_fullsized":"2023-10-19T14:01:40.062576+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:19:01.846781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.21","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.061377+0000","last_change":"2023-10-19T14:01:39.081882+0000","last_active":"2023-10-19T14:01:40.061377+0000","last_peered":"2023-10-19T14:01:40.061377+0000","last_clean":"2023-10-19T14:01:40.061377+0000","last_became_active":"2023-10-19T14:01:39.080530+0000","last_became_peered":"2023-10-19T14:01:39.080530+0000","last_unstale":"2023-10-19T14:01:40.061377+0000","last_undegraded":"2023-10-19T14:01:40.061377+0000","last_fullsized":"2023-10-19T14:01:40.061377+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:30:23.694882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.20","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.242096+0000","last_change":"2023-10-19T14:01:39.073673+0000","last_active":"2023-10-19T14:01:40.242096+0000","last_peered":"2023-10-19T14:01:40.242096+0000","last_clean":"2023-10-19T14:01:40.242096+0000","last_became_active":"2023-10-19T14:01:39.072708+0000","last_became_peered":"2023-10-19T14:01:39.072708+0000","last_unstale":"2023-10-19T14:01:40.242096+0000","last_undegraded":"2023-10-19T14:01:40.242096+0000","last_fullsized":"2023-10-19T14:01:40.242096+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:53:28.733545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1c","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.241224+0000","last_change":"2023-10-19T14:01:39.074536+0000","last_active":"2023-10-19T14:01:40.241224+0000","last_peered":"2023-10-19T14:01:40.241224+0000","last_clean":"2023-10-19T14:01:40.241224+0000","last_became_active":"2023-10-19T14:01:39.074257+0000","last_became_peered":"2023-10-19T14:01:39.074257+0000","last_unstale":"2023-10-19T14:01:40.241224+0000","last_undegraded":"2023-10-19T14:01:40.241224+0000","last_fullsized":"2023-10-19T14:01:40.241224+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:53:12.668683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1f","version":"174'4","reported_seq":16,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.061780+0000","last_change":"2023-10-19T14:01:38.149463+0000","last_active":"2023-10-19T14:01:40.061780+0000","last_peered":"2023-10-19T14:01:40.061780+0000","last_clean":"2023-10-19T14:01:40.061780+0000","last_became_active":"2023-10-19T14:01:38.149187+0000","last_became_peered":"2023-10-19T14:01:38.149187+0000","last_unstale":"2023-10-19T14:01:40.061780+0000","last_undegraded":"2023-10-19T14:01:40.061780+0000","last_fullsized":"2023-10-19T14:01:40.061780+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_clean_scrub_stamp":"2023-10-19T14:01:37.123936+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:32:28.166385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1d","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.241150+0000","last_change":"2023-10-19T14:01:39.080424+0000","last_active":"2023-10-19T14:01:40.241150+0000","last_peered":"2023-10-19T14:01:40.241150+0000","last_clean":"2023-10-19T14:01:40.241150+0000","last_became_active":"2023-10-19T14:01:39.080138+0000","last_became_peered":"2023-10-19T14:01:39.080138+0000","last_unstale":"2023-10-19T14:01:40.241150+0000","last_undegraded":"2023-10-19T14:01:40.241150+0000","last_fullsized":"2023-10-19T14:01:40.241150+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:08:22.714158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1e","version":"174'1","reported_seq":13,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.243017+0000","last_change":"2023-10-19T14:01:38.155102+0000","last_active":"2023-10-19T14:01:40.243017+0000","last_peered":"2023-10-19T14:01:40.243017+0000","last_clean":"2023-10-19T14:01:40.243017+0000","last_became_active":"2023-10-19T14:01:38.154849+0000","last_became_peered":"2023-10-19T14:01:38.154849+0000","last_unstale":"2023-10-19T14:01:40.243017+0000","last_undegraded":"2023-10-19T14:01:40.243017+0000","last_fullsized":"2023-10-19T14:01:40.243017+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_clean_scrub_stamp":"2023-10-19T14:01:37.123936+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:27:11.849330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1e","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.242554+0000","last_change":"2023-10-19T14:01:39.079726+0000","last_active":"2023-10-19T14:01:40.242554+0000","last_peered":"2023-10-19T14:01:40.242554+0000","last_clean":"2023-10-19T14:01:40.242554+0000","last_became_active":"2023-10-19T14:01:39.079439+0000","last_became_peered":"2023-10-19T14:01:39.079439+0000","last_unstale":"2023-10-19T14:01:40.242554+0000","last_undegraded":"2023-10-19T14:01:40.242554+0000","last_fullsized":"2023-10-19T14:01:40.242554+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:10:18.170429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1d","version":"174'2","reported_seq":14,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.243437+0000","last_change":"2023-10-19T14:01:38.154123+0000","last_active":"2023-10-19T14:01:40.243437+0000","last_peered":"2023-10-19T14:01:40.243437+0000","last_clean":"2023-10-19T14:01:40.243437+0000","last_became_active":"2023-10-19T14:01:38.153860+0000","last_became_peered":"2023-10-19T14:01:38.153860+0000","last_unstale":"2023-10-19T14:01:40.243437+0000","last_undegraded":"2023-10-19T14:01:40.243437+0000","last_fullsized":"2023-10-19T14:01:40.243437+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_clean_scrub_stamp":"2023-10-19T14:01:37.123936+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:25:57.506648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1f","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.243438+0000","last_change":"2023-10-19T14:01:39.081167+0000","last_active":"2023-10-19T14:01:40.243438+0000","last_peered":"2023-10-19T14:01:40.243438+0000","last_clean":"2023-10-19T14:01:40.243438+0000","last_became_active":"2023-10-19T14:01:39.080131+0000","last_became_peered":"2023-10-19T14:01:39.080131+0000","last_unstale":"2023-10-19T14:01:40.243438+0000","last_undegraded":"2023-10-19T14:01:40.243438+0000","last_fullsized":"2023-10-19T14:01:40.243438+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T02:00:46.582226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1c","version":"174'2","reported_seq":14,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.241213+0000","last_change":"2023-10-19T14:01:38.154468+0000","last_active":"2023-10-19T14:01:40.241213+0000","last_peered":"2023-10-19T14:01:40.241213+0000","last_clean":"2023-10-19T14:01:40.241213+0000","last_became_active":"2023-10-19T14:01:38.154132+0000","last_became_peered":"2023-10-19T14:01:38.154132+0000","last_unstale":"2023-10-19T14:01:40.241213+0000","last_undegraded":"2023-10-19T14:01:40.241213+0000","last_fullsized":"2023-10-19T14:01:40.241213+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_clean_scrub_stamp":"2023-10-19T14:01:37.123936+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:57:34.773001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.193615+0000","last_change":"2023-10-19T14:01:39.066422+0000","last_active":"2023-10-19T14:01:40.193615+0000","last_peered":"2023-10-19T14:01:40.193615+0000","last_clean":"2023-10-19T14:01:40.193615+0000","last_became_active":"2023-10-19T14:01:39.065943+0000","last_became_peered":"2023-10-19T14:01:39.065943+0000","last_unstale":"2023-10-19T14:01:40.193615+0000","last_undegraded":"2023-10-19T14:01:40.193615+0000","last_fullsized":"2023-10-19T14:01:40.193615+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:02:51.348060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.1b","version":"174'1","reported_seq":13,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.243005+0000","last_change":"2023-10-19T14:01:38.154709+0000","last_active":"2023-10-19T14:01:40.243005+0000","last_peered":"2023-10-19T14:01:40.243005+0000","last_clean":"2023-10-19T14:01:40.243005+0000","last_became_active":"2023-10-19T14:01:38.153296+0000","last_became_peered":"2023-10-19T14:01:38.153296+0000","last_unstale":"2023-10-19T14:01:40.243005+0000","last_undegraded":"2023-10-19T14:01:40.243005+0000","last_fullsized":"2023-10-19T14:01:40.243005+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_clean_scrub_stamp":"2023-10-19T14:01:37.123936+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:10:51.292543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.241912+0000","last_change":"2023-10-19T14:01:39.076795+0000","last_active":"2023-10-19T14:01:40.241912+0000","last_peered":"2023-10-19T14:01:40.241912+0000","last_clean":"2023-10-19T14:01:40.241912+0000","last_became_active":"2023-10-19T14:01:39.076254+0000","last_became_peered":"2023-10-19T14:01:39.076254+0000","last_unstale":"2023-10-19T14:01:40.241912+0000","last_undegraded":"2023-10-19T14:01:40.241912+0000","last_fullsized":"2023-10-19T14:01:40.241912+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:14:16.050479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1a","version":"174'1","reported_seq":13,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.061708+0000","last_change":"2023-10-19T14:01:38.153181+0000","last_active":"2023-10-19T14:01:40.061708+0000","last_peered":"2023-10-19T14:01:40.061708+0000","last_clean":"2023-10-19T14:01:40.061708+0000","last_became_active":"2023-10-19T14:01:38.152855+0000","last_became_peered":"2023-10-19T14:01:38.152855+0000","last_unstale":"2023-10-19T14:01:40.061708+0000","last_undegraded":"2023-10-19T14:01:40.061708+0000","last_fullsized":"2023-10-19T14:01:40.061708+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_clean_scrub_stamp":"2023-10-19T14:01:37.123936+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:20:42.107511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1a","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.240973+0000","last_change":"2023-10-19T14:01:39.067123+0000","last_active":"2023-10-19T14:01:40.240973+0000","last_peered":"2023-10-19T14:01:40.240973+0000","last_clean":"2023-10-19T14:01:40.240973+0000","last_became_active":"2023-10-19T14:01:39.066843+0000","last_became_peered":"2023-10-19T14:01:39.066843+0000","last_unstale":"2023-10-19T14:01:40.240973+0000","last_undegraded":"2023-10-19T14:01:40.240973+0000","last_fullsized":"2023-10-19T14:01:40.240973+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:28:31.752973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":12,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.061406+0000","last_change":"2023-10-19T14:01:38.148197+0000","last_active":"2023-10-19T14:01:40.061406+0000","last_peered":"2023-10-19T14:01:40.061406+0000","last_clean":"2023-10-19T14:01:40.061406+0000","last_became_active":"2023-10-19T14:01:38.147899+0000","last_became_peered":"2023-10-19T14:01:38.147899+0000","last_unstale":"2023-10-19T14:01:40.061406+0000","last_undegraded":"2023-10-19T14:01:40.061406+0000","last_fullsized":"2023-10-19T14:01:40.061406+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_clean_scrub_stamp":"2023-10-19T14:01:37.123936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:09:27.370187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1b","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.061374+0000","last_change":"2023-10-19T14:01:39.073790+0000","last_active":"2023-10-19T14:01:40.061374+0000","last_peered":"2023-10-19T14:01:40.061374+0000","last_clean":"2023-10-19T14:01:40.061374+0000","last_became_active":"2023-10-19T14:01:39.073530+0000","last_became_peered":"2023-10-19T14:01:39.073530+0000","last_unstale":"2023-10-19T14:01:40.061374+0000","last_undegraded":"2023-10-19T14:01:40.061374+0000","last_fullsized":"2023-10-19T14:01:40.061374+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:04:24.165469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":12,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.061278+0000","last_change":"2023-10-19T14:01:38.149972+0000","last_active":"2023-10-19T14:01:40.061278+0000","last_peered":"2023-10-19T14:01:40.061278+0000","last_clean":"2023-10-19T14:01:40.061278+0000","last_became_active":"2023-10-19T14:01:38.149689+0000","last_became_peered":"2023-10-19T14:01:38.149689+0000","last_unstale":"2023-10-19T14:01:40.061278+0000","last_undegraded":"2023-10-19T14:01:40.061278+0000","last_fullsized":"2023-10-19T14:01:40.061278+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_clean_scrub_stamp":"2023-10-19T14:01:37.123936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:28:56.955098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.14","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.243095+0000","last_change":"2023-10-19T14:01:39.074490+0000","last_active":"2023-10-19T14:01:40.243095+0000","last_peered":"2023-10-19T14:01:40.243095+0000","last_clean":"2023-10-19T14:01:40.243095+0000","last_became_active":"2023-10-19T14:01:39.073165+0000","last_became_peered":"2023-10-19T14:01:39.073165+0000","last_unstale":"2023-10-19T14:01:40.243095+0000","last_undegraded":"2023-10-19T14:01:40.243095+0000","last_fullsized":"2023-10-19T14:01:40.243095+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:10:47.398271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":12,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.243541+0000","last_change":"2023-10-19T14:01:38.153657+0000","last_active":"2023-10-19T14:01:40.243541+0000","last_peered":"2023-10-19T14:01:40.243541+0000","last_clean":"2023-10-19T14:01:40.243541+0000","last_became_active":"2023-10-19T14:01:38.153388+0000","last_became_peered":"2023-10-19T14:01:38.153388+0000","last_unstale":"2023-10-19T14:01:40.243541+0000","last_undegraded":"2023-10-19T14:01:40.243541+0000","last_fullsized":"2023-10-19T14:01:40.243541+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_clean_scrub_stamp":"2023-10-19T14:01:37.123936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:11:46.221768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.15","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.062503+0000","last_change":"2023-10-19T14:01:39.075641+0000","last_active":"2023-10-19T14:01:40.062503+0000","last_peered":"2023-10-19T14:01:40.062503+0000","last_clean":"2023-10-19T14:01:40.062503+0000","last_became_active":"2023-10-19T14:01:39.075259+0000","last_became_peered":"2023-10-19T14:01:39.075259+0000","last_unstale":"2023-10-19T14:01:40.062503+0000","last_undegraded":"2023-10-19T14:01:40.062503+0000","last_fullsized":"2023-10-19T14:01:40.062503+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:24:46.342060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":12,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.061731+0000","last_change":"2023-10-19T14:01:38.149138+0000","last_active":"2023-10-19T14:01:40.061731+0000","last_peered":"2023-10-19T14:01:40.061731+0000","last_clean":"2023-10-19T14:01:40.061731+0000","last_became_active":"2023-10-19T14:01:38.148787+0000","last_became_peered":"2023-10-19T14:01:38.148787+0000","last_unstale":"2023-10-19T14:01:40.061731+0000","last_undegraded":"2023-10-19T14:01:40.061731+0000","last_fullsized":"2023-10-19T14:01:40.061731+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_clean_scrub_stamp":"2023-10-19T14:01:37.123936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:09:19.791765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.062605+0000","last_change":"2023-10-19T14:01:39.066640+0000","last_active":"2023-10-19T14:01:40.062605+0000","last_peered":"2023-10-19T14:01:40.062605+0000","last_clean":"2023-10-19T14:01:40.062605+0000","last_became_active":"2023-10-19T14:01:39.066237+0000","last_became_peered":"2023-10-19T14:01:39.066237+0000","last_unstale":"2023-10-19T14:01:40.062605+0000","last_undegraded":"2023-10-19T14:01:40.062605+0000","last_fullsized":"2023-10-19T14:01:40.062605+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:04:01.542818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.15","version":"174'3","reported_seq":15,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.062616+0000","last_change":"2023-10-19T14:01:38.151310+0000","last_active":"2023-10-19T14:01:40.062616+0000","last_peered":"2023-10-19T14:01:40.062616+0000","last_clean":"2023-10-19T14:01:40.062616+0000","last_became_active":"2023-10-19T14:01:38.150998+0000","last_became_peered":"2023-10-19T14:01:38.150998+0000","last_unstale":"2023-10-19T14:01:40.062616+0000","last_undegraded":"2023-10-19T14:01:40.062616+0000","last_fullsized":"2023-10-19T14:01:40.062616+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_clean_scrub_stamp":"2023-10-19T14:01:37.123936+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:05:26.982257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.10","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.242191+0000","last_change":"2023-10-19T14:01:39.078223+0000","last_active":"2023-10-19T14:01:40.242191+0000","last_peered":"2023-10-19T14:01:40.242191+0000","last_clean":"2023-10-19T14:01:40.242191+0000","last_became_active":"2023-10-19T14:01:39.077960+0000","last_became_peered":"2023-10-19T14:01:39.077960+0000","last_unstale":"2023-10-19T14:01:40.242191+0000","last_undegraded":"2023-10-19T14:01:40.242191+0000","last_fullsized":"2023-10-19T14:01:40.242191+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:40:26.710084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.13","version":"174'2","reported_seq":14,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.061914+0000","last_change":"2023-10-19T14:01:38.150112+0000","last_active":"2023-10-19T14:01:40.061914+0000","last_peered":"2023-10-19T14:01:40.061914+0000","last_clean":"2023-10-19T14:01:40.061914+0000","last_became_active":"2023-10-19T14:01:38.149218+0000","last_became_peered":"2023-10-19T14:01:38.149218+0000","last_unstale":"2023-10-19T14:01:40.061914+0000","last_undegraded":"2023-10-19T14:01:40.061914+0000","last_fullsized":"2023-10-19T14:01:40.061914+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_clean_scrub_stamp":"2023-10-19T14:01:37.123936+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:31:42.990421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.241235+0000","last_change":"2023-10-19T14:01:39.081126+0000","last_active":"2023-10-19T14:01:40.241235+0000","last_peered":"2023-10-19T14:01:40.241235+0000","last_clean":"2023-10-19T14:01:40.241235+0000","last_became_active":"2023-10-19T14:01:39.080597+0000","last_became_peered":"2023-10-19T14:01:39.080597+0000","last_unstale":"2023-10-19T14:01:40.241235+0000","last_undegraded":"2023-10-19T14:01:40.241235+0000","last_fullsized":"2023-10-19T14:01:40.241235+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:14:21.486364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":12,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.242645+0000","last_change":"2023-10-19T14:01:38.156939+0000","last_active":"2023-10-19T14:01:40.242645+0000","last_peered":"2023-10-19T14:01:40.242645+0000","last_clean":"2023-10-19T14:01:40.242645+0000","last_became_active":"2023-10-19T14:01:38.156654+0000","last_became_peered":"2023-10-19T14:01:38.156654+0000","last_unstale":"2023-10-19T14:01:40.242645+0000","last_undegraded":"2023-10-19T14:01:40.242645+0000","last_fullsized":"2023-10-19T14:01:40.242645+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_clean_scrub_stamp":"2023-10-19T14:01:37.123936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:27:32.578234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.061693+0000","last_change":"2023-10-19T14:01:39.072644+0000","last_active":"2023-10-19T14:01:40.061693+0000","last_peered":"2023-10-19T14:01:40.061693+0000","last_clean":"2023-10-19T14:01:40.061693+0000","last_became_active":"2023-10-19T14:01:39.072301+0000","last_became_peered":"2023-10-19T14:01:39.072301+0000","last_unstale":"2023-10-19T14:01:40.061693+0000","last_undegraded":"2023-10-19T14:01:40.061693+0000","last_fullsized":"2023-10-19T14:01:40.061693+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:49:07.317364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":12,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.242002+0000","last_change":"2023-10-19T14:01:38.152957+0000","last_active":"2023-10-19T14:01:40.242002+0000","last_peered":"2023-10-19T14:01:40.242002+0000","last_clean":"2023-10-19T14:01:40.242002+0000","last_became_active":"2023-10-19T14:01:38.152708+0000","last_became_peered":"2023-10-19T14:01:38.152708+0000","last_unstale":"2023-10-19T14:01:40.242002+0000","last_undegraded":"2023-10-19T14:01:40.242002+0000","last_fullsized":"2023-10-19T14:01:40.242002+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_clean_scrub_stamp":"2023-10-19T14:01:37.123936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:43:47.404093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.13","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.240990+0000","last_change":"2023-10-19T14:01:39.074287+0000","last_active":"2023-10-19T14:01:40.240990+0000","last_peered":"2023-10-19T14:01:40.240990+0000","last_clean":"2023-10-19T14:01:40.240990+0000","last_became_active":"2023-10-19T14:01:39.073962+0000","last_became_peered":"2023-10-19T14:01:39.073962+0000","last_unstale":"2023-10-19T14:01:40.240990+0000","last_undegraded":"2023-10-19T14:01:40.240990+0000","last_fullsized":"2023-10-19T14:01:40.240990+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:59:15.579667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.10","version":"174'8","reported_seq":20,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.061096+0000","last_change":"2023-10-19T14:01:38.150008+0000","last_active":"2023-10-19T14:01:40.061096+0000","last_peered":"2023-10-19T14:01:40.061096+0000","last_clean":"2023-10-19T14:01:40.061096+0000","last_became_active":"2023-10-19T14:01:38.149708+0000","last_became_peered":"2023-10-19T14:01:38.149708+0000","last_unstale":"2023-10-19T14:01:40.061096+0000","last_undegraded":"2023-10-19T14:01:40.061096+0000","last_fullsized":"2023-10-19T14:01:40.061096+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_clean_scrub_stamp":"2023-10-19T14:01:37.123936+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:22:09.761426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.242341+0000","last_change":"2023-10-19T14:01:39.076798+0000","last_active":"2023-10-19T14:01:40.242341+0000","last_peered":"2023-10-19T14:01:40.242341+0000","last_clean":"2023-10-19T14:01:40.242341+0000","last_became_active":"2023-10-19T14:01:39.075966+0000","last_became_peered":"2023-10-19T14:01:39.075966+0000","last_unstale":"2023-10-19T14:01:40.242341+0000","last_undegraded":"2023-10-19T14:01:40.242341+0000","last_fullsized":"2023-10-19T14:01:40.242341+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:28:46.198601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.f","version":"174'1","reported_seq":13,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.062467+0000","last_change":"2023-10-19T14:01:38.150116+0000","last_active":"2023-10-19T14:01:40.062467+0000","last_peered":"2023-10-19T14:01:40.062467+0000","last_clean":"2023-10-19T14:01:40.062467+0000","last_became_active":"2023-10-19T14:01:38.149778+0000","last_became_peered":"2023-10-19T14:01:38.149778+0000","last_unstale":"2023-10-19T14:01:40.062467+0000","last_undegraded":"2023-10-19T14:01:40.062467+0000","last_fullsized":"2023-10-19T14:01:40.062467+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_clean_scrub_stamp":"2023-10-19T14:01:37.123936+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:46:09.489532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.d","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.243391+0000","last_change":"2023-10-19T14:01:39.082935+0000","last_active":"2023-10-19T14:01:40.243391+0000","last_peered":"2023-10-19T14:01:40.243391+0000","last_clean":"2023-10-19T14:01:40.243391+0000","last_became_active":"2023-10-19T14:01:39.082359+0000","last_became_peered":"2023-10-19T14:01:39.082359+0000","last_unstale":"2023-10-19T14:01:40.243391+0000","last_undegraded":"2023-10-19T14:01:40.243391+0000","last_fullsized":"2023-10-19T14:01:40.243391+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:29:58.142789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.e","version":"174'2","reported_seq":14,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.243125+0000","last_change":"2023-10-19T14:01:38.153428+0000","last_active":"2023-10-19T14:01:40.243125+0000","last_peered":"2023-10-19T14:01:40.243125+0000","last_clean":"2023-10-19T14:01:40.243125+0000","last_became_active":"2023-10-19T14:01:38.153155+0000","last_became_peered":"2023-10-19T14:01:38.153155+0000","last_unstale":"2023-10-19T14:01:40.243125+0000","last_undegraded":"2023-10-19T14:01:40.243125+0000","last_fullsized":"2023-10-19T14:01:40.243125+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_clean_scrub_stamp":"2023-10-19T14:01:37.123936+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:20:27.265338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.e","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.061659+0000","last_change":"2023-10-19T14:01:39.078154+0000","last_active":"2023-10-19T14:01:40.061659+0000","last_peered":"2023-10-19T14:01:40.061659+0000","last_clean":"2023-10-19T14:01:40.061659+0000","last_became_active":"2023-10-19T14:01:39.077803+0000","last_became_peered":"2023-10-19T14:01:39.077803+0000","last_unstale":"2023-10-19T14:01:40.061659+0000","last_undegraded":"2023-10-19T14:01:40.061659+0000","last_fullsized":"2023-10-19T14:01:40.061659+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:17:07.521687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.d","version":"174'1","reported_seq":13,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.241171+0000","last_change":"2023-10-19T14:01:38.153632+0000","last_active":"2023-10-19T14:01:40.241171+0000","last_peered":"2023-10-19T14:01:40.241171+0000","last_clean":"2023-10-19T14:01:40.241171+0000","last_became_active":"2023-10-19T14:01:38.153374+0000","last_became_peered":"2023-10-19T14:01:38.153374+0000","last_unstale":"2023-10-19T14:01:40.241171+0000","last_undegraded":"2023-10-19T14:01:40.241171+0000","last_fullsized":"2023-10-19T14:01:40.241171+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_clean_scrub_stamp":"2023-10-19T14:01:37.123936+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:19:21.478658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.f","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.061761+0000","last_change":"2023-10-19T14:01:39.078701+0000","last_active":"2023-10-19T14:01:40.061761+0000","last_peered":"2023-10-19T14:01:40.061761+0000","last_clean":"2023-10-19T14:01:40.061761+0000","last_became_active":"2023-10-19T14:01:39.078452+0000","last_became_peered":"2023-10-19T14:01:39.078452+0000","last_unstale":"2023-10-19T14:01:40.061761+0000","last_undegraded":"2023-10-19T14:01:40.061761+0000","last_fullsized":"2023-10-19T14:01:40.061761+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:46:36.953152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":12,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.242704+0000","last_change":"2023-10-19T14:01:38.156924+0000","last_active":"2023-10-19T14:01:40.242704+0000","last_peered":"2023-10-19T14:01:40.242704+0000","last_clean":"2023-10-19T14:01:40.242704+0000","last_became_active":"2023-10-19T14:01:38.156662+0000","last_became_peered":"2023-10-19T14:01:38.156662+0000","last_unstale":"2023-10-19T14:01:40.242704+0000","last_undegraded":"2023-10-19T14:01:40.242704+0000","last_fullsized":"2023-10-19T14:01:40.242704+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_clean_scrub_stamp":"2023-10-19T14:01:37.123936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:46:52.638213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.241935+0000","last_change":"2023-10-19T14:01:39.080332+0000","last_active":"2023-10-19T14:01:40.241935+0000","last_peered":"2023-10-19T14:01:40.241935+0000","last_clean":"2023-10-19T14:01:40.241935+0000","last_became_active":"2023-10-19T14:01:39.079874+0000","last_became_peered":"2023-10-19T14:01:39.079874+0000","last_unstale":"2023-10-19T14:01:40.241935+0000","last_undegraded":"2023-10-19T14:01:40.241935+0000","last_fullsized":"2023-10-19T14:01:40.241935+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:29:32.305082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.b","version":"174'2","reported_seq":14,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.193612+0000","last_change":"2023-10-19T14:01:38.155709+0000","last_active":"2023-10-19T14:01:40.193612+0000","last_peered":"2023-10-19T14:01:40.193612+0000","last_clean":"2023-10-19T14:01:40.193612+0000","last_became_active":"2023-10-19T14:01:38.155447+0000","last_became_peered":"2023-10-19T14:01:38.155447+0000","last_unstale":"2023-10-19T14:01:40.193612+0000","last_undegraded":"2023-10-19T14:01:40.193612+0000","last_fullsized":"2023-10-19T14:01:40.193612+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_clean_scrub_stamp":"2023-10-19T14:01:37.123936+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:38:28.395781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.241104+0000","last_change":"2023-10-19T14:01:39.068151+0000","last_active":"2023-10-19T14:01:40.241104+0000","last_peered":"2023-10-19T14:01:40.241104+0000","last_clean":"2023-10-19T14:01:40.241104+0000","last_became_active":"2023-10-19T14:01:39.067791+0000","last_became_peered":"2023-10-19T14:01:39.067791+0000","last_unstale":"2023-10-19T14:01:40.241104+0000","last_undegraded":"2023-10-19T14:01:40.241104+0000","last_fullsized":"2023-10-19T14:01:40.241104+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:04:04.560169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":12,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.242215+0000","last_change":"2023-10-19T14:01:38.151527+0000","last_active":"2023-10-19T14:01:40.242215+0000","last_peered":"2023-10-19T14:01:40.242215+0000","last_clean":"2023-10-19T14:01:40.242215+0000","last_became_active":"2023-10-19T14:01:38.151259+0000","last_became_peered":"2023-10-19T14:01:38.151259+0000","last_unstale":"2023-10-19T14:01:40.242215+0000","last_undegraded":"2023-10-19T14:01:40.242215+0000","last_fullsized":"2023-10-19T14:01:40.242215+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_clean_scrub_stamp":"2023-10-19T14:01:37.123936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:33:19.331015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.242156+0000","last_change":"2023-10-19T14:01:39.081972+0000","last_active":"2023-10-19T14:01:40.242156+0000","last_peered":"2023-10-19T14:01:40.242156+0000","last_clean":"2023-10-19T14:01:40.242156+0000","last_became_active":"2023-10-19T14:01:39.081551+0000","last_became_peered":"2023-10-19T14:01:39.081551+0000","last_unstale":"2023-10-19T14:01:40.242156+0000","last_undegraded":"2023-10-19T14:01:40.242156+0000","last_fullsized":"2023-10-19T14:01:40.242156+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:55:59.374396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":12,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.193453+0000","last_change":"2023-10-19T14:01:38.154227+0000","last_active":"2023-10-19T14:01:40.193453+0000","last_peered":"2023-10-19T14:01:40.193453+0000","last_clean":"2023-10-19T14:01:40.193453+0000","last_became_active":"2023-10-19T14:01:38.153946+0000","last_became_peered":"2023-10-19T14:01:38.153946+0000","last_unstale":"2023-10-19T14:01:40.193453+0000","last_undegraded":"2023-10-19T14:01:40.193453+0000","last_fullsized":"2023-10-19T14:01:40.193453+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_clean_scrub_stamp":"2023-10-19T14:01:37.123936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:23:58.111924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.b","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.240979+0000","last_change":"2023-10-19T14:01:39.081245+0000","last_active":"2023-10-19T14:01:40.240979+0000","last_peered":"2023-10-19T14:01:40.240979+0000","last_clean":"2023-10-19T14:01:40.240979+0000","last_became_active":"2023-10-19T14:01:39.080944+0000","last_became_peered":"2023-10-19T14:01:39.080944+0000","last_unstale":"2023-10-19T14:01:40.240979+0000","last_undegraded":"2023-10-19T14:01:40.240979+0000","last_fullsized":"2023-10-19T14:01:40.240979+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:33:18.166098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":12,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.242431+0000","last_change":"2023-10-19T14:01:38.154468+0000","last_active":"2023-10-19T14:01:40.242431+0000","last_peered":"2023-10-19T14:01:40.242431+0000","last_clean":"2023-10-19T14:01:40.242431+0000","last_became_active":"2023-10-19T14:01:38.154203+0000","last_became_peered":"2023-10-19T14:01:38.154203+0000","last_unstale":"2023-10-19T14:01:40.242431+0000","last_undegraded":"2023-10-19T14:01:40.242431+0000","last_fullsized":"2023-10-19T14:01:40.242431+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_clean_scrub_stamp":"2023-10-19T14:01:37.123936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:14:53.849183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.061623+0000","last_change":"2023-10-19T14:01:39.080523+0000","last_active":"2023-10-19T14:01:40.061623+0000","last_peered":"2023-10-19T14:01:40.061623+0000","last_clean":"2023-10-19T14:01:40.061623+0000","last_became_active":"2023-10-19T14:01:39.079542+0000","last_became_peered":"2023-10-19T14:01:39.079542+0000","last_unstale":"2023-10-19T14:01:40.061623+0000","last_undegraded":"2023-10-19T14:01:40.061623+0000","last_fullsized":"2023-10-19T14:01:40.061623+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:00:06.386114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.7","version":"174'2","reported_seq":14,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.243109+0000","last_change":"2023-10-19T14:01:38.158093+0000","last_active":"2023-10-19T14:01:40.243109+0000","last_peered":"2023-10-19T14:01:40.243109+0000","last_clean":"2023-10-19T14:01:40.243109+0000","last_became_active":"2023-10-19T14:01:38.157861+0000","last_became_peered":"2023-10-19T14:01:38.157861+0000","last_unstale":"2023-10-19T14:01:40.243109+0000","last_undegraded":"2023-10-19T14:01:40.243109+0000","last_fullsized":"2023-10-19T14:01:40.243109+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_clean_scrub_stamp":"2023-10-19T14:01:37.123936+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:05:00.875663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.242829+0000","last_change":"2023-10-19T14:01:39.076188+0000","last_active":"2023-10-19T14:01:40.242829+0000","last_peered":"2023-10-19T14:01:40.242829+0000","last_clean":"2023-10-19T14:01:40.242829+0000","last_became_active":"2023-10-19T14:01:39.075908+0000","last_became_peered":"2023-10-19T14:01:39.075908+0000","last_unstale":"2023-10-19T14:01:40.242829+0000","last_undegraded":"2023-10-19T14:01:40.242829+0000","last_fullsized":"2023-10-19T14:01:40.242829+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:48:55.738928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.6","version":"0'0","reported_seq":12,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.061795+0000","last_change":"2023-10-19T14:01:38.149636+0000","last_active":"2023-10-19T14:01:40.061795+0000","last_peered":"2023-10-19T14:01:40.061795+0000","last_clean":"2023-10-19T14:01:40.061795+0000","last_became_active":"2023-10-19T14:01:38.149360+0000","last_became_peered":"2023-10-19T14:01:38.149360+0000","last_unstale":"2023-10-19T14:01:40.061795+0000","last_undegraded":"2023-10-19T14:01:40.061795+0000","last_fullsized":"2023-10-19T14:01:40.061795+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_clean_scrub_stamp":"2023-10-19T14:01:37.123936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:30:40.775839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.061706+0000","last_change":"2023-10-19T14:01:39.072854+0000","last_active":"2023-10-19T14:01:40.061706+0000","last_peered":"2023-10-19T14:01:40.061706+0000","last_clean":"2023-10-19T14:01:40.061706+0000","last_became_active":"2023-10-19T14:01:39.072507+0000","last_became_peered":"2023-10-19T14:01:39.072507+0000","last_unstale":"2023-10-19T14:01:40.061706+0000","last_undegraded":"2023-10-19T14:01:40.061706+0000","last_fullsized":"2023-10-19T14:01:40.061706+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:53:25.023783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":12,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.241239+0000","last_change":"2023-10-19T14:01:38.152825+0000","last_active":"2023-10-19T14:01:40.241239+0000","last_peered":"2023-10-19T14:01:40.241239+0000","last_clean":"2023-10-19T14:01:40.241239+0000","last_became_active":"2023-10-19T14:01:38.152533+0000","last_became_peered":"2023-10-19T14:01:38.152533+0000","last_unstale":"2023-10-19T14:01:40.241239+0000","last_undegraded":"2023-10-19T14:01:40.241239+0000","last_fullsized":"2023-10-19T14:01:40.241239+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_clean_scrub_stamp":"2023-10-19T14:01:37.123936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:33:51.860498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.7","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.061688+0000","last_change":"2023-10-19T14:01:39.078634+0000","last_active":"2023-10-19T14:01:40.061688+0000","last_peered":"2023-10-19T14:01:40.061688+0000","last_clean":"2023-10-19T14:01:40.061688+0000","last_became_active":"2023-10-19T14:01:39.078348+0000","last_became_peered":"2023-10-19T14:01:39.078348+0000","last_unstale":"2023-10-19T14:01:40.061688+0000","last_undegraded":"2023-10-19T14:01:40.061688+0000","last_fullsized":"2023-10-19T14:01:40.061688+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:49:05.321824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":12,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.061707+0000","last_change":"2023-10-19T14:01:38.155739+0000","last_active":"2023-10-19T14:01:40.061707+0000","last_peered":"2023-10-19T14:01:40.061707+0000","last_clean":"2023-10-19T14:01:40.061707+0000","last_became_active":"2023-10-19T14:01:38.155445+0000","last_became_peered":"2023-10-19T14:01:38.155445+0000","last_unstale":"2023-10-19T14:01:40.061707+0000","last_undegraded":"2023-10-19T14:01:40.061707+0000","last_fullsized":"2023-10-19T14:01:40.061707+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_clean_scrub_stamp":"2023-10-19T14:01:37.123936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:58:02.891758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.242337+0000","last_change":"2023-10-19T14:01:39.073851+0000","last_active":"2023-10-19T14:01:40.242337+0000","last_peered":"2023-10-19T14:01:40.242337+0000","last_clean":"2023-10-19T14:01:40.242337+0000","last_became_active":"2023-10-19T14:01:39.073027+0000","last_became_peered":"2023-10-19T14:01:39.073027+0000","last_unstale":"2023-10-19T14:01:40.242337+0000","last_undegraded":"2023-10-19T14:01:40.242337+0000","last_fullsized":"2023-10-19T14:01:40.242337+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:42:59.302185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":12,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.061382+0000","last_change":"2023-10-19T14:01:38.153359+0000","last_active":"2023-10-19T14:01:40.061382+0000","last_peered":"2023-10-19T14:01:40.061382+0000","last_clean":"2023-10-19T14:01:40.061382+0000","last_became_active":"2023-10-19T14:01:38.153078+0000","last_became_peered":"2023-10-19T14:01:38.153078+0000","last_unstale":"2023-10-19T14:01:40.061382+0000","last_undegraded":"2023-10-19T14:01:40.061382+0000","last_fullsized":"2023-10-19T14:01:40.061382+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_clean_scrub_stamp":"2023-10-19T14:01:37.123936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:33:03.326318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.242083+0000","last_change":"2023-10-19T14:01:39.076610+0000","last_active":"2023-10-19T14:01:40.242083+0000","last_peered":"2023-10-19T14:01:40.242083+0000","last_clean":"2023-10-19T14:01:40.242083+0000","last_became_active":"2023-10-19T14:01:39.075928+0000","last_became_peered":"2023-10-19T14:01:39.075928+0000","last_unstale":"2023-10-19T14:01:40.242083+0000","last_undegraded":"2023-10-19T14:01:40.242083+0000","last_fullsized":"2023-10-19T14:01:40.242083+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:57:05.135549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":12,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.062563+0000","last_change":"2023-10-19T14:01:38.152797+0000","last_active":"2023-10-19T14:01:40.062563+0000","last_peered":"2023-10-19T14:01:40.062563+0000","last_clean":"2023-10-19T14:01:40.062563+0000","last_became_active":"2023-10-19T14:01:38.152527+0000","last_became_peered":"2023-10-19T14:01:38.152527+0000","last_unstale":"2023-10-19T14:01:40.062563+0000","last_undegraded":"2023-10-19T14:01:40.062563+0000","last_fullsized":"2023-10-19T14:01:40.062563+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_clean_scrub_stamp":"2023-10-19T14:01:37.123936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:04:24.064005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.241128+0000","last_change":"2023-10-19T14:01:39.070849+0000","last_active":"2023-10-19T14:01:40.241128+0000","last_peered":"2023-10-19T14:01:40.241128+0000","last_clean":"2023-10-19T14:01:40.241128+0000","last_became_active":"2023-10-19T14:01:39.070554+0000","last_became_peered":"2023-10-19T14:01:39.070554+0000","last_unstale":"2023-10-19T14:01:40.241128+0000","last_undegraded":"2023-10-19T14:01:40.241128+0000","last_fullsized":"2023-10-19T14:01:40.241128+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:35:20.513587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":12,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.061446+0000","last_change":"2023-10-19T14:01:38.149257+0000","last_active":"2023-10-19T14:01:40.061446+0000","last_peered":"2023-10-19T14:01:40.061446+0000","last_clean":"2023-10-19T14:01:40.061446+0000","last_became_active":"2023-10-19T14:01:38.148973+0000","last_became_peered":"2023-10-19T14:01:38.148973+0000","last_unstale":"2023-10-19T14:01:40.061446+0000","last_undegraded":"2023-10-19T14:01:40.061446+0000","last_fullsized":"2023-10-19T14:01:40.061446+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_clean_scrub_stamp":"2023-10-19T14:01:37.123936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:08:54.567239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.242857+0000","last_change":"2023-10-19T14:01:39.075140+0000","last_active":"2023-10-19T14:01:40.242857+0000","last_peered":"2023-10-19T14:01:40.242857+0000","last_clean":"2023-10-19T14:01:40.242857+0000","last_became_active":"2023-10-19T14:01:39.074888+0000","last_became_peered":"2023-10-19T14:01:39.074888+0000","last_unstale":"2023-10-19T14:01:40.242857+0000","last_undegraded":"2023-10-19T14:01:40.242857+0000","last_fullsized":"2023-10-19T14:01:40.242857+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:37:55.217643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":12,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.242035+0000","last_change":"2023-10-19T14:01:38.155747+0000","last_active":"2023-10-19T14:01:40.242035+0000","last_peered":"2023-10-19T14:01:40.242035+0000","last_clean":"2023-10-19T14:01:40.242035+0000","last_became_active":"2023-10-19T14:01:38.155388+0000","last_became_peered":"2023-10-19T14:01:38.155388+0000","last_unstale":"2023-10-19T14:01:40.242035+0000","last_undegraded":"2023-10-19T14:01:40.242035+0000","last_fullsized":"2023-10-19T14:01:40.242035+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_clean_scrub_stamp":"2023-10-19T14:01:37.123936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:17:06.515478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.242623+0000","last_change":"2023-10-19T14:01:39.069645+0000","last_active":"2023-10-19T14:01:40.242623+0000","last_peered":"2023-10-19T14:01:40.242623+0000","last_clean":"2023-10-19T14:01:40.242623+0000","last_became_active":"2023-10-19T14:01:39.069300+0000","last_became_peered":"2023-10-19T14:01:39.069300+0000","last_unstale":"2023-10-19T14:01:40.242623+0000","last_undegraded":"2023-10-19T14:01:40.242623+0000","last_fullsized":"2023-10-19T14:01:40.242623+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:38:15.764494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.14","version":"174'6","reported_seq":18,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.193652+0000","last_change":"2023-10-19T14:01:38.154868+0000","last_active":"2023-10-19T14:01:40.193652+0000","last_peered":"2023-10-19T14:01:40.193652+0000","last_clean":"2023-10-19T14:01:40.193652+0000","last_became_active":"2023-10-19T14:01:38.154616+0000","last_became_peered":"2023-10-19T14:01:38.154616+0000","last_unstale":"2023-10-19T14:01:40.193652+0000","last_undegraded":"2023-10-19T14:01:40.193652+0000","last_fullsized":"2023-10-19T14:01:40.193652+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_clean_scrub_stamp":"2023-10-19T14:01:37.123936+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:26:47.710014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"164'3109","reported_seq":3761,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-10-19T14:01:39.067681+0000","last_change":"2023-10-19T12:44:08.636613+0000","last_active":"2023-10-19T14:01:39.067681+0000","last_peered":"2023-10-19T14:01:39.067681+0000","last_clean":"2023-10-19T14:01:39.067681+0000","last_became_active":"2023-10-19T12:44:08.635697+0000","last_became_peered":"2023-10-19T12:44:08.635697+0000","last_unstale":"2023-10-19T14:01:39.067681+0000","last_undegraded":"2023-10-19T14:01:39.067681+0000","last_fullsized":"2023-10-19T14:01:39.067681+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_clean_scrub_stamp":"2023-10-19T12:44:07.399738+0000","objects_scrubbed":0,"log_size":3109,"log_dups_size":0,"ondisk_log_size":3109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:15:23.685485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1538,"num_read_kb":3182,"num_write":449,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":22,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":21,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1538,"num_read_kb":3182,"num_write":449,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3109,"ondisk_log_size":3109,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575780,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60276,"kb_used_data":2720,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311564,"statfs":{"total":96636764160,"available":96575041536,"internally_reserved":0,"allocated":2785280,"data_stored":1701572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58915275},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:01:29 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.59099999999999997,"15min":0.59699999999999998},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.79900000000000004,"5min":0.85099999999999998,"15min":1.026},"last":0.38},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.34699999999999998,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.83099999999999996,"5min":0.84099999999999997,"15min":1},"last":0.51500000000000001}]},{"osd":1,"last update":"Thu Oct 19 14:01:29 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.59899999999999998,"15min":0.60099999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.873},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.60899999999999999,"15min":0.61199999999999999},"min":{"1min":0.46000000000000002,"5min":0.41999999999999998,"15min":0.376},"max":{"1min":0.84499999999999997,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.45500000000000002}]},{"osd":2,"last update":"Thu Oct 19 14:01:29 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.65900000000000003,"15min":0.65500000000000003},"min":{"1min":0.44500000000000001,"5min":0.44400000000000001,"15min":0.39000000000000001},"max":{"1min":0.875,"5min":0.877,"15min":0.96599999999999997},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.65700000000000003,"15min":0.66000000000000003},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.36799999999999999},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.434}]},{"osd":3,"last update":"Thu Oct 19 14:01:29 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.64300000000000002,"15min":0.64400000000000002},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.76800000000000002,"5min":0.86299999999999999,"15min":1.0129999999999999},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.69499999999999995,"15min":0.69199999999999995},"min":{"1min":0.504,"5min":0.504,"15min":0.48199999999999998},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.92000000000000004},"last":0.65900000000000003}]},{"osd":4,"last update":"Thu Oct 19 14:01:29 2023","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.48099999999999998,"15min":0.46600000000000003},"min":{"1min":0.32000000000000001,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.71799999999999997,"5min":0.73099999999999998,"15min":0.78500000000000003},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.503,"5min":0.51500000000000001,"15min":0.48799999999999999},"min":{"1min":0.35499999999999998,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.69399999999999995,"5min":0.78200000000000003,"15min":0.80900000000000005},"last":0.57299999999999995}]},{"osd":5,"last update":"Thu Oct 19 14:01:29 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.51000000000000001,"15min":0.49299999999999999},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.72899999999999998,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55400000000000005,"15min":0.53100000000000003},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.33900000000000002},"max":{"1min":0.77900000000000003,"5min":0.82699999999999996,"15min":0.86399999999999999},"last":0.628}]},{"osd":6,"last update":"Thu Oct 19 14:01:29 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.58999999999999997,"15min":0.56899999999999995},"min":{"1min":0.38500000000000001,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.81100000000000005,"5min":0.83299999999999996,"15min":0.85799999999999998},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.57399999999999995,"15min":0.55500000000000005},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.33200000000000002},"max":{"1min":0.78000000000000003,"5min":0.81000000000000005,"15min":0.94599999999999995},"last":0.64300000000000002}]}]},{"osd":6,"up_from":13,"seq":55834575780,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37468,"kb_used_data":2128,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334372,"statfs":{"total":96636764160,"available":96598396928,"internally_reserved":0,"allocated":2179072,"data_stored":1104604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:01:30 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.60599999999999998,"15min":0.60799999999999998},"min":{"1min":0.37,"5min":0.37,"15min":0.34599999999999997},"max":{"1min":0.82299999999999995,"5min":0.88100000000000001,"15min":1.3280000000000001},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61699999999999999,"15min":0.61399999999999999},"min":{"1min":0.46500000000000002,"5min":0.46400000000000002,"15min":0.26100000000000001},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":1.4279999999999999},"last":0.51700000000000002}]},{"osd":1,"last update":"Thu Oct 19 14:01:30 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.62,"15min":0.622},"min":{"1min":0.434,"5min":0.42799999999999999,"15min":0.35899999999999999},"max":{"1min":0.78400000000000003,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.624,"15min":0.61799999999999999},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.29299999999999998},"max":{"1min":0.83599999999999997,"5min":0.85399999999999998,"15min":0.89200000000000002},"last":0.60099999999999998}]},{"osd":2,"last update":"Thu Oct 19 14:01:30 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.38200000000000001},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":1.4650000000000001},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.621,"5min":0.65000000000000002,"15min":0.63200000000000001},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.35199999999999998},"max":{"1min":0.82399999999999995,"5min":0.85699999999999998,"15min":1.3979999999999999},"last":0.66800000000000004}]},{"osd":3,"last update":"Thu Oct 19 14:01:30 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.64700000000000002,"15min":0.65300000000000002},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.76600000000000001,"5min":0.82899999999999996,"15min":1.496},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.66900000000000004,"15min":0.66500000000000004},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.38100000000000001},"max":{"1min":0.77500000000000002,"5min":0.83499999999999996,"15min":1.512},"last":0.495}]},{"osd":4,"last update":"Thu Oct 19 14:01:30 2023","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.48499999999999999,"15min":0.47999999999999998},"min":{"1min":0.316,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.51600000000000001,"15min":0.50600000000000001},"min":{"1min":0.36699999999999999,"5min":0.35299999999999998,"15min":0.32800000000000001},"max":{"1min":0.79800000000000004,"5min":0.85699999999999998,"15min":0.93000000000000005},"last":0.42999999999999999}]},{"osd":5,"last update":"Thu Oct 19 14:01:30 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.5,"15min":0.49299999999999999},"min":{"1min":0.35499999999999998,"5min":0.35099999999999998,"15min":0.33700000000000002},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.623},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54200000000000004},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.78800000000000003,"5min":0.83199999999999996,"15min":1.0249999999999999},"last":0.63700000000000001}]},{"osd":7,"last update":"Thu Oct 19 14:01:30 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.54200000000000004,"15min":0.54500000000000004},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.32200000000000001},"max":{"1min":0.81999999999999995,"5min":0.84499999999999997,"15min":0.878},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56399999999999995,"15min":0.55500000000000005},"min":{"1min":0.40500000000000003,"5min":0.373,"15min":0.33400000000000002},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.91100000000000003},"last":0.67900000000000005}]}]},{"osd":5,"up_from":13,"seq":55834575780,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59688,"kb_used_data":2136,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312152,"statfs":{"total":96636764160,"available":96575643648,"internally_reserved":0,"allocated":2187264,"data_stored":1111182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:01:10 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.59399999999999997,"15min":0.61699999999999999},"min":{"1min":0.379,"5min":0.35399999999999998,"15min":0.34499999999999997},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":1.2629999999999999},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.63700000000000001,"15min":0.63300000000000001},"min":{"1min":0.50700000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.88300000000000001,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.55800000000000005}]},{"osd":1,"last update":"Thu Oct 19 14:01:10 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.63700000000000001,"15min":0.63600000000000001},"min":{"1min":0.51700000000000002,"5min":0.435,"15min":0.41699999999999998},"max":{"1min":0.81100000000000005,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.63800000000000001,"15min":0.64000000000000001},"min":{"1min":0.495,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.84899999999999998,"5min":0.87,"15min":1.071},"last":0.627}]},{"osd":2,"last update":"Thu Oct 19 14:01:10 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61899999999999999,"15min":0.628},"min":{"1min":0.46800000000000003,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.83499999999999996,"5min":0.875,"15min":1.044},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.64500000000000002,"15min":0.65300000000000002},"min":{"1min":0.48299999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.81399999999999995,"5min":0.90000000000000002,"15min":1.171},"last":0.68799999999999994}]},{"osd":3,"last update":"Thu Oct 19 14:01:10 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.63500000000000001,"15min":0.64400000000000002},"min":{"1min":0.46600000000000003,"5min":0.44,"15min":0.44},"max":{"1min":0.77800000000000002,"5min":0.85899999999999999,"15min":1.1910000000000001},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.67100000000000004,"15min":0.67500000000000004},"min":{"1min":0.52200000000000002,"5min":0.45500000000000002,"15min":0.43099999999999999},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":1.2210000000000001},"last":0.54600000000000004}]},{"osd":4,"last update":"Thu Oct 19 14:01:10 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.5,"15min":0.48499999999999999},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.65100000000000002,"5min":0.745,"15min":0.75700000000000001},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.501,"5min":0.5,"15min":0.496},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.73899999999999999,"5min":0.76200000000000001,"15min":0.77300000000000002},"last":0.64700000000000002}]},{"osd":6,"last update":"Thu Oct 19 14:01:10 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.48899999999999999,"15min":0.48999999999999999},"min":{"1min":0.32600000000000001,"5min":0.32100000000000001,"15min":0.29699999999999999},"max":{"1min":0.64900000000000002,"5min":0.80000000000000004,"15min":0.83599999999999997},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.53000000000000003,"15min":0.53100000000000003},"min":{"1min":0.38900000000000001,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.80000000000000004},"last":0.70199999999999996}]},{"osd":7,"last update":"Thu Oct 19 14:01:10 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.51500000000000001,"15min":0.51200000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.68000000000000005,"5min":0.79200000000000004,"15min":0.80000000000000004},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.56999999999999995,"15min":0.56299999999999994},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.80100000000000005,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.754}]}]},{"osd":4,"up_from":13,"seq":55834575780,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41652,"kb_used_data":2148,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330188,"statfs":{"total":96636764160,"available":96594112512,"internally_reserved":0,"allocated":2199552,"data_stored":1112794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:00:39 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.59099999999999997,"15min":0.59599999999999997},"min":{"1min":0.45700000000000002,"5min":0.38500000000000001,"15min":0.38400000000000001},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.437},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59899999999999998,"15min":0.59699999999999998},"min":{"1min":0.47899999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.73699999999999999,"5min":0.83199999999999996,"15min":0.98199999999999998},"last":0.41599999999999998}]},{"osd":1,"last update":"Thu Oct 19 14:00:39 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.64300000000000002,"15min":0.64700000000000002},"min":{"1min":0.50600000000000001,"5min":0.443,"15min":0.40600000000000003},"max":{"1min":0.88300000000000001,"5min":0.89400000000000002,"15min":0.92100000000000004},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.65900000000000003,"15min":0.66000000000000003},"min":{"1min":0.53900000000000003,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.57099999999999995}]},{"osd":2,"last update":"Thu Oct 19 14:00:39 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.66300000000000003,"15min":0.65300000000000002},"min":{"1min":0.47099999999999997,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.67300000000000004,"15min":0.66500000000000004},"min":{"1min":0.503,"5min":0.46800000000000003,"15min":0.442},"max":{"1min":1.054,"5min":1.054,"15min":1.054},"last":0.60899999999999999}]},{"osd":3,"last update":"Thu Oct 19 14:00:39 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.65000000000000002,"15min":0.64200000000000002},"min":{"1min":0.48499999999999999,"5min":0.47799999999999998,"15min":0.378},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.93600000000000005},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.67900000000000005,"15min":0.67200000000000004},"min":{"1min":0.58399999999999996,"5min":0.438,"15min":0.438},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.58799999999999997}]},{"osd":5,"last update":"Thu Oct 19 14:00:39 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.46899999999999997,"15min":0.45500000000000002},"min":{"1min":0.33200000000000002,"5min":0.318,"15min":0.29399999999999998},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.498,"15min":0.47999999999999998},"min":{"1min":0.35399999999999998,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.80900000000000005},"last":0.49399999999999999}]},{"osd":6,"last update":"Thu Oct 19 14:00:39 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.49399999999999999,"15min":0.47199999999999998},"min":{"1min":0.375,"5min":0.34000000000000002,"15min":0.32900000000000001},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.62},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.53600000000000003,"15min":0.52000000000000002},"min":{"1min":0.38200000000000001,"5min":0.34499999999999997,"15min":0.32700000000000001},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.64300000000000002}]},{"osd":7,"last update":"Thu Oct 19 14:00:39 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.54000000000000004,"15min":0.52500000000000002},"min":{"1min":0.34200000000000003,"5min":0.33800000000000002,"15min":0.32200000000000001},"max":{"1min":0.71999999999999997,"5min":0.80500000000000005,"15min":0.83699999999999997},"last":0.629},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.60899999999999999,"15min":0.58199999999999996},"min":{"1min":0.46899999999999997,"5min":0.39500000000000002,"15min":0.39200000000000002},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.65500000000000003}]}]},{"osd":3,"up_from":13,"seq":55834575779,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47224,"kb_used_data":2144,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324616,"statfs":{"total":96636764160,"available":96588406784,"internally_reserved":0,"allocated":2195456,"data_stored":1111798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":46135752},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:01:08 2023","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.40999999999999998,"15min":0.40400000000000003},"min":{"1min":0.315,"5min":0.26000000000000001,"15min":0.249},"max":{"1min":0.59199999999999997,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.434,"5min":0.44400000000000001,"15min":0.435},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.26100000000000001},"max":{"1min":0.65400000000000003,"5min":0.66600000000000004,"15min":0.69799999999999995},"last":0.30299999999999999}]},{"osd":1,"last update":"Thu Oct 19 14:01:08 2023","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.42399999999999999,"15min":0.42699999999999999},"min":{"1min":0.30299999999999999,"5min":0.29099999999999998,"15min":0.26200000000000001},"max":{"1min":0.54900000000000004,"5min":0.65600000000000003,"15min":0.72999999999999998},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.47799999999999998,"15min":0.46400000000000002},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.26900000000000002},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72699999999999998},"last":0.34499999999999997}]},{"osd":2,"last update":"Thu Oct 19 14:01:08 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.48999999999999999,"15min":0.48499999999999999},"min":{"1min":0.35399999999999998,"5min":0.33400000000000002,"15min":0.313},"max":{"1min":0.70399999999999996,"5min":0.71899999999999997,"15min":0.72099999999999997},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.51000000000000001,"15min":0.503},"min":{"1min":0.39200000000000002,"5min":0.33600000000000002,"15min":0.32200000000000001},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.42699999999999999}]},{"osd":4,"last update":"Thu Oct 19 14:01:08 2023","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.70399999999999996,"15min":0.69899999999999995},"min":{"1min":0.56599999999999995,"5min":0.46700000000000003,"15min":0.41499999999999998},"max":{"1min":0.81999999999999995,"5min":1.3600000000000001,"15min":1.3600000000000001},"last":0.748},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.71099999999999997,"15min":0.71299999999999997},"min":{"1min":0.56999999999999995,"5min":0.46800000000000003,"15min":0.46300000000000002},"max":{"1min":0.82499999999999996,"5min":1.032,"15min":1.2410000000000001},"last":0.66400000000000003}]},{"osd":5,"last update":"Thu Oct 19 14:01:08 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.69099999999999995,"15min":0.68500000000000005},"min":{"1min":0.54200000000000004,"5min":0.51800000000000002,"15min":0.39700000000000002},"max":{"1min":0.90100000000000002,"5min":1.2450000000000001,"15min":1.2450000000000001},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.67100000000000004,"15min":0.67200000000000004},"min":{"1min":0.502,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.80900000000000005,"5min":1.385,"15min":1.385},"last":0.70199999999999996}]},{"osd":6,"last update":"Thu Oct 19 14:01:08 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.68999999999999995,"15min":0.68500000000000005},"min":{"1min":0.47899999999999998,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.94499999999999995,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.70399999999999996,"15min":0.70299999999999996},"min":{"1min":0.55800000000000005,"5min":0.49099999999999999,"15min":0.42799999999999999},"max":{"1min":0.85799999999999998,"5min":1.0649999999999999,"15min":1.282},"last":0.52800000000000002}]},{"osd":7,"last update":"Thu Oct 19 14:01:08 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.70599999999999996,"15min":0.69399999999999995},"min":{"1min":0.52700000000000002,"5min":0.47499999999999998,"15min":0.46400000000000002},"max":{"1min":0.92000000000000004,"5min":1.3140000000000001,"15min":1.3140000000000001},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72299999999999998},"min":{"1min":0.54500000000000004,"5min":0.49099999999999999,"15min":0.48599999999999999},"max":{"1min":0.88,"5min":0.98899999999999999,"15min":1.304},"last":0.54200000000000004}]}]},{"osd":2,"up_from":13,"seq":55834575781,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59680,"kb_used_data":2132,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312160,"statfs":{"total":96636764160,"available":96575651840,"internally_reserved":0,"allocated":2183168,"data_stored":1104646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:00:56 2023","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.42199999999999999,"15min":0.40899999999999997},"min":{"1min":0.26800000000000002,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.59799999999999998,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.46100000000000002,"15min":0.44600000000000001},"min":{"1min":0.34000000000000002,"5min":0.29799999999999999,"15min":0.26800000000000002},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.42399999999999999}]},{"osd":1,"last update":"Thu Oct 19 14:00:56 2023","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.41799999999999998,"15min":0.41599999999999998},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26700000000000002},"max":{"1min":0.59499999999999997,"5min":0.60799999999999998,"15min":0.66600000000000004},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.48099999999999998,"15min":0.46700000000000003},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.26500000000000001},"max":{"1min":0.69399999999999995,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.35399999999999998}]},{"osd":3,"last update":"Thu Oct 19 14:00:56 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.501,"15min":0.49399999999999999},"min":{"1min":0.33700000000000002,"5min":0.31900000000000001,"15min":0.30199999999999999},"max":{"1min":0.57299999999999995,"5min":0.72099999999999997,"15min":0.75900000000000001},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.52500000000000002,"15min":0.51700000000000002},"min":{"1min":0.40000000000000002,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.66400000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.375}]},{"osd":4,"last update":"Thu Oct 19 14:00:56 2023","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.69799999999999995,"15min":0.68600000000000005},"min":{"1min":0.55300000000000005,"5min":0.46800000000000003,"15min":0.435},"max":{"1min":0.85499999999999998,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.71299999999999997,"15min":0.69499999999999995},"min":{"1min":0.55300000000000005,"5min":0.46600000000000003,"15min":0.44800000000000001},"max":{"1min":1.0580000000000001,"5min":1.2110000000000001,"15min":1.2110000000000001},"last":0.65600000000000003}]},{"osd":5,"last update":"Thu Oct 19 14:00:56 2023","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.72799999999999998,"15min":0.70799999999999996},"min":{"1min":0.53000000000000003,"5min":0.46899999999999997,"15min":0.42299999999999999},"max":{"1min":1.1919999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.69399999999999995},"min":{"1min":0.54400000000000004,"5min":0.48999999999999999,"15min":0.44600000000000001},"max":{"1min":1.1459999999999999,"5min":1.194,"15min":1.194},"last":0.63300000000000001}]},{"osd":6,"last update":"Thu Oct 19 14:00:56 2023","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.72899999999999998,"15min":0.71999999999999997},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.443},"max":{"1min":1.0149999999999999,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.75,"15min":0.72799999999999998},"min":{"1min":0.60199999999999998,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.103,"5min":1.234,"15min":1.234},"last":0.76000000000000001}]},{"osd":7,"last update":"Thu Oct 19 14:00:56 2023","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.75800000000000001,"15min":0.74099999999999999},"min":{"1min":0.65500000000000003,"5min":0.53200000000000003,"15min":0.47699999999999998},"max":{"1min":1.1259999999999999,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.753,"5min":0.76400000000000001,"15min":0.748},"min":{"1min":0.622,"5min":0.505,"15min":0.49399999999999999},"max":{"1min":1.161,"5min":1.161,"15min":1.161},"last":0.70599999999999996}]}]},{"osd":1,"up_from":13,"seq":55834575780,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41632,"kb_used_data":2128,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330208,"statfs":{"total":96636764160,"available":96594132992,"internally_reserved":0,"allocated":2179072,"data_stored":1104604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:00:51 2023","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41499999999999998},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.219},"max":{"1min":0.64500000000000002,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.47399999999999998,"15min":0.46100000000000002},"min":{"1min":0.29899999999999999,"5min":0.25800000000000001,"15min":0.25600000000000001},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.32400000000000001}]},{"osd":2,"last update":"Thu Oct 19 14:00:51 2023","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.44500000000000001,"15min":0.436},"min":{"1min":0.315,"5min":0.29699999999999999,"15min":0.26600000000000001},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68300000000000005},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.5,"5min":0.49399999999999999,"15min":0.48399999999999999},"min":{"1min":0.375,"5min":0.32100000000000001,"15min":0.28499999999999998},"max":{"1min":0.61099999999999999,"5min":0.70599999999999996,"15min":0.73099999999999998},"last":0.40999999999999998}]},{"osd":3,"last update":"Thu Oct 19 14:00:51 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52200000000000002,"15min":0.50900000000000001},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.33800000000000002},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.73899999999999999},"last":0.38},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.45300000000000001,"15min":0.44800000000000001},"min":{"1min":0.33800000000000002,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.67500000000000004},"last":0.36499999999999999}]},{"osd":4,"last update":"Thu Oct 19 14:00:51 2023","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.70599999999999996,"15min":0.69199999999999995},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.434},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.69099999999999995,"15min":0.67800000000000005},"min":{"1min":0.56399999999999995,"5min":0.52300000000000002,"15min":0.40699999999999997},"max":{"1min":0.90200000000000002,"5min":0.90500000000000003,"15min":1.0720000000000001},"last":0.47199999999999998}]},{"osd":5,"last update":"Thu Oct 19 14:00:51 2023","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.71399999999999997,"15min":0.69399999999999995},"min":{"1min":0.56699999999999995,"5min":0.50600000000000001,"15min":0.44},"max":{"1min":1.155,"5min":1.155,"15min":1.254},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.72599999999999998,"15min":0.71099999999999997},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.49099999999999999},"max":{"1min":1.014,"5min":1.014,"15min":1.0760000000000001},"last":0.53700000000000003}]},{"osd":6,"last update":"Thu Oct 19 14:00:51 2023","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.71099999999999997,"15min":0.69399999999999995},"min":{"1min":0.502,"5min":0.502,"15min":0.41899999999999998},"max":{"1min":1.25,"5min":1.25,"15min":1.25},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.73499999999999999,"15min":0.72199999999999998},"min":{"1min":0.56200000000000006,"5min":0.47599999999999998,"15min":0.44500000000000001},"max":{"1min":0.999,"5min":0.999,"15min":1.153},"last":0.56799999999999995}]},{"osd":7,"last update":"Thu Oct 19 14:00:51 2023","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.745,"15min":0.72399999999999998},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.43099999999999999},"max":{"1min":1.1950000000000001,"5min":1.1950000000000001,"15min":1.1950000000000001},"last":0.497},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.746,"15min":0.72699999999999998},"min":{"1min":0.59599999999999997,"5min":0.54500000000000004,"15min":0.49399999999999999},"max":{"1min":1.097,"5min":1.097,"15min":1.097},"last":0.44900000000000001}]}]},{"osd":0,"up_from":13,"seq":55834575781,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38084,"kb_used_data":2736,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333756,"statfs":{"total":96636764160,"available":96597766144,"internally_reserved":0,"allocated":2801664,"data_stored":1703249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Oct 19 14:01:34 2023","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.38900000000000001,"15min":0.39600000000000002},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.45600000000000002,"15min":0.46000000000000002},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.30399999999999999}]},{"osd":2,"last update":"Thu Oct 19 14:01:34 2023","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.47399999999999998,"15min":0.47099999999999997},"min":{"1min":0.312,"5min":0.30299999999999999,"15min":0.28699999999999998},"max":{"1min":0.74199999999999999,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.441,"5min":0.45400000000000001,"15min":0.45100000000000001},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.40500000000000003}]},{"osd":3,"last update":"Thu Oct 19 14:01:34 2023","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.439,"15min":0.43099999999999999},"min":{"1min":0.32700000000000001,"5min":0.30099999999999999,"15min":0.27500000000000002},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.37},{"interface":"front","average":{"1min":0.5,"5min":0.52000000000000002,"15min":0.50900000000000001},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.45600000000000002}]},{"osd":4,"last update":"Thu Oct 19 14:01:34 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.68700000000000006,"15min":0.67400000000000004},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.43099999999999999},"max":{"1min":1.0189999999999999,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.69399999999999995,"15min":0.67900000000000005},"min":{"1min":0.49099999999999999,"5min":0.46800000000000003,"15min":0.45900000000000002},"max":{"1min":0.88900000000000001,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.745}]},{"osd":5,"last update":"Thu Oct 19 14:01:34 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.73399999999999999,"15min":0.71699999999999997},"min":{"1min":0.5,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.069,"5min":1.1970000000000001,"15min":1.1970000000000001},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.73699999999999999,"15min":0.73199999999999998},"min":{"1min":0.51500000000000001,"5min":0.443,"15min":0.443},"max":{"1min":0.95399999999999996,"5min":1.139,"15min":1.139},"last":0.85199999999999998}]},{"osd":6,"last update":"Thu Oct 19 14:01:34 2023","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.71499999999999997,"15min":0.70199999999999996},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.091,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.622},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.71099999999999997,"15min":0.69899999999999995},"min":{"1min":0.503,"5min":0.49399999999999999,"15min":0.443},"max":{"1min":0.98099999999999998,"5min":1.0660000000000001,"15min":1.167},"last":0.79700000000000004}]},{"osd":7,"last update":"Thu Oct 19 14:01:34 2023","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.74199999999999999,"15min":0.72699999999999998},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.145,"5min":1.163,"15min":1.163},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.71599999999999997,"15min":0.70099999999999996},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.123,"5min":1.123,"15min":1.123},"last":0.63200000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-19T14:01:42.152 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-19T14:01:42.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.292+0000 7f9a9f1a9640 1 -- 172.21.15.57:0/1896822712 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a9806ffb0 msgr2=0x7f9a980c96b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:42.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.292+0000 7f9a9f1a9640 1 --2- 172.21.15.57:0/1896822712 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a9806ffb0 0x7f9a980c96b0 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 crypto rx=0x7f9a84005fd0 tx=0x7f9a8402be10 comp rx=0 tx=0).stop 2023-10-19T14:01:42.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.292+0000 7f9a9f1a9640 1 -- 172.21.15.57:0/1896822712 shutdown_connections 2023-10-19T14:01:42.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.292+0000 7f9a9f1a9640 1 --2- 172.21.15.57:0/1896822712 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a9806ffb0 0x7f9a980c96b0 unknown :-1 s=CLOSED pgs=1912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:42.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.292+0000 7f9a9f1a9640 1 --2- 172.21.15.57:0/1896822712 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9a980b1920 0x7f9a9806fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:42.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.292+0000 7f9a9f1a9640 1 -- 172.21.15.57:0/1896822712 >> 172.21.15.57:0/1896822712 conn(0x7f9a98069930 msgr2=0x7f9a98069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:42.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.292+0000 7f9a9f1a9640 1 -- 172.21.15.57:0/1896822712 shutdown_connections 2023-10-19T14:01:42.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.292+0000 7f9a9f1a9640 1 -- 172.21.15.57:0/1896822712 wait complete. 2023-10-19T14:01:42.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.293+0000 7f9a9f1a9640 1 Processor -- start 2023-10-19T14:01:42.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.293+0000 7f9a9f1a9640 1 -- start start 2023-10-19T14:01:42.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.294+0000 7f9a9f1a9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a9806ffb0 0x7f9a98157600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:42.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.294+0000 7f9a9f1a9640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9a980b1920 0x7f9a981514a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:42.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.294+0000 7f9a9f1a9640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9a980c9bf0 0x7f9a981f2470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:42.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.294+0000 7f9a9f1a9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a9806c5c0 con 0x7f9a9806ffb0 2023-10-19T14:01:42.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.294+0000 7f9a9f1a9640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a9815c720 con 0x7f9a980c9bf0 2023-10-19T14:01:42.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.294+0000 7f9a9f1a9640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a981519e0 con 0x7f9a980b1920 2023-10-19T14:01:42.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.294+0000 7f9a9cf45640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a9806ffb0 0x7f9a98157600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:42.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.294+0000 7f9a9cf45640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a9806ffb0 0x7f9a98157600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35292/0 (socket says 172.21.15.57:35292) 2023-10-19T14:01:42.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.294+0000 7f9a9cf45640 1 -- 172.21.15.57:0/4110251044 learned_addr learned my addr 172.21.15.57:0/4110251044 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:01:42.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.294+0000 7f9a97fff640 1 --2- 172.21.15.57:0/4110251044 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9a980b1920 0x7f9a981514a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:42.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.294+0000 7f9a9d746640 1 --2- 172.21.15.57:0/4110251044 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9a980c9bf0 0x7f9a981f2470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:42.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.294+0000 7f9a9cf45640 1 -- 172.21.15.57:0/4110251044 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9a980b1920 msgr2=0x7f9a981514a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:42.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.294+0000 7f9a9cf45640 1 --2- 172.21.15.57:0/4110251044 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9a980b1920 0x7f9a981514a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:42.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.294+0000 7f9a9cf45640 1 -- 172.21.15.57:0/4110251044 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9a980c9bf0 msgr2=0x7f9a981f2470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:42.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.294+0000 7f9a9cf45640 1 --2- 172.21.15.57:0/4110251044 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9a980c9bf0 0x7f9a981f2470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:42.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.294+0000 7f9a9cf45640 1 -- 172.21.15.57:0/4110251044 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a84009560 con 0x7f9a9806ffb0 2023-10-19T14:01:42.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.294+0000 7f9a97fff640 1 --2- 172.21.15.57:0/4110251044 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9a980b1920 0x7f9a981514a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:01:42.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.294+0000 7f9a9d746640 1 --2- 172.21.15.57:0/4110251044 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9a980c9bf0 0x7f9a981f2470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:01:42.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.294+0000 7f9a9cf45640 1 --2- 172.21.15.57:0/4110251044 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a9806ffb0 0x7f9a98157600 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c00e9f0 tx=0x7f9a8c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:42.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.295+0000 7f9a957fa640 1 -- 172.21.15.57:0/4110251044 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a8c009700 con 0x7f9a9806ffb0 2023-10-19T14:01:42.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.295+0000 7f9a9f1a9640 1 -- 172.21.15.57:0/4110251044 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a981f29b0 con 0x7f9a9806ffb0 2023-10-19T14:01:42.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.295+0000 7f9a957fa640 1 -- 172.21.15.57:0/4110251044 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f9a8c00ce70 con 0x7f9a9806ffb0 2023-10-19T14:01:42.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.295+0000 7f9a957fa640 1 -- 172.21.15.57:0/4110251044 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a8c010670 con 0x7f9a9806ffb0 2023-10-19T14:01:42.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.295+0000 7f9a9f1a9640 1 -- 172.21.15.57:0/4110251044 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a98157f10 con 0x7f9a9806ffb0 2023-10-19T14:01:42.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.296+0000 7f9a9f1a9640 1 -- 172.21.15.57:0/4110251044 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a980690e0 con 0x7f9a9806ffb0 2023-10-19T14:01:42.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.296+0000 7f9a957fa640 1 -- 172.21.15.57:0/4110251044 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 91) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f9a8c0107d0 con 0x7f9a9806ffb0 2023-10-19T14:01:42.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.297+0000 7f9a957fa640 1 --2- 172.21.15.57:0/4110251044 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f9a7c06af60 0x7f9a7c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:42.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.297+0000 7f9a957fa640 1 -- 172.21.15.57:0/4110251044 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f9a8c014070 con 0x7f9a9806ffb0 2023-10-19T14:01:42.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.297+0000 7f9a97fff640 1 --2- 172.21.15.57:0/4110251044 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f9a7c06af60 0x7f9a7c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:42.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.298+0000 7f9a97fff640 1 --2- 172.21.15.57:0/4110251044 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f9a7c06af60 0x7f9a7c06d420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9a84035000 tx=0x7f9a84004730 comp rx=0 tx=0).ready entity=mgr.8923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:42.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.301+0000 7f9a957fa640 1 -- 172.21.15.57:0/4110251044 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9a8c051ca0 con 0x7f9a9806ffb0 2023-10-19T14:01:42.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.476+0000 7f9a9f1a9640 1 -- 172.21.15.57:0/4110251044 --> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9a9815c860 con 0x7f9a7c06af60 2023-10-19T14:01:42.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.483+0000 7f9a957fa640 1 -- 172.21.15.57:0/4110251044 <== mgr.8923 v2:172.21.15.179:6832/2037224568 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+290172 (secure 0 0 0) 0x7f9a84004310 con 0x7f9a7c06af60 2023-10-19T14:01:42.488 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:01:42.490 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2023-10-19T14:01:42.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.487+0000 7f9a9f1a9640 1 -- 172.21.15.57:0/4110251044 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f9a7c06af60 msgr2=0x7f9a7c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:42.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.487+0000 7f9a9f1a9640 1 --2- 172.21.15.57:0/4110251044 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f9a7c06af60 0x7f9a7c06d420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9a84035000 tx=0x7f9a84004730 comp rx=0 tx=0).stop 2023-10-19T14:01:42.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.487+0000 7f9a9f1a9640 1 -- 172.21.15.57:0/4110251044 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a9806ffb0 msgr2=0x7f9a98157600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:42.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.487+0000 7f9a9f1a9640 1 --2- 172.21.15.57:0/4110251044 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a9806ffb0 0x7f9a98157600 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c00e9f0 tx=0x7f9a8c00eeb0 comp rx=0 tx=0).stop 2023-10-19T14:01:42.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.487+0000 7f9a9f1a9640 1 -- 172.21.15.57:0/4110251044 shutdown_connections 2023-10-19T14:01:42.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.487+0000 7f9a9f1a9640 1 --2- 172.21.15.57:0/4110251044 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f9a7c06af60 0x7f9a7c06d420 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:42.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.487+0000 7f9a9f1a9640 1 --2- 172.21.15.57:0/4110251044 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9a980c9bf0 0x7f9a981f2470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:42.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.487+0000 7f9a9f1a9640 1 --2- 172.21.15.57:0/4110251044 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9a980b1920 0x7f9a981514a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:42.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.487+0000 7f9a9f1a9640 1 --2- 172.21.15.57:0/4110251044 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a9806ffb0 0x7f9a98157600 unknown :-1 s=CLOSED pgs=1913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:42.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.487+0000 7f9a9f1a9640 1 -- 172.21.15.57:0/4110251044 >> 172.21.15.57:0/4110251044 conn(0x7f9a98069930 msgr2=0x7f9a980cb500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:42.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.487+0000 7f9a9f1a9640 1 -- 172.21.15.57:0/4110251044 shutdown_connections 2023-10-19T14:01:42.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.487+0000 7f9a9f1a9640 1 -- 172.21.15.57:0/4110251044 wait complete. 2023-10-19T14:01:42.500 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2023-10-19T14:01:40.545197+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1538,"num_read_kb":3182,"num_write":493,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3147,"ondisk_log_size":3147,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":148,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":385704,"kb_used_data":18272,"kb_used_omap":12,"kb_used_meta":367283,"kb_avail":754589016,"statfs":{"total":773094113280,"available":772699152384,"internally_reserved":0,"allocated":18710528,"data_stored":10054449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":376098361},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000806"},"pg_stats":[{"pgid":"22.3f","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.061645+0000","last_change":"2023-10-19T14:01:39.071873+0000","last_active":"2023-10-19T14:01:40.061645+0000","last_peered":"2023-10-19T14:01:40.061645+0000","last_clean":"2023-10-19T14:01:40.061645+0000","last_became_active":"2023-10-19T14:01:39.071596+0000","last_became_peered":"2023-10-19T14:01:39.071596+0000","last_unstale":"2023-10-19T14:01:40.061645+0000","last_undegraded":"2023-10-19T14:01:40.061645+0000","last_fullsized":"2023-10-19T14:01:40.061645+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:03:21.766515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.3e","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.241296+0000","last_change":"2023-10-19T14:01:39.080355+0000","last_active":"2023-10-19T14:01:40.241296+0000","last_peered":"2023-10-19T14:01:40.241296+0000","last_clean":"2023-10-19T14:01:40.241296+0000","last_became_active":"2023-10-19T14:01:39.080067+0000","last_became_peered":"2023-10-19T14:01:39.080067+0000","last_unstale":"2023-10-19T14:01:40.241296+0000","last_undegraded":"2023-10-19T14:01:40.241296+0000","last_fullsized":"2023-10-19T14:01:40.241296+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:08:23.201667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.3d","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.241076+0000","last_change":"2023-10-19T14:01:39.075158+0000","last_active":"2023-10-19T14:01:40.241076+0000","last_peered":"2023-10-19T14:01:40.241076+0000","last_clean":"2023-10-19T14:01:40.241076+0000","last_became_active":"2023-10-19T14:01:39.074871+0000","last_became_peered":"2023-10-19T14:01:39.074871+0000","last_unstale":"2023-10-19T14:01:40.241076+0000","last_undegraded":"2023-10-19T14:01:40.241076+0000","last_fullsized":"2023-10-19T14:01:40.241076+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:40:04.433729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.3c","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.242382+0000","last_change":"2023-10-19T14:01:39.077385+0000","last_active":"2023-10-19T14:01:40.242382+0000","last_peered":"2023-10-19T14:01:40.242382+0000","last_clean":"2023-10-19T14:01:40.242382+0000","last_became_active":"2023-10-19T14:01:39.076971+0000","last_became_peered":"2023-10-19T14:01:39.076971+0000","last_unstale":"2023-10-19T14:01:40.242382+0000","last_undegraded":"2023-10-19T14:01:40.242382+0000","last_fullsized":"2023-10-19T14:01:40.242382+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:28:53.177652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.3b","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.061832+0000","last_change":"2023-10-19T14:01:39.079807+0000","last_active":"2023-10-19T14:01:40.061832+0000","last_peered":"2023-10-19T14:01:40.061832+0000","last_clean":"2023-10-19T14:01:40.061832+0000","last_became_active":"2023-10-19T14:01:39.079401+0000","last_became_peered":"2023-10-19T14:01:39.079401+0000","last_unstale":"2023-10-19T14:01:40.061832+0000","last_undegraded":"2023-10-19T14:01:40.061832+0000","last_fullsized":"2023-10-19T14:01:40.061832+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:17:25.334467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.3a","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.061615+0000","last_change":"2023-10-19T14:01:39.080122+0000","last_active":"2023-10-19T14:01:40.061615+0000","last_peered":"2023-10-19T14:01:40.061615+0000","last_clean":"2023-10-19T14:01:40.061615+0000","last_became_active":"2023-10-19T14:01:39.079614+0000","last_became_peered":"2023-10-19T14:01:39.079614+0000","last_unstale":"2023-10-19T14:01:40.061615+0000","last_undegraded":"2023-10-19T14:01:40.061615+0000","last_fullsized":"2023-10-19T14:01:40.061615+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:21:59.248909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.39","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.061330+0000","last_change":"2023-10-19T14:01:39.075475+0000","last_active":"2023-10-19T14:01:40.061330+0000","last_peered":"2023-10-19T14:01:40.061330+0000","last_clean":"2023-10-19T14:01:40.061330+0000","last_became_active":"2023-10-19T14:01:39.074883+0000","last_became_peered":"2023-10-19T14:01:39.074883+0000","last_unstale":"2023-10-19T14:01:40.061330+0000","last_undegraded":"2023-10-19T14:01:40.061330+0000","last_fullsized":"2023-10-19T14:01:40.061330+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:20:28.271547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.38","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.241930+0000","last_change":"2023-10-19T14:01:39.074913+0000","last_active":"2023-10-19T14:01:40.241930+0000","last_peered":"2023-10-19T14:01:40.241930+0000","last_clean":"2023-10-19T14:01:40.241930+0000","last_became_active":"2023-10-19T14:01:39.073378+0000","last_became_peered":"2023-10-19T14:01:39.073378+0000","last_unstale":"2023-10-19T14:01:40.241930+0000","last_undegraded":"2023-10-19T14:01:40.241930+0000","last_fullsized":"2023-10-19T14:01:40.241930+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:40:12.362136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.37","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.061543+0000","last_change":"2023-10-19T14:01:39.075601+0000","last_active":"2023-10-19T14:01:40.061543+0000","last_peered":"2023-10-19T14:01:40.061543+0000","last_clean":"2023-10-19T14:01:40.061543+0000","last_became_active":"2023-10-19T14:01:39.074903+0000","last_became_peered":"2023-10-19T14:01:39.074903+0000","last_unstale":"2023-10-19T14:01:40.061543+0000","last_undegraded":"2023-10-19T14:01:40.061543+0000","last_fullsized":"2023-10-19T14:01:40.061543+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:54:44.077594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.36","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.061464+0000","last_change":"2023-10-19T14:01:39.077407+0000","last_active":"2023-10-19T14:01:40.061464+0000","last_peered":"2023-10-19T14:01:40.061464+0000","last_clean":"2023-10-19T14:01:40.061464+0000","last_became_active":"2023-10-19T14:01:39.077030+0000","last_became_peered":"2023-10-19T14:01:39.077030+0000","last_unstale":"2023-10-19T14:01:40.061464+0000","last_undegraded":"2023-10-19T14:01:40.061464+0000","last_fullsized":"2023-10-19T14:01:40.061464+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:14:13.643726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.35","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.062379+0000","last_change":"2023-10-19T14:01:39.075978+0000","last_active":"2023-10-19T14:01:40.062379+0000","last_peered":"2023-10-19T14:01:40.062379+0000","last_clean":"2023-10-19T14:01:40.062379+0000","last_became_active":"2023-10-19T14:01:39.075623+0000","last_became_peered":"2023-10-19T14:01:39.075623+0000","last_unstale":"2023-10-19T14:01:40.062379+0000","last_undegraded":"2023-10-19T14:01:40.062379+0000","last_fullsized":"2023-10-19T14:01:40.062379+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:23:46.731396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.34","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.242603+0000","last_change":"2023-10-19T14:01:39.077218+0000","last_active":"2023-10-19T14:01:40.242603+0000","last_peered":"2023-10-19T14:01:40.242603+0000","last_clean":"2023-10-19T14:01:40.242603+0000","last_became_active":"2023-10-19T14:01:39.076680+0000","last_became_peered":"2023-10-19T14:01:39.076680+0000","last_unstale":"2023-10-19T14:01:40.242603+0000","last_undegraded":"2023-10-19T14:01:40.242603+0000","last_fullsized":"2023-10-19T14:01:40.242603+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:52:50.282449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.33","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.061857+0000","last_change":"2023-10-19T14:01:39.077328+0000","last_active":"2023-10-19T14:01:40.061857+0000","last_peered":"2023-10-19T14:01:40.061857+0000","last_clean":"2023-10-19T14:01:40.061857+0000","last_became_active":"2023-10-19T14:01:39.075953+0000","last_became_peered":"2023-10-19T14:01:39.075953+0000","last_unstale":"2023-10-19T14:01:40.061857+0000","last_undegraded":"2023-10-19T14:01:40.061857+0000","last_fullsized":"2023-10-19T14:01:40.061857+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:48:15.769444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.32","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.243278+0000","last_change":"2023-10-19T14:01:39.077498+0000","last_active":"2023-10-19T14:01:40.243278+0000","last_peered":"2023-10-19T14:01:40.243278+0000","last_clean":"2023-10-19T14:01:40.243278+0000","last_became_active":"2023-10-19T14:01:39.077146+0000","last_became_peered":"2023-10-19T14:01:39.077146+0000","last_unstale":"2023-10-19T14:01:40.243278+0000","last_undegraded":"2023-10-19T14:01:40.243278+0000","last_fullsized":"2023-10-19T14:01:40.243278+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:34:11.454508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.31","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.240956+0000","last_change":"2023-10-19T14:01:39.081418+0000","last_active":"2023-10-19T14:01:40.240956+0000","last_peered":"2023-10-19T14:01:40.240956+0000","last_clean":"2023-10-19T14:01:40.240956+0000","last_became_active":"2023-10-19T14:01:39.081051+0000","last_became_peered":"2023-10-19T14:01:39.081051+0000","last_unstale":"2023-10-19T14:01:40.240956+0000","last_undegraded":"2023-10-19T14:01:40.240956+0000","last_fullsized":"2023-10-19T14:01:40.240956+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:43:05.627474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.30","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.061249+0000","last_change":"2023-10-19T14:01:39.075843+0000","last_active":"2023-10-19T14:01:40.061249+0000","last_peered":"2023-10-19T14:01:40.061249+0000","last_clean":"2023-10-19T14:01:40.061249+0000","last_became_active":"2023-10-19T14:01:39.075386+0000","last_became_peered":"2023-10-19T14:01:39.075386+0000","last_unstale":"2023-10-19T14:01:40.061249+0000","last_undegraded":"2023-10-19T14:01:40.061249+0000","last_fullsized":"2023-10-19T14:01:40.061249+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:47:13.388172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.2f","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.243015+0000","last_change":"2023-10-19T14:01:39.076284+0000","last_active":"2023-10-19T14:01:40.243015+0000","last_peered":"2023-10-19T14:01:40.243015+0000","last_clean":"2023-10-19T14:01:40.243015+0000","last_became_active":"2023-10-19T14:01:39.075965+0000","last_became_peered":"2023-10-19T14:01:39.075965+0000","last_unstale":"2023-10-19T14:01:40.243015+0000","last_undegraded":"2023-10-19T14:01:40.243015+0000","last_fullsized":"2023-10-19T14:01:40.243015+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:07:41.757569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.2e","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.062463+0000","last_change":"2023-10-19T14:01:39.067100+0000","last_active":"2023-10-19T14:01:40.062463+0000","last_peered":"2023-10-19T14:01:40.062463+0000","last_clean":"2023-10-19T14:01:40.062463+0000","last_became_active":"2023-10-19T14:01:39.066796+0000","last_became_peered":"2023-10-19T14:01:39.066796+0000","last_unstale":"2023-10-19T14:01:40.062463+0000","last_undegraded":"2023-10-19T14:01:40.062463+0000","last_fullsized":"2023-10-19T14:01:40.062463+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:25:36.367891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.2d","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.243360+0000","last_change":"2023-10-19T14:01:39.082921+0000","last_active":"2023-10-19T14:01:40.243360+0000","last_peered":"2023-10-19T14:01:40.243360+0000","last_clean":"2023-10-19T14:01:40.243360+0000","last_became_active":"2023-10-19T14:01:39.082089+0000","last_became_peered":"2023-10-19T14:01:39.082089+0000","last_unstale":"2023-10-19T14:01:40.243360+0000","last_undegraded":"2023-10-19T14:01:40.243360+0000","last_fullsized":"2023-10-19T14:01:40.243360+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:54:37.937188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.2c","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.193612+0000","last_change":"2023-10-19T14:01:39.066830+0000","last_active":"2023-10-19T14:01:40.193612+0000","last_peered":"2023-10-19T14:01:40.193612+0000","last_clean":"2023-10-19T14:01:40.193612+0000","last_became_active":"2023-10-19T14:01:39.066556+0000","last_became_peered":"2023-10-19T14:01:39.066556+0000","last_unstale":"2023-10-19T14:01:40.193612+0000","last_undegraded":"2023-10-19T14:01:40.193612+0000","last_fullsized":"2023-10-19T14:01:40.193612+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:33:15.193395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.2b","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.061698+0000","last_change":"2023-10-19T14:01:39.077198+0000","last_active":"2023-10-19T14:01:40.061698+0000","last_peered":"2023-10-19T14:01:40.061698+0000","last_clean":"2023-10-19T14:01:40.061698+0000","last_became_active":"2023-10-19T14:01:39.076809+0000","last_became_peered":"2023-10-19T14:01:39.076809+0000","last_unstale":"2023-10-19T14:01:40.061698+0000","last_undegraded":"2023-10-19T14:01:40.061698+0000","last_fullsized":"2023-10-19T14:01:40.061698+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:53:06.193155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.2a","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.061339+0000","last_change":"2023-10-19T14:01:39.074528+0000","last_active":"2023-10-19T14:01:40.061339+0000","last_peered":"2023-10-19T14:01:40.061339+0000","last_clean":"2023-10-19T14:01:40.061339+0000","last_became_active":"2023-10-19T14:01:39.074278+0000","last_became_peered":"2023-10-19T14:01:39.074278+0000","last_unstale":"2023-10-19T14:01:40.061339+0000","last_undegraded":"2023-10-19T14:01:40.061339+0000","last_fullsized":"2023-10-19T14:01:40.061339+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:11:47.227977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.29","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.242695+0000","last_change":"2023-10-19T14:01:39.069413+0000","last_active":"2023-10-19T14:01:40.242695+0000","last_peered":"2023-10-19T14:01:40.242695+0000","last_clean":"2023-10-19T14:01:40.242695+0000","last_became_active":"2023-10-19T14:01:39.068967+0000","last_became_peered":"2023-10-19T14:01:39.068967+0000","last_unstale":"2023-10-19T14:01:40.242695+0000","last_undegraded":"2023-10-19T14:01:40.242695+0000","last_fullsized":"2023-10-19T14:01:40.242695+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:46:49.160506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.28","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.062627+0000","last_change":"2023-10-19T14:01:39.075350+0000","last_active":"2023-10-19T14:01:40.062627+0000","last_peered":"2023-10-19T14:01:40.062627+0000","last_clean":"2023-10-19T14:01:40.062627+0000","last_became_active":"2023-10-19T14:01:39.074936+0000","last_became_peered":"2023-10-19T14:01:39.074936+0000","last_unstale":"2023-10-19T14:01:40.062627+0000","last_undegraded":"2023-10-19T14:01:40.062627+0000","last_fullsized":"2023-10-19T14:01:40.062627+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:32:59.224807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.27","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.061670+0000","last_change":"2023-10-19T14:01:39.069828+0000","last_active":"2023-10-19T14:01:40.061670+0000","last_peered":"2023-10-19T14:01:40.061670+0000","last_clean":"2023-10-19T14:01:40.061670+0000","last_became_active":"2023-10-19T14:01:39.069531+0000","last_became_peered":"2023-10-19T14:01:39.069531+0000","last_unstale":"2023-10-19T14:01:40.061670+0000","last_undegraded":"2023-10-19T14:01:40.061670+0000","last_fullsized":"2023-10-19T14:01:40.061670+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:33:57.927060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.26","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.061556+0000","last_change":"2023-10-19T14:01:39.069124+0000","last_active":"2023-10-19T14:01:40.061556+0000","last_peered":"2023-10-19T14:01:40.061556+0000","last_clean":"2023-10-19T14:01:40.061556+0000","last_became_active":"2023-10-19T14:01:39.067572+0000","last_became_peered":"2023-10-19T14:01:39.067572+0000","last_unstale":"2023-10-19T14:01:40.061556+0000","last_undegraded":"2023-10-19T14:01:40.061556+0000","last_fullsized":"2023-10-19T14:01:40.061556+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:27:12.855540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.25","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.061502+0000","last_change":"2023-10-19T14:01:39.075348+0000","last_active":"2023-10-19T14:01:40.061502+0000","last_peered":"2023-10-19T14:01:40.061502+0000","last_clean":"2023-10-19T14:01:40.061502+0000","last_became_active":"2023-10-19T14:01:39.075053+0000","last_became_peered":"2023-10-19T14:01:39.075053+0000","last_unstale":"2023-10-19T14:01:40.061502+0000","last_undegraded":"2023-10-19T14:01:40.061502+0000","last_fullsized":"2023-10-19T14:01:40.061502+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:32:12.211178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.24","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.062677+0000","last_change":"2023-10-19T14:01:39.064768+0000","last_active":"2023-10-19T14:01:40.062677+0000","last_peered":"2023-10-19T14:01:40.062677+0000","last_clean":"2023-10-19T14:01:40.062677+0000","last_became_active":"2023-10-19T14:01:39.064331+0000","last_became_peered":"2023-10-19T14:01:39.064331+0000","last_unstale":"2023-10-19T14:01:40.062677+0000","last_undegraded":"2023-10-19T14:01:40.062677+0000","last_fullsized":"2023-10-19T14:01:40.062677+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:21:33.111873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.23","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.242991+0000","last_change":"2023-10-19T14:01:39.075985+0000","last_active":"2023-10-19T14:01:40.242991+0000","last_peered":"2023-10-19T14:01:40.242991+0000","last_clean":"2023-10-19T14:01:40.242991+0000","last_became_active":"2023-10-19T14:01:39.075344+0000","last_became_peered":"2023-10-19T14:01:39.075344+0000","last_unstale":"2023-10-19T14:01:40.242991+0000","last_undegraded":"2023-10-19T14:01:40.242991+0000","last_fullsized":"2023-10-19T14:01:40.242991+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:00:24.365723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.22","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.062576+0000","last_change":"2023-10-19T14:01:39.078224+0000","last_active":"2023-10-19T14:01:40.062576+0000","last_peered":"2023-10-19T14:01:40.062576+0000","last_clean":"2023-10-19T14:01:40.062576+0000","last_became_active":"2023-10-19T14:01:39.077926+0000","last_became_peered":"2023-10-19T14:01:39.077926+0000","last_unstale":"2023-10-19T14:01:40.062576+0000","last_undegraded":"2023-10-19T14:01:40.062576+0000","last_fullsized":"2023-10-19T14:01:40.062576+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:19:01.846781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.21","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.061377+0000","last_change":"2023-10-19T14:01:39.081882+0000","last_active":"2023-10-19T14:01:40.061377+0000","last_peered":"2023-10-19T14:01:40.061377+0000","last_clean":"2023-10-19T14:01:40.061377+0000","last_became_active":"2023-10-19T14:01:39.080530+0000","last_became_peered":"2023-10-19T14:01:39.080530+0000","last_unstale":"2023-10-19T14:01:40.061377+0000","last_undegraded":"2023-10-19T14:01:40.061377+0000","last_fullsized":"2023-10-19T14:01:40.061377+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:30:23.694882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.20","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.242096+0000","last_change":"2023-10-19T14:01:39.073673+0000","last_active":"2023-10-19T14:01:40.242096+0000","last_peered":"2023-10-19T14:01:40.242096+0000","last_clean":"2023-10-19T14:01:40.242096+0000","last_became_active":"2023-10-19T14:01:39.072708+0000","last_became_peered":"2023-10-19T14:01:39.072708+0000","last_unstale":"2023-10-19T14:01:40.242096+0000","last_undegraded":"2023-10-19T14:01:40.242096+0000","last_fullsized":"2023-10-19T14:01:40.242096+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:53:28.733545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1c","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.241224+0000","last_change":"2023-10-19T14:01:39.074536+0000","last_active":"2023-10-19T14:01:40.241224+0000","last_peered":"2023-10-19T14:01:40.241224+0000","last_clean":"2023-10-19T14:01:40.241224+0000","last_became_active":"2023-10-19T14:01:39.074257+0000","last_became_peered":"2023-10-19T14:01:39.074257+0000","last_unstale":"2023-10-19T14:01:40.241224+0000","last_undegraded":"2023-10-19T14:01:40.241224+0000","last_fullsized":"2023-10-19T14:01:40.241224+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:53:12.668683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1f","version":"174'4","reported_seq":16,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.061780+0000","last_change":"2023-10-19T14:01:38.149463+0000","last_active":"2023-10-19T14:01:40.061780+0000","last_peered":"2023-10-19T14:01:40.061780+0000","last_clean":"2023-10-19T14:01:40.061780+0000","last_became_active":"2023-10-19T14:01:38.149187+0000","last_became_peered":"2023-10-19T14:01:38.149187+0000","last_unstale":"2023-10-19T14:01:40.061780+0000","last_undegraded":"2023-10-19T14:01:40.061780+0000","last_fullsized":"2023-10-19T14:01:40.061780+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_clean_scrub_stamp":"2023-10-19T14:01:37.123936+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:32:28.166385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1d","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.241150+0000","last_change":"2023-10-19T14:01:39.080424+0000","last_active":"2023-10-19T14:01:40.241150+0000","last_peered":"2023-10-19T14:01:40.241150+0000","last_clean":"2023-10-19T14:01:40.241150+0000","last_became_active":"2023-10-19T14:01:39.080138+0000","last_became_peered":"2023-10-19T14:01:39.080138+0000","last_unstale":"2023-10-19T14:01:40.241150+0000","last_undegraded":"2023-10-19T14:01:40.241150+0000","last_fullsized":"2023-10-19T14:01:40.241150+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:08:22.714158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1e","version":"174'1","reported_seq":13,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.243017+0000","last_change":"2023-10-19T14:01:38.155102+0000","last_active":"2023-10-19T14:01:40.243017+0000","last_peered":"2023-10-19T14:01:40.243017+0000","last_clean":"2023-10-19T14:01:40.243017+0000","last_became_active":"2023-10-19T14:01:38.154849+0000","last_became_peered":"2023-10-19T14:01:38.154849+0000","last_unstale":"2023-10-19T14:01:40.243017+0000","last_undegraded":"2023-10-19T14:01:40.243017+0000","last_fullsized":"2023-10-19T14:01:40.243017+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_clean_scrub_stamp":"2023-10-19T14:01:37.123936+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:27:11.849330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1e","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.242554+0000","last_change":"2023-10-19T14:01:39.079726+0000","last_active":"2023-10-19T14:01:40.242554+0000","last_peered":"2023-10-19T14:01:40.242554+0000","last_clean":"2023-10-19T14:01:40.242554+0000","last_became_active":"2023-10-19T14:01:39.079439+0000","last_became_peered":"2023-10-19T14:01:39.079439+0000","last_unstale":"2023-10-19T14:01:40.242554+0000","last_undegraded":"2023-10-19T14:01:40.242554+0000","last_fullsized":"2023-10-19T14:01:40.242554+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:10:18.170429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1d","version":"174'2","reported_seq":14,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.243437+0000","last_change":"2023-10-19T14:01:38.154123+0000","last_active":"2023-10-19T14:01:40.243437+0000","last_peered":"2023-10-19T14:01:40.243437+0000","last_clean":"2023-10-19T14:01:40.243437+0000","last_became_active":"2023-10-19T14:01:38.153860+0000","last_became_peered":"2023-10-19T14:01:38.153860+0000","last_unstale":"2023-10-19T14:01:40.243437+0000","last_undegraded":"2023-10-19T14:01:40.243437+0000","last_fullsized":"2023-10-19T14:01:40.243437+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_clean_scrub_stamp":"2023-10-19T14:01:37.123936+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:25:57.506648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1f","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.243438+0000","last_change":"2023-10-19T14:01:39.081167+0000","last_active":"2023-10-19T14:01:40.243438+0000","last_peered":"2023-10-19T14:01:40.243438+0000","last_clean":"2023-10-19T14:01:40.243438+0000","last_became_active":"2023-10-19T14:01:39.080131+0000","last_became_peered":"2023-10-19T14:01:39.080131+0000","last_unstale":"2023-10-19T14:01:40.243438+0000","last_undegraded":"2023-10-19T14:01:40.243438+0000","last_fullsized":"2023-10-19T14:01:40.243438+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T02:00:46.582226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1c","version":"174'2","reported_seq":14,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.241213+0000","last_change":"2023-10-19T14:01:38.154468+0000","last_active":"2023-10-19T14:01:40.241213+0000","last_peered":"2023-10-19T14:01:40.241213+0000","last_clean":"2023-10-19T14:01:40.241213+0000","last_became_active":"2023-10-19T14:01:38.154132+0000","last_became_peered":"2023-10-19T14:01:38.154132+0000","last_unstale":"2023-10-19T14:01:40.241213+0000","last_undegraded":"2023-10-19T14:01:40.241213+0000","last_fullsized":"2023-10-19T14:01:40.241213+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_clean_scrub_stamp":"2023-10-19T14:01:37.123936+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:57:34.773001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.193615+0000","last_change":"2023-10-19T14:01:39.066422+0000","last_active":"2023-10-19T14:01:40.193615+0000","last_peered":"2023-10-19T14:01:40.193615+0000","last_clean":"2023-10-19T14:01:40.193615+0000","last_became_active":"2023-10-19T14:01:39.065943+0000","last_became_peered":"2023-10-19T14:01:39.065943+0000","last_unstale":"2023-10-19T14:01:40.193615+0000","last_undegraded":"2023-10-19T14:01:40.193615+0000","last_fullsized":"2023-10-19T14:01:40.193615+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:02:51.348060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.1b","version":"174'1","reported_seq":13,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.243005+0000","last_change":"2023-10-19T14:01:38.154709+0000","last_active":"2023-10-19T14:01:40.243005+0000","last_peered":"2023-10-19T14:01:40.243005+0000","last_clean":"2023-10-19T14:01:40.243005+0000","last_became_active":"2023-10-19T14:01:38.153296+0000","last_became_peered":"2023-10-19T14:01:38.153296+0000","last_unstale":"2023-10-19T14:01:40.243005+0000","last_undegraded":"2023-10-19T14:01:40.243005+0000","last_fullsized":"2023-10-19T14:01:40.243005+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_clean_scrub_stamp":"2023-10-19T14:01:37.123936+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:10:51.292543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.241912+0000","last_change":"2023-10-19T14:01:39.076795+0000","last_active":"2023-10-19T14:01:40.241912+0000","last_peered":"2023-10-19T14:01:40.241912+0000","last_clean":"2023-10-19T14:01:40.241912+0000","last_became_active":"2023-10-19T14:01:39.076254+0000","last_became_peered":"2023-10-19T14:01:39.076254+0000","last_unstale":"2023-10-19T14:01:40.241912+0000","last_undegraded":"2023-10-19T14:01:40.241912+0000","last_fullsized":"2023-10-19T14:01:40.241912+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:14:16.050479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1a","version":"174'1","reported_seq":13,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.061708+0000","last_change":"2023-10-19T14:01:38.153181+0000","last_active":"2023-10-19T14:01:40.061708+0000","last_peered":"2023-10-19T14:01:40.061708+0000","last_clean":"2023-10-19T14:01:40.061708+0000","last_became_active":"2023-10-19T14:01:38.152855+0000","last_became_peered":"2023-10-19T14:01:38.152855+0000","last_unstale":"2023-10-19T14:01:40.061708+0000","last_undegraded":"2023-10-19T14:01:40.061708+0000","last_fullsized":"2023-10-19T14:01:40.061708+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_clean_scrub_stamp":"2023-10-19T14:01:37.123936+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:20:42.107511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1a","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.240973+0000","last_change":"2023-10-19T14:01:39.067123+0000","last_active":"2023-10-19T14:01:40.240973+0000","last_peered":"2023-10-19T14:01:40.240973+0000","last_clean":"2023-10-19T14:01:40.240973+0000","last_became_active":"2023-10-19T14:01:39.066843+0000","last_became_peered":"2023-10-19T14:01:39.066843+0000","last_unstale":"2023-10-19T14:01:40.240973+0000","last_undegraded":"2023-10-19T14:01:40.240973+0000","last_fullsized":"2023-10-19T14:01:40.240973+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:28:31.752973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":12,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.061406+0000","last_change":"2023-10-19T14:01:38.148197+0000","last_active":"2023-10-19T14:01:40.061406+0000","last_peered":"2023-10-19T14:01:40.061406+0000","last_clean":"2023-10-19T14:01:40.061406+0000","last_became_active":"2023-10-19T14:01:38.147899+0000","last_became_peered":"2023-10-19T14:01:38.147899+0000","last_unstale":"2023-10-19T14:01:40.061406+0000","last_undegraded":"2023-10-19T14:01:40.061406+0000","last_fullsized":"2023-10-19T14:01:40.061406+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_clean_scrub_stamp":"2023-10-19T14:01:37.123936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:09:27.370187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1b","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.061374+0000","last_change":"2023-10-19T14:01:39.073790+0000","last_active":"2023-10-19T14:01:40.061374+0000","last_peered":"2023-10-19T14:01:40.061374+0000","last_clean":"2023-10-19T14:01:40.061374+0000","last_became_active":"2023-10-19T14:01:39.073530+0000","last_became_peered":"2023-10-19T14:01:39.073530+0000","last_unstale":"2023-10-19T14:01:40.061374+0000","last_undegraded":"2023-10-19T14:01:40.061374+0000","last_fullsized":"2023-10-19T14:01:40.061374+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:04:24.165469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":12,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.061278+0000","last_change":"2023-10-19T14:01:38.149972+0000","last_active":"2023-10-19T14:01:40.061278+0000","last_peered":"2023-10-19T14:01:40.061278+0000","last_clean":"2023-10-19T14:01:40.061278+0000","last_became_active":"2023-10-19T14:01:38.149689+0000","last_became_peered":"2023-10-19T14:01:38.149689+0000","last_unstale":"2023-10-19T14:01:40.061278+0000","last_undegraded":"2023-10-19T14:01:40.061278+0000","last_fullsized":"2023-10-19T14:01:40.061278+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_clean_scrub_stamp":"2023-10-19T14:01:37.123936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:28:56.955098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.14","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.243095+0000","last_change":"2023-10-19T14:01:39.074490+0000","last_active":"2023-10-19T14:01:40.243095+0000","last_peered":"2023-10-19T14:01:40.243095+0000","last_clean":"2023-10-19T14:01:40.243095+0000","last_became_active":"2023-10-19T14:01:39.073165+0000","last_became_peered":"2023-10-19T14:01:39.073165+0000","last_unstale":"2023-10-19T14:01:40.243095+0000","last_undegraded":"2023-10-19T14:01:40.243095+0000","last_fullsized":"2023-10-19T14:01:40.243095+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:10:47.398271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":12,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.243541+0000","last_change":"2023-10-19T14:01:38.153657+0000","last_active":"2023-10-19T14:01:40.243541+0000","last_peered":"2023-10-19T14:01:40.243541+0000","last_clean":"2023-10-19T14:01:40.243541+0000","last_became_active":"2023-10-19T14:01:38.153388+0000","last_became_peered":"2023-10-19T14:01:38.153388+0000","last_unstale":"2023-10-19T14:01:40.243541+0000","last_undegraded":"2023-10-19T14:01:40.243541+0000","last_fullsized":"2023-10-19T14:01:40.243541+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_clean_scrub_stamp":"2023-10-19T14:01:37.123936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:11:46.221768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.15","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.062503+0000","last_change":"2023-10-19T14:01:39.075641+0000","last_active":"2023-10-19T14:01:40.062503+0000","last_peered":"2023-10-19T14:01:40.062503+0000","last_clean":"2023-10-19T14:01:40.062503+0000","last_became_active":"2023-10-19T14:01:39.075259+0000","last_became_peered":"2023-10-19T14:01:39.075259+0000","last_unstale":"2023-10-19T14:01:40.062503+0000","last_undegraded":"2023-10-19T14:01:40.062503+0000","last_fullsized":"2023-10-19T14:01:40.062503+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:24:46.342060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":12,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.061731+0000","last_change":"2023-10-19T14:01:38.149138+0000","last_active":"2023-10-19T14:01:40.061731+0000","last_peered":"2023-10-19T14:01:40.061731+0000","last_clean":"2023-10-19T14:01:40.061731+0000","last_became_active":"2023-10-19T14:01:38.148787+0000","last_became_peered":"2023-10-19T14:01:38.148787+0000","last_unstale":"2023-10-19T14:01:40.061731+0000","last_undegraded":"2023-10-19T14:01:40.061731+0000","last_fullsized":"2023-10-19T14:01:40.061731+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_clean_scrub_stamp":"2023-10-19T14:01:37.123936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:09:19.791765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.062605+0000","last_change":"2023-10-19T14:01:39.066640+0000","last_active":"2023-10-19T14:01:40.062605+0000","last_peered":"2023-10-19T14:01:40.062605+0000","last_clean":"2023-10-19T14:01:40.062605+0000","last_became_active":"2023-10-19T14:01:39.066237+0000","last_became_peered":"2023-10-19T14:01:39.066237+0000","last_unstale":"2023-10-19T14:01:40.062605+0000","last_undegraded":"2023-10-19T14:01:40.062605+0000","last_fullsized":"2023-10-19T14:01:40.062605+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:04:01.542818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.15","version":"174'3","reported_seq":15,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.062616+0000","last_change":"2023-10-19T14:01:38.151310+0000","last_active":"2023-10-19T14:01:40.062616+0000","last_peered":"2023-10-19T14:01:40.062616+0000","last_clean":"2023-10-19T14:01:40.062616+0000","last_became_active":"2023-10-19T14:01:38.150998+0000","last_became_peered":"2023-10-19T14:01:38.150998+0000","last_unstale":"2023-10-19T14:01:40.062616+0000","last_undegraded":"2023-10-19T14:01:40.062616+0000","last_fullsized":"2023-10-19T14:01:40.062616+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_clean_scrub_stamp":"2023-10-19T14:01:37.123936+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:05:26.982257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.10","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.242191+0000","last_change":"2023-10-19T14:01:39.078223+0000","last_active":"2023-10-19T14:01:40.242191+0000","last_peered":"2023-10-19T14:01:40.242191+0000","last_clean":"2023-10-19T14:01:40.242191+0000","last_became_active":"2023-10-19T14:01:39.077960+0000","last_became_peered":"2023-10-19T14:01:39.077960+0000","last_unstale":"2023-10-19T14:01:40.242191+0000","last_undegraded":"2023-10-19T14:01:40.242191+0000","last_fullsized":"2023-10-19T14:01:40.242191+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:40:26.710084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.13","version":"174'2","reported_seq":14,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.061914+0000","last_change":"2023-10-19T14:01:38.150112+0000","last_active":"2023-10-19T14:01:40.061914+0000","last_peered":"2023-10-19T14:01:40.061914+0000","last_clean":"2023-10-19T14:01:40.061914+0000","last_became_active":"2023-10-19T14:01:38.149218+0000","last_became_peered":"2023-10-19T14:01:38.149218+0000","last_unstale":"2023-10-19T14:01:40.061914+0000","last_undegraded":"2023-10-19T14:01:40.061914+0000","last_fullsized":"2023-10-19T14:01:40.061914+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_clean_scrub_stamp":"2023-10-19T14:01:37.123936+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:31:42.990421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.241235+0000","last_change":"2023-10-19T14:01:39.081126+0000","last_active":"2023-10-19T14:01:40.241235+0000","last_peered":"2023-10-19T14:01:40.241235+0000","last_clean":"2023-10-19T14:01:40.241235+0000","last_became_active":"2023-10-19T14:01:39.080597+0000","last_became_peered":"2023-10-19T14:01:39.080597+0000","last_unstale":"2023-10-19T14:01:40.241235+0000","last_undegraded":"2023-10-19T14:01:40.241235+0000","last_fullsized":"2023-10-19T14:01:40.241235+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:14:21.486364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":12,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.242645+0000","last_change":"2023-10-19T14:01:38.156939+0000","last_active":"2023-10-19T14:01:40.242645+0000","last_peered":"2023-10-19T14:01:40.242645+0000","last_clean":"2023-10-19T14:01:40.242645+0000","last_became_active":"2023-10-19T14:01:38.156654+0000","last_became_peered":"2023-10-19T14:01:38.156654+0000","last_unstale":"2023-10-19T14:01:40.242645+0000","last_undegraded":"2023-10-19T14:01:40.242645+0000","last_fullsized":"2023-10-19T14:01:40.242645+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_clean_scrub_stamp":"2023-10-19T14:01:37.123936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:27:32.578234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.061693+0000","last_change":"2023-10-19T14:01:39.072644+0000","last_active":"2023-10-19T14:01:40.061693+0000","last_peered":"2023-10-19T14:01:40.061693+0000","last_clean":"2023-10-19T14:01:40.061693+0000","last_became_active":"2023-10-19T14:01:39.072301+0000","last_became_peered":"2023-10-19T14:01:39.072301+0000","last_unstale":"2023-10-19T14:01:40.061693+0000","last_undegraded":"2023-10-19T14:01:40.061693+0000","last_fullsized":"2023-10-19T14:01:40.061693+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:49:07.317364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":12,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.242002+0000","last_change":"2023-10-19T14:01:38.152957+0000","last_active":"2023-10-19T14:01:40.242002+0000","last_peered":"2023-10-19T14:01:40.242002+0000","last_clean":"2023-10-19T14:01:40.242002+0000","last_became_active":"2023-10-19T14:01:38.152708+0000","last_became_peered":"2023-10-19T14:01:38.152708+0000","last_unstale":"2023-10-19T14:01:40.242002+0000","last_undegraded":"2023-10-19T14:01:40.242002+0000","last_fullsized":"2023-10-19T14:01:40.242002+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_clean_scrub_stamp":"2023-10-19T14:01:37.123936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:43:47.404093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.13","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.240990+0000","last_change":"2023-10-19T14:01:39.074287+0000","last_active":"2023-10-19T14:01:40.240990+0000","last_peered":"2023-10-19T14:01:40.240990+0000","last_clean":"2023-10-19T14:01:40.240990+0000","last_became_active":"2023-10-19T14:01:39.073962+0000","last_became_peered":"2023-10-19T14:01:39.073962+0000","last_unstale":"2023-10-19T14:01:40.240990+0000","last_undegraded":"2023-10-19T14:01:40.240990+0000","last_fullsized":"2023-10-19T14:01:40.240990+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:59:15.579667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.10","version":"174'8","reported_seq":20,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.061096+0000","last_change":"2023-10-19T14:01:38.150008+0000","last_active":"2023-10-19T14:01:40.061096+0000","last_peered":"2023-10-19T14:01:40.061096+0000","last_clean":"2023-10-19T14:01:40.061096+0000","last_became_active":"2023-10-19T14:01:38.149708+0000","last_became_peered":"2023-10-19T14:01:38.149708+0000","last_unstale":"2023-10-19T14:01:40.061096+0000","last_undegraded":"2023-10-19T14:01:40.061096+0000","last_fullsized":"2023-10-19T14:01:40.061096+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_clean_scrub_stamp":"2023-10-19T14:01:37.123936+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:22:09.761426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.242341+0000","last_change":"2023-10-19T14:01:39.076798+0000","last_active":"2023-10-19T14:01:40.242341+0000","last_peered":"2023-10-19T14:01:40.242341+0000","last_clean":"2023-10-19T14:01:40.242341+0000","last_became_active":"2023-10-19T14:01:39.075966+0000","last_became_peered":"2023-10-19T14:01:39.075966+0000","last_unstale":"2023-10-19T14:01:40.242341+0000","last_undegraded":"2023-10-19T14:01:40.242341+0000","last_fullsized":"2023-10-19T14:01:40.242341+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:28:46.198601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.f","version":"174'1","reported_seq":13,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.062467+0000","last_change":"2023-10-19T14:01:38.150116+0000","last_active":"2023-10-19T14:01:40.062467+0000","last_peered":"2023-10-19T14:01:40.062467+0000","last_clean":"2023-10-19T14:01:40.062467+0000","last_became_active":"2023-10-19T14:01:38.149778+0000","last_became_peered":"2023-10-19T14:01:38.149778+0000","last_unstale":"2023-10-19T14:01:40.062467+0000","last_undegraded":"2023-10-19T14:01:40.062467+0000","last_fullsized":"2023-10-19T14:01:40.062467+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_clean_scrub_stamp":"2023-10-19T14:01:37.123936+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:46:09.489532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.d","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.243391+0000","last_change":"2023-10-19T14:01:39.082935+0000","last_active":"2023-10-19T14:01:40.243391+0000","last_peered":"2023-10-19T14:01:40.243391+0000","last_clean":"2023-10-19T14:01:40.243391+0000","last_became_active":"2023-10-19T14:01:39.082359+0000","last_became_peered":"2023-10-19T14:01:39.082359+0000","last_unstale":"2023-10-19T14:01:40.243391+0000","last_undegraded":"2023-10-19T14:01:40.243391+0000","last_fullsized":"2023-10-19T14:01:40.243391+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:29:58.142789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.e","version":"174'2","reported_seq":14,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.243125+0000","last_change":"2023-10-19T14:01:38.153428+0000","last_active":"2023-10-19T14:01:40.243125+0000","last_peered":"2023-10-19T14:01:40.243125+0000","last_clean":"2023-10-19T14:01:40.243125+0000","last_became_active":"2023-10-19T14:01:38.153155+0000","last_became_peered":"2023-10-19T14:01:38.153155+0000","last_unstale":"2023-10-19T14:01:40.243125+0000","last_undegraded":"2023-10-19T14:01:40.243125+0000","last_fullsized":"2023-10-19T14:01:40.243125+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_clean_scrub_stamp":"2023-10-19T14:01:37.123936+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:20:27.265338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.e","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.061659+0000","last_change":"2023-10-19T14:01:39.078154+0000","last_active":"2023-10-19T14:01:40.061659+0000","last_peered":"2023-10-19T14:01:40.061659+0000","last_clean":"2023-10-19T14:01:40.061659+0000","last_became_active":"2023-10-19T14:01:39.077803+0000","last_became_peered":"2023-10-19T14:01:39.077803+0000","last_unstale":"2023-10-19T14:01:40.061659+0000","last_undegraded":"2023-10-19T14:01:40.061659+0000","last_fullsized":"2023-10-19T14:01:40.061659+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:17:07.521687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.d","version":"174'1","reported_seq":13,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.241171+0000","last_change":"2023-10-19T14:01:38.153632+0000","last_active":"2023-10-19T14:01:40.241171+0000","last_peered":"2023-10-19T14:01:40.241171+0000","last_clean":"2023-10-19T14:01:40.241171+0000","last_became_active":"2023-10-19T14:01:38.153374+0000","last_became_peered":"2023-10-19T14:01:38.153374+0000","last_unstale":"2023-10-19T14:01:40.241171+0000","last_undegraded":"2023-10-19T14:01:40.241171+0000","last_fullsized":"2023-10-19T14:01:40.241171+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_clean_scrub_stamp":"2023-10-19T14:01:37.123936+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:19:21.478658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.f","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.061761+0000","last_change":"2023-10-19T14:01:39.078701+0000","last_active":"2023-10-19T14:01:40.061761+0000","last_peered":"2023-10-19T14:01:40.061761+0000","last_clean":"2023-10-19T14:01:40.061761+0000","last_became_active":"2023-10-19T14:01:39.078452+0000","last_became_peered":"2023-10-19T14:01:39.078452+0000","last_unstale":"2023-10-19T14:01:40.061761+0000","last_undegraded":"2023-10-19T14:01:40.061761+0000","last_fullsized":"2023-10-19T14:01:40.061761+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:46:36.953152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":12,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.242704+0000","last_change":"2023-10-19T14:01:38.156924+0000","last_active":"2023-10-19T14:01:40.242704+0000","last_peered":"2023-10-19T14:01:40.242704+0000","last_clean":"2023-10-19T14:01:40.242704+0000","last_became_active":"2023-10-19T14:01:38.156662+0000","last_became_peered":"2023-10-19T14:01:38.156662+0000","last_unstale":"2023-10-19T14:01:40.242704+0000","last_undegraded":"2023-10-19T14:01:40.242704+0000","last_fullsized":"2023-10-19T14:01:40.242704+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_clean_scrub_stamp":"2023-10-19T14:01:37.123936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:46:52.638213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.241935+0000","last_change":"2023-10-19T14:01:39.080332+0000","last_active":"2023-10-19T14:01:40.241935+0000","last_peered":"2023-10-19T14:01:40.241935+0000","last_clean":"2023-10-19T14:01:40.241935+0000","last_became_active":"2023-10-19T14:01:39.079874+0000","last_became_peered":"2023-10-19T14:01:39.079874+0000","last_unstale":"2023-10-19T14:01:40.241935+0000","last_undegraded":"2023-10-19T14:01:40.241935+0000","last_fullsized":"2023-10-19T14:01:40.241935+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:29:32.305082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.b","version":"174'2","reported_seq":14,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.193612+0000","last_change":"2023-10-19T14:01:38.155709+0000","last_active":"2023-10-19T14:01:40.193612+0000","last_peered":"2023-10-19T14:01:40.193612+0000","last_clean":"2023-10-19T14:01:40.193612+0000","last_became_active":"2023-10-19T14:01:38.155447+0000","last_became_peered":"2023-10-19T14:01:38.155447+0000","last_unstale":"2023-10-19T14:01:40.193612+0000","last_undegraded":"2023-10-19T14:01:40.193612+0000","last_fullsized":"2023-10-19T14:01:40.193612+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_clean_scrub_stamp":"2023-10-19T14:01:37.123936+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:38:28.395781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.241104+0000","last_change":"2023-10-19T14:01:39.068151+0000","last_active":"2023-10-19T14:01:40.241104+0000","last_peered":"2023-10-19T14:01:40.241104+0000","last_clean":"2023-10-19T14:01:40.241104+0000","last_became_active":"2023-10-19T14:01:39.067791+0000","last_became_peered":"2023-10-19T14:01:39.067791+0000","last_unstale":"2023-10-19T14:01:40.241104+0000","last_undegraded":"2023-10-19T14:01:40.241104+0000","last_fullsized":"2023-10-19T14:01:40.241104+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:04:04.560169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":12,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.242215+0000","last_change":"2023-10-19T14:01:38.151527+0000","last_active":"2023-10-19T14:01:40.242215+0000","last_peered":"2023-10-19T14:01:40.242215+0000","last_clean":"2023-10-19T14:01:40.242215+0000","last_became_active":"2023-10-19T14:01:38.151259+0000","last_became_peered":"2023-10-19T14:01:38.151259+0000","last_unstale":"2023-10-19T14:01:40.242215+0000","last_undegraded":"2023-10-19T14:01:40.242215+0000","last_fullsized":"2023-10-19T14:01:40.242215+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_clean_scrub_stamp":"2023-10-19T14:01:37.123936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:33:19.331015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.242156+0000","last_change":"2023-10-19T14:01:39.081972+0000","last_active":"2023-10-19T14:01:40.242156+0000","last_peered":"2023-10-19T14:01:40.242156+0000","last_clean":"2023-10-19T14:01:40.242156+0000","last_became_active":"2023-10-19T14:01:39.081551+0000","last_became_peered":"2023-10-19T14:01:39.081551+0000","last_unstale":"2023-10-19T14:01:40.242156+0000","last_undegraded":"2023-10-19T14:01:40.242156+0000","last_fullsized":"2023-10-19T14:01:40.242156+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:55:59.374396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":12,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.193453+0000","last_change":"2023-10-19T14:01:38.154227+0000","last_active":"2023-10-19T14:01:40.193453+0000","last_peered":"2023-10-19T14:01:40.193453+0000","last_clean":"2023-10-19T14:01:40.193453+0000","last_became_active":"2023-10-19T14:01:38.153946+0000","last_became_peered":"2023-10-19T14:01:38.153946+0000","last_unstale":"2023-10-19T14:01:40.193453+0000","last_undegraded":"2023-10-19T14:01:40.193453+0000","last_fullsized":"2023-10-19T14:01:40.193453+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_clean_scrub_stamp":"2023-10-19T14:01:37.123936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:23:58.111924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.b","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.240979+0000","last_change":"2023-10-19T14:01:39.081245+0000","last_active":"2023-10-19T14:01:40.240979+0000","last_peered":"2023-10-19T14:01:40.240979+0000","last_clean":"2023-10-19T14:01:40.240979+0000","last_became_active":"2023-10-19T14:01:39.080944+0000","last_became_peered":"2023-10-19T14:01:39.080944+0000","last_unstale":"2023-10-19T14:01:40.240979+0000","last_undegraded":"2023-10-19T14:01:40.240979+0000","last_fullsized":"2023-10-19T14:01:40.240979+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:33:18.166098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":12,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.242431+0000","last_change":"2023-10-19T14:01:38.154468+0000","last_active":"2023-10-19T14:01:40.242431+0000","last_peered":"2023-10-19T14:01:40.242431+0000","last_clean":"2023-10-19T14:01:40.242431+0000","last_became_active":"2023-10-19T14:01:38.154203+0000","last_became_peered":"2023-10-19T14:01:38.154203+0000","last_unstale":"2023-10-19T14:01:40.242431+0000","last_undegraded":"2023-10-19T14:01:40.242431+0000","last_fullsized":"2023-10-19T14:01:40.242431+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_clean_scrub_stamp":"2023-10-19T14:01:37.123936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:14:53.849183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.061623+0000","last_change":"2023-10-19T14:01:39.080523+0000","last_active":"2023-10-19T14:01:40.061623+0000","last_peered":"2023-10-19T14:01:40.061623+0000","last_clean":"2023-10-19T14:01:40.061623+0000","last_became_active":"2023-10-19T14:01:39.079542+0000","last_became_peered":"2023-10-19T14:01:39.079542+0000","last_unstale":"2023-10-19T14:01:40.061623+0000","last_undegraded":"2023-10-19T14:01:40.061623+0000","last_fullsized":"2023-10-19T14:01:40.061623+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:00:06.386114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.7","version":"174'2","reported_seq":14,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.243109+0000","last_change":"2023-10-19T14:01:38.158093+0000","last_active":"2023-10-19T14:01:40.243109+0000","last_peered":"2023-10-19T14:01:40.243109+0000","last_clean":"2023-10-19T14:01:40.243109+0000","last_became_active":"2023-10-19T14:01:38.157861+0000","last_became_peered":"2023-10-19T14:01:38.157861+0000","last_unstale":"2023-10-19T14:01:40.243109+0000","last_undegraded":"2023-10-19T14:01:40.243109+0000","last_fullsized":"2023-10-19T14:01:40.243109+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_clean_scrub_stamp":"2023-10-19T14:01:37.123936+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:05:00.875663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.242829+0000","last_change":"2023-10-19T14:01:39.076188+0000","last_active":"2023-10-19T14:01:40.242829+0000","last_peered":"2023-10-19T14:01:40.242829+0000","last_clean":"2023-10-19T14:01:40.242829+0000","last_became_active":"2023-10-19T14:01:39.075908+0000","last_became_peered":"2023-10-19T14:01:39.075908+0000","last_unstale":"2023-10-19T14:01:40.242829+0000","last_undegraded":"2023-10-19T14:01:40.242829+0000","last_fullsized":"2023-10-19T14:01:40.242829+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:48:55.738928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.6","version":"0'0","reported_seq":12,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.061795+0000","last_change":"2023-10-19T14:01:38.149636+0000","last_active":"2023-10-19T14:01:40.061795+0000","last_peered":"2023-10-19T14:01:40.061795+0000","last_clean":"2023-10-19T14:01:40.061795+0000","last_became_active":"2023-10-19T14:01:38.149360+0000","last_became_peered":"2023-10-19T14:01:38.149360+0000","last_unstale":"2023-10-19T14:01:40.061795+0000","last_undegraded":"2023-10-19T14:01:40.061795+0000","last_fullsized":"2023-10-19T14:01:40.061795+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_clean_scrub_stamp":"2023-10-19T14:01:37.123936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:30:40.775839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.061706+0000","last_change":"2023-10-19T14:01:39.072854+0000","last_active":"2023-10-19T14:01:40.061706+0000","last_peered":"2023-10-19T14:01:40.061706+0000","last_clean":"2023-10-19T14:01:40.061706+0000","last_became_active":"2023-10-19T14:01:39.072507+0000","last_became_peered":"2023-10-19T14:01:39.072507+0000","last_unstale":"2023-10-19T14:01:40.061706+0000","last_undegraded":"2023-10-19T14:01:40.061706+0000","last_fullsized":"2023-10-19T14:01:40.061706+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:53:25.023783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":12,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.241239+0000","last_change":"2023-10-19T14:01:38.152825+0000","last_active":"2023-10-19T14:01:40.241239+0000","last_peered":"2023-10-19T14:01:40.241239+0000","last_clean":"2023-10-19T14:01:40.241239+0000","last_became_active":"2023-10-19T14:01:38.152533+0000","last_became_peered":"2023-10-19T14:01:38.152533+0000","last_unstale":"2023-10-19T14:01:40.241239+0000","last_undegraded":"2023-10-19T14:01:40.241239+0000","last_fullsized":"2023-10-19T14:01:40.241239+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_clean_scrub_stamp":"2023-10-19T14:01:37.123936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:33:51.860498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.7","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.061688+0000","last_change":"2023-10-19T14:01:39.078634+0000","last_active":"2023-10-19T14:01:40.061688+0000","last_peered":"2023-10-19T14:01:40.061688+0000","last_clean":"2023-10-19T14:01:40.061688+0000","last_became_active":"2023-10-19T14:01:39.078348+0000","last_became_peered":"2023-10-19T14:01:39.078348+0000","last_unstale":"2023-10-19T14:01:40.061688+0000","last_undegraded":"2023-10-19T14:01:40.061688+0000","last_fullsized":"2023-10-19T14:01:40.061688+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:49:05.321824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":12,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.061707+0000","last_change":"2023-10-19T14:01:38.155739+0000","last_active":"2023-10-19T14:01:40.061707+0000","last_peered":"2023-10-19T14:01:40.061707+0000","last_clean":"2023-10-19T14:01:40.061707+0000","last_became_active":"2023-10-19T14:01:38.155445+0000","last_became_peered":"2023-10-19T14:01:38.155445+0000","last_unstale":"2023-10-19T14:01:40.061707+0000","last_undegraded":"2023-10-19T14:01:40.061707+0000","last_fullsized":"2023-10-19T14:01:40.061707+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_clean_scrub_stamp":"2023-10-19T14:01:37.123936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:58:02.891758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.242337+0000","last_change":"2023-10-19T14:01:39.073851+0000","last_active":"2023-10-19T14:01:40.242337+0000","last_peered":"2023-10-19T14:01:40.242337+0000","last_clean":"2023-10-19T14:01:40.242337+0000","last_became_active":"2023-10-19T14:01:39.073027+0000","last_became_peered":"2023-10-19T14:01:39.073027+0000","last_unstale":"2023-10-19T14:01:40.242337+0000","last_undegraded":"2023-10-19T14:01:40.242337+0000","last_fullsized":"2023-10-19T14:01:40.242337+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:42:59.302185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":12,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.061382+0000","last_change":"2023-10-19T14:01:38.153359+0000","last_active":"2023-10-19T14:01:40.061382+0000","last_peered":"2023-10-19T14:01:40.061382+0000","last_clean":"2023-10-19T14:01:40.061382+0000","last_became_active":"2023-10-19T14:01:38.153078+0000","last_became_peered":"2023-10-19T14:01:38.153078+0000","last_unstale":"2023-10-19T14:01:40.061382+0000","last_undegraded":"2023-10-19T14:01:40.061382+0000","last_fullsized":"2023-10-19T14:01:40.061382+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_clean_scrub_stamp":"2023-10-19T14:01:37.123936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:33:03.326318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.242083+0000","last_change":"2023-10-19T14:01:39.076610+0000","last_active":"2023-10-19T14:01:40.242083+0000","last_peered":"2023-10-19T14:01:40.242083+0000","last_clean":"2023-10-19T14:01:40.242083+0000","last_became_active":"2023-10-19T14:01:39.075928+0000","last_became_peered":"2023-10-19T14:01:39.075928+0000","last_unstale":"2023-10-19T14:01:40.242083+0000","last_undegraded":"2023-10-19T14:01:40.242083+0000","last_fullsized":"2023-10-19T14:01:40.242083+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:57:05.135549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":12,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.062563+0000","last_change":"2023-10-19T14:01:38.152797+0000","last_active":"2023-10-19T14:01:40.062563+0000","last_peered":"2023-10-19T14:01:40.062563+0000","last_clean":"2023-10-19T14:01:40.062563+0000","last_became_active":"2023-10-19T14:01:38.152527+0000","last_became_peered":"2023-10-19T14:01:38.152527+0000","last_unstale":"2023-10-19T14:01:40.062563+0000","last_undegraded":"2023-10-19T14:01:40.062563+0000","last_fullsized":"2023-10-19T14:01:40.062563+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_clean_scrub_stamp":"2023-10-19T14:01:37.123936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:04:24.064005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.241128+0000","last_change":"2023-10-19T14:01:39.070849+0000","last_active":"2023-10-19T14:01:40.241128+0000","last_peered":"2023-10-19T14:01:40.241128+0000","last_clean":"2023-10-19T14:01:40.241128+0000","last_became_active":"2023-10-19T14:01:39.070554+0000","last_became_peered":"2023-10-19T14:01:39.070554+0000","last_unstale":"2023-10-19T14:01:40.241128+0000","last_undegraded":"2023-10-19T14:01:40.241128+0000","last_fullsized":"2023-10-19T14:01:40.241128+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:35:20.513587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":12,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.061446+0000","last_change":"2023-10-19T14:01:38.149257+0000","last_active":"2023-10-19T14:01:40.061446+0000","last_peered":"2023-10-19T14:01:40.061446+0000","last_clean":"2023-10-19T14:01:40.061446+0000","last_became_active":"2023-10-19T14:01:38.148973+0000","last_became_peered":"2023-10-19T14:01:38.148973+0000","last_unstale":"2023-10-19T14:01:40.061446+0000","last_undegraded":"2023-10-19T14:01:40.061446+0000","last_fullsized":"2023-10-19T14:01:40.061446+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_clean_scrub_stamp":"2023-10-19T14:01:37.123936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:08:54.567239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.242857+0000","last_change":"2023-10-19T14:01:39.075140+0000","last_active":"2023-10-19T14:01:40.242857+0000","last_peered":"2023-10-19T14:01:40.242857+0000","last_clean":"2023-10-19T14:01:40.242857+0000","last_became_active":"2023-10-19T14:01:39.074888+0000","last_became_peered":"2023-10-19T14:01:39.074888+0000","last_unstale":"2023-10-19T14:01:40.242857+0000","last_undegraded":"2023-10-19T14:01:40.242857+0000","last_fullsized":"2023-10-19T14:01:40.242857+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:37:55.217643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":12,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.242035+0000","last_change":"2023-10-19T14:01:38.155747+0000","last_active":"2023-10-19T14:01:40.242035+0000","last_peered":"2023-10-19T14:01:40.242035+0000","last_clean":"2023-10-19T14:01:40.242035+0000","last_became_active":"2023-10-19T14:01:38.155388+0000","last_became_peered":"2023-10-19T14:01:38.155388+0000","last_unstale":"2023-10-19T14:01:40.242035+0000","last_undegraded":"2023-10-19T14:01:40.242035+0000","last_fullsized":"2023-10-19T14:01:40.242035+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_clean_scrub_stamp":"2023-10-19T14:01:37.123936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:17:06.515478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":11,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.242623+0000","last_change":"2023-10-19T14:01:39.069645+0000","last_active":"2023-10-19T14:01:40.242623+0000","last_peered":"2023-10-19T14:01:40.242623+0000","last_clean":"2023-10-19T14:01:40.242623+0000","last_became_active":"2023-10-19T14:01:39.069300+0000","last_became_peered":"2023-10-19T14:01:39.069300+0000","last_unstale":"2023-10-19T14:01:40.242623+0000","last_undegraded":"2023-10-19T14:01:40.242623+0000","last_fullsized":"2023-10-19T14:01:40.242623+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":173,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:38.130146+0000","last_clean_scrub_stamp":"2023-10-19T14:01:38.130146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:38:15.764494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.14","version":"174'6","reported_seq":18,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-19T14:01:40.193652+0000","last_change":"2023-10-19T14:01:38.154868+0000","last_active":"2023-10-19T14:01:40.193652+0000","last_peered":"2023-10-19T14:01:40.193652+0000","last_clean":"2023-10-19T14:01:40.193652+0000","last_became_active":"2023-10-19T14:01:38.154616+0000","last_became_peered":"2023-10-19T14:01:38.154616+0000","last_unstale":"2023-10-19T14:01:40.193652+0000","last_undegraded":"2023-10-19T14:01:40.193652+0000","last_fullsized":"2023-10-19T14:01:40.193652+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:01:37.123936+0000","last_clean_scrub_stamp":"2023-10-19T14:01:37.123936+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:26:47.710014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"164'3109","reported_seq":3761,"reported_epoch":174,"state":"active+clean","last_fresh":"2023-10-19T14:01:39.067681+0000","last_change":"2023-10-19T12:44:08.636613+0000","last_active":"2023-10-19T14:01:39.067681+0000","last_peered":"2023-10-19T14:01:39.067681+0000","last_clean":"2023-10-19T14:01:39.067681+0000","last_became_active":"2023-10-19T12:44:08.635697+0000","last_became_peered":"2023-10-19T12:44:08.635697+0000","last_unstale":"2023-10-19T14:01:39.067681+0000","last_undegraded":"2023-10-19T14:01:39.067681+0000","last_fullsized":"2023-10-19T14:01:39.067681+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_clean_scrub_stamp":"2023-10-19T12:44:07.399738+0000","objects_scrubbed":0,"log_size":3109,"log_dups_size":0,"ondisk_log_size":3109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:15:23.685485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1538,"num_read_kb":3182,"num_write":449,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":22,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":21,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1538,"num_read_kb":3182,"num_write":449,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3109,"ondisk_log_size":3109,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575780,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60276,"kb_used_data":2720,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311564,"statfs":{"total":96636764160,"available":96575041536,"internally_reserved":0,"allocated":2785280,"data_stored":1701572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58915275},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:01:29 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.59099999999999997,"15min":0.59699999999999998},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.79900000000000004,"5min":0.85099999999999998,"15min":1.026},"last":0.38},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.34699999999999998,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.83099999999999996,"5min":0.84099999999999997,"15min":1},"last":0.51500000000000001}]},{"osd":1,"last update":"Thu Oct 19 14:01:29 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.59899999999999998,"15min":0.60099999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.873},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.60899999999999999,"15min":0.61199999999999999},"min":{"1min":0.46000000000000002,"5min":0.41999999999999998,"15min":0.376},"max":{"1min":0.84499999999999997,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.45500000000000002}]},{"osd":2,"last update":"Thu Oct 19 14:01:29 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.65900000000000003,"15min":0.65500000000000003},"min":{"1min":0.44500000000000001,"5min":0.44400000000000001,"15min":0.39000000000000001},"max":{"1min":0.875,"5min":0.877,"15min":0.96599999999999997},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.65700000000000003,"15min":0.66000000000000003},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.36799999999999999},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.434}]},{"osd":3,"last update":"Thu Oct 19 14:01:29 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.64300000000000002,"15min":0.64400000000000002},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.76800000000000002,"5min":0.86299999999999999,"15min":1.0129999999999999},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.69499999999999995,"15min":0.69199999999999995},"min":{"1min":0.504,"5min":0.504,"15min":0.48199999999999998},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.92000000000000004},"last":0.65900000000000003}]},{"osd":4,"last update":"Thu Oct 19 14:01:29 2023","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.48099999999999998,"15min":0.46600000000000003},"min":{"1min":0.32000000000000001,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.71799999999999997,"5min":0.73099999999999998,"15min":0.78500000000000003},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.503,"5min":0.51500000000000001,"15min":0.48799999999999999},"min":{"1min":0.35499999999999998,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.69399999999999995,"5min":0.78200000000000003,"15min":0.80900000000000005},"last":0.57299999999999995}]},{"osd":5,"last update":"Thu Oct 19 14:01:29 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.51000000000000001,"15min":0.49299999999999999},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.72899999999999998,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55400000000000005,"15min":0.53100000000000003},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.33900000000000002},"max":{"1min":0.77900000000000003,"5min":0.82699999999999996,"15min":0.86399999999999999},"last":0.628}]},{"osd":6,"last update":"Thu Oct 19 14:01:29 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.58999999999999997,"15min":0.56899999999999995},"min":{"1min":0.38500000000000001,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.81100000000000005,"5min":0.83299999999999996,"15min":0.85799999999999998},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.57399999999999995,"15min":0.55500000000000005},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.33200000000000002},"max":{"1min":0.78000000000000003,"5min":0.81000000000000005,"15min":0.94599999999999995},"last":0.64300000000000002}]}]},{"osd":6,"up_from":13,"seq":55834575780,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37468,"kb_used_data":2128,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334372,"statfs":{"total":96636764160,"available":96598396928,"internally_reserved":0,"allocated":2179072,"data_stored":1104604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:01:30 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.60599999999999998,"15min":0.60799999999999998},"min":{"1min":0.37,"5min":0.37,"15min":0.34599999999999997},"max":{"1min":0.82299999999999995,"5min":0.88100000000000001,"15min":1.3280000000000001},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61699999999999999,"15min":0.61399999999999999},"min":{"1min":0.46500000000000002,"5min":0.46400000000000002,"15min":0.26100000000000001},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":1.4279999999999999},"last":0.51700000000000002}]},{"osd":1,"last update":"Thu Oct 19 14:01:30 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.62,"15min":0.622},"min":{"1min":0.434,"5min":0.42799999999999999,"15min":0.35899999999999999},"max":{"1min":0.78400000000000003,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.624,"15min":0.61799999999999999},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.29299999999999998},"max":{"1min":0.83599999999999997,"5min":0.85399999999999998,"15min":0.89200000000000002},"last":0.60099999999999998}]},{"osd":2,"last update":"Thu Oct 19 14:01:30 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.38200000000000001},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":1.4650000000000001},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.621,"5min":0.65000000000000002,"15min":0.63200000000000001},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.35199999999999998},"max":{"1min":0.82399999999999995,"5min":0.85699999999999998,"15min":1.3979999999999999},"last":0.66800000000000004}]},{"osd":3,"last update":"Thu Oct 19 14:01:30 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.64700000000000002,"15min":0.65300000000000002},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.76600000000000001,"5min":0.82899999999999996,"15min":1.496},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.66900000000000004,"15min":0.66500000000000004},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.38100000000000001},"max":{"1min":0.77500000000000002,"5min":0.83499999999999996,"15min":1.512},"last":0.495}]},{"osd":4,"last update":"Thu Oct 19 14:01:30 2023","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.48499999999999999,"15min":0.47999999999999998},"min":{"1min":0.316,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.51600000000000001,"15min":0.50600000000000001},"min":{"1min":0.36699999999999999,"5min":0.35299999999999998,"15min":0.32800000000000001},"max":{"1min":0.79800000000000004,"5min":0.85699999999999998,"15min":0.93000000000000005},"last":0.42999999999999999}]},{"osd":5,"last update":"Thu Oct 19 14:01:30 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.5,"15min":0.49299999999999999},"min":{"1min":0.35499999999999998,"5min":0.35099999999999998,"15min":0.33700000000000002},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.623},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54200000000000004},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.78800000000000003,"5min":0.83199999999999996,"15min":1.0249999999999999},"last":0.63700000000000001}]},{"osd":7,"last update":"Thu Oct 19 14:01:30 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.54200000000000004,"15min":0.54500000000000004},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.32200000000000001},"max":{"1min":0.81999999999999995,"5min":0.84499999999999997,"15min":0.878},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56399999999999995,"15min":0.55500000000000005},"min":{"1min":0.40500000000000003,"5min":0.373,"15min":0.33400000000000002},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.91100000000000003},"last":0.67900000000000005}]}]},{"osd":5,"up_from":13,"seq":55834575780,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59688,"kb_used_data":2136,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312152,"statfs":{"total":96636764160,"available":96575643648,"internally_reserved":0,"allocated":2187264,"data_stored":1111182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:01:10 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.59399999999999997,"15min":0.61699999999999999},"min":{"1min":0.379,"5min":0.35399999999999998,"15min":0.34499999999999997},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":1.2629999999999999},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.63700000000000001,"15min":0.63300000000000001},"min":{"1min":0.50700000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.88300000000000001,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.55800000000000005}]},{"osd":1,"last update":"Thu Oct 19 14:01:10 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.63700000000000001,"15min":0.63600000000000001},"min":{"1min":0.51700000000000002,"5min":0.435,"15min":0.41699999999999998},"max":{"1min":0.81100000000000005,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.63800000000000001,"15min":0.64000000000000001},"min":{"1min":0.495,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.84899999999999998,"5min":0.87,"15min":1.071},"last":0.627}]},{"osd":2,"last update":"Thu Oct 19 14:01:10 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61899999999999999,"15min":0.628},"min":{"1min":0.46800000000000003,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.83499999999999996,"5min":0.875,"15min":1.044},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.64500000000000002,"15min":0.65300000000000002},"min":{"1min":0.48299999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.81399999999999995,"5min":0.90000000000000002,"15min":1.171},"last":0.68799999999999994}]},{"osd":3,"last update":"Thu Oct 19 14:01:10 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.63500000000000001,"15min":0.64400000000000002},"min":{"1min":0.46600000000000003,"5min":0.44,"15min":0.44},"max":{"1min":0.77800000000000002,"5min":0.85899999999999999,"15min":1.1910000000000001},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.67100000000000004,"15min":0.67500000000000004},"min":{"1min":0.52200000000000002,"5min":0.45500000000000002,"15min":0.43099999999999999},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":1.2210000000000001},"last":0.54600000000000004}]},{"osd":4,"last update":"Thu Oct 19 14:01:10 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.5,"15min":0.48499999999999999},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.65100000000000002,"5min":0.745,"15min":0.75700000000000001},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.501,"5min":0.5,"15min":0.496},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.73899999999999999,"5min":0.76200000000000001,"15min":0.77300000000000002},"last":0.64700000000000002}]},{"osd":6,"last update":"Thu Oct 19 14:01:10 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.48899999999999999,"15min":0.48999999999999999},"min":{"1min":0.32600000000000001,"5min":0.32100000000000001,"15min":0.29699999999999999},"max":{"1min":0.64900000000000002,"5min":0.80000000000000004,"15min":0.83599999999999997},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.53000000000000003,"15min":0.53100000000000003},"min":{"1min":0.38900000000000001,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.80000000000000004},"last":0.70199999999999996}]},{"osd":7,"last update":"Thu Oct 19 14:01:10 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.51500000000000001,"15min":0.51200000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.68000000000000005,"5min":0.79200000000000004,"15min":0.80000000000000004},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.56999999999999995,"15min":0.56299999999999994},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.80100000000000005,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.754}]}]},{"osd":4,"up_from":13,"seq":55834575780,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41652,"kb_used_data":2148,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330188,"statfs":{"total":96636764160,"available":96594112512,"internally_reserved":0,"allocated":2199552,"data_stored":1112794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:00:39 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.59099999999999997,"15min":0.59599999999999997},"min":{"1min":0.45700000000000002,"5min":0.38500000000000001,"15min":0.38400000000000001},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.437},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59899999999999998,"15min":0.59699999999999998},"min":{"1min":0.47899999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.73699999999999999,"5min":0.83199999999999996,"15min":0.98199999999999998},"last":0.41599999999999998}]},{"osd":1,"last update":"Thu Oct 19 14:00:39 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.64300000000000002,"15min":0.64700000000000002},"min":{"1min":0.50600000000000001,"5min":0.443,"15min":0.40600000000000003},"max":{"1min":0.88300000000000001,"5min":0.89400000000000002,"15min":0.92100000000000004},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.65900000000000003,"15min":0.66000000000000003},"min":{"1min":0.53900000000000003,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.57099999999999995}]},{"osd":2,"last update":"Thu Oct 19 14:00:39 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.66300000000000003,"15min":0.65300000000000002},"min":{"1min":0.47099999999999997,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.67300000000000004,"15min":0.66500000000000004},"min":{"1min":0.503,"5min":0.46800000000000003,"15min":0.442},"max":{"1min":1.054,"5min":1.054,"15min":1.054},"last":0.60899999999999999}]},{"osd":3,"last update":"Thu Oct 19 14:00:39 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.65000000000000002,"15min":0.64200000000000002},"min":{"1min":0.48499999999999999,"5min":0.47799999999999998,"15min":0.378},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.93600000000000005},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.67900000000000005,"15min":0.67200000000000004},"min":{"1min":0.58399999999999996,"5min":0.438,"15min":0.438},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.58799999999999997}]},{"osd":5,"last update":"Thu Oct 19 14:00:39 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.46899999999999997,"15min":0.45500000000000002},"min":{"1min":0.33200000000000002,"5min":0.318,"15min":0.29399999999999998},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.498,"15min":0.47999999999999998},"min":{"1min":0.35399999999999998,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.80900000000000005},"last":0.49399999999999999}]},{"osd":6,"last update":"Thu Oct 19 14:00:39 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.49399999999999999,"15min":0.47199999999999998},"min":{"1min":0.375,"5min":0.34000000000000002,"15min":0.32900000000000001},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.62},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.53600000000000003,"15min":0.52000000000000002},"min":{"1min":0.38200000000000001,"5min":0.34499999999999997,"15min":0.32700000000000001},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.64300000000000002}]},{"osd":7,"last update":"Thu Oct 19 14:00:39 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.54000000000000004,"15min":0.52500000000000002},"min":{"1min":0.34200000000000003,"5min":0.33800000000000002,"15min":0.32200000000000001},"max":{"1min":0.71999999999999997,"5min":0.80500000000000005,"15min":0.83699999999999997},"last":0.629},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.60899999999999999,"15min":0.58199999999999996},"min":{"1min":0.46899999999999997,"5min":0.39500000000000002,"15min":0.39200000000000002},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.65500000000000003}]}]},{"osd":3,"up_from":13,"seq":55834575779,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47224,"kb_used_data":2144,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324616,"statfs":{"total":96636764160,"available":96588406784,"internally_reserved":0,"allocated":2195456,"data_stored":1111798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":46135752},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:01:08 2023","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.40999999999999998,"15min":0.40400000000000003},"min":{"1min":0.315,"5min":0.26000000000000001,"15min":0.249},"max":{"1min":0.59199999999999997,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.434,"5min":0.44400000000000001,"15min":0.435},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.26100000000000001},"max":{"1min":0.65400000000000003,"5min":0.66600000000000004,"15min":0.69799999999999995},"last":0.30299999999999999}]},{"osd":1,"last update":"Thu Oct 19 14:01:08 2023","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.42399999999999999,"15min":0.42699999999999999},"min":{"1min":0.30299999999999999,"5min":0.29099999999999998,"15min":0.26200000000000001},"max":{"1min":0.54900000000000004,"5min":0.65600000000000003,"15min":0.72999999999999998},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.47799999999999998,"15min":0.46400000000000002},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.26900000000000002},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72699999999999998},"last":0.34499999999999997}]},{"osd":2,"last update":"Thu Oct 19 14:01:08 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.48999999999999999,"15min":0.48499999999999999},"min":{"1min":0.35399999999999998,"5min":0.33400000000000002,"15min":0.313},"max":{"1min":0.70399999999999996,"5min":0.71899999999999997,"15min":0.72099999999999997},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.51000000000000001,"15min":0.503},"min":{"1min":0.39200000000000002,"5min":0.33600000000000002,"15min":0.32200000000000001},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.42699999999999999}]},{"osd":4,"last update":"Thu Oct 19 14:01:08 2023","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.70399999999999996,"15min":0.69899999999999995},"min":{"1min":0.56599999999999995,"5min":0.46700000000000003,"15min":0.41499999999999998},"max":{"1min":0.81999999999999995,"5min":1.3600000000000001,"15min":1.3600000000000001},"last":0.748},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.71099999999999997,"15min":0.71299999999999997},"min":{"1min":0.56999999999999995,"5min":0.46800000000000003,"15min":0.46300000000000002},"max":{"1min":0.82499999999999996,"5min":1.032,"15min":1.2410000000000001},"last":0.66400000000000003}]},{"osd":5,"last update":"Thu Oct 19 14:01:08 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.69099999999999995,"15min":0.68500000000000005},"min":{"1min":0.54200000000000004,"5min":0.51800000000000002,"15min":0.39700000000000002},"max":{"1min":0.90100000000000002,"5min":1.2450000000000001,"15min":1.2450000000000001},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.67100000000000004,"15min":0.67200000000000004},"min":{"1min":0.502,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.80900000000000005,"5min":1.385,"15min":1.385},"last":0.70199999999999996}]},{"osd":6,"last update":"Thu Oct 19 14:01:08 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.68999999999999995,"15min":0.68500000000000005},"min":{"1min":0.47899999999999998,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.94499999999999995,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.70399999999999996,"15min":0.70299999999999996},"min":{"1min":0.55800000000000005,"5min":0.49099999999999999,"15min":0.42799999999999999},"max":{"1min":0.85799999999999998,"5min":1.0649999999999999,"15min":1.282},"last":0.52800000000000002}]},{"osd":7,"last update":"Thu Oct 19 14:01:08 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.70599999999999996,"15min":0.69399999999999995},"min":{"1min":0.52700000000000002,"5min":0.47499999999999998,"15min":0.46400000000000002},"max":{"1min":0.92000000000000004,"5min":1.3140000000000001,"15min":1.3140000000000001},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72299999999999998},"min":{"1min":0.54500000000000004,"5min":0.49099999999999999,"15min":0.48599999999999999},"max":{"1min":0.88,"5min":0.98899999999999999,"15min":1.304},"last":0.54200000000000004}]}]},{"osd":2,"up_from":13,"seq":55834575781,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59680,"kb_used_data":2132,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312160,"statfs":{"total":96636764160,"available":96575651840,"internally_reserved":0,"allocated":2183168,"data_stored":1104646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:00:56 2023","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.42199999999999999,"15min":0.40899999999999997},"min":{"1min":0.26800000000000002,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.59799999999999998,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.46100000000000002,"15min":0.44600000000000001},"min":{"1min":0.34000000000000002,"5min":0.29799999999999999,"15min":0.26800000000000002},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.42399999999999999}]},{"osd":1,"last update":"Thu Oct 19 14:00:56 2023","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.41799999999999998,"15min":0.41599999999999998},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26700000000000002},"max":{"1min":0.59499999999999997,"5min":0.60799999999999998,"15min":0.66600000000000004},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.48099999999999998,"15min":0.46700000000000003},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.26500000000000001},"max":{"1min":0.69399999999999995,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.35399999999999998}]},{"osd":3,"last update":"Thu Oct 19 14:00:56 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.501,"15min":0.49399999999999999},"min":{"1min":0.33700000000000002,"5min":0.31900000000000001,"15min":0.30199999999999999},"max":{"1min":0.57299999999999995,"5min":0.72099999999999997,"15min":0.75900000000000001},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.52500000000000002,"15min":0.51700000000000002},"min":{"1min":0.40000000000000002,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.66400000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.375}]},{"osd":4,"last update":"Thu Oct 19 14:00:56 2023","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.69799999999999995,"15min":0.68600000000000005},"min":{"1min":0.55300000000000005,"5min":0.46800000000000003,"15min":0.435},"max":{"1min":0.85499999999999998,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.71299999999999997,"15min":0.69499999999999995},"min":{"1min":0.55300000000000005,"5min":0.46600000000000003,"15min":0.44800000000000001},"max":{"1min":1.0580000000000001,"5min":1.2110000000000001,"15min":1.2110000000000001},"last":0.65600000000000003}]},{"osd":5,"last update":"Thu Oct 19 14:00:56 2023","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.72799999999999998,"15min":0.70799999999999996},"min":{"1min":0.53000000000000003,"5min":0.46899999999999997,"15min":0.42299999999999999},"max":{"1min":1.1919999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.69399999999999995},"min":{"1min":0.54400000000000004,"5min":0.48999999999999999,"15min":0.44600000000000001},"max":{"1min":1.1459999999999999,"5min":1.194,"15min":1.194},"last":0.63300000000000001}]},{"osd":6,"last update":"Thu Oct 19 14:00:56 2023","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.72899999999999998,"15min":0.71999999999999997},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.443},"max":{"1min":1.0149999999999999,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.75,"15min":0.72799999999999998},"min":{"1min":0.60199999999999998,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.103,"5min":1.234,"15min":1.234},"last":0.76000000000000001}]},{"osd":7,"last update":"Thu Oct 19 14:00:56 2023","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.75800000000000001,"15min":0.74099999999999999},"min":{"1min":0.65500000000000003,"5min":0.53200000000000003,"15min":0.47699999999999998},"max":{"1min":1.1259999999999999,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.753,"5min":0.76400000000000001,"15min":0.748},"min":{"1min":0.622,"5min":0.505,"15min":0.49399999999999999},"max":{"1min":1.161,"5min":1.161,"15min":1.161},"last":0.70599999999999996}]}]},{"osd":1,"up_from":13,"seq":55834575780,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41632,"kb_used_data":2128,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330208,"statfs":{"total":96636764160,"available":96594132992,"internally_reserved":0,"allocated":2179072,"data_stored":1104604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:00:51 2023","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41499999999999998},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.219},"max":{"1min":0.64500000000000002,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.47399999999999998,"15min":0.46100000000000002},"min":{"1min":0.29899999999999999,"5min":0.25800000000000001,"15min":0.25600000000000001},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.32400000000000001}]},{"osd":2,"last update":"Thu Oct 19 14:00:51 2023","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.44500000000000001,"15min":0.436},"min":{"1min":0.315,"5min":0.29699999999999999,"15min":0.26600000000000001},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68300000000000005},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.5,"5min":0.49399999999999999,"15min":0.48399999999999999},"min":{"1min":0.375,"5min":0.32100000000000001,"15min":0.28499999999999998},"max":{"1min":0.61099999999999999,"5min":0.70599999999999996,"15min":0.73099999999999998},"last":0.40999999999999998}]},{"osd":3,"last update":"Thu Oct 19 14:00:51 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52200000000000002,"15min":0.50900000000000001},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.33800000000000002},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.73899999999999999},"last":0.38},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.45300000000000001,"15min":0.44800000000000001},"min":{"1min":0.33800000000000002,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.67500000000000004},"last":0.36499999999999999}]},{"osd":4,"last update":"Thu Oct 19 14:00:51 2023","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.70599999999999996,"15min":0.69199999999999995},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.434},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.69099999999999995,"15min":0.67800000000000005},"min":{"1min":0.56399999999999995,"5min":0.52300000000000002,"15min":0.40699999999999997},"max":{"1min":0.90200000000000002,"5min":0.90500000000000003,"15min":1.0720000000000001},"last":0.47199999999999998}]},{"osd":5,"last update":"Thu Oct 19 14:00:51 2023","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.71399999999999997,"15min":0.69399999999999995},"min":{"1min":0.56699999999999995,"5min":0.50600000000000001,"15min":0.44},"max":{"1min":1.155,"5min":1.155,"15min":1.254},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.72599999999999998,"15min":0.71099999999999997},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.49099999999999999},"max":{"1min":1.014,"5min":1.014,"15min":1.0760000000000001},"last":0.53700000000000003}]},{"osd":6,"last update":"Thu Oct 19 14:00:51 2023","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.71099999999999997,"15min":0.69399999999999995},"min":{"1min":0.502,"5min":0.502,"15min":0.41899999999999998},"max":{"1min":1.25,"5min":1.25,"15min":1.25},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.73499999999999999,"15min":0.72199999999999998},"min":{"1min":0.56200000000000006,"5min":0.47599999999999998,"15min":0.44500000000000001},"max":{"1min":0.999,"5min":0.999,"15min":1.153},"last":0.56799999999999995}]},{"osd":7,"last update":"Thu Oct 19 14:00:51 2023","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.745,"15min":0.72399999999999998},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.43099999999999999},"max":{"1min":1.1950000000000001,"5min":1.1950000000000001,"15min":1.1950000000000001},"last":0.497},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.746,"15min":0.72699999999999998},"min":{"1min":0.59599999999999997,"5min":0.54500000000000004,"15min":0.49399999999999999},"max":{"1min":1.097,"5min":1.097,"15min":1.097},"last":0.44900000000000001}]}]},{"osd":0,"up_from":13,"seq":55834575781,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38084,"kb_used_data":2736,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333756,"statfs":{"total":96636764160,"available":96597766144,"internally_reserved":0,"allocated":2801664,"data_stored":1703249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Oct 19 14:01:34 2023","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.38900000000000001,"15min":0.39600000000000002},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.45600000000000002,"15min":0.46000000000000002},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.30399999999999999}]},{"osd":2,"last update":"Thu Oct 19 14:01:34 2023","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.47399999999999998,"15min":0.47099999999999997},"min":{"1min":0.312,"5min":0.30299999999999999,"15min":0.28699999999999998},"max":{"1min":0.74199999999999999,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.441,"5min":0.45400000000000001,"15min":0.45100000000000001},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.40500000000000003}]},{"osd":3,"last update":"Thu Oct 19 14:01:34 2023","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.439,"15min":0.43099999999999999},"min":{"1min":0.32700000000000001,"5min":0.30099999999999999,"15min":0.27500000000000002},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.37},{"interface":"front","average":{"1min":0.5,"5min":0.52000000000000002,"15min":0.50900000000000001},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.45600000000000002}]},{"osd":4,"last update":"Thu Oct 19 14:01:34 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.68700000000000006,"15min":0.67400000000000004},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.43099999999999999},"max":{"1min":1.0189999999999999,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.69399999999999995,"15min":0.67900000000000005},"min":{"1min":0.49099999999999999,"5min":0.46800000000000003,"15min":0.45900000000000002},"max":{"1min":0.88900000000000001,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.745}]},{"osd":5,"last update":"Thu Oct 19 14:01:34 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.73399999999999999,"15min":0.71699999999999997},"min":{"1min":0.5,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.069,"5min":1.1970000000000001,"15min":1.1970000000000001},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.73699999999999999,"15min":0.73199999999999998},"min":{"1min":0.51500000000000001,"5min":0.443,"15min":0.443},"max":{"1min":0.95399999999999996,"5min":1.139,"15min":1.139},"last":0.85199999999999998}]},{"osd":6,"last update":"Thu Oct 19 14:01:34 2023","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.71499999999999997,"15min":0.70199999999999996},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.091,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.622},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.71099999999999997,"15min":0.69899999999999995},"min":{"1min":0.503,"5min":0.49399999999999999,"15min":0.443},"max":{"1min":0.98099999999999998,"5min":1.0660000000000001,"15min":1.167},"last":0.79700000000000004}]},{"osd":7,"last update":"Thu Oct 19 14:01:34 2023","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.74199999999999999,"15min":0.72699999999999998},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.145,"5min":1.163,"15min":1.163},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.71599999999999997,"15min":0.70099999999999996},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.123,"5min":1.123,"15min":1.123},"last":0.63200000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-19T14:01:42.506 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-19T14:01:42.506 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-19T14:01:42.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.654+0000 7f06151c0640 1 -- 172.21.15.57:0/2245330253 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06100b0530 msgr2=0x7f06100b09a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:42.658 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.655+0000 7f06151c0640 1 --2- 172.21.15.57:0/2245330253 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06100b0530 0x7f06100b09a0 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 crypto rx=0x7f06100cc7e0 tx=0x7f060802c260 comp rx=0 tx=0).stop 2023-10-19T14:01:42.658 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.655+0000 7f06151c0640 1 -- 172.21.15.57:0/2245330253 shutdown_connections 2023-10-19T14:01:42.658 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.655+0000 7f06151c0640 1 --2- 172.21.15.57:0/2245330253 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06100b0530 0x7f06100b09a0 unknown :-1 s=CLOSED pgs=1914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:42.658 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.655+0000 7f06151c0640 1 --2- 172.21.15.57:0/2245330253 >> v2:172.21.15.179:3300/0 conn(0x7f06100cbd20 0x7f0610158300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:42.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.655+0000 7f06151c0640 1 -- 172.21.15.57:0/2245330253 >> 172.21.15.57:0/2245330253 conn(0x7f0610069930 msgr2=0x7f0610069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:42.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.655+0000 7f06151c0640 1 -- 172.21.15.57:0/2245330253 shutdown_connections 2023-10-19T14:01:42.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.655+0000 7f06151c0640 1 -- 172.21.15.57:0/2245330253 wait complete. 2023-10-19T14:01:42.661 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.656+0000 7f06151c0640 1 Processor -- start 2023-10-19T14:01:42.661 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.656+0000 7f06151c0640 1 -- start start 2023-10-19T14:01:42.662 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.657+0000 7f06151c0640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f06100afa20 0x7f0610141b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:42.662 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.657+0000 7f06151c0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06100b0530 0x7f06100ac740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:42.662 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.657+0000 7f06151c0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f06100cbd20 0x7f06100acc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:42.662 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.657+0000 7f06151c0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0610146e80 con 0x7f06100b0530 2023-10-19T14:01:42.662 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.657+0000 7f06151c0640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0610146b30 con 0x7f06100cbd20 2023-10-19T14:01:42.663 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.657+0000 7f06151c0640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f061012c370 con 0x7f06100afa20 2023-10-19T14:01:42.663 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.657+0000 7f0607fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06100b0530 0x7f06100ac740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:42.663 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.657+0000 7f0607fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06100b0530 0x7f06100ac740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35318/0 (socket says 172.21.15.57:35318) 2023-10-19T14:01:42.663 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.657+0000 7f0607fff640 1 -- 172.21.15.57:0/3747127777 learned_addr learned my addr 172.21.15.57:0/3747127777 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:01:42.663 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.657+0000 7f060f59e640 1 --2- 172.21.15.57:0/3747127777 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f06100cbd20 0x7f06100acc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:42.664 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.657+0000 7f060ed9d640 1 --2- 172.21.15.57:0/3747127777 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f06100afa20 0x7f0610141b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:42.664 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.657+0000 7f0607fff640 1 -- 172.21.15.57:0/3747127777 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f06100afa20 msgr2=0x7f0610141b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:42.664 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.657+0000 7f0607fff640 1 --2- 172.21.15.57:0/3747127777 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f06100afa20 0x7f0610141b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:42.664 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.657+0000 7f0607fff640 1 -- 172.21.15.57:0/3747127777 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f06100cbd20 msgr2=0x7f06100acc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:42.664 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.657+0000 7f0607fff640 1 --2- 172.21.15.57:0/3747127777 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f06100cbd20 0x7f06100acc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:42.665 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.657+0000 7f0607fff640 1 -- 172.21.15.57:0/3747127777 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0608009900 con 0x7f06100b0530 2023-10-19T14:01:42.665 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.657+0000 7f0607fff640 1 --2- 172.21.15.57:0/3747127777 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06100b0530 0x7f06100ac740 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7f05f800c410 tx=0x7f05f800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:42.665 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.658+0000 7f06077fe640 1 -- 172.21.15.57:0/3747127777 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05f80035f0 con 0x7f06100b0530 2023-10-19T14:01:42.665 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.658+0000 7f06077fe640 1 -- 172.21.15.57:0/3747127777 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f05f8003750 con 0x7f06100b0530 2023-10-19T14:01:42.665 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.658+0000 7f06151c0640 1 -- 172.21.15.57:0/3747127777 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f061012c650 con 0x7f06100b0530 2023-10-19T14:01:42.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.658+0000 7f06077fe640 1 -- 172.21.15.57:0/3747127777 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05f8003ba0 con 0x7f06100b0530 2023-10-19T14:01:42.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.658+0000 7f06151c0640 1 -- 172.21.15.57:0/3747127777 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0610142040 con 0x7f06100b0530 2023-10-19T14:01:42.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.659+0000 7f06151c0640 1 -- 172.21.15.57:0/3747127777 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06100690e0 con 0x7f06100b0530 2023-10-19T14:01:42.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.659+0000 7f06077fe640 1 -- 172.21.15.57:0/3747127777 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 91) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f05f8003d00 con 0x7f06100b0530 2023-10-19T14:01:42.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.660+0000 7f06077fe640 1 --2- 172.21.15.57:0/3747127777 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f05f006af60 0x7f05f006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:42.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.660+0000 7f06077fe640 1 -- 172.21.15.57:0/3747127777 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f05f8089050 con 0x7f06100b0530 2023-10-19T14:01:42.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.660+0000 7f060ed9d640 1 --2- 172.21.15.57:0/3747127777 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f05f006af60 0x7f05f006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:42.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.660+0000 7f060ed9d640 1 --2- 172.21.15.57:0/3747127777 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f05f006af60 0x7f05f006d420 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f06000058c0 tx=0x7f0600005f60 comp rx=0 tx=0).ready entity=mgr.8923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:42.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.664+0000 7f06077fe640 1 -- 172.21.15.57:0/3747127777 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f05f8053f40 con 0x7f06100b0530 2023-10-19T14:01:42.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.911+0000 7f06151c0640 1 -- 172.21.15.57:0/3747127777 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f061006f700 con 0x7f06100b0530 2023-10-19T14:01:42.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.913+0000 7f06077fe640 1 -- 172.21.15.57:0/3747127777 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v51) v1 ==== 190+0+147 (secure 0 0 0) 0x7f05f8017020 con 0x7f06100b0530 2023-10-19T14:01:42.915 INFO:teuthology.orchestra.run.smithi057.stdout:[client.0] 2023-10-19T14:01:42.916 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQBkJDFlg9yuExAATxiWDfS+p07G1F0pORbJEg== 2023-10-19T14:01:42.916 INFO:teuthology.orchestra.run.smithi057.stdout: caps mds = "allow" 2023-10-19T14:01:42.916 INFO:teuthology.orchestra.run.smithi057.stdout: caps mon = "allow r" 2023-10-19T14:01:42.916 INFO:teuthology.orchestra.run.smithi057.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-19T14:01:42.917 INFO:teuthology.orchestra.run.smithi057.stderr:updated caps for client.0 2023-10-19T14:01:42.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.915+0000 7f06151c0640 1 -- 172.21.15.57:0/3747127777 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f05f006af60 msgr2=0x7f05f006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:42.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.915+0000 7f06151c0640 1 --2- 172.21.15.57:0/3747127777 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f05f006af60 0x7f05f006d420 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f06000058c0 tx=0x7f0600005f60 comp rx=0 tx=0).stop 2023-10-19T14:01:42.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.915+0000 7f06151c0640 1 -- 172.21.15.57:0/3747127777 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06100b0530 msgr2=0x7f06100ac740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:42.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.915+0000 7f06151c0640 1 --2- 172.21.15.57:0/3747127777 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06100b0530 0x7f06100ac740 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7f05f800c410 tx=0x7f05f800c8d0 comp rx=0 tx=0).stop 2023-10-19T14:01:42.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.915+0000 7f06151c0640 1 -- 172.21.15.57:0/3747127777 shutdown_connections 2023-10-19T14:01:42.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.915+0000 7f06151c0640 1 --2- 172.21.15.57:0/3747127777 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f05f006af60 0x7f05f006d420 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:42.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.915+0000 7f06151c0640 1 --2- 172.21.15.57:0/3747127777 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f06100cbd20 0x7f06100acc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:42.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.915+0000 7f06151c0640 1 --2- 172.21.15.57:0/3747127777 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06100b0530 0x7f06100ac740 unknown :-1 s=CLOSED pgs=1915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:42.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.915+0000 7f06151c0640 1 --2- 172.21.15.57:0/3747127777 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f06100afa20 0x7f0610141b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:42.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.915+0000 7f06151c0640 1 -- 172.21.15.57:0/3747127777 >> 172.21.15.57:0/3747127777 conn(0x7f0610069930 msgr2=0x7f06100ca870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:42.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.915+0000 7f06151c0640 1 -- 172.21.15.57:0/3747127777 shutdown_connections 2023-10-19T14:01:42.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:42.915+0000 7f06151c0640 1 -- 172.21.15.57:0/3747127777 wait complete. 2023-10-19T14:01:42.928 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T14:01:43.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.035+0000 7f897c487640 1 -- 172.21.15.57:0/4027815956 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f89740c96b0 msgr2=0x7f89740c9b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:43.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.035+0000 7f897c487640 1 --2- 172.21.15.57:0/4027815956 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f89740c96b0 0x7f89740c9b30 secure :-1 s=READY pgs=1917 cs=0 l=1 rev1=1 crypto rx=0x7f8964009860 tx=0x7f896402bd60 comp rx=0 tx=0).stop 2023-10-19T14:01:43.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.035+0000 7f897c487640 1 -- 172.21.15.57:0/4027815956 shutdown_connections 2023-10-19T14:01:43.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.035+0000 7f897c487640 1 --2- 172.21.15.57:0/4027815956 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f89740c96b0 0x7f89740c9b30 unknown :-1 s=CLOSED pgs=1917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:43.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.035+0000 7f897c487640 1 --2- 172.21.15.57:0/4027815956 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f897406af00 0x7f897406b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:43.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.035+0000 7f897c487640 1 -- 172.21.15.57:0/4027815956 >> 172.21.15.57:0/4027815956 conn(0x7f8974069930 msgr2=0x7f8974069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:43.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.035+0000 7f897c487640 1 -- 172.21.15.57:0/4027815956 shutdown_connections 2023-10-19T14:01:43.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.035+0000 7f897c487640 1 -- 172.21.15.57:0/4027815956 wait complete. 2023-10-19T14:01:43.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.036+0000 7f897c487640 1 Processor -- start 2023-10-19T14:01:43.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.036+0000 7f897c487640 1 -- start start 2023-10-19T14:01:43.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.037+0000 7f897c487640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f897406af00 0x7f8974139220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:43.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.037+0000 7f897c487640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f89740ab500 0x7f89740aa8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:43.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.037+0000 7f897c487640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f89740c96b0 0x7f89740aae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:43.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.037+0000 7f897c487640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89740ace10 con 0x7f89740c96b0 2023-10-19T14:01:43.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.037+0000 7f897c487640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89740ad0f0 con 0x7f89740ab500 2023-10-19T14:01:43.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.037+0000 7f897c487640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89740ad3d0 con 0x7f897406af00 2023-10-19T14:01:43.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.037+0000 7f897aa24640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f89740c96b0 0x7f89740aae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:43.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.037+0000 7f897aa24640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f89740c96b0 0x7f89740aae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35334/0 (socket says 172.21.15.57:35334) 2023-10-19T14:01:43.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.037+0000 7f897aa24640 1 -- 172.21.15.57:0/1951761593 learned_addr learned my addr 172.21.15.57:0/1951761593 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:01:43.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.037+0000 7f8979a22640 1 --2- 172.21.15.57:0/1951761593 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f89740ab500 0x7f89740aa8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:43.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.037+0000 7f897a223640 1 --2- 172.21.15.57:0/1951761593 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f897406af00 0x7f8974139220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:43.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.037+0000 7f897aa24640 1 -- 172.21.15.57:0/1951761593 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f897406af00 msgr2=0x7f8974139220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:43.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.037+0000 7f897aa24640 1 --2- 172.21.15.57:0/1951761593 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f897406af00 0x7f8974139220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:43.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.037+0000 7f897aa24640 1 -- 172.21.15.57:0/1951761593 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f89740ab500 msgr2=0x7f89740aa8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:43.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.037+0000 7f897aa24640 1 --2- 172.21.15.57:0/1951761593 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f89740ab500 0x7f89740aa8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:43.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.037+0000 7f897aa24640 1 -- 172.21.15.57:0/1951761593 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8964009490 con 0x7f89740c96b0 2023-10-19T14:01:43.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.037+0000 7f897a223640 1 --2- 172.21.15.57:0/1951761593 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f897406af00 0x7f8974139220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:01:43.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.037+0000 7f897aa24640 1 --2- 172.21.15.57:0/1951761593 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f89740c96b0 0x7f89740aae00 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7f896c001560 tx=0x7f896c00cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:43.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.037+0000 7f896affd640 1 -- 172.21.15.57:0/1951761593 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f896c003da0 con 0x7f89740c96b0 2023-10-19T14:01:43.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.038+0000 7f896affd640 1 -- 172.21.15.57:0/1951761593 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f896c005070 con 0x7f89740c96b0 2023-10-19T14:01:43.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.038+0000 7f897c487640 1 -- 172.21.15.57:0/1951761593 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89740ad6b0 con 0x7f89740c96b0 2023-10-19T14:01:43.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.038+0000 7f896affd640 1 -- 172.21.15.57:0/1951761593 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f896c005460 con 0x7f89740c96b0 2023-10-19T14:01:43.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.038+0000 7f897c487640 1 -- 172.21.15.57:0/1951761593 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89740adbd0 con 0x7f89740c96b0 2023-10-19T14:01:43.049 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.039+0000 7f897c487640 1 -- 172.21.15.57:0/1951761593 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89740ad8f0 con 0x7f89740c96b0 2023-10-19T14:01:43.049 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.039+0000 7f896affd640 1 -- 172.21.15.57:0/1951761593 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 91) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f896c0055c0 con 0x7f89740c96b0 2023-10-19T14:01:43.049 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.040+0000 7f896affd640 1 --2- 172.21.15.57:0/1951761593 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f895806af90 0x7f895806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:43.049 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.040+0000 7f896affd640 1 -- 172.21.15.57:0/1951761593 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f896c0893f0 con 0x7f89740c96b0 2023-10-19T14:01:43.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.040+0000 7f897a223640 1 --2- 172.21.15.57:0/1951761593 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f895806af90 0x7f895806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:43.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.040+0000 7f897a223640 1 --2- 172.21.15.57:0/1951761593 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f895806af90 0x7f895806d450 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8970005d00 tx=0x7f8970009550 comp rx=0 tx=0).ready entity=mgr.8923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:43.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.044+0000 7f896affd640 1 -- 172.21.15.57:0/1951761593 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f89740ad8f0 con 0x7f89740c96b0 2023-10-19T14:01:43.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.256+0000 7f897c487640 1 -- 172.21.15.57:0/1951761593 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f89740ad8f0 con 0x7f89740c96b0 2023-10-19T14:01:43.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.256+0000 7f896affd640 1 -- 172.21.15.57:0/1951761593 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+4920 (secure 0 0 0) 0x7f89740ad8f0 con 0x7f89740c96b0 2023-10-19T14:01:43.260 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:01:43.260 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":70,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8126,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/1759316409","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1759316409},{"type":"v1","addr":"172.21.15.57:6835","nonce":1759316409}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67},{"gid":8129,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/2462599644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2462599644},{"type":"v1","addr":"172.21.15.57:6834","nonce":2462599644}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67},{"gid":9136,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/373220657","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":373220657},{"type":"v1","addr":"172.21.15.179:6836","nonce":373220657}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T14:01:38.503105+0000","modified":"2023-10-19T14:01:40.495529+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9139},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9139":{"gid":9139,"name":"d","rank":0,"incarnation":68,"state":"up:active","state_seq":3,"addr":"172.21.15.179:6837/1102326590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":1102326590},{"type":"v1","addr":"172.21.15.179:6837","nonce":1102326590}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2023-10-19T14:01:43.262 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 70 2023-10-19T14:01:43.263 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.258+0000 7f897c487640 1 -- 172.21.15.57:0/1951761593 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f895806af90 msgr2=0x7f895806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:43.263 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.258+0000 7f897c487640 1 --2- 172.21.15.57:0/1951761593 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f895806af90 0x7f895806d450 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8970005d00 tx=0x7f8970009550 comp rx=0 tx=0).stop 2023-10-19T14:01:43.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.258+0000 7f897c487640 1 -- 172.21.15.57:0/1951761593 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f89740c96b0 msgr2=0x7f89740aae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:43.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.258+0000 7f897c487640 1 --2- 172.21.15.57:0/1951761593 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f89740c96b0 0x7f89740aae00 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7f896c001560 tx=0x7f896c00cc70 comp rx=0 tx=0).stop 2023-10-19T14:01:43.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.258+0000 7f897c487640 1 -- 172.21.15.57:0/1951761593 shutdown_connections 2023-10-19T14:01:43.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.258+0000 7f897c487640 1 --2- 172.21.15.57:0/1951761593 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f895806af90 0x7f895806d450 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:43.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.258+0000 7f897c487640 1 --2- 172.21.15.57:0/1951761593 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f89740c96b0 0x7f89740aae00 unknown :-1 s=CLOSED pgs=1918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:43.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.258+0000 7f897c487640 1 --2- 172.21.15.57:0/1951761593 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f89740ab500 0x7f89740aa8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:43.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.258+0000 7f897c487640 1 --2- 172.21.15.57:0/1951761593 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f897406af00 0x7f8974139220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:43.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.258+0000 7f897c487640 1 -- 172.21.15.57:0/1951761593 >> 172.21.15.57:0/1951761593 conn(0x7f8974069930 msgr2=0x7f89740ab990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:43.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.258+0000 7f897c487640 1 -- 172.21.15.57:0/1951761593 shutdown_connections 2023-10-19T14:01:43.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.258+0000 7f897c487640 1 -- 172.21.15.57:0/1951761593 wait complete. 2023-10-19T14:01:43.674 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 70, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-19T14:01:38.503105+0000', 'modified': '2023-10-19T14:01:40.495529+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9139}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9139': {'gid': 9139, 'name': 'd', 'rank': 0, 'incarnation': 68, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.179:6837/1102326590', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6835', 'nonce': 1102326590}, {'type': 'v1', 'addr': '172.21.15.179:6837', 'nonce': 1102326590}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-19T14:01:43.674 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-19T14:01:43.675 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2023-10-19T14:01:43.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.785+0000 7f9778075640 1 -- 172.21.15.57:0/2694547780 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f97700b0530 msgr2=0x7f97700b09c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:43.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.785+0000 7f9778075640 1 --2- 172.21.15.57:0/2694547780 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f97700b0530 0x7f97700b09c0 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x7f9768005fd0 tx=0x7f976802c0e0 comp rx=0 tx=0).stop 2023-10-19T14:01:43.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.785+0000 7f9778075640 1 -- 172.21.15.57:0/2694547780 shutdown_connections 2023-10-19T14:01:43.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.785+0000 7f9778075640 1 --2- 172.21.15.57:0/2694547780 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f97700afa20 0x7f9770158300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:43.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.785+0000 7f9778075640 1 --2- 172.21.15.57:0/2694547780 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f97700b0530 0x7f97700b09c0 unknown :-1 s=CLOSED pgs=1920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:43.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.785+0000 7f9778075640 1 -- 172.21.15.57:0/2694547780 >> 172.21.15.57:0/2694547780 conn(0x7f9770069930 msgr2=0x7f9770069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:43.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.785+0000 7f9778075640 1 -- 172.21.15.57:0/2694547780 shutdown_connections 2023-10-19T14:01:43.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.785+0000 7f9778075640 1 -- 172.21.15.57:0/2694547780 wait complete. 2023-10-19T14:01:43.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.786+0000 7f9778075640 1 Processor -- start 2023-10-19T14:01:43.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.786+0000 7f9778075640 1 -- start start 2023-10-19T14:01:43.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.787+0000 7f9778075640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f97700afa20 0x7f9770146680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:43.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.787+0000 7f9778075640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f97700b0530 0x7f977014e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:43.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.787+0000 7f9778075640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f97700cbd20 0x7f97700ae700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:43.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.787+0000 7f9778075640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f977015abf0 con 0x7f97700cbd20 2023-10-19T14:01:43.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.787+0000 7f9778075640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9770146bc0 con 0x7f97700afa20 2023-10-19T14:01:43.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.787+0000 7f9778075640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f977015a070 con 0x7f97700b0530 2023-10-19T14:01:43.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.787+0000 7f9776612640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f97700cbd20 0x7f97700ae700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:43.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.787+0000 7f9776612640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f97700cbd20 0x7f97700ae700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35354/0 (socket says 172.21.15.57:35354) 2023-10-19T14:01:43.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.787+0000 7f9776612640 1 -- 172.21.15.57:0/2072866982 learned_addr learned my addr 172.21.15.57:0/2072866982 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:01:43.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.787+0000 7f9775610640 1 --2- 172.21.15.57:0/2072866982 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f97700b0530 0x7f977014e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:43.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.787+0000 7f9775e11640 1 --2- 172.21.15.57:0/2072866982 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f97700afa20 0x7f9770146680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:43.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.787+0000 7f9776612640 1 -- 172.21.15.57:0/2072866982 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f97700b0530 msgr2=0x7f977014e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:43.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.787+0000 7f9776612640 1 --2- 172.21.15.57:0/2072866982 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f97700b0530 0x7f977014e810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:43.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.787+0000 7f9776612640 1 -- 172.21.15.57:0/2072866982 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f97700afa20 msgr2=0x7f9770146680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:43.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.787+0000 7f9776612640 1 --2- 172.21.15.57:0/2072866982 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f97700afa20 0x7f9770146680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:43.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.787+0000 7f9776612640 1 -- 172.21.15.57:0/2072866982 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9768009840 con 0x7f97700cbd20 2023-10-19T14:01:43.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.787+0000 7f9776612640 1 --2- 172.21.15.57:0/2072866982 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f97700cbd20 0x7f97700ae700 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7f9768036040 tx=0x7f9768004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:43.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.787+0000 7f97667fc640 1 -- 172.21.15.57:0/2072866982 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9768004730 con 0x7f97700cbd20 2023-10-19T14:01:43.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.787+0000 7f97667fc640 1 -- 172.21.15.57:0/2072866982 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f97680345a0 con 0x7f97700cbd20 2023-10-19T14:01:43.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.787+0000 7f9778075640 1 -- 172.21.15.57:0/2072866982 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f977015a350 con 0x7f97700cbd20 2023-10-19T14:01:43.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.787+0000 7f97667fc640 1 -- 172.21.15.57:0/2072866982 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97680349b0 con 0x7f97700cbd20 2023-10-19T14:01:43.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.788+0000 7f9778075640 1 -- 172.21.15.57:0/2072866982 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f977015a910 con 0x7f97700cbd20 2023-10-19T14:01:43.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.789+0000 7f97667fc640 1 -- 172.21.15.57:0/2072866982 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 91) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f9768034b10 con 0x7f97700cbd20 2023-10-19T14:01:43.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.789+0000 7f97667fc640 1 --2- 172.21.15.57:0/2072866982 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f975406af60 0x7f975406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:43.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.789+0000 7f97667fc640 1 -- 172.21.15.57:0/2072866982 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f97680ac9a0 con 0x7f97700cbd20 2023-10-19T14:01:43.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.790+0000 7f9775e11640 1 --2- 172.21.15.57:0/2072866982 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f975406af60 0x7f975406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:43.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.790+0000 7f9775e11640 1 --2- 172.21.15.57:0/2072866982 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f975406af60 0x7f975406d420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f976c0046b0 tx=0x7f976c007eb0 comp rx=0 tx=0).ready entity=mgr.8923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:43.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.791+0000 7f977ac08740 1 Processor -- start 2023-10-19T14:01:43.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.791+0000 7f977ac08740 1 -- start start 2023-10-19T14:01:43.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.791+0000 7f977ac08740 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55c648ab8b40 0x55c648a8bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:43.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.791+0000 7f977ac08740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c648ab8f20 0x55c648abb8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:43.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.791+0000 7f977ac08740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c648a8c020 0x55c648abbe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:43.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.791+0000 7f977ac08740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c64899d730 con 0x55c648ab8f20 2023-10-19T14:01:43.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.791+0000 7f977ac08740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6489fe780 con 0x55c648a8c020 2023-10-19T14:01:43.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.791+0000 7f977ac08740 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c6489fbe70 con 0x55c648ab8b40 2023-10-19T14:01:43.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.792+0000 7f9775610640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c648ab8f20 0x55c648abb8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:43.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.792+0000 7f9775610640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c648ab8f20 0x55c648abb8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35362/0 (socket says 172.21.15.57:35362) 2023-10-19T14:01:43.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.792+0000 7f9775610640 1 -- 172.21.15.57:0/1822620998 learned_addr learned my addr 172.21.15.57:0/1822620998 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:01:43.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.792+0000 7f9775e11640 1 --2- 172.21.15.57:0/1822620998 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55c648ab8b40 0x55c648a8bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:43.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.792+0000 7f9776612640 1 --2- 172.21.15.57:0/1822620998 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c648a8c020 0x55c648abbe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:43.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.792+0000 7f9775610640 1 -- 172.21.15.57:0/1822620998 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55c648ab8b40 msgr2=0x55c648a8bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T14:01:43.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.792+0000 7f9775610640 1 --2- 172.21.15.57:0/1822620998 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55c648ab8b40 0x55c648a8bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:43.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.792+0000 7f9775610640 1 -- 172.21.15.57:0/1822620998 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c648a8c020 msgr2=0x55c648abbe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T14:01:43.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.792+0000 7f9775610640 1 --2- 172.21.15.57:0/1822620998 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c648a8c020 0x55c648abbe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:43.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.792+0000 7f9775610640 1 -- 172.21.15.57:0/1822620998 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c6489fc3d0 con 0x55c648ab8f20 2023-10-19T14:01:43.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.792+0000 7f9775610640 1 --2- 172.21.15.57:0/1822620998 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c648ab8f20 0x55c648abb8d0 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x7f976000bca0 tx=0x7f97600107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4034158cb6166891 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:43.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.792+0000 7f97537fe640 1 -- 172.21.15.57:0/1822620998 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9760005220 con 0x55c648ab8f20 2023-10-19T14:01:43.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.792+0000 7f97537fe640 1 -- 172.21.15.57:0/1822620998 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f9760015070 con 0x55c648ab8f20 2023-10-19T14:01:43.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.792+0000 7f97537fe640 1 -- 172.21.15.57:0/1822620998 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f976000be50 con 0x55c648ab8f20 2023-10-19T14:01:43.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.793+0000 7f977ac08740 1 -- 172.21.15.57:0/1822620998 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c648ab8f20 msgr2=0x55c648abb8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:43.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.793+0000 7f977ac08740 1 --2- 172.21.15.57:0/1822620998 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c648ab8f20 0x55c648abb8d0 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x7f976000bca0 tx=0x7f97600107a0 comp rx=0 tx=0).stop 2023-10-19T14:01:43.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.793+0000 7f977ac08740 1 -- 172.21.15.57:0/1822620998 shutdown_connections 2023-10-19T14:01:43.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.793+0000 7f977ac08740 1 --2- 172.21.15.57:0/1822620998 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c648a8c020 0x55c648abbe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:43.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.793+0000 7f977ac08740 1 --2- 172.21.15.57:0/1822620998 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c648ab8f20 0x55c648abb8d0 unknown :-1 s=CLOSED pgs=1922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:43.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.793+0000 7f977ac08740 1 --2- 172.21.15.57:0/1822620998 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55c648ab8b40 0x55c648a8bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:43.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.793+0000 7f977ac08740 1 -- 172.21.15.57:0/1822620998 >> 172.21.15.57:0/1822620998 conn(0x55c648a8dd50 msgr2=0x55c648ad68d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:43.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.793+0000 7f977ac08740 1 -- 172.21.15.57:0/1822620998 shutdown_connections 2023-10-19T14:01:43.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.793+0000 7f977ac08740 1 -- 172.21.15.57:0/1822620998 wait complete. 2023-10-19T14:01:43.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.793+0000 7f977ac08740 1 -- start start 2023-10-19T14:01:43.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.793+0000 7f977ac08740 1 Processor -- start 2023-10-19T14:01:43.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.794+0000 7f977ac08740 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55c648ab8b40 0x55c648abf120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:43.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.794+0000 7f977ac08740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c648ab8f20 0x55c648aa6d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:43.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.794+0000 7f977ac08740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c648aa6890 0x55c648ad12f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:43.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.794+0000 7f977ac08740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6489f7760 con 0x55c648ab8f20 2023-10-19T14:01:43.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.794+0000 7f977ac08740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6489b28f0 con 0x55c648aa6890 2023-10-19T14:01:43.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.794+0000 7f977ac08740 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c6489b2590 con 0x55c648ab8b40 2023-10-19T14:01:43.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.794+0000 7f977ac08740 10 client.0 _ll_register_callbacks cb 0x55c6489eeee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-19T14:01:43.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.794+0000 7f9775610640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c648ab8f20 0x55c648aa6d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:43.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.795+0000 7f9775610640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c648ab8f20 0x55c648aa6d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35364/0 (socket says 172.21.15.57:35364) 2023-10-19T14:01:43.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.795+0000 7f9775610640 1 -- 172.21.15.57:0/4127490131 learned_addr learned my addr 172.21.15.57:0/4127490131 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:01:43.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.795+0000 7f9775e11640 1 --2- 172.21.15.57:0/4127490131 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55c648ab8b40 0x55c648abf120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:43.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.795+0000 7f9776612640 1 --2- 172.21.15.57:0/4127490131 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c648aa6890 0x55c648ad12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:43.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.795+0000 7f9775610640 1 -- 172.21.15.57:0/4127490131 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55c648ab8b40 msgr2=0x55c648abf120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T14:01:43.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.795+0000 7f9775610640 1 --2- 172.21.15.57:0/4127490131 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55c648ab8b40 0x55c648abf120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:43.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.795+0000 7f9775610640 1 -- 172.21.15.57:0/4127490131 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c648aa6890 msgr2=0x55c648ad12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T14:01:43.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.795+0000 7f9775610640 1 --2- 172.21.15.57:0/4127490131 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c648aa6890 0x55c648ad12f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:43.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.795+0000 7f9775610640 1 -- 172.21.15.57:0/4127490131 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f976000b8a0 con 0x55c648ab8f20 2023-10-19T14:01:43.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.795+0000 7f9775610640 1 --2- 172.21.15.57:0/4127490131 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c648ab8f20 0x55c648aa6d80 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 crypto rx=0x7f9760000c00 tx=0x7f97600056a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b7dc68dcef7737e server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:43.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.795+0000 7f9750ff9640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2023-10-19T14:01:43.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.795+0000 7f9750ff9640 1 -- 172.21.15.57:0/4127490131 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9760020070 con 0x55c648ab8f20 2023-10-19T14:01:43.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.795+0000 7f977ac08740 1 -- 172.21.15.57:0/4127490131 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c6489f0740 con 0x55c648ab8f20 2023-10-19T14:01:43.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.795+0000 7f9750ff9640 1 -- 172.21.15.57:0/4127490131 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(175..175 src has 1..175) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f9760017070 con 0x55c648ab8f20 2023-10-19T14:01:43.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.795+0000 7f9750ff9640 1 -- 172.21.15.57:0/4127490131 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=70) v2 ==== 24+0+0 (secure 0 0 0) 0x7f976000bed0 con 0x55c648ab8f20 2023-10-19T14:01:43.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.795+0000 7f977ac08740 10 client.10275 fetch_fsmap learned FSMap version 70 2023-10-19T14:01:43.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.795+0000 7f977ac08740 1 -- 172.21.15.57:0/4127490131 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=70,monmap=2+}) v3 -- 0x55c6489f5de0 con 0x55c648ab8f20 2023-10-19T14:01:43.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.796+0000 7f9750ff9640 1 -- 172.21.15.57:0/4127490131 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x55c6489f0740 con 0x55c648ab8f20 2023-10-19T14:01:43.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.796+0000 7f9750ff9640 1 -- 172.21.15.57:0/4127490131 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 70) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f9760004250 con 0x55c648ab8f20 2023-10-19T14:01:43.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.796+0000 7f977ac08740 10 client.10275 fetch_fsmap finished waiting for FSMap version 70 2023-10-19T14:01:43.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.796+0000 7f977ac08740 10 client.10275 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-19T14:01:43.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.796+0000 7f977ac08740 20 client.10275 populate_metadata read hostname 'smithi057' 2023-10-19T14:01:43.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.796+0000 7f977ac08740 1 --2- 172.21.15.57:0/4127490131 >> [v2:172.21.15.179:6835/1102326590,v1:172.21.15.179:6837/1102326590] conn(0x55c648aa8e10 0x55c648aa91f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:43.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.796+0000 7f977ac08740 4 client.10275 mds_command: new command op to 9139 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-19T14:01:43.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.796+0000 7f977ac08740 1 -- 172.21.15.57:0/4127490131 --> [v2:172.21.15.179:6835/1102326590,v1:172.21.15.179:6837/1102326590] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c6489f0460 con 0x55c648aa8e10 2023-10-19T14:01:43.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.796+0000 7f9775e11640 1 --2- 172.21.15.57:0/4127490131 >> [v2:172.21.15.179:6835/1102326590,v1:172.21.15.179:6837/1102326590] conn(0x55c648aa8e10 0x55c648aa91f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:43.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.796+0000 7f9750ff9640 10 client.10275 ms_handle_connect on v2:172.21.15.179:6835/1102326590 2023-10-19T14:01:43.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.796+0000 7f9775e11640 1 --2- 172.21.15.57:0/4127490131 >> [v2:172.21.15.179:6835/1102326590,v1:172.21.15.179:6837/1102326590] conn(0x55c648aa8e10 0x55c648aa91f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9017b927ca9d5ce9 server_cookie=af96e028f6548827 in_seq=0 out_seq=0 2023-10-19T14:01:43.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.797+0000 7f9750ff9640 1 -- 172.21.15.57:0/4127490131 <== mds.0 v2:172.21.15.179:6835/1102326590 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55c6489f0460 con 0x55c648aa8e10 2023-10-19T14:01:43.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.797+0000 7f9750ff9640 10 client.10275 handle_command_reply: tid=0 2023-10-19T14:01:43.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.797+0000 7f977ac08740 1 client.10275 shutdown 2023-10-19T14:01:43.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.797+0000 7f977ac08740 1 -- 172.21.15.57:0/4127490131 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c648ab8f20 msgr2=0x55c648aa6d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:43.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.797+0000 7f977ac08740 1 --2- 172.21.15.57:0/4127490131 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c648ab8f20 0x55c648aa6d80 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 crypto rx=0x7f9760000c00 tx=0x7f97600056a0 comp rx=0 tx=0).stop 2023-10-19T14:01:43.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.797+0000 7f977ac08740 1 -- 172.21.15.57:0/4127490131 shutdown_connections 2023-10-19T14:01:43.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.797+0000 7f977ac08740 1 --2- 172.21.15.57:0/4127490131 >> [v2:172.21.15.179:6835/1102326590,v1:172.21.15.179:6837/1102326590] conn(0x55c648aa8e10 0x55c648aa91f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:43.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.797+0000 7f977ac08740 1 --2- 172.21.15.57:0/4127490131 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c648aa6890 0x55c648ad12f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:43.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.797+0000 7f977ac08740 1 --2- 172.21.15.57:0/4127490131 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c648ab8f20 0x55c648aa6d80 unknown :-1 s=CLOSED pgs=1923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:43.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.797+0000 7f9750ff9640 0 client.10275 ms_handle_reset on v2:172.21.15.179:6835/1102326590 2023-10-19T14:01:43.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.797+0000 7f977ac08740 1 --2- 172.21.15.57:0/4127490131 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55c648ab8b40 0x55c648abf120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:43.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.797+0000 7f977ac08740 1 -- 172.21.15.57:0/4127490131 >> 172.21.15.57:0/4127490131 conn(0x55c648a8c020 msgr2=0x55c648a8f9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:43.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.797+0000 7f977ac08740 1 -- 172.21.15.57:0/4127490131 shutdown_connections 2023-10-19T14:01:43.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.798+0000 7f977ac08740 1 -- 172.21.15.57:0/4127490131 wait complete. 2023-10-19T14:01:43.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.798+0000 7f977ac08740 20 client.10275 trim_cache size 0 max 16384 2023-10-19T14:01:43.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.813+0000 7f977ac08740 1 Processor -- start 2023-10-19T14:01:43.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.813+0000 7f977ac08740 1 -- start start 2023-10-19T14:01:43.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.814+0000 7f977ac08740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c648ab8b40 0x55c648aabbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:43.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.814+0000 7f977ac08740 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55c648ab8f20 0x55c648aab200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:43.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.814+0000 7f977ac08740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c648ad23d0 0x55c648ad9de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:43.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.814+0000 7f977ac08740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c648978270 con 0x55c648ab8b40 2023-10-19T14:01:43.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.814+0000 7f977ac08740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c648aa7880 con 0x55c648ad23d0 2023-10-19T14:01:43.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.814+0000 7f977ac08740 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c648a8c740 con 0x55c648ab8f20 2023-10-19T14:01:43.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.814+0000 7f9775e11640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c648ab8b40 0x55c648aabbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:43.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.814+0000 7f9775e11640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c648ab8b40 0x55c648aabbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35376/0 (socket says 172.21.15.57:35376) 2023-10-19T14:01:43.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.814+0000 7f9775e11640 1 -- 172.21.15.57:0/3023213503 learned_addr learned my addr 172.21.15.57:0/3023213503 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:01:43.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.814+0000 7f9775e11640 1 -- 172.21.15.57:0/3023213503 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55c648ab8f20 msgr2=0x55c648aab200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T14:01:43.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.814+0000 7f9775e11640 1 --2- 172.21.15.57:0/3023213503 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55c648ab8f20 0x55c648aab200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:43.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.814+0000 7f9775e11640 1 -- 172.21.15.57:0/3023213503 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c648ad23d0 msgr2=0x55c648ad9de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T14:01:43.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.814+0000 7f9775e11640 1 --2- 172.21.15.57:0/3023213503 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c648ad23d0 0x55c648ad9de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:43.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.814+0000 7f9775e11640 1 -- 172.21.15.57:0/3023213503 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c648994d00 con 0x55c648ab8b40 2023-10-19T14:01:43.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.814+0000 7f9775e11640 1 --2- 172.21.15.57:0/3023213503 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c648ab8b40 0x55c648aabbb0 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7f976c007a40 tx=0x7f976c015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fec99cdc519004b9 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:43.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.814+0000 7f9753fff640 1 -- 172.21.15.57:0/3023213503 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f976c00fcd0 con 0x55c648ab8b40 2023-10-19T14:01:43.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.814+0000 7f9753fff640 1 -- 172.21.15.57:0/3023213503 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f976c029240 con 0x55c648ab8b40 2023-10-19T14:01:43.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.814+0000 7f9753fff640 1 -- 172.21.15.57:0/3023213503 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f976c0297a0 con 0x55c648ab8b40 2023-10-19T14:01:43.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.815+0000 7f977ac08740 1 -- 172.21.15.57:0/3023213503 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c648ab8b40 msgr2=0x55c648aabbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:43.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.815+0000 7f977ac08740 1 --2- 172.21.15.57:0/3023213503 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c648ab8b40 0x55c648aabbb0 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7f976c007a40 tx=0x7f976c015000 comp rx=0 tx=0).stop 2023-10-19T14:01:43.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.815+0000 7f977ac08740 1 -- 172.21.15.57:0/3023213503 shutdown_connections 2023-10-19T14:01:43.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.815+0000 7f977ac08740 1 --2- 172.21.15.57:0/3023213503 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c648ad23d0 0x55c648ad9de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:43.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.815+0000 7f977ac08740 1 --2- 172.21.15.57:0/3023213503 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55c648ab8f20 0x55c648aab200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:43.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.815+0000 7f977ac08740 1 --2- 172.21.15.57:0/3023213503 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c648ab8b40 0x55c648aabbb0 unknown :-1 s=CLOSED pgs=1924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:43.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.815+0000 7f977ac08740 1 -- 172.21.15.57:0/3023213503 >> 172.21.15.57:0/3023213503 conn(0x55c648a8c020 msgr2=0x55c648acfc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:43.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.815+0000 7f977ac08740 1 -- 172.21.15.57:0/3023213503 shutdown_connections 2023-10-19T14:01:43.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.815+0000 7f977ac08740 1 -- 172.21.15.57:0/3023213503 wait complete. 2023-10-19T14:01:43.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.815+0000 7f977ac08740 1 -- start start 2023-10-19T14:01:43.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.815+0000 7f977ac08740 1 Processor -- start 2023-10-19T14:01:43.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.816+0000 7f977ac08740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c648ab8b40 0x55c648b3f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:43.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.816+0000 7f977ac08740 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55c648ab8f20 0x55c648abefe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:43.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.816+0000 7f977ac08740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c648ad23d0 0x55c648abf520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:43.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.816+0000 7f977ac08740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c648b40330 con 0x55c648ab8b40 2023-10-19T14:01:43.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.816+0000 7f977ac08740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c648ae95a0 con 0x55c648ad23d0 2023-10-19T14:01:43.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.816+0000 7f977ac08740 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c648ae9ad0 con 0x55c648ab8f20 2023-10-19T14:01:43.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.816+0000 7f977ac08740 10 client.0 _ll_register_callbacks cb 0x7f976000b770 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-19T14:01:43.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.816+0000 7f9775e11640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c648ab8b40 0x55c648b3f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:43.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.816+0000 7f9775e11640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c648ab8b40 0x55c648b3f7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35378/0 (socket says 172.21.15.57:35378) 2023-10-19T14:01:43.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.816+0000 7f9775e11640 1 -- 172.21.15.57:0/950555368 learned_addr learned my addr 172.21.15.57:0/950555368 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:01:43.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.817+0000 7f9775610640 1 --2- 172.21.15.57:0/950555368 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55c648ab8f20 0x55c648abefe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:43.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.817+0000 7f9776612640 1 --2- 172.21.15.57:0/950555368 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c648ad23d0 0x55c648abf520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:43.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.817+0000 7f9775e11640 1 -- 172.21.15.57:0/950555368 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55c648ab8f20 msgr2=0x55c648abefe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T14:01:43.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.817+0000 7f9775e11640 1 --2- 172.21.15.57:0/950555368 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55c648ab8f20 0x55c648abefe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:43.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.817+0000 7f9775e11640 1 -- 172.21.15.57:0/950555368 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c648ad23d0 msgr2=0x55c648abf520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T14:01:43.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.817+0000 7f9775e11640 1 --2- 172.21.15.57:0/950555368 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c648ad23d0 0x55c648abf520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:43.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.817+0000 7f9775e11640 1 -- 172.21.15.57:0/950555368 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f977006fb50 con 0x55c648ab8b40 2023-10-19T14:01:43.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.817+0000 7f9775e11640 1 --2- 172.21.15.57:0/950555368 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c648ab8b40 0x55c648b3f7e0 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7f976c001590 tx=0x7f976c014680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a30c3d0064d6d3c server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:43.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.817+0000 7f9750ff9640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2023-10-19T14:01:43.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.817+0000 7f9750ff9640 1 -- 172.21.15.57:0/950555368 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f976c014c80 con 0x55c648ab8b40 2023-10-19T14:01:43.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.817+0000 7f9750ff9640 1 -- 172.21.15.57:0/950555368 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(175..175 src has 1..175) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f976c014de0 con 0x55c648ab8b40 2023-10-19T14:01:43.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.817+0000 7f977ac08740 1 -- 172.21.15.57:0/950555368 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9760015070 con 0x55c648ab8b40 2023-10-19T14:01:43.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.817+0000 7f9750ff9640 1 -- 172.21.15.57:0/950555368 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=70) v2 ==== 24+0+0 (secure 0 0 0) 0x7f976c032480 con 0x55c648ab8b40 2023-10-19T14:01:43.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.817+0000 7f977ac08740 10 client.10281 fetch_fsmap learned FSMap version 70 2023-10-19T14:01:43.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.817+0000 7f977ac08740 1 -- 172.21.15.57:0/950555368 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=70,monmap=2+}) v3 -- 0x55c6489ba4f0 con 0x55c648ab8b40 2023-10-19T14:01:43.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.817+0000 7f9750ff9640 1 -- 172.21.15.57:0/950555368 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f9760015070 con 0x55c648ab8b40 2023-10-19T14:01:43.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.817+0000 7f9750ff9640 1 -- 172.21.15.57:0/950555368 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 70) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f976c012070 con 0x55c648ab8b40 2023-10-19T14:01:43.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.817+0000 7f977ac08740 10 client.10281 fetch_fsmap finished waiting for FSMap version 70 2023-10-19T14:01:43.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.817+0000 7f977ac08740 10 client.10281 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-19T14:01:43.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.817+0000 7f977ac08740 20 client.10281 populate_metadata read hostname 'smithi057' 2023-10-19T14:01:43.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.817+0000 7f977ac08740 1 --2- 172.21.15.57:0/950555368 >> [v2:172.21.15.179:6835/1102326590,v1:172.21.15.179:6837/1102326590] conn(0x55c648ad8b40 0x55c648ad8f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:43.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.817+0000 7f977ac08740 4 client.10281 mds_command: new command op to 9139 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-19T14:01:43.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.817+0000 7f977ac08740 1 -- 172.21.15.57:0/950555368 --> [v2:172.21.15.179:6835/1102326590,v1:172.21.15.179:6837/1102326590] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55c6486f96c0 con 0x55c648ad8b40 2023-10-19T14:01:43.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.818+0000 7f9775610640 1 --2- 172.21.15.57:0/950555368 >> [v2:172.21.15.179:6835/1102326590,v1:172.21.15.179:6837/1102326590] conn(0x55c648ad8b40 0x55c648ad8f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:43.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.818+0000 7f9750ff9640 10 client.10281 ms_handle_connect on v2:172.21.15.179:6835/1102326590 2023-10-19T14:01:43.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.818+0000 7f9775610640 1 --2- 172.21.15.57:0/950555368 >> [v2:172.21.15.179:6835/1102326590,v1:172.21.15.179:6837/1102326590] conn(0x55c648ad8b40 0x55c648ad8f20 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c517ae541710baf0 server_cookie=1a5a6117ddcfb59f in_seq=0 out_seq=0 2023-10-19T14:01:43.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.818+0000 7f9750ff9640 1 -- 172.21.15.57:0/950555368 <== mds.0 v2:172.21.15.179:6835/1102326590 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x55c6486f96c0 con 0x55c648ad8b40 2023-10-19T14:01:43.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.818+0000 7f9750ff9640 10 client.10281 handle_command_reply: tid=0 2023-10-19T14:01:43.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.818+0000 7f977ac08740 1 client.10281 shutdown 2023-10-19T14:01:43.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.818+0000 7f977ac08740 1 -- 172.21.15.57:0/950555368 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c648ab8b40 msgr2=0x55c648b3f7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:43.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.818+0000 7f977ac08740 1 --2- 172.21.15.57:0/950555368 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c648ab8b40 0x55c648b3f7e0 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7f976c001590 tx=0x7f976c014680 comp rx=0 tx=0).stop 2023-10-19T14:01:43.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.818+0000 7f977ac08740 1 -- 172.21.15.57:0/950555368 shutdown_connections 2023-10-19T14:01:43.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.818+0000 7f977ac08740 1 --2- 172.21.15.57:0/950555368 >> [v2:172.21.15.179:6835/1102326590,v1:172.21.15.179:6837/1102326590] conn(0x55c648ad8b40 0x55c648ad8f20 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:43.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.818+0000 7f977ac08740 1 --2- 172.21.15.57:0/950555368 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c648ad23d0 0x55c648abf520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:43.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.818+0000 7f977ac08740 1 --2- 172.21.15.57:0/950555368 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55c648ab8f20 0x55c648abefe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:43.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.818+0000 7f977ac08740 1 --2- 172.21.15.57:0/950555368 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c648ab8b40 0x55c648b3f7e0 unknown :-1 s=CLOSED pgs=1925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:43.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.818+0000 7f9750ff9640 0 client.10281 ms_handle_reset on v2:172.21.15.179:6835/1102326590 2023-10-19T14:01:43.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.818+0000 7f977ac08740 1 -- 172.21.15.57:0/950555368 >> 172.21.15.57:0/950555368 conn(0x55c648a8c020 msgr2=0x55c648ad9de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:43.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.819+0000 7f977ac08740 1 -- 172.21.15.57:0/950555368 shutdown_connections 2023-10-19T14:01:43.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.819+0000 7f977ac08740 1 -- 172.21.15.57:0/950555368 wait complete. 2023-10-19T14:01:43.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.819+0000 7f977ac08740 20 client.10281 trim_cache size 0 max 16384 2023-10-19T14:01:43.845 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2023-10-19T14:01:43.845 INFO:teuthology.orchestra.run.smithi057.stdout: "cluster_fsid": "1e9beb19-3260-4dd7-85ef-5b399ce774ac", 2023-10-19T14:01:43.845 INFO:teuthology.orchestra.run.smithi057.stdout: "whoami": 0, 2023-10-19T14:01:43.845 INFO:teuthology.orchestra.run.smithi057.stdout: "id": 9139, 2023-10-19T14:01:43.846 INFO:teuthology.orchestra.run.smithi057.stdout: "want_state": "up:active", 2023-10-19T14:01:43.846 INFO:teuthology.orchestra.run.smithi057.stdout: "state": "up:active", 2023-10-19T14:01:43.846 INFO:teuthology.orchestra.run.smithi057.stdout: "fs_name": "cephfs", 2023-10-19T14:01:43.846 INFO:teuthology.orchestra.run.smithi057.stdout: "rank_uptime": 4.6788451520000001, 2023-10-19T14:01:43.846 INFO:teuthology.orchestra.run.smithi057.stdout: "mdsmap_epoch": 70, 2023-10-19T14:01:43.847 INFO:teuthology.orchestra.run.smithi057.stdout: "osdmap_epoch": 175, 2023-10-19T14:01:43.847 INFO:teuthology.orchestra.run.smithi057.stdout: "osdmap_epoch_barrier": 175, 2023-10-19T14:01:43.847 INFO:teuthology.orchestra.run.smithi057.stdout: "uptime": 9.3684095190000001 2023-10-19T14:01:43.847 INFO:teuthology.orchestra.run.smithi057.stdout:} 2023-10-19T14:01:43.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.819+0000 7f9778075640 1 -- 172.21.15.57:0/2072866982 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f975406af60 msgr2=0x7f975406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:43.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.819+0000 7f9778075640 1 --2- 172.21.15.57:0/2072866982 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f975406af60 0x7f975406d420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f976c0046b0 tx=0x7f976c007eb0 comp rx=0 tx=0).stop 2023-10-19T14:01:43.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.819+0000 7f9778075640 1 -- 172.21.15.57:0/2072866982 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f97700cbd20 msgr2=0x7f97700ae700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:43.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.819+0000 7f9778075640 1 --2- 172.21.15.57:0/2072866982 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f97700cbd20 0x7f97700ae700 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7f9768036040 tx=0x7f9768004270 comp rx=0 tx=0).stop 2023-10-19T14:01:43.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.819+0000 7f9778075640 1 -- 172.21.15.57:0/2072866982 shutdown_connections 2023-10-19T14:01:43.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.819+0000 7f9778075640 1 --2- 172.21.15.57:0/2072866982 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f975406af60 0x7f975406d420 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:43.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.819+0000 7f9778075640 1 --2- 172.21.15.57:0/2072866982 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f97700cbd20 0x7f97700ae700 unknown :-1 s=CLOSED pgs=1921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:43.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.819+0000 7f9778075640 1 --2- 172.21.15.57:0/2072866982 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f97700b0530 0x7f977014e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:43.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.819+0000 7f9778075640 1 --2- 172.21.15.57:0/2072866982 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f97700afa20 0x7f9770146680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:43.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.819+0000 7f9778075640 1 -- 172.21.15.57:0/2072866982 >> 172.21.15.57:0/2072866982 conn(0x7f9770069930 msgr2=0x7f977006c280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:43.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.819+0000 7f9778075640 1 -- 172.21.15.57:0/2072866982 shutdown_connections 2023-10-19T14:01:43.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.819+0000 7f9778075640 1 -- 172.21.15.57:0/2072866982 wait complete. 2023-10-19T14:01:43.852 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-19T14:01:43.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.959+0000 7faa94be9640 1 -- 172.21.15.57:0/468240448 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7faa90070920 msgr2=0x7faa900affe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:43.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.959+0000 7faa94be9640 1 --2- 172.21.15.57:0/468240448 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7faa90070920 0x7faa900affe0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7faa80005fd0 tx=0x7faa8002c050 comp rx=0 tx=0).stop 2023-10-19T14:01:43.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.959+0000 7faa94be9640 1 -- 172.21.15.57:0/468240448 shutdown_connections 2023-10-19T14:01:43.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.959+0000 7faa94be9640 1 --2- 172.21.15.57:0/468240448 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7faa90070920 0x7faa900affe0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:43.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.959+0000 7faa94be9640 1 --2- 172.21.15.57:0/468240448 >> v2:172.21.15.179:3300/0 conn(0x7faa90141690 0x7faa900703e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:43.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.959+0000 7faa94be9640 1 -- 172.21.15.57:0/468240448 >> 172.21.15.57:0/468240448 conn(0x7faa90069930 msgr2=0x7faa90069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:43.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.959+0000 7faa94be9640 1 -- 172.21.15.57:0/468240448 shutdown_connections 2023-10-19T14:01:43.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.959+0000 7faa94be9640 1 -- 172.21.15.57:0/468240448 wait complete. 2023-10-19T14:01:43.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.960+0000 7faa94be9640 1 Processor -- start 2023-10-19T14:01:43.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.960+0000 7faa94be9640 1 -- start start 2023-10-19T14:01:43.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.961+0000 7faa94be9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa90070920 0x7faa901638c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:43.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.961+0000 7faa94be9640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7faa9013e520 0x7faa900ac8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:43.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.961+0000 7faa94be9640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faa90141690 0x7faa901575e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:43.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.961+0000 7faa94be9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa90165370 con 0x7faa90070920 2023-10-19T14:01:43.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.961+0000 7faa94be9640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa90164640 con 0x7faa90141690 2023-10-19T14:01:43.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.961+0000 7faa94be9640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa90164ee0 con 0x7faa9013e520 2023-10-19T14:01:43.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.961+0000 7faa8e59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa90070920 0x7faa901638c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:43.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.961+0000 7faa8e59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa90070920 0x7faa901638c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35380/0 (socket says 172.21.15.57:35380) 2023-10-19T14:01:43.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.961+0000 7faa8e59c640 1 -- 172.21.15.57:0/2238010325 learned_addr learned my addr 172.21.15.57:0/2238010325 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:01:43.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.961+0000 7faa8dd9b640 1 --2- 172.21.15.57:0/2238010325 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7faa9013e520 0x7faa900ac8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:43.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.961+0000 7faa8ed9d640 1 --2- 172.21.15.57:0/2238010325 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faa90141690 0x7faa901575e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:43.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.961+0000 7faa8e59c640 1 -- 172.21.15.57:0/2238010325 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7faa9013e520 msgr2=0x7faa900ac8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:43.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.961+0000 7faa8e59c640 1 --2- 172.21.15.57:0/2238010325 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7faa9013e520 0x7faa900ac8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:43.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.961+0000 7faa8e59c640 1 -- 172.21.15.57:0/2238010325 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faa90141690 msgr2=0x7faa901575e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:43.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.961+0000 7faa8e59c640 1 --2- 172.21.15.57:0/2238010325 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faa90141690 0x7faa901575e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:43.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.961+0000 7faa8e59c640 1 -- 172.21.15.57:0/2238010325 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa84009560 con 0x7faa90070920 2023-10-19T14:01:43.973 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.961+0000 7faa8e59c640 1 --2- 172.21.15.57:0/2238010325 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa90070920 0x7faa901638c0 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7faa8800c430 tx=0x7faa8800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:43.973 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.962+0000 7faa7effd640 1 -- 172.21.15.57:0/2238010325 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa880052d0 con 0x7faa90070920 2023-10-19T14:01:43.973 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.962+0000 7faa7effd640 1 -- 172.21.15.57:0/2238010325 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7faa88007020 con 0x7faa90070920 2023-10-19T14:01:43.974 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.962+0000 7faa94be9640 1 -- 172.21.15.57:0/2238010325 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa80009840 con 0x7faa90070920 2023-10-19T14:01:43.974 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.962+0000 7faa7effd640 1 -- 172.21.15.57:0/2238010325 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa88007430 con 0x7faa90070920 2023-10-19T14:01:43.974 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.962+0000 7faa94be9640 1 -- 172.21.15.57:0/2238010325 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa90164890 con 0x7faa90070920 2023-10-19T14:01:43.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.963+0000 7faa94be9640 1 -- 172.21.15.57:0/2238010325 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa900af8e0 con 0x7faa90070920 2023-10-19T14:01:43.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.963+0000 7faa7effd640 1 -- 172.21.15.57:0/2238010325 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 91) v1 ==== 89331+0+0 (secure 0 0 0) 0x7faa880067d0 con 0x7faa90070920 2023-10-19T14:01:43.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.964+0000 7faa7effd640 1 --2- 172.21.15.57:0/2238010325 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7faa7006af60 0x7faa7006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:43.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.964+0000 7faa7effd640 1 -- 172.21.15.57:0/2238010325 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 6057+0+0 (secure 0 0 0) 0x7faa88088e30 con 0x7faa90070920 2023-10-19T14:01:43.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.964+0000 7faa8dd9b640 1 --2- 172.21.15.57:0/2238010325 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7faa7006af60 0x7faa7006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:43.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.964+0000 7faa8dd9b640 1 --2- 172.21.15.57:0/2238010325 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7faa7006af60 0x7faa7006d420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7faa900c9290 tx=0x7faa84009260 comp rx=0 tx=0).ready entity=mgr.8923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:43.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:43.968+0000 7faa7effd640 1 -- 172.21.15.57:0/2238010325 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7faa88053db0 con 0x7faa90070920 2023-10-19T14:01:44.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:44.181+0000 7faa94be9640 1 -- 172.21.15.57:0/2238010325 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7faa900690e0 con 0x7faa90070920 2023-10-19T14:01:44.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:44.182+0000 7faa7effd640 1 -- 172.21.15.57:0/2238010325 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v70) v1 ==== 53+0+73 (secure 0 0 0) 0x7faa880573d0 con 0x7faa90070920 2023-10-19T14:01:44.185 INFO:teuthology.orchestra.run.smithi057.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-19T14:01:44.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:44.183+0000 7faa94be9640 1 -- 172.21.15.57:0/2238010325 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7faa7006af60 msgr2=0x7faa7006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:44.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:44.183+0000 7faa94be9640 1 --2- 172.21.15.57:0/2238010325 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7faa7006af60 0x7faa7006d420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7faa900c9290 tx=0x7faa84009260 comp rx=0 tx=0).stop 2023-10-19T14:01:44.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:44.183+0000 7faa94be9640 1 -- 172.21.15.57:0/2238010325 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa90070920 msgr2=0x7faa901638c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:44.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:44.183+0000 7faa94be9640 1 --2- 172.21.15.57:0/2238010325 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa90070920 0x7faa901638c0 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7faa8800c430 tx=0x7faa8800c8f0 comp rx=0 tx=0).stop 2023-10-19T14:01:44.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:44.184+0000 7faa94be9640 1 -- 172.21.15.57:0/2238010325 shutdown_connections 2023-10-19T14:01:44.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:44.184+0000 7faa94be9640 1 --2- 172.21.15.57:0/2238010325 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7faa7006af60 0x7faa7006d420 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:44.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:44.184+0000 7faa94be9640 1 --2- 172.21.15.57:0/2238010325 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faa90141690 0x7faa901575e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:44.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:44.184+0000 7faa94be9640 1 --2- 172.21.15.57:0/2238010325 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7faa9013e520 0x7faa900ac8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:44.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:44.184+0000 7faa94be9640 1 --2- 172.21.15.57:0/2238010325 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa90070920 0x7faa901638c0 unknown :-1 s=CLOSED pgs=1926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:44.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:44.184+0000 7faa94be9640 1 -- 172.21.15.57:0/2238010325 >> 172.21.15.57:0/2238010325 conn(0x7faa90069930 msgr2=0x7faa90069d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:44.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:44.184+0000 7faa94be9640 1 -- 172.21.15.57:0/2238010325 shutdown_connections 2023-10-19T14:01:44.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:44.184+0000 7faa94be9640 1 -- 172.21.15.57:0/2238010325 wait complete. 2023-10-19T14:01:44.197 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-10-19T14:01:44.197 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-19T14:01:44.197 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi057.front.sepia.ceph.com 2023-10-19T14:01:44.198 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-19T14:01:44.198 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-19T14:01:44.199 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-10-19T14:01:44.199 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-10-19T14:01:44.199 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-19T14:01:44.200 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-19T14:01:44.200 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:01:44.201 DEBUG:teuthology.orchestra.run.smithi057:> ip addr 2023-10-19T14:01:44.216 INFO:teuthology.orchestra.run.smithi057.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-19T14:01:44.217 INFO:teuthology.orchestra.run.smithi057.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-19T14:01:44.217 INFO:teuthology.orchestra.run.smithi057.stdout: inet 127.0.0.1/8 scope host lo 2023-10-19T14:01:44.217 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T14:01:44.217 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 ::1/128 scope host 2023-10-19T14:01:44.218 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T14:01:44.218 INFO:teuthology.orchestra.run.smithi057.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-19T14:01:44.218 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:6b:ff:86 brd ff:ff:ff:ff:ff:ff 2023-10-19T14:01:44.218 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp7s0f0 2023-10-19T14:01:44.218 INFO:teuthology.orchestra.run.smithi057.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-19T14:01:44.219 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:6b:ff:87 brd ff:ff:ff:ff:ff:ff 2023-10-19T14:01:44.219 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp7s0f1 2023-10-19T14:01:44.219 INFO:teuthology.orchestra.run.smithi057.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-19T14:01:44.219 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:bd:15:62 brd ff:ff:ff:ff:ff:ff 2023-10-19T14:01:44.219 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp1s0f0 2023-10-19T14:01:44.220 INFO:teuthology.orchestra.run.smithi057.stdout: inet 172.21.15.57/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-19T14:01:44.220 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft 38006sec preferred_lft 38006sec 2023-10-19T14:01:44.220 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 fe80::ec4:7aff:febd:1562/64 scope link noprefixroute 2023-10-19T14:01:44.220 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T14:01:44.220 INFO:teuthology.orchestra.run.smithi057.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-19T14:01:44.221 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:bd:15:63 brd ff:ff:ff:ff:ff:ff 2023-10-19T14:01:44.221 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp1s0f1 2023-10-19T14:01:44.221 INFO:teuthology.orchestra.run.smithi057.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-19T14:01:44.221 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0e:fe:f1:b5:41:59 brd ff:ff:ff:ff:ff:ff 2023-10-19T14:01:44.221 INFO:teuthology.orchestra.run.smithi057.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-19T14:01:44.222 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T14:01:44.222 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 fe80::64b1:a2ff:feb3:b08/64 scope link 2023-10-19T14:01:44.222 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T14:01:44.222 INFO:teuthology.orchestra.run.smithi057.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-19T14:01:44.222 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0e:fe:f1:b5:41:59 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T14:01:44.223 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 fe80::cfe:f1ff:feb5:4159/64 scope link 2023-10-19T14:01:44.223 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T14:01:44.223 INFO:teuthology.orchestra.run.smithi057.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-19T14:01:44.223 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 62:53:72:9d:52:71 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-19T14:01:44.224 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 fe80::6053:72ff:fe9d:5271/64 scope link 2023-10-19T14:01:44.224 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T14:01:44.224 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:01:44.225 DEBUG:teuthology.orchestra.run.smithi057:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-19T14:01:44.286 INFO:teuthology.orchestra.run.smithi057.stdout:1 2023-10-19T14:01:44.288 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:01:44.288 DEBUG:teuthology.orchestra.run.smithi057:> route 2023-10-19T14:01:44.345 INFO:teuthology.orchestra.run.smithi057.stdout:Kernel IP routing table 2023-10-19T14:01:44.345 INFO:teuthology.orchestra.run.smithi057.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-19T14:01:44.345 INFO:teuthology.orchestra.run.smithi057.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-19T14:01:44.346 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-19T14:01:44.346 INFO:teuthology.orchestra.run.smithi057.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-19T14:01:44.346 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:01:44.347 DEBUG:teuthology.orchestra.run.smithi057:> (cd / && exec bash -c ' 2023-10-19T14:01:44.348 DEBUG:teuthology.orchestra.run.smithi057:> set -e 2023-10-19T14:01:44.348 DEBUG:teuthology.orchestra.run.smithi057:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-19T14:01:44.348 DEBUG:teuthology.orchestra.run.smithi057:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-19T14:01:44.349 DEBUG:teuthology.orchestra.run.smithi057:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-19T14:01:44.349 DEBUG:teuthology.orchestra.run.smithi057:> ') 2023-10-19T14:01:44.451 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:01:44.452 DEBUG:teuthology.orchestra.run.smithi057:> ip netns list 2023-10-19T14:01:44.467 INFO:teuthology.orchestra.run.smithi057.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-19T14:01:44.468 INFO:teuthology.orchestra.run.smithi057.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-19T14:01:44.468 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-19T14:01:44.469 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:01:44.469 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip link set brx.0 up 2023-10-19T14:01:44.538 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-10-19T14:01:44.539 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-19T14:01:44.539 DEBUG:teuthology.orchestra.run.smithi057:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-19T14:01:44.593 INFO:teuthology.orchestra.run.smithi057.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-19T14:01:44.594 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-19T14:01:44.594 DEBUG:teuthology.orchestra.run.smithi057:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-19T14:01:44.648 DEBUG:teuthology.orchestra.run.smithi057:> sudo modprobe fuse 2023-10-19T14:01:44.719 DEBUG:teuthology.orchestra.run.smithi057:> cat /proc/self/mounts | awk '{print $2}' 2023-10-19T14:01:44.775 INFO:teuthology.orchestra.run.smithi057.stdout:/proc 2023-10-19T14:01:44.775 INFO:teuthology.orchestra.run.smithi057.stdout:/sys 2023-10-19T14:01:44.775 INFO:teuthology.orchestra.run.smithi057.stdout:/dev 2023-10-19T14:01:44.775 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/security 2023-10-19T14:01:44.776 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/shm 2023-10-19T14:01:44.776 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/pts 2023-10-19T14:01:44.776 INFO:teuthology.orchestra.run.smithi057.stdout:/run 2023-10-19T14:01:44.776 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/cgroup 2023-10-19T14:01:44.777 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/pstore 2023-10-19T14:01:44.777 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/bpf 2023-10-19T14:01:44.777 INFO:teuthology.orchestra.run.smithi057.stdout:/ 2023-10-19T14:01:44.777 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/selinux 2023-10-19T14:01:44.777 INFO:teuthology.orchestra.run.smithi057.stdout:/proc/sys/fs/binfmt_misc 2023-10-19T14:01:44.778 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/hugepages 2023-10-19T14:01:44.778 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/mqueue 2023-10-19T14:01:44.778 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/debug 2023-10-19T14:01:44.778 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/tracing 2023-10-19T14:01:44.779 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/fuse/connections 2023-10-19T14:01:44.779 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/config 2023-10-19T14:01:44.779 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-sysctl.service 2023-10-19T14:01:44.779 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-10-19T14:01:44.779 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-10-19T14:01:44.779 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/nfs/rpc_pipefs 2023-10-19T14:01:44.781 INFO:teuthology.orchestra.run.smithi057.stdout:/run/user/1000 2023-10-19T14:01:44.781 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph 2023-10-19T14:01:44.782 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-0 2023-10-19T14:01:44.782 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-1 2023-10-19T14:01:44.782 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-2 2023-10-19T14:01:44.782 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-3 2023-10-19T14:01:44.782 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns 2023-10-19T14:01:44.783 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T14:01:44.783 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T14:01:44.783 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-19T14:01:44.783 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-19T14:01:44.784 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:01:44.784 DEBUG:teuthology.orchestra.run.smithi057:> ls /sys/fs/fuse/connections 2023-10-19T14:01:44.830 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-10-19T14:01:44.830 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-10-19T14:01:44.872 DEBUG:teuthology.orchestra.run.smithi057:> sudo modprobe fuse 2023-10-19T14:01:44.903 DEBUG:teuthology.orchestra.run.smithi057:> cat /proc/self/mounts | awk '{print $2}' 2023-10-19T14:01:44.959 INFO:teuthology.orchestra.run.smithi057.stdout:/proc 2023-10-19T14:01:44.959 INFO:teuthology.orchestra.run.smithi057.stdout:/sys 2023-10-19T14:01:44.960 INFO:teuthology.orchestra.run.smithi057.stdout:/dev 2023-10-19T14:01:44.960 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/security 2023-10-19T14:01:44.960 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/shm 2023-10-19T14:01:44.960 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/pts 2023-10-19T14:01:44.960 INFO:teuthology.orchestra.run.smithi057.stdout:/run 2023-10-19T14:01:44.961 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/cgroup 2023-10-19T14:01:44.961 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/pstore 2023-10-19T14:01:44.961 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/bpf 2023-10-19T14:01:44.961 INFO:teuthology.orchestra.run.smithi057.stdout:/ 2023-10-19T14:01:44.961 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/selinux 2023-10-19T14:01:44.962 INFO:teuthology.orchestra.run.smithi057.stdout:/proc/sys/fs/binfmt_misc 2023-10-19T14:01:44.962 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/hugepages 2023-10-19T14:01:44.962 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/mqueue 2023-10-19T14:01:44.962 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/debug 2023-10-19T14:01:44.962 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/tracing 2023-10-19T14:01:44.963 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/fuse/connections 2023-10-19T14:01:44.963 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/config 2023-10-19T14:01:44.963 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-sysctl.service 2023-10-19T14:01:44.963 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-10-19T14:01:44.963 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-10-19T14:01:44.964 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/nfs/rpc_pipefs 2023-10-19T14:01:44.964 INFO:teuthology.orchestra.run.smithi057.stdout:/run/user/1000 2023-10-19T14:01:44.964 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph 2023-10-19T14:01:44.964 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-0 2023-10-19T14:01:44.964 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-1 2023-10-19T14:01:44.965 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-2 2023-10-19T14:01:44.965 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-3 2023-10-19T14:01:44.965 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns 2023-10-19T14:01:44.965 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T14:01:44.965 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T14:01:44.966 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-19T14:01:44.966 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-19T14:01:44.966 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:01:44.967 DEBUG:teuthology.orchestra.run.smithi057:> ls /sys/fs/fuse/connections 2023-10-19T14:01:44.968 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi057.stderr:2023-10-19T14:01:44.963+0000 7f4f8680c480 -1 init, newargv = 0x5651e5b3fce0 newargc=15 2023-10-19T14:01:44.968 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi057.stderr:ceph-fuse[142079]: starting ceph client 2023-10-19T14:01:44.970 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi057.stderr:ceph-fuse[142079]: starting fuse 2023-10-19T14:01:44.983 INFO:teuthology.orchestra.run.smithi057.stdout:43 2023-10-19T14:01:44.983 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2023-10-19T14:01:44.984 DEBUG:teuthology.orchestra.run.smithi057:> sudo stdin-killer -- python3 -c ' 2023-10-19T14:01:44.984 DEBUG:teuthology.orchestra.run.smithi057:> import glob 2023-10-19T14:01:44.984 DEBUG:teuthology.orchestra.run.smithi057:> import re 2023-10-19T14:01:44.985 DEBUG:teuthology.orchestra.run.smithi057:> import os 2023-10-19T14:01:44.985 DEBUG:teuthology.orchestra.run.smithi057:> import subprocess 2023-10-19T14:01:44.986 DEBUG:teuthology.orchestra.run.smithi057:> 2023-10-19T14:01:44.986 DEBUG:teuthology.orchestra.run.smithi057:> def _find_admin_socket(client_name): 2023-10-19T14:01:44.986 DEBUG:teuthology.orchestra.run.smithi057:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-10-19T14:01:44.987 DEBUG:teuthology.orchestra.run.smithi057:> files = glob.glob(asok_path) 2023-10-19T14:01:44.987 DEBUG:teuthology.orchestra.run.smithi057:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-10-19T14:01:44.987 DEBUG:teuthology.orchestra.run.smithi057:> 2023-10-19T14:01:44.988 DEBUG:teuthology.orchestra.run.smithi057:> # Given a non-glob path, it better be there 2023-10-19T14:01:44.988 DEBUG:teuthology.orchestra.run.smithi057:> if "*" not in asok_path: 2023-10-19T14:01:44.989 DEBUG:teuthology.orchestra.run.smithi057:> assert(len(files) == 1) 2023-10-19T14:01:44.989 DEBUG:teuthology.orchestra.run.smithi057:> return files[0] 2023-10-19T14:01:44.989 DEBUG:teuthology.orchestra.run.smithi057:> 2023-10-19T14:01:44.990 DEBUG:teuthology.orchestra.run.smithi057:> for f in files: 2023-10-19T14:01:44.990 DEBUG:teuthology.orchestra.run.smithi057:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-10-19T14:01:44.990 DEBUG:teuthology.orchestra.run.smithi057:> if os.path.exists("/proc/{0}".format(pid)): 2023-10-19T14:01:44.991 DEBUG:teuthology.orchestra.run.smithi057:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-10-19T14:01:44.991 DEBUG:teuthology.orchestra.run.smithi057:> contents = proc_f.read() 2023-10-19T14:01:44.992 DEBUG:teuthology.orchestra.run.smithi057:> if mountpoint in contents: 2023-10-19T14:01:44.992 DEBUG:teuthology.orchestra.run.smithi057:> return f 2023-10-19T14:01:44.992 DEBUG:teuthology.orchestra.run.smithi057:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-10-19T14:01:44.993 DEBUG:teuthology.orchestra.run.smithi057:> 2023-10-19T14:01:44.993 DEBUG:teuthology.orchestra.run.smithi057:> print(_find_admin_socket("client.0")) 2023-10-19T14:01:44.993 DEBUG:teuthology.orchestra.run.smithi057:> ' 2023-10-19T14:01:45.108 INFO:teuthology.orchestra.run.smithi057.stdout:/var/run/ceph/ceph-client.0.142079.asok 2023-10-19T14:01:45.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2023-10-19T14:01:45.119 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.142079.asok 2023-10-19T14:01:45.119 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:01:45.120 DEBUG:teuthology.orchestra.run.smithi057:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.142079.asok status 2023-10-19T14:01:45.124 INFO:tasks.ceph.mds.d.smithi179.stderr:2023-10-19T14:01:45.122+0000 7f88a6b24640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-19T14:01:45.260 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2023-10-19T14:01:45.261 INFO:teuthology.orchestra.run.smithi057.stdout: "metadata": { 2023-10-19T14:01:45.261 INFO:teuthology.orchestra.run.smithi057.stdout: "ceph_sha1": "f7420d7f8e55ceea786dcbf536c338ce920c21eb", 2023-10-19T14:01:45.261 INFO:teuthology.orchestra.run.smithi057.stdout: "ceph_version": "ceph version 18.0.0-6786-gf7420d7f (f7420d7f8e55ceea786dcbf536c338ce920c21eb) reef (dev)", 2023-10-19T14:01:45.261 INFO:teuthology.orchestra.run.smithi057.stdout: "entity_id": "0", 2023-10-19T14:01:45.262 INFO:teuthology.orchestra.run.smithi057.stdout: "hostname": "smithi057", 2023-10-19T14:01:45.262 INFO:teuthology.orchestra.run.smithi057.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-10-19T14:01:45.262 INFO:teuthology.orchestra.run.smithi057.stdout: "pid": "142079", 2023-10-19T14:01:45.262 INFO:teuthology.orchestra.run.smithi057.stdout: "root": "/" 2023-10-19T14:01:45.262 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:01:45.263 INFO:teuthology.orchestra.run.smithi057.stdout: "dentry_count": 0, 2023-10-19T14:01:45.263 INFO:teuthology.orchestra.run.smithi057.stdout: "dentry_pinned_count": 0, 2023-10-19T14:01:45.263 INFO:teuthology.orchestra.run.smithi057.stdout: "id": 10290, 2023-10-19T14:01:45.263 INFO:teuthology.orchestra.run.smithi057.stdout: "inst": { 2023-10-19T14:01:45.263 INFO:teuthology.orchestra.run.smithi057.stdout: "name": { 2023-10-19T14:01:45.264 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "client", 2023-10-19T14:01:45.264 INFO:teuthology.orchestra.run.smithi057.stdout: "num": 10290 2023-10-19T14:01:45.264 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:01:45.264 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": { 2023-10-19T14:01:45.264 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "v1", 2023-10-19T14:01:45.265 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": "192.168.0.1:0", 2023-10-19T14:01:45.265 INFO:teuthology.orchestra.run.smithi057.stdout: "nonce": 2214762694 2023-10-19T14:01:45.265 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:01:45.265 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:01:45.265 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": { 2023-10-19T14:01:45.266 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "v1", 2023-10-19T14:01:45.266 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": "192.168.0.1:0", 2023-10-19T14:01:45.266 INFO:teuthology.orchestra.run.smithi057.stdout: "nonce": 2214762694 2023-10-19T14:01:45.266 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:01:45.266 INFO:teuthology.orchestra.run.smithi057.stdout: "inst_str": "client.10290 192.168.0.1:0/2214762694", 2023-10-19T14:01:45.267 INFO:teuthology.orchestra.run.smithi057.stdout: "addr_str": "192.168.0.1:0/2214762694", 2023-10-19T14:01:45.267 INFO:teuthology.orchestra.run.smithi057.stdout: "inode_count": 1, 2023-10-19T14:01:45.267 INFO:teuthology.orchestra.run.smithi057.stdout: "mds_epoch": 70, 2023-10-19T14:01:45.267 INFO:teuthology.orchestra.run.smithi057.stdout: "osd_epoch": 175, 2023-10-19T14:01:45.267 INFO:teuthology.orchestra.run.smithi057.stdout: "osd_epoch_barrier": 0, 2023-10-19T14:01:45.267 INFO:teuthology.orchestra.run.smithi057.stdout: "blocklisted": false, 2023-10-19T14:01:45.268 INFO:teuthology.orchestra.run.smithi057.stdout: "fs_name": "cephfs" 2023-10-19T14:01:45.268 INFO:teuthology.orchestra.run.smithi057.stdout:} 2023-10-19T14:01:45.269 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:01:45.270 DEBUG:teuthology.orchestra.run.smithi057:> stat --file-system '--printf=%T 2023-10-19T14:01:45.270 DEBUG:teuthology.orchestra.run.smithi057:> ' -- /home/ubuntu/cephtest/mnt.0 2023-10-19T14:01:45.286 INFO:teuthology.orchestra.run.smithi057.stdout:fuseblk 2023-10-19T14:01:45.286 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-10-19T14:01:45.286 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:01:45.287 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-19T14:01:45.357 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-19T14:01:45.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.506+0000 7f2b44525640 1 -- 172.21.15.57:0/1385188018 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2b28004350 msgr2=0x7f2b28004820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:45.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.506+0000 7f2b44525640 1 --2- 172.21.15.57:0/1385188018 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2b28004350 0x7f2b28004820 secure :-1 s=READY pgs=1930 cs=0 l=1 rev1=1 crypto rx=0x7f2b2c006f70 tx=0x7f2b280057f0 comp rx=0 tx=0).stop 2023-10-19T14:01:45.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.506+0000 7f2b42ac2640 1 -- 172.21.15.57:0/1385188018 reap_dead start 2023-10-19T14:01:45.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.506+0000 7f2b44525640 1 -- 172.21.15.57:0/1385188018 shutdown_connections 2023-10-19T14:01:45.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.506+0000 7f2b44525640 1 -- 172.21.15.57:0/1385188018 >> 172.21.15.57:0/1385188018 conn(0x7f2b3c069930 msgr2=0x7f2b3c069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:45.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.506+0000 7f2b44525640 1 -- 172.21.15.57:0/1385188018 shutdown_connections 2023-10-19T14:01:45.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.506+0000 7f2b44525640 1 -- 172.21.15.57:0/1385188018 wait complete. 2023-10-19T14:01:45.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.507+0000 7f2b44525640 1 Processor -- start 2023-10-19T14:01:45.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.507+0000 7f2b44525640 1 -- start start 2023-10-19T14:01:45.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.508+0000 7f2b44525640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2b28004350 0x7f2b3c136260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:45.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.508+0000 7f2b44525640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2b3c15a570 0x7f2b3c15a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:45.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.508+0000 7f2b44525640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2b3c0ad710 0x7f2b3c0adb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:45.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.508+0000 7f2b44525640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b3c0ae0d0 con 0x7f2b3c0ad710 2023-10-19T14:01:45.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.508+0000 7f2b44525640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b3c1527f0 con 0x7f2b3c15a570 2023-10-19T14:01:45.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.508+0000 7f2b44525640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b3c152ab0 con 0x7f2b28004350 2023-10-19T14:01:45.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.508+0000 7f2b42ac2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2b3c0ad710 0x7f2b3c0adb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:45.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.508+0000 7f2b42ac2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2b3c0ad710 0x7f2b3c0adb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35408/0 (socket says 172.21.15.57:35408) 2023-10-19T14:01:45.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.508+0000 7f2b42ac2640 1 -- 172.21.15.57:0/1902636516 learned_addr learned my addr 172.21.15.57:0/1902636516 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:01:45.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.508+0000 7f2b42ac2640 1 -- 172.21.15.57:0/1902636516 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2b28004350 msgr2=0x7f2b3c136260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:45.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.508+0000 7f2b42ac2640 1 --2- 172.21.15.57:0/1902636516 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2b28004350 0x7f2b3c136260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:45.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.508+0000 7f2b42ac2640 1 -- 172.21.15.57:0/1902636516 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2b3c15a570 msgr2=0x7f2b3c15a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:45.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.508+0000 7f2b42ac2640 1 --2- 172.21.15.57:0/1902636516 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2b3c15a570 0x7f2b3c15a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:45.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.508+0000 7f2b42ac2640 1 -- 172.21.15.57:0/1902636516 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b3c152d90 con 0x7f2b3c0ad710 2023-10-19T14:01:45.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.509+0000 7f2b42ac2640 1 --2- 172.21.15.57:0/1902636516 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2b3c0ad710 0x7f2b3c0adb90 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 crypto rx=0x7f2b3c06d510 tx=0x7f2b34010f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:45.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.509+0000 7f2b32ffd640 1 -- 172.21.15.57:0/1902636516 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b3400eeb0 con 0x7f2b3c0ad710 2023-10-19T14:01:45.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.509+0000 7f2b32ffd640 1 -- 172.21.15.57:0/1902636516 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f2b34004c10 con 0x7f2b3c0ad710 2023-10-19T14:01:45.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.509+0000 7f2b44525640 1 -- 172.21.15.57:0/1902636516 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b3c153070 con 0x7f2b3c0ad710 2023-10-19T14:01:45.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.509+0000 7f2b32ffd640 1 -- 172.21.15.57:0/1902636516 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b34003970 con 0x7f2b3c0ad710 2023-10-19T14:01:45.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.509+0000 7f2b44525640 1 -- 172.21.15.57:0/1902636516 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b3c157720 con 0x7f2b3c0ad710 2023-10-19T14:01:45.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.510+0000 7f2b44525640 1 -- 172.21.15.57:0/1902636516 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b3c153260 con 0x7f2b3c0ad710 2023-10-19T14:01:45.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.510+0000 7f2b32ffd640 1 -- 172.21.15.57:0/1902636516 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 91) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f2b34003ad0 con 0x7f2b3c0ad710 2023-10-19T14:01:45.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.511+0000 7f2b32ffd640 1 --2- 172.21.15.57:0/1902636516 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f2b2006af60 0x7f2b2006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:45.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.511+0000 7f2b32ffd640 1 -- 172.21.15.57:0/1902636516 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f2b34017070 con 0x7f2b3c0ad710 2023-10-19T14:01:45.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.511+0000 7f2b422c1640 1 --2- 172.21.15.57:0/1902636516 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f2b2006af60 0x7f2b2006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:45.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.512+0000 7f2b422c1640 1 --2- 172.21.15.57:0/1902636516 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f2b2006af60 0x7f2b2006d420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2b380069f0 tx=0x7f2b38006aa0 comp rx=0 tx=0).ready entity=mgr.8923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:45.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.515+0000 7f2b32ffd640 1 -- 172.21.15.57:0/1902636516 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2b34055180 con 0x7f2b3c0ad710 2023-10-19T14:01:45.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.713+0000 7f2b44525640 1 -- 172.21.15.57:0/1902636516 --> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f2b3c160e00 con 0x7f2b2006af60 2023-10-19T14:01:45.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.715+0000 7f2b32ffd640 1 -- 172.21.15.57:0/1902636516 <== mgr.8923 v2:172.21.15.179:6832/2037224568 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f2b38006130 con 0x7f2b2006af60 2023-10-19T14:01:45.718 INFO:teuthology.orchestra.run.smithi057.stdout:[ 2023-10-19T14:01:45.718 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:01:45.718 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "cephfs" 2023-10-19T14:01:45.718 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:01:45.719 INFO:teuthology.orchestra.run.smithi057.stdout:] 2023-10-19T14:01:45.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.717+0000 7f2b44525640 1 -- 172.21.15.57:0/1902636516 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f2b2006af60 msgr2=0x7f2b2006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:45.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.717+0000 7f2b44525640 1 --2- 172.21.15.57:0/1902636516 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f2b2006af60 0x7f2b2006d420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2b380069f0 tx=0x7f2b38006aa0 comp rx=0 tx=0).stop 2023-10-19T14:01:45.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.717+0000 7f2b44525640 1 -- 172.21.15.57:0/1902636516 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2b3c0ad710 msgr2=0x7f2b3c0adb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:45.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.717+0000 7f2b44525640 1 --2- 172.21.15.57:0/1902636516 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2b3c0ad710 0x7f2b3c0adb90 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 crypto rx=0x7f2b3c06d510 tx=0x7f2b34010f40 comp rx=0 tx=0).stop 2023-10-19T14:01:45.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.717+0000 7f2b44525640 1 -- 172.21.15.57:0/1902636516 shutdown_connections 2023-10-19T14:01:45.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.717+0000 7f2b44525640 1 --2- 172.21.15.57:0/1902636516 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f2b2006af60 0x7f2b2006d420 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:45.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.717+0000 7f2b44525640 1 --2- 172.21.15.57:0/1902636516 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2b3c0ad710 0x7f2b3c0adb90 unknown :-1 s=CLOSED pgs=1931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:45.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.717+0000 7f2b44525640 1 --2- 172.21.15.57:0/1902636516 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2b3c15a570 0x7f2b3c15a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:45.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.717+0000 7f2b44525640 1 --2- 172.21.15.57:0/1902636516 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2b28004350 0x7f2b3c136260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:45.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.717+0000 7f2b44525640 1 -- 172.21.15.57:0/1902636516 >> 172.21.15.57:0/1902636516 conn(0x7f2b3c069930 msgr2=0x7f2b3c0ca870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:45.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.717+0000 7f2b44525640 1 -- 172.21.15.57:0/1902636516 shutdown_connections 2023-10-19T14:01:45.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.717+0000 7f2b44525640 1 -- 172.21.15.57:0/1902636516 wait complete. 2023-10-19T14:01:45.730 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-19T14:01:45.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.839+0000 7fde46383640 1 -- 172.21.15.57:0/1809347716 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde280047c0 msgr2=0x7fde28004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:45.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.839+0000 7fde46383640 1 --2- 172.21.15.57:0/1809347716 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde280047c0 0x7fde28004c40 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 crypto rx=0x7fde34006fd0 tx=0x7fde280057f0 comp rx=0 tx=0).stop 2023-10-19T14:01:45.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.839+0000 7fde46383640 1 -- 172.21.15.57:0/1809347716 shutdown_connections 2023-10-19T14:01:45.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.839+0000 7fde46383640 1 --2- 172.21.15.57:0/1809347716 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde280047c0 0x7fde28004c40 unknown :-1 s=CLOSED pgs=1933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:45.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.839+0000 7fde46383640 1 -- 172.21.15.57:0/1809347716 >> 172.21.15.57:0/1809347716 conn(0x7fde40069930 msgr2=0x7fde40069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:45.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.839+0000 7fde46383640 1 -- 172.21.15.57:0/1809347716 shutdown_connections 2023-10-19T14:01:45.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.839+0000 7fde46383640 1 -- 172.21.15.57:0/1809347716 wait complete. 2023-10-19T14:01:45.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.840+0000 7fde46383640 1 Processor -- start 2023-10-19T14:01:45.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.840+0000 7fde46383640 1 -- start start 2023-10-19T14:01:45.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.841+0000 7fde46383640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fde280047c0 0x7fde4013d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:45.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.841+0000 7fde46383640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde401e2970 0x7fde401e19a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:45.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.841+0000 7fde46383640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fde401e0fa0 0x7fde400ad7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:45.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.841+0000 7fde46383640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde401e0270 con 0x7fde401e2970 2023-10-19T14:01:45.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.841+0000 7fde46383640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde401e2df0 con 0x7fde280047c0 2023-10-19T14:01:45.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.841+0000 7fde46383640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde401e1ee0 con 0x7fde401e0fa0 2023-10-19T14:01:45.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.841+0000 7fde3f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde401e2970 0x7fde401e19a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:45.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.841+0000 7fde3f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde401e2970 0x7fde401e19a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35424/0 (socket says 172.21.15.57:35424) 2023-10-19T14:01:45.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.841+0000 7fde3f7fe640 1 -- 172.21.15.57:0/3959242069 learned_addr learned my addr 172.21.15.57:0/3959242069 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:01:45.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.841+0000 7fde3ffff640 1 --2- 172.21.15.57:0/3959242069 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fde280047c0 0x7fde4013d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:45.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.841+0000 7fde44920640 1 --2- 172.21.15.57:0/3959242069 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fde401e0fa0 0x7fde400ad7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:45.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.842+0000 7fde3f7fe640 1 -- 172.21.15.57:0/3959242069 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fde401e0fa0 msgr2=0x7fde400ad7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:45.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.842+0000 7fde3f7fe640 1 --2- 172.21.15.57:0/3959242069 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fde401e0fa0 0x7fde400ad7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:45.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.842+0000 7fde3f7fe640 1 -- 172.21.15.57:0/3959242069 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fde280047c0 msgr2=0x7fde4013d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:45.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.842+0000 7fde3f7fe640 1 --2- 172.21.15.57:0/3959242069 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fde280047c0 0x7fde4013d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:45.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.842+0000 7fde3f7fe640 1 -- 172.21.15.57:0/3959242069 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde401e2160 con 0x7fde401e2970 2023-10-19T14:01:45.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.842+0000 7fde3ffff640 1 --2- 172.21.15.57:0/3959242069 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fde280047c0 0x7fde4013d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:01:45.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.842+0000 7fde3f7fe640 1 --2- 172.21.15.57:0/3959242069 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde401e2970 0x7fde401e19a0 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x7fde34001930 tx=0x7fde34001720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:45.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.842+0000 7fde3cff9640 1 -- 172.21.15.57:0/3959242069 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde3403c040 con 0x7fde401e2970 2023-10-19T14:01:45.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.842+0000 7fde3cff9640 1 -- 172.21.15.57:0/3959242069 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fde34003a00 con 0x7fde401e2970 2023-10-19T14:01:45.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.842+0000 7fde3cff9640 1 -- 172.21.15.57:0/3959242069 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde34003e10 con 0x7fde401e2970 2023-10-19T14:01:45.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.842+0000 7fde46383640 1 -- 172.21.15.57:0/3959242069 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde401e04c0 con 0x7fde401e2970 2023-10-19T14:01:45.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.842+0000 7fde46383640 1 -- 172.21.15.57:0/3959242069 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde401e0a80 con 0x7fde401e2970 2023-10-19T14:01:45.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.843+0000 7fde46383640 1 -- 172.21.15.57:0/3959242069 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde401e0c10 con 0x7fde401e2970 2023-10-19T14:01:45.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.844+0000 7fde3cff9640 1 -- 172.21.15.57:0/3959242069 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 91) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fde34032070 con 0x7fde401e2970 2023-10-19T14:01:45.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.844+0000 7fde3cff9640 1 --2- 172.21.15.57:0/3959242069 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7fde2c06af60 0x7fde2c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:45.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.844+0000 7fde3cff9640 1 -- 172.21.15.57:0/3959242069 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fde340ab0f0 con 0x7fde401e2970 2023-10-19T14:01:45.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.844+0000 7fde3ffff640 1 --2- 172.21.15.57:0/3959242069 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7fde2c06af60 0x7fde2c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:45.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.845+0000 7fde3ffff640 1 --2- 172.21.15.57:0/3959242069 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7fde2c06af60 0x7fde2c06d420 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fde30006530 tx=0x7fde30006b30 comp rx=0 tx=0).ready entity=mgr.8923 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:45.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:45.848+0000 7fde3cff9640 1 -- 172.21.15.57:0/3959242069 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fde34075f40 con 0x7fde401e2970 2023-10-19T14:01:46.074 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.071+0000 7fde46383640 1 -- 172.21.15.57:0/3959242069 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7fde400690e0 con 0x7fde401e2970 2023-10-19T14:01:46.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.192+0000 7fde3cff9640 1 -- 172.21.15.57:0/3959242069 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v92) v1 ==== 92+0+0 (secure 0 0 0) 0x7fde34079560 con 0x7fde401e2970 2023-10-19T14:01:46.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.194+0000 7fde46383640 1 -- 172.21.15.57:0/3959242069 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7fde2c06af60 msgr2=0x7fde2c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:46.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.194+0000 7fde46383640 1 --2- 172.21.15.57:0/3959242069 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7fde2c06af60 0x7fde2c06d420 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fde30006530 tx=0x7fde30006b30 comp rx=0 tx=0).stop 2023-10-19T14:01:46.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.194+0000 7fde46383640 1 -- 172.21.15.57:0/3959242069 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde401e2970 msgr2=0x7fde401e19a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:46.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.194+0000 7fde46383640 1 --2- 172.21.15.57:0/3959242069 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde401e2970 0x7fde401e19a0 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x7fde34001930 tx=0x7fde34001720 comp rx=0 tx=0).stop 2023-10-19T14:01:46.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.195+0000 7fde46383640 1 -- 172.21.15.57:0/3959242069 shutdown_connections 2023-10-19T14:01:46.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.195+0000 7fde46383640 1 --2- 172.21.15.57:0/3959242069 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7fde2c06af60 0x7fde2c06d420 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:46.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.195+0000 7fde46383640 1 --2- 172.21.15.57:0/3959242069 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fde401e0fa0 0x7fde400ad7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:46.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.195+0000 7fde46383640 1 --2- 172.21.15.57:0/3959242069 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde401e2970 0x7fde401e19a0 unknown :-1 s=CLOSED pgs=1934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:46.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.195+0000 7fde46383640 1 --2- 172.21.15.57:0/3959242069 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fde280047c0 0x7fde4013d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:46.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.195+0000 7fde46383640 1 -- 172.21.15.57:0/3959242069 >> 172.21.15.57:0/3959242069 conn(0x7fde40069930 msgr2=0x7fde400cbca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:46.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.195+0000 7fde46383640 1 -- 172.21.15.57:0/3959242069 shutdown_connections 2023-10-19T14:01:46.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.195+0000 7fde46383640 1 -- 172.21.15.57:0/3959242069 wait complete. 2023-10-19T14:01:46.212 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-19T14:01:46.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.324+0000 7f3f1925d640 1 -- 172.21.15.57:0/3852005117 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3f140ceb40 msgr2=0x7f3f140b0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:46.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.324+0000 7f3f1925d640 1 --2- 172.21.15.57:0/3852005117 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3f140ceb40 0x7f3f140b0650 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f3f0c005fd0 tx=0x7f3f0c02c050 comp rx=0 tx=0).stop 2023-10-19T14:01:46.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.324+0000 7f3f1925d640 1 -- 172.21.15.57:0/3852005117 shutdown_connections 2023-10-19T14:01:46.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.324+0000 7f3f1925d640 1 --2- 172.21.15.57:0/3852005117 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3f140ceb40 0x7f3f140b0650 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:46.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.324+0000 7f3f1925d640 1 --2- 172.21.15.57:0/3852005117 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3f1415b2a0 0x7f3f140d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:46.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.324+0000 7f3f1925d640 1 -- 172.21.15.57:0/3852005117 >> 172.21.15.57:0/3852005117 conn(0x7f3f140d0d60 msgr2=0x7f3f140d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:46.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.324+0000 7f3f1925d640 1 -- 172.21.15.57:0/3852005117 shutdown_connections 2023-10-19T14:01:46.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.325+0000 7f3f1925d640 1 -- 172.21.15.57:0/3852005117 wait complete. 2023-10-19T14:01:46.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.326+0000 7f3f1925d640 1 Processor -- start 2023-10-19T14:01:46.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.326+0000 7f3f1925d640 1 -- start start 2023-10-19T14:01:46.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.327+0000 7f3f1925d640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3f140ceb40 0x7f3f141f53d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:46.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.327+0000 7f3f1925d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3f140cf340 0x7f3f14151420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:46.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.327+0000 7f3f1925d640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3f1415b2a0 0x7f3f141e5880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:46.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.327+0000 7f3f1925d640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f140cac80 con 0x7f3f140cf340 2023-10-19T14:01:46.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.327+0000 7f3f1925d640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f141f5910 con 0x7f3f140ceb40 2023-10-19T14:01:46.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.327+0000 7f3f1925d640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f14151960 con 0x7f3f1415b2a0 2023-10-19T14:01:46.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.327+0000 7f3f0bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3f140cf340 0x7f3f14151420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:46.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.327+0000 7f3f0bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3f140cf340 0x7f3f14151420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35458/0 (socket says 172.21.15.57:35458) 2023-10-19T14:01:46.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.327+0000 7f3f1359e640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3f1415b2a0 0x7f3f141e5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:46.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.327+0000 7f3f0bfff640 1 -- 172.21.15.57:0/2626132752 learned_addr learned my addr 172.21.15.57:0/2626132752 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:01:46.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.327+0000 7f3f12d9d640 1 --2- 172.21.15.57:0/2626132752 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3f140ceb40 0x7f3f141f53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:46.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.327+0000 7f3f1359e640 1 -- 172.21.15.57:0/2626132752 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3f140ceb40 msgr2=0x7f3f141f53d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:46.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.327+0000 7f3f1359e640 1 --2- 172.21.15.57:0/2626132752 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3f140ceb40 0x7f3f141f53d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:46.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.327+0000 7f3f1359e640 1 -- 172.21.15.57:0/2626132752 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3f140cf340 msgr2=0x7f3f14151420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:46.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.327+0000 7f3f1359e640 1 --2- 172.21.15.57:0/2626132752 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3f140cf340 0x7f3f14151420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:46.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.327+0000 7f3f1359e640 1 -- 172.21.15.57:0/2626132752 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3efc009490 con 0x7f3f1415b2a0 2023-10-19T14:01:46.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.327+0000 7f3f12d9d640 1 --2- 172.21.15.57:0/2626132752 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3f140ceb40 0x7f3f141f53d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T14:01:46.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.327+0000 7f3f0bfff640 1 --2- 172.21.15.57:0/2626132752 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3f140cf340 0x7f3f14151420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T14:01:46.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.327+0000 7f3f1359e640 1 --2- 172.21.15.57:0/2626132752 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3f1415b2a0 0x7f3f141e5880 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f3f14069e00 tx=0x7f3f0c0042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:46.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.328+0000 7f3f0b7fe640 1 -- 172.21.15.57:0/2626132752 <== mon.2 v2:172.21.15.179:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f0c003e80 con 0x7f3f1415b2a0 2023-10-19T14:01:46.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.328+0000 7f3f0b7fe640 1 -- 172.21.15.57:0/2626132752 <== mon.2 v2:172.21.15.179:3301/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f3f0c03e460 con 0x7f3f1415b2a0 2023-10-19T14:01:46.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.328+0000 7f3f0b7fe640 1 -- 172.21.15.57:0/2626132752 <== mon.2 v2:172.21.15.179:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f0c03e7d0 con 0x7f3f1415b2a0 2023-10-19T14:01:46.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.328+0000 7f3f1925d640 1 -- 172.21.15.57:0/2626132752 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f0c009840 con 0x7f3f1415b2a0 2023-10-19T14:01:46.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.328+0000 7f3f1925d640 1 -- 172.21.15.57:0/2626132752 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f140ca2e0 con 0x7f3f1415b2a0 2023-10-19T14:01:46.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.329+0000 7f3f1925d640 1 -- 172.21.15.57:0/2626132752 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f140cadc0 con 0x7f3f1415b2a0 2023-10-19T14:01:46.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.330+0000 7f3f0b7fe640 1 -- 172.21.15.57:0/2626132752 <== mon.2 v2:172.21.15.179:3301/0 4 ==== mgrmap(e 92) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f3f0c03e930 con 0x7f3f1415b2a0 2023-10-19T14:01:46.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.330+0000 7f3f0b7fe640 1 --2- 172.21.15.57:0/2626132752 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f3ef406afb0 0x7f3ef406d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:46.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.330+0000 7f3f0b7fe640 1 -- 172.21.15.57:0/2626132752 <== mon.2 v2:172.21.15.179:3301/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f3f0c0ac670 con 0x7f3f1415b2a0 2023-10-19T14:01:46.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.330+0000 7f3f12d9d640 1 -- 172.21.15.57:0/2626132752 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f3ef406afb0 msgr2=0x7f3ef406d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/2037224568 2023-10-19T14:01:46.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.330+0000 7f3f12d9d640 1 --2- 172.21.15.57:0/2626132752 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f3ef406afb0 0x7f3ef406d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T14:01:46.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.335+0000 7f3f0b7fe640 1 -- 172.21.15.57:0/2626132752 <== mon.2 v2:172.21.15.179:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3f0c0775f0 con 0x7f3f1415b2a0 2023-10-19T14:01:46.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.516+0000 7f3f1925d640 1 -- 172.21.15.57:0/2626132752 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f3f140690e0 con 0x7f3f1415b2a0 2023-10-19T14:01:46.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.520+0000 7f3f0b7fe640 1 -- 172.21.15.57:0/2626132752 <== mon.2 v2:172.21.15.179:3301/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v64)=0 v64) v1 ==== 138+0+0 (secure 0 0 0) 0x7f3f0c0aae60 con 0x7f3f1415b2a0 2023-10-19T14:01:46.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.522+0000 7f3f1925d640 1 -- 172.21.15.57:0/2626132752 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f3ef406afb0 msgr2=0x7f3ef406d470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T14:01:46.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.522+0000 7f3f1925d640 1 --2- 172.21.15.57:0/2626132752 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f3ef406afb0 0x7f3ef406d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:46.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.522+0000 7f3f1925d640 1 -- 172.21.15.57:0/2626132752 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3f1415b2a0 msgr2=0x7f3f141e5880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:46.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.522+0000 7f3f1925d640 1 --2- 172.21.15.57:0/2626132752 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3f1415b2a0 0x7f3f141e5880 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f3f14069e00 tx=0x7f3f0c0042f0 comp rx=0 tx=0).stop 2023-10-19T14:01:46.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.522+0000 7f3f1925d640 1 -- 172.21.15.57:0/2626132752 shutdown_connections 2023-10-19T14:01:46.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.522+0000 7f3f1925d640 1 --2- 172.21.15.57:0/2626132752 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f3ef406afb0 0x7f3ef406d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:46.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.522+0000 7f3f1925d640 1 --2- 172.21.15.57:0/2626132752 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3f1415b2a0 0x7f3f141e5880 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:46.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.522+0000 7f3f1925d640 1 --2- 172.21.15.57:0/2626132752 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3f140cf340 0x7f3f14151420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:46.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.522+0000 7f3f1925d640 1 --2- 172.21.15.57:0/2626132752 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3f140ceb40 0x7f3f141f53d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:46.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.522+0000 7f3f1925d640 1 -- 172.21.15.57:0/2626132752 >> 172.21.15.57:0/2626132752 conn(0x7f3f140d0d60 msgr2=0x7f3f14159ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:46.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.523+0000 7f3f1925d640 1 -- 172.21.15.57:0/2626132752 shutdown_connections 2023-10-19T14:01:46.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.523+0000 7f3f1925d640 1 -- 172.21.15.57:0/2626132752 wait complete. 2023-10-19T14:01:46.536 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-19T14:01:46.581 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:46.579+0000 7f77ed445040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-19T14:01:46.600 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:01:46.597+0000 7fd5fa83e040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-19T14:01:46.626 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:46.624+0000 7f77ed445040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-19T14:01:46.670 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:01:46.646+0000 7fd5fa83e040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-19T14:01:46.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.647+0000 7fbee57b0640 1 -- 172.21.15.57:0/671276675 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbee00cf340 msgr2=0x7fbee00d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:46.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.647+0000 7fbee57b0640 1 --2- 172.21.15.57:0/671276675 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbee00cf340 0x7fbee00d1910 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fbed4009830 tx=0x7fbed402bc50 comp rx=0 tx=0).stop 2023-10-19T14:01:46.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.647+0000 7fbee57b0640 1 -- 172.21.15.57:0/671276675 shutdown_connections 2023-10-19T14:01:46.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.647+0000 7fbee57b0640 1 --2- 172.21.15.57:0/671276675 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbee00cf340 0x7fbee00d1910 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:46.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.647+0000 7fbee57b0640 1 -- 172.21.15.57:0/671276675 >> 172.21.15.57:0/671276675 conn(0x7fbee00d0d60 msgr2=0x7fbee00d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:46.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.647+0000 7fbee57b0640 1 -- 172.21.15.57:0/671276675 shutdown_connections 2023-10-19T14:01:46.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.647+0000 7fbee57b0640 1 -- 172.21.15.57:0/671276675 wait complete. 2023-10-19T14:01:46.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.648+0000 7fbee57b0640 1 Processor -- start 2023-10-19T14:01:46.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.649+0000 7fbee57b0640 1 -- start start 2023-10-19T14:01:46.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.649+0000 7fbee57b0640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbee00ceb40 0x7fbee0150480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:46.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.649+0000 7fbee57b0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbee00cf340 0x7fbee006aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:46.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.649+0000 7fbee57b0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbee015b2a0 0x7fbee006b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:46.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.649+0000 7fbee57b0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbee01f4a30 con 0x7fbee00cf340 2023-10-19T14:01:46.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.649+0000 7fbee57b0640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbee01f4c40 con 0x7fbee015b2a0 2023-10-19T14:01:46.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.649+0000 7fbee57b0640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbee01f4f20 con 0x7fbee00ceb40 2023-10-19T14:01:46.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.649+0000 7fbede7fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbee00cf340 0x7fbee006aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:46.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.649+0000 7fbede7fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbee00cf340 0x7fbee006aea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59860/0 (socket says 172.21.15.57:59860) 2023-10-19T14:01:46.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.649+0000 7fbede7fc640 1 -- 172.21.15.57:0/1635853566 learned_addr learned my addr 172.21.15.57:0/1635853566 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:01:46.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.649+0000 7fbedf7fe640 1 --2- 172.21.15.57:0/1635853566 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbee015b2a0 0x7fbee006b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:46.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.649+0000 7fbedeffd640 1 --2- 172.21.15.57:0/1635853566 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbee00ceb40 0x7fbee0150480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:46.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.650+0000 7fbede7fc640 1 -- 172.21.15.57:0/1635853566 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbee00ceb40 msgr2=0x7fbee0150480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:46.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.650+0000 7fbede7fc640 1 --2- 172.21.15.57:0/1635853566 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbee00ceb40 0x7fbee0150480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:46.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.650+0000 7fbede7fc640 1 -- 172.21.15.57:0/1635853566 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbee015b2a0 msgr2=0x7fbee006b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:46.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.650+0000 7fbede7fc640 1 --2- 172.21.15.57:0/1635853566 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbee015b2a0 0x7fbee006b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:46.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.650+0000 7fbede7fc640 1 -- 172.21.15.57:0/1635853566 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbed4009490 con 0x7fbee00cf340 2023-10-19T14:01:46.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.650+0000 7fbede7fc640 1 --2- 172.21.15.57:0/1635853566 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbee00cf340 0x7fbee006aea0 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 crypto rx=0x7fbed000c6d0 tx=0x7fbed000cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:46.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.650+0000 7fbec77fe640 1 -- 172.21.15.57:0/1635853566 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbed000a840 con 0x7fbee00cf340 2023-10-19T14:01:46.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.650+0000 7fbee57b0640 1 -- 172.21.15.57:0/1635853566 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbee01f5200 con 0x7fbee00cf340 2023-10-19T14:01:46.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.650+0000 7fbec77fe640 1 -- 172.21.15.57:0/1635853566 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fbed000aa90 con 0x7fbee00cf340 2023-10-19T14:01:46.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.650+0000 7fbec77fe640 1 -- 172.21.15.57:0/1635853566 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbed000aea0 con 0x7fbee00cf340 2023-10-19T14:01:46.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.650+0000 7fbee57b0640 1 -- 172.21.15.57:0/1635853566 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbee01f57c0 con 0x7fbee00cf340 2023-10-19T14:01:46.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.651+0000 7fbee57b0640 1 -- 172.21.15.57:0/1635853566 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbee01f5980 con 0x7fbee00cf340 2023-10-19T14:01:46.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.652+0000 7fbec77fe640 1 -- 172.21.15.57:0/1635853566 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 92) v1 ==== 89348+0+0 (secure 0 0 0) 0x7fbed001c460 con 0x7fbee00cf340 2023-10-19T14:01:46.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.652+0000 7fbec77fe640 1 --2- 172.21.15.57:0/1635853566 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7fbecc06afb0 0x7fbecc06d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:46.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.652+0000 7fbec77fe640 1 -- 172.21.15.57:0/1635853566 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fbed0089910 con 0x7fbee00cf340 2023-10-19T14:01:46.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.652+0000 7fbedeffd640 1 -- 172.21.15.57:0/1635853566 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7fbecc06afb0 msgr2=0x7fbecc06d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/2037224568 2023-10-19T14:01:46.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.652+0000 7fbedeffd640 1 --2- 172.21.15.57:0/1635853566 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7fbecc06afb0 0x7fbecc06d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T14:01:46.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.656+0000 7fbec77fe640 1 -- 172.21.15.57:0/1635853566 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fbed0054810 con 0x7fbee00cf340 2023-10-19T14:01:46.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.852+0000 7fbee57b0640 1 -- 172.21.15.57:0/1635853566 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7fbee00690e0 con 0x7fbee00cf340 2023-10-19T14:01:46.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.853+0000 7fbedeffd640 1 -- 172.21.15.57:0/1635853566 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7fbecc06afb0 msgr2=0x7fbecc06d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/2037224568 2023-10-19T14:01:46.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.853+0000 7fbedeffd640 1 --2- 172.21.15.57:0/1635853566 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7fbecc06afb0 0x7fbecc06d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-19T14:01:46.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.856+0000 7fbec77fe640 1 -- 172.21.15.57:0/1635853566 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v65)=0 v65) v1 ==== 133+0+0 (secure 0 0 0) 0x7fbed0057e30 con 0x7fbee00cf340 2023-10-19T14:01:46.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.858+0000 7fbee57b0640 1 -- 172.21.15.57:0/1635853566 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7fbecc06afb0 msgr2=0x7fbecc06d470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T14:01:46.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.858+0000 7fbee57b0640 1 --2- 172.21.15.57:0/1635853566 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7fbecc06afb0 0x7fbecc06d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:46.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.858+0000 7fbee57b0640 1 -- 172.21.15.57:0/1635853566 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbee00cf340 msgr2=0x7fbee006aea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:46.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.858+0000 7fbee57b0640 1 --2- 172.21.15.57:0/1635853566 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbee00cf340 0x7fbee006aea0 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 crypto rx=0x7fbed000c6d0 tx=0x7fbed000cb90 comp rx=0 tx=0).stop 2023-10-19T14:01:46.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.858+0000 7fbee57b0640 1 -- 172.21.15.57:0/1635853566 shutdown_connections 2023-10-19T14:01:46.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.858+0000 7fbee57b0640 1 --2- 172.21.15.57:0/1635853566 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7fbecc06afb0 0x7fbecc06d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:46.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.858+0000 7fbee57b0640 1 --2- 172.21.15.57:0/1635853566 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbee015b2a0 0x7fbee006b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:46.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.858+0000 7fbee57b0640 1 --2- 172.21.15.57:0/1635853566 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbee00cf340 0x7fbee006aea0 unknown :-1 s=CLOSED pgs=1938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:46.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.858+0000 7fbee57b0640 1 --2- 172.21.15.57:0/1635853566 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbee00ceb40 0x7fbee0150480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:46.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.858+0000 7fbee57b0640 1 -- 172.21.15.57:0/1635853566 >> 172.21.15.57:0/1635853566 conn(0x7fbee00d0d60 msgr2=0x7fbee00b2850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:46.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.858+0000 7fbee57b0640 1 -- 172.21.15.57:0/1635853566 shutdown_connections 2023-10-19T14:01:46.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:46.858+0000 7fbee57b0640 1 -- 172.21.15.57:0/1635853566 wait complete. 2023-10-19T14:01:46.871 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:01:46.872 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-10-19T14:01:46.892 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 1h 2023-10-19T14:01:47.015 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:47.013+0000 7f77ed445040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-19T14:01:47.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:47.043+0000 7f986bada640 1 -- 172.21.15.57:0/3229986315 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f98640cddc0 msgr2=0x7f98640ae820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:47.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:47.043+0000 7f986bada640 1 --2- 172.21.15.57:0/3229986315 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f98640cddc0 0x7f98640ae820 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 crypto rx=0x7f9864089890 tx=0x7f985c02c190 comp rx=0 tx=0).stop 2023-10-19T14:01:47.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:47.044+0000 7f986bada640 1 -- 172.21.15.57:0/3229986315 shutdown_connections 2023-10-19T14:01:47.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:47.044+0000 7f986bada640 1 --2- 172.21.15.57:0/3229986315 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f98640cddc0 0x7f98640ae820 unknown :-1 s=CLOSED pgs=1940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:47.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:47.044+0000 7f986bada640 1 --2- 172.21.15.57:0/3229986315 >> v2:172.21.15.179:3300/0 conn(0x7f986415b180 0x7f98640cf350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:47.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:47.044+0000 7f986bada640 1 -- 172.21.15.57:0/3229986315 >> 172.21.15.57:0/3229986315 conn(0x7f9864099ff0 msgr2=0x7f98640d0d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:47.049 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:47.044+0000 7f986bada640 1 -- 172.21.15.57:0/3229986315 shutdown_connections 2023-10-19T14:01:47.049 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:47.044+0000 7f986bada640 1 -- 172.21.15.57:0/3229986315 wait complete. 2023-10-19T14:01:47.049 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:47.045+0000 7f986bada640 1 Processor -- start 2023-10-19T14:01:47.049 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:47.045+0000 7f986bada640 1 -- start start 2023-10-19T14:01:47.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:47.045+0000 7f986bada640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f98640cddc0 0x7f98640b2320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:47.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:47.045+0000 7f986bada640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f98640ce420 0x7f98640c6b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:47.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:47.045+0000 7f986bada640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f986415b180 0x7f98640c70d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:47.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:47.045+0000 7f986bada640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98640c7610 con 0x7f98640cddc0 2023-10-19T14:01:47.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:47.046+0000 7f986bada640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98640c78e0 con 0x7f98640ce420 2023-10-19T14:01:47.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:47.046+0000 7f986bada640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98641e3880 con 0x7f986415b180 2023-10-19T14:01:47.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:47.046+0000 7f9869876640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f98640cddc0 0x7f98640b2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:47.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:47.046+0000 7f9869876640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f98640cddc0 0x7f98640b2320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59880/0 (socket says 172.21.15.57:59880) 2023-10-19T14:01:47.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:47.046+0000 7f9869876640 1 -- 172.21.15.57:0/400034353 learned_addr learned my addr 172.21.15.57:0/400034353 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:01:47.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:47.046+0000 7f9869075640 1 --2- 172.21.15.57:0/400034353 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f98640ce420 0x7f98640c6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:47.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:47.046+0000 7f986a077640 1 --2- 172.21.15.57:0/400034353 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f986415b180 0x7f98640c70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:47.054 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:47.046+0000 7f9869876640 1 -- 172.21.15.57:0/400034353 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f986415b180 msgr2=0x7f98640c70d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:47.054 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:47.046+0000 7f9869876640 1 --2- 172.21.15.57:0/400034353 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f986415b180 0x7f98640c70d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:47.054 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:47.046+0000 7f9869876640 1 -- 172.21.15.57:0/400034353 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f98640ce420 msgr2=0x7f98640c6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:47.054 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:47.046+0000 7f9869876640 1 --2- 172.21.15.57:0/400034353 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f98640ce420 0x7f98640c6b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:47.054 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:47.046+0000 7f9869876640 1 -- 172.21.15.57:0/400034353 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f985c009840 con 0x7f98640cddc0 2023-10-19T14:01:47.055 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:47.046+0000 7f9869876640 1 --2- 172.21.15.57:0/400034353 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f98640cddc0 0x7f98640b2320 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 crypto rx=0x7f986000d740 tx=0x7f986000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:47.055 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:47.046+0000 7f98567fc640 1 -- 172.21.15.57:0/400034353 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9860009c30 con 0x7f98640cddc0 2023-10-19T14:01:47.055 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:47.046+0000 7f98567fc640 1 -- 172.21.15.57:0/400034353 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f9860009e80 con 0x7f98640cddc0 2023-10-19T14:01:47.055 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:47.046+0000 7f98567fc640 1 -- 172.21.15.57:0/400034353 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98600106c0 con 0x7f98640cddc0 2023-10-19T14:01:47.055 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:47.046+0000 7f986bada640 1 -- 172.21.15.57:0/400034353 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98641e3b60 con 0x7f98640cddc0 2023-10-19T14:01:47.056 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:47.046+0000 7f986bada640 1 -- 172.21.15.57:0/400034353 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98641e4120 con 0x7f98640cddc0 2023-10-19T14:01:47.056 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:47.047+0000 7f986bada640 1 -- 172.21.15.57:0/400034353 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9864066960 con 0x7f98640cddc0 2023-10-19T14:01:47.056 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:47.049+0000 7f98567fc640 1 -- 172.21.15.57:0/400034353 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 92) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f9860010820 con 0x7f98640cddc0 2023-10-19T14:01:47.056 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:47.049+0000 7f98567fc640 1 --2- 172.21.15.57:0/400034353 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f984806afb0 0x7f984806d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:47.056 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:47.049+0000 7f9869075640 1 -- 172.21.15.57:0/400034353 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f984806afb0 msgr2=0x7f984806d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/2037224568 2023-10-19T14:01:47.056 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:47.049+0000 7f98567fc640 1 -- 172.21.15.57:0/400034353 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f9860089ee0 con 0x7f98640cddc0 2023-10-19T14:01:47.057 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:47.049+0000 7f9869075640 1 --2- 172.21.15.57:0/400034353 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f984806afb0 0x7f984806d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T14:01:47.057 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:47.054+0000 7f98567fc640 1 -- 172.21.15.57:0/400034353 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9860054cc0 con 0x7f98640cddc0 2023-10-19T14:01:47.060 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:47.058+0000 7f77ed445040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-19T14:01:47.086 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:01:47.083+0000 7fd5fa83e040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-19T14:01:47.105 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:47.103+0000 7f77ed445040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-19T14:01:47.132 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:01:47.130+0000 7fd5fa83e040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-19T14:01:47.157 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:47.155+0000 7f77ed445040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-19T14:01:47.177 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:01:47.175+0000 7fd5fa83e040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-19T14:01:47.198 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:47.196+0000 7f77ed445040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-19T14:01:47.233 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:01:47.230+0000 7fd5fa83e040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-19T14:01:47.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:47.250+0000 7f9869075640 1 -- 172.21.15.57:0/400034353 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f984806afb0 msgr2=0x7f984806d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/2037224568 2023-10-19T14:01:47.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:47.250+0000 7f9869075640 1 --2- 172.21.15.57:0/400034353 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f984806afb0 0x7f984806d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-19T14:01:47.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:47.254+0000 7f986bada640 1 -- 172.21.15.57:0/400034353 --> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1h", "target": ["mon-mgr", ""]}) v1 -- 0x7f98640cfa00 con 0x7f984806afb0 2023-10-19T14:01:47.276 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:01:47.274+0000 7fd5fa83e040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-19T14:01:47.296 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:47.294+0000 7f77ed445040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-19T14:01:47.349 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:47.347+0000 7f77ed445040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-19T14:01:47.379 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:01:47.376+0000 7fd5fa83e040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-19T14:01:47.390 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:47.388+0000 7f77ed445040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-19T14:01:47.435 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:01:47.432+0000 7fd5fa83e040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-19T14:01:47.437 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:47.435+0000 7f77ed445040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-19T14:01:47.479 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:01:47.476+0000 7fd5fa83e040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-19T14:01:47.480 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:47.478+0000 7f77ed445040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-19T14:01:47.529 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:01:47.526+0000 7fd5fa83e040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-19T14:01:47.574 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:01:47.571+0000 7fd5fa83e040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-19T14:01:47.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:47.651+0000 7f9869075640 1 -- 172.21.15.57:0/400034353 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f984806afb0 msgr2=0x7f984806d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/2037224568 2023-10-19T14:01:47.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:47.651+0000 7f9869075640 1 --2- 172.21.15.57:0/400034353 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f984806afb0 0x7f984806d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-19T14:01:47.795 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:47.793+0000 7f77ed445040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-19T14:01:47.907 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:01:47.904+0000 7fd5fa83e040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-19T14:01:47.924 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:47.921+0000 7f77ed445040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-19T14:01:48.014 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:48.012+0000 7f77ed445040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-19T14:01:48.045 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:01:48.043+0000 7fd5fa83e040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-19T14:01:48.057 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:48.054+0000 7f77ed445040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-19T14:01:48.142 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:01:48.139+0000 7fd5fa83e040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-19T14:01:48.151 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:48.149+0000 7f77ed445040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-19T14:01:48.186 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:01:48.184+0000 7fd5fa83e040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-19T14:01:48.242 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:48.240+0000 7f77ed445040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-19T14:01:48.287 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:01:48.284+0000 7fd5fa83e040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-19T14:01:48.380 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:48.377+0000 7f77ed445040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-19T14:01:48.381 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:01:48.379+0000 7fd5fa83e040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-19T14:01:48.421 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:48.418+0000 7f77ed445040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-19T14:01:48.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:48.452+0000 7f9869075640 1 -- 172.21.15.57:0/400034353 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f984806afb0 msgr2=0x7f984806d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/2037224568 2023-10-19T14:01:48.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:48.452+0000 7f9869075640 1 --2- 172.21.15.57:0/400034353 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f984806afb0 0x7f984806d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-19T14:01:48.525 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:01:48.522+0000 7fd5fa83e040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-19T14:01:48.525 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:48.523+0000 7f77ed445040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-19T14:01:48.568 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:01:48.566+0000 7fd5fa83e040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-19T14:01:48.678 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:01:48.676+0000 7fd5fa83e040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-19T14:01:49.144 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:49.141+0000 7f77ed445040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-19T14:01:49.327 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:01:49.325+0000 7fd5fa83e040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-19T14:01:49.404 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:49.401+0000 7f77ed445040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-19T14:01:49.452 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:49.449+0000 7f77ed445040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-19T14:01:49.552 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:49.550+0000 7f77ed445040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-19T14:01:49.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:49.573+0000 7f98567fc640 1 -- 172.21.15.57:0/400034353 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mgrmap(e 93) v1 ==== 88957+0+0 (secure 0 0 0) 0x7f98600882a0 con 0x7f98640cddc0 2023-10-19T14:01:49.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:49.573+0000 7f98567fc640 1 -- 172.21.15.57:0/400034353 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f984806afb0 msgr2=0x7f984806d470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T14:01:49.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:49.573+0000 7f98567fc640 1 --2- 172.21.15.57:0/400034353 >> [v2:172.21.15.179:6832/2037224568,v1:172.21.15.179:6833/2037224568] conn(0x7f984806afb0 0x7f984806d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:49.604 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:01:49.602+0000 7fd5fa83e040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-19T14:01:49.659 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:01:49.656+0000 7fd5fa83e040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-19T14:01:49.677 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:49.674+0000 7f776cf75640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:01:49.678 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:49.674+0000 7f776cf75640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:01:49.678 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:49.674+0000 7f776cf75640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:01:49.679 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:49.674+0000 7f776cf75640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:01:49.679 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:49.674+0000 7f776cf75640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:01:49.679 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:49.675+0000 7f7769f2f640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:01:49.680 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:49.675+0000 7f7769f2f640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:01:49.680 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:49.675+0000 7f7769f2f640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:01:49.680 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:49.675+0000 7f7769f2f640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:01:49.680 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:49.675+0000 7f7769f2f640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:01:49.765 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:01:49.763+0000 7fd5fa83e040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-19T14:01:50.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:50.573+0000 7f98567fc640 1 -- 172.21.15.57:0/400034353 <== mon.0 v2:172.21.15.57:3300/0 8 ==== mgrmap(e 94) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f986008dd70 con 0x7f98640cddc0 2023-10-19T14:01:50.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:50.574+0000 7f98567fc640 1 --2- 172.21.15.57:0/400034353 >> [v2:172.21.15.179:6832/3795018457,v1:172.21.15.179:6833/3795018457] conn(0x7f98480358b0 0x7f98480753e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:50.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:50.574+0000 7f98567fc640 1 -- 172.21.15.57:0/400034353 --> [v2:172.21.15.179:6832/3795018457,v1:172.21.15.179:6833/3795018457] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1h", "target": ["mon-mgr", ""]}) v1 -- 0x7f98640cfa00 con 0x7f98480358b0 2023-10-19T14:01:50.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:50.575+0000 7f9869075640 1 --2- 172.21.15.57:0/400034353 >> [v2:172.21.15.179:6832/3795018457,v1:172.21.15.179:6833/3795018457] conn(0x7f98480358b0 0x7f98480753e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:50.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:50.580+0000 7f9869075640 1 --2- 172.21.15.57:0/400034353 >> [v2:172.21.15.179:6832/3795018457,v1:172.21.15.179:6833/3795018457] conn(0x7f98480358b0 0x7f98480753e0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f9864159710 tx=0x7f9858007e50 comp rx=0 tx=0).ready entity=mgr.8243 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:50.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:50.637+0000 7f98567fc640 1 -- 172.21.15.57:0/400034353 <== mgr.8243 v2:172.21.15.179:6832/3795018457 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f98640cfa00 con 0x7f98480358b0 2023-10-19T14:01:50.640 INFO:teuthology.orchestra.run.smithi057.stdout:Schedule set for path snap_test_dir1 2023-10-19T14:01:50.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:50.639+0000 7f986bada640 1 -- 172.21.15.57:0/400034353 >> [v2:172.21.15.179:6832/3795018457,v1:172.21.15.179:6833/3795018457] conn(0x7f98480358b0 msgr2=0x7f98480753e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:50.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:50.639+0000 7f986bada640 1 --2- 172.21.15.57:0/400034353 >> [v2:172.21.15.179:6832/3795018457,v1:172.21.15.179:6833/3795018457] conn(0x7f98480358b0 0x7f98480753e0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f9864159710 tx=0x7f9858007e50 comp rx=0 tx=0).stop 2023-10-19T14:01:50.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:50.639+0000 7f986bada640 1 -- 172.21.15.57:0/400034353 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f98640cddc0 msgr2=0x7f98640b2320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:50.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:50.639+0000 7f986bada640 1 --2- 172.21.15.57:0/400034353 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f98640cddc0 0x7f98640b2320 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 crypto rx=0x7f986000d740 tx=0x7f986000dc00 comp rx=0 tx=0).stop 2023-10-19T14:01:50.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:50.639+0000 7f986a077640 1 -- 172.21.15.57:0/400034353 reap_dead start 2023-10-19T14:01:50.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:50.639+0000 7f986bada640 1 -- 172.21.15.57:0/400034353 shutdown_connections 2023-10-19T14:01:50.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:50.640+0000 7f986bada640 1 -- 172.21.15.57:0/400034353 >> 172.21.15.57:0/400034353 conn(0x7f9864099ff0 msgr2=0x7f986415c400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:50.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:50.640+0000 7f986bada640 1 -- 172.21.15.57:0/400034353 shutdown_connections 2023-10-19T14:01:50.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:50.640+0000 7f986bada640 1 -- 172.21.15.57:0/400034353 wait complete. 2023-10-19T14:01:50.655 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2023-10-19T14:01:50.655 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove snap_test_dir1 2023-10-19T14:01:50.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:50.767+0000 7fd8404f7640 1 -- 172.21.15.57:0/3387406538 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd8380ceb40 msgr2=0x7fd8380b0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:50.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:50.767+0000 7fd8404f7640 1 --2- 172.21.15.57:0/3387406538 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd8380ceb40 0x7fd8380b0650 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 crypto rx=0x7fd83808a120 tx=0x7fd83002bcf0 comp rx=0 tx=0).stop 2023-10-19T14:01:50.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:50.767+0000 7fd8404f7640 1 -- 172.21.15.57:0/3387406538 shutdown_connections 2023-10-19T14:01:50.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:50.767+0000 7fd8404f7640 1 --2- 172.21.15.57:0/3387406538 >> v2:172.21.15.179:3300/0 conn(0x7fd83815b2a0 0x7fd8380d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:50.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:50.767+0000 7fd8404f7640 1 --2- 172.21.15.57:0/3387406538 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd8380ceb40 0x7fd8380b0650 unknown :-1 s=CLOSED pgs=1944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:50.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:50.767+0000 7fd8404f7640 1 -- 172.21.15.57:0/3387406538 >> 172.21.15.57:0/3387406538 conn(0x7fd8380d0d60 msgr2=0x7fd8380d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:50.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:50.767+0000 7fd8404f7640 1 -- 172.21.15.57:0/3387406538 shutdown_connections 2023-10-19T14:01:50.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:50.767+0000 7fd8404f7640 1 -- 172.21.15.57:0/3387406538 wait complete. 2023-10-19T14:01:50.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:50.768+0000 7fd8404f7640 1 Processor -- start 2023-10-19T14:01:50.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:50.768+0000 7fd8404f7640 1 -- start start 2023-10-19T14:01:50.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:50.769+0000 7fd8404f7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd8380ceb40 0x7fd838154270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:50.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:50.769+0000 7fd8404f7640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd8380cf340 0x7fd8380c6990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:50.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:50.769+0000 7fd8404f7640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd83815b2a0 0x7fd83815ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:50.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:50.769+0000 7fd8404f7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8380b4fe0 con 0x7fd8380ceb40 2023-10-19T14:01:50.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:50.769+0000 7fd8404f7640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8380c94b0 con 0x7fd8380cf340 2023-10-19T14:01:50.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:50.769+0000 7fd8404f7640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8380b4740 con 0x7fd83815b2a0 2023-10-19T14:01:50.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:50.769+0000 7fd83e293640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd8380ceb40 0x7fd838154270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:50.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:50.769+0000 7fd83e293640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd8380ceb40 0x7fd838154270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59896/0 (socket says 172.21.15.57:59896) 2023-10-19T14:01:50.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:50.769+0000 7fd83e293640 1 -- 172.21.15.57:0/2386093283 learned_addr learned my addr 172.21.15.57:0/2386093283 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:01:50.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:50.769+0000 7fd83da92640 1 --2- 172.21.15.57:0/2386093283 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd8380cf340 0x7fd8380c6990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:50.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:50.769+0000 7fd83ea94640 1 --2- 172.21.15.57:0/2386093283 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd83815b2a0 0x7fd83815ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:50.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:50.769+0000 7fd83e293640 1 -- 172.21.15.57:0/2386093283 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd83815b2a0 msgr2=0x7fd83815ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:50.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:50.769+0000 7fd83e293640 1 --2- 172.21.15.57:0/2386093283 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd83815b2a0 0x7fd83815ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:50.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:50.769+0000 7fd83e293640 1 -- 172.21.15.57:0/2386093283 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd8380cf340 msgr2=0x7fd8380c6990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:50.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:50.769+0000 7fd83e293640 1 --2- 172.21.15.57:0/2386093283 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd8380cf340 0x7fd8380c6990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:50.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:50.770+0000 7fd83e293640 1 -- 172.21.15.57:0/2386093283 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd830009490 con 0x7fd8380ceb40 2023-10-19T14:01:50.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:50.770+0000 7fd83da92640 1 --2- 172.21.15.57:0/2386093283 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd8380cf340 0x7fd8380c6990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:01:50.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:50.770+0000 7fd83e293640 1 --2- 172.21.15.57:0/2386093283 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd8380ceb40 0x7fd838154270 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7fd82800cbc0 tx=0x7fd82800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:50.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:50.770+0000 7fd836ffd640 1 -- 172.21.15.57:0/2386093283 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd82800ace0 con 0x7fd8380ceb40 2023-10-19T14:01:50.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:50.770+0000 7fd836ffd640 1 -- 172.21.15.57:0/2386093283 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fd828003970 con 0x7fd8380ceb40 2023-10-19T14:01:50.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:50.770+0000 7fd836ffd640 1 -- 172.21.15.57:0/2386093283 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd828003d60 con 0x7fd8380ceb40 2023-10-19T14:01:50.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:50.770+0000 7fd8404f7640 1 -- 172.21.15.57:0/2386093283 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8380b4a20 con 0x7fd8380ceb40 2023-10-19T14:01:50.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:50.770+0000 7fd8404f7640 1 -- 172.21.15.57:0/2386093283 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8381547b0 con 0x7fd8380ceb40 2023-10-19T14:01:50.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:50.771+0000 7fd8404f7640 1 -- 172.21.15.57:0/2386093283 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8380c9280 con 0x7fd8380ceb40 2023-10-19T14:01:50.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:50.771+0000 7fd836ffd640 1 -- 172.21.15.57:0/2386093283 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 94) v1 ==== 89084+0+0 (secure 0 0 0) 0x7fd828004030 con 0x7fd8380ceb40 2023-10-19T14:01:50.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:50.772+0000 7fd836ffd640 1 --2- 172.21.15.57:0/2386093283 >> [v2:172.21.15.179:6832/3795018457,v1:172.21.15.179:6833/3795018457] conn(0x7fd82406ace0 0x7fd82406d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:50.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:50.772+0000 7fd836ffd640 1 -- 172.21.15.57:0/2386093283 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 6401+0+0 (secure 0 0 0) 0x7fd82808a160 con 0x7fd8380ceb40 2023-10-19T14:01:50.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:50.772+0000 7fd83da92640 1 --2- 172.21.15.57:0/2386093283 >> [v2:172.21.15.179:6832/3795018457,v1:172.21.15.179:6833/3795018457] conn(0x7fd82406ace0 0x7fd82406d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:50.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:50.772+0000 7fd83da92640 1 --2- 172.21.15.57:0/2386093283 >> [v2:172.21.15.179:6832/3795018457,v1:172.21.15.179:6833/3795018457] conn(0x7fd82406ace0 0x7fd82406d1a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fd82c005fd0 tx=0x7fd82c00f360 comp rx=0 tx=0).ready entity=mgr.8243 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:50.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:50.776+0000 7fd836ffd640 1 -- 172.21.15.57:0/2386093283 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd8380c9280 con 0x7fd8380ceb40 2023-10-19T14:01:50.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:50.977+0000 7fd8404f7640 1 -- 172.21.15.57:0/2386093283 --> [v2:172.21.15.179:6832/3795018457,v1:172.21.15.179:6833/3795018457] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7fd8380cc170 con 0x7fd82406ace0 2023-10-19T14:01:51.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.032+0000 7fd836ffd640 1 -- 172.21.15.57:0/2386093283 <== mgr.8243 v2:172.21.15.179:6832/3795018457 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fd82c00b890 con 0x7fd82406ace0 2023-10-19T14:01:51.035 INFO:teuthology.orchestra.run.smithi057.stdout:Schedule removed for path snap_test_dir1 2023-10-19T14:01:51.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.034+0000 7fd8404f7640 1 -- 172.21.15.57:0/2386093283 >> [v2:172.21.15.179:6832/3795018457,v1:172.21.15.179:6833/3795018457] conn(0x7fd82406ace0 msgr2=0x7fd82406d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:51.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.034+0000 7fd8404f7640 1 --2- 172.21.15.57:0/2386093283 >> [v2:172.21.15.179:6832/3795018457,v1:172.21.15.179:6833/3795018457] conn(0x7fd82406ace0 0x7fd82406d1a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fd82c005fd0 tx=0x7fd82c00f360 comp rx=0 tx=0).stop 2023-10-19T14:01:51.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.034+0000 7fd8404f7640 1 -- 172.21.15.57:0/2386093283 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd8380ceb40 msgr2=0x7fd838154270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:51.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.034+0000 7fd8404f7640 1 --2- 172.21.15.57:0/2386093283 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd8380ceb40 0x7fd838154270 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7fd82800cbc0 tx=0x7fd82800a430 comp rx=0 tx=0).stop 2023-10-19T14:01:51.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.034+0000 7fd8404f7640 1 -- 172.21.15.57:0/2386093283 shutdown_connections 2023-10-19T14:01:51.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.034+0000 7fd8404f7640 1 --2- 172.21.15.57:0/2386093283 >> [v2:172.21.15.179:6832/3795018457,v1:172.21.15.179:6833/3795018457] conn(0x7fd82406ace0 0x7fd82406d1a0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:51.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.034+0000 7fd8404f7640 1 --2- 172.21.15.57:0/2386093283 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd83815b2a0 0x7fd83815ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:51.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.034+0000 7fd8404f7640 1 --2- 172.21.15.57:0/2386093283 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd8380cf340 0x7fd8380c6990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:51.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.034+0000 7fd8404f7640 1 --2- 172.21.15.57:0/2386093283 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd8380ceb40 0x7fd838154270 unknown :-1 s=CLOSED pgs=1945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:51.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.034+0000 7fd8404f7640 1 -- 172.21.15.57:0/2386093283 >> 172.21.15.57:0/2386093283 conn(0x7fd8380d0d60 msgr2=0x7fd8380b0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:51.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.034+0000 7fd8404f7640 1 -- 172.21.15.57:0/2386093283 shutdown_connections 2023-10-19T14:01:51.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.034+0000 7fd8404f7640 1 -- 172.21.15.57:0/2386093283 wait complete. 2023-10-19T14:01:51.047 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2023-10-19T14:01:51.048 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list snap_test_dir1 2023-10-19T14:01:51.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.157+0000 7ff1641b6640 1 -- 172.21.15.57:0/2545623859 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff15c06ac70 msgr2=0x7ff15c158060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:51.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.157+0000 7ff1641b6640 1 --2- 172.21.15.57:0/2545623859 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff15c06ac70 0x7ff15c158060 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 crypto rx=0x7ff15c071e20 tx=0x7ff14c02bda0 comp rx=0 tx=0).stop 2023-10-19T14:01:51.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.157+0000 7ff1641b6640 1 -- 172.21.15.57:0/2545623859 shutdown_connections 2023-10-19T14:01:51.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.157+0000 7ff1641b6640 1 --2- 172.21.15.57:0/2545623859 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff15c1585a0 0x7ff15c0c9560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:51.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.157+0000 7ff1641b6640 1 --2- 172.21.15.57:0/2545623859 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff15c06ac70 0x7ff15c158060 unknown :-1 s=CLOSED pgs=1946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:51.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.157+0000 7ff1641b6640 1 --2- 172.21.15.57:0/2545623859 >> v2:172.21.15.179:3300/0 conn(0x7ff15c14eba0 0x7ff15c06a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:51.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.157+0000 7ff1641b6640 1 -- 172.21.15.57:0/2545623859 >> 172.21.15.57:0/2545623859 conn(0x7ff15c06fae0 msgr2=0x7ff15c06fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:51.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.157+0000 7ff1641b6640 1 -- 172.21.15.57:0/2545623859 shutdown_connections 2023-10-19T14:01:51.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.157+0000 7ff1641b6640 1 -- 172.21.15.57:0/2545623859 wait complete. 2023-10-19T14:01:51.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.158+0000 7ff1641b6640 1 Processor -- start 2023-10-19T14:01:51.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.158+0000 7ff1641b6640 1 -- start start 2023-10-19T14:01:51.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.159+0000 7ff1641b6640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff15c06ac70 0x7ff15c158fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:51.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.159+0000 7ff1641b6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff15c14eba0 0x7ff15c143bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:51.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.159+0000 7ff1641b6640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff15c1585a0 0x7ff15c06bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:51.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.159+0000 7ff1641b6640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff15c0ae490 con 0x7ff15c14eba0 2023-10-19T14:01:51.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.159+0000 7ff1641b6640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff15c159510 con 0x7ff15c1585a0 2023-10-19T14:01:51.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.159+0000 7ff1641b6640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff15c0adef0 con 0x7ff15c06ac70 2023-10-19T14:01:51.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.159+0000 7ff161751640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff15c14eba0 0x7ff15c143bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:51.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.159+0000 7ff161751640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff15c14eba0 0x7ff15c143bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59910/0 (socket says 172.21.15.57:59910) 2023-10-19T14:01:51.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.159+0000 7ff161751640 1 -- 172.21.15.57:0/4080740490 learned_addr learned my addr 172.21.15.57:0/4080740490 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:01:51.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.159+0000 7ff162753640 1 --2- 172.21.15.57:0/4080740490 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff15c1585a0 0x7ff15c06bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:51.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.159+0000 7ff161f52640 1 --2- 172.21.15.57:0/4080740490 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff15c06ac70 0x7ff15c158fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:51.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.159+0000 7ff161751640 1 -- 172.21.15.57:0/4080740490 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff15c06ac70 msgr2=0x7ff15c158fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:51.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.159+0000 7ff161751640 1 --2- 172.21.15.57:0/4080740490 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff15c06ac70 0x7ff15c158fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:51.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.159+0000 7ff161751640 1 -- 172.21.15.57:0/4080740490 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff15c1585a0 msgr2=0x7ff15c06bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:51.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.159+0000 7ff161751640 1 --2- 172.21.15.57:0/4080740490 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff15c1585a0 0x7ff15c06bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:51.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.159+0000 7ff161751640 1 -- 172.21.15.57:0/4080740490 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff14c009560 con 0x7ff15c14eba0 2023-10-19T14:01:51.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.159+0000 7ff162753640 1 --2- 172.21.15.57:0/4080740490 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff15c1585a0 0x7ff15c06bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:01:51.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.159+0000 7ff161f52640 1 --2- 172.21.15.57:0/4080740490 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff15c06ac70 0x7ff15c158fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:01:51.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.160+0000 7ff161751640 1 --2- 172.21.15.57:0/4080740490 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff15c14eba0 0x7ff15c143bf0 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x7ff14c036040 tx=0x7ff14c02dac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:51.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.160+0000 7ff1527fc640 1 -- 172.21.15.57:0/4080740490 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff14c003800 con 0x7ff15c14eba0 2023-10-19T14:01:51.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.160+0000 7ff1527fc640 1 -- 172.21.15.57:0/4080740490 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7ff14c003a50 con 0x7ff15c14eba0 2023-10-19T14:01:51.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.160+0000 7ff1641b6640 1 -- 172.21.15.57:0/4080740490 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff15c0cb800 con 0x7ff15c14eba0 2023-10-19T14:01:51.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.160+0000 7ff1527fc640 1 -- 172.21.15.57:0/4080740490 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff14c003de0 con 0x7ff15c14eba0 2023-10-19T14:01:51.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.160+0000 7ff1641b6640 1 -- 172.21.15.57:0/4080740490 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff15c0ae1b0 con 0x7ff15c14eba0 2023-10-19T14:01:51.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.161+0000 7ff1641b6640 1 -- 172.21.15.57:0/4080740490 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff15c0690e0 con 0x7ff15c14eba0 2023-10-19T14:01:51.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.161+0000 7ff1527fc640 1 -- 172.21.15.57:0/4080740490 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 94) v1 ==== 89084+0+0 (secure 0 0 0) 0x7ff14c03e450 con 0x7ff15c14eba0 2023-10-19T14:01:51.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.162+0000 7ff1527fc640 1 --2- 172.21.15.57:0/4080740490 >> [v2:172.21.15.179:6832/3795018457,v1:172.21.15.179:6833/3795018457] conn(0x7ff14006ace0 0x7ff14006d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:51.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.162+0000 7ff1527fc640 1 -- 172.21.15.57:0/4080740490 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 6401+0+0 (secure 0 0 0) 0x7ff14c0ac030 con 0x7ff15c14eba0 2023-10-19T14:01:51.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.162+0000 7ff161f52640 1 --2- 172.21.15.57:0/4080740490 >> [v2:172.21.15.179:6832/3795018457,v1:172.21.15.179:6833/3795018457] conn(0x7ff14006ace0 0x7ff14006d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:51.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.162+0000 7ff161f52640 1 --2- 172.21.15.57:0/4080740490 >> [v2:172.21.15.179:6832/3795018457,v1:172.21.15.179:6833/3795018457] conn(0x7ff14006ace0 0x7ff14006d1a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7ff158009ad0 tx=0x7ff158005f90 comp rx=0 tx=0).ready entity=mgr.8243 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:51.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.166+0000 7ff1527fc640 1 -- 172.21.15.57:0/4080740490 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff14c077160 con 0x7ff15c14eba0 2023-10-19T14:01:51.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.360+0000 7ff1641b6640 1 -- 172.21.15.57:0/4080740490 --> [v2:172.21.15.179:6832/3795018457,v1:172.21.15.179:6833/3795018457] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7ff15c158a30 con 0x7ff14006ace0 2023-10-19T14:01:51.368 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:51.366+0000 7f7773f83640 -1 mgr.server reply reply (2) No such file or directory SnapSchedule for snap_test_dir1 not found 2023-10-19T14:01:51.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.366+0000 7ff1527fc640 1 -- 172.21.15.57:0/4080740490 <== mgr.8243 v2:172.21.15.179:6832/3795018457 1 ==== mgr_command_reply(tid 0: -2 SnapSchedule for snap_test_dir1 not found) v1 ==== 49+0+0 (secure 0 0 0) 0x7ff158009740 con 0x7ff14006ace0 2023-10-19T14:01:51.369 INFO:teuthology.orchestra.run.smithi057.stderr:Error ENOENT: SnapSchedule for snap_test_dir1 not found 2023-10-19T14:01:51.373 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-10-19T14:01:51.374 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:01:51.374 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2023-10-19T14:01:51.394 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-19T14:01:51.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.543+0000 7f06c2125640 1 -- 172.21.15.57:0/997328216 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06a4004d00 msgr2=0x7f06a4005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:51.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.543+0000 7f06c2125640 1 --2- 172.21.15.57:0/997328216 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06a4004d00 0x7f06a4005180 secure :-1 s=READY pgs=1949 cs=0 l=1 rev1=1 crypto rx=0x7f06b4001e90 tx=0x7f06a40057f0 comp rx=0 tx=0).stop 2023-10-19T14:01:51.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.543+0000 7f06bbfff640 1 -- 172.21.15.57:0/997328216 reap_dead start 2023-10-19T14:01:51.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.543+0000 7f06c2125640 1 -- 172.21.15.57:0/997328216 shutdown_connections 2023-10-19T14:01:51.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.543+0000 7f06c2125640 1 -- 172.21.15.57:0/997328216 >> 172.21.15.57:0/997328216 conn(0x7f06bc069930 msgr2=0x7f06bc069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:51.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.543+0000 7f06c2125640 1 -- 172.21.15.57:0/997328216 shutdown_connections 2023-10-19T14:01:51.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.543+0000 7f06c2125640 1 -- 172.21.15.57:0/997328216 wait complete. 2023-10-19T14:01:51.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.544+0000 7f06c2125640 1 Processor -- start 2023-10-19T14:01:51.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.545+0000 7f06c2125640 1 -- start start 2023-10-19T14:01:51.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.545+0000 7f06c2125640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f06bc152830 0x7f06bc152c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:51.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.545+0000 7f06c2125640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06bc1ef2d0 0x7f06bc13a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:51.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.545+0000 7f06c2125640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f06bc14f540 0x7f06bc14f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:51.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.545+0000 7f06c2125640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06bc14ff00 con 0x7f06bc1ef2d0 2023-10-19T14:01:51.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.545+0000 7f06c2125640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06bc153150 con 0x7f06bc14f540 2023-10-19T14:01:51.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.545+0000 7f06c2125640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06bc13aea0 con 0x7f06bc152830 2023-10-19T14:01:51.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.545+0000 7f06baffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06bc1ef2d0 0x7f06bc13a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:51.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.545+0000 7f06baffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06bc1ef2d0 0x7f06bc13a570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59938/0 (socket says 172.21.15.57:59938) 2023-10-19T14:01:51.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.545+0000 7f06baffd640 1 -- 172.21.15.57:0/872744200 learned_addr learned my addr 172.21.15.57:0/872744200 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:01:51.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.545+0000 7f06bbfff640 1 --2- 172.21.15.57:0/872744200 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f06bc14f540 0x7f06bc14f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:51.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.545+0000 7f06bb7fe640 1 --2- 172.21.15.57:0/872744200 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f06bc152830 0x7f06bc152c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:51.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.546+0000 7f06baffd640 1 -- 172.21.15.57:0/872744200 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f06bc152830 msgr2=0x7f06bc152c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:51.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.546+0000 7f06baffd640 1 --2- 172.21.15.57:0/872744200 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f06bc152830 0x7f06bc152c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:51.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.546+0000 7f06baffd640 1 -- 172.21.15.57:0/872744200 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f06bc14f540 msgr2=0x7f06bc14f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:51.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.546+0000 7f06baffd640 1 --2- 172.21.15.57:0/872744200 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f06bc14f540 0x7f06bc14f9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:51.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.546+0000 7f06baffd640 1 -- 172.21.15.57:0/872744200 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06bc13b180 con 0x7f06bc1ef2d0 2023-10-19T14:01:51.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.546+0000 7f06bb7fe640 1 --2- 172.21.15.57:0/872744200 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f06bc152830 0x7f06bc152c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:01:51.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.546+0000 7f06baffd640 1 --2- 172.21.15.57:0/872744200 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06bc1ef2d0 0x7f06bc13a570 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 crypto rx=0x7f06b0003670 tx=0x7f06b0003aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:51.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.546+0000 7f06a3fff640 1 -- 172.21.15.57:0/872744200 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06b0011900 con 0x7f06bc1ef2d0 2023-10-19T14:01:51.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.546+0000 7f06a3fff640 1 -- 172.21.15.57:0/872744200 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f06b0011a60 con 0x7f06bc1ef2d0 2023-10-19T14:01:51.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.546+0000 7f06c2125640 1 -- 172.21.15.57:0/872744200 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06bc13b460 con 0x7f06bc1ef2d0 2023-10-19T14:01:51.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.546+0000 7f06a3fff640 1 -- 172.21.15.57:0/872744200 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06b0011e50 con 0x7f06bc1ef2d0 2023-10-19T14:01:51.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.546+0000 7f06c2125640 1 -- 172.21.15.57:0/872744200 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06bc1503d0 con 0x7f06bc1ef2d0 2023-10-19T14:01:51.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.547+0000 7f06c2125640 1 -- 172.21.15.57:0/872744200 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06bc0cc010 con 0x7f06bc1ef2d0 2023-10-19T14:01:51.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.547+0000 7f06a3fff640 1 -- 172.21.15.57:0/872744200 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 94) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f06b000f430 con 0x7f06bc1ef2d0 2023-10-19T14:01:51.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.548+0000 7f06a3fff640 1 --2- 172.21.15.57:0/872744200 >> [v2:172.21.15.179:6832/3795018457,v1:172.21.15.179:6833/3795018457] conn(0x7f06a806ad10 0x7f06a806d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:51.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.548+0000 7f06a3fff640 1 -- 172.21.15.57:0/872744200 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f06b008a350 con 0x7f06bc1ef2d0 2023-10-19T14:01:51.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.548+0000 7f06bb7fe640 1 --2- 172.21.15.57:0/872744200 >> [v2:172.21.15.179:6832/3795018457,v1:172.21.15.179:6833/3795018457] conn(0x7f06a806ad10 0x7f06a806d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:51.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.549+0000 7f06bb7fe640 1 --2- 172.21.15.57:0/872744200 >> [v2:172.21.15.179:6832/3795018457,v1:172.21.15.179:6833/3795018457] conn(0x7f06a806ad10 0x7f06a806d1d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f06ac009c50 tx=0x7f06ac005f90 comp rx=0 tx=0).ready entity=mgr.8243 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:51.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.552+0000 7f06a3fff640 1 -- 172.21.15.57:0/872744200 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f06b00554b0 con 0x7f06bc1ef2d0 2023-10-19T14:01:51.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:51.778+0000 7f06c2125640 1 -- 172.21.15.57:0/872744200 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f06bc0690e0 con 0x7f06bc1ef2d0 2023-10-19T14:01:52.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:52.582+0000 7f06a3fff640 1 -- 172.21.15.57:0/872744200 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v95) v1 ==== 93+0+0 (secure 0 0 0) 0x7f06b0058ad0 con 0x7f06bc1ef2d0 2023-10-19T14:01:52.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:52.585+0000 7f06a3fff640 1 -- 172.21.15.57:0/872744200 <== mon.0 v2:172.21.15.57:3300/0 8 ==== mgrmap(e 95) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f06b0055880 con 0x7f06bc1ef2d0 2023-10-19T14:01:52.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:52.586+0000 7f06c2125640 1 -- 172.21.15.57:0/872744200 >> [v2:172.21.15.179:6832/3795018457,v1:172.21.15.179:6833/3795018457] conn(0x7f06a806ad10 msgr2=0x7f06a806d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:52.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:52.586+0000 7f06c2125640 1 --2- 172.21.15.57:0/872744200 >> [v2:172.21.15.179:6832/3795018457,v1:172.21.15.179:6833/3795018457] conn(0x7f06a806ad10 0x7f06a806d1d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f06ac009c50 tx=0x7f06ac005f90 comp rx=0 tx=0).stop 2023-10-19T14:01:52.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:52.586+0000 7f06c2125640 1 -- 172.21.15.57:0/872744200 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06bc1ef2d0 msgr2=0x7f06bc13a570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:52.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:52.586+0000 7f06c2125640 1 --2- 172.21.15.57:0/872744200 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06bc1ef2d0 0x7f06bc13a570 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 crypto rx=0x7f06b0003670 tx=0x7f06b0003aa0 comp rx=0 tx=0).stop 2023-10-19T14:01:52.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:52.586+0000 7f06c2125640 1 -- 172.21.15.57:0/872744200 shutdown_connections 2023-10-19T14:01:52.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:52.586+0000 7f06c2125640 1 --2- 172.21.15.57:0/872744200 >> [v2:172.21.15.179:6832/3795018457,v1:172.21.15.179:6833/3795018457] conn(0x7f06a806ad10 0x7f06a806d1d0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:52.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:52.586+0000 7f06c2125640 1 --2- 172.21.15.57:0/872744200 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f06bc14f540 0x7f06bc14f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:52.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:52.586+0000 7f06c2125640 1 --2- 172.21.15.57:0/872744200 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06bc1ef2d0 0x7f06bc13a570 unknown :-1 s=CLOSED pgs=1950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:52.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:52.586+0000 7f06c2125640 1 --2- 172.21.15.57:0/872744200 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f06bc152830 0x7f06bc152c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:52.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:52.586+0000 7f06c2125640 1 -- 172.21.15.57:0/872744200 >> 172.21.15.57:0/872744200 conn(0x7f06bc069930 msgr2=0x7f06bc0adb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:52.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:52.586+0000 7f06c2125640 1 -- 172.21.15.57:0/872744200 shutdown_connections 2023-10-19T14:01:52.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:52.586+0000 7f06c2125640 1 -- 172.21.15.57:0/872744200 wait complete. 2023-10-19T14:01:52.607 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-19T14:01:52.607 DEBUG:teuthology.orchestra.run.smithi057:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-19T14:01:52.655 DEBUG:teuthology.orchestra.run.smithi179:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-19T14:01:52.703 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-19T14:01:52.703 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T14:01:52.704 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T14:01:52.720 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T14:01:52.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:52.869+0000 7f3de812c640 1 -- 172.21.15.57:0/2808788979 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3de015b2a0 msgr2=0x7f3de00cf720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:52.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:52.869+0000 7f3de812c640 1 --2- 172.21.15.57:0/2808788979 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3de015b2a0 0x7f3de00cf720 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f3ddc009830 tx=0x7f3ddc02bcd0 comp rx=0 tx=0).stop 2023-10-19T14:01:52.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:52.869+0000 7f3de812c640 1 -- 172.21.15.57:0/2808788979 shutdown_connections 2023-10-19T14:01:52.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:52.869+0000 7f3de812c640 1 --2- 172.21.15.57:0/2808788979 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3de015b2a0 0x7f3de00cf720 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:52.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:52.869+0000 7f3de812c640 1 --2- 172.21.15.57:0/2808788979 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3de00cf340 0x7f3de00d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:52.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:52.869+0000 7f3de812c640 1 -- 172.21.15.57:0/2808788979 >> 172.21.15.57:0/2808788979 conn(0x7f3de00d0d60 msgr2=0x7f3de00d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:52.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:52.869+0000 7f3de812c640 1 -- 172.21.15.57:0/2808788979 shutdown_connections 2023-10-19T14:01:52.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:52.869+0000 7f3de812c640 1 -- 172.21.15.57:0/2808788979 wait complete. 2023-10-19T14:01:52.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:52.870+0000 7f3de812c640 1 Processor -- start 2023-10-19T14:01:52.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:52.870+0000 7f3de812c640 1 -- start start 2023-10-19T14:01:52.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:52.871+0000 7f3de812c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3de00ceb40 0x7f3de0158350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:52.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:52.871+0000 7f3de812c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3de00cf340 0x7f3de00c9e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:52.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:52.871+0000 7f3de812c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3de015b2a0 0x7f3de00ca380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:52.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:52.871+0000 7f3de812c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3de00b3df0 con 0x7f3de00cf340 2023-10-19T14:01:52.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:52.871+0000 7f3de812c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3de0158890 con 0x7f3de015b2a0 2023-10-19T14:01:52.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:52.871+0000 7f3de812c640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3de015cf00 con 0x7f3de00ceb40 2023-10-19T14:01:52.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:52.871+0000 7f3de56c7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3de00cf340 0x7f3de00c9e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:52.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:52.871+0000 7f3de5ec8640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3de00ceb40 0x7f3de0158350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:52.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:52.871+0000 7f3de56c7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3de00cf340 0x7f3de00c9e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59958/0 (socket says 172.21.15.57:59958) 2023-10-19T14:01:52.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:52.871+0000 7f3de56c7640 1 -- 172.21.15.57:0/881135821 learned_addr learned my addr 172.21.15.57:0/881135821 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:01:52.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:52.871+0000 7f3de66c9640 1 --2- 172.21.15.57:0/881135821 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3de015b2a0 0x7f3de00ca380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:52.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:52.872+0000 7f3de66c9640 1 -- 172.21.15.57:0/881135821 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3de00ceb40 msgr2=0x7f3de0158350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:52.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:52.872+0000 7f3de66c9640 1 --2- 172.21.15.57:0/881135821 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3de00ceb40 0x7f3de0158350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:52.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:52.872+0000 7f3de66c9640 1 -- 172.21.15.57:0/881135821 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3de00cf340 msgr2=0x7f3de00c9e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:52.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:52.872+0000 7f3de66c9640 1 --2- 172.21.15.57:0/881135821 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3de00cf340 0x7f3de00c9e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:52.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:52.872+0000 7f3de66c9640 1 -- 172.21.15.57:0/881135821 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ddc009490 con 0x7f3de015b2a0 2023-10-19T14:01:52.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:52.872+0000 7f3de5ec8640 1 --2- 172.21.15.57:0/881135821 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3de00ceb40 0x7f3de0158350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T14:01:52.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:52.872+0000 7f3de56c7640 1 --2- 172.21.15.57:0/881135821 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3de00cf340 0x7f3de00c9e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T14:01:52.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:52.872+0000 7f3de66c9640 1 --2- 172.21.15.57:0/881135821 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3de015b2a0 0x7f3de00ca380 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f3dd800caa0 tx=0x7f3dd800cf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:52.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:52.872+0000 7f3dd67fc640 1 -- 172.21.15.57:0/881135821 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dd8003da0 con 0x7f3de015b2a0 2023-10-19T14:01:52.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:52.872+0000 7f3dd67fc640 1 -- 172.21.15.57:0/881135821 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f3dd800a790 con 0x7f3de015b2a0 2023-10-19T14:01:52.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:52.872+0000 7f3de812c640 1 -- 172.21.15.57:0/881135821 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3de015d1e0 con 0x7f3de015b2a0 2023-10-19T14:01:52.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:52.872+0000 7f3dd67fc640 1 -- 172.21.15.57:0/881135821 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dd800ac00 con 0x7f3de015b2a0 2023-10-19T14:01:52.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:52.872+0000 7f3de812c640 1 -- 172.21.15.57:0/881135821 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3de015d7a0 con 0x7f3de015b2a0 2023-10-19T14:01:52.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:52.873+0000 7f3de812c640 1 -- 172.21.15.57:0/881135821 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3de00b3fc0 con 0x7f3de015b2a0 2023-10-19T14:01:52.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:52.874+0000 7f3dd67fc640 1 -- 172.21.15.57:0/881135821 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 95) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f3dd8004030 con 0x7f3de015b2a0 2023-10-19T14:01:52.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:52.874+0000 7f3dd67fc640 1 --2- 172.21.15.57:0/881135821 >> [v2:172.21.15.179:6832/3795018457,v1:172.21.15.179:6833/3795018457] conn(0x7f3dc406af60 0x7f3dc406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:52.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:52.874+0000 7f3de5ec8640 1 -- 172.21.15.57:0/881135821 >> [v2:172.21.15.179:6832/3795018457,v1:172.21.15.179:6833/3795018457] conn(0x7f3dc406af60 msgr2=0x7f3dc406d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/3795018457 2023-10-19T14:01:52.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:52.874+0000 7f3dd67fc640 1 -- 172.21.15.57:0/881135821 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f3dd8089fa0 con 0x7f3de015b2a0 2023-10-19T14:01:52.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:52.874+0000 7f3de5ec8640 1 --2- 172.21.15.57:0/881135821 >> [v2:172.21.15.179:6832/3795018457,v1:172.21.15.179:6833/3795018457] conn(0x7f3dc406af60 0x7f3dc406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T14:01:52.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:52.879+0000 7f3dd67fc640 1 -- 172.21.15.57:0/881135821 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3dd80bcca0 con 0x7f3de015b2a0 2023-10-19T14:01:52.977 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:52.975+0000 7f2fcba11040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-19T14:01:52.998 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:01:52.996+0000 7f551f17a040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-19T14:01:53.021 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:53.019+0000 7f2fcba11040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-19T14:01:53.046 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:01:53.043+0000 7f551f17a040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-19T14:01:53.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.057+0000 7f3de812c640 1 -- 172.21.15.57:0/881135821 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3de00690e0 con 0x7f3de015b2a0 2023-10-19T14:01:53.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.058+0000 7f3dd67fc640 1 -- 172.21.15.57:0/881135821 <== mon.1 v2:172.21.15.179:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v176) v1 ==== 74+0+17285 (secure 0 0 0) 0x7f3de00b3fc0 con 0x7f3de015b2a0 2023-10-19T14:01:53.061 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:01:53.061 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":176,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T14:01:49.564595+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2023-10-19T14:01:36.799358+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"174","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_data","create_time":"2023-10-19T14:01:37.481213+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"175","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.179:0/111884799":"2023-10-20T14:01:49.564546+0000","172.21.15.179:0/966948837":"2023-10-20T14:01:49.564546+0000","172.21.15.179:0/1558797744":"2023-10-20T14:01:49.564546+0000","172.21.15.179:6833/2037224568":"2023-10-20T14:01:49.564546+0000","172.21.15.179:6832/2037224568":"2023-10-20T14:01:49.564546+0000","172.21.15.179:0/2965291378":"2023-10-20T14:01:49.564546+0000","172.21.15.179:0/472878921":"2023-10-20T14:01:49.564546+0000","172.21.15.179:0/2416623631":"2023-10-20T14:01:49.564546+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T14:01:53.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.060+0000 7f3de812c640 1 -- 172.21.15.57:0/881135821 >> [v2:172.21.15.179:6832/3795018457,v1:172.21.15.179:6833/3795018457] conn(0x7f3dc406af60 msgr2=0x7f3dc406d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T14:01:53.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.060+0000 7f3de812c640 1 --2- 172.21.15.57:0/881135821 >> [v2:172.21.15.179:6832/3795018457,v1:172.21.15.179:6833/3795018457] conn(0x7f3dc406af60 0x7f3dc406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:53.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.060+0000 7f3de812c640 1 -- 172.21.15.57:0/881135821 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3de015b2a0 msgr2=0x7f3de00ca380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:53.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.060+0000 7f3de812c640 1 --2- 172.21.15.57:0/881135821 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3de015b2a0 0x7f3de00ca380 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f3dd800caa0 tx=0x7f3dd800cf60 comp rx=0 tx=0).stop 2023-10-19T14:01:53.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.060+0000 7f3de812c640 1 -- 172.21.15.57:0/881135821 shutdown_connections 2023-10-19T14:01:53.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.061+0000 7f3de812c640 1 --2- 172.21.15.57:0/881135821 >> [v2:172.21.15.179:6832/3795018457,v1:172.21.15.179:6833/3795018457] conn(0x7f3dc406af60 0x7f3dc406d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:53.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.061+0000 7f3de812c640 1 --2- 172.21.15.57:0/881135821 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3de015b2a0 0x7f3de00ca380 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:53.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.061+0000 7f3de812c640 1 --2- 172.21.15.57:0/881135821 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3de00cf340 0x7f3de00c9e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:53.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.061+0000 7f3de812c640 1 --2- 172.21.15.57:0/881135821 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3de00ceb40 0x7f3de0158350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:53.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.061+0000 7f3de812c640 1 -- 172.21.15.57:0/881135821 >> 172.21.15.57:0/881135821 conn(0x7f3de00d0d60 msgr2=0x7f3de006b350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:53.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.061+0000 7f3de812c640 1 -- 172.21.15.57:0/881135821 shutdown_connections 2023-10-19T14:01:53.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.061+0000 7f3de812c640 1 -- 172.21.15.57:0/881135821 wait complete. 2023-10-19T14:01:53.075 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-19T12:44:07.371186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 21, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-19T14:01:36.799358+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '174', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 22, 'pool_name': 'cephfs_data', 'create_time': '2023-10-19T14:01:37.481213+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '175', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-19T14:01:53.076 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-19T14:01:53.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.186+0000 7fdb3d165640 1 -- 172.21.15.57:0/353092473 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdb3815b2a0 msgr2=0x7fdb380cf720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:53.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.186+0000 7fdb3d165640 1 --2- 172.21.15.57:0/353092473 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdb3815b2a0 0x7fdb380cf720 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fdb28009830 tx=0x7fdb2802bcd0 comp rx=0 tx=0).stop 2023-10-19T14:01:53.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.186+0000 7fdb3d165640 1 -- 172.21.15.57:0/353092473 shutdown_connections 2023-10-19T14:01:53.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.186+0000 7fdb3d165640 1 --2- 172.21.15.57:0/353092473 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdb3815b2a0 0x7fdb380cf720 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:53.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.186+0000 7fdb3d165640 1 --2- 172.21.15.57:0/353092473 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdb380cf340 0x7fdb380d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:53.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.186+0000 7fdb3d165640 1 -- 172.21.15.57:0/353092473 >> 172.21.15.57:0/353092473 conn(0x7fdb380d0d60 msgr2=0x7fdb380d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:53.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.186+0000 7fdb3d165640 1 -- 172.21.15.57:0/353092473 shutdown_connections 2023-10-19T14:01:53.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.186+0000 7fdb3d165640 1 -- 172.21.15.57:0/353092473 wait complete. 2023-10-19T14:01:53.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.187+0000 7fdb3d165640 1 Processor -- start 2023-10-19T14:01:53.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.188+0000 7fdb3d165640 1 -- start start 2023-10-19T14:01:53.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.188+0000 7fdb3d165640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdb380ceb40 0x7fdb381f0900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:53.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.188+0000 7fdb3d165640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdb380cf340 0x7fdb381e4260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:53.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.188+0000 7fdb3d165640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdb3815b2a0 0x7fdb3815c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:53.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.188+0000 7fdb3d165640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb380b4510 con 0x7fdb3815b2a0 2023-10-19T14:01:53.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.188+0000 7fdb3d165640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb380b3f20 con 0x7fdb380ceb40 2023-10-19T14:01:53.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.188+0000 7fdb3d165640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb381e52a0 con 0x7fdb380cf340 2023-10-19T14:01:53.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.188+0000 7fdb3759e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdb3815b2a0 0x7fdb3815c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:53.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.188+0000 7fdb3759e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdb3815b2a0 0x7fdb3815c9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59964/0 (socket says 172.21.15.57:59964) 2023-10-19T14:01:53.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.188+0000 7fdb3659c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdb380cf340 0x7fdb381e4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:53.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.188+0000 7fdb36d9d640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdb380ceb40 0x7fdb381f0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:53.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.188+0000 7fdb3759e640 1 -- 172.21.15.57:0/3991041914 learned_addr learned my addr 172.21.15.57:0/3991041914 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:01:53.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.189+0000 7fdb36d9d640 1 -- 172.21.15.57:0/3991041914 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdb380cf340 msgr2=0x7fdb381e4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:53.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.189+0000 7fdb36d9d640 1 --2- 172.21.15.57:0/3991041914 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdb380cf340 0x7fdb381e4260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:53.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.189+0000 7fdb36d9d640 1 -- 172.21.15.57:0/3991041914 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdb3815b2a0 msgr2=0x7fdb3815c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:53.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.189+0000 7fdb36d9d640 1 --2- 172.21.15.57:0/3991041914 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdb3815b2a0 0x7fdb3815c9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:53.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.189+0000 7fdb36d9d640 1 -- 172.21.15.57:0/3991041914 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb28009490 con 0x7fdb380ceb40 2023-10-19T14:01:53.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.189+0000 7fdb3659c640 1 --2- 172.21.15.57:0/3991041914 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdb380cf340 0x7fdb381e4260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T14:01:53.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.189+0000 7fdb3759e640 1 --2- 172.21.15.57:0/3991041914 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdb3815b2a0 0x7fdb3815c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T14:01:53.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.189+0000 7fdb36d9d640 1 --2- 172.21.15.57:0/3991041914 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdb380ceb40 0x7fdb381f0900 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fdb2c00ea20 tx=0x7fdb2c00eee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:53.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.189+0000 7fdb1f7fe640 1 -- 172.21.15.57:0/3991041914 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb2c00ccc0 con 0x7fdb380ceb40 2023-10-19T14:01:53.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.189+0000 7fdb1f7fe640 1 -- 172.21.15.57:0/3991041914 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fdb2c010430 con 0x7fdb380ceb40 2023-10-19T14:01:53.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.189+0000 7fdb1f7fe640 1 -- 172.21.15.57:0/3991041914 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb2c010800 con 0x7fdb380ceb40 2023-10-19T14:01:53.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.189+0000 7fdb3d165640 1 -- 172.21.15.57:0/3991041914 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb381e4ce0 con 0x7fdb380ceb40 2023-10-19T14:01:53.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.189+0000 7fdb3d165640 1 -- 172.21.15.57:0/3991041914 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb380b4130 con 0x7fdb380ceb40 2023-10-19T14:01:53.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.190+0000 7fdb3d165640 1 -- 172.21.15.57:0/3991041914 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb380690e0 con 0x7fdb380ceb40 2023-10-19T14:01:53.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.191+0000 7fdb1f7fe640 1 -- 172.21.15.57:0/3991041914 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 95) v1 ==== 89332+0+0 (secure 0 0 0) 0x7fdb2c010960 con 0x7fdb380ceb40 2023-10-19T14:01:53.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.191+0000 7fdb1f7fe640 1 --2- 172.21.15.57:0/3991041914 >> [v2:172.21.15.179:6832/3795018457,v1:172.21.15.179:6833/3795018457] conn(0x7fdb2406af60 0x7fdb2406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:53.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.191+0000 7fdb1f7fe640 1 -- 172.21.15.57:0/3991041914 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 6401+0+0 (secure 0 0 0) 0x7fdb2c014070 con 0x7fdb380ceb40 2023-10-19T14:01:53.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.191+0000 7fdb3659c640 1 -- 172.21.15.57:0/3991041914 >> [v2:172.21.15.179:6832/3795018457,v1:172.21.15.179:6833/3795018457] conn(0x7fdb2406af60 msgr2=0x7fdb2406d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/3795018457 2023-10-19T14:01:53.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.191+0000 7fdb3659c640 1 --2- 172.21.15.57:0/3991041914 >> [v2:172.21.15.179:6832/3795018457,v1:172.21.15.179:6833/3795018457] conn(0x7fdb2406af60 0x7fdb2406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T14:01:53.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.196+0000 7fdb1f7fe640 1 -- 172.21.15.57:0/3991041914 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fdb2c051ca0 con 0x7fdb380ceb40 2023-10-19T14:01:53.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.371+0000 7fdb3d165640 1 -- 172.21.15.57:0/3991041914 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fdb380d0b40 con 0x7fdb380ceb40 2023-10-19T14:01:53.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.371+0000 7fdb1f7fe640 1 -- 172.21.15.57:0/3991041914 <== mon.1 v2:172.21.15.179:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v176) v1 ==== 93+0+10 (secure 0 0 0) 0x7fdb2c0552b0 con 0x7fdb380ceb40 2023-10-19T14:01:53.374 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-19T14:01:53.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.373+0000 7fdb3d165640 1 -- 172.21.15.57:0/3991041914 >> [v2:172.21.15.179:6832/3795018457,v1:172.21.15.179:6833/3795018457] conn(0x7fdb2406af60 msgr2=0x7fdb2406d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T14:01:53.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.373+0000 7fdb3d165640 1 --2- 172.21.15.57:0/3991041914 >> [v2:172.21.15.179:6832/3795018457,v1:172.21.15.179:6833/3795018457] conn(0x7fdb2406af60 0x7fdb2406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:53.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.373+0000 7fdb3d165640 1 -- 172.21.15.57:0/3991041914 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdb380ceb40 msgr2=0x7fdb381f0900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:53.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.373+0000 7fdb3d165640 1 --2- 172.21.15.57:0/3991041914 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdb380ceb40 0x7fdb381f0900 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fdb2c00ea20 tx=0x7fdb2c00eee0 comp rx=0 tx=0).stop 2023-10-19T14:01:53.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.373+0000 7fdb3d165640 1 -- 172.21.15.57:0/3991041914 shutdown_connections 2023-10-19T14:01:53.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.373+0000 7fdb3d165640 1 --2- 172.21.15.57:0/3991041914 >> [v2:172.21.15.179:6832/3795018457,v1:172.21.15.179:6833/3795018457] conn(0x7fdb2406af60 0x7fdb2406d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:53.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.373+0000 7fdb3d165640 1 --2- 172.21.15.57:0/3991041914 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdb3815b2a0 0x7fdb3815c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:53.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.373+0000 7fdb3d165640 1 --2- 172.21.15.57:0/3991041914 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdb380cf340 0x7fdb381e4260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:53.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.373+0000 7fdb3d165640 1 --2- 172.21.15.57:0/3991041914 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdb380ceb40 0x7fdb381f0900 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:53.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.373+0000 7fdb3d165640 1 -- 172.21.15.57:0/3991041914 >> 172.21.15.57:0/3991041914 conn(0x7fdb380d0d60 msgr2=0x7fdb380b0ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:53.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.373+0000 7fdb3d165640 1 -- 172.21.15.57:0/3991041914 shutdown_connections 2023-10-19T14:01:53.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.373+0000 7fdb3d165640 1 -- 172.21.15.57:0/3991041914 wait complete. 2023-10-19T14:01:53.387 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-19T14:01:53.414 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:53.412+0000 7f2fcba11040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-19T14:01:53.458 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:53.456+0000 7f2fcba11040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-19T14:01:53.464 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:01:53.461+0000 7f551f17a040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-19T14:01:53.500 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:53.498+0000 7f2fcba11040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-19T14:01:53.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.500+0000 7f1f6d5a9640 1 -- 172.21.15.57:0/2646548807 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1f680cf850 msgr2=0x7f1f680cc9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:53.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.500+0000 7f1f6d5a9640 1 --2- 172.21.15.57:0/2646548807 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1f680cf850 0x7f1f680cc9f0 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 crypto rx=0x7f1f60005fd0 tx=0x7f1f6002c2e0 comp rx=0 tx=0).stop 2023-10-19T14:01:53.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.500+0000 7f1f6d5a9640 1 -- 172.21.15.57:0/2646548807 shutdown_connections 2023-10-19T14:01:53.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.500+0000 7f1f6d5a9640 1 --2- 172.21.15.57:0/2646548807 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1f680cf850 0x7f1f680cc9f0 unknown :-1 s=CLOSED pgs=1953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:53.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.500+0000 7f1f6d5a9640 1 --2- 172.21.15.57:0/2646548807 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1f6815b260 0x7f1f680cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:53.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.500+0000 7f1f6d5a9640 1 -- 172.21.15.57:0/2646548807 >> 172.21.15.57:0/2646548807 conn(0x7f1f6809a030 msgr2=0x7f1f680d0d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:53.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.500+0000 7f1f6d5a9640 1 -- 172.21.15.57:0/2646548807 shutdown_connections 2023-10-19T14:01:53.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.500+0000 7f1f6d5a9640 1 -- 172.21.15.57:0/2646548807 wait complete. 2023-10-19T14:01:53.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.501+0000 7f1f6d5a9640 1 Processor -- start 2023-10-19T14:01:53.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.501+0000 7f1f6d5a9640 1 -- start start 2023-10-19T14:01:53.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.502+0000 7f1f6d5a9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1f680cf850 0x7f1f68150340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:53.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.502+0000 7f1f6d5a9640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1f6815b260 0x7f1f680b6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:53.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.502+0000 7f1f6d5a9640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1f6815b640 0x7f1f6815ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:53.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.502+0000 7f1f6d5a9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f6815cdf0 con 0x7f1f680cf850 2023-10-19T14:01:53.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.502+0000 7f1f6d5a9640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f6815c270 con 0x7f1f6815b260 2023-10-19T14:01:53.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.502+0000 7f1f6d5a9640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f6815c550 con 0x7f1f6815b640 2023-10-19T14:01:53.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.502+0000 7f1f66ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1f680cf850 0x7f1f68150340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:53.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.502+0000 7f1f66ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1f680cf850 0x7f1f68150340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59992/0 (socket says 172.21.15.57:59992) 2023-10-19T14:01:53.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.502+0000 7f1f66ffd640 1 -- 172.21.15.57:0/3696726901 learned_addr learned my addr 172.21.15.57:0/3696726901 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:01:53.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.502+0000 7f1f677fe640 1 --2- 172.21.15.57:0/3696726901 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1f6815b640 0x7f1f6815ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:53.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.502+0000 7f1f667fc640 1 --2- 172.21.15.57:0/3696726901 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1f6815b260 0x7f1f680b6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:53.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.502+0000 7f1f66ffd640 1 -- 172.21.15.57:0/3696726901 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1f6815b640 msgr2=0x7f1f6815ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:53.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.502+0000 7f1f66ffd640 1 --2- 172.21.15.57:0/3696726901 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1f6815b640 0x7f1f6815ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:53.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.502+0000 7f1f66ffd640 1 -- 172.21.15.57:0/3696726901 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1f6815b260 msgr2=0x7f1f680b6380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:53.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.502+0000 7f1f66ffd640 1 --2- 172.21.15.57:0/3696726901 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1f6815b260 0x7f1f680b6380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:53.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.502+0000 7f1f66ffd640 1 -- 172.21.15.57:0/3696726901 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f60009900 con 0x7f1f680cf850 2023-10-19T14:01:53.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.503+0000 7f1f66ffd640 1 --2- 172.21.15.57:0/3696726901 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1f680cf850 0x7f1f68150340 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 crypto rx=0x7f1f5800e9f0 tx=0x7f1f5800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:53.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.503+0000 7f1f4f7fe640 1 -- 172.21.15.57:0/3696726901 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f58009700 con 0x7f1f680cf850 2023-10-19T14:01:53.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.503+0000 7f1f4f7fe640 1 -- 172.21.15.57:0/3696726901 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f1f5800ce70 con 0x7f1f680cf850 2023-10-19T14:01:53.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.503+0000 7f1f6d5a9640 1 -- 172.21.15.57:0/3696726901 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f6815c830 con 0x7f1f680cf850 2023-10-19T14:01:53.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.503+0000 7f1f4f7fe640 1 -- 172.21.15.57:0/3696726901 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f58010670 con 0x7f1f680cf850 2023-10-19T14:01:53.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.503+0000 7f1f6d5a9640 1 -- 172.21.15.57:0/3696726901 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f6815bf60 con 0x7f1f680cf850 2023-10-19T14:01:53.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.504+0000 7f1f6d5a9640 1 -- 172.21.15.57:0/3696726901 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f680b36b0 con 0x7f1f680cf850 2023-10-19T14:01:53.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.504+0000 7f1f4f7fe640 1 -- 172.21.15.57:0/3696726901 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 95) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f1f580107d0 con 0x7f1f680cf850 2023-10-19T14:01:53.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.505+0000 7f1f4f7fe640 1 --2- 172.21.15.57:0/3696726901 >> [v2:172.21.15.179:6832/3795018457,v1:172.21.15.179:6833/3795018457] conn(0x7f1f5406af90 0x7f1f5406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:53.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.505+0000 7f1f4f7fe640 1 -- 172.21.15.57:0/3696726901 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f1f58014070 con 0x7f1f680cf850 2023-10-19T14:01:53.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.505+0000 7f1f667fc640 1 -- 172.21.15.57:0/3696726901 >> [v2:172.21.15.179:6832/3795018457,v1:172.21.15.179:6833/3795018457] conn(0x7f1f5406af90 msgr2=0x7f1f5406d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/3795018457 2023-10-19T14:01:53.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.505+0000 7f1f667fc640 1 --2- 172.21.15.57:0/3696726901 >> [v2:172.21.15.179:6832/3795018457,v1:172.21.15.179:6833/3795018457] conn(0x7f1f5406af90 0x7f1f5406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T14:01:53.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.509+0000 7f1f4f7fe640 1 -- 172.21.15.57:0/3696726901 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f1f58051ca0 con 0x7f1f680cf850 2023-10-19T14:01:53.513 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:01:53.510+0000 7f551f17a040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-19T14:01:53.551 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:53.549+0000 7f2fcba11040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-19T14:01:53.558 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:01:53.556+0000 7f551f17a040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-19T14:01:53.594 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:53.591+0000 7f2fcba11040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-19T14:01:53.614 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:01:53.611+0000 7f551f17a040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-19T14:01:53.658 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:01:53.655+0000 7f551f17a040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-19T14:01:53.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.689+0000 7f1f6d5a9640 1 -- 172.21.15.57:0/3696726901 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f1f680675f0 con 0x7f1f680cf850 2023-10-19T14:01:53.692 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:53.689+0000 7f2fcba11040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-19T14:01:53.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.689+0000 7f1f4f7fe640 1 -- 172.21.15.57:0/3696726901 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v176) v1 ==== 104+0+11 (secure 0 0 0) 0x7f1f58058ea0 con 0x7f1f680cf850 2023-10-19T14:01:53.692 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2023-10-19T14:01:53.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.691+0000 7f1f6d5a9640 1 -- 172.21.15.57:0/3696726901 >> [v2:172.21.15.179:6832/3795018457,v1:172.21.15.179:6833/3795018457] conn(0x7f1f5406af90 msgr2=0x7f1f5406d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T14:01:53.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.691+0000 7f1f6d5a9640 1 --2- 172.21.15.57:0/3696726901 >> [v2:172.21.15.179:6832/3795018457,v1:172.21.15.179:6833/3795018457] conn(0x7f1f5406af90 0x7f1f5406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:53.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.691+0000 7f1f6d5a9640 1 -- 172.21.15.57:0/3696726901 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1f680cf850 msgr2=0x7f1f68150340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:53.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.691+0000 7f1f6d5a9640 1 --2- 172.21.15.57:0/3696726901 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1f680cf850 0x7f1f68150340 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 crypto rx=0x7f1f5800e9f0 tx=0x7f1f5800eeb0 comp rx=0 tx=0).stop 2023-10-19T14:01:53.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.691+0000 7f1f6d5a9640 1 -- 172.21.15.57:0/3696726901 shutdown_connections 2023-10-19T14:01:53.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.691+0000 7f1f6d5a9640 1 --2- 172.21.15.57:0/3696726901 >> [v2:172.21.15.179:6832/3795018457,v1:172.21.15.179:6833/3795018457] conn(0x7f1f5406af90 0x7f1f5406d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:53.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.691+0000 7f1f6d5a9640 1 --2- 172.21.15.57:0/3696726901 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1f6815b640 0x7f1f6815ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:53.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.691+0000 7f1f6d5a9640 1 --2- 172.21.15.57:0/3696726901 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1f6815b260 0x7f1f680b6380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:53.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.691+0000 7f1f6d5a9640 1 --2- 172.21.15.57:0/3696726901 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1f680cf850 0x7f1f68150340 unknown :-1 s=CLOSED pgs=1954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:53.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.691+0000 7f1f6d5a9640 1 -- 172.21.15.57:0/3696726901 >> 172.21.15.57:0/3696726901 conn(0x7f1f6809a030 msgr2=0x7f1f6815a400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:53.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.692+0000 7f1f6d5a9640 1 -- 172.21.15.57:0/3696726901 shutdown_connections 2023-10-19T14:01:53.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.692+0000 7f1f6d5a9640 1 -- 172.21.15.57:0/3696726901 wait complete. 2023-10-19T14:01:53.705 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-19T14:01:53.743 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:53.741+0000 7f2fcba11040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-19T14:01:53.763 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:01:53.761+0000 7f551f17a040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-19T14:01:53.785 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:53.783+0000 7f2fcba11040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-19T14:01:53.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.816+0000 7f7a88844640 1 -- 172.21.15.57:0/1664779183 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7a840cf340 msgr2=0x7f7a840d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:53.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.816+0000 7f7a88844640 1 --2- 172.21.15.57:0/1664779183 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7a840cf340 0x7f7a840d1910 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f7a78009830 tx=0x7f7a7802bcd0 comp rx=0 tx=0).stop 2023-10-19T14:01:53.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.816+0000 7f7a88844640 1 -- 172.21.15.57:0/1664779183 shutdown_connections 2023-10-19T14:01:53.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.816+0000 7f7a88844640 1 --2- 172.21.15.57:0/1664779183 >> v2:172.21.15.179:3300/0 conn(0x7f7a8415b2a0 0x7f7a840cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:53.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.816+0000 7f7a88844640 1 --2- 172.21.15.57:0/1664779183 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7a840cf340 0x7f7a840d1910 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:53.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.816+0000 7f7a88844640 1 -- 172.21.15.57:0/1664779183 >> 172.21.15.57:0/1664779183 conn(0x7f7a840d0d60 msgr2=0x7f7a840d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:53.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.816+0000 7f7a88844640 1 -- 172.21.15.57:0/1664779183 shutdown_connections 2023-10-19T14:01:53.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.816+0000 7f7a88844640 1 -- 172.21.15.57:0/1664779183 wait complete. 2023-10-19T14:01:53.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.817+0000 7f7a88844640 1 Processor -- start 2023-10-19T14:01:53.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.817+0000 7f7a88844640 1 -- start start 2023-10-19T14:01:53.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.818+0000 7f7a88844640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7a840ceb40 0x7f7a84158040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:53.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.818+0000 7f7a88844640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7a840cf340 0x7f7a840b4d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:53.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.818+0000 7f7a88844640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7a8415b2a0 0x7f7a840cc6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:53.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.818+0000 7f7a88844640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a841f5200 con 0x7f7a8415b2a0 2023-10-19T14:01:53.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.818+0000 7f7a88844640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a84158580 con 0x7f7a840cf340 2023-10-19T14:01:53.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.818+0000 7f7a88844640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a841f48d0 con 0x7f7a840ceb40 2023-10-19T14:01:53.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.818+0000 7f7a82d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7a8415b2a0 0x7f7a840cc6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:53.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.818+0000 7f7a82d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7a8415b2a0 0x7f7a840cc6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59998/0 (socket says 172.21.15.57:59998) 2023-10-19T14:01:53.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.818+0000 7f7a82d9d640 1 -- 172.21.15.57:0/3099009052 learned_addr learned my addr 172.21.15.57:0/3099009052 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:01:53.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.818+0000 7f7a8259c640 1 --2- 172.21.15.57:0/3099009052 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7a840ceb40 0x7f7a84158040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:53.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.818+0000 7f7a81d9b640 1 --2- 172.21.15.57:0/3099009052 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7a840cf340 0x7f7a840b4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:53.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.819+0000 7f7a81d9b640 1 -- 172.21.15.57:0/3099009052 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7a840ceb40 msgr2=0x7f7a84158040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:53.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.819+0000 7f7a81d9b640 1 --2- 172.21.15.57:0/3099009052 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7a840ceb40 0x7f7a84158040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:53.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.819+0000 7f7a81d9b640 1 -- 172.21.15.57:0/3099009052 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7a8415b2a0 msgr2=0x7f7a840cc6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:53.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.819+0000 7f7a81d9b640 1 --2- 172.21.15.57:0/3099009052 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7a8415b2a0 0x7f7a840cc6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:53.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.819+0000 7f7a81d9b640 1 -- 172.21.15.57:0/3099009052 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a78009490 con 0x7f7a840cf340 2023-10-19T14:01:53.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.819+0000 7f7a8259c640 1 --2- 172.21.15.57:0/3099009052 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7a840ceb40 0x7f7a84158040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T14:01:53.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.819+0000 7f7a82d9d640 1 --2- 172.21.15.57:0/3099009052 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7a8415b2a0 0x7f7a840cc6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T14:01:53.826 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:01:53.819+0000 7f551f17a040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-19T14:01:53.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.819+0000 7f7a81d9b640 1 --2- 172.21.15.57:0/3099009052 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7a840cf340 0x7f7a840b4d10 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f7a7400e6a0 tx=0x7f7a7400eb60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:53.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.819+0000 7f7a72ffd640 1 -- 172.21.15.57:0/3099009052 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a7400cab0 con 0x7f7a840cf340 2023-10-19T14:01:53.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.819+0000 7f7a72ffd640 1 -- 172.21.15.57:0/3099009052 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f7a7400cd00 con 0x7f7a840cf340 2023-10-19T14:01:53.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.819+0000 7f7a72ffd640 1 -- 172.21.15.57:0/3099009052 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a74010690 con 0x7f7a840cf340 2023-10-19T14:01:53.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.819+0000 7f7a88844640 1 -- 172.21.15.57:0/3099009052 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a841f4bb0 con 0x7f7a840cf340 2023-10-19T14:01:53.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.819+0000 7f7a88844640 1 -- 172.21.15.57:0/3099009052 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a841f5690 con 0x7f7a840cf340 2023-10-19T14:01:53.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.820+0000 7f7a88844640 1 -- 172.21.15.57:0/3099009052 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a841f5370 con 0x7f7a840cf340 2023-10-19T14:01:53.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.821+0000 7f7a72ffd640 1 -- 172.21.15.57:0/3099009052 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 95) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f7a740107f0 con 0x7f7a840cf340 2023-10-19T14:01:53.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.821+0000 7f7a72ffd640 1 --2- 172.21.15.57:0/3099009052 >> [v2:172.21.15.179:6832/3795018457,v1:172.21.15.179:6833/3795018457] conn(0x7f7a6406af60 0x7f7a6406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:53.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.821+0000 7f7a72ffd640 1 -- 172.21.15.57:0/3099009052 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f7a74014070 con 0x7f7a840cf340 2023-10-19T14:01:53.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.821+0000 7f7a8259c640 1 -- 172.21.15.57:0/3099009052 >> [v2:172.21.15.179:6832/3795018457,v1:172.21.15.179:6833/3795018457] conn(0x7f7a6406af60 msgr2=0x7f7a6406d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/3795018457 2023-10-19T14:01:53.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.821+0000 7f7a8259c640 1 --2- 172.21.15.57:0/3099009052 >> [v2:172.21.15.179:6832/3795018457,v1:172.21.15.179:6833/3795018457] conn(0x7f7a6406af60 0x7f7a6406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T14:01:53.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:53.825+0000 7f7a72ffd640 1 -- 172.21.15.57:0/3099009052 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7a74051ca0 con 0x7f7a840cf340 2023-10-19T14:01:53.834 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:53.832+0000 7f2fcba11040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-19T14:01:53.866 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:01:53.864+0000 7f551f17a040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-19T14:01:53.876 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:53.874+0000 7f2fcba11040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-19T14:01:53.917 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:01:53.915+0000 7f551f17a040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-19T14:01:53.962 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:01:53.960+0000 7f551f17a040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-19T14:01:54.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.001+0000 7f7a88844640 1 -- 172.21.15.57:0/3099009052 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7a840690e0 con 0x7f7a840cf340 2023-10-19T14:01:54.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.002+0000 7f7a72ffd640 1 -- 172.21.15.57:0/3099009052 <== mon.1 v2:172.21.15.179:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v176) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7a740550b0 con 0x7f7a840cf340 2023-10-19T14:01:54.005 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 64 2023-10-19T14:01:54.006 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.004+0000 7f7a88844640 1 -- 172.21.15.57:0/3099009052 >> [v2:172.21.15.179:6832/3795018457,v1:172.21.15.179:6833/3795018457] conn(0x7f7a6406af60 msgr2=0x7f7a6406d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T14:01:54.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.004+0000 7f7a88844640 1 --2- 172.21.15.57:0/3099009052 >> [v2:172.21.15.179:6832/3795018457,v1:172.21.15.179:6833/3795018457] conn(0x7f7a6406af60 0x7f7a6406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:54.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.004+0000 7f7a88844640 1 -- 172.21.15.57:0/3099009052 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7a840cf340 msgr2=0x7f7a840b4d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:54.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.004+0000 7f7a88844640 1 --2- 172.21.15.57:0/3099009052 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7a840cf340 0x7f7a840b4d10 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f7a7400e6a0 tx=0x7f7a7400eb60 comp rx=0 tx=0).stop 2023-10-19T14:01:54.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.004+0000 7f7a88844640 1 -- 172.21.15.57:0/3099009052 shutdown_connections 2023-10-19T14:01:54.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.004+0000 7f7a88844640 1 --2- 172.21.15.57:0/3099009052 >> [v2:172.21.15.179:6832/3795018457,v1:172.21.15.179:6833/3795018457] conn(0x7f7a6406af60 0x7f7a6406d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:54.008 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.004+0000 7f7a88844640 1 --2- 172.21.15.57:0/3099009052 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7a8415b2a0 0x7f7a840cc6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:54.008 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.004+0000 7f7a88844640 1 --2- 172.21.15.57:0/3099009052 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7a840cf340 0x7f7a840b4d10 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:54.008 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.004+0000 7f7a88844640 1 --2- 172.21.15.57:0/3099009052 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7a840ceb40 0x7f7a84158040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:54.008 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.004+0000 7f7a88844640 1 -- 172.21.15.57:0/3099009052 >> 172.21.15.57:0/3099009052 conn(0x7f7a840d0d60 msgr2=0x7f7a8415a210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:54.008 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.004+0000 7f7a88844640 1 -- 172.21.15.57:0/3099009052 shutdown_connections 2023-10-19T14:01:54.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.004+0000 7f7a88844640 1 -- 172.21.15.57:0/3099009052 wait complete. 2023-10-19T14:01:54.017 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-19T14:01:54.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.128+0000 7f73aee7c640 1 -- 172.21.15.57:0/4125545255 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f73a815b2a0 msgr2=0x7f73a80cf720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:54.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.128+0000 7f73aee7c640 1 --2- 172.21.15.57:0/4125545255 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f73a815b2a0 0x7f73a80cf720 secure :-1 s=READY pgs=1955 cs=0 l=1 rev1=1 crypto rx=0x7f7394005fd0 tx=0x7f739402bd90 comp rx=0 tx=0).stop 2023-10-19T14:01:54.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.128+0000 7f73aee7c640 1 -- 172.21.15.57:0/4125545255 shutdown_connections 2023-10-19T14:01:54.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.128+0000 7f73aee7c640 1 --2- 172.21.15.57:0/4125545255 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f73a815b2a0 0x7f73a80cf720 unknown :-1 s=CLOSED pgs=1955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:54.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.128+0000 7f73aee7c640 1 --2- 172.21.15.57:0/4125545255 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f73a80cf340 0x7f73a80d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:54.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.128+0000 7f73aee7c640 1 -- 172.21.15.57:0/4125545255 >> 172.21.15.57:0/4125545255 conn(0x7f73a80d0d60 msgr2=0x7f73a80d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:54.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.128+0000 7f73aee7c640 1 -- 172.21.15.57:0/4125545255 shutdown_connections 2023-10-19T14:01:54.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.128+0000 7f73aee7c640 1 -- 172.21.15.57:0/4125545255 wait complete. 2023-10-19T14:01:54.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.129+0000 7f73aee7c640 1 Processor -- start 2023-10-19T14:01:54.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.129+0000 7f73aee7c640 1 -- start start 2023-10-19T14:01:54.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.130+0000 7f73aee7c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f73a80ceb40 0x7f73a8158990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:54.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.130+0000 7f73aee7c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f73a80cf340 0x7f73a80c8840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:54.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.130+0000 7f73aee7c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f73a815b2a0 0x7f73a81f1550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:54.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.130+0000 7f73aee7c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73a8158ed0 con 0x7f73a815b2a0 2023-10-19T14:01:54.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.130+0000 7f73aee7c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73a80c92c0 con 0x7f73a80ceb40 2023-10-19T14:01:54.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.130+0000 7f73aee7c640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73a80c95a0 con 0x7f73a80cf340 2023-10-19T14:01:54.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.130+0000 7f73ad419640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f73a815b2a0 0x7f73a81f1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:54.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.130+0000 7f73a7fff640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f73a80cf340 0x7f73a80c8840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:54.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.130+0000 7f73ad419640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f73a815b2a0 0x7f73a81f1550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:60022/0 (socket says 172.21.15.57:60022) 2023-10-19T14:01:54.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.130+0000 7f73ad419640 1 -- 172.21.15.57:0/1218056266 learned_addr learned my addr 172.21.15.57:0/1218056266 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:01:54.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.130+0000 7f73acc18640 1 --2- 172.21.15.57:0/1218056266 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f73a80ceb40 0x7f73a8158990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:54.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.131+0000 7f73acc18640 1 -- 172.21.15.57:0/1218056266 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f73a80cf340 msgr2=0x7f73a80c8840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:54.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.131+0000 7f73acc18640 1 --2- 172.21.15.57:0/1218056266 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f73a80cf340 0x7f73a80c8840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:54.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.131+0000 7f73acc18640 1 -- 172.21.15.57:0/1218056266 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f73a815b2a0 msgr2=0x7f73a81f1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:54.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.131+0000 7f73acc18640 1 --2- 172.21.15.57:0/1218056266 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f73a815b2a0 0x7f73a81f1550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:54.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.131+0000 7f73acc18640 1 -- 172.21.15.57:0/1218056266 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7394009560 con 0x7f73a80ceb40 2023-10-19T14:01:54.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.131+0000 7f73ad419640 1 --2- 172.21.15.57:0/1218056266 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f73a815b2a0 0x7f73a81f1550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T14:01:54.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.131+0000 7f73acc18640 1 --2- 172.21.15.57:0/1218056266 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f73a80ceb40 0x7f73a8158990 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f739c00e9f0 tx=0x7f739c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:54.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.131+0000 7f73a57fa640 1 -- 172.21.15.57:0/1218056266 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f739c00ccc0 con 0x7f73a80ceb40 2023-10-19T14:01:54.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.131+0000 7f73a57fa640 1 -- 172.21.15.57:0/1218056266 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f739c010430 con 0x7f73a80ceb40 2023-10-19T14:01:54.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.131+0000 7f73aee7c640 1 -- 172.21.15.57:0/1218056266 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73a80c9880 con 0x7f73a80ceb40 2023-10-19T14:01:54.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.131+0000 7f73a57fa640 1 -- 172.21.15.57:0/1218056266 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f739c010820 con 0x7f73a80ceb40 2023-10-19T14:01:54.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.131+0000 7f73aee7c640 1 -- 172.21.15.57:0/1218056266 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73a80c9e40 con 0x7f73a80ceb40 2023-10-19T14:01:54.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.132+0000 7f73aee7c640 1 -- 172.21.15.57:0/1218056266 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73a80c9fd0 con 0x7f73a80ceb40 2023-10-19T14:01:54.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.133+0000 7f73a57fa640 1 -- 172.21.15.57:0/1218056266 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 95) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f739c010980 con 0x7f73a80ceb40 2023-10-19T14:01:54.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.133+0000 7f73a57fa640 1 --2- 172.21.15.57:0/1218056266 >> [v2:172.21.15.179:6832/3795018457,v1:172.21.15.179:6833/3795018457] conn(0x7f738c06af90 0x7f738c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:54.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.133+0000 7f73a7fff640 1 -- 172.21.15.57:0/1218056266 >> [v2:172.21.15.179:6832/3795018457,v1:172.21.15.179:6833/3795018457] conn(0x7f738c06af90 msgr2=0x7f738c06d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/3795018457 2023-10-19T14:01:54.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.133+0000 7f73a57fa640 1 -- 172.21.15.57:0/1218056266 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f739c014070 con 0x7f73a80ceb40 2023-10-19T14:01:54.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.133+0000 7f73a7fff640 1 --2- 172.21.15.57:0/1218056266 >> [v2:172.21.15.179:6832/3795018457,v1:172.21.15.179:6833/3795018457] conn(0x7f738c06af90 0x7f738c06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T14:01:54.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.137+0000 7f73a57fa640 1 -- 172.21.15.57:0/1218056266 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f739c051ca0 con 0x7f73a80ceb40 2023-10-19T14:01:54.194 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:54.192+0000 7f2fcba11040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-19T14:01:54.303 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:01:54.301+0000 7f551f17a040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-19T14:01:54.324 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:54.322+0000 7f2fcba11040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-19T14:01:54.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.327+0000 7f73aee7c640 1 -- 172.21.15.57:0/1218056266 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f73a80690e0 con 0x7f73a80ceb40 2023-10-19T14:01:54.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.328+0000 7f73a57fa640 1 -- 172.21.15.57:0/1218056266 <== mon.1 v2:172.21.15.179:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v176) v1 ==== 80+0+475 (secure 0 0 0) 0x7f739c055230 con 0x7f73a80ceb40 2023-10-19T14:01:54.331 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/111884799 2023-10-20T14:01:49.564546+0000 2023-10-19T14:01:54.331 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/966948837 2023-10-20T14:01:49.564546+0000 2023-10-19T14:01:54.331 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/1558797744 2023-10-20T14:01:49.564546+0000 2023-10-19T14:01:54.332 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:6833/2037224568 2023-10-20T14:01:49.564546+0000 2023-10-19T14:01:54.332 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:6832/2037224568 2023-10-20T14:01:49.564546+0000 2023-10-19T14:01:54.332 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/2965291378 2023-10-20T14:01:49.564546+0000 2023-10-19T14:01:54.332 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/472878921 2023-10-20T14:01:49.564546+0000 2023-10-19T14:01:54.332 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/2416623631 2023-10-20T14:01:49.564546+0000 2023-10-19T14:01:54.333 INFO:teuthology.orchestra.run.smithi057.stderr:listed 8 entries 2023-10-19T14:01:54.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.330+0000 7f73aee7c640 1 -- 172.21.15.57:0/1218056266 >> [v2:172.21.15.179:6832/3795018457,v1:172.21.15.179:6833/3795018457] conn(0x7f738c06af90 msgr2=0x7f738c06d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T14:01:54.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.330+0000 7f73aee7c640 1 --2- 172.21.15.57:0/1218056266 >> [v2:172.21.15.179:6832/3795018457,v1:172.21.15.179:6833/3795018457] conn(0x7f738c06af90 0x7f738c06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:54.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.330+0000 7f73aee7c640 1 -- 172.21.15.57:0/1218056266 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f73a80ceb40 msgr2=0x7f73a8158990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:54.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.330+0000 7f73aee7c640 1 --2- 172.21.15.57:0/1218056266 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f73a80ceb40 0x7f73a8158990 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f739c00e9f0 tx=0x7f739c00eeb0 comp rx=0 tx=0).stop 2023-10-19T14:01:54.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.330+0000 7f73aee7c640 1 -- 172.21.15.57:0/1218056266 shutdown_connections 2023-10-19T14:01:54.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.330+0000 7f73aee7c640 1 --2- 172.21.15.57:0/1218056266 >> [v2:172.21.15.179:6832/3795018457,v1:172.21.15.179:6833/3795018457] conn(0x7f738c06af90 0x7f738c06d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:54.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.330+0000 7f73aee7c640 1 --2- 172.21.15.57:0/1218056266 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f73a815b2a0 0x7f73a81f1550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:54.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.330+0000 7f73aee7c640 1 --2- 172.21.15.57:0/1218056266 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f73a80cf340 0x7f73a80c8840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:54.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.330+0000 7f73aee7c640 1 --2- 172.21.15.57:0/1218056266 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f73a80ceb40 0x7f73a8158990 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:54.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.330+0000 7f73aee7c640 1 -- 172.21.15.57:0/1218056266 >> 172.21.15.57:0/1218056266 conn(0x7f73a80d0d60 msgr2=0x7f73a806a9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:54.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.330+0000 7f73aee7c640 1 -- 172.21.15.57:0/1218056266 shutdown_connections 2023-10-19T14:01:54.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.330+0000 7f73aee7c640 1 -- 172.21.15.57:0/1218056266 wait complete. 2023-10-19T14:01:54.344 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi057.front.sepia.ceph.com... 2023-10-19T14:01:54.344 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:01:54.344 DEBUG:teuthology.orchestra.run.smithi057:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-10-19T14:01:54.373 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi057.stderr:ceph-fuse[142079]: fuse finished with error 0 and tester_r 0 2023-10-19T14:01:54.375 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi057.front.sepia.ceph.com 2023-10-19T14:01:54.376 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:01:54.376 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-19T14:01:54.417 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:54.415+0000 7f2fcba11040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-19T14:01:54.447 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:01:54.445+0000 7f551f17a040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-19T14:01:54.460 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:54.458+0000 7f2fcba11040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-19T14:01:54.544 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-19T14:01:54.544 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T14:01:54.547 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:01:54.545+0000 7f551f17a040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-19T14:01:54.554 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:54.552+0000 7f2fcba11040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-19T14:01:54.560 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054833,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=284 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3247632k,nr_inodes=811908,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-19T14:01:54.560 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi179.front.sepia.ceph.com 2023-10-19T14:01:54.561 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:01:54.561 DEBUG:teuthology.orchestra.run.smithi179:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-10-19T14:01:54.592 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:01:54.590+0000 7f551f17a040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-19T14:01:54.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-19T14:01:54.616 INFO:teuthology.orchestra.run.smithi179.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2023-10-19T14:01:54.616 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T14:01:54.617 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T14:01:54.632 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055614,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=358 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248260k,nr_inodes=812065,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-19T14:01:54.633 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi057.front.sepia.ceph.com 2023-10-19T14:01:54.633 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:01:54.634 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-10-19T14:01:54.651 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:54.648+0000 7f2fcba11040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-19T14:01:54.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-19T14:01:54.688 INFO:teuthology.orchestra.run.smithi057.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2023-10-19T14:01:54.689 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-19T14:01:54.689 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T14:01:54.694 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:01:54.692+0000 7f551f17a040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-19T14:01:54.704 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054833,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=284 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3247632k,nr_inodes=811908,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-19T14:01:54.705 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi179.front.sepia.ceph.com 2023-10-19T14:01:54.705 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:01:54.706 DEBUG:teuthology.orchestra.run.smithi179:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-10-19T14:01:54.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-19T14:01:54.760 INFO:teuthology.orchestra.run.smithi179.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2023-10-19T14:01:54.761 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T14:01:54.790 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:01:54.788+0000 7f551f17a040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-19T14:01:54.791 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:54.789+0000 7f2fcba11040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-19T14:01:54.834 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:54.832+0000 7f2fcba11040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-19T14:01:54.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.868+0000 7fd3975dc640 1 -- 172.21.15.57:0/2970176677 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd3900cf850 msgr2=0x7fd3900cc9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:54.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.868+0000 7fd3975dc640 1 --2- 172.21.15.57:0/2970176677 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd3900cf850 0x7fd3900cc9f0 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7fd3900894b0 tx=0x7fd38802c2f0 comp rx=0 tx=0).stop 2023-10-19T14:01:54.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.868+0000 7fd3975dc640 1 -- 172.21.15.57:0/2970176677 shutdown_connections 2023-10-19T14:01:54.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.868+0000 7fd3975dc640 1 --2- 172.21.15.57:0/2970176677 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd3900cf850 0x7fd3900cc9f0 unknown :-1 s=CLOSED pgs=1956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:54.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.868+0000 7fd3975dc640 1 --2- 172.21.15.57:0/2970176677 >> v2:172.21.15.179:3300/0 conn(0x7fd39015b260 0x7fd3900cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:54.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.868+0000 7fd3975dc640 1 -- 172.21.15.57:0/2970176677 >> 172.21.15.57:0/2970176677 conn(0x7fd39009a030 msgr2=0x7fd3900d0d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:54.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.868+0000 7fd3975dc640 1 -- 172.21.15.57:0/2970176677 shutdown_connections 2023-10-19T14:01:54.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.868+0000 7fd3975dc640 1 -- 172.21.15.57:0/2970176677 wait complete. 2023-10-19T14:01:54.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.870+0000 7fd3975dc640 1 Processor -- start 2023-10-19T14:01:54.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.870+0000 7fd3975dc640 1 -- start start 2023-10-19T14:01:54.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.871+0000 7fd3975dc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd3900cf850 0x7fd390149a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:54.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.871+0000 7fd3975dc640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd39015b260 0x7fd3900c6810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:54.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.871+0000 7fd3975dc640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd39015b640 0x7fd3901485a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:54.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.872+0000 7fd3975dc640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd39015a470 con 0x7fd3900cf850 2023-10-19T14:01:54.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.872+0000 7fd3975dc640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd390148110 con 0x7fd39015b640 2023-10-19T14:01:54.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.872+0000 7fd3975dc640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd390149fd0 con 0x7fd39015b260 2023-10-19T14:01:54.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.872+0000 7fd395378640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd3900cf850 0x7fd390149a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:54.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.872+0000 7fd395378640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd3900cf850 0x7fd390149a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:60042/0 (socket says 172.21.15.57:60042) 2023-10-19T14:01:54.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.872+0000 7fd395378640 1 -- 172.21.15.57:0/1956514332 learned_addr learned my addr 172.21.15.57:0/1956514332 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:01:54.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.872+0000 7fd395b79640 1 --2- 172.21.15.57:0/1956514332 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd39015b640 0x7fd3901485a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:54.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.872+0000 7fd394b77640 1 --2- 172.21.15.57:0/1956514332 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd39015b260 0x7fd3900c6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:54.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.872+0000 7fd395378640 1 -- 172.21.15.57:0/1956514332 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd39015b260 msgr2=0x7fd3900c6810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:54.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.872+0000 7fd395378640 1 --2- 172.21.15.57:0/1956514332 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd39015b260 0x7fd3900c6810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:54.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.872+0000 7fd395378640 1 -- 172.21.15.57:0/1956514332 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd39015b640 msgr2=0x7fd3901485a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:54.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.872+0000 7fd395378640 1 --2- 172.21.15.57:0/1956514332 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd39015b640 0x7fd3901485a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:54.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.872+0000 7fd395378640 1 -- 172.21.15.57:0/1956514332 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd388009900 con 0x7fd3900cf850 2023-10-19T14:01:54.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.872+0000 7fd395b79640 1 --2- 172.21.15.57:0/1956514332 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd39015b640 0x7fd3901485a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T14:01:54.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.872+0000 7fd394b77640 1 --2- 172.21.15.57:0/1956514332 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd39015b260 0x7fd3900c6810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:01:54.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.872+0000 7fd395378640 1 --2- 172.21.15.57:0/1956514332 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd3900cf850 0x7fd390149a90 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7fd38000e9f0 tx=0x7fd38000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:54.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.872+0000 7fd385ffb640 1 -- 172.21.15.57:0/1956514332 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd380009700 con 0x7fd3900cf850 2023-10-19T14:01:54.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.872+0000 7fd385ffb640 1 -- 172.21.15.57:0/1956514332 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fd38000ce70 con 0x7fd3900cf850 2023-10-19T14:01:54.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.872+0000 7fd3975dc640 1 -- 172.21.15.57:0/1956514332 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd390147500 con 0x7fd3900cf850 2023-10-19T14:01:54.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.872+0000 7fd385ffb640 1 -- 172.21.15.57:0/1956514332 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd380010670 con 0x7fd3900cf850 2023-10-19T14:01:54.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.873+0000 7fd3975dc640 1 -- 172.21.15.57:0/1956514332 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd390147ac0 con 0x7fd3900cf850 2023-10-19T14:01:54.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.874+0000 7fd3975dc640 1 -- 172.21.15.57:0/1956514332 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3900675f0 con 0x7fd3900cf850 2023-10-19T14:01:54.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.874+0000 7fd385ffb640 1 -- 172.21.15.57:0/1956514332 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 95) v1 ==== 89332+0+0 (secure 0 0 0) 0x7fd3800107d0 con 0x7fd3900cf850 2023-10-19T14:01:54.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.875+0000 7fd385ffb640 1 --2- 172.21.15.57:0/1956514332 >> [v2:172.21.15.179:6832/3795018457,v1:172.21.15.179:6833/3795018457] conn(0x7fd37406af60 0x7fd37406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:54.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.875+0000 7fd394b77640 1 -- 172.21.15.57:0/1956514332 >> [v2:172.21.15.179:6832/3795018457,v1:172.21.15.179:6833/3795018457] conn(0x7fd37406af60 msgr2=0x7fd37406d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/3795018457 2023-10-19T14:01:54.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.875+0000 7fd385ffb640 1 -- 172.21.15.57:0/1956514332 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 6401+0+0 (secure 0 0 0) 0x7fd380014070 con 0x7fd3900cf850 2023-10-19T14:01:54.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.875+0000 7fd394b77640 1 --2- 172.21.15.57:0/1956514332 >> [v2:172.21.15.179:6832/3795018457,v1:172.21.15.179:6833/3795018457] conn(0x7fd37406af60 0x7fd37406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T14:01:54.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:54.879+0000 7fd385ffb640 1 -- 172.21.15.57:0/1956514332 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd380051ca0 con 0x7fd3900cf850 2023-10-19T14:01:54.938 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:54.936+0000 7f2fcba11040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-19T14:01:54.940 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:01:54.938+0000 7f551f17a040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-19T14:01:54.985 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:01:54.982+0000 7f551f17a040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-19T14:01:55.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.076+0000 7fd394b77640 1 -- 172.21.15.57:0/1956514332 >> [v2:172.21.15.179:6832/3795018457,v1:172.21.15.179:6833/3795018457] conn(0x7fd37406af60 msgr2=0x7fd37406d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/3795018457 2023-10-19T14:01:55.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.076+0000 7fd394b77640 1 --2- 172.21.15.57:0/1956514332 >> [v2:172.21.15.179:6832/3795018457,v1:172.21.15.179:6833/3795018457] conn(0x7fd37406af60 0x7fd37406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-19T14:01:55.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.096+0000 7fd3975dc640 1 -- 172.21.15.57:0/1956514332 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd3900cd8c0 con 0x7fd3900cf850 2023-10-19T14:01:55.099 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:01:55.096+0000 7f551f17a040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-19T14:01:55.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.097+0000 7fd385ffb640 1 -- 172.21.15.57:0/1956514332 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+4920 (secure 0 0 0) 0x7fd380058e00 con 0x7fd3900cf850 2023-10-19T14:01:55.100 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:01:55.100 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":70,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8126,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/1759316409","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1759316409},{"type":"v1","addr":"172.21.15.57:6835","nonce":1759316409}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67},{"gid":8129,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/2462599644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2462599644},{"type":"v1","addr":"172.21.15.57:6834","nonce":2462599644}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67},{"gid":9136,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/373220657","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":373220657},{"type":"v1","addr":"172.21.15.179:6836","nonce":373220657}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T14:01:38.503105+0000","modified":"2023-10-19T14:01:40.495529+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9139},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9139":{"gid":9139,"name":"d","rank":0,"incarnation":68,"state":"up:active","state_seq":3,"addr":"172.21.15.179:6837/1102326590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":1102326590},{"type":"v1","addr":"172.21.15.179:6837","nonce":1102326590}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2023-10-19T14:01:55.101 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 70 2023-10-19T14:01:55.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.099+0000 7fd3975dc640 1 -- 172.21.15.57:0/1956514332 >> [v2:172.21.15.179:6832/3795018457,v1:172.21.15.179:6833/3795018457] conn(0x7fd37406af60 msgr2=0x7fd37406d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T14:01:55.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.099+0000 7fd3975dc640 1 --2- 172.21.15.57:0/1956514332 >> [v2:172.21.15.179:6832/3795018457,v1:172.21.15.179:6833/3795018457] conn(0x7fd37406af60 0x7fd37406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:55.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.099+0000 7fd3975dc640 1 -- 172.21.15.57:0/1956514332 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd3900cf850 msgr2=0x7fd390149a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:55.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.099+0000 7fd3975dc640 1 --2- 172.21.15.57:0/1956514332 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd3900cf850 0x7fd390149a90 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7fd38000e9f0 tx=0x7fd38000eeb0 comp rx=0 tx=0).stop 2023-10-19T14:01:55.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.099+0000 7fd3975dc640 1 -- 172.21.15.57:0/1956514332 shutdown_connections 2023-10-19T14:01:55.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.099+0000 7fd3975dc640 1 --2- 172.21.15.57:0/1956514332 >> [v2:172.21.15.179:6832/3795018457,v1:172.21.15.179:6833/3795018457] conn(0x7fd37406af60 0x7fd37406d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:55.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.099+0000 7fd3975dc640 1 --2- 172.21.15.57:0/1956514332 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd39015b640 0x7fd3901485a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:55.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.099+0000 7fd3975dc640 1 --2- 172.21.15.57:0/1956514332 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd39015b260 0x7fd3900c6810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:55.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.099+0000 7fd3975dc640 1 --2- 172.21.15.57:0/1956514332 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd3900cf850 0x7fd390149a90 unknown :-1 s=CLOSED pgs=1957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:55.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.099+0000 7fd3975dc640 1 -- 172.21.15.57:0/1956514332 >> 172.21.15.57:0/1956514332 conn(0x7fd39009a030 msgr2=0x7fd3900b2090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:55.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.099+0000 7fd3975dc640 1 -- 172.21.15.57:0/1956514332 shutdown_connections 2023-10-19T14:01:55.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.099+0000 7fd3975dc640 1 -- 172.21.15.57:0/1956514332 wait complete. 2023-10-19T14:01:55.112 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T14:01:55.262 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.260+0000 7fc41a5c0640 1 -- 172.21.15.57:0/831488215 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4140cf340 msgr2=0x7fc4140d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:55.263 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.260+0000 7fc41a5c0640 1 --2- 172.21.15.57:0/831488215 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4140cf340 0x7fc4140d1910 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7fc41408a120 tx=0x7fc40402bdb0 comp rx=0 tx=0).stop 2023-10-19T14:01:55.263 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.260+0000 7fc41a5c0640 1 -- 172.21.15.57:0/831488215 shutdown_connections 2023-10-19T14:01:55.263 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.260+0000 7fc41a5c0640 1 --2- 172.21.15.57:0/831488215 >> v2:172.21.15.179:3300/0 conn(0x7fc41415b2a0 0x7fc4140cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:55.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.260+0000 7fc41a5c0640 1 --2- 172.21.15.57:0/831488215 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4140cf340 0x7fc4140d1910 unknown :-1 s=CLOSED pgs=1959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:55.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.260+0000 7fc41a5c0640 1 -- 172.21.15.57:0/831488215 >> 172.21.15.57:0/831488215 conn(0x7fc4140d0d60 msgr2=0x7fc4140d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:55.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.260+0000 7fc41a5c0640 1 -- 172.21.15.57:0/831488215 shutdown_connections 2023-10-19T14:01:55.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.260+0000 7fc41a5c0640 1 -- 172.21.15.57:0/831488215 wait complete. 2023-10-19T14:01:55.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.261+0000 7fc41a5c0640 1 Processor -- start 2023-10-19T14:01:55.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.261+0000 7fc41a5c0640 1 -- start start 2023-10-19T14:01:55.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.262+0000 7fc41a5c0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc4140ceb40 0x7fc4140b4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:55.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.262+0000 7fc41a5c0640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc4140cf340 0x7fc41415d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:55.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.262+0000 7fc41a5c0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc41415b2a0 0x7fc4140b1110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:55.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.262+0000 7fc41a5c0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc41409aa60 con 0x7fc41415b2a0 2023-10-19T14:01:55.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.262+0000 7fc41a5c0640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4141f15f0 con 0x7fc4140ceb40 2023-10-19T14:01:55.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.262+0000 7fc41a5c0640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc41415d5e0 con 0x7fc4140cf340 2023-10-19T14:01:55.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.262+0000 7fc418b5d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc41415b2a0 0x7fc4140b1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:55.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.262+0000 7fc418b5d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc41415b2a0 0x7fc4140b1110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:60056/0 (socket says 172.21.15.57:60056) 2023-10-19T14:01:55.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.262+0000 7fc418b5d640 1 -- 172.21.15.57:0/1757546684 learned_addr learned my addr 172.21.15.57:0/1757546684 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:01:55.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.262+0000 7fc413fff640 1 --2- 172.21.15.57:0/1757546684 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc4140ceb40 0x7fc4140b4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:55.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.262+0000 7fc4137fe640 1 --2- 172.21.15.57:0/1757546684 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc4140cf340 0x7fc41415d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:55.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.262+0000 7fc418b5d640 1 -- 172.21.15.57:0/1757546684 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc4140cf340 msgr2=0x7fc41415d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:55.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.262+0000 7fc418b5d640 1 --2- 172.21.15.57:0/1757546684 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc4140cf340 0x7fc41415d0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:55.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.262+0000 7fc418b5d640 1 -- 172.21.15.57:0/1757546684 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc4140ceb40 msgr2=0x7fc4140b4360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:55.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.262+0000 7fc418b5d640 1 --2- 172.21.15.57:0/1757546684 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc4140ceb40 0x7fc4140b4360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:55.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.262+0000 7fc418b5d640 1 -- 172.21.15.57:0/1757546684 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc404009560 con 0x7fc41415b2a0 2023-10-19T14:01:55.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.263+0000 7fc418b5d640 1 --2- 172.21.15.57:0/1757546684 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc41415b2a0 0x7fc4140b1110 secure :-1 s=READY pgs=1960 cs=0 l=1 rev1=1 crypto rx=0x7fc40c00c730 tx=0x7fc40c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:55.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.263+0000 7fc410ff9640 1 -- 172.21.15.57:0/1757546684 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc40c003b90 con 0x7fc41415b2a0 2023-10-19T14:01:55.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.263+0000 7fc410ff9640 1 -- 172.21.15.57:0/1757546684 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fc40c003cf0 con 0x7fc41415b2a0 2023-10-19T14:01:55.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.263+0000 7fc410ff9640 1 -- 172.21.15.57:0/1757546684 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc40c00aab0 con 0x7fc41415b2a0 2023-10-19T14:01:55.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.263+0000 7fc41a5c0640 1 -- 172.21.15.57:0/1757546684 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4140b4a30 con 0x7fc41415b2a0 2023-10-19T14:01:55.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.263+0000 7fc41a5c0640 1 -- 172.21.15.57:0/1757546684 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4140b5270 con 0x7fc41415b2a0 2023-10-19T14:01:55.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.264+0000 7fc41a5c0640 1 -- 172.21.15.57:0/1757546684 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4140677b0 con 0x7fc41415b2a0 2023-10-19T14:01:55.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.265+0000 7fc410ff9640 1 -- 172.21.15.57:0/1757546684 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 95) v1 ==== 89332+0+0 (secure 0 0 0) 0x7fc40c00ac10 con 0x7fc41415b2a0 2023-10-19T14:01:55.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.265+0000 7fc410ff9640 1 --2- 172.21.15.57:0/1757546684 >> [v2:172.21.15.179:6832/3795018457,v1:172.21.15.179:6833/3795018457] conn(0x7fc40006af60 0x7fc40006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:55.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.265+0000 7fc413fff640 1 -- 172.21.15.57:0/1757546684 >> [v2:172.21.15.179:6832/3795018457,v1:172.21.15.179:6833/3795018457] conn(0x7fc40006af60 msgr2=0x7fc40006d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/3795018457 2023-10-19T14:01:55.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.265+0000 7fc410ff9640 1 -- 172.21.15.57:0/1757546684 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 6401+0+0 (secure 0 0 0) 0x7fc40c089890 con 0x7fc41415b2a0 2023-10-19T14:01:55.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.265+0000 7fc413fff640 1 --2- 172.21.15.57:0/1757546684 >> [v2:172.21.15.179:6832/3795018457,v1:172.21.15.179:6833/3795018457] conn(0x7fc40006af60 0x7fc40006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T14:01:55.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.270+0000 7fc410ff9640 1 -- 172.21.15.57:0/1757546684 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc40c054630 con 0x7fc41415b2a0 2023-10-19T14:01:55.453 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.451+0000 7fc41a5c0640 1 -- 172.21.15.57:0/1757546684 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc4140cb9d0 con 0x7fc41415b2a0 2023-10-19T14:01:55.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.452+0000 7fc410ff9640 1 -- 172.21.15.57:0/1757546684 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v176) v1 ==== 74+0+17285 (secure 0 0 0) 0x7fc40c057c50 con 0x7fc41415b2a0 2023-10-19T14:01:55.455 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:01:55.455 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":176,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T14:01:49.564595+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2023-10-19T14:01:36.799358+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"174","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_data","create_time":"2023-10-19T14:01:37.481213+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"175","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.179:0/111884799":"2023-10-20T14:01:49.564546+0000","172.21.15.179:0/966948837":"2023-10-20T14:01:49.564546+0000","172.21.15.179:0/1558797744":"2023-10-20T14:01:49.564546+0000","172.21.15.179:6833/2037224568":"2023-10-20T14:01:49.564546+0000","172.21.15.179:6832/2037224568":"2023-10-20T14:01:49.564546+0000","172.21.15.179:0/2965291378":"2023-10-20T14:01:49.564546+0000","172.21.15.179:0/472878921":"2023-10-20T14:01:49.564546+0000","172.21.15.179:0/2416623631":"2023-10-20T14:01:49.564546+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T14:01:55.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.455+0000 7fc41a5c0640 1 -- 172.21.15.57:0/1757546684 >> [v2:172.21.15.179:6832/3795018457,v1:172.21.15.179:6833/3795018457] conn(0x7fc40006af60 msgr2=0x7fc40006d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T14:01:55.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.455+0000 7fc41a5c0640 1 --2- 172.21.15.57:0/1757546684 >> [v2:172.21.15.179:6832/3795018457,v1:172.21.15.179:6833/3795018457] conn(0x7fc40006af60 0x7fc40006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:55.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.455+0000 7fc41a5c0640 1 -- 172.21.15.57:0/1757546684 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc41415b2a0 msgr2=0x7fc4140b1110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:55.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.455+0000 7fc41a5c0640 1 --2- 172.21.15.57:0/1757546684 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc41415b2a0 0x7fc4140b1110 secure :-1 s=READY pgs=1960 cs=0 l=1 rev1=1 crypto rx=0x7fc40c00c730 tx=0x7fc40c00cbf0 comp rx=0 tx=0).stop 2023-10-19T14:01:55.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.455+0000 7fc41a5c0640 1 -- 172.21.15.57:0/1757546684 shutdown_connections 2023-10-19T14:01:55.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.455+0000 7fc41a5c0640 1 --2- 172.21.15.57:0/1757546684 >> [v2:172.21.15.179:6832/3795018457,v1:172.21.15.179:6833/3795018457] conn(0x7fc40006af60 0x7fc40006d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:55.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.455+0000 7fc41a5c0640 1 --2- 172.21.15.57:0/1757546684 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc41415b2a0 0x7fc4140b1110 unknown :-1 s=CLOSED pgs=1960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:55.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.455+0000 7fc41a5c0640 1 --2- 172.21.15.57:0/1757546684 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc4140cf340 0x7fc41415d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:55.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.455+0000 7fc41a5c0640 1 --2- 172.21.15.57:0/1757546684 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc4140ceb40 0x7fc4140b4360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:55.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.455+0000 7fc41a5c0640 1 -- 172.21.15.57:0/1757546684 >> 172.21.15.57:0/1757546684 conn(0x7fc4140d0d60 msgr2=0x7fc41415a210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:55.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.455+0000 7fc41a5c0640 1 -- 172.21.15.57:0/1757546684 shutdown_connections 2023-10-19T14:01:55.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.455+0000 7fc41a5c0640 1 -- 172.21.15.57:0/1757546684 wait complete. 2023-10-19T14:01:55.469 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-19T12:44:07.371186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 21, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-19T14:01:36.799358+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '174', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 22, 'pool_name': 'cephfs_data', 'create_time': '2023-10-19T14:01:37.481213+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '175', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-19T14:01:55.469 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-19T14:01:55.557 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:55.555+0000 7f2fcba11040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-19T14:01:55.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.581+0000 7f8ec9675640 1 -- 172.21.15.57:0/2637026958 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ec413bd70 msgr2=0x7f8ec4070ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:55.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.581+0000 7f8ec9675640 1 --2- 172.21.15.57:0/2637026958 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ec413bd70 0x7f8ec4070ce0 secure :-1 s=READY pgs=1961 cs=0 l=1 rev1=1 crypto rx=0x7f8eb8005fd0 tx=0x7f8eb802be10 comp rx=0 tx=0).stop 2023-10-19T14:01:55.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.581+0000 7f8ec9675640 1 -- 172.21.15.57:0/2637026958 shutdown_connections 2023-10-19T14:01:55.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.581+0000 7f8ec9675640 1 --2- 172.21.15.57:0/2637026958 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8ec4069720 0x7f8ec4069b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:55.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.581+0000 7f8ec9675640 1 --2- 172.21.15.57:0/2637026958 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ec413bd70 0x7f8ec4070ce0 unknown :-1 s=CLOSED pgs=1961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:55.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.581+0000 7f8ec9675640 1 -- 172.21.15.57:0/2637026958 >> 172.21.15.57:0/2637026958 conn(0x7f8ec40af730 msgr2=0x7f8ec40afb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:55.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.581+0000 7f8ec9675640 1 -- 172.21.15.57:0/2637026958 shutdown_connections 2023-10-19T14:01:55.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.582+0000 7f8ec9675640 1 -- 172.21.15.57:0/2637026958 wait complete. 2023-10-19T14:01:55.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.583+0000 7f8ec9675640 1 Processor -- start 2023-10-19T14:01:55.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.583+0000 7f8ec9675640 1 -- start start 2023-10-19T14:01:55.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.584+0000 7f8ec9675640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8ec4069720 0x7f8ec4158060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:55.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.584+0000 7f8ec9675640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8ec413bd70 0x7f8ec40b0b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:55.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.584+0000 7f8ec9675640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ec413e4d0 0x7f8ec4137630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:55.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.584+0000 7f8ec9675640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ec40b15f0 con 0x7f8ec413e4d0 2023-10-19T14:01:55.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.584+0000 7f8ec9675640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ec40b18d0 con 0x7f8ec4069720 2023-10-19T14:01:55.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.584+0000 7f8ec9675640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ec40b1bb0 con 0x7f8ec413bd70 2023-10-19T14:01:55.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.584+0000 7f8ec37fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ec413e4d0 0x7f8ec4137630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:55.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.584+0000 7f8ec37fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ec413e4d0 0x7f8ec4137630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:60084/0 (socket says 172.21.15.57:60084) 2023-10-19T14:01:55.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.584+0000 7f8ec37fe640 1 -- 172.21.15.57:0/4230032523 learned_addr learned my addr 172.21.15.57:0/4230032523 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:01:55.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.584+0000 7f8ec27fc640 1 --2- 172.21.15.57:0/4230032523 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8ec413bd70 0x7f8ec40b0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:55.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.584+0000 7f8ec2ffd640 1 --2- 172.21.15.57:0/4230032523 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8ec4069720 0x7f8ec4158060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:55.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.584+0000 7f8ec37fe640 1 -- 172.21.15.57:0/4230032523 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8ec413bd70 msgr2=0x7f8ec40b0b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:55.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.584+0000 7f8ec37fe640 1 --2- 172.21.15.57:0/4230032523 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8ec413bd70 0x7f8ec40b0b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:55.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.584+0000 7f8ec37fe640 1 -- 172.21.15.57:0/4230032523 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8ec4069720 msgr2=0x7f8ec4158060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:55.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.584+0000 7f8ec37fe640 1 --2- 172.21.15.57:0/4230032523 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8ec4069720 0x7f8ec4158060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:55.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.584+0000 7f8ec37fe640 1 -- 172.21.15.57:0/4230032523 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eb8009560 con 0x7f8ec413e4d0 2023-10-19T14:01:55.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.584+0000 7f8ec37fe640 1 --2- 172.21.15.57:0/4230032523 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ec413e4d0 0x7f8ec4137630 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc00c730 tx=0x7f8ebc00cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:55.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.584+0000 7f8eab7fe640 1 -- 172.21.15.57:0/4230032523 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ebc006d50 con 0x7f8ec413e4d0 2023-10-19T14:01:55.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.585+0000 7f8ec9675640 1 -- 172.21.15.57:0/4230032523 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ec40b1e90 con 0x7f8ec413e4d0 2023-10-19T14:01:55.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.585+0000 7f8ec9675640 1 -- 172.21.15.57:0/4230032523 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ec41585a0 con 0x7f8ec413e4d0 2023-10-19T14:01:55.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.585+0000 7f8eab7fe640 1 -- 172.21.15.57:0/4230032523 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f8ebc003970 con 0x7f8ec413e4d0 2023-10-19T14:01:55.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.585+0000 7f8eab7fe640 1 -- 172.21.15.57:0/4230032523 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ebc003d40 con 0x7f8ec413e4d0 2023-10-19T14:01:55.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.586+0000 7f8ec9675640 1 -- 172.21.15.57:0/4230032523 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ea4005250 con 0x7f8ec413e4d0 2023-10-19T14:01:55.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.587+0000 7f8eab7fe640 1 -- 172.21.15.57:0/4230032523 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 95) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f8ebc0050d0 con 0x7f8ec413e4d0 2023-10-19T14:01:55.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.587+0000 7f8eab7fe640 1 --2- 172.21.15.57:0/4230032523 >> [v2:172.21.15.179:6832/3795018457,v1:172.21.15.179:6833/3795018457] conn(0x7f8eb006af60 0x7f8eb006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:55.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.587+0000 7f8eab7fe640 1 -- 172.21.15.57:0/4230032523 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f8ebc089680 con 0x7f8ec413e4d0 2023-10-19T14:01:55.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.587+0000 7f8ec2ffd640 1 -- 172.21.15.57:0/4230032523 >> [v2:172.21.15.179:6832/3795018457,v1:172.21.15.179:6833/3795018457] conn(0x7f8eb006af60 msgr2=0x7f8eb006d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/3795018457 2023-10-19T14:01:55.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.587+0000 7f8ec2ffd640 1 --2- 172.21.15.57:0/4230032523 >> [v2:172.21.15.179:6832/3795018457,v1:172.21.15.179:6833/3795018457] conn(0x7f8eb006af60 0x7f8eb006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T14:01:55.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.594+0000 7f8eab7fe640 1 -- 172.21.15.57:0/4230032523 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8ebc054420 con 0x7f8ec413e4d0 2023-10-19T14:01:55.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.771+0000 7f8ec9675640 1 -- 172.21.15.57:0/4230032523 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8ea4005a80 con 0x7f8ec413e4d0 2023-10-19T14:01:55.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.772+0000 7f8eab7fe640 1 -- 172.21.15.57:0/4230032523 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v176) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8ebc057a40 con 0x7f8ec413e4d0 2023-10-19T14:01:55.775 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-19T14:01:55.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.774+0000 7f8ec9675640 1 -- 172.21.15.57:0/4230032523 >> [v2:172.21.15.179:6832/3795018457,v1:172.21.15.179:6833/3795018457] conn(0x7f8eb006af60 msgr2=0x7f8eb006d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T14:01:55.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.774+0000 7f8ec9675640 1 --2- 172.21.15.57:0/4230032523 >> [v2:172.21.15.179:6832/3795018457,v1:172.21.15.179:6833/3795018457] conn(0x7f8eb006af60 0x7f8eb006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:55.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.774+0000 7f8ec9675640 1 -- 172.21.15.57:0/4230032523 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ec413e4d0 msgr2=0x7f8ec4137630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:55.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.774+0000 7f8ec9675640 1 --2- 172.21.15.57:0/4230032523 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ec413e4d0 0x7f8ec4137630 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc00c730 tx=0x7f8ebc00cb30 comp rx=0 tx=0).stop 2023-10-19T14:01:55.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.774+0000 7f8ec9675640 1 -- 172.21.15.57:0/4230032523 shutdown_connections 2023-10-19T14:01:55.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.774+0000 7f8ec9675640 1 --2- 172.21.15.57:0/4230032523 >> [v2:172.21.15.179:6832/3795018457,v1:172.21.15.179:6833/3795018457] conn(0x7f8eb006af60 0x7f8eb006d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:55.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.774+0000 7f8ec9675640 1 --2- 172.21.15.57:0/4230032523 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ec413e4d0 0x7f8ec4137630 unknown :-1 s=CLOSED pgs=1962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:55.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.774+0000 7f8ec9675640 1 --2- 172.21.15.57:0/4230032523 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8ec413bd70 0x7f8ec40b0b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:55.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.774+0000 7f8ec9675640 1 --2- 172.21.15.57:0/4230032523 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8ec4069720 0x7f8ec4158060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:55.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.774+0000 7f8ec9675640 1 -- 172.21.15.57:0/4230032523 >> 172.21.15.57:0/4230032523 conn(0x7f8ec40af730 msgr2=0x7f8ec40ae2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:55.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.774+0000 7f8ec9675640 1 -- 172.21.15.57:0/4230032523 shutdown_connections 2023-10-19T14:01:55.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.774+0000 7f8ec9675640 1 -- 172.21.15.57:0/4230032523 wait complete. 2023-10-19T14:01:55.779 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:01:55.774+0000 7f551f17a040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-19T14:01:55.787 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-19T14:01:55.820 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:55.818+0000 7f2fcba11040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-19T14:01:55.868 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:55.866+0000 7f2fcba11040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-19T14:01:55.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.898+0000 7fa0fcb05640 1 -- 172.21.15.57:0/3562821739 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa0f80cf340 msgr2=0x7fa0f80d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:55.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.898+0000 7fa0fcb05640 1 --2- 172.21.15.57:0/3562821739 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa0f80cf340 0x7fa0f80d1910 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fa0ec009830 tx=0x7fa0ec02bcd0 comp rx=0 tx=0).stop 2023-10-19T14:01:55.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.898+0000 7fa0fcb05640 1 -- 172.21.15.57:0/3562821739 shutdown_connections 2023-10-19T14:01:55.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.898+0000 7fa0fcb05640 1 --2- 172.21.15.57:0/3562821739 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa0f815b2a0 0x7fa0f80cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:55.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.898+0000 7fa0fcb05640 1 --2- 172.21.15.57:0/3562821739 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa0f80cf340 0x7fa0f80d1910 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:55.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.898+0000 7fa0fcb05640 1 -- 172.21.15.57:0/3562821739 >> 172.21.15.57:0/3562821739 conn(0x7fa0f80d0d60 msgr2=0x7fa0f80d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:55.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.898+0000 7fa0fcb05640 1 -- 172.21.15.57:0/3562821739 shutdown_connections 2023-10-19T14:01:55.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.898+0000 7fa0fcb05640 1 -- 172.21.15.57:0/3562821739 wait complete. 2023-10-19T14:01:55.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.900+0000 7fa0fcb05640 1 Processor -- start 2023-10-19T14:01:55.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.900+0000 7fa0fcb05640 1 -- start start 2023-10-19T14:01:55.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.900+0000 7fa0fcb05640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa0f80ceb40 0x7fa0f80b42f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:55.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.900+0000 7fa0fcb05640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa0f80cf340 0x7fa0f8158690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:55.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.900+0000 7fa0fcb05640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0f815b2a0 0x7fa0f80b12f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:55.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.901+0000 7fa0fcb05640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f8158bd0 con 0x7fa0f815b2a0 2023-10-19T14:01:55.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.901+0000 7fa0fcb05640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f80b48d0 con 0x7fa0f80cf340 2023-10-19T14:01:55.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.901+0000 7fa0fcb05640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f80b4bb0 con 0x7fa0f80ceb40 2023-10-19T14:01:55.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.901+0000 7fa0f6d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0f815b2a0 0x7fa0f80b12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:55.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.901+0000 7fa0f6d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0f815b2a0 0x7fa0f80b12f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:60098/0 (socket says 172.21.15.57:60098) 2023-10-19T14:01:55.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.901+0000 7fa0f6d9d640 1 -- 172.21.15.57:0/1474212634 learned_addr learned my addr 172.21.15.57:0/1474212634 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:01:55.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.901+0000 7fa0f5d9b640 1 --2- 172.21.15.57:0/1474212634 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa0f80cf340 0x7fa0f8158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:55.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.901+0000 7fa0f659c640 1 --2- 172.21.15.57:0/1474212634 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa0f80ceb40 0x7fa0f80b42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:55.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.901+0000 7fa0f6d9d640 1 -- 172.21.15.57:0/1474212634 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa0f80ceb40 msgr2=0x7fa0f80b42f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:55.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.901+0000 7fa0f6d9d640 1 --2- 172.21.15.57:0/1474212634 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa0f80ceb40 0x7fa0f80b42f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:55.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.901+0000 7fa0f6d9d640 1 -- 172.21.15.57:0/1474212634 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa0f80cf340 msgr2=0x7fa0f8158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:55.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.901+0000 7fa0f6d9d640 1 --2- 172.21.15.57:0/1474212634 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa0f80cf340 0x7fa0f8158690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:55.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.901+0000 7fa0f6d9d640 1 -- 172.21.15.57:0/1474212634 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0e8009490 con 0x7fa0f815b2a0 2023-10-19T14:01:55.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.901+0000 7fa0f6d9d640 1 --2- 172.21.15.57:0/1474212634 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0f815b2a0 0x7fa0f80b12f0 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 crypto rx=0x7fa0f000cbc0 tx=0x7fa0f000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:55.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.901+0000 7fa0e6ffd640 1 -- 172.21.15.57:0/1474212634 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0f000aca0 con 0x7fa0f815b2a0 2023-10-19T14:01:55.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.901+0000 7fa0e6ffd640 1 -- 172.21.15.57:0/1474212634 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fa0f0003970 con 0x7fa0f815b2a0 2023-10-19T14:01:55.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.901+0000 7fa0fcb05640 1 -- 172.21.15.57:0/1474212634 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0ec009490 con 0x7fa0f815b2a0 2023-10-19T14:01:55.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.901+0000 7fa0e6ffd640 1 -- 172.21.15.57:0/1474212634 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0f0003d80 con 0x7fa0f815b2a0 2023-10-19T14:01:55.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.901+0000 7fa0fcb05640 1 -- 172.21.15.57:0/1474212634 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0f80b5170 con 0x7fa0f815b2a0 2023-10-19T14:01:55.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.903+0000 7fa0fcb05640 1 -- 172.21.15.57:0/1474212634 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0f80690e0 con 0x7fa0f815b2a0 2023-10-19T14:01:55.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.903+0000 7fa0e6ffd640 1 -- 172.21.15.57:0/1474212634 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 95) v1 ==== 89332+0+0 (secure 0 0 0) 0x7fa0f0004030 con 0x7fa0f815b2a0 2023-10-19T14:01:55.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.904+0000 7fa0e6ffd640 1 --2- 172.21.15.57:0/1474212634 >> [v2:172.21.15.179:6832/3795018457,v1:172.21.15.179:6833/3795018457] conn(0x7fa0d806af60 0x7fa0d806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:55.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.904+0000 7fa0e6ffd640 1 -- 172.21.15.57:0/1474212634 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 6401+0+0 (secure 0 0 0) 0x7fa0f008a410 con 0x7fa0f815b2a0 2023-10-19T14:01:55.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.904+0000 7fa0f659c640 1 -- 172.21.15.57:0/1474212634 >> [v2:172.21.15.179:6832/3795018457,v1:172.21.15.179:6833/3795018457] conn(0x7fa0d806af60 msgr2=0x7fa0d806d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/3795018457 2023-10-19T14:01:55.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.904+0000 7fa0f659c640 1 --2- 172.21.15.57:0/1474212634 >> [v2:172.21.15.179:6832/3795018457,v1:172.21.15.179:6833/3795018457] conn(0x7fa0d806af60 0x7fa0d806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T14:01:55.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.908+0000 7fa0e6ffd640 1 -- 172.21.15.57:0/1474212634 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa0f00551a0 con 0x7fa0f815b2a0 2023-10-19T14:01:55.968 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:55.966+0000 7f2fcba11040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-19T14:01:55.992 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.989+0000 7fa0e6ffd640 1 -- 172.21.15.57:0/1474212634 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mgrmap(e 96) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fa0f0088670 con 0x7fa0f815b2a0 2023-10-19T14:01:55.992 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.989+0000 7fa0e6ffd640 1 -- 172.21.15.57:0/1474212634 >> [v2:172.21.15.179:6832/3795018457,v1:172.21.15.179:6833/3795018457] conn(0x7fa0d806af60 msgr2=0x7fa0d806d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T14:01:55.992 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:55.989+0000 7fa0e6ffd640 1 --2- 172.21.15.57:0/1474212634 >> [v2:172.21.15.179:6832/3795018457,v1:172.21.15.179:6833/3795018457] conn(0x7fa0d806af60 0x7fa0d806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:56.045 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:56.042+0000 7f2f5149f640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:01:56.045 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:56.042+0000 7f2f5149f640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:01:56.045 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:56.042+0000 7f2f5149f640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:01:56.046 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:56.042+0000 7f2f5149f640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:01:56.046 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:56.042+0000 7f2f5149f640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:01:56.046 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:56.042+0000 7f2f4e499640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:01:56.046 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:56.042+0000 7f2f4e499640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:01:56.047 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:56.042+0000 7f2f4e499640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:01:56.047 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:56.042+0000 7f2f4e499640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:01:56.047 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:01:56.042+0000 7f2f4e499640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:01:56.066 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:01:56.064+0000 7f551f17a040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-19T14:01:56.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.091+0000 7fa0fcb05640 1 -- 172.21.15.57:0/1474212634 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa0f80d0b40 con 0x7fa0f815b2a0 2023-10-19T14:01:56.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.091+0000 7fa0e6ffd640 1 -- 172.21.15.57:0/1474212634 <== mon.0 v2:172.21.15.57:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v177) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa0f00726f0 con 0x7fa0f815b2a0 2023-10-19T14:01:56.094 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2023-10-19T14:01:56.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.093+0000 7fa0fcb05640 1 -- 172.21.15.57:0/1474212634 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0f815b2a0 msgr2=0x7fa0f80b12f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:56.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.093+0000 7fa0fcb05640 1 --2- 172.21.15.57:0/1474212634 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0f815b2a0 0x7fa0f80b12f0 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 crypto rx=0x7fa0f000cbc0 tx=0x7fa0f000a060 comp rx=0 tx=0).stop 2023-10-19T14:01:56.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.093+0000 7fa0fcb05640 1 -- 172.21.15.57:0/1474212634 shutdown_connections 2023-10-19T14:01:56.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.093+0000 7fa0fcb05640 1 --2- 172.21.15.57:0/1474212634 >> [v2:172.21.15.179:6832/3795018457,v1:172.21.15.179:6833/3795018457] conn(0x7fa0d806af60 0x7fa0d806d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:56.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.093+0000 7fa0fcb05640 1 --2- 172.21.15.57:0/1474212634 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0f815b2a0 0x7fa0f80b12f0 unknown :-1 s=CLOSED pgs=1964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:56.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.093+0000 7fa0fcb05640 1 --2- 172.21.15.57:0/1474212634 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa0f80cf340 0x7fa0f8158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:56.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.093+0000 7fa0fcb05640 1 --2- 172.21.15.57:0/1474212634 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa0f80ceb40 0x7fa0f80b42f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:56.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.093+0000 7fa0fcb05640 1 -- 172.21.15.57:0/1474212634 >> 172.21.15.57:0/1474212634 conn(0x7fa0f80d0d60 msgr2=0x7fa0f806a9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:56.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.093+0000 7fa0fcb05640 1 -- 172.21.15.57:0/1474212634 shutdown_connections 2023-10-19T14:01:56.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.093+0000 7fa0fcb05640 1 -- 172.21.15.57:0/1474212634 wait complete. 2023-10-19T14:01:56.106 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-19T14:01:56.119 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:01:56.117+0000 7f551f17a040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-19T14:01:56.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.216+0000 7f729567c640 1 -- 172.21.15.57:0/742509446 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f729006af00 msgr2=0x7f729006b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:56.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.216+0000 7f729567c640 1 --2- 172.21.15.57:0/742509446 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f729006af00 0x7f729006b2e0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f7284009860 tx=0x7f728402bd60 comp rx=0 tx=0).stop 2023-10-19T14:01:56.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.216+0000 7f729567c640 1 -- 172.21.15.57:0/742509446 shutdown_connections 2023-10-19T14:01:56.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.216+0000 7f729567c640 1 --2- 172.21.15.57:0/742509446 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f72900c96b0 0x7f72900c9b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:56.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.216+0000 7f729567c640 1 --2- 172.21.15.57:0/742509446 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f729006af00 0x7f729006b2e0 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:56.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.217+0000 7f729567c640 1 -- 172.21.15.57:0/742509446 >> 172.21.15.57:0/742509446 conn(0x7f7290069930 msgr2=0x7f7290069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:56.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.217+0000 7f729567c640 1 -- 172.21.15.57:0/742509446 shutdown_connections 2023-10-19T14:01:56.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.217+0000 7f729567c640 1 -- 172.21.15.57:0/742509446 wait complete. 2023-10-19T14:01:56.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.218+0000 7f729567c640 1 Processor -- start 2023-10-19T14:01:56.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.218+0000 7f729567c640 1 -- start start 2023-10-19T14:01:56.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.219+0000 7f729567c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f729006af00 0x7f72901efc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:56.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.219+0000 7f729567c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f72900ab500 0x7f72901ee9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:56.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.219+0000 7f729567c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f72900c96b0 0x7f7290152650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:56.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.219+0000 7f729567c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72901f03c0 con 0x7f72900ab500 2023-10-19T14:01:56.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.219+0000 7f729567c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72901f01c0 con 0x7f72900c96b0 2023-10-19T14:01:56.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.219+0000 7f729567c640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72901f0610 con 0x7f729006af00 2023-10-19T14:01:56.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.219+0000 7f728e7fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f72900ab500 0x7f72901ee9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:56.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.219+0000 7f728e7fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f72900ab500 0x7f72901ee9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:60114/0 (socket says 172.21.15.57:60114) 2023-10-19T14:01:56.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.219+0000 7f728e7fc640 1 -- 172.21.15.57:0/3005744856 learned_addr learned my addr 172.21.15.57:0/3005744856 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:01:56.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.219+0000 7f728effd640 1 --2- 172.21.15.57:0/3005744856 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f729006af00 0x7f72901efc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:56.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.219+0000 7f728f7fe640 1 --2- 172.21.15.57:0/3005744856 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f72900c96b0 0x7f7290152650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:56.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.219+0000 7f728e7fc640 1 -- 172.21.15.57:0/3005744856 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f729006af00 msgr2=0x7f72901efc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:56.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.219+0000 7f728e7fc640 1 --2- 172.21.15.57:0/3005744856 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f729006af00 0x7f72901efc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:56.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.219+0000 7f728e7fc640 1 -- 172.21.15.57:0/3005744856 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f72900c96b0 msgr2=0x7f7290152650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:56.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.219+0000 7f728e7fc640 1 --2- 172.21.15.57:0/3005744856 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f72900c96b0 0x7f7290152650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:56.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.219+0000 7f728e7fc640 1 -- 172.21.15.57:0/3005744856 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7280009490 con 0x7f72900ab500 2023-10-19T14:01:56.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.219+0000 7f728e7fc640 1 --2- 172.21.15.57:0/3005744856 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f72900ab500 0x7f72901ee9d0 secure :-1 s=READY pgs=1966 cs=0 l=1 rev1=1 crypto rx=0x7f728000ea20 tx=0x7f728000eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:56.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.219+0000 7f72777fe640 1 -- 172.21.15.57:0/3005744856 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7280009750 con 0x7f72900ab500 2023-10-19T14:01:56.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.220+0000 7f729567c640 1 -- 172.21.15.57:0/3005744856 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7284009490 con 0x7f72900ab500 2023-10-19T14:01:56.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.220+0000 7f72777fe640 1 -- 172.21.15.57:0/3005744856 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f728000ce40 con 0x7f72900ab500 2023-10-19T14:01:56.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.220+0000 7f72777fe640 1 -- 172.21.15.57:0/3005744856 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7280004280 con 0x7f72900ab500 2023-10-19T14:01:56.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.220+0000 7f729567c640 1 -- 172.21.15.57:0/3005744856 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72901f0bd0 con 0x7f72900ab500 2023-10-19T14:01:56.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.221+0000 7f729567c640 1 -- 172.21.15.57:0/3005744856 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72900cb6d0 con 0x7f72900ab500 2023-10-19T14:01:56.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.221+0000 7f72777fe640 1 -- 172.21.15.57:0/3005744856 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 96) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f72800035f0 con 0x7f72900ab500 2023-10-19T14:01:56.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.221+0000 7f72777fe640 1 -- 172.21.15.57:0/3005744856 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 6745+0+0 (secure 0 0 0) 0x7f7280014070 con 0x7f72900ab500 2023-10-19T14:01:56.228 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:01:56.225+0000 7f551f17a040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-19T14:01:56.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.226+0000 7f72777fe640 1 -- 172.21.15.57:0/3005744856 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f72800550b0 con 0x7f72900ab500 2023-10-19T14:01:56.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.401+0000 7f729567c640 1 -- 172.21.15.57:0/3005744856 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f72900690e0 con 0x7f72900ab500 2023-10-19T14:01:56.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.402+0000 7f72777fe640 1 -- 172.21.15.57:0/3005744856 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v177) v1 ==== 100+0+11 (secure 0 0 0) 0x7f72800586d0 con 0x7f72900ab500 2023-10-19T14:01:56.405 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 64 2023-10-19T14:01:56.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.403+0000 7f729567c640 1 -- 172.21.15.57:0/3005744856 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f72900ab500 msgr2=0x7f72901ee9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:56.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.403+0000 7f729567c640 1 --2- 172.21.15.57:0/3005744856 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f72900ab500 0x7f72901ee9d0 secure :-1 s=READY pgs=1966 cs=0 l=1 rev1=1 crypto rx=0x7f728000ea20 tx=0x7f728000eee0 comp rx=0 tx=0).stop 2023-10-19T14:01:56.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.403+0000 7f729567c640 1 -- 172.21.15.57:0/3005744856 shutdown_connections 2023-10-19T14:01:56.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.403+0000 7f729567c640 1 --2- 172.21.15.57:0/3005744856 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f72900c96b0 0x7f7290152650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:56.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.403+0000 7f729567c640 1 --2- 172.21.15.57:0/3005744856 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f72900ab500 0x7f72901ee9d0 unknown :-1 s=CLOSED pgs=1966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:56.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.403+0000 7f729567c640 1 --2- 172.21.15.57:0/3005744856 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f729006af00 0x7f72901efc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:56.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.403+0000 7f729567c640 1 -- 172.21.15.57:0/3005744856 >> 172.21.15.57:0/3005744856 conn(0x7f7290069930 msgr2=0x7f72900cafc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:56.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.404+0000 7f729567c640 1 -- 172.21.15.57:0/3005744856 shutdown_connections 2023-10-19T14:01:56.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.404+0000 7f729567c640 1 -- 172.21.15.57:0/3005744856 wait complete. 2023-10-19T14:01:56.417 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T14:01:56.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.563+0000 7fcc8f01a640 1 -- 172.21.15.57:0/3193198097 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcc880ab500 msgr2=0x7fcc880ab990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:56.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.563+0000 7fcc8f01a640 1 --2- 172.21.15.57:0/3193198097 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcc880ab500 0x7fcc880ab990 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 crypto rx=0x7fcc880ca260 tx=0x7fcc8002c2f0 comp rx=0 tx=0).stop 2023-10-19T14:01:56.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.564+0000 7fcc8f01a640 1 -- 172.21.15.57:0/3193198097 shutdown_connections 2023-10-19T14:01:56.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.564+0000 7fcc8f01a640 1 --2- 172.21.15.57:0/3193198097 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcc880ab500 0x7fcc880ab990 unknown :-1 s=CLOSED pgs=1967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:56.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.564+0000 7fcc8f01a640 1 --2- 172.21.15.57:0/3193198097 >> v2:172.21.15.179:3300/0 conn(0x7fcc880c9dc0 0x7fcc8806b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:56.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.564+0000 7fcc8f01a640 1 -- 172.21.15.57:0/3193198097 >> 172.21.15.57:0/3193198097 conn(0x7fcc88069930 msgr2=0x7fcc88069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:56.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.564+0000 7fcc8f01a640 1 -- 172.21.15.57:0/3193198097 shutdown_connections 2023-10-19T14:01:56.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.564+0000 7fcc8f01a640 1 -- 172.21.15.57:0/3193198097 wait complete. 2023-10-19T14:01:56.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.565+0000 7fcc8f01a640 1 Processor -- start 2023-10-19T14:01:56.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.565+0000 7fcc8f01a640 1 -- start start 2023-10-19T14:01:56.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.566+0000 7fcc8f01a640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcc8806af00 0x7fcc880afd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:56.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.566+0000 7fcc8f01a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcc880ab500 0x7fcc881f1620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:56.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.566+0000 7fcc8f01a640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcc880c9dc0 0x7fcc8813d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:56.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.566+0000 7fcc8f01a640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc880b0860 con 0x7fcc880ab500 2023-10-19T14:01:56.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.566+0000 7fcc8f01a640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc880b02c0 con 0x7fcc8806af00 2023-10-19T14:01:56.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.566+0000 7fcc8f01a640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc880b0580 con 0x7fcc880c9dc0 2023-10-19T14:01:56.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.566+0000 7fcc7ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcc880ab500 0x7fcc881f1620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:56.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.566+0000 7fcc7ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcc880ab500 0x7fcc881f1620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:60130/0 (socket says 172.21.15.57:60130) 2023-10-19T14:01:56.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.566+0000 7fcc7ffff640 1 -- 172.21.15.57:0/4018821058 learned_addr learned my addr 172.21.15.57:0/4018821058 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:01:56.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.566+0000 7fcc8d5b7640 1 --2- 172.21.15.57:0/4018821058 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcc880c9dc0 0x7fcc8813d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:56.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.566+0000 7fcc8cdb6640 1 --2- 172.21.15.57:0/4018821058 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcc8806af00 0x7fcc880afd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:56.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.566+0000 7fcc7ffff640 1 -- 172.21.15.57:0/4018821058 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcc880c9dc0 msgr2=0x7fcc8813d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:56.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.566+0000 7fcc7ffff640 1 --2- 172.21.15.57:0/4018821058 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcc880c9dc0 0x7fcc8813d670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:56.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.566+0000 7fcc7ffff640 1 -- 172.21.15.57:0/4018821058 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcc8806af00 msgr2=0x7fcc880afd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:56.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.566+0000 7fcc7ffff640 1 --2- 172.21.15.57:0/4018821058 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcc8806af00 0x7fcc880afd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:56.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.566+0000 7fcc7ffff640 1 -- 172.21.15.57:0/4018821058 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc80009900 con 0x7fcc880ab500 2023-10-19T14:01:56.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.566+0000 7fcc7ffff640 1 --2- 172.21.15.57:0/4018821058 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcc880ab500 0x7fcc881f1620 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 crypto rx=0x7fcc7800e790 tx=0x7fcc7800ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:56.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.566+0000 7fcc7d7fa640 1 -- 172.21.15.57:0/4018821058 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc78009580 con 0x7fcc880ab500 2023-10-19T14:01:56.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.566+0000 7fcc7d7fa640 1 -- 172.21.15.57:0/4018821058 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fcc7800cdb0 con 0x7fcc880ab500 2023-10-19T14:01:56.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.566+0000 7fcc8f01a640 1 -- 172.21.15.57:0/4018821058 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc8806f120 con 0x7fcc880ab500 2023-10-19T14:01:56.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.566+0000 7fcc7d7fa640 1 -- 172.21.15.57:0/4018821058 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc78010690 con 0x7fcc880ab500 2023-10-19T14:01:56.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.566+0000 7fcc8f01a640 1 -- 172.21.15.57:0/4018821058 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc8806f6e0 con 0x7fcc880ab500 2023-10-19T14:01:56.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.567+0000 7fcc8f01a640 1 -- 172.21.15.57:0/4018821058 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc8806c7e0 con 0x7fcc880ab500 2023-10-19T14:01:56.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.568+0000 7fcc7d7fa640 1 -- 172.21.15.57:0/4018821058 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 96) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fcc780107f0 con 0x7fcc880ab500 2023-10-19T14:01:56.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.568+0000 7fcc7d7fa640 1 -- 172.21.15.57:0/4018821058 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 6745+0+0 (secure 0 0 0) 0x7fcc78014070 con 0x7fcc880ab500 2023-10-19T14:01:56.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.572+0000 7fcc7d7fa640 1 -- 172.21.15.57:0/4018821058 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fcc78051ca0 con 0x7fcc880ab500 2023-10-19T14:01:56.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.783+0000 7fcc8f01a640 1 -- 172.21.15.57:0/4018821058 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcc8806c220 con 0x7fcc880ab500 2023-10-19T14:01:56.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.784+0000 7fcc7d7fa640 1 -- 172.21.15.57:0/4018821058 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+4920 (secure 0 0 0) 0x7fcc78054e10 con 0x7fcc880ab500 2023-10-19T14:01:56.787 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:01:56.787 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":70,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8126,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/1759316409","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1759316409},{"type":"v1","addr":"172.21.15.57:6835","nonce":1759316409}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67},{"gid":8129,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/2462599644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2462599644},{"type":"v1","addr":"172.21.15.57:6834","nonce":2462599644}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67},{"gid":9136,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/373220657","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":373220657},{"type":"v1","addr":"172.21.15.179:6836","nonce":373220657}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T14:01:38.503105+0000","modified":"2023-10-19T14:01:40.495529+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9139},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9139":{"gid":9139,"name":"d","rank":0,"incarnation":68,"state":"up:active","state_seq":3,"addr":"172.21.15.179:6837/1102326590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":1102326590},{"type":"v1","addr":"172.21.15.179:6837","nonce":1102326590}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2023-10-19T14:01:56.787 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 70 2023-10-19T14:01:56.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.785+0000 7fcc8f01a640 1 -- 172.21.15.57:0/4018821058 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcc880ab500 msgr2=0x7fcc881f1620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:56.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.785+0000 7fcc8f01a640 1 --2- 172.21.15.57:0/4018821058 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcc880ab500 0x7fcc881f1620 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 crypto rx=0x7fcc7800e790 tx=0x7fcc7800ec50 comp rx=0 tx=0).stop 2023-10-19T14:01:56.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.785+0000 7fcc8f01a640 1 -- 172.21.15.57:0/4018821058 shutdown_connections 2023-10-19T14:01:56.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.785+0000 7fcc8f01a640 1 --2- 172.21.15.57:0/4018821058 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcc880c9dc0 0x7fcc8813d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:56.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.785+0000 7fcc8f01a640 1 --2- 172.21.15.57:0/4018821058 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcc880ab500 0x7fcc881f1620 unknown :-1 s=CLOSED pgs=1968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:56.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.785+0000 7fcc8f01a640 1 --2- 172.21.15.57:0/4018821058 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcc8806af00 0x7fcc880afd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:56.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.785+0000 7fcc8f01a640 1 -- 172.21.15.57:0/4018821058 >> 172.21.15.57:0/4018821058 conn(0x7fcc88069930 msgr2=0x7fcc88069d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:56.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.786+0000 7fcc8f01a640 1 -- 172.21.15.57:0/4018821058 shutdown_connections 2023-10-19T14:01:56.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.786+0000 7fcc8f01a640 1 -- 172.21.15.57:0/4018821058 wait complete. 2023-10-19T14:01:56.798 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T14:01:56.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.905+0000 7fdab6e75640 1 -- 172.21.15.57:0/1307747098 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdab00ab500 msgr2=0x7fdab00ab990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:56.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.905+0000 7fdab6e75640 1 --2- 172.21.15.57:0/1307747098 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdab00ab500 0x7fdab00ab990 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7fdab00ca260 tx=0x7fdaa802c2f0 comp rx=0 tx=0).stop 2023-10-19T14:01:56.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.905+0000 7fdab6e75640 1 -- 172.21.15.57:0/1307747098 shutdown_connections 2023-10-19T14:01:56.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.905+0000 7fdab6e75640 1 --2- 172.21.15.57:0/1307747098 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdab00ab500 0x7fdab00ab990 unknown :-1 s=CLOSED pgs=1969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:56.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.905+0000 7fdab6e75640 1 --2- 172.21.15.57:0/1307747098 >> v2:172.21.15.179:3300/0 conn(0x7fdab00c9dc0 0x7fdab006b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:56.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.905+0000 7fdab6e75640 1 -- 172.21.15.57:0/1307747098 >> 172.21.15.57:0/1307747098 conn(0x7fdab0069930 msgr2=0x7fdab0069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:56.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.905+0000 7fdab6e75640 1 -- 172.21.15.57:0/1307747098 shutdown_connections 2023-10-19T14:01:56.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.905+0000 7fdab6e75640 1 -- 172.21.15.57:0/1307747098 wait complete. 2023-10-19T14:01:56.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.906+0000 7fdab6e75640 1 Processor -- start 2023-10-19T14:01:56.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.906+0000 7fdab6e75640 1 -- start start 2023-10-19T14:01:56.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.907+0000 7fdab6e75640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdab006af00 0x7fdab00aaba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:56.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.907+0000 7fdab6e75640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdab00ab500 0x7fdab013d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:56.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.907+0000 7fdab6e75640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdab00c9dc0 0x7fdab013a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:56.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.907+0000 7fdab6e75640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdab00af780 con 0x7fdab00ab500 2023-10-19T14:01:56.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.907+0000 7fdab6e75640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdab00aec00 con 0x7fdab00c9dc0 2023-10-19T14:01:56.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.907+0000 7fdab6e75640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdab00aeee0 con 0x7fdab006af00 2023-10-19T14:01:56.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.907+0000 7fdaa7fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdab00ab500 0x7fdab013d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:56.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.907+0000 7fdaa7fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdab00ab500 0x7fdab013d670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:47230/0 (socket says 172.21.15.57:47230) 2023-10-19T14:01:56.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.907+0000 7fdaa7fff640 1 -- 172.21.15.57:0/2291566337 learned_addr learned my addr 172.21.15.57:0/2291566337 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:01:56.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.907+0000 7fdab5412640 1 --2- 172.21.15.57:0/2291566337 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdab00c9dc0 0x7fdab013a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:56.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.907+0000 7fdab4c11640 1 --2- 172.21.15.57:0/2291566337 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdab006af00 0x7fdab00aaba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:56.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.907+0000 7fdaa7fff640 1 -- 172.21.15.57:0/2291566337 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdab006af00 msgr2=0x7fdab00aaba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:56.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.907+0000 7fdaa7fff640 1 --2- 172.21.15.57:0/2291566337 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdab006af00 0x7fdab00aaba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:56.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.907+0000 7fdaa7fff640 1 -- 172.21.15.57:0/2291566337 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdab00c9dc0 msgr2=0x7fdab013a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:56.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.907+0000 7fdaa7fff640 1 --2- 172.21.15.57:0/2291566337 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdab00c9dc0 0x7fdab013a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:56.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.907+0000 7fdaa7fff640 1 -- 172.21.15.57:0/2291566337 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaa8009900 con 0x7fdab00ab500 2023-10-19T14:01:56.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.907+0000 7fdaa7fff640 1 --2- 172.21.15.57:0/2291566337 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdab00ab500 0x7fdab013d670 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 crypto rx=0x7fdaa000d6d0 tx=0x7fdaa000db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:56.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.908+0000 7fdaa57fa640 1 -- 172.21.15.57:0/2291566337 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaa0005220 con 0x7fdab00ab500 2023-10-19T14:01:56.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.908+0000 7fdaa57fa640 1 -- 172.21.15.57:0/2291566337 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fdaa0005490 con 0x7fdab00ab500 2023-10-19T14:01:56.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.908+0000 7fdab6e75640 1 -- 172.21.15.57:0/2291566337 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdab00af1c0 con 0x7fdab00ab500 2023-10-19T14:01:56.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.908+0000 7fdaa57fa640 1 -- 172.21.15.57:0/2291566337 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaa0009a80 con 0x7fdab00ab500 2023-10-19T14:01:56.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.908+0000 7fdab6e75640 1 -- 172.21.15.57:0/2291566337 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdab00b0350 con 0x7fdab00ab500 2023-10-19T14:01:56.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.909+0000 7fdab6e75640 1 -- 172.21.15.57:0/2291566337 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdab00a9200 con 0x7fdab00ab500 2023-10-19T14:01:56.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.909+0000 7fdaa57fa640 1 -- 172.21.15.57:0/2291566337 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 96) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fdaa00047a0 con 0x7fdab00ab500 2023-10-19T14:01:56.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.909+0000 7fdaa57fa640 1 -- 172.21.15.57:0/2291566337 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 6745+0+0 (secure 0 0 0) 0x7fdaa0089750 con 0x7fdab00ab500 2023-10-19T14:01:56.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.914+0000 7fdaa57fa640 1 -- 172.21.15.57:0/2291566337 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fdab00a9200 con 0x7fdab00ab500 2023-10-19T14:01:56.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.988+0000 7fdaa57fa640 1 -- 172.21.15.57:0/2291566337 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mgrmap(e 97) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fdaa00543a0 con 0x7fdab00ab500 2023-10-19T14:01:56.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.988+0000 7fdaa57fa640 1 --2- 172.21.15.57:0/2291566337 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7fda9c070b80 0x7fda9c072f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:56.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.989+0000 7fdab4c11640 1 --2- 172.21.15.57:0/2291566337 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7fda9c070b80 0x7fda9c072f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:56.997 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:56.995+0000 7fdab4c11640 1 --2- 172.21.15.57:0/2291566337 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7fda9c070b80 0x7fda9c072f70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fdaac000f60 tx=0x7fdaac009040 comp rx=0 tx=0).ready entity=mgr.8315 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:57.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.094+0000 7fdab6e75640 1 -- 172.21.15.57:0/2291566337 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdab00a9200 con 0x7fdab00ab500 2023-10-19T14:01:57.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.095+0000 7fdaa57fa640 1 -- 172.21.15.57:0/2291566337 <== mon.0 v2:172.21.15.57:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v177) v1 ==== 74+0+17795 (secure 0 0 0) 0x7fdab00a9200 con 0x7fdab00ab500 2023-10-19T14:01:57.098 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:01:57.098 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":177,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T14:01:55.980646+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2023-10-19T14:01:36.799358+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"174","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_data","create_time":"2023-10-19T14:01:37.481213+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"175","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.179:0/3598128118":"2023-10-20T14:01:55.980598+0000","172.21.15.179:0/2108397676":"2023-10-20T14:01:55.980598+0000","172.21.15.179:6832/3795018457":"2023-10-20T14:01:55.980598+0000","172.21.15.179:0/1222555240":"2023-10-20T14:01:55.980598+0000","172.21.15.179:6833/3795018457":"2023-10-20T14:01:55.980598+0000","172.21.15.179:0/111884799":"2023-10-20T14:01:49.564546+0000","172.21.15.179:0/3699846367":"2023-10-20T14:01:55.980598+0000","172.21.15.179:0/966948837":"2023-10-20T14:01:49.564546+0000","172.21.15.179:0/1558797744":"2023-10-20T14:01:49.564546+0000","172.21.15.179:6833/2037224568":"2023-10-20T14:01:49.564546+0000","172.21.15.179:6832/2037224568":"2023-10-20T14:01:49.564546+0000","172.21.15.179:0/2965291378":"2023-10-20T14:01:49.564546+0000","172.21.15.179:0/472878921":"2023-10-20T14:01:49.564546+0000","172.21.15.179:0/3713016428":"2023-10-20T14:01:55.980598+0000","172.21.15.179:0/2930510955":"2023-10-20T14:01:55.980598+0000","172.21.15.179:0/2416623631":"2023-10-20T14:01:49.564546+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T14:01:57.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.097+0000 7fdab6e75640 1 -- 172.21.15.57:0/2291566337 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7fda9c070b80 msgr2=0x7fda9c072f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:57.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.097+0000 7fdab6e75640 1 --2- 172.21.15.57:0/2291566337 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7fda9c070b80 0x7fda9c072f70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fdaac000f60 tx=0x7fdaac009040 comp rx=0 tx=0).stop 2023-10-19T14:01:57.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.097+0000 7fdab6e75640 1 -- 172.21.15.57:0/2291566337 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdab00ab500 msgr2=0x7fdab013d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:57.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.097+0000 7fdab6e75640 1 --2- 172.21.15.57:0/2291566337 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdab00ab500 0x7fdab013d670 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 crypto rx=0x7fdaa000d6d0 tx=0x7fdaa000db90 comp rx=0 tx=0).stop 2023-10-19T14:01:57.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.097+0000 7fdab6e75640 1 -- 172.21.15.57:0/2291566337 shutdown_connections 2023-10-19T14:01:57.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.097+0000 7fdab6e75640 1 --2- 172.21.15.57:0/2291566337 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdab00c9dc0 0x7fdab013a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:57.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.097+0000 7fdab6e75640 1 --2- 172.21.15.57:0/2291566337 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7fda9c070b80 0x7fda9c072f70 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:57.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.097+0000 7fdab6e75640 1 --2- 172.21.15.57:0/2291566337 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdab00ab500 0x7fdab013d670 unknown :-1 s=CLOSED pgs=1970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:57.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.097+0000 7fdab6e75640 1 --2- 172.21.15.57:0/2291566337 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdab006af00 0x7fdab00aaba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:57.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.097+0000 7fdab6e75640 1 -- 172.21.15.57:0/2291566337 >> 172.21.15.57:0/2291566337 conn(0x7fdab0069930 msgr2=0x7fdab014f6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:57.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.097+0000 7fdab6e75640 1 -- 172.21.15.57:0/2291566337 shutdown_connections 2023-10-19T14:01:57.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.097+0000 7fdab6e75640 1 -- 172.21.15.57:0/2291566337 wait complete. 2023-10-19T14:01:57.110 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-19T14:01:57.111 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T14:01:57.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.217+0000 7fef30b5e640 1 -- 172.21.15.57:0/3704634390 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fef2c0c9dc0 msgr2=0x7fef2c06b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:57.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.217+0000 7fef30b5e640 1 --2- 172.21.15.57:0/3704634390 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fef2c0c9dc0 0x7fef2c06b2e0 secure :-1 s=READY pgs=1971 cs=0 l=1 rev1=1 crypto rx=0x7fef2c06d590 tx=0x7fef1402be40 comp rx=0 tx=0).stop 2023-10-19T14:01:57.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.217+0000 7fef30b5e640 1 -- 172.21.15.57:0/3704634390 shutdown_connections 2023-10-19T14:01:57.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.217+0000 7fef30b5e640 1 --2- 172.21.15.57:0/3704634390 >> v2:172.21.15.179:3300/0 conn(0x7fef2c0ab500 0x7fef2c0ab990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:57.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.217+0000 7fef30b5e640 1 --2- 172.21.15.57:0/3704634390 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fef2c0c9dc0 0x7fef2c06b2e0 unknown :-1 s=CLOSED pgs=1971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:57.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.217+0000 7fef30b5e640 1 -- 172.21.15.57:0/3704634390 >> 172.21.15.57:0/3704634390 conn(0x7fef2c069930 msgr2=0x7fef2c069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:57.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.217+0000 7fef30b5e640 1 -- 172.21.15.57:0/3704634390 shutdown_connections 2023-10-19T14:01:57.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.217+0000 7fef30b5e640 1 -- 172.21.15.57:0/3704634390 wait complete. 2023-10-19T14:01:57.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.219+0000 7fef30b5e640 1 Processor -- start 2023-10-19T14:01:57.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.219+0000 7fef30b5e640 1 -- start start 2023-10-19T14:01:57.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.219+0000 7fef30b5e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fef2c06af00 0x7fef2c1522f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:57.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.219+0000 7fef30b5e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fef2c0ab500 0x7fef2c150d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:57.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.219+0000 7fef30b5e640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fef2c0c9dc0 0x7fef2c1370e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:57.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.219+0000 7fef30b5e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef2c0af3b0 con 0x7fef2c0ab500 2023-10-19T14:01:57.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.219+0000 7fef30b5e640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef2c0ae7a0 con 0x7fef2c06af00 2023-10-19T14:01:57.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.219+0000 7fef30b5e640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef2c0aea80 con 0x7fef2c0c9dc0 2023-10-19T14:01:57.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.219+0000 7fef23fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fef2c0ab500 0x7fef2c150d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:57.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.219+0000 7fef23fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fef2c0ab500 0x7fef2c150d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:47248/0 (socket says 172.21.15.57:47248) 2023-10-19T14:01:57.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.219+0000 7fef23fff640 1 -- 172.21.15.57:0/1248789940 learned_addr learned my addr 172.21.15.57:0/1248789940 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:01:57.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.220+0000 7fef2a59c640 1 --2- 172.21.15.57:0/1248789940 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fef2c06af00 0x7fef2c1522f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:57.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.220+0000 7fef2ad9d640 1 --2- 172.21.15.57:0/1248789940 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fef2c0c9dc0 0x7fef2c1370e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:57.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.220+0000 7fef23fff640 1 -- 172.21.15.57:0/1248789940 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fef2c0c9dc0 msgr2=0x7fef2c1370e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:57.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.220+0000 7fef23fff640 1 --2- 172.21.15.57:0/1248789940 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fef2c0c9dc0 0x7fef2c1370e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:57.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.220+0000 7fef23fff640 1 -- 172.21.15.57:0/1248789940 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fef2c06af00 msgr2=0x7fef2c1522f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:57.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.220+0000 7fef23fff640 1 --2- 172.21.15.57:0/1248789940 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fef2c06af00 0x7fef2c1522f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:57.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.220+0000 7fef23fff640 1 -- 172.21.15.57:0/1248789940 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef14009560 con 0x7fef2c0ab500 2023-10-19T14:01:57.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.220+0000 7fef23fff640 1 --2- 172.21.15.57:0/1248789940 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fef2c0ab500 0x7fef2c150d40 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7fef14000c00 tx=0x7fef1402db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:57.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.220+0000 7fef22ffd640 1 -- 172.21.15.57:0/1248789940 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef14003800 con 0x7fef2c0ab500 2023-10-19T14:01:57.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.220+0000 7fef22ffd640 1 -- 172.21.15.57:0/1248789940 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fef14003a50 con 0x7fef2c0ab500 2023-10-19T14:01:57.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.220+0000 7fef30b5e640 1 -- 172.21.15.57:0/1248789940 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef2c0aed60 con 0x7fef2c0ab500 2023-10-19T14:01:57.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.220+0000 7fef22ffd640 1 -- 172.21.15.57:0/1248789940 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef14003e60 con 0x7fef2c0ab500 2023-10-19T14:01:57.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.220+0000 7fef30b5e640 1 -- 172.21.15.57:0/1248789940 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef2c0b0f50 con 0x7fef2c0ab500 2023-10-19T14:01:57.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.221+0000 7fef30b5e640 1 -- 172.21.15.57:0/1248789940 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef2c0b10e0 con 0x7fef2c0ab500 2023-10-19T14:01:57.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.222+0000 7fef22ffd640 1 -- 172.21.15.57:0/1248789940 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 97) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fef1403e450 con 0x7fef2c0ab500 2023-10-19T14:01:57.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.222+0000 7fef22ffd640 1 --2- 172.21.15.57:0/1248789940 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7fef0c06ac90 0x7fef0c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:57.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.222+0000 7fef22ffd640 1 -- 172.21.15.57:0/1248789940 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 6745+0+0 (secure 0 0 0) 0x7fef140ac070 con 0x7fef2c0ab500 2023-10-19T14:01:57.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.222+0000 7fef2a59c640 1 --2- 172.21.15.57:0/1248789940 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7fef0c06ac90 0x7fef0c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:57.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.223+0000 7fef2a59c640 1 --2- 172.21.15.57:0/1248789940 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7fef0c06ac90 0x7fef0c06d150 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fef2c069fc0 tx=0x7fef1c009040 comp rx=0 tx=0).ready entity=mgr.8315 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:57.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.226+0000 7fef22ffd640 1 -- 172.21.15.57:0/1248789940 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fef14077100 con 0x7fef2c0ab500 2023-10-19T14:01:57.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.438+0000 7fef30b5e640 1 -- 172.21.15.57:0/1248789940 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fef2c0690e0 con 0x7fef2c0ab500 2023-10-19T14:01:57.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.438+0000 7fef22ffd640 1 -- 172.21.15.57:0/1248789940 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+4920 (secure 0 0 0) 0x7fef14042020 con 0x7fef2c0ab500 2023-10-19T14:01:57.441 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:01:57.442 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":70,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8126,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/1759316409","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1759316409},{"type":"v1","addr":"172.21.15.57:6835","nonce":1759316409}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67},{"gid":8129,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/2462599644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2462599644},{"type":"v1","addr":"172.21.15.57:6834","nonce":2462599644}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67},{"gid":9136,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/373220657","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":373220657},{"type":"v1","addr":"172.21.15.179:6836","nonce":373220657}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T14:01:38.503105+0000","modified":"2023-10-19T14:01:40.495529+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9139},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9139":{"gid":9139,"name":"d","rank":0,"incarnation":68,"state":"up:active","state_seq":3,"addr":"172.21.15.179:6837/1102326590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":1102326590},{"type":"v1","addr":"172.21.15.179:6837","nonce":1102326590}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2023-10-19T14:01:57.442 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 70 2023-10-19T14:01:57.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.440+0000 7fef30b5e640 1 -- 172.21.15.57:0/1248789940 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7fef0c06ac90 msgr2=0x7fef0c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:57.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.440+0000 7fef30b5e640 1 --2- 172.21.15.57:0/1248789940 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7fef0c06ac90 0x7fef0c06d150 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fef2c069fc0 tx=0x7fef1c009040 comp rx=0 tx=0).stop 2023-10-19T14:01:57.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.440+0000 7fef30b5e640 1 -- 172.21.15.57:0/1248789940 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fef2c0ab500 msgr2=0x7fef2c150d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:57.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.440+0000 7fef30b5e640 1 --2- 172.21.15.57:0/1248789940 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fef2c0ab500 0x7fef2c150d40 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7fef14000c00 tx=0x7fef1402db10 comp rx=0 tx=0).stop 2023-10-19T14:01:57.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.440+0000 7fef30b5e640 1 -- 172.21.15.57:0/1248789940 shutdown_connections 2023-10-19T14:01:57.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.440+0000 7fef30b5e640 1 --2- 172.21.15.57:0/1248789940 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fef2c0c9dc0 0x7fef2c1370e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:57.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.440+0000 7fef30b5e640 1 --2- 172.21.15.57:0/1248789940 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7fef0c06ac90 0x7fef0c06d150 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:57.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.440+0000 7fef30b5e640 1 --2- 172.21.15.57:0/1248789940 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fef2c0ab500 0x7fef2c150d40 unknown :-1 s=CLOSED pgs=1972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:57.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.440+0000 7fef30b5e640 1 --2- 172.21.15.57:0/1248789940 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fef2c06af00 0x7fef2c1522f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:57.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.440+0000 7fef30b5e640 1 -- 172.21.15.57:0/1248789940 >> 172.21.15.57:0/1248789940 conn(0x7fef2c069930 msgr2=0x7fef2c0c9680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:57.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.440+0000 7fef30b5e640 1 -- 172.21.15.57:0/1248789940 shutdown_connections 2023-10-19T14:01:57.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.440+0000 7fef30b5e640 1 -- 172.21.15.57:0/1248789940 wait complete. 2023-10-19T14:01:57.453 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T14:01:57.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.560+0000 7f25a9275640 1 -- 172.21.15.57:0/1064630238 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f25a40c9b60 msgr2=0x7f25a4155200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:57.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.560+0000 7f25a9275640 1 --2- 172.21.15.57:0/1064630238 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f25a40c9b60 0x7f25a4155200 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f259c009be0 tx=0x7f259c02c020 comp rx=0 tx=0).stop 2023-10-19T14:01:57.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.560+0000 7f25a9275640 1 -- 172.21.15.57:0/1064630238 shutdown_connections 2023-10-19T14:01:57.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.560+0000 7f25a9275640 1 --2- 172.21.15.57:0/1064630238 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f25a40c9b60 0x7f25a4155200 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:57.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.560+0000 7f25a9275640 1 -- 172.21.15.57:0/1064630238 >> 172.21.15.57:0/1064630238 conn(0x7f25a4069930 msgr2=0x7f25a4069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:57.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.560+0000 7f25a9275640 1 -- 172.21.15.57:0/1064630238 shutdown_connections 2023-10-19T14:01:57.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.560+0000 7f25a9275640 1 -- 172.21.15.57:0/1064630238 wait complete. 2023-10-19T14:01:57.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.561+0000 7f25a9275640 1 Processor -- start 2023-10-19T14:01:57.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.561+0000 7f25a9275640 1 -- start start 2023-10-19T14:01:57.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.562+0000 7f25a9275640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25a40b1920 0x7f25a40ad070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:57.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.562+0000 7f25a9275640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f25a40c96b0 0x7f25a414a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:57.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.562+0000 7f25a9275640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f25a40c9b60 0x7f25a40adb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:57.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.562+0000 7f25a9275640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25a41535d0 con 0x7f25a40b1920 2023-10-19T14:01:57.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.562+0000 7f25a9275640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25a4153840 con 0x7f25a40c9b60 2023-10-19T14:01:57.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.562+0000 7f25a9275640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25a4153b20 con 0x7f25a40c96b0 2023-10-19T14:01:57.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.562+0000 7f25a2ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25a40b1920 0x7f25a40ad070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:57.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.562+0000 7f25a2ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25a40b1920 0x7f25a40ad070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:47258/0 (socket says 172.21.15.57:47258) 2023-10-19T14:01:57.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.562+0000 7f25a2ffd640 1 -- 172.21.15.57:0/3727492481 learned_addr learned my addr 172.21.15.57:0/3727492481 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:01:57.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.562+0000 7f259a7fc640 1 --2- 172.21.15.57:0/3727492481 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f25a40c96b0 0x7f25a414a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:57.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.562+0000 7f25a37fe640 1 --2- 172.21.15.57:0/3727492481 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f25a40c9b60 0x7f25a40adb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:57.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.562+0000 7f25a2ffd640 1 -- 172.21.15.57:0/3727492481 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f25a40c96b0 msgr2=0x7f25a414a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:57.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.562+0000 7f25a2ffd640 1 --2- 172.21.15.57:0/3727492481 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f25a40c96b0 0x7f25a414a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:57.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.562+0000 7f25a2ffd640 1 -- 172.21.15.57:0/3727492481 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f25a40c9b60 msgr2=0x7f25a40adb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:57.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.562+0000 7f25a2ffd640 1 --2- 172.21.15.57:0/3727492481 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f25a40c9b60 0x7f25a40adb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:57.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.562+0000 7f25a2ffd640 1 -- 172.21.15.57:0/3727492481 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f259c009840 con 0x7f25a40b1920 2023-10-19T14:01:57.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.562+0000 7f25a37fe640 1 --2- 172.21.15.57:0/3727492481 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f25a40c9b60 0x7f25a40adb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:01:57.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.563+0000 7f25a2ffd640 1 --2- 172.21.15.57:0/3727492481 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25a40b1920 0x7f25a40ad070 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 crypto rx=0x7f259400c700 tx=0x7f259400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:57.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.563+0000 7f259bfff640 1 -- 172.21.15.57:0/3727492481 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f259400a840 con 0x7f25a40b1920 2023-10-19T14:01:57.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.563+0000 7f259bfff640 1 -- 172.21.15.57:0/3727492481 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f259400aab0 con 0x7f25a40b1920 2023-10-19T14:01:57.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.563+0000 7f25a9275640 1 -- 172.21.15.57:0/3727492481 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25a4070300 con 0x7f25a40b1920 2023-10-19T14:01:57.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.563+0000 7f259bfff640 1 -- 172.21.15.57:0/3727492481 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2594012420 con 0x7f25a40b1920 2023-10-19T14:01:57.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.563+0000 7f25a9275640 1 -- 172.21.15.57:0/3727492481 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25a414a870 con 0x7f25a40b1920 2023-10-19T14:01:57.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.564+0000 7f25a9275640 1 -- 172.21.15.57:0/3727492481 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25a4070dc0 con 0x7f25a40b1920 2023-10-19T14:01:57.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.564+0000 7f259bfff640 1 -- 172.21.15.57:0/3727492481 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 97) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f259401b400 con 0x7f25a40b1920 2023-10-19T14:01:57.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.565+0000 7f259bfff640 1 --2- 172.21.15.57:0/3727492481 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f258406ac90 0x7f258406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:57.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.565+0000 7f259bfff640 1 -- 172.21.15.57:0/3727492481 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 6745+0+0 (secure 0 0 0) 0x7f2594089d20 con 0x7f25a40b1920 2023-10-19T14:01:57.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.565+0000 7f259a7fc640 1 --2- 172.21.15.57:0/3727492481 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f258406ac90 0x7f258406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:57.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.565+0000 7f259a7fc640 1 --2- 172.21.15.57:0/3727492481 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f258406ac90 0x7f258406d150 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f25a406ca20 tx=0x7f258c009040 comp rx=0 tx=0).ready entity=mgr.8315 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:57.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.569+0000 7f259bfff640 1 -- 172.21.15.57:0/3727492481 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2594054bb0 con 0x7f25a40b1920 2023-10-19T14:01:57.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.782+0000 7f25a9275640 1 -- 172.21.15.57:0/3727492481 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f25a40690e0 con 0x7f25a40b1920 2023-10-19T14:01:57.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.782+0000 7f259bfff640 1 -- 172.21.15.57:0/3727492481 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+4920 (secure 0 0 0) 0x7f25940581d0 con 0x7f25a40b1920 2023-10-19T14:01:57.785 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:01:57.785 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":70,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8126,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/1759316409","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1759316409},{"type":"v1","addr":"172.21.15.57:6835","nonce":1759316409}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67},{"gid":8129,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/2462599644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2462599644},{"type":"v1","addr":"172.21.15.57:6834","nonce":2462599644}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67},{"gid":9136,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/373220657","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":373220657},{"type":"v1","addr":"172.21.15.179:6836","nonce":373220657}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T14:01:38.503105+0000","modified":"2023-10-19T14:01:40.495529+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9139},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9139":{"gid":9139,"name":"d","rank":0,"incarnation":68,"state":"up:active","state_seq":3,"addr":"172.21.15.179:6837/1102326590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":1102326590},{"type":"v1","addr":"172.21.15.179:6837","nonce":1102326590}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2023-10-19T14:01:57.786 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 70 2023-10-19T14:01:57.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.784+0000 7f25a9275640 1 -- 172.21.15.57:0/3727492481 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f258406ac90 msgr2=0x7f258406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:57.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.784+0000 7f25a9275640 1 --2- 172.21.15.57:0/3727492481 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f258406ac90 0x7f258406d150 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f25a406ca20 tx=0x7f258c009040 comp rx=0 tx=0).stop 2023-10-19T14:01:57.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.784+0000 7f25a9275640 1 -- 172.21.15.57:0/3727492481 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25a40b1920 msgr2=0x7f25a40ad070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:57.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.784+0000 7f25a9275640 1 --2- 172.21.15.57:0/3727492481 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25a40b1920 0x7f25a40ad070 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 crypto rx=0x7f259400c700 tx=0x7f259400cbc0 comp rx=0 tx=0).stop 2023-10-19T14:01:57.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.784+0000 7f25a9275640 1 -- 172.21.15.57:0/3727492481 shutdown_connections 2023-10-19T14:01:57.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.784+0000 7f25a9275640 1 --2- 172.21.15.57:0/3727492481 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f25a40c9b60 0x7f25a40adb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:57.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.784+0000 7f25a9275640 1 --2- 172.21.15.57:0/3727492481 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f25a40c96b0 0x7f25a414a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:57.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.784+0000 7f25a9275640 1 --2- 172.21.15.57:0/3727492481 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f258406ac90 0x7f258406d150 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:57.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.784+0000 7f25a9275640 1 --2- 172.21.15.57:0/3727492481 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25a40b1920 0x7f25a40ad070 unknown :-1 s=CLOSED pgs=1974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:57.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.784+0000 7f25a9275640 1 -- 172.21.15.57:0/3727492481 >> 172.21.15.57:0/3727492481 conn(0x7f25a4069930 msgr2=0x7f25a40cb6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:57.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.784+0000 7f25a9275640 1 -- 172.21.15.57:0/3727492481 shutdown_connections 2023-10-19T14:01:57.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.784+0000 7f25a9275640 1 -- 172.21.15.57:0/3727492481 wait complete. 2023-10-19T14:01:57.797 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T14:01:57.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.903+0000 7f62b4432640 1 -- 172.21.15.57:0/2334710860 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f62ac0cb980 msgr2=0x7f62ac13db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:57.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.903+0000 7f62b4432640 1 --2- 172.21.15.57:0/2334710860 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f62ac0cb980 0x7f62ac13db80 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7f62ac06d4f0 tx=0x7f62a402bd70 comp rx=0 tx=0).stop 2023-10-19T14:01:57.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.903+0000 7f62b4432640 1 -- 172.21.15.57:0/2334710860 shutdown_connections 2023-10-19T14:01:57.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.903+0000 7f62b4432640 1 --2- 172.21.15.57:0/2334710860 >> v2:172.21.15.179:3300/0 conn(0x7f62ac070350 0x7f62ac070780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:57.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.903+0000 7f62b4432640 1 --2- 172.21.15.57:0/2334710860 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f62ac0cb980 0x7f62ac13db80 unknown :-1 s=CLOSED pgs=1976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:57.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.903+0000 7f62b4432640 1 -- 172.21.15.57:0/2334710860 >> 172.21.15.57:0/2334710860 conn(0x7f62ac069930 msgr2=0x7f62ac069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:57.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.903+0000 7f62b4432640 1 -- 172.21.15.57:0/2334710860 shutdown_connections 2023-10-19T14:01:57.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.903+0000 7f62b4432640 1 -- 172.21.15.57:0/2334710860 wait complete. 2023-10-19T14:01:57.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.904+0000 7f62b4432640 1 Processor -- start 2023-10-19T14:01:57.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.904+0000 7f62b4432640 1 -- start start 2023-10-19T14:01:57.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.905+0000 7f62b4432640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f62ac070350 0x7f62ac0ad3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:57.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.905+0000 7f62b4432640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f62ac0cb980 0x7f62ac1f3350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:57.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.905+0000 7f62b4432640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f62ac0ca870 0x7f62ac154980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:57.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.905+0000 7f62b4432640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62ac0ad920 con 0x7f62ac0cb980 2023-10-19T14:01:57.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.905+0000 7f62b4432640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62ac0adb30 con 0x7f62ac070350 2023-10-19T14:01:57.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.905+0000 7f62b4432640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62ac0af160 con 0x7f62ac0ca870 2023-10-19T14:01:57.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.905+0000 7f62b19cd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f62ac0cb980 0x7f62ac1f3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:57.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.905+0000 7f62b19cd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f62ac0cb980 0x7f62ac1f3350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:47288/0 (socket says 172.21.15.57:47288) 2023-10-19T14:01:57.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.905+0000 7f62b19cd640 1 -- 172.21.15.57:0/576520508 learned_addr learned my addr 172.21.15.57:0/576520508 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:01:57.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.905+0000 7f62b19cd640 1 -- 172.21.15.57:0/576520508 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f62ac0ca870 msgr2=0x7f62ac154980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:57.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.905+0000 7f62b19cd640 1 --2- 172.21.15.57:0/576520508 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f62ac0ca870 0x7f62ac154980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:57.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.905+0000 7f62b19cd640 1 -- 172.21.15.57:0/576520508 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f62ac070350 msgr2=0x7f62ac0ad3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:57.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.905+0000 7f62b21ce640 1 --2- 172.21.15.57:0/576520508 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f62ac070350 0x7f62ac0ad3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:57.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.905+0000 7f62b19cd640 1 --2- 172.21.15.57:0/576520508 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f62ac070350 0x7f62ac0ad3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:57.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.905+0000 7f62b19cd640 1 -- 172.21.15.57:0/576520508 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62a4009490 con 0x7f62ac0cb980 2023-10-19T14:01:57.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.905+0000 7f62b19cd640 1 --2- 172.21.15.57:0/576520508 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f62ac0cb980 0x7f62ac1f3350 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 crypto rx=0x7f62a000c3f0 tx=0x7f62a000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:57.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.906+0000 7f62aaffd640 1 -- 172.21.15.57:0/576520508 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62a0011790 con 0x7f62ac0cb980 2023-10-19T14:01:57.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.906+0000 7f62aaffd640 1 -- 172.21.15.57:0/576520508 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f62a00119e0 con 0x7f62ac0cb980 2023-10-19T14:01:57.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.906+0000 7f62b4432640 1 -- 172.21.15.57:0/576520508 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62ac1f3890 con 0x7f62ac0cb980 2023-10-19T14:01:57.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.906+0000 7f62aaffd640 1 -- 172.21.15.57:0/576520508 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62a0011dd0 con 0x7f62ac0cb980 2023-10-19T14:01:57.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.906+0000 7f62b4432640 1 -- 172.21.15.57:0/576520508 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62ac0aeb10 con 0x7f62ac0cb980 2023-10-19T14:01:57.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.907+0000 7f62b4432640 1 -- 172.21.15.57:0/576520508 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62ac0690e0 con 0x7f62ac0cb980 2023-10-19T14:01:57.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.907+0000 7f62aaffd640 1 -- 172.21.15.57:0/576520508 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 97) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f62a001c460 con 0x7f62ac0cb980 2023-10-19T14:01:57.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.908+0000 7f62aaffd640 1 --2- 172.21.15.57:0/576520508 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f629806acc0 0x7f629806d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:57.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.908+0000 7f62aaffd640 1 -- 172.21.15.57:0/576520508 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 6745+0+0 (secure 0 0 0) 0x7f62a0089730 con 0x7f62ac0cb980 2023-10-19T14:01:57.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.908+0000 7f62b21ce640 1 --2- 172.21.15.57:0/576520508 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f629806acc0 0x7f629806d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:57.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.908+0000 7f62b21ce640 1 --2- 172.21.15.57:0/576520508 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f629806acc0 0x7f629806d180 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f62a4036000 tx=0x7f62a4004730 comp rx=0 tx=0).ready entity=mgr.8315 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:57.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:57.912+0000 7f62aaffd640 1 -- 172.21.15.57:0/576520508 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f62a0054710 con 0x7f62ac0cb980 2023-10-19T14:01:58.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.095+0000 7f62b4432640 1 -- 172.21.15.57:0/576520508 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f62ac0af2a0 con 0x7f62ac0cb980 2023-10-19T14:01:58.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.096+0000 7f62aaffd640 1 -- 172.21.15.57:0/576520508 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v177) v1 ==== 74+0+17795 (secure 0 0 0) 0x7f62a0057d30 con 0x7f62ac0cb980 2023-10-19T14:01:58.099 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:01:58.099 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":177,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T14:01:55.980646+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2023-10-19T14:01:36.799358+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"174","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_data","create_time":"2023-10-19T14:01:37.481213+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"175","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.179:0/3598128118":"2023-10-20T14:01:55.980598+0000","172.21.15.179:0/2108397676":"2023-10-20T14:01:55.980598+0000","172.21.15.179:6832/3795018457":"2023-10-20T14:01:55.980598+0000","172.21.15.179:0/1222555240":"2023-10-20T14:01:55.980598+0000","172.21.15.179:6833/3795018457":"2023-10-20T14:01:55.980598+0000","172.21.15.179:0/111884799":"2023-10-20T14:01:49.564546+0000","172.21.15.179:0/3699846367":"2023-10-20T14:01:55.980598+0000","172.21.15.179:0/966948837":"2023-10-20T14:01:49.564546+0000","172.21.15.179:0/1558797744":"2023-10-20T14:01:49.564546+0000","172.21.15.179:6833/2037224568":"2023-10-20T14:01:49.564546+0000","172.21.15.179:6832/2037224568":"2023-10-20T14:01:49.564546+0000","172.21.15.179:0/2965291378":"2023-10-20T14:01:49.564546+0000","172.21.15.179:0/472878921":"2023-10-20T14:01:49.564546+0000","172.21.15.179:0/3713016428":"2023-10-20T14:01:55.980598+0000","172.21.15.179:0/2930510955":"2023-10-20T14:01:55.980598+0000","172.21.15.179:0/2416623631":"2023-10-20T14:01:49.564546+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T14:01:58.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.098+0000 7f62b4432640 1 -- 172.21.15.57:0/576520508 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f629806acc0 msgr2=0x7f629806d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:58.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.098+0000 7f62b4432640 1 --2- 172.21.15.57:0/576520508 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f629806acc0 0x7f629806d180 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f62a4036000 tx=0x7f62a4004730 comp rx=0 tx=0).stop 2023-10-19T14:01:58.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.098+0000 7f62b4432640 1 -- 172.21.15.57:0/576520508 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f62ac0cb980 msgr2=0x7f62ac1f3350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:58.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.098+0000 7f62b4432640 1 --2- 172.21.15.57:0/576520508 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f62ac0cb980 0x7f62ac1f3350 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 crypto rx=0x7f62a000c3f0 tx=0x7f62a000c8b0 comp rx=0 tx=0).stop 2023-10-19T14:01:58.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.098+0000 7f62b4432640 1 -- 172.21.15.57:0/576520508 shutdown_connections 2023-10-19T14:01:58.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.098+0000 7f62b4432640 1 --2- 172.21.15.57:0/576520508 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f62ac0ca870 0x7f62ac154980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:58.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.098+0000 7f62b4432640 1 --2- 172.21.15.57:0/576520508 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f629806acc0 0x7f629806d180 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:58.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.098+0000 7f62b4432640 1 --2- 172.21.15.57:0/576520508 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f62ac0cb980 0x7f62ac1f3350 unknown :-1 s=CLOSED pgs=1977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:58.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.098+0000 7f62b4432640 1 --2- 172.21.15.57:0/576520508 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f62ac070350 0x7f62ac0ad3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:58.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.098+0000 7f62b4432640 1 -- 172.21.15.57:0/576520508 >> 172.21.15.57:0/576520508 conn(0x7f62ac069930 msgr2=0x7f62ac158120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:58.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.098+0000 7f62b4432640 1 -- 172.21.15.57:0/576520508 shutdown_connections 2023-10-19T14:01:58.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.098+0000 7f62b4432640 1 -- 172.21.15.57:0/576520508 wait complete. 2023-10-19T14:01:58.112 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-19T14:01:58.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.219+0000 7f7b98694640 1 -- 172.21.15.57:0/2414442404 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7b900cbd20 msgr2=0x7f7b90158300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:58.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.219+0000 7f7b98694640 1 --2- 172.21.15.57:0/2414442404 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7b900cbd20 0x7f7b90158300 secure :-1 s=READY pgs=1978 cs=0 l=1 rev1=1 crypto rx=0x7f7b900cc490 tx=0x7f7b8c02bda0 comp rx=0 tx=0).stop 2023-10-19T14:01:58.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.219+0000 7f7b98694640 1 -- 172.21.15.57:0/2414442404 shutdown_connections 2023-10-19T14:01:58.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.219+0000 7f7b98694640 1 --2- 172.21.15.57:0/2414442404 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7b900b0530 0x7f7b900b09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:58.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.219+0000 7f7b98694640 1 --2- 172.21.15.57:0/2414442404 >> v2:172.21.15.179:3300/0 conn(0x7f7b900afa20 0x7f7b900705a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:58.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.219+0000 7f7b98694640 1 --2- 172.21.15.57:0/2414442404 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7b900cbd20 0x7f7b90158300 unknown :-1 s=CLOSED pgs=1978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:58.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.219+0000 7f7b98694640 1 -- 172.21.15.57:0/2414442404 >> 172.21.15.57:0/2414442404 conn(0x7f7b90069930 msgr2=0x7f7b90069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:58.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.219+0000 7f7b98694640 1 -- 172.21.15.57:0/2414442404 shutdown_connections 2023-10-19T14:01:58.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.219+0000 7f7b98694640 1 -- 172.21.15.57:0/2414442404 wait complete. 2023-10-19T14:01:58.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.221+0000 7f7b98694640 1 Processor -- start 2023-10-19T14:01:58.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.221+0000 7f7b98694640 1 -- start start 2023-10-19T14:01:58.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.221+0000 7f7b98694640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7b900afa20 0x7f7b90141110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:58.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.221+0000 7f7b98694640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7b900b0530 0x7f7b9013d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:58.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.221+0000 7f7b98694640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7b900cbd20 0x7f7b9013d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:58.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.221+0000 7f7b98694640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b901406d0 con 0x7f7b900cbd20 2023-10-19T14:01:58.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.221+0000 7f7b98694640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b9013fe30 con 0x7f7b900afa20 2023-10-19T14:01:58.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.221+0000 7f7b98694640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b90140110 con 0x7f7b900b0530 2023-10-19T14:01:58.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.221+0000 7f7b96c31640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7b900cbd20 0x7f7b9013d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:58.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.222+0000 7f7b96c31640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7b900cbd20 0x7f7b9013d8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:47292/0 (socket says 172.21.15.57:47292) 2023-10-19T14:01:58.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.222+0000 7f7b96c31640 1 -- 172.21.15.57:0/2757609609 learned_addr learned my addr 172.21.15.57:0/2757609609 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:01:58.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.222+0000 7f7b95c2f640 1 --2- 172.21.15.57:0/2757609609 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7b900b0530 0x7f7b9013d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:58.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.222+0000 7f7b96430640 1 --2- 172.21.15.57:0/2757609609 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7b900afa20 0x7f7b90141110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:58.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.222+0000 7f7b96c31640 1 -- 172.21.15.57:0/2757609609 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7b900b0530 msgr2=0x7f7b9013d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:58.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.222+0000 7f7b96c31640 1 --2- 172.21.15.57:0/2757609609 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7b900b0530 0x7f7b9013d360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:58.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.222+0000 7f7b96c31640 1 -- 172.21.15.57:0/2757609609 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7b900afa20 msgr2=0x7f7b90141110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:58.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.222+0000 7f7b96c31640 1 --2- 172.21.15.57:0/2757609609 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7b900afa20 0x7f7b90141110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:58.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.222+0000 7f7b96c31640 1 -- 172.21.15.57:0/2757609609 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b8c009560 con 0x7f7b900cbd20 2023-10-19T14:01:58.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.222+0000 7f7b96430640 1 --2- 172.21.15.57:0/2757609609 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7b900afa20 0x7f7b90141110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:01:58.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.222+0000 7f7b96c31640 1 --2- 172.21.15.57:0/2757609609 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7b900cbd20 0x7f7b9013d8a0 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 crypto rx=0x7f7b8800eaa0 tx=0x7f7b8800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:58.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.222+0000 7f7b86ffd640 1 -- 172.21.15.57:0/2757609609 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b8800cdb0 con 0x7f7b900cbd20 2023-10-19T14:01:58.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.222+0000 7f7b86ffd640 1 -- 172.21.15.57:0/2757609609 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f7b88004b20 con 0x7f7b900cbd20 2023-10-19T14:01:58.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.222+0000 7f7b98694640 1 -- 172.21.15.57:0/2757609609 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b901403f0 con 0x7f7b900cbd20 2023-10-19T14:01:58.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.222+0000 7f7b86ffd640 1 -- 172.21.15.57:0/2757609609 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b88005020 con 0x7f7b900cbd20 2023-10-19T14:01:58.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.222+0000 7f7b98694640 1 -- 172.21.15.57:0/2757609609 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b90141650 con 0x7f7b900cbd20 2023-10-19T14:01:58.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.223+0000 7f7b98694640 1 -- 172.21.15.57:0/2757609609 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b90141a30 con 0x7f7b900cbd20 2023-10-19T14:01:58.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.224+0000 7f7b86ffd640 1 -- 172.21.15.57:0/2757609609 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 97) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f7b88005180 con 0x7f7b900cbd20 2023-10-19T14:01:58.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.224+0000 7f7b86ffd640 1 --2- 172.21.15.57:0/2757609609 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f7b7406ac90 0x7f7b7406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:58.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.224+0000 7f7b86ffd640 1 -- 172.21.15.57:0/2757609609 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 6745+0+0 (secure 0 0 0) 0x7f7b88089f50 con 0x7f7b900cbd20 2023-10-19T14:01:58.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.224+0000 7f7b96430640 1 --2- 172.21.15.57:0/2757609609 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f7b7406ac90 0x7f7b7406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:58.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.225+0000 7f7b96430640 1 --2- 172.21.15.57:0/2757609609 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f7b7406ac90 0x7f7b7406d150 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7b8c035000 tx=0x7f7b8c004730 comp rx=0 tx=0).ready entity=mgr.8315 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:58.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.228+0000 7f7b86ffd640 1 -- 172.21.15.57:0/2757609609 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7b90141a30 con 0x7f7b900cbd20 2023-10-19T14:01:58.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.438+0000 7f7b98694640 1 -- 172.21.15.57:0/2757609609 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f7b90141a30 con 0x7f7b900cbd20 2023-10-19T14:01:58.441 INFO:tasks.ceph.mon.a.smithi057.stderr:2023-10-19T14:01:58.439+0000 7f1253ff3640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-19T14:01:58.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.440+0000 7f7b86ffd640 1 -- 172.21.15.57:0/2757609609 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v71) v1 ==== 161+0+0 (secure 0 0 0) 0x7f7b90141a30 con 0x7f7b900cbd20 2023-10-19T14:01:58.443 INFO:teuthology.orchestra.run.smithi057.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-19T14:01:58.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.442+0000 7f7b98694640 1 -- 172.21.15.57:0/2757609609 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f7b7406ac90 msgr2=0x7f7b7406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:58.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.442+0000 7f7b98694640 1 --2- 172.21.15.57:0/2757609609 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f7b7406ac90 0x7f7b7406d150 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7b8c035000 tx=0x7f7b8c004730 comp rx=0 tx=0).stop 2023-10-19T14:01:58.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.442+0000 7f7b98694640 1 -- 172.21.15.57:0/2757609609 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7b900cbd20 msgr2=0x7f7b9013d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:58.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.442+0000 7f7b98694640 1 --2- 172.21.15.57:0/2757609609 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7b900cbd20 0x7f7b9013d8a0 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 crypto rx=0x7f7b8800eaa0 tx=0x7f7b8800ef60 comp rx=0 tx=0).stop 2023-10-19T14:01:58.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.442+0000 7f7b98694640 1 -- 172.21.15.57:0/2757609609 shutdown_connections 2023-10-19T14:01:58.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.442+0000 7f7b98694640 1 --2- 172.21.15.57:0/2757609609 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f7b7406ac90 0x7f7b7406d150 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:58.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.442+0000 7f7b98694640 1 --2- 172.21.15.57:0/2757609609 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7b900cbd20 0x7f7b9013d8a0 unknown :-1 s=CLOSED pgs=1979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:58.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.442+0000 7f7b98694640 1 --2- 172.21.15.57:0/2757609609 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7b900b0530 0x7f7b9013d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:58.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.442+0000 7f7b98694640 1 --2- 172.21.15.57:0/2757609609 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7b900afa20 0x7f7b90141110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:58.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.442+0000 7f7b98694640 1 -- 172.21.15.57:0/2757609609 >> 172.21.15.57:0/2757609609 conn(0x7f7b90069930 msgr2=0x7f7b90069d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:58.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.442+0000 7f7b98694640 1 -- 172.21.15.57:0/2757609609 shutdown_connections 2023-10-19T14:01:58.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.442+0000 7f7b98694640 1 -- 172.21.15.57:0/2757609609 wait complete. 2023-10-19T14:01:58.451 INFO:tasks.ceph.mds.d.smithi179.stderr: -2615> 2023-10-19T14:01:45.122+0000 7f88a6b24640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-19T14:01:58.456 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-19T14:01:58.479 INFO:tasks.ceph.mds.d.smithi179.stdout:starting mds.d at 2023-10-19T14:01:58.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.565+0000 7fb6983a5640 1 -- 172.21.15.57:0/3989981946 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb69006af00 msgr2=0x7fb69006b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:58.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.565+0000 7fb6983a5640 1 --2- 172.21.15.57:0/3989981946 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb69006af00 0x7fb69006b2e0 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 crypto rx=0x7fb68c009830 tx=0x7fb68c02bcd0 comp rx=0 tx=0).stop 2023-10-19T14:01:58.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.565+0000 7fb6983a5640 1 -- 172.21.15.57:0/3989981946 shutdown_connections 2023-10-19T14:01:58.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.565+0000 7fb6983a5640 1 --2- 172.21.15.57:0/3989981946 >> v2:172.21.15.179:3300/0 conn(0x7fb6900c96b0 0x7fb6900c9b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:58.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.565+0000 7fb6983a5640 1 --2- 172.21.15.57:0/3989981946 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb69006af00 0x7fb69006b2e0 unknown :-1 s=CLOSED pgs=1981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:58.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.565+0000 7fb6983a5640 1 -- 172.21.15.57:0/3989981946 >> 172.21.15.57:0/3989981946 conn(0x7fb690069930 msgr2=0x7fb690069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:58.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.565+0000 7fb6983a5640 1 -- 172.21.15.57:0/3989981946 shutdown_connections 2023-10-19T14:01:58.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.565+0000 7fb6983a5640 1 -- 172.21.15.57:0/3989981946 wait complete. 2023-10-19T14:01:58.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.566+0000 7fb6983a5640 1 Processor -- start 2023-10-19T14:01:58.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.566+0000 7fb6983a5640 1 -- start start 2023-10-19T14:01:58.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.567+0000 7fb6983a5640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb69006af00 0x7fb6900a7ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:58.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.567+0000 7fb6983a5640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb6900ab500 0x7fb69013aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:58.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.567+0000 7fb6983a5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb6900c96b0 0x7fb6900a9080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:58.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.567+0000 7fb6983a5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6900a8020 con 0x7fb6900c96b0 2023-10-19T14:01:58.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.567+0000 7fb6983a5640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6900a7650 con 0x7fb6900ab500 2023-10-19T14:01:58.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.567+0000 7fb6983a5640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb690130ce0 con 0x7fb69006af00 2023-10-19T14:01:58.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.567+0000 7fb696942640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb6900c96b0 0x7fb6900a9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:58.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.567+0000 7fb696942640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb6900c96b0 0x7fb6900a9080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:47306/0 (socket says 172.21.15.57:47306) 2023-10-19T14:01:58.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.567+0000 7fb696942640 1 -- 172.21.15.57:0/1653584989 learned_addr learned my addr 172.21.15.57:0/1653584989 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:01:58.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.567+0000 7fb695940640 1 --2- 172.21.15.57:0/1653584989 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb6900ab500 0x7fb69013aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:58.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.567+0000 7fb696141640 1 --2- 172.21.15.57:0/1653584989 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb69006af00 0x7fb6900a7ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:58.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.567+0000 7fb696942640 1 -- 172.21.15.57:0/1653584989 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb69006af00 msgr2=0x7fb6900a7ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:58.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.567+0000 7fb696942640 1 --2- 172.21.15.57:0/1653584989 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb69006af00 0x7fb6900a7ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:58.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.567+0000 7fb696942640 1 -- 172.21.15.57:0/1653584989 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb6900ab500 msgr2=0x7fb69013aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:58.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.567+0000 7fb696942640 1 --2- 172.21.15.57:0/1653584989 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb6900ab500 0x7fb69013aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:58.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.567+0000 7fb696942640 1 -- 172.21.15.57:0/1653584989 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb68c009490 con 0x7fb6900c96b0 2023-10-19T14:01:58.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.568+0000 7fb696942640 1 --2- 172.21.15.57:0/1653584989 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb6900c96b0 0x7fb6900a9080 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7fb68800cbc0 tx=0x7fb68800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:58.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.568+0000 7fb686ffd640 1 -- 172.21.15.57:0/1653584989 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb68800aca0 con 0x7fb6900c96b0 2023-10-19T14:01:58.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.568+0000 7fb686ffd640 1 -- 172.21.15.57:0/1653584989 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fb688003970 con 0x7fb6900c96b0 2023-10-19T14:01:58.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.568+0000 7fb6983a5640 1 -- 172.21.15.57:0/1653584989 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6900a6d20 con 0x7fb6900c96b0 2023-10-19T14:01:58.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.568+0000 7fb686ffd640 1 -- 172.21.15.57:0/1653584989 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb688003d80 con 0x7fb6900c96b0 2023-10-19T14:01:58.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.568+0000 7fb6983a5640 1 -- 172.21.15.57:0/1653584989 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6900a72e0 con 0x7fb6900c96b0 2023-10-19T14:01:58.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.569+0000 7fb6983a5640 1 -- 172.21.15.57:0/1653584989 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6900a7800 con 0x7fb6900c96b0 2023-10-19T14:01:58.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.569+0000 7fb686ffd640 1 -- 172.21.15.57:0/1653584989 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 97) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fb688004030 con 0x7fb6900c96b0 2023-10-19T14:01:58.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.570+0000 7fb686ffd640 1 --2- 172.21.15.57:0/1653584989 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7fb67406ac90 0x7fb67406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:58.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.570+0000 7fb686ffd640 1 -- 172.21.15.57:0/1653584989 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 6831+0+0 (secure 0 0 0) 0x7fb68808a130 con 0x7fb6900c96b0 2023-10-19T14:01:58.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.570+0000 7fb696141640 1 --2- 172.21.15.57:0/1653584989 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7fb67406ac90 0x7fb67406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:58.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.570+0000 7fb696141640 1 --2- 172.21.15.57:0/1653584989 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7fb67406ac90 0x7fb67406d150 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb68c036000 tx=0x7fb68c004730 comp rx=0 tx=0).ready entity=mgr.8315 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:58.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.574+0000 7fb686ffd640 1 -- 172.21.15.57:0/1653584989 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb6900a7800 con 0x7fb6900c96b0 2023-10-19T14:01:58.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.787+0000 7fb6983a5640 1 -- 172.21.15.57:0/1653584989 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fb6900a7800 con 0x7fb6900c96b0 2023-10-19T14:01:59.001 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:58.999+0000 7fb686ffd640 1 -- 172.21.15.57:0/1653584989 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mgrmap(e 98) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fb688055490 con 0x7fb6900c96b0 2023-10-19T14:01:59.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:59.443+0000 7fb686ffd640 1 -- 172.21.15.57:0/1653584989 <== mon.0 v2:172.21.15.57:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v72) v1 ==== 104+0+0 (secure 0 0 0) 0x7fb688072c30 con 0x7fb6900c96b0 2023-10-19T14:01:59.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:59.446+0000 7fb6983a5640 1 -- 172.21.15.57:0/1653584989 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7fb67406ac90 msgr2=0x7fb67406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:59.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:59.446+0000 7fb6983a5640 1 --2- 172.21.15.57:0/1653584989 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7fb67406ac90 0x7fb67406d150 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb68c036000 tx=0x7fb68c004730 comp rx=0 tx=0).stop 2023-10-19T14:01:59.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:59.446+0000 7fb6983a5640 1 -- 172.21.15.57:0/1653584989 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb6900c96b0 msgr2=0x7fb6900a9080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:59.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:59.446+0000 7fb6983a5640 1 --2- 172.21.15.57:0/1653584989 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb6900c96b0 0x7fb6900a9080 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7fb68800cbc0 tx=0x7fb68800a060 comp rx=0 tx=0).stop 2023-10-19T14:01:59.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:59.446+0000 7fb6983a5640 1 -- 172.21.15.57:0/1653584989 shutdown_connections 2023-10-19T14:01:59.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:59.446+0000 7fb6983a5640 1 --2- 172.21.15.57:0/1653584989 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7fb67406ac90 0x7fb67406d150 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:59.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:59.446+0000 7fb6983a5640 1 --2- 172.21.15.57:0/1653584989 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb6900c96b0 0x7fb6900a9080 unknown :-1 s=CLOSED pgs=1982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:59.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:59.446+0000 7fb6983a5640 1 --2- 172.21.15.57:0/1653584989 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb6900ab500 0x7fb69013aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:59.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:59.446+0000 7fb6983a5640 1 --2- 172.21.15.57:0/1653584989 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb69006af00 0x7fb6900a7ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:59.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:59.446+0000 7fb6983a5640 1 -- 172.21.15.57:0/1653584989 >> 172.21.15.57:0/1653584989 conn(0x7fb690069930 msgr2=0x7fb690069d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:59.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:59.446+0000 7fb6983a5640 1 -- 172.21.15.57:0/1653584989 shutdown_connections 2023-10-19T14:01:59.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:59.446+0000 7fb6983a5640 1 -- 172.21.15.57:0/1653584989 wait complete. 2023-10-19T14:01:59.462 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-19T14:01:59.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:59.579+0000 7f17c33ce640 1 -- 172.21.15.57:0/249229510 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f17bc140660 msgr2=0x7f17bc0c97c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:59.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:59.579+0000 7f17c33ce640 1 --2- 172.21.15.57:0/249229510 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f17bc140660 0x7f17bc0c97c0 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 crypto rx=0x7f17b0005fd0 tx=0x7f17b002be10 comp rx=0 tx=0).stop 2023-10-19T14:01:59.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:59.579+0000 7f17c33ce640 1 -- 172.21.15.57:0/249229510 shutdown_connections 2023-10-19T14:01:59.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:59.579+0000 7f17c33ce640 1 --2- 172.21.15.57:0/249229510 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f17bc140660 0x7f17bc0c97c0 unknown :-1 s=CLOSED pgs=1984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:59.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:59.579+0000 7f17c33ce640 1 --2- 172.21.15.57:0/249229510 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f17bc13d360 0x7f17bc15a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:59.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:59.579+0000 7f17c33ce640 1 -- 172.21.15.57:0/249229510 >> 172.21.15.57:0/249229510 conn(0x7f17bc069930 msgr2=0x7f17bc069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:01:59.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:59.579+0000 7f17c33ce640 1 -- 172.21.15.57:0/249229510 shutdown_connections 2023-10-19T14:01:59.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:59.579+0000 7f17c33ce640 1 -- 172.21.15.57:0/249229510 wait complete. 2023-10-19T14:01:59.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:59.580+0000 7f17c33ce640 1 Processor -- start 2023-10-19T14:01:59.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:59.580+0000 7f17c33ce640 1 -- start start 2023-10-19T14:01:59.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:59.581+0000 7f17c33ce640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f17bc0c9d00 0x7f17bc1566b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:59.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:59.581+0000 7f17c33ce640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f17bc13d360 0x7f17bc157b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:59.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:59.581+0000 7f17c33ce640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f17bc140660 0x7f17bc13ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:59.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:59.581+0000 7f17c33ce640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17bc1585c0 con 0x7f17bc13d360 2023-10-19T14:01:59.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:59.581+0000 7f17c33ce640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17bc1588a0 con 0x7f17bc0c9d00 2023-10-19T14:01:59.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:59.581+0000 7f17c33ce640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17bc158b80 con 0x7f17bc140660 2023-10-19T14:01:59.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:59.581+0000 7f17c0969640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f17bc13d360 0x7f17bc157b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:59.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:59.581+0000 7f17c0969640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f17bc13d360 0x7f17bc157b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:47330/0 (socket says 172.21.15.57:47330) 2023-10-19T14:01:59.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:59.581+0000 7f17c0969640 1 -- 172.21.15.57:0/1391584518 learned_addr learned my addr 172.21.15.57:0/1391584518 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:01:59.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:59.581+0000 7f17c116a640 1 --2- 172.21.15.57:0/1391584518 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f17bc0c9d00 0x7f17bc1566b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:59.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:59.581+0000 7f17c196b640 1 --2- 172.21.15.57:0/1391584518 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f17bc140660 0x7f17bc13ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:59.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:59.581+0000 7f17c0969640 1 -- 172.21.15.57:0/1391584518 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f17bc140660 msgr2=0x7f17bc13ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:59.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:59.581+0000 7f17c0969640 1 --2- 172.21.15.57:0/1391584518 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f17bc140660 0x7f17bc13ae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:59.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:59.581+0000 7f17c0969640 1 -- 172.21.15.57:0/1391584518 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f17bc0c9d00 msgr2=0x7f17bc1566b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:01:59.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:59.581+0000 7f17c0969640 1 --2- 172.21.15.57:0/1391584518 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f17bc0c9d00 0x7f17bc1566b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:01:59.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:59.581+0000 7f17c0969640 1 -- 172.21.15.57:0/1391584518 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17b0009560 con 0x7f17bc13d360 2023-10-19T14:01:59.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:59.582+0000 7f17c0969640 1 --2- 172.21.15.57:0/1391584518 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f17bc13d360 0x7f17bc157b00 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7f17b0036040 tx=0x7f17b002dbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:59.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:59.582+0000 7f17b9ffb640 1 -- 172.21.15.57:0/1391584518 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17b00035f0 con 0x7f17bc13d360 2023-10-19T14:01:59.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:59.582+0000 7f17b9ffb640 1 -- 172.21.15.57:0/1391584518 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f17b0003840 con 0x7f17bc13d360 2023-10-19T14:01:59.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:59.582+0000 7f17b9ffb640 1 -- 172.21.15.57:0/1391584518 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17b0003c50 con 0x7f17bc13d360 2023-10-19T14:01:59.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:59.582+0000 7f17c33ce640 1 -- 172.21.15.57:0/1391584518 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17bc158e60 con 0x7f17bc13d360 2023-10-19T14:01:59.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:59.582+0000 7f17c33ce640 1 -- 172.21.15.57:0/1391584518 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17bc156bf0 con 0x7f17bc13d360 2023-10-19T14:01:59.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:59.583+0000 7f17c33ce640 1 -- 172.21.15.57:0/1391584518 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17bc159380 con 0x7f17bc13d360 2023-10-19T14:01:59.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:59.584+0000 7f17b9ffb640 1 -- 172.21.15.57:0/1391584518 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 98) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f17b003e450 con 0x7f17bc13d360 2023-10-19T14:01:59.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:59.584+0000 7f17b9ffb640 1 --2- 172.21.15.57:0/1391584518 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f17a806af00 0x7f17a806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:01:59.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:59.584+0000 7f17b9ffb640 1 -- 172.21.15.57:0/1391584518 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 6831+0+0 (secure 0 0 0) 0x7f17b00ac6e0 con 0x7f17bc13d360 2023-10-19T14:01:59.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:59.584+0000 7f17c116a640 1 --2- 172.21.15.57:0/1391584518 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f17a806af00 0x7f17a806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:01:59.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:59.585+0000 7f17c116a640 1 --2- 172.21.15.57:0/1391584518 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f17a806af00 0x7f17a806d3c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f17b4005fd0 tx=0x7f17b4005850 comp rx=0 tx=0).ready entity=mgr.8315 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:01:59.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:59.589+0000 7f17b9ffb640 1 -- 172.21.15.57:0/1391584518 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f17bc159380 con 0x7f17bc13d360 2023-10-19T14:01:59.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:01:59.768+0000 7f17c33ce640 1 -- 172.21.15.57:0/1391584518 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f17bc159380 con 0x7f17bc13d360 2023-10-19T14:02:00.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:00.601+0000 7f17b9ffb640 1 -- 172.21.15.57:0/1391584518 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v179) v1 ==== 181+0+0 (secure 0 0 0) 0x7f17bc159380 con 0x7f17bc13d360 2023-10-19T14:02:00.603 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'cephfs_metadata' removed 2023-10-19T14:02:00.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:00.605+0000 7f17c33ce640 1 -- 172.21.15.57:0/1391584518 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f17a806af00 msgr2=0x7f17a806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:00.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:00.605+0000 7f17c33ce640 1 --2- 172.21.15.57:0/1391584518 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f17a806af00 0x7f17a806d3c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f17b4005fd0 tx=0x7f17b4005850 comp rx=0 tx=0).stop 2023-10-19T14:02:00.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:00.605+0000 7f17c33ce640 1 -- 172.21.15.57:0/1391584518 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f17bc13d360 msgr2=0x7f17bc157b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:00.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:00.605+0000 7f17c33ce640 1 --2- 172.21.15.57:0/1391584518 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f17bc13d360 0x7f17bc157b00 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7f17b0036040 tx=0x7f17b002dbb0 comp rx=0 tx=0).stop 2023-10-19T14:02:00.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:00.607+0000 7f17c33ce640 1 -- 172.21.15.57:0/1391584518 shutdown_connections 2023-10-19T14:02:00.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:00.607+0000 7f17c33ce640 1 --2- 172.21.15.57:0/1391584518 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f17bc140660 0x7f17bc13ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:00.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:00.608+0000 7f17c33ce640 1 --2- 172.21.15.57:0/1391584518 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f17a806af00 0x7f17a806d3c0 secure :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f17b4005fd0 tx=0x7f17b4005850 comp rx=0 tx=0).stop 2023-10-19T14:02:00.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:00.608+0000 7f17c33ce640 1 --2- 172.21.15.57:0/1391584518 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f17bc13d360 0x7f17bc157b00 secure :-1 s=CLOSED pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7f17b0036040 tx=0x7f17b002dbb0 comp rx=0 tx=0).stop 2023-10-19T14:02:00.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:00.608+0000 7f17c33ce640 1 --2- 172.21.15.57:0/1391584518 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f17bc0c9d00 0x7f17bc1566b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:00.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:00.608+0000 7f17c33ce640 1 -- 172.21.15.57:0/1391584518 >> 172.21.15.57:0/1391584518 conn(0x7f17bc069930 msgr2=0x7f17bc13c3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:02:00.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:00.608+0000 7f17c33ce640 1 -- 172.21.15.57:0/1391584518 shutdown_connections 2023-10-19T14:02:00.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:00.609+0000 7f17c33ce640 1 -- 172.21.15.57:0/1391584518 wait complete. 2023-10-19T14:02:00.625 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-19T14:02:00.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:00.742+0000 7fbe469e2640 1 -- 172.21.15.57:0/3383766149 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbe400c9dc0 msgr2=0x7fbe400ab500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:00.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:00.742+0000 7fbe469e2640 1 --2- 172.21.15.57:0/3383766149 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbe400c9dc0 0x7fbe400ab500 secure :-1 s=READY pgs=1987 cs=0 l=1 rev1=1 crypto rx=0x7fbe40071370 tx=0x7fbe3002be40 comp rx=0 tx=0).stop 2023-10-19T14:02:00.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:00.742+0000 7fbe469e2640 1 -- 172.21.15.57:0/3383766149 shutdown_connections 2023-10-19T14:02:00.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:00.742+0000 7fbe469e2640 1 --2- 172.21.15.57:0/3383766149 >> v2:172.21.15.179:3300/0 conn(0x7fbe401533a0 0x7fbe400aba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:00.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:00.742+0000 7fbe469e2640 1 --2- 172.21.15.57:0/3383766149 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbe400c9dc0 0x7fbe400ab500 unknown :-1 s=CLOSED pgs=1987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:00.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:00.742+0000 7fbe469e2640 1 -- 172.21.15.57:0/3383766149 >> 172.21.15.57:0/3383766149 conn(0x7fbe40069930 msgr2=0x7fbe40069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:02:00.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:00.742+0000 7fbe469e2640 1 -- 172.21.15.57:0/3383766149 shutdown_connections 2023-10-19T14:02:00.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:00.742+0000 7fbe469e2640 1 -- 172.21.15.57:0/3383766149 wait complete. 2023-10-19T14:02:00.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:00.743+0000 7fbe469e2640 1 Processor -- start 2023-10-19T14:02:00.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:00.743+0000 7fbe469e2640 1 -- start start 2023-10-19T14:02:00.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:00.744+0000 7fbe469e2640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbe400c99e0 0x7fbe401477b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:00.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:00.744+0000 7fbe469e2640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbe400c9dc0 0x7fbe400aab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:00.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:00.744+0000 7fbe469e2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbe401533a0 0x7fbe4006f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:00.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:00.744+0000 7fbe469e2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe400b1240 con 0x7fbe401533a0 2023-10-19T14:02:00.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:00.744+0000 7fbe469e2640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe40147cf0 con 0x7fbe400c9dc0 2023-10-19T14:02:00.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:00.744+0000 7fbe469e2640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe40155560 con 0x7fbe400c99e0 2023-10-19T14:02:00.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:00.744+0000 7fbe44f7f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbe401533a0 0x7fbe4006f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:00.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:00.744+0000 7fbe44f7f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbe401533a0 0x7fbe4006f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:47340/0 (socket says 172.21.15.57:47340) 2023-10-19T14:02:00.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:00.744+0000 7fbe44f7f640 1 -- 172.21.15.57:0/1437347961 learned_addr learned my addr 172.21.15.57:0/1437347961 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:02:00.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:00.744+0000 7fbe3ffff640 1 --2- 172.21.15.57:0/1437347961 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbe400c99e0 0x7fbe401477b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:00.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:00.744+0000 7fbe44f7f640 1 -- 172.21.15.57:0/1437347961 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbe400c99e0 msgr2=0x7fbe401477b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:00.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:00.744+0000 7fbe44f7f640 1 --2- 172.21.15.57:0/1437347961 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbe400c99e0 0x7fbe401477b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:00.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:00.744+0000 7fbe3f7fe640 1 --2- 172.21.15.57:0/1437347961 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbe400c9dc0 0x7fbe400aab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:00.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:00.744+0000 7fbe44f7f640 1 -- 172.21.15.57:0/1437347961 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbe400c9dc0 msgr2=0x7fbe400aab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:00.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:00.744+0000 7fbe44f7f640 1 --2- 172.21.15.57:0/1437347961 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbe400c9dc0 0x7fbe400aab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:00.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:00.744+0000 7fbe44f7f640 1 -- 172.21.15.57:0/1437347961 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe30009560 con 0x7fbe401533a0 2023-10-19T14:02:00.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:00.744+0000 7fbe3f7fe640 1 --2- 172.21.15.57:0/1437347961 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbe400c9dc0 0x7fbe400aab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:02:00.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:00.745+0000 7fbe44f7f640 1 --2- 172.21.15.57:0/1437347961 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbe401533a0 0x7fbe4006f080 secure :-1 s=READY pgs=1988 cs=0 l=1 rev1=1 crypto rx=0x7fbe3800cad0 tx=0x7fbe3800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:02:00.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:00.745+0000 7fbe3cff9640 1 -- 172.21.15.57:0/1437347961 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe3800ae20 con 0x7fbe401533a0 2023-10-19T14:02:00.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:00.745+0000 7fbe3cff9640 1 -- 172.21.15.57:0/1437347961 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fbe3800f450 con 0x7fbe401533a0 2023-10-19T14:02:00.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:00.745+0000 7fbe469e2640 1 -- 172.21.15.57:0/1437347961 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe40155840 con 0x7fbe401533a0 2023-10-19T14:02:00.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:00.745+0000 7fbe3cff9640 1 -- 172.21.15.57:0/1437347961 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe3800f8c0 con 0x7fbe401533a0 2023-10-19T14:02:00.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:00.745+0000 7fbe469e2640 1 -- 172.21.15.57:0/1437347961 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe40155e00 con 0x7fbe401533a0 2023-10-19T14:02:00.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:00.746+0000 7fbe469e2640 1 -- 172.21.15.57:0/1437347961 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe400690e0 con 0x7fbe401533a0 2023-10-19T14:02:00.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:00.746+0000 7fbe3cff9640 1 -- 172.21.15.57:0/1437347961 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 99) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fbe3800fa20 con 0x7fbe401533a0 2023-10-19T14:02:00.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:00.747+0000 7fbe3cff9640 1 --2- 172.21.15.57:0/1437347961 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7fbe2c06af90 0x7fbe2c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:00.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:00.747+0000 7fbe3cff9640 1 -- 172.21.15.57:0/1437347961 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 6393+0+0 (secure 0 0 0) 0x7fbe3808a810 con 0x7fbe401533a0 2023-10-19T14:02:00.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:00.747+0000 7fbe3ffff640 1 --2- 172.21.15.57:0/1437347961 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7fbe2c06af90 0x7fbe2c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:00.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:00.748+0000 7fbe3ffff640 1 --2- 172.21.15.57:0/1437347961 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7fbe2c06af90 0x7fbe2c06d450 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fbe340010e0 tx=0x7fbe34009040 comp rx=0 tx=0).ready entity=mgr.8315 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:02:00.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:00.751+0000 7fbe3cff9640 1 -- 172.21.15.57:0/1437347961 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fbe38055350 con 0x7fbe401533a0 2023-10-19T14:02:00.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:00.926+0000 7fbe469e2640 1 -- 172.21.15.57:0/1437347961 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fbe400cc420 con 0x7fbe401533a0 2023-10-19T14:02:01.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:01.602+0000 7fbe3cff9640 1 -- 172.21.15.57:0/1437347961 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v180) v1 ==== 169+0+0 (secure 0 0 0) 0x7fbe38058970 con 0x7fbe401533a0 2023-10-19T14:02:01.605 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'cephfs_data' removed 2023-10-19T14:02:01.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:01.607+0000 7fbe469e2640 1 -- 172.21.15.57:0/1437347961 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7fbe2c06af90 msgr2=0x7fbe2c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:01.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:01.607+0000 7fbe469e2640 1 --2- 172.21.15.57:0/1437347961 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7fbe2c06af90 0x7fbe2c06d450 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fbe340010e0 tx=0x7fbe34009040 comp rx=0 tx=0).stop 2023-10-19T14:02:01.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:01.607+0000 7fbe469e2640 1 -- 172.21.15.57:0/1437347961 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbe401533a0 msgr2=0x7fbe4006f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:01.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:01.607+0000 7fbe469e2640 1 --2- 172.21.15.57:0/1437347961 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbe401533a0 0x7fbe4006f080 secure :-1 s=READY pgs=1988 cs=0 l=1 rev1=1 crypto rx=0x7fbe3800cad0 tx=0x7fbe3800cf90 comp rx=0 tx=0).stop 2023-10-19T14:02:01.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:01.607+0000 7fbe469e2640 1 -- 172.21.15.57:0/1437347961 shutdown_connections 2023-10-19T14:02:01.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:01.607+0000 7fbe469e2640 1 --2- 172.21.15.57:0/1437347961 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7fbe2c06af90 0x7fbe2c06d450 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:01.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:01.607+0000 7fbe469e2640 1 --2- 172.21.15.57:0/1437347961 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbe401533a0 0x7fbe4006f080 unknown :-1 s=CLOSED pgs=1988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:01.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:01.607+0000 7fbe469e2640 1 --2- 172.21.15.57:0/1437347961 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbe400c9dc0 0x7fbe400aab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:01.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:01.607+0000 7fbe469e2640 1 --2- 172.21.15.57:0/1437347961 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbe400c99e0 0x7fbe401477b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:01.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:01.607+0000 7fbe469e2640 1 -- 172.21.15.57:0/1437347961 >> 172.21.15.57:0/1437347961 conn(0x7fbe40069930 msgr2=0x7fbe40069d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:02:01.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:01.608+0000 7fbe469e2640 1 -- 172.21.15.57:0/1437347961 shutdown_connections 2023-10-19T14:02:01.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:01.608+0000 7fbe469e2640 1 -- 172.21.15.57:0/1437347961 wait complete. 2023-10-19T14:02:01.626 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 30 2023-10-19T14:02:01.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:01.739+0000 7f8d80e1a640 1 -- 172.21.15.57:0/3420708910 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8d7c0cbd20 msgr2=0x7f8d7c158300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:01.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:01.739+0000 7f8d80e1a640 1 --2- 172.21.15.57:0/3420708910 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8d7c0cbd20 0x7f8d7c158300 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 crypto rx=0x7f8d6c005fd0 tx=0x7f8d6c02c2e0 comp rx=0 tx=0).stop 2023-10-19T14:02:01.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:01.739+0000 7f8d80e1a640 1 -- 172.21.15.57:0/3420708910 shutdown_connections 2023-10-19T14:02:01.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:01.739+0000 7f8d80e1a640 1 --2- 172.21.15.57:0/3420708910 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8d7c0b0530 0x7f8d7c0b09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:01.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:01.739+0000 7f8d80e1a640 1 --2- 172.21.15.57:0/3420708910 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8d7c0cbd20 0x7f8d7c158300 unknown :-1 s=CLOSED pgs=1989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:01.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:01.739+0000 7f8d80e1a640 1 -- 172.21.15.57:0/3420708910 >> 172.21.15.57:0/3420708910 conn(0x7f8d7c069930 msgr2=0x7f8d7c069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:02:01.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:01.739+0000 7f8d80e1a640 1 -- 172.21.15.57:0/3420708910 shutdown_connections 2023-10-19T14:02:01.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:01.739+0000 7f8d80e1a640 1 -- 172.21.15.57:0/3420708910 wait complete. 2023-10-19T14:02:01.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:01.741+0000 7f8d80e1a640 1 Processor -- start 2023-10-19T14:02:01.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:01.741+0000 7f8d80e1a640 1 -- start start 2023-10-19T14:02:01.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:01.741+0000 7f8d80e1a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8d7c0afa20 0x7f8d7c148f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:01.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:01.741+0000 7f8d80e1a640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8d7c0b0530 0x7f8d7c150d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:01.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:01.741+0000 7f8d80e1a640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8d7c0cbd20 0x7f8d7c141ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:01.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:01.741+0000 7f8d80e1a640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d7c151260 con 0x7f8d7c0afa20 2023-10-19T14:02:01.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:01.741+0000 7f8d80e1a640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d7c142430 con 0x7f8d7c0b0530 2023-10-19T14:02:01.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:01.741+0000 7f8d80e1a640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d7c15a280 con 0x7f8d7c0cbd20 2023-10-19T14:02:01.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:01.742+0000 7f8d7a59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8d7c0afa20 0x7f8d7c148f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:01.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:01.742+0000 7f8d7a59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8d7c0afa20 0x7f8d7c148f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:47370/0 (socket says 172.21.15.57:47370) 2023-10-19T14:02:01.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:01.742+0000 7f8d7a59c640 1 -- 172.21.15.57:0/802280319 learned_addr learned my addr 172.21.15.57:0/802280319 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:02:01.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:01.742+0000 7f8d7ad9d640 1 --2- 172.21.15.57:0/802280319 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8d7c0cbd20 0x7f8d7c141ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:01.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:01.742+0000 7f8d79d9b640 1 --2- 172.21.15.57:0/802280319 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8d7c0b0530 0x7f8d7c150d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:01.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:01.742+0000 7f8d7a59c640 1 -- 172.21.15.57:0/802280319 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8d7c0cbd20 msgr2=0x7f8d7c141ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:01.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:01.742+0000 7f8d7a59c640 1 --2- 172.21.15.57:0/802280319 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8d7c0cbd20 0x7f8d7c141ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:01.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:01.742+0000 7f8d7a59c640 1 -- 172.21.15.57:0/802280319 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8d7c0b0530 msgr2=0x7f8d7c150d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:01.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:01.742+0000 7f8d7a59c640 1 --2- 172.21.15.57:0/802280319 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8d7c0b0530 0x7f8d7c150d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:01.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:01.742+0000 7f8d7a59c640 1 -- 172.21.15.57:0/802280319 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d6c009900 con 0x7f8d7c0afa20 2023-10-19T14:02:01.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:01.742+0000 7f8d79d9b640 1 --2- 172.21.15.57:0/802280319 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8d7c0b0530 0x7f8d7c150d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:02:01.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:01.742+0000 7f8d7ad9d640 1 --2- 172.21.15.57:0/802280319 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8d7c0cbd20 0x7f8d7c141ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:02:01.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:01.742+0000 7f8d7a59c640 1 --2- 172.21.15.57:0/802280319 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8d7c0afa20 0x7f8d7c148f30 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 crypto rx=0x7f8d6c006010 tx=0x7f8d6c035b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:02:01.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:01.742+0000 7f8d72ffd640 1 -- 172.21.15.57:0/802280319 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d6c004240 con 0x7f8d7c0afa20 2023-10-19T14:02:01.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:01.742+0000 7f8d72ffd640 1 -- 172.21.15.57:0/802280319 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f8d6c004490 con 0x7f8d7c0afa20 2023-10-19T14:02:01.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:01.742+0000 7f8d72ffd640 1 -- 172.21.15.57:0/802280319 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d6c004860 con 0x7f8d7c0afa20 2023-10-19T14:02:01.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:01.742+0000 7f8d80e1a640 1 -- 172.21.15.57:0/802280319 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d7c15a560 con 0x7f8d7c0afa20 2023-10-19T14:02:01.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:01.742+0000 7f8d80e1a640 1 -- 172.21.15.57:0/802280319 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d7c142720 con 0x7f8d7c0afa20 2023-10-19T14:02:01.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:01.743+0000 7f8d80e1a640 1 -- 172.21.15.57:0/802280319 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d7c06a270 con 0x7f8d7c0afa20 2023-10-19T14:02:01.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:01.744+0000 7f8d72ffd640 1 -- 172.21.15.57:0/802280319 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 99) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f8d6c03b040 con 0x7f8d7c0afa20 2023-10-19T14:02:01.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:01.744+0000 7f8d72ffd640 1 --2- 172.21.15.57:0/802280319 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f8d5c06af60 0x7f8d5c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:01.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:01.744+0000 7f8d72ffd640 1 -- 172.21.15.57:0/802280319 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5979+0+0 (secure 0 0 0) 0x7f8d6c0ac590 con 0x7f8d7c0afa20 2023-10-19T14:02:01.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:01.744+0000 7f8d79d9b640 1 --2- 172.21.15.57:0/802280319 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f8d5c06af60 0x7f8d5c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:01.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:01.745+0000 7f8d79d9b640 1 --2- 172.21.15.57:0/802280319 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f8d5c06af60 0x7f8d5c06d420 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c0af730 tx=0x7f8d64009040 comp rx=0 tx=0).ready entity=mgr.8315 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:02:01.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:01.748+0000 7f8d72ffd640 1 -- 172.21.15.57:0/802280319 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8d6c0774b0 con 0x7f8d7c0afa20 2023-10-19T14:02:01.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:01.924+0000 7f8d80e1a640 1 -- 172.21.15.57:0/802280319 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "config reset", "num": 30} v 0) v1 -- 0x7f8d7c0700d0 con 0x7f8d7c0afa20 2023-10-19T14:02:01.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:01.929+0000 7f8d72ffd640 1 -- 172.21.15.57:0/802280319 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 30}]=0 v66) v1 ==== 71+0+0 (secure 0 0 0) 0x7f8d6c07aad0 con 0x7f8d7c0afa20 2023-10-19T14:02:01.934 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:01.931+0000 7f8d80e1a640 1 -- 172.21.15.57:0/802280319 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f8d5c06af60 msgr2=0x7f8d5c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:01.934 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:01.931+0000 7f8d80e1a640 1 --2- 172.21.15.57:0/802280319 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f8d5c06af60 0x7f8d5c06d420 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c0af730 tx=0x7f8d64009040 comp rx=0 tx=0).stop 2023-10-19T14:02:01.935 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:01.931+0000 7f8d80e1a640 1 -- 172.21.15.57:0/802280319 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8d7c0afa20 msgr2=0x7f8d7c148f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:01.935 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:01.931+0000 7f8d80e1a640 1 --2- 172.21.15.57:0/802280319 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8d7c0afa20 0x7f8d7c148f30 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 crypto rx=0x7f8d6c006010 tx=0x7f8d6c035b50 comp rx=0 tx=0).stop 2023-10-19T14:02:01.935 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:01.931+0000 7f8d80e1a640 1 -- 172.21.15.57:0/802280319 shutdown_connections 2023-10-19T14:02:01.935 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:01.931+0000 7f8d80e1a640 1 --2- 172.21.15.57:0/802280319 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8d7c0cbd20 0x7f8d7c141ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:01.935 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:01.931+0000 7f8d80e1a640 1 --2- 172.21.15.57:0/802280319 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8d7c0b0530 0x7f8d7c150d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:01.936 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:01.931+0000 7f8d80e1a640 1 --2- 172.21.15.57:0/802280319 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f8d5c06af60 0x7f8d5c06d420 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:01.936 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:01.931+0000 7f8d80e1a640 1 --2- 172.21.15.57:0/802280319 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8d7c0afa20 0x7f8d7c148f30 unknown :-1 s=CLOSED pgs=1990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:01.936 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:01.931+0000 7f8d80e1a640 1 -- 172.21.15.57:0/802280319 >> 172.21.15.57:0/802280319 conn(0x7f8d7c069930 msgr2=0x7f8d7c1483c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:02:01.936 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:01.932+0000 7f8d80e1a640 1 -- 172.21.15.57:0/802280319 shutdown_connections 2023-10-19T14:02:01.936 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:01.932+0000 7f8d80e1a640 1 -- 172.21.15.57:0/802280319 wait complete. 2023-10-19T14:02:01.944 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove' 2023-10-19T14:02:02.055 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.052+0000 7fcc43632640 1 -- 172.21.15.57:0/725895557 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcc28004d00 msgr2=0x7fcc28005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:02.055 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.052+0000 7fcc43632640 1 --2- 172.21.15.57:0/725895557 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcc28004d00 0x7fcc28005180 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 crypto rx=0x7fcc34001d80 tx=0x7fcc280057f0 comp rx=0 tx=0).stop 2023-10-19T14:02:02.055 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.052+0000 7fcc41bcf640 1 -- 172.21.15.57:0/725895557 reap_dead start 2023-10-19T14:02:02.056 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.052+0000 7fcc43632640 1 -- 172.21.15.57:0/725895557 shutdown_connections 2023-10-19T14:02:02.056 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.052+0000 7fcc43632640 1 -- 172.21.15.57:0/725895557 >> 172.21.15.57:0/725895557 conn(0x7fcc3c069930 msgr2=0x7fcc3c069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:02:02.056 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.052+0000 7fcc43632640 1 -- 172.21.15.57:0/725895557 shutdown_connections 2023-10-19T14:02:02.056 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.052+0000 7fcc43632640 1 -- 172.21.15.57:0/725895557 wait complete. 2023-10-19T14:02:02.058 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.053+0000 7fcc43632640 1 Processor -- start 2023-10-19T14:02:02.058 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.053+0000 7fcc43632640 1 -- start start 2023-10-19T14:02:02.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.054+0000 7fcc43632640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcc3c13a450 0x7fcc3c0a7b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:02.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.054+0000 7fcc43632640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcc3c0a8060 0x7fcc3c0b0c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:02.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.054+0000 7fcc43632640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcc3c0b11c0 0x7fcc3c152ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:02.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.054+0000 7fcc43632640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc3c0ac040 con 0x7fcc3c0a8060 2023-10-19T14:02:02.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.054+0000 7fcc43632640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc3c0b1670 con 0x7fcc3c0b11c0 2023-10-19T14:02:02.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.054+0000 7fcc43632640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc3c1ef260 con 0x7fcc3c13a450 2023-10-19T14:02:02.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.054+0000 7fcc40bcd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcc3c0a8060 0x7fcc3c0b0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:02.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.054+0000 7fcc40bcd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcc3c0a8060 0x7fcc3c0b0c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:47388/0 (socket says 172.21.15.57:47388) 2023-10-19T14:02:02.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.054+0000 7fcc40bcd640 1 -- 172.21.15.57:0/915859272 learned_addr learned my addr 172.21.15.57:0/915859272 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:02:02.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.054+0000 7fcc413ce640 1 --2- 172.21.15.57:0/915859272 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcc3c13a450 0x7fcc3c0a7b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:02.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.054+0000 7fcc41bcf640 1 --2- 172.21.15.57:0/915859272 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcc3c0b11c0 0x7fcc3c152ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:02.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.054+0000 7fcc40bcd640 1 -- 172.21.15.57:0/915859272 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcc3c13a450 msgr2=0x7fcc3c0a7b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:02.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.054+0000 7fcc40bcd640 1 --2- 172.21.15.57:0/915859272 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcc3c13a450 0x7fcc3c0a7b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:02.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.054+0000 7fcc40bcd640 1 -- 172.21.15.57:0/915859272 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcc3c0b11c0 msgr2=0x7fcc3c152ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:02.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.054+0000 7fcc40bcd640 1 --2- 172.21.15.57:0/915859272 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcc3c0b11c0 0x7fcc3c152ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:02.062 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.055+0000 7fcc40bcd640 1 -- 172.21.15.57:0/915859272 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc3c1f0210 con 0x7fcc3c0a8060 2023-10-19T14:02:02.062 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.055+0000 7fcc40bcd640 1 --2- 172.21.15.57:0/915859272 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcc3c0a8060 0x7fcc3c0b0c80 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7fcc2c00e930 tx=0x7fcc2c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:02:02.062 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.055+0000 7fcc31ffb640 1 -- 172.21.15.57:0/915859272 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc2c004440 con 0x7fcc3c0a8060 2023-10-19T14:02:02.062 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.055+0000 7fcc31ffb640 1 -- 172.21.15.57:0/915859272 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fcc2c00c840 con 0x7fcc3c0a8060 2023-10-19T14:02:02.062 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.055+0000 7fcc43632640 1 -- 172.21.15.57:0/915859272 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc3c1ef450 con 0x7fcc3c0a8060 2023-10-19T14:02:02.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.055+0000 7fcc31ffb640 1 -- 172.21.15.57:0/915859272 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc2c00cc30 con 0x7fcc3c0a8060 2023-10-19T14:02:02.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.055+0000 7fcc43632640 1 -- 172.21.15.57:0/915859272 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc3c1efa10 con 0x7fcc3c0a8060 2023-10-19T14:02:02.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.056+0000 7fcc43632640 1 -- 172.21.15.57:0/915859272 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc3c0690e0 con 0x7fcc3c0a8060 2023-10-19T14:02:02.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.056+0000 7fcc31ffb640 1 -- 172.21.15.57:0/915859272 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 99) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fcc2c0036d0 con 0x7fcc3c0a8060 2023-10-19T14:02:02.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.057+0000 7fcc31ffb640 1 --2- 172.21.15.57:0/915859272 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7fcc2006af60 0x7fcc2006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:02.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.057+0000 7fcc31ffb640 1 -- 172.21.15.57:0/915859272 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5979+0+0 (secure 0 0 0) 0x7fcc2c08a180 con 0x7fcc3c0a8060 2023-10-19T14:02:02.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.057+0000 7fcc413ce640 1 --2- 172.21.15.57:0/915859272 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7fcc2006af60 0x7fcc2006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:02.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.057+0000 7fcc413ce640 1 --2- 172.21.15.57:0/915859272 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7fcc2006af60 0x7fcc2006d420 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fcc38003700 tx=0x7fcc3800b260 comp rx=0 tx=0).ready entity=mgr.8315 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:02:02.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.061+0000 7fcc31ffb640 1 -- 172.21.15.57:0/915859272 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fcc2c054f00 con 0x7fcc3c0a8060 2023-10-19T14:02:02.160 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:02:02.163 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:02:02.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.303+0000 7fcc43632640 1 -- 172.21.15.57:0/915859272 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove"]} v 0) v1 -- 0x7fcc3c0cc420 con 0x7fcc3c0a8060 2023-10-19T14:02:02.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.612+0000 7fcc31ffb640 1 -- 172.21.15.57:0/915859272 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove"]}]=0 v2537) v1 ==== 174+0+0 (secure 0 0 0) 0x7fcc2c058520 con 0x7fcc3c0a8060 2023-10-19T14:02:02.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.614+0000 7fcc43632640 1 -- 172.21.15.57:0/915859272 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7fcc2006af60 msgr2=0x7fcc2006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:02.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.614+0000 7fcc43632640 1 --2- 172.21.15.57:0/915859272 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7fcc2006af60 0x7fcc2006d420 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fcc38003700 tx=0x7fcc3800b260 comp rx=0 tx=0).stop 2023-10-19T14:02:02.618 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.614+0000 7fcc43632640 1 -- 172.21.15.57:0/915859272 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcc3c0a8060 msgr2=0x7fcc3c0b0c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:02.618 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.614+0000 7fcc43632640 1 --2- 172.21.15.57:0/915859272 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcc3c0a8060 0x7fcc3c0b0c80 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7fcc2c00e930 tx=0x7fcc2c00edf0 comp rx=0 tx=0).stop 2023-10-19T14:02:02.618 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.615+0000 7fcc43632640 1 -- 172.21.15.57:0/915859272 shutdown_connections 2023-10-19T14:02:02.618 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.615+0000 7fcc43632640 1 --2- 172.21.15.57:0/915859272 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcc3c0b11c0 0x7fcc3c152ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:02.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.615+0000 7fcc43632640 1 --2- 172.21.15.57:0/915859272 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7fcc2006af60 0x7fcc2006d420 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:02.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.615+0000 7fcc43632640 1 --2- 172.21.15.57:0/915859272 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcc3c0a8060 0x7fcc3c0b0c80 unknown :-1 s=CLOSED pgs=1993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:02.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.615+0000 7fcc43632640 1 --2- 172.21.15.57:0/915859272 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcc3c13a450 0x7fcc3c0a7b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:02.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.615+0000 7fcc43632640 1 -- 172.21.15.57:0/915859272 >> 172.21.15.57:0/915859272 conn(0x7fcc3c069930 msgr2=0x7fcc3c0ca870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:02:02.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.615+0000 7fcc43632640 1 -- 172.21.15.57:0/915859272 shutdown_connections 2023-10-19T14:02:02.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.615+0000 7fcc43632640 1 -- 172.21.15.57:0/915859272 wait complete. 2023-10-19T14:02:02.628 INFO:tasks.cephfs_test_runner:Test listing snap schedules post removal of a schedule ... ok 2023-10-19T14:02:02.629 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_with_mgr_restart (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test that snap schedule is resumed after mgr restart 2023-10-19T14:02:02.629 INFO:tasks.cephfs_test_runner:test_snap_schedule_with_mgr_restart (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-10-19T14:02:02.630 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T14:02:02.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.737+0000 7f8c0cb05640 1 -- 172.21.15.57:0/3873405045 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8c080ceb40 msgr2=0x7f8c080b0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:02.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.737+0000 7f8c0cb05640 1 --2- 172.21.15.57:0/3873405045 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8c080ceb40 0x7f8c080b0650 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 crypto rx=0x7f8c0808a120 tx=0x7f8c0002c1d0 comp rx=0 tx=0).stop 2023-10-19T14:02:02.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.737+0000 7f8c0cb05640 1 -- 172.21.15.57:0/3873405045 shutdown_connections 2023-10-19T14:02:02.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.737+0000 7f8c0cb05640 1 --2- 172.21.15.57:0/3873405045 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8c080ceb40 0x7f8c080b0650 unknown :-1 s=CLOSED pgs=1994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:02.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.737+0000 7f8c0cb05640 1 --2- 172.21.15.57:0/3873405045 >> v2:172.21.15.179:3300/0 conn(0x7f8c0815b2a0 0x7f8c080d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:02.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.737+0000 7f8c0cb05640 1 -- 172.21.15.57:0/3873405045 >> 172.21.15.57:0/3873405045 conn(0x7f8c080d0d60 msgr2=0x7f8c080d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:02:02.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.737+0000 7f8c0cb05640 1 -- 172.21.15.57:0/3873405045 shutdown_connections 2023-10-19T14:02:02.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.737+0000 7f8c0cb05640 1 -- 172.21.15.57:0/3873405045 wait complete. 2023-10-19T14:02:02.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.739+0000 7f8c0cb05640 1 Processor -- start 2023-10-19T14:02:02.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.739+0000 7f8c0cb05640 1 -- start start 2023-10-19T14:02:02.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.739+0000 7f8c0cb05640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8c080ceb40 0x7f8c08158040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:02.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.739+0000 7f8c0cb05640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8c080cf340 0x7f8c0815cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:02.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.739+0000 7f8c0cb05640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8c0815b2a0 0x7f8c080ae860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:02.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.739+0000 7f8c0cb05640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c080b42a0 con 0x7f8c080cf340 2023-10-19T14:02:02.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.739+0000 7f8c0cb05640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c080b4540 con 0x7f8c0815b2a0 2023-10-19T14:02:02.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.739+0000 7f8c0cb05640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c080b4820 con 0x7f8c080ceb40 2023-10-19T14:02:02.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.740+0000 7f8bfffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8c080cf340 0x7f8c0815cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:02.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.740+0000 7f8bfffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8c080cf340 0x7f8c0815cf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:47400/0 (socket says 172.21.15.57:47400) 2023-10-19T14:02:02.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.740+0000 7f8bfffff640 1 -- 172.21.15.57:0/3762984520 learned_addr learned my addr 172.21.15.57:0/3762984520 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:02:02.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.740+0000 7f8c06d9d640 1 --2- 172.21.15.57:0/3762984520 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8c0815b2a0 0x7f8c080ae860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:02.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.740+0000 7f8c0659c640 1 --2- 172.21.15.57:0/3762984520 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8c080ceb40 0x7f8c08158040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:02.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.740+0000 7f8bfffff640 1 -- 172.21.15.57:0/3762984520 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8c080ceb40 msgr2=0x7f8c08158040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:02.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.740+0000 7f8bfffff640 1 --2- 172.21.15.57:0/3762984520 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8c080ceb40 0x7f8c08158040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:02.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.740+0000 7f8bfffff640 1 -- 172.21.15.57:0/3762984520 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8c0815b2a0 msgr2=0x7f8c080ae860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:02.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.740+0000 7f8bfffff640 1 --2- 172.21.15.57:0/3762984520 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8c0815b2a0 0x7f8c080ae860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:02.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.740+0000 7f8bfffff640 1 -- 172.21.15.57:0/3762984520 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c00009900 con 0x7f8c080cf340 2023-10-19T14:02:02.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.740+0000 7f8bfffff640 1 --2- 172.21.15.57:0/3762984520 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8c080cf340 0x7f8c0815cf60 secure :-1 s=READY pgs=1995 cs=0 l=1 rev1=1 crypto rx=0x7f8bf000d6d0 tx=0x7f8bf000db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:02:02.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.740+0000 7f8bfeffd640 1 -- 172.21.15.57:0/3762984520 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bf0005220 con 0x7f8c080cf340 2023-10-19T14:02:02.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.740+0000 7f8bfeffd640 1 -- 172.21.15.57:0/3762984520 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f8bf0005490 con 0x7f8c080cf340 2023-10-19T14:02:02.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.740+0000 7f8bfeffd640 1 -- 172.21.15.57:0/3762984520 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bf0009a80 con 0x7f8c080cf340 2023-10-19T14:02:02.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.740+0000 7f8c0cb05640 1 -- 172.21.15.57:0/3762984520 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c080b4b00 con 0x7f8c080cf340 2023-10-19T14:02:02.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.740+0000 7f8c0cb05640 1 -- 172.21.15.57:0/3762984520 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c080b50c0 con 0x7f8c080cf340 2023-10-19T14:02:02.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.741+0000 7f8c0cb05640 1 -- 172.21.15.57:0/3762984520 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c080690e0 con 0x7f8c080cf340 2023-10-19T14:02:02.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.742+0000 7f8bfeffd640 1 -- 172.21.15.57:0/3762984520 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 99) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f8bf00047a0 con 0x7f8c080cf340 2023-10-19T14:02:02.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.742+0000 7f8bfeffd640 1 --2- 172.21.15.57:0/3762984520 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f8be806af60 0x7f8be806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:02.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.742+0000 7f8bfeffd640 1 -- 172.21.15.57:0/3762984520 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5979+0+0 (secure 0 0 0) 0x7f8bf0089d00 con 0x7f8c080cf340 2023-10-19T14:02:02.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.742+0000 7f8c0659c640 1 --2- 172.21.15.57:0/3762984520 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f8be806af60 0x7f8be806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:02.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.743+0000 7f8c0659c640 1 --2- 172.21.15.57:0/3762984520 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f8be806af60 0x7f8be806d420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8bf8004ea0 tx=0x7f8bf8009040 comp rx=0 tx=0).ready entity=mgr.8315 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:02:02.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.746+0000 7f8bfeffd640 1 -- 172.21.15.57:0/3762984520 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8bf0054b30 con 0x7f8c080cf340 2023-10-19T14:02:02.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.926+0000 7f8c0cb05640 1 -- 172.21.15.57:0/3762984520 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8c080b0d00 con 0x7f8c080cf340 2023-10-19T14:02:02.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.927+0000 7f8bfeffd640 1 -- 172.21.15.57:0/3762984520 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v180) v1 ==== 74+0+14275 (secure 0 0 0) 0x7f8bf0017020 con 0x7f8c080cf340 2023-10-19T14:02:02.930 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:02:02.930 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":180,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T14:02:01.601969+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.179:6837/1102326590":"2023-10-20T14:01:58.439814+0000","172.21.15.179:0/3598128118":"2023-10-20T14:01:55.980598+0000","172.21.15.179:0/2108397676":"2023-10-20T14:01:55.980598+0000","172.21.15.179:6832/3795018457":"2023-10-20T14:01:55.980598+0000","172.21.15.179:0/1222555240":"2023-10-20T14:01:55.980598+0000","172.21.15.179:6835/1102326590":"2023-10-20T14:01:58.439814+0000","172.21.15.179:6833/3795018457":"2023-10-20T14:01:55.980598+0000","172.21.15.179:0/111884799":"2023-10-20T14:01:49.564546+0000","172.21.15.179:0/3699846367":"2023-10-20T14:01:55.980598+0000","172.21.15.179:0/966948837":"2023-10-20T14:01:49.564546+0000","172.21.15.179:0/1558797744":"2023-10-20T14:01:49.564546+0000","172.21.15.179:6833/2037224568":"2023-10-20T14:01:49.564546+0000","172.21.15.179:6832/2037224568":"2023-10-20T14:01:49.564546+0000","172.21.15.179:0/2965291378":"2023-10-20T14:01:49.564546+0000","172.21.15.179:0/472878921":"2023-10-20T14:01:49.564546+0000","172.21.15.179:0/3713016428":"2023-10-20T14:01:55.980598+0000","172.21.15.179:0/2930510955":"2023-10-20T14:01:55.980598+0000","172.21.15.179:0/2416623631":"2023-10-20T14:01:49.564546+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T14:02:02.932 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.929+0000 7f8c0cb05640 1 -- 172.21.15.57:0/3762984520 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f8be806af60 msgr2=0x7f8be806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:02.932 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.929+0000 7f8c0cb05640 1 --2- 172.21.15.57:0/3762984520 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f8be806af60 0x7f8be806d420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8bf8004ea0 tx=0x7f8bf8009040 comp rx=0 tx=0).stop 2023-10-19T14:02:02.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.929+0000 7f8c0cb05640 1 -- 172.21.15.57:0/3762984520 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8c080cf340 msgr2=0x7f8c0815cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:02.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.929+0000 7f8c0cb05640 1 --2- 172.21.15.57:0/3762984520 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8c080cf340 0x7f8c0815cf60 secure :-1 s=READY pgs=1995 cs=0 l=1 rev1=1 crypto rx=0x7f8bf000d6d0 tx=0x7f8bf000db90 comp rx=0 tx=0).stop 2023-10-19T14:02:02.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.930+0000 7f8c0cb05640 1 -- 172.21.15.57:0/3762984520 shutdown_connections 2023-10-19T14:02:02.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.930+0000 7f8c0cb05640 1 --2- 172.21.15.57:0/3762984520 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8c0815b2a0 0x7f8c080ae860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:02.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.930+0000 7f8c0cb05640 1 --2- 172.21.15.57:0/3762984520 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f8be806af60 0x7f8be806d420 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:02.934 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.930+0000 7f8c0cb05640 1 --2- 172.21.15.57:0/3762984520 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8c080cf340 0x7f8c0815cf60 unknown :-1 s=CLOSED pgs=1995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:02.934 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.930+0000 7f8c0cb05640 1 --2- 172.21.15.57:0/3762984520 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8c080ceb40 0x7f8c08158040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:02.934 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.930+0000 7f8c0cb05640 1 -- 172.21.15.57:0/3762984520 >> 172.21.15.57:0/3762984520 conn(0x7f8c080d0d60 msgr2=0x7f8c080c9e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:02:02.934 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.930+0000 7f8c0cb05640 1 -- 172.21.15.57:0/3762984520 shutdown_connections 2023-10-19T14:02:02.934 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:02.930+0000 7f8c0cb05640 1 -- 172.21.15.57:0/3762984520 wait complete. 2023-10-19T14:02:02.943 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-19T12:44:07.371186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-19T14:02:02.943 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-19T14:02:03.055 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.052+0000 7f7ae1087640 1 -- 172.21.15.57:0/2204828221 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7adc0b0530 msgr2=0x7f7adc0b09c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:03.056 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.052+0000 7f7ae1087640 1 --2- 172.21.15.57:0/2204828221 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7adc0b0530 0x7f7adc0b09c0 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 crypto rx=0x7f7ad4005fd0 tx=0x7f7ad402d400 comp rx=0 tx=0).stop 2023-10-19T14:02:03.056 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.053+0000 7f7ae1087640 1 -- 172.21.15.57:0/2204828221 shutdown_connections 2023-10-19T14:02:03.056 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.053+0000 7f7ae1087640 1 --2- 172.21.15.57:0/2204828221 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7adc0afa20 0x7f7adc158300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:03.056 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.053+0000 7f7ae1087640 1 --2- 172.21.15.57:0/2204828221 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7adc0b0530 0x7f7adc0b09c0 unknown :-1 s=CLOSED pgs=1997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:03.057 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.053+0000 7f7ae1087640 1 -- 172.21.15.57:0/2204828221 >> 172.21.15.57:0/2204828221 conn(0x7f7adc069930 msgr2=0x7f7adc069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:02:03.057 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.053+0000 7f7ae1087640 1 -- 172.21.15.57:0/2204828221 shutdown_connections 2023-10-19T14:02:03.057 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.053+0000 7f7ae1087640 1 -- 172.21.15.57:0/2204828221 wait complete. 2023-10-19T14:02:03.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.054+0000 7f7ae1087640 1 Processor -- start 2023-10-19T14:02:03.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.054+0000 7f7ae1087640 1 -- start start 2023-10-19T14:02:03.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.055+0000 7f7ae1087640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7adc0afa20 0x7f7adc1f25b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:03.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.055+0000 7f7ae1087640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7adc0b0530 0x7f7adc14cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:03.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.055+0000 7f7ae1087640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7adc0cbd20 0x7f7adc14a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:03.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.055+0000 7f7ae1087640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7adc1f3550 con 0x7f7adc0b0530 2023-10-19T14:02:03.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.055+0000 7f7ae1087640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7adc150a60 con 0x7f7adc0cbd20 2023-10-19T14:02:03.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.055+0000 7f7ae1087640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7adc14d000 con 0x7f7adc0afa20 2023-10-19T14:02:03.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.055+0000 7f7ad259c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7adc0b0530 0x7f7adc14cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:03.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.055+0000 7f7ad259c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7adc0b0530 0x7f7adc14cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:47414/0 (socket says 172.21.15.57:47414) 2023-10-19T14:02:03.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.055+0000 7f7ad259c640 1 -- 172.21.15.57:0/3822037703 learned_addr learned my addr 172.21.15.57:0/3822037703 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:02:03.062 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.055+0000 7f7adad9d640 1 --2- 172.21.15.57:0/3822037703 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7adc0afa20 0x7f7adc1f25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:03.062 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.055+0000 7f7adb59e640 1 --2- 172.21.15.57:0/3822037703 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7adc0cbd20 0x7f7adc14a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:03.062 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.055+0000 7f7ad259c640 1 -- 172.21.15.57:0/3822037703 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7adc0afa20 msgr2=0x7f7adc1f25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:03.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.055+0000 7f7ad259c640 1 --2- 172.21.15.57:0/3822037703 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7adc0afa20 0x7f7adc1f25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:03.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.055+0000 7f7ad259c640 1 -- 172.21.15.57:0/3822037703 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7adc0cbd20 msgr2=0x7f7adc14a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:03.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.055+0000 7f7ad259c640 1 --2- 172.21.15.57:0/3822037703 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7adc0cbd20 0x7f7adc14a600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:03.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.055+0000 7f7ad259c640 1 -- 172.21.15.57:0/3822037703 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ad400a950 con 0x7f7adc0b0530 2023-10-19T14:02:03.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.055+0000 7f7adad9d640 1 --2- 172.21.15.57:0/3822037703 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7adc0afa20 0x7f7adc1f25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:02:03.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.055+0000 7f7ad259c640 1 --2- 172.21.15.57:0/3822037703 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7adc0b0530 0x7f7adc14cac0 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7f7ac400d950 tx=0x7f7ac400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:02:03.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.056+0000 7f7ad3fff640 1 -- 172.21.15.57:0/3822037703 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ac4009c30 con 0x7f7adc0b0530 2023-10-19T14:02:03.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.056+0000 7f7ad3fff640 1 -- 172.21.15.57:0/3822037703 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f7ac4009e80 con 0x7f7adc0b0530 2023-10-19T14:02:03.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.056+0000 7f7ae1087640 1 -- 172.21.15.57:0/3822037703 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7adc145eb0 con 0x7f7adc0b0530 2023-10-19T14:02:03.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.056+0000 7f7ad3fff640 1 -- 172.21.15.57:0/3822037703 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ac40106c0 con 0x7f7adc0b0530 2023-10-19T14:02:03.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.056+0000 7f7ae1087640 1 -- 172.21.15.57:0/3822037703 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7adc1f2f90 con 0x7f7adc0b0530 2023-10-19T14:02:03.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.057+0000 7f7ae1087640 1 -- 172.21.15.57:0/3822037703 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7adc145c80 con 0x7f7adc0b0530 2023-10-19T14:02:03.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.057+0000 7f7ad3fff640 1 -- 172.21.15.57:0/3822037703 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 99) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f7ac4010820 con 0x7f7adc0b0530 2023-10-19T14:02:03.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.058+0000 7f7ad3fff640 1 --2- 172.21.15.57:0/3822037703 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f7abc06af60 0x7f7abc06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:03.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.058+0000 7f7ad3fff640 1 -- 172.21.15.57:0/3822037703 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5979+0+0 (secure 0 0 0) 0x7f7ac4089ea0 con 0x7f7adc0b0530 2023-10-19T14:02:03.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.058+0000 7f7adad9d640 1 --2- 172.21.15.57:0/3822037703 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f7abc06af60 0x7f7abc06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:03.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.059+0000 7f7adad9d640 1 --2- 172.21.15.57:0/3822037703 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f7abc06af60 0x7f7abc06d420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f7acc001660 tx=0x7f7acc009040 comp rx=0 tx=0).ready entity=mgr.8315 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:02:03.067 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.062+0000 7f7ad3fff640 1 -- 172.21.15.57:0/3822037703 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7adc145c80 con 0x7f7adc0b0530 2023-10-19T14:02:03.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.244+0000 7f7ae1087640 1 -- 172.21.15.57:0/3822037703 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7adc145c80 con 0x7f7adc0b0530 2023-10-19T14:02:03.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.244+0000 7f7ad3fff640 1 -- 172.21.15.57:0/3822037703 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v180) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7adc145c80 con 0x7f7adc0b0530 2023-10-19T14:02:03.247 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-19T14:02:03.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.246+0000 7f7ae1087640 1 -- 172.21.15.57:0/3822037703 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f7abc06af60 msgr2=0x7f7abc06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:03.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.246+0000 7f7ae1087640 1 --2- 172.21.15.57:0/3822037703 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f7abc06af60 0x7f7abc06d420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f7acc001660 tx=0x7f7acc009040 comp rx=0 tx=0).stop 2023-10-19T14:02:03.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.246+0000 7f7ae1087640 1 -- 172.21.15.57:0/3822037703 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7adc0b0530 msgr2=0x7f7adc14cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:03.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.246+0000 7f7ae1087640 1 --2- 172.21.15.57:0/3822037703 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7adc0b0530 0x7f7adc14cac0 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7f7ac400d950 tx=0x7f7ac400de10 comp rx=0 tx=0).stop 2023-10-19T14:02:03.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.246+0000 7f7ae1087640 1 -- 172.21.15.57:0/3822037703 shutdown_connections 2023-10-19T14:02:03.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.246+0000 7f7ae1087640 1 --2- 172.21.15.57:0/3822037703 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7adc0cbd20 0x7f7adc14a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:03.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.246+0000 7f7ae1087640 1 --2- 172.21.15.57:0/3822037703 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f7abc06af60 0x7f7abc06d420 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:03.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.246+0000 7f7ae1087640 1 --2- 172.21.15.57:0/3822037703 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7adc0b0530 0x7f7adc14cac0 unknown :-1 s=CLOSED pgs=1998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:03.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.246+0000 7f7ae1087640 1 --2- 172.21.15.57:0/3822037703 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7adc0afa20 0x7f7adc1f25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:03.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.246+0000 7f7ae1087640 1 -- 172.21.15.57:0/3822037703 >> 172.21.15.57:0/3822037703 conn(0x7f7adc069930 msgr2=0x7f7adc0ac7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:02:03.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.246+0000 7f7ae1087640 1 -- 172.21.15.57:0/3822037703 shutdown_connections 2023-10-19T14:02:03.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.246+0000 7f7ae1087640 1 -- 172.21.15.57:0/3822037703 wait complete. 2023-10-19T14:02:03.260 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart' 2023-10-19T14:02:03.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.367+0000 7fdfe795e640 1 -- 172.21.15.57:0/2233198605 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdfe0141690 msgr2=0x7fdfe00703e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:03.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.367+0000 7fdfe795e640 1 --2- 172.21.15.57:0/2233198605 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdfe0141690 0x7fdfe00703e0 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 crypto rx=0x7fdfe00b20f0 tx=0x7fdfd002bd30 comp rx=0 tx=0).stop 2023-10-19T14:02:03.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.367+0000 7fdfe795e640 1 -- 172.21.15.57:0/2233198605 shutdown_connections 2023-10-19T14:02:03.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.367+0000 7fdfe795e640 1 --2- 172.21.15.57:0/2233198605 >> v2:172.21.15.179:3300/0 conn(0x7fdfe0070920 0x7fdfe00affe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:03.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.367+0000 7fdfe795e640 1 --2- 172.21.15.57:0/2233198605 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdfe0141690 0x7fdfe00703e0 unknown :-1 s=CLOSED pgs=1999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:03.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.367+0000 7fdfe795e640 1 -- 172.21.15.57:0/2233198605 >> 172.21.15.57:0/2233198605 conn(0x7fdfe0069930 msgr2=0x7fdfe0069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:02:03.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.367+0000 7fdfe795e640 1 -- 172.21.15.57:0/2233198605 shutdown_connections 2023-10-19T14:02:03.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.367+0000 7fdfe795e640 1 -- 172.21.15.57:0/2233198605 wait complete. 2023-10-19T14:02:03.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.368+0000 7fdfe795e640 1 Processor -- start 2023-10-19T14:02:03.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.368+0000 7fdfe795e640 1 -- start start 2023-10-19T14:02:03.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.369+0000 7fdfe795e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdfe0070920 0x7fdfe01d8290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:03.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.369+0000 7fdfe795e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdfe013e520 0x7fdfe00a77a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:03.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.369+0000 7fdfe795e640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdfe0141690 0x7fdfe00ad6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:03.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.369+0000 7fdfe795e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfe00c8c40 con 0x7fdfe0070920 2023-10-19T14:02:03.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.369+0000 7fdfe795e640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfe00a9430 con 0x7fdfe013e520 2023-10-19T14:02:03.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.369+0000 7fdfe795e640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfe00c80c0 con 0x7fdfe0141690 2023-10-19T14:02:03.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.369+0000 7fdfe56fa640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdfe0070920 0x7fdfe01d8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:03.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.369+0000 7fdfe56fa640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdfe0070920 0x7fdfe01d8290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:47420/0 (socket says 172.21.15.57:47420) 2023-10-19T14:02:03.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.369+0000 7fdfe56fa640 1 -- 172.21.15.57:0/2538232676 learned_addr learned my addr 172.21.15.57:0/2538232676 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:02:03.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.369+0000 7fdfe5efb640 1 --2- 172.21.15.57:0/2538232676 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdfe0141690 0x7fdfe00ad6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:03.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.369+0000 7fdfe4ef9640 1 --2- 172.21.15.57:0/2538232676 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdfe013e520 0x7fdfe00a77a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:03.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.369+0000 7fdfe56fa640 1 -- 172.21.15.57:0/2538232676 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdfe0141690 msgr2=0x7fdfe00ad6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:03.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.369+0000 7fdfe56fa640 1 --2- 172.21.15.57:0/2538232676 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdfe0141690 0x7fdfe00ad6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:03.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.369+0000 7fdfe56fa640 1 -- 172.21.15.57:0/2538232676 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdfe013e520 msgr2=0x7fdfe00a77a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:03.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.369+0000 7fdfe56fa640 1 --2- 172.21.15.57:0/2538232676 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdfe013e520 0x7fdfe00a77a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:03.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.369+0000 7fdfe56fa640 1 -- 172.21.15.57:0/2538232676 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfd0009560 con 0x7fdfe0070920 2023-10-19T14:02:03.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.369+0000 7fdfe56fa640 1 --2- 172.21.15.57:0/2538232676 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdfe0070920 0x7fdfe01d8290 secure :-1 s=READY pgs=2000 cs=0 l=1 rev1=1 crypto rx=0x7fdfdc00c410 tx=0x7fdfdc00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:02:03.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.369+0000 7fdfd5ffb640 1 -- 172.21.15.57:0/2538232676 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfdc0035f0 con 0x7fdfe0070920 2023-10-19T14:02:03.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.369+0000 7fdfd5ffb640 1 -- 172.21.15.57:0/2538232676 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fdfdc003750 con 0x7fdfe0070920 2023-10-19T14:02:03.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.369+0000 7fdfe795e640 1 -- 172.21.15.57:0/2538232676 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfe00c83a0 con 0x7fdfe0070920 2023-10-19T14:02:03.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.369+0000 7fdfd5ffb640 1 -- 172.21.15.57:0/2538232676 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfdc003ba0 con 0x7fdfe0070920 2023-10-19T14:02:03.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.370+0000 7fdfe795e640 1 -- 172.21.15.57:0/2538232676 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfe00c8960 con 0x7fdfe0070920 2023-10-19T14:02:03.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.371+0000 7fdfe795e640 1 -- 172.21.15.57:0/2538232676 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfe00690e0 con 0x7fdfe0070920 2023-10-19T14:02:03.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.371+0000 7fdfd5ffb640 1 -- 172.21.15.57:0/2538232676 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 99) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fdfdc003d00 con 0x7fdfe0070920 2023-10-19T14:02:03.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.371+0000 7fdfd5ffb640 1 --2- 172.21.15.57:0/2538232676 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7fdfc406af60 0x7fdfc406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:03.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.371+0000 7fdfd5ffb640 1 -- 172.21.15.57:0/2538232676 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5979+0+0 (secure 0 0 0) 0x7fdfdc089030 con 0x7fdfe0070920 2023-10-19T14:02:03.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.371+0000 7fdfe4ef9640 1 --2- 172.21.15.57:0/2538232676 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7fdfc406af60 0x7fdfc406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:03.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.372+0000 7fdfe4ef9640 1 --2- 172.21.15.57:0/2538232676 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7fdfc406af60 0x7fdfc406d420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fdfd0035000 tx=0x7fdfd0004730 comp rx=0 tx=0).ready entity=mgr.8315 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:02:03.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.375+0000 7fdfd5ffb640 1 -- 172.21.15.57:0/2538232676 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fdfdc053f70 con 0x7fdfe0070920 2023-10-19T14:02:03.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:03.620+0000 7fdfe795e640 1 -- 172.21.15.57:0/2538232676 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart"]} v 0) v1 -- 0x7fdfe006b140 con 0x7fdfe0070920 2023-10-19T14:02:04.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:04.620+0000 7fdfd5ffb640 1 -- 172.21.15.57:0/2538232676 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart"]}]=0 v2539) v1 ==== 168+0+0 (secure 0 0 0) 0x7fdfdc0170e0 con 0x7fdfe0070920 2023-10-19T14:02:04.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:04.623+0000 7fdfe795e640 1 -- 172.21.15.57:0/2538232676 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7fdfc406af60 msgr2=0x7fdfc406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:04.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:04.623+0000 7fdfe795e640 1 --2- 172.21.15.57:0/2538232676 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7fdfc406af60 0x7fdfc406d420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fdfd0035000 tx=0x7fdfd0004730 comp rx=0 tx=0).stop 2023-10-19T14:02:04.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:04.623+0000 7fdfe795e640 1 -- 172.21.15.57:0/2538232676 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdfe0070920 msgr2=0x7fdfe01d8290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:04.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:04.623+0000 7fdfe795e640 1 --2- 172.21.15.57:0/2538232676 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdfe0070920 0x7fdfe01d8290 secure :-1 s=READY pgs=2000 cs=0 l=1 rev1=1 crypto rx=0x7fdfdc00c410 tx=0x7fdfdc00c8d0 comp rx=0 tx=0).stop 2023-10-19T14:02:04.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:04.623+0000 7fdfe795e640 1 -- 172.21.15.57:0/2538232676 shutdown_connections 2023-10-19T14:02:04.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:04.623+0000 7fdfe795e640 1 --2- 172.21.15.57:0/2538232676 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdfe0141690 0x7fdfe00ad6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:04.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:04.623+0000 7fdfe795e640 1 --2- 172.21.15.57:0/2538232676 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdfe013e520 0x7fdfe00a77a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:04.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:04.623+0000 7fdfe795e640 1 --2- 172.21.15.57:0/2538232676 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7fdfc406af60 0x7fdfc406d420 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:04.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:04.623+0000 7fdfe795e640 1 --2- 172.21.15.57:0/2538232676 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdfe0070920 0x7fdfe01d8290 unknown :-1 s=CLOSED pgs=2000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:04.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:04.623+0000 7fdfe795e640 1 -- 172.21.15.57:0/2538232676 >> 172.21.15.57:0/2538232676 conn(0x7fdfe0069930 msgr2=0x7fdfe00aef90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:02:04.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:04.623+0000 7fdfe795e640 1 -- 172.21.15.57:0/2538232676 shutdown_connections 2023-10-19T14:02:04.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:04.623+0000 7fdfe795e640 1 -- 172.21.15.57:0/2538232676 wait complete. 2023-10-19T14:02:04.640 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2023-10-19T14:02:04.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:04.755+0000 7fcdc7bd0640 1 -- 172.21.15.57:0/989243990 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcdc013d360 msgr2=0x7fcdc015a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:04.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:04.755+0000 7fcdc7bd0640 1 --2- 172.21.15.57:0/989243990 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcdc013d360 0x7fcdc015a9f0 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x7fcdc006f010 tx=0x7fcdb802c630 comp rx=0 tx=0).stop 2023-10-19T14:02:04.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:04.755+0000 7fcdc7bd0640 1 -- 172.21.15.57:0/989243990 shutdown_connections 2023-10-19T14:02:04.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:04.755+0000 7fcdc7bd0640 1 --2- 172.21.15.57:0/989243990 >> v2:172.21.15.179:3300/0 conn(0x7fcdc00c9ed0 0x7fcdc006b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:04.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:04.755+0000 7fcdc7bd0640 1 --2- 172.21.15.57:0/989243990 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcdc013d360 0x7fcdc015a9f0 unknown :-1 s=CLOSED pgs=2001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:04.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:04.755+0000 7fcdc7bd0640 1 -- 172.21.15.57:0/989243990 >> 172.21.15.57:0/989243990 conn(0x7fcdc0069930 msgr2=0x7fcdc0069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:02:04.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:04.755+0000 7fcdc7bd0640 1 -- 172.21.15.57:0/989243990 shutdown_connections 2023-10-19T14:02:04.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:04.755+0000 7fcdc7bd0640 1 -- 172.21.15.57:0/989243990 wait complete. 2023-10-19T14:02:04.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:04.756+0000 7fcdc7bd0640 1 Processor -- start 2023-10-19T14:02:04.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:04.757+0000 7fcdc7bd0640 1 -- start start 2023-10-19T14:02:04.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:04.757+0000 7fcdc7bd0640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcdc00c9ed0 0x7fcdc0158b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:04.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:04.757+0000 7fcdc7bd0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcdc013d360 0x7fcdc00aa8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:04.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:04.757+0000 7fcdc7bd0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcdc0140660 0x7fcdc00afe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:04.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:04.757+0000 7fcdc7bd0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdc0159080 con 0x7fcdc0140660 2023-10-19T14:02:04.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:04.757+0000 7fcdc7bd0640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdc00abf90 con 0x7fcdc013d360 2023-10-19T14:02:04.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:04.757+0000 7fcdc7bd0640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdc00ab380 con 0x7fcdc00c9ed0 2023-10-19T14:02:04.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:04.757+0000 7fcdc616d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcdc0140660 0x7fcdc00afe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:04.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:04.757+0000 7fcdc616d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcdc0140660 0x7fcdc00afe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:47434/0 (socket says 172.21.15.57:47434) 2023-10-19T14:02:04.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:04.757+0000 7fcdc616d640 1 -- 172.21.15.57:0/4231790573 learned_addr learned my addr 172.21.15.57:0/4231790573 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:02:04.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:04.758+0000 7fcdc516b640 1 --2- 172.21.15.57:0/4231790573 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcdc013d360 0x7fcdc00aa8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:04.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:04.758+0000 7fcdc596c640 1 --2- 172.21.15.57:0/4231790573 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcdc00c9ed0 0x7fcdc0158b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:04.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:04.758+0000 7fcdc616d640 1 -- 172.21.15.57:0/4231790573 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcdc00c9ed0 msgr2=0x7fcdc0158b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:04.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:04.758+0000 7fcdc616d640 1 --2- 172.21.15.57:0/4231790573 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcdc00c9ed0 0x7fcdc0158b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:04.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:04.758+0000 7fcdc616d640 1 -- 172.21.15.57:0/4231790573 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcdc013d360 msgr2=0x7fcdc00aa8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:04.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:04.758+0000 7fcdc616d640 1 --2- 172.21.15.57:0/4231790573 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcdc013d360 0x7fcdc00aa8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:04.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:04.758+0000 7fcdc616d640 1 -- 172.21.15.57:0/4231790573 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdb8009df0 con 0x7fcdc0140660 2023-10-19T14:02:04.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:04.758+0000 7fcdc516b640 1 --2- 172.21.15.57:0/4231790573 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcdc013d360 0x7fcdc00aa8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:02:04.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:04.758+0000 7fcdc616d640 1 --2- 172.21.15.57:0/4231790573 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcdc0140660 0x7fcdc00afe10 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 crypto rx=0x7fcdb000cce0 tx=0x7fcdb000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:02:04.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:04.758+0000 7fcdbe7fc640 1 -- 172.21.15.57:0/4231790573 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdb000f450 con 0x7fcdc0140660 2023-10-19T14:02:04.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:04.758+0000 7fcdc7bd0640 1 -- 172.21.15.57:0/4231790573 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdc00ab660 con 0x7fcdc0140660 2023-10-19T14:02:04.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:04.758+0000 7fcdbe7fc640 1 -- 172.21.15.57:0/4231790573 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fcdb000f6c0 con 0x7fcdc0140660 2023-10-19T14:02:04.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:04.758+0000 7fcdbe7fc640 1 -- 172.21.15.57:0/4231790573 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdb000fad0 con 0x7fcdc0140660 2023-10-19T14:02:04.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:04.758+0000 7fcdc7bd0640 1 -- 172.21.15.57:0/4231790573 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdc00abc20 con 0x7fcdc0140660 2023-10-19T14:02:04.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:04.759+0000 7fcdc7bd0640 1 -- 172.21.15.57:0/4231790573 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdc00ac140 con 0x7fcdc0140660 2023-10-19T14:02:04.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:04.760+0000 7fcdbe7fc640 1 -- 172.21.15.57:0/4231790573 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 99) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fcdb000fc30 con 0x7fcdc0140660 2023-10-19T14:02:04.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:04.760+0000 7fcdbe7fc640 1 --2- 172.21.15.57:0/4231790573 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7fcdac06af60 0x7fcdac06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:04.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:04.760+0000 7fcdbe7fc640 1 -- 172.21.15.57:0/4231790573 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5979+0+0 (secure 0 0 0) 0x7fcdb008a610 con 0x7fcdc0140660 2023-10-19T14:02:04.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:04.760+0000 7fcdc596c640 1 --2- 172.21.15.57:0/4231790573 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7fcdac06af60 0x7fcdac06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:04.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:04.761+0000 7fcdc596c640 1 --2- 172.21.15.57:0/4231790573 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7fcdac06af60 0x7fcdac06d420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fcdb8035000 tx=0x7fcdb8004730 comp rx=0 tx=0).ready entity=mgr.8315 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:02:04.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:04.764+0000 7fcdbe7fc640 1 -- 172.21.15.57:0/4231790573 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fcdc00ac140 con 0x7fcdc0140660 2023-10-19T14:02:04.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:04.941+0000 7fcdc7bd0640 1 -- 172.21.15.57:0/4231790573 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "config dump"} v 0) v1 -- 0x7fcdc00ac100 con 0x7fcdc0140660 2023-10-19T14:02:04.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:04.942+0000 7fcdbe7fc640 1 -- 172.21.15.57:0/4231790573 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v66) v1 ==== 59+0+2220 (secure 0 0 0) 0x7fcdb001d8c0 con 0x7fcdc0140660 2023-10-19T14:02:04.945 INFO:teuthology.orchestra.run.smithi057.stdout:WHO MASK LEVEL OPTION VALUE RO 2023-10-19T14:02:04.945 INFO:teuthology.orchestra.run.smithi057.stdout:mon advanced mon_op_complaint_time 120 2023-10-19T14:02:04.945 INFO:teuthology.orchestra.run.smithi057.stdout:osd advanced osd_op_complaint_time 180.000000 2023-10-19T14:02:04.945 INFO:teuthology.orchestra.run.smithi057.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 4775.995373 2023-10-19T14:02:04.946 INFO:teuthology.orchestra.run.smithi057.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 4208.118410 2023-10-19T14:02:04.946 INFO:teuthology.orchestra.run.smithi057.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 4541.123030 2023-10-19T14:02:04.946 INFO:teuthology.orchestra.run.smithi057.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 4877.592206 2023-10-19T14:02:04.946 INFO:teuthology.orchestra.run.smithi057.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 5019.748032 2023-10-19T14:02:04.947 INFO:teuthology.orchestra.run.smithi057.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 5000.074376 2023-10-19T14:02:04.947 INFO:teuthology.orchestra.run.smithi057.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 5153.479439 2023-10-19T14:02:04.947 INFO:teuthology.orchestra.run.smithi057.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 4977.363845 2023-10-19T14:02:04.947 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced debug_mds 20/20 2023-10-19T14:02:04.948 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced debug_mds_balancer 20/20 2023-10-19T14:02:04.948 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced debug_ms 1/1 2023-10-19T14:02:04.948 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced mds_bal_fragment_size_max 10000 2023-10-19T14:02:04.948 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced mds_bal_merge_size 5 2023-10-19T14:02:04.948 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced mds_bal_split_bits 3 2023-10-19T14:02:04.949 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced mds_bal_split_size 100 2023-10-19T14:02:04.949 INFO:teuthology.orchestra.run.smithi057.stdout:mds dev mds_debug_frag true 2023-10-19T14:02:04.949 INFO:teuthology.orchestra.run.smithi057.stdout:mds dev mds_debug_scatterstat true 2023-10-19T14:02:04.949 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced mds_op_complaint_time 180.000000 2023-10-19T14:02:04.949 INFO:teuthology.orchestra.run.smithi057.stdout:mds dev mds_verify_scatter true 2023-10-19T14:02:04.950 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced osd_op_complaint_time 180.000000 2023-10-19T14:02:04.950 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced rados_mon_op_timeout 900 2023-10-19T14:02:04.950 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced rados_osd_op_timeout 900 2023-10-19T14:02:04.950 INFO:teuthology.orchestra.run.smithi057.stdout:client advanced client_mount_timeout 600 2023-10-19T14:02:04.950 INFO:teuthology.orchestra.run.smithi057.stdout:client advanced debug_client 20/20 2023-10-19T14:02:04.951 INFO:teuthology.orchestra.run.smithi057.stdout:client advanced debug_ms 1/1 2023-10-19T14:02:04.951 INFO:teuthology.orchestra.run.smithi057.stdout:client advanced rados_mon_op_timeout 900 2023-10-19T14:02:04.951 INFO:teuthology.orchestra.run.smithi057.stdout:client advanced rados_osd_op_timeout 900 2023-10-19T14:02:04.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:04.944+0000 7fcdc7bd0640 1 -- 172.21.15.57:0/4231790573 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7fcdac06af60 msgr2=0x7fcdac06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:04.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:04.944+0000 7fcdc7bd0640 1 --2- 172.21.15.57:0/4231790573 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7fcdac06af60 0x7fcdac06d420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fcdb8035000 tx=0x7fcdb8004730 comp rx=0 tx=0).stop 2023-10-19T14:02:04.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:04.944+0000 7fcdc7bd0640 1 -- 172.21.15.57:0/4231790573 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcdc0140660 msgr2=0x7fcdc00afe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:04.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:04.944+0000 7fcdc7bd0640 1 --2- 172.21.15.57:0/4231790573 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcdc0140660 0x7fcdc00afe10 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 crypto rx=0x7fcdb000cce0 tx=0x7fcdb000a620 comp rx=0 tx=0).stop 2023-10-19T14:02:04.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:04.944+0000 7fcdc7bd0640 1 -- 172.21.15.57:0/4231790573 shutdown_connections 2023-10-19T14:02:04.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:04.944+0000 7fcdc7bd0640 1 --2- 172.21.15.57:0/4231790573 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7fcdac06af60 0x7fcdac06d420 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:04.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:04.944+0000 7fcdc7bd0640 1 --2- 172.21.15.57:0/4231790573 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcdc0140660 0x7fcdc00afe10 unknown :-1 s=CLOSED pgs=2002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:04.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:04.944+0000 7fcdc7bd0640 1 --2- 172.21.15.57:0/4231790573 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcdc013d360 0x7fcdc00aa8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:04.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:04.944+0000 7fcdc7bd0640 1 --2- 172.21.15.57:0/4231790573 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcdc00c9ed0 0x7fcdc0158b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:04.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:04.944+0000 7fcdc7bd0640 1 -- 172.21.15.57:0/4231790573 >> 172.21.15.57:0/4231790573 conn(0x7fcdc0069930 msgr2=0x7fcdc00cbdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:02:04.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:04.944+0000 7fcdc7bd0640 1 -- 172.21.15.57:0/4231790573 shutdown_connections 2023-10-19T14:02:04.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:04.944+0000 7fcdc7bd0640 1 -- 172.21.15.57:0/4231790573 wait complete. 2023-10-19T14:02:04.956 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-19T14:02:05.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.066+0000 7f6a93844640 1 -- 172.21.15.57:0/576029630 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6a8c0ab500 msgr2=0x7f6a8c0ab9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:05.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.066+0000 7f6a93844640 1 --2- 172.21.15.57:0/576029630 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6a8c0ab500 0x7f6a8c0ab9b0 secure :-1 s=READY pgs=2003 cs=0 l=1 rev1=1 crypto rx=0x7f6a7c005fd0 tx=0x7f6a7c02bd90 comp rx=0 tx=0).stop 2023-10-19T14:02:05.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.066+0000 7f6a93844640 1 -- 172.21.15.57:0/576029630 shutdown_connections 2023-10-19T14:02:05.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.066+0000 7f6a93844640 1 --2- 172.21.15.57:0/576029630 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6a8c0ab500 0x7f6a8c0ab9b0 unknown :-1 s=CLOSED pgs=2003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:05.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.066+0000 7f6a93844640 1 --2- 172.21.15.57:0/576029630 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6a8c0c9dc0 0x7f6a8c06b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:05.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.066+0000 7f6a93844640 1 -- 172.21.15.57:0/576029630 >> 172.21.15.57:0/576029630 conn(0x7f6a8c069930 msgr2=0x7f6a8c069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:02:05.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.066+0000 7f6a93844640 1 -- 172.21.15.57:0/576029630 shutdown_connections 2023-10-19T14:02:05.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.066+0000 7f6a93844640 1 -- 172.21.15.57:0/576029630 wait complete. 2023-10-19T14:02:05.072 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.067+0000 7f6a93844640 1 Processor -- start 2023-10-19T14:02:05.072 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.067+0000 7f6a93844640 1 -- start start 2023-10-19T14:02:05.073 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.068+0000 7f6a93844640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6a8c06af00 0x7f6a8c139200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:05.073 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.068+0000 7f6a93844640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6a8c0ab500 0x7f6a8c0b11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:05.073 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.068+0000 7f6a93844640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6a8c0c9dc0 0x7f6a8c14f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:05.073 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.068+0000 7f6a93844640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a8c0b1830 con 0x7f6a8c0c9dc0 2023-10-19T14:02:05.073 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.068+0000 7f6a93844640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a8c0ac690 con 0x7f6a8c0ab500 2023-10-19T14:02:05.074 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.068+0000 7f6a93844640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a8c0b0350 con 0x7f6a8c06af00 2023-10-19T14:02:05.074 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.068+0000 7f6a91de1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6a8c0c9dc0 0x7f6a8c14f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:05.074 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.068+0000 7f6a91de1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6a8c0c9dc0 0x7f6a8c14f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:47458/0 (socket says 172.21.15.57:47458) 2023-10-19T14:02:05.074 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.068+0000 7f6a91de1640 1 -- 172.21.15.57:0/3417851797 learned_addr learned my addr 172.21.15.57:0/3417851797 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:02:05.074 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.068+0000 7f6a915e0640 1 --2- 172.21.15.57:0/3417851797 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6a8c06af00 0x7f6a8c139200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:05.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.068+0000 7f6a90ddf640 1 --2- 172.21.15.57:0/3417851797 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6a8c0ab500 0x7f6a8c0b11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:05.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.068+0000 7f6a91de1640 1 -- 172.21.15.57:0/3417851797 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6a8c06af00 msgr2=0x7f6a8c139200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:05.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.068+0000 7f6a91de1640 1 --2- 172.21.15.57:0/3417851797 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6a8c06af00 0x7f6a8c139200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:05.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.068+0000 7f6a91de1640 1 -- 172.21.15.57:0/3417851797 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6a8c0ab500 msgr2=0x7f6a8c0b11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:05.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.068+0000 7f6a91de1640 1 --2- 172.21.15.57:0/3417851797 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6a8c0ab500 0x7f6a8c0b11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:05.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.068+0000 7f6a91de1640 1 -- 172.21.15.57:0/3417851797 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a7c009560 con 0x7f6a8c0c9dc0 2023-10-19T14:02:05.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.068+0000 7f6a90ddf640 1 --2- 172.21.15.57:0/3417851797 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6a8c0ab500 0x7f6a8c0b11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:02:05.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.068+0000 7f6a915e0640 1 --2- 172.21.15.57:0/3417851797 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6a8c06af00 0x7f6a8c139200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:02:05.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.069+0000 7f6a91de1640 1 --2- 172.21.15.57:0/3417851797 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6a8c0c9dc0 0x7f6a8c14f740 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7f6a7c006010 tx=0x7f6a7c02dce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:02:05.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.069+0000 7f6a89ffb640 1 -- 172.21.15.57:0/3417851797 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a7c005690 con 0x7f6a8c0c9dc0 2023-10-19T14:02:05.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.069+0000 7f6a89ffb640 1 -- 172.21.15.57:0/3417851797 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f6a7c003800 con 0x7f6a8c0c9dc0 2023-10-19T14:02:05.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.069+0000 7f6a89ffb640 1 -- 172.21.15.57:0/3417851797 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a7c003bb0 con 0x7f6a8c0c9dc0 2023-10-19T14:02:05.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.069+0000 7f6a93844640 1 -- 172.21.15.57:0/3417851797 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a8c0b0630 con 0x7f6a8c0c9dc0 2023-10-19T14:02:05.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.069+0000 7f6a93844640 1 -- 172.21.15.57:0/3417851797 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a8c158e80 con 0x7f6a8c0c9dc0 2023-10-19T14:02:05.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.070+0000 7f6a93844640 1 -- 172.21.15.57:0/3417851797 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a8c0b0490 con 0x7f6a8c0c9dc0 2023-10-19T14:02:05.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.070+0000 7f6a89ffb640 1 -- 172.21.15.57:0/3417851797 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 99) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6a7c003d10 con 0x7f6a8c0c9dc0 2023-10-19T14:02:05.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.071+0000 7f6a89ffb640 1 --2- 172.21.15.57:0/3417851797 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f6a7806af60 0x7f6a7806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:05.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.071+0000 7f6a89ffb640 1 -- 172.21.15.57:0/3417851797 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5979+0+0 (secure 0 0 0) 0x7f6a7c0ac590 con 0x7f6a8c0c9dc0 2023-10-19T14:02:05.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.071+0000 7f6a915e0640 1 --2- 172.21.15.57:0/3417851797 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f6a7806af60 0x7f6a7806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:05.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.072+0000 7f6a915e0640 1 --2- 172.21.15.57:0/3417851797 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f6a7806af60 0x7f6a7806d420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6a840045b0 tx=0x7f6a84009040 comp rx=0 tx=0).ready entity=mgr.8315 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:02:05.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.075+0000 7f6a89ffb640 1 -- 172.21.15.57:0/3417851797 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6a8c0b0490 con 0x7f6a8c0c9dc0 2023-10-19T14:02:05.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.249+0000 7f6a93844640 1 -- 172.21.15.57:0/3417851797 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f6a8c0b0490 con 0x7f6a8c0c9dc0 2023-10-19T14:02:05.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.252+0000 7f6a89ffb640 1 -- 172.21.15.57:0/3417851797 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v67)=0 v67) v1 ==== 122+0+0 (secure 0 0 0) 0x7f6a8c0b0490 con 0x7f6a8c0c9dc0 2023-10-19T14:02:05.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.254+0000 7f6a93844640 1 -- 172.21.15.57:0/3417851797 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f6a7806af60 msgr2=0x7f6a7806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:05.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.255+0000 7f6a93844640 1 --2- 172.21.15.57:0/3417851797 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f6a7806af60 0x7f6a7806d420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6a840045b0 tx=0x7f6a84009040 comp rx=0 tx=0).stop 2023-10-19T14:02:05.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.255+0000 7f6a93844640 1 -- 172.21.15.57:0/3417851797 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6a8c0c9dc0 msgr2=0x7f6a8c14f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:05.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.255+0000 7f6a93844640 1 --2- 172.21.15.57:0/3417851797 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6a8c0c9dc0 0x7f6a8c14f740 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7f6a7c006010 tx=0x7f6a7c02dce0 comp rx=0 tx=0).stop 2023-10-19T14:02:05.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.255+0000 7f6a93844640 1 -- 172.21.15.57:0/3417851797 shutdown_connections 2023-10-19T14:02:05.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.255+0000 7f6a93844640 1 --2- 172.21.15.57:0/3417851797 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f6a7806af60 0x7f6a7806d420 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:05.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.255+0000 7f6a93844640 1 --2- 172.21.15.57:0/3417851797 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6a8c0c9dc0 0x7f6a8c14f740 unknown :-1 s=CLOSED pgs=2004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:05.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.255+0000 7f6a93844640 1 --2- 172.21.15.57:0/3417851797 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6a8c0ab500 0x7f6a8c0b11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:05.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.255+0000 7f6a93844640 1 --2- 172.21.15.57:0/3417851797 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6a8c06af00 0x7f6a8c139200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:05.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.255+0000 7f6a93844640 1 -- 172.21.15.57:0/3417851797 >> 172.21.15.57:0/3417851797 conn(0x7f6a8c069930 msgr2=0x7f6a8c069d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:02:05.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.255+0000 7f6a93844640 1 -- 172.21.15.57:0/3417851797 shutdown_connections 2023-10-19T14:02:05.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.255+0000 7f6a93844640 1 -- 172.21.15.57:0/3417851797 wait complete. 2023-10-19T14:02:05.267 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-19T14:02:05.268 DEBUG:teuthology.orchestra.run.smithi057:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-19T14:02:05.314 DEBUG:teuthology.orchestra.run.smithi179:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-19T14:02:05.353 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-19T14:02:05.353 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T14:02:05.354 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T14:02:05.369 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055614,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=358 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248260k,nr_inodes=812065,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-19T14:02:05.370 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-19T14:02:05.370 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T14:02:05.410 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054833,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=284 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3247632k,nr_inodes=811908,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-19T14:02:05.411 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T14:02:05.411 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T14:02:05.426 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055614,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=358 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248260k,nr_inodes=812065,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-19T14:02:05.427 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-19T14:02:05.427 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T14:02:05.467 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054833,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=284 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3247632k,nr_inodes=811908,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-19T14:02:05.468 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-19T14:02:05.468 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T14:02:05.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.576+0000 7f5d2964f640 1 -- 172.21.15.57:0/3960385265 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5d0c003970 msgr2=0x7f5d0c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:05.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.576+0000 7f5d2964f640 1 --2- 172.21.15.57:0/3960385265 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5d0c003970 0x7f5d0c003d90 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c00c290 tx=0x7f5d0c0057f0 comp rx=0 tx=0).stop 2023-10-19T14:02:05.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.576+0000 7f5d237fe640 1 -- 172.21.15.57:0/3960385265 reap_dead start 2023-10-19T14:02:05.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.576+0000 7f5d2964f640 1 -- 172.21.15.57:0/3960385265 shutdown_connections 2023-10-19T14:02:05.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.576+0000 7f5d2964f640 1 -- 172.21.15.57:0/3960385265 >> 172.21.15.57:0/3960385265 conn(0x7f5d24069930 msgr2=0x7f5d24069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:02:05.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.576+0000 7f5d2964f640 1 -- 172.21.15.57:0/3960385265 shutdown_connections 2023-10-19T14:02:05.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.576+0000 7f5d2964f640 1 -- 172.21.15.57:0/3960385265 wait complete. 2023-10-19T14:02:05.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.577+0000 7f5d2964f640 1 Processor -- start 2023-10-19T14:02:05.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.577+0000 7f5d2964f640 1 -- start start 2023-10-19T14:02:05.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.578+0000 7f5d2964f640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5d0c003970 0x7f5d24151ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:05.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.578+0000 7f5d2964f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5d240aa0c0 0x7f5d240aa540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:05.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.578+0000 7f5d2964f640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5d24139ad0 0x7f5d24139f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:05.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.578+0000 7f5d2964f640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d240aaa80 con 0x7f5d240aa0c0 2023-10-19T14:02:05.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.578+0000 7f5d2964f640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d2413a490 con 0x7f5d0c003970 2023-10-19T14:02:05.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.578+0000 7f5d2964f640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d24139480 con 0x7f5d24139ad0 2023-10-19T14:02:05.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.578+0000 7f5d227fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5d240aa0c0 0x7f5d240aa540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:05.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.578+0000 7f5d227fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5d240aa0c0 0x7f5d240aa540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:47482/0 (socket says 172.21.15.57:47482) 2023-10-19T14:02:05.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.578+0000 7f5d227fc640 1 -- 172.21.15.57:0/283348605 learned_addr learned my addr 172.21.15.57:0/283348605 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:02:05.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.578+0000 7f5d22ffd640 1 --2- 172.21.15.57:0/283348605 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5d0c003970 0x7f5d24151ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:05.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.578+0000 7f5d237fe640 1 --2- 172.21.15.57:0/283348605 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5d24139ad0 0x7f5d24139f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:05.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.578+0000 7f5d227fc640 1 -- 172.21.15.57:0/283348605 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5d24139ad0 msgr2=0x7f5d24139f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:05.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.579+0000 7f5d227fc640 1 --2- 172.21.15.57:0/283348605 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5d24139ad0 0x7f5d24139f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:05.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.579+0000 7f5d227fc640 1 -- 172.21.15.57:0/283348605 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5d0c003970 msgr2=0x7f5d24151ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:05.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.579+0000 7f5d227fc640 1 --2- 172.21.15.57:0/283348605 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5d0c003970 0x7f5d24151ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:05.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.579+0000 7f5d227fc640 1 -- 172.21.15.57:0/283348605 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d240aac80 con 0x7f5d240aa0c0 2023-10-19T14:02:05.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.579+0000 7f5d22ffd640 1 --2- 172.21.15.57:0/283348605 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5d0c003970 0x7f5d24151ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:02:05.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.579+0000 7f5d237fe640 1 --2- 172.21.15.57:0/283348605 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5d24139ad0 0x7f5d24139f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:02:05.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.579+0000 7f5d227fc640 1 --2- 172.21.15.57:0/283348605 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5d240aa0c0 0x7f5d240aa540 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 crypto rx=0x7f5d18011b20 tx=0x7f5d1800f3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:02:05.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.579+0000 7f5d0b7fe640 1 -- 172.21.15.57:0/283348605 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d1800fe00 con 0x7f5d240aa0c0 2023-10-19T14:02:05.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.579+0000 7f5d0b7fe640 1 -- 172.21.15.57:0/283348605 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f5d18019410 con 0x7f5d240aa0c0 2023-10-19T14:02:05.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.579+0000 7f5d0b7fe640 1 -- 172.21.15.57:0/283348605 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d180197c0 con 0x7f5d240aa0c0 2023-10-19T14:02:05.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.579+0000 7f5d2964f640 1 -- 172.21.15.57:0/283348605 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d240aaf60 con 0x7f5d240aa0c0 2023-10-19T14:02:05.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.579+0000 7f5d2964f640 1 -- 172.21.15.57:0/283348605 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d240a6550 con 0x7f5d240aa0c0 2023-10-19T14:02:05.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.580+0000 7f5d2964f640 1 -- 172.21.15.57:0/283348605 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d240690e0 con 0x7f5d240aa0c0 2023-10-19T14:02:05.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.580+0000 7f5d0b7fe640 1 -- 172.21.15.57:0/283348605 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 99) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f5d18019920 con 0x7f5d240aa0c0 2023-10-19T14:02:05.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.581+0000 7f5d0b7fe640 1 --2- 172.21.15.57:0/283348605 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f5d1006af90 0x7f5d1006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:05.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.581+0000 7f5d0b7fe640 1 -- 172.21.15.57:0/283348605 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5979+0+0 (secure 0 0 0) 0x7f5d1808a5d0 con 0x7f5d240aa0c0 2023-10-19T14:02:05.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.581+0000 7f5d22ffd640 1 --2- 172.21.15.57:0/283348605 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f5d1006af90 0x7f5d1006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:05.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.582+0000 7f5d22ffd640 1 --2- 172.21.15.57:0/283348605 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f5d1006af90 0x7f5d1006d450 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c034000 tx=0x7f5d1c001a70 comp rx=0 tx=0).ready entity=mgr.8315 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:02:05.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.585+0000 7f5d0b7fe640 1 -- 172.21.15.57:0/283348605 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f5d18055400 con 0x7f5d240aa0c0 2023-10-19T14:02:05.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.798+0000 7f5d2964f640 1 -- 172.21.15.57:0/283348605 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5d240cc420 con 0x7f5d240aa0c0 2023-10-19T14:02:05.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.798+0000 7f5d0b7fe640 1 -- 172.21.15.57:0/283348605 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+3702 (secure 0 0 0) 0x7f5d18058a20 con 0x7f5d240aa0c0 2023-10-19T14:02:05.801 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:02:05.801 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":72,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8126,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/1759316409","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1759316409},{"type":"v1","addr":"172.21.15.57:6835","nonce":1759316409}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67},{"gid":8129,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/2462599644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2462599644},{"type":"v1","addr":"172.21.15.57:6834","nonce":2462599644}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67},{"gid":8432,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/3940342107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":3940342107},{"type":"v1","addr":"172.21.15.179:6837","nonce":3940342107}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":72},{"gid":9136,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/373220657","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":373220657},{"type":"v1","addr":"172.21.15.179:6836","nonce":373220657}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67}],"filesystems":[]} 2023-10-19T14:02:05.802 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 72 2023-10-19T14:02:05.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.800+0000 7f5d2964f640 1 -- 172.21.15.57:0/283348605 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f5d1006af90 msgr2=0x7f5d1006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:05.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.800+0000 7f5d2964f640 1 --2- 172.21.15.57:0/283348605 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f5d1006af90 0x7f5d1006d450 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c034000 tx=0x7f5d1c001a70 comp rx=0 tx=0).stop 2023-10-19T14:02:05.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.800+0000 7f5d2964f640 1 -- 172.21.15.57:0/283348605 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5d240aa0c0 msgr2=0x7f5d240aa540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:05.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.800+0000 7f5d2964f640 1 --2- 172.21.15.57:0/283348605 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5d240aa0c0 0x7f5d240aa540 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 crypto rx=0x7f5d18011b20 tx=0x7f5d1800f3c0 comp rx=0 tx=0).stop 2023-10-19T14:02:05.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.800+0000 7f5d2964f640 1 -- 172.21.15.57:0/283348605 shutdown_connections 2023-10-19T14:02:05.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.800+0000 7f5d2964f640 1 --2- 172.21.15.57:0/283348605 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5d24139ad0 0x7f5d24139f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:05.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.800+0000 7f5d2964f640 1 --2- 172.21.15.57:0/283348605 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f5d1006af90 0x7f5d1006d450 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:05.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.800+0000 7f5d2964f640 1 --2- 172.21.15.57:0/283348605 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5d240aa0c0 0x7f5d240aa540 unknown :-1 s=CLOSED pgs=2007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:05.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.800+0000 7f5d2964f640 1 --2- 172.21.15.57:0/283348605 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5d0c003970 0x7f5d24151ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:05.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.800+0000 7f5d2964f640 1 -- 172.21.15.57:0/283348605 >> 172.21.15.57:0/283348605 conn(0x7f5d24069930 msgr2=0x7f5d2413b580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:02:05.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.800+0000 7f5d2964f640 1 -- 172.21.15.57:0/283348605 shutdown_connections 2023-10-19T14:02:05.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.800+0000 7f5d2964f640 1 -- 172.21.15.57:0/283348605 wait complete. 2023-10-19T14:02:05.813 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-19T14:02:05.814 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-19T14:02:05.814 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-19T14:02:05.814 INFO:teuthology.orchestra.run:waiting for 300 2023-10-19T14:02:05.815 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-19T14:02:05.815 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-19T14:02:05.815 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-19T14:02:05.815 INFO:teuthology.orchestra.run:waiting for 300 2023-10-19T14:02:05.816 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-19T14:02:05.816 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-19T14:02:05.816 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-19T14:02:05.817 INFO:teuthology.orchestra.run:waiting for 300 2023-10-19T14:02:05.817 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-19T14:02:05.817 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-19T14:02:05.817 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-19T14:02:05.818 INFO:teuthology.orchestra.run:waiting for 300 2023-10-19T14:02:05.823 INFO:tasks.ceph.mds.a:Stopped 2023-10-19T14:02:05.823 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-19T14:02:05.824 INFO:tasks.ceph.mds.c:Stopped 2023-10-19T14:02:05.825 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-19T14:02:05.828 INFO:tasks.ceph.mds.b:Stopped 2023-10-19T14:02:05.828 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-19T14:02:05.829 INFO:tasks.ceph.mds.d:Stopped 2023-10-19T14:02:05.830 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-19T14:02:05.866 INFO:tasks.ceph.mds.a:Started 2023-10-19T14:02:05.867 DEBUG:teuthology.parallel:result is None 2023-10-19T14:02:05.868 INFO:tasks.ceph.mds.c:Started 2023-10-19T14:02:05.868 DEBUG:teuthology.parallel:result is None 2023-10-19T14:02:05.871 INFO:tasks.ceph.mds.b:Started 2023-10-19T14:02:05.871 DEBUG:teuthology.parallel:result is None 2023-10-19T14:02:05.872 INFO:tasks.ceph.mds.d:Started 2023-10-19T14:02:05.872 DEBUG:teuthology.parallel:result is None 2023-10-19T14:02:05.872 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-19T14:02:05.949 INFO:tasks.ceph.mds.c.smithi057.stdout:starting mds.c at 2023-10-19T14:02:05.950 INFO:tasks.ceph.mds.a.smithi057.stdout:starting mds.a at 2023-10-19T14:02:05.954 INFO:tasks.ceph.mds.b.smithi179.stdout:starting mds.b at 2023-10-19T14:02:05.954 INFO:tasks.ceph.mds.d.smithi179.stdout:starting mds.d at 2023-10-19T14:02:05.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.984+0000 7ff03de32640 1 -- 172.21.15.57:0/1492479955 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff03815b2a0 msgr2=0x7ff0380cf720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:05.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.984+0000 7ff03de32640 1 --2- 172.21.15.57:0/1492479955 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff03815b2a0 0x7ff0380cf720 secure :-1 s=READY pgs=2014 cs=0 l=1 rev1=1 crypto rx=0x7ff03808a810 tx=0x7ff02c02bd10 comp rx=0 tx=0).stop 2023-10-19T14:02:05.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.984+0000 7ff03de32640 1 -- 172.21.15.57:0/1492479955 shutdown_connections 2023-10-19T14:02:05.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.984+0000 7ff03de32640 1 --2- 172.21.15.57:0/1492479955 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff0380ceb40 0x7ff0380b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:05.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.984+0000 7ff03de32640 1 --2- 172.21.15.57:0/1492479955 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff03815b2a0 0x7ff0380cf720 unknown :-1 s=CLOSED pgs=2014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:05.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.984+0000 7ff03de32640 1 --2- 172.21.15.57:0/1492479955 >> v2:172.21.15.179:3300/0 conn(0x7ff0380cf340 0x7ff0380d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:05.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.984+0000 7ff03de32640 1 -- 172.21.15.57:0/1492479955 >> 172.21.15.57:0/1492479955 conn(0x7ff0380d0d60 msgr2=0x7ff0380d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:02:05.989 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.984+0000 7ff03de32640 1 -- 172.21.15.57:0/1492479955 shutdown_connections 2023-10-19T14:02:05.989 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.984+0000 7ff03de32640 1 -- 172.21.15.57:0/1492479955 wait complete. 2023-10-19T14:02:05.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.985+0000 7ff03de32640 1 Processor -- start 2023-10-19T14:02:05.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.986+0000 7ff03de32640 1 -- start start 2023-10-19T14:02:05.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.986+0000 7ff03de32640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff0380ceb40 0x7ff03814d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:05.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.986+0000 7ff03de32640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff0380cf340 0x7ff03814dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:05.992 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.986+0000 7ff03de32640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff03815b2a0 0x7ff03815d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:05.992 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.986+0000 7ff03de32640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0380c8900 con 0x7ff03815b2a0 2023-10-19T14:02:05.992 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.986+0000 7ff03de32640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0380ca750 con 0x7ff0380cf340 2023-10-19T14:02:05.992 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.986+0000 7ff03de32640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0380c7f60 con 0x7ff0380ceb40 2023-10-19T14:02:05.993 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.986+0000 7ff037fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff03815b2a0 0x7ff03815d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:05.993 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.987+0000 7ff037fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff03815b2a0 0x7ff03815d6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:47522/0 (socket says 172.21.15.57:47522) 2023-10-19T14:02:05.993 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.987+0000 7ff037fff640 1 -- 172.21.15.57:0/1623912215 learned_addr learned my addr 172.21.15.57:0/1623912215 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:02:05.993 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.987+0000 7ff036ffd640 1 --2- 172.21.15.57:0/1623912215 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff0380cf340 0x7ff03814dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:05.993 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.987+0000 7ff0377fe640 1 --2- 172.21.15.57:0/1623912215 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff0380ceb40 0x7ff03814d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:05.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.987+0000 7ff037fff640 1 -- 172.21.15.57:0/1623912215 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff0380ceb40 msgr2=0x7ff03814d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:05.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.987+0000 7ff037fff640 1 --2- 172.21.15.57:0/1623912215 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff0380ceb40 0x7ff03814d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:05.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.987+0000 7ff037fff640 1 -- 172.21.15.57:0/1623912215 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff0380cf340 msgr2=0x7ff03814dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:05.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.987+0000 7ff037fff640 1 --2- 172.21.15.57:0/1623912215 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff0380cf340 0x7ff03814dc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:05.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.987+0000 7ff037fff640 1 -- 172.21.15.57:0/1623912215 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff02c009560 con 0x7ff03815b2a0 2023-10-19T14:02:05.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.987+0000 7ff036ffd640 1 --2- 172.21.15.57:0/1623912215 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff0380cf340 0x7ff03814dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:02:05.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.987+0000 7ff037fff640 1 --2- 172.21.15.57:0/1623912215 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff03815b2a0 0x7ff03815d6e0 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 crypto rx=0x7ff03000ca40 tx=0x7ff03000cf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:02:05.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.987+0000 7ff01ffff640 1 -- 172.21.15.57:0/1623912215 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff03000adc0 con 0x7ff03815b2a0 2023-10-19T14:02:05.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.987+0000 7ff01ffff640 1 -- 172.21.15.57:0/1623912215 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7ff030004c50 con 0x7ff03815b2a0 2023-10-19T14:02:05.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.987+0000 7ff03de32640 1 -- 172.21.15.57:0/1623912215 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0380c8240 con 0x7ff03815b2a0 2023-10-19T14:02:05.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.987+0000 7ff01ffff640 1 -- 172.21.15.57:0/1623912215 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff030005280 con 0x7ff03815b2a0 2023-10-19T14:02:05.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.987+0000 7ff03de32640 1 -- 172.21.15.57:0/1623912215 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff03814d5d0 con 0x7ff03815b2a0 2023-10-19T14:02:05.997 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.988+0000 7ff03de32640 1 -- 172.21.15.57:0/1623912215 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0380c8a40 con 0x7ff03815b2a0 2023-10-19T14:02:05.997 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.989+0000 7ff01ffff640 1 -- 172.21.15.57:0/1623912215 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 99) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff0300053e0 con 0x7ff03815b2a0 2023-10-19T14:02:05.998 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.989+0000 7ff01ffff640 1 --2- 172.21.15.57:0/1623912215 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7ff02406af60 0x7ff02406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:05.998 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.989+0000 7ff01ffff640 1 -- 172.21.15.57:0/1623912215 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5979+0+0 (secure 0 0 0) 0x7ff03008a1e0 con 0x7ff03815b2a0 2023-10-19T14:02:05.998 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.989+0000 7ff0377fe640 1 --2- 172.21.15.57:0/1623912215 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7ff02406af60 0x7ff02406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:05.998 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.990+0000 7ff0377fe640 1 --2- 172.21.15.57:0/1623912215 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7ff02406af60 0x7ff02406d420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff028005e10 tx=0x7ff028005da0 comp rx=0 tx=0).ready entity=mgr.8315 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:02:05.999 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:05.993+0000 7ff01ffff640 1 -- 172.21.15.57:0/1623912215 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff0380c8a40 con 0x7ff03815b2a0 2023-10-19T14:02:06.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:06.182+0000 7ff03de32640 1 -- 172.21.15.57:0/1623912215 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7ff0380c8a40 con 0x7ff03815b2a0 2023-10-19T14:02:06.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:06.634+0000 7ff01ffff640 1 -- 172.21.15.57:0/1623912215 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v182) v1 ==== 97+0+0 (secure 0 0 0) 0x7ff0380c8a40 con 0x7ff03815b2a0 2023-10-19T14:02:06.637 INFO:teuthology.orchestra.run.smithi057.stderr: removed all blocklist entries 2023-10-19T14:02:06.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:06.637+0000 7ff03de32640 1 -- 172.21.15.57:0/1623912215 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7ff02406af60 msgr2=0x7ff02406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:06.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:06.637+0000 7ff03de32640 1 --2- 172.21.15.57:0/1623912215 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7ff02406af60 0x7ff02406d420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff028005e10 tx=0x7ff028005da0 comp rx=0 tx=0).stop 2023-10-19T14:02:06.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:06.637+0000 7ff03de32640 1 -- 172.21.15.57:0/1623912215 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff03815b2a0 msgr2=0x7ff03815d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:06.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:06.637+0000 7ff03de32640 1 --2- 172.21.15.57:0/1623912215 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff03815b2a0 0x7ff03815d6e0 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 crypto rx=0x7ff03000ca40 tx=0x7ff03000cf00 comp rx=0 tx=0).stop 2023-10-19T14:02:06.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:06.637+0000 7ff03de32640 1 -- 172.21.15.57:0/1623912215 shutdown_connections 2023-10-19T14:02:06.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:06.637+0000 7ff03de32640 1 --2- 172.21.15.57:0/1623912215 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7ff02406af60 0x7ff02406d420 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:06.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:06.637+0000 7ff03de32640 1 --2- 172.21.15.57:0/1623912215 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff03815b2a0 0x7ff03815d6e0 unknown :-1 s=CLOSED pgs=2015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:06.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:06.637+0000 7ff03de32640 1 --2- 172.21.15.57:0/1623912215 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff0380cf340 0x7ff03814dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:06.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:06.637+0000 7ff03de32640 1 --2- 172.21.15.57:0/1623912215 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff0380ceb40 0x7ff03814d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:06.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:06.637+0000 7ff03de32640 1 -- 172.21.15.57:0/1623912215 >> 172.21.15.57:0/1623912215 conn(0x7ff0380d0d60 msgr2=0x7ff0380b1bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:02:06.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:06.637+0000 7ff03de32640 1 -- 172.21.15.57:0/1623912215 shutdown_connections 2023-10-19T14:02:06.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:06.637+0000 7ff03de32640 1 -- 172.21.15.57:0/1623912215 wait complete. 2023-10-19T14:02:06.654 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-19T14:02:06.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:06.763+0000 7f6e85444640 1 -- 172.21.15.57:0/34751734 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6e800cf850 msgr2=0x7f6e800cc9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:06.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:06.763+0000 7f6e85444640 1 --2- 172.21.15.57:0/34751734 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6e800cf850 0x7f6e800cc9f0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f6e70005fd0 tx=0x7f6e7002c2e0 comp rx=0 tx=0).stop 2023-10-19T14:02:06.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:06.763+0000 7f6e85444640 1 -- 172.21.15.57:0/34751734 shutdown_connections 2023-10-19T14:02:06.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:06.763+0000 7f6e85444640 1 --2- 172.21.15.57:0/34751734 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6e800cf850 0x7f6e800cc9f0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:06.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:06.763+0000 7f6e85444640 1 --2- 172.21.15.57:0/34751734 >> v2:172.21.15.179:3300/0 conn(0x7f6e8015b260 0x7f6e800cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:06.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:06.763+0000 7f6e85444640 1 -- 172.21.15.57:0/34751734 >> 172.21.15.57:0/34751734 conn(0x7f6e8009a030 msgr2=0x7f6e800d0d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:02:06.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:06.764+0000 7f6e85444640 1 -- 172.21.15.57:0/34751734 shutdown_connections 2023-10-19T14:02:06.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:06.764+0000 7f6e85444640 1 -- 172.21.15.57:0/34751734 wait complete. 2023-10-19T14:02:06.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:06.765+0000 7f6e85444640 1 Processor -- start 2023-10-19T14:02:06.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:06.765+0000 7f6e85444640 1 -- start start 2023-10-19T14:02:06.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:06.766+0000 7f6e85444640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6e800cf850 0x7f6e800c7df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:06.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:06.766+0000 7f6e85444640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6e8015b260 0x7f6e800b5f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:06.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:06.766+0000 7f6e85444640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6e8015b640 0x7f6e800c96e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:06.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:06.766+0000 7f6e85444640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e800c8640 con 0x7f6e8015b260 2023-10-19T14:02:06.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:06.766+0000 7f6e85444640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e800c8920 con 0x7f6e8015b640 2023-10-19T14:02:06.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:06.766+0000 7f6e85444640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e800c8c00 con 0x7f6e800cf850 2023-10-19T14:02:06.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:06.766+0000 7f6e7e7fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6e8015b260 0x7f6e800b5f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:06.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:06.766+0000 7f6e7e7fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6e8015b260 0x7f6e800b5f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59546/0 (socket says 172.21.15.57:59546) 2023-10-19T14:02:06.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:06.766+0000 7f6e7e7fc640 1 -- 172.21.15.57:0/881114202 learned_addr learned my addr 172.21.15.57:0/881114202 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:02:06.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:06.766+0000 7f6e7effd640 1 --2- 172.21.15.57:0/881114202 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6e800cf850 0x7f6e800c7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:06.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:06.766+0000 7f6e7f7fe640 1 --2- 172.21.15.57:0/881114202 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6e8015b640 0x7f6e800c96e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:06.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:06.766+0000 7f6e7e7fc640 1 -- 172.21.15.57:0/881114202 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6e800cf850 msgr2=0x7f6e800c7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:06.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:06.766+0000 7f6e7e7fc640 1 --2- 172.21.15.57:0/881114202 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6e800cf850 0x7f6e800c7df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:06.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:06.766+0000 7f6e7e7fc640 1 -- 172.21.15.57:0/881114202 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6e8015b640 msgr2=0x7f6e800c96e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:06.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:06.766+0000 7f6e7e7fc640 1 --2- 172.21.15.57:0/881114202 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6e8015b640 0x7f6e800c96e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:06.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:06.766+0000 7f6e7e7fc640 1 -- 172.21.15.57:0/881114202 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e70009900 con 0x7f6e8015b260 2023-10-19T14:02:06.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:06.766+0000 7f6e7e7fc640 1 --2- 172.21.15.57:0/881114202 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6e8015b260 0x7f6e800b5f40 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 crypto rx=0x7f6e6800c3c0 tx=0x7f6e6800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:02:06.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:06.766+0000 7f6e777fe640 1 -- 172.21.15.57:0/881114202 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e680052d0 con 0x7f6e8015b260 2023-10-19T14:02:06.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:06.766+0000 7f6e777fe640 1 -- 172.21.15.57:0/881114202 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f6e68007020 con 0x7f6e8015b260 2023-10-19T14:02:06.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:06.766+0000 7f6e85444640 1 -- 172.21.15.57:0/881114202 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e800c8ee0 con 0x7f6e8015b260 2023-10-19T14:02:06.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:06.766+0000 7f6e777fe640 1 -- 172.21.15.57:0/881114202 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e68007470 con 0x7f6e8015b260 2023-10-19T14:02:06.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:06.766+0000 7f6e85444640 1 -- 172.21.15.57:0/881114202 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e800c8330 con 0x7f6e8015b260 2023-10-19T14:02:06.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:06.767+0000 7f6e85444640 1 -- 172.21.15.57:0/881114202 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e800c9400 con 0x7f6e8015b260 2023-10-19T14:02:06.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:06.768+0000 7f6e777fe640 1 -- 172.21.15.57:0/881114202 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 99) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6e680067d0 con 0x7f6e8015b260 2023-10-19T14:02:06.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:06.768+0000 7f6e777fe640 1 --2- 172.21.15.57:0/881114202 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f6e6006af60 0x7f6e6006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:06.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:06.768+0000 7f6e777fe640 1 -- 172.21.15.57:0/881114202 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f6e68088ba0 con 0x7f6e8015b260 2023-10-19T14:02:06.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:06.768+0000 7f6e7effd640 1 --2- 172.21.15.57:0/881114202 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f6e6006af60 0x7f6e6006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:06.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:06.769+0000 7f6e7effd640 1 --2- 172.21.15.57:0/881114202 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f6e6006af60 0x7f6e6006d420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6e78009b60 tx=0x7f6e78005f90 comp rx=0 tx=0).ready entity=mgr.8315 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:02:06.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:06.772+0000 7f6e777fe640 1 -- 172.21.15.57:0/881114202 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6e800c9400 con 0x7f6e8015b260 2023-10-19T14:02:07.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.012+0000 7f6e85444640 1 -- 172.21.15.57:0/881114202 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f6e800c9400 con 0x7f6e8015b260 2023-10-19T14:02:07.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.013+0000 7f6e777fe640 1 -- 172.21.15.57:0/881114202 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v51) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f6e800c9400 con 0x7f6e8015b260 2023-10-19T14:02:07.016 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:02:07.016 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2023-10-19T14:02:07.016 INFO:teuthology.orchestra.run.smithi057.stdout: "auth_dump": [ 2023-10-19T14:02:07.017 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:02:07.017 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "mds.a", 2023-10-19T14:02:07.017 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBjJDFlmbVANhAAP/ynHlU/OxaRq5lQoAa8Ng==", 2023-10-19T14:02:07.017 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:02:07.017 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T14:02:07.018 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow *", 2023-10-19T14:02:07.018 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow *", 2023-10-19T14:02:07.018 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:02:07.018 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:02:07.018 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:02:07.019 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:02:07.019 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "mds.b", 2023-10-19T14:02:07.019 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFl9l7RBxAAC0geI2tJ1SIZZr1++kS/BA==", 2023-10-19T14:02:07.019 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:02:07.019 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T14:02:07.020 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow *", 2023-10-19T14:02:07.020 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow *", 2023-10-19T14:02:07.020 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:02:07.020 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:02:07.020 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:02:07.021 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:02:07.021 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "mds.c", 2023-10-19T14:02:07.021 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFlFkaoAhAA/PBizkJ4bm958oP4CmPuIQ==", 2023-10-19T14:02:07.021 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:02:07.021 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T14:02:07.022 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow *", 2023-10-19T14:02:07.022 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow *", 2023-10-19T14:02:07.022 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:02:07.022 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:02:07.022 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:02:07.023 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:02:07.023 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "mds.d", 2023-10-19T14:02:07.023 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFlx8skDxAAAfCHTYtYJgJik0WHSWeQfQ==", 2023-10-19T14:02:07.023 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:02:07.023 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T14:02:07.024 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow *", 2023-10-19T14:02:07.024 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow *", 2023-10-19T14:02:07.024 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:02:07.024 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:02:07.024 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:02:07.024 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:02:07.025 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.0", 2023-10-19T14:02:07.025 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBrJDFlCo9sBxAAQN2cID4nCQgRwGaGFEV8LQ==", 2023-10-19T14:02:07.025 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:02:07.025 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T14:02:07.025 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T14:02:07.026 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:02:07.026 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:02:07.026 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:02:07.026 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:02:07.026 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.1", 2023-10-19T14:02:07.027 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBuJDFl4fsLCBAAp3Pi80yZtc5sju0GisEuVA==", 2023-10-19T14:02:07.027 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:02:07.027 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T14:02:07.027 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T14:02:07.027 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:02:07.028 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:02:07.028 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:02:07.028 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:02:07.028 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.2", 2023-10-19T14:02:07.028 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBxJDFl+KAKCBAAd4N9cluARxPVfBLU+7Hd7A==", 2023-10-19T14:02:07.029 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:02:07.029 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T14:02:07.029 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T14:02:07.029 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:02:07.029 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:02:07.029 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:02:07.030 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:02:07.030 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.3", 2023-10-19T14:02:07.030 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQB0JDFlU/toCBAAP0Ou1zcHKNx5SfbybJkh4Q==", 2023-10-19T14:02:07.030 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:02:07.030 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T14:02:07.031 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T14:02:07.031 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:02:07.031 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:02:07.031 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:02:07.031 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:02:07.032 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.4", 2023-10-19T14:02:07.032 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQB6JDFlc6stEBAALkgNyxoiCTzZnKpBSPxI6w==", 2023-10-19T14:02:07.032 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:02:07.032 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T14:02:07.032 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T14:02:07.033 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:02:07.033 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:02:07.033 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:02:07.033 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:02:07.033 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.5", 2023-10-19T14:02:07.033 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQB9JDFlI3cNDhAAuG3tHJ6RvPwAzPE76wTIVQ==", 2023-10-19T14:02:07.034 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:02:07.034 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T14:02:07.034 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T14:02:07.034 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:02:07.034 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:02:07.035 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:02:07.035 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:02:07.035 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.6", 2023-10-19T14:02:07.035 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQCAJDFlLPumCxAA/iy08t/2N9rXfn71KtbSQw==", 2023-10-19T14:02:07.035 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:02:07.036 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T14:02:07.036 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T14:02:07.036 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:02:07.036 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:02:07.036 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:02:07.036 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:02:07.037 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.7", 2023-10-19T14:02:07.037 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQCDJDFlHmrbCRAA+hpWygGIJydfU6cHWlfx/Q==", 2023-10-19T14:02:07.037 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:02:07.037 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T14:02:07.037 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T14:02:07.038 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:02:07.038 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:02:07.038 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:02:07.038 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:02:07.038 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "client.0", 2023-10-19T14:02:07.039 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFlg9yuExAATxiWDfS+p07G1F0pORbJEg==", 2023-10-19T14:02:07.039 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:02:07.039 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T14:02:07.039 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow r", 2023-10-19T14:02:07.039 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-19T14:02:07.039 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:02:07.040 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:02:07.040 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:02:07.040 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "client.1", 2023-10-19T14:02:07.040 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFl9IwYGxAAHPituiJSPkHDuD0QyG4Hww==", 2023-10-19T14:02:07.040 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:02:07.040 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T14:02:07.041 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow r", 2023-10-19T14:02:07.041 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow rw", 2023-10-19T14:02:07.041 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow rwx" 2023-10-19T14:02:07.041 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:02:07.041 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:02:07.042 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:02:07.042 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "client.2", 2023-10-19T14:02:07.042 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFlMd+AFxAAS/UTESDt76GmrhhN2o/Fdw==", 2023-10-19T14:02:07.042 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:02:07.042 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T14:02:07.043 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow r", 2023-10-19T14:02:07.043 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow rw", 2023-10-19T14:02:07.043 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow rwx" 2023-10-19T14:02:07.043 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:02:07.043 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:02:07.044 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:02:07.044 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "client.3", 2023-10-19T14:02:07.044 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFlHZ3fIBAA42B7n8opLS5smXIwJyVYzw==", 2023-10-19T14:02:07.044 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:02:07.044 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T14:02:07.044 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow r", 2023-10-19T14:02:07.045 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow rw", 2023-10-19T14:02:07.045 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow rwx" 2023-10-19T14:02:07.045 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:02:07.045 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:02:07.045 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:02:07.046 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "client.admin", 2023-10-19T14:02:07.046 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBjJDFlBgasFxAAMA24j8KszcXMEpYRQDXj4g==", 2023-10-19T14:02:07.046 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:02:07.046 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow *", 2023-10-19T14:02:07.046 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow *", 2023-10-19T14:02:07.047 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow *", 2023-10-19T14:02:07.047 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:02:07.047 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:02:07.047 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:02:07.047 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:02:07.047 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "mgr.x", 2023-10-19T14:02:07.048 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBjJDFllFBfMhAA5+KPXWmYZqcESmLObaUbBA==", 2023-10-19T14:02:07.048 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:02:07.048 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow *", 2023-10-19T14:02:07.048 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile mgr", 2023-10-19T14:02:07.048 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:02:07.049 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:02:07.049 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:02:07.049 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:02:07.049 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "mgr.y", 2023-10-19T14:02:07.049 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBjJDFlxRzJLRAAGjG58o7qXAHvIEIRYwl6QA==", 2023-10-19T14:02:07.050 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:02:07.050 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow *", 2023-10-19T14:02:07.050 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile mgr", 2023-10-19T14:02:07.050 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:02:07.050 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:02:07.050 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:02:07.051 INFO:teuthology.orchestra.run.smithi057.stdout: ] 2023-10-19T14:02:07.051 INFO:teuthology.orchestra.run.smithi057.stdout:} 2023-10-19T14:02:07.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.015+0000 7f6e85444640 1 -- 172.21.15.57:0/881114202 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f6e6006af60 msgr2=0x7f6e6006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:07.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.015+0000 7f6e85444640 1 --2- 172.21.15.57:0/881114202 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f6e6006af60 0x7f6e6006d420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6e78009b60 tx=0x7f6e78005f90 comp rx=0 tx=0).stop 2023-10-19T14:02:07.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.015+0000 7f6e85444640 1 -- 172.21.15.57:0/881114202 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6e8015b260 msgr2=0x7f6e800b5f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:07.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.015+0000 7f6e85444640 1 --2- 172.21.15.57:0/881114202 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6e8015b260 0x7f6e800b5f40 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 crypto rx=0x7f6e6800c3c0 tx=0x7f6e6800c880 comp rx=0 tx=0).stop 2023-10-19T14:02:07.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.015+0000 7f6e85444640 1 -- 172.21.15.57:0/881114202 shutdown_connections 2023-10-19T14:02:07.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.015+0000 7f6e85444640 1 --2- 172.21.15.57:0/881114202 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6e8015b640 0x7f6e800c96e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:07.054 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.015+0000 7f6e85444640 1 --2- 172.21.15.57:0/881114202 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f6e6006af60 0x7f6e6006d420 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:07.054 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.015+0000 7f6e85444640 1 --2- 172.21.15.57:0/881114202 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6e8015b260 0x7f6e800b5f40 unknown :-1 s=CLOSED pgs=2016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:07.054 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.015+0000 7f6e85444640 1 --2- 172.21.15.57:0/881114202 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6e800cf850 0x7f6e800c7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:07.054 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.015+0000 7f6e85444640 1 -- 172.21.15.57:0/881114202 >> 172.21.15.57:0/881114202 conn(0x7f6e8009a030 msgr2=0x7f6e800b1710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:02:07.055 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.015+0000 7f6e85444640 1 -- 172.21.15.57:0/881114202 shutdown_connections 2023-10-19T14:02:07.055 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.015+0000 7f6e85444640 1 -- 172.21.15.57:0/881114202 wait complete. 2023-10-19T14:02:07.055 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T14:02:07.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.163+0000 7fd5f2eaf640 1 -- 172.21.15.57:0/3803911427 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5dc0047c0 msgr2=0x7fd5dc004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:07.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.163+0000 7fd5f2eaf640 1 --2- 172.21.15.57:0/3803911427 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5dc0047c0 0x7fd5dc004c40 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 crypto rx=0x7fd5d8001650 tx=0x7fd5dc0057f0 comp rx=0 tx=0).stop 2023-10-19T14:02:07.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.163+0000 7fd5f2eaf640 1 -- 172.21.15.57:0/3803911427 shutdown_connections 2023-10-19T14:02:07.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.163+0000 7fd5f2eaf640 1 --2- 172.21.15.57:0/3803911427 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5dc0047c0 0x7fd5dc004c40 unknown :-1 s=CLOSED pgs=2018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:07.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.163+0000 7fd5f2eaf640 1 -- 172.21.15.57:0/3803911427 >> 172.21.15.57:0/3803911427 conn(0x7fd5ec0d0d60 msgr2=0x7fd5ec0d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:02:07.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.163+0000 7fd5f2eaf640 1 -- 172.21.15.57:0/3803911427 shutdown_connections 2023-10-19T14:02:07.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.163+0000 7fd5f2eaf640 1 -- 172.21.15.57:0/3803911427 wait complete. 2023-10-19T14:02:07.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.164+0000 7fd5f2eaf640 1 Processor -- start 2023-10-19T14:02:07.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.164+0000 7fd5f2eaf640 1 -- start start 2023-10-19T14:02:07.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.165+0000 7fd5f2eaf640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5dc0047c0 0x7fd5ec0c9570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:07.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.165+0000 7fd5f2eaf640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd5ec15c1a0 0x7fd5ec1f5840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:07.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.165+0000 7fd5f2eaf640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd5ec1561e0 0x7fd5ec0b2830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:07.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.165+0000 7fd5f2eaf640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5ec15d140 con 0x7fd5dc0047c0 2023-10-19T14:02:07.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.165+0000 7fd5f2eaf640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5ec156bc0 con 0x7fd5ec15c1a0 2023-10-19T14:02:07.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.165+0000 7fd5f2eaf640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5ec15c620 con 0x7fd5ec1561e0 2023-10-19T14:02:07.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.165+0000 7fd5f0c4b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5dc0047c0 0x7fd5ec0c9570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:07.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.165+0000 7fd5f0c4b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5dc0047c0 0x7fd5ec0c9570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59564/0 (socket says 172.21.15.57:59564) 2023-10-19T14:02:07.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.165+0000 7fd5f0c4b640 1 -- 172.21.15.57:0/2432857711 learned_addr learned my addr 172.21.15.57:0/2432857711 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:02:07.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.165+0000 7fd5ebfff640 1 --2- 172.21.15.57:0/2432857711 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd5ec15c1a0 0x7fd5ec1f5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:07.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.165+0000 7fd5f144c640 1 --2- 172.21.15.57:0/2432857711 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd5ec1561e0 0x7fd5ec0b2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:07.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.165+0000 7fd5f0c4b640 1 -- 172.21.15.57:0/2432857711 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd5ec1561e0 msgr2=0x7fd5ec0b2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:07.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.165+0000 7fd5f0c4b640 1 --2- 172.21.15.57:0/2432857711 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd5ec1561e0 0x7fd5ec0b2830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:07.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.165+0000 7fd5f0c4b640 1 -- 172.21.15.57:0/2432857711 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd5ec15c1a0 msgr2=0x7fd5ec1f5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:07.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.165+0000 7fd5f0c4b640 1 --2- 172.21.15.57:0/2432857711 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd5ec15c1a0 0x7fd5ec1f5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:07.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.165+0000 7fd5f0c4b640 1 -- 172.21.15.57:0/2432857711 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5ec15c8a0 con 0x7fd5dc0047c0 2023-10-19T14:02:07.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.165+0000 7fd5f144c640 1 --2- 172.21.15.57:0/2432857711 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd5ec1561e0 0x7fd5ec0b2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:02:07.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.166+0000 7fd5f0c4b640 1 --2- 172.21.15.57:0/2432857711 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5dc0047c0 0x7fd5ec0c9570 secure :-1 s=READY pgs=2019 cs=0 l=1 rev1=1 crypto rx=0x7fd5e0006660 tx=0x7fd5e00039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:02:07.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.166+0000 7fd5e97fa640 1 -- 172.21.15.57:0/2432857711 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5e0011700 con 0x7fd5dc0047c0 2023-10-19T14:02:07.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.166+0000 7fd5e97fa640 1 -- 172.21.15.57:0/2432857711 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fd5e0011860 con 0x7fd5dc0047c0 2023-10-19T14:02:07.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.166+0000 7fd5f2eaf640 1 -- 172.21.15.57:0/2432857711 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5ec15cb80 con 0x7fd5dc0047c0 2023-10-19T14:02:07.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.166+0000 7fd5e97fa640 1 -- 172.21.15.57:0/2432857711 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5e0011c10 con 0x7fd5dc0047c0 2023-10-19T14:02:07.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.166+0000 7fd5f2eaf640 1 -- 172.21.15.57:0/2432857711 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5ec156660 con 0x7fd5dc0047c0 2023-10-19T14:02:07.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.167+0000 7fd5f2eaf640 1 -- 172.21.15.57:0/2432857711 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5ec08a3c0 con 0x7fd5dc0047c0 2023-10-19T14:02:07.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.167+0000 7fd5e97fa640 1 -- 172.21.15.57:0/2432857711 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 99) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd5e0010420 con 0x7fd5dc0047c0 2023-10-19T14:02:07.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.168+0000 7fd5e97fa640 1 --2- 172.21.15.57:0/2432857711 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7fd5d006af60 0x7fd5d006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:07.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.168+0000 7fd5e97fa640 1 -- 172.21.15.57:0/2432857711 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 5205+0+0 (secure 0 0 0) 0x7fd5e008a3b0 con 0x7fd5dc0047c0 2023-10-19T14:02:07.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.168+0000 7fd5ebfff640 1 --2- 172.21.15.57:0/2432857711 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7fd5d006af60 0x7fd5d006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:07.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.168+0000 7fd5ebfff640 1 --2- 172.21.15.57:0/2432857711 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7fd5d006af60 0x7fd5d006d420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fd5d8034000 tx=0x7fd5d8038040 comp rx=0 tx=0).ready entity=mgr.8315 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:02:07.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.172+0000 7fd5e97fa640 1 -- 172.21.15.57:0/2432857711 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd5e0055440 con 0x7fd5dc0047c0 2023-10-19T14:02:07.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.349+0000 7fd5f2eaf640 1 -- 172.21.15.57:0/2432857711 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd5ec0d07d0 con 0x7fd5dc0047c0 2023-10-19T14:02:07.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.349+0000 7fd5e97fa640 1 -- 172.21.15.57:0/2432857711 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v184) v1 ==== 74+0+13127 (secure 0 0 0) 0x7fd5e0058a60 con 0x7fd5dc0047c0 2023-10-19T14:02:07.352 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:02:07.352 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":184,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T14:02:06.643492+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T14:02:07.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.351+0000 7fd5f2eaf640 1 -- 172.21.15.57:0/2432857711 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7fd5d006af60 msgr2=0x7fd5d006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:07.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.351+0000 7fd5f2eaf640 1 --2- 172.21.15.57:0/2432857711 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7fd5d006af60 0x7fd5d006d420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fd5d8034000 tx=0x7fd5d8038040 comp rx=0 tx=0).stop 2023-10-19T14:02:07.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.351+0000 7fd5f2eaf640 1 -- 172.21.15.57:0/2432857711 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5dc0047c0 msgr2=0x7fd5ec0c9570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:07.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.351+0000 7fd5f2eaf640 1 --2- 172.21.15.57:0/2432857711 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5dc0047c0 0x7fd5ec0c9570 secure :-1 s=READY pgs=2019 cs=0 l=1 rev1=1 crypto rx=0x7fd5e0006660 tx=0x7fd5e00039e0 comp rx=0 tx=0).stop 2023-10-19T14:02:07.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.351+0000 7fd5f2eaf640 1 -- 172.21.15.57:0/2432857711 shutdown_connections 2023-10-19T14:02:07.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.351+0000 7fd5f2eaf640 1 --2- 172.21.15.57:0/2432857711 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd5ec1561e0 0x7fd5ec0b2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:07.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.351+0000 7fd5f2eaf640 1 --2- 172.21.15.57:0/2432857711 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd5ec15c1a0 0x7fd5ec1f5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:07.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.351+0000 7fd5f2eaf640 1 --2- 172.21.15.57:0/2432857711 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7fd5d006af60 0x7fd5d006d420 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:07.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.351+0000 7fd5f2eaf640 1 --2- 172.21.15.57:0/2432857711 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5dc0047c0 0x7fd5ec0c9570 unknown :-1 s=CLOSED pgs=2019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:07.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.351+0000 7fd5f2eaf640 1 -- 172.21.15.57:0/2432857711 >> 172.21.15.57:0/2432857711 conn(0x7fd5ec0d0d60 msgr2=0x7fd5ec0ad8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:02:07.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.351+0000 7fd5f2eaf640 1 -- 172.21.15.57:0/2432857711 shutdown_connections 2023-10-19T14:02:07.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.351+0000 7fd5f2eaf640 1 -- 172.21.15.57:0/2432857711 wait complete. 2023-10-19T14:02:07.365 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-19T12:44:07.371186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-19T14:02:07.365 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-19T14:02:07.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.474+0000 7ff9fcb89640 1 -- 172.21.15.57:0/1876609925 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9f815c1f0 msgr2=0x7ff9f806a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:07.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.474+0000 7ff9fcb89640 1 --2- 172.21.15.57:0/1876609925 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9f815c1f0 0x7ff9f806a730 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 crypto rx=0x7ff9f80662b0 tx=0x7ff9ec02be40 comp rx=0 tx=0).stop 2023-10-19T14:02:07.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.475+0000 7ff9fcb89640 1 -- 172.21.15.57:0/1876609925 shutdown_connections 2023-10-19T14:02:07.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.475+0000 7ff9fcb89640 1 --2- 172.21.15.57:0/1876609925 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9f815c1f0 0x7ff9f806a730 unknown :-1 s=CLOSED pgs=2020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:07.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.475+0000 7ff9fcb89640 1 --2- 172.21.15.57:0/1876609925 >> v2:172.21.15.179:3300/0 conn(0x7ff9f81579f0 0x7ff9f80b0e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:07.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.475+0000 7ff9fcb89640 1 -- 172.21.15.57:0/1876609925 >> 172.21.15.57:0/1876609925 conn(0x7ff9f806fae0 msgr2=0x7ff9f806fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:02:07.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.475+0000 7ff9fcb89640 1 -- 172.21.15.57:0/1876609925 shutdown_connections 2023-10-19T14:02:07.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.475+0000 7ff9fcb89640 1 -- 172.21.15.57:0/1876609925 wait complete. 2023-10-19T14:02:07.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.476+0000 7ff9fcb89640 1 Processor -- start 2023-10-19T14:02:07.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.476+0000 7ff9fcb89640 1 -- start start 2023-10-19T14:02:07.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.477+0000 7ff9fcb89640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9f806ac70 0x7ff9f8156480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:07.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.477+0000 7ff9fcb89640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff9f81579f0 0x7ff9f81e13b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:07.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.477+0000 7ff9fcb89640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff9f815c1f0 0x7ff9f80cb8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:07.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.477+0000 7ff9fcb89640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9f8153fe0 con 0x7ff9f806ac70 2023-10-19T14:02:07.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.477+0000 7ff9fcb89640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9f81e30a0 con 0x7ff9f815c1f0 2023-10-19T14:02:07.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.477+0000 7ff9fcb89640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9f813fce0 con 0x7ff9f81579f0 2023-10-19T14:02:07.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.477+0000 7ff9f659c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9f806ac70 0x7ff9f8156480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:07.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.477+0000 7ff9f659c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9f806ac70 0x7ff9f8156480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59580/0 (socket says 172.21.15.57:59580) 2023-10-19T14:02:07.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.477+0000 7ff9f659c640 1 -- 172.21.15.57:0/3179113767 learned_addr learned my addr 172.21.15.57:0/3179113767 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:02:07.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.477+0000 7ff9f6d9d640 1 --2- 172.21.15.57:0/3179113767 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff9f815c1f0 0x7ff9f80cb8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:07.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.477+0000 7ff9f5d9b640 1 --2- 172.21.15.57:0/3179113767 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff9f81579f0 0x7ff9f81e13b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:07.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.477+0000 7ff9f659c640 1 -- 172.21.15.57:0/3179113767 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff9f81579f0 msgr2=0x7ff9f81e13b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:07.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.477+0000 7ff9f659c640 1 --2- 172.21.15.57:0/3179113767 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff9f81579f0 0x7ff9f81e13b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:07.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.477+0000 7ff9f659c640 1 -- 172.21.15.57:0/3179113767 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff9f815c1f0 msgr2=0x7ff9f80cb8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:07.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.477+0000 7ff9f659c640 1 --2- 172.21.15.57:0/3179113767 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff9f815c1f0 0x7ff9f80cb8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:07.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.477+0000 7ff9f659c640 1 -- 172.21.15.57:0/3179113767 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9ec009560 con 0x7ff9f806ac70 2023-10-19T14:02:07.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.477+0000 7ff9f659c640 1 --2- 172.21.15.57:0/3179113767 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9f806ac70 0x7ff9f8156480 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7ff9e800ebf0 tx=0x7ff9e800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:02:07.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.477+0000 7ff9e6ffd640 1 -- 172.21.15.57:0/3179113767 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9e8010630 con 0x7ff9f806ac70 2023-10-19T14:02:07.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.477+0000 7ff9e6ffd640 1 -- 172.21.15.57:0/3179113767 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7ff9e800ced0 con 0x7ff9f806ac70 2023-10-19T14:02:07.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.477+0000 7ff9fcb89640 1 -- 172.21.15.57:0/3179113767 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9f813ffc0 con 0x7ff9f806ac70 2023-10-19T14:02:07.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.477+0000 7ff9e6ffd640 1 -- 172.21.15.57:0/3179113767 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9e8010b20 con 0x7ff9f806ac70 2023-10-19T14:02:07.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.477+0000 7ff9fcb89640 1 -- 172.21.15.57:0/3179113767 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9f8140580 con 0x7ff9f806ac70 2023-10-19T14:02:07.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.478+0000 7ff9fcb89640 1 -- 172.21.15.57:0/3179113767 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9f80b11f0 con 0x7ff9f806ac70 2023-10-19T14:02:07.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.479+0000 7ff9e6ffd640 1 -- 172.21.15.57:0/3179113767 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 99) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff9e8010c80 con 0x7ff9f806ac70 2023-10-19T14:02:07.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.479+0000 7ff9e6ffd640 1 --2- 172.21.15.57:0/3179113767 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7ff9d806af60 0x7ff9d806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:07.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.479+0000 7ff9e6ffd640 1 -- 172.21.15.57:0/3179113767 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 5205+0+0 (secure 0 0 0) 0x7ff9e8014070 con 0x7ff9f806ac70 2023-10-19T14:02:07.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.479+0000 7ff9f5d9b640 1 --2- 172.21.15.57:0/3179113767 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7ff9d806af60 0x7ff9d806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:07.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.480+0000 7ff9f5d9b640 1 --2- 172.21.15.57:0/3179113767 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7ff9d806af60 0x7ff9d806d420 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff9ec035000 tx=0x7ff9ec004730 comp rx=0 tx=0).ready entity=mgr.8315 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:02:07.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.484+0000 7ff9e6ffd640 1 -- 172.21.15.57:0/3179113767 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff9e8051ca0 con 0x7ff9f806ac70 2023-10-19T14:02:07.664 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.662+0000 7ff9fcb89640 1 -- 172.21.15.57:0/3179113767 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff9f80690e0 con 0x7ff9f806ac70 2023-10-19T14:02:07.665 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.662+0000 7ff9e6ffd640 1 -- 172.21.15.57:0/3179113767 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v184) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff9e805f070 con 0x7ff9f806ac70 2023-10-19T14:02:07.665 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-19T14:02:07.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.664+0000 7ff9fcb89640 1 -- 172.21.15.57:0/3179113767 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7ff9d806af60 msgr2=0x7ff9d806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:07.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.664+0000 7ff9fcb89640 1 --2- 172.21.15.57:0/3179113767 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7ff9d806af60 0x7ff9d806d420 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff9ec035000 tx=0x7ff9ec004730 comp rx=0 tx=0).stop 2023-10-19T14:02:07.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.664+0000 7ff9fcb89640 1 -- 172.21.15.57:0/3179113767 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9f806ac70 msgr2=0x7ff9f8156480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:07.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.664+0000 7ff9fcb89640 1 --2- 172.21.15.57:0/3179113767 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9f806ac70 0x7ff9f8156480 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7ff9e800ebf0 tx=0x7ff9e800c6a0 comp rx=0 tx=0).stop 2023-10-19T14:02:07.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.664+0000 7ff9fcb89640 1 -- 172.21.15.57:0/3179113767 shutdown_connections 2023-10-19T14:02:07.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.664+0000 7ff9fcb89640 1 --2- 172.21.15.57:0/3179113767 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff9f815c1f0 0x7ff9f80cb8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:07.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.664+0000 7ff9fcb89640 1 --2- 172.21.15.57:0/3179113767 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff9f81579f0 0x7ff9f81e13b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:07.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.664+0000 7ff9fcb89640 1 --2- 172.21.15.57:0/3179113767 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7ff9d806af60 0x7ff9d806d420 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:07.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.664+0000 7ff9fcb89640 1 --2- 172.21.15.57:0/3179113767 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9f806ac70 0x7ff9f8156480 unknown :-1 s=CLOSED pgs=2021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:07.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.664+0000 7ff9fcb89640 1 -- 172.21.15.57:0/3179113767 >> 172.21.15.57:0/3179113767 conn(0x7ff9f806fae0 msgr2=0x7ff9f8153510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:02:07.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.664+0000 7ff9fcb89640 1 -- 172.21.15.57:0/3179113767 shutdown_connections 2023-10-19T14:02:07.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.664+0000 7ff9fcb89640 1 -- 172.21.15.57:0/3179113767 wait complete. 2023-10-19T14:02:07.677 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-19T14:02:07.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.784+0000 7fb4f196c640 1 -- 172.21.15.57:0/1973807998 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4ec0d1910 msgr2=0x7fb4ec0b0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:07.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.784+0000 7fb4f196c640 1 --2- 172.21.15.57:0/1973807998 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4ec0d1910 0x7fb4ec0b0650 secure :-1 s=READY pgs=2023 cs=0 l=1 rev1=1 crypto rx=0x7fb4dc005fd0 tx=0x7fb4dc02c250 comp rx=0 tx=0).stop 2023-10-19T14:02:07.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.784+0000 7fb4f196c640 1 -- 172.21.15.57:0/1973807998 shutdown_connections 2023-10-19T14:02:07.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.784+0000 7fb4f196c640 1 --2- 172.21.15.57:0/1973807998 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4ec0d1910 0x7fb4ec0b0650 unknown :-1 s=CLOSED pgs=2023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:07.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.784+0000 7fb4f196c640 1 -- 172.21.15.57:0/1973807998 >> 172.21.15.57:0/1973807998 conn(0x7fb4ec0d0d60 msgr2=0x7fb4ec0d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:02:07.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.784+0000 7fb4f196c640 1 -- 172.21.15.57:0/1973807998 shutdown_connections 2023-10-19T14:02:07.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.784+0000 7fb4f196c640 1 -- 172.21.15.57:0/1973807998 wait complete. 2023-10-19T14:02:07.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.785+0000 7fb4f196c640 1 Processor -- start 2023-10-19T14:02:07.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.786+0000 7fb4f196c640 1 -- start start 2023-10-19T14:02:07.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.786+0000 7fb4f196c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4ec0cf340 0x7fb4ec155bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:07.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.786+0000 7fb4f196c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb4ec0d1910 0x7fb4ec15d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:07.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.786+0000 7fb4f196c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb4ec15b2a0 0x7fb4ec15dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:07.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.786+0000 7fb4f196c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4ec0cae60 con 0x7fb4ec0cf340 2023-10-19T14:02:07.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.786+0000 7fb4f196c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4ec0c7e30 con 0x7fb4ec0d1910 2023-10-19T14:02:07.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.786+0000 7fb4f196c640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4ec0b5f70 con 0x7fb4ec15b2a0 2023-10-19T14:02:07.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.786+0000 7fb4eaffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4ec0cf340 0x7fb4ec155bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:07.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.786+0000 7fb4eaffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4ec0cf340 0x7fb4ec155bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59602/0 (socket says 172.21.15.57:59602) 2023-10-19T14:02:07.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.786+0000 7fb4eaffd640 1 -- 172.21.15.57:0/171586109 learned_addr learned my addr 172.21.15.57:0/171586109 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:02:07.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.787+0000 7fb4ea7fc640 1 --2- 172.21.15.57:0/171586109 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb4ec0d1910 0x7fb4ec15d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:07.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.787+0000 7fb4eaffd640 1 -- 172.21.15.57:0/171586109 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb4ec15b2a0 msgr2=0x7fb4ec15dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:07.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.787+0000 7fb4eb7fe640 1 --2- 172.21.15.57:0/171586109 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb4ec15b2a0 0x7fb4ec15dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:07.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.787+0000 7fb4eaffd640 1 --2- 172.21.15.57:0/171586109 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb4ec15b2a0 0x7fb4ec15dd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:07.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.787+0000 7fb4eaffd640 1 -- 172.21.15.57:0/171586109 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb4ec0d1910 msgr2=0x7fb4ec15d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:07.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.787+0000 7fb4eaffd640 1 --2- 172.21.15.57:0/171586109 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb4ec0d1910 0x7fb4ec15d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:07.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.787+0000 7fb4eaffd640 1 -- 172.21.15.57:0/171586109 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4dc009900 con 0x7fb4ec0cf340 2023-10-19T14:02:07.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.787+0000 7fb4ea7fc640 1 --2- 172.21.15.57:0/171586109 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb4ec0d1910 0x7fb4ec15d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:02:07.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.787+0000 7fb4eaffd640 1 --2- 172.21.15.57:0/171586109 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4ec0cf340 0x7fb4ec155bd0 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 crypto rx=0x7fb4e400c520 tx=0x7fb4e400c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:02:07.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.787+0000 7fb4d3fff640 1 -- 172.21.15.57:0/171586109 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4e40052d0 con 0x7fb4ec0cf340 2023-10-19T14:02:07.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.787+0000 7fb4d3fff640 1 -- 172.21.15.57:0/171586109 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fb4e4012630 con 0x7fb4ec0cf340 2023-10-19T14:02:07.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.787+0000 7fb4f196c640 1 -- 172.21.15.57:0/171586109 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4ec0b6250 con 0x7fb4ec0cf340 2023-10-19T14:02:07.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.787+0000 7fb4d3fff640 1 -- 172.21.15.57:0/171586109 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4e4012a80 con 0x7fb4ec0cf340 2023-10-19T14:02:07.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.787+0000 7fb4f196c640 1 -- 172.21.15.57:0/171586109 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4ec0b6810 con 0x7fb4ec0cf340 2023-10-19T14:02:07.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.788+0000 7fb4f196c640 1 -- 172.21.15.57:0/171586109 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4ec0690e0 con 0x7fb4ec0cf340 2023-10-19T14:02:07.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.789+0000 7fb4d3fff640 1 -- 172.21.15.57:0/171586109 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 99) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb4e4003610 con 0x7fb4ec0cf340 2023-10-19T14:02:07.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.789+0000 7fb4d3fff640 1 --2- 172.21.15.57:0/171586109 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7fb4d806af60 0x7fb4d806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:07.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.789+0000 7fb4d3fff640 1 -- 172.21.15.57:0/171586109 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 5205+0+0 (secure 0 0 0) 0x7fb4e4088db0 con 0x7fb4ec0cf340 2023-10-19T14:02:07.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.789+0000 7fb4ea7fc640 1 --2- 172.21.15.57:0/171586109 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7fb4d806af60 0x7fb4d806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:07.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.790+0000 7fb4ea7fc640 1 --2- 172.21.15.57:0/171586109 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7fb4d806af60 0x7fb4d806d420 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb4ec15acb0 tx=0x7fb4e0008040 comp rx=0 tx=0).ready entity=mgr.8315 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:02:07.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.794+0000 7fb4d3fff640 1 -- 172.21.15.57:0/171586109 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb4e4053f50 con 0x7fb4ec0cf340 2023-10-19T14:02:07.973 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.970+0000 7fb4f196c640 1 -- 172.21.15.57:0/171586109 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fb4ec0d0b40 con 0x7fb4ec0cf340 2023-10-19T14:02:07.973 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.971+0000 7fb4d3fff640 1 -- 172.21.15.57:0/171586109 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v184) v1 ==== 84+0+65 (secure 0 0 0) 0x7fb4e4017090 con 0x7fb4ec0cf340 2023-10-19T14:02:07.974 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:02:07.974 INFO:teuthology.orchestra.run.smithi057.stdout:[ 2023-10-19T14:02:07.974 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:02:07.974 INFO:teuthology.orchestra.run.smithi057.stdout: "poolnum": 1, 2023-10-19T14:02:07.975 INFO:teuthology.orchestra.run.smithi057.stdout: "poolname": ".mgr" 2023-10-19T14:02:07.975 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:02:07.975 INFO:teuthology.orchestra.run.smithi057.stdout:] 2023-10-19T14:02:07.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.973+0000 7fb4f196c640 1 -- 172.21.15.57:0/171586109 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7fb4d806af60 msgr2=0x7fb4d806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:07.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.973+0000 7fb4f196c640 1 --2- 172.21.15.57:0/171586109 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7fb4d806af60 0x7fb4d806d420 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb4ec15acb0 tx=0x7fb4e0008040 comp rx=0 tx=0).stop 2023-10-19T14:02:07.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.973+0000 7fb4f196c640 1 -- 172.21.15.57:0/171586109 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4ec0cf340 msgr2=0x7fb4ec155bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:07.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.973+0000 7fb4f196c640 1 --2- 172.21.15.57:0/171586109 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4ec0cf340 0x7fb4ec155bd0 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 crypto rx=0x7fb4e400c520 tx=0x7fb4e400c9e0 comp rx=0 tx=0).stop 2023-10-19T14:02:07.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.973+0000 7fb4f196c640 1 -- 172.21.15.57:0/171586109 shutdown_connections 2023-10-19T14:02:07.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.973+0000 7fb4f196c640 1 --2- 172.21.15.57:0/171586109 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb4ec15b2a0 0x7fb4ec15dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:07.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.973+0000 7fb4f196c640 1 --2- 172.21.15.57:0/171586109 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb4ec0d1910 0x7fb4ec15d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:07.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.973+0000 7fb4f196c640 1 --2- 172.21.15.57:0/171586109 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7fb4d806af60 0x7fb4d806d420 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:07.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.973+0000 7fb4f196c640 1 --2- 172.21.15.57:0/171586109 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4ec0cf340 0x7fb4ec155bd0 unknown :-1 s=CLOSED pgs=2024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:07.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.973+0000 7fb4f196c640 1 -- 172.21.15.57:0/171586109 >> 172.21.15.57:0/171586109 conn(0x7fb4ec0d0d60 msgr2=0x7fb4ec0b0da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:02:07.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.973+0000 7fb4f196c640 1 -- 172.21.15.57:0/171586109 shutdown_connections 2023-10-19T14:02:07.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:07.973+0000 7fb4f196c640 1 -- 172.21.15.57:0/171586109 wait complete. 2023-10-19T14:02:07.986 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-19T14:02:07.986 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-19T14:02:08.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.097+0000 7f812645d640 1 -- 172.21.15.57:0/3721732938 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f81201579f0 msgr2=0x7f81200b0e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:08.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.097+0000 7f812645d640 1 --2- 172.21.15.57:0/3721732938 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f81201579f0 0x7f81200b0e90 secure :-1 s=READY pgs=2025 cs=0 l=1 rev1=1 crypto rx=0x7f81200b0160 tx=0x7f811002be40 comp rx=0 tx=0).stop 2023-10-19T14:02:08.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.098+0000 7f812645d640 1 -- 172.21.15.57:0/3721732938 shutdown_connections 2023-10-19T14:02:08.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.098+0000 7f812645d640 1 --2- 172.21.15.57:0/3721732938 >> v2:172.21.15.179:3300/0 conn(0x7f812006ae40 0x7f8120157dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:08.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.098+0000 7f812645d640 1 --2- 172.21.15.57:0/3721732938 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f81201579f0 0x7f81200b0e90 unknown :-1 s=CLOSED pgs=2025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:08.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.098+0000 7f812645d640 1 -- 172.21.15.57:0/3721732938 >> 172.21.15.57:0/3721732938 conn(0x7f812006fae0 msgr2=0x7f812006fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:02:08.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.098+0000 7f812645d640 1 -- 172.21.15.57:0/3721732938 shutdown_connections 2023-10-19T14:02:08.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.098+0000 7f812645d640 1 -- 172.21.15.57:0/3721732938 wait complete. 2023-10-19T14:02:08.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.099+0000 7f812645d640 1 Processor -- start 2023-10-19T14:02:08.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.099+0000 7f812645d640 1 -- start start 2023-10-19T14:02:08.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.100+0000 7f812645d640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f812006ae40 0x7f812015a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:08.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.100+0000 7f812645d640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f81201579f0 0x7f812013fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:08.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.100+0000 7f812645d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f812015c1f0 0x7f812006cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:08.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.100+0000 7f812645d640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8120140220 con 0x7f812015c1f0 2023-10-19T14:02:08.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.100+0000 7f812645d640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81200c27d0 con 0x7f81201579f0 2023-10-19T14:02:08.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.100+0000 7f812645d640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f812006bfc0 con 0x7f812006ae40 2023-10-19T14:02:08.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.100+0000 7f81249fa640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f812015c1f0 0x7f812006cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:08.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.100+0000 7f81249fa640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f812015c1f0 0x7f812006cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59622/0 (socket says 172.21.15.57:59622) 2023-10-19T14:02:08.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.100+0000 7f81249fa640 1 -- 172.21.15.57:0/922771744 learned_addr learned my addr 172.21.15.57:0/922771744 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:02:08.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.100+0000 7f811f7fe640 1 --2- 172.21.15.57:0/922771744 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f81201579f0 0x7f812013fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:08.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.100+0000 7f81249fa640 1 -- 172.21.15.57:0/922771744 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f812006ae40 msgr2=0x7f812015a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:08.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.100+0000 7f81249fa640 1 --2- 172.21.15.57:0/922771744 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f812006ae40 0x7f812015a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:08.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.100+0000 7f81249fa640 1 -- 172.21.15.57:0/922771744 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f81201579f0 msgr2=0x7f812013fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:08.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.100+0000 7f81249fa640 1 --2- 172.21.15.57:0/922771744 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f81201579f0 0x7f812013fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:08.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.100+0000 7f81249fa640 1 -- 172.21.15.57:0/922771744 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8110009560 con 0x7f812015c1f0 2023-10-19T14:02:08.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.100+0000 7f811f7fe640 1 --2- 172.21.15.57:0/922771744 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f81201579f0 0x7f812013fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:02:08.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.100+0000 7f81249fa640 1 --2- 172.21.15.57:0/922771744 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f812015c1f0 0x7f812006cd80 secure :-1 s=READY pgs=2026 cs=0 l=1 rev1=1 crypto rx=0x7f8118010fc0 tx=0x7f811800e490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:02:08.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.100+0000 7f811cff9640 1 -- 172.21.15.57:0/922771744 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f811800bea0 con 0x7f812015c1f0 2023-10-19T14:02:08.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.100+0000 7f811cff9640 1 -- 172.21.15.57:0/922771744 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f8118011040 con 0x7f812015c1f0 2023-10-19T14:02:08.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.100+0000 7f812645d640 1 -- 172.21.15.57:0/922771744 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f812006c2a0 con 0x7f812015c1f0 2023-10-19T14:02:08.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.100+0000 7f811cff9640 1 -- 172.21.15.57:0/922771744 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f811800b290 con 0x7f812015c1f0 2023-10-19T14:02:08.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.101+0000 7f812645d640 1 -- 172.21.15.57:0/922771744 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f812006c860 con 0x7f812015c1f0 2023-10-19T14:02:08.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.101+0000 7f812645d640 1 -- 172.21.15.57:0/922771744 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f812006ca80 con 0x7f812015c1f0 2023-10-19T14:02:08.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.102+0000 7f811cff9640 1 -- 172.21.15.57:0/922771744 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 99) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f8118016480 con 0x7f812015c1f0 2023-10-19T14:02:08.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.102+0000 7f811cff9640 1 --2- 172.21.15.57:0/922771744 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f810c06af60 0x7f810c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:08.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.102+0000 7f811cff9640 1 -- 172.21.15.57:0/922771744 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f811808bfe0 con 0x7f812015c1f0 2023-10-19T14:02:08.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.103+0000 7f811ffff640 1 --2- 172.21.15.57:0/922771744 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f810c06af60 0x7f810c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:08.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.103+0000 7f811ffff640 1 --2- 172.21.15.57:0/922771744 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f810c06af60 0x7f810c06d420 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8110035000 tx=0x7f81100046c0 comp rx=0 tx=0).ready entity=mgr.8315 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:02:08.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.106+0000 7f811cff9640 1 -- 172.21.15.57:0/922771744 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8118056fc0 con 0x7f812015c1f0 2023-10-19T14:02:08.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.282+0000 7f812645d640 1 -- 172.21.15.57:0/922771744 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f81200690e0 con 0x7f812015c1f0 2023-10-19T14:02:08.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.643+0000 7f811cff9640 1 -- 172.21.15.57:0/922771744 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v185) v1 ==== 138+0+0 (secure 0 0 0) 0x7f811801a020 con 0x7f812015c1f0 2023-10-19T14:02:08.646 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'cephfs_metadata' created 2023-10-19T14:02:08.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.645+0000 7f812645d640 1 -- 172.21.15.57:0/922771744 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f810c06af60 msgr2=0x7f810c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:08.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.645+0000 7f812645d640 1 --2- 172.21.15.57:0/922771744 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f810c06af60 0x7f810c06d420 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8110035000 tx=0x7f81100046c0 comp rx=0 tx=0).stop 2023-10-19T14:02:08.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.645+0000 7f812645d640 1 -- 172.21.15.57:0/922771744 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f812015c1f0 msgr2=0x7f812006cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:08.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.645+0000 7f812645d640 1 --2- 172.21.15.57:0/922771744 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f812015c1f0 0x7f812006cd80 secure :-1 s=READY pgs=2026 cs=0 l=1 rev1=1 crypto rx=0x7f8118010fc0 tx=0x7f811800e490 comp rx=0 tx=0).stop 2023-10-19T14:02:08.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.645+0000 7f812645d640 1 -- 172.21.15.57:0/922771744 shutdown_connections 2023-10-19T14:02:08.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.645+0000 7f812645d640 1 --2- 172.21.15.57:0/922771744 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f810c06af60 0x7f810c06d420 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:08.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.645+0000 7f812645d640 1 --2- 172.21.15.57:0/922771744 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f812015c1f0 0x7f812006cd80 unknown :-1 s=CLOSED pgs=2026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:08.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.645+0000 7f812645d640 1 --2- 172.21.15.57:0/922771744 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f81201579f0 0x7f812013fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:08.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.645+0000 7f812645d640 1 --2- 172.21.15.57:0/922771744 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f812006ae40 0x7f812015a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:08.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.645+0000 7f812645d640 1 -- 172.21.15.57:0/922771744 >> 172.21.15.57:0/922771744 conn(0x7f812006fae0 msgr2=0x7f8120150e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:02:08.650 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.645+0000 7f812645d640 1 -- 172.21.15.57:0/922771744 shutdown_connections 2023-10-19T14:02:08.650 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.645+0000 7f812645d640 1 -- 172.21.15.57:0/922771744 wait complete. 2023-10-19T14:02:08.663 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-19T14:02:08.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.777+0000 7f73def05640 1 -- 172.21.15.57:0/381051651 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f73d80afa20 msgr2=0x7f73d80705a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:08.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.777+0000 7f73def05640 1 --2- 172.21.15.57:0/381051651 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f73d80afa20 0x7f73d80705a0 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 crypto rx=0x7f73d80cc7e0 tx=0x7f73c402d160 comp rx=0 tx=0).stop 2023-10-19T14:02:08.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.777+0000 7f73def05640 1 -- 172.21.15.57:0/381051651 shutdown_connections 2023-10-19T14:02:08.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.777+0000 7f73def05640 1 --2- 172.21.15.57:0/381051651 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f73d80afa20 0x7f73d80705a0 unknown :-1 s=CLOSED pgs=2027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:08.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.777+0000 7f73def05640 1 --2- 172.21.15.57:0/381051651 >> v2:172.21.15.179:3300/0 conn(0x7f73d80cbd20 0x7f73d8158300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:08.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.777+0000 7f73def05640 1 -- 172.21.15.57:0/381051651 >> 172.21.15.57:0/381051651 conn(0x7f73d8069930 msgr2=0x7f73d8069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:02:08.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.777+0000 7f73def05640 1 -- 172.21.15.57:0/381051651 shutdown_connections 2023-10-19T14:02:08.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.777+0000 7f73def05640 1 -- 172.21.15.57:0/381051651 wait complete. 2023-10-19T14:02:08.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.779+0000 7f73def05640 1 Processor -- start 2023-10-19T14:02:08.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.779+0000 7f73def05640 1 -- start start 2023-10-19T14:02:08.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.779+0000 7f73def05640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f73d80afa20 0x7f73d8135b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:08.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.779+0000 7f73def05640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f73d80b0530 0x7f73d814f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:08.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.779+0000 7f73def05640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f73d80cbd20 0x7f73d81e0e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:08.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.779+0000 7f73def05640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73d81e13d0 con 0x7f73d80cbd20 2023-10-19T14:02:08.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.779+0000 7f73def05640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73d814fe00 con 0x7f73d80b0530 2023-10-19T14:02:08.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.779+0000 7f73def05640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73d8149a20 con 0x7f73d80afa20 2023-10-19T14:02:08.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.780+0000 7f73dd4a2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f73d80cbd20 0x7f73d81e0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:08.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.780+0000 7f73dd4a2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f73d80cbd20 0x7f73d81e0e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59638/0 (socket says 172.21.15.57:59638) 2023-10-19T14:02:08.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.780+0000 7f73dd4a2640 1 -- 172.21.15.57:0/2180970901 learned_addr learned my addr 172.21.15.57:0/2180970901 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:02:08.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.780+0000 7f73dcca1640 1 --2- 172.21.15.57:0/2180970901 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f73d80afa20 0x7f73d8135b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:08.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.780+0000 7f73d7fff640 1 --2- 172.21.15.57:0/2180970901 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f73d80b0530 0x7f73d814f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:08.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.780+0000 7f73dd4a2640 1 -- 172.21.15.57:0/2180970901 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f73d80afa20 msgr2=0x7f73d8135b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:08.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.780+0000 7f73dd4a2640 1 --2- 172.21.15.57:0/2180970901 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f73d80afa20 0x7f73d8135b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:08.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.780+0000 7f73dd4a2640 1 -- 172.21.15.57:0/2180970901 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f73d80b0530 msgr2=0x7f73d814f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:08.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.780+0000 7f73dd4a2640 1 --2- 172.21.15.57:0/2180970901 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f73d80b0530 0x7f73d814f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:08.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.780+0000 7f73dd4a2640 1 -- 172.21.15.57:0/2180970901 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73c400a670 con 0x7f73d80cbd20 2023-10-19T14:02:08.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.780+0000 7f73d7fff640 1 --2- 172.21.15.57:0/2180970901 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f73d80b0530 0x7f73d814f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:02:08.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.780+0000 7f73dd4a2640 1 --2- 172.21.15.57:0/2180970901 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f73d80cbd20 0x7f73d81e0e90 secure :-1 s=READY pgs=2028 cs=0 l=1 rev1=1 crypto rx=0x7f73d000c7a0 tx=0x7f73d000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:02:08.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.780+0000 7f73d57fa640 1 -- 172.21.15.57:0/2180970901 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73d0003b70 con 0x7f73d80cbd20 2023-10-19T14:02:08.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.780+0000 7f73d57fa640 1 -- 172.21.15.57:0/2180970901 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f73d0003cf0 con 0x7f73d80cbd20 2023-10-19T14:02:08.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.780+0000 7f73def05640 1 -- 172.21.15.57:0/2180970901 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73d8149d00 con 0x7f73d80cbd20 2023-10-19T14:02:08.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.780+0000 7f73d57fa640 1 -- 172.21.15.57:0/2180970901 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73d000aab0 con 0x7f73d80cbd20 2023-10-19T14:02:08.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.780+0000 7f73def05640 1 -- 172.21.15.57:0/2180970901 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73d814a2c0 con 0x7f73d80cbd20 2023-10-19T14:02:08.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.781+0000 7f73def05640 1 -- 172.21.15.57:0/2180970901 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73d80690e0 con 0x7f73d80cbd20 2023-10-19T14:02:08.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.782+0000 7f73d57fa640 1 -- 172.21.15.57:0/2180970901 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 99) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f73d000ac10 con 0x7f73d80cbd20 2023-10-19T14:02:08.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.782+0000 7f73d57fa640 1 --2- 172.21.15.57:0/2180970901 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f73bc06af60 0x7f73bc06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:08.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.782+0000 7f73d57fa640 1 -- 172.21.15.57:0/2180970901 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 5575+0+0 (secure 0 0 0) 0x7f73d00897d0 con 0x7f73d80cbd20 2023-10-19T14:02:08.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.783+0000 7f73dcca1640 1 --2- 172.21.15.57:0/2180970901 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f73bc06af60 0x7f73bc06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:08.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.783+0000 7f73dcca1640 1 --2- 172.21.15.57:0/2180970901 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f73bc06af60 0x7f73bc06d420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f73cc005fd0 tx=0x7f73cc005850 comp rx=0 tx=0).ready entity=mgr.8315 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:02:08.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.787+0000 7f73d57fa640 1 -- 172.21.15.57:0/2180970901 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f73d0054750 con 0x7f73d80cbd20 2023-10-19T14:02:08.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:08.962+0000 7f73def05640 1 -- 172.21.15.57:0/2180970901 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f73d806f700 con 0x7f73d80cbd20 2023-10-19T14:02:09.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:09.646+0000 7f73d57fa640 1 -- 172.21.15.57:0/2180970901 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v186) v1 ==== 170+0+0 (secure 0 0 0) 0x7f73d0057d70 con 0x7f73d80cbd20 2023-10-19T14:02:09.649 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'cephfs_data' created 2023-10-19T14:02:09.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:09.650+0000 7f73def05640 1 -- 172.21.15.57:0/2180970901 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f73bc06af60 msgr2=0x7f73bc06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:09.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:09.650+0000 7f73def05640 1 --2- 172.21.15.57:0/2180970901 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f73bc06af60 0x7f73bc06d420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f73cc005fd0 tx=0x7f73cc005850 comp rx=0 tx=0).stop 2023-10-19T14:02:09.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:09.650+0000 7f73def05640 1 -- 172.21.15.57:0/2180970901 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f73d80cbd20 msgr2=0x7f73d81e0e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:09.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:09.650+0000 7f73def05640 1 --2- 172.21.15.57:0/2180970901 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f73d80cbd20 0x7f73d81e0e90 secure :-1 s=READY pgs=2028 cs=0 l=1 rev1=1 crypto rx=0x7f73d000c7a0 tx=0x7f73d000cc60 comp rx=0 tx=0).stop 2023-10-19T14:02:09.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:09.651+0000 7f73def05640 1 -- 172.21.15.57:0/2180970901 shutdown_connections 2023-10-19T14:02:09.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:09.652+0000 7f73def05640 1 --2- 172.21.15.57:0/2180970901 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f73bc06af60 0x7f73bc06d420 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:09.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:09.652+0000 7f73def05640 1 --2- 172.21.15.57:0/2180970901 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f73d80cbd20 0x7f73d81e0e90 unknown :-1 s=CLOSED pgs=2028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:09.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:09.652+0000 7f73def05640 1 --2- 172.21.15.57:0/2180970901 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f73d80b0530 0x7f73d814f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:09.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:09.652+0000 7f73def05640 1 --2- 172.21.15.57:0/2180970901 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f73d80afa20 0x7f73d8135b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:09.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:09.652+0000 7f73def05640 1 -- 172.21.15.57:0/2180970901 >> 172.21.15.57:0/2180970901 conn(0x7f73d8069930 msgr2=0x7f73d8069d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:02:09.658 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:09.656+0000 7f73def05640 1 -- 172.21.15.57:0/2180970901 shutdown_connections 2023-10-19T14:02:09.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:09.657+0000 7f73def05640 1 -- 172.21.15.57:0/2180970901 wait complete. 2023-10-19T14:02:09.684 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-19T14:02:09.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:09.799+0000 7f27cedce640 1 -- 172.21.15.57:0/2985079801 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f27c80afa20 msgr2=0x7f27c80c97a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:09.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:09.799+0000 7f27cedce640 1 --2- 172.21.15.57:0/2985079801 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f27c80afa20 0x7f27c80c97a0 secure :-1 s=READY pgs=2029 cs=0 l=1 rev1=1 crypto rx=0x7f27c806a780 tx=0x7f27b002bda0 comp rx=0 tx=0).stop 2023-10-19T14:02:09.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:09.799+0000 7f27cedce640 1 -- 172.21.15.57:0/2985079801 shutdown_connections 2023-10-19T14:02:09.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:09.799+0000 7f27cedce640 1 --2- 172.21.15.57:0/2985079801 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f27c80afa20 0x7f27c80c97a0 unknown :-1 s=CLOSED pgs=2029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:09.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:09.799+0000 7f27cedce640 1 --2- 172.21.15.57:0/2985079801 >> v2:172.21.15.179:3300/0 conn(0x7f27c8140a00 0x7f27c806fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:09.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:09.799+0000 7f27cedce640 1 --2- 172.21.15.57:0/2985079801 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f27c8140620 0x7f27c80703e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:09.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:09.799+0000 7f27cedce640 1 -- 172.21.15.57:0/2985079801 >> 172.21.15.57:0/2985079801 conn(0x7f27c8069930 msgr2=0x7f27c8069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:02:09.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:09.799+0000 7f27cedce640 1 -- 172.21.15.57:0/2985079801 shutdown_connections 2023-10-19T14:02:09.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:09.799+0000 7f27cedce640 1 -- 172.21.15.57:0/2985079801 wait complete. 2023-10-19T14:02:09.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:09.800+0000 7f27cedce640 1 Processor -- start 2023-10-19T14:02:09.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:09.800+0000 7f27cedce640 1 -- start start 2023-10-19T14:02:09.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:09.801+0000 7f27cedce640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f27c80afa20 0x7f27c81566f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:09.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:09.801+0000 7f27cedce640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f27c8140620 0x7f27c81f4370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:09.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:09.801+0000 7f27cedce640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f27c8140a00 0x7f27c80aeec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:09.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:09.801+0000 7f27cedce640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27c8158570 con 0x7f27c8140a00 2023-10-19T14:02:09.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:09.801+0000 7f27cedce640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27c8156c30 con 0x7f27c8140620 2023-10-19T14:02:09.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:09.801+0000 7f27cedce640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27c81f4ba0 con 0x7f27c80afa20 2023-10-19T14:02:09.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:09.801+0000 7f27cd36b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f27c8140a00 0x7f27c80aeec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:09.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:09.801+0000 7f27cd36b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f27c8140a00 0x7f27c80aeec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59660/0 (socket says 172.21.15.57:59660) 2023-10-19T14:02:09.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:09.801+0000 7f27cd36b640 1 -- 172.21.15.57:0/3241330334 learned_addr learned my addr 172.21.15.57:0/3241330334 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:02:09.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:09.801+0000 7f27ccb6a640 1 --2- 172.21.15.57:0/3241330334 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f27c80afa20 0x7f27c81566f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:09.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:09.802+0000 7f27c7fff640 1 --2- 172.21.15.57:0/3241330334 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f27c8140620 0x7f27c81f4370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:09.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:09.802+0000 7f27cd36b640 1 -- 172.21.15.57:0/3241330334 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f27c80afa20 msgr2=0x7f27c81566f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:09.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:09.802+0000 7f27cd36b640 1 --2- 172.21.15.57:0/3241330334 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f27c80afa20 0x7f27c81566f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:09.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:09.802+0000 7f27cd36b640 1 -- 172.21.15.57:0/3241330334 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f27c8140620 msgr2=0x7f27c81f4370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:09.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:09.802+0000 7f27cd36b640 1 --2- 172.21.15.57:0/3241330334 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f27c8140620 0x7f27c81f4370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:09.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:09.802+0000 7f27cd36b640 1 -- 172.21.15.57:0/3241330334 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27b0009560 con 0x7f27c8140a00 2023-10-19T14:02:09.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:09.802+0000 7f27cd36b640 1 --2- 172.21.15.57:0/3241330334 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f27c8140a00 0x7f27c80aeec0 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 crypto rx=0x7f27b0009230 tx=0x7f27b002dac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:02:09.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:09.802+0000 7f27c57fa640 1 -- 172.21.15.57:0/3241330334 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27b0003800 con 0x7f27c8140a00 2023-10-19T14:02:09.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:09.802+0000 7f27c57fa640 1 -- 172.21.15.57:0/3241330334 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f27b0003a50 con 0x7f27c8140a00 2023-10-19T14:02:09.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:09.802+0000 7f27cedce640 1 -- 172.21.15.57:0/3241330334 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27c81f4e80 con 0x7f27c8140a00 2023-10-19T14:02:09.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:09.802+0000 7f27c57fa640 1 -- 172.21.15.57:0/3241330334 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27b0003e60 con 0x7f27c8140a00 2023-10-19T14:02:09.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:09.802+0000 7f27cedce640 1 -- 172.21.15.57:0/3241330334 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27c8158190 con 0x7f27c8140a00 2023-10-19T14:02:09.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:09.803+0000 7f27cedce640 1 -- 172.21.15.57:0/3241330334 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27c81586b0 con 0x7f27c8140a00 2023-10-19T14:02:09.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:09.804+0000 7f27c57fa640 1 -- 172.21.15.57:0/3241330334 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 99) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f27b003e450 con 0x7f27c8140a00 2023-10-19T14:02:09.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:09.804+0000 7f27c57fa640 1 --2- 172.21.15.57:0/3241330334 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f27a806af60 0x7f27a806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:09.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:09.804+0000 7f27c57fa640 1 -- 172.21.15.57:0/3241330334 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 5957+0+0 (secure 0 0 0) 0x7f27b00ac610 con 0x7f27c8140a00 2023-10-19T14:02:09.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:09.805+0000 7f27ccb6a640 1 --2- 172.21.15.57:0/3241330334 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f27a806af60 0x7f27a806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:09.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:09.805+0000 7f27ccb6a640 1 --2- 172.21.15.57:0/3241330334 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f27a806af60 0x7f27a806d420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f27c0005fd0 tx=0x7f27c000b390 comp rx=0 tx=0).ready entity=mgr.8315 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:02:09.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:09.809+0000 7f27c57fa640 1 -- 172.21.15.57:0/3241330334 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f27b0077760 con 0x7f27c8140a00 2023-10-19T14:02:10.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:10.022+0000 7f27cedce640 1 -- 172.21.15.57:0/3241330334 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f27c80b0150 con 0x7f27c8140a00 2023-10-19T14:02:10.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:10.670+0000 7f27c57fa640 1 -- 172.21.15.57:0/3241330334 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 23 and data pool 24 v75) v1 ==== 174+0+0 (secure 0 0 0) 0x7f27b00420c0 con 0x7f27c8140a00 2023-10-19T14:02:10.675 INFO:teuthology.orchestra.run.smithi057.stderr:new fs with metadata pool 23 and data pool 24 2023-10-19T14:02:10.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:10.680+0000 7f27cedce640 1 -- 172.21.15.57:0/3241330334 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f27a806af60 msgr2=0x7f27a806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:10.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:10.680+0000 7f27cedce640 1 --2- 172.21.15.57:0/3241330334 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f27a806af60 0x7f27a806d420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f27c0005fd0 tx=0x7f27c000b390 comp rx=0 tx=0).stop 2023-10-19T14:02:10.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:10.680+0000 7f27cedce640 1 -- 172.21.15.57:0/3241330334 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f27c8140a00 msgr2=0x7f27c80aeec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:10.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:10.680+0000 7f27cedce640 1 --2- 172.21.15.57:0/3241330334 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f27c8140a00 0x7f27c80aeec0 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 crypto rx=0x7f27b0009230 tx=0x7f27b002dac0 comp rx=0 tx=0).stop 2023-10-19T14:02:10.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:10.681+0000 7f27cedce640 1 -- 172.21.15.57:0/3241330334 shutdown_connections 2023-10-19T14:02:10.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:10.681+0000 7f27cedce640 1 --2- 172.21.15.57:0/3241330334 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f27a806af60 0x7f27a806d420 secure :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f27c0005fd0 tx=0x7f27c000b390 comp rx=0 tx=0).stop 2023-10-19T14:02:10.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:10.681+0000 7f27cedce640 1 --2- 172.21.15.57:0/3241330334 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f27c8140a00 0x7f27c80aeec0 unknown :-1 s=CLOSED pgs=2030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:10.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:10.681+0000 7f27cedce640 1 --2- 172.21.15.57:0/3241330334 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f27c8140620 0x7f27c81f4370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:10.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:10.681+0000 7f27cedce640 1 --2- 172.21.15.57:0/3241330334 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f27c80afa20 0x7f27c81566f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:10.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:10.681+0000 7f27cedce640 1 -- 172.21.15.57:0/3241330334 >> 172.21.15.57:0/3241330334 conn(0x7f27c8069930 msgr2=0x7f27c8156ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:02:10.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:10.684+0000 7f27cedce640 1 -- 172.21.15.57:0/3241330334 shutdown_connections 2023-10-19T14:02:10.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:10.684+0000 7f27cedce640 1 -- 172.21.15.57:0/3241330334 wait complete. 2023-10-19T14:02:10.706 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T14:02:10.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:10.818+0000 7f2b788c2640 1 -- 172.21.15.57:0/1775303813 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2b74070920 msgr2=0x7f2b740affe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:10.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:10.818+0000 7f2b788c2640 1 --2- 172.21.15.57:0/1775303813 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2b74070920 0x7f2b740affe0 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 crypto rx=0x7f2b6c005fd0 tx=0x7f2b6c02c1b0 comp rx=0 tx=0).stop 2023-10-19T14:02:10.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:10.818+0000 7f2b788c2640 1 -- 172.21.15.57:0/1775303813 shutdown_connections 2023-10-19T14:02:10.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:10.818+0000 7f2b788c2640 1 --2- 172.21.15.57:0/1775303813 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2b74070920 0x7f2b740affe0 unknown :-1 s=CLOSED pgs=2031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:10.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:10.818+0000 7f2b788c2640 1 --2- 172.21.15.57:0/1775303813 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2b74141690 0x7f2b740703e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:10.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:10.818+0000 7f2b788c2640 1 -- 172.21.15.57:0/1775303813 >> 172.21.15.57:0/1775303813 conn(0x7f2b74069930 msgr2=0x7f2b74069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:02:10.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:10.818+0000 7f2b788c2640 1 -- 172.21.15.57:0/1775303813 shutdown_connections 2023-10-19T14:02:10.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:10.818+0000 7f2b788c2640 1 -- 172.21.15.57:0/1775303813 wait complete. 2023-10-19T14:02:10.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:10.820+0000 7f2b788c2640 1 Processor -- start 2023-10-19T14:02:10.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:10.820+0000 7f2b788c2640 1 -- start start 2023-10-19T14:02:10.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:10.821+0000 7f2b788c2640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2b74070920 0x7f2b741db670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:10.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:10.821+0000 7f2b788c2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2b7413e520 0x7f2b740ac660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:10.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:10.821+0000 7f2b788c2640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2b74141690 0x7f2b740ca870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:10.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:10.821+0000 7f2b788c2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b741dbbb0 con 0x7f2b7413e520 2023-10-19T14:02:10.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:10.821+0000 7f2b788c2640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b740aa8c0 con 0x7f2b74070920 2023-10-19T14:02:10.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:10.821+0000 7f2b788c2640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b741e4990 con 0x7f2b74141690 2023-10-19T14:02:10.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:10.821+0000 7f2b69d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2b7413e520 0x7f2b740ac660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:10.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:10.821+0000 7f2b69d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2b7413e520 0x7f2b740ac660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59678/0 (socket says 172.21.15.57:59678) 2023-10-19T14:02:10.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:10.821+0000 7f2b69d9b640 1 -- 172.21.15.57:0/2800914552 learned_addr learned my addr 172.21.15.57:0/2800914552 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:02:10.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:10.821+0000 7f2b72d9d640 1 --2- 172.21.15.57:0/2800914552 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2b74141690 0x7f2b740ca870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:10.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:10.821+0000 7f2b7259c640 1 --2- 172.21.15.57:0/2800914552 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2b74070920 0x7f2b741db670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:10.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:10.821+0000 7f2b69d9b640 1 -- 172.21.15.57:0/2800914552 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2b74141690 msgr2=0x7f2b740ca870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:10.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:10.821+0000 7f2b69d9b640 1 --2- 172.21.15.57:0/2800914552 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2b74141690 0x7f2b740ca870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:10.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:10.821+0000 7f2b69d9b640 1 -- 172.21.15.57:0/2800914552 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2b74070920 msgr2=0x7f2b741db670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:10.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:10.821+0000 7f2b69d9b640 1 --2- 172.21.15.57:0/2800914552 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2b74070920 0x7f2b741db670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:10.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:10.821+0000 7f2b69d9b640 1 -- 172.21.15.57:0/2800914552 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b6c009900 con 0x7f2b7413e520 2023-10-19T14:02:10.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:10.821+0000 7f2b69d9b640 1 --2- 172.21.15.57:0/2800914552 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2b7413e520 0x7f2b740ac660 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c00b410 tx=0x7f2b5c00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:02:10.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:10.822+0000 7f2b6b7fe640 1 -- 172.21.15.57:0/2800914552 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b5c011830 con 0x7f2b7413e520 2023-10-19T14:02:10.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:10.822+0000 7f2b6b7fe640 1 -- 172.21.15.57:0/2800914552 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f2b5c011aa0 con 0x7f2b7413e520 2023-10-19T14:02:10.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:10.822+0000 7f2b788c2640 1 -- 172.21.15.57:0/2800914552 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b741e4c70 con 0x7f2b7413e520 2023-10-19T14:02:10.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:10.822+0000 7f2b6b7fe640 1 -- 172.21.15.57:0/2800914552 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b5c010380 con 0x7f2b7413e520 2023-10-19T14:02:10.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:10.822+0000 7f2b788c2640 1 -- 172.21.15.57:0/2800914552 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b741e5230 con 0x7f2b7413e520 2023-10-19T14:02:10.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:10.823+0000 7f2b788c2640 1 -- 172.21.15.57:0/2800914552 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b740aaae0 con 0x7f2b7413e520 2023-10-19T14:02:10.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:10.823+0000 7f2b6b7fe640 1 -- 172.21.15.57:0/2800914552 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 99) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f2b5c0104e0 con 0x7f2b7413e520 2023-10-19T14:02:10.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:10.824+0000 7f2b6b7fe640 1 --2- 172.21.15.57:0/2800914552 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f2b5406af60 0x7f2b5406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:10.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:10.824+0000 7f2b6b7fe640 1 -- 172.21.15.57:0/2800914552 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f2b5c089c10 con 0x7f2b7413e520 2023-10-19T14:02:10.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:10.824+0000 7f2b7259c640 1 --2- 172.21.15.57:0/2800914552 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f2b5406af60 0x7f2b5406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:10.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:10.824+0000 7f2b7259c640 1 --2- 172.21.15.57:0/2800914552 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f2b5406af60 0x7f2b5406d420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2b640017f0 tx=0x7f2b64009040 comp rx=0 tx=0).ready entity=mgr.8315 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:02:10.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:10.828+0000 7f2b6b7fe640 1 -- 172.21.15.57:0/2800914552 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2b740aaae0 con 0x7f2b7413e520 2023-10-19T14:02:11.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.004+0000 7f2b788c2640 1 -- 172.21.15.57:0/2800914552 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2b740aaa80 con 0x7f2b7413e520 2023-10-19T14:02:11.008 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.005+0000 7f2b6b7fe640 1 -- 172.21.15.57:0/2800914552 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v187) v1 ==== 74+0+16784 (secure 0 0 0) 0x7f2b5c054a70 con 0x7f2b7413e520 2023-10-19T14:02:11.008 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:02:11.009 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":187,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T14:02:10.024344+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2023-10-19T14:02:08.318466+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"187","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2023-10-19T14:02:08.995986+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"187","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T14:02:11.010 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.007+0000 7f2b788c2640 1 -- 172.21.15.57:0/2800914552 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f2b5406af60 msgr2=0x7f2b5406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:11.010 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.007+0000 7f2b788c2640 1 --2- 172.21.15.57:0/2800914552 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f2b5406af60 0x7f2b5406d420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2b640017f0 tx=0x7f2b64009040 comp rx=0 tx=0).stop 2023-10-19T14:02:11.011 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.007+0000 7f2b788c2640 1 -- 172.21.15.57:0/2800914552 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2b7413e520 msgr2=0x7f2b740ac660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:11.011 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.007+0000 7f2b788c2640 1 --2- 172.21.15.57:0/2800914552 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2b7413e520 0x7f2b740ac660 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c00b410 tx=0x7f2b5c00b8d0 comp rx=0 tx=0).stop 2023-10-19T14:02:11.011 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.008+0000 7f2b788c2640 1 -- 172.21.15.57:0/2800914552 shutdown_connections 2023-10-19T14:02:11.011 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.008+0000 7f2b788c2640 1 --2- 172.21.15.57:0/2800914552 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2b74141690 0x7f2b740ca870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:11.011 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.008+0000 7f2b788c2640 1 --2- 172.21.15.57:0/2800914552 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f2b5406af60 0x7f2b5406d420 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:11.012 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.008+0000 7f2b788c2640 1 --2- 172.21.15.57:0/2800914552 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2b7413e520 0x7f2b740ac660 unknown :-1 s=CLOSED pgs=2032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:11.012 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.008+0000 7f2b788c2640 1 --2- 172.21.15.57:0/2800914552 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2b74070920 0x7f2b741db670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:11.012 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.008+0000 7f2b788c2640 1 -- 172.21.15.57:0/2800914552 >> 172.21.15.57:0/2800914552 conn(0x7f2b74069930 msgr2=0x7f2b74069d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:02:11.012 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.008+0000 7f2b788c2640 1 -- 172.21.15.57:0/2800914552 shutdown_connections 2023-10-19T14:02:11.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.008+0000 7f2b788c2640 1 -- 172.21.15.57:0/2800914552 wait complete. 2023-10-19T14:02:11.021 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T14:02:11.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.128+0000 7f10b2773640 1 -- 172.21.15.57:0/2369431690 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f10ac0c9dc0 msgr2=0x7f10ac06b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:11.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.128+0000 7f10b2773640 1 --2- 172.21.15.57:0/2369431690 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f10ac0c9dc0 0x7f10ac06b2e0 secure :-1 s=READY pgs=2033 cs=0 l=1 rev1=1 crypto rx=0x7f109c005fd0 tx=0x7f109c02be10 comp rx=0 tx=0).stop 2023-10-19T14:02:11.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.128+0000 7f10b2773640 1 -- 172.21.15.57:0/2369431690 shutdown_connections 2023-10-19T14:02:11.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.128+0000 7f10b2773640 1 --2- 172.21.15.57:0/2369431690 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f10ac0ab500 0x7f10ac0ab990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:11.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.128+0000 7f10b2773640 1 --2- 172.21.15.57:0/2369431690 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f10ac0c9dc0 0x7f10ac06b2e0 unknown :-1 s=CLOSED pgs=2033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:11.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.128+0000 7f10b2773640 1 -- 172.21.15.57:0/2369431690 >> 172.21.15.57:0/2369431690 conn(0x7f10ac069930 msgr2=0x7f10ac069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:02:11.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.128+0000 7f10b2773640 1 -- 172.21.15.57:0/2369431690 shutdown_connections 2023-10-19T14:02:11.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.128+0000 7f10b2773640 1 -- 172.21.15.57:0/2369431690 wait complete. 2023-10-19T14:02:11.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.129+0000 7f10b2773640 1 Processor -- start 2023-10-19T14:02:11.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.129+0000 7f10b2773640 1 -- start start 2023-10-19T14:02:11.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.130+0000 7f10b2773640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f10ac06af00 0x7f10ac0a85f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:11.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.130+0000 7f10b2773640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f10ac0ab500 0x7f10ac1eef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:11.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.130+0000 7f10b2773640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f10ac0c9dc0 0x7f10ac0ad720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:11.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.130+0000 7f10b2773640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10ac0a8b30 con 0x7f10ac0c9dc0 2023-10-19T14:02:11.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.130+0000 7f10b2773640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10ac0af240 con 0x7f10ac0ab500 2023-10-19T14:02:11.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.130+0000 7f10b2773640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10ac0af520 con 0x7f10ac06af00 2023-10-19T14:02:11.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.130+0000 7f10b0d10640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f10ac0c9dc0 0x7f10ac0ad720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:11.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.130+0000 7f10b0d10640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f10ac0c9dc0 0x7f10ac0ad720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59686/0 (socket says 172.21.15.57:59686) 2023-10-19T14:02:11.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.130+0000 7f10b0d10640 1 -- 172.21.15.57:0/1018569158 learned_addr learned my addr 172.21.15.57:0/1018569158 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:02:11.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.130+0000 7f10ab7fe640 1 --2- 172.21.15.57:0/1018569158 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f10ac0ab500 0x7f10ac1eef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:11.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.130+0000 7f10abfff640 1 --2- 172.21.15.57:0/1018569158 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f10ac06af00 0x7f10ac0a85f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:11.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.130+0000 7f10b0d10640 1 -- 172.21.15.57:0/1018569158 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f10ac06af00 msgr2=0x7f10ac0a85f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:11.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.130+0000 7f10b0d10640 1 --2- 172.21.15.57:0/1018569158 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f10ac06af00 0x7f10ac0a85f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:11.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.130+0000 7f10b0d10640 1 -- 172.21.15.57:0/1018569158 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f10ac0ab500 msgr2=0x7f10ac1eef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:11.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.130+0000 7f10b0d10640 1 --2- 172.21.15.57:0/1018569158 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f10ac0ab500 0x7f10ac1eef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:11.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.130+0000 7f10b0d10640 1 -- 172.21.15.57:0/1018569158 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f109c009560 con 0x7f10ac0c9dc0 2023-10-19T14:02:11.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.130+0000 7f10ab7fe640 1 --2- 172.21.15.57:0/1018569158 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f10ac0ab500 0x7f10ac1eef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:02:11.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.131+0000 7f10b0d10640 1 --2- 172.21.15.57:0/1018569158 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f10ac0c9dc0 0x7f10ac0ad720 secure :-1 s=READY pgs=2034 cs=0 l=1 rev1=1 crypto rx=0x7f10a400eb10 tx=0x7f10a400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:02:11.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.131+0000 7f10a8ff9640 1 -- 172.21.15.57:0/1018569158 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10a4004b20 con 0x7f10ac0c9dc0 2023-10-19T14:02:11.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.131+0000 7f10a8ff9640 1 -- 172.21.15.57:0/1018569158 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f10a4004ca0 con 0x7f10ac0c9dc0 2023-10-19T14:02:11.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.131+0000 7f10b2773640 1 -- 172.21.15.57:0/1018569158 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10ac0af800 con 0x7f10ac0c9dc0 2023-10-19T14:02:11.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.131+0000 7f10a8ff9640 1 -- 172.21.15.57:0/1018569158 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10a4005280 con 0x7f10ac0c9dc0 2023-10-19T14:02:11.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.131+0000 7f10b2773640 1 -- 172.21.15.57:0/1018569158 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10ac0afdc0 con 0x7f10ac0c9dc0 2023-10-19T14:02:11.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.132+0000 7f10b2773640 1 -- 172.21.15.57:0/1018569158 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10ac0affe0 con 0x7f10ac0c9dc0 2023-10-19T14:02:11.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.132+0000 7f10a8ff9640 1 -- 172.21.15.57:0/1018569158 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 99) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f10a40053e0 con 0x7f10ac0c9dc0 2023-10-19T14:02:11.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.133+0000 7f10a8ff9640 1 --2- 172.21.15.57:0/1018569158 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f108c06af60 0x7f108c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:11.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.133+0000 7f10a8ff9640 1 -- 172.21.15.57:0/1018569158 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f10a4014070 con 0x7f10ac0c9dc0 2023-10-19T14:02:11.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.133+0000 7f10abfff640 1 --2- 172.21.15.57:0/1018569158 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f108c06af60 0x7f108c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:11.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.133+0000 7f10abfff640 1 --2- 172.21.15.57:0/1018569158 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f108c06af60 0x7f108c06d420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1094001640 tx=0x7f1094009040 comp rx=0 tx=0).ready entity=mgr.8315 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:02:11.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.137+0000 7f10a8ff9640 1 -- 172.21.15.57:0/1018569158 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f10a4055500 con 0x7f10ac0c9dc0 2023-10-19T14:02:11.315 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.313+0000 7f10b2773640 1 -- 172.21.15.57:0/1018569158 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f10ac0690e0 con 0x7f10ac0c9dc0 2023-10-19T14:02:11.316 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.314+0000 7f10a8ff9640 1 -- 172.21.15.57:0/1018569158 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v187) v1 ==== 74+0+16784 (secure 0 0 0) 0x7f10a4058b20 con 0x7f10ac0c9dc0 2023-10-19T14:02:11.317 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:02:11.317 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":187,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T14:02:10.024344+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2023-10-19T14:02:08.318466+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"187","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2023-10-19T14:02:08.995986+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"187","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T14:02:11.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.316+0000 7f10b2773640 1 -- 172.21.15.57:0/1018569158 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f108c06af60 msgr2=0x7f108c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:11.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.316+0000 7f10b2773640 1 --2- 172.21.15.57:0/1018569158 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f108c06af60 0x7f108c06d420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1094001640 tx=0x7f1094009040 comp rx=0 tx=0).stop 2023-10-19T14:02:11.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.316+0000 7f10b2773640 1 -- 172.21.15.57:0/1018569158 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f10ac0c9dc0 msgr2=0x7f10ac0ad720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:11.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.316+0000 7f10b2773640 1 --2- 172.21.15.57:0/1018569158 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f10ac0c9dc0 0x7f10ac0ad720 secure :-1 s=READY pgs=2034 cs=0 l=1 rev1=1 crypto rx=0x7f10a400eb10 tx=0x7f10a400c2d0 comp rx=0 tx=0).stop 2023-10-19T14:02:11.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.316+0000 7f10b2773640 1 -- 172.21.15.57:0/1018569158 shutdown_connections 2023-10-19T14:02:11.320 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.316+0000 7f10b2773640 1 --2- 172.21.15.57:0/1018569158 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f108c06af60 0x7f108c06d420 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:11.320 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.316+0000 7f10b2773640 1 --2- 172.21.15.57:0/1018569158 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f10ac0c9dc0 0x7f10ac0ad720 unknown :-1 s=CLOSED pgs=2034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:11.320 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.316+0000 7f10b2773640 1 --2- 172.21.15.57:0/1018569158 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f10ac0ab500 0x7f10ac1eef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:11.320 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.316+0000 7f10b2773640 1 --2- 172.21.15.57:0/1018569158 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f10ac06af00 0x7f10ac0a85f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:11.320 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.316+0000 7f10b2773640 1 -- 172.21.15.57:0/1018569158 >> 172.21.15.57:0/1018569158 conn(0x7f10ac069930 msgr2=0x7f10ac15d070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:02:11.320 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.316+0000 7f10b2773640 1 -- 172.21.15.57:0/1018569158 shutdown_connections 2023-10-19T14:02:11.321 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.316+0000 7f10b2773640 1 -- 172.21.15.57:0/1018569158 wait complete. 2023-10-19T14:02:11.329 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-19T14:02:11.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.439+0000 7f9924832640 1 -- 172.21.15.57:0/3325708540 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9920158b50 msgr2=0x7f992006c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:11.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.439+0000 7f9924832640 1 --2- 172.21.15.57:0/3325708540 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9920158b50 0x7f992006c960 secure :-1 s=READY pgs=2035 cs=0 l=1 rev1=1 crypto rx=0x7f9914005fd0 tx=0x7f991402be10 comp rx=0 tx=0).stop 2023-10-19T14:02:11.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.439+0000 7f9924832640 1 -- 172.21.15.57:0/3325708540 shutdown_connections 2023-10-19T14:02:11.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.439+0000 7f9924832640 1 --2- 172.21.15.57:0/3325708540 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f992006cea0 0x7f99200b0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:11.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.439+0000 7f9924832640 1 --2- 172.21.15.57:0/3325708540 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9920158b50 0x7f992006c960 unknown :-1 s=CLOSED pgs=2035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:11.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.439+0000 7f9924832640 1 -- 172.21.15.57:0/3325708540 >> 172.21.15.57:0/3325708540 conn(0x7f99200cc970 msgr2=0x7f99200ccd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:02:11.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.439+0000 7f9924832640 1 -- 172.21.15.57:0/3325708540 shutdown_connections 2023-10-19T14:02:11.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.439+0000 7f9924832640 1 -- 172.21.15.57:0/3325708540 wait complete. 2023-10-19T14:02:11.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.440+0000 7f9924832640 1 Processor -- start 2023-10-19T14:02:11.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.440+0000 7f9924832640 1 -- start start 2023-10-19T14:02:11.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.441+0000 7f9924832640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f992006cea0 0x7f99200a64b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:11.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.441+0000 7f9924832640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9920158770 0x7f99201496b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:11.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.441+0000 7f9924832640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9920158b50 0x7f9920156c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:11.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.441+0000 7f9924832640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99200a69f0 con 0x7f992006cea0 2023-10-19T14:02:11.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.441+0000 7f9924832640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99200ca0f0 con 0x7f9920158b50 2023-10-19T14:02:11.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.441+0000 7f9924832640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99200ca3d0 con 0x7f9920158770 2023-10-19T14:02:11.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.441+0000 7f991e59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f992006cea0 0x7f99200a64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:11.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.441+0000 7f991e59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f992006cea0 0x7f99200a64b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59710/0 (socket says 172.21.15.57:59710) 2023-10-19T14:02:11.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.441+0000 7f991e59c640 1 -- 172.21.15.57:0/1249438220 learned_addr learned my addr 172.21.15.57:0/1249438220 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:02:11.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.441+0000 7f991dd9b640 1 --2- 172.21.15.57:0/1249438220 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9920158770 0x7f99201496b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:11.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.441+0000 7f991ed9d640 1 --2- 172.21.15.57:0/1249438220 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9920158b50 0x7f9920156c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:11.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.442+0000 7f991e59c640 1 -- 172.21.15.57:0/1249438220 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9920158770 msgr2=0x7f99201496b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:11.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.442+0000 7f991e59c640 1 --2- 172.21.15.57:0/1249438220 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9920158770 0x7f99201496b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:11.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.442+0000 7f991e59c640 1 -- 172.21.15.57:0/1249438220 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9920158b50 msgr2=0x7f9920156c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:11.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.442+0000 7f991e59c640 1 --2- 172.21.15.57:0/1249438220 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9920158b50 0x7f9920156c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:11.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.442+0000 7f991e59c640 1 -- 172.21.15.57:0/1249438220 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9914009560 con 0x7f992006cea0 2023-10-19T14:02:11.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.442+0000 7f991e59c640 1 --2- 172.21.15.57:0/1249438220 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f992006cea0 0x7f99200a64b0 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 crypto rx=0x7f991800c450 tx=0x7f991800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:02:11.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.442+0000 7f990effd640 1 -- 172.21.15.57:0/1249438220 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99180052d0 con 0x7f992006cea0 2023-10-19T14:02:11.452 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.442+0000 7f990effd640 1 -- 172.21.15.57:0/1249438220 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f9918005430 con 0x7f992006cea0 2023-10-19T14:02:11.452 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.442+0000 7f9924832640 1 -- 172.21.15.57:0/1249438220 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99200ca6b0 con 0x7f992006cea0 2023-10-19T14:02:11.452 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.442+0000 7f990effd640 1 -- 172.21.15.57:0/1249438220 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9918012890 con 0x7f992006cea0 2023-10-19T14:02:11.452 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.442+0000 7f9924832640 1 -- 172.21.15.57:0/1249438220 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99200cac70 con 0x7f992006cea0 2023-10-19T14:02:11.452 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.443+0000 7f9924832640 1 -- 172.21.15.57:0/1249438220 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f992004dae0 con 0x7f992006cea0 2023-10-19T14:02:11.453 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.444+0000 7f990effd640 1 -- 172.21.15.57:0/1249438220 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 99) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f99180035f0 con 0x7f992006cea0 2023-10-19T14:02:11.453 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.444+0000 7f990effd640 1 --2- 172.21.15.57:0/1249438220 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f990006af60 0x7f990006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:11.453 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.444+0000 7f990effd640 1 -- 172.21.15.57:0/1249438220 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f99180894d0 con 0x7f992006cea0 2023-10-19T14:02:11.453 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.444+0000 7f991dd9b640 1 --2- 172.21.15.57:0/1249438220 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f990006af60 0x7f990006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:11.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.445+0000 7f991dd9b640 1 --2- 172.21.15.57:0/1249438220 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f990006af60 0x7f990006d420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9914035000 tx=0x7f9914004730 comp rx=0 tx=0).ready entity=mgr.8315 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:02:11.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.448+0000 7f990effd640 1 -- 172.21.15.57:0/1249438220 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f99180543c0 con 0x7f992006cea0 2023-10-19T14:02:11.663 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.661+0000 7f9924832640 1 -- 172.21.15.57:0/1249438220 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f99200af230 con 0x7f992006cea0 2023-10-19T14:02:11.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.681+0000 7f990effd640 1 -- 172.21.15.57:0/1249438220 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v76) v1 ==== 118+0+0 (secure 0 0 0) 0x7f991805ce60 con 0x7f992006cea0 2023-10-19T14:02:11.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.683+0000 7f9924832640 1 -- 172.21.15.57:0/1249438220 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f990006af60 msgr2=0x7f990006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:11.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.683+0000 7f9924832640 1 --2- 172.21.15.57:0/1249438220 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f990006af60 0x7f990006d420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9914035000 tx=0x7f9914004730 comp rx=0 tx=0).stop 2023-10-19T14:02:11.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.683+0000 7f9924832640 1 -- 172.21.15.57:0/1249438220 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f992006cea0 msgr2=0x7f99200a64b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:11.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.683+0000 7f9924832640 1 --2- 172.21.15.57:0/1249438220 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f992006cea0 0x7f99200a64b0 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 crypto rx=0x7f991800c450 tx=0x7f991800c910 comp rx=0 tx=0).stop 2023-10-19T14:02:11.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.683+0000 7f9924832640 1 -- 172.21.15.57:0/1249438220 shutdown_connections 2023-10-19T14:02:11.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.683+0000 7f9924832640 1 --2- 172.21.15.57:0/1249438220 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9920158b50 0x7f9920156c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:11.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.683+0000 7f9924832640 1 --2- 172.21.15.57:0/1249438220 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9920158770 0x7f99201496b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:11.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.683+0000 7f9924832640 1 --2- 172.21.15.57:0/1249438220 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f990006af60 0x7f990006d420 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:11.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.683+0000 7f9924832640 1 --2- 172.21.15.57:0/1249438220 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f992006cea0 0x7f99200a64b0 unknown :-1 s=CLOSED pgs=2036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:11.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.683+0000 7f9924832640 1 -- 172.21.15.57:0/1249438220 >> 172.21.15.57:0/1249438220 conn(0x7f99200cc970 msgr2=0x7f99200ae700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:02:11.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.683+0000 7f9924832640 1 -- 172.21.15.57:0/1249438220 shutdown_connections 2023-10-19T14:02:11.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.683+0000 7f9924832640 1 -- 172.21.15.57:0/1249438220 wait complete. 2023-10-19T14:02:11.696 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-19T14:02:11.696 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-19T14:02:11.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.806+0000 7f1086064640 1 -- 172.21.15.57:0/3256904160 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f108013d350 msgr2=0x7f10800cbd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:11.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.806+0000 7f1086064640 1 --2- 172.21.15.57:0/3256904160 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f108013d350 0x7f10800cbd60 secure :-1 s=READY pgs=2037 cs=0 l=1 rev1=1 crypto rx=0x7f108006fa40 tx=0x7f107402bda0 comp rx=0 tx=0).stop 2023-10-19T14:02:11.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.806+0000 7f1086064640 1 -- 172.21.15.57:0/3256904160 shutdown_connections 2023-10-19T14:02:11.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.806+0000 7f1086064640 1 --2- 172.21.15.57:0/3256904160 >> v2:172.21.15.179:3300/0 conn(0x7f10800c96b0 0x7f10800c9b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:11.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.806+0000 7f1086064640 1 --2- 172.21.15.57:0/3256904160 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f108013d350 0x7f10800cbd60 unknown :-1 s=CLOSED pgs=2037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:11.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.806+0000 7f1086064640 1 --2- 172.21.15.57:0/3256904160 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f10800cb980 0x7f108013db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:11.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.806+0000 7f1086064640 1 -- 172.21.15.57:0/3256904160 >> 172.21.15.57:0/3256904160 conn(0x7f1080069930 msgr2=0x7f1080069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:02:11.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.806+0000 7f1086064640 1 -- 172.21.15.57:0/3256904160 shutdown_connections 2023-10-19T14:02:11.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.806+0000 7f1086064640 1 -- 172.21.15.57:0/3256904160 wait complete. 2023-10-19T14:02:11.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.807+0000 7f1086064640 1 Processor -- start 2023-10-19T14:02:11.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.807+0000 7f1086064640 1 -- start start 2023-10-19T14:02:11.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.808+0000 7f1086064640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f10800c96b0 0x7f1080159160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:11.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.808+0000 7f1086064640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f10800cb980 0x7f10801364f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:11.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.808+0000 7f1086064640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1080136a30 0x7f1080134b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:11.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.808+0000 7f1086064640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f108013b0a0 con 0x7f1080136a30 2023-10-19T14:02:11.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.808+0000 7f1086064640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f108013bbe0 con 0x7f10800c96b0 2023-10-19T14:02:11.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.808+0000 7f1086064640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f108006cc10 con 0x7f10800cb980 2023-10-19T14:02:11.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.808+0000 7f107ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1080136a30 0x7f1080134b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:11.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.808+0000 7f107ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1080136a30 0x7f1080134b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59722/0 (socket says 172.21.15.57:59722) 2023-10-19T14:02:11.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.808+0000 7f107ffff640 1 -- 172.21.15.57:0/279820281 learned_addr learned my addr 172.21.15.57:0/279820281 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:02:11.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.808+0000 7f107f7fe640 1 --2- 172.21.15.57:0/279820281 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f10800c96b0 0x7f1080159160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:11.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.808+0000 7f107effd640 1 --2- 172.21.15.57:0/279820281 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f10800cb980 0x7f10801364f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:11.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.808+0000 7f107ffff640 1 -- 172.21.15.57:0/279820281 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f10800cb980 msgr2=0x7f10801364f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:11.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.808+0000 7f107ffff640 1 --2- 172.21.15.57:0/279820281 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f10800cb980 0x7f10801364f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:11.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.808+0000 7f107ffff640 1 -- 172.21.15.57:0/279820281 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f10800c96b0 msgr2=0x7f1080159160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:11.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.808+0000 7f107ffff640 1 --2- 172.21.15.57:0/279820281 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f10800c96b0 0x7f1080159160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:11.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.808+0000 7f107ffff640 1 -- 172.21.15.57:0/279820281 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1074009560 con 0x7f1080136a30 2023-10-19T14:02:11.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.808+0000 7f107effd640 1 --2- 172.21.15.57:0/279820281 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f10800cb980 0x7f10801364f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:02:11.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.809+0000 7f107ffff640 1 --2- 172.21.15.57:0/279820281 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1080136a30 0x7f1080134b80 secure :-1 s=READY pgs=2038 cs=0 l=1 rev1=1 crypto rx=0x7f107800e980 tx=0x7f107800ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:02:11.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.809+0000 7f1067fff640 1 -- 172.21.15.57:0/279820281 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1078004d20 con 0x7f1080136a30 2023-10-19T14:02:11.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.809+0000 7f1067fff640 1 -- 172.21.15.57:0/279820281 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f107800cb20 con 0x7f1080136a30 2023-10-19T14:02:11.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.809+0000 7f1086064640 1 -- 172.21.15.57:0/279820281 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f108013b2b0 con 0x7f1080136a30 2023-10-19T14:02:11.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.809+0000 7f1067fff640 1 -- 172.21.15.57:0/279820281 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1078005020 con 0x7f1080136a30 2023-10-19T14:02:11.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.809+0000 7f1086064640 1 -- 172.21.15.57:0/279820281 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f108013b870 con 0x7f1080136a30 2023-10-19T14:02:11.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.810+0000 7f1086064640 1 -- 172.21.15.57:0/279820281 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f108013bd90 con 0x7f1080136a30 2023-10-19T14:02:11.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.810+0000 7f1067fff640 1 -- 172.21.15.57:0/279820281 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 99) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f1078005180 con 0x7f1080136a30 2023-10-19T14:02:11.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.811+0000 7f1067fff640 1 --2- 172.21.15.57:0/279820281 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f106c06af60 0x7f106c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:11.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.811+0000 7f1067fff640 1 -- 172.21.15.57:0/279820281 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f107808a360 con 0x7f1080136a30 2023-10-19T14:02:11.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.811+0000 7f107f7fe640 1 --2- 172.21.15.57:0/279820281 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f106c06af60 0x7f106c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:11.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.812+0000 7f107f7fe640 1 --2- 172.21.15.57:0/279820281 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f106c06af60 0x7f106c06d420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f107000bbe0 tx=0x7f107000b390 comp rx=0 tx=0).ready entity=mgr.8315 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:02:11.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:11.815+0000 7f1067fff640 1 -- 172.21.15.57:0/279820281 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f108013bd90 con 0x7f1080136a30 2023-10-19T14:02:12.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:12.028+0000 7f1086064640 1 -- 172.21.15.57:0/279820281 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f108013bd90 con 0x7f1080136a30 2023-10-19T14:02:12.645 INFO:tasks.ceph.mds.d.smithi179.stderr:2023-10-19T14:02:12.642+0000 7fb19be3c640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-19T14:02:12.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:12.692+0000 7f1067fff640 1 -- 172.21.15.57:0/279820281 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v77) v1 ==== 122+0+0 (secure 0 0 0) 0x7f108013bd90 con 0x7f1080136a30 2023-10-19T14:02:12.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:12.694+0000 7f1086064640 1 -- 172.21.15.57:0/279820281 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f106c06af60 msgr2=0x7f106c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:12.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:12.694+0000 7f1086064640 1 --2- 172.21.15.57:0/279820281 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f106c06af60 0x7f106c06d420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f107000bbe0 tx=0x7f107000b390 comp rx=0 tx=0).stop 2023-10-19T14:02:12.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:12.695+0000 7f1086064640 1 -- 172.21.15.57:0/279820281 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1080136a30 msgr2=0x7f1080134b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:12.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:12.695+0000 7f1086064640 1 --2- 172.21.15.57:0/279820281 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1080136a30 0x7f1080134b80 secure :-1 s=READY pgs=2038 cs=0 l=1 rev1=1 crypto rx=0x7f107800e980 tx=0x7f107800ee40 comp rx=0 tx=0).stop 2023-10-19T14:02:12.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:12.695+0000 7f1086064640 1 -- 172.21.15.57:0/279820281 shutdown_connections 2023-10-19T14:02:12.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:12.695+0000 7f1086064640 1 --2- 172.21.15.57:0/279820281 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f106c06af60 0x7f106c06d420 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:12.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:12.695+0000 7f1086064640 1 --2- 172.21.15.57:0/279820281 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1080136a30 0x7f1080134b80 unknown :-1 s=CLOSED pgs=2038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:12.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:12.695+0000 7f1086064640 1 --2- 172.21.15.57:0/279820281 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f10800cb980 0x7f10801364f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:12.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:12.695+0000 7f1086064640 1 --2- 172.21.15.57:0/279820281 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f10800c96b0 0x7f1080159160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:12.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:12.695+0000 7f1086064640 1 -- 172.21.15.57:0/279820281 >> 172.21.15.57:0/279820281 conn(0x7f1080069930 msgr2=0x7f10800702c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:02:12.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:12.695+0000 7f1086064640 1 -- 172.21.15.57:0/279820281 shutdown_connections 2023-10-19T14:02:12.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:12.695+0000 7f1086064640 1 -- 172.21.15.57:0/279820281 wait complete. 2023-10-19T14:02:12.711 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-19T14:02:12.711 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T14:02:12.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:12.824+0000 7ff8d5221640 1 -- 172.21.15.57:0/692135436 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff8d00cf340 msgr2=0x7ff8d00d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:12.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:12.824+0000 7ff8d5221640 1 --2- 172.21.15.57:0/692135436 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff8d00cf340 0x7ff8d00d1910 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 crypto rx=0x7ff8d008a810 tx=0x7ff8c802bd10 comp rx=0 tx=0).stop 2023-10-19T14:02:12.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:12.824+0000 7ff8d5221640 1 -- 172.21.15.57:0/692135436 shutdown_connections 2023-10-19T14:02:12.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:12.824+0000 7ff8d5221640 1 --2- 172.21.15.57:0/692135436 >> v2:172.21.15.179:3300/0 conn(0x7ff8d015b2a0 0x7ff8d00cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:12.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:12.824+0000 7ff8d5221640 1 --2- 172.21.15.57:0/692135436 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff8d00cf340 0x7ff8d00d1910 unknown :-1 s=CLOSED pgs=2039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:12.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:12.824+0000 7ff8d5221640 1 -- 172.21.15.57:0/692135436 >> 172.21.15.57:0/692135436 conn(0x7ff8d00d0d60 msgr2=0x7ff8d00d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:02:12.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:12.824+0000 7ff8d5221640 1 -- 172.21.15.57:0/692135436 shutdown_connections 2023-10-19T14:02:12.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:12.824+0000 7ff8d5221640 1 -- 172.21.15.57:0/692135436 wait complete. 2023-10-19T14:02:12.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:12.826+0000 7ff8d5221640 1 Processor -- start 2023-10-19T14:02:12.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:12.826+0000 7ff8d5221640 1 -- start start 2023-10-19T14:02:12.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:12.826+0000 7ff8d5221640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff8d00ceb40 0x7ff8d0154dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:12.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:12.826+0000 7ff8d5221640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff8d00cf340 0x7ff8d0157e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:12.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:12.826+0000 7ff8d5221640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff8d015b2a0 0x7ff8d0158340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:12.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:12.826+0000 7ff8d5221640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d0155310 con 0x7ff8d015b2a0 2023-10-19T14:02:12.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:12.826+0000 7ff8d5221640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d015c180 con 0x7ff8d00cf340 2023-10-19T14:02:12.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:12.826+0000 7ff8d5221640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d015c420 con 0x7ff8d00ceb40 2023-10-19T14:02:12.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:12.827+0000 7ff8cf59e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff8d015b2a0 0x7ff8d0158340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:12.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:12.827+0000 7ff8cf59e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff8d015b2a0 0x7ff8d0158340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59726/0 (socket says 172.21.15.57:59726) 2023-10-19T14:02:12.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:12.827+0000 7ff8cf59e640 1 -- 172.21.15.57:0/3409662595 learned_addr learned my addr 172.21.15.57:0/3409662595 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:02:12.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:12.827+0000 7ff8ced9d640 1 --2- 172.21.15.57:0/3409662595 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff8d00ceb40 0x7ff8d0154dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:12.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:12.827+0000 7ff8ce59c640 1 --2- 172.21.15.57:0/3409662595 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff8d00cf340 0x7ff8d0157e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:12.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:12.827+0000 7ff8cf59e640 1 -- 172.21.15.57:0/3409662595 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff8d00ceb40 msgr2=0x7ff8d0154dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:12.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:12.827+0000 7ff8cf59e640 1 --2- 172.21.15.57:0/3409662595 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff8d00ceb40 0x7ff8d0154dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:12.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:12.827+0000 7ff8cf59e640 1 -- 172.21.15.57:0/3409662595 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff8d00cf340 msgr2=0x7ff8d0157e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:12.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:12.827+0000 7ff8cf59e640 1 --2- 172.21.15.57:0/3409662595 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff8d00cf340 0x7ff8d0157e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:12.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:12.827+0000 7ff8cf59e640 1 -- 172.21.15.57:0/3409662595 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8c8009560 con 0x7ff8d015b2a0 2023-10-19T14:02:12.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:12.827+0000 7ff8cf59e640 1 --2- 172.21.15.57:0/3409662595 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff8d015b2a0 0x7ff8d0158340 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 crypto rx=0x7ff8c000c780 tx=0x7ff8c000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:02:12.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:12.827+0000 7ff8b77fe640 1 -- 172.21.15.57:0/3409662595 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8c0003d80 con 0x7ff8d015b2a0 2023-10-19T14:02:12.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:12.827+0000 7ff8b77fe640 1 -- 172.21.15.57:0/3409662595 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7ff8c000ce90 con 0x7ff8d015b2a0 2023-10-19T14:02:12.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:12.827+0000 7ff8d5221640 1 -- 172.21.15.57:0/3409662595 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8d015c700 con 0x7ff8d015b2a0 2023-10-19T14:02:12.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:12.827+0000 7ff8b77fe640 1 -- 172.21.15.57:0/3409662595 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8c00052d0 con 0x7ff8d015b2a0 2023-10-19T14:02:12.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:12.827+0000 7ff8d5221640 1 -- 172.21.15.57:0/3409662595 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8d015ccc0 con 0x7ff8d015b2a0 2023-10-19T14:02:12.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:12.828+0000 7ff8d5221640 1 -- 172.21.15.57:0/3409662595 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8d015cee0 con 0x7ff8d015b2a0 2023-10-19T14:02:12.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:12.829+0000 7ff8b77fe640 1 -- 172.21.15.57:0/3409662595 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 99) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff8c0005430 con 0x7ff8d015b2a0 2023-10-19T14:02:12.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:12.829+0000 7ff8b77fe640 1 --2- 172.21.15.57:0/3409662595 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7ff8bc06af90 0x7ff8bc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:12.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:12.829+0000 7ff8b77fe640 1 -- 172.21.15.57:0/3409662595 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6057+0+0 (secure 0 0 0) 0x7ff8c0088f10 con 0x7ff8d015b2a0 2023-10-19T14:02:12.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:12.830+0000 7ff8ced9d640 1 --2- 172.21.15.57:0/3409662595 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7ff8bc06af90 0x7ff8bc06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:12.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:12.830+0000 7ff8ced9d640 1 --2- 172.21.15.57:0/3409662595 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7ff8bc06af90 0x7ff8bc06d450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff8c8035000 tx=0x7ff8c8004730 comp rx=0 tx=0).ready entity=mgr.8315 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:02:12.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:12.834+0000 7ff8b77fe640 1 -- 172.21.15.57:0/3409662595 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff8c0053d80 con 0x7ff8d015b2a0 2023-10-19T14:02:13.049 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.046+0000 7ff8d5221640 1 -- 172.21.15.57:0/3409662595 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff8d00cb9d0 con 0x7ff8d015b2a0 2023-10-19T14:02:13.049 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.047+0000 7ff8b77fe640 1 -- 172.21.15.57:0/3409662595 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+4924 (secure 0 0 0) 0x7ff8c00573a0 con 0x7ff8d015b2a0 2023-10-19T14:02:13.050 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:02:13.050 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":77,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8480,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6833/2675481384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2675481384},{"type":"v1","addr":"172.21.15.57:6833","nonce":2675481384}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74},{"gid":8486,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/2884298100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2884298100},{"type":"v1","addr":"172.21.15.179:6836","nonce":2884298100}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":73},{"gid":10641,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/505361151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":505361151},{"type":"v1","addr":"172.21.15.57:6835","nonce":505361151}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T14:02:10.024658+0000","modified":"2023-10-19T14:02:12.029940+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":10647},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10647":{"gid":10647,"name":"d","rank":0,"incarnation":75,"state":"up:active","state_seq":3,"addr":"172.21.15.179:6837/3993214977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":3993214977},{"type":"v1","addr":"172.21.15.179:6837","nonce":3993214977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2023-10-19T14:02:13.050 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 77 2023-10-19T14:02:13.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.049+0000 7ff8d5221640 1 -- 172.21.15.57:0/3409662595 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7ff8bc06af90 msgr2=0x7ff8bc06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:13.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.049+0000 7ff8d5221640 1 --2- 172.21.15.57:0/3409662595 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7ff8bc06af90 0x7ff8bc06d450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff8c8035000 tx=0x7ff8c8004730 comp rx=0 tx=0).stop 2023-10-19T14:02:13.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.049+0000 7ff8d5221640 1 -- 172.21.15.57:0/3409662595 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff8d015b2a0 msgr2=0x7ff8d0158340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:13.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.049+0000 7ff8d5221640 1 --2- 172.21.15.57:0/3409662595 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff8d015b2a0 0x7ff8d0158340 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 crypto rx=0x7ff8c000c780 tx=0x7ff8c000cc40 comp rx=0 tx=0).stop 2023-10-19T14:02:13.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.049+0000 7ff8d5221640 1 -- 172.21.15.57:0/3409662595 shutdown_connections 2023-10-19T14:02:13.052 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.049+0000 7ff8d5221640 1 --2- 172.21.15.57:0/3409662595 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7ff8bc06af90 0x7ff8bc06d450 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:13.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.049+0000 7ff8d5221640 1 --2- 172.21.15.57:0/3409662595 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff8d015b2a0 0x7ff8d0158340 unknown :-1 s=CLOSED pgs=2040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:13.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.049+0000 7ff8d5221640 1 --2- 172.21.15.57:0/3409662595 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff8d00cf340 0x7ff8d0157e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:13.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.049+0000 7ff8d5221640 1 --2- 172.21.15.57:0/3409662595 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff8d00ceb40 0x7ff8d0154dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:13.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.049+0000 7ff8d5221640 1 -- 172.21.15.57:0/3409662595 >> 172.21.15.57:0/3409662595 conn(0x7ff8d00d0d60 msgr2=0x7ff8d006b1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:02:13.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.049+0000 7ff8d5221640 1 -- 172.21.15.57:0/3409662595 shutdown_connections 2023-10-19T14:02:13.054 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.049+0000 7ff8d5221640 1 -- 172.21.15.57:0/3409662595 wait complete. 2023-10-19T14:02:13.062 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T14:02:13.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.171+0000 7f9b73983640 1 -- 172.21.15.57:0/2845218922 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b54003f90 msgr2=0x7f9b540047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:13.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.171+0000 7f9b73983640 1 --2- 172.21.15.57:0/2845218922 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b54003f90 0x7f9b540047c0 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 crypto rx=0x7f9b60006fa0 tx=0x7f9b540057f0 comp rx=0 tx=0).stop 2023-10-19T14:02:13.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.171+0000 7f9b71f20640 1 -- 172.21.15.57:0/2845218922 reap_dead start 2023-10-19T14:02:13.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.171+0000 7f9b73983640 1 -- 172.21.15.57:0/2845218922 shutdown_connections 2023-10-19T14:02:13.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.171+0000 7f9b73983640 1 -- 172.21.15.57:0/2845218922 >> 172.21.15.57:0/2845218922 conn(0x7f9b6c0d0d60 msgr2=0x7f9b6c0d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:02:13.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.171+0000 7f9b73983640 1 -- 172.21.15.57:0/2845218922 shutdown_connections 2023-10-19T14:02:13.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.171+0000 7f9b73983640 1 -- 172.21.15.57:0/2845218922 wait complete. 2023-10-19T14:02:13.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.172+0000 7f9b73983640 1 Processor -- start 2023-10-19T14:02:13.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.172+0000 7f9b73983640 1 -- start start 2023-10-19T14:02:13.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.173+0000 7f9b73983640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9b54003f90 0x7f9b6c0b3950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:13.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.173+0000 7f9b73983640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b6c15d6b0 0x7f9b6c1525a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:13.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.173+0000 7f9b73983640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9b6c06a770 0x7f9b6c06abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:13.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.173+0000 7f9b73983640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b6c06b130 con 0x7f9b6c15d6b0 2023-10-19T14:02:13.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.173+0000 7f9b73983640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b6c0b41a0 con 0x7f9b54003f90 2023-10-19T14:02:13.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.173+0000 7f9b73983640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b6c0b4480 con 0x7f9b6c06a770 2023-10-19T14:02:13.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.173+0000 7f9b70f1e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b6c15d6b0 0x7f9b6c1525a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:13.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.173+0000 7f9b70f1e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b6c15d6b0 0x7f9b6c1525a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59748/0 (socket says 172.21.15.57:59748) 2023-10-19T14:02:13.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.173+0000 7f9b70f1e640 1 -- 172.21.15.57:0/4212410669 learned_addr learned my addr 172.21.15.57:0/4212410669 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:02:13.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.173+0000 7f9b70f1e640 1 -- 172.21.15.57:0/4212410669 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9b6c06a770 msgr2=0x7f9b6c06abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:13.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.173+0000 7f9b70f1e640 1 --2- 172.21.15.57:0/4212410669 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9b6c06a770 0x7f9b6c06abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:13.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.173+0000 7f9b70f1e640 1 -- 172.21.15.57:0/4212410669 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9b54003f90 msgr2=0x7f9b6c0b3950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:13.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.173+0000 7f9b7171f640 1 --2- 172.21.15.57:0/4212410669 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9b54003f90 0x7f9b6c0b3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:13.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.173+0000 7f9b70f1e640 1 --2- 172.21.15.57:0/4212410669 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9b54003f90 0x7f9b6c0b3950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:13.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.173+0000 7f9b70f1e640 1 -- 172.21.15.57:0/4212410669 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b6c0b4760 con 0x7f9b6c15d6b0 2023-10-19T14:02:13.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.173+0000 7f9b7171f640 1 --2- 172.21.15.57:0/4212410669 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9b54003f90 0x7f9b6c0b3950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:02:13.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.173+0000 7f9b70f1e640 1 --2- 172.21.15.57:0/4212410669 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b6c15d6b0 0x7f9b6c1525a0 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 crypto rx=0x7f9b6000c320 tx=0x7f9b60001010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:02:13.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.173+0000 7f9b69ffb640 1 -- 172.21.15.57:0/4212410669 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b60001670 con 0x7f9b6c15d6b0 2023-10-19T14:02:13.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.173+0000 7f9b69ffb640 1 -- 172.21.15.57:0/4212410669 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f9b60006460 con 0x7f9b6c15d6b0 2023-10-19T14:02:13.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.173+0000 7f9b73983640 1 -- 172.21.15.57:0/4212410669 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b6c0b4a40 con 0x7f9b6c15d6b0 2023-10-19T14:02:13.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.173+0000 7f9b69ffb640 1 -- 172.21.15.57:0/4212410669 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b60003850 con 0x7f9b6c15d6b0 2023-10-19T14:02:13.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.174+0000 7f9b73983640 1 -- 172.21.15.57:0/4212410669 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b6c152ae0 con 0x7f9b6c15d6b0 2023-10-19T14:02:13.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.175+0000 7f9b73983640 1 -- 172.21.15.57:0/4212410669 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b6c0690e0 con 0x7f9b6c15d6b0 2023-10-19T14:02:13.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.175+0000 7f9b69ffb640 1 -- 172.21.15.57:0/4212410669 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 99) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f9b60031030 con 0x7f9b6c15d6b0 2023-10-19T14:02:13.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.175+0000 7f9b69ffb640 1 --2- 172.21.15.57:0/4212410669 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f9b5806af60 0x7f9b5806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:13.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.176+0000 7f9b69ffb640 1 -- 172.21.15.57:0/4212410669 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f9b600ab460 con 0x7f9b6c15d6b0 2023-10-19T14:02:13.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.176+0000 7f9b7171f640 1 --2- 172.21.15.57:0/4212410669 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f9b5806af60 0x7f9b5806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:13.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.176+0000 7f9b7171f640 1 --2- 172.21.15.57:0/4212410669 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f9b5806af60 0x7f9b5806d420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9b6400bb50 tx=0x7f9b6400b2e0 comp rx=0 tx=0).ready entity=mgr.8315 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:02:13.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.180+0000 7f9b69ffb640 1 -- 172.21.15.57:0/4212410669 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9b60079c20 con 0x7f9b6c15d6b0 2023-10-19T14:02:13.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.359+0000 7f9b73983640 1 -- 172.21.15.57:0/4212410669 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9b6c0d0b40 con 0x7f9b6c15d6b0 2023-10-19T14:02:13.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.360+0000 7f9b69ffb640 1 -- 172.21.15.57:0/4212410669 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v188) v1 ==== 74+0+16771 (secure 0 0 0) 0x7f9b600065f0 con 0x7f9b6c15d6b0 2023-10-19T14:02:13.363 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:02:13.363 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":188,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T14:02:11.669605+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2023-10-19T14:02:08.318466+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"187","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2023-10-19T14:02:08.995986+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"188","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T14:02:13.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.362+0000 7f9b73983640 1 -- 172.21.15.57:0/4212410669 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f9b5806af60 msgr2=0x7f9b5806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:13.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.362+0000 7f9b73983640 1 --2- 172.21.15.57:0/4212410669 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f9b5806af60 0x7f9b5806d420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9b6400bb50 tx=0x7f9b6400b2e0 comp rx=0 tx=0).stop 2023-10-19T14:02:13.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.362+0000 7f9b73983640 1 -- 172.21.15.57:0/4212410669 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b6c15d6b0 msgr2=0x7f9b6c1525a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:13.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.362+0000 7f9b73983640 1 --2- 172.21.15.57:0/4212410669 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b6c15d6b0 0x7f9b6c1525a0 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 crypto rx=0x7f9b6000c320 tx=0x7f9b60001010 comp rx=0 tx=0).stop 2023-10-19T14:02:13.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.362+0000 7f9b73983640 1 -- 172.21.15.57:0/4212410669 shutdown_connections 2023-10-19T14:02:13.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.362+0000 7f9b73983640 1 --2- 172.21.15.57:0/4212410669 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9b6c06a770 0x7f9b6c06abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:13.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.362+0000 7f9b73983640 1 --2- 172.21.15.57:0/4212410669 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f9b5806af60 0x7f9b5806d420 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:13.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.362+0000 7f9b73983640 1 --2- 172.21.15.57:0/4212410669 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b6c15d6b0 0x7f9b6c1525a0 unknown :-1 s=CLOSED pgs=2043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:13.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.362+0000 7f9b73983640 1 --2- 172.21.15.57:0/4212410669 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9b54003f90 0x7f9b6c0b3950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:13.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.362+0000 7f9b73983640 1 -- 172.21.15.57:0/4212410669 >> 172.21.15.57:0/4212410669 conn(0x7f9b6c0d0d60 msgr2=0x7f9b6c06b570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:02:13.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.362+0000 7f9b73983640 1 -- 172.21.15.57:0/4212410669 shutdown_connections 2023-10-19T14:02:13.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.362+0000 7f9b73983640 1 -- 172.21.15.57:0/4212410669 wait complete. 2023-10-19T14:02:13.376 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-19T14:02:13.376 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-19T14:02:13.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.482+0000 7f436a1e2640 1 -- 172.21.15.57:0/3191806236 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4364070a60 msgr2=0x7f436406bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:13.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.483+0000 7f436a1e2640 1 --2- 172.21.15.57:0/3191806236 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4364070a60 0x7f436406bd90 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 crypto rx=0x7f4364071940 tx=0x7f435c02c1d0 comp rx=0 tx=0).stop 2023-10-19T14:02:13.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.483+0000 7f436a1e2640 1 -- 172.21.15.57:0/3191806236 shutdown_connections 2023-10-19T14:02:13.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.483+0000 7f436a1e2640 1 --2- 172.21.15.57:0/3191806236 >> v2:172.21.15.179:3300/0 conn(0x7f43640cb2f0 0x7f4364070520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:13.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.483+0000 7f436a1e2640 1 --2- 172.21.15.57:0/3191806236 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4364070a60 0x7f436406bd90 unknown :-1 s=CLOSED pgs=2045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:13.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.483+0000 7f436a1e2640 1 -- 172.21.15.57:0/3191806236 >> 172.21.15.57:0/3191806236 conn(0x7f43640aec40 msgr2=0x7f43640af050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:02:13.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.483+0000 7f436a1e2640 1 -- 172.21.15.57:0/3191806236 shutdown_connections 2023-10-19T14:02:13.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.483+0000 7f436a1e2640 1 -- 172.21.15.57:0/3191806236 wait complete. 2023-10-19T14:02:13.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.484+0000 7f436a1e2640 1 Processor -- start 2023-10-19T14:02:13.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.484+0000 7f436a1e2640 1 -- start start 2023-10-19T14:02:13.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.485+0000 7f436a1e2640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4364070a60 0x7f43640ab2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:13.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.485+0000 7f436a1e2640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f43640cb2f0 0x7f43641437e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:13.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.485+0000 7f436a1e2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4364157c00 0x7f43641562f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:13.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.485+0000 7f436a1e2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f436414eed0 con 0x7f4364157c00 2023-10-19T14:02:13.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.485+0000 7f436a1e2640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43640ab890 con 0x7f43640cb2f0 2023-10-19T14:02:13.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.485+0000 7f436a1e2640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43640abb70 con 0x7f4364070a60 2023-10-19T14:02:13.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.485+0000 7f4363fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4364157c00 0x7f43641562f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:13.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.485+0000 7f4363fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4364157c00 0x7f43641562f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59764/0 (socket says 172.21.15.57:59764) 2023-10-19T14:02:13.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.485+0000 7f4363fff640 1 -- 172.21.15.57:0/3969010661 learned_addr learned my addr 172.21.15.57:0/3969010661 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:02:13.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.485+0000 7f4362ffd640 1 --2- 172.21.15.57:0/3969010661 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f43640cb2f0 0x7f43641437e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:13.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.485+0000 7f43637fe640 1 --2- 172.21.15.57:0/3969010661 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4364070a60 0x7f43640ab2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:13.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.485+0000 7f4363fff640 1 -- 172.21.15.57:0/3969010661 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4364070a60 msgr2=0x7f43640ab2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:13.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.485+0000 7f4363fff640 1 --2- 172.21.15.57:0/3969010661 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4364070a60 0x7f43640ab2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:13.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.485+0000 7f4363fff640 1 -- 172.21.15.57:0/3969010661 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f43640cb2f0 msgr2=0x7f43641437e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:13.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.485+0000 7f4363fff640 1 --2- 172.21.15.57:0/3969010661 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f43640cb2f0 0x7f43641437e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:13.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.485+0000 7f4363fff640 1 -- 172.21.15.57:0/3969010661 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f435c009900 con 0x7f4364157c00 2023-10-19T14:02:13.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.485+0000 7f43637fe640 1 --2- 172.21.15.57:0/3969010661 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4364070a60 0x7f43640ab2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:02:13.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.485+0000 7f4363fff640 1 --2- 172.21.15.57:0/3969010661 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4364157c00 0x7f43641562f0 secure :-1 s=READY pgs=2046 cs=0 l=1 rev1=1 crypto rx=0x7f435c009d60 tx=0x7f435c003c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:02:13.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.485+0000 7f434bfff640 1 -- 172.21.15.57:0/3969010661 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f435c004240 con 0x7f4364157c00 2023-10-19T14:02:13.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.485+0000 7f434bfff640 1 -- 172.21.15.57:0/3969010661 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f435c004490 con 0x7f4364157c00 2023-10-19T14:02:13.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.485+0000 7f436a1e2640 1 -- 172.21.15.57:0/3969010661 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43640abe50 con 0x7f4364157c00 2023-10-19T14:02:13.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.485+0000 7f434bfff640 1 -- 172.21.15.57:0/3969010661 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f435c0048a0 con 0x7f4364157c00 2023-10-19T14:02:13.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.486+0000 7f436a1e2640 1 -- 172.21.15.57:0/3969010661 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4364156830 con 0x7f4364157c00 2023-10-19T14:02:13.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.486+0000 7f436a1e2640 1 -- 172.21.15.57:0/3969010661 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f436414f040 con 0x7f4364157c00 2023-10-19T14:02:13.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.487+0000 7f434bfff640 1 -- 172.21.15.57:0/3969010661 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 99) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f435c00a040 con 0x7f4364157c00 2023-10-19T14:02:13.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.487+0000 7f434bfff640 1 --2- 172.21.15.57:0/3969010661 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f435006af60 0x7f435006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:13.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.487+0000 7f434bfff640 1 -- 172.21.15.57:0/3969010661 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f435c0acaa0 con 0x7f4364157c00 2023-10-19T14:02:13.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.488+0000 7f43637fe640 1 --2- 172.21.15.57:0/3969010661 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f435006af60 0x7f435006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:13.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.488+0000 7f43637fe640 1 --2- 172.21.15.57:0/3969010661 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f435006af60 0x7f435006d420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f436406d190 tx=0x7f4354009040 comp rx=0 tx=0).ready entity=mgr.8315 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:02:13.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.491+0000 7f434bfff640 1 -- 172.21.15.57:0/3969010661 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f435c077a90 con 0x7f4364157c00 2023-10-19T14:02:13.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.667+0000 7f436a1e2640 1 -- 172.21.15.57:0/3969010661 --> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f43640abce0 con 0x7f435006af60 2023-10-19T14:02:13.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.675+0000 7f434bfff640 1 -- 172.21.15.57:0/3969010661 <== mgr.8315 v2:172.21.15.179:6832/1426553632 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286841 (secure 0 0 0) 0x7f43640abce0 con 0x7f435006af60 2023-10-19T14:02:13.681 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:02:13.682 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2023-10-19T14:02:13.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.679+0000 7f436a1e2640 1 -- 172.21.15.57:0/3969010661 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f435006af60 msgr2=0x7f435006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:13.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.679+0000 7f436a1e2640 1 --2- 172.21.15.57:0/3969010661 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f435006af60 0x7f435006d420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f436406d190 tx=0x7f4354009040 comp rx=0 tx=0).stop 2023-10-19T14:02:13.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.679+0000 7f436a1e2640 1 -- 172.21.15.57:0/3969010661 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4364157c00 msgr2=0x7f43641562f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:13.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.679+0000 7f436a1e2640 1 --2- 172.21.15.57:0/3969010661 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4364157c00 0x7f43641562f0 secure :-1 s=READY pgs=2046 cs=0 l=1 rev1=1 crypto rx=0x7f435c009d60 tx=0x7f435c003c00 comp rx=0 tx=0).stop 2023-10-19T14:02:13.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.679+0000 7f436a1e2640 1 -- 172.21.15.57:0/3969010661 shutdown_connections 2023-10-19T14:02:13.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.679+0000 7f436a1e2640 1 --2- 172.21.15.57:0/3969010661 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f435006af60 0x7f435006d420 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:13.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.679+0000 7f436a1e2640 1 --2- 172.21.15.57:0/3969010661 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4364157c00 0x7f43641562f0 unknown :-1 s=CLOSED pgs=2046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:13.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.679+0000 7f436a1e2640 1 --2- 172.21.15.57:0/3969010661 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f43640cb2f0 0x7f43641437e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:13.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.679+0000 7f436a1e2640 1 --2- 172.21.15.57:0/3969010661 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4364070a60 0x7f43640ab2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:13.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.679+0000 7f436a1e2640 1 -- 172.21.15.57:0/3969010661 >> 172.21.15.57:0/3969010661 conn(0x7f43640aec40 msgr2=0x7f436414dcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:02:13.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.679+0000 7f436a1e2640 1 -- 172.21.15.57:0/3969010661 shutdown_connections 2023-10-19T14:02:13.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.679+0000 7f436a1e2640 1 -- 172.21.15.57:0/3969010661 wait complete. 2023-10-19T14:02:13.692 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2023-10-19T14:02:11.999492+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1730,"num_read_kb":3594,"num_write":525,"num_write_kb":2422,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3160,"ondisk_log_size":3160,"up":158,"acting":158,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":74,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":390936,"kb_used_data":19348,"kb_used_omap":12,"kb_used_meta":371443,"kb_avail":754583784,"statfs":{"total":773094113280,"available":772693794816,"internally_reserved":0,"allocated":19812352,"data_stored":10611779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":380358201},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000865"},"pg_stats":[{"pgid":"24.3f","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.685267+0000","last_change":"2023-10-19T14:02:10.680090+0000","last_active":"2023-10-19T14:02:10.685267+0000","last_peered":"2023-10-19T14:02:10.685267+0000","last_clean":"2023-10-19T14:02:10.685267+0000","last_became_active":"2023-10-19T14:02:10.679811+0000","last_became_peered":"2023-10-19T14:02:10.679811+0000","last_unstale":"2023-10-19T14:02:10.685267+0000","last_undegraded":"2023-10-19T14:02:10.685267+0000","last_fullsized":"2023-10-19T14:02:10.685267+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:05:33.395729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.3e","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.677900+0000","last_change":"2023-10-19T14:02:10.676966+0000","last_active":"2023-10-19T14:02:10.677900+0000","last_peered":"2023-10-19T14:02:10.677900+0000","last_clean":"2023-10-19T14:02:10.677900+0000","last_became_active":"2023-10-19T14:02:10.676689+0000","last_became_peered":"2023-10-19T14:02:10.676689+0000","last_unstale":"2023-10-19T14:02:10.677900+0000","last_undegraded":"2023-10-19T14:02:10.677900+0000","last_fullsized":"2023-10-19T14:02:10.677900+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:32:15.510487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3d","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.685628+0000","last_change":"2023-10-19T14:02:10.683123+0000","last_active":"2023-10-19T14:02:10.685628+0000","last_peered":"2023-10-19T14:02:10.685628+0000","last_clean":"2023-10-19T14:02:10.685628+0000","last_became_active":"2023-10-19T14:02:10.682822+0000","last_became_peered":"2023-10-19T14:02:10.682822+0000","last_unstale":"2023-10-19T14:02:10.685628+0000","last_undegraded":"2023-10-19T14:02:10.685628+0000","last_fullsized":"2023-10-19T14:02:10.685628+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:15:35.564507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.3c","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.685634+0000","last_change":"2023-10-19T14:02:10.682016+0000","last_active":"2023-10-19T14:02:10.685634+0000","last_peered":"2023-10-19T14:02:10.685634+0000","last_clean":"2023-10-19T14:02:10.685634+0000","last_became_active":"2023-10-19T14:02:10.681752+0000","last_became_peered":"2023-10-19T14:02:10.681752+0000","last_unstale":"2023-10-19T14:02:10.685634+0000","last_undegraded":"2023-10-19T14:02:10.685634+0000","last_fullsized":"2023-10-19T14:02:10.685634+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:47:25.158279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:02:09.644002+0000","last_change":"2023-10-19T14:02:09.644002+0000","last_active":"2023-10-19T14:02:09.644002+0000","last_peered":"2023-10-19T14:02:09.644002+0000","last_clean":"2023-10-19T14:02:09.644002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:02:09.644002+0000","last_undegraded":"2023-10-19T14:02:09.644002+0000","last_fullsized":"2023-10-19T14:02:09.644002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.3a","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.688713+0000","last_change":"2023-10-19T14:02:10.687039+0000","last_active":"2023-10-19T14:02:10.688713+0000","last_peered":"2023-10-19T14:02:10.688713+0000","last_clean":"2023-10-19T14:02:10.688713+0000","last_became_active":"2023-10-19T14:02:10.686737+0000","last_became_peered":"2023-10-19T14:02:10.686737+0000","last_unstale":"2023-10-19T14:02:10.688713+0000","last_undegraded":"2023-10-19T14:02:10.688713+0000","last_fullsized":"2023-10-19T14:02:10.688713+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:30:55.208738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.39","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.683909+0000","last_change":"2023-10-19T14:02:10.681884+0000","last_active":"2023-10-19T14:02:10.683909+0000","last_peered":"2023-10-19T14:02:10.683909+0000","last_clean":"2023-10-19T14:02:10.683909+0000","last_became_active":"2023-10-19T14:02:10.681470+0000","last_became_peered":"2023-10-19T14:02:10.681470+0000","last_unstale":"2023-10-19T14:02:10.683909+0000","last_undegraded":"2023-10-19T14:02:10.683909+0000","last_fullsized":"2023-10-19T14:02:10.683909+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:20:59.785404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.38","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.690451+0000","last_change":"2023-10-19T14:02:10.688431+0000","last_active":"2023-10-19T14:02:10.690451+0000","last_peered":"2023-10-19T14:02:10.690451+0000","last_clean":"2023-10-19T14:02:10.690451+0000","last_became_active":"2023-10-19T14:02:10.687693+0000","last_became_peered":"2023-10-19T14:02:10.687693+0000","last_unstale":"2023-10-19T14:02:10.690451+0000","last_undegraded":"2023-10-19T14:02:10.690451+0000","last_fullsized":"2023-10-19T14:02:10.690451+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:33:35.846384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.37","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.682038+0000","last_change":"2023-10-19T14:02:10.679157+0000","last_active":"2023-10-19T14:02:10.682038+0000","last_peered":"2023-10-19T14:02:10.682038+0000","last_clean":"2023-10-19T14:02:10.682038+0000","last_became_active":"2023-10-19T14:02:10.678735+0000","last_became_peered":"2023-10-19T14:02:10.678735+0000","last_unstale":"2023-10-19T14:02:10.682038+0000","last_undegraded":"2023-10-19T14:02:10.682038+0000","last_fullsized":"2023-10-19T14:02:10.682038+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:12:18.741834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.36","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.690625+0000","last_change":"2023-10-19T14:02:10.688854+0000","last_active":"2023-10-19T14:02:10.690625+0000","last_peered":"2023-10-19T14:02:10.690625+0000","last_clean":"2023-10-19T14:02:10.690625+0000","last_became_active":"2023-10-19T14:02:10.688575+0000","last_became_peered":"2023-10-19T14:02:10.688575+0000","last_unstale":"2023-10-19T14:02:10.690625+0000","last_undegraded":"2023-10-19T14:02:10.690625+0000","last_fullsized":"2023-10-19T14:02:10.690625+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:04:55.679325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.35","version":"0'0","reported_seq":11,"reported_epoch":188,"state":"active+clean","last_fresh":"2023-10-19T14:02:11.676554+0000","last_change":"2023-10-19T14:02:10.669419+0000","last_active":"2023-10-19T14:02:11.676554+0000","last_peered":"2023-10-19T14:02:11.676554+0000","last_clean":"2023-10-19T14:02:11.676554+0000","last_became_active":"2023-10-19T14:02:10.668994+0000","last_became_peered":"2023-10-19T14:02:10.668994+0000","last_unstale":"2023-10-19T14:02:11.676554+0000","last_undegraded":"2023-10-19T14:02:11.676554+0000","last_fullsized":"2023-10-19T14:02:11.676554+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:32:43.725034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.34","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.688751+0000","last_change":"2023-10-19T14:02:10.687045+0000","last_active":"2023-10-19T14:02:10.688751+0000","last_peered":"2023-10-19T14:02:10.688751+0000","last_clean":"2023-10-19T14:02:10.688751+0000","last_became_active":"2023-10-19T14:02:10.686758+0000","last_became_peered":"2023-10-19T14:02:10.686758+0000","last_unstale":"2023-10-19T14:02:10.688751+0000","last_undegraded":"2023-10-19T14:02:10.688751+0000","last_fullsized":"2023-10-19T14:02:10.688751+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:47:44.902028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.33","version":"0'0","reported_seq":11,"reported_epoch":188,"state":"active+clean","last_fresh":"2023-10-19T14:02:11.676817+0000","last_change":"2023-10-19T14:02:10.682320+0000","last_active":"2023-10-19T14:02:11.676817+0000","last_peered":"2023-10-19T14:02:11.676817+0000","last_clean":"2023-10-19T14:02:11.676817+0000","last_became_active":"2023-10-19T14:02:10.679611+0000","last_became_peered":"2023-10-19T14:02:10.679611+0000","last_unstale":"2023-10-19T14:02:11.676817+0000","last_undegraded":"2023-10-19T14:02:11.676817+0000","last_fullsized":"2023-10-19T14:02:11.676817+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:26:53.697061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:02:09.644002+0000","last_change":"2023-10-19T14:02:09.644002+0000","last_active":"2023-10-19T14:02:09.644002+0000","last_peered":"2023-10-19T14:02:09.644002+0000","last_clean":"2023-10-19T14:02:09.644002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:02:09.644002+0000","last_undegraded":"2023-10-19T14:02:09.644002+0000","last_fullsized":"2023-10-19T14:02:09.644002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.31","version":"0'0","reported_seq":11,"reported_epoch":188,"state":"active+clean","last_fresh":"2023-10-19T14:02:11.676135+0000","last_change":"2023-10-19T14:02:10.684424+0000","last_active":"2023-10-19T14:02:11.676135+0000","last_peered":"2023-10-19T14:02:11.676135+0000","last_clean":"2023-10-19T14:02:11.676135+0000","last_became_active":"2023-10-19T14:02:10.684125+0000","last_became_peered":"2023-10-19T14:02:10.684125+0000","last_unstale":"2023-10-19T14:02:11.676135+0000","last_undegraded":"2023-10-19T14:02:11.676135+0000","last_fullsized":"2023-10-19T14:02:11.676135+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:22:58.263925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.30","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.697012+0000","last_change":"2023-10-19T14:02:10.694706+0000","last_active":"2023-10-19T14:02:10.697012+0000","last_peered":"2023-10-19T14:02:10.697012+0000","last_clean":"2023-10-19T14:02:10.697012+0000","last_became_active":"2023-10-19T14:02:10.694360+0000","last_became_peered":"2023-10-19T14:02:10.694360+0000","last_unstale":"2023-10-19T14:02:10.697012+0000","last_undegraded":"2023-10-19T14:02:10.697012+0000","last_fullsized":"2023-10-19T14:02:10.697012+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:44:19.924159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.2f","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.682022+0000","last_change":"2023-10-19T14:02:10.678976+0000","last_active":"2023-10-19T14:02:10.682022+0000","last_peered":"2023-10-19T14:02:10.682022+0000","last_clean":"2023-10-19T14:02:10.682022+0000","last_became_active":"2023-10-19T14:02:10.678447+0000","last_became_peered":"2023-10-19T14:02:10.678447+0000","last_unstale":"2023-10-19T14:02:10.682022+0000","last_undegraded":"2023-10-19T14:02:10.682022+0000","last_fullsized":"2023-10-19T14:02:10.682022+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:09:52.311831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:02:09.644002+0000","last_change":"2023-10-19T14:02:09.644002+0000","last_active":"2023-10-19T14:02:09.644002+0000","last_peered":"2023-10-19T14:02:09.644002+0000","last_clean":"2023-10-19T14:02:09.644002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:02:09.644002+0000","last_undegraded":"2023-10-19T14:02:09.644002+0000","last_fullsized":"2023-10-19T14:02:09.644002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.2d","version":"0'0","reported_seq":11,"reported_epoch":188,"state":"active+clean","last_fresh":"2023-10-19T14:02:11.676637+0000","last_change":"2023-10-19T14:02:10.685211+0000","last_active":"2023-10-19T14:02:11.676637+0000","last_peered":"2023-10-19T14:02:11.676637+0000","last_clean":"2023-10-19T14:02:11.676637+0000","last_became_active":"2023-10-19T14:02:10.684944+0000","last_became_peered":"2023-10-19T14:02:10.684944+0000","last_unstale":"2023-10-19T14:02:11.676637+0000","last_undegraded":"2023-10-19T14:02:11.676637+0000","last_fullsized":"2023-10-19T14:02:11.676637+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:22:30.762765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.2c","version":"0'0","reported_seq":11,"reported_epoch":188,"state":"active+clean","last_fresh":"2023-10-19T14:02:11.677539+0000","last_change":"2023-10-19T14:02:10.677499+0000","last_active":"2023-10-19T14:02:11.677539+0000","last_peered":"2023-10-19T14:02:11.677539+0000","last_clean":"2023-10-19T14:02:11.677539+0000","last_became_active":"2023-10-19T14:02:10.677245+0000","last_became_peered":"2023-10-19T14:02:10.677245+0000","last_unstale":"2023-10-19T14:02:11.677539+0000","last_undegraded":"2023-10-19T14:02:11.677539+0000","last_fullsized":"2023-10-19T14:02:11.677539+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:32:56.038053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.2b","version":"0'0","reported_seq":11,"reported_epoch":188,"state":"active+clean","last_fresh":"2023-10-19T14:02:11.676840+0000","last_change":"2023-10-19T14:02:10.686175+0000","last_active":"2023-10-19T14:02:11.676840+0000","last_peered":"2023-10-19T14:02:11.676840+0000","last_clean":"2023-10-19T14:02:11.676840+0000","last_became_active":"2023-10-19T14:02:10.685847+0000","last_became_peered":"2023-10-19T14:02:10.685847+0000","last_unstale":"2023-10-19T14:02:11.676840+0000","last_undegraded":"2023-10-19T14:02:11.676840+0000","last_fullsized":"2023-10-19T14:02:11.676840+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:01:51.421272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.2a","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.677928+0000","last_change":"2023-10-19T14:02:10.677165+0000","last_active":"2023-10-19T14:02:10.677928+0000","last_peered":"2023-10-19T14:02:10.677928+0000","last_clean":"2023-10-19T14:02:10.677928+0000","last_became_active":"2023-10-19T14:02:10.676908+0000","last_became_peered":"2023-10-19T14:02:10.676908+0000","last_unstale":"2023-10-19T14:02:10.677928+0000","last_undegraded":"2023-10-19T14:02:10.677928+0000","last_fullsized":"2023-10-19T14:02:10.677928+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:14:10.545049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:02:09.644002+0000","last_change":"2023-10-19T14:02:09.644002+0000","last_active":"2023-10-19T14:02:09.644002+0000","last_peered":"2023-10-19T14:02:09.644002+0000","last_clean":"2023-10-19T14:02:09.644002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:02:09.644002+0000","last_undegraded":"2023-10-19T14:02:09.644002+0000","last_fullsized":"2023-10-19T14:02:09.644002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.28","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.685597+0000","last_change":"2023-10-19T14:02:10.683726+0000","last_active":"2023-10-19T14:02:10.685597+0000","last_peered":"2023-10-19T14:02:10.685597+0000","last_clean":"2023-10-19T14:02:10.685597+0000","last_became_active":"2023-10-19T14:02:10.683452+0000","last_became_peered":"2023-10-19T14:02:10.683452+0000","last_unstale":"2023-10-19T14:02:10.685597+0000","last_undegraded":"2023-10-19T14:02:10.685597+0000","last_fullsized":"2023-10-19T14:02:10.685597+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:28:05.098300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.27","version":"0'0","reported_seq":11,"reported_epoch":188,"state":"active+clean","last_fresh":"2023-10-19T14:02:11.677064+0000","last_change":"2023-10-19T14:02:10.678793+0000","last_active":"2023-10-19T14:02:11.677064+0000","last_peered":"2023-10-19T14:02:11.677064+0000","last_clean":"2023-10-19T14:02:11.677064+0000","last_became_active":"2023-10-19T14:02:10.678473+0000","last_became_peered":"2023-10-19T14:02:10.678473+0000","last_unstale":"2023-10-19T14:02:11.677064+0000","last_undegraded":"2023-10-19T14:02:11.677064+0000","last_fullsized":"2023-10-19T14:02:11.677064+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:49:09.882058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.26","version":"0'0","reported_seq":11,"reported_epoch":188,"state":"active+clean","last_fresh":"2023-10-19T14:02:11.677709+0000","last_change":"2023-10-19T14:02:10.687721+0000","last_active":"2023-10-19T14:02:11.677709+0000","last_peered":"2023-10-19T14:02:11.677709+0000","last_clean":"2023-10-19T14:02:11.677709+0000","last_became_active":"2023-10-19T14:02:10.687270+0000","last_became_peered":"2023-10-19T14:02:10.687270+0000","last_unstale":"2023-10-19T14:02:11.677709+0000","last_undegraded":"2023-10-19T14:02:11.677709+0000","last_fullsized":"2023-10-19T14:02:11.677709+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:30:27.924431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.25","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.685487+0000","last_change":"2023-10-19T14:02:10.683270+0000","last_active":"2023-10-19T14:02:10.685487+0000","last_peered":"2023-10-19T14:02:10.685487+0000","last_clean":"2023-10-19T14:02:10.685487+0000","last_became_active":"2023-10-19T14:02:10.682994+0000","last_became_peered":"2023-10-19T14:02:10.682994+0000","last_unstale":"2023-10-19T14:02:10.685487+0000","last_undegraded":"2023-10-19T14:02:10.685487+0000","last_fullsized":"2023-10-19T14:02:10.685487+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:14:45.157582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.24","version":"0'0","reported_seq":11,"reported_epoch":188,"state":"active+clean","last_fresh":"2023-10-19T14:02:11.676818+0000","last_change":"2023-10-19T14:02:10.695300+0000","last_active":"2023-10-19T14:02:11.676818+0000","last_peered":"2023-10-19T14:02:11.676818+0000","last_clean":"2023-10-19T14:02:11.676818+0000","last_became_active":"2023-10-19T14:02:10.694960+0000","last_became_peered":"2023-10-19T14:02:10.694960+0000","last_unstale":"2023-10-19T14:02:11.676818+0000","last_undegraded":"2023-10-19T14:02:11.676818+0000","last_fullsized":"2023-10-19T14:02:11.676818+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:15:56.950509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.23","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.681808+0000","last_change":"2023-10-19T14:02:10.680553+0000","last_active":"2023-10-19T14:02:10.681808+0000","last_peered":"2023-10-19T14:02:10.681808+0000","last_clean":"2023-10-19T14:02:10.681808+0000","last_became_active":"2023-10-19T14:02:10.680292+0000","last_became_peered":"2023-10-19T14:02:10.680292+0000","last_unstale":"2023-10-19T14:02:10.681808+0000","last_undegraded":"2023-10-19T14:02:10.681808+0000","last_fullsized":"2023-10-19T14:02:10.681808+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:11:05.529745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:02:09.644002+0000","last_change":"2023-10-19T14:02:09.644002+0000","last_active":"2023-10-19T14:02:09.644002+0000","last_peered":"2023-10-19T14:02:09.644002+0000","last_clean":"2023-10-19T14:02:09.644002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:02:09.644002+0000","last_undegraded":"2023-10-19T14:02:09.644002+0000","last_fullsized":"2023-10-19T14:02:09.644002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.21","version":"0'0","reported_seq":11,"reported_epoch":188,"state":"active+clean","last_fresh":"2023-10-19T14:02:11.676226+0000","last_change":"2023-10-19T14:02:10.681523+0000","last_active":"2023-10-19T14:02:11.676226+0000","last_peered":"2023-10-19T14:02:11.676226+0000","last_clean":"2023-10-19T14:02:11.676226+0000","last_became_active":"2023-10-19T14:02:10.681243+0000","last_became_peered":"2023-10-19T14:02:10.681243+0000","last_unstale":"2023-10-19T14:02:11.676226+0000","last_undegraded":"2023-10-19T14:02:11.676226+0000","last_fullsized":"2023-10-19T14:02:11.676226+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:30:55.208738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:02:09.644002+0000","last_change":"2023-10-19T14:02:09.644002+0000","last_active":"2023-10-19T14:02:09.644002+0000","last_peered":"2023-10-19T14:02:09.644002+0000","last_clean":"2023-10-19T14:02:09.644002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:02:09.644002+0000","last_undegraded":"2023-10-19T14:02:09.644002+0000","last_fullsized":"2023-10-19T14:02:09.644002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.10","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.689546+0000","last_change":"2023-10-19T14:02:10.689114+0000","last_active":"2023-10-19T14:02:10.689546+0000","last_peered":"2023-10-19T14:02:10.689546+0000","last_clean":"2023-10-19T14:02:10.689546+0000","last_became_active":"2023-10-19T14:02:10.688259+0000","last_became_peered":"2023-10-19T14:02:10.688259+0000","last_unstale":"2023-10-19T14:02:10.689546+0000","last_undegraded":"2023-10-19T14:02:10.689546+0000","last_fullsized":"2023-10-19T14:02:10.689546+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:58:35.411824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1f","version":"187'4","reported_seq":15,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.696376+0000","last_change":"2023-10-19T14:02:09.668623+0000","last_active":"2023-10-19T14:02:10.696376+0000","last_peered":"2023-10-19T14:02:10.696376+0000","last_clean":"2023-10-19T14:02:10.696376+0000","last_became_active":"2023-10-19T14:02:09.668354+0000","last_became_peered":"2023-10-19T14:02:09.668354+0000","last_unstale":"2023-10-19T14:02:10.696376+0000","last_undegraded":"2023-10-19T14:02:10.696376+0000","last_fullsized":"2023-10-19T14:02:10.696376+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_clean_scrub_stamp":"2023-10-19T14:02:08.638576+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:57:35.643980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":11,"reported_epoch":188,"state":"active+clean","last_fresh":"2023-10-19T14:02:11.676570+0000","last_change":"2023-10-19T14:02:10.678806+0000","last_active":"2023-10-19T14:02:11.676570+0000","last_peered":"2023-10-19T14:02:11.676570+0000","last_clean":"2023-10-19T14:02:11.676570+0000","last_became_active":"2023-10-19T14:02:10.678408+0000","last_became_peered":"2023-10-19T14:02:10.678408+0000","last_unstale":"2023-10-19T14:02:11.676570+0000","last_undegraded":"2023-10-19T14:02:11.676570+0000","last_fullsized":"2023-10-19T14:02:11.676570+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:24:23.166190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:02:08.638576+0000","last_change":"2023-10-19T14:02:08.638576+0000","last_active":"2023-10-19T14:02:08.638576+0000","last_peered":"2023-10-19T14:02:08.638576+0000","last_clean":"2023-10-19T14:02:08.638576+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:02:08.638576+0000","last_undegraded":"2023-10-19T14:02:08.638576+0000","last_fullsized":"2023-10-19T14:02:08.638576+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_clean_scrub_stamp":"2023-10-19T14:02:08.638576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:02:09.644002+0000","last_change":"2023-10-19T14:02:09.644002+0000","last_active":"2023-10-19T14:02:09.644002+0000","last_peered":"2023-10-19T14:02:09.644002+0000","last_clean":"2023-10-19T14:02:09.644002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:02:09.644002+0000","last_undegraded":"2023-10-19T14:02:09.644002+0000","last_fullsized":"2023-10-19T14:02:09.644002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.1d","version":"187'2","reported_seq":14,"reported_epoch":188,"state":"active+clean","last_fresh":"2023-10-19T14:02:11.676647+0000","last_change":"2023-10-19T14:02:09.659670+0000","last_active":"2023-10-19T14:02:11.676647+0000","last_peered":"2023-10-19T14:02:11.676647+0000","last_clean":"2023-10-19T14:02:11.676647+0000","last_became_active":"2023-10-19T14:02:09.659397+0000","last_became_peered":"2023-10-19T14:02:09.659397+0000","last_unstale":"2023-10-19T14:02:11.676647+0000","last_undegraded":"2023-10-19T14:02:11.676647+0000","last_fullsized":"2023-10-19T14:02:11.676647+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_clean_scrub_stamp":"2023-10-19T14:02:08.638576+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:21:13.622150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.13","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.688379+0000","last_change":"2023-10-19T14:02:10.686240+0000","last_active":"2023-10-19T14:02:10.688379+0000","last_peered":"2023-10-19T14:02:10.688379+0000","last_clean":"2023-10-19T14:02:10.688379+0000","last_became_active":"2023-10-19T14:02:10.685908+0000","last_became_peered":"2023-10-19T14:02:10.685908+0000","last_unstale":"2023-10-19T14:02:10.688379+0000","last_undegraded":"2023-10-19T14:02:10.688379+0000","last_fullsized":"2023-10-19T14:02:10.688379+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:29:29.475164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1c","version":"187'2","reported_seq":14,"reported_epoch":188,"state":"active+clean","last_fresh":"2023-10-19T14:02:11.677513+0000","last_change":"2023-10-19T14:02:09.663273+0000","last_active":"2023-10-19T14:02:11.677513+0000","last_peered":"2023-10-19T14:02:11.677513+0000","last_clean":"2023-10-19T14:02:11.677513+0000","last_became_active":"2023-10-19T14:02:09.662981+0000","last_became_peered":"2023-10-19T14:02:09.662981+0000","last_unstale":"2023-10-19T14:02:11.677513+0000","last_undegraded":"2023-10-19T14:02:11.677513+0000","last_fullsized":"2023-10-19T14:02:11.677513+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_clean_scrub_stamp":"2023-10-19T14:02:08.638576+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:43:29.810615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.14","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.692347+0000","last_change":"2023-10-19T14:02:10.688031+0000","last_active":"2023-10-19T14:02:10.692347+0000","last_peered":"2023-10-19T14:02:10.692347+0000","last_clean":"2023-10-19T14:02:10.692347+0000","last_became_active":"2023-10-19T14:02:10.687232+0000","last_became_peered":"2023-10-19T14:02:10.687232+0000","last_unstale":"2023-10-19T14:02:10.692347+0000","last_undegraded":"2023-10-19T14:02:10.692347+0000","last_fullsized":"2023-10-19T14:02:10.692347+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:31:13.295905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1b","version":"187'1","reported_seq":12,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.692939+0000","last_change":"2023-10-19T14:02:09.669838+0000","last_active":"2023-10-19T14:02:10.692939+0000","last_peered":"2023-10-19T14:02:10.692939+0000","last_clean":"2023-10-19T14:02:10.692939+0000","last_became_active":"2023-10-19T14:02:09.669582+0000","last_became_peered":"2023-10-19T14:02:09.669582+0000","last_unstale":"2023-10-19T14:02:10.692939+0000","last_undegraded":"2023-10-19T14:02:10.692939+0000","last_fullsized":"2023-10-19T14:02:10.692939+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_clean_scrub_stamp":"2023-10-19T14:02:08.638576+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:53:55.532213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.15","version":"0'0","reported_seq":11,"reported_epoch":188,"state":"active+clean","last_fresh":"2023-10-19T14:02:11.677677+0000","last_change":"2023-10-19T14:02:10.678950+0000","last_active":"2023-10-19T14:02:11.677677+0000","last_peered":"2023-10-19T14:02:11.677677+0000","last_clean":"2023-10-19T14:02:11.677677+0000","last_became_active":"2023-10-19T14:02:10.678670+0000","last_became_peered":"2023-10-19T14:02:10.678670+0000","last_unstale":"2023-10-19T14:02:11.677677+0000","last_undegraded":"2023-10-19T14:02:11.677677+0000","last_fullsized":"2023-10-19T14:02:11.677677+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:38:14.577922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:02:08.638576+0000","last_change":"2023-10-19T14:02:08.638576+0000","last_active":"2023-10-19T14:02:08.638576+0000","last_peered":"2023-10-19T14:02:08.638576+0000","last_clean":"2023-10-19T14:02:08.638576+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:02:08.638576+0000","last_undegraded":"2023-10-19T14:02:08.638576+0000","last_fullsized":"2023-10-19T14:02:08.638576+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_clean_scrub_stamp":"2023-10-19T14:02:08.638576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":11,"reported_epoch":188,"state":"active+clean","last_fresh":"2023-10-19T14:02:11.677728+0000","last_change":"2023-10-19T14:02:10.682862+0000","last_active":"2023-10-19T14:02:11.677728+0000","last_peered":"2023-10-19T14:02:11.677728+0000","last_clean":"2023-10-19T14:02:11.677728+0000","last_became_active":"2023-10-19T14:02:10.682558+0000","last_became_peered":"2023-10-19T14:02:10.682558+0000","last_unstale":"2023-10-19T14:02:11.677728+0000","last_undegraded":"2023-10-19T14:02:11.677728+0000","last_fullsized":"2023-10-19T14:02:11.677728+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:51:17.752537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2023-10-19T14:02:11.677355+0000","last_change":"2023-10-19T14:02:09.661271+0000","last_active":"2023-10-19T14:02:11.677355+0000","last_peered":"2023-10-19T14:02:11.677355+0000","last_clean":"2023-10-19T14:02:11.677355+0000","last_became_active":"2023-10-19T14:02:09.661012+0000","last_became_peered":"2023-10-19T14:02:09.661012+0000","last_unstale":"2023-10-19T14:02:11.677355+0000","last_undegraded":"2023-10-19T14:02:11.677355+0000","last_fullsized":"2023-10-19T14:02:11.677355+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_clean_scrub_stamp":"2023-10-19T14:02:08.638576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:38:46.272924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.685431+0000","last_change":"2023-10-19T14:02:10.681882+0000","last_active":"2023-10-19T14:02:10.685431+0000","last_peered":"2023-10-19T14:02:10.685431+0000","last_clean":"2023-10-19T14:02:10.685431+0000","last_became_active":"2023-10-19T14:02:10.681603+0000","last_became_peered":"2023-10-19T14:02:10.681603+0000","last_unstale":"2023-10-19T14:02:10.685431+0000","last_undegraded":"2023-10-19T14:02:10.685431+0000","last_fullsized":"2023-10-19T14:02:10.685431+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:40:09.919763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.672552+0000","last_change":"2023-10-19T14:02:09.663748+0000","last_active":"2023-10-19T14:02:10.672552+0000","last_peered":"2023-10-19T14:02:10.672552+0000","last_clean":"2023-10-19T14:02:10.672552+0000","last_became_active":"2023-10-19T14:02:09.663456+0000","last_became_peered":"2023-10-19T14:02:09.663456+0000","last_unstale":"2023-10-19T14:02:10.672552+0000","last_undegraded":"2023-10-19T14:02:10.672552+0000","last_fullsized":"2023-10-19T14:02:10.672552+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_clean_scrub_stamp":"2023-10-19T14:02:08.638576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:34:28.435490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.688744+0000","last_change":"2023-10-19T14:02:10.688225+0000","last_active":"2023-10-19T14:02:10.688744+0000","last_peered":"2023-10-19T14:02:10.688744+0000","last_clean":"2023-10-19T14:02:10.688744+0000","last_became_active":"2023-10-19T14:02:10.687857+0000","last_became_peered":"2023-10-19T14:02:10.687857+0000","last_unstale":"2023-10-19T14:02:10.688744+0000","last_undegraded":"2023-10-19T14:02:10.688744+0000","last_fullsized":"2023-10-19T14:02:10.688744+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:33:00.686451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2023-10-19T14:02:11.677005+0000","last_change":"2023-10-19T14:02:09.670452+0000","last_active":"2023-10-19T14:02:11.677005+0000","last_peered":"2023-10-19T14:02:11.677005+0000","last_clean":"2023-10-19T14:02:11.677005+0000","last_became_active":"2023-10-19T14:02:09.670133+0000","last_became_peered":"2023-10-19T14:02:09.670133+0000","last_unstale":"2023-10-19T14:02:11.677005+0000","last_undegraded":"2023-10-19T14:02:11.677005+0000","last_fullsized":"2023-10-19T14:02:11.677005+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_clean_scrub_stamp":"2023-10-19T14:02:08.638576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:57:58.962523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1a","version":"0'0","reported_seq":11,"reported_epoch":188,"state":"active+clean","last_fresh":"2023-10-19T14:02:11.676681+0000","last_change":"2023-10-19T14:02:10.677844+0000","last_active":"2023-10-19T14:02:11.676681+0000","last_peered":"2023-10-19T14:02:11.676681+0000","last_clean":"2023-10-19T14:02:11.676681+0000","last_became_active":"2023-10-19T14:02:10.677559+0000","last_became_peered":"2023-10-19T14:02:10.677559+0000","last_unstale":"2023-10-19T14:02:11.676681+0000","last_undegraded":"2023-10-19T14:02:11.676681+0000","last_fullsized":"2023-10-19T14:02:11.676681+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:12:49.139706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.15","version":"187'3","reported_seq":14,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.691713+0000","last_change":"2023-10-19T14:02:09.662732+0000","last_active":"2023-10-19T14:02:10.691713+0000","last_peered":"2023-10-19T14:02:10.691713+0000","last_clean":"2023-10-19T14:02:10.691713+0000","last_became_active":"2023-10-19T14:02:09.662431+0000","last_became_peered":"2023-10-19T14:02:09.662431+0000","last_unstale":"2023-10-19T14:02:10.691713+0000","last_undegraded":"2023-10-19T14:02:10.691713+0000","last_fullsized":"2023-10-19T14:02:10.691713+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_clean_scrub_stamp":"2023-10-19T14:02:08.638576+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:14:46.558909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1b","version":"0'0","reported_seq":11,"reported_epoch":188,"state":"active+clean","last_fresh":"2023-10-19T14:02:11.677539+0000","last_change":"2023-10-19T14:02:10.680074+0000","last_active":"2023-10-19T14:02:11.677539+0000","last_peered":"2023-10-19T14:02:11.677539+0000","last_clean":"2023-10-19T14:02:11.677539+0000","last_became_active":"2023-10-19T14:02:10.679791+0000","last_became_peered":"2023-10-19T14:02:10.679791+0000","last_unstale":"2023-10-19T14:02:11.677539+0000","last_undegraded":"2023-10-19T14:02:11.677539+0000","last_fullsized":"2023-10-19T14:02:11.677539+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:13:42.324779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.14","version":"187'6","reported_seq":18,"reported_epoch":188,"state":"active+clean","last_fresh":"2023-10-19T14:02:11.676960+0000","last_change":"2023-10-19T14:02:09.670227+0000","last_active":"2023-10-19T14:02:11.676960+0000","last_peered":"2023-10-19T14:02:11.676960+0000","last_clean":"2023-10-19T14:02:11.676960+0000","last_became_active":"2023-10-19T14:02:09.669940+0000","last_became_peered":"2023-10-19T14:02:09.669940+0000","last_unstale":"2023-10-19T14:02:11.676960+0000","last_undegraded":"2023-10-19T14:02:11.676960+0000","last_fullsized":"2023-10-19T14:02:11.676960+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_clean_scrub_stamp":"2023-10-19T14:02:08.638576+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:50:48.588866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:02:09.644002+0000","last_change":"2023-10-19T14:02:09.644002+0000","last_active":"2023-10-19T14:02:09.644002+0000","last_peered":"2023-10-19T14:02:09.644002+0000","last_clean":"2023-10-19T14:02:09.644002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:02:09.644002+0000","last_undegraded":"2023-10-19T14:02:09.644002+0000","last_fullsized":"2023-10-19T14:02:09.644002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:02:08.638576+0000","last_change":"2023-10-19T14:02:08.638576+0000","last_active":"2023-10-19T14:02:08.638576+0000","last_peered":"2023-10-19T14:02:08.638576+0000","last_clean":"2023-10-19T14:02:08.638576+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:02:08.638576+0000","last_undegraded":"2023-10-19T14:02:08.638576+0000","last_fullsized":"2023-10-19T14:02:08.638576+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_clean_scrub_stamp":"2023-10-19T14:02:08.638576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.1d","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.690591+0000","last_change":"2023-10-19T14:02:10.686088+0000","last_active":"2023-10-19T14:02:10.690591+0000","last_peered":"2023-10-19T14:02:10.690591+0000","last_clean":"2023-10-19T14:02:10.690591+0000","last_became_active":"2023-10-19T14:02:10.684607+0000","last_became_peered":"2023-10-19T14:02:10.684607+0000","last_unstale":"2023-10-19T14:02:10.690591+0000","last_undegraded":"2023-10-19T14:02:10.690591+0000","last_fullsized":"2023-10-19T14:02:10.690591+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:47:08.467008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2023-10-19T14:02:11.676769+0000","last_change":"2023-10-19T14:02:09.664080+0000","last_active":"2023-10-19T14:02:11.676769+0000","last_peered":"2023-10-19T14:02:11.676769+0000","last_clean":"2023-10-19T14:02:11.676769+0000","last_became_active":"2023-10-19T14:02:09.663801+0000","last_became_peered":"2023-10-19T14:02:09.663801+0000","last_unstale":"2023-10-19T14:02:11.676769+0000","last_undegraded":"2023-10-19T14:02:11.676769+0000","last_fullsized":"2023-10-19T14:02:11.676769+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_clean_scrub_stamp":"2023-10-19T14:02:08.638576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:56:20.643608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1e","version":"0'0","reported_seq":11,"reported_epoch":188,"state":"active+clean","last_fresh":"2023-10-19T14:02:11.676569+0000","last_change":"2023-10-19T14:02:10.674303+0000","last_active":"2023-10-19T14:02:11.676569+0000","last_peered":"2023-10-19T14:02:11.676569+0000","last_clean":"2023-10-19T14:02:11.676569+0000","last_became_active":"2023-10-19T14:02:10.674038+0000","last_became_peered":"2023-10-19T14:02:10.674038+0000","last_unstale":"2023-10-19T14:02:11.676569+0000","last_undegraded":"2023-10-19T14:02:11.676569+0000","last_fullsized":"2023-10-19T14:02:11.676569+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:55:15.591450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2023-10-19T14:02:11.676228+0000","last_change":"2023-10-19T14:02:09.660406+0000","last_active":"2023-10-19T14:02:11.676228+0000","last_peered":"2023-10-19T14:02:11.676228+0000","last_clean":"2023-10-19T14:02:11.676228+0000","last_became_active":"2023-10-19T14:02:09.660096+0000","last_became_peered":"2023-10-19T14:02:09.660096+0000","last_unstale":"2023-10-19T14:02:11.676228+0000","last_undegraded":"2023-10-19T14:02:11.676228+0000","last_fullsized":"2023-10-19T14:02:11.676228+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_clean_scrub_stamp":"2023-10-19T14:02:08.638576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:32:59.681025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.1f","version":"0'0","reported_seq":11,"reported_epoch":188,"state":"active+clean","last_fresh":"2023-10-19T14:02:11.677770+0000","last_change":"2023-10-19T14:02:10.689906+0000","last_active":"2023-10-19T14:02:11.677770+0000","last_peered":"2023-10-19T14:02:11.677770+0000","last_clean":"2023-10-19T14:02:11.677770+0000","last_became_active":"2023-10-19T14:02:10.689462+0000","last_became_peered":"2023-10-19T14:02:10.689462+0000","last_unstale":"2023-10-19T14:02:11.677770+0000","last_undegraded":"2023-10-19T14:02:11.677770+0000","last_fullsized":"2023-10-19T14:02:11.677770+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:41:10.616800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.10","version":"187'8","reported_seq":19,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.695050+0000","last_change":"2023-10-19T14:02:09.663071+0000","last_active":"2023-10-19T14:02:10.695050+0000","last_peered":"2023-10-19T14:02:10.695050+0000","last_clean":"2023-10-19T14:02:10.695050+0000","last_became_active":"2023-10-19T14:02:09.662349+0000","last_became_peered":"2023-10-19T14:02:09.662349+0000","last_unstale":"2023-10-19T14:02:10.695050+0000","last_undegraded":"2023-10-19T14:02:10.695050+0000","last_fullsized":"2023-10-19T14:02:10.695050+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_clean_scrub_stamp":"2023-10-19T14:02:08.638576+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:03:52.274945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.680089+0000","last_change":"2023-10-19T14:02:10.677622+0000","last_active":"2023-10-19T14:02:10.680089+0000","last_peered":"2023-10-19T14:02:10.680089+0000","last_clean":"2023-10-19T14:02:10.680089+0000","last_became_active":"2023-10-19T14:02:10.677267+0000","last_became_peered":"2023-10-19T14:02:10.677267+0000","last_unstale":"2023-10-19T14:02:10.680089+0000","last_undegraded":"2023-10-19T14:02:10.680089+0000","last_fullsized":"2023-10-19T14:02:10.680089+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:38:12.275859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.f","version":"187'1","reported_seq":13,"reported_epoch":188,"state":"active+clean","last_fresh":"2023-10-19T14:02:11.677085+0000","last_change":"2023-10-19T14:02:09.665590+0000","last_active":"2023-10-19T14:02:11.677085+0000","last_peered":"2023-10-19T14:02:11.677085+0000","last_clean":"2023-10-19T14:02:11.677085+0000","last_became_active":"2023-10-19T14:02:09.665266+0000","last_became_peered":"2023-10-19T14:02:09.665266+0000","last_unstale":"2023-10-19T14:02:11.677085+0000","last_undegraded":"2023-10-19T14:02:11.677085+0000","last_fullsized":"2023-10-19T14:02:11.677085+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_clean_scrub_stamp":"2023-10-19T14:02:08.638576+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:59:47.216132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:02:09.644002+0000","last_change":"2023-10-19T14:02:09.644002+0000","last_active":"2023-10-19T14:02:09.644002+0000","last_peered":"2023-10-19T14:02:09.644002+0000","last_clean":"2023-10-19T14:02:09.644002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:02:09.644002+0000","last_undegraded":"2023-10-19T14:02:09.644002+0000","last_fullsized":"2023-10-19T14:02:09.644002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.e","version":"187'2","reported_seq":13,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.692093+0000","last_change":"2023-10-19T14:02:09.669029+0000","last_active":"2023-10-19T14:02:10.692093+0000","last_peered":"2023-10-19T14:02:10.692093+0000","last_clean":"2023-10-19T14:02:10.692093+0000","last_became_active":"2023-10-19T14:02:09.668456+0000","last_became_peered":"2023-10-19T14:02:09.668456+0000","last_unstale":"2023-10-19T14:02:10.692093+0000","last_undegraded":"2023-10-19T14:02:10.692093+0000","last_fullsized":"2023-10-19T14:02:10.692093+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_clean_scrub_stamp":"2023-10-19T14:02:08.638576+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:17:38.030117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":11,"reported_epoch":188,"state":"active+clean","last_fresh":"2023-10-19T14:02:11.676190+0000","last_change":"2023-10-19T14:02:10.684928+0000","last_active":"2023-10-19T14:02:11.676190+0000","last_peered":"2023-10-19T14:02:11.676190+0000","last_clean":"2023-10-19T14:02:11.676190+0000","last_became_active":"2023-10-19T14:02:10.684656+0000","last_became_peered":"2023-10-19T14:02:10.684656+0000","last_unstale":"2023-10-19T14:02:11.676190+0000","last_undegraded":"2023-10-19T14:02:11.676190+0000","last_fullsized":"2023-10-19T14:02:11.676190+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:47:44.902028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:02:08.638576+0000","last_change":"2023-10-19T14:02:08.638576+0000","last_active":"2023-10-19T14:02:08.638576+0000","last_peered":"2023-10-19T14:02:08.638576+0000","last_clean":"2023-10-19T14:02:08.638576+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:02:08.638576+0000","last_undegraded":"2023-10-19T14:02:08.638576+0000","last_fullsized":"2023-10-19T14:02:08.638576+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_clean_scrub_stamp":"2023-10-19T14:02:08.638576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":11,"reported_epoch":188,"state":"active+clean","last_fresh":"2023-10-19T14:02:11.676393+0000","last_change":"2023-10-19T14:02:10.666725+0000","last_active":"2023-10-19T14:02:11.676393+0000","last_peered":"2023-10-19T14:02:11.676393+0000","last_clean":"2023-10-19T14:02:11.676393+0000","last_became_active":"2023-10-19T14:02:10.666353+0000","last_became_peered":"2023-10-19T14:02:10.666353+0000","last_unstale":"2023-10-19T14:02:11.676393+0000","last_undegraded":"2023-10-19T14:02:11.676393+0000","last_fullsized":"2023-10-19T14:02:11.676393+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:14:45.157582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.672241+0000","last_change":"2023-10-19T14:02:09.671519+0000","last_active":"2023-10-19T14:02:10.672241+0000","last_peered":"2023-10-19T14:02:10.672241+0000","last_clean":"2023-10-19T14:02:10.672241+0000","last_became_active":"2023-10-19T14:02:09.671226+0000","last_became_peered":"2023-10-19T14:02:09.671226+0000","last_unstale":"2023-10-19T14:02:10.672241+0000","last_undegraded":"2023-10-19T14:02:10.672241+0000","last_fullsized":"2023-10-19T14:02:10.672241+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_clean_scrub_stamp":"2023-10-19T14:02:08.638576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:27:43.363970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":11,"reported_epoch":188,"state":"active+clean","last_fresh":"2023-10-19T14:02:11.676947+0000","last_change":"2023-10-19T14:02:10.694215+0000","last_active":"2023-10-19T14:02:11.676947+0000","last_peered":"2023-10-19T14:02:11.676947+0000","last_clean":"2023-10-19T14:02:11.676947+0000","last_became_active":"2023-10-19T14:02:10.693907+0000","last_became_peered":"2023-10-19T14:02:10.693907+0000","last_unstale":"2023-10-19T14:02:11.676947+0000","last_undegraded":"2023-10-19T14:02:11.676947+0000","last_fullsized":"2023-10-19T14:02:11.676947+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:08:43.952742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:02:08.638576+0000","last_change":"2023-10-19T14:02:08.638576+0000","last_active":"2023-10-19T14:02:08.638576+0000","last_peered":"2023-10-19T14:02:08.638576+0000","last_clean":"2023-10-19T14:02:08.638576+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:02:08.638576+0000","last_undegraded":"2023-10-19T14:02:08.638576+0000","last_fullsized":"2023-10-19T14:02:08.638576+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_clean_scrub_stamp":"2023-10-19T14:02:08.638576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:02:09.644002+0000","last_change":"2023-10-19T14:02:09.644002+0000","last_active":"2023-10-19T14:02:09.644002+0000","last_peered":"2023-10-19T14:02:09.644002+0000","last_clean":"2023-10-19T14:02:09.644002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:02:09.644002+0000","last_undegraded":"2023-10-19T14:02:09.644002+0000","last_fullsized":"2023-10-19T14:02:09.644002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2023-10-19T14:02:11.676225+0000","last_change":"2023-10-19T14:02:09.670957+0000","last_active":"2023-10-19T14:02:11.676225+0000","last_peered":"2023-10-19T14:02:11.676225+0000","last_clean":"2023-10-19T14:02:11.676225+0000","last_became_active":"2023-10-19T14:02:09.670658+0000","last_became_peered":"2023-10-19T14:02:09.670658+0000","last_unstale":"2023-10-19T14:02:11.676225+0000","last_undegraded":"2023-10-19T14:02:11.676225+0000","last_fullsized":"2023-10-19T14:02:11.676225+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_clean_scrub_stamp":"2023-10-19T14:02:08.638576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:11:04.524319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:02:09.644002+0000","last_change":"2023-10-19T14:02:09.644002+0000","last_active":"2023-10-19T14:02:09.644002+0000","last_peered":"2023-10-19T14:02:09.644002+0000","last_clean":"2023-10-19T14:02:09.644002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:02:09.644002+0000","last_undegraded":"2023-10-19T14:02:09.644002+0000","last_fullsized":"2023-10-19T14:02:09.644002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2023-10-19T14:02:11.677433+0000","last_change":"2023-10-19T14:02:09.659752+0000","last_active":"2023-10-19T14:02:11.677433+0000","last_peered":"2023-10-19T14:02:11.677433+0000","last_clean":"2023-10-19T14:02:11.677433+0000","last_became_active":"2023-10-19T14:02:09.659434+0000","last_became_peered":"2023-10-19T14:02:09.659434+0000","last_unstale":"2023-10-19T14:02:11.677433+0000","last_undegraded":"2023-10-19T14:02:11.677433+0000","last_fullsized":"2023-10-19T14:02:11.677433+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_clean_scrub_stamp":"2023-10-19T14:02:08.638576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:47:19.668936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.7","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.683831+0000","last_change":"2023-10-19T14:02:10.682388+0000","last_active":"2023-10-19T14:02:10.683831+0000","last_peered":"2023-10-19T14:02:10.683831+0000","last_clean":"2023-10-19T14:02:10.683831+0000","last_became_active":"2023-10-19T14:02:10.682102+0000","last_became_peered":"2023-10-19T14:02:10.682102+0000","last_unstale":"2023-10-19T14:02:10.683831+0000","last_undegraded":"2023-10-19T14:02:10.683831+0000","last_fullsized":"2023-10-19T14:02:10.683831+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:22:58.263925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.670151+0000","last_change":"2023-10-19T14:02:09.665481+0000","last_active":"2023-10-19T14:02:10.670151+0000","last_peered":"2023-10-19T14:02:10.670151+0000","last_clean":"2023-10-19T14:02:10.670151+0000","last_became_active":"2023-10-19T14:02:09.665241+0000","last_became_peered":"2023-10-19T14:02:09.665241+0000","last_unstale":"2023-10-19T14:02:10.670151+0000","last_undegraded":"2023-10-19T14:02:10.670151+0000","last_fullsized":"2023-10-19T14:02:10.670151+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_clean_scrub_stamp":"2023-10-19T14:02:08.638576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:21:13.622150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":11,"reported_epoch":188,"state":"active+clean","last_fresh":"2023-10-19T14:02:11.676118+0000","last_change":"2023-10-19T14:02:10.671405+0000","last_active":"2023-10-19T14:02:11.676118+0000","last_peered":"2023-10-19T14:02:11.676118+0000","last_clean":"2023-10-19T14:02:11.676118+0000","last_became_active":"2023-10-19T14:02:10.671121+0000","last_became_peered":"2023-10-19T14:02:10.671121+0000","last_unstale":"2023-10-19T14:02:11.676118+0000","last_undegraded":"2023-10-19T14:02:11.676118+0000","last_fullsized":"2023-10-19T14:02:11.676118+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:58:35.411824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:02:08.638576+0000","last_change":"2023-10-19T14:02:08.638576+0000","last_active":"2023-10-19T14:02:08.638576+0000","last_peered":"2023-10-19T14:02:08.638576+0000","last_clean":"2023-10-19T14:02:08.638576+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:02:08.638576+0000","last_undegraded":"2023-10-19T14:02:08.638576+0000","last_fullsized":"2023-10-19T14:02:08.638576+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_clean_scrub_stamp":"2023-10-19T14:02:08.638576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.681988+0000","last_change":"2023-10-19T14:02:10.680661+0000","last_active":"2023-10-19T14:02:10.681988+0000","last_peered":"2023-10-19T14:02:10.681988+0000","last_clean":"2023-10-19T14:02:10.681988+0000","last_became_active":"2023-10-19T14:02:10.680407+0000","last_became_peered":"2023-10-19T14:02:10.680407+0000","last_unstale":"2023-10-19T14:02:10.681988+0000","last_undegraded":"2023-10-19T14:02:10.681988+0000","last_fullsized":"2023-10-19T14:02:10.681988+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:15:39.858994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.6","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2023-10-19T14:02:11.676860+0000","last_change":"2023-10-19T14:02:09.670892+0000","last_active":"2023-10-19T14:02:11.676860+0000","last_peered":"2023-10-19T14:02:11.676860+0000","last_clean":"2023-10-19T14:02:11.676860+0000","last_became_active":"2023-10-19T14:02:09.670599+0000","last_became_peered":"2023-10-19T14:02:09.670599+0000","last_unstale":"2023-10-19T14:02:11.676860+0000","last_undegraded":"2023-10-19T14:02:11.676860+0000","last_fullsized":"2023-10-19T14:02:11.676860+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_clean_scrub_stamp":"2023-10-19T14:02:08.638576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:50:35.829632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":11,"reported_epoch":188,"state":"active+clean","last_fresh":"2023-10-19T14:02:11.677464+0000","last_change":"2023-10-19T14:02:10.671805+0000","last_active":"2023-10-19T14:02:11.677464+0000","last_peered":"2023-10-19T14:02:11.677464+0000","last_clean":"2023-10-19T14:02:11.677464+0000","last_became_active":"2023-10-19T14:02:10.671534+0000","last_became_peered":"2023-10-19T14:02:10.671534+0000","last_unstale":"2023-10-19T14:02:11.677464+0000","last_undegraded":"2023-10-19T14:02:11.677464+0000","last_fullsized":"2023-10-19T14:02:11.677464+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:18:19.596195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2023-10-19T14:02:11.676842+0000","last_change":"2023-10-19T14:02:09.657923+0000","last_active":"2023-10-19T14:02:11.676842+0000","last_peered":"2023-10-19T14:02:11.676842+0000","last_clean":"2023-10-19T14:02:11.676842+0000","last_became_active":"2023-10-19T14:02:09.657637+0000","last_became_peered":"2023-10-19T14:02:09.657637+0000","last_unstale":"2023-10-19T14:02:11.676842+0000","last_undegraded":"2023-10-19T14:02:11.676842+0000","last_fullsized":"2023-10-19T14:02:11.676842+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_clean_scrub_stamp":"2023-10-19T14:02:08.638576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:02:00.400790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.b","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.681682+0000","last_change":"2023-10-19T14:02:10.678147+0000","last_active":"2023-10-19T14:02:10.681682+0000","last_peered":"2023-10-19T14:02:10.681682+0000","last_clean":"2023-10-19T14:02:10.681682+0000","last_became_active":"2023-10-19T14:02:10.677778+0000","last_became_peered":"2023-10-19T14:02:10.677778+0000","last_unstale":"2023-10-19T14:02:10.681682+0000","last_undegraded":"2023-10-19T14:02:10.681682+0000","last_fullsized":"2023-10-19T14:02:10.681682+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:42:33.481800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2023-10-19T14:02:11.677623+0000","last_change":"2023-10-19T14:02:09.663832+0000","last_active":"2023-10-19T14:02:11.677623+0000","last_peered":"2023-10-19T14:02:11.677623+0000","last_clean":"2023-10-19T14:02:11.677623+0000","last_became_active":"2023-10-19T14:02:09.663232+0000","last_became_peered":"2023-10-19T14:02:09.663232+0000","last_unstale":"2023-10-19T14:02:11.677623+0000","last_undegraded":"2023-10-19T14:02:11.677623+0000","last_fullsized":"2023-10-19T14:02:11.677623+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_clean_scrub_stamp":"2023-10-19T14:02:08.638576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:00:54.874154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.685383+0000","last_change":"2023-10-19T14:02:10.681779+0000","last_active":"2023-10-19T14:02:10.685383+0000","last_peered":"2023-10-19T14:02:10.685383+0000","last_clean":"2023-10-19T14:02:10.685383+0000","last_became_active":"2023-10-19T14:02:10.681508+0000","last_became_peered":"2023-10-19T14:02:10.681508+0000","last_unstale":"2023-10-19T14:02:10.685383+0000","last_undegraded":"2023-10-19T14:02:10.685383+0000","last_fullsized":"2023-10-19T14:02:10.685383+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:18:55.719249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.672729+0000","last_change":"2023-10-19T14:02:09.660687+0000","last_active":"2023-10-19T14:02:10.672729+0000","last_peered":"2023-10-19T14:02:10.672729+0000","last_clean":"2023-10-19T14:02:10.672729+0000","last_became_active":"2023-10-19T14:02:09.660428+0000","last_became_peered":"2023-10-19T14:02:09.660428+0000","last_unstale":"2023-10-19T14:02:10.672729+0000","last_undegraded":"2023-10-19T14:02:10.672729+0000","last_fullsized":"2023-10-19T14:02:10.672729+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_clean_scrub_stamp":"2023-10-19T14:02:08.638576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:06:06.545431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.d","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.689416+0000","last_change":"2023-10-19T14:02:10.688510+0000","last_active":"2023-10-19T14:02:10.689416+0000","last_peered":"2023-10-19T14:02:10.689416+0000","last_clean":"2023-10-19T14:02:10.689416+0000","last_became_active":"2023-10-19T14:02:10.687717+0000","last_became_peered":"2023-10-19T14:02:10.687717+0000","last_unstale":"2023-10-19T14:02:10.689416+0000","last_undegraded":"2023-10-19T14:02:10.689416+0000","last_fullsized":"2023-10-19T14:02:10.689416+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:32:43.725034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2023-10-19T14:02:11.676858+0000","last_change":"2023-10-19T14:02:09.669522+0000","last_active":"2023-10-19T14:02:11.676858+0000","last_peered":"2023-10-19T14:02:11.676858+0000","last_clean":"2023-10-19T14:02:11.676858+0000","last_became_active":"2023-10-19T14:02:09.669110+0000","last_became_peered":"2023-10-19T14:02:09.669110+0000","last_unstale":"2023-10-19T14:02:11.676858+0000","last_undegraded":"2023-10-19T14:02:11.676858+0000","last_fullsized":"2023-10-19T14:02:11.676858+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_clean_scrub_stamp":"2023-10-19T14:02:08.638576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:03:02.136067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.e","version":"0'0","reported_seq":11,"reported_epoch":188,"state":"active+clean","last_fresh":"2023-10-19T14:02:11.676947+0000","last_change":"2023-10-19T14:02:10.694078+0000","last_active":"2023-10-19T14:02:11.676947+0000","last_peered":"2023-10-19T14:02:11.676947+0000","last_clean":"2023-10-19T14:02:11.676947+0000","last_became_active":"2023-10-19T14:02:10.693764+0000","last_became_peered":"2023-10-19T14:02:10.693764+0000","last_unstale":"2023-10-19T14:02:11.676947+0000","last_undegraded":"2023-10-19T14:02:11.676947+0000","last_fullsized":"2023-10-19T14:02:11.676947+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:25:36.235834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2023-10-19T14:02:11.676322+0000","last_change":"2023-10-19T14:02:09.661342+0000","last_active":"2023-10-19T14:02:11.676322+0000","last_peered":"2023-10-19T14:02:11.676322+0000","last_clean":"2023-10-19T14:02:11.676322+0000","last_became_active":"2023-10-19T14:02:09.661084+0000","last_became_peered":"2023-10-19T14:02:09.661084+0000","last_unstale":"2023-10-19T14:02:11.676322+0000","last_undegraded":"2023-10-19T14:02:11.676322+0000","last_fullsized":"2023-10-19T14:02:11.676322+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_clean_scrub_stamp":"2023-10-19T14:02:08.638576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:32:14.505061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.f","version":"0'0","reported_seq":11,"reported_epoch":188,"state":"active+clean","last_fresh":"2023-10-19T14:02:11.677804+0000","last_change":"2023-10-19T14:02:10.676495+0000","last_active":"2023-10-19T14:02:11.677804+0000","last_peered":"2023-10-19T14:02:11.677804+0000","last_clean":"2023-10-19T14:02:11.677804+0000","last_became_active":"2023-10-19T14:02:10.676222+0000","last_became_peered":"2023-10-19T14:02:10.676222+0000","last_unstale":"2023-10-19T14:02:11.677804+0000","last_undegraded":"2023-10-19T14:02:11.677804+0000","last_fullsized":"2023-10-19T14:02:11.677804+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:02:10.131510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:02:08.638576+0000","last_change":"2023-10-19T14:02:08.638576+0000","last_active":"2023-10-19T14:02:08.638576+0000","last_peered":"2023-10-19T14:02:08.638576+0000","last_clean":"2023-10-19T14:02:08.638576+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:02:08.638576+0000","last_undegraded":"2023-10-19T14:02:08.638576+0000","last_fullsized":"2023-10-19T14:02:08.638576+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_clean_scrub_stamp":"2023-10-19T14:02:08.638576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.680999+0000","last_change":"2023-10-19T14:02:10.679293+0000","last_active":"2023-10-19T14:02:10.680999+0000","last_peered":"2023-10-19T14:02:10.680999+0000","last_clean":"2023-10-19T14:02:10.680999+0000","last_became_active":"2023-10-19T14:02:10.679040+0000","last_became_peered":"2023-10-19T14:02:10.679040+0000","last_unstale":"2023-10-19T14:02:10.680999+0000","last_undegraded":"2023-10-19T14:02:10.680999+0000","last_fullsized":"2023-10-19T14:02:10.680999+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T02:00:25.823282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2023-10-19T14:02:11.676768+0000","last_change":"2023-10-19T14:02:09.671482+0000","last_active":"2023-10-19T14:02:11.676768+0000","last_peered":"2023-10-19T14:02:11.676768+0000","last_clean":"2023-10-19T14:02:11.676768+0000","last_became_active":"2023-10-19T14:02:09.671225+0000","last_became_peered":"2023-10-19T14:02:09.671225+0000","last_unstale":"2023-10-19T14:02:11.676768+0000","last_undegraded":"2023-10-19T14:02:11.676768+0000","last_fullsized":"2023-10-19T14:02:11.676768+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_clean_scrub_stamp":"2023-10-19T14:02:08.638576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:05:14.796788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"177'3131","reported_seq":3848,"reported_epoch":186,"state":"active+clean","last_fresh":"2023-10-19T14:02:09.654340+0000","last_change":"2023-10-19T12:44:08.636613+0000","last_active":"2023-10-19T14:02:09.654340+0000","last_peered":"2023-10-19T14:02:09.654340+0000","last_clean":"2023-10-19T14:02:09.654340+0000","last_became_active":"2023-10-19T12:44:08.635697+0000","last_became_peered":"2023-10-19T12:44:08.635697+0000","last_unstale":"2023-10-19T14:02:09.654340+0000","last_undegraded":"2023-10-19T14:02:09.654340+0000","last_fullsized":"2023-10-19T14:02:09.654340+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_clean_scrub_stamp":"2023-10-19T12:44:07.399738+0000","objects_scrubbed":0,"log_size":3131,"log_dups_size":0,"ondisk_log_size":3131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:15:23.685485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1730,"num_read_kb":3594,"num_write":489,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":24,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":106,"acting":106,"num_store_stats":6},{"poolid":23,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":4716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":29,"ondisk_log_size":29,"up":50,"acting":50,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1730,"num_read_kb":3594,"num_write":489,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3131,"ondisk_log_size":3131,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575786,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60416,"kb_used_data":2868,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311424,"statfs":{"total":96636764160,"available":96574898176,"internally_reserved":0,"allocated":2936832,"data_stored":1779468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58915275},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:01:29 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.59099999999999997,"15min":0.59699999999999998},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.79900000000000004,"5min":0.85099999999999998,"15min":1.026},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.34699999999999998,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.83099999999999996,"5min":0.84099999999999997,"15min":1},"last":0.64200000000000002}]},{"osd":1,"last update":"Thu Oct 19 14:01:29 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.59899999999999998,"15min":0.60099999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.873},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.60899999999999999,"15min":0.61199999999999999},"min":{"1min":0.46000000000000002,"5min":0.41999999999999998,"15min":0.376},"max":{"1min":0.84499999999999997,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.54400000000000004}]},{"osd":2,"last update":"Thu Oct 19 14:01:29 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.65900000000000003,"15min":0.65500000000000003},"min":{"1min":0.44500000000000001,"5min":0.44400000000000001,"15min":0.39000000000000001},"max":{"1min":0.875,"5min":0.877,"15min":0.96599999999999997},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.65700000000000003,"15min":0.66000000000000003},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.36799999999999999},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.70499999999999996}]},{"osd":3,"last update":"Thu Oct 19 14:01:29 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.64300000000000002,"15min":0.64400000000000002},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.76800000000000002,"5min":0.86299999999999999,"15min":1.0129999999999999},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.69499999999999995,"15min":0.69199999999999995},"min":{"1min":0.504,"5min":0.504,"15min":0.48199999999999998},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.92000000000000004},"last":0.71899999999999997}]},{"osd":4,"last update":"Thu Oct 19 14:01:29 2023","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.48099999999999998,"15min":0.46600000000000003},"min":{"1min":0.32000000000000001,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.71799999999999997,"5min":0.73099999999999998,"15min":0.78500000000000003},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.503,"5min":0.51500000000000001,"15min":0.48799999999999999},"min":{"1min":0.35499999999999998,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.69399999999999995,"5min":0.78200000000000003,"15min":0.80900000000000005},"last":0.76800000000000002}]},{"osd":5,"last update":"Thu Oct 19 14:01:29 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.51000000000000001,"15min":0.49299999999999999},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.72899999999999998,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.624},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55400000000000005,"15min":0.53100000000000003},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.33900000000000002},"max":{"1min":0.77900000000000003,"5min":0.82699999999999996,"15min":0.86399999999999999},"last":0.78200000000000003}]},{"osd":6,"last update":"Thu Oct 19 14:01:29 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.58999999999999997,"15min":0.56899999999999995},"min":{"1min":0.38500000000000001,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.81100000000000005,"5min":0.83299999999999996,"15min":0.85799999999999998},"last":0.75},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.57399999999999995,"15min":0.55500000000000005},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.33200000000000002},"max":{"1min":0.78000000000000003,"5min":0.81000000000000005,"15min":0.94599999999999995},"last":0.58799999999999997}]}]},{"osd":6,"up_from":13,"seq":55834575785,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37548,"kb_used_data":2204,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334292,"statfs":{"total":96636764160,"available":96598315008,"internally_reserved":0,"allocated":2256896,"data_stored":1146173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:01:30 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.60599999999999998,"15min":0.60799999999999998},"min":{"1min":0.37,"5min":0.37,"15min":0.34599999999999997},"max":{"1min":0.82299999999999995,"5min":0.88100000000000001,"15min":1.3280000000000001},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61699999999999999,"15min":0.61399999999999999},"min":{"1min":0.46500000000000002,"5min":0.46400000000000002,"15min":0.26100000000000001},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":1.4279999999999999},"last":0.38600000000000001}]},{"osd":1,"last update":"Thu Oct 19 14:01:30 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.62,"15min":0.622},"min":{"1min":0.434,"5min":0.42799999999999999,"15min":0.35899999999999999},"max":{"1min":0.78400000000000003,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.624,"15min":0.61799999999999999},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.29299999999999998},"max":{"1min":0.83599999999999997,"5min":0.85399999999999998,"15min":0.89200000000000002},"last":0.47899999999999998}]},{"osd":2,"last update":"Thu Oct 19 14:01:30 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.38200000000000001},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":1.4650000000000001},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.621,"5min":0.65000000000000002,"15min":0.63200000000000001},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.35199999999999998},"max":{"1min":0.82399999999999995,"5min":0.85699999999999998,"15min":1.3979999999999999},"last":0.504}]},{"osd":3,"last update":"Thu Oct 19 14:01:30 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.64700000000000002,"15min":0.65300000000000002},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.76600000000000001,"5min":0.82899999999999996,"15min":1.496},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.66900000000000004,"15min":0.66500000000000004},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.38100000000000001},"max":{"1min":0.77500000000000002,"5min":0.83499999999999996,"15min":1.512},"last":0.61399999999999999}]},{"osd":4,"last update":"Thu Oct 19 14:01:30 2023","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.48499999999999999,"15min":0.47999999999999998},"min":{"1min":0.316,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.51600000000000001,"15min":0.50600000000000001},"min":{"1min":0.36699999999999999,"5min":0.35299999999999998,"15min":0.32800000000000001},"max":{"1min":0.79800000000000004,"5min":0.85699999999999998,"15min":0.93000000000000005},"last":0.434}]},{"osd":5,"last update":"Thu Oct 19 14:01:30 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.5,"15min":0.49299999999999999},"min":{"1min":0.35499999999999998,"5min":0.35099999999999998,"15min":0.33700000000000002},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54200000000000004},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.78800000000000003,"5min":0.83199999999999996,"15min":1.0249999999999999},"last":0.73999999999999999}]},{"osd":7,"last update":"Thu Oct 19 14:01:30 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.54200000000000004,"15min":0.54500000000000004},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.32200000000000001},"max":{"1min":0.81999999999999995,"5min":0.84499999999999997,"15min":0.878},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56399999999999995,"15min":0.55500000000000005},"min":{"1min":0.40500000000000003,"5min":0.373,"15min":0.33400000000000002},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.91100000000000003},"last":0.40600000000000003}]}]},{"osd":5,"up_from":13,"seq":55834575786,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59852,"kb_used_data":2296,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311988,"statfs":{"total":96636764160,"available":96575475712,"internally_reserved":0,"allocated":2351104,"data_stored":1189224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:01:10 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.59399999999999997,"15min":0.61699999999999999},"min":{"1min":0.379,"5min":0.35399999999999998,"15min":0.34499999999999997},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":1.2629999999999999},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.63700000000000001,"15min":0.63300000000000001},"min":{"1min":0.50700000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.88300000000000001,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.71799999999999997}]},{"osd":1,"last update":"Thu Oct 19 14:01:10 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.63700000000000001,"15min":0.63600000000000001},"min":{"1min":0.51700000000000002,"5min":0.435,"15min":0.41699999999999998},"max":{"1min":0.81100000000000005,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.63800000000000001,"15min":0.64000000000000001},"min":{"1min":0.495,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.84899999999999998,"5min":0.87,"15min":1.071},"last":0.64600000000000002}]},{"osd":2,"last update":"Thu Oct 19 14:01:10 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61899999999999999,"15min":0.628},"min":{"1min":0.46800000000000003,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.83499999999999996,"5min":0.875,"15min":1.044},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.64500000000000002,"15min":0.65300000000000002},"min":{"1min":0.48299999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.81399999999999995,"5min":0.90000000000000002,"15min":1.171},"last":0.67500000000000004}]},{"osd":3,"last update":"Thu Oct 19 14:01:10 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.63500000000000001,"15min":0.64400000000000002},"min":{"1min":0.46600000000000003,"5min":0.44,"15min":0.44},"max":{"1min":0.77800000000000002,"5min":0.85899999999999999,"15min":1.1910000000000001},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.67100000000000004,"15min":0.67500000000000004},"min":{"1min":0.52200000000000002,"5min":0.45500000000000002,"15min":0.43099999999999999},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":1.2210000000000001},"last":0.501}]},{"osd":4,"last update":"Thu Oct 19 14:01:10 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.5,"15min":0.48499999999999999},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.65100000000000002,"5min":0.745,"15min":0.75700000000000001},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.501,"5min":0.5,"15min":0.496},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.73899999999999999,"5min":0.76200000000000001,"15min":0.77300000000000002},"last":0.60699999999999998}]},{"osd":6,"last update":"Thu Oct 19 14:01:10 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.48899999999999999,"15min":0.48999999999999999},"min":{"1min":0.32600000000000001,"5min":0.32100000000000001,"15min":0.29699999999999999},"max":{"1min":0.64900000000000002,"5min":0.80000000000000004,"15min":0.83599999999999997},"last":0.746},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.53000000000000003,"15min":0.53100000000000003},"min":{"1min":0.38900000000000001,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.80000000000000004},"last":0.70299999999999996}]},{"osd":7,"last update":"Thu Oct 19 14:01:10 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.51500000000000001,"15min":0.51200000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.68000000000000005,"5min":0.79200000000000004,"15min":0.80000000000000004},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.56999999999999995,"15min":0.56299999999999994},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.80100000000000005,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.65800000000000003}]}]},{"osd":4,"up_from":13,"seq":55834575786,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41812,"kb_used_data":2304,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330028,"statfs":{"total":96636764160,"available":96593948672,"internally_reserved":0,"allocated":2359296,"data_stored":1189864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:01:45 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.57099999999999995,"15min":0.59099999999999997},"min":{"1min":0.437,"5min":0.38500000000000001,"15min":0.38400000000000001},"max":{"1min":0.76700000000000002,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.58399999999999996,"15min":0.59299999999999997},"min":{"1min":0.39400000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.70399999999999996,"5min":0.83199999999999996,"15min":0.98199999999999998},"last":0.77100000000000002}]},{"osd":1,"last update":"Thu Oct 19 14:01:45 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.621,"15min":0.64200000000000002},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.78600000000000003,"5min":0.88300000000000001,"15min":0.92100000000000004},"last":0.626},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.64500000000000002,"15min":0.65500000000000003},"min":{"1min":0.44500000000000001,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.79700000000000004,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.54100000000000004}]},{"osd":2,"last update":"Thu Oct 19 14:01:45 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.64300000000000002,"15min":0.64800000000000002},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.76900000000000002,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.66100000000000003,"15min":0.66200000000000003},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.751,"5min":1.054,"15min":1.054},"last":0.58499999999999996}]},{"osd":3,"last update":"Thu Oct 19 14:01:45 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.628,"15min":0.63700000000000001},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.72099999999999997,"5min":0.92000000000000004,"15min":0.93600000000000005},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.67200000000000004,"15min":0.67000000000000004},"min":{"1min":0.45600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":0.80700000000000005,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.56899999999999995}]},{"osd":5,"last update":"Thu Oct 19 14:01:45 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.47299999999999998,"15min":0.45800000000000002},"min":{"1min":0.318,"5min":0.318,"15min":0.29399999999999998},"max":{"1min":0.68300000000000005,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.496,"15min":0.48099999999999998},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.64300000000000002,"5min":0.77300000000000002,"15min":0.80900000000000005},"last":0.61199999999999999}]},{"osd":6,"last update":"Thu Oct 19 14:01:45 2023","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.49199999999999999,"15min":0.47299999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.72399999999999998,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.505,"5min":0.53400000000000003,"15min":0.52100000000000002},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.73799999999999999,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.67300000000000004}]},{"osd":7,"last update":"Thu Oct 19 14:01:45 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.53900000000000003,"15min":0.52600000000000002},"min":{"1min":0.34200000000000003,"5min":0.33800000000000002,"15min":0.32200000000000001},"max":{"1min":0.70499999999999996,"5min":0.79600000000000004,"15min":0.83699999999999997},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.61199999999999999,"15min":0.58499999999999996},"min":{"1min":0.40899999999999997,"5min":0.39500000000000002,"15min":0.39200000000000002},"max":{"1min":0.79100000000000004,"5min":1.02,"15min":1.02},"last":0.71299999999999997}]}]},{"osd":3,"up_from":13,"seq":55834575785,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51544,"kb_used_data":2308,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320296,"statfs":{"total":96636764160,"available":96583983104,"internally_reserved":0,"allocated":2363392,"data_stored":1191316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50395592},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:02:09 2023","interfaces":[{"interface":"back","average":{"1min":0.33200000000000002,"5min":0.39700000000000002,"15min":0.40100000000000002},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.51000000000000001,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.34699999999999998,"5min":0.42299999999999999,"15min":0.42899999999999999},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.45000000000000001,"5min":0.66600000000000004,"15min":0.69799999999999995},"last":0.32300000000000001}]},{"osd":1,"last update":"Thu Oct 19 14:02:09 2023","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.40100000000000002,"15min":0.42299999999999999},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.46600000000000003,"5min":0.58999999999999997,"15min":0.72999999999999998},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.379,"5min":0.45400000000000001,"15min":0.46000000000000002},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.56100000000000005,"5min":0.72599999999999998,"15min":0.72699999999999998},"last":0.308}]},{"osd":2,"last update":"Thu Oct 19 14:02:09 2023","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.46999999999999997,"15min":0.48099999999999998},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.53600000000000003,"5min":0.70399999999999996,"15min":0.72099999999999997},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.48999999999999999,"15min":0.499},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.56599999999999995,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.34599999999999997}]},{"osd":4,"last update":"Thu Oct 19 14:02:09 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.68700000000000006,"15min":0.69199999999999995},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.82499999999999996,"5min":1.3600000000000001,"15min":1.3600000000000001},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.68799999999999994,"15min":0.70699999999999996},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.82599999999999996,"5min":1.0209999999999999,"15min":1.2410000000000001},"last":0.629}]},{"osd":5,"last update":"Thu Oct 19 14:02:09 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.66400000000000003,"15min":0.67800000000000005},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.39700000000000002},"max":{"1min":0.85499999999999998,"5min":1.2450000000000001,"15min":1.2450000000000001},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.65400000000000003,"15min":0.66700000000000004},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.73299999999999998,"5min":1.385,"15min":1.385},"last":0.72499999999999998}]},{"osd":6,"last update":"Thu Oct 19 14:02:09 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.66800000000000004,"15min":0.68000000000000005},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.749,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.67300000000000004,"15min":0.69399999999999995},"min":{"1min":0.435,"5min":0.435,"15min":0.42799999999999999},"max":{"1min":0.78500000000000003,"5min":1.0649999999999999,"15min":1.282},"last":0.59699999999999998}]},{"osd":7,"last update":"Thu Oct 19 14:02:09 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.67900000000000005,"15min":0.68700000000000006},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.78100000000000003,"5min":1.3140000000000001,"15min":1.3140000000000001},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.69899999999999995,"15min":0.71499999999999997},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.80800000000000005,"5min":0.98899999999999999,"15min":1.304},"last":0.69399999999999995}]}]},{"osd":2,"up_from":13,"seq":55834575787,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59840,"kb_used_data":2288,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312000,"statfs":{"total":96636764160,"available":96575488000,"internally_reserved":0,"allocated":2342912,"data_stored":1189100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:01:58 2023","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.41399999999999998,"15min":0.40699999999999997},"min":{"1min":0.24199999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.63400000000000001,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.20999999999999999},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.44700000000000001,"15min":0.441},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.56899999999999995,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.35499999999999998}]},{"osd":1,"last update":"Thu Oct 19 14:01:58 2023","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.40500000000000003,"15min":0.41299999999999998},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.52100000000000002,"5min":0.60799999999999998,"15min":0.66600000000000004},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.46800000000000003,"15min":0.46400000000000002},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.26500000000000001},"max":{"1min":0.67300000000000004,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.25}]},{"osd":3,"last update":"Thu Oct 19 14:01:58 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.48699999999999999,"15min":0.49199999999999999},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.68300000000000005,"5min":0.72099999999999997,"15min":0.75900000000000001},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.503,"15min":0.51300000000000001},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.66200000000000003,"5min":0.69699999999999995,"15min":0.78700000000000003},"last":0.38800000000000001}]},{"osd":4,"last update":"Thu Oct 19 14:01:58 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.67000000000000004,"15min":0.68100000000000005},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.032,"5min":1.032,"15min":1.0640000000000001},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.68799999999999994,"15min":0.68899999999999995},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.86799999999999999,"5min":1.1619999999999999,"15min":1.2110000000000001},"last":0.57799999999999996}]},{"osd":5,"last update":"Thu Oct 19 14:01:58 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.70599999999999996,"15min":0.70399999999999996},"min":{"1min":0.433,"5min":0.433,"15min":0.42299999999999999},"max":{"1min":0.85099999999999998,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.622,"5min":0.68100000000000005,"15min":0.68999999999999995},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.44600000000000001},"max":{"1min":1.1140000000000001,"5min":1.1459999999999999,"15min":1.194},"last":0.505}]},{"osd":6,"last update":"Thu Oct 19 14:01:58 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.70099999999999996,"15min":0.71499999999999997},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.98399999999999999,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.72399999999999998,"15min":0.72199999999999998},"min":{"1min":0.48199999999999998,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.056,"5min":1.222,"15min":1.234},"last":0.435}]},{"osd":7,"last update":"Thu Oct 19 14:01:58 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.73199999999999998,"15min":0.73599999999999999},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.47699999999999998},"max":{"1min":1.0189999999999999,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.73099999999999998,"15min":0.74099999999999999},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.98999999999999999,"5min":1.161,"15min":1.161},"last":0.46200000000000002}]}]},{"osd":1,"up_from":13,"seq":55834575786,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41800,"kb_used_data":2300,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330040,"statfs":{"total":96636764160,"available":96593960960,"internally_reserved":0,"allocated":2355200,"data_stored":1190712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:01:52 2023","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.40799999999999997,"15min":0.41499999999999998},"min":{"1min":0.27400000000000002,"5min":0.26000000000000001,"15min":0.219},"max":{"1min":0.628,"5min":0.64500000000000002,"15min":0.69599999999999995},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.45100000000000001,"15min":0.45600000000000002},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.55400000000000005,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.45700000000000002}]},{"osd":2,"last update":"Thu Oct 19 14:01:52 2023","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.42599999999999999,"15min":0.434},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.53200000000000003,"5min":0.68000000000000005,"15min":0.68300000000000005},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.432,"5min":0.48399999999999999,"15min":0.48099999999999998},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.28499999999999998},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.52400000000000002}]},{"osd":3,"last update":"Thu Oct 19 14:01:52 2023","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.504,"15min":0.505},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.627,"5min":0.70299999999999996,"15min":0.73899999999999999},"last":0.5},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.44,"15min":0.44600000000000001},"min":{"1min":0.29899999999999999,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.67500000000000004},"last":0.42699999999999999}]},{"osd":4,"last update":"Thu Oct 19 14:01:52 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.67800000000000005,"15min":0.68600000000000005},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.83899999999999997,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.67800000000000005,"15min":0.67200000000000004},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.40699999999999997},"max":{"1min":0.751,"5min":0.90500000000000003,"15min":1.0720000000000001},"last":0.89500000000000002}]},{"osd":5,"last update":"Thu Oct 19 14:01:52 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.69699999999999995,"15min":0.69199999999999995},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.44},"max":{"1min":1.2470000000000001,"5min":1.2470000000000001,"15min":1.254},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.628,"5min":0.70499999999999996,"15min":0.70499999999999996},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.137,"5min":1.137,"15min":1.137},"last":0.78600000000000003}]},{"osd":6,"last update":"Thu Oct 19 14:01:52 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.68600000000000005,"15min":0.68799999999999994},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.41899999999999998},"max":{"1min":0.79700000000000004,"5min":1.25,"15min":1.25},"last":0.873},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.70499999999999996,"15min":0.71399999999999997},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.79300000000000004,"5min":0.999,"15min":1.153},"last":0.68300000000000005}]},{"osd":7,"last update":"Thu Oct 19 14:01:52 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.72099999999999997,"15min":0.71799999999999997},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.76900000000000002,"5min":1.1950000000000001,"15min":1.1950000000000001},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.624,"5min":0.72099999999999997,"15min":0.71899999999999997},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.878,"5min":1.097,"15min":1.097},"last":0.84699999999999998}]}]},{"osd":0,"up_from":13,"seq":55834575786,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38124,"kb_used_data":2780,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333716,"statfs":{"total":96636764160,"available":96597725184,"internally_reserved":0,"allocated":2846720,"data_stored":1735922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Oct 19 14:01:34 2023","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.38900000000000001,"15min":0.39600000000000002},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.45600000000000002,"15min":0.46000000000000002},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.49299999999999999}]},{"osd":2,"last update":"Thu Oct 19 14:01:34 2023","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.47399999999999998,"15min":0.47099999999999997},"min":{"1min":0.312,"5min":0.30299999999999999,"15min":0.28699999999999998},"max":{"1min":0.74199999999999999,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.441,"5min":0.45400000000000001,"15min":0.45100000000000001},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.46800000000000003}]},{"osd":3,"last update":"Thu Oct 19 14:01:34 2023","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.439,"15min":0.43099999999999999},"min":{"1min":0.32700000000000001,"5min":0.30099999999999999,"15min":0.27500000000000002},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.5,"5min":0.52000000000000002,"15min":0.50900000000000001},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.56699999999999995}]},{"osd":4,"last update":"Thu Oct 19 14:01:34 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.68700000000000006,"15min":0.67400000000000004},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.43099999999999999},"max":{"1min":1.0189999999999999,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.69399999999999995,"15min":0.67900000000000005},"min":{"1min":0.49099999999999999,"5min":0.46800000000000003,"15min":0.45900000000000002},"max":{"1min":0.88900000000000001,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.76600000000000001}]},{"osd":5,"last update":"Thu Oct 19 14:01:34 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.73399999999999999,"15min":0.71699999999999997},"min":{"1min":0.5,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.069,"5min":1.1970000000000001,"15min":1.1970000000000001},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.73699999999999999,"15min":0.73199999999999998},"min":{"1min":0.51500000000000001,"5min":0.443,"15min":0.443},"max":{"1min":0.95399999999999996,"5min":1.139,"15min":1.139},"last":0.91100000000000003}]},{"osd":6,"last update":"Thu Oct 19 14:01:34 2023","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.71499999999999997,"15min":0.70199999999999996},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.091,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.71099999999999997,"15min":0.69899999999999995},"min":{"1min":0.503,"5min":0.49399999999999999,"15min":0.443},"max":{"1min":0.98099999999999998,"5min":1.0660000000000001,"15min":1.167},"last":0.878}]},{"osd":7,"last update":"Thu Oct 19 14:01:34 2023","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.74199999999999999,"15min":0.72699999999999998},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.145,"5min":1.163,"15min":1.163},"last":0.755},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.71599999999999997,"15min":0.70099999999999996},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.123,"5min":1.123,"15min":1.123},"last":0.89900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-19T14:02:13.699 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-19T14:02:13.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.839+0000 7f197954c640 1 -- 172.21.15.57:0/999290820 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f19740c96b0 msgr2=0x7f19740c9b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:13.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.839+0000 7f197954c640 1 --2- 172.21.15.57:0/999290820 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f19740c96b0 0x7f19740c9b60 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 crypto rx=0x7f195c009830 tx=0x7f195c02bcd0 comp rx=0 tx=0).stop 2023-10-19T14:02:13.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.839+0000 7f197954c640 1 -- 172.21.15.57:0/999290820 shutdown_connections 2023-10-19T14:02:13.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.839+0000 7f197954c640 1 --2- 172.21.15.57:0/999290820 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f19740c96b0 0x7f19740c9b60 unknown :-1 s=CLOSED pgs=2048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:13.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.839+0000 7f197954c640 1 --2- 172.21.15.57:0/999290820 >> v2:172.21.15.179:3300/0 conn(0x7f197406af00 0x7f197406b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:13.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.839+0000 7f197954c640 1 -- 172.21.15.57:0/999290820 >> 172.21.15.57:0/999290820 conn(0x7f1974069930 msgr2=0x7f1974069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:02:13.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.840+0000 7f197954c640 1 -- 172.21.15.57:0/999290820 shutdown_connections 2023-10-19T14:02:13.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.840+0000 7f197954c640 1 -- 172.21.15.57:0/999290820 wait complete. 2023-10-19T14:02:13.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.841+0000 7f197954c640 1 Processor -- start 2023-10-19T14:02:13.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.841+0000 7f197954c640 1 -- start start 2023-10-19T14:02:13.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.841+0000 7f197954c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f197406af00 0x7f1974157f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:13.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.841+0000 7f197954c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f19740ab500 0x7f19740aa8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:13.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.842+0000 7f197954c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f19740c96b0 0x7f19740aae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:13.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.842+0000 7f197954c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1974071010 con 0x7f19740c96b0 2023-10-19T14:02:13.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.842+0000 7f197954c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f197406ff70 con 0x7f197406af00 2023-10-19T14:02:13.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.842+0000 7f197954c640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1974070250 con 0x7f19740ab500 2023-10-19T14:02:13.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.842+0000 7f19737fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f19740c96b0 0x7f19740aae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:13.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.842+0000 7f19737fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f19740c96b0 0x7f19740aae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59778/0 (socket says 172.21.15.57:59778) 2023-10-19T14:02:13.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.842+0000 7f19737fe640 1 -- 172.21.15.57:0/2555011117 learned_addr learned my addr 172.21.15.57:0/2555011117 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:02:13.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.842+0000 7f1972ffd640 1 --2- 172.21.15.57:0/2555011117 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f197406af00 0x7f1974157f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:13.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.842+0000 7f196a7fc640 1 --2- 172.21.15.57:0/2555011117 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f19740ab500 0x7f19740aa8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:13.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.842+0000 7f19737fe640 1 -- 172.21.15.57:0/2555011117 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f19740ab500 msgr2=0x7f19740aa8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:13.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.842+0000 7f19737fe640 1 --2- 172.21.15.57:0/2555011117 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f19740ab500 0x7f19740aa8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:13.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.842+0000 7f19737fe640 1 -- 172.21.15.57:0/2555011117 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f197406af00 msgr2=0x7f1974157f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:13.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.842+0000 7f19737fe640 1 --2- 172.21.15.57:0/2555011117 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f197406af00 0x7f1974157f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:13.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.842+0000 7f19737fe640 1 -- 172.21.15.57:0/2555011117 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f195c009490 con 0x7f19740c96b0 2023-10-19T14:02:13.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.842+0000 7f19737fe640 1 --2- 172.21.15.57:0/2555011117 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f19740c96b0 0x7f19740aae00 secure :-1 s=READY pgs=2049 cs=0 l=1 rev1=1 crypto rx=0x7f196c00cbc0 tx=0x7f196c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:02:13.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.842+0000 7f196bfff640 1 -- 172.21.15.57:0/2555011117 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f196c00ad40 con 0x7f19740c96b0 2023-10-19T14:02:13.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.842+0000 7f196bfff640 1 -- 172.21.15.57:0/2555011117 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f196c012420 con 0x7f19740c96b0 2023-10-19T14:02:13.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.842+0000 7f197954c640 1 -- 172.21.15.57:0/2555011117 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1974070530 con 0x7f19740c96b0 2023-10-19T14:02:13.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.842+0000 7f196bfff640 1 -- 172.21.15.57:0/2555011117 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f196c012850 con 0x7f19740c96b0 2023-10-19T14:02:13.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.842+0000 7f197954c640 1 -- 172.21.15.57:0/2555011117 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1974070af0 con 0x7f19740c96b0 2023-10-19T14:02:13.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.843+0000 7f197954c640 1 -- 172.21.15.57:0/2555011117 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1974070ce0 con 0x7f19740c96b0 2023-10-19T14:02:13.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.844+0000 7f196bfff640 1 -- 172.21.15.57:0/2555011117 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 99) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f196c01a3b0 con 0x7f19740c96b0 2023-10-19T14:02:13.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.844+0000 7f196bfff640 1 --2- 172.21.15.57:0/2555011117 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f195406af90 0x7f195406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:13.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.844+0000 7f196bfff640 1 -- 172.21.15.57:0/2555011117 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f196c08a070 con 0x7f19740c96b0 2023-10-19T14:02:13.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.845+0000 7f1972ffd640 1 --2- 172.21.15.57:0/2555011117 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f195406af90 0x7f195406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:13.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.845+0000 7f1972ffd640 1 --2- 172.21.15.57:0/2555011117 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f195406af90 0x7f195406d450 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f196400bbe0 tx=0x7f196400b390 comp rx=0 tx=0).ready entity=mgr.8315 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:02:13.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:13.849+0000 7f196bfff640 1 -- 172.21.15.57:0/2555011117 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f196c0bcca0 con 0x7f19740c96b0 2023-10-19T14:02:14.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.023+0000 7f197954c640 1 -- 172.21.15.57:0/2555011117 --> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1974159c50 con 0x7f195406af90 2023-10-19T14:02:14.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.031+0000 7f196bfff640 1 -- 172.21.15.57:0/2555011117 <== mgr.8315 v2:172.21.15.179:6832/1426553632 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289863 (secure 0 0 0) 0x7f1964015070 con 0x7f195406af90 2023-10-19T14:02:14.036 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:02:14.038 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2023-10-19T14:02:14.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.034+0000 7f197954c640 1 -- 172.21.15.57:0/2555011117 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f195406af90 msgr2=0x7f195406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:14.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.034+0000 7f197954c640 1 --2- 172.21.15.57:0/2555011117 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f195406af90 0x7f195406d450 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f196400bbe0 tx=0x7f196400b390 comp rx=0 tx=0).stop 2023-10-19T14:02:14.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.034+0000 7f197954c640 1 -- 172.21.15.57:0/2555011117 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f19740c96b0 msgr2=0x7f19740aae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:14.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.034+0000 7f197954c640 1 --2- 172.21.15.57:0/2555011117 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f19740c96b0 0x7f19740aae00 secure :-1 s=READY pgs=2049 cs=0 l=1 rev1=1 crypto rx=0x7f196c00cbc0 tx=0x7f196c00a430 comp rx=0 tx=0).stop 2023-10-19T14:02:14.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.034+0000 7f197954c640 1 -- 172.21.15.57:0/2555011117 shutdown_connections 2023-10-19T14:02:14.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.034+0000 7f197954c640 1 --2- 172.21.15.57:0/2555011117 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f195406af90 0x7f195406d450 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:14.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.034+0000 7f197954c640 1 --2- 172.21.15.57:0/2555011117 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f19740c96b0 0x7f19740aae00 unknown :-1 s=CLOSED pgs=2049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:14.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.034+0000 7f197954c640 1 --2- 172.21.15.57:0/2555011117 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f19740ab500 0x7f19740aa8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:14.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.034+0000 7f197954c640 1 --2- 172.21.15.57:0/2555011117 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f197406af00 0x7f1974157f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:14.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.034+0000 7f197954c640 1 -- 172.21.15.57:0/2555011117 >> 172.21.15.57:0/2555011117 conn(0x7f1974069930 msgr2=0x7f197406cd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:02:14.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.034+0000 7f197954c640 1 -- 172.21.15.57:0/2555011117 shutdown_connections 2023-10-19T14:02:14.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.035+0000 7f197954c640 1 -- 172.21.15.57:0/2555011117 wait complete. 2023-10-19T14:02:14.047 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2023-10-19T14:02:13.999975+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1730,"num_read_kb":3594,"num_write":533,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3169,"ondisk_log_size":3169,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":74,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":391116,"kb_used_data":19524,"kb_used_omap":12,"kb_used_meta":371443,"kb_avail":754583604,"statfs":{"total":773094113280,"available":772693610496,"internally_reserved":0,"allocated":19992576,"data_stored":10697737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":380358201},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.350491"},"pg_stats":[{"pgid":"24.3f","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.685267+0000","last_change":"2023-10-19T14:02:10.680090+0000","last_active":"2023-10-19T14:02:10.685267+0000","last_peered":"2023-10-19T14:02:10.685267+0000","last_clean":"2023-10-19T14:02:10.685267+0000","last_became_active":"2023-10-19T14:02:10.679811+0000","last_became_peered":"2023-10-19T14:02:10.679811+0000","last_unstale":"2023-10-19T14:02:10.685267+0000","last_undegraded":"2023-10-19T14:02:10.685267+0000","last_fullsized":"2023-10-19T14:02:10.685267+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:05:33.395729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.3e","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.677900+0000","last_change":"2023-10-19T14:02:10.676966+0000","last_active":"2023-10-19T14:02:10.677900+0000","last_peered":"2023-10-19T14:02:10.677900+0000","last_clean":"2023-10-19T14:02:10.677900+0000","last_became_active":"2023-10-19T14:02:10.676689+0000","last_became_peered":"2023-10-19T14:02:10.676689+0000","last_unstale":"2023-10-19T14:02:10.677900+0000","last_undegraded":"2023-10-19T14:02:10.677900+0000","last_fullsized":"2023-10-19T14:02:10.677900+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:32:15.510487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3d","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.685628+0000","last_change":"2023-10-19T14:02:10.683123+0000","last_active":"2023-10-19T14:02:10.685628+0000","last_peered":"2023-10-19T14:02:10.685628+0000","last_clean":"2023-10-19T14:02:10.685628+0000","last_became_active":"2023-10-19T14:02:10.682822+0000","last_became_peered":"2023-10-19T14:02:10.682822+0000","last_unstale":"2023-10-19T14:02:10.685628+0000","last_undegraded":"2023-10-19T14:02:10.685628+0000","last_fullsized":"2023-10-19T14:02:10.685628+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:15:35.564507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.3c","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.685634+0000","last_change":"2023-10-19T14:02:10.682016+0000","last_active":"2023-10-19T14:02:10.685634+0000","last_peered":"2023-10-19T14:02:10.685634+0000","last_clean":"2023-10-19T14:02:10.685634+0000","last_became_active":"2023-10-19T14:02:10.681752+0000","last_became_peered":"2023-10-19T14:02:10.681752+0000","last_unstale":"2023-10-19T14:02:10.685634+0000","last_undegraded":"2023-10-19T14:02:10.685634+0000","last_fullsized":"2023-10-19T14:02:10.685634+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:47:25.158279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.3b","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.690071+0000","last_change":"2023-10-19T14:02:10.684486+0000","last_active":"2023-10-19T14:02:10.690071+0000","last_peered":"2023-10-19T14:02:10.690071+0000","last_clean":"2023-10-19T14:02:10.690071+0000","last_became_active":"2023-10-19T14:02:10.684223+0000","last_became_peered":"2023-10-19T14:02:10.684223+0000","last_unstale":"2023-10-19T14:02:10.690071+0000","last_undegraded":"2023-10-19T14:02:10.690071+0000","last_fullsized":"2023-10-19T14:02:10.690071+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:28:32.842244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.3a","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.688713+0000","last_change":"2023-10-19T14:02:10.687039+0000","last_active":"2023-10-19T14:02:10.688713+0000","last_peered":"2023-10-19T14:02:10.688713+0000","last_clean":"2023-10-19T14:02:10.688713+0000","last_became_active":"2023-10-19T14:02:10.686737+0000","last_became_peered":"2023-10-19T14:02:10.686737+0000","last_unstale":"2023-10-19T14:02:10.688713+0000","last_undegraded":"2023-10-19T14:02:10.688713+0000","last_fullsized":"2023-10-19T14:02:10.688713+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:30:55.208738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.39","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.683909+0000","last_change":"2023-10-19T14:02:10.681884+0000","last_active":"2023-10-19T14:02:10.683909+0000","last_peered":"2023-10-19T14:02:10.683909+0000","last_clean":"2023-10-19T14:02:10.683909+0000","last_became_active":"2023-10-19T14:02:10.681470+0000","last_became_peered":"2023-10-19T14:02:10.681470+0000","last_unstale":"2023-10-19T14:02:10.683909+0000","last_undegraded":"2023-10-19T14:02:10.683909+0000","last_fullsized":"2023-10-19T14:02:10.683909+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:20:59.785404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.38","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.690451+0000","last_change":"2023-10-19T14:02:10.688431+0000","last_active":"2023-10-19T14:02:10.690451+0000","last_peered":"2023-10-19T14:02:10.690451+0000","last_clean":"2023-10-19T14:02:10.690451+0000","last_became_active":"2023-10-19T14:02:10.687693+0000","last_became_peered":"2023-10-19T14:02:10.687693+0000","last_unstale":"2023-10-19T14:02:10.690451+0000","last_undegraded":"2023-10-19T14:02:10.690451+0000","last_fullsized":"2023-10-19T14:02:10.690451+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:33:35.846384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.37","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.682038+0000","last_change":"2023-10-19T14:02:10.679157+0000","last_active":"2023-10-19T14:02:10.682038+0000","last_peered":"2023-10-19T14:02:10.682038+0000","last_clean":"2023-10-19T14:02:10.682038+0000","last_became_active":"2023-10-19T14:02:10.678735+0000","last_became_peered":"2023-10-19T14:02:10.678735+0000","last_unstale":"2023-10-19T14:02:10.682038+0000","last_undegraded":"2023-10-19T14:02:10.682038+0000","last_fullsized":"2023-10-19T14:02:10.682038+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:12:18.741834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.36","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.690625+0000","last_change":"2023-10-19T14:02:10.688854+0000","last_active":"2023-10-19T14:02:10.690625+0000","last_peered":"2023-10-19T14:02:10.690625+0000","last_clean":"2023-10-19T14:02:10.690625+0000","last_became_active":"2023-10-19T14:02:10.688575+0000","last_became_peered":"2023-10-19T14:02:10.688575+0000","last_unstale":"2023-10-19T14:02:10.690625+0000","last_undegraded":"2023-10-19T14:02:10.690625+0000","last_fullsized":"2023-10-19T14:02:10.690625+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:04:55.679325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.35","version":"0'0","reported_seq":11,"reported_epoch":188,"state":"active+clean","last_fresh":"2023-10-19T14:02:11.676554+0000","last_change":"2023-10-19T14:02:10.669419+0000","last_active":"2023-10-19T14:02:11.676554+0000","last_peered":"2023-10-19T14:02:11.676554+0000","last_clean":"2023-10-19T14:02:11.676554+0000","last_became_active":"2023-10-19T14:02:10.668994+0000","last_became_peered":"2023-10-19T14:02:10.668994+0000","last_unstale":"2023-10-19T14:02:11.676554+0000","last_undegraded":"2023-10-19T14:02:11.676554+0000","last_fullsized":"2023-10-19T14:02:11.676554+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:32:43.725034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.34","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.688751+0000","last_change":"2023-10-19T14:02:10.687045+0000","last_active":"2023-10-19T14:02:10.688751+0000","last_peered":"2023-10-19T14:02:10.688751+0000","last_clean":"2023-10-19T14:02:10.688751+0000","last_became_active":"2023-10-19T14:02:10.686758+0000","last_became_peered":"2023-10-19T14:02:10.686758+0000","last_unstale":"2023-10-19T14:02:10.688751+0000","last_undegraded":"2023-10-19T14:02:10.688751+0000","last_fullsized":"2023-10-19T14:02:10.688751+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:47:44.902028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.33","version":"0'0","reported_seq":11,"reported_epoch":188,"state":"active+clean","last_fresh":"2023-10-19T14:02:11.676817+0000","last_change":"2023-10-19T14:02:10.682320+0000","last_active":"2023-10-19T14:02:11.676817+0000","last_peered":"2023-10-19T14:02:11.676817+0000","last_clean":"2023-10-19T14:02:11.676817+0000","last_became_active":"2023-10-19T14:02:10.679611+0000","last_became_peered":"2023-10-19T14:02:10.679611+0000","last_unstale":"2023-10-19T14:02:11.676817+0000","last_undegraded":"2023-10-19T14:02:11.676817+0000","last_fullsized":"2023-10-19T14:02:11.676817+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:26:53.697061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.32","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.675349+0000","last_change":"2023-10-19T14:02:10.670051+0000","last_active":"2023-10-19T14:02:10.675349+0000","last_peered":"2023-10-19T14:02:10.675349+0000","last_clean":"2023-10-19T14:02:10.675349+0000","last_became_active":"2023-10-19T14:02:10.669801+0000","last_became_peered":"2023-10-19T14:02:10.669801+0000","last_unstale":"2023-10-19T14:02:10.675349+0000","last_undegraded":"2023-10-19T14:02:10.675349+0000","last_fullsized":"2023-10-19T14:02:10.675349+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:07:10.151502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.31","version":"0'0","reported_seq":11,"reported_epoch":188,"state":"active+clean","last_fresh":"2023-10-19T14:02:11.676135+0000","last_change":"2023-10-19T14:02:10.684424+0000","last_active":"2023-10-19T14:02:11.676135+0000","last_peered":"2023-10-19T14:02:11.676135+0000","last_clean":"2023-10-19T14:02:11.676135+0000","last_became_active":"2023-10-19T14:02:10.684125+0000","last_became_peered":"2023-10-19T14:02:10.684125+0000","last_unstale":"2023-10-19T14:02:11.676135+0000","last_undegraded":"2023-10-19T14:02:11.676135+0000","last_fullsized":"2023-10-19T14:02:11.676135+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:22:58.263925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.30","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.697012+0000","last_change":"2023-10-19T14:02:10.694706+0000","last_active":"2023-10-19T14:02:10.697012+0000","last_peered":"2023-10-19T14:02:10.697012+0000","last_clean":"2023-10-19T14:02:10.697012+0000","last_became_active":"2023-10-19T14:02:10.694360+0000","last_became_peered":"2023-10-19T14:02:10.694360+0000","last_unstale":"2023-10-19T14:02:10.697012+0000","last_undegraded":"2023-10-19T14:02:10.697012+0000","last_fullsized":"2023-10-19T14:02:10.697012+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:44:19.924159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.2f","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.682022+0000","last_change":"2023-10-19T14:02:10.678976+0000","last_active":"2023-10-19T14:02:10.682022+0000","last_peered":"2023-10-19T14:02:10.682022+0000","last_clean":"2023-10-19T14:02:10.682022+0000","last_became_active":"2023-10-19T14:02:10.678447+0000","last_became_peered":"2023-10-19T14:02:10.678447+0000","last_unstale":"2023-10-19T14:02:10.682022+0000","last_undegraded":"2023-10-19T14:02:10.682022+0000","last_fullsized":"2023-10-19T14:02:10.682022+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:09:52.311831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.2e","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.682038+0000","last_change":"2023-10-19T14:02:10.681400+0000","last_active":"2023-10-19T14:02:10.682038+0000","last_peered":"2023-10-19T14:02:10.682038+0000","last_clean":"2023-10-19T14:02:10.682038+0000","last_became_active":"2023-10-19T14:02:10.681123+0000","last_became_peered":"2023-10-19T14:02:10.681123+0000","last_unstale":"2023-10-19T14:02:10.682038+0000","last_undegraded":"2023-10-19T14:02:10.682038+0000","last_fullsized":"2023-10-19T14:02:10.682038+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:53:44.182540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.2d","version":"0'0","reported_seq":11,"reported_epoch":188,"state":"active+clean","last_fresh":"2023-10-19T14:02:11.676637+0000","last_change":"2023-10-19T14:02:10.685211+0000","last_active":"2023-10-19T14:02:11.676637+0000","last_peered":"2023-10-19T14:02:11.676637+0000","last_clean":"2023-10-19T14:02:11.676637+0000","last_became_active":"2023-10-19T14:02:10.684944+0000","last_became_peered":"2023-10-19T14:02:10.684944+0000","last_unstale":"2023-10-19T14:02:11.676637+0000","last_undegraded":"2023-10-19T14:02:11.676637+0000","last_fullsized":"2023-10-19T14:02:11.676637+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:22:30.762765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.2c","version":"0'0","reported_seq":11,"reported_epoch":188,"state":"active+clean","last_fresh":"2023-10-19T14:02:11.677539+0000","last_change":"2023-10-19T14:02:10.677499+0000","last_active":"2023-10-19T14:02:11.677539+0000","last_peered":"2023-10-19T14:02:11.677539+0000","last_clean":"2023-10-19T14:02:11.677539+0000","last_became_active":"2023-10-19T14:02:10.677245+0000","last_became_peered":"2023-10-19T14:02:10.677245+0000","last_unstale":"2023-10-19T14:02:11.677539+0000","last_undegraded":"2023-10-19T14:02:11.677539+0000","last_fullsized":"2023-10-19T14:02:11.677539+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:32:56.038053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.2b","version":"0'0","reported_seq":11,"reported_epoch":188,"state":"active+clean","last_fresh":"2023-10-19T14:02:11.676840+0000","last_change":"2023-10-19T14:02:10.686175+0000","last_active":"2023-10-19T14:02:11.676840+0000","last_peered":"2023-10-19T14:02:11.676840+0000","last_clean":"2023-10-19T14:02:11.676840+0000","last_became_active":"2023-10-19T14:02:10.685847+0000","last_became_peered":"2023-10-19T14:02:10.685847+0000","last_unstale":"2023-10-19T14:02:11.676840+0000","last_undegraded":"2023-10-19T14:02:11.676840+0000","last_fullsized":"2023-10-19T14:02:11.676840+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:01:51.421272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.2a","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.677928+0000","last_change":"2023-10-19T14:02:10.677165+0000","last_active":"2023-10-19T14:02:10.677928+0000","last_peered":"2023-10-19T14:02:10.677928+0000","last_clean":"2023-10-19T14:02:10.677928+0000","last_became_active":"2023-10-19T14:02:10.676908+0000","last_became_peered":"2023-10-19T14:02:10.676908+0000","last_unstale":"2023-10-19T14:02:10.677928+0000","last_undegraded":"2023-10-19T14:02:10.677928+0000","last_fullsized":"2023-10-19T14:02:10.677928+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:14:10.545049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.29","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.681891+0000","last_change":"2023-10-19T14:02:10.679340+0000","last_active":"2023-10-19T14:02:10.681891+0000","last_peered":"2023-10-19T14:02:10.681891+0000","last_clean":"2023-10-19T14:02:10.681891+0000","last_became_active":"2023-10-19T14:02:10.679056+0000","last_became_peered":"2023-10-19T14:02:10.679056+0000","last_unstale":"2023-10-19T14:02:10.681891+0000","last_undegraded":"2023-10-19T14:02:10.681891+0000","last_fullsized":"2023-10-19T14:02:10.681891+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:53:37.707011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.28","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.685597+0000","last_change":"2023-10-19T14:02:10.683726+0000","last_active":"2023-10-19T14:02:10.685597+0000","last_peered":"2023-10-19T14:02:10.685597+0000","last_clean":"2023-10-19T14:02:10.685597+0000","last_became_active":"2023-10-19T14:02:10.683452+0000","last_became_peered":"2023-10-19T14:02:10.683452+0000","last_unstale":"2023-10-19T14:02:10.685597+0000","last_undegraded":"2023-10-19T14:02:10.685597+0000","last_fullsized":"2023-10-19T14:02:10.685597+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:28:05.098300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.27","version":"0'0","reported_seq":11,"reported_epoch":188,"state":"active+clean","last_fresh":"2023-10-19T14:02:11.677064+0000","last_change":"2023-10-19T14:02:10.678793+0000","last_active":"2023-10-19T14:02:11.677064+0000","last_peered":"2023-10-19T14:02:11.677064+0000","last_clean":"2023-10-19T14:02:11.677064+0000","last_became_active":"2023-10-19T14:02:10.678473+0000","last_became_peered":"2023-10-19T14:02:10.678473+0000","last_unstale":"2023-10-19T14:02:11.677064+0000","last_undegraded":"2023-10-19T14:02:11.677064+0000","last_fullsized":"2023-10-19T14:02:11.677064+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:49:09.882058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.26","version":"0'0","reported_seq":11,"reported_epoch":188,"state":"active+clean","last_fresh":"2023-10-19T14:02:11.677709+0000","last_change":"2023-10-19T14:02:10.687721+0000","last_active":"2023-10-19T14:02:11.677709+0000","last_peered":"2023-10-19T14:02:11.677709+0000","last_clean":"2023-10-19T14:02:11.677709+0000","last_became_active":"2023-10-19T14:02:10.687270+0000","last_became_peered":"2023-10-19T14:02:10.687270+0000","last_unstale":"2023-10-19T14:02:11.677709+0000","last_undegraded":"2023-10-19T14:02:11.677709+0000","last_fullsized":"2023-10-19T14:02:11.677709+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:30:27.924431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.25","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.685487+0000","last_change":"2023-10-19T14:02:10.683270+0000","last_active":"2023-10-19T14:02:10.685487+0000","last_peered":"2023-10-19T14:02:10.685487+0000","last_clean":"2023-10-19T14:02:10.685487+0000","last_became_active":"2023-10-19T14:02:10.682994+0000","last_became_peered":"2023-10-19T14:02:10.682994+0000","last_unstale":"2023-10-19T14:02:10.685487+0000","last_undegraded":"2023-10-19T14:02:10.685487+0000","last_fullsized":"2023-10-19T14:02:10.685487+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:14:45.157582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.24","version":"0'0","reported_seq":11,"reported_epoch":188,"state":"active+clean","last_fresh":"2023-10-19T14:02:11.676818+0000","last_change":"2023-10-19T14:02:10.695300+0000","last_active":"2023-10-19T14:02:11.676818+0000","last_peered":"2023-10-19T14:02:11.676818+0000","last_clean":"2023-10-19T14:02:11.676818+0000","last_became_active":"2023-10-19T14:02:10.694960+0000","last_became_peered":"2023-10-19T14:02:10.694960+0000","last_unstale":"2023-10-19T14:02:11.676818+0000","last_undegraded":"2023-10-19T14:02:11.676818+0000","last_fullsized":"2023-10-19T14:02:11.676818+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:15:56.950509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.23","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.681808+0000","last_change":"2023-10-19T14:02:10.680553+0000","last_active":"2023-10-19T14:02:10.681808+0000","last_peered":"2023-10-19T14:02:10.681808+0000","last_clean":"2023-10-19T14:02:10.681808+0000","last_became_active":"2023-10-19T14:02:10.680292+0000","last_became_peered":"2023-10-19T14:02:10.680292+0000","last_unstale":"2023-10-19T14:02:10.681808+0000","last_undegraded":"2023-10-19T14:02:10.681808+0000","last_fullsized":"2023-10-19T14:02:10.681808+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:11:05.529745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.22","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.681590+0000","last_change":"2023-10-19T14:02:10.679166+0000","last_active":"2023-10-19T14:02:10.681590+0000","last_peered":"2023-10-19T14:02:10.681590+0000","last_clean":"2023-10-19T14:02:10.681590+0000","last_became_active":"2023-10-19T14:02:10.678781+0000","last_became_peered":"2023-10-19T14:02:10.678781+0000","last_unstale":"2023-10-19T14:02:10.681590+0000","last_undegraded":"2023-10-19T14:02:10.681590+0000","last_fullsized":"2023-10-19T14:02:10.681590+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:35:52.027443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.21","version":"0'0","reported_seq":11,"reported_epoch":188,"state":"active+clean","last_fresh":"2023-10-19T14:02:11.676226+0000","last_change":"2023-10-19T14:02:10.681523+0000","last_active":"2023-10-19T14:02:11.676226+0000","last_peered":"2023-10-19T14:02:11.676226+0000","last_clean":"2023-10-19T14:02:11.676226+0000","last_became_active":"2023-10-19T14:02:10.681243+0000","last_became_peered":"2023-10-19T14:02:10.681243+0000","last_unstale":"2023-10-19T14:02:11.676226+0000","last_undegraded":"2023-10-19T14:02:11.676226+0000","last_fullsized":"2023-10-19T14:02:11.676226+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:30:55.208738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.20","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.680064+0000","last_change":"2023-10-19T14:02:10.678496+0000","last_active":"2023-10-19T14:02:10.680064+0000","last_peered":"2023-10-19T14:02:10.680064+0000","last_clean":"2023-10-19T14:02:10.680064+0000","last_became_active":"2023-10-19T14:02:10.678190+0000","last_became_peered":"2023-10-19T14:02:10.678190+0000","last_unstale":"2023-10-19T14:02:10.680064+0000","last_undegraded":"2023-10-19T14:02:10.680064+0000","last_fullsized":"2023-10-19T14:02:10.680064+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:40:35.947585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.10","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.689546+0000","last_change":"2023-10-19T14:02:10.689114+0000","last_active":"2023-10-19T14:02:10.689546+0000","last_peered":"2023-10-19T14:02:10.689546+0000","last_clean":"2023-10-19T14:02:10.689546+0000","last_became_active":"2023-10-19T14:02:10.688259+0000","last_became_peered":"2023-10-19T14:02:10.688259+0000","last_unstale":"2023-10-19T14:02:10.689546+0000","last_undegraded":"2023-10-19T14:02:10.689546+0000","last_fullsized":"2023-10-19T14:02:10.689546+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:58:35.411824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1f","version":"187'4","reported_seq":15,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.696376+0000","last_change":"2023-10-19T14:02:09.668623+0000","last_active":"2023-10-19T14:02:10.696376+0000","last_peered":"2023-10-19T14:02:10.696376+0000","last_clean":"2023-10-19T14:02:10.696376+0000","last_became_active":"2023-10-19T14:02:09.668354+0000","last_became_peered":"2023-10-19T14:02:09.668354+0000","last_unstale":"2023-10-19T14:02:10.696376+0000","last_undegraded":"2023-10-19T14:02:10.696376+0000","last_fullsized":"2023-10-19T14:02:10.696376+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_clean_scrub_stamp":"2023-10-19T14:02:08.638576+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:57:35.643980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":11,"reported_epoch":188,"state":"active+clean","last_fresh":"2023-10-19T14:02:11.676570+0000","last_change":"2023-10-19T14:02:10.678806+0000","last_active":"2023-10-19T14:02:11.676570+0000","last_peered":"2023-10-19T14:02:11.676570+0000","last_clean":"2023-10-19T14:02:11.676570+0000","last_became_active":"2023-10-19T14:02:10.678408+0000","last_became_peered":"2023-10-19T14:02:10.678408+0000","last_unstale":"2023-10-19T14:02:11.676570+0000","last_undegraded":"2023-10-19T14:02:11.676570+0000","last_fullsized":"2023-10-19T14:02:11.676570+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:24:23.166190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1e","version":"187'1","reported_seq":12,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.694091+0000","last_change":"2023-10-19T14:02:09.670885+0000","last_active":"2023-10-19T14:02:10.694091+0000","last_peered":"2023-10-19T14:02:10.694091+0000","last_clean":"2023-10-19T14:02:10.694091+0000","last_became_active":"2023-10-19T14:02:09.670580+0000","last_became_peered":"2023-10-19T14:02:09.670580+0000","last_unstale":"2023-10-19T14:02:10.694091+0000","last_undegraded":"2023-10-19T14:02:10.694091+0000","last_fullsized":"2023-10-19T14:02:10.694091+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_clean_scrub_stamp":"2023-10-19T14:02:08.638576+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:51:10.920122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.675391+0000","last_change":"2023-10-19T14:02:10.670047+0000","last_active":"2023-10-19T14:02:10.675391+0000","last_peered":"2023-10-19T14:02:10.675391+0000","last_clean":"2023-10-19T14:02:10.675391+0000","last_became_active":"2023-10-19T14:02:10.669756+0000","last_became_peered":"2023-10-19T14:02:10.669756+0000","last_unstale":"2023-10-19T14:02:10.675391+0000","last_undegraded":"2023-10-19T14:02:10.675391+0000","last_fullsized":"2023-10-19T14:02:10.675391+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:24:21.948670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.1d","version":"187'2","reported_seq":14,"reported_epoch":188,"state":"active+clean","last_fresh":"2023-10-19T14:02:11.676647+0000","last_change":"2023-10-19T14:02:09.659670+0000","last_active":"2023-10-19T14:02:11.676647+0000","last_peered":"2023-10-19T14:02:11.676647+0000","last_clean":"2023-10-19T14:02:11.676647+0000","last_became_active":"2023-10-19T14:02:09.659397+0000","last_became_peered":"2023-10-19T14:02:09.659397+0000","last_unstale":"2023-10-19T14:02:11.676647+0000","last_undegraded":"2023-10-19T14:02:11.676647+0000","last_fullsized":"2023-10-19T14:02:11.676647+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_clean_scrub_stamp":"2023-10-19T14:02:08.638576+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:21:13.622150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.13","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.688379+0000","last_change":"2023-10-19T14:02:10.686240+0000","last_active":"2023-10-19T14:02:10.688379+0000","last_peered":"2023-10-19T14:02:10.688379+0000","last_clean":"2023-10-19T14:02:10.688379+0000","last_became_active":"2023-10-19T14:02:10.685908+0000","last_became_peered":"2023-10-19T14:02:10.685908+0000","last_unstale":"2023-10-19T14:02:10.688379+0000","last_undegraded":"2023-10-19T14:02:10.688379+0000","last_fullsized":"2023-10-19T14:02:10.688379+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:29:29.475164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1c","version":"187'2","reported_seq":14,"reported_epoch":188,"state":"active+clean","last_fresh":"2023-10-19T14:02:11.677513+0000","last_change":"2023-10-19T14:02:09.663273+0000","last_active":"2023-10-19T14:02:11.677513+0000","last_peered":"2023-10-19T14:02:11.677513+0000","last_clean":"2023-10-19T14:02:11.677513+0000","last_became_active":"2023-10-19T14:02:09.662981+0000","last_became_peered":"2023-10-19T14:02:09.662981+0000","last_unstale":"2023-10-19T14:02:11.677513+0000","last_undegraded":"2023-10-19T14:02:11.677513+0000","last_fullsized":"2023-10-19T14:02:11.677513+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_clean_scrub_stamp":"2023-10-19T14:02:08.638576+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:43:29.810615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.14","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.692347+0000","last_change":"2023-10-19T14:02:10.688031+0000","last_active":"2023-10-19T14:02:10.692347+0000","last_peered":"2023-10-19T14:02:10.692347+0000","last_clean":"2023-10-19T14:02:10.692347+0000","last_became_active":"2023-10-19T14:02:10.687232+0000","last_became_peered":"2023-10-19T14:02:10.687232+0000","last_unstale":"2023-10-19T14:02:10.692347+0000","last_undegraded":"2023-10-19T14:02:10.692347+0000","last_fullsized":"2023-10-19T14:02:10.692347+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:31:13.295905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1b","version":"187'1","reported_seq":12,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.692939+0000","last_change":"2023-10-19T14:02:09.669838+0000","last_active":"2023-10-19T14:02:10.692939+0000","last_peered":"2023-10-19T14:02:10.692939+0000","last_clean":"2023-10-19T14:02:10.692939+0000","last_became_active":"2023-10-19T14:02:09.669582+0000","last_became_peered":"2023-10-19T14:02:09.669582+0000","last_unstale":"2023-10-19T14:02:10.692939+0000","last_undegraded":"2023-10-19T14:02:10.692939+0000","last_fullsized":"2023-10-19T14:02:10.692939+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_clean_scrub_stamp":"2023-10-19T14:02:08.638576+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:53:55.532213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.15","version":"0'0","reported_seq":11,"reported_epoch":188,"state":"active+clean","last_fresh":"2023-10-19T14:02:11.677677+0000","last_change":"2023-10-19T14:02:10.678950+0000","last_active":"2023-10-19T14:02:11.677677+0000","last_peered":"2023-10-19T14:02:11.677677+0000","last_clean":"2023-10-19T14:02:11.677677+0000","last_became_active":"2023-10-19T14:02:10.678670+0000","last_became_peered":"2023-10-19T14:02:10.678670+0000","last_unstale":"2023-10-19T14:02:11.677677+0000","last_undegraded":"2023-10-19T14:02:11.677677+0000","last_fullsized":"2023-10-19T14:02:11.677677+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:38:14.577922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.1a","version":"187'1","reported_seq":12,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.696106+0000","last_change":"2023-10-19T14:02:09.666855+0000","last_active":"2023-10-19T14:02:10.696106+0000","last_peered":"2023-10-19T14:02:10.696106+0000","last_clean":"2023-10-19T14:02:10.696106+0000","last_became_active":"2023-10-19T14:02:09.666569+0000","last_became_peered":"2023-10-19T14:02:09.666569+0000","last_unstale":"2023-10-19T14:02:10.696106+0000","last_undegraded":"2023-10-19T14:02:10.696106+0000","last_fullsized":"2023-10-19T14:02:10.696106+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_clean_scrub_stamp":"2023-10-19T14:02:08.638576+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:00:51.177349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":11,"reported_epoch":188,"state":"active+clean","last_fresh":"2023-10-19T14:02:11.677728+0000","last_change":"2023-10-19T14:02:10.682862+0000","last_active":"2023-10-19T14:02:11.677728+0000","last_peered":"2023-10-19T14:02:11.677728+0000","last_clean":"2023-10-19T14:02:11.677728+0000","last_became_active":"2023-10-19T14:02:10.682558+0000","last_became_peered":"2023-10-19T14:02:10.682558+0000","last_unstale":"2023-10-19T14:02:11.677728+0000","last_undegraded":"2023-10-19T14:02:11.677728+0000","last_fullsized":"2023-10-19T14:02:11.677728+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:51:17.752537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2023-10-19T14:02:11.677355+0000","last_change":"2023-10-19T14:02:09.661271+0000","last_active":"2023-10-19T14:02:11.677355+0000","last_peered":"2023-10-19T14:02:11.677355+0000","last_clean":"2023-10-19T14:02:11.677355+0000","last_became_active":"2023-10-19T14:02:09.661012+0000","last_became_peered":"2023-10-19T14:02:09.661012+0000","last_unstale":"2023-10-19T14:02:11.677355+0000","last_undegraded":"2023-10-19T14:02:11.677355+0000","last_fullsized":"2023-10-19T14:02:11.677355+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_clean_scrub_stamp":"2023-10-19T14:02:08.638576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:38:46.272924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.685431+0000","last_change":"2023-10-19T14:02:10.681882+0000","last_active":"2023-10-19T14:02:10.685431+0000","last_peered":"2023-10-19T14:02:10.685431+0000","last_clean":"2023-10-19T14:02:10.685431+0000","last_became_active":"2023-10-19T14:02:10.681603+0000","last_became_peered":"2023-10-19T14:02:10.681603+0000","last_unstale":"2023-10-19T14:02:10.685431+0000","last_undegraded":"2023-10-19T14:02:10.685431+0000","last_fullsized":"2023-10-19T14:02:10.685431+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:40:09.919763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.672552+0000","last_change":"2023-10-19T14:02:09.663748+0000","last_active":"2023-10-19T14:02:10.672552+0000","last_peered":"2023-10-19T14:02:10.672552+0000","last_clean":"2023-10-19T14:02:10.672552+0000","last_became_active":"2023-10-19T14:02:09.663456+0000","last_became_peered":"2023-10-19T14:02:09.663456+0000","last_unstale":"2023-10-19T14:02:10.672552+0000","last_undegraded":"2023-10-19T14:02:10.672552+0000","last_fullsized":"2023-10-19T14:02:10.672552+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_clean_scrub_stamp":"2023-10-19T14:02:08.638576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:34:28.435490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.688744+0000","last_change":"2023-10-19T14:02:10.688225+0000","last_active":"2023-10-19T14:02:10.688744+0000","last_peered":"2023-10-19T14:02:10.688744+0000","last_clean":"2023-10-19T14:02:10.688744+0000","last_became_active":"2023-10-19T14:02:10.687857+0000","last_became_peered":"2023-10-19T14:02:10.687857+0000","last_unstale":"2023-10-19T14:02:10.688744+0000","last_undegraded":"2023-10-19T14:02:10.688744+0000","last_fullsized":"2023-10-19T14:02:10.688744+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:33:00.686451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2023-10-19T14:02:11.677005+0000","last_change":"2023-10-19T14:02:09.670452+0000","last_active":"2023-10-19T14:02:11.677005+0000","last_peered":"2023-10-19T14:02:11.677005+0000","last_clean":"2023-10-19T14:02:11.677005+0000","last_became_active":"2023-10-19T14:02:09.670133+0000","last_became_peered":"2023-10-19T14:02:09.670133+0000","last_unstale":"2023-10-19T14:02:11.677005+0000","last_undegraded":"2023-10-19T14:02:11.677005+0000","last_fullsized":"2023-10-19T14:02:11.677005+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_clean_scrub_stamp":"2023-10-19T14:02:08.638576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:57:58.962523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1a","version":"0'0","reported_seq":11,"reported_epoch":188,"state":"active+clean","last_fresh":"2023-10-19T14:02:11.676681+0000","last_change":"2023-10-19T14:02:10.677844+0000","last_active":"2023-10-19T14:02:11.676681+0000","last_peered":"2023-10-19T14:02:11.676681+0000","last_clean":"2023-10-19T14:02:11.676681+0000","last_became_active":"2023-10-19T14:02:10.677559+0000","last_became_peered":"2023-10-19T14:02:10.677559+0000","last_unstale":"2023-10-19T14:02:11.676681+0000","last_undegraded":"2023-10-19T14:02:11.676681+0000","last_fullsized":"2023-10-19T14:02:11.676681+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:12:49.139706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.15","version":"187'3","reported_seq":14,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.691713+0000","last_change":"2023-10-19T14:02:09.662732+0000","last_active":"2023-10-19T14:02:10.691713+0000","last_peered":"2023-10-19T14:02:10.691713+0000","last_clean":"2023-10-19T14:02:10.691713+0000","last_became_active":"2023-10-19T14:02:09.662431+0000","last_became_peered":"2023-10-19T14:02:09.662431+0000","last_unstale":"2023-10-19T14:02:10.691713+0000","last_undegraded":"2023-10-19T14:02:10.691713+0000","last_fullsized":"2023-10-19T14:02:10.691713+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_clean_scrub_stamp":"2023-10-19T14:02:08.638576+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:14:46.558909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1b","version":"0'0","reported_seq":11,"reported_epoch":188,"state":"active+clean","last_fresh":"2023-10-19T14:02:11.677539+0000","last_change":"2023-10-19T14:02:10.680074+0000","last_active":"2023-10-19T14:02:11.677539+0000","last_peered":"2023-10-19T14:02:11.677539+0000","last_clean":"2023-10-19T14:02:11.677539+0000","last_became_active":"2023-10-19T14:02:10.679791+0000","last_became_peered":"2023-10-19T14:02:10.679791+0000","last_unstale":"2023-10-19T14:02:11.677539+0000","last_undegraded":"2023-10-19T14:02:11.677539+0000","last_fullsized":"2023-10-19T14:02:11.677539+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:13:42.324779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.14","version":"187'6","reported_seq":18,"reported_epoch":188,"state":"active+clean","last_fresh":"2023-10-19T14:02:11.676960+0000","last_change":"2023-10-19T14:02:09.670227+0000","last_active":"2023-10-19T14:02:11.676960+0000","last_peered":"2023-10-19T14:02:11.676960+0000","last_clean":"2023-10-19T14:02:11.676960+0000","last_became_active":"2023-10-19T14:02:09.669940+0000","last_became_peered":"2023-10-19T14:02:09.669940+0000","last_unstale":"2023-10-19T14:02:11.676960+0000","last_undegraded":"2023-10-19T14:02:11.676960+0000","last_fullsized":"2023-10-19T14:02:11.676960+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_clean_scrub_stamp":"2023-10-19T14:02:08.638576+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:50:48.588866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1c","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.679964+0000","last_change":"2023-10-19T14:02:10.677767+0000","last_active":"2023-10-19T14:02:10.679964+0000","last_peered":"2023-10-19T14:02:10.679964+0000","last_clean":"2023-10-19T14:02:10.679964+0000","last_became_active":"2023-10-19T14:02:10.677505+0000","last_became_peered":"2023-10-19T14:02:10.677505+0000","last_unstale":"2023-10-19T14:02:10.679964+0000","last_undegraded":"2023-10-19T14:02:10.679964+0000","last_fullsized":"2023-10-19T14:02:10.679964+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:04:36.074025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.13","version":"187'2","reported_seq":13,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.694448+0000","last_change":"2023-10-19T14:02:09.665674+0000","last_active":"2023-10-19T14:02:10.694448+0000","last_peered":"2023-10-19T14:02:10.694448+0000","last_clean":"2023-10-19T14:02:10.694448+0000","last_became_active":"2023-10-19T14:02:09.664962+0000","last_became_peered":"2023-10-19T14:02:09.664962+0000","last_unstale":"2023-10-19T14:02:10.694448+0000","last_undegraded":"2023-10-19T14:02:10.694448+0000","last_fullsized":"2023-10-19T14:02:10.694448+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_clean_scrub_stamp":"2023-10-19T14:02:08.638576+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:11:07.265862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1d","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.690591+0000","last_change":"2023-10-19T14:02:10.686088+0000","last_active":"2023-10-19T14:02:10.690591+0000","last_peered":"2023-10-19T14:02:10.690591+0000","last_clean":"2023-10-19T14:02:10.690591+0000","last_became_active":"2023-10-19T14:02:10.684607+0000","last_became_peered":"2023-10-19T14:02:10.684607+0000","last_unstale":"2023-10-19T14:02:10.690591+0000","last_undegraded":"2023-10-19T14:02:10.690591+0000","last_fullsized":"2023-10-19T14:02:10.690591+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:47:08.467008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2023-10-19T14:02:11.676769+0000","last_change":"2023-10-19T14:02:09.664080+0000","last_active":"2023-10-19T14:02:11.676769+0000","last_peered":"2023-10-19T14:02:11.676769+0000","last_clean":"2023-10-19T14:02:11.676769+0000","last_became_active":"2023-10-19T14:02:09.663801+0000","last_became_peered":"2023-10-19T14:02:09.663801+0000","last_unstale":"2023-10-19T14:02:11.676769+0000","last_undegraded":"2023-10-19T14:02:11.676769+0000","last_fullsized":"2023-10-19T14:02:11.676769+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_clean_scrub_stamp":"2023-10-19T14:02:08.638576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:56:20.643608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1e","version":"0'0","reported_seq":11,"reported_epoch":188,"state":"active+clean","last_fresh":"2023-10-19T14:02:11.676569+0000","last_change":"2023-10-19T14:02:10.674303+0000","last_active":"2023-10-19T14:02:11.676569+0000","last_peered":"2023-10-19T14:02:11.676569+0000","last_clean":"2023-10-19T14:02:11.676569+0000","last_became_active":"2023-10-19T14:02:10.674038+0000","last_became_peered":"2023-10-19T14:02:10.674038+0000","last_unstale":"2023-10-19T14:02:11.676569+0000","last_undegraded":"2023-10-19T14:02:11.676569+0000","last_fullsized":"2023-10-19T14:02:11.676569+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:55:15.591450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2023-10-19T14:02:11.676228+0000","last_change":"2023-10-19T14:02:09.660406+0000","last_active":"2023-10-19T14:02:11.676228+0000","last_peered":"2023-10-19T14:02:11.676228+0000","last_clean":"2023-10-19T14:02:11.676228+0000","last_became_active":"2023-10-19T14:02:09.660096+0000","last_became_peered":"2023-10-19T14:02:09.660096+0000","last_unstale":"2023-10-19T14:02:11.676228+0000","last_undegraded":"2023-10-19T14:02:11.676228+0000","last_fullsized":"2023-10-19T14:02:11.676228+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_clean_scrub_stamp":"2023-10-19T14:02:08.638576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:32:59.681025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.1f","version":"0'0","reported_seq":11,"reported_epoch":188,"state":"active+clean","last_fresh":"2023-10-19T14:02:11.677770+0000","last_change":"2023-10-19T14:02:10.689906+0000","last_active":"2023-10-19T14:02:11.677770+0000","last_peered":"2023-10-19T14:02:11.677770+0000","last_clean":"2023-10-19T14:02:11.677770+0000","last_became_active":"2023-10-19T14:02:10.689462+0000","last_became_peered":"2023-10-19T14:02:10.689462+0000","last_unstale":"2023-10-19T14:02:11.677770+0000","last_undegraded":"2023-10-19T14:02:11.677770+0000","last_fullsized":"2023-10-19T14:02:11.677770+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:41:10.616800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.10","version":"187'8","reported_seq":19,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.695050+0000","last_change":"2023-10-19T14:02:09.663071+0000","last_active":"2023-10-19T14:02:10.695050+0000","last_peered":"2023-10-19T14:02:10.695050+0000","last_clean":"2023-10-19T14:02:10.695050+0000","last_became_active":"2023-10-19T14:02:09.662349+0000","last_became_peered":"2023-10-19T14:02:09.662349+0000","last_unstale":"2023-10-19T14:02:10.695050+0000","last_undegraded":"2023-10-19T14:02:10.695050+0000","last_fullsized":"2023-10-19T14:02:10.695050+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_clean_scrub_stamp":"2023-10-19T14:02:08.638576+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:03:52.274945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.680089+0000","last_change":"2023-10-19T14:02:10.677622+0000","last_active":"2023-10-19T14:02:10.680089+0000","last_peered":"2023-10-19T14:02:10.680089+0000","last_clean":"2023-10-19T14:02:10.680089+0000","last_became_active":"2023-10-19T14:02:10.677267+0000","last_became_peered":"2023-10-19T14:02:10.677267+0000","last_unstale":"2023-10-19T14:02:10.680089+0000","last_undegraded":"2023-10-19T14:02:10.680089+0000","last_fullsized":"2023-10-19T14:02:10.680089+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:38:12.275859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.f","version":"187'1","reported_seq":13,"reported_epoch":188,"state":"active+clean","last_fresh":"2023-10-19T14:02:11.677085+0000","last_change":"2023-10-19T14:02:09.665590+0000","last_active":"2023-10-19T14:02:11.677085+0000","last_peered":"2023-10-19T14:02:11.677085+0000","last_clean":"2023-10-19T14:02:11.677085+0000","last_became_active":"2023-10-19T14:02:09.665266+0000","last_became_peered":"2023-10-19T14:02:09.665266+0000","last_unstale":"2023-10-19T14:02:11.677085+0000","last_undegraded":"2023-10-19T14:02:11.677085+0000","last_fullsized":"2023-10-19T14:02:11.677085+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_clean_scrub_stamp":"2023-10-19T14:02:08.638576+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:59:47.216132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.679996+0000","last_change":"2023-10-19T14:02:10.678345+0000","last_active":"2023-10-19T14:02:10.679996+0000","last_peered":"2023-10-19T14:02:10.679996+0000","last_clean":"2023-10-19T14:02:10.679996+0000","last_became_active":"2023-10-19T14:02:10.678058+0000","last_became_peered":"2023-10-19T14:02:10.678058+0000","last_unstale":"2023-10-19T14:02:10.679996+0000","last_undegraded":"2023-10-19T14:02:10.679996+0000","last_fullsized":"2023-10-19T14:02:10.679996+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:08:54.228015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.e","version":"187'2","reported_seq":13,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.692093+0000","last_change":"2023-10-19T14:02:09.669029+0000","last_active":"2023-10-19T14:02:10.692093+0000","last_peered":"2023-10-19T14:02:10.692093+0000","last_clean":"2023-10-19T14:02:10.692093+0000","last_became_active":"2023-10-19T14:02:09.668456+0000","last_became_peered":"2023-10-19T14:02:09.668456+0000","last_unstale":"2023-10-19T14:02:10.692093+0000","last_undegraded":"2023-10-19T14:02:10.692093+0000","last_fullsized":"2023-10-19T14:02:10.692093+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_clean_scrub_stamp":"2023-10-19T14:02:08.638576+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:17:38.030117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":11,"reported_epoch":188,"state":"active+clean","last_fresh":"2023-10-19T14:02:11.676190+0000","last_change":"2023-10-19T14:02:10.684928+0000","last_active":"2023-10-19T14:02:11.676190+0000","last_peered":"2023-10-19T14:02:11.676190+0000","last_clean":"2023-10-19T14:02:11.676190+0000","last_became_active":"2023-10-19T14:02:10.684656+0000","last_became_peered":"2023-10-19T14:02:10.684656+0000","last_unstale":"2023-10-19T14:02:11.676190+0000","last_undegraded":"2023-10-19T14:02:11.676190+0000","last_fullsized":"2023-10-19T14:02:11.676190+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:47:44.902028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.d","version":"187'1","reported_seq":12,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.696670+0000","last_change":"2023-10-19T14:02:09.665974+0000","last_active":"2023-10-19T14:02:10.696670+0000","last_peered":"2023-10-19T14:02:10.696670+0000","last_clean":"2023-10-19T14:02:10.696670+0000","last_became_active":"2023-10-19T14:02:09.665007+0000","last_became_peered":"2023-10-19T14:02:09.665007+0000","last_unstale":"2023-10-19T14:02:10.696670+0000","last_undegraded":"2023-10-19T14:02:10.696670+0000","last_fullsized":"2023-10-19T14:02:10.696670+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_clean_scrub_stamp":"2023-10-19T14:02:08.638576+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:47:29.671420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":11,"reported_epoch":188,"state":"active+clean","last_fresh":"2023-10-19T14:02:11.676393+0000","last_change":"2023-10-19T14:02:10.666725+0000","last_active":"2023-10-19T14:02:11.676393+0000","last_peered":"2023-10-19T14:02:11.676393+0000","last_clean":"2023-10-19T14:02:11.676393+0000","last_became_active":"2023-10-19T14:02:10.666353+0000","last_became_peered":"2023-10-19T14:02:10.666353+0000","last_unstale":"2023-10-19T14:02:11.676393+0000","last_undegraded":"2023-10-19T14:02:11.676393+0000","last_fullsized":"2023-10-19T14:02:11.676393+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:14:45.157582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.672241+0000","last_change":"2023-10-19T14:02:09.671519+0000","last_active":"2023-10-19T14:02:10.672241+0000","last_peered":"2023-10-19T14:02:10.672241+0000","last_clean":"2023-10-19T14:02:10.672241+0000","last_became_active":"2023-10-19T14:02:09.671226+0000","last_became_peered":"2023-10-19T14:02:09.671226+0000","last_unstale":"2023-10-19T14:02:10.672241+0000","last_undegraded":"2023-10-19T14:02:10.672241+0000","last_fullsized":"2023-10-19T14:02:10.672241+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_clean_scrub_stamp":"2023-10-19T14:02:08.638576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:27:43.363970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":11,"reported_epoch":188,"state":"active+clean","last_fresh":"2023-10-19T14:02:11.676947+0000","last_change":"2023-10-19T14:02:10.694215+0000","last_active":"2023-10-19T14:02:11.676947+0000","last_peered":"2023-10-19T14:02:11.676947+0000","last_clean":"2023-10-19T14:02:11.676947+0000","last_became_active":"2023-10-19T14:02:10.693907+0000","last_became_peered":"2023-10-19T14:02:10.693907+0000","last_unstale":"2023-10-19T14:02:11.676947+0000","last_undegraded":"2023-10-19T14:02:11.676947+0000","last_fullsized":"2023-10-19T14:02:11.676947+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:08:43.952742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.b","version":"187'2","reported_seq":13,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.693463+0000","last_change":"2023-10-19T14:02:09.662023+0000","last_active":"2023-10-19T14:02:10.693463+0000","last_peered":"2023-10-19T14:02:10.693463+0000","last_clean":"2023-10-19T14:02:10.693463+0000","last_became_active":"2023-10-19T14:02:09.661766+0000","last_became_peered":"2023-10-19T14:02:09.661766+0000","last_unstale":"2023-10-19T14:02:10.693463+0000","last_undegraded":"2023-10-19T14:02:10.693463+0000","last_fullsized":"2023-10-19T14:02:10.693463+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_clean_scrub_stamp":"2023-10-19T14:02:08.638576+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:29:14.114099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.687100+0000","last_change":"2023-10-19T14:02:10.681993+0000","last_active":"2023-10-19T14:02:10.687100+0000","last_peered":"2023-10-19T14:02:10.687100+0000","last_clean":"2023-10-19T14:02:10.687100+0000","last_became_active":"2023-10-19T14:02:10.681604+0000","last_became_peered":"2023-10-19T14:02:10.681604+0000","last_unstale":"2023-10-19T14:02:10.687100+0000","last_undegraded":"2023-10-19T14:02:10.687100+0000","last_fullsized":"2023-10-19T14:02:10.687100+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:48:47.283300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2023-10-19T14:02:11.676225+0000","last_change":"2023-10-19T14:02:09.670957+0000","last_active":"2023-10-19T14:02:11.676225+0000","last_peered":"2023-10-19T14:02:11.676225+0000","last_clean":"2023-10-19T14:02:11.676225+0000","last_became_active":"2023-10-19T14:02:09.670658+0000","last_became_peered":"2023-10-19T14:02:09.670658+0000","last_unstale":"2023-10-19T14:02:11.676225+0000","last_undegraded":"2023-10-19T14:02:11.676225+0000","last_fullsized":"2023-10-19T14:02:11.676225+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_clean_scrub_stamp":"2023-10-19T14:02:08.638576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:11:04.524319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.6","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.687060+0000","last_change":"2023-10-19T14:02:10.681236+0000","last_active":"2023-10-19T14:02:10.687060+0000","last_peered":"2023-10-19T14:02:10.687060+0000","last_clean":"2023-10-19T14:02:10.687060+0000","last_became_active":"2023-10-19T14:02:10.680948+0000","last_became_peered":"2023-10-19T14:02:10.680948+0000","last_unstale":"2023-10-19T14:02:10.687060+0000","last_undegraded":"2023-10-19T14:02:10.687060+0000","last_fullsized":"2023-10-19T14:02:10.687060+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:53:43.428321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2023-10-19T14:02:11.677433+0000","last_change":"2023-10-19T14:02:09.659752+0000","last_active":"2023-10-19T14:02:11.677433+0000","last_peered":"2023-10-19T14:02:11.677433+0000","last_clean":"2023-10-19T14:02:11.677433+0000","last_became_active":"2023-10-19T14:02:09.659434+0000","last_became_peered":"2023-10-19T14:02:09.659434+0000","last_unstale":"2023-10-19T14:02:11.677433+0000","last_undegraded":"2023-10-19T14:02:11.677433+0000","last_fullsized":"2023-10-19T14:02:11.677433+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_clean_scrub_stamp":"2023-10-19T14:02:08.638576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:47:19.668936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.7","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.683831+0000","last_change":"2023-10-19T14:02:10.682388+0000","last_active":"2023-10-19T14:02:10.683831+0000","last_peered":"2023-10-19T14:02:10.683831+0000","last_clean":"2023-10-19T14:02:10.683831+0000","last_became_active":"2023-10-19T14:02:10.682102+0000","last_became_peered":"2023-10-19T14:02:10.682102+0000","last_unstale":"2023-10-19T14:02:10.683831+0000","last_undegraded":"2023-10-19T14:02:10.683831+0000","last_fullsized":"2023-10-19T14:02:10.683831+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:22:58.263925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.670151+0000","last_change":"2023-10-19T14:02:09.665481+0000","last_active":"2023-10-19T14:02:10.670151+0000","last_peered":"2023-10-19T14:02:10.670151+0000","last_clean":"2023-10-19T14:02:10.670151+0000","last_became_active":"2023-10-19T14:02:09.665241+0000","last_became_peered":"2023-10-19T14:02:09.665241+0000","last_unstale":"2023-10-19T14:02:10.670151+0000","last_undegraded":"2023-10-19T14:02:10.670151+0000","last_fullsized":"2023-10-19T14:02:10.670151+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_clean_scrub_stamp":"2023-10-19T14:02:08.638576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:21:13.622150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":11,"reported_epoch":188,"state":"active+clean","last_fresh":"2023-10-19T14:02:11.676118+0000","last_change":"2023-10-19T14:02:10.671405+0000","last_active":"2023-10-19T14:02:11.676118+0000","last_peered":"2023-10-19T14:02:11.676118+0000","last_clean":"2023-10-19T14:02:11.676118+0000","last_became_active":"2023-10-19T14:02:10.671121+0000","last_became_peered":"2023-10-19T14:02:10.671121+0000","last_unstale":"2023-10-19T14:02:11.676118+0000","last_undegraded":"2023-10-19T14:02:11.676118+0000","last_fullsized":"2023-10-19T14:02:11.676118+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:58:35.411824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.7","version":"187'2","reported_seq":13,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.692855+0000","last_change":"2023-10-19T14:02:09.658010+0000","last_active":"2023-10-19T14:02:10.692855+0000","last_peered":"2023-10-19T14:02:10.692855+0000","last_clean":"2023-10-19T14:02:10.692855+0000","last_became_active":"2023-10-19T14:02:09.657682+0000","last_became_peered":"2023-10-19T14:02:09.657682+0000","last_unstale":"2023-10-19T14:02:10.692855+0000","last_undegraded":"2023-10-19T14:02:10.692855+0000","last_fullsized":"2023-10-19T14:02:10.692855+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_clean_scrub_stamp":"2023-10-19T14:02:08.638576+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:17:55.842897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.681988+0000","last_change":"2023-10-19T14:02:10.680661+0000","last_active":"2023-10-19T14:02:10.681988+0000","last_peered":"2023-10-19T14:02:10.681988+0000","last_clean":"2023-10-19T14:02:10.681988+0000","last_became_active":"2023-10-19T14:02:10.680407+0000","last_became_peered":"2023-10-19T14:02:10.680407+0000","last_unstale":"2023-10-19T14:02:10.681988+0000","last_undegraded":"2023-10-19T14:02:10.681988+0000","last_fullsized":"2023-10-19T14:02:10.681988+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:15:39.858994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.6","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2023-10-19T14:02:11.676860+0000","last_change":"2023-10-19T14:02:09.670892+0000","last_active":"2023-10-19T14:02:11.676860+0000","last_peered":"2023-10-19T14:02:11.676860+0000","last_clean":"2023-10-19T14:02:11.676860+0000","last_became_active":"2023-10-19T14:02:09.670599+0000","last_became_peered":"2023-10-19T14:02:09.670599+0000","last_unstale":"2023-10-19T14:02:11.676860+0000","last_undegraded":"2023-10-19T14:02:11.676860+0000","last_fullsized":"2023-10-19T14:02:11.676860+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_clean_scrub_stamp":"2023-10-19T14:02:08.638576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:50:35.829632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":11,"reported_epoch":188,"state":"active+clean","last_fresh":"2023-10-19T14:02:11.677464+0000","last_change":"2023-10-19T14:02:10.671805+0000","last_active":"2023-10-19T14:02:11.677464+0000","last_peered":"2023-10-19T14:02:11.677464+0000","last_clean":"2023-10-19T14:02:11.677464+0000","last_became_active":"2023-10-19T14:02:10.671534+0000","last_became_peered":"2023-10-19T14:02:10.671534+0000","last_unstale":"2023-10-19T14:02:11.677464+0000","last_undegraded":"2023-10-19T14:02:11.677464+0000","last_fullsized":"2023-10-19T14:02:11.677464+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:18:19.596195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2023-10-19T14:02:11.676842+0000","last_change":"2023-10-19T14:02:09.657923+0000","last_active":"2023-10-19T14:02:11.676842+0000","last_peered":"2023-10-19T14:02:11.676842+0000","last_clean":"2023-10-19T14:02:11.676842+0000","last_became_active":"2023-10-19T14:02:09.657637+0000","last_became_peered":"2023-10-19T14:02:09.657637+0000","last_unstale":"2023-10-19T14:02:11.676842+0000","last_undegraded":"2023-10-19T14:02:11.676842+0000","last_fullsized":"2023-10-19T14:02:11.676842+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_clean_scrub_stamp":"2023-10-19T14:02:08.638576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:02:00.400790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.b","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.681682+0000","last_change":"2023-10-19T14:02:10.678147+0000","last_active":"2023-10-19T14:02:10.681682+0000","last_peered":"2023-10-19T14:02:10.681682+0000","last_clean":"2023-10-19T14:02:10.681682+0000","last_became_active":"2023-10-19T14:02:10.677778+0000","last_became_peered":"2023-10-19T14:02:10.677778+0000","last_unstale":"2023-10-19T14:02:10.681682+0000","last_undegraded":"2023-10-19T14:02:10.681682+0000","last_fullsized":"2023-10-19T14:02:10.681682+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:42:33.481800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2023-10-19T14:02:11.677623+0000","last_change":"2023-10-19T14:02:09.663832+0000","last_active":"2023-10-19T14:02:11.677623+0000","last_peered":"2023-10-19T14:02:11.677623+0000","last_clean":"2023-10-19T14:02:11.677623+0000","last_became_active":"2023-10-19T14:02:09.663232+0000","last_became_peered":"2023-10-19T14:02:09.663232+0000","last_unstale":"2023-10-19T14:02:11.677623+0000","last_undegraded":"2023-10-19T14:02:11.677623+0000","last_fullsized":"2023-10-19T14:02:11.677623+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_clean_scrub_stamp":"2023-10-19T14:02:08.638576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:00:54.874154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.685383+0000","last_change":"2023-10-19T14:02:10.681779+0000","last_active":"2023-10-19T14:02:10.685383+0000","last_peered":"2023-10-19T14:02:10.685383+0000","last_clean":"2023-10-19T14:02:10.685383+0000","last_became_active":"2023-10-19T14:02:10.681508+0000","last_became_peered":"2023-10-19T14:02:10.681508+0000","last_unstale":"2023-10-19T14:02:10.685383+0000","last_undegraded":"2023-10-19T14:02:10.685383+0000","last_fullsized":"2023-10-19T14:02:10.685383+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:18:55.719249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.672729+0000","last_change":"2023-10-19T14:02:09.660687+0000","last_active":"2023-10-19T14:02:10.672729+0000","last_peered":"2023-10-19T14:02:10.672729+0000","last_clean":"2023-10-19T14:02:10.672729+0000","last_became_active":"2023-10-19T14:02:09.660428+0000","last_became_peered":"2023-10-19T14:02:09.660428+0000","last_unstale":"2023-10-19T14:02:10.672729+0000","last_undegraded":"2023-10-19T14:02:10.672729+0000","last_fullsized":"2023-10-19T14:02:10.672729+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_clean_scrub_stamp":"2023-10-19T14:02:08.638576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:06:06.545431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.d","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.689416+0000","last_change":"2023-10-19T14:02:10.688510+0000","last_active":"2023-10-19T14:02:10.689416+0000","last_peered":"2023-10-19T14:02:10.689416+0000","last_clean":"2023-10-19T14:02:10.689416+0000","last_became_active":"2023-10-19T14:02:10.687717+0000","last_became_peered":"2023-10-19T14:02:10.687717+0000","last_unstale":"2023-10-19T14:02:10.689416+0000","last_undegraded":"2023-10-19T14:02:10.689416+0000","last_fullsized":"2023-10-19T14:02:10.689416+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:32:43.725034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2023-10-19T14:02:11.676858+0000","last_change":"2023-10-19T14:02:09.669522+0000","last_active":"2023-10-19T14:02:11.676858+0000","last_peered":"2023-10-19T14:02:11.676858+0000","last_clean":"2023-10-19T14:02:11.676858+0000","last_became_active":"2023-10-19T14:02:09.669110+0000","last_became_peered":"2023-10-19T14:02:09.669110+0000","last_unstale":"2023-10-19T14:02:11.676858+0000","last_undegraded":"2023-10-19T14:02:11.676858+0000","last_fullsized":"2023-10-19T14:02:11.676858+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_clean_scrub_stamp":"2023-10-19T14:02:08.638576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:03:02.136067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.e","version":"0'0","reported_seq":11,"reported_epoch":188,"state":"active+clean","last_fresh":"2023-10-19T14:02:11.676947+0000","last_change":"2023-10-19T14:02:10.694078+0000","last_active":"2023-10-19T14:02:11.676947+0000","last_peered":"2023-10-19T14:02:11.676947+0000","last_clean":"2023-10-19T14:02:11.676947+0000","last_became_active":"2023-10-19T14:02:10.693764+0000","last_became_peered":"2023-10-19T14:02:10.693764+0000","last_unstale":"2023-10-19T14:02:11.676947+0000","last_undegraded":"2023-10-19T14:02:11.676947+0000","last_fullsized":"2023-10-19T14:02:11.676947+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:25:36.235834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2023-10-19T14:02:11.676322+0000","last_change":"2023-10-19T14:02:09.661342+0000","last_active":"2023-10-19T14:02:11.676322+0000","last_peered":"2023-10-19T14:02:11.676322+0000","last_clean":"2023-10-19T14:02:11.676322+0000","last_became_active":"2023-10-19T14:02:09.661084+0000","last_became_peered":"2023-10-19T14:02:09.661084+0000","last_unstale":"2023-10-19T14:02:11.676322+0000","last_undegraded":"2023-10-19T14:02:11.676322+0000","last_fullsized":"2023-10-19T14:02:11.676322+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_clean_scrub_stamp":"2023-10-19T14:02:08.638576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:32:14.505061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.f","version":"0'0","reported_seq":11,"reported_epoch":188,"state":"active+clean","last_fresh":"2023-10-19T14:02:11.677804+0000","last_change":"2023-10-19T14:02:10.676495+0000","last_active":"2023-10-19T14:02:11.677804+0000","last_peered":"2023-10-19T14:02:11.677804+0000","last_clean":"2023-10-19T14:02:11.677804+0000","last_became_active":"2023-10-19T14:02:10.676222+0000","last_became_peered":"2023-10-19T14:02:10.676222+0000","last_unstale":"2023-10-19T14:02:11.677804+0000","last_undegraded":"2023-10-19T14:02:11.677804+0000","last_fullsized":"2023-10-19T14:02:11.677804+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:02:10.131510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":11,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.662928+0000","last_change":"2023-10-19T14:02:09.661418+0000","last_active":"2023-10-19T14:02:10.662928+0000","last_peered":"2023-10-19T14:02:10.662928+0000","last_clean":"2023-10-19T14:02:10.662928+0000","last_became_active":"2023-10-19T14:02:09.661138+0000","last_became_peered":"2023-10-19T14:02:09.661138+0000","last_unstale":"2023-10-19T14:02:10.662928+0000","last_undegraded":"2023-10-19T14:02:10.662928+0000","last_fullsized":"2023-10-19T14:02:10.662928+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_clean_scrub_stamp":"2023-10-19T14:02:08.638576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:49:37.825794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":9,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-19T14:02:10.680999+0000","last_change":"2023-10-19T14:02:10.679293+0000","last_active":"2023-10-19T14:02:10.680999+0000","last_peered":"2023-10-19T14:02:10.680999+0000","last_clean":"2023-10-19T14:02:10.680999+0000","last_became_active":"2023-10-19T14:02:10.679040+0000","last_became_peered":"2023-10-19T14:02:10.679040+0000","last_unstale":"2023-10-19T14:02:10.680999+0000","last_undegraded":"2023-10-19T14:02:10.680999+0000","last_fullsized":"2023-10-19T14:02:10.680999+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:09.644002+0000","last_clean_scrub_stamp":"2023-10-19T14:02:09.644002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T02:00:25.823282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2023-10-19T14:02:11.676768+0000","last_change":"2023-10-19T14:02:09.671482+0000","last_active":"2023-10-19T14:02:11.676768+0000","last_peered":"2023-10-19T14:02:11.676768+0000","last_clean":"2023-10-19T14:02:11.676768+0000","last_became_active":"2023-10-19T14:02:09.671225+0000","last_became_peered":"2023-10-19T14:02:09.671225+0000","last_unstale":"2023-10-19T14:02:11.676768+0000","last_undegraded":"2023-10-19T14:02:11.676768+0000","last_fullsized":"2023-10-19T14:02:11.676768+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:02:08.638576+0000","last_clean_scrub_stamp":"2023-10-19T14:02:08.638576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:05:14.796788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"177'3131","reported_seq":3848,"reported_epoch":186,"state":"active+clean","last_fresh":"2023-10-19T14:02:09.654340+0000","last_change":"2023-10-19T12:44:08.636613+0000","last_active":"2023-10-19T14:02:09.654340+0000","last_peered":"2023-10-19T14:02:09.654340+0000","last_clean":"2023-10-19T14:02:09.654340+0000","last_became_active":"2023-10-19T12:44:08.635697+0000","last_became_peered":"2023-10-19T12:44:08.635697+0000","last_unstale":"2023-10-19T14:02:09.654340+0000","last_undegraded":"2023-10-19T14:02:09.654340+0000","last_fullsized":"2023-10-19T14:02:09.654340+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_clean_scrub_stamp":"2023-10-19T12:44:07.399738+0000","objects_scrubbed":0,"log_size":3131,"log_dups_size":0,"ondisk_log_size":3131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:15:23.685485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1730,"num_read_kb":3594,"num_write":489,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":24,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":23,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1730,"num_read_kb":3594,"num_write":489,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3131,"ondisk_log_size":3131,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575786,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60416,"kb_used_data":2868,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311424,"statfs":{"total":96636764160,"available":96574898176,"internally_reserved":0,"allocated":2936832,"data_stored":1779468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58915275},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:01:29 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.59099999999999997,"15min":0.59699999999999998},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.79900000000000004,"5min":0.85099999999999998,"15min":1.026},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.34699999999999998,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.83099999999999996,"5min":0.84099999999999997,"15min":1},"last":0.64200000000000002}]},{"osd":1,"last update":"Thu Oct 19 14:01:29 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.59899999999999998,"15min":0.60099999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.873},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.60899999999999999,"15min":0.61199999999999999},"min":{"1min":0.46000000000000002,"5min":0.41999999999999998,"15min":0.376},"max":{"1min":0.84499999999999997,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.54400000000000004}]},{"osd":2,"last update":"Thu Oct 19 14:01:29 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.65900000000000003,"15min":0.65500000000000003},"min":{"1min":0.44500000000000001,"5min":0.44400000000000001,"15min":0.39000000000000001},"max":{"1min":0.875,"5min":0.877,"15min":0.96599999999999997},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.65700000000000003,"15min":0.66000000000000003},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.36799999999999999},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.70499999999999996}]},{"osd":3,"last update":"Thu Oct 19 14:01:29 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.64300000000000002,"15min":0.64400000000000002},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.76800000000000002,"5min":0.86299999999999999,"15min":1.0129999999999999},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.69499999999999995,"15min":0.69199999999999995},"min":{"1min":0.504,"5min":0.504,"15min":0.48199999999999998},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.92000000000000004},"last":0.71899999999999997}]},{"osd":4,"last update":"Thu Oct 19 14:01:29 2023","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.48099999999999998,"15min":0.46600000000000003},"min":{"1min":0.32000000000000001,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.71799999999999997,"5min":0.73099999999999998,"15min":0.78500000000000003},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.503,"5min":0.51500000000000001,"15min":0.48799999999999999},"min":{"1min":0.35499999999999998,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.69399999999999995,"5min":0.78200000000000003,"15min":0.80900000000000005},"last":0.76800000000000002}]},{"osd":5,"last update":"Thu Oct 19 14:01:29 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.51000000000000001,"15min":0.49299999999999999},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.72899999999999998,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.624},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55400000000000005,"15min":0.53100000000000003},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.33900000000000002},"max":{"1min":0.77900000000000003,"5min":0.82699999999999996,"15min":0.86399999999999999},"last":0.78200000000000003}]},{"osd":6,"last update":"Thu Oct 19 14:01:29 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.58999999999999997,"15min":0.56899999999999995},"min":{"1min":0.38500000000000001,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.81100000000000005,"5min":0.83299999999999996,"15min":0.85799999999999998},"last":0.75},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.57399999999999995,"15min":0.55500000000000005},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.33200000000000002},"max":{"1min":0.78000000000000003,"5min":0.81000000000000005,"15min":0.94599999999999995},"last":0.58799999999999997}]}]},{"osd":6,"up_from":13,"seq":55834575786,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37632,"kb_used_data":2288,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334208,"statfs":{"total":96636764160,"available":96598228992,"internally_reserved":0,"allocated":2342912,"data_stored":1189100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:01:30 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.60599999999999998,"15min":0.60799999999999998},"min":{"1min":0.37,"5min":0.37,"15min":0.34599999999999997},"max":{"1min":0.82299999999999995,"5min":0.88100000000000001,"15min":1.3280000000000001},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61699999999999999,"15min":0.61399999999999999},"min":{"1min":0.46500000000000002,"5min":0.46400000000000002,"15min":0.26100000000000001},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":1.4279999999999999},"last":0.63300000000000001}]},{"osd":1,"last update":"Thu Oct 19 14:01:30 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.62,"15min":0.622},"min":{"1min":0.434,"5min":0.42799999999999999,"15min":0.35899999999999999},"max":{"1min":0.78400000000000003,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.624,"15min":0.61799999999999999},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.29299999999999998},"max":{"1min":0.83599999999999997,"5min":0.85399999999999998,"15min":0.89200000000000002},"last":0.67700000000000005}]},{"osd":2,"last update":"Thu Oct 19 14:01:30 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.38200000000000001},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":1.4650000000000001},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.621,"5min":0.65000000000000002,"15min":0.63200000000000001},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.35199999999999998},"max":{"1min":0.82399999999999995,"5min":0.85699999999999998,"15min":1.3979999999999999},"last":0.61499999999999999}]},{"osd":3,"last update":"Thu Oct 19 14:01:30 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.64700000000000002,"15min":0.65300000000000002},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.76600000000000001,"5min":0.82899999999999996,"15min":1.496},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.66900000000000004,"15min":0.66500000000000004},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.38100000000000001},"max":{"1min":0.77500000000000002,"5min":0.83499999999999996,"15min":1.512},"last":0.74299999999999999}]},{"osd":4,"last update":"Thu Oct 19 14:01:30 2023","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.48499999999999999,"15min":0.47999999999999998},"min":{"1min":0.316,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.51600000000000001,"15min":0.50600000000000001},"min":{"1min":0.36699999999999999,"5min":0.35299999999999998,"15min":0.32800000000000001},"max":{"1min":0.79800000000000004,"5min":0.85699999999999998,"15min":0.93000000000000005},"last":0.38500000000000001}]},{"osd":5,"last update":"Thu Oct 19 14:01:30 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.5,"15min":0.49299999999999999},"min":{"1min":0.35499999999999998,"5min":0.35099999999999998,"15min":0.33700000000000002},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54200000000000004},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.78800000000000003,"5min":0.83199999999999996,"15min":1.0249999999999999},"last":0.82699999999999996}]},{"osd":7,"last update":"Thu Oct 19 14:01:30 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.54200000000000004,"15min":0.54500000000000004},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.32200000000000001},"max":{"1min":0.81999999999999995,"5min":0.84499999999999997,"15min":0.878},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56399999999999995,"15min":0.55500000000000005},"min":{"1min":0.40500000000000003,"5min":0.373,"15min":0.33400000000000002},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.91100000000000003},"last":0.65400000000000003}]}]},{"osd":5,"up_from":13,"seq":55834575786,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59852,"kb_used_data":2296,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311988,"statfs":{"total":96636764160,"available":96575475712,"internally_reserved":0,"allocated":2351104,"data_stored":1189224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:01:10 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.59399999999999997,"15min":0.61699999999999999},"min":{"1min":0.379,"5min":0.35399999999999998,"15min":0.34499999999999997},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":1.2629999999999999},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.63700000000000001,"15min":0.63300000000000001},"min":{"1min":0.50700000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.88300000000000001,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.71799999999999997}]},{"osd":1,"last update":"Thu Oct 19 14:01:10 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.63700000000000001,"15min":0.63600000000000001},"min":{"1min":0.51700000000000002,"5min":0.435,"15min":0.41699999999999998},"max":{"1min":0.81100000000000005,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.63800000000000001,"15min":0.64000000000000001},"min":{"1min":0.495,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.84899999999999998,"5min":0.87,"15min":1.071},"last":0.64600000000000002}]},{"osd":2,"last update":"Thu Oct 19 14:01:10 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61899999999999999,"15min":0.628},"min":{"1min":0.46800000000000003,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.83499999999999996,"5min":0.875,"15min":1.044},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.64500000000000002,"15min":0.65300000000000002},"min":{"1min":0.48299999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.81399999999999995,"5min":0.90000000000000002,"15min":1.171},"last":0.67500000000000004}]},{"osd":3,"last update":"Thu Oct 19 14:01:10 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.63500000000000001,"15min":0.64400000000000002},"min":{"1min":0.46600000000000003,"5min":0.44,"15min":0.44},"max":{"1min":0.77800000000000002,"5min":0.85899999999999999,"15min":1.1910000000000001},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.67100000000000004,"15min":0.67500000000000004},"min":{"1min":0.52200000000000002,"5min":0.45500000000000002,"15min":0.43099999999999999},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":1.2210000000000001},"last":0.501}]},{"osd":4,"last update":"Thu Oct 19 14:01:10 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.5,"15min":0.48499999999999999},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.65100000000000002,"5min":0.745,"15min":0.75700000000000001},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.501,"5min":0.5,"15min":0.496},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.73899999999999999,"5min":0.76200000000000001,"15min":0.77300000000000002},"last":0.60699999999999998}]},{"osd":6,"last update":"Thu Oct 19 14:01:10 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.48899999999999999,"15min":0.48999999999999999},"min":{"1min":0.32600000000000001,"5min":0.32100000000000001,"15min":0.29699999999999999},"max":{"1min":0.64900000000000002,"5min":0.80000000000000004,"15min":0.83599999999999997},"last":0.746},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.53000000000000003,"15min":0.53100000000000003},"min":{"1min":0.38900000000000001,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.80000000000000004},"last":0.70299999999999996}]},{"osd":7,"last update":"Thu Oct 19 14:01:10 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.51500000000000001,"15min":0.51200000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.68000000000000005,"5min":0.79200000000000004,"15min":0.80000000000000004},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.56999999999999995,"15min":0.56299999999999994},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.80100000000000005,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.65800000000000003}]}]},{"osd":4,"up_from":13,"seq":55834575786,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41812,"kb_used_data":2304,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330028,"statfs":{"total":96636764160,"available":96593948672,"internally_reserved":0,"allocated":2359296,"data_stored":1189864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:01:45 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.57099999999999995,"15min":0.59099999999999997},"min":{"1min":0.437,"5min":0.38500000000000001,"15min":0.38400000000000001},"max":{"1min":0.76700000000000002,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.58399999999999996,"15min":0.59299999999999997},"min":{"1min":0.39400000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.70399999999999996,"5min":0.83199999999999996,"15min":0.98199999999999998},"last":0.77100000000000002}]},{"osd":1,"last update":"Thu Oct 19 14:01:45 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.621,"15min":0.64200000000000002},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.78600000000000003,"5min":0.88300000000000001,"15min":0.92100000000000004},"last":0.626},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.64500000000000002,"15min":0.65500000000000003},"min":{"1min":0.44500000000000001,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.79700000000000004,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.54100000000000004}]},{"osd":2,"last update":"Thu Oct 19 14:01:45 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.64300000000000002,"15min":0.64800000000000002},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.76900000000000002,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.66100000000000003,"15min":0.66200000000000003},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.751,"5min":1.054,"15min":1.054},"last":0.58499999999999996}]},{"osd":3,"last update":"Thu Oct 19 14:01:45 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.628,"15min":0.63700000000000001},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.72099999999999997,"5min":0.92000000000000004,"15min":0.93600000000000005},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.67200000000000004,"15min":0.67000000000000004},"min":{"1min":0.45600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":0.80700000000000005,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.56899999999999995}]},{"osd":5,"last update":"Thu Oct 19 14:01:45 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.47299999999999998,"15min":0.45800000000000002},"min":{"1min":0.318,"5min":0.318,"15min":0.29399999999999998},"max":{"1min":0.68300000000000005,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.496,"15min":0.48099999999999998},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.64300000000000002,"5min":0.77300000000000002,"15min":0.80900000000000005},"last":0.61199999999999999}]},{"osd":6,"last update":"Thu Oct 19 14:01:45 2023","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.49199999999999999,"15min":0.47299999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.72399999999999998,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.505,"5min":0.53400000000000003,"15min":0.52100000000000002},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.73799999999999999,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.67300000000000004}]},{"osd":7,"last update":"Thu Oct 19 14:01:45 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.53900000000000003,"15min":0.52600000000000002},"min":{"1min":0.34200000000000003,"5min":0.33800000000000002,"15min":0.32200000000000001},"max":{"1min":0.70499999999999996,"5min":0.79600000000000004,"15min":0.83699999999999997},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.61199999999999999,"15min":0.58499999999999996},"min":{"1min":0.40899999999999997,"5min":0.39500000000000002,"15min":0.39200000000000002},"max":{"1min":0.79100000000000004,"5min":1.02,"15min":1.02},"last":0.71299999999999997}]}]},{"osd":3,"up_from":13,"seq":55834575785,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51544,"kb_used_data":2308,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320296,"statfs":{"total":96636764160,"available":96583983104,"internally_reserved":0,"allocated":2363392,"data_stored":1191316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50395592},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:02:09 2023","interfaces":[{"interface":"back","average":{"1min":0.33200000000000002,"5min":0.39700000000000002,"15min":0.40100000000000002},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.51000000000000001,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.34699999999999998,"5min":0.42299999999999999,"15min":0.42899999999999999},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.45000000000000001,"5min":0.66600000000000004,"15min":0.69799999999999995},"last":0.32300000000000001}]},{"osd":1,"last update":"Thu Oct 19 14:02:09 2023","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.40100000000000002,"15min":0.42299999999999999},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.46600000000000003,"5min":0.58999999999999997,"15min":0.72999999999999998},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.379,"5min":0.45400000000000001,"15min":0.46000000000000002},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.56100000000000005,"5min":0.72599999999999998,"15min":0.72699999999999998},"last":0.308}]},{"osd":2,"last update":"Thu Oct 19 14:02:09 2023","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.46999999999999997,"15min":0.48099999999999998},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.53600000000000003,"5min":0.70399999999999996,"15min":0.72099999999999997},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.48999999999999999,"15min":0.499},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.56599999999999995,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.34599999999999997}]},{"osd":4,"last update":"Thu Oct 19 14:02:09 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.68700000000000006,"15min":0.69199999999999995},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.82499999999999996,"5min":1.3600000000000001,"15min":1.3600000000000001},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.68799999999999994,"15min":0.70699999999999996},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.82599999999999996,"5min":1.0209999999999999,"15min":1.2410000000000001},"last":0.629}]},{"osd":5,"last update":"Thu Oct 19 14:02:09 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.66400000000000003,"15min":0.67800000000000005},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.39700000000000002},"max":{"1min":0.85499999999999998,"5min":1.2450000000000001,"15min":1.2450000000000001},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.65400000000000003,"15min":0.66700000000000004},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.73299999999999998,"5min":1.385,"15min":1.385},"last":0.72499999999999998}]},{"osd":6,"last update":"Thu Oct 19 14:02:09 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.66800000000000004,"15min":0.68000000000000005},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.749,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.67300000000000004,"15min":0.69399999999999995},"min":{"1min":0.435,"5min":0.435,"15min":0.42799999999999999},"max":{"1min":0.78500000000000003,"5min":1.0649999999999999,"15min":1.282},"last":0.59699999999999998}]},{"osd":7,"last update":"Thu Oct 19 14:02:09 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.67900000000000005,"15min":0.68700000000000006},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.78100000000000003,"5min":1.3140000000000001,"15min":1.3140000000000001},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.69899999999999995,"15min":0.71499999999999997},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.80800000000000005,"5min":0.98899999999999999,"15min":1.304},"last":0.69399999999999995}]}]},{"osd":2,"up_from":13,"seq":55834575787,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59840,"kb_used_data":2288,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312000,"statfs":{"total":96636764160,"available":96575488000,"internally_reserved":0,"allocated":2342912,"data_stored":1189100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:01:58 2023","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.41399999999999998,"15min":0.40699999999999997},"min":{"1min":0.24199999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.63400000000000001,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.20999999999999999},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.44700000000000001,"15min":0.441},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.56899999999999995,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.35499999999999998}]},{"osd":1,"last update":"Thu Oct 19 14:01:58 2023","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.40500000000000003,"15min":0.41299999999999998},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.52100000000000002,"5min":0.60799999999999998,"15min":0.66600000000000004},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.46800000000000003,"15min":0.46400000000000002},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.26500000000000001},"max":{"1min":0.67300000000000004,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.25}]},{"osd":3,"last update":"Thu Oct 19 14:01:58 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.48699999999999999,"15min":0.49199999999999999},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.68300000000000005,"5min":0.72099999999999997,"15min":0.75900000000000001},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.503,"15min":0.51300000000000001},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.66200000000000003,"5min":0.69699999999999995,"15min":0.78700000000000003},"last":0.38800000000000001}]},{"osd":4,"last update":"Thu Oct 19 14:01:58 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.67000000000000004,"15min":0.68100000000000005},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.032,"5min":1.032,"15min":1.0640000000000001},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.68799999999999994,"15min":0.68899999999999995},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.86799999999999999,"5min":1.1619999999999999,"15min":1.2110000000000001},"last":0.57799999999999996}]},{"osd":5,"last update":"Thu Oct 19 14:01:58 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.70599999999999996,"15min":0.70399999999999996},"min":{"1min":0.433,"5min":0.433,"15min":0.42299999999999999},"max":{"1min":0.85099999999999998,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.622,"5min":0.68100000000000005,"15min":0.68999999999999995},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.44600000000000001},"max":{"1min":1.1140000000000001,"5min":1.1459999999999999,"15min":1.194},"last":0.505}]},{"osd":6,"last update":"Thu Oct 19 14:01:58 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.70099999999999996,"15min":0.71499999999999997},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.98399999999999999,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.72399999999999998,"15min":0.72199999999999998},"min":{"1min":0.48199999999999998,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.056,"5min":1.222,"15min":1.234},"last":0.435}]},{"osd":7,"last update":"Thu Oct 19 14:01:58 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.73199999999999998,"15min":0.73599999999999999},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.47699999999999998},"max":{"1min":1.0189999999999999,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.73099999999999998,"15min":0.74099999999999999},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.98999999999999999,"5min":1.161,"15min":1.161},"last":0.46200000000000002}]}]},{"osd":1,"up_from":13,"seq":55834575786,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41800,"kb_used_data":2300,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330040,"statfs":{"total":96636764160,"available":96593960960,"internally_reserved":0,"allocated":2355200,"data_stored":1190712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:01:52 2023","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.40799999999999997,"15min":0.41499999999999998},"min":{"1min":0.27400000000000002,"5min":0.26000000000000001,"15min":0.219},"max":{"1min":0.628,"5min":0.64500000000000002,"15min":0.69599999999999995},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.45100000000000001,"15min":0.45600000000000002},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.55400000000000005,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.45700000000000002}]},{"osd":2,"last update":"Thu Oct 19 14:01:52 2023","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.42599999999999999,"15min":0.434},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.53200000000000003,"5min":0.68000000000000005,"15min":0.68300000000000005},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.432,"5min":0.48399999999999999,"15min":0.48099999999999998},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.28499999999999998},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.52400000000000002}]},{"osd":3,"last update":"Thu Oct 19 14:01:52 2023","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.504,"15min":0.505},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.627,"5min":0.70299999999999996,"15min":0.73899999999999999},"last":0.5},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.44,"15min":0.44600000000000001},"min":{"1min":0.29899999999999999,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.67500000000000004},"last":0.42699999999999999}]},{"osd":4,"last update":"Thu Oct 19 14:01:52 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.67800000000000005,"15min":0.68600000000000005},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.83899999999999997,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.67800000000000005,"15min":0.67200000000000004},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.40699999999999997},"max":{"1min":0.751,"5min":0.90500000000000003,"15min":1.0720000000000001},"last":0.89500000000000002}]},{"osd":5,"last update":"Thu Oct 19 14:01:52 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.69699999999999995,"15min":0.69199999999999995},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.44},"max":{"1min":1.2470000000000001,"5min":1.2470000000000001,"15min":1.254},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.628,"5min":0.70499999999999996,"15min":0.70499999999999996},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.137,"5min":1.137,"15min":1.137},"last":0.78600000000000003}]},{"osd":6,"last update":"Thu Oct 19 14:01:52 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.68600000000000005,"15min":0.68799999999999994},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.41899999999999998},"max":{"1min":0.79700000000000004,"5min":1.25,"15min":1.25},"last":0.873},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.70499999999999996,"15min":0.71399999999999997},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.79300000000000004,"5min":0.999,"15min":1.153},"last":0.68300000000000005}]},{"osd":7,"last update":"Thu Oct 19 14:01:52 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.72099999999999997,"15min":0.71799999999999997},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.76900000000000002,"5min":1.1950000000000001,"15min":1.1950000000000001},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.624,"5min":0.72099999999999997,"15min":0.71899999999999997},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.878,"5min":1.097,"15min":1.097},"last":0.84699999999999998}]}]},{"osd":0,"up_from":13,"seq":55834575787,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38220,"kb_used_data":2872,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333620,"statfs":{"total":96636764160,"available":96597626880,"internally_reserved":0,"allocated":2940928,"data_stored":1778953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Oct 19 14:01:34 2023","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.38900000000000001,"15min":0.39600000000000002},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.314},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.45600000000000002,"15min":0.46000000000000002},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.379}]},{"osd":2,"last update":"Thu Oct 19 14:01:34 2023","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.47399999999999998,"15min":0.47099999999999997},"min":{"1min":0.312,"5min":0.30299999999999999,"15min":0.28699999999999998},"max":{"1min":0.74199999999999999,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.441,"5min":0.45400000000000001,"15min":0.45100000000000001},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.40300000000000002}]},{"osd":3,"last update":"Thu Oct 19 14:01:34 2023","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.439,"15min":0.43099999999999999},"min":{"1min":0.32700000000000001,"5min":0.30099999999999999,"15min":0.27500000000000002},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.5,"5min":0.52000000000000002,"15min":0.50900000000000001},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.435}]},{"osd":4,"last update":"Thu Oct 19 14:01:34 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.68700000000000006,"15min":0.67400000000000004},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.43099999999999999},"max":{"1min":1.0189999999999999,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.622},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.69399999999999995,"15min":0.67900000000000005},"min":{"1min":0.49099999999999999,"5min":0.46800000000000003,"15min":0.45900000000000002},"max":{"1min":0.88900000000000001,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.78200000000000003}]},{"osd":5,"last update":"Thu Oct 19 14:01:34 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.73399999999999999,"15min":0.71699999999999997},"min":{"1min":0.5,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.069,"5min":1.1970000000000001,"15min":1.1970000000000001},"last":0.496},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.73699999999999999,"15min":0.73199999999999998},"min":{"1min":0.51500000000000001,"5min":0.443,"15min":0.443},"max":{"1min":0.95399999999999996,"5min":1.139,"15min":1.139},"last":0.745}]},{"osd":6,"last update":"Thu Oct 19 14:01:34 2023","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.71499999999999997,"15min":0.70199999999999996},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.091,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.71099999999999997,"15min":0.69899999999999995},"min":{"1min":0.503,"5min":0.49399999999999999,"15min":0.443},"max":{"1min":0.98099999999999998,"5min":1.0660000000000001,"15min":1.167},"last":0.68600000000000005}]},{"osd":7,"last update":"Thu Oct 19 14:01:34 2023","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.74199999999999999,"15min":0.72699999999999998},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.145,"5min":1.163,"15min":1.163},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.71599999999999997,"15min":0.70099999999999996},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.123,"5min":1.123,"15min":1.123},"last":0.71099999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-19T14:02:14.053 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-19T14:02:14.054 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-19T14:02:14.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.201+0000 7f495f5c0640 1 -- 172.21.15.57:0/1691969874 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f495815b2a0 msgr2=0x7f49580cf720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:14.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.201+0000 7f495f5c0640 1 --2- 172.21.15.57:0/1691969874 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f495815b2a0 0x7f49580cf720 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 crypto rx=0x7f495808a810 tx=0x7f494c02bd10 comp rx=0 tx=0).stop 2023-10-19T14:02:14.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.201+0000 7f495f5c0640 1 -- 172.21.15.57:0/1691969874 shutdown_connections 2023-10-19T14:02:14.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.201+0000 7f495f5c0640 1 --2- 172.21.15.57:0/1691969874 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f49580ceb40 0x7f49580b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:14.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.201+0000 7f495f5c0640 1 --2- 172.21.15.57:0/1691969874 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f495815b2a0 0x7f49580cf720 unknown :-1 s=CLOSED pgs=2050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:14.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.201+0000 7f495f5c0640 1 --2- 172.21.15.57:0/1691969874 >> v2:172.21.15.179:3300/0 conn(0x7f49580cf340 0x7f49580d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:14.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.201+0000 7f495f5c0640 1 -- 172.21.15.57:0/1691969874 >> 172.21.15.57:0/1691969874 conn(0x7f49580d0d60 msgr2=0x7f49580d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:02:14.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.201+0000 7f495f5c0640 1 -- 172.21.15.57:0/1691969874 shutdown_connections 2023-10-19T14:02:14.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.201+0000 7f495f5c0640 1 -- 172.21.15.57:0/1691969874 wait complete. 2023-10-19T14:02:14.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.203+0000 7f495f5c0640 1 Processor -- start 2023-10-19T14:02:14.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.203+0000 7f495f5c0640 1 -- start start 2023-10-19T14:02:14.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.203+0000 7f495f5c0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49580ceb40 0x7f49581f0de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:14.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.203+0000 7f495f5c0640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f49580cf340 0x7f49581587d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:14.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.203+0000 7f495f5c0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f495815b2a0 0x7f4958158d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:14.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.203+0000 7f495f5c0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f495814fb50 con 0x7f49580ceb40 2023-10-19T14:02:14.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.203+0000 7f495f5c0640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49581f5430 con 0x7f495815b2a0 2023-10-19T14:02:14.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.203+0000 7f495f5c0640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49581f4b00 con 0x7f49580cf340 2023-10-19T14:02:14.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.203+0000 7f495d35c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49580ceb40 0x7f49581f0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:14.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.203+0000 7f495d35c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49580ceb40 0x7f49581f0de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59786/0 (socket says 172.21.15.57:59786) 2023-10-19T14:02:14.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.203+0000 7f495d35c640 1 -- 172.21.15.57:0/2597564584 learned_addr learned my addr 172.21.15.57:0/2597564584 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:02:14.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.204+0000 7f495cb5b640 1 --2- 172.21.15.57:0/2597564584 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f49580cf340 0x7f49581587d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:14.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.204+0000 7f495db5d640 1 --2- 172.21.15.57:0/2597564584 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f495815b2a0 0x7f4958158d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:14.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.204+0000 7f495d35c640 1 -- 172.21.15.57:0/2597564584 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f49580cf340 msgr2=0x7f49581587d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:14.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.204+0000 7f495d35c640 1 --2- 172.21.15.57:0/2597564584 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f49580cf340 0x7f49581587d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:14.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.204+0000 7f495d35c640 1 -- 172.21.15.57:0/2597564584 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f495815b2a0 msgr2=0x7f4958158d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:14.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.204+0000 7f495d35c640 1 --2- 172.21.15.57:0/2597564584 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f495815b2a0 0x7f4958158d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:14.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.204+0000 7f495d35c640 1 -- 172.21.15.57:0/2597564584 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f494c009560 con 0x7f49580ceb40 2023-10-19T14:02:14.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.204+0000 7f495cb5b640 1 --2- 172.21.15.57:0/2597564584 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f49580cf340 0x7f49581587d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:02:14.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.204+0000 7f495d35c640 1 --2- 172.21.15.57:0/2597564584 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49580ceb40 0x7f49581f0de0 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7f495000d950 tx=0x7f495000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:02:14.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.204+0000 7f4955ffb640 1 -- 172.21.15.57:0/2597564584 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4950009c30 con 0x7f49580ceb40 2023-10-19T14:02:14.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.204+0000 7f4955ffb640 1 -- 172.21.15.57:0/2597564584 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f4950009e80 con 0x7f49580ceb40 2023-10-19T14:02:14.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.204+0000 7f495f5c0640 1 -- 172.21.15.57:0/2597564584 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49581f4de0 con 0x7f49580ceb40 2023-10-19T14:02:14.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.204+0000 7f4955ffb640 1 -- 172.21.15.57:0/2597564584 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4950004280 con 0x7f49580ceb40 2023-10-19T14:02:14.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.204+0000 7f495f5c0640 1 -- 172.21.15.57:0/2597564584 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49581f1320 con 0x7f49580ceb40 2023-10-19T14:02:14.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.205+0000 7f495f5c0640 1 -- 172.21.15.57:0/2597564584 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49580690e0 con 0x7f49580ceb40 2023-10-19T14:02:14.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.206+0000 7f4955ffb640 1 -- 172.21.15.57:0/2597564584 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 99) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f49500035f0 con 0x7f49580ceb40 2023-10-19T14:02:14.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.206+0000 7f4955ffb640 1 --2- 172.21.15.57:0/2597564584 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f494406af60 0x7f494406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:14.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.206+0000 7f4955ffb640 1 -- 172.21.15.57:0/2597564584 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f4950089c30 con 0x7f49580ceb40 2023-10-19T14:02:14.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.206+0000 7f495cb5b640 1 --2- 172.21.15.57:0/2597564584 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f494406af60 0x7f494406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:14.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.207+0000 7f495cb5b640 1 --2- 172.21.15.57:0/2597564584 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f494406af60 0x7f494406d420 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f494c035000 tx=0x7f494c004730 comp rx=0 tx=0).ready entity=mgr.8315 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:02:14.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.210+0000 7f4955ffb640 1 -- 172.21.15.57:0/2597564584 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4950054960 con 0x7f49580ceb40 2023-10-19T14:02:14.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.458+0000 7f495f5c0640 1 -- 172.21.15.57:0/2597564584 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f49580d0b40 con 0x7f49580ceb40 2023-10-19T14:02:14.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.464+0000 7f4955ffb640 1 -- 172.21.15.57:0/2597564584 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v52) v1 ==== 190+0+147 (secure 0 0 0) 0x7f49500170e0 con 0x7f49580ceb40 2023-10-19T14:02:14.467 INFO:teuthology.orchestra.run.smithi057.stdout:[client.0] 2023-10-19T14:02:14.467 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQBkJDFlg9yuExAATxiWDfS+p07G1F0pORbJEg== 2023-10-19T14:02:14.467 INFO:teuthology.orchestra.run.smithi057.stdout: caps mds = "allow" 2023-10-19T14:02:14.467 INFO:teuthology.orchestra.run.smithi057.stdout: caps mon = "allow r" 2023-10-19T14:02:14.468 INFO:teuthology.orchestra.run.smithi057.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-19T14:02:14.468 INFO:teuthology.orchestra.run.smithi057.stderr:updated caps for client.0 2023-10-19T14:02:14.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.466+0000 7f495f5c0640 1 -- 172.21.15.57:0/2597564584 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f494406af60 msgr2=0x7f494406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:14.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.466+0000 7f495f5c0640 1 --2- 172.21.15.57:0/2597564584 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f494406af60 0x7f494406d420 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f494c035000 tx=0x7f494c004730 comp rx=0 tx=0).stop 2023-10-19T14:02:14.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.466+0000 7f495f5c0640 1 -- 172.21.15.57:0/2597564584 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49580ceb40 msgr2=0x7f49581f0de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:14.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.466+0000 7f495f5c0640 1 --2- 172.21.15.57:0/2597564584 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49580ceb40 0x7f49581f0de0 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7f495000d950 tx=0x7f495000de10 comp rx=0 tx=0).stop 2023-10-19T14:02:14.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.466+0000 7f495f5c0640 1 -- 172.21.15.57:0/2597564584 shutdown_connections 2023-10-19T14:02:14.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.466+0000 7f495f5c0640 1 --2- 172.21.15.57:0/2597564584 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f495815b2a0 0x7f4958158d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:14.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.466+0000 7f495f5c0640 1 --2- 172.21.15.57:0/2597564584 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f49580cf340 0x7f49581587d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:14.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.466+0000 7f495f5c0640 1 --2- 172.21.15.57:0/2597564584 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f494406af60 0x7f494406d420 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:14.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.466+0000 7f495f5c0640 1 --2- 172.21.15.57:0/2597564584 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49580ceb40 0x7f49581f0de0 unknown :-1 s=CLOSED pgs=2051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:14.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.466+0000 7f495f5c0640 1 -- 172.21.15.57:0/2597564584 >> 172.21.15.57:0/2597564584 conn(0x7f49580d0d60 msgr2=0x7f495815a080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:02:14.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.467+0000 7f495f5c0640 1 -- 172.21.15.57:0/2597564584 shutdown_connections 2023-10-19T14:02:14.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.467+0000 7f495f5c0640 1 -- 172.21.15.57:0/2597564584 wait complete. 2023-10-19T14:02:14.480 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T14:02:14.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.586+0000 7f33358e5640 1 -- 172.21.15.57:0/3520944706 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f33300cf340 msgr2=0x7f33300d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:14.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.586+0000 7f33358e5640 1 --2- 172.21.15.57:0/3520944706 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f33300cf340 0x7f33300d1910 secure :-1 s=READY pgs=2052 cs=0 l=1 rev1=1 crypto rx=0x7f3320005fd0 tx=0x7f332002bd90 comp rx=0 tx=0).stop 2023-10-19T14:02:14.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.586+0000 7f33358e5640 1 -- 172.21.15.57:0/3520944706 shutdown_connections 2023-10-19T14:02:14.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.586+0000 7f33358e5640 1 --2- 172.21.15.57:0/3520944706 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f33300ceb40 0x7f33300b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:14.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.586+0000 7f33358e5640 1 --2- 172.21.15.57:0/3520944706 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f33300cf340 0x7f33300d1910 unknown :-1 s=CLOSED pgs=2052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:14.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.586+0000 7f33358e5640 1 -- 172.21.15.57:0/3520944706 >> 172.21.15.57:0/3520944706 conn(0x7f33300d0d60 msgr2=0x7f33300d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:02:14.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.586+0000 7f33358e5640 1 -- 172.21.15.57:0/3520944706 shutdown_connections 2023-10-19T14:02:14.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.586+0000 7f33358e5640 1 -- 172.21.15.57:0/3520944706 wait complete. 2023-10-19T14:02:14.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.587+0000 7f33358e5640 1 Processor -- start 2023-10-19T14:02:14.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.588+0000 7f33358e5640 1 -- start start 2023-10-19T14:02:14.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.588+0000 7f33358e5640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f33300ceb40 0x7f3330153bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:14.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.588+0000 7f33358e5640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f33300cf340 0x7f33300c9e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:14.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.588+0000 7f33358e5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f333015b2a0 0x7f33300ca380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:14.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.588+0000 7f33358e5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f333015c5a0 con 0x7f333015b2a0 2023-10-19T14:02:14.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.588+0000 7f33358e5640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f333015b6d0 con 0x7f33300ceb40 2023-10-19T14:02:14.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.588+0000 7f33358e5640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3330156770 con 0x7f33300cf340 2023-10-19T14:02:14.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.588+0000 7f332f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f333015b2a0 0x7f33300ca380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:14.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.588+0000 7f332f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f333015b2a0 0x7f33300ca380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59794/0 (socket says 172.21.15.57:59794) 2023-10-19T14:02:14.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.588+0000 7f332f7fe640 1 -- 172.21.15.57:0/1447289540 learned_addr learned my addr 172.21.15.57:0/1447289540 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:02:14.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.589+0000 7f332effd640 1 --2- 172.21.15.57:0/1447289540 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f33300ceb40 0x7f3330153bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:14.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.589+0000 7f332e7fc640 1 --2- 172.21.15.57:0/1447289540 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f33300cf340 0x7f33300c9e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:14.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.589+0000 7f332f7fe640 1 -- 172.21.15.57:0/1447289540 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f33300cf340 msgr2=0x7f33300c9e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:14.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.589+0000 7f332f7fe640 1 --2- 172.21.15.57:0/1447289540 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f33300cf340 0x7f33300c9e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:14.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.589+0000 7f332f7fe640 1 -- 172.21.15.57:0/1447289540 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f33300ceb40 msgr2=0x7f3330153bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:14.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.589+0000 7f332f7fe640 1 --2- 172.21.15.57:0/1447289540 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f33300ceb40 0x7f3330153bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:14.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.589+0000 7f332f7fe640 1 -- 172.21.15.57:0/1447289540 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3320009560 con 0x7f333015b2a0 2023-10-19T14:02:14.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.589+0000 7f332effd640 1 --2- 172.21.15.57:0/1447289540 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f33300ceb40 0x7f3330153bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:02:14.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.589+0000 7f332e7fc640 1 --2- 172.21.15.57:0/1447289540 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f33300cf340 0x7f33300c9e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:02:14.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.589+0000 7f332f7fe640 1 --2- 172.21.15.57:0/1447289540 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f333015b2a0 0x7f33300ca380 secure :-1 s=READY pgs=2053 cs=0 l=1 rev1=1 crypto rx=0x7f332800ead0 tx=0x7f332800ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:02:14.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.589+0000 7f3317fff640 1 -- 172.21.15.57:0/1447289540 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f332800cde0 con 0x7f333015b2a0 2023-10-19T14:02:14.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.589+0000 7f3317fff640 1 -- 172.21.15.57:0/1447289540 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f3328004c10 con 0x7f333015b2a0 2023-10-19T14:02:14.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.589+0000 7f33358e5640 1 -- 172.21.15.57:0/1447289540 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33300b0d50 con 0x7f333015b2a0 2023-10-19T14:02:14.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.589+0000 7f3317fff640 1 -- 172.21.15.57:0/1447289540 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3328005020 con 0x7f333015b2a0 2023-10-19T14:02:14.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.589+0000 7f33358e5640 1 -- 172.21.15.57:0/1447289540 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f333015bc00 con 0x7f333015b2a0 2023-10-19T14:02:14.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.590+0000 7f33358e5640 1 -- 172.21.15.57:0/1447289540 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f333015c6e0 con 0x7f333015b2a0 2023-10-19T14:02:14.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.591+0000 7f3317fff640 1 -- 172.21.15.57:0/1447289540 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 99) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f3328005180 con 0x7f333015b2a0 2023-10-19T14:02:14.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.591+0000 7f3317fff640 1 --2- 172.21.15.57:0/1447289540 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f331c06af60 0x7f331c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:14.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.591+0000 7f3317fff640 1 -- 172.21.15.57:0/1447289540 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f332808a3a0 con 0x7f333015b2a0 2023-10-19T14:02:14.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.591+0000 7f332effd640 1 --2- 172.21.15.57:0/1447289540 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f331c06af60 0x7f331c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:14.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.592+0000 7f332effd640 1 --2- 172.21.15.57:0/1447289540 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f331c06af60 0x7f331c06d420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3320035000 tx=0x7f3320004730 comp rx=0 tx=0).ready entity=mgr.8315 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:02:14.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.595+0000 7f3317fff640 1 -- 172.21.15.57:0/1447289540 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3328055050 con 0x7f333015b2a0 2023-10-19T14:02:14.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.807+0000 7f33358e5640 1 -- 172.21.15.57:0/1447289540 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f33300690e0 con 0x7f333015b2a0 2023-10-19T14:02:14.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.808+0000 7f3317fff640 1 -- 172.21.15.57:0/1447289540 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+4924 (secure 0 0 0) 0x7f3328058670 con 0x7f333015b2a0 2023-10-19T14:02:14.811 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:02:14.811 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":77,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8480,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6833/2675481384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2675481384},{"type":"v1","addr":"172.21.15.57:6833","nonce":2675481384}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74},{"gid":8486,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/2884298100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2884298100},{"type":"v1","addr":"172.21.15.179:6836","nonce":2884298100}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":73},{"gid":10641,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/505361151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":505361151},{"type":"v1","addr":"172.21.15.57:6835","nonce":505361151}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T14:02:10.024658+0000","modified":"2023-10-19T14:02:12.029940+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":10647},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10647":{"gid":10647,"name":"d","rank":0,"incarnation":75,"state":"up:active","state_seq":3,"addr":"172.21.15.179:6837/3993214977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":3993214977},{"type":"v1","addr":"172.21.15.179:6837","nonce":3993214977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2023-10-19T14:02:14.811 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 77 2023-10-19T14:02:14.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.809+0000 7f33358e5640 1 -- 172.21.15.57:0/1447289540 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f331c06af60 msgr2=0x7f331c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:14.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.809+0000 7f33358e5640 1 --2- 172.21.15.57:0/1447289540 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f331c06af60 0x7f331c06d420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3320035000 tx=0x7f3320004730 comp rx=0 tx=0).stop 2023-10-19T14:02:14.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.809+0000 7f33358e5640 1 -- 172.21.15.57:0/1447289540 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f333015b2a0 msgr2=0x7f33300ca380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:14.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.809+0000 7f33358e5640 1 --2- 172.21.15.57:0/1447289540 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f333015b2a0 0x7f33300ca380 secure :-1 s=READY pgs=2053 cs=0 l=1 rev1=1 crypto rx=0x7f332800ead0 tx=0x7f332800ef90 comp rx=0 tx=0).stop 2023-10-19T14:02:14.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.810+0000 7f33358e5640 1 -- 172.21.15.57:0/1447289540 shutdown_connections 2023-10-19T14:02:14.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.810+0000 7f33358e5640 1 --2- 172.21.15.57:0/1447289540 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f331c06af60 0x7f331c06d420 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:14.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.810+0000 7f33358e5640 1 --2- 172.21.15.57:0/1447289540 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f333015b2a0 0x7f33300ca380 unknown :-1 s=CLOSED pgs=2053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:14.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.810+0000 7f33358e5640 1 --2- 172.21.15.57:0/1447289540 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f33300cf340 0x7f33300c9e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:14.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.810+0000 7f33358e5640 1 --2- 172.21.15.57:0/1447289540 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f33300ceb40 0x7f3330153bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:14.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.810+0000 7f33358e5640 1 -- 172.21.15.57:0/1447289540 >> 172.21.15.57:0/1447289540 conn(0x7f33300d0d60 msgr2=0x7f333006aac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:02:14.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.810+0000 7f33358e5640 1 -- 172.21.15.57:0/1447289540 shutdown_connections 2023-10-19T14:02:14.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:14.810+0000 7f33358e5640 1 -- 172.21.15.57:0/1447289540 wait complete. 2023-10-19T14:02:15.226 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 77, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-19T14:02:10.024658+0000', 'modified': '2023-10-19T14:02:12.029940+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 10647}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10647': {'gid': 10647, 'name': 'd', 'rank': 0, 'incarnation': 75, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.179:6837/3993214977', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6835', 'nonce': 3993214977}, {'type': 'v1', 'addr': '172.21.15.179:6837', 'nonce': 3993214977}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [24], 'metadata_pool': 23, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-19T14:02:15.226 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-19T14:02:15.227 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2023-10-19T14:02:15.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.340+0000 7fb8b5dd7640 1 -- 172.21.15.57:0/1966344675 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8b015b200 msgr2=0x7fb8b00b00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:15.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.340+0000 7fb8b5dd7640 1 --2- 172.21.15.57:0/1966344675 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8b015b200 0x7fb8b00b00f0 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 crypto rx=0x7fb8a0005fd0 tx=0x7fb8a002bea0 comp rx=0 tx=0).stop 2023-10-19T14:02:15.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.340+0000 7fb8b5dd7640 1 -- 172.21.15.57:0/1966344675 shutdown_connections 2023-10-19T14:02:15.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.340+0000 7fb8b5dd7640 1 --2- 172.21.15.57:0/1966344675 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb8b00cf2a0 0x7fb8b00cf730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:15.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.340+0000 7fb8b5dd7640 1 --2- 172.21.15.57:0/1966344675 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8b015b200 0x7fb8b00b00f0 unknown :-1 s=CLOSED pgs=2054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:15.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.340+0000 7fb8b5dd7640 1 -- 172.21.15.57:0/1966344675 >> 172.21.15.57:0/1966344675 conn(0x7fb8b008a650 msgr2=0x7fb8b0099f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:02:15.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.340+0000 7fb8b5dd7640 1 -- 172.21.15.57:0/1966344675 shutdown_connections 2023-10-19T14:02:15.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.340+0000 7fb8b5dd7640 1 -- 172.21.15.57:0/1966344675 wait complete. 2023-10-19T14:02:15.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.341+0000 7fb8b5dd7640 1 Processor -- start 2023-10-19T14:02:15.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.342+0000 7fb8b5dd7640 1 -- start start 2023-10-19T14:02:15.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.342+0000 7fb8b5dd7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8b00cf2a0 0x7fb8b00c60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:15.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.342+0000 7fb8b5dd7640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb8b015b200 0x7fb8b0153860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:15.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.342+0000 7fb8b5dd7640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb8b015eba0 0x7fb8b00ae740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:15.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.342+0000 7fb8b5dd7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8b00c6e70 con 0x7fb8b00cf2a0 2023-10-19T14:02:15.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.342+0000 7fb8b5dd7640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8b00adcf0 con 0x7fb8b015b200 2023-10-19T14:02:15.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.342+0000 7fb8b5dd7640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8b00c6630 con 0x7fb8b015eba0 2023-10-19T14:02:15.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.342+0000 7fb8af7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8b00cf2a0 0x7fb8b00c60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:15.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.342+0000 7fb8af7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8b00cf2a0 0x7fb8b00c60f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59820/0 (socket says 172.21.15.57:59820) 2023-10-19T14:02:15.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.342+0000 7fb8af7fe640 1 -- 172.21.15.57:0/2981146150 learned_addr learned my addr 172.21.15.57:0/2981146150 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:02:15.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.343+0000 7fb8affff640 1 --2- 172.21.15.57:0/2981146150 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb8b015eba0 0x7fb8b00ae740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:15.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.343+0000 7fb8aeffd640 1 --2- 172.21.15.57:0/2981146150 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb8b015b200 0x7fb8b0153860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:15.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.343+0000 7fb8af7fe640 1 -- 172.21.15.57:0/2981146150 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb8b015eba0 msgr2=0x7fb8b00ae740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:15.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.343+0000 7fb8af7fe640 1 --2- 172.21.15.57:0/2981146150 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb8b015eba0 0x7fb8b00ae740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:15.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.343+0000 7fb8af7fe640 1 -- 172.21.15.57:0/2981146150 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb8b015b200 msgr2=0x7fb8b0153860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:15.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.343+0000 7fb8af7fe640 1 --2- 172.21.15.57:0/2981146150 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb8b015b200 0x7fb8b0153860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:15.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.343+0000 7fb8af7fe640 1 -- 172.21.15.57:0/2981146150 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8a0009560 con 0x7fb8b00cf2a0 2023-10-19T14:02:15.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.343+0000 7fb8af7fe640 1 --2- 172.21.15.57:0/2981146150 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8b00cf2a0 0x7fb8b00c60f0 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 crypto rx=0x7fb8a00099f0 tx=0x7fb8a00038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:02:15.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.343+0000 7fb897fff640 1 -- 172.21.15.57:0/2981146150 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8a0003ca0 con 0x7fb8b00cf2a0 2023-10-19T14:02:15.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.343+0000 7fb897fff640 1 -- 172.21.15.57:0/2981146150 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fb8a0005020 con 0x7fb8b00cf2a0 2023-10-19T14:02:15.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.343+0000 7fb8b5dd7640 1 -- 172.21.15.57:0/2981146150 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8b0153da0 con 0x7fb8b00cf2a0 2023-10-19T14:02:15.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.343+0000 7fb897fff640 1 -- 172.21.15.57:0/2981146150 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8a0005210 con 0x7fb8b00cf2a0 2023-10-19T14:02:15.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.343+0000 7fb8b5dd7640 1 -- 172.21.15.57:0/2981146150 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8b00c6b90 con 0x7fb8b00cf2a0 2023-10-19T14:02:15.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.345+0000 7fb897fff640 1 -- 172.21.15.57:0/2981146150 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 99) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb8a0005370 con 0x7fb8b00cf2a0 2023-10-19T14:02:15.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.345+0000 7fb897fff640 1 --2- 172.21.15.57:0/2981146150 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7fb89c06af60 0x7fb89c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:15.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.345+0000 7fb897fff640 1 -- 172.21.15.57:0/2981146150 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fb8a00ac3b0 con 0x7fb8b00cf2a0 2023-10-19T14:02:15.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.345+0000 7fb8aeffd640 1 --2- 172.21.15.57:0/2981146150 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7fb89c06af60 0x7fb89c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:15.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.346+0000 7fb8aeffd640 1 --2- 172.21.15.57:0/2981146150 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7fb89c06af60 0x7fb89c06d420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb8a4007fd0 tx=0x7fb8a4009040 comp rx=0 tx=0).ready entity=mgr.8315 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:02:15.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.346+0000 7fb8b8969740 1 Processor -- start 2023-10-19T14:02:15.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.346+0000 7fb8b8969740 1 -- start start 2023-10-19T14:02:15.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.347+0000 7fb8b8969740 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55fa5d574b40 0x55fa5d547ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:15.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.347+0000 7fb8b8969740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fa5d574f20 0x55fa5d5778d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:15.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.347+0000 7fb8b8969740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fa5d548020 0x55fa5d577e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:15.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.347+0000 7fb8b8969740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa5d459730 con 0x55fa5d548020 2023-10-19T14:02:15.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.347+0000 7fb8b8969740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa5d4ba780 con 0x55fa5d574f20 2023-10-19T14:02:15.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.347+0000 7fb8b8969740 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x55fa5d4b7e70 con 0x55fa5d574b40 2023-10-19T14:02:15.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.347+0000 7fb8affff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fa5d548020 0x55fa5d577e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:15.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.347+0000 7fb8affff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fa5d548020 0x55fa5d577e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59834/0 (socket says 172.21.15.57:59834) 2023-10-19T14:02:15.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.347+0000 7fb8affff640 1 -- 172.21.15.57:0/579428732 learned_addr learned my addr 172.21.15.57:0/579428732 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:02:15.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.347+0000 7fb8aeffd640 1 --2- 172.21.15.57:0/579428732 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fa5d574f20 0x55fa5d5778d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:15.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.347+0000 7fb8af7fe640 1 --2- 172.21.15.57:0/579428732 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55fa5d574b40 0x55fa5d547ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:15.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.347+0000 7fb8affff640 1 -- 172.21.15.57:0/579428732 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55fa5d574b40 msgr2=0x55fa5d547ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T14:02:15.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.347+0000 7fb8affff640 1 --2- 172.21.15.57:0/579428732 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55fa5d574b40 0x55fa5d547ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:15.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.348+0000 7fb8affff640 1 -- 172.21.15.57:0/579428732 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fa5d574f20 msgr2=0x55fa5d5778d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T14:02:15.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.348+0000 7fb8affff640 1 --2- 172.21.15.57:0/579428732 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fa5d574f20 0x55fa5d5778d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:15.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.348+0000 7fb8affff640 1 -- 172.21.15.57:0/579428732 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fa5d4b83d0 con 0x55fa5d548020 2023-10-19T14:02:15.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.348+0000 7fb8affff640 1 --2- 172.21.15.57:0/579428732 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fa5d548020 0x55fa5d577e10 secure :-1 s=READY pgs=2056 cs=0 l=1 rev1=1 crypto rx=0x7fb8a8009f10 tx=0x7fb8a8010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4724312c15a10a4a server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:02:15.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.348+0000 7fb8957fa640 1 -- 172.21.15.57:0/579428732 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8a8004d20 con 0x55fa5d548020 2023-10-19T14:02:15.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.348+0000 7fb8957fa640 1 -- 172.21.15.57:0/579428732 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fb8a8005020 con 0x55fa5d548020 2023-10-19T14:02:15.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.348+0000 7fb8957fa640 1 -- 172.21.15.57:0/579428732 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8a8005410 con 0x55fa5d548020 2023-10-19T14:02:15.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.348+0000 7fb8b8969740 1 -- 172.21.15.57:0/579428732 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fa5d548020 msgr2=0x55fa5d577e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:15.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.348+0000 7fb8b8969740 1 --2- 172.21.15.57:0/579428732 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fa5d548020 0x55fa5d577e10 secure :-1 s=READY pgs=2056 cs=0 l=1 rev1=1 crypto rx=0x7fb8a8009f10 tx=0x7fb8a8010740 comp rx=0 tx=0).stop 2023-10-19T14:02:15.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.349+0000 7fb8b8969740 1 -- 172.21.15.57:0/579428732 shutdown_connections 2023-10-19T14:02:15.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.349+0000 7fb8b8969740 1 --2- 172.21.15.57:0/579428732 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fa5d548020 0x55fa5d577e10 unknown :-1 s=CLOSED pgs=2056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:15.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.349+0000 7fb8b8969740 1 --2- 172.21.15.57:0/579428732 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fa5d574f20 0x55fa5d5778d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:15.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.349+0000 7fb8b8969740 1 --2- 172.21.15.57:0/579428732 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55fa5d574b40 0x55fa5d547ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:15.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.349+0000 7fb8b8969740 1 -- 172.21.15.57:0/579428732 >> 172.21.15.57:0/579428732 conn(0x55fa5d549d50 msgr2=0x55fa5d5928d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:02:15.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.349+0000 7fb8b8969740 1 -- 172.21.15.57:0/579428732 shutdown_connections 2023-10-19T14:02:15.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.349+0000 7fb8b8969740 1 -- 172.21.15.57:0/579428732 wait complete. 2023-10-19T14:02:15.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.349+0000 7fb8b8969740 1 -- start start 2023-10-19T14:02:15.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.349+0000 7fb8b8969740 1 Processor -- start 2023-10-19T14:02:15.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.350+0000 7fb8b8969740 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55fa5d574b40 0x55fa5d57b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:15.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.350+0000 7fb8b8969740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fa5d574f20 0x55fa5d5961e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:15.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.350+0000 7fb8b8969740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fa5d567a50 0x55fa5d566ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:15.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.350+0000 7fb8b8969740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa5d4b3760 con 0x55fa5d574f20 2023-10-19T14:02:15.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.350+0000 7fb8b8969740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa5d46e8f0 con 0x55fa5d567a50 2023-10-19T14:02:15.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.350+0000 7fb8b8969740 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x55fa5d46e590 con 0x55fa5d574b40 2023-10-19T14:02:15.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.350+0000 7fb8b8969740 10 client.0 _ll_register_callbacks cb 0x55fa5d4aaee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-19T14:02:15.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.350+0000 7fb8aeffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fa5d574f20 0x55fa5d5961e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:15.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.350+0000 7fb8aeffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fa5d574f20 0x55fa5d5961e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59842/0 (socket says 172.21.15.57:59842) 2023-10-19T14:02:15.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.350+0000 7fb8aeffd640 1 -- 172.21.15.57:0/2477004382 learned_addr learned my addr 172.21.15.57:0/2477004382 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:02:15.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.350+0000 7fb8affff640 1 --2- 172.21.15.57:0/2477004382 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fa5d567a50 0x55fa5d566ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:15.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.350+0000 7fb8af7fe640 1 --2- 172.21.15.57:0/2477004382 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55fa5d574b40 0x55fa5d57b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:15.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.350+0000 7fb8aeffd640 1 -- 172.21.15.57:0/2477004382 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55fa5d574b40 msgr2=0x55fa5d57b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T14:02:15.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.350+0000 7fb8aeffd640 1 --2- 172.21.15.57:0/2477004382 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55fa5d574b40 0x55fa5d57b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:15.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.350+0000 7fb8aeffd640 1 -- 172.21.15.57:0/2477004382 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fa5d567a50 msgr2=0x55fa5d566ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T14:02:15.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.350+0000 7fb8aeffd640 1 --2- 172.21.15.57:0/2477004382 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fa5d567a50 0x55fa5d566ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:15.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.350+0000 7fb8aeffd640 1 -- 172.21.15.57:0/2477004382 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8a8009b40 con 0x55fa5d574f20 2023-10-19T14:02:15.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.351+0000 7fb8aeffd640 1 --2- 172.21.15.57:0/2477004382 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fa5d574f20 0x55fa5d5961e0 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 crypto rx=0x7fb8b015e830 tx=0x7fb8a4000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f53e39f73baa35bd server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:02:15.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.351+0000 7fb88e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2023-10-19T14:02:15.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.351+0000 7fb88e7fc640 1 -- 172.21.15.57:0/2477004382 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8a4015110 con 0x55fa5d574f20 2023-10-19T14:02:15.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.351+0000 7fb88e7fc640 1 -- 172.21.15.57:0/2477004382 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(188..188 src has 1..188) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fb8a4015270 con 0x55fa5d574f20 2023-10-19T14:02:15.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.351+0000 7fb8b8969740 1 -- 172.21.15.57:0/2477004382 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55fa5d4ac740 con 0x55fa5d574f20 2023-10-19T14:02:15.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.351+0000 7fb88e7fc640 1 -- 172.21.15.57:0/2477004382 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=77) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb8a4015990 con 0x55fa5d574f20 2023-10-19T14:02:15.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.351+0000 7fb8b8969740 10 client.10773 fetch_fsmap learned FSMap version 77 2023-10-19T14:02:15.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.351+0000 7fb8b8969740 1 -- 172.21.15.57:0/2477004382 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=77,monmap=2+}) v3 -- 0x55fa5d4b1de0 con 0x55fa5d574f20 2023-10-19T14:02:15.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.351+0000 7fb88e7fc640 1 -- 172.21.15.57:0/2477004382 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x55fa5d4ac740 con 0x55fa5d574f20 2023-10-19T14:02:15.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.351+0000 7fb88e7fc640 1 -- 172.21.15.57:0/2477004382 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 77) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fb8a4015be0 con 0x55fa5d574f20 2023-10-19T14:02:15.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.351+0000 7fb8b8969740 10 client.10773 fetch_fsmap finished waiting for FSMap version 77 2023-10-19T14:02:15.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.351+0000 7fb8b8969740 10 client.10773 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-19T14:02:15.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.351+0000 7fb8b8969740 20 client.10773 populate_metadata read hostname 'smithi057' 2023-10-19T14:02:15.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.351+0000 7fb8b8969740 1 --2- 172.21.15.57:0/2477004382 >> [v2:172.21.15.179:6835/3993214977,v1:172.21.15.179:6837/3993214977] conn(0x55fa5d579b50 0x55fa5d579f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:15.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.351+0000 7fb8b8969740 4 client.10773 mds_command: new command op to 10647 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-19T14:02:15.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.351+0000 7fb8b8969740 1 -- 172.21.15.57:0/2477004382 --> [v2:172.21.15.179:6835/3993214977,v1:172.21.15.179:6837/3993214977] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fa5d4ac460 con 0x55fa5d579b50 2023-10-19T14:02:15.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.351+0000 7fb8af7fe640 1 --2- 172.21.15.57:0/2477004382 >> [v2:172.21.15.179:6835/3993214977,v1:172.21.15.179:6837/3993214977] conn(0x55fa5d579b50 0x55fa5d579f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:15.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.352+0000 7fb88e7fc640 10 client.10773 ms_handle_connect on v2:172.21.15.179:6835/3993214977 2023-10-19T14:02:15.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.352+0000 7fb8af7fe640 1 --2- 172.21.15.57:0/2477004382 >> [v2:172.21.15.179:6835/3993214977,v1:172.21.15.179:6837/3993214977] conn(0x55fa5d579b50 0x55fa5d579f30 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=927510debd7b5f3c server_cookie=64262d517c3b7d08 in_seq=0 out_seq=0 2023-10-19T14:02:15.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.352+0000 7fb88e7fc640 1 -- 172.21.15.57:0/2477004382 <== mds.0 v2:172.21.15.179:6835/3993214977 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55fa5d4ac460 con 0x55fa5d579b50 2023-10-19T14:02:15.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.352+0000 7fb88e7fc640 10 client.10773 handle_command_reply: tid=0 2023-10-19T14:02:15.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.352+0000 7fb8b8969740 1 client.10773 shutdown 2023-10-19T14:02:15.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.353+0000 7fb8b8969740 1 -- 172.21.15.57:0/2477004382 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fa5d574f20 msgr2=0x55fa5d5961e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:15.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.353+0000 7fb8b8969740 1 --2- 172.21.15.57:0/2477004382 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fa5d574f20 0x55fa5d5961e0 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 crypto rx=0x7fb8b015e830 tx=0x7fb8a4000bc0 comp rx=0 tx=0).stop 2023-10-19T14:02:15.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.353+0000 7fb8b8969740 1 -- 172.21.15.57:0/2477004382 shutdown_connections 2023-10-19T14:02:15.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.353+0000 7fb8b8969740 1 --2- 172.21.15.57:0/2477004382 >> [v2:172.21.15.179:6835/3993214977,v1:172.21.15.179:6837/3993214977] conn(0x55fa5d579b50 0x55fa5d579f30 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:15.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.353+0000 7fb8b8969740 1 --2- 172.21.15.57:0/2477004382 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fa5d567a50 0x55fa5d566ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:15.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.353+0000 7fb8b8969740 1 --2- 172.21.15.57:0/2477004382 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fa5d574f20 0x55fa5d5961e0 unknown :-1 s=CLOSED pgs=2057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:15.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.353+0000 7fb88e7fc640 0 client.10773 ms_handle_reset on v2:172.21.15.179:6835/3993214977 2023-10-19T14:02:15.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.353+0000 7fb8b8969740 1 --2- 172.21.15.57:0/2477004382 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55fa5d574b40 0x55fa5d57b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:15.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.353+0000 7fb8b8969740 1 -- 172.21.15.57:0/2477004382 >> 172.21.15.57:0/2477004382 conn(0x55fa5d548020 msgr2=0x55fa5d54bb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:02:15.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.353+0000 7fb8b8969740 1 -- 172.21.15.57:0/2477004382 shutdown_connections 2023-10-19T14:02:15.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.353+0000 7fb8b8969740 1 -- 172.21.15.57:0/2477004382 wait complete. 2023-10-19T14:02:15.375 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.353+0000 7fb8b8969740 20 client.10773 trim_cache size 0 max 16384 2023-10-19T14:02:15.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.368+0000 7fb8b8969740 1 Processor -- start 2023-10-19T14:02:15.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.368+0000 7fb8b8969740 1 -- start start 2023-10-19T14:02:15.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.369+0000 7fb8b8969740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fa5d567a50 0x55fa5d5630c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:15.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.369+0000 7fb8b8969740 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55fa5d574b40 0x55fa5d5773a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:15.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.369+0000 7fb8b8969740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fa5d574f20 0x55fa5d573a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:15.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.369+0000 7fb8b8969740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa5d434270 con 0x55fa5d574f20 2023-10-19T14:02:15.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.369+0000 7fb8b8969740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa5d54df80 con 0x55fa5d567a50 2023-10-19T14:02:15.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.369+0000 7fb8b8969740 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x55fa5d577a60 con 0x55fa5d574b40 2023-10-19T14:02:15.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.369+0000 7fb8affff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fa5d574f20 0x55fa5d573a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:15.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.369+0000 7fb8affff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fa5d574f20 0x55fa5d573a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59844/0 (socket says 172.21.15.57:59844) 2023-10-19T14:02:15.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.369+0000 7fb8affff640 1 -- 172.21.15.57:0/1349497026 learned_addr learned my addr 172.21.15.57:0/1349497026 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:02:15.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.369+0000 7fb8af7fe640 1 --2- 172.21.15.57:0/1349497026 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fa5d567a50 0x55fa5d5630c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:15.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.369+0000 7fb8aeffd640 1 --2- 172.21.15.57:0/1349497026 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55fa5d574b40 0x55fa5d5773a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:15.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.369+0000 7fb8affff640 1 -- 172.21.15.57:0/1349497026 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55fa5d574b40 msgr2=0x55fa5d5773a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T14:02:15.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.369+0000 7fb8affff640 1 --2- 172.21.15.57:0/1349497026 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55fa5d574b40 0x55fa5d5773a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:15.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.369+0000 7fb8affff640 1 -- 172.21.15.57:0/1349497026 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fa5d567a50 msgr2=0x55fa5d5630c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T14:02:15.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.369+0000 7fb8affff640 1 --2- 172.21.15.57:0/1349497026 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fa5d567a50 0x55fa5d5630c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:15.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.369+0000 7fb8affff640 1 -- 172.21.15.57:0/1349497026 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fa5d450d00 con 0x55fa5d574f20 2023-10-19T14:02:15.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.370+0000 7fb8affff640 1 --2- 172.21.15.57:0/1349497026 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fa5d574f20 0x55fa5d573a80 secure :-1 s=READY pgs=2058 cs=0 l=1 rev1=1 crypto rx=0x7fb8a8010c40 tx=0x7fb8a8004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9deced26e3d9efcf server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:02:15.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.370+0000 7fb895ffb640 1 -- 172.21.15.57:0/1349497026 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8a801f040 con 0x55fa5d574f20 2023-10-19T14:02:15.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.370+0000 7fb895ffb640 1 -- 172.21.15.57:0/1349497026 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fb8a8005190 con 0x55fa5d574f20 2023-10-19T14:02:15.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.370+0000 7fb895ffb640 1 -- 172.21.15.57:0/1349497026 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8a8005520 con 0x55fa5d574f20 2023-10-19T14:02:15.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.370+0000 7fb8b8969740 1 -- 172.21.15.57:0/1349497026 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fa5d574f20 msgr2=0x55fa5d573a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:15.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.370+0000 7fb8b8969740 1 --2- 172.21.15.57:0/1349497026 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fa5d574f20 0x55fa5d573a80 secure :-1 s=READY pgs=2058 cs=0 l=1 rev1=1 crypto rx=0x7fb8a8010c40 tx=0x7fb8a8004ab0 comp rx=0 tx=0).stop 2023-10-19T14:02:15.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.370+0000 7fb8b8969740 1 -- 172.21.15.57:0/1349497026 shutdown_connections 2023-10-19T14:02:15.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.370+0000 7fb8b8969740 1 --2- 172.21.15.57:0/1349497026 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fa5d574f20 0x55fa5d573a80 unknown :-1 s=CLOSED pgs=2058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:15.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.370+0000 7fb8b8969740 1 --2- 172.21.15.57:0/1349497026 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55fa5d574b40 0x55fa5d5773a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:15.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.370+0000 7fb8b8969740 1 --2- 172.21.15.57:0/1349497026 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fa5d567a50 0x55fa5d5630c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:15.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.370+0000 7fb8b8969740 1 -- 172.21.15.57:0/1349497026 >> 172.21.15.57:0/1349497026 conn(0x55fa5d548020 msgr2=0x55fa5d58d2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:02:15.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.371+0000 7fb8b8969740 1 -- 172.21.15.57:0/1349497026 shutdown_connections 2023-10-19T14:02:15.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.371+0000 7fb8b8969740 1 -- 172.21.15.57:0/1349497026 wait complete. 2023-10-19T14:02:15.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.371+0000 7fb8b8969740 1 -- start start 2023-10-19T14:02:15.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.371+0000 7fb8b8969740 1 Processor -- start 2023-10-19T14:02:15.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.372+0000 7fb8b8969740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fa5d567a50 0x55fa5d594340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:15.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.372+0000 7fb8b8969740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fa5d574b40 0x55fa5d5f48e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:15.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.372+0000 7fb8b8969740 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55fa5d574f20 0x55fa5d5f4e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:15.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.372+0000 7fb8b8969740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa5d5f5360 con 0x55fa5d574b40 2023-10-19T14:02:15.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.372+0000 7fb8b8969740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa5d54cd80 con 0x55fa5d567a50 2023-10-19T14:02:15.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.372+0000 7fb8b8969740 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x55fa5d54d1e0 con 0x55fa5d574f20 2023-10-19T14:02:15.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.372+0000 7fb8b8969740 10 client.0 _ll_register_callbacks cb 0x7fb8a8009680 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-19T14:02:15.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.372+0000 7fb8aeffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fa5d574b40 0x55fa5d5f48e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:15.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.372+0000 7fb8aeffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fa5d574b40 0x55fa5d5f48e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59862/0 (socket says 172.21.15.57:59862) 2023-10-19T14:02:15.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.372+0000 7fb8aeffd640 1 -- 172.21.15.57:0/236837223 learned_addr learned my addr 172.21.15.57:0/236837223 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:02:15.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.372+0000 7fb8af7fe640 1 --2- 172.21.15.57:0/236837223 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fa5d567a50 0x55fa5d594340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:15.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.372+0000 7fb8affff640 1 --2- 172.21.15.57:0/236837223 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55fa5d574f20 0x55fa5d5f4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:15.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.372+0000 7fb8aeffd640 1 -- 172.21.15.57:0/236837223 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55fa5d574f20 msgr2=0x55fa5d5f4e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T14:02:15.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.372+0000 7fb8aeffd640 1 --2- 172.21.15.57:0/236837223 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55fa5d574f20 0x55fa5d5f4e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:15.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.372+0000 7fb8aeffd640 1 -- 172.21.15.57:0/236837223 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fa5d567a50 msgr2=0x55fa5d594340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T14:02:15.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.372+0000 7fb8aeffd640 1 --2- 172.21.15.57:0/236837223 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fa5d567a50 0x55fa5d594340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:15.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.372+0000 7fb8aeffd640 1 -- 172.21.15.57:0/236837223 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fa5d4b83d0 con 0x55fa5d574b40 2023-10-19T14:02:15.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.372+0000 7fb8aeffd640 1 --2- 172.21.15.57:0/236837223 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fa5d574b40 0x55fa5d5f48e0 secure :-1 s=READY pgs=2059 cs=0 l=1 rev1=1 crypto rx=0x7fb8a4003fd0 tx=0x7fb8a4005550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=539f7e6cc9cdaee2 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:02:15.391 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.372+0000 7fb88e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2023-10-19T14:02:15.391 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.373+0000 7fb88e7fc640 1 -- 172.21.15.57:0/236837223 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8a400e160 con 0x55fa5d574b40 2023-10-19T14:02:15.391 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.373+0000 7fb88e7fc640 1 -- 172.21.15.57:0/236837223 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(188..188 src has 1..188) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fb8a4001220 con 0x55fa5d574b40 2023-10-19T14:02:15.391 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.373+0000 7fb8b8969740 1 -- 172.21.15.57:0/236837223 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb8a8005020 con 0x55fa5d574b40 2023-10-19T14:02:15.391 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.373+0000 7fb88e7fc640 1 -- 172.21.15.57:0/236837223 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=77) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb8a4007c70 con 0x55fa5d574b40 2023-10-19T14:02:15.392 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.373+0000 7fb8b8969740 10 client.10779 fetch_fsmap learned FSMap version 77 2023-10-19T14:02:15.392 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.373+0000 7fb8b8969740 1 -- 172.21.15.57:0/236837223 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=77,monmap=2+}) v3 -- 0x55fa5d4764f0 con 0x55fa5d574b40 2023-10-19T14:02:15.392 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.373+0000 7fb88e7fc640 1 -- 172.21.15.57:0/236837223 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fb8a8005020 con 0x55fa5d574b40 2023-10-19T14:02:15.392 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.373+0000 7fb88e7fc640 1 -- 172.21.15.57:0/236837223 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 77) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fb8a4015a20 con 0x55fa5d574b40 2023-10-19T14:02:15.392 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.373+0000 7fb8b8969740 10 client.10779 fetch_fsmap finished waiting for FSMap version 77 2023-10-19T14:02:15.392 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.373+0000 7fb8b8969740 10 client.10779 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-19T14:02:15.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.373+0000 7fb8b8969740 20 client.10779 populate_metadata read hostname 'smithi057' 2023-10-19T14:02:15.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.373+0000 7fb8b8969740 1 --2- 172.21.15.57:0/236837223 >> [v2:172.21.15.179:6835/3993214977,v1:172.21.15.179:6837/3993214977] conn(0x55fa5d5fea70 0x55fa5d5fee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:15.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.373+0000 7fb8b8969740 4 client.10779 mds_command: new command op to 10647 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-19T14:02:15.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.373+0000 7fb8b8969740 1 -- 172.21.15.57:0/236837223 --> [v2:172.21.15.179:6835/3993214977,v1:172.21.15.179:6837/3993214977] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55fa5d1b56c0 con 0x55fa5d5fea70 2023-10-19T14:02:15.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.373+0000 7fb8af7fe640 1 --2- 172.21.15.57:0/236837223 >> [v2:172.21.15.179:6835/3993214977,v1:172.21.15.179:6837/3993214977] conn(0x55fa5d5fea70 0x55fa5d5fee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:15.394 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.374+0000 7fb88e7fc640 10 client.10779 ms_handle_connect on v2:172.21.15.179:6835/3993214977 2023-10-19T14:02:15.394 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.374+0000 7fb8af7fe640 1 --2- 172.21.15.57:0/236837223 >> [v2:172.21.15.179:6835/3993214977,v1:172.21.15.179:6837/3993214977] conn(0x55fa5d5fea70 0x55fa5d5fee50 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c2c7ef6c9d531326 server_cookie=bd8933ffff47d79b in_seq=0 out_seq=0 2023-10-19T14:02:15.394 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.374+0000 7fb88e7fc640 1 -- 172.21.15.57:0/236837223 <== mds.0 v2:172.21.15.179:6835/3993214977 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+334 (crc 0 0 0) 0x55fa5d1b56c0 con 0x55fa5d5fea70 2023-10-19T14:02:15.394 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.374+0000 7fb88e7fc640 10 client.10779 handle_command_reply: tid=0 2023-10-19T14:02:15.394 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.374+0000 7fb8b8969740 1 client.10779 shutdown 2023-10-19T14:02:15.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.374+0000 7fb8b8969740 1 -- 172.21.15.57:0/236837223 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fa5d574b40 msgr2=0x55fa5d5f48e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:15.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.374+0000 7fb8b8969740 1 --2- 172.21.15.57:0/236837223 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fa5d574b40 0x55fa5d5f48e0 secure :-1 s=READY pgs=2059 cs=0 l=1 rev1=1 crypto rx=0x7fb8a4003fd0 tx=0x7fb8a4005550 comp rx=0 tx=0).stop 2023-10-19T14:02:15.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.374+0000 7fb8b8969740 1 -- 172.21.15.57:0/236837223 shutdown_connections 2023-10-19T14:02:15.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.374+0000 7fb8b8969740 1 --2- 172.21.15.57:0/236837223 >> [v2:172.21.15.179:6835/3993214977,v1:172.21.15.179:6837/3993214977] conn(0x55fa5d5fea70 0x55fa5d5fee50 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:15.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.374+0000 7fb8b8969740 1 --2- 172.21.15.57:0/236837223 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55fa5d574f20 0x55fa5d5f4e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:15.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.374+0000 7fb88e7fc640 0 client.10779 ms_handle_reset on v2:172.21.15.179:6835/3993214977 2023-10-19T14:02:15.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.374+0000 7fb8b8969740 1 --2- 172.21.15.57:0/236837223 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fa5d574b40 0x55fa5d5f48e0 unknown :-1 s=CLOSED pgs=2059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:15.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.374+0000 7fb8b8969740 1 --2- 172.21.15.57:0/236837223 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fa5d567a50 0x55fa5d594340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:15.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.374+0000 7fb8b8969740 1 -- 172.21.15.57:0/236837223 >> 172.21.15.57:0/236837223 conn(0x55fa5d548020 msgr2=0x55fa5d573b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:02:15.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.374+0000 7fb8b8969740 1 -- 172.21.15.57:0/236837223 shutdown_connections 2023-10-19T14:02:15.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.374+0000 7fb8b8969740 1 -- 172.21.15.57:0/236837223 wait complete. 2023-10-19T14:02:15.397 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.374+0000 7fb8b8969740 20 client.10779 trim_cache size 0 max 16384 2023-10-19T14:02:15.397 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2023-10-19T14:02:15.397 INFO:teuthology.orchestra.run.smithi057.stdout: "cluster_fsid": "1e9beb19-3260-4dd7-85ef-5b399ce774ac", 2023-10-19T14:02:15.398 INFO:teuthology.orchestra.run.smithi057.stdout: "whoami": 0, 2023-10-19T14:02:15.398 INFO:teuthology.orchestra.run.smithi057.stdout: "id": 10647, 2023-10-19T14:02:15.398 INFO:teuthology.orchestra.run.smithi057.stdout: "want_state": "up:active", 2023-10-19T14:02:15.398 INFO:teuthology.orchestra.run.smithi057.stdout: "state": "up:active", 2023-10-19T14:02:15.398 INFO:teuthology.orchestra.run.smithi057.stdout: "fs_name": "cephfs", 2023-10-19T14:02:15.399 INFO:teuthology.orchestra.run.smithi057.stdout: "rank_uptime": 4.7036479099999999, 2023-10-19T14:02:15.399 INFO:teuthology.orchestra.run.smithi057.stdout: "mdsmap_epoch": 77, 2023-10-19T14:02:15.399 INFO:teuthology.orchestra.run.smithi057.stdout: "osdmap_epoch": 188, 2023-10-19T14:02:15.399 INFO:teuthology.orchestra.run.smithi057.stdout: "osdmap_epoch_barrier": 188, 2023-10-19T14:02:15.399 INFO:teuthology.orchestra.run.smithi057.stdout: "uptime": 9.4210113730000007 2023-10-19T14:02:15.399 INFO:teuthology.orchestra.run.smithi057.stdout:} 2023-10-19T14:02:15.401 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.375+0000 7fb8b5dd7640 1 -- 172.21.15.57:0/2981146150 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7fb89c06af60 msgr2=0x7fb89c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:15.401 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.375+0000 7fb8b5dd7640 1 --2- 172.21.15.57:0/2981146150 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7fb89c06af60 0x7fb89c06d420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb8a4007fd0 tx=0x7fb8a4009040 comp rx=0 tx=0).stop 2023-10-19T14:02:15.401 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.375+0000 7fb8b5dd7640 1 -- 172.21.15.57:0/2981146150 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8b00cf2a0 msgr2=0x7fb8b00c60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:15.401 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.375+0000 7fb8b5dd7640 1 --2- 172.21.15.57:0/2981146150 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8b00cf2a0 0x7fb8b00c60f0 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 crypto rx=0x7fb8a00099f0 tx=0x7fb8a00038a0 comp rx=0 tx=0).stop 2023-10-19T14:02:15.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.375+0000 7fb8b5dd7640 1 -- 172.21.15.57:0/2981146150 shutdown_connections 2023-10-19T14:02:15.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.375+0000 7fb8b5dd7640 1 --2- 172.21.15.57:0/2981146150 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb8b015eba0 0x7fb8b00ae740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:15.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.375+0000 7fb8b5dd7640 1 --2- 172.21.15.57:0/2981146150 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb8b015b200 0x7fb8b0153860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:15.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.375+0000 7fb8b5dd7640 1 --2- 172.21.15.57:0/2981146150 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7fb89c06af60 0x7fb89c06d420 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:15.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.375+0000 7fb8b5dd7640 1 --2- 172.21.15.57:0/2981146150 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8b00cf2a0 0x7fb8b00c60f0 unknown :-1 s=CLOSED pgs=2055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:15.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.375+0000 7fb8b5dd7640 1 -- 172.21.15.57:0/2981146150 >> 172.21.15.57:0/2981146150 conn(0x7fb8b008a650 msgr2=0x7fb8b0065490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:02:15.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.375+0000 7fb8b5dd7640 1 -- 172.21.15.57:0/2981146150 shutdown_connections 2023-10-19T14:02:15.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.375+0000 7fb8b5dd7640 1 -- 172.21.15.57:0/2981146150 wait complete. 2023-10-19T14:02:15.404 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-19T14:02:15.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.510+0000 7f6f9c5d0640 1 -- 172.21.15.57:0/1883038626 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6f7c003f20 msgr2=0x7f6f7c0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:15.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.510+0000 7f6f9c5d0640 1 --2- 172.21.15.57:0/1883038626 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6f7c003f20 0x7f6f7c0047c0 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 crypto rx=0x7f6f88009b50 tx=0x7f6f7c0057f0 comp rx=0 tx=0).stop 2023-10-19T14:02:15.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.510+0000 7f6f9ab6d640 1 -- 172.21.15.57:0/1883038626 reap_dead start 2023-10-19T14:02:15.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.511+0000 7f6f9c5d0640 1 -- 172.21.15.57:0/1883038626 shutdown_connections 2023-10-19T14:02:15.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.511+0000 7f6f9c5d0640 1 -- 172.21.15.57:0/1883038626 >> 172.21.15.57:0/1883038626 conn(0x7f6f940aec40 msgr2=0x7f6f940af050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:02:15.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.511+0000 7f6f9c5d0640 1 -- 172.21.15.57:0/1883038626 shutdown_connections 2023-10-19T14:02:15.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.511+0000 7f6f9c5d0640 1 -- 172.21.15.57:0/1883038626 wait complete. 2023-10-19T14:02:15.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.512+0000 7f6f9c5d0640 1 Processor -- start 2023-10-19T14:02:15.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.512+0000 7f6f9c5d0640 1 -- start start 2023-10-19T14:02:15.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.513+0000 7f6f9c5d0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6f7c003f20 0x7f6f9414bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:15.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.513+0000 7f6f9c5d0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6f94149920 0x7f6f94154280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:15.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.513+0000 7f6f9c5d0640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6f94149da0 0x7f6f941547c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:15.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.513+0000 7f6f9c5d0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f9414aec0 con 0x7f6f94149920 2023-10-19T14:02:15.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.513+0000 7f6f9c5d0640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f9414c410 con 0x7f6f7c003f20 2023-10-19T14:02:15.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.513+0000 7f6f9c5d0640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f9414b110 con 0x7f6f94149da0 2023-10-19T14:02:15.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.513+0000 7f6f99b6b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6f94149920 0x7f6f94154280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:15.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.513+0000 7f6f99b6b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6f94149920 0x7f6f94154280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59882/0 (socket says 172.21.15.57:59882) 2023-10-19T14:02:15.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.513+0000 7f6f99b6b640 1 -- 172.21.15.57:0/891603906 learned_addr learned my addr 172.21.15.57:0/891603906 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:02:15.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.513+0000 7f6f9a36c640 1 --2- 172.21.15.57:0/891603906 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6f7c003f20 0x7f6f9414bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:15.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.513+0000 7f6f9ab6d640 1 --2- 172.21.15.57:0/891603906 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6f94149da0 0x7f6f941547c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:15.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.513+0000 7f6f99b6b640 1 -- 172.21.15.57:0/891603906 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6f94149da0 msgr2=0x7f6f941547c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:15.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.513+0000 7f6f99b6b640 1 --2- 172.21.15.57:0/891603906 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6f94149da0 0x7f6f941547c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:15.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.513+0000 7f6f99b6b640 1 -- 172.21.15.57:0/891603906 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6f7c003f20 msgr2=0x7f6f9414bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:15.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.513+0000 7f6f99b6b640 1 --2- 172.21.15.57:0/891603906 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6f7c003f20 0x7f6f9414bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:15.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.513+0000 7f6f99b6b640 1 -- 172.21.15.57:0/891603906 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f9414b3f0 con 0x7f6f94149920 2023-10-19T14:02:15.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.513+0000 7f6f99b6b640 1 --2- 172.21.15.57:0/891603906 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6f94149920 0x7f6f94154280 secure :-1 s=READY pgs=2062 cs=0 l=1 rev1=1 crypto rx=0x7f6f8802cb70 tx=0x7f6f88033c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:02:15.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.513+0000 7f6f92ffd640 1 -- 172.21.15.57:0/891603906 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f88004440 con 0x7f6f94149920 2023-10-19T14:02:15.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.513+0000 7f6f92ffd640 1 -- 172.21.15.57:0/891603906 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f6f88033e40 con 0x7f6f94149920 2023-10-19T14:02:15.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.513+0000 7f6f9c5d0640 1 -- 172.21.15.57:0/891603906 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f9414b6d0 con 0x7f6f94149920 2023-10-19T14:02:15.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.513+0000 7f6f92ffd640 1 -- 172.21.15.57:0/891603906 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f88003850 con 0x7f6f94149920 2023-10-19T14:02:15.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.514+0000 7f6f9c5d0640 1 -- 172.21.15.57:0/891603906 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f9414c660 con 0x7f6f94149920 2023-10-19T14:02:15.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.514+0000 7f6f9c5d0640 1 -- 172.21.15.57:0/891603906 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f940690e0 con 0x7f6f94149920 2023-10-19T14:02:15.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.515+0000 7f6f92ffd640 1 -- 172.21.15.57:0/891603906 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 99) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6f880039b0 con 0x7f6f94149920 2023-10-19T14:02:15.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.515+0000 7f6f92ffd640 1 --2- 172.21.15.57:0/891603906 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f6f8006af90 0x7f6f8006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:15.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.515+0000 7f6f92ffd640 1 -- 172.21.15.57:0/891603906 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f6f880ac710 con 0x7f6f94149920 2023-10-19T14:02:15.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.516+0000 7f6f9a36c640 1 --2- 172.21.15.57:0/891603906 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f6f8006af90 0x7f6f8006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:15.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.516+0000 7f6f9a36c640 1 --2- 172.21.15.57:0/891603906 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f6f8006af90 0x7f6f8006d450 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6f9406c630 tx=0x7f6f8c0014f0 comp rx=0 tx=0).ready entity=mgr.8315 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:02:15.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.520+0000 7f6f92ffd640 1 -- 172.21.15.57:0/891603906 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6f88077450 con 0x7f6f94149920 2023-10-19T14:02:15.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.731+0000 7f6f9c5d0640 1 -- 172.21.15.57:0/891603906 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f6f940cadf0 con 0x7f6f94149920 2023-10-19T14:02:15.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.731+0000 7f6f92ffd640 1 -- 172.21.15.57:0/891603906 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v77) v1 ==== 53+0+73 (secure 0 0 0) 0x7f6f880aae60 con 0x7f6f94149920 2023-10-19T14:02:15.734 INFO:teuthology.orchestra.run.smithi057.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-19T14:02:15.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.733+0000 7f6f9c5d0640 1 -- 172.21.15.57:0/891603906 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f6f8006af90 msgr2=0x7f6f8006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:15.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.733+0000 7f6f9c5d0640 1 --2- 172.21.15.57:0/891603906 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f6f8006af90 0x7f6f8006d450 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6f9406c630 tx=0x7f6f8c0014f0 comp rx=0 tx=0).stop 2023-10-19T14:02:15.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.733+0000 7f6f9c5d0640 1 -- 172.21.15.57:0/891603906 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6f94149920 msgr2=0x7f6f94154280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:15.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.733+0000 7f6f9c5d0640 1 --2- 172.21.15.57:0/891603906 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6f94149920 0x7f6f94154280 secure :-1 s=READY pgs=2062 cs=0 l=1 rev1=1 crypto rx=0x7f6f8802cb70 tx=0x7f6f88033c10 comp rx=0 tx=0).stop 2023-10-19T14:02:15.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.733+0000 7f6f9c5d0640 1 -- 172.21.15.57:0/891603906 shutdown_connections 2023-10-19T14:02:15.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.733+0000 7f6f9c5d0640 1 --2- 172.21.15.57:0/891603906 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6f94149da0 0x7f6f941547c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:15.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.733+0000 7f6f9c5d0640 1 --2- 172.21.15.57:0/891603906 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f6f8006af90 0x7f6f8006d450 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:15.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.733+0000 7f6f9c5d0640 1 --2- 172.21.15.57:0/891603906 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6f94149920 0x7f6f94154280 unknown :-1 s=CLOSED pgs=2062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:15.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.733+0000 7f6f9c5d0640 1 --2- 172.21.15.57:0/891603906 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6f7c003f20 0x7f6f9414bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:15.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.733+0000 7f6f9c5d0640 1 -- 172.21.15.57:0/891603906 >> 172.21.15.57:0/891603906 conn(0x7f6f940aec40 msgr2=0x7f6f9414a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:02:15.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.733+0000 7f6f9c5d0640 1 -- 172.21.15.57:0/891603906 shutdown_connections 2023-10-19T14:02:15.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:15.733+0000 7f6f9c5d0640 1 -- 172.21.15.57:0/891603906 wait complete. 2023-10-19T14:02:15.746 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-10-19T14:02:15.747 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-19T14:02:15.747 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi057.front.sepia.ceph.com 2023-10-19T14:02:15.747 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-19T14:02:15.748 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-19T14:02:15.748 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-10-19T14:02:15.748 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-10-19T14:02:15.749 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-19T14:02:15.749 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-19T14:02:15.749 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:02:15.750 DEBUG:teuthology.orchestra.run.smithi057:> ip addr 2023-10-19T14:02:15.766 INFO:teuthology.orchestra.run.smithi057.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-19T14:02:15.766 INFO:teuthology.orchestra.run.smithi057.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-19T14:02:15.767 INFO:teuthology.orchestra.run.smithi057.stdout: inet 127.0.0.1/8 scope host lo 2023-10-19T14:02:15.767 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T14:02:15.767 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 ::1/128 scope host 2023-10-19T14:02:15.767 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T14:02:15.767 INFO:teuthology.orchestra.run.smithi057.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-19T14:02:15.768 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:6b:ff:86 brd ff:ff:ff:ff:ff:ff 2023-10-19T14:02:15.768 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp7s0f0 2023-10-19T14:02:15.768 INFO:teuthology.orchestra.run.smithi057.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-19T14:02:15.768 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:6b:ff:87 brd ff:ff:ff:ff:ff:ff 2023-10-19T14:02:15.768 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp7s0f1 2023-10-19T14:02:15.769 INFO:teuthology.orchestra.run.smithi057.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-19T14:02:15.769 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:bd:15:62 brd ff:ff:ff:ff:ff:ff 2023-10-19T14:02:15.769 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp1s0f0 2023-10-19T14:02:15.769 INFO:teuthology.orchestra.run.smithi057.stdout: inet 172.21.15.57/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-19T14:02:15.769 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft 37975sec preferred_lft 37975sec 2023-10-19T14:02:15.770 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 fe80::ec4:7aff:febd:1562/64 scope link noprefixroute 2023-10-19T14:02:15.770 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T14:02:15.770 INFO:teuthology.orchestra.run.smithi057.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-19T14:02:15.770 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:bd:15:63 brd ff:ff:ff:ff:ff:ff 2023-10-19T14:02:15.771 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp1s0f1 2023-10-19T14:02:15.771 INFO:teuthology.orchestra.run.smithi057.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-19T14:02:15.771 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0e:fe:f1:b5:41:59 brd ff:ff:ff:ff:ff:ff 2023-10-19T14:02:15.771 INFO:teuthology.orchestra.run.smithi057.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-19T14:02:15.771 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T14:02:15.772 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 fe80::64b1:a2ff:feb3:b08/64 scope link 2023-10-19T14:02:15.772 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T14:02:15.772 INFO:teuthology.orchestra.run.smithi057.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-19T14:02:15.772 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0e:fe:f1:b5:41:59 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T14:02:15.773 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 fe80::cfe:f1ff:feb5:4159/64 scope link 2023-10-19T14:02:15.773 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T14:02:15.773 INFO:teuthology.orchestra.run.smithi057.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-19T14:02:15.773 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 62:53:72:9d:52:71 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-19T14:02:15.773 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 fe80::6053:72ff:fe9d:5271/64 scope link 2023-10-19T14:02:15.773 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T14:02:15.774 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:02:15.774 DEBUG:teuthology.orchestra.run.smithi057:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-19T14:02:15.835 INFO:teuthology.orchestra.run.smithi057.stdout:1 2023-10-19T14:02:15.837 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:02:15.837 DEBUG:teuthology.orchestra.run.smithi057:> route 2023-10-19T14:02:15.893 INFO:teuthology.orchestra.run.smithi057.stdout:Kernel IP routing table 2023-10-19T14:02:15.893 INFO:teuthology.orchestra.run.smithi057.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-19T14:02:15.893 INFO:teuthology.orchestra.run.smithi057.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-19T14:02:15.894 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-19T14:02:15.894 INFO:teuthology.orchestra.run.smithi057.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-19T14:02:15.894 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:02:15.895 DEBUG:teuthology.orchestra.run.smithi057:> (cd / && exec bash -c ' 2023-10-19T14:02:15.895 DEBUG:teuthology.orchestra.run.smithi057:> set -e 2023-10-19T14:02:15.895 DEBUG:teuthology.orchestra.run.smithi057:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-19T14:02:15.896 DEBUG:teuthology.orchestra.run.smithi057:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-19T14:02:15.896 DEBUG:teuthology.orchestra.run.smithi057:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-19T14:02:15.896 DEBUG:teuthology.orchestra.run.smithi057:> ') 2023-10-19T14:02:15.999 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:02:15.999 DEBUG:teuthology.orchestra.run.smithi057:> ip netns list 2023-10-19T14:02:16.014 INFO:teuthology.orchestra.run.smithi057.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-19T14:02:16.015 INFO:teuthology.orchestra.run.smithi057.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-19T14:02:16.015 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-19T14:02:16.015 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:02:16.016 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip link set brx.0 up 2023-10-19T14:02:16.087 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-10-19T14:02:16.087 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-19T14:02:16.087 DEBUG:teuthology.orchestra.run.smithi057:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-19T14:02:16.142 INFO:teuthology.orchestra.run.smithi057.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-19T14:02:16.143 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-19T14:02:16.143 DEBUG:teuthology.orchestra.run.smithi057:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-19T14:02:16.198 DEBUG:teuthology.orchestra.run.smithi057:> sudo modprobe fuse 2023-10-19T14:02:16.269 DEBUG:teuthology.orchestra.run.smithi057:> cat /proc/self/mounts | awk '{print $2}' 2023-10-19T14:02:16.325 INFO:teuthology.orchestra.run.smithi057.stdout:/proc 2023-10-19T14:02:16.325 INFO:teuthology.orchestra.run.smithi057.stdout:/sys 2023-10-19T14:02:16.326 INFO:teuthology.orchestra.run.smithi057.stdout:/dev 2023-10-19T14:02:16.326 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/security 2023-10-19T14:02:16.326 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/shm 2023-10-19T14:02:16.326 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/pts 2023-10-19T14:02:16.326 INFO:teuthology.orchestra.run.smithi057.stdout:/run 2023-10-19T14:02:16.327 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/cgroup 2023-10-19T14:02:16.327 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/pstore 2023-10-19T14:02:16.327 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/bpf 2023-10-19T14:02:16.327 INFO:teuthology.orchestra.run.smithi057.stdout:/ 2023-10-19T14:02:16.327 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/selinux 2023-10-19T14:02:16.327 INFO:teuthology.orchestra.run.smithi057.stdout:/proc/sys/fs/binfmt_misc 2023-10-19T14:02:16.328 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/hugepages 2023-10-19T14:02:16.328 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/mqueue 2023-10-19T14:02:16.328 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/debug 2023-10-19T14:02:16.328 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/tracing 2023-10-19T14:02:16.328 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/fuse/connections 2023-10-19T14:02:16.329 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/config 2023-10-19T14:02:16.329 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-sysctl.service 2023-10-19T14:02:16.329 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-10-19T14:02:16.329 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-10-19T14:02:16.329 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/nfs/rpc_pipefs 2023-10-19T14:02:16.330 INFO:teuthology.orchestra.run.smithi057.stdout:/run/user/1000 2023-10-19T14:02:16.330 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph 2023-10-19T14:02:16.330 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-0 2023-10-19T14:02:16.330 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-1 2023-10-19T14:02:16.330 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-2 2023-10-19T14:02:16.331 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-3 2023-10-19T14:02:16.331 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns 2023-10-19T14:02:16.331 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T14:02:16.331 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T14:02:16.331 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-19T14:02:16.332 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-19T14:02:16.332 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:02:16.332 DEBUG:teuthology.orchestra.run.smithi057:> ls /sys/fs/fuse/connections 2023-10-19T14:02:16.380 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-10-19T14:02:16.381 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-10-19T14:02:16.423 DEBUG:teuthology.orchestra.run.smithi057:> sudo modprobe fuse 2023-10-19T14:02:16.453 DEBUG:teuthology.orchestra.run.smithi057:> cat /proc/self/mounts | awk '{print $2}' 2023-10-19T14:02:16.509 INFO:teuthology.orchestra.run.smithi057.stdout:/proc 2023-10-19T14:02:16.509 INFO:teuthology.orchestra.run.smithi057.stdout:/sys 2023-10-19T14:02:16.510 INFO:teuthology.orchestra.run.smithi057.stdout:/dev 2023-10-19T14:02:16.510 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/security 2023-10-19T14:02:16.510 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/shm 2023-10-19T14:02:16.510 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/pts 2023-10-19T14:02:16.510 INFO:teuthology.orchestra.run.smithi057.stdout:/run 2023-10-19T14:02:16.511 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/cgroup 2023-10-19T14:02:16.511 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/pstore 2023-10-19T14:02:16.511 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/bpf 2023-10-19T14:02:16.511 INFO:teuthology.orchestra.run.smithi057.stdout:/ 2023-10-19T14:02:16.511 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/selinux 2023-10-19T14:02:16.512 INFO:teuthology.orchestra.run.smithi057.stdout:/proc/sys/fs/binfmt_misc 2023-10-19T14:02:16.512 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/hugepages 2023-10-19T14:02:16.512 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/mqueue 2023-10-19T14:02:16.512 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/debug 2023-10-19T14:02:16.512 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/tracing 2023-10-19T14:02:16.513 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/fuse/connections 2023-10-19T14:02:16.513 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/config 2023-10-19T14:02:16.513 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-sysctl.service 2023-10-19T14:02:16.513 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-10-19T14:02:16.513 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-10-19T14:02:16.514 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/nfs/rpc_pipefs 2023-10-19T14:02:16.514 INFO:teuthology.orchestra.run.smithi057.stdout:/run/user/1000 2023-10-19T14:02:16.514 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph 2023-10-19T14:02:16.514 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-0 2023-10-19T14:02:16.514 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-1 2023-10-19T14:02:16.515 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-2 2023-10-19T14:02:16.515 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-3 2023-10-19T14:02:16.515 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns 2023-10-19T14:02:16.515 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T14:02:16.515 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T14:02:16.516 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-19T14:02:16.516 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-19T14:02:16.516 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:02:16.516 DEBUG:teuthology.orchestra.run.smithi057:> ls /sys/fs/fuse/connections 2023-10-19T14:02:16.518 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi057.stderr:2023-10-19T14:02:16.513+0000 7f19d8e3d480 -1 init, newargv = 0x563010039cd0 newargc=15 2023-10-19T14:02:16.518 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi057.stderr:ceph-fuse[145152]: starting ceph client 2023-10-19T14:02:16.520 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi057.stderr:ceph-fuse[145152]: starting fuse 2023-10-19T14:02:16.533 INFO:teuthology.orchestra.run.smithi057.stdout:43 2023-10-19T14:02:16.533 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2023-10-19T14:02:16.533 DEBUG:teuthology.orchestra.run.smithi057:> sudo stdin-killer -- python3 -c ' 2023-10-19T14:02:16.534 DEBUG:teuthology.orchestra.run.smithi057:> import glob 2023-10-19T14:02:16.534 DEBUG:teuthology.orchestra.run.smithi057:> import re 2023-10-19T14:02:16.535 DEBUG:teuthology.orchestra.run.smithi057:> import os 2023-10-19T14:02:16.535 DEBUG:teuthology.orchestra.run.smithi057:> import subprocess 2023-10-19T14:02:16.535 DEBUG:teuthology.orchestra.run.smithi057:> 2023-10-19T14:02:16.536 DEBUG:teuthology.orchestra.run.smithi057:> def _find_admin_socket(client_name): 2023-10-19T14:02:16.536 DEBUG:teuthology.orchestra.run.smithi057:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-10-19T14:02:16.536 DEBUG:teuthology.orchestra.run.smithi057:> files = glob.glob(asok_path) 2023-10-19T14:02:16.537 DEBUG:teuthology.orchestra.run.smithi057:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-10-19T14:02:16.537 DEBUG:teuthology.orchestra.run.smithi057:> 2023-10-19T14:02:16.538 DEBUG:teuthology.orchestra.run.smithi057:> # Given a non-glob path, it better be there 2023-10-19T14:02:16.538 DEBUG:teuthology.orchestra.run.smithi057:> if "*" not in asok_path: 2023-10-19T14:02:16.538 DEBUG:teuthology.orchestra.run.smithi057:> assert(len(files) == 1) 2023-10-19T14:02:16.539 DEBUG:teuthology.orchestra.run.smithi057:> return files[0] 2023-10-19T14:02:16.539 DEBUG:teuthology.orchestra.run.smithi057:> 2023-10-19T14:02:16.539 DEBUG:teuthology.orchestra.run.smithi057:> for f in files: 2023-10-19T14:02:16.540 DEBUG:teuthology.orchestra.run.smithi057:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-10-19T14:02:16.540 DEBUG:teuthology.orchestra.run.smithi057:> if os.path.exists("/proc/{0}".format(pid)): 2023-10-19T14:02:16.540 DEBUG:teuthology.orchestra.run.smithi057:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-10-19T14:02:16.541 DEBUG:teuthology.orchestra.run.smithi057:> contents = proc_f.read() 2023-10-19T14:02:16.541 DEBUG:teuthology.orchestra.run.smithi057:> if mountpoint in contents: 2023-10-19T14:02:16.542 DEBUG:teuthology.orchestra.run.smithi057:> return f 2023-10-19T14:02:16.542 DEBUG:teuthology.orchestra.run.smithi057:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-10-19T14:02:16.542 DEBUG:teuthology.orchestra.run.smithi057:> 2023-10-19T14:02:16.543 DEBUG:teuthology.orchestra.run.smithi057:> print(_find_admin_socket("client.0")) 2023-10-19T14:02:16.543 DEBUG:teuthology.orchestra.run.smithi057:> ' 2023-10-19T14:02:16.658 INFO:teuthology.orchestra.run.smithi057.stdout:/var/run/ceph/ceph-client.0.145152.asok 2023-10-19T14:02:16.661 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:16 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2023-10-19T14:02:16.669 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.145152.asok 2023-10-19T14:02:16.669 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:02:16.670 DEBUG:teuthology.orchestra.run.smithi057:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.145152.asok status 2023-10-19T14:02:16.810 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2023-10-19T14:02:16.811 INFO:teuthology.orchestra.run.smithi057.stdout: "metadata": { 2023-10-19T14:02:16.811 INFO:teuthology.orchestra.run.smithi057.stdout: "ceph_sha1": "f7420d7f8e55ceea786dcbf536c338ce920c21eb", 2023-10-19T14:02:16.811 INFO:teuthology.orchestra.run.smithi057.stdout: "ceph_version": "ceph version 18.0.0-6786-gf7420d7f (f7420d7f8e55ceea786dcbf536c338ce920c21eb) reef (dev)", 2023-10-19T14:02:16.811 INFO:teuthology.orchestra.run.smithi057.stdout: "entity_id": "0", 2023-10-19T14:02:16.812 INFO:teuthology.orchestra.run.smithi057.stdout: "hostname": "smithi057", 2023-10-19T14:02:16.812 INFO:teuthology.orchestra.run.smithi057.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-10-19T14:02:16.812 INFO:teuthology.orchestra.run.smithi057.stdout: "pid": "145152", 2023-10-19T14:02:16.812 INFO:teuthology.orchestra.run.smithi057.stdout: "root": "/" 2023-10-19T14:02:16.813 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:02:16.813 INFO:teuthology.orchestra.run.smithi057.stdout: "dentry_count": 0, 2023-10-19T14:02:16.813 INFO:teuthology.orchestra.run.smithi057.stdout: "dentry_pinned_count": 0, 2023-10-19T14:02:16.813 INFO:teuthology.orchestra.run.smithi057.stdout: "id": 9670, 2023-10-19T14:02:16.813 INFO:teuthology.orchestra.run.smithi057.stdout: "inst": { 2023-10-19T14:02:16.813 INFO:teuthology.orchestra.run.smithi057.stdout: "name": { 2023-10-19T14:02:16.814 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "client", 2023-10-19T14:02:16.814 INFO:teuthology.orchestra.run.smithi057.stdout: "num": 9670 2023-10-19T14:02:16.814 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:02:16.814 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": { 2023-10-19T14:02:16.814 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "v1", 2023-10-19T14:02:16.815 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": "172.21.15.57:0", 2023-10-19T14:02:16.815 INFO:teuthology.orchestra.run.smithi057.stdout: "nonce": 4268064765 2023-10-19T14:02:16.815 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:02:16.815 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:02:16.815 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": { 2023-10-19T14:02:16.816 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "v1", 2023-10-19T14:02:16.816 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": "172.21.15.57:0", 2023-10-19T14:02:16.816 INFO:teuthology.orchestra.run.smithi057.stdout: "nonce": 4268064765 2023-10-19T14:02:16.816 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:02:16.816 INFO:teuthology.orchestra.run.smithi057.stdout: "inst_str": "client.9670 172.21.15.57:0/4268064765", 2023-10-19T14:02:16.817 INFO:teuthology.orchestra.run.smithi057.stdout: "addr_str": "172.21.15.57:0/4268064765", 2023-10-19T14:02:16.817 INFO:teuthology.orchestra.run.smithi057.stdout: "inode_count": 1, 2023-10-19T14:02:16.817 INFO:teuthology.orchestra.run.smithi057.stdout: "mds_epoch": 77, 2023-10-19T14:02:16.817 INFO:teuthology.orchestra.run.smithi057.stdout: "osd_epoch": 188, 2023-10-19T14:02:16.817 INFO:teuthology.orchestra.run.smithi057.stdout: "osd_epoch_barrier": 0, 2023-10-19T14:02:16.817 INFO:teuthology.orchestra.run.smithi057.stdout: "blocklisted": false, 2023-10-19T14:02:16.818 INFO:teuthology.orchestra.run.smithi057.stdout: "fs_name": "cephfs" 2023-10-19T14:02:16.818 INFO:teuthology.orchestra.run.smithi057.stdout:} 2023-10-19T14:02:16.819 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:02:16.819 DEBUG:teuthology.orchestra.run.smithi057:> stat --file-system '--printf=%T 2023-10-19T14:02:16.820 DEBUG:teuthology.orchestra.run.smithi057:> ' -- /home/ubuntu/cephtest/mnt.0 2023-10-19T14:02:16.836 INFO:teuthology.orchestra.run.smithi057.stdout:fuseblk 2023-10-19T14:02:16.836 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-10-19T14:02:16.837 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:02:16.837 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-19T14:02:16.908 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-19T14:02:17.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.057+0000 7f82f3afc640 1 -- 172.21.15.57:0/586024035 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f82d80058d0 msgr2=0x7f82d8005d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:17.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.057+0000 7f82f3afc640 1 --2- 172.21.15.57:0/586024035 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f82d80058d0 0x7f82d8005d50 secure :-1 s=READY pgs=2064 cs=0 l=1 rev1=1 crypto rx=0x7f82dc009b50 tx=0x7f82d8006900 comp rx=0 tx=0).stop 2023-10-19T14:02:17.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.057+0000 7f82f3afc640 1 -- 172.21.15.57:0/586024035 shutdown_connections 2023-10-19T14:02:17.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.057+0000 7f82f3afc640 1 --2- 172.21.15.57:0/586024035 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f82d80058d0 0x7f82d8005d50 unknown :-1 s=CLOSED pgs=2064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:17.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.057+0000 7f82f3afc640 1 -- 172.21.15.57:0/586024035 >> 172.21.15.57:0/586024035 conn(0x7f82ec069930 msgr2=0x7f82ec069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:02:17.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.057+0000 7f82f3afc640 1 -- 172.21.15.57:0/586024035 shutdown_connections 2023-10-19T14:02:17.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.057+0000 7f82f3afc640 1 -- 172.21.15.57:0/586024035 wait complete. 2023-10-19T14:02:17.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.059+0000 7f82f3afc640 1 Processor -- start 2023-10-19T14:02:17.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.059+0000 7f82f3afc640 1 -- start start 2023-10-19T14:02:17.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.059+0000 7f82f3afc640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f82d80058d0 0x7f82ec13d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:17.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.059+0000 7f82f3afc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f82ec14b730 0x7f82ec14bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:17.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.059+0000 7f82f3afc640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f82ec1e50a0 0x7f82ec1e5520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:17.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.059+0000 7f82f3afc640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82ec1e5a60 con 0x7f82ec14b730 2023-10-19T14:02:17.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.059+0000 7f82f3afc640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82ec148a90 con 0x7f82ec1e50a0 2023-10-19T14:02:17.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.059+0000 7f82f3afc640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82ec0ac730 con 0x7f82d80058d0 2023-10-19T14:02:17.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.059+0000 7f82f1097640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f82ec14b730 0x7f82ec14bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:17.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.059+0000 7f82f1097640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f82ec14b730 0x7f82ec14bbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50082/0 (socket says 172.21.15.57:50082) 2023-10-19T14:02:17.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.059+0000 7f82f1097640 1 -- 172.21.15.57:0/4685420 learned_addr learned my addr 172.21.15.57:0/4685420 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:02:17.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.060+0000 7f82f1898640 1 --2- 172.21.15.57:0/4685420 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f82d80058d0 0x7f82ec13d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:17.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.060+0000 7f82f2099640 1 --2- 172.21.15.57:0/4685420 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f82ec1e50a0 0x7f82ec1e5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:17.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.060+0000 7f82f1097640 1 -- 172.21.15.57:0/4685420 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f82d80058d0 msgr2=0x7f82ec13d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:17.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.060+0000 7f82f1097640 1 --2- 172.21.15.57:0/4685420 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f82d80058d0 0x7f82ec13d360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:17.067 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.060+0000 7f82f1097640 1 -- 172.21.15.57:0/4685420 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f82ec1e50a0 msgr2=0x7f82ec1e5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:17.067 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.060+0000 7f82f1097640 1 --2- 172.21.15.57:0/4685420 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f82ec1e50a0 0x7f82ec1e5520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:17.067 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.060+0000 7f82f1097640 1 -- 172.21.15.57:0/4685420 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82ec0aca10 con 0x7f82ec14b730 2023-10-19T14:02:17.067 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.060+0000 7f82f1097640 1 --2- 172.21.15.57:0/4685420 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f82ec14b730 0x7f82ec14bbb0 secure :-1 s=READY pgs=2065 cs=0 l=1 rev1=1 crypto rx=0x7f82dc02cb70 tx=0x7f82dc033c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:02:17.067 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.060+0000 7f82e27fc640 1 -- 172.21.15.57:0/4685420 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82dc004440 con 0x7f82ec14b730 2023-10-19T14:02:17.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.060+0000 7f82e27fc640 1 -- 172.21.15.57:0/4685420 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f82dc0035f0 con 0x7f82ec14b730 2023-10-19T14:02:17.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.060+0000 7f82f3afc640 1 -- 172.21.15.57:0/4685420 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82ec0accf0 con 0x7f82ec14b730 2023-10-19T14:02:17.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.060+0000 7f82e27fc640 1 -- 172.21.15.57:0/4685420 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82dc0039e0 con 0x7f82ec14b730 2023-10-19T14:02:17.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.060+0000 7f82f3afc640 1 -- 172.21.15.57:0/4685420 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82ec0ad2b0 con 0x7f82ec14b730 2023-10-19T14:02:17.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.061+0000 7f82f3afc640 1 -- 172.21.15.57:0/4685420 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82ec0690e0 con 0x7f82ec14b730 2023-10-19T14:02:17.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.062+0000 7f82e27fc640 1 -- 172.21.15.57:0/4685420 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 99) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f82dc003b40 con 0x7f82ec14b730 2023-10-19T14:02:17.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.062+0000 7f82e27fc640 1 --2- 172.21.15.57:0/4685420 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f82d006af60 0x7f82d006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:17.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.062+0000 7f82e27fc640 1 -- 172.21.15.57:0/4685420 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f82dc0acab0 con 0x7f82ec14b730 2023-10-19T14:02:17.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.062+0000 7f82f1898640 1 --2- 172.21.15.57:0/4685420 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f82d006af60 0x7f82d006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:17.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.063+0000 7f82f1898640 1 --2- 172.21.15.57:0/4685420 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f82d006af60 0x7f82d006d420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f82ec1f49d0 tx=0x7f82e4001c70 comp rx=0 tx=0).ready entity=mgr.8315 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:02:17.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.066+0000 7f82e27fc640 1 -- 172.21.15.57:0/4685420 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f82dc077890 con 0x7f82ec14b730 2023-10-19T14:02:17.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.266+0000 7f82f3afc640 1 -- 172.21.15.57:0/4685420 --> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f82ec1e5a60 con 0x7f82d006af60 2023-10-19T14:02:17.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.269+0000 7f82e27fc640 1 -- 172.21.15.57:0/4685420 <== mgr.8315 v2:172.21.15.179:6832/1426553632 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f82ec1e5a60 con 0x7f82d006af60 2023-10-19T14:02:17.271 INFO:teuthology.orchestra.run.smithi057.stdout:[ 2023-10-19T14:02:17.271 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:02:17.271 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "cephfs" 2023-10-19T14:02:17.271 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:02:17.272 INFO:teuthology.orchestra.run.smithi057.stdout:] 2023-10-19T14:02:17.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.270+0000 7f82f3afc640 1 -- 172.21.15.57:0/4685420 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f82d006af60 msgr2=0x7f82d006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:17.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.270+0000 7f82f3afc640 1 --2- 172.21.15.57:0/4685420 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f82d006af60 0x7f82d006d420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f82ec1f49d0 tx=0x7f82e4001c70 comp rx=0 tx=0).stop 2023-10-19T14:02:17.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.270+0000 7f82f3afc640 1 -- 172.21.15.57:0/4685420 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f82ec14b730 msgr2=0x7f82ec14bbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:17.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.270+0000 7f82f3afc640 1 --2- 172.21.15.57:0/4685420 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f82ec14b730 0x7f82ec14bbb0 secure :-1 s=READY pgs=2065 cs=0 l=1 rev1=1 crypto rx=0x7f82dc02cb70 tx=0x7f82dc033c10 comp rx=0 tx=0).stop 2023-10-19T14:02:17.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.270+0000 7f82f3afc640 1 -- 172.21.15.57:0/4685420 shutdown_connections 2023-10-19T14:02:17.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.270+0000 7f82f3afc640 1 --2- 172.21.15.57:0/4685420 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f82ec1e50a0 0x7f82ec1e5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:17.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.270+0000 7f82f3afc640 1 --2- 172.21.15.57:0/4685420 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f82d006af60 0x7f82d006d420 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:17.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.270+0000 7f82f3afc640 1 --2- 172.21.15.57:0/4685420 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f82ec14b730 0x7f82ec14bbb0 unknown :-1 s=CLOSED pgs=2065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:17.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.270+0000 7f82f3afc640 1 --2- 172.21.15.57:0/4685420 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f82d80058d0 0x7f82ec13d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:17.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.270+0000 7f82f3afc640 1 -- 172.21.15.57:0/4685420 >> 172.21.15.57:0/4685420 conn(0x7f82ec069930 msgr2=0x7f82ec069d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:02:17.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.270+0000 7f82f3afc640 1 -- 172.21.15.57:0/4685420 shutdown_connections 2023-10-19T14:02:17.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.270+0000 7f82f3afc640 1 -- 172.21.15.57:0/4685420 wait complete. 2023-10-19T14:02:17.283 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-19T14:02:17.394 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.392+0000 7f02eb339640 1 -- 172.21.15.57:0/1478316395 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f02e4159bd0 msgr2=0x7f02e415c770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:17.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.392+0000 7f02eb339640 1 --2- 172.21.15.57:0/1478316395 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f02e4159bd0 0x7f02e415c770 secure :-1 s=READY pgs=2066 cs=0 l=1 rev1=1 crypto rx=0x7f02e40cbd30 tx=0x7f02d402bd30 comp rx=0 tx=0).stop 2023-10-19T14:02:17.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.392+0000 7f02eb339640 1 -- 172.21.15.57:0/1478316395 shutdown_connections 2023-10-19T14:02:17.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.392+0000 7f02eb339640 1 --2- 172.21.15.57:0/1478316395 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f02e4159bd0 0x7f02e415c770 unknown :-1 s=CLOSED pgs=2066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:17.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.392+0000 7f02eb339640 1 --2- 172.21.15.57:0/1478316395 >> v2:172.21.15.179:3300/0 conn(0x7f02e406b030 0x7f02e406b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:17.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.392+0000 7f02eb339640 1 -- 172.21.15.57:0/1478316395 >> 172.21.15.57:0/1478316395 conn(0x7f02e4069930 msgr2=0x7f02e4069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:02:17.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.392+0000 7f02eb339640 1 -- 172.21.15.57:0/1478316395 shutdown_connections 2023-10-19T14:02:17.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.392+0000 7f02eb339640 1 -- 172.21.15.57:0/1478316395 wait complete. 2023-10-19T14:02:17.398 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.393+0000 7f02eb339640 1 Processor -- start 2023-10-19T14:02:17.398 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.393+0000 7f02eb339640 1 -- start start 2023-10-19T14:02:17.399 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.394+0000 7f02eb339640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f02e406b030 0x7f02e4158a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:17.399 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.394+0000 7f02eb339640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f02e4159bd0 0x7f02e413a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:17.399 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.394+0000 7f02eb339640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f02e415ccb0 0x7f02e40ca870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:17.399 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.394+0000 7f02eb339640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02e4158f80 con 0x7f02e406b030 2023-10-19T14:02:17.399 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.394+0000 7f02eb339640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02e40aa3d0 con 0x7f02e415ccb0 2023-10-19T14:02:17.400 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.394+0000 7f02eb339640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02e4157c80 con 0x7f02e4159bd0 2023-10-19T14:02:17.400 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.394+0000 7f02e90d5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f02e406b030 0x7f02e4158a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:17.400 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.394+0000 7f02e90d5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f02e406b030 0x7f02e4158a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50096/0 (socket says 172.21.15.57:50096) 2023-10-19T14:02:17.400 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.394+0000 7f02e90d5640 1 -- 172.21.15.57:0/3992801425 learned_addr learned my addr 172.21.15.57:0/3992801425 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:02:17.400 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.394+0000 7f02e88d4640 1 --2- 172.21.15.57:0/3992801425 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f02e4159bd0 0x7f02e413a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:17.400 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.394+0000 7f02e98d6640 1 --2- 172.21.15.57:0/3992801425 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f02e415ccb0 0x7f02e40ca870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:17.401 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.394+0000 7f02e90d5640 1 -- 172.21.15.57:0/3992801425 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f02e4159bd0 msgr2=0x7f02e413a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:17.401 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.394+0000 7f02e90d5640 1 --2- 172.21.15.57:0/3992801425 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f02e4159bd0 0x7f02e413a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:17.401 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.394+0000 7f02e90d5640 1 -- 172.21.15.57:0/3992801425 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f02e415ccb0 msgr2=0x7f02e40ca870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:17.401 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.394+0000 7f02e90d5640 1 --2- 172.21.15.57:0/3992801425 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f02e415ccb0 0x7f02e40ca870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:17.401 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.394+0000 7f02e90d5640 1 -- 172.21.15.57:0/3992801425 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02d4009560 con 0x7f02e406b030 2023-10-19T14:02:17.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.394+0000 7f02e88d4640 1 --2- 172.21.15.57:0/3992801425 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f02e4159bd0 0x7f02e413a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:02:17.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.394+0000 7f02e98d6640 1 --2- 172.21.15.57:0/3992801425 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f02e415ccb0 0x7f02e40ca870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:02:17.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.394+0000 7f02e90d5640 1 --2- 172.21.15.57:0/3992801425 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f02e406b030 0x7f02e4158a40 secure :-1 s=READY pgs=2067 cs=0 l=1 rev1=1 crypto rx=0x7f02e000d770 tx=0x7f02e000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:02:17.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.394+0000 7f02d9ffb640 1 -- 172.21.15.57:0/3992801425 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02e0009c30 con 0x7f02e406b030 2023-10-19T14:02:17.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.394+0000 7f02d9ffb640 1 -- 172.21.15.57:0/3992801425 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f02e000de80 con 0x7f02e406b030 2023-10-19T14:02:17.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.394+0000 7f02eb339640 1 -- 172.21.15.57:0/3992801425 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02e4157f60 con 0x7f02e406b030 2023-10-19T14:02:17.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.394+0000 7f02d9ffb640 1 -- 172.21.15.57:0/3992801425 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02e0005280 con 0x7f02e406b030 2023-10-19T14:02:17.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.395+0000 7f02eb339640 1 -- 172.21.15.57:0/3992801425 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02e4158520 con 0x7f02e406b030 2023-10-19T14:02:17.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.396+0000 7f02eb339640 1 -- 172.21.15.57:0/3992801425 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02e406c570 con 0x7f02e406b030 2023-10-19T14:02:17.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.396+0000 7f02d9ffb640 1 -- 172.21.15.57:0/3992801425 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 99) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f02e00053e0 con 0x7f02e406b030 2023-10-19T14:02:17.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.396+0000 7f02d9ffb640 1 --2- 172.21.15.57:0/3992801425 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f02c806af90 0x7f02c806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:17.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.396+0000 7f02d9ffb640 1 -- 172.21.15.57:0/3992801425 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f02e0089bf0 con 0x7f02e406b030 2023-10-19T14:02:17.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.397+0000 7f02e88d4640 1 --2- 172.21.15.57:0/3992801425 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f02c806af90 0x7f02c806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:17.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.398+0000 7f02e88d4640 1 --2- 172.21.15.57:0/3992801425 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f02c806af90 0x7f02c806d450 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f02d4035000 tx=0x7f02d4004730 comp rx=0 tx=0).ready entity=mgr.8315 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:02:17.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.401+0000 7f02d9ffb640 1 -- 172.21.15.57:0/3992801425 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f02e0054920 con 0x7f02e406b030 2023-10-19T14:02:17.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:17.628+0000 7f02eb339640 1 -- 172.21.15.57:0/3992801425 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f02e406c850 con 0x7f02e406b030 2023-10-19T14:02:18.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.098+0000 7f02d9ffb640 1 -- 172.21.15.57:0/3992801425 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v100) v1 ==== 92+0+0 (secure 0 0 0) 0x7f02e0017020 con 0x7f02e406b030 2023-10-19T14:02:18.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.101+0000 7f02eb339640 1 -- 172.21.15.57:0/3992801425 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f02c806af90 msgr2=0x7f02c806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:18.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.101+0000 7f02eb339640 1 --2- 172.21.15.57:0/3992801425 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f02c806af90 0x7f02c806d450 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f02d4035000 tx=0x7f02d4004730 comp rx=0 tx=0).stop 2023-10-19T14:02:18.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.101+0000 7f02eb339640 1 -- 172.21.15.57:0/3992801425 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f02e406b030 msgr2=0x7f02e4158a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:18.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.101+0000 7f02eb339640 1 --2- 172.21.15.57:0/3992801425 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f02e406b030 0x7f02e4158a40 secure :-1 s=READY pgs=2067 cs=0 l=1 rev1=1 crypto rx=0x7f02e000d770 tx=0x7f02e000dc30 comp rx=0 tx=0).stop 2023-10-19T14:02:18.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.101+0000 7f02eb339640 1 -- 172.21.15.57:0/3992801425 shutdown_connections 2023-10-19T14:02:18.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.101+0000 7f02eb339640 1 --2- 172.21.15.57:0/3992801425 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f02e415ccb0 0x7f02e40ca870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:18.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.101+0000 7f02eb339640 1 --2- 172.21.15.57:0/3992801425 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f02e4159bd0 0x7f02e413a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:18.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.101+0000 7f02eb339640 1 --2- 172.21.15.57:0/3992801425 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f02c806af90 0x7f02c806d450 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:18.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.101+0000 7f02eb339640 1 --2- 172.21.15.57:0/3992801425 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f02e406b030 0x7f02e4158a40 secure :-1 s=CLOSED pgs=2067 cs=0 l=1 rev1=1 crypto rx=0x7f02e000d770 tx=0x7f02e000dc30 comp rx=0 tx=0).stop 2023-10-19T14:02:18.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.101+0000 7f02eb339640 1 -- 172.21.15.57:0/3992801425 >> 172.21.15.57:0/3992801425 conn(0x7f02e4069930 msgr2=0x7f02e40ade10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:02:18.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.101+0000 7f02eb339640 1 -- 172.21.15.57:0/3992801425 shutdown_connections 2023-10-19T14:02:18.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.101+0000 7f02eb339640 1 -- 172.21.15.57:0/3992801425 wait complete. 2023-10-19T14:02:18.117 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-19T14:02:18.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.234+0000 7f6a4359e640 1 -- 172.21.15.57:0/2816516562 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6a3c0cd890 msgr2=0x7f6a3c0cc730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:18.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.234+0000 7f6a4359e640 1 --2- 172.21.15.57:0/2816516562 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6a3c0cd890 0x7f6a3c0cc730 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f6a38009830 tx=0x7f6a3802bc20 comp rx=0 tx=0).stop 2023-10-19T14:02:18.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.234+0000 7f6a4359e640 1 -- 172.21.15.57:0/2816516562 shutdown_connections 2023-10-19T14:02:18.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.234+0000 7f6a4359e640 1 --2- 172.21.15.57:0/2816516562 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6a3c15e840 0x7f6a3c0c4780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:18.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.234+0000 7f6a4359e640 1 --2- 172.21.15.57:0/2816516562 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6a3c0cd890 0x7f6a3c0cc730 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:18.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.234+0000 7f6a4359e640 1 -- 172.21.15.57:0/2816516562 >> 172.21.15.57:0/2816516562 conn(0x7f6a3c0d0c30 msgr2=0x7f6a3c0d1040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:02:18.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.234+0000 7f6a4359e640 1 -- 172.21.15.57:0/2816516562 shutdown_connections 2023-10-19T14:02:18.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.234+0000 7f6a4359e640 1 -- 172.21.15.57:0/2816516562 wait complete. 2023-10-19T14:02:18.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.235+0000 7f6a4359e640 1 Processor -- start 2023-10-19T14:02:18.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.235+0000 7f6a4359e640 1 -- start start 2023-10-19T14:02:18.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.236+0000 7f6a4359e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6a3c0cd890 0x7f6a3c1f4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:18.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.236+0000 7f6a4359e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6a3c0cdc70 0x7f6a3c0b60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:18.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.236+0000 7f6a4359e640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6a3c15e840 0x7f6a3c0b4220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:18.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.236+0000 7f6a4359e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a3c1f5440 con 0x7f6a3c0cdc70 2023-10-19T14:02:18.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.236+0000 7f6a4359e640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a3c0b65e0 con 0x7f6a3c0cd890 2023-10-19T14:02:18.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.236+0000 7f6a4359e640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a3c0b3860 con 0x7f6a3c15e840 2023-10-19T14:02:18.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.236+0000 7f6a41d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6a3c0cdc70 0x7f6a3c0b60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:18.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.236+0000 7f6a41d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6a3c0cdc70 0x7f6a3c0b60a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50118/0 (socket says 172.21.15.57:50118) 2023-10-19T14:02:18.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.236+0000 7f6a41d9b640 1 -- 172.21.15.57:0/2213714866 learned_addr learned my addr 172.21.15.57:0/2213714866 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:02:18.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.236+0000 7f6a4259c640 1 --2- 172.21.15.57:0/2213714866 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6a3c0cd890 0x7f6a3c1f4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:18.243 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.236+0000 7f6a42d9d640 1 --2- 172.21.15.57:0/2213714866 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6a3c15e840 0x7f6a3c0b4220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:18.243 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.236+0000 7f6a41d9b640 1 -- 172.21.15.57:0/2213714866 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6a3c15e840 msgr2=0x7f6a3c0b4220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:18.243 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.236+0000 7f6a41d9b640 1 --2- 172.21.15.57:0/2213714866 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6a3c15e840 0x7f6a3c0b4220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:18.243 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.236+0000 7f6a41d9b640 1 -- 172.21.15.57:0/2213714866 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6a3c0cd890 msgr2=0x7f6a3c1f4f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:18.243 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.236+0000 7f6a41d9b640 1 --2- 172.21.15.57:0/2213714866 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6a3c0cd890 0x7f6a3c1f4f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:18.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.236+0000 7f6a41d9b640 1 -- 172.21.15.57:0/2213714866 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a38009490 con 0x7f6a3c0cdc70 2023-10-19T14:02:18.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.237+0000 7f6a41d9b640 1 --2- 172.21.15.57:0/2213714866 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6a3c0cdc70 0x7f6a3c0b60a0 secure :-1 s=READY pgs=2073 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c00c780 tx=0x7f6a2c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:02:18.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.237+0000 7f6a32ffd640 1 -- 172.21.15.57:0/2213714866 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a2c005270 con 0x7f6a3c0cdc70 2023-10-19T14:02:18.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.237+0000 7f6a32ffd640 1 -- 172.21.15.57:0/2213714866 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f6a2c00a640 con 0x7f6a3c0cdc70 2023-10-19T14:02:18.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.237+0000 7f6a4359e640 1 -- 172.21.15.57:0/2213714866 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a3c0b33d0 con 0x7f6a3c0cdc70 2023-10-19T14:02:18.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.237+0000 7f6a32ffd640 1 -- 172.21.15.57:0/2213714866 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a2c00aa50 con 0x7f6a3c0cdc70 2023-10-19T14:02:18.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.237+0000 7f6a4359e640 1 -- 172.21.15.57:0/2213714866 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a3c0b2d80 con 0x7f6a3c0cdc70 2023-10-19T14:02:18.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.238+0000 7f6a4359e640 1 -- 172.21.15.57:0/2213714866 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a3c066960 con 0x7f6a3c0cdc70 2023-10-19T14:02:18.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.238+0000 7f6a32ffd640 1 -- 172.21.15.57:0/2213714866 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 100) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f6a2c00abb0 con 0x7f6a3c0cdc70 2023-10-19T14:02:18.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.239+0000 7f6a32ffd640 1 --2- 172.21.15.57:0/2213714866 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f6a2006afb0 0x7f6a2006d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:18.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.239+0000 7f6a4259c640 1 -- 172.21.15.57:0/2213714866 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f6a2006afb0 msgr2=0x7f6a2006d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/1426553632 2023-10-19T14:02:18.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.239+0000 7f6a32ffd640 1 -- 172.21.15.57:0/2213714866 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f6a2c0892e0 con 0x7f6a3c0cdc70 2023-10-19T14:02:18.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.239+0000 7f6a4259c640 1 --2- 172.21.15.57:0/2213714866 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f6a2006afb0 0x7f6a2006d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T14:02:18.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.243+0000 7f6a32ffd640 1 -- 172.21.15.57:0/2213714866 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6a2c054150 con 0x7f6a3c0cdc70 2023-10-19T14:02:18.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.422+0000 7f6a4359e640 1 -- 172.21.15.57:0/2213714866 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f6a3c0b3560 con 0x7f6a3c0cdc70 2023-10-19T14:02:18.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.425+0000 7f6a32ffd640 1 -- 172.21.15.57:0/2213714866 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v68)=0 v68) v1 ==== 138+0+0 (secure 0 0 0) 0x7f6a2c00ae60 con 0x7f6a3c0cdc70 2023-10-19T14:02:18.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.428+0000 7f6a4359e640 1 -- 172.21.15.57:0/2213714866 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f6a2006afb0 msgr2=0x7f6a2006d470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T14:02:18.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.428+0000 7f6a4359e640 1 --2- 172.21.15.57:0/2213714866 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f6a2006afb0 0x7f6a2006d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:18.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.428+0000 7f6a4359e640 1 -- 172.21.15.57:0/2213714866 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6a3c0cdc70 msgr2=0x7f6a3c0b60a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:18.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.428+0000 7f6a4359e640 1 --2- 172.21.15.57:0/2213714866 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6a3c0cdc70 0x7f6a3c0b60a0 secure :-1 s=READY pgs=2073 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c00c780 tx=0x7f6a2c00cc40 comp rx=0 tx=0).stop 2023-10-19T14:02:18.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.428+0000 7f6a4359e640 1 -- 172.21.15.57:0/2213714866 shutdown_connections 2023-10-19T14:02:18.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.428+0000 7f6a4359e640 1 --2- 172.21.15.57:0/2213714866 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6a3c15e840 0x7f6a3c0b4220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:18.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.428+0000 7f6a4359e640 1 --2- 172.21.15.57:0/2213714866 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f6a2006afb0 0x7f6a2006d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:18.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.428+0000 7f6a4359e640 1 --2- 172.21.15.57:0/2213714866 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6a3c0cdc70 0x7f6a3c0b60a0 unknown :-1 s=CLOSED pgs=2073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:18.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.428+0000 7f6a4359e640 1 --2- 172.21.15.57:0/2213714866 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6a3c0cd890 0x7f6a3c1f4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:18.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.428+0000 7f6a4359e640 1 -- 172.21.15.57:0/2213714866 >> 172.21.15.57:0/2213714866 conn(0x7f6a3c0d0c30 msgr2=0x7f6a3c15df00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:02:18.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.428+0000 7f6a4359e640 1 -- 172.21.15.57:0/2213714866 shutdown_connections 2023-10-19T14:02:18.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.428+0000 7f6a4359e640 1 -- 172.21.15.57:0/2213714866 wait complete. 2023-10-19T14:02:18.441 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-19T14:02:18.492 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:02:18.490+0000 7f2d92039040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-19T14:02:18.507 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:02:18.505+0000 7fdb7681f040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-19T14:02:18.536 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:02:18.535+0000 7f2d92039040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-19T14:02:18.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.553+0000 7f8b0601a640 1 -- 172.21.15.57:0/1655185067 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8aec004d00 msgr2=0x7f8aec005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:18.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.553+0000 7f8b0601a640 1 --2- 172.21.15.57:0/1655185067 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8aec004d00 0x7f8aec005180 secure :-1 s=READY pgs=2075 cs=0 l=1 rev1=1 crypto rx=0x7f8af8009ed0 tx=0x7f8aec0057f0 comp rx=0 tx=0).stop 2023-10-19T14:02:18.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.553+0000 7f8afffff640 1 -- 172.21.15.57:0/1655185067 reap_dead start 2023-10-19T14:02:18.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.553+0000 7f8b0601a640 1 -- 172.21.15.57:0/1655185067 shutdown_connections 2023-10-19T14:02:18.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.553+0000 7f8b0601a640 1 -- 172.21.15.57:0/1655185067 >> 172.21.15.57:0/1655185067 conn(0x7f8b000d0d60 msgr2=0x7f8b000d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:02:18.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.553+0000 7f8b0601a640 1 -- 172.21.15.57:0/1655185067 shutdown_connections 2023-10-19T14:02:18.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.553+0000 7f8b0601a640 1 -- 172.21.15.57:0/1655185067 wait complete. 2023-10-19T14:02:18.557 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:02:18.555+0000 7fdb7681f040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-19T14:02:18.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.555+0000 7f8b0601a640 1 Processor -- start 2023-10-19T14:02:18.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.555+0000 7f8b0601a640 1 -- start start 2023-10-19T14:02:18.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.556+0000 7f8b0601a640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8aec004d00 0x7f8b0015df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:18.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.556+0000 7f8b0601a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8b0015c6d0 0x7f8b001f0d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:18.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.556+0000 7f8b0601a640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8b001f1250 0x7f8b000b5960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:18.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.556+0000 7f8b0601a640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b0015cb90 con 0x7f8b0015c6d0 2023-10-19T14:02:18.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.556+0000 7f8b0601a640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b0015ce70 con 0x7f8b001f1250 2023-10-19T14:02:18.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.557+0000 7f8b0601a640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b0015d150 con 0x7f8aec004d00 2023-10-19T14:02:18.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.557+0000 7f8af6ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8b0015c6d0 0x7f8b001f0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:18.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.557+0000 7f8af6ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8b0015c6d0 0x7f8b001f0d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50124/0 (socket says 172.21.15.57:50124) 2023-10-19T14:02:18.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.557+0000 7f8aff7fe640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8aec004d00 0x7f8b0015df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:18.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.557+0000 7f8af6ffd640 1 -- 172.21.15.57:0/3288407612 learned_addr learned my addr 172.21.15.57:0/3288407612 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:02:18.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.557+0000 7f8afffff640 1 --2- 172.21.15.57:0/3288407612 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8b001f1250 0x7f8b000b5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:18.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.557+0000 7f8af6ffd640 1 -- 172.21.15.57:0/3288407612 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8aec004d00 msgr2=0x7f8b0015df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:18.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.557+0000 7f8af6ffd640 1 --2- 172.21.15.57:0/3288407612 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8aec004d00 0x7f8b0015df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:18.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.557+0000 7f8af6ffd640 1 -- 172.21.15.57:0/3288407612 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8b001f1250 msgr2=0x7f8b000b5960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:18.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.557+0000 7f8af6ffd640 1 --2- 172.21.15.57:0/3288407612 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8b001f1250 0x7f8b000b5960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:18.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.557+0000 7f8af6ffd640 1 -- 172.21.15.57:0/3288407612 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b0015d430 con 0x7f8b0015c6d0 2023-10-19T14:02:18.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.557+0000 7f8af6ffd640 1 --2- 172.21.15.57:0/3288407612 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8b0015c6d0 0x7f8b001f0d10 secure :-1 s=READY pgs=2076 cs=0 l=1 rev1=1 crypto rx=0x7f8ae8011ae0 tx=0x7f8ae8009f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:02:18.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.557+0000 7f8afcff9640 1 -- 172.21.15.57:0/3288407612 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ae800fc30 con 0x7f8b0015c6d0 2023-10-19T14:02:18.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.557+0000 7f8afcff9640 1 -- 172.21.15.57:0/3288407612 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f8ae800fd90 con 0x7f8b0015c6d0 2023-10-19T14:02:18.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.557+0000 7f8b0601a640 1 -- 172.21.15.57:0/3288407612 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b0015e490 con 0x7f8b0015c6d0 2023-10-19T14:02:18.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.557+0000 7f8afcff9640 1 -- 172.21.15.57:0/3288407612 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ae8004280 con 0x7f8b0015c6d0 2023-10-19T14:02:18.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.557+0000 7f8b0601a640 1 -- 172.21.15.57:0/3288407612 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b000b6080 con 0x7f8b0015c6d0 2023-10-19T14:02:18.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.558+0000 7f8b0601a640 1 -- 172.21.15.57:0/3288407612 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b0015d5c0 con 0x7f8b0015c6d0 2023-10-19T14:02:18.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.559+0000 7f8afcff9640 1 -- 172.21.15.57:0/3288407612 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 100) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f8ae80035f0 con 0x7f8b0015c6d0 2023-10-19T14:02:18.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.559+0000 7f8afcff9640 1 --2- 172.21.15.57:0/3288407612 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f8ae006afb0 0x7f8ae006d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:18.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.559+0000 7f8aff7fe640 1 -- 172.21.15.57:0/3288407612 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f8ae006afb0 msgr2=0x7f8ae006d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/1426553632 2023-10-19T14:02:18.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.560+0000 7f8afcff9640 1 -- 172.21.15.57:0/3288407612 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f8ae808a570 con 0x7f8b0015c6d0 2023-10-19T14:02:18.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.560+0000 7f8aff7fe640 1 --2- 172.21.15.57:0/3288407612 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f8ae006afb0 0x7f8ae006d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T14:02:18.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.564+0000 7f8afcff9640 1 -- 172.21.15.57:0/3288407612 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8ae80552b0 con 0x7f8b0015c6d0 2023-10-19T14:02:18.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.744+0000 7f8b0601a640 1 -- 172.21.15.57:0/3288407612 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f8b000690e0 con 0x7f8b0015c6d0 2023-10-19T14:02:18.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.748+0000 7f8afcff9640 1 -- 172.21.15.57:0/3288407612 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v69)=0 v69) v1 ==== 133+0+0 (secure 0 0 0) 0x7f8ae80588d0 con 0x7f8b0015c6d0 2023-10-19T14:02:18.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.750+0000 7f8b0601a640 1 -- 172.21.15.57:0/3288407612 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f8ae006afb0 msgr2=0x7f8ae006d470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T14:02:18.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.750+0000 7f8b0601a640 1 --2- 172.21.15.57:0/3288407612 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f8ae006afb0 0x7f8ae006d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:18.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.750+0000 7f8b0601a640 1 -- 172.21.15.57:0/3288407612 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8b0015c6d0 msgr2=0x7f8b001f0d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:18.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.750+0000 7f8b0601a640 1 --2- 172.21.15.57:0/3288407612 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8b0015c6d0 0x7f8b001f0d10 secure :-1 s=READY pgs=2076 cs=0 l=1 rev1=1 crypto rx=0x7f8ae8011ae0 tx=0x7f8ae8009f10 comp rx=0 tx=0).stop 2023-10-19T14:02:18.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.751+0000 7f8b0601a640 1 -- 172.21.15.57:0/3288407612 shutdown_connections 2023-10-19T14:02:18.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.751+0000 7f8b0601a640 1 --2- 172.21.15.57:0/3288407612 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8b001f1250 0x7f8b000b5960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:18.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.751+0000 7f8b0601a640 1 --2- 172.21.15.57:0/3288407612 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f8ae006afb0 0x7f8ae006d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:18.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.751+0000 7f8b0601a640 1 --2- 172.21.15.57:0/3288407612 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8b0015c6d0 0x7f8b001f0d10 unknown :-1 s=CLOSED pgs=2076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:18.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.751+0000 7f8b0601a640 1 --2- 172.21.15.57:0/3288407612 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8aec004d00 0x7f8b0015df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:18.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.751+0000 7f8b0601a640 1 -- 172.21.15.57:0/3288407612 >> 172.21.15.57:0/3288407612 conn(0x7f8b000d0d60 msgr2=0x7f8b0006b540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:02:18.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.751+0000 7f8b0601a640 1 -- 172.21.15.57:0/3288407612 shutdown_connections 2023-10-19T14:02:18.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.751+0000 7f8b0601a640 1 -- 172.21.15.57:0/3288407612 wait complete. 2023-10-19T14:02:18.764 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:02:18.765 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-10-19T14:02:18.785 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:02:18.786 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/test_restart) 2023-10-19T14:02:18.844 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/test_restart 1m 2023-10-19T14:02:18.931 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:02:18.929+0000 7f2d92039040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-19T14:02:18.974 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:02:18.972+0000 7f2d92039040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-19T14:02:18.977 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:02:18.975+0000 7fdb7681f040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-19T14:02:18.998 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.995+0000 7f23bfb97640 1 -- 172.21.15.57:0/2845747394 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f23b815b260 msgr2=0x7f23b80cf310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:18.998 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.995+0000 7f23bfb97640 1 --2- 172.21.15.57:0/2845747394 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f23b815b260 0x7f23b80cf310 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f23a8009860 tx=0x7f23a802bde0 comp rx=0 tx=0).stop 2023-10-19T14:02:18.999 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.995+0000 7f23bfb97640 1 -- 172.21.15.57:0/2845747394 shutdown_connections 2023-10-19T14:02:18.999 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.995+0000 7f23bfb97640 1 --2- 172.21.15.57:0/2845747394 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f23b815b260 0x7f23b80cf310 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:18.999 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.995+0000 7f23bfb97640 1 -- 172.21.15.57:0/2845747394 >> 172.21.15.57:0/2845747394 conn(0x7f23b809a030 msgr2=0x7f23b80d0d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:02:18.999 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.995+0000 7f23bfb97640 1 -- 172.21.15.57:0/2845747394 shutdown_connections 2023-10-19T14:02:18.999 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.995+0000 7f23bfb97640 1 -- 172.21.15.57:0/2845747394 wait complete. 2023-10-19T14:02:19.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.996+0000 7f23bfb97640 1 Processor -- start 2023-10-19T14:02:19.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.997+0000 7f23bfb97640 1 -- start start 2023-10-19T14:02:19.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.997+0000 7f23bfb97640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f23b80cf850 0x7f23b80b3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:19.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.997+0000 7f23bfb97640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f23b815b260 0x7f23b815e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:19.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.997+0000 7f23bfb97640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f23b815b640 0x7f23b8159100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:19.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.997+0000 7f23bfb97640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23b81f54d0 con 0x7f23b815b260 2023-10-19T14:02:19.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.997+0000 7f23bfb97640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23b815bac0 con 0x7f23b815b640 2023-10-19T14:02:19.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.997+0000 7f23bfb97640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23b815bda0 con 0x7f23b80cf850 2023-10-19T14:02:19.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.997+0000 7f23bd132640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f23b815b260 0x7f23b815e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:19.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.997+0000 7f23bd132640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f23b815b260 0x7f23b815e1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50126/0 (socket says 172.21.15.57:50126) 2023-10-19T14:02:19.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.997+0000 7f23bd132640 1 -- 172.21.15.57:0/2662469376 learned_addr learned my addr 172.21.15.57:0/2662469376 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:02:19.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.997+0000 7f23bd933640 1 --2- 172.21.15.57:0/2662469376 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f23b80cf850 0x7f23b80b3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:19.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.997+0000 7f23be134640 1 --2- 172.21.15.57:0/2662469376 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f23b815b640 0x7f23b8159100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:19.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.998+0000 7f23bd132640 1 -- 172.21.15.57:0/2662469376 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f23b80cf850 msgr2=0x7f23b80b3ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:19.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.998+0000 7f23bd132640 1 --2- 172.21.15.57:0/2662469376 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f23b80cf850 0x7f23b80b3ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:19.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.998+0000 7f23bd132640 1 -- 172.21.15.57:0/2662469376 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f23b815b640 msgr2=0x7f23b8159100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:19.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.998+0000 7f23bd132640 1 --2- 172.21.15.57:0/2662469376 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f23b815b640 0x7f23b8159100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:19.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.998+0000 7f23bd132640 1 -- 172.21.15.57:0/2662469376 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23a8009490 con 0x7f23b815b260 2023-10-19T14:02:19.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.998+0000 7f23bd132640 1 --2- 172.21.15.57:0/2662469376 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f23b815b260 0x7f23b815e1b0 secure :-1 s=READY pgs=2078 cs=0 l=1 rev1=1 crypto rx=0x7f23b400c410 tx=0x7f23b400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:02:19.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.998+0000 7f23ae7fc640 1 -- 172.21.15.57:0/2662469376 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23b40035f0 con 0x7f23b815b260 2023-10-19T14:02:19.006 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.998+0000 7f23ae7fc640 1 -- 172.21.15.57:0/2662469376 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f23b4003770 con 0x7f23b815b260 2023-10-19T14:02:19.006 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.998+0000 7f23bfb97640 1 -- 172.21.15.57:0/2662469376 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23b815c080 con 0x7f23b815b260 2023-10-19T14:02:19.006 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.998+0000 7f23ae7fc640 1 -- 172.21.15.57:0/2662469376 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23b4003b80 con 0x7f23b815b260 2023-10-19T14:02:19.006 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.998+0000 7f23bfb97640 1 -- 172.21.15.57:0/2662469376 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23b80b40e0 con 0x7f23b815b260 2023-10-19T14:02:19.006 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:18.999+0000 7f23bfb97640 1 -- 172.21.15.57:0/2662469376 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23b80b4270 con 0x7f23b815b260 2023-10-19T14:02:19.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:19.000+0000 7f23ae7fc640 1 -- 172.21.15.57:0/2662469376 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 100) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f23b4003ce0 con 0x7f23b815b260 2023-10-19T14:02:19.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:19.000+0000 7f23ae7fc640 1 --2- 172.21.15.57:0/2662469376 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f239c06afb0 0x7f239c06d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:19.008 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:19.000+0000 7f23bd933640 1 -- 172.21.15.57:0/2662469376 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f239c06afb0 msgr2=0x7f239c06d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/1426553632 2023-10-19T14:02:19.008 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:19.000+0000 7f23ae7fc640 1 -- 172.21.15.57:0/2662469376 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f23b4089040 con 0x7f23b815b260 2023-10-19T14:02:19.008 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:19.000+0000 7f23bd933640 1 --2- 172.21.15.57:0/2662469376 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f239c06afb0 0x7f239c06d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T14:02:19.008 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:19.005+0000 7f23ae7fc640 1 -- 172.21.15.57:0/2662469376 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f23b4053f30 con 0x7f23b815b260 2023-10-19T14:02:19.018 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:02:19.016+0000 7f2d92039040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-19T14:02:19.025 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:02:19.023+0000 7fdb7681f040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-19T14:02:19.070 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:02:19.067+0000 7f2d92039040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-19T14:02:19.072 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:02:19.070+0000 7fdb7681f040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-19T14:02:19.113 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:02:19.111+0000 7f2d92039040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-19T14:02:19.128 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:02:19.126+0000 7fdb7681f040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-19T14:02:19.172 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:02:19.170+0000 7fdb7681f040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-19T14:02:19.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:19.201+0000 7f23bd933640 1 -- 172.21.15.57:0/2662469376 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f239c06afb0 msgr2=0x7f239c06d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/1426553632 2023-10-19T14:02:19.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:19.201+0000 7f23bd933640 1 --2- 172.21.15.57:0/2662469376 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f239c06afb0 0x7f239c06d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-19T14:02:19.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:19.203+0000 7f23bfb97640 1 -- 172.21.15.57:0/2662469376 --> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/test_restart", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f23b80675f0 con 0x7f239c06afb0 2023-10-19T14:02:19.211 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:02:19.209+0000 7f2d92039040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-19T14:02:19.264 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:02:19.261+0000 7f2d92039040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-19T14:02:19.274 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:02:19.272+0000 7fdb7681f040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-19T14:02:19.304 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:02:19.302+0000 7f2d92039040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-19T14:02:19.330 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:02:19.327+0000 7fdb7681f040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-19T14:02:19.352 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:02:19.350+0000 7f2d92039040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-19T14:02:19.373 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:02:19.371+0000 7fdb7681f040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-19T14:02:19.395 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:02:19.393+0000 7f2d92039040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-19T14:02:19.423 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:02:19.421+0000 7fdb7681f040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-19T14:02:19.468 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:02:19.465+0000 7fdb7681f040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-19T14:02:19.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:19.602+0000 7f23bd933640 1 -- 172.21.15.57:0/2662469376 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f239c06afb0 msgr2=0x7f239c06d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/1426553632 2023-10-19T14:02:19.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:19.602+0000 7f23bd933640 1 --2- 172.21.15.57:0/2662469376 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f239c06afb0 0x7f239c06d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-19T14:02:19.713 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:02:19.711+0000 7f2d92039040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-19T14:02:19.800 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:02:19.798+0000 7fdb7681f040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-19T14:02:19.846 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:02:19.841+0000 7f2d92039040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-19T14:02:19.933 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:02:19.931+0000 7f2d92039040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-19T14:02:19.938 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:02:19.936+0000 7fdb7681f040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-19T14:02:19.975 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:02:19.973+0000 7f2d92039040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-19T14:02:20.035 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:02:20.033+0000 7fdb7681f040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-19T14:02:20.069 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:02:20.066+0000 7f2d92039040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-19T14:02:20.080 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:02:20.078+0000 7fdb7681f040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-19T14:02:20.160 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:02:20.157+0000 7f2d92039040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-19T14:02:20.180 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:02:20.178+0000 7fdb7681f040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-19T14:02:20.275 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:02:20.273+0000 7fdb7681f040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-19T14:02:20.296 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:02:20.293+0000 7f2d92039040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-19T14:02:20.337 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:02:20.335+0000 7f2d92039040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-19T14:02:20.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:20.403+0000 7f23bd933640 1 -- 172.21.15.57:0/2662469376 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f239c06afb0 msgr2=0x7f239c06d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/1426553632 2023-10-19T14:02:20.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:20.403+0000 7f23bd933640 1 --2- 172.21.15.57:0/2662469376 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f239c06afb0 0x7f239c06d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-19T14:02:20.418 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:02:20.416+0000 7fdb7681f040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-19T14:02:20.442 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:02:20.439+0000 7f2d92039040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-19T14:02:20.462 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:02:20.459+0000 7fdb7681f040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-19T14:02:20.571 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:02:20.568+0000 7fdb7681f040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-19T14:02:21.061 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:02:21.058+0000 7f2d92039040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-19T14:02:21.218 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:02:21.216+0000 7fdb7681f040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-19T14:02:21.321 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:02:21.318+0000 7f2d92039040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-19T14:02:21.369 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:02:21.367+0000 7f2d92039040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-19T14:02:21.469 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:02:21.467+0000 7f2d92039040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-19T14:02:21.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:21.492+0000 7f23ae7fc640 1 -- 172.21.15.57:0/2662469376 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mgrmap(e 101) v1 ==== 88957+0+0 (secure 0 0 0) 0x7f23b4087400 con 0x7f23b815b260 2023-10-19T14:02:21.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:21.492+0000 7f23ae7fc640 1 -- 172.21.15.57:0/2662469376 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f239c06afb0 msgr2=0x7f239c06d470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T14:02:21.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:21.492+0000 7f23ae7fc640 1 --2- 172.21.15.57:0/2662469376 >> [v2:172.21.15.179:6832/1426553632,v1:172.21.15.179:6833/1426553632] conn(0x7f239c06afb0 0x7f239c06d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:21.496 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:02:21.493+0000 7fdb7681f040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-19T14:02:21.551 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:02:21.549+0000 7fdb7681f040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-19T14:02:21.592 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:02:21.589+0000 7f2d0ea9b640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:02:21.592 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:02:21.589+0000 7f2d0ea9b640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:02:21.592 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:02:21.589+0000 7f2d0ea9b640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:02:21.592 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:02:21.589+0000 7f2d0ea9b640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:02:21.593 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:02:21.589+0000 7f2d0ea9b640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:02:21.593 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:02:21.589+0000 7f2d11ae1640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:02:21.593 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:02:21.589+0000 7f2d11ae1640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:02:21.593 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:02:21.589+0000 7f2d11ae1640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:02:21.593 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:02:21.589+0000 7f2d11ae1640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:02:21.594 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:02:21.589+0000 7f2d11ae1640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:02:21.662 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:02:21.660+0000 7fdb7681f040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-19T14:02:22.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:22.494+0000 7f23ae7fc640 1 -- 172.21.15.57:0/2662469376 <== mon.0 v2:172.21.15.57:3300/0 8 ==== mgrmap(e 102) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f23b4054300 con 0x7f23b815b260 2023-10-19T14:02:22.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:22.495+0000 7f23ae7fc640 1 --2- 172.21.15.57:0/2662469376 >> [v2:172.21.15.179:6832/2587699336,v1:172.21.15.179:6833/2587699336] conn(0x7f239c035850 0x7f239c0753e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:22.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:22.495+0000 7f23ae7fc640 1 -- 172.21.15.57:0/2662469376 --> [v2:172.21.15.179:6832/2587699336,v1:172.21.15.179:6833/2587699336] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/test_restart", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f23b80675f0 con 0x7f239c035850 2023-10-19T14:02:22.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:22.497+0000 7f23bd933640 1 --2- 172.21.15.57:0/2662469376 >> [v2:172.21.15.179:6832/2587699336,v1:172.21.15.179:6833/2587699336] conn(0x7f239c035850 0x7f239c0753e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:22.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:22.500+0000 7f23bd933640 1 --2- 172.21.15.57:0/2662469376 >> [v2:172.21.15.179:6832/2587699336,v1:172.21.15.179:6833/2587699336] conn(0x7f239c035850 0x7f239c0753e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f23a8036000 tx=0x7f23a8004730 comp rx=0 tx=0).ready entity=mgr.8591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:02:22.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:22.560+0000 7f23ae7fc640 1 -- 172.21.15.57:0/2662469376 <== mgr.8591 v2:172.21.15.179:6832/2587699336 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f23b80675f0 con 0x7f239c035850 2023-10-19T14:02:22.563 INFO:teuthology.orchestra.run.smithi057.stdout:Schedule set for path /snap_test_dir1/test_restart 2023-10-19T14:02:22.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:22.562+0000 7f23bfb97640 1 -- 172.21.15.57:0/2662469376 >> [v2:172.21.15.179:6832/2587699336,v1:172.21.15.179:6833/2587699336] conn(0x7f239c035850 msgr2=0x7f239c0753e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:22.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:22.562+0000 7f23bfb97640 1 --2- 172.21.15.57:0/2662469376 >> [v2:172.21.15.179:6832/2587699336,v1:172.21.15.179:6833/2587699336] conn(0x7f239c035850 0x7f239c0753e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f23a8036000 tx=0x7f23a8004730 comp rx=0 tx=0).stop 2023-10-19T14:02:22.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:22.562+0000 7f23bfb97640 1 -- 172.21.15.57:0/2662469376 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f23b815b260 msgr2=0x7f23b815e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:22.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:22.562+0000 7f23bfb97640 1 --2- 172.21.15.57:0/2662469376 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f23b815b260 0x7f23b815e1b0 secure :-1 s=READY pgs=2078 cs=0 l=1 rev1=1 crypto rx=0x7f23b400c410 tx=0x7f23b400c8d0 comp rx=0 tx=0).stop 2023-10-19T14:02:22.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:22.562+0000 7f23be134640 1 -- 172.21.15.57:0/2662469376 reap_dead start 2023-10-19T14:02:22.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:22.562+0000 7f23bfb97640 1 -- 172.21.15.57:0/2662469376 shutdown_connections 2023-10-19T14:02:22.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:22.562+0000 7f23bfb97640 1 -- 172.21.15.57:0/2662469376 >> 172.21.15.57:0/2662469376 conn(0x7f23b809a030 msgr2=0x7f23b80af200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:02:22.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:22.562+0000 7f23bfb97640 1 -- 172.21.15.57:0/2662469376 shutdown_connections 2023-10-19T14:02:22.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:22.562+0000 7f23bfb97640 1 -- 172.21.15.57:0/2662469376 wait complete. 2023-10-19T14:02:22.576 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/test_restart 2023-10-19T14:02:22.577 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/test_restart 1m 2023-10-19T14:02:22.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:22.689+0000 7fd83a264640 1 -- 172.21.15.57:0/1885025257 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd8340abc10 msgr2=0x7fd83406b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:22.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:22.689+0000 7fd83a264640 1 --2- 172.21.15.57:0/1885025257 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd8340abc10 0x7fd83406b2e0 secure :-1 s=READY pgs=2081 cs=0 l=1 rev1=1 crypto rx=0x7fd82c005fd0 tx=0x7fd82c02c170 comp rx=0 tx=0).stop 2023-10-19T14:02:22.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:22.689+0000 7fd83a264640 1 -- 172.21.15.57:0/1885025257 shutdown_connections 2023-10-19T14:02:22.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:22.689+0000 7fd83a264640 1 --2- 172.21.15.57:0/1885025257 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd8340abc10 0x7fd83406b2e0 unknown :-1 s=CLOSED pgs=2081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:22.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:22.689+0000 7fd83a264640 1 -- 172.21.15.57:0/1885025257 >> 172.21.15.57:0/1885025257 conn(0x7fd834069930 msgr2=0x7fd834069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:02:22.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:22.689+0000 7fd83a264640 1 -- 172.21.15.57:0/1885025257 shutdown_connections 2023-10-19T14:02:22.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:22.689+0000 7fd83a264640 1 -- 172.21.15.57:0/1885025257 wait complete. 2023-10-19T14:02:22.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:22.690+0000 7fd83a264640 1 Processor -- start 2023-10-19T14:02:22.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:22.690+0000 7fd83a264640 1 -- start start 2023-10-19T14:02:22.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:22.691+0000 7fd83a264640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd83406af00 0x7fd8341ef1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:22.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:22.691+0000 7fd83a264640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd8340abc10 0x7fd8340aa890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:22.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:22.691+0000 7fd83a264640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd8340c9dc0 0x7fd83413d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:22.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:22.691+0000 7fd83a264640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd83413ab70 con 0x7fd83406af00 2023-10-19T14:02:22.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:22.691+0000 7fd83a264640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8340ab310 con 0x7fd8340abc10 2023-10-19T14:02:22.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:22.691+0000 7fd83a264640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd834150460 con 0x7fd8340c9dc0 2023-10-19T14:02:22.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:22.691+0000 7fd833fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd83406af00 0x7fd8341ef1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:22.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:22.691+0000 7fd833fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd83406af00 0x7fd8341ef1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50142/0 (socket says 172.21.15.57:50142) 2023-10-19T14:02:22.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:22.691+0000 7fd833fff640 1 -- 172.21.15.57:0/177781163 learned_addr learned my addr 172.21.15.57:0/177781163 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:02:22.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:22.691+0000 7fd8337fe640 1 --2- 172.21.15.57:0/177781163 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd8340abc10 0x7fd8340aa890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:22.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:22.691+0000 7fd838801640 1 --2- 172.21.15.57:0/177781163 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd8340c9dc0 0x7fd83413d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:22.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:22.691+0000 7fd833fff640 1 -- 172.21.15.57:0/177781163 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd8340c9dc0 msgr2=0x7fd83413d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:22.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:22.691+0000 7fd833fff640 1 --2- 172.21.15.57:0/177781163 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd8340c9dc0 0x7fd83413d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:22.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:22.691+0000 7fd833fff640 1 -- 172.21.15.57:0/177781163 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd8340abc10 msgr2=0x7fd8340aa890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:22.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:22.691+0000 7fd833fff640 1 --2- 172.21.15.57:0/177781163 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd8340abc10 0x7fd8340aa890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:22.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:22.691+0000 7fd833fff640 1 -- 172.21.15.57:0/177781163 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd82c009840 con 0x7fd83406af00 2023-10-19T14:02:22.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:22.691+0000 7fd833fff640 1 --2- 172.21.15.57:0/177781163 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd83406af00 0x7fd8341ef1a0 secure :-1 s=READY pgs=2082 cs=0 l=1 rev1=1 crypto rx=0x7fd82400c500 tx=0x7fd82400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:02:22.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:22.692+0000 7fd830ff9640 1 -- 172.21.15.57:0/177781163 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8240052d0 con 0x7fd83406af00 2023-10-19T14:02:22.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:22.692+0000 7fd830ff9640 1 -- 172.21.15.57:0/177781163 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fd824005450 con 0x7fd83406af00 2023-10-19T14:02:22.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:22.692+0000 7fd83a264640 1 -- 172.21.15.57:0/177781163 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd834150700 con 0x7fd83406af00 2023-10-19T14:02:22.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:22.692+0000 7fd830ff9640 1 -- 172.21.15.57:0/177781163 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd824012890 con 0x7fd83406af00 2023-10-19T14:02:22.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:22.692+0000 7fd83a264640 1 -- 172.21.15.57:0/177781163 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd834150cc0 con 0x7fd83406af00 2023-10-19T14:02:22.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:22.693+0000 7fd83a264640 1 -- 172.21.15.57:0/177781163 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8340690e0 con 0x7fd83406af00 2023-10-19T14:02:22.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:22.693+0000 7fd830ff9640 1 -- 172.21.15.57:0/177781163 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 102) v1 ==== 89084+0+0 (secure 0 0 0) 0x7fd8240035f0 con 0x7fd83406af00 2023-10-19T14:02:22.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:22.693+0000 7fd830ff9640 1 --2- 172.21.15.57:0/177781163 >> [v2:172.21.15.179:6832/2587699336,v1:172.21.15.179:6833/2587699336] conn(0x7fd82006ace0 0x7fd82006d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:02:22.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:22.694+0000 7fd830ff9640 1 -- 172.21.15.57:0/177781163 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 6401+0+0 (secure 0 0 0) 0x7fd8240891f0 con 0x7fd83406af00 2023-10-19T14:02:22.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:22.694+0000 7fd8337fe640 1 --2- 172.21.15.57:0/177781163 >> [v2:172.21.15.179:6832/2587699336,v1:172.21.15.179:6833/2587699336] conn(0x7fd82006ace0 0x7fd82006d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:02:22.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:22.694+0000 7fd8337fe640 1 --2- 172.21.15.57:0/177781163 >> [v2:172.21.15.179:6832/2587699336,v1:172.21.15.179:6833/2587699336] conn(0x7fd82006ace0 0x7fd82006d1a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd834159100 tx=0x7fd828007d20 comp rx=0 tx=0).ready entity=mgr.8591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:02:22.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:22.698+0000 7fd830ff9640 1 -- 172.21.15.57:0/177781163 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd824054350 con 0x7fd83406af00 2023-10-19T14:02:22.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:22.897+0000 7fd83a264640 1 -- 172.21.15.57:0/177781163 --> [v2:172.21.15.179:6832/2587699336,v1:172.21.15.179:6833/2587699336] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/test_restart", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fd834159a00 con 0x7fd82006ace0 2023-10-19T14:02:22.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:22.939+0000 7fd830ff9640 1 -- 172.21.15.57:0/177781163 <== mgr.8591 v2:172.21.15.179:6832/2587699336 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+57 (secure 0 0 0) 0x7fd834159a00 con 0x7fd82006ace0 2023-10-19T14:02:22.942 INFO:teuthology.orchestra.run.smithi057.stdout:Schedule activated for path /snap_test_dir1/test_restart 2023-10-19T14:02:22.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:22.941+0000 7fd83a264640 1 -- 172.21.15.57:0/177781163 >> [v2:172.21.15.179:6832/2587699336,v1:172.21.15.179:6833/2587699336] conn(0x7fd82006ace0 msgr2=0x7fd82006d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:22.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:22.941+0000 7fd83a264640 1 --2- 172.21.15.57:0/177781163 >> [v2:172.21.15.179:6832/2587699336,v1:172.21.15.179:6833/2587699336] conn(0x7fd82006ace0 0x7fd82006d1a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd834159100 tx=0x7fd828007d20 comp rx=0 tx=0).stop 2023-10-19T14:02:22.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:22.941+0000 7fd83a264640 1 -- 172.21.15.57:0/177781163 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd83406af00 msgr2=0x7fd8341ef1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:02:22.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:22.941+0000 7fd83a264640 1 --2- 172.21.15.57:0/177781163 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd83406af00 0x7fd8341ef1a0 secure :-1 s=READY pgs=2082 cs=0 l=1 rev1=1 crypto rx=0x7fd82400c500 tx=0x7fd82400c9c0 comp rx=0 tx=0).stop 2023-10-19T14:02:22.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:22.941+0000 7fd83a264640 1 -- 172.21.15.57:0/177781163 shutdown_connections 2023-10-19T14:02:22.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:22.941+0000 7fd83a264640 1 --2- 172.21.15.57:0/177781163 >> [v2:172.21.15.179:6832/2587699336,v1:172.21.15.179:6833/2587699336] conn(0x7fd82006ace0 0x7fd82006d1a0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:22.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:22.941+0000 7fd83a264640 1 --2- 172.21.15.57:0/177781163 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd8340c9dc0 0x7fd83413d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:22.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:22.941+0000 7fd83a264640 1 --2- 172.21.15.57:0/177781163 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd8340abc10 0x7fd8340aa890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:22.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:22.941+0000 7fd83a264640 1 --2- 172.21.15.57:0/177781163 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd83406af00 0x7fd8341ef1a0 unknown :-1 s=CLOSED pgs=2082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:02:22.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:22.941+0000 7fd83a264640 1 -- 172.21.15.57:0/177781163 >> 172.21.15.57:0/177781163 conn(0x7fd834069930 msgr2=0x7fd8340c9680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:02:22.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:22.941+0000 7fd83a264640 1 -- 172.21.15.57:0/177781163 shutdown_connections 2023-10-19T14:02:22.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:02:22.941+0000 7fd83a264640 1 -- 172.21.15.57:0/177781163 wait complete. 2023-10-19T14:02:22.954 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/test_restart 2023-10-19T14:02:32.229 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:02:32.233 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:03:00.952 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:03:00.951+0000 7f2d00a7f640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:03:00.953 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:03:00.951+0000 7f2d00a7f640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:03:00.953 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:03:00.951+0000 7f2d00a7f640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:03:00.953 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:03:00.951+0000 7f2d00a7f640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:03:00.954 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:03:00.951+0000 7f2d00a7f640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:03:02.306 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:03:02.309 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:03:32.371 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:03:32.375 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:04:02.515 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:04:02.517 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:04:32.821 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:04:32.825 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:05:02.953 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:05:02.956 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:05:33.083 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:05:33.087 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:06:03.214 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:06:03.218 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:06:33.280 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:06:33.284 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:07:03.409 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:07:03.413 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:07:33.484 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:07:33.487 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:08:03.556 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:08:03.559 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:08:33.676 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:08:33.679 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:09:03.759 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:09:03.762 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:09:33.880 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:09:33.884 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:10:03.962 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:10:03.967 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:10:34.109 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:10:34.113 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:11:04.174 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:11:04.177 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:11:34.313 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:11:34.316 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:12:04.494 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:12:04.497 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:12:34.592 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:12:34.596 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:13:04.706 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:13:04.709 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:13:15.956 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:13:15.956 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/test_restart/.snap) 2023-10-19T14:13:15.982 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_03_00_UTC 2023-10-19T14:13:15.982 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_04_00_UTC 2023-10-19T14:13:15.983 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_05_00_UTC 2023-10-19T14:13:15.983 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_06_00_UTC 2023-10-19T14:13:15.983 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_07_00_UTC 2023-10-19T14:13:15.983 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_08_00_UTC 2023-10-19T14:13:15.983 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_09_00_UTC 2023-10-19T14:13:15.984 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_10_00_UTC 2023-10-19T14:13:15.984 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_11_00_UTC 2023-10-19T14:13:15.984 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_12_00_UTC 2023-10-19T14:13:15.984 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_13_00_UTC 2023-10-19T14:13:15.985 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-19-14_03_00_UTC', 'scheduled-2023-10-19-14_04_00_UTC', 'scheduled-2023-10-19-14_05_00_UTC', 'scheduled-2023-10-19-14_06_00_UTC', 'scheduled-2023-10-19-14_07_00_UTC', 'scheduled-2023-10-19-14_08_00_UTC', 'scheduled-2023-10-19-14_09_00_UTC', 'scheduled-2023-10-19-14_10_00_UTC', 'scheduled-2023-10-19-14_11_00_UTC', 'scheduled-2023-10-19-14_12_00_UTC', 'scheduled-2023-10-19-14_13_00_UTC'] 2023-10-19T14:13:15.986 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/test_restart 2023-10-19T14:13:16.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.144+0000 7f1cdb0e5640 1 -- 172.21.15.57:0/196409107 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1cd413e520 msgr2=0x7f1cd406bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:13:16.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.144+0000 7f1cdb0e5640 1 --2- 172.21.15.57:0/196409107 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1cd413e520 0x7f1cd406bb60 secure :-1 s=READY pgs=2084 cs=0 l=1 rev1=1 crypto rx=0x7f1cc8009830 tx=0x7f1cc802bcd0 comp rx=0 tx=0).stop 2023-10-19T14:13:16.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.144+0000 7f1cdb0e5640 1 -- 172.21.15.57:0/196409107 shutdown_connections 2023-10-19T14:13:16.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.144+0000 7f1cdb0e5640 1 --2- 172.21.15.57:0/196409107 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1cd413e520 0x7f1cd406bb60 unknown :-1 s=CLOSED pgs=2084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:13:16.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.144+0000 7f1cdb0e5640 1 -- 172.21.15.57:0/196409107 >> 172.21.15.57:0/196409107 conn(0x7f1cd4069930 msgr2=0x7f1cd4069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:13:16.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.144+0000 7f1cdb0e5640 1 -- 172.21.15.57:0/196409107 shutdown_connections 2023-10-19T14:13:16.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.144+0000 7f1cdb0e5640 1 -- 172.21.15.57:0/196409107 wait complete. 2023-10-19T14:13:16.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.145+0000 7f1cdb0e5640 1 Processor -- start 2023-10-19T14:13:16.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.145+0000 7f1cdb0e5640 1 -- start start 2023-10-19T14:13:16.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.146+0000 7f1cdb0e5640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1cd4070af0 0x7f1cd41f5a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:13:16.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.146+0000 7f1cdb0e5640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1cd413e520 0x7f1cd413c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:13:16.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.146+0000 7f1cdb0e5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1cd4141690 0x7f1cd413adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:13:16.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.146+0000 7f1cdb0e5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cd40aa960 con 0x7f1cd4141690 2023-10-19T14:13:16.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.146+0000 7f1cdb0e5640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cd40aac40 con 0x7f1cd4070af0 2023-10-19T14:13:16.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.146+0000 7f1cdb0e5640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cd40aaf20 con 0x7f1cd413e520 2023-10-19T14:13:16.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.146+0000 7f1cd9682640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1cd4141690 0x7f1cd413adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:13:16.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.146+0000 7f1cd9682640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1cd4141690 0x7f1cd413adf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59714/0 (socket says 172.21.15.57:59714) 2023-10-19T14:13:16.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.146+0000 7f1cd9682640 1 -- 172.21.15.57:0/140795712 learned_addr learned my addr 172.21.15.57:0/140795712 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:13:16.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.146+0000 7f1cd8e81640 1 --2- 172.21.15.57:0/140795712 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1cd4070af0 0x7f1cd41f5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:13:16.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.146+0000 7f1cd3fff640 1 --2- 172.21.15.57:0/140795712 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1cd413e520 0x7f1cd413c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:13:16.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.146+0000 7f1cd9682640 1 -- 172.21.15.57:0/140795712 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1cd413e520 msgr2=0x7f1cd413c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:13:16.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.146+0000 7f1cd9682640 1 --2- 172.21.15.57:0/140795712 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1cd413e520 0x7f1cd413c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:13:16.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.146+0000 7f1cd9682640 1 -- 172.21.15.57:0/140795712 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1cd4070af0 msgr2=0x7f1cd41f5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:13:16.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.146+0000 7f1cd9682640 1 --2- 172.21.15.57:0/140795712 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1cd4070af0 0x7f1cd41f5a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:13:16.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.146+0000 7f1cd9682640 1 -- 172.21.15.57:0/140795712 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cc8009490 con 0x7f1cd4141690 2023-10-19T14:13:16.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.146+0000 7f1cd8e81640 1 --2- 172.21.15.57:0/140795712 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1cd4070af0 0x7f1cd41f5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:13:16.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.146+0000 7f1cd9682640 1 --2- 172.21.15.57:0/140795712 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1cd4141690 0x7f1cd413adf0 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 crypto rx=0x7f1ccc00c780 tx=0x7f1ccc00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:13:16.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.147+0000 7f1cd17fa640 1 -- 172.21.15.57:0/140795712 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ccc004b60 con 0x7f1cd4141690 2023-10-19T14:13:16.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.147+0000 7f1cd17fa640 1 -- 172.21.15.57:0/140795712 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f1ccc00ce90 con 0x7f1cd4141690 2023-10-19T14:13:16.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.147+0000 7f1cdb0e5640 1 -- 172.21.15.57:0/140795712 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cd40ab200 con 0x7f1cd4141690 2023-10-19T14:13:16.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.147+0000 7f1cd17fa640 1 -- 172.21.15.57:0/140795712 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ccc00aab0 con 0x7f1cd4141690 2023-10-19T14:13:16.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.147+0000 7f1cdb0e5640 1 -- 172.21.15.57:0/140795712 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cd413c820 con 0x7f1cd4141690 2023-10-19T14:13:16.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.148+0000 7f1cdb0e5640 1 -- 172.21.15.57:0/140795712 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cd4067930 con 0x7f1cd4141690 2023-10-19T14:13:16.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.148+0000 7f1cd17fa640 1 -- 172.21.15.57:0/140795712 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 106) v1 ==== 89360+0+0 (secure 0 0 0) 0x7f1ccc00ac10 con 0x7f1cd4141690 2023-10-19T14:13:16.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.149+0000 7f1cd17fa640 1 --2- 172.21.15.57:0/140795712 >> [v2:172.21.15.179:6832/2587699336,v1:172.21.15.179:6833/2587699336] conn(0x7f1cb806afb0 0x7f1cb806d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:13:16.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.149+0000 7f1cd17fa640 1 -- 172.21.15.57:0/140795712 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f1ccc08a230 con 0x7f1cd4141690 2023-10-19T14:13:16.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.149+0000 7f1cd8e81640 1 --2- 172.21.15.57:0/140795712 >> [v2:172.21.15.179:6832/2587699336,v1:172.21.15.179:6833/2587699336] conn(0x7f1cb806afb0 0x7f1cb806d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:13:16.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.149+0000 7f1cd8e81640 1 --2- 172.21.15.57:0/140795712 >> [v2:172.21.15.179:6832/2587699336,v1:172.21.15.179:6833/2587699336] conn(0x7f1cb806afb0 0x7f1cb806d470 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f1cc8036000 tx=0x7f1cc8004730 comp rx=0 tx=0).ready entity=mgr.8591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:13:16.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.153+0000 7f1cd17fa640 1 -- 172.21.15.57:0/140795712 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f1ccc054e70 con 0x7f1cd4141690 2023-10-19T14:13:16.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.355+0000 7f1cdb0e5640 1 -- 172.21.15.57:0/140795712 --> [v2:172.21.15.179:6832/2587699336,v1:172.21.15.179:6833/2587699336] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/test_restart", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1cd40690e0 con 0x7f1cb806afb0 2023-10-19T14:13:16.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.363+0000 7f1cd17fa640 1 -- 172.21.15.57:0/140795712 <== mgr.8591 v2:172.21.15.179:6832/2587699336 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f1cd40690e0 con 0x7f1cb806afb0 2023-10-19T14:13:16.366 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:13:16.367 INFO:teuthology.orchestra.run.smithi057.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/test_restart", "rel_path": "/snap_test_dir1/test_restart", "schedule": "1m", "retention": {}, "start": "2023-10-19T00:00:00", "created": "2023-10-19T14:02:22", "first": "2023-10-19T14:03:00", "last": "2023-10-19T14:13:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": true}] 2023-10-19T14:13:16.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.366+0000 7f1cdb0e5640 1 -- 172.21.15.57:0/140795712 >> [v2:172.21.15.179:6832/2587699336,v1:172.21.15.179:6833/2587699336] conn(0x7f1cb806afb0 msgr2=0x7f1cb806d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:13:16.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.366+0000 7f1cdb0e5640 1 --2- 172.21.15.57:0/140795712 >> [v2:172.21.15.179:6832/2587699336,v1:172.21.15.179:6833/2587699336] conn(0x7f1cb806afb0 0x7f1cb806d470 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f1cc8036000 tx=0x7f1cc8004730 comp rx=0 tx=0).stop 2023-10-19T14:13:16.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.366+0000 7f1cdb0e5640 1 -- 172.21.15.57:0/140795712 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1cd4141690 msgr2=0x7f1cd413adf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:13:16.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.366+0000 7f1cdb0e5640 1 --2- 172.21.15.57:0/140795712 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1cd4141690 0x7f1cd413adf0 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 crypto rx=0x7f1ccc00c780 tx=0x7f1ccc00cc40 comp rx=0 tx=0).stop 2023-10-19T14:13:16.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.366+0000 7f1cdb0e5640 1 -- 172.21.15.57:0/140795712 shutdown_connections 2023-10-19T14:13:16.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.366+0000 7f1cdb0e5640 1 --2- 172.21.15.57:0/140795712 >> [v2:172.21.15.179:6832/2587699336,v1:172.21.15.179:6833/2587699336] conn(0x7f1cb806afb0 0x7f1cb806d470 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:13:16.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.366+0000 7f1cdb0e5640 1 --2- 172.21.15.57:0/140795712 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1cd4141690 0x7f1cd413adf0 unknown :-1 s=CLOSED pgs=2085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:13:16.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.366+0000 7f1cdb0e5640 1 --2- 172.21.15.57:0/140795712 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1cd413e520 0x7f1cd413c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:13:16.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.366+0000 7f1cdb0e5640 1 --2- 172.21.15.57:0/140795712 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1cd4070af0 0x7f1cd41f5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:13:16.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.366+0000 7f1cdb0e5640 1 -- 172.21.15.57:0/140795712 >> 172.21.15.57:0/140795712 conn(0x7f1cd4069930 msgr2=0x7f1cd4069d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:13:16.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.366+0000 7f1cdb0e5640 1 -- 172.21.15.57:0/140795712 shutdown_connections 2023-10-19T14:13:16.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.366+0000 7f1cdb0e5640 1 -- 172.21.15.57:0/140795712 wait complete. 2023-10-19T14:13:16.379 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/test_restart", "rel_path": "/snap_test_dir1/test_restart", "schedule": "1m", "retention": {}, "start": "2023-10-19T00:00:00", "created": "2023-10-19T14:02:22", "first": "2023-10-19T14:03:00", "last": "2023-10-19T14:13:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": true}] 2023-10-19T14:13:16.379 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/test_restart', 'rel_path': '/snap_test_dir1/test_restart', 'schedule': '1m', 'retention': {}, 'start': '2023-10-19T00:00:00', 'created': '2023-10-19T14:02:22', 'first': '2023-10-19T14:03:00', 'last': '2023-10-19T14:13:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': True} 2023-10-19T14:13:16.379 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-10-19T14:13:16.380 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-10-19T14:13:16.380 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-10-19T14:13:16.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.487+0000 7ff36c5b7640 1 -- 172.21.15.57:0/3235800714 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff350003970 msgr2=0x7ff350003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:13:16.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.487+0000 7ff36c5b7640 1 --2- 172.21.15.57:0/3235800714 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff350003970 0x7ff350003d90 secure :-1 s=READY pgs=2087 cs=0 l=1 rev1=1 crypto rx=0x7ff360009b50 tx=0x7ff3500057f0 comp rx=0 tx=0).stop 2023-10-19T14:13:16.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.487+0000 7ff36ab54640 1 -- 172.21.15.57:0/3235800714 reap_dead start 2023-10-19T14:13:16.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.487+0000 7ff36c5b7640 1 -- 172.21.15.57:0/3235800714 shutdown_connections 2023-10-19T14:13:16.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.487+0000 7ff36c5b7640 1 -- 172.21.15.57:0/3235800714 >> 172.21.15.57:0/3235800714 conn(0x7ff364069930 msgr2=0x7ff364069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:13:16.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.487+0000 7ff36c5b7640 1 -- 172.21.15.57:0/3235800714 shutdown_connections 2023-10-19T14:13:16.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.487+0000 7ff36c5b7640 1 -- 172.21.15.57:0/3235800714 wait complete. 2023-10-19T14:13:16.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.488+0000 7ff36c5b7640 1 Processor -- start 2023-10-19T14:13:16.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.488+0000 7ff36c5b7640 1 -- start start 2023-10-19T14:13:16.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.489+0000 7ff36c5b7640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff350003970 0x7ff3640addc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:13:16.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.489+0000 7ff36c5b7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff3641df160 0x7ff36413c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:13:16.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.489+0000 7ff36c5b7640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff3641df5c0 0x7ff3640ac590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:13:16.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.489+0000 7ff36c5b7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff36413de20 con 0x7ff3641df160 2023-10-19T14:13:16.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.489+0000 7ff36c5b7640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff36413db40 con 0x7ff3641df5c0 2023-10-19T14:13:16.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.489+0000 7ff36c5b7640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff36413bb70 con 0x7ff350003970 2023-10-19T14:13:16.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.489+0000 7ff369b52640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff3641df160 0x7ff36413c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:13:16.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.489+0000 7ff369b52640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff3641df160 0x7ff36413c630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59732/0 (socket says 172.21.15.57:59732) 2023-10-19T14:13:16.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.489+0000 7ff369b52640 1 -- 172.21.15.57:0/3756442873 learned_addr learned my addr 172.21.15.57:0/3756442873 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:13:16.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.489+0000 7ff36a353640 1 --2- 172.21.15.57:0/3756442873 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff350003970 0x7ff3640addc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:13:16.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.489+0000 7ff36ab54640 1 --2- 172.21.15.57:0/3756442873 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff3641df5c0 0x7ff3640ac590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:13:16.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.490+0000 7ff369b52640 1 -- 172.21.15.57:0/3756442873 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff350003970 msgr2=0x7ff3640addc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:13:16.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.490+0000 7ff369b52640 1 --2- 172.21.15.57:0/3756442873 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff350003970 0x7ff3640addc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:13:16.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.490+0000 7ff369b52640 1 -- 172.21.15.57:0/3756442873 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff3641df5c0 msgr2=0x7ff3640ac590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:13:16.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.490+0000 7ff369b52640 1 --2- 172.21.15.57:0/3756442873 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff3641df5c0 0x7ff3640ac590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:13:16.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.490+0000 7ff369b52640 1 -- 172.21.15.57:0/3756442873 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff36413be50 con 0x7ff3641df160 2023-10-19T14:13:16.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.490+0000 7ff369b52640 1 --2- 172.21.15.57:0/3756442873 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff3641df160 0x7ff36413c630 secure :-1 s=READY pgs=2088 cs=0 l=1 rev1=1 crypto rx=0x7ff354011a50 tx=0x7ff354011f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:13:16.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.490+0000 7ff35affd640 1 -- 172.21.15.57:0/3756442873 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff35400fd90 con 0x7ff3641df160 2023-10-19T14:13:16.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.490+0000 7ff35affd640 1 -- 172.21.15.57:0/3756442873 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7ff354012450 con 0x7ff3641df160 2023-10-19T14:13:16.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.490+0000 7ff36c5b7640 1 -- 172.21.15.57:0/3756442873 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff36413c130 con 0x7ff3641df160 2023-10-19T14:13:16.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.490+0000 7ff35affd640 1 -- 172.21.15.57:0/3756442873 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff354012800 con 0x7ff3641df160 2023-10-19T14:13:16.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.490+0000 7ff36c5b7640 1 -- 172.21.15.57:0/3756442873 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff36413d710 con 0x7ff3641df160 2023-10-19T14:13:16.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.491+0000 7ff36c5b7640 1 -- 172.21.15.57:0/3756442873 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff36413b9d0 con 0x7ff3641df160 2023-10-19T14:13:16.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.491+0000 7ff35affd640 1 -- 172.21.15.57:0/3756442873 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 106) v1 ==== 89360+0+0 (secure 0 0 0) 0x7ff354012960 con 0x7ff3641df160 2023-10-19T14:13:16.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.492+0000 7ff35affd640 1 --2- 172.21.15.57:0/3756442873 >> [v2:172.21.15.179:6832/2587699336,v1:172.21.15.179:6833/2587699336] conn(0x7ff34806afb0 0x7ff34806d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:13:16.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.492+0000 7ff35affd640 1 -- 172.21.15.57:0/3756442873 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 6401+0+0 (secure 0 0 0) 0x7ff35408a8e0 con 0x7ff3641df160 2023-10-19T14:13:16.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.492+0000 7ff36a353640 1 --2- 172.21.15.57:0/3756442873 >> [v2:172.21.15.179:6832/2587699336,v1:172.21.15.179:6833/2587699336] conn(0x7ff34806afb0 0x7ff34806d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:13:16.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.493+0000 7ff36a353640 1 --2- 172.21.15.57:0/3756442873 >> [v2:172.21.15.179:6832/2587699336,v1:172.21.15.179:6833/2587699336] conn(0x7ff34806afb0 0x7ff34806d470 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff360036000 tx=0x7ff3600046c0 comp rx=0 tx=0).ready entity=mgr.8591 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:13:16.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.496+0000 7ff35affd640 1 -- 172.21.15.57:0/3756442873 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff36413b9d0 con 0x7ff3641df160 2023-10-19T14:13:16.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.719+0000 7ff36c5b7640 1 -- 172.21.15.57:0/3756442873 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7ff36413b9d0 con 0x7ff3641df160 2023-10-19T14:13:16.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.722+0000 7ff35affd640 1 -- 172.21.15.57:0/3756442873 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v106) v1 ==== 74+0+171248 (secure 0 0 0) 0x7ff36413b9d0 con 0x7ff3641df160 2023-10-19T14:13:16.726 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:13:16.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.724+0000 7ff36c5b7640 1 -- 172.21.15.57:0/3756442873 >> [v2:172.21.15.179:6832/2587699336,v1:172.21.15.179:6833/2587699336] conn(0x7ff34806afb0 msgr2=0x7ff34806d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:13:16.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.724+0000 7ff36c5b7640 1 --2- 172.21.15.57:0/3756442873 >> [v2:172.21.15.179:6832/2587699336,v1:172.21.15.179:6833/2587699336] conn(0x7ff34806afb0 0x7ff34806d470 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff360036000 tx=0x7ff3600046c0 comp rx=0 tx=0).stop 2023-10-19T14:13:16.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.724+0000 7ff36c5b7640 1 -- 172.21.15.57:0/3756442873 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff3641df160 msgr2=0x7ff36413c630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:13:16.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.724+0000 7ff36c5b7640 1 --2- 172.21.15.57:0/3756442873 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff3641df160 0x7ff36413c630 secure :-1 s=READY pgs=2088 cs=0 l=1 rev1=1 crypto rx=0x7ff354011a50 tx=0x7ff354011f10 comp rx=0 tx=0).stop 2023-10-19T14:13:16.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.724+0000 7ff36c5b7640 1 -- 172.21.15.57:0/3756442873 shutdown_connections 2023-10-19T14:13:16.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.724+0000 7ff36c5b7640 1 --2- 172.21.15.57:0/3756442873 >> [v2:172.21.15.179:6832/2587699336,v1:172.21.15.179:6833/2587699336] conn(0x7ff34806afb0 0x7ff34806d470 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:13:16.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.724+0000 7ff36c5b7640 1 --2- 172.21.15.57:0/3756442873 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff3641df5c0 0x7ff3640ac590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:13:16.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.724+0000 7ff36c5b7640 1 --2- 172.21.15.57:0/3756442873 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff3641df160 0x7ff36413c630 unknown :-1 s=CLOSED pgs=2088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:13:16.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.724+0000 7ff36c5b7640 1 --2- 172.21.15.57:0/3756442873 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff350003970 0x7ff3640addc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:13:16.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.724+0000 7ff36c5b7640 1 -- 172.21.15.57:0/3756442873 >> 172.21.15.57:0/3756442873 conn(0x7ff364069930 msgr2=0x7ff3641e7110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:13:16.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.724+0000 7ff36c5b7640 1 -- 172.21.15.57:0/3756442873 shutdown_connections 2023-10-19T14:13:16.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:13:16.724+0000 7ff36c5b7640 1 -- 172.21.15.57:0/3756442873 wait complete. 2023-10-19T14:13:16.737 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":106,"active_gid":8591,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":2587699336},{"type":"v1","addr":"172.21.15.179:6833","nonce":2587699336}]},"active_addr":"172.21.15.179:6833/2587699336","active_change":"2023-10-19T14:02:21.481815+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":10812,"name":"y","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful","snap_schedule"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":189,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.179:0","nonce":1009979748}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.179:0","nonce":1362323478}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.179:0","nonce":2295535972}]},{"name":"snap_schedule","addrvec":[{"type":"v2","addr":"172.21.15.179:0","nonce":1094895803}]},{"name":"snap_schedule","addrvec":[{"type":"v2","addr":"172.21.15.179:0","nonce":2961439032}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.179:0","nonce":2642945549}]}]} 2023-10-19T14:13:16.771 DEBUG:tasks.cephfs.test_snap_schedules:restarting active mgr: x 2023-10-19T14:13:16.772 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-10-19T14:13:16.772 INFO:tasks.ceph.mgr.x:Stopping old one... 2023-10-19T14:13:16.773 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-10-19T14:13:16.773 INFO:teuthology.orchestra.run:waiting for 300 2023-10-19T14:13:16.807 INFO:tasks.ceph.mgr.x:Stopped 2023-10-19T14:13:16.808 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-10-19T14:13:16.850 INFO:tasks.ceph.mgr.x:Started 2023-10-19T14:13:17.263 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:13:17.261+0000 7f4136a2f040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-19T14:13:17.308 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:13:17.306+0000 7f4136a2f040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-19T14:13:17.697 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:13:17.694+0000 7f4136a2f040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-19T14:13:17.740 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:13:17.737+0000 7f4136a2f040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-19T14:13:17.781 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:13:17.779+0000 7f4136a2f040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-19T14:13:17.831 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:13:17.829+0000 7f4136a2f040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-19T14:13:17.871 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:13:17.869+0000 7f4136a2f040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-19T14:13:17.967 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:13:17.965+0000 7f4136a2f040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-19T14:13:18.019 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:13:18.017+0000 7f4136a2f040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-19T14:13:18.060 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:13:18.058+0000 7f4136a2f040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-19T14:13:18.108 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:13:18.105+0000 7f4136a2f040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-19T14:13:18.150 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:13:18.148+0000 7f4136a2f040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-19T14:13:18.463 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:13:18.461+0000 7f4136a2f040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-19T14:13:18.592 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:13:18.589+0000 7f4136a2f040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-19T14:13:18.682 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:13:18.679+0000 7f4136a2f040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-19T14:13:18.723 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:13:18.721+0000 7f4136a2f040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-19T14:13:18.817 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:13:18.815+0000 7f4136a2f040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-19T14:13:18.906 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:13:18.903+0000 7f4136a2f040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-19T14:13:19.040 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:13:19.037+0000 7f4136a2f040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-19T14:13:19.081 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:13:19.078+0000 7f4136a2f040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-19T14:13:19.182 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:13:19.180+0000 7f4136a2f040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-19T14:13:19.797 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:13:19.795+0000 7f4136a2f040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-19T14:13:20.055 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:13:20.053+0000 7f4136a2f040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-19T14:13:20.102 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:13:20.100+0000 7f4136a2f040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-19T14:13:20.201 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:13:20.199+0000 7f4136a2f040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-19T14:13:20.360 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:13:20.357+0000 7f40b24e8640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:13:20.360 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:13:20.357+0000 7f40b24e8640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:13:20.360 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:13:20.357+0000 7f40b24e8640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:13:20.360 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:13:20.357+0000 7f40b24e8640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:13:20.361 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:13:20.357+0000 7f40b24e8640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:13:20.361 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:13:20.357+0000 7f40b552e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:13:20.361 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:13:20.357+0000 7f40b552e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:13:20.361 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:13:20.357+0000 7f40b552e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:13:20.362 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:13:20.357+0000 7f40b552e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:13:20.362 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:13:20.357+0000 7f40b552e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:13:34.794 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:13:34.797 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:14:00.359 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:14:00.356+0000 7f40bd53e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:14:00.359 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:14:00.356+0000 7f40bd53e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:14:00.360 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:14:00.356+0000 7f40bd53e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:14:00.360 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:14:00.356+0000 7f40bd53e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:14:00.360 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:14:00.356+0000 7f40bd53e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:14:04.927 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:14:04.930 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:14:35.000 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:14:35.003 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:15:05.139 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:15:05.142 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:15:35.225 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:15:35.228 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:16:05.342 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:16:05.346 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:16:35.412 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:16:35.415 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:17:05.522 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:17:05.525 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:17:35.602 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:17:35.606 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:18:05.734 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:18:05.737 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:18:16.851 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/test_restart 1m 2023-10-19T14:18:16.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:16.974+0000 7f8a788a9640 1 -- 172.21.15.57:0/2157448286 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8a60005180 msgr2=0x7f8a6000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:16.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:16.974+0000 7f8a788a9640 1 --2- 172.21.15.57:0/2157448286 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8a60005180 0x7f8a6000b980 secure :-1 s=READY pgs=2091 cs=0 l=1 rev1=1 crypto rx=0x7f8a6c009b50 tx=0x7f8a600057f0 comp rx=0 tx=0).stop 2023-10-19T14:18:16.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:16.974+0000 7f8a788a9640 1 -- 172.21.15.57:0/2157448286 shutdown_connections 2023-10-19T14:18:16.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:16.974+0000 7f8a788a9640 1 --2- 172.21.15.57:0/2157448286 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8a60005180 0x7f8a6000b980 unknown :-1 s=CLOSED pgs=2091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:16.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:16.974+0000 7f8a788a9640 1 -- 172.21.15.57:0/2157448286 >> 172.21.15.57:0/2157448286 conn(0x7f8a74069930 msgr2=0x7f8a74069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:16.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:16.974+0000 7f8a788a9640 1 -- 172.21.15.57:0/2157448286 shutdown_connections 2023-10-19T14:18:16.979 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:16.974+0000 7f8a788a9640 1 -- 172.21.15.57:0/2157448286 wait complete. 2023-10-19T14:18:16.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:16.975+0000 7f8a788a9640 1 Processor -- start 2023-10-19T14:18:16.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:16.975+0000 7f8a788a9640 1 -- start start 2023-10-19T14:18:16.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:16.976+0000 7f8a788a9640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8a60005180 0x7f8a7414ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:16.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:16.976+0000 7f8a788a9640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8a74151370 0x7f8a741556b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:16.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:16.976+0000 7f8a788a9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8a74155bf0 0x7f8a740ac870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:16.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:16.976+0000 7f8a788a9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a74152440 con 0x7f8a74155bf0 2023-10-19T14:18:16.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:16.976+0000 7f8a788a9640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a74151830 con 0x7f8a60005180 2023-10-19T14:18:16.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:16.976+0000 7f8a788a9640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a74151b10 con 0x7f8a74151370 2023-10-19T14:18:16.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:16.976+0000 7f8a72d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8a74155bf0 0x7f8a740ac870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:16.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:16.976+0000 7f8a72d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8a74155bf0 0x7f8a740ac870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39864/0 (socket says 172.21.15.57:39864) 2023-10-19T14:18:16.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:16.976+0000 7f8a72d9d640 1 -- 172.21.15.57:0/2089318926 learned_addr learned my addr 172.21.15.57:0/2089318926 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:18:16.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:16.976+0000 7f8a7259c640 1 --2- 172.21.15.57:0/2089318926 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8a60005180 0x7f8a7414ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:16.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:16.976+0000 7f8a71d9b640 1 --2- 172.21.15.57:0/2089318926 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8a74151370 0x7f8a741556b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:16.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:16.977+0000 7f8a72d9d640 1 -- 172.21.15.57:0/2089318926 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8a74151370 msgr2=0x7f8a741556b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:16.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:16.977+0000 7f8a72d9d640 1 --2- 172.21.15.57:0/2089318926 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8a74151370 0x7f8a741556b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:16.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:16.977+0000 7f8a72d9d640 1 -- 172.21.15.57:0/2089318926 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8a60005180 msgr2=0x7f8a7414ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:16.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:16.977+0000 7f8a72d9d640 1 --2- 172.21.15.57:0/2089318926 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8a60005180 0x7f8a7414ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:16.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:16.977+0000 7f8a72d9d640 1 -- 172.21.15.57:0/2089318926 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a74151df0 con 0x7f8a74155bf0 2023-10-19T14:18:16.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:16.977+0000 7f8a72d9d640 1 --2- 172.21.15.57:0/2089318926 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8a74155bf0 0x7f8a740ac870 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 crypto rx=0x7f8a6c02cc90 tx=0x7f8a6c003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:16.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:16.977+0000 7f8a6affd640 1 -- 172.21.15.57:0/2089318926 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a6c003b20 con 0x7f8a74155bf0 2023-10-19T14:18:16.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:16.977+0000 7f8a6affd640 1 -- 172.21.15.57:0/2089318926 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f8a6c003d70 con 0x7f8a74155bf0 2023-10-19T14:18:16.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:16.977+0000 7f8a788a9640 1 -- 172.21.15.57:0/2089318926 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a741520d0 con 0x7f8a74155bf0 2023-10-19T14:18:16.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:16.977+0000 7f8a6affd640 1 -- 172.21.15.57:0/2089318926 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a6c03e6d0 con 0x7f8a74155bf0 2023-10-19T14:18:16.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:16.977+0000 7f8a788a9640 1 -- 172.21.15.57:0/2089318926 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a740acdb0 con 0x7f8a74155bf0 2023-10-19T14:18:16.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:16.978+0000 7f8a788a9640 1 -- 172.21.15.57:0/2089318926 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a741525f0 con 0x7f8a74155bf0 2023-10-19T14:18:16.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:16.979+0000 7f8a6affd640 1 -- 172.21.15.57:0/2089318926 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 112) v1 ==== 89360+0+0 (secure 0 0 0) 0x7f8a6c03e830 con 0x7f8a74155bf0 2023-10-19T14:18:16.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:16.979+0000 7f8a6affd640 1 --2- 172.21.15.57:0/2089318926 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7f8a5406afe0 0x7f8a5406d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:16.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:16.979+0000 7f8a6affd640 1 -- 172.21.15.57:0/2089318926 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 6745+0+0 (secure 0 0 0) 0x7f8a6c0ac620 con 0x7f8a74155bf0 2023-10-19T14:18:16.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:16.979+0000 7f8a7259c640 1 --2- 172.21.15.57:0/2089318926 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7f8a5406afe0 0x7f8a5406d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:16.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:16.980+0000 7f8a7259c640 1 --2- 172.21.15.57:0/2089318926 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7f8a5406afe0 0x7f8a5406d4a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8a64000f80 tx=0x7f8a64006eb0 comp rx=0 tx=0).ready entity=mgr.9784 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:16.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:16.984+0000 7f8a6affd640 1 -- 172.21.15.57:0/2089318926 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8a741525f0 con 0x7f8a74155bf0 2023-10-19T14:18:17.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.183+0000 7f8a788a9640 1 -- 172.21.15.57:0/2089318926 --> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/test_restart", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f8a7414f590 con 0x7f8a5406afe0 2023-10-19T14:18:17.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.241+0000 7f8a6affd640 1 -- 172.21.15.57:0/2089318926 <== mgr.9784 v2:172.21.15.179:6832/4086068793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f8a7414f590 con 0x7f8a5406afe0 2023-10-19T14:18:17.244 INFO:teuthology.orchestra.run.smithi057.stdout:Schedule deactivated for path /snap_test_dir1/test_restart 2023-10-19T14:18:17.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.243+0000 7f8a788a9640 1 -- 172.21.15.57:0/2089318926 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7f8a5406afe0 msgr2=0x7f8a5406d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:17.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.243+0000 7f8a788a9640 1 --2- 172.21.15.57:0/2089318926 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7f8a5406afe0 0x7f8a5406d4a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8a64000f80 tx=0x7f8a64006eb0 comp rx=0 tx=0).stop 2023-10-19T14:18:17.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.243+0000 7f8a788a9640 1 -- 172.21.15.57:0/2089318926 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8a74155bf0 msgr2=0x7f8a740ac870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:17.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.243+0000 7f8a788a9640 1 --2- 172.21.15.57:0/2089318926 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8a74155bf0 0x7f8a740ac870 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 crypto rx=0x7f8a6c02cc90 tx=0x7f8a6c003820 comp rx=0 tx=0).stop 2023-10-19T14:18:17.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.243+0000 7f8a788a9640 1 -- 172.21.15.57:0/2089318926 shutdown_connections 2023-10-19T14:18:17.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.243+0000 7f8a788a9640 1 --2- 172.21.15.57:0/2089318926 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7f8a5406afe0 0x7f8a5406d4a0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:17.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.243+0000 7f8a788a9640 1 --2- 172.21.15.57:0/2089318926 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8a74155bf0 0x7f8a740ac870 unknown :-1 s=CLOSED pgs=2092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:17.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.243+0000 7f8a788a9640 1 --2- 172.21.15.57:0/2089318926 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8a74151370 0x7f8a741556b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:17.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.243+0000 7f8a788a9640 1 --2- 172.21.15.57:0/2089318926 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8a60005180 0x7f8a7414ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:17.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.243+0000 7f8a788a9640 1 -- 172.21.15.57:0/2089318926 >> 172.21.15.57:0/2089318926 conn(0x7f8a74069930 msgr2=0x7f8a74069d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:17.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.243+0000 7f8a788a9640 1 -- 172.21.15.57:0/2089318926 shutdown_connections 2023-10-19T14:18:17.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.243+0000 7f8a788a9640 1 -- 172.21.15.57:0/2089318926 wait complete. 2023-10-19T14:18:17.256 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/test_restart 2023-10-19T14:18:17.257 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:18:17.257 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/test_restart/.snap) 2023-10-19T14:18:17.277 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_03_00_UTC 2023-10-19T14:18:17.278 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_04_00_UTC 2023-10-19T14:18:17.278 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_05_00_UTC 2023-10-19T14:18:17.278 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_06_00_UTC 2023-10-19T14:18:17.278 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_07_00_UTC 2023-10-19T14:18:17.279 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_08_00_UTC 2023-10-19T14:18:17.279 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_09_00_UTC 2023-10-19T14:18:17.279 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_10_00_UTC 2023-10-19T14:18:17.279 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_11_00_UTC 2023-10-19T14:18:17.279 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_12_00_UTC 2023-10-19T14:18:17.279 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_13_00_UTC 2023-10-19T14:18:17.280 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_14_00_UTC 2023-10-19T14:18:17.280 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_15_00_UTC 2023-10-19T14:18:17.280 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_16_00_UTC 2023-10-19T14:18:17.280 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_17_00_UTC 2023-10-19T14:18:17.280 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_18_00_UTC 2023-10-19T14:18:17.281 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-19-14_03_00_UTC', 'scheduled-2023-10-19-14_04_00_UTC', 'scheduled-2023-10-19-14_05_00_UTC', 'scheduled-2023-10-19-14_06_00_UTC', 'scheduled-2023-10-19-14_07_00_UTC', 'scheduled-2023-10-19-14_08_00_UTC', 'scheduled-2023-10-19-14_09_00_UTC', 'scheduled-2023-10-19-14_10_00_UTC', 'scheduled-2023-10-19-14_11_00_UTC', 'scheduled-2023-10-19-14_12_00_UTC', 'scheduled-2023-10-19-14_13_00_UTC', 'scheduled-2023-10-19-14_14_00_UTC', 'scheduled-2023-10-19-14_15_00_UTC', 'scheduled-2023-10-19-14_16_00_UTC', 'scheduled-2023-10-19-14_17_00_UTC', 'scheduled-2023-10-19-14_18_00_UTC'] 2023-10-19T14:18:17.281 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/test_restart 2023-10-19T14:18:17.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.427+0000 7f00b4a4b640 1 -- 172.21.15.57:0/1898461567 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f00980039a0 msgr2=0x7f0098003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:17.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.427+0000 7f00b4a4b640 1 --2- 172.21.15.57:0/1898461567 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f00980039a0 0x7f0098003dc0 secure :-1 s=READY pgs=2094 cs=0 l=1 rev1=1 crypto rx=0x7f00a8001680 tx=0x7f00980057f0 comp rx=0 tx=0).stop 2023-10-19T14:18:17.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.427+0000 7f00aed9d640 1 -- 172.21.15.57:0/1898461567 reap_dead start 2023-10-19T14:18:17.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.428+0000 7f00b4a4b640 1 -- 172.21.15.57:0/1898461567 shutdown_connections 2023-10-19T14:18:17.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.428+0000 7f00b4a4b640 1 -- 172.21.15.57:0/1898461567 >> 172.21.15.57:0/1898461567 conn(0x7f00b0069930 msgr2=0x7f00b0069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:17.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.428+0000 7f00b4a4b640 1 -- 172.21.15.57:0/1898461567 shutdown_connections 2023-10-19T14:18:17.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.428+0000 7f00b4a4b640 1 -- 172.21.15.57:0/1898461567 wait complete. 2023-10-19T14:18:17.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.429+0000 7f00b4a4b640 1 Processor -- start 2023-10-19T14:18:17.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.429+0000 7f00b4a4b640 1 -- start start 2023-10-19T14:18:17.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.430+0000 7f00b4a4b640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f00980039a0 0x7f00b00a6a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:17.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.430+0000 7f00b4a4b640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f00b012b960 0x7f00b00ca870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:17.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.430+0000 7f00b4a4b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f00b00cadb0 0x7f00b015dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:17.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.430+0000 7f00b4a4b640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00b00cb230 con 0x7f00b00cadb0 2023-10-19T14:18:17.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.430+0000 7f00b4a4b640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00b0157f90 con 0x7f00980039a0 2023-10-19T14:18:17.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.430+0000 7f00b4a4b640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00b00a6f90 con 0x7f00b012b960 2023-10-19T14:18:17.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.430+0000 7f00aed9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f00b00cadb0 0x7f00b015dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:17.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.430+0000 7f00aed9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f00b00cadb0 0x7f00b015dd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39890/0 (socket says 172.21.15.57:39890) 2023-10-19T14:18:17.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.430+0000 7f00aed9d640 1 -- 172.21.15.57:0/3594550102 learned_addr learned my addr 172.21.15.57:0/3594550102 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:18:17.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.430+0000 7f00ae59c640 1 --2- 172.21.15.57:0/3594550102 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f00980039a0 0x7f00b00a6a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:17.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.430+0000 7f00add9b640 1 --2- 172.21.15.57:0/3594550102 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f00b012b960 0x7f00b00ca870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:17.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.430+0000 7f00aed9d640 1 -- 172.21.15.57:0/3594550102 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f00b012b960 msgr2=0x7f00b00ca870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:17.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.430+0000 7f00aed9d640 1 --2- 172.21.15.57:0/3594550102 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f00b012b960 0x7f00b00ca870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:17.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.430+0000 7f00aed9d640 1 -- 172.21.15.57:0/3594550102 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f00980039a0 msgr2=0x7f00b00a6a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:17.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.430+0000 7f00aed9d640 1 --2- 172.21.15.57:0/3594550102 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f00980039a0 0x7f00b00a6a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:17.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.430+0000 7f00aed9d640 1 -- 172.21.15.57:0/3594550102 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00b015ea20 con 0x7f00b00cadb0 2023-10-19T14:18:17.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.430+0000 7f00aed9d640 1 --2- 172.21.15.57:0/3594550102 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f00b00cadb0 0x7f00b015dd30 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 crypto rx=0x7f00a0011a50 tx=0x7f00a0011f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:17.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.430+0000 7f009effd640 1 -- 172.21.15.57:0/3594550102 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00a000fe90 con 0x7f00b00cadb0 2023-10-19T14:18:17.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.430+0000 7f009effd640 1 -- 172.21.15.57:0/3594550102 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f00a0004b20 con 0x7f00b00cadb0 2023-10-19T14:18:17.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.430+0000 7f00b4a4b640 1 -- 172.21.15.57:0/3594550102 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00b015ed00 con 0x7f00b00cadb0 2023-10-19T14:18:17.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.430+0000 7f009effd640 1 -- 172.21.15.57:0/3594550102 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00a0005020 con 0x7f00b00cadb0 2023-10-19T14:18:17.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.430+0000 7f00b4a4b640 1 -- 172.21.15.57:0/3594550102 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00b015f2c0 con 0x7f00b00cadb0 2023-10-19T14:18:17.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.431+0000 7f00b4a4b640 1 -- 172.21.15.57:0/3594550102 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00b00b1830 con 0x7f00b00cadb0 2023-10-19T14:18:17.439 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.432+0000 7f009effd640 1 -- 172.21.15.57:0/3594550102 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 112) v1 ==== 89360+0+0 (secure 0 0 0) 0x7f00a0018030 con 0x7f00b00cadb0 2023-10-19T14:18:17.439 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.432+0000 7f009effd640 1 --2- 172.21.15.57:0/3594550102 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7f009006afb0 0x7f009006d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:17.439 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.432+0000 7f009effd640 1 -- 172.21.15.57:0/3594550102 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 6745+0+0 (secure 0 0 0) 0x7f00a008a870 con 0x7f00b00cadb0 2023-10-19T14:18:17.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.433+0000 7f00ae59c640 1 --2- 172.21.15.57:0/3594550102 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7f009006afb0 0x7f009006d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:17.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.433+0000 7f00ae59c640 1 --2- 172.21.15.57:0/3594550102 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7f009006afb0 0x7f009006d470 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f00a8034000 tx=0x7f00a8001190 comp rx=0 tx=0).ready entity=mgr.9784 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:17.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.437+0000 7f009effd640 1 -- 172.21.15.57:0/3594550102 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f00a00552e0 con 0x7f00b00cadb0 2023-10-19T14:18:17.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.633+0000 7f00b4a4b640 1 -- 172.21.15.57:0/3594550102 --> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/test_restart", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f00b00b12e0 con 0x7f009006afb0 2023-10-19T14:18:17.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.640+0000 7f009effd640 1 -- 172.21.15.57:0/3594550102 <== mgr.9784 v2:172.21.15.179:6832/4086068793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+361 (secure 0 0 0) 0x7f00a80035f0 con 0x7f009006afb0 2023-10-19T14:18:17.643 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:18:17.643 INFO:teuthology.orchestra.run.smithi057.stdout:[{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/test_restart", "rel_path": "/snap_test_dir1/test_restart", "schedule": "1m", "retention": {}, "start": "2023-10-19T00:00:00", "created": "2023-10-19T14:02:22", "first": "2023-10-19T14:03:00", "last": "2023-10-19T14:18:00", "last_pruned": null, "created_count": 16, "pruned_count": 0, "active": false}] 2023-10-19T14:18:17.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.642+0000 7f00b4a4b640 1 -- 172.21.15.57:0/3594550102 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7f009006afb0 msgr2=0x7f009006d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:17.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.642+0000 7f00b4a4b640 1 --2- 172.21.15.57:0/3594550102 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7f009006afb0 0x7f009006d470 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f00a8034000 tx=0x7f00a8001190 comp rx=0 tx=0).stop 2023-10-19T14:18:17.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.642+0000 7f00b4a4b640 1 -- 172.21.15.57:0/3594550102 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f00b00cadb0 msgr2=0x7f00b015dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:17.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.642+0000 7f00b4a4b640 1 --2- 172.21.15.57:0/3594550102 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f00b00cadb0 0x7f00b015dd30 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 crypto rx=0x7f00a0011a50 tx=0x7f00a0011f10 comp rx=0 tx=0).stop 2023-10-19T14:18:17.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.642+0000 7f00b4a4b640 1 -- 172.21.15.57:0/3594550102 shutdown_connections 2023-10-19T14:18:17.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.642+0000 7f00b4a4b640 1 --2- 172.21.15.57:0/3594550102 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7f009006afb0 0x7f009006d470 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:17.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.642+0000 7f00b4a4b640 1 --2- 172.21.15.57:0/3594550102 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f00b00cadb0 0x7f00b015dd30 unknown :-1 s=CLOSED pgs=2095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:17.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.642+0000 7f00b4a4b640 1 --2- 172.21.15.57:0/3594550102 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f00b012b960 0x7f00b00ca870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:17.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.642+0000 7f00b4a4b640 1 --2- 172.21.15.57:0/3594550102 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f00980039a0 0x7f00b00a6a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:17.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.642+0000 7f00b4a4b640 1 -- 172.21.15.57:0/3594550102 >> 172.21.15.57:0/3594550102 conn(0x7f00b0069930 msgr2=0x7f00b00703e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:17.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.642+0000 7f00b4a4b640 1 -- 172.21.15.57:0/3594550102 shutdown_connections 2023-10-19T14:18:17.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.642+0000 7f00b4a4b640 1 -- 172.21.15.57:0/3594550102 wait complete. 2023-10-19T14:18:17.655 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "path": "/snap_test_dir1/test_restart", "rel_path": "/snap_test_dir1/test_restart", "schedule": "1m", "retention": {}, "start": "2023-10-19T00:00:00", "created": "2023-10-19T14:02:22", "first": "2023-10-19T14:03:00", "last": "2023-10-19T14:18:00", "last_pruned": null, "created_count": 16, "pruned_count": 0, "active": false}] 2023-10-19T14:18:17.656 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'path': '/snap_test_dir1/test_restart', 'rel_path': '/snap_test_dir1/test_restart', 'schedule': '1m', 'retention': {}, 'start': '2023-10-19T00:00:00', 'created': '2023-10-19T14:02:22', 'first': '2023-10-19T14:03:00', 'last': '2023-10-19T14:18:00', 'last_pruned': None, 'created_count': 16, 'pruned_count': 0, 'active': False} 2023-10-19T14:18:17.656 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 16 2023-10-19T14:18:17.656 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 16 2023-10-19T14:18:17.657 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/test_restart 1m 2023-10-19T14:18:17.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.767+0000 7f2b4454c640 1 -- 172.21.15.57:0/3419070795 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2b3c0afa20 msgr2=0x7f2b3c158300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:17.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.767+0000 7f2b4454c640 1 --2- 172.21.15.57:0/3419070795 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2b3c0afa20 0x7f2b3c158300 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f2b30005fd0 tx=0x7f2b3002be10 comp rx=0 tx=0).stop 2023-10-19T14:18:17.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.767+0000 7f2b4454c640 1 -- 172.21.15.57:0/3419070795 shutdown_connections 2023-10-19T14:18:17.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.767+0000 7f2b4454c640 1 --2- 172.21.15.57:0/3419070795 >> v2:172.21.15.179:3300/0 conn(0x7f2b3c0b0530 0x7f2b3c0b09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:17.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.767+0000 7f2b4454c640 1 --2- 172.21.15.57:0/3419070795 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2b3c0afa20 0x7f2b3c158300 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:17.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.767+0000 7f2b4454c640 1 -- 172.21.15.57:0/3419070795 >> 172.21.15.57:0/3419070795 conn(0x7f2b3c069930 msgr2=0x7f2b3c069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:17.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.767+0000 7f2b4454c640 1 -- 172.21.15.57:0/3419070795 shutdown_connections 2023-10-19T14:18:17.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.767+0000 7f2b4454c640 1 -- 172.21.15.57:0/3419070795 wait complete. 2023-10-19T14:18:17.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.768+0000 7f2b4454c640 1 Processor -- start 2023-10-19T14:18:17.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.769+0000 7f2b4454c640 1 -- start start 2023-10-19T14:18:17.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.769+0000 7f2b4454c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2b3c0afa20 0x7f2b3c1311a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:17.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.769+0000 7f2b4454c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2b3c0b0530 0x7f2b3c06c210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:17.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.769+0000 7f2b4454c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2b3c0cbd20 0x7f2b3c06c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:17.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.769+0000 7f2b4454c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b3c13eeb0 con 0x7f2b3c0cbd20 2023-10-19T14:18:17.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.769+0000 7f2b4454c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b3c146b40 con 0x7f2b3c0afa20 2023-10-19T14:18:17.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.769+0000 7f2b4454c640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b3c146e20 con 0x7f2b3c0b0530 2023-10-19T14:18:17.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.769+0000 7f2b42ae9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2b3c0cbd20 0x7f2b3c06c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:17.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.769+0000 7f2b42ae9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2b3c0cbd20 0x7f2b3c06c750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39906/0 (socket says 172.21.15.57:39906) 2023-10-19T14:18:17.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.769+0000 7f2b42ae9640 1 -- 172.21.15.57:0/67904854 learned_addr learned my addr 172.21.15.57:0/67904854 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:18:17.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.769+0000 7f2b422e8640 1 --2- 172.21.15.57:0/67904854 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2b3c0afa20 0x7f2b3c1311a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:17.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.769+0000 7f2b41ae7640 1 --2- 172.21.15.57:0/67904854 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2b3c0b0530 0x7f2b3c06c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:17.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.770+0000 7f2b42ae9640 1 -- 172.21.15.57:0/67904854 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2b3c0b0530 msgr2=0x7f2b3c06c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:17.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.770+0000 7f2b42ae9640 1 --2- 172.21.15.57:0/67904854 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2b3c0b0530 0x7f2b3c06c210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:17.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.770+0000 7f2b42ae9640 1 -- 172.21.15.57:0/67904854 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2b3c0afa20 msgr2=0x7f2b3c1311a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:17.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.770+0000 7f2b42ae9640 1 --2- 172.21.15.57:0/67904854 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2b3c0afa20 0x7f2b3c1311a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:17.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.770+0000 7f2b42ae9640 1 -- 172.21.15.57:0/67904854 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b2c009840 con 0x7f2b3c0cbd20 2023-10-19T14:18:17.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.770+0000 7f2b42ae9640 1 --2- 172.21.15.57:0/67904854 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2b3c0cbd20 0x7f2b3c06c750 secure :-1 s=READY pgs=2096 cs=0 l=1 rev1=1 crypto rx=0x7f2b2c00efd0 tx=0x7f2b2c00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:17.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.770+0000 7f2b3affd640 1 -- 172.21.15.57:0/67904854 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b2c009b40 con 0x7f2b3c0cbd20 2023-10-19T14:18:17.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.770+0000 7f2b3affd640 1 -- 172.21.15.57:0/67904854 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f2b2c003c60 con 0x7f2b3c0cbd20 2023-10-19T14:18:17.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.770+0000 7f2b4454c640 1 -- 172.21.15.57:0/67904854 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b30009560 con 0x7f2b3c0cbd20 2023-10-19T14:18:17.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.770+0000 7f2b3affd640 1 -- 172.21.15.57:0/67904854 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b2c004020 con 0x7f2b3c0cbd20 2023-10-19T14:18:17.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.770+0000 7f2b4454c640 1 -- 172.21.15.57:0/67904854 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b3c1473e0 con 0x7f2b3c0cbd20 2023-10-19T14:18:17.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.771+0000 7f2b4454c640 1 -- 172.21.15.57:0/67904854 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b3c147900 con 0x7f2b3c0cbd20 2023-10-19T14:18:17.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.771+0000 7f2b3affd640 1 -- 172.21.15.57:0/67904854 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 112) v1 ==== 89360+0+0 (secure 0 0 0) 0x7f2b2c0041a0 con 0x7f2b3c0cbd20 2023-10-19T14:18:17.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.772+0000 7f2b3affd640 1 --2- 172.21.15.57:0/67904854 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7f2b2806afb0 0x7f2b2806d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:17.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.772+0000 7f2b3affd640 1 -- 172.21.15.57:0/67904854 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 6745+0+0 (secure 0 0 0) 0x7f2b2c014070 con 0x7f2b3c0cbd20 2023-10-19T14:18:17.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.772+0000 7f2b422e8640 1 --2- 172.21.15.57:0/67904854 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7f2b2806afb0 0x7f2b2806d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:17.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.773+0000 7f2b422e8640 1 --2- 172.21.15.57:0/67904854 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7f2b2806afb0 0x7f2b2806d470 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2b340016e0 tx=0x7f2b34009040 comp rx=0 tx=0).ready entity=mgr.9784 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:17.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.776+0000 7f2b3affd640 1 -- 172.21.15.57:0/67904854 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2b3c147900 con 0x7f2b3c0cbd20 2023-10-19T14:18:17.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:17.972+0000 7f2b4454c640 1 -- 172.21.15.57:0/67904854 --> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/test_restart", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f2b3c146f90 con 0x7f2b2806afb0 2023-10-19T14:18:18.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:18.041+0000 7f2b3affd640 1 -- 172.21.15.57:0/67904854 <== mgr.9784 v2:172.21.15.179:6832/4086068793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+55 (secure 0 0 0) 0x7f2b3c146f90 con 0x7f2b2806afb0 2023-10-19T14:18:18.045 INFO:teuthology.orchestra.run.smithi057.stdout:Schedule removed for path /snap_test_dir1/test_restart 2023-10-19T14:18:18.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:18.043+0000 7f2b4454c640 1 -- 172.21.15.57:0/67904854 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7f2b2806afb0 msgr2=0x7f2b2806d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:18.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:18.043+0000 7f2b4454c640 1 --2- 172.21.15.57:0/67904854 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7f2b2806afb0 0x7f2b2806d470 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2b340016e0 tx=0x7f2b34009040 comp rx=0 tx=0).stop 2023-10-19T14:18:18.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:18.043+0000 7f2b4454c640 1 -- 172.21.15.57:0/67904854 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2b3c0cbd20 msgr2=0x7f2b3c06c750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:18.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:18.043+0000 7f2b4454c640 1 --2- 172.21.15.57:0/67904854 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2b3c0cbd20 0x7f2b3c06c750 secure :-1 s=READY pgs=2096 cs=0 l=1 rev1=1 crypto rx=0x7f2b2c00efd0 tx=0x7f2b2c00c6a0 comp rx=0 tx=0).stop 2023-10-19T14:18:18.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:18.043+0000 7f2b4454c640 1 -- 172.21.15.57:0/67904854 shutdown_connections 2023-10-19T14:18:18.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:18.043+0000 7f2b4454c640 1 --2- 172.21.15.57:0/67904854 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7f2b2806afb0 0x7f2b2806d470 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:18.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:18.043+0000 7f2b4454c640 1 --2- 172.21.15.57:0/67904854 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2b3c0cbd20 0x7f2b3c06c750 unknown :-1 s=CLOSED pgs=2096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:18.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:18.043+0000 7f2b4454c640 1 --2- 172.21.15.57:0/67904854 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2b3c0b0530 0x7f2b3c06c210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:18.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:18.043+0000 7f2b4454c640 1 --2- 172.21.15.57:0/67904854 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2b3c0afa20 0x7f2b3c1311a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:18.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:18.043+0000 7f2b4454c640 1 -- 172.21.15.57:0/67904854 >> 172.21.15.57:0/67904854 conn(0x7f2b3c069930 msgr2=0x7f2b3c0ace90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:18.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:18.043+0000 7f2b4454c640 1 -- 172.21.15.57:0/67904854 shutdown_connections 2023-10-19T14:18:18.049 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:18.044+0000 7f2b4454c640 1 -- 172.21.15.57:0/67904854 wait complete. 2023-10-19T14:18:18.056 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/test_restart 2023-10-19T14:18:18.057 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:18:18.057 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/test_restart/.snap) 2023-10-19T14:18:18.078 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_03_00_UTC 2023-10-19T14:18:18.078 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_04_00_UTC 2023-10-19T14:18:18.078 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_05_00_UTC 2023-10-19T14:18:18.078 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_06_00_UTC 2023-10-19T14:18:18.079 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_07_00_UTC 2023-10-19T14:18:18.079 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_08_00_UTC 2023-10-19T14:18:18.079 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_09_00_UTC 2023-10-19T14:18:18.079 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_10_00_UTC 2023-10-19T14:18:18.080 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_11_00_UTC 2023-10-19T14:18:18.080 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_12_00_UTC 2023-10-19T14:18:18.080 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_13_00_UTC 2023-10-19T14:18:18.080 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_14_00_UTC 2023-10-19T14:18:18.080 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_15_00_UTC 2023-10-19T14:18:18.080 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_16_00_UTC 2023-10-19T14:18:18.081 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_17_00_UTC 2023-10-19T14:18:18.081 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_18_00_UTC 2023-10-19T14:18:18.081 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-19-14_03_00_UTC 2023-10-19T14:18:18.082 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:18:18.082 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-19-14_03_00_UTC) 2023-10-19T14:18:18.141 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-19-14_04_00_UTC 2023-10-19T14:18:18.141 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:18:18.142 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-19-14_04_00_UTC) 2023-10-19T14:18:18.204 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-19-14_05_00_UTC 2023-10-19T14:18:18.204 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:18:18.204 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-19-14_05_00_UTC) 2023-10-19T14:18:18.265 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-19-14_06_00_UTC 2023-10-19T14:18:18.266 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:18:18.266 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-19-14_06_00_UTC) 2023-10-19T14:18:18.327 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-19-14_07_00_UTC 2023-10-19T14:18:18.328 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:18:18.328 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-19-14_07_00_UTC) 2023-10-19T14:18:18.390 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-19-14_08_00_UTC 2023-10-19T14:18:18.390 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:18:18.390 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-19-14_08_00_UTC) 2023-10-19T14:18:18.451 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-19-14_09_00_UTC 2023-10-19T14:18:18.451 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:18:18.452 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-19-14_09_00_UTC) 2023-10-19T14:18:18.513 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-19-14_10_00_UTC 2023-10-19T14:18:18.514 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:18:18.514 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-19-14_10_00_UTC) 2023-10-19T14:18:18.575 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-19-14_11_00_UTC 2023-10-19T14:18:18.576 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:18:18.576 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-19-14_11_00_UTC) 2023-10-19T14:18:18.637 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-19-14_12_00_UTC 2023-10-19T14:18:18.638 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:18:18.638 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-19-14_12_00_UTC) 2023-10-19T14:18:18.699 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-19-14_13_00_UTC 2023-10-19T14:18:18.700 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:18:18.700 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-19-14_13_00_UTC) 2023-10-19T14:18:18.762 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-19-14_14_00_UTC 2023-10-19T14:18:18.762 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:18:18.763 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-19-14_14_00_UTC) 2023-10-19T14:18:18.824 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-19-14_15_00_UTC 2023-10-19T14:18:18.825 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:18:18.825 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-19-14_15_00_UTC) 2023-10-19T14:18:18.887 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-19-14_16_00_UTC 2023-10-19T14:18:18.888 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:18:18.888 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-19-14_16_00_UTC) 2023-10-19T14:18:18.950 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-19-14_17_00_UTC 2023-10-19T14:18:18.951 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:18:18.951 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-19-14_17_00_UTC) 2023-10-19T14:18:19.013 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-19-14_18_00_UTC 2023-10-19T14:18:19.013 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:18:19.014 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-19-14_18_00_UTC) 2023-10-19T14:18:19.075 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:18:19.075 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart) 2023-10-19T14:18:19.133 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-19T14:18:19.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:19.281+0000 7fe07ecd7640 1 -- 172.21.15.57:0/1850797956 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe07806b030 msgr2=0x7fe07806b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:19.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:19.281+0000 7fe07ecd7640 1 --2- 172.21.15.57:0/1850797956 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe07806b030 0x7fe07806b410 secure :-1 s=READY pgs=2098 cs=0 l=1 rev1=1 crypto rx=0x7fe0780cb9b0 tx=0x7fe06c02bdb0 comp rx=0 tx=0).stop 2023-10-19T14:18:19.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:19.281+0000 7fe07ecd7640 1 -- 172.21.15.57:0/1850797956 shutdown_connections 2023-10-19T14:18:19.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:19.281+0000 7fe07ecd7640 1 --2- 172.21.15.57:0/1850797956 >> v2:172.21.15.179:3300/0 conn(0x7fe078159bd0 0x7fe07815c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:19.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:19.281+0000 7fe07ecd7640 1 --2- 172.21.15.57:0/1850797956 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe07806b030 0x7fe07806b410 unknown :-1 s=CLOSED pgs=2098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:19.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:19.281+0000 7fe07ecd7640 1 -- 172.21.15.57:0/1850797956 >> 172.21.15.57:0/1850797956 conn(0x7fe078069930 msgr2=0x7fe078069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:19.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:19.281+0000 7fe07ecd7640 1 -- 172.21.15.57:0/1850797956 shutdown_connections 2023-10-19T14:18:19.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:19.281+0000 7fe07ecd7640 1 -- 172.21.15.57:0/1850797956 wait complete. 2023-10-19T14:18:19.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:19.282+0000 7fe07ecd7640 1 Processor -- start 2023-10-19T14:18:19.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:19.283+0000 7fe07ecd7640 1 -- start start 2023-10-19T14:18:19.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:19.283+0000 7fe07ecd7640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe07806b030 0x7fe0780aab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:19.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:19.283+0000 7fe07ecd7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe078159bd0 0x7fe07813cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:19.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:19.283+0000 7fe07ecd7640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe07815ccb0 0x7fe07813cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:19.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:19.283+0000 7fe07ecd7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe07813e6d0 con 0x7fe078159bd0 2023-10-19T14:18:19.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:19.283+0000 7fe07ecd7640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe07813e240 con 0x7fe07806b030 2023-10-19T14:18:19.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:19.283+0000 7fe07ecd7640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe07813eb70 con 0x7fe07815ccb0 2023-10-19T14:18:19.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:19.284+0000 7fe077fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe078159bd0 0x7fe07813cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:19.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:19.284+0000 7fe077fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe078159bd0 0x7fe07813cab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39926/0 (socket says 172.21.15.57:39926) 2023-10-19T14:18:19.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:19.284+0000 7fe077fff640 1 -- 172.21.15.57:0/1132172930 learned_addr learned my addr 172.21.15.57:0/1132172930 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:18:19.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:19.284+0000 7fe07ca73640 1 --2- 172.21.15.57:0/1132172930 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe07806b030 0x7fe0780aab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:19.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:19.284+0000 7fe07d274640 1 --2- 172.21.15.57:0/1132172930 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe07815ccb0 0x7fe07813cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:19.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:19.284+0000 7fe077fff640 1 -- 172.21.15.57:0/1132172930 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe07815ccb0 msgr2=0x7fe07813cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:19.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:19.284+0000 7fe077fff640 1 --2- 172.21.15.57:0/1132172930 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe07815ccb0 0x7fe07813cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:19.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:19.284+0000 7fe077fff640 1 -- 172.21.15.57:0/1132172930 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe07806b030 msgr2=0x7fe0780aab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:19.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:19.284+0000 7fe077fff640 1 --2- 172.21.15.57:0/1132172930 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe07806b030 0x7fe0780aab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:19.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:19.284+0000 7fe077fff640 1 -- 172.21.15.57:0/1132172930 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe06c009560 con 0x7fe078159bd0 2023-10-19T14:18:19.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:19.284+0000 7fe07d274640 1 --2- 172.21.15.57:0/1132172930 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe07815ccb0 0x7fe07813cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:18:19.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:19.284+0000 7fe077fff640 1 --2- 172.21.15.57:0/1132172930 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe078159bd0 0x7fe07813cab0 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 crypto rx=0x7fe06400c920 tx=0x7fe06400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:19.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:19.284+0000 7fe0757fa640 1 -- 172.21.15.57:0/1132172930 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe064007c70 con 0x7fe078159bd0 2023-10-19T14:18:19.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:19.284+0000 7fe0757fa640 1 -- 172.21.15.57:0/1132172930 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fe06400f450 con 0x7fe078159bd0 2023-10-19T14:18:19.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:19.284+0000 7fe07ecd7640 1 -- 172.21.15.57:0/1132172930 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe07813d630 con 0x7fe078159bd0 2023-10-19T14:18:19.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:19.284+0000 7fe0757fa640 1 -- 172.21.15.57:0/1132172930 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe06400f860 con 0x7fe078159bd0 2023-10-19T14:18:19.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:19.284+0000 7fe07ecd7640 1 -- 172.21.15.57:0/1132172930 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe07813dbf0 con 0x7fe078159bd0 2023-10-19T14:18:19.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:19.285+0000 7fe07ecd7640 1 -- 172.21.15.57:0/1132172930 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe07806c290 con 0x7fe078159bd0 2023-10-19T14:18:19.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:19.286+0000 7fe0757fa640 1 -- 172.21.15.57:0/1132172930 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 112) v1 ==== 89360+0+0 (secure 0 0 0) 0x7fe06400f9c0 con 0x7fe078159bd0 2023-10-19T14:18:19.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:19.286+0000 7fe0757fa640 1 --2- 172.21.15.57:0/1132172930 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7fe05c06afb0 0x7fe05c06d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:19.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:19.286+0000 7fe0757fa640 1 -- 172.21.15.57:0/1132172930 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 6745+0+0 (secure 0 0 0) 0x7fe06408a4c0 con 0x7fe078159bd0 2023-10-19T14:18:19.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:19.286+0000 7fe07ca73640 1 --2- 172.21.15.57:0/1132172930 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7fe05c06afb0 0x7fe05c06d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:19.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:19.287+0000 7fe07ca73640 1 --2- 172.21.15.57:0/1132172930 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7fe05c06afb0 0x7fe05c06d470 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe06c035000 tx=0x7fe06c004730 comp rx=0 tx=0).ready entity=mgr.9784 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:19.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:19.290+0000 7fe0757fa640 1 -- 172.21.15.57:0/1132172930 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe064054fd0 con 0x7fe078159bd0 2023-10-19T14:18:19.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:19.514+0000 7fe07ecd7640 1 -- 172.21.15.57:0/1132172930 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7fe0780690e0 con 0x7fe078159bd0 2023-10-19T14:18:20.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.377+0000 7fe0757fa640 1 -- 172.21.15.57:0/1132172930 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v113) v1 ==== 93+0+0 (secure 0 0 0) 0x7fe0640585f0 con 0x7fe078159bd0 2023-10-19T14:18:20.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.380+0000 7fe07ecd7640 1 -- 172.21.15.57:0/1132172930 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7fe05c06afb0 msgr2=0x7fe05c06d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:20.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.380+0000 7fe07ecd7640 1 --2- 172.21.15.57:0/1132172930 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7fe05c06afb0 0x7fe05c06d470 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe06c035000 tx=0x7fe06c004730 comp rx=0 tx=0).stop 2023-10-19T14:18:20.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.380+0000 7fe07ecd7640 1 -- 172.21.15.57:0/1132172930 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe078159bd0 msgr2=0x7fe07813cab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:20.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.380+0000 7fe07ecd7640 1 --2- 172.21.15.57:0/1132172930 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe078159bd0 0x7fe07813cab0 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 crypto rx=0x7fe06400c920 tx=0x7fe06400cde0 comp rx=0 tx=0).stop 2023-10-19T14:18:20.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.381+0000 7fe07ecd7640 1 -- 172.21.15.57:0/1132172930 shutdown_connections 2023-10-19T14:18:20.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.381+0000 7fe07ecd7640 1 --2- 172.21.15.57:0/1132172930 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7fe05c06afb0 0x7fe05c06d470 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:20.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.381+0000 7fe07ecd7640 1 --2- 172.21.15.57:0/1132172930 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe07815ccb0 0x7fe07813cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:20.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.381+0000 7fe07ecd7640 1 --2- 172.21.15.57:0/1132172930 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe078159bd0 0x7fe07813cab0 unknown :-1 s=CLOSED pgs=2099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:20.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.381+0000 7fe07ecd7640 1 --2- 172.21.15.57:0/1132172930 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe07806b030 0x7fe0780aab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:20.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.381+0000 7fe07ecd7640 1 -- 172.21.15.57:0/1132172930 >> 172.21.15.57:0/1132172930 conn(0x7fe078069930 msgr2=0x7fe0780ad7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:20.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.381+0000 7fe07ecd7640 1 -- 172.21.15.57:0/1132172930 shutdown_connections 2023-10-19T14:18:20.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.382+0000 7fe07ecd7640 1 -- 172.21.15.57:0/1132172930 wait complete. 2023-10-19T14:18:20.397 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-19T14:18:20.398 DEBUG:teuthology.orchestra.run.smithi057:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-19T14:18:20.457 DEBUG:teuthology.orchestra.run.smithi179:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-19T14:18:20.499 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-19T14:18:20.499 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T14:18:20.500 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T14:18:20.516 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T14:18:20.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.663+0000 7fd8c6bd7640 1 -- 172.21.15.57:0/3250217254 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd8c00cf340 msgr2=0x7fd8c00d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:20.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.663+0000 7fd8c6bd7640 1 --2- 172.21.15.57:0/3250217254 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd8c00cf340 0x7fd8c00d1910 secure :-1 s=READY pgs=2102 cs=0 l=1 rev1=1 crypto rx=0x7fd8b4005fd0 tx=0x7fd8b402bd90 comp rx=0 tx=0).stop 2023-10-19T14:18:20.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.663+0000 7fd8c6bd7640 1 -- 172.21.15.57:0/3250217254 shutdown_connections 2023-10-19T14:18:20.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.663+0000 7fd8c6bd7640 1 --2- 172.21.15.57:0/3250217254 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd8c015b2a0 0x7fd8c00cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:20.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.663+0000 7fd8c6bd7640 1 --2- 172.21.15.57:0/3250217254 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd8c00cf340 0x7fd8c00d1910 unknown :-1 s=CLOSED pgs=2102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:20.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.663+0000 7fd8c6bd7640 1 -- 172.21.15.57:0/3250217254 >> 172.21.15.57:0/3250217254 conn(0x7fd8c00d0d60 msgr2=0x7fd8c00d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:20.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.663+0000 7fd8c6bd7640 1 -- 172.21.15.57:0/3250217254 shutdown_connections 2023-10-19T14:18:20.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.663+0000 7fd8c6bd7640 1 -- 172.21.15.57:0/3250217254 wait complete. 2023-10-19T14:18:20.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.664+0000 7fd8c6bd7640 1 Processor -- start 2023-10-19T14:18:20.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.664+0000 7fd8c6bd7640 1 -- start start 2023-10-19T14:18:20.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.665+0000 7fd8c6bd7640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd8c00ceb40 0x7fd8c00b5510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:20.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.665+0000 7fd8c6bd7640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd8c00cf340 0x7fd8c0158d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:20.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.665+0000 7fd8c6bd7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd8c015b2a0 0x7fd8c0157f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:20.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.665+0000 7fd8c6bd7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c015c3b0 con 0x7fd8c015b2a0 2023-10-19T14:18:20.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.665+0000 7fd8c6bd7640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c015b770 con 0x7fd8c00ceb40 2023-10-19T14:18:20.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.665+0000 7fd8c6bd7640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c015ba10 con 0x7fd8c00cf340 2023-10-19T14:18:20.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.665+0000 7fd8c5174640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd8c015b2a0 0x7fd8c0157f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:20.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.665+0000 7fd8c5174640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd8c015b2a0 0x7fd8c0157f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39972/0 (socket says 172.21.15.57:39972) 2023-10-19T14:18:20.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.665+0000 7fd8c5174640 1 -- 172.21.15.57:0/883556704 learned_addr learned my addr 172.21.15.57:0/883556704 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:18:20.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.665+0000 7fd8bffff640 1 --2- 172.21.15.57:0/883556704 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd8c00cf340 0x7fd8c0158d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:20.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.665+0000 7fd8c4973640 1 --2- 172.21.15.57:0/883556704 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd8c00ceb40 0x7fd8c00b5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:20.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.666+0000 7fd8bffff640 1 -- 172.21.15.57:0/883556704 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd8c00ceb40 msgr2=0x7fd8c00b5510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:20.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.666+0000 7fd8bffff640 1 --2- 172.21.15.57:0/883556704 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd8c00ceb40 0x7fd8c00b5510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:20.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.666+0000 7fd8bffff640 1 -- 172.21.15.57:0/883556704 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd8c015b2a0 msgr2=0x7fd8c0157f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:20.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.666+0000 7fd8bffff640 1 --2- 172.21.15.57:0/883556704 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd8c015b2a0 0x7fd8c0157f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:20.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.666+0000 7fd8bffff640 1 -- 172.21.15.57:0/883556704 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8b4009560 con 0x7fd8c00cf340 2023-10-19T14:18:20.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.666+0000 7fd8c5174640 1 --2- 172.21.15.57:0/883556704 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd8c015b2a0 0x7fd8c0157f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T14:18:20.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.666+0000 7fd8bffff640 1 --2- 172.21.15.57:0/883556704 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd8c00cf340 0x7fd8c0158d70 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fd8ac00e6d0 tx=0x7fd8ac00eb90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:20.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.666+0000 7fd8bd7fa640 1 -- 172.21.15.57:0/883556704 <== mon.2 v2:172.21.15.179:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8ac00cab0 con 0x7fd8c00cf340 2023-10-19T14:18:20.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.666+0000 7fd8bd7fa640 1 -- 172.21.15.57:0/883556704 <== mon.2 v2:172.21.15.179:3301/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fd8ac00cd00 con 0x7fd8c00cf340 2023-10-19T14:18:20.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.666+0000 7fd8c6bd7640 1 -- 172.21.15.57:0/883556704 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8c015bcf0 con 0x7fd8c00cf340 2023-10-19T14:18:20.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.666+0000 7fd8bd7fa640 1 -- 172.21.15.57:0/883556704 <== mon.2 v2:172.21.15.179:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8ac010690 con 0x7fd8c00cf340 2023-10-19T14:18:20.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.666+0000 7fd8c6bd7640 1 -- 172.21.15.57:0/883556704 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8c00b5a50 con 0x7fd8c00cf340 2023-10-19T14:18:20.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.667+0000 7fd8c6bd7640 1 -- 172.21.15.57:0/883556704 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8c00690e0 con 0x7fd8c00cf340 2023-10-19T14:18:20.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.668+0000 7fd8bd7fa640 1 -- 172.21.15.57:0/883556704 <== mon.2 v2:172.21.15.179:3301/0 4 ==== mgrmap(e 113) v1 ==== 89343+0+0 (secure 0 0 0) 0x7fd8ac0107f0 con 0x7fd8c00cf340 2023-10-19T14:18:20.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.668+0000 7fd8bd7fa640 1 --2- 172.21.15.57:0/883556704 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7fd8a406af60 0x7fd8a406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:20.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.668+0000 7fd8bd7fa640 1 -- 172.21.15.57:0/883556704 <== mon.2 v2:172.21.15.179:3301/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 6745+0+0 (secure 0 0 0) 0x7fd8ac014070 con 0x7fd8c00cf340 2023-10-19T14:18:20.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.668+0000 7fd8c4973640 1 -- 172.21.15.57:0/883556704 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7fd8a406af60 msgr2=0x7fd8a406d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/4086068793 2023-10-19T14:18:20.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.668+0000 7fd8c4973640 1 --2- 172.21.15.57:0/883556704 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7fd8a406af60 0x7fd8a406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T14:18:20.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.673+0000 7fd8bd7fa640 1 -- 172.21.15.57:0/883556704 <== mon.2 v2:172.21.15.179:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd8ac051ca0 con 0x7fd8c00cf340 2023-10-19T14:18:20.765 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:18:20.763+0000 7f90fd439040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-19T14:18:20.788 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:18:20.786+0000 7fbd89e45040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-19T14:18:20.810 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:18:20.808+0000 7f90fd439040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-19T14:18:20.835 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:18:20.833+0000 7fbd89e45040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-19T14:18:20.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.851+0000 7fd8c6bd7640 1 -- 172.21.15.57:0/883556704 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd8c00d0b40 con 0x7fd8c00cf340 2023-10-19T14:18:20.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.852+0000 7fd8bd7fa640 1 -- 172.21.15.57:0/883556704 <== mon.2 v2:172.21.15.179:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v190) v1 ==== 74+0+17788 (secure 0 0 0) 0x7fd8ac055120 con 0x7fd8c00cf340 2023-10-19T14:18:20.855 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:18:20.856 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":190,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T14:13:20.213698+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2023-10-19T14:02:08.318466+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"187","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2023-10-19T14:02:08.995986+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"188","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.179:0/1094895803":"2023-10-20T14:13:20.213663+0000","172.21.15.179:6832/2587699336":"2023-10-20T14:13:20.213663+0000","172.21.15.179:0/2295535972":"2023-10-20T14:13:20.213663+0000","172.21.15.179:0/1009979748":"2023-10-20T14:13:20.213663+0000","172.21.15.179:6833/2587699336":"2023-10-20T14:13:20.213663+0000","172.21.15.179:0/2961439032":"2023-10-20T14:13:20.213663+0000","172.21.15.179:0/1362323478":"2023-10-20T14:13:20.213663+0000","172.21.15.179:0/2642945549":"2023-10-20T14:13:20.213663+0000","172.21.15.179:0/1066702535":"2023-10-20T14:02:21.481635+0000","172.21.15.179:0/904747968":"2023-10-20T14:02:21.481635+0000","172.21.15.179:0/1902799271":"2023-10-20T14:02:21.481635+0000","172.21.15.179:0/2939688089":"2023-10-20T14:02:21.481635+0000","172.21.15.179:0/682180483":"2023-10-20T14:02:21.481635+0000","172.21.15.179:6833/1426553632":"2023-10-20T14:02:21.481635+0000","172.21.15.179:6832/1426553632":"2023-10-20T14:02:21.481635+0000","172.21.15.179:0/2321890842":"2023-10-20T14:02:21.481635+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T14:18:20.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.854+0000 7fd8c6bd7640 1 -- 172.21.15.57:0/883556704 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7fd8a406af60 msgr2=0x7fd8a406d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T14:18:20.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.854+0000 7fd8c6bd7640 1 --2- 172.21.15.57:0/883556704 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7fd8a406af60 0x7fd8a406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:20.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.854+0000 7fd8c6bd7640 1 -- 172.21.15.57:0/883556704 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd8c00cf340 msgr2=0x7fd8c0158d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:20.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.854+0000 7fd8c6bd7640 1 --2- 172.21.15.57:0/883556704 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd8c00cf340 0x7fd8c0158d70 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fd8ac00e6d0 tx=0x7fd8ac00eb90 comp rx=0 tx=0).stop 2023-10-19T14:18:20.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.854+0000 7fd8c6bd7640 1 -- 172.21.15.57:0/883556704 shutdown_connections 2023-10-19T14:18:20.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.854+0000 7fd8c6bd7640 1 --2- 172.21.15.57:0/883556704 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7fd8a406af60 0x7fd8a406d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:20.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.854+0000 7fd8c6bd7640 1 --2- 172.21.15.57:0/883556704 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd8c015b2a0 0x7fd8c0157f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:20.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.854+0000 7fd8c6bd7640 1 --2- 172.21.15.57:0/883556704 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd8c00cf340 0x7fd8c0158d70 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:20.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.854+0000 7fd8c6bd7640 1 --2- 172.21.15.57:0/883556704 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd8c00ceb40 0x7fd8c00b5510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:20.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.854+0000 7fd8c6bd7640 1 -- 172.21.15.57:0/883556704 >> 172.21.15.57:0/883556704 conn(0x7fd8c00d0d60 msgr2=0x7fd8c015a2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:20.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.854+0000 7fd8c6bd7640 1 -- 172.21.15.57:0/883556704 shutdown_connections 2023-10-19T14:18:20.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.854+0000 7fd8c6bd7640 1 -- 172.21.15.57:0/883556704 wait complete. 2023-10-19T14:18:20.868 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-19T12:44:07.371186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 23, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-19T14:02:08.318466+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '187', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 24, 'pool_name': 'cephfs_data', 'create_time': '2023-10-19T14:02:08.995986+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '188', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-19T14:18:20.869 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-19T14:18:20.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.980+0000 7ff9b7abb640 1 -- 172.21.15.57:0/2287518130 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9b00cf340 msgr2=0x7ff9b00d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:20.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.980+0000 7ff9b7abb640 1 --2- 172.21.15.57:0/2287518130 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9b00cf340 0x7ff9b00d1910 secure :-1 s=READY pgs=2103 cs=0 l=1 rev1=1 crypto rx=0x7ff9b006c850 tx=0x7ff9ac02bd10 comp rx=0 tx=0).stop 2023-10-19T14:18:20.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.980+0000 7ff9b7abb640 1 -- 172.21.15.57:0/2287518130 shutdown_connections 2023-10-19T14:18:20.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.980+0000 7ff9b7abb640 1 --2- 172.21.15.57:0/2287518130 >> v2:172.21.15.179:3300/0 conn(0x7ff9b00ceb40 0x7ff9b00b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:20.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.980+0000 7ff9b7abb640 1 --2- 172.21.15.57:0/2287518130 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff9b015b2a0 0x7ff9b00cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:20.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.980+0000 7ff9b7abb640 1 --2- 172.21.15.57:0/2287518130 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9b00cf340 0x7ff9b00d1910 unknown :-1 s=CLOSED pgs=2103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:20.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.980+0000 7ff9b7abb640 1 -- 172.21.15.57:0/2287518130 >> 172.21.15.57:0/2287518130 conn(0x7ff9b00d0d60 msgr2=0x7ff9b00d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:20.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.980+0000 7ff9b7abb640 1 -- 172.21.15.57:0/2287518130 shutdown_connections 2023-10-19T14:18:20.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.980+0000 7ff9b7abb640 1 -- 172.21.15.57:0/2287518130 wait complete. 2023-10-19T14:18:20.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.981+0000 7ff9b7abb640 1 Processor -- start 2023-10-19T14:18:20.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.981+0000 7ff9b7abb640 1 -- start start 2023-10-19T14:18:20.989 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.982+0000 7ff9b7abb640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff9b00ceb40 0x7ff9b0157d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:20.989 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.982+0000 7ff9b7abb640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9b00cf340 0x7ff9b00ccd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:20.989 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.982+0000 7ff9b7abb640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff9b015b2a0 0x7ff9b00cd270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:20.990 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.982+0000 7ff9b7abb640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b01e4ed0 con 0x7ff9b00cf340 2023-10-19T14:18:20.990 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.982+0000 7ff9b7abb640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b0158920 con 0x7ff9b015b2a0 2023-10-19T14:18:20.990 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.982+0000 7ff9b7abb640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b01e48a0 con 0x7ff9b00ceb40 2023-10-19T14:18:20.990 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.982+0000 7ff9b5056640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9b00cf340 0x7ff9b00ccd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:20.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.982+0000 7ff9b5056640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9b00cf340 0x7ff9b00ccd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39980/0 (socket says 172.21.15.57:39980) 2023-10-19T14:18:20.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.982+0000 7ff9b5056640 1 -- 172.21.15.57:0/3403368383 learned_addr learned my addr 172.21.15.57:0/3403368383 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:18:20.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.982+0000 7ff9b5857640 1 --2- 172.21.15.57:0/3403368383 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff9b00ceb40 0x7ff9b0157d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:20.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.982+0000 7ff9b6058640 1 --2- 172.21.15.57:0/3403368383 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff9b015b2a0 0x7ff9b00cd270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:20.992 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.983+0000 7ff9b5056640 1 -- 172.21.15.57:0/3403368383 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff9b00ceb40 msgr2=0x7ff9b0157d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:20.992 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.983+0000 7ff9b5056640 1 --2- 172.21.15.57:0/3403368383 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff9b00ceb40 0x7ff9b0157d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:20.992 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.983+0000 7ff9b5056640 1 -- 172.21.15.57:0/3403368383 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff9b015b2a0 msgr2=0x7ff9b00cd270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:20.992 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.983+0000 7ff9b5056640 1 --2- 172.21.15.57:0/3403368383 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff9b015b2a0 0x7ff9b00cd270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:20.992 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.983+0000 7ff9b5056640 1 -- 172.21.15.57:0/3403368383 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9ac009560 con 0x7ff9b00cf340 2023-10-19T14:18:20.993 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.983+0000 7ff9b5056640 1 --2- 172.21.15.57:0/3403368383 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9b00cf340 0x7ff9b00ccd30 secure :-1 s=READY pgs=2104 cs=0 l=1 rev1=1 crypto rx=0x7ff9a000d740 tx=0x7ff9a000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:20.993 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.983+0000 7ff9a67fc640 1 -- 172.21.15.57:0/3403368383 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9a0009bd0 con 0x7ff9b00cf340 2023-10-19T14:18:20.993 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.983+0000 7ff9a67fc640 1 -- 172.21.15.57:0/3403368383 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7ff9a0009e20 con 0x7ff9b00cf340 2023-10-19T14:18:20.993 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.983+0000 7ff9b7abb640 1 -- 172.21.15.57:0/3403368383 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9b00cad30 con 0x7ff9b00cf340 2023-10-19T14:18:20.993 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.983+0000 7ff9a67fc640 1 -- 172.21.15.57:0/3403368383 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9a00106c0 con 0x7ff9b00cf340 2023-10-19T14:18:20.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.983+0000 7ff9b7abb640 1 -- 172.21.15.57:0/3403368383 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9b01e4af0 con 0x7ff9b00cf340 2023-10-19T14:18:20.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.984+0000 7ff9b7abb640 1 -- 172.21.15.57:0/3403368383 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9b00690e0 con 0x7ff9b00cf340 2023-10-19T14:18:20.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.985+0000 7ff9a67fc640 1 -- 172.21.15.57:0/3403368383 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 113) v1 ==== 89343+0+0 (secure 0 0 0) 0x7ff9a0010820 con 0x7ff9b00cf340 2023-10-19T14:18:20.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.985+0000 7ff9a67fc640 1 --2- 172.21.15.57:0/3403368383 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7ff99406af60 0x7ff99406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:20.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.985+0000 7ff9b5857640 1 -- 172.21.15.57:0/3403368383 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7ff99406af60 msgr2=0x7ff99406d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/4086068793 2023-10-19T14:18:20.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.985+0000 7ff9a67fc640 1 -- 172.21.15.57:0/3403368383 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 6745+0+0 (secure 0 0 0) 0x7ff9a0089fe0 con 0x7ff9b00cf340 2023-10-19T14:18:20.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.985+0000 7ff9b5857640 1 --2- 172.21.15.57:0/3403368383 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7ff99406af60 0x7ff99406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T14:18:20.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:20.990+0000 7ff9a67fc640 1 -- 172.21.15.57:0/3403368383 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff9a0054ba0 con 0x7ff9b00cf340 2023-10-19T14:18:21.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.186+0000 7ff9b5857640 1 -- 172.21.15.57:0/3403368383 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7ff99406af60 msgr2=0x7ff99406d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/4086068793 2023-10-19T14:18:21.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.186+0000 7ff9b5857640 1 --2- 172.21.15.57:0/3403368383 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7ff99406af60 0x7ff99406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-19T14:18:21.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.192+0000 7ff9b7abb640 1 -- 172.21.15.57:0/3403368383 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff9b00d0b40 con 0x7ff9b00cf340 2023-10-19T14:18:21.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.192+0000 7ff9a67fc640 1 -- 172.21.15.57:0/3403368383 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v190) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff9a00170a0 con 0x7ff9b00cf340 2023-10-19T14:18:21.195 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-19T14:18:21.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.194+0000 7ff9b7abb640 1 -- 172.21.15.57:0/3403368383 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7ff99406af60 msgr2=0x7ff99406d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T14:18:21.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.194+0000 7ff9b7abb640 1 --2- 172.21.15.57:0/3403368383 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7ff99406af60 0x7ff99406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:21.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.194+0000 7ff9b7abb640 1 -- 172.21.15.57:0/3403368383 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9b00cf340 msgr2=0x7ff9b00ccd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:21.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.194+0000 7ff9b7abb640 1 --2- 172.21.15.57:0/3403368383 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9b00cf340 0x7ff9b00ccd30 secure :-1 s=READY pgs=2104 cs=0 l=1 rev1=1 crypto rx=0x7ff9a000d740 tx=0x7ff9a000dc00 comp rx=0 tx=0).stop 2023-10-19T14:18:21.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.194+0000 7ff9b7abb640 1 -- 172.21.15.57:0/3403368383 shutdown_connections 2023-10-19T14:18:21.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.194+0000 7ff9b7abb640 1 --2- 172.21.15.57:0/3403368383 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7ff99406af60 0x7ff99406d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:21.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.194+0000 7ff9b7abb640 1 --2- 172.21.15.57:0/3403368383 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff9b015b2a0 0x7ff9b00cd270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:21.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.194+0000 7ff9b7abb640 1 --2- 172.21.15.57:0/3403368383 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9b00cf340 0x7ff9b00ccd30 unknown :-1 s=CLOSED pgs=2104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:21.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.194+0000 7ff9b7abb640 1 --2- 172.21.15.57:0/3403368383 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff9b00ceb40 0x7ff9b0157d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:21.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.194+0000 7ff9b7abb640 1 -- 172.21.15.57:0/3403368383 >> 172.21.15.57:0/3403368383 conn(0x7ff9b00d0d60 msgr2=0x7ff9b0069f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:21.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.194+0000 7ff9b7abb640 1 -- 172.21.15.57:0/3403368383 shutdown_connections 2023-10-19T14:18:21.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.194+0000 7ff9b7abb640 1 -- 172.21.15.57:0/3403368383 wait complete. 2023-10-19T14:18:21.202 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:18:21.200+0000 7f90fd439040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-19T14:18:21.208 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-19T14:18:21.246 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:18:21.243+0000 7f90fd439040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-19T14:18:21.275 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:18:21.272+0000 7fbd89e45040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-19T14:18:21.288 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:18:21.285+0000 7f90fd439040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-19T14:18:21.321 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.318+0000 7f0ea1d1d640 1 -- 172.21.15.57:0/601441816 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0e9c0cf340 msgr2=0x7f0e9c0d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:21.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.319+0000 7f0ea1d1d640 1 --2- 172.21.15.57:0/601441816 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0e9c0cf340 0x7f0e9c0d1910 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c009830 tx=0x7f0e8c02bb90 comp rx=0 tx=0).stop 2023-10-19T14:18:21.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.319+0000 7f0ea1d1d640 1 -- 172.21.15.57:0/601441816 shutdown_connections 2023-10-19T14:18:21.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.319+0000 7f0ea1d1d640 1 --2- 172.21.15.57:0/601441816 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0e9c0ceb40 0x7f0e9c0b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:21.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.319+0000 7f0ea1d1d640 1 --2- 172.21.15.57:0/601441816 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0e9c0cf340 0x7f0e9c0d1910 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:21.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.319+0000 7f0ea1d1d640 1 -- 172.21.15.57:0/601441816 >> 172.21.15.57:0/601441816 conn(0x7f0e9c0d0d60 msgr2=0x7f0e9c0d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:21.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.319+0000 7f0ea1d1d640 1 -- 172.21.15.57:0/601441816 shutdown_connections 2023-10-19T14:18:21.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.319+0000 7f0ea1d1d640 1 -- 172.21.15.57:0/601441816 wait complete. 2023-10-19T14:18:21.324 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:18:21.319+0000 7fbd89e45040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-19T14:18:21.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.320+0000 7f0ea1d1d640 1 Processor -- start 2023-10-19T14:18:21.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.320+0000 7f0ea1d1d640 1 -- start start 2023-10-19T14:18:21.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.321+0000 7f0ea1d1d640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0e9c0ceb40 0x7f0e9c15dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:21.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.321+0000 7f0ea1d1d640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0e9c0cf340 0x7f0e9c1493a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:21.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.321+0000 7f0ea1d1d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0e9c15b2a0 0x7f0e9c152190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:21.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.321+0000 7f0ea1d1d640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e9c1498e0 con 0x7f0e9c15b2a0 2023-10-19T14:18:21.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.321+0000 7f0ea1d1d640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e9c1526d0 con 0x7f0e9c0ceb40 2023-10-19T14:18:21.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.321+0000 7f0ea1d1d640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e9c152900 con 0x7f0e9c0cf340 2023-10-19T14:18:21.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.321+0000 7f0e9bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0e9c15b2a0 0x7f0e9c152190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:21.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.321+0000 7f0e9bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0e9c15b2a0 0x7f0e9c152190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39992/0 (socket says 172.21.15.57:39992) 2023-10-19T14:18:21.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.321+0000 7f0e9bfff640 1 -- 172.21.15.57:0/2920143950 learned_addr learned my addr 172.21.15.57:0/2920143950 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:18:21.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.321+0000 7f0e9b7fe640 1 --2- 172.21.15.57:0/2920143950 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0e9c0ceb40 0x7f0e9c15dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:21.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.321+0000 7f0e9affd640 1 --2- 172.21.15.57:0/2920143950 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0e9c0cf340 0x7f0e9c1493a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:21.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.321+0000 7f0e9bfff640 1 -- 172.21.15.57:0/2920143950 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0e9c0cf340 msgr2=0x7f0e9c1493a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:21.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.321+0000 7f0e9bfff640 1 --2- 172.21.15.57:0/2920143950 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0e9c0cf340 0x7f0e9c1493a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:21.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.321+0000 7f0e9bfff640 1 -- 172.21.15.57:0/2920143950 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0e9c0ceb40 msgr2=0x7f0e9c15dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:21.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.321+0000 7f0e9bfff640 1 --2- 172.21.15.57:0/2920143950 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0e9c0ceb40 0x7f0e9c15dbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:21.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.321+0000 7f0e9bfff640 1 -- 172.21.15.57:0/2920143950 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e8c009490 con 0x7f0e9c15b2a0 2023-10-19T14:18:21.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.322+0000 7f0e9bfff640 1 --2- 172.21.15.57:0/2920143950 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0e9c15b2a0 0x7f0e9c152190 secure :-1 s=READY pgs=2106 cs=0 l=1 rev1=1 crypto rx=0x7f0e9400eb10 tx=0x7f0e9400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:21.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.322+0000 7f0e93fff640 1 -- 172.21.15.57:0/2920143950 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e94004b20 con 0x7f0e9c15b2a0 2023-10-19T14:18:21.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.322+0000 7f0e93fff640 1 -- 172.21.15.57:0/2920143950 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f0e94004ca0 con 0x7f0e9c15b2a0 2023-10-19T14:18:21.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.322+0000 7f0e93fff640 1 -- 172.21.15.57:0/2920143950 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e94005280 con 0x7f0e9c15b2a0 2023-10-19T14:18:21.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.322+0000 7f0ea1d1d640 1 -- 172.21.15.57:0/2920143950 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e9c0b0d50 con 0x7f0e9c15b2a0 2023-10-19T14:18:21.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.322+0000 7f0ea1d1d640 1 -- 172.21.15.57:0/2920143950 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e9c152ec0 con 0x7f0e9c15b2a0 2023-10-19T14:18:21.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.323+0000 7f0ea1d1d640 1 -- 172.21.15.57:0/2920143950 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e9c15ae00 con 0x7f0e9c15b2a0 2023-10-19T14:18:21.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.323+0000 7f0e93fff640 1 -- 172.21.15.57:0/2920143950 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 113) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f0e940053e0 con 0x7f0e9c15b2a0 2023-10-19T14:18:21.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.324+0000 7f0e93fff640 1 --2- 172.21.15.57:0/2920143950 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7f0e7c06af60 0x7f0e7c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:21.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.324+0000 7f0e9b7fe640 1 -- 172.21.15.57:0/2920143950 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7f0e7c06af60 msgr2=0x7f0e7c06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/4086068793 2023-10-19T14:18:21.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.324+0000 7f0e93fff640 1 -- 172.21.15.57:0/2920143950 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 6745+0+0 (secure 0 0 0) 0x7f0e94014070 con 0x7f0e9c15b2a0 2023-10-19T14:18:21.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.324+0000 7f0e9b7fe640 1 --2- 172.21.15.57:0/2920143950 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7f0e7c06af60 0x7f0e7c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T14:18:21.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.328+0000 7f0e93fff640 1 -- 172.21.15.57:0/2920143950 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0e94055450 con 0x7f0e9c15b2a0 2023-10-19T14:18:21.339 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:18:21.337+0000 7f90fd439040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-19T14:18:21.368 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:18:21.365+0000 7fbd89e45040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-19T14:18:21.381 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:18:21.378+0000 7f90fd439040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-19T14:18:21.454 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:18:21.452+0000 7fbd89e45040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-19T14:18:21.504 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:18:21.501+0000 7fbd89e45040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-19T14:18:21.526 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:18:21.523+0000 7f90fd439040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-19T14:18:21.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.525+0000 7f0e9b7fe640 1 -- 172.21.15.57:0/2920143950 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7f0e7c06af60 msgr2=0x7f0e7c06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/4086068793 2023-10-19T14:18:21.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.525+0000 7f0e9b7fe640 1 --2- 172.21.15.57:0/2920143950 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7f0e7c06af60 0x7f0e7c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-19T14:18:21.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.552+0000 7f0ea1d1d640 1 -- 172.21.15.57:0/2920143950 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f0e9c0d04f0 con 0x7f0e9c15b2a0 2023-10-19T14:18:21.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.552+0000 7f0e93fff640 1 -- 172.21.15.57:0/2920143950 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v191) v1 ==== 104+0+11 (secure 0 0 0) 0x7f0e94058a70 con 0x7f0e9c15b2a0 2023-10-19T14:18:21.555 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2023-10-19T14:18:21.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.554+0000 7f0ea1d1d640 1 -- 172.21.15.57:0/2920143950 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7f0e7c06af60 msgr2=0x7f0e7c06d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T14:18:21.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.554+0000 7f0ea1d1d640 1 --2- 172.21.15.57:0/2920143950 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7f0e7c06af60 0x7f0e7c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:21.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.554+0000 7f0ea1d1d640 1 -- 172.21.15.57:0/2920143950 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0e9c15b2a0 msgr2=0x7f0e9c152190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:21.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.554+0000 7f0ea1d1d640 1 --2- 172.21.15.57:0/2920143950 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0e9c15b2a0 0x7f0e9c152190 secure :-1 s=READY pgs=2106 cs=0 l=1 rev1=1 crypto rx=0x7f0e9400eb10 tx=0x7f0e9400c2d0 comp rx=0 tx=0).stop 2023-10-19T14:18:21.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.554+0000 7f0ea1d1d640 1 -- 172.21.15.57:0/2920143950 shutdown_connections 2023-10-19T14:18:21.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.554+0000 7f0ea1d1d640 1 --2- 172.21.15.57:0/2920143950 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7f0e7c06af60 0x7f0e7c06d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:21.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.554+0000 7f0ea1d1d640 1 --2- 172.21.15.57:0/2920143950 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0e9c15b2a0 0x7f0e9c152190 unknown :-1 s=CLOSED pgs=2106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:21.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.554+0000 7f0ea1d1d640 1 --2- 172.21.15.57:0/2920143950 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0e9c0cf340 0x7f0e9c1493a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:21.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.554+0000 7f0ea1d1d640 1 --2- 172.21.15.57:0/2920143950 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0e9c0ceb40 0x7f0e9c15dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:21.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.554+0000 7f0ea1d1d640 1 -- 172.21.15.57:0/2920143950 >> 172.21.15.57:0/2920143950 conn(0x7f0e9c0d0d60 msgr2=0x7f0e9c0ad8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:21.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.554+0000 7f0ea1d1d640 1 -- 172.21.15.57:0/2920143950 shutdown_connections 2023-10-19T14:18:21.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.554+0000 7f0ea1d1d640 1 -- 172.21.15.57:0/2920143950 wait complete. 2023-10-19T14:18:21.568 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-19T14:18:21.579 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:18:21.577+0000 7f90fd439040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-19T14:18:21.626 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:18:21.623+0000 7f90fd439040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-19T14:18:21.630 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:18:21.627+0000 7fbd89e45040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-19T14:18:21.673 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:18:21.671+0000 7f90fd439040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-19T14:18:21.688 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:18:21.685+0000 7fbd89e45040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-19T14:18:21.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.692+0000 7f7910501640 1 -- 172.21.15.57:0/1100234915 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f79080cf340 msgr2=0x7f79080d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:21.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.692+0000 7f7910501640 1 --2- 172.21.15.57:0/1100234915 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f79080cf340 0x7f79080d1910 secure :-1 s=READY pgs=2107 cs=0 l=1 rev1=1 crypto rx=0x7f78f8005fd0 tx=0x7f78f802bd90 comp rx=0 tx=0).stop 2023-10-19T14:18:21.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.692+0000 7f7910501640 1 -- 172.21.15.57:0/1100234915 shutdown_connections 2023-10-19T14:18:21.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.692+0000 7f7910501640 1 --2- 172.21.15.57:0/1100234915 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f790815b2a0 0x7f79080cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:21.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.692+0000 7f7910501640 1 --2- 172.21.15.57:0/1100234915 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f79080cf340 0x7f79080d1910 unknown :-1 s=CLOSED pgs=2107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:21.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.692+0000 7f7910501640 1 -- 172.21.15.57:0/1100234915 >> 172.21.15.57:0/1100234915 conn(0x7f79080d0d60 msgr2=0x7f79080d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:21.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.692+0000 7f7910501640 1 -- 172.21.15.57:0/1100234915 shutdown_connections 2023-10-19T14:18:21.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.692+0000 7f7910501640 1 -- 172.21.15.57:0/1100234915 wait complete. 2023-10-19T14:18:21.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.693+0000 7f7910501640 1 Processor -- start 2023-10-19T14:18:21.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.693+0000 7f7910501640 1 -- start start 2023-10-19T14:18:21.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.694+0000 7f7910501640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f79080ceb40 0x7f79081582c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:21.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.694+0000 7f7910501640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f79080cf340 0x7f79081f4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:21.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.694+0000 7f7910501640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f790815b2a0 0x7f79081f52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:21.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.694+0000 7f7910501640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f790815bd00 con 0x7f790815b2a0 2023-10-19T14:18:21.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.694+0000 7f7910501640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79080ca3a0 con 0x7f79080cf340 2023-10-19T14:18:21.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.694+0000 7f7910501640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f790815b6d0 con 0x7f79080ceb40 2023-10-19T14:18:21.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.694+0000 7f790ea9e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f790815b2a0 0x7f79081f52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:21.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.694+0000 7f790ea9e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f790815b2a0 0x7f79081f52d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40008/0 (socket says 172.21.15.57:40008) 2023-10-19T14:18:21.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.694+0000 7f790ea9e640 1 -- 172.21.15.57:0/4139466622 learned_addr learned my addr 172.21.15.57:0/4139466622 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:18:21.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.694+0000 7f790e29d640 1 --2- 172.21.15.57:0/4139466622 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f79080ceb40 0x7f79081582c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:21.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.694+0000 7f790da9c640 1 --2- 172.21.15.57:0/4139466622 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f79080cf340 0x7f79081f4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:21.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.695+0000 7f790ea9e640 1 -- 172.21.15.57:0/4139466622 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f79080ceb40 msgr2=0x7f79081582c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:21.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.695+0000 7f790ea9e640 1 --2- 172.21.15.57:0/4139466622 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f79080ceb40 0x7f79081582c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:21.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.695+0000 7f790ea9e640 1 -- 172.21.15.57:0/4139466622 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f79080cf340 msgr2=0x7f79081f4d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:21.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.695+0000 7f790ea9e640 1 --2- 172.21.15.57:0/4139466622 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f79080cf340 0x7f79081f4d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:21.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.695+0000 7f790ea9e640 1 -- 172.21.15.57:0/4139466622 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78f8009560 con 0x7f790815b2a0 2023-10-19T14:18:21.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.695+0000 7f790ea9e640 1 --2- 172.21.15.57:0/4139466622 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f790815b2a0 0x7f79081f52d0 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 crypto rx=0x7f790000c740 tx=0x7f790000cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:21.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.695+0000 7f78feffd640 1 -- 172.21.15.57:0/4139466622 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7900006d50 con 0x7f790815b2a0 2023-10-19T14:18:21.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.695+0000 7f78feffd640 1 -- 172.21.15.57:0/4139466622 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f790000ce50 con 0x7f790815b2a0 2023-10-19T14:18:21.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.695+0000 7f78feffd640 1 -- 172.21.15.57:0/4139466622 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7900003bd0 con 0x7f790815b2a0 2023-10-19T14:18:21.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.695+0000 7f7910501640 1 -- 172.21.15.57:0/4139466622 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79081f5bd0 con 0x7f790815b2a0 2023-10-19T14:18:21.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.695+0000 7f7910501640 1 -- 172.21.15.57:0/4139466622 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79080ca0c0 con 0x7f790815b2a0 2023-10-19T14:18:21.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.696+0000 7f7910501640 1 -- 172.21.15.57:0/4139466622 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f790815be40 con 0x7f790815b2a0 2023-10-19T14:18:21.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.697+0000 7f78feffd640 1 -- 172.21.15.57:0/4139466622 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 113) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f7900003d30 con 0x7f790815b2a0 2023-10-19T14:18:21.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.697+0000 7f78feffd640 1 --2- 172.21.15.57:0/4139466622 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7f78ec06af60 0x7f78ec06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:21.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.697+0000 7f790e29d640 1 -- 172.21.15.57:0/4139466622 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7f78ec06af60 msgr2=0x7f78ec06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/4086068793 2023-10-19T14:18:21.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.697+0000 7f78feffd640 1 -- 172.21.15.57:0/4139466622 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(191..191 src has 1..191) v4 ==== 6833+0+0 (secure 0 0 0) 0x7f79000897d0 con 0x7f790815b2a0 2023-10-19T14:18:21.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.697+0000 7f790e29d640 1 --2- 172.21.15.57:0/4139466622 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7f78ec06af60 0x7f78ec06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T14:18:21.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.701+0000 7f78feffd640 1 -- 172.21.15.57:0/4139466622 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f790815be40 con 0x7f790815b2a0 2023-10-19T14:18:21.717 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:18:21.714+0000 7f90fd439040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-19T14:18:21.734 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:18:21.731+0000 7fbd89e45040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-19T14:18:21.785 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:18:21.782+0000 7fbd89e45040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-19T14:18:21.850 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:18:21.848+0000 7fbd89e45040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-19T14:18:21.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.898+0000 7f790e29d640 1 -- 172.21.15.57:0/4139466622 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7f78ec06af60 msgr2=0x7f78ec06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/4086068793 2023-10-19T14:18:21.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.898+0000 7f790e29d640 1 --2- 172.21.15.57:0/4139466622 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7f78ec06af60 0x7f78ec06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-19T14:18:21.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.902+0000 7f7910501640 1 -- 172.21.15.57:0/4139466622 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f790815be40 con 0x7f790815b2a0 2023-10-19T14:18:21.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.902+0000 7f78feffd640 1 -- 172.21.15.57:0/4139466622 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v191) v1 ==== 100+0+11 (secure 0 0 0) 0x7f790815be40 con 0x7f790815b2a0 2023-10-19T14:18:21.905 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 64 2023-10-19T14:18:21.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.904+0000 7f7910501640 1 -- 172.21.15.57:0/4139466622 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7f78ec06af60 msgr2=0x7f78ec06d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T14:18:21.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.904+0000 7f7910501640 1 --2- 172.21.15.57:0/4139466622 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7f78ec06af60 0x7f78ec06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:21.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.904+0000 7f7910501640 1 -- 172.21.15.57:0/4139466622 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f790815b2a0 msgr2=0x7f79081f52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:21.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.904+0000 7f7910501640 1 --2- 172.21.15.57:0/4139466622 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f790815b2a0 0x7f79081f52d0 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 crypto rx=0x7f790000c740 tx=0x7f790000cc00 comp rx=0 tx=0).stop 2023-10-19T14:18:21.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.904+0000 7f7910501640 1 -- 172.21.15.57:0/4139466622 shutdown_connections 2023-10-19T14:18:21.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.904+0000 7f7910501640 1 --2- 172.21.15.57:0/4139466622 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7f78ec06af60 0x7f78ec06d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:21.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.904+0000 7f7910501640 1 --2- 172.21.15.57:0/4139466622 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f790815b2a0 0x7f79081f52d0 unknown :-1 s=CLOSED pgs=2108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:21.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.904+0000 7f7910501640 1 --2- 172.21.15.57:0/4139466622 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f79080cf340 0x7f79081f4d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:21.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.904+0000 7f7910501640 1 --2- 172.21.15.57:0/4139466622 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f79080ceb40 0x7f79081582c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:21.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.905+0000 7f7910501640 1 -- 172.21.15.57:0/4139466622 >> 172.21.15.57:0/4139466622 conn(0x7f79080d0d60 msgr2=0x7f790815a340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:21.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.905+0000 7f7910501640 1 -- 172.21.15.57:0/4139466622 shutdown_connections 2023-10-19T14:18:21.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:21.905+0000 7f7910501640 1 -- 172.21.15.57:0/4139466622 wait complete. 2023-10-19T14:18:21.918 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-19T14:18:22.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.026+0000 7f224ba61640 1 -- 172.21.15.57:0/2613421457 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f224415d400 msgr2=0x7f22440cdfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:22.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.026+0000 7f224ba61640 1 --2- 172.21.15.57:0/2613421457 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f224415d400 0x7f22440cdfb0 secure :-1 s=READY pgs=2109 cs=0 l=1 rev1=1 crypto rx=0x7f223c005fd0 tx=0x7f223c02c240 comp rx=0 tx=0).stop 2023-10-19T14:18:22.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.027+0000 7f224ba61640 1 -- 172.21.15.57:0/2613421457 shutdown_connections 2023-10-19T14:18:22.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.027+0000 7f224ba61640 1 --2- 172.21.15.57:0/2613421457 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f224415d400 0x7f22440cdfb0 unknown :-1 s=CLOSED pgs=2109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:22.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.027+0000 7f224ba61640 1 --2- 172.21.15.57:0/2613421457 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2244159060 0x7f224406d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:22.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.027+0000 7f224ba61640 1 -- 172.21.15.57:0/2613421457 >> 172.21.15.57:0/2613421457 conn(0x7f22440b0670 msgr2=0x7f22440b0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:22.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.027+0000 7f224ba61640 1 -- 172.21.15.57:0/2613421457 shutdown_connections 2023-10-19T14:18:22.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.027+0000 7f224ba61640 1 -- 172.21.15.57:0/2613421457 wait complete. 2023-10-19T14:18:22.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.028+0000 7f224ba61640 1 Processor -- start 2023-10-19T14:18:22.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.028+0000 7f224ba61640 1 -- start start 2023-10-19T14:18:22.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.029+0000 7f224ba61640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f22440cdb30 0x7f22440c19d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:22.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.029+0000 7f224ba61640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2244159060 0x7f2244150b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:22.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.029+0000 7f224ba61640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f224415d400 0x7f22440c27a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:22.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.029+0000 7f224ba61640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22441f1790 con 0x7f22440cdb30 2023-10-19T14:18:22.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.029+0000 7f224ba61640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22441510d0 con 0x7f224415d400 2023-10-19T14:18:22.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.029+0000 7f224ba61640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22440b1440 con 0x7f2244159060 2023-10-19T14:18:22.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.029+0000 7f22497fd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f22440cdb30 0x7f22440c19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:22.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.029+0000 7f22497fd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f22440cdb30 0x7f22440c19d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40032/0 (socket says 172.21.15.57:40032) 2023-10-19T14:18:22.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.029+0000 7f22497fd640 1 -- 172.21.15.57:0/3824844932 learned_addr learned my addr 172.21.15.57:0/3824844932 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:18:22.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.029+0000 7f2249ffe640 1 --2- 172.21.15.57:0/3824844932 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f224415d400 0x7f22440c27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:22.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.029+0000 7f2248ffc640 1 --2- 172.21.15.57:0/3824844932 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2244159060 0x7f2244150b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:22.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.029+0000 7f22497fd640 1 -- 172.21.15.57:0/3824844932 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2244159060 msgr2=0x7f2244150b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:22.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.029+0000 7f22497fd640 1 --2- 172.21.15.57:0/3824844932 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2244159060 0x7f2244150b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:22.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.029+0000 7f22497fd640 1 -- 172.21.15.57:0/3824844932 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f224415d400 msgr2=0x7f22440c27a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:22.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.029+0000 7f22497fd640 1 --2- 172.21.15.57:0/3824844932 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f224415d400 0x7f22440c27a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:22.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.029+0000 7f22497fd640 1 -- 172.21.15.57:0/3824844932 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f223c009900 con 0x7f22440cdb30 2023-10-19T14:18:22.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.029+0000 7f22497fd640 1 --2- 172.21.15.57:0/3824844932 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f22440cdb30 0x7f22440c19d0 secure :-1 s=READY pgs=2110 cs=0 l=1 rev1=1 crypto rx=0x7f223400e9f0 tx=0x7f223400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:22.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.029+0000 7f2239ffb640 1 -- 172.21.15.57:0/3824844932 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2234009700 con 0x7f22440cdb30 2023-10-19T14:18:22.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.029+0000 7f2239ffb640 1 -- 172.21.15.57:0/3824844932 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f223400ce70 con 0x7f22440cdb30 2023-10-19T14:18:22.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.029+0000 7f224ba61640 1 -- 172.21.15.57:0/3824844932 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22440b16e0 con 0x7f22440cdb30 2023-10-19T14:18:22.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.029+0000 7f2239ffb640 1 -- 172.21.15.57:0/3824844932 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2234010670 con 0x7f22440cdb30 2023-10-19T14:18:22.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.030+0000 7f224ba61640 1 -- 172.21.15.57:0/3824844932 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22441f5ce0 con 0x7f22440cdb30 2023-10-19T14:18:22.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.030+0000 7f224ba61640 1 -- 172.21.15.57:0/3824844932 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22441588c0 con 0x7f22440cdb30 2023-10-19T14:18:22.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.031+0000 7f2239ffb640 1 -- 172.21.15.57:0/3824844932 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 113) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f22340107d0 con 0x7f22440cdb30 2023-10-19T14:18:22.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.032+0000 7f2239ffb640 1 --2- 172.21.15.57:0/3824844932 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7f222806af60 0x7f222806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:22.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.032+0000 7f2248ffc640 1 -- 172.21.15.57:0/3824844932 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7f222806af60 msgr2=0x7f222806d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/4086068793 2023-10-19T14:18:22.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.032+0000 7f2239ffb640 1 -- 172.21.15.57:0/3824844932 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(191..191 src has 1..191) v4 ==== 6833+0+0 (secure 0 0 0) 0x7f2234014070 con 0x7f22440cdb30 2023-10-19T14:18:22.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.032+0000 7f2248ffc640 1 --2- 172.21.15.57:0/3824844932 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7f222806af60 0x7f222806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T14:18:22.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.035+0000 7f2239ffb640 1 -- 172.21.15.57:0/3824844932 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2234055cf0 con 0x7f22440cdb30 2023-10-19T14:18:22.041 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:18:22.038+0000 7f90fd439040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-19T14:18:22.185 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:18:22.183+0000 7f90fd439040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-19T14:18:22.200 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:18:22.197+0000 7fbd89e45040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-19T14:18:22.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.233+0000 7f2248ffc640 1 -- 172.21.15.57:0/3824844932 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7f222806af60 msgr2=0x7f222806d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/4086068793 2023-10-19T14:18:22.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.233+0000 7f2248ffc640 1 --2- 172.21.15.57:0/3824844932 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7f222806af60 0x7f222806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-19T14:18:22.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.236+0000 7f224ba61640 1 -- 172.21.15.57:0/3824844932 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f22440675f0 con 0x7f22440cdb30 2023-10-19T14:18:22.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.236+0000 7f2239ffb640 1 -- 172.21.15.57:0/3824844932 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v191) v1 ==== 81+0+954 (secure 0 0 0) 0x7f2234026020 con 0x7f22440cdb30 2023-10-19T14:18:22.239 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/1094895803 2023-10-20T14:13:20.213663+0000 2023-10-19T14:18:22.240 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:6832/2587699336 2023-10-20T14:13:20.213663+0000 2023-10-19T14:18:22.240 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/2295535972 2023-10-20T14:13:20.213663+0000 2023-10-19T14:18:22.240 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/1009979748 2023-10-20T14:13:20.213663+0000 2023-10-19T14:18:22.240 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:6833/2587699336 2023-10-20T14:13:20.213663+0000 2023-10-19T14:18:22.241 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/2961439032 2023-10-20T14:13:20.213663+0000 2023-10-19T14:18:22.241 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/1362323478 2023-10-20T14:13:20.213663+0000 2023-10-19T14:18:22.241 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/2642945549 2023-10-20T14:13:20.213663+0000 2023-10-19T14:18:22.241 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/1066702535 2023-10-20T14:02:21.481635+0000 2023-10-19T14:18:22.241 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/904747968 2023-10-20T14:02:21.481635+0000 2023-10-19T14:18:22.242 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/1902799271 2023-10-20T14:02:21.481635+0000 2023-10-19T14:18:22.242 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/2939688089 2023-10-20T14:02:21.481635+0000 2023-10-19T14:18:22.242 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/682180483 2023-10-20T14:02:21.481635+0000 2023-10-19T14:18:22.242 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:6833/1426553632 2023-10-20T14:02:21.481635+0000 2023-10-19T14:18:22.242 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:6832/1426553632 2023-10-20T14:02:21.481635+0000 2023-10-19T14:18:22.243 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/2321890842 2023-10-20T14:02:21.481635+0000 2023-10-19T14:18:22.244 INFO:teuthology.orchestra.run.smithi057.stderr:listed 16 entries 2023-10-19T14:18:22.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.238+0000 7f224ba61640 1 -- 172.21.15.57:0/3824844932 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7f222806af60 msgr2=0x7f222806d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T14:18:22.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.238+0000 7f224ba61640 1 --2- 172.21.15.57:0/3824844932 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7f222806af60 0x7f222806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:22.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.238+0000 7f224ba61640 1 -- 172.21.15.57:0/3824844932 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f22440cdb30 msgr2=0x7f22440c19d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:22.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.238+0000 7f224ba61640 1 --2- 172.21.15.57:0/3824844932 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f22440cdb30 0x7f22440c19d0 secure :-1 s=READY pgs=2110 cs=0 l=1 rev1=1 crypto rx=0x7f223400e9f0 tx=0x7f223400eeb0 comp rx=0 tx=0).stop 2023-10-19T14:18:22.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.238+0000 7f224ba61640 1 -- 172.21.15.57:0/3824844932 shutdown_connections 2023-10-19T14:18:22.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.238+0000 7f224ba61640 1 --2- 172.21.15.57:0/3824844932 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7f222806af60 0x7f222806d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:22.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.238+0000 7f224ba61640 1 --2- 172.21.15.57:0/3824844932 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f224415d400 0x7f22440c27a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:22.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.238+0000 7f224ba61640 1 --2- 172.21.15.57:0/3824844932 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2244159060 0x7f2244150b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:22.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.238+0000 7f224ba61640 1 --2- 172.21.15.57:0/3824844932 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f22440cdb30 0x7f22440c19d0 unknown :-1 s=CLOSED pgs=2110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:22.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.238+0000 7f224ba61640 1 -- 172.21.15.57:0/3824844932 >> 172.21.15.57:0/3824844932 conn(0x7f22440b0670 msgr2=0x7f2244157e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:22.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.238+0000 7f224ba61640 1 -- 172.21.15.57:0/3824844932 shutdown_connections 2023-10-19T14:18:22.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.238+0000 7f224ba61640 1 -- 172.21.15.57:0/3824844932 wait complete. 2023-10-19T14:18:22.252 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi057.front.sepia.ceph.com... 2023-10-19T14:18:22.252 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:18:22.253 DEBUG:teuthology.orchestra.run.smithi057:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-10-19T14:18:22.277 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:18:22.274+0000 7f90fd439040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-19T14:18:22.282 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi057.stderr:ceph-fuse[145152]: fuse finished with error 0 and tester_r 0 2023-10-19T14:18:22.284 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi057.front.sepia.ceph.com 2023-10-19T14:18:22.285 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:18:22.285 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-19T14:18:22.319 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:18:22.316+0000 7f90fd439040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-19T14:18:22.342 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:18:22.339+0000 7fbd89e45040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-19T14:18:22.357 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-19T14:18:22.357 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T14:18:22.372 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054833,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=284 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3247632k,nr_inodes=811908,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-19T14:18:22.373 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi179.front.sepia.ceph.com 2023-10-19T14:18:22.373 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:18:22.374 DEBUG:teuthology.orchestra.run.smithi179:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-10-19T14:18:22.421 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:18:22.419+0000 7f90fd439040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-19T14:18:22.426 INFO:teuthology.orchestra.run.smithi179.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2023-10-19T14:18:22.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-19T14:18:22.428 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T14:18:22.429 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T14:18:22.444 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055614,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=358 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248260k,nr_inodes=812065,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-19T14:18:22.445 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi057.front.sepia.ceph.com 2023-10-19T14:18:22.445 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:18:22.448 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-10-19T14:18:22.449 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:18:22.443+0000 7fbd89e45040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-19T14:18:22.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-19T14:18:22.465 INFO:teuthology.orchestra.run.smithi057.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2023-10-19T14:18:22.465 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-19T14:18:22.466 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T14:18:22.484 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054833,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=284 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3247632k,nr_inodes=811908,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-19T14:18:22.484 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi179.front.sepia.ceph.com 2023-10-19T14:18:22.485 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:18:22.485 DEBUG:teuthology.orchestra.run.smithi179:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-10-19T14:18:22.492 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:18:22.489+0000 7fbd89e45040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-19T14:18:22.518 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:18:22.515+0000 7f90fd439040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-19T14:18:22.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-19T14:18:22.534 INFO:teuthology.orchestra.run.smithi179.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2023-10-19T14:18:22.534 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T14:18:22.596 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:18:22.593+0000 7fbd89e45040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-19T14:18:22.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.645+0000 7f7ace475640 1 -- 172.21.15.57:0/543532509 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ac80ceb40 msgr2=0x7f7ac80b0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:22.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.645+0000 7f7ace475640 1 --2- 172.21.15.57:0/543532509 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ac80ceb40 0x7f7ac80b0650 secure :-1 s=READY pgs=2111 cs=0 l=1 rev1=1 crypto rx=0x7f7ac808a810 tx=0x7f7ab802bdb0 comp rx=0 tx=0).stop 2023-10-19T14:18:22.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.645+0000 7f7ace475640 1 -- 172.21.15.57:0/543532509 shutdown_connections 2023-10-19T14:18:22.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.645+0000 7f7ace475640 1 --2- 172.21.15.57:0/543532509 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ac80ceb40 0x7f7ac80b0650 unknown :-1 s=CLOSED pgs=2111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:22.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.645+0000 7f7ace475640 1 --2- 172.21.15.57:0/543532509 >> v2:172.21.15.179:3300/0 conn(0x7f7ac80cf340 0x7f7ac80d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:22.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.645+0000 7f7ace475640 1 -- 172.21.15.57:0/543532509 >> 172.21.15.57:0/543532509 conn(0x7f7ac80d0d60 msgr2=0x7f7ac80d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:22.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.645+0000 7f7ace475640 1 -- 172.21.15.57:0/543532509 shutdown_connections 2023-10-19T14:18:22.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.645+0000 7f7ace475640 1 -- 172.21.15.57:0/543532509 wait complete. 2023-10-19T14:18:22.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.646+0000 7f7ace475640 1 Processor -- start 2023-10-19T14:18:22.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.646+0000 7f7ace475640 1 -- start start 2023-10-19T14:18:22.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.647+0000 7f7ace475640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7ac80ceb40 0x7f7ac80c9a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:22.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.647+0000 7f7ace475640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7ac80cf340 0x7f7ac80ca320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:22.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.647+0000 7f7ace475640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ac815b2a0 0x7f7ac8159e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:22.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.647+0000 7f7ace475640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ac80b5010 con 0x7f7ac815b2a0 2023-10-19T14:18:22.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.647+0000 7f7ace475640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ac80b63e0 con 0x7f7ac80ceb40 2023-10-19T14:18:22.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.647+0000 7f7ace475640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ac80b4390 con 0x7f7ac80cf340 2023-10-19T14:18:22.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.647+0000 7f7acca12640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ac815b2a0 0x7f7ac8159e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:22.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.647+0000 7f7ac7fff640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7ac80ceb40 0x7f7ac80c9a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:22.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.647+0000 7f7acca12640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ac815b2a0 0x7f7ac8159e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40042/0 (socket says 172.21.15.57:40042) 2023-10-19T14:18:22.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.647+0000 7f7acca12640 1 -- 172.21.15.57:0/3931644148 learned_addr learned my addr 172.21.15.57:0/3931644148 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:18:22.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.647+0000 7f7ac77fe640 1 --2- 172.21.15.57:0/3931644148 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7ac80cf340 0x7f7ac80ca320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:22.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.648+0000 7f7acca12640 1 -- 172.21.15.57:0/3931644148 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7ac80cf340 msgr2=0x7f7ac80ca320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:22.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.648+0000 7f7acca12640 1 --2- 172.21.15.57:0/3931644148 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7ac80cf340 0x7f7ac80ca320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:22.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.648+0000 7f7acca12640 1 -- 172.21.15.57:0/3931644148 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7ac80ceb40 msgr2=0x7f7ac80c9a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:22.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.648+0000 7f7acca12640 1 --2- 172.21.15.57:0/3931644148 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7ac80ceb40 0x7f7ac80c9a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:22.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.648+0000 7f7acca12640 1 -- 172.21.15.57:0/3931644148 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ab8009560 con 0x7f7ac815b2a0 2023-10-19T14:18:22.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.648+0000 7f7ac7fff640 1 --2- 172.21.15.57:0/3931644148 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7ac80ceb40 0x7f7ac80c9a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T14:18:22.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.648+0000 7f7acca12640 1 --2- 172.21.15.57:0/3931644148 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ac815b2a0 0x7f7ac8159e80 secure :-1 s=READY pgs=2112 cs=0 l=1 rev1=1 crypto rx=0x7f7ab8036040 tx=0x7f7ab802da10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:22.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.648+0000 7f7ac4ff9640 1 -- 172.21.15.57:0/3931644148 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ab8003800 con 0x7f7ac815b2a0 2023-10-19T14:18:22.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.648+0000 7f7ac4ff9640 1 -- 172.21.15.57:0/3931644148 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f7ab8003a90 con 0x7f7ac815b2a0 2023-10-19T14:18:22.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.648+0000 7f7ace475640 1 -- 172.21.15.57:0/3931644148 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ac80b4670 con 0x7f7ac815b2a0 2023-10-19T14:18:22.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.648+0000 7f7ac4ff9640 1 -- 172.21.15.57:0/3931644148 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ab8003ea0 con 0x7f7ac815b2a0 2023-10-19T14:18:22.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.648+0000 7f7ace475640 1 -- 172.21.15.57:0/3931644148 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ac80b4c30 con 0x7f7ac815b2a0 2023-10-19T14:18:22.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.649+0000 7f7ace475640 1 -- 172.21.15.57:0/3931644148 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ac80b5150 con 0x7f7ac815b2a0 2023-10-19T14:18:22.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.650+0000 7f7ac4ff9640 1 -- 172.21.15.57:0/3931644148 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 113) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f7ab803e450 con 0x7f7ac815b2a0 2023-10-19T14:18:22.658 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.651+0000 7f7ac4ff9640 1 --2- 172.21.15.57:0/3931644148 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7f7aa806af60 0x7f7aa806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:22.658 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.651+0000 7f7ac4ff9640 1 -- 172.21.15.57:0/3931644148 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 6817+0+0 (secure 0 0 0) 0x7f7ab80ac630 con 0x7f7ac815b2a0 2023-10-19T14:18:22.658 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.651+0000 7f7ac7fff640 1 -- 172.21.15.57:0/3931644148 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7f7aa806af60 msgr2=0x7f7aa806d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/4086068793 2023-10-19T14:18:22.658 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.651+0000 7f7ac7fff640 1 --2- 172.21.15.57:0/3931644148 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7f7aa806af60 0x7f7aa806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T14:18:22.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.655+0000 7f7ac4ff9640 1 -- 172.21.15.57:0/3931644148 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7ac80b5150 con 0x7f7ac815b2a0 2023-10-19T14:18:22.660 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:18:22.658+0000 7f90fd439040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-19T14:18:22.696 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:18:22.694+0000 7fbd89e45040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-19T14:18:22.702 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:18:22.700+0000 7f90fd439040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-19T14:18:22.807 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:18:22.804+0000 7f90fd439040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-19T14:18:22.841 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:18:22.838+0000 7fbd89e45040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-19T14:18:22.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.852+0000 7f7ac7fff640 1 -- 172.21.15.57:0/3931644148 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7f7aa806af60 msgr2=0x7f7aa806d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/4086068793 2023-10-19T14:18:22.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.852+0000 7f7ac7fff640 1 --2- 172.21.15.57:0/3931644148 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7f7aa806af60 0x7f7aa806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-19T14:18:22.869 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.866+0000 7f7ace475640 1 -- 172.21.15.57:0/3931644148 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7ac80b5150 con 0x7f7ac815b2a0 2023-10-19T14:18:22.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.867+0000 7f7ac4ff9640 1 -- 172.21.15.57:0/3931644148 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+4924 (secure 0 0 0) 0x7f7ac80b5150 con 0x7f7ac815b2a0 2023-10-19T14:18:22.870 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:18:22.870 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":77,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8480,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6833/2675481384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2675481384},{"type":"v1","addr":"172.21.15.57:6833","nonce":2675481384}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74},{"gid":8486,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/2884298100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2884298100},{"type":"v1","addr":"172.21.15.179:6836","nonce":2884298100}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":73},{"gid":10641,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/505361151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":505361151},{"type":"v1","addr":"172.21.15.57:6835","nonce":505361151}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T14:02:10.024658+0000","modified":"2023-10-19T14:02:12.029940+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":10647},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10647":{"gid":10647,"name":"d","rank":0,"incarnation":75,"state":"up:active","state_seq":3,"addr":"172.21.15.179:6837/3993214977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":3993214977},{"type":"v1","addr":"172.21.15.179:6837","nonce":3993214977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2023-10-19T14:18:22.871 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 77 2023-10-19T14:18:22.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.869+0000 7f7ace475640 1 -- 172.21.15.57:0/3931644148 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7f7aa806af60 msgr2=0x7f7aa806d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T14:18:22.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.869+0000 7f7ace475640 1 --2- 172.21.15.57:0/3931644148 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7f7aa806af60 0x7f7aa806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:22.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.869+0000 7f7ace475640 1 -- 172.21.15.57:0/3931644148 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ac815b2a0 msgr2=0x7f7ac8159e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:22.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.869+0000 7f7ace475640 1 --2- 172.21.15.57:0/3931644148 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ac815b2a0 0x7f7ac8159e80 secure :-1 s=READY pgs=2112 cs=0 l=1 rev1=1 crypto rx=0x7f7ab8036040 tx=0x7f7ab802da10 comp rx=0 tx=0).stop 2023-10-19T14:18:22.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.869+0000 7f7ace475640 1 -- 172.21.15.57:0/3931644148 shutdown_connections 2023-10-19T14:18:22.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.869+0000 7f7ace475640 1 --2- 172.21.15.57:0/3931644148 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7f7aa806af60 0x7f7aa806d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:22.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.869+0000 7f7ace475640 1 --2- 172.21.15.57:0/3931644148 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ac815b2a0 0x7f7ac8159e80 unknown :-1 s=CLOSED pgs=2112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:22.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.869+0000 7f7ace475640 1 --2- 172.21.15.57:0/3931644148 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7ac80cf340 0x7f7ac80ca320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:22.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.869+0000 7f7ace475640 1 --2- 172.21.15.57:0/3931644148 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7ac80ceb40 0x7f7ac80c9a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:22.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.869+0000 7f7ace475640 1 -- 172.21.15.57:0/3931644148 >> 172.21.15.57:0/3931644148 conn(0x7f7ac80d0d60 msgr2=0x7f7ac80af070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:22.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.869+0000 7f7ace475640 1 -- 172.21.15.57:0/3931644148 shutdown_connections 2023-10-19T14:18:22.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.869+0000 7f7ace475640 1 -- 172.21.15.57:0/3931644148 wait complete. 2023-10-19T14:18:22.883 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T14:18:22.886 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:18:22.883+0000 7fbd89e45040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-19T14:18:22.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.993+0000 7fac2050f640 1 -- 172.21.15.57:0/2079023028 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fac1815b2a0 msgr2=0x7fac180d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:22.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.993+0000 7fac2050f640 1 --2- 172.21.15.57:0/2079023028 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fac1815b2a0 0x7fac180d1910 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fac08009830 tx=0x7fac0802bd60 comp rx=0 tx=0).stop 2023-10-19T14:18:22.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.993+0000 7fac2050f640 1 -- 172.21.15.57:0/2079023028 shutdown_connections 2023-10-19T14:18:22.997 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.993+0000 7fac2050f640 1 --2- 172.21.15.57:0/2079023028 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fac1815b2a0 0x7fac180d1910 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:22.997 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.993+0000 7fac2050f640 1 -- 172.21.15.57:0/2079023028 >> 172.21.15.57:0/2079023028 conn(0x7fac180d0d60 msgr2=0x7fac180d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:22.997 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.993+0000 7fac2050f640 1 -- 172.21.15.57:0/2079023028 shutdown_connections 2023-10-19T14:18:22.997 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.993+0000 7fac2050f640 1 -- 172.21.15.57:0/2079023028 wait complete. 2023-10-19T14:18:22.999 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.994+0000 7fac2050f640 1 Processor -- start 2023-10-19T14:18:23.000 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.994+0000 7fac2050f640 1 -- start start 2023-10-19T14:18:23.000 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.995+0000 7fac2050f640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fac180ceb40 0x7fac181517c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:23.000 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.995+0000 7fac2050f640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fac180cf340 0x7fac180ae430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:23.000 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.995+0000 7fac2050f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac1815b2a0 0x7fac180ae970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:23.001 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.995+0000 7fac2050f640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac180c8ae0 con 0x7fac1815b2a0 2023-10-19T14:18:23.001 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.995+0000 7fac2050f640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac180c7ed0 con 0x7fac180cf340 2023-10-19T14:18:23.001 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.995+0000 7fac2050f640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac180c81b0 con 0x7fac180ceb40 2023-10-19T14:18:23.001 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.995+0000 7fac1eaac640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac1815b2a0 0x7fac180ae970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:23.001 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.995+0000 7fac1eaac640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac1815b2a0 0x7fac180ae970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40052/0 (socket says 172.21.15.57:40052) 2023-10-19T14:18:23.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.995+0000 7fac1eaac640 1 -- 172.21.15.57:0/3556270761 learned_addr learned my addr 172.21.15.57:0/3556270761 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:18:23.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.995+0000 7fac1e2ab640 1 --2- 172.21.15.57:0/3556270761 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fac180ceb40 0x7fac181517c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:23.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.995+0000 7fac1daaa640 1 --2- 172.21.15.57:0/3556270761 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fac180cf340 0x7fac180ae430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:23.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.995+0000 7fac1eaac640 1 -- 172.21.15.57:0/3556270761 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fac180ceb40 msgr2=0x7fac181517c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:23.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.995+0000 7fac1eaac640 1 --2- 172.21.15.57:0/3556270761 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fac180ceb40 0x7fac181517c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:23.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.995+0000 7fac1eaac640 1 -- 172.21.15.57:0/3556270761 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fac180cf340 msgr2=0x7fac180ae430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:23.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.995+0000 7fac1eaac640 1 --2- 172.21.15.57:0/3556270761 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fac180cf340 0x7fac180ae430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:23.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.995+0000 7fac1eaac640 1 -- 172.21.15.57:0/3556270761 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac08009490 con 0x7fac1815b2a0 2023-10-19T14:18:23.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.995+0000 7fac1eaac640 1 --2- 172.21.15.57:0/3556270761 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac1815b2a0 0x7fac180ae970 secure :-1 s=READY pgs=2114 cs=0 l=1 rev1=1 crypto rx=0x7fac1000c790 tx=0x7fac1000cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:23.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.995+0000 7fac0effd640 1 -- 172.21.15.57:0/3556270761 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac1000cea0 con 0x7fac1815b2a0 2023-10-19T14:18:23.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.996+0000 7fac2050f640 1 -- 172.21.15.57:0/3556270761 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac180c8490 con 0x7fac1815b2a0 2023-10-19T14:18:23.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.996+0000 7fac0effd640 1 -- 172.21.15.57:0/3556270761 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fac10003b70 con 0x7fac1815b2a0 2023-10-19T14:18:23.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.996+0000 7fac0effd640 1 -- 172.21.15.57:0/3556270761 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac1000a850 con 0x7fac1815b2a0 2023-10-19T14:18:23.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.996+0000 7fac2050f640 1 -- 172.21.15.57:0/3556270761 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac181f11a0 con 0x7fac1815b2a0 2023-10-19T14:18:23.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.997+0000 7fac2050f640 1 -- 172.21.15.57:0/3556270761 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac181f1390 con 0x7fac1815b2a0 2023-10-19T14:18:23.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.997+0000 7fac0effd640 1 -- 172.21.15.57:0/3556270761 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 113) v1 ==== 89343+0+0 (secure 0 0 0) 0x7fac1000a9b0 con 0x7fac1815b2a0 2023-10-19T14:18:23.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.997+0000 7fac0effd640 1 --2- 172.21.15.57:0/3556270761 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7fabfc06af60 0x7fabfc06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:23.006 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.998+0000 7fac1e2ab640 1 -- 172.21.15.57:0/3556270761 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7fabfc06af60 msgr2=0x7fabfc06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/4086068793 2023-10-19T14:18:23.006 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.998+0000 7fac0effd640 1 -- 172.21.15.57:0/3556270761 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 6817+0+0 (secure 0 0 0) 0x7fac10089b70 con 0x7fac1815b2a0 2023-10-19T14:18:23.006 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:22.998+0000 7fac1e2ab640 1 --2- 172.21.15.57:0/3556270761 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7fabfc06af60 0x7fabfc06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T14:18:23.006 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:18:22.999+0000 7fbd89e45040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-19T14:18:23.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.002+0000 7fac0effd640 1 -- 172.21.15.57:0/3556270761 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fac181f1390 con 0x7fac1815b2a0 2023-10-19T14:18:23.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.177+0000 7fac2050f640 1 -- 172.21.15.57:0/3556270761 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fac181f1390 con 0x7fac1815b2a0 2023-10-19T14:18:23.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.179+0000 7fac0effd640 1 -- 172.21.15.57:0/3556270761 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v192) v1 ==== 74+0+17927 (secure 0 0 0) 0x7fac181f1390 con 0x7fac1815b2a0 2023-10-19T14:18:23.182 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:18:23.183 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":192,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T14:18:22.379487+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2023-10-19T14:02:08.318466+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"187","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2023-10-19T14:02:08.995986+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"192","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":33,"snap_epoch":192,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.179:0/1094895803":"2023-10-20T14:13:20.213663+0000","172.21.15.179:6832/2587699336":"2023-10-20T14:13:20.213663+0000","172.21.15.179:0/2295535972":"2023-10-20T14:13:20.213663+0000","172.21.15.179:0/1009979748":"2023-10-20T14:13:20.213663+0000","172.21.15.179:6833/2587699336":"2023-10-20T14:13:20.213663+0000","172.21.15.179:0/2961439032":"2023-10-20T14:13:20.213663+0000","172.21.15.179:0/1362323478":"2023-10-20T14:13:20.213663+0000","172.21.15.179:0/2642945549":"2023-10-20T14:13:20.213663+0000","172.21.15.179:0/1066702535":"2023-10-20T14:02:21.481635+0000","172.21.15.179:0/904747968":"2023-10-20T14:02:21.481635+0000","172.21.15.179:0/1902799271":"2023-10-20T14:02:21.481635+0000","172.21.15.179:0/2939688089":"2023-10-20T14:02:21.481635+0000","172.21.15.179:0/682180483":"2023-10-20T14:02:21.481635+0000","172.21.15.179:6833/1426553632":"2023-10-20T14:02:21.481635+0000","172.21.15.179:6832/1426553632":"2023-10-20T14:02:21.481635+0000","172.21.15.179:0/2321890842":"2023-10-20T14:02:21.481635+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[{"pool":24,"snaps":[{"begin":17,"length":1},{"begin":33,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T14:18:23.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.181+0000 7fac2050f640 1 -- 172.21.15.57:0/3556270761 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7fabfc06af60 msgr2=0x7fabfc06d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T14:18:23.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.181+0000 7fac2050f640 1 --2- 172.21.15.57:0/3556270761 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7fabfc06af60 0x7fabfc06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:23.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.181+0000 7fac2050f640 1 -- 172.21.15.57:0/3556270761 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac1815b2a0 msgr2=0x7fac180ae970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:23.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.181+0000 7fac2050f640 1 --2- 172.21.15.57:0/3556270761 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac1815b2a0 0x7fac180ae970 secure :-1 s=READY pgs=2114 cs=0 l=1 rev1=1 crypto rx=0x7fac1000c790 tx=0x7fac1000cc50 comp rx=0 tx=0).stop 2023-10-19T14:18:23.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.181+0000 7fac2050f640 1 -- 172.21.15.57:0/3556270761 shutdown_connections 2023-10-19T14:18:23.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.181+0000 7fac2050f640 1 --2- 172.21.15.57:0/3556270761 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7fabfc06af60 0x7fabfc06d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:23.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.181+0000 7fac2050f640 1 --2- 172.21.15.57:0/3556270761 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac1815b2a0 0x7fac180ae970 unknown :-1 s=CLOSED pgs=2114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:23.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.181+0000 7fac2050f640 1 --2- 172.21.15.57:0/3556270761 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fac180cf340 0x7fac180ae430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:23.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.181+0000 7fac2050f640 1 --2- 172.21.15.57:0/3556270761 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fac180ceb40 0x7fac181517c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:23.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.181+0000 7fac2050f640 1 -- 172.21.15.57:0/3556270761 >> 172.21.15.57:0/3556270761 conn(0x7fac180d0d60 msgr2=0x7fac1806a3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:23.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.181+0000 7fac2050f640 1 -- 172.21.15.57:0/3556270761 shutdown_connections 2023-10-19T14:18:23.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.181+0000 7fac2050f640 1 -- 172.21.15.57:0/3556270761 wait complete. 2023-10-19T14:18:23.195 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-19T12:44:07.371186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 23, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-19T14:02:08.318466+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '187', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 24, 'pool_name': 'cephfs_data', 'create_time': '2023-10-19T14:02:08.995986+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '192', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 33, 'snap_epoch': 192, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-19T14:18:23.196 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-19T14:18:23.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.307+0000 7fe1b18ec640 1 -- 172.21.15.57:0/3077662483 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe194003b30 msgr2=0x7fe194003f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:23.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.307+0000 7fe1b18ec640 1 --2- 172.21.15.57:0/3077662483 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe194003b30 0x7fe194003f30 secure :-1 s=READY pgs=2116 cs=0 l=1 rev1=1 crypto rx=0x7fe1a0006f60 tx=0x7fe1940057f0 comp rx=0 tx=0).stop 2023-10-19T14:18:23.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.307+0000 7fe1ab7fe640 1 -- 172.21.15.57:0/3077662483 reap_dead start 2023-10-19T14:18:23.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.307+0000 7fe1b18ec640 1 -- 172.21.15.57:0/3077662483 shutdown_connections 2023-10-19T14:18:23.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.307+0000 7fe1b18ec640 1 -- 172.21.15.57:0/3077662483 >> 172.21.15.57:0/3077662483 conn(0x7fe1ac0d0c50 msgr2=0x7fe1ac0d1060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:23.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.307+0000 7fe1b18ec640 1 -- 172.21.15.57:0/3077662483 shutdown_connections 2023-10-19T14:18:23.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.307+0000 7fe1b18ec640 1 -- 172.21.15.57:0/3077662483 wait complete. 2023-10-19T14:18:23.314 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.308+0000 7fe1b18ec640 1 Processor -- start 2023-10-19T14:18:23.314 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.308+0000 7fe1b18ec640 1 -- start start 2023-10-19T14:18:23.314 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.309+0000 7fe1b18ec640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe194003b30 0x7fe1ac0c6ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:23.315 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.309+0000 7fe1b18ec640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe1ac0c86a0 0x7fe1ac0c8b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:23.315 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.309+0000 7fe1b18ec640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe1ac1f19a0 0x7fe1ac1f1e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:23.315 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.309+0000 7fe1b18ec640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1ac0c9060 con 0x7fe194003b30 2023-10-19T14:18:23.315 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.309+0000 7fe1b18ec640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1ac1f2740 con 0x7fe1ac0c86a0 2023-10-19T14:18:23.315 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.309+0000 7fe1b18ec640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1ac1f2d20 con 0x7fe1ac1f19a0 2023-10-19T14:18:23.316 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.309+0000 7fe1aaffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe194003b30 0x7fe1ac0c6ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:23.316 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.309+0000 7fe1aaffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe194003b30 0x7fe1ac0c6ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40080/0 (socket says 172.21.15.57:40080) 2023-10-19T14:18:23.316 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.309+0000 7fe1aaffd640 1 -- 172.21.15.57:0/3309592361 learned_addr learned my addr 172.21.15.57:0/3309592361 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:18:23.316 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.309+0000 7fe1aa7fc640 1 --2- 172.21.15.57:0/3309592361 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe1ac0c86a0 0x7fe1ac0c8b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:23.316 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.309+0000 7fe1ab7fe640 1 --2- 172.21.15.57:0/3309592361 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe1ac1f19a0 0x7fe1ac1f1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:23.317 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.310+0000 7fe1aaffd640 1 -- 172.21.15.57:0/3309592361 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe1ac1f19a0 msgr2=0x7fe1ac1f1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:23.317 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.310+0000 7fe1aaffd640 1 --2- 172.21.15.57:0/3309592361 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe1ac1f19a0 0x7fe1ac1f1e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:23.317 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.310+0000 7fe1aaffd640 1 -- 172.21.15.57:0/3309592361 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe1ac0c86a0 msgr2=0x7fe1ac0c8b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:23.317 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.310+0000 7fe1aaffd640 1 --2- 172.21.15.57:0/3309592361 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe1ac0c86a0 0x7fe1ac0c8b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:23.317 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.310+0000 7fe1aaffd640 1 -- 172.21.15.57:0/3309592361 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1ac1f3000 con 0x7fe194003b30 2023-10-19T14:18:23.317 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.310+0000 7fe1aaffd640 1 --2- 172.21.15.57:0/3309592361 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe194003b30 0x7fe1ac0c6ed0 secure :-1 s=READY pgs=2117 cs=0 l=1 rev1=1 crypto rx=0x7fe1a002c850 tx=0x7fe1a0001720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:23.318 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.310+0000 7fe193fff640 1 -- 172.21.15.57:0/3309592361 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1a0042070 con 0x7fe194003b30 2023-10-19T14:18:23.318 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.310+0000 7fe193fff640 1 -- 172.21.15.57:0/3309592361 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fe1a0004980 con 0x7fe194003b30 2023-10-19T14:18:23.318 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.310+0000 7fe1b18ec640 1 -- 172.21.15.57:0/3309592361 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1ac1f32e0 con 0x7fe194003b30 2023-10-19T14:18:23.318 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.310+0000 7fe193fff640 1 -- 172.21.15.57:0/3309592361 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1a0004d90 con 0x7fe194003b30 2023-10-19T14:18:23.318 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.310+0000 7fe1b18ec640 1 -- 172.21.15.57:0/3309592361 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1ac1f2390 con 0x7fe194003b30 2023-10-19T14:18:23.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.311+0000 7fe1b18ec640 1 -- 172.21.15.57:0/3309592361 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1ac0d0440 con 0x7fe194003b30 2023-10-19T14:18:23.320 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.312+0000 7fe193fff640 1 -- 172.21.15.57:0/3309592361 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 113) v1 ==== 89343+0+0 (secure 0 0 0) 0x7fe1a0032070 con 0x7fe194003b30 2023-10-19T14:18:23.320 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.312+0000 7fe193fff640 1 --2- 172.21.15.57:0/3309592361 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7fe19806af60 0x7fe19806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:23.320 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.313+0000 7fe1aa7fc640 1 -- 172.21.15.57:0/3309592361 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7fe19806af60 msgr2=0x7fe19806d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/4086068793 2023-10-19T14:18:23.320 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.313+0000 7fe193fff640 1 -- 172.21.15.57:0/3309592361 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 6817+0+0 (secure 0 0 0) 0x7fe1a00ab230 con 0x7fe194003b30 2023-10-19T14:18:23.320 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.313+0000 7fe1aa7fc640 1 --2- 172.21.15.57:0/3309592361 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7fe19806af60 0x7fe19806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T14:18:23.321 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.317+0000 7fe193fff640 1 -- 172.21.15.57:0/3309592361 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe1a0075e40 con 0x7fe194003b30 2023-10-19T14:18:23.434 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:18:23.432+0000 7f90fd439040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-19T14:18:23.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.494+0000 7fe1b18ec640 1 -- 172.21.15.57:0/3309592361 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe1ac0690e0 con 0x7fe194003b30 2023-10-19T14:18:23.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.495+0000 7fe193fff640 1 -- 172.21.15.57:0/3309592361 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v192) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe1a0079460 con 0x7fe194003b30 2023-10-19T14:18:23.498 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-19T14:18:23.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.497+0000 7fe1b18ec640 1 -- 172.21.15.57:0/3309592361 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7fe19806af60 msgr2=0x7fe19806d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T14:18:23.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.497+0000 7fe1b18ec640 1 --2- 172.21.15.57:0/3309592361 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7fe19806af60 0x7fe19806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:23.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.497+0000 7fe1b18ec640 1 -- 172.21.15.57:0/3309592361 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe194003b30 msgr2=0x7fe1ac0c6ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:23.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.497+0000 7fe1b18ec640 1 --2- 172.21.15.57:0/3309592361 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe194003b30 0x7fe1ac0c6ed0 secure :-1 s=READY pgs=2117 cs=0 l=1 rev1=1 crypto rx=0x7fe1a002c850 tx=0x7fe1a0001720 comp rx=0 tx=0).stop 2023-10-19T14:18:23.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.497+0000 7fe1b18ec640 1 -- 172.21.15.57:0/3309592361 shutdown_connections 2023-10-19T14:18:23.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.497+0000 7fe1b18ec640 1 --2- 172.21.15.57:0/3309592361 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7fe19806af60 0x7fe19806d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:23.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.497+0000 7fe1b18ec640 1 --2- 172.21.15.57:0/3309592361 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe1ac1f19a0 0x7fe1ac1f1e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:23.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.497+0000 7fe1b18ec640 1 --2- 172.21.15.57:0/3309592361 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe1ac0c86a0 0x7fe1ac0c8b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:23.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.497+0000 7fe1b18ec640 1 --2- 172.21.15.57:0/3309592361 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe194003b30 0x7fe1ac0c6ed0 unknown :-1 s=CLOSED pgs=2117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:23.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.497+0000 7fe1b18ec640 1 -- 172.21.15.57:0/3309592361 >> 172.21.15.57:0/3309592361 conn(0x7fe1ac0d0c50 msgr2=0x7fe1ac0cf1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:23.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.497+0000 7fe1b18ec640 1 -- 172.21.15.57:0/3309592361 shutdown_connections 2023-10-19T14:18:23.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.497+0000 7fe1b18ec640 1 -- 172.21.15.57:0/3309592361 wait complete. 2023-10-19T14:18:23.510 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-19T14:18:23.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.620+0000 7f6822cc2640 1 -- 172.21.15.57:0/3106654468 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f681c0cf340 msgr2=0x7f681c0d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:23.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.620+0000 7f6822cc2640 1 --2- 172.21.15.57:0/3106654468 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f681c0cf340 0x7f681c0d1910 secure :-1 s=READY pgs=2119 cs=0 l=1 rev1=1 crypto rx=0x7f6810009830 tx=0x7f681002bd60 comp rx=0 tx=0).stop 2023-10-19T14:18:23.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.620+0000 7f6822cc2640 1 -- 172.21.15.57:0/3106654468 shutdown_connections 2023-10-19T14:18:23.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.620+0000 7f6822cc2640 1 --2- 172.21.15.57:0/3106654468 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f681c0cf340 0x7f681c0d1910 unknown :-1 s=CLOSED pgs=2119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:23.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.620+0000 7f6822cc2640 1 -- 172.21.15.57:0/3106654468 >> 172.21.15.57:0/3106654468 conn(0x7f681c0d0d60 msgr2=0x7f681c0d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:23.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.620+0000 7f6822cc2640 1 -- 172.21.15.57:0/3106654468 shutdown_connections 2023-10-19T14:18:23.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.620+0000 7f6822cc2640 1 -- 172.21.15.57:0/3106654468 wait complete. 2023-10-19T14:18:23.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.621+0000 7f6822cc2640 1 Processor -- start 2023-10-19T14:18:23.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.621+0000 7f6822cc2640 1 -- start start 2023-10-19T14:18:23.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.622+0000 7f6822cc2640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f681c0ceb40 0x7f681c158570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:23.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.622+0000 7f6822cc2640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f681c0cf340 0x7f681c0ca7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:23.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.622+0000 7f6822cc2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f681c15b2a0 0x7f681c0b4b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:23.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.622+0000 7f6822cc2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f681c0b3a80 con 0x7f681c15b2a0 2023-10-19T14:18:23.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.622+0000 7f6822cc2640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f681c0b3d60 con 0x7f681c0ceb40 2023-10-19T14:18:23.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.622+0000 7f6822cc2640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f681c0b4040 con 0x7f681c0cf340 2023-10-19T14:18:23.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.622+0000 7f682125f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f681c15b2a0 0x7f681c0b4b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:23.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.622+0000 7f682125f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f681c15b2a0 0x7f681c0b4b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40108/0 (socket says 172.21.15.57:40108) 2023-10-19T14:18:23.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.622+0000 7f682125f640 1 -- 172.21.15.57:0/1568884769 learned_addr learned my addr 172.21.15.57:0/1568884769 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:18:23.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.622+0000 7f6820a5e640 1 --2- 172.21.15.57:0/1568884769 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f681c0ceb40 0x7f681c158570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:23.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.622+0000 7f681bfff640 1 --2- 172.21.15.57:0/1568884769 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f681c0cf340 0x7f681c0ca7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:23.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.623+0000 7f682125f640 1 -- 172.21.15.57:0/1568884769 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f681c0cf340 msgr2=0x7f681c0ca7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:23.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.623+0000 7f682125f640 1 --2- 172.21.15.57:0/1568884769 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f681c0cf340 0x7f681c0ca7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:23.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.623+0000 7f682125f640 1 -- 172.21.15.57:0/1568884769 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f681c0ceb40 msgr2=0x7f681c158570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:23.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.623+0000 7f682125f640 1 --2- 172.21.15.57:0/1568884769 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f681c0ceb40 0x7f681c158570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:23.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.623+0000 7f682125f640 1 -- 172.21.15.57:0/1568884769 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6810009490 con 0x7f681c15b2a0 2023-10-19T14:18:23.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.623+0000 7f682125f640 1 --2- 172.21.15.57:0/1568884769 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f681c15b2a0 0x7f681c0b4b20 secure :-1 s=READY pgs=2120 cs=0 l=1 rev1=1 crypto rx=0x7f681400c7a0 tx=0x7f681400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:23.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.623+0000 7f68197fa640 1 -- 172.21.15.57:0/1568884769 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6814003b70 con 0x7f681c15b2a0 2023-10-19T14:18:23.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.623+0000 7f68197fa640 1 -- 172.21.15.57:0/1568884769 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f6814003cf0 con 0x7f681c15b2a0 2023-10-19T14:18:23.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.623+0000 7f68197fa640 1 -- 172.21.15.57:0/1568884769 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f681400aab0 con 0x7f681c15b2a0 2023-10-19T14:18:23.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.623+0000 7f6822cc2640 1 -- 172.21.15.57:0/1568884769 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f681c0b4320 con 0x7f681c15b2a0 2023-10-19T14:18:23.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.623+0000 7f6822cc2640 1 -- 172.21.15.57:0/1568884769 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f681c158ab0 con 0x7f681c15b2a0 2023-10-19T14:18:23.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.624+0000 7f6822cc2640 1 -- 172.21.15.57:0/1568884769 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f681c0b4840 con 0x7f681c15b2a0 2023-10-19T14:18:23.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.624+0000 7f68197fa640 1 -- 172.21.15.57:0/1568884769 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 113) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f681400ac10 con 0x7f681c15b2a0 2023-10-19T14:18:23.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.625+0000 7f68197fa640 1 --2- 172.21.15.57:0/1568884769 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7f680006af60 0x7f680006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:23.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.625+0000 7f6820a5e640 1 -- 172.21.15.57:0/1568884769 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7f680006af60 msgr2=0x7f680006d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/4086068793 2023-10-19T14:18:23.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.625+0000 7f68197fa640 1 -- 172.21.15.57:0/1568884769 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 6817+0+0 (secure 0 0 0) 0x7f6814089a70 con 0x7f681c15b2a0 2023-10-19T14:18:23.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.625+0000 7f6820a5e640 1 --2- 172.21.15.57:0/1568884769 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7f680006af60 0x7f680006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T14:18:23.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.629+0000 7f68197fa640 1 -- 172.21.15.57:0/1568884769 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f681c0b4840 con 0x7f681c15b2a0 2023-10-19T14:18:23.671 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:18:23.668+0000 7fbd89e45040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-19T14:18:23.694 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:18:23.692+0000 7f90fd439040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-19T14:18:23.744 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:18:23.741+0000 7f90fd439040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-19T14:18:23.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.807+0000 7f6822cc2640 1 -- 172.21.15.57:0/1568884769 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f681c0b4840 con 0x7f681c15b2a0 2023-10-19T14:18:23.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.807+0000 7f68197fa640 1 -- 172.21.15.57:0/1568884769 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v192) v1 ==== 104+0+11 (secure 0 0 0) 0x7f681c0b4840 con 0x7f681c15b2a0 2023-10-19T14:18:23.811 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2023-10-19T14:18:23.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.809+0000 7f6822cc2640 1 -- 172.21.15.57:0/1568884769 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7f680006af60 msgr2=0x7f680006d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T14:18:23.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.809+0000 7f6822cc2640 1 --2- 172.21.15.57:0/1568884769 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7f680006af60 0x7f680006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:23.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.809+0000 7f6822cc2640 1 -- 172.21.15.57:0/1568884769 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f681c15b2a0 msgr2=0x7f681c0b4b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:23.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.809+0000 7f6822cc2640 1 --2- 172.21.15.57:0/1568884769 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f681c15b2a0 0x7f681c0b4b20 secure :-1 s=READY pgs=2120 cs=0 l=1 rev1=1 crypto rx=0x7f681400c7a0 tx=0x7f681400cc60 comp rx=0 tx=0).stop 2023-10-19T14:18:23.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.809+0000 7f6822cc2640 1 -- 172.21.15.57:0/1568884769 shutdown_connections 2023-10-19T14:18:23.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.809+0000 7f6822cc2640 1 --2- 172.21.15.57:0/1568884769 >> [v2:172.21.15.179:6832/4086068793,v1:172.21.15.179:6833/4086068793] conn(0x7f680006af60 0x7f680006d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:23.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.809+0000 7f6822cc2640 1 --2- 172.21.15.57:0/1568884769 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f681c15b2a0 0x7f681c0b4b20 unknown :-1 s=CLOSED pgs=2120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:23.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.809+0000 7f6822cc2640 1 --2- 172.21.15.57:0/1568884769 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f681c0cf340 0x7f681c0ca7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:23.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.809+0000 7f6822cc2640 1 --2- 172.21.15.57:0/1568884769 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f681c0ceb40 0x7f681c158570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:23.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.809+0000 7f6822cc2640 1 -- 172.21.15.57:0/1568884769 >> 172.21.15.57:0/1568884769 conn(0x7f681c0d0d60 msgr2=0x7f681c159580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:23.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.809+0000 7f6822cc2640 1 -- 172.21.15.57:0/1568884769 shutdown_connections 2023-10-19T14:18:23.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.809+0000 7f6822cc2640 1 -- 172.21.15.57:0/1568884769 wait complete. 2023-10-19T14:18:23.823 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-19T14:18:23.844 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:18:23.842+0000 7f90fd439040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-19T14:18:23.922 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:18:23.919+0000 7f90826ec640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:18:23.923 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:18:23.919+0000 7f90826ec640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:18:23.923 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:18:23.919+0000 7f90826ec640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:18:23.923 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:18:23.919+0000 7f90826ec640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:18:23.923 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:18:23.919+0000 7f90826ec640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:18:23.924 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:18:23.920+0000 7f907f6e6640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:18:23.924 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:18:23.920+0000 7f907f6e6640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:18:23.924 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:18:23.920+0000 7f907f6e6640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:18:23.924 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:18:23.920+0000 7f907f6e6640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:18:23.924 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:18:23.920+0000 7f907f6e6640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:18:23.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.939+0000 7fd1884f2640 1 -- 172.21.15.57:0/931879701 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd18015b2a0 msgr2=0x7fd1800d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:23.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.939+0000 7fd1884f2640 1 --2- 172.21.15.57:0/931879701 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd18015b2a0 0x7fd1800d1910 secure :-1 s=READY pgs=2122 cs=0 l=1 rev1=1 crypto rx=0x7fd17c009830 tx=0x7fd17c02bcd0 comp rx=0 tx=0).stop 2023-10-19T14:18:23.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.939+0000 7fd1884f2640 1 -- 172.21.15.57:0/931879701 shutdown_connections 2023-10-19T14:18:23.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.939+0000 7fd1884f2640 1 --2- 172.21.15.57:0/931879701 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd18015b2a0 0x7fd1800d1910 unknown :-1 s=CLOSED pgs=2122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:23.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.939+0000 7fd1884f2640 1 -- 172.21.15.57:0/931879701 >> 172.21.15.57:0/931879701 conn(0x7fd1800d0d60 msgr2=0x7fd1800d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:23.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.939+0000 7fd1884f2640 1 -- 172.21.15.57:0/931879701 shutdown_connections 2023-10-19T14:18:23.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.939+0000 7fd1884f2640 1 -- 172.21.15.57:0/931879701 wait complete. 2023-10-19T14:18:23.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.941+0000 7fd1884f2640 1 Processor -- start 2023-10-19T14:18:23.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.941+0000 7fd1884f2640 1 -- start start 2023-10-19T14:18:23.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.941+0000 7fd1884f2640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd1800ceb40 0x7fd1801573d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:23.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.941+0000 7fd1884f2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd1800cf340 0x7fd1800b5860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:23.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.941+0000 7fd1884f2640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd18015b2a0 0x7fd1800ae460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:23.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.941+0000 7fd1884f2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1800b6340 con 0x7fd1800cf340 2023-10-19T14:18:23.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.941+0000 7fd1884f2640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1800b6620 con 0x7fd1800ceb40 2023-10-19T14:18:23.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.941+0000 7fd1884f2640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd18015b770 con 0x7fd18015b2a0 2023-10-19T14:18:23.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.942+0000 7fd185a8d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd1800cf340 0x7fd1800b5860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:23.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.942+0000 7fd185a8d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd1800cf340 0x7fd1800b5860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40130/0 (socket says 172.21.15.57:40130) 2023-10-19T14:18:23.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.942+0000 7fd185a8d640 1 -- 172.21.15.57:0/450768817 learned_addr learned my addr 172.21.15.57:0/450768817 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:18:23.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.942+0000 7fd18628e640 1 --2- 172.21.15.57:0/450768817 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd1800ceb40 0x7fd1801573d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:23.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.942+0000 7fd186a8f640 1 --2- 172.21.15.57:0/450768817 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd18015b2a0 0x7fd1800ae460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:23.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.942+0000 7fd185a8d640 1 -- 172.21.15.57:0/450768817 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd18015b2a0 msgr2=0x7fd1800ae460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:23.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.942+0000 7fd185a8d640 1 --2- 172.21.15.57:0/450768817 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd18015b2a0 0x7fd1800ae460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:23.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.942+0000 7fd185a8d640 1 -- 172.21.15.57:0/450768817 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd1800ceb40 msgr2=0x7fd1801573d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:23.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.942+0000 7fd185a8d640 1 --2- 172.21.15.57:0/450768817 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd1800ceb40 0x7fd1801573d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:23.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.942+0000 7fd185a8d640 1 -- 172.21.15.57:0/450768817 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd17c009490 con 0x7fd1800cf340 2023-10-19T14:18:23.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.942+0000 7fd185a8d640 1 --2- 172.21.15.57:0/450768817 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd1800cf340 0x7fd1800b5860 secure :-1 s=READY pgs=2123 cs=0 l=1 rev1=1 crypto rx=0x7fd17c009780 tx=0x7fd17c0038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:23.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.942+0000 7fd176ffd640 1 -- 172.21.15.57:0/450768817 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd17c003ea0 con 0x7fd1800cf340 2023-10-19T14:18:23.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.942+0000 7fd176ffd640 1 -- 172.21.15.57:0/450768817 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fd17c034550 con 0x7fd1800cf340 2023-10-19T14:18:23.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.942+0000 7fd1884f2640 1 -- 172.21.15.57:0/450768817 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd18015ba50 con 0x7fd1800cf340 2023-10-19T14:18:23.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.942+0000 7fd176ffd640 1 -- 172.21.15.57:0/450768817 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd17c034960 con 0x7fd1800cf340 2023-10-19T14:18:23.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.942+0000 7fd1884f2640 1 -- 172.21.15.57:0/450768817 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1800b6900 con 0x7fd1800cf340 2023-10-19T14:18:23.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.943+0000 7fd1884f2640 1 -- 172.21.15.57:0/450768817 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1800690e0 con 0x7fd1800cf340 2023-10-19T14:18:23.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.944+0000 7fd176ffd640 1 -- 172.21.15.57:0/450768817 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 114) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fd17c03b040 con 0x7fd1800cf340 2023-10-19T14:18:23.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.944+0000 7fd176ffd640 1 -- 172.21.15.57:0/450768817 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 7117+0+0 (secure 0 0 0) 0x7fd17c0abb70 con 0x7fd1800cf340 2023-10-19T14:18:23.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:23.948+0000 7fd176ffd640 1 -- 172.21.15.57:0/450768817 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd17c076b20 con 0x7fd1800cf340 2023-10-19T14:18:23.960 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:18:23.957+0000 7fbd89e45040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-19T14:18:24.012 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:18:24.010+0000 7fbd89e45040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-19T14:18:24.118 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:18:24.116+0000 7fbd89e45040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-19T14:18:24.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.123+0000 7fd1884f2640 1 -- 172.21.15.57:0/450768817 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd1800d0b40 con 0x7fd1800cf340 2023-10-19T14:18:24.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.124+0000 7fd176ffd640 1 -- 172.21.15.57:0/450768817 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v193) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd17c07a140 con 0x7fd1800cf340 2023-10-19T14:18:24.127 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 64 2023-10-19T14:18:24.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.126+0000 7fd1884f2640 1 -- 172.21.15.57:0/450768817 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd1800cf340 msgr2=0x7fd1800b5860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:24.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.126+0000 7fd1884f2640 1 --2- 172.21.15.57:0/450768817 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd1800cf340 0x7fd1800b5860 secure :-1 s=READY pgs=2123 cs=0 l=1 rev1=1 crypto rx=0x7fd17c009780 tx=0x7fd17c0038a0 comp rx=0 tx=0).stop 2023-10-19T14:18:24.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.126+0000 7fd1884f2640 1 -- 172.21.15.57:0/450768817 shutdown_connections 2023-10-19T14:18:24.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.126+0000 7fd1884f2640 1 --2- 172.21.15.57:0/450768817 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd18015b2a0 0x7fd1800ae460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:24.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.126+0000 7fd1884f2640 1 --2- 172.21.15.57:0/450768817 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd1800cf340 0x7fd1800b5860 unknown :-1 s=CLOSED pgs=2123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:24.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.126+0000 7fd1884f2640 1 --2- 172.21.15.57:0/450768817 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd1800ceb40 0x7fd1801573d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:24.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.126+0000 7fd1884f2640 1 -- 172.21.15.57:0/450768817 >> 172.21.15.57:0/450768817 conn(0x7fd1800d0d60 msgr2=0x7fd1800b0630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:24.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.126+0000 7fd1884f2640 1 -- 172.21.15.57:0/450768817 shutdown_connections 2023-10-19T14:18:24.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.126+0000 7fd1884f2640 1 -- 172.21.15.57:0/450768817 wait complete. 2023-10-19T14:18:24.140 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T14:18:24.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.246+0000 7f9878965640 1 -- 172.21.15.57:0/4200127627 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9874069720 msgr2=0x7f9874069bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:24.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.246+0000 7f9878965640 1 --2- 172.21.15.57:0/4200127627 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9874069720 0x7f9874069bb0 secure :-1 s=READY pgs=2124 cs=0 l=1 rev1=1 crypto rx=0x7f9874061510 tx=0x7f986c02c1d0 comp rx=0 tx=0).stop 2023-10-19T14:18:24.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.246+0000 7f9878965640 1 -- 172.21.15.57:0/4200127627 shutdown_connections 2023-10-19T14:18:24.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.246+0000 7f9878965640 1 --2- 172.21.15.57:0/4200127627 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9874069720 0x7f9874069bb0 unknown :-1 s=CLOSED pgs=2124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:24.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.246+0000 7f9878965640 1 --2- 172.21.15.57:0/4200127627 >> v2:172.21.15.179:3300/0 conn(0x7f98740cd6d0 0x7f98740cdb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:24.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.246+0000 7f9878965640 1 --2- 172.21.15.57:0/4200127627 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9874142850 0x7f9874159440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:24.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.246+0000 7f9878965640 1 -- 172.21.15.57:0/4200127627 >> 172.21.15.57:0/4200127627 conn(0x7f98740af7a0 msgr2=0x7f98740afb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:24.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.247+0000 7f9878965640 1 -- 172.21.15.57:0/4200127627 shutdown_connections 2023-10-19T14:18:24.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.247+0000 7f9878965640 1 -- 172.21.15.57:0/4200127627 wait complete. 2023-10-19T14:18:24.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.248+0000 7f9878965640 1 Processor -- start 2023-10-19T14:18:24.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.248+0000 7f9878965640 1 -- start start 2023-10-19T14:18:24.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.249+0000 7f9878965640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9874069720 0x7f98741f0c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:24.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.249+0000 7f9878965640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f98740cd6d0 0x7f9874137ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:24.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.249+0000 7f9878965640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9874142850 0x7f98740abbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:24.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.249+0000 7f9878965640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98740aa8c0 con 0x7f9874142850 2023-10-19T14:18:24.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.249+0000 7f9878965640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9874158f50 con 0x7f9874069720 2023-10-19T14:18:24.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.249+0000 7f9878965640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98740aab10 con 0x7f98740cd6d0 2023-10-19T14:18:24.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.249+0000 7f9872d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9874142850 0x7f98740abbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:24.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.249+0000 7f9872d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9874142850 0x7f98740abbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40146/0 (socket says 172.21.15.57:40146) 2023-10-19T14:18:24.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.249+0000 7f9872d9d640 1 -- 172.21.15.57:0/1797817589 learned_addr learned my addr 172.21.15.57:0/1797817589 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:18:24.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.249+0000 7f986bfff640 1 --2- 172.21.15.57:0/1797817589 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f98740cd6d0 0x7f9874137ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:24.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.249+0000 7f987259c640 1 --2- 172.21.15.57:0/1797817589 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9874069720 0x7f98741f0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:24.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.249+0000 7f9872d9d640 1 -- 172.21.15.57:0/1797817589 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f98740cd6d0 msgr2=0x7f9874137ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:24.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.249+0000 7f9872d9d640 1 --2- 172.21.15.57:0/1797817589 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f98740cd6d0 0x7f9874137ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:24.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.249+0000 7f9872d9d640 1 -- 172.21.15.57:0/1797817589 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9874069720 msgr2=0x7f98741f0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:24.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.249+0000 7f9872d9d640 1 --2- 172.21.15.57:0/1797817589 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9874069720 0x7f98741f0c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:24.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.249+0000 7f9872d9d640 1 -- 172.21.15.57:0/1797817589 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f986c009900 con 0x7f9874142850 2023-10-19T14:18:24.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.249+0000 7f9872d9d640 1 --2- 172.21.15.57:0/1797817589 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9874142850 0x7f98740abbb0 secure :-1 s=READY pgs=2125 cs=0 l=1 rev1=1 crypto rx=0x7f986c037040 tx=0x7f986c035db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:24.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.249+0000 7f986affd640 1 -- 172.21.15.57:0/1797817589 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f986c034440 con 0x7f9874142850 2023-10-19T14:18:24.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.249+0000 7f986affd640 1 -- 172.21.15.57:0/1797817589 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f986c004540 con 0x7f9874142850 2023-10-19T14:18:24.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.249+0000 7f986affd640 1 -- 172.21.15.57:0/1797817589 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f986c003e60 con 0x7f9874142850 2023-10-19T14:18:24.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.249+0000 7f9878965640 1 -- 172.21.15.57:0/1797817589 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98740aadf0 con 0x7f9874142850 2023-10-19T14:18:24.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.249+0000 7f9878965640 1 -- 172.21.15.57:0/1797817589 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98740ab3b0 con 0x7f9874142850 2023-10-19T14:18:24.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.250+0000 7f9878965640 1 -- 172.21.15.57:0/1797817589 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98740690e0 con 0x7f9874142850 2023-10-19T14:18:24.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.251+0000 7f986affd640 1 -- 172.21.15.57:0/1797817589 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 114) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f986c03b040 con 0x7f9874142850 2023-10-19T14:18:24.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.251+0000 7f986affd640 1 -- 172.21.15.57:0/1797817589 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 7117+0+0 (secure 0 0 0) 0x7f986c0ac2a0 con 0x7f9874142850 2023-10-19T14:18:24.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.256+0000 7f986affd640 1 -- 172.21.15.57:0/1797817589 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f986c0771e0 con 0x7f9874142850 2023-10-19T14:18:24.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.470+0000 7f9878965640 1 -- 172.21.15.57:0/1797817589 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f98740af550 con 0x7f9874142850 2023-10-19T14:18:24.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.470+0000 7f986affd640 1 -- 172.21.15.57:0/1797817589 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+4924 (secure 0 0 0) 0x7f986c07a800 con 0x7f9874142850 2023-10-19T14:18:24.474 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:18:24.474 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":77,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8480,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6833/2675481384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2675481384},{"type":"v1","addr":"172.21.15.57:6833","nonce":2675481384}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74},{"gid":8486,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/2884298100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2884298100},{"type":"v1","addr":"172.21.15.179:6836","nonce":2884298100}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":73},{"gid":10641,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/505361151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":505361151},{"type":"v1","addr":"172.21.15.57:6835","nonce":505361151}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T14:02:10.024658+0000","modified":"2023-10-19T14:02:12.029940+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":10647},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10647":{"gid":10647,"name":"d","rank":0,"incarnation":75,"state":"up:active","state_seq":3,"addr":"172.21.15.179:6837/3993214977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":3993214977},{"type":"v1","addr":"172.21.15.179:6837","nonce":3993214977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2023-10-19T14:18:24.474 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 77 2023-10-19T14:18:24.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.472+0000 7f9878965640 1 -- 172.21.15.57:0/1797817589 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9874142850 msgr2=0x7f98740abbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:24.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.472+0000 7f9878965640 1 --2- 172.21.15.57:0/1797817589 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9874142850 0x7f98740abbb0 secure :-1 s=READY pgs=2125 cs=0 l=1 rev1=1 crypto rx=0x7f986c037040 tx=0x7f986c035db0 comp rx=0 tx=0).stop 2023-10-19T14:18:24.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.472+0000 7f9878965640 1 -- 172.21.15.57:0/1797817589 shutdown_connections 2023-10-19T14:18:24.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.472+0000 7f9878965640 1 --2- 172.21.15.57:0/1797817589 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9874142850 0x7f98740abbb0 unknown :-1 s=CLOSED pgs=2125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:24.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.472+0000 7f9878965640 1 --2- 172.21.15.57:0/1797817589 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f98740cd6d0 0x7f9874137ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:24.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.472+0000 7f9878965640 1 --2- 172.21.15.57:0/1797817589 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9874069720 0x7f98741f0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:24.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.472+0000 7f9878965640 1 -- 172.21.15.57:0/1797817589 >> 172.21.15.57:0/1797817589 conn(0x7f98740af7a0 msgr2=0x7f98740b2300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:24.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.472+0000 7f9878965640 1 -- 172.21.15.57:0/1797817589 shutdown_connections 2023-10-19T14:18:24.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.472+0000 7f9878965640 1 -- 172.21.15.57:0/1797817589 wait complete. 2023-10-19T14:18:24.486 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T14:18:24.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.592+0000 7f3fe0ba5640 1 -- 172.21.15.57:0/2718477244 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3fc80047c0 msgr2=0x7f3fc8004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:24.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.592+0000 7f3fe0ba5640 1 --2- 172.21.15.57:0/2718477244 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3fc80047c0 0x7f3fc8004c40 secure :-1 s=READY pgs=2127 cs=0 l=1 rev1=1 crypto rx=0x7f3fc4006fa0 tx=0x7f3fc80057f0 comp rx=0 tx=0).stop 2023-10-19T14:18:24.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.592+0000 7f3fe0ba5640 1 -- 172.21.15.57:0/2718477244 shutdown_connections 2023-10-19T14:18:24.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.592+0000 7f3fe0ba5640 1 --2- 172.21.15.57:0/2718477244 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3fc80047c0 0x7f3fc8004c40 unknown :-1 s=CLOSED pgs=2127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:24.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.592+0000 7f3fe0ba5640 1 -- 172.21.15.57:0/2718477244 >> 172.21.15.57:0/2718477244 conn(0x7f3fdc069930 msgr2=0x7f3fdc069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:24.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.592+0000 7f3fe0ba5640 1 -- 172.21.15.57:0/2718477244 shutdown_connections 2023-10-19T14:18:24.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.592+0000 7f3fe0ba5640 1 -- 172.21.15.57:0/2718477244 wait complete. 2023-10-19T14:18:24.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.594+0000 7f3fe0ba5640 1 Processor -- start 2023-10-19T14:18:24.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.594+0000 7f3fe0ba5640 1 -- start start 2023-10-19T14:18:24.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.594+0000 7f3fe0ba5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3fc80047c0 0x7f3fdc0adb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:24.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.594+0000 7f3fe0ba5640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3fdc13ada0 0x7f3fdc13b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:24.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.594+0000 7f3fe0ba5640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3fdc0ab720 0x7f3fdc0abba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:24.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.594+0000 7f3fe0ba5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fdc13b760 con 0x7f3fc80047c0 2023-10-19T14:18:24.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.594+0000 7f3fe0ba5640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fdc0ac350 con 0x7f3fdc0ab720 2023-10-19T14:18:24.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.594+0000 7f3fe0ba5640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fdc0ac0e0 con 0x7f3fdc13ada0 2023-10-19T14:18:24.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.594+0000 7f3fda59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3fc80047c0 0x7f3fdc0adb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:24.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.595+0000 7f3fda59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3fc80047c0 0x7f3fdc0adb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40160/0 (socket says 172.21.15.57:40160) 2023-10-19T14:18:24.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.595+0000 7f3fda59c640 1 -- 172.21.15.57:0/2837417145 learned_addr learned my addr 172.21.15.57:0/2837417145 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:18:24.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.595+0000 7f3fd3fff640 1 --2- 172.21.15.57:0/2837417145 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3fdc13ada0 0x7f3fdc13b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:24.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.595+0000 7f3fdad9d640 1 --2- 172.21.15.57:0/2837417145 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3fdc0ab720 0x7f3fdc0abba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:24.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.595+0000 7f3fda59c640 1 -- 172.21.15.57:0/2837417145 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3fdc13ada0 msgr2=0x7f3fdc13b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:24.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.595+0000 7f3fda59c640 1 --2- 172.21.15.57:0/2837417145 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3fdc13ada0 0x7f3fdc13b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:24.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.595+0000 7f3fda59c640 1 -- 172.21.15.57:0/2837417145 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3fdc0ab720 msgr2=0x7f3fdc0abba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:24.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.595+0000 7f3fda59c640 1 --2- 172.21.15.57:0/2837417145 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3fdc0ab720 0x7f3fdc0abba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:24.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.595+0000 7f3fda59c640 1 -- 172.21.15.57:0/2837417145 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fdc0a8fa0 con 0x7f3fc80047c0 2023-10-19T14:18:24.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.595+0000 7f3fda59c640 1 --2- 172.21.15.57:0/2837417145 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3fc80047c0 0x7f3fdc0adb20 secure :-1 s=READY pgs=2128 cs=0 l=1 rev1=1 crypto rx=0x7f3fcc003ec0 tx=0x7f3fcc001630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:24.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.595+0000 7f3fd2ffd640 1 -- 172.21.15.57:0/2837417145 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fcc006030 con 0x7f3fc80047c0 2023-10-19T14:18:24.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.595+0000 7f3fd2ffd640 1 -- 172.21.15.57:0/2837417145 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f3fcc011640 con 0x7f3fc80047c0 2023-10-19T14:18:24.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.595+0000 7f3fe0ba5640 1 -- 172.21.15.57:0/2837417145 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fdc0a65e0 con 0x7f3fc80047c0 2023-10-19T14:18:24.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.595+0000 7f3fd2ffd640 1 -- 172.21.15.57:0/2837417145 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fcc011a10 con 0x7f3fc80047c0 2023-10-19T14:18:24.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.595+0000 7f3fe0ba5640 1 -- 172.21.15.57:0/2837417145 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fdc0a6ba0 con 0x7f3fc80047c0 2023-10-19T14:18:24.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.596+0000 7f3fe0ba5640 1 -- 172.21.15.57:0/2837417145 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fdc0690e0 con 0x7f3fc80047c0 2023-10-19T14:18:24.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.597+0000 7f3fd2ffd640 1 -- 172.21.15.57:0/2837417145 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 114) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f3fcc011b70 con 0x7f3fc80047c0 2023-10-19T14:18:24.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.597+0000 7f3fd2ffd640 1 -- 172.21.15.57:0/2837417145 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 7117+0+0 (secure 0 0 0) 0x7f3fcc08a210 con 0x7f3fc80047c0 2023-10-19T14:18:24.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.601+0000 7f3fd2ffd640 1 -- 172.21.15.57:0/2837417145 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3fcc055240 con 0x7f3fc80047c0 2023-10-19T14:18:24.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.776+0000 7f3fe0ba5640 1 -- 172.21.15.57:0/2837417145 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3fdc0b0980 con 0x7f3fc80047c0 2023-10-19T14:18:24.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.778+0000 7f3fd2ffd640 1 -- 172.21.15.57:0/2837417145 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v193) v1 ==== 74+0+18367 (secure 0 0 0) 0x7f3fcc015070 con 0x7f3fc80047c0 2023-10-19T14:18:24.781 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:18:24.781 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":193,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T14:18:23.856506+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2023-10-19T14:02:08.318466+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"187","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2023-10-19T14:02:08.995986+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"192","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":33,"snap_epoch":192,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.179:0/1944205818":"2023-10-20T14:18:23.856456+0000","172.21.15.179:0/530749640":"2023-10-20T14:18:23.856456+0000","172.21.15.179:0/3372019790":"2023-10-20T14:18:23.856456+0000","172.21.15.179:6832/4086068793":"2023-10-20T14:18:23.856456+0000","172.21.15.179:0/4245102210":"2023-10-20T14:18:23.856456+0000","172.21.15.179:0/1094895803":"2023-10-20T14:13:20.213663+0000","172.21.15.179:6832/2587699336":"2023-10-20T14:13:20.213663+0000","172.21.15.179:0/2295535972":"2023-10-20T14:13:20.213663+0000","172.21.15.179:6833/4086068793":"2023-10-20T14:18:23.856456+0000","172.21.15.179:0/1009979748":"2023-10-20T14:13:20.213663+0000","172.21.15.179:6833/2587699336":"2023-10-20T14:13:20.213663+0000","172.21.15.179:0/2961439032":"2023-10-20T14:13:20.213663+0000","172.21.15.179:0/1362323478":"2023-10-20T14:13:20.213663+0000","172.21.15.179:0/1220569576":"2023-10-20T14:18:23.856456+0000","172.21.15.179:0/2642945549":"2023-10-20T14:13:20.213663+0000","172.21.15.179:0/2377816178":"2023-10-20T14:18:23.856456+0000","172.21.15.179:0/1066702535":"2023-10-20T14:02:21.481635+0000","172.21.15.179:0/904747968":"2023-10-20T14:02:21.481635+0000","172.21.15.179:0/1902799271":"2023-10-20T14:02:21.481635+0000","172.21.15.179:0/2939688089":"2023-10-20T14:02:21.481635+0000","172.21.15.179:0/682180483":"2023-10-20T14:02:21.481635+0000","172.21.15.179:6833/1426553632":"2023-10-20T14:02:21.481635+0000","172.21.15.179:6832/1426553632":"2023-10-20T14:02:21.481635+0000","172.21.15.179:0/2321890842":"2023-10-20T14:02:21.481635+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T14:18:24.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.780+0000 7f3fe0ba5640 1 -- 172.21.15.57:0/2837417145 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3fc80047c0 msgr2=0x7f3fdc0adb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:24.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.780+0000 7f3fe0ba5640 1 --2- 172.21.15.57:0/2837417145 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3fc80047c0 0x7f3fdc0adb20 secure :-1 s=READY pgs=2128 cs=0 l=1 rev1=1 crypto rx=0x7f3fcc003ec0 tx=0x7f3fcc001630 comp rx=0 tx=0).stop 2023-10-19T14:18:24.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.780+0000 7f3fe0ba5640 1 -- 172.21.15.57:0/2837417145 shutdown_connections 2023-10-19T14:18:24.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.780+0000 7f3fe0ba5640 1 --2- 172.21.15.57:0/2837417145 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3fdc0ab720 0x7f3fdc0abba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:24.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.780+0000 7f3fe0ba5640 1 --2- 172.21.15.57:0/2837417145 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3fdc13ada0 0x7f3fdc13b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:24.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.780+0000 7f3fe0ba5640 1 --2- 172.21.15.57:0/2837417145 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3fc80047c0 0x7f3fdc0adb20 unknown :-1 s=CLOSED pgs=2128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:24.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.780+0000 7f3fe0ba5640 1 -- 172.21.15.57:0/2837417145 >> 172.21.15.57:0/2837417145 conn(0x7f3fdc069930 msgr2=0x7f3fdc069d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:24.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.780+0000 7f3fe0ba5640 1 -- 172.21.15.57:0/2837417145 shutdown_connections 2023-10-19T14:18:24.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.780+0000 7f3fe0ba5640 1 -- 172.21.15.57:0/2837417145 wait complete. 2023-10-19T14:18:24.793 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-19T14:18:24.794 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T14:18:24.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.903+0000 7fa18b59e640 1 -- 172.21.15.57:0/204882557 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa18c0ad7b0 msgr2=0x7fa18c0adc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:24.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.903+0000 7fa18b59e640 1 --2- 172.21.15.57:0/204882557 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa18c0ad7b0 0x7fa18c0adc10 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fa174009860 tx=0x7fa17402bde0 comp rx=0 tx=0).stop 2023-10-19T14:18:24.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.903+0000 7fa18b59e640 1 -- 172.21.15.57:0/204882557 shutdown_connections 2023-10-19T14:18:24.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.903+0000 7fa18b59e640 1 --2- 172.21.15.57:0/204882557 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa18c0ad7b0 0x7fa18c0adc10 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:24.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.903+0000 7fa18b59e640 1 -- 172.21.15.57:0/204882557 >> 172.21.15.57:0/204882557 conn(0x7fa18c0b3db0 msgr2=0x7fa18c0b41c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:24.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.903+0000 7fa18b59e640 1 -- 172.21.15.57:0/204882557 shutdown_connections 2023-10-19T14:18:24.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.903+0000 7fa18b59e640 1 -- 172.21.15.57:0/204882557 wait complete. 2023-10-19T14:18:24.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.904+0000 7fa18b59e640 1 Processor -- start 2023-10-19T14:18:24.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.904+0000 7fa18b59e640 1 -- start start 2023-10-19T14:18:24.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.905+0000 7fa18b59e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa18c0ad7b0 0x7fa18c14a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:24.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.905+0000 7fa18b59e640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa18c0ae400 0x7fa18c14ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:24.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.905+0000 7fa18b59e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa18c158f00 0x7fa18c14bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:24.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.905+0000 7fa18b59e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa18c14d2a0 con 0x7fa18c0ad7b0 2023-10-19T14:18:24.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.905+0000 7fa18b59e640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa18c14d580 con 0x7fa18c158f00 2023-10-19T14:18:24.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.905+0000 7fa18b59e640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa18c14d860 con 0x7fa18c0ae400 2023-10-19T14:18:24.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.905+0000 7fa18a59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa18c0ad7b0 0x7fa18c14a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:24.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.905+0000 7fa18a59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa18c0ad7b0 0x7fa18c14a3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40174/0 (socket says 172.21.15.57:40174) 2023-10-19T14:18:24.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.905+0000 7fa18a59c640 1 -- 172.21.15.57:0/2014498659 learned_addr learned my addr 172.21.15.57:0/2014498659 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:18:24.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.905+0000 7fa181d9b640 1 --2- 172.21.15.57:0/2014498659 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa18c0ae400 0x7fa18c14ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:24.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.905+0000 7fa18ad9d640 1 --2- 172.21.15.57:0/2014498659 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa18c158f00 0x7fa18c14bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:24.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.905+0000 7fa18a59c640 1 -- 172.21.15.57:0/2014498659 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa18c0ae400 msgr2=0x7fa18c14ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:24.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.905+0000 7fa18a59c640 1 --2- 172.21.15.57:0/2014498659 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa18c0ae400 0x7fa18c14ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:24.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.905+0000 7fa18a59c640 1 -- 172.21.15.57:0/2014498659 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa18c158f00 msgr2=0x7fa18c14bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:24.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.905+0000 7fa18a59c640 1 --2- 172.21.15.57:0/2014498659 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa18c158f00 0x7fa18c14bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:24.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.905+0000 7fa18a59c640 1 -- 172.21.15.57:0/2014498659 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa174009490 con 0x7fa18c0ad7b0 2023-10-19T14:18:24.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.905+0000 7fa18a59c640 1 --2- 172.21.15.57:0/2014498659 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa18c0ad7b0 0x7fa18c14a3f0 secure :-1 s=READY pgs=2130 cs=0 l=1 rev1=1 crypto rx=0x7fa17c00c430 tx=0x7fa17c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:24.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.906+0000 7fa1837fe640 1 -- 172.21.15.57:0/2014498659 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa17c0052d0 con 0x7fa18c0ad7b0 2023-10-19T14:18:24.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.906+0000 7fa1837fe640 1 -- 172.21.15.57:0/2014498659 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fa17c0035f0 con 0x7fa18c0ad7b0 2023-10-19T14:18:24.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.906+0000 7fa18b59e640 1 -- 172.21.15.57:0/2014498659 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa18c14db40 con 0x7fa18c0ad7b0 2023-10-19T14:18:24.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.906+0000 7fa1837fe640 1 -- 172.21.15.57:0/2014498659 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa17c003a00 con 0x7fa18c0ad7b0 2023-10-19T14:18:24.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.906+0000 7fa18b59e640 1 -- 172.21.15.57:0/2014498659 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa18c14a930 con 0x7fa18c0ad7b0 2023-10-19T14:18:24.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.907+0000 7fa18b59e640 1 -- 172.21.15.57:0/2014498659 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa18c0b0390 con 0x7fa18c0ad7b0 2023-10-19T14:18:24.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.907+0000 7fa1837fe640 1 -- 172.21.15.57:0/2014498659 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 115) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fa17c003b60 con 0x7fa18c0ad7b0 2023-10-19T14:18:24.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.908+0000 7fa1837fe640 1 --2- 172.21.15.57:0/2014498659 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fa16c06acc0 0x7fa16c06d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:24.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.908+0000 7fa1837fe640 1 -- 172.21.15.57:0/2014498659 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 7117+0+0 (secure 0 0 0) 0x7fa17c088f00 con 0x7fa18c0ad7b0 2023-10-19T14:18:24.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.908+0000 7fa181d9b640 1 --2- 172.21.15.57:0/2014498659 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fa16c06acc0 0x7fa16c06d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:24.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.908+0000 7fa181d9b640 1 --2- 172.21.15.57:0/2014498659 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fa16c06acc0 0x7fa16c06d180 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fa174036000 tx=0x7fa174004730 comp rx=0 tx=0).ready entity=mgr.8744 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:24.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:24.912+0000 7fa1837fe640 1 -- 172.21.15.57:0/2014498659 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa17c053e20 con 0x7fa18c0ad7b0 2023-10-19T14:18:25.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.125+0000 7fa18b59e640 1 -- 172.21.15.57:0/2014498659 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa18c14e060 con 0x7fa18c0ad7b0 2023-10-19T14:18:25.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.125+0000 7fa1837fe640 1 -- 172.21.15.57:0/2014498659 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+4924 (secure 0 0 0) 0x7fa18c14e060 con 0x7fa18c0ad7b0 2023-10-19T14:18:25.129 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:18:25.129 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":77,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8480,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6833/2675481384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2675481384},{"type":"v1","addr":"172.21.15.57:6833","nonce":2675481384}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74},{"gid":8486,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/2884298100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2884298100},{"type":"v1","addr":"172.21.15.179:6836","nonce":2884298100}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":73},{"gid":10641,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/505361151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":505361151},{"type":"v1","addr":"172.21.15.57:6835","nonce":505361151}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T14:02:10.024658+0000","modified":"2023-10-19T14:02:12.029940+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":10647},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10647":{"gid":10647,"name":"d","rank":0,"incarnation":75,"state":"up:active","state_seq":3,"addr":"172.21.15.179:6837/3993214977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":3993214977},{"type":"v1","addr":"172.21.15.179:6837","nonce":3993214977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2023-10-19T14:18:25.129 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 77 2023-10-19T14:18:25.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.127+0000 7fa18b59e640 1 -- 172.21.15.57:0/2014498659 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fa16c06acc0 msgr2=0x7fa16c06d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:25.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.127+0000 7fa18b59e640 1 --2- 172.21.15.57:0/2014498659 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fa16c06acc0 0x7fa16c06d180 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fa174036000 tx=0x7fa174004730 comp rx=0 tx=0).stop 2023-10-19T14:18:25.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.127+0000 7fa18b59e640 1 -- 172.21.15.57:0/2014498659 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa18c0ad7b0 msgr2=0x7fa18c14a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:25.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.127+0000 7fa18b59e640 1 --2- 172.21.15.57:0/2014498659 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa18c0ad7b0 0x7fa18c14a3f0 secure :-1 s=READY pgs=2130 cs=0 l=1 rev1=1 crypto rx=0x7fa17c00c430 tx=0x7fa17c00c8f0 comp rx=0 tx=0).stop 2023-10-19T14:18:25.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.127+0000 7fa18b59e640 1 -- 172.21.15.57:0/2014498659 shutdown_connections 2023-10-19T14:18:25.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.127+0000 7fa18b59e640 1 --2- 172.21.15.57:0/2014498659 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fa16c06acc0 0x7fa16c06d180 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:25.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.127+0000 7fa18b59e640 1 --2- 172.21.15.57:0/2014498659 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa18c158f00 0x7fa18c14bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:25.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.127+0000 7fa18b59e640 1 --2- 172.21.15.57:0/2014498659 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa18c0ae400 0x7fa18c14ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:25.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.127+0000 7fa18b59e640 1 --2- 172.21.15.57:0/2014498659 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa18c0ad7b0 0x7fa18c14a3f0 unknown :-1 s=CLOSED pgs=2130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:25.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.127+0000 7fa18b59e640 1 -- 172.21.15.57:0/2014498659 >> 172.21.15.57:0/2014498659 conn(0x7fa18c0b3db0 msgr2=0x7fa18c1563d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:25.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.127+0000 7fa18b59e640 1 -- 172.21.15.57:0/2014498659 shutdown_connections 2023-10-19T14:18:25.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.127+0000 7fa18b59e640 1 -- 172.21.15.57:0/2014498659 wait complete. 2023-10-19T14:18:25.141 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T14:18:25.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.247+0000 7f0a62844640 1 -- 172.21.15.57:0/372881063 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0a5c0abc10 msgr2=0x7f0a5c0c9b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:25.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.247+0000 7f0a62844640 1 --2- 172.21.15.57:0/372881063 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0a5c0abc10 0x7f0a5c0c9b30 secure :-1 s=READY pgs=2131 cs=0 l=1 rev1=1 crypto rx=0x7f0a54009ca0 tx=0x7f0a5402c230 comp rx=0 tx=0).stop 2023-10-19T14:18:25.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.247+0000 7f0a62844640 1 -- 172.21.15.57:0/372881063 shutdown_connections 2023-10-19T14:18:25.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.247+0000 7f0a62844640 1 --2- 172.21.15.57:0/372881063 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0a5c0abc10 0x7f0a5c0c9b30 unknown :-1 s=CLOSED pgs=2131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:25.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.247+0000 7f0a62844640 1 --2- 172.21.15.57:0/372881063 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0a5c06af00 0x7f0a5c06b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:25.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.247+0000 7f0a62844640 1 -- 172.21.15.57:0/372881063 >> 172.21.15.57:0/372881063 conn(0x7f0a5c069930 msgr2=0x7f0a5c069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:25.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.247+0000 7f0a62844640 1 -- 172.21.15.57:0/372881063 shutdown_connections 2023-10-19T14:18:25.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.247+0000 7f0a62844640 1 -- 172.21.15.57:0/372881063 wait complete. 2023-10-19T14:18:25.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.248+0000 7f0a62844640 1 Processor -- start 2023-10-19T14:18:25.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.248+0000 7f0a62844640 1 -- start start 2023-10-19T14:18:25.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.249+0000 7f0a62844640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0a5c06af00 0x7f0a5c0ab3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:25.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.249+0000 7f0a62844640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0a5c0abc10 0x7f0a5c1ef340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:25.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.249+0000 7f0a62844640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0a5c0c96b0 0x7f0a5c13cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:25.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.249+0000 7f0a62844640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a5c0ab910 con 0x7f0a5c06af00 2023-10-19T14:18:25.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.249+0000 7f0a62844640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a5c12fb10 con 0x7f0a5c0c96b0 2023-10-19T14:18:25.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.249+0000 7f0a62844640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a5c12fdf0 con 0x7f0a5c0abc10 2023-10-19T14:18:25.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.249+0000 7f0a5bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0a5c06af00 0x7f0a5c0ab3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:25.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.249+0000 7f0a5bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0a5c06af00 0x7f0a5c0ab3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40188/0 (socket says 172.21.15.57:40188) 2023-10-19T14:18:25.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.249+0000 7f0a5bfff640 1 -- 172.21.15.57:0/4183499116 learned_addr learned my addr 172.21.15.57:0/4183499116 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:18:25.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.249+0000 7f0a60de1640 1 --2- 172.21.15.57:0/4183499116 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0a5c0c96b0 0x7f0a5c13cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:25.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.249+0000 7f0a5b7fe640 1 --2- 172.21.15.57:0/4183499116 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0a5c0abc10 0x7f0a5c1ef340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:25.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.249+0000 7f0a5bfff640 1 -- 172.21.15.57:0/4183499116 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0a5c0abc10 msgr2=0x7f0a5c1ef340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:25.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.249+0000 7f0a5bfff640 1 --2- 172.21.15.57:0/4183499116 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0a5c0abc10 0x7f0a5c1ef340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:25.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.249+0000 7f0a5bfff640 1 -- 172.21.15.57:0/4183499116 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0a5c0c96b0 msgr2=0x7f0a5c13cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:25.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.249+0000 7f0a5bfff640 1 --2- 172.21.15.57:0/4183499116 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0a5c0c96b0 0x7f0a5c13cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:25.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.250+0000 7f0a5bfff640 1 -- 172.21.15.57:0/4183499116 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a54009900 con 0x7f0a5c06af00 2023-10-19T14:18:25.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.250+0000 7f0a5bfff640 1 --2- 172.21.15.57:0/4183499116 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0a5c06af00 0x7f0a5c0ab3d0 secure :-1 s=READY pgs=2132 cs=0 l=1 rev1=1 crypto rx=0x7f0a4c00b4a0 tx=0x7f0a4c00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:25.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.250+0000 7f0a58ff9640 1 -- 172.21.15.57:0/4183499116 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a4c010610 con 0x7f0a5c06af00 2023-10-19T14:18:25.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.250+0000 7f0a58ff9640 1 -- 172.21.15.57:0/4183499116 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f0a4c010860 con 0x7f0a5c06af00 2023-10-19T14:18:25.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.250+0000 7f0a62844640 1 -- 172.21.15.57:0/4183499116 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a5c1300d0 con 0x7f0a5c06af00 2023-10-19T14:18:25.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.250+0000 7f0a58ff9640 1 -- 172.21.15.57:0/4183499116 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a4c010c70 con 0x7f0a5c06af00 2023-10-19T14:18:25.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.250+0000 7f0a62844640 1 -- 172.21.15.57:0/4183499116 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a5c130690 con 0x7f0a5c06af00 2023-10-19T14:18:25.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.251+0000 7f0a62844640 1 -- 172.21.15.57:0/4183499116 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a5c1308b0 con 0x7f0a5c06af00 2023-10-19T14:18:25.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.252+0000 7f0a58ff9640 1 -- 172.21.15.57:0/4183499116 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 115) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f0a4c017360 con 0x7f0a5c06af00 2023-10-19T14:18:25.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.252+0000 7f0a58ff9640 1 --2- 172.21.15.57:0/4183499116 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f0a3c06ac90 0x7f0a3c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:25.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.252+0000 7f0a58ff9640 1 -- 172.21.15.57:0/4183499116 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 7117+0+0 (secure 0 0 0) 0x7f0a4c0896a0 con 0x7f0a5c06af00 2023-10-19T14:18:25.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.252+0000 7f0a5b7fe640 1 --2- 172.21.15.57:0/4183499116 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f0a3c06ac90 0x7f0a3c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:25.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.253+0000 7f0a5b7fe640 1 --2- 172.21.15.57:0/4183499116 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f0a3c06ac90 0x7f0a3c06d150 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f0a44006fd0 tx=0x7f0a44008040 comp rx=0 tx=0).ready entity=mgr.8744 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:25.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.256+0000 7f0a58ff9640 1 -- 172.21.15.57:0/4183499116 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0a4c057fa0 con 0x7f0a5c06af00 2023-10-19T14:18:25.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.470+0000 7f0a62844640 1 -- 172.21.15.57:0/4183499116 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0a5c06c7e0 con 0x7f0a5c06af00 2023-10-19T14:18:25.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.470+0000 7f0a58ff9640 1 -- 172.21.15.57:0/4183499116 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+4924 (secure 0 0 0) 0x7f0a4c057b30 con 0x7f0a5c06af00 2023-10-19T14:18:25.474 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:18:25.474 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":77,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8480,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6833/2675481384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2675481384},{"type":"v1","addr":"172.21.15.57:6833","nonce":2675481384}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74},{"gid":8486,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/2884298100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2884298100},{"type":"v1","addr":"172.21.15.179:6836","nonce":2884298100}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":73},{"gid":10641,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/505361151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":505361151},{"type":"v1","addr":"172.21.15.57:6835","nonce":505361151}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T14:02:10.024658+0000","modified":"2023-10-19T14:02:12.029940+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":10647},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10647":{"gid":10647,"name":"d","rank":0,"incarnation":75,"state":"up:active","state_seq":3,"addr":"172.21.15.179:6837/3993214977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":3993214977},{"type":"v1","addr":"172.21.15.179:6837","nonce":3993214977}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2023-10-19T14:18:25.474 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 77 2023-10-19T14:18:25.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.473+0000 7f0a62844640 1 -- 172.21.15.57:0/4183499116 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f0a3c06ac90 msgr2=0x7f0a3c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:25.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.473+0000 7f0a62844640 1 --2- 172.21.15.57:0/4183499116 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f0a3c06ac90 0x7f0a3c06d150 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f0a44006fd0 tx=0x7f0a44008040 comp rx=0 tx=0).stop 2023-10-19T14:18:25.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.473+0000 7f0a62844640 1 -- 172.21.15.57:0/4183499116 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0a5c06af00 msgr2=0x7f0a5c0ab3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:25.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.473+0000 7f0a62844640 1 --2- 172.21.15.57:0/4183499116 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0a5c06af00 0x7f0a5c0ab3d0 secure :-1 s=READY pgs=2132 cs=0 l=1 rev1=1 crypto rx=0x7f0a4c00b4a0 tx=0x7f0a4c00b960 comp rx=0 tx=0).stop 2023-10-19T14:18:25.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.473+0000 7f0a62844640 1 -- 172.21.15.57:0/4183499116 shutdown_connections 2023-10-19T14:18:25.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.473+0000 7f0a62844640 1 --2- 172.21.15.57:0/4183499116 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f0a3c06ac90 0x7f0a3c06d150 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:25.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.473+0000 7f0a62844640 1 --2- 172.21.15.57:0/4183499116 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0a5c0c96b0 0x7f0a5c13cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:25.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.473+0000 7f0a62844640 1 --2- 172.21.15.57:0/4183499116 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0a5c0abc10 0x7f0a5c1ef340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:25.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.473+0000 7f0a62844640 1 --2- 172.21.15.57:0/4183499116 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0a5c06af00 0x7f0a5c0ab3d0 unknown :-1 s=CLOSED pgs=2132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:25.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.473+0000 7f0a62844640 1 -- 172.21.15.57:0/4183499116 >> 172.21.15.57:0/4183499116 conn(0x7f0a5c069930 msgr2=0x7f0a5c0cb1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:25.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.473+0000 7f0a62844640 1 -- 172.21.15.57:0/4183499116 shutdown_connections 2023-10-19T14:18:25.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.473+0000 7f0a62844640 1 -- 172.21.15.57:0/4183499116 wait complete. 2023-10-19T14:18:25.486 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T14:18:25.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.592+0000 7fc4691d0640 1 -- 172.21.15.57:0/1461223325 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc46413d350 msgr2=0x7fc4640cbd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:25.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.592+0000 7fc4691d0640 1 --2- 172.21.15.57:0/1461223325 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc46413d350 0x7fc4640cbd60 secure :-1 s=READY pgs=2134 cs=0 l=1 rev1=1 crypto rx=0x7fc458005fd0 tx=0x7fc45802be10 comp rx=0 tx=0).stop 2023-10-19T14:18:25.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.592+0000 7fc4691d0640 1 -- 172.21.15.57:0/1461223325 shutdown_connections 2023-10-19T14:18:25.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.592+0000 7fc4691d0640 1 --2- 172.21.15.57:0/1461223325 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc46413d350 0x7fc4640cbd60 unknown :-1 s=CLOSED pgs=2134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:25.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.592+0000 7fc4691d0640 1 --2- 172.21.15.57:0/1461223325 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc4640cb980 0x7fc46413db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:25.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.592+0000 7fc4691d0640 1 -- 172.21.15.57:0/1461223325 >> 172.21.15.57:0/1461223325 conn(0x7fc464069930 msgr2=0x7fc464069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:25.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.592+0000 7fc4691d0640 1 -- 172.21.15.57:0/1461223325 shutdown_connections 2023-10-19T14:18:25.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.592+0000 7fc4691d0640 1 -- 172.21.15.57:0/1461223325 wait complete. 2023-10-19T14:18:25.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.593+0000 7fc4691d0640 1 Processor -- start 2023-10-19T14:18:25.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.593+0000 7fc4691d0640 1 -- start start 2023-10-19T14:18:25.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.594+0000 7fc4691d0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4640c96b0 0x7fc4641eea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:25.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.594+0000 7fc4691d0640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc4640cb980 0x7fc46415a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:25.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.594+0000 7fc4691d0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc4641592d0 0x7fc464154780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:25.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.594+0000 7fc4691d0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc46415ae20 con 0x7fc4640c96b0 2023-10-19T14:18:25.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.594+0000 7fc4691d0640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc464158230 con 0x7fc4641592d0 2023-10-19T14:18:25.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.594+0000 7fc4691d0640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc464158510 con 0x7fc4640cb980 2023-10-19T14:18:25.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.594+0000 7fc462d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4640c96b0 0x7fc4641eea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:25.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.594+0000 7fc462d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4640c96b0 0x7fc4641eea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40204/0 (socket says 172.21.15.57:40204) 2023-10-19T14:18:25.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.594+0000 7fc462d9d640 1 -- 172.21.15.57:0/3762156886 learned_addr learned my addr 172.21.15.57:0/3762156886 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:18:25.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.594+0000 7fc46359e640 1 --2- 172.21.15.57:0/3762156886 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc4641592d0 0x7fc464154780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:25.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.594+0000 7fc46259c640 1 --2- 172.21.15.57:0/3762156886 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc4640cb980 0x7fc46415a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:25.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.594+0000 7fc462d9d640 1 -- 172.21.15.57:0/3762156886 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc4640cb980 msgr2=0x7fc46415a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:25.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.594+0000 7fc462d9d640 1 --2- 172.21.15.57:0/3762156886 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc4640cb980 0x7fc46415a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:25.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.594+0000 7fc462d9d640 1 -- 172.21.15.57:0/3762156886 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc4641592d0 msgr2=0x7fc464154780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:25.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.594+0000 7fc462d9d640 1 --2- 172.21.15.57:0/3762156886 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc4641592d0 0x7fc464154780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:25.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.594+0000 7fc462d9d640 1 -- 172.21.15.57:0/3762156886 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc458009560 con 0x7fc4640c96b0 2023-10-19T14:18:25.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.595+0000 7fc462d9d640 1 --2- 172.21.15.57:0/3762156886 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4640c96b0 0x7fc4641eea40 secure :-1 s=READY pgs=2135 cs=0 l=1 rev1=1 crypto rx=0x7fc45c00d950 tx=0x7fc45c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:25.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.595+0000 7fc44b7fe640 1 -- 172.21.15.57:0/3762156886 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc45c009c30 con 0x7fc4640c96b0 2023-10-19T14:18:25.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.595+0000 7fc44b7fe640 1 -- 172.21.15.57:0/3762156886 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fc45c009e80 con 0x7fc4640c96b0 2023-10-19T14:18:25.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.595+0000 7fc4691d0640 1 -- 172.21.15.57:0/3762156886 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4641587f0 con 0x7fc4640c96b0 2023-10-19T14:18:25.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.595+0000 7fc44b7fe640 1 -- 172.21.15.57:0/3762156886 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc45c0106c0 con 0x7fc4640c96b0 2023-10-19T14:18:25.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.595+0000 7fc4691d0640 1 -- 172.21.15.57:0/3762156886 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc464158db0 con 0x7fc4640c96b0 2023-10-19T14:18:25.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.596+0000 7fc4691d0640 1 -- 172.21.15.57:0/3762156886 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4640690e0 con 0x7fc4640c96b0 2023-10-19T14:18:25.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.596+0000 7fc44b7fe640 1 -- 172.21.15.57:0/3762156886 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 115) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fc45c010820 con 0x7fc4640c96b0 2023-10-19T14:18:25.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.597+0000 7fc44b7fe640 1 --2- 172.21.15.57:0/3762156886 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fc45006acc0 0x7fc45006d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:25.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.597+0000 7fc44b7fe640 1 -- 172.21.15.57:0/3762156886 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 7117+0+0 (secure 0 0 0) 0x7fc45c089d40 con 0x7fc4640c96b0 2023-10-19T14:18:25.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.597+0000 7fc46259c640 1 --2- 172.21.15.57:0/3762156886 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fc45006acc0 0x7fc45006d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:25.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.598+0000 7fc46259c640 1 --2- 172.21.15.57:0/3762156886 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fc45006acc0 0x7fc45006d180 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc458035000 tx=0x7fc458004730 comp rx=0 tx=0).ready entity=mgr.8744 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:25.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.601+0000 7fc44b7fe640 1 -- 172.21.15.57:0/3762156886 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc45c054ae0 con 0x7fc4640c96b0 2023-10-19T14:18:25.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.779+0000 7fc4691d0640 1 -- 172.21.15.57:0/3762156886 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc46406ff80 con 0x7fc4640c96b0 2023-10-19T14:18:25.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.780+0000 7fc44b7fe640 1 -- 172.21.15.57:0/3762156886 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v193) v1 ==== 74+0+18367 (secure 0 0 0) 0x7fc45c017090 con 0x7fc4640c96b0 2023-10-19T14:18:25.784 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:18:25.784 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":193,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T14:18:23.856506+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2023-10-19T14:02:08.318466+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"187","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2023-10-19T14:02:08.995986+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"192","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":33,"snap_epoch":192,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.179:0/1944205818":"2023-10-20T14:18:23.856456+0000","172.21.15.179:0/530749640":"2023-10-20T14:18:23.856456+0000","172.21.15.179:0/3372019790":"2023-10-20T14:18:23.856456+0000","172.21.15.179:6832/4086068793":"2023-10-20T14:18:23.856456+0000","172.21.15.179:0/4245102210":"2023-10-20T14:18:23.856456+0000","172.21.15.179:0/1094895803":"2023-10-20T14:13:20.213663+0000","172.21.15.179:6832/2587699336":"2023-10-20T14:13:20.213663+0000","172.21.15.179:0/2295535972":"2023-10-20T14:13:20.213663+0000","172.21.15.179:6833/4086068793":"2023-10-20T14:18:23.856456+0000","172.21.15.179:0/1009979748":"2023-10-20T14:13:20.213663+0000","172.21.15.179:6833/2587699336":"2023-10-20T14:13:20.213663+0000","172.21.15.179:0/2961439032":"2023-10-20T14:13:20.213663+0000","172.21.15.179:0/1362323478":"2023-10-20T14:13:20.213663+0000","172.21.15.179:0/1220569576":"2023-10-20T14:18:23.856456+0000","172.21.15.179:0/2642945549":"2023-10-20T14:13:20.213663+0000","172.21.15.179:0/2377816178":"2023-10-20T14:18:23.856456+0000","172.21.15.179:0/1066702535":"2023-10-20T14:02:21.481635+0000","172.21.15.179:0/904747968":"2023-10-20T14:02:21.481635+0000","172.21.15.179:0/1902799271":"2023-10-20T14:02:21.481635+0000","172.21.15.179:0/2939688089":"2023-10-20T14:02:21.481635+0000","172.21.15.179:0/682180483":"2023-10-20T14:02:21.481635+0000","172.21.15.179:6833/1426553632":"2023-10-20T14:02:21.481635+0000","172.21.15.179:6832/1426553632":"2023-10-20T14:02:21.481635+0000","172.21.15.179:0/2321890842":"2023-10-20T14:02:21.481635+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T14:18:25.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.783+0000 7fc4691d0640 1 -- 172.21.15.57:0/3762156886 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fc45006acc0 msgr2=0x7fc45006d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:25.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.783+0000 7fc4691d0640 1 --2- 172.21.15.57:0/3762156886 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fc45006acc0 0x7fc45006d180 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc458035000 tx=0x7fc458004730 comp rx=0 tx=0).stop 2023-10-19T14:18:25.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.783+0000 7fc4691d0640 1 -- 172.21.15.57:0/3762156886 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4640c96b0 msgr2=0x7fc4641eea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:25.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.783+0000 7fc4691d0640 1 --2- 172.21.15.57:0/3762156886 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4640c96b0 0x7fc4641eea40 secure :-1 s=READY pgs=2135 cs=0 l=1 rev1=1 crypto rx=0x7fc45c00d950 tx=0x7fc45c00de10 comp rx=0 tx=0).stop 2023-10-19T14:18:25.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.783+0000 7fc4691d0640 1 -- 172.21.15.57:0/3762156886 shutdown_connections 2023-10-19T14:18:25.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.783+0000 7fc4691d0640 1 --2- 172.21.15.57:0/3762156886 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fc45006acc0 0x7fc45006d180 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:25.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.783+0000 7fc4691d0640 1 --2- 172.21.15.57:0/3762156886 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc4641592d0 0x7fc464154780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:25.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.783+0000 7fc4691d0640 1 --2- 172.21.15.57:0/3762156886 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc4640cb980 0x7fc46415a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:25.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.783+0000 7fc4691d0640 1 --2- 172.21.15.57:0/3762156886 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4640c96b0 0x7fc4641eea40 unknown :-1 s=CLOSED pgs=2135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:25.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.783+0000 7fc4691d0640 1 -- 172.21.15.57:0/3762156886 >> 172.21.15.57:0/3762156886 conn(0x7fc464069930 msgr2=0x7fc464069d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:25.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.783+0000 7fc4691d0640 1 -- 172.21.15.57:0/3762156886 shutdown_connections 2023-10-19T14:18:25.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.783+0000 7fc4691d0640 1 -- 172.21.15.57:0/3762156886 wait complete. 2023-10-19T14:18:25.796 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-19T14:18:25.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.904+0000 7fb94c300640 1 -- 172.21.15.57:0/1572980515 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb94415b2a0 msgr2=0x7fb9440cf720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:25.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.904+0000 7fb94c300640 1 --2- 172.21.15.57:0/1572980515 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb94415b2a0 0x7fb9440cf720 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fb934009830 tx=0x7fb93402bcd0 comp rx=0 tx=0).stop 2023-10-19T14:18:25.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.905+0000 7fb94c300640 1 -- 172.21.15.57:0/1572980515 shutdown_connections 2023-10-19T14:18:25.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.905+0000 7fb94c300640 1 --2- 172.21.15.57:0/1572980515 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb94415b2a0 0x7fb9440cf720 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:25.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.905+0000 7fb94c300640 1 --2- 172.21.15.57:0/1572980515 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb9440cf340 0x7fb9440d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:25.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.905+0000 7fb94c300640 1 -- 172.21.15.57:0/1572980515 >> 172.21.15.57:0/1572980515 conn(0x7fb9440d0d60 msgr2=0x7fb9440d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:25.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.905+0000 7fb94c300640 1 -- 172.21.15.57:0/1572980515 shutdown_connections 2023-10-19T14:18:25.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.905+0000 7fb94c300640 1 -- 172.21.15.57:0/1572980515 wait complete. 2023-10-19T14:18:25.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.906+0000 7fb94c300640 1 Processor -- start 2023-10-19T14:18:25.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.906+0000 7fb94c300640 1 -- start start 2023-10-19T14:18:25.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.906+0000 7fb94c300640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb9440ceb40 0x7fb944158520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:25.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.906+0000 7fb94c300640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb9440cf340 0x7fb9441f10d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:25.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.906+0000 7fb94c300640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb94415b2a0 0x7fb944158ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:25.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.906+0000 7fb94c300640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9441f5e60 con 0x7fb94415b2a0 2023-10-19T14:18:25.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.906+0000 7fb94c300640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb944158100 con 0x7fb9440cf340 2023-10-19T14:18:25.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.906+0000 7fb94c300640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb944157760 con 0x7fb9440ceb40 2023-10-19T14:18:25.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.907+0000 7fb94a89d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb94415b2a0 0x7fb944158ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:25.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.907+0000 7fb94a09c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb9440ceb40 0x7fb944158520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:25.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.907+0000 7fb94989b640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb9440cf340 0x7fb9441f10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:25.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.907+0000 7fb94a89d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb94415b2a0 0x7fb944158ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40216/0 (socket says 172.21.15.57:40216) 2023-10-19T14:18:25.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.907+0000 7fb94a89d640 1 -- 172.21.15.57:0/2763878201 learned_addr learned my addr 172.21.15.57:0/2763878201 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:18:25.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.907+0000 7fb94989b640 1 -- 172.21.15.57:0/2763878201 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb9440ceb40 msgr2=0x7fb944158520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:25.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.907+0000 7fb94989b640 1 --2- 172.21.15.57:0/2763878201 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb9440ceb40 0x7fb944158520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:25.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.907+0000 7fb94989b640 1 -- 172.21.15.57:0/2763878201 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb94415b2a0 msgr2=0x7fb944158ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:25.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.907+0000 7fb94989b640 1 --2- 172.21.15.57:0/2763878201 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb94415b2a0 0x7fb944158ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:25.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.907+0000 7fb94989b640 1 -- 172.21.15.57:0/2763878201 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb940009560 con 0x7fb9440cf340 2023-10-19T14:18:25.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.907+0000 7fb94a89d640 1 --2- 172.21.15.57:0/2763878201 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb94415b2a0 0x7fb944158ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T14:18:25.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.907+0000 7fb94989b640 1 --2- 172.21.15.57:0/2763878201 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb9440cf340 0x7fb9441f10d0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fb934033000 tx=0x7fb9340038a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:25.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.908+0000 7fb93affd640 1 -- 172.21.15.57:0/2763878201 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb934003ea0 con 0x7fb9440cf340 2023-10-19T14:18:25.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.908+0000 7fb93affd640 1 -- 172.21.15.57:0/2763878201 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fb934034440 con 0x7fb9440cf340 2023-10-19T14:18:25.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.908+0000 7fb94c300640 1 -- 172.21.15.57:0/2763878201 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb934009490 con 0x7fb9440cf340 2023-10-19T14:18:25.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.908+0000 7fb93affd640 1 -- 172.21.15.57:0/2763878201 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb934034830 con 0x7fb9440cf340 2023-10-19T14:18:25.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.908+0000 7fb94c300640 1 -- 172.21.15.57:0/2763878201 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb944157d20 con 0x7fb9440cf340 2023-10-19T14:18:25.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.909+0000 7fb94c300640 1 -- 172.21.15.57:0/2763878201 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9440b6c70 con 0x7fb9440cf340 2023-10-19T14:18:25.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.909+0000 7fb93affd640 1 -- 172.21.15.57:0/2763878201 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 115) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fb934034990 con 0x7fb9440cf340 2023-10-19T14:18:25.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.910+0000 7fb93affd640 1 --2- 172.21.15.57:0/2763878201 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fb92806ac90 0x7fb92806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:25.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.910+0000 7fb93affd640 1 -- 172.21.15.57:0/2763878201 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 7117+0+0 (secure 0 0 0) 0x7fb9340abfe0 con 0x7fb9440cf340 2023-10-19T14:18:25.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.910+0000 7fb94a09c640 1 --2- 172.21.15.57:0/2763878201 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fb92806ac90 0x7fb92806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:25.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.910+0000 7fb94a09c640 1 --2- 172.21.15.57:0/2763878201 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fb92806ac90 0x7fb92806d150 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fb944153ff0 tx=0x7fb940009260 comp rx=0 tx=0).ready entity=mgr.8744 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:25.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:25.914+0000 7fb93affd640 1 -- 172.21.15.57:0/2763878201 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb934076e30 con 0x7fb9440cf340 2023-10-19T14:18:26.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:26.129+0000 7fb94c300640 1 -- 172.21.15.57:0/2763878201 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fb944158240 con 0x7fb9440cf340 2023-10-19T14:18:26.133 INFO:tasks.ceph.mon.a.smithi057.stderr:2023-10-19T14:18:26.130+0000 7f1253ff3640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-19T14:18:26.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:26.136+0000 7fb93affd640 1 -- 172.21.15.57:0/2763878201 <== mon.1 v2:172.21.15.179:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v78) v1 ==== 161+0+0 (secure 0 0 0) 0x7fb944158240 con 0x7fb9440cf340 2023-10-19T14:18:26.139 INFO:teuthology.orchestra.run.smithi057.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-19T14:18:26.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:26.138+0000 7fb94c300640 1 -- 172.21.15.57:0/2763878201 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fb92806ac90 msgr2=0x7fb92806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:26.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:26.138+0000 7fb94c300640 1 --2- 172.21.15.57:0/2763878201 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fb92806ac90 0x7fb92806d150 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fb944153ff0 tx=0x7fb940009260 comp rx=0 tx=0).stop 2023-10-19T14:18:26.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:26.138+0000 7fb94c300640 1 -- 172.21.15.57:0/2763878201 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb9440cf340 msgr2=0x7fb9441f10d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:26.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:26.138+0000 7fb94c300640 1 --2- 172.21.15.57:0/2763878201 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb9440cf340 0x7fb9441f10d0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fb934033000 tx=0x7fb9340038a0 comp rx=0 tx=0).stop 2023-10-19T14:18:26.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:26.138+0000 7fb94c300640 1 -- 172.21.15.57:0/2763878201 shutdown_connections 2023-10-19T14:18:26.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:26.138+0000 7fb94c300640 1 --2- 172.21.15.57:0/2763878201 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fb92806ac90 0x7fb92806d150 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:26.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:26.138+0000 7fb94c300640 1 --2- 172.21.15.57:0/2763878201 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb94415b2a0 0x7fb944158ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:26.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:26.138+0000 7fb94c300640 1 --2- 172.21.15.57:0/2763878201 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb9440cf340 0x7fb9441f10d0 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:26.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:26.138+0000 7fb94c300640 1 --2- 172.21.15.57:0/2763878201 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb9440ceb40 0x7fb944158520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:26.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:26.138+0000 7fb94c300640 1 -- 172.21.15.57:0/2763878201 >> 172.21.15.57:0/2763878201 conn(0x7fb9440d0d60 msgr2=0x7fb94406b500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:26.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:26.138+0000 7fb94c300640 1 -- 172.21.15.57:0/2763878201 shutdown_connections 2023-10-19T14:18:26.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:26.138+0000 7fb94c300640 1 -- 172.21.15.57:0/2763878201 wait complete. 2023-10-19T14:18:26.151 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-19T14:18:26.173 INFO:tasks.ceph.mds.d.smithi179.stdout:starting mds.d at 2023-10-19T14:18:26.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:26.260+0000 7fb83c0f2640 1 -- 172.21.15.57:0/3346052473 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8340ab500 msgr2=0x7fb8340ab990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:26.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:26.260+0000 7fb83c0f2640 1 --2- 172.21.15.57:0/3346052473 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8340ab500 0x7fb8340ab990 secure :-1 s=READY pgs=2137 cs=0 l=1 rev1=1 crypto rx=0x7fb8340ca260 tx=0x7fb82c02c2f0 comp rx=0 tx=0).stop 2023-10-19T14:18:26.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:26.260+0000 7fb83c0f2640 1 -- 172.21.15.57:0/3346052473 shutdown_connections 2023-10-19T14:18:26.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:26.260+0000 7fb83c0f2640 1 --2- 172.21.15.57:0/3346052473 >> v2:172.21.15.179:3300/0 conn(0x7fb8340c9dc0 0x7fb83406b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:26.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:26.260+0000 7fb83c0f2640 1 --2- 172.21.15.57:0/3346052473 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8340ab500 0x7fb8340ab990 unknown :-1 s=CLOSED pgs=2137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:26.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:26.260+0000 7fb83c0f2640 1 -- 172.21.15.57:0/3346052473 >> 172.21.15.57:0/3346052473 conn(0x7fb834069930 msgr2=0x7fb834069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:26.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:26.260+0000 7fb83c0f2640 1 -- 172.21.15.57:0/3346052473 shutdown_connections 2023-10-19T14:18:26.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:26.260+0000 7fb83c0f2640 1 -- 172.21.15.57:0/3346052473 wait complete. 2023-10-19T14:18:26.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:26.262+0000 7fb83c0f2640 1 Processor -- start 2023-10-19T14:18:26.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:26.262+0000 7fb83c0f2640 1 -- start start 2023-10-19T14:18:26.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:26.262+0000 7fb83c0f2640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb83406af00 0x7fb8340a6e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:26.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:26.262+0000 7fb83c0f2640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb8340ab500 0x7fb83413a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:26.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:26.262+0000 7fb83c0f2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8340c9dc0 0x7fb83413a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:26.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:26.262+0000 7fb83c0f2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8340b0230 con 0x7fb8340c9dc0 2023-10-19T14:18:26.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:26.262+0000 7fb83c0f2640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8340af6b0 con 0x7fb8340ab500 2023-10-19T14:18:26.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:26.262+0000 7fb83c0f2640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8340af990 con 0x7fb83406af00 2023-10-19T14:18:26.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:26.262+0000 7fb83a68f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8340c9dc0 0x7fb83413a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:26.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:26.262+0000 7fb83a68f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8340c9dc0 0x7fb83413a600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40232/0 (socket says 172.21.15.57:40232) 2023-10-19T14:18:26.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:26.262+0000 7fb83a68f640 1 -- 172.21.15.57:0/1928032605 learned_addr learned my addr 172.21.15.57:0/1928032605 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:18:26.272 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:26.263+0000 7fb83968d640 1 --2- 172.21.15.57:0/1928032605 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb8340ab500 0x7fb83413a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:26.272 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:26.263+0000 7fb839e8e640 1 --2- 172.21.15.57:0/1928032605 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb83406af00 0x7fb8340a6e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:26.272 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:26.263+0000 7fb83a68f640 1 -- 172.21.15.57:0/1928032605 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb83406af00 msgr2=0x7fb8340a6e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:26.272 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:26.263+0000 7fb83a68f640 1 --2- 172.21.15.57:0/1928032605 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb83406af00 0x7fb8340a6e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:26.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:26.263+0000 7fb83a68f640 1 -- 172.21.15.57:0/1928032605 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb8340ab500 msgr2=0x7fb83413a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:26.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:26.263+0000 7fb83a68f640 1 --2- 172.21.15.57:0/1928032605 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb8340ab500 0x7fb83413a0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:26.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:26.263+0000 7fb83a68f640 1 -- 172.21.15.57:0/1928032605 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb82c009900 con 0x7fb8340c9dc0 2023-10-19T14:18:26.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:26.263+0000 7fb83968d640 1 --2- 172.21.15.57:0/1928032605 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb8340ab500 0x7fb83413a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:18:26.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:26.263+0000 7fb83a68f640 1 --2- 172.21.15.57:0/1928032605 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8340c9dc0 0x7fb83413a600 secure :-1 s=READY pgs=2138 cs=0 l=1 rev1=1 crypto rx=0x7fb82c037040 tx=0x7fb82c035e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:26.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:26.263+0000 7fb82a7fc640 1 -- 172.21.15.57:0/1928032605 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb82c02cea0 con 0x7fb8340c9dc0 2023-10-19T14:18:26.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:26.263+0000 7fb82a7fc640 1 -- 172.21.15.57:0/1928032605 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fb82c004250 con 0x7fb8340c9dc0 2023-10-19T14:18:26.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:26.263+0000 7fb82a7fc640 1 -- 172.21.15.57:0/1928032605 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb82c004640 con 0x7fb8340c9dc0 2023-10-19T14:18:26.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:26.263+0000 7fb83c0f2640 1 -- 172.21.15.57:0/1928032605 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8340afc70 con 0x7fb8340c9dc0 2023-10-19T14:18:26.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:26.263+0000 7fb83c0f2640 1 -- 172.21.15.57:0/1928032605 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8340a73c0 con 0x7fb8340c9dc0 2023-10-19T14:18:26.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:26.264+0000 7fb83c0f2640 1 -- 172.21.15.57:0/1928032605 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8340abe50 con 0x7fb8340c9dc0 2023-10-19T14:18:26.275 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:26.265+0000 7fb82a7fc640 1 -- 172.21.15.57:0/1928032605 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 115) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fb82c03b040 con 0x7fb8340c9dc0 2023-10-19T14:18:26.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:26.265+0000 7fb82a7fc640 1 --2- 172.21.15.57:0/1928032605 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fb81806ac90 0x7fb81806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:26.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:26.265+0000 7fb82a7fc640 1 -- 172.21.15.57:0/1928032605 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(194..194 src has 1..194) v4 ==== 7203+0+0 (secure 0 0 0) 0x7fb82c0ac570 con 0x7fb8340c9dc0 2023-10-19T14:18:26.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:26.265+0000 7fb839e8e640 1 --2- 172.21.15.57:0/1928032605 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fb81806ac90 0x7fb81806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:26.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:26.266+0000 7fb839e8e640 1 --2- 172.21.15.57:0/1928032605 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fb81806ac90 0x7fb81806d150 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb8341f2a60 tx=0x7fb830009040 comp rx=0 tx=0).ready entity=mgr.8744 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:26.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:26.270+0000 7fb82a7fc640 1 -- 172.21.15.57:0/1928032605 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb82c077380 con 0x7fb8340c9dc0 2023-10-19T14:18:26.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:26.482+0000 7fb83c0f2640 1 -- 172.21.15.57:0/1928032605 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fb8340690e0 con 0x7fb8340c9dc0 2023-10-19T14:18:26.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:26.876+0000 7fb82a7fc640 1 -- 172.21.15.57:0/1928032605 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mgrmap(e 116) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fb82c00a040 con 0x7fb8340c9dc0 2023-10-19T14:18:27.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:27.134+0000 7fb82a7fc640 1 -- 172.21.15.57:0/1928032605 <== mon.0 v2:172.21.15.57:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v79) v1 ==== 104+0+0 (secure 0 0 0) 0x7fb82c0a4f10 con 0x7fb8340c9dc0 2023-10-19T14:18:27.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:27.137+0000 7fb83c0f2640 1 -- 172.21.15.57:0/1928032605 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fb81806ac90 msgr2=0x7fb81806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:27.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:27.137+0000 7fb83c0f2640 1 --2- 172.21.15.57:0/1928032605 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fb81806ac90 0x7fb81806d150 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb8341f2a60 tx=0x7fb830009040 comp rx=0 tx=0).stop 2023-10-19T14:18:27.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:27.137+0000 7fb83c0f2640 1 -- 172.21.15.57:0/1928032605 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8340c9dc0 msgr2=0x7fb83413a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:27.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:27.137+0000 7fb83c0f2640 1 --2- 172.21.15.57:0/1928032605 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8340c9dc0 0x7fb83413a600 secure :-1 s=READY pgs=2138 cs=0 l=1 rev1=1 crypto rx=0x7fb82c037040 tx=0x7fb82c035e50 comp rx=0 tx=0).stop 2023-10-19T14:18:27.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:27.137+0000 7fb83c0f2640 1 -- 172.21.15.57:0/1928032605 shutdown_connections 2023-10-19T14:18:27.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:27.137+0000 7fb83c0f2640 1 --2- 172.21.15.57:0/1928032605 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fb81806ac90 0x7fb81806d150 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:27.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:27.137+0000 7fb83c0f2640 1 --2- 172.21.15.57:0/1928032605 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8340c9dc0 0x7fb83413a600 unknown :-1 s=CLOSED pgs=2138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:27.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:27.137+0000 7fb83c0f2640 1 --2- 172.21.15.57:0/1928032605 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb8340ab500 0x7fb83413a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:27.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:27.137+0000 7fb83c0f2640 1 --2- 172.21.15.57:0/1928032605 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb83406af00 0x7fb8340a6e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:27.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:27.137+0000 7fb83c0f2640 1 -- 172.21.15.57:0/1928032605 >> 172.21.15.57:0/1928032605 conn(0x7fb834069930 msgr2=0x7fb834069d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:27.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:27.137+0000 7fb83c0f2640 1 -- 172.21.15.57:0/1928032605 shutdown_connections 2023-10-19T14:18:27.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:27.137+0000 7fb83c0f2640 1 -- 172.21.15.57:0/1928032605 wait complete. 2023-10-19T14:18:27.152 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-19T14:18:27.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:27.268+0000 7fd322773640 1 -- 172.21.15.57:0/1421940081 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd31c0cf340 msgr2=0x7fd31c0d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:27.272 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:27.268+0000 7fd322773640 1 --2- 172.21.15.57:0/1421940081 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd31c0cf340 0x7fd31c0d1910 secure :-1 s=READY pgs=2140 cs=0 l=1 rev1=1 crypto rx=0x7fd30c005fd0 tx=0x7fd30c02bd90 comp rx=0 tx=0).stop 2023-10-19T14:18:27.272 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:27.268+0000 7fd322773640 1 -- 172.21.15.57:0/1421940081 shutdown_connections 2023-10-19T14:18:27.272 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:27.268+0000 7fd322773640 1 --2- 172.21.15.57:0/1421940081 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd31c0ceb40 0x7fd31c0b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:27.272 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:27.268+0000 7fd322773640 1 --2- 172.21.15.57:0/1421940081 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd31c0cf340 0x7fd31c0d1910 unknown :-1 s=CLOSED pgs=2140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:27.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:27.268+0000 7fd322773640 1 -- 172.21.15.57:0/1421940081 >> 172.21.15.57:0/1421940081 conn(0x7fd31c0d0d60 msgr2=0x7fd31c0d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:27.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:27.268+0000 7fd322773640 1 -- 172.21.15.57:0/1421940081 shutdown_connections 2023-10-19T14:18:27.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:27.268+0000 7fd322773640 1 -- 172.21.15.57:0/1421940081 wait complete. 2023-10-19T14:18:27.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:27.269+0000 7fd322773640 1 Processor -- start 2023-10-19T14:18:27.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:27.270+0000 7fd322773640 1 -- start start 2023-10-19T14:18:27.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:27.270+0000 7fd322773640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd31c0ceb40 0x7fd31c15c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:27.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:27.270+0000 7fd322773640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd31c0cf340 0x7fd31c155340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:27.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:27.270+0000 7fd322773640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd31c15b2a0 0x7fd31c0ca3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:27.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:27.270+0000 7fd322773640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd31c0cac30 con 0x7fd31c0ceb40 2023-10-19T14:18:27.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:27.270+0000 7fd322773640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd31c0b49d0 con 0x7fd31c15b2a0 2023-10-19T14:18:27.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:27.270+0000 7fd322773640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd31c0b4cb0 con 0x7fd31c0cf340 2023-10-19T14:18:27.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:27.270+0000 7fd31bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd31c0ceb40 0x7fd31c15c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:27.278 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:27.270+0000 7fd31bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd31c0ceb40 0x7fd31c15c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51228/0 (socket says 172.21.15.57:51228) 2023-10-19T14:18:27.278 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:27.270+0000 7fd31bfff640 1 -- 172.21.15.57:0/3247864389 learned_addr learned my addr 172.21.15.57:0/3247864389 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:18:27.278 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:27.271+0000 7fd31b7fe640 1 --2- 172.21.15.57:0/3247864389 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd31c0cf340 0x7fd31c155340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:27.278 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:27.271+0000 7fd320d10640 1 --2- 172.21.15.57:0/3247864389 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd31c15b2a0 0x7fd31c0ca3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:27.279 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:27.271+0000 7fd31bfff640 1 -- 172.21.15.57:0/3247864389 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd31c0cf340 msgr2=0x7fd31c155340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:27.279 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:27.271+0000 7fd31bfff640 1 --2- 172.21.15.57:0/3247864389 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd31c0cf340 0x7fd31c155340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:27.279 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:27.271+0000 7fd31bfff640 1 -- 172.21.15.57:0/3247864389 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd31c15b2a0 msgr2=0x7fd31c0ca3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:27.279 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:27.271+0000 7fd31bfff640 1 --2- 172.21.15.57:0/3247864389 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd31c15b2a0 0x7fd31c0ca3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:27.279 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:27.271+0000 7fd31bfff640 1 -- 172.21.15.57:0/3247864389 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd30c009560 con 0x7fd31c0ceb40 2023-10-19T14:18:27.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:27.271+0000 7fd31bfff640 1 --2- 172.21.15.57:0/3247864389 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd31c0ceb40 0x7fd31c15c420 secure :-1 s=READY pgs=2141 cs=0 l=1 rev1=1 crypto rx=0x7fd30c006010 tx=0x7fd30c02db30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:27.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:27.271+0000 7fd318ff9640 1 -- 172.21.15.57:0/3247864389 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd30c0035f0 con 0x7fd31c0ceb40 2023-10-19T14:18:27.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:27.271+0000 7fd318ff9640 1 -- 172.21.15.57:0/3247864389 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fd30c003840 con 0x7fd31c0ceb40 2023-10-19T14:18:27.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:27.271+0000 7fd322773640 1 -- 172.21.15.57:0/3247864389 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd31c0b0d50 con 0x7fd31c0ceb40 2023-10-19T14:18:27.282 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:27.271+0000 7fd318ff9640 1 -- 172.21.15.57:0/3247864389 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd30c003c50 con 0x7fd31c0ceb40 2023-10-19T14:18:27.282 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:27.271+0000 7fd322773640 1 -- 172.21.15.57:0/3247864389 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd31c0b5270 con 0x7fd31c0ceb40 2023-10-19T14:18:27.282 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:27.272+0000 7fd322773640 1 -- 172.21.15.57:0/3247864389 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd31c0690e0 con 0x7fd31c0ceb40 2023-10-19T14:18:27.282 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:27.273+0000 7fd318ff9640 1 -- 172.21.15.57:0/3247864389 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 116) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fd30c03e450 con 0x7fd31c0ceb40 2023-10-19T14:18:27.282 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:27.273+0000 7fd318ff9640 1 --2- 172.21.15.57:0/3247864389 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fd30806aed0 0x7fd30806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:27.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:27.273+0000 7fd318ff9640 1 -- 172.21.15.57:0/3247864389 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(194..194 src has 1..194) v4 ==== 7203+0+0 (secure 0 0 0) 0x7fd30c0ac5f0 con 0x7fd31c0ceb40 2023-10-19T14:18:27.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:27.274+0000 7fd31b7fe640 1 --2- 172.21.15.57:0/3247864389 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fd30806aed0 0x7fd30806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:27.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:27.274+0000 7fd31b7fe640 1 --2- 172.21.15.57:0/3247864389 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fd30806aed0 0x7fd30806d390 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd31c08af90 tx=0x7fd310009040 comp rx=0 tx=0).ready entity=mgr.8744 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:27.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:27.278+0000 7fd318ff9640 1 -- 172.21.15.57:0/3247864389 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd30c0772a0 con 0x7fd31c0ceb40 2023-10-19T14:18:27.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:27.458+0000 7fd322773640 1 -- 172.21.15.57:0/3247864389 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fd31c0d0b40 con 0x7fd31c0ceb40 2023-10-19T14:18:28.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:28.412+0000 7fd318ff9640 1 -- 172.21.15.57:0/3247864389 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v195) v1 ==== 181+0+0 (secure 0 0 0) 0x7fd30c0430d0 con 0x7fd31c0ceb40 2023-10-19T14:18:28.415 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'cephfs_metadata' removed 2023-10-19T14:18:28.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:28.415+0000 7fd322773640 1 -- 172.21.15.57:0/3247864389 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fd30806aed0 msgr2=0x7fd30806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:28.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:28.415+0000 7fd322773640 1 --2- 172.21.15.57:0/3247864389 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fd30806aed0 0x7fd30806d390 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd31c08af90 tx=0x7fd310009040 comp rx=0 tx=0).stop 2023-10-19T14:18:28.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:28.415+0000 7fd322773640 1 -- 172.21.15.57:0/3247864389 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd31c0ceb40 msgr2=0x7fd31c15c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:28.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:28.415+0000 7fd322773640 1 --2- 172.21.15.57:0/3247864389 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd31c0ceb40 0x7fd31c15c420 secure :-1 s=READY pgs=2141 cs=0 l=1 rev1=1 crypto rx=0x7fd30c006010 tx=0x7fd30c02db30 comp rx=0 tx=0).stop 2023-10-19T14:18:28.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:28.415+0000 7fd322773640 1 -- 172.21.15.57:0/3247864389 shutdown_connections 2023-10-19T14:18:28.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:28.415+0000 7fd322773640 1 --2- 172.21.15.57:0/3247864389 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fd30806aed0 0x7fd30806d390 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:28.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:28.415+0000 7fd322773640 1 --2- 172.21.15.57:0/3247864389 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd31c15b2a0 0x7fd31c0ca3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:28.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:28.415+0000 7fd322773640 1 --2- 172.21.15.57:0/3247864389 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd31c0cf340 0x7fd31c155340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:28.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:28.415+0000 7fd322773640 1 --2- 172.21.15.57:0/3247864389 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd31c0ceb40 0x7fd31c15c420 unknown :-1 s=CLOSED pgs=2141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:28.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:28.415+0000 7fd322773640 1 -- 172.21.15.57:0/3247864389 >> 172.21.15.57:0/3247864389 conn(0x7fd31c0d0d60 msgr2=0x7fd31c06b5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:28.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:28.415+0000 7fd322773640 1 -- 172.21.15.57:0/3247864389 shutdown_connections 2023-10-19T14:18:28.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:28.415+0000 7fd322773640 1 -- 172.21.15.57:0/3247864389 wait complete. 2023-10-19T14:18:28.439 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-19T14:18:28.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:28.556+0000 7fefe5d00640 1 -- 172.21.15.57:0/3019929192 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fefe0141690 msgr2=0x7fefe00703e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:28.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:28.556+0000 7fefe5d00640 1 --2- 172.21.15.57:0/3019929192 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fefe0141690 0x7fefe00703e0 secure :-1 s=READY pgs=2143 cs=0 l=1 rev1=1 crypto rx=0x7fefe00cbd00 tx=0x7fefd402bda0 comp rx=0 tx=0).stop 2023-10-19T14:18:28.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:28.556+0000 7fefe5d00640 1 -- 172.21.15.57:0/3019929192 shutdown_connections 2023-10-19T14:18:28.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:28.556+0000 7fefe5d00640 1 --2- 172.21.15.57:0/3019929192 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fefe0141690 0x7fefe00703e0 unknown :-1 s=CLOSED pgs=2143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:28.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:28.556+0000 7fefe5d00640 1 --2- 172.21.15.57:0/3019929192 >> v2:172.21.15.179:3300/0 conn(0x7fefe013e520 0x7fefe006bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:28.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:28.556+0000 7fefe5d00640 1 -- 172.21.15.57:0/3019929192 >> 172.21.15.57:0/3019929192 conn(0x7fefe0069930 msgr2=0x7fefe0069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:28.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:28.556+0000 7fefe5d00640 1 -- 172.21.15.57:0/3019929192 shutdown_connections 2023-10-19T14:18:28.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:28.556+0000 7fefe5d00640 1 -- 172.21.15.57:0/3019929192 wait complete. 2023-10-19T14:18:28.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:28.557+0000 7fefe5d00640 1 Processor -- start 2023-10-19T14:18:28.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:28.557+0000 7fefe5d00640 1 -- start start 2023-10-19T14:18:28.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:28.558+0000 7fefe5d00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fefe006c0a0 0x7fefe013b110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:28.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:28.558+0000 7fefe5d00640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fefe013e520 0x7fefe01e4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:28.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:28.558+0000 7fefe5d00640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fefe0141690 0x7fefe00c6de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:28.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:28.558+0000 7fefe5d00640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefe01e0da0 con 0x7fefe013e520 2023-10-19T14:18:28.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:28.558+0000 7fefe5d00640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefe01e5970 con 0x7fefe006c0a0 2023-10-19T14:18:28.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:28.558+0000 7fefe5d00640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefe01e5c50 con 0x7fefe0141690 2023-10-19T14:18:28.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:28.558+0000 7fefdeffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fefe013e520 0x7fefe01e4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:28.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:28.558+0000 7fefdeffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fefe013e520 0x7fefe01e4ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51238/0 (socket says 172.21.15.57:51238) 2023-10-19T14:18:28.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:28.558+0000 7fefdeffd640 1 -- 172.21.15.57:0/1226842673 learned_addr learned my addr 172.21.15.57:0/1226842673 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:18:28.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:28.558+0000 7fefdffff640 1 --2- 172.21.15.57:0/1226842673 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fefe0141690 0x7fefe00c6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:28.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:28.558+0000 7fefdf7fe640 1 --2- 172.21.15.57:0/1226842673 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fefe006c0a0 0x7fefe013b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:28.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:28.558+0000 7fefdeffd640 1 -- 172.21.15.57:0/1226842673 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fefe0141690 msgr2=0x7fefe00c6de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:28.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:28.558+0000 7fefdeffd640 1 --2- 172.21.15.57:0/1226842673 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fefe0141690 0x7fefe00c6de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:28.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:28.558+0000 7fefdeffd640 1 -- 172.21.15.57:0/1226842673 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fefe006c0a0 msgr2=0x7fefe013b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:28.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:28.558+0000 7fefdeffd640 1 --2- 172.21.15.57:0/1226842673 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fefe006c0a0 0x7fefe013b110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:28.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:28.558+0000 7fefdeffd640 1 -- 172.21.15.57:0/1226842673 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefd4009560 con 0x7fefe013e520 2023-10-19T14:18:28.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:28.559+0000 7fefdeffd640 1 --2- 172.21.15.57:0/1226842673 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fefe013e520 0x7fefe01e4ef0 secure :-1 s=READY pgs=2144 cs=0 l=1 rev1=1 crypto rx=0x7fefd4009850 tx=0x7fefd402dad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:28.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:28.559+0000 7fefc7fff640 1 -- 172.21.15.57:0/1226842673 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefd4003800 con 0x7fefe013e520 2023-10-19T14:18:28.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:28.559+0000 7fefc7fff640 1 -- 172.21.15.57:0/1226842673 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fefd4003a50 con 0x7fefe013e520 2023-10-19T14:18:28.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:28.559+0000 7fefc7fff640 1 -- 172.21.15.57:0/1226842673 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefd4003e20 con 0x7fefe013e520 2023-10-19T14:18:28.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:28.559+0000 7fefe5d00640 1 -- 172.21.15.57:0/1226842673 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefe01e1010 con 0x7fefe013e520 2023-10-19T14:18:28.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:28.559+0000 7fefe5d00640 1 -- 172.21.15.57:0/1226842673 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefe01f4c50 con 0x7fefe013e520 2023-10-19T14:18:28.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:28.560+0000 7fefe5d00640 1 -- 172.21.15.57:0/1226842673 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefe00b1bd0 con 0x7fefe013e520 2023-10-19T14:18:28.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:28.560+0000 7fefc7fff640 1 -- 172.21.15.57:0/1226842673 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 117) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fefd403e450 con 0x7fefe013e520 2023-10-19T14:18:28.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:28.561+0000 7fefc7fff640 1 --2- 172.21.15.57:0/1226842673 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fefcc06af60 0x7fefcc06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:28.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:28.561+0000 7fefc7fff640 1 -- 172.21.15.57:0/1226842673 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 6765+0+0 (secure 0 0 0) 0x7fefd40ac700 con 0x7fefe013e520 2023-10-19T14:18:28.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:28.561+0000 7fefdf7fe640 1 --2- 172.21.15.57:0/1226842673 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fefcc06af60 0x7fefcc06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:28.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:28.561+0000 7fefdf7fe640 1 --2- 172.21.15.57:0/1226842673 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fefcc06af60 0x7fefcc06d420 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fefd0005fd0 tx=0x7fefd0005850 comp rx=0 tx=0).ready entity=mgr.8744 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:28.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:28.565+0000 7fefc7fff640 1 -- 172.21.15.57:0/1226842673 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fefd4077260 con 0x7fefe013e520 2023-10-19T14:18:28.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:28.747+0000 7fefe5d00640 1 -- 172.21.15.57:0/1226842673 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fefe00aef70 con 0x7fefe013e520 2023-10-19T14:18:29.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.298+0000 7fefc7fff640 1 -- 172.21.15.57:0/1226842673 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v196) v1 ==== 169+0+0 (secure 0 0 0) 0x7fefd407a880 con 0x7fefe013e520 2023-10-19T14:18:29.300 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'cephfs_data' removed 2023-10-19T14:18:29.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.305+0000 7fefe5d00640 1 -- 172.21.15.57:0/1226842673 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fefcc06af60 msgr2=0x7fefcc06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:29.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.305+0000 7fefe5d00640 1 --2- 172.21.15.57:0/1226842673 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fefcc06af60 0x7fefcc06d420 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fefd0005fd0 tx=0x7fefd0005850 comp rx=0 tx=0).stop 2023-10-19T14:18:29.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.305+0000 7fefe5d00640 1 -- 172.21.15.57:0/1226842673 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fefe013e520 msgr2=0x7fefe01e4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:29.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.305+0000 7fefe5d00640 1 --2- 172.21.15.57:0/1226842673 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fefe013e520 0x7fefe01e4ef0 secure :-1 s=READY pgs=2144 cs=0 l=1 rev1=1 crypto rx=0x7fefd4009850 tx=0x7fefd402dad0 comp rx=0 tx=0).stop 2023-10-19T14:18:29.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.305+0000 7fefe5d00640 1 -- 172.21.15.57:0/1226842673 shutdown_connections 2023-10-19T14:18:29.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.305+0000 7fefe5d00640 1 --2- 172.21.15.57:0/1226842673 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fefcc06af60 0x7fefcc06d420 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:29.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.305+0000 7fefe5d00640 1 --2- 172.21.15.57:0/1226842673 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fefe0141690 0x7fefe00c6de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:29.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.305+0000 7fefe5d00640 1 --2- 172.21.15.57:0/1226842673 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fefe013e520 0x7fefe01e4ef0 unknown :-1 s=CLOSED pgs=2144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:29.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.305+0000 7fefe5d00640 1 --2- 172.21.15.57:0/1226842673 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fefe006c0a0 0x7fefe013b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:29.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.305+0000 7fefe5d00640 1 -- 172.21.15.57:0/1226842673 >> 172.21.15.57:0/1226842673 conn(0x7fefe0069930 msgr2=0x7fefe01e2be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:29.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.306+0000 7fefe5d00640 1 -- 172.21.15.57:0/1226842673 shutdown_connections 2023-10-19T14:18:29.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.306+0000 7fefe5d00640 1 -- 172.21.15.57:0/1226842673 wait complete. 2023-10-19T14:18:29.324 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 30 2023-10-19T14:18:29.439 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.437+0000 7f7d0714c640 1 -- 172.21.15.57:0/1578169166 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ce80039a0 msgr2=0x7f7ce8003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:29.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.437+0000 7f7d0714c640 1 --2- 172.21.15.57:0/1578169166 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ce80039a0 0x7f7ce8003dc0 secure :-1 s=READY pgs=2146 cs=0 l=1 rev1=1 crypto rx=0x7f7cf8001d50 tx=0x7f7ce80057f0 comp rx=0 tx=0).stop 2023-10-19T14:18:29.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.437+0000 7f7d056e9640 1 -- 172.21.15.57:0/1578169166 reap_dead start 2023-10-19T14:18:29.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.437+0000 7f7d0714c640 1 -- 172.21.15.57:0/1578169166 shutdown_connections 2023-10-19T14:18:29.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.437+0000 7f7d0714c640 1 -- 172.21.15.57:0/1578169166 >> 172.21.15.57:0/1578169166 conn(0x7f7d00069930 msgr2=0x7f7d00069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:29.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.437+0000 7f7d0714c640 1 -- 172.21.15.57:0/1578169166 shutdown_connections 2023-10-19T14:18:29.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.437+0000 7f7d0714c640 1 -- 172.21.15.57:0/1578169166 wait complete. 2023-10-19T14:18:29.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.439+0000 7f7d0714c640 1 Processor -- start 2023-10-19T14:18:29.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.439+0000 7f7d0714c640 1 -- start start 2023-10-19T14:18:29.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.439+0000 7f7d0714c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7ce80039a0 0x7f7d001eece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:29.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.439+0000 7f7d0714c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7d000aa4f0 0x7f7d000aa970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:29.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.439+0000 7f7d0714c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7d000aaeb0 0x7f7d0015a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:29.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.439+0000 7f7d0714c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d0015bc50 con 0x7f7d000aaeb0 2023-10-19T14:18:29.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.439+0000 7f7d0714c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d001ef220 con 0x7f7ce80039a0 2023-10-19T14:18:29.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.439+0000 7f7d0714c640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d00154660 con 0x7f7d000aa4f0 2023-10-19T14:18:29.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.440+0000 7f7d056e9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7d000aaeb0 0x7f7d0015a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:29.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.440+0000 7f7d056e9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7d000aaeb0 0x7f7d0015a3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51250/0 (socket says 172.21.15.57:51250) 2023-10-19T14:18:29.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.440+0000 7f7d056e9640 1 -- 172.21.15.57:0/1808263565 learned_addr learned my addr 172.21.15.57:0/1808263565 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:18:29.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.440+0000 7f7d04ee8640 1 --2- 172.21.15.57:0/1808263565 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7ce80039a0 0x7f7d001eece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:29.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.440+0000 7f7cfffff640 1 --2- 172.21.15.57:0/1808263565 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7d000aa4f0 0x7f7d000aa970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:29.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.440+0000 7f7d056e9640 1 -- 172.21.15.57:0/1808263565 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7d000aa4f0 msgr2=0x7f7d000aa970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:29.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.440+0000 7f7d056e9640 1 --2- 172.21.15.57:0/1808263565 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7d000aa4f0 0x7f7d000aa970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:29.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.440+0000 7f7d056e9640 1 -- 172.21.15.57:0/1808263565 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7ce80039a0 msgr2=0x7f7d001eece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:29.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.440+0000 7f7d056e9640 1 --2- 172.21.15.57:0/1808263565 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7ce80039a0 0x7f7d001eece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:29.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.440+0000 7f7d056e9640 1 -- 172.21.15.57:0/1808263565 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d00153a50 con 0x7f7d000aaeb0 2023-10-19T14:18:29.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.440+0000 7f7d056e9640 1 --2- 172.21.15.57:0/1808263565 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7d000aaeb0 0x7f7d0015a3b0 secure :-1 s=READY pgs=2147 cs=0 l=1 rev1=1 crypto rx=0x7f7d000712f0 tx=0x7f7cf0011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:29.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.440+0000 7f7cfd7fa640 1 -- 172.21.15.57:0/1808263565 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cf0004440 con 0x7f7d000aaeb0 2023-10-19T14:18:29.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.440+0000 7f7cfd7fa640 1 -- 172.21.15.57:0/1808263565 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f7cf000f8f0 con 0x7f7d000aaeb0 2023-10-19T14:18:29.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.440+0000 7f7cfd7fa640 1 -- 172.21.15.57:0/1808263565 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cf000fd00 con 0x7f7d000aaeb0 2023-10-19T14:18:29.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.440+0000 7f7d0714c640 1 -- 172.21.15.57:0/1808263565 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d00153d30 con 0x7f7d000aaeb0 2023-10-19T14:18:29.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.440+0000 7f7d0714c640 1 -- 172.21.15.57:0/1808263565 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d001542f0 con 0x7f7d000aaeb0 2023-10-19T14:18:29.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.441+0000 7f7d0714c640 1 -- 172.21.15.57:0/1808263565 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d00154480 con 0x7f7d000aaeb0 2023-10-19T14:18:29.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.442+0000 7f7cfd7fa640 1 -- 172.21.15.57:0/1808263565 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 117) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f7cf00035f0 con 0x7f7d000aaeb0 2023-10-19T14:18:29.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.442+0000 7f7cfd7fa640 1 --2- 172.21.15.57:0/1808263565 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f7cec06af60 0x7f7cec06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:29.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.442+0000 7f7cfd7fa640 1 -- 172.21.15.57:0/1808263565 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 6351+0+0 (secure 0 0 0) 0x7f7cf008a600 con 0x7f7d000aaeb0 2023-10-19T14:18:29.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.442+0000 7f7d04ee8640 1 --2- 172.21.15.57:0/1808263565 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f7cec06af60 0x7f7cec06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:29.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.443+0000 7f7d04ee8640 1 --2- 172.21.15.57:0/1808263565 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f7cec06af60 0x7f7cec06d420 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7cf8034000 tx=0x7f7cf8001510 comp rx=0 tx=0).ready entity=mgr.8744 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:29.452 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.446+0000 7f7cfd7fa640 1 -- 172.21.15.57:0/1808263565 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7cf00552c0 con 0x7f7d000aaeb0 2023-10-19T14:18:29.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.621+0000 7f7d0714c640 1 -- 172.21.15.57:0/1808263565 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "config reset", "num": 30} v 0) v1 -- 0x7f7d000690e0 con 0x7f7d000aaeb0 2023-10-19T14:18:29.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.624+0000 7f7cfd7fa640 1 -- 172.21.15.57:0/1808263565 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 30}]=0 v70) v1 ==== 71+0+0 (secure 0 0 0) 0x7f7cf00588e0 con 0x7f7d000aaeb0 2023-10-19T14:18:29.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.626+0000 7f7d0714c640 1 -- 172.21.15.57:0/1808263565 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f7cec06af60 msgr2=0x7f7cec06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:29.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.627+0000 7f7d0714c640 1 --2- 172.21.15.57:0/1808263565 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f7cec06af60 0x7f7cec06d420 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7cf8034000 tx=0x7f7cf8001510 comp rx=0 tx=0).stop 2023-10-19T14:18:29.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.627+0000 7f7d0714c640 1 -- 172.21.15.57:0/1808263565 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7d000aaeb0 msgr2=0x7f7d0015a3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:29.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.627+0000 7f7d0714c640 1 --2- 172.21.15.57:0/1808263565 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7d000aaeb0 0x7f7d0015a3b0 secure :-1 s=READY pgs=2147 cs=0 l=1 rev1=1 crypto rx=0x7f7d000712f0 tx=0x7f7cf0011b50 comp rx=0 tx=0).stop 2023-10-19T14:18:29.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.627+0000 7f7d0714c640 1 -- 172.21.15.57:0/1808263565 shutdown_connections 2023-10-19T14:18:29.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.627+0000 7f7d0714c640 1 --2- 172.21.15.57:0/1808263565 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f7cec06af60 0x7f7cec06d420 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:29.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.627+0000 7f7d0714c640 1 --2- 172.21.15.57:0/1808263565 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7d000aaeb0 0x7f7d0015a3b0 unknown :-1 s=CLOSED pgs=2147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:29.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.627+0000 7f7d0714c640 1 --2- 172.21.15.57:0/1808263565 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7d000aa4f0 0x7f7d000aa970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:29.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.627+0000 7f7d0714c640 1 --2- 172.21.15.57:0/1808263565 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7ce80039a0 0x7f7d001eece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:29.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.627+0000 7f7d0714c640 1 -- 172.21.15.57:0/1808263565 >> 172.21.15.57:0/1808263565 conn(0x7f7d00069930 msgr2=0x7f7d0013c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:29.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.627+0000 7f7d0714c640 1 -- 172.21.15.57:0/1808263565 shutdown_connections 2023-10-19T14:18:29.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.627+0000 7f7d0714c640 1 -- 172.21.15.57:0/1808263565 wait complete. 2023-10-19T14:18:29.639 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart' 2023-10-19T14:18:29.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.748+0000 7f7ef2165640 1 -- 172.21.15.57:0/2983452180 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ed4004d00 msgr2=0x7f7ed4005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:29.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.748+0000 7f7ef2165640 1 --2- 172.21.15.57:0/2983452180 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ed4004d00 0x7f7ed4005180 secure :-1 s=READY pgs=2149 cs=0 l=1 rev1=1 crypto rx=0x7f7ee400c640 tx=0x7f7ed40057f0 comp rx=0 tx=0).stop 2023-10-19T14:18:29.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.748+0000 7f7eebfff640 1 -- 172.21.15.57:0/2983452180 reap_dead start 2023-10-19T14:18:29.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.748+0000 7f7ef2165640 1 -- 172.21.15.57:0/2983452180 shutdown_connections 2023-10-19T14:18:29.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.748+0000 7f7ef2165640 1 -- 172.21.15.57:0/2983452180 >> 172.21.15.57:0/2983452180 conn(0x7f7eec069930 msgr2=0x7f7eec069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:29.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.748+0000 7f7ef2165640 1 -- 172.21.15.57:0/2983452180 shutdown_connections 2023-10-19T14:18:29.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.748+0000 7f7ef2165640 1 -- 172.21.15.57:0/2983452180 wait complete. 2023-10-19T14:18:29.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.749+0000 7f7ef2165640 1 Processor -- start 2023-10-19T14:18:29.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.749+0000 7f7ef2165640 1 -- start start 2023-10-19T14:18:29.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.750+0000 7f7ef2165640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7eec135b70 0x7f7eec14f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:29.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.750+0000 7f7ef2165640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7eec1583b0 0x7f7eec158830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:29.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.750+0000 7f7ef2165640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7eec0ad5e0 0x7f7eec0ada60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:29.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.750+0000 7f7ef2165640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eec150f80 con 0x7f7eec0ad5e0 2023-10-19T14:18:29.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.750+0000 7f7ef2165640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eec158d70 con 0x7f7eec1583b0 2023-10-19T14:18:29.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.750+0000 7f7ef2165640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7eec1500b0 con 0x7f7eec135b70 2023-10-19T14:18:29.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.750+0000 7f7eebfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7eec0ad5e0 0x7f7eec0ada60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:29.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.750+0000 7f7eebfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7eec0ad5e0 0x7f7eec0ada60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51264/0 (socket says 172.21.15.57:51264) 2023-10-19T14:18:29.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.750+0000 7f7eebfff640 1 -- 172.21.15.57:0/1503137576 learned_addr learned my addr 172.21.15.57:0/1503137576 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:18:29.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.750+0000 7f7eeb7fe640 1 --2- 172.21.15.57:0/1503137576 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7eec135b70 0x7f7eec14f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:29.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.750+0000 7f7eeaffd640 1 --2- 172.21.15.57:0/1503137576 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7eec1583b0 0x7f7eec158830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:29.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.750+0000 7f7eebfff640 1 -- 172.21.15.57:0/1503137576 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7eec135b70 msgr2=0x7f7eec14f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:29.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.750+0000 7f7eebfff640 1 --2- 172.21.15.57:0/1503137576 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7eec135b70 0x7f7eec14f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:29.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.750+0000 7f7eebfff640 1 -- 172.21.15.57:0/1503137576 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7eec1583b0 msgr2=0x7f7eec158830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:29.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.750+0000 7f7eebfff640 1 --2- 172.21.15.57:0/1503137576 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7eec1583b0 0x7f7eec158830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:29.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.750+0000 7f7eebfff640 1 -- 172.21.15.57:0/1503137576 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7eec150300 con 0x7f7eec0ad5e0 2023-10-19T14:18:29.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.750+0000 7f7eebfff640 1 --2- 172.21.15.57:0/1503137576 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7eec0ad5e0 0x7f7eec0ada60 secure :-1 s=READY pgs=2150 cs=0 l=1 rev1=1 crypto rx=0x7f7ee4001cb0 tx=0x7f7ee4001290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:29.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.750+0000 7f7ed3fff640 1 -- 172.21.15.57:0/1503137576 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ee40019f0 con 0x7f7eec0ad5e0 2023-10-19T14:18:29.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.751+0000 7f7ed3fff640 1 -- 172.21.15.57:0/1503137576 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f7ee4003970 con 0x7f7eec0ad5e0 2023-10-19T14:18:29.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.751+0000 7f7ef2165640 1 -- 172.21.15.57:0/1503137576 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7eec1505e0 con 0x7f7eec0ad5e0 2023-10-19T14:18:29.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.751+0000 7f7ed3fff640 1 -- 172.21.15.57:0/1503137576 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ee4003d40 con 0x7f7eec0ad5e0 2023-10-19T14:18:29.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.751+0000 7f7ef2165640 1 -- 172.21.15.57:0/1503137576 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7eec150ba0 con 0x7f7eec0ad5e0 2023-10-19T14:18:29.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.752+0000 7f7ef2165640 1 -- 172.21.15.57:0/1503137576 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7eec0690e0 con 0x7f7eec0ad5e0 2023-10-19T14:18:29.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.752+0000 7f7ed3fff640 1 -- 172.21.15.57:0/1503137576 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 117) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f7ee4031030 con 0x7f7eec0ad5e0 2023-10-19T14:18:29.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.752+0000 7f7ed3fff640 1 --2- 172.21.15.57:0/1503137576 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f7ed806af60 0x7f7ed806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:29.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.753+0000 7f7ed3fff640 1 -- 172.21.15.57:0/1503137576 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 6351+0+0 (secure 0 0 0) 0x7f7ee40aa6f0 con 0x7f7eec0ad5e0 2023-10-19T14:18:29.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.753+0000 7f7eeb7fe640 1 --2- 172.21.15.57:0/1503137576 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f7ed806af60 0x7f7ed806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:29.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.753+0000 7f7eeb7fe640 1 --2- 172.21.15.57:0/1503137576 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f7ed806af60 0x7f7ed806d420 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f7edc003640 tx=0x7f7edc00b260 comp rx=0 tx=0).ready entity=mgr.8744 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:29.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.757+0000 7f7ed3fff640 1 -- 172.21.15.57:0/1503137576 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7ee4075260 con 0x7f7eec0ad5e0 2023-10-19T14:18:30.001 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:29.999+0000 7f7ef2165640 1 -- 172.21.15.57:0/1503137576 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart"]} v 0) v1 -- 0x7f7eec06ff80 con 0x7f7eec0ad5e0 2023-10-19T14:18:30.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.414+0000 7f7ed3fff640 1 -- 172.21.15.57:0/1503137576 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart"]}]=0 v3051) v1 ==== 165+0+0 (secure 0 0 0) 0x7f7ee4078880 con 0x7f7eec0ad5e0 2023-10-19T14:18:30.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.416+0000 7f7ef2165640 1 -- 172.21.15.57:0/1503137576 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f7ed806af60 msgr2=0x7f7ed806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:30.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.416+0000 7f7ef2165640 1 --2- 172.21.15.57:0/1503137576 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f7ed806af60 0x7f7ed806d420 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f7edc003640 tx=0x7f7edc00b260 comp rx=0 tx=0).stop 2023-10-19T14:18:30.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.416+0000 7f7ef2165640 1 -- 172.21.15.57:0/1503137576 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7eec0ad5e0 msgr2=0x7f7eec0ada60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:30.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.416+0000 7f7ef2165640 1 --2- 172.21.15.57:0/1503137576 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7eec0ad5e0 0x7f7eec0ada60 secure :-1 s=READY pgs=2150 cs=0 l=1 rev1=1 crypto rx=0x7f7ee4001cb0 tx=0x7f7ee4001290 comp rx=0 tx=0).stop 2023-10-19T14:18:30.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.417+0000 7f7ef2165640 1 -- 172.21.15.57:0/1503137576 shutdown_connections 2023-10-19T14:18:30.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.417+0000 7f7ef2165640 1 --2- 172.21.15.57:0/1503137576 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f7ed806af60 0x7f7ed806d420 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:30.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.417+0000 7f7ef2165640 1 --2- 172.21.15.57:0/1503137576 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7eec0ad5e0 0x7f7eec0ada60 unknown :-1 s=CLOSED pgs=2150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:30.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.417+0000 7f7ef2165640 1 --2- 172.21.15.57:0/1503137576 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7eec1583b0 0x7f7eec158830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:30.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.417+0000 7f7ef2165640 1 --2- 172.21.15.57:0/1503137576 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7eec135b70 0x7f7eec14f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:30.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.417+0000 7f7ef2165640 1 -- 172.21.15.57:0/1503137576 >> 172.21.15.57:0/1503137576 conn(0x7f7eec069930 msgr2=0x7f7eec06c790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:30.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.417+0000 7f7ef2165640 1 -- 172.21.15.57:0/1503137576 shutdown_connections 2023-10-19T14:18:30.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.417+0000 7f7ef2165640 1 -- 172.21.15.57:0/1503137576 wait complete. 2023-10-19T14:18:30.431 INFO:tasks.cephfs_test_runner:Test that snap schedule is resumed after mgr restart ... ok 2023-10-19T14:18:30.432 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_with_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test scheduled snapshots along with rentention policy 2023-10-19T14:18:30.432 INFO:tasks.cephfs_test_runner:test_snap_schedule_with_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-10-19T14:18:30.433 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T14:18:30.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.542+0000 7f5199900640 1 -- 172.21.15.57:0/2965038821 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51940b1920 msgr2=0x7f519406fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:30.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.542+0000 7f5199900640 1 --2- 172.21.15.57:0/2965038821 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51940b1920 0x7f519406fa70 secure :-1 s=READY pgs=2151 cs=0 l=1 rev1=1 crypto rx=0x7f5184005fd0 tx=0x7f518402be10 comp rx=0 tx=0).stop 2023-10-19T14:18:30.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.542+0000 7f5199900640 1 -- 172.21.15.57:0/2965038821 shutdown_connections 2023-10-19T14:18:30.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.542+0000 7f5199900640 1 --2- 172.21.15.57:0/2965038821 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f51940c9dc0 0x7f5194154af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:30.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.542+0000 7f5199900640 1 --2- 172.21.15.57:0/2965038821 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51940b1920 0x7f519406fa70 unknown :-1 s=CLOSED pgs=2151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:30.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.542+0000 7f5199900640 1 -- 172.21.15.57:0/2965038821 >> 172.21.15.57:0/2965038821 conn(0x7f5194069930 msgr2=0x7f5194069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:30.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.542+0000 7f5199900640 1 -- 172.21.15.57:0/2965038821 shutdown_connections 2023-10-19T14:18:30.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.542+0000 7f5199900640 1 -- 172.21.15.57:0/2965038821 wait complete. 2023-10-19T14:18:30.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.543+0000 7f5199900640 1 Processor -- start 2023-10-19T14:18:30.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.543+0000 7f5199900640 1 -- start start 2023-10-19T14:18:30.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.544+0000 7f5199900640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f519406ffb0 0x7f51941446f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:30.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.544+0000 7f5199900640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f51940b1920 0x7f5194153750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:30.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.544+0000 7f5199900640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51940c9dc0 0x7f519406be50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:30.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.544+0000 7f5199900640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51941f4d70 con 0x7f51940c9dc0 2023-10-19T14:18:30.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.544+0000 7f5199900640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51941f5050 con 0x7f519406ffb0 2023-10-19T14:18:30.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.544+0000 7f5199900640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51941f5330 con 0x7f51940b1920 2023-10-19T14:18:30.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.544+0000 7f51937fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51940c9dc0 0x7f519406be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:30.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.544+0000 7f51937fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51940c9dc0 0x7f519406be50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51276/0 (socket says 172.21.15.57:51276) 2023-10-19T14:18:30.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.544+0000 7f51937fe640 1 -- 172.21.15.57:0/2342471886 learned_addr learned my addr 172.21.15.57:0/2342471886 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:18:30.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.544+0000 7f5192ffd640 1 --2- 172.21.15.57:0/2342471886 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f519406ffb0 0x7f51941446f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:30.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.544+0000 7f518bfff640 1 --2- 172.21.15.57:0/2342471886 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f51940b1920 0x7f5194153750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:30.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.544+0000 7f51937fe640 1 -- 172.21.15.57:0/2342471886 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f51940b1920 msgr2=0x7f5194153750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:30.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.544+0000 7f51937fe640 1 --2- 172.21.15.57:0/2342471886 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f51940b1920 0x7f5194153750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:30.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.544+0000 7f51937fe640 1 -- 172.21.15.57:0/2342471886 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f519406ffb0 msgr2=0x7f51941446f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:30.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.544+0000 7f51937fe640 1 --2- 172.21.15.57:0/2342471886 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f519406ffb0 0x7f51941446f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:30.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.544+0000 7f51937fe640 1 -- 172.21.15.57:0/2342471886 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5184009560 con 0x7f51940c9dc0 2023-10-19T14:18:30.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.544+0000 7f518bfff640 1 --2- 172.21.15.57:0/2342471886 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f51940b1920 0x7f5194153750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:18:30.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.544+0000 7f5192ffd640 1 --2- 172.21.15.57:0/2342471886 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f519406ffb0 0x7f51941446f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:18:30.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.544+0000 7f51937fe640 1 --2- 172.21.15.57:0/2342471886 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51940c9dc0 0x7f519406be50 secure :-1 s=READY pgs=2152 cs=0 l=1 rev1=1 crypto rx=0x7f518c00ea10 tx=0x7f518c00eed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:30.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.544+0000 7f5190ff9640 1 -- 172.21.15.57:0/2342471886 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f518c00cde0 con 0x7f51940c9dc0 2023-10-19T14:18:30.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.544+0000 7f5190ff9640 1 -- 172.21.15.57:0/2342471886 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f518c004c10 con 0x7f51940c9dc0 2023-10-19T14:18:30.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.544+0000 7f5190ff9640 1 -- 172.21.15.57:0/2342471886 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f518c005020 con 0x7f51940c9dc0 2023-10-19T14:18:30.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.544+0000 7f5199900640 1 -- 172.21.15.57:0/2342471886 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51941f5610 con 0x7f51940c9dc0 2023-10-19T14:18:30.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.545+0000 7f5199900640 1 -- 172.21.15.57:0/2342471886 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51940ae450 con 0x7f51940c9dc0 2023-10-19T14:18:30.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.545+0000 7f5199900640 1 -- 172.21.15.57:0/2342471886 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51941f5b30 con 0x7f51940c9dc0 2023-10-19T14:18:30.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.546+0000 7f5190ff9640 1 -- 172.21.15.57:0/2342471886 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 117) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f518c005180 con 0x7f51940c9dc0 2023-10-19T14:18:30.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.546+0000 7f5190ff9640 1 --2- 172.21.15.57:0/2342471886 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f517406af60 0x7f517406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:30.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.547+0000 7f5190ff9640 1 -- 172.21.15.57:0/2342471886 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 6351+0+0 (secure 0 0 0) 0x7f518c014070 con 0x7f51940c9dc0 2023-10-19T14:18:30.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.547+0000 7f5192ffd640 1 --2- 172.21.15.57:0/2342471886 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f517406af60 0x7f517406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:30.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.547+0000 7f5192ffd640 1 --2- 172.21.15.57:0/2342471886 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f517406af60 0x7f517406d420 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f5184035000 tx=0x7f5184004730 comp rx=0 tx=0).ready entity=mgr.8744 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:30.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.551+0000 7f5190ff9640 1 -- 172.21.15.57:0/2342471886 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f51941f5b30 con 0x7f51940c9dc0 2023-10-19T14:18:30.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.730+0000 7f5199900640 1 -- 172.21.15.57:0/2342471886 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f51941f5b30 con 0x7f51940c9dc0 2023-10-19T14:18:30.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.731+0000 7f5190ff9640 1 -- 172.21.15.57:0/2342471886 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v196) v1 ==== 74+0+14830 (secure 0 0 0) 0x7f51941f5b30 con 0x7f51940c9dc0 2023-10-19T14:18:30.734 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:18:30.734 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":196,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T14:18:29.296634+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.179:6837/3993214977":"2023-10-20T14:18:26.131465+0000","172.21.15.179:0/1944205818":"2023-10-20T14:18:23.856456+0000","172.21.15.179:0/530749640":"2023-10-20T14:18:23.856456+0000","172.21.15.179:0/3372019790":"2023-10-20T14:18:23.856456+0000","172.21.15.179:6832/4086068793":"2023-10-20T14:18:23.856456+0000","172.21.15.179:6835/3993214977":"2023-10-20T14:18:26.131465+0000","172.21.15.179:0/4245102210":"2023-10-20T14:18:23.856456+0000","172.21.15.179:0/1094895803":"2023-10-20T14:13:20.213663+0000","172.21.15.179:6832/2587699336":"2023-10-20T14:13:20.213663+0000","172.21.15.179:0/2295535972":"2023-10-20T14:13:20.213663+0000","172.21.15.179:6833/4086068793":"2023-10-20T14:18:23.856456+0000","172.21.15.179:0/1009979748":"2023-10-20T14:13:20.213663+0000","172.21.15.179:6833/2587699336":"2023-10-20T14:13:20.213663+0000","172.21.15.179:0/2961439032":"2023-10-20T14:13:20.213663+0000","172.21.15.179:0/1362323478":"2023-10-20T14:13:20.213663+0000","172.21.15.179:0/1220569576":"2023-10-20T14:18:23.856456+0000","172.21.15.179:0/2642945549":"2023-10-20T14:13:20.213663+0000","172.21.15.179:0/2377816178":"2023-10-20T14:18:23.856456+0000","172.21.15.179:0/1066702535":"2023-10-20T14:02:21.481635+0000","172.21.15.179:0/904747968":"2023-10-20T14:02:21.481635+0000","172.21.15.179:0/1902799271":"2023-10-20T14:02:21.481635+0000","172.21.15.179:0/2939688089":"2023-10-20T14:02:21.481635+0000","172.21.15.179:0/682180483":"2023-10-20T14:02:21.481635+0000","172.21.15.179:6833/1426553632":"2023-10-20T14:02:21.481635+0000","172.21.15.179:6832/1426553632":"2023-10-20T14:02:21.481635+0000","172.21.15.179:0/2321890842":"2023-10-20T14:02:21.481635+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[{"pool":24,"snaps":[{"begin":2,"length":32}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T14:18:30.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.732+0000 7f5199900640 1 -- 172.21.15.57:0/2342471886 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f517406af60 msgr2=0x7f517406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:30.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.732+0000 7f5199900640 1 --2- 172.21.15.57:0/2342471886 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f517406af60 0x7f517406d420 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f5184035000 tx=0x7f5184004730 comp rx=0 tx=0).stop 2023-10-19T14:18:30.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.732+0000 7f5199900640 1 -- 172.21.15.57:0/2342471886 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51940c9dc0 msgr2=0x7f519406be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:30.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.732+0000 7f5199900640 1 --2- 172.21.15.57:0/2342471886 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51940c9dc0 0x7f519406be50 secure :-1 s=READY pgs=2152 cs=0 l=1 rev1=1 crypto rx=0x7f518c00ea10 tx=0x7f518c00eed0 comp rx=0 tx=0).stop 2023-10-19T14:18:30.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.733+0000 7f5199900640 1 -- 172.21.15.57:0/2342471886 shutdown_connections 2023-10-19T14:18:30.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.733+0000 7f5199900640 1 --2- 172.21.15.57:0/2342471886 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f517406af60 0x7f517406d420 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:30.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.733+0000 7f5199900640 1 --2- 172.21.15.57:0/2342471886 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51940c9dc0 0x7f519406be50 unknown :-1 s=CLOSED pgs=2152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:30.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.733+0000 7f5199900640 1 --2- 172.21.15.57:0/2342471886 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f51940b1920 0x7f5194153750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:30.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.733+0000 7f5199900640 1 --2- 172.21.15.57:0/2342471886 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f519406ffb0 0x7f51941446f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:30.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.733+0000 7f5199900640 1 -- 172.21.15.57:0/2342471886 >> 172.21.15.57:0/2342471886 conn(0x7f5194069930 msgr2=0x7f51940c8d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:30.737 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.733+0000 7f5199900640 1 -- 172.21.15.57:0/2342471886 shutdown_connections 2023-10-19T14:18:30.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.733+0000 7f5199900640 1 -- 172.21.15.57:0/2342471886 wait complete. 2023-10-19T14:18:30.746 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-19T12:44:07.371186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-19T14:18:30.747 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-19T14:18:30.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.856+0000 7fcd81f8e640 1 -- 172.21.15.57:0/4196739926 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcd64005180 msgr2=0x7fcd6400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:30.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.856+0000 7fcd81f8e640 1 --2- 172.21.15.57:0/4196739926 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcd64005180 0x7fcd6400b980 secure :-1 s=READY pgs=2154 cs=0 l=1 rev1=1 crypto rx=0x7fcd74001d80 tx=0x7fcd640057f0 comp rx=0 tx=0).stop 2023-10-19T14:18:30.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.856+0000 7fcd81f8e640 1 -- 172.21.15.57:0/4196739926 shutdown_connections 2023-10-19T14:18:30.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.856+0000 7fcd81f8e640 1 --2- 172.21.15.57:0/4196739926 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcd64005180 0x7fcd6400b980 unknown :-1 s=CLOSED pgs=2154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:30.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.856+0000 7fcd81f8e640 1 -- 172.21.15.57:0/4196739926 >> 172.21.15.57:0/4196739926 conn(0x7fcd7c069930 msgr2=0x7fcd7c069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:30.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.856+0000 7fcd81f8e640 1 -- 172.21.15.57:0/4196739926 shutdown_connections 2023-10-19T14:18:30.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.856+0000 7fcd81f8e640 1 -- 172.21.15.57:0/4196739926 wait complete. 2023-10-19T14:18:30.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.857+0000 7fcd81f8e640 1 Processor -- start 2023-10-19T14:18:30.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.857+0000 7fcd81f8e640 1 -- start start 2023-10-19T14:18:30.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.858+0000 7fcd81f8e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcd64005180 0x7fcd7c138fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:30.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.858+0000 7fcd81f8e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcd7c0ad2b0 0x7fcd7c0aebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:30.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.858+0000 7fcd81f8e640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcd7c0af110 0x7fcd7c1eefa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:30.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.858+0000 7fcd81f8e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd7c0af590 con 0x7fcd7c0ad2b0 2023-10-19T14:18:30.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.858+0000 7fcd81f8e640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd7c151c50 con 0x7fcd64005180 2023-10-19T14:18:30.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.858+0000 7fcd81f8e640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd7c0b1740 con 0x7fcd7c0af110 2023-10-19T14:18:30.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.858+0000 7fcd7affd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcd7c0ad2b0 0x7fcd7c0aebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:30.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.858+0000 7fcd7affd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcd7c0ad2b0 0x7fcd7c0aebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51284/0 (socket says 172.21.15.57:51284) 2023-10-19T14:18:30.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.858+0000 7fcd7affd640 1 -- 172.21.15.57:0/2000528377 learned_addr learned my addr 172.21.15.57:0/2000528377 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:18:30.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.858+0000 7fcd7bfff640 1 --2- 172.21.15.57:0/2000528377 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcd7c0af110 0x7fcd7c1eefa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:30.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.858+0000 7fcd7b7fe640 1 --2- 172.21.15.57:0/2000528377 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcd64005180 0x7fcd7c138fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:30.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.858+0000 7fcd7affd640 1 -- 172.21.15.57:0/2000528377 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcd7c0af110 msgr2=0x7fcd7c1eefa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:30.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.858+0000 7fcd7affd640 1 --2- 172.21.15.57:0/2000528377 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcd7c0af110 0x7fcd7c1eefa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:30.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.858+0000 7fcd7affd640 1 -- 172.21.15.57:0/2000528377 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcd64005180 msgr2=0x7fcd7c138fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:30.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.858+0000 7fcd7affd640 1 --2- 172.21.15.57:0/2000528377 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcd64005180 0x7fcd7c138fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:30.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.858+0000 7fcd7affd640 1 -- 172.21.15.57:0/2000528377 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd7c151ef0 con 0x7fcd7c0ad2b0 2023-10-19T14:18:30.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.858+0000 7fcd7affd640 1 --2- 172.21.15.57:0/2000528377 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcd7c0ad2b0 0x7fcd7c0aebd0 secure :-1 s=READY pgs=2155 cs=0 l=1 rev1=1 crypto rx=0x7fcd7000e510 tx=0x7fcd7000e9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:30.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.859+0000 7fcd63fff640 1 -- 172.21.15.57:0/2000528377 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd70010890 con 0x7fcd7c0ad2b0 2023-10-19T14:18:30.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.859+0000 7fcd63fff640 1 -- 172.21.15.57:0/2000528377 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fcd70010ae0 con 0x7fcd7c0ad2b0 2023-10-19T14:18:30.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.859+0000 7fcd63fff640 1 -- 172.21.15.57:0/2000528377 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd70011460 con 0x7fcd7c0ad2b0 2023-10-19T14:18:30.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.859+0000 7fcd81f8e640 1 -- 172.21.15.57:0/2000528377 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd7c1521d0 con 0x7fcd7c0ad2b0 2023-10-19T14:18:30.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.859+0000 7fcd81f8e640 1 -- 172.21.15.57:0/2000528377 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd7c0af930 con 0x7fcd7c0ad2b0 2023-10-19T14:18:30.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.860+0000 7fcd81f8e640 1 -- 172.21.15.57:0/2000528377 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd7c1526f0 con 0x7fcd7c0ad2b0 2023-10-19T14:18:30.868 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.860+0000 7fcd63fff640 1 -- 172.21.15.57:0/2000528377 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 117) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fcd700115c0 con 0x7fcd7c0ad2b0 2023-10-19T14:18:30.868 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.861+0000 7fcd63fff640 1 --2- 172.21.15.57:0/2000528377 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fcd6806af90 0x7fcd6806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:30.868 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.861+0000 7fcd63fff640 1 -- 172.21.15.57:0/2000528377 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 6351+0+0 (secure 0 0 0) 0x7fcd7008a2e0 con 0x7fcd7c0ad2b0 2023-10-19T14:18:30.868 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.861+0000 7fcd7b7fe640 1 --2- 172.21.15.57:0/2000528377 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fcd6806af90 0x7fcd6806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:30.869 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.861+0000 7fcd7b7fe640 1 --2- 172.21.15.57:0/2000528377 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fcd6806af90 0x7fcd6806d450 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fcd7c153050 tx=0x7fcd6c005f90 comp rx=0 tx=0).ready entity=mgr.8744 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:30.869 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:30.865+0000 7fcd63fff640 1 -- 172.21.15.57:0/2000528377 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fcd7c1526f0 con 0x7fcd7c0ad2b0 2023-10-19T14:18:31.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:31.040+0000 7fcd81f8e640 1 -- 172.21.15.57:0/2000528377 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fcd7c1526f0 con 0x7fcd7c0ad2b0 2023-10-19T14:18:31.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:31.041+0000 7fcd63fff640 1 -- 172.21.15.57:0/2000528377 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v196) v1 ==== 93+0+10 (secure 0 0 0) 0x7fcd7c1526f0 con 0x7fcd7c0ad2b0 2023-10-19T14:18:31.044 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-19T14:18:31.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:31.042+0000 7fcd81f8e640 1 -- 172.21.15.57:0/2000528377 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fcd6806af90 msgr2=0x7fcd6806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:31.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:31.042+0000 7fcd81f8e640 1 --2- 172.21.15.57:0/2000528377 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fcd6806af90 0x7fcd6806d450 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fcd7c153050 tx=0x7fcd6c005f90 comp rx=0 tx=0).stop 2023-10-19T14:18:31.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:31.042+0000 7fcd81f8e640 1 -- 172.21.15.57:0/2000528377 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcd7c0ad2b0 msgr2=0x7fcd7c0aebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:31.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:31.042+0000 7fcd81f8e640 1 --2- 172.21.15.57:0/2000528377 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcd7c0ad2b0 0x7fcd7c0aebd0 secure :-1 s=READY pgs=2155 cs=0 l=1 rev1=1 crypto rx=0x7fcd7000e510 tx=0x7fcd7000e9d0 comp rx=0 tx=0).stop 2023-10-19T14:18:31.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:31.043+0000 7fcd81f8e640 1 -- 172.21.15.57:0/2000528377 shutdown_connections 2023-10-19T14:18:31.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:31.043+0000 7fcd81f8e640 1 --2- 172.21.15.57:0/2000528377 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fcd6806af90 0x7fcd6806d450 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:31.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:31.043+0000 7fcd81f8e640 1 --2- 172.21.15.57:0/2000528377 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcd7c0af110 0x7fcd7c1eefa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:31.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:31.043+0000 7fcd81f8e640 1 --2- 172.21.15.57:0/2000528377 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcd7c0ad2b0 0x7fcd7c0aebd0 unknown :-1 s=CLOSED pgs=2155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:31.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:31.043+0000 7fcd81f8e640 1 --2- 172.21.15.57:0/2000528377 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcd64005180 0x7fcd7c138fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:31.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:31.043+0000 7fcd81f8e640 1 -- 172.21.15.57:0/2000528377 >> 172.21.15.57:0/2000528377 conn(0x7fcd7c069930 msgr2=0x7fcd7c13b130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:31.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:31.043+0000 7fcd81f8e640 1 -- 172.21.15.57:0/2000528377 shutdown_connections 2023-10-19T14:18:31.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:31.043+0000 7fcd81f8e640 1 -- 172.21.15.57:0/2000528377 wait complete. 2023-10-19T14:18:31.056 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention' 2023-10-19T14:18:31.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:31.163+0000 7fa74e9f5640 1 -- 172.21.15.57:0/3660323276 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa734003970 msgr2=0x7fa734003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:31.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:31.163+0000 7fa74e9f5640 1 --2- 172.21.15.57:0/3660323276 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa734003970 0x7fa734003d90 secure :-1 s=READY pgs=2157 cs=0 l=1 rev1=1 crypto rx=0x7fa73000c260 tx=0x7fa7340057f0 comp rx=0 tx=0).stop 2023-10-19T14:18:31.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:31.163+0000 7fa74cf92640 1 -- 172.21.15.57:0/3660323276 reap_dead start 2023-10-19T14:18:31.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:31.163+0000 7fa74e9f5640 1 -- 172.21.15.57:0/3660323276 shutdown_connections 2023-10-19T14:18:31.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:31.163+0000 7fa74e9f5640 1 -- 172.21.15.57:0/3660323276 >> 172.21.15.57:0/3660323276 conn(0x7fa74809a030 msgr2=0x7fa7480d0d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:31.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:31.163+0000 7fa74e9f5640 1 -- 172.21.15.57:0/3660323276 shutdown_connections 2023-10-19T14:18:31.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:31.163+0000 7fa74e9f5640 1 -- 172.21.15.57:0/3660323276 wait complete. 2023-10-19T14:18:31.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:31.164+0000 7fa74e9f5640 1 Processor -- start 2023-10-19T14:18:31.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:31.164+0000 7fa74e9f5640 1 -- start start 2023-10-19T14:18:31.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:31.165+0000 7fa74e9f5640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa734003970 0x7fa74814fe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:31.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:31.165+0000 7fa74e9f5640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa7480c9900 0x7fa748153350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:31.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:31.165+0000 7fa74e9f5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa7480c9d80 0x7fa748153890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:31.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:31.165+0000 7fa74e9f5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7481f2540 con 0x7fa7480c9d80 2023-10-19T14:18:31.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:31.165+0000 7fa74e9f5640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7481f18c0 con 0x7fa7480c9900 2023-10-19T14:18:31.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:31.165+0000 7fa74e9f5640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7481f1ba0 con 0x7fa734003970 2023-10-19T14:18:31.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:31.165+0000 7fa74cf92640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa7480c9d80 0x7fa748153890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:31.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:31.165+0000 7fa74cf92640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa7480c9d80 0x7fa748153890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51302/0 (socket says 172.21.15.57:51302) 2023-10-19T14:18:31.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:31.165+0000 7fa74cf92640 1 -- 172.21.15.57:0/4268460297 learned_addr learned my addr 172.21.15.57:0/4268460297 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:18:31.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:31.165+0000 7fa747fff640 1 --2- 172.21.15.57:0/4268460297 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa734003970 0x7fa74814fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:31.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:31.165+0000 7fa7477fe640 1 --2- 172.21.15.57:0/4268460297 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa7480c9900 0x7fa748153350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:31.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:31.165+0000 7fa74cf92640 1 -- 172.21.15.57:0/4268460297 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa734003970 msgr2=0x7fa74814fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:31.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:31.165+0000 7fa74cf92640 1 --2- 172.21.15.57:0/4268460297 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa734003970 0x7fa74814fe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:31.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:31.165+0000 7fa74cf92640 1 -- 172.21.15.57:0/4268460297 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa7480c9900 msgr2=0x7fa748153350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:31.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:31.165+0000 7fa74cf92640 1 --2- 172.21.15.57:0/4268460297 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa7480c9900 0x7fa748153350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:31.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:31.165+0000 7fa74cf92640 1 -- 172.21.15.57:0/4268460297 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7481f1e80 con 0x7fa7480c9d80 2023-10-19T14:18:31.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:31.165+0000 7fa747fff640 1 --2- 172.21.15.57:0/4268460297 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa734003970 0x7fa74814fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:18:31.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:31.165+0000 7fa74cf92640 1 --2- 172.21.15.57:0/4268460297 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa7480c9d80 0x7fa748153890 secure :-1 s=READY pgs=2158 cs=0 l=1 rev1=1 crypto rx=0x7fa7480cfef0 tx=0x7fa74000f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:31.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:31.165+0000 7fa744ff9640 1 -- 172.21.15.57:0/4268460297 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa740003d80 con 0x7fa7480c9d80 2023-10-19T14:18:31.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:31.166+0000 7fa744ff9640 1 -- 172.21.15.57:0/4268460297 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fa740004b20 con 0x7fa7480c9d80 2023-10-19T14:18:31.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:31.166+0000 7fa74e9f5640 1 -- 172.21.15.57:0/4268460297 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7481f2160 con 0x7fa7480c9d80 2023-10-19T14:18:31.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:31.166+0000 7fa744ff9640 1 -- 172.21.15.57:0/4268460297 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa740004030 con 0x7fa7480c9d80 2023-10-19T14:18:31.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:31.166+0000 7fa74e9f5640 1 -- 172.21.15.57:0/4268460297 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7481f1520 con 0x7fa7480c9d80 2023-10-19T14:18:31.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:31.167+0000 7fa74e9f5640 1 -- 172.21.15.57:0/4268460297 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7480675f0 con 0x7fa7480c9d80 2023-10-19T14:18:31.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:31.167+0000 7fa744ff9640 1 -- 172.21.15.57:0/4268460297 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 117) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fa740004190 con 0x7fa7480c9d80 2023-10-19T14:18:31.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:31.167+0000 7fa744ff9640 1 --2- 172.21.15.57:0/4268460297 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fa72806af60 0x7fa72806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:31.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:31.168+0000 7fa744ff9640 1 -- 172.21.15.57:0/4268460297 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 6351+0+0 (secure 0 0 0) 0x7fa74008a9e0 con 0x7fa7480c9d80 2023-10-19T14:18:31.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:31.168+0000 7fa747fff640 1 --2- 172.21.15.57:0/4268460297 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fa72806af60 0x7fa72806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:31.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:31.168+0000 7fa747fff640 1 --2- 172.21.15.57:0/4268460297 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fa72806af60 0x7fa72806d420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fa730034000 tx=0x7fa730001a00 comp rx=0 tx=0).ready entity=mgr.8744 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:31.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:31.172+0000 7fa744ff9640 1 -- 172.21.15.57:0/4268460297 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa7400555e0 con 0x7fa7480c9d80 2023-10-19T14:18:31.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:31.422+0000 7fa74e9f5640 1 -- 172.21.15.57:0/4268460297 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention"]} v 0) v1 -- 0x7fa7480b13a0 con 0x7fa7480c9d80 2023-10-19T14:18:32.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.431+0000 7fa744ff9640 1 -- 172.21.15.57:0/4268460297 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention"]}]=0 v3053) v1 ==== 166+0+0 (secure 0 0 0) 0x7fa740058c00 con 0x7fa7480c9d80 2023-10-19T14:18:32.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.434+0000 7fa74e9f5640 1 -- 172.21.15.57:0/4268460297 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fa72806af60 msgr2=0x7fa72806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:32.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.434+0000 7fa74e9f5640 1 --2- 172.21.15.57:0/4268460297 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fa72806af60 0x7fa72806d420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fa730034000 tx=0x7fa730001a00 comp rx=0 tx=0).stop 2023-10-19T14:18:32.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.434+0000 7fa74e9f5640 1 -- 172.21.15.57:0/4268460297 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa7480c9d80 msgr2=0x7fa748153890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:32.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.434+0000 7fa74e9f5640 1 --2- 172.21.15.57:0/4268460297 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa7480c9d80 0x7fa748153890 secure :-1 s=READY pgs=2158 cs=0 l=1 rev1=1 crypto rx=0x7fa7480cfef0 tx=0x7fa74000f790 comp rx=0 tx=0).stop 2023-10-19T14:18:32.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.434+0000 7fa74e9f5640 1 -- 172.21.15.57:0/4268460297 shutdown_connections 2023-10-19T14:18:32.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.434+0000 7fa74e9f5640 1 --2- 172.21.15.57:0/4268460297 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fa72806af60 0x7fa72806d420 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:32.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.434+0000 7fa74e9f5640 1 --2- 172.21.15.57:0/4268460297 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa7480c9d80 0x7fa748153890 unknown :-1 s=CLOSED pgs=2158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:32.439 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.434+0000 7fa74e9f5640 1 --2- 172.21.15.57:0/4268460297 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa7480c9900 0x7fa748153350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:32.439 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.434+0000 7fa74e9f5640 1 --2- 172.21.15.57:0/4268460297 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa734003970 0x7fa74814fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:32.439 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.434+0000 7fa74e9f5640 1 -- 172.21.15.57:0/4268460297 >> 172.21.15.57:0/4268460297 conn(0x7fa74809a030 msgr2=0x7fa7480af150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:32.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.434+0000 7fa74e9f5640 1 -- 172.21.15.57:0/4268460297 shutdown_connections 2023-10-19T14:18:32.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.434+0000 7fa74e9f5640 1 -- 172.21.15.57:0/4268460297 wait complete. 2023-10-19T14:18:32.451 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2023-10-19T14:18:32.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.564+0000 7fcff92a2640 1 -- 172.21.15.57:0/334460551 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcff4070920 msgr2=0x7fcff40affe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:32.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.564+0000 7fcff92a2640 1 --2- 172.21.15.57:0/334460551 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcff4070920 0x7fcff40affe0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fcfec005fd0 tx=0x7fcfec02c050 comp rx=0 tx=0).stop 2023-10-19T14:18:32.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.564+0000 7fcff92a2640 1 -- 172.21.15.57:0/334460551 shutdown_connections 2023-10-19T14:18:32.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.564+0000 7fcff92a2640 1 --2- 172.21.15.57:0/334460551 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcff4070920 0x7fcff40affe0 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:32.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.564+0000 7fcff92a2640 1 --2- 172.21.15.57:0/334460551 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcff4141690 0x7fcff40703e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:32.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.564+0000 7fcff92a2640 1 -- 172.21.15.57:0/334460551 >> 172.21.15.57:0/334460551 conn(0x7fcff4069930 msgr2=0x7fcff4069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:32.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.564+0000 7fcff92a2640 1 -- 172.21.15.57:0/334460551 shutdown_connections 2023-10-19T14:18:32.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.564+0000 7fcff92a2640 1 -- 172.21.15.57:0/334460551 wait complete. 2023-10-19T14:18:32.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.565+0000 7fcff92a2640 1 Processor -- start 2023-10-19T14:18:32.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.565+0000 7fcff92a2640 1 -- start start 2023-10-19T14:18:32.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.566+0000 7fcff92a2640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcff4070920 0x7fcff41e1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:32.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.566+0000 7fcff92a2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcff413e520 0x7fcff413a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:32.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.566+0000 7fcff92a2640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcff4141690 0x7fcff41dcec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:32.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.566+0000 7fcff92a2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcff413ac50 con 0x7fcff413e520 2023-10-19T14:18:32.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.566+0000 7fcff92a2640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcff41e0f00 con 0x7fcff4070920 2023-10-19T14:18:32.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.566+0000 7fcff92a2640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcff41e11e0 con 0x7fcff4141690 2023-10-19T14:18:32.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.566+0000 7fcfea7fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcff413e520 0x7fcff413a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:32.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.566+0000 7fcfea7fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcff413e520 0x7fcff413a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51318/0 (socket says 172.21.15.57:51318) 2023-10-19T14:18:32.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.566+0000 7fcfea7fc640 1 -- 172.21.15.57:0/3399506367 learned_addr learned my addr 172.21.15.57:0/3399506367 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:18:32.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.566+0000 7fcff2ffd640 1 --2- 172.21.15.57:0/3399506367 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcff4070920 0x7fcff41e1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:32.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.567+0000 7fcff37fe640 1 --2- 172.21.15.57:0/3399506367 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcff4141690 0x7fcff41dcec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:32.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.567+0000 7fcfea7fc640 1 -- 172.21.15.57:0/3399506367 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcff4141690 msgr2=0x7fcff41dcec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:32.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.567+0000 7fcfea7fc640 1 --2- 172.21.15.57:0/3399506367 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcff4141690 0x7fcff41dcec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:32.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.567+0000 7fcfea7fc640 1 -- 172.21.15.57:0/3399506367 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcff4070920 msgr2=0x7fcff41e1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:32.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.567+0000 7fcfea7fc640 1 --2- 172.21.15.57:0/3399506367 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcff4070920 0x7fcff41e1fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:32.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.567+0000 7fcfea7fc640 1 -- 172.21.15.57:0/3399506367 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfdc009490 con 0x7fcff413e520 2023-10-19T14:18:32.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.567+0000 7fcff2ffd640 1 --2- 172.21.15.57:0/3399506367 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcff4070920 0x7fcff41e1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:18:32.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.567+0000 7fcfea7fc640 1 --2- 172.21.15.57:0/3399506367 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcff413e520 0x7fcff413a710 secure :-1 s=READY pgs=2159 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc00e9f0 tx=0x7fcfdc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:32.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.567+0000 7fcfebfff640 1 -- 172.21.15.57:0/3399506367 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfdc00cd80 con 0x7fcff413e520 2023-10-19T14:18:32.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.567+0000 7fcfebfff640 1 -- 172.21.15.57:0/3399506367 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fcfdc004020 con 0x7fcff413e520 2023-10-19T14:18:32.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.567+0000 7fcff92a2640 1 -- 172.21.15.57:0/3399506367 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfec009840 con 0x7fcff413e520 2023-10-19T14:18:32.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.567+0000 7fcfebfff640 1 -- 172.21.15.57:0/3399506367 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfdc00cee0 con 0x7fcff413e520 2023-10-19T14:18:32.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.567+0000 7fcff92a2640 1 -- 172.21.15.57:0/3399506367 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcff41e17a0 con 0x7fcff413e520 2023-10-19T14:18:32.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.568+0000 7fcff92a2640 1 -- 172.21.15.57:0/3399506367 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcff40690e0 con 0x7fcff413e520 2023-10-19T14:18:32.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.569+0000 7fcfebfff640 1 -- 172.21.15.57:0/3399506367 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 117) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fcfdc0035f0 con 0x7fcff413e520 2023-10-19T14:18:32.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.569+0000 7fcfebfff640 1 --2- 172.21.15.57:0/3399506367 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fcfd406af90 0x7fcfd406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:32.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.569+0000 7fcfebfff640 1 -- 172.21.15.57:0/3399506367 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 6351+0+0 (secure 0 0 0) 0x7fcfdc014070 con 0x7fcff413e520 2023-10-19T14:18:32.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.569+0000 7fcff2ffd640 1 --2- 172.21.15.57:0/3399506367 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fcfd406af90 0x7fcfd406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:32.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.570+0000 7fcff2ffd640 1 --2- 172.21.15.57:0/3399506367 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fcfd406af90 0x7fcfd406d450 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4001710 tx=0x7fcfe4009040 comp rx=0 tx=0).ready entity=mgr.8744 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:32.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.573+0000 7fcfebfff640 1 -- 172.21.15.57:0/3399506367 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fcfdc055270 con 0x7fcff413e520 2023-10-19T14:18:32.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.749+0000 7fcff92a2640 1 -- 172.21.15.57:0/3399506367 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "config dump"} v 0) v1 -- 0x7fcff406bb60 con 0x7fcff413e520 2023-10-19T14:18:32.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.749+0000 7fcfebfff640 1 -- 172.21.15.57:0/3399506367 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v70) v1 ==== 59+0+2220 (secure 0 0 0) 0x7fcfdc058890 con 0x7fcff413e520 2023-10-19T14:18:32.753 INFO:teuthology.orchestra.run.smithi057.stdout:WHO MASK LEVEL OPTION VALUE RO 2023-10-19T14:18:32.753 INFO:teuthology.orchestra.run.smithi057.stdout:mon advanced mon_op_complaint_time 120 2023-10-19T14:18:32.754 INFO:teuthology.orchestra.run.smithi057.stdout:osd advanced osd_op_complaint_time 180.000000 2023-10-19T14:18:32.754 INFO:teuthology.orchestra.run.smithi057.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 4775.995373 2023-10-19T14:18:32.754 INFO:teuthology.orchestra.run.smithi057.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 4208.118410 2023-10-19T14:18:32.754 INFO:teuthology.orchestra.run.smithi057.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 4541.123030 2023-10-19T14:18:32.755 INFO:teuthology.orchestra.run.smithi057.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 4877.592206 2023-10-19T14:18:32.755 INFO:teuthology.orchestra.run.smithi057.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 5019.748032 2023-10-19T14:18:32.755 INFO:teuthology.orchestra.run.smithi057.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 5000.074376 2023-10-19T14:18:32.755 INFO:teuthology.orchestra.run.smithi057.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 5153.479439 2023-10-19T14:18:32.755 INFO:teuthology.orchestra.run.smithi057.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 4977.363845 2023-10-19T14:18:32.755 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced debug_mds 20/20 2023-10-19T14:18:32.756 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced debug_mds_balancer 20/20 2023-10-19T14:18:32.756 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced debug_ms 1/1 2023-10-19T14:18:32.756 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced mds_bal_fragment_size_max 10000 2023-10-19T14:18:32.756 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced mds_bal_merge_size 5 2023-10-19T14:18:32.756 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced mds_bal_split_bits 3 2023-10-19T14:18:32.757 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced mds_bal_split_size 100 2023-10-19T14:18:32.757 INFO:teuthology.orchestra.run.smithi057.stdout:mds dev mds_debug_frag true 2023-10-19T14:18:32.757 INFO:teuthology.orchestra.run.smithi057.stdout:mds dev mds_debug_scatterstat true 2023-10-19T14:18:32.757 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced mds_op_complaint_time 180.000000 2023-10-19T14:18:32.757 INFO:teuthology.orchestra.run.smithi057.stdout:mds dev mds_verify_scatter true 2023-10-19T14:18:32.757 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced osd_op_complaint_time 180.000000 2023-10-19T14:18:32.758 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced rados_mon_op_timeout 900 2023-10-19T14:18:32.758 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced rados_osd_op_timeout 900 2023-10-19T14:18:32.758 INFO:teuthology.orchestra.run.smithi057.stdout:client advanced client_mount_timeout 600 2023-10-19T14:18:32.758 INFO:teuthology.orchestra.run.smithi057.stdout:client advanced debug_client 20/20 2023-10-19T14:18:32.758 INFO:teuthology.orchestra.run.smithi057.stdout:client advanced debug_ms 1/1 2023-10-19T14:18:32.759 INFO:teuthology.orchestra.run.smithi057.stdout:client advanced rados_mon_op_timeout 900 2023-10-19T14:18:32.759 INFO:teuthology.orchestra.run.smithi057.stdout:client advanced rados_osd_op_timeout 900 2023-10-19T14:18:32.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.751+0000 7fcff92a2640 1 -- 172.21.15.57:0/3399506367 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fcfd406af90 msgr2=0x7fcfd406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:32.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.751+0000 7fcff92a2640 1 --2- 172.21.15.57:0/3399506367 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fcfd406af90 0x7fcfd406d450 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4001710 tx=0x7fcfe4009040 comp rx=0 tx=0).stop 2023-10-19T14:18:32.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.751+0000 7fcff92a2640 1 -- 172.21.15.57:0/3399506367 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcff413e520 msgr2=0x7fcff413a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:32.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.751+0000 7fcff92a2640 1 --2- 172.21.15.57:0/3399506367 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcff413e520 0x7fcff413a710 secure :-1 s=READY pgs=2159 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc00e9f0 tx=0x7fcfdc00eeb0 comp rx=0 tx=0).stop 2023-10-19T14:18:32.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.751+0000 7fcff92a2640 1 -- 172.21.15.57:0/3399506367 shutdown_connections 2023-10-19T14:18:32.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.751+0000 7fcff92a2640 1 --2- 172.21.15.57:0/3399506367 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fcfd406af90 0x7fcfd406d450 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:32.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.751+0000 7fcff92a2640 1 --2- 172.21.15.57:0/3399506367 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcff4141690 0x7fcff41dcec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:32.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.751+0000 7fcff92a2640 1 --2- 172.21.15.57:0/3399506367 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcff413e520 0x7fcff413a710 unknown :-1 s=CLOSED pgs=2159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:32.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.751+0000 7fcff92a2640 1 --2- 172.21.15.57:0/3399506367 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcff4070920 0x7fcff41e1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:32.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.751+0000 7fcff92a2640 1 -- 172.21.15.57:0/3399506367 >> 172.21.15.57:0/3399506367 conn(0x7fcff4069930 msgr2=0x7fcff41df760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:32.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.751+0000 7fcff92a2640 1 -- 172.21.15.57:0/3399506367 shutdown_connections 2023-10-19T14:18:32.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.751+0000 7fcff92a2640 1 -- 172.21.15.57:0/3399506367 wait complete. 2023-10-19T14:18:32.764 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-19T14:18:32.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.873+0000 7fcf86924640 1 -- 172.21.15.57:0/3986272128 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcf68003970 msgr2=0x7fcf68003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:32.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.873+0000 7fcf86924640 1 --2- 172.21.15.57:0/3986272128 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcf68003970 0x7fcf68003d90 secure :-1 s=READY pgs=2161 cs=0 l=1 rev1=1 crypto rx=0x7fcf74009b50 tx=0x7fcf680057f0 comp rx=0 tx=0).stop 2023-10-19T14:18:32.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.874+0000 7fcf86924640 1 -- 172.21.15.57:0/3986272128 shutdown_connections 2023-10-19T14:18:32.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.874+0000 7fcf86924640 1 --2- 172.21.15.57:0/3986272128 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcf68003970 0x7fcf68003d90 unknown :-1 s=CLOSED pgs=2161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:32.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.874+0000 7fcf86924640 1 -- 172.21.15.57:0/3986272128 >> 172.21.15.57:0/3986272128 conn(0x7fcf80069930 msgr2=0x7fcf80069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:32.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.874+0000 7fcf86924640 1 -- 172.21.15.57:0/3986272128 shutdown_connections 2023-10-19T14:18:32.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.874+0000 7fcf86924640 1 -- 172.21.15.57:0/3986272128 wait complete. 2023-10-19T14:18:32.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.875+0000 7fcf86924640 1 Processor -- start 2023-10-19T14:18:32.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.875+0000 7fcf86924640 1 -- start start 2023-10-19T14:18:32.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.876+0000 7fcf86924640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcf68003970 0x7fcf8013ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:32.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.876+0000 7fcf86924640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcf800af3e0 0x7fcf800b1380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:32.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.876+0000 7fcf86924640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcf801ef620 0x7fcf801efab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:32.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.876+0000 7fcf86924640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf800af8a0 con 0x7fcf801ef620 2023-10-19T14:18:32.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.876+0000 7fcf86924640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf800afb80 con 0x7fcf800af3e0 2023-10-19T14:18:32.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.876+0000 7fcf86924640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf800afe60 con 0x7fcf68003970 2023-10-19T14:18:32.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.876+0000 7fcf84ec1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcf801ef620 0x7fcf801efab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:32.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.876+0000 7fcf84ec1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcf801ef620 0x7fcf801efab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51330/0 (socket says 172.21.15.57:51330) 2023-10-19T14:18:32.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.876+0000 7fcf84ec1640 1 -- 172.21.15.57:0/2030555558 learned_addr learned my addr 172.21.15.57:0/2030555558 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:18:32.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.876+0000 7fcf84ec1640 1 -- 172.21.15.57:0/2030555558 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcf68003970 msgr2=0x7fcf8013ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:32.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.876+0000 7fcf84ec1640 1 --2- 172.21.15.57:0/2030555558 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcf68003970 0x7fcf8013ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:32.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.876+0000 7fcf84ec1640 1 -- 172.21.15.57:0/2030555558 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcf800af3e0 msgr2=0x7fcf800b1380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:32.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.876+0000 7fcf7f7fe640 1 --2- 172.21.15.57:0/2030555558 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcf800af3e0 0x7fcf800b1380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:32.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.876+0000 7fcf84ec1640 1 --2- 172.21.15.57:0/2030555558 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcf800af3e0 0x7fcf800b1380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:32.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.876+0000 7fcf84ec1640 1 -- 172.21.15.57:0/2030555558 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf800b0140 con 0x7fcf801ef620 2023-10-19T14:18:32.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.876+0000 7fcf7f7fe640 1 --2- 172.21.15.57:0/2030555558 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcf800af3e0 0x7fcf800b1380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:18:32.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.876+0000 7fcf84ec1640 1 --2- 172.21.15.57:0/2030555558 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcf801ef620 0x7fcf801efab0 secure :-1 s=READY pgs=2162 cs=0 l=1 rev1=1 crypto rx=0x7fcf800cc150 tx=0x7fcf7800eb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:32.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.876+0000 7fcf7cff9640 1 -- 172.21.15.57:0/2030555558 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf7800ca10 con 0x7fcf801ef620 2023-10-19T14:18:32.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.876+0000 7fcf7cff9640 1 -- 172.21.15.57:0/2030555558 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fcf7800cc80 con 0x7fcf801ef620 2023-10-19T14:18:32.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.876+0000 7fcf86924640 1 -- 172.21.15.57:0/2030555558 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf800b0420 con 0x7fcf801ef620 2023-10-19T14:18:32.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.876+0000 7fcf7cff9640 1 -- 172.21.15.57:0/2030555558 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf78011460 con 0x7fcf801ef620 2023-10-19T14:18:32.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.877+0000 7fcf86924640 1 -- 172.21.15.57:0/2030555558 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf8013b2a0 con 0x7fcf801ef620 2023-10-19T14:18:32.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.878+0000 7fcf86924640 1 -- 172.21.15.57:0/2030555558 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf8006c220 con 0x7fcf801ef620 2023-10-19T14:18:32.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.878+0000 7fcf7cff9640 1 -- 172.21.15.57:0/2030555558 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 117) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fcf780115c0 con 0x7fcf801ef620 2023-10-19T14:18:32.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.878+0000 7fcf7cff9640 1 --2- 172.21.15.57:0/2030555558 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fcf6c06af90 0x7fcf6c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:32.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.879+0000 7fcf7cff9640 1 -- 172.21.15.57:0/2030555558 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 6351+0+0 (secure 0 0 0) 0x7fcf7808a030 con 0x7fcf801ef620 2023-10-19T14:18:32.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.879+0000 7fcf7ffff640 1 --2- 172.21.15.57:0/2030555558 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fcf6c06af90 0x7fcf6c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:32.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.879+0000 7fcf7ffff640 1 --2- 172.21.15.57:0/2030555558 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fcf6c06af90 0x7fcf6c06d450 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fcf74036000 tx=0x7fcf74035040 comp rx=0 tx=0).ready entity=mgr.8744 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:32.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:32.883+0000 7fcf7cff9640 1 -- 172.21.15.57:0/2030555558 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fcf78054c40 con 0x7fcf801ef620 2023-10-19T14:18:33.062 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.059+0000 7fcf86924640 1 -- 172.21.15.57:0/2030555558 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fcf8013b2a0 con 0x7fcf801ef620 2023-10-19T14:18:33.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.064+0000 7fcf7cff9640 1 -- 172.21.15.57:0/2030555558 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v71)=0 v71) v1 ==== 122+0+0 (secure 0 0 0) 0x7fcf8013b2a0 con 0x7fcf801ef620 2023-10-19T14:18:33.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.066+0000 7fcf86924640 1 -- 172.21.15.57:0/2030555558 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fcf6c06af90 msgr2=0x7fcf6c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:33.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.066+0000 7fcf86924640 1 --2- 172.21.15.57:0/2030555558 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fcf6c06af90 0x7fcf6c06d450 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fcf74036000 tx=0x7fcf74035040 comp rx=0 tx=0).stop 2023-10-19T14:18:33.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.066+0000 7fcf86924640 1 -- 172.21.15.57:0/2030555558 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcf801ef620 msgr2=0x7fcf801efab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:33.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.066+0000 7fcf86924640 1 --2- 172.21.15.57:0/2030555558 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcf801ef620 0x7fcf801efab0 secure :-1 s=READY pgs=2162 cs=0 l=1 rev1=1 crypto rx=0x7fcf800cc150 tx=0x7fcf7800eb60 comp rx=0 tx=0).stop 2023-10-19T14:18:33.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.066+0000 7fcf86924640 1 -- 172.21.15.57:0/2030555558 shutdown_connections 2023-10-19T14:18:33.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.066+0000 7fcf86924640 1 --2- 172.21.15.57:0/2030555558 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fcf6c06af90 0x7fcf6c06d450 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:33.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.066+0000 7fcf86924640 1 --2- 172.21.15.57:0/2030555558 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcf801ef620 0x7fcf801efab0 unknown :-1 s=CLOSED pgs=2162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:33.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.066+0000 7fcf86924640 1 --2- 172.21.15.57:0/2030555558 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcf800af3e0 0x7fcf800b1380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:33.072 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.066+0000 7fcf86924640 1 --2- 172.21.15.57:0/2030555558 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcf68003970 0x7fcf8013ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:33.072 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.066+0000 7fcf86924640 1 -- 172.21.15.57:0/2030555558 >> 172.21.15.57:0/2030555558 conn(0x7fcf80069930 msgr2=0x7fcf8015b340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:33.072 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.066+0000 7fcf86924640 1 -- 172.21.15.57:0/2030555558 shutdown_connections 2023-10-19T14:18:33.073 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.066+0000 7fcf86924640 1 -- 172.21.15.57:0/2030555558 wait complete. 2023-10-19T14:18:33.079 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-19T14:18:33.080 DEBUG:teuthology.orchestra.run.smithi057:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-19T14:18:33.135 DEBUG:teuthology.orchestra.run.smithi179:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-19T14:18:33.181 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-19T14:18:33.181 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T14:18:33.182 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T14:18:33.197 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055614,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=358 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248260k,nr_inodes=812065,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-19T14:18:33.198 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-19T14:18:33.198 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T14:18:33.216 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054833,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=284 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3247632k,nr_inodes=811908,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-19T14:18:33.216 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T14:18:33.217 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T14:18:33.252 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055614,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=358 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248260k,nr_inodes=812065,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-19T14:18:33.253 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-19T14:18:33.253 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T14:18:33.273 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054833,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=284 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3247632k,nr_inodes=811908,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-19T14:18:33.274 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-19T14:18:33.274 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T14:18:33.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.399+0000 7f813fbf0640 1 -- 172.21.15.57:0/2711372745 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8138157220 msgr2=0x7f81380ca870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:33.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.399+0000 7f813fbf0640 1 --2- 172.21.15.57:0/2711372745 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8138157220 0x7f81380ca870 secure :-1 s=READY pgs=2163 cs=0 l=1 rev1=1 crypto rx=0x7f8130005fd0 tx=0x7f813002bcf0 comp rx=0 tx=0).stop 2023-10-19T14:18:33.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.399+0000 7f813fbf0640 1 -- 172.21.15.57:0/2711372745 shutdown_connections 2023-10-19T14:18:33.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.399+0000 7f813fbf0640 1 --2- 172.21.15.57:0/2711372745 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f813806c3b0 0x7f81380b0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:33.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.399+0000 7f813fbf0640 1 --2- 172.21.15.57:0/2711372745 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8138157220 0x7f81380ca870 unknown :-1 s=CLOSED pgs=2163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:33.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.399+0000 7f813fbf0640 1 -- 172.21.15.57:0/2711372745 >> 172.21.15.57:0/2711372745 conn(0x7f8138069930 msgr2=0x7f8138069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:33.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.399+0000 7f813fbf0640 1 -- 172.21.15.57:0/2711372745 shutdown_connections 2023-10-19T14:18:33.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.399+0000 7f813fbf0640 1 -- 172.21.15.57:0/2711372745 wait complete. 2023-10-19T14:18:33.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.400+0000 7f813fbf0640 1 Processor -- start 2023-10-19T14:18:33.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.400+0000 7f813fbf0640 1 -- start start 2023-10-19T14:18:33.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.401+0000 7f813fbf0640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f813806c3b0 0x7f813813b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:33.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.401+0000 7f813fbf0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f81380cadb0 0x7f81380aa690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:33.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.401+0000 7f813fbf0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8138157220 0x7f81380aabd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:33.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.401+0000 7f813fbf0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81381e3e80 con 0x7f81380cadb0 2023-10-19T14:18:33.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.401+0000 7f813fbf0640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f813813f150 con 0x7f8138157220 2023-10-19T14:18:33.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.401+0000 7f813fbf0640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81381e40d0 con 0x7f813806c3b0 2023-10-19T14:18:33.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.401+0000 7f813d18b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f81380cadb0 0x7f81380aa690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:33.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.401+0000 7f813d18b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f81380cadb0 0x7f81380aa690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51346/0 (socket says 172.21.15.57:51346) 2023-10-19T14:18:33.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.401+0000 7f813d18b640 1 -- 172.21.15.57:0/2812646738 learned_addr learned my addr 172.21.15.57:0/2812646738 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:18:33.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.401+0000 7f813d98c640 1 --2- 172.21.15.57:0/2812646738 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f813806c3b0 0x7f813813b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:33.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.401+0000 7f813e18d640 1 --2- 172.21.15.57:0/2812646738 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8138157220 0x7f81380aabd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:33.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.401+0000 7f813d18b640 1 -- 172.21.15.57:0/2812646738 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f813806c3b0 msgr2=0x7f813813b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:33.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.401+0000 7f813d18b640 1 --2- 172.21.15.57:0/2812646738 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f813806c3b0 0x7f813813b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:33.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.401+0000 7f813d18b640 1 -- 172.21.15.57:0/2812646738 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8138157220 msgr2=0x7f81380aabd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:33.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.401+0000 7f813d18b640 1 --2- 172.21.15.57:0/2812646738 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8138157220 0x7f81380aabd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:33.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.401+0000 7f813d18b640 1 -- 172.21.15.57:0/2812646738 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8130009560 con 0x7f81380cadb0 2023-10-19T14:18:33.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.401+0000 7f813d18b640 1 --2- 172.21.15.57:0/2812646738 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f81380cadb0 0x7f81380aa690 secure :-1 s=READY pgs=2164 cs=0 l=1 rev1=1 crypto rx=0x7f812800c410 tx=0x7f812800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:33.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.402+0000 7f812e7fc640 1 -- 172.21.15.57:0/2812646738 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81280035f0 con 0x7f81380cadb0 2023-10-19T14:18:33.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.402+0000 7f812e7fc640 1 -- 172.21.15.57:0/2812646738 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f8128003750 con 0x7f81380cadb0 2023-10-19T14:18:33.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.402+0000 7f813fbf0640 1 -- 172.21.15.57:0/2812646738 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81381e43b0 con 0x7f81380cadb0 2023-10-19T14:18:33.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.402+0000 7f812e7fc640 1 -- 172.21.15.57:0/2812646738 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8128003ba0 con 0x7f81380cadb0 2023-10-19T14:18:33.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.402+0000 7f813fbf0640 1 -- 172.21.15.57:0/2812646738 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f813813bea0 con 0x7f81380cadb0 2023-10-19T14:18:33.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.403+0000 7f813fbf0640 1 -- 172.21.15.57:0/2812646738 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81381e48d0 con 0x7f81380cadb0 2023-10-19T14:18:33.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.403+0000 7f812e7fc640 1 -- 172.21.15.57:0/2812646738 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 117) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f8128003d00 con 0x7f81380cadb0 2023-10-19T14:18:33.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.403+0000 7f812e7fc640 1 --2- 172.21.15.57:0/2812646738 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f811c06af60 0x7f811c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:33.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.404+0000 7f812e7fc640 1 -- 172.21.15.57:0/2812646738 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 6351+0+0 (secure 0 0 0) 0x7f8128089170 con 0x7f81380cadb0 2023-10-19T14:18:33.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.404+0000 7f813d98c640 1 --2- 172.21.15.57:0/2812646738 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f811c06af60 0x7f811c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:33.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.404+0000 7f813d98c640 1 --2- 172.21.15.57:0/2812646738 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f811c06af60 0x7f811c06d420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f8130035000 tx=0x7f8130004730 comp rx=0 tx=0).ready entity=mgr.8744 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:33.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.408+0000 7f812e7fc640 1 -- 172.21.15.57:0/2812646738 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f81381e48d0 con 0x7f81380cadb0 2023-10-19T14:18:33.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.620+0000 7f813fbf0640 1 -- 172.21.15.57:0/2812646738 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f81381e48d0 con 0x7f81380cadb0 2023-10-19T14:18:33.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.620+0000 7f812e7fc640 1 -- 172.21.15.57:0/2812646738 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+3704 (secure 0 0 0) 0x7f81381e48d0 con 0x7f81380cadb0 2023-10-19T14:18:33.625 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:18:33.625 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":79,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8480,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6833/2675481384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2675481384},{"type":"v1","addr":"172.21.15.57:6833","nonce":2675481384}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74},{"gid":8486,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/2884298100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2884298100},{"type":"v1","addr":"172.21.15.179:6836","nonce":2884298100}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":73},{"gid":10000,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/1620626927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":1620626927},{"type":"v1","addr":"172.21.15.179:6837","nonce":1620626927}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":79},{"gid":10641,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/505361151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":505361151},{"type":"v1","addr":"172.21.15.57:6835","nonce":505361151}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74}],"filesystems":[]} 2023-10-19T14:18:33.626 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 79 2023-10-19T14:18:33.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.622+0000 7f813fbf0640 1 -- 172.21.15.57:0/2812646738 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f811c06af60 msgr2=0x7f811c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:33.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.622+0000 7f813fbf0640 1 --2- 172.21.15.57:0/2812646738 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f811c06af60 0x7f811c06d420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f8130035000 tx=0x7f8130004730 comp rx=0 tx=0).stop 2023-10-19T14:18:33.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.622+0000 7f813fbf0640 1 -- 172.21.15.57:0/2812646738 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f81380cadb0 msgr2=0x7f81380aa690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:33.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.622+0000 7f813fbf0640 1 --2- 172.21.15.57:0/2812646738 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f81380cadb0 0x7f81380aa690 secure :-1 s=READY pgs=2164 cs=0 l=1 rev1=1 crypto rx=0x7f812800c410 tx=0x7f812800c8d0 comp rx=0 tx=0).stop 2023-10-19T14:18:33.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.622+0000 7f813fbf0640 1 -- 172.21.15.57:0/2812646738 shutdown_connections 2023-10-19T14:18:33.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.622+0000 7f813fbf0640 1 --2- 172.21.15.57:0/2812646738 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f811c06af60 0x7f811c06d420 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:33.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.622+0000 7f813fbf0640 1 --2- 172.21.15.57:0/2812646738 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8138157220 0x7f81380aabd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:33.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.622+0000 7f813fbf0640 1 --2- 172.21.15.57:0/2812646738 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f81380cadb0 0x7f81380aa690 unknown :-1 s=CLOSED pgs=2164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:33.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.622+0000 7f813fbf0640 1 --2- 172.21.15.57:0/2812646738 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f813806c3b0 0x7f813813b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:33.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.622+0000 7f813fbf0640 1 -- 172.21.15.57:0/2812646738 >> 172.21.15.57:0/2812646738 conn(0x7f8138069930 msgr2=0x7f8138069d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:33.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.622+0000 7f813fbf0640 1 -- 172.21.15.57:0/2812646738 shutdown_connections 2023-10-19T14:18:33.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.622+0000 7f813fbf0640 1 -- 172.21.15.57:0/2812646738 wait complete. 2023-10-19T14:18:33.636 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-19T14:18:33.636 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-19T14:18:33.637 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-19T14:18:33.637 INFO:teuthology.orchestra.run:waiting for 300 2023-10-19T14:18:33.637 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-19T14:18:33.637 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-19T14:18:33.638 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-19T14:18:33.638 INFO:teuthology.orchestra.run:waiting for 300 2023-10-19T14:18:33.638 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-19T14:18:33.638 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-19T14:18:33.639 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-19T14:18:33.639 INFO:teuthology.orchestra.run:waiting for 300 2023-10-19T14:18:33.639 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-19T14:18:33.640 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-19T14:18:33.640 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-19T14:18:33.640 INFO:teuthology.orchestra.run:waiting for 300 2023-10-19T14:18:33.646 INFO:tasks.ceph.mds.a:Stopped 2023-10-19T14:18:33.646 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-19T14:18:33.647 INFO:tasks.ceph.mds.c:Stopped 2023-10-19T14:18:33.648 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-19T14:18:33.651 INFO:tasks.ceph.mds.b:Stopped 2023-10-19T14:18:33.651 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-19T14:18:33.652 INFO:tasks.ceph.mds.d:Stopped 2023-10-19T14:18:33.652 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-19T14:18:33.689 INFO:tasks.ceph.mds.a:Started 2023-10-19T14:18:33.689 DEBUG:teuthology.parallel:result is None 2023-10-19T14:18:33.690 INFO:tasks.ceph.mds.c:Started 2023-10-19T14:18:33.690 DEBUG:teuthology.parallel:result is None 2023-10-19T14:18:33.693 INFO:tasks.ceph.mds.b:Started 2023-10-19T14:18:33.694 DEBUG:teuthology.parallel:result is None 2023-10-19T14:18:33.694 INFO:tasks.ceph.mds.d:Started 2023-10-19T14:18:33.695 DEBUG:teuthology.parallel:result is None 2023-10-19T14:18:33.695 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-19T14:18:33.771 INFO:tasks.ceph.mds.c.smithi057.stdout:starting mds.c at 2023-10-19T14:18:33.772 INFO:tasks.ceph.mds.a.smithi057.stdout:starting mds.a at 2023-10-19T14:18:33.773 INFO:tasks.ceph.mds.d.smithi179.stdout:starting mds.d at 2023-10-19T14:18:33.774 INFO:tasks.ceph.mds.b.smithi179.stdout:starting mds.b at 2023-10-19T14:18:33.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.807+0000 7ff0cfd05640 1 -- 172.21.15.57:0/1693092552 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff0b4003970 msgr2=0x7ff0b4003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:33.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.807+0000 7ff0cfd05640 1 --2- 172.21.15.57:0/1693092552 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff0b4003970 0x7ff0b4003d90 secure :-1 s=READY pgs=2171 cs=0 l=1 rev1=1 crypto rx=0x7ff0c0006fa0 tx=0x7ff0b40057f0 comp rx=0 tx=0).stop 2023-10-19T14:18:33.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.807+0000 7ff0cfd05640 1 -- 172.21.15.57:0/1693092552 shutdown_connections 2023-10-19T14:18:33.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.807+0000 7ff0cfd05640 1 --2- 172.21.15.57:0/1693092552 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff0b4003970 0x7ff0b4003d90 unknown :-1 s=CLOSED pgs=2171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:33.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.807+0000 7ff0cfd05640 1 -- 172.21.15.57:0/1693092552 >> 172.21.15.57:0/1693092552 conn(0x7ff0c8069930 msgr2=0x7ff0c8069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:33.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.807+0000 7ff0cfd05640 1 -- 172.21.15.57:0/1693092552 shutdown_connections 2023-10-19T14:18:33.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.807+0000 7ff0cfd05640 1 -- 172.21.15.57:0/1693092552 wait complete. 2023-10-19T14:18:33.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.808+0000 7ff0cfd05640 1 Processor -- start 2023-10-19T14:18:33.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.808+0000 7ff0cfd05640 1 -- start start 2023-10-19T14:18:33.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.809+0000 7ff0cfd05640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff0b4003970 0x7ff0c813d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:33.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.809+0000 7ff0cfd05640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff0c81ddf70 0x7ff0c81dd8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:33.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.809+0000 7ff0cfd05640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff0c80a8110 0x7ff0c80aca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:33.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.809+0000 7ff0cfd05640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c81ddde0 con 0x7ff0c80a8110 2023-10-19T14:18:33.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.809+0000 7ff0cfd05640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c80a8e40 con 0x7ff0b4003970 2023-10-19T14:18:33.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.809+0000 7ff0cfd05640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c80a8880 con 0x7ff0c81ddf70 2023-10-19T14:18:33.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.809+0000 7ff0ce2a2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff0c80a8110 0x7ff0c80aca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:33.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.809+0000 7ff0ce2a2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff0c80a8110 0x7ff0c80aca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51378/0 (socket says 172.21.15.57:51378) 2023-10-19T14:18:33.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.809+0000 7ff0ce2a2640 1 -- 172.21.15.57:0/3480321322 learned_addr learned my addr 172.21.15.57:0/3480321322 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:18:33.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.809+0000 7ff0cdaa1640 1 --2- 172.21.15.57:0/3480321322 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff0b4003970 0x7ff0c813d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:33.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.809+0000 7ff0cd2a0640 1 --2- 172.21.15.57:0/3480321322 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff0c81ddf70 0x7ff0c81dd8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:33.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.809+0000 7ff0ce2a2640 1 -- 172.21.15.57:0/3480321322 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff0c81ddf70 msgr2=0x7ff0c81dd8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:33.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.809+0000 7ff0ce2a2640 1 --2- 172.21.15.57:0/3480321322 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff0c81ddf70 0x7ff0c81dd8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:33.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.809+0000 7ff0ce2a2640 1 -- 172.21.15.57:0/3480321322 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff0b4003970 msgr2=0x7ff0c813d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:33.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.809+0000 7ff0ce2a2640 1 --2- 172.21.15.57:0/3480321322 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff0b4003970 0x7ff0c813d350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:33.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.809+0000 7ff0ce2a2640 1 -- 172.21.15.57:0/3480321322 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0c80a8b60 con 0x7ff0c80a8110 2023-10-19T14:18:33.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.809+0000 7ff0cd2a0640 1 --2- 172.21.15.57:0/3480321322 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff0c81ddf70 0x7ff0c81dd8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:18:33.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.809+0000 7ff0ce2a2640 1 --2- 172.21.15.57:0/3480321322 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff0c80a8110 0x7ff0c80aca10 secure :-1 s=READY pgs=2172 cs=0 l=1 rev1=1 crypto rx=0x7ff0c806d570 tx=0x7ff0c400da80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:33.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.809+0000 7ff0be7fc640 1 -- 172.21.15.57:0/3480321322 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0c4015030 con 0x7ff0c80a8110 2023-10-19T14:18:33.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.809+0000 7ff0be7fc640 1 -- 172.21.15.57:0/3480321322 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7ff0c4003e70 con 0x7ff0c80a8110 2023-10-19T14:18:33.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.810+0000 7ff0cfd05640 1 -- 172.21.15.57:0/3480321322 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0c813cb50 con 0x7ff0c80a8110 2023-10-19T14:18:33.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.810+0000 7ff0be7fc640 1 -- 172.21.15.57:0/3480321322 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0c4006350 con 0x7ff0c80a8110 2023-10-19T14:18:33.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.810+0000 7ff0cfd05640 1 -- 172.21.15.57:0/3480321322 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0c80a8590 con 0x7ff0c80a8110 2023-10-19T14:18:33.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.811+0000 7ff0cfd05640 1 -- 172.21.15.57:0/3480321322 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0c806a190 con 0x7ff0c80a8110 2023-10-19T14:18:33.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.811+0000 7ff0be7fc640 1 -- 172.21.15.57:0/3480321322 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 117) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff0c4018070 con 0x7ff0c80a8110 2023-10-19T14:18:33.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.811+0000 7ff0be7fc640 1 --2- 172.21.15.57:0/3480321322 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7ff0ac06af60 0x7ff0ac06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:33.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.811+0000 7ff0be7fc640 1 -- 172.21.15.57:0/3480321322 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 6351+0+0 (secure 0 0 0) 0x7ff0c4016070 con 0x7ff0c80a8110 2023-10-19T14:18:33.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.812+0000 7ff0cdaa1640 1 --2- 172.21.15.57:0/3480321322 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7ff0ac06af60 0x7ff0ac06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:33.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.812+0000 7ff0cdaa1640 1 --2- 172.21.15.57:0/3480321322 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7ff0ac06af60 0x7ff0ac06d420 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff0c0034000 tx=0x7ff0c002e000 comp rx=0 tx=0).ready entity=mgr.8744 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:33.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:33.815+0000 7ff0be7fc640 1 -- 172.21.15.57:0/3480321322 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff0c4058220 con 0x7ff0c80a8110 2023-10-19T14:18:34.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:34.004+0000 7ff0cfd05640 1 -- 172.21.15.57:0/3480321322 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7ff0c80690e0 con 0x7ff0c80a8110 2023-10-19T14:18:35.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.325+0000 7ff0be7fc640 1 -- 172.21.15.57:0/3480321322 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v201) v1 ==== 97+0+0 (secure 0 0 0) 0x7ff0c40014a0 con 0x7ff0c80a8110 2023-10-19T14:18:35.328 INFO:teuthology.orchestra.run.smithi057.stderr: removed all blocklist entries 2023-10-19T14:18:35.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.328+0000 7ff0cfd05640 1 -- 172.21.15.57:0/3480321322 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7ff0ac06af60 msgr2=0x7ff0ac06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:35.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.328+0000 7ff0cfd05640 1 --2- 172.21.15.57:0/3480321322 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7ff0ac06af60 0x7ff0ac06d420 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff0c0034000 tx=0x7ff0c002e000 comp rx=0 tx=0).stop 2023-10-19T14:18:35.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.328+0000 7ff0cfd05640 1 -- 172.21.15.57:0/3480321322 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff0c80a8110 msgr2=0x7ff0c80aca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:35.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.328+0000 7ff0cfd05640 1 --2- 172.21.15.57:0/3480321322 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff0c80a8110 0x7ff0c80aca10 secure :-1 s=READY pgs=2172 cs=0 l=1 rev1=1 crypto rx=0x7ff0c806d570 tx=0x7ff0c400da80 comp rx=0 tx=0).stop 2023-10-19T14:18:35.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.328+0000 7ff0cfd05640 1 -- 172.21.15.57:0/3480321322 shutdown_connections 2023-10-19T14:18:35.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.328+0000 7ff0cfd05640 1 --2- 172.21.15.57:0/3480321322 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7ff0ac06af60 0x7ff0ac06d420 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:35.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.328+0000 7ff0cfd05640 1 --2- 172.21.15.57:0/3480321322 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff0c80a8110 0x7ff0c80aca10 unknown :-1 s=CLOSED pgs=2172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:35.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.328+0000 7ff0cfd05640 1 --2- 172.21.15.57:0/3480321322 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff0c81ddf70 0x7ff0c81dd8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:35.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.328+0000 7ff0cfd05640 1 --2- 172.21.15.57:0/3480321322 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff0b4003970 0x7ff0c813d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:35.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.328+0000 7ff0cfd05640 1 -- 172.21.15.57:0/3480321322 >> 172.21.15.57:0/3480321322 conn(0x7ff0c8069930 msgr2=0x7ff0c80b0660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:35.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.328+0000 7ff0cfd05640 1 -- 172.21.15.57:0/3480321322 shutdown_connections 2023-10-19T14:18:35.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.328+0000 7ff0cfd05640 1 -- 172.21.15.57:0/3480321322 wait complete. 2023-10-19T14:18:35.346 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-19T14:18:35.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.463+0000 7f7466c39640 1 -- 172.21.15.57:0/3548437376 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f746013e520 msgr2=0x7f746006bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:35.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.463+0000 7f7466c39640 1 --2- 172.21.15.57:0/3548437376 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f746013e520 0x7f746006bb60 secure :-1 s=READY pgs=2174 cs=0 l=1 rev1=1 crypto rx=0x7f7450009830 tx=0x7f745002bcd0 comp rx=0 tx=0).stop 2023-10-19T14:18:35.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.463+0000 7f7466c39640 1 -- 172.21.15.57:0/3548437376 shutdown_connections 2023-10-19T14:18:35.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.463+0000 7f7466c39640 1 --2- 172.21.15.57:0/3548437376 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7460070af0 0x7f74600affe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:35.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.463+0000 7f7466c39640 1 --2- 172.21.15.57:0/3548437376 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f746013e520 0x7f746006bb60 unknown :-1 s=CLOSED pgs=2174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:35.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.463+0000 7f7466c39640 1 -- 172.21.15.57:0/3548437376 >> 172.21.15.57:0/3548437376 conn(0x7f7460069930 msgr2=0x7f7460069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:35.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.463+0000 7f7466c39640 1 -- 172.21.15.57:0/3548437376 shutdown_connections 2023-10-19T14:18:35.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.463+0000 7f7466c39640 1 -- 172.21.15.57:0/3548437376 wait complete. 2023-10-19T14:18:35.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.464+0000 7f7466c39640 1 Processor -- start 2023-10-19T14:18:35.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.464+0000 7f7466c39640 1 -- start start 2023-10-19T14:18:35.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.465+0000 7f7466c39640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7460070af0 0x7f74601dc7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:35.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.465+0000 7f7466c39640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f746013e520 0x7f746012b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:35.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.465+0000 7f7466c39640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7460141690 0x7f74601f25c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:35.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.465+0000 7f7466c39640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74600ab4a0 con 0x7f7460141690 2023-10-19T14:18:35.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.465+0000 7f7466c39640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74601f30c0 con 0x7f746013e520 2023-10-19T14:18:35.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.465+0000 7f7466c39640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74601f33a0 con 0x7f7460070af0 2023-10-19T14:18:35.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.465+0000 7f74651d6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7460141690 0x7f74601f25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:35.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.465+0000 7f74651d6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7460141690 0x7f74601f25c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51388/0 (socket says 172.21.15.57:51388) 2023-10-19T14:18:35.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.465+0000 7f74651d6640 1 -- 172.21.15.57:0/4187804049 learned_addr learned my addr 172.21.15.57:0/4187804049 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:18:35.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.465+0000 7f74649d5640 1 --2- 172.21.15.57:0/4187804049 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7460070af0 0x7f74601dc7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:35.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.466+0000 7f745ffff640 1 --2- 172.21.15.57:0/4187804049 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f746013e520 0x7f746012b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:35.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.466+0000 7f74651d6640 1 -- 172.21.15.57:0/4187804049 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7460070af0 msgr2=0x7f74601dc7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:35.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.466+0000 7f74651d6640 1 --2- 172.21.15.57:0/4187804049 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7460070af0 0x7f74601dc7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:35.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.466+0000 7f74651d6640 1 -- 172.21.15.57:0/4187804049 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f746013e520 msgr2=0x7f746012b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:35.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.466+0000 7f74651d6640 1 --2- 172.21.15.57:0/4187804049 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f746013e520 0x7f746012b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:35.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.466+0000 7f74651d6640 1 -- 172.21.15.57:0/4187804049 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7450009490 con 0x7f7460141690 2023-10-19T14:18:35.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.466+0000 7f74651d6640 1 --2- 172.21.15.57:0/4187804049 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7460141690 0x7f74601f25c0 secure :-1 s=READY pgs=2175 cs=0 l=1 rev1=1 crypto rx=0x7f745800cad0 tx=0x7f745800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:35.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.466+0000 7f745d7fa640 1 -- 172.21.15.57:0/4187804049 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f745800ae20 con 0x7f7460141690 2023-10-19T14:18:35.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.466+0000 7f745d7fa640 1 -- 172.21.15.57:0/4187804049 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f7458003aa0 con 0x7f7460141690 2023-10-19T14:18:35.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.466+0000 7f7466c39640 1 -- 172.21.15.57:0/4187804049 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74600ab0c0 con 0x7f7460141690 2023-10-19T14:18:35.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.466+0000 7f745d7fa640 1 -- 172.21.15.57:0/4187804049 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7458004030 con 0x7f7460141690 2023-10-19T14:18:35.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.466+0000 7f7466c39640 1 -- 172.21.15.57:0/4187804049 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74601dccf0 con 0x7f7460141690 2023-10-19T14:18:35.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.467+0000 7f7466c39640 1 -- 172.21.15.57:0/4187804049 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74600690e0 con 0x7f7460141690 2023-10-19T14:18:35.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.468+0000 7f745d7fa640 1 -- 172.21.15.57:0/4187804049 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 117) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f7458004190 con 0x7f7460141690 2023-10-19T14:18:35.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.468+0000 7f745d7fa640 1 --2- 172.21.15.57:0/4187804049 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f744c06af60 0x7f744c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:35.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.468+0000 7f745d7fa640 1 -- 172.21.15.57:0/4187804049 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f7458089dc0 con 0x7f7460141690 2023-10-19T14:18:35.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.468+0000 7f74649d5640 1 --2- 172.21.15.57:0/4187804049 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f744c06af60 0x7f744c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:35.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.469+0000 7f74649d5640 1 --2- 172.21.15.57:0/4187804049 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f744c06af60 0x7f744c06d420 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7450036000 tx=0x7f7450004730 comp rx=0 tx=0).ready entity=mgr.8744 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:35.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.472+0000 7f745d7fa640 1 -- 172.21.15.57:0/4187804049 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7458054f80 con 0x7f7460141690 2023-10-19T14:18:35.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.713+0000 7f7466c39640 1 -- 172.21.15.57:0/4187804049 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f746006bb60 con 0x7f7460141690 2023-10-19T14:18:35.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.713+0000 7f745d7fa640 1 -- 172.21.15.57:0/4187804049 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v57) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f745801d020 con 0x7f7460141690 2023-10-19T14:18:35.717 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:18:35.717 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2023-10-19T14:18:35.717 INFO:teuthology.orchestra.run.smithi057.stdout: "auth_dump": [ 2023-10-19T14:18:35.717 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:18:35.717 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "mds.a", 2023-10-19T14:18:35.718 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBjJDFlmbVANhAAP/ynHlU/OxaRq5lQoAa8Ng==", 2023-10-19T14:18:35.718 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:18:35.718 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T14:18:35.718 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow *", 2023-10-19T14:18:35.718 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow *", 2023-10-19T14:18:35.719 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:18:35.719 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:18:35.719 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:18:35.719 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:18:35.719 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "mds.b", 2023-10-19T14:18:35.720 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFl9l7RBxAAC0geI2tJ1SIZZr1++kS/BA==", 2023-10-19T14:18:35.720 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:18:35.720 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T14:18:35.720 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow *", 2023-10-19T14:18:35.720 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow *", 2023-10-19T14:18:35.720 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:18:35.721 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:18:35.721 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:18:35.721 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:18:35.721 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "mds.c", 2023-10-19T14:18:35.721 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFlFkaoAhAA/PBizkJ4bm958oP4CmPuIQ==", 2023-10-19T14:18:35.722 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:18:35.722 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T14:18:35.722 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow *", 2023-10-19T14:18:35.722 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow *", 2023-10-19T14:18:35.722 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:18:35.722 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:18:35.723 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:18:35.723 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:18:35.723 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "mds.d", 2023-10-19T14:18:35.723 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFlx8skDxAAAfCHTYtYJgJik0WHSWeQfQ==", 2023-10-19T14:18:35.723 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:18:35.724 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T14:18:35.724 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow *", 2023-10-19T14:18:35.724 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow *", 2023-10-19T14:18:35.724 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:18:35.724 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:18:35.725 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:18:35.725 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:18:35.725 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.0", 2023-10-19T14:18:35.725 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBrJDFlCo9sBxAAQN2cID4nCQgRwGaGFEV8LQ==", 2023-10-19T14:18:35.725 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:18:35.725 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T14:18:35.726 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T14:18:35.726 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:18:35.726 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:18:35.726 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:18:35.726 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:18:35.727 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.1", 2023-10-19T14:18:35.727 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBuJDFl4fsLCBAAp3Pi80yZtc5sju0GisEuVA==", 2023-10-19T14:18:35.727 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:18:35.727 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T14:18:35.727 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T14:18:35.727 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:18:35.728 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:18:35.728 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:18:35.728 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:18:35.728 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.2", 2023-10-19T14:18:35.729 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBxJDFl+KAKCBAAd4N9cluARxPVfBLU+7Hd7A==", 2023-10-19T14:18:35.729 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:18:35.729 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T14:18:35.729 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T14:18:35.729 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:18:35.730 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:18:35.730 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:18:35.730 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:18:35.730 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.3", 2023-10-19T14:18:35.730 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQB0JDFlU/toCBAAP0Ou1zcHKNx5SfbybJkh4Q==", 2023-10-19T14:18:35.730 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:18:35.731 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T14:18:35.731 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T14:18:35.731 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:18:35.731 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:18:35.731 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:18:35.732 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:18:35.732 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.4", 2023-10-19T14:18:35.732 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQB6JDFlc6stEBAALkgNyxoiCTzZnKpBSPxI6w==", 2023-10-19T14:18:35.732 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:18:35.732 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T14:18:35.732 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T14:18:35.733 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:18:35.733 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:18:35.733 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:18:35.733 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:18:35.733 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.5", 2023-10-19T14:18:35.734 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQB9JDFlI3cNDhAAuG3tHJ6RvPwAzPE76wTIVQ==", 2023-10-19T14:18:35.734 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:18:35.734 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T14:18:35.734 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T14:18:35.734 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:18:35.734 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:18:35.735 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:18:35.735 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:18:35.735 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.6", 2023-10-19T14:18:35.735 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQCAJDFlLPumCxAA/iy08t/2N9rXfn71KtbSQw==", 2023-10-19T14:18:35.735 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:18:35.736 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T14:18:35.736 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T14:18:35.736 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:18:35.736 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:18:35.736 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:18:35.737 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:18:35.737 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.7", 2023-10-19T14:18:35.737 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQCDJDFlHmrbCRAA+hpWygGIJydfU6cHWlfx/Q==", 2023-10-19T14:18:35.737 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:18:35.737 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T14:18:35.737 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T14:18:35.738 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:18:35.738 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:18:35.738 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:18:35.738 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:18:35.739 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "client.0", 2023-10-19T14:18:35.739 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFlg9yuExAATxiWDfS+p07G1F0pORbJEg==", 2023-10-19T14:18:35.739 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:18:35.739 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T14:18:35.739 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow r", 2023-10-19T14:18:35.740 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-19T14:18:35.740 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:18:35.740 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:18:35.740 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:18:35.741 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "client.1", 2023-10-19T14:18:35.741 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFl9IwYGxAAHPituiJSPkHDuD0QyG4Hww==", 2023-10-19T14:18:35.741 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:18:35.741 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T14:18:35.741 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow r", 2023-10-19T14:18:35.742 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow rw", 2023-10-19T14:18:35.742 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow rwx" 2023-10-19T14:18:35.742 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:18:35.742 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:18:35.743 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:18:35.743 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "client.2", 2023-10-19T14:18:35.743 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFlMd+AFxAAS/UTESDt76GmrhhN2o/Fdw==", 2023-10-19T14:18:35.743 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:18:35.743 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T14:18:35.744 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow r", 2023-10-19T14:18:35.744 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow rw", 2023-10-19T14:18:35.744 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow rwx" 2023-10-19T14:18:35.744 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:18:35.744 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:18:35.744 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:18:35.745 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "client.3", 2023-10-19T14:18:35.745 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFlHZ3fIBAA42B7n8opLS5smXIwJyVYzw==", 2023-10-19T14:18:35.745 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:18:35.745 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T14:18:35.746 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow r", 2023-10-19T14:18:35.746 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow rw", 2023-10-19T14:18:35.746 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow rwx" 2023-10-19T14:18:35.746 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:18:35.746 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:18:35.746 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:18:35.747 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "client.admin", 2023-10-19T14:18:35.747 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBjJDFlBgasFxAAMA24j8KszcXMEpYRQDXj4g==", 2023-10-19T14:18:35.747 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:18:35.747 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow *", 2023-10-19T14:18:35.747 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow *", 2023-10-19T14:18:35.748 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow *", 2023-10-19T14:18:35.748 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:18:35.748 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:18:35.748 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:18:35.748 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:18:35.748 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "mgr.x", 2023-10-19T14:18:35.749 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBjJDFllFBfMhAA5+KPXWmYZqcESmLObaUbBA==", 2023-10-19T14:18:35.749 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:18:35.749 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow *", 2023-10-19T14:18:35.749 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile mgr", 2023-10-19T14:18:35.749 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:18:35.750 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:18:35.750 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:18:35.750 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:18:35.750 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "mgr.y", 2023-10-19T14:18:35.750 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBjJDFlxRzJLRAAGjG58o7qXAHvIEIRYwl6QA==", 2023-10-19T14:18:35.750 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:18:35.751 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow *", 2023-10-19T14:18:35.751 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile mgr", 2023-10-19T14:18:35.751 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:18:35.751 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:18:35.752 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:18:35.752 INFO:teuthology.orchestra.run.smithi057.stdout: ] 2023-10-19T14:18:35.752 INFO:teuthology.orchestra.run.smithi057.stdout:} 2023-10-19T14:18:35.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.715+0000 7f7466c39640 1 -- 172.21.15.57:0/4187804049 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f744c06af60 msgr2=0x7f744c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:35.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.715+0000 7f7466c39640 1 --2- 172.21.15.57:0/4187804049 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f744c06af60 0x7f744c06d420 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7450036000 tx=0x7f7450004730 comp rx=0 tx=0).stop 2023-10-19T14:18:35.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.715+0000 7f7466c39640 1 -- 172.21.15.57:0/4187804049 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7460141690 msgr2=0x7f74601f25c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:35.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.715+0000 7f7466c39640 1 --2- 172.21.15.57:0/4187804049 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7460141690 0x7f74601f25c0 secure :-1 s=READY pgs=2175 cs=0 l=1 rev1=1 crypto rx=0x7f745800cad0 tx=0x7f745800cf90 comp rx=0 tx=0).stop 2023-10-19T14:18:35.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.715+0000 7f7466c39640 1 -- 172.21.15.57:0/4187804049 shutdown_connections 2023-10-19T14:18:35.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.715+0000 7f7466c39640 1 --2- 172.21.15.57:0/4187804049 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f744c06af60 0x7f744c06d420 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:35.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.715+0000 7f7466c39640 1 --2- 172.21.15.57:0/4187804049 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7460141690 0x7f74601f25c0 unknown :-1 s=CLOSED pgs=2175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:35.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.715+0000 7f7466c39640 1 --2- 172.21.15.57:0/4187804049 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f746013e520 0x7f746012b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:35.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.715+0000 7f7466c39640 1 --2- 172.21.15.57:0/4187804049 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7460070af0 0x7f74601dc7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:35.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.715+0000 7f7466c39640 1 -- 172.21.15.57:0/4187804049 >> 172.21.15.57:0/4187804049 conn(0x7f7460069930 msgr2=0x7f74600ad9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:35.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.715+0000 7f7466c39640 1 -- 172.21.15.57:0/4187804049 shutdown_connections 2023-10-19T14:18:35.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.715+0000 7f7466c39640 1 -- 172.21.15.57:0/4187804049 wait complete. 2023-10-19T14:18:35.757 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T14:18:35.798 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:18:35.809 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:18:35.869 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.865+0000 7f8d469f5640 1 -- 172.21.15.57:0/568025521 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8d40157d20 msgr2=0x7f8d40069820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:35.869 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.865+0000 7f8d469f5640 1 --2- 172.21.15.57:0/568025521 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8d40157d20 0x7f8d40069820 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f8d38005fd0 tx=0x7f8d3802c0e0 comp rx=0 tx=0).stop 2023-10-19T14:18:35.869 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.866+0000 7f8d469f5640 1 -- 172.21.15.57:0/568025521 shutdown_connections 2023-10-19T14:18:35.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.866+0000 7f8d469f5640 1 --2- 172.21.15.57:0/568025521 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8d40157d20 0x7f8d40069820 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:35.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.866+0000 7f8d469f5640 1 --2- 172.21.15.57:0/568025521 >> v2:172.21.15.179:3300/0 conn(0x7f8d400ad170 0x7f8d400af770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:35.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.866+0000 7f8d469f5640 1 -- 172.21.15.57:0/568025521 >> 172.21.15.57:0/568025521 conn(0x7f8d400b0cd0 msgr2=0x7f8d400b1f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:35.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.866+0000 7f8d469f5640 1 -- 172.21.15.57:0/568025521 shutdown_connections 2023-10-19T14:18:35.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.866+0000 7f8d469f5640 1 -- 172.21.15.57:0/568025521 wait complete. 2023-10-19T14:18:35.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.867+0000 7f8d469f5640 1 Processor -- start 2023-10-19T14:18:35.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.867+0000 7f8d469f5640 1 -- start start 2023-10-19T14:18:35.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.868+0000 7f8d469f5640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8d4006fee0 0x7f8d401411c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:35.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.868+0000 7f8d469f5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8d400ad170 0x7f8d4013f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:35.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.868+0000 7f8d469f5640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8d40157d20 0x7f8d4006cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:35.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.868+0000 7f8d469f5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d40151cb0 con 0x7f8d400ad170 2023-10-19T14:18:35.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.868+0000 7f8d469f5640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d40140120 con 0x7f8d40157d20 2023-10-19T14:18:35.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.868+0000 7f8d469f5640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d40140400 con 0x7f8d4006fee0 2023-10-19T14:18:35.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.868+0000 7f8d3f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8d400ad170 0x7f8d4013f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:35.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.868+0000 7f8d3f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8d400ad170 0x7f8d4013f820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51396/0 (socket says 172.21.15.57:51396) 2023-10-19T14:18:35.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.868+0000 7f8d3f7fe640 1 -- 172.21.15.57:0/2264739040 learned_addr learned my addr 172.21.15.57:0/2264739040 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:18:35.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.868+0000 7f8d3ffff640 1 --2- 172.21.15.57:0/2264739040 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8d4006fee0 0x7f8d401411c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:35.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.868+0000 7f8d44f92640 1 --2- 172.21.15.57:0/2264739040 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8d40157d20 0x7f8d4006cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:35.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.868+0000 7f8d3f7fe640 1 -- 172.21.15.57:0/2264739040 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8d4006fee0 msgr2=0x7f8d401411c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:35.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.868+0000 7f8d3f7fe640 1 --2- 172.21.15.57:0/2264739040 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8d4006fee0 0x7f8d401411c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:35.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.868+0000 7f8d3f7fe640 1 -- 172.21.15.57:0/2264739040 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8d40157d20 msgr2=0x7f8d4006cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:35.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.868+0000 7f8d3f7fe640 1 --2- 172.21.15.57:0/2264739040 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8d40157d20 0x7f8d4006cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:35.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.868+0000 7f8d3f7fe640 1 -- 172.21.15.57:0/2264739040 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d30009560 con 0x7f8d400ad170 2023-10-19T14:18:35.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.868+0000 7f8d3f7fe640 1 --2- 172.21.15.57:0/2264739040 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8d400ad170 0x7f8d4013f820 secure :-1 s=READY pgs=2176 cs=0 l=1 rev1=1 crypto rx=0x7f8d3400c400 tx=0x7f8d3400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:35.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.868+0000 7f8d3cff9640 1 -- 172.21.15.57:0/2264739040 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d340052d0 con 0x7f8d400ad170 2023-10-19T14:18:35.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.869+0000 7f8d3cff9640 1 -- 172.21.15.57:0/2264739040 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f8d340035f0 con 0x7f8d400ad170 2023-10-19T14:18:35.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.869+0000 7f8d469f5640 1 -- 172.21.15.57:0/2264739040 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d38009840 con 0x7f8d400ad170 2023-10-19T14:18:35.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.869+0000 7f8d3cff9640 1 -- 172.21.15.57:0/2264739040 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d34003a40 con 0x7f8d400ad170 2023-10-19T14:18:35.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.869+0000 7f8d469f5640 1 -- 172.21.15.57:0/2264739040 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d401409c0 con 0x7f8d400ad170 2023-10-19T14:18:35.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.869+0000 7f8d469f5640 1 -- 172.21.15.57:0/2264739040 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d4006a840 con 0x7f8d400ad170 2023-10-19T14:18:35.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.870+0000 7f8d3cff9640 1 -- 172.21.15.57:0/2264739040 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 117) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f8d34003ba0 con 0x7f8d400ad170 2023-10-19T14:18:35.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.870+0000 7f8d3cff9640 1 --2- 172.21.15.57:0/2264739040 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f8d2c06af90 0x7f8d2c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:35.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.871+0000 7f8d3cff9640 1 -- 172.21.15.57:0/2264739040 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f8d34088d60 con 0x7f8d400ad170 2023-10-19T14:18:35.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.871+0000 7f8d3ffff640 1 --2- 172.21.15.57:0/2264739040 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f8d2c06af90 0x7f8d2c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:35.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.871+0000 7f8d3ffff640 1 --2- 172.21.15.57:0/2264739040 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f8d2c06af90 0x7f8d2c06d450 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f8d30009530 tx=0x7f8d30009260 comp rx=0 tx=0).ready entity=mgr.8744 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:35.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:35.875+0000 7f8d3cff9640 1 -- 172.21.15.57:0/2264739040 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8d34053f90 con 0x7f8d400ad170 2023-10-19T14:18:36.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.051+0000 7f8d469f5640 1 -- 172.21.15.57:0/2264739040 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8d400675f0 con 0x7f8d400ad170 2023-10-19T14:18:36.054 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.052+0000 7f8d3cff9640 1 -- 172.21.15.57:0/2264739040 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v201) v1 ==== 74+0+13127 (secure 0 0 0) 0x7f8d340575b0 con 0x7f8d400ad170 2023-10-19T14:18:36.055 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:18:36.055 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":201,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T14:18:35.324106+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T14:18:36.056 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.053+0000 7f8d469f5640 1 -- 172.21.15.57:0/2264739040 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f8d2c06af90 msgr2=0x7f8d2c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:36.056 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.053+0000 7f8d469f5640 1 --2- 172.21.15.57:0/2264739040 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f8d2c06af90 0x7f8d2c06d450 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f8d30009530 tx=0x7f8d30009260 comp rx=0 tx=0).stop 2023-10-19T14:18:36.057 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.053+0000 7f8d469f5640 1 -- 172.21.15.57:0/2264739040 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8d400ad170 msgr2=0x7f8d4013f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:36.057 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.053+0000 7f8d469f5640 1 --2- 172.21.15.57:0/2264739040 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8d400ad170 0x7f8d4013f820 secure :-1 s=READY pgs=2176 cs=0 l=1 rev1=1 crypto rx=0x7f8d3400c400 tx=0x7f8d3400c8c0 comp rx=0 tx=0).stop 2023-10-19T14:18:36.057 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.053+0000 7f8d469f5640 1 -- 172.21.15.57:0/2264739040 shutdown_connections 2023-10-19T14:18:36.057 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.053+0000 7f8d469f5640 1 --2- 172.21.15.57:0/2264739040 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f8d2c06af90 0x7f8d2c06d450 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:36.058 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.054+0000 7f8d469f5640 1 --2- 172.21.15.57:0/2264739040 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8d40157d20 0x7f8d4006cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:36.058 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.054+0000 7f8d469f5640 1 --2- 172.21.15.57:0/2264739040 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8d400ad170 0x7f8d4013f820 unknown :-1 s=CLOSED pgs=2176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:36.058 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.054+0000 7f8d469f5640 1 --2- 172.21.15.57:0/2264739040 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8d4006fee0 0x7f8d401411c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:36.058 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.054+0000 7f8d469f5640 1 -- 172.21.15.57:0/2264739040 >> 172.21.15.57:0/2264739040 conn(0x7f8d400b0cd0 msgr2=0x7f8d4006b920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:36.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.054+0000 7f8d469f5640 1 -- 172.21.15.57:0/2264739040 shutdown_connections 2023-10-19T14:18:36.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.054+0000 7f8d469f5640 1 -- 172.21.15.57:0/2264739040 wait complete. 2023-10-19T14:18:36.067 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-19T12:44:07.371186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-19T14:18:36.068 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-19T14:18:36.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.221+0000 7f714bd97640 1 -- 172.21.15.57:0/483400045 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7130004d00 msgr2=0x7f7130005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:36.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.221+0000 7f714bd97640 1 --2- 172.21.15.57:0/483400045 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7130004d00 0x7f7130005180 secure :-1 s=READY pgs=2178 cs=0 l=1 rev1=1 crypto rx=0x7f713c00c640 tx=0x7f71300057f0 comp rx=0 tx=0).stop 2023-10-19T14:18:36.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.221+0000 7f714a334640 1 -- 172.21.15.57:0/483400045 reap_dead start 2023-10-19T14:18:36.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.221+0000 7f714bd97640 1 -- 172.21.15.57:0/483400045 shutdown_connections 2023-10-19T14:18:36.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.221+0000 7f714bd97640 1 -- 172.21.15.57:0/483400045 >> 172.21.15.57:0/483400045 conn(0x7f7144069930 msgr2=0x7f7144069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:36.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.221+0000 7f714bd97640 1 -- 172.21.15.57:0/483400045 shutdown_connections 2023-10-19T14:18:36.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.221+0000 7f714bd97640 1 -- 172.21.15.57:0/483400045 wait complete. 2023-10-19T14:18:36.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.222+0000 7f714bd97640 1 Processor -- start 2023-10-19T14:18:36.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.222+0000 7f714bd97640 1 -- start start 2023-10-19T14:18:36.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.223+0000 7f714bd97640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f71440b01c0 0x7f71441e2520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:36.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.223+0000 7f714bd97640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f71440b05a0 0x7f71440aa8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:36.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.223+0000 7f714bd97640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f71441e3060 0x7f7144157600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:36.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.223+0000 7f714bd97640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71441e2a60 con 0x7f71440b01c0 2023-10-19T14:18:36.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.223+0000 7f714bd97640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71441e3be0 con 0x7f71440b05a0 2023-10-19T14:18:36.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.223+0000 7f714bd97640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71441e3620 con 0x7f71441e3060 2023-10-19T14:18:36.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.223+0000 7f7149b33640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f71440b01c0 0x7f71441e2520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:36.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.223+0000 7f7149b33640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f71440b01c0 0x7f71441e2520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51416/0 (socket says 172.21.15.57:51416) 2023-10-19T14:18:36.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.223+0000 7f7149b33640 1 -- 172.21.15.57:0/4189462854 learned_addr learned my addr 172.21.15.57:0/4189462854 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:18:36.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.223+0000 7f7149332640 1 --2- 172.21.15.57:0/4189462854 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f71440b05a0 0x7f71440aa8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:36.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.223+0000 7f7149b33640 1 -- 172.21.15.57:0/4189462854 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f71441e3060 msgr2=0x7f7144157600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:36.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.223+0000 7f714a334640 1 --2- 172.21.15.57:0/4189462854 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f71441e3060 0x7f7144157600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:36.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.223+0000 7f7149b33640 1 --2- 172.21.15.57:0/4189462854 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f71441e3060 0x7f7144157600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:36.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.223+0000 7f7149b33640 1 -- 172.21.15.57:0/4189462854 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f71440b05a0 msgr2=0x7f71440aa8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:36.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.223+0000 7f7149b33640 1 --2- 172.21.15.57:0/4189462854 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f71440b05a0 0x7f71440aa8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:36.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.223+0000 7f7149b33640 1 -- 172.21.15.57:0/4189462854 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71441e3900 con 0x7f71440b01c0 2023-10-19T14:18:36.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.224+0000 7f7149332640 1 --2- 172.21.15.57:0/4189462854 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f71440b05a0 0x7f71440aa8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:18:36.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.224+0000 7f714a334640 1 --2- 172.21.15.57:0/4189462854 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f71441e3060 0x7f7144157600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:18:36.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.224+0000 7f7149b33640 1 --2- 172.21.15.57:0/4189462854 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f71440b01c0 0x7f71441e2520 secure :-1 s=READY pgs=2179 cs=0 l=1 rev1=1 crypto rx=0x7f7140011ae0 tx=0x7f7140009e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:36.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.224+0000 7f713a7fc640 1 -- 172.21.15.57:0/4189462854 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f714000fe70 con 0x7f71440b01c0 2023-10-19T14:18:36.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.224+0000 7f713a7fc640 1 -- 172.21.15.57:0/4189462854 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f7140004050 con 0x7f71440b01c0 2023-10-19T14:18:36.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.224+0000 7f714bd97640 1 -- 172.21.15.57:0/4189462854 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f714413c7a0 con 0x7f71440b01c0 2023-10-19T14:18:36.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.224+0000 7f713a7fc640 1 -- 172.21.15.57:0/4189462854 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7140004460 con 0x7f71440b01c0 2023-10-19T14:18:36.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.224+0000 7f714bd97640 1 -- 172.21.15.57:0/4189462854 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71440aae00 con 0x7f71440b01c0 2023-10-19T14:18:36.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.225+0000 7f714bd97640 1 -- 172.21.15.57:0/4189462854 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f714406b740 con 0x7f71440b01c0 2023-10-19T14:18:36.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.225+0000 7f713a7fc640 1 -- 172.21.15.57:0/4189462854 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 117) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f71400035f0 con 0x7f71440b01c0 2023-10-19T14:18:36.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.226+0000 7f713a7fc640 1 --2- 172.21.15.57:0/4189462854 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f712806af60 0x7f712806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:36.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.226+0000 7f713a7fc640 1 -- 172.21.15.57:0/4189462854 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f714008a3b0 con 0x7f71440b01c0 2023-10-19T14:18:36.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.226+0000 7f7149332640 1 --2- 172.21.15.57:0/4189462854 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f712806af60 0x7f712806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:36.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.226+0000 7f7149332640 1 --2- 172.21.15.57:0/4189462854 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f712806af60 0x7f712806d420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f714406a420 tx=0x7f7134005f90 comp rx=0 tx=0).ready entity=mgr.8744 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:36.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.230+0000 7f713a7fc640 1 -- 172.21.15.57:0/4189462854 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7140055630 con 0x7f71440b01c0 2023-10-19T14:18:36.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.407+0000 7f714bd97640 1 -- 172.21.15.57:0/4189462854 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f71440b1bd0 con 0x7f71440b01c0 2023-10-19T14:18:36.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.407+0000 7f713a7fc640 1 -- 172.21.15.57:0/4189462854 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v201) v1 ==== 93+0+10 (secure 0 0 0) 0x7f714001e020 con 0x7f71440b01c0 2023-10-19T14:18:36.411 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-19T14:18:36.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.409+0000 7f714bd97640 1 -- 172.21.15.57:0/4189462854 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f712806af60 msgr2=0x7f712806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:36.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.409+0000 7f714bd97640 1 --2- 172.21.15.57:0/4189462854 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f712806af60 0x7f712806d420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f714406a420 tx=0x7f7134005f90 comp rx=0 tx=0).stop 2023-10-19T14:18:36.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.409+0000 7f714bd97640 1 -- 172.21.15.57:0/4189462854 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f71440b01c0 msgr2=0x7f71441e2520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:36.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.409+0000 7f714bd97640 1 --2- 172.21.15.57:0/4189462854 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f71440b01c0 0x7f71441e2520 secure :-1 s=READY pgs=2179 cs=0 l=1 rev1=1 crypto rx=0x7f7140011ae0 tx=0x7f7140009e90 comp rx=0 tx=0).stop 2023-10-19T14:18:36.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.409+0000 7f714bd97640 1 -- 172.21.15.57:0/4189462854 shutdown_connections 2023-10-19T14:18:36.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.409+0000 7f714bd97640 1 --2- 172.21.15.57:0/4189462854 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f712806af60 0x7f712806d420 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:36.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.409+0000 7f714bd97640 1 --2- 172.21.15.57:0/4189462854 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f71441e3060 0x7f7144157600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:36.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.409+0000 7f714bd97640 1 --2- 172.21.15.57:0/4189462854 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f71440b05a0 0x7f71440aa8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:36.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.409+0000 7f714bd97640 1 --2- 172.21.15.57:0/4189462854 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f71440b01c0 0x7f71441e2520 unknown :-1 s=CLOSED pgs=2179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:36.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.409+0000 7f714bd97640 1 -- 172.21.15.57:0/4189462854 >> 172.21.15.57:0/4189462854 conn(0x7f7144069930 msgr2=0x7f71440ac660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:36.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.409+0000 7f714bd97640 1 -- 172.21.15.57:0/4189462854 shutdown_connections 2023-10-19T14:18:36.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.409+0000 7f714bd97640 1 -- 172.21.15.57:0/4189462854 wait complete. 2023-10-19T14:18:36.423 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-19T14:18:36.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.530+0000 7f299cb83640 1 -- 172.21.15.57:0/4068891102 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f29800039a0 msgr2=0x7f2980003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:36.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.530+0000 7f299cb83640 1 --2- 172.21.15.57:0/4068891102 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f29800039a0 0x7f2980003dc0 secure :-1 s=READY pgs=2181 cs=0 l=1 rev1=1 crypto rx=0x7f2988001650 tx=0x7f29800057f0 comp rx=0 tx=0).stop 2023-10-19T14:18:36.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.530+0000 7f2996d9d640 1 -- 172.21.15.57:0/4068891102 reap_dead start 2023-10-19T14:18:36.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.530+0000 7f299cb83640 1 -- 172.21.15.57:0/4068891102 shutdown_connections 2023-10-19T14:18:36.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.530+0000 7f299cb83640 1 -- 172.21.15.57:0/4068891102 >> 172.21.15.57:0/4068891102 conn(0x7f299806fae0 msgr2=0x7f299806fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:36.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.530+0000 7f299cb83640 1 -- 172.21.15.57:0/4068891102 shutdown_connections 2023-10-19T14:18:36.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.530+0000 7f299cb83640 1 -- 172.21.15.57:0/4068891102 wait complete. 2023-10-19T14:18:36.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.531+0000 7f299cb83640 1 Processor -- start 2023-10-19T14:18:36.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.532+0000 7f299cb83640 1 -- start start 2023-10-19T14:18:36.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.532+0000 7f299cb83640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f29800039a0 0x7f2998149700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:36.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.532+0000 7f299cb83640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2998152010 0x7f2998152490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:36.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.532+0000 7f299cb83640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f29980aa8c0 0x7f29980aad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:36.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.532+0000 7f299cb83640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29980ab280 con 0x7f29980aa8c0 2023-10-19T14:18:36.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.532+0000 7f299cb83640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29980b0580 con 0x7f2998152010 2023-10-19T14:18:36.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.532+0000 7f299cb83640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29980b0860 con 0x7f29800039a0 2023-10-19T14:18:36.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.532+0000 7f2996d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f29980aa8c0 0x7f29980aad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:36.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.532+0000 7f2996d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f29980aa8c0 0x7f29980aad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51434/0 (socket says 172.21.15.57:51434) 2023-10-19T14:18:36.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.532+0000 7f2996d9d640 1 -- 172.21.15.57:0/2272834921 learned_addr learned my addr 172.21.15.57:0/2272834921 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:18:36.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.533+0000 7f299659c640 1 --2- 172.21.15.57:0/2272834921 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f29800039a0 0x7f2998149700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:36.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.533+0000 7f2995d9b640 1 --2- 172.21.15.57:0/2272834921 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2998152010 0x7f2998152490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:36.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.533+0000 7f2996d9d640 1 -- 172.21.15.57:0/2272834921 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f29800039a0 msgr2=0x7f2998149700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:36.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.533+0000 7f2996d9d640 1 --2- 172.21.15.57:0/2272834921 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f29800039a0 0x7f2998149700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:36.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.533+0000 7f2996d9d640 1 -- 172.21.15.57:0/2272834921 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2998152010 msgr2=0x7f2998152490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:36.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.533+0000 7f2996d9d640 1 --2- 172.21.15.57:0/2272834921 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2998152010 0x7f2998152490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:36.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.533+0000 7f2996d9d640 1 -- 172.21.15.57:0/2272834921 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f299806b0d0 con 0x7f29980aa8c0 2023-10-19T14:18:36.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.533+0000 7f2996d9d640 1 --2- 172.21.15.57:0/2272834921 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f29980aa8c0 0x7f29980aad40 secure :-1 s=READY pgs=2182 cs=0 l=1 rev1=1 crypto rx=0x7f2990005fd0 tx=0x7f2990011f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:36.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.533+0000 7f2986ffd640 1 -- 172.21.15.57:0/2272834921 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f299000fd70 con 0x7f29980aa8c0 2023-10-19T14:18:36.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.533+0000 7f2986ffd640 1 -- 172.21.15.57:0/2272834921 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f299000fed0 con 0x7f29980aa8c0 2023-10-19T14:18:36.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.533+0000 7f2986ffd640 1 -- 172.21.15.57:0/2272834921 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2990004d60 con 0x7f29980aa8c0 2023-10-19T14:18:36.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.533+0000 7f299cb83640 1 -- 172.21.15.57:0/2272834921 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2998151380 con 0x7f29980aa8c0 2023-10-19T14:18:36.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.533+0000 7f299cb83640 1 -- 172.21.15.57:0/2272834921 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2998151940 con 0x7f29980aa8c0 2023-10-19T14:18:36.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.534+0000 7f299cb83640 1 -- 172.21.15.57:0/2272834921 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29980b09f0 con 0x7f29980aa8c0 2023-10-19T14:18:36.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.534+0000 7f2986ffd640 1 -- 172.21.15.57:0/2272834921 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 117) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f2990019410 con 0x7f29980aa8c0 2023-10-19T14:18:36.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.535+0000 7f2986ffd640 1 --2- 172.21.15.57:0/2272834921 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f297806af90 0x7f297806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:36.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.535+0000 7f2986ffd640 1 -- 172.21.15.57:0/2272834921 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f299008a0e0 con 0x7f29980aa8c0 2023-10-19T14:18:36.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.535+0000 7f299659c640 1 --2- 172.21.15.57:0/2272834921 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f297806af90 0x7f297806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:36.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.536+0000 7f299659c640 1 --2- 172.21.15.57:0/2272834921 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f297806af90 0x7f297806d450 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2988034000 tx=0x7f2988039040 comp rx=0 tx=0).ready entity=mgr.8744 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:36.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.539+0000 7f2986ffd640 1 -- 172.21.15.57:0/2272834921 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f29900551f0 con 0x7f29980aa8c0 2023-10-19T14:18:36.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.716+0000 7f299cb83640 1 -- 172.21.15.57:0/2272834921 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f29980690e0 con 0x7f29980aa8c0 2023-10-19T14:18:36.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.717+0000 7f2986ffd640 1 -- 172.21.15.57:0/2272834921 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v201) v1 ==== 84+0+65 (secure 0 0 0) 0x7f2990005e30 con 0x7f29980aa8c0 2023-10-19T14:18:36.720 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:18:36.720 INFO:teuthology.orchestra.run.smithi057.stdout:[ 2023-10-19T14:18:36.721 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:18:36.721 INFO:teuthology.orchestra.run.smithi057.stdout: "poolnum": 1, 2023-10-19T14:18:36.721 INFO:teuthology.orchestra.run.smithi057.stdout: "poolname": ".mgr" 2023-10-19T14:18:36.721 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:18:36.722 INFO:teuthology.orchestra.run.smithi057.stdout:] 2023-10-19T14:18:36.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.719+0000 7f299cb83640 1 -- 172.21.15.57:0/2272834921 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f297806af90 msgr2=0x7f297806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:36.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.719+0000 7f299cb83640 1 --2- 172.21.15.57:0/2272834921 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f297806af90 0x7f297806d450 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2988034000 tx=0x7f2988039040 comp rx=0 tx=0).stop 2023-10-19T14:18:36.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.719+0000 7f299cb83640 1 -- 172.21.15.57:0/2272834921 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f29980aa8c0 msgr2=0x7f29980aad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:36.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.719+0000 7f299cb83640 1 --2- 172.21.15.57:0/2272834921 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f29980aa8c0 0x7f29980aad40 secure :-1 s=READY pgs=2182 cs=0 l=1 rev1=1 crypto rx=0x7f2990005fd0 tx=0x7f2990011f90 comp rx=0 tx=0).stop 2023-10-19T14:18:36.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.719+0000 7f299cb83640 1 -- 172.21.15.57:0/2272834921 shutdown_connections 2023-10-19T14:18:36.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.719+0000 7f299cb83640 1 --2- 172.21.15.57:0/2272834921 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f297806af90 0x7f297806d450 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:36.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.719+0000 7f299cb83640 1 --2- 172.21.15.57:0/2272834921 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f29980aa8c0 0x7f29980aad40 unknown :-1 s=CLOSED pgs=2182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:36.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.719+0000 7f299cb83640 1 --2- 172.21.15.57:0/2272834921 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2998152010 0x7f2998152490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:36.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.719+0000 7f299cb83640 1 --2- 172.21.15.57:0/2272834921 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f29800039a0 0x7f2998149700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:36.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.719+0000 7f299cb83640 1 -- 172.21.15.57:0/2272834921 >> 172.21.15.57:0/2272834921 conn(0x7f299806fae0 msgr2=0x7f29981581f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:36.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.719+0000 7f299cb83640 1 -- 172.21.15.57:0/2272834921 shutdown_connections 2023-10-19T14:18:36.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.719+0000 7f299cb83640 1 -- 172.21.15.57:0/2272834921 wait complete. 2023-10-19T14:18:36.732 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-19T14:18:36.733 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-19T14:18:36.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.843+0000 7f71062af640 1 -- 172.21.15.57:0/1357840902 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f71000c9750 msgr2=0x7f71000c9b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:36.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.843+0000 7f71062af640 1 --2- 172.21.15.57:0/1357840902 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f71000c9750 0x7f71000c9b70 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 crypto rx=0x7f71000cbd30 tx=0x7f70f802c260 comp rx=0 tx=0).stop 2023-10-19T14:18:36.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.843+0000 7f71062af640 1 -- 172.21.15.57:0/1357840902 shutdown_connections 2023-10-19T14:18:36.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.843+0000 7f71062af640 1 --2- 172.21.15.57:0/1357840902 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f71000c9750 0x7f71000c9b70 unknown :-1 s=CLOSED pgs=2184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:36.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.843+0000 7f71062af640 1 --2- 172.21.15.57:0/1357840902 >> v2:172.21.15.179:3300/0 conn(0x7f710006b030 0x7f710006b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:36.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.843+0000 7f71062af640 1 -- 172.21.15.57:0/1357840902 >> 172.21.15.57:0/1357840902 conn(0x7f7100069930 msgr2=0x7f7100069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:36.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.843+0000 7f71062af640 1 -- 172.21.15.57:0/1357840902 shutdown_connections 2023-10-19T14:18:36.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.843+0000 7f71062af640 1 -- 172.21.15.57:0/1357840902 wait complete. 2023-10-19T14:18:36.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.844+0000 7f71062af640 1 Processor -- start 2023-10-19T14:18:36.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.844+0000 7f71062af640 1 -- start start 2023-10-19T14:18:36.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.845+0000 7f71062af640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f710006b030 0x7f71000a64b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:36.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.845+0000 7f71062af640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f71000c9750 0x7f71001f2d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:36.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.845+0000 7f71062af640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7100159bd0 0x7f71001f32b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:36.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.845+0000 7f71062af640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7100150c00 con 0x7f710006b030 2023-10-19T14:18:36.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.845+0000 7f71062af640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71000ab270 con 0x7f7100159bd0 2023-10-19T14:18:36.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.845+0000 7f71062af640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71001502d0 con 0x7f71000c9750 2023-10-19T14:18:36.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.845+0000 7f70fffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f710006b030 0x7f71000a64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:36.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.845+0000 7f70fffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f710006b030 0x7f71000a64b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55400/0 (socket says 172.21.15.57:55400) 2023-10-19T14:18:36.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.845+0000 7f70fffff640 1 -- 172.21.15.57:0/4152473153 learned_addr learned my addr 172.21.15.57:0/4152473153 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:18:36.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.846+0000 7f70fffff640 1 -- 172.21.15.57:0/4152473153 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f71000c9750 msgr2=0x7f71001f2d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:36.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.846+0000 7f70fffff640 1 --2- 172.21.15.57:0/4152473153 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f71000c9750 0x7f71001f2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:36.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.846+0000 7f70fffff640 1 -- 172.21.15.57:0/4152473153 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7100159bd0 msgr2=0x7f71001f32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:36.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.846+0000 7f710484c640 1 --2- 172.21.15.57:0/4152473153 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7100159bd0 0x7f71001f32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:36.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.846+0000 7f70fffff640 1 --2- 172.21.15.57:0/4152473153 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7100159bd0 0x7f71001f32b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:36.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.846+0000 7f70fffff640 1 -- 172.21.15.57:0/4152473153 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70f8009900 con 0x7f710006b030 2023-10-19T14:18:36.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.846+0000 7f70fffff640 1 --2- 172.21.15.57:0/4152473153 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f710006b030 0x7f71000a64b0 secure :-1 s=READY pgs=2185 cs=0 l=1 rev1=1 crypto rx=0x7f70f000d950 tx=0x7f70f000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:36.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.846+0000 7f70fcff9640 1 -- 172.21.15.57:0/4152473153 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70f0009c30 con 0x7f710006b030 2023-10-19T14:18:36.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.846+0000 7f70fcff9640 1 -- 172.21.15.57:0/4152473153 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f70f0009e80 con 0x7f710006b030 2023-10-19T14:18:36.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.846+0000 7f71062af640 1 -- 172.21.15.57:0/4152473153 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71001505b0 con 0x7f710006b030 2023-10-19T14:18:36.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.846+0000 7f70fcff9640 1 -- 172.21.15.57:0/4152473153 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70f00106c0 con 0x7f710006b030 2023-10-19T14:18:36.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.846+0000 7f71062af640 1 -- 172.21.15.57:0/4152473153 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71000a69f0 con 0x7f710006b030 2023-10-19T14:18:36.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.847+0000 7f71062af640 1 -- 172.21.15.57:0/4152473153 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71001e4590 con 0x7f710006b030 2023-10-19T14:18:36.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.847+0000 7f70fcff9640 1 -- 172.21.15.57:0/4152473153 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 117) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f70f0010820 con 0x7f710006b030 2023-10-19T14:18:36.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.848+0000 7f70fcff9640 1 --2- 172.21.15.57:0/4152473153 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f70e006af60 0x7f70e006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:36.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.848+0000 7f70fcff9640 1 -- 172.21.15.57:0/4152473153 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f70f0089b40 con 0x7f710006b030 2023-10-19T14:18:36.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.848+0000 7f70ff7fe640 1 --2- 172.21.15.57:0/4152473153 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f70e006af60 0x7f70e006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:36.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.848+0000 7f70ff7fe640 1 --2- 172.21.15.57:0/4152473153 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f70e006af60 0x7f70e006d420 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f70e8000f60 tx=0x7f70e80079b0 comp rx=0 tx=0).ready entity=mgr.8744 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:36.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:36.852+0000 7f70fcff9640 1 -- 172.21.15.57:0/4152473153 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f70f0054c50 con 0x7f710006b030 2023-10-19T14:18:37.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:37.030+0000 7f71062af640 1 -- 172.21.15.57:0/4152473153 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f710006c850 con 0x7f710006b030 2023-10-19T14:18:37.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:37.454+0000 7f70fcff9640 1 -- 172.21.15.57:0/4152473153 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v202) v1 ==== 138+0+0 (secure 0 0 0) 0x7f70f00170b0 con 0x7f710006b030 2023-10-19T14:18:37.457 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'cephfs_metadata' created 2023-10-19T14:18:37.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:37.457+0000 7f71062af640 1 -- 172.21.15.57:0/4152473153 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f70e006af60 msgr2=0x7f70e006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:37.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:37.457+0000 7f71062af640 1 --2- 172.21.15.57:0/4152473153 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f70e006af60 0x7f70e006d420 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f70e8000f60 tx=0x7f70e80079b0 comp rx=0 tx=0).stop 2023-10-19T14:18:37.461 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:37.457+0000 7f71062af640 1 -- 172.21.15.57:0/4152473153 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f710006b030 msgr2=0x7f71000a64b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:37.461 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:37.457+0000 7f71062af640 1 --2- 172.21.15.57:0/4152473153 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f710006b030 0x7f71000a64b0 secure :-1 s=READY pgs=2185 cs=0 l=1 rev1=1 crypto rx=0x7f70f000d950 tx=0x7f70f000de10 comp rx=0 tx=0).stop 2023-10-19T14:18:37.461 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:37.457+0000 7f71062af640 1 -- 172.21.15.57:0/4152473153 shutdown_connections 2023-10-19T14:18:37.461 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:37.457+0000 7f71062af640 1 --2- 172.21.15.57:0/4152473153 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f70e006af60 0x7f70e006d420 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:37.461 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:37.457+0000 7f71062af640 1 --2- 172.21.15.57:0/4152473153 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7100159bd0 0x7f71001f32b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:37.462 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:37.457+0000 7f71062af640 1 --2- 172.21.15.57:0/4152473153 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f71000c9750 0x7f71001f2d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:37.462 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:37.457+0000 7f71062af640 1 --2- 172.21.15.57:0/4152473153 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f710006b030 0x7f71000a64b0 unknown :-1 s=CLOSED pgs=2185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:37.462 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:37.457+0000 7f71062af640 1 -- 172.21.15.57:0/4152473153 >> 172.21.15.57:0/4152473153 conn(0x7f7100069930 msgr2=0x7f710015baf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:37.462 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:37.458+0000 7f71062af640 1 -- 172.21.15.57:0/4152473153 shutdown_connections 2023-10-19T14:18:37.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:37.458+0000 7f71062af640 1 -- 172.21.15.57:0/4152473153 wait complete. 2023-10-19T14:18:37.475 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-19T14:18:37.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:37.589+0000 7fd5e2e64640 1 -- 172.21.15.57:0/1527255324 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5dc0ceb40 msgr2=0x7fd5dc0b0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:37.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:37.589+0000 7fd5e2e64640 1 --2- 172.21.15.57:0/1527255324 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5dc0ceb40 0x7fd5dc0b0650 secure :-1 s=READY pgs=2186 cs=0 l=1 rev1=1 crypto rx=0x7fd5dc08a810 tx=0x7fd5d402bdb0 comp rx=0 tx=0).stop 2023-10-19T14:18:37.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:37.589+0000 7fd5e2e64640 1 -- 172.21.15.57:0/1527255324 shutdown_connections 2023-10-19T14:18:37.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:37.589+0000 7fd5e2e64640 1 --2- 172.21.15.57:0/1527255324 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5dc0ceb40 0x7fd5dc0b0650 unknown :-1 s=CLOSED pgs=2186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:37.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:37.589+0000 7fd5e2e64640 1 --2- 172.21.15.57:0/1527255324 >> v2:172.21.15.179:3300/0 conn(0x7fd5dc0cf340 0x7fd5dc0d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:37.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:37.589+0000 7fd5e2e64640 1 -- 172.21.15.57:0/1527255324 >> 172.21.15.57:0/1527255324 conn(0x7fd5dc0d0d60 msgr2=0x7fd5dc0d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:37.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:37.589+0000 7fd5e2e64640 1 -- 172.21.15.57:0/1527255324 shutdown_connections 2023-10-19T14:18:37.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:37.589+0000 7fd5e2e64640 1 -- 172.21.15.57:0/1527255324 wait complete. 2023-10-19T14:18:37.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:37.591+0000 7fd5e2e64640 1 Processor -- start 2023-10-19T14:18:37.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:37.591+0000 7fd5e2e64640 1 -- start start 2023-10-19T14:18:37.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:37.591+0000 7fd5e2e64640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd5dc0ceb40 0x7fd5dc157310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:37.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:37.591+0000 7fd5e2e64640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5dc0cf340 0x7fd5dc0cc790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:37.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:37.591+0000 7fd5e2e64640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd5dc15b2a0 0x7fd5dc0cccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:37.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:37.591+0000 7fd5e2e64640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5dc1586e0 con 0x7fd5dc0cf340 2023-10-19T14:18:37.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:37.591+0000 7fd5e2e64640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5dc0b3a80 con 0x7fd5dc0ceb40 2023-10-19T14:18:37.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:37.591+0000 7fd5e2e64640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5dc0caac0 con 0x7fd5dc15b2a0 2023-10-19T14:18:37.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:37.592+0000 7fd5dbfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5dc0cf340 0x7fd5dc0cc790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:37.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:37.592+0000 7fd5dbfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5dc0cf340 0x7fd5dc0cc790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55412/0 (socket says 172.21.15.57:55412) 2023-10-19T14:18:37.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:37.592+0000 7fd5dbfff640 1 -- 172.21.15.57:0/2380838897 learned_addr learned my addr 172.21.15.57:0/2380838897 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:18:37.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:37.592+0000 7fd5e0c00640 1 --2- 172.21.15.57:0/2380838897 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd5dc0ceb40 0x7fd5dc157310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:37.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:37.592+0000 7fd5e1401640 1 --2- 172.21.15.57:0/2380838897 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd5dc15b2a0 0x7fd5dc0cccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:37.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:37.592+0000 7fd5dbfff640 1 -- 172.21.15.57:0/2380838897 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd5dc15b2a0 msgr2=0x7fd5dc0cccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:37.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:37.592+0000 7fd5dbfff640 1 --2- 172.21.15.57:0/2380838897 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd5dc15b2a0 0x7fd5dc0cccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:37.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:37.592+0000 7fd5dbfff640 1 -- 172.21.15.57:0/2380838897 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd5dc0ceb40 msgr2=0x7fd5dc157310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:37.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:37.592+0000 7fd5dbfff640 1 --2- 172.21.15.57:0/2380838897 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd5dc0ceb40 0x7fd5dc157310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:37.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:37.592+0000 7fd5dbfff640 1 -- 172.21.15.57:0/2380838897 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5d4009560 con 0x7fd5dc0cf340 2023-10-19T14:18:37.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:37.592+0000 7fd5dbfff640 1 --2- 172.21.15.57:0/2380838897 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5dc0cf340 0x7fd5dc0cc790 secure :-1 s=READY pgs=2187 cs=0 l=1 rev1=1 crypto rx=0x7fd5c400c400 tx=0x7fd5c400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:37.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:37.592+0000 7fd5d97fa640 1 -- 172.21.15.57:0/2380838897 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5c40052d0 con 0x7fd5dc0cf340 2023-10-19T14:18:37.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:37.592+0000 7fd5d97fa640 1 -- 172.21.15.57:0/2380838897 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fd5c40035f0 con 0x7fd5dc0cf340 2023-10-19T14:18:37.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:37.592+0000 7fd5d97fa640 1 -- 172.21.15.57:0/2380838897 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5c4003a00 con 0x7fd5dc0cf340 2023-10-19T14:18:37.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:37.592+0000 7fd5e2e64640 1 -- 172.21.15.57:0/2380838897 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5dc157b60 con 0x7fd5dc0cf340 2023-10-19T14:18:37.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:37.592+0000 7fd5e2e64640 1 -- 172.21.15.57:0/2380838897 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5dc158120 con 0x7fd5dc0cf340 2023-10-19T14:18:37.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:37.593+0000 7fd5e2e64640 1 -- 172.21.15.57:0/2380838897 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5dc0690e0 con 0x7fd5dc0cf340 2023-10-19T14:18:37.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:37.594+0000 7fd5d97fa640 1 -- 172.21.15.57:0/2380838897 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 117) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd5c4003b60 con 0x7fd5dc0cf340 2023-10-19T14:18:37.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:37.595+0000 7fd5d97fa640 1 --2- 172.21.15.57:0/2380838897 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fd5bc06af60 0x7fd5bc06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:37.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:37.595+0000 7fd5d97fa640 1 -- 172.21.15.57:0/2380838897 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 5575+0+0 (secure 0 0 0) 0x7fd5c4089240 con 0x7fd5dc0cf340 2023-10-19T14:18:37.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:37.595+0000 7fd5e0c00640 1 --2- 172.21.15.57:0/2380838897 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fd5bc06af60 0x7fd5bc06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:37.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:37.596+0000 7fd5e0c00640 1 --2- 172.21.15.57:0/2380838897 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fd5bc06af60 0x7fd5bc06d420 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd5cc005fd0 tx=0x7fd5cc009500 comp rx=0 tx=0).ready entity=mgr.8744 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:37.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:37.599+0000 7fd5d97fa640 1 -- 172.21.15.57:0/2380838897 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd5c40543f0 con 0x7fd5dc0cf340 2023-10-19T14:18:37.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:37.775+0000 7fd5e2e64640 1 -- 172.21.15.57:0/2380838897 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fd5dc0b0d00 con 0x7fd5dc0cf340 2023-10-19T14:18:38.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:38.461+0000 7fd5d97fa640 1 -- 172.21.15.57:0/2380838897 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v203) v1 ==== 170+0+0 (secure 0 0 0) 0x7fd5c4057a10 con 0x7fd5dc0cf340 2023-10-19T14:18:38.464 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'cephfs_data' created 2023-10-19T14:18:38.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:38.465+0000 7fd5e2e64640 1 -- 172.21.15.57:0/2380838897 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fd5bc06af60 msgr2=0x7fd5bc06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:38.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:38.465+0000 7fd5e2e64640 1 --2- 172.21.15.57:0/2380838897 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fd5bc06af60 0x7fd5bc06d420 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd5cc005fd0 tx=0x7fd5cc009500 comp rx=0 tx=0).stop 2023-10-19T14:18:38.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:38.465+0000 7fd5e2e64640 1 -- 172.21.15.57:0/2380838897 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5dc0cf340 msgr2=0x7fd5dc0cc790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:38.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:38.465+0000 7fd5e2e64640 1 --2- 172.21.15.57:0/2380838897 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5dc0cf340 0x7fd5dc0cc790 secure :-1 s=READY pgs=2187 cs=0 l=1 rev1=1 crypto rx=0x7fd5c400c400 tx=0x7fd5c400c8c0 comp rx=0 tx=0).stop 2023-10-19T14:18:38.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:38.466+0000 7fd5e2e64640 1 -- 172.21.15.57:0/2380838897 shutdown_connections 2023-10-19T14:18:38.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:38.466+0000 7fd5e2e64640 1 --2- 172.21.15.57:0/2380838897 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fd5bc06af60 0x7fd5bc06d420 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:38.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:38.466+0000 7fd5e2e64640 1 --2- 172.21.15.57:0/2380838897 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd5dc15b2a0 0x7fd5dc0cccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:38.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:38.466+0000 7fd5e2e64640 1 --2- 172.21.15.57:0/2380838897 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5dc0cf340 0x7fd5dc0cc790 unknown :-1 s=CLOSED pgs=2187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:38.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:38.466+0000 7fd5e2e64640 1 --2- 172.21.15.57:0/2380838897 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd5dc0ceb40 0x7fd5dc157310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:38.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:38.466+0000 7fd5e2e64640 1 -- 172.21.15.57:0/2380838897 >> 172.21.15.57:0/2380838897 conn(0x7fd5dc0d0d60 msgr2=0x7fd5dc15a2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:38.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:38.466+0000 7fd5e2e64640 1 -- 172.21.15.57:0/2380838897 shutdown_connections 2023-10-19T14:18:38.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:38.467+0000 7fd5e2e64640 1 -- 172.21.15.57:0/2380838897 wait complete. 2023-10-19T14:18:38.496 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-19T14:18:38.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:38.610+0000 7f5c5f0b4640 1 -- 172.21.15.57:0/868360824 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5c5815beb0 msgr2=0x7f5c58070a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:38.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:38.610+0000 7f5c5f0b4640 1 --2- 172.21.15.57:0/868360824 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5c5815beb0 0x7f5c58070a60 secure :-1 s=READY pgs=2189 cs=0 l=1 rev1=1 crypto rx=0x7f5c580b2630 tx=0x7f5c4002bdb0 comp rx=0 tx=0).stop 2023-10-19T14:18:38.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:38.610+0000 7f5c5f0b4640 1 -- 172.21.15.57:0/868360824 shutdown_connections 2023-10-19T14:18:38.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:38.610+0000 7f5c5f0b4640 1 --2- 172.21.15.57:0/868360824 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5c5815beb0 0x7f5c58070a60 unknown :-1 s=CLOSED pgs=2189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:38.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:38.610+0000 7f5c5f0b4640 1 --2- 172.21.15.57:0/868360824 >> v2:172.21.15.179:3300/0 conn(0x7f5c58157bb0 0x7f5c58070520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:38.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:38.610+0000 7f5c5f0b4640 1 -- 172.21.15.57:0/868360824 >> 172.21.15.57:0/868360824 conn(0x7f5c580aec40 msgr2=0x7f5c580af050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:38.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:38.610+0000 7f5c5f0b4640 1 -- 172.21.15.57:0/868360824 shutdown_connections 2023-10-19T14:18:38.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:38.610+0000 7f5c5f0b4640 1 -- 172.21.15.57:0/868360824 wait complete. 2023-10-19T14:18:38.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:38.611+0000 7f5c5f0b4640 1 Processor -- start 2023-10-19T14:18:38.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:38.611+0000 7f5c5f0b4640 1 -- start start 2023-10-19T14:18:38.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:38.612+0000 7f5c5f0b4640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5c5806c960 0x7f5c581e3e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:38.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:38.612+0000 7f5c5f0b4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5c58157bb0 0x7f5c58148030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:38.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:38.612+0000 7f5c5f0b4640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5c5815beb0 0x7f5c58148570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:38.618 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:38.612+0000 7f5c5f0b4640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c581570d0 con 0x7f5c58157bb0 2023-10-19T14:18:38.618 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:38.612+0000 7f5c5f0b4640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c581e4370 con 0x7f5c5806c960 2023-10-19T14:18:38.618 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:38.612+0000 7f5c5f0b4640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c581564c0 con 0x7f5c5815beb0 2023-10-19T14:18:38.618 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:38.612+0000 7f5c57fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5c58157bb0 0x7f5c58148030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:38.618 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:38.612+0000 7f5c57fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5c58157bb0 0x7f5c58148030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55436/0 (socket says 172.21.15.57:55436) 2023-10-19T14:18:38.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:38.612+0000 7f5c57fff640 1 -- 172.21.15.57:0/1859688008 learned_addr learned my addr 172.21.15.57:0/1859688008 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:18:38.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:38.612+0000 7f5c5ce50640 1 --2- 172.21.15.57:0/1859688008 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5c5806c960 0x7f5c581e3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:38.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:38.612+0000 7f5c5d651640 1 --2- 172.21.15.57:0/1859688008 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5c5815beb0 0x7f5c58148570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:38.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:38.613+0000 7f5c57fff640 1 -- 172.21.15.57:0/1859688008 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5c5815beb0 msgr2=0x7f5c58148570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:38.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:38.613+0000 7f5c57fff640 1 --2- 172.21.15.57:0/1859688008 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5c5815beb0 0x7f5c58148570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:38.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:38.613+0000 7f5c57fff640 1 -- 172.21.15.57:0/1859688008 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5c5806c960 msgr2=0x7f5c581e3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:38.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:38.613+0000 7f5c57fff640 1 --2- 172.21.15.57:0/1859688008 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5c5806c960 0x7f5c581e3e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:38.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:38.613+0000 7f5c57fff640 1 -- 172.21.15.57:0/1859688008 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c40009560 con 0x7f5c58157bb0 2023-10-19T14:18:38.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:38.613+0000 7f5c5d651640 1 --2- 172.21.15.57:0/1859688008 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5c5815beb0 0x7f5c58148570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:18:38.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:38.613+0000 7f5c57fff640 1 --2- 172.21.15.57:0/1859688008 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5c58157bb0 0x7f5c58148030 secure :-1 s=READY pgs=2190 cs=0 l=1 rev1=1 crypto rx=0x7f5c4800c4b0 tx=0x7f5c4800c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:38.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:38.613+0000 7f5c557fa640 1 -- 172.21.15.57:0/1859688008 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c48005480 con 0x7f5c58157bb0 2023-10-19T14:18:38.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:38.613+0000 7f5c557fa640 1 -- 172.21.15.57:0/1859688008 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f5c480036e0 con 0x7f5c58157bb0 2023-10-19T14:18:38.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:38.613+0000 7f5c5f0b4640 1 -- 172.21.15.57:0/1859688008 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c581567a0 con 0x7f5c58157bb0 2023-10-19T14:18:38.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:38.613+0000 7f5c557fa640 1 -- 172.21.15.57:0/1859688008 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c48003b30 con 0x7f5c58157bb0 2023-10-19T14:18:38.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:38.613+0000 7f5c5f0b4640 1 -- 172.21.15.57:0/1859688008 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c58156d60 con 0x7f5c58157bb0 2023-10-19T14:18:38.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:38.614+0000 7f5c5f0b4640 1 -- 172.21.15.57:0/1859688008 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c580ca910 con 0x7f5c58157bb0 2023-10-19T14:18:38.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:38.615+0000 7f5c557fa640 1 -- 172.21.15.57:0/1859688008 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 117) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f5c48003c90 con 0x7f5c58157bb0 2023-10-19T14:18:38.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:38.615+0000 7f5c557fa640 1 --2- 172.21.15.57:0/1859688008 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f5c3806af60 0x7f5c3806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:38.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:38.615+0000 7f5c557fa640 1 -- 172.21.15.57:0/1859688008 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(203..203 src has 1..203) v4 ==== 5957+0+0 (secure 0 0 0) 0x7f5c480894d0 con 0x7f5c58157bb0 2023-10-19T14:18:38.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:38.615+0000 7f5c5ce50640 1 --2- 172.21.15.57:0/1859688008 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f5c3806af60 0x7f5c3806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:38.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:38.616+0000 7f5c5ce50640 1 --2- 172.21.15.57:0/1859688008 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f5c3806af60 0x7f5c3806d420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5c580ad530 tx=0x7f5c500075b0 comp rx=0 tx=0).ready entity=mgr.8744 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:38.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:38.619+0000 7f5c557fa640 1 -- 172.21.15.57:0/1859688008 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f5c48054420 con 0x7f5c58157bb0 2023-10-19T14:18:38.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:38.832+0000 7f5c5f0b4640 1 -- 172.21.15.57:0/1859688008 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f5c580690e0 con 0x7f5c58157bb0 2023-10-19T14:18:39.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.466+0000 7f5c557fa640 1 -- 172.21.15.57:0/1859688008 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 25 and data pool 26 v81) v1 ==== 174+0+0 (secure 0 0 0) 0x7f5c48057a40 con 0x7f5c58157bb0 2023-10-19T14:18:39.470 INFO:teuthology.orchestra.run.smithi057.stderr:new fs with metadata pool 25 and data pool 26 2023-10-19T14:18:39.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.470+0000 7f5c5f0b4640 1 -- 172.21.15.57:0/1859688008 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f5c3806af60 msgr2=0x7f5c3806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:39.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.470+0000 7f5c5f0b4640 1 --2- 172.21.15.57:0/1859688008 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f5c3806af60 0x7f5c3806d420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5c580ad530 tx=0x7f5c500075b0 comp rx=0 tx=0).stop 2023-10-19T14:18:39.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.470+0000 7f5c5f0b4640 1 -- 172.21.15.57:0/1859688008 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5c58157bb0 msgr2=0x7f5c58148030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:39.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.470+0000 7f5c5f0b4640 1 --2- 172.21.15.57:0/1859688008 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5c58157bb0 0x7f5c58148030 secure :-1 s=READY pgs=2190 cs=0 l=1 rev1=1 crypto rx=0x7f5c4800c4b0 tx=0x7f5c4800c970 comp rx=0 tx=0).stop 2023-10-19T14:18:39.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.470+0000 7f5c5f0b4640 1 -- 172.21.15.57:0/1859688008 shutdown_connections 2023-10-19T14:18:39.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.470+0000 7f5c5f0b4640 1 --2- 172.21.15.57:0/1859688008 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f5c3806af60 0x7f5c3806d420 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:39.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.470+0000 7f5c5f0b4640 1 --2- 172.21.15.57:0/1859688008 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5c5815beb0 0x7f5c58148570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:39.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.470+0000 7f5c5f0b4640 1 --2- 172.21.15.57:0/1859688008 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5c58157bb0 0x7f5c58148030 unknown :-1 s=CLOSED pgs=2190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:39.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.470+0000 7f5c5f0b4640 1 --2- 172.21.15.57:0/1859688008 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5c5806c960 0x7f5c581e3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:39.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.470+0000 7f5c5f0b4640 1 -- 172.21.15.57:0/1859688008 >> 172.21.15.57:0/1859688008 conn(0x7f5c580aec40 msgr2=0x7f5c5806f510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:39.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.470+0000 7f5c5f0b4640 1 -- 172.21.15.57:0/1859688008 shutdown_connections 2023-10-19T14:18:39.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.470+0000 7f5c5f0b4640 1 -- 172.21.15.57:0/1859688008 wait complete. 2023-10-19T14:18:39.491 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T14:18:39.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.603+0000 7f4fc6383640 1 -- 172.21.15.57:0/4199895335 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4fc00cbd20 msgr2=0x7f4fc0158300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:39.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.603+0000 7f4fc6383640 1 --2- 172.21.15.57:0/4199895335 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4fc00cbd20 0x7f4fc0158300 secure :-1 s=READY pgs=2192 cs=0 l=1 rev1=1 crypto rx=0x7f4fc00cc490 tx=0x7f4fb402d0a0 comp rx=0 tx=0).stop 2023-10-19T14:18:39.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.603+0000 7f4fc6383640 1 -- 172.21.15.57:0/4199895335 shutdown_connections 2023-10-19T14:18:39.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.603+0000 7f4fc6383640 1 --2- 172.21.15.57:0/4199895335 >> v2:172.21.15.179:3300/0 conn(0x7f4fc00afa20 0x7f4fc00705a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:39.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.603+0000 7f4fc6383640 1 --2- 172.21.15.57:0/4199895335 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4fc00cbd20 0x7f4fc0158300 unknown :-1 s=CLOSED pgs=2192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:39.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.603+0000 7f4fc6383640 1 -- 172.21.15.57:0/4199895335 >> 172.21.15.57:0/4199895335 conn(0x7f4fc0069930 msgr2=0x7f4fc0069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:39.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.603+0000 7f4fc6383640 1 -- 172.21.15.57:0/4199895335 shutdown_connections 2023-10-19T14:18:39.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.603+0000 7f4fc6383640 1 -- 172.21.15.57:0/4199895335 wait complete. 2023-10-19T14:18:39.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.604+0000 7f4fc6383640 1 Processor -- start 2023-10-19T14:18:39.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.604+0000 7f4fc6383640 1 -- start start 2023-10-19T14:18:39.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.605+0000 7f4fc6383640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4fc00afa20 0x7f4fc0149bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:39.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.605+0000 7f4fc6383640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4fc00b0530 0x7f4fc01ec410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:39.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.605+0000 7f4fc6383640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4fc00cbd20 0x7f4fc013c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:39.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.605+0000 7f4fc6383640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fc014b010 con 0x7f4fc00b0530 2023-10-19T14:18:39.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.605+0000 7f4fc6383640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fc014a110 con 0x7f4fc00cbd20 2023-10-19T14:18:39.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.605+0000 7f4fc6383640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fc014a390 con 0x7f4fc00afa20 2023-10-19T14:18:39.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.605+0000 7f4fbf7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4fc00b0530 0x7f4fc01ec410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:39.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.605+0000 7f4fbf7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4fc00b0530 0x7f4fc01ec410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55464/0 (socket says 172.21.15.57:55464) 2023-10-19T14:18:39.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.605+0000 7f4fbf7fe640 1 -- 172.21.15.57:0/1547966668 learned_addr learned my addr 172.21.15.57:0/1547966668 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:18:39.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.605+0000 7f4fbf7fe640 1 -- 172.21.15.57:0/1547966668 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4fc00afa20 msgr2=0x7f4fc0149bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:39.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.605+0000 7f4fbf7fe640 1 --2- 172.21.15.57:0/1547966668 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4fc00afa20 0x7f4fc0149bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:39.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.605+0000 7f4fbf7fe640 1 -- 172.21.15.57:0/1547966668 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4fc00cbd20 msgr2=0x7f4fc013c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:39.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.605+0000 7f4fbf7fe640 1 --2- 172.21.15.57:0/1547966668 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4fc00cbd20 0x7f4fc013c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:39.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.605+0000 7f4fbf7fe640 1 -- 172.21.15.57:0/1547966668 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fb400a5a0 con 0x7f4fc00b0530 2023-10-19T14:18:39.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.605+0000 7f4fbf7fe640 1 --2- 172.21.15.57:0/1547966668 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4fc00b0530 0x7f4fc01ec410 secure :-1 s=READY pgs=2193 cs=0 l=1 rev1=1 crypto rx=0x7f4fb000bea0 tx=0x7f4fb000bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:39.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.606+0000 7f4fbcff9640 1 -- 172.21.15.57:0/1547966668 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fb0009c30 con 0x7f4fc00b0530 2023-10-19T14:18:39.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.606+0000 7f4fbcff9640 1 -- 172.21.15.57:0/1547966668 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f4fb0014070 con 0x7f4fc00b0530 2023-10-19T14:18:39.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.606+0000 7f4fc6383640 1 -- 172.21.15.57:0/1547966668 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fc014a670 con 0x7f4fc00b0530 2023-10-19T14:18:39.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.606+0000 7f4fbcff9640 1 -- 172.21.15.57:0/1547966668 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fb0010560 con 0x7f4fc00b0530 2023-10-19T14:18:39.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.606+0000 7f4fc6383640 1 -- 172.21.15.57:0/1547966668 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fc014ac30 con 0x7f4fc00b0530 2023-10-19T14:18:39.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.607+0000 7f4fc6383640 1 -- 172.21.15.57:0/1547966668 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fc00690e0 con 0x7f4fc00b0530 2023-10-19T14:18:39.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.607+0000 7f4fbcff9640 1 -- 172.21.15.57:0/1547966668 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 117) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4fb00106c0 con 0x7f4fc00b0530 2023-10-19T14:18:39.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.608+0000 7f4fbcff9640 1 --2- 172.21.15.57:0/1547966668 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f4fac06af60 0x7f4fac06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:39.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.608+0000 7f4fbcff9640 1 -- 172.21.15.57:0/1547966668 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(204..204 src has 1..204) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f4fb008bb50 con 0x7f4fc00b0530 2023-10-19T14:18:39.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.608+0000 7f4fbffff640 1 --2- 172.21.15.57:0/1547966668 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f4fac06af60 0x7f4fac06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:39.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.609+0000 7f4fbffff640 1 --2- 172.21.15.57:0/1547966668 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f4fac06af60 0x7f4fac06d420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4fb4037000 tx=0x7f4fb40046c0 comp rx=0 tx=0).ready entity=mgr.8744 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:39.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.612+0000 7f4fbcff9640 1 -- 172.21.15.57:0/1547966668 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4fb0056a20 con 0x7f4fc00b0530 2023-10-19T14:18:39.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.791+0000 7f4fc6383640 1 -- 172.21.15.57:0/1547966668 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4fc006f700 con 0x7f4fc00b0530 2023-10-19T14:18:39.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.792+0000 7f4fbcff9640 1 -- 172.21.15.57:0/1547966668 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v204) v1 ==== 74+0+16788 (secure 0 0 0) 0x7f4fb005a040 con 0x7f4fc00b0530 2023-10-19T14:18:39.796 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:18:39.796 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":204,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T14:18:38.834159+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2023-10-19T14:18:37.065110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"204","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_data","create_time":"2023-10-19T14:18:37.809854+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"204","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T14:18:39.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.795+0000 7f4fc6383640 1 -- 172.21.15.57:0/1547966668 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f4fac06af60 msgr2=0x7f4fac06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:39.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.795+0000 7f4fc6383640 1 --2- 172.21.15.57:0/1547966668 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f4fac06af60 0x7f4fac06d420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4fb4037000 tx=0x7f4fb40046c0 comp rx=0 tx=0).stop 2023-10-19T14:18:39.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.795+0000 7f4fc6383640 1 -- 172.21.15.57:0/1547966668 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4fc00b0530 msgr2=0x7f4fc01ec410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:39.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.795+0000 7f4fc6383640 1 --2- 172.21.15.57:0/1547966668 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4fc00b0530 0x7f4fc01ec410 secure :-1 s=READY pgs=2193 cs=0 l=1 rev1=1 crypto rx=0x7f4fb000bea0 tx=0x7f4fb000bf90 comp rx=0 tx=0).stop 2023-10-19T14:18:39.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.795+0000 7f4fc6383640 1 -- 172.21.15.57:0/1547966668 shutdown_connections 2023-10-19T14:18:39.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.795+0000 7f4fc6383640 1 --2- 172.21.15.57:0/1547966668 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f4fac06af60 0x7f4fac06d420 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:39.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.795+0000 7f4fc6383640 1 --2- 172.21.15.57:0/1547966668 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4fc00cbd20 0x7f4fc013c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:39.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.795+0000 7f4fc6383640 1 --2- 172.21.15.57:0/1547966668 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4fc00b0530 0x7f4fc01ec410 unknown :-1 s=CLOSED pgs=2193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:39.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.795+0000 7f4fc6383640 1 --2- 172.21.15.57:0/1547966668 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4fc00afa20 0x7f4fc0149bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:39.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.795+0000 7f4fc6383640 1 -- 172.21.15.57:0/1547966668 >> 172.21.15.57:0/1547966668 conn(0x7f4fc0069930 msgr2=0x7f4fc00c9730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:39.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.795+0000 7f4fc6383640 1 -- 172.21.15.57:0/1547966668 shutdown_connections 2023-10-19T14:18:39.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.795+0000 7f4fc6383640 1 -- 172.21.15.57:0/1547966668 wait complete. 2023-10-19T14:18:39.809 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T14:18:39.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.915+0000 7f6924b68640 1 -- 172.21.15.57:0/3179240361 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f69200cf340 msgr2=0x7f69200d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:39.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.915+0000 7f6924b68640 1 --2- 172.21.15.57:0/3179240361 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f69200cf340 0x7f69200d1910 secure :-1 s=READY pgs=2194 cs=0 l=1 rev1=1 crypto rx=0x7f692015b680 tx=0x7f691002bdb0 comp rx=0 tx=0).stop 2023-10-19T14:18:39.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.915+0000 7f6924b68640 1 -- 172.21.15.57:0/3179240361 shutdown_connections 2023-10-19T14:18:39.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.915+0000 7f6924b68640 1 --2- 172.21.15.57:0/3179240361 >> v2:172.21.15.179:3300/0 conn(0x7f69200ceb40 0x7f69200b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:39.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.915+0000 7f6924b68640 1 --2- 172.21.15.57:0/3179240361 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f69200cf340 0x7f69200d1910 unknown :-1 s=CLOSED pgs=2194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:39.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.915+0000 7f6924b68640 1 -- 172.21.15.57:0/3179240361 >> 172.21.15.57:0/3179240361 conn(0x7f69200d0d60 msgr2=0x7f69200d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:39.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.915+0000 7f6924b68640 1 -- 172.21.15.57:0/3179240361 shutdown_connections 2023-10-19T14:18:39.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.915+0000 7f6924b68640 1 -- 172.21.15.57:0/3179240361 wait complete. 2023-10-19T14:18:39.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.917+0000 7f6924b68640 1 Processor -- start 2023-10-19T14:18:39.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.917+0000 7f6924b68640 1 -- start start 2023-10-19T14:18:39.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.917+0000 7f6924b68640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f69200ceb40 0x7f6920157e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:39.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.917+0000 7f6924b68640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f69200cf340 0x7f69200b4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:39.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.917+0000 7f6924b68640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f692015b2a0 0x7f69200cc790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:39.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.917+0000 7f6924b68640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f692015bd20 con 0x7f692015b2a0 2023-10-19T14:18:39.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.917+0000 7f6924b68640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6920158360 con 0x7f69200ceb40 2023-10-19T14:18:39.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.917+0000 7f6924b68640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69201f1470 con 0x7f69200cf340 2023-10-19T14:18:39.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.917+0000 7f691ed9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f692015b2a0 0x7f69200cc790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:39.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.918+0000 7f691ed9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f692015b2a0 0x7f69200cc790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55482/0 (socket says 172.21.15.57:55482) 2023-10-19T14:18:39.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.918+0000 7f691ed9d640 1 -- 172.21.15.57:0/2093321793 learned_addr learned my addr 172.21.15.57:0/2093321793 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:18:39.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.918+0000 7f691e59c640 1 --2- 172.21.15.57:0/2093321793 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f69200ceb40 0x7f6920157e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:39.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.918+0000 7f691dd9b640 1 --2- 172.21.15.57:0/2093321793 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f69200cf340 0x7f69200b4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:39.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.918+0000 7f691ed9d640 1 -- 172.21.15.57:0/2093321793 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f69200cf340 msgr2=0x7f69200b4360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:39.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.918+0000 7f691ed9d640 1 --2- 172.21.15.57:0/2093321793 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f69200cf340 0x7f69200b4360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:39.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.918+0000 7f691ed9d640 1 -- 172.21.15.57:0/2093321793 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f69200ceb40 msgr2=0x7f6920157e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:39.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.918+0000 7f691ed9d640 1 --2- 172.21.15.57:0/2093321793 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f69200ceb40 0x7f6920157e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:39.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.918+0000 7f691ed9d640 1 -- 172.21.15.57:0/2093321793 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6910009560 con 0x7f692015b2a0 2023-10-19T14:18:39.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.918+0000 7f691ed9d640 1 --2- 172.21.15.57:0/2093321793 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f692015b2a0 0x7f69200cc790 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 crypto rx=0x7f691800eb10 tx=0x7f691800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:39.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.918+0000 7f690effd640 1 -- 172.21.15.57:0/2093321793 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6918004b20 con 0x7f692015b2a0 2023-10-19T14:18:39.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.918+0000 7f690effd640 1 -- 172.21.15.57:0/2093321793 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f6918004ca0 con 0x7f692015b2a0 2023-10-19T14:18:39.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.918+0000 7f6924b68640 1 -- 172.21.15.57:0/2093321793 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69201587e0 con 0x7f692015b2a0 2023-10-19T14:18:39.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.918+0000 7f690effd640 1 -- 172.21.15.57:0/2093321793 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6918005280 con 0x7f692015b2a0 2023-10-19T14:18:39.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.918+0000 7f6924b68640 1 -- 172.21.15.57:0/2093321793 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69200b5b60 con 0x7f692015b2a0 2023-10-19T14:18:39.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.919+0000 7f6924b68640 1 -- 172.21.15.57:0/2093321793 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f692015be90 con 0x7f692015b2a0 2023-10-19T14:18:39.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.920+0000 7f690effd640 1 -- 172.21.15.57:0/2093321793 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 117) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f69180053e0 con 0x7f692015b2a0 2023-10-19T14:18:39.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.920+0000 7f690effd640 1 --2- 172.21.15.57:0/2093321793 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f690006af60 0x7f690006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:39.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.920+0000 7f690effd640 1 -- 172.21.15.57:0/2093321793 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(204..204 src has 1..204) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f6918014070 con 0x7f692015b2a0 2023-10-19T14:18:39.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.920+0000 7f691e59c640 1 --2- 172.21.15.57:0/2093321793 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f690006af60 0x7f690006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:39.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.921+0000 7f691e59c640 1 --2- 172.21.15.57:0/2093321793 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f690006af60 0x7f690006d420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6910035000 tx=0x7f6910004730 comp rx=0 tx=0).ready entity=mgr.8744 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:39.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:39.924+0000 7f690effd640 1 -- 172.21.15.57:0/2093321793 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6918055420 con 0x7f692015b2a0 2023-10-19T14:18:40.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.100+0000 7f6924b68640 1 -- 172.21.15.57:0/2093321793 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f69200b0d00 con 0x7f692015b2a0 2023-10-19T14:18:40.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.101+0000 7f690effd640 1 -- 172.21.15.57:0/2093321793 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v204) v1 ==== 74+0+16788 (secure 0 0 0) 0x7f6918058a40 con 0x7f692015b2a0 2023-10-19T14:18:40.105 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:18:40.105 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":204,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T14:18:38.834159+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2023-10-19T14:18:37.065110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"204","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_data","create_time":"2023-10-19T14:18:37.809854+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"204","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T14:18:40.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.103+0000 7f6924b68640 1 -- 172.21.15.57:0/2093321793 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f690006af60 msgr2=0x7f690006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:40.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.103+0000 7f6924b68640 1 --2- 172.21.15.57:0/2093321793 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f690006af60 0x7f690006d420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6910035000 tx=0x7f6910004730 comp rx=0 tx=0).stop 2023-10-19T14:18:40.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.103+0000 7f6924b68640 1 -- 172.21.15.57:0/2093321793 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f692015b2a0 msgr2=0x7f69200cc790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:40.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.103+0000 7f6924b68640 1 --2- 172.21.15.57:0/2093321793 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f692015b2a0 0x7f69200cc790 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 crypto rx=0x7f691800eb10 tx=0x7f691800c2d0 comp rx=0 tx=0).stop 2023-10-19T14:18:40.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.103+0000 7f6924b68640 1 -- 172.21.15.57:0/2093321793 shutdown_connections 2023-10-19T14:18:40.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.103+0000 7f6924b68640 1 --2- 172.21.15.57:0/2093321793 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f690006af60 0x7f690006d420 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:40.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.103+0000 7f6924b68640 1 --2- 172.21.15.57:0/2093321793 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f692015b2a0 0x7f69200cc790 unknown :-1 s=CLOSED pgs=2195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:40.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.103+0000 7f6924b68640 1 --2- 172.21.15.57:0/2093321793 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f69200cf340 0x7f69200b4360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:40.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.103+0000 7f6924b68640 1 --2- 172.21.15.57:0/2093321793 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f69200ceb40 0x7f6920157e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:40.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.103+0000 7f6924b68640 1 -- 172.21.15.57:0/2093321793 >> 172.21.15.57:0/2093321793 conn(0x7f69200d0d60 msgr2=0x7f692015a210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:40.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.103+0000 7f6924b68640 1 -- 172.21.15.57:0/2093321793 shutdown_connections 2023-10-19T14:18:40.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.103+0000 7f6924b68640 1 -- 172.21.15.57:0/2093321793 wait complete. 2023-10-19T14:18:40.117 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-19T14:18:40.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.226+0000 7f69740f2640 1 -- 172.21.15.57:0/2680869686 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f696c15b2a0 msgr2=0x7f696c0cf720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:40.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.226+0000 7f69740f2640 1 --2- 172.21.15.57:0/2680869686 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f696c15b2a0 0x7f696c0cf720 secure :-1 s=READY pgs=2196 cs=0 l=1 rev1=1 crypto rx=0x7f695c005fd0 tx=0x7f695c02bcf0 comp rx=0 tx=0).stop 2023-10-19T14:18:40.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.226+0000 7f69740f2640 1 -- 172.21.15.57:0/2680869686 shutdown_connections 2023-10-19T14:18:40.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.226+0000 7f69740f2640 1 --2- 172.21.15.57:0/2680869686 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f696c15b2a0 0x7f696c0cf720 unknown :-1 s=CLOSED pgs=2196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:40.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.226+0000 7f69740f2640 1 --2- 172.21.15.57:0/2680869686 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f696c0cf340 0x7f696c0d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:40.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.226+0000 7f69740f2640 1 -- 172.21.15.57:0/2680869686 >> 172.21.15.57:0/2680869686 conn(0x7f696c0d0d60 msgr2=0x7f696c0d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:40.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.226+0000 7f69740f2640 1 -- 172.21.15.57:0/2680869686 shutdown_connections 2023-10-19T14:18:40.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.226+0000 7f69740f2640 1 -- 172.21.15.57:0/2680869686 wait complete. 2023-10-19T14:18:40.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.228+0000 7f69740f2640 1 Processor -- start 2023-10-19T14:18:40.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.228+0000 7f69740f2640 1 -- start start 2023-10-19T14:18:40.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.228+0000 7f69740f2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f696c0ceb40 0x7f696c1f0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:40.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.228+0000 7f69740f2640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f696c0cf340 0x7f696c0ca830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:40.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.228+0000 7f69740f2640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f696c15b2a0 0x7f696c0cad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:40.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.228+0000 7f69740f2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f696c0ca180 con 0x7f696c0ceb40 2023-10-19T14:18:40.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.228+0000 7f69740f2640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f696c0ca610 con 0x7f696c0cf340 2023-10-19T14:18:40.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.228+0000 7f69740f2640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f696c0c9570 con 0x7f696c15b2a0 2023-10-19T14:18:40.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.229+0000 7f6971e8e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f696c0ceb40 0x7f696c1f0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:40.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.229+0000 7f6971e8e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f696c0ceb40 0x7f696c1f0ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55504/0 (socket says 172.21.15.57:55504) 2023-10-19T14:18:40.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.229+0000 7f6971e8e640 1 -- 172.21.15.57:0/1258665073 learned_addr learned my addr 172.21.15.57:0/1258665073 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:18:40.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.229+0000 7f697268f640 1 --2- 172.21.15.57:0/1258665073 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f696c15b2a0 0x7f696c0cad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:40.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.229+0000 7f697168d640 1 --2- 172.21.15.57:0/1258665073 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f696c0cf340 0x7f696c0ca830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:40.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.229+0000 7f6971e8e640 1 -- 172.21.15.57:0/1258665073 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f696c15b2a0 msgr2=0x7f696c0cad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:40.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.229+0000 7f6971e8e640 1 --2- 172.21.15.57:0/1258665073 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f696c15b2a0 0x7f696c0cad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:40.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.229+0000 7f6971e8e640 1 -- 172.21.15.57:0/1258665073 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f696c0cf340 msgr2=0x7f696c0ca830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:40.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.229+0000 7f6971e8e640 1 --2- 172.21.15.57:0/1258665073 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f696c0cf340 0x7f696c0ca830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:40.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.229+0000 7f6971e8e640 1 -- 172.21.15.57:0/1258665073 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f695c009560 con 0x7f696c0ceb40 2023-10-19T14:18:40.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.229+0000 7f6971e8e640 1 --2- 172.21.15.57:0/1258665073 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f696c0ceb40 0x7f696c1f0ee0 secure :-1 s=READY pgs=2197 cs=0 l=1 rev1=1 crypto rx=0x7f696800d770 tx=0x7f696800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:40.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.229+0000 7f69627fc640 1 -- 172.21.15.57:0/1258665073 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6968009c30 con 0x7f696c0ceb40 2023-10-19T14:18:40.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.229+0000 7f69627fc640 1 -- 172.21.15.57:0/1258665073 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f696800de80 con 0x7f696c0ceb40 2023-10-19T14:18:40.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.229+0000 7f69740f2640 1 -- 172.21.15.57:0/1258665073 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f696c0c9850 con 0x7f696c0ceb40 2023-10-19T14:18:40.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.229+0000 7f69627fc640 1 -- 172.21.15.57:0/1258665073 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6968005280 con 0x7f696c0ceb40 2023-10-19T14:18:40.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.229+0000 7f69740f2640 1 -- 172.21.15.57:0/1258665073 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f696c0c9e10 con 0x7f696c0ceb40 2023-10-19T14:18:40.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.230+0000 7f69740f2640 1 -- 172.21.15.57:0/1258665073 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f696c0690e0 con 0x7f696c0ceb40 2023-10-19T14:18:40.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.231+0000 7f69627fc640 1 -- 172.21.15.57:0/1258665073 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 117) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f69680053e0 con 0x7f696c0ceb40 2023-10-19T14:18:40.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.231+0000 7f69627fc640 1 --2- 172.21.15.57:0/1258665073 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f695006af60 0x7f695006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:40.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.231+0000 7f69627fc640 1 -- 172.21.15.57:0/1258665073 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(204..204 src has 1..204) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f6968089bb0 con 0x7f696c0ceb40 2023-10-19T14:18:40.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.231+0000 7f697168d640 1 --2- 172.21.15.57:0/1258665073 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f695006af60 0x7f695006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:40.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.232+0000 7f697168d640 1 --2- 172.21.15.57:0/1258665073 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f695006af60 0x7f695006d420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f695c035000 tx=0x7f695c004730 comp rx=0 tx=0).ready entity=mgr.8744 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:40.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.235+0000 7f69627fc640 1 -- 172.21.15.57:0/1258665073 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6968054990 con 0x7f696c0ceb40 2023-10-19T14:18:40.452 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.449+0000 7f69740f2640 1 -- 172.21.15.57:0/1258665073 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f696c0d0b40 con 0x7f696c0ceb40 2023-10-19T14:18:40.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.485+0000 7f69627fc640 1 -- 172.21.15.57:0/1258665073 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v82) v1 ==== 118+0+0 (secure 0 0 0) 0x7f6968017020 con 0x7f696c0ceb40 2023-10-19T14:18:40.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.487+0000 7f69740f2640 1 -- 172.21.15.57:0/1258665073 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f695006af60 msgr2=0x7f695006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:40.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.487+0000 7f69740f2640 1 --2- 172.21.15.57:0/1258665073 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f695006af60 0x7f695006d420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f695c035000 tx=0x7f695c004730 comp rx=0 tx=0).stop 2023-10-19T14:18:40.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.487+0000 7f69740f2640 1 -- 172.21.15.57:0/1258665073 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f696c0ceb40 msgr2=0x7f696c1f0ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:40.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.487+0000 7f69740f2640 1 --2- 172.21.15.57:0/1258665073 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f696c0ceb40 0x7f696c1f0ee0 secure :-1 s=READY pgs=2197 cs=0 l=1 rev1=1 crypto rx=0x7f696800d770 tx=0x7f696800dc30 comp rx=0 tx=0).stop 2023-10-19T14:18:40.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.487+0000 7f69740f2640 1 -- 172.21.15.57:0/1258665073 shutdown_connections 2023-10-19T14:18:40.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.487+0000 7f69740f2640 1 --2- 172.21.15.57:0/1258665073 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f695006af60 0x7f695006d420 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:40.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.487+0000 7f69740f2640 1 --2- 172.21.15.57:0/1258665073 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f696c15b2a0 0x7f696c0cad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:40.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.487+0000 7f69740f2640 1 --2- 172.21.15.57:0/1258665073 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f696c0cf340 0x7f696c0ca830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:40.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.487+0000 7f69740f2640 1 --2- 172.21.15.57:0/1258665073 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f696c0ceb40 0x7f696c1f0ee0 unknown :-1 s=CLOSED pgs=2197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:40.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.487+0000 7f69740f2640 1 -- 172.21.15.57:0/1258665073 >> 172.21.15.57:0/1258665073 conn(0x7f696c0d0d60 msgr2=0x7f696c06b560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:40.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.487+0000 7f69740f2640 1 -- 172.21.15.57:0/1258665073 shutdown_connections 2023-10-19T14:18:40.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.487+0000 7f69740f2640 1 -- 172.21.15.57:0/1258665073 wait complete. 2023-10-19T14:18:40.501 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-19T14:18:40.501 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-19T14:18:40.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.610+0000 7fb59559e640 1 -- 172.21.15.57:0/1978354255 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb5900adc10 msgr2=0x7fb5900aeb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:40.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.610+0000 7fb59559e640 1 --2- 172.21.15.57:0/1978354255 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb5900adc10 0x7fb5900aeb10 secure :-1 s=READY pgs=2199 cs=0 l=1 rev1=1 crypto rx=0x7fb588005fd0 tx=0x7fb58802c2e0 comp rx=0 tx=0).stop 2023-10-19T14:18:40.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.610+0000 7fb59559e640 1 -- 172.21.15.57:0/1978354255 shutdown_connections 2023-10-19T14:18:40.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.610+0000 7fb59559e640 1 --2- 172.21.15.57:0/1978354255 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb5900adc10 0x7fb5900aeb10 unknown :-1 s=CLOSED pgs=2199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:40.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.610+0000 7fb59559e640 1 -- 172.21.15.57:0/1978354255 >> 172.21.15.57:0/1978354255 conn(0x7fb5900b3db0 msgr2=0x7fb5900b41c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:40.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.610+0000 7fb59559e640 1 -- 172.21.15.57:0/1978354255 shutdown_connections 2023-10-19T14:18:40.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.610+0000 7fb59559e640 1 -- 172.21.15.57:0/1978354255 wait complete. 2023-10-19T14:18:40.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.611+0000 7fb59559e640 1 Processor -- start 2023-10-19T14:18:40.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.612+0000 7fb59559e640 1 -- start start 2023-10-19T14:18:40.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.612+0000 7fb59559e640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb5900ad7b0 0x7fb59014a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:40.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.612+0000 7fb59559e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb5900adc10 0x7fb5901504e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:40.618 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.612+0000 7fb59559e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb590158f00 0x7fb5901e38d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:40.618 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.612+0000 7fb59559e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5901f1990 con 0x7fb590158f00 2023-10-19T14:18:40.618 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.612+0000 7fb59559e640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5901f27b0 con 0x7fb5900adc10 2023-10-19T14:18:40.618 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.612+0000 7fb59559e640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5901f1ba0 con 0x7fb5900ad7b0 2023-10-19T14:18:40.618 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.612+0000 7fb594d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb590158f00 0x7fb5901e38d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:40.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.612+0000 7fb594d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb590158f00 0x7fb5901e38d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55520/0 (socket says 172.21.15.57:55520) 2023-10-19T14:18:40.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.612+0000 7fb594d9d640 1 -- 172.21.15.57:0/1474710131 learned_addr learned my addr 172.21.15.57:0/1474710131 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:18:40.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.613+0000 7fb58f7fe640 1 --2- 172.21.15.57:0/1474710131 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb5900adc10 0x7fb5901504e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:40.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.613+0000 7fb58ffff640 1 --2- 172.21.15.57:0/1474710131 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb5900ad7b0 0x7fb59014a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:40.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.613+0000 7fb594d9d640 1 -- 172.21.15.57:0/1474710131 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb5900ad7b0 msgr2=0x7fb59014a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:40.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.613+0000 7fb594d9d640 1 --2- 172.21.15.57:0/1474710131 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb5900ad7b0 0x7fb59014a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:40.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.613+0000 7fb594d9d640 1 -- 172.21.15.57:0/1474710131 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb5900adc10 msgr2=0x7fb5901504e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:40.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.613+0000 7fb594d9d640 1 --2- 172.21.15.57:0/1474710131 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb5900adc10 0x7fb5901504e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:40.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.613+0000 7fb594d9d640 1 -- 172.21.15.57:0/1474710131 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb588009900 con 0x7fb590158f00 2023-10-19T14:18:40.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.613+0000 7fb594d9d640 1 --2- 172.21.15.57:0/1474710131 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb590158f00 0x7fb5901e38d0 secure :-1 s=READY pgs=2200 cs=0 l=1 rev1=1 crypto rx=0x7fb588009bf0 tx=0x7fb588035b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:40.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.613+0000 7fb58cff9640 1 -- 172.21.15.57:0/1474710131 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb588004240 con 0x7fb590158f00 2023-10-19T14:18:40.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.613+0000 7fb58cff9640 1 -- 172.21.15.57:0/1474710131 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fb588004490 con 0x7fb590158f00 2023-10-19T14:18:40.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.613+0000 7fb59559e640 1 -- 172.21.15.57:0/1474710131 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5901f1e80 con 0x7fb590158f00 2023-10-19T14:18:40.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.613+0000 7fb58cff9640 1 -- 172.21.15.57:0/1474710131 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb588004860 con 0x7fb590158f00 2023-10-19T14:18:40.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.613+0000 7fb59559e640 1 -- 172.21.15.57:0/1474710131 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5901f2440 con 0x7fb590158f00 2023-10-19T14:18:40.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.614+0000 7fb59559e640 1 -- 172.21.15.57:0/1474710131 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb59004da70 con 0x7fb590158f00 2023-10-19T14:18:40.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.615+0000 7fb58cff9640 1 -- 172.21.15.57:0/1474710131 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 117) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb58803b050 con 0x7fb590158f00 2023-10-19T14:18:40.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.615+0000 7fb58cff9640 1 --2- 172.21.15.57:0/1474710131 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fb57c06af60 0x7fb57c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:40.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.615+0000 7fb58cff9640 1 -- 172.21.15.57:0/1474710131 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fb5880ac7c0 con 0x7fb590158f00 2023-10-19T14:18:40.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.615+0000 7fb58ffff640 1 --2- 172.21.15.57:0/1474710131 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fb57c06af60 0x7fb57c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:40.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.616+0000 7fb58ffff640 1 --2- 172.21.15.57:0/1474710131 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fb57c06af60 0x7fb57c06d420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb584007fd0 tx=0x7fb584009040 comp rx=0 tx=0).ready entity=mgr.8744 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:40.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.619+0000 7fb58cff9640 1 -- 172.21.15.57:0/1474710131 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb588077600 con 0x7fb590158f00 2023-10-19T14:18:40.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:40.831+0000 7fb59559e640 1 -- 172.21.15.57:0/1474710131 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fb5900b1d20 con 0x7fb590158f00 2023-10-19T14:18:41.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.491+0000 7fb58cff9640 1 -- 172.21.15.57:0/1474710131 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v83) v1 ==== 122+0+0 (secure 0 0 0) 0x7fb58807ac20 con 0x7fb590158f00 2023-10-19T14:18:41.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.493+0000 7fb59559e640 1 -- 172.21.15.57:0/1474710131 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fb57c06af60 msgr2=0x7fb57c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:41.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.493+0000 7fb59559e640 1 --2- 172.21.15.57:0/1474710131 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fb57c06af60 0x7fb57c06d420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb584007fd0 tx=0x7fb584009040 comp rx=0 tx=0).stop 2023-10-19T14:18:41.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.493+0000 7fb59559e640 1 -- 172.21.15.57:0/1474710131 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb590158f00 msgr2=0x7fb5901e38d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:41.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.493+0000 7fb59559e640 1 --2- 172.21.15.57:0/1474710131 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb590158f00 0x7fb5901e38d0 secure :-1 s=READY pgs=2200 cs=0 l=1 rev1=1 crypto rx=0x7fb588009bf0 tx=0x7fb588035b50 comp rx=0 tx=0).stop 2023-10-19T14:18:41.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.493+0000 7fb59559e640 1 -- 172.21.15.57:0/1474710131 shutdown_connections 2023-10-19T14:18:41.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.493+0000 7fb59559e640 1 --2- 172.21.15.57:0/1474710131 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fb57c06af60 0x7fb57c06d420 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:41.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.493+0000 7fb59559e640 1 --2- 172.21.15.57:0/1474710131 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb590158f00 0x7fb5901e38d0 unknown :-1 s=CLOSED pgs=2200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:41.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.493+0000 7fb59559e640 1 --2- 172.21.15.57:0/1474710131 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb5900adc10 0x7fb5901504e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:41.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.493+0000 7fb59559e640 1 --2- 172.21.15.57:0/1474710131 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb5900ad7b0 0x7fb59014a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:41.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.493+0000 7fb59559e640 1 -- 172.21.15.57:0/1474710131 >> 172.21.15.57:0/1474710131 conn(0x7fb5900b3db0 msgr2=0x7fb59014c560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:41.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.494+0000 7fb59559e640 1 -- 172.21.15.57:0/1474710131 shutdown_connections 2023-10-19T14:18:41.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.494+0000 7fb59559e640 1 -- 172.21.15.57:0/1474710131 wait complete. 2023-10-19T14:18:41.510 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-19T14:18:41.510 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T14:18:41.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.623+0000 7f614fbb6640 1 -- 172.21.15.57:0/1424502936 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f614806b030 msgr2=0x7f614806b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:41.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.623+0000 7f614fbb6640 1 --2- 172.21.15.57:0/1424502936 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f614806b030 0x7f614806b410 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f6144005fd0 tx=0x7f614402bd90 comp rx=0 tx=0).stop 2023-10-19T14:18:41.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.623+0000 7f614fbb6640 1 -- 172.21.15.57:0/1424502936 shutdown_connections 2023-10-19T14:18:41.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.623+0000 7f614fbb6640 1 --2- 172.21.15.57:0/1424502936 >> v2:172.21.15.179:3300/0 conn(0x7f6148159bd0 0x7f614815c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:41.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.623+0000 7f614fbb6640 1 --2- 172.21.15.57:0/1424502936 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f614806b030 0x7f614806b410 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:41.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.623+0000 7f614fbb6640 1 -- 172.21.15.57:0/1424502936 >> 172.21.15.57:0/1424502936 conn(0x7f6148069930 msgr2=0x7f6148069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:41.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.623+0000 7f614fbb6640 1 -- 172.21.15.57:0/1424502936 shutdown_connections 2023-10-19T14:18:41.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.623+0000 7f614fbb6640 1 -- 172.21.15.57:0/1424502936 wait complete. 2023-10-19T14:18:41.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.624+0000 7f614fbb6640 1 Processor -- start 2023-10-19T14:18:41.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.624+0000 7f614fbb6640 1 -- start start 2023-10-19T14:18:41.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.625+0000 7f614fbb6640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f614806b030 0x7f61480c8390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:41.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.625+0000 7f614fbb6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6148159bd0 0x7f6148138260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:41.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.625+0000 7f614fbb6640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f614815ccb0 0x7f614812d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:41.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.625+0000 7f614fbb6640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f614812be70 con 0x7f6148159bd0 2023-10-19T14:18:41.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.625+0000 7f614fbb6640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f614812b540 con 0x7f614815ccb0 2023-10-19T14:18:41.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.625+0000 7f614fbb6640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f614812b820 con 0x7f614806b030 2023-10-19T14:18:41.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.625+0000 7f614d151640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6148159bd0 0x7f6148138260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:41.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.625+0000 7f614d151640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6148159bd0 0x7f6148138260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55534/0 (socket says 172.21.15.57:55534) 2023-10-19T14:18:41.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.625+0000 7f614d151640 1 -- 172.21.15.57:0/886567490 learned_addr learned my addr 172.21.15.57:0/886567490 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:18:41.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.625+0000 7f614d952640 1 --2- 172.21.15.57:0/886567490 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f614806b030 0x7f61480c8390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:41.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.625+0000 7f614e153640 1 --2- 172.21.15.57:0/886567490 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f614815ccb0 0x7f614812d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:41.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.626+0000 7f614d151640 1 -- 172.21.15.57:0/886567490 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f614806b030 msgr2=0x7f61480c8390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:41.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.626+0000 7f614d151640 1 --2- 172.21.15.57:0/886567490 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f614806b030 0x7f61480c8390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:41.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.626+0000 7f614d151640 1 -- 172.21.15.57:0/886567490 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f614815ccb0 msgr2=0x7f614812d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:41.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.626+0000 7f614d151640 1 --2- 172.21.15.57:0/886567490 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f614815ccb0 0x7f614812d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:41.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.626+0000 7f614d151640 1 -- 172.21.15.57:0/886567490 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6138009490 con 0x7f6148159bd0 2023-10-19T14:18:41.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.626+0000 7f614d151640 1 --2- 172.21.15.57:0/886567490 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6148159bd0 0x7f6148138260 secure :-1 s=READY pgs=2201 cs=0 l=1 rev1=1 crypto rx=0x7f613800ea60 tx=0x7f613800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:41.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.626+0000 7f613e7fc640 1 -- 172.21.15.57:0/886567490 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f613800cde0 con 0x7f6148159bd0 2023-10-19T14:18:41.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.626+0000 7f613e7fc640 1 -- 172.21.15.57:0/886567490 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f6138004020 con 0x7f6148159bd0 2023-10-19T14:18:41.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.626+0000 7f614fbb6640 1 -- 172.21.15.57:0/886567490 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6144009560 con 0x7f6148159bd0 2023-10-19T14:18:41.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.626+0000 7f613e7fc640 1 -- 172.21.15.57:0/886567490 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6138004410 con 0x7f6148159bd0 2023-10-19T14:18:41.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.626+0000 7f614fbb6640 1 -- 172.21.15.57:0/886567490 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f614812e290 con 0x7f6148159bd0 2023-10-19T14:18:41.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.627+0000 7f614fbb6640 1 -- 172.21.15.57:0/886567490 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f614806c290 con 0x7f6148159bd0 2023-10-19T14:18:41.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.628+0000 7f613e7fc640 1 -- 172.21.15.57:0/886567490 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 117) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f61380035f0 con 0x7f6148159bd0 2023-10-19T14:18:41.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.628+0000 7f613e7fc640 1 --2- 172.21.15.57:0/886567490 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f612c06af90 0x7f612c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:41.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.628+0000 7f613e7fc640 1 -- 172.21.15.57:0/886567490 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f6138014070 con 0x7f6148159bd0 2023-10-19T14:18:41.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.628+0000 7f614d952640 1 --2- 172.21.15.57:0/886567490 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f612c06af90 0x7f612c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:41.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.629+0000 7f614d952640 1 --2- 172.21.15.57:0/886567490 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f612c06af90 0x7f612c06d450 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f6144035000 tx=0x7f6144004730 comp rx=0 tx=0).ready entity=mgr.8744 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:41.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.632+0000 7f613e7fc640 1 -- 172.21.15.57:0/886567490 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f61380552a0 con 0x7f6148159bd0 2023-10-19T14:18:41.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.845+0000 7f614fbb6640 1 -- 172.21.15.57:0/886567490 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f61480ab4d0 con 0x7f6148159bd0 2023-10-19T14:18:41.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.845+0000 7f613e7fc640 1 -- 172.21.15.57:0/886567490 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+4926 (secure 0 0 0) 0x7f61380588c0 con 0x7f6148159bd0 2023-10-19T14:18:41.848 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:18:41.849 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":83,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/2313260224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2313260224},{"type":"v1","addr":"172.21.15.179:6836","nonce":2313260224}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":10081,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/494432208","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":494432208},{"type":"v1","addr":"172.21.15.179:6837","nonce":494432208}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":11214,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/3080658822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":3080658822},{"type":"v1","addr":"172.21.15.57:6834","nonce":3080658822}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T14:18:38.834485+0000","modified":"2023-10-19T14:18:40.833363+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11217":{"gid":11217,"name":"a","rank":0,"incarnation":81,"state":"up:active","state_seq":3,"addr":"172.21.15.57:6835/1674945162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1674945162},{"type":"v1","addr":"172.21.15.57:6835","nonce":1674945162}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2023-10-19T14:18:41.849 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 83 2023-10-19T14:18:41.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.847+0000 7f614fbb6640 1 -- 172.21.15.57:0/886567490 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f612c06af90 msgr2=0x7f612c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:41.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.847+0000 7f614fbb6640 1 --2- 172.21.15.57:0/886567490 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f612c06af90 0x7f612c06d450 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f6144035000 tx=0x7f6144004730 comp rx=0 tx=0).stop 2023-10-19T14:18:41.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.847+0000 7f614fbb6640 1 -- 172.21.15.57:0/886567490 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6148159bd0 msgr2=0x7f6148138260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:41.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.847+0000 7f614fbb6640 1 --2- 172.21.15.57:0/886567490 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6148159bd0 0x7f6148138260 secure :-1 s=READY pgs=2201 cs=0 l=1 rev1=1 crypto rx=0x7f613800ea60 tx=0x7f613800ef20 comp rx=0 tx=0).stop 2023-10-19T14:18:41.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.847+0000 7f614fbb6640 1 -- 172.21.15.57:0/886567490 shutdown_connections 2023-10-19T14:18:41.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.847+0000 7f614fbb6640 1 --2- 172.21.15.57:0/886567490 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f612c06af90 0x7f612c06d450 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:41.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.847+0000 7f614fbb6640 1 --2- 172.21.15.57:0/886567490 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f614815ccb0 0x7f614812d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:41.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.847+0000 7f614fbb6640 1 --2- 172.21.15.57:0/886567490 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6148159bd0 0x7f6148138260 unknown :-1 s=CLOSED pgs=2201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:41.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.847+0000 7f614fbb6640 1 --2- 172.21.15.57:0/886567490 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f614806b030 0x7f61480c8390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:41.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.847+0000 7f614fbb6640 1 -- 172.21.15.57:0/886567490 >> 172.21.15.57:0/886567490 conn(0x7f6148069930 msgr2=0x7f614815c590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:41.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.848+0000 7f614fbb6640 1 -- 172.21.15.57:0/886567490 shutdown_connections 2023-10-19T14:18:41.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.848+0000 7f614fbb6640 1 -- 172.21.15.57:0/886567490 wait complete. 2023-10-19T14:18:41.861 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T14:18:41.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.968+0000 7fbd36f1d640 1 -- 172.21.15.57:0/2489383722 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd3006ae40 msgr2=0x7fbd300b0e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:41.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.968+0000 7fbd36f1d640 1 --2- 172.21.15.57:0/2489383722 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd3006ae40 0x7fbd300b0e90 secure :-1 s=READY pgs=2202 cs=0 l=1 rev1=1 crypto rx=0x7fbd1c005fd0 tx=0x7fbd1c02be10 comp rx=0 tx=0).stop 2023-10-19T14:18:41.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.968+0000 7fbd36f1d640 1 -- 172.21.15.57:0/2489383722 shutdown_connections 2023-10-19T14:18:41.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.968+0000 7fbd36f1d640 1 --2- 172.21.15.57:0/2489383722 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbd3015c1f0 0x7fbd30157dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:41.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.968+0000 7fbd36f1d640 1 --2- 172.21.15.57:0/2489383722 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd3006ae40 0x7fbd300b0e90 unknown :-1 s=CLOSED pgs=2202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:41.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.968+0000 7fbd36f1d640 1 -- 172.21.15.57:0/2489383722 >> 172.21.15.57:0/2489383722 conn(0x7fbd3006fae0 msgr2=0x7fbd3006fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:41.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.968+0000 7fbd36f1d640 1 -- 172.21.15.57:0/2489383722 shutdown_connections 2023-10-19T14:18:41.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.968+0000 7fbd36f1d640 1 -- 172.21.15.57:0/2489383722 wait complete. 2023-10-19T14:18:41.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.969+0000 7fbd36f1d640 1 Processor -- start 2023-10-19T14:18:41.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.969+0000 7fbd36f1d640 1 -- start start 2023-10-19T14:18:41.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.970+0000 7fbd36f1d640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbd3006ae40 0x7fbd300ae840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:41.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.970+0000 7fbd36f1d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd301579f0 0x7fbd30146cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:41.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.970+0000 7fbd36f1d640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbd3015c1f0 0x7fbd30147440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:41.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.970+0000 7fbd36f1d640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd30147980 con 0x7fbd301579f0 2023-10-19T14:18:41.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.970+0000 7fbd36f1d640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd300aeed0 con 0x7fbd3015c1f0 2023-10-19T14:18:41.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.970+0000 7fbd36f1d640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd300aceb0 con 0x7fbd3006ae40 2023-10-19T14:18:41.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.970+0000 7fbd2ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd301579f0 0x7fbd30146cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:41.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.970+0000 7fbd2ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd301579f0 0x7fbd30146cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55554/0 (socket says 172.21.15.57:55554) 2023-10-19T14:18:41.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.970+0000 7fbd2ffff640 1 -- 172.21.15.57:0/1755933786 learned_addr learned my addr 172.21.15.57:0/1755933786 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:18:41.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.970+0000 7fbd2ffff640 1 -- 172.21.15.57:0/1755933786 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbd3006ae40 msgr2=0x7fbd300ae840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:41.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.970+0000 7fbd34cb9640 1 --2- 172.21.15.57:0/1755933786 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbd3006ae40 0x7fbd300ae840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:41.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.970+0000 7fbd354ba640 1 --2- 172.21.15.57:0/1755933786 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbd3015c1f0 0x7fbd30147440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:41.979 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.970+0000 7fbd2ffff640 1 --2- 172.21.15.57:0/1755933786 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbd3006ae40 0x7fbd300ae840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:41.979 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.970+0000 7fbd2ffff640 1 -- 172.21.15.57:0/1755933786 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbd3015c1f0 msgr2=0x7fbd30147440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:41.979 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.970+0000 7fbd2ffff640 1 --2- 172.21.15.57:0/1755933786 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbd3015c1f0 0x7fbd30147440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:41.979 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.970+0000 7fbd2ffff640 1 -- 172.21.15.57:0/1755933786 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd1c009560 con 0x7fbd301579f0 2023-10-19T14:18:41.979 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.970+0000 7fbd354ba640 1 --2- 172.21.15.57:0/1755933786 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbd3015c1f0 0x7fbd30147440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:18:41.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.970+0000 7fbd34cb9640 1 --2- 172.21.15.57:0/1755933786 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbd3006ae40 0x7fbd300ae840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:18:41.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.970+0000 7fbd2ffff640 1 --2- 172.21.15.57:0/1755933786 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd301579f0 0x7fbd30146cc0 secure :-1 s=READY pgs=2203 cs=0 l=1 rev1=1 crypto rx=0x7fbd1c006010 tx=0x7fbd1c02dbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:41.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.970+0000 7fbd2d7fa640 1 -- 172.21.15.57:0/1755933786 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd1c0035f0 con 0x7fbd301579f0 2023-10-19T14:18:41.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.971+0000 7fbd2d7fa640 1 -- 172.21.15.57:0/1755933786 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fbd1c003840 con 0x7fbd301579f0 2023-10-19T14:18:41.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.971+0000 7fbd36f1d640 1 -- 172.21.15.57:0/1755933786 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd300ad190 con 0x7fbd301579f0 2023-10-19T14:18:41.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.971+0000 7fbd2d7fa640 1 -- 172.21.15.57:0/1755933786 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd1c003c50 con 0x7fbd301579f0 2023-10-19T14:18:41.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.971+0000 7fbd36f1d640 1 -- 172.21.15.57:0/1755933786 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd300ad750 con 0x7fbd301579f0 2023-10-19T14:18:41.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.972+0000 7fbd36f1d640 1 -- 172.21.15.57:0/1755933786 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd300690e0 con 0x7fbd301579f0 2023-10-19T14:18:41.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.972+0000 7fbd2d7fa640 1 -- 172.21.15.57:0/1755933786 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 117) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fbd1c03e450 con 0x7fbd301579f0 2023-10-19T14:18:41.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.972+0000 7fbd2d7fa640 1 --2- 172.21.15.57:0/1755933786 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fbd1406af60 0x7fbd1406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:41.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.973+0000 7fbd2d7fa640 1 -- 172.21.15.57:0/1755933786 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fbd1c0ac590 con 0x7fbd301579f0 2023-10-19T14:18:41.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.973+0000 7fbd34cb9640 1 --2- 172.21.15.57:0/1755933786 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fbd1406af60 0x7fbd1406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:41.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.973+0000 7fbd34cb9640 1 --2- 172.21.15.57:0/1755933786 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fbd1406af60 0x7fbd1406d420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fbd301f4410 tx=0x7fbd24009040 comp rx=0 tx=0).ready entity=mgr.8744 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:41.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:41.977+0000 7fbd2d7fa640 1 -- 172.21.15.57:0/1755933786 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fbd1c077590 con 0x7fbd301579f0 2023-10-19T14:18:42.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.153+0000 7fbd36f1d640 1 -- 172.21.15.57:0/1755933786 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbd30153790 con 0x7fbd301579f0 2023-10-19T14:18:42.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.154+0000 7fbd2d7fa640 1 -- 172.21.15.57:0/1755933786 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v205) v1 ==== 74+0+16775 (secure 0 0 0) 0x7fbd1c003db0 con 0x7fbd301579f0 2023-10-19T14:18:42.157 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:18:42.158 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":205,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T14:18:40.300486+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2023-10-19T14:18:37.065110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"204","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_data","create_time":"2023-10-19T14:18:37.809854+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"205","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T14:18:42.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.156+0000 7fbd36f1d640 1 -- 172.21.15.57:0/1755933786 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fbd1406af60 msgr2=0x7fbd1406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:42.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.156+0000 7fbd36f1d640 1 --2- 172.21.15.57:0/1755933786 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fbd1406af60 0x7fbd1406d420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fbd301f4410 tx=0x7fbd24009040 comp rx=0 tx=0).stop 2023-10-19T14:18:42.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.156+0000 7fbd36f1d640 1 -- 172.21.15.57:0/1755933786 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd301579f0 msgr2=0x7fbd30146cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:42.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.156+0000 7fbd36f1d640 1 --2- 172.21.15.57:0/1755933786 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd301579f0 0x7fbd30146cc0 secure :-1 s=READY pgs=2203 cs=0 l=1 rev1=1 crypto rx=0x7fbd1c006010 tx=0x7fbd1c02dbb0 comp rx=0 tx=0).stop 2023-10-19T14:18:42.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.157+0000 7fbd36f1d640 1 -- 172.21.15.57:0/1755933786 shutdown_connections 2023-10-19T14:18:42.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.157+0000 7fbd36f1d640 1 --2- 172.21.15.57:0/1755933786 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fbd1406af60 0x7fbd1406d420 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:42.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.157+0000 7fbd36f1d640 1 --2- 172.21.15.57:0/1755933786 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbd3015c1f0 0x7fbd30147440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:42.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.157+0000 7fbd36f1d640 1 --2- 172.21.15.57:0/1755933786 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd301579f0 0x7fbd30146cc0 unknown :-1 s=CLOSED pgs=2203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:42.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.157+0000 7fbd36f1d640 1 --2- 172.21.15.57:0/1755933786 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbd3006ae40 0x7fbd300ae840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:42.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.157+0000 7fbd36f1d640 1 -- 172.21.15.57:0/1755933786 >> 172.21.15.57:0/1755933786 conn(0x7fbd3006fae0 msgr2=0x7fbd30149200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:42.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.157+0000 7fbd36f1d640 1 -- 172.21.15.57:0/1755933786 shutdown_connections 2023-10-19T14:18:42.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.157+0000 7fbd36f1d640 1 -- 172.21.15.57:0/1755933786 wait complete. 2023-10-19T14:18:42.170 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-19T14:18:42.171 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-19T14:18:42.279 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.276+0000 7f350d8c2640 1 -- 172.21.15.57:0/3272915741 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f350815b2a0 msgr2=0x7f35080d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:42.280 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.276+0000 7f350d8c2640 1 --2- 172.21.15.57:0/3272915741 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f350815b2a0 0x7f35080d1910 secure :-1 s=READY pgs=2204 cs=0 l=1 rev1=1 crypto rx=0x7f350815b680 tx=0x7f34f002bda0 comp rx=0 tx=0).stop 2023-10-19T14:18:42.280 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.276+0000 7f350d8c2640 1 -- 172.21.15.57:0/3272915741 shutdown_connections 2023-10-19T14:18:42.280 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.276+0000 7f350d8c2640 1 --2- 172.21.15.57:0/3272915741 >> v2:172.21.15.179:3300/0 conn(0x7f35080ceb40 0x7f35080b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:42.280 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.276+0000 7f350d8c2640 1 --2- 172.21.15.57:0/3272915741 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f350815b2a0 0x7f35080d1910 unknown :-1 s=CLOSED pgs=2204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:42.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.276+0000 7f350d8c2640 1 -- 172.21.15.57:0/3272915741 >> 172.21.15.57:0/3272915741 conn(0x7f35080d0d60 msgr2=0x7f35080d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:42.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.276+0000 7f350d8c2640 1 -- 172.21.15.57:0/3272915741 shutdown_connections 2023-10-19T14:18:42.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.276+0000 7f350d8c2640 1 -- 172.21.15.57:0/3272915741 wait complete. 2023-10-19T14:18:42.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.277+0000 7f350d8c2640 1 Processor -- start 2023-10-19T14:18:42.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.278+0000 7f350d8c2640 1 -- start start 2023-10-19T14:18:42.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.278+0000 7f350d8c2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f35080ceb40 0x7f35081f4560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:42.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.278+0000 7f350d8c2640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f35080cf340 0x7f3508145fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:42.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.278+0000 7f350d8c2640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f350815b2a0 0x7f350815dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:42.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.278+0000 7f350d8c2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f350815d910 con 0x7f35080ceb40 2023-10-19T14:18:42.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.278+0000 7f350d8c2640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f350815cf70 con 0x7f350815b2a0 2023-10-19T14:18:42.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.278+0000 7f350d8c2640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f350815d250 con 0x7f35080cf340 2023-10-19T14:18:42.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.278+0000 7f3506ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f35080ceb40 0x7f35081f4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:42.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.278+0000 7f3506ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f35080ceb40 0x7f35081f4560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55570/0 (socket says 172.21.15.57:55570) 2023-10-19T14:18:42.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.278+0000 7f3506ffd640 1 -- 172.21.15.57:0/3616023984 learned_addr learned my addr 172.21.15.57:0/3616023984 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:18:42.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.279+0000 7f35077fe640 1 --2- 172.21.15.57:0/3616023984 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f350815b2a0 0x7f350815dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:42.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.279+0000 7f34fffff640 1 --2- 172.21.15.57:0/3616023984 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f35080cf340 0x7f3508145fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:42.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.279+0000 7f3506ffd640 1 -- 172.21.15.57:0/3616023984 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f35080cf340 msgr2=0x7f3508145fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:42.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.279+0000 7f3506ffd640 1 --2- 172.21.15.57:0/3616023984 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f35080cf340 0x7f3508145fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:42.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.279+0000 7f3506ffd640 1 -- 172.21.15.57:0/3616023984 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f350815b2a0 msgr2=0x7f350815dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:42.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.279+0000 7f3506ffd640 1 --2- 172.21.15.57:0/3616023984 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f350815b2a0 0x7f350815dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:42.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.279+0000 7f3506ffd640 1 -- 172.21.15.57:0/3616023984 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34f0009560 con 0x7f35080ceb40 2023-10-19T14:18:42.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.279+0000 7f34fffff640 1 --2- 172.21.15.57:0/3616023984 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f35080cf340 0x7f3508145fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:18:42.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.279+0000 7f3506ffd640 1 --2- 172.21.15.57:0/3616023984 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f35080ceb40 0x7f35081f4560 secure :-1 s=READY pgs=2205 cs=0 l=1 rev1=1 crypto rx=0x7f34f800c450 tx=0x7f34f800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:42.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.279+0000 7f3504ff9640 1 -- 172.21.15.57:0/3616023984 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34f80052d0 con 0x7f35080ceb40 2023-10-19T14:18:42.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.279+0000 7f3504ff9640 1 -- 172.21.15.57:0/3616023984 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f34f8005430 con 0x7f35080ceb40 2023-10-19T14:18:42.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.279+0000 7f350d8c2640 1 -- 172.21.15.57:0/3616023984 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f350815d530 con 0x7f35080ceb40 2023-10-19T14:18:42.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.279+0000 7f3504ff9640 1 -- 172.21.15.57:0/3616023984 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34f8012890 con 0x7f35080ceb40 2023-10-19T14:18:42.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.279+0000 7f350d8c2640 1 -- 172.21.15.57:0/3616023984 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3508146520 con 0x7f35080ceb40 2023-10-19T14:18:42.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.280+0000 7f350d8c2640 1 -- 172.21.15.57:0/3616023984 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35080690e0 con 0x7f35080ceb40 2023-10-19T14:18:42.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.281+0000 7f3504ff9640 1 -- 172.21.15.57:0/3616023984 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 117) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f34f80035f0 con 0x7f35080ceb40 2023-10-19T14:18:42.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.281+0000 7f3504ff9640 1 --2- 172.21.15.57:0/3616023984 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f34e806af60 0x7f34e806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:42.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.281+0000 7f3504ff9640 1 -- 172.21.15.57:0/3616023984 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f34f8089470 con 0x7f35080ceb40 2023-10-19T14:18:42.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.281+0000 7f34fffff640 1 --2- 172.21.15.57:0/3616023984 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f34e806af60 0x7f34e806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:42.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.282+0000 7f34fffff640 1 --2- 172.21.15.57:0/3616023984 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f34e806af60 0x7f34e806d420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f34f0035000 tx=0x7f34f0004730 comp rx=0 tx=0).ready entity=mgr.8744 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:42.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.285+0000 7f3504ff9640 1 -- 172.21.15.57:0/3616023984 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f34f8054340 con 0x7f35080ceb40 2023-10-19T14:18:42.462 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.460+0000 7f350d8c2640 1 -- 172.21.15.57:0/3616023984 --> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f35080b0d00 con 0x7f34e806af60 2023-10-19T14:18:42.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.468+0000 7f3504ff9640 1 -- 172.21.15.57:0/3616023984 <== mgr.8744 v2:172.21.15.179:6832/2730461479 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289958 (secure 0 0 0) 0x7f35080b0d00 con 0x7f34e806af60 2023-10-19T14:18:42.474 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:18:42.476 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2023-10-19T14:18:42.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.471+0000 7f350d8c2640 1 -- 172.21.15.57:0/3616023984 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f34e806af60 msgr2=0x7f34e806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:42.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.471+0000 7f350d8c2640 1 --2- 172.21.15.57:0/3616023984 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f34e806af60 0x7f34e806d420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f34f0035000 tx=0x7f34f0004730 comp rx=0 tx=0).stop 2023-10-19T14:18:42.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.471+0000 7f350d8c2640 1 -- 172.21.15.57:0/3616023984 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f35080ceb40 msgr2=0x7f35081f4560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:42.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.471+0000 7f350d8c2640 1 --2- 172.21.15.57:0/3616023984 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f35080ceb40 0x7f35081f4560 secure :-1 s=READY pgs=2205 cs=0 l=1 rev1=1 crypto rx=0x7f34f800c450 tx=0x7f34f800c910 comp rx=0 tx=0).stop 2023-10-19T14:18:42.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.471+0000 7f350d8c2640 1 -- 172.21.15.57:0/3616023984 shutdown_connections 2023-10-19T14:18:42.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.471+0000 7f350d8c2640 1 --2- 172.21.15.57:0/3616023984 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f34e806af60 0x7f34e806d420 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:42.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.471+0000 7f350d8c2640 1 --2- 172.21.15.57:0/3616023984 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f350815b2a0 0x7f350815dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:42.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.471+0000 7f350d8c2640 1 --2- 172.21.15.57:0/3616023984 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f35080cf340 0x7f3508145fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:42.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.471+0000 7f350d8c2640 1 --2- 172.21.15.57:0/3616023984 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f35080ceb40 0x7f35081f4560 unknown :-1 s=CLOSED pgs=2205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:42.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.471+0000 7f350d8c2640 1 -- 172.21.15.57:0/3616023984 >> 172.21.15.57:0/3616023984 conn(0x7f35080d0d60 msgr2=0x7f350806b5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:42.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.471+0000 7f350d8c2640 1 -- 172.21.15.57:0/3616023984 shutdown_connections 2023-10-19T14:18:42.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.471+0000 7f350d8c2640 1 -- 172.21.15.57:0/3616023984 wait complete. 2023-10-19T14:18:42.485 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-10-19T14:18:41.875178+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2018,"num_read_kb":4212,"num_write":593,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3202,"ondisk_log_size":3202,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":56,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":446900,"kb_used_data":21172,"kb_used_omap":12,"kb_used_meta":425587,"kb_avail":754527820,"statfs":{"total":773094113280,"available":772636487680,"internally_reserved":0,"allocated":21680128,"data_stored":11622649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":435801657},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.416866"},"pg_stats":[{"pgid":"26.3f","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.324443+0000","last_change":"2023-10-19T14:18:39.322803+0000","last_active":"2023-10-19T14:18:39.324443+0000","last_peered":"2023-10-19T14:18:39.324443+0000","last_clean":"2023-10-19T14:18:39.324443+0000","last_became_active":"2023-10-19T14:18:39.322536+0000","last_became_peered":"2023-10-19T14:18:39.322536+0000","last_unstale":"2023-10-19T14:18:39.324443+0000","last_undegraded":"2023-10-19T14:18:39.324443+0000","last_fullsized":"2023-10-19T14:18:39.324443+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:59:58.753855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.3e","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.319203+0000","last_change":"2023-10-19T14:18:39.315553+0000","last_active":"2023-10-19T14:18:39.319203+0000","last_peered":"2023-10-19T14:18:39.319203+0000","last_clean":"2023-10-19T14:18:39.319203+0000","last_became_active":"2023-10-19T14:18:39.315289+0000","last_became_peered":"2023-10-19T14:18:39.315289+0000","last_unstale":"2023-10-19T14:18:39.319203+0000","last_undegraded":"2023-10-19T14:18:39.319203+0000","last_fullsized":"2023-10-19T14:18:39.319203+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:12:51.586440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.3d","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.321113+0000","last_change":"2023-10-19T14:18:39.316231+0000","last_active":"2023-10-19T14:18:39.321113+0000","last_peered":"2023-10-19T14:18:39.321113+0000","last_clean":"2023-10-19T14:18:39.321113+0000","last_became_active":"2023-10-19T14:18:39.315932+0000","last_became_peered":"2023-10-19T14:18:39.315932+0000","last_unstale":"2023-10-19T14:18:39.321113+0000","last_undegraded":"2023-10-19T14:18:39.321113+0000","last_fullsized":"2023-10-19T14:18:39.321113+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:16:48.927842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.3c","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.324172+0000","last_change":"2023-10-19T14:18:39.321383+0000","last_active":"2023-10-19T14:18:39.324172+0000","last_peered":"2023-10-19T14:18:39.324172+0000","last_clean":"2023-10-19T14:18:39.324172+0000","last_became_active":"2023-10-19T14:18:39.321129+0000","last_became_peered":"2023-10-19T14:18:39.321129+0000","last_unstale":"2023-10-19T14:18:39.324172+0000","last_undegraded":"2023-10-19T14:18:39.324172+0000","last_fullsized":"2023-10-19T14:18:39.324172+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:52:51.790069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.3b","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.315083+0000","last_change":"2023-10-19T14:18:39.313841+0000","last_active":"2023-10-19T14:18:39.315083+0000","last_peered":"2023-10-19T14:18:39.315083+0000","last_clean":"2023-10-19T14:18:39.315083+0000","last_became_active":"2023-10-19T14:18:39.313566+0000","last_became_peered":"2023-10-19T14:18:39.313566+0000","last_unstale":"2023-10-19T14:18:39.315083+0000","last_undegraded":"2023-10-19T14:18:39.315083+0000","last_fullsized":"2023-10-19T14:18:39.315083+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:51:46.164589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.3a","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.321815+0000","last_change":"2023-10-19T14:18:39.316958+0000","last_active":"2023-10-19T14:18:39.321815+0000","last_peered":"2023-10-19T14:18:39.321815+0000","last_clean":"2023-10-19T14:18:39.321815+0000","last_became_active":"2023-10-19T14:18:39.316704+0000","last_became_peered":"2023-10-19T14:18:39.316704+0000","last_unstale":"2023-10-19T14:18:39.321815+0000","last_undegraded":"2023-10-19T14:18:39.321815+0000","last_fullsized":"2023-10-19T14:18:39.321815+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:48:12.573140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.39","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.323581+0000","last_change":"2023-10-19T14:18:39.318686+0000","last_active":"2023-10-19T14:18:39.323581+0000","last_peered":"2023-10-19T14:18:39.323581+0000","last_clean":"2023-10-19T14:18:39.323581+0000","last_became_active":"2023-10-19T14:18:39.318421+0000","last_became_peered":"2023-10-19T14:18:39.318421+0000","last_unstale":"2023-10-19T14:18:39.323581+0000","last_undegraded":"2023-10-19T14:18:39.323581+0000","last_fullsized":"2023-10-19T14:18:39.323581+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:33:02.890838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.38","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.324708+0000","last_change":"2023-10-19T14:18:39.320746+0000","last_active":"2023-10-19T14:18:39.324708+0000","last_peered":"2023-10-19T14:18:39.324708+0000","last_clean":"2023-10-19T14:18:39.324708+0000","last_became_active":"2023-10-19T14:18:39.320078+0000","last_became_peered":"2023-10-19T14:18:39.320078+0000","last_unstale":"2023-10-19T14:18:39.324708+0000","last_undegraded":"2023-10-19T14:18:39.324708+0000","last_fullsized":"2023-10-19T14:18:39.324708+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:24:12.867737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.37","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.322942+0000","last_change":"2023-10-19T14:18:39.321093+0000","last_active":"2023-10-19T14:18:39.322942+0000","last_peered":"2023-10-19T14:18:39.322942+0000","last_clean":"2023-10-19T14:18:39.322942+0000","last_became_active":"2023-10-19T14:18:39.317029+0000","last_became_peered":"2023-10-19T14:18:39.317029+0000","last_unstale":"2023-10-19T14:18:39.322942+0000","last_undegraded":"2023-10-19T14:18:39.322942+0000","last_fullsized":"2023-10-19T14:18:39.322942+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:52:37.893529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.36","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.323859+0000","last_change":"2023-10-19T14:18:39.323088+0000","last_active":"2023-10-19T14:18:39.323859+0000","last_peered":"2023-10-19T14:18:39.323859+0000","last_clean":"2023-10-19T14:18:39.323859+0000","last_became_active":"2023-10-19T14:18:39.322826+0000","last_became_peered":"2023-10-19T14:18:39.322826+0000","last_unstale":"2023-10-19T14:18:39.323859+0000","last_undegraded":"2023-10-19T14:18:39.323859+0000","last_fullsized":"2023-10-19T14:18:39.323859+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:41:18.044931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.35","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.320711+0000","last_change":"2023-10-19T14:18:39.318272+0000","last_active":"2023-10-19T14:18:39.320711+0000","last_peered":"2023-10-19T14:18:39.320711+0000","last_clean":"2023-10-19T14:18:39.320711+0000","last_became_active":"2023-10-19T14:18:39.317967+0000","last_became_peered":"2023-10-19T14:18:39.317967+0000","last_unstale":"2023-10-19T14:18:39.320711+0000","last_undegraded":"2023-10-19T14:18:39.320711+0000","last_fullsized":"2023-10-19T14:18:39.320711+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T02:02:07.160434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.34","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.312253+0000","last_change":"2023-10-19T14:18:39.309562+0000","last_active":"2023-10-19T14:18:39.312253+0000","last_peered":"2023-10-19T14:18:39.312253+0000","last_clean":"2023-10-19T14:18:39.312253+0000","last_became_active":"2023-10-19T14:18:39.309302+0000","last_became_peered":"2023-10-19T14:18:39.309302+0000","last_unstale":"2023-10-19T14:18:39.312253+0000","last_undegraded":"2023-10-19T14:18:39.312253+0000","last_fullsized":"2023-10-19T14:18:39.312253+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:36:22.594185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.33","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.326719+0000","last_change":"2023-10-19T14:18:39.322907+0000","last_active":"2023-10-19T14:18:39.326719+0000","last_peered":"2023-10-19T14:18:39.326719+0000","last_clean":"2023-10-19T14:18:39.326719+0000","last_became_active":"2023-10-19T14:18:39.322386+0000","last_became_peered":"2023-10-19T14:18:39.322386+0000","last_unstale":"2023-10-19T14:18:39.326719+0000","last_undegraded":"2023-10-19T14:18:39.326719+0000","last_fullsized":"2023-10-19T14:18:39.326719+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:44:23.406435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.32","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.323734+0000","last_change":"2023-10-19T14:18:39.320215+0000","last_active":"2023-10-19T14:18:39.323734+0000","last_peered":"2023-10-19T14:18:39.323734+0000","last_clean":"2023-10-19T14:18:39.323734+0000","last_became_active":"2023-10-19T14:18:39.319939+0000","last_became_peered":"2023-10-19T14:18:39.319939+0000","last_unstale":"2023-10-19T14:18:39.323734+0000","last_undegraded":"2023-10-19T14:18:39.323734+0000","last_fullsized":"2023-10-19T14:18:39.323734+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:16:35.591881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.31","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.320606+0000","last_change":"2023-10-19T14:18:39.318398+0000","last_active":"2023-10-19T14:18:39.320606+0000","last_peered":"2023-10-19T14:18:39.320606+0000","last_clean":"2023-10-19T14:18:39.320606+0000","last_became_active":"2023-10-19T14:18:39.318113+0000","last_became_peered":"2023-10-19T14:18:39.318113+0000","last_unstale":"2023-10-19T14:18:39.320606+0000","last_undegraded":"2023-10-19T14:18:39.320606+0000","last_fullsized":"2023-10-19T14:18:39.320606+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:28:53.413157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.30","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.315874+0000","last_change":"2023-10-19T14:18:39.314377+0000","last_active":"2023-10-19T14:18:39.315874+0000","last_peered":"2023-10-19T14:18:39.315874+0000","last_clean":"2023-10-19T14:18:39.315874+0000","last_became_active":"2023-10-19T14:18:39.314119+0000","last_became_peered":"2023-10-19T14:18:39.314119+0000","last_unstale":"2023-10-19T14:18:39.315874+0000","last_undegraded":"2023-10-19T14:18:39.315874+0000","last_fullsized":"2023-10-19T14:18:39.315874+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:20:04.481873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.2f","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.319792+0000","last_change":"2023-10-19T14:18:39.315448+0000","last_active":"2023-10-19T14:18:39.319792+0000","last_peered":"2023-10-19T14:18:39.319792+0000","last_clean":"2023-10-19T14:18:39.319792+0000","last_became_active":"2023-10-19T14:18:39.315108+0000","last_became_peered":"2023-10-19T14:18:39.315108+0000","last_unstale":"2023-10-19T14:18:39.319792+0000","last_undegraded":"2023-10-19T14:18:39.319792+0000","last_fullsized":"2023-10-19T14:18:39.319792+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:06:20.776277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.2e","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.326637+0000","last_change":"2023-10-19T14:18:39.322035+0000","last_active":"2023-10-19T14:18:39.326637+0000","last_peered":"2023-10-19T14:18:39.326637+0000","last_clean":"2023-10-19T14:18:39.326637+0000","last_became_active":"2023-10-19T14:18:39.321764+0000","last_became_peered":"2023-10-19T14:18:39.321764+0000","last_unstale":"2023-10-19T14:18:39.326637+0000","last_undegraded":"2023-10-19T14:18:39.326637+0000","last_fullsized":"2023-10-19T14:18:39.326637+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:05:13.696675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.2d","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.323140+0000","last_change":"2023-10-19T14:18:39.317826+0000","last_active":"2023-10-19T14:18:39.323140+0000","last_peered":"2023-10-19T14:18:39.323140+0000","last_clean":"2023-10-19T14:18:39.323140+0000","last_became_active":"2023-10-19T14:18:39.317325+0000","last_became_peered":"2023-10-19T14:18:39.317325+0000","last_unstale":"2023-10-19T14:18:39.323140+0000","last_undegraded":"2023-10-19T14:18:39.323140+0000","last_fullsized":"2023-10-19T14:18:39.323140+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:52:36.474116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.2c","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.325351+0000","last_change":"2023-10-19T14:18:39.324937+0000","last_active":"2023-10-19T14:18:39.325351+0000","last_peered":"2023-10-19T14:18:39.325351+0000","last_clean":"2023-10-19T14:18:39.325351+0000","last_became_active":"2023-10-19T14:18:39.324523+0000","last_became_peered":"2023-10-19T14:18:39.324523+0000","last_unstale":"2023-10-19T14:18:39.325351+0000","last_undegraded":"2023-10-19T14:18:39.325351+0000","last_fullsized":"2023-10-19T14:18:39.325351+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:09:16.621219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.2b","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.320947+0000","last_change":"2023-10-19T14:18:39.319111+0000","last_active":"2023-10-19T14:18:39.320947+0000","last_peered":"2023-10-19T14:18:39.320947+0000","last_clean":"2023-10-19T14:18:39.320947+0000","last_became_active":"2023-10-19T14:18:39.318343+0000","last_became_peered":"2023-10-19T14:18:39.318343+0000","last_unstale":"2023-10-19T14:18:39.320947+0000","last_undegraded":"2023-10-19T14:18:39.320947+0000","last_fullsized":"2023-10-19T14:18:39.320947+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:33:47.182338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.2a","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.320581+0000","last_change":"2023-10-19T14:18:39.317915+0000","last_active":"2023-10-19T14:18:39.320581+0000","last_peered":"2023-10-19T14:18:39.320581+0000","last_clean":"2023-10-19T14:18:39.320581+0000","last_became_active":"2023-10-19T14:18:39.317638+0000","last_became_peered":"2023-10-19T14:18:39.317638+0000","last_unstale":"2023-10-19T14:18:39.320581+0000","last_undegraded":"2023-10-19T14:18:39.320581+0000","last_fullsized":"2023-10-19T14:18:39.320581+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:31:02.673391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.29","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.324374+0000","last_change":"2023-10-19T14:18:39.321228+0000","last_active":"2023-10-19T14:18:39.324374+0000","last_peered":"2023-10-19T14:18:39.324374+0000","last_clean":"2023-10-19T14:18:39.324374+0000","last_became_active":"2023-10-19T14:18:39.320925+0000","last_became_peered":"2023-10-19T14:18:39.320925+0000","last_unstale":"2023-10-19T14:18:39.324374+0000","last_undegraded":"2023-10-19T14:18:39.324374+0000","last_fullsized":"2023-10-19T14:18:39.324374+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:58:28.744223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.28","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.324404+0000","last_change":"2023-10-19T14:18:39.323433+0000","last_active":"2023-10-19T14:18:39.324404+0000","last_peered":"2023-10-19T14:18:39.324404+0000","last_clean":"2023-10-19T14:18:39.324404+0000","last_became_active":"2023-10-19T14:18:39.323048+0000","last_became_peered":"2023-10-19T14:18:39.323048+0000","last_unstale":"2023-10-19T14:18:39.324404+0000","last_undegraded":"2023-10-19T14:18:39.324404+0000","last_fullsized":"2023-10-19T14:18:39.324404+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:26:44.530531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.27","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.323612+0000","last_change":"2023-10-19T14:18:39.320272+0000","last_active":"2023-10-19T14:18:39.323612+0000","last_peered":"2023-10-19T14:18:39.323612+0000","last_clean":"2023-10-19T14:18:39.323612+0000","last_became_active":"2023-10-19T14:18:39.319840+0000","last_became_peered":"2023-10-19T14:18:39.319840+0000","last_unstale":"2023-10-19T14:18:39.323612+0000","last_undegraded":"2023-10-19T14:18:39.323612+0000","last_fullsized":"2023-10-19T14:18:39.323612+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:46:10.846757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.26","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.315303+0000","last_change":"2023-10-19T14:18:39.314106+0000","last_active":"2023-10-19T14:18:39.315303+0000","last_peered":"2023-10-19T14:18:39.315303+0000","last_clean":"2023-10-19T14:18:39.315303+0000","last_became_active":"2023-10-19T14:18:39.313685+0000","last_became_peered":"2023-10-19T14:18:39.313685+0000","last_unstale":"2023-10-19T14:18:39.315303+0000","last_undegraded":"2023-10-19T14:18:39.315303+0000","last_fullsized":"2023-10-19T14:18:39.315303+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:23:30.741166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.25","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.320804+0000","last_change":"2023-10-19T14:18:39.317419+0000","last_active":"2023-10-19T14:18:39.320804+0000","last_peered":"2023-10-19T14:18:39.320804+0000","last_clean":"2023-10-19T14:18:39.320804+0000","last_became_active":"2023-10-19T14:18:39.317038+0000","last_became_peered":"2023-10-19T14:18:39.317038+0000","last_unstale":"2023-10-19T14:18:39.320804+0000","last_undegraded":"2023-10-19T14:18:39.320804+0000","last_fullsized":"2023-10-19T14:18:39.320804+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:02:47.020095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.24","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.323098+0000","last_change":"2023-10-19T14:18:39.321201+0000","last_active":"2023-10-19T14:18:39.323098+0000","last_peered":"2023-10-19T14:18:39.323098+0000","last_clean":"2023-10-19T14:18:39.323098+0000","last_became_active":"2023-10-19T14:18:39.319434+0000","last_became_peered":"2023-10-19T14:18:39.319434+0000","last_unstale":"2023-10-19T14:18:39.323098+0000","last_undegraded":"2023-10-19T14:18:39.323098+0000","last_fullsized":"2023-10-19T14:18:39.323098+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:57:26.768932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.23","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.315313+0000","last_change":"2023-10-19T14:18:39.312602+0000","last_active":"2023-10-19T14:18:39.315313+0000","last_peered":"2023-10-19T14:18:39.315313+0000","last_clean":"2023-10-19T14:18:39.315313+0000","last_became_active":"2023-10-19T14:18:39.312266+0000","last_became_peered":"2023-10-19T14:18:39.312266+0000","last_unstale":"2023-10-19T14:18:39.315313+0000","last_undegraded":"2023-10-19T14:18:39.315313+0000","last_fullsized":"2023-10-19T14:18:39.315313+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:32:11.986790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.22","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.315681+0000","last_change":"2023-10-19T14:18:39.311988+0000","last_active":"2023-10-19T14:18:39.315681+0000","last_peered":"2023-10-19T14:18:39.315681+0000","last_clean":"2023-10-19T14:18:39.315681+0000","last_became_active":"2023-10-19T14:18:39.311704+0000","last_became_peered":"2023-10-19T14:18:39.311704+0000","last_unstale":"2023-10-19T14:18:39.315681+0000","last_undegraded":"2023-10-19T14:18:39.315681+0000","last_fullsized":"2023-10-19T14:18:39.315681+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:34:55.930079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.21","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.314806+0000","last_change":"2023-10-19T14:18:39.311776+0000","last_active":"2023-10-19T14:18:39.314806+0000","last_peered":"2023-10-19T14:18:39.314806+0000","last_clean":"2023-10-19T14:18:39.314806+0000","last_became_active":"2023-10-19T14:18:39.311465+0000","last_became_peered":"2023-10-19T14:18:39.311465+0000","last_unstale":"2023-10-19T14:18:39.314806+0000","last_undegraded":"2023-10-19T14:18:39.314806+0000","last_fullsized":"2023-10-19T14:18:39.314806+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:30:13.687170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.20","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.324582+0000","last_change":"2023-10-19T14:18:39.323494+0000","last_active":"2023-10-19T14:18:39.324582+0000","last_peered":"2023-10-19T14:18:39.324582+0000","last_clean":"2023-10-19T14:18:39.324582+0000","last_became_active":"2023-10-19T14:18:39.321826+0000","last_became_peered":"2023-10-19T14:18:39.321826+0000","last_unstale":"2023-10-19T14:18:39.324582+0000","last_undegraded":"2023-10-19T14:18:39.324582+0000","last_fullsized":"2023-10-19T14:18:39.324582+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:48:53.137786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1c","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.324469+0000","last_change":"2023-10-19T14:18:39.321449+0000","last_active":"2023-10-19T14:18:39.324469+0000","last_peered":"2023-10-19T14:18:39.324469+0000","last_clean":"2023-10-19T14:18:39.324469+0000","last_became_active":"2023-10-19T14:18:39.321172+0000","last_became_peered":"2023-10-19T14:18:39.321172+0000","last_unstale":"2023-10-19T14:18:39.324469+0000","last_undegraded":"2023-10-19T14:18:39.324469+0000","last_fullsized":"2023-10-19T14:18:39.324469+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:36:46.457621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.1f","version":"204'4","reported_seq":15,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.481238+0000","last_change":"2023-10-19T14:18:38.485999+0000","last_active":"2023-10-19T14:18:39.481238+0000","last_peered":"2023-10-19T14:18:39.481238+0000","last_clean":"2023-10-19T14:18:39.481238+0000","last_became_active":"2023-10-19T14:18:38.485732+0000","last_became_peered":"2023-10-19T14:18:38.485732+0000","last_unstale":"2023-10-19T14:18:39.481238+0000","last_undegraded":"2023-10-19T14:18:39.481238+0000","last_fullsized":"2023-10-19T14:18:39.481238+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_clean_scrub_stamp":"2023-10-19T14:18:37.448968+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:09:08.257110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1d","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.324399+0000","last_change":"2023-10-19T14:18:39.321188+0000","last_active":"2023-10-19T14:18:39.324399+0000","last_peered":"2023-10-19T14:18:39.324399+0000","last_clean":"2023-10-19T14:18:39.324399+0000","last_became_active":"2023-10-19T14:18:39.320859+0000","last_became_peered":"2023-10-19T14:18:39.320859+0000","last_unstale":"2023-10-19T14:18:39.324399+0000","last_undegraded":"2023-10-19T14:18:39.324399+0000","last_fullsized":"2023-10-19T14:18:39.324399+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T02:14:35.102807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1e","version":"204'1","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.477262+0000","last_change":"2023-10-19T14:18:38.475466+0000","last_active":"2023-10-19T14:18:39.477262+0000","last_peered":"2023-10-19T14:18:39.477262+0000","last_clean":"2023-10-19T14:18:39.477262+0000","last_became_active":"2023-10-19T14:18:38.475088+0000","last_became_peered":"2023-10-19T14:18:38.475088+0000","last_unstale":"2023-10-19T14:18:39.477262+0000","last_undegraded":"2023-10-19T14:18:39.477262+0000","last_fullsized":"2023-10-19T14:18:39.477262+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_clean_scrub_stamp":"2023-10-19T14:18:37.448968+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:30:38.136460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1e","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.321890+0000","last_change":"2023-10-19T14:18:39.320557+0000","last_active":"2023-10-19T14:18:39.321890+0000","last_peered":"2023-10-19T14:18:39.321890+0000","last_clean":"2023-10-19T14:18:39.321890+0000","last_became_active":"2023-10-19T14:18:39.320236+0000","last_became_peered":"2023-10-19T14:18:39.320236+0000","last_unstale":"2023-10-19T14:18:39.321890+0000","last_undegraded":"2023-10-19T14:18:39.321890+0000","last_fullsized":"2023-10-19T14:18:39.321890+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:18:55.511235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.1d","version":"204'2","reported_seq":13,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.477241+0000","last_change":"2023-10-19T14:18:38.479859+0000","last_active":"2023-10-19T14:18:39.477241+0000","last_peered":"2023-10-19T14:18:39.477241+0000","last_clean":"2023-10-19T14:18:39.477241+0000","last_became_active":"2023-10-19T14:18:38.479322+0000","last_became_peered":"2023-10-19T14:18:38.479322+0000","last_unstale":"2023-10-19T14:18:39.477241+0000","last_undegraded":"2023-10-19T14:18:39.477241+0000","last_fullsized":"2023-10-19T14:18:39.477241+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_clean_scrub_stamp":"2023-10-19T14:18:37.448968+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:30:58.065702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1f","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.321482+0000","last_change":"2023-10-19T14:18:39.316917+0000","last_active":"2023-10-19T14:18:39.321482+0000","last_peered":"2023-10-19T14:18:39.321482+0000","last_clean":"2023-10-19T14:18:39.321482+0000","last_became_active":"2023-10-19T14:18:39.316620+0000","last_became_peered":"2023-10-19T14:18:39.316620+0000","last_unstale":"2023-10-19T14:18:39.321482+0000","last_undegraded":"2023-10-19T14:18:39.321482+0000","last_fullsized":"2023-10-19T14:18:39.321482+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:40:21.367319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1c","version":"204'2","reported_seq":13,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.481289+0000","last_change":"2023-10-19T14:18:38.485790+0000","last_active":"2023-10-19T14:18:39.481289+0000","last_peered":"2023-10-19T14:18:39.481289+0000","last_clean":"2023-10-19T14:18:39.481289+0000","last_became_active":"2023-10-19T14:18:38.485478+0000","last_became_peered":"2023-10-19T14:18:38.485478+0000","last_unstale":"2023-10-19T14:18:39.481289+0000","last_undegraded":"2023-10-19T14:18:39.481289+0000","last_fullsized":"2023-10-19T14:18:39.481289+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_clean_scrub_stamp":"2023-10-19T14:18:37.448968+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T02:04:51.639909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.319288+0000","last_change":"2023-10-19T14:18:39.318138+0000","last_active":"2023-10-19T14:18:39.319288+0000","last_peered":"2023-10-19T14:18:39.319288+0000","last_clean":"2023-10-19T14:18:39.319288+0000","last_became_active":"2023-10-19T14:18:39.317869+0000","last_became_peered":"2023-10-19T14:18:39.317869+0000","last_unstale":"2023-10-19T14:18:39.319288+0000","last_undegraded":"2023-10-19T14:18:39.319288+0000","last_fullsized":"2023-10-19T14:18:39.319288+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:10:10.162980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.1b","version":"204'1","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.476702+0000","last_change":"2023-10-19T14:18:38.474975+0000","last_active":"2023-10-19T14:18:39.476702+0000","last_peered":"2023-10-19T14:18:39.476702+0000","last_clean":"2023-10-19T14:18:39.476702+0000","last_became_active":"2023-10-19T14:18:38.473518+0000","last_became_peered":"2023-10-19T14:18:38.473518+0000","last_unstale":"2023-10-19T14:18:39.476702+0000","last_undegraded":"2023-10-19T14:18:39.476702+0000","last_fullsized":"2023-10-19T14:18:39.476702+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_clean_scrub_stamp":"2023-10-19T14:18:37.448968+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:42:15.965695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.312442+0000","last_change":"2023-10-19T14:18:39.311444+0000","last_active":"2023-10-19T14:18:39.312442+0000","last_peered":"2023-10-19T14:18:39.312442+0000","last_clean":"2023-10-19T14:18:39.312442+0000","last_became_active":"2023-10-19T14:18:39.311184+0000","last_became_peered":"2023-10-19T14:18:39.311184+0000","last_unstale":"2023-10-19T14:18:39.312442+0000","last_undegraded":"2023-10-19T14:18:39.312442+0000","last_fullsized":"2023-10-19T14:18:39.312442+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:21:35.591006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1a","version":"204'1","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.477439+0000","last_change":"2023-10-19T14:18:38.483165+0000","last_active":"2023-10-19T14:18:39.477439+0000","last_peered":"2023-10-19T14:18:39.477439+0000","last_clean":"2023-10-19T14:18:39.477439+0000","last_became_active":"2023-10-19T14:18:38.482417+0000","last_became_peered":"2023-10-19T14:18:38.482417+0000","last_unstale":"2023-10-19T14:18:39.477439+0000","last_undegraded":"2023-10-19T14:18:39.477439+0000","last_fullsized":"2023-10-19T14:18:39.477439+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_clean_scrub_stamp":"2023-10-19T14:18:37.448968+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:58:27.737188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1a","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.323767+0000","last_change":"2023-10-19T14:18:39.322322+0000","last_active":"2023-10-19T14:18:39.323767+0000","last_peered":"2023-10-19T14:18:39.323767+0000","last_clean":"2023-10-19T14:18:39.323767+0000","last_became_active":"2023-10-19T14:18:39.321983+0000","last_became_peered":"2023-10-19T14:18:39.321983+0000","last_unstale":"2023-10-19T14:18:39.323767+0000","last_undegraded":"2023-10-19T14:18:39.323767+0000","last_fullsized":"2023-10-19T14:18:39.323767+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:49:19.779416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.19","version":"0'0","reported_seq":11,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.310595+0000","last_change":"2023-10-19T14:18:38.476189+0000","last_active":"2023-10-19T14:18:39.310595+0000","last_peered":"2023-10-19T14:18:39.310595+0000","last_clean":"2023-10-19T14:18:39.310595+0000","last_became_active":"2023-10-19T14:18:38.475250+0000","last_became_peered":"2023-10-19T14:18:38.475250+0000","last_unstale":"2023-10-19T14:18:39.310595+0000","last_undegraded":"2023-10-19T14:18:39.310595+0000","last_fullsized":"2023-10-19T14:18:39.310595+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_clean_scrub_stamp":"2023-10-19T14:18:37.448968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:11:51.980596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.14","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.324487+0000","last_change":"2023-10-19T14:18:39.322238+0000","last_active":"2023-10-19T14:18:39.324487+0000","last_peered":"2023-10-19T14:18:39.324487+0000","last_clean":"2023-10-19T14:18:39.324487+0000","last_became_active":"2023-10-19T14:18:39.321870+0000","last_became_peered":"2023-10-19T14:18:39.321870+0000","last_unstale":"2023-10-19T14:18:39.324487+0000","last_undegraded":"2023-10-19T14:18:39.324487+0000","last_fullsized":"2023-10-19T14:18:39.324487+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:18:25.998173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.17","version":"0'0","reported_seq":11,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.305798+0000","last_change":"2023-10-19T14:18:38.473330+0000","last_active":"2023-10-19T14:18:39.305798+0000","last_peered":"2023-10-19T14:18:39.305798+0000","last_clean":"2023-10-19T14:18:39.305798+0000","last_became_active":"2023-10-19T14:18:38.472987+0000","last_became_peered":"2023-10-19T14:18:38.472987+0000","last_unstale":"2023-10-19T14:18:39.305798+0000","last_undegraded":"2023-10-19T14:18:39.305798+0000","last_fullsized":"2023-10-19T14:18:39.305798+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_clean_scrub_stamp":"2023-10-19T14:18:37.448968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:24:20.646403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.15","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.315033+0000","last_change":"2023-10-19T14:18:39.313590+0000","last_active":"2023-10-19T14:18:39.315033+0000","last_peered":"2023-10-19T14:18:39.315033+0000","last_clean":"2023-10-19T14:18:39.315033+0000","last_became_active":"2023-10-19T14:18:39.313328+0000","last_became_peered":"2023-10-19T14:18:39.313328+0000","last_unstale":"2023-10-19T14:18:39.315033+0000","last_undegraded":"2023-10-19T14:18:39.315033+0000","last_fullsized":"2023-10-19T14:18:39.315033+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T02:00:02.465134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.16","version":"0'0","reported_seq":11,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.309629+0000","last_change":"2023-10-19T14:18:38.482895+0000","last_active":"2023-10-19T14:18:39.309629+0000","last_peered":"2023-10-19T14:18:39.309629+0000","last_clean":"2023-10-19T14:18:39.309629+0000","last_became_active":"2023-10-19T14:18:38.482645+0000","last_became_peered":"2023-10-19T14:18:38.482645+0000","last_unstale":"2023-10-19T14:18:39.309629+0000","last_undegraded":"2023-10-19T14:18:39.309629+0000","last_fullsized":"2023-10-19T14:18:39.309629+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_clean_scrub_stamp":"2023-10-19T14:18:37.448968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:56:48.037930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.318883+0000","last_change":"2023-10-19T14:18:39.315849+0000","last_active":"2023-10-19T14:18:39.318883+0000","last_peered":"2023-10-19T14:18:39.318883+0000","last_clean":"2023-10-19T14:18:39.318883+0000","last_became_active":"2023-10-19T14:18:39.315597+0000","last_became_peered":"2023-10-19T14:18:39.315597+0000","last_unstale":"2023-10-19T14:18:39.318883+0000","last_undegraded":"2023-10-19T14:18:39.318883+0000","last_fullsized":"2023-10-19T14:18:39.318883+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:25:34.591207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.15","version":"204'3","reported_seq":14,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.480973+0000","last_change":"2023-10-19T14:18:38.482908+0000","last_active":"2023-10-19T14:18:39.480973+0000","last_peered":"2023-10-19T14:18:39.480973+0000","last_clean":"2023-10-19T14:18:39.480973+0000","last_became_active":"2023-10-19T14:18:38.482612+0000","last_became_peered":"2023-10-19T14:18:38.482612+0000","last_unstale":"2023-10-19T14:18:39.480973+0000","last_undegraded":"2023-10-19T14:18:39.480973+0000","last_fullsized":"2023-10-19T14:18:39.480973+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_clean_scrub_stamp":"2023-10-19T14:18:37.448968+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:35:21.292510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.320491+0000","last_change":"2023-10-19T14:18:39.316748+0000","last_active":"2023-10-19T14:18:39.320491+0000","last_peered":"2023-10-19T14:18:39.320491+0000","last_clean":"2023-10-19T14:18:39.320491+0000","last_became_active":"2023-10-19T14:18:39.316075+0000","last_became_peered":"2023-10-19T14:18:39.316075+0000","last_unstale":"2023-10-19T14:18:39.320491+0000","last_undegraded":"2023-10-19T14:18:39.320491+0000","last_fullsized":"2023-10-19T14:18:39.320491+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:16:45.929714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.14","version":"204'6","reported_seq":17,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.483016+0000","last_change":"2023-10-19T14:18:38.477382+0000","last_active":"2023-10-19T14:18:39.483016+0000","last_peered":"2023-10-19T14:18:39.483016+0000","last_clean":"2023-10-19T14:18:39.483016+0000","last_became_active":"2023-10-19T14:18:38.476733+0000","last_became_peered":"2023-10-19T14:18:38.476733+0000","last_unstale":"2023-10-19T14:18:39.483016+0000","last_undegraded":"2023-10-19T14:18:39.483016+0000","last_fullsized":"2023-10-19T14:18:39.483016+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_clean_scrub_stamp":"2023-10-19T14:18:37.448968+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:38:11.517639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.10","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.324576+0000","last_change":"2023-10-19T14:18:39.321629+0000","last_active":"2023-10-19T14:18:39.324576+0000","last_peered":"2023-10-19T14:18:39.324576+0000","last_clean":"2023-10-19T14:18:39.324576+0000","last_became_active":"2023-10-19T14:18:39.321356+0000","last_became_peered":"2023-10-19T14:18:39.321356+0000","last_unstale":"2023-10-19T14:18:39.324576+0000","last_undegraded":"2023-10-19T14:18:39.324576+0000","last_fullsized":"2023-10-19T14:18:39.324576+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:52:51.699459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.13","version":"204'2","reported_seq":13,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.481294+0000","last_change":"2023-10-19T14:18:38.475966+0000","last_active":"2023-10-19T14:18:39.481294+0000","last_peered":"2023-10-19T14:18:39.481294+0000","last_clean":"2023-10-19T14:18:39.481294+0000","last_became_active":"2023-10-19T14:18:38.475239+0000","last_became_peered":"2023-10-19T14:18:38.475239+0000","last_unstale":"2023-10-19T14:18:39.481294+0000","last_undegraded":"2023-10-19T14:18:39.481294+0000","last_fullsized":"2023-10-19T14:18:39.481294+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_clean_scrub_stamp":"2023-10-19T14:18:37.448968+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:16:47.920807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.320104+0000","last_change":"2023-10-19T14:18:39.316015+0000","last_active":"2023-10-19T14:18:39.320104+0000","last_peered":"2023-10-19T14:18:39.320104+0000","last_clean":"2023-10-19T14:18:39.320104+0000","last_became_active":"2023-10-19T14:18:39.315724+0000","last_became_peered":"2023-10-19T14:18:39.315724+0000","last_unstale":"2023-10-19T14:18:39.320104+0000","last_undegraded":"2023-10-19T14:18:39.320104+0000","last_fullsized":"2023-10-19T14:18:39.320104+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:51:15.427846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.12","version":"0'0","reported_seq":11,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.308847+0000","last_change":"2023-10-19T14:18:38.485409+0000","last_active":"2023-10-19T14:18:39.308847+0000","last_peered":"2023-10-19T14:18:39.308847+0000","last_clean":"2023-10-19T14:18:39.308847+0000","last_became_active":"2023-10-19T14:18:38.485133+0000","last_became_peered":"2023-10-19T14:18:38.485133+0000","last_unstale":"2023-10-19T14:18:39.308847+0000","last_undegraded":"2023-10-19T14:18:39.308847+0000","last_fullsized":"2023-10-19T14:18:39.308847+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_clean_scrub_stamp":"2023-10-19T14:18:37.448968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:50:41.815531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.320846+0000","last_change":"2023-10-19T14:18:39.319370+0000","last_active":"2023-10-19T14:18:39.320846+0000","last_peered":"2023-10-19T14:18:39.320846+0000","last_clean":"2023-10-19T14:18:39.320846+0000","last_became_active":"2023-10-19T14:18:39.318772+0000","last_became_peered":"2023-10-19T14:18:39.318772+0000","last_unstale":"2023-10-19T14:18:39.320846+0000","last_undegraded":"2023-10-19T14:18:39.320846+0000","last_fullsized":"2023-10-19T14:18:39.320846+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:18:25.389622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.11","version":"0'0","reported_seq":11,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.302590+0000","last_change":"2023-10-19T14:18:38.482814+0000","last_active":"2023-10-19T14:18:39.302590+0000","last_peered":"2023-10-19T14:18:39.302590+0000","last_clean":"2023-10-19T14:18:39.302590+0000","last_became_active":"2023-10-19T14:18:38.482538+0000","last_became_peered":"2023-10-19T14:18:38.482538+0000","last_unstale":"2023-10-19T14:18:39.302590+0000","last_undegraded":"2023-10-19T14:18:39.302590+0000","last_fullsized":"2023-10-19T14:18:39.302590+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_clean_scrub_stamp":"2023-10-19T14:18:37.448968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:47:17.141705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.13","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.326836+0000","last_change":"2023-10-19T14:18:39.323926+0000","last_active":"2023-10-19T14:18:39.326836+0000","last_peered":"2023-10-19T14:18:39.326836+0000","last_clean":"2023-10-19T14:18:39.326836+0000","last_became_active":"2023-10-19T14:18:39.323223+0000","last_became_peered":"2023-10-19T14:18:39.323223+0000","last_unstale":"2023-10-19T14:18:39.326836+0000","last_undegraded":"2023-10-19T14:18:39.326836+0000","last_fullsized":"2023-10-19T14:18:39.326836+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:25:56.932484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.10","version":"204'8","reported_seq":19,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.482607+0000","last_change":"2023-10-19T14:18:38.473672+0000","last_active":"2023-10-19T14:18:39.482607+0000","last_peered":"2023-10-19T14:18:39.482607+0000","last_clean":"2023-10-19T14:18:39.482607+0000","last_became_active":"2023-10-19T14:18:38.473346+0000","last_became_peered":"2023-10-19T14:18:38.473346+0000","last_unstale":"2023-10-19T14:18:39.482607+0000","last_undegraded":"2023-10-19T14:18:39.482607+0000","last_fullsized":"2023-10-19T14:18:39.482607+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_clean_scrub_stamp":"2023-10-19T14:18:37.448968+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:16:28.811592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.323885+0000","last_change":"2023-10-19T14:18:39.319381+0000","last_active":"2023-10-19T14:18:39.323885+0000","last_peered":"2023-10-19T14:18:39.323885+0000","last_clean":"2023-10-19T14:18:39.323885+0000","last_became_active":"2023-10-19T14:18:39.319091+0000","last_became_peered":"2023-10-19T14:18:39.319091+0000","last_unstale":"2023-10-19T14:18:39.323885+0000","last_undegraded":"2023-10-19T14:18:39.323885+0000","last_fullsized":"2023-10-19T14:18:39.323885+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:00:13.209777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.f","version":"204'1","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.481928+0000","last_change":"2023-10-19T14:18:38.473810+0000","last_active":"2023-10-19T14:18:39.481928+0000","last_peered":"2023-10-19T14:18:39.481928+0000","last_clean":"2023-10-19T14:18:39.481928+0000","last_became_active":"2023-10-19T14:18:38.473546+0000","last_became_peered":"2023-10-19T14:18:38.473546+0000","last_unstale":"2023-10-19T14:18:39.481928+0000","last_undegraded":"2023-10-19T14:18:39.481928+0000","last_fullsized":"2023-10-19T14:18:39.481928+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_clean_scrub_stamp":"2023-10-19T14:18:37.448968+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:02:52.762600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.d","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.322242+0000","last_change":"2023-10-19T14:18:39.321385+0000","last_active":"2023-10-19T14:18:39.322242+0000","last_peered":"2023-10-19T14:18:39.322242+0000","last_clean":"2023-10-19T14:18:39.322242+0000","last_became_active":"2023-10-19T14:18:39.319376+0000","last_became_peered":"2023-10-19T14:18:39.319376+0000","last_unstale":"2023-10-19T14:18:39.322242+0000","last_undegraded":"2023-10-19T14:18:39.322242+0000","last_fullsized":"2023-10-19T14:18:39.322242+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:41:05.496979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.e","version":"204'2","reported_seq":13,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.480708+0000","last_change":"2023-10-19T14:18:38.483495+0000","last_active":"2023-10-19T14:18:39.480708+0000","last_peered":"2023-10-19T14:18:39.480708+0000","last_clean":"2023-10-19T14:18:39.480708+0000","last_became_active":"2023-10-19T14:18:38.483195+0000","last_became_peered":"2023-10-19T14:18:38.483195+0000","last_unstale":"2023-10-19T14:18:39.480708+0000","last_undegraded":"2023-10-19T14:18:39.480708+0000","last_fullsized":"2023-10-19T14:18:39.480708+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_clean_scrub_stamp":"2023-10-19T14:18:37.448968+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:50:29.357721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.e","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.315282+0000","last_change":"2023-10-19T14:18:39.313942+0000","last_active":"2023-10-19T14:18:39.315282+0000","last_peered":"2023-10-19T14:18:39.315282+0000","last_clean":"2023-10-19T14:18:39.315282+0000","last_became_active":"2023-10-19T14:18:39.313320+0000","last_became_peered":"2023-10-19T14:18:39.313320+0000","last_unstale":"2023-10-19T14:18:39.315282+0000","last_undegraded":"2023-10-19T14:18:39.315282+0000","last_fullsized":"2023-10-19T14:18:39.315282+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:42:16.940277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.d","version":"204'1","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.477874+0000","last_change":"2023-10-19T14:18:38.479263+0000","last_active":"2023-10-19T14:18:39.477874+0000","last_peered":"2023-10-19T14:18:39.477874+0000","last_clean":"2023-10-19T14:18:39.477874+0000","last_became_active":"2023-10-19T14:18:38.478937+0000","last_became_peered":"2023-10-19T14:18:38.478937+0000","last_unstale":"2023-10-19T14:18:39.477874+0000","last_undegraded":"2023-10-19T14:18:39.477874+0000","last_fullsized":"2023-10-19T14:18:39.477874+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_clean_scrub_stamp":"2023-10-19T14:18:37.448968+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:42:50.809620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.f","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.320520+0000","last_change":"2023-10-19T14:18:39.317173+0000","last_active":"2023-10-19T14:18:39.320520+0000","last_peered":"2023-10-19T14:18:39.320520+0000","last_clean":"2023-10-19T14:18:39.320520+0000","last_became_active":"2023-10-19T14:18:39.316464+0000","last_became_peered":"2023-10-19T14:18:39.316464+0000","last_unstale":"2023-10-19T14:18:39.320520+0000","last_undegraded":"2023-10-19T14:18:39.320520+0000","last_fullsized":"2023-10-19T14:18:39.320520+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T02:16:33.786472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.c","version":"0'0","reported_seq":11,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.303947+0000","last_change":"2023-10-19T14:18:38.474681+0000","last_active":"2023-10-19T14:18:39.303947+0000","last_peered":"2023-10-19T14:18:39.303947+0000","last_clean":"2023-10-19T14:18:39.303947+0000","last_became_active":"2023-10-19T14:18:38.474417+0000","last_became_peered":"2023-10-19T14:18:38.474417+0000","last_unstale":"2023-10-19T14:18:39.303947+0000","last_undegraded":"2023-10-19T14:18:39.303947+0000","last_fullsized":"2023-10-19T14:18:39.303947+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_clean_scrub_stamp":"2023-10-19T14:18:37.448968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:34:33.625057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.320521+0000","last_change":"2023-10-19T14:18:39.317469+0000","last_active":"2023-10-19T14:18:39.320521+0000","last_peered":"2023-10-19T14:18:39.320521+0000","last_clean":"2023-10-19T14:18:39.320521+0000","last_became_active":"2023-10-19T14:18:39.317185+0000","last_became_peered":"2023-10-19T14:18:39.317185+0000","last_unstale":"2023-10-19T14:18:39.320521+0000","last_undegraded":"2023-10-19T14:18:39.320521+0000","last_fullsized":"2023-10-19T14:18:39.320521+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:48:54.489239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.b","version":"204'2","reported_seq":13,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.478145+0000","last_change":"2023-10-19T14:18:38.486381+0000","last_active":"2023-10-19T14:18:39.478145+0000","last_peered":"2023-10-19T14:18:39.478145+0000","last_clean":"2023-10-19T14:18:39.478145+0000","last_became_active":"2023-10-19T14:18:38.486144+0000","last_became_peered":"2023-10-19T14:18:38.486144+0000","last_unstale":"2023-10-19T14:18:39.478145+0000","last_undegraded":"2023-10-19T14:18:39.478145+0000","last_fullsized":"2023-10-19T14:18:39.478145+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_clean_scrub_stamp":"2023-10-19T14:18:37.448968+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:40:20.360283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.317797+0000","last_change":"2023-10-19T14:18:39.317090+0000","last_active":"2023-10-19T14:18:39.317797+0000","last_peered":"2023-10-19T14:18:39.317797+0000","last_clean":"2023-10-19T14:18:39.317797+0000","last_became_active":"2023-10-19T14:18:39.316792+0000","last_became_peered":"2023-10-19T14:18:39.316792+0000","last_unstale":"2023-10-19T14:18:39.317797+0000","last_undegraded":"2023-10-19T14:18:39.317797+0000","last_fullsized":"2023-10-19T14:18:39.317797+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:52:07.280534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.a","version":"0'0","reported_seq":11,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.302667+0000","last_change":"2023-10-19T14:18:38.484108+0000","last_active":"2023-10-19T14:18:39.302667+0000","last_peered":"2023-10-19T14:18:39.302667+0000","last_clean":"2023-10-19T14:18:39.302667+0000","last_became_active":"2023-10-19T14:18:38.483844+0000","last_became_peered":"2023-10-19T14:18:38.483844+0000","last_unstale":"2023-10-19T14:18:39.302667+0000","last_undegraded":"2023-10-19T14:18:39.302667+0000","last_fullsized":"2023-10-19T14:18:39.302667+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_clean_scrub_stamp":"2023-10-19T14:18:37.448968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:36:38.326149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.312368+0000","last_change":"2023-10-19T14:18:39.309759+0000","last_active":"2023-10-19T14:18:39.312368+0000","last_peered":"2023-10-19T14:18:39.312368+0000","last_clean":"2023-10-19T14:18:39.312368+0000","last_became_active":"2023-10-19T14:18:39.307978+0000","last_became_peered":"2023-10-19T14:18:39.307978+0000","last_unstale":"2023-10-19T14:18:39.312368+0000","last_undegraded":"2023-10-19T14:18:39.312368+0000","last_fullsized":"2023-10-19T14:18:39.312368+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:58:36.439264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.9","version":"0'0","reported_seq":11,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.312051+0000","last_change":"2023-10-19T14:18:38.482203+0000","last_active":"2023-10-19T14:18:39.312051+0000","last_peered":"2023-10-19T14:18:39.312051+0000","last_clean":"2023-10-19T14:18:39.312051+0000","last_became_active":"2023-10-19T14:18:38.481856+0000","last_became_peered":"2023-10-19T14:18:38.481856+0000","last_unstale":"2023-10-19T14:18:39.312051+0000","last_undegraded":"2023-10-19T14:18:39.312051+0000","last_fullsized":"2023-10-19T14:18:39.312051+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_clean_scrub_stamp":"2023-10-19T14:18:37.448968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:42:05.821944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.b","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.320131+0000","last_change":"2023-10-19T14:18:39.316324+0000","last_active":"2023-10-19T14:18:39.320131+0000","last_peered":"2023-10-19T14:18:39.320131+0000","last_clean":"2023-10-19T14:18:39.320131+0000","last_became_active":"2023-10-19T14:18:39.315913+0000","last_became_peered":"2023-10-19T14:18:39.315913+0000","last_unstale":"2023-10-19T14:18:39.320131+0000","last_undegraded":"2023-10-19T14:18:39.320131+0000","last_fullsized":"2023-10-19T14:18:39.320131+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:17:32.830503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.8","version":"0'0","reported_seq":11,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.307796+0000","last_change":"2023-10-19T14:18:38.476672+0000","last_active":"2023-10-19T14:18:39.307796+0000","last_peered":"2023-10-19T14:18:39.307796+0000","last_clean":"2023-10-19T14:18:39.307796+0000","last_became_active":"2023-10-19T14:18:38.476410+0000","last_became_peered":"2023-10-19T14:18:38.476410+0000","last_unstale":"2023-10-19T14:18:39.307796+0000","last_undegraded":"2023-10-19T14:18:39.307796+0000","last_fullsized":"2023-10-19T14:18:39.307796+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_clean_scrub_stamp":"2023-10-19T14:18:37.448968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:28:00.507785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.315739+0000","last_change":"2023-10-19T14:18:39.315337+0000","last_active":"2023-10-19T14:18:39.315739+0000","last_peered":"2023-10-19T14:18:39.315739+0000","last_clean":"2023-10-19T14:18:39.315739+0000","last_became_active":"2023-10-19T14:18:39.315012+0000","last_became_peered":"2023-10-19T14:18:39.315012+0000","last_unstale":"2023-10-19T14:18:39.315739+0000","last_undegraded":"2023-10-19T14:18:39.315739+0000","last_fullsized":"2023-10-19T14:18:39.315739+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:29:51.042961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.7","version":"204'2","reported_seq":13,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.480716+0000","last_change":"2023-10-19T14:18:38.486645+0000","last_active":"2023-10-19T14:18:39.480716+0000","last_peered":"2023-10-19T14:18:39.480716+0000","last_clean":"2023-10-19T14:18:39.480716+0000","last_became_active":"2023-10-19T14:18:38.486374+0000","last_became_peered":"2023-10-19T14:18:38.486374+0000","last_unstale":"2023-10-19T14:18:39.480716+0000","last_undegraded":"2023-10-19T14:18:39.480716+0000","last_fullsized":"2023-10-19T14:18:39.480716+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_clean_scrub_stamp":"2023-10-19T14:18:37.448968+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:18:49.996920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.321001+0000","last_change":"2023-10-19T14:18:39.318905+0000","last_active":"2023-10-19T14:18:39.321001+0000","last_peered":"2023-10-19T14:18:39.321001+0000","last_clean":"2023-10-19T14:18:39.321001+0000","last_became_active":"2023-10-19T14:18:39.318288+0000","last_became_peered":"2023-10-19T14:18:39.318288+0000","last_unstale":"2023-10-19T14:18:39.321001+0000","last_undegraded":"2023-10-19T14:18:39.321001+0000","last_fullsized":"2023-10-19T14:18:39.321001+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:18:28.117122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.6","version":"0'0","reported_seq":11,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.312325+0000","last_change":"2023-10-19T14:18:38.482090+0000","last_active":"2023-10-19T14:18:39.312325+0000","last_peered":"2023-10-19T14:18:39.312325+0000","last_clean":"2023-10-19T14:18:39.312325+0000","last_became_active":"2023-10-19T14:18:38.481768+0000","last_became_peered":"2023-10-19T14:18:38.481768+0000","last_unstale":"2023-10-19T14:18:39.312325+0000","last_undegraded":"2023-10-19T14:18:39.312325+0000","last_fullsized":"2023-10-19T14:18:39.312325+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_clean_scrub_stamp":"2023-10-19T14:18:37.448968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:32:00.626249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.6","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.320691+0000","last_change":"2023-10-19T14:18:39.318510+0000","last_active":"2023-10-19T14:18:39.320691+0000","last_peered":"2023-10-19T14:18:39.320691+0000","last_clean":"2023-10-19T14:18:39.320691+0000","last_became_active":"2023-10-19T14:18:39.318181+0000","last_became_peered":"2023-10-19T14:18:39.318181+0000","last_unstale":"2023-10-19T14:18:39.320691+0000","last_undegraded":"2023-10-19T14:18:39.320691+0000","last_fullsized":"2023-10-19T14:18:39.320691+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T02:06:53.489258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.5","version":"0'0","reported_seq":11,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.306459+0000","last_change":"2023-10-19T14:18:38.475968+0000","last_active":"2023-10-19T14:18:39.306459+0000","last_peered":"2023-10-19T14:18:39.306459+0000","last_clean":"2023-10-19T14:18:39.306459+0000","last_became_active":"2023-10-19T14:18:38.472924+0000","last_became_peered":"2023-10-19T14:18:38.472924+0000","last_unstale":"2023-10-19T14:18:39.306459+0000","last_undegraded":"2023-10-19T14:18:39.306459+0000","last_fullsized":"2023-10-19T14:18:39.306459+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_clean_scrub_stamp":"2023-10-19T14:18:37.448968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:33:30.282629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.7","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.321005+0000","last_change":"2023-10-19T14:18:39.317307+0000","last_active":"2023-10-19T14:18:39.321005+0000","last_peered":"2023-10-19T14:18:39.321005+0000","last_clean":"2023-10-19T14:18:39.321005+0000","last_became_active":"2023-10-19T14:18:39.317017+0000","last_became_peered":"2023-10-19T14:18:39.317017+0000","last_unstale":"2023-10-19T14:18:39.321005+0000","last_undegraded":"2023-10-19T14:18:39.321005+0000","last_fullsized":"2023-10-19T14:18:39.321005+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:19:09.730366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.4","version":"0'0","reported_seq":11,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.310919+0000","last_change":"2023-10-19T14:18:38.478452+0000","last_active":"2023-10-19T14:18:39.310919+0000","last_peered":"2023-10-19T14:18:39.310919+0000","last_clean":"2023-10-19T14:18:39.310919+0000","last_became_active":"2023-10-19T14:18:38.473482+0000","last_became_peered":"2023-10-19T14:18:38.473482+0000","last_unstale":"2023-10-19T14:18:39.310919+0000","last_undegraded":"2023-10-19T14:18:39.310919+0000","last_fullsized":"2023-10-19T14:18:39.310919+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_clean_scrub_stamp":"2023-10-19T14:18:37.448968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:42:38.791498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.321083+0000","last_change":"2023-10-19T14:18:39.315487+0000","last_active":"2023-10-19T14:18:39.321083+0000","last_peered":"2023-10-19T14:18:39.321083+0000","last_clean":"2023-10-19T14:18:39.321083+0000","last_became_active":"2023-10-19T14:18:39.315195+0000","last_became_peered":"2023-10-19T14:18:39.315195+0000","last_unstale":"2023-10-19T14:18:39.321083+0000","last_undegraded":"2023-10-19T14:18:39.321083+0000","last_fullsized":"2023-10-19T14:18:39.321083+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:44:23.406435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3","version":"0'0","reported_seq":11,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.311946+0000","last_change":"2023-10-19T14:18:38.477660+0000","last_active":"2023-10-19T14:18:39.311946+0000","last_peered":"2023-10-19T14:18:39.311946+0000","last_clean":"2023-10-19T14:18:39.311946+0000","last_became_active":"2023-10-19T14:18:38.477401+0000","last_became_peered":"2023-10-19T14:18:38.477401+0000","last_unstale":"2023-10-19T14:18:39.311946+0000","last_undegraded":"2023-10-19T14:18:39.311946+0000","last_fullsized":"2023-10-19T14:18:39.311946+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_clean_scrub_stamp":"2023-10-19T14:18:37.448968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:49:09.667974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.326705+0000","last_change":"2023-10-19T14:18:39.322736+0000","last_active":"2023-10-19T14:18:39.326705+0000","last_peered":"2023-10-19T14:18:39.326705+0000","last_clean":"2023-10-19T14:18:39.326705+0000","last_became_active":"2023-10-19T14:18:39.322307+0000","last_became_peered":"2023-10-19T14:18:39.322307+0000","last_unstale":"2023-10-19T14:18:39.326705+0000","last_undegraded":"2023-10-19T14:18:39.326705+0000","last_fullsized":"2023-10-19T14:18:39.326705+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:41:26.208568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.2","version":"0'0","reported_seq":11,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.309923+0000","last_change":"2023-10-19T14:18:38.479907+0000","last_active":"2023-10-19T14:18:39.309923+0000","last_peered":"2023-10-19T14:18:39.309923+0000","last_clean":"2023-10-19T14:18:39.309923+0000","last_became_active":"2023-10-19T14:18:38.479210+0000","last_became_peered":"2023-10-19T14:18:38.479210+0000","last_unstale":"2023-10-19T14:18:39.309923+0000","last_undegraded":"2023-10-19T14:18:39.309923+0000","last_fullsized":"2023-10-19T14:18:39.309923+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_clean_scrub_stamp":"2023-10-19T14:18:37.448968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:21:14.725719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.321829+0000","last_change":"2023-10-19T14:18:39.319325+0000","last_active":"2023-10-19T14:18:39.321829+0000","last_peered":"2023-10-19T14:18:39.321829+0000","last_clean":"2023-10-19T14:18:39.321829+0000","last_became_active":"2023-10-19T14:18:39.317811+0000","last_became_peered":"2023-10-19T14:18:39.317811+0000","last_unstale":"2023-10-19T14:18:39.321829+0000","last_undegraded":"2023-10-19T14:18:39.321829+0000","last_fullsized":"2023-10-19T14:18:39.321829+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:03:00.677322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1","version":"0'0","reported_seq":11,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.312195+0000","last_change":"2023-10-19T14:18:38.480193+0000","last_active":"2023-10-19T14:18:39.312195+0000","last_peered":"2023-10-19T14:18:39.312195+0000","last_clean":"2023-10-19T14:18:39.312195+0000","last_became_active":"2023-10-19T14:18:38.479721+0000","last_became_peered":"2023-10-19T14:18:38.479721+0000","last_unstale":"2023-10-19T14:18:39.312195+0000","last_undegraded":"2023-10-19T14:18:39.312195+0000","last_fullsized":"2023-10-19T14:18:39.312195+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_clean_scrub_stamp":"2023-10-19T14:18:37.448968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:38:35.012627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.324428+0000","last_change":"2023-10-19T14:18:39.320923+0000","last_active":"2023-10-19T14:18:39.324428+0000","last_peered":"2023-10-19T14:18:39.324428+0000","last_clean":"2023-10-19T14:18:39.324428+0000","last_became_active":"2023-10-19T14:18:39.320490+0000","last_became_peered":"2023-10-19T14:18:39.320490+0000","last_unstale":"2023-10-19T14:18:39.324428+0000","last_undegraded":"2023-10-19T14:18:39.324428+0000","last_fullsized":"2023-10-19T14:18:39.324428+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:17:08.446391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.0","version":"0'0","reported_seq":11,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.307932+0000","last_change":"2023-10-19T14:18:38.475854+0000","last_active":"2023-10-19T14:18:39.307932+0000","last_peered":"2023-10-19T14:18:39.307932+0000","last_clean":"2023-10-19T14:18:39.307932+0000","last_became_active":"2023-10-19T14:18:38.475321+0000","last_became_peered":"2023-10-19T14:18:38.475321+0000","last_unstale":"2023-10-19T14:18:39.307932+0000","last_undegraded":"2023-10-19T14:18:39.307932+0000","last_fullsized":"2023-10-19T14:18:39.307932+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_clean_scrub_stamp":"2023-10-19T14:18:37.448968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:21:55.114037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.1b","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.320529+0000","last_change":"2023-10-19T14:18:39.313806+0000","last_active":"2023-10-19T14:18:39.320529+0000","last_peered":"2023-10-19T14:18:39.320529+0000","last_clean":"2023-10-19T14:18:39.320529+0000","last_became_active":"2023-10-19T14:18:39.313428+0000","last_became_peered":"2023-10-19T14:18:39.313428+0000","last_unstale":"2023-10-19T14:18:39.320529+0000","last_undegraded":"2023-10-19T14:18:39.320529+0000","last_fullsized":"2023-10-19T14:18:39.320529+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:42:20.542292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.18","version":"0'0","reported_seq":11,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.302416+0000","last_change":"2023-10-19T14:18:38.475260+0000","last_active":"2023-10-19T14:18:39.302416+0000","last_peered":"2023-10-19T14:18:39.302416+0000","last_clean":"2023-10-19T14:18:39.302416+0000","last_became_active":"2023-10-19T14:18:38.474945+0000","last_became_peered":"2023-10-19T14:18:38.474945+0000","last_unstale":"2023-10-19T14:18:39.302416+0000","last_undegraded":"2023-10-19T14:18:39.302416+0000","last_fullsized":"2023-10-19T14:18:39.302416+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_clean_scrub_stamp":"2023-10-19T14:18:37.448968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:49:09.667974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"193'3164","reported_seq":3990,"reported_epoch":203,"state":"active+clean","last_fresh":"2023-10-19T14:18:38.472236+0000","last_change":"2023-10-19T12:44:08.636613+0000","last_active":"2023-10-19T14:18:38.472236+0000","last_peered":"2023-10-19T14:18:38.472236+0000","last_clean":"2023-10-19T14:18:38.472236+0000","last_became_active":"2023-10-19T12:44:08.635697+0000","last_became_peered":"2023-10-19T12:44:08.635697+0000","last_unstale":"2023-10-19T14:18:38.472236+0000","last_undegraded":"2023-10-19T14:18:38.472236+0000","last_fullsized":"2023-10-19T14:18:38.472236+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_clean_scrub_stamp":"2023-10-19T12:44:07.399738+0000","objects_scrubbed":0,"log_size":3164,"log_dups_size":0,"ondisk_log_size":3164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:15:23.685485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2018,"num_read_kb":4212,"num_write":549,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":26,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":25,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2018,"num_read_kb":4212,"num_write":549,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3164,"ondisk_log_size":3164,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575982,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64792,"kb_used_data":3080,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94307048,"statfs":{"total":96636764160,"available":96570417152,"internally_reserved":0,"allocated":3153920,"data_stored":1897150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":63175115},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:18:05 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.59099999999999997,"15min":0.59399999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.32200000000000001},"max":{"1min":0.78800000000000003,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.622,"5min":0.61599999999999999,"15min":0.61799999999999999},"min":{"1min":0.505,"5min":0.39900000000000002,"15min":0.312},"max":{"1min":0.72399999999999998,"5min":0.81899999999999995,"15min":0.86599999999999999},"last":0.50600000000000001}]},{"osd":1,"last update":"Thu Oct 19 14:18:05 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.59999999999999998,"15min":0.60299999999999998},"min":{"1min":0.48699999999999999,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.60199999999999998,"15min":0.60299999999999998},"min":{"1min":0.40500000000000003,"5min":0.377,"15min":0.33400000000000002},"max":{"1min":0.76400000000000001,"5min":0.78600000000000003,"15min":0.88400000000000001},"last":0.745}]},{"osd":2,"last update":"Thu Oct 19 14:18:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.66900000000000004,"15min":0.65300000000000002},"min":{"1min":0.51900000000000002,"5min":0.42399999999999999,"15min":0.38400000000000001},"max":{"1min":0.76500000000000001,"5min":0.84699999999999998,"15min":0.85499999999999998},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.67400000000000004,"15min":0.66600000000000004},"min":{"1min":0.57599999999999996,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.75700000000000001,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.64300000000000002}]},{"osd":3,"last update":"Thu Oct 19 14:18:05 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.65400000000000003,"15min":0.64400000000000002},"min":{"1min":0.48699999999999999,"5min":0.47999999999999998,"15min":0.36399999999999999},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.873},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.68999999999999995,"15min":0.68799999999999994},"min":{"1min":0.54400000000000004,"5min":0.51700000000000002,"15min":0.47299999999999998},"max":{"1min":0.76700000000000002,"5min":0.876,"15min":0.89600000000000002},"last":0.78300000000000003}]},{"osd":4,"last update":"Thu Oct 19 14:18:05 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.48799999999999999,"15min":0.48399999999999999},"min":{"1min":0.33100000000000002,"5min":0.32900000000000001,"15min":0.32000000000000001},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.83799999999999997},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.501,"15min":0.496},"min":{"1min":0.36199999999999999,"5min":0.35299999999999998,"15min":0.33000000000000002},"max":{"1min":0.621,"5min":0.80200000000000005,"15min":0.81699999999999995},"last":0.53000000000000003}]},{"osd":5,"last update":"Thu Oct 19 14:18:05 2023","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.50700000000000001,"15min":0.498},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.32900000000000001},"max":{"1min":0.73699999999999999,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.53700000000000003,"15min":0.54100000000000004},"min":{"1min":0.377,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.71999999999999997,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.66600000000000004}]},{"osd":6,"last update":"Thu Oct 19 14:18:05 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.57599999999999996,"15min":0.56999999999999995},"min":{"1min":0.432,"5min":0.432,"15min":0.35999999999999999},"max":{"1min":0.76900000000000002,"5min":0.82999999999999996,"15min":0.85099999999999998},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.56699999999999995,"15min":0.56000000000000005},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.33500000000000002},"max":{"1min":0.80600000000000005,"5min":0.85199999999999998,"15min":0.85999999999999999},"last":0.71699999999999997}]}]},{"osd":6,"up_from":13,"seq":55834575982,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37848,"kb_used_data":2500,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333992,"statfs":{"total":96636764160,"available":96598007808,"internally_reserved":0,"allocated":2560000,"data_stored":1306782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:17:56 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.61299999999999999,"15min":0.61599999999999999},"min":{"1min":0.39200000000000002,"5min":0.35899999999999999,"15min":0.27100000000000002},"max":{"1min":0.746,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.60999999999999999,"15min":0.621},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.70099999999999996,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.627}]},{"osd":1,"last update":"Thu Oct 19 14:17:56 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.61299999999999999,"15min":0.622},"min":{"1min":0.41899999999999998,"5min":0.38,"15min":0.35799999999999998},"max":{"1min":0.77500000000000002,"5min":0.84699999999999998,"15min":1.165},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.629,"15min":0.63700000000000001},"min":{"1min":0.44900000000000001,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.81399999999999995,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.68700000000000006}]},{"osd":2,"last update":"Thu Oct 19 14:17:56 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.60299999999999998,"15min":0.61199999999999999},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.67300000000000004,"5min":0.83499999999999996,"15min":0.873},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.63,"15min":0.63500000000000001},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.80100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.64800000000000002}]},{"osd":3,"last update":"Thu Oct 19 14:17:56 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.64700000000000002,"15min":0.66300000000000003},"min":{"1min":0.372,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.76700000000000002,"5min":0.84399999999999997,"15min":0.85899999999999999},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.65300000000000002,"15min":0.66300000000000003},"min":{"1min":0.45800000000000002,"5min":0.44800000000000001,"15min":0.40100000000000002},"max":{"1min":0.76700000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.59499999999999997}]},{"osd":4,"last update":"Thu Oct 19 14:17:56 2023","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46200000000000002,"15min":0.47999999999999998},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.65800000000000003,"5min":0.77100000000000002,"15min":0.79600000000000004},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.48499999999999999,"15min":0.50800000000000001},"min":{"1min":0.34699999999999998,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.78800000000000003,"5min":0.85699999999999998,"15min":0.86499999999999999},"last":0.77100000000000002}]},{"osd":5,"last update":"Thu Oct 19 14:17:56 2023","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.47899999999999998,"15min":0.495},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.32500000000000001},"max":{"1min":0.66700000000000004,"5min":0.70099999999999996,"15min":0.77900000000000003},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52900000000000003,"15min":0.54000000000000004},"min":{"1min":0.35799999999999998,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.74199999999999999,"5min":0.878,"15min":0.878},"last":0.67700000000000005}]},{"osd":7,"last update":"Thu Oct 19 14:17:56 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.53200000000000003,"15min":0.54500000000000004},"min":{"1min":0.42099999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.90100000000000002},"last":0.749},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.54300000000000004,"15min":0.55100000000000005},"min":{"1min":0.41499999999999998,"5min":0.38200000000000001,"15min":0.36199999999999999},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.89500000000000002},"last":0.70499999999999996}]}]},{"osd":5,"up_from":13,"seq":55834575982,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47584,"kb_used_data":2516,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324256,"statfs":{"total":96636764160,"available":96588038144,"internally_reserved":0,"allocated":2576384,"data_stored":1308416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:17:45 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.61599999999999999,"15min":0.622},"min":{"1min":0.45900000000000002,"5min":0.41099999999999998,"15min":0.35399999999999998},"max":{"1min":0.77200000000000002,"5min":0.85199999999999998,"15min":0.86399999999999999},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.64500000000000002,"15min":0.64600000000000002},"min":{"1min":0.48899999999999999,"5min":0.45100000000000001,"15min":0.39000000000000001},"max":{"1min":0.84399999999999997,"5min":0.84499999999999997,"15min":0.85999999999999999},"last":0.55400000000000005}]},{"osd":1,"last update":"Thu Oct 19 14:17:45 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.61899999999999999,"15min":0.63600000000000001},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.80300000000000005,"5min":0.83699999999999997,"15min":0.877},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.625,"5min":0.65000000000000002,"15min":0.64600000000000002},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.41199999999999998},"max":{"1min":0.752,"5min":0.84399999999999997,"15min":0.88500000000000001},"last":0.60499999999999998}]},{"osd":2,"last update":"Thu Oct 19 14:17:45 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.62,"15min":0.628},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.83799999999999997,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.65700000000000003,"15min":0.66100000000000003},"min":{"1min":0.495,"5min":0.47299999999999998,"15min":0.46200000000000002},"max":{"1min":0.76100000000000001,"5min":0.81399999999999995,"15min":0.90800000000000003},"last":0.63400000000000001}]},{"osd":3,"last update":"Thu Oct 19 14:17:45 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.65300000000000002,"15min":0.64800000000000002},"min":{"1min":0.496,"5min":0.44800000000000001,"15min":0.44500000000000001},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.90100000000000002},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.57199999999999995,"5min":0.46100000000000002,"15min":0.42899999999999999},"max":{"1min":0.876,"5min":0.876,"15min":0.90900000000000003},"last":0.44500000000000001}]},{"osd":4,"last update":"Thu Oct 19 14:17:45 2023","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.48599999999999999,"15min":0.48499999999999999},"min":{"1min":0.36299999999999999,"5min":0.34699999999999998,"15min":0.32500000000000001},"max":{"1min":0.60599999999999998,"5min":0.69099999999999995,"15min":0.77500000000000002},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.496,"5min":0.49199999999999999,"15min":0.503},"min":{"1min":0.33500000000000002,"5min":0.29399999999999998,"15min":0.28599999999999998},"max":{"1min":0.71999999999999997,"5min":0.77000000000000002,"15min":0.82899999999999996},"last":0.64900000000000002}]},{"osd":6,"last update":"Thu Oct 19 14:17:45 2023","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.48899999999999999,"15min":0.49399999999999999},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.76500000000000001,"5min":0.84699999999999998,"15min":0.88800000000000001},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.502,"5min":0.53300000000000003,"15min":0.53800000000000003},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.625,"5min":0.86799999999999999,"15min":0.88100000000000001},"last":0.56999999999999995}]},{"osd":7,"last update":"Thu Oct 19 14:17:45 2023","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.50900000000000001,"15min":0.51900000000000002},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.64000000000000001,"5min":0.79500000000000004,"15min":0.82899999999999996},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.56599999999999995,"15min":0.56699999999999995},"min":{"1min":0.38400000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.752,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.61799999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575982,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64200,"kb_used_data":2488,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94307640,"statfs":{"total":96636764160,"available":96571023360,"internally_reserved":0,"allocated":2547712,"data_stored":1299659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":63175111},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:18:09 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.56299999999999994,"15min":0.57799999999999996},"min":{"1min":0.33100000000000002,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.78200000000000003,"5min":0.879,"15min":0.879},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.58599999999999997,"15min":0.59799999999999998},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.75700000000000001,"5min":0.86699999999999999,"15min":0.873},"last":0.69299999999999995}]},{"osd":1,"last update":"Thu Oct 19 14:18:09 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.629,"15min":0.64600000000000002},"min":{"1min":0.41299999999999998,"5min":0.372,"15min":0.32600000000000001},"max":{"1min":0.80400000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.63700000000000001,"15min":0.65000000000000002},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.82199999999999995,"5min":0.93400000000000005,"15min":1.0940000000000001},"last":0.63100000000000001}]},{"osd":2,"last update":"Thu Oct 19 14:18:09 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.63800000000000001,"15min":0.65200000000000002},"min":{"1min":0.46200000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.79600000000000004,"5min":0.82399999999999995,"15min":0.92800000000000005},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.66000000000000003,"15min":0.67400000000000004},"min":{"1min":0.48599999999999999,"5min":0.46200000000000002,"15min":0.38},"max":{"1min":0.80300000000000005,"5min":0.88600000000000001,"15min":1.008},"last":0.68300000000000005}]},{"osd":3,"last update":"Thu Oct 19 14:18:09 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.629,"15min":0.63900000000000001},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.81699999999999995,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.749},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.66500000000000004,"15min":0.67000000000000004},"min":{"1min":0.496,"5min":0.437,"15min":0.437},"max":{"1min":0.77500000000000002,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.67500000000000004}]},{"osd":5,"last update":"Thu Oct 19 14:18:09 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.48199999999999998,"15min":0.47299999999999998},"min":{"1min":0.33100000000000002,"5min":0.31,"15min":0.31},"max":{"1min":0.73299999999999998,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.5,"5min":0.501,"15min":0.48999999999999999},"min":{"1min":0.33700000000000002,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.66700000000000004,"5min":0.85599999999999998,"15min":0.86399999999999999},"last":0.60799999999999998}]},{"osd":6,"last update":"Thu Oct 19 14:18:09 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.49299999999999999,"15min":0.48899999999999999},"min":{"1min":0.36899999999999999,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.71699999999999997,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.54600000000000004,"15min":0.53300000000000003},"min":{"1min":0.38300000000000001,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.71399999999999997,"5min":0.84799999999999998,"15min":1.2969999999999999},"last":0.65400000000000003}]},{"osd":7,"last update":"Thu Oct 19 14:18:09 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.55000000000000004,"15min":0.53900000000000003},"min":{"1min":0.35699999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.65200000000000002,"5min":0.82299999999999995,"15min":0.95699999999999996},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.60299999999999998,"15min":0.59299999999999997},"min":{"1min":0.32100000000000001,"5min":0.318,"15min":0.318},"max":{"1min":0.72499999999999998,"5min":0.94499999999999995,"15min":1.032},"last":0.73999999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575981,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51744,"kb_used_data":2508,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320096,"statfs":{"total":96636764160,"available":96583778304,"internally_reserved":0,"allocated":2568192,"data_stored":1306942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50395592},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:17:42 2023","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.441,"15min":0.42199999999999999},"min":{"1min":0.29899999999999999,"5min":0.253,"15min":0.253},"max":{"1min":0.56899999999999995,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.435},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.44500000000000001,"15min":0.436},"min":{"1min":0.29499999999999998,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.58399999999999996,"5min":0.752,"15min":0.752},"last":0.28100000000000003}]},{"osd":1,"last update":"Thu Oct 19 14:17:42 2023","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.45700000000000002,"15min":0.44},"min":{"1min":0.28799999999999998,"5min":0.28000000000000003,"15min":0.27700000000000002},"max":{"1min":0.64700000000000002,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.504,"5min":0.497,"15min":0.47999999999999998},"min":{"1min":0.33100000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.73099999999999998,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.33900000000000002}]},{"osd":2,"last update":"Thu Oct 19 14:17:42 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.498,"15min":0.49099999999999999},"min":{"1min":0.375,"5min":0.34000000000000002,"15min":0.32800000000000001},"max":{"1min":0.64600000000000002,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.38},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.52600000000000002,"15min":0.51300000000000001},"min":{"1min":0.40100000000000002,"5min":0.36799999999999999,"15min":0.316},"max":{"1min":0.67000000000000004,"5min":0.77700000000000002,"15min":0.83699999999999997},"last":0.47199999999999998}]},{"osd":4,"last update":"Thu Oct 19 14:17:42 2023","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.73099999999999998,"15min":0.71099999999999997},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.018,"5min":1.157,"15min":1.157},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.755,"5min":0.75800000000000001,"15min":0.73599999999999999},"min":{"1min":0.55900000000000005,"5min":0.52800000000000002,"15min":0.50700000000000001},"max":{"1min":0.996,"5min":1.173,"15min":1.173},"last":0.65300000000000002}]},{"osd":5,"last update":"Thu Oct 19 14:17:42 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.70799999999999996,"15min":0.69599999999999995},"min":{"1min":0.56899999999999995,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.97299999999999998,"5min":0.997,"15min":1.036},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.71099999999999997,"15min":0.69299999999999995},"min":{"1min":0.57199999999999995,"5min":0.501,"15min":0.42599999999999999},"max":{"1min":0.873,"5min":0.98499999999999999,"15min":1.105},"last":0.54600000000000004}]},{"osd":6,"last update":"Thu Oct 19 14:17:42 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.71199999999999997,"15min":0.69899999999999995},"min":{"1min":0.54500000000000004,"5min":0.504,"15min":0.47099999999999997},"max":{"1min":0.88900000000000001,"5min":1.018,"15min":1.0669999999999999},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.73199999999999998,"15min":0.72099999999999997},"min":{"1min":0.61699999999999999,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":1.079},"last":0.60199999999999998}]},{"osd":7,"last update":"Thu Oct 19 14:17:42 2023","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.72199999999999998,"15min":0.70799999999999996},"min":{"1min":0.51700000000000002,"5min":0.499,"15min":0.45900000000000002},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":1.085},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.753,"5min":0.755,"15min":0.73899999999999999},"min":{"1min":0.60299999999999998,"5min":0.51900000000000002,"15min":0.42799999999999999},"max":{"1min":0.92900000000000005,"5min":0.98199999999999998,"15min":1.1739999999999999},"last":0.71799999999999997}]}]},{"osd":2,"up_from":13,"seq":55834575983,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60040,"kb_used_data":2488,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311800,"statfs":{"total":96636764160,"available":96575283200,"internally_reserved":0,"allocated":2547712,"data_stored":1299659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:18:34 2023","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.39600000000000002,"15min":0.40300000000000002},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.66600000000000004,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.443,"15min":0.439},"min":{"1min":0.249,"5min":0.249,"15min":0.223},"max":{"1min":0.53700000000000003,"5min":0.71399999999999997,"15min":0.73599999999999999},"last":0.53700000000000003}]},{"osd":1,"last update":"Thu Oct 19 14:18:34 2023","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.40500000000000003,"15min":0.41599999999999998},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.53500000000000003,"5min":0.61699999999999999,"15min":0.65400000000000003},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.46300000000000002,"15min":0.46600000000000003},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.59799999999999998,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":1.742}]},{"osd":3,"last update":"Thu Oct 19 14:18:34 2023","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.49299999999999999,"15min":0.48799999999999999},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.65900000000000003,"5min":0.69299999999999995,"15min":0.81499999999999995},"last":0.317},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.502,"15min":0.51300000000000001},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.315},"max":{"1min":0.65400000000000003,"5min":0.73499999999999999,"15min":0.83599999999999997},"last":0.49099999999999999}]},{"osd":4,"last update":"Thu Oct 19 14:18:34 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.66300000000000003,"15min":0.67200000000000004},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39500000000000002},"max":{"1min":0.89200000000000002,"5min":0.93899999999999995,"15min":0.998},"last":0.622},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.67800000000000005,"15min":0.68999999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.35199999999999998},"max":{"1min":0.91800000000000004,"5min":0.98799999999999999,"15min":1.083},"last":0.51700000000000002}]},{"osd":5,"last update":"Thu Oct 19 14:18:34 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.69999999999999996,"15min":0.70499999999999996},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.40600000000000003},"max":{"1min":0.93999999999999995,"5min":1.008,"15min":1.0289999999999999},"last":1.4330000000000001},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.67000000000000004,"15min":0.68999999999999995},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.39300000000000002},"max":{"1min":0.81499999999999995,"5min":1.036,"15min":1.04},"last":1.595}]},{"osd":6,"last update":"Thu Oct 19 14:18:34 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.70299999999999996,"15min":0.71699999999999997},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.92800000000000005,"5min":0.93200000000000005,"15min":1.0940000000000001},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.72699999999999998,"15min":0.72799999999999998},"min":{"1min":0.433,"5min":0.433,"15min":0.40600000000000003},"max":{"1min":0.92000000000000004,"5min":0.998,"15min":1.1140000000000001},"last":1.4930000000000001}]},{"osd":7,"last update":"Thu Oct 19 14:18:34 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.72499999999999998,"15min":0.73399999999999999},"min":{"1min":0.55600000000000005,"5min":0.51600000000000001,"15min":0.41799999999999998},"max":{"1min":0.95399999999999996,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.73299999999999998,"15min":0.73599999999999999},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.98099999999999998,"5min":1.054,"15min":1.101},"last":1.53}]}]},{"osd":1,"up_from":13,"seq":55834575982,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64212,"kb_used_data":2500,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94307628,"statfs":{"total":96636764160,"available":96571011072,"internally_reserved":0,"allocated":2560000,"data_stored":1306782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":63175110},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:18:33 2023","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.40799999999999997,"15min":0.40899999999999997},"min":{"1min":0.27900000000000003,"5min":0.26700000000000002,"15min":0.23100000000000001},"max":{"1min":0.61599999999999999,"5min":0.67400000000000004,"15min":0.67800000000000005},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.46800000000000003,"15min":0.46400000000000002},"min":{"1min":0.27500000000000002,"5min":0.26400000000000001,"15min":0.251},"max":{"1min":0.64200000000000002,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.32800000000000001}]},{"osd":2,"last update":"Thu Oct 19 14:18:33 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.45000000000000001,"15min":0.44700000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.27700000000000002},"max":{"1min":0.56299999999999994,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.498,"5min":0.502,"15min":0.49299999999999999},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.68700000000000006,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.432}]},{"osd":3,"last update":"Thu Oct 19 14:18:33 2023","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51800000000000002,"15min":0.50800000000000001},"min":{"1min":0.36499999999999999,"5min":0.35299999999999998,"15min":0.29499999999999998},"max":{"1min":0.74399999999999999,"5min":0.76000000000000001,"15min":0.78900000000000003},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.439,"5min":0.45200000000000001,"15min":0.45300000000000001},"min":{"1min":0.32600000000000001,"5min":0.314,"15min":0.26100000000000001},"max":{"1min":0.56100000000000005,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.40899999999999997}]},{"osd":4,"last update":"Thu Oct 19 14:18:33 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.68999999999999995,"15min":0.67800000000000005},"min":{"1min":0.45100000000000001,"5min":0.372,"15min":0.372},"max":{"1min":0.95999999999999996,"5min":1.042,"15min":1.042},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.621,"5min":0.66700000000000004,"15min":0.67200000000000004},"min":{"1min":0.46100000000000002,"5min":0.41299999999999998,"15min":0.32300000000000001},"max":{"1min":0.79800000000000004,"5min":0.93500000000000005,"15min":1.1659999999999999},"last":0.64500000000000002}]},{"osd":5,"last update":"Thu Oct 19 14:18:33 2023","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.69299999999999995,"15min":0.68999999999999995},"min":{"1min":0.47199999999999998,"5min":0.42799999999999999,"15min":0.42499999999999999},"max":{"1min":0.88400000000000001,"5min":1.0589999999999999,"15min":1.3500000000000001},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.72699999999999998,"15min":0.71599999999999997},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.40500000000000003},"max":{"1min":0.92800000000000005,"5min":1.0149999999999999,"15min":1.121},"last":0.67200000000000004}]},{"osd":6,"last update":"Thu Oct 19 14:18:33 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.68799999999999994,"15min":0.69099999999999995},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.2310000000000001},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.72699999999999998,"15min":0.72199999999999998},"min":{"1min":0.48199999999999998,"5min":0.441,"15min":0.441},"max":{"1min":0.88500000000000001,"5min":1.107,"15min":1.274},"last":0.52800000000000002}]},{"osd":7,"last update":"Thu Oct 19 14:18:33 2023","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.74099999999999999,"15min":0.72999999999999998},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.99099999999999999,"5min":1.075,"15min":1.075},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.71399999999999997,"15min":0.70699999999999996},"min":{"1min":0.498,"5min":0.46100000000000002,"15min":0.44400000000000001},"max":{"1min":0.97599999999999998,"5min":1.0169999999999999,"15min":1.117},"last":0.53500000000000003}]}]},{"osd":0,"up_from":13,"seq":55834575983,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56480,"kb_used_data":3092,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315360,"statfs":{"total":96636764160,"available":96578928640,"internally_reserved":0,"allocated":3166208,"data_stored":1897259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Oct 19 14:18:05 2023","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.41599999999999998,"15min":0.41499999999999998},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.253},"max":{"1min":0.59999999999999998,"5min":0.624,"15min":0.628},"last":0.374},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.46600000000000003,"15min":0.47199999999999998},"min":{"1min":0.32200000000000001,"5min":0.29799999999999999,"15min":0.27800000000000002},"max":{"1min":0.58899999999999997,"5min":0.70899999999999996,"15min":0.81100000000000005},"last":0.25700000000000001}]},{"osd":2,"last update":"Thu Oct 19 14:18:05 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.50900000000000001,"15min":0.48799999999999999},"min":{"1min":0.35399999999999998,"5min":0.316,"15min":0.29599999999999999},"max":{"1min":0.73099999999999998,"5min":0.754,"15min":0.81200000000000006},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.45500000000000002,"15min":0.45600000000000002},"min":{"1min":0.33000000000000002,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.59099999999999997,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.32800000000000001}]},{"osd":3,"last update":"Thu Oct 19 14:18:05 2023","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.45300000000000001,"15min":0.443},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.28999999999999998},"max":{"1min":0.54900000000000004,"5min":0.61099999999999999,"15min":0.66800000000000004},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52100000000000002,"15min":0.51700000000000002},"min":{"1min":0.376,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.78100000000000003,"5min":0.79800000000000004,"15min":0.84599999999999997},"last":0.39100000000000001}]},{"osd":4,"last update":"Thu Oct 19 14:18:05 2023","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.68500000000000005,"15min":0.68200000000000005},"min":{"1min":0.495,"5min":0.46200000000000002,"15min":0.44},"max":{"1min":0.85799999999999998,"5min":1.0069999999999999,"15min":1.1890000000000001},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.69299999999999995,"15min":0.69699999999999995},"min":{"1min":0.54200000000000004,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.93400000000000005,"5min":0.996,"15min":1.2150000000000001},"last":0.75900000000000001}]},{"osd":5,"last update":"Thu Oct 19 14:18:05 2023","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.73299999999999998,"15min":0.73299999999999998},"min":{"1min":0.54300000000000004,"5min":0.48399999999999999,"15min":0.45500000000000002},"max":{"1min":0.95499999999999996,"5min":1.0189999999999999,"15min":1.1579999999999999},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.72599999999999998,"15min":0.73299999999999998},"min":{"1min":0.46400000000000002,"5min":0.44,"15min":0.44},"max":{"1min":0.98299999999999998,"5min":1.038,"15min":1.2310000000000001},"last":0.68200000000000005}]},{"osd":6,"last update":"Thu Oct 19 14:18:05 2023","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.71299999999999997,"15min":0.71499999999999997},"min":{"1min":0.55000000000000004,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.91300000000000003,"5min":0.97699999999999998,"15min":1.1739999999999999},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.72299999999999998,"15min":0.71899999999999997},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46300000000000002},"max":{"1min":0.89900000000000002,"5min":1.01,"15min":1.0189999999999999},"last":0.59899999999999998}]},{"osd":7,"last update":"Thu Oct 19 14:18:05 2023","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.72399999999999998,"15min":0.72799999999999998},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.874,"5min":1.014,"15min":1.1180000000000001},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.72399999999999998,"15min":0.71599999999999997},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.45100000000000001},"max":{"1min":0.93799999999999994,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.72699999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-19T14:18:42.491 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-19T14:18:42.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.632+0000 7f203c2da640 1 -- 172.21.15.57:0/1243722872 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f20340cbd20 msgr2=0x7f2034158300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:42.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.632+0000 7f203c2da640 1 --2- 172.21.15.57:0/1243722872 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f20340cbd20 0x7f2034158300 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f203000a940 tx=0x7f203002d090 comp rx=0 tx=0).stop 2023-10-19T14:18:42.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.632+0000 7f203c2da640 1 -- 172.21.15.57:0/1243722872 shutdown_connections 2023-10-19T14:18:42.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.632+0000 7f203c2da640 1 --2- 172.21.15.57:0/1243722872 >> v2:172.21.15.179:3300/0 conn(0x7f20340afa20 0x7f20340705a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:42.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.632+0000 7f203c2da640 1 --2- 172.21.15.57:0/1243722872 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f20340cbd20 0x7f2034158300 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:42.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.632+0000 7f203c2da640 1 -- 172.21.15.57:0/1243722872 >> 172.21.15.57:0/1243722872 conn(0x7f2034069930 msgr2=0x7f2034069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:42.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.632+0000 7f203c2da640 1 -- 172.21.15.57:0/1243722872 shutdown_connections 2023-10-19T14:18:42.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.632+0000 7f203c2da640 1 -- 172.21.15.57:0/1243722872 wait complete. 2023-10-19T14:18:42.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.633+0000 7f203c2da640 1 Processor -- start 2023-10-19T14:18:42.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.633+0000 7f203c2da640 1 -- start start 2023-10-19T14:18:42.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.634+0000 7f203c2da640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f20340afa20 0x7f2034143910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:42.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.634+0000 7f203c2da640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f20340b0530 0x7f203414dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:42.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.634+0000 7f203c2da640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f20340cbd20 0x7f2034159b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:42.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.634+0000 7f203c2da640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2034147550 con 0x7f20340cbd20 2023-10-19T14:18:42.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.634+0000 7f203c2da640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2034147830 con 0x7f20340b0530 2023-10-19T14:18:42.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.634+0000 7f203c2da640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2034147b10 con 0x7f20340afa20 2023-10-19T14:18:42.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.634+0000 7f203a877640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f20340cbd20 0x7f2034159b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:42.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.634+0000 7f203a877640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f20340cbd20 0x7f2034159b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55574/0 (socket says 172.21.15.57:55574) 2023-10-19T14:18:42.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.634+0000 7f203a877640 1 -- 172.21.15.57:0/2425915243 learned_addr learned my addr 172.21.15.57:0/2425915243 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:18:42.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.634+0000 7f203a076640 1 --2- 172.21.15.57:0/2425915243 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f20340afa20 0x7f2034143910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:42.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.634+0000 7f2039875640 1 --2- 172.21.15.57:0/2425915243 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f20340b0530 0x7f203414dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:42.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.634+0000 7f203a877640 1 -- 172.21.15.57:0/2425915243 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f20340afa20 msgr2=0x7f2034143910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:42.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.635+0000 7f203a877640 1 --2- 172.21.15.57:0/2425915243 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f20340afa20 0x7f2034143910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:42.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.635+0000 7f203a877640 1 -- 172.21.15.57:0/2425915243 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f20340b0530 msgr2=0x7f203414dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:42.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.635+0000 7f203a877640 1 --2- 172.21.15.57:0/2425915243 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f20340b0530 0x7f203414dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:42.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.635+0000 7f203a877640 1 -- 172.21.15.57:0/2425915243 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2024009490 con 0x7f20340cbd20 2023-10-19T14:18:42.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.635+0000 7f2039875640 1 --2- 172.21.15.57:0/2425915243 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f20340b0530 0x7f203414dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T14:18:42.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.635+0000 7f203a877640 1 --2- 172.21.15.57:0/2425915243 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f20340cbd20 0x7f2034159b30 secure :-1 s=READY pgs=2206 cs=0 l=1 rev1=1 crypto rx=0x7f202c00ede0 tx=0x7f202c00c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:42.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.635+0000 7f202affd640 1 -- 172.21.15.57:0/2425915243 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f202c011670 con 0x7f20340cbd20 2023-10-19T14:18:42.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.635+0000 7f202affd640 1 -- 172.21.15.57:0/2425915243 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f202c0118c0 con 0x7f20340cbd20 2023-10-19T14:18:42.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.635+0000 7f202affd640 1 -- 172.21.15.57:0/2425915243 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f202c011cf0 con 0x7f20340cbd20 2023-10-19T14:18:42.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.635+0000 7f203c2da640 1 -- 172.21.15.57:0/2425915243 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f203000a5a0 con 0x7f20340cbd20 2023-10-19T14:18:42.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.635+0000 7f203c2da640 1 -- 172.21.15.57:0/2425915243 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2034070cf0 con 0x7f20340cbd20 2023-10-19T14:18:42.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.636+0000 7f203c2da640 1 -- 172.21.15.57:0/2425915243 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2034067930 con 0x7f20340cbd20 2023-10-19T14:18:42.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.636+0000 7f202affd640 1 -- 172.21.15.57:0/2425915243 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 117) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f202c0243f0 con 0x7f20340cbd20 2023-10-19T14:18:42.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.637+0000 7f202affd640 1 --2- 172.21.15.57:0/2425915243 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f201806af60 0x7f201806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:42.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.637+0000 7f202affd640 1 -- 172.21.15.57:0/2425915243 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f202c08c860 con 0x7f20340cbd20 2023-10-19T14:18:42.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.637+0000 7f203a076640 1 --2- 172.21.15.57:0/2425915243 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f201806af60 0x7f201806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:42.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.638+0000 7f203a076640 1 --2- 172.21.15.57:0/2425915243 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f201806af60 0x7f201806d420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2030037000 tx=0x7f2030004730 comp rx=0 tx=0).ready entity=mgr.8744 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:42.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.641+0000 7f202affd640 1 -- 172.21.15.57:0/2425915243 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f202c05b0c0 con 0x7f20340cbd20 2023-10-19T14:18:42.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.817+0000 7f203c2da640 1 -- 172.21.15.57:0/2425915243 --> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2034144640 con 0x7f201806af60 2023-10-19T14:18:42.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.825+0000 7f202affd640 1 -- 172.21.15.57:0/2425915243 <== mgr.8744 v2:172.21.15.179:6832/2730461479 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289958 (secure 0 0 0) 0x7f2034144640 con 0x7f201806af60 2023-10-19T14:18:42.831 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:18:42.832 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2023-10-19T14:18:42.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.828+0000 7f203c2da640 1 -- 172.21.15.57:0/2425915243 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f201806af60 msgr2=0x7f201806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:42.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.828+0000 7f203c2da640 1 --2- 172.21.15.57:0/2425915243 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f201806af60 0x7f201806d420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2030037000 tx=0x7f2030004730 comp rx=0 tx=0).stop 2023-10-19T14:18:42.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.829+0000 7f203c2da640 1 -- 172.21.15.57:0/2425915243 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f20340cbd20 msgr2=0x7f2034159b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:42.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.829+0000 7f203c2da640 1 --2- 172.21.15.57:0/2425915243 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f20340cbd20 0x7f2034159b30 secure :-1 s=READY pgs=2206 cs=0 l=1 rev1=1 crypto rx=0x7f202c00ede0 tx=0x7f202c00c640 comp rx=0 tx=0).stop 2023-10-19T14:18:42.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.829+0000 7f203c2da640 1 -- 172.21.15.57:0/2425915243 shutdown_connections 2023-10-19T14:18:42.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.829+0000 7f203c2da640 1 --2- 172.21.15.57:0/2425915243 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f201806af60 0x7f201806d420 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:42.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.829+0000 7f203c2da640 1 --2- 172.21.15.57:0/2425915243 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f20340cbd20 0x7f2034159b30 unknown :-1 s=CLOSED pgs=2206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:42.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.829+0000 7f203c2da640 1 --2- 172.21.15.57:0/2425915243 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f20340b0530 0x7f203414dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:42.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.829+0000 7f203c2da640 1 --2- 172.21.15.57:0/2425915243 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f20340afa20 0x7f2034143910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:42.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.829+0000 7f203c2da640 1 -- 172.21.15.57:0/2425915243 >> 172.21.15.57:0/2425915243 conn(0x7f2034069930 msgr2=0x7f20340ca870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:42.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.829+0000 7f203c2da640 1 -- 172.21.15.57:0/2425915243 shutdown_connections 2023-10-19T14:18:42.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.829+0000 7f203c2da640 1 -- 172.21.15.57:0/2425915243 wait complete. 2023-10-19T14:18:42.842 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-10-19T14:18:41.875178+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2018,"num_read_kb":4212,"num_write":593,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3202,"ondisk_log_size":3202,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":56,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":446900,"kb_used_data":21172,"kb_used_omap":12,"kb_used_meta":425587,"kb_avail":754527820,"statfs":{"total":773094113280,"available":772636487680,"internally_reserved":0,"allocated":21680128,"data_stored":11622649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":435801657},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.416866"},"pg_stats":[{"pgid":"26.3f","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.324443+0000","last_change":"2023-10-19T14:18:39.322803+0000","last_active":"2023-10-19T14:18:39.324443+0000","last_peered":"2023-10-19T14:18:39.324443+0000","last_clean":"2023-10-19T14:18:39.324443+0000","last_became_active":"2023-10-19T14:18:39.322536+0000","last_became_peered":"2023-10-19T14:18:39.322536+0000","last_unstale":"2023-10-19T14:18:39.324443+0000","last_undegraded":"2023-10-19T14:18:39.324443+0000","last_fullsized":"2023-10-19T14:18:39.324443+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:59:58.753855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.3e","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.319203+0000","last_change":"2023-10-19T14:18:39.315553+0000","last_active":"2023-10-19T14:18:39.319203+0000","last_peered":"2023-10-19T14:18:39.319203+0000","last_clean":"2023-10-19T14:18:39.319203+0000","last_became_active":"2023-10-19T14:18:39.315289+0000","last_became_peered":"2023-10-19T14:18:39.315289+0000","last_unstale":"2023-10-19T14:18:39.319203+0000","last_undegraded":"2023-10-19T14:18:39.319203+0000","last_fullsized":"2023-10-19T14:18:39.319203+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:12:51.586440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.3d","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.321113+0000","last_change":"2023-10-19T14:18:39.316231+0000","last_active":"2023-10-19T14:18:39.321113+0000","last_peered":"2023-10-19T14:18:39.321113+0000","last_clean":"2023-10-19T14:18:39.321113+0000","last_became_active":"2023-10-19T14:18:39.315932+0000","last_became_peered":"2023-10-19T14:18:39.315932+0000","last_unstale":"2023-10-19T14:18:39.321113+0000","last_undegraded":"2023-10-19T14:18:39.321113+0000","last_fullsized":"2023-10-19T14:18:39.321113+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:16:48.927842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.3c","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.324172+0000","last_change":"2023-10-19T14:18:39.321383+0000","last_active":"2023-10-19T14:18:39.324172+0000","last_peered":"2023-10-19T14:18:39.324172+0000","last_clean":"2023-10-19T14:18:39.324172+0000","last_became_active":"2023-10-19T14:18:39.321129+0000","last_became_peered":"2023-10-19T14:18:39.321129+0000","last_unstale":"2023-10-19T14:18:39.324172+0000","last_undegraded":"2023-10-19T14:18:39.324172+0000","last_fullsized":"2023-10-19T14:18:39.324172+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:52:51.790069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.3b","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.315083+0000","last_change":"2023-10-19T14:18:39.313841+0000","last_active":"2023-10-19T14:18:39.315083+0000","last_peered":"2023-10-19T14:18:39.315083+0000","last_clean":"2023-10-19T14:18:39.315083+0000","last_became_active":"2023-10-19T14:18:39.313566+0000","last_became_peered":"2023-10-19T14:18:39.313566+0000","last_unstale":"2023-10-19T14:18:39.315083+0000","last_undegraded":"2023-10-19T14:18:39.315083+0000","last_fullsized":"2023-10-19T14:18:39.315083+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:51:46.164589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.3a","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.321815+0000","last_change":"2023-10-19T14:18:39.316958+0000","last_active":"2023-10-19T14:18:39.321815+0000","last_peered":"2023-10-19T14:18:39.321815+0000","last_clean":"2023-10-19T14:18:39.321815+0000","last_became_active":"2023-10-19T14:18:39.316704+0000","last_became_peered":"2023-10-19T14:18:39.316704+0000","last_unstale":"2023-10-19T14:18:39.321815+0000","last_undegraded":"2023-10-19T14:18:39.321815+0000","last_fullsized":"2023-10-19T14:18:39.321815+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:48:12.573140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.39","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.323581+0000","last_change":"2023-10-19T14:18:39.318686+0000","last_active":"2023-10-19T14:18:39.323581+0000","last_peered":"2023-10-19T14:18:39.323581+0000","last_clean":"2023-10-19T14:18:39.323581+0000","last_became_active":"2023-10-19T14:18:39.318421+0000","last_became_peered":"2023-10-19T14:18:39.318421+0000","last_unstale":"2023-10-19T14:18:39.323581+0000","last_undegraded":"2023-10-19T14:18:39.323581+0000","last_fullsized":"2023-10-19T14:18:39.323581+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:33:02.890838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.38","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.324708+0000","last_change":"2023-10-19T14:18:39.320746+0000","last_active":"2023-10-19T14:18:39.324708+0000","last_peered":"2023-10-19T14:18:39.324708+0000","last_clean":"2023-10-19T14:18:39.324708+0000","last_became_active":"2023-10-19T14:18:39.320078+0000","last_became_peered":"2023-10-19T14:18:39.320078+0000","last_unstale":"2023-10-19T14:18:39.324708+0000","last_undegraded":"2023-10-19T14:18:39.324708+0000","last_fullsized":"2023-10-19T14:18:39.324708+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:24:12.867737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.37","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.322942+0000","last_change":"2023-10-19T14:18:39.321093+0000","last_active":"2023-10-19T14:18:39.322942+0000","last_peered":"2023-10-19T14:18:39.322942+0000","last_clean":"2023-10-19T14:18:39.322942+0000","last_became_active":"2023-10-19T14:18:39.317029+0000","last_became_peered":"2023-10-19T14:18:39.317029+0000","last_unstale":"2023-10-19T14:18:39.322942+0000","last_undegraded":"2023-10-19T14:18:39.322942+0000","last_fullsized":"2023-10-19T14:18:39.322942+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:52:37.893529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.36","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.323859+0000","last_change":"2023-10-19T14:18:39.323088+0000","last_active":"2023-10-19T14:18:39.323859+0000","last_peered":"2023-10-19T14:18:39.323859+0000","last_clean":"2023-10-19T14:18:39.323859+0000","last_became_active":"2023-10-19T14:18:39.322826+0000","last_became_peered":"2023-10-19T14:18:39.322826+0000","last_unstale":"2023-10-19T14:18:39.323859+0000","last_undegraded":"2023-10-19T14:18:39.323859+0000","last_fullsized":"2023-10-19T14:18:39.323859+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:41:18.044931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.35","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.320711+0000","last_change":"2023-10-19T14:18:39.318272+0000","last_active":"2023-10-19T14:18:39.320711+0000","last_peered":"2023-10-19T14:18:39.320711+0000","last_clean":"2023-10-19T14:18:39.320711+0000","last_became_active":"2023-10-19T14:18:39.317967+0000","last_became_peered":"2023-10-19T14:18:39.317967+0000","last_unstale":"2023-10-19T14:18:39.320711+0000","last_undegraded":"2023-10-19T14:18:39.320711+0000","last_fullsized":"2023-10-19T14:18:39.320711+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T02:02:07.160434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.34","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.312253+0000","last_change":"2023-10-19T14:18:39.309562+0000","last_active":"2023-10-19T14:18:39.312253+0000","last_peered":"2023-10-19T14:18:39.312253+0000","last_clean":"2023-10-19T14:18:39.312253+0000","last_became_active":"2023-10-19T14:18:39.309302+0000","last_became_peered":"2023-10-19T14:18:39.309302+0000","last_unstale":"2023-10-19T14:18:39.312253+0000","last_undegraded":"2023-10-19T14:18:39.312253+0000","last_fullsized":"2023-10-19T14:18:39.312253+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:36:22.594185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.33","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.326719+0000","last_change":"2023-10-19T14:18:39.322907+0000","last_active":"2023-10-19T14:18:39.326719+0000","last_peered":"2023-10-19T14:18:39.326719+0000","last_clean":"2023-10-19T14:18:39.326719+0000","last_became_active":"2023-10-19T14:18:39.322386+0000","last_became_peered":"2023-10-19T14:18:39.322386+0000","last_unstale":"2023-10-19T14:18:39.326719+0000","last_undegraded":"2023-10-19T14:18:39.326719+0000","last_fullsized":"2023-10-19T14:18:39.326719+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:44:23.406435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.32","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.323734+0000","last_change":"2023-10-19T14:18:39.320215+0000","last_active":"2023-10-19T14:18:39.323734+0000","last_peered":"2023-10-19T14:18:39.323734+0000","last_clean":"2023-10-19T14:18:39.323734+0000","last_became_active":"2023-10-19T14:18:39.319939+0000","last_became_peered":"2023-10-19T14:18:39.319939+0000","last_unstale":"2023-10-19T14:18:39.323734+0000","last_undegraded":"2023-10-19T14:18:39.323734+0000","last_fullsized":"2023-10-19T14:18:39.323734+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:16:35.591881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.31","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.320606+0000","last_change":"2023-10-19T14:18:39.318398+0000","last_active":"2023-10-19T14:18:39.320606+0000","last_peered":"2023-10-19T14:18:39.320606+0000","last_clean":"2023-10-19T14:18:39.320606+0000","last_became_active":"2023-10-19T14:18:39.318113+0000","last_became_peered":"2023-10-19T14:18:39.318113+0000","last_unstale":"2023-10-19T14:18:39.320606+0000","last_undegraded":"2023-10-19T14:18:39.320606+0000","last_fullsized":"2023-10-19T14:18:39.320606+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:28:53.413157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.30","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.315874+0000","last_change":"2023-10-19T14:18:39.314377+0000","last_active":"2023-10-19T14:18:39.315874+0000","last_peered":"2023-10-19T14:18:39.315874+0000","last_clean":"2023-10-19T14:18:39.315874+0000","last_became_active":"2023-10-19T14:18:39.314119+0000","last_became_peered":"2023-10-19T14:18:39.314119+0000","last_unstale":"2023-10-19T14:18:39.315874+0000","last_undegraded":"2023-10-19T14:18:39.315874+0000","last_fullsized":"2023-10-19T14:18:39.315874+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:20:04.481873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.2f","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.319792+0000","last_change":"2023-10-19T14:18:39.315448+0000","last_active":"2023-10-19T14:18:39.319792+0000","last_peered":"2023-10-19T14:18:39.319792+0000","last_clean":"2023-10-19T14:18:39.319792+0000","last_became_active":"2023-10-19T14:18:39.315108+0000","last_became_peered":"2023-10-19T14:18:39.315108+0000","last_unstale":"2023-10-19T14:18:39.319792+0000","last_undegraded":"2023-10-19T14:18:39.319792+0000","last_fullsized":"2023-10-19T14:18:39.319792+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:06:20.776277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.2e","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.326637+0000","last_change":"2023-10-19T14:18:39.322035+0000","last_active":"2023-10-19T14:18:39.326637+0000","last_peered":"2023-10-19T14:18:39.326637+0000","last_clean":"2023-10-19T14:18:39.326637+0000","last_became_active":"2023-10-19T14:18:39.321764+0000","last_became_peered":"2023-10-19T14:18:39.321764+0000","last_unstale":"2023-10-19T14:18:39.326637+0000","last_undegraded":"2023-10-19T14:18:39.326637+0000","last_fullsized":"2023-10-19T14:18:39.326637+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:05:13.696675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.2d","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.323140+0000","last_change":"2023-10-19T14:18:39.317826+0000","last_active":"2023-10-19T14:18:39.323140+0000","last_peered":"2023-10-19T14:18:39.323140+0000","last_clean":"2023-10-19T14:18:39.323140+0000","last_became_active":"2023-10-19T14:18:39.317325+0000","last_became_peered":"2023-10-19T14:18:39.317325+0000","last_unstale":"2023-10-19T14:18:39.323140+0000","last_undegraded":"2023-10-19T14:18:39.323140+0000","last_fullsized":"2023-10-19T14:18:39.323140+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:52:36.474116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.2c","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.325351+0000","last_change":"2023-10-19T14:18:39.324937+0000","last_active":"2023-10-19T14:18:39.325351+0000","last_peered":"2023-10-19T14:18:39.325351+0000","last_clean":"2023-10-19T14:18:39.325351+0000","last_became_active":"2023-10-19T14:18:39.324523+0000","last_became_peered":"2023-10-19T14:18:39.324523+0000","last_unstale":"2023-10-19T14:18:39.325351+0000","last_undegraded":"2023-10-19T14:18:39.325351+0000","last_fullsized":"2023-10-19T14:18:39.325351+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:09:16.621219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.2b","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.320947+0000","last_change":"2023-10-19T14:18:39.319111+0000","last_active":"2023-10-19T14:18:39.320947+0000","last_peered":"2023-10-19T14:18:39.320947+0000","last_clean":"2023-10-19T14:18:39.320947+0000","last_became_active":"2023-10-19T14:18:39.318343+0000","last_became_peered":"2023-10-19T14:18:39.318343+0000","last_unstale":"2023-10-19T14:18:39.320947+0000","last_undegraded":"2023-10-19T14:18:39.320947+0000","last_fullsized":"2023-10-19T14:18:39.320947+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:33:47.182338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.2a","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.320581+0000","last_change":"2023-10-19T14:18:39.317915+0000","last_active":"2023-10-19T14:18:39.320581+0000","last_peered":"2023-10-19T14:18:39.320581+0000","last_clean":"2023-10-19T14:18:39.320581+0000","last_became_active":"2023-10-19T14:18:39.317638+0000","last_became_peered":"2023-10-19T14:18:39.317638+0000","last_unstale":"2023-10-19T14:18:39.320581+0000","last_undegraded":"2023-10-19T14:18:39.320581+0000","last_fullsized":"2023-10-19T14:18:39.320581+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:31:02.673391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.29","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.324374+0000","last_change":"2023-10-19T14:18:39.321228+0000","last_active":"2023-10-19T14:18:39.324374+0000","last_peered":"2023-10-19T14:18:39.324374+0000","last_clean":"2023-10-19T14:18:39.324374+0000","last_became_active":"2023-10-19T14:18:39.320925+0000","last_became_peered":"2023-10-19T14:18:39.320925+0000","last_unstale":"2023-10-19T14:18:39.324374+0000","last_undegraded":"2023-10-19T14:18:39.324374+0000","last_fullsized":"2023-10-19T14:18:39.324374+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:58:28.744223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.28","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.324404+0000","last_change":"2023-10-19T14:18:39.323433+0000","last_active":"2023-10-19T14:18:39.324404+0000","last_peered":"2023-10-19T14:18:39.324404+0000","last_clean":"2023-10-19T14:18:39.324404+0000","last_became_active":"2023-10-19T14:18:39.323048+0000","last_became_peered":"2023-10-19T14:18:39.323048+0000","last_unstale":"2023-10-19T14:18:39.324404+0000","last_undegraded":"2023-10-19T14:18:39.324404+0000","last_fullsized":"2023-10-19T14:18:39.324404+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:26:44.530531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.27","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.323612+0000","last_change":"2023-10-19T14:18:39.320272+0000","last_active":"2023-10-19T14:18:39.323612+0000","last_peered":"2023-10-19T14:18:39.323612+0000","last_clean":"2023-10-19T14:18:39.323612+0000","last_became_active":"2023-10-19T14:18:39.319840+0000","last_became_peered":"2023-10-19T14:18:39.319840+0000","last_unstale":"2023-10-19T14:18:39.323612+0000","last_undegraded":"2023-10-19T14:18:39.323612+0000","last_fullsized":"2023-10-19T14:18:39.323612+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:46:10.846757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.26","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.315303+0000","last_change":"2023-10-19T14:18:39.314106+0000","last_active":"2023-10-19T14:18:39.315303+0000","last_peered":"2023-10-19T14:18:39.315303+0000","last_clean":"2023-10-19T14:18:39.315303+0000","last_became_active":"2023-10-19T14:18:39.313685+0000","last_became_peered":"2023-10-19T14:18:39.313685+0000","last_unstale":"2023-10-19T14:18:39.315303+0000","last_undegraded":"2023-10-19T14:18:39.315303+0000","last_fullsized":"2023-10-19T14:18:39.315303+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:23:30.741166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.25","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.320804+0000","last_change":"2023-10-19T14:18:39.317419+0000","last_active":"2023-10-19T14:18:39.320804+0000","last_peered":"2023-10-19T14:18:39.320804+0000","last_clean":"2023-10-19T14:18:39.320804+0000","last_became_active":"2023-10-19T14:18:39.317038+0000","last_became_peered":"2023-10-19T14:18:39.317038+0000","last_unstale":"2023-10-19T14:18:39.320804+0000","last_undegraded":"2023-10-19T14:18:39.320804+0000","last_fullsized":"2023-10-19T14:18:39.320804+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:02:47.020095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.24","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.323098+0000","last_change":"2023-10-19T14:18:39.321201+0000","last_active":"2023-10-19T14:18:39.323098+0000","last_peered":"2023-10-19T14:18:39.323098+0000","last_clean":"2023-10-19T14:18:39.323098+0000","last_became_active":"2023-10-19T14:18:39.319434+0000","last_became_peered":"2023-10-19T14:18:39.319434+0000","last_unstale":"2023-10-19T14:18:39.323098+0000","last_undegraded":"2023-10-19T14:18:39.323098+0000","last_fullsized":"2023-10-19T14:18:39.323098+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:57:26.768932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.23","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.315313+0000","last_change":"2023-10-19T14:18:39.312602+0000","last_active":"2023-10-19T14:18:39.315313+0000","last_peered":"2023-10-19T14:18:39.315313+0000","last_clean":"2023-10-19T14:18:39.315313+0000","last_became_active":"2023-10-19T14:18:39.312266+0000","last_became_peered":"2023-10-19T14:18:39.312266+0000","last_unstale":"2023-10-19T14:18:39.315313+0000","last_undegraded":"2023-10-19T14:18:39.315313+0000","last_fullsized":"2023-10-19T14:18:39.315313+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:32:11.986790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.22","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.315681+0000","last_change":"2023-10-19T14:18:39.311988+0000","last_active":"2023-10-19T14:18:39.315681+0000","last_peered":"2023-10-19T14:18:39.315681+0000","last_clean":"2023-10-19T14:18:39.315681+0000","last_became_active":"2023-10-19T14:18:39.311704+0000","last_became_peered":"2023-10-19T14:18:39.311704+0000","last_unstale":"2023-10-19T14:18:39.315681+0000","last_undegraded":"2023-10-19T14:18:39.315681+0000","last_fullsized":"2023-10-19T14:18:39.315681+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:34:55.930079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.21","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.314806+0000","last_change":"2023-10-19T14:18:39.311776+0000","last_active":"2023-10-19T14:18:39.314806+0000","last_peered":"2023-10-19T14:18:39.314806+0000","last_clean":"2023-10-19T14:18:39.314806+0000","last_became_active":"2023-10-19T14:18:39.311465+0000","last_became_peered":"2023-10-19T14:18:39.311465+0000","last_unstale":"2023-10-19T14:18:39.314806+0000","last_undegraded":"2023-10-19T14:18:39.314806+0000","last_fullsized":"2023-10-19T14:18:39.314806+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:30:13.687170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.20","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.324582+0000","last_change":"2023-10-19T14:18:39.323494+0000","last_active":"2023-10-19T14:18:39.324582+0000","last_peered":"2023-10-19T14:18:39.324582+0000","last_clean":"2023-10-19T14:18:39.324582+0000","last_became_active":"2023-10-19T14:18:39.321826+0000","last_became_peered":"2023-10-19T14:18:39.321826+0000","last_unstale":"2023-10-19T14:18:39.324582+0000","last_undegraded":"2023-10-19T14:18:39.324582+0000","last_fullsized":"2023-10-19T14:18:39.324582+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:48:53.137786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1c","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.324469+0000","last_change":"2023-10-19T14:18:39.321449+0000","last_active":"2023-10-19T14:18:39.324469+0000","last_peered":"2023-10-19T14:18:39.324469+0000","last_clean":"2023-10-19T14:18:39.324469+0000","last_became_active":"2023-10-19T14:18:39.321172+0000","last_became_peered":"2023-10-19T14:18:39.321172+0000","last_unstale":"2023-10-19T14:18:39.324469+0000","last_undegraded":"2023-10-19T14:18:39.324469+0000","last_fullsized":"2023-10-19T14:18:39.324469+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:36:46.457621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.1f","version":"204'4","reported_seq":15,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.481238+0000","last_change":"2023-10-19T14:18:38.485999+0000","last_active":"2023-10-19T14:18:39.481238+0000","last_peered":"2023-10-19T14:18:39.481238+0000","last_clean":"2023-10-19T14:18:39.481238+0000","last_became_active":"2023-10-19T14:18:38.485732+0000","last_became_peered":"2023-10-19T14:18:38.485732+0000","last_unstale":"2023-10-19T14:18:39.481238+0000","last_undegraded":"2023-10-19T14:18:39.481238+0000","last_fullsized":"2023-10-19T14:18:39.481238+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_clean_scrub_stamp":"2023-10-19T14:18:37.448968+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:09:08.257110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1d","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.324399+0000","last_change":"2023-10-19T14:18:39.321188+0000","last_active":"2023-10-19T14:18:39.324399+0000","last_peered":"2023-10-19T14:18:39.324399+0000","last_clean":"2023-10-19T14:18:39.324399+0000","last_became_active":"2023-10-19T14:18:39.320859+0000","last_became_peered":"2023-10-19T14:18:39.320859+0000","last_unstale":"2023-10-19T14:18:39.324399+0000","last_undegraded":"2023-10-19T14:18:39.324399+0000","last_fullsized":"2023-10-19T14:18:39.324399+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T02:14:35.102807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1e","version":"204'1","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.477262+0000","last_change":"2023-10-19T14:18:38.475466+0000","last_active":"2023-10-19T14:18:39.477262+0000","last_peered":"2023-10-19T14:18:39.477262+0000","last_clean":"2023-10-19T14:18:39.477262+0000","last_became_active":"2023-10-19T14:18:38.475088+0000","last_became_peered":"2023-10-19T14:18:38.475088+0000","last_unstale":"2023-10-19T14:18:39.477262+0000","last_undegraded":"2023-10-19T14:18:39.477262+0000","last_fullsized":"2023-10-19T14:18:39.477262+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_clean_scrub_stamp":"2023-10-19T14:18:37.448968+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:30:38.136460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1e","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.321890+0000","last_change":"2023-10-19T14:18:39.320557+0000","last_active":"2023-10-19T14:18:39.321890+0000","last_peered":"2023-10-19T14:18:39.321890+0000","last_clean":"2023-10-19T14:18:39.321890+0000","last_became_active":"2023-10-19T14:18:39.320236+0000","last_became_peered":"2023-10-19T14:18:39.320236+0000","last_unstale":"2023-10-19T14:18:39.321890+0000","last_undegraded":"2023-10-19T14:18:39.321890+0000","last_fullsized":"2023-10-19T14:18:39.321890+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:18:55.511235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.1d","version":"204'2","reported_seq":13,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.477241+0000","last_change":"2023-10-19T14:18:38.479859+0000","last_active":"2023-10-19T14:18:39.477241+0000","last_peered":"2023-10-19T14:18:39.477241+0000","last_clean":"2023-10-19T14:18:39.477241+0000","last_became_active":"2023-10-19T14:18:38.479322+0000","last_became_peered":"2023-10-19T14:18:38.479322+0000","last_unstale":"2023-10-19T14:18:39.477241+0000","last_undegraded":"2023-10-19T14:18:39.477241+0000","last_fullsized":"2023-10-19T14:18:39.477241+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_clean_scrub_stamp":"2023-10-19T14:18:37.448968+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:30:58.065702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1f","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.321482+0000","last_change":"2023-10-19T14:18:39.316917+0000","last_active":"2023-10-19T14:18:39.321482+0000","last_peered":"2023-10-19T14:18:39.321482+0000","last_clean":"2023-10-19T14:18:39.321482+0000","last_became_active":"2023-10-19T14:18:39.316620+0000","last_became_peered":"2023-10-19T14:18:39.316620+0000","last_unstale":"2023-10-19T14:18:39.321482+0000","last_undegraded":"2023-10-19T14:18:39.321482+0000","last_fullsized":"2023-10-19T14:18:39.321482+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:40:21.367319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1c","version":"204'2","reported_seq":13,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.481289+0000","last_change":"2023-10-19T14:18:38.485790+0000","last_active":"2023-10-19T14:18:39.481289+0000","last_peered":"2023-10-19T14:18:39.481289+0000","last_clean":"2023-10-19T14:18:39.481289+0000","last_became_active":"2023-10-19T14:18:38.485478+0000","last_became_peered":"2023-10-19T14:18:38.485478+0000","last_unstale":"2023-10-19T14:18:39.481289+0000","last_undegraded":"2023-10-19T14:18:39.481289+0000","last_fullsized":"2023-10-19T14:18:39.481289+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_clean_scrub_stamp":"2023-10-19T14:18:37.448968+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T02:04:51.639909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.319288+0000","last_change":"2023-10-19T14:18:39.318138+0000","last_active":"2023-10-19T14:18:39.319288+0000","last_peered":"2023-10-19T14:18:39.319288+0000","last_clean":"2023-10-19T14:18:39.319288+0000","last_became_active":"2023-10-19T14:18:39.317869+0000","last_became_peered":"2023-10-19T14:18:39.317869+0000","last_unstale":"2023-10-19T14:18:39.319288+0000","last_undegraded":"2023-10-19T14:18:39.319288+0000","last_fullsized":"2023-10-19T14:18:39.319288+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:10:10.162980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.1b","version":"204'1","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.476702+0000","last_change":"2023-10-19T14:18:38.474975+0000","last_active":"2023-10-19T14:18:39.476702+0000","last_peered":"2023-10-19T14:18:39.476702+0000","last_clean":"2023-10-19T14:18:39.476702+0000","last_became_active":"2023-10-19T14:18:38.473518+0000","last_became_peered":"2023-10-19T14:18:38.473518+0000","last_unstale":"2023-10-19T14:18:39.476702+0000","last_undegraded":"2023-10-19T14:18:39.476702+0000","last_fullsized":"2023-10-19T14:18:39.476702+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_clean_scrub_stamp":"2023-10-19T14:18:37.448968+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:42:15.965695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.312442+0000","last_change":"2023-10-19T14:18:39.311444+0000","last_active":"2023-10-19T14:18:39.312442+0000","last_peered":"2023-10-19T14:18:39.312442+0000","last_clean":"2023-10-19T14:18:39.312442+0000","last_became_active":"2023-10-19T14:18:39.311184+0000","last_became_peered":"2023-10-19T14:18:39.311184+0000","last_unstale":"2023-10-19T14:18:39.312442+0000","last_undegraded":"2023-10-19T14:18:39.312442+0000","last_fullsized":"2023-10-19T14:18:39.312442+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:21:35.591006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1a","version":"204'1","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.477439+0000","last_change":"2023-10-19T14:18:38.483165+0000","last_active":"2023-10-19T14:18:39.477439+0000","last_peered":"2023-10-19T14:18:39.477439+0000","last_clean":"2023-10-19T14:18:39.477439+0000","last_became_active":"2023-10-19T14:18:38.482417+0000","last_became_peered":"2023-10-19T14:18:38.482417+0000","last_unstale":"2023-10-19T14:18:39.477439+0000","last_undegraded":"2023-10-19T14:18:39.477439+0000","last_fullsized":"2023-10-19T14:18:39.477439+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_clean_scrub_stamp":"2023-10-19T14:18:37.448968+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:58:27.737188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1a","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.323767+0000","last_change":"2023-10-19T14:18:39.322322+0000","last_active":"2023-10-19T14:18:39.323767+0000","last_peered":"2023-10-19T14:18:39.323767+0000","last_clean":"2023-10-19T14:18:39.323767+0000","last_became_active":"2023-10-19T14:18:39.321983+0000","last_became_peered":"2023-10-19T14:18:39.321983+0000","last_unstale":"2023-10-19T14:18:39.323767+0000","last_undegraded":"2023-10-19T14:18:39.323767+0000","last_fullsized":"2023-10-19T14:18:39.323767+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:49:19.779416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.19","version":"0'0","reported_seq":11,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.310595+0000","last_change":"2023-10-19T14:18:38.476189+0000","last_active":"2023-10-19T14:18:39.310595+0000","last_peered":"2023-10-19T14:18:39.310595+0000","last_clean":"2023-10-19T14:18:39.310595+0000","last_became_active":"2023-10-19T14:18:38.475250+0000","last_became_peered":"2023-10-19T14:18:38.475250+0000","last_unstale":"2023-10-19T14:18:39.310595+0000","last_undegraded":"2023-10-19T14:18:39.310595+0000","last_fullsized":"2023-10-19T14:18:39.310595+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_clean_scrub_stamp":"2023-10-19T14:18:37.448968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:11:51.980596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.14","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.324487+0000","last_change":"2023-10-19T14:18:39.322238+0000","last_active":"2023-10-19T14:18:39.324487+0000","last_peered":"2023-10-19T14:18:39.324487+0000","last_clean":"2023-10-19T14:18:39.324487+0000","last_became_active":"2023-10-19T14:18:39.321870+0000","last_became_peered":"2023-10-19T14:18:39.321870+0000","last_unstale":"2023-10-19T14:18:39.324487+0000","last_undegraded":"2023-10-19T14:18:39.324487+0000","last_fullsized":"2023-10-19T14:18:39.324487+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:18:25.998173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.17","version":"0'0","reported_seq":11,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.305798+0000","last_change":"2023-10-19T14:18:38.473330+0000","last_active":"2023-10-19T14:18:39.305798+0000","last_peered":"2023-10-19T14:18:39.305798+0000","last_clean":"2023-10-19T14:18:39.305798+0000","last_became_active":"2023-10-19T14:18:38.472987+0000","last_became_peered":"2023-10-19T14:18:38.472987+0000","last_unstale":"2023-10-19T14:18:39.305798+0000","last_undegraded":"2023-10-19T14:18:39.305798+0000","last_fullsized":"2023-10-19T14:18:39.305798+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_clean_scrub_stamp":"2023-10-19T14:18:37.448968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:24:20.646403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.15","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.315033+0000","last_change":"2023-10-19T14:18:39.313590+0000","last_active":"2023-10-19T14:18:39.315033+0000","last_peered":"2023-10-19T14:18:39.315033+0000","last_clean":"2023-10-19T14:18:39.315033+0000","last_became_active":"2023-10-19T14:18:39.313328+0000","last_became_peered":"2023-10-19T14:18:39.313328+0000","last_unstale":"2023-10-19T14:18:39.315033+0000","last_undegraded":"2023-10-19T14:18:39.315033+0000","last_fullsized":"2023-10-19T14:18:39.315033+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T02:00:02.465134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.16","version":"0'0","reported_seq":11,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.309629+0000","last_change":"2023-10-19T14:18:38.482895+0000","last_active":"2023-10-19T14:18:39.309629+0000","last_peered":"2023-10-19T14:18:39.309629+0000","last_clean":"2023-10-19T14:18:39.309629+0000","last_became_active":"2023-10-19T14:18:38.482645+0000","last_became_peered":"2023-10-19T14:18:38.482645+0000","last_unstale":"2023-10-19T14:18:39.309629+0000","last_undegraded":"2023-10-19T14:18:39.309629+0000","last_fullsized":"2023-10-19T14:18:39.309629+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_clean_scrub_stamp":"2023-10-19T14:18:37.448968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:56:48.037930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.318883+0000","last_change":"2023-10-19T14:18:39.315849+0000","last_active":"2023-10-19T14:18:39.318883+0000","last_peered":"2023-10-19T14:18:39.318883+0000","last_clean":"2023-10-19T14:18:39.318883+0000","last_became_active":"2023-10-19T14:18:39.315597+0000","last_became_peered":"2023-10-19T14:18:39.315597+0000","last_unstale":"2023-10-19T14:18:39.318883+0000","last_undegraded":"2023-10-19T14:18:39.318883+0000","last_fullsized":"2023-10-19T14:18:39.318883+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:25:34.591207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.15","version":"204'3","reported_seq":14,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.480973+0000","last_change":"2023-10-19T14:18:38.482908+0000","last_active":"2023-10-19T14:18:39.480973+0000","last_peered":"2023-10-19T14:18:39.480973+0000","last_clean":"2023-10-19T14:18:39.480973+0000","last_became_active":"2023-10-19T14:18:38.482612+0000","last_became_peered":"2023-10-19T14:18:38.482612+0000","last_unstale":"2023-10-19T14:18:39.480973+0000","last_undegraded":"2023-10-19T14:18:39.480973+0000","last_fullsized":"2023-10-19T14:18:39.480973+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_clean_scrub_stamp":"2023-10-19T14:18:37.448968+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:35:21.292510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.320491+0000","last_change":"2023-10-19T14:18:39.316748+0000","last_active":"2023-10-19T14:18:39.320491+0000","last_peered":"2023-10-19T14:18:39.320491+0000","last_clean":"2023-10-19T14:18:39.320491+0000","last_became_active":"2023-10-19T14:18:39.316075+0000","last_became_peered":"2023-10-19T14:18:39.316075+0000","last_unstale":"2023-10-19T14:18:39.320491+0000","last_undegraded":"2023-10-19T14:18:39.320491+0000","last_fullsized":"2023-10-19T14:18:39.320491+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:16:45.929714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.14","version":"204'6","reported_seq":17,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.483016+0000","last_change":"2023-10-19T14:18:38.477382+0000","last_active":"2023-10-19T14:18:39.483016+0000","last_peered":"2023-10-19T14:18:39.483016+0000","last_clean":"2023-10-19T14:18:39.483016+0000","last_became_active":"2023-10-19T14:18:38.476733+0000","last_became_peered":"2023-10-19T14:18:38.476733+0000","last_unstale":"2023-10-19T14:18:39.483016+0000","last_undegraded":"2023-10-19T14:18:39.483016+0000","last_fullsized":"2023-10-19T14:18:39.483016+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_clean_scrub_stamp":"2023-10-19T14:18:37.448968+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:38:11.517639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.10","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.324576+0000","last_change":"2023-10-19T14:18:39.321629+0000","last_active":"2023-10-19T14:18:39.324576+0000","last_peered":"2023-10-19T14:18:39.324576+0000","last_clean":"2023-10-19T14:18:39.324576+0000","last_became_active":"2023-10-19T14:18:39.321356+0000","last_became_peered":"2023-10-19T14:18:39.321356+0000","last_unstale":"2023-10-19T14:18:39.324576+0000","last_undegraded":"2023-10-19T14:18:39.324576+0000","last_fullsized":"2023-10-19T14:18:39.324576+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:52:51.699459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.13","version":"204'2","reported_seq":13,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.481294+0000","last_change":"2023-10-19T14:18:38.475966+0000","last_active":"2023-10-19T14:18:39.481294+0000","last_peered":"2023-10-19T14:18:39.481294+0000","last_clean":"2023-10-19T14:18:39.481294+0000","last_became_active":"2023-10-19T14:18:38.475239+0000","last_became_peered":"2023-10-19T14:18:38.475239+0000","last_unstale":"2023-10-19T14:18:39.481294+0000","last_undegraded":"2023-10-19T14:18:39.481294+0000","last_fullsized":"2023-10-19T14:18:39.481294+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_clean_scrub_stamp":"2023-10-19T14:18:37.448968+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:16:47.920807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.320104+0000","last_change":"2023-10-19T14:18:39.316015+0000","last_active":"2023-10-19T14:18:39.320104+0000","last_peered":"2023-10-19T14:18:39.320104+0000","last_clean":"2023-10-19T14:18:39.320104+0000","last_became_active":"2023-10-19T14:18:39.315724+0000","last_became_peered":"2023-10-19T14:18:39.315724+0000","last_unstale":"2023-10-19T14:18:39.320104+0000","last_undegraded":"2023-10-19T14:18:39.320104+0000","last_fullsized":"2023-10-19T14:18:39.320104+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:51:15.427846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.12","version":"0'0","reported_seq":11,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.308847+0000","last_change":"2023-10-19T14:18:38.485409+0000","last_active":"2023-10-19T14:18:39.308847+0000","last_peered":"2023-10-19T14:18:39.308847+0000","last_clean":"2023-10-19T14:18:39.308847+0000","last_became_active":"2023-10-19T14:18:38.485133+0000","last_became_peered":"2023-10-19T14:18:38.485133+0000","last_unstale":"2023-10-19T14:18:39.308847+0000","last_undegraded":"2023-10-19T14:18:39.308847+0000","last_fullsized":"2023-10-19T14:18:39.308847+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_clean_scrub_stamp":"2023-10-19T14:18:37.448968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:50:41.815531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.320846+0000","last_change":"2023-10-19T14:18:39.319370+0000","last_active":"2023-10-19T14:18:39.320846+0000","last_peered":"2023-10-19T14:18:39.320846+0000","last_clean":"2023-10-19T14:18:39.320846+0000","last_became_active":"2023-10-19T14:18:39.318772+0000","last_became_peered":"2023-10-19T14:18:39.318772+0000","last_unstale":"2023-10-19T14:18:39.320846+0000","last_undegraded":"2023-10-19T14:18:39.320846+0000","last_fullsized":"2023-10-19T14:18:39.320846+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:18:25.389622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.11","version":"0'0","reported_seq":11,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.302590+0000","last_change":"2023-10-19T14:18:38.482814+0000","last_active":"2023-10-19T14:18:39.302590+0000","last_peered":"2023-10-19T14:18:39.302590+0000","last_clean":"2023-10-19T14:18:39.302590+0000","last_became_active":"2023-10-19T14:18:38.482538+0000","last_became_peered":"2023-10-19T14:18:38.482538+0000","last_unstale":"2023-10-19T14:18:39.302590+0000","last_undegraded":"2023-10-19T14:18:39.302590+0000","last_fullsized":"2023-10-19T14:18:39.302590+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_clean_scrub_stamp":"2023-10-19T14:18:37.448968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:47:17.141705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.13","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.326836+0000","last_change":"2023-10-19T14:18:39.323926+0000","last_active":"2023-10-19T14:18:39.326836+0000","last_peered":"2023-10-19T14:18:39.326836+0000","last_clean":"2023-10-19T14:18:39.326836+0000","last_became_active":"2023-10-19T14:18:39.323223+0000","last_became_peered":"2023-10-19T14:18:39.323223+0000","last_unstale":"2023-10-19T14:18:39.326836+0000","last_undegraded":"2023-10-19T14:18:39.326836+0000","last_fullsized":"2023-10-19T14:18:39.326836+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:25:56.932484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.10","version":"204'8","reported_seq":19,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.482607+0000","last_change":"2023-10-19T14:18:38.473672+0000","last_active":"2023-10-19T14:18:39.482607+0000","last_peered":"2023-10-19T14:18:39.482607+0000","last_clean":"2023-10-19T14:18:39.482607+0000","last_became_active":"2023-10-19T14:18:38.473346+0000","last_became_peered":"2023-10-19T14:18:38.473346+0000","last_unstale":"2023-10-19T14:18:39.482607+0000","last_undegraded":"2023-10-19T14:18:39.482607+0000","last_fullsized":"2023-10-19T14:18:39.482607+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_clean_scrub_stamp":"2023-10-19T14:18:37.448968+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:16:28.811592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.323885+0000","last_change":"2023-10-19T14:18:39.319381+0000","last_active":"2023-10-19T14:18:39.323885+0000","last_peered":"2023-10-19T14:18:39.323885+0000","last_clean":"2023-10-19T14:18:39.323885+0000","last_became_active":"2023-10-19T14:18:39.319091+0000","last_became_peered":"2023-10-19T14:18:39.319091+0000","last_unstale":"2023-10-19T14:18:39.323885+0000","last_undegraded":"2023-10-19T14:18:39.323885+0000","last_fullsized":"2023-10-19T14:18:39.323885+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:00:13.209777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.f","version":"204'1","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.481928+0000","last_change":"2023-10-19T14:18:38.473810+0000","last_active":"2023-10-19T14:18:39.481928+0000","last_peered":"2023-10-19T14:18:39.481928+0000","last_clean":"2023-10-19T14:18:39.481928+0000","last_became_active":"2023-10-19T14:18:38.473546+0000","last_became_peered":"2023-10-19T14:18:38.473546+0000","last_unstale":"2023-10-19T14:18:39.481928+0000","last_undegraded":"2023-10-19T14:18:39.481928+0000","last_fullsized":"2023-10-19T14:18:39.481928+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_clean_scrub_stamp":"2023-10-19T14:18:37.448968+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:02:52.762600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.d","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.322242+0000","last_change":"2023-10-19T14:18:39.321385+0000","last_active":"2023-10-19T14:18:39.322242+0000","last_peered":"2023-10-19T14:18:39.322242+0000","last_clean":"2023-10-19T14:18:39.322242+0000","last_became_active":"2023-10-19T14:18:39.319376+0000","last_became_peered":"2023-10-19T14:18:39.319376+0000","last_unstale":"2023-10-19T14:18:39.322242+0000","last_undegraded":"2023-10-19T14:18:39.322242+0000","last_fullsized":"2023-10-19T14:18:39.322242+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:41:05.496979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.e","version":"204'2","reported_seq":13,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.480708+0000","last_change":"2023-10-19T14:18:38.483495+0000","last_active":"2023-10-19T14:18:39.480708+0000","last_peered":"2023-10-19T14:18:39.480708+0000","last_clean":"2023-10-19T14:18:39.480708+0000","last_became_active":"2023-10-19T14:18:38.483195+0000","last_became_peered":"2023-10-19T14:18:38.483195+0000","last_unstale":"2023-10-19T14:18:39.480708+0000","last_undegraded":"2023-10-19T14:18:39.480708+0000","last_fullsized":"2023-10-19T14:18:39.480708+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_clean_scrub_stamp":"2023-10-19T14:18:37.448968+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:50:29.357721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.e","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.315282+0000","last_change":"2023-10-19T14:18:39.313942+0000","last_active":"2023-10-19T14:18:39.315282+0000","last_peered":"2023-10-19T14:18:39.315282+0000","last_clean":"2023-10-19T14:18:39.315282+0000","last_became_active":"2023-10-19T14:18:39.313320+0000","last_became_peered":"2023-10-19T14:18:39.313320+0000","last_unstale":"2023-10-19T14:18:39.315282+0000","last_undegraded":"2023-10-19T14:18:39.315282+0000","last_fullsized":"2023-10-19T14:18:39.315282+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:42:16.940277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.d","version":"204'1","reported_seq":12,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.477874+0000","last_change":"2023-10-19T14:18:38.479263+0000","last_active":"2023-10-19T14:18:39.477874+0000","last_peered":"2023-10-19T14:18:39.477874+0000","last_clean":"2023-10-19T14:18:39.477874+0000","last_became_active":"2023-10-19T14:18:38.478937+0000","last_became_peered":"2023-10-19T14:18:38.478937+0000","last_unstale":"2023-10-19T14:18:39.477874+0000","last_undegraded":"2023-10-19T14:18:39.477874+0000","last_fullsized":"2023-10-19T14:18:39.477874+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_clean_scrub_stamp":"2023-10-19T14:18:37.448968+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:42:50.809620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.f","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.320520+0000","last_change":"2023-10-19T14:18:39.317173+0000","last_active":"2023-10-19T14:18:39.320520+0000","last_peered":"2023-10-19T14:18:39.320520+0000","last_clean":"2023-10-19T14:18:39.320520+0000","last_became_active":"2023-10-19T14:18:39.316464+0000","last_became_peered":"2023-10-19T14:18:39.316464+0000","last_unstale":"2023-10-19T14:18:39.320520+0000","last_undegraded":"2023-10-19T14:18:39.320520+0000","last_fullsized":"2023-10-19T14:18:39.320520+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T02:16:33.786472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.c","version":"0'0","reported_seq":11,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.303947+0000","last_change":"2023-10-19T14:18:38.474681+0000","last_active":"2023-10-19T14:18:39.303947+0000","last_peered":"2023-10-19T14:18:39.303947+0000","last_clean":"2023-10-19T14:18:39.303947+0000","last_became_active":"2023-10-19T14:18:38.474417+0000","last_became_peered":"2023-10-19T14:18:38.474417+0000","last_unstale":"2023-10-19T14:18:39.303947+0000","last_undegraded":"2023-10-19T14:18:39.303947+0000","last_fullsized":"2023-10-19T14:18:39.303947+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_clean_scrub_stamp":"2023-10-19T14:18:37.448968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:34:33.625057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.320521+0000","last_change":"2023-10-19T14:18:39.317469+0000","last_active":"2023-10-19T14:18:39.320521+0000","last_peered":"2023-10-19T14:18:39.320521+0000","last_clean":"2023-10-19T14:18:39.320521+0000","last_became_active":"2023-10-19T14:18:39.317185+0000","last_became_peered":"2023-10-19T14:18:39.317185+0000","last_unstale":"2023-10-19T14:18:39.320521+0000","last_undegraded":"2023-10-19T14:18:39.320521+0000","last_fullsized":"2023-10-19T14:18:39.320521+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:48:54.489239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.b","version":"204'2","reported_seq":13,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.478145+0000","last_change":"2023-10-19T14:18:38.486381+0000","last_active":"2023-10-19T14:18:39.478145+0000","last_peered":"2023-10-19T14:18:39.478145+0000","last_clean":"2023-10-19T14:18:39.478145+0000","last_became_active":"2023-10-19T14:18:38.486144+0000","last_became_peered":"2023-10-19T14:18:38.486144+0000","last_unstale":"2023-10-19T14:18:39.478145+0000","last_undegraded":"2023-10-19T14:18:39.478145+0000","last_fullsized":"2023-10-19T14:18:39.478145+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_clean_scrub_stamp":"2023-10-19T14:18:37.448968+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:40:20.360283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.317797+0000","last_change":"2023-10-19T14:18:39.317090+0000","last_active":"2023-10-19T14:18:39.317797+0000","last_peered":"2023-10-19T14:18:39.317797+0000","last_clean":"2023-10-19T14:18:39.317797+0000","last_became_active":"2023-10-19T14:18:39.316792+0000","last_became_peered":"2023-10-19T14:18:39.316792+0000","last_unstale":"2023-10-19T14:18:39.317797+0000","last_undegraded":"2023-10-19T14:18:39.317797+0000","last_fullsized":"2023-10-19T14:18:39.317797+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:52:07.280534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.a","version":"0'0","reported_seq":11,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.302667+0000","last_change":"2023-10-19T14:18:38.484108+0000","last_active":"2023-10-19T14:18:39.302667+0000","last_peered":"2023-10-19T14:18:39.302667+0000","last_clean":"2023-10-19T14:18:39.302667+0000","last_became_active":"2023-10-19T14:18:38.483844+0000","last_became_peered":"2023-10-19T14:18:38.483844+0000","last_unstale":"2023-10-19T14:18:39.302667+0000","last_undegraded":"2023-10-19T14:18:39.302667+0000","last_fullsized":"2023-10-19T14:18:39.302667+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_clean_scrub_stamp":"2023-10-19T14:18:37.448968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:36:38.326149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.312368+0000","last_change":"2023-10-19T14:18:39.309759+0000","last_active":"2023-10-19T14:18:39.312368+0000","last_peered":"2023-10-19T14:18:39.312368+0000","last_clean":"2023-10-19T14:18:39.312368+0000","last_became_active":"2023-10-19T14:18:39.307978+0000","last_became_peered":"2023-10-19T14:18:39.307978+0000","last_unstale":"2023-10-19T14:18:39.312368+0000","last_undegraded":"2023-10-19T14:18:39.312368+0000","last_fullsized":"2023-10-19T14:18:39.312368+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:58:36.439264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.9","version":"0'0","reported_seq":11,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.312051+0000","last_change":"2023-10-19T14:18:38.482203+0000","last_active":"2023-10-19T14:18:39.312051+0000","last_peered":"2023-10-19T14:18:39.312051+0000","last_clean":"2023-10-19T14:18:39.312051+0000","last_became_active":"2023-10-19T14:18:38.481856+0000","last_became_peered":"2023-10-19T14:18:38.481856+0000","last_unstale":"2023-10-19T14:18:39.312051+0000","last_undegraded":"2023-10-19T14:18:39.312051+0000","last_fullsized":"2023-10-19T14:18:39.312051+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_clean_scrub_stamp":"2023-10-19T14:18:37.448968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:42:05.821944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.b","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.320131+0000","last_change":"2023-10-19T14:18:39.316324+0000","last_active":"2023-10-19T14:18:39.320131+0000","last_peered":"2023-10-19T14:18:39.320131+0000","last_clean":"2023-10-19T14:18:39.320131+0000","last_became_active":"2023-10-19T14:18:39.315913+0000","last_became_peered":"2023-10-19T14:18:39.315913+0000","last_unstale":"2023-10-19T14:18:39.320131+0000","last_undegraded":"2023-10-19T14:18:39.320131+0000","last_fullsized":"2023-10-19T14:18:39.320131+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:17:32.830503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.8","version":"0'0","reported_seq":11,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.307796+0000","last_change":"2023-10-19T14:18:38.476672+0000","last_active":"2023-10-19T14:18:39.307796+0000","last_peered":"2023-10-19T14:18:39.307796+0000","last_clean":"2023-10-19T14:18:39.307796+0000","last_became_active":"2023-10-19T14:18:38.476410+0000","last_became_peered":"2023-10-19T14:18:38.476410+0000","last_unstale":"2023-10-19T14:18:39.307796+0000","last_undegraded":"2023-10-19T14:18:39.307796+0000","last_fullsized":"2023-10-19T14:18:39.307796+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_clean_scrub_stamp":"2023-10-19T14:18:37.448968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:28:00.507785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.315739+0000","last_change":"2023-10-19T14:18:39.315337+0000","last_active":"2023-10-19T14:18:39.315739+0000","last_peered":"2023-10-19T14:18:39.315739+0000","last_clean":"2023-10-19T14:18:39.315739+0000","last_became_active":"2023-10-19T14:18:39.315012+0000","last_became_peered":"2023-10-19T14:18:39.315012+0000","last_unstale":"2023-10-19T14:18:39.315739+0000","last_undegraded":"2023-10-19T14:18:39.315739+0000","last_fullsized":"2023-10-19T14:18:39.315739+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:29:51.042961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.7","version":"204'2","reported_seq":13,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.480716+0000","last_change":"2023-10-19T14:18:38.486645+0000","last_active":"2023-10-19T14:18:39.480716+0000","last_peered":"2023-10-19T14:18:39.480716+0000","last_clean":"2023-10-19T14:18:39.480716+0000","last_became_active":"2023-10-19T14:18:38.486374+0000","last_became_peered":"2023-10-19T14:18:38.486374+0000","last_unstale":"2023-10-19T14:18:39.480716+0000","last_undegraded":"2023-10-19T14:18:39.480716+0000","last_fullsized":"2023-10-19T14:18:39.480716+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_clean_scrub_stamp":"2023-10-19T14:18:37.448968+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:18:49.996920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.321001+0000","last_change":"2023-10-19T14:18:39.318905+0000","last_active":"2023-10-19T14:18:39.321001+0000","last_peered":"2023-10-19T14:18:39.321001+0000","last_clean":"2023-10-19T14:18:39.321001+0000","last_became_active":"2023-10-19T14:18:39.318288+0000","last_became_peered":"2023-10-19T14:18:39.318288+0000","last_unstale":"2023-10-19T14:18:39.321001+0000","last_undegraded":"2023-10-19T14:18:39.321001+0000","last_fullsized":"2023-10-19T14:18:39.321001+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:18:28.117122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.6","version":"0'0","reported_seq":11,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.312325+0000","last_change":"2023-10-19T14:18:38.482090+0000","last_active":"2023-10-19T14:18:39.312325+0000","last_peered":"2023-10-19T14:18:39.312325+0000","last_clean":"2023-10-19T14:18:39.312325+0000","last_became_active":"2023-10-19T14:18:38.481768+0000","last_became_peered":"2023-10-19T14:18:38.481768+0000","last_unstale":"2023-10-19T14:18:39.312325+0000","last_undegraded":"2023-10-19T14:18:39.312325+0000","last_fullsized":"2023-10-19T14:18:39.312325+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_clean_scrub_stamp":"2023-10-19T14:18:37.448968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:32:00.626249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.6","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.320691+0000","last_change":"2023-10-19T14:18:39.318510+0000","last_active":"2023-10-19T14:18:39.320691+0000","last_peered":"2023-10-19T14:18:39.320691+0000","last_clean":"2023-10-19T14:18:39.320691+0000","last_became_active":"2023-10-19T14:18:39.318181+0000","last_became_peered":"2023-10-19T14:18:39.318181+0000","last_unstale":"2023-10-19T14:18:39.320691+0000","last_undegraded":"2023-10-19T14:18:39.320691+0000","last_fullsized":"2023-10-19T14:18:39.320691+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T02:06:53.489258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.5","version":"0'0","reported_seq":11,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.306459+0000","last_change":"2023-10-19T14:18:38.475968+0000","last_active":"2023-10-19T14:18:39.306459+0000","last_peered":"2023-10-19T14:18:39.306459+0000","last_clean":"2023-10-19T14:18:39.306459+0000","last_became_active":"2023-10-19T14:18:38.472924+0000","last_became_peered":"2023-10-19T14:18:38.472924+0000","last_unstale":"2023-10-19T14:18:39.306459+0000","last_undegraded":"2023-10-19T14:18:39.306459+0000","last_fullsized":"2023-10-19T14:18:39.306459+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_clean_scrub_stamp":"2023-10-19T14:18:37.448968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:33:30.282629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.7","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.321005+0000","last_change":"2023-10-19T14:18:39.317307+0000","last_active":"2023-10-19T14:18:39.321005+0000","last_peered":"2023-10-19T14:18:39.321005+0000","last_clean":"2023-10-19T14:18:39.321005+0000","last_became_active":"2023-10-19T14:18:39.317017+0000","last_became_peered":"2023-10-19T14:18:39.317017+0000","last_unstale":"2023-10-19T14:18:39.321005+0000","last_undegraded":"2023-10-19T14:18:39.321005+0000","last_fullsized":"2023-10-19T14:18:39.321005+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:19:09.730366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.4","version":"0'0","reported_seq":11,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.310919+0000","last_change":"2023-10-19T14:18:38.478452+0000","last_active":"2023-10-19T14:18:39.310919+0000","last_peered":"2023-10-19T14:18:39.310919+0000","last_clean":"2023-10-19T14:18:39.310919+0000","last_became_active":"2023-10-19T14:18:38.473482+0000","last_became_peered":"2023-10-19T14:18:38.473482+0000","last_unstale":"2023-10-19T14:18:39.310919+0000","last_undegraded":"2023-10-19T14:18:39.310919+0000","last_fullsized":"2023-10-19T14:18:39.310919+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_clean_scrub_stamp":"2023-10-19T14:18:37.448968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:42:38.791498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.321083+0000","last_change":"2023-10-19T14:18:39.315487+0000","last_active":"2023-10-19T14:18:39.321083+0000","last_peered":"2023-10-19T14:18:39.321083+0000","last_clean":"2023-10-19T14:18:39.321083+0000","last_became_active":"2023-10-19T14:18:39.315195+0000","last_became_peered":"2023-10-19T14:18:39.315195+0000","last_unstale":"2023-10-19T14:18:39.321083+0000","last_undegraded":"2023-10-19T14:18:39.321083+0000","last_fullsized":"2023-10-19T14:18:39.321083+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:44:23.406435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3","version":"0'0","reported_seq":11,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.311946+0000","last_change":"2023-10-19T14:18:38.477660+0000","last_active":"2023-10-19T14:18:39.311946+0000","last_peered":"2023-10-19T14:18:39.311946+0000","last_clean":"2023-10-19T14:18:39.311946+0000","last_became_active":"2023-10-19T14:18:38.477401+0000","last_became_peered":"2023-10-19T14:18:38.477401+0000","last_unstale":"2023-10-19T14:18:39.311946+0000","last_undegraded":"2023-10-19T14:18:39.311946+0000","last_fullsized":"2023-10-19T14:18:39.311946+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_clean_scrub_stamp":"2023-10-19T14:18:37.448968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:49:09.667974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.326705+0000","last_change":"2023-10-19T14:18:39.322736+0000","last_active":"2023-10-19T14:18:39.326705+0000","last_peered":"2023-10-19T14:18:39.326705+0000","last_clean":"2023-10-19T14:18:39.326705+0000","last_became_active":"2023-10-19T14:18:39.322307+0000","last_became_peered":"2023-10-19T14:18:39.322307+0000","last_unstale":"2023-10-19T14:18:39.326705+0000","last_undegraded":"2023-10-19T14:18:39.326705+0000","last_fullsized":"2023-10-19T14:18:39.326705+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:41:26.208568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.2","version":"0'0","reported_seq":11,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.309923+0000","last_change":"2023-10-19T14:18:38.479907+0000","last_active":"2023-10-19T14:18:39.309923+0000","last_peered":"2023-10-19T14:18:39.309923+0000","last_clean":"2023-10-19T14:18:39.309923+0000","last_became_active":"2023-10-19T14:18:38.479210+0000","last_became_peered":"2023-10-19T14:18:38.479210+0000","last_unstale":"2023-10-19T14:18:39.309923+0000","last_undegraded":"2023-10-19T14:18:39.309923+0000","last_fullsized":"2023-10-19T14:18:39.309923+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_clean_scrub_stamp":"2023-10-19T14:18:37.448968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:21:14.725719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.321829+0000","last_change":"2023-10-19T14:18:39.319325+0000","last_active":"2023-10-19T14:18:39.321829+0000","last_peered":"2023-10-19T14:18:39.321829+0000","last_clean":"2023-10-19T14:18:39.321829+0000","last_became_active":"2023-10-19T14:18:39.317811+0000","last_became_peered":"2023-10-19T14:18:39.317811+0000","last_unstale":"2023-10-19T14:18:39.321829+0000","last_undegraded":"2023-10-19T14:18:39.321829+0000","last_fullsized":"2023-10-19T14:18:39.321829+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:03:00.677322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1","version":"0'0","reported_seq":11,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.312195+0000","last_change":"2023-10-19T14:18:38.480193+0000","last_active":"2023-10-19T14:18:39.312195+0000","last_peered":"2023-10-19T14:18:39.312195+0000","last_clean":"2023-10-19T14:18:39.312195+0000","last_became_active":"2023-10-19T14:18:38.479721+0000","last_became_peered":"2023-10-19T14:18:38.479721+0000","last_unstale":"2023-10-19T14:18:39.312195+0000","last_undegraded":"2023-10-19T14:18:39.312195+0000","last_fullsized":"2023-10-19T14:18:39.312195+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_clean_scrub_stamp":"2023-10-19T14:18:37.448968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:38:35.012627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.324428+0000","last_change":"2023-10-19T14:18:39.320923+0000","last_active":"2023-10-19T14:18:39.324428+0000","last_peered":"2023-10-19T14:18:39.324428+0000","last_clean":"2023-10-19T14:18:39.324428+0000","last_became_active":"2023-10-19T14:18:39.320490+0000","last_became_peered":"2023-10-19T14:18:39.320490+0000","last_unstale":"2023-10-19T14:18:39.324428+0000","last_undegraded":"2023-10-19T14:18:39.324428+0000","last_fullsized":"2023-10-19T14:18:39.324428+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:17:08.446391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.0","version":"0'0","reported_seq":11,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.307932+0000","last_change":"2023-10-19T14:18:38.475854+0000","last_active":"2023-10-19T14:18:39.307932+0000","last_peered":"2023-10-19T14:18:39.307932+0000","last_clean":"2023-10-19T14:18:39.307932+0000","last_became_active":"2023-10-19T14:18:38.475321+0000","last_became_peered":"2023-10-19T14:18:38.475321+0000","last_unstale":"2023-10-19T14:18:39.307932+0000","last_undegraded":"2023-10-19T14:18:39.307932+0000","last_fullsized":"2023-10-19T14:18:39.307932+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_clean_scrub_stamp":"2023-10-19T14:18:37.448968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:21:55.114037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.1b","version":"0'0","reported_seq":9,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.320529+0000","last_change":"2023-10-19T14:18:39.313806+0000","last_active":"2023-10-19T14:18:39.320529+0000","last_peered":"2023-10-19T14:18:39.320529+0000","last_clean":"2023-10-19T14:18:39.320529+0000","last_became_active":"2023-10-19T14:18:39.313428+0000","last_became_peered":"2023-10-19T14:18:39.313428+0000","last_unstale":"2023-10-19T14:18:39.320529+0000","last_undegraded":"2023-10-19T14:18:39.320529+0000","last_fullsized":"2023-10-19T14:18:39.320529+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:38.456003+0000","last_clean_scrub_stamp":"2023-10-19T14:18:38.456003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:42:20.542292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.18","version":"0'0","reported_seq":11,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-10-19T14:18:39.302416+0000","last_change":"2023-10-19T14:18:38.475260+0000","last_active":"2023-10-19T14:18:39.302416+0000","last_peered":"2023-10-19T14:18:39.302416+0000","last_clean":"2023-10-19T14:18:39.302416+0000","last_became_active":"2023-10-19T14:18:38.474945+0000","last_became_peered":"2023-10-19T14:18:38.474945+0000","last_unstale":"2023-10-19T14:18:39.302416+0000","last_undegraded":"2023-10-19T14:18:39.302416+0000","last_fullsized":"2023-10-19T14:18:39.302416+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:18:37.448968+0000","last_clean_scrub_stamp":"2023-10-19T14:18:37.448968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:49:09.667974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"193'3164","reported_seq":3990,"reported_epoch":203,"state":"active+clean","last_fresh":"2023-10-19T14:18:38.472236+0000","last_change":"2023-10-19T12:44:08.636613+0000","last_active":"2023-10-19T14:18:38.472236+0000","last_peered":"2023-10-19T14:18:38.472236+0000","last_clean":"2023-10-19T14:18:38.472236+0000","last_became_active":"2023-10-19T12:44:08.635697+0000","last_became_peered":"2023-10-19T12:44:08.635697+0000","last_unstale":"2023-10-19T14:18:38.472236+0000","last_undegraded":"2023-10-19T14:18:38.472236+0000","last_fullsized":"2023-10-19T14:18:38.472236+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_clean_scrub_stamp":"2023-10-19T12:44:07.399738+0000","objects_scrubbed":0,"log_size":3164,"log_dups_size":0,"ondisk_log_size":3164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:15:23.685485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2018,"num_read_kb":4212,"num_write":549,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":26,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":25,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2018,"num_read_kb":4212,"num_write":549,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3164,"ondisk_log_size":3164,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575982,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64792,"kb_used_data":3080,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94307048,"statfs":{"total":96636764160,"available":96570417152,"internally_reserved":0,"allocated":3153920,"data_stored":1897150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":63175115},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:18:05 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.59099999999999997,"15min":0.59399999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.32200000000000001},"max":{"1min":0.78800000000000003,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.622,"5min":0.61599999999999999,"15min":0.61799999999999999},"min":{"1min":0.505,"5min":0.39900000000000002,"15min":0.312},"max":{"1min":0.72399999999999998,"5min":0.81899999999999995,"15min":0.86599999999999999},"last":0.50600000000000001}]},{"osd":1,"last update":"Thu Oct 19 14:18:05 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.59999999999999998,"15min":0.60299999999999998},"min":{"1min":0.48699999999999999,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.60199999999999998,"15min":0.60299999999999998},"min":{"1min":0.40500000000000003,"5min":0.377,"15min":0.33400000000000002},"max":{"1min":0.76400000000000001,"5min":0.78600000000000003,"15min":0.88400000000000001},"last":0.745}]},{"osd":2,"last update":"Thu Oct 19 14:18:05 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.66900000000000004,"15min":0.65300000000000002},"min":{"1min":0.51900000000000002,"5min":0.42399999999999999,"15min":0.38400000000000001},"max":{"1min":0.76500000000000001,"5min":0.84699999999999998,"15min":0.85499999999999998},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.67400000000000004,"15min":0.66600000000000004},"min":{"1min":0.57599999999999996,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.75700000000000001,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.64300000000000002}]},{"osd":3,"last update":"Thu Oct 19 14:18:05 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.65400000000000003,"15min":0.64400000000000002},"min":{"1min":0.48699999999999999,"5min":0.47999999999999998,"15min":0.36399999999999999},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.873},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.68999999999999995,"15min":0.68799999999999994},"min":{"1min":0.54400000000000004,"5min":0.51700000000000002,"15min":0.47299999999999998},"max":{"1min":0.76700000000000002,"5min":0.876,"15min":0.89600000000000002},"last":0.78300000000000003}]},{"osd":4,"last update":"Thu Oct 19 14:18:05 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.48799999999999999,"15min":0.48399999999999999},"min":{"1min":0.33100000000000002,"5min":0.32900000000000001,"15min":0.32000000000000001},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.83799999999999997},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.501,"15min":0.496},"min":{"1min":0.36199999999999999,"5min":0.35299999999999998,"15min":0.33000000000000002},"max":{"1min":0.621,"5min":0.80200000000000005,"15min":0.81699999999999995},"last":0.53000000000000003}]},{"osd":5,"last update":"Thu Oct 19 14:18:05 2023","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.50700000000000001,"15min":0.498},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.32900000000000001},"max":{"1min":0.73699999999999999,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.53700000000000003,"15min":0.54100000000000004},"min":{"1min":0.377,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.71999999999999997,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.66600000000000004}]},{"osd":6,"last update":"Thu Oct 19 14:18:05 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.57599999999999996,"15min":0.56999999999999995},"min":{"1min":0.432,"5min":0.432,"15min":0.35999999999999999},"max":{"1min":0.76900000000000002,"5min":0.82999999999999996,"15min":0.85099999999999998},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.56699999999999995,"15min":0.56000000000000005},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.33500000000000002},"max":{"1min":0.80600000000000005,"5min":0.85199999999999998,"15min":0.85999999999999999},"last":0.71699999999999997}]}]},{"osd":6,"up_from":13,"seq":55834575982,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37848,"kb_used_data":2500,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333992,"statfs":{"total":96636764160,"available":96598007808,"internally_reserved":0,"allocated":2560000,"data_stored":1306782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:17:56 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.61299999999999999,"15min":0.61599999999999999},"min":{"1min":0.39200000000000002,"5min":0.35899999999999999,"15min":0.27100000000000002},"max":{"1min":0.746,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.60999999999999999,"15min":0.621},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.70099999999999996,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.627}]},{"osd":1,"last update":"Thu Oct 19 14:17:56 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.61299999999999999,"15min":0.622},"min":{"1min":0.41899999999999998,"5min":0.38,"15min":0.35799999999999998},"max":{"1min":0.77500000000000002,"5min":0.84699999999999998,"15min":1.165},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.629,"15min":0.63700000000000001},"min":{"1min":0.44900000000000001,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.81399999999999995,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.68700000000000006}]},{"osd":2,"last update":"Thu Oct 19 14:17:56 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.60299999999999998,"15min":0.61199999999999999},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.67300000000000004,"5min":0.83499999999999996,"15min":0.873},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.63,"15min":0.63500000000000001},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.80100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.64800000000000002}]},{"osd":3,"last update":"Thu Oct 19 14:17:56 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.64700000000000002,"15min":0.66300000000000003},"min":{"1min":0.372,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.76700000000000002,"5min":0.84399999999999997,"15min":0.85899999999999999},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.65300000000000002,"15min":0.66300000000000003},"min":{"1min":0.45800000000000002,"5min":0.44800000000000001,"15min":0.40100000000000002},"max":{"1min":0.76700000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.59499999999999997}]},{"osd":4,"last update":"Thu Oct 19 14:17:56 2023","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46200000000000002,"15min":0.47999999999999998},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.65800000000000003,"5min":0.77100000000000002,"15min":0.79600000000000004},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.48499999999999999,"15min":0.50800000000000001},"min":{"1min":0.34699999999999998,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.78800000000000003,"5min":0.85699999999999998,"15min":0.86499999999999999},"last":0.77100000000000002}]},{"osd":5,"last update":"Thu Oct 19 14:17:56 2023","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.47899999999999998,"15min":0.495},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.32500000000000001},"max":{"1min":0.66700000000000004,"5min":0.70099999999999996,"15min":0.77900000000000003},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52900000000000003,"15min":0.54000000000000004},"min":{"1min":0.35799999999999998,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.74199999999999999,"5min":0.878,"15min":0.878},"last":0.67700000000000005}]},{"osd":7,"last update":"Thu Oct 19 14:17:56 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.53200000000000003,"15min":0.54500000000000004},"min":{"1min":0.42099999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.90100000000000002},"last":0.749},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.54300000000000004,"15min":0.55100000000000005},"min":{"1min":0.41499999999999998,"5min":0.38200000000000001,"15min":0.36199999999999999},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.89500000000000002},"last":0.70499999999999996}]}]},{"osd":5,"up_from":13,"seq":55834575982,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47584,"kb_used_data":2516,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324256,"statfs":{"total":96636764160,"available":96588038144,"internally_reserved":0,"allocated":2576384,"data_stored":1308416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:17:45 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.61599999999999999,"15min":0.622},"min":{"1min":0.45900000000000002,"5min":0.41099999999999998,"15min":0.35399999999999998},"max":{"1min":0.77200000000000002,"5min":0.85199999999999998,"15min":0.86399999999999999},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.64500000000000002,"15min":0.64600000000000002},"min":{"1min":0.48899999999999999,"5min":0.45100000000000001,"15min":0.39000000000000001},"max":{"1min":0.84399999999999997,"5min":0.84499999999999997,"15min":0.85999999999999999},"last":0.55400000000000005}]},{"osd":1,"last update":"Thu Oct 19 14:17:45 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.61899999999999999,"15min":0.63600000000000001},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.80300000000000005,"5min":0.83699999999999997,"15min":0.877},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.625,"5min":0.65000000000000002,"15min":0.64600000000000002},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.41199999999999998},"max":{"1min":0.752,"5min":0.84399999999999997,"15min":0.88500000000000001},"last":0.60499999999999998}]},{"osd":2,"last update":"Thu Oct 19 14:17:45 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.62,"15min":0.628},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.83799999999999997,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.65700000000000003,"15min":0.66100000000000003},"min":{"1min":0.495,"5min":0.47299999999999998,"15min":0.46200000000000002},"max":{"1min":0.76100000000000001,"5min":0.81399999999999995,"15min":0.90800000000000003},"last":0.63400000000000001}]},{"osd":3,"last update":"Thu Oct 19 14:17:45 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.65300000000000002,"15min":0.64800000000000002},"min":{"1min":0.496,"5min":0.44800000000000001,"15min":0.44500000000000001},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.90100000000000002},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.57199999999999995,"5min":0.46100000000000002,"15min":0.42899999999999999},"max":{"1min":0.876,"5min":0.876,"15min":0.90900000000000003},"last":0.44500000000000001}]},{"osd":4,"last update":"Thu Oct 19 14:17:45 2023","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.48599999999999999,"15min":0.48499999999999999},"min":{"1min":0.36299999999999999,"5min":0.34699999999999998,"15min":0.32500000000000001},"max":{"1min":0.60599999999999998,"5min":0.69099999999999995,"15min":0.77500000000000002},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.496,"5min":0.49199999999999999,"15min":0.503},"min":{"1min":0.33500000000000002,"5min":0.29399999999999998,"15min":0.28599999999999998},"max":{"1min":0.71999999999999997,"5min":0.77000000000000002,"15min":0.82899999999999996},"last":0.64900000000000002}]},{"osd":6,"last update":"Thu Oct 19 14:17:45 2023","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.48899999999999999,"15min":0.49399999999999999},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.76500000000000001,"5min":0.84699999999999998,"15min":0.88800000000000001},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.502,"5min":0.53300000000000003,"15min":0.53800000000000003},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.625,"5min":0.86799999999999999,"15min":0.88100000000000001},"last":0.56999999999999995}]},{"osd":7,"last update":"Thu Oct 19 14:17:45 2023","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.50900000000000001,"15min":0.51900000000000002},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.64000000000000001,"5min":0.79500000000000004,"15min":0.82899999999999996},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.56599999999999995,"15min":0.56699999999999995},"min":{"1min":0.38400000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.752,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.61799999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575982,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64200,"kb_used_data":2488,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94307640,"statfs":{"total":96636764160,"available":96571023360,"internally_reserved":0,"allocated":2547712,"data_stored":1299659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":63175111},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:18:09 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.56299999999999994,"15min":0.57799999999999996},"min":{"1min":0.33100000000000002,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.78200000000000003,"5min":0.879,"15min":0.879},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.58599999999999997,"15min":0.59799999999999998},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.75700000000000001,"5min":0.86699999999999999,"15min":0.873},"last":0.69299999999999995}]},{"osd":1,"last update":"Thu Oct 19 14:18:09 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.629,"15min":0.64600000000000002},"min":{"1min":0.41299999999999998,"5min":0.372,"15min":0.32600000000000001},"max":{"1min":0.80400000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.63700000000000001,"15min":0.65000000000000002},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.82199999999999995,"5min":0.93400000000000005,"15min":1.0940000000000001},"last":0.63100000000000001}]},{"osd":2,"last update":"Thu Oct 19 14:18:09 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.63800000000000001,"15min":0.65200000000000002},"min":{"1min":0.46200000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.79600000000000004,"5min":0.82399999999999995,"15min":0.92800000000000005},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.66000000000000003,"15min":0.67400000000000004},"min":{"1min":0.48599999999999999,"5min":0.46200000000000002,"15min":0.38},"max":{"1min":0.80300000000000005,"5min":0.88600000000000001,"15min":1.008},"last":0.68300000000000005}]},{"osd":3,"last update":"Thu Oct 19 14:18:09 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.629,"15min":0.63900000000000001},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.81699999999999995,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.749},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.66500000000000004,"15min":0.67000000000000004},"min":{"1min":0.496,"5min":0.437,"15min":0.437},"max":{"1min":0.77500000000000002,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.67500000000000004}]},{"osd":5,"last update":"Thu Oct 19 14:18:09 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.48199999999999998,"15min":0.47299999999999998},"min":{"1min":0.33100000000000002,"5min":0.31,"15min":0.31},"max":{"1min":0.73299999999999998,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.5,"5min":0.501,"15min":0.48999999999999999},"min":{"1min":0.33700000000000002,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.66700000000000004,"5min":0.85599999999999998,"15min":0.86399999999999999},"last":0.60799999999999998}]},{"osd":6,"last update":"Thu Oct 19 14:18:09 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.49299999999999999,"15min":0.48899999999999999},"min":{"1min":0.36899999999999999,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.71699999999999997,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.54600000000000004,"15min":0.53300000000000003},"min":{"1min":0.38300000000000001,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.71399999999999997,"5min":0.84799999999999998,"15min":1.2969999999999999},"last":0.65400000000000003}]},{"osd":7,"last update":"Thu Oct 19 14:18:09 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.55000000000000004,"15min":0.53900000000000003},"min":{"1min":0.35699999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.65200000000000002,"5min":0.82299999999999995,"15min":0.95699999999999996},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.60299999999999998,"15min":0.59299999999999997},"min":{"1min":0.32100000000000001,"5min":0.318,"15min":0.318},"max":{"1min":0.72499999999999998,"5min":0.94499999999999995,"15min":1.032},"last":0.73999999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575981,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51744,"kb_used_data":2508,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320096,"statfs":{"total":96636764160,"available":96583778304,"internally_reserved":0,"allocated":2568192,"data_stored":1306942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50395592},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:17:42 2023","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.441,"15min":0.42199999999999999},"min":{"1min":0.29899999999999999,"5min":0.253,"15min":0.253},"max":{"1min":0.56899999999999995,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.435},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.44500000000000001,"15min":0.436},"min":{"1min":0.29499999999999998,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.58399999999999996,"5min":0.752,"15min":0.752},"last":0.28100000000000003}]},{"osd":1,"last update":"Thu Oct 19 14:17:42 2023","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.45700000000000002,"15min":0.44},"min":{"1min":0.28799999999999998,"5min":0.28000000000000003,"15min":0.27700000000000002},"max":{"1min":0.64700000000000002,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.504,"5min":0.497,"15min":0.47999999999999998},"min":{"1min":0.33100000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.73099999999999998,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.33900000000000002}]},{"osd":2,"last update":"Thu Oct 19 14:17:42 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.498,"15min":0.49099999999999999},"min":{"1min":0.375,"5min":0.34000000000000002,"15min":0.32800000000000001},"max":{"1min":0.64600000000000002,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.38},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.52600000000000002,"15min":0.51300000000000001},"min":{"1min":0.40100000000000002,"5min":0.36799999999999999,"15min":0.316},"max":{"1min":0.67000000000000004,"5min":0.77700000000000002,"15min":0.83699999999999997},"last":0.47199999999999998}]},{"osd":4,"last update":"Thu Oct 19 14:17:42 2023","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.73099999999999998,"15min":0.71099999999999997},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.018,"5min":1.157,"15min":1.157},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.755,"5min":0.75800000000000001,"15min":0.73599999999999999},"min":{"1min":0.55900000000000005,"5min":0.52800000000000002,"15min":0.50700000000000001},"max":{"1min":0.996,"5min":1.173,"15min":1.173},"last":0.65300000000000002}]},{"osd":5,"last update":"Thu Oct 19 14:17:42 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.70799999999999996,"15min":0.69599999999999995},"min":{"1min":0.56899999999999995,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.97299999999999998,"5min":0.997,"15min":1.036},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.71099999999999997,"15min":0.69299999999999995},"min":{"1min":0.57199999999999995,"5min":0.501,"15min":0.42599999999999999},"max":{"1min":0.873,"5min":0.98499999999999999,"15min":1.105},"last":0.54600000000000004}]},{"osd":6,"last update":"Thu Oct 19 14:17:42 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.71199999999999997,"15min":0.69899999999999995},"min":{"1min":0.54500000000000004,"5min":0.504,"15min":0.47099999999999997},"max":{"1min":0.88900000000000001,"5min":1.018,"15min":1.0669999999999999},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.73199999999999998,"15min":0.72099999999999997},"min":{"1min":0.61699999999999999,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":1.079},"last":0.60199999999999998}]},{"osd":7,"last update":"Thu Oct 19 14:17:42 2023","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.72199999999999998,"15min":0.70799999999999996},"min":{"1min":0.51700000000000002,"5min":0.499,"15min":0.45900000000000002},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":1.085},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.753,"5min":0.755,"15min":0.73899999999999999},"min":{"1min":0.60299999999999998,"5min":0.51900000000000002,"15min":0.42799999999999999},"max":{"1min":0.92900000000000005,"5min":0.98199999999999998,"15min":1.1739999999999999},"last":0.71799999999999997}]}]},{"osd":2,"up_from":13,"seq":55834575983,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60040,"kb_used_data":2488,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311800,"statfs":{"total":96636764160,"available":96575283200,"internally_reserved":0,"allocated":2547712,"data_stored":1299659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:18:34 2023","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.39600000000000002,"15min":0.40300000000000002},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.66600000000000004,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.443,"15min":0.439},"min":{"1min":0.249,"5min":0.249,"15min":0.223},"max":{"1min":0.53700000000000003,"5min":0.71399999999999997,"15min":0.73599999999999999},"last":0.53700000000000003}]},{"osd":1,"last update":"Thu Oct 19 14:18:34 2023","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.40500000000000003,"15min":0.41599999999999998},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.53500000000000003,"5min":0.61699999999999999,"15min":0.65400000000000003},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.46300000000000002,"15min":0.46600000000000003},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.59799999999999998,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":1.742}]},{"osd":3,"last update":"Thu Oct 19 14:18:34 2023","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.49299999999999999,"15min":0.48799999999999999},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.65900000000000003,"5min":0.69299999999999995,"15min":0.81499999999999995},"last":0.317},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.502,"15min":0.51300000000000001},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.315},"max":{"1min":0.65400000000000003,"5min":0.73499999999999999,"15min":0.83599999999999997},"last":0.49099999999999999}]},{"osd":4,"last update":"Thu Oct 19 14:18:34 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.66300000000000003,"15min":0.67200000000000004},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39500000000000002},"max":{"1min":0.89200000000000002,"5min":0.93899999999999995,"15min":0.998},"last":0.622},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.67800000000000005,"15min":0.68999999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.35199999999999998},"max":{"1min":0.91800000000000004,"5min":0.98799999999999999,"15min":1.083},"last":0.51700000000000002}]},{"osd":5,"last update":"Thu Oct 19 14:18:34 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.69999999999999996,"15min":0.70499999999999996},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.40600000000000003},"max":{"1min":0.93999999999999995,"5min":1.008,"15min":1.0289999999999999},"last":1.4330000000000001},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.67000000000000004,"15min":0.68999999999999995},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.39300000000000002},"max":{"1min":0.81499999999999995,"5min":1.036,"15min":1.04},"last":1.595}]},{"osd":6,"last update":"Thu Oct 19 14:18:34 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.70299999999999996,"15min":0.71699999999999997},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.92800000000000005,"5min":0.93200000000000005,"15min":1.0940000000000001},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.72699999999999998,"15min":0.72799999999999998},"min":{"1min":0.433,"5min":0.433,"15min":0.40600000000000003},"max":{"1min":0.92000000000000004,"5min":0.998,"15min":1.1140000000000001},"last":1.4930000000000001}]},{"osd":7,"last update":"Thu Oct 19 14:18:34 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.72499999999999998,"15min":0.73399999999999999},"min":{"1min":0.55600000000000005,"5min":0.51600000000000001,"15min":0.41799999999999998},"max":{"1min":0.95399999999999996,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.73299999999999998,"15min":0.73599999999999999},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.98099999999999998,"5min":1.054,"15min":1.101},"last":1.53}]}]},{"osd":1,"up_from":13,"seq":55834575982,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64212,"kb_used_data":2500,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94307628,"statfs":{"total":96636764160,"available":96571011072,"internally_reserved":0,"allocated":2560000,"data_stored":1306782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":63175110},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:18:33 2023","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.40799999999999997,"15min":0.40899999999999997},"min":{"1min":0.27900000000000003,"5min":0.26700000000000002,"15min":0.23100000000000001},"max":{"1min":0.61599999999999999,"5min":0.67400000000000004,"15min":0.67800000000000005},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.46800000000000003,"15min":0.46400000000000002},"min":{"1min":0.27500000000000002,"5min":0.26400000000000001,"15min":0.251},"max":{"1min":0.64200000000000002,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.32800000000000001}]},{"osd":2,"last update":"Thu Oct 19 14:18:33 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.45000000000000001,"15min":0.44700000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.27700000000000002},"max":{"1min":0.56299999999999994,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.498,"5min":0.502,"15min":0.49299999999999999},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.68700000000000006,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.432}]},{"osd":3,"last update":"Thu Oct 19 14:18:33 2023","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51800000000000002,"15min":0.50800000000000001},"min":{"1min":0.36499999999999999,"5min":0.35299999999999998,"15min":0.29499999999999998},"max":{"1min":0.74399999999999999,"5min":0.76000000000000001,"15min":0.78900000000000003},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.439,"5min":0.45200000000000001,"15min":0.45300000000000001},"min":{"1min":0.32600000000000001,"5min":0.314,"15min":0.26100000000000001},"max":{"1min":0.56100000000000005,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.40899999999999997}]},{"osd":4,"last update":"Thu Oct 19 14:18:33 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.68999999999999995,"15min":0.67800000000000005},"min":{"1min":0.45100000000000001,"5min":0.372,"15min":0.372},"max":{"1min":0.95999999999999996,"5min":1.042,"15min":1.042},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.621,"5min":0.66700000000000004,"15min":0.67200000000000004},"min":{"1min":0.46100000000000002,"5min":0.41299999999999998,"15min":0.32300000000000001},"max":{"1min":0.79800000000000004,"5min":0.93500000000000005,"15min":1.1659999999999999},"last":0.64500000000000002}]},{"osd":5,"last update":"Thu Oct 19 14:18:33 2023","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.69299999999999995,"15min":0.68999999999999995},"min":{"1min":0.47199999999999998,"5min":0.42799999999999999,"15min":0.42499999999999999},"max":{"1min":0.88400000000000001,"5min":1.0589999999999999,"15min":1.3500000000000001},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.72699999999999998,"15min":0.71599999999999997},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.40500000000000003},"max":{"1min":0.92800000000000005,"5min":1.0149999999999999,"15min":1.121},"last":0.67200000000000004}]},{"osd":6,"last update":"Thu Oct 19 14:18:33 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.68799999999999994,"15min":0.69099999999999995},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.2310000000000001},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.72699999999999998,"15min":0.72199999999999998},"min":{"1min":0.48199999999999998,"5min":0.441,"15min":0.441},"max":{"1min":0.88500000000000001,"5min":1.107,"15min":1.274},"last":0.52800000000000002}]},{"osd":7,"last update":"Thu Oct 19 14:18:33 2023","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.74099999999999999,"15min":0.72999999999999998},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.99099999999999999,"5min":1.075,"15min":1.075},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.71399999999999997,"15min":0.70699999999999996},"min":{"1min":0.498,"5min":0.46100000000000002,"15min":0.44400000000000001},"max":{"1min":0.97599999999999998,"5min":1.0169999999999999,"15min":1.117},"last":0.53500000000000003}]}]},{"osd":0,"up_from":13,"seq":55834575983,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56480,"kb_used_data":3092,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315360,"statfs":{"total":96636764160,"available":96578928640,"internally_reserved":0,"allocated":3166208,"data_stored":1897259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Oct 19 14:18:05 2023","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.41599999999999998,"15min":0.41499999999999998},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.253},"max":{"1min":0.59999999999999998,"5min":0.624,"15min":0.628},"last":0.374},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.46600000000000003,"15min":0.47199999999999998},"min":{"1min":0.32200000000000001,"5min":0.29799999999999999,"15min":0.27800000000000002},"max":{"1min":0.58899999999999997,"5min":0.70899999999999996,"15min":0.81100000000000005},"last":0.25700000000000001}]},{"osd":2,"last update":"Thu Oct 19 14:18:05 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.50900000000000001,"15min":0.48799999999999999},"min":{"1min":0.35399999999999998,"5min":0.316,"15min":0.29599999999999999},"max":{"1min":0.73099999999999998,"5min":0.754,"15min":0.81200000000000006},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.45500000000000002,"15min":0.45600000000000002},"min":{"1min":0.33000000000000002,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.59099999999999997,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.32800000000000001}]},{"osd":3,"last update":"Thu Oct 19 14:18:05 2023","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.45300000000000001,"15min":0.443},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.28999999999999998},"max":{"1min":0.54900000000000004,"5min":0.61099999999999999,"15min":0.66800000000000004},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52100000000000002,"15min":0.51700000000000002},"min":{"1min":0.376,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.78100000000000003,"5min":0.79800000000000004,"15min":0.84599999999999997},"last":0.39100000000000001}]},{"osd":4,"last update":"Thu Oct 19 14:18:05 2023","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.68500000000000005,"15min":0.68200000000000005},"min":{"1min":0.495,"5min":0.46200000000000002,"15min":0.44},"max":{"1min":0.85799999999999998,"5min":1.0069999999999999,"15min":1.1890000000000001},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.69299999999999995,"15min":0.69699999999999995},"min":{"1min":0.54200000000000004,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.93400000000000005,"5min":0.996,"15min":1.2150000000000001},"last":0.75900000000000001}]},{"osd":5,"last update":"Thu Oct 19 14:18:05 2023","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.73299999999999998,"15min":0.73299999999999998},"min":{"1min":0.54300000000000004,"5min":0.48399999999999999,"15min":0.45500000000000002},"max":{"1min":0.95499999999999996,"5min":1.0189999999999999,"15min":1.1579999999999999},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.72599999999999998,"15min":0.73299999999999998},"min":{"1min":0.46400000000000002,"5min":0.44,"15min":0.44},"max":{"1min":0.98299999999999998,"5min":1.038,"15min":1.2310000000000001},"last":0.68200000000000005}]},{"osd":6,"last update":"Thu Oct 19 14:18:05 2023","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.71299999999999997,"15min":0.71499999999999997},"min":{"1min":0.55000000000000004,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.91300000000000003,"5min":0.97699999999999998,"15min":1.1739999999999999},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.72299999999999998,"15min":0.71899999999999997},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46300000000000002},"max":{"1min":0.89900000000000002,"5min":1.01,"15min":1.0189999999999999},"last":0.59899999999999998}]},{"osd":7,"last update":"Thu Oct 19 14:18:05 2023","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.72399999999999998,"15min":0.72799999999999998},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.874,"5min":1.014,"15min":1.1180000000000001},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.72399999999999998,"15min":0.71599999999999997},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.45100000000000001},"max":{"1min":0.93799999999999994,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.72699999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-19T14:18:42.850 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-19T14:18:42.851 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-19T14:18:42.999 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.996+0000 7f9b439e7640 1 -- 172.21.15.57:0/2846529269 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b28004d00 msgr2=0x7f9b28005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:43.000 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.996+0000 7f9b439e7640 1 --2- 172.21.15.57:0/2846529269 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b28004d00 0x7f9b28005180 secure :-1 s=READY pgs=2208 cs=0 l=1 rev1=1 crypto rx=0x7f9b34001650 tx=0x7f9b280057f0 comp rx=0 tx=0).stop 2023-10-19T14:18:43.000 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.996+0000 7f9b41f84640 1 -- 172.21.15.57:0/2846529269 reap_dead start 2023-10-19T14:18:43.000 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.996+0000 7f9b439e7640 1 -- 172.21.15.57:0/2846529269 shutdown_connections 2023-10-19T14:18:43.000 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.996+0000 7f9b439e7640 1 -- 172.21.15.57:0/2846529269 >> 172.21.15.57:0/2846529269 conn(0x7f9b3c069930 msgr2=0x7f9b3c069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:43.001 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.996+0000 7f9b439e7640 1 -- 172.21.15.57:0/2846529269 shutdown_connections 2023-10-19T14:18:43.001 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.996+0000 7f9b439e7640 1 -- 172.21.15.57:0/2846529269 wait complete. 2023-10-19T14:18:43.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.998+0000 7f9b439e7640 1 Processor -- start 2023-10-19T14:18:43.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.998+0000 7f9b439e7640 1 -- start start 2023-10-19T14:18:43.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.998+0000 7f9b439e7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b28004d00 0x7f9b3c151d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:43.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.998+0000 7f9b439e7640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9b3c139190 0x7f9b3c1e4670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:43.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.998+0000 7f9b439e7640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9b3c139610 0x7f9b3c1e4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:43.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.998+0000 7f9b439e7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b3c1f3b00 con 0x7f9b28004d00 2023-10-19T14:18:43.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.998+0000 7f9b439e7640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b3c1f2f80 con 0x7f9b3c139610 2023-10-19T14:18:43.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.998+0000 7f9b439e7640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b3c1f3260 con 0x7f9b3c139190 2023-10-19T14:18:43.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.998+0000 7f9b41783640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b28004d00 0x7f9b3c151d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:43.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.999+0000 7f9b41783640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b28004d00 0x7f9b3c151d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55598/0 (socket says 172.21.15.57:55598) 2023-10-19T14:18:43.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.999+0000 7f9b41783640 1 -- 172.21.15.57:0/2423737756 learned_addr learned my addr 172.21.15.57:0/2423737756 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:18:43.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.999+0000 7f9b40f82640 1 --2- 172.21.15.57:0/2423737756 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9b3c139190 0x7f9b3c1e4670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:43.006 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.999+0000 7f9b41f84640 1 --2- 172.21.15.57:0/2423737756 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9b3c139610 0x7f9b3c1e4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:43.006 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.999+0000 7f9b41783640 1 -- 172.21.15.57:0/2423737756 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9b3c139190 msgr2=0x7f9b3c1e4670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:43.006 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.999+0000 7f9b41783640 1 --2- 172.21.15.57:0/2423737756 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9b3c139190 0x7f9b3c1e4670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:43.006 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.999+0000 7f9b41783640 1 -- 172.21.15.57:0/2423737756 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9b3c139610 msgr2=0x7f9b3c1e4bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:43.006 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.999+0000 7f9b41783640 1 --2- 172.21.15.57:0/2423737756 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9b3c139610 0x7f9b3c1e4bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:43.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.999+0000 7f9b41783640 1 -- 172.21.15.57:0/2423737756 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b3c1f3540 con 0x7f9b28004d00 2023-10-19T14:18:43.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.999+0000 7f9b41f84640 1 --2- 172.21.15.57:0/2423737756 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9b3c139610 0x7f9b3c1e4bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:18:43.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.999+0000 7f9b41783640 1 --2- 172.21.15.57:0/2423737756 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b28004d00 0x7f9b3c151d90 secure :-1 s=READY pgs=2209 cs=0 l=1 rev1=1 crypto rx=0x7f9b38011d60 tx=0x7f9b3800f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:43.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.999+0000 7f9b31ffb640 1 -- 172.21.15.57:0/2423737756 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b38003d80 con 0x7f9b28004d00 2023-10-19T14:18:43.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.999+0000 7f9b31ffb640 1 -- 172.21.15.57:0/2423737756 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f9b38004b20 con 0x7f9b28004d00 2023-10-19T14:18:43.008 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.999+0000 7f9b439e7640 1 -- 172.21.15.57:0/2423737756 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b3c1f3820 con 0x7f9b28004d00 2023-10-19T14:18:43.008 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.999+0000 7f9b31ffb640 1 -- 172.21.15.57:0/2423737756 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b38004050 con 0x7f9b28004d00 2023-10-19T14:18:43.008 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:42.999+0000 7f9b439e7640 1 -- 172.21.15.57:0/2423737756 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b3c1f2ba0 con 0x7f9b28004d00 2023-10-19T14:18:43.008 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:43.000+0000 7f9b439e7640 1 -- 172.21.15.57:0/2423737756 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b3c0690e0 con 0x7f9b28004d00 2023-10-19T14:18:43.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:43.001+0000 7f9b31ffb640 1 -- 172.21.15.57:0/2423737756 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 117) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f9b380041b0 con 0x7f9b28004d00 2023-10-19T14:18:43.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:43.001+0000 7f9b31ffb640 1 --2- 172.21.15.57:0/2423737756 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f9b2006af60 0x7f9b2006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:43.010 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:43.001+0000 7f9b31ffb640 1 -- 172.21.15.57:0/2423737756 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f9b3808ac90 con 0x7f9b28004d00 2023-10-19T14:18:43.010 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:43.001+0000 7f9b40f82640 1 --2- 172.21.15.57:0/2423737756 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f9b2006af60 0x7f9b2006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:43.010 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:43.002+0000 7f9b40f82640 1 --2- 172.21.15.57:0/2423737756 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f9b2006af60 0x7f9b2006d420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c0cc910 tx=0x7f9b2c000bc0 comp rx=0 tx=0).ready entity=mgr.8744 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:43.010 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:43.005+0000 7f9b31ffb640 1 -- 172.21.15.57:0/2423737756 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9b380559c0 con 0x7f9b28004d00 2023-10-19T14:18:43.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:43.254+0000 7f9b439e7640 1 -- 172.21.15.57:0/2423737756 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f9b3c152b90 con 0x7f9b28004d00 2023-10-19T14:18:43.263 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:43.261+0000 7f9b31ffb640 1 -- 172.21.15.57:0/2423737756 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v58) v1 ==== 190+0+147 (secure 0 0 0) 0x7f9b38058fe0 con 0x7f9b28004d00 2023-10-19T14:18:43.264 INFO:teuthology.orchestra.run.smithi057.stdout:[client.0] 2023-10-19T14:18:43.264 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQBkJDFlg9yuExAATxiWDfS+p07G1F0pORbJEg== 2023-10-19T14:18:43.264 INFO:teuthology.orchestra.run.smithi057.stdout: caps mds = "allow" 2023-10-19T14:18:43.264 INFO:teuthology.orchestra.run.smithi057.stdout: caps mon = "allow r" 2023-10-19T14:18:43.265 INFO:teuthology.orchestra.run.smithi057.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-19T14:18:43.265 INFO:teuthology.orchestra.run.smithi057.stderr:updated caps for client.0 2023-10-19T14:18:43.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:43.263+0000 7f9b439e7640 1 -- 172.21.15.57:0/2423737756 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f9b2006af60 msgr2=0x7f9b2006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:43.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:43.263+0000 7f9b439e7640 1 --2- 172.21.15.57:0/2423737756 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f9b2006af60 0x7f9b2006d420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c0cc910 tx=0x7f9b2c000bc0 comp rx=0 tx=0).stop 2023-10-19T14:18:43.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:43.263+0000 7f9b439e7640 1 -- 172.21.15.57:0/2423737756 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b28004d00 msgr2=0x7f9b3c151d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:43.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:43.263+0000 7f9b439e7640 1 --2- 172.21.15.57:0/2423737756 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b28004d00 0x7f9b3c151d90 secure :-1 s=READY pgs=2209 cs=0 l=1 rev1=1 crypto rx=0x7f9b38011d60 tx=0x7f9b3800f790 comp rx=0 tx=0).stop 2023-10-19T14:18:43.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:43.263+0000 7f9b439e7640 1 -- 172.21.15.57:0/2423737756 shutdown_connections 2023-10-19T14:18:43.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:43.263+0000 7f9b439e7640 1 --2- 172.21.15.57:0/2423737756 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f9b2006af60 0x7f9b2006d420 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:43.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:43.263+0000 7f9b439e7640 1 --2- 172.21.15.57:0/2423737756 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9b3c139610 0x7f9b3c1e4bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:43.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:43.263+0000 7f9b439e7640 1 --2- 172.21.15.57:0/2423737756 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9b3c139190 0x7f9b3c1e4670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:43.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:43.263+0000 7f9b439e7640 1 --2- 172.21.15.57:0/2423737756 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b28004d00 0x7f9b3c151d90 unknown :-1 s=CLOSED pgs=2209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:43.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:43.263+0000 7f9b439e7640 1 -- 172.21.15.57:0/2423737756 >> 172.21.15.57:0/2423737756 conn(0x7f9b3c069930 msgr2=0x7f9b3c069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:43.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:43.263+0000 7f9b439e7640 1 -- 172.21.15.57:0/2423737756 shutdown_connections 2023-10-19T14:18:43.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:43.263+0000 7f9b439e7640 1 -- 172.21.15.57:0/2423737756 wait complete. 2023-10-19T14:18:43.276 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T14:18:43.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:43.384+0000 7f7d6c105640 1 -- 172.21.15.57:0/2194952521 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7d500047c0 msgr2=0x7f7d50004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:43.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:43.384+0000 7f7d6c105640 1 --2- 172.21.15.57:0/2194952521 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7d500047c0 0x7f7d50004c40 secure :-1 s=READY pgs=2211 cs=0 l=1 rev1=1 crypto rx=0x7f7d54006f80 tx=0x7f7d500057f0 comp rx=0 tx=0).stop 2023-10-19T14:18:43.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:43.384+0000 7f7d6c105640 1 -- 172.21.15.57:0/2194952521 shutdown_connections 2023-10-19T14:18:43.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:43.384+0000 7f7d6c105640 1 --2- 172.21.15.57:0/2194952521 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7d500047c0 0x7f7d50004c40 unknown :-1 s=CLOSED pgs=2211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:43.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:43.384+0000 7f7d6c105640 1 -- 172.21.15.57:0/2194952521 >> 172.21.15.57:0/2194952521 conn(0x7f7d640d0d60 msgr2=0x7f7d640d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:43.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:43.384+0000 7f7d6c105640 1 -- 172.21.15.57:0/2194952521 shutdown_connections 2023-10-19T14:18:43.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:43.384+0000 7f7d6c105640 1 -- 172.21.15.57:0/2194952521 wait complete. 2023-10-19T14:18:43.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:43.385+0000 7f7d6c105640 1 Processor -- start 2023-10-19T14:18:43.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:43.385+0000 7f7d6c105640 1 -- start start 2023-10-19T14:18:43.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:43.386+0000 7f7d6c105640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7d500047c0 0x7f7d640b5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:43.391 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:43.386+0000 7f7d6c105640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7d641f4680 0x7f7d6415d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:43.391 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:43.386+0000 7f7d6c105640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7d640caa60 0x7f7d6414ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:43.391 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:43.386+0000 7f7d6c105640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d641f1090 con 0x7f7d641f4680 2023-10-19T14:18:43.391 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:43.386+0000 7f7d6c105640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d640b5590 con 0x7f7d500047c0 2023-10-19T14:18:43.392 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:43.386+0000 7f7d6c105640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d641f4b10 con 0x7f7d640caa60 2023-10-19T14:18:43.392 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:43.386+0000 7f7d696a0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7d641f4680 0x7f7d6415d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:43.392 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:43.386+0000 7f7d696a0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7d641f4680 0x7f7d6415d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55616/0 (socket says 172.21.15.57:55616) 2023-10-19T14:18:43.392 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:43.386+0000 7f7d696a0640 1 -- 172.21.15.57:0/642899714 learned_addr learned my addr 172.21.15.57:0/642899714 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:18:43.392 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:43.386+0000 7f7d6a6a2640 1 --2- 172.21.15.57:0/642899714 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7d640caa60 0x7f7d6414ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:43.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:43.386+0000 7f7d69ea1640 1 --2- 172.21.15.57:0/642899714 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7d500047c0 0x7f7d640b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:43.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:43.386+0000 7f7d696a0640 1 -- 172.21.15.57:0/642899714 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7d640caa60 msgr2=0x7f7d6414ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:43.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:43.386+0000 7f7d696a0640 1 --2- 172.21.15.57:0/642899714 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7d640caa60 0x7f7d6414ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:43.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:43.386+0000 7f7d696a0640 1 -- 172.21.15.57:0/642899714 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7d500047c0 msgr2=0x7f7d640b5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:43.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:43.386+0000 7f7d696a0640 1 --2- 172.21.15.57:0/642899714 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7d500047c0 0x7f7d640b5050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:43.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:43.386+0000 7f7d696a0640 1 -- 172.21.15.57:0/642899714 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d641f4df0 con 0x7f7d641f4680 2023-10-19T14:18:43.394 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:43.386+0000 7f7d696a0640 1 --2- 172.21.15.57:0/642899714 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7d641f4680 0x7f7d6415d900 secure :-1 s=READY pgs=2212 cs=0 l=1 rev1=1 crypto rx=0x7f7d5402c860 tx=0x7f7d540065f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:43.394 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:43.386+0000 7f7d5a7fc640 1 -- 172.21.15.57:0/642899714 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d54003a00 con 0x7f7d641f4680 2023-10-19T14:18:43.394 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:43.386+0000 7f7d5a7fc640 1 -- 172.21.15.57:0/642899714 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f7d54003c50 con 0x7f7d641f4680 2023-10-19T14:18:43.394 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:43.386+0000 7f7d5a7fc640 1 -- 172.21.15.57:0/642899714 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d54035470 con 0x7f7d641f4680 2023-10-19T14:18:43.394 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:43.386+0000 7f7d6c105640 1 -- 172.21.15.57:0/642899714 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d641f50d0 con 0x7f7d641f4680 2023-10-19T14:18:43.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:43.386+0000 7f7d6c105640 1 -- 172.21.15.57:0/642899714 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d6415de40 con 0x7f7d641f4680 2023-10-19T14:18:43.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:43.387+0000 7f7d6c105640 1 -- 172.21.15.57:0/642899714 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d641f55f0 con 0x7f7d641f4680 2023-10-19T14:18:43.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:43.388+0000 7f7d5a7fc640 1 -- 172.21.15.57:0/642899714 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 117) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f7d54032070 con 0x7f7d641f4680 2023-10-19T14:18:43.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:43.388+0000 7f7d5a7fc640 1 --2- 172.21.15.57:0/642899714 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f7d4806af60 0x7f7d4806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:43.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:43.388+0000 7f7d5a7fc640 1 -- 172.21.15.57:0/642899714 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f7d540ab330 con 0x7f7d641f4680 2023-10-19T14:18:43.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:43.388+0000 7f7d69ea1640 1 --2- 172.21.15.57:0/642899714 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f7d4806af60 0x7f7d4806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:43.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:43.389+0000 7f7d69ea1640 1 --2- 172.21.15.57:0/642899714 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f7d4806af60 0x7f7d4806d420 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7d640682a0 tx=0x7f7d5c005f90 comp rx=0 tx=0).ready entity=mgr.8744 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:43.397 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:43.393+0000 7f7d5a7fc640 1 -- 172.21.15.57:0/642899714 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7d54076150 con 0x7f7d641f4680 2023-10-19T14:18:43.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:43.604+0000 7f7d6c105640 1 -- 172.21.15.57:0/642899714 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7d6415dfd0 con 0x7f7d641f4680 2023-10-19T14:18:43.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:43.605+0000 7f7d5a7fc640 1 -- 172.21.15.57:0/642899714 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+4926 (secure 0 0 0) 0x7f7d54044020 con 0x7f7d641f4680 2023-10-19T14:18:43.608 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:18:43.608 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":83,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/2313260224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2313260224},{"type":"v1","addr":"172.21.15.179:6836","nonce":2313260224}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":10081,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/494432208","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":494432208},{"type":"v1","addr":"172.21.15.179:6837","nonce":494432208}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":11214,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/3080658822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":3080658822},{"type":"v1","addr":"172.21.15.57:6834","nonce":3080658822}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T14:18:38.834485+0000","modified":"2023-10-19T14:18:40.833363+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11217":{"gid":11217,"name":"a","rank":0,"incarnation":81,"state":"up:active","state_seq":3,"addr":"172.21.15.57:6835/1674945162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1674945162},{"type":"v1","addr":"172.21.15.57:6835","nonce":1674945162}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2023-10-19T14:18:43.609 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 83 2023-10-19T14:18:43.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:43.607+0000 7f7d6c105640 1 -- 172.21.15.57:0/642899714 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f7d4806af60 msgr2=0x7f7d4806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:43.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:43.607+0000 7f7d6c105640 1 --2- 172.21.15.57:0/642899714 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f7d4806af60 0x7f7d4806d420 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7d640682a0 tx=0x7f7d5c005f90 comp rx=0 tx=0).stop 2023-10-19T14:18:43.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:43.607+0000 7f7d6c105640 1 -- 172.21.15.57:0/642899714 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7d641f4680 msgr2=0x7f7d6415d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:43.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:43.607+0000 7f7d6c105640 1 --2- 172.21.15.57:0/642899714 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7d641f4680 0x7f7d6415d900 secure :-1 s=READY pgs=2212 cs=0 l=1 rev1=1 crypto rx=0x7f7d5402c860 tx=0x7f7d540065f0 comp rx=0 tx=0).stop 2023-10-19T14:18:43.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:43.607+0000 7f7d6c105640 1 -- 172.21.15.57:0/642899714 shutdown_connections 2023-10-19T14:18:43.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:43.607+0000 7f7d6c105640 1 --2- 172.21.15.57:0/642899714 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f7d4806af60 0x7f7d4806d420 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:43.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:43.607+0000 7f7d6c105640 1 --2- 172.21.15.57:0/642899714 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7d640caa60 0x7f7d6414ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:43.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:43.607+0000 7f7d6c105640 1 --2- 172.21.15.57:0/642899714 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7d641f4680 0x7f7d6415d900 unknown :-1 s=CLOSED pgs=2212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:43.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:43.607+0000 7f7d6c105640 1 --2- 172.21.15.57:0/642899714 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7d500047c0 0x7f7d640b5050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:43.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:43.607+0000 7f7d6c105640 1 -- 172.21.15.57:0/642899714 >> 172.21.15.57:0/642899714 conn(0x7f7d640d0d60 msgr2=0x7f7d6406a8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:43.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:43.607+0000 7f7d6c105640 1 -- 172.21.15.57:0/642899714 shutdown_connections 2023-10-19T14:18:43.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:43.607+0000 7f7d6c105640 1 -- 172.21.15.57:0/642899714 wait complete. 2023-10-19T14:18:44.022 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 83, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-19T14:18:38.834485+0000', 'modified': '2023-10-19T14:18:40.833363+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 11217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_11217': {'gid': 11217, 'name': 'a', 'rank': 0, 'incarnation': 81, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.57:6835/1674945162', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6833', 'nonce': 1674945162}, {'type': 'v1', 'addr': '172.21.15.57:6835', 'nonce': 1674945162}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [26], 'metadata_pool': 25, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-19T14:18:44.023 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-19T14:18:44.023 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2023-10-19T14:18:44.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.135+0000 7fbf7bc26640 1 -- 172.21.15.57:0/2409267377 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbf74070350 msgr2=0x7fbf74070780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:44.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.135+0000 7fbf7bc26640 1 --2- 172.21.15.57:0/2409267377 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbf74070350 0x7fbf74070780 secure :-1 s=READY pgs=2214 cs=0 l=1 rev1=1 crypto rx=0x7fbf7406f4d0 tx=0x7fbf6c02c2f0 comp rx=0 tx=0).stop 2023-10-19T14:18:44.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.135+0000 7fbf7bc26640 1 -- 172.21.15.57:0/2409267377 shutdown_connections 2023-10-19T14:18:44.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.135+0000 7fbf7bc26640 1 --2- 172.21.15.57:0/2409267377 >> v2:172.21.15.179:3300/0 conn(0x7fbf7413d350 0x7fbf740cbd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:44.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.136+0000 7fbf7bc26640 1 --2- 172.21.15.57:0/2409267377 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbf74070350 0x7fbf74070780 unknown :-1 s=CLOSED pgs=2214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:44.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.136+0000 7fbf7bc26640 1 -- 172.21.15.57:0/2409267377 >> 172.21.15.57:0/2409267377 conn(0x7fbf74069930 msgr2=0x7fbf74069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:44.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.136+0000 7fbf7bc26640 1 -- 172.21.15.57:0/2409267377 shutdown_connections 2023-10-19T14:18:44.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.136+0000 7fbf7bc26640 1 -- 172.21.15.57:0/2409267377 wait complete. 2023-10-19T14:18:44.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.137+0000 7fbf7bc26640 1 Processor -- start 2023-10-19T14:18:44.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.137+0000 7fbf7bc26640 1 -- start start 2023-10-19T14:18:44.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.137+0000 7fbf7bc26640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbf74070350 0x7fbf741f2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:44.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.137+0000 7fbf7bc26640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbf740cb980 0x7fbf74135fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:44.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.137+0000 7fbf7bc26640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbf74157a20 0x7fbf74140b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:44.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.138+0000 7fbf7bc26640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf74155750 con 0x7fbf740cb980 2023-10-19T14:18:44.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.138+0000 7fbf7bc26640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf741f3500 con 0x7fbf74070350 2023-10-19T14:18:44.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.138+0000 7fbf7bc26640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf74155190 con 0x7fbf74157a20 2023-10-19T14:18:44.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.138+0000 7fbf791c1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbf740cb980 0x7fbf74135fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:44.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.138+0000 7fbf791c1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbf740cb980 0x7fbf74135fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55638/0 (socket says 172.21.15.57:55638) 2023-10-19T14:18:44.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.138+0000 7fbf791c1640 1 -- 172.21.15.57:0/731022855 learned_addr learned my addr 172.21.15.57:0/731022855 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:18:44.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.138+0000 7fbf7a1c3640 1 --2- 172.21.15.57:0/731022855 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbf74157a20 0x7fbf74140b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:44.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.138+0000 7fbf799c2640 1 --2- 172.21.15.57:0/731022855 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbf74070350 0x7fbf741f2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:44.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.138+0000 7fbf791c1640 1 -- 172.21.15.57:0/731022855 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbf74157a20 msgr2=0x7fbf74140b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:44.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.138+0000 7fbf791c1640 1 --2- 172.21.15.57:0/731022855 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbf74157a20 0x7fbf74140b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:44.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.138+0000 7fbf791c1640 1 -- 172.21.15.57:0/731022855 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbf74070350 msgr2=0x7fbf741f2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:44.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.138+0000 7fbf791c1640 1 --2- 172.21.15.57:0/731022855 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbf74070350 0x7fbf741f2fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:44.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.138+0000 7fbf791c1640 1 -- 172.21.15.57:0/731022855 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf6c009900 con 0x7fbf740cb980 2023-10-19T14:18:44.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.138+0000 7fbf791c1640 1 --2- 172.21.15.57:0/731022855 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbf740cb980 0x7fbf74135fc0 secure :-1 s=READY pgs=2215 cs=0 l=1 rev1=1 crypto rx=0x7fbf6400b410 tx=0x7fbf6400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:44.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.138+0000 7fbf6a7fc640 1 -- 172.21.15.57:0/731022855 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf64011830 con 0x7fbf740cb980 2023-10-19T14:18:44.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.138+0000 7fbf6a7fc640 1 -- 172.21.15.57:0/731022855 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fbf64011aa0 con 0x7fbf740cb980 2023-10-19T14:18:44.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.138+0000 7fbf7bc26640 1 -- 172.21.15.57:0/731022855 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf74155470 con 0x7fbf740cb980 2023-10-19T14:18:44.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.138+0000 7fbf6a7fc640 1 -- 172.21.15.57:0/731022855 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf64010380 con 0x7fbf740cb980 2023-10-19T14:18:44.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.138+0000 7fbf7bc26640 1 -- 172.21.15.57:0/731022855 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf741f2aa0 con 0x7fbf740cb980 2023-10-19T14:18:44.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.140+0000 7fbf6a7fc640 1 -- 172.21.15.57:0/731022855 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 117) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fbf640104e0 con 0x7fbf740cb980 2023-10-19T14:18:44.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.140+0000 7fbf6a7fc640 1 --2- 172.21.15.57:0/731022855 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fbf5806af60 0x7fbf5806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:44.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.140+0000 7fbf6a7fc640 1 -- 172.21.15.57:0/731022855 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fbf64089c70 con 0x7fbf740cb980 2023-10-19T14:18:44.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.140+0000 7fbf799c2640 1 --2- 172.21.15.57:0/731022855 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fbf5806af60 0x7fbf5806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:44.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.141+0000 7fbf799c2640 1 --2- 172.21.15.57:0/731022855 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fbf5806af60 0x7fbf5806d420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fbf7406f900 tx=0x7fbf70009040 comp rx=0 tx=0).ready entity=mgr.8744 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:44.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.141+0000 7fbf7e7bb740 1 Processor -- start 2023-10-19T14:18:44.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.141+0000 7fbf7e7bb740 1 -- start start 2023-10-19T14:18:44.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.142+0000 7fbf7e7bb740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56044d32cb40 0x56044d2ffae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:44.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.142+0000 7fbf7e7bb740 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x56044d32cf20 0x56044d32f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:44.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.142+0000 7fbf7e7bb740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56044d300020 0x56044d32fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:44.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.142+0000 7fbf7e7bb740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x56044d211730 con 0x56044d300020 2023-10-19T14:18:44.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.142+0000 7fbf7e7bb740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x56044d272780 con 0x56044d32cb40 2023-10-19T14:18:44.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.142+0000 7fbf7e7bb740 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x56044d26fe70 con 0x56044d32cf20 2023-10-19T14:18:44.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.142+0000 7fbf7a1c3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56044d300020 0x56044d32fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:44.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.142+0000 7fbf7a1c3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56044d300020 0x56044d32fe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55654/0 (socket says 172.21.15.57:55654) 2023-10-19T14:18:44.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.142+0000 7fbf7a1c3640 1 -- 172.21.15.57:0/1934664317 learned_addr learned my addr 172.21.15.57:0/1934664317 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:18:44.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.142+0000 7fbf799c2640 1 --2- 172.21.15.57:0/1934664317 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56044d32cb40 0x56044d2ffae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:44.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.142+0000 7fbf791c1640 1 --2- 172.21.15.57:0/1934664317 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x56044d32cf20 0x56044d32f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:44.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.142+0000 7fbf7a1c3640 1 -- 172.21.15.57:0/1934664317 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x56044d32cf20 msgr2=0x56044d32f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T14:18:44.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.142+0000 7fbf7a1c3640 1 --2- 172.21.15.57:0/1934664317 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x56044d32cf20 0x56044d32f8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:44.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.142+0000 7fbf7a1c3640 1 -- 172.21.15.57:0/1934664317 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56044d32cb40 msgr2=0x56044d2ffae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T14:18:44.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.142+0000 7fbf7a1c3640 1 --2- 172.21.15.57:0/1934664317 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56044d32cb40 0x56044d2ffae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:44.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.142+0000 7fbf7a1c3640 1 -- 172.21.15.57:0/1934664317 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56044d2703d0 con 0x56044d300020 2023-10-19T14:18:44.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.143+0000 7fbf799c2640 1 --2- 172.21.15.57:0/1934664317 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56044d32cb40 0x56044d2ffae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:18:44.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.143+0000 7fbf791c1640 1 --2- 172.21.15.57:0/1934664317 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x56044d32cf20 0x56044d32f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:18:44.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.143+0000 7fbf7a1c3640 1 --2- 172.21.15.57:0/1934664317 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56044d300020 0x56044d32fe10 secure :-1 s=READY pgs=2216 cs=0 l=1 rev1=1 crypto rx=0x7fbf6c009d60 tx=0x7fbf6c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9ed99d39468e01 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:44.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.143+0000 7fbf577fe640 1 -- 172.21.15.57:0/1934664317 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf6c02cea0 con 0x56044d300020 2023-10-19T14:18:44.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.143+0000 7fbf577fe640 1 -- 172.21.15.57:0/1934664317 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fbf6c003c60 con 0x56044d300020 2023-10-19T14:18:44.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.143+0000 7fbf577fe640 1 -- 172.21.15.57:0/1934664317 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf6c0348a0 con 0x56044d300020 2023-10-19T14:18:44.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.143+0000 7fbf7e7bb740 1 -- 172.21.15.57:0/1934664317 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56044d300020 msgr2=0x56044d32fe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:44.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.143+0000 7fbf7e7bb740 1 --2- 172.21.15.57:0/1934664317 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56044d300020 0x56044d32fe10 secure :-1 s=READY pgs=2216 cs=0 l=1 rev1=1 crypto rx=0x7fbf6c009d60 tx=0x7fbf6c004a40 comp rx=0 tx=0).stop 2023-10-19T14:18:44.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.143+0000 7fbf7e7bb740 1 -- 172.21.15.57:0/1934664317 shutdown_connections 2023-10-19T14:18:44.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.143+0000 7fbf7e7bb740 1 --2- 172.21.15.57:0/1934664317 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56044d300020 0x56044d32fe10 unknown :-1 s=CLOSED pgs=2216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:44.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.143+0000 7fbf7e7bb740 1 --2- 172.21.15.57:0/1934664317 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x56044d32cf20 0x56044d32f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:44.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.143+0000 7fbf7e7bb740 1 --2- 172.21.15.57:0/1934664317 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56044d32cb40 0x56044d2ffae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:44.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.143+0000 7fbf7e7bb740 1 -- 172.21.15.57:0/1934664317 >> 172.21.15.57:0/1934664317 conn(0x56044d301d50 msgr2=0x56044d34a8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:44.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.144+0000 7fbf7e7bb740 1 -- 172.21.15.57:0/1934664317 shutdown_connections 2023-10-19T14:18:44.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.144+0000 7fbf7e7bb740 1 -- 172.21.15.57:0/1934664317 wait complete. 2023-10-19T14:18:44.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.144+0000 7fbf7e7bb740 1 -- start start 2023-10-19T14:18:44.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.144+0000 7fbf7e7bb740 1 Processor -- start 2023-10-19T14:18:44.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.145+0000 7fbf7e7bb740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56044d32cb40 0x56044d3421d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:44.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.145+0000 7fbf7e7bb740 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x56044d32cf20 0x56044d32ead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:44.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.145+0000 7fbf7e7bb740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56044d35ac10 0x56044d35b070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:44.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.145+0000 7fbf7e7bb740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x56044d26b760 con 0x56044d35ac10 2023-10-19T14:18:44.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.145+0000 7fbf7e7bb740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x56044d2268f0 con 0x56044d32cb40 2023-10-19T14:18:44.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.145+0000 7fbf7e7bb740 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x56044d226590 con 0x56044d32cf20 2023-10-19T14:18:44.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.145+0000 7fbf7e7bb740 10 client.0 _ll_register_callbacks cb 0x56044d262ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-19T14:18:44.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.145+0000 7fbf7a1c3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56044d35ac10 0x56044d35b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:44.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.145+0000 7fbf7a1c3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56044d35ac10 0x56044d35b070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55662/0 (socket says 172.21.15.57:55662) 2023-10-19T14:18:44.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.145+0000 7fbf7a1c3640 1 -- 172.21.15.57:0/1910593664 learned_addr learned my addr 172.21.15.57:0/1910593664 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:18:44.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.145+0000 7fbf791c1640 1 --2- 172.21.15.57:0/1910593664 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x56044d32cf20 0x56044d32ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:44.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.145+0000 7fbf799c2640 1 --2- 172.21.15.57:0/1910593664 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56044d32cb40 0x56044d3421d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:44.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.146+0000 7fbf7a1c3640 1 -- 172.21.15.57:0/1910593664 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x56044d32cf20 msgr2=0x56044d32ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T14:18:44.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.146+0000 7fbf7a1c3640 1 --2- 172.21.15.57:0/1910593664 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x56044d32cf20 0x56044d32ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:44.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.146+0000 7fbf7a1c3640 1 -- 172.21.15.57:0/1910593664 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56044d32cb40 msgr2=0x56044d3421d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T14:18:44.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.146+0000 7fbf7a1c3640 1 --2- 172.21.15.57:0/1910593664 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56044d32cb40 0x56044d3421d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:44.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.146+0000 7fbf7a1c3640 1 -- 172.21.15.57:0/1910593664 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf7415b8f0 con 0x56044d35ac10 2023-10-19T14:18:44.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.146+0000 7fbf7a1c3640 1 --2- 172.21.15.57:0/1910593664 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56044d35ac10 0x56044d35b070 secure :-1 s=READY pgs=2217 cs=0 l=1 rev1=1 crypto rx=0x7fbf6c005c30 tx=0x7fbf6c005c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35650d3b24fbbe1f server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:44.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.146+0000 7fbf54ff9640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2023-10-19T14:18:44.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.146+0000 7fbf54ff9640 1 -- 172.21.15.57:0/1910593664 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf6c02cea0 con 0x56044d35ac10 2023-10-19T14:18:44.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.146+0000 7fbf7e7bb740 1 -- 172.21.15.57:0/1910593664 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56044d264740 con 0x56044d35ac10 2023-10-19T14:18:44.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.146+0000 7fbf54ff9640 1 -- 172.21.15.57:0/1910593664 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(205..205 src has 1..205) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fbf6c047860 con 0x56044d35ac10 2023-10-19T14:18:44.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.146+0000 7fbf54ff9640 1 -- 172.21.15.57:0/1910593664 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=83) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbf6c0039f0 con 0x56044d35ac10 2023-10-19T14:18:44.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.146+0000 7fbf7e7bb740 10 client.11352 fetch_fsmap learned FSMap version 83 2023-10-19T14:18:44.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.146+0000 7fbf7e7bb740 1 -- 172.21.15.57:0/1910593664 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=83,monmap=2+}) v3 -- 0x56044d269de0 con 0x56044d35ac10 2023-10-19T14:18:44.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.146+0000 7fbf54ff9640 1 -- 172.21.15.57:0/1910593664 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x56044d264740 con 0x56044d35ac10 2023-10-19T14:18:44.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.146+0000 7fbf54ff9640 1 -- 172.21.15.57:0/1910593664 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 83) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fbf6c047a20 con 0x56044d35ac10 2023-10-19T14:18:44.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.146+0000 7fbf7e7bb740 10 client.11352 fetch_fsmap finished waiting for FSMap version 83 2023-10-19T14:18:44.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.146+0000 7fbf7e7bb740 10 client.11352 resolve_mds: resolved name 'a' to daemon mds.a 2023-10-19T14:18:44.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.146+0000 7fbf7e7bb740 20 client.11352 populate_metadata read hostname 'smithi057' 2023-10-19T14:18:44.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.146+0000 7fbf7e7bb740 1 --2- 172.21.15.57:0/1910593664 >> [v2:172.21.15.57:6833/1674945162,v1:172.21.15.57:6835/1674945162] conn(0x56044d35d580 0x56044d3685a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:44.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.146+0000 7fbf7e7bb740 4 client.11352 mds_command: new command op to 11217 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-19T14:18:44.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.146+0000 7fbf7e7bb740 1 -- 172.21.15.57:0/1910593664 --> [v2:172.21.15.57:6833/1674945162,v1:172.21.15.57:6835/1674945162] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56044d264460 con 0x56044d35d580 2023-10-19T14:18:44.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.147+0000 7fbf799c2640 1 --2- 172.21.15.57:0/1910593664 >> [v2:172.21.15.57:6833/1674945162,v1:172.21.15.57:6835/1674945162] conn(0x56044d35d580 0x56044d3685a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:44.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.147+0000 7fbf54ff9640 10 client.11352 ms_handle_connect on v2:172.21.15.57:6833/1674945162 2023-10-19T14:18:44.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.147+0000 7fbf799c2640 1 --2- 172.21.15.57:0/1910593664 >> [v2:172.21.15.57:6833/1674945162,v1:172.21.15.57:6835/1674945162] conn(0x56044d35d580 0x56044d3685a0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=66f1dce685f57ff8 server_cookie=828576fb067926fb in_seq=0 out_seq=0 2023-10-19T14:18:44.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.147+0000 7fbf54ff9640 1 -- 172.21.15.57:0/1910593664 <== mds.0 v2:172.21.15.57:6833/1674945162 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x56044d264460 con 0x56044d35d580 2023-10-19T14:18:44.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.147+0000 7fbf54ff9640 10 client.11352 handle_command_reply: tid=0 2023-10-19T14:18:44.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.147+0000 7fbf7e7bb740 1 client.11352 shutdown 2023-10-19T14:18:44.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.148+0000 7fbf7e7bb740 1 -- 172.21.15.57:0/1910593664 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56044d35ac10 msgr2=0x56044d35b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:44.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.148+0000 7fbf7e7bb740 1 --2- 172.21.15.57:0/1910593664 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56044d35ac10 0x56044d35b070 secure :-1 s=READY pgs=2217 cs=0 l=1 rev1=1 crypto rx=0x7fbf6c005c30 tx=0x7fbf6c005c60 comp rx=0 tx=0).stop 2023-10-19T14:18:44.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.148+0000 7fbf7e7bb740 1 -- 172.21.15.57:0/1910593664 shutdown_connections 2023-10-19T14:18:44.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.148+0000 7fbf7e7bb740 1 --2- 172.21.15.57:0/1910593664 >> [v2:172.21.15.57:6833/1674945162,v1:172.21.15.57:6835/1674945162] conn(0x56044d35d580 0x56044d3685a0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:44.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.148+0000 7fbf7e7bb740 1 --2- 172.21.15.57:0/1910593664 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56044d35ac10 0x56044d35b070 unknown :-1 s=CLOSED pgs=2217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:44.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.148+0000 7fbf7e7bb740 1 --2- 172.21.15.57:0/1910593664 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x56044d32cf20 0x56044d32ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:44.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.148+0000 7fbf54ff9640 0 client.11352 ms_handle_reset on v2:172.21.15.57:6833/1674945162 2023-10-19T14:18:44.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.148+0000 7fbf7e7bb740 1 --2- 172.21.15.57:0/1910593664 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56044d32cb40 0x56044d3421d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:44.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.148+0000 7fbf7e7bb740 1 -- 172.21.15.57:0/1910593664 >> 172.21.15.57:0/1910593664 conn(0x56044d300020 msgr2=0x56044d303b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:44.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.148+0000 7fbf7e7bb740 1 -- 172.21.15.57:0/1910593664 shutdown_connections 2023-10-19T14:18:44.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.148+0000 7fbf7e7bb740 1 -- 172.21.15.57:0/1910593664 wait complete. 2023-10-19T14:18:44.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.148+0000 7fbf7e7bb740 20 client.11352 trim_cache size 0 max 16384 2023-10-19T14:18:44.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.163+0000 7fbf7e7bb740 1 Processor -- start 2023-10-19T14:18:44.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.163+0000 7fbf7e7bb740 1 -- start start 2023-10-19T14:18:44.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.164+0000 7fbf7e7bb740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56044d32cb40 0x56044d34dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:44.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.164+0000 7fbf7e7bb740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56044d32cf20 0x56044d36c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:44.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.164+0000 7fbf7e7bb740 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x56044d35ac10 0x56044d3b2d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:44.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.164+0000 7fbf7e7bb740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x56044d1ec270 con 0x56044d32cf20 2023-10-19T14:18:44.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.164+0000 7fbf7e7bb740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x56044d35dba0 con 0x56044d32cb40 2023-10-19T14:18:44.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.164+0000 7fbf7e7bb740 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x56044d307040 con 0x56044d35ac10 2023-10-19T14:18:44.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.164+0000 7fbf791c1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56044d32cf20 0x56044d36c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:44.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.164+0000 7fbf791c1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56044d32cf20 0x56044d36c5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55664/0 (socket says 172.21.15.57:55664) 2023-10-19T14:18:44.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.164+0000 7fbf791c1640 1 -- 172.21.15.57:0/3777102723 learned_addr learned my addr 172.21.15.57:0/3777102723 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:18:44.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.164+0000 7fbf7a1c3640 1 --2- 172.21.15.57:0/3777102723 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x56044d35ac10 0x56044d3b2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:44.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.164+0000 7fbf799c2640 1 --2- 172.21.15.57:0/3777102723 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56044d32cb40 0x56044d34dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:44.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.164+0000 7fbf791c1640 1 -- 172.21.15.57:0/3777102723 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x56044d35ac10 msgr2=0x56044d3b2d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T14:18:44.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.164+0000 7fbf791c1640 1 --2- 172.21.15.57:0/3777102723 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x56044d35ac10 0x56044d3b2d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:44.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.164+0000 7fbf791c1640 1 -- 172.21.15.57:0/3777102723 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56044d32cb40 msgr2=0x56044d34dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T14:18:44.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.164+0000 7fbf791c1640 1 --2- 172.21.15.57:0/3777102723 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56044d32cb40 0x56044d34dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:44.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.164+0000 7fbf791c1640 1 -- 172.21.15.57:0/3777102723 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56044d208d00 con 0x56044d32cf20 2023-10-19T14:18:44.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.164+0000 7fbf799c2640 1 --2- 172.21.15.57:0/3777102723 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56044d32cb40 0x56044d34dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:18:44.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.164+0000 7fbf7a1c3640 1 --2- 172.21.15.57:0/3777102723 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x56044d35ac10 0x56044d3b2d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:18:44.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.165+0000 7fbf791c1640 1 --2- 172.21.15.57:0/3777102723 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56044d32cf20 0x56044d36c5c0 secure :-1 s=READY pgs=2218 cs=0 l=1 rev1=1 crypto rx=0x7fbf64054c70 tx=0x7fbf64058670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b9d16487dd03a7e server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:44.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.165+0000 7fbf57fff640 1 -- 172.21.15.57:0/3777102723 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf64088030 con 0x56044d32cf20 2023-10-19T14:18:44.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.165+0000 7fbf57fff640 1 -- 172.21.15.57:0/3777102723 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fbf64054e80 con 0x56044d32cf20 2023-10-19T14:18:44.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.165+0000 7fbf57fff640 1 -- 172.21.15.57:0/3777102723 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf64057ea0 con 0x56044d32cf20 2023-10-19T14:18:44.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.165+0000 7fbf7e7bb740 1 -- 172.21.15.57:0/3777102723 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56044d32cf20 msgr2=0x56044d36c5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:44.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.165+0000 7fbf7e7bb740 1 --2- 172.21.15.57:0/3777102723 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56044d32cf20 0x56044d36c5c0 secure :-1 s=READY pgs=2218 cs=0 l=1 rev1=1 crypto rx=0x7fbf64054c70 tx=0x7fbf64058670 comp rx=0 tx=0).stop 2023-10-19T14:18:44.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.165+0000 7fbf7e7bb740 1 -- 172.21.15.57:0/3777102723 shutdown_connections 2023-10-19T14:18:44.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.165+0000 7fbf7e7bb740 1 --2- 172.21.15.57:0/3777102723 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x56044d35ac10 0x56044d3b2d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:44.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.165+0000 7fbf7e7bb740 1 --2- 172.21.15.57:0/3777102723 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56044d32cf20 0x56044d36c5c0 unknown :-1 s=CLOSED pgs=2218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:44.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.165+0000 7fbf7e7bb740 1 --2- 172.21.15.57:0/3777102723 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56044d32cb40 0x56044d34dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:44.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.165+0000 7fbf7e7bb740 1 -- 172.21.15.57:0/3777102723 >> 172.21.15.57:0/3777102723 conn(0x56044d300020 msgr2=0x56044d32ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:44.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.165+0000 7fbf7e7bb740 1 -- 172.21.15.57:0/3777102723 shutdown_connections 2023-10-19T14:18:44.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.165+0000 7fbf7e7bb740 1 -- 172.21.15.57:0/3777102723 wait complete. 2023-10-19T14:18:44.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.166+0000 7fbf7e7bb740 1 -- start start 2023-10-19T14:18:44.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.166+0000 7fbf7e7bb740 1 Processor -- start 2023-10-19T14:18:44.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.167+0000 7fbf7e7bb740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56044d32cb40 0x56044d362ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:44.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.167+0000 7fbf7e7bb740 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x56044d32cf20 0x56044d3630e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:44.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.167+0000 7fbf7e7bb740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56044d35ac10 0x56044d327420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:44.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.167+0000 7fbf7e7bb740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x56044d34cf00 con 0x56044d32cb40 2023-10-19T14:18:44.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.167+0000 7fbf7e7bb740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x56044d363e10 con 0x56044d35ac10 2023-10-19T14:18:44.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.167+0000 7fbf7e7bb740 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x56044d3b9140 con 0x56044d32cf20 2023-10-19T14:18:44.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.167+0000 7fbf7e7bb740 10 client.0 _ll_register_callbacks cb 0x7fbf6c037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-19T14:18:44.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.167+0000 7fbf799c2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56044d32cb40 0x56044d362ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:44.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.167+0000 7fbf799c2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56044d32cb40 0x56044d362ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55676/0 (socket says 172.21.15.57:55676) 2023-10-19T14:18:44.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.167+0000 7fbf799c2640 1 -- 172.21.15.57:0/3574473217 learned_addr learned my addr 172.21.15.57:0/3574473217 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:18:44.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.167+0000 7fbf791c1640 1 --2- 172.21.15.57:0/3574473217 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x56044d32cf20 0x56044d3630e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:44.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.167+0000 7fbf7a1c3640 1 --2- 172.21.15.57:0/3574473217 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56044d35ac10 0x56044d327420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:44.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.167+0000 7fbf799c2640 1 -- 172.21.15.57:0/3574473217 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x56044d32cf20 msgr2=0x56044d3630e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T14:18:44.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.167+0000 7fbf799c2640 1 --2- 172.21.15.57:0/3574473217 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x56044d32cf20 0x56044d3630e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:44.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.167+0000 7fbf799c2640 1 -- 172.21.15.57:0/3574473217 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56044d35ac10 msgr2=0x56044d327420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T14:18:44.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.167+0000 7fbf799c2640 1 --2- 172.21.15.57:0/3574473217 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56044d35ac10 0x56044d327420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:44.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.167+0000 7fbf799c2640 1 -- 172.21.15.57:0/3574473217 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf741f2aa0 con 0x56044d32cb40 2023-10-19T14:18:44.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.167+0000 7fbf7a1c3640 1 --2- 172.21.15.57:0/3574473217 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56044d35ac10 0x56044d327420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:18:44.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.168+0000 7fbf799c2640 1 --2- 172.21.15.57:0/3574473217 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56044d32cb40 0x56044d362ba0 secure :-1 s=READY pgs=2219 cs=0 l=1 rev1=1 crypto rx=0x7fbf70007ca0 tx=0x7fbf70000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d0d1556f7b505c8 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:44.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.168+0000 7fbf54ff9640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2023-10-19T14:18:44.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.168+0000 7fbf54ff9640 1 -- 172.21.15.57:0/3574473217 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf70025e10 con 0x56044d32cb40 2023-10-19T14:18:44.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.168+0000 7fbf54ff9640 1 -- 172.21.15.57:0/3574473217 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(205..205 src has 1..205) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fbf70025040 con 0x56044d32cb40 2023-10-19T14:18:44.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.168+0000 7fbf7e7bb740 1 -- 172.21.15.57:0/3574473217 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbf6c003c60 con 0x56044d32cb40 2023-10-19T14:18:44.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.168+0000 7fbf54ff9640 1 -- 172.21.15.57:0/3574473217 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=83) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbf70025200 con 0x56044d32cb40 2023-10-19T14:18:44.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.168+0000 7fbf7e7bb740 10 client.11358 fetch_fsmap learned FSMap version 83 2023-10-19T14:18:44.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.168+0000 7fbf7e7bb740 1 -- 172.21.15.57:0/3574473217 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=83,monmap=2+}) v3 -- 0x56044d22e4f0 con 0x56044d32cb40 2023-10-19T14:18:44.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.168+0000 7fbf54ff9640 1 -- 172.21.15.57:0/3574473217 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fbf6c003c60 con 0x56044d32cb40 2023-10-19T14:18:44.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.168+0000 7fbf54ff9640 1 -- 172.21.15.57:0/3574473217 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 83) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fbf70020040 con 0x56044d32cb40 2023-10-19T14:18:44.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.168+0000 7fbf7e7bb740 10 client.11358 fetch_fsmap finished waiting for FSMap version 83 2023-10-19T14:18:44.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.168+0000 7fbf7e7bb740 10 client.11358 resolve_mds: resolved name 'a' to daemon mds.a 2023-10-19T14:18:44.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.168+0000 7fbf7e7bb740 20 client.11358 populate_metadata read hostname 'smithi057' 2023-10-19T14:18:44.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.168+0000 7fbf7e7bb740 1 --2- 172.21.15.57:0/3574473217 >> [v2:172.21.15.57:6833/1674945162,v1:172.21.15.57:6835/1674945162] conn(0x56044d327ab0 0x56044d327e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:44.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.168+0000 7fbf7e7bb740 4 client.11358 mds_command: new command op to 11217 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-19T14:18:44.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.168+0000 7fbf7e7bb740 1 -- 172.21.15.57:0/3574473217 --> [v2:172.21.15.57:6833/1674945162,v1:172.21.15.57:6835/1674945162] -- command(tid 0: {"prefix": "status"}) v1 -- 0x56044cf6d6c0 con 0x56044d327ab0 2023-10-19T14:18:44.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.168+0000 7fbf791c1640 1 --2- 172.21.15.57:0/3574473217 >> [v2:172.21.15.57:6833/1674945162,v1:172.21.15.57:6835/1674945162] conn(0x56044d327ab0 0x56044d327e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:44.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.169+0000 7fbf791c1640 1 --2- 172.21.15.57:0/3574473217 >> [v2:172.21.15.57:6833/1674945162,v1:172.21.15.57:6835/1674945162] conn(0x56044d327ab0 0x56044d327e90 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=70f9b3316ee1dcf0 server_cookie=71a1c0079e071578 in_seq=0 out_seq=0 2023-10-19T14:18:44.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.169+0000 7fbf54ff9640 10 client.11358 ms_handle_connect on v2:172.21.15.57:6833/1674945162 2023-10-19T14:18:44.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.169+0000 7fbf54ff9640 1 -- 172.21.15.57:0/3574473217 <== mds.0 v2:172.21.15.57:6833/1674945162 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+328 (crc 0 0 0) 0x56044cf6d6c0 con 0x56044d327ab0 2023-10-19T14:18:44.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.169+0000 7fbf54ff9640 10 client.11358 handle_command_reply: tid=0 2023-10-19T14:18:44.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.169+0000 7fbf7e7bb740 1 client.11358 shutdown 2023-10-19T14:18:44.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.169+0000 7fbf7e7bb740 1 -- 172.21.15.57:0/3574473217 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56044d32cb40 msgr2=0x56044d362ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:44.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.169+0000 7fbf7e7bb740 1 --2- 172.21.15.57:0/3574473217 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56044d32cb40 0x56044d362ba0 secure :-1 s=READY pgs=2219 cs=0 l=1 rev1=1 crypto rx=0x7fbf70007ca0 tx=0x7fbf70000ed0 comp rx=0 tx=0).stop 2023-10-19T14:18:44.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.169+0000 7fbf7e7bb740 1 -- 172.21.15.57:0/3574473217 shutdown_connections 2023-10-19T14:18:44.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.169+0000 7fbf7e7bb740 1 --2- 172.21.15.57:0/3574473217 >> [v2:172.21.15.57:6833/1674945162,v1:172.21.15.57:6835/1674945162] conn(0x56044d327ab0 0x56044d327e90 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:44.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.169+0000 7fbf7e7bb740 1 --2- 172.21.15.57:0/3574473217 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56044d35ac10 0x56044d327420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:44.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.169+0000 7fbf7e7bb740 1 --2- 172.21.15.57:0/3574473217 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x56044d32cf20 0x56044d3630e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:44.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.169+0000 7fbf54ff9640 0 client.11358 ms_handle_reset on v2:172.21.15.57:6833/1674945162 2023-10-19T14:18:44.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.169+0000 7fbf7e7bb740 1 --2- 172.21.15.57:0/3574473217 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56044d32cb40 0x56044d362ba0 unknown :-1 s=CLOSED pgs=2219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:44.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.169+0000 7fbf7e7bb740 1 -- 172.21.15.57:0/3574473217 >> 172.21.15.57:0/3574473217 conn(0x56044d300020 msgr2=0x56044d342270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:44.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.169+0000 7fbf7e7bb740 1 -- 172.21.15.57:0/3574473217 shutdown_connections 2023-10-19T14:18:44.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.169+0000 7fbf7e7bb740 1 -- 172.21.15.57:0/3574473217 wait complete. 2023-10-19T14:18:44.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.169+0000 7fbf7e7bb740 20 client.11358 trim_cache size 0 max 16384 2023-10-19T14:18:44.193 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2023-10-19T14:18:44.194 INFO:teuthology.orchestra.run.smithi057.stdout: "cluster_fsid": "1e9beb19-3260-4dd7-85ef-5b399ce774ac", 2023-10-19T14:18:44.194 INFO:teuthology.orchestra.run.smithi057.stdout: "whoami": 0, 2023-10-19T14:18:44.194 INFO:teuthology.orchestra.run.smithi057.stdout: "id": 11217, 2023-10-19T14:18:44.194 INFO:teuthology.orchestra.run.smithi057.stdout: "want_state": "up:active", 2023-10-19T14:18:44.194 INFO:teuthology.orchestra.run.smithi057.stdout: "state": "up:active", 2023-10-19T14:18:44.195 INFO:teuthology.orchestra.run.smithi057.stdout: "fs_name": "cephfs", 2023-10-19T14:18:44.195 INFO:teuthology.orchestra.run.smithi057.stdout: "rank_uptime": 4.7018354479999998, 2023-10-19T14:18:44.195 INFO:teuthology.orchestra.run.smithi057.stdout: "mdsmap_epoch": 83, 2023-10-19T14:18:44.195 INFO:teuthology.orchestra.run.smithi057.stdout: "osdmap_epoch": 205, 2023-10-19T14:18:44.195 INFO:teuthology.orchestra.run.smithi057.stdout: "osdmap_epoch_barrier": 205, 2023-10-19T14:18:44.196 INFO:teuthology.orchestra.run.smithi057.stdout: "uptime": 10.398862611 2023-10-19T14:18:44.196 INFO:teuthology.orchestra.run.smithi057.stdout:} 2023-10-19T14:18:44.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.170+0000 7fbf7bc26640 1 -- 172.21.15.57:0/731022855 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fbf5806af60 msgr2=0x7fbf5806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:44.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.170+0000 7fbf7bc26640 1 --2- 172.21.15.57:0/731022855 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fbf5806af60 0x7fbf5806d420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fbf7406f900 tx=0x7fbf70009040 comp rx=0 tx=0).stop 2023-10-19T14:18:44.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.170+0000 7fbf7bc26640 1 -- 172.21.15.57:0/731022855 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbf740cb980 msgr2=0x7fbf74135fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:44.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.170+0000 7fbf7bc26640 1 --2- 172.21.15.57:0/731022855 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbf740cb980 0x7fbf74135fc0 secure :-1 s=READY pgs=2215 cs=0 l=1 rev1=1 crypto rx=0x7fbf6400b410 tx=0x7fbf6400b8d0 comp rx=0 tx=0).stop 2023-10-19T14:18:44.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.170+0000 7fbf7bc26640 1 -- 172.21.15.57:0/731022855 shutdown_connections 2023-10-19T14:18:44.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.170+0000 7fbf7bc26640 1 --2- 172.21.15.57:0/731022855 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fbf5806af60 0x7fbf5806d420 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:44.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.170+0000 7fbf7bc26640 1 --2- 172.21.15.57:0/731022855 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbf74157a20 0x7fbf74140b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:44.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.170+0000 7fbf7bc26640 1 --2- 172.21.15.57:0/731022855 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbf740cb980 0x7fbf74135fc0 unknown :-1 s=CLOSED pgs=2215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:44.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.170+0000 7fbf7bc26640 1 --2- 172.21.15.57:0/731022855 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbf74070350 0x7fbf741f2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:44.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.170+0000 7fbf7bc26640 1 -- 172.21.15.57:0/731022855 >> 172.21.15.57:0/731022855 conn(0x7fbf74069930 msgr2=0x7fbf7406c6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:44.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.170+0000 7fbf7bc26640 1 -- 172.21.15.57:0/731022855 shutdown_connections 2023-10-19T14:18:44.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.170+0000 7fbf7bc26640 1 -- 172.21.15.57:0/731022855 wait complete. 2023-10-19T14:18:44.200 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-19T14:18:44.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.307+0000 7fac7677d640 1 -- 172.21.15.57:0/1823748858 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac58004d00 msgr2=0x7fac58005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:44.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.307+0000 7fac7677d640 1 --2- 172.21.15.57:0/1823748858 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac58004d00 0x7fac58005180 secure :-1 s=READY pgs=2221 cs=0 l=1 rev1=1 crypto rx=0x7fac68009ed0 tx=0x7fac580057f0 comp rx=0 tx=0).stop 2023-10-19T14:18:44.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.307+0000 7fac74d1a640 1 -- 172.21.15.57:0/1823748858 reap_dead start 2023-10-19T14:18:44.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.307+0000 7fac7677d640 1 -- 172.21.15.57:0/1823748858 shutdown_connections 2023-10-19T14:18:44.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.307+0000 7fac7677d640 1 -- 172.21.15.57:0/1823748858 >> 172.21.15.57:0/1823748858 conn(0x7fac70069930 msgr2=0x7fac70069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:44.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.307+0000 7fac7677d640 1 -- 172.21.15.57:0/1823748858 shutdown_connections 2023-10-19T14:18:44.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.307+0000 7fac7677d640 1 -- 172.21.15.57:0/1823748858 wait complete. 2023-10-19T14:18:44.313 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.308+0000 7fac7677d640 1 Processor -- start 2023-10-19T14:18:44.314 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.308+0000 7fac7677d640 1 -- start start 2023-10-19T14:18:44.314 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.309+0000 7fac7677d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac70148eb0 0x7fac7014ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:44.314 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.309+0000 7fac7677d640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fac7015a410 0x7fac7014bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:44.314 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.309+0000 7fac7677d640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fac7015ae00 0x7fac7015b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:44.314 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.309+0000 7fac7677d640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac7014b810 con 0x7fac70148eb0 2023-10-19T14:18:44.315 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.309+0000 7fac7677d640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac7015b7c0 con 0x7fac7015a410 2023-10-19T14:18:44.315 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.309+0000 7fac7677d640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac7014af60 con 0x7fac7015ae00 2023-10-19T14:18:44.315 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.309+0000 7fac6ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac70148eb0 0x7fac7014ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:44.315 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.309+0000 7fac6ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac70148eb0 0x7fac7014ce20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55694/0 (socket says 172.21.15.57:55694) 2023-10-19T14:18:44.316 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.309+0000 7fac6ffff640 1 -- 172.21.15.57:0/2400530388 learned_addr learned my addr 172.21.15.57:0/2400530388 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:18:44.316 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.309+0000 7fac6f7fe640 1 --2- 172.21.15.57:0/2400530388 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fac7015a410 0x7fac7014bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:44.316 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.309+0000 7fac74d1a640 1 --2- 172.21.15.57:0/2400530388 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fac7015ae00 0x7fac7015b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:44.316 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.309+0000 7fac6ffff640 1 -- 172.21.15.57:0/2400530388 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fac7015ae00 msgr2=0x7fac7015b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:44.316 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.309+0000 7fac6ffff640 1 --2- 172.21.15.57:0/2400530388 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fac7015ae00 0x7fac7015b280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:44.316 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.309+0000 7fac6ffff640 1 -- 172.21.15.57:0/2400530388 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fac7015a410 msgr2=0x7fac7014bc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:44.317 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.309+0000 7fac6ffff640 1 --2- 172.21.15.57:0/2400530388 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fac7015a410 0x7fac7014bc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:44.317 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.309+0000 7fac6ffff640 1 -- 172.21.15.57:0/2400530388 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac7015ab80 con 0x7fac70148eb0 2023-10-19T14:18:44.317 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.310+0000 7fac74d1a640 1 --2- 172.21.15.57:0/2400530388 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fac7015ae00 0x7fac7015b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:18:44.317 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.310+0000 7fac6f7fe640 1 --2- 172.21.15.57:0/2400530388 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fac7015a410 0x7fac7014bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:18:44.317 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.310+0000 7fac6ffff640 1 --2- 172.21.15.57:0/2400530388 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac70148eb0 0x7fac7014ce20 secure :-1 s=READY pgs=2222 cs=0 l=1 rev1=1 crypto rx=0x7fac6000e6f0 tx=0x7fac6000ebb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:44.318 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.310+0000 7fac6cff9640 1 -- 172.21.15.57:0/2400530388 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac6000ca40 con 0x7fac70148eb0 2023-10-19T14:18:44.318 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.310+0000 7fac6cff9640 1 -- 172.21.15.57:0/2400530388 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fac6000cc90 con 0x7fac70148eb0 2023-10-19T14:18:44.318 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.310+0000 7fac7677d640 1 -- 172.21.15.57:0/2400530388 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac7014b190 con 0x7fac70148eb0 2023-10-19T14:18:44.318 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.310+0000 7fac6cff9640 1 -- 172.21.15.57:0/2400530388 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac600116c0 con 0x7fac70148eb0 2023-10-19T14:18:44.318 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.310+0000 7fac7677d640 1 -- 172.21.15.57:0/2400530388 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac7014b430 con 0x7fac70148eb0 2023-10-19T14:18:44.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.311+0000 7fac7677d640 1 -- 172.21.15.57:0/2400530388 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac7014ba80 con 0x7fac70148eb0 2023-10-19T14:18:44.320 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.311+0000 7fac6cff9640 1 -- 172.21.15.57:0/2400530388 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 117) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fac60011820 con 0x7fac70148eb0 2023-10-19T14:18:44.320 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.312+0000 7fac6cff9640 1 --2- 172.21.15.57:0/2400530388 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fac5c06af60 0x7fac5c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:44.320 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.312+0000 7fac6cff9640 1 -- 172.21.15.57:0/2400530388 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fac6008a2e0 con 0x7fac70148eb0 2023-10-19T14:18:44.320 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.312+0000 7fac6f7fe640 1 --2- 172.21.15.57:0/2400530388 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fac5c06af60 0x7fac5c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:44.320 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.312+0000 7fac6f7fe640 1 --2- 172.21.15.57:0/2400530388 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fac5c06af60 0x7fac5c06d420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fac70150790 tx=0x7fac64005f90 comp rx=0 tx=0).ready entity=mgr.8744 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:44.321 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.316+0000 7fac6cff9640 1 -- 172.21.15.57:0/2400530388 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fac60055010 con 0x7fac70148eb0 2023-10-19T14:18:44.446 INFO:tasks.ceph.mds.a.smithi057.stderr:2023-10-19T14:18:44.444+0000 7f7d88e3c640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-19T14:18:44.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.528+0000 7fac7677d640 1 -- 172.21.15.57:0/2400530388 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fac700690e0 con 0x7fac70148eb0 2023-10-19T14:18:44.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.529+0000 7fac6cff9640 1 -- 172.21.15.57:0/2400530388 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v83) v1 ==== 53+0+73 (secure 0 0 0) 0x7fac60058630 con 0x7fac70148eb0 2023-10-19T14:18:44.532 INFO:teuthology.orchestra.run.smithi057.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-19T14:18:44.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.531+0000 7fac7677d640 1 -- 172.21.15.57:0/2400530388 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fac5c06af60 msgr2=0x7fac5c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:44.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.531+0000 7fac7677d640 1 --2- 172.21.15.57:0/2400530388 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fac5c06af60 0x7fac5c06d420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fac70150790 tx=0x7fac64005f90 comp rx=0 tx=0).stop 2023-10-19T14:18:44.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.531+0000 7fac7677d640 1 -- 172.21.15.57:0/2400530388 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac70148eb0 msgr2=0x7fac7014ce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:44.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.531+0000 7fac7677d640 1 --2- 172.21.15.57:0/2400530388 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac70148eb0 0x7fac7014ce20 secure :-1 s=READY pgs=2222 cs=0 l=1 rev1=1 crypto rx=0x7fac6000e6f0 tx=0x7fac6000ebb0 comp rx=0 tx=0).stop 2023-10-19T14:18:44.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.531+0000 7fac7677d640 1 -- 172.21.15.57:0/2400530388 shutdown_connections 2023-10-19T14:18:44.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.531+0000 7fac7677d640 1 --2- 172.21.15.57:0/2400530388 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fac5c06af60 0x7fac5c06d420 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:44.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.531+0000 7fac7677d640 1 --2- 172.21.15.57:0/2400530388 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fac7015ae00 0x7fac7015b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:44.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.531+0000 7fac7677d640 1 --2- 172.21.15.57:0/2400530388 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fac7015a410 0x7fac7014bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:44.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.531+0000 7fac7677d640 1 --2- 172.21.15.57:0/2400530388 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac70148eb0 0x7fac7014ce20 unknown :-1 s=CLOSED pgs=2222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:44.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.531+0000 7fac7677d640 1 -- 172.21.15.57:0/2400530388 >> 172.21.15.57:0/2400530388 conn(0x7fac70069930 msgr2=0x7fac70069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:44.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.531+0000 7fac7677d640 1 -- 172.21.15.57:0/2400530388 shutdown_connections 2023-10-19T14:18:44.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:44.531+0000 7fac7677d640 1 -- 172.21.15.57:0/2400530388 wait complete. 2023-10-19T14:18:44.544 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-10-19T14:18:44.544 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-19T14:18:44.545 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi057.front.sepia.ceph.com 2023-10-19T14:18:44.545 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-19T14:18:44.545 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-19T14:18:44.546 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-10-19T14:18:44.546 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-10-19T14:18:44.546 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-19T14:18:44.547 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-19T14:18:44.547 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:18:44.548 DEBUG:teuthology.orchestra.run.smithi057:> ip addr 2023-10-19T14:18:44.564 INFO:teuthology.orchestra.run.smithi057.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-19T14:18:44.564 INFO:teuthology.orchestra.run.smithi057.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-19T14:18:44.564 INFO:teuthology.orchestra.run.smithi057.stdout: inet 127.0.0.1/8 scope host lo 2023-10-19T14:18:44.564 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T14:18:44.565 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 ::1/128 scope host 2023-10-19T14:18:44.565 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T14:18:44.565 INFO:teuthology.orchestra.run.smithi057.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-19T14:18:44.565 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:6b:ff:86 brd ff:ff:ff:ff:ff:ff 2023-10-19T14:18:44.566 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp7s0f0 2023-10-19T14:18:44.566 INFO:teuthology.orchestra.run.smithi057.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-19T14:18:44.566 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:6b:ff:87 brd ff:ff:ff:ff:ff:ff 2023-10-19T14:18:44.566 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp7s0f1 2023-10-19T14:18:44.567 INFO:teuthology.orchestra.run.smithi057.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-19T14:18:44.567 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:bd:15:62 brd ff:ff:ff:ff:ff:ff 2023-10-19T14:18:44.567 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp1s0f0 2023-10-19T14:18:44.567 INFO:teuthology.orchestra.run.smithi057.stdout: inet 172.21.15.57/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-19T14:18:44.568 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft 36986sec preferred_lft 36986sec 2023-10-19T14:18:44.568 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 fe80::ec4:7aff:febd:1562/64 scope link noprefixroute 2023-10-19T14:18:44.568 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T14:18:44.568 INFO:teuthology.orchestra.run.smithi057.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-19T14:18:44.569 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:bd:15:63 brd ff:ff:ff:ff:ff:ff 2023-10-19T14:18:44.569 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp1s0f1 2023-10-19T14:18:44.569 INFO:teuthology.orchestra.run.smithi057.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-19T14:18:44.570 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0e:fe:f1:b5:41:59 brd ff:ff:ff:ff:ff:ff 2023-10-19T14:18:44.570 INFO:teuthology.orchestra.run.smithi057.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-19T14:18:44.570 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T14:18:44.570 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 fe80::64b1:a2ff:feb3:b08/64 scope link 2023-10-19T14:18:44.570 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T14:18:44.571 INFO:teuthology.orchestra.run.smithi057.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-19T14:18:44.571 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0e:fe:f1:b5:41:59 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T14:18:44.571 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 fe80::cfe:f1ff:feb5:4159/64 scope link 2023-10-19T14:18:44.571 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T14:18:44.571 INFO:teuthology.orchestra.run.smithi057.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-19T14:18:44.572 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 62:53:72:9d:52:71 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-19T14:18:44.572 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 fe80::6053:72ff:fe9d:5271/64 scope link 2023-10-19T14:18:44.572 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T14:18:44.573 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:18:44.573 DEBUG:teuthology.orchestra.run.smithi057:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-19T14:18:44.633 INFO:teuthology.orchestra.run.smithi057.stdout:1 2023-10-19T14:18:44.635 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:18:44.635 DEBUG:teuthology.orchestra.run.smithi057:> route 2023-10-19T14:18:44.691 INFO:teuthology.orchestra.run.smithi057.stdout:Kernel IP routing table 2023-10-19T14:18:44.691 INFO:teuthology.orchestra.run.smithi057.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-19T14:18:44.691 INFO:teuthology.orchestra.run.smithi057.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-19T14:18:44.691 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-19T14:18:44.692 INFO:teuthology.orchestra.run.smithi057.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-19T14:18:44.692 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:18:44.693 DEBUG:teuthology.orchestra.run.smithi057:> (cd / && exec bash -c ' 2023-10-19T14:18:44.693 DEBUG:teuthology.orchestra.run.smithi057:> set -e 2023-10-19T14:18:44.694 DEBUG:teuthology.orchestra.run.smithi057:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-19T14:18:44.694 DEBUG:teuthology.orchestra.run.smithi057:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-19T14:18:44.695 DEBUG:teuthology.orchestra.run.smithi057:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-19T14:18:44.695 DEBUG:teuthology.orchestra.run.smithi057:> ') 2023-10-19T14:18:44.796 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:18:44.796 DEBUG:teuthology.orchestra.run.smithi057:> ip netns list 2023-10-19T14:18:44.811 INFO:teuthology.orchestra.run.smithi057.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-19T14:18:44.811 INFO:teuthology.orchestra.run.smithi057.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-19T14:18:44.812 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-19T14:18:44.812 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:18:44.813 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip link set brx.0 up 2023-10-19T14:18:44.882 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-10-19T14:18:44.883 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-19T14:18:44.883 DEBUG:teuthology.orchestra.run.smithi057:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-19T14:18:44.938 INFO:teuthology.orchestra.run.smithi057.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-19T14:18:44.938 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-19T14:18:44.939 DEBUG:teuthology.orchestra.run.smithi057:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-19T14:18:44.993 DEBUG:teuthology.orchestra.run.smithi057:> sudo modprobe fuse 2023-10-19T14:18:45.064 DEBUG:teuthology.orchestra.run.smithi057:> cat /proc/self/mounts | awk '{print $2}' 2023-10-19T14:18:45.120 INFO:teuthology.orchestra.run.smithi057.stdout:/proc 2023-10-19T14:18:45.121 INFO:teuthology.orchestra.run.smithi057.stdout:/sys 2023-10-19T14:18:45.121 INFO:teuthology.orchestra.run.smithi057.stdout:/dev 2023-10-19T14:18:45.121 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/security 2023-10-19T14:18:45.121 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/shm 2023-10-19T14:18:45.122 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/pts 2023-10-19T14:18:45.122 INFO:teuthology.orchestra.run.smithi057.stdout:/run 2023-10-19T14:18:45.122 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/cgroup 2023-10-19T14:18:45.122 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/pstore 2023-10-19T14:18:45.123 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/bpf 2023-10-19T14:18:45.123 INFO:teuthology.orchestra.run.smithi057.stdout:/ 2023-10-19T14:18:45.123 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/selinux 2023-10-19T14:18:45.123 INFO:teuthology.orchestra.run.smithi057.stdout:/proc/sys/fs/binfmt_misc 2023-10-19T14:18:45.123 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/hugepages 2023-10-19T14:18:45.124 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/mqueue 2023-10-19T14:18:45.124 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/debug 2023-10-19T14:18:45.124 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/tracing 2023-10-19T14:18:45.124 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/fuse/connections 2023-10-19T14:18:45.124 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/config 2023-10-19T14:18:45.125 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-sysctl.service 2023-10-19T14:18:45.125 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-10-19T14:18:45.125 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-10-19T14:18:45.125 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/nfs/rpc_pipefs 2023-10-19T14:18:45.125 INFO:teuthology.orchestra.run.smithi057.stdout:/run/user/1000 2023-10-19T14:18:45.126 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph 2023-10-19T14:18:45.126 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-0 2023-10-19T14:18:45.126 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-1 2023-10-19T14:18:45.126 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-2 2023-10-19T14:18:45.126 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-3 2023-10-19T14:18:45.127 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns 2023-10-19T14:18:45.127 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T14:18:45.127 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T14:18:45.127 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-19T14:18:45.127 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-19T14:18:45.128 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:18:45.128 DEBUG:teuthology.orchestra.run.smithi057:> ls /sys/fs/fuse/connections 2023-10-19T14:18:45.175 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-10-19T14:18:45.176 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-10-19T14:18:45.218 DEBUG:teuthology.orchestra.run.smithi057:> sudo modprobe fuse 2023-10-19T14:18:45.249 DEBUG:teuthology.orchestra.run.smithi057:> cat /proc/self/mounts | awk '{print $2}' 2023-10-19T14:18:45.304 INFO:teuthology.orchestra.run.smithi057.stdout:/proc 2023-10-19T14:18:45.304 INFO:teuthology.orchestra.run.smithi057.stdout:/sys 2023-10-19T14:18:45.305 INFO:teuthology.orchestra.run.smithi057.stdout:/dev 2023-10-19T14:18:45.305 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/security 2023-10-19T14:18:45.305 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/shm 2023-10-19T14:18:45.305 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/pts 2023-10-19T14:18:45.305 INFO:teuthology.orchestra.run.smithi057.stdout:/run 2023-10-19T14:18:45.306 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/cgroup 2023-10-19T14:18:45.306 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/pstore 2023-10-19T14:18:45.306 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/bpf 2023-10-19T14:18:45.306 INFO:teuthology.orchestra.run.smithi057.stdout:/ 2023-10-19T14:18:45.307 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/selinux 2023-10-19T14:18:45.307 INFO:teuthology.orchestra.run.smithi057.stdout:/proc/sys/fs/binfmt_misc 2023-10-19T14:18:45.307 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/hugepages 2023-10-19T14:18:45.307 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/mqueue 2023-10-19T14:18:45.307 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/debug 2023-10-19T14:18:45.307 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/tracing 2023-10-19T14:18:45.308 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/fuse/connections 2023-10-19T14:18:45.308 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/config 2023-10-19T14:18:45.308 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-sysctl.service 2023-10-19T14:18:45.308 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-10-19T14:18:45.308 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-10-19T14:18:45.309 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/nfs/rpc_pipefs 2023-10-19T14:18:45.309 INFO:teuthology.orchestra.run.smithi057.stdout:/run/user/1000 2023-10-19T14:18:45.309 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph 2023-10-19T14:18:45.309 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-0 2023-10-19T14:18:45.309 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-1 2023-10-19T14:18:45.310 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-2 2023-10-19T14:18:45.310 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-3 2023-10-19T14:18:45.310 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns 2023-10-19T14:18:45.310 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T14:18:45.310 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T14:18:45.311 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-19T14:18:45.311 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-19T14:18:45.311 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:18:45.312 DEBUG:teuthology.orchestra.run.smithi057:> ls /sys/fs/fuse/connections 2023-10-19T14:18:45.313 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi057.stderr:2023-10-19T14:18:45.308+0000 7f01729fb480 -1 init, newargv = 0x55a9a4bcd910 newargc=15 2023-10-19T14:18:45.313 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi057.stderr:ceph-fuse[149328]: starting ceph client 2023-10-19T14:18:45.315 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi057.stderr:ceph-fuse[149328]: starting fuse 2023-10-19T14:18:45.327 INFO:teuthology.orchestra.run.smithi057.stdout:43 2023-10-19T14:18:45.328 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2023-10-19T14:18:45.328 DEBUG:teuthology.orchestra.run.smithi057:> sudo stdin-killer -- python3 -c ' 2023-10-19T14:18:45.328 DEBUG:teuthology.orchestra.run.smithi057:> import glob 2023-10-19T14:18:45.329 DEBUG:teuthology.orchestra.run.smithi057:> import re 2023-10-19T14:18:45.329 DEBUG:teuthology.orchestra.run.smithi057:> import os 2023-10-19T14:18:45.330 DEBUG:teuthology.orchestra.run.smithi057:> import subprocess 2023-10-19T14:18:45.330 DEBUG:teuthology.orchestra.run.smithi057:> 2023-10-19T14:18:45.330 DEBUG:teuthology.orchestra.run.smithi057:> def _find_admin_socket(client_name): 2023-10-19T14:18:45.331 DEBUG:teuthology.orchestra.run.smithi057:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-10-19T14:18:45.331 DEBUG:teuthology.orchestra.run.smithi057:> files = glob.glob(asok_path) 2023-10-19T14:18:45.332 DEBUG:teuthology.orchestra.run.smithi057:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-10-19T14:18:45.332 DEBUG:teuthology.orchestra.run.smithi057:> 2023-10-19T14:18:45.332 DEBUG:teuthology.orchestra.run.smithi057:> # Given a non-glob path, it better be there 2023-10-19T14:18:45.333 DEBUG:teuthology.orchestra.run.smithi057:> if "*" not in asok_path: 2023-10-19T14:18:45.333 DEBUG:teuthology.orchestra.run.smithi057:> assert(len(files) == 1) 2023-10-19T14:18:45.333 DEBUG:teuthology.orchestra.run.smithi057:> return files[0] 2023-10-19T14:18:45.334 DEBUG:teuthology.orchestra.run.smithi057:> 2023-10-19T14:18:45.334 DEBUG:teuthology.orchestra.run.smithi057:> for f in files: 2023-10-19T14:18:45.335 DEBUG:teuthology.orchestra.run.smithi057:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-10-19T14:18:45.335 DEBUG:teuthology.orchestra.run.smithi057:> if os.path.exists("/proc/{0}".format(pid)): 2023-10-19T14:18:45.335 DEBUG:teuthology.orchestra.run.smithi057:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-10-19T14:18:45.336 DEBUG:teuthology.orchestra.run.smithi057:> contents = proc_f.read() 2023-10-19T14:18:45.336 DEBUG:teuthology.orchestra.run.smithi057:> if mountpoint in contents: 2023-10-19T14:18:45.337 DEBUG:teuthology.orchestra.run.smithi057:> return f 2023-10-19T14:18:45.337 DEBUG:teuthology.orchestra.run.smithi057:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-10-19T14:18:45.337 DEBUG:teuthology.orchestra.run.smithi057:> 2023-10-19T14:18:45.338 DEBUG:teuthology.orchestra.run.smithi057:> print(_find_admin_socket("client.0")) 2023-10-19T14:18:45.338 DEBUG:teuthology.orchestra.run.smithi057:> ' 2023-10-19T14:18:45.452 INFO:teuthology.orchestra.run.smithi057.stdout:/var/run/ceph/ceph-client.0.149328.asok 2023-10-19T14:18:45.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2023-10-19T14:18:45.462 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.149328.asok 2023-10-19T14:18:45.463 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:18:45.464 DEBUG:teuthology.orchestra.run.smithi057:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.149328.asok status 2023-10-19T14:18:45.604 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2023-10-19T14:18:45.604 INFO:teuthology.orchestra.run.smithi057.stdout: "metadata": { 2023-10-19T14:18:45.604 INFO:teuthology.orchestra.run.smithi057.stdout: "ceph_sha1": "f7420d7f8e55ceea786dcbf536c338ce920c21eb", 2023-10-19T14:18:45.605 INFO:teuthology.orchestra.run.smithi057.stdout: "ceph_version": "ceph version 18.0.0-6786-gf7420d7f (f7420d7f8e55ceea786dcbf536c338ce920c21eb) reef (dev)", 2023-10-19T14:18:45.605 INFO:teuthology.orchestra.run.smithi057.stdout: "entity_id": "0", 2023-10-19T14:18:45.605 INFO:teuthology.orchestra.run.smithi057.stdout: "hostname": "smithi057", 2023-10-19T14:18:45.605 INFO:teuthology.orchestra.run.smithi057.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-10-19T14:18:45.605 INFO:teuthology.orchestra.run.smithi057.stdout: "pid": "149328", 2023-10-19T14:18:45.606 INFO:teuthology.orchestra.run.smithi057.stdout: "root": "/" 2023-10-19T14:18:45.606 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:18:45.606 INFO:teuthology.orchestra.run.smithi057.stdout: "dentry_count": 0, 2023-10-19T14:18:45.606 INFO:teuthology.orchestra.run.smithi057.stdout: "dentry_pinned_count": 0, 2023-10-19T14:18:45.606 INFO:teuthology.orchestra.run.smithi057.stdout: "id": 11370, 2023-10-19T14:18:45.607 INFO:teuthology.orchestra.run.smithi057.stdout: "inst": { 2023-10-19T14:18:45.607 INFO:teuthology.orchestra.run.smithi057.stdout: "name": { 2023-10-19T14:18:45.607 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "client", 2023-10-19T14:18:45.607 INFO:teuthology.orchestra.run.smithi057.stdout: "num": 11370 2023-10-19T14:18:45.607 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:18:45.608 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": { 2023-10-19T14:18:45.608 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "v1", 2023-10-19T14:18:45.608 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": "192.168.0.1:0", 2023-10-19T14:18:45.608 INFO:teuthology.orchestra.run.smithi057.stdout: "nonce": 2505102404 2023-10-19T14:18:45.608 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:18:45.609 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:18:45.609 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": { 2023-10-19T14:18:45.609 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "v1", 2023-10-19T14:18:45.609 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": "192.168.0.1:0", 2023-10-19T14:18:45.609 INFO:teuthology.orchestra.run.smithi057.stdout: "nonce": 2505102404 2023-10-19T14:18:45.609 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:18:45.610 INFO:teuthology.orchestra.run.smithi057.stdout: "inst_str": "client.11370 192.168.0.1:0/2505102404", 2023-10-19T14:18:45.610 INFO:teuthology.orchestra.run.smithi057.stdout: "addr_str": "192.168.0.1:0/2505102404", 2023-10-19T14:18:45.610 INFO:teuthology.orchestra.run.smithi057.stdout: "inode_count": 1, 2023-10-19T14:18:45.610 INFO:teuthology.orchestra.run.smithi057.stdout: "mds_epoch": 83, 2023-10-19T14:18:45.611 INFO:teuthology.orchestra.run.smithi057.stdout: "osd_epoch": 205, 2023-10-19T14:18:45.611 INFO:teuthology.orchestra.run.smithi057.stdout: "osd_epoch_barrier": 0, 2023-10-19T14:18:45.611 INFO:teuthology.orchestra.run.smithi057.stdout: "blocklisted": false, 2023-10-19T14:18:45.611 INFO:teuthology.orchestra.run.smithi057.stdout: "fs_name": "cephfs" 2023-10-19T14:18:45.611 INFO:teuthology.orchestra.run.smithi057.stdout:} 2023-10-19T14:18:45.612 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:18:45.613 DEBUG:teuthology.orchestra.run.smithi057:> stat --file-system '--printf=%T 2023-10-19T14:18:45.613 DEBUG:teuthology.orchestra.run.smithi057:> ' -- /home/ubuntu/cephtest/mnt.0 2023-10-19T14:18:45.629 INFO:teuthology.orchestra.run.smithi057.stdout:fuseblk 2023-10-19T14:18:45.629 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-10-19T14:18:45.630 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:18:45.630 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-19T14:18:45.701 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-19T14:18:45.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:45.847+0000 7fe42d04b640 1 -- 172.21.15.57:0/2434039875 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe4280b0530 msgr2=0x7fe4280b09a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:45.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:45.847+0000 7fe42d04b640 1 --2- 172.21.15.57:0/2434039875 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe4280b0530 0x7fe4280b09a0 secure :-1 s=READY pgs=2226 cs=0 l=1 rev1=1 crypto rx=0x7fe4280cc490 tx=0x7fe42002c260 comp rx=0 tx=0).stop 2023-10-19T14:18:45.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:45.847+0000 7fe42d04b640 1 -- 172.21.15.57:0/2434039875 shutdown_connections 2023-10-19T14:18:45.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:45.847+0000 7fe42d04b640 1 --2- 172.21.15.57:0/2434039875 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe4280b0530 0x7fe4280b09a0 unknown :-1 s=CLOSED pgs=2226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:45.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:45.847+0000 7fe42d04b640 1 --2- 172.21.15.57:0/2434039875 >> v2:172.21.15.179:3300/0 conn(0x7fe4280afa20 0x7fe428158300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:45.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:45.847+0000 7fe42d04b640 1 -- 172.21.15.57:0/2434039875 >> 172.21.15.57:0/2434039875 conn(0x7fe428069930 msgr2=0x7fe428069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:45.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:45.847+0000 7fe42d04b640 1 -- 172.21.15.57:0/2434039875 shutdown_connections 2023-10-19T14:18:45.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:45.847+0000 7fe42d04b640 1 -- 172.21.15.57:0/2434039875 wait complete. 2023-10-19T14:18:45.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:45.849+0000 7fe42d04b640 1 Processor -- start 2023-10-19T14:18:45.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:45.849+0000 7fe42d04b640 1 -- start start 2023-10-19T14:18:45.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:45.849+0000 7fe42d04b640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe4280afa20 0x7fe4280cb530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:45.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:45.849+0000 7fe42d04b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe4280b0530 0x7fe428144710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:45.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:45.849+0000 7fe42d04b640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe4280cbd20 0x7fe4281f2040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:45.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:45.849+0000 7fe42d04b640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe428147e70 con 0x7fe4280b0530 2023-10-19T14:18:45.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:45.850+0000 7fe42d04b640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4281f2580 con 0x7fe4280afa20 2023-10-19T14:18:45.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:45.850+0000 7fe42d04b640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe428144da0 con 0x7fe4280cbd20 2023-10-19T14:18:45.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:45.850+0000 7fe42659c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe4280b0530 0x7fe428144710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:45.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:45.850+0000 7fe42659c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe4280b0530 0x7fe428144710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55710/0 (socket says 172.21.15.57:55710) 2023-10-19T14:18:45.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:45.850+0000 7fe42659c640 1 -- 172.21.15.57:0/1542145460 learned_addr learned my addr 172.21.15.57:0/1542145460 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:18:45.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:45.850+0000 7fe42759e640 1 --2- 172.21.15.57:0/1542145460 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe4280cbd20 0x7fe4281f2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:45.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:45.850+0000 7fe426d9d640 1 --2- 172.21.15.57:0/1542145460 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe4280afa20 0x7fe4280cb530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:45.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:45.850+0000 7fe42659c640 1 -- 172.21.15.57:0/1542145460 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe4280cbd20 msgr2=0x7fe4281f2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:45.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:45.850+0000 7fe42659c640 1 --2- 172.21.15.57:0/1542145460 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe4280cbd20 0x7fe4281f2040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:45.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:45.850+0000 7fe42659c640 1 -- 172.21.15.57:0/1542145460 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe4280afa20 msgr2=0x7fe4280cb530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:45.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:45.850+0000 7fe42659c640 1 --2- 172.21.15.57:0/1542145460 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe4280afa20 0x7fe4280cb530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:45.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:45.850+0000 7fe42659c640 1 -- 172.21.15.57:0/1542145460 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe420009900 con 0x7fe4280b0530 2023-10-19T14:18:45.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:45.850+0000 7fe426d9d640 1 --2- 172.21.15.57:0/1542145460 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe4280afa20 0x7fe4280cb530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:18:45.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:45.850+0000 7fe42659c640 1 --2- 172.21.15.57:0/1542145460 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe4280b0530 0x7fe428144710 secure :-1 s=READY pgs=2227 cs=0 l=1 rev1=1 crypto rx=0x7fe41c00b4a0 tx=0x7fe41c00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:45.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:45.850+0000 7fe40f7fe640 1 -- 172.21.15.57:0/1542145460 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe41c010610 con 0x7fe4280b0530 2023-10-19T14:18:45.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:45.850+0000 7fe40f7fe640 1 -- 172.21.15.57:0/1542145460 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fe41c010860 con 0x7fe4280b0530 2023-10-19T14:18:45.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:45.850+0000 7fe42d04b640 1 -- 172.21.15.57:0/1542145460 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe428145080 con 0x7fe4280b0530 2023-10-19T14:18:45.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:45.850+0000 7fe40f7fe640 1 -- 172.21.15.57:0/1542145460 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe41c010c70 con 0x7fe4280b0530 2023-10-19T14:18:45.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:45.850+0000 7fe42d04b640 1 -- 172.21.15.57:0/1542145460 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe428145640 con 0x7fe4280b0530 2023-10-19T14:18:45.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:45.851+0000 7fe42d04b640 1 -- 172.21.15.57:0/1542145460 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4280690e0 con 0x7fe4280b0530 2023-10-19T14:18:45.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:45.852+0000 7fe40f7fe640 1 -- 172.21.15.57:0/1542145460 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 117) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe41c017360 con 0x7fe4280b0530 2023-10-19T14:18:45.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:45.852+0000 7fe40f7fe640 1 --2- 172.21.15.57:0/1542145460 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fe41406af60 0x7fe41406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:45.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:45.853+0000 7fe40f7fe640 1 -- 172.21.15.57:0/1542145460 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fe41c0896f0 con 0x7fe4280b0530 2023-10-19T14:18:45.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:45.853+0000 7fe426d9d640 1 --2- 172.21.15.57:0/1542145460 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fe41406af60 0x7fe41406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:45.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:45.853+0000 7fe426d9d640 1 --2- 172.21.15.57:0/1542145460 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fe41406af60 0x7fe41406d420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe418006fd0 tx=0x7fe418008040 comp rx=0 tx=0).ready entity=mgr.8744 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:45.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:45.857+0000 7fe40f7fe640 1 -- 172.21.15.57:0/1542145460 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe41c058070 con 0x7fe4280b0530 2023-10-19T14:18:46.058 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.055+0000 7fe42d04b640 1 -- 172.21.15.57:0/1542145460 --> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fe4280cc100 con 0x7fe41406af60 2023-10-19T14:18:46.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.057+0000 7fe40f7fe640 1 -- 172.21.15.57:0/1542145460 <== mgr.8744 v2:172.21.15.179:6832/2730461479 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fe418009e00 con 0x7fe41406af60 2023-10-19T14:18:46.060 INFO:teuthology.orchestra.run.smithi057.stdout:[ 2023-10-19T14:18:46.061 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:18:46.061 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "cephfs" 2023-10-19T14:18:46.061 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:18:46.062 INFO:teuthology.orchestra.run.smithi057.stdout:] 2023-10-19T14:18:46.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.059+0000 7fe42d04b640 1 -- 172.21.15.57:0/1542145460 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fe41406af60 msgr2=0x7fe41406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:46.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.059+0000 7fe42d04b640 1 --2- 172.21.15.57:0/1542145460 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fe41406af60 0x7fe41406d420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe418006fd0 tx=0x7fe418008040 comp rx=0 tx=0).stop 2023-10-19T14:18:46.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.059+0000 7fe42d04b640 1 -- 172.21.15.57:0/1542145460 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe4280b0530 msgr2=0x7fe428144710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:46.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.059+0000 7fe42d04b640 1 --2- 172.21.15.57:0/1542145460 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe4280b0530 0x7fe428144710 secure :-1 s=READY pgs=2227 cs=0 l=1 rev1=1 crypto rx=0x7fe41c00b4a0 tx=0x7fe41c00b960 comp rx=0 tx=0).stop 2023-10-19T14:18:46.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.059+0000 7fe42d04b640 1 -- 172.21.15.57:0/1542145460 shutdown_connections 2023-10-19T14:18:46.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.059+0000 7fe42d04b640 1 --2- 172.21.15.57:0/1542145460 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fe41406af60 0x7fe41406d420 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:46.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.059+0000 7fe42d04b640 1 --2- 172.21.15.57:0/1542145460 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe4280cbd20 0x7fe4281f2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:46.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.059+0000 7fe42d04b640 1 --2- 172.21.15.57:0/1542145460 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe4280b0530 0x7fe428144710 unknown :-1 s=CLOSED pgs=2227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:46.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.059+0000 7fe42d04b640 1 --2- 172.21.15.57:0/1542145460 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe4280afa20 0x7fe4280cb530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:46.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.059+0000 7fe42d04b640 1 -- 172.21.15.57:0/1542145460 >> 172.21.15.57:0/1542145460 conn(0x7fe428069930 msgr2=0x7fe42814cd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:46.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.059+0000 7fe42d04b640 1 -- 172.21.15.57:0/1542145460 shutdown_connections 2023-10-19T14:18:46.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.059+0000 7fe42d04b640 1 -- 172.21.15.57:0/1542145460 wait complete. 2023-10-19T14:18:46.072 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-19T14:18:46.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.180+0000 7f10289f5640 1 -- 172.21.15.57:0/673859971 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f102413e520 msgr2=0x7f102406bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:46.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.180+0000 7f10289f5640 1 --2- 172.21.15.57:0/673859971 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f102413e520 0x7f102406bb60 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f101c009830 tx=0x7f101c02bcd0 comp rx=0 tx=0).stop 2023-10-19T14:18:46.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.181+0000 7f10289f5640 1 -- 172.21.15.57:0/673859971 shutdown_connections 2023-10-19T14:18:46.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.181+0000 7f10289f5640 1 --2- 172.21.15.57:0/673859971 >> v2:172.21.15.179:3300/0 conn(0x7f1024070af0 0x7f10240affe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:46.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.181+0000 7f10289f5640 1 --2- 172.21.15.57:0/673859971 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f102413e520 0x7f102406bb60 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:46.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.181+0000 7f10289f5640 1 -- 172.21.15.57:0/673859971 >> 172.21.15.57:0/673859971 conn(0x7f1024069930 msgr2=0x7f1024069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:46.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.181+0000 7f10289f5640 1 -- 172.21.15.57:0/673859971 shutdown_connections 2023-10-19T14:18:46.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.181+0000 7f10289f5640 1 -- 172.21.15.57:0/673859971 wait complete. 2023-10-19T14:18:46.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.182+0000 7f10289f5640 1 Processor -- start 2023-10-19T14:18:46.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.182+0000 7f10289f5640 1 -- start start 2023-10-19T14:18:46.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.183+0000 7f10289f5640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1024070af0 0x7f102413b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:46.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.183+0000 7f10289f5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f102413e520 0x7f10240abb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:46.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.183+0000 7f10289f5640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1024141690 0x7f10241e2580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:46.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.183+0000 7f10289f5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10240a69b0 con 0x7f102413e520 2023-10-19T14:18:46.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.183+0000 7f10289f5640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10240a6c90 con 0x7f1024070af0 2023-10-19T14:18:46.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.183+0000 7f10289f5640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10240a6f70 con 0x7f1024141690 2023-10-19T14:18:46.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.183+0000 7f1021d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f102413e520 0x7f10240abb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:46.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.183+0000 7f1021d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f102413e520 0x7f10240abb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55720/0 (socket says 172.21.15.57:55720) 2023-10-19T14:18:46.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.183+0000 7f1021d9b640 1 -- 172.21.15.57:0/2052373839 learned_addr learned my addr 172.21.15.57:0/2052373839 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:18:46.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.183+0000 7f102259c640 1 --2- 172.21.15.57:0/2052373839 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1024070af0 0x7f102413b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:46.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.183+0000 7f1022d9d640 1 --2- 172.21.15.57:0/2052373839 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1024141690 0x7f10241e2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:46.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.183+0000 7f1021d9b640 1 -- 172.21.15.57:0/2052373839 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1024141690 msgr2=0x7f10241e2580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:46.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.183+0000 7f1021d9b640 1 --2- 172.21.15.57:0/2052373839 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1024141690 0x7f10241e2580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:46.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.183+0000 7f1021d9b640 1 -- 172.21.15.57:0/2052373839 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1024070af0 msgr2=0x7f102413b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:46.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.183+0000 7f1021d9b640 1 --2- 172.21.15.57:0/2052373839 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1024070af0 0x7f102413b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:46.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.183+0000 7f1021d9b640 1 -- 172.21.15.57:0/2052373839 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f101c009490 con 0x7f102413e520 2023-10-19T14:18:46.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.183+0000 7f1022d9d640 1 --2- 172.21.15.57:0/2052373839 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1024141690 0x7f10241e2580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:18:46.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.183+0000 7f1021d9b640 1 --2- 172.21.15.57:0/2052373839 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f102413e520 0x7f10240abb60 secure :-1 s=READY pgs=2228 cs=0 l=1 rev1=1 crypto rx=0x7f100c00c3c0 tx=0x7f100c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:46.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.183+0000 7f101affd640 1 -- 172.21.15.57:0/2052373839 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f100c004230 con 0x7f102413e520 2023-10-19T14:18:46.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.184+0000 7f101affd640 1 -- 172.21.15.57:0/2052373839 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f100c0044a0 con 0x7f102413e520 2023-10-19T14:18:46.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.184+0000 7f10289f5640 1 -- 172.21.15.57:0/2052373839 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10240a7250 con 0x7f102413e520 2023-10-19T14:18:46.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.184+0000 7f101affd640 1 -- 172.21.15.57:0/2052373839 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f100c003850 con 0x7f102413e520 2023-10-19T14:18:46.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.184+0000 7f10289f5640 1 -- 172.21.15.57:0/2052373839 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f102413b5c0 con 0x7f102413e520 2023-10-19T14:18:46.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.184+0000 7f10289f5640 1 -- 172.21.15.57:0/2052373839 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10240a7770 con 0x7f102413e520 2023-10-19T14:18:46.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.185+0000 7f101affd640 1 -- 172.21.15.57:0/2052373839 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 117) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f100c0039b0 con 0x7f102413e520 2023-10-19T14:18:46.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.185+0000 7f101affd640 1 --2- 172.21.15.57:0/2052373839 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f100406af60 0x7f100406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:46.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.185+0000 7f101affd640 1 -- 172.21.15.57:0/2052373839 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f100c0890f0 con 0x7f102413e520 2023-10-19T14:18:46.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.186+0000 7f102259c640 1 --2- 172.21.15.57:0/2052373839 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f100406af60 0x7f100406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:46.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.186+0000 7f102259c640 1 --2- 172.21.15.57:0/2052373839 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f100406af60 0x7f100406d420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f101c036000 tx=0x7f101c004730 comp rx=0 tx=0).ready entity=mgr.8744 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:46.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.190+0000 7f101affd640 1 -- 172.21.15.57:0/2052373839 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f10240a7770 con 0x7f102413e520 2023-10-19T14:18:46.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.413+0000 7f10289f5640 1 -- 172.21.15.57:0/2052373839 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f10240a7770 con 0x7f102413e520 2023-10-19T14:18:46.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.533+0000 7f101affd640 1 -- 172.21.15.57:0/2052373839 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v118) v1 ==== 92+0+0 (secure 0 0 0) 0x7f10240a7770 con 0x7f102413e520 2023-10-19T14:18:46.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.535+0000 7f10289f5640 1 -- 172.21.15.57:0/2052373839 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f100406af60 msgr2=0x7f100406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:46.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.535+0000 7f10289f5640 1 --2- 172.21.15.57:0/2052373839 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f100406af60 0x7f100406d420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f101c036000 tx=0x7f101c004730 comp rx=0 tx=0).stop 2023-10-19T14:18:46.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.535+0000 7f10289f5640 1 -- 172.21.15.57:0/2052373839 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f102413e520 msgr2=0x7f10240abb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:46.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.535+0000 7f10289f5640 1 --2- 172.21.15.57:0/2052373839 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f102413e520 0x7f10240abb60 secure :-1 s=READY pgs=2228 cs=0 l=1 rev1=1 crypto rx=0x7f100c00c3c0 tx=0x7f100c00c880 comp rx=0 tx=0).stop 2023-10-19T14:18:46.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.536+0000 7f10289f5640 1 -- 172.21.15.57:0/2052373839 shutdown_connections 2023-10-19T14:18:46.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.536+0000 7f10289f5640 1 --2- 172.21.15.57:0/2052373839 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7f100406af60 0x7f100406d420 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:46.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.536+0000 7f10289f5640 1 --2- 172.21.15.57:0/2052373839 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1024141690 0x7f10241e2580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:46.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.536+0000 7f10289f5640 1 --2- 172.21.15.57:0/2052373839 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f102413e520 0x7f10240abb60 unknown :-1 s=CLOSED pgs=2228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:46.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.536+0000 7f10289f5640 1 --2- 172.21.15.57:0/2052373839 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1024070af0 0x7f102413b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:46.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.536+0000 7f10289f5640 1 -- 172.21.15.57:0/2052373839 >> 172.21.15.57:0/2052373839 conn(0x7f1024069930 msgr2=0x7f10241df740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:46.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.536+0000 7f10289f5640 1 -- 172.21.15.57:0/2052373839 shutdown_connections 2023-10-19T14:18:46.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.536+0000 7f10289f5640 1 -- 172.21.15.57:0/2052373839 wait complete. 2023-10-19T14:18:46.551 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-19T14:18:46.665 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.662+0000 7feb0cff5640 1 -- 172.21.15.57:0/1102747145 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feb08070080 msgr2=0x7feb08159f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:46.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.662+0000 7feb0cff5640 1 --2- 172.21.15.57:0/1102747145 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feb08070080 0x7feb08159f40 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7feaf8009830 tx=0x7feaf802bd60 comp rx=0 tx=0).stop 2023-10-19T14:18:46.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.662+0000 7feb0cff5640 1 -- 172.21.15.57:0/1102747145 shutdown_connections 2023-10-19T14:18:46.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.662+0000 7feb0cff5640 1 --2- 172.21.15.57:0/1102747145 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feb08070080 0x7feb08159f40 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:46.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.662+0000 7feb0cff5640 1 --2- 172.21.15.57:0/1102747145 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7feb081424c0 0x7feb0806c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:46.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.662+0000 7feb0cff5640 1 -- 172.21.15.57:0/1102747145 >> 172.21.15.57:0/1102747145 conn(0x7feb080af730 msgr2=0x7feb080afb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:46.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.662+0000 7feb0cff5640 1 -- 172.21.15.57:0/1102747145 shutdown_connections 2023-10-19T14:18:46.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.662+0000 7feb0cff5640 1 -- 172.21.15.57:0/1102747145 wait complete. 2023-10-19T14:18:46.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.664+0000 7feb0cff5640 1 Processor -- start 2023-10-19T14:18:46.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.664+0000 7feb0cff5640 1 -- start start 2023-10-19T14:18:46.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.664+0000 7feb0cff5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feb08069720 0x7feb08156890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:46.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.664+0000 7feb0cff5640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7feb08070080 0x7feb0813e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:46.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.665+0000 7feb0cff5640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feb081424c0 0x7feb080cda00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:46.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.665+0000 7feb0cff5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb080a73c0 con 0x7feb08069720 2023-10-19T14:18:46.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.665+0000 7feb0cff5640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb080a76a0 con 0x7feb081424c0 2023-10-19T14:18:46.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.665+0000 7feb0cff5640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb080a7980 con 0x7feb08070080 2023-10-19T14:18:46.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.665+0000 7feb0659c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feb08069720 0x7feb08156890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:46.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.665+0000 7feb0659c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feb08069720 0x7feb08156890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:36848/0 (socket says 172.21.15.57:36848) 2023-10-19T14:18:46.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.665+0000 7feb0659c640 1 -- 172.21.15.57:0/470091730 learned_addr learned my addr 172.21.15.57:0/470091730 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:18:46.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.665+0000 7feb05d9b640 1 --2- 172.21.15.57:0/470091730 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7feb08070080 0x7feb0813e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:46.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.665+0000 7feb06d9d640 1 --2- 172.21.15.57:0/470091730 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feb081424c0 0x7feb080cda00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:46.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.665+0000 7feb0659c640 1 -- 172.21.15.57:0/470091730 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7feb08070080 msgr2=0x7feb0813e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:46.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.665+0000 7feb0659c640 1 --2- 172.21.15.57:0/470091730 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7feb08070080 0x7feb0813e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:46.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.665+0000 7feb0659c640 1 -- 172.21.15.57:0/470091730 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feb081424c0 msgr2=0x7feb080cda00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:46.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.665+0000 7feb0659c640 1 --2- 172.21.15.57:0/470091730 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feb081424c0 0x7feb080cda00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:46.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.665+0000 7feb0659c640 1 -- 172.21.15.57:0/470091730 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaf8009490 con 0x7feb08069720 2023-10-19T14:18:46.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.665+0000 7feb0659c640 1 --2- 172.21.15.57:0/470091730 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feb08069720 0x7feb08156890 secure :-1 s=READY pgs=2232 cs=0 l=1 rev1=1 crypto rx=0x7feafc00e790 tx=0x7feafc00ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:46.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.665+0000 7feaf6ffd640 1 -- 172.21.15.57:0/470091730 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feafc009580 con 0x7feb08069720 2023-10-19T14:18:46.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.665+0000 7feb0cff5640 1 -- 172.21.15.57:0/470091730 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb080a7c60 con 0x7feb08069720 2023-10-19T14:18:46.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.665+0000 7feaf6ffd640 1 -- 172.21.15.57:0/470091730 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7feafc00cdb0 con 0x7feb08069720 2023-10-19T14:18:46.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.665+0000 7feaf6ffd640 1 -- 172.21.15.57:0/470091730 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feafc010690 con 0x7feb08069720 2023-10-19T14:18:46.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.666+0000 7feb0cff5640 1 -- 172.21.15.57:0/470091730 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb08156dd0 con 0x7feb08069720 2023-10-19T14:18:46.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.666+0000 7feb0cff5640 1 -- 172.21.15.57:0/470091730 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb080cad20 con 0x7feb08069720 2023-10-19T14:18:46.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.667+0000 7feaf6ffd640 1 -- 172.21.15.57:0/470091730 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 118) v1 ==== 89348+0+0 (secure 0 0 0) 0x7feafc0107f0 con 0x7feb08069720 2023-10-19T14:18:46.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.668+0000 7feaf6ffd640 1 --2- 172.21.15.57:0/470091730 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7feae806afe0 0x7feae806d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:46.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.668+0000 7feaf6ffd640 1 -- 172.21.15.57:0/470091730 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 6057+0+0 (secure 0 0 0) 0x7feafc014070 con 0x7feb08069720 2023-10-19T14:18:46.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.668+0000 7feb05d9b640 1 -- 172.21.15.57:0/470091730 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7feae806afe0 msgr2=0x7feae806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/2730461479 2023-10-19T14:18:46.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.668+0000 7feb05d9b640 1 --2- 172.21.15.57:0/470091730 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7feae806afe0 0x7feae806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T14:18:46.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.672+0000 7feaf6ffd640 1 -- 172.21.15.57:0/470091730 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7feafc051ca0 con 0x7feb08069720 2023-10-19T14:18:46.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.852+0000 7feb0cff5640 1 -- 172.21.15.57:0/470091730 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7feb080675f0 con 0x7feb08069720 2023-10-19T14:18:46.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.859+0000 7feaf6ffd640 1 -- 172.21.15.57:0/470091730 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v72)=0 v72) v1 ==== 138+0+0 (secure 0 0 0) 0x7feafc055230 con 0x7feb08069720 2023-10-19T14:18:46.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.861+0000 7feb0cff5640 1 -- 172.21.15.57:0/470091730 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7feae806afe0 msgr2=0x7feae806d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T14:18:46.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.861+0000 7feb0cff5640 1 --2- 172.21.15.57:0/470091730 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7feae806afe0 0x7feae806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:46.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.861+0000 7feb0cff5640 1 -- 172.21.15.57:0/470091730 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feb08069720 msgr2=0x7feb08156890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:46.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.861+0000 7feb0cff5640 1 --2- 172.21.15.57:0/470091730 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feb08069720 0x7feb08156890 secure :-1 s=READY pgs=2232 cs=0 l=1 rev1=1 crypto rx=0x7feafc00e790 tx=0x7feafc00ec50 comp rx=0 tx=0).stop 2023-10-19T14:18:46.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.861+0000 7feb0cff5640 1 -- 172.21.15.57:0/470091730 shutdown_connections 2023-10-19T14:18:46.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.861+0000 7feb0cff5640 1 --2- 172.21.15.57:0/470091730 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7feae806afe0 0x7feae806d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:46.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.861+0000 7feb0cff5640 1 --2- 172.21.15.57:0/470091730 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feb081424c0 0x7feb080cda00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:46.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.861+0000 7feb0cff5640 1 --2- 172.21.15.57:0/470091730 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7feb08070080 0x7feb0813e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:46.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.862+0000 7feb0cff5640 1 --2- 172.21.15.57:0/470091730 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feb08069720 0x7feb08156890 unknown :-1 s=CLOSED pgs=2232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:46.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.862+0000 7feb0cff5640 1 -- 172.21.15.57:0/470091730 >> 172.21.15.57:0/470091730 conn(0x7feb080af730 msgr2=0x7feb0813bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:46.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.862+0000 7feb0cff5640 1 -- 172.21.15.57:0/470091730 shutdown_connections 2023-10-19T14:18:46.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.862+0000 7feb0cff5640 1 -- 172.21.15.57:0/470091730 wait complete. 2023-10-19T14:18:46.879 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-19T14:18:46.925 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:18:46.922+0000 7f0a4b817040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-19T14:18:46.946 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:18:46.941+0000 7f8b3c435040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-19T14:18:46.969 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:18:46.967+0000 7f0a4b817040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-19T14:18:46.992 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.989+0000 7fb3c759e640 1 -- 172.21.15.57:0/2248589073 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3c815e840 msgr2=0x7fb3c80c4780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:46.993 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.989+0000 7fb3c759e640 1 --2- 172.21.15.57:0/2248589073 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3c815e840 0x7fb3c80c4780 secure :-1 s=READY pgs=2233 cs=0 l=1 rev1=1 crypto rx=0x7fb3b8005fd0 tx=0x7fb3b802c2e0 comp rx=0 tx=0).stop 2023-10-19T14:18:46.993 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.989+0000 7fb3c759e640 1 -- 172.21.15.57:0/2248589073 shutdown_connections 2023-10-19T14:18:46.993 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.989+0000 7fb3c759e640 1 --2- 172.21.15.57:0/2248589073 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3c815e840 0x7fb3c80c4780 unknown :-1 s=CLOSED pgs=2233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:46.993 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.989+0000 7fb3c759e640 1 --2- 172.21.15.57:0/2248589073 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb3c80cd890 0x7fb3c80cc730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:46.993 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.989+0000 7fb3c759e640 1 -- 172.21.15.57:0/2248589073 >> 172.21.15.57:0/2248589073 conn(0x7fb3c80d0c30 msgr2=0x7fb3c80d1040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:46.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.989+0000 7fb3c759e640 1 -- 172.21.15.57:0/2248589073 shutdown_connections 2023-10-19T14:18:46.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.990+0000 7fb3c759e640 1 -- 172.21.15.57:0/2248589073 wait complete. 2023-10-19T14:18:46.994 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:18:46.991+0000 7f8b3c435040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-19T14:18:46.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.991+0000 7fb3c759e640 1 Processor -- start 2023-10-19T14:18:46.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.991+0000 7fb3c759e640 1 -- start start 2023-10-19T14:18:46.997 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.992+0000 7fb3c759e640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb3c80cd890 0x7fb3c8149050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:46.997 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.992+0000 7fb3c759e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3c80cdc70 0x7fb3c81e39c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:46.997 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.992+0000 7fb3c759e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb3c815e840 0x7fb3c80c85b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:46.997 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.992+0000 7fb3c759e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3c8149590 con 0x7fb3c80cdc70 2023-10-19T14:18:46.997 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.992+0000 7fb3c759e640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3c8157ab0 con 0x7fb3c815e840 2023-10-19T14:18:46.998 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.992+0000 7fb3c759e640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3c8157d90 con 0x7fb3c80cd890 2023-10-19T14:18:46.998 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.992+0000 7fb3c5d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3c80cdc70 0x7fb3c81e39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:46.998 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.992+0000 7fb3c5d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3c80cdc70 0x7fb3c81e39c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:36868/0 (socket says 172.21.15.57:36868) 2023-10-19T14:18:46.998 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.992+0000 7fb3c5d9b640 1 -- 172.21.15.57:0/880299893 learned_addr learned my addr 172.21.15.57:0/880299893 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:18:46.998 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.992+0000 7fb3c659c640 1 --2- 172.21.15.57:0/880299893 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb3c80cd890 0x7fb3c8149050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:46.999 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.992+0000 7fb3c6d9d640 1 --2- 172.21.15.57:0/880299893 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb3c815e840 0x7fb3c80c85b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:46.999 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.992+0000 7fb3c5d9b640 1 -- 172.21.15.57:0/880299893 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb3c80cd890 msgr2=0x7fb3c8149050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:46.999 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.992+0000 7fb3c5d9b640 1 --2- 172.21.15.57:0/880299893 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb3c80cd890 0x7fb3c8149050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:46.999 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.992+0000 7fb3c5d9b640 1 -- 172.21.15.57:0/880299893 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb3c815e840 msgr2=0x7fb3c80c85b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:46.999 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.992+0000 7fb3c5d9b640 1 --2- 172.21.15.57:0/880299893 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb3c815e840 0x7fb3c80c85b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:47.000 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.992+0000 7fb3c5d9b640 1 -- 172.21.15.57:0/880299893 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3b8009900 con 0x7fb3c80cdc70 2023-10-19T14:18:47.000 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.992+0000 7fb3c5d9b640 1 --2- 172.21.15.57:0/880299893 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3c80cdc70 0x7fb3c81e39c0 secure :-1 s=READY pgs=2234 cs=0 l=1 rev1=1 crypto rx=0x7fb3b000c400 tx=0x7fb3b000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:47.000 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.992+0000 7fb3beffd640 1 -- 172.21.15.57:0/880299893 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3b00052d0 con 0x7fb3c80cdc70 2023-10-19T14:18:47.000 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.992+0000 7fb3beffd640 1 -- 172.21.15.57:0/880299893 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fb3b0007020 con 0x7fb3c80cdc70 2023-10-19T14:18:47.001 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.992+0000 7fb3beffd640 1 -- 172.21.15.57:0/880299893 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3b0007470 con 0x7fb3c80cdc70 2023-10-19T14:18:47.001 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.992+0000 7fb3c759e640 1 -- 172.21.15.57:0/880299893 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3c8158070 con 0x7fb3c80cdc70 2023-10-19T14:18:47.001 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.993+0000 7fb3c759e640 1 -- 172.21.15.57:0/880299893 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3c8158630 con 0x7fb3c80cdc70 2023-10-19T14:18:47.001 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.994+0000 7fb3c759e640 1 -- 172.21.15.57:0/880299893 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3c8066960 con 0x7fb3c80cdc70 2023-10-19T14:18:47.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.994+0000 7fb3beffd640 1 -- 172.21.15.57:0/880299893 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 118) v1 ==== 89348+0+0 (secure 0 0 0) 0x7fb3b00067d0 con 0x7fb3c80cdc70 2023-10-19T14:18:47.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.995+0000 7fb3beffd640 1 --2- 172.21.15.57:0/880299893 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fb3a806afe0 0x7fb3a806d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:47.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.995+0000 7fb3c659c640 1 -- 172.21.15.57:0/880299893 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fb3a806afe0 msgr2=0x7fb3a806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/2730461479 2023-10-19T14:18:47.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.995+0000 7fb3beffd640 1 -- 172.21.15.57:0/880299893 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fb3b0088f00 con 0x7fb3c80cdc70 2023-10-19T14:18:47.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.995+0000 7fb3c659c640 1 --2- 172.21.15.57:0/880299893 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fb3a806afe0 0x7fb3a806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T14:18:47.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:46.999+0000 7fb3beffd640 1 -- 172.21.15.57:0/880299893 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb3b0053df0 con 0x7fb3c80cdc70 2023-10-19T14:18:47.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:47.177+0000 7fb3c759e640 1 -- 172.21.15.57:0/880299893 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7fb3c804da70 con 0x7fb3c80cdc70 2023-10-19T14:18:47.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:47.184+0000 7fb3beffd640 1 -- 172.21.15.57:0/880299893 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v73)=0 v73) v1 ==== 133+0+0 (secure 0 0 0) 0x7fb3b0057410 con 0x7fb3c80cdc70 2023-10-19T14:18:47.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:47.186+0000 7fb3c759e640 1 -- 172.21.15.57:0/880299893 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fb3a806afe0 msgr2=0x7fb3a806d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T14:18:47.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:47.186+0000 7fb3c759e640 1 --2- 172.21.15.57:0/880299893 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fb3a806afe0 0x7fb3a806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:47.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:47.186+0000 7fb3c759e640 1 -- 172.21.15.57:0/880299893 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3c80cdc70 msgr2=0x7fb3c81e39c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:47.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:47.186+0000 7fb3c759e640 1 --2- 172.21.15.57:0/880299893 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3c80cdc70 0x7fb3c81e39c0 secure :-1 s=READY pgs=2234 cs=0 l=1 rev1=1 crypto rx=0x7fb3b000c400 tx=0x7fb3b000c8c0 comp rx=0 tx=0).stop 2023-10-19T14:18:47.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:47.186+0000 7fb3c759e640 1 -- 172.21.15.57:0/880299893 shutdown_connections 2023-10-19T14:18:47.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:47.186+0000 7fb3c759e640 1 --2- 172.21.15.57:0/880299893 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7fb3a806afe0 0x7fb3a806d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:47.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:47.186+0000 7fb3c759e640 1 --2- 172.21.15.57:0/880299893 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb3c815e840 0x7fb3c80c85b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:47.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:47.186+0000 7fb3c759e640 1 --2- 172.21.15.57:0/880299893 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3c80cdc70 0x7fb3c81e39c0 unknown :-1 s=CLOSED pgs=2234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:47.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:47.186+0000 7fb3c759e640 1 --2- 172.21.15.57:0/880299893 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb3c80cd890 0x7fb3c8149050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:47.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:47.186+0000 7fb3c759e640 1 -- 172.21.15.57:0/880299893 >> 172.21.15.57:0/880299893 conn(0x7fb3c80d0c30 msgr2=0x7fb3c815c9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:47.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:47.187+0000 7fb3c759e640 1 -- 172.21.15.57:0/880299893 shutdown_connections 2023-10-19T14:18:47.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:47.187+0000 7fb3c759e640 1 -- 172.21.15.57:0/880299893 wait complete. 2023-10-19T14:18:47.200 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:18:47.201 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-10-19T14:18:47.222 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 1m 2023-10-19T14:18:47.363 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:18:47.361+0000 7f0a4b817040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-19T14:18:47.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:47.376+0000 7faf6874c640 1 -- 172.21.15.57:0/4053686825 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faf600cdbf0 msgr2=0x7faf600ae820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:47.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:47.376+0000 7faf6874c640 1 --2- 172.21.15.57:0/4053686825 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faf600cdbf0 0x7faf600ae820 secure :-1 s=READY pgs=2235 cs=0 l=1 rev1=1 crypto rx=0x7faf58009ca0 tx=0x7faf5802c360 comp rx=0 tx=0).stop 2023-10-19T14:18:47.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:47.376+0000 7faf6874c640 1 -- 172.21.15.57:0/4053686825 shutdown_connections 2023-10-19T14:18:47.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:47.376+0000 7faf6874c640 1 --2- 172.21.15.57:0/4053686825 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faf600cdbf0 0x7faf600ae820 unknown :-1 s=CLOSED pgs=2235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:47.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:47.376+0000 7faf6874c640 1 --2- 172.21.15.57:0/4053686825 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7faf600ce420 0x7faf600cd6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:47.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:47.376+0000 7faf6874c640 1 -- 172.21.15.57:0/4053686825 >> 172.21.15.57:0/4053686825 conn(0x7faf60099ff0 msgr2=0x7faf600d0d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:47.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:47.376+0000 7faf6874c640 1 -- 172.21.15.57:0/4053686825 shutdown_connections 2023-10-19T14:18:47.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:47.376+0000 7faf6874c640 1 -- 172.21.15.57:0/4053686825 wait complete. 2023-10-19T14:18:47.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:47.377+0000 7faf6874c640 1 Processor -- start 2023-10-19T14:18:47.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:47.378+0000 7faf6874c640 1 -- start start 2023-10-19T14:18:47.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:47.378+0000 7faf6874c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faf600cdbf0 0x7faf60151200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:47.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:47.378+0000 7faf6874c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7faf600ce420 0x7faf600b28b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:47.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:47.378+0000 7faf6874c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faf6015b180 0x7faf6014d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:47.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:47.378+0000 7faf6874c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf600b2490 con 0x7faf6015b180 2023-10-19T14:18:47.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:47.378+0000 7faf6874c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf60151740 con 0x7faf600cdbf0 2023-10-19T14:18:47.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:47.378+0000 7faf6874c640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf600b15e0 con 0x7faf600ce420 2023-10-19T14:18:47.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:47.378+0000 7faf66ce9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faf6015b180 0x7faf6014d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:47.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:47.378+0000 7faf66ce9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faf6015b180 0x7faf6014d1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:36884/0 (socket says 172.21.15.57:36884) 2023-10-19T14:18:47.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:47.378+0000 7faf66ce9640 1 -- 172.21.15.57:0/491597351 learned_addr learned my addr 172.21.15.57:0/491597351 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:18:47.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:47.378+0000 7faf65ce7640 1 --2- 172.21.15.57:0/491597351 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7faf600ce420 0x7faf600b28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:47.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:47.378+0000 7faf664e8640 1 --2- 172.21.15.57:0/491597351 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faf600cdbf0 0x7faf60151200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:47.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:47.379+0000 7faf66ce9640 1 -- 172.21.15.57:0/491597351 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7faf600ce420 msgr2=0x7faf600b28b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:47.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:47.379+0000 7faf66ce9640 1 --2- 172.21.15.57:0/491597351 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7faf600ce420 0x7faf600b28b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:47.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:47.379+0000 7faf66ce9640 1 -- 172.21.15.57:0/491597351 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faf600cdbf0 msgr2=0x7faf60151200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:47.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:47.379+0000 7faf66ce9640 1 --2- 172.21.15.57:0/491597351 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faf600cdbf0 0x7faf60151200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:47.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:47.379+0000 7faf66ce9640 1 -- 172.21.15.57:0/491597351 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf58009900 con 0x7faf6015b180 2023-10-19T14:18:47.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:47.379+0000 7faf65ce7640 1 --2- 172.21.15.57:0/491597351 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7faf600ce420 0x7faf600b28b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T14:18:47.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:47.379+0000 7faf66ce9640 1 --2- 172.21.15.57:0/491597351 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faf6015b180 0x7faf6014d1f0 secure :-1 s=READY pgs=2236 cs=0 l=1 rev1=1 crypto rx=0x7faf58006010 tx=0x7faf58035b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:47.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:47.379+0000 7faf52ffd640 1 -- 172.21.15.57:0/491597351 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf58004240 con 0x7faf6015b180 2023-10-19T14:18:47.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:47.379+0000 7faf52ffd640 1 -- 172.21.15.57:0/491597351 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7faf58004490 con 0x7faf6015b180 2023-10-19T14:18:47.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:47.379+0000 7faf52ffd640 1 -- 172.21.15.57:0/491597351 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf58004820 con 0x7faf6015b180 2023-10-19T14:18:47.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:47.379+0000 7faf6874c640 1 -- 172.21.15.57:0/491597351 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf601f13e0 con 0x7faf6015b180 2023-10-19T14:18:47.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:47.379+0000 7faf6874c640 1 -- 172.21.15.57:0/491597351 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf600b1af0 con 0x7faf6015b180 2023-10-19T14:18:47.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:47.380+0000 7faf6874c640 1 -- 172.21.15.57:0/491597351 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf60066960 con 0x7faf6015b180 2023-10-19T14:18:47.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:47.381+0000 7faf52ffd640 1 -- 172.21.15.57:0/491597351 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 118) v1 ==== 89348+0+0 (secure 0 0 0) 0x7faf5803b040 con 0x7faf6015b180 2023-10-19T14:18:47.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:47.381+0000 7faf52ffd640 1 --2- 172.21.15.57:0/491597351 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7faf4406afb0 0x7faf4406d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:47.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:47.381+0000 7faf52ffd640 1 -- 172.21.15.57:0/491597351 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 6057+0+0 (secure 0 0 0) 0x7faf580ac440 con 0x7faf6015b180 2023-10-19T14:18:47.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:47.381+0000 7faf664e8640 1 -- 172.21.15.57:0/491597351 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7faf4406afb0 msgr2=0x7faf4406d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/2730461479 2023-10-19T14:18:47.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:47.381+0000 7faf664e8640 1 --2- 172.21.15.57:0/491597351 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7faf4406afb0 0x7faf4406d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T14:18:47.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:47.385+0000 7faf52ffd640 1 -- 172.21.15.57:0/491597351 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7faf58077330 con 0x7faf6015b180 2023-10-19T14:18:47.408 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:18:47.405+0000 7f0a4b817040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-19T14:18:47.414 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:18:47.412+0000 7f8b3c435040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-19T14:18:47.450 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:18:47.448+0000 7f0a4b817040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-19T14:18:47.461 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:18:47.459+0000 7f8b3c435040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-19T14:18:47.502 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:18:47.500+0000 7f0a4b817040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-19T14:18:47.507 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:18:47.505+0000 7f8b3c435040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-19T14:18:47.545 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:18:47.543+0000 7f0a4b817040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-19T14:18:47.562 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:18:47.560+0000 7f8b3c435040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-19T14:18:47.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:47.582+0000 7faf664e8640 1 -- 172.21.15.57:0/491597351 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7faf4406afb0 msgr2=0x7faf4406d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/2730461479 2023-10-19T14:18:47.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:47.582+0000 7faf664e8640 1 --2- 172.21.15.57:0/491597351 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7faf4406afb0 0x7faf4406d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-19T14:18:47.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:47.583+0000 7faf6874c640 1 -- 172.21.15.57:0/491597351 --> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7faf600b25d0 con 0x7faf4406afb0 2023-10-19T14:18:47.606 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:18:47.604+0000 7f8b3c435040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-19T14:18:47.644 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:18:47.642+0000 7f0a4b817040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-19T14:18:47.697 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:18:47.695+0000 7f0a4b817040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-19T14:18:47.709 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:18:47.707+0000 7f8b3c435040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-19T14:18:47.738 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:18:47.737+0000 7f0a4b817040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-19T14:18:47.765 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:18:47.762+0000 7f8b3c435040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-19T14:18:47.786 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:18:47.784+0000 7f0a4b817040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-19T14:18:47.809 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:18:47.806+0000 7f8b3c435040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-19T14:18:47.828 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:18:47.826+0000 7f0a4b817040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-19T14:18:47.859 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:18:47.857+0000 7f8b3c435040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-19T14:18:47.904 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:18:47.901+0000 7f8b3c435040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-19T14:18:47.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:47.983+0000 7faf664e8640 1 -- 172.21.15.57:0/491597351 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7faf4406afb0 msgr2=0x7faf4406d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/2730461479 2023-10-19T14:18:47.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:47.983+0000 7faf664e8640 1 --2- 172.21.15.57:0/491597351 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7faf4406afb0 0x7faf4406d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-19T14:18:48.144 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:18:48.142+0000 7f0a4b817040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-19T14:18:48.238 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:18:48.235+0000 7f8b3c435040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-19T14:18:48.273 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:18:48.271+0000 7f0a4b817040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-19T14:18:48.365 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:18:48.363+0000 7f0a4b817040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-19T14:18:48.377 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:18:48.374+0000 7f8b3c435040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-19T14:18:48.407 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:18:48.405+0000 7f0a4b817040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-19T14:18:48.474 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:18:48.471+0000 7f8b3c435040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-19T14:18:48.502 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:18:48.501+0000 7f0a4b817040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-19T14:18:48.518 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:18:48.516+0000 7f8b3c435040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-19T14:18:48.593 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:18:48.591+0000 7f0a4b817040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-19T14:18:48.618 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:18:48.615+0000 7f8b3c435040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-19T14:18:48.714 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:18:48.711+0000 7f8b3c435040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-19T14:18:48.729 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:18:48.728+0000 7f0a4b817040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-19T14:18:48.771 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:18:48.769+0000 7f0a4b817040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-19T14:18:48.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:48.784+0000 7faf664e8640 1 -- 172.21.15.57:0/491597351 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7faf4406afb0 msgr2=0x7faf4406d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/2730461479 2023-10-19T14:18:48.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:48.784+0000 7faf664e8640 1 --2- 172.21.15.57:0/491597351 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7faf4406afb0 0x7faf4406d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-19T14:18:48.858 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:18:48.855+0000 7f8b3c435040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-19T14:18:48.876 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:18:48.874+0000 7f0a4b817040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-19T14:18:48.902 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:18:48.899+0000 7f8b3c435040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-19T14:18:49.012 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:18:49.009+0000 7f8b3c435040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-19T14:18:49.494 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:18:49.492+0000 7f0a4b817040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-19T14:18:49.662 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:18:49.660+0000 7f8b3c435040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-19T14:18:49.754 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:18:49.753+0000 7f0a4b817040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-19T14:18:49.803 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:18:49.802+0000 7f0a4b817040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-19T14:18:49.903 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:18:49.902+0000 7f0a4b817040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-19T14:18:49.932 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:49.930+0000 7faf52ffd640 1 -- 172.21.15.57:0/491597351 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mgrmap(e 119) v1 ==== 88957+0+0 (secure 0 0 0) 0x7faf5800a040 con 0x7faf6015b180 2023-10-19T14:18:49.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:49.930+0000 7faf52ffd640 1 -- 172.21.15.57:0/491597351 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7faf4406afb0 msgr2=0x7faf4406d470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T14:18:49.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:49.930+0000 7faf52ffd640 1 --2- 172.21.15.57:0/491597351 >> [v2:172.21.15.179:6832/2730461479,v1:172.21.15.179:6833/2730461479] conn(0x7faf4406afb0 0x7faf4406d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:49.941 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:18:49.938+0000 7f8b3c435040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-19T14:18:49.997 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:18:49.994+0000 7f8b3c435040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-19T14:18:50.032 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:18:50.030+0000 7f09c800d640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:18:50.032 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:18:50.030+0000 7f09c800d640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:18:50.033 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:18:50.030+0000 7f09c800d640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:18:50.033 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:18:50.030+0000 7f09c800d640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:18:50.033 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:18:50.030+0000 7f09c800d640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:18:50.033 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:18:50.030+0000 7f09cb053640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:18:50.033 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:18:50.030+0000 7f09cb053640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:18:50.033 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:18:50.030+0000 7f09cb053640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:18:50.034 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:18:50.030+0000 7f09cb053640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:18:50.034 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:18:50.030+0000 7f09cb053640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:18:50.106 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:18:50.104+0000 7f8b3c435040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-19T14:18:50.932 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:50.929+0000 7faf52ffd640 1 -- 172.21.15.57:0/491597351 <== mon.0 v2:172.21.15.57:3300/0 8 ==== mgrmap(e 120) v1 ==== 89084+0+0 (secure 0 0 0) 0x7faf580aa800 con 0x7faf6015b180 2023-10-19T14:18:50.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:50.930+0000 7faf52ffd640 1 --2- 172.21.15.57:0/491597351 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7faf44035850 0x7faf440753e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:50.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:50.930+0000 7faf52ffd640 1 -- 172.21.15.57:0/491597351 --> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7faf600b25d0 con 0x7faf44035850 2023-10-19T14:18:50.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:50.931+0000 7faf664e8640 1 --2- 172.21.15.57:0/491597351 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7faf44035850 0x7faf440753e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:50.935 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:50.932+0000 7faf664e8640 1 --2- 172.21.15.57:0/491597351 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7faf44035850 0x7faf440753e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7faf54000f60 tx=0x7faf54007e30 comp rx=0 tx=0).ready entity=mgr.10231 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:50.993 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:50.991+0000 7faf52ffd640 1 -- 172.21.15.57:0/491597351 <== mgr.10231 v2:172.21.15.179:6832/366917243 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7faf600b25d0 con 0x7faf44035850 2023-10-19T14:18:50.994 INFO:teuthology.orchestra.run.smithi057.stdout:Schedule set for path snap_test_dir1 2023-10-19T14:18:50.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:50.993+0000 7faf6874c640 1 -- 172.21.15.57:0/491597351 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7faf44035850 msgr2=0x7faf440753e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:50.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:50.993+0000 7faf6874c640 1 --2- 172.21.15.57:0/491597351 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7faf44035850 0x7faf440753e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7faf54000f60 tx=0x7faf54007e30 comp rx=0 tx=0).stop 2023-10-19T14:18:50.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:50.993+0000 7faf6874c640 1 -- 172.21.15.57:0/491597351 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faf6015b180 msgr2=0x7faf6014d1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:50.997 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:50.993+0000 7faf6874c640 1 --2- 172.21.15.57:0/491597351 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faf6015b180 0x7faf6014d1f0 secure :-1 s=READY pgs=2236 cs=0 l=1 rev1=1 crypto rx=0x7faf58006010 tx=0x7faf58035b50 comp rx=0 tx=0).stop 2023-10-19T14:18:50.997 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:50.993+0000 7faf66ce9640 1 -- 172.21.15.57:0/491597351 reap_dead start 2023-10-19T14:18:50.997 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:50.993+0000 7faf6874c640 1 -- 172.21.15.57:0/491597351 shutdown_connections 2023-10-19T14:18:50.997 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:50.993+0000 7faf6874c640 1 -- 172.21.15.57:0/491597351 >> 172.21.15.57:0/491597351 conn(0x7faf60099ff0 msgr2=0x7faf600b59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:50.997 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:50.993+0000 7faf6874c640 1 -- 172.21.15.57:0/491597351 shutdown_connections 2023-10-19T14:18:50.998 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:50.993+0000 7faf6874c640 1 -- 172.21.15.57:0/491597351 wait complete. 2023-10-19T14:18:51.008 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2023-10-19T14:18:51.008 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention add snap_test_dir1 1m 2023-10-19T14:18:51.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.121+0000 7f0844b6e640 1 -- 172.21.15.57:0/2704878037 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f08400d1910 msgr2=0x7f08400b0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:51.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.121+0000 7f0844b6e640 1 --2- 172.21.15.57:0/2704878037 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f08400d1910 0x7f08400b0650 secure :-1 s=READY pgs=2238 cs=0 l=1 rev1=1 crypto rx=0x7f0838005fd0 tx=0x7f083802c0e0 comp rx=0 tx=0).stop 2023-10-19T14:18:51.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.121+0000 7f0844b6e640 1 -- 172.21.15.57:0/2704878037 shutdown_connections 2023-10-19T14:18:51.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.121+0000 7f0844b6e640 1 --2- 172.21.15.57:0/2704878037 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f08400d1910 0x7f08400b0650 unknown :-1 s=CLOSED pgs=2238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:51.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.121+0000 7f0844b6e640 1 -- 172.21.15.57:0/2704878037 >> 172.21.15.57:0/2704878037 conn(0x7f08400d0d60 msgr2=0x7f08400d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:51.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.121+0000 7f0844b6e640 1 -- 172.21.15.57:0/2704878037 shutdown_connections 2023-10-19T14:18:51.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.121+0000 7f0844b6e640 1 -- 172.21.15.57:0/2704878037 wait complete. 2023-10-19T14:18:51.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.123+0000 7f0844b6e640 1 Processor -- start 2023-10-19T14:18:51.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.123+0000 7f0844b6e640 1 -- start start 2023-10-19T14:18:51.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.123+0000 7f0844b6e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f08400cf340 0x7f08400c9390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:51.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.123+0000 7f0844b6e640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f08400d1910 0x7f08401f1c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:51.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.123+0000 7f0844b6e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f084015b2a0 0x7f084015de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:51.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.123+0000 7f0844b6e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08400c8f00 con 0x7f08400cf340 2023-10-19T14:18:51.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.123+0000 7f0844b6e640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08401f21b0 con 0x7f084015b2a0 2023-10-19T14:18:51.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.123+0000 7f0844b6e640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08400c9ef0 con 0x7f08400d1910 2023-10-19T14:18:51.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.123+0000 7f083e59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f08400cf340 0x7f08400c9390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:51.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.123+0000 7f083e59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f08400cf340 0x7f08400c9390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:36900/0 (socket says 172.21.15.57:36900) 2023-10-19T14:18:51.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.123+0000 7f083e59c640 1 -- 172.21.15.57:0/2291750519 learned_addr learned my addr 172.21.15.57:0/2291750519 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:18:51.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.124+0000 7f0835d9b640 1 --2- 172.21.15.57:0/2291750519 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f08400d1910 0x7f08401f1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:51.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.124+0000 7f083ed9d640 1 --2- 172.21.15.57:0/2291750519 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f084015b2a0 0x7f084015de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:51.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.124+0000 7f083e59c640 1 -- 172.21.15.57:0/2291750519 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f08400d1910 msgr2=0x7f08401f1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:51.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.124+0000 7f083e59c640 1 --2- 172.21.15.57:0/2291750519 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f08400d1910 0x7f08401f1c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:51.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.124+0000 7f083e59c640 1 -- 172.21.15.57:0/2291750519 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f084015b2a0 msgr2=0x7f084015de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:51.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.124+0000 7f083e59c640 1 --2- 172.21.15.57:0/2291750519 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f084015b2a0 0x7f084015de50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:51.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.124+0000 7f083e59c640 1 -- 172.21.15.57:0/2291750519 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0838009840 con 0x7f08400cf340 2023-10-19T14:18:51.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.124+0000 7f083e59c640 1 --2- 172.21.15.57:0/2291750519 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f08400cf340 0x7f08400c9390 secure :-1 s=READY pgs=2239 cs=0 l=1 rev1=1 crypto rx=0x7f083000c450 tx=0x7f083000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:51.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.124+0000 7f08377fe640 1 -- 172.21.15.57:0/2291750519 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0830012830 con 0x7f08400cf340 2023-10-19T14:18:51.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.124+0000 7f08377fe640 1 -- 172.21.15.57:0/2291750519 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f0830012a80 con 0x7f08400cf340 2023-10-19T14:18:51.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.124+0000 7f0844b6e640 1 -- 172.21.15.57:0/2291750519 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08400c82f0 con 0x7f08400cf340 2023-10-19T14:18:51.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.124+0000 7f08377fe640 1 -- 172.21.15.57:0/2291750519 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0830012e50 con 0x7f08400cf340 2023-10-19T14:18:51.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.124+0000 7f0844b6e640 1 -- 172.21.15.57:0/2291750519 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08400c88b0 con 0x7f08400cf340 2023-10-19T14:18:51.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.125+0000 7f0844b6e640 1 -- 172.21.15.57:0/2291750519 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08400690e0 con 0x7f08400cf340 2023-10-19T14:18:51.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.126+0000 7f08377fe640 1 -- 172.21.15.57:0/2291750519 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 120) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f083001b400 con 0x7f08400cf340 2023-10-19T14:18:51.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.126+0000 7f08377fe640 1 --2- 172.21.15.57:0/2291750519 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7f082006ace0 0x7f082006d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:51.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.126+0000 7f08377fe640 1 -- 172.21.15.57:0/2291750519 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(206..206 src has 1..206) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f0830089a40 con 0x7f08400cf340 2023-10-19T14:18:51.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.126+0000 7f0835d9b640 1 --2- 172.21.15.57:0/2291750519 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7f082006ace0 0x7f082006d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:51.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.127+0000 7f0835d9b640 1 --2- 172.21.15.57:0/2291750519 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7f082006ace0 0x7f082006d1a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0828007fd0 tx=0x7f0828009040 comp rx=0 tx=0).ready entity=mgr.10231 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:51.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.130+0000 7f08377fe640 1 -- 172.21.15.57:0/2291750519 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0830054ae0 con 0x7f08400cf340 2023-10-19T14:18:51.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.327+0000 7f0844b6e640 1 -- 172.21.15.57:0/2291750519 --> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "path": "snap_test_dir1", "retention_spec_or_period": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f08400b1b80 con 0x7f082006ace0 2023-10-19T14:18:51.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.374+0000 7f08377fe640 1 -- 172.21.15.57:0/2291750519 <== mgr.10231 v2:172.21.15.179:6832/366917243 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+39 (secure 0 0 0) 0x7f08400b1b80 con 0x7f082006ace0 2023-10-19T14:18:51.377 INFO:teuthology.orchestra.run.smithi057.stdout:Retention added to path snap_test_dir1 2023-10-19T14:18:51.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.376+0000 7f0844b6e640 1 -- 172.21.15.57:0/2291750519 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7f082006ace0 msgr2=0x7f082006d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:51.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.376+0000 7f0844b6e640 1 --2- 172.21.15.57:0/2291750519 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7f082006ace0 0x7f082006d1a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0828007fd0 tx=0x7f0828009040 comp rx=0 tx=0).stop 2023-10-19T14:18:51.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.376+0000 7f0844b6e640 1 -- 172.21.15.57:0/2291750519 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f08400cf340 msgr2=0x7f08400c9390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:51.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.376+0000 7f0844b6e640 1 --2- 172.21.15.57:0/2291750519 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f08400cf340 0x7f08400c9390 secure :-1 s=READY pgs=2239 cs=0 l=1 rev1=1 crypto rx=0x7f083000c450 tx=0x7f083000c910 comp rx=0 tx=0).stop 2023-10-19T14:18:51.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.376+0000 7f0844b6e640 1 -- 172.21.15.57:0/2291750519 shutdown_connections 2023-10-19T14:18:51.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.376+0000 7f0844b6e640 1 --2- 172.21.15.57:0/2291750519 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f084015b2a0 0x7f084015de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:51.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.376+0000 7f0844b6e640 1 --2- 172.21.15.57:0/2291750519 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7f082006ace0 0x7f082006d1a0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:51.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.376+0000 7f0844b6e640 1 --2- 172.21.15.57:0/2291750519 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f08400d1910 0x7f08401f1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:51.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.376+0000 7f0844b6e640 1 --2- 172.21.15.57:0/2291750519 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f08400cf340 0x7f08400c9390 unknown :-1 s=CLOSED pgs=2239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:51.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.376+0000 7f0844b6e640 1 -- 172.21.15.57:0/2291750519 >> 172.21.15.57:0/2291750519 conn(0x7f08400d0d60 msgr2=0x7f084015a4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:51.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.376+0000 7f0844b6e640 1 -- 172.21.15.57:0/2291750519 shutdown_connections 2023-10-19T14:18:51.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.376+0000 7f0844b6e640 1 -- 172.21.15.57:0/2291750519 wait complete. 2023-10-19T14:18:51.389 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention added to path snap_test_dir1 2023-10-19T14:18:51.390 DEBUG:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2023-10-19-14_19_00 in ~9s... 2023-10-19T14:18:51.390 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1m'], expected_retention: [{'m': 1}] 2023-10-19T14:18:51.390 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --format json snap_test_dir1 2023-10-19T14:18:51.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.499+0000 7f727ace5640 1 -- 172.21.15.57:0/2701926418 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f727415b2a0 msgr2=0x7f72740cf720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:51.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.499+0000 7f727ace5640 1 --2- 172.21.15.57:0/2701926418 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f727415b2a0 0x7f72740cf720 secure :-1 s=READY pgs=2241 cs=0 l=1 rev1=1 crypto rx=0x7f7264009830 tx=0x7f726402bcd0 comp rx=0 tx=0).stop 2023-10-19T14:18:51.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.499+0000 7f727ace5640 1 -- 172.21.15.57:0/2701926418 shutdown_connections 2023-10-19T14:18:51.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.499+0000 7f727ace5640 1 --2- 172.21.15.57:0/2701926418 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f727415b2a0 0x7f72740cf720 unknown :-1 s=CLOSED pgs=2241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:51.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.499+0000 7f727ace5640 1 --2- 172.21.15.57:0/2701926418 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f72740cf340 0x7f72740d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:51.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.499+0000 7f727ace5640 1 -- 172.21.15.57:0/2701926418 >> 172.21.15.57:0/2701926418 conn(0x7f72740d0d60 msgr2=0x7f72740d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:51.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.499+0000 7f727ace5640 1 -- 172.21.15.57:0/2701926418 shutdown_connections 2023-10-19T14:18:51.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.499+0000 7f727ace5640 1 -- 172.21.15.57:0/2701926418 wait complete. 2023-10-19T14:18:51.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.500+0000 7f727ace5640 1 Processor -- start 2023-10-19T14:18:51.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.500+0000 7f727ace5640 1 -- start start 2023-10-19T14:18:51.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.501+0000 7f727ace5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f72740ceb40 0x7f72740c9e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:51.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.501+0000 7f727ace5640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f72740cf340 0x7f72740c7e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:51.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.501+0000 7f727ace5640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f727415b2a0 0x7f72741f0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:51.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.501+0000 7f727ace5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72740c9490 con 0x7f72740ceb40 2023-10-19T14:18:51.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.501+0000 7f727ace5640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72740c8880 con 0x7f72740cf340 2023-10-19T14:18:51.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.501+0000 7f727ace5640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72740c8b60 con 0x7f727415b2a0 2023-10-19T14:18:51.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.501+0000 7f7278a81640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f72740ceb40 0x7f72740c9e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:51.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.501+0000 7f7278a81640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f72740ceb40 0x7f72740c9e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:36928/0 (socket says 172.21.15.57:36928) 2023-10-19T14:18:51.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.501+0000 7f7278a81640 1 -- 172.21.15.57:0/2285156256 learned_addr learned my addr 172.21.15.57:0/2285156256 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:18:51.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.502+0000 7f726bfff640 1 --2- 172.21.15.57:0/2285156256 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f72740cf340 0x7f72740c7e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:51.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.502+0000 7f7278a81640 1 -- 172.21.15.57:0/2285156256 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f727415b2a0 msgr2=0x7f72741f0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:51.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.502+0000 7f7279282640 1 --2- 172.21.15.57:0/2285156256 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f727415b2a0 0x7f72741f0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:51.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.502+0000 7f7278a81640 1 --2- 172.21.15.57:0/2285156256 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f727415b2a0 0x7f72741f0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:51.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.502+0000 7f7278a81640 1 -- 172.21.15.57:0/2285156256 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f72740cf340 msgr2=0x7f72740c7e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:51.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.502+0000 7f7278a81640 1 --2- 172.21.15.57:0/2285156256 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f72740cf340 0x7f72740c7e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:51.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.502+0000 7f7278a81640 1 -- 172.21.15.57:0/2285156256 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7264009490 con 0x7f72740ceb40 2023-10-19T14:18:51.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.502+0000 7f726bfff640 1 --2- 172.21.15.57:0/2285156256 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f72740cf340 0x7f72740c7e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:18:51.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.502+0000 7f7279282640 1 --2- 172.21.15.57:0/2285156256 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f727415b2a0 0x7f72741f0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:18:51.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.502+0000 7f7278a81640 1 --2- 172.21.15.57:0/2285156256 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f72740ceb40 0x7f72740c9e80 secure :-1 s=READY pgs=2242 cs=0 l=1 rev1=1 crypto rx=0x7f727000c890 tx=0x7f727000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:51.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.502+0000 7f72697fa640 1 -- 172.21.15.57:0/2285156256 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7270004220 con 0x7f72740ceb40 2023-10-19T14:18:51.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.502+0000 7f72697fa640 1 -- 172.21.15.57:0/2285156256 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f7270004470 con 0x7f72740ceb40 2023-10-19T14:18:51.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.502+0000 7f72697fa640 1 -- 172.21.15.57:0/2285156256 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7270007c10 con 0x7f72740ceb40 2023-10-19T14:18:51.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.502+0000 7f727ace5640 1 -- 172.21.15.57:0/2285156256 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72740c8e40 con 0x7f72740ceb40 2023-10-19T14:18:51.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.502+0000 7f727ace5640 1 -- 172.21.15.57:0/2285156256 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72741504e0 con 0x7f72740ceb40 2023-10-19T14:18:51.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.503+0000 7f727ace5640 1 -- 172.21.15.57:0/2285156256 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72740690e0 con 0x7f72740ceb40 2023-10-19T14:18:51.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.503+0000 7f72697fa640 1 -- 172.21.15.57:0/2285156256 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 120) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f7270005020 con 0x7f72740ceb40 2023-10-19T14:18:51.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.504+0000 7f72697fa640 1 --2- 172.21.15.57:0/2285156256 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7f726006ace0 0x7f726006d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:18:51.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.504+0000 7f72697fa640 1 -- 172.21.15.57:0/2285156256 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(206..206 src has 1..206) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f7270089ac0 con 0x7f72740ceb40 2023-10-19T14:18:51.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.504+0000 7f726bfff640 1 --2- 172.21.15.57:0/2285156256 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7f726006ace0 0x7f726006d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:18:51.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.504+0000 7f726bfff640 1 --2- 172.21.15.57:0/2285156256 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7f726006ace0 0x7f726006d1a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f7264036000 tx=0x7f7264004730 comp rx=0 tx=0).ready entity=mgr.10231 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:18:51.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.508+0000 7f72697fa640 1 -- 172.21.15.57:0/2285156256 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7270054bf0 con 0x7f72740ceb40 2023-10-19T14:18:51.707 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.704+0000 7f727ace5640 1 -- 172.21.15.57:0/2285156256 --> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f72740cf0f0 con 0x7f726006ace0 2023-10-19T14:18:51.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.711+0000 7f72697fa640 1 -- 172.21.15.57:0/2285156256 <== mgr.10231 v2:172.21.15.179:6832/366917243 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+72 (secure 0 0 0) 0x7f7264004140 con 0x7f726006ace0 2023-10-19T14:18:51.714 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:18:51.714 INFO:teuthology.orchestra.run.smithi057.stdout:{"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{"m": 1}]} 2023-10-19T14:18:51.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.712+0000 7f727ace5640 1 -- 172.21.15.57:0/2285156256 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7f726006ace0 msgr2=0x7f726006d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:51.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.712+0000 7f727ace5640 1 --2- 172.21.15.57:0/2285156256 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7f726006ace0 0x7f726006d1a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f7264036000 tx=0x7f7264004730 comp rx=0 tx=0).stop 2023-10-19T14:18:51.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.713+0000 7f727ace5640 1 -- 172.21.15.57:0/2285156256 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f72740ceb40 msgr2=0x7f72740c9e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:18:51.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.713+0000 7f727ace5640 1 --2- 172.21.15.57:0/2285156256 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f72740ceb40 0x7f72740c9e80 secure :-1 s=READY pgs=2242 cs=0 l=1 rev1=1 crypto rx=0x7f727000c890 tx=0x7f727000cd50 comp rx=0 tx=0).stop 2023-10-19T14:18:51.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.713+0000 7f727ace5640 1 -- 172.21.15.57:0/2285156256 shutdown_connections 2023-10-19T14:18:51.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.713+0000 7f727ace5640 1 --2- 172.21.15.57:0/2285156256 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7f726006ace0 0x7f726006d1a0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:51.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.713+0000 7f727ace5640 1 --2- 172.21.15.57:0/2285156256 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f727415b2a0 0x7f72741f0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:51.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.713+0000 7f727ace5640 1 --2- 172.21.15.57:0/2285156256 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f72740cf340 0x7f72740c7e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:51.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.713+0000 7f727ace5640 1 --2- 172.21.15.57:0/2285156256 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f72740ceb40 0x7f72740c9e80 unknown :-1 s=CLOSED pgs=2242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:18:51.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.713+0000 7f727ace5640 1 -- 172.21.15.57:0/2285156256 >> 172.21.15.57:0/2285156256 conn(0x7f72740d0d60 msgr2=0x7f72740ae180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:18:51.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.713+0000 7f727ace5640 1 -- 172.21.15.57:0/2285156256 shutdown_connections 2023-10-19T14:18:51.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:18:51.713+0000 7f727ace5640 1 -- 172.21.15.57:0/2285156256 wait complete. 2023-10-19T14:18:51.726 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{"m": 1}]} 2023-10-19T14:18:51.727 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': 'snap_test_dir1', 'schedule': ['1m'], 'retention': [{'m': 1}]} 2023-10-19T14:18:51.727 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:18:51.727 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T14:18:51.747 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T14:18:51.747 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T14:18:51.747 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T14:18:52.749 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:18:52.749 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T14:18:52.773 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T14:18:52.774 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T14:18:52.774 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T14:18:53.775 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:18:53.776 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T14:18:53.800 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T14:18:53.800 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T14:18:53.801 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T14:18:54.802 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:18:54.802 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T14:18:54.827 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T14:18:54.827 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T14:18:54.828 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T14:18:55.829 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:18:55.830 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T14:18:55.854 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T14:18:55.854 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T14:18:55.855 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T14:18:56.856 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:18:56.857 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T14:18:56.881 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T14:18:56.881 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T14:18:56.882 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T14:18:57.882 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:18:57.883 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T14:18:57.908 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T14:18:57.909 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T14:18:57.909 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T14:18:58.910 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:18:58.911 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T14:18:58.935 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T14:18:58.935 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T14:18:58.936 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T14:18:59.937 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:18:59.938 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T14:18:59.962 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-19T14:18:59.962 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T14:18:59.963 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T14:19:00.387 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:19:00.384+0000 7f09b9ff1640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:19:00.387 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:19:00.384+0000 7f09b9ff1640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:19:00.387 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:19:00.384+0000 7f09b9ff1640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:19:00.388 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:19:00.384+0000 7f09b9ff1640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:19:00.388 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:19:00.384+0000 7f09b9ff1640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:19:00.964 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:19:00.965 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T14:19:00.988 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_19_00_UTC 2023-10-19T14:19:00.989 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-14_19_00_UTC'} 2023-10-19T14:19:00.989 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2023-10-19-14_19_00_UTC'} 2023-10-19T14:19:00.989 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T14:19:00.990 DEBUG:tasks.cephfs.test_snap_schedules:snapshots added=['scheduled-2023-10-19-14_19_00_UTC'] 2023-10-19T14:19:00.990 DEBUG:tasks.cephfs.test_snap_schedules:exec=1697725131.3901973, now = 1697725140.990377, timo = 9 2023-10-19T14:19:01.991 DEBUG:tasks.cephfs.test_snap_schedules:expecting removal of snap snap_test_dir1/.snap/scheduled-2023-10-19-14_19_00 in ~69s... 2023-10-19T14:19:01.992 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:19:01.992 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T14:19:02.016 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_19_00_UTC 2023-10-19T14:19:02.016 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-14_19_00_UTC'} 2023-10-19T14:19:02.017 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T14:19:02.017 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T14:19:03.018 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:19:03.019 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T14:19:03.042 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_19_00_UTC 2023-10-19T14:19:03.043 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-14_19_00_UTC'} 2023-10-19T14:19:03.044 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T14:19:03.044 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T14:19:04.045 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:19:04.045 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T14:19:04.070 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_19_00_UTC 2023-10-19T14:19:04.070 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-14_19_00_UTC'} 2023-10-19T14:19:04.070 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T14:19:04.071 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T14:19:05.072 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:19:05.072 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T14:19:05.095 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_19_00_UTC 2023-10-19T14:19:05.096 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-14_19_00_UTC'} 2023-10-19T14:19:05.097 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T14:19:05.097 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T14:19:05.873 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:19:05.876 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:19:06.098 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:19:06.099 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T14:19:06.122 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_19_00_UTC 2023-10-19T14:19:06.123 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-14_19_00_UTC'} 2023-10-19T14:19:06.123 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T14:19:06.123 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T14:19:07.125 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:19:07.125 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T14:19:07.149 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_19_00_UTC 2023-10-19T14:19:07.150 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-14_19_00_UTC'} 2023-10-19T14:19:07.150 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T14:19:07.150 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T14:19:08.151 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:19:08.152 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T14:19:08.175 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_19_00_UTC 2023-10-19T14:19:08.176 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-14_19_00_UTC'} 2023-10-19T14:19:08.176 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T14:19:08.177 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T14:19:09.178 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:19:09.179 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T14:19:09.202 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_19_00_UTC 2023-10-19T14:19:09.203 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-14_19_00_UTC'} 2023-10-19T14:19:09.203 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T14:19:09.204 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T14:19:10.205 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:19:10.206 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T14:19:10.228 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_19_00_UTC 2023-10-19T14:19:10.229 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-14_19_00_UTC'} 2023-10-19T14:19:10.230 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T14:19:10.230 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T14:19:11.231 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:19:11.232 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T14:19:11.255 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_19_00_UTC 2023-10-19T14:19:11.256 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-14_19_00_UTC'} 2023-10-19T14:19:11.256 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T14:19:11.256 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T14:19:12.257 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:19:12.258 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T14:19:12.281 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_19_00_UTC 2023-10-19T14:19:12.282 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-14_19_00_UTC'} 2023-10-19T14:19:12.282 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T14:19:12.282 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T14:19:13.284 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:19:13.285 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T14:19:13.308 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_19_00_UTC 2023-10-19T14:19:13.309 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-14_19_00_UTC'} 2023-10-19T14:19:13.309 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T14:19:13.310 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T14:19:14.311 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:19:14.311 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T14:19:14.335 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_19_00_UTC 2023-10-19T14:19:14.335 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-14_19_00_UTC'} 2023-10-19T14:19:14.335 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T14:19:14.336 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T14:19:15.336 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:19:15.337 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T14:19:15.362 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_19_00_UTC 2023-10-19T14:19:15.363 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-14_19_00_UTC'} 2023-10-19T14:19:15.364 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T14:19:15.364 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T14:19:16.365 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:19:16.366 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T14:19:16.390 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_19_00_UTC 2023-10-19T14:19:16.391 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-14_19_00_UTC'} 2023-10-19T14:19:16.391 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T14:19:16.392 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T14:19:17.394 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:19:17.394 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T14:19:17.418 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_19_00_UTC 2023-10-19T14:19:17.419 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-14_19_00_UTC'} 2023-10-19T14:19:17.419 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T14:19:17.419 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T14:19:18.421 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:19:18.421 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T14:19:18.445 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_19_00_UTC 2023-10-19T14:19:18.446 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-14_19_00_UTC'} 2023-10-19T14:19:18.446 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T14:19:18.446 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T14:19:19.448 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:19:19.448 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T14:19:19.472 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_19_00_UTC 2023-10-19T14:19:19.473 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-14_19_00_UTC'} 2023-10-19T14:19:19.474 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T14:19:19.474 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T14:19:20.475 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:19:20.476 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T14:19:20.499 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_19_00_UTC 2023-10-19T14:19:20.500 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-14_19_00_UTC'} 2023-10-19T14:19:20.500 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T14:19:20.501 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T14:19:21.502 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:19:21.502 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T14:19:21.525 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_19_00_UTC 2023-10-19T14:19:21.527 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-14_19_00_UTC'} 2023-10-19T14:19:21.527 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T14:19:21.527 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T14:19:22.529 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:19:22.529 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T14:19:22.553 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_19_00_UTC 2023-10-19T14:19:22.554 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-14_19_00_UTC'} 2023-10-19T14:19:22.554 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T14:19:22.555 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T14:19:23.556 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:19:23.557 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T14:19:23.580 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_19_00_UTC 2023-10-19T14:19:23.581 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-14_19_00_UTC'} 2023-10-19T14:19:23.582 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T14:19:23.582 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T14:19:24.583 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:19:24.583 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T14:19:24.607 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_19_00_UTC 2023-10-19T14:19:24.607 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-14_19_00_UTC'} 2023-10-19T14:19:24.607 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T14:19:24.608 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T14:19:25.609 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:19:25.609 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T14:19:25.633 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_19_00_UTC 2023-10-19T14:19:25.633 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-14_19_00_UTC'} 2023-10-19T14:19:25.634 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T14:19:25.634 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T14:19:26.636 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:19:26.636 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T14:19:26.660 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_19_00_UTC 2023-10-19T14:19:26.660 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-14_19_00_UTC'} 2023-10-19T14:19:26.660 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T14:19:26.661 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T14:19:27.662 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:19:27.663 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T14:19:27.686 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_19_00_UTC 2023-10-19T14:19:27.687 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-14_19_00_UTC'} 2023-10-19T14:19:27.688 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T14:19:27.688 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T14:19:28.689 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:19:28.689 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T14:19:28.712 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_19_00_UTC 2023-10-19T14:19:28.714 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-14_19_00_UTC'} 2023-10-19T14:19:28.714 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T14:19:28.714 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T14:19:29.716 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:19:29.716 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T14:19:29.739 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_19_00_UTC 2023-10-19T14:19:29.740 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-14_19_00_UTC'} 2023-10-19T14:19:29.740 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T14:19:29.740 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T14:19:30.742 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:19:30.742 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T14:19:30.766 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_19_00_UTC 2023-10-19T14:19:30.766 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-14_19_00_UTC'} 2023-10-19T14:19:30.767 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T14:19:30.767 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T14:19:31.768 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:19:31.768 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T14:19:31.792 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_19_00_UTC 2023-10-19T14:19:31.792 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-14_19_00_UTC'} 2023-10-19T14:19:31.793 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T14:19:31.793 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T14:19:32.794 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:19:32.795 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T14:19:32.819 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_19_00_UTC 2023-10-19T14:19:32.819 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-14_19_00_UTC'} 2023-10-19T14:19:32.819 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T14:19:32.820 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T14:19:33.820 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:19:33.821 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T14:19:33.844 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_19_00_UTC 2023-10-19T14:19:33.845 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-14_19_00_UTC'} 2023-10-19T14:19:33.845 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T14:19:33.846 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T14:19:34.847 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:19:34.848 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T14:19:34.871 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_19_00_UTC 2023-10-19T14:19:34.872 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-14_19_00_UTC'} 2023-10-19T14:19:34.872 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T14:19:34.873 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T14:19:35.874 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:19:35.874 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T14:19:35.898 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_19_00_UTC 2023-10-19T14:19:35.898 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-14_19_00_UTC'} 2023-10-19T14:19:35.899 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T14:19:35.899 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T14:19:35.994 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:19:35.997 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:19:36.900 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:19:36.901 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T14:19:36.925 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_19_00_UTC 2023-10-19T14:19:36.925 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-14_19_00_UTC'} 2023-10-19T14:19:36.925 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T14:19:36.926 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T14:19:37.927 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:19:37.928 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T14:19:37.955 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_19_00_UTC 2023-10-19T14:19:37.955 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-14_19_00_UTC'} 2023-10-19T14:19:37.955 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T14:19:37.956 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T14:19:38.957 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:19:38.957 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T14:19:38.980 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_19_00_UTC 2023-10-19T14:19:38.981 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-14_19_00_UTC'} 2023-10-19T14:19:38.981 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T14:19:38.982 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T14:19:39.983 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:19:39.984 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T14:19:40.007 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_19_00_UTC 2023-10-19T14:19:40.008 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-14_19_00_UTC'} 2023-10-19T14:19:40.008 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T14:19:40.008 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T14:19:41.009 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:19:41.010 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T14:19:41.033 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_19_00_UTC 2023-10-19T14:19:41.034 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-14_19_00_UTC'} 2023-10-19T14:19:41.034 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T14:19:41.034 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T14:19:42.036 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:19:42.037 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T14:19:42.060 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_19_00_UTC 2023-10-19T14:19:42.060 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-14_19_00_UTC'} 2023-10-19T14:19:42.061 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T14:19:42.061 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T14:19:43.062 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:19:43.063 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T14:19:43.087 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_19_00_UTC 2023-10-19T14:19:43.087 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-14_19_00_UTC'} 2023-10-19T14:19:43.087 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T14:19:43.088 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T14:19:44.089 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:19:44.090 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T14:19:44.114 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_19_00_UTC 2023-10-19T14:19:44.114 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-14_19_00_UTC'} 2023-10-19T14:19:44.115 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T14:19:44.115 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T14:19:45.116 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:19:45.116 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T14:19:45.140 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_19_00_UTC 2023-10-19T14:19:45.140 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-14_19_00_UTC'} 2023-10-19T14:19:45.141 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T14:19:45.141 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T14:19:46.142 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:19:46.143 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T14:19:46.166 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_19_00_UTC 2023-10-19T14:19:46.166 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-14_19_00_UTC'} 2023-10-19T14:19:46.167 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T14:19:46.167 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T14:19:47.168 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:19:47.168 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T14:19:47.192 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_19_00_UTC 2023-10-19T14:19:47.192 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-14_19_00_UTC'} 2023-10-19T14:19:47.192 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T14:19:47.193 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T14:19:48.194 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:19:48.194 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T14:19:48.218 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_19_00_UTC 2023-10-19T14:19:48.219 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-14_19_00_UTC'} 2023-10-19T14:19:48.219 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T14:19:48.219 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T14:19:49.220 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:19:49.221 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T14:19:49.245 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_19_00_UTC 2023-10-19T14:19:49.245 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-14_19_00_UTC'} 2023-10-19T14:19:49.245 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T14:19:49.246 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T14:19:50.247 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:19:50.247 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T14:19:50.271 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_19_00_UTC 2023-10-19T14:19:50.272 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-14_19_00_UTC'} 2023-10-19T14:19:50.272 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T14:19:50.272 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T14:19:51.273 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:19:51.274 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T14:19:51.297 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_19_00_UTC 2023-10-19T14:19:51.298 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-14_19_00_UTC'} 2023-10-19T14:19:51.298 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T14:19:51.298 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T14:19:52.300 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:19:52.300 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T14:19:52.323 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_19_00_UTC 2023-10-19T14:19:52.324 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-14_19_00_UTC'} 2023-10-19T14:19:52.324 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T14:19:52.325 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T14:19:53.326 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:19:53.326 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T14:19:53.350 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_19_00_UTC 2023-10-19T14:19:53.350 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-14_19_00_UTC'} 2023-10-19T14:19:53.350 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T14:19:53.351 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T14:19:54.352 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:19:54.353 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T14:19:54.376 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_19_00_UTC 2023-10-19T14:19:54.376 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-14_19_00_UTC'} 2023-10-19T14:19:54.377 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T14:19:54.377 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T14:19:55.378 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:19:55.379 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T14:19:55.402 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_19_00_UTC 2023-10-19T14:19:55.403 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-14_19_00_UTC'} 2023-10-19T14:19:55.403 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T14:19:55.403 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T14:19:56.404 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:19:56.405 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T14:19:56.427 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_19_00_UTC 2023-10-19T14:19:56.428 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-14_19_00_UTC'} 2023-10-19T14:19:56.429 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T14:19:56.429 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T14:19:57.430 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:19:57.431 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T14:19:57.454 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_19_00_UTC 2023-10-19T14:19:57.455 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-14_19_00_UTC'} 2023-10-19T14:19:57.455 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T14:19:57.455 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T14:19:58.457 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:19:58.457 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T14:19:58.481 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_19_00_UTC 2023-10-19T14:19:58.483 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-14_19_00_UTC'} 2023-10-19T14:19:58.483 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T14:19:58.483 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T14:19:59.485 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:19:59.486 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T14:19:59.511 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_19_00_UTC 2023-10-19T14:19:59.511 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-14_19_00_UTC'} 2023-10-19T14:19:59.512 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-19T14:19:59.512 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-19T14:20:00.513 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:20:00.514 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T14:20:00.535 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_20_00_UTC 2023-10-19T14:20:00.536 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-19-14_20_00_UTC'} 2023-10-19T14:20:00.537 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2023-10-19-14_20_00_UTC'} 2023-10-19T14:20:00.537 INFO:tasks.cephfs.test_snap_schedules:removed: {'scheduled-2023-10-19-14_19_00_UTC'} 2023-10-19T14:20:00.537 DEBUG:tasks.cephfs.test_snap_schedules:snapshots removed=['scheduled-2023-10-19-14_19_00_UTC'] 2023-10-19T14:20:00.538 DEBUG:tasks.cephfs.test_snap_schedules:exec=1697725131.3901973, now = 1697725200.5381722, timo = 69 2023-10-19T14:20:01.539 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove snap_test_dir1 2023-10-19T14:20:01.665 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:01.661+0000 7f2604c32640 1 -- 172.21.15.57:0/2324933280 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f26000cbd20 msgr2=0x7f2600158300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:01.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:01.661+0000 7f2604c32640 1 --2- 172.21.15.57:0/2324933280 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f26000cbd20 0x7f2600158300 secure :-1 s=READY pgs=2243 cs=0 l=1 rev1=1 crypto rx=0x7f25f8005fd0 tx=0x7f25f802bd90 comp rx=0 tx=0).stop 2023-10-19T14:20:01.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:01.662+0000 7f2604c32640 1 -- 172.21.15.57:0/2324933280 shutdown_connections 2023-10-19T14:20:01.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:01.662+0000 7f2604c32640 1 --2- 172.21.15.57:0/2324933280 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f26000afa20 0x7f26000705a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:01.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:01.662+0000 7f2604c32640 1 --2- 172.21.15.57:0/2324933280 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f26000cbd20 0x7f2600158300 unknown :-1 s=CLOSED pgs=2243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:01.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:01.662+0000 7f2604c32640 1 -- 172.21.15.57:0/2324933280 >> 172.21.15.57:0/2324933280 conn(0x7f2600069930 msgr2=0x7f2600069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:01.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:01.662+0000 7f2604c32640 1 -- 172.21.15.57:0/2324933280 shutdown_connections 2023-10-19T14:20:01.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:01.662+0000 7f2604c32640 1 -- 172.21.15.57:0/2324933280 wait complete. 2023-10-19T14:20:01.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:01.663+0000 7f2604c32640 1 Processor -- start 2023-10-19T14:20:01.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:01.663+0000 7f2604c32640 1 -- start start 2023-10-19T14:20:01.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:01.664+0000 7f2604c32640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f26000afa20 0x7f26000ac8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:01.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:01.664+0000 7f2604c32640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f26000b0530 0x7f2600142020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:01.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:01.664+0000 7f2604c32640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f26000cbd20 0x7f2600142560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:01.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:01.664+0000 7f2604c32640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2600151020 con 0x7f26000b0530 2023-10-19T14:20:01.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:01.664+0000 7f2604c32640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f260014ecf0 con 0x7f26000afa20 2023-10-19T14:20:01.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:01.664+0000 7f2604c32640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f260015a9b0 con 0x7f26000cbd20 2023-10-19T14:20:01.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:01.664+0000 7f25fdd9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f26000b0530 0x7f2600142020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:01.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:01.664+0000 7f25fdd9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f26000b0530 0x7f2600142020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:47874/0 (socket says 172.21.15.57:47874) 2023-10-19T14:20:01.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:01.664+0000 7f25fdd9b640 1 -- 172.21.15.57:0/2828989772 learned_addr learned my addr 172.21.15.57:0/2828989772 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:01.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:01.664+0000 7f25fed9d640 1 --2- 172.21.15.57:0/2828989772 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f26000cbd20 0x7f2600142560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:01.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:01.664+0000 7f25fe59c640 1 --2- 172.21.15.57:0/2828989772 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f26000afa20 0x7f26000ac8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:01.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:01.664+0000 7f25fdd9b640 1 -- 172.21.15.57:0/2828989772 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f26000cbd20 msgr2=0x7f2600142560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:01.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:01.664+0000 7f25fdd9b640 1 --2- 172.21.15.57:0/2828989772 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f26000cbd20 0x7f2600142560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:01.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:01.664+0000 7f25fdd9b640 1 -- 172.21.15.57:0/2828989772 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f26000afa20 msgr2=0x7f26000ac8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:01.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:01.664+0000 7f25fdd9b640 1 --2- 172.21.15.57:0/2828989772 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f26000afa20 0x7f26000ac8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:01.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:01.664+0000 7f25fdd9b640 1 -- 172.21.15.57:0/2828989772 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25f8009560 con 0x7f26000b0530 2023-10-19T14:20:01.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:01.664+0000 7f25fed9d640 1 --2- 172.21.15.57:0/2828989772 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f26000cbd20 0x7f2600142560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:20:01.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:01.664+0000 7f25fe59c640 1 --2- 172.21.15.57:0/2828989772 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f26000afa20 0x7f26000ac8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:20:01.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:01.665+0000 7f25fdd9b640 1 --2- 172.21.15.57:0/2828989772 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f26000b0530 0x7f2600142020 secure :-1 s=READY pgs=2244 cs=0 l=1 rev1=1 crypto rx=0x7f25e800dbc0 tx=0x7f25e800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:01.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:01.665+0000 7f25f6ffd640 1 -- 172.21.15.57:0/2828989772 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25e8005220 con 0x7f26000b0530 2023-10-19T14:20:01.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:01.665+0000 7f25f6ffd640 1 -- 172.21.15.57:0/2828989772 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f25e8005470 con 0x7f26000b0530 2023-10-19T14:20:01.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:01.665+0000 7f2604c32640 1 -- 172.21.15.57:0/2828989772 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2600070cf0 con 0x7f26000b0530 2023-10-19T14:20:01.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:01.665+0000 7f25f6ffd640 1 -- 172.21.15.57:0/2828989772 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25e8004a00 con 0x7f26000b0530 2023-10-19T14:20:01.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:01.665+0000 7f2604c32640 1 -- 172.21.15.57:0/2828989772 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2600147ae0 con 0x7f26000b0530 2023-10-19T14:20:01.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:01.666+0000 7f2604c32640 1 -- 172.21.15.57:0/2828989772 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2600147670 con 0x7f26000b0530 2023-10-19T14:20:01.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:01.666+0000 7f25f6ffd640 1 -- 172.21.15.57:0/2828989772 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 123) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f25e8004b60 con 0x7f26000b0530 2023-10-19T14:20:01.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:01.667+0000 7f25f6ffd640 1 --2- 172.21.15.57:0/2828989772 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7f25e006b0d0 0x7f25e006d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:01.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:01.667+0000 7f25f6ffd640 1 -- 172.21.15.57:0/2828989772 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 6488+0+0 (secure 0 0 0) 0x7f25e808a560 con 0x7f26000b0530 2023-10-19T14:20:01.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:01.667+0000 7f25fe59c640 1 --2- 172.21.15.57:0/2828989772 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7f25e006b0d0 0x7f25e006d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:01.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:01.668+0000 7f25fe59c640 1 --2- 172.21.15.57:0/2828989772 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7f25e006b0d0 0x7f25e006d590 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f25f8035000 tx=0x7f25f8004730 comp rx=0 tx=0).ready entity=mgr.10231 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:01.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:01.672+0000 7f25f6ffd640 1 -- 172.21.15.57:0/2828989772 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f25e8055030 con 0x7f26000b0530 2023-10-19T14:20:01.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:01.870+0000 7f2604c32640 1 -- 172.21.15.57:0/2828989772 --> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f26000cc1e0 con 0x7f25e006b0d0 2023-10-19T14:20:01.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:01.936+0000 7f25f6ffd640 1 -- 172.21.15.57:0/2828989772 <== mgr.10231 v2:172.21.15.179:6832/366917243 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f26000cc1e0 con 0x7f25e006b0d0 2023-10-19T14:20:01.940 INFO:teuthology.orchestra.run.smithi057.stdout:Schedule removed for path snap_test_dir1 2023-10-19T14:20:01.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:01.938+0000 7f2604c32640 1 -- 172.21.15.57:0/2828989772 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7f25e006b0d0 msgr2=0x7f25e006d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:01.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:01.938+0000 7f2604c32640 1 --2- 172.21.15.57:0/2828989772 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7f25e006b0d0 0x7f25e006d590 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f25f8035000 tx=0x7f25f8004730 comp rx=0 tx=0).stop 2023-10-19T14:20:01.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:01.938+0000 7f2604c32640 1 -- 172.21.15.57:0/2828989772 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f26000b0530 msgr2=0x7f2600142020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:01.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:01.938+0000 7f2604c32640 1 --2- 172.21.15.57:0/2828989772 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f26000b0530 0x7f2600142020 secure :-1 s=READY pgs=2244 cs=0 l=1 rev1=1 crypto rx=0x7f25e800dbc0 tx=0x7f25e800a620 comp rx=0 tx=0).stop 2023-10-19T14:20:01.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:01.939+0000 7f2604c32640 1 -- 172.21.15.57:0/2828989772 shutdown_connections 2023-10-19T14:20:01.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:01.939+0000 7f2604c32640 1 --2- 172.21.15.57:0/2828989772 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7f25e006b0d0 0x7f25e006d590 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:01.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:01.939+0000 7f2604c32640 1 --2- 172.21.15.57:0/2828989772 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f26000cbd20 0x7f2600142560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:01.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:01.939+0000 7f2604c32640 1 --2- 172.21.15.57:0/2828989772 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f26000b0530 0x7f2600142020 unknown :-1 s=CLOSED pgs=2244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:01.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:01.939+0000 7f2604c32640 1 --2- 172.21.15.57:0/2828989772 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f26000afa20 0x7f26000ac8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:01.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:01.939+0000 7f2604c32640 1 -- 172.21.15.57:0/2828989772 >> 172.21.15.57:0/2828989772 conn(0x7f2600069930 msgr2=0x7f26000cad50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:01.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:01.939+0000 7f2604c32640 1 -- 172.21.15.57:0/2828989772 shutdown_connections 2023-10-19T14:20:01.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:01.939+0000 7f2604c32640 1 -- 172.21.15.57:0/2828989772 wait complete. 2023-10-19T14:20:01.952 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2023-10-19T14:20:01.952 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:20:01.953 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T14:20:01.971 INFO:teuthology.orchestra.run.smithi057.stdout:scheduled-2023-10-19-14_20_00_UTC 2023-10-19T14:20:01.972 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-19-14_20_00_UTC 2023-10-19T14:20:01.972 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:20:01.973 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-19-14_20_00_UTC) 2023-10-19T14:20:02.034 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:20:02.034 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2023-10-19T14:20:02.092 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-19T14:20:02.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.241+0000 7f1843c87640 1 -- 172.21.15.57:0/3826070422 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1824005180 msgr2=0x7f182400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:02.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.241+0000 7f1843c87640 1 --2- 172.21.15.57:0/3826070422 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1824005180 0x7f182400b980 secure :-1 s=READY pgs=2246 cs=0 l=1 rev1=1 crypto rx=0x7f182c001d50 tx=0x7f18240057f0 comp rx=0 tx=0).stop 2023-10-19T14:20:02.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.242+0000 7f1843c87640 1 -- 172.21.15.57:0/3826070422 shutdown_connections 2023-10-19T14:20:02.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.242+0000 7f1843c87640 1 --2- 172.21.15.57:0/3826070422 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1824005180 0x7f182400b980 unknown :-1 s=CLOSED pgs=2246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:02.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.242+0000 7f1843c87640 1 -- 172.21.15.57:0/3826070422 >> 172.21.15.57:0/3826070422 conn(0x7f183c06fae0 msgr2=0x7f183c06fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:02.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.242+0000 7f1843c87640 1 -- 172.21.15.57:0/3826070422 shutdown_connections 2023-10-19T14:20:02.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.242+0000 7f1843c87640 1 -- 172.21.15.57:0/3826070422 wait complete. 2023-10-19T14:20:02.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.243+0000 7f1843c87640 1 Processor -- start 2023-10-19T14:20:02.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.243+0000 7f1843c87640 1 -- start start 2023-10-19T14:20:02.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.243+0000 7f1843c87640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1824005180 0x7f183c14ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:02.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.243+0000 7f1843c87640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f183c148c40 0x7f183c06b980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:02.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.244+0000 7f1843c87640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f183c1490a0 0x7f183c06bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:02.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.244+0000 7f1843c87640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f183c149a70 con 0x7f183c1490a0 2023-10-19T14:20:02.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.244+0000 7f1843c87640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f183c149d50 con 0x7f183c148c40 2023-10-19T14:20:02.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.244+0000 7f1843c87640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f183c14a030 con 0x7f1824005180 2023-10-19T14:20:02.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.244+0000 7f1842224640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f183c1490a0 0x7f183c06bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:02.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.244+0000 7f1842224640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f183c1490a0 0x7f183c06bec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:47894/0 (socket says 172.21.15.57:47894) 2023-10-19T14:20:02.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.244+0000 7f1842224640 1 -- 172.21.15.57:0/4008303129 learned_addr learned my addr 172.21.15.57:0/4008303129 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:02.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.244+0000 7f1841a23640 1 --2- 172.21.15.57:0/4008303129 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1824005180 0x7f183c14ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:02.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.244+0000 7f1841222640 1 --2- 172.21.15.57:0/4008303129 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f183c148c40 0x7f183c06b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:02.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.244+0000 7f1842224640 1 -- 172.21.15.57:0/4008303129 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1824005180 msgr2=0x7f183c14ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:02.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.244+0000 7f1842224640 1 --2- 172.21.15.57:0/4008303129 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1824005180 0x7f183c14ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:02.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.244+0000 7f1842224640 1 -- 172.21.15.57:0/4008303129 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f183c148c40 msgr2=0x7f183c06b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:02.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.244+0000 7f1842224640 1 --2- 172.21.15.57:0/4008303129 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f183c148c40 0x7f183c06b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:02.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.244+0000 7f1842224640 1 -- 172.21.15.57:0/4008303129 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f183c0b2330 con 0x7f183c1490a0 2023-10-19T14:20:02.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.244+0000 7f1842224640 1 --2- 172.21.15.57:0/4008303129 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f183c1490a0 0x7f183c06bec0 secure :-1 s=READY pgs=2247 cs=0 l=1 rev1=1 crypto rx=0x7f182c001d50 tx=0x7f182c003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:02.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.244+0000 7f183a7fc640 1 -- 172.21.15.57:0/4008303129 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f182c0062d0 con 0x7f183c1490a0 2023-10-19T14:20:02.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.244+0000 7f183a7fc640 1 -- 172.21.15.57:0/4008303129 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f182c006520 con 0x7f183c1490a0 2023-10-19T14:20:02.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.244+0000 7f1843c87640 1 -- 172.21.15.57:0/4008303129 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f183c14a310 con 0x7f183c1490a0 2023-10-19T14:20:02.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.244+0000 7f183a7fc640 1 -- 172.21.15.57:0/4008303129 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f182c034410 con 0x7f183c1490a0 2023-10-19T14:20:02.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.244+0000 7f1843c87640 1 -- 172.21.15.57:0/4008303129 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f183c14b300 con 0x7f183c1490a0 2023-10-19T14:20:02.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.245+0000 7f1843c87640 1 -- 172.21.15.57:0/4008303129 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f183c14a830 con 0x7f183c1490a0 2023-10-19T14:20:02.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.246+0000 7f183a7fc640 1 -- 172.21.15.57:0/4008303129 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 123) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f182c034570 con 0x7f183c1490a0 2023-10-19T14:20:02.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.246+0000 7f183a7fc640 1 --2- 172.21.15.57:0/4008303129 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7f182806b100 0x7f182806d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:02.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.246+0000 7f183a7fc640 1 -- 172.21.15.57:0/4008303129 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 6488+0+0 (secure 0 0 0) 0x7f182c0ab280 con 0x7f183c1490a0 2023-10-19T14:20:02.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.247+0000 7f1841a23640 1 --2- 172.21.15.57:0/4008303129 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7f182806b100 0x7f182806d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:02.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.247+0000 7f1841a23640 1 --2- 172.21.15.57:0/4008303129 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7f182806b100 0x7f182806d5c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1834009a20 tx=0x7f1834005f90 comp rx=0 tx=0).ready entity=mgr.10231 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:02.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.251+0000 7f183a7fc640 1 -- 172.21.15.57:0/4008303129 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f183c14a830 con 0x7f183c1490a0 2023-10-19T14:20:02.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.474+0000 7f1843c87640 1 -- 172.21.15.57:0/4008303129 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f183c14a830 con 0x7f183c1490a0 2023-10-19T14:20:02.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.583+0000 7f183a7fc640 1 -- 172.21.15.57:0/4008303129 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v124) v1 ==== 93+0+0 (secure 0 0 0) 0x7f183c14a830 con 0x7f183c1490a0 2023-10-19T14:20:02.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.586+0000 7f1843c87640 1 -- 172.21.15.57:0/4008303129 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7f182806b100 msgr2=0x7f182806d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:02.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.586+0000 7f1843c87640 1 --2- 172.21.15.57:0/4008303129 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7f182806b100 0x7f182806d5c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1834009a20 tx=0x7f1834005f90 comp rx=0 tx=0).stop 2023-10-19T14:20:02.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.586+0000 7f1843c87640 1 -- 172.21.15.57:0/4008303129 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f183c1490a0 msgr2=0x7f183c06bec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:02.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.586+0000 7f1843c87640 1 --2- 172.21.15.57:0/4008303129 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f183c1490a0 0x7f183c06bec0 secure :-1 s=READY pgs=2247 cs=0 l=1 rev1=1 crypto rx=0x7f182c001d50 tx=0x7f182c003f10 comp rx=0 tx=0).stop 2023-10-19T14:20:02.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.586+0000 7f1843c87640 1 -- 172.21.15.57:0/4008303129 shutdown_connections 2023-10-19T14:20:02.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.586+0000 7f1843c87640 1 --2- 172.21.15.57:0/4008303129 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f183c1490a0 0x7f183c06bec0 unknown :-1 s=CLOSED pgs=2247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:02.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.586+0000 7f1843c87640 1 --2- 172.21.15.57:0/4008303129 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f183c148c40 0x7f183c06b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:02.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.586+0000 7f1843c87640 1 --2- 172.21.15.57:0/4008303129 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7f182806b100 0x7f182806d5c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:02.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.586+0000 7f1843c87640 1 --2- 172.21.15.57:0/4008303129 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1824005180 0x7f183c14ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:02.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.586+0000 7f1843c87640 1 -- 172.21.15.57:0/4008303129 >> 172.21.15.57:0/4008303129 conn(0x7f183c06fae0 msgr2=0x7f183c159cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:02.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.586+0000 7f1843c87640 1 -- 172.21.15.57:0/4008303129 shutdown_connections 2023-10-19T14:20:02.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.587+0000 7f1843c87640 1 -- 172.21.15.57:0/4008303129 wait complete. 2023-10-19T14:20:02.601 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-19T14:20:02.602 DEBUG:teuthology.orchestra.run.smithi057:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-19T14:20:02.656 DEBUG:teuthology.orchestra.run.smithi179:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-19T14:20:02.702 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-19T14:20:02.703 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T14:20:02.703 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T14:20:02.719 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T14:20:02.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.867+0000 7f3cd129b640 1 -- 172.21.15.57:0/3059774782 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3ccc15e5e0 msgr2=0x7f3ccc0ce9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:02.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.867+0000 7f3cd129b640 1 --2- 172.21.15.57:0/3059774782 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3ccc15e5e0 0x7f3ccc0ce9a0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f3cc0009860 tx=0x7f3cc002bde0 comp rx=0 tx=0).stop 2023-10-19T14:20:02.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.867+0000 7f3cd129b640 1 -- 172.21.15.57:0/3059774782 shutdown_connections 2023-10-19T14:20:02.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.867+0000 7f3cd129b640 1 --2- 172.21.15.57:0/3059774782 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3ccc15e5e0 0x7f3ccc0ce9a0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:02.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.867+0000 7f3cd129b640 1 -- 172.21.15.57:0/3059774782 >> 172.21.15.57:0/3059774782 conn(0x7f3ccc0d0c50 msgr2=0x7f3ccc0d1060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:02.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.867+0000 7f3cd129b640 1 -- 172.21.15.57:0/3059774782 shutdown_connections 2023-10-19T14:20:02.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.867+0000 7f3cd129b640 1 -- 172.21.15.57:0/3059774782 wait complete. 2023-10-19T14:20:02.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.869+0000 7f3cd129b640 1 Processor -- start 2023-10-19T14:20:02.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.869+0000 7f3cd129b640 1 -- start start 2023-10-19T14:20:02.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.869+0000 7f3cd129b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3ccc0b1ca0 0x7f3ccc09a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:02.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.869+0000 7f3cd129b640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3ccc0cf8b0 0x7f3ccc149790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:02.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.869+0000 7f3cd129b640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3ccc15e5e0 0x7f3ccc09d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:02.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.869+0000 7f3cd129b640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ccc14a0c0 con 0x7f3ccc0b1ca0 2023-10-19T14:20:02.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.869+0000 7f3cd129b640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ccc14a3a0 con 0x7f3ccc15e5e0 2023-10-19T14:20:02.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.869+0000 7f3cd129b640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ccc14a680 con 0x7f3ccc0cf8b0 2023-10-19T14:20:02.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.869+0000 7f3ccaffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3ccc0b1ca0 0x7f3ccc09a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:02.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.869+0000 7f3ccaffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3ccc0b1ca0 0x7f3ccc09a950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:47936/0 (socket says 172.21.15.57:47936) 2023-10-19T14:20:02.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.869+0000 7f3ccaffd640 1 -- 172.21.15.57:0/489333616 learned_addr learned my addr 172.21.15.57:0/489333616 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:02.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.870+0000 7f3ccb7fe640 1 --2- 172.21.15.57:0/489333616 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3ccc15e5e0 0x7f3ccc09d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:02.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.870+0000 7f3cca7fc640 1 --2- 172.21.15.57:0/489333616 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3ccc0cf8b0 0x7f3ccc149790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:02.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.870+0000 7f3ccaffd640 1 -- 172.21.15.57:0/489333616 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3ccc0cf8b0 msgr2=0x7f3ccc149790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:02.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.870+0000 7f3ccaffd640 1 --2- 172.21.15.57:0/489333616 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3ccc0cf8b0 0x7f3ccc149790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:02.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.870+0000 7f3ccaffd640 1 -- 172.21.15.57:0/489333616 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3ccc15e5e0 msgr2=0x7f3ccc09d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:02.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.870+0000 7f3ccaffd640 1 --2- 172.21.15.57:0/489333616 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3ccc15e5e0 0x7f3ccc09d0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:02.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.870+0000 7f3ccaffd640 1 -- 172.21.15.57:0/489333616 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cc0009490 con 0x7f3ccc0b1ca0 2023-10-19T14:20:02.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.870+0000 7f3ccaffd640 1 --2- 172.21.15.57:0/489333616 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3ccc0b1ca0 0x7f3ccc09a950 secure :-1 s=READY pgs=2252 cs=0 l=1 rev1=1 crypto rx=0x7f3cc0009980 tx=0x7f3cc00038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:02.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.870+0000 7f3cb37fe640 1 -- 172.21.15.57:0/489333616 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cc0003ea0 con 0x7f3ccc0b1ca0 2023-10-19T14:20:02.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.870+0000 7f3cd129b640 1 -- 172.21.15.57:0/489333616 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ccc14a960 con 0x7f3ccc0b1ca0 2023-10-19T14:20:02.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.870+0000 7f3cb37fe640 1 -- 172.21.15.57:0/489333616 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f3cc0005020 con 0x7f3ccc0b1ca0 2023-10-19T14:20:02.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.870+0000 7f3cb37fe640 1 -- 172.21.15.57:0/489333616 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cc0005430 con 0x7f3ccc0b1ca0 2023-10-19T14:20:02.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.870+0000 7f3cd129b640 1 -- 172.21.15.57:0/489333616 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ccc149cd0 con 0x7f3ccc0b1ca0 2023-10-19T14:20:02.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.871+0000 7f3cd129b640 1 -- 172.21.15.57:0/489333616 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ccc14ae80 con 0x7f3ccc0b1ca0 2023-10-19T14:20:02.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.872+0000 7f3cb37fe640 1 -- 172.21.15.57:0/489333616 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 124) v1 ==== 89445+0+0 (secure 0 0 0) 0x7f3cc002dd50 con 0x7f3ccc0b1ca0 2023-10-19T14:20:02.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.872+0000 7f3cb37fe640 1 --2- 172.21.15.57:0/489333616 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7f3cb806b080 0x7f3cb806d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:02.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.872+0000 7f3cca7fc640 1 -- 172.21.15.57:0/489333616 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7f3cb806b080 msgr2=0x7f3cb806d540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/366917243 2023-10-19T14:20:02.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.872+0000 7f3cb37fe640 1 -- 172.21.15.57:0/489333616 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 6488+0+0 (secure 0 0 0) 0x7f3cc00abde0 con 0x7f3ccc0b1ca0 2023-10-19T14:20:02.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.872+0000 7f3cca7fc640 1 --2- 172.21.15.57:0/489333616 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7f3cb806b080 0x7f3cb806d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T14:20:02.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:02.880+0000 7f3cb37fe640 1 -- 172.21.15.57:0/489333616 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3ccc14ae80 con 0x7f3ccc0b1ca0 2023-10-19T14:20:02.977 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:02.974+0000 7fa04d811040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-19T14:20:02.989 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:20:02.986+0000 7f2bba22c040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-19T14:20:03.021 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:03.019+0000 7fa04d811040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-19T14:20:03.036 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:20:03.034+0000 7f2bba22c040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-19T14:20:03.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.058+0000 7f3cd129b640 1 -- 172.21.15.57:0/489333616 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3ccc14ae80 con 0x7f3ccc0b1ca0 2023-10-19T14:20:03.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.060+0000 7f3cb37fe640 1 -- 172.21.15.57:0/489333616 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v208) v1 ==== 74+0+17425 (secure 0 0 0) 0x7f3ccc14ae80 con 0x7f3ccc0b1ca0 2023-10-19T14:20:03.064 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:20:03.064 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":208,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T14:19:51.475004+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2023-10-19T14:18:37.065110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"204","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_data","create_time":"2023-10-19T14:18:37.809854+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"205","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"26.f","mappings":[{"from":1,"to":0}]},{"pgid":"26.2a","mappings":[{"from":1,"to":0}]},{"pgid":"26.3a","mappings":[{"from":3,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.179:0/2246649044":"2023-10-20T14:18:49.915871+0000","172.21.15.179:0/1186360784":"2023-10-20T14:18:49.915871+0000","172.21.15.179:0/1525973956":"2023-10-20T14:18:49.915871+0000","172.21.15.179:0/1500866753":"2023-10-20T14:18:49.915871+0000","172.21.15.179:6833/2730461479":"2023-10-20T14:18:49.915871+0000","172.21.15.179:6832/2730461479":"2023-10-20T14:18:49.915871+0000","172.21.15.179:0/3327639073":"2023-10-20T14:18:49.915871+0000","172.21.15.179:0/1497317399":"2023-10-20T14:18:49.915871+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T14:20:03.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.063+0000 7f3cd129b640 1 -- 172.21.15.57:0/489333616 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7f3cb806b080 msgr2=0x7f3cb806d540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T14:20:03.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.063+0000 7f3cd129b640 1 --2- 172.21.15.57:0/489333616 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7f3cb806b080 0x7f3cb806d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:03.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.063+0000 7f3cd129b640 1 -- 172.21.15.57:0/489333616 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3ccc0b1ca0 msgr2=0x7f3ccc09a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:03.067 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.063+0000 7f3cd129b640 1 --2- 172.21.15.57:0/489333616 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3ccc0b1ca0 0x7f3ccc09a950 secure :-1 s=READY pgs=2252 cs=0 l=1 rev1=1 crypto rx=0x7f3cc0009980 tx=0x7f3cc00038a0 comp rx=0 tx=0).stop 2023-10-19T14:20:03.067 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.063+0000 7f3cd129b640 1 -- 172.21.15.57:0/489333616 shutdown_connections 2023-10-19T14:20:03.067 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.063+0000 7f3cd129b640 1 --2- 172.21.15.57:0/489333616 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3ccc15e5e0 0x7f3ccc09d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:03.067 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.063+0000 7f3cd129b640 1 --2- 172.21.15.57:0/489333616 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7f3cb806b080 0x7f3cb806d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:03.067 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.063+0000 7f3cd129b640 1 --2- 172.21.15.57:0/489333616 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3ccc0cf8b0 0x7f3ccc149790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:03.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.063+0000 7f3cd129b640 1 --2- 172.21.15.57:0/489333616 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3ccc0b1ca0 0x7f3ccc09a950 unknown :-1 s=CLOSED pgs=2252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:03.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.063+0000 7f3cd129b640 1 -- 172.21.15.57:0/489333616 >> 172.21.15.57:0/489333616 conn(0x7f3ccc0d0c50 msgr2=0x7f3ccc15adf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:03.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.063+0000 7f3cd129b640 1 -- 172.21.15.57:0/489333616 shutdown_connections 2023-10-19T14:20:03.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.063+0000 7f3cd129b640 1 -- 172.21.15.57:0/489333616 wait complete. 2023-10-19T14:20:03.077 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-19T12:44:07.371186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 25, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-19T14:18:37.065110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '204', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 26, 'pool_name': 'cephfs_data', 'create_time': '2023-10-19T14:18:37.809854+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '205', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-19T14:20:03.079 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-19T14:20:03.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.189+0000 7f54c621a640 1 -- 172.21.15.57:0/2995005 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f54c00d1910 msgr2=0x7f54c00b0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:03.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.189+0000 7f54c621a640 1 --2- 172.21.15.57:0/2995005 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f54c00d1910 0x7f54c00b0650 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f54b8005fd0 tx=0x7f54b802bfc0 comp rx=0 tx=0).stop 2023-10-19T14:20:03.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.189+0000 7f54c621a640 1 -- 172.21.15.57:0/2995005 shutdown_connections 2023-10-19T14:20:03.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.189+0000 7f54c621a640 1 --2- 172.21.15.57:0/2995005 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f54c00d1910 0x7f54c00b0650 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:03.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.189+0000 7f54c621a640 1 -- 172.21.15.57:0/2995005 >> 172.21.15.57:0/2995005 conn(0x7f54c00d0d60 msgr2=0x7f54c00d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:03.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.189+0000 7f54c621a640 1 -- 172.21.15.57:0/2995005 shutdown_connections 2023-10-19T14:20:03.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.189+0000 7f54c621a640 1 -- 172.21.15.57:0/2995005 wait complete. 2023-10-19T14:20:03.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.190+0000 7f54c621a640 1 Processor -- start 2023-10-19T14:20:03.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.191+0000 7f54c621a640 1 -- start start 2023-10-19T14:20:03.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.191+0000 7f54c621a640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f54c00cf340 0x7f54c0154a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:03.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.191+0000 7f54c621a640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f54c00d1910 0x7f54c014c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:03.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.191+0000 7f54c621a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f54c015b2a0 0x7f54c00ca2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:03.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.191+0000 7f54c621a640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54c014c5a0 con 0x7f54c015b2a0 2023-10-19T14:20:03.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.191+0000 7f54c621a640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54c015c830 con 0x7f54c00d1910 2023-10-19T14:20:03.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.191+0000 7f54c621a640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54c015cb10 con 0x7f54c00cf340 2023-10-19T14:20:03.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.191+0000 7f54bffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f54c015b2a0 0x7f54c00ca2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:03.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.191+0000 7f54bffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f54c015b2a0 0x7f54c00ca2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:47948/0 (socket says 172.21.15.57:47948) 2023-10-19T14:20:03.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.191+0000 7f54bffff640 1 -- 172.21.15.57:0/2066728414 learned_addr learned my addr 172.21.15.57:0/2066728414 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:03.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.192+0000 7f54beffd640 1 --2- 172.21.15.57:0/2066728414 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f54c00d1910 0x7f54c014c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:03.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.192+0000 7f54bf7fe640 1 --2- 172.21.15.57:0/2066728414 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f54c00cf340 0x7f54c0154a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:03.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.192+0000 7f54bffff640 1 -- 172.21.15.57:0/2066728414 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f54c00cf340 msgr2=0x7f54c0154a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:03.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.192+0000 7f54bffff640 1 --2- 172.21.15.57:0/2066728414 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f54c00cf340 0x7f54c0154a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:03.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.192+0000 7f54bffff640 1 -- 172.21.15.57:0/2066728414 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f54c00d1910 msgr2=0x7f54c014c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:03.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.192+0000 7f54bffff640 1 --2- 172.21.15.57:0/2066728414 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f54c00d1910 0x7f54c014c060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:03.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.192+0000 7f54bffff640 1 -- 172.21.15.57:0/2066728414 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54b8009840 con 0x7f54c015b2a0 2023-10-19T14:20:03.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.192+0000 7f54bffff640 1 --2- 172.21.15.57:0/2066728414 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f54c015b2a0 0x7f54c00ca2e0 secure :-1 s=READY pgs=2254 cs=0 l=1 rev1=1 crypto rx=0x7f54b8006010 tx=0x7f54b80040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:03.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.192+0000 7f54a7fff640 1 -- 172.21.15.57:0/2066728414 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54b80044c0 con 0x7f54c015b2a0 2023-10-19T14:20:03.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.192+0000 7f54a7fff640 1 -- 172.21.15.57:0/2066728414 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f54b8004730 con 0x7f54c015b2a0 2023-10-19T14:20:03.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.192+0000 7f54c621a640 1 -- 172.21.15.57:0/2066728414 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54c015cdf0 con 0x7f54c015b2a0 2023-10-19T14:20:03.201 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.192+0000 7f54a7fff640 1 -- 172.21.15.57:0/2066728414 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54b803e6b0 con 0x7f54c015b2a0 2023-10-19T14:20:03.201 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.192+0000 7f54c621a640 1 -- 172.21.15.57:0/2066728414 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54c015d3b0 con 0x7f54c015b2a0 2023-10-19T14:20:03.201 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.193+0000 7f54c621a640 1 -- 172.21.15.57:0/2066728414 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54c015d5d0 con 0x7f54c015b2a0 2023-10-19T14:20:03.202 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.194+0000 7f54a7fff640 1 -- 172.21.15.57:0/2066728414 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 124) v1 ==== 89445+0+0 (secure 0 0 0) 0x7f54b803e810 con 0x7f54c015b2a0 2023-10-19T14:20:03.202 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.194+0000 7f54a7fff640 1 --2- 172.21.15.57:0/2066728414 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7f54ac06b080 0x7f54ac06d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:03.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.194+0000 7f54bf7fe640 1 -- 172.21.15.57:0/2066728414 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7f54ac06b080 msgr2=0x7f54ac06d540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/366917243 2023-10-19T14:20:03.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.194+0000 7f54a7fff640 1 -- 172.21.15.57:0/2066728414 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 6488+0+0 (secure 0 0 0) 0x7f54b80acb60 con 0x7f54c015b2a0 2023-10-19T14:20:03.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.194+0000 7f54bf7fe640 1 --2- 172.21.15.57:0/2066728414 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7f54ac06b080 0x7f54ac06d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T14:20:03.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.203+0000 7f54a7fff640 1 -- 172.21.15.57:0/2066728414 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f54b8077770 con 0x7f54c015b2a0 2023-10-19T14:20:03.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.380+0000 7f54c621a640 1 -- 172.21.15.57:0/2066728414 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f54c015ae00 con 0x7f54c015b2a0 2023-10-19T14:20:03.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.380+0000 7f54a7fff640 1 -- 172.21.15.57:0/2066728414 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v208) v1 ==== 93+0+10 (secure 0 0 0) 0x7f54b8043020 con 0x7f54c015b2a0 2023-10-19T14:20:03.383 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-19T14:20:03.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.382+0000 7f54c621a640 1 -- 172.21.15.57:0/2066728414 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7f54ac06b080 msgr2=0x7f54ac06d540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T14:20:03.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.382+0000 7f54c621a640 1 --2- 172.21.15.57:0/2066728414 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7f54ac06b080 0x7f54ac06d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:03.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.382+0000 7f54c621a640 1 -- 172.21.15.57:0/2066728414 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f54c015b2a0 msgr2=0x7f54c00ca2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:03.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.382+0000 7f54c621a640 1 --2- 172.21.15.57:0/2066728414 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f54c015b2a0 0x7f54c00ca2e0 secure :-1 s=READY pgs=2254 cs=0 l=1 rev1=1 crypto rx=0x7f54b8006010 tx=0x7f54b80040c0 comp rx=0 tx=0).stop 2023-10-19T14:20:03.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.382+0000 7f54c621a640 1 -- 172.21.15.57:0/2066728414 shutdown_connections 2023-10-19T14:20:03.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.382+0000 7f54c621a640 1 --2- 172.21.15.57:0/2066728414 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f54c015b2a0 0x7f54c00ca2e0 unknown :-1 s=CLOSED pgs=2254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:03.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.382+0000 7f54c621a640 1 --2- 172.21.15.57:0/2066728414 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f54c00d1910 0x7f54c014c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:03.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.382+0000 7f54c621a640 1 --2- 172.21.15.57:0/2066728414 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7f54ac06b080 0x7f54ac06d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:03.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.382+0000 7f54c621a640 1 --2- 172.21.15.57:0/2066728414 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f54c00cf340 0x7f54c0154a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:03.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.382+0000 7f54c621a640 1 -- 172.21.15.57:0/2066728414 >> 172.21.15.57:0/2066728414 conn(0x7f54c00d0d60 msgr2=0x7f54c00b0da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:03.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.382+0000 7f54c621a640 1 -- 172.21.15.57:0/2066728414 shutdown_connections 2023-10-19T14:20:03.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.382+0000 7f54c621a640 1 -- 172.21.15.57:0/2066728414 wait complete. 2023-10-19T14:20:03.397 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-19T14:20:03.408 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:03.406+0000 7fa04d811040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-19T14:20:03.451 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:03.449+0000 7fa04d811040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-19T14:20:03.460 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:20:03.457+0000 7f2bba22c040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-19T14:20:03.493 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:03.490+0000 7fa04d811040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-19T14:20:03.507 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:20:03.504+0000 7f2bba22c040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-19T14:20:03.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.507+0000 7f23743e2640 1 -- 172.21.15.57:0/3455408115 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f236c15b2a0 msgr2=0x7f236c0cf720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:03.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.507+0000 7f23743e2640 1 --2- 172.21.15.57:0/3455408115 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f236c15b2a0 0x7f236c0cf720 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f235c009830 tx=0x7f235c02bcd0 comp rx=0 tx=0).stop 2023-10-19T14:20:03.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.508+0000 7f23743e2640 1 -- 172.21.15.57:0/3455408115 shutdown_connections 2023-10-19T14:20:03.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.508+0000 7f23743e2640 1 --2- 172.21.15.57:0/3455408115 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f236c15b2a0 0x7f236c0cf720 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:03.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.508+0000 7f23743e2640 1 --2- 172.21.15.57:0/3455408115 >> v2:172.21.15.179:3300/0 conn(0x7f236c0cf340 0x7f236c0d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:03.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.508+0000 7f23743e2640 1 -- 172.21.15.57:0/3455408115 >> 172.21.15.57:0/3455408115 conn(0x7f236c0d0d60 msgr2=0x7f236c0d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:03.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.508+0000 7f23743e2640 1 -- 172.21.15.57:0/3455408115 shutdown_connections 2023-10-19T14:20:03.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.508+0000 7f23743e2640 1 -- 172.21.15.57:0/3455408115 wait complete. 2023-10-19T14:20:03.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.509+0000 7f23743e2640 1 Processor -- start 2023-10-19T14:20:03.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.509+0000 7f23743e2640 1 -- start start 2023-10-19T14:20:03.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.509+0000 7f23743e2640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f236c0ceb40 0x7f236c0b65b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:03.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.509+0000 7f23743e2640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f236c0cf340 0x7f236c157f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:03.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.509+0000 7f23743e2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f236c15b2a0 0x7f236c1e4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:03.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.509+0000 7f23743e2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f236c15d6e0 con 0x7f236c15b2a0 2023-10-19T14:20:03.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.510+0000 7f23743e2640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f236c1e4400 con 0x7f236c0cf340 2023-10-19T14:20:03.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.510+0000 7f23743e2640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f236c1e46e0 con 0x7f236c0ceb40 2023-10-19T14:20:03.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.510+0000 7f237297f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f236c15b2a0 0x7f236c1e4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:03.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.510+0000 7f237217e640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f236c0ceb40 0x7f236c0b65b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:03.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.510+0000 7f237297f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f236c15b2a0 0x7f236c1e4ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:47952/0 (socket says 172.21.15.57:47952) 2023-10-19T14:20:03.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.510+0000 7f237297f640 1 -- 172.21.15.57:0/1585377415 learned_addr learned my addr 172.21.15.57:0/1585377415 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:03.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.510+0000 7f237197d640 1 --2- 172.21.15.57:0/1585377415 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f236c0cf340 0x7f236c157f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:03.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.510+0000 7f237297f640 1 -- 172.21.15.57:0/1585377415 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f236c0ceb40 msgr2=0x7f236c0b65b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:03.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.510+0000 7f237297f640 1 --2- 172.21.15.57:0/1585377415 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f236c0ceb40 0x7f236c0b65b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:03.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.510+0000 7f237297f640 1 -- 172.21.15.57:0/1585377415 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f236c0cf340 msgr2=0x7f236c157f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:03.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.510+0000 7f237297f640 1 --2- 172.21.15.57:0/1585377415 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f236c0cf340 0x7f236c157f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:03.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.510+0000 7f237297f640 1 -- 172.21.15.57:0/1585377415 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2364009560 con 0x7f236c15b2a0 2023-10-19T14:20:03.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.510+0000 7f237297f640 1 --2- 172.21.15.57:0/1585377415 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f236c15b2a0 0x7f236c1e4ee0 secure :-1 s=READY pgs=2255 cs=0 l=1 rev1=1 crypto rx=0x7f236800c7a0 tx=0x7f236800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:03.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.510+0000 7f2362ffd640 1 -- 172.21.15.57:0/1585377415 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2368003b70 con 0x7f236c15b2a0 2023-10-19T14:20:03.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.510+0000 7f2362ffd640 1 -- 172.21.15.57:0/1585377415 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f2368003cf0 con 0x7f236c15b2a0 2023-10-19T14:20:03.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.510+0000 7f23743e2640 1 -- 172.21.15.57:0/1585377415 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f235c009490 con 0x7f236c15b2a0 2023-10-19T14:20:03.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.510+0000 7f2362ffd640 1 -- 172.21.15.57:0/1585377415 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f236800aab0 con 0x7f236c15b2a0 2023-10-19T14:20:03.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.510+0000 7f23743e2640 1 -- 172.21.15.57:0/1585377415 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f236c0b6af0 con 0x7f236c15b2a0 2023-10-19T14:20:03.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.511+0000 7f23743e2640 1 -- 172.21.15.57:0/1585377415 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f236c0b6c80 con 0x7f236c15b2a0 2023-10-19T14:20:03.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.512+0000 7f2362ffd640 1 -- 172.21.15.57:0/1585377415 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 124) v1 ==== 89445+0+0 (secure 0 0 0) 0x7f236800ac10 con 0x7f236c15b2a0 2023-10-19T14:20:03.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.512+0000 7f2362ffd640 1 --2- 172.21.15.57:0/1585377415 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7f235006b080 0x7f235006d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:03.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.512+0000 7f237217e640 1 -- 172.21.15.57:0/1585377415 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7f235006b080 msgr2=0x7f235006d540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/366917243 2023-10-19T14:20:03.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.512+0000 7f2362ffd640 1 -- 172.21.15.57:0/1585377415 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 6488+0+0 (secure 0 0 0) 0x7f2368089c90 con 0x7f236c15b2a0 2023-10-19T14:20:03.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.512+0000 7f237217e640 1 --2- 172.21.15.57:0/1585377415 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7f235006b080 0x7f235006d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T14:20:03.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.517+0000 7f2362ffd640 1 -- 172.21.15.57:0/1585377415 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f23680546f0 con 0x7f236c15b2a0 2023-10-19T14:20:03.545 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:03.542+0000 7fa04d811040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-19T14:20:03.553 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:20:03.550+0000 7f2bba22c040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-19T14:20:03.586 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:03.583+0000 7fa04d811040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-19T14:20:03.608 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:20:03.605+0000 7f2bba22c040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-19T14:20:03.651 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:20:03.649+0000 7f2bba22c040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-19T14:20:03.683 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:03.681+0000 7fa04d811040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-19T14:20:03.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.693+0000 7f23743e2640 1 -- 172.21.15.57:0/1585377415 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f236c1e4c00 con 0x7f236c15b2a0 2023-10-19T14:20:03.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.693+0000 7f2362ffd640 1 -- 172.21.15.57:0/1585377415 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v208) v1 ==== 104+0+11 (secure 0 0 0) 0x7f236c1e4c00 con 0x7f236c15b2a0 2023-10-19T14:20:03.697 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2023-10-19T14:20:03.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.695+0000 7f23743e2640 1 -- 172.21.15.57:0/1585377415 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7f235006b080 msgr2=0x7f235006d540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T14:20:03.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.695+0000 7f23743e2640 1 --2- 172.21.15.57:0/1585377415 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7f235006b080 0x7f235006d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:03.699 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.695+0000 7f23743e2640 1 -- 172.21.15.57:0/1585377415 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f236c15b2a0 msgr2=0x7f236c1e4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:03.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.695+0000 7f23743e2640 1 --2- 172.21.15.57:0/1585377415 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f236c15b2a0 0x7f236c1e4ee0 secure :-1 s=READY pgs=2255 cs=0 l=1 rev1=1 crypto rx=0x7f236800c7a0 tx=0x7f236800cc60 comp rx=0 tx=0).stop 2023-10-19T14:20:03.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.695+0000 7f23743e2640 1 -- 172.21.15.57:0/1585377415 shutdown_connections 2023-10-19T14:20:03.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.695+0000 7f23743e2640 1 --2- 172.21.15.57:0/1585377415 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f236c15b2a0 0x7f236c1e4ee0 unknown :-1 s=CLOSED pgs=2255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:03.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.695+0000 7f23743e2640 1 --2- 172.21.15.57:0/1585377415 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f236c0cf340 0x7f236c157f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:03.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.695+0000 7f23743e2640 1 --2- 172.21.15.57:0/1585377415 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7f235006b080 0x7f235006d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:03.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.695+0000 7f23743e2640 1 --2- 172.21.15.57:0/1585377415 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f236c0ceb40 0x7f236c0b65b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:03.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.695+0000 7f23743e2640 1 -- 172.21.15.57:0/1585377415 >> 172.21.15.57:0/1585377415 conn(0x7f236c0d0d60 msgr2=0x7f236c06b2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:03.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.695+0000 7f23743e2640 1 -- 172.21.15.57:0/1585377415 shutdown_connections 2023-10-19T14:20:03.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.695+0000 7f23743e2640 1 -- 172.21.15.57:0/1585377415 wait complete. 2023-10-19T14:20:03.709 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-19T14:20:03.735 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:03.733+0000 7fa04d811040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-19T14:20:03.756 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:20:03.753+0000 7f2bba22c040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-19T14:20:03.776 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:03.774+0000 7fa04d811040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-19T14:20:03.814 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:20:03.811+0000 7f2bba22c040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-19T14:20:03.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.821+0000 7f5e451d7640 1 -- 172.21.15.57:0/3681671052 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5e4015b2a0 msgr2=0x7f5e400d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:03.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.821+0000 7f5e451d7640 1 --2- 172.21.15.57:0/3681671052 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5e4015b2a0 0x7f5e400d1910 secure :-1 s=READY pgs=2257 cs=0 l=1 rev1=1 crypto rx=0x7f5e30009830 tx=0x7f5e3002bd60 comp rx=0 tx=0).stop 2023-10-19T14:20:03.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.821+0000 7f5e451d7640 1 -- 172.21.15.57:0/3681671052 shutdown_connections 2023-10-19T14:20:03.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.821+0000 7f5e451d7640 1 --2- 172.21.15.57:0/3681671052 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5e4015b2a0 0x7f5e400d1910 unknown :-1 s=CLOSED pgs=2257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:03.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.821+0000 7f5e451d7640 1 -- 172.21.15.57:0/3681671052 >> 172.21.15.57:0/3681671052 conn(0x7f5e400d0d60 msgr2=0x7f5e400d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:03.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.821+0000 7f5e451d7640 1 -- 172.21.15.57:0/3681671052 shutdown_connections 2023-10-19T14:20:03.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.821+0000 7f5e451d7640 1 -- 172.21.15.57:0/3681671052 wait complete. 2023-10-19T14:20:03.826 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:03.822+0000 7fa04d811040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-19T14:20:03.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.822+0000 7f5e451d7640 1 Processor -- start 2023-10-19T14:20:03.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.822+0000 7f5e451d7640 1 -- start start 2023-10-19T14:20:03.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.823+0000 7f5e451d7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5e400ceb40 0x7f5e4015e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:03.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.823+0000 7f5e451d7640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5e400cf340 0x7f5e400c90e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:03.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.823+0000 7f5e451d7640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5e4015b2a0 0x7f5e40158ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:03.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.823+0000 7f5e451d7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e4015ba70 con 0x7f5e400ceb40 2023-10-19T14:20:03.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.823+0000 7f5e451d7640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e4015e610 con 0x7f5e400cf340 2023-10-19T14:20:03.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.823+0000 7f5e451d7640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e400c9620 con 0x7f5e4015b2a0 2023-10-19T14:20:03.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.823+0000 7f5e3ed9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5e400ceb40 0x7f5e4015e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:03.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.823+0000 7f5e3ed9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5e400ceb40 0x7f5e4015e0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:47966/0 (socket says 172.21.15.57:47966) 2023-10-19T14:20:03.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.823+0000 7f5e3ed9d640 1 -- 172.21.15.57:0/2996433466 learned_addr learned my addr 172.21.15.57:0/2996433466 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:03.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.823+0000 7f5e3e59c640 1 --2- 172.21.15.57:0/2996433466 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5e400cf340 0x7f5e400c90e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:03.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.823+0000 7f5e3f59e640 1 --2- 172.21.15.57:0/2996433466 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5e4015b2a0 0x7f5e40158ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:03.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.823+0000 7f5e3ed9d640 1 -- 172.21.15.57:0/2996433466 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5e4015b2a0 msgr2=0x7f5e40158ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:03.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.823+0000 7f5e3ed9d640 1 --2- 172.21.15.57:0/2996433466 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5e4015b2a0 0x7f5e40158ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:03.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.823+0000 7f5e3ed9d640 1 -- 172.21.15.57:0/2996433466 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5e400cf340 msgr2=0x7f5e400c90e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:03.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.823+0000 7f5e3ed9d640 1 --2- 172.21.15.57:0/2996433466 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5e400cf340 0x7f5e400c90e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:03.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.823+0000 7f5e3ed9d640 1 -- 172.21.15.57:0/2996433466 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e30009490 con 0x7f5e400ceb40 2023-10-19T14:20:03.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.824+0000 7f5e3ed9d640 1 --2- 172.21.15.57:0/2996433466 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5e400ceb40 0x7f5e4015e0d0 secure :-1 s=READY pgs=2258 cs=0 l=1 rev1=1 crypto rx=0x7f5e2800c460 tx=0x7f5e2800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:03.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.824+0000 7f5e377fe640 1 -- 172.21.15.57:0/2996433466 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e28005480 con 0x7f5e400ceb40 2023-10-19T14:20:03.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.824+0000 7f5e451d7640 1 -- 172.21.15.57:0/2996433466 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e40159430 con 0x7f5e400ceb40 2023-10-19T14:20:03.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.824+0000 7f5e377fe640 1 -- 172.21.15.57:0/2996433466 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f5e280035f0 con 0x7f5e400ceb40 2023-10-19T14:20:03.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.824+0000 7f5e377fe640 1 -- 172.21.15.57:0/2996433466 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e28003a00 con 0x7f5e400ceb40 2023-10-19T14:20:03.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.824+0000 7f5e451d7640 1 -- 172.21.15.57:0/2996433466 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e4015d310 con 0x7f5e400ceb40 2023-10-19T14:20:03.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.825+0000 7f5e451d7640 1 -- 172.21.15.57:0/2996433466 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e400690e0 con 0x7f5e400ceb40 2023-10-19T14:20:03.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.826+0000 7f5e377fe640 1 -- 172.21.15.57:0/2996433466 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 124) v1 ==== 89445+0+0 (secure 0 0 0) 0x7f5e28003b60 con 0x7f5e400ceb40 2023-10-19T14:20:03.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.826+0000 7f5e377fe640 1 --2- 172.21.15.57:0/2996433466 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7f5e2006b080 0x7f5e2006d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:03.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.826+0000 7f5e3e59c640 1 -- 172.21.15.57:0/2996433466 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7f5e2006b080 msgr2=0x7f5e2006d540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/366917243 2023-10-19T14:20:03.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.826+0000 7f5e377fe640 1 -- 172.21.15.57:0/2996433466 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 6488+0+0 (secure 0 0 0) 0x7f5e28089920 con 0x7f5e400ceb40 2023-10-19T14:20:03.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.826+0000 7f5e3e59c640 1 --2- 172.21.15.57:0/2996433466 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7f5e2006b080 0x7f5e2006d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T14:20:03.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:03.834+0000 7f5e377fe640 1 -- 172.21.15.57:0/2996433466 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f5e28054390 con 0x7f5e400ceb40 2023-10-19T14:20:03.863 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:20:03.861+0000 7f2bba22c040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-19T14:20:03.867 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:03.864+0000 7fa04d811040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-19T14:20:03.914 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:20:03.912+0000 7f2bba22c040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-19T14:20:03.959 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:20:03.957+0000 7f2bba22c040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-19T14:20:04.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.010+0000 7f5e451d7640 1 -- 172.21.15.57:0/2996433466 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5e400c84b0 con 0x7f5e400ceb40 2023-10-19T14:20:04.014 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.011+0000 7f5e377fe640 1 -- 172.21.15.57:0/2996433466 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v208) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5e280579b0 con 0x7f5e400ceb40 2023-10-19T14:20:04.014 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 64 2023-10-19T14:20:04.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.013+0000 7f5e451d7640 1 -- 172.21.15.57:0/2996433466 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7f5e2006b080 msgr2=0x7f5e2006d540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T14:20:04.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.013+0000 7f5e451d7640 1 --2- 172.21.15.57:0/2996433466 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7f5e2006b080 0x7f5e2006d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:04.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.013+0000 7f5e451d7640 1 -- 172.21.15.57:0/2996433466 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5e400ceb40 msgr2=0x7f5e4015e0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:04.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.013+0000 7f5e451d7640 1 --2- 172.21.15.57:0/2996433466 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5e400ceb40 0x7f5e4015e0d0 secure :-1 s=READY pgs=2258 cs=0 l=1 rev1=1 crypto rx=0x7f5e2800c460 tx=0x7f5e2800c920 comp rx=0 tx=0).stop 2023-10-19T14:20:04.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.013+0000 7f5e451d7640 1 -- 172.21.15.57:0/2996433466 shutdown_connections 2023-10-19T14:20:04.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.013+0000 7f5e451d7640 1 --2- 172.21.15.57:0/2996433466 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7f5e2006b080 0x7f5e2006d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:04.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.013+0000 7f5e451d7640 1 --2- 172.21.15.57:0/2996433466 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5e4015b2a0 0x7f5e40158ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:04.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.013+0000 7f5e451d7640 1 --2- 172.21.15.57:0/2996433466 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5e400cf340 0x7f5e400c90e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:04.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.013+0000 7f5e451d7640 1 --2- 172.21.15.57:0/2996433466 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5e400ceb40 0x7f5e4015e0d0 unknown :-1 s=CLOSED pgs=2258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:04.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.013+0000 7f5e451d7640 1 -- 172.21.15.57:0/2996433466 >> 172.21.15.57:0/2996433466 conn(0x7f5e400d0d60 msgr2=0x7f5e400d1500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:04.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.013+0000 7f5e451d7640 1 -- 172.21.15.57:0/2996433466 shutdown_connections 2023-10-19T14:20:04.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.013+0000 7f5e451d7640 1 -- 172.21.15.57:0/2996433466 wait complete. 2023-10-19T14:20:04.027 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-19T14:20:04.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.135+0000 7f5fea96e640 1 -- 172.21.15.57:0/4182368951 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5fe415b640 msgr2=0x7f5fe40cf850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:04.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.135+0000 7f5fea96e640 1 --2- 172.21.15.57:0/4182368951 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5fe415b640 0x7f5fe40cf850 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f5fcc009830 tx=0x7f5fcc02bd60 comp rx=0 tx=0).stop 2023-10-19T14:20:04.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.135+0000 7f5fea96e640 1 -- 172.21.15.57:0/4182368951 shutdown_connections 2023-10-19T14:20:04.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.135+0000 7f5fea96e640 1 --2- 172.21.15.57:0/4182368951 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5fe415b640 0x7f5fe40cf850 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:04.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.136+0000 7f5fea96e640 1 --2- 172.21.15.57:0/4182368951 >> v2:172.21.15.179:3300/0 conn(0x7f5fe415b260 0x7f5fe40cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:04.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.136+0000 7f5fea96e640 1 -- 172.21.15.57:0/4182368951 >> 172.21.15.57:0/4182368951 conn(0x7f5fe409a030 msgr2=0x7f5fe40d0d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:04.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.136+0000 7f5fea96e640 1 -- 172.21.15.57:0/4182368951 shutdown_connections 2023-10-19T14:20:04.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.136+0000 7f5fea96e640 1 -- 172.21.15.57:0/4182368951 wait complete. 2023-10-19T14:20:04.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.137+0000 7f5fea96e640 1 Processor -- start 2023-10-19T14:20:04.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.137+0000 7f5fea96e640 1 -- start start 2023-10-19T14:20:04.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.138+0000 7f5fea96e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5fe40b0b10 0x7f5fe4157bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:04.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.138+0000 7f5fea96e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5fe415b260 0x7f5fe40b3940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:04.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.138+0000 7f5fea96e640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5fe415b640 0x7f5fe40b52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:04.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.138+0000 7f5fea96e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe40b4ec0 con 0x7f5fe40b0b10 2023-10-19T14:20:04.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.138+0000 7f5fea96e640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe409abb0 con 0x7f5fe415b260 2023-10-19T14:20:04.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.138+0000 7f5fea96e640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe41580f0 con 0x7f5fe415b640 2023-10-19T14:20:04.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.138+0000 7f5fe3fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5fe40b0b10 0x7f5fe4157bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:04.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.138+0000 7f5fe3fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5fe40b0b10 0x7f5fe4157bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:47980/0 (socket says 172.21.15.57:47980) 2023-10-19T14:20:04.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.138+0000 7f5fe3fff640 1 -- 172.21.15.57:0/286327174 learned_addr learned my addr 172.21.15.57:0/286327174 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:04.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.138+0000 7f5fe37fe640 1 --2- 172.21.15.57:0/286327174 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5fe415b260 0x7f5fe40b3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:04.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.138+0000 7f5fe8f0b640 1 --2- 172.21.15.57:0/286327174 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5fe415b640 0x7f5fe40b52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:04.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.138+0000 7f5fe3fff640 1 -- 172.21.15.57:0/286327174 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5fe415b640 msgr2=0x7f5fe40b52e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:04.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.138+0000 7f5fe3fff640 1 --2- 172.21.15.57:0/286327174 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5fe415b640 0x7f5fe40b52e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:04.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.138+0000 7f5fe3fff640 1 -- 172.21.15.57:0/286327174 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5fe415b260 msgr2=0x7f5fe40b3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:04.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.138+0000 7f5fe3fff640 1 --2- 172.21.15.57:0/286327174 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5fe415b260 0x7f5fe40b3940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:04.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.138+0000 7f5fe3fff640 1 -- 172.21.15.57:0/286327174 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fcc009490 con 0x7f5fe40b0b10 2023-10-19T14:20:04.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.138+0000 7f5fe3fff640 1 --2- 172.21.15.57:0/286327174 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5fe40b0b10 0x7f5fe4157bb0 secure :-1 s=READY pgs=2260 cs=0 l=1 rev1=1 crypto rx=0x7f5fd400ea20 tx=0x7f5fd400eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:04.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.138+0000 7f5fe0ff9640 1 -- 172.21.15.57:0/286327174 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fd4009700 con 0x7f5fe40b0b10 2023-10-19T14:20:04.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.138+0000 7f5fe0ff9640 1 -- 172.21.15.57:0/286327174 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f5fd400ce70 con 0x7f5fe40b0b10 2023-10-19T14:20:04.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.139+0000 7f5fea96e640 1 -- 172.21.15.57:0/286327174 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fe40b4240 con 0x7f5fe40b0b10 2023-10-19T14:20:04.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.139+0000 7f5fe0ff9640 1 -- 172.21.15.57:0/286327174 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fd4010690 con 0x7f5fe40b0b10 2023-10-19T14:20:04.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.139+0000 7f5fea96e640 1 -- 172.21.15.57:0/286327174 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fe40b4800 con 0x7f5fe40b0b10 2023-10-19T14:20:04.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.139+0000 7f5fea96e640 1 -- 172.21.15.57:0/286327174 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fe40675f0 con 0x7f5fe40b0b10 2023-10-19T14:20:04.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.140+0000 7f5fe0ff9640 1 -- 172.21.15.57:0/286327174 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 124) v1 ==== 89445+0+0 (secure 0 0 0) 0x7f5fd40107f0 con 0x7f5fe40b0b10 2023-10-19T14:20:04.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.141+0000 7f5fe0ff9640 1 --2- 172.21.15.57:0/286327174 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7f5fc406b080 0x7f5fc406d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:04.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.141+0000 7f5fe0ff9640 1 -- 172.21.15.57:0/286327174 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 6488+0+0 (secure 0 0 0) 0x7f5fd4014070 con 0x7f5fe40b0b10 2023-10-19T14:20:04.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.141+0000 7f5fe37fe640 1 -- 172.21.15.57:0/286327174 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7f5fc406b080 msgr2=0x7f5fc406d540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/366917243 2023-10-19T14:20:04.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.141+0000 7f5fe37fe640 1 --2- 172.21.15.57:0/286327174 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7f5fc406b080 0x7f5fc406d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T14:20:04.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.145+0000 7f5fe0ff9640 1 -- 172.21.15.57:0/286327174 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f5fd4051ca0 con 0x7f5fe40b0b10 2023-10-19T14:20:04.180 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:04.177+0000 7fa04d811040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-19T14:20:04.298 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:20:04.295+0000 7f2bba22c040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-19T14:20:04.309 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:04.306+0000 7fa04d811040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-19T14:20:04.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.337+0000 7f5fea96e640 1 -- 172.21.15.57:0/286327174 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f5fe40b34b0 con 0x7f5fe40b0b10 2023-10-19T14:20:04.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.337+0000 7f5fe0ff9640 1 -- 172.21.15.57:0/286327174 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v208) v1 ==== 80+0+478 (secure 0 0 0) 0x7f5fd405de60 con 0x7f5fe40b0b10 2023-10-19T14:20:04.341 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/2246649044 2023-10-20T14:18:49.915871+0000 2023-10-19T14:20:04.341 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/1186360784 2023-10-20T14:18:49.915871+0000 2023-10-19T14:20:04.341 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/1525973956 2023-10-20T14:18:49.915871+0000 2023-10-19T14:20:04.341 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/1500866753 2023-10-20T14:18:49.915871+0000 2023-10-19T14:20:04.341 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:6833/2730461479 2023-10-20T14:18:49.915871+0000 2023-10-19T14:20:04.342 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:6832/2730461479 2023-10-20T14:18:49.915871+0000 2023-10-19T14:20:04.342 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/3327639073 2023-10-20T14:18:49.915871+0000 2023-10-19T14:20:04.342 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/1497317399 2023-10-20T14:18:49.915871+0000 2023-10-19T14:20:04.343 INFO:teuthology.orchestra.run.smithi057.stderr:listed 8 entries 2023-10-19T14:20:04.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.339+0000 7f5fea96e640 1 -- 172.21.15.57:0/286327174 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7f5fc406b080 msgr2=0x7f5fc406d540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T14:20:04.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.339+0000 7f5fea96e640 1 --2- 172.21.15.57:0/286327174 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7f5fc406b080 0x7f5fc406d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:04.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.339+0000 7f5fea96e640 1 -- 172.21.15.57:0/286327174 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5fe40b0b10 msgr2=0x7f5fe4157bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:04.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.339+0000 7f5fea96e640 1 --2- 172.21.15.57:0/286327174 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5fe40b0b10 0x7f5fe4157bb0 secure :-1 s=READY pgs=2260 cs=0 l=1 rev1=1 crypto rx=0x7f5fd400ea20 tx=0x7f5fd400eee0 comp rx=0 tx=0).stop 2023-10-19T14:20:04.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.339+0000 7f5fea96e640 1 -- 172.21.15.57:0/286327174 shutdown_connections 2023-10-19T14:20:04.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.339+0000 7f5fea96e640 1 --2- 172.21.15.57:0/286327174 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7f5fc406b080 0x7f5fc406d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:04.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.339+0000 7f5fea96e640 1 --2- 172.21.15.57:0/286327174 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5fe415b640 0x7f5fe40b52e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:04.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.339+0000 7f5fea96e640 1 --2- 172.21.15.57:0/286327174 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5fe415b260 0x7f5fe40b3940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:04.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.339+0000 7f5fea96e640 1 --2- 172.21.15.57:0/286327174 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5fe40b0b10 0x7f5fe4157bb0 unknown :-1 s=CLOSED pgs=2260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:04.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.339+0000 7f5fea96e640 1 -- 172.21.15.57:0/286327174 >> 172.21.15.57:0/286327174 conn(0x7f5fe409a030 msgr2=0x7f5fe40b24a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:04.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.339+0000 7f5fea96e640 1 -- 172.21.15.57:0/286327174 shutdown_connections 2023-10-19T14:20:04.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.339+0000 7f5fea96e640 1 -- 172.21.15.57:0/286327174 wait complete. 2023-10-19T14:20:04.353 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi057.front.sepia.ceph.com... 2023-10-19T14:20:04.353 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:20:04.354 DEBUG:teuthology.orchestra.run.smithi057:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-10-19T14:20:04.384 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi057.stderr:ceph-fuse[149328]: fuse finished with error 0 and tester_r 0 2023-10-19T14:20:04.385 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi057.front.sepia.ceph.com 2023-10-19T14:20:04.385 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:20:04.386 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-19T14:20:04.402 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:04.399+0000 7fa04d811040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-19T14:20:04.441 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:20:04.438+0000 7f2bba22c040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-19T14:20:04.444 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:04.441+0000 7fa04d811040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-19T14:20:04.455 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-19T14:20:04.455 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T14:20:04.470 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054833,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=284 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3247632k,nr_inodes=811908,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-19T14:20:04.471 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi179.front.sepia.ceph.com 2023-10-19T14:20:04.471 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:20:04.472 DEBUG:teuthology.orchestra.run.smithi179:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-10-19T14:20:04.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-19T14:20:04.525 INFO:teuthology.orchestra.run.smithi179.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2023-10-19T14:20:04.526 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T14:20:04.526 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T14:20:04.538 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:20:04.535+0000 7f2bba22c040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-19T14:20:04.542 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055614,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=358 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248260k,nr_inodes=812065,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-19T14:20:04.542 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi057.front.sepia.ceph.com 2023-10-19T14:20:04.543 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:20:04.543 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-10-19T14:20:04.544 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:04.540+0000 7fa04d811040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-19T14:20:04.583 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:20:04.580+0000 7f2bba22c040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-19T14:20:04.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-19T14:20:04.602 INFO:teuthology.orchestra.run.smithi057.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2023-10-19T14:20:04.602 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-19T14:20:04.602 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T14:20:04.617 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054833,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=284 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3247632k,nr_inodes=811908,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-19T14:20:04.618 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi179.front.sepia.ceph.com 2023-10-19T14:20:04.618 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:20:04.619 DEBUG:teuthology.orchestra.run.smithi179:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-10-19T14:20:04.640 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:04.637+0000 7fa04d811040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-19T14:20:04.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-19T14:20:04.654 INFO:teuthology.orchestra.run.smithi179.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2023-10-19T14:20:04.655 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T14:20:04.693 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:20:04.690+0000 7f2bba22c040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-19T14:20:04.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.765+0000 7f9f36b2c640 1 -- 172.21.15.57:0/1152720634 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9f3015b2a0 msgr2=0x7f9f300d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:04.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.765+0000 7f9f36b2c640 1 --2- 172.21.15.57:0/1152720634 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9f3015b2a0 0x7f9f300d1910 secure :-1 s=READY pgs=2262 cs=0 l=1 rev1=1 crypto rx=0x7f9f3015b680 tx=0x7f9f1c02bcf0 comp rx=0 tx=0).stop 2023-10-19T14:20:04.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.766+0000 7f9f36b2c640 1 -- 172.21.15.57:0/1152720634 shutdown_connections 2023-10-19T14:20:04.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.766+0000 7f9f36b2c640 1 --2- 172.21.15.57:0/1152720634 >> v2:172.21.15.179:3300/0 conn(0x7f9f300ceb40 0x7f9f300b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:04.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.766+0000 7f9f36b2c640 1 --2- 172.21.15.57:0/1152720634 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9f3015b2a0 0x7f9f300d1910 unknown :-1 s=CLOSED pgs=2262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:04.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.766+0000 7f9f36b2c640 1 -- 172.21.15.57:0/1152720634 >> 172.21.15.57:0/1152720634 conn(0x7f9f300d0d60 msgr2=0x7f9f300d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:04.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.766+0000 7f9f36b2c640 1 -- 172.21.15.57:0/1152720634 shutdown_connections 2023-10-19T14:20:04.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.766+0000 7f9f36b2c640 1 -- 172.21.15.57:0/1152720634 wait complete. 2023-10-19T14:20:04.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.767+0000 7f9f36b2c640 1 Processor -- start 2023-10-19T14:20:04.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.767+0000 7f9f36b2c640 1 -- start start 2023-10-19T14:20:04.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.768+0000 7f9f36b2c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9f300ceb40 0x7f9f3015b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:04.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.768+0000 7f9f36b2c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9f300cf340 0x7f9f300b4a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:04.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.768+0000 7f9f36b2c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9f3015b2a0 0x7f9f30155640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:04.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.768+0000 7f9f36b2c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f300b4f70 con 0x7f9f300ceb40 2023-10-19T14:20:04.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.768+0000 7f9f36b2c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f3015c9e0 con 0x7f9f3015b2a0 2023-10-19T14:20:04.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.768+0000 7f9f36b2c640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f3015bdd0 con 0x7f9f300cf340 2023-10-19T14:20:04.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.768+0000 7f9f348c8640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9f300ceb40 0x7f9f3015b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:04.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.768+0000 7f9f348c8640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9f300ceb40 0x7f9f3015b6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:48004/0 (socket says 172.21.15.57:48004) 2023-10-19T14:20:04.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.768+0000 7f9f348c8640 1 -- 172.21.15.57:0/3169596676 learned_addr learned my addr 172.21.15.57:0/3169596676 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:04.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.768+0000 7f9f350c9640 1 --2- 172.21.15.57:0/3169596676 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9f3015b2a0 0x7f9f30155640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:04.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.768+0000 7f9f2ffff640 1 --2- 172.21.15.57:0/3169596676 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9f300cf340 0x7f9f300b4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:04.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.768+0000 7f9f348c8640 1 -- 172.21.15.57:0/3169596676 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9f300cf340 msgr2=0x7f9f300b4a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:04.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.768+0000 7f9f348c8640 1 --2- 172.21.15.57:0/3169596676 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9f300cf340 0x7f9f300b4a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:04.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.768+0000 7f9f348c8640 1 -- 172.21.15.57:0/3169596676 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9f3015b2a0 msgr2=0x7f9f30155640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:04.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.768+0000 7f9f348c8640 1 --2- 172.21.15.57:0/3169596676 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9f3015b2a0 0x7f9f30155640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:04.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.768+0000 7f9f348c8640 1 -- 172.21.15.57:0/3169596676 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f1c009490 con 0x7f9f300ceb40 2023-10-19T14:20:04.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.768+0000 7f9f348c8640 1 --2- 172.21.15.57:0/3169596676 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9f300ceb40 0x7f9f3015b6d0 secure :-1 s=READY pgs=2263 cs=0 l=1 rev1=1 crypto rx=0x7f9f2400c490 tx=0x7f9f2400c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:04.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.768+0000 7f9f2d7fa640 1 -- 172.21.15.57:0/3169596676 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f24012830 con 0x7f9f300ceb40 2023-10-19T14:20:04.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.768+0000 7f9f2d7fa640 1 -- 172.21.15.57:0/3169596676 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f9f24012a80 con 0x7f9f300ceb40 2023-10-19T14:20:04.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.768+0000 7f9f36b2c640 1 -- 172.21.15.57:0/3169596676 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f3015c0b0 con 0x7f9f300ceb40 2023-10-19T14:20:04.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.769+0000 7f9f2d7fa640 1 -- 172.21.15.57:0/3169596676 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f24011380 con 0x7f9f300ceb40 2023-10-19T14:20:04.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.769+0000 7f9f36b2c640 1 -- 172.21.15.57:0/3169596676 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f3015c670 con 0x7f9f300ceb40 2023-10-19T14:20:04.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.769+0000 7f9f36b2c640 1 -- 172.21.15.57:0/3169596676 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f300b0400 con 0x7f9f300ceb40 2023-10-19T14:20:04.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.770+0000 7f9f2d7fa640 1 -- 172.21.15.57:0/3169596676 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 124) v1 ==== 89445+0+0 (secure 0 0 0) 0x7f9f240114e0 con 0x7f9f300ceb40 2023-10-19T14:20:04.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.771+0000 7f9f2d7fa640 1 --2- 172.21.15.57:0/3169596676 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7f9f1406b0b0 0x7f9f1406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:04.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.771+0000 7f9f2ffff640 1 -- 172.21.15.57:0/3169596676 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7f9f1406b0b0 msgr2=0x7f9f1406d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/366917243 2023-10-19T14:20:04.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.771+0000 7f9f2d7fa640 1 -- 172.21.15.57:0/3169596676 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6576+0+0 (secure 0 0 0) 0x7f9f24089a30 con 0x7f9f300ceb40 2023-10-19T14:20:04.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.771+0000 7f9f2ffff640 1 --2- 172.21.15.57:0/3169596676 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7f9f1406b0b0 0x7f9f1406d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T14:20:04.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.775+0000 7f9f2d7fa640 1 -- 172.21.15.57:0/3169596676 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9f24058030 con 0x7f9f300ceb40 2023-10-19T14:20:04.782 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:04.780+0000 7fa04d811040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-19T14:20:04.791 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:20:04.789+0000 7f2bba22c040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-19T14:20:04.824 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:04.821+0000 7fa04d811040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-19T14:20:04.927 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:04.924+0000 7fa04d811040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-19T14:20:04.936 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:20:04.933+0000 7f2bba22c040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-19T14:20:04.974 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.972+0000 7f9f2ffff640 1 -- 172.21.15.57:0/3169596676 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7f9f1406b0b0 msgr2=0x7f9f1406d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/366917243 2023-10-19T14:20:04.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.972+0000 7f9f2ffff640 1 --2- 172.21.15.57:0/3169596676 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7f9f1406b0b0 0x7f9f1406d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-19T14:20:04.980 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:20:04.977+0000 7f2bba22c040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-19T14:20:04.990 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.988+0000 7f9f36b2c640 1 -- 172.21.15.57:0/3169596676 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9f300b0d00 con 0x7f9f300ceb40 2023-10-19T14:20:04.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.988+0000 7f9f2d7fa640 1 -- 172.21.15.57:0/3169596676 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+4926 (secure 0 0 0) 0x7f9f24016090 con 0x7f9f300ceb40 2023-10-19T14:20:04.992 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:20:04.992 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":83,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/2313260224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2313260224},{"type":"v1","addr":"172.21.15.179:6836","nonce":2313260224}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":10081,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/494432208","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":494432208},{"type":"v1","addr":"172.21.15.179:6837","nonce":494432208}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":11214,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/3080658822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":3080658822},{"type":"v1","addr":"172.21.15.57:6834","nonce":3080658822}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T14:18:38.834485+0000","modified":"2023-10-19T14:18:40.833363+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11217":{"gid":11217,"name":"a","rank":0,"incarnation":81,"state":"up:active","state_seq":3,"addr":"172.21.15.57:6835/1674945162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1674945162},{"type":"v1","addr":"172.21.15.57:6835","nonce":1674945162}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2023-10-19T14:20:04.992 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 83 2023-10-19T14:20:04.993 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.990+0000 7f9f36b2c640 1 -- 172.21.15.57:0/3169596676 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7f9f1406b0b0 msgr2=0x7f9f1406d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T14:20:04.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.990+0000 7f9f36b2c640 1 --2- 172.21.15.57:0/3169596676 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7f9f1406b0b0 0x7f9f1406d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:04.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.990+0000 7f9f36b2c640 1 -- 172.21.15.57:0/3169596676 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9f300ceb40 msgr2=0x7f9f3015b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:04.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.990+0000 7f9f36b2c640 1 --2- 172.21.15.57:0/3169596676 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9f300ceb40 0x7f9f3015b6d0 secure :-1 s=READY pgs=2263 cs=0 l=1 rev1=1 crypto rx=0x7f9f2400c490 tx=0x7f9f2400c950 comp rx=0 tx=0).stop 2023-10-19T14:20:04.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.991+0000 7f9f36b2c640 1 -- 172.21.15.57:0/3169596676 shutdown_connections 2023-10-19T14:20:04.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.991+0000 7f9f36b2c640 1 --2- 172.21.15.57:0/3169596676 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9f3015b2a0 0x7f9f30155640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:04.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.991+0000 7f9f36b2c640 1 --2- 172.21.15.57:0/3169596676 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7f9f1406b0b0 0x7f9f1406d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:04.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.991+0000 7f9f36b2c640 1 --2- 172.21.15.57:0/3169596676 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9f300cf340 0x7f9f300b4a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:04.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.991+0000 7f9f36b2c640 1 --2- 172.21.15.57:0/3169596676 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9f300ceb40 0x7f9f3015b6d0 unknown :-1 s=CLOSED pgs=2263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:04.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.991+0000 7f9f36b2c640 1 -- 172.21.15.57:0/3169596676 >> 172.21.15.57:0/3169596676 conn(0x7f9f300d0d60 msgr2=0x7f9f30069f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:04.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.991+0000 7f9f36b2c640 1 -- 172.21.15.57:0/3169596676 shutdown_connections 2023-10-19T14:20:04.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:04.991+0000 7f9f36b2c640 1 -- 172.21.15.57:0/3169596676 wait complete. 2023-10-19T14:20:05.004 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T14:20:05.093 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:20:05.090+0000 7f2bba22c040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-19T14:20:05.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.153+0000 7f647a9d7640 1 -- 172.21.15.57:0/520474816 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f64740ceb40 msgr2=0x7f64740b0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:05.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.153+0000 7f647a9d7640 1 --2- 172.21.15.57:0/520474816 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f64740ceb40 0x7f64740b0650 secure :-1 s=READY pgs=2265 cs=0 l=1 rev1=1 crypto rx=0x7f647408a810 tx=0x7f646c02c080 comp rx=0 tx=0).stop 2023-10-19T14:20:05.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.153+0000 7f647a9d7640 1 -- 172.21.15.57:0/520474816 shutdown_connections 2023-10-19T14:20:05.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.153+0000 7f647a9d7640 1 --2- 172.21.15.57:0/520474816 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f64740ceb40 0x7f64740b0650 unknown :-1 s=CLOSED pgs=2265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:05.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.153+0000 7f647a9d7640 1 --2- 172.21.15.57:0/520474816 >> v2:172.21.15.179:3300/0 conn(0x7f64740cf340 0x7f64740d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:05.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.153+0000 7f647a9d7640 1 -- 172.21.15.57:0/520474816 >> 172.21.15.57:0/520474816 conn(0x7f64740d0d60 msgr2=0x7f64740d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:05.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.154+0000 7f647a9d7640 1 -- 172.21.15.57:0/520474816 shutdown_connections 2023-10-19T14:20:05.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.154+0000 7f647a9d7640 1 -- 172.21.15.57:0/520474816 wait complete. 2023-10-19T14:20:05.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.155+0000 7f647a9d7640 1 Processor -- start 2023-10-19T14:20:05.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.155+0000 7f647a9d7640 1 -- start start 2023-10-19T14:20:05.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.155+0000 7f647a9d7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f64740ceb40 0x7f64740b42f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:05.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.156+0000 7f647a9d7640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f64740cf340 0x7f647415d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:05.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.156+0000 7f647a9d7640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f647415b2a0 0x7f647415dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:05.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.156+0000 7f647a9d7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6474158d50 con 0x7f64740ceb40 2023-10-19T14:20:05.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.156+0000 7f647a9d7640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f647415c6d0 con 0x7f64740cf340 2023-10-19T14:20:05.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.156+0000 7f647a9d7640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f647415c9b0 con 0x7f647415b2a0 2023-10-19T14:20:05.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.156+0000 7f6473fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f64740ceb40 0x7f64740b42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:05.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.156+0000 7f6473fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f64740ceb40 0x7f64740b42f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:48028/0 (socket says 172.21.15.57:48028) 2023-10-19T14:20:05.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.156+0000 7f6473fff640 1 -- 172.21.15.57:0/1083295352 learned_addr learned my addr 172.21.15.57:0/1083295352 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:05.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.156+0000 7f64737fe640 1 --2- 172.21.15.57:0/1083295352 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f64740cf340 0x7f647415d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:05.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.156+0000 7f6478f74640 1 --2- 172.21.15.57:0/1083295352 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f647415b2a0 0x7f647415dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:05.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.156+0000 7f6473fff640 1 -- 172.21.15.57:0/1083295352 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f647415b2a0 msgr2=0x7f647415dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:05.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.156+0000 7f6473fff640 1 --2- 172.21.15.57:0/1083295352 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f647415b2a0 0x7f647415dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:05.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.156+0000 7f6473fff640 1 -- 172.21.15.57:0/1083295352 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f64740cf340 msgr2=0x7f647415d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:05.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.156+0000 7f6473fff640 1 --2- 172.21.15.57:0/1083295352 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f64740cf340 0x7f647415d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:05.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.156+0000 7f6473fff640 1 -- 172.21.15.57:0/1083295352 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f646c009840 con 0x7f64740ceb40 2023-10-19T14:20:05.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.156+0000 7f6473fff640 1 --2- 172.21.15.57:0/1083295352 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f64740ceb40 0x7f64740b42f0 secure :-1 s=READY pgs=2266 cs=0 l=1 rev1=1 crypto rx=0x7f646400e9f0 tx=0x7f646400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:05.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.156+0000 7f6470ff9640 1 -- 172.21.15.57:0/1083295352 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6464009700 con 0x7f64740ceb40 2023-10-19T14:20:05.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.156+0000 7f6470ff9640 1 -- 172.21.15.57:0/1083295352 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f646400ce70 con 0x7f64740ceb40 2023-10-19T14:20:05.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.156+0000 7f647a9d7640 1 -- 172.21.15.57:0/1083295352 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f647415cc90 con 0x7f64740ceb40 2023-10-19T14:20:05.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.156+0000 7f6470ff9640 1 -- 172.21.15.57:0/1083295352 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6464010670 con 0x7f64740ceb40 2023-10-19T14:20:05.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.156+0000 7f647a9d7640 1 -- 172.21.15.57:0/1083295352 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f647415d250 con 0x7f64740ceb40 2023-10-19T14:20:05.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.157+0000 7f647a9d7640 1 -- 172.21.15.57:0/1083295352 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64740690e0 con 0x7f64740ceb40 2023-10-19T14:20:05.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.158+0000 7f6470ff9640 1 -- 172.21.15.57:0/1083295352 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 124) v1 ==== 89445+0+0 (secure 0 0 0) 0x7f64640107d0 con 0x7f64740ceb40 2023-10-19T14:20:05.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.158+0000 7f6470ff9640 1 --2- 172.21.15.57:0/1083295352 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7f646006b0b0 0x7f646006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:05.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.159+0000 7f64737fe640 1 -- 172.21.15.57:0/1083295352 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7f646006b0b0 msgr2=0x7f646006d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/366917243 2023-10-19T14:20:05.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.159+0000 7f6470ff9640 1 -- 172.21.15.57:0/1083295352 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6576+0+0 (secure 0 0 0) 0x7f6464014070 con 0x7f64740ceb40 2023-10-19T14:20:05.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.159+0000 7f64737fe640 1 --2- 172.21.15.57:0/1083295352 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7f646006b0b0 0x7f646006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T14:20:05.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.163+0000 7f6470ff9640 1 -- 172.21.15.57:0/1083295352 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6464051ca0 con 0x7f64740ceb40 2023-10-19T14:20:05.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.341+0000 7f647a9d7640 1 -- 172.21.15.57:0/1083295352 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f64740b0d00 con 0x7f64740ceb40 2023-10-19T14:20:05.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.342+0000 7f6470ff9640 1 -- 172.21.15.57:0/1083295352 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v209) v1 ==== 74+0+17583 (secure 0 0 0) 0x7f64640553b0 con 0x7f64740ceb40 2023-10-19T14:20:05.345 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:20:05.345 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":209,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T14:20:04.586243+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2023-10-19T14:18:37.065110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"204","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_data","create_time":"2023-10-19T14:18:37.809854+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"209","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":4,"snap_epoch":209,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"26.f","mappings":[{"from":1,"to":0}]},{"pgid":"26.2a","mappings":[{"from":1,"to":0}]},{"pgid":"26.3a","mappings":[{"from":3,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.179:0/2246649044":"2023-10-20T14:18:49.915871+0000","172.21.15.179:0/1186360784":"2023-10-20T14:18:49.915871+0000","172.21.15.179:0/1525973956":"2023-10-20T14:18:49.915871+0000","172.21.15.179:0/1500866753":"2023-10-20T14:18:49.915871+0000","172.21.15.179:6833/2730461479":"2023-10-20T14:18:49.915871+0000","172.21.15.179:6832/2730461479":"2023-10-20T14:18:49.915871+0000","172.21.15.179:0/3327639073":"2023-10-20T14:18:49.915871+0000","172.21.15.179:0/1497317399":"2023-10-20T14:18:49.915871+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":26,"snaps":[{"begin":2,"length":1},{"begin":4,"length":1}]}],"new_removed_snaps":[{"pool":26,"snaps":[{"begin":2,"length":1},{"begin":4,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T14:20:05.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.344+0000 7f647a9d7640 1 -- 172.21.15.57:0/1083295352 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7f646006b0b0 msgr2=0x7f646006d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T14:20:05.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.344+0000 7f647a9d7640 1 --2- 172.21.15.57:0/1083295352 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7f646006b0b0 0x7f646006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:05.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.344+0000 7f647a9d7640 1 -- 172.21.15.57:0/1083295352 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f64740ceb40 msgr2=0x7f64740b42f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:05.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.344+0000 7f647a9d7640 1 --2- 172.21.15.57:0/1083295352 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f64740ceb40 0x7f64740b42f0 secure :-1 s=READY pgs=2266 cs=0 l=1 rev1=1 crypto rx=0x7f646400e9f0 tx=0x7f646400eeb0 comp rx=0 tx=0).stop 2023-10-19T14:20:05.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.344+0000 7f647a9d7640 1 -- 172.21.15.57:0/1083295352 shutdown_connections 2023-10-19T14:20:05.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.344+0000 7f647a9d7640 1 --2- 172.21.15.57:0/1083295352 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7f646006b0b0 0x7f646006d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:05.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.344+0000 7f647a9d7640 1 --2- 172.21.15.57:0/1083295352 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f647415b2a0 0x7f647415dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:05.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.344+0000 7f647a9d7640 1 --2- 172.21.15.57:0/1083295352 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f64740cf340 0x7f647415d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:05.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.344+0000 7f647a9d7640 1 --2- 172.21.15.57:0/1083295352 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f64740ceb40 0x7f64740b42f0 unknown :-1 s=CLOSED pgs=2266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:05.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.344+0000 7f647a9d7640 1 -- 172.21.15.57:0/1083295352 >> 172.21.15.57:0/1083295352 conn(0x7f64740d0d60 msgr2=0x7f64740b2850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:05.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.344+0000 7f647a9d7640 1 -- 172.21.15.57:0/1083295352 shutdown_connections 2023-10-19T14:20:05.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.344+0000 7f647a9d7640 1 -- 172.21.15.57:0/1083295352 wait complete. 2023-10-19T14:20:05.358 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-19T12:44:07.371186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 25, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-19T14:18:37.065110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '204', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 26, 'pool_name': 'cephfs_data', 'create_time': '2023-10-19T14:18:37.809854+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '209', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 4, 'snap_epoch': 209, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-19T14:20:05.359 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-19T14:20:05.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.470+0000 7f45f6cc2640 1 -- 172.21.15.57:0/2476290290 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f45f00cf340 msgr2=0x7f45f00d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:05.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.470+0000 7f45f6cc2640 1 --2- 172.21.15.57:0/2476290290 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f45f00cf340 0x7f45f00d1910 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f45e4009830 tx=0x7f45e402bcd0 comp rx=0 tx=0).stop 2023-10-19T14:20:05.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.470+0000 7f45f6cc2640 1 -- 172.21.15.57:0/2476290290 shutdown_connections 2023-10-19T14:20:05.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.470+0000 7f45f6cc2640 1 --2- 172.21.15.57:0/2476290290 >> v2:172.21.15.179:3300/0 conn(0x7f45f00ceb40 0x7f45f00b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:05.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.470+0000 7f45f6cc2640 1 --2- 172.21.15.57:0/2476290290 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f45f00cf340 0x7f45f00d1910 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:05.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.470+0000 7f45f6cc2640 1 -- 172.21.15.57:0/2476290290 >> 172.21.15.57:0/2476290290 conn(0x7f45f00d0d60 msgr2=0x7f45f00d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:05.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.470+0000 7f45f6cc2640 1 -- 172.21.15.57:0/2476290290 shutdown_connections 2023-10-19T14:20:05.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.470+0000 7f45f6cc2640 1 -- 172.21.15.57:0/2476290290 wait complete. 2023-10-19T14:20:05.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.472+0000 7f45f6cc2640 1 Processor -- start 2023-10-19T14:20:05.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.472+0000 7f45f6cc2640 1 -- start start 2023-10-19T14:20:05.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.472+0000 7f45f6cc2640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f45f00ceb40 0x7f45f015c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:05.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.472+0000 7f45f6cc2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f45f00cf340 0x7f45f00c8040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:05.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.472+0000 7f45f6cc2640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f45f015b2a0 0x7f45f0150fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:05.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.472+0000 7f45f6cc2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45f0151510 con 0x7f45f00cf340 2023-10-19T14:20:05.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.472+0000 7f45f6cc2640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45f015df10 con 0x7f45f015b2a0 2023-10-19T14:20:05.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.472+0000 7f45f6cc2640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45f015ce70 con 0x7f45f00ceb40 2023-10-19T14:20:05.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.473+0000 7f45effff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f45f00cf340 0x7f45f00c8040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:05.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.473+0000 7f45effff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f45f00cf340 0x7f45f00c8040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:48044/0 (socket says 172.21.15.57:48044) 2023-10-19T14:20:05.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.473+0000 7f45f4a5e640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f45f00ceb40 0x7f45f015c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:05.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.473+0000 7f45effff640 1 -- 172.21.15.57:0/2484780960 learned_addr learned my addr 172.21.15.57:0/2484780960 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:05.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.473+0000 7f45f525f640 1 --2- 172.21.15.57:0/2484780960 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f45f015b2a0 0x7f45f0150fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:05.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.473+0000 7f45f4a5e640 1 -- 172.21.15.57:0/2484780960 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f45f015b2a0 msgr2=0x7f45f0150fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:05.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.473+0000 7f45f4a5e640 1 --2- 172.21.15.57:0/2484780960 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f45f015b2a0 0x7f45f0150fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:05.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.473+0000 7f45f4a5e640 1 -- 172.21.15.57:0/2484780960 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f45f00cf340 msgr2=0x7f45f00c8040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:05.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.473+0000 7f45f4a5e640 1 --2- 172.21.15.57:0/2484780960 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f45f00cf340 0x7f45f00c8040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:05.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.473+0000 7f45f4a5e640 1 -- 172.21.15.57:0/2484780960 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45e4009490 con 0x7f45f00ceb40 2023-10-19T14:20:05.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.473+0000 7f45effff640 1 --2- 172.21.15.57:0/2484780960 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f45f00cf340 0x7f45f00c8040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T14:20:05.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.473+0000 7f45f4a5e640 1 --2- 172.21.15.57:0/2484780960 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f45f00ceb40 0x7f45f015c790 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f45e4033000 tx=0x7f45e40038a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:05.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.473+0000 7f45ed7fa640 1 -- 172.21.15.57:0/2484780960 <== mon.2 v2:172.21.15.179:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45e4003ea0 con 0x7f45f00ceb40 2023-10-19T14:20:05.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.473+0000 7f45ed7fa640 1 -- 172.21.15.57:0/2484780960 <== mon.2 v2:172.21.15.179:3301/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f45e4034440 con 0x7f45f00ceb40 2023-10-19T14:20:05.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.473+0000 7f45f6cc2640 1 -- 172.21.15.57:0/2484780960 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45f015d150 con 0x7f45f00ceb40 2023-10-19T14:20:05.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.473+0000 7f45ed7fa640 1 -- 172.21.15.57:0/2484780960 <== mon.2 v2:172.21.15.179:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45e4034850 con 0x7f45f00ceb40 2023-10-19T14:20:05.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.473+0000 7f45f6cc2640 1 -- 172.21.15.57:0/2484780960 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45f015d710 con 0x7f45f00ceb40 2023-10-19T14:20:05.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.474+0000 7f45f6cc2640 1 -- 172.21.15.57:0/2484780960 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45f00690e0 con 0x7f45f00ceb40 2023-10-19T14:20:05.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.475+0000 7f45ed7fa640 1 -- 172.21.15.57:0/2484780960 <== mon.2 v2:172.21.15.179:3301/0 4 ==== mgrmap(e 124) v1 ==== 89445+0+0 (secure 0 0 0) 0x7f45e40349b0 con 0x7f45f00ceb40 2023-10-19T14:20:05.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.475+0000 7f45ed7fa640 1 --2- 172.21.15.57:0/2484780960 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7f45d406b080 0x7f45d406d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:05.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.475+0000 7f45ed7fa640 1 -- 172.21.15.57:0/2484780960 <== mon.2 v2:172.21.15.179:3301/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6576+0+0 (secure 0 0 0) 0x7f45e40ac440 con 0x7f45f00ceb40 2023-10-19T14:20:05.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.475+0000 7f45effff640 1 -- 172.21.15.57:0/2484780960 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7f45d406b080 msgr2=0x7f45d406d540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/366917243 2023-10-19T14:20:05.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.475+0000 7f45effff640 1 --2- 172.21.15.57:0/2484780960 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7f45d406b080 0x7f45d406d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T14:20:05.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.480+0000 7f45ed7fa640 1 -- 172.21.15.57:0/2484780960 <== mon.2 v2:172.21.15.179:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f45e4076e50 con 0x7f45f00ceb40 2023-10-19T14:20:05.540 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:05.538+0000 7fa04d811040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-19T14:20:05.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.676+0000 7f45effff640 1 -- 172.21.15.57:0/2484780960 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7f45d406b080 msgr2=0x7f45d406d540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/366917243 2023-10-19T14:20:05.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.676+0000 7f45effff640 1 --2- 172.21.15.57:0/2484780960 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7f45d406b080 0x7f45d406d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-19T14:20:05.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.686+0000 7f45f6cc2640 1 -- 172.21.15.57:0/2484780960 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f45f00d0b40 con 0x7f45f00ceb40 2023-10-19T14:20:05.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.686+0000 7f45ed7fa640 1 -- 172.21.15.57:0/2484780960 <== mon.2 v2:172.21.15.179:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v210) v1 ==== 93+0+10 (secure 0 0 0) 0x7f45e4039020 con 0x7f45f00ceb40 2023-10-19T14:20:05.689 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-19T14:20:05.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.688+0000 7f45f6cc2640 1 -- 172.21.15.57:0/2484780960 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7f45d406b080 msgr2=0x7f45d406d540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T14:20:05.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.688+0000 7f45f6cc2640 1 --2- 172.21.15.57:0/2484780960 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7f45d406b080 0x7f45d406d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:05.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.688+0000 7f45f6cc2640 1 -- 172.21.15.57:0/2484780960 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f45f00ceb40 msgr2=0x7f45f015c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:05.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.688+0000 7f45f6cc2640 1 --2- 172.21.15.57:0/2484780960 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f45f00ceb40 0x7f45f015c790 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f45e4033000 tx=0x7f45e40038a0 comp rx=0 tx=0).stop 2023-10-19T14:20:05.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.688+0000 7f45f6cc2640 1 -- 172.21.15.57:0/2484780960 shutdown_connections 2023-10-19T14:20:05.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.688+0000 7f45f6cc2640 1 --2- 172.21.15.57:0/2484780960 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f45f015b2a0 0x7f45f0150fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:05.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.688+0000 7f45f6cc2640 1 --2- 172.21.15.57:0/2484780960 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f45f00cf340 0x7f45f00c8040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:05.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.688+0000 7f45f6cc2640 1 --2- 172.21.15.57:0/2484780960 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7f45d406b080 0x7f45d406d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:05.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.688+0000 7f45f6cc2640 1 --2- 172.21.15.57:0/2484780960 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f45f00ceb40 0x7f45f015c790 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:05.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.688+0000 7f45f6cc2640 1 -- 172.21.15.57:0/2484780960 >> 172.21.15.57:0/2484780960 conn(0x7f45f00d0d60 msgr2=0x7f45f0159fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:05.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.689+0000 7f45f6cc2640 1 -- 172.21.15.57:0/2484780960 shutdown_connections 2023-10-19T14:20:05.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.689+0000 7f45f6cc2640 1 -- 172.21.15.57:0/2484780960 wait complete. 2023-10-19T14:20:05.702 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-19T14:20:05.787 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:20:05.784+0000 7f2bba22c040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-19T14:20:05.802 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:05.800+0000 7fa04d811040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-19T14:20:05.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.813+0000 7fb6f6694640 1 -- 172.21.15.57:0/3654980226 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb6f00cf340 msgr2=0x7fb6f00d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:05.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.813+0000 7fb6f6694640 1 --2- 172.21.15.57:0/3654980226 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb6f00cf340 0x7fb6f00d1910 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7fb6e0009830 tx=0x7fb6e002bcd0 comp rx=0 tx=0).stop 2023-10-19T14:20:05.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.813+0000 7fb6f6694640 1 -- 172.21.15.57:0/3654980226 shutdown_connections 2023-10-19T14:20:05.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.813+0000 7fb6f6694640 1 --2- 172.21.15.57:0/3654980226 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb6f015b2a0 0x7fb6f00cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:05.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.813+0000 7fb6f6694640 1 --2- 172.21.15.57:0/3654980226 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb6f00cf340 0x7fb6f00d1910 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:05.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.813+0000 7fb6f6694640 1 -- 172.21.15.57:0/3654980226 >> 172.21.15.57:0/3654980226 conn(0x7fb6f00d0d60 msgr2=0x7fb6f00d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:05.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.813+0000 7fb6f6694640 1 -- 172.21.15.57:0/3654980226 shutdown_connections 2023-10-19T14:20:05.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.813+0000 7fb6f6694640 1 -- 172.21.15.57:0/3654980226 wait complete. 2023-10-19T14:20:05.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.814+0000 7fb6f6694640 1 Processor -- start 2023-10-19T14:20:05.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.814+0000 7fb6f6694640 1 -- start start 2023-10-19T14:20:05.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.815+0000 7fb6f6694640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb6f00ceb40 0x7fb6f0158480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:05.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.815+0000 7fb6f6694640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb6f00cf340 0x7fb6f014dd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:05.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.815+0000 7fb6f6694640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb6f015b2a0 0x7fb6f01e3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:05.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.815+0000 7fb6f6694640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6f00b69c0 con 0x7fb6f015b2a0 2023-10-19T14:20:05.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.815+0000 7fb6f6694640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6f014e2c0 con 0x7fb6f00ceb40 2023-10-19T14:20:05.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.815+0000 7fb6f6694640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6f00b5d40 con 0x7fb6f00cf340 2023-10-19T14:20:05.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.815+0000 7fb6f4c31640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb6f015b2a0 0x7fb6f01e3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:05.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.815+0000 7fb6f4c31640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb6f015b2a0 0x7fb6f01e3870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:48060/0 (socket says 172.21.15.57:48060) 2023-10-19T14:20:05.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.815+0000 7fb6f4c31640 1 -- 172.21.15.57:0/2134530263 learned_addr learned my addr 172.21.15.57:0/2134530263 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:05.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.815+0000 7fb6ef7fe640 1 --2- 172.21.15.57:0/2134530263 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb6f00cf340 0x7fb6f014dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:05.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.815+0000 7fb6effff640 1 --2- 172.21.15.57:0/2134530263 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb6f00ceb40 0x7fb6f0158480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:05.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.815+0000 7fb6f4c31640 1 -- 172.21.15.57:0/2134530263 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb6f00cf340 msgr2=0x7fb6f014dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:05.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.815+0000 7fb6f4c31640 1 --2- 172.21.15.57:0/2134530263 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb6f00cf340 0x7fb6f014dd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:05.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.815+0000 7fb6f4c31640 1 -- 172.21.15.57:0/2134530263 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb6f00ceb40 msgr2=0x7fb6f0158480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:05.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.815+0000 7fb6f4c31640 1 --2- 172.21.15.57:0/2134530263 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb6f00ceb40 0x7fb6f0158480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:05.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.815+0000 7fb6f4c31640 1 -- 172.21.15.57:0/2134530263 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6e4009490 con 0x7fb6f015b2a0 2023-10-19T14:20:05.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.816+0000 7fb6f4c31640 1 --2- 172.21.15.57:0/2134530263 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb6f015b2a0 0x7fb6f01e3870 secure :-1 s=READY pgs=2269 cs=0 l=1 rev1=1 crypto rx=0x7fb6e800cbc0 tx=0x7fb6e800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:05.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.816+0000 7fb6ecff9640 1 -- 172.21.15.57:0/2134530263 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6e800ad40 con 0x7fb6f015b2a0 2023-10-19T14:20:05.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.816+0000 7fb6ecff9640 1 -- 172.21.15.57:0/2134530263 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fb6e8012420 con 0x7fb6f015b2a0 2023-10-19T14:20:05.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.816+0000 7fb6ecff9640 1 -- 172.21.15.57:0/2134530263 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6e8012850 con 0x7fb6f015b2a0 2023-10-19T14:20:05.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.816+0000 7fb6f6694640 1 -- 172.21.15.57:0/2134530263 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6e0009490 con 0x7fb6f015b2a0 2023-10-19T14:20:05.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.816+0000 7fb6f6694640 1 -- 172.21.15.57:0/2134530263 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6f00b6300 con 0x7fb6f015b2a0 2023-10-19T14:20:05.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.817+0000 7fb6f6694640 1 -- 172.21.15.57:0/2134530263 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6f00b6b00 con 0x7fb6f015b2a0 2023-10-19T14:20:05.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.818+0000 7fb6ecff9640 1 -- 172.21.15.57:0/2134530263 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 124) v1 ==== 89445+0+0 (secure 0 0 0) 0x7fb6e801a3b0 con 0x7fb6f015b2a0 2023-10-19T14:20:05.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.818+0000 7fb6ecff9640 1 --2- 172.21.15.57:0/2134530263 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7fb6dc06b080 0x7fb6dc06d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:05.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.818+0000 7fb6effff640 1 -- 172.21.15.57:0/2134530263 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7fb6dc06b080 msgr2=0x7fb6dc06d540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/366917243 2023-10-19T14:20:05.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.818+0000 7fb6ecff9640 1 -- 172.21.15.57:0/2134530263 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 6560+0+0 (secure 0 0 0) 0x7fb6e808a4b0 con 0x7fb6f015b2a0 2023-10-19T14:20:05.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.818+0000 7fb6effff640 1 --2- 172.21.15.57:0/2134530263 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7fb6dc06b080 0x7fb6dc06d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T14:20:05.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.822+0000 7fb6ecff9640 1 -- 172.21.15.57:0/2134530263 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb6f00b6b00 con 0x7fb6f015b2a0 2023-10-19T14:20:05.852 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:05.849+0000 7fa04d811040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-19T14:20:05.950 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:05.948+0000 7fa04d811040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-19T14:20:05.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.974+0000 7fb6ecff9640 1 -- 172.21.15.57:0/2134530263 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mgrmap(e 125) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fb6e8055340 con 0x7fb6f015b2a0 2023-10-19T14:20:05.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.974+0000 7fb6ecff9640 1 -- 172.21.15.57:0/2134530263 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7fb6dc06b080 msgr2=0x7fb6dc06d540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T14:20:05.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:05.974+0000 7fb6ecff9640 1 --2- 172.21.15.57:0/2134530263 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7fb6dc06b080 0x7fb6dc06d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:06.006 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.003+0000 7fb6f6694640 1 -- 172.21.15.57:0/2134530263 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb6f00b6b00 con 0x7fb6f015b2a0 2023-10-19T14:20:06.006 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.003+0000 7fb6ecff9640 1 -- 172.21.15.57:0/2134530263 <== mon.0 v2:172.21.15.57:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v211) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb6f00b6b00 con 0x7fb6f015b2a0 2023-10-19T14:20:06.007 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2023-10-19T14:20:06.008 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.005+0000 7fb6f6694640 1 -- 172.21.15.57:0/2134530263 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb6f015b2a0 msgr2=0x7fb6f01e3870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:06.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.005+0000 7fb6f6694640 1 --2- 172.21.15.57:0/2134530263 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb6f015b2a0 0x7fb6f01e3870 secure :-1 s=READY pgs=2269 cs=0 l=1 rev1=1 crypto rx=0x7fb6e800cbc0 tx=0x7fb6e800a430 comp rx=0 tx=0).stop 2023-10-19T14:20:06.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.005+0000 7fb6f6694640 1 -- 172.21.15.57:0/2134530263 shutdown_connections 2023-10-19T14:20:06.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.005+0000 7fb6f6694640 1 --2- 172.21.15.57:0/2134530263 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb6f015b2a0 0x7fb6f01e3870 unknown :-1 s=CLOSED pgs=2269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:06.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.005+0000 7fb6f6694640 1 --2- 172.21.15.57:0/2134530263 >> [v2:172.21.15.179:6832/366917243,v1:172.21.15.179:6833/366917243] conn(0x7fb6dc06b080 0x7fb6dc06d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:06.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.005+0000 7fb6f6694640 1 --2- 172.21.15.57:0/2134530263 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb6f00cf340 0x7fb6f014dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:06.010 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.005+0000 7fb6f6694640 1 --2- 172.21.15.57:0/2134530263 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb6f00ceb40 0x7fb6f0158480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:06.010 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.005+0000 7fb6f6694640 1 -- 172.21.15.57:0/2134530263 >> 172.21.15.57:0/2134530263 conn(0x7fb6f00d0d60 msgr2=0x7fb6f015a2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:06.010 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.005+0000 7fb6f6694640 1 -- 172.21.15.57:0/2134530263 shutdown_connections 2023-10-19T14:20:06.010 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.005+0000 7fb6f6694640 1 -- 172.21.15.57:0/2134530263 wait complete. 2023-10-19T14:20:06.019 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-19T14:20:06.034 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:06.031+0000 7f9fd329f640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:20:06.034 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:06.031+0000 7f9fd329f640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:20:06.035 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:06.031+0000 7f9fd329f640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:20:06.035 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:06.031+0000 7f9fd0299640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:20:06.035 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:06.031+0000 7f9fd329f640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:20:06.035 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:06.031+0000 7f9fd0299640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:20:06.035 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:06.032+0000 7f9fd0299640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:20:06.036 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:06.032+0000 7f9fd0299640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:20:06.036 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:06.032+0000 7f9fd329f640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:20:06.036 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:06.032+0000 7f9fd0299640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:20:06.073 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:20:06.070+0000 7f2bba22c040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-19T14:20:06.075 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:20:06.077 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:20:06.126 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:20:06.124+0000 7f2bba22c040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-19T14:20:06.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.131+0000 7f936fcaf640 1 -- 172.21.15.57:0/3002111620 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f93680cf340 msgr2=0x7f93680d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:06.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.131+0000 7f936fcaf640 1 --2- 172.21.15.57:0/3002111620 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f93680cf340 0x7f93680d1910 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f9358009830 tx=0x7f935802bd60 comp rx=0 tx=0).stop 2023-10-19T14:20:06.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.131+0000 7f936fcaf640 1 -- 172.21.15.57:0/3002111620 shutdown_connections 2023-10-19T14:20:06.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.131+0000 7f936fcaf640 1 --2- 172.21.15.57:0/3002111620 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f93680cf340 0x7f93680d1910 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:06.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.131+0000 7f936fcaf640 1 -- 172.21.15.57:0/3002111620 >> 172.21.15.57:0/3002111620 conn(0x7f93680d0d60 msgr2=0x7f93680d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:06.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.131+0000 7f936fcaf640 1 -- 172.21.15.57:0/3002111620 shutdown_connections 2023-10-19T14:20:06.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.131+0000 7f936fcaf640 1 -- 172.21.15.57:0/3002111620 wait complete. 2023-10-19T14:20:06.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.132+0000 7f936fcaf640 1 Processor -- start 2023-10-19T14:20:06.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.132+0000 7f936fcaf640 1 -- start start 2023-10-19T14:20:06.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.133+0000 7f936fcaf640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f93680ceb40 0x7f93681f11b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:06.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.133+0000 7f936fcaf640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f93680cf340 0x7f93680b66c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:06.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.133+0000 7f936fcaf640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f936815b2a0 0x7f9368157480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:06.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.133+0000 7f936fcaf640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93681f16f0 con 0x7f936815b2a0 2023-10-19T14:20:06.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.133+0000 7f936fcaf640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93681f4970 con 0x7f93680ceb40 2023-10-19T14:20:06.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.133+0000 7f936fcaf640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93681f2180 con 0x7f93680cf340 2023-10-19T14:20:06.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.133+0000 7f936e24c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f936815b2a0 0x7f9368157480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:06.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.133+0000 7f936e24c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f936815b2a0 0x7f9368157480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:48072/0 (socket says 172.21.15.57:48072) 2023-10-19T14:20:06.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.133+0000 7f936e24c640 1 -- 172.21.15.57:0/268086442 learned_addr learned my addr 172.21.15.57:0/268086442 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:06.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.133+0000 7f936da4b640 1 --2- 172.21.15.57:0/268086442 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f93680ceb40 0x7f93681f11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:06.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.133+0000 7f936d24a640 1 --2- 172.21.15.57:0/268086442 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f93680cf340 0x7f93680b66c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:06.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.133+0000 7f936e24c640 1 -- 172.21.15.57:0/268086442 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f93680cf340 msgr2=0x7f93680b66c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:06.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.133+0000 7f936e24c640 1 --2- 172.21.15.57:0/268086442 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f93680cf340 0x7f93680b66c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:06.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.133+0000 7f936e24c640 1 -- 172.21.15.57:0/268086442 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f93680ceb40 msgr2=0x7f93681f11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:06.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.133+0000 7f936e24c640 1 --2- 172.21.15.57:0/268086442 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f93680ceb40 0x7f93681f11b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:06.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.133+0000 7f936e24c640 1 -- 172.21.15.57:0/268086442 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9358009490 con 0x7f936815b2a0 2023-10-19T14:20:06.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.133+0000 7f936e24c640 1 --2- 172.21.15.57:0/268086442 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f936815b2a0 0x7f9368157480 secure :-1 s=READY pgs=2272 cs=0 l=1 rev1=1 crypto rx=0x7f936000c7a0 tx=0x7f936000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:06.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.134+0000 7f935e7fc640 1 -- 172.21.15.57:0/268086442 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9360003b70 con 0x7f936815b2a0 2023-10-19T14:20:06.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.134+0000 7f936fcaf640 1 -- 172.21.15.57:0/268086442 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93680b6c00 con 0x7f936815b2a0 2023-10-19T14:20:06.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.134+0000 7f935e7fc640 1 -- 172.21.15.57:0/268086442 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f9360003cf0 con 0x7f936815b2a0 2023-10-19T14:20:06.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.134+0000 7f935e7fc640 1 -- 172.21.15.57:0/268086442 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f936000aab0 con 0x7f936815b2a0 2023-10-19T14:20:06.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.134+0000 7f936fcaf640 1 -- 172.21.15.57:0/268086442 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93681f1bc0 con 0x7f936815b2a0 2023-10-19T14:20:06.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.135+0000 7f936fcaf640 1 -- 172.21.15.57:0/268086442 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93680cca00 con 0x7f936815b2a0 2023-10-19T14:20:06.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.135+0000 7f935e7fc640 1 -- 172.21.15.57:0/268086442 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 125) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f936000ac10 con 0x7f936815b2a0 2023-10-19T14:20:06.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.136+0000 7f935e7fc640 1 -- 172.21.15.57:0/268086442 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(211..211 src has 1..211) v4 ==== 6946+0+0 (secure 0 0 0) 0x7f93600894a0 con 0x7f936815b2a0 2023-10-19T14:20:06.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.140+0000 7f935e7fc640 1 -- 172.21.15.57:0/268086442 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f93600544f0 con 0x7f936815b2a0 2023-10-19T14:20:06.236 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:20:06.233+0000 7f2bba22c040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-19T14:20:06.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.321+0000 7f936fcaf640 1 -- 172.21.15.57:0/268086442 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f93680690e0 con 0x7f936815b2a0 2023-10-19T14:20:06.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.322+0000 7f935e7fc640 1 -- 172.21.15.57:0/268086442 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v211) v1 ==== 100+0+11 (secure 0 0 0) 0x7f9360057b10 con 0x7f936815b2a0 2023-10-19T14:20:06.325 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 64 2023-10-19T14:20:06.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.324+0000 7f936fcaf640 1 -- 172.21.15.57:0/268086442 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f936815b2a0 msgr2=0x7f9368157480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:06.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.324+0000 7f936fcaf640 1 --2- 172.21.15.57:0/268086442 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f936815b2a0 0x7f9368157480 secure :-1 s=READY pgs=2272 cs=0 l=1 rev1=1 crypto rx=0x7f936000c7a0 tx=0x7f936000cc60 comp rx=0 tx=0).stop 2023-10-19T14:20:06.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.324+0000 7f936fcaf640 1 -- 172.21.15.57:0/268086442 shutdown_connections 2023-10-19T14:20:06.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.324+0000 7f936fcaf640 1 --2- 172.21.15.57:0/268086442 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f936815b2a0 0x7f9368157480 unknown :-1 s=CLOSED pgs=2272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:06.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.324+0000 7f936fcaf640 1 --2- 172.21.15.57:0/268086442 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f93680cf340 0x7f93680b66c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:06.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.324+0000 7f936fcaf640 1 --2- 172.21.15.57:0/268086442 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f93680ceb40 0x7f93681f11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:06.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.324+0000 7f936fcaf640 1 -- 172.21.15.57:0/268086442 >> 172.21.15.57:0/268086442 conn(0x7f93680d0d60 msgr2=0x7f93680ad8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:06.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.324+0000 7f936fcaf640 1 -- 172.21.15.57:0/268086442 shutdown_connections 2023-10-19T14:20:06.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.324+0000 7f936fcaf640 1 -- 172.21.15.57:0/268086442 wait complete. 2023-10-19T14:20:06.337 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T14:20:06.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.443+0000 7fef399a9640 1 -- 172.21.15.57:0/2752061942 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fef34155580 msgr2=0x7fef340b0d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:06.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.443+0000 7fef399a9640 1 --2- 172.21.15.57:0/2752061942 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fef34155580 0x7fef340b0d70 secure :-1 s=READY pgs=2274 cs=0 l=1 rev1=1 crypto rx=0x7fef2c009830 tx=0x7fef2c02bdf0 comp rx=0 tx=0).stop 2023-10-19T14:20:06.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.443+0000 7fef399a9640 1 -- 172.21.15.57:0/2752061942 shutdown_connections 2023-10-19T14:20:06.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.443+0000 7fef399a9640 1 --2- 172.21.15.57:0/2752061942 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fef34155580 0x7fef340b0d70 unknown :-1 s=CLOSED pgs=2274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:06.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.443+0000 7fef399a9640 1 -- 172.21.15.57:0/2752061942 >> 172.21.15.57:0/2752061942 conn(0x7fef3406fae0 msgr2=0x7fef3406fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:06.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.443+0000 7fef399a9640 1 -- 172.21.15.57:0/2752061942 shutdown_connections 2023-10-19T14:20:06.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.443+0000 7fef399a9640 1 -- 172.21.15.57:0/2752061942 wait complete. 2023-10-19T14:20:06.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.444+0000 7fef399a9640 1 Processor -- start 2023-10-19T14:20:06.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.445+0000 7fef399a9640 1 -- start start 2023-10-19T14:20:06.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.445+0000 7fef399a9640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fef3406ae40 0x7fef3413ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:06.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.445+0000 7fef399a9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fef340b12b0 0x7fef3414f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:06.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.445+0000 7fef399a9640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fef34155580 0x7fef34153c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:06.452 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.445+0000 7fef399a9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef3413b1d0 con 0x7fef340b12b0 2023-10-19T14:20:06.452 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.445+0000 7fef399a9640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef3414e860 con 0x7fef3406ae40 2023-10-19T14:20:06.452 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.445+0000 7fef399a9640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef3414eb40 con 0x7fef34155580 2023-10-19T14:20:06.452 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.445+0000 7fef327fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fef340b12b0 0x7fef3414f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:06.452 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.445+0000 7fef327fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fef340b12b0 0x7fef3414f900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:48088/0 (socket says 172.21.15.57:48088) 2023-10-19T14:20:06.453 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.445+0000 7fef327fc640 1 -- 172.21.15.57:0/1733286325 learned_addr learned my addr 172.21.15.57:0/1733286325 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:06.453 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.446+0000 7fef32ffd640 1 --2- 172.21.15.57:0/1733286325 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fef3406ae40 0x7fef3413ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:06.453 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.446+0000 7fef337fe640 1 --2- 172.21.15.57:0/1733286325 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fef34155580 0x7fef34153c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:06.453 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.446+0000 7fef327fc640 1 -- 172.21.15.57:0/1733286325 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fef34155580 msgr2=0x7fef34153c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:06.453 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.446+0000 7fef327fc640 1 --2- 172.21.15.57:0/1733286325 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fef34155580 0x7fef34153c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:06.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.446+0000 7fef327fc640 1 -- 172.21.15.57:0/1733286325 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fef3406ae40 msgr2=0x7fef3413ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:06.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.446+0000 7fef327fc640 1 --2- 172.21.15.57:0/1733286325 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fef3406ae40 0x7fef3413ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:06.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.446+0000 7fef327fc640 1 -- 172.21.15.57:0/1733286325 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef2c009490 con 0x7fef340b12b0 2023-10-19T14:20:06.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.446+0000 7fef327fc640 1 --2- 172.21.15.57:0/1733286325 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fef340b12b0 0x7fef3414f900 secure :-1 s=READY pgs=2275 cs=0 l=1 rev1=1 crypto rx=0x7fef1c00c450 tx=0x7fef1c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:06.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.446+0000 7fef2bfff640 1 -- 172.21.15.57:0/1733286325 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef1c0052d0 con 0x7fef340b12b0 2023-10-19T14:20:06.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.446+0000 7fef2bfff640 1 -- 172.21.15.57:0/1733286325 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fef1c005430 con 0x7fef340b12b0 2023-10-19T14:20:06.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.446+0000 7fef399a9640 1 -- 172.21.15.57:0/1733286325 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef3414ee20 con 0x7fef340b12b0 2023-10-19T14:20:06.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.446+0000 7fef2bfff640 1 -- 172.21.15.57:0/1733286325 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef1c012890 con 0x7fef340b12b0 2023-10-19T14:20:06.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.446+0000 7fef399a9640 1 -- 172.21.15.57:0/1733286325 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef3414f3e0 con 0x7fef340b12b0 2023-10-19T14:20:06.456 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.447+0000 7fef399a9640 1 -- 172.21.15.57:0/1733286325 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef3414f600 con 0x7fef340b12b0 2023-10-19T14:20:06.456 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.447+0000 7fef2bfff640 1 -- 172.21.15.57:0/1733286325 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 125) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fef1c0035f0 con 0x7fef340b12b0 2023-10-19T14:20:06.456 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.448+0000 7fef2bfff640 1 -- 172.21.15.57:0/1733286325 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(211..211 src has 1..211) v4 ==== 6946+0+0 (secure 0 0 0) 0x7fef1c089190 con 0x7fef340b12b0 2023-10-19T14:20:06.456 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.452+0000 7fef2bfff640 1 -- 172.21.15.57:0/1733286325 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fef1c0541b0 con 0x7fef340b12b0 2023-10-19T14:20:06.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.666+0000 7fef399a9640 1 -- 172.21.15.57:0/1733286325 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fef340690e0 con 0x7fef340b12b0 2023-10-19T14:20:06.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.667+0000 7fef2bfff640 1 -- 172.21.15.57:0/1733286325 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+4926 (secure 0 0 0) 0x7fef1c0577d0 con 0x7fef340b12b0 2023-10-19T14:20:06.671 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:20:06.671 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":83,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/2313260224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2313260224},{"type":"v1","addr":"172.21.15.179:6836","nonce":2313260224}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":10081,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/494432208","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":494432208},{"type":"v1","addr":"172.21.15.179:6837","nonce":494432208}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":11214,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/3080658822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":3080658822},{"type":"v1","addr":"172.21.15.57:6834","nonce":3080658822}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T14:18:38.834485+0000","modified":"2023-10-19T14:18:40.833363+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11217":{"gid":11217,"name":"a","rank":0,"incarnation":81,"state":"up:active","state_seq":3,"addr":"172.21.15.57:6835/1674945162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1674945162},{"type":"v1","addr":"172.21.15.57:6835","nonce":1674945162}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2023-10-19T14:20:06.672 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 83 2023-10-19T14:20:06.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.668+0000 7fef399a9640 1 -- 172.21.15.57:0/1733286325 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fef340b12b0 msgr2=0x7fef3414f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:06.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.668+0000 7fef399a9640 1 --2- 172.21.15.57:0/1733286325 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fef340b12b0 0x7fef3414f900 secure :-1 s=READY pgs=2275 cs=0 l=1 rev1=1 crypto rx=0x7fef1c00c450 tx=0x7fef1c00c910 comp rx=0 tx=0).stop 2023-10-19T14:20:06.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.668+0000 7fef399a9640 1 -- 172.21.15.57:0/1733286325 shutdown_connections 2023-10-19T14:20:06.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.668+0000 7fef399a9640 1 --2- 172.21.15.57:0/1733286325 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fef34155580 0x7fef34153c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:06.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.668+0000 7fef399a9640 1 --2- 172.21.15.57:0/1733286325 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fef340b12b0 0x7fef3414f900 unknown :-1 s=CLOSED pgs=2275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:06.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.668+0000 7fef399a9640 1 --2- 172.21.15.57:0/1733286325 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fef3406ae40 0x7fef3413ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:06.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.668+0000 7fef399a9640 1 -- 172.21.15.57:0/1733286325 >> 172.21.15.57:0/1733286325 conn(0x7fef3406fae0 msgr2=0x7fef340b0120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:06.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.669+0000 7fef399a9640 1 -- 172.21.15.57:0/1733286325 shutdown_connections 2023-10-19T14:20:06.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.669+0000 7fef399a9640 1 -- 172.21.15.57:0/1733286325 wait complete. 2023-10-19T14:20:06.682 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T14:20:06.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.791+0000 7fc7f698e640 1 -- 172.21.15.57:0/2763403001 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc7f015c1f0 msgr2=0x7fc7f00b0e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:06.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.791+0000 7fc7f698e640 1 --2- 172.21.15.57:0/2763403001 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc7f015c1f0 0x7fc7f00b0e90 secure :-1 s=READY pgs=2277 cs=0 l=1 rev1=1 crypto rx=0x7fc7e8005fd0 tx=0x7fc7e802c170 comp rx=0 tx=0).stop 2023-10-19T14:20:06.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.791+0000 7fc7f698e640 1 -- 172.21.15.57:0/2763403001 shutdown_connections 2023-10-19T14:20:06.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.791+0000 7fc7f698e640 1 --2- 172.21.15.57:0/2763403001 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc7f015c1f0 0x7fc7f00b0e90 unknown :-1 s=CLOSED pgs=2277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:06.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.791+0000 7fc7f698e640 1 -- 172.21.15.57:0/2763403001 >> 172.21.15.57:0/2763403001 conn(0x7fc7f006fae0 msgr2=0x7fc7f006fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:06.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.791+0000 7fc7f698e640 1 -- 172.21.15.57:0/2763403001 shutdown_connections 2023-10-19T14:20:06.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.791+0000 7fc7f698e640 1 -- 172.21.15.57:0/2763403001 wait complete. 2023-10-19T14:20:06.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.792+0000 7fc7f698e640 1 Processor -- start 2023-10-19T14:20:06.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.792+0000 7fc7f698e640 1 -- start start 2023-10-19T14:20:06.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.793+0000 7fc7f698e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc7f006ae40 0x7fc7f01555c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:06.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.793+0000 7fc7f698e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc7f01579f0 0x7fc7f0153270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:06.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.793+0000 7fc7f698e640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc7f015c1f0 0x7fc7f0152070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:06.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.793+0000 7fc7f698e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7f01542d0 con 0x7fc7f01579f0 2023-10-19T14:20:06.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.793+0000 7fc7f698e640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7f0154520 con 0x7fc7f006ae40 2023-10-19T14:20:06.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.793+0000 7fc7f698e640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7f0154800 con 0x7fc7f015c1f0 2023-10-19T14:20:06.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.793+0000 7fc7ef7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc7f01579f0 0x7fc7f0153270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:06.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.793+0000 7fc7ef7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc7f01579f0 0x7fc7f0153270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35480/0 (socket says 172.21.15.57:35480) 2023-10-19T14:20:06.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.793+0000 7fc7ef7fe640 1 -- 172.21.15.57:0/1365991745 learned_addr learned my addr 172.21.15.57:0/1365991745 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:06.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.793+0000 7fc7f4f2b640 1 --2- 172.21.15.57:0/1365991745 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc7f015c1f0 0x7fc7f0152070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:06.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.793+0000 7fc7effff640 1 --2- 172.21.15.57:0/1365991745 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc7f006ae40 0x7fc7f01555c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:06.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.793+0000 7fc7ef7fe640 1 -- 172.21.15.57:0/1365991745 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc7f015c1f0 msgr2=0x7fc7f0152070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:06.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.793+0000 7fc7ef7fe640 1 --2- 172.21.15.57:0/1365991745 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc7f015c1f0 0x7fc7f0152070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:06.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.793+0000 7fc7ef7fe640 1 -- 172.21.15.57:0/1365991745 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc7f006ae40 msgr2=0x7fc7f01555c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:06.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.793+0000 7fc7ef7fe640 1 --2- 172.21.15.57:0/1365991745 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc7f006ae40 0x7fc7f01555c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:06.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.793+0000 7fc7ef7fe640 1 -- 172.21.15.57:0/1365991745 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7e8009840 con 0x7fc7f01579f0 2023-10-19T14:20:06.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.793+0000 7fc7ef7fe640 1 --2- 172.21.15.57:0/1365991745 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc7f01579f0 0x7fc7f0153270 secure :-1 s=READY pgs=2278 cs=0 l=1 rev1=1 crypto rx=0x7fc7e400c400 tx=0x7fc7e400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:06.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.794+0000 7fc7ecff9640 1 -- 172.21.15.57:0/1365991745 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7e40052d0 con 0x7fc7f01579f0 2023-10-19T14:20:06.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.794+0000 7fc7ecff9640 1 -- 172.21.15.57:0/1365991745 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fc7e4007020 con 0x7fc7f01579f0 2023-10-19T14:20:06.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.794+0000 7fc7f698e640 1 -- 172.21.15.57:0/1365991745 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7f0154ae0 con 0x7fc7f01579f0 2023-10-19T14:20:06.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.794+0000 7fc7ecff9640 1 -- 172.21.15.57:0/1365991745 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7e4007470 con 0x7fc7f01579f0 2023-10-19T14:20:06.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.794+0000 7fc7f698e640 1 -- 172.21.15.57:0/1365991745 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7f01550a0 con 0x7fc7f01579f0 2023-10-19T14:20:06.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.795+0000 7fc7f698e640 1 -- 172.21.15.57:0/1365991745 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7f01552c0 con 0x7fc7f01579f0 2023-10-19T14:20:06.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.795+0000 7fc7ecff9640 1 -- 172.21.15.57:0/1365991745 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 125) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fc7e40067d0 con 0x7fc7f01579f0 2023-10-19T14:20:06.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.796+0000 7fc7ecff9640 1 -- 172.21.15.57:0/1365991745 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(211..211 src has 1..211) v4 ==== 6946+0+0 (secure 0 0 0) 0x7fc7e4088bd0 con 0x7fc7f01579f0 2023-10-19T14:20:06.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.800+0000 7fc7ecff9640 1 -- 172.21.15.57:0/1365991745 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc7e4053bf0 con 0x7fc7f01579f0 2023-10-19T14:20:06.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.975+0000 7fc7ecff9640 1 -- 172.21.15.57:0/1365991745 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mgrmap(e 126) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fc7e40536a0 con 0x7fc7f01579f0 2023-10-19T14:20:06.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.975+0000 7fc7ecff9640 1 --2- 172.21.15.57:0/1365991745 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fc7dc070df0 0x7fc7dc0731e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:06.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.977+0000 7fc7effff640 1 --2- 172.21.15.57:0/1365991745 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fc7dc070df0 0x7fc7dc0731e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:06.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.979+0000 7fc7effff640 1 --2- 172.21.15.57:0/1365991745 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fc7dc070df0 0x7fc7dc0731e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc7e00011e0 tx=0x7fc7e0009040 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:06.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.983+0000 7fc7f698e640 1 -- 172.21.15.57:0/1365991745 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc7f00690e0 con 0x7fc7f01579f0 2023-10-19T14:20:06.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.984+0000 7fc7ecff9640 1 -- 172.21.15.57:0/1365991745 <== mon.0 v2:172.21.15.57:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v211) v1 ==== 74+0+18123 (secure 0 0 0) 0x7fc7e40740f0 con 0x7fc7f01579f0 2023-10-19T14:20:06.988 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:20:06.988 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":211,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T14:20:05.962894+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2023-10-19T14:18:37.065110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"204","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_data","create_time":"2023-10-19T14:18:37.809854+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"210","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":210,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"26.f","mappings":[{"from":1,"to":0}]},{"pgid":"26.2a","mappings":[{"from":1,"to":0}]},{"pgid":"26.3a","mappings":[{"from":3,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.179:0/31901178":"2023-10-20T14:20:05.962854+0000","172.21.15.179:0/2690789813":"2023-10-20T14:20:05.962854+0000","172.21.15.179:0/3626712747":"2023-10-20T14:20:05.962854+0000","172.21.15.179:0/613069221":"2023-10-20T14:20:05.962854+0000","172.21.15.179:6833/366917243":"2023-10-20T14:20:05.962854+0000","172.21.15.179:6832/366917243":"2023-10-20T14:20:05.962854+0000","172.21.15.179:0/3217199253":"2023-10-20T14:20:05.962854+0000","172.21.15.179:0/3870514757":"2023-10-20T14:20:05.962854+0000","172.21.15.179:0/2246649044":"2023-10-20T14:18:49.915871+0000","172.21.15.179:0/1186360784":"2023-10-20T14:18:49.915871+0000","172.21.15.179:0/1525973956":"2023-10-20T14:18:49.915871+0000","172.21.15.179:0/1500866753":"2023-10-20T14:18:49.915871+0000","172.21.15.179:0/940398488":"2023-10-20T14:20:05.962854+0000","172.21.15.179:6833/2730461479":"2023-10-20T14:18:49.915871+0000","172.21.15.179:6832/2730461479":"2023-10-20T14:18:49.915871+0000","172.21.15.179:0/1627936534":"2023-10-20T14:20:05.962854+0000","172.21.15.179:0/3327639073":"2023-10-20T14:18:49.915871+0000","172.21.15.179:0/1497317399":"2023-10-20T14:18:49.915871+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T14:20:06.990 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.987+0000 7fc7f698e640 1 -- 172.21.15.57:0/1365991745 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fc7dc070df0 msgr2=0x7fc7dc0731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:06.990 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.987+0000 7fc7f698e640 1 --2- 172.21.15.57:0/1365991745 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fc7dc070df0 0x7fc7dc0731e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc7e00011e0 tx=0x7fc7e0009040 comp rx=0 tx=0).stop 2023-10-19T14:20:06.990 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.987+0000 7fc7f698e640 1 -- 172.21.15.57:0/1365991745 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc7f01579f0 msgr2=0x7fc7f0153270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:06.990 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.987+0000 7fc7f698e640 1 --2- 172.21.15.57:0/1365991745 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc7f01579f0 0x7fc7f0153270 secure :-1 s=READY pgs=2278 cs=0 l=1 rev1=1 crypto rx=0x7fc7e400c400 tx=0x7fc7e400c8c0 comp rx=0 tx=0).stop 2023-10-19T14:20:06.990 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.987+0000 7fc7f698e640 1 -- 172.21.15.57:0/1365991745 shutdown_connections 2023-10-19T14:20:06.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.987+0000 7fc7f698e640 1 --2- 172.21.15.57:0/1365991745 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fc7dc070df0 0x7fc7dc0731e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:06.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.987+0000 7fc7f698e640 1 --2- 172.21.15.57:0/1365991745 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc7f015c1f0 0x7fc7f0152070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:06.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.987+0000 7fc7f698e640 1 --2- 172.21.15.57:0/1365991745 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc7f01579f0 0x7fc7f0153270 unknown :-1 s=CLOSED pgs=2278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:06.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.987+0000 7fc7f698e640 1 --2- 172.21.15.57:0/1365991745 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc7f006ae40 0x7fc7f01555c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:06.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.987+0000 7fc7f698e640 1 -- 172.21.15.57:0/1365991745 >> 172.21.15.57:0/1365991745 conn(0x7fc7f006fae0 msgr2=0x7fc7f0159120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:06.992 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.987+0000 7fc7f698e640 1 -- 172.21.15.57:0/1365991745 shutdown_connections 2023-10-19T14:20:06.992 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:06.987+0000 7fc7f698e640 1 -- 172.21.15.57:0/1365991745 wait complete. 2023-10-19T14:20:07.000 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-19T14:20:07.001 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T14:20:07.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.107+0000 7f59713e2640 1 -- 172.21.15.57:0/2892301276 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f596c0ceb40 msgr2=0x7f596c0b0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:07.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.107+0000 7f59713e2640 1 --2- 172.21.15.57:0/2892301276 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f596c0ceb40 0x7f596c0b0650 secure :-1 s=READY pgs=2280 cs=0 l=1 rev1=1 crypto rx=0x7f5964005fd0 tx=0x7f596402bd90 comp rx=0 tx=0).stop 2023-10-19T14:20:07.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.107+0000 7f59713e2640 1 -- 172.21.15.57:0/2892301276 shutdown_connections 2023-10-19T14:20:07.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.107+0000 7f59713e2640 1 --2- 172.21.15.57:0/2892301276 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f596c15b2a0 0x7f596c0d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:07.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.107+0000 7f59713e2640 1 --2- 172.21.15.57:0/2892301276 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f596c0ceb40 0x7f596c0b0650 unknown :-1 s=CLOSED pgs=2280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:07.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.107+0000 7f59713e2640 1 -- 172.21.15.57:0/2892301276 >> 172.21.15.57:0/2892301276 conn(0x7f596c0d0d60 msgr2=0x7f596c0d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:07.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.107+0000 7f59713e2640 1 -- 172.21.15.57:0/2892301276 shutdown_connections 2023-10-19T14:20:07.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.107+0000 7f59713e2640 1 -- 172.21.15.57:0/2892301276 wait complete. 2023-10-19T14:20:07.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.108+0000 7f59713e2640 1 Processor -- start 2023-10-19T14:20:07.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.108+0000 7f59713e2640 1 -- start start 2023-10-19T14:20:07.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.109+0000 7f59713e2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f596c0ceb40 0x7f596c1f0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:07.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.109+0000 7f59713e2640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f596c0cf340 0x7f596c1f5650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:07.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.109+0000 7f59713e2640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f596c15b2a0 0x7f596c158b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:07.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.109+0000 7f59713e2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f596c0c9860 con 0x7f596c0ceb40 2023-10-19T14:20:07.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.109+0000 7f59713e2640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f596c0caa20 con 0x7f596c15b2a0 2023-10-19T14:20:07.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.109+0000 7f59713e2640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f596c0c9b00 con 0x7f596c0cf340 2023-10-19T14:20:07.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.109+0000 7f596affd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f596c0ceb40 0x7f596c1f0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:07.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.109+0000 7f596affd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f596c0ceb40 0x7f596c1f0ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35492/0 (socket says 172.21.15.57:35492) 2023-10-19T14:20:07.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.109+0000 7f596affd640 1 -- 172.21.15.57:0/484101192 learned_addr learned my addr 172.21.15.57:0/484101192 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:07.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.109+0000 7f596a7fc640 1 --2- 172.21.15.57:0/484101192 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f596c0cf340 0x7f596c1f5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:07.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.110+0000 7f596b7fe640 1 --2- 172.21.15.57:0/484101192 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f596c15b2a0 0x7f596c158b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:07.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.110+0000 7f596affd640 1 -- 172.21.15.57:0/484101192 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f596c0cf340 msgr2=0x7f596c1f5650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:07.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.110+0000 7f596affd640 1 --2- 172.21.15.57:0/484101192 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f596c0cf340 0x7f596c1f5650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:07.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.110+0000 7f596affd640 1 -- 172.21.15.57:0/484101192 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f596c15b2a0 msgr2=0x7f596c158b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:07.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.110+0000 7f596affd640 1 --2- 172.21.15.57:0/484101192 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f596c15b2a0 0x7f596c158b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:07.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.110+0000 7f596affd640 1 -- 172.21.15.57:0/484101192 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5964009560 con 0x7f596c0ceb40 2023-10-19T14:20:07.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.110+0000 7f596b7fe640 1 --2- 172.21.15.57:0/484101192 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f596c15b2a0 0x7f596c158b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:20:07.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.110+0000 7f596affd640 1 --2- 172.21.15.57:0/484101192 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f596c0ceb40 0x7f596c1f0ff0 secure :-1 s=READY pgs=2281 cs=0 l=1 rev1=1 crypto rx=0x7f595c00c870 tx=0x7f595c00cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:07.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.110+0000 7f59637fe640 1 -- 172.21.15.57:0/484101192 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f595c003b70 con 0x7f596c0ceb40 2023-10-19T14:20:07.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.110+0000 7f59637fe640 1 -- 172.21.15.57:0/484101192 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f595c003cf0 con 0x7f596c0ceb40 2023-10-19T14:20:07.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.110+0000 7f59637fe640 1 -- 172.21.15.57:0/484101192 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f595c003ea0 con 0x7f596c0ceb40 2023-10-19T14:20:07.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.110+0000 7f59713e2640 1 -- 172.21.15.57:0/484101192 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f596c0b0d50 con 0x7f596c0ceb40 2023-10-19T14:20:07.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.110+0000 7f59713e2640 1 -- 172.21.15.57:0/484101192 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f596c0cad00 con 0x7f596c0ceb40 2023-10-19T14:20:07.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.111+0000 7f59713e2640 1 -- 172.21.15.57:0/484101192 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f596c0690e0 con 0x7f596c0ceb40 2023-10-19T14:20:07.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.112+0000 7f59637fe640 1 -- 172.21.15.57:0/484101192 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 126) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f595c00aab0 con 0x7f596c0ceb40 2023-10-19T14:20:07.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.112+0000 7f59637fe640 1 --2- 172.21.15.57:0/484101192 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f594c06ac90 0x7f594c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:07.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.112+0000 7f59637fe640 1 -- 172.21.15.57:0/484101192 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(211..211 src has 1..211) v4 ==== 6946+0+0 (secure 0 0 0) 0x7f595c089680 con 0x7f596c0ceb40 2023-10-19T14:20:07.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.112+0000 7f596a7fc640 1 --2- 172.21.15.57:0/484101192 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f594c06ac90 0x7f594c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:07.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.113+0000 7f596a7fc640 1 --2- 172.21.15.57:0/484101192 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f594c06ac90 0x7f594c06d150 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f596c0ae5d0 tx=0x7f595400b2e0 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:07.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.116+0000 7f59637fe640 1 -- 172.21.15.57:0/484101192 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f595c054590 con 0x7f596c0ceb40 2023-10-19T14:20:07.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.333+0000 7f59713e2640 1 -- 172.21.15.57:0/484101192 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f596c0d0b40 con 0x7f596c0ceb40 2023-10-19T14:20:07.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.334+0000 7f59637fe640 1 -- 172.21.15.57:0/484101192 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+4926 (secure 0 0 0) 0x7f595c057bb0 con 0x7f596c0ceb40 2023-10-19T14:20:07.337 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:20:07.338 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":83,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/2313260224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2313260224},{"type":"v1","addr":"172.21.15.179:6836","nonce":2313260224}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":10081,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/494432208","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":494432208},{"type":"v1","addr":"172.21.15.179:6837","nonce":494432208}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":11214,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/3080658822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":3080658822},{"type":"v1","addr":"172.21.15.57:6834","nonce":3080658822}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T14:18:38.834485+0000","modified":"2023-10-19T14:18:40.833363+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11217":{"gid":11217,"name":"a","rank":0,"incarnation":81,"state":"up:active","state_seq":3,"addr":"172.21.15.57:6835/1674945162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1674945162},{"type":"v1","addr":"172.21.15.57:6835","nonce":1674945162}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2023-10-19T14:20:07.338 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 83 2023-10-19T14:20:07.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.336+0000 7f59713e2640 1 -- 172.21.15.57:0/484101192 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f594c06ac90 msgr2=0x7f594c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:07.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.336+0000 7f59713e2640 1 --2- 172.21.15.57:0/484101192 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f594c06ac90 0x7f594c06d150 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f596c0ae5d0 tx=0x7f595400b2e0 comp rx=0 tx=0).stop 2023-10-19T14:20:07.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.336+0000 7f59713e2640 1 -- 172.21.15.57:0/484101192 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f596c0ceb40 msgr2=0x7f596c1f0ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:07.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.336+0000 7f59713e2640 1 --2- 172.21.15.57:0/484101192 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f596c0ceb40 0x7f596c1f0ff0 secure :-1 s=READY pgs=2281 cs=0 l=1 rev1=1 crypto rx=0x7f595c00c870 tx=0x7f595c00cd30 comp rx=0 tx=0).stop 2023-10-19T14:20:07.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.336+0000 7f59713e2640 1 -- 172.21.15.57:0/484101192 shutdown_connections 2023-10-19T14:20:07.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.336+0000 7f59713e2640 1 --2- 172.21.15.57:0/484101192 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f594c06ac90 0x7f594c06d150 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:07.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.336+0000 7f59713e2640 1 --2- 172.21.15.57:0/484101192 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f596c15b2a0 0x7f596c158b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:07.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.336+0000 7f59713e2640 1 --2- 172.21.15.57:0/484101192 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f596c0cf340 0x7f596c1f5650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:07.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.336+0000 7f59713e2640 1 --2- 172.21.15.57:0/484101192 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f596c0ceb40 0x7f596c1f0ff0 unknown :-1 s=CLOSED pgs=2281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:07.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.336+0000 7f59713e2640 1 -- 172.21.15.57:0/484101192 >> 172.21.15.57:0/484101192 conn(0x7f596c0d0d60 msgr2=0x7f596c159fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:07.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.336+0000 7f59713e2640 1 -- 172.21.15.57:0/484101192 shutdown_connections 2023-10-19T14:20:07.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.336+0000 7f59713e2640 1 -- 172.21.15.57:0/484101192 wait complete. 2023-10-19T14:20:07.350 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T14:20:07.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.455+0000 7f8ebfc44640 1 -- 172.21.15.57:0/1431270005 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8eb80cf340 msgr2=0x7f8eb80d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:07.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.455+0000 7f8ebfc44640 1 --2- 172.21.15.57:0/1431270005 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8eb80cf340 0x7f8eb80d1910 secure :-1 s=READY pgs=2282 cs=0 l=1 rev1=1 crypto rx=0x7f8eb4005fd0 tx=0x7f8eb402bd90 comp rx=0 tx=0).stop 2023-10-19T14:20:07.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.455+0000 7f8ebfc44640 1 -- 172.21.15.57:0/1431270005 shutdown_connections 2023-10-19T14:20:07.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.455+0000 7f8ebfc44640 1 --2- 172.21.15.57:0/1431270005 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8eb80ceb40 0x7f8eb80b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:07.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.455+0000 7f8ebfc44640 1 --2- 172.21.15.57:0/1431270005 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8eb80cf340 0x7f8eb80d1910 unknown :-1 s=CLOSED pgs=2282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:07.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.455+0000 7f8ebfc44640 1 -- 172.21.15.57:0/1431270005 >> 172.21.15.57:0/1431270005 conn(0x7f8eb80d0d60 msgr2=0x7f8eb80d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:07.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.455+0000 7f8ebfc44640 1 -- 172.21.15.57:0/1431270005 shutdown_connections 2023-10-19T14:20:07.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.455+0000 7f8ebfc44640 1 -- 172.21.15.57:0/1431270005 wait complete. 2023-10-19T14:20:07.462 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.456+0000 7f8ebfc44640 1 Processor -- start 2023-10-19T14:20:07.462 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.457+0000 7f8ebfc44640 1 -- start start 2023-10-19T14:20:07.462 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.457+0000 7f8ebfc44640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8eb80ceb40 0x7f8eb80c7270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:07.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.457+0000 7f8ebfc44640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8eb80cf340 0x7f8eb81e5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:07.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.457+0000 7f8ebfc44640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8eb815b2a0 0x7f8eb81469e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:07.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.457+0000 7f8ebfc44640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb81f1660 con 0x7f8eb815b2a0 2023-10-19T14:20:07.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.457+0000 7f8ebfc44640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb815b860 con 0x7f8eb80cf340 2023-10-19T14:20:07.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.457+0000 7f8ebfc44640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb815bb40 con 0x7f8eb80ceb40 2023-10-19T14:20:07.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.457+0000 7f8ebe1e1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8eb815b2a0 0x7f8eb81469e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:07.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.457+0000 7f8ebe1e1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8eb815b2a0 0x7f8eb81469e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35496/0 (socket says 172.21.15.57:35496) 2023-10-19T14:20:07.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.457+0000 7f8ebe1e1640 1 -- 172.21.15.57:0/2167446351 learned_addr learned my addr 172.21.15.57:0/2167446351 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:07.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.458+0000 7f8ebd9e0640 1 --2- 172.21.15.57:0/2167446351 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8eb80ceb40 0x7f8eb80c7270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:07.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.458+0000 7f8ebd1df640 1 --2- 172.21.15.57:0/2167446351 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8eb80cf340 0x7f8eb81e5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:07.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.458+0000 7f8ebe1e1640 1 -- 172.21.15.57:0/2167446351 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8eb80ceb40 msgr2=0x7f8eb80c7270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:07.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.458+0000 7f8ebe1e1640 1 --2- 172.21.15.57:0/2167446351 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8eb80ceb40 0x7f8eb80c7270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:07.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.458+0000 7f8ebe1e1640 1 -- 172.21.15.57:0/2167446351 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8eb80cf340 msgr2=0x7f8eb81e5ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:07.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.458+0000 7f8ebe1e1640 1 --2- 172.21.15.57:0/2167446351 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8eb80cf340 0x7f8eb81e5ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:07.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.458+0000 7f8ebe1e1640 1 -- 172.21.15.57:0/2167446351 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eb4009560 con 0x7f8eb815b2a0 2023-10-19T14:20:07.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.458+0000 7f8ebd1df640 1 --2- 172.21.15.57:0/2167446351 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8eb80cf340 0x7f8eb81e5ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:20:07.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.458+0000 7f8ebd9e0640 1 --2- 172.21.15.57:0/2167446351 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8eb80ceb40 0x7f8eb80c7270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:20:07.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.458+0000 7f8ebe1e1640 1 --2- 172.21.15.57:0/2167446351 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8eb815b2a0 0x7f8eb81469e0 secure :-1 s=READY pgs=2283 cs=0 l=1 rev1=1 crypto rx=0x7f8eb000da70 tx=0x7f8eb000df30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:07.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.458+0000 7f8eae7fc640 1 -- 172.21.15.57:0/2167446351 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eb000bdd0 con 0x7f8eb815b2a0 2023-10-19T14:20:07.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.458+0000 7f8eae7fc640 1 -- 172.21.15.57:0/2167446351 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f8eb0003a60 con 0x7f8eb815b2a0 2023-10-19T14:20:07.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.458+0000 7f8ebfc44640 1 -- 172.21.15.57:0/2167446351 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8eb80b0d50 con 0x7f8eb815b2a0 2023-10-19T14:20:07.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.458+0000 7f8eae7fc640 1 -- 172.21.15.57:0/2167446351 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eb0003e70 con 0x7f8eb815b2a0 2023-10-19T14:20:07.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.458+0000 7f8ebfc44640 1 -- 172.21.15.57:0/2167446351 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8eb81e56c0 con 0x7f8eb815b2a0 2023-10-19T14:20:07.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.459+0000 7f8ebfc44640 1 -- 172.21.15.57:0/2167446351 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8eb80690e0 con 0x7f8eb815b2a0 2023-10-19T14:20:07.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.459+0000 7f8eae7fc640 1 -- 172.21.15.57:0/2167446351 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 126) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f8eb0004000 con 0x7f8eb815b2a0 2023-10-19T14:20:07.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.460+0000 7f8eae7fc640 1 --2- 172.21.15.57:0/2167446351 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f8e9c06acc0 0x7f8e9c06d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:07.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.460+0000 7f8eae7fc640 1 -- 172.21.15.57:0/2167446351 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(211..211 src has 1..211) v4 ==== 6946+0+0 (secure 0 0 0) 0x7f8eb0089d50 con 0x7f8eb815b2a0 2023-10-19T14:20:07.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.460+0000 7f8ebd9e0640 1 --2- 172.21.15.57:0/2167446351 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f8e9c06acc0 0x7f8e9c06d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:07.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.461+0000 7f8ebd9e0640 1 --2- 172.21.15.57:0/2167446351 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f8e9c06acc0 0x7f8e9c06d180 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8eb4035000 tx=0x7f8eb4004730 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:07.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.464+0000 7f8eae7fc640 1 -- 172.21.15.57:0/2167446351 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8eb0054be0 con 0x7f8eb815b2a0 2023-10-19T14:20:07.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.676+0000 7f8ebfc44640 1 -- 172.21.15.57:0/2167446351 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8eb80d0b40 con 0x7f8eb815b2a0 2023-10-19T14:20:07.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.676+0000 7f8eae7fc640 1 -- 172.21.15.57:0/2167446351 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+4926 (secure 0 0 0) 0x7f8eb0058200 con 0x7f8eb815b2a0 2023-10-19T14:20:07.679 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:20:07.680 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":83,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/2313260224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2313260224},{"type":"v1","addr":"172.21.15.179:6836","nonce":2313260224}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":10081,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/494432208","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":494432208},{"type":"v1","addr":"172.21.15.179:6837","nonce":494432208}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":11214,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/3080658822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":3080658822},{"type":"v1","addr":"172.21.15.57:6834","nonce":3080658822}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T14:18:38.834485+0000","modified":"2023-10-19T14:18:40.833363+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11217":{"gid":11217,"name":"a","rank":0,"incarnation":81,"state":"up:active","state_seq":3,"addr":"172.21.15.57:6835/1674945162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1674945162},{"type":"v1","addr":"172.21.15.57:6835","nonce":1674945162}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2023-10-19T14:20:07.680 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 83 2023-10-19T14:20:07.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.678+0000 7f8ebfc44640 1 -- 172.21.15.57:0/2167446351 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f8e9c06acc0 msgr2=0x7f8e9c06d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:07.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.678+0000 7f8ebfc44640 1 --2- 172.21.15.57:0/2167446351 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f8e9c06acc0 0x7f8e9c06d180 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8eb4035000 tx=0x7f8eb4004730 comp rx=0 tx=0).stop 2023-10-19T14:20:07.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.678+0000 7f8ebfc44640 1 -- 172.21.15.57:0/2167446351 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8eb815b2a0 msgr2=0x7f8eb81469e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:07.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.678+0000 7f8ebfc44640 1 --2- 172.21.15.57:0/2167446351 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8eb815b2a0 0x7f8eb81469e0 secure :-1 s=READY pgs=2283 cs=0 l=1 rev1=1 crypto rx=0x7f8eb000da70 tx=0x7f8eb000df30 comp rx=0 tx=0).stop 2023-10-19T14:20:07.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.678+0000 7f8ebfc44640 1 -- 172.21.15.57:0/2167446351 shutdown_connections 2023-10-19T14:20:07.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.678+0000 7f8ebfc44640 1 --2- 172.21.15.57:0/2167446351 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f8e9c06acc0 0x7f8e9c06d180 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:07.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.678+0000 7f8ebfc44640 1 --2- 172.21.15.57:0/2167446351 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8eb815b2a0 0x7f8eb81469e0 unknown :-1 s=CLOSED pgs=2283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:07.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.678+0000 7f8ebfc44640 1 --2- 172.21.15.57:0/2167446351 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8eb80cf340 0x7f8eb81e5ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:07.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.678+0000 7f8ebfc44640 1 --2- 172.21.15.57:0/2167446351 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8eb80ceb40 0x7f8eb80c7270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:07.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.678+0000 7f8ebfc44640 1 -- 172.21.15.57:0/2167446351 >> 172.21.15.57:0/2167446351 conn(0x7f8eb80d0d60 msgr2=0x7f8eb80ad8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:07.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.678+0000 7f8ebfc44640 1 -- 172.21.15.57:0/2167446351 shutdown_connections 2023-10-19T14:20:07.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.678+0000 7f8ebfc44640 1 -- 172.21.15.57:0/2167446351 wait complete. 2023-10-19T14:20:07.691 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T14:20:07.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.797+0000 7f77d9b0f640 1 -- 172.21.15.57:0/3189170940 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f77d40cf340 msgr2=0x7f77d40d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:07.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.797+0000 7f77d9b0f640 1 --2- 172.21.15.57:0/3189170940 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f77d40cf340 0x7f77d40d1910 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f77c4005fd0 tx=0x7f77c402bd90 comp rx=0 tx=0).stop 2023-10-19T14:20:07.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.798+0000 7f77d9b0f640 1 -- 172.21.15.57:0/3189170940 shutdown_connections 2023-10-19T14:20:07.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.798+0000 7f77d9b0f640 1 --2- 172.21.15.57:0/3189170940 >> v2:172.21.15.179:3300/0 conn(0x7f77d40ceb40 0x7f77d40b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:07.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.798+0000 7f77d9b0f640 1 --2- 172.21.15.57:0/3189170940 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f77d40cf340 0x7f77d40d1910 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:07.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.798+0000 7f77d9b0f640 1 -- 172.21.15.57:0/3189170940 >> 172.21.15.57:0/3189170940 conn(0x7f77d40d0d60 msgr2=0x7f77d40d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:07.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.798+0000 7f77d9b0f640 1 -- 172.21.15.57:0/3189170940 shutdown_connections 2023-10-19T14:20:07.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.798+0000 7f77d9b0f640 1 -- 172.21.15.57:0/3189170940 wait complete. 2023-10-19T14:20:07.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.799+0000 7f77d9b0f640 1 Processor -- start 2023-10-19T14:20:07.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.799+0000 7f77d9b0f640 1 -- start start 2023-10-19T14:20:07.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.800+0000 7f77d9b0f640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f77d40ceb40 0x7f77d40b4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:07.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.800+0000 7f77d9b0f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f77d40cf340 0x7f77d40ca0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:07.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.800+0000 7f77d9b0f640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f77d415b2a0 0x7f77d4157fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:07.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.800+0000 7f77d9b0f640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77d415e100 con 0x7f77d40cf340 2023-10-19T14:20:07.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.800+0000 7f77d9b0f640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77d40b4a70 con 0x7f77d415b2a0 2023-10-19T14:20:07.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.800+0000 7f77d9b0f640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77d40b4d50 con 0x7f77d40ceb40 2023-10-19T14:20:07.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.800+0000 7f77d2ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f77d40cf340 0x7f77d40ca0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:07.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.800+0000 7f77d2ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f77d40cf340 0x7f77d40ca0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35506/0 (socket says 172.21.15.57:35506) 2023-10-19T14:20:07.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.800+0000 7f77d2ffd640 1 -- 172.21.15.57:0/782757209 learned_addr learned my addr 172.21.15.57:0/782757209 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:07.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.800+0000 7f77d37fe640 1 --2- 172.21.15.57:0/782757209 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f77d40ceb40 0x7f77d40b4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:07.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.800+0000 7f77d3fff640 1 --2- 172.21.15.57:0/782757209 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f77d415b2a0 0x7f77d4157fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:07.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.800+0000 7f77d2ffd640 1 -- 172.21.15.57:0/782757209 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f77d40ceb40 msgr2=0x7f77d40b4360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:07.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.800+0000 7f77d2ffd640 1 --2- 172.21.15.57:0/782757209 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f77d40ceb40 0x7f77d40b4360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:07.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.800+0000 7f77d2ffd640 1 -- 172.21.15.57:0/782757209 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f77d415b2a0 msgr2=0x7f77d4157fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:07.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.800+0000 7f77d2ffd640 1 --2- 172.21.15.57:0/782757209 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f77d415b2a0 0x7f77d4157fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:07.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.800+0000 7f77d2ffd640 1 -- 172.21.15.57:0/782757209 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77cc009840 con 0x7f77d40cf340 2023-10-19T14:20:07.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.800+0000 7f77d3fff640 1 --2- 172.21.15.57:0/782757209 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f77d415b2a0 0x7f77d4157fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:20:07.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.800+0000 7f77d37fe640 1 --2- 172.21.15.57:0/782757209 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f77d40ceb40 0x7f77d40b4360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:20:07.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.800+0000 7f77d2ffd640 1 --2- 172.21.15.57:0/782757209 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f77d40cf340 0x7f77d40ca0b0 secure :-1 s=READY pgs=2284 cs=0 l=1 rev1=1 crypto rx=0x7f77c800c440 tx=0x7f77c800c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:07.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.800+0000 7f77bbfff640 1 -- 172.21.15.57:0/782757209 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77c80052d0 con 0x7f77d40cf340 2023-10-19T14:20:07.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.800+0000 7f77d9b0f640 1 -- 172.21.15.57:0/782757209 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77c4009560 con 0x7f77d40cf340 2023-10-19T14:20:07.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.800+0000 7f77bbfff640 1 -- 172.21.15.57:0/782757209 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f77c8007020 con 0x7f77d40cf340 2023-10-19T14:20:07.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.800+0000 7f77bbfff640 1 -- 172.21.15.57:0/782757209 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77c8007430 con 0x7f77d40cf340 2023-10-19T14:20:07.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.801+0000 7f77d9b0f640 1 -- 172.21.15.57:0/782757209 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77d40b5310 con 0x7f77d40cf340 2023-10-19T14:20:07.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.801+0000 7f77d9b0f640 1 -- 172.21.15.57:0/782757209 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77d40cfbc0 con 0x7f77d40cf340 2023-10-19T14:20:07.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.802+0000 7f77bbfff640 1 -- 172.21.15.57:0/782757209 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 126) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f77c80067d0 con 0x7f77d40cf340 2023-10-19T14:20:07.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.802+0000 7f77bbfff640 1 --2- 172.21.15.57:0/782757209 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f77c006ac90 0x7f77c006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:07.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.802+0000 7f77bbfff640 1 -- 172.21.15.57:0/782757209 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(211..211 src has 1..211) v4 ==== 6946+0+0 (secure 0 0 0) 0x7f77c8088d50 con 0x7f77d40cf340 2023-10-19T14:20:07.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.802+0000 7f77d37fe640 1 --2- 172.21.15.57:0/782757209 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f77c006ac90 0x7f77c006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:07.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.803+0000 7f77d37fe640 1 --2- 172.21.15.57:0/782757209 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f77c006ac90 0x7f77c006d150 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f77c4035000 tx=0x7f77c4004730 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:07.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.806+0000 7f77bbfff640 1 -- 172.21.15.57:0/782757209 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f77c8053be0 con 0x7f77d40cf340 2023-10-19T14:20:07.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.985+0000 7f77d9b0f640 1 -- 172.21.15.57:0/782757209 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f77d40cb9b0 con 0x7f77d40cf340 2023-10-19T14:20:07.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.987+0000 7f77bbfff640 1 -- 172.21.15.57:0/782757209 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v211) v1 ==== 74+0+18123 (secure 0 0 0) 0x7f77c8057200 con 0x7f77d40cf340 2023-10-19T14:20:07.999 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:20:07.999 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":211,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T14:20:05.962894+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2023-10-19T14:18:37.065110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"204","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_data","create_time":"2023-10-19T14:18:37.809854+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"210","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":210,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"26.f","mappings":[{"from":1,"to":0}]},{"pgid":"26.2a","mappings":[{"from":1,"to":0}]},{"pgid":"26.3a","mappings":[{"from":3,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.179:0/31901178":"2023-10-20T14:20:05.962854+0000","172.21.15.179:0/2690789813":"2023-10-20T14:20:05.962854+0000","172.21.15.179:0/3626712747":"2023-10-20T14:20:05.962854+0000","172.21.15.179:0/613069221":"2023-10-20T14:20:05.962854+0000","172.21.15.179:6833/366917243":"2023-10-20T14:20:05.962854+0000","172.21.15.179:6832/366917243":"2023-10-20T14:20:05.962854+0000","172.21.15.179:0/3217199253":"2023-10-20T14:20:05.962854+0000","172.21.15.179:0/3870514757":"2023-10-20T14:20:05.962854+0000","172.21.15.179:0/2246649044":"2023-10-20T14:18:49.915871+0000","172.21.15.179:0/1186360784":"2023-10-20T14:18:49.915871+0000","172.21.15.179:0/1525973956":"2023-10-20T14:18:49.915871+0000","172.21.15.179:0/1500866753":"2023-10-20T14:18:49.915871+0000","172.21.15.179:0/940398488":"2023-10-20T14:20:05.962854+0000","172.21.15.179:6833/2730461479":"2023-10-20T14:18:49.915871+0000","172.21.15.179:6832/2730461479":"2023-10-20T14:18:49.915871+0000","172.21.15.179:0/1627936534":"2023-10-20T14:20:05.962854+0000","172.21.15.179:0/3327639073":"2023-10-20T14:18:49.915871+0000","172.21.15.179:0/1497317399":"2023-10-20T14:18:49.915871+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T14:20:08.001 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.989+0000 7f77d9b0f640 1 -- 172.21.15.57:0/782757209 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f77c006ac90 msgr2=0x7f77c006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:08.001 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.989+0000 7f77d9b0f640 1 --2- 172.21.15.57:0/782757209 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f77c006ac90 0x7f77c006d150 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f77c4035000 tx=0x7f77c4004730 comp rx=0 tx=0).stop 2023-10-19T14:20:08.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.989+0000 7f77d9b0f640 1 -- 172.21.15.57:0/782757209 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f77d40cf340 msgr2=0x7f77d40ca0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:08.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.989+0000 7f77d9b0f640 1 --2- 172.21.15.57:0/782757209 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f77d40cf340 0x7f77d40ca0b0 secure :-1 s=READY pgs=2284 cs=0 l=1 rev1=1 crypto rx=0x7f77c800c440 tx=0x7f77c800c900 comp rx=0 tx=0).stop 2023-10-19T14:20:08.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.989+0000 7f77d9b0f640 1 -- 172.21.15.57:0/782757209 shutdown_connections 2023-10-19T14:20:08.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.989+0000 7f77d9b0f640 1 --2- 172.21.15.57:0/782757209 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f77c006ac90 0x7f77c006d150 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:08.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.989+0000 7f77d9b0f640 1 --2- 172.21.15.57:0/782757209 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f77d415b2a0 0x7f77d4157fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:08.006 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.989+0000 7f77d9b0f640 1 --2- 172.21.15.57:0/782757209 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f77d40cf340 0x7f77d40ca0b0 unknown :-1 s=CLOSED pgs=2284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:08.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.989+0000 7f77d9b0f640 1 --2- 172.21.15.57:0/782757209 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f77d40ceb40 0x7f77d40b4360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:08.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.989+0000 7f77d9b0f640 1 -- 172.21.15.57:0/782757209 >> 172.21.15.57:0/782757209 conn(0x7f77d40d0d60 msgr2=0x7f77d415a1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:08.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.989+0000 7f77d9b0f640 1 -- 172.21.15.57:0/782757209 shutdown_connections 2023-10-19T14:20:08.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:07.989+0000 7f77d9b0f640 1 -- 172.21.15.57:0/782757209 wait complete. 2023-10-19T14:20:08.017 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-19T14:20:08.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.135+0000 7f461057d640 1 -- 172.21.15.57:0/499590435 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f46080abc10 msgr2=0x7f460806b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:08.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.135+0000 7f461057d640 1 --2- 172.21.15.57:0/499590435 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f46080abc10 0x7f460806b2e0 secure :-1 s=READY pgs=2286 cs=0 l=1 rev1=1 crypto rx=0x7f4600005fd0 tx=0x7f460002c2e0 comp rx=0 tx=0).stop 2023-10-19T14:20:08.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.135+0000 7f461057d640 1 -- 172.21.15.57:0/499590435 shutdown_connections 2023-10-19T14:20:08.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.135+0000 7f461057d640 1 --2- 172.21.15.57:0/499590435 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f46080abc10 0x7f460806b2e0 unknown :-1 s=CLOSED pgs=2286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:08.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.135+0000 7f461057d640 1 -- 172.21.15.57:0/499590435 >> 172.21.15.57:0/499590435 conn(0x7f4608069930 msgr2=0x7f4608069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:08.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.135+0000 7f461057d640 1 -- 172.21.15.57:0/499590435 shutdown_connections 2023-10-19T14:20:08.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.135+0000 7f461057d640 1 -- 172.21.15.57:0/499590435 wait complete. 2023-10-19T14:20:08.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.137+0000 7f461057d640 1 Processor -- start 2023-10-19T14:20:08.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.137+0000 7f461057d640 1 -- start start 2023-10-19T14:20:08.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.137+0000 7f461057d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f460806af00 0x7f460815b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:08.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.137+0000 7f461057d640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f46080abc10 0x7f46080a64b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:08.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.137+0000 7f461057d640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f46080c9dc0 0x7f46080b08d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:08.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.137+0000 7f461057d640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46081ef7c0 con 0x7f460806af00 2023-10-19T14:20:08.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.137+0000 7f461057d640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46081516c0 con 0x7f46080c9dc0 2023-10-19T14:20:08.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.137+0000 7f461057d640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46081519a0 con 0x7f46080abc10 2023-10-19T14:20:08.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.137+0000 7f460e319640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f460806af00 0x7f460815b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:08.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.137+0000 7f460e319640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f460806af00 0x7f460815b390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35522/0 (socket says 172.21.15.57:35522) 2023-10-19T14:20:08.147 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.137+0000 7f460e319640 1 -- 172.21.15.57:0/2298021989 learned_addr learned my addr 172.21.15.57:0/2298021989 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:08.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.138+0000 7f460eb1a640 1 --2- 172.21.15.57:0/2298021989 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f46080c9dc0 0x7f46080b08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:08.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.138+0000 7f460db18640 1 --2- 172.21.15.57:0/2298021989 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f46080abc10 0x7f46080a64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:08.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.138+0000 7f460e319640 1 -- 172.21.15.57:0/2298021989 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f46080abc10 msgr2=0x7f46080a64b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:08.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.138+0000 7f460e319640 1 --2- 172.21.15.57:0/2298021989 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f46080abc10 0x7f46080a64b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:08.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.138+0000 7f460e319640 1 -- 172.21.15.57:0/2298021989 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f46080c9dc0 msgr2=0x7f46080b08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:08.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.138+0000 7f460e319640 1 --2- 172.21.15.57:0/2298021989 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f46080c9dc0 0x7f46080b08d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:08.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.138+0000 7f460e319640 1 -- 172.21.15.57:0/2298021989 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4600009900 con 0x7f460806af00 2023-10-19T14:20:08.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.138+0000 7f460e319640 1 --2- 172.21.15.57:0/2298021989 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f460806af00 0x7f460815b390 secure :-1 s=READY pgs=2287 cs=0 l=1 rev1=1 crypto rx=0x7f460400c400 tx=0x7f460400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:08.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.138+0000 7f45feffd640 1 -- 172.21.15.57:0/2298021989 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46040052d0 con 0x7f460806af00 2023-10-19T14:20:08.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.138+0000 7f45feffd640 1 -- 172.21.15.57:0/2298021989 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f4604007020 con 0x7f460806af00 2023-10-19T14:20:08.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.138+0000 7f461057d640 1 -- 172.21.15.57:0/2298021989 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4608159a20 con 0x7f460806af00 2023-10-19T14:20:08.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.138+0000 7f45feffd640 1 -- 172.21.15.57:0/2298021989 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4604007470 con 0x7f460806af00 2023-10-19T14:20:08.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.138+0000 7f461057d640 1 -- 172.21.15.57:0/2298021989 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4608151f60 con 0x7f460806af00 2023-10-19T14:20:08.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.139+0000 7f461057d640 1 -- 172.21.15.57:0/2298021989 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46080690e0 con 0x7f460806af00 2023-10-19T14:20:08.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.140+0000 7f45feffd640 1 -- 172.21.15.57:0/2298021989 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 126) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f46040067d0 con 0x7f460806af00 2023-10-19T14:20:08.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.140+0000 7f45feffd640 1 --2- 172.21.15.57:0/2298021989 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f45ec06ac90 0x7f45ec06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:08.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.140+0000 7f45feffd640 1 -- 172.21.15.57:0/2298021989 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(211..211 src has 1..211) v4 ==== 6946+0+0 (secure 0 0 0) 0x7f4604088d40 con 0x7f460806af00 2023-10-19T14:20:08.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.140+0000 7f460db18640 1 --2- 172.21.15.57:0/2298021989 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f45ec06ac90 0x7f45ec06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:08.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.141+0000 7f460db18640 1 --2- 172.21.15.57:0/2298021989 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f45ec06ac90 0x7f45ec06d150 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f46081f2ce0 tx=0x7f45f8008040 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:08.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.144+0000 7f45feffd640 1 -- 172.21.15.57:0/2298021989 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f460408c030 con 0x7f460806af00 2023-10-19T14:20:08.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.358+0000 7f461057d640 1 -- 172.21.15.57:0/2298021989 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f46080c8d40 con 0x7f460806af00 2023-10-19T14:20:08.362 INFO:tasks.ceph.mon.a.smithi057.stderr:2023-10-19T14:20:08.359+0000 7f1253ff3640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-19T14:20:08.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.361+0000 7f45feffd640 1 -- 172.21.15.57:0/2298021989 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v84) v1 ==== 161+0+0 (secure 0 0 0) 0x7f4604053c50 con 0x7f460806af00 2023-10-19T14:20:08.364 INFO:teuthology.orchestra.run.smithi057.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-19T14:20:08.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.363+0000 7f461057d640 1 -- 172.21.15.57:0/2298021989 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f45ec06ac90 msgr2=0x7f45ec06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:08.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.363+0000 7f461057d640 1 --2- 172.21.15.57:0/2298021989 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f45ec06ac90 0x7f45ec06d150 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f46081f2ce0 tx=0x7f45f8008040 comp rx=0 tx=0).stop 2023-10-19T14:20:08.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.363+0000 7f461057d640 1 -- 172.21.15.57:0/2298021989 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f460806af00 msgr2=0x7f460815b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:08.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.363+0000 7f461057d640 1 --2- 172.21.15.57:0/2298021989 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f460806af00 0x7f460815b390 secure :-1 s=READY pgs=2287 cs=0 l=1 rev1=1 crypto rx=0x7f460400c400 tx=0x7f460400c8c0 comp rx=0 tx=0).stop 2023-10-19T14:20:08.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.363+0000 7f461057d640 1 -- 172.21.15.57:0/2298021989 shutdown_connections 2023-10-19T14:20:08.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.364+0000 7f461057d640 1 --2- 172.21.15.57:0/2298021989 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f45ec06ac90 0x7f45ec06d150 secure :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f46081f2ce0 tx=0x7f45f8008040 comp rx=0 tx=0).stop 2023-10-19T14:20:08.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.364+0000 7f461057d640 1 --2- 172.21.15.57:0/2298021989 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f46080c9dc0 0x7f46080b08d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:08.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.364+0000 7f461057d640 1 --2- 172.21.15.57:0/2298021989 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f46080abc10 0x7f46080a64b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:08.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.364+0000 7f461057d640 1 --2- 172.21.15.57:0/2298021989 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f460806af00 0x7f460815b390 secure :-1 s=CLOSED pgs=2287 cs=0 l=1 rev1=1 crypto rx=0x7f460400c400 tx=0x7f460400c8c0 comp rx=0 tx=0).stop 2023-10-19T14:20:08.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.364+0000 7f461057d640 1 -- 172.21.15.57:0/2298021989 >> 172.21.15.57:0/2298021989 conn(0x7f4608069930 msgr2=0x7f460814fb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:08.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.364+0000 7f461057d640 1 -- 172.21.15.57:0/2298021989 shutdown_connections 2023-10-19T14:20:08.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.365+0000 7f461057d640 1 -- 172.21.15.57:0/2298021989 wait complete. 2023-10-19T14:20:08.379 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-19T14:20:08.403 INFO:tasks.ceph.mds.a.smithi057.stdout:starting mds.a at 2023-10-19T14:20:08.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.514+0000 7fdf5bba9640 1 -- 172.21.15.57:0/1170392312 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdf540cf340 msgr2=0x7fdf540d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:08.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.514+0000 7fdf5bba9640 1 --2- 172.21.15.57:0/1170392312 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdf540cf340 0x7fdf540d1910 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fdf4c009830 tx=0x7fdf4c02bcd0 comp rx=0 tx=0).stop 2023-10-19T14:20:08.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.514+0000 7fdf5bba9640 1 -- 172.21.15.57:0/1170392312 shutdown_connections 2023-10-19T14:20:08.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.514+0000 7fdf5bba9640 1 --2- 172.21.15.57:0/1170392312 >> v2:172.21.15.179:3300/0 conn(0x7fdf5415b2a0 0x7fdf540cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:08.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.514+0000 7fdf5bba9640 1 --2- 172.21.15.57:0/1170392312 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdf540cf340 0x7fdf540d1910 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:08.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.514+0000 7fdf5bba9640 1 -- 172.21.15.57:0/1170392312 >> 172.21.15.57:0/1170392312 conn(0x7fdf540d0d60 msgr2=0x7fdf540d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:08.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.514+0000 7fdf5bba9640 1 -- 172.21.15.57:0/1170392312 shutdown_connections 2023-10-19T14:20:08.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.514+0000 7fdf5bba9640 1 -- 172.21.15.57:0/1170392312 wait complete. 2023-10-19T14:20:08.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.515+0000 7fdf5bba9640 1 Processor -- start 2023-10-19T14:20:08.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.515+0000 7fdf5bba9640 1 -- start start 2023-10-19T14:20:08.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.516+0000 7fdf5bba9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdf540ceb40 0x7fdf540b6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:08.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.516+0000 7fdf5bba9640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdf540cf340 0x7fdf54158270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:08.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.516+0000 7fdf5bba9640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdf5415b2a0 0x7fdf541591d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:08.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.516+0000 7fdf5bba9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf5415cc40 con 0x7fdf540ceb40 2023-10-19T14:20:08.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.516+0000 7fdf5bba9640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf5415cf20 con 0x7fdf540cf340 2023-10-19T14:20:08.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.516+0000 7fdf5bba9640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf5415d200 con 0x7fdf5415b2a0 2023-10-19T14:20:08.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.516+0000 7fdf59945640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdf540ceb40 0x7fdf540b6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:08.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.516+0000 7fdf59945640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdf540ceb40 0x7fdf540b6380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35556/0 (socket says 172.21.15.57:35556) 2023-10-19T14:20:08.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.516+0000 7fdf59945640 1 -- 172.21.15.57:0/3204344563 learned_addr learned my addr 172.21.15.57:0/3204344563 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:08.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.516+0000 7fdf5a146640 1 --2- 172.21.15.57:0/3204344563 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdf5415b2a0 0x7fdf541591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:08.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.516+0000 7fdf59144640 1 --2- 172.21.15.57:0/3204344563 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdf540cf340 0x7fdf54158270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:08.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.516+0000 7fdf59945640 1 -- 172.21.15.57:0/3204344563 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdf5415b2a0 msgr2=0x7fdf541591d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:08.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.516+0000 7fdf59945640 1 --2- 172.21.15.57:0/3204344563 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdf5415b2a0 0x7fdf541591d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:08.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.516+0000 7fdf59945640 1 -- 172.21.15.57:0/3204344563 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdf540cf340 msgr2=0x7fdf54158270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:08.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.516+0000 7fdf59945640 1 --2- 172.21.15.57:0/3204344563 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdf540cf340 0x7fdf54158270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:08.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.516+0000 7fdf59945640 1 -- 172.21.15.57:0/3204344563 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf4c009490 con 0x7fdf540ceb40 2023-10-19T14:20:08.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.516+0000 7fdf5a146640 1 --2- 172.21.15.57:0/3204344563 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdf5415b2a0 0x7fdf541591d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:20:08.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.516+0000 7fdf59144640 1 --2- 172.21.15.57:0/3204344563 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdf540cf340 0x7fdf54158270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:20:08.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.516+0000 7fdf59945640 1 --2- 172.21.15.57:0/3204344563 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdf540ceb40 0x7fdf540b6380 secure :-1 s=READY pgs=2291 cs=0 l=1 rev1=1 crypto rx=0x7fdf4c009950 tx=0x7fdf4c0038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:08.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.516+0000 7fdf527fc640 1 -- 172.21.15.57:0/3204344563 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf4c003ea0 con 0x7fdf540ceb40 2023-10-19T14:20:08.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.516+0000 7fdf527fc640 1 -- 172.21.15.57:0/3204344563 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fdf4c005c90 con 0x7fdf540ceb40 2023-10-19T14:20:08.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.516+0000 7fdf5bba9640 1 -- 172.21.15.57:0/3204344563 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf5415d4e0 con 0x7fdf540ceb40 2023-10-19T14:20:08.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.517+0000 7fdf527fc640 1 -- 172.21.15.57:0/3204344563 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf4c005280 con 0x7fdf540ceb40 2023-10-19T14:20:08.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.517+0000 7fdf5bba9640 1 -- 172.21.15.57:0/3204344563 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf540b68c0 con 0x7fdf540ceb40 2023-10-19T14:20:08.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.517+0000 7fdf5bba9640 1 -- 172.21.15.57:0/3204344563 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf5415da00 con 0x7fdf540ceb40 2023-10-19T14:20:08.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.518+0000 7fdf527fc640 1 -- 172.21.15.57:0/3204344563 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 126) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fdf4c0053e0 con 0x7fdf540ceb40 2023-10-19T14:20:08.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.518+0000 7fdf527fc640 1 --2- 172.21.15.57:0/3204344563 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fdf4006acc0 0x7fdf4006d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:08.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.518+0000 7fdf527fc640 1 -- 172.21.15.57:0/3204344563 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 7032+0+0 (secure 0 0 0) 0x7fdf4c0ac380 con 0x7fdf540ceb40 2023-10-19T14:20:08.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.519+0000 7fdf59144640 1 --2- 172.21.15.57:0/3204344563 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fdf4006acc0 0x7fdf4006d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:08.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.519+0000 7fdf59144640 1 --2- 172.21.15.57:0/3204344563 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fdf4006acc0 0x7fdf4006d180 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fdf540b49f0 tx=0x7fdf48005f90 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:08.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.523+0000 7fdf527fc640 1 -- 172.21.15.57:0/3204344563 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fdf5415da00 con 0x7fdf540ceb40 2023-10-19T14:20:08.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.736+0000 7fdf5bba9640 1 -- 172.21.15.57:0/3204344563 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fdf5415da00 con 0x7fdf540ceb40 2023-10-19T14:20:08.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:08.988+0000 7fdf527fc640 1 -- 172.21.15.57:0/3204344563 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mgrmap(e 127) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fdf4c077540 con 0x7fdf540ceb40 2023-10-19T14:20:09.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:09.362+0000 7fdf527fc640 1 -- 172.21.15.57:0/3204344563 <== mon.0 v2:172.21.15.57:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v85) v1 ==== 104+0+0 (secure 0 0 0) 0x7fdf4c07ea10 con 0x7fdf540ceb40 2023-10-19T14:20:09.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:09.364+0000 7fdf5bba9640 1 -- 172.21.15.57:0/3204344563 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fdf4006acc0 msgr2=0x7fdf4006d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:09.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:09.364+0000 7fdf5bba9640 1 --2- 172.21.15.57:0/3204344563 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fdf4006acc0 0x7fdf4006d180 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fdf540b49f0 tx=0x7fdf48005f90 comp rx=0 tx=0).stop 2023-10-19T14:20:09.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:09.364+0000 7fdf5bba9640 1 -- 172.21.15.57:0/3204344563 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdf540ceb40 msgr2=0x7fdf540b6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:09.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:09.364+0000 7fdf5bba9640 1 --2- 172.21.15.57:0/3204344563 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdf540ceb40 0x7fdf540b6380 secure :-1 s=READY pgs=2291 cs=0 l=1 rev1=1 crypto rx=0x7fdf4c009950 tx=0x7fdf4c0038a0 comp rx=0 tx=0).stop 2023-10-19T14:20:09.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:09.365+0000 7fdf5bba9640 1 -- 172.21.15.57:0/3204344563 shutdown_connections 2023-10-19T14:20:09.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:09.365+0000 7fdf5bba9640 1 --2- 172.21.15.57:0/3204344563 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fdf4006acc0 0x7fdf4006d180 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:09.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:09.365+0000 7fdf5bba9640 1 --2- 172.21.15.57:0/3204344563 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdf5415b2a0 0x7fdf541591d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:09.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:09.365+0000 7fdf5bba9640 1 --2- 172.21.15.57:0/3204344563 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdf540cf340 0x7fdf54158270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:09.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:09.365+0000 7fdf5bba9640 1 --2- 172.21.15.57:0/3204344563 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdf540ceb40 0x7fdf540b6380 unknown :-1 s=CLOSED pgs=2291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:09.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:09.365+0000 7fdf5bba9640 1 -- 172.21.15.57:0/3204344563 >> 172.21.15.57:0/3204344563 conn(0x7fdf540d0d60 msgr2=0x7fdf5415a2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:09.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:09.365+0000 7fdf5bba9640 1 -- 172.21.15.57:0/3204344563 shutdown_connections 2023-10-19T14:20:09.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:09.365+0000 7fdf5bba9640 1 -- 172.21.15.57:0/3204344563 wait complete. 2023-10-19T14:20:09.380 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-19T14:20:09.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:09.497+0000 7f30e5955640 1 -- 172.21.15.57:0/1601399298 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f30e0142850 msgr2=0x7f30e0159440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:09.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:09.497+0000 7f30e5955640 1 --2- 172.21.15.57:0/1601399298 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f30e0142850 0x7f30e0159440 secure :-1 s=READY pgs=2292 cs=0 l=1 rev1=1 crypto rx=0x7f30e0072c60 tx=0x7f30d002c1d0 comp rx=0 tx=0).stop 2023-10-19T14:20:09.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:09.497+0000 7f30e5955640 1 -- 172.21.15.57:0/1601399298 shutdown_connections 2023-10-19T14:20:09.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:09.497+0000 7f30e5955640 1 --2- 172.21.15.57:0/1601399298 >> v2:172.21.15.179:3300/0 conn(0x7f30e00cd6d0 0x7f30e00cdb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:09.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:09.497+0000 7f30e5955640 1 --2- 172.21.15.57:0/1601399298 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f30e0142850 0x7f30e0159440 unknown :-1 s=CLOSED pgs=2292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:09.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:09.497+0000 7f30e5955640 1 -- 172.21.15.57:0/1601399298 >> 172.21.15.57:0/1601399298 conn(0x7f30e00af7a0 msgr2=0x7f30e00afb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:09.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:09.497+0000 7f30e5955640 1 -- 172.21.15.57:0/1601399298 shutdown_connections 2023-10-19T14:20:09.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:09.497+0000 7f30e5955640 1 -- 172.21.15.57:0/1601399298 wait complete. 2023-10-19T14:20:09.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:09.498+0000 7f30e5955640 1 Processor -- start 2023-10-19T14:20:09.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:09.498+0000 7f30e5955640 1 -- start start 2023-10-19T14:20:09.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:09.499+0000 7f30e5955640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f30e0069720 0x7f30e0158640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:09.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:09.499+0000 7f30e5955640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f30e00cd6d0 0x7f30e00a6a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:09.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:09.499+0000 7f30e5955640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f30e0142850 0x7f30e00ac6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:09.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:09.499+0000 7f30e5955640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30e01e26f0 con 0x7f30e0142850 2023-10-19T14:20:09.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:09.499+0000 7f30e5955640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30e01e29d0 con 0x7f30e0069720 2023-10-19T14:20:09.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:09.499+0000 7f30e5955640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30e01e2cb0 con 0x7f30e00cd6d0 2023-10-19T14:20:09.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:09.499+0000 7f30df7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f30e0142850 0x7f30e00ac6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:09.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:09.499+0000 7f30df7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f30e0142850 0x7f30e00ac6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35568/0 (socket says 172.21.15.57:35568) 2023-10-19T14:20:09.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:09.499+0000 7f30df7fe640 1 -- 172.21.15.57:0/1610963676 learned_addr learned my addr 172.21.15.57:0/1610963676 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:09.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:09.499+0000 7f30df7fe640 1 -- 172.21.15.57:0/1610963676 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f30e00cd6d0 msgr2=0x7f30e00a6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:09.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:09.499+0000 7f30deffd640 1 --2- 172.21.15.57:0/1610963676 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f30e0069720 0x7f30e0158640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:09.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:09.499+0000 7f30de7fc640 1 --2- 172.21.15.57:0/1610963676 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f30e00cd6d0 0x7f30e00a6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:09.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:09.499+0000 7f30df7fe640 1 --2- 172.21.15.57:0/1610963676 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f30e00cd6d0 0x7f30e00a6a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:09.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:09.499+0000 7f30df7fe640 1 -- 172.21.15.57:0/1610963676 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f30e0069720 msgr2=0x7f30e0158640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:09.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:09.499+0000 7f30df7fe640 1 --2- 172.21.15.57:0/1610963676 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f30e0069720 0x7f30e0158640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:09.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:09.499+0000 7f30df7fe640 1 -- 172.21.15.57:0/1610963676 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30d0009900 con 0x7f30e0142850 2023-10-19T14:20:09.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:09.499+0000 7f30deffd640 1 --2- 172.21.15.57:0/1610963676 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f30e0069720 0x7f30e0158640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:20:09.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:09.499+0000 7f30de7fc640 1 --2- 172.21.15.57:0/1610963676 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f30e00cd6d0 0x7f30e00a6a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:20:09.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:09.500+0000 7f30df7fe640 1 --2- 172.21.15.57:0/1610963676 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f30e0142850 0x7f30e00ac6a0 secure :-1 s=READY pgs=2293 cs=0 l=1 rev1=1 crypto rx=0x7f30d800c390 tx=0x7f30d800c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:09.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:09.500+0000 7f30d7fff640 1 -- 172.21.15.57:0/1610963676 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30d80035f0 con 0x7f30e0142850 2023-10-19T14:20:09.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:09.500+0000 7f30d7fff640 1 -- 172.21.15.57:0/1610963676 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f30d8003750 con 0x7f30e0142850 2023-10-19T14:20:09.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:09.500+0000 7f30d7fff640 1 -- 172.21.15.57:0/1610963676 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30d8003b60 con 0x7f30e0142850 2023-10-19T14:20:09.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:09.500+0000 7f30e5955640 1 -- 172.21.15.57:0/1610963676 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30e01e2f90 con 0x7f30e0142850 2023-10-19T14:20:09.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:09.500+0000 7f30e5955640 1 -- 172.21.15.57:0/1610963676 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30e01e3550 con 0x7f30e0142850 2023-10-19T14:20:09.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:09.501+0000 7f30e5955640 1 -- 172.21.15.57:0/1610963676 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30e00aee30 con 0x7f30e0142850 2023-10-19T14:20:09.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:09.501+0000 7f30d7fff640 1 -- 172.21.15.57:0/1610963676 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 127) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f30d8003cc0 con 0x7f30e0142850 2023-10-19T14:20:09.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:09.502+0000 7f30d7fff640 1 --2- 172.21.15.57:0/1610963676 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f30c006aed0 0x7f30c006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:09.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:09.502+0000 7f30d7fff640 1 -- 172.21.15.57:0/1610963676 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 7032+0+0 (secure 0 0 0) 0x7f30d8088e20 con 0x7f30e0142850 2023-10-19T14:20:09.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:09.502+0000 7f30deffd640 1 --2- 172.21.15.57:0/1610963676 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f30c006aed0 0x7f30c006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:09.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:09.503+0000 7f30deffd640 1 --2- 172.21.15.57:0/1610963676 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f30c006aed0 0x7f30c006d390 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f30d0036000 tx=0x7f30d0004ab0 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:09.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:09.506+0000 7f30d7fff640 1 -- 172.21.15.57:0/1610963676 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f30d8053a40 con 0x7f30e0142850 2023-10-19T14:20:09.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:09.687+0000 7f30e5955640 1 -- 172.21.15.57:0/1610963676 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f30e00690e0 con 0x7f30e0142850 2023-10-19T14:20:10.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:10.622+0000 7f30d7fff640 1 -- 172.21.15.57:0/1610963676 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v214) v1 ==== 181+0+0 (secure 0 0 0) 0x7f30d8057060 con 0x7f30e0142850 2023-10-19T14:20:10.625 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'cephfs_metadata' removed 2023-10-19T14:20:10.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:10.629+0000 7f30e5955640 1 -- 172.21.15.57:0/1610963676 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f30c006aed0 msgr2=0x7f30c006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:10.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:10.629+0000 7f30e5955640 1 --2- 172.21.15.57:0/1610963676 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f30c006aed0 0x7f30c006d390 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f30d0036000 tx=0x7f30d0004ab0 comp rx=0 tx=0).stop 2023-10-19T14:20:10.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:10.629+0000 7f30e5955640 1 -- 172.21.15.57:0/1610963676 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f30e0142850 msgr2=0x7f30e00ac6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:10.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:10.630+0000 7f30d7fff640 1 -- 172.21.15.57:0/1610963676 <== mon.0 v2:172.21.15.57:3300/0 8 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f30d8053e10 con 0x7f30e0142850 2023-10-19T14:20:10.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:10.630+0000 7f30e5955640 1 --2- 172.21.15.57:0/1610963676 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f30e0142850 0x7f30e00ac6a0 secure :-1 s=READY pgs=2293 cs=0 l=1 rev1=1 crypto rx=0x7f30d800c390 tx=0x7f30d800c850 comp rx=0 tx=0).stop 2023-10-19T14:20:10.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:10.630+0000 7f30d7fff640 1 --2- 172.21.15.57:0/1610963676 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f30c006aed0 0x7f30c006d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:10.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:10.631+0000 7f30e5955640 1 -- 172.21.15.57:0/1610963676 shutdown_connections 2023-10-19T14:20:10.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:10.631+0000 7f30e5955640 1 --2- 172.21.15.57:0/1610963676 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f30c006aed0 0x7f30c006d2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:10.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:10.631+0000 7f30e5955640 1 --2- 172.21.15.57:0/1610963676 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f30e0142850 0x7f30e00ac6a0 unknown :-1 s=CLOSED pgs=2293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:10.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:10.631+0000 7f30e5955640 1 --2- 172.21.15.57:0/1610963676 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f30e00cd6d0 0x7f30e00a6a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:10.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:10.631+0000 7f30e5955640 1 --2- 172.21.15.57:0/1610963676 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f30e0069720 0x7f30e0158640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:10.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:10.631+0000 7f30e5955640 1 -- 172.21.15.57:0/1610963676 >> 172.21.15.57:0/1610963676 conn(0x7f30e00af7a0 msgr2=0x7f30e00cc8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:10.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:10.631+0000 7f30d7fff640 1 -- 172.21.15.57:0/1610963676 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f30c006aed0 msgr2=0x7f30c006d2f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-10-19T14:20:10.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:10.631+0000 7f30d7fff640 1 --2- 172.21.15.57:0/1610963676 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f30c006aed0 0x7f30c006d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:10.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:10.631+0000 7f30e5955640 1 -- 172.21.15.57:0/1610963676 shutdown_connections 2023-10-19T14:20:10.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:10.631+0000 7f30e5955640 1 -- 172.21.15.57:0/1610963676 wait complete. 2023-10-19T14:20:10.649 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-19T14:20:10.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:10.765+0000 7f05a9f5e640 1 -- 172.21.15.57:0/735228337 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f05a415b2a0 msgr2=0x7f05a40cf720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:10.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:10.765+0000 7f05a9f5e640 1 --2- 172.21.15.57:0/735228337 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f05a415b2a0 0x7f05a40cf720 secure :-1 s=READY pgs=2294 cs=0 l=1 rev1=1 crypto rx=0x7f0598005fd0 tx=0x7f059802bcf0 comp rx=0 tx=0).stop 2023-10-19T14:20:10.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:10.765+0000 7f05a9f5e640 1 -- 172.21.15.57:0/735228337 shutdown_connections 2023-10-19T14:20:10.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:10.765+0000 7f05a9f5e640 1 --2- 172.21.15.57:0/735228337 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f05a415b2a0 0x7f05a40cf720 unknown :-1 s=CLOSED pgs=2294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:10.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:10.765+0000 7f05a9f5e640 1 --2- 172.21.15.57:0/735228337 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f05a40cf340 0x7f05a40d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:10.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:10.765+0000 7f05a9f5e640 1 -- 172.21.15.57:0/735228337 >> 172.21.15.57:0/735228337 conn(0x7f05a40d0d60 msgr2=0x7f05a40d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:10.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:10.765+0000 7f05a9f5e640 1 -- 172.21.15.57:0/735228337 shutdown_connections 2023-10-19T14:20:10.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:10.765+0000 7f05a9f5e640 1 -- 172.21.15.57:0/735228337 wait complete. 2023-10-19T14:20:10.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:10.766+0000 7f05a9f5e640 1 Processor -- start 2023-10-19T14:20:10.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:10.767+0000 7f05a9f5e640 1 -- start start 2023-10-19T14:20:10.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:10.767+0000 7f05a9f5e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f05a40ceb40 0x7f05a4157fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:10.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:10.767+0000 7f05a9f5e640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f05a40cf340 0x7f05a40b3980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:10.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:10.767+0000 7f05a9f5e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f05a415b2a0 0x7f05a40c9fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:10.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:10.767+0000 7f05a9f5e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05a40b4d70 con 0x7f05a40ceb40 2023-10-19T14:20:10.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:10.767+0000 7f05a9f5e640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05a40b3ec0 con 0x7f05a415b2a0 2023-10-19T14:20:10.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:10.767+0000 7f05a9f5e640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05a40ca4f0 con 0x7f05a40cf340 2023-10-19T14:20:10.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:10.767+0000 7f05a37fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f05a40ceb40 0x7f05a4157fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:10.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:10.768+0000 7f05a37fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f05a40ceb40 0x7f05a4157fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35578/0 (socket says 172.21.15.57:35578) 2023-10-19T14:20:10.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:10.768+0000 7f05a37fe640 1 -- 172.21.15.57:0/3624473214 learned_addr learned my addr 172.21.15.57:0/3624473214 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:10.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:10.768+0000 7f05a2ffd640 1 --2- 172.21.15.57:0/3624473214 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f05a40cf340 0x7f05a40b3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:10.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:10.768+0000 7f05a3fff640 1 --2- 172.21.15.57:0/3624473214 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f05a415b2a0 0x7f05a40c9fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:10.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:10.768+0000 7f05a37fe640 1 -- 172.21.15.57:0/3624473214 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f05a40cf340 msgr2=0x7f05a40b3980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:10.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:10.768+0000 7f05a37fe640 1 --2- 172.21.15.57:0/3624473214 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f05a40cf340 0x7f05a40b3980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:10.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:10.768+0000 7f05a37fe640 1 -- 172.21.15.57:0/3624473214 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f05a415b2a0 msgr2=0x7f05a40c9fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:10.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:10.768+0000 7f05a37fe640 1 --2- 172.21.15.57:0/3624473214 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f05a415b2a0 0x7f05a40c9fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:10.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:10.768+0000 7f05a37fe640 1 -- 172.21.15.57:0/3624473214 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0598009560 con 0x7f05a40ceb40 2023-10-19T14:20:10.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:10.768+0000 7f05a37fe640 1 --2- 172.21.15.57:0/3624473214 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f05a40ceb40 0x7f05a4157fb0 secure :-1 s=READY pgs=2295 cs=0 l=1 rev1=1 crypto rx=0x7f059400c970 tx=0x7f059400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:10.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:10.768+0000 7f058bfff640 1 -- 172.21.15.57:0/3624473214 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0594007bb0 con 0x7f05a40ceb40 2023-10-19T14:20:10.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:10.768+0000 7f058bfff640 1 -- 172.21.15.57:0/3624473214 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f0594007e20 con 0x7f05a40ceb40 2023-10-19T14:20:10.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:10.768+0000 7f05a9f5e640 1 -- 172.21.15.57:0/3624473214 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05a40b40f0 con 0x7f05a40ceb40 2023-10-19T14:20:10.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:10.768+0000 7f058bfff640 1 -- 172.21.15.57:0/3624473214 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0594005280 con 0x7f05a40ceb40 2023-10-19T14:20:10.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:10.768+0000 7f05a9f5e640 1 -- 172.21.15.57:0/3624473214 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05a40b46b0 con 0x7f05a40ceb40 2023-10-19T14:20:10.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:10.769+0000 7f05a9f5e640 1 -- 172.21.15.57:0/3624473214 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05a40690e0 con 0x7f05a40ceb40 2023-10-19T14:20:10.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:10.770+0000 7f058bfff640 1 -- 172.21.15.57:0/3624473214 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f05940053e0 con 0x7f05a40ceb40 2023-10-19T14:20:10.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:10.770+0000 7f058bfff640 1 --2- 172.21.15.57:0/3624473214 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f059006af60 0x7f059006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:10.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:10.770+0000 7f058bfff640 1 -- 172.21.15.57:0/3624473214 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6566+0+0 (secure 0 0 0) 0x7f059408a150 con 0x7f05a40ceb40 2023-10-19T14:20:10.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:10.770+0000 7f05a2ffd640 1 --2- 172.21.15.57:0/3624473214 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f059006af60 0x7f059006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:10.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:10.771+0000 7f05a2ffd640 1 --2- 172.21.15.57:0/3624473214 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f059006af60 0x7f059006d420 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0598035000 tx=0x7f0598004730 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:10.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:10.774+0000 7f058bfff640 1 -- 172.21.15.57:0/3624473214 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0594054d30 con 0x7f05a40ceb40 2023-10-19T14:20:10.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:10.951+0000 7f05a9f5e640 1 -- 172.21.15.57:0/3624473214 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f05a40d0b40 con 0x7f05a40ceb40 2023-10-19T14:20:11.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:11.630+0000 7f058bfff640 1 -- 172.21.15.57:0/3624473214 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v215) v1 ==== 169+0+0 (secure 0 0 0) 0x7f059401d0d0 con 0x7f05a40ceb40 2023-10-19T14:20:11.633 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'cephfs_data' removed 2023-10-19T14:20:11.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:11.634+0000 7f05a9f5e640 1 -- 172.21.15.57:0/3624473214 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f059006af60 msgr2=0x7f059006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:11.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:11.634+0000 7f05a9f5e640 1 --2- 172.21.15.57:0/3624473214 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f059006af60 0x7f059006d420 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0598035000 tx=0x7f0598004730 comp rx=0 tx=0).stop 2023-10-19T14:20:11.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:11.634+0000 7f05a9f5e640 1 -- 172.21.15.57:0/3624473214 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f05a40ceb40 msgr2=0x7f05a4157fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:11.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:11.634+0000 7f05a9f5e640 1 --2- 172.21.15.57:0/3624473214 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f05a40ceb40 0x7f05a4157fb0 secure :-1 s=READY pgs=2295 cs=0 l=1 rev1=1 crypto rx=0x7f059400c970 tx=0x7f059400ce30 comp rx=0 tx=0).stop 2023-10-19T14:20:11.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:11.635+0000 7f05a9f5e640 1 -- 172.21.15.57:0/3624473214 shutdown_connections 2023-10-19T14:20:11.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:11.635+0000 7f05a9f5e640 1 --2- 172.21.15.57:0/3624473214 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f059006af60 0x7f059006d420 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:11.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:11.635+0000 7f05a9f5e640 1 --2- 172.21.15.57:0/3624473214 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f05a415b2a0 0x7f05a40c9fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:11.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:11.635+0000 7f05a9f5e640 1 --2- 172.21.15.57:0/3624473214 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f05a40cf340 0x7f05a40b3980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:11.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:11.635+0000 7f05a9f5e640 1 --2- 172.21.15.57:0/3624473214 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f05a40ceb40 0x7f05a4157fb0 unknown :-1 s=CLOSED pgs=2295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:11.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:11.635+0000 7f05a9f5e640 1 -- 172.21.15.57:0/3624473214 >> 172.21.15.57:0/3624473214 conn(0x7f05a40d0d60 msgr2=0x7f05a4159750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:11.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:11.637+0000 7f05a9f5e640 1 -- 172.21.15.57:0/3624473214 shutdown_connections 2023-10-19T14:20:11.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:11.638+0000 7f05a9f5e640 1 -- 172.21.15.57:0/3624473214 wait complete. 2023-10-19T14:20:11.656 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 30 2023-10-19T14:20:11.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:11.769+0000 7fe36504f640 1 -- 172.21.15.57:0/3331074225 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3600c9dc0 msgr2=0x7fe36006b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:11.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:11.769+0000 7fe36504f640 1 --2- 172.21.15.57:0/3331074225 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3600c9dc0 0x7fe36006b2e0 secure :-1 s=READY pgs=2297 cs=0 l=1 rev1=1 crypto rx=0x7fe354009830 tx=0x7fe35402bdf0 comp rx=0 tx=0).stop 2023-10-19T14:20:11.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:11.769+0000 7fe36504f640 1 -- 172.21.15.57:0/3331074225 shutdown_connections 2023-10-19T14:20:11.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:11.769+0000 7fe36504f640 1 --2- 172.21.15.57:0/3331074225 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3600c9dc0 0x7fe36006b2e0 unknown :-1 s=CLOSED pgs=2297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:11.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:11.769+0000 7fe36504f640 1 -- 172.21.15.57:0/3331074225 >> 172.21.15.57:0/3331074225 conn(0x7fe360069930 msgr2=0x7fe360069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:11.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:11.769+0000 7fe36504f640 1 -- 172.21.15.57:0/3331074225 shutdown_connections 2023-10-19T14:20:11.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:11.769+0000 7fe36504f640 1 -- 172.21.15.57:0/3331074225 wait complete. 2023-10-19T14:20:11.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:11.770+0000 7fe36504f640 1 Processor -- start 2023-10-19T14:20:11.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:11.770+0000 7fe36504f640 1 -- start start 2023-10-19T14:20:11.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:11.771+0000 7fe36504f640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe36006af00 0x7fe36013a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:11.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:11.771+0000 7fe36504f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3600ab500 0x7fe3600aecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:11.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:11.771+0000 7fe36504f640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe3600c9dc0 0x7fe360151f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:11.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:11.771+0000 7fe36504f640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe360151a80 con 0x7fe3600ab500 2023-10-19T14:20:11.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:11.771+0000 7fe36504f640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3600af230 con 0x7fe36006af00 2023-10-19T14:20:11.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:11.771+0000 7fe36504f640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe360150e70 con 0x7fe3600c9dc0 2023-10-19T14:20:11.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:11.771+0000 7fe35e59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3600ab500 0x7fe3600aecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:11.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:11.771+0000 7fe35e59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3600ab500 0x7fe3600aecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35604/0 (socket says 172.21.15.57:35604) 2023-10-19T14:20:11.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:11.771+0000 7fe35e59c640 1 -- 172.21.15.57:0/1080321017 learned_addr learned my addr 172.21.15.57:0/1080321017 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:11.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:11.772+0000 7fe35ed9d640 1 --2- 172.21.15.57:0/1080321017 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe36006af00 0x7fe36013a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:11.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:11.772+0000 7fe35e59c640 1 -- 172.21.15.57:0/1080321017 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe3600c9dc0 msgr2=0x7fe360151f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:11.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:11.772+0000 7fe35f59e640 1 --2- 172.21.15.57:0/1080321017 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe3600c9dc0 0x7fe360151f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:11.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:11.772+0000 7fe35e59c640 1 --2- 172.21.15.57:0/1080321017 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe3600c9dc0 0x7fe360151f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:11.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:11.772+0000 7fe35e59c640 1 -- 172.21.15.57:0/1080321017 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe36006af00 msgr2=0x7fe36013a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:11.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:11.772+0000 7fe35e59c640 1 --2- 172.21.15.57:0/1080321017 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe36006af00 0x7fe36013a5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:11.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:11.772+0000 7fe35e59c640 1 -- 172.21.15.57:0/1080321017 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe354009490 con 0x7fe3600ab500 2023-10-19T14:20:11.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:11.772+0000 7fe35e59c640 1 --2- 172.21.15.57:0/1080321017 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3600ab500 0x7fe3600aecf0 secure :-1 s=READY pgs=2298 cs=0 l=1 rev1=1 crypto rx=0x7fe354009780 tx=0x7fe3540038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:11.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:11.772+0000 7fe3477fe640 1 -- 172.21.15.57:0/1080321017 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe354003ea0 con 0x7fe3600ab500 2023-10-19T14:20:11.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:11.772+0000 7fe3477fe640 1 -- 172.21.15.57:0/1080321017 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fe354034550 con 0x7fe3600ab500 2023-10-19T14:20:11.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:11.772+0000 7fe36504f640 1 -- 172.21.15.57:0/1080321017 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe360151150 con 0x7fe3600ab500 2023-10-19T14:20:11.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:11.772+0000 7fe3477fe640 1 -- 172.21.15.57:0/1080321017 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe354034960 con 0x7fe3600ab500 2023-10-19T14:20:11.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:11.772+0000 7fe36504f640 1 -- 172.21.15.57:0/1080321017 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe360151710 con 0x7fe3600ab500 2023-10-19T14:20:11.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:11.773+0000 7fe36504f640 1 -- 172.21.15.57:0/1080321017 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3600690e0 con 0x7fe3600ab500 2023-10-19T14:20:11.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:11.774+0000 7fe3477fe640 1 -- 172.21.15.57:0/1080321017 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe35403b040 con 0x7fe3600ab500 2023-10-19T14:20:11.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:11.774+0000 7fe3477fe640 1 --2- 172.21.15.57:0/1080321017 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fe34c06af90 0x7fe34c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:11.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:11.774+0000 7fe3477fe640 1 -- 172.21.15.57:0/1080321017 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 6065+0+0 (secure 0 0 0) 0x7fe3540abff0 con 0x7fe3600ab500 2023-10-19T14:20:11.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:11.774+0000 7fe35ed9d640 1 --2- 172.21.15.57:0/1080321017 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fe34c06af90 0x7fe34c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:11.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:11.775+0000 7fe35ed9d640 1 --2- 172.21.15.57:0/1080321017 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fe34c06af90 0x7fe34c06d450 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe358000f60 tx=0x7fe358009040 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:11.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:11.778+0000 7fe3477fe640 1 -- 172.21.15.57:0/1080321017 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe354076da0 con 0x7fe3600ab500 2023-10-19T14:20:11.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:11.956+0000 7fe36504f640 1 -- 172.21.15.57:0/1080321017 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "config reset", "num": 30} v 0) v1 -- 0x7fe3600cc420 con 0x7fe3600ab500 2023-10-19T14:20:11.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:11.960+0000 7fe3477fe640 1 -- 172.21.15.57:0/1080321017 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 30}]=0 v74) v1 ==== 71+0+0 (secure 0 0 0) 0x7fe35407a3c0 con 0x7fe3600ab500 2023-10-19T14:20:11.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:11.962+0000 7fe36504f640 1 -- 172.21.15.57:0/1080321017 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fe34c06af90 msgr2=0x7fe34c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:11.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:11.962+0000 7fe36504f640 1 --2- 172.21.15.57:0/1080321017 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fe34c06af90 0x7fe34c06d450 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe358000f60 tx=0x7fe358009040 comp rx=0 tx=0).stop 2023-10-19T14:20:11.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:11.962+0000 7fe36504f640 1 -- 172.21.15.57:0/1080321017 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3600ab500 msgr2=0x7fe3600aecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:11.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:11.962+0000 7fe36504f640 1 --2- 172.21.15.57:0/1080321017 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3600ab500 0x7fe3600aecf0 secure :-1 s=READY pgs=2298 cs=0 l=1 rev1=1 crypto rx=0x7fe354009780 tx=0x7fe3540038a0 comp rx=0 tx=0).stop 2023-10-19T14:20:11.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:11.962+0000 7fe36504f640 1 -- 172.21.15.57:0/1080321017 shutdown_connections 2023-10-19T14:20:11.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:11.962+0000 7fe36504f640 1 --2- 172.21.15.57:0/1080321017 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fe34c06af90 0x7fe34c06d450 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:11.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:11.962+0000 7fe36504f640 1 --2- 172.21.15.57:0/1080321017 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe3600c9dc0 0x7fe360151f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:11.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:11.962+0000 7fe36504f640 1 --2- 172.21.15.57:0/1080321017 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3600ab500 0x7fe3600aecf0 unknown :-1 s=CLOSED pgs=2298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:11.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:11.962+0000 7fe36504f640 1 --2- 172.21.15.57:0/1080321017 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe36006af00 0x7fe36013a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:11.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:11.962+0000 7fe36504f640 1 -- 172.21.15.57:0/1080321017 >> 172.21.15.57:0/1080321017 conn(0x7fe360069930 msgr2=0x7fe36014fb20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:11.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:11.962+0000 7fe36504f640 1 -- 172.21.15.57:0/1080321017 shutdown_connections 2023-10-19T14:20:11.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:11.962+0000 7fe36504f640 1 -- 172.21.15.57:0/1080321017 wait complete. 2023-10-19T14:20:11.975 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention' 2023-10-19T14:20:12.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.082+0000 7f053618a640 1 -- 172.21.15.57:0/3575679922 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f053006af00 msgr2=0x7f053006b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:12.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.082+0000 7f053618a640 1 --2- 172.21.15.57:0/3575679922 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f053006af00 0x7f053006b2e0 secure :-1 s=READY pgs=2299 cs=0 l=1 rev1=1 crypto rx=0x7f0520009900 tx=0x7f052002bd70 comp rx=0 tx=0).stop 2023-10-19T14:20:12.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.082+0000 7f053618a640 1 -- 172.21.15.57:0/3575679922 shutdown_connections 2023-10-19T14:20:12.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.082+0000 7f053618a640 1 --2- 172.21.15.57:0/3575679922 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f05300abc10 0x7f05300c9b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:12.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.082+0000 7f053618a640 1 --2- 172.21.15.57:0/3575679922 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f053006af00 0x7f053006b2e0 unknown :-1 s=CLOSED pgs=2299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:12.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.082+0000 7f053618a640 1 -- 172.21.15.57:0/3575679922 >> 172.21.15.57:0/3575679922 conn(0x7f0530069930 msgr2=0x7f0530069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:12.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.082+0000 7f053618a640 1 -- 172.21.15.57:0/3575679922 shutdown_connections 2023-10-19T14:20:12.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.082+0000 7f053618a640 1 -- 172.21.15.57:0/3575679922 wait complete. 2023-10-19T14:20:12.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.083+0000 7f053618a640 1 Processor -- start 2023-10-19T14:20:12.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.083+0000 7f053618a640 1 -- start start 2023-10-19T14:20:12.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.084+0000 7f053618a640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f053006af00 0x7f05301eedf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:12.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.084+0000 7f053618a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f05300abc10 0x7f05301353e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:12.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.084+0000 7f053618a640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f05300c96b0 0x7f05301345b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:12.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.084+0000 7f053618a640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0530136d60 con 0x7f05300abc10 2023-10-19T14:20:12.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.084+0000 7f053618a640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05300b0d40 con 0x7f05300c96b0 2023-10-19T14:20:12.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.084+0000 7f053618a640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05300b1020 con 0x7f053006af00 2023-10-19T14:20:12.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.084+0000 7f052effd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f05300abc10 0x7f05301353e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:12.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.084+0000 7f052effd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f05300abc10 0x7f05301353e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35624/0 (socket says 172.21.15.57:35624) 2023-10-19T14:20:12.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.084+0000 7f052effd640 1 -- 172.21.15.57:0/515147023 learned_addr learned my addr 172.21.15.57:0/515147023 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:12.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.084+0000 7f052f7fe640 1 --2- 172.21.15.57:0/515147023 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f053006af00 0x7f05301eedf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:12.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.084+0000 7f052ffff640 1 --2- 172.21.15.57:0/515147023 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f05300c96b0 0x7f05301345b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:12.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.084+0000 7f052effd640 1 -- 172.21.15.57:0/515147023 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f053006af00 msgr2=0x7f05301eedf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:12.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.084+0000 7f052effd640 1 --2- 172.21.15.57:0/515147023 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f053006af00 0x7f05301eedf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:12.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.084+0000 7f052effd640 1 -- 172.21.15.57:0/515147023 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f05300c96b0 msgr2=0x7f05301345b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:12.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.084+0000 7f052effd640 1 --2- 172.21.15.57:0/515147023 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f05300c96b0 0x7f05301345b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:12.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.084+0000 7f052effd640 1 -- 172.21.15.57:0/515147023 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0520009560 con 0x7f05300abc10 2023-10-19T14:20:12.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.084+0000 7f052effd640 1 --2- 172.21.15.57:0/515147023 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f05300abc10 0x7f05301353e0 secure :-1 s=READY pgs=2300 cs=0 l=1 rev1=1 crypto rx=0x7f052400b3f0 tx=0x7f052400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:12.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.084+0000 7f0517fff640 1 -- 172.21.15.57:0/515147023 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0524011830 con 0x7f05300abc10 2023-10-19T14:20:12.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.084+0000 7f0517fff640 1 -- 172.21.15.57:0/515147023 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f0524011a80 con 0x7f05300abc10 2023-10-19T14:20:12.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.084+0000 7f053618a640 1 -- 172.21.15.57:0/515147023 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0530136710 con 0x7f05300abc10 2023-10-19T14:20:12.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.084+0000 7f0517fff640 1 -- 172.21.15.57:0/515147023 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0524011e90 con 0x7f05300abc10 2023-10-19T14:20:12.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.084+0000 7f053618a640 1 -- 172.21.15.57:0/515147023 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0530135920 con 0x7f05300abc10 2023-10-19T14:20:12.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.086+0000 7f053618a640 1 -- 172.21.15.57:0/515147023 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f053013baf0 con 0x7f05300abc10 2023-10-19T14:20:12.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.086+0000 7f0517fff640 1 -- 172.21.15.57:0/515147023 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f052401a400 con 0x7f05300abc10 2023-10-19T14:20:12.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.086+0000 7f0517fff640 1 --2- 172.21.15.57:0/515147023 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f051c06af90 0x7f051c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:12.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.086+0000 7f0517fff640 1 -- 172.21.15.57:0/515147023 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 6065+0+0 (secure 0 0 0) 0x7f0524089930 con 0x7f05300abc10 2023-10-19T14:20:12.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.087+0000 7f052f7fe640 1 --2- 172.21.15.57:0/515147023 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f051c06af90 0x7f051c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:12.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.087+0000 7f052f7fe640 1 --2- 172.21.15.57:0/515147023 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f051c06af90 0x7f051c06d450 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0520035000 tx=0x7f0520004730 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:12.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.091+0000 7f0517fff640 1 -- 172.21.15.57:0/515147023 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f05240547f0 con 0x7f05300abc10 2023-10-19T14:20:12.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.343+0000 7f053618a640 1 -- 172.21.15.57:0/515147023 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention"]} v 0) v1 -- 0x7f05300690e0 con 0x7f05300abc10 2023-10-19T14:20:12.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.632+0000 7f0517fff640 1 -- 172.21.15.57:0/515147023 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention"]}]=0 v3121) v1 ==== 163+0+0 (secure 0 0 0) 0x7f05240150e0 con 0x7f05300abc10 2023-10-19T14:20:12.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.634+0000 7f053618a640 1 -- 172.21.15.57:0/515147023 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f051c06af90 msgr2=0x7f051c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:12.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.634+0000 7f053618a640 1 --2- 172.21.15.57:0/515147023 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f051c06af90 0x7f051c06d450 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0520035000 tx=0x7f0520004730 comp rx=0 tx=0).stop 2023-10-19T14:20:12.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.634+0000 7f053618a640 1 -- 172.21.15.57:0/515147023 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f05300abc10 msgr2=0x7f05301353e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:12.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.634+0000 7f053618a640 1 --2- 172.21.15.57:0/515147023 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f05300abc10 0x7f05301353e0 secure :-1 s=READY pgs=2300 cs=0 l=1 rev1=1 crypto rx=0x7f052400b3f0 tx=0x7f052400b8b0 comp rx=0 tx=0).stop 2023-10-19T14:20:12.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.634+0000 7f053618a640 1 -- 172.21.15.57:0/515147023 shutdown_connections 2023-10-19T14:20:12.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.634+0000 7f053618a640 1 --2- 172.21.15.57:0/515147023 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f051c06af90 0x7f051c06d450 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:12.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.634+0000 7f053618a640 1 --2- 172.21.15.57:0/515147023 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f05300c96b0 0x7f05301345b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:12.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.634+0000 7f053618a640 1 --2- 172.21.15.57:0/515147023 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f05300abc10 0x7f05301353e0 unknown :-1 s=CLOSED pgs=2300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:12.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.634+0000 7f053618a640 1 --2- 172.21.15.57:0/515147023 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f053006af00 0x7f05301eedf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:12.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.634+0000 7f053618a640 1 -- 172.21.15.57:0/515147023 >> 172.21.15.57:0/515147023 conn(0x7f0530069930 msgr2=0x7f0530069d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:12.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.634+0000 7f053618a640 1 -- 172.21.15.57:0/515147023 shutdown_connections 2023-10-19T14:20:12.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.634+0000 7f053618a640 1 -- 172.21.15.57:0/515147023 wait complete. 2023-10-19T14:20:12.648 INFO:tasks.cephfs_test_runner:Test scheduled snapshots along with rentention policy ... ok 2023-10-19T14:20:12.648 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_for_non_default_fs (tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument) Test command succes with --fs argument for non-default fs 2023-10-19T14:20:12.649 INFO:tasks.cephfs_test_runner:test_snap_schedule_for_non_default_fs (tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument) 2023-10-19T14:20:12.649 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T14:20:12.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.756+0000 7fb4f3656640 1 -- 172.21.15.57:0/1085343169 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4ec0d1910 msgr2=0x7fb4ec0b0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:12.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.756+0000 7fb4f3656640 1 --2- 172.21.15.57:0/1085343169 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4ec0d1910 0x7fb4ec0b0650 secure :-1 s=READY pgs=2302 cs=0 l=1 rev1=1 crypto rx=0x7fb4e4005fd0 tx=0x7fb4e402c050 comp rx=0 tx=0).stop 2023-10-19T14:20:12.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.756+0000 7fb4f3656640 1 -- 172.21.15.57:0/1085343169 shutdown_connections 2023-10-19T14:20:12.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.756+0000 7fb4f3656640 1 --2- 172.21.15.57:0/1085343169 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4ec0d1910 0x7fb4ec0b0650 unknown :-1 s=CLOSED pgs=2302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:12.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.756+0000 7fb4f3656640 1 -- 172.21.15.57:0/1085343169 >> 172.21.15.57:0/1085343169 conn(0x7fb4ec0d0d60 msgr2=0x7fb4ec0d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:12.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.756+0000 7fb4f3656640 1 -- 172.21.15.57:0/1085343169 shutdown_connections 2023-10-19T14:20:12.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.756+0000 7fb4f3656640 1 -- 172.21.15.57:0/1085343169 wait complete. 2023-10-19T14:20:12.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.758+0000 7fb4f3656640 1 Processor -- start 2023-10-19T14:20:12.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.758+0000 7fb4f3656640 1 -- start start 2023-10-19T14:20:12.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.758+0000 7fb4f3656640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb4ec0cf340 0x7fb4ec158040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:12.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.758+0000 7fb4f3656640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4ec0d1910 0x7fb4ec0c97c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:12.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.758+0000 7fb4f3656640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb4ec15b2a0 0x7fb4ec0c9d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:12.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.758+0000 7fb4f3656640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4ec158580 con 0x7fb4ec0d1910 2023-10-19T14:20:12.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.758+0000 7fb4f3656640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4ec1e3980 con 0x7fb4ec15b2a0 2023-10-19T14:20:12.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.758+0000 7fb4f3656640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4ec1e3c60 con 0x7fb4ec0cf340 2023-10-19T14:20:12.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.758+0000 7fb4f0bf1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4ec0d1910 0x7fb4ec0c97c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:12.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.758+0000 7fb4f0bf1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4ec0d1910 0x7fb4ec0c97c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35630/0 (socket says 172.21.15.57:35630) 2023-10-19T14:20:12.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.758+0000 7fb4f0bf1640 1 -- 172.21.15.57:0/2847453564 learned_addr learned my addr 172.21.15.57:0/2847453564 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:12.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.759+0000 7fb4f1bf3640 1 --2- 172.21.15.57:0/2847453564 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb4ec15b2a0 0x7fb4ec0c9d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:12.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.759+0000 7fb4f13f2640 1 --2- 172.21.15.57:0/2847453564 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb4ec0cf340 0x7fb4ec158040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:12.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.759+0000 7fb4f0bf1640 1 -- 172.21.15.57:0/2847453564 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb4ec0cf340 msgr2=0x7fb4ec158040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:12.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.759+0000 7fb4f0bf1640 1 --2- 172.21.15.57:0/2847453564 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb4ec0cf340 0x7fb4ec158040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:12.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.759+0000 7fb4f0bf1640 1 -- 172.21.15.57:0/2847453564 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb4ec15b2a0 msgr2=0x7fb4ec0c9d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:12.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.759+0000 7fb4f0bf1640 1 --2- 172.21.15.57:0/2847453564 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb4ec15b2a0 0x7fb4ec0c9d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:12.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.759+0000 7fb4f0bf1640 1 -- 172.21.15.57:0/2847453564 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4e4009840 con 0x7fb4ec0d1910 2023-10-19T14:20:12.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.759+0000 7fb4f1bf3640 1 --2- 172.21.15.57:0/2847453564 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb4ec15b2a0 0x7fb4ec0c9d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:20:12.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.759+0000 7fb4f0bf1640 1 --2- 172.21.15.57:0/2847453564 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4ec0d1910 0x7fb4ec0c97c0 secure :-1 s=READY pgs=2303 cs=0 l=1 rev1=1 crypto rx=0x7fb4dc00b3e0 tx=0x7fb4dc00b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:12.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.759+0000 7fb4e1ffb640 1 -- 172.21.15.57:0/2847453564 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4dc011830 con 0x7fb4ec0d1910 2023-10-19T14:20:12.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.759+0000 7fb4e1ffb640 1 -- 172.21.15.57:0/2847453564 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fb4dc011aa0 con 0x7fb4ec0d1910 2023-10-19T14:20:12.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.759+0000 7fb4e1ffb640 1 -- 172.21.15.57:0/2847453564 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4dc010380 con 0x7fb4ec0d1910 2023-10-19T14:20:12.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.759+0000 7fb4f3656640 1 -- 172.21.15.57:0/2847453564 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4ec1e3f40 con 0x7fb4ec0d1910 2023-10-19T14:20:12.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.759+0000 7fb4f3656640 1 -- 172.21.15.57:0/2847453564 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4ec0ca7c0 con 0x7fb4ec0d1910 2023-10-19T14:20:12.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.760+0000 7fb4f3656640 1 -- 172.21.15.57:0/2847453564 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4ec1e4460 con 0x7fb4ec0d1910 2023-10-19T14:20:12.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.761+0000 7fb4e1ffb640 1 -- 172.21.15.57:0/2847453564 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb4dc0104e0 con 0x7fb4ec0d1910 2023-10-19T14:20:12.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.761+0000 7fb4e1ffb640 1 --2- 172.21.15.57:0/2847453564 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fb4d006af60 0x7fb4d006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:12.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.761+0000 7fb4e1ffb640 1 -- 172.21.15.57:0/2847453564 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 6065+0+0 (secure 0 0 0) 0x7fb4dc089c80 con 0x7fb4ec0d1910 2023-10-19T14:20:12.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.762+0000 7fb4f13f2640 1 --2- 172.21.15.57:0/2847453564 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fb4d006af60 0x7fb4d006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:12.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.762+0000 7fb4f13f2640 1 --2- 172.21.15.57:0/2847453564 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fb4d006af60 0x7fb4d006d420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb4e80046b0 tx=0x7fb4e8009040 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:12.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.765+0000 7fb4e1ffb640 1 -- 172.21.15.57:0/2847453564 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb4ec1e4460 con 0x7fb4ec0d1910 2023-10-19T14:20:12.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.942+0000 7fb4f3656640 1 -- 172.21.15.57:0/2847453564 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb4ec1e4460 con 0x7fb4ec0d1910 2023-10-19T14:20:12.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.943+0000 7fb4e1ffb640 1 -- 172.21.15.57:0/2847453564 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v215) v1 ==== 74+0+14396 (secure 0 0 0) 0x7fb4ec1e4460 con 0x7fb4ec0d1910 2023-10-19T14:20:12.946 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:20:12.947 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":215,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T14:20:11.623679+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.57:6833/1674945162":"2023-10-20T14:20:08.360375+0000","172.21.15.179:0/31901178":"2023-10-20T14:20:05.962854+0000","172.21.15.179:0/2690789813":"2023-10-20T14:20:05.962854+0000","172.21.15.179:0/3626712747":"2023-10-20T14:20:05.962854+0000","172.21.15.179:0/613069221":"2023-10-20T14:20:05.962854+0000","172.21.15.179:6833/366917243":"2023-10-20T14:20:05.962854+0000","172.21.15.179:6832/366917243":"2023-10-20T14:20:05.962854+0000","172.21.15.179:0/3217199253":"2023-10-20T14:20:05.962854+0000","172.21.15.179:0/3870514757":"2023-10-20T14:20:05.962854+0000","172.21.15.179:0/2246649044":"2023-10-20T14:18:49.915871+0000","172.21.15.179:0/1186360784":"2023-10-20T14:18:49.915871+0000","172.21.15.179:0/1525973956":"2023-10-20T14:18:49.915871+0000","172.21.15.179:0/1500866753":"2023-10-20T14:18:49.915871+0000","172.21.15.179:0/940398488":"2023-10-20T14:20:05.962854+0000","172.21.15.179:6833/2730461479":"2023-10-20T14:18:49.915871+0000","172.21.15.57:6835/1674945162":"2023-10-20T14:20:08.360375+0000","172.21.15.179:6832/2730461479":"2023-10-20T14:18:49.915871+0000","172.21.15.179:0/1627936534":"2023-10-20T14:20:05.962854+0000","172.21.15.179:0/3327639073":"2023-10-20T14:18:49.915871+0000","172.21.15.179:0/1497317399":"2023-10-20T14:18:49.915871+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T14:20:12.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.945+0000 7fb4f3656640 1 -- 172.21.15.57:0/2847453564 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fb4d006af60 msgr2=0x7fb4d006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:12.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.945+0000 7fb4f3656640 1 --2- 172.21.15.57:0/2847453564 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fb4d006af60 0x7fb4d006d420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb4e80046b0 tx=0x7fb4e8009040 comp rx=0 tx=0).stop 2023-10-19T14:20:12.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.946+0000 7fb4f3656640 1 -- 172.21.15.57:0/2847453564 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4ec0d1910 msgr2=0x7fb4ec0c97c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:12.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.946+0000 7fb4f3656640 1 --2- 172.21.15.57:0/2847453564 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4ec0d1910 0x7fb4ec0c97c0 secure :-1 s=READY pgs=2303 cs=0 l=1 rev1=1 crypto rx=0x7fb4dc00b3e0 tx=0x7fb4dc00b8a0 comp rx=0 tx=0).stop 2023-10-19T14:20:12.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.946+0000 7fb4f3656640 1 -- 172.21.15.57:0/2847453564 shutdown_connections 2023-10-19T14:20:12.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.946+0000 7fb4f3656640 1 --2- 172.21.15.57:0/2847453564 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fb4d006af60 0x7fb4d006d420 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:12.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.946+0000 7fb4f3656640 1 --2- 172.21.15.57:0/2847453564 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb4ec15b2a0 0x7fb4ec0c9d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:12.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.946+0000 7fb4f3656640 1 --2- 172.21.15.57:0/2847453564 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4ec0d1910 0x7fb4ec0c97c0 unknown :-1 s=CLOSED pgs=2303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:12.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.946+0000 7fb4f3656640 1 --2- 172.21.15.57:0/2847453564 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb4ec0cf340 0x7fb4ec158040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:12.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.946+0000 7fb4f3656640 1 -- 172.21.15.57:0/2847453564 >> 172.21.15.57:0/2847453564 conn(0x7fb4ec0d0d60 msgr2=0x7fb4ec15a2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:12.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.946+0000 7fb4f3656640 1 -- 172.21.15.57:0/2847453564 shutdown_connections 2023-10-19T14:20:12.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:12.946+0000 7fb4f3656640 1 -- 172.21.15.57:0/2847453564 wait complete. 2023-10-19T14:20:12.959 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-19T12:44:07.371186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-19T14:20:12.960 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-19T14:20:13.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.068+0000 7faa4acd7640 1 -- 172.21.15.57:0/3161841297 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa4406af00 msgr2=0x7faa4406b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:13.072 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.068+0000 7faa4acd7640 1 --2- 172.21.15.57:0/3161841297 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa4406af00 0x7faa4406b2e0 secure :-1 s=READY pgs=2305 cs=0 l=1 rev1=1 crypto rx=0x7faa3c009860 tx=0x7faa3c02bd60 comp rx=0 tx=0).stop 2023-10-19T14:20:13.072 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.068+0000 7faa4acd7640 1 -- 172.21.15.57:0/3161841297 shutdown_connections 2023-10-19T14:20:13.072 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.068+0000 7faa4acd7640 1 --2- 172.21.15.57:0/3161841297 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa4406af00 0x7faa4406b2e0 unknown :-1 s=CLOSED pgs=2305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:13.072 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.068+0000 7faa4acd7640 1 -- 172.21.15.57:0/3161841297 >> 172.21.15.57:0/3161841297 conn(0x7faa44069930 msgr2=0x7faa44069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:13.072 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.068+0000 7faa4acd7640 1 -- 172.21.15.57:0/3161841297 shutdown_connections 2023-10-19T14:20:13.073 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.068+0000 7faa4acd7640 1 -- 172.21.15.57:0/3161841297 wait complete. 2023-10-19T14:20:13.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.069+0000 7faa4acd7640 1 Processor -- start 2023-10-19T14:20:13.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.070+0000 7faa4acd7640 1 -- start start 2023-10-19T14:20:13.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.070+0000 7faa4acd7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa4406af00 0x7faa440b1090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:13.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.070+0000 7faa4acd7640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faa440abc10 0x7faa440af210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:13.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.070+0000 7faa4acd7640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7faa440c96b0 0x7faa44070340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:13.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.070+0000 7faa4acd7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa440afff0 con 0x7faa4406af00 2023-10-19T14:20:13.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.070+0000 7faa4acd7640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa440b02d0 con 0x7faa440abc10 2023-10-19T14:20:13.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.070+0000 7faa4acd7640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa440b05b0 con 0x7faa440c96b0 2023-10-19T14:20:13.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.070+0000 7faa48a73640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa4406af00 0x7faa440b1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:13.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.070+0000 7faa48a73640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa4406af00 0x7faa440b1090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35642/0 (socket says 172.21.15.57:35642) 2023-10-19T14:20:13.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.070+0000 7faa48a73640 1 -- 172.21.15.57:0/1417359594 learned_addr learned my addr 172.21.15.57:0/1417359594 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:13.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.071+0000 7faa48a73640 1 -- 172.21.15.57:0/1417359594 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7faa440c96b0 msgr2=0x7faa44070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:13.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.071+0000 7faa48a73640 1 --2- 172.21.15.57:0/1417359594 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7faa440c96b0 0x7faa44070340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:13.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.071+0000 7faa48a73640 1 -- 172.21.15.57:0/1417359594 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faa440abc10 msgr2=0x7faa440af210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:13.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.071+0000 7faa48a73640 1 --2- 172.21.15.57:0/1417359594 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faa440abc10 0x7faa440af210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:13.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.071+0000 7faa48a73640 1 -- 172.21.15.57:0/1417359594 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa3c009490 con 0x7faa4406af00 2023-10-19T14:20:13.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.071+0000 7faa48a73640 1 --2- 172.21.15.57:0/1417359594 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa4406af00 0x7faa440b1090 secure :-1 s=READY pgs=2306 cs=0 l=1 rev1=1 crypto rx=0x7faa3c009980 tx=0x7faa3c003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:13.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.071+0000 7faa417fa640 1 -- 172.21.15.57:0/1417359594 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa3c003e20 con 0x7faa4406af00 2023-10-19T14:20:13.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.071+0000 7faa4acd7640 1 -- 172.21.15.57:0/1417359594 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa440b0890 con 0x7faa4406af00 2023-10-19T14:20:13.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.071+0000 7faa417fa640 1 -- 172.21.15.57:0/1417359594 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7faa3c005110 con 0x7faa4406af00 2023-10-19T14:20:13.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.071+0000 7faa417fa640 1 -- 172.21.15.57:0/1417359594 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa3c005520 con 0x7faa4406af00 2023-10-19T14:20:13.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.071+0000 7faa4acd7640 1 -- 172.21.15.57:0/1417359594 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa440af750 con 0x7faa4406af00 2023-10-19T14:20:13.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.072+0000 7faa4acd7640 1 -- 172.21.15.57:0/1417359594 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa440b0db0 con 0x7faa4406af00 2023-10-19T14:20:13.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.073+0000 7faa417fa640 1 -- 172.21.15.57:0/1417359594 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7faa3c0047a0 con 0x7faa4406af00 2023-10-19T14:20:13.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.073+0000 7faa417fa640 1 --2- 172.21.15.57:0/1417359594 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7faa2406af60 0x7faa2406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:13.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.073+0000 7faa417fa640 1 -- 172.21.15.57:0/1417359594 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 6065+0+0 (secure 0 0 0) 0x7faa3c0ac300 con 0x7faa4406af00 2023-10-19T14:20:13.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.073+0000 7faa43fff640 1 --2- 172.21.15.57:0/1417359594 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7faa2406af60 0x7faa2406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:13.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.074+0000 7faa43fff640 1 --2- 172.21.15.57:0/1417359594 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7faa2406af60 0x7faa2406d420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7faa340012e0 tx=0x7faa34007d30 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:13.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.077+0000 7faa417fa640 1 -- 172.21.15.57:0/1417359594 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7faa3c0770b0 con 0x7faa4406af00 2023-10-19T14:20:13.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.251+0000 7faa4acd7640 1 -- 172.21.15.57:0/1417359594 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7faa440690e0 con 0x7faa4406af00 2023-10-19T14:20:13.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.252+0000 7faa417fa640 1 -- 172.21.15.57:0/1417359594 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v215) v1 ==== 93+0+10 (secure 0 0 0) 0x7faa3c07a6d0 con 0x7faa4406af00 2023-10-19T14:20:13.255 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-19T14:20:13.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.254+0000 7faa4acd7640 1 -- 172.21.15.57:0/1417359594 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7faa2406af60 msgr2=0x7faa2406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:13.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.254+0000 7faa4acd7640 1 --2- 172.21.15.57:0/1417359594 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7faa2406af60 0x7faa2406d420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7faa340012e0 tx=0x7faa34007d30 comp rx=0 tx=0).stop 2023-10-19T14:20:13.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.254+0000 7faa4acd7640 1 -- 172.21.15.57:0/1417359594 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa4406af00 msgr2=0x7faa440b1090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:13.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.254+0000 7faa4acd7640 1 --2- 172.21.15.57:0/1417359594 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa4406af00 0x7faa440b1090 secure :-1 s=READY pgs=2306 cs=0 l=1 rev1=1 crypto rx=0x7faa3c009980 tx=0x7faa3c003820 comp rx=0 tx=0).stop 2023-10-19T14:20:13.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.254+0000 7faa4acd7640 1 -- 172.21.15.57:0/1417359594 shutdown_connections 2023-10-19T14:20:13.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.254+0000 7faa4acd7640 1 --2- 172.21.15.57:0/1417359594 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7faa2406af60 0x7faa2406d420 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:13.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.254+0000 7faa4acd7640 1 --2- 172.21.15.57:0/1417359594 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7faa440c96b0 0x7faa44070340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:13.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.254+0000 7faa4acd7640 1 --2- 172.21.15.57:0/1417359594 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faa440abc10 0x7faa440af210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:13.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.254+0000 7faa4acd7640 1 --2- 172.21.15.57:0/1417359594 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa4406af00 0x7faa440b1090 unknown :-1 s=CLOSED pgs=2306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:13.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.254+0000 7faa4acd7640 1 -- 172.21.15.57:0/1417359594 >> 172.21.15.57:0/1417359594 conn(0x7faa44069930 msgr2=0x7faa440adea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:13.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.254+0000 7faa4acd7640 1 -- 172.21.15.57:0/1417359594 shutdown_connections 2023-10-19T14:20:13.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.254+0000 7faa4acd7640 1 -- 172.21.15.57:0/1417359594 wait complete. 2023-10-19T14:20:13.268 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs' 2023-10-19T14:20:13.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.374+0000 7f789cc0e640 1 -- 172.21.15.57:0/2252124422 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7898070af0 msgr2=0x7f78980affe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:13.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.374+0000 7f789cc0e640 1 --2- 172.21.15.57:0/2252124422 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7898070af0 0x7f78980affe0 secure :-1 s=READY pgs=2308 cs=0 l=1 rev1=1 crypto rx=0x7f78980cbd00 tx=0x7f788802c080 comp rx=0 tx=0).stop 2023-10-19T14:20:13.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.374+0000 7f789cc0e640 1 -- 172.21.15.57:0/2252124422 shutdown_connections 2023-10-19T14:20:13.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.374+0000 7f789cc0e640 1 --2- 172.21.15.57:0/2252124422 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7898070af0 0x7f78980affe0 unknown :-1 s=CLOSED pgs=2308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:13.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.374+0000 7f789cc0e640 1 --2- 172.21.15.57:0/2252124422 >> v2:172.21.15.179:3300/0 conn(0x7f789813e520 0x7f789806bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:13.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.374+0000 7f789cc0e640 1 -- 172.21.15.57:0/2252124422 >> 172.21.15.57:0/2252124422 conn(0x7f7898069930 msgr2=0x7f7898069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:13.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.375+0000 7f789cc0e640 1 -- 172.21.15.57:0/2252124422 shutdown_connections 2023-10-19T14:20:13.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.375+0000 7f789cc0e640 1 -- 172.21.15.57:0/2252124422 wait complete. 2023-10-19T14:20:13.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.376+0000 7f789cc0e640 1 Processor -- start 2023-10-19T14:20:13.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.376+0000 7f789cc0e640 1 -- start start 2023-10-19T14:20:13.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.377+0000 7f789cc0e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7898070af0 0x7f78981f1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:13.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.377+0000 7f789cc0e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f789813e520 0x7f78981e2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:13.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.377+0000 7f789cc0e640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7898141690 0x7f78981e2950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:13.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.377+0000 7f789cc0e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78981dbc90 con 0x7f7898070af0 2023-10-19T14:20:13.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.377+0000 7f789cc0e640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f789813f320 con 0x7f789813e520 2023-10-19T14:20:13.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.377+0000 7f789cc0e640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f789813f600 con 0x7f7898141690 2023-10-19T14:20:13.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.377+0000 7f789659c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7898070af0 0x7f78981f1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:13.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.377+0000 7f789659c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7898070af0 0x7f78981f1ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35662/0 (socket says 172.21.15.57:35662) 2023-10-19T14:20:13.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.377+0000 7f789659c640 1 -- 172.21.15.57:0/3332645320 learned_addr learned my addr 172.21.15.57:0/3332645320 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:13.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.377+0000 7f789659c640 1 -- 172.21.15.57:0/3332645320 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7898141690 msgr2=0x7f78981e2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:13.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.377+0000 7f789659c640 1 --2- 172.21.15.57:0/3332645320 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7898141690 0x7f78981e2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:13.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.377+0000 7f789659c640 1 -- 172.21.15.57:0/3332645320 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f789813e520 msgr2=0x7f78981e2410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:13.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.377+0000 7f789659c640 1 --2- 172.21.15.57:0/3332645320 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f789813e520 0x7f78981e2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:13.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.377+0000 7f789659c640 1 -- 172.21.15.57:0/3332645320 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7888009840 con 0x7f7898070af0 2023-10-19T14:20:13.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.377+0000 7f789659c640 1 --2- 172.21.15.57:0/3332645320 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7898070af0 0x7f78981f1ea0 secure :-1 s=READY pgs=2309 cs=0 l=1 rev1=1 crypto rx=0x7f789000d950 tx=0x7f789000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:13.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.377+0000 7f7886ffd640 1 -- 172.21.15.57:0/3332645320 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7890009c30 con 0x7f7898070af0 2023-10-19T14:20:13.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.377+0000 7f7886ffd640 1 -- 172.21.15.57:0/3332645320 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f7890009e80 con 0x7f7898070af0 2023-10-19T14:20:13.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.377+0000 7f789cc0e640 1 -- 172.21.15.57:0/3332645320 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f789813f8e0 con 0x7f7898070af0 2023-10-19T14:20:13.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.377+0000 7f7886ffd640 1 -- 172.21.15.57:0/3332645320 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7890004280 con 0x7f7898070af0 2023-10-19T14:20:13.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.377+0000 7f789cc0e640 1 -- 172.21.15.57:0/3332645320 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f789813fea0 con 0x7f7898070af0 2023-10-19T14:20:13.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.378+0000 7f789cc0e640 1 -- 172.21.15.57:0/3332645320 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7898140060 con 0x7f7898070af0 2023-10-19T14:20:13.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.379+0000 7f7886ffd640 1 -- 172.21.15.57:0/3332645320 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f78900035f0 con 0x7f7898070af0 2023-10-19T14:20:13.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.379+0000 7f7886ffd640 1 --2- 172.21.15.57:0/3332645320 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f787806af60 0x7f787806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:13.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.379+0000 7f7886ffd640 1 -- 172.21.15.57:0/3332645320 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 6065+0+0 (secure 0 0 0) 0x7f7890089be0 con 0x7f7898070af0 2023-10-19T14:20:13.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.379+0000 7f7895d9b640 1 --2- 172.21.15.57:0/3332645320 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f787806af60 0x7f787806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:13.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.380+0000 7f7895d9b640 1 --2- 172.21.15.57:0/3332645320 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f787806af60 0x7f787806d420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f788c0046b0 tx=0x7f788c007a70 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:13.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.383+0000 7f7886ffd640 1 -- 172.21.15.57:0/3332645320 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7890054900 con 0x7f7898070af0 2023-10-19T14:20:13.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.626+0000 7f789cc0e640 1 -- 172.21.15.57:0/3332645320 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs"]} v 0) v1 -- 0x7f78980690e0 con 0x7f7898070af0 2023-10-19T14:20:13.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.634+0000 7f7886ffd640 1 -- 172.21.15.57:0/3332645320 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs"]}]=0 v3122) v1 ==== 189+0+0 (secure 0 0 0) 0x7f7890017100 con 0x7f7898070af0 2023-10-19T14:20:13.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.636+0000 7f789cc0e640 1 -- 172.21.15.57:0/3332645320 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f787806af60 msgr2=0x7f787806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:13.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.636+0000 7f789cc0e640 1 --2- 172.21.15.57:0/3332645320 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f787806af60 0x7f787806d420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f788c0046b0 tx=0x7f788c007a70 comp rx=0 tx=0).stop 2023-10-19T14:20:13.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.636+0000 7f789cc0e640 1 -- 172.21.15.57:0/3332645320 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7898070af0 msgr2=0x7f78981f1ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:13.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.636+0000 7f789cc0e640 1 --2- 172.21.15.57:0/3332645320 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7898070af0 0x7f78981f1ea0 secure :-1 s=READY pgs=2309 cs=0 l=1 rev1=1 crypto rx=0x7f789000d950 tx=0x7f789000de10 comp rx=0 tx=0).stop 2023-10-19T14:20:13.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.636+0000 7f789cc0e640 1 -- 172.21.15.57:0/3332645320 shutdown_connections 2023-10-19T14:20:13.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.636+0000 7f789cc0e640 1 --2- 172.21.15.57:0/3332645320 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f787806af60 0x7f787806d420 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:13.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.636+0000 7f789cc0e640 1 --2- 172.21.15.57:0/3332645320 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7898141690 0x7f78981e2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:13.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.636+0000 7f789cc0e640 1 --2- 172.21.15.57:0/3332645320 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f789813e520 0x7f78981e2410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:13.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.636+0000 7f789cc0e640 1 --2- 172.21.15.57:0/3332645320 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7898070af0 0x7f78981f1ea0 unknown :-1 s=CLOSED pgs=2309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:13.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.636+0000 7f789cc0e640 1 -- 172.21.15.57:0/3332645320 >> 172.21.15.57:0/3332645320 conn(0x7f7898069930 msgr2=0x7f78980b0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:13.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.636+0000 7f789cc0e640 1 -- 172.21.15.57:0/3332645320 shutdown_connections 2023-10-19T14:20:13.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.636+0000 7f789cc0e640 1 -- 172.21.15.57:0/3332645320 wait complete. 2023-10-19T14:20:13.650 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2023-10-19T14:20:13.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.755+0000 7f894eb73640 1 -- 172.21.15.57:0/111916045 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f89480cf340 msgr2=0x7f89480d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:13.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.755+0000 7f894eb73640 1 --2- 172.21.15.57:0/111916045 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f89480cf340 0x7f89480d1910 secure :-1 s=READY pgs=2310 cs=0 l=1 rev1=1 crypto rx=0x7f894808a120 tx=0x7f894002c1d0 comp rx=0 tx=0).stop 2023-10-19T14:20:13.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.755+0000 7f894eb73640 1 -- 172.21.15.57:0/111916045 shutdown_connections 2023-10-19T14:20:13.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.755+0000 7f894eb73640 1 --2- 172.21.15.57:0/111916045 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f89480ceb40 0x7f89480b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:13.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.755+0000 7f894eb73640 1 --2- 172.21.15.57:0/111916045 >> v2:172.21.15.179:3300/0 conn(0x7f894815b2a0 0x7f89480cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:13.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.755+0000 7f894eb73640 1 --2- 172.21.15.57:0/111916045 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f89480cf340 0x7f89480d1910 unknown :-1 s=CLOSED pgs=2310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:13.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.755+0000 7f894eb73640 1 -- 172.21.15.57:0/111916045 >> 172.21.15.57:0/111916045 conn(0x7f89480d0d60 msgr2=0x7f89480d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:13.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.755+0000 7f894eb73640 1 -- 172.21.15.57:0/111916045 shutdown_connections 2023-10-19T14:20:13.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.755+0000 7f894eb73640 1 -- 172.21.15.57:0/111916045 wait complete. 2023-10-19T14:20:13.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.756+0000 7f894eb73640 1 Processor -- start 2023-10-19T14:20:13.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.756+0000 7f894eb73640 1 -- start start 2023-10-19T14:20:13.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.757+0000 7f894eb73640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f89480ceb40 0x7f8948153940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:13.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.757+0000 7f894eb73640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f89480cf340 0x7f8948158270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:13.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.757+0000 7f894eb73640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f894815b2a0 0x7f894815d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:13.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.757+0000 7f894eb73640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8948157de0 con 0x7f89480cf340 2023-10-19T14:20:13.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.757+0000 7f894eb73640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89481577d0 con 0x7f894815b2a0 2023-10-19T14:20:13.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.757+0000 7f894eb73640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8948153e80 con 0x7f89480ceb40 2023-10-19T14:20:13.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.757+0000 7f8947fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f89480cf340 0x7f8948158270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:13.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.757+0000 7f8947fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f89480cf340 0x7f8948158270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35688/0 (socket says 172.21.15.57:35688) 2023-10-19T14:20:13.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.758+0000 7f8947fff640 1 -- 172.21.15.57:0/3666585459 learned_addr learned my addr 172.21.15.57:0/3666585459 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:13.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.758+0000 7f894c90f640 1 --2- 172.21.15.57:0/3666585459 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f89480ceb40 0x7f8948153940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:13.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.758+0000 7f894d110640 1 --2- 172.21.15.57:0/3666585459 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f894815b2a0 0x7f894815d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:13.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.758+0000 7f8947fff640 1 -- 172.21.15.57:0/3666585459 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f89480ceb40 msgr2=0x7f8948153940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:13.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.758+0000 7f8947fff640 1 --2- 172.21.15.57:0/3666585459 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f89480ceb40 0x7f8948153940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:13.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.758+0000 7f8947fff640 1 -- 172.21.15.57:0/3666585459 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f894815b2a0 msgr2=0x7f894815d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:13.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.758+0000 7f8947fff640 1 --2- 172.21.15.57:0/3666585459 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f894815b2a0 0x7f894815d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:13.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.758+0000 7f8947fff640 1 -- 172.21.15.57:0/3666585459 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8940009900 con 0x7f89480cf340 2023-10-19T14:20:13.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.758+0000 7f8947fff640 1 --2- 172.21.15.57:0/3666585459 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f89480cf340 0x7f8948158270 secure :-1 s=READY pgs=2311 cs=0 l=1 rev1=1 crypto rx=0x7f893800c970 tx=0x7f893800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:13.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.758+0000 7f89457fa640 1 -- 172.21.15.57:0/3666585459 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8938007bb0 con 0x7f89480cf340 2023-10-19T14:20:13.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.758+0000 7f89457fa640 1 -- 172.21.15.57:0/3666585459 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f8938007e00 con 0x7f89480cf340 2023-10-19T14:20:13.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.758+0000 7f894eb73640 1 -- 172.21.15.57:0/3666585459 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89480ca640 con 0x7f89480cf340 2023-10-19T14:20:13.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.758+0000 7f89457fa640 1 -- 172.21.15.57:0/3666585459 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8938005280 con 0x7f89480cf340 2023-10-19T14:20:13.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.758+0000 7f894eb73640 1 -- 172.21.15.57:0/3666585459 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8948157a70 con 0x7f89480cf340 2023-10-19T14:20:13.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.759+0000 7f894eb73640 1 -- 172.21.15.57:0/3666585459 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8948157630 con 0x7f89480cf340 2023-10-19T14:20:13.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.760+0000 7f89457fa640 1 -- 172.21.15.57:0/3666585459 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f89380053e0 con 0x7f89480cf340 2023-10-19T14:20:13.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.760+0000 7f89457fa640 1 --2- 172.21.15.57:0/3666585459 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f892806af60 0x7f892806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:13.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.760+0000 7f89457fa640 1 -- 172.21.15.57:0/3666585459 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 6065+0+0 (secure 0 0 0) 0x7f8938089e80 con 0x7f89480cf340 2023-10-19T14:20:13.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.760+0000 7f894c90f640 1 --2- 172.21.15.57:0/3666585459 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f892806af60 0x7f892806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:13.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.761+0000 7f894c90f640 1 --2- 172.21.15.57:0/3666585459 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f892806af60 0x7f892806d420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f8940036000 tx=0x7f8940004ab0 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:13.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.764+0000 7f89457fa640 1 -- 172.21.15.57:0/3666585459 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8948157630 con 0x7f89480cf340 2023-10-19T14:20:13.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.941+0000 7f894eb73640 1 -- 172.21.15.57:0/3666585459 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "config dump"} v 0) v1 -- 0x7f8948157630 con 0x7f89480cf340 2023-10-19T14:20:13.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.941+0000 7f89457fa640 1 -- 172.21.15.57:0/3666585459 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v74) v1 ==== 59+0+2220 (secure 0 0 0) 0x7f8948157630 con 0x7f89480cf340 2023-10-19T14:20:13.945 INFO:teuthology.orchestra.run.smithi057.stdout:WHO MASK LEVEL OPTION VALUE RO 2023-10-19T14:20:13.945 INFO:teuthology.orchestra.run.smithi057.stdout:mon advanced mon_op_complaint_time 120 2023-10-19T14:20:13.945 INFO:teuthology.orchestra.run.smithi057.stdout:osd advanced osd_op_complaint_time 180.000000 2023-10-19T14:20:13.945 INFO:teuthology.orchestra.run.smithi057.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 4775.995373 2023-10-19T14:20:13.946 INFO:teuthology.orchestra.run.smithi057.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 4208.118410 2023-10-19T14:20:13.946 INFO:teuthology.orchestra.run.smithi057.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 4541.123030 2023-10-19T14:20:13.946 INFO:teuthology.orchestra.run.smithi057.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 4877.592206 2023-10-19T14:20:13.946 INFO:teuthology.orchestra.run.smithi057.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 5019.748032 2023-10-19T14:20:13.946 INFO:teuthology.orchestra.run.smithi057.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 5000.074376 2023-10-19T14:20:13.947 INFO:teuthology.orchestra.run.smithi057.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 5153.479439 2023-10-19T14:20:13.947 INFO:teuthology.orchestra.run.smithi057.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 4977.363845 2023-10-19T14:20:13.947 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced debug_mds 20/20 2023-10-19T14:20:13.947 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced debug_mds_balancer 20/20 2023-10-19T14:20:13.947 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced debug_ms 1/1 2023-10-19T14:20:13.947 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced mds_bal_fragment_size_max 10000 2023-10-19T14:20:13.948 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced mds_bal_merge_size 5 2023-10-19T14:20:13.948 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced mds_bal_split_bits 3 2023-10-19T14:20:13.948 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced mds_bal_split_size 100 2023-10-19T14:20:13.948 INFO:teuthology.orchestra.run.smithi057.stdout:mds dev mds_debug_frag true 2023-10-19T14:20:13.948 INFO:teuthology.orchestra.run.smithi057.stdout:mds dev mds_debug_scatterstat true 2023-10-19T14:20:13.949 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced mds_op_complaint_time 180.000000 2023-10-19T14:20:13.949 INFO:teuthology.orchestra.run.smithi057.stdout:mds dev mds_verify_scatter true 2023-10-19T14:20:13.949 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced osd_op_complaint_time 180.000000 2023-10-19T14:20:13.949 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced rados_mon_op_timeout 900 2023-10-19T14:20:13.949 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced rados_osd_op_timeout 900 2023-10-19T14:20:13.950 INFO:teuthology.orchestra.run.smithi057.stdout:client advanced client_mount_timeout 600 2023-10-19T14:20:13.950 INFO:teuthology.orchestra.run.smithi057.stdout:client advanced debug_client 20/20 2023-10-19T14:20:13.950 INFO:teuthology.orchestra.run.smithi057.stdout:client advanced debug_ms 1/1 2023-10-19T14:20:13.950 INFO:teuthology.orchestra.run.smithi057.stdout:client advanced rados_mon_op_timeout 900 2023-10-19T14:20:13.950 INFO:teuthology.orchestra.run.smithi057.stdout:client advanced rados_osd_op_timeout 900 2023-10-19T14:20:13.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.943+0000 7f894eb73640 1 -- 172.21.15.57:0/3666585459 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f892806af60 msgr2=0x7f892806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:13.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.944+0000 7f894eb73640 1 --2- 172.21.15.57:0/3666585459 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f892806af60 0x7f892806d420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f8940036000 tx=0x7f8940004ab0 comp rx=0 tx=0).stop 2023-10-19T14:20:13.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.944+0000 7f894eb73640 1 -- 172.21.15.57:0/3666585459 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f89480cf340 msgr2=0x7f8948158270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:13.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.944+0000 7f894eb73640 1 --2- 172.21.15.57:0/3666585459 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f89480cf340 0x7f8948158270 secure :-1 s=READY pgs=2311 cs=0 l=1 rev1=1 crypto rx=0x7f893800c970 tx=0x7f893800ce30 comp rx=0 tx=0).stop 2023-10-19T14:20:13.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.944+0000 7f894eb73640 1 -- 172.21.15.57:0/3666585459 shutdown_connections 2023-10-19T14:20:13.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.944+0000 7f894eb73640 1 --2- 172.21.15.57:0/3666585459 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f892806af60 0x7f892806d420 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:13.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.944+0000 7f894eb73640 1 --2- 172.21.15.57:0/3666585459 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f894815b2a0 0x7f894815d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:13.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.944+0000 7f894eb73640 1 --2- 172.21.15.57:0/3666585459 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f89480cf340 0x7f8948158270 unknown :-1 s=CLOSED pgs=2311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:13.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.944+0000 7f894eb73640 1 --2- 172.21.15.57:0/3666585459 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f89480ceb40 0x7f8948153940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:13.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.944+0000 7f894eb73640 1 -- 172.21.15.57:0/3666585459 >> 172.21.15.57:0/3666585459 conn(0x7f89480d0d60 msgr2=0x7f894815a440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:13.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.944+0000 7f894eb73640 1 -- 172.21.15.57:0/3666585459 shutdown_connections 2023-10-19T14:20:13.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:13.944+0000 7f894eb73640 1 -- 172.21.15.57:0/3666585459 wait complete. 2023-10-19T14:20:13.957 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-19T14:20:14.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.066+0000 7fedf2afc640 1 -- 172.21.15.57:0/268106625 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fedec0cbd20 msgr2=0x7fedec158300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:14.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.066+0000 7fedf2afc640 1 --2- 172.21.15.57:0/268106625 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fedec0cbd20 0x7fedec158300 secure :-1 s=READY pgs=2313 cs=0 l=1 rev1=1 crypto rx=0x7fedec0b10f0 tx=0x7fede802bd70 comp rx=0 tx=0).stop 2023-10-19T14:20:14.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.066+0000 7fedf2afc640 1 -- 172.21.15.57:0/268106625 shutdown_connections 2023-10-19T14:20:14.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.066+0000 7fedf2afc640 1 --2- 172.21.15.57:0/268106625 >> v2:172.21.15.179:3300/0 conn(0x7fedec0b0530 0x7fedec0b09a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:14.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.066+0000 7fedf2afc640 1 --2- 172.21.15.57:0/268106625 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fedec0cbd20 0x7fedec158300 unknown :-1 s=CLOSED pgs=2313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:14.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.066+0000 7fedf2afc640 1 -- 172.21.15.57:0/268106625 >> 172.21.15.57:0/268106625 conn(0x7fedec069930 msgr2=0x7fedec069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:14.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.066+0000 7fedf2afc640 1 -- 172.21.15.57:0/268106625 shutdown_connections 2023-10-19T14:20:14.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.066+0000 7fedf2afc640 1 -- 172.21.15.57:0/268106625 wait complete. 2023-10-19T14:20:14.072 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.067+0000 7fedf2afc640 1 Processor -- start 2023-10-19T14:20:14.073 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.067+0000 7fedf2afc640 1 -- start start 2023-10-19T14:20:14.073 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.068+0000 7fedf2afc640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fedec0afa20 0x7fedec13d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:14.073 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.068+0000 7fedf2afc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fedec0b0530 0x7fedec1562c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:14.073 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.068+0000 7fedf2afc640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fedec0cbd20 0x7fedec156800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:14.074 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.068+0000 7fedf2afc640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedec15aa30 con 0x7fedec0b0530 2023-10-19T14:20:14.074 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.068+0000 7fedf2afc640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedec159c70 con 0x7fedec0afa20 2023-10-19T14:20:14.074 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.068+0000 7fedf2afc640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedec159f50 con 0x7fedec0cbd20 2023-10-19T14:20:14.074 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.068+0000 7fede3fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fedec0b0530 0x7fedec1562c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:14.074 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.068+0000 7fede3fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fedec0b0530 0x7fedec1562c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35712/0 (socket says 172.21.15.57:35712) 2023-10-19T14:20:14.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.068+0000 7fede3fff640 1 -- 172.21.15.57:0/2072966080 learned_addr learned my addr 172.21.15.57:0/2072966080 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:14.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.068+0000 7fedf1099640 1 --2- 172.21.15.57:0/2072966080 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fedec0cbd20 0x7fedec156800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:14.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.068+0000 7fede3fff640 1 -- 172.21.15.57:0/2072966080 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fedec0cbd20 msgr2=0x7fedec156800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:14.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.068+0000 7fedf0898640 1 --2- 172.21.15.57:0/2072966080 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fedec0afa20 0x7fedec13d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:14.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.068+0000 7fede3fff640 1 --2- 172.21.15.57:0/2072966080 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fedec0cbd20 0x7fedec156800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:14.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.068+0000 7fede3fff640 1 -- 172.21.15.57:0/2072966080 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fedec0afa20 msgr2=0x7fedec13d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:14.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.068+0000 7fede3fff640 1 --2- 172.21.15.57:0/2072966080 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fedec0afa20 0x7fedec13d130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:14.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.068+0000 7fede3fff640 1 -- 172.21.15.57:0/2072966080 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fede8009490 con 0x7fedec0b0530 2023-10-19T14:20:14.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.068+0000 7fedf0898640 1 --2- 172.21.15.57:0/2072966080 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fedec0afa20 0x7fedec13d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:20:14.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.068+0000 7fedf1099640 1 --2- 172.21.15.57:0/2072966080 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fedec0cbd20 0x7fedec156800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:20:14.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.068+0000 7fede3fff640 1 --2- 172.21.15.57:0/2072966080 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fedec0b0530 0x7fedec1562c0 secure :-1 s=READY pgs=2314 cs=0 l=1 rev1=1 crypto rx=0x7feddc00c4b0 tx=0x7feddc00c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:14.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.068+0000 7fede17fa640 1 -- 172.21.15.57:0/2072966080 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feddc005480 con 0x7fedec0b0530 2023-10-19T14:20:14.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.069+0000 7fede17fa640 1 -- 172.21.15.57:0/2072966080 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7feddc0036e0 con 0x7fedec0b0530 2023-10-19T14:20:14.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.069+0000 7fede17fa640 1 -- 172.21.15.57:0/2072966080 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feddc003b30 con 0x7fedec0b0530 2023-10-19T14:20:14.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.069+0000 7fedf2afc640 1 -- 172.21.15.57:0/2072966080 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedec15a230 con 0x7fedec0b0530 2023-10-19T14:20:14.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.069+0000 7fedf2afc640 1 -- 172.21.15.57:0/2072966080 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedec13d670 con 0x7fedec0b0530 2023-10-19T14:20:14.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.070+0000 7fedf2afc640 1 -- 172.21.15.57:0/2072966080 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedec0690e0 con 0x7fedec0b0530 2023-10-19T14:20:14.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.070+0000 7fede17fa640 1 -- 172.21.15.57:0/2072966080 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7feddc003c90 con 0x7fedec0b0530 2023-10-19T14:20:14.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.070+0000 7fede17fa640 1 --2- 172.21.15.57:0/2072966080 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fedd806af60 0x7fedd806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:14.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.070+0000 7fede17fa640 1 -- 172.21.15.57:0/2072966080 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 6065+0+0 (secure 0 0 0) 0x7feddc0890f0 con 0x7fedec0b0530 2023-10-19T14:20:14.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.071+0000 7fedf0898640 1 --2- 172.21.15.57:0/2072966080 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fedd806af60 0x7fedd806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:14.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.071+0000 7fedf0898640 1 --2- 172.21.15.57:0/2072966080 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fedd806af60 0x7fedd806d420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fede8036000 tx=0x7fede8004730 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:14.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.075+0000 7fede17fa640 1 -- 172.21.15.57:0/2072966080 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7feddc053fb0 con 0x7fedec0b0530 2023-10-19T14:20:14.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.250+0000 7fedf2afc640 1 -- 172.21.15.57:0/2072966080 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fedec06ae00 con 0x7fedec0b0530 2023-10-19T14:20:14.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.254+0000 7fede17fa640 1 -- 172.21.15.57:0/2072966080 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v75)=0 v75) v1 ==== 122+0+0 (secure 0 0 0) 0x7feddc0575d0 con 0x7fedec0b0530 2023-10-19T14:20:14.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.256+0000 7fedf2afc640 1 -- 172.21.15.57:0/2072966080 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fedd806af60 msgr2=0x7fedd806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:14.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.256+0000 7fedf2afc640 1 --2- 172.21.15.57:0/2072966080 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fedd806af60 0x7fedd806d420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fede8036000 tx=0x7fede8004730 comp rx=0 tx=0).stop 2023-10-19T14:20:14.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.256+0000 7fedf2afc640 1 -- 172.21.15.57:0/2072966080 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fedec0b0530 msgr2=0x7fedec1562c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:14.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.256+0000 7fedf2afc640 1 --2- 172.21.15.57:0/2072966080 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fedec0b0530 0x7fedec1562c0 secure :-1 s=READY pgs=2314 cs=0 l=1 rev1=1 crypto rx=0x7feddc00c4b0 tx=0x7feddc00c970 comp rx=0 tx=0).stop 2023-10-19T14:20:14.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.256+0000 7fedf2afc640 1 -- 172.21.15.57:0/2072966080 shutdown_connections 2023-10-19T14:20:14.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.256+0000 7fedf2afc640 1 --2- 172.21.15.57:0/2072966080 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fedd806af60 0x7fedd806d420 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:14.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.256+0000 7fedf2afc640 1 --2- 172.21.15.57:0/2072966080 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fedec0cbd20 0x7fedec156800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:14.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.256+0000 7fedf2afc640 1 --2- 172.21.15.57:0/2072966080 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fedec0b0530 0x7fedec1562c0 unknown :-1 s=CLOSED pgs=2314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:14.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.256+0000 7fedf2afc640 1 --2- 172.21.15.57:0/2072966080 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fedec0afa20 0x7fedec13d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:14.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.256+0000 7fedf2afc640 1 -- 172.21.15.57:0/2072966080 >> 172.21.15.57:0/2072966080 conn(0x7fedec069930 msgr2=0x7fedec0c9730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:14.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.256+0000 7fedf2afc640 1 -- 172.21.15.57:0/2072966080 shutdown_connections 2023-10-19T14:20:14.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.256+0000 7fedf2afc640 1 -- 172.21.15.57:0/2072966080 wait complete. 2023-10-19T14:20:14.270 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-19T14:20:14.270 DEBUG:teuthology.orchestra.run.smithi057:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-19T14:20:14.319 DEBUG:teuthology.orchestra.run.smithi179:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-19T14:20:14.369 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-19T14:20:14.370 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T14:20:14.370 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T14:20:14.386 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055614,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=358 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248260k,nr_inodes=812065,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-19T14:20:14.387 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-19T14:20:14.387 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T14:20:14.404 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054833,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=284 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3247632k,nr_inodes=811908,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-19T14:20:14.405 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T14:20:14.405 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T14:20:14.442 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055614,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=358 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248260k,nr_inodes=812065,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-19T14:20:14.443 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-19T14:20:14.443 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T14:20:14.461 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054833,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=284 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3247632k,nr_inodes=811908,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-19T14:20:14.462 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-19T14:20:14.463 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T14:20:14.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.589+0000 7f6c6504b640 1 -- 172.21.15.57:0/1147993011 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6c4c004a80 msgr2=0x7f6c4c004ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:14.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.589+0000 7f6c6504b640 1 --2- 172.21.15.57:0/1147993011 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6c4c004a80 0x7f6c4c004ea0 secure :-1 s=READY pgs=2316 cs=0 l=1 rev1=1 crypto rx=0x7f6c50001680 tx=0x7f6c4c006900 comp rx=0 tx=0).stop 2023-10-19T14:20:14.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.589+0000 7f6c6504b640 1 -- 172.21.15.57:0/1147993011 shutdown_connections 2023-10-19T14:20:14.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.589+0000 7f6c6504b640 1 --2- 172.21.15.57:0/1147993011 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6c4c004a80 0x7f6c4c004ea0 unknown :-1 s=CLOSED pgs=2316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:14.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.589+0000 7f6c6504b640 1 -- 172.21.15.57:0/1147993011 >> 172.21.15.57:0/1147993011 conn(0x7f6c60069930 msgr2=0x7f6c60069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:14.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.589+0000 7f6c6504b640 1 -- 172.21.15.57:0/1147993011 shutdown_connections 2023-10-19T14:20:14.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.589+0000 7f6c6504b640 1 -- 172.21.15.57:0/1147993011 wait complete. 2023-10-19T14:20:14.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.590+0000 7f6c6504b640 1 Processor -- start 2023-10-19T14:20:14.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.590+0000 7f6c6504b640 1 -- start start 2023-10-19T14:20:14.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.591+0000 7f6c6504b640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6c4c004a80 0x7f6c601568b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:14.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.591+0000 7f6c6504b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6c60147c40 0x7f6c60159020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:14.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.591+0000 7f6c6504b640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6c6014a7e0 0x7f6c600aed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:14.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.591+0000 7f6c6504b640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c6014a350 con 0x7f6c60147c40 2023-10-19T14:20:14.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.591+0000 7f6c6504b640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c6015ab80 con 0x7f6c6014a7e0 2023-10-19T14:20:14.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.591+0000 7f6c6504b640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c60159720 con 0x7f6c4c004a80 2023-10-19T14:20:14.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.591+0000 7f6c57fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6c60147c40 0x7f6c60159020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:14.599 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.591+0000 7f6c57fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6c60147c40 0x7f6c60159020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35716/0 (socket says 172.21.15.57:35716) 2023-10-19T14:20:14.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.591+0000 7f6c57fff640 1 -- 172.21.15.57:0/1380064866 learned_addr learned my addr 172.21.15.57:0/1380064866 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:14.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.591+0000 7f6c5ed9d640 1 --2- 172.21.15.57:0/1380064866 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6c4c004a80 0x7f6c601568b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:14.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.591+0000 7f6c5f59e640 1 --2- 172.21.15.57:0/1380064866 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6c6014a7e0 0x7f6c600aed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:14.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.591+0000 7f6c57fff640 1 -- 172.21.15.57:0/1380064866 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6c4c004a80 msgr2=0x7f6c601568b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:14.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.591+0000 7f6c57fff640 1 --2- 172.21.15.57:0/1380064866 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6c4c004a80 0x7f6c601568b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:14.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.591+0000 7f6c57fff640 1 -- 172.21.15.57:0/1380064866 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6c6014a7e0 msgr2=0x7f6c600aed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:14.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.591+0000 7f6c57fff640 1 --2- 172.21.15.57:0/1380064866 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6c6014a7e0 0x7f6c600aed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:14.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.592+0000 7f6c57fff640 1 -- 172.21.15.57:0/1380064866 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c60159a00 con 0x7f6c60147c40 2023-10-19T14:20:14.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.592+0000 7f6c5ed9d640 1 --2- 172.21.15.57:0/1380064866 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6c4c004a80 0x7f6c601568b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:20:14.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.592+0000 7f6c57fff640 1 --2- 172.21.15.57:0/1380064866 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6c60147c40 0x7f6c60159020 secure :-1 s=READY pgs=2317 cs=0 l=1 rev1=1 crypto rx=0x7f6c48011a50 tx=0x7f6c48011f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:14.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.592+0000 7f6c577fe640 1 -- 172.21.15.57:0/1380064866 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c4800fd90 con 0x7f6c60147c40 2023-10-19T14:20:14.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.592+0000 7f6c577fe640 1 -- 172.21.15.57:0/1380064866 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f6c48012450 con 0x7f6c60147c40 2023-10-19T14:20:14.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.592+0000 7f6c6504b640 1 -- 172.21.15.57:0/1380064866 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c60159ce0 con 0x7f6c60147c40 2023-10-19T14:20:14.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.592+0000 7f6c577fe640 1 -- 172.21.15.57:0/1380064866 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c48012800 con 0x7f6c60147c40 2023-10-19T14:20:14.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.592+0000 7f6c6504b640 1 -- 172.21.15.57:0/1380064866 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c6014ac60 con 0x7f6c60147c40 2023-10-19T14:20:14.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.593+0000 7f6c6504b640 1 -- 172.21.15.57:0/1380064866 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c6014a4c0 con 0x7f6c60147c40 2023-10-19T14:20:14.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.593+0000 7f6c577fe640 1 -- 172.21.15.57:0/1380064866 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6c48012960 con 0x7f6c60147c40 2023-10-19T14:20:14.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.594+0000 7f6c577fe640 1 --2- 172.21.15.57:0/1380064866 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f6c4006af90 0x7f6c4006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:14.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.594+0000 7f6c577fe640 1 -- 172.21.15.57:0/1380064866 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 6065+0+0 (secure 0 0 0) 0x7f6c4808a720 con 0x7f6c60147c40 2023-10-19T14:20:14.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.594+0000 7f6c5ed9d640 1 --2- 172.21.15.57:0/1380064866 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f6c4006af90 0x7f6c4006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:14.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.595+0000 7f6c5ed9d640 1 --2- 172.21.15.57:0/1380064866 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f6c4006af90 0x7f6c4006d450 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6c50034000 tx=0x7f6c5002e040 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:14.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.598+0000 7f6c577fe640 1 -- 172.21.15.57:0/1380064866 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6c48055600 con 0x7f6c60147c40 2023-10-19T14:20:14.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.813+0000 7f6c6504b640 1 -- 172.21.15.57:0/1380064866 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6c600690e0 con 0x7f6c60147c40 2023-10-19T14:20:14.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.813+0000 7f6c577fe640 1 -- 172.21.15.57:0/1380064866 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+3706 (secure 0 0 0) 0x7f6c48058c20 con 0x7f6c60147c40 2023-10-19T14:20:14.817 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:20:14.817 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":85,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/2313260224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2313260224},{"type":"v1","addr":"172.21.15.179:6836","nonce":2313260224}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":10081,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/494432208","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":494432208},{"type":"v1","addr":"172.21.15.179:6837","nonce":494432208}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":11214,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/3080658822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":3080658822},{"type":"v1","addr":"172.21.15.57:6834","nonce":3080658822}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":11631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/2572870511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2572870511},{"type":"v1","addr":"172.21.15.57:6835","nonce":2572870511}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85}],"filesystems":[]} 2023-10-19T14:20:14.818 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 85 2023-10-19T14:20:14.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.815+0000 7f6c6504b640 1 -- 172.21.15.57:0/1380064866 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f6c4006af90 msgr2=0x7f6c4006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:14.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.815+0000 7f6c6504b640 1 --2- 172.21.15.57:0/1380064866 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f6c4006af90 0x7f6c4006d450 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6c50034000 tx=0x7f6c5002e040 comp rx=0 tx=0).stop 2023-10-19T14:20:14.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.815+0000 7f6c6504b640 1 -- 172.21.15.57:0/1380064866 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6c60147c40 msgr2=0x7f6c60159020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:14.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.815+0000 7f6c6504b640 1 --2- 172.21.15.57:0/1380064866 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6c60147c40 0x7f6c60159020 secure :-1 s=READY pgs=2317 cs=0 l=1 rev1=1 crypto rx=0x7f6c48011a50 tx=0x7f6c48011f10 comp rx=0 tx=0).stop 2023-10-19T14:20:14.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.815+0000 7f6c6504b640 1 -- 172.21.15.57:0/1380064866 shutdown_connections 2023-10-19T14:20:14.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.815+0000 7f6c6504b640 1 --2- 172.21.15.57:0/1380064866 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f6c4006af90 0x7f6c4006d450 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:14.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.815+0000 7f6c6504b640 1 --2- 172.21.15.57:0/1380064866 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6c6014a7e0 0x7f6c600aed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:14.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.815+0000 7f6c6504b640 1 --2- 172.21.15.57:0/1380064866 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6c60147c40 0x7f6c60159020 unknown :-1 s=CLOSED pgs=2317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:14.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.815+0000 7f6c6504b640 1 --2- 172.21.15.57:0/1380064866 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6c4c004a80 0x7f6c601568b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:14.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.815+0000 7f6c6504b640 1 -- 172.21.15.57:0/1380064866 >> 172.21.15.57:0/1380064866 conn(0x7f6c60069930 msgr2=0x7f6c6006b9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:14.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.815+0000 7f6c6504b640 1 -- 172.21.15.57:0/1380064866 shutdown_connections 2023-10-19T14:20:14.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:14.815+0000 7f6c6504b640 1 -- 172.21.15.57:0/1380064866 wait complete. 2023-10-19T14:20:14.829 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-19T14:20:14.829 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-19T14:20:14.829 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-19T14:20:14.830 INFO:teuthology.orchestra.run:waiting for 300 2023-10-19T14:20:14.830 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-19T14:20:14.830 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-19T14:20:14.831 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-19T14:20:14.831 INFO:teuthology.orchestra.run:waiting for 300 2023-10-19T14:20:14.831 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-19T14:20:14.831 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-19T14:20:14.832 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-19T14:20:14.832 INFO:teuthology.orchestra.run:waiting for 300 2023-10-19T14:20:14.832 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-19T14:20:14.833 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-19T14:20:14.833 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-19T14:20:14.833 INFO:teuthology.orchestra.run:waiting for 300 2023-10-19T14:20:14.838 INFO:tasks.ceph.mds.a:Stopped 2023-10-19T14:20:14.839 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-19T14:20:14.840 INFO:tasks.ceph.mds.c:Stopped 2023-10-19T14:20:14.840 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-19T14:20:14.847 INFO:tasks.ceph.mds.b:Stopped 2023-10-19T14:20:14.847 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-19T14:20:14.848 INFO:tasks.ceph.mds.d:Stopped 2023-10-19T14:20:14.848 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-19T14:20:14.883 INFO:tasks.ceph.mds.a:Started 2023-10-19T14:20:14.884 DEBUG:teuthology.parallel:result is None 2023-10-19T14:20:14.885 INFO:tasks.ceph.mds.c:Started 2023-10-19T14:20:14.886 DEBUG:teuthology.parallel:result is None 2023-10-19T14:20:14.890 INFO:tasks.ceph.mds.b:Started 2023-10-19T14:20:14.890 DEBUG:teuthology.parallel:result is None 2023-10-19T14:20:14.891 INFO:tasks.ceph.mds.d:Started 2023-10-19T14:20:14.891 DEBUG:teuthology.parallel:result is None 2023-10-19T14:20:14.892 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-19T14:20:14.963 INFO:tasks.ceph.mds.c.smithi057.stdout:starting mds.c at 2023-10-19T14:20:14.963 INFO:tasks.ceph.mds.a.smithi057.stdout:starting mds.a at 2023-10-19T14:20:14.972 INFO:tasks.ceph.mds.b.smithi179.stdout:starting mds.b at 2023-10-19T14:20:14.973 INFO:tasks.ceph.mds.d.smithi179.stdout:starting mds.d at 2023-10-19T14:20:15.006 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.003+0000 7ffad4368640 1 -- 172.21.15.57:0/3375076464 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ffacc15b3e0 msgr2=0x7ffacc0d1810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:15.006 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.003+0000 7ffad4368640 1 --2- 172.21.15.57:0/3375076464 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ffacc15b3e0 0x7ffacc0d1810 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7ffac4009be0 tx=0x7ffac402c160 comp rx=0 tx=0).stop 2023-10-19T14:20:15.006 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.003+0000 7ffad4368640 1 -- 172.21.15.57:0/3375076464 shutdown_connections 2023-10-19T14:20:15.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.003+0000 7ffad4368640 1 --2- 172.21.15.57:0/3375076464 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ffacc15b3e0 0x7ffacc0d1810 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:15.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.003+0000 7ffad4368640 1 -- 172.21.15.57:0/3375076464 >> 172.21.15.57:0/3375076464 conn(0x7ffacc0d0cf0 msgr2=0x7ffacc0d1100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:15.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.003+0000 7ffad4368640 1 -- 172.21.15.57:0/3375076464 shutdown_connections 2023-10-19T14:20:15.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.003+0000 7ffad4368640 1 -- 172.21.15.57:0/3375076464 wait complete. 2023-10-19T14:20:15.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.004+0000 7ffad4368640 1 Processor -- start 2023-10-19T14:20:15.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.004+0000 7ffad4368640 1 -- start start 2023-10-19T14:20:15.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.005+0000 7ffad4368640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffacc0cf9e0 0x7ffacc151b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:15.010 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.005+0000 7ffad4368640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffacc15b000 0x7ffacc15b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:15.010 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.005+0000 7ffad4368640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ffacc15b3e0 0x7ffacc0c6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:15.010 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.005+0000 7ffad4368640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffacc0c9c10 con 0x7ffacc15b000 2023-10-19T14:20:15.010 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.005+0000 7ffad4368640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffacc0c7dc0 con 0x7ffacc0cf9e0 2023-10-19T14:20:15.010 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.005+0000 7ffad4368640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffacc0c90d0 con 0x7ffacc15b3e0 2023-10-19T14:20:15.011 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.005+0000 7ffad1903640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffacc15b000 0x7ffacc15b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:15.011 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.005+0000 7ffad1903640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffacc15b000 0x7ffacc15b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35766/0 (socket says 172.21.15.57:35766) 2023-10-19T14:20:15.011 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.005+0000 7ffad1903640 1 -- 172.21.15.57:0/830873787 learned_addr learned my addr 172.21.15.57:0/830873787 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:15.011 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.005+0000 7ffad2104640 1 --2- 172.21.15.57:0/830873787 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffacc0cf9e0 0x7ffacc151b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:15.011 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.005+0000 7ffad2905640 1 --2- 172.21.15.57:0/830873787 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ffacc15b3e0 0x7ffacc0c6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:15.011 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.005+0000 7ffad1903640 1 -- 172.21.15.57:0/830873787 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ffacc15b3e0 msgr2=0x7ffacc0c6070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:15.012 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.005+0000 7ffad1903640 1 --2- 172.21.15.57:0/830873787 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ffacc15b3e0 0x7ffacc0c6070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:15.012 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.005+0000 7ffad1903640 1 -- 172.21.15.57:0/830873787 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffacc0cf9e0 msgr2=0x7ffacc151b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:15.012 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.005+0000 7ffad1903640 1 --2- 172.21.15.57:0/830873787 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffacc0cf9e0 0x7ffacc151b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:15.012 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.005+0000 7ffad1903640 1 -- 172.21.15.57:0/830873787 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffac4009840 con 0x7ffacc15b000 2023-10-19T14:20:15.012 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.005+0000 7ffad2905640 1 --2- 172.21.15.57:0/830873787 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ffacc15b3e0 0x7ffacc0c6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:20:15.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.006+0000 7ffad1903640 1 --2- 172.21.15.57:0/830873787 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffacc15b000 0x7ffacc15b9d0 secure :-1 s=READY pgs=2323 cs=0 l=1 rev1=1 crypto rx=0x7ffac800c490 tx=0x7ffac800c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:15.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.006+0000 7ffac2ffd640 1 -- 172.21.15.57:0/830873787 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffac80052d0 con 0x7ffacc15b000 2023-10-19T14:20:15.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.006+0000 7ffac2ffd640 1 -- 172.21.15.57:0/830873787 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7ffac8005430 con 0x7ffacc15b000 2023-10-19T14:20:15.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.006+0000 7ffad4368640 1 -- 172.21.15.57:0/830873787 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffacc0c9370 con 0x7ffacc15b000 2023-10-19T14:20:15.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.006+0000 7ffac2ffd640 1 -- 172.21.15.57:0/830873787 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffac8012890 con 0x7ffacc15b000 2023-10-19T14:20:15.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.006+0000 7ffad4368640 1 -- 172.21.15.57:0/830873787 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffacc0c9930 con 0x7ffacc15b000 2023-10-19T14:20:15.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.007+0000 7ffad4368640 1 -- 172.21.15.57:0/830873787 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffacc0c7f30 con 0x7ffacc15b000 2023-10-19T14:20:15.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.007+0000 7ffac2ffd640 1 -- 172.21.15.57:0/830873787 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ffac80035f0 con 0x7ffacc15b000 2023-10-19T14:20:15.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.008+0000 7ffac2ffd640 1 --2- 172.21.15.57:0/830873787 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7ffab006af60 0x7ffab006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:15.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.008+0000 7ffac2ffd640 1 -- 172.21.15.57:0/830873787 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 6065+0+0 (secure 0 0 0) 0x7ffac8089660 con 0x7ffacc15b000 2023-10-19T14:20:15.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.008+0000 7ffad2104640 1 --2- 172.21.15.57:0/830873787 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7ffab006af60 0x7ffab006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:15.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.009+0000 7ffad2104640 1 --2- 172.21.15.57:0/830873787 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7ffab006af60 0x7ffab006d420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ffabc0011e0 tx=0x7ffabc007d20 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:15.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.012+0000 7ffac2ffd640 1 -- 172.21.15.57:0/830873787 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ffac8054400 con 0x7ffacc15b000 2023-10-19T14:20:15.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.201+0000 7ffad4368640 1 -- 172.21.15.57:0/830873787 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7ffacc15a860 con 0x7ffacc15b000 2023-10-19T14:20:15.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.673+0000 7ffac2ffd640 1 -- 172.21.15.57:0/830873787 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v217) v1 ==== 97+0+0 (secure 0 0 0) 0x7ffac8057a20 con 0x7ffacc15b000 2023-10-19T14:20:15.676 INFO:teuthology.orchestra.run.smithi057.stderr: removed all blocklist entries 2023-10-19T14:20:15.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.675+0000 7ffad4368640 1 -- 172.21.15.57:0/830873787 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7ffab006af60 msgr2=0x7ffab006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:15.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.675+0000 7ffad4368640 1 --2- 172.21.15.57:0/830873787 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7ffab006af60 0x7ffab006d420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ffabc0011e0 tx=0x7ffabc007d20 comp rx=0 tx=0).stop 2023-10-19T14:20:15.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.675+0000 7ffad4368640 1 -- 172.21.15.57:0/830873787 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffacc15b000 msgr2=0x7ffacc15b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:15.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.675+0000 7ffad4368640 1 --2- 172.21.15.57:0/830873787 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffacc15b000 0x7ffacc15b9d0 secure :-1 s=READY pgs=2323 cs=0 l=1 rev1=1 crypto rx=0x7ffac800c490 tx=0x7ffac800c950 comp rx=0 tx=0).stop 2023-10-19T14:20:15.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.675+0000 7ffad4368640 1 -- 172.21.15.57:0/830873787 shutdown_connections 2023-10-19T14:20:15.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.675+0000 7ffad4368640 1 --2- 172.21.15.57:0/830873787 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7ffab006af60 0x7ffab006d420 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:15.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.675+0000 7ffad4368640 1 --2- 172.21.15.57:0/830873787 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ffacc15b3e0 0x7ffacc0c6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:15.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.675+0000 7ffad4368640 1 --2- 172.21.15.57:0/830873787 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffacc15b000 0x7ffacc15b9d0 unknown :-1 s=CLOSED pgs=2323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:15.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.675+0000 7ffad4368640 1 --2- 172.21.15.57:0/830873787 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffacc0cf9e0 0x7ffacc151b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:15.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.675+0000 7ffad4368640 1 -- 172.21.15.57:0/830873787 >> 172.21.15.57:0/830873787 conn(0x7ffacc0d0cf0 msgr2=0x7ffacc06b590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:15.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.675+0000 7ffad4368640 1 -- 172.21.15.57:0/830873787 shutdown_connections 2023-10-19T14:20:15.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.675+0000 7ffad4368640 1 -- 172.21.15.57:0/830873787 wait complete. 2023-10-19T14:20:15.691 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-19T14:20:15.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.799+0000 7fb8a7c90640 1 -- 172.21.15.57:0/1539255510 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8a0070af0 msgr2=0x7fb8a00affe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:15.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.799+0000 7fb8a7c90640 1 --2- 172.21.15.57:0/1539255510 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8a0070af0 0x7fb8a00affe0 secure :-1 s=READY pgs=2324 cs=0 l=1 rev1=1 crypto rx=0x7fb890005fd0 tx=0x7fb89002c1b0 comp rx=0 tx=0).stop 2023-10-19T14:20:15.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.799+0000 7fb8a7c90640 1 -- 172.21.15.57:0/1539255510 shutdown_connections 2023-10-19T14:20:15.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.799+0000 7fb8a7c90640 1 --2- 172.21.15.57:0/1539255510 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8a0070af0 0x7fb8a00affe0 unknown :-1 s=CLOSED pgs=2324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:15.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.799+0000 7fb8a7c90640 1 --2- 172.21.15.57:0/1539255510 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb8a013e520 0x7fb8a006bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:15.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.799+0000 7fb8a7c90640 1 -- 172.21.15.57:0/1539255510 >> 172.21.15.57:0/1539255510 conn(0x7fb8a0069930 msgr2=0x7fb8a0069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:15.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.799+0000 7fb8a7c90640 1 -- 172.21.15.57:0/1539255510 shutdown_connections 2023-10-19T14:20:15.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.799+0000 7fb8a7c90640 1 -- 172.21.15.57:0/1539255510 wait complete. 2023-10-19T14:20:15.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.800+0000 7fb8a7c90640 1 Processor -- start 2023-10-19T14:20:15.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.800+0000 7fb8a7c90640 1 -- start start 2023-10-19T14:20:15.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.801+0000 7fb8a7c90640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb8a0070af0 0x7fb8a00ae0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:15.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.801+0000 7fb8a7c90640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb8a013e520 0x7fb8a00a9c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:15.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.801+0000 7fb8a7c90640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8a0141690 0x7fb8a00ab000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:15.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.801+0000 7fb8a7c90640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a00a8bc0 con 0x7fb8a0141690 2023-10-19T14:20:15.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.801+0000 7fb8a7c90640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a00a8ea0 con 0x7fb8a013e520 2023-10-19T14:20:15.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.801+0000 7fb8a7c90640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a00a9180 con 0x7fb8a0070af0 2023-10-19T14:20:15.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.801+0000 7fb8a622d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8a0141690 0x7fb8a00ab000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:15.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.801+0000 7fb8a622d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8a0141690 0x7fb8a00ab000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35776/0 (socket says 172.21.15.57:35776) 2023-10-19T14:20:15.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.801+0000 7fb8a622d640 1 -- 172.21.15.57:0/1661255324 learned_addr learned my addr 172.21.15.57:0/1661255324 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:15.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.801+0000 7fb8a522b640 1 --2- 172.21.15.57:0/1661255324 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb8a013e520 0x7fb8a00a9c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:15.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.801+0000 7fb8a5a2c640 1 --2- 172.21.15.57:0/1661255324 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb8a0070af0 0x7fb8a00ae0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:15.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.801+0000 7fb8a622d640 1 -- 172.21.15.57:0/1661255324 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb8a0070af0 msgr2=0x7fb8a00ae0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:15.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.802+0000 7fb8a622d640 1 --2- 172.21.15.57:0/1661255324 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb8a0070af0 0x7fb8a00ae0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:15.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.802+0000 7fb8a622d640 1 -- 172.21.15.57:0/1661255324 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb8a013e520 msgr2=0x7fb8a00a9c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:15.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.802+0000 7fb8a622d640 1 --2- 172.21.15.57:0/1661255324 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb8a013e520 0x7fb8a00a9c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:15.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.802+0000 7fb8a622d640 1 -- 172.21.15.57:0/1661255324 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb890009900 con 0x7fb8a0141690 2023-10-19T14:20:15.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.802+0000 7fb8a622d640 1 --2- 172.21.15.57:0/1661255324 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8a0141690 0x7fb8a00ab000 secure :-1 s=READY pgs=2325 cs=0 l=1 rev1=1 crypto rx=0x7fb890037040 tx=0x7fb890035db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:15.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.802+0000 7fb89e7fc640 1 -- 172.21.15.57:0/1661255324 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb890034440 con 0x7fb8a0141690 2023-10-19T14:20:15.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.802+0000 7fb8a7c90640 1 -- 172.21.15.57:0/1661255324 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8a00a9460 con 0x7fb8a0141690 2023-10-19T14:20:15.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.802+0000 7fb89e7fc640 1 -- 172.21.15.57:0/1661255324 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fb890004540 con 0x7fb8a0141690 2023-10-19T14:20:15.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.802+0000 7fb89e7fc640 1 -- 172.21.15.57:0/1661255324 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb89003f440 con 0x7fb8a0141690 2023-10-19T14:20:15.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.802+0000 7fb8a7c90640 1 -- 172.21.15.57:0/1661255324 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8a01d5520 con 0x7fb8a0141690 2023-10-19T14:20:15.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.803+0000 7fb8a7c90640 1 -- 172.21.15.57:0/1661255324 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8a00a9980 con 0x7fb8a0141690 2023-10-19T14:20:15.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.803+0000 7fb89e7fc640 1 -- 172.21.15.57:0/1661255324 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb89003b040 con 0x7fb8a0141690 2023-10-19T14:20:15.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.804+0000 7fb89e7fc640 1 --2- 172.21.15.57:0/1661255324 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fb88c06af60 0x7fb88c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:15.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.804+0000 7fb89e7fc640 1 -- 172.21.15.57:0/1661255324 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 5205+0+0 (secure 0 0 0) 0x7fb8900ac360 con 0x7fb8a0141690 2023-10-19T14:20:15.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.804+0000 7fb8a5a2c640 1 --2- 172.21.15.57:0/1661255324 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fb88c06af60 0x7fb88c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:15.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.805+0000 7fb8a5a2c640 1 --2- 172.21.15.57:0/1661255324 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fb88c06af60 0x7fb88c06d420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb89800abe0 tx=0x7fb89800a2d0 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:15.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:15.808+0000 7fb89e7fc640 1 -- 172.21.15.57:0/1661255324 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb8a00a9980 con 0x7fb8a0141690 2023-10-19T14:20:16.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.047+0000 7fb8a7c90640 1 -- 172.21.15.57:0/1661255324 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fb8a00a9980 con 0x7fb8a0141690 2023-10-19T14:20:16.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.048+0000 7fb89e7fc640 1 -- 172.21.15.57:0/1661255324 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v59) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fb8a00a9980 con 0x7fb8a0141690 2023-10-19T14:20:16.051 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:20:16.051 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2023-10-19T14:20:16.052 INFO:teuthology.orchestra.run.smithi057.stdout: "auth_dump": [ 2023-10-19T14:20:16.052 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:20:16.052 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "mds.a", 2023-10-19T14:20:16.052 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBjJDFlmbVANhAAP/ynHlU/OxaRq5lQoAa8Ng==", 2023-10-19T14:20:16.052 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:20:16.053 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T14:20:16.053 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow *", 2023-10-19T14:20:16.053 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow *", 2023-10-19T14:20:16.053 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:20:16.053 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:20:16.054 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:20:16.054 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:20:16.054 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "mds.b", 2023-10-19T14:20:16.054 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFl9l7RBxAAC0geI2tJ1SIZZr1++kS/BA==", 2023-10-19T14:20:16.054 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:20:16.055 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T14:20:16.055 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow *", 2023-10-19T14:20:16.055 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow *", 2023-10-19T14:20:16.055 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:20:16.055 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:20:16.055 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:20:16.056 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:20:16.056 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "mds.c", 2023-10-19T14:20:16.056 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFlFkaoAhAA/PBizkJ4bm958oP4CmPuIQ==", 2023-10-19T14:20:16.056 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:20:16.056 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T14:20:16.057 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow *", 2023-10-19T14:20:16.057 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow *", 2023-10-19T14:20:16.057 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:20:16.057 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:20:16.057 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:20:16.058 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:20:16.058 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "mds.d", 2023-10-19T14:20:16.058 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFlx8skDxAAAfCHTYtYJgJik0WHSWeQfQ==", 2023-10-19T14:20:16.058 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:20:16.058 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T14:20:16.059 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow *", 2023-10-19T14:20:16.059 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow *", 2023-10-19T14:20:16.059 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:20:16.059 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:20:16.059 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:20:16.059 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:20:16.060 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.0", 2023-10-19T14:20:16.060 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBrJDFlCo9sBxAAQN2cID4nCQgRwGaGFEV8LQ==", 2023-10-19T14:20:16.060 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:20:16.060 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T14:20:16.060 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T14:20:16.060 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:20:16.061 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:20:16.061 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:20:16.061 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:20:16.061 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.1", 2023-10-19T14:20:16.061 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBuJDFl4fsLCBAAp3Pi80yZtc5sju0GisEuVA==", 2023-10-19T14:20:16.062 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:20:16.062 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T14:20:16.062 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T14:20:16.062 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:20:16.062 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:20:16.063 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:20:16.063 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:20:16.063 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.2", 2023-10-19T14:20:16.063 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBxJDFl+KAKCBAAd4N9cluARxPVfBLU+7Hd7A==", 2023-10-19T14:20:16.063 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:20:16.064 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T14:20:16.064 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T14:20:16.064 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:20:16.064 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:20:16.064 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:20:16.064 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:20:16.065 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.3", 2023-10-19T14:20:16.065 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQB0JDFlU/toCBAAP0Ou1zcHKNx5SfbybJkh4Q==", 2023-10-19T14:20:16.065 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:20:16.065 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T14:20:16.065 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T14:20:16.066 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:20:16.066 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:20:16.066 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:20:16.066 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:20:16.066 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.4", 2023-10-19T14:20:16.067 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQB6JDFlc6stEBAALkgNyxoiCTzZnKpBSPxI6w==", 2023-10-19T14:20:16.067 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:20:16.067 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T14:20:16.067 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T14:20:16.067 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:20:16.068 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:20:16.068 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:20:16.068 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:20:16.068 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.5", 2023-10-19T14:20:16.068 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQB9JDFlI3cNDhAAuG3tHJ6RvPwAzPE76wTIVQ==", 2023-10-19T14:20:16.068 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:20:16.069 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T14:20:16.069 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T14:20:16.069 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:20:16.069 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:20:16.069 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:20:16.070 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:20:16.070 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.6", 2023-10-19T14:20:16.070 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQCAJDFlLPumCxAA/iy08t/2N9rXfn71KtbSQw==", 2023-10-19T14:20:16.070 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:20:16.070 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T14:20:16.071 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T14:20:16.071 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:20:16.071 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:20:16.071 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:20:16.071 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:20:16.072 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.7", 2023-10-19T14:20:16.072 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQCDJDFlHmrbCRAA+hpWygGIJydfU6cHWlfx/Q==", 2023-10-19T14:20:16.072 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:20:16.072 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T14:20:16.072 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T14:20:16.073 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:20:16.073 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:20:16.073 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:20:16.073 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:20:16.073 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "client.0", 2023-10-19T14:20:16.073 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFlg9yuExAATxiWDfS+p07G1F0pORbJEg==", 2023-10-19T14:20:16.074 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:20:16.074 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T14:20:16.074 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow r", 2023-10-19T14:20:16.074 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-19T14:20:16.074 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:20:16.075 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:20:16.075 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:20:16.075 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "client.1", 2023-10-19T14:20:16.075 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFl9IwYGxAAHPituiJSPkHDuD0QyG4Hww==", 2023-10-19T14:20:16.075 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:20:16.076 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T14:20:16.076 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow r", 2023-10-19T14:20:16.076 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow rw", 2023-10-19T14:20:16.076 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow rwx" 2023-10-19T14:20:16.076 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:20:16.076 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:20:16.077 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:20:16.077 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "client.2", 2023-10-19T14:20:16.077 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFlMd+AFxAAS/UTESDt76GmrhhN2o/Fdw==", 2023-10-19T14:20:16.077 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:20:16.077 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T14:20:16.078 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow r", 2023-10-19T14:20:16.078 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow rw", 2023-10-19T14:20:16.078 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow rwx" 2023-10-19T14:20:16.078 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:20:16.078 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:20:16.079 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:20:16.079 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "client.3", 2023-10-19T14:20:16.079 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFlHZ3fIBAA42B7n8opLS5smXIwJyVYzw==", 2023-10-19T14:20:16.079 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:20:16.079 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T14:20:16.079 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow r", 2023-10-19T14:20:16.080 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow rw", 2023-10-19T14:20:16.080 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow rwx" 2023-10-19T14:20:16.080 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:20:16.080 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:20:16.080 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:20:16.081 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "client.admin", 2023-10-19T14:20:16.081 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBjJDFlBgasFxAAMA24j8KszcXMEpYRQDXj4g==", 2023-10-19T14:20:16.081 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:20:16.081 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow *", 2023-10-19T14:20:16.081 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow *", 2023-10-19T14:20:16.082 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow *", 2023-10-19T14:20:16.082 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:20:16.082 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:20:16.082 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:20:16.082 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:20:16.083 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "mgr.x", 2023-10-19T14:20:16.083 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBjJDFllFBfMhAA5+KPXWmYZqcESmLObaUbBA==", 2023-10-19T14:20:16.083 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:20:16.083 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow *", 2023-10-19T14:20:16.083 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile mgr", 2023-10-19T14:20:16.084 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:20:16.084 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:20:16.084 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:20:16.084 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:20:16.084 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "mgr.y", 2023-10-19T14:20:16.085 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBjJDFlxRzJLRAAGjG58o7qXAHvIEIRYwl6QA==", 2023-10-19T14:20:16.085 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:20:16.085 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow *", 2023-10-19T14:20:16.085 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile mgr", 2023-10-19T14:20:16.085 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:20:16.085 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:20:16.086 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:20:16.086 INFO:teuthology.orchestra.run.smithi057.stdout: ] 2023-10-19T14:20:16.086 INFO:teuthology.orchestra.run.smithi057.stdout:} 2023-10-19T14:20:16.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.050+0000 7fb8a7c90640 1 -- 172.21.15.57:0/1661255324 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fb88c06af60 msgr2=0x7fb88c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:16.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.050+0000 7fb8a7c90640 1 --2- 172.21.15.57:0/1661255324 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fb88c06af60 0x7fb88c06d420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb89800abe0 tx=0x7fb89800a2d0 comp rx=0 tx=0).stop 2023-10-19T14:20:16.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.050+0000 7fb8a7c90640 1 -- 172.21.15.57:0/1661255324 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8a0141690 msgr2=0x7fb8a00ab000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:16.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.050+0000 7fb8a7c90640 1 --2- 172.21.15.57:0/1661255324 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8a0141690 0x7fb8a00ab000 secure :-1 s=READY pgs=2325 cs=0 l=1 rev1=1 crypto rx=0x7fb890037040 tx=0x7fb890035db0 comp rx=0 tx=0).stop 2023-10-19T14:20:16.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.050+0000 7fb8a7c90640 1 -- 172.21.15.57:0/1661255324 shutdown_connections 2023-10-19T14:20:16.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.050+0000 7fb8a7c90640 1 --2- 172.21.15.57:0/1661255324 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fb88c06af60 0x7fb88c06d420 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:16.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.050+0000 7fb8a7c90640 1 --2- 172.21.15.57:0/1661255324 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8a0141690 0x7fb8a00ab000 unknown :-1 s=CLOSED pgs=2325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:16.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.050+0000 7fb8a7c90640 1 --2- 172.21.15.57:0/1661255324 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb8a013e520 0x7fb8a00a9c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:16.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.050+0000 7fb8a7c90640 1 --2- 172.21.15.57:0/1661255324 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb8a0070af0 0x7fb8a00ae0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:16.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.050+0000 7fb8a7c90640 1 -- 172.21.15.57:0/1661255324 >> 172.21.15.57:0/1661255324 conn(0x7fb8a0069930 msgr2=0x7fb8a0069d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:16.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.050+0000 7fb8a7c90640 1 -- 172.21.15.57:0/1661255324 shutdown_connections 2023-10-19T14:20:16.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.050+0000 7fb8a7c90640 1 -- 172.21.15.57:0/1661255324 wait complete. 2023-10-19T14:20:16.090 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T14:20:16.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.197+0000 7fc2d2cd7640 1 -- 172.21.15.57:0/2615580913 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc2cc13d350 msgr2=0x7fc2cc0cbd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:16.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.197+0000 7fc2d2cd7640 1 --2- 172.21.15.57:0/2615580913 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc2cc13d350 0x7fc2cc0cbd60 secure :-1 s=READY pgs=2327 cs=0 l=1 rev1=1 crypto rx=0x7fc2b8009830 tx=0x7fc2b802bd60 comp rx=0 tx=0).stop 2023-10-19T14:20:16.201 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.197+0000 7fc2d2cd7640 1 -- 172.21.15.57:0/2615580913 shutdown_connections 2023-10-19T14:20:16.201 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.197+0000 7fc2d2cd7640 1 --2- 172.21.15.57:0/2615580913 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc2cc13d350 0x7fc2cc0cbd60 unknown :-1 s=CLOSED pgs=2327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:16.201 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.197+0000 7fc2d2cd7640 1 --2- 172.21.15.57:0/2615580913 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc2cc0cb980 0x7fc2cc13db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:16.201 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.197+0000 7fc2d2cd7640 1 -- 172.21.15.57:0/2615580913 >> 172.21.15.57:0/2615580913 conn(0x7fc2cc069930 msgr2=0x7fc2cc069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:16.201 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.197+0000 7fc2d2cd7640 1 -- 172.21.15.57:0/2615580913 shutdown_connections 2023-10-19T14:20:16.202 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.197+0000 7fc2d2cd7640 1 -- 172.21.15.57:0/2615580913 wait complete. 2023-10-19T14:20:16.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.198+0000 7fc2d2cd7640 1 Processor -- start 2023-10-19T14:20:16.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.198+0000 7fc2d2cd7640 1 -- start start 2023-10-19T14:20:16.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.199+0000 7fc2d2cd7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc2cc0c96b0 0x7fc2cc14df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:16.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.199+0000 7fc2d2cd7640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc2cc0cb980 0x7fc2cc155440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:16.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.199+0000 7fc2d2cd7640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc2cc156d20 0x7fc2cc1571b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:16.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.199+0000 7fc2d2cd7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2cc1576f0 con 0x7fc2cc0c96b0 2023-10-19T14:20:16.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.199+0000 7fc2d2cd7640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2cc1540c0 con 0x7fc2cc156d20 2023-10-19T14:20:16.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.199+0000 7fc2d2cd7640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2cc1543a0 con 0x7fc2cc0cb980 2023-10-19T14:20:16.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.199+0000 7fc2d0a73640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc2cc0c96b0 0x7fc2cc14df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:16.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.199+0000 7fc2d0a73640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc2cc0c96b0 0x7fc2cc14df20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35798/0 (socket says 172.21.15.57:35798) 2023-10-19T14:20:16.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.199+0000 7fc2d0a73640 1 -- 172.21.15.57:0/548683720 learned_addr learned my addr 172.21.15.57:0/548683720 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:16.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.199+0000 7fc2cbfff640 1 --2- 172.21.15.57:0/548683720 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc2cc0cb980 0x7fc2cc155440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:16.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.199+0000 7fc2d1274640 1 --2- 172.21.15.57:0/548683720 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc2cc156d20 0x7fc2cc1571b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:16.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.199+0000 7fc2d0a73640 1 -- 172.21.15.57:0/548683720 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc2cc0cb980 msgr2=0x7fc2cc155440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:16.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.199+0000 7fc2d0a73640 1 --2- 172.21.15.57:0/548683720 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc2cc0cb980 0x7fc2cc155440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:16.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.199+0000 7fc2d0a73640 1 -- 172.21.15.57:0/548683720 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc2cc156d20 msgr2=0x7fc2cc1571b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:16.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.199+0000 7fc2d0a73640 1 --2- 172.21.15.57:0/548683720 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc2cc156d20 0x7fc2cc1571b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:16.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.199+0000 7fc2d0a73640 1 -- 172.21.15.57:0/548683720 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2b8009490 con 0x7fc2cc0c96b0 2023-10-19T14:20:16.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.199+0000 7fc2d0a73640 1 --2- 172.21.15.57:0/548683720 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc2cc0c96b0 0x7fc2cc14df20 secure :-1 s=READY pgs=2328 cs=0 l=1 rev1=1 crypto rx=0x7fc2c000d950 tx=0x7fc2c000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:16.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.200+0000 7fc2c97fa640 1 -- 172.21.15.57:0/548683720 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2c0009c30 con 0x7fc2cc0c96b0 2023-10-19T14:20:16.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.200+0000 7fc2c97fa640 1 -- 172.21.15.57:0/548683720 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fc2c0009e80 con 0x7fc2cc0c96b0 2023-10-19T14:20:16.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.200+0000 7fc2d2cd7640 1 -- 172.21.15.57:0/548683720 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2cc154680 con 0x7fc2cc0c96b0 2023-10-19T14:20:16.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.200+0000 7fc2c97fa640 1 -- 172.21.15.57:0/548683720 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2c00106c0 con 0x7fc2cc0c96b0 2023-10-19T14:20:16.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.200+0000 7fc2d2cd7640 1 -- 172.21.15.57:0/548683720 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2cc154c40 con 0x7fc2cc0c96b0 2023-10-19T14:20:16.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.201+0000 7fc2d2cd7640 1 -- 172.21.15.57:0/548683720 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2cc0690e0 con 0x7fc2cc0c96b0 2023-10-19T14:20:16.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.201+0000 7fc2c97fa640 1 -- 172.21.15.57:0/548683720 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc2c0010820 con 0x7fc2cc0c96b0 2023-10-19T14:20:16.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.202+0000 7fc2c97fa640 1 --2- 172.21.15.57:0/548683720 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fc2b006af60 0x7fc2b006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:16.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.202+0000 7fc2c97fa640 1 -- 172.21.15.57:0/548683720 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 5205+0+0 (secure 0 0 0) 0x7fc2c0089c00 con 0x7fc2cc0c96b0 2023-10-19T14:20:16.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.202+0000 7fc2cbfff640 1 --2- 172.21.15.57:0/548683720 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fc2b006af60 0x7fc2b006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:16.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.202+0000 7fc2cbfff640 1 --2- 172.21.15.57:0/548683720 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fc2b006af60 0x7fc2b006d420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc2b8036000 tx=0x7fc2b8004730 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:16.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.206+0000 7fc2c97fa640 1 -- 172.21.15.57:0/548683720 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc2c0058700 con 0x7fc2cc0c96b0 2023-10-19T14:20:16.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.382+0000 7fc2d2cd7640 1 -- 172.21.15.57:0/548683720 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc2cc15bf00 con 0x7fc2cc0c96b0 2023-10-19T14:20:16.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.383+0000 7fc2c97fa640 1 -- 172.21.15.57:0/548683720 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v219) v1 ==== 74+0+13127 (secure 0 0 0) 0x7fc2c0017090 con 0x7fc2cc0c96b0 2023-10-19T14:20:16.386 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:20:16.387 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":219,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T14:20:15.691165+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T14:20:16.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.385+0000 7fc2d2cd7640 1 -- 172.21.15.57:0/548683720 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fc2b006af60 msgr2=0x7fc2b006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:16.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.385+0000 7fc2d2cd7640 1 --2- 172.21.15.57:0/548683720 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fc2b006af60 0x7fc2b006d420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc2b8036000 tx=0x7fc2b8004730 comp rx=0 tx=0).stop 2023-10-19T14:20:16.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.385+0000 7fc2d2cd7640 1 -- 172.21.15.57:0/548683720 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc2cc0c96b0 msgr2=0x7fc2cc14df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:16.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.385+0000 7fc2d2cd7640 1 --2- 172.21.15.57:0/548683720 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc2cc0c96b0 0x7fc2cc14df20 secure :-1 s=READY pgs=2328 cs=0 l=1 rev1=1 crypto rx=0x7fc2c000d950 tx=0x7fc2c000de10 comp rx=0 tx=0).stop 2023-10-19T14:20:16.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.385+0000 7fc2d2cd7640 1 -- 172.21.15.57:0/548683720 shutdown_connections 2023-10-19T14:20:16.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.385+0000 7fc2d2cd7640 1 --2- 172.21.15.57:0/548683720 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fc2b006af60 0x7fc2b006d420 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:16.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.385+0000 7fc2d2cd7640 1 --2- 172.21.15.57:0/548683720 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc2cc156d20 0x7fc2cc1571b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:16.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.385+0000 7fc2d2cd7640 1 --2- 172.21.15.57:0/548683720 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc2cc0cb980 0x7fc2cc155440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:16.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.385+0000 7fc2d2cd7640 1 --2- 172.21.15.57:0/548683720 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc2cc0c96b0 0x7fc2cc14df20 unknown :-1 s=CLOSED pgs=2328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:16.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.385+0000 7fc2d2cd7640 1 -- 172.21.15.57:0/548683720 >> 172.21.15.57:0/548683720 conn(0x7fc2cc069930 msgr2=0x7fc2cc158190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:16.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.385+0000 7fc2d2cd7640 1 -- 172.21.15.57:0/548683720 shutdown_connections 2023-10-19T14:20:16.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.385+0000 7fc2d2cd7640 1 -- 172.21.15.57:0/548683720 wait complete. 2023-10-19T14:20:16.399 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-19T12:44:07.371186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-19T14:20:16.400 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-19T14:20:16.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.509+0000 7fe3b621a640 1 -- 172.21.15.57:0/1070528500 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3b0141690 msgr2=0x7fe3b00703e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:16.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.509+0000 7fe3b621a640 1 --2- 172.21.15.57:0/1070528500 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3b0141690 0x7fe3b00703e0 secure :-1 s=READY pgs=2330 cs=0 l=1 rev1=1 crypto rx=0x7fe3a4009830 tx=0x7fe3a402bcd0 comp rx=0 tx=0).stop 2023-10-19T14:20:16.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.509+0000 7fe3b621a640 1 -- 172.21.15.57:0/1070528500 shutdown_connections 2023-10-19T14:20:16.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.509+0000 7fe3b621a640 1 --2- 172.21.15.57:0/1070528500 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe3b0070920 0x7fe3b00affe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:16.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.509+0000 7fe3b621a640 1 --2- 172.21.15.57:0/1070528500 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3b0141690 0x7fe3b00703e0 unknown :-1 s=CLOSED pgs=2330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:16.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.509+0000 7fe3b621a640 1 -- 172.21.15.57:0/1070528500 >> 172.21.15.57:0/1070528500 conn(0x7fe3b0069930 msgr2=0x7fe3b0069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:16.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.509+0000 7fe3b621a640 1 -- 172.21.15.57:0/1070528500 shutdown_connections 2023-10-19T14:20:16.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.509+0000 7fe3b621a640 1 -- 172.21.15.57:0/1070528500 wait complete. 2023-10-19T14:20:16.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.510+0000 7fe3b621a640 1 Processor -- start 2023-10-19T14:20:16.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.510+0000 7fe3b621a640 1 -- start start 2023-10-19T14:20:16.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.511+0000 7fe3b621a640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe3b0070920 0x7fe3b00c6750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:16.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.511+0000 7fe3b621a640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe3b013e520 0x7fe3b01e1660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:16.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.511+0000 7fe3b621a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3b0141690 0x7fe3b013cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:16.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.511+0000 7fe3b621a640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3b00a6b70 con 0x7fe3b0141690 2023-10-19T14:20:16.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.511+0000 7fe3b621a640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3b00c5da0 con 0x7fe3b0070920 2023-10-19T14:20:16.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.511+0000 7fe3b621a640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3b00c6080 con 0x7fe3b013e520 2023-10-19T14:20:16.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.511+0000 7fe3affff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3b0141690 0x7fe3b013cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:16.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.511+0000 7fe3affff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3b0141690 0x7fe3b013cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35812/0 (socket says 172.21.15.57:35812) 2023-10-19T14:20:16.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.511+0000 7fe3affff640 1 -- 172.21.15.57:0/2159870267 learned_addr learned my addr 172.21.15.57:0/2159870267 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:16.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.511+0000 7fe3aeffd640 1 --2- 172.21.15.57:0/2159870267 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe3b013e520 0x7fe3b01e1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:16.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.511+0000 7fe3af7fe640 1 --2- 172.21.15.57:0/2159870267 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe3b0070920 0x7fe3b00c6750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:16.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.512+0000 7fe3affff640 1 -- 172.21.15.57:0/2159870267 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe3b013e520 msgr2=0x7fe3b01e1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:16.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.512+0000 7fe3affff640 1 --2- 172.21.15.57:0/2159870267 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe3b013e520 0x7fe3b01e1660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:16.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.512+0000 7fe3affff640 1 -- 172.21.15.57:0/2159870267 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe3b0070920 msgr2=0x7fe3b00c6750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:16.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.512+0000 7fe3affff640 1 --2- 172.21.15.57:0/2159870267 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe3b0070920 0x7fe3b00c6750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:16.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.512+0000 7fe3affff640 1 -- 172.21.15.57:0/2159870267 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3a4009490 con 0x7fe3b0141690 2023-10-19T14:20:16.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.512+0000 7fe3affff640 1 --2- 172.21.15.57:0/2159870267 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3b0141690 0x7fe3b013cee0 secure :-1 s=READY pgs=2331 cs=0 l=1 rev1=1 crypto rx=0x7fe3a800eaa0 tx=0x7fe3a800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:16.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.512+0000 7fe397fff640 1 -- 172.21.15.57:0/2159870267 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3a800cdb0 con 0x7fe3b0141690 2023-10-19T14:20:16.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.512+0000 7fe397fff640 1 -- 172.21.15.57:0/2159870267 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fe3a8004b20 con 0x7fe3b0141690 2023-10-19T14:20:16.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.512+0000 7fe3b621a640 1 -- 172.21.15.57:0/2159870267 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3b00c6360 con 0x7fe3b0141690 2023-10-19T14:20:16.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.512+0000 7fe397fff640 1 -- 172.21.15.57:0/2159870267 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3a8005020 con 0x7fe3b0141690 2023-10-19T14:20:16.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.512+0000 7fe3b621a640 1 -- 172.21.15.57:0/2159870267 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3b00a6790 con 0x7fe3b0141690 2023-10-19T14:20:16.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.513+0000 7fe3b621a640 1 -- 172.21.15.57:0/2159870267 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3b00a6cb0 con 0x7fe3b0141690 2023-10-19T14:20:16.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.513+0000 7fe397fff640 1 -- 172.21.15.57:0/2159870267 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe3a8005180 con 0x7fe3b0141690 2023-10-19T14:20:16.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.514+0000 7fe397fff640 1 --2- 172.21.15.57:0/2159870267 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fe39c06af60 0x7fe39c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:16.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.514+0000 7fe397fff640 1 -- 172.21.15.57:0/2159870267 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 5205+0+0 (secure 0 0 0) 0x7fe3a808a000 con 0x7fe3b0141690 2023-10-19T14:20:16.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.514+0000 7fe3af7fe640 1 --2- 172.21.15.57:0/2159870267 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fe39c06af60 0x7fe39c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:16.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.515+0000 7fe3af7fe640 1 --2- 172.21.15.57:0/2159870267 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fe39c06af60 0x7fe39c06d420 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe3a0001640 tx=0x7fe3a0009040 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:16.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.518+0000 7fe397fff640 1 -- 172.21.15.57:0/2159870267 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe3b00a6cb0 con 0x7fe3b0141690 2023-10-19T14:20:16.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.698+0000 7fe3b621a640 1 -- 172.21.15.57:0/2159870267 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe3b00a6cb0 con 0x7fe3b0141690 2023-10-19T14:20:16.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.698+0000 7fe397fff640 1 -- 172.21.15.57:0/2159870267 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v219) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe3b00a6cb0 con 0x7fe3b0141690 2023-10-19T14:20:16.701 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-19T14:20:16.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.700+0000 7fe3b621a640 1 -- 172.21.15.57:0/2159870267 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fe39c06af60 msgr2=0x7fe39c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:16.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.700+0000 7fe3b621a640 1 --2- 172.21.15.57:0/2159870267 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fe39c06af60 0x7fe39c06d420 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe3a0001640 tx=0x7fe3a0009040 comp rx=0 tx=0).stop 2023-10-19T14:20:16.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.700+0000 7fe3b621a640 1 -- 172.21.15.57:0/2159870267 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3b0141690 msgr2=0x7fe3b013cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:16.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.700+0000 7fe3b621a640 1 --2- 172.21.15.57:0/2159870267 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3b0141690 0x7fe3b013cee0 secure :-1 s=READY pgs=2331 cs=0 l=1 rev1=1 crypto rx=0x7fe3a800eaa0 tx=0x7fe3a800ef60 comp rx=0 tx=0).stop 2023-10-19T14:20:16.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.700+0000 7fe3b621a640 1 -- 172.21.15.57:0/2159870267 shutdown_connections 2023-10-19T14:20:16.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.700+0000 7fe3b621a640 1 --2- 172.21.15.57:0/2159870267 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fe39c06af60 0x7fe39c06d420 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:16.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.700+0000 7fe3b621a640 1 --2- 172.21.15.57:0/2159870267 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3b0141690 0x7fe3b013cee0 unknown :-1 s=CLOSED pgs=2331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:16.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.700+0000 7fe3b621a640 1 --2- 172.21.15.57:0/2159870267 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe3b013e520 0x7fe3b01e1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:16.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.700+0000 7fe3b621a640 1 --2- 172.21.15.57:0/2159870267 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe3b0070920 0x7fe3b00c6750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:16.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.700+0000 7fe3b621a640 1 -- 172.21.15.57:0/2159870267 >> 172.21.15.57:0/2159870267 conn(0x7fe3b0069930 msgr2=0x7fe3b006bb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:16.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.700+0000 7fe3b621a640 1 -- 172.21.15.57:0/2159870267 shutdown_connections 2023-10-19T14:20:16.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.700+0000 7fe3b621a640 1 -- 172.21.15.57:0/2159870267 wait complete. 2023-10-19T14:20:16.715 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-19T14:20:16.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.822+0000 7f57575b6640 1 -- 172.21.15.57:0/270699507 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f57500ceb40 msgr2=0x7f57500b0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:16.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.822+0000 7f57575b6640 1 --2- 172.21.15.57:0/270699507 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f57500ceb40 0x7f57500b0650 secure :-1 s=READY pgs=2332 cs=0 l=1 rev1=1 crypto rx=0x7f575008a810 tx=0x7f574c02bd10 comp rx=0 tx=0).stop 2023-10-19T14:20:16.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.822+0000 7f57575b6640 1 -- 172.21.15.57:0/270699507 shutdown_connections 2023-10-19T14:20:16.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.822+0000 7f57575b6640 1 --2- 172.21.15.57:0/270699507 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f57500ceb40 0x7f57500b0650 unknown :-1 s=CLOSED pgs=2332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:16.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.822+0000 7f57575b6640 1 --2- 172.21.15.57:0/270699507 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f575015b2a0 0x7f57500cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:16.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.822+0000 7f57575b6640 1 --2- 172.21.15.57:0/270699507 >> v2:172.21.15.179:3300/0 conn(0x7f57500cf340 0x7f57500d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:16.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.822+0000 7f57575b6640 1 -- 172.21.15.57:0/270699507 >> 172.21.15.57:0/270699507 conn(0x7f57500d0d60 msgr2=0x7f57500d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:16.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.822+0000 7f57575b6640 1 -- 172.21.15.57:0/270699507 shutdown_connections 2023-10-19T14:20:16.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.822+0000 7f57575b6640 1 -- 172.21.15.57:0/270699507 wait complete. 2023-10-19T14:20:16.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.823+0000 7f57575b6640 1 Processor -- start 2023-10-19T14:20:16.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.824+0000 7f57575b6640 1 -- start start 2023-10-19T14:20:16.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.824+0000 7f57575b6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f57500ceb40 0x7f57500b38e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:16.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.824+0000 7f57575b6640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f57500cf340 0x7f57500af020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:16.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.824+0000 7f57575b6640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f575015b2a0 0x7f575015c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:16.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.824+0000 7f57575b6640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57500b4a80 con 0x7f57500ceb40 2023-10-19T14:20:16.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.824+0000 7f57575b6640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57500b41e0 con 0x7f575015b2a0 2023-10-19T14:20:16.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.824+0000 7f57575b6640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57500b44c0 con 0x7f57500cf340 2023-10-19T14:20:16.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.824+0000 7f5755352640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f57500ceb40 0x7f57500b38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:16.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.824+0000 7f5755352640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f57500ceb40 0x7f57500b38e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:41908/0 (socket says 172.21.15.57:41908) 2023-10-19T14:20:16.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.824+0000 7f5755352640 1 -- 172.21.15.57:0/3588745223 learned_addr learned my addr 172.21.15.57:0/3588745223 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:16.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.825+0000 7f5755b53640 1 --2- 172.21.15.57:0/3588745223 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f575015b2a0 0x7f575015c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:16.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.825+0000 7f5754b51640 1 --2- 172.21.15.57:0/3588745223 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f57500cf340 0x7f57500af020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:16.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.825+0000 7f5755352640 1 -- 172.21.15.57:0/3588745223 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f57500cf340 msgr2=0x7f57500af020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:16.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.825+0000 7f5755352640 1 --2- 172.21.15.57:0/3588745223 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f57500cf340 0x7f57500af020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:16.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.825+0000 7f5755352640 1 -- 172.21.15.57:0/3588745223 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f575015b2a0 msgr2=0x7f575015c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:16.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.825+0000 7f5755352640 1 --2- 172.21.15.57:0/3588745223 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f575015b2a0 0x7f575015c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:16.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.825+0000 7f5755352640 1 -- 172.21.15.57:0/3588745223 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f574c009560 con 0x7f57500ceb40 2023-10-19T14:20:16.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.825+0000 7f5755b53640 1 --2- 172.21.15.57:0/3588745223 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f575015b2a0 0x7f575015c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:20:16.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.825+0000 7f5754b51640 1 --2- 172.21.15.57:0/3588745223 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f57500cf340 0x7f57500af020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:20:16.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.825+0000 7f5755352640 1 --2- 172.21.15.57:0/3588745223 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f57500ceb40 0x7f57500b38e0 secure :-1 s=READY pgs=2333 cs=0 l=1 rev1=1 crypto rx=0x7f574800ba50 tx=0x7f574800bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:16.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.825+0000 7f5745ffb640 1 -- 172.21.15.57:0/3588745223 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f574800cbe0 con 0x7f57500ceb40 2023-10-19T14:20:16.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.825+0000 7f5745ffb640 1 -- 172.21.15.57:0/3588745223 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f574800ce50 con 0x7f57500ceb40 2023-10-19T14:20:16.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.825+0000 7f57575b6640 1 -- 172.21.15.57:0/3588745223 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57500b47a0 con 0x7f57500ceb40 2023-10-19T14:20:16.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.825+0000 7f5745ffb640 1 -- 172.21.15.57:0/3588745223 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5748014680 con 0x7f57500ceb40 2023-10-19T14:20:16.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.825+0000 7f57575b6640 1 -- 172.21.15.57:0/3588745223 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57500b3e20 con 0x7f57500ceb40 2023-10-19T14:20:16.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.826+0000 7f57575b6640 1 -- 172.21.15.57:0/3588745223 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f575009b1b0 con 0x7f57500ceb40 2023-10-19T14:20:16.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.827+0000 7f5745ffb640 1 -- 172.21.15.57:0/3588745223 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f574801d3b0 con 0x7f57500ceb40 2023-10-19T14:20:16.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.827+0000 7f5745ffb640 1 --2- 172.21.15.57:0/3588745223 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f573406af90 0x7f573406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:16.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.827+0000 7f5745ffb640 1 -- 172.21.15.57:0/3588745223 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f5748089840 con 0x7f57500ceb40 2023-10-19T14:20:16.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.828+0000 7f5754b51640 1 --2- 172.21.15.57:0/3588745223 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f573406af90 0x7f573406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:16.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.828+0000 7f5754b51640 1 --2- 172.21.15.57:0/3588745223 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f573406af90 0x7f573406d450 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f57500c9e60 tx=0x7f5740008040 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:16.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:16.832+0000 7f5745ffb640 1 -- 172.21.15.57:0/3588745223 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f5748054930 con 0x7f57500ceb40 2023-10-19T14:20:17.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.010+0000 7f57575b6640 1 -- 172.21.15.57:0/3588745223 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f57500d07d0 con 0x7f57500ceb40 2023-10-19T14:20:17.014 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.010+0000 7f5745ffb640 1 -- 172.21.15.57:0/3588745223 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v219) v1 ==== 84+0+65 (secure 0 0 0) 0x7f5748019090 con 0x7f57500ceb40 2023-10-19T14:20:17.014 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:20:17.014 INFO:teuthology.orchestra.run.smithi057.stdout:[ 2023-10-19T14:20:17.014 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:20:17.015 INFO:teuthology.orchestra.run.smithi057.stdout: "poolnum": 1, 2023-10-19T14:20:17.015 INFO:teuthology.orchestra.run.smithi057.stdout: "poolname": ".mgr" 2023-10-19T14:20:17.015 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:20:17.015 INFO:teuthology.orchestra.run.smithi057.stdout:] 2023-10-19T14:20:17.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.012+0000 7f57575b6640 1 -- 172.21.15.57:0/3588745223 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f573406af90 msgr2=0x7f573406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:17.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.012+0000 7f57575b6640 1 --2- 172.21.15.57:0/3588745223 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f573406af90 0x7f573406d450 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f57500c9e60 tx=0x7f5740008040 comp rx=0 tx=0).stop 2023-10-19T14:20:17.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.012+0000 7f57575b6640 1 -- 172.21.15.57:0/3588745223 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f57500ceb40 msgr2=0x7f57500b38e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:17.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.012+0000 7f57575b6640 1 --2- 172.21.15.57:0/3588745223 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f57500ceb40 0x7f57500b38e0 secure :-1 s=READY pgs=2333 cs=0 l=1 rev1=1 crypto rx=0x7f574800ba50 tx=0x7f574800bf10 comp rx=0 tx=0).stop 2023-10-19T14:20:17.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.012+0000 7f57575b6640 1 -- 172.21.15.57:0/3588745223 shutdown_connections 2023-10-19T14:20:17.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.012+0000 7f57575b6640 1 --2- 172.21.15.57:0/3588745223 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f573406af90 0x7f573406d450 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:17.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.012+0000 7f57575b6640 1 --2- 172.21.15.57:0/3588745223 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f575015b2a0 0x7f575015c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:17.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.012+0000 7f57575b6640 1 --2- 172.21.15.57:0/3588745223 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f57500cf340 0x7f57500af020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:17.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.012+0000 7f57575b6640 1 --2- 172.21.15.57:0/3588745223 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f57500ceb40 0x7f57500b38e0 unknown :-1 s=CLOSED pgs=2333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:17.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.012+0000 7f57575b6640 1 -- 172.21.15.57:0/3588745223 >> 172.21.15.57:0/3588745223 conn(0x7f57500d0d60 msgr2=0x7f57500ae180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:17.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.012+0000 7f57575b6640 1 -- 172.21.15.57:0/3588745223 shutdown_connections 2023-10-19T14:20:17.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.013+0000 7f57575b6640 1 -- 172.21.15.57:0/3588745223 wait complete. 2023-10-19T14:20:17.026 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-19T14:20:17.026 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-19T14:20:17.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.137+0000 7fdb83b83640 1 -- 172.21.15.57:0/1404896178 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdb68004090 msgr2=0x7fdb680047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:17.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.137+0000 7fdb83b83640 1 --2- 172.21.15.57:0/1404896178 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdb68004090 0x7fdb680047c0 secure :-1 s=READY pgs=2335 cs=0 l=1 rev1=1 crypto rx=0x7fdb6c009b50 tx=0x7fdb680057f0 comp rx=0 tx=0).stop 2023-10-19T14:20:17.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.137+0000 7fdb82120640 1 -- 172.21.15.57:0/1404896178 reap_dead start 2023-10-19T14:20:17.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.137+0000 7fdb83b83640 1 -- 172.21.15.57:0/1404896178 shutdown_connections 2023-10-19T14:20:17.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.137+0000 7fdb83b83640 1 -- 172.21.15.57:0/1404896178 >> 172.21.15.57:0/1404896178 conn(0x7fdb7c0d0d60 msgr2=0x7fdb7c0d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:17.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.137+0000 7fdb83b83640 1 -- 172.21.15.57:0/1404896178 shutdown_connections 2023-10-19T14:20:17.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.137+0000 7fdb83b83640 1 -- 172.21.15.57:0/1404896178 wait complete. 2023-10-19T14:20:17.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.138+0000 7fdb83b83640 1 Processor -- start 2023-10-19T14:20:17.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.138+0000 7fdb83b83640 1 -- start start 2023-10-19T14:20:17.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.139+0000 7fdb83b83640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdb68004090 0x7fdb7c0b3d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:17.149 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.139+0000 7fdb83b83640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdb7c0ca9d0 0x7fdb7c09e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:17.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.139+0000 7fdb83b83640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdb7c158900 0x7fdb7c158d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:17.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.139+0000 7fdb83b83640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb7c1592c0 con 0x7fdb68004090 2023-10-19T14:20:17.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.139+0000 7fdb83b83640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb7c0b3b70 con 0x7fdb7c0ca9d0 2023-10-19T14:20:17.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.139+0000 7fdb83b83640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb7c0b4450 con 0x7fdb7c158900 2023-10-19T14:20:17.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.139+0000 7fdb8191f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdb68004090 0x7fdb7c0b3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:17.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.139+0000 7fdb8191f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdb68004090 0x7fdb7c0b3d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:41930/0 (socket says 172.21.15.57:41930) 2023-10-19T14:20:17.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.139+0000 7fdb8191f640 1 -- 172.21.15.57:0/4164021261 learned_addr learned my addr 172.21.15.57:0/4164021261 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:17.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.139+0000 7fdb8111e640 1 --2- 172.21.15.57:0/4164021261 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdb7c0ca9d0 0x7fdb7c09e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:17.151 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.139+0000 7fdb8191f640 1 -- 172.21.15.57:0/4164021261 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdb7c158900 msgr2=0x7fdb7c158d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:17.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.139+0000 7fdb82120640 1 --2- 172.21.15.57:0/4164021261 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdb7c158900 0x7fdb7c158d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:17.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.139+0000 7fdb8191f640 1 --2- 172.21.15.57:0/4164021261 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdb7c158900 0x7fdb7c158d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:17.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.139+0000 7fdb8191f640 1 -- 172.21.15.57:0/4164021261 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdb7c0ca9d0 msgr2=0x7fdb7c09e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:17.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.139+0000 7fdb8191f640 1 --2- 172.21.15.57:0/4164021261 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdb7c0ca9d0 0x7fdb7c09e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:17.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.139+0000 7fdb8191f640 1 -- 172.21.15.57:0/4164021261 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb7c0b4730 con 0x7fdb68004090 2023-10-19T14:20:17.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.139+0000 7fdb8111e640 1 --2- 172.21.15.57:0/4164021261 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdb7c0ca9d0 0x7fdb7c09e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:20:17.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.140+0000 7fdb8191f640 1 --2- 172.21.15.57:0/4164021261 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdb68004090 0x7fdb7c0b3d90 secure :-1 s=READY pgs=2336 cs=0 l=1 rev1=1 crypto rx=0x7fdb7800e510 tx=0x7fdb7800e9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:17.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.140+0000 7fdb727fc640 1 -- 172.21.15.57:0/4164021261 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb78010890 con 0x7fdb68004090 2023-10-19T14:20:17.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.140+0000 7fdb727fc640 1 -- 172.21.15.57:0/4164021261 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fdb78010ae0 con 0x7fdb68004090 2023-10-19T14:20:17.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.140+0000 7fdb83b83640 1 -- 172.21.15.57:0/4164021261 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb7c0b4a10 con 0x7fdb68004090 2023-10-19T14:20:17.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.140+0000 7fdb727fc640 1 -- 172.21.15.57:0/4164021261 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb78011460 con 0x7fdb68004090 2023-10-19T14:20:17.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.140+0000 7fdb83b83640 1 -- 172.21.15.57:0/4164021261 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb7c0b4fd0 con 0x7fdb68004090 2023-10-19T14:20:17.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.140+0000 7fdb83b83640 1 -- 172.21.15.57:0/4164021261 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb7c0c7db0 con 0x7fdb68004090 2023-10-19T14:20:17.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.141+0000 7fdb727fc640 1 -- 172.21.15.57:0/4164021261 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fdb780115c0 con 0x7fdb68004090 2023-10-19T14:20:17.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.141+0000 7fdb727fc640 1 --2- 172.21.15.57:0/4164021261 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fdb6006af60 0x7fdb6006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:17.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.142+0000 7fdb727fc640 1 -- 172.21.15.57:0/4164021261 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 5205+0+0 (secure 0 0 0) 0x7fdb78089ec0 con 0x7fdb68004090 2023-10-19T14:20:17.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.142+0000 7fdb8111e640 1 --2- 172.21.15.57:0/4164021261 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fdb6006af60 0x7fdb6006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:17.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.142+0000 7fdb8111e640 1 --2- 172.21.15.57:0/4164021261 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fdb6006af60 0x7fdb6006d420 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fdb6c036000 tx=0x7fdb6c035040 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:17.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.146+0000 7fdb727fc640 1 -- 172.21.15.57:0/4164021261 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fdb7c0c7db0 con 0x7fdb68004090 2023-10-19T14:20:17.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.322+0000 7fdb83b83640 1 -- 172.21.15.57:0/4164021261 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fdb7c0c7db0 con 0x7fdb68004090 2023-10-19T14:20:17.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.694+0000 7fdb727fc640 1 -- 172.21.15.57:0/4164021261 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v220) v1 ==== 138+0+0 (secure 0 0 0) 0x7fdb7c0c7db0 con 0x7fdb68004090 2023-10-19T14:20:17.697 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'cephfs_metadata' created 2023-10-19T14:20:17.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.697+0000 7fdb83b83640 1 -- 172.21.15.57:0/4164021261 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fdb6006af60 msgr2=0x7fdb6006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:17.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.697+0000 7fdb83b83640 1 --2- 172.21.15.57:0/4164021261 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fdb6006af60 0x7fdb6006d420 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fdb6c036000 tx=0x7fdb6c035040 comp rx=0 tx=0).stop 2023-10-19T14:20:17.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.697+0000 7fdb83b83640 1 -- 172.21.15.57:0/4164021261 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdb68004090 msgr2=0x7fdb7c0b3d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:17.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.697+0000 7fdb83b83640 1 --2- 172.21.15.57:0/4164021261 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdb68004090 0x7fdb7c0b3d90 secure :-1 s=READY pgs=2336 cs=0 l=1 rev1=1 crypto rx=0x7fdb7800e510 tx=0x7fdb7800e9d0 comp rx=0 tx=0).stop 2023-10-19T14:20:17.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.697+0000 7fdb83b83640 1 -- 172.21.15.57:0/4164021261 shutdown_connections 2023-10-19T14:20:17.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.697+0000 7fdb83b83640 1 --2- 172.21.15.57:0/4164021261 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fdb6006af60 0x7fdb6006d420 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:17.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.697+0000 7fdb83b83640 1 --2- 172.21.15.57:0/4164021261 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdb7c158900 0x7fdb7c158d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:17.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.697+0000 7fdb83b83640 1 --2- 172.21.15.57:0/4164021261 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdb7c0ca9d0 0x7fdb7c09e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:17.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.697+0000 7fdb83b83640 1 --2- 172.21.15.57:0/4164021261 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdb68004090 0x7fdb7c0b3d90 unknown :-1 s=CLOSED pgs=2336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:17.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.697+0000 7fdb83b83640 1 -- 172.21.15.57:0/4164021261 >> 172.21.15.57:0/4164021261 conn(0x7fdb7c0d0d60 msgr2=0x7fdb7c0cf720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:17.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.697+0000 7fdb83b83640 1 -- 172.21.15.57:0/4164021261 shutdown_connections 2023-10-19T14:20:17.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.697+0000 7fdb83b83640 1 -- 172.21.15.57:0/4164021261 wait complete. 2023-10-19T14:20:17.715 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-19T14:20:17.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.829+0000 7f69c6976640 1 -- 172.21.15.57:0/3276999578 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f69a8003fd0 msgr2=0x7f69a80047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:17.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.829+0000 7f69c6976640 1 --2- 172.21.15.57:0/3276999578 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f69a8003fd0 0x7f69a80047c0 secure :-1 s=READY pgs=2338 cs=0 l=1 rev1=1 crypto rx=0x7f69b4006fd0 tx=0x7f69a80057f0 comp rx=0 tx=0).stop 2023-10-19T14:20:17.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.829+0000 7f69c4f13640 1 -- 172.21.15.57:0/3276999578 reap_dead start 2023-10-19T14:20:17.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.829+0000 7f69c6976640 1 -- 172.21.15.57:0/3276999578 shutdown_connections 2023-10-19T14:20:17.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.829+0000 7f69c6976640 1 -- 172.21.15.57:0/3276999578 >> 172.21.15.57:0/3276999578 conn(0x7f69c006fae0 msgr2=0x7f69c006fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:17.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.829+0000 7f69c6976640 1 -- 172.21.15.57:0/3276999578 shutdown_connections 2023-10-19T14:20:17.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.829+0000 7f69c6976640 1 -- 172.21.15.57:0/3276999578 wait complete. 2023-10-19T14:20:17.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.830+0000 7f69c6976640 1 Processor -- start 2023-10-19T14:20:17.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.830+0000 7f69c6976640 1 -- start start 2023-10-19T14:20:17.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.831+0000 7f69c6976640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f69a8003fd0 0x7f69c0156c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:17.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.831+0000 7f69c6976640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f69c0140930 0x7f69c013d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:17.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.831+0000 7f69c6976640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f69c015a360 0x7f69c015a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:17.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.831+0000 7f69c6976640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69c015ad50 con 0x7f69c015a360 2023-10-19T14:20:17.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.831+0000 7f69c6976640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69c0157280 con 0x7f69c0140930 2023-10-19T14:20:17.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.831+0000 7f69c6976640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69c013db80 con 0x7f69a8003fd0 2023-10-19T14:20:17.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.831+0000 7f69c4f13640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f69c015a360 0x7f69c015a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:17.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.831+0000 7f69c4f13640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f69c015a360 0x7f69c015a810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:41940/0 (socket says 172.21.15.57:41940) 2023-10-19T14:20:17.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.831+0000 7f69c4f13640 1 -- 172.21.15.57:0/3379285943 learned_addr learned my addr 172.21.15.57:0/3379285943 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:17.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.831+0000 7f69bf7fe640 1 --2- 172.21.15.57:0/3379285943 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f69c0140930 0x7f69c013d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:17.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.831+0000 7f69bffff640 1 --2- 172.21.15.57:0/3379285943 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f69a8003fd0 0x7f69c0156c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:17.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.832+0000 7f69c4f13640 1 -- 172.21.15.57:0/3379285943 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f69a8003fd0 msgr2=0x7f69c0156c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:17.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.832+0000 7f69c4f13640 1 --2- 172.21.15.57:0/3379285943 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f69a8003fd0 0x7f69c0156c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:17.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.832+0000 7f69c4f13640 1 -- 172.21.15.57:0/3379285943 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f69c0140930 msgr2=0x7f69c013d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:17.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.832+0000 7f69c4f13640 1 --2- 172.21.15.57:0/3379285943 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f69c0140930 0x7f69c013d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:17.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.832+0000 7f69c4f13640 1 -- 172.21.15.57:0/3379285943 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69c013dfc0 con 0x7f69c015a360 2023-10-19T14:20:17.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.832+0000 7f69c4f13640 1 --2- 172.21.15.57:0/3379285943 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f69c015a360 0x7f69c015a810 secure :-1 s=READY pgs=2339 cs=0 l=1 rev1=1 crypto rx=0x7f69c00b0160 tx=0x7f69b8010f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:17.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.832+0000 7f69bcff9640 1 -- 172.21.15.57:0/3379285943 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b800ee70 con 0x7f69c015a360 2023-10-19T14:20:17.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.832+0000 7f69bcff9640 1 -- 172.21.15.57:0/3379285943 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f69b8004b20 con 0x7f69c015a360 2023-10-19T14:20:17.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.832+0000 7f69c6976640 1 -- 172.21.15.57:0/3379285943 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69c013e2a0 con 0x7f69c015a360 2023-10-19T14:20:17.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.832+0000 7f69bcff9640 1 -- 172.21.15.57:0/3379285943 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b8003970 con 0x7f69c015a360 2023-10-19T14:20:17.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.832+0000 7f69c6976640 1 -- 172.21.15.57:0/3379285943 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69c013e860 con 0x7f69c015a360 2023-10-19T14:20:17.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.833+0000 7f69c6976640 1 -- 172.21.15.57:0/3379285943 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69c013ea80 con 0x7f69c015a360 2023-10-19T14:20:17.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.834+0000 7f69bcff9640 1 -- 172.21.15.57:0/3379285943 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f69b8003ad0 con 0x7f69c015a360 2023-10-19T14:20:17.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.834+0000 7f69bcff9640 1 --2- 172.21.15.57:0/3379285943 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f69ac06af90 0x7f69ac06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:17.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.834+0000 7f69bcff9640 1 -- 172.21.15.57:0/3379285943 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(220..220 src has 1..220) v4 ==== 5575+0+0 (secure 0 0 0) 0x7f69b808a3a0 con 0x7f69c015a360 2023-10-19T14:20:17.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.835+0000 7f69bffff640 1 --2- 172.21.15.57:0/3379285943 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f69ac06af90 0x7f69ac06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:17.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.835+0000 7f69bffff640 1 --2- 172.21.15.57:0/3379285943 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f69ac06af90 0x7f69ac06d450 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f69b0006fd0 tx=0x7f69b0006b30 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:17.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:17.839+0000 7f69bcff9640 1 -- 172.21.15.57:0/3379285943 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f69b80552d0 con 0x7f69c015a360 2023-10-19T14:20:18.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:18.014+0000 7f69c6976640 1 -- 172.21.15.57:0/3379285943 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f69c00690e0 con 0x7f69c015a360 2023-10-19T14:20:18.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:18.703+0000 7f69bcff9640 1 -- 172.21.15.57:0/3379285943 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v221) v1 ==== 170+0+0 (secure 0 0 0) 0x7f69b80588f0 con 0x7f69c015a360 2023-10-19T14:20:18.706 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'cephfs_data' created 2023-10-19T14:20:18.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:18.706+0000 7f69c6976640 1 -- 172.21.15.57:0/3379285943 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f69ac06af90 msgr2=0x7f69ac06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:18.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:18.706+0000 7f69c6976640 1 --2- 172.21.15.57:0/3379285943 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f69ac06af90 0x7f69ac06d450 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f69b0006fd0 tx=0x7f69b0006b30 comp rx=0 tx=0).stop 2023-10-19T14:20:18.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:18.706+0000 7f69c6976640 1 -- 172.21.15.57:0/3379285943 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f69c015a360 msgr2=0x7f69c015a810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:18.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:18.706+0000 7f69c6976640 1 --2- 172.21.15.57:0/3379285943 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f69c015a360 0x7f69c015a810 secure :-1 s=READY pgs=2339 cs=0 l=1 rev1=1 crypto rx=0x7f69c00b0160 tx=0x7f69b8010f00 comp rx=0 tx=0).stop 2023-10-19T14:20:18.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:18.706+0000 7f69c6976640 1 -- 172.21.15.57:0/3379285943 shutdown_connections 2023-10-19T14:20:18.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:18.706+0000 7f69c6976640 1 --2- 172.21.15.57:0/3379285943 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f69ac06af90 0x7f69ac06d450 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:18.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:18.706+0000 7f69c6976640 1 --2- 172.21.15.57:0/3379285943 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f69c015a360 0x7f69c015a810 unknown :-1 s=CLOSED pgs=2339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:18.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:18.706+0000 7f69c6976640 1 --2- 172.21.15.57:0/3379285943 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f69c0140930 0x7f69c013d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:18.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:18.706+0000 7f69c6976640 1 --2- 172.21.15.57:0/3379285943 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f69a8003fd0 0x7f69c0156c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:18.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:18.706+0000 7f69c6976640 1 -- 172.21.15.57:0/3379285943 >> 172.21.15.57:0/3379285943 conn(0x7f69c006fae0 msgr2=0x7f69c0069720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:18.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:18.707+0000 7f69c6976640 1 -- 172.21.15.57:0/3379285943 shutdown_connections 2023-10-19T14:20:18.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:18.707+0000 7f69c6976640 1 -- 172.21.15.57:0/3379285943 wait complete. 2023-10-19T14:20:18.736 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-19T14:20:18.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:18.849+0000 7f3e56325640 1 -- 172.21.15.57:0/469909725 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3e3c0047c0 msgr2=0x7f3e3c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:18.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:18.849+0000 7f3e56325640 1 --2- 172.21.15.57:0/469909725 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3e3c0047c0 0x7f3e3c004c40 secure :-1 s=READY pgs=2341 cs=0 l=1 rev1=1 crypto rx=0x7f3e3800bb20 tx=0x7f3e3c0057f0 comp rx=0 tx=0).stop 2023-10-19T14:20:18.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:18.849+0000 7f3e56325640 1 -- 172.21.15.57:0/469909725 shutdown_connections 2023-10-19T14:20:18.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:18.849+0000 7f3e56325640 1 --2- 172.21.15.57:0/469909725 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3e3c0047c0 0x7f3e3c004c40 unknown :-1 s=CLOSED pgs=2341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:18.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:18.849+0000 7f3e56325640 1 -- 172.21.15.57:0/469909725 >> 172.21.15.57:0/469909725 conn(0x7f3e50069930 msgr2=0x7f3e50069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:18.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:18.849+0000 7f3e56325640 1 -- 172.21.15.57:0/469909725 shutdown_connections 2023-10-19T14:20:18.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:18.849+0000 7f3e56325640 1 -- 172.21.15.57:0/469909725 wait complete. 2023-10-19T14:20:18.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:18.850+0000 7f3e56325640 1 Processor -- start 2023-10-19T14:20:18.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:18.850+0000 7f3e56325640 1 -- start start 2023-10-19T14:20:18.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:18.851+0000 7f3e56325640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3e3c0047c0 0x7f3e5014e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:18.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:18.851+0000 7f3e56325640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3e5014aef0 0x7f3e500a7510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:18.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:18.851+0000 7f3e56325640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3e5014b350 0x7f3e500a7a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:18.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:18.851+0000 7f3e56325640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e5014d650 con 0x7f3e3c0047c0 2023-10-19T14:20:18.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:18.851+0000 7f3e56325640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e5014e4b0 con 0x7f3e5014aef0 2023-10-19T14:20:18.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:18.851+0000 7f3e56325640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e5014d8a0 con 0x7f3e5014b350 2023-10-19T14:20:18.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:18.851+0000 7f3e4ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3e3c0047c0 0x7f3e5014e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:18.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:18.851+0000 7f3e4ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3e3c0047c0 0x7f3e5014e940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:41966/0 (socket says 172.21.15.57:41966) 2023-10-19T14:20:18.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:18.851+0000 7f3e4ffff640 1 -- 172.21.15.57:0/1251403044 learned_addr learned my addr 172.21.15.57:0/1251403044 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:18.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:18.851+0000 7f3e4f7fe640 1 --2- 172.21.15.57:0/1251403044 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3e5014aef0 0x7f3e500a7510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:18.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:18.851+0000 7f3e548c2640 1 --2- 172.21.15.57:0/1251403044 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3e5014b350 0x7f3e500a7a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:18.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:18.851+0000 7f3e4ffff640 1 -- 172.21.15.57:0/1251403044 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3e5014b350 msgr2=0x7f3e500a7a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:18.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:18.851+0000 7f3e4ffff640 1 --2- 172.21.15.57:0/1251403044 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3e5014b350 0x7f3e500a7a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:18.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:18.851+0000 7f3e4ffff640 1 -- 172.21.15.57:0/1251403044 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3e5014aef0 msgr2=0x7f3e500a7510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:18.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:18.851+0000 7f3e4ffff640 1 --2- 172.21.15.57:0/1251403044 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3e5014aef0 0x7f3e500a7510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:18.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:18.851+0000 7f3e4ffff640 1 -- 172.21.15.57:0/1251403044 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e5014db80 con 0x7f3e3c0047c0 2023-10-19T14:20:18.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:18.851+0000 7f3e4ffff640 1 --2- 172.21.15.57:0/1251403044 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3e3c0047c0 0x7f3e5014e940 secure :-1 s=READY pgs=2342 cs=0 l=1 rev1=1 crypto rx=0x7f3e40004cb0 tx=0x7f3e40001630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:18.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:18.852+0000 7f3e4cff9640 1 -- 172.21.15.57:0/1251403044 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e40011660 con 0x7f3e3c0047c0 2023-10-19T14:20:18.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:18.852+0000 7f3e4cff9640 1 -- 172.21.15.57:0/1251403044 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f3e400118b0 con 0x7f3e3c0047c0 2023-10-19T14:20:18.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:18.852+0000 7f3e56325640 1 -- 172.21.15.57:0/1251403044 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e5014de60 con 0x7f3e3c0047c0 2023-10-19T14:20:18.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:18.852+0000 7f3e4cff9640 1 -- 172.21.15.57:0/1251403044 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e40011c80 con 0x7f3e3c0047c0 2023-10-19T14:20:18.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:18.852+0000 7f3e56325640 1 -- 172.21.15.57:0/1251403044 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e5014ee80 con 0x7f3e3c0047c0 2023-10-19T14:20:18.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:18.853+0000 7f3e56325640 1 -- 172.21.15.57:0/1251403044 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e500690e0 con 0x7f3e3c0047c0 2023-10-19T14:20:18.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:18.853+0000 7f3e4cff9640 1 -- 172.21.15.57:0/1251403044 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f3e40024440 con 0x7f3e3c0047c0 2023-10-19T14:20:18.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:18.854+0000 7f3e4cff9640 1 --2- 172.21.15.57:0/1251403044 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f3e3006af60 0x7f3e3006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:18.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:18.854+0000 7f3e4cff9640 1 -- 172.21.15.57:0/1251403044 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(221..221 src has 1..221) v4 ==== 5957+0+0 (secure 0 0 0) 0x7f3e4008a9d0 con 0x7f3e3c0047c0 2023-10-19T14:20:18.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:18.854+0000 7f3e4f7fe640 1 --2- 172.21.15.57:0/1251403044 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f3e3006af60 0x7f3e3006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:18.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:18.855+0000 7f3e4f7fe640 1 --2- 172.21.15.57:0/1251403044 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f3e3006af60 0x7f3e3006d420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3e38037000 tx=0x7f3e3802e040 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:18.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:18.858+0000 7f3e4cff9640 1 -- 172.21.15.57:0/1251403044 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3e40055980 con 0x7f3e3c0047c0 2023-10-19T14:20:19.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:19.075+0000 7f3e56325640 1 -- 172.21.15.57:0/1251403044 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f3e5006a1b0 con 0x7f3e3c0047c0 2023-10-19T14:20:19.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:19.706+0000 7f3e4cff9640 1 -- 172.21.15.57:0/1251403044 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 27 and data pool 28 v88) v1 ==== 174+0+0 (secure 0 0 0) 0x7f3e40058fa0 con 0x7f3e3c0047c0 2023-10-19T14:20:19.709 INFO:teuthology.orchestra.run.smithi057.stderr:new fs with metadata pool 27 and data pool 28 2023-10-19T14:20:19.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:19.709+0000 7f3e56325640 1 -- 172.21.15.57:0/1251403044 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f3e3006af60 msgr2=0x7f3e3006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:19.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:19.709+0000 7f3e56325640 1 --2- 172.21.15.57:0/1251403044 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f3e3006af60 0x7f3e3006d420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3e38037000 tx=0x7f3e3802e040 comp rx=0 tx=0).stop 2023-10-19T14:20:19.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:19.709+0000 7f3e56325640 1 -- 172.21.15.57:0/1251403044 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3e3c0047c0 msgr2=0x7f3e5014e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:19.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:19.709+0000 7f3e56325640 1 --2- 172.21.15.57:0/1251403044 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3e3c0047c0 0x7f3e5014e940 secure :-1 s=READY pgs=2342 cs=0 l=1 rev1=1 crypto rx=0x7f3e40004cb0 tx=0x7f3e40001630 comp rx=0 tx=0).stop 2023-10-19T14:20:19.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:19.709+0000 7f3e56325640 1 -- 172.21.15.57:0/1251403044 shutdown_connections 2023-10-19T14:20:19.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:19.709+0000 7f3e56325640 1 --2- 172.21.15.57:0/1251403044 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f3e3006af60 0x7f3e3006d420 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:19.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:19.709+0000 7f3e56325640 1 --2- 172.21.15.57:0/1251403044 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3e5014b350 0x7f3e500a7a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:19.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:19.709+0000 7f3e56325640 1 --2- 172.21.15.57:0/1251403044 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3e5014aef0 0x7f3e500a7510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:19.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:19.709+0000 7f3e56325640 1 --2- 172.21.15.57:0/1251403044 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3e3c0047c0 0x7f3e5014e940 unknown :-1 s=CLOSED pgs=2342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:19.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:19.709+0000 7f3e56325640 1 -- 172.21.15.57:0/1251403044 >> 172.21.15.57:0/1251403044 conn(0x7f3e50069930 msgr2=0x7f3e5006f900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:19.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:19.709+0000 7f3e56325640 1 -- 172.21.15.57:0/1251403044 shutdown_connections 2023-10-19T14:20:19.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:19.709+0000 7f3e56325640 1 -- 172.21.15.57:0/1251403044 wait complete. 2023-10-19T14:20:19.730 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T14:20:19.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:19.842+0000 7faf3e373640 1 -- 172.21.15.57:0/821615036 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faf380b0c40 msgr2=0x7faf38158300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:19.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:19.842+0000 7faf3e373640 1 --2- 172.21.15.57:0/821615036 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faf380b0c40 0x7faf38158300 secure :-1 s=READY pgs=2344 cs=0 l=1 rev1=1 crypto rx=0x7faf30005fd0 tx=0x7faf3002c170 comp rx=0 tx=0).stop 2023-10-19T14:20:19.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:19.842+0000 7faf3e373640 1 -- 172.21.15.57:0/821615036 shutdown_connections 2023-10-19T14:20:19.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:19.842+0000 7faf3e373640 1 --2- 172.21.15.57:0/821615036 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faf380b0c40 0x7faf38158300 unknown :-1 s=CLOSED pgs=2344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:19.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:19.842+0000 7faf3e373640 1 -- 172.21.15.57:0/821615036 >> 172.21.15.57:0/821615036 conn(0x7faf38069930 msgr2=0x7faf38069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:19.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:19.842+0000 7faf3e373640 1 -- 172.21.15.57:0/821615036 shutdown_connections 2023-10-19T14:20:19.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:19.842+0000 7faf3e373640 1 -- 172.21.15.57:0/821615036 wait complete. 2023-10-19T14:20:19.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:19.843+0000 7faf3e373640 1 Processor -- start 2023-10-19T14:20:19.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:19.844+0000 7faf3e373640 1 -- start start 2023-10-19T14:20:19.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:19.844+0000 7faf3e373640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faf380afa20 0x7faf3814d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:19.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:19.844+0000 7faf3e373640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faf380b0c40 0x7faf381532b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:19.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:19.844+0000 7faf3e373640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7faf380cbd20 0x7faf381562f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:19.851 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:19.844+0000 7faf3e373640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf381537f0 con 0x7faf380afa20 2023-10-19T14:20:19.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:19.844+0000 7faf3e373640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf3814e470 con 0x7faf380b0c40 2023-10-19T14:20:19.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:19.844+0000 7faf3e373640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf38155250 con 0x7faf380cbd20 2023-10-19T14:20:19.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:19.844+0000 7faf37fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faf380afa20 0x7faf3814d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:19.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:19.844+0000 7faf37fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faf380afa20 0x7faf3814d710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:41990/0 (socket says 172.21.15.57:41990) 2023-10-19T14:20:19.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:19.844+0000 7faf37fff640 1 -- 172.21.15.57:0/3619380292 learned_addr learned my addr 172.21.15.57:0/3619380292 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:19.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:19.845+0000 7faf377fe640 1 --2- 172.21.15.57:0/3619380292 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faf380b0c40 0x7faf381532b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:19.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:19.845+0000 7faf3c910640 1 --2- 172.21.15.57:0/3619380292 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7faf380cbd20 0x7faf381562f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:19.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:19.845+0000 7faf37fff640 1 -- 172.21.15.57:0/3619380292 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7faf380cbd20 msgr2=0x7faf381562f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:19.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:19.845+0000 7faf37fff640 1 --2- 172.21.15.57:0/3619380292 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7faf380cbd20 0x7faf381562f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:19.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:19.845+0000 7faf37fff640 1 -- 172.21.15.57:0/3619380292 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faf380b0c40 msgr2=0x7faf381532b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:19.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:19.845+0000 7faf37fff640 1 --2- 172.21.15.57:0/3619380292 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faf380b0c40 0x7faf381532b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:19.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:19.845+0000 7faf37fff640 1 -- 172.21.15.57:0/3619380292 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf30009840 con 0x7faf380afa20 2023-10-19T14:20:19.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:19.845+0000 7faf37fff640 1 --2- 172.21.15.57:0/3619380292 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faf380afa20 0x7faf3814d710 secure :-1 s=READY pgs=2345 cs=0 l=1 rev1=1 crypto rx=0x7faf2800c3c0 tx=0x7faf2800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:19.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:19.845+0000 7faf34ff9640 1 -- 172.21.15.57:0/3619380292 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf280035f0 con 0x7faf380afa20 2023-10-19T14:20:19.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:19.845+0000 7faf34ff9640 1 -- 172.21.15.57:0/3619380292 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7faf28003770 con 0x7faf380afa20 2023-10-19T14:20:19.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:19.845+0000 7faf3e373640 1 -- 172.21.15.57:0/3619380292 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf38155530 con 0x7faf380afa20 2023-10-19T14:20:19.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:19.845+0000 7faf34ff9640 1 -- 172.21.15.57:0/3619380292 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf28003b80 con 0x7faf380afa20 2023-10-19T14:20:19.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:19.845+0000 7faf3e373640 1 -- 172.21.15.57:0/3619380292 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf38155af0 con 0x7faf380afa20 2023-10-19T14:20:19.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:19.846+0000 7faf3e373640 1 -- 172.21.15.57:0/3619380292 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf380690e0 con 0x7faf380afa20 2023-10-19T14:20:19.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:19.847+0000 7faf34ff9640 1 -- 172.21.15.57:0/3619380292 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7faf28003ce0 con 0x7faf380afa20 2023-10-19T14:20:19.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:19.847+0000 7faf34ff9640 1 --2- 172.21.15.57:0/3619380292 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7faf1806af60 0x7faf1806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:19.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:19.847+0000 7faf34ff9640 1 -- 172.21.15.57:0/3619380292 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(222..222 src has 1..222) v4 ==== 6057+0+0 (secure 0 0 0) 0x7faf28089070 con 0x7faf380afa20 2023-10-19T14:20:19.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:19.847+0000 7faf377fe640 1 --2- 172.21.15.57:0/3619380292 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7faf1806af60 0x7faf1806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:19.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:19.848+0000 7faf377fe640 1 --2- 172.21.15.57:0/3619380292 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7faf1806af60 0x7faf1806d420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7faf381520d0 tx=0x7faf20009040 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:19.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:19.851+0000 7faf34ff9640 1 -- 172.21.15.57:0/3619380292 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7faf28053e30 con 0x7faf380afa20 2023-10-19T14:20:20.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.026+0000 7faf3e373640 1 -- 172.21.15.57:0/3619380292 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faf3806a040 con 0x7faf380afa20 2023-10-19T14:20:20.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.028+0000 7faf34ff9640 1 -- 172.21.15.57:0/3619380292 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v222) v1 ==== 74+0+16852 (secure 0 0 0) 0x7faf28057450 con 0x7faf380afa20 2023-10-19T14:20:20.031 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:20:20.031 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":222,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T14:20:19.077088+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2023-10-19T14:20:17.357433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"222","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2023-10-19T14:20:18.049516+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"222","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T14:20:20.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.029+0000 7faf3e373640 1 -- 172.21.15.57:0/3619380292 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7faf1806af60 msgr2=0x7faf1806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:20.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.029+0000 7faf3e373640 1 --2- 172.21.15.57:0/3619380292 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7faf1806af60 0x7faf1806d420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7faf381520d0 tx=0x7faf20009040 comp rx=0 tx=0).stop 2023-10-19T14:20:20.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.029+0000 7faf3e373640 1 -- 172.21.15.57:0/3619380292 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faf380afa20 msgr2=0x7faf3814d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:20.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.029+0000 7faf3e373640 1 --2- 172.21.15.57:0/3619380292 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faf380afa20 0x7faf3814d710 secure :-1 s=READY pgs=2345 cs=0 l=1 rev1=1 crypto rx=0x7faf2800c3c0 tx=0x7faf2800c880 comp rx=0 tx=0).stop 2023-10-19T14:20:20.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.030+0000 7faf3e373640 1 -- 172.21.15.57:0/3619380292 shutdown_connections 2023-10-19T14:20:20.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.030+0000 7faf3e373640 1 --2- 172.21.15.57:0/3619380292 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7faf1806af60 0x7faf1806d420 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:20.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.030+0000 7faf3e373640 1 --2- 172.21.15.57:0/3619380292 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7faf380cbd20 0x7faf381562f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:20.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.030+0000 7faf3e373640 1 --2- 172.21.15.57:0/3619380292 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faf380b0c40 0x7faf381532b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:20.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.030+0000 7faf3e373640 1 --2- 172.21.15.57:0/3619380292 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faf380afa20 0x7faf3814d710 unknown :-1 s=CLOSED pgs=2345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:20.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.030+0000 7faf3e373640 1 -- 172.21.15.57:0/3619380292 >> 172.21.15.57:0/3619380292 conn(0x7faf38069930 msgr2=0x7faf380afed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:20.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.030+0000 7faf3e373640 1 -- 172.21.15.57:0/3619380292 shutdown_connections 2023-10-19T14:20:20.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.030+0000 7faf3e373640 1 -- 172.21.15.57:0/3619380292 wait complete. 2023-10-19T14:20:20.043 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T14:20:20.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.151+0000 7f85c0561640 1 -- 172.21.15.57:0/2375969236 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85a0006290 msgr2=0x7f85a000ca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:20.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.151+0000 7f85c0561640 1 --2- 172.21.15.57:0/2375969236 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85a0006290 0x7f85a000ca90 secure :-1 s=READY pgs=2347 cs=0 l=1 rev1=1 crypto rx=0x7f85a800c640 tx=0x7f85a0006900 comp rx=0 tx=0).stop 2023-10-19T14:20:20.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.151+0000 7f85c0561640 1 -- 172.21.15.57:0/2375969236 shutdown_connections 2023-10-19T14:20:20.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.151+0000 7f85c0561640 1 --2- 172.21.15.57:0/2375969236 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85a0006290 0x7f85a000ca90 unknown :-1 s=CLOSED pgs=2347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:20.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.151+0000 7f85c0561640 1 -- 172.21.15.57:0/2375969236 >> 172.21.15.57:0/2375969236 conn(0x7f85b8069930 msgr2=0x7f85b8069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:20.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.151+0000 7f85c0561640 1 -- 172.21.15.57:0/2375969236 shutdown_connections 2023-10-19T14:20:20.155 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.151+0000 7f85c0561640 1 -- 172.21.15.57:0/2375969236 wait complete. 2023-10-19T14:20:20.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.152+0000 7f85c0561640 1 Processor -- start 2023-10-19T14:20:20.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.152+0000 7f85c0561640 1 -- start start 2023-10-19T14:20:20.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.152+0000 7f85c0561640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f85a0006290 0x7f85b8137d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:20.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.153+0000 7f85c0561640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85b8138940 0x7f85b814bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:20.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.153+0000 7f85c0561640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f85b814fdd0 0x7f85b8150230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:20.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.153+0000 7f85c0561640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85b8138250 con 0x7f85b8138940 2023-10-19T14:20:20.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.153+0000 7f85c0561640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85b8150770 con 0x7f85a0006290 2023-10-19T14:20:20.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.153+0000 7f85c0561640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85b814d0f0 con 0x7f85b814fdd0 2023-10-19T14:20:20.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.153+0000 7f85bdafc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85b8138940 0x7f85b814bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:20.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.153+0000 7f85bdafc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85b8138940 0x7f85b814bf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42020/0 (socket says 172.21.15.57:42020) 2023-10-19T14:20:20.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.153+0000 7f85bdafc640 1 -- 172.21.15.57:0/3874124258 learned_addr learned my addr 172.21.15.57:0/3874124258 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:20.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.153+0000 7f85be2fd640 1 --2- 172.21.15.57:0/3874124258 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f85a0006290 0x7f85b8137d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:20.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.153+0000 7f85beafe640 1 --2- 172.21.15.57:0/3874124258 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f85b814fdd0 0x7f85b8150230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:20.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.153+0000 7f85bdafc640 1 -- 172.21.15.57:0/3874124258 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f85b814fdd0 msgr2=0x7f85b8150230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:20.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.153+0000 7f85bdafc640 1 --2- 172.21.15.57:0/3874124258 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f85b814fdd0 0x7f85b8150230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:20.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.153+0000 7f85bdafc640 1 -- 172.21.15.57:0/3874124258 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f85a0006290 msgr2=0x7f85b8137d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:20.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.153+0000 7f85bdafc640 1 --2- 172.21.15.57:0/3874124258 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f85a0006290 0x7f85b8137d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:20.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.153+0000 7f85bdafc640 1 -- 172.21.15.57:0/3874124258 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85b814c850 con 0x7f85b8138940 2023-10-19T14:20:20.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.153+0000 7f85beafe640 1 --2- 172.21.15.57:0/3874124258 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f85b814fdd0 0x7f85b8150230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:20:20.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.153+0000 7f85bdafc640 1 --2- 172.21.15.57:0/3874124258 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85b8138940 0x7f85b814bf00 secure :-1 s=READY pgs=2348 cs=0 l=1 rev1=1 crypto rx=0x7f85ac005f60 tx=0x7f85ac011cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:20.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.153+0000 7f85b6ffd640 1 -- 172.21.15.57:0/3874124258 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85ac00fd50 con 0x7f85b8138940 2023-10-19T14:20:20.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.153+0000 7f85b6ffd640 1 -- 172.21.15.57:0/3874124258 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f85ac012450 con 0x7f85b8138940 2023-10-19T14:20:20.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.153+0000 7f85c0561640 1 -- 172.21.15.57:0/3874124258 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85b814cb30 con 0x7f85b8138940 2023-10-19T14:20:20.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.153+0000 7f85b6ffd640 1 -- 172.21.15.57:0/3874124258 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85ac012860 con 0x7f85b8138940 2023-10-19T14:20:20.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.153+0000 7f85c0561640 1 -- 172.21.15.57:0/3874124258 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85b8135e80 con 0x7f85b8138940 2023-10-19T14:20:20.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.154+0000 7f85c0561640 1 -- 172.21.15.57:0/3874124258 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85b80690e0 con 0x7f85b8138940 2023-10-19T14:20:20.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.155+0000 7f85b6ffd640 1 -- 172.21.15.57:0/3874124258 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f85ac0129c0 con 0x7f85b8138940 2023-10-19T14:20:20.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.155+0000 7f85b6ffd640 1 --2- 172.21.15.57:0/3874124258 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f85a406af90 0x7f85a406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:20.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.155+0000 7f85b6ffd640 1 -- 172.21.15.57:0/3874124258 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(222..222 src has 1..222) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f85ac08a620 con 0x7f85b8138940 2023-10-19T14:20:20.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.155+0000 7f85be2fd640 1 --2- 172.21.15.57:0/3874124258 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f85a406af90 0x7f85a406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:20.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.156+0000 7f85be2fd640 1 --2- 172.21.15.57:0/3874124258 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f85a406af90 0x7f85a406d450 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f85b0000c00 tx=0x7f85b0006c60 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:20.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.159+0000 7f85b6ffd640 1 -- 172.21.15.57:0/3874124258 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f85ac0555a0 con 0x7f85b8138940 2023-10-19T14:20:20.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.335+0000 7f85c0561640 1 -- 172.21.15.57:0/3874124258 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f85b806f700 con 0x7f85b8138940 2023-10-19T14:20:20.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.337+0000 7f85b6ffd640 1 -- 172.21.15.57:0/3874124258 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v223) v1 ==== 74+0+16839 (secure 0 0 0) 0x7f85ac01f090 con 0x7f85b8138940 2023-10-19T14:20:20.340 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:20:20.340 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":223,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T14:20:20.325160+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2023-10-19T14:20:17.357433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"222","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2023-10-19T14:20:18.049516+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"223","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T14:20:20.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.338+0000 7f85c0561640 1 -- 172.21.15.57:0/3874124258 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f85a406af90 msgr2=0x7f85a406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:20.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.338+0000 7f85c0561640 1 --2- 172.21.15.57:0/3874124258 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f85a406af90 0x7f85a406d450 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f85b0000c00 tx=0x7f85b0006c60 comp rx=0 tx=0).stop 2023-10-19T14:20:20.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.338+0000 7f85c0561640 1 -- 172.21.15.57:0/3874124258 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85b8138940 msgr2=0x7f85b814bf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:20.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.338+0000 7f85c0561640 1 --2- 172.21.15.57:0/3874124258 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85b8138940 0x7f85b814bf00 secure :-1 s=READY pgs=2348 cs=0 l=1 rev1=1 crypto rx=0x7f85ac005f60 tx=0x7f85ac011cb0 comp rx=0 tx=0).stop 2023-10-19T14:20:20.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.338+0000 7f85c0561640 1 -- 172.21.15.57:0/3874124258 shutdown_connections 2023-10-19T14:20:20.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.338+0000 7f85c0561640 1 --2- 172.21.15.57:0/3874124258 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f85a406af90 0x7f85a406d450 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:20.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.338+0000 7f85c0561640 1 --2- 172.21.15.57:0/3874124258 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f85b814fdd0 0x7f85b8150230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:20.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.338+0000 7f85c0561640 1 --2- 172.21.15.57:0/3874124258 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85b8138940 0x7f85b814bf00 unknown :-1 s=CLOSED pgs=2348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:20.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.339+0000 7f85c0561640 1 --2- 172.21.15.57:0/3874124258 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f85a0006290 0x7f85b8137d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:20.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.339+0000 7f85c0561640 1 -- 172.21.15.57:0/3874124258 >> 172.21.15.57:0/3874124258 conn(0x7f85b8069930 msgr2=0x7f85b8069d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:20.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.339+0000 7f85c0561640 1 -- 172.21.15.57:0/3874124258 shutdown_connections 2023-10-19T14:20:20.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.339+0000 7f85c0561640 1 -- 172.21.15.57:0/3874124258 wait complete. 2023-10-19T14:20:20.352 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-19T14:20:20.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.462+0000 7f9fc5075640 1 -- 172.21.15.57:0/1696865121 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9fc015b2a0 msgr2=0x7f9fc00cf720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:20.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.462+0000 7f9fc5075640 1 --2- 172.21.15.57:0/1696865121 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9fc015b2a0 0x7f9fc00cf720 secure :-1 s=READY pgs=2349 cs=0 l=1 rev1=1 crypto rx=0x7f9fc008a810 tx=0x7f9fb402bd10 comp rx=0 tx=0).stop 2023-10-19T14:20:20.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.462+0000 7f9fc5075640 1 -- 172.21.15.57:0/1696865121 shutdown_connections 2023-10-19T14:20:20.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.462+0000 7f9fc5075640 1 --2- 172.21.15.57:0/1696865121 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9fc00ceb40 0x7f9fc00b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:20.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.462+0000 7f9fc5075640 1 --2- 172.21.15.57:0/1696865121 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9fc015b2a0 0x7f9fc00cf720 unknown :-1 s=CLOSED pgs=2349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:20.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.462+0000 7f9fc5075640 1 --2- 172.21.15.57:0/1696865121 >> v2:172.21.15.179:3300/0 conn(0x7f9fc00cf340 0x7f9fc00d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:20.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.462+0000 7f9fc5075640 1 -- 172.21.15.57:0/1696865121 >> 172.21.15.57:0/1696865121 conn(0x7f9fc00d0d60 msgr2=0x7f9fc00d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:20.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.462+0000 7f9fc5075640 1 -- 172.21.15.57:0/1696865121 shutdown_connections 2023-10-19T14:20:20.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.462+0000 7f9fc5075640 1 -- 172.21.15.57:0/1696865121 wait complete. 2023-10-19T14:20:20.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.463+0000 7f9fc5075640 1 Processor -- start 2023-10-19T14:20:20.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.463+0000 7f9fc5075640 1 -- start start 2023-10-19T14:20:20.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.464+0000 7f9fc5075640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9fc00ceb40 0x7f9fc01f22a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:20.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.464+0000 7f9fc5075640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9fc00cf340 0x7f9fc00cc630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:20.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.464+0000 7f9fc5075640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9fc015b2a0 0x7f9fc00ccb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:20.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.464+0000 7f9fc5075640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fc00b6b40 con 0x7f9fc00ceb40 2023-10-19T14:20:20.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.464+0000 7f9fc5075640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fc0159bc0 con 0x7f9fc015b2a0 2023-10-19T14:20:20.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.464+0000 7f9fc5075640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fc00b6580 con 0x7f9fc00cf340 2023-10-19T14:20:20.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.464+0000 7f9fbed9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9fc00ceb40 0x7f9fc01f22a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:20.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.464+0000 7f9fbed9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9fc00ceb40 0x7f9fc01f22a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42034/0 (socket says 172.21.15.57:42034) 2023-10-19T14:20:20.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.464+0000 7f9fbed9d640 1 -- 172.21.15.57:0/2020969185 learned_addr learned my addr 172.21.15.57:0/2020969185 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:20.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.464+0000 7f9fbe59c640 1 --2- 172.21.15.57:0/2020969185 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9fc00cf340 0x7f9fc00cc630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:20.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.464+0000 7f9fbf59e640 1 --2- 172.21.15.57:0/2020969185 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9fc015b2a0 0x7f9fc00ccb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:20.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.465+0000 7f9fbed9d640 1 -- 172.21.15.57:0/2020969185 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9fc00cf340 msgr2=0x7f9fc00cc630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:20.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.465+0000 7f9fbed9d640 1 --2- 172.21.15.57:0/2020969185 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9fc00cf340 0x7f9fc00cc630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:20.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.465+0000 7f9fbed9d640 1 -- 172.21.15.57:0/2020969185 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9fc015b2a0 msgr2=0x7f9fc00ccb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:20.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.465+0000 7f9fbed9d640 1 --2- 172.21.15.57:0/2020969185 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9fc015b2a0 0x7f9fc00ccb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:20.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.465+0000 7f9fbed9d640 1 -- 172.21.15.57:0/2020969185 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fb4009560 con 0x7f9fc00ceb40 2023-10-19T14:20:20.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.465+0000 7f9fbed9d640 1 --2- 172.21.15.57:0/2020969185 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9fc00ceb40 0x7f9fc01f22a0 secure :-1 s=READY pgs=2350 cs=0 l=1 rev1=1 crypto rx=0x7f9fb800c920 tx=0x7f9fb800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:20.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.465+0000 7f9fa77fe640 1 -- 172.21.15.57:0/2020969185 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fb8007c70 con 0x7f9fc00ceb40 2023-10-19T14:20:20.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.465+0000 7f9fa77fe640 1 -- 172.21.15.57:0/2020969185 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f9fb800f450 con 0x7f9fc00ceb40 2023-10-19T14:20:20.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.465+0000 7f9fa77fe640 1 -- 172.21.15.57:0/2020969185 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fb800f860 con 0x7f9fc00ceb40 2023-10-19T14:20:20.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.465+0000 7f9fc5075640 1 -- 172.21.15.57:0/2020969185 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fc00b6860 con 0x7f9fc00ceb40 2023-10-19T14:20:20.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.465+0000 7f9fc5075640 1 -- 172.21.15.57:0/2020969185 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fc00b3cf0 con 0x7f9fc00ceb40 2023-10-19T14:20:20.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.466+0000 7f9fc5075640 1 -- 172.21.15.57:0/2020969185 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fc00b3ee0 con 0x7f9fc00ceb40 2023-10-19T14:20:20.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.467+0000 7f9fa77fe640 1 -- 172.21.15.57:0/2020969185 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f9fb800f9c0 con 0x7f9fc00ceb40 2023-10-19T14:20:20.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.467+0000 7f9fa77fe640 1 --2- 172.21.15.57:0/2020969185 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f9fac06af60 0x7f9fac06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:20.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.467+0000 7f9fa77fe640 1 -- 172.21.15.57:0/2020969185 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f9fb808a210 con 0x7f9fc00ceb40 2023-10-19T14:20:20.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.467+0000 7f9fbe59c640 1 --2- 172.21.15.57:0/2020969185 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f9fac06af60 0x7f9fac06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:20.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.468+0000 7f9fbe59c640 1 --2- 172.21.15.57:0/2020969185 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f9fac06af60 0x7f9fac06d420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9fb4035000 tx=0x7f9fb4004730 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:20.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.471+0000 7f9fa77fe640 1 -- 172.21.15.57:0/2020969185 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9fb8054fd0 con 0x7f9fc00ceb40 2023-10-19T14:20:20.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.682+0000 7f9fc5075640 1 -- 172.21.15.57:0/2020969185 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f9fc00690e0 con 0x7f9fc00ceb40 2023-10-19T14:20:20.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.721+0000 7f9fa77fe640 1 -- 172.21.15.57:0/2020969185 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v89) v1 ==== 118+0+0 (secure 0 0 0) 0x7f9fb80585f0 con 0x7f9fc00ceb40 2023-10-19T14:20:20.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.723+0000 7f9fc5075640 1 -- 172.21.15.57:0/2020969185 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f9fac06af60 msgr2=0x7f9fac06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:20.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.723+0000 7f9fc5075640 1 --2- 172.21.15.57:0/2020969185 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f9fac06af60 0x7f9fac06d420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9fb4035000 tx=0x7f9fb4004730 comp rx=0 tx=0).stop 2023-10-19T14:20:20.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.723+0000 7f9fc5075640 1 -- 172.21.15.57:0/2020969185 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9fc00ceb40 msgr2=0x7f9fc01f22a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:20.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.723+0000 7f9fc5075640 1 --2- 172.21.15.57:0/2020969185 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9fc00ceb40 0x7f9fc01f22a0 secure :-1 s=READY pgs=2350 cs=0 l=1 rev1=1 crypto rx=0x7f9fb800c920 tx=0x7f9fb800cde0 comp rx=0 tx=0).stop 2023-10-19T14:20:20.727 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.723+0000 7f9fc5075640 1 -- 172.21.15.57:0/2020969185 shutdown_connections 2023-10-19T14:20:20.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.723+0000 7f9fc5075640 1 --2- 172.21.15.57:0/2020969185 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f9fac06af60 0x7f9fac06d420 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:20.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.723+0000 7f9fc5075640 1 --2- 172.21.15.57:0/2020969185 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9fc015b2a0 0x7f9fc00ccb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:20.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.724+0000 7f9fc5075640 1 --2- 172.21.15.57:0/2020969185 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9fc00cf340 0x7f9fc00cc630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:20.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.724+0000 7f9fc5075640 1 --2- 172.21.15.57:0/2020969185 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9fc00ceb40 0x7f9fc01f22a0 unknown :-1 s=CLOSED pgs=2350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:20.728 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.724+0000 7f9fc5075640 1 -- 172.21.15.57:0/2020969185 >> 172.21.15.57:0/2020969185 conn(0x7f9fc00d0d60 msgr2=0x7f9fc006b320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:20.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.724+0000 7f9fc5075640 1 -- 172.21.15.57:0/2020969185 shutdown_connections 2023-10-19T14:20:20.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.724+0000 7f9fc5075640 1 -- 172.21.15.57:0/2020969185 wait complete. 2023-10-19T14:20:20.737 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-19T14:20:20.738 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-19T14:20:20.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.848+0000 7fd944464640 1 -- 172.21.15.57:0/4094263158 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd928004d00 msgr2=0x7fd928005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:20.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.848+0000 7fd944464640 1 --2- 172.21.15.57:0/4094263158 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd928004d00 0x7fd928005180 secure :-1 s=READY pgs=2352 cs=0 l=1 rev1=1 crypto rx=0x7fd934001d80 tx=0x7fd9280057f0 comp rx=0 tx=0).stop 2023-10-19T14:20:20.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.848+0000 7fd942a01640 1 -- 172.21.15.57:0/4094263158 reap_dead start 2023-10-19T14:20:20.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.848+0000 7fd944464640 1 -- 172.21.15.57:0/4094263158 shutdown_connections 2023-10-19T14:20:20.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.848+0000 7fd944464640 1 -- 172.21.15.57:0/4094263158 >> 172.21.15.57:0/4094263158 conn(0x7fd93c069930 msgr2=0x7fd93c069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:20.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.848+0000 7fd944464640 1 -- 172.21.15.57:0/4094263158 shutdown_connections 2023-10-19T14:20:20.853 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.848+0000 7fd944464640 1 -- 172.21.15.57:0/4094263158 wait complete. 2023-10-19T14:20:20.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.849+0000 7fd944464640 1 Processor -- start 2023-10-19T14:20:20.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.849+0000 7fd944464640 1 -- start start 2023-10-19T14:20:20.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.850+0000 7fd944464640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd928004d00 0x7fd93c0ae8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:20.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.850+0000 7fd944464640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd93c0adf20 0x7fd93c13a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:20.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.850+0000 7fd944464640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd93c1e1950 0x7fd93c1e1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:20.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.850+0000 7fd944464640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd93c1e0c90 con 0x7fd93c1e1950 2023-10-19T14:20:20.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.850+0000 7fd944464640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd93c0ae3b0 con 0x7fd93c0adf20 2023-10-19T14:20:20.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.850+0000 7fd944464640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd93c1e0e70 con 0x7fd928004d00 2023-10-19T14:20:20.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.850+0000 7fd942a01640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd93c1e1950 0x7fd93c1e1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:20.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.850+0000 7fd942a01640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd93c1e1950 0x7fd93c1e1dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42054/0 (socket says 172.21.15.57:42054) 2023-10-19T14:20:20.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.850+0000 7fd942a01640 1 -- 172.21.15.57:0/2926176213 learned_addr learned my addr 172.21.15.57:0/2926176213 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:20.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.850+0000 7fd9419ff640 1 --2- 172.21.15.57:0/2926176213 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd93c0adf20 0x7fd93c13a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:20.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.850+0000 7fd942200640 1 --2- 172.21.15.57:0/2926176213 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd928004d00 0x7fd93c0ae8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:20.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.850+0000 7fd942a01640 1 -- 172.21.15.57:0/2926176213 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd928004d00 msgr2=0x7fd93c0ae8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:20.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.850+0000 7fd942a01640 1 --2- 172.21.15.57:0/2926176213 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd928004d00 0x7fd93c0ae8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:20.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.850+0000 7fd942a01640 1 -- 172.21.15.57:0/2926176213 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd93c0adf20 msgr2=0x7fd93c13a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:20.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.850+0000 7fd942a01640 1 --2- 172.21.15.57:0/2926176213 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd93c0adf20 0x7fd93c13a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:20.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.850+0000 7fd942a01640 1 -- 172.21.15.57:0/2926176213 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd93c1e1150 con 0x7fd93c1e1950 2023-10-19T14:20:20.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.851+0000 7fd942a01640 1 --2- 172.21.15.57:0/2926176213 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd93c1e1950 0x7fd93c1e1dd0 secure :-1 s=READY pgs=2353 cs=0 l=1 rev1=1 crypto rx=0x7fd934001ea0 tx=0x7fd934003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:20.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.851+0000 7fd932ffd640 1 -- 172.21.15.57:0/2926176213 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd93402ccd0 con 0x7fd93c1e1950 2023-10-19T14:20:20.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.851+0000 7fd932ffd640 1 -- 172.21.15.57:0/2926176213 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fd9340060b0 con 0x7fd93c1e1950 2023-10-19T14:20:20.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.851+0000 7fd944464640 1 -- 172.21.15.57:0/2926176213 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd93c1e1430 con 0x7fd93c1e1950 2023-10-19T14:20:20.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.851+0000 7fd932ffd640 1 -- 172.21.15.57:0/2926176213 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd934006480 con 0x7fd93c1e1950 2023-10-19T14:20:20.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.851+0000 7fd944464640 1 -- 172.21.15.57:0/2926176213 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd93c149a60 con 0x7fd93c1e1950 2023-10-19T14:20:20.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.852+0000 7fd944464640 1 -- 172.21.15.57:0/2926176213 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd93c06bb60 con 0x7fd93c1e1950 2023-10-19T14:20:20.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.852+0000 7fd932ffd640 1 -- 172.21.15.57:0/2926176213 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd9340065e0 con 0x7fd93c1e1950 2023-10-19T14:20:20.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.853+0000 7fd932ffd640 1 --2- 172.21.15.57:0/2926176213 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fd92006af90 0x7fd92006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:20.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.853+0000 7fd932ffd640 1 -- 172.21.15.57:0/2926176213 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fd9340aa9c0 con 0x7fd93c1e1950 2023-10-19T14:20:20.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.853+0000 7fd942200640 1 --2- 172.21.15.57:0/2926176213 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fd92006af90 0x7fd92006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:20.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.853+0000 7fd942200640 1 --2- 172.21.15.57:0/2926176213 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fd92006af90 0x7fd92006d450 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd9380017a0 tx=0x7fd938001550 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:20.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:20.857+0000 7fd932ffd640 1 -- 172.21.15.57:0/2926176213 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd934075890 con 0x7fd93c1e1950 2023-10-19T14:20:21.073 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:21.071+0000 7fd944464640 1 -- 172.21.15.57:0/2926176213 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fd93c0690e0 con 0x7fd93c1e1950 2023-10-19T14:20:21.672 INFO:tasks.ceph.mds.a.smithi057.stderr:2023-10-19T14:20:21.670+0000 7f2368f2e640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-19T14:20:21.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:21.723+0000 7fd932ffd640 1 -- 172.21.15.57:0/2926176213 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v90) v1 ==== 122+0+0 (secure 0 0 0) 0x7fd934044020 con 0x7fd93c1e1950 2023-10-19T14:20:21.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:21.726+0000 7fd944464640 1 -- 172.21.15.57:0/2926176213 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fd92006af90 msgr2=0x7fd92006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:21.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:21.726+0000 7fd944464640 1 --2- 172.21.15.57:0/2926176213 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fd92006af90 0x7fd92006d450 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd9380017a0 tx=0x7fd938001550 comp rx=0 tx=0).stop 2023-10-19T14:20:21.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:21.726+0000 7fd944464640 1 -- 172.21.15.57:0/2926176213 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd93c1e1950 msgr2=0x7fd93c1e1dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:21.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:21.726+0000 7fd944464640 1 --2- 172.21.15.57:0/2926176213 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd93c1e1950 0x7fd93c1e1dd0 secure :-1 s=READY pgs=2353 cs=0 l=1 rev1=1 crypto rx=0x7fd934001ea0 tx=0x7fd934003ba0 comp rx=0 tx=0).stop 2023-10-19T14:20:21.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:21.726+0000 7fd944464640 1 -- 172.21.15.57:0/2926176213 shutdown_connections 2023-10-19T14:20:21.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:21.726+0000 7fd944464640 1 --2- 172.21.15.57:0/2926176213 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fd92006af90 0x7fd92006d450 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:21.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:21.726+0000 7fd944464640 1 --2- 172.21.15.57:0/2926176213 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd93c1e1950 0x7fd93c1e1dd0 unknown :-1 s=CLOSED pgs=2353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:21.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:21.726+0000 7fd944464640 1 --2- 172.21.15.57:0/2926176213 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd93c0adf20 0x7fd93c13a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:21.730 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:21.726+0000 7fd944464640 1 --2- 172.21.15.57:0/2926176213 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd928004d00 0x7fd93c0ae8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:21.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:21.726+0000 7fd944464640 1 -- 172.21.15.57:0/2926176213 >> 172.21.15.57:0/2926176213 conn(0x7fd93c069930 msgr2=0x7fd93c06b200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:21.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:21.726+0000 7fd944464640 1 -- 172.21.15.57:0/2926176213 shutdown_connections 2023-10-19T14:20:21.731 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:21.726+0000 7fd944464640 1 -- 172.21.15.57:0/2926176213 wait complete. 2023-10-19T14:20:21.742 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-19T14:20:21.743 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T14:20:21.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:21.856+0000 7f2c82a07640 1 -- 172.21.15.57:0/522548467 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2c7c0b1920 msgr2=0x7f2c7c06fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:21.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:21.856+0000 7f2c82a07640 1 --2- 172.21.15.57:0/522548467 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2c7c0b1920 0x7f2c7c06fa70 secure :-1 s=READY pgs=2355 cs=0 l=1 rev1=1 crypto rx=0x7f2c7c0abdc0 tx=0x7f2c6402bd70 comp rx=0 tx=0).stop 2023-10-19T14:20:21.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:21.856+0000 7f2c82a07640 1 -- 172.21.15.57:0/522548467 shutdown_connections 2023-10-19T14:20:21.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:21.856+0000 7f2c82a07640 1 --2- 172.21.15.57:0/522548467 >> v2:172.21.15.179:3300/0 conn(0x7f2c7c06ffb0 0x7f2c7c0c96b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:21.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:21.856+0000 7f2c82a07640 1 --2- 172.21.15.57:0/522548467 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2c7c0b1920 0x7f2c7c06fa70 unknown :-1 s=CLOSED pgs=2355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:21.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:21.856+0000 7f2c82a07640 1 -- 172.21.15.57:0/522548467 >> 172.21.15.57:0/522548467 conn(0x7f2c7c069930 msgr2=0x7f2c7c069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:21.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:21.856+0000 7f2c82a07640 1 -- 172.21.15.57:0/522548467 shutdown_connections 2023-10-19T14:20:21.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:21.856+0000 7f2c82a07640 1 -- 172.21.15.57:0/522548467 wait complete. 2023-10-19T14:20:21.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:21.857+0000 7f2c82a07640 1 Processor -- start 2023-10-19T14:20:21.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:21.857+0000 7f2c82a07640 1 -- start start 2023-10-19T14:20:21.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:21.858+0000 7f2c82a07640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2c7c06ffb0 0x7f2c7c0c8e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:21.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:21.858+0000 7f2c82a07640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2c7c0b1920 0x7f2c7c0c1990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:21.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:21.858+0000 7f2c82a07640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2c7c0c9bf0 0x7f2c7c1f4b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:21.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:21.858+0000 7f2c82a07640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c7c15c6d0 con 0x7f2c7c0c9bf0 2023-10-19T14:20:21.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:21.858+0000 7f2c82a07640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c7c1564a0 con 0x7f2c7c0b1920 2023-10-19T14:20:21.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:21.858+0000 7f2c82a07640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c7c155f20 con 0x7f2c7c06ffb0 2023-10-19T14:20:21.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:21.858+0000 7f2c80fa4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2c7c0c9bf0 0x7f2c7c1f4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:21.868 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:21.858+0000 7f2c80fa4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2c7c0c9bf0 0x7f2c7c1f4b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42076/0 (socket says 172.21.15.57:42076) 2023-10-19T14:20:21.868 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:21.858+0000 7f2c80fa4640 1 -- 172.21.15.57:0/3391418628 learned_addr learned my addr 172.21.15.57:0/3391418628 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:21.868 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:21.858+0000 7f2c7b7fe640 1 --2- 172.21.15.57:0/3391418628 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2c7c0b1920 0x7f2c7c0c1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:21.868 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:21.859+0000 7f2c7bfff640 1 --2- 172.21.15.57:0/3391418628 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2c7c06ffb0 0x7f2c7c0c8e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:21.869 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:21.859+0000 7f2c80fa4640 1 -- 172.21.15.57:0/3391418628 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2c7c06ffb0 msgr2=0x7f2c7c0c8e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:21.869 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:21.859+0000 7f2c80fa4640 1 --2- 172.21.15.57:0/3391418628 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2c7c06ffb0 0x7f2c7c0c8e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:21.869 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:21.859+0000 7f2c80fa4640 1 -- 172.21.15.57:0/3391418628 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2c7c0b1920 msgr2=0x7f2c7c0c1990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:21.869 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:21.859+0000 7f2c80fa4640 1 --2- 172.21.15.57:0/3391418628 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2c7c0b1920 0x7f2c7c0c1990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:21.869 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:21.859+0000 7f2c80fa4640 1 -- 172.21.15.57:0/3391418628 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c64009490 con 0x7f2c7c0c9bf0 2023-10-19T14:20:21.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:21.859+0000 7f2c80fa4640 1 --2- 172.21.15.57:0/3391418628 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2c7c0c9bf0 0x7f2c7c1f4b10 secure :-1 s=READY pgs=2356 cs=0 l=1 rev1=1 crypto rx=0x7f2c7400cb40 tx=0x7f2c7400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:21.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:21.859+0000 7f2c78ff9640 1 -- 172.21.15.57:0/3391418628 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c7400aca0 con 0x7f2c7c0c9bf0 2023-10-19T14:20:21.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:21.859+0000 7f2c78ff9640 1 -- 172.21.15.57:0/3391418628 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f2c74012420 con 0x7f2c7c0c9bf0 2023-10-19T14:20:21.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:21.859+0000 7f2c82a07640 1 -- 172.21.15.57:0/3391418628 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c7c0c1ed0 con 0x7f2c7c0c9bf0 2023-10-19T14:20:21.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:21.859+0000 7f2c78ff9640 1 -- 172.21.15.57:0/3391418628 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c74012850 con 0x7f2c7c0c9bf0 2023-10-19T14:20:21.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:21.859+0000 7f2c82a07640 1 -- 172.21.15.57:0/3391418628 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c7c1561c0 con 0x7f2c7c0c9bf0 2023-10-19T14:20:21.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:21.860+0000 7f2c82a07640 1 -- 172.21.15.57:0/3391418628 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c7c15c840 con 0x7f2c7c0c9bf0 2023-10-19T14:20:21.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:21.861+0000 7f2c78ff9640 1 -- 172.21.15.57:0/3391418628 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f2c7401a3b0 con 0x7f2c7c0c9bf0 2023-10-19T14:20:21.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:21.861+0000 7f2c78ff9640 1 --2- 172.21.15.57:0/3391418628 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f2c5c06af60 0x7f2c5c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:21.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:21.861+0000 7f2c78ff9640 1 -- 172.21.15.57:0/3391418628 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f2c74089ed0 con 0x7f2c7c0c9bf0 2023-10-19T14:20:21.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:21.861+0000 7f2c7bfff640 1 --2- 172.21.15.57:0/3391418628 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f2c5c06af60 0x7f2c5c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:21.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:21.862+0000 7f2c7bfff640 1 --2- 172.21.15.57:0/3391418628 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f2c5c06af60 0x7f2c5c06d420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f2c64036000 tx=0x7f2c64004730 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:21.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:21.865+0000 7f2c78ff9640 1 -- 172.21.15.57:0/3391418628 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2c7c15c840 con 0x7f2c7c0c9bf0 2023-10-19T14:20:22.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.078+0000 7f2c82a07640 1 -- 172.21.15.57:0/3391418628 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2c7c15c360 con 0x7f2c7c0c9bf0 2023-10-19T14:20:22.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.079+0000 7f2c78ff9640 1 -- 172.21.15.57:0/3391418628 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+4928 (secure 0 0 0) 0x7f2c74054da0 con 0x7f2c7c0c9bf0 2023-10-19T14:20:22.084 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:20:22.084 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":90,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9248,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/1249807745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":1249807745},{"type":"v1","addr":"172.21.15.179:6836","nonce":1249807745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87},{"gid":10513,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/1406825818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":1406825818},{"type":"v1","addr":"172.21.15.179:6837","nonce":1406825818}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":86},{"gid":11715,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/2467807803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2467807803},{"type":"v1","addr":"172.21.15.57:6834","nonce":2467807803}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T14:20:19.077396+0000","modified":"2023-10-19T14:20:21.072483+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11718},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11718":{"gid":11718,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.57:6835/1529368280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1529368280},{"type":"v1","addr":"172.21.15.57:6835","nonce":1529368280}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13}]} 2023-10-19T14:20:22.085 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 90 2023-10-19T14:20:22.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.081+0000 7f2c82a07640 1 -- 172.21.15.57:0/3391418628 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f2c5c06af60 msgr2=0x7f2c5c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:22.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.081+0000 7f2c82a07640 1 --2- 172.21.15.57:0/3391418628 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f2c5c06af60 0x7f2c5c06d420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f2c64036000 tx=0x7f2c64004730 comp rx=0 tx=0).stop 2023-10-19T14:20:22.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.081+0000 7f2c82a07640 1 -- 172.21.15.57:0/3391418628 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2c7c0c9bf0 msgr2=0x7f2c7c1f4b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:22.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.081+0000 7f2c82a07640 1 --2- 172.21.15.57:0/3391418628 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2c7c0c9bf0 0x7f2c7c1f4b10 secure :-1 s=READY pgs=2356 cs=0 l=1 rev1=1 crypto rx=0x7f2c7400cb40 tx=0x7f2c7400a060 comp rx=0 tx=0).stop 2023-10-19T14:20:22.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.081+0000 7f2c82a07640 1 -- 172.21.15.57:0/3391418628 shutdown_connections 2023-10-19T14:20:22.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.081+0000 7f2c82a07640 1 --2- 172.21.15.57:0/3391418628 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f2c5c06af60 0x7f2c5c06d420 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:22.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.081+0000 7f2c82a07640 1 --2- 172.21.15.57:0/3391418628 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2c7c0c9bf0 0x7f2c7c1f4b10 unknown :-1 s=CLOSED pgs=2356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:22.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.081+0000 7f2c82a07640 1 --2- 172.21.15.57:0/3391418628 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2c7c0b1920 0x7f2c7c0c1990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:22.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.081+0000 7f2c82a07640 1 --2- 172.21.15.57:0/3391418628 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2c7c06ffb0 0x7f2c7c0c8e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:22.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.081+0000 7f2c82a07640 1 -- 172.21.15.57:0/3391418628 >> 172.21.15.57:0/3391418628 conn(0x7f2c7c069930 msgr2=0x7f2c7c069d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:22.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.081+0000 7f2c82a07640 1 -- 172.21.15.57:0/3391418628 shutdown_connections 2023-10-19T14:20:22.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.081+0000 7f2c82a07640 1 -- 172.21.15.57:0/3391418628 wait complete. 2023-10-19T14:20:22.095 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T14:20:22.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.202+0000 7fe56ec90640 1 -- 172.21.15.57:0/2176109616 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe56806ae40 msgr2=0x7fe5680b0e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:22.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.202+0000 7fe56ec90640 1 --2- 172.21.15.57:0/2176109616 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe56806ae40 0x7fe5680b0e90 secure :-1 s=READY pgs=2358 cs=0 l=1 rev1=1 crypto rx=0x7fe5680acd30 tx=0x7fe55402bd70 comp rx=0 tx=0).stop 2023-10-19T14:20:22.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.202+0000 7fe56ec90640 1 -- 172.21.15.57:0/2176109616 shutdown_connections 2023-10-19T14:20:22.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.202+0000 7fe56ec90640 1 --2- 172.21.15.57:0/2176109616 >> v2:172.21.15.179:3300/0 conn(0x7fe56815c1f0 0x7fe568157dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:22.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.202+0000 7fe56ec90640 1 --2- 172.21.15.57:0/2176109616 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe56806ae40 0x7fe5680b0e90 unknown :-1 s=CLOSED pgs=2358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:22.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.202+0000 7fe56ec90640 1 -- 172.21.15.57:0/2176109616 >> 172.21.15.57:0/2176109616 conn(0x7fe56806fae0 msgr2=0x7fe56806fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:22.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.202+0000 7fe56ec90640 1 -- 172.21.15.57:0/2176109616 shutdown_connections 2023-10-19T14:20:22.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.202+0000 7fe56ec90640 1 -- 172.21.15.57:0/2176109616 wait complete. 2023-10-19T14:20:22.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.203+0000 7fe56ec90640 1 Processor -- start 2023-10-19T14:20:22.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.203+0000 7fe56ec90640 1 -- start start 2023-10-19T14:20:22.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.204+0000 7fe56ec90640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe56806ae40 0x7fe56814ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:22.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.204+0000 7fe56ec90640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe5681579f0 0x7fe568159bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:22.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.204+0000 7fe56ec90640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe56815c1f0 0x7fe56815a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:22.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.204+0000 7fe56ec90640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe56814ae20 con 0x7fe56815c1f0 2023-10-19T14:20:22.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.204+0000 7fe56ec90640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5680ab940 con 0x7fe56806ae40 2023-10-19T14:20:22.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.204+0000 7fe56ec90640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5680abc20 con 0x7fe5681579f0 2023-10-19T14:20:22.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.204+0000 7fe56d22d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe56815c1f0 0x7fe56815a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:22.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.204+0000 7fe56d22d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe56815c1f0 0x7fe56815a110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42088/0 (socket says 172.21.15.57:42088) 2023-10-19T14:20:22.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.204+0000 7fe56d22d640 1 -- 172.21.15.57:0/2615058469 learned_addr learned my addr 172.21.15.57:0/2615058469 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:22.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.205+0000 7fe56ca2c640 1 --2- 172.21.15.57:0/2615058469 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe56806ae40 0x7fe56814ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:22.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.205+0000 7fe567fff640 1 --2- 172.21.15.57:0/2615058469 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe5681579f0 0x7fe568159bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:22.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.205+0000 7fe56d22d640 1 -- 172.21.15.57:0/2615058469 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe5681579f0 msgr2=0x7fe568159bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:22.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.205+0000 7fe56d22d640 1 --2- 172.21.15.57:0/2615058469 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe5681579f0 0x7fe568159bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:22.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.205+0000 7fe56d22d640 1 -- 172.21.15.57:0/2615058469 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe56806ae40 msgr2=0x7fe56814ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:22.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.205+0000 7fe56d22d640 1 --2- 172.21.15.57:0/2615058469 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe56806ae40 0x7fe56814ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:22.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.205+0000 7fe56d22d640 1 -- 172.21.15.57:0/2615058469 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe554009490 con 0x7fe56815c1f0 2023-10-19T14:20:22.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.205+0000 7fe56d22d640 1 --2- 172.21.15.57:0/2615058469 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe56815c1f0 0x7fe56815a110 secure :-1 s=READY pgs=2359 cs=0 l=1 rev1=1 crypto rx=0x7fe56000cab0 tx=0x7fe56000cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:22.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.205+0000 7fe5657fa640 1 -- 172.21.15.57:0/2615058469 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe560007d90 con 0x7fe56815c1f0 2023-10-19T14:20:22.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.205+0000 7fe5657fa640 1 -- 172.21.15.57:0/2615058469 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fe56000f450 con 0x7fe56815c1f0 2023-10-19T14:20:22.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.205+0000 7fe56ec90640 1 -- 172.21.15.57:0/2615058469 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5680abf00 con 0x7fe56815c1f0 2023-10-19T14:20:22.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.205+0000 7fe5657fa640 1 -- 172.21.15.57:0/2615058469 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe56000f840 con 0x7fe56815c1f0 2023-10-19T14:20:22.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.205+0000 7fe56ec90640 1 -- 172.21.15.57:0/2615058469 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe56814d390 con 0x7fe56815c1f0 2023-10-19T14:20:22.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.206+0000 7fe56ec90640 1 -- 172.21.15.57:0/2615058469 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe56814d580 con 0x7fe56815c1f0 2023-10-19T14:20:22.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.206+0000 7fe5657fa640 1 -- 172.21.15.57:0/2615058469 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe56000f9a0 con 0x7fe56815c1f0 2023-10-19T14:20:22.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.207+0000 7fe5657fa640 1 --2- 172.21.15.57:0/2615058469 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fe54c06af60 0x7fe54c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:22.216 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.207+0000 7fe5657fa640 1 -- 172.21.15.57:0/2615058469 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fe56008a4b0 con 0x7fe56815c1f0 2023-10-19T14:20:22.216 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.207+0000 7fe56ca2c640 1 --2- 172.21.15.57:0/2615058469 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fe54c06af60 0x7fe54c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:22.216 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.208+0000 7fe56ca2c640 1 --2- 172.21.15.57:0/2615058469 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fe54c06af60 0x7fe54c06d420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe55c000f60 tx=0x7fe55c009040 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:22.217 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.211+0000 7fe5657fa640 1 -- 172.21.15.57:0/2615058469 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe56814d580 con 0x7fe56815c1f0 2023-10-19T14:20:22.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.387+0000 7fe56ec90640 1 -- 172.21.15.57:0/2615058469 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe56814d580 con 0x7fe56815c1f0 2023-10-19T14:20:22.391 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.388+0000 7fe5657fa640 1 -- 172.21.15.57:0/2615058469 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v223) v1 ==== 74+0+16839 (secure 0 0 0) 0x7fe56814d580 con 0x7fe56815c1f0 2023-10-19T14:20:22.392 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:20:22.392 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":223,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T14:20:20.325160+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2023-10-19T14:20:17.357433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"222","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2023-10-19T14:20:18.049516+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"223","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T14:20:22.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.390+0000 7fe56ec90640 1 -- 172.21.15.57:0/2615058469 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fe54c06af60 msgr2=0x7fe54c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:22.394 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.390+0000 7fe56ec90640 1 --2- 172.21.15.57:0/2615058469 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fe54c06af60 0x7fe54c06d420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe55c000f60 tx=0x7fe55c009040 comp rx=0 tx=0).stop 2023-10-19T14:20:22.394 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.390+0000 7fe56ec90640 1 -- 172.21.15.57:0/2615058469 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe56815c1f0 msgr2=0x7fe56815a110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:22.394 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.390+0000 7fe56ec90640 1 --2- 172.21.15.57:0/2615058469 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe56815c1f0 0x7fe56815a110 secure :-1 s=READY pgs=2359 cs=0 l=1 rev1=1 crypto rx=0x7fe56000cab0 tx=0x7fe56000cf70 comp rx=0 tx=0).stop 2023-10-19T14:20:22.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.390+0000 7fe56ec90640 1 -- 172.21.15.57:0/2615058469 shutdown_connections 2023-10-19T14:20:22.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.390+0000 7fe56ec90640 1 --2- 172.21.15.57:0/2615058469 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fe54c06af60 0x7fe54c06d420 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:22.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.390+0000 7fe56ec90640 1 --2- 172.21.15.57:0/2615058469 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe56815c1f0 0x7fe56815a110 unknown :-1 s=CLOSED pgs=2359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:22.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.390+0000 7fe56ec90640 1 --2- 172.21.15.57:0/2615058469 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe5681579f0 0x7fe568159bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:22.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.390+0000 7fe56ec90640 1 --2- 172.21.15.57:0/2615058469 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe56806ae40 0x7fe56814ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:22.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.390+0000 7fe56ec90640 1 -- 172.21.15.57:0/2615058469 >> 172.21.15.57:0/2615058469 conn(0x7fe56806fae0 msgr2=0x7fe568069b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:22.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.391+0000 7fe56ec90640 1 -- 172.21.15.57:0/2615058469 shutdown_connections 2023-10-19T14:20:22.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.391+0000 7fe56ec90640 1 -- 172.21.15.57:0/2615058469 wait complete. 2023-10-19T14:20:22.404 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-19T14:20:22.405 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-19T14:20:22.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.512+0000 7f392f3a9640 1 -- 172.21.15.57:0/701231903 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f39280b0530 msgr2=0x7f39280b09a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:22.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.512+0000 7f392f3a9640 1 --2- 172.21.15.57:0/701231903 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f39280b0530 0x7f39280b09a0 secure :-1 s=READY pgs=2360 cs=0 l=1 rev1=1 crypto rx=0x7f39280cc490 tx=0x7f392002c2f0 comp rx=0 tx=0).stop 2023-10-19T14:20:22.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.512+0000 7f392f3a9640 1 -- 172.21.15.57:0/701231903 shutdown_connections 2023-10-19T14:20:22.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.512+0000 7f392f3a9640 1 --2- 172.21.15.57:0/701231903 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f39280b0530 0x7f39280b09a0 unknown :-1 s=CLOSED pgs=2360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:22.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.512+0000 7f392f3a9640 1 --2- 172.21.15.57:0/701231903 >> v2:172.21.15.179:3300/0 conn(0x7f39280afa20 0x7f3928158300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:22.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.512+0000 7f392f3a9640 1 -- 172.21.15.57:0/701231903 >> 172.21.15.57:0/701231903 conn(0x7f3928069930 msgr2=0x7f3928069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:22.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.512+0000 7f392f3a9640 1 -- 172.21.15.57:0/701231903 shutdown_connections 2023-10-19T14:20:22.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.512+0000 7f392f3a9640 1 -- 172.21.15.57:0/701231903 wait complete. 2023-10-19T14:20:22.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.513+0000 7f392f3a9640 1 Processor -- start 2023-10-19T14:20:22.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.513+0000 7f392f3a9640 1 -- start start 2023-10-19T14:20:22.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.514+0000 7f392f3a9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f39280afa20 0x7f39280abd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:22.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.514+0000 7f392f3a9640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f39280b0530 0x7f39281e2240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:22.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.514+0000 7f392f3a9640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f39280cbd20 0x7f39281e3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:22.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.514+0000 7f392f3a9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39281eef50 con 0x7f39280afa20 2023-10-19T14:20:22.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.514+0000 7f392f3a9640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39281e3770 con 0x7f39280cbd20 2023-10-19T14:20:22.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.514+0000 7f392f3a9640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39281e2bf0 con 0x7f39280b0530 2023-10-19T14:20:22.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.514+0000 7f392d145640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f39280afa20 0x7f39280abd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:22.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.514+0000 7f392d145640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f39280afa20 0x7f39280abd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42102/0 (socket says 172.21.15.57:42102) 2023-10-19T14:20:22.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.514+0000 7f392d145640 1 -- 172.21.15.57:0/3577002225 learned_addr learned my addr 172.21.15.57:0/3577002225 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:22.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.514+0000 7f392d145640 1 -- 172.21.15.57:0/3577002225 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f39280b0530 msgr2=0x7f39281e2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:22.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.514+0000 7f392d145640 1 --2- 172.21.15.57:0/3577002225 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f39280b0530 0x7f39281e2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:22.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.514+0000 7f392d946640 1 --2- 172.21.15.57:0/3577002225 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f39280cbd20 0x7f39281e3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:22.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.514+0000 7f392d145640 1 -- 172.21.15.57:0/3577002225 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f39280cbd20 msgr2=0x7f39281e3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:22.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.514+0000 7f392d145640 1 --2- 172.21.15.57:0/3577002225 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f39280cbd20 0x7f39281e3ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:22.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.514+0000 7f392d145640 1 -- 172.21.15.57:0/3577002225 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3920009900 con 0x7f39280afa20 2023-10-19T14:20:22.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.514+0000 7f392d946640 1 --2- 172.21.15.57:0/3577002225 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f39280cbd20 0x7f39281e3ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:20:22.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.514+0000 7f392d145640 1 --2- 172.21.15.57:0/3577002225 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f39280afa20 0x7f39280abd10 secure :-1 s=READY pgs=2361 cs=0 l=1 rev1=1 crypto rx=0x7f392400c440 tx=0x7f392400c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:22.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.515+0000 7f391dffb640 1 -- 172.21.15.57:0/3577002225 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39240052d0 con 0x7f39280afa20 2023-10-19T14:20:22.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.515+0000 7f391dffb640 1 -- 172.21.15.57:0/3577002225 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f39240035f0 con 0x7f39280afa20 2023-10-19T14:20:22.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.515+0000 7f392f3a9640 1 -- 172.21.15.57:0/3577002225 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39281e2ed0 con 0x7f39280afa20 2023-10-19T14:20:22.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.515+0000 7f391dffb640 1 -- 172.21.15.57:0/3577002225 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3924003a00 con 0x7f39280afa20 2023-10-19T14:20:22.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.515+0000 7f392f3a9640 1 -- 172.21.15.57:0/3577002225 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39281e3490 con 0x7f39280afa20 2023-10-19T14:20:22.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.516+0000 7f392f3a9640 1 -- 172.21.15.57:0/3577002225 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39281ef0c0 con 0x7f39280afa20 2023-10-19T14:20:22.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.516+0000 7f391dffb640 1 -- 172.21.15.57:0/3577002225 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f3924003b60 con 0x7f39280afa20 2023-10-19T14:20:22.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.517+0000 7f391dffb640 1 --2- 172.21.15.57:0/3577002225 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f390c06af60 0x7f390c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:22.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.517+0000 7f391dffb640 1 -- 172.21.15.57:0/3577002225 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f3924089170 con 0x7f39280afa20 2023-10-19T14:20:22.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.517+0000 7f392c944640 1 --2- 172.21.15.57:0/3577002225 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f390c06af60 0x7f390c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:22.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.517+0000 7f392c944640 1 --2- 172.21.15.57:0/3577002225 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f390c06af60 0x7f390c06d420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f3918005fd0 tx=0x7f3918005850 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:22.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.521+0000 7f391dffb640 1 -- 172.21.15.57:0/3577002225 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f39281ef0c0 con 0x7f39280afa20 2023-10-19T14:20:22.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.695+0000 7f392f3a9640 1 -- 172.21.15.57:0/3577002225 --> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f39280cc100 con 0x7f390c06af60 2023-10-19T14:20:22.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.703+0000 7f391dffb640 1 -- 172.21.15.57:0/3577002225 <== mgr.9107 v2:172.21.15.179:6832/3232482678 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+290075 (secure 0 0 0) 0x7f3918012070 con 0x7f390c06af60 2023-10-19T14:20:22.711 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:20:22.713 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2023-10-19T14:20:22.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.706+0000 7f392f3a9640 1 -- 172.21.15.57:0/3577002225 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f390c06af60 msgr2=0x7f390c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:22.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.706+0000 7f392f3a9640 1 --2- 172.21.15.57:0/3577002225 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f390c06af60 0x7f390c06d420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f3918005fd0 tx=0x7f3918005850 comp rx=0 tx=0).stop 2023-10-19T14:20:22.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.706+0000 7f392f3a9640 1 -- 172.21.15.57:0/3577002225 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f39280afa20 msgr2=0x7f39280abd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:22.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.706+0000 7f392f3a9640 1 --2- 172.21.15.57:0/3577002225 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f39280afa20 0x7f39280abd10 secure :-1 s=READY pgs=2361 cs=0 l=1 rev1=1 crypto rx=0x7f392400c440 tx=0x7f392400c900 comp rx=0 tx=0).stop 2023-10-19T14:20:22.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.706+0000 7f392f3a9640 1 -- 172.21.15.57:0/3577002225 shutdown_connections 2023-10-19T14:20:22.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.706+0000 7f392f3a9640 1 --2- 172.21.15.57:0/3577002225 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f390c06af60 0x7f390c06d420 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:22.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.706+0000 7f392f3a9640 1 --2- 172.21.15.57:0/3577002225 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f39280cbd20 0x7f39281e3ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:22.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.706+0000 7f392f3a9640 1 --2- 172.21.15.57:0/3577002225 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f39280b0530 0x7f39281e2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:22.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.706+0000 7f392f3a9640 1 --2- 172.21.15.57:0/3577002225 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f39280afa20 0x7f39280abd10 unknown :-1 s=CLOSED pgs=2361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:22.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.706+0000 7f392f3a9640 1 -- 172.21.15.57:0/3577002225 >> 172.21.15.57:0/3577002225 conn(0x7f3928069930 msgr2=0x7f3928069d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:22.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.706+0000 7f392f3a9640 1 -- 172.21.15.57:0/3577002225 shutdown_connections 2023-10-19T14:20:22.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.706+0000 7f392f3a9640 1 -- 172.21.15.57:0/3577002225 wait complete. 2023-10-19T14:20:22.720 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2023-10-19T14:20:21.988889+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2210,"num_read_kb":4624,"num_write":633,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3224,"ondisk_log_size":3224,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":156,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":411380,"kb_used_data":23096,"kb_used_omap":12,"kb_used_meta":388147,"kb_avail":754563340,"statfs":{"total":773094113280,"available":772672860160,"internally_reserved":0,"allocated":23650304,"data_stored":12655921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":397463097},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.291526"},"pg_stats":[{"pgid":"28.3f","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.332591+0000","last_change":"2023-10-19T14:20:19.348564+0000","last_active":"2023-10-19T14:20:20.332591+0000","last_peered":"2023-10-19T14:20:20.332591+0000","last_clean":"2023-10-19T14:20:20.332591+0000","last_became_active":"2023-10-19T14:20:19.348281+0000","last_became_peered":"2023-10-19T14:20:19.348281+0000","last_unstale":"2023-10-19T14:20:20.332591+0000","last_undegraded":"2023-10-19T14:20:20.332591+0000","last_fullsized":"2023-10-19T14:20:20.332591+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:15:21.019725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3e","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.331656+0000","last_change":"2023-10-19T14:20:19.346014+0000","last_active":"2023-10-19T14:20:20.331656+0000","last_peered":"2023-10-19T14:20:20.331656+0000","last_clean":"2023-10-19T14:20:20.331656+0000","last_became_active":"2023-10-19T14:20:19.345686+0000","last_became_peered":"2023-10-19T14:20:19.345686+0000","last_unstale":"2023-10-19T14:20:20.331656+0000","last_undegraded":"2023-10-19T14:20:20.331656+0000","last_fullsized":"2023-10-19T14:20:20.331656+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:29:26.735942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.3d","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.922247+0000","last_change":"2023-10-19T14:20:19.349659+0000","last_active":"2023-10-19T14:20:20.922247+0000","last_peered":"2023-10-19T14:20:20.922247+0000","last_clean":"2023-10-19T14:20:20.922247+0000","last_became_active":"2023-10-19T14:20:19.349362+0000","last_became_peered":"2023-10-19T14:20:19.349362+0000","last_unstale":"2023-10-19T14:20:20.922247+0000","last_undegraded":"2023-10-19T14:20:20.922247+0000","last_fullsized":"2023-10-19T14:20:20.922247+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:23:13.051759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.3c","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.020977+0000","last_change":"2023-10-19T14:20:19.351919+0000","last_active":"2023-10-19T14:20:21.020977+0000","last_peered":"2023-10-19T14:20:21.020977+0000","last_clean":"2023-10-19T14:20:21.020977+0000","last_became_active":"2023-10-19T14:20:19.351634+0000","last_became_peered":"2023-10-19T14:20:19.351634+0000","last_unstale":"2023-10-19T14:20:21.020977+0000","last_undegraded":"2023-10-19T14:20:21.020977+0000","last_fullsized":"2023-10-19T14:20:21.020977+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:40:33.512260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.3b","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.382538+0000","last_change":"2023-10-19T14:20:19.344979+0000","last_active":"2023-10-19T14:20:21.382538+0000","last_peered":"2023-10-19T14:20:21.382538+0000","last_clean":"2023-10-19T14:20:21.382538+0000","last_became_active":"2023-10-19T14:20:19.344717+0000","last_became_peered":"2023-10-19T14:20:19.344717+0000","last_unstale":"2023-10-19T14:20:21.382538+0000","last_undegraded":"2023-10-19T14:20:21.382538+0000","last_fullsized":"2023-10-19T14:20:21.382538+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:14:31.826091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.3a","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.020630+0000","last_change":"2023-10-19T14:20:19.344720+0000","last_active":"2023-10-19T14:20:21.020630+0000","last_peered":"2023-10-19T14:20:21.020630+0000","last_clean":"2023-10-19T14:20:21.020630+0000","last_became_active":"2023-10-19T14:20:19.344415+0000","last_became_peered":"2023-10-19T14:20:19.344415+0000","last_unstale":"2023-10-19T14:20:21.020630+0000","last_undegraded":"2023-10-19T14:20:21.020630+0000","last_fullsized":"2023-10-19T14:20:21.020630+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:01:45.199351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.39","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.382402+0000","last_change":"2023-10-19T14:20:19.343249+0000","last_active":"2023-10-19T14:20:21.382402+0000","last_peered":"2023-10-19T14:20:21.382402+0000","last_clean":"2023-10-19T14:20:21.382402+0000","last_became_active":"2023-10-19T14:20:19.342989+0000","last_became_peered":"2023-10-19T14:20:19.342989+0000","last_unstale":"2023-10-19T14:20:21.382402+0000","last_undegraded":"2023-10-19T14:20:21.382402+0000","last_fullsized":"2023-10-19T14:20:21.382402+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:54:18.133180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.38","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.383386+0000","last_change":"2023-10-19T14:20:19.347387+0000","last_active":"2023-10-19T14:20:21.383386+0000","last_peered":"2023-10-19T14:20:21.383386+0000","last_clean":"2023-10-19T14:20:21.383386+0000","last_became_active":"2023-10-19T14:20:19.347104+0000","last_became_peered":"2023-10-19T14:20:19.347104+0000","last_unstale":"2023-10-19T14:20:21.383386+0000","last_undegraded":"2023-10-19T14:20:21.383386+0000","last_fullsized":"2023-10-19T14:20:21.383386+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:13:49.536943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.37","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.328659+0000","last_change":"2023-10-19T14:20:19.335860+0000","last_active":"2023-10-19T14:20:20.328659+0000","last_peered":"2023-10-19T14:20:20.328659+0000","last_clean":"2023-10-19T14:20:20.328659+0000","last_became_active":"2023-10-19T14:20:19.335522+0000","last_became_peered":"2023-10-19T14:20:19.335522+0000","last_unstale":"2023-10-19T14:20:20.328659+0000","last_undegraded":"2023-10-19T14:20:20.328659+0000","last_fullsized":"2023-10-19T14:20:20.328659+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:36:28.937721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.36","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.383938+0000","last_change":"2023-10-19T14:20:19.350276+0000","last_active":"2023-10-19T14:20:21.383938+0000","last_peered":"2023-10-19T14:20:21.383938+0000","last_clean":"2023-10-19T14:20:21.383938+0000","last_became_active":"2023-10-19T14:20:19.349877+0000","last_became_peered":"2023-10-19T14:20:19.349877+0000","last_unstale":"2023-10-19T14:20:21.383938+0000","last_undegraded":"2023-10-19T14:20:21.383938+0000","last_fullsized":"2023-10-19T14:20:21.383938+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:25:10.980817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.35","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.922282+0000","last_change":"2023-10-19T14:20:19.347686+0000","last_active":"2023-10-19T14:20:20.922282+0000","last_peered":"2023-10-19T14:20:20.922282+0000","last_clean":"2023-10-19T14:20:20.922282+0000","last_became_active":"2023-10-19T14:20:19.347432+0000","last_became_peered":"2023-10-19T14:20:19.347432+0000","last_unstale":"2023-10-19T14:20:20.922282+0000","last_undegraded":"2023-10-19T14:20:20.922282+0000","last_fullsized":"2023-10-19T14:20:20.922282+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:46:20.720183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.34","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.382533+0000","last_change":"2023-10-19T14:20:19.350634+0000","last_active":"2023-10-19T14:20:21.382533+0000","last_peered":"2023-10-19T14:20:21.382533+0000","last_clean":"2023-10-19T14:20:21.382533+0000","last_became_active":"2023-10-19T14:20:19.350362+0000","last_became_peered":"2023-10-19T14:20:19.350362+0000","last_unstale":"2023-10-19T14:20:21.382533+0000","last_undegraded":"2023-10-19T14:20:21.382533+0000","last_fullsized":"2023-10-19T14:20:21.382533+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:30:33.652808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.33","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.020809+0000","last_change":"2023-10-19T14:20:19.351639+0000","last_active":"2023-10-19T14:20:21.020809+0000","last_peered":"2023-10-19T14:20:21.020809+0000","last_clean":"2023-10-19T14:20:21.020809+0000","last_became_active":"2023-10-19T14:20:19.350821+0000","last_became_peered":"2023-10-19T14:20:19.350821+0000","last_unstale":"2023-10-19T14:20:21.020809+0000","last_undegraded":"2023-10-19T14:20:21.020809+0000","last_fullsized":"2023-10-19T14:20:21.020809+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T02:16:40.560667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.32","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.383562+0000","last_change":"2023-10-19T14:20:19.346548+0000","last_active":"2023-10-19T14:20:21.383562+0000","last_peered":"2023-10-19T14:20:21.383562+0000","last_clean":"2023-10-19T14:20:21.383562+0000","last_became_active":"2023-10-19T14:20:19.345001+0000","last_became_peered":"2023-10-19T14:20:19.345001+0000","last_unstale":"2023-10-19T14:20:21.383562+0000","last_undegraded":"2023-10-19T14:20:21.383562+0000","last_fullsized":"2023-10-19T14:20:21.383562+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T02:03:47.400085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.31","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.382911+0000","last_change":"2023-10-19T14:20:19.349749+0000","last_active":"2023-10-19T14:20:21.382911+0000","last_peered":"2023-10-19T14:20:21.382911+0000","last_clean":"2023-10-19T14:20:21.382911+0000","last_became_active":"2023-10-19T14:20:19.349481+0000","last_became_peered":"2023-10-19T14:20:19.349481+0000","last_unstale":"2023-10-19T14:20:21.382911+0000","last_undegraded":"2023-10-19T14:20:21.382911+0000","last_fullsized":"2023-10-19T14:20:21.382911+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:20:35.750886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.30","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.922028+0000","last_change":"2023-10-19T14:20:19.338108+0000","last_active":"2023-10-19T14:20:20.922028+0000","last_peered":"2023-10-19T14:20:20.922028+0000","last_clean":"2023-10-19T14:20:20.922028+0000","last_became_active":"2023-10-19T14:20:19.337834+0000","last_became_peered":"2023-10-19T14:20:19.337834+0000","last_unstale":"2023-10-19T14:20:20.922028+0000","last_undegraded":"2023-10-19T14:20:20.922028+0000","last_fullsized":"2023-10-19T14:20:20.922028+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:15:21.019725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.2f","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.328664+0000","last_change":"2023-10-19T14:20:19.344630+0000","last_active":"2023-10-19T14:20:20.328664+0000","last_peered":"2023-10-19T14:20:20.328664+0000","last_clean":"2023-10-19T14:20:20.328664+0000","last_became_active":"2023-10-19T14:20:19.344354+0000","last_became_peered":"2023-10-19T14:20:19.344354+0000","last_unstale":"2023-10-19T14:20:20.328664+0000","last_undegraded":"2023-10-19T14:20:20.328664+0000","last_fullsized":"2023-10-19T14:20:20.328664+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:10:30.171693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.2e","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.328574+0000","last_change":"2023-10-19T14:20:19.341197+0000","last_active":"2023-10-19T14:20:20.328574+0000","last_peered":"2023-10-19T14:20:20.328574+0000","last_clean":"2023-10-19T14:20:20.328574+0000","last_became_active":"2023-10-19T14:20:19.340929+0000","last_became_peered":"2023-10-19T14:20:19.340929+0000","last_unstale":"2023-10-19T14:20:20.328574+0000","last_undegraded":"2023-10-19T14:20:20.328574+0000","last_fullsized":"2023-10-19T14:20:20.328574+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:44:28.147164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.2d","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.384001+0000","last_change":"2023-10-19T14:20:19.352942+0000","last_active":"2023-10-19T14:20:21.384001+0000","last_peered":"2023-10-19T14:20:21.384001+0000","last_clean":"2023-10-19T14:20:21.384001+0000","last_became_active":"2023-10-19T14:20:19.352678+0000","last_became_peered":"2023-10-19T14:20:19.352678+0000","last_unstale":"2023-10-19T14:20:21.384001+0000","last_undegraded":"2023-10-19T14:20:21.384001+0000","last_fullsized":"2023-10-19T14:20:21.384001+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:19:02.500176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2c","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.383918+0000","last_change":"2023-10-19T14:20:19.343670+0000","last_active":"2023-10-19T14:20:21.383918+0000","last_peered":"2023-10-19T14:20:21.383918+0000","last_clean":"2023-10-19T14:20:21.383918+0000","last_became_active":"2023-10-19T14:20:19.342491+0000","last_became_peered":"2023-10-19T14:20:19.342491+0000","last_unstale":"2023-10-19T14:20:21.383918+0000","last_undegraded":"2023-10-19T14:20:21.383918+0000","last_fullsized":"2023-10-19T14:20:21.383918+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:32:39.312387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2b","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.383630+0000","last_change":"2023-10-19T14:20:19.351751+0000","last_active":"2023-10-19T14:20:21.383630+0000","last_peered":"2023-10-19T14:20:21.383630+0000","last_clean":"2023-10-19T14:20:21.383630+0000","last_became_active":"2023-10-19T14:20:19.350197+0000","last_became_peered":"2023-10-19T14:20:19.350197+0000","last_unstale":"2023-10-19T14:20:21.383630+0000","last_undegraded":"2023-10-19T14:20:21.383630+0000","last_fullsized":"2023-10-19T14:20:21.383630+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:18:10.058277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2a","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.383884+0000","last_change":"2023-10-19T14:20:19.345667+0000","last_active":"2023-10-19T14:20:21.383884+0000","last_peered":"2023-10-19T14:20:21.383884+0000","last_clean":"2023-10-19T14:20:21.383884+0000","last_became_active":"2023-10-19T14:20:19.345343+0000","last_became_peered":"2023-10-19T14:20:19.345343+0000","last_unstale":"2023-10-19T14:20:21.383884+0000","last_undegraded":"2023-10-19T14:20:21.383884+0000","last_fullsized":"2023-10-19T14:20:21.383884+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:36:14.871743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.29","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.331283+0000","last_change":"2023-10-19T14:20:19.333467+0000","last_active":"2023-10-19T14:20:20.331283+0000","last_peered":"2023-10-19T14:20:20.331283+0000","last_clean":"2023-10-19T14:20:20.331283+0000","last_became_active":"2023-10-19T14:20:19.333176+0000","last_became_peered":"2023-10-19T14:20:19.333176+0000","last_unstale":"2023-10-19T14:20:20.331283+0000","last_undegraded":"2023-10-19T14:20:20.331283+0000","last_fullsized":"2023-10-19T14:20:20.331283+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:39:10.256824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.28","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.382366+0000","last_change":"2023-10-19T14:20:19.341549+0000","last_active":"2023-10-19T14:20:21.382366+0000","last_peered":"2023-10-19T14:20:21.382366+0000","last_clean":"2023-10-19T14:20:21.382366+0000","last_became_active":"2023-10-19T14:20:19.341276+0000","last_became_peered":"2023-10-19T14:20:19.341276+0000","last_unstale":"2023-10-19T14:20:21.382366+0000","last_undegraded":"2023-10-19T14:20:21.382366+0000","last_fullsized":"2023-10-19T14:20:21.382366+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:23:15.830657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.27","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.328747+0000","last_change":"2023-10-19T14:20:19.339558+0000","last_active":"2023-10-19T14:20:20.328747+0000","last_peered":"2023-10-19T14:20:20.328747+0000","last_clean":"2023-10-19T14:20:20.328747+0000","last_became_active":"2023-10-19T14:20:19.339298+0000","last_became_peered":"2023-10-19T14:20:19.339298+0000","last_unstale":"2023-10-19T14:20:20.328747+0000","last_undegraded":"2023-10-19T14:20:20.328747+0000","last_fullsized":"2023-10-19T14:20:20.328747+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:27:05.614080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.26","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.332703+0000","last_change":"2023-10-19T14:20:19.348136+0000","last_active":"2023-10-19T14:20:20.332703+0000","last_peered":"2023-10-19T14:20:20.332703+0000","last_clean":"2023-10-19T14:20:20.332703+0000","last_became_active":"2023-10-19T14:20:19.347832+0000","last_became_peered":"2023-10-19T14:20:19.347832+0000","last_unstale":"2023-10-19T14:20:20.332703+0000","last_undegraded":"2023-10-19T14:20:20.332703+0000","last_fullsized":"2023-10-19T14:20:20.332703+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:16:46.603067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.25","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.922367+0000","last_change":"2023-10-19T14:20:19.347669+0000","last_active":"2023-10-19T14:20:20.922367+0000","last_peered":"2023-10-19T14:20:20.922367+0000","last_clean":"2023-10-19T14:20:20.922367+0000","last_became_active":"2023-10-19T14:20:19.347397+0000","last_became_peered":"2023-10-19T14:20:19.347397+0000","last_unstale":"2023-10-19T14:20:20.922367+0000","last_undegraded":"2023-10-19T14:20:20.922367+0000","last_fullsized":"2023-10-19T14:20:20.922367+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:47:46.746053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.24","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.382730+0000","last_change":"2023-10-19T14:20:19.345949+0000","last_active":"2023-10-19T14:20:21.382730+0000","last_peered":"2023-10-19T14:20:21.382730+0000","last_clean":"2023-10-19T14:20:21.382730+0000","last_became_active":"2023-10-19T14:20:19.345664+0000","last_became_peered":"2023-10-19T14:20:19.345664+0000","last_unstale":"2023-10-19T14:20:21.382730+0000","last_undegraded":"2023-10-19T14:20:21.382730+0000","last_fullsized":"2023-10-19T14:20:21.382730+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:52:23.062217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.23","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.328362+0000","last_change":"2023-10-19T14:20:19.342642+0000","last_active":"2023-10-19T14:20:20.328362+0000","last_peered":"2023-10-19T14:20:20.328362+0000","last_clean":"2023-10-19T14:20:20.328362+0000","last_became_active":"2023-10-19T14:20:19.342379+0000","last_became_peered":"2023-10-19T14:20:19.342379+0000","last_unstale":"2023-10-19T14:20:20.328362+0000","last_undegraded":"2023-10-19T14:20:20.328362+0000","last_fullsized":"2023-10-19T14:20:20.328362+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:33:34.581616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.22","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.382958+0000","last_change":"2023-10-19T14:20:19.342297+0000","last_active":"2023-10-19T14:20:21.382958+0000","last_peered":"2023-10-19T14:20:21.382958+0000","last_clean":"2023-10-19T14:20:21.382958+0000","last_became_active":"2023-10-19T14:20:19.342026+0000","last_became_peered":"2023-10-19T14:20:19.342026+0000","last_unstale":"2023-10-19T14:20:21.382958+0000","last_undegraded":"2023-10-19T14:20:21.382958+0000","last_fullsized":"2023-10-19T14:20:21.382958+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:43:57.179927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.21","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.020598+0000","last_change":"2023-10-19T14:20:19.349579+0000","last_active":"2023-10-19T14:20:21.020598+0000","last_peered":"2023-10-19T14:20:21.020598+0000","last_clean":"2023-10-19T14:20:21.020598+0000","last_became_active":"2023-10-19T14:20:19.349321+0000","last_became_peered":"2023-10-19T14:20:19.349321+0000","last_unstale":"2023-10-19T14:20:21.020598+0000","last_undegraded":"2023-10-19T14:20:21.020598+0000","last_fullsized":"2023-10-19T14:20:21.020598+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:44:03.588824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.20","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.020600+0000","last_change":"2023-10-19T14:20:19.348899+0000","last_active":"2023-10-19T14:20:21.020600+0000","last_peered":"2023-10-19T14:20:21.020600+0000","last_clean":"2023-10-19T14:20:21.020600+0000","last_became_active":"2023-10-19T14:20:19.348453+0000","last_became_peered":"2023-10-19T14:20:19.348453+0000","last_unstale":"2023-10-19T14:20:21.020600+0000","last_undegraded":"2023-10-19T14:20:21.020600+0000","last_fullsized":"2023-10-19T14:20:21.020600+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:01:53.425378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.331310+0000","last_change":"2023-10-19T14:20:19.333259+0000","last_active":"2023-10-19T14:20:20.331310+0000","last_peered":"2023-10-19T14:20:20.331310+0000","last_clean":"2023-10-19T14:20:20.331310+0000","last_became_active":"2023-10-19T14:20:19.332735+0000","last_became_peered":"2023-10-19T14:20:19.332735+0000","last_unstale":"2023-10-19T14:20:20.331310+0000","last_undegraded":"2023-10-19T14:20:20.331310+0000","last_fullsized":"2023-10-19T14:20:20.331310+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:46:26.663119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.1f","version":"222'4","reported_seq":16,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.383194+0000","last_change":"2023-10-19T14:20:18.725431+0000","last_active":"2023-10-19T14:20:21.383194+0000","last_peered":"2023-10-19T14:20:21.383194+0000","last_clean":"2023-10-19T14:20:21.383194+0000","last_became_active":"2023-10-19T14:20:18.725142+0000","last_became_peered":"2023-10-19T14:20:18.725142+0000","last_unstale":"2023-10-19T14:20:21.383194+0000","last_undegraded":"2023-10-19T14:20:21.383194+0000","last_fullsized":"2023-10-19T14:20:21.383194+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T02:03:46.398046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.332233+0000","last_change":"2023-10-19T14:20:19.343393+0000","last_active":"2023-10-19T14:20:20.332233+0000","last_peered":"2023-10-19T14:20:20.332233+0000","last_clean":"2023-10-19T14:20:20.332233+0000","last_became_active":"2023-10-19T14:20:19.343011+0000","last_became_peered":"2023-10-19T14:20:19.343011+0000","last_unstale":"2023-10-19T14:20:20.332233+0000","last_undegraded":"2023-10-19T14:20:20.332233+0000","last_fullsized":"2023-10-19T14:20:20.332233+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:22:47.864868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1e","version":"222'1","reported_seq":13,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.332751+0000","last_change":"2023-10-19T14:20:18.719165+0000","last_active":"2023-10-19T14:20:20.332751+0000","last_peered":"2023-10-19T14:20:20.332751+0000","last_clean":"2023-10-19T14:20:20.332751+0000","last_became_active":"2023-10-19T14:20:18.717146+0000","last_became_peered":"2023-10-19T14:20:18.717146+0000","last_unstale":"2023-10-19T14:20:20.332751+0000","last_undegraded":"2023-10-19T14:20:20.332751+0000","last_fullsized":"2023-10-19T14:20:20.332751+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:57:14.320702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.1a","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.383778+0000","last_change":"2023-10-19T14:20:19.346318+0000","last_active":"2023-10-19T14:20:21.383778+0000","last_peered":"2023-10-19T14:20:21.383778+0000","last_clean":"2023-10-19T14:20:21.383778+0000","last_became_active":"2023-10-19T14:20:19.346014+0000","last_became_peered":"2023-10-19T14:20:19.346014+0000","last_unstale":"2023-10-19T14:20:21.383778+0000","last_undegraded":"2023-10-19T14:20:21.383778+0000","last_fullsized":"2023-10-19T14:20:21.383778+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:20:35.750886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1d","version":"222'2","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.331596+0000","last_change":"2023-10-19T14:20:18.717342+0000","last_active":"2023-10-19T14:20:20.331596+0000","last_peered":"2023-10-19T14:20:20.331596+0000","last_clean":"2023-10-19T14:20:20.331596+0000","last_became_active":"2023-10-19T14:20:18.717047+0000","last_became_peered":"2023-10-19T14:20:18.717047+0000","last_unstale":"2023-10-19T14:20:20.331596+0000","last_undegraded":"2023-10-19T14:20:20.331596+0000","last_fullsized":"2023-10-19T14:20:20.331596+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:28:55.961320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.1b","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.382554+0000","last_change":"2023-10-19T14:20:19.348254+0000","last_active":"2023-10-19T14:20:21.382554+0000","last_peered":"2023-10-19T14:20:21.382554+0000","last_clean":"2023-10-19T14:20:21.382554+0000","last_became_active":"2023-10-19T14:20:19.347974+0000","last_became_peered":"2023-10-19T14:20:19.347974+0000","last_unstale":"2023-10-19T14:20:21.382554+0000","last_undegraded":"2023-10-19T14:20:21.382554+0000","last_fullsized":"2023-10-19T14:20:21.382554+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:32:42.913042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1c","version":"222'2","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.328509+0000","last_change":"2023-10-19T14:20:18.722673+0000","last_active":"2023-10-19T14:20:20.328509+0000","last_peered":"2023-10-19T14:20:20.328509+0000","last_clean":"2023-10-19T14:20:20.328509+0000","last_became_active":"2023-10-19T14:20:18.722418+0000","last_became_peered":"2023-10-19T14:20:18.722418+0000","last_unstale":"2023-10-19T14:20:20.328509+0000","last_undegraded":"2023-10-19T14:20:20.328509+0000","last_fullsized":"2023-10-19T14:20:20.328509+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:40:32.510221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1c","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.383205+0000","last_change":"2023-10-19T14:20:19.349815+0000","last_active":"2023-10-19T14:20:21.383205+0000","last_peered":"2023-10-19T14:20:21.383205+0000","last_clean":"2023-10-19T14:20:21.383205+0000","last_became_active":"2023-10-19T14:20:19.349511+0000","last_became_peered":"2023-10-19T14:20:19.349511+0000","last_unstale":"2023-10-19T14:20:21.383205+0000","last_undegraded":"2023-10-19T14:20:21.383205+0000","last_fullsized":"2023-10-19T14:20:21.383205+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:32:39.312387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.1b","version":"222'1","reported_seq":13,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.020920+0000","last_change":"2023-10-19T14:20:18.722476+0000","last_active":"2023-10-19T14:20:21.020920+0000","last_peered":"2023-10-19T14:20:21.020920+0000","last_clean":"2023-10-19T14:20:21.020920+0000","last_became_active":"2023-10-19T14:20:18.722195+0000","last_became_peered":"2023-10-19T14:20:18.722195+0000","last_unstale":"2023-10-19T14:20:21.020920+0000","last_undegraded":"2023-10-19T14:20:21.020920+0000","last_fullsized":"2023-10-19T14:20:21.020920+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:38:13.906613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1e","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.383161+0000","last_change":"2023-10-19T14:20:19.343379+0000","last_active":"2023-10-19T14:20:21.383161+0000","last_peered":"2023-10-19T14:20:21.383161+0000","last_clean":"2023-10-19T14:20:21.383161+0000","last_became_active":"2023-10-19T14:20:19.343117+0000","last_became_peered":"2023-10-19T14:20:19.343117+0000","last_unstale":"2023-10-19T14:20:21.383161+0000","last_undegraded":"2023-10-19T14:20:21.383161+0000","last_fullsized":"2023-10-19T14:20:21.383161+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:33:52.226441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":12,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.020524+0000","last_change":"2023-10-19T14:20:18.721875+0000","last_active":"2023-10-19T14:20:21.020524+0000","last_peered":"2023-10-19T14:20:21.020524+0000","last_clean":"2023-10-19T14:20:21.020524+0000","last_became_active":"2023-10-19T14:20:18.721611+0000","last_became_peered":"2023-10-19T14:20:18.721611+0000","last_unstale":"2023-10-19T14:20:21.020524+0000","last_undegraded":"2023-10-19T14:20:21.020524+0000","last_fullsized":"2023-10-19T14:20:21.020524+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:46:19.718145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1f","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.328790+0000","last_change":"2023-10-19T14:20:19.346582+0000","last_active":"2023-10-19T14:20:20.328790+0000","last_peered":"2023-10-19T14:20:20.328790+0000","last_clean":"2023-10-19T14:20:20.328790+0000","last_became_active":"2023-10-19T14:20:19.346246+0000","last_became_peered":"2023-10-19T14:20:19.346246+0000","last_unstale":"2023-10-19T14:20:20.328790+0000","last_undegraded":"2023-10-19T14:20:20.328790+0000","last_fullsized":"2023-10-19T14:20:20.328790+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:37:58.222092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":12,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.922070+0000","last_change":"2023-10-19T14:20:18.718913+0000","last_active":"2023-10-19T14:20:20.922070+0000","last_peered":"2023-10-19T14:20:20.922070+0000","last_clean":"2023-10-19T14:20:20.922070+0000","last_became_active":"2023-10-19T14:20:18.718641+0000","last_became_peered":"2023-10-19T14:20:18.718641+0000","last_unstale":"2023-10-19T14:20:20.922070+0000","last_undegraded":"2023-10-19T14:20:20.922070+0000","last_fullsized":"2023-10-19T14:20:20.922070+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:54:40.217789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.10","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.382991+0000","last_change":"2023-10-19T14:20:19.351668+0000","last_active":"2023-10-19T14:20:21.382991+0000","last_peered":"2023-10-19T14:20:21.382991+0000","last_clean":"2023-10-19T14:20:21.382991+0000","last_became_active":"2023-10-19T14:20:19.351298+0000","last_became_peered":"2023-10-19T14:20:19.351298+0000","last_unstale":"2023-10-19T14:20:21.382991+0000","last_undegraded":"2023-10-19T14:20:21.382991+0000","last_fullsized":"2023-10-19T14:20:21.382991+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:36:14.871743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":12,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.382838+0000","last_change":"2023-10-19T14:20:18.722070+0000","last_active":"2023-10-19T14:20:21.382838+0000","last_peered":"2023-10-19T14:20:21.382838+0000","last_clean":"2023-10-19T14:20:21.382838+0000","last_became_active":"2023-10-19T14:20:18.721791+0000","last_became_peered":"2023-10-19T14:20:18.721791+0000","last_unstale":"2023-10-19T14:20:21.382838+0000","last_undegraded":"2023-10-19T14:20:21.382838+0000","last_fullsized":"2023-10-19T14:20:21.382838+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:17:55.048333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.922093+0000","last_change":"2023-10-19T14:20:19.339048+0000","last_active":"2023-10-19T14:20:20.922093+0000","last_peered":"2023-10-19T14:20:20.922093+0000","last_clean":"2023-10-19T14:20:20.922093+0000","last_became_active":"2023-10-19T14:20:19.338786+0000","last_became_peered":"2023-10-19T14:20:19.338786+0000","last_unstale":"2023-10-19T14:20:20.922093+0000","last_undegraded":"2023-10-19T14:20:20.922093+0000","last_fullsized":"2023-10-19T14:20:20.922093+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:16:46.603067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":12,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.021070+0000","last_change":"2023-10-19T14:20:18.721992+0000","last_active":"2023-10-19T14:20:21.021070+0000","last_peered":"2023-10-19T14:20:21.021070+0000","last_clean":"2023-10-19T14:20:21.021070+0000","last_became_active":"2023-10-19T14:20:18.721742+0000","last_became_peered":"2023-10-19T14:20:18.721742+0000","last_unstale":"2023-10-19T14:20:21.021070+0000","last_undegraded":"2023-10-19T14:20:21.021070+0000","last_fullsized":"2023-10-19T14:20:21.021070+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:16:37.567429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.328317+0000","last_change":"2023-10-19T14:20:19.334253+0000","last_active":"2023-10-19T14:20:20.328317+0000","last_peered":"2023-10-19T14:20:20.328317+0000","last_clean":"2023-10-19T14:20:20.328317+0000","last_became_active":"2023-10-19T14:20:19.333981+0000","last_became_peered":"2023-10-19T14:20:19.333981+0000","last_unstale":"2023-10-19T14:20:20.328317+0000","last_undegraded":"2023-10-19T14:20:20.328317+0000","last_fullsized":"2023-10-19T14:20:20.328317+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:48:00.239750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.15","version":"222'3","reported_seq":15,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.328592+0000","last_change":"2023-10-19T14:20:18.723409+0000","last_active":"2023-10-19T14:20:20.328592+0000","last_peered":"2023-10-19T14:20:20.328592+0000","last_clean":"2023-10-19T14:20:20.328592+0000","last_became_active":"2023-10-19T14:20:18.723148+0000","last_became_peered":"2023-10-19T14:20:18.723148+0000","last_unstale":"2023-10-19T14:20:20.328592+0000","last_undegraded":"2023-10-19T14:20:20.328592+0000","last_fullsized":"2023-10-19T14:20:20.328592+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:13:53.110830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.13","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.382663+0000","last_change":"2023-10-19T14:20:19.350427+0000","last_active":"2023-10-19T14:20:21.382663+0000","last_peered":"2023-10-19T14:20:21.382663+0000","last_clean":"2023-10-19T14:20:21.382663+0000","last_became_active":"2023-10-19T14:20:19.350112+0000","last_became_peered":"2023-10-19T14:20:19.350112+0000","last_unstale":"2023-10-19T14:20:21.382663+0000","last_undegraded":"2023-10-19T14:20:21.382663+0000","last_fullsized":"2023-10-19T14:20:21.382663+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:01:38.993506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.14","version":"222'6","reported_seq":18,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.020941+0000","last_change":"2023-10-19T14:20:18.727225+0000","last_active":"2023-10-19T14:20:21.020941+0000","last_peered":"2023-10-19T14:20:21.020941+0000","last_clean":"2023-10-19T14:20:21.020941+0000","last_became_active":"2023-10-19T14:20:18.726839+0000","last_became_peered":"2023-10-19T14:20:18.726839+0000","last_unstale":"2023-10-19T14:20:21.020941+0000","last_undegraded":"2023-10-19T14:20:21.020941+0000","last_fullsized":"2023-10-19T14:20:21.020941+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:10:54.369994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.14","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.331654+0000","last_change":"2023-10-19T14:20:19.340471+0000","last_active":"2023-10-19T14:20:20.331654+0000","last_peered":"2023-10-19T14:20:20.331654+0000","last_clean":"2023-10-19T14:20:20.331654+0000","last_became_active":"2023-10-19T14:20:19.340187+0000","last_became_peered":"2023-10-19T14:20:19.340187+0000","last_unstale":"2023-10-19T14:20:20.331654+0000","last_undegraded":"2023-10-19T14:20:20.331654+0000","last_fullsized":"2023-10-19T14:20:20.331654+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:53:57.462566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.13","version":"222'2","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.331524+0000","last_change":"2023-10-19T14:20:18.713594+0000","last_active":"2023-10-19T14:20:20.331524+0000","last_peered":"2023-10-19T14:20:20.331524+0000","last_clean":"2023-10-19T14:20:20.331524+0000","last_became_active":"2023-10-19T14:20:18.713263+0000","last_became_peered":"2023-10-19T14:20:18.713263+0000","last_unstale":"2023-10-19T14:20:20.331524+0000","last_undegraded":"2023-10-19T14:20:20.331524+0000","last_fullsized":"2023-10-19T14:20:20.331524+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:32:02.753539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.15","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.922354+0000","last_change":"2023-10-19T14:20:19.349089+0000","last_active":"2023-10-19T14:20:20.922354+0000","last_peered":"2023-10-19T14:20:20.922354+0000","last_clean":"2023-10-19T14:20:20.922354+0000","last_became_active":"2023-10-19T14:20:19.348826+0000","last_became_peered":"2023-10-19T14:20:19.348826+0000","last_unstale":"2023-10-19T14:20:20.922354+0000","last_undegraded":"2023-10-19T14:20:20.922354+0000","last_fullsized":"2023-10-19T14:20:20.922354+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:10:55.372033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":12,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.922121+0000","last_change":"2023-10-19T14:20:18.709786+0000","last_active":"2023-10-19T14:20:20.922121+0000","last_peered":"2023-10-19T14:20:20.922121+0000","last_clean":"2023-10-19T14:20:20.922121+0000","last_became_active":"2023-10-19T14:20:18.709496+0000","last_became_peered":"2023-10-19T14:20:18.709496+0000","last_unstale":"2023-10-19T14:20:20.922121+0000","last_undegraded":"2023-10-19T14:20:20.922121+0000","last_fullsized":"2023-10-19T14:20:20.922121+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:46:59.026062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.328599+0000","last_change":"2023-10-19T14:20:19.343343+0000","last_active":"2023-10-19T14:20:20.328599+0000","last_peered":"2023-10-19T14:20:20.328599+0000","last_clean":"2023-10-19T14:20:20.328599+0000","last_became_active":"2023-10-19T14:20:19.343059+0000","last_became_peered":"2023-10-19T14:20:19.343059+0000","last_unstale":"2023-10-19T14:20:20.328599+0000","last_undegraded":"2023-10-19T14:20:20.328599+0000","last_fullsized":"2023-10-19T14:20:20.328599+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:31:59.664720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":12,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.328232+0000","last_change":"2023-10-19T14:20:18.717768+0000","last_active":"2023-10-19T14:20:20.328232+0000","last_peered":"2023-10-19T14:20:20.328232+0000","last_clean":"2023-10-19T14:20:20.328232+0000","last_became_active":"2023-10-19T14:20:18.717375+0000","last_became_peered":"2023-10-19T14:20:18.717375+0000","last_unstale":"2023-10-19T14:20:20.328232+0000","last_undegraded":"2023-10-19T14:20:20.328232+0000","last_fullsized":"2023-10-19T14:20:20.328232+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:44:02.586786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.020980+0000","last_change":"2023-10-19T14:20:19.353248+0000","last_active":"2023-10-19T14:20:21.020980+0000","last_peered":"2023-10-19T14:20:21.020980+0000","last_clean":"2023-10-19T14:20:21.020980+0000","last_became_active":"2023-10-19T14:20:19.352973+0000","last_became_peered":"2023-10-19T14:20:19.352973+0000","last_unstale":"2023-10-19T14:20:21.020980+0000","last_undegraded":"2023-10-19T14:20:21.020980+0000","last_fullsized":"2023-10-19T14:20:21.020980+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:07:18.725863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.10","version":"222'8","reported_seq":20,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.382313+0000","last_change":"2023-10-19T14:20:18.722891+0000","last_active":"2023-10-19T14:20:21.382313+0000","last_peered":"2023-10-19T14:20:21.382313+0000","last_clean":"2023-10-19T14:20:21.382313+0000","last_became_active":"2023-10-19T14:20:18.722633+0000","last_became_peered":"2023-10-19T14:20:18.722633+0000","last_unstale":"2023-10-19T14:20:21.382313+0000","last_undegraded":"2023-10-19T14:20:21.382313+0000","last_fullsized":"2023-10-19T14:20:21.382313+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:32:21.255137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.383389+0000","last_change":"2023-10-19T14:20:19.349188+0000","last_active":"2023-10-19T14:20:21.383389+0000","last_peered":"2023-10-19T14:20:21.383389+0000","last_clean":"2023-10-19T14:20:21.383389+0000","last_became_active":"2023-10-19T14:20:19.348905+0000","last_became_peered":"2023-10-19T14:20:19.348905+0000","last_unstale":"2023-10-19T14:20:21.383389+0000","last_undegraded":"2023-10-19T14:20:21.383389+0000","last_fullsized":"2023-10-19T14:20:21.383389+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:36:36.169729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.f","version":"222'1","reported_seq":13,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.332614+0000","last_change":"2023-10-19T14:20:18.719346+0000","last_active":"2023-10-19T14:20:20.332614+0000","last_peered":"2023-10-19T14:20:20.332614+0000","last_clean":"2023-10-19T14:20:20.332614+0000","last_became_active":"2023-10-19T14:20:18.719101+0000","last_became_peered":"2023-10-19T14:20:18.719101+0000","last_unstale":"2023-10-19T14:20:20.332614+0000","last_undegraded":"2023-10-19T14:20:20.332614+0000","last_fullsized":"2023-10-19T14:20:20.332614+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:01:11.976346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.332287+0000","last_change":"2023-10-19T14:20:19.348569+0000","last_active":"2023-10-19T14:20:20.332287+0000","last_peered":"2023-10-19T14:20:20.332287+0000","last_clean":"2023-10-19T14:20:20.332287+0000","last_became_active":"2023-10-19T14:20:19.348268+0000","last_became_peered":"2023-10-19T14:20:19.348268+0000","last_unstale":"2023-10-19T14:20:20.332287+0000","last_undegraded":"2023-10-19T14:20:20.332287+0000","last_fullsized":"2023-10-19T14:20:20.332287+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:54:41.219828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.e","version":"222'2","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.332760+0000","last_change":"2023-10-19T14:20:18.721114+0000","last_active":"2023-10-19T14:20:20.332760+0000","last_peered":"2023-10-19T14:20:20.332760+0000","last_clean":"2023-10-19T14:20:20.332760+0000","last_became_active":"2023-10-19T14:20:18.720840+0000","last_became_peered":"2023-10-19T14:20:18.720840+0000","last_unstale":"2023-10-19T14:20:20.332760+0000","last_undegraded":"2023-10-19T14:20:20.332760+0000","last_fullsized":"2023-10-19T14:20:20.332760+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:44:50.967568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.332574+0000","last_change":"2023-10-19T14:20:19.332710+0000","last_active":"2023-10-19T14:20:20.332574+0000","last_peered":"2023-10-19T14:20:20.332574+0000","last_clean":"2023-10-19T14:20:20.332574+0000","last_became_active":"2023-10-19T14:20:19.332132+0000","last_became_peered":"2023-10-19T14:20:19.332132+0000","last_unstale":"2023-10-19T14:20:20.332574+0000","last_undegraded":"2023-10-19T14:20:20.332574+0000","last_fullsized":"2023-10-19T14:20:20.332574+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:36:15.681007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.d","version":"222'1","reported_seq":13,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.331623+0000","last_change":"2023-10-19T14:20:18.719857+0000","last_active":"2023-10-19T14:20:20.331623+0000","last_peered":"2023-10-19T14:20:20.331623+0000","last_clean":"2023-10-19T14:20:20.331623+0000","last_became_active":"2023-10-19T14:20:18.719586+0000","last_became_peered":"2023-10-19T14:20:18.719586+0000","last_unstale":"2023-10-19T14:20:20.331623+0000","last_undegraded":"2023-10-19T14:20:20.331623+0000","last_fullsized":"2023-10-19T14:20:20.331623+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:12:37.668857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.b","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.383814+0000","last_change":"2023-10-19T14:20:19.352403+0000","last_active":"2023-10-19T14:20:21.383814+0000","last_peered":"2023-10-19T14:20:21.383814+0000","last_clean":"2023-10-19T14:20:21.383814+0000","last_became_active":"2023-10-19T14:20:19.351892+0000","last_became_peered":"2023-10-19T14:20:19.351892+0000","last_unstale":"2023-10-19T14:20:21.383814+0000","last_undegraded":"2023-10-19T14:20:21.383814+0000","last_fullsized":"2023-10-19T14:20:21.383814+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:33:52.226441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":12,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.382772+0000","last_change":"2023-10-19T14:20:18.717037+0000","last_active":"2023-10-19T14:20:21.382772+0000","last_peered":"2023-10-19T14:20:21.382772+0000","last_clean":"2023-10-19T14:20:21.382772+0000","last_became_active":"2023-10-19T14:20:18.716731+0000","last_became_peered":"2023-10-19T14:20:18.716731+0000","last_unstale":"2023-10-19T14:20:21.382772+0000","last_undegraded":"2023-10-19T14:20:21.382772+0000","last_fullsized":"2023-10-19T14:20:21.382772+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:50:59.017027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.331668+0000","last_change":"2023-10-19T14:20:19.346026+0000","last_active":"2023-10-19T14:20:20.331668+0000","last_peered":"2023-10-19T14:20:20.331668+0000","last_clean":"2023-10-19T14:20:20.331668+0000","last_became_active":"2023-10-19T14:20:19.345762+0000","last_became_peered":"2023-10-19T14:20:19.345762+0000","last_unstale":"2023-10-19T14:20:20.331668+0000","last_undegraded":"2023-10-19T14:20:20.331668+0000","last_fullsized":"2023-10-19T14:20:20.331668+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:28:21.209211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.b","version":"222'2","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.382612+0000","last_change":"2023-10-19T14:20:18.719825+0000","last_active":"2023-10-19T14:20:21.382612+0000","last_peered":"2023-10-19T14:20:21.382612+0000","last_clean":"2023-10-19T14:20:21.382612+0000","last_became_active":"2023-10-19T14:20:18.719549+0000","last_became_peered":"2023-10-19T14:20:18.719549+0000","last_unstale":"2023-10-19T14:20:21.382612+0000","last_undegraded":"2023-10-19T14:20:21.382612+0000","last_fullsized":"2023-10-19T14:20:21.382612+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T02:18:13.024084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.d","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.020866+0000","last_change":"2023-10-19T14:20:19.350548+0000","last_active":"2023-10-19T14:20:21.020866+0000","last_peered":"2023-10-19T14:20:21.020866+0000","last_clean":"2023-10-19T14:20:21.020866+0000","last_became_active":"2023-10-19T14:20:19.349339+0000","last_became_peered":"2023-10-19T14:20:19.349339+0000","last_unstale":"2023-10-19T14:20:21.020866+0000","last_undegraded":"2023-10-19T14:20:21.020866+0000","last_fullsized":"2023-10-19T14:20:21.020866+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:13:54.112869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":12,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.331433+0000","last_change":"2023-10-19T14:20:18.711157+0000","last_active":"2023-10-19T14:20:20.331433+0000","last_peered":"2023-10-19T14:20:20.331433+0000","last_clean":"2023-10-19T14:20:20.331433+0000","last_became_active":"2023-10-19T14:20:18.710747+0000","last_became_peered":"2023-10-19T14:20:18.710747+0000","last_unstale":"2023-10-19T14:20:20.331433+0000","last_undegraded":"2023-10-19T14:20:20.331433+0000","last_fullsized":"2023-10-19T14:20:20.331433+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:46:05.844692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.e","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.328663+0000","last_change":"2023-10-19T14:20:19.345329+0000","last_active":"2023-10-19T14:20:20.328663+0000","last_peered":"2023-10-19T14:20:20.328663+0000","last_clean":"2023-10-19T14:20:20.328663+0000","last_became_active":"2023-10-19T14:20:19.344983+0000","last_became_peered":"2023-10-19T14:20:19.344983+0000","last_unstale":"2023-10-19T14:20:20.328663+0000","last_undegraded":"2023-10-19T14:20:20.328663+0000","last_fullsized":"2023-10-19T14:20:20.328663+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:17:53.794653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":12,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.382937+0000","last_change":"2023-10-19T14:20:18.727394+0000","last_active":"2023-10-19T14:20:21.382937+0000","last_peered":"2023-10-19T14:20:21.382937+0000","last_clean":"2023-10-19T14:20:21.382937+0000","last_became_active":"2023-10-19T14:20:18.727050+0000","last_became_peered":"2023-10-19T14:20:18.727050+0000","last_unstale":"2023-10-19T14:20:21.382937+0000","last_undegraded":"2023-10-19T14:20:21.382937+0000","last_fullsized":"2023-10-19T14:20:21.382937+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:13:48.534905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.f","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.384115+0000","last_change":"2023-10-19T14:20:19.346087+0000","last_active":"2023-10-19T14:20:21.384115+0000","last_peered":"2023-10-19T14:20:21.384115+0000","last_clean":"2023-10-19T14:20:21.384115+0000","last_became_active":"2023-10-19T14:20:19.345704+0000","last_became_peered":"2023-10-19T14:20:19.345704+0000","last_unstale":"2023-10-19T14:20:21.384115+0000","last_undegraded":"2023-10-19T14:20:21.384115+0000","last_fullsized":"2023-10-19T14:20:21.384115+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:53:30.184157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":12,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.332172+0000","last_change":"2023-10-19T14:20:18.715329+0000","last_active":"2023-10-19T14:20:20.332172+0000","last_peered":"2023-10-19T14:20:20.332172+0000","last_clean":"2023-10-19T14:20:20.332172+0000","last_became_active":"2023-10-19T14:20:18.715068+0000","last_became_peered":"2023-10-19T14:20:18.715068+0000","last_unstale":"2023-10-19T14:20:20.332172+0000","last_undegraded":"2023-10-19T14:20:20.332172+0000","last_fullsized":"2023-10-19T14:20:20.332172+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:35:10.527276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.383603+0000","last_change":"2023-10-19T14:20:19.349129+0000","last_active":"2023-10-19T14:20:21.383603+0000","last_peered":"2023-10-19T14:20:21.383603+0000","last_clean":"2023-10-19T14:20:21.383603+0000","last_became_active":"2023-10-19T14:20:19.348846+0000","last_became_peered":"2023-10-19T14:20:19.348846+0000","last_unstale":"2023-10-19T14:20:21.383603+0000","last_undegraded":"2023-10-19T14:20:21.383603+0000","last_fullsized":"2023-10-19T14:20:21.383603+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:04:34.009285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.7","version":"222'2","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.328793+0000","last_change":"2023-10-19T14:20:18.724203+0000","last_active":"2023-10-19T14:20:20.328793+0000","last_peered":"2023-10-19T14:20:20.328793+0000","last_clean":"2023-10-19T14:20:20.328793+0000","last_became_active":"2023-10-19T14:20:18.723883+0000","last_became_peered":"2023-10-19T14:20:18.723883+0000","last_unstale":"2023-10-19T14:20:20.328793+0000","last_undegraded":"2023-10-19T14:20:20.328793+0000","last_fullsized":"2023-10-19T14:20:20.328793+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:38:31.255506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.922094+0000","last_change":"2023-10-19T14:20:19.344287+0000","last_active":"2023-10-19T14:20:20.922094+0000","last_peered":"2023-10-19T14:20:20.922094+0000","last_clean":"2023-10-19T14:20:20.922094+0000","last_became_active":"2023-10-19T14:20:19.344030+0000","last_became_peered":"2023-10-19T14:20:19.344030+0000","last_unstale":"2023-10-19T14:20:20.922094+0000","last_undegraded":"2023-10-19T14:20:20.922094+0000","last_fullsized":"2023-10-19T14:20:20.922094+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:35:58.143784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.6","version":"0'0","reported_seq":12,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.383985+0000","last_change":"2023-10-19T14:20:18.722733+0000","last_active":"2023-10-19T14:20:21.383985+0000","last_peered":"2023-10-19T14:20:21.383985+0000","last_clean":"2023-10-19T14:20:21.383985+0000","last_became_active":"2023-10-19T14:20:18.722447+0000","last_became_peered":"2023-10-19T14:20:18.722447+0000","last_unstale":"2023-10-19T14:20:21.383985+0000","last_undegraded":"2023-10-19T14:20:21.383985+0000","last_fullsized":"2023-10-19T14:20:21.383985+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:28:23.768143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.328321+0000","last_change":"2023-10-19T14:20:19.340725+0000","last_active":"2023-10-19T14:20:20.328321+0000","last_peered":"2023-10-19T14:20:20.328321+0000","last_clean":"2023-10-19T14:20:20.328321+0000","last_became_active":"2023-10-19T14:20:19.340284+0000","last_became_peered":"2023-10-19T14:20:19.340284+0000","last_unstale":"2023-10-19T14:20:20.328321+0000","last_undegraded":"2023-10-19T14:20:20.328321+0000","last_fullsized":"2023-10-19T14:20:20.328321+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:16:20.216590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":12,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.383019+0000","last_change":"2023-10-19T14:20:18.724127+0000","last_active":"2023-10-19T14:20:21.383019+0000","last_peered":"2023-10-19T14:20:21.383019+0000","last_clean":"2023-10-19T14:20:21.383019+0000","last_became_active":"2023-10-19T14:20:18.723847+0000","last_became_peered":"2023-10-19T14:20:18.723847+0000","last_unstale":"2023-10-19T14:20:21.383019+0000","last_undegraded":"2023-10-19T14:20:21.383019+0000","last_fullsized":"2023-10-19T14:20:21.383019+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:36:35.167691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.332503+0000","last_change":"2023-10-19T14:20:19.352214+0000","last_active":"2023-10-19T14:20:20.332503+0000","last_peered":"2023-10-19T14:20:20.332503+0000","last_clean":"2023-10-19T14:20:20.332503+0000","last_became_active":"2023-10-19T14:20:19.351789+0000","last_became_peered":"2023-10-19T14:20:19.351789+0000","last_unstale":"2023-10-19T14:20:20.332503+0000","last_undegraded":"2023-10-19T14:20:20.332503+0000","last_fullsized":"2023-10-19T14:20:20.332503+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:05:58.978059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":12,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.332482+0000","last_change":"2023-10-19T14:20:18.718760+0000","last_active":"2023-10-19T14:20:20.332482+0000","last_peered":"2023-10-19T14:20:20.332482+0000","last_clean":"2023-10-19T14:20:20.332482+0000","last_became_active":"2023-10-19T14:20:18.718459+0000","last_became_peered":"2023-10-19T14:20:18.718459+0000","last_unstale":"2023-10-19T14:20:20.332482+0000","last_undegraded":"2023-10-19T14:20:20.332482+0000","last_fullsized":"2023-10-19T14:20:20.332482+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:32:18.381107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.383815+0000","last_change":"2023-10-19T14:20:19.345692+0000","last_active":"2023-10-19T14:20:21.383815+0000","last_peered":"2023-10-19T14:20:21.383815+0000","last_clean":"2023-10-19T14:20:21.383815+0000","last_became_active":"2023-10-19T14:20:19.344861+0000","last_became_peered":"2023-10-19T14:20:19.344861+0000","last_unstale":"2023-10-19T14:20:21.383815+0000","last_undegraded":"2023-10-19T14:20:21.383815+0000","last_fullsized":"2023-10-19T14:20:21.383815+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:43:57.179927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":12,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.383799+0000","last_change":"2023-10-19T14:20:18.723898+0000","last_active":"2023-10-19T14:20:21.383799+0000","last_peered":"2023-10-19T14:20:21.383799+0000","last_clean":"2023-10-19T14:20:21.383799+0000","last_became_active":"2023-10-19T14:20:18.722703+0000","last_became_peered":"2023-10-19T14:20:18.722703+0000","last_unstale":"2023-10-19T14:20:21.383799+0000","last_undegraded":"2023-10-19T14:20:21.383799+0000","last_fullsized":"2023-10-19T14:20:21.383799+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:25:51.334964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.021015+0000","last_change":"2023-10-19T14:20:19.344708+0000","last_active":"2023-10-19T14:20:21.021015+0000","last_peered":"2023-10-19T14:20:21.021015+0000","last_clean":"2023-10-19T14:20:21.021015+0000","last_became_active":"2023-10-19T14:20:19.344380+0000","last_became_peered":"2023-10-19T14:20:19.344380+0000","last_unstale":"2023-10-19T14:20:21.021015+0000","last_undegraded":"2023-10-19T14:20:21.021015+0000","last_fullsized":"2023-10-19T14:20:21.021015+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:37:55.455379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":12,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.328440+0000","last_change":"2023-10-19T14:20:18.722453+0000","last_active":"2023-10-19T14:20:20.328440+0000","last_peered":"2023-10-19T14:20:20.328440+0000","last_clean":"2023-10-19T14:20:20.328440+0000","last_became_active":"2023-10-19T14:20:18.722162+0000","last_became_peered":"2023-10-19T14:20:18.722162+0000","last_unstale":"2023-10-19T14:20:20.328440+0000","last_undegraded":"2023-10-19T14:20:20.328440+0000","last_fullsized":"2023-10-19T14:20:20.328440+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:49:03.077978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.021092+0000","last_change":"2023-10-19T14:20:19.345421+0000","last_active":"2023-10-19T14:20:21.021092+0000","last_peered":"2023-10-19T14:20:21.021092+0000","last_clean":"2023-10-19T14:20:21.021092+0000","last_became_active":"2023-10-19T14:20:19.345091+0000","last_became_peered":"2023-10-19T14:20:19.345091+0000","last_unstale":"2023-10-19T14:20:21.021092+0000","last_undegraded":"2023-10-19T14:20:21.021092+0000","last_fullsized":"2023-10-19T14:20:21.021092+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:49:04.080016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":12,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.332384+0000","last_change":"2023-10-19T14:20:18.723906+0000","last_active":"2023-10-19T14:20:20.332384+0000","last_peered":"2023-10-19T14:20:20.332384+0000","last_clean":"2023-10-19T14:20:20.332384+0000","last_became_active":"2023-10-19T14:20:18.723585+0000","last_became_peered":"2023-10-19T14:20:18.723585+0000","last_unstale":"2023-10-19T14:20:20.332384+0000","last_undegraded":"2023-10-19T14:20:20.332384+0000","last_fullsized":"2023-10-19T14:20:20.332384+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:28:48.261352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.7","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.922350+0000","last_change":"2023-10-19T14:20:19.347562+0000","last_active":"2023-10-19T14:20:20.922350+0000","last_peered":"2023-10-19T14:20:20.922350+0000","last_clean":"2023-10-19T14:20:20.922350+0000","last_became_active":"2023-10-19T14:20:19.347286+0000","last_became_peered":"2023-10-19T14:20:19.347286+0000","last_unstale":"2023-10-19T14:20:20.922350+0000","last_undegraded":"2023-10-19T14:20:20.922350+0000","last_fullsized":"2023-10-19T14:20:20.922350+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:06:12.825567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":12,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.382416+0000","last_change":"2023-10-19T14:20:18.722143+0000","last_active":"2023-10-19T14:20:21.382416+0000","last_peered":"2023-10-19T14:20:21.382416+0000","last_clean":"2023-10-19T14:20:21.382416+0000","last_became_active":"2023-10-19T14:20:18.721853+0000","last_became_peered":"2023-10-19T14:20:18.721853+0000","last_unstale":"2023-10-19T14:20:21.382416+0000","last_undegraded":"2023-10-19T14:20:21.382416+0000","last_fullsized":"2023-10-19T14:20:21.382416+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:01:31.494525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.1d","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.332568+0000","last_change":"2023-10-19T14:20:19.350783+0000","last_active":"2023-10-19T14:20:20.332568+0000","last_peered":"2023-10-19T14:20:20.332568+0000","last_clean":"2023-10-19T14:20:20.332568+0000","last_became_active":"2023-10-19T14:20:19.350441+0000","last_became_peered":"2023-10-19T14:20:19.350441+0000","last_unstale":"2023-10-19T14:20:20.332568+0000","last_undegraded":"2023-10-19T14:20:20.332568+0000","last_fullsized":"2023-10-19T14:20:20.332568+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:47:00.028101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1a","version":"222'1","reported_seq":13,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.328426+0000","last_change":"2023-10-19T14:20:18.723791+0000","last_active":"2023-10-19T14:20:20.328426+0000","last_peered":"2023-10-19T14:20:20.328426+0000","last_clean":"2023-10-19T14:20:20.328426+0000","last_became_active":"2023-10-19T14:20:18.723476+0000","last_became_peered":"2023-10-19T14:20:18.723476+0000","last_unstale":"2023-10-19T14:20:20.328426+0000","last_undegraded":"2023-10-19T14:20:20.328426+0000","last_fullsized":"2023-10-19T14:20:20.328426+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:07:17.723824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"211'3186","reported_seq":4083,"reported_epoch":222,"state":"active+clean","last_fresh":"2023-10-19T14:20:19.342159+0000","last_change":"2023-10-19T12:44:08.636613+0000","last_active":"2023-10-19T14:20:19.342159+0000","last_peered":"2023-10-19T14:20:19.342159+0000","last_clean":"2023-10-19T14:20:19.342159+0000","last_became_active":"2023-10-19T12:44:08.635697+0000","last_became_peered":"2023-10-19T12:44:08.635697+0000","last_unstale":"2023-10-19T14:20:19.342159+0000","last_undegraded":"2023-10-19T14:20:19.342159+0000","last_fullsized":"2023-10-19T14:20:19.342159+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_clean_scrub_stamp":"2023-10-19T12:44:07.399738+0000","objects_scrubbed":0,"log_size":3186,"log_dups_size":0,"ondisk_log_size":3186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:15:23.685485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2210,"num_read_kb":4624,"num_write":589,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":28,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":27,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2210,"num_read_kb":4624,"num_write":589,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3186,"ondisk_log_size":3186,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576002,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48400,"kb_used_data":3332,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323440,"statfs":{"total":96636764160,"available":96587202560,"internally_reserved":0,"allocated":3411968,"data_stored":2026754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":46135755},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:20:09 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.56999999999999995,"15min":0.58499999999999996},"min":{"1min":0.35799999999999998,"5min":0.34100000000000003,"15min":0.32200000000000001},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.86199999999999999},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.60199999999999998,"15min":0.60999999999999999},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.312},"max":{"1min":0.77600000000000002,"5min":0.81899999999999995,"15min":0.86599999999999999},"last":0.54800000000000004}]},{"osd":1,"last update":"Thu Oct 19 14:20:09 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.59799999999999998,"15min":0.59999999999999998},"min":{"1min":0.46500000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.76800000000000002,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.59499999999999997,"15min":0.59799999999999998},"min":{"1min":0.373,"5min":0.373,"15min":0.33500000000000002},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.78500000000000003}]},{"osd":2,"last update":"Thu Oct 19 14:20:09 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64900000000000002,"15min":0.64800000000000002},"min":{"1min":0.50900000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.85499999999999998},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.66600000000000004,"15min":0.66200000000000003},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.40100000000000002},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.92100000000000004}]},{"osd":3,"last update":"Thu Oct 19 14:20:09 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.63300000000000001,"15min":0.63700000000000001},"min":{"1min":0.51100000000000001,"5min":0.42099999999999999,"15min":0.36399999999999999},"max":{"1min":0.82699999999999996,"5min":0.84799999999999998,"15min":0.873},"last":0.95599999999999996},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.68799999999999994,"15min":0.68700000000000006},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":0.873,"5min":0.876,"15min":0.89600000000000002},"last":0.83299999999999996}]},{"osd":4,"last update":"Thu Oct 19 14:20:09 2023","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.50600000000000001,"15min":0.49299999999999999},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.62,"5min":0.78100000000000003,"15min":0.83799999999999997},"last":0.98399999999999999},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.51800000000000002,"15min":0.50600000000000001},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.71899999999999997,"5min":0.80200000000000005,"15min":0.81699999999999995},"last":1.0329999999999999}]},{"osd":5,"last update":"Thu Oct 19 14:20:09 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51400000000000001,"15min":0.50800000000000001},"min":{"1min":0.36799999999999999,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.65600000000000003,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.89100000000000001},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.55600000000000005,"15min":0.54900000000000004},"min":{"1min":0.377,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.70999999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":1.054}]},{"osd":6,"last update":"Thu Oct 19 14:20:09 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57699999999999996,"15min":0.57399999999999995},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.35999999999999999},"max":{"1min":0.73199999999999998,"5min":0.82999999999999996,"15min":0.85099999999999998},"last":1.0129999999999999},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.58099999999999996,"15min":0.56799999999999995},"min":{"1min":0.38800000000000001,"5min":0.35599999999999998,"15min":0.33500000000000002},"max":{"1min":0.78200000000000003,"5min":0.85199999999999998,"15min":0.85999999999999999},"last":1.1180000000000001}]}]},{"osd":6,"up_from":13,"seq":55834576002,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38092,"kb_used_data":2752,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333748,"statfs":{"total":96636764160,"available":96597757952,"internally_reserved":0,"allocated":2818048,"data_stored":1436898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:20:00 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.60799999999999998,"15min":0.61499999999999999},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.28799999999999998},"max":{"1min":0.81200000000000006,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.59399999999999997,"15min":0.61499999999999999},"min":{"1min":0.40699999999999997,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.42899999999999999}]},{"osd":1,"last update":"Thu Oct 19 14:20:00 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.621,"15min":0.621},"min":{"1min":0.52300000000000002,"5min":0.38200000000000001,"15min":0.35799999999999998},"max":{"1min":0.79500000000000004,"5min":0.80200000000000005,"15min":1.165},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.624,"15min":0.63500000000000001},"min":{"1min":0.503,"5min":0.439,"15min":0.36899999999999999},"max":{"1min":0.81599999999999995,"5min":0.83699999999999997,"15min":0.91300000000000003},"last":0.66600000000000004}]},{"osd":2,"last update":"Thu Oct 19 14:20:00 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.60099999999999998,"15min":0.60899999999999999},"min":{"1min":0.442,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.873},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.626,"15min":0.629},"min":{"1min":0.45000000000000001,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.93100000000000005},"last":0.55700000000000005}]},{"osd":3,"last update":"Thu Oct 19 14:20:00 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.65500000000000003,"15min":0.65800000000000003},"min":{"1min":0.53000000000000003,"5min":0.372,"15min":0.34300000000000003},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.65600000000000003,"15min":0.66100000000000003},"min":{"1min":0.48299999999999998,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.84399999999999997,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.67900000000000005}]},{"osd":4,"last update":"Thu Oct 19 14:20:00 2023","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.47999999999999998,"15min":0.48199999999999998},"min":{"1min":0.34499999999999997,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.79600000000000004},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.502,"15min":0.50800000000000001},"min":{"1min":0.35399999999999998,"5min":0.34300000000000003,"15min":0.30399999999999999},"max":{"1min":0.74199999999999999,"5min":0.82199999999999995,"15min":0.85699999999999998},"last":0.53500000000000003}]},{"osd":5,"last update":"Thu Oct 19 14:20:00 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.5,"15min":0.5},"min":{"1min":0.35599999999999998,"5min":0.32700000000000001,"15min":0.32500000000000001},"max":{"1min":0.747,"5min":1.115,"15min":1.115},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.55100000000000005,"15min":0.54300000000000004},"min":{"1min":0.40000000000000002,"5min":0.35799999999999998,"15min":0.32400000000000001},"max":{"1min":0.78700000000000003,"5min":0.86499999999999999,"15min":0.878},"last":0.65000000000000002}]},{"osd":7,"last update":"Thu Oct 19 14:20:00 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.55000000000000004,"15min":0.54600000000000004},"min":{"1min":0.39900000000000002,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.77600000000000002,"5min":0.86399999999999999,"15min":0.90100000000000002},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.55700000000000005,"15min":0.55200000000000005},"min":{"1min":0.39400000000000002,"5min":0.38200000000000001,"15min":0.36199999999999999},"max":{"1min":0.85099999999999998,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.621}]}]},{"osd":5,"up_from":13,"seq":55834576002,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47820,"kb_used_data":2744,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324020,"statfs":{"total":96636764160,"available":96587796480,"internally_reserved":0,"allocated":2809856,"data_stored":1434760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:19:46 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.59899999999999998,"15min":0.621},"min":{"1min":0.44900000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.79700000000000004,"5min":0.82599999999999996,"15min":0.86399999999999999},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.63200000000000001,"15min":0.64500000000000002},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.82499999999999996,"5min":0.85899999999999999,"15min":0.85999999999999999},"last":0.68000000000000005}]},{"osd":1,"last update":"Thu Oct 19 14:19:46 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.61799999999999999,"15min":0.63500000000000001},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.78800000000000003,"5min":0.83699999999999997,"15min":0.877},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.64100000000000001,"15min":0.64500000000000002},"min":{"1min":0.47999999999999998,"5min":0.44700000000000001,"15min":0.41199999999999998},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.88500000000000001},"last":0.81599999999999995}]},{"osd":2,"last update":"Thu Oct 19 14:19:46 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.61499999999999999,"15min":0.627},"min":{"1min":0.44800000000000001,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.81299999999999994,"5min":0.83799999999999997,"15min":0.85099999999999998},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.65200000000000002,"15min":0.66100000000000003},"min":{"1min":0.52000000000000002,"5min":0.49399999999999999,"15min":0.47099999999999997},"max":{"1min":0.79500000000000004,"5min":0.83299999999999996,"15min":0.90800000000000003},"last":0.63800000000000001}]},{"osd":3,"last update":"Thu Oct 19 14:19:46 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.63900000000000001,"15min":0.64700000000000002},"min":{"1min":0.51100000000000001,"5min":0.443,"15min":0.443},"max":{"1min":0.84399999999999997,"5min":0.86099999999999999,"15min":0.90100000000000002},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.66500000000000004,"15min":0.67400000000000004},"min":{"1min":0.53300000000000003,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.84899999999999998,"5min":0.876,"15min":0.90900000000000003},"last":0.69999999999999996}]},{"osd":4,"last update":"Thu Oct 19 14:19:46 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.497,"15min":0.48699999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.32500000000000001},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50900000000000001,"15min":0.50600000000000001},"min":{"1min":0.35499999999999998,"5min":0.318,"15min":0.28599999999999998},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.82899999999999996},"last":0.51500000000000001}]},{"osd":6,"last update":"Thu Oct 19 14:19:46 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51200000000000001,"15min":0.497},"min":{"1min":0.37,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.88800000000000001},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.54100000000000004,"15min":0.53900000000000003},"min":{"1min":0.39100000000000001,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.77900000000000003,"5min":0.78000000000000003,"15min":0.88100000000000001},"last":0.59599999999999997}]},{"osd":7,"last update":"Thu Oct 19 14:19:46 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.51500000000000001,"15min":0.52000000000000002},"min":{"1min":0.41799999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.749,"5min":0.749,"15min":0.82899999999999996},"last":0.625},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.58999999999999997,"15min":0.57499999999999996},"min":{"1min":0.47099999999999997,"5min":0.38400000000000001,"15min":0.34300000000000003},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.66100000000000003}]}]},{"osd":4,"up_from":13,"seq":55834576002,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47796,"kb_used_data":2728,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324044,"statfs":{"total":96636764160,"available":96587821056,"internally_reserved":0,"allocated":2793472,"data_stored":1428638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.55500000000000005,"15min":0.57599999999999996},"min":{"1min":0.35499999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.80800000000000005,"5min":0.81799999999999995,"15min":0.879},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.58499999999999996,"15min":0.59799999999999998},"min":{"1min":0.499,"5min":0.31,"15min":0.31},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.873},"last":0.379}]},{"osd":1,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.61499999999999999,"15min":0.64100000000000001},"min":{"1min":0.504,"5min":0.372,"15min":0.32600000000000001},"max":{"1min":0.80400000000000005,"5min":0.80700000000000005,"15min":0.92700000000000005},"last":0.441},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.623,"15min":0.64400000000000002},"min":{"1min":0.443,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":1.0940000000000001},"last":0.50700000000000001}]},{"osd":2,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.63400000000000001,"15min":0.65000000000000002},"min":{"1min":0.46400000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.64800000000000002,"15min":0.66900000000000004},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":1.008},"last":0.628}]},{"osd":3,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.61299999999999999,"15min":0.63400000000000001},"min":{"1min":0.47399999999999998,"5min":0.37,"15min":0.37},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.91900000000000004},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.66600000000000004,"15min":0.66700000000000004},"min":{"1min":0.54000000000000004,"5min":0.45400000000000001,"15min":0.437},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.60799999999999998}]},{"osd":5,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.48299999999999998,"15min":0.47099999999999997},"min":{"1min":0.34000000000000002,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.504,"15min":0.49199999999999999},"min":{"1min":0.34300000000000003,"5min":0.33700000000000002,"15min":0.29199999999999998},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.57699999999999996}]},{"osd":6,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.496,"15min":0.48599999999999999},"min":{"1min":0.34799999999999998,"5min":0.33900000000000002,"15min":0.28199999999999997},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.53700000000000003},"min":{"1min":0.39100000000000001,"5min":0.375,"15min":0.33300000000000002},"max":{"1min":0.81299999999999994,"5min":0.84799999999999998,"15min":1.2969999999999999},"last":0.67900000000000005}]},{"osd":7,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.55100000000000005,"15min":0.54100000000000004},"min":{"1min":0.36499999999999999,"5min":0.35699999999999998,"15min":0.32600000000000001},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60499999999999998,"15min":0.59199999999999997},"min":{"1min":0.47099999999999997,"5min":0.32100000000000001,"15min":0.318},"max":{"1min":0.80400000000000005,"5min":0.88600000000000001,"15min":1.032},"last":0.66700000000000004}]}]},{"osd":3,"up_from":13,"seq":55834576001,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56148,"kb_used_data":2748,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315692,"statfs":{"total":96636764160,"available":96579268608,"internally_reserved":0,"allocated":2813952,"data_stored":1434830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54655432},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:19:48 2023","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42299999999999999},"min":{"1min":0.316,"5min":0.253,"15min":0.253},"max":{"1min":0.60599999999999998,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.42599999999999999,"15min":0.432},"min":{"1min":0.29399999999999998,"5min":0.219,"15min":0.219},"max":{"1min":0.57599999999999996,"5min":0.752,"15min":0.752},"last":0.317}]},{"osd":1,"last update":"Thu Oct 19 14:19:48 2023","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.439,"15min":0.433},"min":{"1min":0.29599999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.56599999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.30099999999999999,"5min":0.29399999999999998,"15min":0.26400000000000001},"max":{"1min":0.56599999999999995,"5min":0.73099999999999998,"15min":0.81799999999999995},"last":0.51700000000000002}]},{"osd":2,"last update":"Thu Oct 19 14:19:48 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.47699999999999998,"15min":0.48599999999999999},"min":{"1min":0.39600000000000002,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.64300000000000002,"5min":0.64600000000000002,"15min":0.73099999999999998},"last":0.441},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.499,"15min":0.50800000000000001},"min":{"1min":0.33500000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.66800000000000004,"5min":0.67000000000000004,"15min":0.83699999999999997},"last":0.46700000000000003}]},{"osd":4,"last update":"Thu Oct 19 14:19:48 2023","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.71099999999999997,"15min":0.70899999999999996},"min":{"1min":0.45000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.96099999999999997,"5min":1.018,"15min":1.157},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.57999999999999996,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.93100000000000005,"5min":0.996,"15min":1.173},"last":0.64200000000000002}]},{"osd":5,"last update":"Thu Oct 19 14:19:48 2023","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.69499999999999995,"15min":0.69199999999999995},"min":{"1min":0.55000000000000004,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.93100000000000005,"5min":0.997,"15min":1.036},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.69799999999999995,"15min":0.69299999999999995},"min":{"1min":0.47399999999999998,"5min":0.42999999999999999,"15min":0.42599999999999999},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":1.105},"last":0.54200000000000004}]},{"osd":6,"last update":"Thu Oct 19 14:19:48 2023","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.68500000000000005,"15min":0.69499999999999995},"min":{"1min":0.51900000000000002,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.99099999999999999,"5min":1.018,"15min":1.0669999999999999},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.55800000000000005,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.88800000000000001,"5min":0.91500000000000004,"15min":1.079},"last":0.61099999999999999}]},{"osd":7,"last update":"Thu Oct 19 14:19:48 2023","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.70799999999999996,"15min":0.70499999999999996},"min":{"1min":0.51600000000000001,"5min":0.434,"15min":0.434},"max":{"1min":0.94899999999999995,"5min":0.96899999999999997,"15min":1.085},"last":0.627},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.73199999999999998,"15min":0.73599999999999999},"min":{"1min":0.57099999999999995,"5min":0.51900000000000002,"15min":0.51300000000000001},"max":{"1min":0.96499999999999997,"5min":0.97199999999999998,"15min":1.1739999999999999},"last":0.71499999999999997}]}]},{"osd":2,"up_from":13,"seq":55834576003,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64448,"kb_used_data":2740,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94307392,"statfs":{"total":96636764160,"available":96570769408,"internally_reserved":0,"allocated":2805760,"data_stored":1434794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":63175110},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.39900000000000002,"15min":0.40200000000000002},"min":{"1min":0.28199999999999997,"5min":0.218,"15min":0.218},"max":{"1min":0.623,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.44500000000000001,"15min":0.441},"min":{"1min":0.34100000000000003,"5min":0.249,"15min":0.223},"max":{"1min":0.628,"5min":0.71399999999999997,"15min":0.73599999999999999},"last":0.39600000000000002}]},{"osd":1,"last update":"Thu Oct 19 14:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.40500000000000003,"15min":0.41599999999999998},"min":{"1min":0.28999999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.48999999999999999,"5min":0.61699999999999999,"15min":0.65400000000000003},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.47999999999999998,"15min":0.47099999999999997},"min":{"1min":0.33000000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":1.742,"5min":1.742,"15min":1.742},"last":0.38100000000000001}]},{"osd":3,"last update":"Thu Oct 19 14:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.48199999999999998,"15min":0.48399999999999999},"min":{"1min":0.313,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.56999999999999995,"5min":0.69299999999999995,"15min":0.74399999999999999},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.499,"15min":0.51100000000000001},"min":{"1min":0.38200000000000001,"5min":0.32000000000000001,"15min":0.315},"max":{"1min":0.60599999999999998,"5min":0.67600000000000005,"15min":0.83599999999999997},"last":0.36399999999999999}]},{"osd":4,"last update":"Thu Oct 19 14:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.66900000000000004,"15min":0.67000000000000004},"min":{"1min":0.44900000000000001,"5min":0.39600000000000002,"15min":0.39500000000000002},"max":{"1min":0.88600000000000001,"5min":0.93899999999999995,"15min":0.998},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.67300000000000004,"15min":0.68200000000000005},"min":{"1min":0.51700000000000002,"5min":0.38900000000000001,"15min":0.35199999999999998},"max":{"1min":0.79500000000000004,"5min":0.97099999999999997,"15min":1.0289999999999999},"last":0.47299999999999998}]},{"osd":5,"last update":"Thu Oct 19 14:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.70499999999999996,"15min":0.70199999999999996},"min":{"1min":0.53300000000000003,"5min":0.45100000000000001,"15min":0.40600000000000003},"max":{"1min":1.4330000000000001,"5min":1.4330000000000001,"15min":1.4330000000000001},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.68400000000000005,"15min":0.68899999999999995},"min":{"1min":0.48199999999999998,"5min":0.45100000000000001,"15min":0.39300000000000002},"max":{"1min":1.595,"5min":1.595,"15min":1.595},"last":0.62}]},{"osd":6,"last update":"Thu Oct 19 14:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.70299999999999996,"15min":0.71199999999999997},"min":{"1min":0.46500000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.85999999999999999,"5min":0.92800000000000005,"15min":1.0189999999999999},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.72899999999999998,"15min":0.72599999999999998},"min":{"1min":0.47899999999999998,"5min":0.433,"15min":0.40600000000000003},"max":{"1min":1.4930000000000001,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":0.48899999999999999}]},{"osd":7,"last update":"Thu Oct 19 14:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.71999999999999997,"15min":0.72899999999999998},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.41799999999999998},"max":{"1min":0.97999999999999998,"5min":1.0129999999999999,"15min":1.0289999999999999},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.73899999999999999,"15min":0.73499999999999999},"min":{"1min":0.503,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.53,"5min":1.53,"15min":1.53},"last":0.57999999999999996}]}]},{"osd":1,"up_from":13,"seq":55834576002,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47804,"kb_used_data":2736,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324036,"statfs":{"total":96636764160,"available":96587812864,"internally_reserved":0,"allocated":2801664,"data_stored":1434704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:19:35 2023","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.40300000000000002,"15min":0.40699999999999997},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.23100000000000001},"max":{"1min":0.48799999999999999,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.37},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.45400000000000001,"15min":0.45700000000000002},"min":{"1min":0.28299999999999997,"5min":0.27500000000000002,"15min":0.251},"max":{"1min":0.61399999999999999,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.46899999999999997}]},{"osd":2,"last update":"Thu Oct 19 14:19:35 2023","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.44500000000000001,"15min":0.44600000000000001},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.58599999999999997,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.495,"15min":0.49199999999999999},"min":{"1min":0.34399999999999997,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.64400000000000002,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.42399999999999999}]},{"osd":3,"last update":"Thu Oct 19 14:19:35 2023","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.51600000000000001,"15min":0.50800000000000001},"min":{"1min":0.38400000000000001,"5min":0.36299999999999999,"15min":0.29499999999999998},"max":{"1min":0.64400000000000002,"5min":0.76000000000000001,"15min":0.78900000000000003},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.44600000000000001,"15min":0.44900000000000001},"min":{"1min":0.32000000000000001,"5min":0.315,"15min":0.26100000000000001},"max":{"1min":0.65200000000000002,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.38400000000000001}]},{"osd":4,"last update":"Thu Oct 19 14:19:35 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.67000000000000004,"15min":0.67100000000000004},"min":{"1min":0.39900000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.84399999999999997,"5min":1.042,"15min":1.042},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"min":{"1min":0.45100000000000001,"5min":0.41299999999999998,"15min":0.32300000000000001},"max":{"1min":0.88800000000000001,"5min":0.93500000000000005,"15min":1.1659999999999999},"last":0.52100000000000002}]},{"osd":5,"last update":"Thu Oct 19 14:19:35 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.69199999999999995,"15min":0.68700000000000006},"min":{"1min":0.45900000000000002,"5min":0.42799999999999999,"15min":0.42499999999999999},"max":{"1min":0.83099999999999996,"5min":1.0589999999999999,"15min":1.3500000000000001},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.71199999999999997,"15min":0.70799999999999996},"min":{"1min":0.44,"5min":0.44,"15min":0.40500000000000003},"max":{"1min":0.78600000000000003,"5min":1.0149999999999999,"15min":1.121},"last":0.58999999999999997}]},{"osd":6,"last update":"Thu Oct 19 14:19:35 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.67300000000000004,"15min":0.68300000000000005},"min":{"1min":0.42499999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.77200000000000002,"5min":1.0129999999999999,"15min":1.2310000000000001},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.71999999999999997,"15min":0.71499999999999997},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.86399999999999999,"5min":1.107,"15min":1.274},"last":0.60299999999999998}]},{"osd":7,"last update":"Thu Oct 19 14:19:35 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.72799999999999998,"15min":0.72399999999999998},"min":{"1min":0.46999999999999997,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.78600000000000003,"5min":1.075,"15min":1.075},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.47199999999999998,"5min":0.46100000000000002,"15min":0.44400000000000001},"max":{"1min":0.79600000000000004,"5min":1.0169999999999999,"15min":1.117},"last":0.53100000000000003}]}]},{"osd":0,"up_from":13,"seq":55834576003,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60872,"kb_used_data":3316,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310968,"statfs":{"total":96636764160,"available":96574431232,"internally_reserved":0,"allocated":3395584,"data_stored":2024543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.40699999999999997,"15min":0.41299999999999998},"min":{"1min":0.29599999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.56299999999999994,"5min":0.61499999999999999,"15min":0.624},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.441,"15min":0.46100000000000002},"min":{"1min":0.26800000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.56799999999999995,"5min":0.70899999999999996,"15min":0.81100000000000005},"last":0.42399999999999999}]},{"osd":2,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.496,"15min":0.48399999999999999},"min":{"1min":0.39300000000000002,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.71499999999999997,"5min":0.754,"15min":0.754},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.44800000000000001},"min":{"1min":0.32700000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.60199999999999998,"5min":0.63100000000000001,"15min":0.69099999999999995},"last":0.45100000000000001}]},{"osd":3,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.432,"15min":0.44},"min":{"1min":0.26900000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.58799999999999997,"5min":0.61099999999999999,"15min":0.66800000000000004},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.501,"5min":0.5,"15min":0.51200000000000001},"min":{"1min":0.35199999999999998,"5min":0.254,"15min":0.254},"max":{"1min":0.78100000000000003,"5min":0.79800000000000004,"15min":0.84599999999999997},"last":0.54100000000000004}]},{"osd":4,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.64800000000000002,"15min":0.67700000000000005},"min":{"1min":0.48799999999999999,"5min":0.44600000000000001,"15min":0.44},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":1.1890000000000001},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.65800000000000003,"15min":0.68200000000000005},"min":{"1min":0.47499999999999998,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.751,"5min":0.95399999999999996,"15min":1.2150000000000001},"last":0.56399999999999995}]},{"osd":5,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.68999999999999995,"15min":0.72199999999999998},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":1.1579999999999999},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.69499999999999995,"15min":0.71999999999999997},"min":{"1min":0.57799999999999996,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.85399999999999998,"5min":0.98299999999999998,"15min":1.2310000000000001},"last":0.64600000000000002}]},{"osd":6,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.68700000000000006,"15min":0.70599999999999996},"min":{"1min":0.47799999999999998,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":1.1739999999999999},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.70399999999999996},"min":{"1min":0.54100000000000004,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.88900000000000001,"5min":0.97799999999999998,"15min":1.0189999999999999},"last":0.60499999999999998}]},{"osd":7,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.68600000000000005,"15min":0.71599999999999997},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":1.1180000000000001},"last":0.628},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.68200000000000005,"15min":0.70399999999999996},"min":{"1min":0.53600000000000003,"5min":0.46899999999999997,"15min":0.45100000000000001},"max":{"1min":0.90600000000000003,"5min":0.93799999999999994,"15min":1.0349999999999999},"last":0.71399999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-19T14:20:22.727 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-19T14:20:22.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.866+0000 7f6e00f9e640 1 -- 172.21.15.57:0/535137865 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6dfc0ab500 msgr2=0x7f6dfc0ab990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:22.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.866+0000 7f6e00f9e640 1 --2- 172.21.15.57:0/535137865 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6dfc0ab500 0x7f6dfc0ab990 secure :-1 s=READY pgs=2362 cs=0 l=1 rev1=1 crypto rx=0x7f6dfc0ca260 tx=0x7f6dec02c260 comp rx=0 tx=0).stop 2023-10-19T14:20:22.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.866+0000 7f6e00f9e640 1 -- 172.21.15.57:0/535137865 shutdown_connections 2023-10-19T14:20:22.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.866+0000 7f6e00f9e640 1 --2- 172.21.15.57:0/535137865 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6dfc0ab500 0x7f6dfc0ab990 unknown :-1 s=CLOSED pgs=2362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:22.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.866+0000 7f6e00f9e640 1 --2- 172.21.15.57:0/535137865 >> v2:172.21.15.179:3300/0 conn(0x7f6dfc0c9dc0 0x7f6dfc06b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:22.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.866+0000 7f6e00f9e640 1 -- 172.21.15.57:0/535137865 >> 172.21.15.57:0/535137865 conn(0x7f6dfc069930 msgr2=0x7f6dfc069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:22.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.866+0000 7f6e00f9e640 1 -- 172.21.15.57:0/535137865 shutdown_connections 2023-10-19T14:20:22.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.866+0000 7f6e00f9e640 1 -- 172.21.15.57:0/535137865 wait complete. 2023-10-19T14:20:22.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.868+0000 7f6e00f9e640 1 Processor -- start 2023-10-19T14:20:22.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.868+0000 7f6e00f9e640 1 -- start start 2023-10-19T14:20:22.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.868+0000 7f6e00f9e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6dfc06af00 0x7f6dfc070990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:22.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.868+0000 7f6e00f9e640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6dfc0ab500 0x7f6dfc1f1820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:22.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.868+0000 7f6e00f9e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6dfc0c9dc0 0x7f6dfc06f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:22.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.868+0000 7f6e00f9e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dfc06f120 con 0x7f6dfc06af00 2023-10-19T14:20:22.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.868+0000 7f6e00f9e640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dfc1f20c0 con 0x7f6dfc0c9dc0 2023-10-19T14:20:22.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.868+0000 7f6e00f9e640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6dfc1f23a0 con 0x7f6dfc0ab500 2023-10-19T14:20:22.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.869+0000 7f6dfa59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6dfc06af00 0x7f6dfc070990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:22.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.869+0000 7f6dfa59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6dfc06af00 0x7f6dfc070990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42114/0 (socket says 172.21.15.57:42114) 2023-10-19T14:20:22.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.869+0000 7f6dfa59c640 1 -- 172.21.15.57:0/2317603023 learned_addr learned my addr 172.21.15.57:0/2317603023 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:22.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.869+0000 7f6df9d9b640 1 --2- 172.21.15.57:0/2317603023 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6dfc0ab500 0x7f6dfc1f1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:22.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.869+0000 7f6dfad9d640 1 --2- 172.21.15.57:0/2317603023 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6dfc0c9dc0 0x7f6dfc06f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:22.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.869+0000 7f6dfa59c640 1 -- 172.21.15.57:0/2317603023 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6dfc0ab500 msgr2=0x7f6dfc1f1820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:22.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.869+0000 7f6dfa59c640 1 --2- 172.21.15.57:0/2317603023 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6dfc0ab500 0x7f6dfc1f1820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:22.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.869+0000 7f6dfa59c640 1 -- 172.21.15.57:0/2317603023 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6dfc0c9dc0 msgr2=0x7f6dfc06f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:22.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.869+0000 7f6dfa59c640 1 --2- 172.21.15.57:0/2317603023 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6dfc0c9dc0 0x7f6dfc06f8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:22.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.869+0000 7f6dfa59c640 1 -- 172.21.15.57:0/2317603023 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dec009900 con 0x7f6dfc06af00 2023-10-19T14:20:22.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.869+0000 7f6df9d9b640 1 --2- 172.21.15.57:0/2317603023 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6dfc0ab500 0x7f6dfc1f1820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:20:22.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.869+0000 7f6dfad9d640 1 --2- 172.21.15.57:0/2317603023 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6dfc0c9dc0 0x7f6dfc06f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:20:22.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.869+0000 7f6dfa59c640 1 --2- 172.21.15.57:0/2317603023 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6dfc06af00 0x7f6dfc070990 secure :-1 s=READY pgs=2363 cs=0 l=1 rev1=1 crypto rx=0x7f6df400c430 tx=0x7f6df400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:22.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.869+0000 7f6df2ffd640 1 -- 172.21.15.57:0/2317603023 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6df4011790 con 0x7f6dfc06af00 2023-10-19T14:20:22.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.869+0000 7f6df2ffd640 1 -- 172.21.15.57:0/2317603023 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f6df40119e0 con 0x7f6dfc06af00 2023-10-19T14:20:22.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.869+0000 7f6e00f9e640 1 -- 172.21.15.57:0/2317603023 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6dfc06f390 con 0x7f6dfc06af00 2023-10-19T14:20:22.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.869+0000 7f6df2ffd640 1 -- 172.21.15.57:0/2317603023 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6df4011e30 con 0x7f6dfc06af00 2023-10-19T14:20:22.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.869+0000 7f6e00f9e640 1 -- 172.21.15.57:0/2317603023 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dfc1f1d60 con 0x7f6dfc06af00 2023-10-19T14:20:22.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.870+0000 7f6e00f9e640 1 -- 172.21.15.57:0/2317603023 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6dfc06c7e0 con 0x7f6dfc06af00 2023-10-19T14:20:22.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.871+0000 7f6df2ffd640 1 -- 172.21.15.57:0/2317603023 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6df401c460 con 0x7f6dfc06af00 2023-10-19T14:20:22.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.871+0000 7f6df2ffd640 1 --2- 172.21.15.57:0/2317603023 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f6ddc06af60 0x7f6ddc06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:22.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.871+0000 7f6df2ffd640 1 -- 172.21.15.57:0/2317603023 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f6df4089bc0 con 0x7f6dfc06af00 2023-10-19T14:20:22.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.871+0000 7f6df9d9b640 1 --2- 172.21.15.57:0/2317603023 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f6ddc06af60 0x7f6ddc06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:22.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.872+0000 7f6df9d9b640 1 --2- 172.21.15.57:0/2317603023 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f6ddc06af60 0x7f6ddc06d420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6de4005fd0 tx=0x7f6de4005ea0 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:22.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:22.876+0000 7f6df2ffd640 1 -- 172.21.15.57:0/2317603023 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6df4054980 con 0x7f6dfc06af00 2023-10-19T14:20:23.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.049+0000 7f6e00f9e640 1 -- 172.21.15.57:0/2317603023 --> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6dfc0abed0 con 0x7f6ddc06af60 2023-10-19T14:20:23.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.057+0000 7f6df2ffd640 1 -- 172.21.15.57:0/2317603023 <== mgr.9107 v2:172.21.15.179:6832/3232482678 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+290075 (secure 0 0 0) 0x7f6de400a310 con 0x7f6ddc06af60 2023-10-19T14:20:23.063 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:20:23.064 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2023-10-19T14:20:23.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.060+0000 7f6e00f9e640 1 -- 172.21.15.57:0/2317603023 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f6ddc06af60 msgr2=0x7f6ddc06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:23.067 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.060+0000 7f6e00f9e640 1 --2- 172.21.15.57:0/2317603023 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f6ddc06af60 0x7f6ddc06d420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6de4005fd0 tx=0x7f6de4005ea0 comp rx=0 tx=0).stop 2023-10-19T14:20:23.067 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.060+0000 7f6e00f9e640 1 -- 172.21.15.57:0/2317603023 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6dfc06af00 msgr2=0x7f6dfc070990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:23.067 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.060+0000 7f6e00f9e640 1 --2- 172.21.15.57:0/2317603023 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6dfc06af00 0x7f6dfc070990 secure :-1 s=READY pgs=2363 cs=0 l=1 rev1=1 crypto rx=0x7f6df400c430 tx=0x7f6df400c8f0 comp rx=0 tx=0).stop 2023-10-19T14:20:23.067 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.060+0000 7f6e00f9e640 1 -- 172.21.15.57:0/2317603023 shutdown_connections 2023-10-19T14:20:23.067 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.060+0000 7f6e00f9e640 1 --2- 172.21.15.57:0/2317603023 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f6ddc06af60 0x7f6ddc06d420 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:23.067 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.060+0000 7f6e00f9e640 1 --2- 172.21.15.57:0/2317603023 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6dfc0c9dc0 0x7f6dfc06f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:23.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.060+0000 7f6e00f9e640 1 --2- 172.21.15.57:0/2317603023 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6dfc0ab500 0x7f6dfc1f1820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:23.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.060+0000 7f6e00f9e640 1 --2- 172.21.15.57:0/2317603023 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6dfc06af00 0x7f6dfc070990 unknown :-1 s=CLOSED pgs=2363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:23.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.060+0000 7f6e00f9e640 1 -- 172.21.15.57:0/2317603023 >> 172.21.15.57:0/2317603023 conn(0x7f6dfc069930 msgr2=0x7f6dfc0ac550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:23.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.060+0000 7f6e00f9e640 1 -- 172.21.15.57:0/2317603023 shutdown_connections 2023-10-19T14:20:23.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.060+0000 7f6e00f9e640 1 -- 172.21.15.57:0/2317603023 wait complete. 2023-10-19T14:20:23.073 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2023-10-19T14:20:21.988889+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2210,"num_read_kb":4624,"num_write":633,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3224,"ondisk_log_size":3224,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":156,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":411380,"kb_used_data":23096,"kb_used_omap":12,"kb_used_meta":388147,"kb_avail":754563340,"statfs":{"total":773094113280,"available":772672860160,"internally_reserved":0,"allocated":23650304,"data_stored":12655921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":397463097},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.291526"},"pg_stats":[{"pgid":"28.3f","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.332591+0000","last_change":"2023-10-19T14:20:19.348564+0000","last_active":"2023-10-19T14:20:20.332591+0000","last_peered":"2023-10-19T14:20:20.332591+0000","last_clean":"2023-10-19T14:20:20.332591+0000","last_became_active":"2023-10-19T14:20:19.348281+0000","last_became_peered":"2023-10-19T14:20:19.348281+0000","last_unstale":"2023-10-19T14:20:20.332591+0000","last_undegraded":"2023-10-19T14:20:20.332591+0000","last_fullsized":"2023-10-19T14:20:20.332591+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:15:21.019725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3e","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.331656+0000","last_change":"2023-10-19T14:20:19.346014+0000","last_active":"2023-10-19T14:20:20.331656+0000","last_peered":"2023-10-19T14:20:20.331656+0000","last_clean":"2023-10-19T14:20:20.331656+0000","last_became_active":"2023-10-19T14:20:19.345686+0000","last_became_peered":"2023-10-19T14:20:19.345686+0000","last_unstale":"2023-10-19T14:20:20.331656+0000","last_undegraded":"2023-10-19T14:20:20.331656+0000","last_fullsized":"2023-10-19T14:20:20.331656+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:29:26.735942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.3d","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.922247+0000","last_change":"2023-10-19T14:20:19.349659+0000","last_active":"2023-10-19T14:20:20.922247+0000","last_peered":"2023-10-19T14:20:20.922247+0000","last_clean":"2023-10-19T14:20:20.922247+0000","last_became_active":"2023-10-19T14:20:19.349362+0000","last_became_peered":"2023-10-19T14:20:19.349362+0000","last_unstale":"2023-10-19T14:20:20.922247+0000","last_undegraded":"2023-10-19T14:20:20.922247+0000","last_fullsized":"2023-10-19T14:20:20.922247+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:23:13.051759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.3c","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.020977+0000","last_change":"2023-10-19T14:20:19.351919+0000","last_active":"2023-10-19T14:20:21.020977+0000","last_peered":"2023-10-19T14:20:21.020977+0000","last_clean":"2023-10-19T14:20:21.020977+0000","last_became_active":"2023-10-19T14:20:19.351634+0000","last_became_peered":"2023-10-19T14:20:19.351634+0000","last_unstale":"2023-10-19T14:20:21.020977+0000","last_undegraded":"2023-10-19T14:20:21.020977+0000","last_fullsized":"2023-10-19T14:20:21.020977+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:40:33.512260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.3b","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.382538+0000","last_change":"2023-10-19T14:20:19.344979+0000","last_active":"2023-10-19T14:20:21.382538+0000","last_peered":"2023-10-19T14:20:21.382538+0000","last_clean":"2023-10-19T14:20:21.382538+0000","last_became_active":"2023-10-19T14:20:19.344717+0000","last_became_peered":"2023-10-19T14:20:19.344717+0000","last_unstale":"2023-10-19T14:20:21.382538+0000","last_undegraded":"2023-10-19T14:20:21.382538+0000","last_fullsized":"2023-10-19T14:20:21.382538+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:14:31.826091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.3a","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.020630+0000","last_change":"2023-10-19T14:20:19.344720+0000","last_active":"2023-10-19T14:20:21.020630+0000","last_peered":"2023-10-19T14:20:21.020630+0000","last_clean":"2023-10-19T14:20:21.020630+0000","last_became_active":"2023-10-19T14:20:19.344415+0000","last_became_peered":"2023-10-19T14:20:19.344415+0000","last_unstale":"2023-10-19T14:20:21.020630+0000","last_undegraded":"2023-10-19T14:20:21.020630+0000","last_fullsized":"2023-10-19T14:20:21.020630+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:01:45.199351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.39","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.382402+0000","last_change":"2023-10-19T14:20:19.343249+0000","last_active":"2023-10-19T14:20:21.382402+0000","last_peered":"2023-10-19T14:20:21.382402+0000","last_clean":"2023-10-19T14:20:21.382402+0000","last_became_active":"2023-10-19T14:20:19.342989+0000","last_became_peered":"2023-10-19T14:20:19.342989+0000","last_unstale":"2023-10-19T14:20:21.382402+0000","last_undegraded":"2023-10-19T14:20:21.382402+0000","last_fullsized":"2023-10-19T14:20:21.382402+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:54:18.133180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.38","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.383386+0000","last_change":"2023-10-19T14:20:19.347387+0000","last_active":"2023-10-19T14:20:21.383386+0000","last_peered":"2023-10-19T14:20:21.383386+0000","last_clean":"2023-10-19T14:20:21.383386+0000","last_became_active":"2023-10-19T14:20:19.347104+0000","last_became_peered":"2023-10-19T14:20:19.347104+0000","last_unstale":"2023-10-19T14:20:21.383386+0000","last_undegraded":"2023-10-19T14:20:21.383386+0000","last_fullsized":"2023-10-19T14:20:21.383386+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:13:49.536943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.37","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.328659+0000","last_change":"2023-10-19T14:20:19.335860+0000","last_active":"2023-10-19T14:20:20.328659+0000","last_peered":"2023-10-19T14:20:20.328659+0000","last_clean":"2023-10-19T14:20:20.328659+0000","last_became_active":"2023-10-19T14:20:19.335522+0000","last_became_peered":"2023-10-19T14:20:19.335522+0000","last_unstale":"2023-10-19T14:20:20.328659+0000","last_undegraded":"2023-10-19T14:20:20.328659+0000","last_fullsized":"2023-10-19T14:20:20.328659+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:36:28.937721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.36","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.383938+0000","last_change":"2023-10-19T14:20:19.350276+0000","last_active":"2023-10-19T14:20:21.383938+0000","last_peered":"2023-10-19T14:20:21.383938+0000","last_clean":"2023-10-19T14:20:21.383938+0000","last_became_active":"2023-10-19T14:20:19.349877+0000","last_became_peered":"2023-10-19T14:20:19.349877+0000","last_unstale":"2023-10-19T14:20:21.383938+0000","last_undegraded":"2023-10-19T14:20:21.383938+0000","last_fullsized":"2023-10-19T14:20:21.383938+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:25:10.980817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.35","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.922282+0000","last_change":"2023-10-19T14:20:19.347686+0000","last_active":"2023-10-19T14:20:20.922282+0000","last_peered":"2023-10-19T14:20:20.922282+0000","last_clean":"2023-10-19T14:20:20.922282+0000","last_became_active":"2023-10-19T14:20:19.347432+0000","last_became_peered":"2023-10-19T14:20:19.347432+0000","last_unstale":"2023-10-19T14:20:20.922282+0000","last_undegraded":"2023-10-19T14:20:20.922282+0000","last_fullsized":"2023-10-19T14:20:20.922282+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:46:20.720183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.34","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.382533+0000","last_change":"2023-10-19T14:20:19.350634+0000","last_active":"2023-10-19T14:20:21.382533+0000","last_peered":"2023-10-19T14:20:21.382533+0000","last_clean":"2023-10-19T14:20:21.382533+0000","last_became_active":"2023-10-19T14:20:19.350362+0000","last_became_peered":"2023-10-19T14:20:19.350362+0000","last_unstale":"2023-10-19T14:20:21.382533+0000","last_undegraded":"2023-10-19T14:20:21.382533+0000","last_fullsized":"2023-10-19T14:20:21.382533+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:30:33.652808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.33","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.020809+0000","last_change":"2023-10-19T14:20:19.351639+0000","last_active":"2023-10-19T14:20:21.020809+0000","last_peered":"2023-10-19T14:20:21.020809+0000","last_clean":"2023-10-19T14:20:21.020809+0000","last_became_active":"2023-10-19T14:20:19.350821+0000","last_became_peered":"2023-10-19T14:20:19.350821+0000","last_unstale":"2023-10-19T14:20:21.020809+0000","last_undegraded":"2023-10-19T14:20:21.020809+0000","last_fullsized":"2023-10-19T14:20:21.020809+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T02:16:40.560667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.32","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.383562+0000","last_change":"2023-10-19T14:20:19.346548+0000","last_active":"2023-10-19T14:20:21.383562+0000","last_peered":"2023-10-19T14:20:21.383562+0000","last_clean":"2023-10-19T14:20:21.383562+0000","last_became_active":"2023-10-19T14:20:19.345001+0000","last_became_peered":"2023-10-19T14:20:19.345001+0000","last_unstale":"2023-10-19T14:20:21.383562+0000","last_undegraded":"2023-10-19T14:20:21.383562+0000","last_fullsized":"2023-10-19T14:20:21.383562+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T02:03:47.400085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.31","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.382911+0000","last_change":"2023-10-19T14:20:19.349749+0000","last_active":"2023-10-19T14:20:21.382911+0000","last_peered":"2023-10-19T14:20:21.382911+0000","last_clean":"2023-10-19T14:20:21.382911+0000","last_became_active":"2023-10-19T14:20:19.349481+0000","last_became_peered":"2023-10-19T14:20:19.349481+0000","last_unstale":"2023-10-19T14:20:21.382911+0000","last_undegraded":"2023-10-19T14:20:21.382911+0000","last_fullsized":"2023-10-19T14:20:21.382911+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:20:35.750886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.30","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.922028+0000","last_change":"2023-10-19T14:20:19.338108+0000","last_active":"2023-10-19T14:20:20.922028+0000","last_peered":"2023-10-19T14:20:20.922028+0000","last_clean":"2023-10-19T14:20:20.922028+0000","last_became_active":"2023-10-19T14:20:19.337834+0000","last_became_peered":"2023-10-19T14:20:19.337834+0000","last_unstale":"2023-10-19T14:20:20.922028+0000","last_undegraded":"2023-10-19T14:20:20.922028+0000","last_fullsized":"2023-10-19T14:20:20.922028+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:15:21.019725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.2f","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.328664+0000","last_change":"2023-10-19T14:20:19.344630+0000","last_active":"2023-10-19T14:20:20.328664+0000","last_peered":"2023-10-19T14:20:20.328664+0000","last_clean":"2023-10-19T14:20:20.328664+0000","last_became_active":"2023-10-19T14:20:19.344354+0000","last_became_peered":"2023-10-19T14:20:19.344354+0000","last_unstale":"2023-10-19T14:20:20.328664+0000","last_undegraded":"2023-10-19T14:20:20.328664+0000","last_fullsized":"2023-10-19T14:20:20.328664+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:10:30.171693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.2e","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.328574+0000","last_change":"2023-10-19T14:20:19.341197+0000","last_active":"2023-10-19T14:20:20.328574+0000","last_peered":"2023-10-19T14:20:20.328574+0000","last_clean":"2023-10-19T14:20:20.328574+0000","last_became_active":"2023-10-19T14:20:19.340929+0000","last_became_peered":"2023-10-19T14:20:19.340929+0000","last_unstale":"2023-10-19T14:20:20.328574+0000","last_undegraded":"2023-10-19T14:20:20.328574+0000","last_fullsized":"2023-10-19T14:20:20.328574+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:44:28.147164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.2d","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.384001+0000","last_change":"2023-10-19T14:20:19.352942+0000","last_active":"2023-10-19T14:20:21.384001+0000","last_peered":"2023-10-19T14:20:21.384001+0000","last_clean":"2023-10-19T14:20:21.384001+0000","last_became_active":"2023-10-19T14:20:19.352678+0000","last_became_peered":"2023-10-19T14:20:19.352678+0000","last_unstale":"2023-10-19T14:20:21.384001+0000","last_undegraded":"2023-10-19T14:20:21.384001+0000","last_fullsized":"2023-10-19T14:20:21.384001+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:19:02.500176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2c","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.383918+0000","last_change":"2023-10-19T14:20:19.343670+0000","last_active":"2023-10-19T14:20:21.383918+0000","last_peered":"2023-10-19T14:20:21.383918+0000","last_clean":"2023-10-19T14:20:21.383918+0000","last_became_active":"2023-10-19T14:20:19.342491+0000","last_became_peered":"2023-10-19T14:20:19.342491+0000","last_unstale":"2023-10-19T14:20:21.383918+0000","last_undegraded":"2023-10-19T14:20:21.383918+0000","last_fullsized":"2023-10-19T14:20:21.383918+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:32:39.312387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2b","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.383630+0000","last_change":"2023-10-19T14:20:19.351751+0000","last_active":"2023-10-19T14:20:21.383630+0000","last_peered":"2023-10-19T14:20:21.383630+0000","last_clean":"2023-10-19T14:20:21.383630+0000","last_became_active":"2023-10-19T14:20:19.350197+0000","last_became_peered":"2023-10-19T14:20:19.350197+0000","last_unstale":"2023-10-19T14:20:21.383630+0000","last_undegraded":"2023-10-19T14:20:21.383630+0000","last_fullsized":"2023-10-19T14:20:21.383630+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:18:10.058277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2a","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.383884+0000","last_change":"2023-10-19T14:20:19.345667+0000","last_active":"2023-10-19T14:20:21.383884+0000","last_peered":"2023-10-19T14:20:21.383884+0000","last_clean":"2023-10-19T14:20:21.383884+0000","last_became_active":"2023-10-19T14:20:19.345343+0000","last_became_peered":"2023-10-19T14:20:19.345343+0000","last_unstale":"2023-10-19T14:20:21.383884+0000","last_undegraded":"2023-10-19T14:20:21.383884+0000","last_fullsized":"2023-10-19T14:20:21.383884+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:36:14.871743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.29","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.331283+0000","last_change":"2023-10-19T14:20:19.333467+0000","last_active":"2023-10-19T14:20:20.331283+0000","last_peered":"2023-10-19T14:20:20.331283+0000","last_clean":"2023-10-19T14:20:20.331283+0000","last_became_active":"2023-10-19T14:20:19.333176+0000","last_became_peered":"2023-10-19T14:20:19.333176+0000","last_unstale":"2023-10-19T14:20:20.331283+0000","last_undegraded":"2023-10-19T14:20:20.331283+0000","last_fullsized":"2023-10-19T14:20:20.331283+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:39:10.256824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.28","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.382366+0000","last_change":"2023-10-19T14:20:19.341549+0000","last_active":"2023-10-19T14:20:21.382366+0000","last_peered":"2023-10-19T14:20:21.382366+0000","last_clean":"2023-10-19T14:20:21.382366+0000","last_became_active":"2023-10-19T14:20:19.341276+0000","last_became_peered":"2023-10-19T14:20:19.341276+0000","last_unstale":"2023-10-19T14:20:21.382366+0000","last_undegraded":"2023-10-19T14:20:21.382366+0000","last_fullsized":"2023-10-19T14:20:21.382366+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:23:15.830657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.27","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.328747+0000","last_change":"2023-10-19T14:20:19.339558+0000","last_active":"2023-10-19T14:20:20.328747+0000","last_peered":"2023-10-19T14:20:20.328747+0000","last_clean":"2023-10-19T14:20:20.328747+0000","last_became_active":"2023-10-19T14:20:19.339298+0000","last_became_peered":"2023-10-19T14:20:19.339298+0000","last_unstale":"2023-10-19T14:20:20.328747+0000","last_undegraded":"2023-10-19T14:20:20.328747+0000","last_fullsized":"2023-10-19T14:20:20.328747+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:27:05.614080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.26","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.332703+0000","last_change":"2023-10-19T14:20:19.348136+0000","last_active":"2023-10-19T14:20:20.332703+0000","last_peered":"2023-10-19T14:20:20.332703+0000","last_clean":"2023-10-19T14:20:20.332703+0000","last_became_active":"2023-10-19T14:20:19.347832+0000","last_became_peered":"2023-10-19T14:20:19.347832+0000","last_unstale":"2023-10-19T14:20:20.332703+0000","last_undegraded":"2023-10-19T14:20:20.332703+0000","last_fullsized":"2023-10-19T14:20:20.332703+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:16:46.603067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.25","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.922367+0000","last_change":"2023-10-19T14:20:19.347669+0000","last_active":"2023-10-19T14:20:20.922367+0000","last_peered":"2023-10-19T14:20:20.922367+0000","last_clean":"2023-10-19T14:20:20.922367+0000","last_became_active":"2023-10-19T14:20:19.347397+0000","last_became_peered":"2023-10-19T14:20:19.347397+0000","last_unstale":"2023-10-19T14:20:20.922367+0000","last_undegraded":"2023-10-19T14:20:20.922367+0000","last_fullsized":"2023-10-19T14:20:20.922367+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:47:46.746053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.24","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.382730+0000","last_change":"2023-10-19T14:20:19.345949+0000","last_active":"2023-10-19T14:20:21.382730+0000","last_peered":"2023-10-19T14:20:21.382730+0000","last_clean":"2023-10-19T14:20:21.382730+0000","last_became_active":"2023-10-19T14:20:19.345664+0000","last_became_peered":"2023-10-19T14:20:19.345664+0000","last_unstale":"2023-10-19T14:20:21.382730+0000","last_undegraded":"2023-10-19T14:20:21.382730+0000","last_fullsized":"2023-10-19T14:20:21.382730+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:52:23.062217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.23","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.328362+0000","last_change":"2023-10-19T14:20:19.342642+0000","last_active":"2023-10-19T14:20:20.328362+0000","last_peered":"2023-10-19T14:20:20.328362+0000","last_clean":"2023-10-19T14:20:20.328362+0000","last_became_active":"2023-10-19T14:20:19.342379+0000","last_became_peered":"2023-10-19T14:20:19.342379+0000","last_unstale":"2023-10-19T14:20:20.328362+0000","last_undegraded":"2023-10-19T14:20:20.328362+0000","last_fullsized":"2023-10-19T14:20:20.328362+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:33:34.581616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.22","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.382958+0000","last_change":"2023-10-19T14:20:19.342297+0000","last_active":"2023-10-19T14:20:21.382958+0000","last_peered":"2023-10-19T14:20:21.382958+0000","last_clean":"2023-10-19T14:20:21.382958+0000","last_became_active":"2023-10-19T14:20:19.342026+0000","last_became_peered":"2023-10-19T14:20:19.342026+0000","last_unstale":"2023-10-19T14:20:21.382958+0000","last_undegraded":"2023-10-19T14:20:21.382958+0000","last_fullsized":"2023-10-19T14:20:21.382958+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:43:57.179927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.21","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.020598+0000","last_change":"2023-10-19T14:20:19.349579+0000","last_active":"2023-10-19T14:20:21.020598+0000","last_peered":"2023-10-19T14:20:21.020598+0000","last_clean":"2023-10-19T14:20:21.020598+0000","last_became_active":"2023-10-19T14:20:19.349321+0000","last_became_peered":"2023-10-19T14:20:19.349321+0000","last_unstale":"2023-10-19T14:20:21.020598+0000","last_undegraded":"2023-10-19T14:20:21.020598+0000","last_fullsized":"2023-10-19T14:20:21.020598+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:44:03.588824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.20","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.020600+0000","last_change":"2023-10-19T14:20:19.348899+0000","last_active":"2023-10-19T14:20:21.020600+0000","last_peered":"2023-10-19T14:20:21.020600+0000","last_clean":"2023-10-19T14:20:21.020600+0000","last_became_active":"2023-10-19T14:20:19.348453+0000","last_became_peered":"2023-10-19T14:20:19.348453+0000","last_unstale":"2023-10-19T14:20:21.020600+0000","last_undegraded":"2023-10-19T14:20:21.020600+0000","last_fullsized":"2023-10-19T14:20:21.020600+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:01:53.425378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.331310+0000","last_change":"2023-10-19T14:20:19.333259+0000","last_active":"2023-10-19T14:20:20.331310+0000","last_peered":"2023-10-19T14:20:20.331310+0000","last_clean":"2023-10-19T14:20:20.331310+0000","last_became_active":"2023-10-19T14:20:19.332735+0000","last_became_peered":"2023-10-19T14:20:19.332735+0000","last_unstale":"2023-10-19T14:20:20.331310+0000","last_undegraded":"2023-10-19T14:20:20.331310+0000","last_fullsized":"2023-10-19T14:20:20.331310+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:46:26.663119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.1f","version":"222'4","reported_seq":16,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.383194+0000","last_change":"2023-10-19T14:20:18.725431+0000","last_active":"2023-10-19T14:20:21.383194+0000","last_peered":"2023-10-19T14:20:21.383194+0000","last_clean":"2023-10-19T14:20:21.383194+0000","last_became_active":"2023-10-19T14:20:18.725142+0000","last_became_peered":"2023-10-19T14:20:18.725142+0000","last_unstale":"2023-10-19T14:20:21.383194+0000","last_undegraded":"2023-10-19T14:20:21.383194+0000","last_fullsized":"2023-10-19T14:20:21.383194+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T02:03:46.398046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.332233+0000","last_change":"2023-10-19T14:20:19.343393+0000","last_active":"2023-10-19T14:20:20.332233+0000","last_peered":"2023-10-19T14:20:20.332233+0000","last_clean":"2023-10-19T14:20:20.332233+0000","last_became_active":"2023-10-19T14:20:19.343011+0000","last_became_peered":"2023-10-19T14:20:19.343011+0000","last_unstale":"2023-10-19T14:20:20.332233+0000","last_undegraded":"2023-10-19T14:20:20.332233+0000","last_fullsized":"2023-10-19T14:20:20.332233+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:22:47.864868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1e","version":"222'1","reported_seq":13,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.332751+0000","last_change":"2023-10-19T14:20:18.719165+0000","last_active":"2023-10-19T14:20:20.332751+0000","last_peered":"2023-10-19T14:20:20.332751+0000","last_clean":"2023-10-19T14:20:20.332751+0000","last_became_active":"2023-10-19T14:20:18.717146+0000","last_became_peered":"2023-10-19T14:20:18.717146+0000","last_unstale":"2023-10-19T14:20:20.332751+0000","last_undegraded":"2023-10-19T14:20:20.332751+0000","last_fullsized":"2023-10-19T14:20:20.332751+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:57:14.320702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.1a","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.383778+0000","last_change":"2023-10-19T14:20:19.346318+0000","last_active":"2023-10-19T14:20:21.383778+0000","last_peered":"2023-10-19T14:20:21.383778+0000","last_clean":"2023-10-19T14:20:21.383778+0000","last_became_active":"2023-10-19T14:20:19.346014+0000","last_became_peered":"2023-10-19T14:20:19.346014+0000","last_unstale":"2023-10-19T14:20:21.383778+0000","last_undegraded":"2023-10-19T14:20:21.383778+0000","last_fullsized":"2023-10-19T14:20:21.383778+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:20:35.750886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1d","version":"222'2","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.331596+0000","last_change":"2023-10-19T14:20:18.717342+0000","last_active":"2023-10-19T14:20:20.331596+0000","last_peered":"2023-10-19T14:20:20.331596+0000","last_clean":"2023-10-19T14:20:20.331596+0000","last_became_active":"2023-10-19T14:20:18.717047+0000","last_became_peered":"2023-10-19T14:20:18.717047+0000","last_unstale":"2023-10-19T14:20:20.331596+0000","last_undegraded":"2023-10-19T14:20:20.331596+0000","last_fullsized":"2023-10-19T14:20:20.331596+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:28:55.961320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.1b","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.382554+0000","last_change":"2023-10-19T14:20:19.348254+0000","last_active":"2023-10-19T14:20:21.382554+0000","last_peered":"2023-10-19T14:20:21.382554+0000","last_clean":"2023-10-19T14:20:21.382554+0000","last_became_active":"2023-10-19T14:20:19.347974+0000","last_became_peered":"2023-10-19T14:20:19.347974+0000","last_unstale":"2023-10-19T14:20:21.382554+0000","last_undegraded":"2023-10-19T14:20:21.382554+0000","last_fullsized":"2023-10-19T14:20:21.382554+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:32:42.913042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1c","version":"222'2","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.328509+0000","last_change":"2023-10-19T14:20:18.722673+0000","last_active":"2023-10-19T14:20:20.328509+0000","last_peered":"2023-10-19T14:20:20.328509+0000","last_clean":"2023-10-19T14:20:20.328509+0000","last_became_active":"2023-10-19T14:20:18.722418+0000","last_became_peered":"2023-10-19T14:20:18.722418+0000","last_unstale":"2023-10-19T14:20:20.328509+0000","last_undegraded":"2023-10-19T14:20:20.328509+0000","last_fullsized":"2023-10-19T14:20:20.328509+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:40:32.510221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1c","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.383205+0000","last_change":"2023-10-19T14:20:19.349815+0000","last_active":"2023-10-19T14:20:21.383205+0000","last_peered":"2023-10-19T14:20:21.383205+0000","last_clean":"2023-10-19T14:20:21.383205+0000","last_became_active":"2023-10-19T14:20:19.349511+0000","last_became_peered":"2023-10-19T14:20:19.349511+0000","last_unstale":"2023-10-19T14:20:21.383205+0000","last_undegraded":"2023-10-19T14:20:21.383205+0000","last_fullsized":"2023-10-19T14:20:21.383205+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:32:39.312387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.1b","version":"222'1","reported_seq":13,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.020920+0000","last_change":"2023-10-19T14:20:18.722476+0000","last_active":"2023-10-19T14:20:21.020920+0000","last_peered":"2023-10-19T14:20:21.020920+0000","last_clean":"2023-10-19T14:20:21.020920+0000","last_became_active":"2023-10-19T14:20:18.722195+0000","last_became_peered":"2023-10-19T14:20:18.722195+0000","last_unstale":"2023-10-19T14:20:21.020920+0000","last_undegraded":"2023-10-19T14:20:21.020920+0000","last_fullsized":"2023-10-19T14:20:21.020920+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:38:13.906613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1e","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.383161+0000","last_change":"2023-10-19T14:20:19.343379+0000","last_active":"2023-10-19T14:20:21.383161+0000","last_peered":"2023-10-19T14:20:21.383161+0000","last_clean":"2023-10-19T14:20:21.383161+0000","last_became_active":"2023-10-19T14:20:19.343117+0000","last_became_peered":"2023-10-19T14:20:19.343117+0000","last_unstale":"2023-10-19T14:20:21.383161+0000","last_undegraded":"2023-10-19T14:20:21.383161+0000","last_fullsized":"2023-10-19T14:20:21.383161+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:33:52.226441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":12,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.020524+0000","last_change":"2023-10-19T14:20:18.721875+0000","last_active":"2023-10-19T14:20:21.020524+0000","last_peered":"2023-10-19T14:20:21.020524+0000","last_clean":"2023-10-19T14:20:21.020524+0000","last_became_active":"2023-10-19T14:20:18.721611+0000","last_became_peered":"2023-10-19T14:20:18.721611+0000","last_unstale":"2023-10-19T14:20:21.020524+0000","last_undegraded":"2023-10-19T14:20:21.020524+0000","last_fullsized":"2023-10-19T14:20:21.020524+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:46:19.718145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1f","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.328790+0000","last_change":"2023-10-19T14:20:19.346582+0000","last_active":"2023-10-19T14:20:20.328790+0000","last_peered":"2023-10-19T14:20:20.328790+0000","last_clean":"2023-10-19T14:20:20.328790+0000","last_became_active":"2023-10-19T14:20:19.346246+0000","last_became_peered":"2023-10-19T14:20:19.346246+0000","last_unstale":"2023-10-19T14:20:20.328790+0000","last_undegraded":"2023-10-19T14:20:20.328790+0000","last_fullsized":"2023-10-19T14:20:20.328790+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:37:58.222092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":12,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.922070+0000","last_change":"2023-10-19T14:20:18.718913+0000","last_active":"2023-10-19T14:20:20.922070+0000","last_peered":"2023-10-19T14:20:20.922070+0000","last_clean":"2023-10-19T14:20:20.922070+0000","last_became_active":"2023-10-19T14:20:18.718641+0000","last_became_peered":"2023-10-19T14:20:18.718641+0000","last_unstale":"2023-10-19T14:20:20.922070+0000","last_undegraded":"2023-10-19T14:20:20.922070+0000","last_fullsized":"2023-10-19T14:20:20.922070+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:54:40.217789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.10","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.382991+0000","last_change":"2023-10-19T14:20:19.351668+0000","last_active":"2023-10-19T14:20:21.382991+0000","last_peered":"2023-10-19T14:20:21.382991+0000","last_clean":"2023-10-19T14:20:21.382991+0000","last_became_active":"2023-10-19T14:20:19.351298+0000","last_became_peered":"2023-10-19T14:20:19.351298+0000","last_unstale":"2023-10-19T14:20:21.382991+0000","last_undegraded":"2023-10-19T14:20:21.382991+0000","last_fullsized":"2023-10-19T14:20:21.382991+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:36:14.871743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":12,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.382838+0000","last_change":"2023-10-19T14:20:18.722070+0000","last_active":"2023-10-19T14:20:21.382838+0000","last_peered":"2023-10-19T14:20:21.382838+0000","last_clean":"2023-10-19T14:20:21.382838+0000","last_became_active":"2023-10-19T14:20:18.721791+0000","last_became_peered":"2023-10-19T14:20:18.721791+0000","last_unstale":"2023-10-19T14:20:21.382838+0000","last_undegraded":"2023-10-19T14:20:21.382838+0000","last_fullsized":"2023-10-19T14:20:21.382838+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:17:55.048333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.922093+0000","last_change":"2023-10-19T14:20:19.339048+0000","last_active":"2023-10-19T14:20:20.922093+0000","last_peered":"2023-10-19T14:20:20.922093+0000","last_clean":"2023-10-19T14:20:20.922093+0000","last_became_active":"2023-10-19T14:20:19.338786+0000","last_became_peered":"2023-10-19T14:20:19.338786+0000","last_unstale":"2023-10-19T14:20:20.922093+0000","last_undegraded":"2023-10-19T14:20:20.922093+0000","last_fullsized":"2023-10-19T14:20:20.922093+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:16:46.603067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":12,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.021070+0000","last_change":"2023-10-19T14:20:18.721992+0000","last_active":"2023-10-19T14:20:21.021070+0000","last_peered":"2023-10-19T14:20:21.021070+0000","last_clean":"2023-10-19T14:20:21.021070+0000","last_became_active":"2023-10-19T14:20:18.721742+0000","last_became_peered":"2023-10-19T14:20:18.721742+0000","last_unstale":"2023-10-19T14:20:21.021070+0000","last_undegraded":"2023-10-19T14:20:21.021070+0000","last_fullsized":"2023-10-19T14:20:21.021070+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:16:37.567429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.328317+0000","last_change":"2023-10-19T14:20:19.334253+0000","last_active":"2023-10-19T14:20:20.328317+0000","last_peered":"2023-10-19T14:20:20.328317+0000","last_clean":"2023-10-19T14:20:20.328317+0000","last_became_active":"2023-10-19T14:20:19.333981+0000","last_became_peered":"2023-10-19T14:20:19.333981+0000","last_unstale":"2023-10-19T14:20:20.328317+0000","last_undegraded":"2023-10-19T14:20:20.328317+0000","last_fullsized":"2023-10-19T14:20:20.328317+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:48:00.239750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.15","version":"222'3","reported_seq":15,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.328592+0000","last_change":"2023-10-19T14:20:18.723409+0000","last_active":"2023-10-19T14:20:20.328592+0000","last_peered":"2023-10-19T14:20:20.328592+0000","last_clean":"2023-10-19T14:20:20.328592+0000","last_became_active":"2023-10-19T14:20:18.723148+0000","last_became_peered":"2023-10-19T14:20:18.723148+0000","last_unstale":"2023-10-19T14:20:20.328592+0000","last_undegraded":"2023-10-19T14:20:20.328592+0000","last_fullsized":"2023-10-19T14:20:20.328592+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:13:53.110830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.13","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.382663+0000","last_change":"2023-10-19T14:20:19.350427+0000","last_active":"2023-10-19T14:20:21.382663+0000","last_peered":"2023-10-19T14:20:21.382663+0000","last_clean":"2023-10-19T14:20:21.382663+0000","last_became_active":"2023-10-19T14:20:19.350112+0000","last_became_peered":"2023-10-19T14:20:19.350112+0000","last_unstale":"2023-10-19T14:20:21.382663+0000","last_undegraded":"2023-10-19T14:20:21.382663+0000","last_fullsized":"2023-10-19T14:20:21.382663+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:01:38.993506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.14","version":"222'6","reported_seq":18,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.020941+0000","last_change":"2023-10-19T14:20:18.727225+0000","last_active":"2023-10-19T14:20:21.020941+0000","last_peered":"2023-10-19T14:20:21.020941+0000","last_clean":"2023-10-19T14:20:21.020941+0000","last_became_active":"2023-10-19T14:20:18.726839+0000","last_became_peered":"2023-10-19T14:20:18.726839+0000","last_unstale":"2023-10-19T14:20:21.020941+0000","last_undegraded":"2023-10-19T14:20:21.020941+0000","last_fullsized":"2023-10-19T14:20:21.020941+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:10:54.369994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.14","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.331654+0000","last_change":"2023-10-19T14:20:19.340471+0000","last_active":"2023-10-19T14:20:20.331654+0000","last_peered":"2023-10-19T14:20:20.331654+0000","last_clean":"2023-10-19T14:20:20.331654+0000","last_became_active":"2023-10-19T14:20:19.340187+0000","last_became_peered":"2023-10-19T14:20:19.340187+0000","last_unstale":"2023-10-19T14:20:20.331654+0000","last_undegraded":"2023-10-19T14:20:20.331654+0000","last_fullsized":"2023-10-19T14:20:20.331654+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:53:57.462566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.13","version":"222'2","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.331524+0000","last_change":"2023-10-19T14:20:18.713594+0000","last_active":"2023-10-19T14:20:20.331524+0000","last_peered":"2023-10-19T14:20:20.331524+0000","last_clean":"2023-10-19T14:20:20.331524+0000","last_became_active":"2023-10-19T14:20:18.713263+0000","last_became_peered":"2023-10-19T14:20:18.713263+0000","last_unstale":"2023-10-19T14:20:20.331524+0000","last_undegraded":"2023-10-19T14:20:20.331524+0000","last_fullsized":"2023-10-19T14:20:20.331524+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:32:02.753539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.15","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.922354+0000","last_change":"2023-10-19T14:20:19.349089+0000","last_active":"2023-10-19T14:20:20.922354+0000","last_peered":"2023-10-19T14:20:20.922354+0000","last_clean":"2023-10-19T14:20:20.922354+0000","last_became_active":"2023-10-19T14:20:19.348826+0000","last_became_peered":"2023-10-19T14:20:19.348826+0000","last_unstale":"2023-10-19T14:20:20.922354+0000","last_undegraded":"2023-10-19T14:20:20.922354+0000","last_fullsized":"2023-10-19T14:20:20.922354+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:10:55.372033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":12,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.922121+0000","last_change":"2023-10-19T14:20:18.709786+0000","last_active":"2023-10-19T14:20:20.922121+0000","last_peered":"2023-10-19T14:20:20.922121+0000","last_clean":"2023-10-19T14:20:20.922121+0000","last_became_active":"2023-10-19T14:20:18.709496+0000","last_became_peered":"2023-10-19T14:20:18.709496+0000","last_unstale":"2023-10-19T14:20:20.922121+0000","last_undegraded":"2023-10-19T14:20:20.922121+0000","last_fullsized":"2023-10-19T14:20:20.922121+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:46:59.026062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.328599+0000","last_change":"2023-10-19T14:20:19.343343+0000","last_active":"2023-10-19T14:20:20.328599+0000","last_peered":"2023-10-19T14:20:20.328599+0000","last_clean":"2023-10-19T14:20:20.328599+0000","last_became_active":"2023-10-19T14:20:19.343059+0000","last_became_peered":"2023-10-19T14:20:19.343059+0000","last_unstale":"2023-10-19T14:20:20.328599+0000","last_undegraded":"2023-10-19T14:20:20.328599+0000","last_fullsized":"2023-10-19T14:20:20.328599+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:31:59.664720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":12,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.328232+0000","last_change":"2023-10-19T14:20:18.717768+0000","last_active":"2023-10-19T14:20:20.328232+0000","last_peered":"2023-10-19T14:20:20.328232+0000","last_clean":"2023-10-19T14:20:20.328232+0000","last_became_active":"2023-10-19T14:20:18.717375+0000","last_became_peered":"2023-10-19T14:20:18.717375+0000","last_unstale":"2023-10-19T14:20:20.328232+0000","last_undegraded":"2023-10-19T14:20:20.328232+0000","last_fullsized":"2023-10-19T14:20:20.328232+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:44:02.586786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.020980+0000","last_change":"2023-10-19T14:20:19.353248+0000","last_active":"2023-10-19T14:20:21.020980+0000","last_peered":"2023-10-19T14:20:21.020980+0000","last_clean":"2023-10-19T14:20:21.020980+0000","last_became_active":"2023-10-19T14:20:19.352973+0000","last_became_peered":"2023-10-19T14:20:19.352973+0000","last_unstale":"2023-10-19T14:20:21.020980+0000","last_undegraded":"2023-10-19T14:20:21.020980+0000","last_fullsized":"2023-10-19T14:20:21.020980+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:07:18.725863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.10","version":"222'8","reported_seq":20,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.382313+0000","last_change":"2023-10-19T14:20:18.722891+0000","last_active":"2023-10-19T14:20:21.382313+0000","last_peered":"2023-10-19T14:20:21.382313+0000","last_clean":"2023-10-19T14:20:21.382313+0000","last_became_active":"2023-10-19T14:20:18.722633+0000","last_became_peered":"2023-10-19T14:20:18.722633+0000","last_unstale":"2023-10-19T14:20:21.382313+0000","last_undegraded":"2023-10-19T14:20:21.382313+0000","last_fullsized":"2023-10-19T14:20:21.382313+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:32:21.255137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.383389+0000","last_change":"2023-10-19T14:20:19.349188+0000","last_active":"2023-10-19T14:20:21.383389+0000","last_peered":"2023-10-19T14:20:21.383389+0000","last_clean":"2023-10-19T14:20:21.383389+0000","last_became_active":"2023-10-19T14:20:19.348905+0000","last_became_peered":"2023-10-19T14:20:19.348905+0000","last_unstale":"2023-10-19T14:20:21.383389+0000","last_undegraded":"2023-10-19T14:20:21.383389+0000","last_fullsized":"2023-10-19T14:20:21.383389+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:36:36.169729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.f","version":"222'1","reported_seq":13,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.332614+0000","last_change":"2023-10-19T14:20:18.719346+0000","last_active":"2023-10-19T14:20:20.332614+0000","last_peered":"2023-10-19T14:20:20.332614+0000","last_clean":"2023-10-19T14:20:20.332614+0000","last_became_active":"2023-10-19T14:20:18.719101+0000","last_became_peered":"2023-10-19T14:20:18.719101+0000","last_unstale":"2023-10-19T14:20:20.332614+0000","last_undegraded":"2023-10-19T14:20:20.332614+0000","last_fullsized":"2023-10-19T14:20:20.332614+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:01:11.976346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.332287+0000","last_change":"2023-10-19T14:20:19.348569+0000","last_active":"2023-10-19T14:20:20.332287+0000","last_peered":"2023-10-19T14:20:20.332287+0000","last_clean":"2023-10-19T14:20:20.332287+0000","last_became_active":"2023-10-19T14:20:19.348268+0000","last_became_peered":"2023-10-19T14:20:19.348268+0000","last_unstale":"2023-10-19T14:20:20.332287+0000","last_undegraded":"2023-10-19T14:20:20.332287+0000","last_fullsized":"2023-10-19T14:20:20.332287+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:54:41.219828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.e","version":"222'2","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.332760+0000","last_change":"2023-10-19T14:20:18.721114+0000","last_active":"2023-10-19T14:20:20.332760+0000","last_peered":"2023-10-19T14:20:20.332760+0000","last_clean":"2023-10-19T14:20:20.332760+0000","last_became_active":"2023-10-19T14:20:18.720840+0000","last_became_peered":"2023-10-19T14:20:18.720840+0000","last_unstale":"2023-10-19T14:20:20.332760+0000","last_undegraded":"2023-10-19T14:20:20.332760+0000","last_fullsized":"2023-10-19T14:20:20.332760+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:44:50.967568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.332574+0000","last_change":"2023-10-19T14:20:19.332710+0000","last_active":"2023-10-19T14:20:20.332574+0000","last_peered":"2023-10-19T14:20:20.332574+0000","last_clean":"2023-10-19T14:20:20.332574+0000","last_became_active":"2023-10-19T14:20:19.332132+0000","last_became_peered":"2023-10-19T14:20:19.332132+0000","last_unstale":"2023-10-19T14:20:20.332574+0000","last_undegraded":"2023-10-19T14:20:20.332574+0000","last_fullsized":"2023-10-19T14:20:20.332574+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:36:15.681007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.d","version":"222'1","reported_seq":13,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.331623+0000","last_change":"2023-10-19T14:20:18.719857+0000","last_active":"2023-10-19T14:20:20.331623+0000","last_peered":"2023-10-19T14:20:20.331623+0000","last_clean":"2023-10-19T14:20:20.331623+0000","last_became_active":"2023-10-19T14:20:18.719586+0000","last_became_peered":"2023-10-19T14:20:18.719586+0000","last_unstale":"2023-10-19T14:20:20.331623+0000","last_undegraded":"2023-10-19T14:20:20.331623+0000","last_fullsized":"2023-10-19T14:20:20.331623+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:12:37.668857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.b","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.383814+0000","last_change":"2023-10-19T14:20:19.352403+0000","last_active":"2023-10-19T14:20:21.383814+0000","last_peered":"2023-10-19T14:20:21.383814+0000","last_clean":"2023-10-19T14:20:21.383814+0000","last_became_active":"2023-10-19T14:20:19.351892+0000","last_became_peered":"2023-10-19T14:20:19.351892+0000","last_unstale":"2023-10-19T14:20:21.383814+0000","last_undegraded":"2023-10-19T14:20:21.383814+0000","last_fullsized":"2023-10-19T14:20:21.383814+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:33:52.226441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":12,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.382772+0000","last_change":"2023-10-19T14:20:18.717037+0000","last_active":"2023-10-19T14:20:21.382772+0000","last_peered":"2023-10-19T14:20:21.382772+0000","last_clean":"2023-10-19T14:20:21.382772+0000","last_became_active":"2023-10-19T14:20:18.716731+0000","last_became_peered":"2023-10-19T14:20:18.716731+0000","last_unstale":"2023-10-19T14:20:21.382772+0000","last_undegraded":"2023-10-19T14:20:21.382772+0000","last_fullsized":"2023-10-19T14:20:21.382772+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:50:59.017027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.331668+0000","last_change":"2023-10-19T14:20:19.346026+0000","last_active":"2023-10-19T14:20:20.331668+0000","last_peered":"2023-10-19T14:20:20.331668+0000","last_clean":"2023-10-19T14:20:20.331668+0000","last_became_active":"2023-10-19T14:20:19.345762+0000","last_became_peered":"2023-10-19T14:20:19.345762+0000","last_unstale":"2023-10-19T14:20:20.331668+0000","last_undegraded":"2023-10-19T14:20:20.331668+0000","last_fullsized":"2023-10-19T14:20:20.331668+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:28:21.209211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.b","version":"222'2","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.382612+0000","last_change":"2023-10-19T14:20:18.719825+0000","last_active":"2023-10-19T14:20:21.382612+0000","last_peered":"2023-10-19T14:20:21.382612+0000","last_clean":"2023-10-19T14:20:21.382612+0000","last_became_active":"2023-10-19T14:20:18.719549+0000","last_became_peered":"2023-10-19T14:20:18.719549+0000","last_unstale":"2023-10-19T14:20:21.382612+0000","last_undegraded":"2023-10-19T14:20:21.382612+0000","last_fullsized":"2023-10-19T14:20:21.382612+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T02:18:13.024084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.d","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.020866+0000","last_change":"2023-10-19T14:20:19.350548+0000","last_active":"2023-10-19T14:20:21.020866+0000","last_peered":"2023-10-19T14:20:21.020866+0000","last_clean":"2023-10-19T14:20:21.020866+0000","last_became_active":"2023-10-19T14:20:19.349339+0000","last_became_peered":"2023-10-19T14:20:19.349339+0000","last_unstale":"2023-10-19T14:20:21.020866+0000","last_undegraded":"2023-10-19T14:20:21.020866+0000","last_fullsized":"2023-10-19T14:20:21.020866+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:13:54.112869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":12,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.331433+0000","last_change":"2023-10-19T14:20:18.711157+0000","last_active":"2023-10-19T14:20:20.331433+0000","last_peered":"2023-10-19T14:20:20.331433+0000","last_clean":"2023-10-19T14:20:20.331433+0000","last_became_active":"2023-10-19T14:20:18.710747+0000","last_became_peered":"2023-10-19T14:20:18.710747+0000","last_unstale":"2023-10-19T14:20:20.331433+0000","last_undegraded":"2023-10-19T14:20:20.331433+0000","last_fullsized":"2023-10-19T14:20:20.331433+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:46:05.844692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.e","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.328663+0000","last_change":"2023-10-19T14:20:19.345329+0000","last_active":"2023-10-19T14:20:20.328663+0000","last_peered":"2023-10-19T14:20:20.328663+0000","last_clean":"2023-10-19T14:20:20.328663+0000","last_became_active":"2023-10-19T14:20:19.344983+0000","last_became_peered":"2023-10-19T14:20:19.344983+0000","last_unstale":"2023-10-19T14:20:20.328663+0000","last_undegraded":"2023-10-19T14:20:20.328663+0000","last_fullsized":"2023-10-19T14:20:20.328663+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:17:53.794653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":12,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.382937+0000","last_change":"2023-10-19T14:20:18.727394+0000","last_active":"2023-10-19T14:20:21.382937+0000","last_peered":"2023-10-19T14:20:21.382937+0000","last_clean":"2023-10-19T14:20:21.382937+0000","last_became_active":"2023-10-19T14:20:18.727050+0000","last_became_peered":"2023-10-19T14:20:18.727050+0000","last_unstale":"2023-10-19T14:20:21.382937+0000","last_undegraded":"2023-10-19T14:20:21.382937+0000","last_fullsized":"2023-10-19T14:20:21.382937+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:13:48.534905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.f","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.384115+0000","last_change":"2023-10-19T14:20:19.346087+0000","last_active":"2023-10-19T14:20:21.384115+0000","last_peered":"2023-10-19T14:20:21.384115+0000","last_clean":"2023-10-19T14:20:21.384115+0000","last_became_active":"2023-10-19T14:20:19.345704+0000","last_became_peered":"2023-10-19T14:20:19.345704+0000","last_unstale":"2023-10-19T14:20:21.384115+0000","last_undegraded":"2023-10-19T14:20:21.384115+0000","last_fullsized":"2023-10-19T14:20:21.384115+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:53:30.184157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":12,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.332172+0000","last_change":"2023-10-19T14:20:18.715329+0000","last_active":"2023-10-19T14:20:20.332172+0000","last_peered":"2023-10-19T14:20:20.332172+0000","last_clean":"2023-10-19T14:20:20.332172+0000","last_became_active":"2023-10-19T14:20:18.715068+0000","last_became_peered":"2023-10-19T14:20:18.715068+0000","last_unstale":"2023-10-19T14:20:20.332172+0000","last_undegraded":"2023-10-19T14:20:20.332172+0000","last_fullsized":"2023-10-19T14:20:20.332172+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:35:10.527276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.383603+0000","last_change":"2023-10-19T14:20:19.349129+0000","last_active":"2023-10-19T14:20:21.383603+0000","last_peered":"2023-10-19T14:20:21.383603+0000","last_clean":"2023-10-19T14:20:21.383603+0000","last_became_active":"2023-10-19T14:20:19.348846+0000","last_became_peered":"2023-10-19T14:20:19.348846+0000","last_unstale":"2023-10-19T14:20:21.383603+0000","last_undegraded":"2023-10-19T14:20:21.383603+0000","last_fullsized":"2023-10-19T14:20:21.383603+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:04:34.009285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.7","version":"222'2","reported_seq":14,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.328793+0000","last_change":"2023-10-19T14:20:18.724203+0000","last_active":"2023-10-19T14:20:20.328793+0000","last_peered":"2023-10-19T14:20:20.328793+0000","last_clean":"2023-10-19T14:20:20.328793+0000","last_became_active":"2023-10-19T14:20:18.723883+0000","last_became_peered":"2023-10-19T14:20:18.723883+0000","last_unstale":"2023-10-19T14:20:20.328793+0000","last_undegraded":"2023-10-19T14:20:20.328793+0000","last_fullsized":"2023-10-19T14:20:20.328793+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:38:31.255506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.922094+0000","last_change":"2023-10-19T14:20:19.344287+0000","last_active":"2023-10-19T14:20:20.922094+0000","last_peered":"2023-10-19T14:20:20.922094+0000","last_clean":"2023-10-19T14:20:20.922094+0000","last_became_active":"2023-10-19T14:20:19.344030+0000","last_became_peered":"2023-10-19T14:20:19.344030+0000","last_unstale":"2023-10-19T14:20:20.922094+0000","last_undegraded":"2023-10-19T14:20:20.922094+0000","last_fullsized":"2023-10-19T14:20:20.922094+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:35:58.143784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.6","version":"0'0","reported_seq":12,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.383985+0000","last_change":"2023-10-19T14:20:18.722733+0000","last_active":"2023-10-19T14:20:21.383985+0000","last_peered":"2023-10-19T14:20:21.383985+0000","last_clean":"2023-10-19T14:20:21.383985+0000","last_became_active":"2023-10-19T14:20:18.722447+0000","last_became_peered":"2023-10-19T14:20:18.722447+0000","last_unstale":"2023-10-19T14:20:21.383985+0000","last_undegraded":"2023-10-19T14:20:21.383985+0000","last_fullsized":"2023-10-19T14:20:21.383985+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:28:23.768143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.328321+0000","last_change":"2023-10-19T14:20:19.340725+0000","last_active":"2023-10-19T14:20:20.328321+0000","last_peered":"2023-10-19T14:20:20.328321+0000","last_clean":"2023-10-19T14:20:20.328321+0000","last_became_active":"2023-10-19T14:20:19.340284+0000","last_became_peered":"2023-10-19T14:20:19.340284+0000","last_unstale":"2023-10-19T14:20:20.328321+0000","last_undegraded":"2023-10-19T14:20:20.328321+0000","last_fullsized":"2023-10-19T14:20:20.328321+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:16:20.216590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":12,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.383019+0000","last_change":"2023-10-19T14:20:18.724127+0000","last_active":"2023-10-19T14:20:21.383019+0000","last_peered":"2023-10-19T14:20:21.383019+0000","last_clean":"2023-10-19T14:20:21.383019+0000","last_became_active":"2023-10-19T14:20:18.723847+0000","last_became_peered":"2023-10-19T14:20:18.723847+0000","last_unstale":"2023-10-19T14:20:21.383019+0000","last_undegraded":"2023-10-19T14:20:21.383019+0000","last_fullsized":"2023-10-19T14:20:21.383019+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:36:35.167691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.332503+0000","last_change":"2023-10-19T14:20:19.352214+0000","last_active":"2023-10-19T14:20:20.332503+0000","last_peered":"2023-10-19T14:20:20.332503+0000","last_clean":"2023-10-19T14:20:20.332503+0000","last_became_active":"2023-10-19T14:20:19.351789+0000","last_became_peered":"2023-10-19T14:20:19.351789+0000","last_unstale":"2023-10-19T14:20:20.332503+0000","last_undegraded":"2023-10-19T14:20:20.332503+0000","last_fullsized":"2023-10-19T14:20:20.332503+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:05:58.978059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":12,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.332482+0000","last_change":"2023-10-19T14:20:18.718760+0000","last_active":"2023-10-19T14:20:20.332482+0000","last_peered":"2023-10-19T14:20:20.332482+0000","last_clean":"2023-10-19T14:20:20.332482+0000","last_became_active":"2023-10-19T14:20:18.718459+0000","last_became_peered":"2023-10-19T14:20:18.718459+0000","last_unstale":"2023-10-19T14:20:20.332482+0000","last_undegraded":"2023-10-19T14:20:20.332482+0000","last_fullsized":"2023-10-19T14:20:20.332482+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:32:18.381107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.383815+0000","last_change":"2023-10-19T14:20:19.345692+0000","last_active":"2023-10-19T14:20:21.383815+0000","last_peered":"2023-10-19T14:20:21.383815+0000","last_clean":"2023-10-19T14:20:21.383815+0000","last_became_active":"2023-10-19T14:20:19.344861+0000","last_became_peered":"2023-10-19T14:20:19.344861+0000","last_unstale":"2023-10-19T14:20:21.383815+0000","last_undegraded":"2023-10-19T14:20:21.383815+0000","last_fullsized":"2023-10-19T14:20:21.383815+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:43:57.179927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":12,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.383799+0000","last_change":"2023-10-19T14:20:18.723898+0000","last_active":"2023-10-19T14:20:21.383799+0000","last_peered":"2023-10-19T14:20:21.383799+0000","last_clean":"2023-10-19T14:20:21.383799+0000","last_became_active":"2023-10-19T14:20:18.722703+0000","last_became_peered":"2023-10-19T14:20:18.722703+0000","last_unstale":"2023-10-19T14:20:21.383799+0000","last_undegraded":"2023-10-19T14:20:21.383799+0000","last_fullsized":"2023-10-19T14:20:21.383799+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:25:51.334964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.021015+0000","last_change":"2023-10-19T14:20:19.344708+0000","last_active":"2023-10-19T14:20:21.021015+0000","last_peered":"2023-10-19T14:20:21.021015+0000","last_clean":"2023-10-19T14:20:21.021015+0000","last_became_active":"2023-10-19T14:20:19.344380+0000","last_became_peered":"2023-10-19T14:20:19.344380+0000","last_unstale":"2023-10-19T14:20:21.021015+0000","last_undegraded":"2023-10-19T14:20:21.021015+0000","last_fullsized":"2023-10-19T14:20:21.021015+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:37:55.455379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":12,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.328440+0000","last_change":"2023-10-19T14:20:18.722453+0000","last_active":"2023-10-19T14:20:20.328440+0000","last_peered":"2023-10-19T14:20:20.328440+0000","last_clean":"2023-10-19T14:20:20.328440+0000","last_became_active":"2023-10-19T14:20:18.722162+0000","last_became_peered":"2023-10-19T14:20:18.722162+0000","last_unstale":"2023-10-19T14:20:20.328440+0000","last_undegraded":"2023-10-19T14:20:20.328440+0000","last_fullsized":"2023-10-19T14:20:20.328440+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:49:03.077978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.021092+0000","last_change":"2023-10-19T14:20:19.345421+0000","last_active":"2023-10-19T14:20:21.021092+0000","last_peered":"2023-10-19T14:20:21.021092+0000","last_clean":"2023-10-19T14:20:21.021092+0000","last_became_active":"2023-10-19T14:20:19.345091+0000","last_became_peered":"2023-10-19T14:20:19.345091+0000","last_unstale":"2023-10-19T14:20:21.021092+0000","last_undegraded":"2023-10-19T14:20:21.021092+0000","last_fullsized":"2023-10-19T14:20:21.021092+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:49:04.080016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":12,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.332384+0000","last_change":"2023-10-19T14:20:18.723906+0000","last_active":"2023-10-19T14:20:20.332384+0000","last_peered":"2023-10-19T14:20:20.332384+0000","last_clean":"2023-10-19T14:20:20.332384+0000","last_became_active":"2023-10-19T14:20:18.723585+0000","last_became_peered":"2023-10-19T14:20:18.723585+0000","last_unstale":"2023-10-19T14:20:20.332384+0000","last_undegraded":"2023-10-19T14:20:20.332384+0000","last_fullsized":"2023-10-19T14:20:20.332384+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:28:48.261352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.7","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.922350+0000","last_change":"2023-10-19T14:20:19.347562+0000","last_active":"2023-10-19T14:20:20.922350+0000","last_peered":"2023-10-19T14:20:20.922350+0000","last_clean":"2023-10-19T14:20:20.922350+0000","last_became_active":"2023-10-19T14:20:19.347286+0000","last_became_peered":"2023-10-19T14:20:19.347286+0000","last_unstale":"2023-10-19T14:20:20.922350+0000","last_undegraded":"2023-10-19T14:20:20.922350+0000","last_fullsized":"2023-10-19T14:20:20.922350+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:06:12.825567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":12,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:21.382416+0000","last_change":"2023-10-19T14:20:18.722143+0000","last_active":"2023-10-19T14:20:21.382416+0000","last_peered":"2023-10-19T14:20:21.382416+0000","last_clean":"2023-10-19T14:20:21.382416+0000","last_became_active":"2023-10-19T14:20:18.721853+0000","last_became_peered":"2023-10-19T14:20:18.721853+0000","last_unstale":"2023-10-19T14:20:21.382416+0000","last_undegraded":"2023-10-19T14:20:21.382416+0000","last_fullsized":"2023-10-19T14:20:21.382416+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:01:31.494525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.1d","version":"0'0","reported_seq":11,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.332568+0000","last_change":"2023-10-19T14:20:19.350783+0000","last_active":"2023-10-19T14:20:20.332568+0000","last_peered":"2023-10-19T14:20:20.332568+0000","last_clean":"2023-10-19T14:20:20.332568+0000","last_became_active":"2023-10-19T14:20:19.350441+0000","last_became_peered":"2023-10-19T14:20:19.350441+0000","last_unstale":"2023-10-19T14:20:20.332568+0000","last_undegraded":"2023-10-19T14:20:20.332568+0000","last_fullsized":"2023-10-19T14:20:20.332568+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:47:00.028101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1a","version":"222'1","reported_seq":13,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-19T14:20:20.328426+0000","last_change":"2023-10-19T14:20:18.723791+0000","last_active":"2023-10-19T14:20:20.328426+0000","last_peered":"2023-10-19T14:20:20.328426+0000","last_clean":"2023-10-19T14:20:20.328426+0000","last_became_active":"2023-10-19T14:20:18.723476+0000","last_became_peered":"2023-10-19T14:20:18.723476+0000","last_unstale":"2023-10-19T14:20:20.328426+0000","last_undegraded":"2023-10-19T14:20:20.328426+0000","last_fullsized":"2023-10-19T14:20:20.328426+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:07:17.723824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"211'3186","reported_seq":4083,"reported_epoch":222,"state":"active+clean","last_fresh":"2023-10-19T14:20:19.342159+0000","last_change":"2023-10-19T12:44:08.636613+0000","last_active":"2023-10-19T14:20:19.342159+0000","last_peered":"2023-10-19T14:20:19.342159+0000","last_clean":"2023-10-19T14:20:19.342159+0000","last_became_active":"2023-10-19T12:44:08.635697+0000","last_became_peered":"2023-10-19T12:44:08.635697+0000","last_unstale":"2023-10-19T14:20:19.342159+0000","last_undegraded":"2023-10-19T14:20:19.342159+0000","last_fullsized":"2023-10-19T14:20:19.342159+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_clean_scrub_stamp":"2023-10-19T12:44:07.399738+0000","objects_scrubbed":0,"log_size":3186,"log_dups_size":0,"ondisk_log_size":3186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:15:23.685485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2210,"num_read_kb":4624,"num_write":589,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":28,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":27,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2210,"num_read_kb":4624,"num_write":589,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3186,"ondisk_log_size":3186,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576002,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48400,"kb_used_data":3332,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323440,"statfs":{"total":96636764160,"available":96587202560,"internally_reserved":0,"allocated":3411968,"data_stored":2026754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":46135755},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:20:09 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.56999999999999995,"15min":0.58499999999999996},"min":{"1min":0.35799999999999998,"5min":0.34100000000000003,"15min":0.32200000000000001},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.86199999999999999},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.60199999999999998,"15min":0.60999999999999999},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.312},"max":{"1min":0.77600000000000002,"5min":0.81899999999999995,"15min":0.86599999999999999},"last":0.54800000000000004}]},{"osd":1,"last update":"Thu Oct 19 14:20:09 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.59799999999999998,"15min":0.59999999999999998},"min":{"1min":0.46500000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.76800000000000002,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.59499999999999997,"15min":0.59799999999999998},"min":{"1min":0.373,"5min":0.373,"15min":0.33500000000000002},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.78500000000000003}]},{"osd":2,"last update":"Thu Oct 19 14:20:09 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64900000000000002,"15min":0.64800000000000002},"min":{"1min":0.50900000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.85499999999999998},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.66600000000000004,"15min":0.66200000000000003},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.40100000000000002},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.92100000000000004}]},{"osd":3,"last update":"Thu Oct 19 14:20:09 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.63300000000000001,"15min":0.63700000000000001},"min":{"1min":0.51100000000000001,"5min":0.42099999999999999,"15min":0.36399999999999999},"max":{"1min":0.82699999999999996,"5min":0.84799999999999998,"15min":0.873},"last":0.95599999999999996},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.68799999999999994,"15min":0.68700000000000006},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":0.873,"5min":0.876,"15min":0.89600000000000002},"last":0.83299999999999996}]},{"osd":4,"last update":"Thu Oct 19 14:20:09 2023","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.50600000000000001,"15min":0.49299999999999999},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.62,"5min":0.78100000000000003,"15min":0.83799999999999997},"last":0.98399999999999999},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.51800000000000002,"15min":0.50600000000000001},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.71899999999999997,"5min":0.80200000000000005,"15min":0.81699999999999995},"last":1.0329999999999999}]},{"osd":5,"last update":"Thu Oct 19 14:20:09 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51400000000000001,"15min":0.50800000000000001},"min":{"1min":0.36799999999999999,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.65600000000000003,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.89100000000000001},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.55600000000000005,"15min":0.54900000000000004},"min":{"1min":0.377,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.70999999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":1.054}]},{"osd":6,"last update":"Thu Oct 19 14:20:09 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57699999999999996,"15min":0.57399999999999995},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.35999999999999999},"max":{"1min":0.73199999999999998,"5min":0.82999999999999996,"15min":0.85099999999999998},"last":1.0129999999999999},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.58099999999999996,"15min":0.56799999999999995},"min":{"1min":0.38800000000000001,"5min":0.35599999999999998,"15min":0.33500000000000002},"max":{"1min":0.78200000000000003,"5min":0.85199999999999998,"15min":0.85999999999999999},"last":1.1180000000000001}]}]},{"osd":6,"up_from":13,"seq":55834576002,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38092,"kb_used_data":2752,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333748,"statfs":{"total":96636764160,"available":96597757952,"internally_reserved":0,"allocated":2818048,"data_stored":1436898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:20:00 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.60799999999999998,"15min":0.61499999999999999},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.28799999999999998},"max":{"1min":0.81200000000000006,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.59399999999999997,"15min":0.61499999999999999},"min":{"1min":0.40699999999999997,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.42899999999999999}]},{"osd":1,"last update":"Thu Oct 19 14:20:00 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.621,"15min":0.621},"min":{"1min":0.52300000000000002,"5min":0.38200000000000001,"15min":0.35799999999999998},"max":{"1min":0.79500000000000004,"5min":0.80200000000000005,"15min":1.165},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.624,"15min":0.63500000000000001},"min":{"1min":0.503,"5min":0.439,"15min":0.36899999999999999},"max":{"1min":0.81599999999999995,"5min":0.83699999999999997,"15min":0.91300000000000003},"last":0.66600000000000004}]},{"osd":2,"last update":"Thu Oct 19 14:20:00 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.60099999999999998,"15min":0.60899999999999999},"min":{"1min":0.442,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.873},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.626,"15min":0.629},"min":{"1min":0.45000000000000001,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.93100000000000005},"last":0.55700000000000005}]},{"osd":3,"last update":"Thu Oct 19 14:20:00 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.65500000000000003,"15min":0.65800000000000003},"min":{"1min":0.53000000000000003,"5min":0.372,"15min":0.34300000000000003},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.65600000000000003,"15min":0.66100000000000003},"min":{"1min":0.48299999999999998,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.84399999999999997,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.67900000000000005}]},{"osd":4,"last update":"Thu Oct 19 14:20:00 2023","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.47999999999999998,"15min":0.48199999999999998},"min":{"1min":0.34499999999999997,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.79600000000000004},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.502,"15min":0.50800000000000001},"min":{"1min":0.35399999999999998,"5min":0.34300000000000003,"15min":0.30399999999999999},"max":{"1min":0.74199999999999999,"5min":0.82199999999999995,"15min":0.85699999999999998},"last":0.53500000000000003}]},{"osd":5,"last update":"Thu Oct 19 14:20:00 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.5,"15min":0.5},"min":{"1min":0.35599999999999998,"5min":0.32700000000000001,"15min":0.32500000000000001},"max":{"1min":0.747,"5min":1.115,"15min":1.115},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.55100000000000005,"15min":0.54300000000000004},"min":{"1min":0.40000000000000002,"5min":0.35799999999999998,"15min":0.32400000000000001},"max":{"1min":0.78700000000000003,"5min":0.86499999999999999,"15min":0.878},"last":0.65000000000000002}]},{"osd":7,"last update":"Thu Oct 19 14:20:00 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.55000000000000004,"15min":0.54600000000000004},"min":{"1min":0.39900000000000002,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.77600000000000002,"5min":0.86399999999999999,"15min":0.90100000000000002},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.55700000000000005,"15min":0.55200000000000005},"min":{"1min":0.39400000000000002,"5min":0.38200000000000001,"15min":0.36199999999999999},"max":{"1min":0.85099999999999998,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.621}]}]},{"osd":5,"up_from":13,"seq":55834576002,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47820,"kb_used_data":2744,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324020,"statfs":{"total":96636764160,"available":96587796480,"internally_reserved":0,"allocated":2809856,"data_stored":1434760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:19:46 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.59899999999999998,"15min":0.621},"min":{"1min":0.44900000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.79700000000000004,"5min":0.82599999999999996,"15min":0.86399999999999999},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.63200000000000001,"15min":0.64500000000000002},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.82499999999999996,"5min":0.85899999999999999,"15min":0.85999999999999999},"last":0.68000000000000005}]},{"osd":1,"last update":"Thu Oct 19 14:19:46 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.61799999999999999,"15min":0.63500000000000001},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.78800000000000003,"5min":0.83699999999999997,"15min":0.877},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.64100000000000001,"15min":0.64500000000000002},"min":{"1min":0.47999999999999998,"5min":0.44700000000000001,"15min":0.41199999999999998},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.88500000000000001},"last":0.81599999999999995}]},{"osd":2,"last update":"Thu Oct 19 14:19:46 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.61499999999999999,"15min":0.627},"min":{"1min":0.44800000000000001,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.81299999999999994,"5min":0.83799999999999997,"15min":0.85099999999999998},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.65200000000000002,"15min":0.66100000000000003},"min":{"1min":0.52000000000000002,"5min":0.49399999999999999,"15min":0.47099999999999997},"max":{"1min":0.79500000000000004,"5min":0.83299999999999996,"15min":0.90800000000000003},"last":0.63800000000000001}]},{"osd":3,"last update":"Thu Oct 19 14:19:46 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.63900000000000001,"15min":0.64700000000000002},"min":{"1min":0.51100000000000001,"5min":0.443,"15min":0.443},"max":{"1min":0.84399999999999997,"5min":0.86099999999999999,"15min":0.90100000000000002},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.66500000000000004,"15min":0.67400000000000004},"min":{"1min":0.53300000000000003,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.84899999999999998,"5min":0.876,"15min":0.90900000000000003},"last":0.69999999999999996}]},{"osd":4,"last update":"Thu Oct 19 14:19:46 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.497,"15min":0.48699999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.32500000000000001},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50900000000000001,"15min":0.50600000000000001},"min":{"1min":0.35499999999999998,"5min":0.318,"15min":0.28599999999999998},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.82899999999999996},"last":0.51500000000000001}]},{"osd":6,"last update":"Thu Oct 19 14:19:46 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51200000000000001,"15min":0.497},"min":{"1min":0.37,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.88800000000000001},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.54100000000000004,"15min":0.53900000000000003},"min":{"1min":0.39100000000000001,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.77900000000000003,"5min":0.78000000000000003,"15min":0.88100000000000001},"last":0.59599999999999997}]},{"osd":7,"last update":"Thu Oct 19 14:19:46 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.51500000000000001,"15min":0.52000000000000002},"min":{"1min":0.41799999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.749,"5min":0.749,"15min":0.82899999999999996},"last":0.625},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.58999999999999997,"15min":0.57499999999999996},"min":{"1min":0.47099999999999997,"5min":0.38400000000000001,"15min":0.34300000000000003},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.66100000000000003}]}]},{"osd":4,"up_from":13,"seq":55834576002,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47796,"kb_used_data":2728,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324044,"statfs":{"total":96636764160,"available":96587821056,"internally_reserved":0,"allocated":2793472,"data_stored":1428638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.55500000000000005,"15min":0.57599999999999996},"min":{"1min":0.35499999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.80800000000000005,"5min":0.81799999999999995,"15min":0.879},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.58499999999999996,"15min":0.59799999999999998},"min":{"1min":0.499,"5min":0.31,"15min":0.31},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.873},"last":0.379}]},{"osd":1,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.61499999999999999,"15min":0.64100000000000001},"min":{"1min":0.504,"5min":0.372,"15min":0.32600000000000001},"max":{"1min":0.80400000000000005,"5min":0.80700000000000005,"15min":0.92700000000000005},"last":0.441},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.623,"15min":0.64400000000000002},"min":{"1min":0.443,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":1.0940000000000001},"last":0.50700000000000001}]},{"osd":2,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.63400000000000001,"15min":0.65000000000000002},"min":{"1min":0.46400000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.64800000000000002,"15min":0.66900000000000004},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":1.008},"last":0.628}]},{"osd":3,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.61299999999999999,"15min":0.63400000000000001},"min":{"1min":0.47399999999999998,"5min":0.37,"15min":0.37},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.91900000000000004},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.66600000000000004,"15min":0.66700000000000004},"min":{"1min":0.54000000000000004,"5min":0.45400000000000001,"15min":0.437},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.60799999999999998}]},{"osd":5,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.48299999999999998,"15min":0.47099999999999997},"min":{"1min":0.34000000000000002,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.504,"15min":0.49199999999999999},"min":{"1min":0.34300000000000003,"5min":0.33700000000000002,"15min":0.29199999999999998},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.57699999999999996}]},{"osd":6,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.496,"15min":0.48599999999999999},"min":{"1min":0.34799999999999998,"5min":0.33900000000000002,"15min":0.28199999999999997},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.53700000000000003},"min":{"1min":0.39100000000000001,"5min":0.375,"15min":0.33300000000000002},"max":{"1min":0.81299999999999994,"5min":0.84799999999999998,"15min":1.2969999999999999},"last":0.67900000000000005}]},{"osd":7,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.55100000000000005,"15min":0.54100000000000004},"min":{"1min":0.36499999999999999,"5min":0.35699999999999998,"15min":0.32600000000000001},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60499999999999998,"15min":0.59199999999999997},"min":{"1min":0.47099999999999997,"5min":0.32100000000000001,"15min":0.318},"max":{"1min":0.80400000000000005,"5min":0.88600000000000001,"15min":1.032},"last":0.66700000000000004}]}]},{"osd":3,"up_from":13,"seq":55834576001,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56148,"kb_used_data":2748,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315692,"statfs":{"total":96636764160,"available":96579268608,"internally_reserved":0,"allocated":2813952,"data_stored":1434830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54655432},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:19:48 2023","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42299999999999999},"min":{"1min":0.316,"5min":0.253,"15min":0.253},"max":{"1min":0.60599999999999998,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.42599999999999999,"15min":0.432},"min":{"1min":0.29399999999999998,"5min":0.219,"15min":0.219},"max":{"1min":0.57599999999999996,"5min":0.752,"15min":0.752},"last":0.317}]},{"osd":1,"last update":"Thu Oct 19 14:19:48 2023","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.439,"15min":0.433},"min":{"1min":0.29599999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.56599999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.30099999999999999,"5min":0.29399999999999998,"15min":0.26400000000000001},"max":{"1min":0.56599999999999995,"5min":0.73099999999999998,"15min":0.81799999999999995},"last":0.51700000000000002}]},{"osd":2,"last update":"Thu Oct 19 14:19:48 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.47699999999999998,"15min":0.48599999999999999},"min":{"1min":0.39600000000000002,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.64300000000000002,"5min":0.64600000000000002,"15min":0.73099999999999998},"last":0.441},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.499,"15min":0.50800000000000001},"min":{"1min":0.33500000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.66800000000000004,"5min":0.67000000000000004,"15min":0.83699999999999997},"last":0.46700000000000003}]},{"osd":4,"last update":"Thu Oct 19 14:19:48 2023","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.71099999999999997,"15min":0.70899999999999996},"min":{"1min":0.45000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.96099999999999997,"5min":1.018,"15min":1.157},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.57999999999999996,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.93100000000000005,"5min":0.996,"15min":1.173},"last":0.64200000000000002}]},{"osd":5,"last update":"Thu Oct 19 14:19:48 2023","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.69499999999999995,"15min":0.69199999999999995},"min":{"1min":0.55000000000000004,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.93100000000000005,"5min":0.997,"15min":1.036},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.69799999999999995,"15min":0.69299999999999995},"min":{"1min":0.47399999999999998,"5min":0.42999999999999999,"15min":0.42599999999999999},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":1.105},"last":0.54200000000000004}]},{"osd":6,"last update":"Thu Oct 19 14:19:48 2023","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.68500000000000005,"15min":0.69499999999999995},"min":{"1min":0.51900000000000002,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.99099999999999999,"5min":1.018,"15min":1.0669999999999999},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.55800000000000005,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.88800000000000001,"5min":0.91500000000000004,"15min":1.079},"last":0.61099999999999999}]},{"osd":7,"last update":"Thu Oct 19 14:19:48 2023","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.70799999999999996,"15min":0.70499999999999996},"min":{"1min":0.51600000000000001,"5min":0.434,"15min":0.434},"max":{"1min":0.94899999999999995,"5min":0.96899999999999997,"15min":1.085},"last":0.627},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.73199999999999998,"15min":0.73599999999999999},"min":{"1min":0.57099999999999995,"5min":0.51900000000000002,"15min":0.51300000000000001},"max":{"1min":0.96499999999999997,"5min":0.97199999999999998,"15min":1.1739999999999999},"last":0.71499999999999997}]}]},{"osd":2,"up_from":13,"seq":55834576003,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64448,"kb_used_data":2740,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94307392,"statfs":{"total":96636764160,"available":96570769408,"internally_reserved":0,"allocated":2805760,"data_stored":1434794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":63175110},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.39900000000000002,"15min":0.40200000000000002},"min":{"1min":0.28199999999999997,"5min":0.218,"15min":0.218},"max":{"1min":0.623,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.44500000000000001,"15min":0.441},"min":{"1min":0.34100000000000003,"5min":0.249,"15min":0.223},"max":{"1min":0.628,"5min":0.71399999999999997,"15min":0.73599999999999999},"last":0.39600000000000002}]},{"osd":1,"last update":"Thu Oct 19 14:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.40500000000000003,"15min":0.41599999999999998},"min":{"1min":0.28999999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.48999999999999999,"5min":0.61699999999999999,"15min":0.65400000000000003},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.47999999999999998,"15min":0.47099999999999997},"min":{"1min":0.33000000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":1.742,"5min":1.742,"15min":1.742},"last":0.38100000000000001}]},{"osd":3,"last update":"Thu Oct 19 14:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.48199999999999998,"15min":0.48399999999999999},"min":{"1min":0.313,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.56999999999999995,"5min":0.69299999999999995,"15min":0.74399999999999999},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.499,"15min":0.51100000000000001},"min":{"1min":0.38200000000000001,"5min":0.32000000000000001,"15min":0.315},"max":{"1min":0.60599999999999998,"5min":0.67600000000000005,"15min":0.83599999999999997},"last":0.36399999999999999}]},{"osd":4,"last update":"Thu Oct 19 14:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.66900000000000004,"15min":0.67000000000000004},"min":{"1min":0.44900000000000001,"5min":0.39600000000000002,"15min":0.39500000000000002},"max":{"1min":0.88600000000000001,"5min":0.93899999999999995,"15min":0.998},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.67300000000000004,"15min":0.68200000000000005},"min":{"1min":0.51700000000000002,"5min":0.38900000000000001,"15min":0.35199999999999998},"max":{"1min":0.79500000000000004,"5min":0.97099999999999997,"15min":1.0289999999999999},"last":0.47299999999999998}]},{"osd":5,"last update":"Thu Oct 19 14:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.70499999999999996,"15min":0.70199999999999996},"min":{"1min":0.53300000000000003,"5min":0.45100000000000001,"15min":0.40600000000000003},"max":{"1min":1.4330000000000001,"5min":1.4330000000000001,"15min":1.4330000000000001},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.68400000000000005,"15min":0.68899999999999995},"min":{"1min":0.48199999999999998,"5min":0.45100000000000001,"15min":0.39300000000000002},"max":{"1min":1.595,"5min":1.595,"15min":1.595},"last":0.62}]},{"osd":6,"last update":"Thu Oct 19 14:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.70299999999999996,"15min":0.71199999999999997},"min":{"1min":0.46500000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.85999999999999999,"5min":0.92800000000000005,"15min":1.0189999999999999},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.72899999999999998,"15min":0.72599999999999998},"min":{"1min":0.47899999999999998,"5min":0.433,"15min":0.40600000000000003},"max":{"1min":1.4930000000000001,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":0.48899999999999999}]},{"osd":7,"last update":"Thu Oct 19 14:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.71999999999999997,"15min":0.72899999999999998},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.41799999999999998},"max":{"1min":0.97999999999999998,"5min":1.0129999999999999,"15min":1.0289999999999999},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.73899999999999999,"15min":0.73499999999999999},"min":{"1min":0.503,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.53,"5min":1.53,"15min":1.53},"last":0.57999999999999996}]}]},{"osd":1,"up_from":13,"seq":55834576002,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47804,"kb_used_data":2736,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324036,"statfs":{"total":96636764160,"available":96587812864,"internally_reserved":0,"allocated":2801664,"data_stored":1434704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:19:35 2023","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.40300000000000002,"15min":0.40699999999999997},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.23100000000000001},"max":{"1min":0.48799999999999999,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.37},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.45400000000000001,"15min":0.45700000000000002},"min":{"1min":0.28299999999999997,"5min":0.27500000000000002,"15min":0.251},"max":{"1min":0.61399999999999999,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.46899999999999997}]},{"osd":2,"last update":"Thu Oct 19 14:19:35 2023","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.44500000000000001,"15min":0.44600000000000001},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.58599999999999997,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.495,"15min":0.49199999999999999},"min":{"1min":0.34399999999999997,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.64400000000000002,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.42399999999999999}]},{"osd":3,"last update":"Thu Oct 19 14:19:35 2023","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.51600000000000001,"15min":0.50800000000000001},"min":{"1min":0.38400000000000001,"5min":0.36299999999999999,"15min":0.29499999999999998},"max":{"1min":0.64400000000000002,"5min":0.76000000000000001,"15min":0.78900000000000003},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.44600000000000001,"15min":0.44900000000000001},"min":{"1min":0.32000000000000001,"5min":0.315,"15min":0.26100000000000001},"max":{"1min":0.65200000000000002,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.38400000000000001}]},{"osd":4,"last update":"Thu Oct 19 14:19:35 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.67000000000000004,"15min":0.67100000000000004},"min":{"1min":0.39900000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.84399999999999997,"5min":1.042,"15min":1.042},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"min":{"1min":0.45100000000000001,"5min":0.41299999999999998,"15min":0.32300000000000001},"max":{"1min":0.88800000000000001,"5min":0.93500000000000005,"15min":1.1659999999999999},"last":0.52100000000000002}]},{"osd":5,"last update":"Thu Oct 19 14:19:35 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.69199999999999995,"15min":0.68700000000000006},"min":{"1min":0.45900000000000002,"5min":0.42799999999999999,"15min":0.42499999999999999},"max":{"1min":0.83099999999999996,"5min":1.0589999999999999,"15min":1.3500000000000001},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.71199999999999997,"15min":0.70799999999999996},"min":{"1min":0.44,"5min":0.44,"15min":0.40500000000000003},"max":{"1min":0.78600000000000003,"5min":1.0149999999999999,"15min":1.121},"last":0.58999999999999997}]},{"osd":6,"last update":"Thu Oct 19 14:19:35 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.67300000000000004,"15min":0.68300000000000005},"min":{"1min":0.42499999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.77200000000000002,"5min":1.0129999999999999,"15min":1.2310000000000001},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.71999999999999997,"15min":0.71499999999999997},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.86399999999999999,"5min":1.107,"15min":1.274},"last":0.60299999999999998}]},{"osd":7,"last update":"Thu Oct 19 14:19:35 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.72799999999999998,"15min":0.72399999999999998},"min":{"1min":0.46999999999999997,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.78600000000000003,"5min":1.075,"15min":1.075},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.47199999999999998,"5min":0.46100000000000002,"15min":0.44400000000000001},"max":{"1min":0.79600000000000004,"5min":1.0169999999999999,"15min":1.117},"last":0.53100000000000003}]}]},{"osd":0,"up_from":13,"seq":55834576003,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60872,"kb_used_data":3316,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310968,"statfs":{"total":96636764160,"available":96574431232,"internally_reserved":0,"allocated":3395584,"data_stored":2024543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.40699999999999997,"15min":0.41299999999999998},"min":{"1min":0.29599999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.56299999999999994,"5min":0.61499999999999999,"15min":0.624},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.441,"15min":0.46100000000000002},"min":{"1min":0.26800000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.56799999999999995,"5min":0.70899999999999996,"15min":0.81100000000000005},"last":0.42399999999999999}]},{"osd":2,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.496,"15min":0.48399999999999999},"min":{"1min":0.39300000000000002,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.71499999999999997,"5min":0.754,"15min":0.754},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.44800000000000001},"min":{"1min":0.32700000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.60199999999999998,"5min":0.63100000000000001,"15min":0.69099999999999995},"last":0.45100000000000001}]},{"osd":3,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.432,"15min":0.44},"min":{"1min":0.26900000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.58799999999999997,"5min":0.61099999999999999,"15min":0.66800000000000004},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.501,"5min":0.5,"15min":0.51200000000000001},"min":{"1min":0.35199999999999998,"5min":0.254,"15min":0.254},"max":{"1min":0.78100000000000003,"5min":0.79800000000000004,"15min":0.84599999999999997},"last":0.54100000000000004}]},{"osd":4,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.64800000000000002,"15min":0.67700000000000005},"min":{"1min":0.48799999999999999,"5min":0.44600000000000001,"15min":0.44},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":1.1890000000000001},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.65800000000000003,"15min":0.68200000000000005},"min":{"1min":0.47499999999999998,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.751,"5min":0.95399999999999996,"15min":1.2150000000000001},"last":0.56399999999999995}]},{"osd":5,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.68999999999999995,"15min":0.72199999999999998},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":1.1579999999999999},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.69499999999999995,"15min":0.71999999999999997},"min":{"1min":0.57799999999999996,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.85399999999999998,"5min":0.98299999999999998,"15min":1.2310000000000001},"last":0.64600000000000002}]},{"osd":6,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.68700000000000006,"15min":0.70599999999999996},"min":{"1min":0.47799999999999998,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":1.1739999999999999},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.70399999999999996},"min":{"1min":0.54100000000000004,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.88900000000000001,"5min":0.97799999999999998,"15min":1.0189999999999999},"last":0.60499999999999998}]},{"osd":7,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.68600000000000005,"15min":0.71599999999999997},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":1.1180000000000001},"last":0.628},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.68200000000000005,"15min":0.70399999999999996},"min":{"1min":0.53600000000000003,"5min":0.46899999999999997,"15min":0.45100000000000001},"max":{"1min":0.90600000000000003,"5min":0.93799999999999994,"15min":1.0349999999999999},"last":0.71399999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-19T14:20:23.082 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-19T14:20:23.082 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-19T14:20:23.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.226+0000 7fd06f0c9640 1 -- 172.21.15.57:0/3761264217 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd0680afa20 msgr2=0x7fd0680705a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:23.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.226+0000 7fd06f0c9640 1 --2- 172.21.15.57:0/3761264217 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd0680afa20 0x7fd0680705a0 secure :-1 s=READY pgs=2364 cs=0 l=1 rev1=1 crypto rx=0x7fd0680cc7e0 tx=0x7fd05402d0e0 comp rx=0 tx=0).stop 2023-10-19T14:20:23.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.226+0000 7fd06f0c9640 1 -- 172.21.15.57:0/3761264217 shutdown_connections 2023-10-19T14:20:23.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.226+0000 7fd06f0c9640 1 --2- 172.21.15.57:0/3761264217 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd0680afa20 0x7fd0680705a0 unknown :-1 s=CLOSED pgs=2364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:23.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.226+0000 7fd06f0c9640 1 --2- 172.21.15.57:0/3761264217 >> v2:172.21.15.179:3300/0 conn(0x7fd0680cbd20 0x7fd068158300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:23.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.226+0000 7fd06f0c9640 1 -- 172.21.15.57:0/3761264217 >> 172.21.15.57:0/3761264217 conn(0x7fd068069930 msgr2=0x7fd068069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:23.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.226+0000 7fd06f0c9640 1 -- 172.21.15.57:0/3761264217 shutdown_connections 2023-10-19T14:20:23.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.226+0000 7fd06f0c9640 1 -- 172.21.15.57:0/3761264217 wait complete. 2023-10-19T14:20:23.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.227+0000 7fd06f0c9640 1 Processor -- start 2023-10-19T14:20:23.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.227+0000 7fd06f0c9640 1 -- start start 2023-10-19T14:20:23.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.228+0000 7fd06f0c9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd0680afa20 0x7fd068156530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:23.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.228+0000 7fd06f0c9640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd0680b0530 0x7fd068148290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:23.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.228+0000 7fd06f0c9640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd0680cbd20 0x7fd06814abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:23.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.228+0000 7fd06f0c9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0681595c0 con 0x7fd0680afa20 2023-10-19T14:20:23.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.228+0000 7fd06f0c9640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0681e2f30 con 0x7fd0680cbd20 2023-10-19T14:20:23.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.228+0000 7fd06f0c9640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd06814d6a0 con 0x7fd0680b0530 2023-10-19T14:20:23.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.228+0000 7fd06ce65640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd0680afa20 0x7fd068156530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:23.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.228+0000 7fd06ce65640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd0680afa20 0x7fd068156530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42136/0 (socket says 172.21.15.57:42136) 2023-10-19T14:20:23.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.228+0000 7fd06ce65640 1 -- 172.21.15.57:0/164985180 learned_addr learned my addr 172.21.15.57:0/164985180 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:23.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.229+0000 7fd06d666640 1 --2- 172.21.15.57:0/164985180 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd0680cbd20 0x7fd06814abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:23.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.229+0000 7fd067fff640 1 --2- 172.21.15.57:0/164985180 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd0680b0530 0x7fd068148290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:23.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.229+0000 7fd06ce65640 1 -- 172.21.15.57:0/164985180 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd0680b0530 msgr2=0x7fd068148290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:23.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.229+0000 7fd06ce65640 1 --2- 172.21.15.57:0/164985180 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd0680b0530 0x7fd068148290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:23.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.229+0000 7fd06ce65640 1 -- 172.21.15.57:0/164985180 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd0680cbd20 msgr2=0x7fd06814abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:23.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.229+0000 7fd06ce65640 1 --2- 172.21.15.57:0/164985180 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd0680cbd20 0x7fd06814abd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:23.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.229+0000 7fd06ce65640 1 -- 172.21.15.57:0/164985180 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd05400a670 con 0x7fd0680afa20 2023-10-19T14:20:23.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.229+0000 7fd06d666640 1 --2- 172.21.15.57:0/164985180 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd0680cbd20 0x7fd06814abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:20:23.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.229+0000 7fd06ce65640 1 --2- 172.21.15.57:0/164985180 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd0680afa20 0x7fd068156530 secure :-1 s=READY pgs=2365 cs=0 l=1 rev1=1 crypto rx=0x7fd05c00b440 tx=0x7fd05c00b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:23.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.229+0000 7fd0657fa640 1 -- 172.21.15.57:0/164985180 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd05c011890 con 0x7fd0680afa20 2023-10-19T14:20:23.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.229+0000 7fd0657fa640 1 -- 172.21.15.57:0/164985180 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fd05c011ae0 con 0x7fd0680afa20 2023-10-19T14:20:23.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.229+0000 7fd0657fa640 1 -- 172.21.15.57:0/164985180 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd05c0103e0 con 0x7fd0680afa20 2023-10-19T14:20:23.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.229+0000 7fd06f0c9640 1 -- 172.21.15.57:0/164985180 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0681589e0 con 0x7fd0680afa20 2023-10-19T14:20:23.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.229+0000 7fd06f0c9640 1 -- 172.21.15.57:0/164985180 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0681591e0 con 0x7fd0680afa20 2023-10-19T14:20:23.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.230+0000 7fd06f0c9640 1 -- 172.21.15.57:0/164985180 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0680690e0 con 0x7fd0680afa20 2023-10-19T14:20:23.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.231+0000 7fd0657fa640 1 -- 172.21.15.57:0/164985180 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd05c010540 con 0x7fd0680afa20 2023-10-19T14:20:23.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.231+0000 7fd0657fa640 1 --2- 172.21.15.57:0/164985180 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fd04c06af60 0x7fd04c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:23.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.231+0000 7fd0657fa640 1 -- 172.21.15.57:0/164985180 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fd05c089d10 con 0x7fd0680afa20 2023-10-19T14:20:23.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.231+0000 7fd067fff640 1 --2- 172.21.15.57:0/164985180 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fd04c06af60 0x7fd04c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:23.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.232+0000 7fd067fff640 1 --2- 172.21.15.57:0/164985180 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fd04c06af60 0x7fd04c06d420 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd054036000 tx=0x7fd054004730 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:23.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.235+0000 7fd0657fa640 1 -- 172.21.15.57:0/164985180 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd05c054b50 con 0x7fd0680afa20 2023-10-19T14:20:23.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.480+0000 7fd06f0c9640 1 -- 172.21.15.57:0/164985180 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fd0681589b0 con 0x7fd0680afa20 2023-10-19T14:20:23.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.482+0000 7fd0657fa640 1 -- 172.21.15.57:0/164985180 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v60) v1 ==== 190+0+147 (secure 0 0 0) 0x7fd05c0150e0 con 0x7fd0680afa20 2023-10-19T14:20:23.485 INFO:teuthology.orchestra.run.smithi057.stdout:[client.0] 2023-10-19T14:20:23.485 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQBkJDFlg9yuExAATxiWDfS+p07G1F0pORbJEg== 2023-10-19T14:20:23.486 INFO:teuthology.orchestra.run.smithi057.stdout: caps mds = "allow" 2023-10-19T14:20:23.486 INFO:teuthology.orchestra.run.smithi057.stdout: caps mon = "allow r" 2023-10-19T14:20:23.486 INFO:teuthology.orchestra.run.smithi057.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-19T14:20:23.487 INFO:teuthology.orchestra.run.smithi057.stderr:updated caps for client.0 2023-10-19T14:20:23.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.484+0000 7fd06f0c9640 1 -- 172.21.15.57:0/164985180 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fd04c06af60 msgr2=0x7fd04c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:23.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.484+0000 7fd06f0c9640 1 --2- 172.21.15.57:0/164985180 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fd04c06af60 0x7fd04c06d420 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd054036000 tx=0x7fd054004730 comp rx=0 tx=0).stop 2023-10-19T14:20:23.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.484+0000 7fd06f0c9640 1 -- 172.21.15.57:0/164985180 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd0680afa20 msgr2=0x7fd068156530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:23.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.484+0000 7fd06f0c9640 1 --2- 172.21.15.57:0/164985180 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd0680afa20 0x7fd068156530 secure :-1 s=READY pgs=2365 cs=0 l=1 rev1=1 crypto rx=0x7fd05c00b440 tx=0x7fd05c00b900 comp rx=0 tx=0).stop 2023-10-19T14:20:23.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.484+0000 7fd06f0c9640 1 -- 172.21.15.57:0/164985180 shutdown_connections 2023-10-19T14:20:23.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.484+0000 7fd06f0c9640 1 --2- 172.21.15.57:0/164985180 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fd04c06af60 0x7fd04c06d420 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:23.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.484+0000 7fd06f0c9640 1 --2- 172.21.15.57:0/164985180 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd0680cbd20 0x7fd06814abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:23.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.484+0000 7fd06f0c9640 1 --2- 172.21.15.57:0/164985180 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd0680b0530 0x7fd068148290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:23.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.484+0000 7fd06f0c9640 1 --2- 172.21.15.57:0/164985180 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd0680afa20 0x7fd068156530 unknown :-1 s=CLOSED pgs=2365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:23.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.484+0000 7fd06f0c9640 1 -- 172.21.15.57:0/164985180 >> 172.21.15.57:0/164985180 conn(0x7fd068069930 msgr2=0x7fd068069d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:23.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.484+0000 7fd06f0c9640 1 -- 172.21.15.57:0/164985180 shutdown_connections 2023-10-19T14:20:23.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.484+0000 7fd06f0c9640 1 -- 172.21.15.57:0/164985180 wait complete. 2023-10-19T14:20:23.498 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T14:20:23.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.605+0000 7f7a86f76640 1 -- 172.21.15.57:0/2131047316 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7a800afa20 msgr2=0x7f7a800705a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:23.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.605+0000 7f7a86f76640 1 --2- 172.21.15.57:0/2131047316 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7a800afa20 0x7f7a800705a0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f7a6c009830 tx=0x7f7a6c02bd60 comp rx=0 tx=0).stop 2023-10-19T14:20:23.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.605+0000 7f7a86f76640 1 -- 172.21.15.57:0/2131047316 shutdown_connections 2023-10-19T14:20:23.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.605+0000 7f7a86f76640 1 --2- 172.21.15.57:0/2131047316 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7a800afa20 0x7f7a800705a0 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:23.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.605+0000 7f7a86f76640 1 --2- 172.21.15.57:0/2131047316 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7a800cbd20 0x7f7a80158300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:23.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.605+0000 7f7a86f76640 1 -- 172.21.15.57:0/2131047316 >> 172.21.15.57:0/2131047316 conn(0x7f7a80069930 msgr2=0x7f7a80069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:23.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.605+0000 7f7a86f76640 1 -- 172.21.15.57:0/2131047316 shutdown_connections 2023-10-19T14:20:23.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.605+0000 7f7a86f76640 1 -- 172.21.15.57:0/2131047316 wait complete. 2023-10-19T14:20:23.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.606+0000 7f7a86f76640 1 Processor -- start 2023-10-19T14:20:23.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.606+0000 7f7a86f76640 1 -- start start 2023-10-19T14:20:23.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.607+0000 7f7a86f76640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7a800afa20 0x7f7a800adab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:23.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.607+0000 7f7a86f76640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7a800b0530 0x7f7a8013eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:23.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.607+0000 7f7a86f76640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7a800cbd20 0x7f7a8013be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:23.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.607+0000 7f7a86f76640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a8013faa0 con 0x7f7a800cbd20 2023-10-19T14:20:23.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.607+0000 7f7a86f76640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a8013f4e0 con 0x7f7a800afa20 2023-10-19T14:20:23.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.607+0000 7f7a86f76640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a8013f7c0 con 0x7f7a800b0530 2023-10-19T14:20:23.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.607+0000 7f7a85513640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7a800cbd20 0x7f7a8013be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:23.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.607+0000 7f7a85513640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7a800cbd20 0x7f7a8013be00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42142/0 (socket says 172.21.15.57:42142) 2023-10-19T14:20:23.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.607+0000 7f7a85513640 1 -- 172.21.15.57:0/516612147 learned_addr learned my addr 172.21.15.57:0/516612147 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:23.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.607+0000 7f7a7ffff640 1 --2- 172.21.15.57:0/516612147 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7a800b0530 0x7f7a8013eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:23.618 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.607+0000 7f7a84d12640 1 --2- 172.21.15.57:0/516612147 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7a800afa20 0x7f7a800adab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:23.618 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.607+0000 7f7a85513640 1 -- 172.21.15.57:0/516612147 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7a800b0530 msgr2=0x7f7a8013eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:23.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.607+0000 7f7a85513640 1 --2- 172.21.15.57:0/516612147 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7a800b0530 0x7f7a8013eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:23.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.607+0000 7f7a85513640 1 -- 172.21.15.57:0/516612147 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7a800afa20 msgr2=0x7f7a800adab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:23.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.607+0000 7f7a85513640 1 --2- 172.21.15.57:0/516612147 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7a800afa20 0x7f7a800adab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:23.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.607+0000 7f7a85513640 1 -- 172.21.15.57:0/516612147 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a7400a5a0 con 0x7f7a800cbd20 2023-10-19T14:20:23.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.608+0000 7f7a85513640 1 --2- 172.21.15.57:0/516612147 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7a800cbd20 0x7f7a8013be00 secure :-1 s=READY pgs=2366 cs=0 l=1 rev1=1 crypto rx=0x7f7a7800cbc0 tx=0x7f7a7800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:23.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.608+0000 7f7a7d7fa640 1 -- 172.21.15.57:0/516612147 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a7800ad40 con 0x7f7a800cbd20 2023-10-19T14:20:23.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.608+0000 7f7a7d7fa640 1 -- 172.21.15.57:0/516612147 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f7a78012420 con 0x7f7a800cbd20 2023-10-19T14:20:23.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.608+0000 7f7a86f76640 1 -- 172.21.15.57:0/516612147 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a6c009490 con 0x7f7a800cbd20 2023-10-19T14:20:23.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.608+0000 7f7a7d7fa640 1 -- 172.21.15.57:0/516612147 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a78012850 con 0x7f7a800cbd20 2023-10-19T14:20:23.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.608+0000 7f7a86f76640 1 -- 172.21.15.57:0/516612147 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a80070cf0 con 0x7f7a800cbd20 2023-10-19T14:20:23.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.609+0000 7f7a86f76640 1 -- 172.21.15.57:0/516612147 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a8013ca70 con 0x7f7a800cbd20 2023-10-19T14:20:23.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.609+0000 7f7a7d7fa640 1 -- 172.21.15.57:0/516612147 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f7a7801a3b0 con 0x7f7a800cbd20 2023-10-19T14:20:23.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.610+0000 7f7a7d7fa640 1 --2- 172.21.15.57:0/516612147 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f7a6406af60 0x7f7a6406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:23.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.610+0000 7f7a7d7fa640 1 -- 172.21.15.57:0/516612147 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f7a7808a1a0 con 0x7f7a800cbd20 2023-10-19T14:20:23.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.610+0000 7f7a84d12640 1 --2- 172.21.15.57:0/516612147 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f7a6406af60 0x7f7a6406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:23.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.610+0000 7f7a84d12640 1 --2- 172.21.15.57:0/516612147 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f7a6406af60 0x7f7a6406d420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7a7400acc0 tx=0x7f7a74005f90 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:23.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.614+0000 7f7a7d7fa640 1 -- 172.21.15.57:0/516612147 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7a8013ca70 con 0x7f7a800cbd20 2023-10-19T14:20:23.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.826+0000 7f7a86f76640 1 -- 172.21.15.57:0/516612147 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7a8013ca70 con 0x7f7a800cbd20 2023-10-19T14:20:23.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.826+0000 7f7a7d7fa640 1 -- 172.21.15.57:0/516612147 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+4928 (secure 0 0 0) 0x7f7a8013ca70 con 0x7f7a800cbd20 2023-10-19T14:20:23.830 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:20:23.830 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":90,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9248,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/1249807745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":1249807745},{"type":"v1","addr":"172.21.15.179:6836","nonce":1249807745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87},{"gid":10513,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/1406825818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":1406825818},{"type":"v1","addr":"172.21.15.179:6837","nonce":1406825818}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":86},{"gid":11715,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/2467807803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2467807803},{"type":"v1","addr":"172.21.15.57:6834","nonce":2467807803}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T14:20:19.077396+0000","modified":"2023-10-19T14:20:21.072483+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11718},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11718":{"gid":11718,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.57:6835/1529368280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1529368280},{"type":"v1","addr":"172.21.15.57:6835","nonce":1529368280}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13}]} 2023-10-19T14:20:23.831 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 90 2023-10-19T14:20:23.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.828+0000 7f7a86f76640 1 -- 172.21.15.57:0/516612147 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f7a6406af60 msgr2=0x7f7a6406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:23.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.828+0000 7f7a86f76640 1 --2- 172.21.15.57:0/516612147 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f7a6406af60 0x7f7a6406d420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7a7400acc0 tx=0x7f7a74005f90 comp rx=0 tx=0).stop 2023-10-19T14:20:23.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.828+0000 7f7a86f76640 1 -- 172.21.15.57:0/516612147 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7a800cbd20 msgr2=0x7f7a8013be00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:23.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.828+0000 7f7a86f76640 1 --2- 172.21.15.57:0/516612147 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7a800cbd20 0x7f7a8013be00 secure :-1 s=READY pgs=2366 cs=0 l=1 rev1=1 crypto rx=0x7f7a7800cbc0 tx=0x7f7a7800a430 comp rx=0 tx=0).stop 2023-10-19T14:20:23.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.828+0000 7f7a86f76640 1 -- 172.21.15.57:0/516612147 shutdown_connections 2023-10-19T14:20:23.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.828+0000 7f7a86f76640 1 --2- 172.21.15.57:0/516612147 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f7a6406af60 0x7f7a6406d420 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:23.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.828+0000 7f7a86f76640 1 --2- 172.21.15.57:0/516612147 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7a800cbd20 0x7f7a8013be00 unknown :-1 s=CLOSED pgs=2366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:23.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.828+0000 7f7a86f76640 1 --2- 172.21.15.57:0/516612147 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7a800b0530 0x7f7a8013eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:23.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.828+0000 7f7a86f76640 1 --2- 172.21.15.57:0/516612147 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7a800afa20 0x7f7a800adab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:23.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.828+0000 7f7a86f76640 1 -- 172.21.15.57:0/516612147 >> 172.21.15.57:0/516612147 conn(0x7f7a80069930 msgr2=0x7f7a800cb310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:23.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.828+0000 7f7a86f76640 1 -- 172.21.15.57:0/516612147 shutdown_connections 2023-10-19T14:20:23.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:23.828+0000 7f7a86f76640 1 -- 172.21.15.57:0/516612147 wait complete. 2023-10-19T14:20:24.244 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 90, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-19T14:20:19.077396+0000', 'modified': '2023-10-19T14:20:21.072483+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 11718}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_11718': {'gid': 11718, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.57:6835/1529368280', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6833', 'nonce': 1529368280}, {'type': 'v1', 'addr': '172.21.15.57:6835', 'nonce': 1529368280}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [28], 'metadata_pool': 27, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-19T14:20:24.245 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-19T14:20:24.245 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2023-10-19T14:20:24.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.357+0000 7f574882b640 1 -- 172.21.15.57:0/253475967 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f574406c0a0 msgr2=0x7f5744070920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:24.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.357+0000 7f574882b640 1 --2- 172.21.15.57:0/253475967 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f574406c0a0 0x7f5744070920 secure :-1 s=READY pgs=2367 cs=0 l=1 rev1=1 crypto rx=0x7f574406a780 tx=0x7f573402bc80 comp rx=0 tx=0).stop 2023-10-19T14:20:24.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.357+0000 7f574882b640 1 -- 172.21.15.57:0/253475967 shutdown_connections 2023-10-19T14:20:24.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.357+0000 7f574882b640 1 --2- 172.21.15.57:0/253475967 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f574406c0a0 0x7f5744070920 unknown :-1 s=CLOSED pgs=2367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:24.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.357+0000 7f574882b640 1 --2- 172.21.15.57:0/253475967 >> v2:172.21.15.179:3300/0 conn(0x7f5744141690 0x7f57440703e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:24.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.357+0000 7f574882b640 1 --2- 172.21.15.57:0/253475967 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f574413e520 0x7f574406bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:24.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.357+0000 7f574882b640 1 -- 172.21.15.57:0/253475967 >> 172.21.15.57:0/253475967 conn(0x7f5744069930 msgr2=0x7f5744069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:24.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.357+0000 7f574882b640 1 -- 172.21.15.57:0/253475967 shutdown_connections 2023-10-19T14:20:24.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.357+0000 7f574882b640 1 -- 172.21.15.57:0/253475967 wait complete. 2023-10-19T14:20:24.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.358+0000 7f574882b640 1 Processor -- start 2023-10-19T14:20:24.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.358+0000 7f574882b640 1 -- start start 2023-10-19T14:20:24.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.359+0000 7f574882b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f574406c0a0 0x7f5744157b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:24.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.359+0000 7f574882b640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f574413e520 0x7f574413aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:24.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.359+0000 7f574882b640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5744141690 0x7f574413fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:24.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.359+0000 7f574882b640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f574413ab60 con 0x7f574406c0a0 2023-10-19T14:20:24.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.359+0000 7f574882b640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5744140640 con 0x7f574413e520 2023-10-19T14:20:24.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.359+0000 7f574882b640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5744140920 con 0x7f5744141690 2023-10-19T14:20:24.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.359+0000 7f574259c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f574406c0a0 0x7f5744157b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:24.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.359+0000 7f574259c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f574406c0a0 0x7f5744157b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42156/0 (socket says 172.21.15.57:42156) 2023-10-19T14:20:24.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.359+0000 7f574259c640 1 -- 172.21.15.57:0/4108747631 learned_addr learned my addr 172.21.15.57:0/4108747631 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:24.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.359+0000 7f5742d9d640 1 --2- 172.21.15.57:0/4108747631 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5744141690 0x7f574413fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:24.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.359+0000 7f5741d9b640 1 --2- 172.21.15.57:0/4108747631 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f574413e520 0x7f574413aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:24.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.359+0000 7f574259c640 1 -- 172.21.15.57:0/4108747631 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5744141690 msgr2=0x7f574413fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:24.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.359+0000 7f574259c640 1 --2- 172.21.15.57:0/4108747631 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5744141690 0x7f574413fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:24.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.359+0000 7f574259c640 1 -- 172.21.15.57:0/4108747631 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f574413e520 msgr2=0x7f574413aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:24.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.359+0000 7f574259c640 1 --2- 172.21.15.57:0/4108747631 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f574413e520 0x7f574413aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:24.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.359+0000 7f574259c640 1 -- 172.21.15.57:0/4108747631 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5734009560 con 0x7f574406c0a0 2023-10-19T14:20:24.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.359+0000 7f5742d9d640 1 --2- 172.21.15.57:0/4108747631 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5744141690 0x7f574413fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:20:24.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.359+0000 7f5741d9b640 1 --2- 172.21.15.57:0/4108747631 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f574413e520 0x7f574413aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:20:24.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.360+0000 7f574259c640 1 --2- 172.21.15.57:0/4108747631 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f574406c0a0 0x7f5744157b10 secure :-1 s=READY pgs=2368 cs=0 l=1 rev1=1 crypto rx=0x7f573c00ba50 tx=0x7f573c00bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:24.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.360+0000 7f5732ffd640 1 -- 172.21.15.57:0/4108747631 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f573c00cbe0 con 0x7f574406c0a0 2023-10-19T14:20:24.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.360+0000 7f5732ffd640 1 -- 172.21.15.57:0/4108747631 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f573c00ce50 con 0x7f574406c0a0 2023-10-19T14:20:24.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.360+0000 7f574882b640 1 -- 172.21.15.57:0/4108747631 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f574413a510 con 0x7f574406c0a0 2023-10-19T14:20:24.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.360+0000 7f5732ffd640 1 -- 172.21.15.57:0/4108747631 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f573c014680 con 0x7f574406c0a0 2023-10-19T14:20:24.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.360+0000 7f574882b640 1 -- 172.21.15.57:0/4108747631 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f574415e5c0 con 0x7f574406c0a0 2023-10-19T14:20:24.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.362+0000 7f5732ffd640 1 -- 172.21.15.57:0/4108747631 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f573c01d3b0 con 0x7f574406c0a0 2023-10-19T14:20:24.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.362+0000 7f5732ffd640 1 --2- 172.21.15.57:0/4108747631 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f572406af90 0x7f572406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:24.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.362+0000 7f5732ffd640 1 -- 172.21.15.57:0/4108747631 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f573c089b80 con 0x7f574406c0a0 2023-10-19T14:20:24.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.363+0000 7f5741d9b640 1 --2- 172.21.15.57:0/4108747631 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f572406af90 0x7f572406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:24.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.363+0000 7f574b3bd740 1 Processor -- start 2023-10-19T14:20:24.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.363+0000 7f574b3bd740 1 -- start start 2023-10-19T14:20:24.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.363+0000 7f5741d9b640 1 --2- 172.21.15.57:0/4108747631 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f572406af90 0x7f572406d450 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f57440afa90 tx=0x7f5738008040 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:24.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.364+0000 7f574b3bd740 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55da63c2ab40 0x55da63bfdae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:24.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.364+0000 7f574b3bd740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55da63c2af20 0x55da63c2d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:24.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.364+0000 7f574b3bd740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55da63bfe020 0x55da63c2de10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:24.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.364+0000 7f574b3bd740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da63b0f730 con 0x55da63c2af20 2023-10-19T14:20:24.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.364+0000 7f574b3bd740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da63b70780 con 0x55da63bfe020 2023-10-19T14:20:24.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.364+0000 7f574b3bd740 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x55da63b6de70 con 0x55da63c2ab40 2023-10-19T14:20:24.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.364+0000 7f5741d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55da63c2af20 0x55da63c2d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:24.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.364+0000 7f5741d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55da63c2af20 0x55da63c2d8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42160/0 (socket says 172.21.15.57:42160) 2023-10-19T14:20:24.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.364+0000 7f5741d9b640 1 -- 172.21.15.57:0/1787602064 learned_addr learned my addr 172.21.15.57:0/1787602064 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:24.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.364+0000 7f574259c640 1 --2- 172.21.15.57:0/1787602064 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55da63c2ab40 0x55da63bfdae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:24.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.364+0000 7f5742d9d640 1 --2- 172.21.15.57:0/1787602064 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55da63bfe020 0x55da63c2de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:24.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.364+0000 7f5741d9b640 1 -- 172.21.15.57:0/1787602064 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55da63c2ab40 msgr2=0x55da63bfdae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T14:20:24.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.364+0000 7f5741d9b640 1 --2- 172.21.15.57:0/1787602064 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55da63c2ab40 0x55da63bfdae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:24.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.364+0000 7f5741d9b640 1 -- 172.21.15.57:0/1787602064 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55da63bfe020 msgr2=0x55da63c2de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T14:20:24.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.364+0000 7f5741d9b640 1 --2- 172.21.15.57:0/1787602064 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55da63bfe020 0x55da63c2de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:24.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.364+0000 7f5741d9b640 1 -- 172.21.15.57:0/1787602064 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55da63b6e3d0 con 0x55da63c2af20 2023-10-19T14:20:24.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.364+0000 7f5742d9d640 1 --2- 172.21.15.57:0/1787602064 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55da63bfe020 0x55da63c2de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:20:24.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.364+0000 7f5741d9b640 1 --2- 172.21.15.57:0/1787602064 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55da63c2af20 0x55da63c2d8d0 secure :-1 s=READY pgs=2369 cs=0 l=1 rev1=1 crypto rx=0x7f573800ca30 tx=0x7f573800f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10aa0ba3d51c9c27 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:24.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.365+0000 7f5723fff640 1 -- 172.21.15.57:0/1787602064 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5738013db0 con 0x55da63c2af20 2023-10-19T14:20:24.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.365+0000 7f5723fff640 1 -- 172.21.15.57:0/1787602064 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f573801e400 con 0x55da63c2af20 2023-10-19T14:20:24.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.365+0000 7f5723fff640 1 -- 172.21.15.57:0/1787602064 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f573801e7b0 con 0x55da63c2af20 2023-10-19T14:20:24.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.365+0000 7f574b3bd740 1 -- 172.21.15.57:0/1787602064 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55da63c2af20 msgr2=0x55da63c2d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:24.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.365+0000 7f574b3bd740 1 --2- 172.21.15.57:0/1787602064 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55da63c2af20 0x55da63c2d8d0 secure :-1 s=READY pgs=2369 cs=0 l=1 rev1=1 crypto rx=0x7f573800ca30 tx=0x7f573800f000 comp rx=0 tx=0).stop 2023-10-19T14:20:24.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.365+0000 7f574b3bd740 1 -- 172.21.15.57:0/1787602064 shutdown_connections 2023-10-19T14:20:24.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.365+0000 7f574b3bd740 1 --2- 172.21.15.57:0/1787602064 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55da63bfe020 0x55da63c2de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:24.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.365+0000 7f574b3bd740 1 --2- 172.21.15.57:0/1787602064 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55da63c2af20 0x55da63c2d8d0 unknown :-1 s=CLOSED pgs=2369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:24.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.365+0000 7f574b3bd740 1 --2- 172.21.15.57:0/1787602064 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55da63c2ab40 0x55da63bfdae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:24.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.365+0000 7f574b3bd740 1 -- 172.21.15.57:0/1787602064 >> 172.21.15.57:0/1787602064 conn(0x55da63bffd50 msgr2=0x55da63c488d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:24.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.365+0000 7f574b3bd740 1 -- 172.21.15.57:0/1787602064 shutdown_connections 2023-10-19T14:20:24.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.365+0000 7f574b3bd740 1 -- 172.21.15.57:0/1787602064 wait complete. 2023-10-19T14:20:24.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.366+0000 7f574b3bd740 1 -- start start 2023-10-19T14:20:24.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.366+0000 7f574b3bd740 1 Processor -- start 2023-10-19T14:20:24.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.367+0000 7f574b3bd740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55da63c2ab40 0x55da63c30830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:24.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.367+0000 7f574b3bd740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55da63c2af20 0x55da63c1e990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:24.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.367+0000 7f574b3bd740 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55da63c1eed0 0x55da63c42920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:24.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.367+0000 7f574b3bd740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da63b69760 con 0x55da63c2ab40 2023-10-19T14:20:24.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.367+0000 7f574b3bd740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da63b248f0 con 0x55da63c2af20 2023-10-19T14:20:24.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.367+0000 7f574b3bd740 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x55da63b24590 con 0x55da63c1eed0 2023-10-19T14:20:24.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.367+0000 7f574b3bd740 10 client.0 _ll_register_callbacks cb 0x55da63b60ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-19T14:20:24.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.367+0000 7f574259c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55da63c2ab40 0x55da63c30830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:24.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.367+0000 7f574259c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55da63c2ab40 0x55da63c30830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42172/0 (socket says 172.21.15.57:42172) 2023-10-19T14:20:24.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.367+0000 7f574259c640 1 -- 172.21.15.57:0/2940188824 learned_addr learned my addr 172.21.15.57:0/2940188824 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:24.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.367+0000 7f5741d9b640 1 --2- 172.21.15.57:0/2940188824 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55da63c2af20 0x55da63c1e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:24.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.367+0000 7f5742d9d640 1 --2- 172.21.15.57:0/2940188824 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55da63c1eed0 0x55da63c42920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:24.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.367+0000 7f574259c640 1 -- 172.21.15.57:0/2940188824 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55da63c1eed0 msgr2=0x55da63c42920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T14:20:24.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.367+0000 7f574259c640 1 --2- 172.21.15.57:0/2940188824 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55da63c1eed0 0x55da63c42920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:24.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.367+0000 7f574259c640 1 -- 172.21.15.57:0/2940188824 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55da63c2af20 msgr2=0x55da63c1e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T14:20:24.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.367+0000 7f574259c640 1 --2- 172.21.15.57:0/2940188824 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55da63c2af20 0x55da63c1e990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:24.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.367+0000 7f574259c640 1 -- 172.21.15.57:0/2940188824 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f573800c7a0 con 0x55da63c2ab40 2023-10-19T14:20:24.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.367+0000 7f574259c640 1 --2- 172.21.15.57:0/2940188824 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55da63c2ab40 0x55da63c30830 secure :-1 s=READY pgs=2370 cs=0 l=1 rev1=1 crypto rx=0x7f573c054af0 tx=0x7f573c060a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f3932f6cb9f8f44 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:24.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.368+0000 7f57217fa640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2023-10-19T14:20:24.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.368+0000 7f57217fa640 1 -- 172.21.15.57:0/2940188824 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f573c0604f0 con 0x55da63c2ab40 2023-10-19T14:20:24.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.368+0000 7f57217fa640 1 -- 172.21.15.57:0/2940188824 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(223..223 src has 1..223) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f573c060650 con 0x55da63c2ab40 2023-10-19T14:20:24.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.368+0000 7f574b3bd740 1 -- 172.21.15.57:0/2940188824 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55da63b62740 con 0x55da63c2ab40 2023-10-19T14:20:24.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.368+0000 7f57217fa640 1 -- 172.21.15.57:0/2940188824 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=90) v2 ==== 24+0+0 (secure 0 0 0) 0x7f573c0543e0 con 0x55da63c2ab40 2023-10-19T14:20:24.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.368+0000 7f574b3bd740 10 client.11865 fetch_fsmap learned FSMap version 90 2023-10-19T14:20:24.391 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.368+0000 7f574b3bd740 1 -- 172.21.15.57:0/2940188824 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=90,monmap=2+}) v3 -- 0x55da63b67de0 con 0x55da63c2ab40 2023-10-19T14:20:24.391 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.368+0000 7f57217fa640 1 -- 172.21.15.57:0/2940188824 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x55da63b62740 con 0x55da63c2ab40 2023-10-19T14:20:24.391 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.368+0000 7f57217fa640 1 -- 172.21.15.57:0/2940188824 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 90) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f573c054540 con 0x55da63c2ab40 2023-10-19T14:20:24.391 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.368+0000 7f574b3bd740 10 client.11865 fetch_fsmap finished waiting for FSMap version 90 2023-10-19T14:20:24.391 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.369+0000 7f574b3bd740 10 client.11865 resolve_mds: resolved name 'a' to daemon mds.a 2023-10-19T14:20:24.391 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.369+0000 7f574b3bd740 20 client.11865 populate_metadata read hostname 'smithi057' 2023-10-19T14:20:24.392 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.369+0000 7f574b3bd740 1 --2- 172.21.15.57:0/2940188824 >> [v2:172.21.15.57:6833/1529368280,v1:172.21.15.57:6835/1529368280] conn(0x55da63c61bd0 0x55da63c65bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:24.392 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.369+0000 7f574b3bd740 4 client.11865 mds_command: new command op to 11718 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-19T14:20:24.392 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.369+0000 7f574b3bd740 1 -- 172.21.15.57:0/2940188824 --> [v2:172.21.15.57:6833/1529368280,v1:172.21.15.57:6835/1529368280] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55da63b62460 con 0x55da63c61bd0 2023-10-19T14:20:24.392 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.369+0000 7f5741d9b640 1 --2- 172.21.15.57:0/2940188824 >> [v2:172.21.15.57:6833/1529368280,v1:172.21.15.57:6835/1529368280] conn(0x55da63c61bd0 0x55da63c65bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:24.392 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.369+0000 7f5741d9b640 1 --2- 172.21.15.57:0/2940188824 >> [v2:172.21.15.57:6833/1529368280,v1:172.21.15.57:6835/1529368280] conn(0x55da63c61bd0 0x55da63c65bf0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e7c983b49ea92f91 server_cookie=a5dccf8a700a94dc in_seq=0 out_seq=0 2023-10-19T14:20:24.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.369+0000 7f57217fa640 10 client.11865 ms_handle_connect on v2:172.21.15.57:6833/1529368280 2023-10-19T14:20:24.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.369+0000 7f57217fa640 1 -- 172.21.15.57:0/2940188824 <== mds.0 v2:172.21.15.57:6833/1529368280 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55da63b62460 con 0x55da63c61bd0 2023-10-19T14:20:24.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.369+0000 7f57217fa640 10 client.11865 handle_command_reply: tid=0 2023-10-19T14:20:24.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.370+0000 7f574b3bd740 1 client.11865 shutdown 2023-10-19T14:20:24.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.370+0000 7f574b3bd740 1 -- 172.21.15.57:0/2940188824 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55da63c2ab40 msgr2=0x55da63c30830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:24.394 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.370+0000 7f574b3bd740 1 --2- 172.21.15.57:0/2940188824 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55da63c2ab40 0x55da63c30830 secure :-1 s=READY pgs=2370 cs=0 l=1 rev1=1 crypto rx=0x7f573c054af0 tx=0x7f573c060a40 comp rx=0 tx=0).stop 2023-10-19T14:20:24.394 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.370+0000 7f574b3bd740 1 -- 172.21.15.57:0/2940188824 shutdown_connections 2023-10-19T14:20:24.394 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.370+0000 7f574b3bd740 1 --2- 172.21.15.57:0/2940188824 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55da63c1eed0 0x55da63c42920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:24.394 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.370+0000 7f574b3bd740 1 --2- 172.21.15.57:0/2940188824 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55da63c2af20 0x55da63c1e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:24.394 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.370+0000 7f574b3bd740 1 --2- 172.21.15.57:0/2940188824 >> [v2:172.21.15.57:6833/1529368280,v1:172.21.15.57:6835/1529368280] conn(0x55da63c61bd0 0x55da63c65bf0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:24.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.370+0000 7f574b3bd740 1 --2- 172.21.15.57:0/2940188824 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55da63c2ab40 0x55da63c30830 unknown :-1 s=CLOSED pgs=2370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:24.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.370+0000 7f574b3bd740 1 -- 172.21.15.57:0/2940188824 >> 172.21.15.57:0/2940188824 conn(0x55da63bfe020 msgr2=0x55da63c018f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:24.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.370+0000 7f57217fa640 0 client.11865 ms_handle_reset on v2:172.21.15.57:6833/1529368280 2023-10-19T14:20:24.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.370+0000 7f574b3bd740 1 -- 172.21.15.57:0/2940188824 shutdown_connections 2023-10-19T14:20:24.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.370+0000 7f574b3bd740 1 -- 172.21.15.57:0/2940188824 wait complete. 2023-10-19T14:20:24.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.370+0000 7f574b3bd740 20 client.11865 trim_cache size 0 max 16384 2023-10-19T14:20:24.401 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.385+0000 7f574b3bd740 1 Processor -- start 2023-10-19T14:20:24.401 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.385+0000 7f574b3bd740 1 -- start start 2023-10-19T14:20:24.401 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.386+0000 7f574b3bd740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55da63c1eed0 0x55da63c43910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:24.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.386+0000 7f574b3bd740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55da63c2ab40 0x55da63c69c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:24.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.386+0000 7f574b3bd740 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55da63c2af20 0x55da63c01830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:24.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.386+0000 7f574b3bd740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da63aea270 con 0x55da63c2ab40 2023-10-19T14:20:24.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.386+0000 7f574b3bd740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da63c3fab0 con 0x55da63c1eed0 2023-10-19T14:20:24.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.386+0000 7f574b3bd740 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x55da63c4cd90 con 0x55da63c2af20 2023-10-19T14:20:24.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.386+0000 7f5741d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55da63c2ab40 0x55da63c69c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:24.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.386+0000 7f5741d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55da63c2ab40 0x55da63c69c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42188/0 (socket says 172.21.15.57:42188) 2023-10-19T14:20:24.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.386+0000 7f5741d9b640 1 -- 172.21.15.57:0/1252134782 learned_addr learned my addr 172.21.15.57:0/1252134782 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:24.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.386+0000 7f5742d9d640 1 --2- 172.21.15.57:0/1252134782 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55da63c2af20 0x55da63c01830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:24.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.386+0000 7f574259c640 1 --2- 172.21.15.57:0/1252134782 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55da63c1eed0 0x55da63c43910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:24.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.386+0000 7f5741d9b640 1 -- 172.21.15.57:0/1252134782 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55da63c2af20 msgr2=0x55da63c01830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T14:20:24.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.386+0000 7f5741d9b640 1 --2- 172.21.15.57:0/1252134782 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55da63c2af20 0x55da63c01830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:24.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.386+0000 7f5741d9b640 1 -- 172.21.15.57:0/1252134782 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55da63c1eed0 msgr2=0x55da63c43910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T14:20:24.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.386+0000 7f5741d9b640 1 --2- 172.21.15.57:0/1252134782 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55da63c1eed0 0x55da63c43910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:24.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.386+0000 7f5741d9b640 1 -- 172.21.15.57:0/1252134782 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55da63b06d00 con 0x55da63c2ab40 2023-10-19T14:20:24.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.386+0000 7f5742d9d640 1 --2- 172.21.15.57:0/1252134782 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55da63c2af20 0x55da63c01830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:20:24.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.386+0000 7f574259c640 1 --2- 172.21.15.57:0/1252134782 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55da63c1eed0 0x55da63c43910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:20:24.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.387+0000 7f5741d9b640 1 --2- 172.21.15.57:0/1252134782 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55da63c2ab40 0x55da63c69c80 secure :-1 s=READY pgs=2371 cs=0 l=1 rev1=1 crypto rx=0x7f5738021040 tx=0x7f57380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e4e61a81ea33e03 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:24.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.387+0000 7f5730ff9640 1 -- 172.21.15.57:0/1252134782 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5738013bb0 con 0x55da63c2ab40 2023-10-19T14:20:24.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.387+0000 7f5730ff9640 1 -- 172.21.15.57:0/1252134782 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f5738017070 con 0x55da63c2ab40 2023-10-19T14:20:24.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.387+0000 7f5730ff9640 1 -- 172.21.15.57:0/1252134782 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f573801eae0 con 0x55da63c2ab40 2023-10-19T14:20:24.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.387+0000 7f574b3bd740 1 -- 172.21.15.57:0/1252134782 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55da63c2ab40 msgr2=0x55da63c69c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:24.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.387+0000 7f574b3bd740 1 --2- 172.21.15.57:0/1252134782 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55da63c2ab40 0x55da63c69c80 secure :-1 s=READY pgs=2371 cs=0 l=1 rev1=1 crypto rx=0x7f5738021040 tx=0x7f57380046c0 comp rx=0 tx=0).stop 2023-10-19T14:20:24.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.387+0000 7f574b3bd740 1 -- 172.21.15.57:0/1252134782 shutdown_connections 2023-10-19T14:20:24.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.387+0000 7f574b3bd740 1 --2- 172.21.15.57:0/1252134782 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55da63c2af20 0x55da63c01830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:24.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.387+0000 7f574b3bd740 1 --2- 172.21.15.57:0/1252134782 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55da63c2ab40 0x55da63c69c80 unknown :-1 s=CLOSED pgs=2371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:24.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.387+0000 7f574b3bd740 1 --2- 172.21.15.57:0/1252134782 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55da63c1eed0 0x55da63c43910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:24.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.387+0000 7f574b3bd740 1 -- 172.21.15.57:0/1252134782 >> 172.21.15.57:0/1252134782 conn(0x55da63bfe020 msgr2=0x55da63c49090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:24.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.387+0000 7f574b3bd740 1 -- 172.21.15.57:0/1252134782 shutdown_connections 2023-10-19T14:20:24.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.387+0000 7f574b3bd740 1 -- 172.21.15.57:0/1252134782 wait complete. 2023-10-19T14:20:24.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.388+0000 7f574b3bd740 1 -- start start 2023-10-19T14:20:24.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.388+0000 7f574b3bd740 1 Processor -- start 2023-10-19T14:20:24.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.389+0000 7f574b3bd740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55da63c1eed0 0x55da63caa480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:24.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.389+0000 7f574b3bd740 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55da63c2ab40 0x55da63c027e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:24.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.389+0000 7f574b3bd740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55da63c2af20 0x55da63c02d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:24.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.389+0000 7f574b3bd740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da63bfd3f0 con 0x55da63c1eed0 2023-10-19T14:20:24.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.389+0000 7f574b3bd740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da63cb0f40 con 0x55da63c2af20 2023-10-19T14:20:24.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.389+0000 7f574b3bd740 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x55da63c42ce0 con 0x55da63c2ab40 2023-10-19T14:20:24.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.389+0000 7f574b3bd740 10 client.0 _ll_register_callbacks cb 0x7f573800adc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-19T14:20:24.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.389+0000 7f574259c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55da63c1eed0 0x55da63caa480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:24.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.389+0000 7f574259c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55da63c1eed0 0x55da63caa480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42194/0 (socket says 172.21.15.57:42194) 2023-10-19T14:20:24.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.389+0000 7f574259c640 1 -- 172.21.15.57:0/2256227635 learned_addr learned my addr 172.21.15.57:0/2256227635 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:24.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.389+0000 7f5741d9b640 1 --2- 172.21.15.57:0/2256227635 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55da63c2ab40 0x55da63c027e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:24.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.389+0000 7f5742d9d640 1 --2- 172.21.15.57:0/2256227635 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55da63c2af20 0x55da63c02d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:24.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.389+0000 7f574259c640 1 -- 172.21.15.57:0/2256227635 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55da63c2ab40 msgr2=0x55da63c027e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T14:20:24.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.389+0000 7f574259c640 1 --2- 172.21.15.57:0/2256227635 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55da63c2ab40 0x55da63c027e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:24.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.389+0000 7f574259c640 1 -- 172.21.15.57:0/2256227635 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55da63c2af20 msgr2=0x55da63c02d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T14:20:24.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.389+0000 7f574259c640 1 --2- 172.21.15.57:0/2256227635 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55da63c2af20 0x55da63c02d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:24.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.389+0000 7f574259c640 1 -- 172.21.15.57:0/2256227635 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55da63b6e3d0 con 0x55da63c1eed0 2023-10-19T14:20:24.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.390+0000 7f574259c640 1 --2- 172.21.15.57:0/2256227635 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55da63c1eed0 0x55da63caa480 secure :-1 s=READY pgs=2372 cs=0 l=1 rev1=1 crypto rx=0x7f573c0580b0 tx=0x7f573c054fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7cd735bc36e3652 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:24.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.390+0000 7f57217fa640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2023-10-19T14:20:24.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.390+0000 7f57217fa640 1 -- 172.21.15.57:0/2256227635 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f573c0545e0 con 0x55da63c1eed0 2023-10-19T14:20:24.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.390+0000 7f57217fa640 1 -- 172.21.15.57:0/2256227635 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(223..223 src has 1..223) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f573c05c740 con 0x55da63c1eed0 2023-10-19T14:20:24.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.390+0000 7f574b3bd740 1 -- 172.21.15.57:0/2256227635 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f573801e400 con 0x55da63c1eed0 2023-10-19T14:20:24.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.390+0000 7f57217fa640 1 -- 172.21.15.57:0/2256227635 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=90) v2 ==== 24+0+0 (secure 0 0 0) 0x7f573c054d00 con 0x55da63c1eed0 2023-10-19T14:20:24.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.390+0000 7f574b3bd740 10 client.11871 fetch_fsmap learned FSMap version 90 2023-10-19T14:20:24.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.390+0000 7f574b3bd740 1 -- 172.21.15.57:0/2256227635 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=90,monmap=2+}) v3 -- 0x55da63b2c4f0 con 0x55da63c1eed0 2023-10-19T14:20:24.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.390+0000 7f57217fa640 1 -- 172.21.15.57:0/2256227635 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f573801e400 con 0x55da63c1eed0 2023-10-19T14:20:24.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.390+0000 7f57217fa640 1 -- 172.21.15.57:0/2256227635 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 90) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f573c081070 con 0x55da63c1eed0 2023-10-19T14:20:24.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.390+0000 7f574b3bd740 10 client.11871 fetch_fsmap finished waiting for FSMap version 90 2023-10-19T14:20:24.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.390+0000 7f574b3bd740 10 client.11871 resolve_mds: resolved name 'a' to daemon mds.a 2023-10-19T14:20:24.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.390+0000 7f574b3bd740 20 client.11871 populate_metadata read hostname 'smithi057' 2023-10-19T14:20:24.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.390+0000 7f574b3bd740 1 --2- 172.21.15.57:0/2256227635 >> [v2:172.21.15.57:6833/1529368280,v1:172.21.15.57:6835/1529368280] conn(0x55da63cb4310 0x55da63cb46f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:24.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.390+0000 7f574b3bd740 4 client.11871 mds_command: new command op to 11718 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-19T14:20:24.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.390+0000 7f574b3bd740 1 -- 172.21.15.57:0/2256227635 --> [v2:172.21.15.57:6833/1529368280,v1:172.21.15.57:6835/1529368280] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55da6386b6c0 con 0x55da63cb4310 2023-10-19T14:20:24.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.390+0000 7f5741d9b640 1 --2- 172.21.15.57:0/2256227635 >> [v2:172.21.15.57:6833/1529368280,v1:172.21.15.57:6835/1529368280] conn(0x55da63cb4310 0x55da63cb46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:24.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.391+0000 7f57217fa640 10 client.11871 ms_handle_connect on v2:172.21.15.57:6833/1529368280 2023-10-19T14:20:24.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.391+0000 7f5741d9b640 1 --2- 172.21.15.57:0/2256227635 >> [v2:172.21.15.57:6833/1529368280,v1:172.21.15.57:6835/1529368280] conn(0x55da63cb4310 0x55da63cb46f0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=611d484e0e783ad server_cookie=ffd0cb194a4c9a69 in_seq=0 out_seq=0 2023-10-19T14:20:24.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.391+0000 7f57217fa640 1 -- 172.21.15.57:0/2256227635 <== mds.0 v2:172.21.15.57:6833/1529368280 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+334 (crc 0 0 0) 0x55da6386b6c0 con 0x55da63cb4310 2023-10-19T14:20:24.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.391+0000 7f57217fa640 10 client.11871 handle_command_reply: tid=0 2023-10-19T14:20:24.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.391+0000 7f574b3bd740 1 client.11871 shutdown 2023-10-19T14:20:24.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.391+0000 7f574b3bd740 1 -- 172.21.15.57:0/2256227635 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55da63c1eed0 msgr2=0x55da63caa480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:24.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.391+0000 7f574b3bd740 1 --2- 172.21.15.57:0/2256227635 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55da63c1eed0 0x55da63caa480 secure :-1 s=READY pgs=2372 cs=0 l=1 rev1=1 crypto rx=0x7f573c0580b0 tx=0x7f573c054fc0 comp rx=0 tx=0).stop 2023-10-19T14:20:24.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.391+0000 7f574b3bd740 1 -- 172.21.15.57:0/2256227635 shutdown_connections 2023-10-19T14:20:24.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.391+0000 7f574b3bd740 1 --2- 172.21.15.57:0/2256227635 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55da63c2af20 0x55da63c02d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:24.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.391+0000 7f574b3bd740 1 --2- 172.21.15.57:0/2256227635 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55da63c2ab40 0x55da63c027e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:24.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.391+0000 7f574b3bd740 1 --2- 172.21.15.57:0/2256227635 >> [v2:172.21.15.57:6833/1529368280,v1:172.21.15.57:6835/1529368280] conn(0x55da63cb4310 0x55da63cb46f0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:24.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.391+0000 7f574b3bd740 1 --2- 172.21.15.57:0/2256227635 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55da63c1eed0 0x55da63caa480 unknown :-1 s=CLOSED pgs=2372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:24.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.391+0000 7f574b3bd740 1 -- 172.21.15.57:0/2256227635 >> 172.21.15.57:0/2256227635 conn(0x55da63bfe020 msgr2=0x55da63c01830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:24.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.391+0000 7f57217fa640 0 client.11871 ms_handle_reset on v2:172.21.15.57:6833/1529368280 2023-10-19T14:20:24.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.391+0000 7f574b3bd740 1 -- 172.21.15.57:0/2256227635 shutdown_connections 2023-10-19T14:20:24.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.391+0000 7f574b3bd740 1 -- 172.21.15.57:0/2256227635 wait complete. 2023-10-19T14:20:24.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.391+0000 7f574b3bd740 20 client.11871 trim_cache size 0 max 16384 2023-10-19T14:20:24.421 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2023-10-19T14:20:24.421 INFO:teuthology.orchestra.run.smithi057.stdout: "cluster_fsid": "1e9beb19-3260-4dd7-85ef-5b399ce774ac", 2023-10-19T14:20:24.421 INFO:teuthology.orchestra.run.smithi057.stdout: "whoami": 0, 2023-10-19T14:20:24.421 INFO:teuthology.orchestra.run.smithi057.stdout: "id": 11718, 2023-10-19T14:20:24.422 INFO:teuthology.orchestra.run.smithi057.stdout: "want_state": "up:active", 2023-10-19T14:20:24.422 INFO:teuthology.orchestra.run.smithi057.stdout: "state": "up:active", 2023-10-19T14:20:24.422 INFO:teuthology.orchestra.run.smithi057.stdout: "fs_name": "cephfs", 2023-10-19T14:20:24.422 INFO:teuthology.orchestra.run.smithi057.stdout: "rank_uptime": 4.6841176669999998, 2023-10-19T14:20:24.422 INFO:teuthology.orchestra.run.smithi057.stdout: "mdsmap_epoch": 90, 2023-10-19T14:20:24.423 INFO:teuthology.orchestra.run.smithi057.stdout: "osdmap_epoch": 223, 2023-10-19T14:20:24.423 INFO:teuthology.orchestra.run.smithi057.stdout: "osdmap_epoch_barrier": 223, 2023-10-19T14:20:24.423 INFO:teuthology.orchestra.run.smithi057.stdout: "uptime": 9.4285045079999996 2023-10-19T14:20:24.423 INFO:teuthology.orchestra.run.smithi057.stdout:} 2023-10-19T14:20:24.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.392+0000 7f574882b640 1 -- 172.21.15.57:0/4108747631 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f572406af90 msgr2=0x7f572406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:24.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.392+0000 7f574882b640 1 --2- 172.21.15.57:0/4108747631 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f572406af90 0x7f572406d450 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f57440afa90 tx=0x7f5738008040 comp rx=0 tx=0).stop 2023-10-19T14:20:24.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.392+0000 7f574882b640 1 -- 172.21.15.57:0/4108747631 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f574406c0a0 msgr2=0x7f5744157b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:24.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.392+0000 7f574882b640 1 --2- 172.21.15.57:0/4108747631 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f574406c0a0 0x7f5744157b10 secure :-1 s=READY pgs=2368 cs=0 l=1 rev1=1 crypto rx=0x7f573c00ba50 tx=0x7f573c00bf10 comp rx=0 tx=0).stop 2023-10-19T14:20:24.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.392+0000 7f574882b640 1 -- 172.21.15.57:0/4108747631 shutdown_connections 2023-10-19T14:20:24.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.392+0000 7f574882b640 1 --2- 172.21.15.57:0/4108747631 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f572406af90 0x7f572406d450 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:24.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.392+0000 7f574882b640 1 --2- 172.21.15.57:0/4108747631 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5744141690 0x7f574413fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:24.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.392+0000 7f574882b640 1 --2- 172.21.15.57:0/4108747631 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f574413e520 0x7f574413aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:24.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.392+0000 7f574882b640 1 --2- 172.21.15.57:0/4108747631 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f574406c0a0 0x7f5744157b10 unknown :-1 s=CLOSED pgs=2368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:24.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.392+0000 7f574882b640 1 -- 172.21.15.57:0/4108747631 >> 172.21.15.57:0/4108747631 conn(0x7f5744069930 msgr2=0x7f574413d750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:24.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.392+0000 7f574882b640 1 -- 172.21.15.57:0/4108747631 shutdown_connections 2023-10-19T14:20:24.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.392+0000 7f574882b640 1 -- 172.21.15.57:0/4108747631 wait complete. 2023-10-19T14:20:24.428 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-19T14:20:24.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.533+0000 7fc21970f640 1 -- 172.21.15.57:0/1946952308 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc214141690 msgr2=0x7fc2140703e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:24.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.533+0000 7fc21970f640 1 --2- 172.21.15.57:0/1946952308 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc214141690 0x7fc2140703e0 secure :-1 s=READY pgs=2374 cs=0 l=1 rev1=1 crypto rx=0x7fc208009830 tx=0x7fc20802bcd0 comp rx=0 tx=0).stop 2023-10-19T14:20:24.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.533+0000 7fc21970f640 1 -- 172.21.15.57:0/1946952308 shutdown_connections 2023-10-19T14:20:24.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.533+0000 7fc21970f640 1 --2- 172.21.15.57:0/1946952308 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc214141690 0x7fc2140703e0 unknown :-1 s=CLOSED pgs=2374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:24.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.533+0000 7fc21970f640 1 -- 172.21.15.57:0/1946952308 >> 172.21.15.57:0/1946952308 conn(0x7fc214069930 msgr2=0x7fc214069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:24.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.533+0000 7fc21970f640 1 -- 172.21.15.57:0/1946952308 shutdown_connections 2023-10-19T14:20:24.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.533+0000 7fc21970f640 1 -- 172.21.15.57:0/1946952308 wait complete. 2023-10-19T14:20:24.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.535+0000 7fc21970f640 1 Processor -- start 2023-10-19T14:20:24.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.535+0000 7fc21970f640 1 -- start start 2023-10-19T14:20:24.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.535+0000 7fc21970f640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc214070920 0x7fc2141f3040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:24.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.535+0000 7fc21970f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc21413e520 0x7fc2141dba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:24.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.535+0000 7fc21970f640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc214141690 0x7fc2140ac770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:24.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.535+0000 7fc21970f640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2141e2be0 con 0x7fc21413e520 2023-10-19T14:20:24.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.535+0000 7fc21970f640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc214138ab0 con 0x7fc214070920 2023-10-19T14:20:24.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.535+0000 7fc21970f640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc214139070 con 0x7fc214141690 2023-10-19T14:20:24.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.536+0000 7fc2127fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc21413e520 0x7fc2141dba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:24.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.536+0000 7fc2127fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc21413e520 0x7fc2141dba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42218/0 (socket says 172.21.15.57:42218) 2023-10-19T14:20:24.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.536+0000 7fc2127fc640 1 -- 172.21.15.57:0/1397203841 learned_addr learned my addr 172.21.15.57:0/1397203841 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:24.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.536+0000 7fc212ffd640 1 --2- 172.21.15.57:0/1397203841 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc214070920 0x7fc2141f3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:24.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.536+0000 7fc2137fe640 1 --2- 172.21.15.57:0/1397203841 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc214141690 0x7fc2140ac770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:24.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.536+0000 7fc2127fc640 1 -- 172.21.15.57:0/1397203841 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc214141690 msgr2=0x7fc2140ac770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:24.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.536+0000 7fc2127fc640 1 --2- 172.21.15.57:0/1397203841 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc214141690 0x7fc2140ac770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:24.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.536+0000 7fc2127fc640 1 -- 172.21.15.57:0/1397203841 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc214070920 msgr2=0x7fc2141f3040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:24.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.536+0000 7fc2127fc640 1 --2- 172.21.15.57:0/1397203841 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc214070920 0x7fc2141f3040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:24.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.536+0000 7fc2127fc640 1 -- 172.21.15.57:0/1397203841 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc208009490 con 0x7fc21413e520 2023-10-19T14:20:24.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.536+0000 7fc2127fc640 1 --2- 172.21.15.57:0/1397203841 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc21413e520 0x7fc2141dba70 secure :-1 s=READY pgs=2375 cs=0 l=1 rev1=1 crypto rx=0x7fc208000c00 tx=0x7fc2080038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:24.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.536+0000 7fc1fb7fe640 1 -- 172.21.15.57:0/1397203841 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc208003ea0 con 0x7fc21413e520 2023-10-19T14:20:24.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.536+0000 7fc1fb7fe640 1 -- 172.21.15.57:0/1397203841 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fc208034530 con 0x7fc21413e520 2023-10-19T14:20:24.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.536+0000 7fc21970f640 1 -- 172.21.15.57:0/1397203841 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2141f3580 con 0x7fc21413e520 2023-10-19T14:20:24.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.536+0000 7fc1fb7fe640 1 -- 172.21.15.57:0/1397203841 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc208034940 con 0x7fc21413e520 2023-10-19T14:20:24.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.536+0000 7fc21970f640 1 -- 172.21.15.57:0/1397203841 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2141e30d0 con 0x7fc21413e520 2023-10-19T14:20:24.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.537+0000 7fc21970f640 1 -- 172.21.15.57:0/1397203841 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2141391e0 con 0x7fc21413e520 2023-10-19T14:20:24.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.538+0000 7fc1fb7fe640 1 -- 172.21.15.57:0/1397203841 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc208034aa0 con 0x7fc21413e520 2023-10-19T14:20:24.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.538+0000 7fc1fb7fe640 1 --2- 172.21.15.57:0/1397203841 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fc20006af60 0x7fc20006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:24.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.538+0000 7fc1fb7fe640 1 -- 172.21.15.57:0/1397203841 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fc2080ac080 con 0x7fc21413e520 2023-10-19T14:20:24.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.538+0000 7fc212ffd640 1 --2- 172.21.15.57:0/1397203841 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fc20006af60 0x7fc20006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:24.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.539+0000 7fc212ffd640 1 --2- 172.21.15.57:0/1397203841 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fc20006af60 0x7fc20006d420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc204005660 tx=0x7fc204003d90 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:24.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.543+0000 7fc1fb7fe640 1 -- 172.21.15.57:0/1397203841 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc208077000 con 0x7fc21413e520 2023-10-19T14:20:24.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.754+0000 7fc21970f640 1 -- 172.21.15.57:0/1397203841 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fc2140690e0 con 0x7fc21413e520 2023-10-19T14:20:24.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.754+0000 7fc1fb7fe640 1 -- 172.21.15.57:0/1397203841 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v90) v1 ==== 53+0+73 (secure 0 0 0) 0x7fc208042060 con 0x7fc21413e520 2023-10-19T14:20:24.758 INFO:teuthology.orchestra.run.smithi057.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-19T14:20:24.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.756+0000 7fc21970f640 1 -- 172.21.15.57:0/1397203841 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fc20006af60 msgr2=0x7fc20006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:24.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.756+0000 7fc21970f640 1 --2- 172.21.15.57:0/1397203841 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fc20006af60 0x7fc20006d420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc204005660 tx=0x7fc204003d90 comp rx=0 tx=0).stop 2023-10-19T14:20:24.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.756+0000 7fc21970f640 1 -- 172.21.15.57:0/1397203841 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc21413e520 msgr2=0x7fc2141dba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:24.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.756+0000 7fc21970f640 1 --2- 172.21.15.57:0/1397203841 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc21413e520 0x7fc2141dba70 secure :-1 s=READY pgs=2375 cs=0 l=1 rev1=1 crypto rx=0x7fc208000c00 tx=0x7fc2080038a0 comp rx=0 tx=0).stop 2023-10-19T14:20:24.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.756+0000 7fc21970f640 1 -- 172.21.15.57:0/1397203841 shutdown_connections 2023-10-19T14:20:24.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.756+0000 7fc21970f640 1 --2- 172.21.15.57:0/1397203841 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fc20006af60 0x7fc20006d420 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:24.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.756+0000 7fc21970f640 1 --2- 172.21.15.57:0/1397203841 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc214141690 0x7fc2140ac770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:24.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.756+0000 7fc21970f640 1 --2- 172.21.15.57:0/1397203841 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc21413e520 0x7fc2141dba70 unknown :-1 s=CLOSED pgs=2375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:24.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.756+0000 7fc21970f640 1 --2- 172.21.15.57:0/1397203841 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc214070920 0x7fc2141f3040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:24.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.756+0000 7fc21970f640 1 -- 172.21.15.57:0/1397203841 >> 172.21.15.57:0/1397203841 conn(0x7fc214069930 msgr2=0x7fc21413cb20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:24.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.756+0000 7fc21970f640 1 -- 172.21.15.57:0/1397203841 shutdown_connections 2023-10-19T14:20:24.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:24.756+0000 7fc21970f640 1 -- 172.21.15.57:0/1397203841 wait complete. 2023-10-19T14:20:24.770 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-10-19T14:20:24.771 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-19T14:20:24.771 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi057.front.sepia.ceph.com 2023-10-19T14:20:24.772 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-19T14:20:24.772 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-19T14:20:24.773 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-10-19T14:20:24.773 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-10-19T14:20:24.774 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-19T14:20:24.774 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-19T14:20:24.775 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:20:24.776 DEBUG:teuthology.orchestra.run.smithi057:> ip addr 2023-10-19T14:20:24.791 INFO:teuthology.orchestra.run.smithi057.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-19T14:20:24.792 INFO:teuthology.orchestra.run.smithi057.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-19T14:20:24.792 INFO:teuthology.orchestra.run.smithi057.stdout: inet 127.0.0.1/8 scope host lo 2023-10-19T14:20:24.792 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T14:20:24.792 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 ::1/128 scope host 2023-10-19T14:20:24.793 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T14:20:24.793 INFO:teuthology.orchestra.run.smithi057.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-19T14:20:24.793 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:6b:ff:86 brd ff:ff:ff:ff:ff:ff 2023-10-19T14:20:24.793 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp7s0f0 2023-10-19T14:20:24.794 INFO:teuthology.orchestra.run.smithi057.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-19T14:20:24.794 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:6b:ff:87 brd ff:ff:ff:ff:ff:ff 2023-10-19T14:20:24.794 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp7s0f1 2023-10-19T14:20:24.794 INFO:teuthology.orchestra.run.smithi057.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-19T14:20:24.795 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:bd:15:62 brd ff:ff:ff:ff:ff:ff 2023-10-19T14:20:24.795 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp1s0f0 2023-10-19T14:20:24.795 INFO:teuthology.orchestra.run.smithi057.stdout: inet 172.21.15.57/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-19T14:20:24.795 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft 36886sec preferred_lft 36886sec 2023-10-19T14:20:24.795 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 fe80::ec4:7aff:febd:1562/64 scope link noprefixroute 2023-10-19T14:20:24.796 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T14:20:24.796 INFO:teuthology.orchestra.run.smithi057.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-19T14:20:24.796 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:bd:15:63 brd ff:ff:ff:ff:ff:ff 2023-10-19T14:20:24.796 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp1s0f1 2023-10-19T14:20:24.796 INFO:teuthology.orchestra.run.smithi057.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-19T14:20:24.797 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0e:fe:f1:b5:41:59 brd ff:ff:ff:ff:ff:ff 2023-10-19T14:20:24.797 INFO:teuthology.orchestra.run.smithi057.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-19T14:20:24.797 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T14:20:24.797 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 fe80::64b1:a2ff:feb3:b08/64 scope link 2023-10-19T14:20:24.797 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T14:20:24.797 INFO:teuthology.orchestra.run.smithi057.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-19T14:20:24.798 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0e:fe:f1:b5:41:59 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T14:20:24.798 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 fe80::cfe:f1ff:feb5:4159/64 scope link 2023-10-19T14:20:24.798 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T14:20:24.798 INFO:teuthology.orchestra.run.smithi057.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-19T14:20:24.798 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 62:53:72:9d:52:71 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-19T14:20:24.799 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 fe80::6053:72ff:fe9d:5271/64 scope link 2023-10-19T14:20:24.799 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T14:20:24.799 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:20:24.800 DEBUG:teuthology.orchestra.run.smithi057:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-19T14:20:24.861 INFO:teuthology.orchestra.run.smithi057.stdout:1 2023-10-19T14:20:24.862 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:20:24.862 DEBUG:teuthology.orchestra.run.smithi057:> route 2023-10-19T14:20:24.918 INFO:teuthology.orchestra.run.smithi057.stdout:Kernel IP routing table 2023-10-19T14:20:24.918 INFO:teuthology.orchestra.run.smithi057.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-19T14:20:24.919 INFO:teuthology.orchestra.run.smithi057.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-19T14:20:24.919 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-19T14:20:24.919 INFO:teuthology.orchestra.run.smithi057.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-19T14:20:24.919 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:20:24.920 DEBUG:teuthology.orchestra.run.smithi057:> (cd / && exec bash -c ' 2023-10-19T14:20:24.920 DEBUG:teuthology.orchestra.run.smithi057:> set -e 2023-10-19T14:20:24.921 DEBUG:teuthology.orchestra.run.smithi057:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-19T14:20:24.921 DEBUG:teuthology.orchestra.run.smithi057:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-19T14:20:24.921 DEBUG:teuthology.orchestra.run.smithi057:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-19T14:20:24.922 DEBUG:teuthology.orchestra.run.smithi057:> ') 2023-10-19T14:20:25.024 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:20:25.025 DEBUG:teuthology.orchestra.run.smithi057:> ip netns list 2023-10-19T14:20:25.039 INFO:teuthology.orchestra.run.smithi057.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-19T14:20:25.040 INFO:teuthology.orchestra.run.smithi057.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-19T14:20:25.040 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-19T14:20:25.040 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:20:25.041 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip link set brx.0 up 2023-10-19T14:20:25.111 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-10-19T14:20:25.111 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-19T14:20:25.112 DEBUG:teuthology.orchestra.run.smithi057:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-19T14:20:25.167 INFO:teuthology.orchestra.run.smithi057.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-19T14:20:25.167 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-19T14:20:25.168 DEBUG:teuthology.orchestra.run.smithi057:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-19T14:20:25.222 DEBUG:teuthology.orchestra.run.smithi057:> sudo modprobe fuse 2023-10-19T14:20:25.293 DEBUG:teuthology.orchestra.run.smithi057:> cat /proc/self/mounts | awk '{print $2}' 2023-10-19T14:20:25.350 INFO:teuthology.orchestra.run.smithi057.stdout:/proc 2023-10-19T14:20:25.350 INFO:teuthology.orchestra.run.smithi057.stdout:/sys 2023-10-19T14:20:25.350 INFO:teuthology.orchestra.run.smithi057.stdout:/dev 2023-10-19T14:20:25.350 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/security 2023-10-19T14:20:25.351 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/shm 2023-10-19T14:20:25.351 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/pts 2023-10-19T14:20:25.351 INFO:teuthology.orchestra.run.smithi057.stdout:/run 2023-10-19T14:20:25.351 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/cgroup 2023-10-19T14:20:25.351 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/pstore 2023-10-19T14:20:25.352 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/bpf 2023-10-19T14:20:25.352 INFO:teuthology.orchestra.run.smithi057.stdout:/ 2023-10-19T14:20:25.353 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/selinux 2023-10-19T14:20:25.353 INFO:teuthology.orchestra.run.smithi057.stdout:/proc/sys/fs/binfmt_misc 2023-10-19T14:20:25.353 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/hugepages 2023-10-19T14:20:25.353 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/mqueue 2023-10-19T14:20:25.353 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/debug 2023-10-19T14:20:25.354 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/tracing 2023-10-19T14:20:25.354 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/fuse/connections 2023-10-19T14:20:25.354 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/config 2023-10-19T14:20:25.354 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-sysctl.service 2023-10-19T14:20:25.355 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-10-19T14:20:25.355 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-10-19T14:20:25.355 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/nfs/rpc_pipefs 2023-10-19T14:20:25.355 INFO:teuthology.orchestra.run.smithi057.stdout:/run/user/1000 2023-10-19T14:20:25.355 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph 2023-10-19T14:20:25.356 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-0 2023-10-19T14:20:25.356 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-1 2023-10-19T14:20:25.356 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-2 2023-10-19T14:20:25.356 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-3 2023-10-19T14:20:25.356 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns 2023-10-19T14:20:25.357 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T14:20:25.357 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T14:20:25.357 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-19T14:20:25.357 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-19T14:20:25.358 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:20:25.358 DEBUG:teuthology.orchestra.run.smithi057:> ls /sys/fs/fuse/connections 2023-10-19T14:20:25.405 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-10-19T14:20:25.405 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-10-19T14:20:25.448 DEBUG:teuthology.orchestra.run.smithi057:> sudo modprobe fuse 2023-10-19T14:20:25.479 DEBUG:teuthology.orchestra.run.smithi057:> cat /proc/self/mounts | awk '{print $2}' 2023-10-19T14:20:25.534 INFO:teuthology.orchestra.run.smithi057.stdout:/proc 2023-10-19T14:20:25.535 INFO:teuthology.orchestra.run.smithi057.stdout:/sys 2023-10-19T14:20:25.535 INFO:teuthology.orchestra.run.smithi057.stdout:/dev 2023-10-19T14:20:25.535 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/security 2023-10-19T14:20:25.535 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/shm 2023-10-19T14:20:25.536 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/pts 2023-10-19T14:20:25.536 INFO:teuthology.orchestra.run.smithi057.stdout:/run 2023-10-19T14:20:25.536 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/cgroup 2023-10-19T14:20:25.536 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/pstore 2023-10-19T14:20:25.537 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/bpf 2023-10-19T14:20:25.537 INFO:teuthology.orchestra.run.smithi057.stdout:/ 2023-10-19T14:20:25.537 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/selinux 2023-10-19T14:20:25.537 INFO:teuthology.orchestra.run.smithi057.stdout:/proc/sys/fs/binfmt_misc 2023-10-19T14:20:25.537 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/hugepages 2023-10-19T14:20:25.538 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/mqueue 2023-10-19T14:20:25.538 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/debug 2023-10-19T14:20:25.538 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/tracing 2023-10-19T14:20:25.538 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/fuse/connections 2023-10-19T14:20:25.539 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/config 2023-10-19T14:20:25.539 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-sysctl.service 2023-10-19T14:20:25.539 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-10-19T14:20:25.539 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-10-19T14:20:25.539 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/nfs/rpc_pipefs 2023-10-19T14:20:25.540 INFO:teuthology.orchestra.run.smithi057.stdout:/run/user/1000 2023-10-19T14:20:25.540 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph 2023-10-19T14:20:25.540 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-0 2023-10-19T14:20:25.540 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-1 2023-10-19T14:20:25.540 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-2 2023-10-19T14:20:25.541 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-3 2023-10-19T14:20:25.541 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns 2023-10-19T14:20:25.541 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T14:20:25.541 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T14:20:25.541 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-19T14:20:25.542 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-19T14:20:25.542 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:20:25.542 DEBUG:teuthology.orchestra.run.smithi057:> ls /sys/fs/fuse/connections 2023-10-19T14:20:25.544 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi057.stderr:2023-10-19T14:20:25.538+0000 7f023e61f480 -1 init, newargv = 0x55a3e8678ac0 newargc=15 2023-10-19T14:20:25.544 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi057.stderr:ceph-fuse[153701]: starting ceph client 2023-10-19T14:20:25.545 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi057.stderr:ceph-fuse[153701]: starting fuse 2023-10-19T14:20:25.559 INFO:teuthology.orchestra.run.smithi057.stdout:43 2023-10-19T14:20:25.559 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2023-10-19T14:20:25.560 DEBUG:teuthology.orchestra.run.smithi057:> sudo stdin-killer -- python3 -c ' 2023-10-19T14:20:25.560 DEBUG:teuthology.orchestra.run.smithi057:> import glob 2023-10-19T14:20:25.560 DEBUG:teuthology.orchestra.run.smithi057:> import re 2023-10-19T14:20:25.561 DEBUG:teuthology.orchestra.run.smithi057:> import os 2023-10-19T14:20:25.561 DEBUG:teuthology.orchestra.run.smithi057:> import subprocess 2023-10-19T14:20:25.562 DEBUG:teuthology.orchestra.run.smithi057:> 2023-10-19T14:20:25.562 DEBUG:teuthology.orchestra.run.smithi057:> def _find_admin_socket(client_name): 2023-10-19T14:20:25.562 DEBUG:teuthology.orchestra.run.smithi057:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-10-19T14:20:25.563 DEBUG:teuthology.orchestra.run.smithi057:> files = glob.glob(asok_path) 2023-10-19T14:20:25.563 DEBUG:teuthology.orchestra.run.smithi057:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-10-19T14:20:25.564 DEBUG:teuthology.orchestra.run.smithi057:> 2023-10-19T14:20:25.564 DEBUG:teuthology.orchestra.run.smithi057:> # Given a non-glob path, it better be there 2023-10-19T14:20:25.564 DEBUG:teuthology.orchestra.run.smithi057:> if "*" not in asok_path: 2023-10-19T14:20:25.565 DEBUG:teuthology.orchestra.run.smithi057:> assert(len(files) == 1) 2023-10-19T14:20:25.565 DEBUG:teuthology.orchestra.run.smithi057:> return files[0] 2023-10-19T14:20:25.566 DEBUG:teuthology.orchestra.run.smithi057:> 2023-10-19T14:20:25.566 DEBUG:teuthology.orchestra.run.smithi057:> for f in files: 2023-10-19T14:20:25.566 DEBUG:teuthology.orchestra.run.smithi057:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-10-19T14:20:25.567 DEBUG:teuthology.orchestra.run.smithi057:> if os.path.exists("/proc/{0}".format(pid)): 2023-10-19T14:20:25.567 DEBUG:teuthology.orchestra.run.smithi057:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-10-19T14:20:25.568 DEBUG:teuthology.orchestra.run.smithi057:> contents = proc_f.read() 2023-10-19T14:20:25.568 DEBUG:teuthology.orchestra.run.smithi057:> if mountpoint in contents: 2023-10-19T14:20:25.568 DEBUG:teuthology.orchestra.run.smithi057:> return f 2023-10-19T14:20:25.569 DEBUG:teuthology.orchestra.run.smithi057:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-10-19T14:20:25.569 DEBUG:teuthology.orchestra.run.smithi057:> 2023-10-19T14:20:25.569 DEBUG:teuthology.orchestra.run.smithi057:> print(_find_admin_socket("client.0")) 2023-10-19T14:20:25.570 DEBUG:teuthology.orchestra.run.smithi057:> ' 2023-10-19T14:20:25.684 INFO:teuthology.orchestra.run.smithi057.stdout:/var/run/ceph/ceph-client.0.153701.asok 2023-10-19T14:20:25.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:25 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2023-10-19T14:20:25.695 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.153701.asok 2023-10-19T14:20:25.695 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:20:25.696 DEBUG:teuthology.orchestra.run.smithi057:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.153701.asok status 2023-10-19T14:20:25.836 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2023-10-19T14:20:25.837 INFO:teuthology.orchestra.run.smithi057.stdout: "metadata": { 2023-10-19T14:20:25.837 INFO:teuthology.orchestra.run.smithi057.stdout: "ceph_sha1": "f7420d7f8e55ceea786dcbf536c338ce920c21eb", 2023-10-19T14:20:25.837 INFO:teuthology.orchestra.run.smithi057.stdout: "ceph_version": "ceph version 18.0.0-6786-gf7420d7f (f7420d7f8e55ceea786dcbf536c338ce920c21eb) reef (dev)", 2023-10-19T14:20:25.837 INFO:teuthology.orchestra.run.smithi057.stdout: "entity_id": "0", 2023-10-19T14:20:25.838 INFO:teuthology.orchestra.run.smithi057.stdout: "hostname": "smithi057", 2023-10-19T14:20:25.838 INFO:teuthology.orchestra.run.smithi057.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-10-19T14:20:25.838 INFO:teuthology.orchestra.run.smithi057.stdout: "pid": "153701", 2023-10-19T14:20:25.838 INFO:teuthology.orchestra.run.smithi057.stdout: "root": "/" 2023-10-19T14:20:25.839 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:20:25.839 INFO:teuthology.orchestra.run.smithi057.stdout: "dentry_count": 0, 2023-10-19T14:20:25.839 INFO:teuthology.orchestra.run.smithi057.stdout: "dentry_pinned_count": 0, 2023-10-19T14:20:25.839 INFO:teuthology.orchestra.run.smithi057.stdout: "id": 11883, 2023-10-19T14:20:25.839 INFO:teuthology.orchestra.run.smithi057.stdout: "inst": { 2023-10-19T14:20:25.840 INFO:teuthology.orchestra.run.smithi057.stdout: "name": { 2023-10-19T14:20:25.840 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "client", 2023-10-19T14:20:25.840 INFO:teuthology.orchestra.run.smithi057.stdout: "num": 11883 2023-10-19T14:20:25.840 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:20:25.841 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": { 2023-10-19T14:20:25.841 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "v1", 2023-10-19T14:20:25.841 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": "192.168.0.1:0", 2023-10-19T14:20:25.841 INFO:teuthology.orchestra.run.smithi057.stdout: "nonce": 649836429 2023-10-19T14:20:25.841 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:20:25.842 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:20:25.842 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": { 2023-10-19T14:20:25.842 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "v1", 2023-10-19T14:20:25.842 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": "192.168.0.1:0", 2023-10-19T14:20:25.843 INFO:teuthology.orchestra.run.smithi057.stdout: "nonce": 649836429 2023-10-19T14:20:25.843 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:20:25.843 INFO:teuthology.orchestra.run.smithi057.stdout: "inst_str": "client.11883 192.168.0.1:0/649836429", 2023-10-19T14:20:25.843 INFO:teuthology.orchestra.run.smithi057.stdout: "addr_str": "192.168.0.1:0/649836429", 2023-10-19T14:20:25.843 INFO:teuthology.orchestra.run.smithi057.stdout: "inode_count": 1, 2023-10-19T14:20:25.844 INFO:teuthology.orchestra.run.smithi057.stdout: "mds_epoch": 90, 2023-10-19T14:20:25.844 INFO:teuthology.orchestra.run.smithi057.stdout: "osd_epoch": 223, 2023-10-19T14:20:25.844 INFO:teuthology.orchestra.run.smithi057.stdout: "osd_epoch_barrier": 0, 2023-10-19T14:20:25.844 INFO:teuthology.orchestra.run.smithi057.stdout: "blocklisted": false, 2023-10-19T14:20:25.844 INFO:teuthology.orchestra.run.smithi057.stdout: "fs_name": "cephfs" 2023-10-19T14:20:25.845 INFO:teuthology.orchestra.run.smithi057.stdout:} 2023-10-19T14:20:25.846 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:20:25.847 DEBUG:teuthology.orchestra.run.smithi057:> stat --file-system '--printf=%T 2023-10-19T14:20:25.847 DEBUG:teuthology.orchestra.run.smithi057:> ' -- /home/ubuntu/cephtest/mnt.0 2023-10-19T14:20:25.862 INFO:teuthology.orchestra.run.smithi057.stdout:fuseblk 2023-10-19T14:20:25.863 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-10-19T14:20:25.863 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:20:25.864 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-19T14:20:25.934 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2023-10-19T14:20:26.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.084+0000 7fa396d9e640 1 -- 172.21.15.57:0/3041737596 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa3900ab500 msgr2=0x7fa3900ab990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:26.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.084+0000 7fa396d9e640 1 --2- 172.21.15.57:0/3041737596 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa3900ab500 0x7fa3900ab990 secure :-1 s=READY pgs=2377 cs=0 l=1 rev1=1 crypto rx=0x7fa3900713a0 tx=0x7fa38002bda0 comp rx=0 tx=0).stop 2023-10-19T14:20:26.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.084+0000 7fa396d9e640 1 -- 172.21.15.57:0/3041737596 shutdown_connections 2023-10-19T14:20:26.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.084+0000 7fa396d9e640 1 --2- 172.21.15.57:0/3041737596 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa3900ab500 0x7fa3900ab990 unknown :-1 s=CLOSED pgs=2377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:26.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.084+0000 7fa396d9e640 1 --2- 172.21.15.57:0/3041737596 >> v2:172.21.15.179:3300/0 conn(0x7fa3900c96b0 0x7fa3900c9b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:26.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.084+0000 7fa396d9e640 1 --2- 172.21.15.57:0/3041737596 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa39006af00 0x7fa39006b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:26.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.084+0000 7fa396d9e640 1 -- 172.21.15.57:0/3041737596 >> 172.21.15.57:0/3041737596 conn(0x7fa390069930 msgr2=0x7fa390069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:26.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.084+0000 7fa396d9e640 1 -- 172.21.15.57:0/3041737596 shutdown_connections 2023-10-19T14:20:26.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.084+0000 7fa396d9e640 1 -- 172.21.15.57:0/3041737596 wait complete. 2023-10-19T14:20:26.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.085+0000 7fa396d9e640 1 Processor -- start 2023-10-19T14:20:26.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.085+0000 7fa396d9e640 1 -- start start 2023-10-19T14:20:26.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.086+0000 7fa396d9e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa39006af00 0x7fa39013a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:26.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.086+0000 7fa396d9e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa3900ab500 0x7fa3900709a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:26.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.086+0000 7fa396d9e640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa3900c96b0 0x7fa3900aa8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:26.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.086+0000 7fa396d9e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa39013a970 con 0x7fa39006af00 2023-10-19T14:20:26.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.086+0000 7fa396d9e640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3901f1490 con 0x7fa3900ab500 2023-10-19T14:20:26.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.086+0000 7fa396d9e640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3901f1770 con 0x7fa3900c96b0 2023-10-19T14:20:26.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.086+0000 7fa394b3a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa39006af00 0x7fa39013a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:26.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.086+0000 7fa394b3a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa39006af00 0x7fa39013a430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42234/0 (socket says 172.21.15.57:42234) 2023-10-19T14:20:26.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.086+0000 7fa394b3a640 1 -- 172.21.15.57:0/2654084759 learned_addr learned my addr 172.21.15.57:0/2654084759 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:26.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.086+0000 7fa39533b640 1 --2- 172.21.15.57:0/2654084759 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa3900c96b0 0x7fa3900aa8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:26.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.086+0000 7fa394b3a640 1 -- 172.21.15.57:0/2654084759 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa3900c96b0 msgr2=0x7fa3900aa8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:26.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.086+0000 7fa38ffff640 1 --2- 172.21.15.57:0/2654084759 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa3900ab500 0x7fa3900709a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:26.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.086+0000 7fa394b3a640 1 --2- 172.21.15.57:0/2654084759 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa3900c96b0 0x7fa3900aa8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:26.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.086+0000 7fa394b3a640 1 -- 172.21.15.57:0/2654084759 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa3900ab500 msgr2=0x7fa3900709a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:26.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.086+0000 7fa394b3a640 1 --2- 172.21.15.57:0/2654084759 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa3900ab500 0x7fa3900709a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:26.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.086+0000 7fa394b3a640 1 -- 172.21.15.57:0/2654084759 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa380009560 con 0x7fa39006af00 2023-10-19T14:20:26.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.087+0000 7fa39533b640 1 --2- 172.21.15.57:0/2654084759 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa3900c96b0 0x7fa3900aa8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:20:26.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.087+0000 7fa38ffff640 1 --2- 172.21.15.57:0/2654084759 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa3900ab500 0x7fa3900709a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:20:26.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.087+0000 7fa394b3a640 1 --2- 172.21.15.57:0/2654084759 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa39006af00 0x7fa39013a430 secure :-1 s=READY pgs=2378 cs=0 l=1 rev1=1 crypto rx=0x7fa38800ea80 tx=0x7fa38800ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:26.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.087+0000 7fa38d7fa640 1 -- 172.21.15.57:0/2654084759 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa38800ce40 con 0x7fa39006af00 2023-10-19T14:20:26.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.087+0000 7fa38d7fa640 1 -- 172.21.15.57:0/2654084759 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fa388004c10 con 0x7fa39006af00 2023-10-19T14:20:26.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.087+0000 7fa396d9e640 1 -- 172.21.15.57:0/2654084759 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3901f1a50 con 0x7fa39006af00 2023-10-19T14:20:26.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.087+0000 7fa38d7fa640 1 -- 172.21.15.57:0/2654084759 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa388005020 con 0x7fa39006af00 2023-10-19T14:20:26.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.087+0000 7fa396d9e640 1 -- 172.21.15.57:0/2654084759 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3901f2010 con 0x7fa39006af00 2023-10-19T14:20:26.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.088+0000 7fa396d9e640 1 -- 172.21.15.57:0/2654084759 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa390159020 con 0x7fa39006af00 2023-10-19T14:20:26.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.088+0000 7fa38d7fa640 1 -- 172.21.15.57:0/2654084759 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fa388005180 con 0x7fa39006af00 2023-10-19T14:20:26.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.089+0000 7fa38d7fa640 1 --2- 172.21.15.57:0/2654084759 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fa37006af60 0x7fa37006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:26.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.089+0000 7fa38d7fa640 1 -- 172.21.15.57:0/2654084759 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fa38808a500 con 0x7fa39006af00 2023-10-19T14:20:26.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.089+0000 7fa38ffff640 1 --2- 172.21.15.57:0/2654084759 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fa37006af60 0x7fa37006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:26.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.090+0000 7fa38ffff640 1 --2- 172.21.15.57:0/2654084759 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fa37006af60 0x7fa37006d420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa3900cb720 tx=0x7fa378005d30 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:26.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.093+0000 7fa38d7fa640 1 -- 172.21.15.57:0/2654084759 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa388055230 con 0x7fa39006af00 2023-10-19T14:20:26.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.308+0000 7fa396d9e640 1 -- 172.21.15.57:0/2654084759 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fa3901f2230 con 0x7fa39006af00 2023-10-19T14:20:26.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.759+0000 7fa38d7fa640 1 -- 172.21.15.57:0/2654084759 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v91) v1 ==== 136+0+0 (secure 0 0 0) 0x7fa388058850 con 0x7fa39006af00 2023-10-19T14:20:26.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.762+0000 7fa396d9e640 1 -- 172.21.15.57:0/2654084759 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fa37006af60 msgr2=0x7fa37006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:26.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.762+0000 7fa396d9e640 1 --2- 172.21.15.57:0/2654084759 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fa37006af60 0x7fa37006d420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa3900cb720 tx=0x7fa378005d30 comp rx=0 tx=0).stop 2023-10-19T14:20:26.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.762+0000 7fa396d9e640 1 -- 172.21.15.57:0/2654084759 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa39006af00 msgr2=0x7fa39013a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:26.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.762+0000 7fa396d9e640 1 --2- 172.21.15.57:0/2654084759 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa39006af00 0x7fa39013a430 secure :-1 s=READY pgs=2378 cs=0 l=1 rev1=1 crypto rx=0x7fa38800ea80 tx=0x7fa38800ef40 comp rx=0 tx=0).stop 2023-10-19T14:20:26.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.762+0000 7fa396d9e640 1 -- 172.21.15.57:0/2654084759 shutdown_connections 2023-10-19T14:20:26.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.762+0000 7fa396d9e640 1 --2- 172.21.15.57:0/2654084759 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fa37006af60 0x7fa37006d420 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:26.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.762+0000 7fa396d9e640 1 --2- 172.21.15.57:0/2654084759 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa3900c96b0 0x7fa3900aa8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:26.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.762+0000 7fa396d9e640 1 --2- 172.21.15.57:0/2654084759 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa3900ab500 0x7fa3900709a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:26.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.762+0000 7fa396d9e640 1 --2- 172.21.15.57:0/2654084759 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa39006af00 0x7fa39013a430 unknown :-1 s=CLOSED pgs=2378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:26.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.762+0000 7fa396d9e640 1 -- 172.21.15.57:0/2654084759 >> 172.21.15.57:0/2654084759 conn(0x7fa390069930 msgr2=0x7fa39013ea60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:26.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.762+0000 7fa396d9e640 1 -- 172.21.15.57:0/2654084759 shutdown_connections 2023-10-19T14:20:26.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.762+0000 7fa396d9e640 1 -- 172.21.15.57:0/2654084759 wait complete. 2023-10-19T14:20:26.776 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T14:20:26.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.886+0000 7f88f6bd7640 1 -- 172.21.15.57:0/2333906577 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f88dc004d00 msgr2=0x7f88dc005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:26.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.886+0000 7f88f6bd7640 1 --2- 172.21.15.57:0/2333906577 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f88dc004d00 0x7f88dc005180 secure :-1 s=READY pgs=2380 cs=0 l=1 rev1=1 crypto rx=0x7f88e000c260 tx=0x7f88dc0057f0 comp rx=0 tx=0).stop 2023-10-19T14:20:26.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.886+0000 7f88f5174640 1 -- 172.21.15.57:0/2333906577 reap_dead start 2023-10-19T14:20:26.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.886+0000 7f88f6bd7640 1 -- 172.21.15.57:0/2333906577 shutdown_connections 2023-10-19T14:20:26.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.886+0000 7f88f6bd7640 1 -- 172.21.15.57:0/2333906577 >> 172.21.15.57:0/2333906577 conn(0x7f88f0069930 msgr2=0x7f88f0069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:26.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.886+0000 7f88f6bd7640 1 -- 172.21.15.57:0/2333906577 shutdown_connections 2023-10-19T14:20:26.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.887+0000 7f88f6bd7640 1 -- 172.21.15.57:0/2333906577 wait complete. 2023-10-19T14:20:26.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.888+0000 7f88f6bd7640 1 Processor -- start 2023-10-19T14:20:26.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.888+0000 7f88f6bd7640 1 -- start start 2023-10-19T14:20:26.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.889+0000 7f88f6bd7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f88dc004d00 0x7f88f0135500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:26.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.889+0000 7f88f6bd7640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f88f013f3a0 0x7f88f0151200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:26.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.889+0000 7f88f6bd7640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f88f0133e60 0x7f88f0134310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:26.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.889+0000 7f88f6bd7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88f0134850 con 0x7f88dc004d00 2023-10-19T14:20:26.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.889+0000 7f88f6bd7640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88f013fb40 con 0x7f88f013f3a0 2023-10-19T14:20:26.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.889+0000 7f88f6bd7640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88f013fe20 con 0x7f88f0133e60 2023-10-19T14:20:26.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.889+0000 7f88f4973640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f88dc004d00 0x7f88f0135500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:26.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.889+0000 7f88f4973640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f88dc004d00 0x7f88f0135500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:58678/0 (socket says 172.21.15.57:58678) 2023-10-19T14:20:26.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.889+0000 7f88f4973640 1 -- 172.21.15.57:0/4182526943 learned_addr learned my addr 172.21.15.57:0/4182526943 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:26.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.889+0000 7f88effff640 1 --2- 172.21.15.57:0/4182526943 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f88f013f3a0 0x7f88f0151200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:26.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.889+0000 7f88f5174640 1 --2- 172.21.15.57:0/4182526943 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f88f0133e60 0x7f88f0134310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:26.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.889+0000 7f88f4973640 1 -- 172.21.15.57:0/4182526943 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f88f0133e60 msgr2=0x7f88f0134310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:26.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.889+0000 7f88f4973640 1 --2- 172.21.15.57:0/4182526943 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f88f0133e60 0x7f88f0134310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:26.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.889+0000 7f88f4973640 1 -- 172.21.15.57:0/4182526943 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f88f013f3a0 msgr2=0x7f88f0151200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:26.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.889+0000 7f88f4973640 1 --2- 172.21.15.57:0/4182526943 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f88f013f3a0 0x7f88f0151200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:26.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.889+0000 7f88f4973640 1 -- 172.21.15.57:0/4182526943 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88f0140100 con 0x7f88dc004d00 2023-10-19T14:20:26.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.889+0000 7f88effff640 1 --2- 172.21.15.57:0/4182526943 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f88f013f3a0 0x7f88f0151200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:20:26.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.889+0000 7f88f4973640 1 --2- 172.21.15.57:0/4182526943 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f88dc004d00 0x7f88f0135500 secure :-1 s=READY pgs=2381 cs=0 l=1 rev1=1 crypto rx=0x7f88e8010bc0 tx=0x7f88e800e6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:26.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.889+0000 7f88ed7fa640 1 -- 172.21.15.57:0/4182526943 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88e8003b70 con 0x7f88dc004d00 2023-10-19T14:20:26.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.889+0000 7f88ed7fa640 1 -- 172.21.15.57:0/4182526943 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f88e8003cd0 con 0x7f88dc004d00 2023-10-19T14:20:26.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.889+0000 7f88f6bd7640 1 -- 172.21.15.57:0/4182526943 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88f01403e0 con 0x7f88dc004d00 2023-10-19T14:20:26.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.889+0000 7f88ed7fa640 1 -- 172.21.15.57:0/4182526943 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88e80042b0 con 0x7f88dc004d00 2023-10-19T14:20:26.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.889+0000 7f88f6bd7640 1 -- 172.21.15.57:0/4182526943 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88f013f820 con 0x7f88dc004d00 2023-10-19T14:20:26.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.891+0000 7f88f6bd7640 1 -- 172.21.15.57:0/4182526943 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88f00690e0 con 0x7f88dc004d00 2023-10-19T14:20:26.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.891+0000 7f88ed7fa640 1 -- 172.21.15.57:0/4182526943 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f88e8004410 con 0x7f88dc004d00 2023-10-19T14:20:26.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.891+0000 7f88ed7fa640 1 --2- 172.21.15.57:0/4182526943 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f88d006af90 0x7f88d006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:26.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.891+0000 7f88ed7fa640 1 -- 172.21.15.57:0/4182526943 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f88e808aca0 con 0x7f88dc004d00 2023-10-19T14:20:26.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.892+0000 7f88effff640 1 --2- 172.21.15.57:0/4182526943 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f88d006af90 0x7f88d006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:26.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.892+0000 7f88effff640 1 --2- 172.21.15.57:0/4182526943 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f88d006af90 0x7f88d006d450 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f88f00ae6b0 tx=0x7f88d800b260 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:26.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:26.896+0000 7f88ed7fa640 1 -- 172.21.15.57:0/4182526943 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f88e80559d0 con 0x7f88dc004d00 2023-10-19T14:20:27.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.077+0000 7f88f6bd7640 1 -- 172.21.15.57:0/4182526943 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f88f00c8c30 con 0x7f88dc004d00 2023-10-19T14:20:27.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.078+0000 7f88ed7fa640 1 -- 172.21.15.57:0/4182526943 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v223) v1 ==== 74+0+16839 (secure 0 0 0) 0x7f88e8003e60 con 0x7f88dc004d00 2023-10-19T14:20:27.081 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:20:27.081 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":223,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T14:20:20.325160+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2023-10-19T14:20:17.357433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"222","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2023-10-19T14:20:18.049516+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"223","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T14:20:27.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.080+0000 7f88f6bd7640 1 -- 172.21.15.57:0/4182526943 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f88d006af90 msgr2=0x7f88d006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:27.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.080+0000 7f88f6bd7640 1 --2- 172.21.15.57:0/4182526943 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f88d006af90 0x7f88d006d450 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f88f00ae6b0 tx=0x7f88d800b260 comp rx=0 tx=0).stop 2023-10-19T14:20:27.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.080+0000 7f88f6bd7640 1 -- 172.21.15.57:0/4182526943 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f88dc004d00 msgr2=0x7f88f0135500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:27.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.080+0000 7f88f6bd7640 1 --2- 172.21.15.57:0/4182526943 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f88dc004d00 0x7f88f0135500 secure :-1 s=READY pgs=2381 cs=0 l=1 rev1=1 crypto rx=0x7f88e8010bc0 tx=0x7f88e800e6c0 comp rx=0 tx=0).stop 2023-10-19T14:20:27.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.080+0000 7f88f6bd7640 1 -- 172.21.15.57:0/4182526943 shutdown_connections 2023-10-19T14:20:27.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.080+0000 7f88f6bd7640 1 --2- 172.21.15.57:0/4182526943 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f88d006af90 0x7f88d006d450 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:27.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.080+0000 7f88f6bd7640 1 --2- 172.21.15.57:0/4182526943 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f88f0133e60 0x7f88f0134310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:27.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.080+0000 7f88f6bd7640 1 --2- 172.21.15.57:0/4182526943 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f88f013f3a0 0x7f88f0151200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:27.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.080+0000 7f88f6bd7640 1 --2- 172.21.15.57:0/4182526943 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f88dc004d00 0x7f88f0135500 unknown :-1 s=CLOSED pgs=2381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:27.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.080+0000 7f88f6bd7640 1 -- 172.21.15.57:0/4182526943 >> 172.21.15.57:0/4182526943 conn(0x7f88f0069930 msgr2=0x7f88f00acd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:27.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.080+0000 7f88f6bd7640 1 -- 172.21.15.57:0/4182526943 shutdown_connections 2023-10-19T14:20:27.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.080+0000 7f88f6bd7640 1 -- 172.21.15.57:0/4182526943 wait complete. 2023-10-19T14:20:27.094 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-19T12:44:07.371186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 27, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-19T14:20:17.357433+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '222', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 28, 'pool_name': 'cephfs_data', 'create_time': '2023-10-19T14:20:18.049516+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '223', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-19T14:20:27.095 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-19T14:20:27.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.203+0000 7f093c96e640 1 -- 172.21.15.57:0/3234773724 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f09380afa20 msgr2=0x7f09380705a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:27.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.203+0000 7f093c96e640 1 --2- 172.21.15.57:0/3234773724 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f09380afa20 0x7f09380705a0 secure :-1 s=READY pgs=2383 cs=0 l=1 rev1=1 crypto rx=0x7f09380cc7e0 tx=0x7f092002be40 comp rx=0 tx=0).stop 2023-10-19T14:20:27.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.203+0000 7f093c96e640 1 -- 172.21.15.57:0/3234773724 shutdown_connections 2023-10-19T14:20:27.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.203+0000 7f093c96e640 1 --2- 172.21.15.57:0/3234773724 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f09380afa20 0x7f09380705a0 unknown :-1 s=CLOSED pgs=2383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:27.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.203+0000 7f093c96e640 1 --2- 172.21.15.57:0/3234773724 >> v2:172.21.15.179:3300/0 conn(0x7f09380cbd20 0x7f0938158300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:27.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.203+0000 7f093c96e640 1 -- 172.21.15.57:0/3234773724 >> 172.21.15.57:0/3234773724 conn(0x7f0938069930 msgr2=0x7f0938069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:27.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.204+0000 7f093c96e640 1 -- 172.21.15.57:0/3234773724 shutdown_connections 2023-10-19T14:20:27.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.204+0000 7f093c96e640 1 -- 172.21.15.57:0/3234773724 wait complete. 2023-10-19T14:20:27.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.205+0000 7f093c96e640 1 Processor -- start 2023-10-19T14:20:27.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.205+0000 7f093c96e640 1 -- start start 2023-10-19T14:20:27.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.206+0000 7f093c96e640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f09380afa20 0x7f09381511d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:27.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.206+0000 7f093c96e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f09380b0530 0x7f09381405b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:27.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.206+0000 7f093c96e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f09380cbd20 0x7f0938145060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:27.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.206+0000 7f093c96e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09381411c0 con 0x7f09380b0530 2023-10-19T14:20:27.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.206+0000 7f093c96e640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0938146bb0 con 0x7f09380cbd20 2023-10-19T14:20:27.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.206+0000 7f093c96e640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0938151710 con 0x7f09380afa20 2023-10-19T14:20:27.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.206+0000 7f092dd9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f09380b0530 0x7f09381405b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:27.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.206+0000 7f092dd9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f09380b0530 0x7f09381405b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:58684/0 (socket says 172.21.15.57:58684) 2023-10-19T14:20:27.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.206+0000 7f092dd9b640 1 -- 172.21.15.57:0/842568063 learned_addr learned my addr 172.21.15.57:0/842568063 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:27.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.206+0000 7f0936d9d640 1 --2- 172.21.15.57:0/842568063 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f09380cbd20 0x7f0938145060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:27.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.206+0000 7f093659c640 1 --2- 172.21.15.57:0/842568063 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f09380afa20 0x7f09381511d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:27.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.206+0000 7f092dd9b640 1 -- 172.21.15.57:0/842568063 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f09380afa20 msgr2=0x7f09381511d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:27.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.206+0000 7f092dd9b640 1 --2- 172.21.15.57:0/842568063 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f09380afa20 0x7f09381511d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:27.216 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.206+0000 7f092dd9b640 1 -- 172.21.15.57:0/842568063 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f09380cbd20 msgr2=0x7f0938145060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:27.216 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.206+0000 7f092dd9b640 1 --2- 172.21.15.57:0/842568063 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f09380cbd20 0x7f0938145060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:27.216 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.206+0000 7f092dd9b640 1 -- 172.21.15.57:0/842568063 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0920009560 con 0x7f09380b0530 2023-10-19T14:20:27.216 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.206+0000 7f0936d9d640 1 --2- 172.21.15.57:0/842568063 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f09380cbd20 0x7f0938145060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:20:27.216 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.206+0000 7f092dd9b640 1 --2- 172.21.15.57:0/842568063 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f09380b0530 0x7f09381405b0 secure :-1 s=READY pgs=2384 cs=0 l=1 rev1=1 crypto rx=0x7f0920000c00 tx=0x7f092002dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:27.217 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.206+0000 7f092f7fe640 1 -- 172.21.15.57:0/842568063 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0920003800 con 0x7f09380b0530 2023-10-19T14:20:27.217 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.206+0000 7f092f7fe640 1 -- 172.21.15.57:0/842568063 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f0920003a50 con 0x7f09380b0530 2023-10-19T14:20:27.217 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.207+0000 7f092f7fe640 1 -- 172.21.15.57:0/842568063 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0920003e60 con 0x7f09380b0530 2023-10-19T14:20:27.217 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.207+0000 7f093c96e640 1 -- 172.21.15.57:0/842568063 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09381426b0 con 0x7f09380b0530 2023-10-19T14:20:27.217 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.207+0000 7f093c96e640 1 -- 172.21.15.57:0/842568063 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0938146e20 con 0x7f09380b0530 2023-10-19T14:20:27.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.207+0000 7f093c96e640 1 -- 172.21.15.57:0/842568063 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09380690e0 con 0x7f09380b0530 2023-10-19T14:20:27.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.208+0000 7f092f7fe640 1 -- 172.21.15.57:0/842568063 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f092003e450 con 0x7f09380b0530 2023-10-19T14:20:27.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.208+0000 7f092f7fe640 1 --2- 172.21.15.57:0/842568063 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f091806af60 0x7f091806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:27.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.208+0000 7f092f7fe640 1 -- 172.21.15.57:0/842568063 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f09200ac430 con 0x7f09380b0530 2023-10-19T14:20:27.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.208+0000 7f093659c640 1 --2- 172.21.15.57:0/842568063 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f091806af60 0x7f091806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:27.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.209+0000 7f093659c640 1 --2- 172.21.15.57:0/842568063 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f091806af60 0x7f091806d420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0928005fd0 tx=0x7f0928005850 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:27.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.212+0000 7f092f7fe640 1 -- 172.21.15.57:0/842568063 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0920077390 con 0x7f09380b0530 2023-10-19T14:20:27.392 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.389+0000 7f093c96e640 1 -- 172.21.15.57:0/842568063 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0938157350 con 0x7f09380b0530 2023-10-19T14:20:27.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.390+0000 7f092f7fe640 1 -- 172.21.15.57:0/842568063 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v223) v1 ==== 93+0+10 (secure 0 0 0) 0x7f092007a9b0 con 0x7f09380b0530 2023-10-19T14:20:27.393 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-19T14:20:27.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.392+0000 7f093c96e640 1 -- 172.21.15.57:0/842568063 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f091806af60 msgr2=0x7f091806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:27.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.392+0000 7f093c96e640 1 --2- 172.21.15.57:0/842568063 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f091806af60 0x7f091806d420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0928005fd0 tx=0x7f0928005850 comp rx=0 tx=0).stop 2023-10-19T14:20:27.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.392+0000 7f093c96e640 1 -- 172.21.15.57:0/842568063 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f09380b0530 msgr2=0x7f09381405b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:27.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.392+0000 7f093c96e640 1 --2- 172.21.15.57:0/842568063 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f09380b0530 0x7f09381405b0 secure :-1 s=READY pgs=2384 cs=0 l=1 rev1=1 crypto rx=0x7f0920000c00 tx=0x7f092002dbe0 comp rx=0 tx=0).stop 2023-10-19T14:20:27.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.392+0000 7f093c96e640 1 -- 172.21.15.57:0/842568063 shutdown_connections 2023-10-19T14:20:27.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.392+0000 7f093c96e640 1 --2- 172.21.15.57:0/842568063 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f091806af60 0x7f091806d420 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:27.397 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.392+0000 7f093c96e640 1 --2- 172.21.15.57:0/842568063 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f09380cbd20 0x7f0938145060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:27.397 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.392+0000 7f093c96e640 1 --2- 172.21.15.57:0/842568063 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f09380b0530 0x7f09381405b0 unknown :-1 s=CLOSED pgs=2384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:27.397 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.392+0000 7f093c96e640 1 --2- 172.21.15.57:0/842568063 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f09380afa20 0x7f09381511d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:27.397 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.392+0000 7f093c96e640 1 -- 172.21.15.57:0/842568063 >> 172.21.15.57:0/842568063 conn(0x7f0938069930 msgr2=0x7f0938065490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:27.398 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.392+0000 7f093c96e640 1 -- 172.21.15.57:0/842568063 shutdown_connections 2023-10-19T14:20:27.398 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.392+0000 7f093c96e640 1 -- 172.21.15.57:0/842568063 wait complete. 2023-10-19T14:20:27.405 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-19T14:20:27.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.514+0000 7fc85c1c0640 1 -- 172.21.15.57:0/209692815 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc8400039a0 msgr2=0x7fc840003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:27.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.514+0000 7fc85c1c0640 1 --2- 172.21.15.57:0/209692815 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc8400039a0 0x7fc840003dc0 secure :-1 s=READY pgs=2386 cs=0 l=1 rev1=1 crypto rx=0x7fc850006fa0 tx=0x7fc8400057f0 comp rx=0 tx=0).stop 2023-10-19T14:20:27.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.514+0000 7fc85a75d640 1 -- 172.21.15.57:0/209692815 reap_dead start 2023-10-19T14:20:27.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.514+0000 7fc85c1c0640 1 -- 172.21.15.57:0/209692815 shutdown_connections 2023-10-19T14:20:27.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.514+0000 7fc85c1c0640 1 -- 172.21.15.57:0/209692815 >> 172.21.15.57:0/209692815 conn(0x7fc854069930 msgr2=0x7fc854069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:27.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.514+0000 7fc85c1c0640 1 -- 172.21.15.57:0/209692815 shutdown_connections 2023-10-19T14:20:27.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.514+0000 7fc85c1c0640 1 -- 172.21.15.57:0/209692815 wait complete. 2023-10-19T14:20:27.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.516+0000 7fc85c1c0640 1 Processor -- start 2023-10-19T14:20:27.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.516+0000 7fc85c1c0640 1 -- start start 2023-10-19T14:20:27.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.516+0000 7fc85c1c0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc8400039a0 0x7fc85413b110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:27.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.516+0000 7fc85c1c0640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc85414e1c0 0x7fc85414b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:27.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.516+0000 7fc85c1c0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc85414e640 0x7fc85414b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:27.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.516+0000 7fc85c1c0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc85413b650 con 0x7fc8400039a0 2023-10-19T14:20:27.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.516+0000 7fc85c1c0640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc85414ec00 con 0x7fc85414e640 2023-10-19T14:20:27.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.516+0000 7fc85c1c0640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc85414eee0 con 0x7fc85414e1c0 2023-10-19T14:20:27.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.516+0000 7fc859f5c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc8400039a0 0x7fc85413b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:27.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.517+0000 7fc859f5c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc8400039a0 0x7fc85413b110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:58692/0 (socket says 172.21.15.57:58692) 2023-10-19T14:20:27.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.517+0000 7fc859f5c640 1 -- 172.21.15.57:0/562709195 learned_addr learned my addr 172.21.15.57:0/562709195 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:27.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.517+0000 7fc85a75d640 1 --2- 172.21.15.57:0/562709195 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc85414e640 0x7fc85414b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:27.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.517+0000 7fc85975b640 1 --2- 172.21.15.57:0/562709195 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc85414e1c0 0x7fc85414b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:27.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.517+0000 7fc859f5c640 1 -- 172.21.15.57:0/562709195 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc85414e1c0 msgr2=0x7fc85414b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:27.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.517+0000 7fc859f5c640 1 --2- 172.21.15.57:0/562709195 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc85414e1c0 0x7fc85414b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:27.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.517+0000 7fc859f5c640 1 -- 172.21.15.57:0/562709195 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc85414e640 msgr2=0x7fc85414b640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:27.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.517+0000 7fc859f5c640 1 --2- 172.21.15.57:0/562709195 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc85414e640 0x7fc85414b640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:27.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.517+0000 7fc859f5c640 1 -- 172.21.15.57:0/562709195 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc85414f1c0 con 0x7fc8400039a0 2023-10-19T14:20:27.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.517+0000 7fc85a75d640 1 --2- 172.21.15.57:0/562709195 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc85414e640 0x7fc85414b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:20:27.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.517+0000 7fc859f5c640 1 --2- 172.21.15.57:0/562709195 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc8400039a0 0x7fc85413b110 secure :-1 s=READY pgs=2387 cs=0 l=1 rev1=1 crypto rx=0x7fc850000c00 tx=0x7fc8500016e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:27.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.517+0000 7fc84a7fc640 1 -- 172.21.15.57:0/562709195 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8500062e0 con 0x7fc8400039a0 2023-10-19T14:20:27.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.517+0000 7fc84a7fc640 1 -- 172.21.15.57:0/562709195 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fc850006440 con 0x7fc8400039a0 2023-10-19T14:20:27.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.517+0000 7fc84a7fc640 1 -- 172.21.15.57:0/562709195 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc850003850 con 0x7fc8400039a0 2023-10-19T14:20:27.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.517+0000 7fc85c1c0640 1 -- 172.21.15.57:0/562709195 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc85414f4a0 con 0x7fc8400039a0 2023-10-19T14:20:27.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.517+0000 7fc85c1c0640 1 -- 172.21.15.57:0/562709195 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8541e16a0 con 0x7fc8400039a0 2023-10-19T14:20:27.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.518+0000 7fc85c1c0640 1 -- 172.21.15.57:0/562709195 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc85414f020 con 0x7fc8400039a0 2023-10-19T14:20:27.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.519+0000 7fc84a7fc640 1 -- 172.21.15.57:0/562709195 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc850032030 con 0x7fc8400039a0 2023-10-19T14:20:27.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.519+0000 7fc84a7fc640 1 --2- 172.21.15.57:0/562709195 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fc83806af60 0x7fc83806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:27.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.519+0000 7fc84a7fc640 1 -- 172.21.15.57:0/562709195 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fc8500abe00 con 0x7fc8400039a0 2023-10-19T14:20:27.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.519+0000 7fc85975b640 1 --2- 172.21.15.57:0/562709195 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fc83806af60 0x7fc83806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:27.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.520+0000 7fc85975b640 1 --2- 172.21.15.57:0/562709195 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fc83806af60 0x7fc83806d420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc85414c0f0 tx=0x7fc844006b30 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:27.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.523+0000 7fc84a7fc640 1 -- 172.21.15.57:0/562709195 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc85414f020 con 0x7fc8400039a0 2023-10-19T14:20:27.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.697+0000 7fc85c1c0640 1 -- 172.21.15.57:0/562709195 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc85414f020 con 0x7fc8400039a0 2023-10-19T14:20:27.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.698+0000 7fc84a7fc640 1 -- 172.21.15.57:0/562709195 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v223) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc85414f020 con 0x7fc8400039a0 2023-10-19T14:20:27.701 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2023-10-19T14:20:27.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.700+0000 7fc85c1c0640 1 -- 172.21.15.57:0/562709195 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fc83806af60 msgr2=0x7fc83806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:27.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.700+0000 7fc85c1c0640 1 --2- 172.21.15.57:0/562709195 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fc83806af60 0x7fc83806d420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc85414c0f0 tx=0x7fc844006b30 comp rx=0 tx=0).stop 2023-10-19T14:20:27.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.700+0000 7fc85c1c0640 1 -- 172.21.15.57:0/562709195 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc8400039a0 msgr2=0x7fc85413b110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:27.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.700+0000 7fc85c1c0640 1 --2- 172.21.15.57:0/562709195 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc8400039a0 0x7fc85413b110 secure :-1 s=READY pgs=2387 cs=0 l=1 rev1=1 crypto rx=0x7fc850000c00 tx=0x7fc8500016e0 comp rx=0 tx=0).stop 2023-10-19T14:20:27.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.700+0000 7fc85c1c0640 1 -- 172.21.15.57:0/562709195 shutdown_connections 2023-10-19T14:20:27.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.700+0000 7fc85c1c0640 1 --2- 172.21.15.57:0/562709195 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fc83806af60 0x7fc83806d420 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:27.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.700+0000 7fc85c1c0640 1 --2- 172.21.15.57:0/562709195 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc85414e640 0x7fc85414b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:27.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.700+0000 7fc85c1c0640 1 --2- 172.21.15.57:0/562709195 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc85414e1c0 0x7fc85414b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:27.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.700+0000 7fc85c1c0640 1 --2- 172.21.15.57:0/562709195 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc8400039a0 0x7fc85413b110 unknown :-1 s=CLOSED pgs=2387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:27.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.700+0000 7fc85c1c0640 1 -- 172.21.15.57:0/562709195 >> 172.21.15.57:0/562709195 conn(0x7fc854069930 msgr2=0x7fc8540aff00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:27.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.700+0000 7fc85c1c0640 1 -- 172.21.15.57:0/562709195 shutdown_connections 2023-10-19T14:20:27.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.700+0000 7fc85c1c0640 1 -- 172.21.15.57:0/562709195 wait complete. 2023-10-19T14:20:27.713 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-19T14:20:27.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.824+0000 7f74ce0af640 1 -- 172.21.15.57:0/603245582 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f74b4004d00 msgr2=0x7f74b4005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:27.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.824+0000 7f74ce0af640 1 --2- 172.21.15.57:0/603245582 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f74b4004d00 0x7f74b4005180 secure :-1 s=READY pgs=2389 cs=0 l=1 rev1=1 crypto rx=0x7f74c806d510 tx=0x7f74b40057f0 comp rx=0 tx=0).stop 2023-10-19T14:20:27.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.824+0000 7f74c7fff640 1 -- 172.21.15.57:0/603245582 reap_dead start 2023-10-19T14:20:27.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.824+0000 7f74ce0af640 1 -- 172.21.15.57:0/603245582 shutdown_connections 2023-10-19T14:20:27.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.824+0000 7f74ce0af640 1 -- 172.21.15.57:0/603245582 >> 172.21.15.57:0/603245582 conn(0x7f74c8069930 msgr2=0x7f74c8069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:27.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.824+0000 7f74ce0af640 1 -- 172.21.15.57:0/603245582 shutdown_connections 2023-10-19T14:20:27.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.824+0000 7f74ce0af640 1 -- 172.21.15.57:0/603245582 wait complete. 2023-10-19T14:20:27.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.825+0000 7f74ce0af640 1 Processor -- start 2023-10-19T14:20:27.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.825+0000 7f74ce0af640 1 -- start start 2023-10-19T14:20:27.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.826+0000 7f74ce0af640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f74c80aae10 0x7f74c80ca870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:27.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.826+0000 7f74ce0af640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f74c80cadb0 0x7f74c80ab1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:27.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.826+0000 7f74ce0af640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f74c80ab730 0x7f74c80b07f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:27.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.826+0000 7f74ce0af640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74c81394c0 con 0x7f74c80aae10 2023-10-19T14:20:27.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.826+0000 7f74ce0af640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74c80cb210 con 0x7f74c80cadb0 2023-10-19T14:20:27.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.826+0000 7f74ce0af640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74c8139950 con 0x7f74c80ab730 2023-10-19T14:20:27.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.826+0000 7f74c77fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f74c80aae10 0x7f74c80ca870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:27.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.826+0000 7f74c77fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f74c80aae10 0x7f74c80ca870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:58706/0 (socket says 172.21.15.57:58706) 2023-10-19T14:20:27.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.826+0000 7f74c77fe640 1 -- 172.21.15.57:0/3900277850 learned_addr learned my addr 172.21.15.57:0/3900277850 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:27.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.826+0000 7f74c7fff640 1 --2- 172.21.15.57:0/3900277850 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f74c80ab730 0x7f74c80b07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:27.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.826+0000 7f74c6ffd640 1 --2- 172.21.15.57:0/3900277850 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f74c80cadb0 0x7f74c80ab1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:27.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.826+0000 7f74c77fe640 1 -- 172.21.15.57:0/3900277850 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f74c80ab730 msgr2=0x7f74c80b07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:27.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.826+0000 7f74c77fe640 1 --2- 172.21.15.57:0/3900277850 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f74c80ab730 0x7f74c80b07f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:27.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.826+0000 7f74c77fe640 1 -- 172.21.15.57:0/3900277850 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f74c80cadb0 msgr2=0x7f74c80ab1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:27.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.826+0000 7f74c77fe640 1 --2- 172.21.15.57:0/3900277850 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f74c80cadb0 0x7f74c80ab1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:27.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.826+0000 7f74c77fe640 1 -- 172.21.15.57:0/3900277850 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74c81388b0 con 0x7f74c80aae10 2023-10-19T14:20:27.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.826+0000 7f74c7fff640 1 --2- 172.21.15.57:0/3900277850 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f74c80ab730 0x7f74c80b07f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:20:27.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.826+0000 7f74c6ffd640 1 --2- 172.21.15.57:0/3900277850 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f74c80cadb0 0x7f74c80ab1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:20:27.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.827+0000 7f74c77fe640 1 --2- 172.21.15.57:0/3900277850 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f74c80aae10 0x7f74c80ca870 secure :-1 s=READY pgs=2390 cs=0 l=1 rev1=1 crypto rx=0x7f74b8006660 tx=0x7f74b8003aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:27.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.827+0000 7f74bffff640 1 -- 172.21.15.57:0/3900277850 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74b800f7e0 con 0x7f74c80aae10 2023-10-19T14:20:27.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.827+0000 7f74bffff640 1 -- 172.21.15.57:0/3900277850 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f74b800f940 con 0x7f74c80aae10 2023-10-19T14:20:27.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.827+0000 7f74ce0af640 1 -- 172.21.15.57:0/3900277850 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74c8138b90 con 0x7f74c80aae10 2023-10-19T14:20:27.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.827+0000 7f74bffff640 1 -- 172.21.15.57:0/3900277850 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74b800fd50 con 0x7f74c80aae10 2023-10-19T14:20:27.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.827+0000 7f74ce0af640 1 -- 172.21.15.57:0/3900277850 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74c8139150 con 0x7f74c80aae10 2023-10-19T14:20:27.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.828+0000 7f74ce0af640 1 -- 172.21.15.57:0/3900277850 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74c8139310 con 0x7f74c80aae10 2023-10-19T14:20:27.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.828+0000 7f74bffff640 1 -- 172.21.15.57:0/3900277850 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f74b8011430 con 0x7f74c80aae10 2023-10-19T14:20:27.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.829+0000 7f74bffff640 1 --2- 172.21.15.57:0/3900277850 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f74a806af90 0x7f74a806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:27.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.829+0000 7f74bffff640 1 -- 172.21.15.57:0/3900277850 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f74b808a7c0 con 0x7f74c80aae10 2023-10-19T14:20:27.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.829+0000 7f74c6ffd640 1 --2- 172.21.15.57:0/3900277850 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f74a806af90 0x7f74a806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:27.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.829+0000 7f74c6ffd640 1 --2- 172.21.15.57:0/3900277850 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f74a806af90 0x7f74a806d450 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f74b0006990 tx=0x7f74b0006b60 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:27.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:27.833+0000 7f74bffff640 1 -- 172.21.15.57:0/3900277850 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f74b8055500 con 0x7f74c80aae10 2023-10-19T14:20:28.011 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.008+0000 7f74ce0af640 1 -- 172.21.15.57:0/3900277850 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f74c80690e0 con 0x7f74c80aae10 2023-10-19T14:20:28.011 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.008+0000 7f74bffff640 1 -- 172.21.15.57:0/3900277850 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v223) v1 ==== 100+0+11 (secure 0 0 0) 0x7f74b8015020 con 0x7f74c80aae10 2023-10-19T14:20:28.012 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 64 2023-10-19T14:20:28.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.010+0000 7f74ce0af640 1 -- 172.21.15.57:0/3900277850 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f74a806af90 msgr2=0x7f74a806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:28.014 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.010+0000 7f74ce0af640 1 --2- 172.21.15.57:0/3900277850 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f74a806af90 0x7f74a806d450 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f74b0006990 tx=0x7f74b0006b60 comp rx=0 tx=0).stop 2023-10-19T14:20:28.014 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.010+0000 7f74ce0af640 1 -- 172.21.15.57:0/3900277850 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f74c80aae10 msgr2=0x7f74c80ca870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:28.014 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.010+0000 7f74ce0af640 1 --2- 172.21.15.57:0/3900277850 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f74c80aae10 0x7f74c80ca870 secure :-1 s=READY pgs=2390 cs=0 l=1 rev1=1 crypto rx=0x7f74b8006660 tx=0x7f74b8003aa0 comp rx=0 tx=0).stop 2023-10-19T14:20:28.014 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.010+0000 7f74ce0af640 1 -- 172.21.15.57:0/3900277850 shutdown_connections 2023-10-19T14:20:28.014 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.010+0000 7f74ce0af640 1 --2- 172.21.15.57:0/3900277850 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f74a806af90 0x7f74a806d450 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:28.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.010+0000 7f74ce0af640 1 --2- 172.21.15.57:0/3900277850 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f74c80ab730 0x7f74c80b07f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:28.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.010+0000 7f74ce0af640 1 --2- 172.21.15.57:0/3900277850 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f74c80cadb0 0x7f74c80ab1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:28.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.010+0000 7f74ce0af640 1 --2- 172.21.15.57:0/3900277850 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f74c80aae10 0x7f74c80ca870 unknown :-1 s=CLOSED pgs=2390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:28.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.010+0000 7f74ce0af640 1 -- 172.21.15.57:0/3900277850 >> 172.21.15.57:0/3900277850 conn(0x7f74c8069930 msgr2=0x7f74c80adc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:28.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.010+0000 7f74ce0af640 1 -- 172.21.15.57:0/3900277850 shutdown_connections 2023-10-19T14:20:28.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.010+0000 7f74ce0af640 1 -- 172.21.15.57:0/3900277850 wait complete. 2023-10-19T14:20:28.024 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-19T14:20:28.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.130+0000 7ffafe305640 1 -- 172.21.15.57:0/471877448 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffaf806af00 msgr2=0x7ffaf806b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:28.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.130+0000 7ffafe305640 1 --2- 172.21.15.57:0/471877448 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffaf806af00 0x7ffaf806b2e0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7ffaec009860 tx=0x7ffaec02bd60 comp rx=0 tx=0).stop 2023-10-19T14:20:28.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.130+0000 7ffafe305640 1 -- 172.21.15.57:0/471877448 shutdown_connections 2023-10-19T14:20:28.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.130+0000 7ffafe305640 1 --2- 172.21.15.57:0/471877448 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ffaf80abc10 0x7ffaf80c9b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:28.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.130+0000 7ffafe305640 1 --2- 172.21.15.57:0/471877448 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffaf806af00 0x7ffaf806b2e0 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:28.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.130+0000 7ffafe305640 1 -- 172.21.15.57:0/471877448 >> 172.21.15.57:0/471877448 conn(0x7ffaf8069930 msgr2=0x7ffaf8069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:28.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.130+0000 7ffafe305640 1 -- 172.21.15.57:0/471877448 shutdown_connections 2023-10-19T14:20:28.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.130+0000 7ffafe305640 1 -- 172.21.15.57:0/471877448 wait complete. 2023-10-19T14:20:28.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.131+0000 7ffafe305640 1 Processor -- start 2023-10-19T14:20:28.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.132+0000 7ffafe305640 1 -- start start 2023-10-19T14:20:28.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.132+0000 7ffafe305640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ffaf806af00 0x7ffaf8151d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:28.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.132+0000 7ffafe305640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffaf80abc10 0x7ffaf80aa8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:28.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.132+0000 7ffafe305640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffaf80c96b0 0x7ffaf80aae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:28.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.132+0000 7ffafe305640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf8070a10 con 0x7ffaf80abc10 2023-10-19T14:20:28.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.132+0000 7ffafe305640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf806fd90 con 0x7ffaf80c96b0 2023-10-19T14:20:28.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.132+0000 7ffafe305640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf8070070 con 0x7ffaf806af00 2023-10-19T14:20:28.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.132+0000 7ffaf77fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffaf80abc10 0x7ffaf80aa8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:28.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.132+0000 7ffaf77fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffaf80abc10 0x7ffaf80aa8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:58710/0 (socket says 172.21.15.57:58710) 2023-10-19T14:20:28.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.132+0000 7ffaf77fe640 1 -- 172.21.15.57:0/1286295698 learned_addr learned my addr 172.21.15.57:0/1286295698 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:28.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.133+0000 7ffafc8a2640 1 --2- 172.21.15.57:0/1286295698 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffaf80c96b0 0x7ffaf80aae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:28.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.133+0000 7ffaf7fff640 1 --2- 172.21.15.57:0/1286295698 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ffaf806af00 0x7ffaf8151d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:28.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.133+0000 7ffaf77fe640 1 -- 172.21.15.57:0/1286295698 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ffaf806af00 msgr2=0x7ffaf8151d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:28.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.133+0000 7ffaf77fe640 1 --2- 172.21.15.57:0/1286295698 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ffaf806af00 0x7ffaf8151d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:28.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.133+0000 7ffaf77fe640 1 -- 172.21.15.57:0/1286295698 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffaf80c96b0 msgr2=0x7ffaf80aae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:28.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.133+0000 7ffaf77fe640 1 --2- 172.21.15.57:0/1286295698 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffaf80c96b0 0x7ffaf80aae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:28.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.133+0000 7ffaf77fe640 1 -- 172.21.15.57:0/1286295698 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaf0009840 con 0x7ffaf80abc10 2023-10-19T14:20:28.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.133+0000 7ffaf77fe640 1 --2- 172.21.15.57:0/1286295698 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffaf80abc10 0x7ffaf80aa8c0 secure :-1 s=READY pgs=2391 cs=0 l=1 rev1=1 crypto rx=0x7ffae800c410 tx=0x7ffae800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:28.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.133+0000 7ffaf4ff9640 1 -- 172.21.15.57:0/1286295698 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffae80035f0 con 0x7ffaf80abc10 2023-10-19T14:20:28.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.133+0000 7ffaf4ff9640 1 -- 172.21.15.57:0/1286295698 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7ffae8003770 con 0x7ffaf80abc10 2023-10-19T14:20:28.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.133+0000 7ffafe305640 1 -- 172.21.15.57:0/1286295698 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffaec009490 con 0x7ffaf80abc10 2023-10-19T14:20:28.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.133+0000 7ffaf4ff9640 1 -- 172.21.15.57:0/1286295698 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffae8003b80 con 0x7ffaf80abc10 2023-10-19T14:20:28.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.133+0000 7ffafe305640 1 -- 172.21.15.57:0/1286295698 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffaf8070630 con 0x7ffaf80abc10 2023-10-19T14:20:28.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.134+0000 7ffafe305640 1 -- 172.21.15.57:0/1286295698 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffaf8070b50 con 0x7ffaf80abc10 2023-10-19T14:20:28.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.135+0000 7ffaf4ff9640 1 -- 172.21.15.57:0/1286295698 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ffae8003ce0 con 0x7ffaf80abc10 2023-10-19T14:20:28.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.135+0000 7ffaf4ff9640 1 --2- 172.21.15.57:0/1286295698 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7ffae406af60 0x7ffae406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:28.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.135+0000 7ffaf4ff9640 1 -- 172.21.15.57:0/1286295698 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 6057+0+0 (secure 0 0 0) 0x7ffae80890a0 con 0x7ffaf80abc10 2023-10-19T14:20:28.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.135+0000 7ffaf7fff640 1 --2- 172.21.15.57:0/1286295698 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7ffae406af60 0x7ffae406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:28.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.136+0000 7ffaf7fff640 1 --2- 172.21.15.57:0/1286295698 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7ffae406af60 0x7ffae406d420 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ffaec036000 tx=0x7ffaec004730 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:28.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.139+0000 7ffaf4ff9640 1 -- 172.21.15.57:0/1286295698 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ffaf8070b50 con 0x7ffaf80abc10 2023-10-19T14:20:28.321 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.318+0000 7ffafe305640 1 -- 172.21.15.57:0/1286295698 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7ffaf8070b50 con 0x7ffaf80abc10 2023-10-19T14:20:28.321 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.318+0000 7ffaf4ff9640 1 -- 172.21.15.57:0/1286295698 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v223) v1 ==== 84+0+209 (secure 0 0 0) 0x7ffaf8070b50 con 0x7ffaf80abc10 2023-10-19T14:20:28.321 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:20:28.322 INFO:teuthology.orchestra.run.smithi057.stdout:[ 2023-10-19T14:20:28.322 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:20:28.322 INFO:teuthology.orchestra.run.smithi057.stdout: "poolnum": 1, 2023-10-19T14:20:28.322 INFO:teuthology.orchestra.run.smithi057.stdout: "poolname": ".mgr" 2023-10-19T14:20:28.323 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:20:28.323 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:20:28.323 INFO:teuthology.orchestra.run.smithi057.stdout: "poolnum": 27, 2023-10-19T14:20:28.323 INFO:teuthology.orchestra.run.smithi057.stdout: "poolname": "cephfs_metadata" 2023-10-19T14:20:28.323 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:20:28.323 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:20:28.324 INFO:teuthology.orchestra.run.smithi057.stdout: "poolnum": 28, 2023-10-19T14:20:28.324 INFO:teuthology.orchestra.run.smithi057.stdout: "poolname": "cephfs_data" 2023-10-19T14:20:28.324 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:20:28.324 INFO:teuthology.orchestra.run.smithi057.stdout:] 2023-10-19T14:20:28.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.320+0000 7ffafe305640 1 -- 172.21.15.57:0/1286295698 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7ffae406af60 msgr2=0x7ffae406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:28.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.320+0000 7ffafe305640 1 --2- 172.21.15.57:0/1286295698 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7ffae406af60 0x7ffae406d420 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ffaec036000 tx=0x7ffaec004730 comp rx=0 tx=0).stop 2023-10-19T14:20:28.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.320+0000 7ffafe305640 1 -- 172.21.15.57:0/1286295698 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffaf80abc10 msgr2=0x7ffaf80aa8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:28.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.320+0000 7ffafe305640 1 --2- 172.21.15.57:0/1286295698 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffaf80abc10 0x7ffaf80aa8c0 secure :-1 s=READY pgs=2391 cs=0 l=1 rev1=1 crypto rx=0x7ffae800c410 tx=0x7ffae800c8d0 comp rx=0 tx=0).stop 2023-10-19T14:20:28.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.320+0000 7ffafe305640 1 -- 172.21.15.57:0/1286295698 shutdown_connections 2023-10-19T14:20:28.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.320+0000 7ffafe305640 1 --2- 172.21.15.57:0/1286295698 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7ffae406af60 0x7ffae406d420 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:28.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.320+0000 7ffafe305640 1 --2- 172.21.15.57:0/1286295698 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffaf80c96b0 0x7ffaf80aae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:28.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.320+0000 7ffafe305640 1 --2- 172.21.15.57:0/1286295698 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffaf80abc10 0x7ffaf80aa8c0 unknown :-1 s=CLOSED pgs=2391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:28.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.320+0000 7ffafe305640 1 --2- 172.21.15.57:0/1286295698 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ffaf806af00 0x7ffaf8151d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:28.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.320+0000 7ffafe305640 1 -- 172.21.15.57:0/1286295698 >> 172.21.15.57:0/1286295698 conn(0x7ffaf8069930 msgr2=0x7ffaf80cb0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:28.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.320+0000 7ffafe305640 1 -- 172.21.15.57:0/1286295698 shutdown_connections 2023-10-19T14:20:28.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.320+0000 7ffafe305640 1 -- 172.21.15.57:0/1286295698 wait complete. 2023-10-19T14:20:28.334 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'backup_fs' 2023-10-19T14:20:28.334 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create backup_fs_metadata --pg_num_min 64 2023-10-19T14:20:28.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.446+0000 7f20c3fb6640 1 -- 172.21.15.57:0/3384954559 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f20a8005000 msgr2=0x7f20a80058d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:28.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.446+0000 7f20c3fb6640 1 --2- 172.21.15.57:0/3384954559 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f20a8005000 0x7f20a80058d0 secure :-1 s=READY pgs=2393 cs=0 l=1 rev1=1 crypto rx=0x7f20ac009b50 tx=0x7f20a8006900 comp rx=0 tx=0).stop 2023-10-19T14:20:28.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.446+0000 7f20c2553640 1 -- 172.21.15.57:0/3384954559 reap_dead start 2023-10-19T14:20:28.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.446+0000 7f20c3fb6640 1 -- 172.21.15.57:0/3384954559 shutdown_connections 2023-10-19T14:20:28.452 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.446+0000 7f20c3fb6640 1 -- 172.21.15.57:0/3384954559 >> 172.21.15.57:0/3384954559 conn(0x7f20bc069930 msgr2=0x7f20bc069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:28.452 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.446+0000 7f20c3fb6640 1 -- 172.21.15.57:0/3384954559 shutdown_connections 2023-10-19T14:20:28.452 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.446+0000 7f20c3fb6640 1 -- 172.21.15.57:0/3384954559 wait complete. 2023-10-19T14:20:28.452 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.447+0000 7f20c3fb6640 1 Processor -- start 2023-10-19T14:20:28.452 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.448+0000 7f20c3fb6640 1 -- start start 2023-10-19T14:20:28.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.448+0000 7f20c3fb6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f20a8005000 0x7f20bc13ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:28.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.448+0000 7f20c3fb6640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f20bc12caa0 0x7f20bc12cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:28.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.448+0000 7f20c3fb6640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f20bc12dae0 0x7f20bc12df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:28.456 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.448+0000 7f20c3fb6640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20bc12f460 con 0x7f20a8005000 2023-10-19T14:20:28.456 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.448+0000 7f20c3fb6640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20bc12d460 con 0x7f20bc12caa0 2023-10-19T14:20:28.456 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.448+0000 7f20c3fb6640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20bc12e850 con 0x7f20bc12dae0 2023-10-19T14:20:28.456 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.448+0000 7f20c1d52640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f20a8005000 0x7f20bc13ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:28.456 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.448+0000 7f20c1d52640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f20a8005000 0x7f20bc13ef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:58732/0 (socket says 172.21.15.57:58732) 2023-10-19T14:20:28.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.448+0000 7f20c1d52640 1 -- 172.21.15.57:0/71057262 learned_addr learned my addr 172.21.15.57:0/71057262 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:28.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.449+0000 7f20c2553640 1 --2- 172.21.15.57:0/71057262 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f20bc12dae0 0x7f20bc12df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:28.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.449+0000 7f20c1551640 1 --2- 172.21.15.57:0/71057262 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f20bc12caa0 0x7f20bc12cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:28.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.449+0000 7f20c1d52640 1 -- 172.21.15.57:0/71057262 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f20bc12dae0 msgr2=0x7f20bc12df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:28.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.449+0000 7f20c1d52640 1 --2- 172.21.15.57:0/71057262 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f20bc12dae0 0x7f20bc12df60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:28.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.449+0000 7f20c1d52640 1 -- 172.21.15.57:0/71057262 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f20bc12caa0 msgr2=0x7f20bc12cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:28.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.449+0000 7f20c1d52640 1 --2- 172.21.15.57:0/71057262 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f20bc12caa0 0x7f20bc12cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:28.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.449+0000 7f20c1d52640 1 -- 172.21.15.57:0/71057262 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20bc12eb30 con 0x7f20a8005000 2023-10-19T14:20:28.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.449+0000 7f20c1d52640 1 --2- 172.21.15.57:0/71057262 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f20a8005000 0x7f20bc13ef10 secure :-1 s=READY pgs=2394 cs=0 l=1 rev1=1 crypto rx=0x7f20b8011a50 tx=0x7f20b8011f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:28.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.449+0000 7f20b27fc640 1 -- 172.21.15.57:0/71057262 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20b800fd90 con 0x7f20a8005000 2023-10-19T14:20:28.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.449+0000 7f20b27fc640 1 -- 172.21.15.57:0/71057262 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f20b8012450 con 0x7f20a8005000 2023-10-19T14:20:28.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.449+0000 7f20c3fb6640 1 -- 172.21.15.57:0/71057262 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20bc12ee10 con 0x7f20a8005000 2023-10-19T14:20:28.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.449+0000 7f20b27fc640 1 -- 172.21.15.57:0/71057262 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20b8012800 con 0x7f20a8005000 2023-10-19T14:20:28.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.449+0000 7f20c3fb6640 1 -- 172.21.15.57:0/71057262 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20bc12e4a0 con 0x7f20a8005000 2023-10-19T14:20:28.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.450+0000 7f20c3fb6640 1 -- 172.21.15.57:0/71057262 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20bc0690e0 con 0x7f20a8005000 2023-10-19T14:20:28.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.451+0000 7f20b27fc640 1 -- 172.21.15.57:0/71057262 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f20b8012960 con 0x7f20a8005000 2023-10-19T14:20:28.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.451+0000 7f20b27fc640 1 --2- 172.21.15.57:0/71057262 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f20a006af90 0x7f20a006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:28.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.451+0000 7f20b27fc640 1 -- 172.21.15.57:0/71057262 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f20b808a700 con 0x7f20a8005000 2023-10-19T14:20:28.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.451+0000 7f20c1551640 1 --2- 172.21.15.57:0/71057262 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f20a006af90 0x7f20a006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:28.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.452+0000 7f20c1551640 1 --2- 172.21.15.57:0/71057262 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f20a006af90 0x7f20a006d450 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f20ac036000 tx=0x7f20ac035040 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:28.461 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.455+0000 7f20b27fc640 1 -- 172.21.15.57:0/71057262 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f20b80555f0 con 0x7f20a8005000 2023-10-19T14:20:28.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.632+0000 7f20c3fb6640 1 -- 172.21.15.57:0/71057262 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "backup_fs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f20bc1402b0 con 0x7f20a8005000 2023-10-19T14:20:28.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.769+0000 7f20b27fc640 1 -- 172.21.15.57:0/71057262 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "backup_fs_metadata", "pg_num_min": 64}]=0 pool 'backup_fs_metadata' created v224) v1 ==== 144+0+0 (secure 0 0 0) 0x7f20b8058c10 con 0x7f20a8005000 2023-10-19T14:20:28.772 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'backup_fs_metadata' created 2023-10-19T14:20:28.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.771+0000 7f20c3fb6640 1 -- 172.21.15.57:0/71057262 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f20a006af90 msgr2=0x7f20a006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:28.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.771+0000 7f20c3fb6640 1 --2- 172.21.15.57:0/71057262 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f20a006af90 0x7f20a006d450 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f20ac036000 tx=0x7f20ac035040 comp rx=0 tx=0).stop 2023-10-19T14:20:28.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.771+0000 7f20c3fb6640 1 -- 172.21.15.57:0/71057262 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f20a8005000 msgr2=0x7f20bc13ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:28.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.771+0000 7f20c3fb6640 1 --2- 172.21.15.57:0/71057262 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f20a8005000 0x7f20bc13ef10 secure :-1 s=READY pgs=2394 cs=0 l=1 rev1=1 crypto rx=0x7f20b8011a50 tx=0x7f20b8011f10 comp rx=0 tx=0).stop 2023-10-19T14:20:28.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.771+0000 7f20c3fb6640 1 -- 172.21.15.57:0/71057262 shutdown_connections 2023-10-19T14:20:28.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.771+0000 7f20c3fb6640 1 --2- 172.21.15.57:0/71057262 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f20a006af90 0x7f20a006d450 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:28.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.771+0000 7f20c3fb6640 1 --2- 172.21.15.57:0/71057262 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f20bc12dae0 0x7f20bc12df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:28.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.771+0000 7f20c3fb6640 1 --2- 172.21.15.57:0/71057262 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f20bc12caa0 0x7f20bc12cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:28.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.771+0000 7f20c3fb6640 1 --2- 172.21.15.57:0/71057262 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f20a8005000 0x7f20bc13ef10 unknown :-1 s=CLOSED pgs=2394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:28.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.771+0000 7f20c3fb6640 1 -- 172.21.15.57:0/71057262 >> 172.21.15.57:0/71057262 conn(0x7f20bc069930 msgr2=0x7f20bc13c1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:28.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.771+0000 7f20c3fb6640 1 -- 172.21.15.57:0/71057262 shutdown_connections 2023-10-19T14:20:28.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.771+0000 7f20c3fb6640 1 -- 172.21.15.57:0/71057262 wait complete. 2023-10-19T14:20:28.788 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create backup_fs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-19T14:20:28.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.902+0000 7fa7cf339640 1 -- 172.21.15.57:0/159929785 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa7b4005180 msgr2=0x7fa7b400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:28.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.902+0000 7fa7cf339640 1 --2- 172.21.15.57:0/159929785 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa7b4005180 0x7fa7b400b980 secure :-1 s=READY pgs=2396 cs=0 l=1 rev1=1 crypto rx=0x7fa7c0009ed0 tx=0x7fa7b40057f0 comp rx=0 tx=0).stop 2023-10-19T14:20:28.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.902+0000 7fa7cf339640 1 -- 172.21.15.57:0/159929785 shutdown_connections 2023-10-19T14:20:28.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.902+0000 7fa7cf339640 1 --2- 172.21.15.57:0/159929785 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa7b4005180 0x7fa7b400b980 unknown :-1 s=CLOSED pgs=2396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:28.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.902+0000 7fa7cf339640 1 -- 172.21.15.57:0/159929785 >> 172.21.15.57:0/159929785 conn(0x7fa7c80aec40 msgr2=0x7fa7c80af050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:28.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.903+0000 7fa7cf339640 1 -- 172.21.15.57:0/159929785 shutdown_connections 2023-10-19T14:20:28.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.903+0000 7fa7cf339640 1 -- 172.21.15.57:0/159929785 wait complete. 2023-10-19T14:20:28.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.904+0000 7fa7cf339640 1 Processor -- start 2023-10-19T14:20:28.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.904+0000 7fa7cf339640 1 -- start start 2023-10-19T14:20:28.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.904+0000 7fa7cf339640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa7b4005180 0x7fa7c81f2810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:28.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.904+0000 7fa7cf339640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa7c814dd60 0x7fa7c80adc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:28.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.904+0000 7fa7cf339640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa7c80ae160 0x7fa7c815a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:28.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.905+0000 7fa7cf339640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c80ae5e0 con 0x7fa7c80ae160 2023-10-19T14:20:28.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.905+0000 7fa7cf339640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c806b890 con 0x7fa7b4005180 2023-10-19T14:20:28.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.905+0000 7fa7cf339640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c806bb70 con 0x7fa7c814dd60 2023-10-19T14:20:28.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.905+0000 7fa7cd8d6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa7c80ae160 0x7fa7c815a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:28.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.905+0000 7fa7cd8d6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa7c80ae160 0x7fa7c815a260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:58738/0 (socket says 172.21.15.57:58738) 2023-10-19T14:20:28.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.905+0000 7fa7cd8d6640 1 -- 172.21.15.57:0/4226037526 learned_addr learned my addr 172.21.15.57:0/4226037526 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:28.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.905+0000 7fa7cd0d5640 1 --2- 172.21.15.57:0/4226037526 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa7b4005180 0x7fa7c81f2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:28.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.905+0000 7fa7cc8d4640 1 --2- 172.21.15.57:0/4226037526 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa7c814dd60 0x7fa7c80adc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:28.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.905+0000 7fa7cd8d6640 1 -- 172.21.15.57:0/4226037526 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa7c814dd60 msgr2=0x7fa7c80adc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:28.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.905+0000 7fa7cd8d6640 1 --2- 172.21.15.57:0/4226037526 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa7c814dd60 0x7fa7c80adc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:28.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.905+0000 7fa7cd8d6640 1 -- 172.21.15.57:0/4226037526 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa7b4005180 msgr2=0x7fa7c81f2810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:28.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.905+0000 7fa7cd8d6640 1 --2- 172.21.15.57:0/4226037526 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa7b4005180 0x7fa7c81f2810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:28.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.905+0000 7fa7cd8d6640 1 -- 172.21.15.57:0/4226037526 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7c806be50 con 0x7fa7c80ae160 2023-10-19T14:20:28.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.905+0000 7fa7cd8d6640 1 --2- 172.21.15.57:0/4226037526 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa7c80ae160 0x7fa7c815a260 secure :-1 s=READY pgs=2397 cs=0 l=1 rev1=1 crypto rx=0x7fa7c0009e20 tx=0x7fa7c0003b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:28.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.905+0000 7fa7bdffb640 1 -- 172.21.15.57:0/4226037526 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7c003d070 con 0x7fa7c80ae160 2023-10-19T14:20:28.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.905+0000 7fa7bdffb640 1 -- 172.21.15.57:0/4226037526 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fa7c00043f0 con 0x7fa7c80ae160 2023-10-19T14:20:28.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.905+0000 7fa7cf339640 1 -- 172.21.15.57:0/4226037526 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7c806c130 con 0x7fa7c80ae160 2023-10-19T14:20:28.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.905+0000 7fa7bdffb640 1 -- 172.21.15.57:0/4226037526 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7c0004780 con 0x7fa7c80ae160 2023-10-19T14:20:28.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.906+0000 7fa7cf339640 1 -- 172.21.15.57:0/4226037526 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7c814e1c0 con 0x7fa7c80ae160 2023-10-19T14:20:28.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.906+0000 7fa7cf339640 1 -- 172.21.15.57:0/4226037526 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7c806c650 con 0x7fa7c80ae160 2023-10-19T14:20:28.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.908+0000 7fa7bdffb640 1 -- 172.21.15.57:0/4226037526 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fa7c00304c0 con 0x7fa7c80ae160 2023-10-19T14:20:28.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.908+0000 7fa7bdffb640 1 --2- 172.21.15.57:0/4226037526 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fa7ac06af90 0x7fa7ac06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:28.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.908+0000 7fa7bdffb640 1 -- 172.21.15.57:0/4226037526 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(224..224 src has 1..224) v4 ==== 6430+0+0 (secure 0 0 0) 0x7fa7c00abe60 con 0x7fa7c80ae160 2023-10-19T14:20:28.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.908+0000 7fa7cd0d5640 1 --2- 172.21.15.57:0/4226037526 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fa7ac06af90 0x7fa7ac06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:28.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.909+0000 7fa7cd0d5640 1 --2- 172.21.15.57:0/4226037526 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fa7ac06af90 0x7fa7ac06d450 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa7c40017a0 tx=0x7fa7c4006040 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:28.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:28.912+0000 7fa7bdffb640 1 -- 172.21.15.57:0/4226037526 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa7c0076bf0 con 0x7fa7c80ae160 2023-10-19T14:20:29.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:29.088+0000 7fa7cf339640 1 -- 172.21.15.57:0/4226037526 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "backup_fs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fa7c806c650 con 0x7fa7c80ae160 2023-10-19T14:20:29.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:29.774+0000 7fa7bdffb640 1 -- 172.21.15.57:0/4226037526 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "backup_fs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'backup_fs_data' created v225) v1 ==== 176+0+0 (secure 0 0 0) 0x7fa7c806c650 con 0x7fa7c80ae160 2023-10-19T14:20:29.778 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'backup_fs_data' created 2023-10-19T14:20:29.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:29.778+0000 7fa7cf339640 1 -- 172.21.15.57:0/4226037526 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fa7ac06af90 msgr2=0x7fa7ac06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:29.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:29.778+0000 7fa7cf339640 1 --2- 172.21.15.57:0/4226037526 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fa7ac06af90 0x7fa7ac06d450 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa7c40017a0 tx=0x7fa7c4006040 comp rx=0 tx=0).stop 2023-10-19T14:20:29.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:29.778+0000 7fa7cf339640 1 -- 172.21.15.57:0/4226037526 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa7c80ae160 msgr2=0x7fa7c815a260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:29.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:29.778+0000 7fa7cf339640 1 --2- 172.21.15.57:0/4226037526 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa7c80ae160 0x7fa7c815a260 secure :-1 s=READY pgs=2397 cs=0 l=1 rev1=1 crypto rx=0x7fa7c0009e20 tx=0x7fa7c0003b80 comp rx=0 tx=0).stop 2023-10-19T14:20:29.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:29.778+0000 7fa7cf339640 1 -- 172.21.15.57:0/4226037526 shutdown_connections 2023-10-19T14:20:29.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:29.778+0000 7fa7cf339640 1 --2- 172.21.15.57:0/4226037526 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fa7ac06af90 0x7fa7ac06d450 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:29.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:29.778+0000 7fa7cf339640 1 --2- 172.21.15.57:0/4226037526 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa7c80ae160 0x7fa7c815a260 unknown :-1 s=CLOSED pgs=2397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:29.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:29.778+0000 7fa7cf339640 1 --2- 172.21.15.57:0/4226037526 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa7c814dd60 0x7fa7c80adc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:29.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:29.778+0000 7fa7cf339640 1 --2- 172.21.15.57:0/4226037526 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa7b4005180 0x7fa7c81f2810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:29.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:29.778+0000 7fa7cf339640 1 -- 172.21.15.57:0/4226037526 >> 172.21.15.57:0/4226037526 conn(0x7fa7c80aec40 msgr2=0x7fa7c80af020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:29.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:29.779+0000 7fa7cf339640 1 -- 172.21.15.57:0/4226037526 shutdown_connections 2023-10-19T14:20:29.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:29.779+0000 7fa7cf339640 1 -- 172.21.15.57:0/4226037526 wait complete. 2023-10-19T14:20:29.808 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new backup_fs backup_fs_metadata backup_fs_data 2023-10-19T14:20:29.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:29.922+0000 7effb0118640 1 -- 172.21.15.57:0/3416084183 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7effa8070350 msgr2=0x7effa80707c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:29.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:29.922+0000 7effb0118640 1 --2- 172.21.15.57:0/3416084183 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7effa8070350 0x7effa80707c0 secure :-1 s=READY pgs=2398 cs=0 l=1 rev1=1 crypto rx=0x7effa813e2f0 tx=0x7eff9802be40 comp rx=0 tx=0).stop 2023-10-19T14:20:29.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:29.922+0000 7effb0118640 1 -- 172.21.15.57:0/3416084183 shutdown_connections 2023-10-19T14:20:29.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:29.922+0000 7effb0118640 1 --2- 172.21.15.57:0/3416084183 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7effa8070350 0x7effa80707c0 unknown :-1 s=CLOSED pgs=2398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:29.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:29.922+0000 7effb0118640 1 --2- 172.21.15.57:0/3416084183 >> v2:172.21.15.179:3300/0 conn(0x7effa813d350 0x7effa80cbd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:29.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:29.922+0000 7effb0118640 1 -- 172.21.15.57:0/3416084183 >> 172.21.15.57:0/3416084183 conn(0x7effa8069930 msgr2=0x7effa8069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:29.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:29.922+0000 7effb0118640 1 -- 172.21.15.57:0/3416084183 shutdown_connections 2023-10-19T14:20:29.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:29.922+0000 7effb0118640 1 -- 172.21.15.57:0/3416084183 wait complete. 2023-10-19T14:20:29.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:29.924+0000 7effb0118640 1 Processor -- start 2023-10-19T14:20:29.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:29.924+0000 7effb0118640 1 -- start start 2023-10-19T14:20:29.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:29.924+0000 7effb0118640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7effa8070350 0x7effa8153ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:29.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:29.924+0000 7effb0118640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7effa80cb980 0x7effa8135e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:29.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:29.924+0000 7effb0118640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7effa81f3820 0x7effa8136340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:29.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:29.924+0000 7effb0118640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effa81f2af0 con 0x7effa81f3820 2023-10-19T14:20:29.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:29.924+0000 7effb0118640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effa80b1430 con 0x7effa80cb980 2023-10-19T14:20:29.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:29.924+0000 7effb0118640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effa81f2d40 con 0x7effa8070350 2023-10-19T14:20:29.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:29.925+0000 7effae6b5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7effa81f3820 0x7effa8136340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:29.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:29.925+0000 7effae6b5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7effa81f3820 0x7effa8136340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:58754/0 (socket says 172.21.15.57:58754) 2023-10-19T14:20:29.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:29.925+0000 7effae6b5640 1 -- 172.21.15.57:0/445473836 learned_addr learned my addr 172.21.15.57:0/445473836 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:29.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:29.925+0000 7effadeb4640 1 --2- 172.21.15.57:0/445473836 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7effa8070350 0x7effa8153ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:29.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:29.925+0000 7effad6b3640 1 --2- 172.21.15.57:0/445473836 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7effa80cb980 0x7effa8135e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:29.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:29.925+0000 7effae6b5640 1 -- 172.21.15.57:0/445473836 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7effa8070350 msgr2=0x7effa8153ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:29.932 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:29.925+0000 7effae6b5640 1 --2- 172.21.15.57:0/445473836 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7effa8070350 0x7effa8153ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:29.932 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:29.925+0000 7effae6b5640 1 -- 172.21.15.57:0/445473836 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7effa80cb980 msgr2=0x7effa8135e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:29.932 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:29.925+0000 7effae6b5640 1 --2- 172.21.15.57:0/445473836 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7effa80cb980 0x7effa8135e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:29.932 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:29.925+0000 7effae6b5640 1 -- 172.21.15.57:0/445473836 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff98009560 con 0x7effa81f3820 2023-10-19T14:20:29.932 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:29.925+0000 7effae6b5640 1 --2- 172.21.15.57:0/445473836 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7effa81f3820 0x7effa8136340 secure :-1 s=READY pgs=2399 cs=0 l=1 rev1=1 crypto rx=0x7eff98036040 tx=0x7eff9802db70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:29.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:29.925+0000 7effa67fc640 1 -- 172.21.15.57:0/445473836 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff98003800 con 0x7effa81f3820 2023-10-19T14:20:29.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:29.925+0000 7effa67fc640 1 -- 172.21.15.57:0/445473836 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7eff98003a50 con 0x7effa81f3820 2023-10-19T14:20:29.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:29.925+0000 7effb0118640 1 -- 172.21.15.57:0/445473836 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effa81f3020 con 0x7effa81f3820 2023-10-19T14:20:29.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:29.925+0000 7effa67fc640 1 -- 172.21.15.57:0/445473836 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff98003ea0 con 0x7effa81f3820 2023-10-19T14:20:29.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:29.925+0000 7effb0118640 1 -- 172.21.15.57:0/445473836 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effa80b0700 con 0x7effa81f3820 2023-10-19T14:20:29.934 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:29.926+0000 7effb0118640 1 -- 172.21.15.57:0/445473836 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effa81f3540 con 0x7effa81f3820 2023-10-19T14:20:29.935 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:29.927+0000 7effa67fc640 1 -- 172.21.15.57:0/445473836 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7eff9803e450 con 0x7effa81f3820 2023-10-19T14:20:29.935 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:29.928+0000 7effa67fc640 1 --2- 172.21.15.57:0/445473836 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7eff9406af60 0x7eff9406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:29.935 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:29.928+0000 7effa67fc640 1 -- 172.21.15.57:0/445473836 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(225..225 src has 1..225) v4 ==== 6815+0+0 (secure 0 0 0) 0x7eff980ac870 con 0x7effa81f3820 2023-10-19T14:20:29.935 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:29.928+0000 7effadeb4640 1 --2- 172.21.15.57:0/445473836 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7eff9406af60 0x7eff9406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:29.935 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:29.929+0000 7effadeb4640 1 --2- 172.21.15.57:0/445473836 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7eff9406af60 0x7eff9406d420 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7effa0001f10 tx=0x7effa0009040 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:29.936 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:29.932+0000 7effa67fc640 1 -- 172.21.15.57:0/445473836 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7eff980774e0 con 0x7effa81f3820 2023-10-19T14:20:30.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:30.143+0000 7effb0118640 1 -- 172.21.15.57:0/445473836 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "backup_fs", "metadata": "backup_fs_metadata", "data": "backup_fs_data"} v 0) v1 -- 0x7effa81f3540 con 0x7effa81f3820 2023-10-19T14:20:30.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:30.800+0000 7effa67fc640 1 -- 172.21.15.57:0/445473836 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "backup_fs", "metadata": "backup_fs_metadata", "data": "backup_fs_data"}]=0 new fs with metadata pool 29 and data pool 30 v92) v1 ==== 183+0+0 (secure 0 0 0) 0x7effa81f3540 con 0x7effa81f3820 2023-10-19T14:20:30.807 INFO:teuthology.orchestra.run.smithi057.stderr:new fs with metadata pool 29 and data pool 30 2023-10-19T14:20:30.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:30.807+0000 7effb0118640 1 -- 172.21.15.57:0/445473836 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7eff9406af60 msgr2=0x7eff9406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:30.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:30.807+0000 7effb0118640 1 --2- 172.21.15.57:0/445473836 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7eff9406af60 0x7eff9406d420 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7effa0001f10 tx=0x7effa0009040 comp rx=0 tx=0).stop 2023-10-19T14:20:30.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:30.807+0000 7effb0118640 1 -- 172.21.15.57:0/445473836 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7effa81f3820 msgr2=0x7effa8136340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:30.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:30.807+0000 7effb0118640 1 --2- 172.21.15.57:0/445473836 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7effa81f3820 0x7effa8136340 secure :-1 s=READY pgs=2399 cs=0 l=1 rev1=1 crypto rx=0x7eff98036040 tx=0x7eff9802db70 comp rx=0 tx=0).stop 2023-10-19T14:20:30.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:30.807+0000 7effb0118640 1 -- 172.21.15.57:0/445473836 shutdown_connections 2023-10-19T14:20:30.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:30.807+0000 7effb0118640 1 --2- 172.21.15.57:0/445473836 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7eff9406af60 0x7eff9406d420 secure :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7effa0001f10 tx=0x7effa0009040 comp rx=0 tx=0).stop 2023-10-19T14:20:30.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:30.807+0000 7effb0118640 1 --2- 172.21.15.57:0/445473836 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7effa81f3820 0x7effa8136340 unknown :-1 s=CLOSED pgs=2399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:30.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:30.807+0000 7effb0118640 1 --2- 172.21.15.57:0/445473836 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7effa80cb980 0x7effa8135e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:30.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:30.807+0000 7effb0118640 1 --2- 172.21.15.57:0/445473836 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7effa8070350 0x7effa8153ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:30.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:30.807+0000 7effb0118640 1 -- 172.21.15.57:0/445473836 >> 172.21.15.57:0/445473836 conn(0x7effa8069930 msgr2=0x7effa806c160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:30.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:30.808+0000 7effb0118640 1 -- 172.21.15.57:0/445473836 shutdown_connections 2023-10-19T14:20:30.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:30.808+0000 7effb0118640 1 -- 172.21.15.57:0/445473836 wait complete. 2023-10-19T14:20:30.824 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T14:20:30.936 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:30.932+0000 7f11f40bb640 1 -- 172.21.15.57:0/2954107360 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f11d40039a0 msgr2=0x7f11d4003d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:30.936 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:30.932+0000 7f11f40bb640 1 --2- 172.21.15.57:0/2954107360 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f11d40039a0 0x7f11d4003d80 secure :-1 s=READY pgs=2401 cs=0 l=1 rev1=1 crypto rx=0x7f11e4001790 tx=0x7f11d40057f0 comp rx=0 tx=0).stop 2023-10-19T14:20:30.936 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:30.932+0000 7f11f2658640 1 -- 172.21.15.57:0/2954107360 reap_dead start 2023-10-19T14:20:30.936 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:30.933+0000 7f11f40bb640 1 -- 172.21.15.57:0/2954107360 shutdown_connections 2023-10-19T14:20:30.937 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:30.933+0000 7f11f40bb640 1 -- 172.21.15.57:0/2954107360 >> 172.21.15.57:0/2954107360 conn(0x7f11ec06fae0 msgr2=0x7f11ec06fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:30.937 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:30.933+0000 7f11f40bb640 1 -- 172.21.15.57:0/2954107360 shutdown_connections 2023-10-19T14:20:30.937 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:30.933+0000 7f11f40bb640 1 -- 172.21.15.57:0/2954107360 wait complete. 2023-10-19T14:20:30.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:30.934+0000 7f11f40bb640 1 Processor -- start 2023-10-19T14:20:30.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:30.934+0000 7f11f40bb640 1 -- start start 2023-10-19T14:20:30.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:30.934+0000 7f11f40bb640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f11d40039a0 0x7f11ec1f0b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:30.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:30.934+0000 7f11f40bb640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f11ec14b8e0 0x7f11ec0cc680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:30.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:30.934+0000 7f11f40bb640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f11ec1f47d0 0x7f11ec1f4c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:30.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:30.934+0000 7f11f40bb640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11ec1f5190 con 0x7f11ec14b8e0 2023-10-19T14:20:30.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:30.934+0000 7f11f40bb640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11ec14b450 con 0x7f11ec1f47d0 2023-10-19T14:20:30.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:30.935+0000 7f11f40bb640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11ec1f10b0 con 0x7f11d40039a0 2023-10-19T14:20:30.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:30.935+0000 7f11f1656640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f11ec14b8e0 0x7f11ec0cc680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:30.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:30.935+0000 7f11f1656640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f11ec14b8e0 0x7f11ec0cc680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:58772/0 (socket says 172.21.15.57:58772) 2023-10-19T14:20:30.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:30.935+0000 7f11f1656640 1 -- 172.21.15.57:0/563245789 learned_addr learned my addr 172.21.15.57:0/563245789 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:30.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:30.935+0000 7f11f2658640 1 --2- 172.21.15.57:0/563245789 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f11ec1f47d0 0x7f11ec1f4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:30.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:30.935+0000 7f11f1e57640 1 --2- 172.21.15.57:0/563245789 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f11d40039a0 0x7f11ec1f0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:30.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:30.935+0000 7f11f1656640 1 -- 172.21.15.57:0/563245789 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f11d40039a0 msgr2=0x7f11ec1f0b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:30.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:30.935+0000 7f11f1656640 1 --2- 172.21.15.57:0/563245789 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f11d40039a0 0x7f11ec1f0b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:30.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:30.935+0000 7f11f1656640 1 -- 172.21.15.57:0/563245789 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f11ec1f47d0 msgr2=0x7f11ec1f4c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:30.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:30.935+0000 7f11f1656640 1 --2- 172.21.15.57:0/563245789 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f11ec1f47d0 0x7f11ec1f4c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:30.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:30.935+0000 7f11f1656640 1 -- 172.21.15.57:0/563245789 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11ec1f55a0 con 0x7f11ec14b8e0 2023-10-19T14:20:30.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:30.935+0000 7f11f1e57640 1 --2- 172.21.15.57:0/563245789 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f11d40039a0 0x7f11ec1f0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:20:30.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:30.935+0000 7f11f1656640 1 --2- 172.21.15.57:0/563245789 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f11ec14b8e0 0x7f11ec0cc680 secure :-1 s=READY pgs=2402 cs=0 l=1 rev1=1 crypto rx=0x7f11e0011b10 tx=0x7f11e0009e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:30.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:30.935+0000 7f11ea7fc640 1 -- 172.21.15.57:0/563245789 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11e000fea0 con 0x7f11ec14b8e0 2023-10-19T14:20:30.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:30.935+0000 7f11ea7fc640 1 -- 172.21.15.57:0/563245789 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f11e0004050 con 0x7f11ec14b8e0 2023-10-19T14:20:30.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:30.935+0000 7f11f40bb640 1 -- 172.21.15.57:0/563245789 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11ec14a840 con 0x7f11ec14b8e0 2023-10-19T14:20:30.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:30.935+0000 7f11ea7fc640 1 -- 172.21.15.57:0/563245789 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11e0004460 con 0x7f11ec14b8e0 2023-10-19T14:20:30.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:30.935+0000 7f11f40bb640 1 -- 172.21.15.57:0/563245789 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11ec14ae00 con 0x7f11ec14b8e0 2023-10-19T14:20:30.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:30.936+0000 7f11f40bb640 1 -- 172.21.15.57:0/563245789 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11ec0690e0 con 0x7f11ec14b8e0 2023-10-19T14:20:30.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:30.937+0000 7f11ea7fc640 1 -- 172.21.15.57:0/563245789 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f11e00035f0 con 0x7f11ec14b8e0 2023-10-19T14:20:30.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:30.937+0000 7f11ea7fc640 1 --2- 172.21.15.57:0/563245789 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f11d806af90 0x7f11d806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:30.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:30.938+0000 7f11ea7fc640 1 -- 172.21.15.57:0/563245789 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(226..226 src has 1..226) v4 ==== 6921+0+0 (secure 0 0 0) 0x7f11e008aa60 con 0x7f11ec14b8e0 2023-10-19T14:20:30.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:30.938+0000 7f11f1e57640 1 --2- 172.21.15.57:0/563245789 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f11d806af90 0x7f11d806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:30.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:30.938+0000 7f11f1e57640 1 --2- 172.21.15.57:0/563245789 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f11d806af90 0x7f11d806d450 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f11e4034000 tx=0x7f11e4039040 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:30.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:30.942+0000 7f11ea7fc640 1 -- 172.21.15.57:0/563245789 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f11e0055520 con 0x7f11ec14b8e0 2023-10-19T14:20:31.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.118+0000 7f11f40bb640 1 -- 172.21.15.57:0/563245789 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f11ec0707b0 con 0x7f11ec14b8e0 2023-10-19T14:20:31.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.120+0000 7f11ea7fc640 1 -- 172.21.15.57:0/563245789 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v226) v1 ==== 74+0+20508 (secure 0 0 0) 0x7f11e0058b40 con 0x7f11ec14b8e0 2023-10-19T14:20:31.123 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:20:31.123 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":226,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T14:20:30.144813+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2023-10-19T14:20:17.357433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"222","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2023-10-19T14:20:18.049516+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"223","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_metadata","create_time":"2023-10-19T14:20:28.667664+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"226","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"backup_fs_data","create_time":"2023-10-19T14:20:29.122584+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"226","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T14:20:31.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.122+0000 7f11f40bb640 1 -- 172.21.15.57:0/563245789 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f11d806af90 msgr2=0x7f11d806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:31.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.122+0000 7f11f40bb640 1 --2- 172.21.15.57:0/563245789 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f11d806af90 0x7f11d806d450 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f11e4034000 tx=0x7f11e4039040 comp rx=0 tx=0).stop 2023-10-19T14:20:31.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.122+0000 7f11f40bb640 1 -- 172.21.15.57:0/563245789 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f11ec14b8e0 msgr2=0x7f11ec0cc680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:31.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.122+0000 7f11f40bb640 1 --2- 172.21.15.57:0/563245789 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f11ec14b8e0 0x7f11ec0cc680 secure :-1 s=READY pgs=2402 cs=0 l=1 rev1=1 crypto rx=0x7f11e0011b10 tx=0x7f11e0009e90 comp rx=0 tx=0).stop 2023-10-19T14:20:31.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.122+0000 7f11f40bb640 1 -- 172.21.15.57:0/563245789 shutdown_connections 2023-10-19T14:20:31.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.122+0000 7f11f40bb640 1 --2- 172.21.15.57:0/563245789 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f11d806af90 0x7f11d806d450 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:31.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.122+0000 7f11f40bb640 1 --2- 172.21.15.57:0/563245789 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f11ec1f47d0 0x7f11ec1f4c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:31.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.122+0000 7f11f40bb640 1 --2- 172.21.15.57:0/563245789 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f11ec14b8e0 0x7f11ec0cc680 unknown :-1 s=CLOSED pgs=2402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:31.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.122+0000 7f11f40bb640 1 --2- 172.21.15.57:0/563245789 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f11d40039a0 0x7f11ec1f0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:31.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.122+0000 7f11f40bb640 1 -- 172.21.15.57:0/563245789 >> 172.21.15.57:0/563245789 conn(0x7f11ec06fae0 msgr2=0x7f11ec1544a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:31.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.122+0000 7f11f40bb640 1 -- 172.21.15.57:0/563245789 shutdown_connections 2023-10-19T14:20:31.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.122+0000 7f11f40bb640 1 -- 172.21.15.57:0/563245789 wait complete. 2023-10-19T14:20:31.135 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T14:20:31.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.241+0000 7f24665d0640 1 -- 172.21.15.57:0/3121267103 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f24600705a0 msgr2=0x7f24600b0530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:31.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.241+0000 7f24665d0640 1 --2- 172.21.15.57:0/3121267103 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f24600705a0 0x7f24600b0530 secure :-1 s=READY pgs=2404 cs=0 l=1 rev1=1 crypto rx=0x7f2458005fd0 tx=0x7f245802c0e0 comp rx=0 tx=0).stop 2023-10-19T14:20:31.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.241+0000 7f24665d0640 1 -- 172.21.15.57:0/3121267103 shutdown_connections 2023-10-19T14:20:31.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.241+0000 7f24665d0640 1 --2- 172.21.15.57:0/3121267103 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f24600cbd20 0x7f2460158300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:31.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.241+0000 7f24665d0640 1 --2- 172.21.15.57:0/3121267103 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f24600705a0 0x7f24600b0530 unknown :-1 s=CLOSED pgs=2404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:31.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.241+0000 7f24665d0640 1 -- 172.21.15.57:0/3121267103 >> 172.21.15.57:0/3121267103 conn(0x7f2460069930 msgr2=0x7f2460069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:31.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.241+0000 7f24665d0640 1 -- 172.21.15.57:0/3121267103 shutdown_connections 2023-10-19T14:20:31.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.242+0000 7f24665d0640 1 -- 172.21.15.57:0/3121267103 wait complete. 2023-10-19T14:20:31.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.243+0000 7f24665d0640 1 Processor -- start 2023-10-19T14:20:31.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.243+0000 7f24665d0640 1 -- start start 2023-10-19T14:20:31.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.243+0000 7f24665d0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f24600705a0 0x7f246015d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:31.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.243+0000 7f24665d0640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f24600afa20 0x7f24600ad8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:31.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.243+0000 7f24665d0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f24600cbd20 0x7f246014e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:31.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.243+0000 7f24665d0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24601e4020 con 0x7f24600cbd20 2023-10-19T14:20:31.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.243+0000 7f24665d0640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f246015dc70 con 0x7f24600705a0 2023-10-19T14:20:31.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.243+0000 7f24665d0640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24601e33a0 con 0x7f24600afa20 2023-10-19T14:20:31.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.244+0000 7f2464b6d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f24600cbd20 0x7f246014e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:31.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.244+0000 7f2464b6d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f24600cbd20 0x7f246014e950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:58804/0 (socket says 172.21.15.57:58804) 2023-10-19T14:20:31.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.244+0000 7f2464b6d640 1 -- 172.21.15.57:0/1567310247 learned_addr learned my addr 172.21.15.57:0/1567310247 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:31.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.244+0000 7f245f7fe640 1 --2- 172.21.15.57:0/1567310247 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f24600afa20 0x7f24600ad8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:31.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.244+0000 7f245ffff640 1 --2- 172.21.15.57:0/1567310247 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f24600705a0 0x7f246015d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:31.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.244+0000 7f2464b6d640 1 -- 172.21.15.57:0/1567310247 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f24600afa20 msgr2=0x7f24600ad8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:31.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.244+0000 7f2464b6d640 1 --2- 172.21.15.57:0/1567310247 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f24600afa20 0x7f24600ad8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:31.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.244+0000 7f2464b6d640 1 -- 172.21.15.57:0/1567310247 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f24600705a0 msgr2=0x7f246015d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:31.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.244+0000 7f2464b6d640 1 --2- 172.21.15.57:0/1567310247 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f24600705a0 0x7f246015d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:31.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.244+0000 7f2464b6d640 1 -- 172.21.15.57:0/1567310247 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2458009840 con 0x7f24600cbd20 2023-10-19T14:20:31.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.244+0000 7f2464b6d640 1 --2- 172.21.15.57:0/1567310247 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f24600cbd20 0x7f246014e950 secure :-1 s=READY pgs=2405 cs=0 l=1 rev1=1 crypto rx=0x7f2458006010 tx=0x7f24580042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:31.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.244+0000 7f245cff9640 1 -- 172.21.15.57:0/1567310247 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24580046f0 con 0x7f24600cbd20 2023-10-19T14:20:31.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.244+0000 7f245cff9640 1 -- 172.21.15.57:0/1567310247 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f24580345a0 con 0x7f24600cbd20 2023-10-19T14:20:31.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.244+0000 7f24665d0640 1 -- 172.21.15.57:0/1567310247 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24601e3680 con 0x7f24600cbd20 2023-10-19T14:20:31.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.244+0000 7f245cff9640 1 -- 172.21.15.57:0/1567310247 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2458034990 con 0x7f24600cbd20 2023-10-19T14:20:31.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.244+0000 7f24665d0640 1 -- 172.21.15.57:0/1567310247 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24601e3c40 con 0x7f24600cbd20 2023-10-19T14:20:31.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.245+0000 7f24665d0640 1 -- 172.21.15.57:0/1567310247 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24601e4160 con 0x7f24600cbd20 2023-10-19T14:20:31.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.246+0000 7f245cff9640 1 -- 172.21.15.57:0/1567310247 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f2458034af0 con 0x7f24600cbd20 2023-10-19T14:20:31.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.246+0000 7f245cff9640 1 --2- 172.21.15.57:0/1567310247 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f244c06af60 0x7f244c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:31.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.246+0000 7f245cff9640 1 -- 172.21.15.57:0/1567310247 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(226..226 src has 1..226) v4 ==== 6921+0+0 (secure 0 0 0) 0x7f24580acd20 con 0x7f24600cbd20 2023-10-19T14:20:31.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.247+0000 7f245ffff640 1 --2- 172.21.15.57:0/1567310247 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f244c06af60 0x7f244c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:31.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.247+0000 7f245ffff640 1 --2- 172.21.15.57:0/1567310247 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f244c06af60 0x7f244c06d420 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f246014f9f0 tx=0x7f2450007e50 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:31.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.251+0000 7f245cff9640 1 -- 172.21.15.57:0/1567310247 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f24580778a0 con 0x7f24600cbd20 2023-10-19T14:20:31.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.427+0000 7f24665d0640 1 -- 172.21.15.57:0/1567310247 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2460067930 con 0x7f24600cbd20 2023-10-19T14:20:31.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.428+0000 7f245cff9640 1 -- 172.21.15.57:0/1567310247 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v226) v1 ==== 74+0+20508 (secure 0 0 0) 0x7f2458043090 con 0x7f24600cbd20 2023-10-19T14:20:31.431 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:20:31.431 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":226,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T14:20:30.144813+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2023-10-19T14:20:17.357433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"222","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2023-10-19T14:20:18.049516+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"223","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_metadata","create_time":"2023-10-19T14:20:28.667664+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"226","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"backup_fs_data","create_time":"2023-10-19T14:20:29.122584+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"226","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T14:20:31.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.430+0000 7f24665d0640 1 -- 172.21.15.57:0/1567310247 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f244c06af60 msgr2=0x7f244c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:31.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.430+0000 7f24665d0640 1 --2- 172.21.15.57:0/1567310247 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f244c06af60 0x7f244c06d420 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f246014f9f0 tx=0x7f2450007e50 comp rx=0 tx=0).stop 2023-10-19T14:20:31.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.430+0000 7f24665d0640 1 -- 172.21.15.57:0/1567310247 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f24600cbd20 msgr2=0x7f246014e950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:31.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.430+0000 7f24665d0640 1 --2- 172.21.15.57:0/1567310247 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f24600cbd20 0x7f246014e950 secure :-1 s=READY pgs=2405 cs=0 l=1 rev1=1 crypto rx=0x7f2458006010 tx=0x7f24580042f0 comp rx=0 tx=0).stop 2023-10-19T14:20:31.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.430+0000 7f24665d0640 1 -- 172.21.15.57:0/1567310247 shutdown_connections 2023-10-19T14:20:31.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.430+0000 7f24665d0640 1 --2- 172.21.15.57:0/1567310247 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f244c06af60 0x7f244c06d420 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:31.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.430+0000 7f24665d0640 1 --2- 172.21.15.57:0/1567310247 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f24600cbd20 0x7f246014e950 unknown :-1 s=CLOSED pgs=2405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:31.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.430+0000 7f24665d0640 1 --2- 172.21.15.57:0/1567310247 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f24600afa20 0x7f24600ad8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:31.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.430+0000 7f24665d0640 1 --2- 172.21.15.57:0/1567310247 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f24600705a0 0x7f246015d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:31.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.430+0000 7f24665d0640 1 -- 172.21.15.57:0/1567310247 >> 172.21.15.57:0/1567310247 conn(0x7f2460069930 msgr2=0x7f24600aff00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:31.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.430+0000 7f24665d0640 1 -- 172.21.15.57:0/1567310247 shutdown_connections 2023-10-19T14:20:31.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.430+0000 7f24665d0640 1 -- 172.21.15.57:0/1567310247 wait complete. 2023-10-19T14:20:31.444 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set backup_fs standby_count_wanted 0 2023-10-19T14:20:31.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.553+0000 7f6b83dc7640 1 -- 172.21.15.57:0/2969371927 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6b7c0ceb40 msgr2=0x7f6b7c0b0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:31.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.553+0000 7f6b83dc7640 1 --2- 172.21.15.57:0/2969371927 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6b7c0ceb40 0x7f6b7c0b0650 secure :-1 s=READY pgs=2406 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c08a120 tx=0x7f6b6c02c260 comp rx=0 tx=0).stop 2023-10-19T14:20:31.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.553+0000 7f6b83dc7640 1 -- 172.21.15.57:0/2969371927 shutdown_connections 2023-10-19T14:20:31.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.553+0000 7f6b83dc7640 1 --2- 172.21.15.57:0/2969371927 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6b7c0ceb40 0x7f6b7c0b0650 unknown :-1 s=CLOSED pgs=2406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:31.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.553+0000 7f6b83dc7640 1 --2- 172.21.15.57:0/2969371927 >> v2:172.21.15.179:3300/0 conn(0x7f6b7c15b2a0 0x7f6b7c0d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:31.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.553+0000 7f6b83dc7640 1 -- 172.21.15.57:0/2969371927 >> 172.21.15.57:0/2969371927 conn(0x7f6b7c0d0d60 msgr2=0x7f6b7c0d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:31.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.553+0000 7f6b83dc7640 1 -- 172.21.15.57:0/2969371927 shutdown_connections 2023-10-19T14:20:31.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.553+0000 7f6b83dc7640 1 -- 172.21.15.57:0/2969371927 wait complete. 2023-10-19T14:20:31.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.554+0000 7f6b83dc7640 1 Processor -- start 2023-10-19T14:20:31.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.554+0000 7f6b83dc7640 1 -- start start 2023-10-19T14:20:31.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.555+0000 7f6b83dc7640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6b7c0ceb40 0x7f6b7c0cad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:31.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.555+0000 7f6b83dc7640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6b7c0cf340 0x7f6b7c1f19c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:31.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.555+0000 7f6b83dc7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6b7c15b2a0 0x7f6b7c0b42f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:31.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.555+0000 7f6b83dc7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b7c0b4d70 con 0x7f6b7c15b2a0 2023-10-19T14:20:31.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.555+0000 7f6b83dc7640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b7c0b4830 con 0x7f6b7c0ceb40 2023-10-19T14:20:31.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.555+0000 7f6b83dc7640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b7c0b4a90 con 0x7f6b7c0cf340 2023-10-19T14:20:31.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.555+0000 7f6b82364640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6b7c15b2a0 0x7f6b7c0b42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:31.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.555+0000 7f6b82364640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6b7c15b2a0 0x7f6b7c0b42f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:58820/0 (socket says 172.21.15.57:58820) 2023-10-19T14:20:31.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.555+0000 7f6b82364640 1 -- 172.21.15.57:0/84282667 learned_addr learned my addr 172.21.15.57:0/84282667 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:31.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.555+0000 7f6b81362640 1 --2- 172.21.15.57:0/84282667 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6b7c0cf340 0x7f6b7c1f19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:31.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.555+0000 7f6b81b63640 1 --2- 172.21.15.57:0/84282667 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6b7c0ceb40 0x7f6b7c0cad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:31.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.556+0000 7f6b82364640 1 -- 172.21.15.57:0/84282667 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6b7c0cf340 msgr2=0x7f6b7c1f19c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:31.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.556+0000 7f6b82364640 1 --2- 172.21.15.57:0/84282667 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6b7c0cf340 0x7f6b7c1f19c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:31.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.556+0000 7f6b82364640 1 -- 172.21.15.57:0/84282667 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6b7c0ceb40 msgr2=0x7f6b7c0cad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:31.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.556+0000 7f6b82364640 1 --2- 172.21.15.57:0/84282667 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6b7c0ceb40 0x7f6b7c0cad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:31.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.556+0000 7f6b82364640 1 -- 172.21.15.57:0/84282667 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b6c009900 con 0x7f6b7c15b2a0 2023-10-19T14:20:31.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.556+0000 7f6b82364640 1 --2- 172.21.15.57:0/84282667 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6b7c15b2a0 0x7f6b7c0b42f0 secure :-1 s=READY pgs=2407 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c037040 tx=0x7f6b6c003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:31.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.556+0000 7f6b7a7fc640 1 -- 172.21.15.57:0/84282667 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b6c004240 con 0x7f6b7c15b2a0 2023-10-19T14:20:31.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.556+0000 7f6b7a7fc640 1 -- 172.21.15.57:0/84282667 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f6b6c004490 con 0x7f6b7c15b2a0 2023-10-19T14:20:31.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.556+0000 7f6b83dc7640 1 -- 172.21.15.57:0/84282667 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b7c0c7f60 con 0x7f6b7c15b2a0 2023-10-19T14:20:31.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.556+0000 7f6b7a7fc640 1 -- 172.21.15.57:0/84282667 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b6c004880 con 0x7f6b7c15b2a0 2023-10-19T14:20:31.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.556+0000 7f6b83dc7640 1 -- 172.21.15.57:0/84282667 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b7c0c8520 con 0x7f6b7c15b2a0 2023-10-19T14:20:31.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.557+0000 7f6b83dc7640 1 -- 172.21.15.57:0/84282667 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b7c0c8710 con 0x7f6b7c15b2a0 2023-10-19T14:20:31.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.557+0000 7f6b7a7fc640 1 -- 172.21.15.57:0/84282667 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6b6c03b040 con 0x7f6b7c15b2a0 2023-10-19T14:20:31.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.558+0000 7f6b7a7fc640 1 --2- 172.21.15.57:0/84282667 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f6b6806af60 0x7f6b6806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:31.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.558+0000 7f6b7a7fc640 1 -- 172.21.15.57:0/84282667 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(226..226 src has 1..226) v4 ==== 6921+0+0 (secure 0 0 0) 0x7f6b6c0ac8d0 con 0x7f6b7c15b2a0 2023-10-19T14:20:31.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.558+0000 7f6b81b63640 1 --2- 172.21.15.57:0/84282667 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f6b6806af60 0x7f6b6806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:31.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.559+0000 7f6b81b63640 1 --2- 172.21.15.57:0/84282667 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f6b6806af60 0x7f6b6806d420 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6b74000f60 tx=0x7f6b74009040 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:31.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.562+0000 7f6b7a7fc640 1 -- 172.21.15.57:0/84282667 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6b6c077440 con 0x7f6b7c15b2a0 2023-10-19T14:20:31.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.773+0000 7f6b83dc7640 1 -- 172.21.15.57:0/84282667 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "backup_fs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f6b7c0b0d00 con 0x7f6b7c15b2a0 2023-10-19T14:20:31.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.793+0000 7f6b7a7fc640 1 -- 172.21.15.57:0/84282667 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "backup_fs", "var": "standby_count_wanted", "val": "0"}]=0 v93) v1 ==== 121+0+0 (secure 0 0 0) 0x7f6b6c07aa60 con 0x7f6b7c15b2a0 2023-10-19T14:20:31.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.796+0000 7f6b83dc7640 1 -- 172.21.15.57:0/84282667 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f6b6806af60 msgr2=0x7f6b6806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:31.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.796+0000 7f6b83dc7640 1 --2- 172.21.15.57:0/84282667 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f6b6806af60 0x7f6b6806d420 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6b74000f60 tx=0x7f6b74009040 comp rx=0 tx=0).stop 2023-10-19T14:20:31.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.796+0000 7f6b83dc7640 1 -- 172.21.15.57:0/84282667 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6b7c15b2a0 msgr2=0x7f6b7c0b42f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:31.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.796+0000 7f6b83dc7640 1 --2- 172.21.15.57:0/84282667 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6b7c15b2a0 0x7f6b7c0b42f0 secure :-1 s=READY pgs=2407 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c037040 tx=0x7f6b6c003ba0 comp rx=0 tx=0).stop 2023-10-19T14:20:31.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.796+0000 7f6b83dc7640 1 -- 172.21.15.57:0/84282667 shutdown_connections 2023-10-19T14:20:31.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.796+0000 7f6b83dc7640 1 --2- 172.21.15.57:0/84282667 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f6b6806af60 0x7f6b6806d420 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:31.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.796+0000 7f6b83dc7640 1 --2- 172.21.15.57:0/84282667 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6b7c15b2a0 0x7f6b7c0b42f0 unknown :-1 s=CLOSED pgs=2407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:31.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.796+0000 7f6b83dc7640 1 --2- 172.21.15.57:0/84282667 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6b7c0cf340 0x7f6b7c1f19c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:31.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.796+0000 7f6b83dc7640 1 --2- 172.21.15.57:0/84282667 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6b7c0ceb40 0x7f6b7c0cad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:31.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.796+0000 7f6b83dc7640 1 -- 172.21.15.57:0/84282667 >> 172.21.15.57:0/84282667 conn(0x7f6b7c0d0d60 msgr2=0x7f6b7c06a930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:31.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.796+0000 7f6b83dc7640 1 -- 172.21.15.57:0/84282667 shutdown_connections 2023-10-19T14:20:31.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.796+0000 7f6b83dc7640 1 -- 172.21.15.57:0/84282667 wait complete. 2023-10-19T14:20:31.809 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-19T14:20:31.810 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set backup_fs allow_standby_replay false 2023-10-19T14:20:31.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.919+0000 7f346ada9640 1 -- 172.21.15.57:0/2178007945 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f346415c1f0 msgr2=0x7f3464157dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:31.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.919+0000 7f346ada9640 1 --2- 172.21.15.57:0/2178007945 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f346415c1f0 0x7f3464157dd0 secure :-1 s=READY pgs=2408 cs=0 l=1 rev1=1 crypto rx=0x7f345c005fd0 tx=0x7f345c02c240 comp rx=0 tx=0).stop 2023-10-19T14:20:31.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.919+0000 7f346ada9640 1 -- 172.21.15.57:0/2178007945 shutdown_connections 2023-10-19T14:20:31.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.919+0000 7f346ada9640 1 --2- 172.21.15.57:0/2178007945 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f346415c1f0 0x7f3464157dd0 unknown :-1 s=CLOSED pgs=2408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:31.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.919+0000 7f346ada9640 1 --2- 172.21.15.57:0/2178007945 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f346406ae40 0x7f34640b0e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:31.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.919+0000 7f346ada9640 1 -- 172.21.15.57:0/2178007945 >> 172.21.15.57:0/2178007945 conn(0x7f346406fae0 msgr2=0x7f346406fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:31.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.919+0000 7f346ada9640 1 -- 172.21.15.57:0/2178007945 shutdown_connections 2023-10-19T14:20:31.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.919+0000 7f346ada9640 1 -- 172.21.15.57:0/2178007945 wait complete. 2023-10-19T14:20:31.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.920+0000 7f346ada9640 1 Processor -- start 2023-10-19T14:20:31.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.920+0000 7f346ada9640 1 -- start start 2023-10-19T14:20:31.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.921+0000 7f346ada9640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f346406ae40 0x7f34640a64b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:31.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.921+0000 7f346ada9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f34641579f0 0x7f34640ae0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:31.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.921+0000 7f346ada9640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f346415c1f0 0x7f34640ae5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:31.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.921+0000 7f346ada9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3464151910 con 0x7f34641579f0 2023-10-19T14:20:31.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.921+0000 7f346ada9640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3464151390 con 0x7f346415c1f0 2023-10-19T14:20:31.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.921+0000 7f346ada9640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34640a69f0 con 0x7f346406ae40 2023-10-19T14:20:31.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.921+0000 7f3463fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f34641579f0 0x7f34640ae0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:31.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.921+0000 7f3463fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f34641579f0 0x7f34640ae0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:58836/0 (socket says 172.21.15.57:58836) 2023-10-19T14:20:31.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.921+0000 7f3463fff640 1 -- 172.21.15.57:0/959050526 learned_addr learned my addr 172.21.15.57:0/959050526 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:31.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.921+0000 7f3469346640 1 --2- 172.21.15.57:0/959050526 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f346415c1f0 0x7f34640ae5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:31.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.921+0000 7f3468b45640 1 --2- 172.21.15.57:0/959050526 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f346406ae40 0x7f34640a64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:31.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.921+0000 7f3463fff640 1 -- 172.21.15.57:0/959050526 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f346406ae40 msgr2=0x7f34640a64b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:31.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.921+0000 7f3463fff640 1 --2- 172.21.15.57:0/959050526 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f346406ae40 0x7f34640a64b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:31.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.921+0000 7f3463fff640 1 -- 172.21.15.57:0/959050526 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f346415c1f0 msgr2=0x7f34640ae5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:31.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.921+0000 7f3463fff640 1 --2- 172.21.15.57:0/959050526 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f346415c1f0 0x7f34640ae5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:31.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.922+0000 7f3463fff640 1 -- 172.21.15.57:0/959050526 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f345c009900 con 0x7f34641579f0 2023-10-19T14:20:31.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.922+0000 7f3469346640 1 --2- 172.21.15.57:0/959050526 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f346415c1f0 0x7f34640ae5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:20:31.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.922+0000 7f3468b45640 1 --2- 172.21.15.57:0/959050526 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f346406ae40 0x7f34640a64b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:20:31.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.922+0000 7f3463fff640 1 --2- 172.21.15.57:0/959050526 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f34641579f0 0x7f34640ae0b0 secure :-1 s=READY pgs=2409 cs=0 l=1 rev1=1 crypto rx=0x7f345800b4a0 tx=0x7f345800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:31.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.922+0000 7f34617fa640 1 -- 172.21.15.57:0/959050526 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3458010610 con 0x7f34641579f0 2023-10-19T14:20:31.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.922+0000 7f34617fa640 1 -- 172.21.15.57:0/959050526 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f3458010860 con 0x7f34641579f0 2023-10-19T14:20:31.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.922+0000 7f346ada9640 1 -- 172.21.15.57:0/959050526 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34640b2330 con 0x7f34641579f0 2023-10-19T14:20:31.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.922+0000 7f34617fa640 1 -- 172.21.15.57:0/959050526 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3458010c70 con 0x7f34641579f0 2023-10-19T14:20:31.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.922+0000 7f346ada9640 1 -- 172.21.15.57:0/959050526 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34641563f0 con 0x7f34641579f0 2023-10-19T14:20:31.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.923+0000 7f346ada9640 1 -- 172.21.15.57:0/959050526 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34640690e0 con 0x7f34641579f0 2023-10-19T14:20:31.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.923+0000 7f34617fa640 1 -- 172.21.15.57:0/959050526 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f3458017360 con 0x7f34641579f0 2023-10-19T14:20:31.932 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.924+0000 7f34617fa640 1 --2- 172.21.15.57:0/959050526 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f345006af60 0x7f345006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:31.932 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.924+0000 7f34617fa640 1 -- 172.21.15.57:0/959050526 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 6921+0+0 (secure 0 0 0) 0x7f3458089a60 con 0x7f34641579f0 2023-10-19T14:20:31.932 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.924+0000 7f3468b45640 1 --2- 172.21.15.57:0/959050526 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f345006af60 0x7f345006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:31.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.925+0000 7f3468b45640 1 --2- 172.21.15.57:0/959050526 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f345006af60 0x7f345006d420 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f3454007fd0 tx=0x7f3454009040 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:31.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:31.928+0000 7f34617fa640 1 -- 172.21.15.57:0/959050526 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f34580545d0 con 0x7f34641579f0 2023-10-19T14:20:32.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:32.140+0000 7f346ada9640 1 -- 172.21.15.57:0/959050526 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "backup_fs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f3464153950 con 0x7f34641579f0 2023-10-19T14:20:32.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:32.815+0000 7f34617fa640 1 -- 172.21.15.57:0/959050526 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "backup_fs", "var": "allow_standby_replay", "val": "false"}]=0 v94) v1 ==== 125+0+0 (secure 0 0 0) 0x7f3458057bf0 con 0x7f34641579f0 2023-10-19T14:20:32.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:32.818+0000 7f346ada9640 1 -- 172.21.15.57:0/959050526 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f345006af60 msgr2=0x7f345006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:32.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:32.818+0000 7f346ada9640 1 --2- 172.21.15.57:0/959050526 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f345006af60 0x7f345006d420 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f3454007fd0 tx=0x7f3454009040 comp rx=0 tx=0).stop 2023-10-19T14:20:32.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:32.818+0000 7f346ada9640 1 -- 172.21.15.57:0/959050526 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f34641579f0 msgr2=0x7f34640ae0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:32.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:32.818+0000 7f346ada9640 1 --2- 172.21.15.57:0/959050526 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f34641579f0 0x7f34640ae0b0 secure :-1 s=READY pgs=2409 cs=0 l=1 rev1=1 crypto rx=0x7f345800b4a0 tx=0x7f345800b960 comp rx=0 tx=0).stop 2023-10-19T14:20:32.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:32.818+0000 7f346ada9640 1 -- 172.21.15.57:0/959050526 shutdown_connections 2023-10-19T14:20:32.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:32.818+0000 7f346ada9640 1 --2- 172.21.15.57:0/959050526 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f345006af60 0x7f345006d420 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:32.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:32.818+0000 7f346ada9640 1 --2- 172.21.15.57:0/959050526 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f346415c1f0 0x7f34640ae5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:32.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:32.818+0000 7f346ada9640 1 --2- 172.21.15.57:0/959050526 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f34641579f0 0x7f34640ae0b0 unknown :-1 s=CLOSED pgs=2409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:32.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:32.818+0000 7f346ada9640 1 --2- 172.21.15.57:0/959050526 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f346406ae40 0x7f34640a64b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:32.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:32.818+0000 7f346ada9640 1 -- 172.21.15.57:0/959050526 >> 172.21.15.57:0/959050526 conn(0x7f346406fae0 msgr2=0x7f346415aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:32.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:32.818+0000 7f346ada9640 1 -- 172.21.15.57:0/959050526 shutdown_connections 2023-10-19T14:20:32.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:32.818+0000 7f346ada9640 1 -- 172.21.15.57:0/959050526 wait complete. 2023-10-19T14:20:32.834 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'backup_fs' 2023-10-19T14:20:32.834 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T14:20:32.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:32.947+0000 7f6e1092c640 1 -- 172.21.15.57:0/2868220005 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6e0c0b0530 msgr2=0x7f6e0c0b09c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:32.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:32.947+0000 7f6e1092c640 1 --2- 172.21.15.57:0/2868220005 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6e0c0b0530 0x7f6e0c0b09c0 secure :-1 s=READY pgs=2411 cs=0 l=1 rev1=1 crypto rx=0x7f6e04005fd0 tx=0x7f6e0402c0e0 comp rx=0 tx=0).stop 2023-10-19T14:20:32.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:32.947+0000 7f6e1092c640 1 -- 172.21.15.57:0/2868220005 shutdown_connections 2023-10-19T14:20:32.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:32.947+0000 7f6e1092c640 1 --2- 172.21.15.57:0/2868220005 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6e0c0b0530 0x7f6e0c0b09c0 unknown :-1 s=CLOSED pgs=2411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:32.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:32.947+0000 7f6e1092c640 1 --2- 172.21.15.57:0/2868220005 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6e0c0cbd20 0x7f6e0c158300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:32.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:32.947+0000 7f6e1092c640 1 -- 172.21.15.57:0/2868220005 >> 172.21.15.57:0/2868220005 conn(0x7f6e0c069930 msgr2=0x7f6e0c069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:32.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:32.947+0000 7f6e1092c640 1 -- 172.21.15.57:0/2868220005 shutdown_connections 2023-10-19T14:20:32.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:32.947+0000 7f6e1092c640 1 -- 172.21.15.57:0/2868220005 wait complete. 2023-10-19T14:20:32.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:32.949+0000 7f6e1092c640 1 Processor -- start 2023-10-19T14:20:32.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:32.949+0000 7f6e1092c640 1 -- start start 2023-10-19T14:20:32.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:32.949+0000 7f6e1092c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6e0c0afa20 0x7f6e0c13e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:32.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:32.949+0000 7f6e1092c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6e0c0b0530 0x7f6e0c147a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:32.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:32.949+0000 7f6e1092c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6e0c0cbd20 0x7f6e0c147f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:32.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:32.949+0000 7f6e1092c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e0c1598e0 con 0x7f6e0c0afa20 2023-10-19T14:20:32.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:32.949+0000 7f6e1092c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e0c13e6b0 con 0x7f6e0c0cbd20 2023-10-19T14:20:32.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:32.949+0000 7f6e1092c640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e0c13f230 con 0x7f6e0c0b0530 2023-10-19T14:20:32.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:32.950+0000 7f6e0a59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6e0c0afa20 0x7f6e0c13e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:32.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:32.950+0000 7f6e0a59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6e0c0afa20 0x7f6e0c13e040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:58860/0 (socket says 172.21.15.57:58860) 2023-10-19T14:20:32.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:32.950+0000 7f6e0a59c640 1 -- 172.21.15.57:0/2060343569 learned_addr learned my addr 172.21.15.57:0/2060343569 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:32.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:32.950+0000 7f6e03fff640 1 --2- 172.21.15.57:0/2060343569 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6e0c0b0530 0x7f6e0c147a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:32.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:32.950+0000 7f6e0ad9d640 1 --2- 172.21.15.57:0/2060343569 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6e0c0cbd20 0x7f6e0c147f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:32.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:32.950+0000 7f6e0a59c640 1 -- 172.21.15.57:0/2060343569 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6e0c0b0530 msgr2=0x7f6e0c147a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:32.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:32.950+0000 7f6e0a59c640 1 --2- 172.21.15.57:0/2060343569 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6e0c0b0530 0x7f6e0c147a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:32.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:32.950+0000 7f6e0a59c640 1 -- 172.21.15.57:0/2060343569 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6e0c0cbd20 msgr2=0x7f6e0c147f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:32.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:32.950+0000 7f6e0a59c640 1 --2- 172.21.15.57:0/2060343569 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6e0c0cbd20 0x7f6e0c147f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:32.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:32.950+0000 7f6e0a59c640 1 -- 172.21.15.57:0/2060343569 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e04009840 con 0x7f6e0c0afa20 2023-10-19T14:20:32.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:32.950+0000 7f6e0ad9d640 1 --2- 172.21.15.57:0/2060343569 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6e0c0cbd20 0x7f6e0c147f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:20:32.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:32.950+0000 7f6e0a59c640 1 --2- 172.21.15.57:0/2060343569 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6e0c0afa20 0x7f6e0c13e040 secure :-1 s=READY pgs=2412 cs=0 l=1 rev1=1 crypto rx=0x7f6dfc00b410 tx=0x7f6dfc00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:32.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:32.950+0000 7f6e02ffd640 1 -- 172.21.15.57:0/2060343569 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dfc011830 con 0x7f6e0c0afa20 2023-10-19T14:20:32.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:32.950+0000 7f6e02ffd640 1 -- 172.21.15.57:0/2060343569 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f6dfc011aa0 con 0x7f6e0c0afa20 2023-10-19T14:20:32.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:32.950+0000 7f6e1092c640 1 -- 172.21.15.57:0/2060343569 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e0c13e900 con 0x7f6e0c0afa20 2023-10-19T14:20:32.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:32.950+0000 7f6e02ffd640 1 -- 172.21.15.57:0/2060343569 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dfc010380 con 0x7f6e0c0afa20 2023-10-19T14:20:32.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:32.950+0000 7f6e1092c640 1 -- 172.21.15.57:0/2060343569 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e0c13eec0 con 0x7f6e0c0afa20 2023-10-19T14:20:32.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:32.951+0000 7f6e1092c640 1 -- 172.21.15.57:0/2060343569 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e0c0690e0 con 0x7f6e0c0afa20 2023-10-19T14:20:32.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:32.952+0000 7f6e02ffd640 1 -- 172.21.15.57:0/2060343569 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6dfc0104e0 con 0x7f6e0c0afa20 2023-10-19T14:20:32.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:32.952+0000 7f6e02ffd640 1 --2- 172.21.15.57:0/2060343569 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f6dec06af60 0x7f6dec06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:32.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:32.952+0000 7f6e02ffd640 1 -- 172.21.15.57:0/2060343569 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 6921+0+0 (secure 0 0 0) 0x7f6dfc08a1e0 con 0x7f6e0c0afa20 2023-10-19T14:20:32.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:32.952+0000 7f6e03fff640 1 --2- 172.21.15.57:0/2060343569 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f6dec06af60 0x7f6dec06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:32.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:32.953+0000 7f6e03fff640 1 --2- 172.21.15.57:0/2060343569 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f6dec06af60 0x7f6dec06d420 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6e0c1f4b50 tx=0x7f6df4009040 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:32.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:32.956+0000 7f6e02ffd640 1 -- 172.21.15.57:0/2060343569 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6dfc054ba0 con 0x7f6e0c0afa20 2023-10-19T14:20:33.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.169+0000 7f6e1092c640 1 -- 172.21.15.57:0/2060343569 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6e0c165080 con 0x7f6e0c0afa20 2023-10-19T14:20:33.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.170+0000 7f6e02ffd640 1 -- 172.21.15.57:0/2060343569 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+6152 (secure 0 0 0) 0x7f6dfc0581c0 con 0x7f6e0c0afa20 2023-10-19T14:20:33.173 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:20:33.173 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":94,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9248,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/1249807745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":1249807745},{"type":"v1","addr":"172.21.15.179:6836","nonce":1249807745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87},{"gid":10513,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/1406825818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":1406825818},{"type":"v1","addr":"172.21.15.179:6837","nonce":1406825818}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":86}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T14:20:19.077396+0000","modified":"2023-10-19T14:20:21.072483+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11718},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11718":{"gid":11718,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.57:6835/1529368280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1529368280},{"type":"v1","addr":"172.21.15.57:6835","nonce":1529368280}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13},{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T14:20:30.145135+0000","modified":"2023-10-19T14:20:32.141937+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11715},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11715":{"gid":11715,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":5,"addr":"172.21.15.57:6834/2467807803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2467807803},{"type":"v1","addr":"172.21.15.57:6834","nonce":2467807803}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2023-10-19T14:20:33.174 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 94 2023-10-19T14:20:33.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.172+0000 7f6e1092c640 1 -- 172.21.15.57:0/2060343569 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f6dec06af60 msgr2=0x7f6dec06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:33.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.172+0000 7f6e1092c640 1 --2- 172.21.15.57:0/2060343569 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f6dec06af60 0x7f6dec06d420 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6e0c1f4b50 tx=0x7f6df4009040 comp rx=0 tx=0).stop 2023-10-19T14:20:33.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.172+0000 7f6e1092c640 1 -- 172.21.15.57:0/2060343569 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6e0c0afa20 msgr2=0x7f6e0c13e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:33.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.172+0000 7f6e1092c640 1 --2- 172.21.15.57:0/2060343569 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6e0c0afa20 0x7f6e0c13e040 secure :-1 s=READY pgs=2412 cs=0 l=1 rev1=1 crypto rx=0x7f6dfc00b410 tx=0x7f6dfc00b8d0 comp rx=0 tx=0).stop 2023-10-19T14:20:33.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.172+0000 7f6e1092c640 1 -- 172.21.15.57:0/2060343569 shutdown_connections 2023-10-19T14:20:33.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.172+0000 7f6e1092c640 1 --2- 172.21.15.57:0/2060343569 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f6dec06af60 0x7f6dec06d420 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:33.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.172+0000 7f6e1092c640 1 --2- 172.21.15.57:0/2060343569 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6e0c0cbd20 0x7f6e0c147f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:33.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.172+0000 7f6e1092c640 1 --2- 172.21.15.57:0/2060343569 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6e0c0b0530 0x7f6e0c147a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:33.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.172+0000 7f6e1092c640 1 --2- 172.21.15.57:0/2060343569 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6e0c0afa20 0x7f6e0c13e040 unknown :-1 s=CLOSED pgs=2412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:33.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.172+0000 7f6e1092c640 1 -- 172.21.15.57:0/2060343569 >> 172.21.15.57:0/2060343569 conn(0x7f6e0c069930 msgr2=0x7f6e0c1495b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:33.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.172+0000 7f6e1092c640 1 -- 172.21.15.57:0/2060343569 shutdown_connections 2023-10-19T14:20:33.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.172+0000 7f6e1092c640 1 -- 172.21.15.57:0/2060343569 wait complete. 2023-10-19T14:20:33.185 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T14:20:33.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.292+0000 7f2d5a9a0640 1 -- 172.21.15.57:0/371304744 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2d540b1b40 msgr2=0x7f2d540ca4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:33.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.292+0000 7f2d5a9a0640 1 --2- 172.21.15.57:0/371304744 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2d540b1b40 0x7f2d540ca4a0 secure :-1 s=READY pgs=2414 cs=0 l=1 rev1=1 crypto rx=0x7f2d54070010 tx=0x7f2d5002bcf0 comp rx=0 tx=0).stop 2023-10-19T14:20:33.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.292+0000 7f2d5a9a0640 1 -- 172.21.15.57:0/371304744 shutdown_connections 2023-10-19T14:20:33.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.292+0000 7f2d5a9a0640 1 --2- 172.21.15.57:0/371304744 >> v2:172.21.15.179:3300/0 conn(0x7f2d541413b0 0x7f2d5406c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:33.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.292+0000 7f2d5a9a0640 1 --2- 172.21.15.57:0/371304744 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2d540b1b40 0x7f2d540ca4a0 unknown :-1 s=CLOSED pgs=2414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:33.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.292+0000 7f2d5a9a0640 1 -- 172.21.15.57:0/371304744 >> 172.21.15.57:0/371304744 conn(0x7f2d54070080 msgr2=0x7f2d54070490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:33.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.292+0000 7f2d5a9a0640 1 -- 172.21.15.57:0/371304744 shutdown_connections 2023-10-19T14:20:33.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.292+0000 7f2d5a9a0640 1 -- 172.21.15.57:0/371304744 wait complete. 2023-10-19T14:20:33.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.293+0000 7f2d5a9a0640 1 Processor -- start 2023-10-19T14:20:33.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.293+0000 7f2d5a9a0640 1 -- start start 2023-10-19T14:20:33.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.294+0000 7f2d5a9a0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2d540b1b40 0x7f2d5413c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:33.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.294+0000 7f2d5a9a0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2d541413b0 0x7f2d54140040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:33.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.294+0000 7f2d5a9a0640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2d54140580 0x7f2d540ad5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:33.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.294+0000 7f2d5a9a0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d541542f0 con 0x7f2d541413b0 2023-10-19T14:20:33.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.294+0000 7f2d5a9a0640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d541545d0 con 0x7f2d540b1b40 2023-10-19T14:20:33.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.294+0000 7f2d5a9a0640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d541548b0 con 0x7f2d54140580 2023-10-19T14:20:33.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.294+0000 7f2d5919d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2d541413b0 0x7f2d54140040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:33.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.294+0000 7f2d5919d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2d541413b0 0x7f2d54140040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:58878/0 (socket says 172.21.15.57:58878) 2023-10-19T14:20:33.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.294+0000 7f2d5919d640 1 -- 172.21.15.57:0/2553427644 learned_addr learned my addr 172.21.15.57:0/2553427644 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:33.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.294+0000 7f2d5999e640 1 --2- 172.21.15.57:0/2553427644 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2d540b1b40 0x7f2d5413c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:33.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.294+0000 7f2d5a19f640 1 --2- 172.21.15.57:0/2553427644 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2d54140580 0x7f2d540ad5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:33.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.294+0000 7f2d5919d640 1 -- 172.21.15.57:0/2553427644 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2d54140580 msgr2=0x7f2d540ad5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:33.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.294+0000 7f2d5919d640 1 --2- 172.21.15.57:0/2553427644 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2d54140580 0x7f2d540ad5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:33.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.294+0000 7f2d5919d640 1 -- 172.21.15.57:0/2553427644 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2d540b1b40 msgr2=0x7f2d5413c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:33.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.294+0000 7f2d5919d640 1 --2- 172.21.15.57:0/2553427644 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2d540b1b40 0x7f2d5413c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:33.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.294+0000 7f2d5919d640 1 -- 172.21.15.57:0/2553427644 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d50009490 con 0x7f2d541413b0 2023-10-19T14:20:33.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.294+0000 7f2d5a19f640 1 --2- 172.21.15.57:0/2553427644 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2d54140580 0x7f2d540ad5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:20:33.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.294+0000 7f2d5919d640 1 --2- 172.21.15.57:0/2553427644 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2d541413b0 0x7f2d54140040 secure :-1 s=READY pgs=2415 cs=0 l=1 rev1=1 crypto rx=0x7f2d4400c430 tx=0x7f2d4400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:33.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.294+0000 7f2d4a7fc640 1 -- 172.21.15.57:0/2553427644 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d44004230 con 0x7f2d541413b0 2023-10-19T14:20:33.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.294+0000 7f2d4a7fc640 1 -- 172.21.15.57:0/2553427644 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f2d440044a0 con 0x7f2d541413b0 2023-10-19T14:20:33.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.294+0000 7f2d5a9a0640 1 -- 172.21.15.57:0/2553427644 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d54154b90 con 0x7f2d541413b0 2023-10-19T14:20:33.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.294+0000 7f2d4a7fc640 1 -- 172.21.15.57:0/2553427644 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d44003850 con 0x7f2d541413b0 2023-10-19T14:20:33.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.294+0000 7f2d5a9a0640 1 -- 172.21.15.57:0/2553427644 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d5413c7f0 con 0x7f2d541413b0 2023-10-19T14:20:33.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.295+0000 7f2d5a9a0640 1 -- 172.21.15.57:0/2553427644 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d54071fe0 con 0x7f2d541413b0 2023-10-19T14:20:33.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.296+0000 7f2d4a7fc640 1 -- 172.21.15.57:0/2553427644 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f2d440039f0 con 0x7f2d541413b0 2023-10-19T14:20:33.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.296+0000 7f2d4a7fc640 1 --2- 172.21.15.57:0/2553427644 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f2d3806af90 0x7f2d3806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:33.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.296+0000 7f2d4a7fc640 1 -- 172.21.15.57:0/2553427644 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 6921+0+0 (secure 0 0 0) 0x7f2d44089430 con 0x7f2d541413b0 2023-10-19T14:20:33.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.296+0000 7f2d5999e640 1 --2- 172.21.15.57:0/2553427644 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f2d3806af90 0x7f2d3806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:33.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.297+0000 7f2d5999e640 1 --2- 172.21.15.57:0/2553427644 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f2d3806af90 0x7f2d3806d450 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2d50036000 tx=0x7f2d50004730 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:33.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.301+0000 7f2d4a7fc640 1 -- 172.21.15.57:0/2553427644 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2d54071fe0 con 0x7f2d541413b0 2023-10-19T14:20:33.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.478+0000 7f2d5a9a0640 1 -- 172.21.15.57:0/2553427644 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2d541550b0 con 0x7f2d541413b0 2023-10-19T14:20:33.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.479+0000 7f2d4a7fc640 1 -- 172.21.15.57:0/2553427644 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v227) v1 ==== 74+0+20495 (secure 0 0 0) 0x7f2d541550b0 con 0x7f2d541413b0 2023-10-19T14:20:33.482 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:20:33.483 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":227,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T14:20:31.777665+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2023-10-19T14:20:17.357433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"222","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2023-10-19T14:20:18.049516+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"223","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_metadata","create_time":"2023-10-19T14:20:28.667664+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"226","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"backup_fs_data","create_time":"2023-10-19T14:20:29.122584+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"227","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T14:20:33.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.481+0000 7f2d5a9a0640 1 -- 172.21.15.57:0/2553427644 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f2d3806af90 msgr2=0x7f2d3806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:33.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.481+0000 7f2d5a9a0640 1 --2- 172.21.15.57:0/2553427644 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f2d3806af90 0x7f2d3806d450 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2d50036000 tx=0x7f2d50004730 comp rx=0 tx=0).stop 2023-10-19T14:20:33.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.481+0000 7f2d5a9a0640 1 -- 172.21.15.57:0/2553427644 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2d541413b0 msgr2=0x7f2d54140040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:33.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.481+0000 7f2d5a9a0640 1 --2- 172.21.15.57:0/2553427644 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2d541413b0 0x7f2d54140040 secure :-1 s=READY pgs=2415 cs=0 l=1 rev1=1 crypto rx=0x7f2d4400c430 tx=0x7f2d4400c8f0 comp rx=0 tx=0).stop 2023-10-19T14:20:33.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.481+0000 7f2d5a9a0640 1 -- 172.21.15.57:0/2553427644 shutdown_connections 2023-10-19T14:20:33.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.481+0000 7f2d5a9a0640 1 --2- 172.21.15.57:0/2553427644 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f2d3806af90 0x7f2d3806d450 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:33.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.481+0000 7f2d5a9a0640 1 --2- 172.21.15.57:0/2553427644 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2d54140580 0x7f2d540ad5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:33.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.481+0000 7f2d5a9a0640 1 --2- 172.21.15.57:0/2553427644 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2d541413b0 0x7f2d54140040 unknown :-1 s=CLOSED pgs=2415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:33.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.481+0000 7f2d5a9a0640 1 --2- 172.21.15.57:0/2553427644 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2d540b1b40 0x7f2d5413c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:33.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.481+0000 7f2d5a9a0640 1 -- 172.21.15.57:0/2553427644 >> 172.21.15.57:0/2553427644 conn(0x7f2d54070080 msgr2=0x7f2d541570b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:33.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.481+0000 7f2d5a9a0640 1 -- 172.21.15.57:0/2553427644 shutdown_connections 2023-10-19T14:20:33.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.481+0000 7f2d5a9a0640 1 -- 172.21.15.57:0/2553427644 wait complete. 2023-10-19T14:20:33.495 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-19T14:20:33.495 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-19T14:20:33.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.603+0000 7fbefe87c640 1 -- 172.21.15.57:0/2791381217 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbee0004d00 msgr2=0x7fbee0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:33.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.603+0000 7fbefe87c640 1 --2- 172.21.15.57:0/2791381217 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbee0004d00 0x7fbee0005180 secure :-1 s=READY pgs=2417 cs=0 l=1 rev1=1 crypto rx=0x7fbef0009ed0 tx=0x7fbee00057f0 comp rx=0 tx=0).stop 2023-10-19T14:20:33.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.603+0000 7fbefce19640 1 -- 172.21.15.57:0/2791381217 reap_dead start 2023-10-19T14:20:33.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.603+0000 7fbefe87c640 1 -- 172.21.15.57:0/2791381217 shutdown_connections 2023-10-19T14:20:33.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.603+0000 7fbefe87c640 1 -- 172.21.15.57:0/2791381217 >> 172.21.15.57:0/2791381217 conn(0x7fbef80d0d60 msgr2=0x7fbef80d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:33.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.603+0000 7fbefe87c640 1 -- 172.21.15.57:0/2791381217 shutdown_connections 2023-10-19T14:20:33.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.603+0000 7fbefe87c640 1 -- 172.21.15.57:0/2791381217 wait complete. 2023-10-19T14:20:33.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.603+0000 7fbefe87c640 1 Processor -- start 2023-10-19T14:20:33.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.603+0000 7fbefe87c640 1 -- start start 2023-10-19T14:20:33.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.604+0000 7fbefe87c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbee0004d00 0x7fbef814f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:33.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.604+0000 7fbefe87c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbef80c7e30 0x7fbef80c82b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:33.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.604+0000 7fbefe87c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbef80c87f0 0x7fbef80b49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:33.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.604+0000 7fbefe87c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbef814f8b0 con 0x7fbef80c87f0 2023-10-19T14:20:33.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.604+0000 7fbefe87c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbef81e4c60 con 0x7fbef80c7e30 2023-10-19T14:20:33.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.604+0000 7fbefe87c640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbef80c8c70 con 0x7fbee0004d00 2023-10-19T14:20:33.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.604+0000 7fbefce19640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbef80c87f0 0x7fbef80b49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:33.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.604+0000 7fbefce19640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbef80c87f0 0x7fbef80b49e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:58892/0 (socket says 172.21.15.57:58892) 2023-10-19T14:20:33.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.604+0000 7fbefce19640 1 -- 172.21.15.57:0/2474120208 learned_addr learned my addr 172.21.15.57:0/2474120208 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:33.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.604+0000 7fbef77fe640 1 --2- 172.21.15.57:0/2474120208 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbef80c7e30 0x7fbef80c82b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:33.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.604+0000 7fbef7fff640 1 --2- 172.21.15.57:0/2474120208 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbee0004d00 0x7fbef814f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:33.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.604+0000 7fbefce19640 1 -- 172.21.15.57:0/2474120208 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbee0004d00 msgr2=0x7fbef814f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:33.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.604+0000 7fbefce19640 1 --2- 172.21.15.57:0/2474120208 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbee0004d00 0x7fbef814f370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:33.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.604+0000 7fbefce19640 1 -- 172.21.15.57:0/2474120208 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbef80c7e30 msgr2=0x7fbef80c82b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:33.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.604+0000 7fbefce19640 1 --2- 172.21.15.57:0/2474120208 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbef80c7e30 0x7fbef80c82b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:33.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.604+0000 7fbefce19640 1 -- 172.21.15.57:0/2474120208 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbef81e4050 con 0x7fbef80c87f0 2023-10-19T14:20:33.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.604+0000 7fbef77fe640 1 --2- 172.21.15.57:0/2474120208 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbef80c7e30 0x7fbef80c82b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:20:33.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.604+0000 7fbefce19640 1 --2- 172.21.15.57:0/2474120208 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbef80c87f0 0x7fbef80b49e0 secure :-1 s=READY pgs=2418 cs=0 l=1 rev1=1 crypto rx=0x7fbef0031750 tx=0x7fbef0031f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:33.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.604+0000 7fbef4ff9640 1 -- 172.21.15.57:0/2474120208 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef003f070 con 0x7fbef80c87f0 2023-10-19T14:20:33.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.605+0000 7fbef4ff9640 1 -- 172.21.15.57:0/2474120208 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fbef0009200 con 0x7fbef80c87f0 2023-10-19T14:20:33.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.605+0000 7fbef4ff9640 1 -- 172.21.15.57:0/2474120208 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef0003bb0 con 0x7fbef80c87f0 2023-10-19T14:20:33.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.605+0000 7fbefe87c640 1 -- 172.21.15.57:0/2474120208 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbef81e4330 con 0x7fbef80c87f0 2023-10-19T14:20:33.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.605+0000 7fbefe87c640 1 -- 172.21.15.57:0/2474120208 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbef81e48f0 con 0x7fbef80c87f0 2023-10-19T14:20:33.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.606+0000 7fbefe87c640 1 -- 172.21.15.57:0/2474120208 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbef81e4a80 con 0x7fbef80c87f0 2023-10-19T14:20:33.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.606+0000 7fbef4ff9640 1 -- 172.21.15.57:0/2474120208 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fbef0003d10 con 0x7fbef80c87f0 2023-10-19T14:20:33.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.606+0000 7fbef4ff9640 1 --2- 172.21.15.57:0/2474120208 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fbee406af90 0x7fbee406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:33.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.607+0000 7fbef4ff9640 1 -- 172.21.15.57:0/2474120208 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 6921+0+0 (secure 0 0 0) 0x7fbef00aca00 con 0x7fbef80c87f0 2023-10-19T14:20:33.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.607+0000 7fbef7fff640 1 --2- 172.21.15.57:0/2474120208 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fbee406af90 0x7fbee406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:33.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.607+0000 7fbef7fff640 1 --2- 172.21.15.57:0/2474120208 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fbee406af90 0x7fbee406d450 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fbee8001ce0 tx=0x7fbee8001bd0 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:33.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.611+0000 7fbef4ff9640 1 -- 172.21.15.57:0/2474120208 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fbef00773c0 con 0x7fbef80c87f0 2023-10-19T14:20:33.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.783+0000 7fbefe87c640 1 -- 172.21.15.57:0/2474120208 --> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbef80690e0 con 0x7fbee406af90 2023-10-19T14:20:33.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.798+0000 7fbef4ff9640 1 -- 172.21.15.57:0/2474120208 <== mgr.9107 v2:172.21.15.179:6832/3232482678 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+528945 (secure 0 0 0) 0x7fbef80690e0 con 0x7fbee406af90 2023-10-19T14:20:33.806 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:20:33.810 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2023-10-19T14:20:33.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.802+0000 7fbefe87c640 1 -- 172.21.15.57:0/2474120208 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fbee406af90 msgr2=0x7fbee406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:33.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.802+0000 7fbefe87c640 1 --2- 172.21.15.57:0/2474120208 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fbee406af90 0x7fbee406d450 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fbee8001ce0 tx=0x7fbee8001bd0 comp rx=0 tx=0).stop 2023-10-19T14:20:33.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.802+0000 7fbefe87c640 1 -- 172.21.15.57:0/2474120208 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbef80c87f0 msgr2=0x7fbef80b49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:33.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.802+0000 7fbefe87c640 1 --2- 172.21.15.57:0/2474120208 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbef80c87f0 0x7fbef80b49e0 secure :-1 s=READY pgs=2418 cs=0 l=1 rev1=1 crypto rx=0x7fbef0031750 tx=0x7fbef0031f90 comp rx=0 tx=0).stop 2023-10-19T14:20:33.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.802+0000 7fbefe87c640 1 -- 172.21.15.57:0/2474120208 shutdown_connections 2023-10-19T14:20:33.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.802+0000 7fbefe87c640 1 --2- 172.21.15.57:0/2474120208 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fbee406af90 0x7fbee406d450 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:33.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.802+0000 7fbefe87c640 1 --2- 172.21.15.57:0/2474120208 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbef80c87f0 0x7fbef80b49e0 unknown :-1 s=CLOSED pgs=2418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:33.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.802+0000 7fbefe87c640 1 --2- 172.21.15.57:0/2474120208 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbef80c7e30 0x7fbef80c82b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:33.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.802+0000 7fbefe87c640 1 --2- 172.21.15.57:0/2474120208 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbee0004d00 0x7fbef814f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:33.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.802+0000 7fbefe87c640 1 -- 172.21.15.57:0/2474120208 >> 172.21.15.57:0/2474120208 conn(0x7fbef80d0d60 msgr2=0x7fbef80ce820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:33.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.802+0000 7fbefe87c640 1 -- 172.21.15.57:0/2474120208 shutdown_connections 2023-10-19T14:20:33.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.802+0000 7fbefe87c640 1 -- 172.21.15.57:0/2474120208 wait complete. 2023-10-19T14:20:33.815 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":32,"stamp":"2023-10-19T14:20:31.991749+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596444,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":2210,"num_read_kb":4624,"num_write":680,"num_write_kb":2446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3265,"ondisk_log_size":3265,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":328,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":411736,"kb_used_data":23460,"kb_used_omap":12,"kb_used_meta":388147,"kb_avail":754562984,"statfs":{"total":773094113280,"available":772672495616,"internally_reserved":0,"allocated":24023040,"data_stored":12849901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":397463097},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3225,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001099"},"pg_stats":[{"pgid":"30.1f","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.784610+0000","last_change":"2023-10-19T14:20:30.803400+0000","last_active":"2023-10-19T14:20:31.784610+0000","last_peered":"2023-10-19T14:20:31.784610+0000","last_clean":"2023-10-19T14:20:31.784610+0000","last_became_active":"2023-10-19T14:20:30.803141+0000","last_became_peered":"2023-10-19T14:20:30.803141+0000","last_unstale":"2023-10-19T14:20:31.784610+0000","last_undegraded":"2023-10-19T14:20:31.784610+0000","last_fullsized":"2023-10-19T14:20:31.784610+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:44:52.163309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"211'3186","reported_seq":4086,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.786003+0000","last_change":"2023-10-19T12:44:08.636613+0000","last_active":"2023-10-19T14:20:29.786003+0000","last_peered":"2023-10-19T14:20:29.786003+0000","last_clean":"2023-10-19T14:20:29.786003+0000","last_became_active":"2023-10-19T12:44:08.635697+0000","last_became_peered":"2023-10-19T12:44:08.635697+0000","last_unstale":"2023-10-19T14:20:29.786003+0000","last_undegraded":"2023-10-19T14:20:29.786003+0000","last_fullsized":"2023-10-19T14:20:29.786003+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_clean_scrub_stamp":"2023-10-19T12:44:07.399738+0000","objects_scrubbed":0,"log_size":3186,"log_dups_size":0,"ondisk_log_size":3186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:15:23.685485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2210,"num_read_kb":4624,"num_write":589,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1a","version":"222'1","reported_seq":17,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.786649+0000","last_change":"2023-10-19T14:20:18.723791+0000","last_active":"2023-10-19T14:20:30.786649+0000","last_peered":"2023-10-19T14:20:30.786649+0000","last_clean":"2023-10-19T14:20:30.786649+0000","last_became_active":"2023-10-19T14:20:18.723476+0000","last_became_peered":"2023-10-19T14:20:18.723476+0000","last_unstale":"2023-10-19T14:20:30.786649+0000","last_undegraded":"2023-10-19T14:20:30.786649+0000","last_fullsized":"2023-10-19T14:20:30.786649+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:07:17.723824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1d","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.782044+0000","last_change":"2023-10-19T14:20:19.350783+0000","last_active":"2023-10-19T14:20:30.782044+0000","last_peered":"2023-10-19T14:20:30.782044+0000","last_clean":"2023-10-19T14:20:30.782044+0000","last_became_active":"2023-10-19T14:20:19.350441+0000","last_became_peered":"2023-10-19T14:20:19.350441+0000","last_unstale":"2023-10-19T14:20:30.782044+0000","last_undegraded":"2023-10-19T14:20:30.782044+0000","last_fullsized":"2023-10-19T14:20:30.782044+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:47:00.028101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.1c","version":"226'2","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.812900+0000","last_change":"2023-10-19T14:20:29.796419+0000","last_active":"2023-10-19T14:20:30.812900+0000","last_peered":"2023-10-19T14:20:30.812900+0000","last_clean":"2023-10-19T14:20:30.812900+0000","last_became_active":"2023-10-19T14:20:29.796167+0000","last_became_peered":"2023-10-19T14:20:29.796167+0000","last_unstale":"2023-10-19T14:20:30.812900+0000","last_undegraded":"2023-10-19T14:20:30.812900+0000","last_fullsized":"2023-10-19T14:20:30.812900+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_clean_scrub_stamp":"2023-10-19T14:20:28.763908+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:47:00.349856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.5","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.804883+0000","last_change":"2023-10-19T14:20:30.798451+0000","last_active":"2023-10-19T14:20:30.804883+0000","last_peered":"2023-10-19T14:20:30.804883+0000","last_clean":"2023-10-19T14:20:30.804883+0000","last_became_active":"2023-10-19T14:20:30.796684+0000","last_became_peered":"2023-10-19T14:20:30.796684+0000","last_unstale":"2023-10-19T14:20:30.804883+0000","last_undegraded":"2023-10-19T14:20:30.804883+0000","last_fullsized":"2023-10-19T14:20:30.804883+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:10:41.246540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":15,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.782348+0000","last_change":"2023-10-19T14:20:18.722143+0000","last_active":"2023-10-19T14:20:29.782348+0000","last_peered":"2023-10-19T14:20:29.782348+0000","last_clean":"2023-10-19T14:20:29.782348+0000","last_became_active":"2023-10-19T14:20:18.721853+0000","last_became_peered":"2023-10-19T14:20:18.721853+0000","last_unstale":"2023-10-19T14:20:29.782348+0000","last_undegraded":"2023-10-19T14:20:29.782348+0000","last_fullsized":"2023-10-19T14:20:29.782348+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:01:31.494525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.7","version":"0'0","reported_seq":14,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.788060+0000","last_change":"2023-10-19T14:20:19.347562+0000","last_active":"2023-10-19T14:20:29.788060+0000","last_peered":"2023-10-19T14:20:29.788060+0000","last_clean":"2023-10-19T14:20:29.788060+0000","last_became_active":"2023-10-19T14:20:19.347286+0000","last_became_peered":"2023-10-19T14:20:19.347286+0000","last_unstale":"2023-10-19T14:20:29.788060+0000","last_undegraded":"2023-10-19T14:20:29.788060+0000","last_fullsized":"2023-10-19T14:20:29.788060+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:06:12.825567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.6","version":"0'0","reported_seq":11,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.788851+0000","last_change":"2023-10-19T14:20:29.796583+0000","last_active":"2023-10-19T14:20:30.788851+0000","last_peered":"2023-10-19T14:20:30.788851+0000","last_clean":"2023-10-19T14:20:30.788851+0000","last_became_active":"2023-10-19T14:20:29.796107+0000","last_became_peered":"2023-10-19T14:20:29.796107+0000","last_unstale":"2023-10-19T14:20:30.788851+0000","last_undegraded":"2023-10-19T14:20:30.788851+0000","last_fullsized":"2023-10-19T14:20:30.788851+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_clean_scrub_stamp":"2023-10-19T14:20:28.763908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:26:10.775357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.4","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.785627+0000","last_change":"2023-10-19T14:20:30.794356+0000","last_active":"2023-10-19T14:20:31.785627+0000","last_peered":"2023-10-19T14:20:31.785627+0000","last_clean":"2023-10-19T14:20:31.785627+0000","last_became_active":"2023-10-19T14:20:30.794101+0000","last_became_peered":"2023-10-19T14:20:30.794101+0000","last_unstale":"2023-10-19T14:20:31.785627+0000","last_undegraded":"2023-10-19T14:20:31.785627+0000","last_fullsized":"2023-10-19T14:20:31.785627+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:13:36.493553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":16,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.782732+0000","last_change":"2023-10-19T14:20:18.723906+0000","last_active":"2023-10-19T14:20:30.782732+0000","last_peered":"2023-10-19T14:20:30.782732+0000","last_clean":"2023-10-19T14:20:30.782732+0000","last_became_active":"2023-10-19T14:20:18.723585+0000","last_became_peered":"2023-10-19T14:20:18.723585+0000","last_unstale":"2023-10-19T14:20:30.782732+0000","last_undegraded":"2023-10-19T14:20:30.782732+0000","last_fullsized":"2023-10-19T14:20:30.782732+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:28:48.261352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":14,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.787461+0000","last_change":"2023-10-19T14:20:19.345421+0000","last_active":"2023-10-19T14:20:29.787461+0000","last_peered":"2023-10-19T14:20:29.787461+0000","last_clean":"2023-10-19T14:20:29.787461+0000","last_became_active":"2023-10-19T14:20:19.345091+0000","last_became_peered":"2023-10-19T14:20:19.345091+0000","last_unstale":"2023-10-19T14:20:29.787461+0000","last_undegraded":"2023-10-19T14:20:29.787461+0000","last_fullsized":"2023-10-19T14:20:29.787461+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:49:04.080016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.7","version":"226'2","reported_seq":14,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.814420+0000","last_change":"2023-10-19T14:20:29.800281+0000","last_active":"2023-10-19T14:20:31.814420+0000","last_peered":"2023-10-19T14:20:31.814420+0000","last_clean":"2023-10-19T14:20:31.814420+0000","last_became_active":"2023-10-19T14:20:29.799527+0000","last_became_peered":"2023-10-19T14:20:29.799527+0000","last_unstale":"2023-10-19T14:20:31.814420+0000","last_undegraded":"2023-10-19T14:20:31.814420+0000","last_fullsized":"2023-10-19T14:20:31.814420+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_clean_scrub_stamp":"2023-10-19T14:20:28.763908+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:32:29.418946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.7","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.794051+0000","last_change":"2023-10-19T14:20:30.792541+0000","last_active":"2023-10-19T14:20:30.794051+0000","last_peered":"2023-10-19T14:20:30.794051+0000","last_clean":"2023-10-19T14:20:30.794051+0000","last_became_active":"2023-10-19T14:20:30.792275+0000","last_became_peered":"2023-10-19T14:20:30.792275+0000","last_unstale":"2023-10-19T14:20:30.794051+0000","last_undegraded":"2023-10-19T14:20:30.794051+0000","last_fullsized":"2023-10-19T14:20:30.794051+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:34:03.301288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":16,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.786736+0000","last_change":"2023-10-19T14:20:18.722453+0000","last_active":"2023-10-19T14:20:30.786736+0000","last_peered":"2023-10-19T14:20:30.786736+0000","last_clean":"2023-10-19T14:20:30.786736+0000","last_became_active":"2023-10-19T14:20:18.722162+0000","last_became_peered":"2023-10-19T14:20:18.722162+0000","last_unstale":"2023-10-19T14:20:30.786736+0000","last_undegraded":"2023-10-19T14:20:30.786736+0000","last_fullsized":"2023-10-19T14:20:30.786736+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:49:03.077978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":14,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.787669+0000","last_change":"2023-10-19T14:20:19.344708+0000","last_active":"2023-10-19T14:20:29.787669+0000","last_peered":"2023-10-19T14:20:29.787669+0000","last_clean":"2023-10-19T14:20:29.787669+0000","last_became_active":"2023-10-19T14:20:19.344380+0000","last_became_peered":"2023-10-19T14:20:19.344380+0000","last_unstale":"2023-10-19T14:20:29.787669+0000","last_undegraded":"2023-10-19T14:20:29.787669+0000","last_fullsized":"2023-10-19T14:20:29.787669+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:37:55.455379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.4","version":"0'0","reported_seq":12,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.814069+0000","last_change":"2023-10-19T14:20:29.798930+0000","last_active":"2023-10-19T14:20:31.814069+0000","last_peered":"2023-10-19T14:20:31.814069+0000","last_clean":"2023-10-19T14:20:31.814069+0000","last_became_active":"2023-10-19T14:20:29.798658+0000","last_became_peered":"2023-10-19T14:20:29.798658+0000","last_unstale":"2023-10-19T14:20:31.814069+0000","last_undegraded":"2023-10-19T14:20:31.814069+0000","last_fullsized":"2023-10-19T14:20:31.814069+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_clean_scrub_stamp":"2023-10-19T14:20:28.763908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:36:37.487929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.6","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.782950+0000","last_change":"2023-10-19T14:20:30.801925+0000","last_active":"2023-10-19T14:20:31.782950+0000","last_peered":"2023-10-19T14:20:31.782950+0000","last_clean":"2023-10-19T14:20:31.782950+0000","last_became_active":"2023-10-19T14:20:30.801564+0000","last_became_peered":"2023-10-19T14:20:30.801564+0000","last_unstale":"2023-10-19T14:20:31.782950+0000","last_undegraded":"2023-10-19T14:20:31.782950+0000","last_fullsized":"2023-10-19T14:20:31.782950+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:00:34.306103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":16,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.800098+0000","last_change":"2023-10-19T14:20:18.723898+0000","last_active":"2023-10-19T14:20:30.800098+0000","last_peered":"2023-10-19T14:20:30.800098+0000","last_clean":"2023-10-19T14:20:30.800098+0000","last_became_active":"2023-10-19T14:20:18.722703+0000","last_became_peered":"2023-10-19T14:20:18.722703+0000","last_unstale":"2023-10-19T14:20:30.800098+0000","last_undegraded":"2023-10-19T14:20:30.800098+0000","last_fullsized":"2023-10-19T14:20:30.800098+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:25:51.334964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.797684+0000","last_change":"2023-10-19T14:20:19.345692+0000","last_active":"2023-10-19T14:20:30.797684+0000","last_peered":"2023-10-19T14:20:30.797684+0000","last_clean":"2023-10-19T14:20:30.797684+0000","last_became_active":"2023-10-19T14:20:19.344861+0000","last_became_peered":"2023-10-19T14:20:19.344861+0000","last_unstale":"2023-10-19T14:20:30.797684+0000","last_undegraded":"2023-10-19T14:20:30.797684+0000","last_fullsized":"2023-10-19T14:20:30.797684+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:43:57.179927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.5","version":"0'0","reported_seq":11,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.791341+0000","last_change":"2023-10-19T14:20:29.793288+0000","last_active":"2023-10-19T14:20:30.791341+0000","last_peered":"2023-10-19T14:20:30.791341+0000","last_clean":"2023-10-19T14:20:30.791341+0000","last_became_active":"2023-10-19T14:20:29.792978+0000","last_became_peered":"2023-10-19T14:20:29.792978+0000","last_unstale":"2023-10-19T14:20:30.791341+0000","last_undegraded":"2023-10-19T14:20:30.791341+0000","last_fullsized":"2023-10-19T14:20:30.791341+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_clean_scrub_stamp":"2023-10-19T14:20:28.763908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:21:28.975578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.1","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.784439+0000","last_change":"2023-10-19T14:20:30.799002+0000","last_active":"2023-10-19T14:20:31.784439+0000","last_peered":"2023-10-19T14:20:31.784439+0000","last_clean":"2023-10-19T14:20:31.784439+0000","last_became_active":"2023-10-19T14:20:30.798128+0000","last_became_peered":"2023-10-19T14:20:30.798128+0000","last_unstale":"2023-10-19T14:20:31.784439+0000","last_undegraded":"2023-10-19T14:20:31.784439+0000","last_fullsized":"2023-10-19T14:20:31.784439+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:18:44.195844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":16,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.782167+0000","last_change":"2023-10-19T14:20:18.718760+0000","last_active":"2023-10-19T14:20:30.782167+0000","last_peered":"2023-10-19T14:20:30.782167+0000","last_clean":"2023-10-19T14:20:30.782167+0000","last_became_active":"2023-10-19T14:20:18.718459+0000","last_became_peered":"2023-10-19T14:20:18.718459+0000","last_unstale":"2023-10-19T14:20:30.782167+0000","last_undegraded":"2023-10-19T14:20:30.782167+0000","last_fullsized":"2023-10-19T14:20:30.782167+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:32:18.381107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.782276+0000","last_change":"2023-10-19T14:20:19.352214+0000","last_active":"2023-10-19T14:20:30.782276+0000","last_peered":"2023-10-19T14:20:30.782276+0000","last_clean":"2023-10-19T14:20:30.782276+0000","last_became_active":"2023-10-19T14:20:19.351789+0000","last_became_peered":"2023-10-19T14:20:19.351789+0000","last_unstale":"2023-10-19T14:20:30.782276+0000","last_undegraded":"2023-10-19T14:20:30.782276+0000","last_fullsized":"2023-10-19T14:20:30.782276+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:05:58.978059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.2","version":"0'0","reported_seq":11,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.786923+0000","last_change":"2023-10-19T14:20:29.792399+0000","last_active":"2023-10-19T14:20:30.786923+0000","last_peered":"2023-10-19T14:20:30.786923+0000","last_clean":"2023-10-19T14:20:30.786923+0000","last_became_active":"2023-10-19T14:20:29.792120+0000","last_became_peered":"2023-10-19T14:20:29.792120+0000","last_unstale":"2023-10-19T14:20:30.786923+0000","last_undegraded":"2023-10-19T14:20:30.786923+0000","last_fullsized":"2023-10-19T14:20:30.786923+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_clean_scrub_stamp":"2023-10-19T14:20:28.763908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:26:03.175641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.0","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.784658+0000","last_change":"2023-10-19T14:20:30.803671+0000","last_active":"2023-10-19T14:20:31.784658+0000","last_peered":"2023-10-19T14:20:31.784658+0000","last_clean":"2023-10-19T14:20:31.784658+0000","last_became_active":"2023-10-19T14:20:30.803352+0000","last_became_peered":"2023-10-19T14:20:30.803352+0000","last_unstale":"2023-10-19T14:20:31.784658+0000","last_undegraded":"2023-10-19T14:20:31.784658+0000","last_fullsized":"2023-10-19T14:20:31.784658+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:59:44.340796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":15,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.787522+0000","last_change":"2023-10-19T14:20:18.724127+0000","last_active":"2023-10-19T14:20:29.787522+0000","last_peered":"2023-10-19T14:20:29.787522+0000","last_clean":"2023-10-19T14:20:29.787522+0000","last_became_active":"2023-10-19T14:20:18.723847+0000","last_became_peered":"2023-10-19T14:20:18.723847+0000","last_unstale":"2023-10-19T14:20:29.787522+0000","last_undegraded":"2023-10-19T14:20:29.787522+0000","last_fullsized":"2023-10-19T14:20:29.787522+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:36:35.167691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.786444+0000","last_change":"2023-10-19T14:20:19.340725+0000","last_active":"2023-10-19T14:20:30.786444+0000","last_peered":"2023-10-19T14:20:30.786444+0000","last_clean":"2023-10-19T14:20:30.786444+0000","last_became_active":"2023-10-19T14:20:19.340284+0000","last_became_peered":"2023-10-19T14:20:19.340284+0000","last_unstale":"2023-10-19T14:20:30.786444+0000","last_undegraded":"2023-10-19T14:20:30.786444+0000","last_fullsized":"2023-10-19T14:20:30.786444+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:16:20.216590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.3","version":"0'0","reported_seq":12,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.813807+0000","last_change":"2023-10-19T14:20:29.802371+0000","last_active":"2023-10-19T14:20:31.813807+0000","last_peered":"2023-10-19T14:20:31.813807+0000","last_clean":"2023-10-19T14:20:31.813807+0000","last_became_active":"2023-10-19T14:20:29.800479+0000","last_became_peered":"2023-10-19T14:20:29.800479+0000","last_unstale":"2023-10-19T14:20:31.813807+0000","last_undegraded":"2023-10-19T14:20:31.813807+0000","last_fullsized":"2023-10-19T14:20:31.813807+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_clean_scrub_stamp":"2023-10-19T14:20:28.763908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:24:02.062518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.814024+0000","last_change":"2023-10-19T14:20:30.801362+0000","last_active":"2023-10-19T14:20:31.814024+0000","last_peered":"2023-10-19T14:20:31.814024+0000","last_clean":"2023-10-19T14:20:31.814024+0000","last_became_active":"2023-10-19T14:20:30.801064+0000","last_became_peered":"2023-10-19T14:20:30.801064+0000","last_unstale":"2023-10-19T14:20:31.814024+0000","last_undegraded":"2023-10-19T14:20:31.814024+0000","last_fullsized":"2023-10-19T14:20:31.814024+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:28:21.802555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.6","version":"223'1","reported_seq":17,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.797899+0000","last_change":"2023-10-19T14:20:18.722733+0000","last_active":"2023-10-19T14:20:30.797899+0000","last_peered":"2023-10-19T14:20:30.797899+0000","last_clean":"2023-10-19T14:20:30.797899+0000","last_became_active":"2023-10-19T14:20:18.722447+0000","last_became_peered":"2023-10-19T14:20:18.722447+0000","last_unstale":"2023-10-19T14:20:30.797899+0000","last_undegraded":"2023-10-19T14:20:30.797899+0000","last_fullsized":"2023-10-19T14:20:30.797899+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:28:23.768143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1","version":"0'0","reported_seq":14,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.784233+0000","last_change":"2023-10-19T14:20:19.344287+0000","last_active":"2023-10-19T14:20:29.784233+0000","last_peered":"2023-10-19T14:20:29.784233+0000","last_clean":"2023-10-19T14:20:29.784233+0000","last_became_active":"2023-10-19T14:20:19.344030+0000","last_became_peered":"2023-10-19T14:20:19.344030+0000","last_unstale":"2023-10-19T14:20:29.784233+0000","last_undegraded":"2023-10-19T14:20:29.784233+0000","last_fullsized":"2023-10-19T14:20:29.784233+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:35:58.143784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.0","version":"0'0","reported_seq":12,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.813395+0000","last_change":"2023-10-19T14:20:29.797496+0000","last_active":"2023-10-19T14:20:31.813395+0000","last_peered":"2023-10-19T14:20:31.813395+0000","last_clean":"2023-10-19T14:20:31.813395+0000","last_became_active":"2023-10-19T14:20:29.797194+0000","last_became_peered":"2023-10-19T14:20:29.797194+0000","last_unstale":"2023-10-19T14:20:31.813395+0000","last_undegraded":"2023-10-19T14:20:31.813395+0000","last_fullsized":"2023-10-19T14:20:31.813395+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_clean_scrub_stamp":"2023-10-19T14:20:28.763908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:42:03.650411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.794174+0000","last_change":"2023-10-19T14:20:30.788501+0000","last_active":"2023-10-19T14:20:30.794174+0000","last_peered":"2023-10-19T14:20:30.794174+0000","last_clean":"2023-10-19T14:20:30.794174+0000","last_became_active":"2023-10-19T14:20:30.788225+0000","last_became_peered":"2023-10-19T14:20:30.788225+0000","last_unstale":"2023-10-19T14:20:30.794174+0000","last_undegraded":"2023-10-19T14:20:30.794174+0000","last_fullsized":"2023-10-19T14:20:30.794174+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T02:03:58.474932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.7","version":"222'2","reported_seq":18,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.788923+0000","last_change":"2023-10-19T14:20:18.724203+0000","last_active":"2023-10-19T14:20:30.788923+0000","last_peered":"2023-10-19T14:20:30.788923+0000","last_clean":"2023-10-19T14:20:30.788923+0000","last_became_active":"2023-10-19T14:20:18.723883+0000","last_became_peered":"2023-10-19T14:20:18.723883+0000","last_unstale":"2023-10-19T14:20:30.788923+0000","last_undegraded":"2023-10-19T14:20:30.788923+0000","last_fullsized":"2023-10-19T14:20:30.788923+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:38:31.255506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.800298+0000","last_change":"2023-10-19T14:20:19.349129+0000","last_active":"2023-10-19T14:20:30.800298+0000","last_peered":"2023-10-19T14:20:30.800298+0000","last_clean":"2023-10-19T14:20:30.800298+0000","last_became_active":"2023-10-19T14:20:19.348846+0000","last_became_peered":"2023-10-19T14:20:19.348846+0000","last_unstale":"2023-10-19T14:20:30.800298+0000","last_undegraded":"2023-10-19T14:20:30.800298+0000","last_fullsized":"2023-10-19T14:20:30.800298+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:04:34.009285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1","version":"0'0","reported_seq":11,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.787548+0000","last_change":"2023-10-19T14:20:29.796999+0000","last_active":"2023-10-19T14:20:30.787548+0000","last_peered":"2023-10-19T14:20:30.787548+0000","last_clean":"2023-10-19T14:20:30.787548+0000","last_became_active":"2023-10-19T14:20:29.796769+0000","last_became_peered":"2023-10-19T14:20:29.796769+0000","last_unstale":"2023-10-19T14:20:30.787548+0000","last_undegraded":"2023-10-19T14:20:30.787548+0000","last_fullsized":"2023-10-19T14:20:30.787548+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_clean_scrub_stamp":"2023-10-19T14:20:28.763908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:23:35.556147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.d","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.804445+0000","last_change":"2023-10-19T14:20:30.797600+0000","last_active":"2023-10-19T14:20:30.804445+0000","last_peered":"2023-10-19T14:20:30.804445+0000","last_clean":"2023-10-19T14:20:30.804445+0000","last_became_active":"2023-10-19T14:20:30.797358+0000","last_became_peered":"2023-10-19T14:20:30.797358+0000","last_unstale":"2023-10-19T14:20:30.804445+0000","last_undegraded":"2023-10-19T14:20:30.804445+0000","last_fullsized":"2023-10-19T14:20:30.804445+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:32:10.739567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":16,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.782953+0000","last_change":"2023-10-19T14:20:18.715329+0000","last_active":"2023-10-19T14:20:30.782953+0000","last_peered":"2023-10-19T14:20:30.782953+0000","last_clean":"2023-10-19T14:20:30.782953+0000","last_became_active":"2023-10-19T14:20:18.715068+0000","last_became_peered":"2023-10-19T14:20:18.715068+0000","last_unstale":"2023-10-19T14:20:30.782953+0000","last_undegraded":"2023-10-19T14:20:30.782953+0000","last_fullsized":"2023-10-19T14:20:30.782953+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:35:10.527276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.f","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.797934+0000","last_change":"2023-10-19T14:20:19.346087+0000","last_active":"2023-10-19T14:20:30.797934+0000","last_peered":"2023-10-19T14:20:30.797934+0000","last_clean":"2023-10-19T14:20:30.797934+0000","last_became_active":"2023-10-19T14:20:19.345704+0000","last_became_peered":"2023-10-19T14:20:19.345704+0000","last_unstale":"2023-10-19T14:20:30.797934+0000","last_undegraded":"2023-10-19T14:20:30.797934+0000","last_fullsized":"2023-10-19T14:20:30.797934+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:53:30.184157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.e","version":"226'2","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.810714+0000","last_change":"2023-10-19T14:20:29.792465+0000","last_active":"2023-10-19T14:20:30.810714+0000","last_peered":"2023-10-19T14:20:30.810714+0000","last_clean":"2023-10-19T14:20:30.810714+0000","last_became_active":"2023-10-19T14:20:29.792174+0000","last_became_peered":"2023-10-19T14:20:29.792174+0000","last_unstale":"2023-10-19T14:20:30.810714+0000","last_undegraded":"2023-10-19T14:20:30.810714+0000","last_fullsized":"2023-10-19T14:20:30.810714+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_clean_scrub_stamp":"2023-10-19T14:20:28.763908+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:36:24.939997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.c","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.785757+0000","last_change":"2023-10-19T14:20:30.802369+0000","last_active":"2023-10-19T14:20:31.785757+0000","last_peered":"2023-10-19T14:20:31.785757+0000","last_clean":"2023-10-19T14:20:31.785757+0000","last_became_active":"2023-10-19T14:20:30.802102+0000","last_became_peered":"2023-10-19T14:20:30.802102+0000","last_unstale":"2023-10-19T14:20:31.785757+0000","last_undegraded":"2023-10-19T14:20:31.785757+0000","last_fullsized":"2023-10-19T14:20:31.785757+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:02:08.150394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":15,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.785119+0000","last_change":"2023-10-19T14:20:18.727394+0000","last_active":"2023-10-19T14:20:29.785119+0000","last_peered":"2023-10-19T14:20:29.785119+0000","last_clean":"2023-10-19T14:20:29.785119+0000","last_became_active":"2023-10-19T14:20:18.727050+0000","last_became_peered":"2023-10-19T14:20:18.727050+0000","last_unstale":"2023-10-19T14:20:29.785119+0000","last_undegraded":"2023-10-19T14:20:29.785119+0000","last_fullsized":"2023-10-19T14:20:29.785119+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:13:48.534905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.e","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.787156+0000","last_change":"2023-10-19T14:20:19.345329+0000","last_active":"2023-10-19T14:20:30.787156+0000","last_peered":"2023-10-19T14:20:30.787156+0000","last_clean":"2023-10-19T14:20:30.787156+0000","last_became_active":"2023-10-19T14:20:19.344983+0000","last_became_peered":"2023-10-19T14:20:19.344983+0000","last_unstale":"2023-10-19T14:20:30.787156+0000","last_undegraded":"2023-10-19T14:20:30.787156+0000","last_fullsized":"2023-10-19T14:20:30.787156+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:17:53.794653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.f","version":"226'1","reported_seq":13,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.785140+0000","last_change":"2023-10-19T14:20:29.795271+0000","last_active":"2023-10-19T14:20:31.785140+0000","last_peered":"2023-10-19T14:20:31.785140+0000","last_clean":"2023-10-19T14:20:31.785140+0000","last_became_active":"2023-10-19T14:20:29.794970+0000","last_became_peered":"2023-10-19T14:20:29.794970+0000","last_unstale":"2023-10-19T14:20:31.785140+0000","last_undegraded":"2023-10-19T14:20:31.785140+0000","last_fullsized":"2023-10-19T14:20:31.785140+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_clean_scrub_stamp":"2023-10-19T14:20:28.763908+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:52:55.038174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.f","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.814575+0000","last_change":"2023-10-19T14:20:30.801356+0000","last_active":"2023-10-19T14:20:31.814575+0000","last_peered":"2023-10-19T14:20:31.814575+0000","last_clean":"2023-10-19T14:20:31.814575+0000","last_became_active":"2023-10-19T14:20:30.801095+0000","last_became_peered":"2023-10-19T14:20:30.801095+0000","last_unstale":"2023-10-19T14:20:31.814575+0000","last_undegraded":"2023-10-19T14:20:31.814575+0000","last_fullsized":"2023-10-19T14:20:31.814575+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:08:12.090774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":16,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.783128+0000","last_change":"2023-10-19T14:20:18.711157+0000","last_active":"2023-10-19T14:20:30.783128+0000","last_peered":"2023-10-19T14:20:30.783128+0000","last_clean":"2023-10-19T14:20:30.783128+0000","last_became_active":"2023-10-19T14:20:18.710747+0000","last_became_peered":"2023-10-19T14:20:18.710747+0000","last_unstale":"2023-10-19T14:20:30.783128+0000","last_undegraded":"2023-10-19T14:20:30.783128+0000","last_fullsized":"2023-10-19T14:20:30.783128+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:46:05.844692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.d","version":"0'0","reported_seq":14,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.787526+0000","last_change":"2023-10-19T14:20:19.350548+0000","last_active":"2023-10-19T14:20:29.787526+0000","last_peered":"2023-10-19T14:20:29.787526+0000","last_clean":"2023-10-19T14:20:29.787526+0000","last_became_active":"2023-10-19T14:20:19.349339+0000","last_became_peered":"2023-10-19T14:20:19.349339+0000","last_unstale":"2023-10-19T14:20:29.787526+0000","last_undegraded":"2023-10-19T14:20:29.787526+0000","last_fullsized":"2023-10-19T14:20:29.787526+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:13:54.112869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.c","version":"0'0","reported_seq":12,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.813971+0000","last_change":"2023-10-19T14:20:29.798678+0000","last_active":"2023-10-19T14:20:31.813971+0000","last_peered":"2023-10-19T14:20:31.813971+0000","last_clean":"2023-10-19T14:20:31.813971+0000","last_became_active":"2023-10-19T14:20:29.798386+0000","last_became_peered":"2023-10-19T14:20:29.798386+0000","last_unstale":"2023-10-19T14:20:31.813971+0000","last_undegraded":"2023-10-19T14:20:31.813971+0000","last_fullsized":"2023-10-19T14:20:31.813971+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_clean_scrub_stamp":"2023-10-19T14:20:28.763908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:08:50.934693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.e","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.785275+0000","last_change":"2023-10-19T14:20:30.800273+0000","last_active":"2023-10-19T14:20:31.785275+0000","last_peered":"2023-10-19T14:20:31.785275+0000","last_clean":"2023-10-19T14:20:31.785275+0000","last_became_active":"2023-10-19T14:20:30.799631+0000","last_became_peered":"2023-10-19T14:20:30.799631+0000","last_unstale":"2023-10-19T14:20:31.785275+0000","last_undegraded":"2023-10-19T14:20:31.785275+0000","last_fullsized":"2023-10-19T14:20:31.785275+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:30:11.739707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.b","version":"222'2","reported_seq":17,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.782562+0000","last_change":"2023-10-19T14:20:18.719825+0000","last_active":"2023-10-19T14:20:29.782562+0000","last_peered":"2023-10-19T14:20:29.782562+0000","last_clean":"2023-10-19T14:20:29.782562+0000","last_became_active":"2023-10-19T14:20:18.719549+0000","last_became_peered":"2023-10-19T14:20:18.719549+0000","last_unstale":"2023-10-19T14:20:29.782562+0000","last_undegraded":"2023-10-19T14:20:29.782562+0000","last_fullsized":"2023-10-19T14:20:29.782562+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T02:18:13.024084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.783414+0000","last_change":"2023-10-19T14:20:19.346026+0000","last_active":"2023-10-19T14:20:30.783414+0000","last_peered":"2023-10-19T14:20:30.783414+0000","last_clean":"2023-10-19T14:20:30.783414+0000","last_became_active":"2023-10-19T14:20:19.345762+0000","last_became_peered":"2023-10-19T14:20:19.345762+0000","last_unstale":"2023-10-19T14:20:30.783414+0000","last_undegraded":"2023-10-19T14:20:30.783414+0000","last_fullsized":"2023-10-19T14:20:30.783414+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:28:21.209211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.d","version":"226'1","reported_seq":13,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.814086+0000","last_change":"2023-10-19T14:20:29.800987+0000","last_active":"2023-10-19T14:20:31.814086+0000","last_peered":"2023-10-19T14:20:31.814086+0000","last_clean":"2023-10-19T14:20:31.814086+0000","last_became_active":"2023-10-19T14:20:29.800691+0000","last_became_peered":"2023-10-19T14:20:29.800691+0000","last_unstale":"2023-10-19T14:20:31.814086+0000","last_undegraded":"2023-10-19T14:20:31.814086+0000","last_fullsized":"2023-10-19T14:20:31.814086+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_clean_scrub_stamp":"2023-10-19T14:20:28.763908+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:50:50.574035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.9","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.804333+0000","last_change":"2023-10-19T14:20:30.796772+0000","last_active":"2023-10-19T14:20:30.804333+0000","last_peered":"2023-10-19T14:20:30.804333+0000","last_clean":"2023-10-19T14:20:30.804333+0000","last_became_active":"2023-10-19T14:20:30.796517+0000","last_became_peered":"2023-10-19T14:20:30.796517+0000","last_unstale":"2023-10-19T14:20:30.804333+0000","last_undegraded":"2023-10-19T14:20:30.804333+0000","last_fullsized":"2023-10-19T14:20:30.804333+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:02:53.373912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":15,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.781908+0000","last_change":"2023-10-19T14:20:18.717037+0000","last_active":"2023-10-19T14:20:29.781908+0000","last_peered":"2023-10-19T14:20:29.781908+0000","last_clean":"2023-10-19T14:20:29.781908+0000","last_became_active":"2023-10-19T14:20:18.716731+0000","last_became_peered":"2023-10-19T14:20:18.716731+0000","last_unstale":"2023-10-19T14:20:29.781908+0000","last_undegraded":"2023-10-19T14:20:29.781908+0000","last_fullsized":"2023-10-19T14:20:29.781908+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:50:59.017027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.b","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.800006+0000","last_change":"2023-10-19T14:20:19.352403+0000","last_active":"2023-10-19T14:20:30.800006+0000","last_peered":"2023-10-19T14:20:30.800006+0000","last_clean":"2023-10-19T14:20:30.800006+0000","last_became_active":"2023-10-19T14:20:19.351892+0000","last_became_peered":"2023-10-19T14:20:19.351892+0000","last_unstale":"2023-10-19T14:20:30.800006+0000","last_undegraded":"2023-10-19T14:20:30.800006+0000","last_fullsized":"2023-10-19T14:20:30.800006+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:33:52.226441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.a","version":"0'0","reported_seq":11,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.787934+0000","last_change":"2023-10-19T14:20:29.788981+0000","last_active":"2023-10-19T14:20:30.787934+0000","last_peered":"2023-10-19T14:20:30.787934+0000","last_clean":"2023-10-19T14:20:30.787934+0000","last_became_active":"2023-10-19T14:20:29.788699+0000","last_became_peered":"2023-10-19T14:20:29.788699+0000","last_unstale":"2023-10-19T14:20:30.787934+0000","last_undegraded":"2023-10-19T14:20:30.787934+0000","last_fullsized":"2023-10-19T14:20:30.787934+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_clean_scrub_stamp":"2023-10-19T14:20:28.763908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:49:14.148270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.8","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.813308+0000","last_change":"2023-10-19T14:20:30.800475+0000","last_active":"2023-10-19T14:20:31.813308+0000","last_peered":"2023-10-19T14:20:31.813308+0000","last_clean":"2023-10-19T14:20:31.813308+0000","last_became_active":"2023-10-19T14:20:30.797151+0000","last_became_peered":"2023-10-19T14:20:30.797151+0000","last_unstale":"2023-10-19T14:20:31.813308+0000","last_undegraded":"2023-10-19T14:20:31.813308+0000","last_fullsized":"2023-10-19T14:20:31.813308+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:41:36.778953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.d","version":"222'1","reported_seq":17,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.788115+0000","last_change":"2023-10-19T14:20:18.719857+0000","last_active":"2023-10-19T14:20:30.788115+0000","last_peered":"2023-10-19T14:20:30.788115+0000","last_clean":"2023-10-19T14:20:30.788115+0000","last_became_active":"2023-10-19T14:20:18.719586+0000","last_became_peered":"2023-10-19T14:20:18.719586+0000","last_unstale":"2023-10-19T14:20:30.788115+0000","last_undegraded":"2023-10-19T14:20:30.788115+0000","last_fullsized":"2023-10-19T14:20:30.788115+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:12:37.668857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.782665+0000","last_change":"2023-10-19T14:20:19.332710+0000","last_active":"2023-10-19T14:20:30.782665+0000","last_peered":"2023-10-19T14:20:30.782665+0000","last_clean":"2023-10-19T14:20:30.782665+0000","last_became_active":"2023-10-19T14:20:19.332132+0000","last_became_peered":"2023-10-19T14:20:19.332132+0000","last_unstale":"2023-10-19T14:20:30.782665+0000","last_undegraded":"2023-10-19T14:20:30.782665+0000","last_fullsized":"2023-10-19T14:20:30.782665+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:36:15.681007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.b","version":"226'2","reported_seq":14,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.813825+0000","last_change":"2023-10-19T14:20:29.801095+0000","last_active":"2023-10-19T14:20:31.813825+0000","last_peered":"2023-10-19T14:20:31.813825+0000","last_clean":"2023-10-19T14:20:31.813825+0000","last_became_active":"2023-10-19T14:20:29.799487+0000","last_became_peered":"2023-10-19T14:20:29.799487+0000","last_unstale":"2023-10-19T14:20:31.813825+0000","last_undegraded":"2023-10-19T14:20:31.813825+0000","last_fullsized":"2023-10-19T14:20:31.813825+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_clean_scrub_stamp":"2023-10-19T14:20:28.763908+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:48:54.823238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.b","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.784907+0000","last_change":"2023-10-19T14:20:30.804612+0000","last_active":"2023-10-19T14:20:31.784907+0000","last_peered":"2023-10-19T14:20:31.784907+0000","last_clean":"2023-10-19T14:20:31.784907+0000","last_became_active":"2023-10-19T14:20:30.804342+0000","last_became_peered":"2023-10-19T14:20:30.804342+0000","last_unstale":"2023-10-19T14:20:31.784907+0000","last_undegraded":"2023-10-19T14:20:31.784907+0000","last_fullsized":"2023-10-19T14:20:31.784907+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:36:59.617065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.e","version":"222'2","reported_seq":18,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.781973+0000","last_change":"2023-10-19T14:20:18.721114+0000","last_active":"2023-10-19T14:20:30.781973+0000","last_peered":"2023-10-19T14:20:30.781973+0000","last_clean":"2023-10-19T14:20:30.781973+0000","last_became_active":"2023-10-19T14:20:18.720840+0000","last_became_peered":"2023-10-19T14:20:18.720840+0000","last_unstale":"2023-10-19T14:20:30.781973+0000","last_undegraded":"2023-10-19T14:20:30.781973+0000","last_fullsized":"2023-10-19T14:20:30.781973+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:44:50.967568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.782688+0000","last_change":"2023-10-19T14:20:19.348569+0000","last_active":"2023-10-19T14:20:30.782688+0000","last_peered":"2023-10-19T14:20:30.782688+0000","last_clean":"2023-10-19T14:20:30.782688+0000","last_became_active":"2023-10-19T14:20:19.348268+0000","last_became_peered":"2023-10-19T14:20:19.348268+0000","last_unstale":"2023-10-19T14:20:30.782688+0000","last_undegraded":"2023-10-19T14:20:30.782688+0000","last_fullsized":"2023-10-19T14:20:30.782688+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:54:41.219828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.8","version":"0'0","reported_seq":11,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.780209+0000","last_change":"2023-10-19T14:20:29.792764+0000","last_active":"2023-10-19T14:20:30.780209+0000","last_peered":"2023-10-19T14:20:30.780209+0000","last_clean":"2023-10-19T14:20:30.780209+0000","last_became_active":"2023-10-19T14:20:29.792199+0000","last_became_peered":"2023-10-19T14:20:29.792199+0000","last_unstale":"2023-10-19T14:20:30.780209+0000","last_undegraded":"2023-10-19T14:20:30.780209+0000","last_fullsized":"2023-10-19T14:20:30.780209+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_clean_scrub_stamp":"2023-10-19T14:20:28.763908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:13:59.605197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.a","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.785546+0000","last_change":"2023-10-19T14:20:30.803004+0000","last_active":"2023-10-19T14:20:31.785546+0000","last_peered":"2023-10-19T14:20:31.785546+0000","last_clean":"2023-10-19T14:20:31.785546+0000","last_became_active":"2023-10-19T14:20:30.802745+0000","last_became_peered":"2023-10-19T14:20:30.802745+0000","last_unstale":"2023-10-19T14:20:31.785546+0000","last_undegraded":"2023-10-19T14:20:31.785546+0000","last_fullsized":"2023-10-19T14:20:31.785546+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:54:08.144842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.f","version":"222'1","reported_seq":17,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.781971+0000","last_change":"2023-10-19T14:20:18.719346+0000","last_active":"2023-10-19T14:20:30.781971+0000","last_peered":"2023-10-19T14:20:30.781971+0000","last_clean":"2023-10-19T14:20:30.781971+0000","last_became_active":"2023-10-19T14:20:18.719101+0000","last_became_peered":"2023-10-19T14:20:18.719101+0000","last_unstale":"2023-10-19T14:20:30.781971+0000","last_undegraded":"2023-10-19T14:20:30.781971+0000","last_fullsized":"2023-10-19T14:20:30.781971+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:01:11.976346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.800218+0000","last_change":"2023-10-19T14:20:19.349188+0000","last_active":"2023-10-19T14:20:30.800218+0000","last_peered":"2023-10-19T14:20:30.800218+0000","last_clean":"2023-10-19T14:20:30.800218+0000","last_became_active":"2023-10-19T14:20:19.348905+0000","last_became_peered":"2023-10-19T14:20:19.348905+0000","last_unstale":"2023-10-19T14:20:30.800218+0000","last_undegraded":"2023-10-19T14:20:30.800218+0000","last_fullsized":"2023-10-19T14:20:30.800218+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:36:36.169729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.9","version":"0'0","reported_seq":11,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.788328+0000","last_change":"2023-10-19T14:20:29.797428+0000","last_active":"2023-10-19T14:20:30.788328+0000","last_peered":"2023-10-19T14:20:30.788328+0000","last_clean":"2023-10-19T14:20:30.788328+0000","last_became_active":"2023-10-19T14:20:29.797186+0000","last_became_peered":"2023-10-19T14:20:29.797186+0000","last_unstale":"2023-10-19T14:20:30.788328+0000","last_undegraded":"2023-10-19T14:20:30.788328+0000","last_fullsized":"2023-10-19T14:20:30.788328+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_clean_scrub_stamp":"2023-10-19T14:20:28.763908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:18:59.806364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.15","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.803321+0000","last_change":"2023-10-19T14:20:30.797685+0000","last_active":"2023-10-19T14:20:30.803321+0000","last_peered":"2023-10-19T14:20:30.803321+0000","last_clean":"2023-10-19T14:20:30.803321+0000","last_became_active":"2023-10-19T14:20:30.797418+0000","last_became_peered":"2023-10-19T14:20:30.797418+0000","last_unstale":"2023-10-19T14:20:30.803321+0000","last_undegraded":"2023-10-19T14:20:30.803321+0000","last_fullsized":"2023-10-19T14:20:30.803321+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:53:03.050006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.10","version":"222'8","reported_seq":23,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.782257+0000","last_change":"2023-10-19T14:20:18.722891+0000","last_active":"2023-10-19T14:20:29.782257+0000","last_peered":"2023-10-19T14:20:29.782257+0000","last_clean":"2023-10-19T14:20:29.782257+0000","last_became_active":"2023-10-19T14:20:18.722633+0000","last_became_peered":"2023-10-19T14:20:18.722633+0000","last_unstale":"2023-10-19T14:20:29.782257+0000","last_undegraded":"2023-10-19T14:20:29.782257+0000","last_fullsized":"2023-10-19T14:20:29.782257+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:32:21.255137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":14,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.788128+0000","last_change":"2023-10-19T14:20:19.353248+0000","last_active":"2023-10-19T14:20:29.788128+0000","last_peered":"2023-10-19T14:20:29.788128+0000","last_clean":"2023-10-19T14:20:29.788128+0000","last_became_active":"2023-10-19T14:20:19.352973+0000","last_became_peered":"2023-10-19T14:20:19.352973+0000","last_unstale":"2023-10-19T14:20:29.788128+0000","last_undegraded":"2023-10-19T14:20:29.788128+0000","last_fullsized":"2023-10-19T14:20:29.788128+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:07:18.725863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.16","version":"0'0","reported_seq":11,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.789745+0000","last_change":"2023-10-19T14:20:29.796603+0000","last_active":"2023-10-19T14:20:30.789745+0000","last_peered":"2023-10-19T14:20:30.789745+0000","last_clean":"2023-10-19T14:20:30.789745+0000","last_became_active":"2023-10-19T14:20:29.796056+0000","last_became_peered":"2023-10-19T14:20:29.796056+0000","last_unstale":"2023-10-19T14:20:30.789745+0000","last_undegraded":"2023-10-19T14:20:30.789745+0000","last_fullsized":"2023-10-19T14:20:30.789745+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_clean_scrub_stamp":"2023-10-19T14:20:28.763908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:18:58.754296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.14","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.806260+0000","last_change":"2023-10-19T14:20:30.800049+0000","last_active":"2023-10-19T14:20:30.806260+0000","last_peered":"2023-10-19T14:20:30.806260+0000","last_clean":"2023-10-19T14:20:30.806260+0000","last_became_active":"2023-10-19T14:20:30.799751+0000","last_became_peered":"2023-10-19T14:20:30.799751+0000","last_unstale":"2023-10-19T14:20:30.806260+0000","last_undegraded":"2023-10-19T14:20:30.806260+0000","last_fullsized":"2023-10-19T14:20:30.806260+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T02:16:51.635514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":16,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.786371+0000","last_change":"2023-10-19T14:20:18.717768+0000","last_active":"2023-10-19T14:20:30.786371+0000","last_peered":"2023-10-19T14:20:30.786371+0000","last_clean":"2023-10-19T14:20:30.786371+0000","last_became_active":"2023-10-19T14:20:18.717375+0000","last_became_peered":"2023-10-19T14:20:18.717375+0000","last_unstale":"2023-10-19T14:20:30.786371+0000","last_undegraded":"2023-10-19T14:20:30.786371+0000","last_fullsized":"2023-10-19T14:20:30.786371+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:44:02.586786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.787066+0000","last_change":"2023-10-19T14:20:19.343343+0000","last_active":"2023-10-19T14:20:30.787066+0000","last_peered":"2023-10-19T14:20:30.787066+0000","last_clean":"2023-10-19T14:20:30.787066+0000","last_became_active":"2023-10-19T14:20:19.343059+0000","last_became_peered":"2023-10-19T14:20:19.343059+0000","last_unstale":"2023-10-19T14:20:30.787066+0000","last_undegraded":"2023-10-19T14:20:30.787066+0000","last_fullsized":"2023-10-19T14:20:30.787066+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:31:59.664720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.17","version":"0'0","reported_seq":11,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.786321+0000","last_change":"2023-10-19T14:20:29.799948+0000","last_active":"2023-10-19T14:20:30.786321+0000","last_peered":"2023-10-19T14:20:30.786321+0000","last_clean":"2023-10-19T14:20:30.786321+0000","last_became_active":"2023-10-19T14:20:29.799698+0000","last_became_peered":"2023-10-19T14:20:29.799698+0000","last_unstale":"2023-10-19T14:20:30.786321+0000","last_undegraded":"2023-10-19T14:20:30.786321+0000","last_fullsized":"2023-10-19T14:20:30.786321+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_clean_scrub_stamp":"2023-10-19T14:20:28.763908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:18:03.862907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.17","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.795489+0000","last_change":"2023-10-19T14:20:30.794377+0000","last_active":"2023-10-19T14:20:30.795489+0000","last_peered":"2023-10-19T14:20:30.795489+0000","last_clean":"2023-10-19T14:20:30.795489+0000","last_became_active":"2023-10-19T14:20:30.794122+0000","last_became_peered":"2023-10-19T14:20:30.794122+0000","last_unstale":"2023-10-19T14:20:30.795489+0000","last_undegraded":"2023-10-19T14:20:30.795489+0000","last_fullsized":"2023-10-19T14:20:30.795489+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:23:07.047251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":15,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.787462+0000","last_change":"2023-10-19T14:20:18.709786+0000","last_active":"2023-10-19T14:20:29.787462+0000","last_peered":"2023-10-19T14:20:29.787462+0000","last_clean":"2023-10-19T14:20:29.787462+0000","last_became_active":"2023-10-19T14:20:18.709496+0000","last_became_peered":"2023-10-19T14:20:18.709496+0000","last_unstale":"2023-10-19T14:20:29.787462+0000","last_undegraded":"2023-10-19T14:20:29.787462+0000","last_fullsized":"2023-10-19T14:20:29.787462+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:46:59.026062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.15","version":"0'0","reported_seq":14,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.790018+0000","last_change":"2023-10-19T14:20:19.349089+0000","last_active":"2023-10-19T14:20:29.790018+0000","last_peered":"2023-10-19T14:20:29.790018+0000","last_clean":"2023-10-19T14:20:29.790018+0000","last_became_active":"2023-10-19T14:20:19.348826+0000","last_became_peered":"2023-10-19T14:20:19.348826+0000","last_unstale":"2023-10-19T14:20:29.790018+0000","last_undegraded":"2023-10-19T14:20:29.790018+0000","last_fullsized":"2023-10-19T14:20:29.790018+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:10:55.372033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.14","version":"226'6","reported_seq":18,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.782786+0000","last_change":"2023-10-19T14:20:29.795087+0000","last_active":"2023-10-19T14:20:31.782786+0000","last_peered":"2023-10-19T14:20:31.782786+0000","last_clean":"2023-10-19T14:20:31.782786+0000","last_became_active":"2023-10-19T14:20:29.794764+0000","last_became_peered":"2023-10-19T14:20:29.794764+0000","last_unstale":"2023-10-19T14:20:31.782786+0000","last_undegraded":"2023-10-19T14:20:31.782786+0000","last_fullsized":"2023-10-19T14:20:31.782786+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_clean_scrub_stamp":"2023-10-19T14:20:28.763908+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T02:16:50.628921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.16","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.785386+0000","last_change":"2023-10-19T14:20:30.802097+0000","last_active":"2023-10-19T14:20:31.785386+0000","last_peered":"2023-10-19T14:20:31.785386+0000","last_clean":"2023-10-19T14:20:31.785386+0000","last_became_active":"2023-10-19T14:20:30.801764+0000","last_became_peered":"2023-10-19T14:20:30.801764+0000","last_unstale":"2023-10-19T14:20:31.785386+0000","last_undegraded":"2023-10-19T14:20:31.785386+0000","last_fullsized":"2023-10-19T14:20:31.785386+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:17:27.514421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.13","version":"222'2","reported_seq":18,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.783682+0000","last_change":"2023-10-19T14:20:18.713594+0000","last_active":"2023-10-19T14:20:30.783682+0000","last_peered":"2023-10-19T14:20:30.783682+0000","last_clean":"2023-10-19T14:20:30.783682+0000","last_became_active":"2023-10-19T14:20:18.713263+0000","last_became_peered":"2023-10-19T14:20:18.713263+0000","last_unstale":"2023-10-19T14:20:30.783682+0000","last_undegraded":"2023-10-19T14:20:30.783682+0000","last_fullsized":"2023-10-19T14:20:30.783682+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:32:02.753539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.14","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.783314+0000","last_change":"2023-10-19T14:20:19.340471+0000","last_active":"2023-10-19T14:20:30.783314+0000","last_peered":"2023-10-19T14:20:30.783314+0000","last_clean":"2023-10-19T14:20:30.783314+0000","last_became_active":"2023-10-19T14:20:19.340187+0000","last_became_peered":"2023-10-19T14:20:19.340187+0000","last_unstale":"2023-10-19T14:20:30.783314+0000","last_undegraded":"2023-10-19T14:20:30.783314+0000","last_fullsized":"2023-10-19T14:20:30.783314+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:53:57.462566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.15","version":"226'3","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.812302+0000","last_change":"2023-10-19T14:20:29.796471+0000","last_active":"2023-10-19T14:20:30.812302+0000","last_peered":"2023-10-19T14:20:30.812302+0000","last_clean":"2023-10-19T14:20:30.812302+0000","last_became_active":"2023-10-19T14:20:29.796232+0000","last_became_peered":"2023-10-19T14:20:29.796232+0000","last_unstale":"2023-10-19T14:20:30.812302+0000","last_undegraded":"2023-10-19T14:20:30.812302+0000","last_fullsized":"2023-10-19T14:20:30.812302+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_clean_scrub_stamp":"2023-10-19T14:20:28.763908+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:27:15.682334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.11","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.813383+0000","last_change":"2023-10-19T14:20:30.800576+0000","last_active":"2023-10-19T14:20:31.813383+0000","last_peered":"2023-10-19T14:20:31.813383+0000","last_clean":"2023-10-19T14:20:31.813383+0000","last_became_active":"2023-10-19T14:20:30.800156+0000","last_became_peered":"2023-10-19T14:20:30.800156+0000","last_unstale":"2023-10-19T14:20:31.813383+0000","last_undegraded":"2023-10-19T14:20:31.813383+0000","last_fullsized":"2023-10-19T14:20:31.813383+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:02:04.231680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.14","version":"225'8","reported_seq":23,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.967684+0000","last_change":"2023-10-19T14:20:18.727225+0000","last_active":"2023-10-19T14:20:29.967684+0000","last_peered":"2023-10-19T14:20:29.967684+0000","last_clean":"2023-10-19T14:20:29.967684+0000","last_became_active":"2023-10-19T14:20:18.726839+0000","last_became_peered":"2023-10-19T14:20:18.726839+0000","last_unstale":"2023-10-19T14:20:29.967684+0000","last_undegraded":"2023-10-19T14:20:29.967684+0000","last_fullsized":"2023-10-19T14:20:29.967684+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:10:54.369994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2868,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.13","version":"0'0","reported_seq":14,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.782468+0000","last_change":"2023-10-19T14:20:19.350427+0000","last_active":"2023-10-19T14:20:29.782468+0000","last_peered":"2023-10-19T14:20:29.782468+0000","last_clean":"2023-10-19T14:20:29.782468+0000","last_became_active":"2023-10-19T14:20:19.350112+0000","last_became_peered":"2023-10-19T14:20:19.350112+0000","last_unstale":"2023-10-19T14:20:29.782468+0000","last_undegraded":"2023-10-19T14:20:29.782468+0000","last_fullsized":"2023-10-19T14:20:29.782468+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:01:38.993506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.12","version":"0'0","reported_seq":12,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.785538+0000","last_change":"2023-10-19T14:20:29.784629+0000","last_active":"2023-10-19T14:20:31.785538+0000","last_peered":"2023-10-19T14:20:31.785538+0000","last_clean":"2023-10-19T14:20:31.785538+0000","last_became_active":"2023-10-19T14:20:29.784364+0000","last_became_peered":"2023-10-19T14:20:29.784364+0000","last_unstale":"2023-10-19T14:20:31.785538+0000","last_undegraded":"2023-10-19T14:20:31.785538+0000","last_fullsized":"2023-10-19T14:20:31.785538+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_clean_scrub_stamp":"2023-10-19T14:20:28.763908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:07:07.108020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.10","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.785348+0000","last_change":"2023-10-19T14:20:30.795623+0000","last_active":"2023-10-19T14:20:31.785348+0000","last_peered":"2023-10-19T14:20:31.785348+0000","last_clean":"2023-10-19T14:20:31.785348+0000","last_became_active":"2023-10-19T14:20:30.795347+0000","last_became_peered":"2023-10-19T14:20:30.795347+0000","last_unstale":"2023-10-19T14:20:31.785348+0000","last_undegraded":"2023-10-19T14:20:31.785348+0000","last_fullsized":"2023-10-19T14:20:31.785348+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:44:04.797792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.15","version":"222'3","reported_seq":19,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.789401+0000","last_change":"2023-10-19T14:20:18.723409+0000","last_active":"2023-10-19T14:20:30.789401+0000","last_peered":"2023-10-19T14:20:30.789401+0000","last_clean":"2023-10-19T14:20:30.789401+0000","last_became_active":"2023-10-19T14:20:18.723148+0000","last_became_peered":"2023-10-19T14:20:18.723148+0000","last_unstale":"2023-10-19T14:20:30.789401+0000","last_undegraded":"2023-10-19T14:20:30.789401+0000","last_fullsized":"2023-10-19T14:20:30.789401+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:13:53.110830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.786554+0000","last_change":"2023-10-19T14:20:19.334253+0000","last_active":"2023-10-19T14:20:30.786554+0000","last_peered":"2023-10-19T14:20:30.786554+0000","last_clean":"2023-10-19T14:20:30.786554+0000","last_became_active":"2023-10-19T14:20:19.333981+0000","last_became_peered":"2023-10-19T14:20:19.333981+0000","last_unstale":"2023-10-19T14:20:30.786554+0000","last_undegraded":"2023-10-19T14:20:30.786554+0000","last_fullsized":"2023-10-19T14:20:30.786554+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:48:00.239750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.13","version":"226'2","reported_seq":14,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.813695+0000","last_change":"2023-10-19T14:20:29.800227+0000","last_active":"2023-10-19T14:20:31.813695+0000","last_peered":"2023-10-19T14:20:31.813695+0000","last_clean":"2023-10-19T14:20:31.813695+0000","last_became_active":"2023-10-19T14:20:29.799696+0000","last_became_peered":"2023-10-19T14:20:29.799696+0000","last_unstale":"2023-10-19T14:20:31.813695+0000","last_undegraded":"2023-10-19T14:20:31.813695+0000","last_fullsized":"2023-10-19T14:20:31.813695+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_clean_scrub_stamp":"2023-10-19T14:20:28.763908+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:37:21.704109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.13","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.797344+0000","last_change":"2023-10-19T14:20:30.796333+0000","last_active":"2023-10-19T14:20:30.797344+0000","last_peered":"2023-10-19T14:20:30.797344+0000","last_clean":"2023-10-19T14:20:30.797344+0000","last_became_active":"2023-10-19T14:20:30.796046+0000","last_became_peered":"2023-10-19T14:20:30.796046+0000","last_unstale":"2023-10-19T14:20:30.797344+0000","last_undegraded":"2023-10-19T14:20:30.797344+0000","last_fullsized":"2023-10-19T14:20:30.797344+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:14:05.187716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":15,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.787532+0000","last_change":"2023-10-19T14:20:18.721992+0000","last_active":"2023-10-19T14:20:29.787532+0000","last_peered":"2023-10-19T14:20:29.787532+0000","last_clean":"2023-10-19T14:20:29.787532+0000","last_became_active":"2023-10-19T14:20:18.721742+0000","last_became_peered":"2023-10-19T14:20:18.721742+0000","last_unstale":"2023-10-19T14:20:29.787532+0000","last_undegraded":"2023-10-19T14:20:29.787532+0000","last_fullsized":"2023-10-19T14:20:29.787532+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:16:37.567429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":14,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.784142+0000","last_change":"2023-10-19T14:20:19.339048+0000","last_active":"2023-10-19T14:20:29.784142+0000","last_peered":"2023-10-19T14:20:29.784142+0000","last_clean":"2023-10-19T14:20:29.784142+0000","last_became_active":"2023-10-19T14:20:19.338786+0000","last_became_peered":"2023-10-19T14:20:19.338786+0000","last_unstale":"2023-10-19T14:20:29.784142+0000","last_undegraded":"2023-10-19T14:20:29.784142+0000","last_fullsized":"2023-10-19T14:20:29.784142+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:16:46.603067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.10","version":"226'8","reported_seq":20,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.813335+0000","last_change":"2023-10-19T14:20:29.798135+0000","last_active":"2023-10-19T14:20:31.813335+0000","last_peered":"2023-10-19T14:20:31.813335+0000","last_clean":"2023-10-19T14:20:31.813335+0000","last_became_active":"2023-10-19T14:20:29.797692+0000","last_became_peered":"2023-10-19T14:20:29.797692+0000","last_unstale":"2023-10-19T14:20:31.813335+0000","last_undegraded":"2023-10-19T14:20:31.813335+0000","last_fullsized":"2023-10-19T14:20:31.813335+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_clean_scrub_stamp":"2023-10-19T14:20:28.763908+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:38:29.957301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.12","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.784527+0000","last_change":"2023-10-19T14:20:30.790680+0000","last_active":"2023-10-19T14:20:31.784527+0000","last_peered":"2023-10-19T14:20:31.784527+0000","last_clean":"2023-10-19T14:20:31.784527+0000","last_became_active":"2023-10-19T14:20:30.790400+0000","last_became_peered":"2023-10-19T14:20:30.790400+0000","last_unstale":"2023-10-19T14:20:31.784527+0000","last_undegraded":"2023-10-19T14:20:31.784527+0000","last_fullsized":"2023-10-19T14:20:31.784527+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:26:05.573883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":15,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.782121+0000","last_change":"2023-10-19T14:20:18.722070+0000","last_active":"2023-10-19T14:20:29.782121+0000","last_peered":"2023-10-19T14:20:29.782121+0000","last_clean":"2023-10-19T14:20:29.782121+0000","last_became_active":"2023-10-19T14:20:18.721791+0000","last_became_peered":"2023-10-19T14:20:18.721791+0000","last_unstale":"2023-10-19T14:20:29.782121+0000","last_undegraded":"2023-10-19T14:20:29.782121+0000","last_fullsized":"2023-10-19T14:20:29.782121+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:17:55.048333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.10","version":"0'0","reported_seq":14,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.784494+0000","last_change":"2023-10-19T14:20:19.351668+0000","last_active":"2023-10-19T14:20:29.784494+0000","last_peered":"2023-10-19T14:20:29.784494+0000","last_clean":"2023-10-19T14:20:29.784494+0000","last_became_active":"2023-10-19T14:20:19.351298+0000","last_became_peered":"2023-10-19T14:20:19.351298+0000","last_unstale":"2023-10-19T14:20:29.784494+0000","last_undegraded":"2023-10-19T14:20:29.784494+0000","last_fullsized":"2023-10-19T14:20:29.784494+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:36:14.871743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.11","version":"0'0","reported_seq":12,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.781994+0000","last_change":"2023-10-19T14:20:29.790327+0000","last_active":"2023-10-19T14:20:31.781994+0000","last_peered":"2023-10-19T14:20:31.781994+0000","last_clean":"2023-10-19T14:20:31.781994+0000","last_became_active":"2023-10-19T14:20:29.790038+0000","last_became_peered":"2023-10-19T14:20:29.790038+0000","last_unstale":"2023-10-19T14:20:31.781994+0000","last_undegraded":"2023-10-19T14:20:31.781994+0000","last_fullsized":"2023-10-19T14:20:31.781994+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_clean_scrub_stamp":"2023-10-19T14:20:28.763908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:40:43.580514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.1d","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.784800+0000","last_change":"2023-10-19T14:20:30.801236+0000","last_active":"2023-10-19T14:20:31.784800+0000","last_peered":"2023-10-19T14:20:31.784800+0000","last_clean":"2023-10-19T14:20:31.784800+0000","last_became_active":"2023-10-19T14:20:30.800928+0000","last_became_peered":"2023-10-19T14:20:30.800928+0000","last_unstale":"2023-10-19T14:20:31.784800+0000","last_undegraded":"2023-10-19T14:20:31.784800+0000","last_fullsized":"2023-10-19T14:20:31.784800+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:52:54.676865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.1e","version":"226'1","reported_seq":13,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.814413+0000","last_change":"2023-10-19T14:20:29.798101+0000","last_active":"2023-10-19T14:20:31.814413+0000","last_peered":"2023-10-19T14:20:31.814413+0000","last_clean":"2023-10-19T14:20:31.814413+0000","last_became_active":"2023-10-19T14:20:29.797834+0000","last_became_peered":"2023-10-19T14:20:29.797834+0000","last_unstale":"2023-10-19T14:20:31.814413+0000","last_undegraded":"2023-10-19T14:20:31.814413+0000","last_fullsized":"2023-10-19T14:20:31.814413+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_clean_scrub_stamp":"2023-10-19T14:20:28.763908+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:50:11.018716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":15,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.784192+0000","last_change":"2023-10-19T14:20:18.718913+0000","last_active":"2023-10-19T14:20:29.784192+0000","last_peered":"2023-10-19T14:20:29.784192+0000","last_clean":"2023-10-19T14:20:29.784192+0000","last_became_active":"2023-10-19T14:20:18.718641+0000","last_became_peered":"2023-10-19T14:20:18.718641+0000","last_unstale":"2023-10-19T14:20:29.784192+0000","last_undegraded":"2023-10-19T14:20:29.784192+0000","last_fullsized":"2023-10-19T14:20:29.784192+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:54:40.217789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1f","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.788798+0000","last_change":"2023-10-19T14:20:19.346582+0000","last_active":"2023-10-19T14:20:30.788798+0000","last_peered":"2023-10-19T14:20:30.788798+0000","last_clean":"2023-10-19T14:20:30.788798+0000","last_became_active":"2023-10-19T14:20:19.346246+0000","last_became_peered":"2023-10-19T14:20:19.346246+0000","last_unstale":"2023-10-19T14:20:30.788798+0000","last_undegraded":"2023-10-19T14:20:30.788798+0000","last_fullsized":"2023-10-19T14:20:30.788798+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:37:58.222092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.1c","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.784738+0000","last_change":"2023-10-19T14:20:30.788933+0000","last_active":"2023-10-19T14:20:31.784738+0000","last_peered":"2023-10-19T14:20:31.784738+0000","last_clean":"2023-10-19T14:20:31.784738+0000","last_became_active":"2023-10-19T14:20:30.788378+0000","last_became_peered":"2023-10-19T14:20:30.788378+0000","last_unstale":"2023-10-19T14:20:31.784738+0000","last_undegraded":"2023-10-19T14:20:31.784738+0000","last_fullsized":"2023-10-19T14:20:31.784738+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:41:13.232644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.1f","version":"226'4","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.812922+0000","last_change":"2023-10-19T14:20:29.787116+0000","last_active":"2023-10-19T14:20:30.812922+0000","last_peered":"2023-10-19T14:20:30.812922+0000","last_clean":"2023-10-19T14:20:30.812922+0000","last_became_active":"2023-10-19T14:20:29.786827+0000","last_became_peered":"2023-10-19T14:20:29.786827+0000","last_unstale":"2023-10-19T14:20:30.812922+0000","last_undegraded":"2023-10-19T14:20:30.812922+0000","last_fullsized":"2023-10-19T14:20:30.812922+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_clean_scrub_stamp":"2023-10-19T14:20:28.763908+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:44:07.248181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":15,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.786373+0000","last_change":"2023-10-19T14:20:18.721875+0000","last_active":"2023-10-19T14:20:29.786373+0000","last_peered":"2023-10-19T14:20:29.786373+0000","last_clean":"2023-10-19T14:20:29.786373+0000","last_became_active":"2023-10-19T14:20:18.721611+0000","last_became_peered":"2023-10-19T14:20:18.721611+0000","last_unstale":"2023-10-19T14:20:29.786373+0000","last_undegraded":"2023-10-19T14:20:29.786373+0000","last_fullsized":"2023-10-19T14:20:29.786373+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:46:19.718145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1e","version":"0'0","reported_seq":14,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.786008+0000","last_change":"2023-10-19T14:20:19.343379+0000","last_active":"2023-10-19T14:20:29.786008+0000","last_peered":"2023-10-19T14:20:29.786008+0000","last_clean":"2023-10-19T14:20:29.786008+0000","last_became_active":"2023-10-19T14:20:19.343117+0000","last_became_peered":"2023-10-19T14:20:19.343117+0000","last_unstale":"2023-10-19T14:20:29.786008+0000","last_undegraded":"2023-10-19T14:20:29.786008+0000","last_fullsized":"2023-10-19T14:20:29.786008+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:33:52.226441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1e","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.814298+0000","last_change":"2023-10-19T14:20:30.802110+0000","last_active":"2023-10-19T14:20:31.814298+0000","last_peered":"2023-10-19T14:20:31.814298+0000","last_clean":"2023-10-19T14:20:31.814298+0000","last_became_active":"2023-10-19T14:20:30.801833+0000","last_became_peered":"2023-10-19T14:20:30.801833+0000","last_unstale":"2023-10-19T14:20:31.814298+0000","last_undegraded":"2023-10-19T14:20:31.814298+0000","last_fullsized":"2023-10-19T14:20:31.814298+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:50:03.887637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1b","version":"222'1","reported_seq":16,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.786436+0000","last_change":"2023-10-19T14:20:18.722476+0000","last_active":"2023-10-19T14:20:29.786436+0000","last_peered":"2023-10-19T14:20:29.786436+0000","last_clean":"2023-10-19T14:20:29.786436+0000","last_became_active":"2023-10-19T14:20:18.722195+0000","last_became_peered":"2023-10-19T14:20:18.722195+0000","last_unstale":"2023-10-19T14:20:29.786436+0000","last_undegraded":"2023-10-19T14:20:29.786436+0000","last_fullsized":"2023-10-19T14:20:29.786436+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:38:13.906613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1c","version":"0'0","reported_seq":14,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.786205+0000","last_change":"2023-10-19T14:20:19.349815+0000","last_active":"2023-10-19T14:20:29.786205+0000","last_peered":"2023-10-19T14:20:29.786205+0000","last_clean":"2023-10-19T14:20:29.786205+0000","last_became_active":"2023-10-19T14:20:19.349511+0000","last_became_peered":"2023-10-19T14:20:19.349511+0000","last_unstale":"2023-10-19T14:20:29.786205+0000","last_undegraded":"2023-10-19T14:20:29.786205+0000","last_fullsized":"2023-10-19T14:20:29.786205+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:32:39.312387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.1d","version":"226'2","reported_seq":14,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.782733+0000","last_change":"2023-10-19T14:20:29.802016+0000","last_active":"2023-10-19T14:20:31.782733+0000","last_peered":"2023-10-19T14:20:31.782733+0000","last_clean":"2023-10-19T14:20:31.782733+0000","last_became_active":"2023-10-19T14:20:29.801737+0000","last_became_peered":"2023-10-19T14:20:29.801737+0000","last_unstale":"2023-10-19T14:20:31.782733+0000","last_undegraded":"2023-10-19T14:20:31.782733+0000","last_fullsized":"2023-10-19T14:20:31.782733+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_clean_scrub_stamp":"2023-10-19T14:20:28.763908+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:01:55.267605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.19","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.813383+0000","last_change":"2023-10-19T14:20:30.802232+0000","last_active":"2023-10-19T14:20:31.813383+0000","last_peered":"2023-10-19T14:20:31.813383+0000","last_clean":"2023-10-19T14:20:31.813383+0000","last_became_active":"2023-10-19T14:20:30.801966+0000","last_became_peered":"2023-10-19T14:20:30.801966+0000","last_unstale":"2023-10-19T14:20:31.813383+0000","last_undegraded":"2023-10-19T14:20:31.813383+0000","last_fullsized":"2023-10-19T14:20:31.813383+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:44:41.933734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1c","version":"222'2","reported_seq":18,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.786824+0000","last_change":"2023-10-19T14:20:18.722673+0000","last_active":"2023-10-19T14:20:30.786824+0000","last_peered":"2023-10-19T14:20:30.786824+0000","last_clean":"2023-10-19T14:20:30.786824+0000","last_became_active":"2023-10-19T14:20:18.722418+0000","last_became_peered":"2023-10-19T14:20:18.722418+0000","last_unstale":"2023-10-19T14:20:30.786824+0000","last_undegraded":"2023-10-19T14:20:30.786824+0000","last_fullsized":"2023-10-19T14:20:30.786824+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:40:32.510221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1b","version":"0'0","reported_seq":14,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.782383+0000","last_change":"2023-10-19T14:20:19.348254+0000","last_active":"2023-10-19T14:20:29.782383+0000","last_peered":"2023-10-19T14:20:29.782383+0000","last_clean":"2023-10-19T14:20:29.782383+0000","last_became_active":"2023-10-19T14:20:19.347974+0000","last_became_peered":"2023-10-19T14:20:19.347974+0000","last_unstale":"2023-10-19T14:20:29.782383+0000","last_undegraded":"2023-10-19T14:20:29.782383+0000","last_fullsized":"2023-10-19T14:20:29.782383+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:32:42.913042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.1a","version":"226'1","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.813571+0000","last_change":"2023-10-19T14:20:29.795524+0000","last_active":"2023-10-19T14:20:30.813571+0000","last_peered":"2023-10-19T14:20:30.813571+0000","last_clean":"2023-10-19T14:20:30.813571+0000","last_became_active":"2023-10-19T14:20:29.795205+0000","last_became_peered":"2023-10-19T14:20:29.795205+0000","last_unstale":"2023-10-19T14:20:30.813571+0000","last_undegraded":"2023-10-19T14:20:30.813571+0000","last_fullsized":"2023-10-19T14:20:30.813571+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_clean_scrub_stamp":"2023-10-19T14:20:28.763908+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:16:30.284844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.18","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.784275+0000","last_change":"2023-10-19T14:20:30.791002+0000","last_active":"2023-10-19T14:20:31.784275+0000","last_peered":"2023-10-19T14:20:31.784275+0000","last_clean":"2023-10-19T14:20:31.784275+0000","last_became_active":"2023-10-19T14:20:30.790717+0000","last_became_peered":"2023-10-19T14:20:30.790717+0000","last_unstale":"2023-10-19T14:20:31.784275+0000","last_undegraded":"2023-10-19T14:20:31.784275+0000","last_fullsized":"2023-10-19T14:20:31.784275+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:13:13.631186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.1d","version":"222'2","reported_seq":18,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.787815+0000","last_change":"2023-10-19T14:20:18.717342+0000","last_active":"2023-10-19T14:20:30.787815+0000","last_peered":"2023-10-19T14:20:30.787815+0000","last_clean":"2023-10-19T14:20:30.787815+0000","last_became_active":"2023-10-19T14:20:18.717047+0000","last_became_peered":"2023-10-19T14:20:18.717047+0000","last_unstale":"2023-10-19T14:20:30.787815+0000","last_undegraded":"2023-10-19T14:20:30.787815+0000","last_fullsized":"2023-10-19T14:20:30.787815+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:28:55.961320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.1a","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.799358+0000","last_change":"2023-10-19T14:20:19.346318+0000","last_active":"2023-10-19T14:20:30.799358+0000","last_peered":"2023-10-19T14:20:30.799358+0000","last_clean":"2023-10-19T14:20:30.799358+0000","last_became_active":"2023-10-19T14:20:19.346014+0000","last_became_peered":"2023-10-19T14:20:19.346014+0000","last_unstale":"2023-10-19T14:20:30.799358+0000","last_undegraded":"2023-10-19T14:20:30.799358+0000","last_fullsized":"2023-10-19T14:20:30.799358+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:20:35.750886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1b","version":"226'1","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.813306+0000","last_change":"2023-10-19T14:20:29.796824+0000","last_active":"2023-10-19T14:20:30.813306+0000","last_peered":"2023-10-19T14:20:30.813306+0000","last_clean":"2023-10-19T14:20:30.813306+0000","last_became_active":"2023-10-19T14:20:29.796469+0000","last_became_peered":"2023-10-19T14:20:29.796469+0000","last_unstale":"2023-10-19T14:20:30.813306+0000","last_undegraded":"2023-10-19T14:20:30.813306+0000","last_fullsized":"2023-10-19T14:20:30.813306+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_clean_scrub_stamp":"2023-10-19T14:20:28.763908+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:40:43.580514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.1b","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.790310+0000","last_change":"2023-10-19T14:20:30.787300+0000","last_active":"2023-10-19T14:20:30.790310+0000","last_peered":"2023-10-19T14:20:30.790310+0000","last_clean":"2023-10-19T14:20:30.790310+0000","last_became_active":"2023-10-19T14:20:30.786994+0000","last_became_peered":"2023-10-19T14:20:30.786994+0000","last_unstale":"2023-10-19T14:20:30.790310+0000","last_undegraded":"2023-10-19T14:20:30.790310+0000","last_fullsized":"2023-10-19T14:20:30.790310+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:18:21.133124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1e","version":"222'1","reported_seq":17,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.781879+0000","last_change":"2023-10-19T14:20:18.719165+0000","last_active":"2023-10-19T14:20:30.781879+0000","last_peered":"2023-10-19T14:20:30.781879+0000","last_clean":"2023-10-19T14:20:30.781879+0000","last_became_active":"2023-10-19T14:20:18.717146+0000","last_became_peered":"2023-10-19T14:20:18.717146+0000","last_unstale":"2023-10-19T14:20:30.781879+0000","last_undegraded":"2023-10-19T14:20:30.781879+0000","last_fullsized":"2023-10-19T14:20:30.781879+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:57:14.320702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.782644+0000","last_change":"2023-10-19T14:20:19.343393+0000","last_active":"2023-10-19T14:20:30.782644+0000","last_peered":"2023-10-19T14:20:30.782644+0000","last_clean":"2023-10-19T14:20:30.782644+0000","last_became_active":"2023-10-19T14:20:19.343011+0000","last_became_peered":"2023-10-19T14:20:19.343011+0000","last_unstale":"2023-10-19T14:20:30.782644+0000","last_undegraded":"2023-10-19T14:20:30.782644+0000","last_fullsized":"2023-10-19T14:20:30.782644+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:22:47.864868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.18","version":"0'0","reported_seq":12,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.784505+0000","last_change":"2023-10-19T14:20:29.794071+0000","last_active":"2023-10-19T14:20:31.784505+0000","last_peered":"2023-10-19T14:20:31.784505+0000","last_clean":"2023-10-19T14:20:31.784505+0000","last_became_active":"2023-10-19T14:20:29.793702+0000","last_became_peered":"2023-10-19T14:20:29.793702+0000","last_unstale":"2023-10-19T14:20:31.784505+0000","last_undegraded":"2023-10-19T14:20:31.784505+0000","last_fullsized":"2023-10-19T14:20:31.784505+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_clean_scrub_stamp":"2023-10-19T14:20:28.763908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:20:52.649351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.1a","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.797372+0000","last_change":"2023-10-19T14:20:30.795544+0000","last_active":"2023-10-19T14:20:30.797372+0000","last_peered":"2023-10-19T14:20:30.797372+0000","last_clean":"2023-10-19T14:20:30.797372+0000","last_became_active":"2023-10-19T14:20:30.795273+0000","last_became_peered":"2023-10-19T14:20:30.795273+0000","last_unstale":"2023-10-19T14:20:30.797372+0000","last_undegraded":"2023-10-19T14:20:30.797372+0000","last_fullsized":"2023-10-19T14:20:30.797372+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:07:29.800710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.1f","version":"222'4","reported_seq":19,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.786147+0000","last_change":"2023-10-19T14:20:18.725431+0000","last_active":"2023-10-19T14:20:29.786147+0000","last_peered":"2023-10-19T14:20:29.786147+0000","last_clean":"2023-10-19T14:20:29.786147+0000","last_became_active":"2023-10-19T14:20:18.725142+0000","last_became_peered":"2023-10-19T14:20:18.725142+0000","last_unstale":"2023-10-19T14:20:29.786147+0000","last_undegraded":"2023-10-19T14:20:29.786147+0000","last_fullsized":"2023-10-19T14:20:29.786147+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T02:03:46.398046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.785197+0000","last_change":"2023-10-19T14:20:19.333259+0000","last_active":"2023-10-19T14:20:30.785197+0000","last_peered":"2023-10-19T14:20:30.785197+0000","last_clean":"2023-10-19T14:20:30.785197+0000","last_became_active":"2023-10-19T14:20:19.332735+0000","last_became_peered":"2023-10-19T14:20:19.332735+0000","last_unstale":"2023-10-19T14:20:30.785197+0000","last_undegraded":"2023-10-19T14:20:30.785197+0000","last_fullsized":"2023-10-19T14:20:30.785197+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:46:26.663119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.19","version":"0'0","reported_seq":11,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.788500+0000","last_change":"2023-10-19T14:20:29.796237+0000","last_active":"2023-10-19T14:20:30.788500+0000","last_peered":"2023-10-19T14:20:30.788500+0000","last_clean":"2023-10-19T14:20:30.788500+0000","last_became_active":"2023-10-19T14:20:29.795731+0000","last_became_peered":"2023-10-19T14:20:29.795731+0000","last_unstale":"2023-10-19T14:20:30.788500+0000","last_undegraded":"2023-10-19T14:20:30.788500+0000","last_fullsized":"2023-10-19T14:20:30.788500+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_clean_scrub_stamp":"2023-10-19T14:20:28.763908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:38:36.765526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.22","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.782363+0000","last_change":"2023-10-19T14:20:30.803463+0000","last_active":"2023-10-19T14:20:31.782363+0000","last_peered":"2023-10-19T14:20:31.782363+0000","last_clean":"2023-10-19T14:20:31.782363+0000","last_became_active":"2023-10-19T14:20:30.803169+0000","last_became_peered":"2023-10-19T14:20:30.803169+0000","last_unstale":"2023-10-19T14:20:31.782363+0000","last_undegraded":"2023-10-19T14:20:31.782363+0000","last_fullsized":"2023-10-19T14:20:31.782363+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:19:00.812957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.20","version":"0'0","reported_seq":14,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.785838+0000","last_change":"2023-10-19T14:20:19.348899+0000","last_active":"2023-10-19T14:20:29.785838+0000","last_peered":"2023-10-19T14:20:29.785838+0000","last_clean":"2023-10-19T14:20:29.785838+0000","last_became_active":"2023-10-19T14:20:19.348453+0000","last_became_peered":"2023-10-19T14:20:19.348453+0000","last_unstale":"2023-10-19T14:20:29.785838+0000","last_undegraded":"2023-10-19T14:20:29.785838+0000","last_fullsized":"2023-10-19T14:20:29.785838+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:01:53.425378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.23","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.785706+0000","last_change":"2023-10-19T14:20:30.790486+0000","last_active":"2023-10-19T14:20:31.785706+0000","last_peered":"2023-10-19T14:20:31.785706+0000","last_clean":"2023-10-19T14:20:31.785706+0000","last_became_active":"2023-10-19T14:20:30.790164+0000","last_became_peered":"2023-10-19T14:20:30.790164+0000","last_unstale":"2023-10-19T14:20:31.785706+0000","last_undegraded":"2023-10-19T14:20:31.785706+0000","last_fullsized":"2023-10-19T14:20:31.785706+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:12:50.182135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.21","version":"0'0","reported_seq":14,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.785838+0000","last_change":"2023-10-19T14:20:19.349579+0000","last_active":"2023-10-19T14:20:29.785838+0000","last_peered":"2023-10-19T14:20:29.785838+0000","last_clean":"2023-10-19T14:20:29.785838+0000","last_became_active":"2023-10-19T14:20:19.349321+0000","last_became_peered":"2023-10-19T14:20:19.349321+0000","last_unstale":"2023-10-19T14:20:29.785838+0000","last_undegraded":"2023-10-19T14:20:29.785838+0000","last_fullsized":"2023-10-19T14:20:29.785838+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:44:03.588824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.20","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.782192+0000","last_change":"2023-10-19T14:20:30.799253+0000","last_active":"2023-10-19T14:20:31.782192+0000","last_peered":"2023-10-19T14:20:31.782192+0000","last_clean":"2023-10-19T14:20:31.782192+0000","last_became_active":"2023-10-19T14:20:30.799000+0000","last_became_peered":"2023-10-19T14:20:30.799000+0000","last_unstale":"2023-10-19T14:20:31.782192+0000","last_undegraded":"2023-10-19T14:20:31.782192+0000","last_fullsized":"2023-10-19T14:20:31.782192+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:28:29.776053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.22","version":"0'0","reported_seq":14,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.784544+0000","last_change":"2023-10-19T14:20:19.342297+0000","last_active":"2023-10-19T14:20:29.784544+0000","last_peered":"2023-10-19T14:20:29.784544+0000","last_clean":"2023-10-19T14:20:29.784544+0000","last_became_active":"2023-10-19T14:20:19.342026+0000","last_became_peered":"2023-10-19T14:20:19.342026+0000","last_unstale":"2023-10-19T14:20:29.784544+0000","last_undegraded":"2023-10-19T14:20:29.784544+0000","last_fullsized":"2023-10-19T14:20:29.784544+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:43:57.179927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.21","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.793992+0000","last_change":"2023-10-19T14:20:30.791876+0000","last_active":"2023-10-19T14:20:30.793992+0000","last_peered":"2023-10-19T14:20:30.793992+0000","last_clean":"2023-10-19T14:20:30.793992+0000","last_became_active":"2023-10-19T14:20:30.791563+0000","last_became_peered":"2023-10-19T14:20:30.791563+0000","last_unstale":"2023-10-19T14:20:30.793992+0000","last_undegraded":"2023-10-19T14:20:30.793992+0000","last_fullsized":"2023-10-19T14:20:30.793992+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:04:45.084132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.23","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.786920+0000","last_change":"2023-10-19T14:20:19.342642+0000","last_active":"2023-10-19T14:20:30.786920+0000","last_peered":"2023-10-19T14:20:30.786920+0000","last_clean":"2023-10-19T14:20:30.786920+0000","last_became_active":"2023-10-19T14:20:19.342379+0000","last_became_peered":"2023-10-19T14:20:19.342379+0000","last_unstale":"2023-10-19T14:20:30.786920+0000","last_undegraded":"2023-10-19T14:20:30.786920+0000","last_fullsized":"2023-10-19T14:20:30.786920+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:33:34.581616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.26","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.785208+0000","last_change":"2023-10-19T14:20:30.807457+0000","last_active":"2023-10-19T14:20:31.785208+0000","last_peered":"2023-10-19T14:20:31.785208+0000","last_clean":"2023-10-19T14:20:31.785208+0000","last_became_active":"2023-10-19T14:20:30.807150+0000","last_became_peered":"2023-10-19T14:20:30.807150+0000","last_unstale":"2023-10-19T14:20:31.785208+0000","last_undegraded":"2023-10-19T14:20:31.785208+0000","last_fullsized":"2023-10-19T14:20:31.785208+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:13:06.353666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.24","version":"0'0","reported_seq":14,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.781754+0000","last_change":"2023-10-19T14:20:19.345949+0000","last_active":"2023-10-19T14:20:29.781754+0000","last_peered":"2023-10-19T14:20:29.781754+0000","last_clean":"2023-10-19T14:20:29.781754+0000","last_became_active":"2023-10-19T14:20:19.345664+0000","last_became_peered":"2023-10-19T14:20:19.345664+0000","last_unstale":"2023-10-19T14:20:29.781754+0000","last_undegraded":"2023-10-19T14:20:29.781754+0000","last_fullsized":"2023-10-19T14:20:29.781754+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:52:23.062217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.27","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.804953+0000","last_change":"2023-10-19T14:20:30.798702+0000","last_active":"2023-10-19T14:20:30.804953+0000","last_peered":"2023-10-19T14:20:30.804953+0000","last_clean":"2023-10-19T14:20:30.804953+0000","last_became_active":"2023-10-19T14:20:30.798425+0000","last_became_peered":"2023-10-19T14:20:30.798425+0000","last_unstale":"2023-10-19T14:20:30.804953+0000","last_undegraded":"2023-10-19T14:20:30.804953+0000","last_fullsized":"2023-10-19T14:20:30.804953+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:36:40.012568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.25","version":"0'0","reported_seq":14,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.788699+0000","last_change":"2023-10-19T14:20:19.347669+0000","last_active":"2023-10-19T14:20:29.788699+0000","last_peered":"2023-10-19T14:20:29.788699+0000","last_clean":"2023-10-19T14:20:29.788699+0000","last_became_active":"2023-10-19T14:20:19.347397+0000","last_became_peered":"2023-10-19T14:20:19.347397+0000","last_unstale":"2023-10-19T14:20:29.788699+0000","last_undegraded":"2023-10-19T14:20:29.788699+0000","last_fullsized":"2023-10-19T14:20:29.788699+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:47:46.746053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.24","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.813987+0000","last_change":"2023-10-19T14:20:30.803549+0000","last_active":"2023-10-19T14:20:31.813987+0000","last_peered":"2023-10-19T14:20:31.813987+0000","last_clean":"2023-10-19T14:20:31.813987+0000","last_became_active":"2023-10-19T14:20:30.803130+0000","last_became_peered":"2023-10-19T14:20:30.803130+0000","last_unstale":"2023-10-19T14:20:31.813987+0000","last_undegraded":"2023-10-19T14:20:31.813987+0000","last_fullsized":"2023-10-19T14:20:31.813987+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:20:12.233298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.26","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.781764+0000","last_change":"2023-10-19T14:20:19.348136+0000","last_active":"2023-10-19T14:20:30.781764+0000","last_peered":"2023-10-19T14:20:30.781764+0000","last_clean":"2023-10-19T14:20:30.781764+0000","last_became_active":"2023-10-19T14:20:19.347832+0000","last_became_peered":"2023-10-19T14:20:19.347832+0000","last_unstale":"2023-10-19T14:20:30.781764+0000","last_undegraded":"2023-10-19T14:20:30.781764+0000","last_fullsized":"2023-10-19T14:20:30.781764+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:16:46.603067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.25","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.801797+0000","last_change":"2023-10-19T14:20:30.799093+0000","last_active":"2023-10-19T14:20:30.801797+0000","last_peered":"2023-10-19T14:20:30.801797+0000","last_clean":"2023-10-19T14:20:30.801797+0000","last_became_active":"2023-10-19T14:20:30.798726+0000","last_became_peered":"2023-10-19T14:20:30.798726+0000","last_unstale":"2023-10-19T14:20:30.801797+0000","last_undegraded":"2023-10-19T14:20:30.801797+0000","last_fullsized":"2023-10-19T14:20:30.801797+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T02:08:44.803755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.27","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.788702+0000","last_change":"2023-10-19T14:20:19.339558+0000","last_active":"2023-10-19T14:20:30.788702+0000","last_peered":"2023-10-19T14:20:30.788702+0000","last_clean":"2023-10-19T14:20:30.788702+0000","last_became_active":"2023-10-19T14:20:19.339298+0000","last_became_peered":"2023-10-19T14:20:19.339298+0000","last_unstale":"2023-10-19T14:20:30.788702+0000","last_undegraded":"2023-10-19T14:20:30.788702+0000","last_fullsized":"2023-10-19T14:20:30.788702+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:27:05.614080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.2a","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.798377+0000","last_change":"2023-10-19T14:20:30.797190+0000","last_active":"2023-10-19T14:20:30.798377+0000","last_peered":"2023-10-19T14:20:30.798377+0000","last_clean":"2023-10-19T14:20:30.798377+0000","last_became_active":"2023-10-19T14:20:30.796902+0000","last_became_peered":"2023-10-19T14:20:30.796902+0000","last_unstale":"2023-10-19T14:20:30.798377+0000","last_undegraded":"2023-10-19T14:20:30.798377+0000","last_fullsized":"2023-10-19T14:20:30.798377+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:38:06.530226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.28","version":"0'0","reported_seq":14,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.781321+0000","last_change":"2023-10-19T14:20:19.341549+0000","last_active":"2023-10-19T14:20:29.781321+0000","last_peered":"2023-10-19T14:20:29.781321+0000","last_clean":"2023-10-19T14:20:29.781321+0000","last_became_active":"2023-10-19T14:20:19.341276+0000","last_became_peered":"2023-10-19T14:20:19.341276+0000","last_unstale":"2023-10-19T14:20:29.781321+0000","last_undegraded":"2023-10-19T14:20:29.781321+0000","last_fullsized":"2023-10-19T14:20:29.781321+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:23:15.830657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.2b","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.784598+0000","last_change":"2023-10-19T14:20:30.791962+0000","last_active":"2023-10-19T14:20:31.784598+0000","last_peered":"2023-10-19T14:20:31.784598+0000","last_clean":"2023-10-19T14:20:31.784598+0000","last_became_active":"2023-10-19T14:20:30.791621+0000","last_became_peered":"2023-10-19T14:20:30.791621+0000","last_unstale":"2023-10-19T14:20:31.784598+0000","last_undegraded":"2023-10-19T14:20:31.784598+0000","last_fullsized":"2023-10-19T14:20:31.784598+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:32:05.192356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.29","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.782836+0000","last_change":"2023-10-19T14:20:19.333467+0000","last_active":"2023-10-19T14:20:30.782836+0000","last_peered":"2023-10-19T14:20:30.782836+0000","last_clean":"2023-10-19T14:20:30.782836+0000","last_became_active":"2023-10-19T14:20:19.333176+0000","last_became_peered":"2023-10-19T14:20:19.333176+0000","last_unstale":"2023-10-19T14:20:30.782836+0000","last_undegraded":"2023-10-19T14:20:30.782836+0000","last_fullsized":"2023-10-19T14:20:30.782836+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:39:10.256824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.28","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.795475+0000","last_change":"2023-10-19T14:20:30.793171+0000","last_active":"2023-10-19T14:20:30.795475+0000","last_peered":"2023-10-19T14:20:30.795475+0000","last_clean":"2023-10-19T14:20:30.795475+0000","last_became_active":"2023-10-19T14:20:30.792382+0000","last_became_peered":"2023-10-19T14:20:30.792382+0000","last_unstale":"2023-10-19T14:20:30.795475+0000","last_undegraded":"2023-10-19T14:20:30.795475+0000","last_fullsized":"2023-10-19T14:20:30.795475+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:48:11.314597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.2a","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.799275+0000","last_change":"2023-10-19T14:20:19.345667+0000","last_active":"2023-10-19T14:20:30.799275+0000","last_peered":"2023-10-19T14:20:30.799275+0000","last_clean":"2023-10-19T14:20:30.799275+0000","last_became_active":"2023-10-19T14:20:19.345343+0000","last_became_peered":"2023-10-19T14:20:19.345343+0000","last_unstale":"2023-10-19T14:20:30.799275+0000","last_undegraded":"2023-10-19T14:20:30.799275+0000","last_fullsized":"2023-10-19T14:20:30.799275+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:36:14.871743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.29","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.800683+0000","last_change":"2023-10-19T14:20:30.799002+0000","last_active":"2023-10-19T14:20:30.800683+0000","last_peered":"2023-10-19T14:20:30.800683+0000","last_clean":"2023-10-19T14:20:30.800683+0000","last_became_active":"2023-10-19T14:20:30.798780+0000","last_became_peered":"2023-10-19T14:20:30.798780+0000","last_unstale":"2023-10-19T14:20:30.800683+0000","last_undegraded":"2023-10-19T14:20:30.800683+0000","last_fullsized":"2023-10-19T14:20:30.800683+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:36:47.244576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.2b","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.800012+0000","last_change":"2023-10-19T14:20:19.351751+0000","last_active":"2023-10-19T14:20:30.800012+0000","last_peered":"2023-10-19T14:20:30.800012+0000","last_clean":"2023-10-19T14:20:30.800012+0000","last_became_active":"2023-10-19T14:20:19.350197+0000","last_became_peered":"2023-10-19T14:20:19.350197+0000","last_unstale":"2023-10-19T14:20:30.800012+0000","last_undegraded":"2023-10-19T14:20:30.800012+0000","last_fullsized":"2023-10-19T14:20:30.800012+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:18:10.058277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2e","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.785156+0000","last_change":"2023-10-19T14:20:30.800432+0000","last_active":"2023-10-19T14:20:31.785156+0000","last_peered":"2023-10-19T14:20:31.785156+0000","last_clean":"2023-10-19T14:20:31.785156+0000","last_became_active":"2023-10-19T14:20:30.799997+0000","last_became_peered":"2023-10-19T14:20:30.799997+0000","last_unstale":"2023-10-19T14:20:31.785156+0000","last_undegraded":"2023-10-19T14:20:31.785156+0000","last_fullsized":"2023-10-19T14:20:31.785156+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:06:24.375786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.2c","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.797597+0000","last_change":"2023-10-19T14:20:19.343670+0000","last_active":"2023-10-19T14:20:30.797597+0000","last_peered":"2023-10-19T14:20:30.797597+0000","last_clean":"2023-10-19T14:20:30.797597+0000","last_became_active":"2023-10-19T14:20:19.342491+0000","last_became_peered":"2023-10-19T14:20:19.342491+0000","last_unstale":"2023-10-19T14:20:30.797597+0000","last_undegraded":"2023-10-19T14:20:30.797597+0000","last_fullsized":"2023-10-19T14:20:30.797597+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:32:39.312387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2f","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.814351+0000","last_change":"2023-10-19T14:20:30.801539+0000","last_active":"2023-10-19T14:20:31.814351+0000","last_peered":"2023-10-19T14:20:31.814351+0000","last_clean":"2023-10-19T14:20:31.814351+0000","last_became_active":"2023-10-19T14:20:30.801273+0000","last_became_peered":"2023-10-19T14:20:30.801273+0000","last_unstale":"2023-10-19T14:20:31.814351+0000","last_undegraded":"2023-10-19T14:20:31.814351+0000","last_fullsized":"2023-10-19T14:20:31.814351+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:50:44.452283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2d","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.799030+0000","last_change":"2023-10-19T14:20:19.352942+0000","last_active":"2023-10-19T14:20:30.799030+0000","last_peered":"2023-10-19T14:20:30.799030+0000","last_clean":"2023-10-19T14:20:30.799030+0000","last_became_active":"2023-10-19T14:20:19.352678+0000","last_became_peered":"2023-10-19T14:20:19.352678+0000","last_unstale":"2023-10-19T14:20:30.799030+0000","last_undegraded":"2023-10-19T14:20:30.799030+0000","last_fullsized":"2023-10-19T14:20:30.799030+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:19:02.500176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2c","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.782792+0000","last_change":"2023-10-19T14:20:30.799925+0000","last_active":"2023-10-19T14:20:31.782792+0000","last_peered":"2023-10-19T14:20:31.782792+0000","last_clean":"2023-10-19T14:20:31.782792+0000","last_became_active":"2023-10-19T14:20:30.799680+0000","last_became_peered":"2023-10-19T14:20:30.799680+0000","last_unstale":"2023-10-19T14:20:31.782792+0000","last_undegraded":"2023-10-19T14:20:31.782792+0000","last_fullsized":"2023-10-19T14:20:31.782792+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:21:29.982170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.2e","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.786969+0000","last_change":"2023-10-19T14:20:19.341197+0000","last_active":"2023-10-19T14:20:30.786969+0000","last_peered":"2023-10-19T14:20:30.786969+0000","last_clean":"2023-10-19T14:20:30.786969+0000","last_became_active":"2023-10-19T14:20:19.340929+0000","last_became_peered":"2023-10-19T14:20:19.340929+0000","last_unstale":"2023-10-19T14:20:30.786969+0000","last_undegraded":"2023-10-19T14:20:30.786969+0000","last_fullsized":"2023-10-19T14:20:30.786969+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:44:28.147164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.2d","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.804426+0000","last_change":"2023-10-19T14:20:30.797134+0000","last_active":"2023-10-19T14:20:30.804426+0000","last_peered":"2023-10-19T14:20:30.804426+0000","last_clean":"2023-10-19T14:20:30.804426+0000","last_became_active":"2023-10-19T14:20:30.796900+0000","last_became_peered":"2023-10-19T14:20:30.796900+0000","last_unstale":"2023-10-19T14:20:30.804426+0000","last_undegraded":"2023-10-19T14:20:30.804426+0000","last_fullsized":"2023-10-19T14:20:30.804426+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:38:09.296939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.2f","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.788609+0000","last_change":"2023-10-19T14:20:19.344630+0000","last_active":"2023-10-19T14:20:30.788609+0000","last_peered":"2023-10-19T14:20:30.788609+0000","last_clean":"2023-10-19T14:20:30.788609+0000","last_became_active":"2023-10-19T14:20:19.344354+0000","last_became_peered":"2023-10-19T14:20:19.344354+0000","last_unstale":"2023-10-19T14:20:30.788609+0000","last_undegraded":"2023-10-19T14:20:30.788609+0000","last_fullsized":"2023-10-19T14:20:30.788609+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:10:30.171693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.32","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.813589+0000","last_change":"2023-10-19T14:20:30.810659+0000","last_active":"2023-10-19T14:20:31.813589+0000","last_peered":"2023-10-19T14:20:31.813589+0000","last_clean":"2023-10-19T14:20:31.813589+0000","last_became_active":"2023-10-19T14:20:30.810384+0000","last_became_peered":"2023-10-19T14:20:30.810384+0000","last_unstale":"2023-10-19T14:20:31.813589+0000","last_undegraded":"2023-10-19T14:20:31.813589+0000","last_fullsized":"2023-10-19T14:20:31.813589+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:29:51.812187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.30","version":"0'0","reported_seq":14,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.784095+0000","last_change":"2023-10-19T14:20:19.338108+0000","last_active":"2023-10-19T14:20:29.784095+0000","last_peered":"2023-10-19T14:20:29.784095+0000","last_clean":"2023-10-19T14:20:29.784095+0000","last_became_active":"2023-10-19T14:20:19.337834+0000","last_became_peered":"2023-10-19T14:20:19.337834+0000","last_unstale":"2023-10-19T14:20:29.784095+0000","last_undegraded":"2023-10-19T14:20:29.784095+0000","last_fullsized":"2023-10-19T14:20:29.784095+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:15:21.019725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.33","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.782574+0000","last_change":"2023-10-19T14:20:30.791799+0000","last_active":"2023-10-19T14:20:31.782574+0000","last_peered":"2023-10-19T14:20:31.782574+0000","last_clean":"2023-10-19T14:20:31.782574+0000","last_became_active":"2023-10-19T14:20:30.791484+0000","last_became_peered":"2023-10-19T14:20:30.791484+0000","last_unstale":"2023-10-19T14:20:31.782574+0000","last_undegraded":"2023-10-19T14:20:31.782574+0000","last_fullsized":"2023-10-19T14:20:31.782574+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:53:03.050006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.31","version":"0'0","reported_seq":14,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.784960+0000","last_change":"2023-10-19T14:20:19.349749+0000","last_active":"2023-10-19T14:20:29.784960+0000","last_peered":"2023-10-19T14:20:29.784960+0000","last_clean":"2023-10-19T14:20:29.784960+0000","last_became_active":"2023-10-19T14:20:19.349481+0000","last_became_peered":"2023-10-19T14:20:19.349481+0000","last_unstale":"2023-10-19T14:20:29.784960+0000","last_undegraded":"2023-10-19T14:20:29.784960+0000","last_fullsized":"2023-10-19T14:20:29.784960+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:20:35.750886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.30","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.781976+0000","last_change":"2023-10-19T14:20:30.790210+0000","last_active":"2023-10-19T14:20:31.781976+0000","last_peered":"2023-10-19T14:20:31.781976+0000","last_clean":"2023-10-19T14:20:31.781976+0000","last_became_active":"2023-10-19T14:20:30.789906+0000","last_became_peered":"2023-10-19T14:20:30.789906+0000","last_unstale":"2023-10-19T14:20:31.781976+0000","last_undegraded":"2023-10-19T14:20:31.781976+0000","last_fullsized":"2023-10-19T14:20:31.781976+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:49:15.154863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.32","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.799189+0000","last_change":"2023-10-19T14:20:19.346548+0000","last_active":"2023-10-19T14:20:30.799189+0000","last_peered":"2023-10-19T14:20:30.799189+0000","last_clean":"2023-10-19T14:20:30.799189+0000","last_became_active":"2023-10-19T14:20:19.345001+0000","last_became_peered":"2023-10-19T14:20:19.345001+0000","last_unstale":"2023-10-19T14:20:30.799189+0000","last_undegraded":"2023-10-19T14:20:30.799189+0000","last_fullsized":"2023-10-19T14:20:30.799189+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T02:03:47.400085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.31","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.806474+0000","last_change":"2023-10-19T14:20:30.803998+0000","last_active":"2023-10-19T14:20:30.806474+0000","last_peered":"2023-10-19T14:20:30.806474+0000","last_clean":"2023-10-19T14:20:30.806474+0000","last_became_active":"2023-10-19T14:20:30.803734+0000","last_became_peered":"2023-10-19T14:20:30.803734+0000","last_unstale":"2023-10-19T14:20:30.806474+0000","last_undegraded":"2023-10-19T14:20:30.806474+0000","last_fullsized":"2023-10-19T14:20:30.806474+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:28:29.776053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.33","version":"0'0","reported_seq":14,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.786335+0000","last_change":"2023-10-19T14:20:19.351639+0000","last_active":"2023-10-19T14:20:29.786335+0000","last_peered":"2023-10-19T14:20:29.786335+0000","last_clean":"2023-10-19T14:20:29.786335+0000","last_became_active":"2023-10-19T14:20:19.350821+0000","last_became_peered":"2023-10-19T14:20:19.350821+0000","last_unstale":"2023-10-19T14:20:29.786335+0000","last_undegraded":"2023-10-19T14:20:29.786335+0000","last_fullsized":"2023-10-19T14:20:29.786335+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T02:16:40.560667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.36","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.785082+0000","last_change":"2023-10-19T14:20:30.789975+0000","last_active":"2023-10-19T14:20:31.785082+0000","last_peered":"2023-10-19T14:20:31.785082+0000","last_clean":"2023-10-19T14:20:31.785082+0000","last_became_active":"2023-10-19T14:20:30.789652+0000","last_became_peered":"2023-10-19T14:20:30.789652+0000","last_unstale":"2023-10-19T14:20:31.785082+0000","last_undegraded":"2023-10-19T14:20:31.785082+0000","last_fullsized":"2023-10-19T14:20:31.785082+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:37:17.323203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.34","version":"0'0","reported_seq":14,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.781664+0000","last_change":"2023-10-19T14:20:19.350634+0000","last_active":"2023-10-19T14:20:29.781664+0000","last_peered":"2023-10-19T14:20:29.781664+0000","last_clean":"2023-10-19T14:20:29.781664+0000","last_became_active":"2023-10-19T14:20:19.350362+0000","last_became_peered":"2023-10-19T14:20:19.350362+0000","last_unstale":"2023-10-19T14:20:29.781664+0000","last_undegraded":"2023-10-19T14:20:29.781664+0000","last_fullsized":"2023-10-19T14:20:29.781664+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:30:33.652808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.37","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.799021+0000","last_change":"2023-10-19T14:20:30.798066+0000","last_active":"2023-10-19T14:20:30.799021+0000","last_peered":"2023-10-19T14:20:30.799021+0000","last_clean":"2023-10-19T14:20:30.799021+0000","last_became_active":"2023-10-19T14:20:30.797781+0000","last_became_peered":"2023-10-19T14:20:30.797781+0000","last_unstale":"2023-10-19T14:20:30.799021+0000","last_undegraded":"2023-10-19T14:20:30.799021+0000","last_fullsized":"2023-10-19T14:20:30.799021+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:20:46.825732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.35","version":"0'0","reported_seq":14,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.788609+0000","last_change":"2023-10-19T14:20:19.347686+0000","last_active":"2023-10-19T14:20:29.788609+0000","last_peered":"2023-10-19T14:20:29.788609+0000","last_clean":"2023-10-19T14:20:29.788609+0000","last_became_active":"2023-10-19T14:20:19.347432+0000","last_became_peered":"2023-10-19T14:20:19.347432+0000","last_unstale":"2023-10-19T14:20:29.788609+0000","last_undegraded":"2023-10-19T14:20:29.788609+0000","last_fullsized":"2023-10-19T14:20:29.788609+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:46:20.720183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.34","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.813864+0000","last_change":"2023-10-19T14:20:30.803838+0000","last_active":"2023-10-19T14:20:31.813864+0000","last_peered":"2023-10-19T14:20:31.813864+0000","last_clean":"2023-10-19T14:20:31.813864+0000","last_became_active":"2023-10-19T14:20:30.803512+0000","last_became_peered":"2023-10-19T14:20:30.803512+0000","last_unstale":"2023-10-19T14:20:31.813864+0000","last_undegraded":"2023-10-19T14:20:31.813864+0000","last_fullsized":"2023-10-19T14:20:31.813864+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:50:16.345467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.36","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.797815+0000","last_change":"2023-10-19T14:20:19.350276+0000","last_active":"2023-10-19T14:20:30.797815+0000","last_peered":"2023-10-19T14:20:30.797815+0000","last_clean":"2023-10-19T14:20:30.797815+0000","last_became_active":"2023-10-19T14:20:19.349877+0000","last_became_peered":"2023-10-19T14:20:19.349877+0000","last_unstale":"2023-10-19T14:20:30.797815+0000","last_undegraded":"2023-10-19T14:20:30.797815+0000","last_fullsized":"2023-10-19T14:20:30.797815+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:25:10.980817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.35","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.800120+0000","last_change":"2023-10-19T14:20:30.795581+0000","last_active":"2023-10-19T14:20:30.800120+0000","last_peered":"2023-10-19T14:20:30.800120+0000","last_clean":"2023-10-19T14:20:30.800120+0000","last_became_active":"2023-10-19T14:20:30.795262+0000","last_became_peered":"2023-10-19T14:20:30.795262+0000","last_unstale":"2023-10-19T14:20:30.800120+0000","last_undegraded":"2023-10-19T14:20:30.800120+0000","last_fullsized":"2023-10-19T14:20:30.800120+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:04:38.334593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.37","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.787262+0000","last_change":"2023-10-19T14:20:19.335860+0000","last_active":"2023-10-19T14:20:30.787262+0000","last_peered":"2023-10-19T14:20:30.787262+0000","last_clean":"2023-10-19T14:20:30.787262+0000","last_became_active":"2023-10-19T14:20:19.335522+0000","last_became_peered":"2023-10-19T14:20:19.335522+0000","last_unstale":"2023-10-19T14:20:30.787262+0000","last_undegraded":"2023-10-19T14:20:30.787262+0000","last_fullsized":"2023-10-19T14:20:30.787262+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:36:28.937721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.3a","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.782414+0000","last_change":"2023-10-19T14:20:30.788800+0000","last_active":"2023-10-19T14:20:31.782414+0000","last_peered":"2023-10-19T14:20:31.782414+0000","last_clean":"2023-10-19T14:20:31.782414+0000","last_became_active":"2023-10-19T14:20:30.788544+0000","last_became_peered":"2023-10-19T14:20:30.788544+0000","last_unstale":"2023-10-19T14:20:31.782414+0000","last_undegraded":"2023-10-19T14:20:31.782414+0000","last_fullsized":"2023-10-19T14:20:31.782414+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:26:11.781950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.38","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.800130+0000","last_change":"2023-10-19T14:20:19.347387+0000","last_active":"2023-10-19T14:20:30.800130+0000","last_peered":"2023-10-19T14:20:30.800130+0000","last_clean":"2023-10-19T14:20:30.800130+0000","last_became_active":"2023-10-19T14:20:19.347104+0000","last_became_peered":"2023-10-19T14:20:19.347104+0000","last_unstale":"2023-10-19T14:20:30.800130+0000","last_undegraded":"2023-10-19T14:20:30.800130+0000","last_fullsized":"2023-10-19T14:20:30.800130+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:13:49.536943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3b","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.798107+0000","last_change":"2023-10-19T14:20:30.795221+0000","last_active":"2023-10-19T14:20:30.798107+0000","last_peered":"2023-10-19T14:20:30.798107+0000","last_clean":"2023-10-19T14:20:30.798107+0000","last_became_active":"2023-10-19T14:20:30.794905+0000","last_became_peered":"2023-10-19T14:20:30.794905+0000","last_unstale":"2023-10-19T14:20:30.798107+0000","last_undegraded":"2023-10-19T14:20:30.798107+0000","last_fullsized":"2023-10-19T14:20:30.798107+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:33:45.656463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.39","version":"0'0","reported_seq":14,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.781557+0000","last_change":"2023-10-19T14:20:19.343249+0000","last_active":"2023-10-19T14:20:29.781557+0000","last_peered":"2023-10-19T14:20:29.781557+0000","last_clean":"2023-10-19T14:20:29.781557+0000","last_became_active":"2023-10-19T14:20:19.342989+0000","last_became_peered":"2023-10-19T14:20:19.342989+0000","last_unstale":"2023-10-19T14:20:29.781557+0000","last_undegraded":"2023-10-19T14:20:29.781557+0000","last_fullsized":"2023-10-19T14:20:29.781557+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:54:18.133180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.38","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.781967+0000","last_change":"2023-10-19T14:20:30.792568+0000","last_active":"2023-10-19T14:20:31.781967+0000","last_peered":"2023-10-19T14:20:31.781967+0000","last_clean":"2023-10-19T14:20:31.781967+0000","last_became_active":"2023-10-19T14:20:30.792070+0000","last_became_peered":"2023-10-19T14:20:30.792070+0000","last_unstale":"2023-10-19T14:20:31.781967+0000","last_undegraded":"2023-10-19T14:20:31.781967+0000","last_fullsized":"2023-10-19T14:20:31.781967+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:38:06.530226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3a","version":"0'0","reported_seq":14,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.786856+0000","last_change":"2023-10-19T14:20:19.344720+0000","last_active":"2023-10-19T14:20:29.786856+0000","last_peered":"2023-10-19T14:20:29.786856+0000","last_clean":"2023-10-19T14:20:29.786856+0000","last_became_active":"2023-10-19T14:20:19.344415+0000","last_became_peered":"2023-10-19T14:20:19.344415+0000","last_unstale":"2023-10-19T14:20:29.786856+0000","last_undegraded":"2023-10-19T14:20:29.786856+0000","last_fullsized":"2023-10-19T14:20:29.786856+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:01:45.199351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.39","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.784816+0000","last_change":"2023-10-19T14:20:30.792844+0000","last_active":"2023-10-19T14:20:31.784816+0000","last_peered":"2023-10-19T14:20:31.784816+0000","last_clean":"2023-10-19T14:20:31.784816+0000","last_became_active":"2023-10-19T14:20:30.792560+0000","last_became_peered":"2023-10-19T14:20:30.792560+0000","last_unstale":"2023-10-19T14:20:31.784816+0000","last_undegraded":"2023-10-19T14:20:31.784816+0000","last_fullsized":"2023-10-19T14:20:31.784816+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:36:43.318354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.3b","version":"0'0","reported_seq":14,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.782298+0000","last_change":"2023-10-19T14:20:19.344979+0000","last_active":"2023-10-19T14:20:29.782298+0000","last_peered":"2023-10-19T14:20:29.782298+0000","last_clean":"2023-10-19T14:20:29.782298+0000","last_became_active":"2023-10-19T14:20:19.344717+0000","last_became_peered":"2023-10-19T14:20:19.344717+0000","last_unstale":"2023-10-19T14:20:29.782298+0000","last_undegraded":"2023-10-19T14:20:29.782298+0000","last_fullsized":"2023-10-19T14:20:29.782298+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:14:31.826091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.3e","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.785876+0000","last_change":"2023-10-19T14:20:30.797158+0000","last_active":"2023-10-19T14:20:31.785876+0000","last_peered":"2023-10-19T14:20:31.785876+0000","last_clean":"2023-10-19T14:20:31.785876+0000","last_became_active":"2023-10-19T14:20:30.796872+0000","last_became_peered":"2023-10-19T14:20:30.796872+0000","last_unstale":"2023-10-19T14:20:31.785876+0000","last_undegraded":"2023-10-19T14:20:31.785876+0000","last_fullsized":"2023-10-19T14:20:31.785876+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T02:12:07.536009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.3c","version":"0'0","reported_seq":14,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.787311+0000","last_change":"2023-10-19T14:20:19.351919+0000","last_active":"2023-10-19T14:20:29.787311+0000","last_peered":"2023-10-19T14:20:29.787311+0000","last_clean":"2023-10-19T14:20:29.787311+0000","last_became_active":"2023-10-19T14:20:19.351634+0000","last_became_peered":"2023-10-19T14:20:19.351634+0000","last_unstale":"2023-10-19T14:20:29.787311+0000","last_undegraded":"2023-10-19T14:20:29.787311+0000","last_fullsized":"2023-10-19T14:20:29.787311+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:40:33.512260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.3f","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.793889+0000","last_change":"2023-10-19T14:20:30.791868+0000","last_active":"2023-10-19T14:20:30.793889+0000","last_peered":"2023-10-19T14:20:30.793889+0000","last_clean":"2023-10-19T14:20:30.793889+0000","last_became_active":"2023-10-19T14:20:30.791557+0000","last_became_peered":"2023-10-19T14:20:30.791557+0000","last_unstale":"2023-10-19T14:20:30.793889+0000","last_undegraded":"2023-10-19T14:20:30.793889+0000","last_fullsized":"2023-10-19T14:20:30.793889+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:44:39.222010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.3d","version":"0'0","reported_seq":14,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.788515+0000","last_change":"2023-10-19T14:20:19.349659+0000","last_active":"2023-10-19T14:20:29.788515+0000","last_peered":"2023-10-19T14:20:29.788515+0000","last_clean":"2023-10-19T14:20:29.788515+0000","last_became_active":"2023-10-19T14:20:19.349362+0000","last_became_peered":"2023-10-19T14:20:19.349362+0000","last_unstale":"2023-10-19T14:20:29.788515+0000","last_undegraded":"2023-10-19T14:20:29.788515+0000","last_fullsized":"2023-10-19T14:20:29.788515+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:23:13.051759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.3c","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.788732+0000","last_change":"2023-10-19T14:20:30.787587+0000","last_active":"2023-10-19T14:20:30.788732+0000","last_peered":"2023-10-19T14:20:30.788732+0000","last_clean":"2023-10-19T14:20:30.788732+0000","last_became_active":"2023-10-19T14:20:30.787318+0000","last_became_peered":"2023-10-19T14:20:30.787318+0000","last_unstale":"2023-10-19T14:20:30.788732+0000","last_undegraded":"2023-10-19T14:20:30.788732+0000","last_fullsized":"2023-10-19T14:20:30.788732+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:38:03.480323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.3e","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.785209+0000","last_change":"2023-10-19T14:20:19.346014+0000","last_active":"2023-10-19T14:20:30.785209+0000","last_peered":"2023-10-19T14:20:30.785209+0000","last_clean":"2023-10-19T14:20:30.785209+0000","last_became_active":"2023-10-19T14:20:19.345686+0000","last_became_peered":"2023-10-19T14:20:19.345686+0000","last_unstale":"2023-10-19T14:20:30.785209+0000","last_undegraded":"2023-10-19T14:20:30.785209+0000","last_fullsized":"2023-10-19T14:20:30.785209+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:29:26.735942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.3d","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.784998+0000","last_change":"2023-10-19T14:20:30.794749+0000","last_active":"2023-10-19T14:20:31.784998+0000","last_peered":"2023-10-19T14:20:31.784998+0000","last_clean":"2023-10-19T14:20:31.784998+0000","last_became_active":"2023-10-19T14:20:30.794474+0000","last_became_peered":"2023-10-19T14:20:30.794474+0000","last_unstale":"2023-10-19T14:20:31.784998+0000","last_undegraded":"2023-10-19T14:20:31.784998+0000","last_fullsized":"2023-10-19T14:20:31.784998+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:36:27.153729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.3f","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.781865+0000","last_change":"2023-10-19T14:20:19.348564+0000","last_active":"2023-10-19T14:20:30.781865+0000","last_peered":"2023-10-19T14:20:30.781865+0000","last_clean":"2023-10-19T14:20:30.781865+0000","last_became_active":"2023-10-19T14:20:19.348281+0000","last_became_peered":"2023-10-19T14:20:19.348281+0000","last_unstale":"2023-10-19T14:20:30.781865+0000","last_undegraded":"2023-10-19T14:20:30.781865+0000","last_fullsized":"2023-10-19T14:20:30.781865+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:15:21.019725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":30,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":29,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":28,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":27,"num_pg":32,"stat_sum":{"num_bytes":3666,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2210,"num_read_kb":4624,"num_write":589,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3186,"ondisk_log_size":3186,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576004,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48448,"kb_used_data":3376,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323392,"statfs":{"total":96636764160,"available":96587153408,"internally_reserved":0,"allocated":3457024,"data_stored":2051182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":46135755},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:20:09 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.56999999999999995,"15min":0.58499999999999996},"min":{"1min":0.35799999999999998,"5min":0.34100000000000003,"15min":0.32200000000000001},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.86199999999999999},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.60199999999999998,"15min":0.60999999999999999},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.312},"max":{"1min":0.77600000000000002,"5min":0.81899999999999995,"15min":0.86599999999999999},"last":0.50700000000000001}]},{"osd":1,"last update":"Thu Oct 19 14:20:09 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.59799999999999998,"15min":0.59999999999999998},"min":{"1min":0.46500000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.76800000000000002,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.59499999999999997,"15min":0.59799999999999998},"min":{"1min":0.373,"5min":0.373,"15min":0.33500000000000002},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.443}]},{"osd":2,"last update":"Thu Oct 19 14:20:09 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64900000000000002,"15min":0.64800000000000002},"min":{"1min":0.50900000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.85499999999999998},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.66600000000000004,"15min":0.66200000000000003},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.40100000000000002},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.46000000000000002}]},{"osd":3,"last update":"Thu Oct 19 14:20:09 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.63300000000000001,"15min":0.63700000000000001},"min":{"1min":0.51100000000000001,"5min":0.42099999999999999,"15min":0.36399999999999999},"max":{"1min":0.82699999999999996,"5min":0.84799999999999998,"15min":0.873},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.68799999999999994,"15min":0.68700000000000006},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":0.873,"5min":0.876,"15min":0.89600000000000002},"last":0.497}]},{"osd":4,"last update":"Thu Oct 19 14:20:09 2023","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.50600000000000001,"15min":0.49299999999999999},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.62,"5min":0.78100000000000003,"15min":0.83799999999999997},"last":0.311},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.51800000000000002,"15min":0.50600000000000001},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.71899999999999997,"5min":0.80200000000000005,"15min":0.81699999999999995},"last":0.36299999999999999}]},{"osd":5,"last update":"Thu Oct 19 14:20:09 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51400000000000001,"15min":0.50800000000000001},"min":{"1min":0.36799999999999999,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.65600000000000003,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.55600000000000005,"15min":0.54900000000000004},"min":{"1min":0.377,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.70999999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.42599999999999999}]},{"osd":6,"last update":"Thu Oct 19 14:20:09 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57699999999999996,"15min":0.57399999999999995},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.35999999999999999},"max":{"1min":0.73199999999999998,"5min":0.82999999999999996,"15min":0.85099999999999998},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.58099999999999996,"15min":0.56799999999999995},"min":{"1min":0.38800000000000001,"5min":0.35599999999999998,"15min":0.33500000000000002},"max":{"1min":0.78200000000000003,"5min":0.85199999999999998,"15min":0.85999999999999999},"last":0.45000000000000001}]}]},{"osd":6,"up_from":13,"seq":55834576004,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38152,"kb_used_data":2812,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333688,"statfs":{"total":96636764160,"available":96597696512,"internally_reserved":0,"allocated":2879488,"data_stored":1462392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:20:00 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.60799999999999998,"15min":0.61499999999999999},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.28799999999999998},"max":{"1min":0.81200000000000006,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.59399999999999997,"15min":0.61499999999999999},"min":{"1min":0.40699999999999997,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.436}]},{"osd":1,"last update":"Thu Oct 19 14:20:00 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.621,"15min":0.621},"min":{"1min":0.52300000000000002,"5min":0.38200000000000001,"15min":0.35799999999999998},"max":{"1min":0.79500000000000004,"5min":0.80200000000000005,"15min":1.165},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.624,"15min":0.63500000000000001},"min":{"1min":0.503,"5min":0.439,"15min":0.36899999999999999},"max":{"1min":0.81599999999999995,"5min":0.83699999999999997,"15min":0.91300000000000003},"last":0.33700000000000002}]},{"osd":2,"last update":"Thu Oct 19 14:20:00 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.60099999999999998,"15min":0.60899999999999999},"min":{"1min":0.442,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.873},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.626,"15min":0.629},"min":{"1min":0.45000000000000001,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.93100000000000005},"last":0.56399999999999995}]},{"osd":3,"last update":"Thu Oct 19 14:20:00 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.65500000000000003,"15min":0.65800000000000003},"min":{"1min":0.53000000000000003,"5min":0.372,"15min":0.34300000000000003},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.65600000000000003,"15min":0.66100000000000003},"min":{"1min":0.48299999999999998,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.84399999999999997,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.622}]},{"osd":4,"last update":"Thu Oct 19 14:20:00 2023","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.47999999999999998,"15min":0.48199999999999998},"min":{"1min":0.34499999999999997,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.79600000000000004},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.502,"15min":0.50800000000000001},"min":{"1min":0.35399999999999998,"5min":0.34300000000000003,"15min":0.30399999999999999},"max":{"1min":0.74199999999999999,"5min":0.82199999999999995,"15min":0.85699999999999998},"last":0.69999999999999996}]},{"osd":5,"last update":"Thu Oct 19 14:20:00 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.5,"15min":0.5},"min":{"1min":0.35599999999999998,"5min":0.32700000000000001,"15min":0.32500000000000001},"max":{"1min":0.747,"5min":1.115,"15min":1.115},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.55100000000000005,"15min":0.54300000000000004},"min":{"1min":0.40000000000000002,"5min":0.35799999999999998,"15min":0.32400000000000001},"max":{"1min":0.78700000000000003,"5min":0.86499999999999999,"15min":0.878},"last":0.749}]},{"osd":7,"last update":"Thu Oct 19 14:20:00 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.55000000000000004,"15min":0.54600000000000004},"min":{"1min":0.39900000000000002,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.77600000000000002,"5min":0.86399999999999999,"15min":0.90100000000000002},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.55700000000000005,"15min":0.55200000000000005},"min":{"1min":0.39400000000000002,"5min":0.38200000000000001,"15min":0.36199999999999999},"max":{"1min":0.85099999999999998,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.66600000000000004}]}]},{"osd":5,"up_from":13,"seq":55834576004,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47864,"kb_used_data":2796,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323976,"statfs":{"total":96636764160,"available":96587751424,"internally_reserved":0,"allocated":2863104,"data_stored":1458942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:19:46 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.59899999999999998,"15min":0.621},"min":{"1min":0.44900000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.79700000000000004,"5min":0.82599999999999996,"15min":0.86399999999999999},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.63200000000000001,"15min":0.64500000000000002},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.82499999999999996,"5min":0.85899999999999999,"15min":0.85999999999999999},"last":0.34599999999999997}]},{"osd":1,"last update":"Thu Oct 19 14:19:46 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.61799999999999999,"15min":0.63500000000000001},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.78800000000000003,"5min":0.83699999999999997,"15min":0.877},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.64100000000000001,"15min":0.64500000000000002},"min":{"1min":0.47999999999999998,"5min":0.44700000000000001,"15min":0.41199999999999998},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.88500000000000001},"last":0.504}]},{"osd":2,"last update":"Thu Oct 19 14:19:46 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.61499999999999999,"15min":0.627},"min":{"1min":0.44800000000000001,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.81299999999999994,"5min":0.83799999999999997,"15min":0.85099999999999998},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.65200000000000002,"15min":0.66100000000000003},"min":{"1min":0.52000000000000002,"5min":0.49399999999999999,"15min":0.47099999999999997},"max":{"1min":0.79500000000000004,"5min":0.83299999999999996,"15min":0.90800000000000003},"last":0.45000000000000001}]},{"osd":3,"last update":"Thu Oct 19 14:19:46 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.63900000000000001,"15min":0.64700000000000002},"min":{"1min":0.51100000000000001,"5min":0.443,"15min":0.443},"max":{"1min":0.84399999999999997,"5min":0.86099999999999999,"15min":0.90100000000000002},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.66500000000000004,"15min":0.67400000000000004},"min":{"1min":0.53300000000000003,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.84899999999999998,"5min":0.876,"15min":0.90900000000000003},"last":0.436}]},{"osd":4,"last update":"Thu Oct 19 14:19:46 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.497,"15min":0.48699999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.32500000000000001},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.371},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50900000000000001,"15min":0.50600000000000001},"min":{"1min":0.35499999999999998,"5min":0.318,"15min":0.28599999999999998},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.82899999999999996},"last":0.55600000000000005}]},{"osd":6,"last update":"Thu Oct 19 14:19:46 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51200000000000001,"15min":0.497},"min":{"1min":0.37,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.88800000000000001},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.54100000000000004,"15min":0.53900000000000003},"min":{"1min":0.39100000000000001,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.77900000000000003,"5min":0.78000000000000003,"15min":0.88100000000000001},"last":0.65100000000000002}]},{"osd":7,"last update":"Thu Oct 19 14:19:46 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.51500000000000001,"15min":0.52000000000000002},"min":{"1min":0.41799999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.749,"5min":0.749,"15min":0.82899999999999996},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.58999999999999997,"15min":0.57499999999999996},"min":{"1min":0.47099999999999997,"5min":0.38400000000000001,"15min":0.34300000000000003},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.61799999999999999}]}]},{"osd":4,"up_from":13,"seq":55834576004,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47856,"kb_used_data":2780,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323984,"statfs":{"total":96636764160,"available":96587759616,"internally_reserved":0,"allocated":2846720,"data_stored":1457876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.55500000000000005,"15min":0.57599999999999996},"min":{"1min":0.35499999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.80800000000000005,"5min":0.81799999999999995,"15min":0.879},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.58499999999999996,"15min":0.59799999999999998},"min":{"1min":0.499,"5min":0.31,"15min":0.31},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.873},"last":0.51000000000000001}]},{"osd":1,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.61499999999999999,"15min":0.64100000000000001},"min":{"1min":0.504,"5min":0.372,"15min":0.32600000000000001},"max":{"1min":0.80400000000000005,"5min":0.80700000000000005,"15min":0.92700000000000005},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.623,"15min":0.64400000000000002},"min":{"1min":0.443,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":1.0940000000000001},"last":0.56599999999999995}]},{"osd":2,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.63400000000000001,"15min":0.65000000000000002},"min":{"1min":0.46400000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.64800000000000002,"15min":0.66900000000000004},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":1.008},"last":0.48499999999999999}]},{"osd":3,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.61299999999999999,"15min":0.63400000000000001},"min":{"1min":0.47399999999999998,"5min":0.37,"15min":0.37},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.91900000000000004},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.66600000000000004,"15min":0.66700000000000004},"min":{"1min":0.54000000000000004,"5min":0.45400000000000001,"15min":0.437},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.57899999999999996}]},{"osd":5,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.48299999999999998,"15min":0.47099999999999997},"min":{"1min":0.34000000000000002,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.504,"15min":0.49199999999999999},"min":{"1min":0.34300000000000003,"5min":0.33700000000000002,"15min":0.29199999999999998},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.38700000000000001}]},{"osd":6,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.496,"15min":0.48599999999999999},"min":{"1min":0.34799999999999998,"5min":0.33900000000000002,"15min":0.28199999999999997},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.496},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.53700000000000003},"min":{"1min":0.39100000000000001,"5min":0.375,"15min":0.33300000000000002},"max":{"1min":0.81299999999999994,"5min":0.84799999999999998,"15min":1.2969999999999999},"last":0.52900000000000003}]},{"osd":7,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.55100000000000005,"15min":0.54100000000000004},"min":{"1min":0.36499999999999999,"5min":0.35699999999999998,"15min":0.32600000000000001},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60499999999999998,"15min":0.59199999999999997},"min":{"1min":0.47099999999999997,"5min":0.32100000000000001,"15min":0.318},"max":{"1min":0.80400000000000005,"5min":0.88600000000000001,"15min":1.032},"last":0.60199999999999998}]}]},{"osd":3,"up_from":13,"seq":55834576003,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56180,"kb_used_data":2788,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315660,"statfs":{"total":96636764160,"available":96579235840,"internally_reserved":0,"allocated":2854912,"data_stored":1457352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54655432},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:19:48 2023","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42299999999999999},"min":{"1min":0.316,"5min":0.253,"15min":0.253},"max":{"1min":0.60599999999999998,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.42599999999999999,"15min":0.432},"min":{"1min":0.29399999999999998,"5min":0.219,"15min":0.219},"max":{"1min":0.57599999999999996,"5min":0.752,"15min":0.752},"last":0.32800000000000001}]},{"osd":1,"last update":"Thu Oct 19 14:19:48 2023","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.439,"15min":0.433},"min":{"1min":0.29599999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.56599999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.374},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.30099999999999999,"5min":0.29399999999999998,"15min":0.26400000000000001},"max":{"1min":0.56599999999999995,"5min":0.73099999999999998,"15min":0.81799999999999995},"last":0.39000000000000001}]},{"osd":2,"last update":"Thu Oct 19 14:19:48 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.47699999999999998,"15min":0.48599999999999999},"min":{"1min":0.39600000000000002,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.64300000000000002,"5min":0.64600000000000002,"15min":0.73099999999999998},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.499,"15min":0.50800000000000001},"min":{"1min":0.33500000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.66800000000000004,"5min":0.67000000000000004,"15min":0.83699999999999997},"last":0.36099999999999999}]},{"osd":4,"last update":"Thu Oct 19 14:19:48 2023","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.71099999999999997,"15min":0.70899999999999996},"min":{"1min":0.45000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.96099999999999997,"5min":1.018,"15min":1.157},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.57999999999999996,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.93100000000000005,"5min":0.996,"15min":1.173},"last":0.56799999999999995}]},{"osd":5,"last update":"Thu Oct 19 14:19:48 2023","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.69499999999999995,"15min":0.69199999999999995},"min":{"1min":0.55000000000000004,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.93100000000000005,"5min":0.997,"15min":1.036},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.69799999999999995,"15min":0.69299999999999995},"min":{"1min":0.47399999999999998,"5min":0.42999999999999999,"15min":0.42599999999999999},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":1.105},"last":0.502}]},{"osd":6,"last update":"Thu Oct 19 14:19:48 2023","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.68500000000000005,"15min":0.69499999999999995},"min":{"1min":0.51900000000000002,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.99099999999999999,"5min":1.018,"15min":1.0669999999999999},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.55800000000000005,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.88800000000000001,"5min":0.91500000000000004,"15min":1.079},"last":0.53500000000000003}]},{"osd":7,"last update":"Thu Oct 19 14:19:48 2023","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.70799999999999996,"15min":0.70499999999999996},"min":{"1min":0.51600000000000001,"5min":0.434,"15min":0.434},"max":{"1min":0.94899999999999995,"5min":0.96899999999999997,"15min":1.085},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.73199999999999998,"15min":0.73599999999999999},"min":{"1min":0.57099999999999995,"5min":0.51900000000000002,"15min":0.51300000000000001},"max":{"1min":0.96499999999999997,"5min":0.97199999999999998,"15min":1.1739999999999999},"last":0.51700000000000002}]}]},{"osd":2,"up_from":13,"seq":55834576005,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64492,"kb_used_data":2784,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94307348,"statfs":{"total":96636764160,"available":96570724352,"internally_reserved":0,"allocated":2850816,"data_stored":1457910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":63175110},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.39900000000000002,"15min":0.40200000000000002},"min":{"1min":0.28199999999999997,"5min":0.218,"15min":0.218},"max":{"1min":0.623,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.254},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.44500000000000001,"15min":0.441},"min":{"1min":0.34100000000000003,"5min":0.249,"15min":0.223},"max":{"1min":0.628,"5min":0.71399999999999997,"15min":0.73599999999999999},"last":0.38300000000000001}]},{"osd":1,"last update":"Thu Oct 19 14:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.40500000000000003,"15min":0.41599999999999998},"min":{"1min":0.28999999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.48999999999999999,"5min":0.61699999999999999,"15min":0.65400000000000003},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.47999999999999998,"15min":0.47099999999999997},"min":{"1min":0.33000000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":1.742,"5min":1.742,"15min":1.742},"last":0.32900000000000001}]},{"osd":3,"last update":"Thu Oct 19 14:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.48199999999999998,"15min":0.48399999999999999},"min":{"1min":0.313,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.56999999999999995,"5min":0.69299999999999995,"15min":0.74399999999999999},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.499,"15min":0.51100000000000001},"min":{"1min":0.38200000000000001,"5min":0.32000000000000001,"15min":0.315},"max":{"1min":0.60599999999999998,"5min":0.67600000000000005,"15min":0.83599999999999997},"last":0.36499999999999999}]},{"osd":4,"last update":"Thu Oct 19 14:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.66900000000000004,"15min":0.67000000000000004},"min":{"1min":0.44900000000000001,"5min":0.39600000000000002,"15min":0.39500000000000002},"max":{"1min":0.88600000000000001,"5min":0.93899999999999995,"15min":0.998},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.67300000000000004,"15min":0.68200000000000005},"min":{"1min":0.51700000000000002,"5min":0.38900000000000001,"15min":0.35199999999999998},"max":{"1min":0.79500000000000004,"5min":0.97099999999999997,"15min":1.0289999999999999},"last":0.65900000000000003}]},{"osd":5,"last update":"Thu Oct 19 14:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.70499999999999996,"15min":0.70199999999999996},"min":{"1min":0.53300000000000003,"5min":0.45100000000000001,"15min":0.40600000000000003},"max":{"1min":1.4330000000000001,"5min":1.4330000000000001,"15min":1.4330000000000001},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.68400000000000005,"15min":0.68899999999999995},"min":{"1min":0.48199999999999998,"5min":0.45100000000000001,"15min":0.39300000000000002},"max":{"1min":1.595,"5min":1.595,"15min":1.595},"last":0.72499999999999998}]},{"osd":6,"last update":"Thu Oct 19 14:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.70299999999999996,"15min":0.71199999999999997},"min":{"1min":0.46500000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.85999999999999999,"5min":0.92800000000000005,"15min":1.0189999999999999},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.72899999999999998,"15min":0.72599999999999998},"min":{"1min":0.47899999999999998,"5min":0.433,"15min":0.40600000000000003},"max":{"1min":1.4930000000000001,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":0.63900000000000001}]},{"osd":7,"last update":"Thu Oct 19 14:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.71999999999999997,"15min":0.72899999999999998},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.41799999999999998},"max":{"1min":0.97999999999999998,"5min":1.0129999999999999,"15min":1.0289999999999999},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.73899999999999999,"15min":0.73499999999999999},"min":{"1min":0.503,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.53,"5min":1.53,"15min":1.53},"last":0.91100000000000003}]}]},{"osd":1,"up_from":13,"seq":55834576004,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47844,"kb_used_data":2772,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323996,"statfs":{"total":96636764160,"available":96587771904,"internally_reserved":0,"allocated":2838528,"data_stored":1457204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:19:35 2023","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.40300000000000002,"15min":0.40699999999999997},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.23100000000000001},"max":{"1min":0.48799999999999999,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.45400000000000001,"15min":0.45700000000000002},"min":{"1min":0.28299999999999997,"5min":0.27500000000000002,"15min":0.251},"max":{"1min":0.61399999999999999,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.26800000000000002}]},{"osd":2,"last update":"Thu Oct 19 14:19:35 2023","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.44500000000000001,"15min":0.44600000000000001},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.58599999999999997,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.251},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.495,"15min":0.49199999999999999},"min":{"1min":0.34399999999999997,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.64400000000000002,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.33700000000000002}]},{"osd":3,"last update":"Thu Oct 19 14:19:35 2023","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.51600000000000001,"15min":0.50800000000000001},"min":{"1min":0.38400000000000001,"5min":0.36299999999999999,"15min":0.29499999999999998},"max":{"1min":0.64400000000000002,"5min":0.76000000000000001,"15min":0.78900000000000003},"last":0.31},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.44600000000000001,"15min":0.44900000000000001},"min":{"1min":0.32000000000000001,"5min":0.315,"15min":0.26100000000000001},"max":{"1min":0.65200000000000002,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.38}]},{"osd":4,"last update":"Thu Oct 19 14:19:35 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.67000000000000004,"15min":0.67100000000000004},"min":{"1min":0.39900000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.84399999999999997,"5min":1.042,"15min":1.042},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"min":{"1min":0.45100000000000001,"5min":0.41299999999999998,"15min":0.32300000000000001},"max":{"1min":0.88800000000000001,"5min":0.93500000000000005,"15min":1.1659999999999999},"last":0.747}]},{"osd":5,"last update":"Thu Oct 19 14:19:35 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.69199999999999995,"15min":0.68700000000000006},"min":{"1min":0.45900000000000002,"5min":0.42799999999999999,"15min":0.42499999999999999},"max":{"1min":0.83099999999999996,"5min":1.0589999999999999,"15min":1.3500000000000001},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.71199999999999997,"15min":0.70799999999999996},"min":{"1min":0.44,"5min":0.44,"15min":0.40500000000000003},"max":{"1min":0.78600000000000003,"5min":1.0149999999999999,"15min":1.121},"last":0.61699999999999999}]},{"osd":6,"last update":"Thu Oct 19 14:19:35 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.67300000000000004,"15min":0.68300000000000005},"min":{"1min":0.42499999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.77200000000000002,"5min":1.0129999999999999,"15min":1.2310000000000001},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.71999999999999997,"15min":0.71499999999999997},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.86399999999999999,"5min":1.107,"15min":1.274},"last":0.67900000000000005}]},{"osd":7,"last update":"Thu Oct 19 14:19:35 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.72799999999999998,"15min":0.72399999999999998},"min":{"1min":0.46999999999999997,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.78600000000000003,"5min":1.075,"15min":1.075},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.47199999999999998,"5min":0.46100000000000002,"15min":0.44400000000000001},"max":{"1min":0.79600000000000004,"5min":1.0169999999999999,"15min":1.117},"last":0.72799999999999998}]}]},{"osd":0,"up_from":13,"seq":55834576005,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60900,"kb_used_data":3352,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310940,"statfs":{"total":96636764160,"available":96574402560,"internally_reserved":0,"allocated":3432448,"data_stored":2047043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.40699999999999997,"15min":0.41299999999999998},"min":{"1min":0.29599999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.56299999999999994,"5min":0.61499999999999999,"15min":0.624},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.441,"15min":0.46100000000000002},"min":{"1min":0.26800000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.56799999999999995,"5min":0.70899999999999996,"15min":0.81100000000000005},"last":0.33000000000000002}]},{"osd":2,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.496,"15min":0.48399999999999999},"min":{"1min":0.39300000000000002,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.71499999999999997,"5min":0.754,"15min":0.754},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.44800000000000001},"min":{"1min":0.32700000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.60199999999999998,"5min":0.63100000000000001,"15min":0.69099999999999995},"last":0.38800000000000001}]},{"osd":3,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.432,"15min":0.44},"min":{"1min":0.26900000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.58799999999999997,"5min":0.61099999999999999,"15min":0.66800000000000004},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.501,"5min":0.5,"15min":0.51200000000000001},"min":{"1min":0.35199999999999998,"5min":0.254,"15min":0.254},"max":{"1min":0.78100000000000003,"5min":0.79800000000000004,"15min":0.84599999999999997},"last":0.37}]},{"osd":4,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.64800000000000002,"15min":0.67700000000000005},"min":{"1min":0.48799999999999999,"5min":0.44600000000000001,"15min":0.44},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":1.1890000000000001},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.65800000000000003,"15min":0.68200000000000005},"min":{"1min":0.47499999999999998,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.751,"5min":0.95399999999999996,"15min":1.2150000000000001},"last":0.57399999999999995}]},{"osd":5,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.68999999999999995,"15min":0.72199999999999998},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":1.1579999999999999},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.69499999999999995,"15min":0.71999999999999997},"min":{"1min":0.57799999999999996,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.85399999999999998,"5min":0.98299999999999998,"15min":1.2310000000000001},"last":0.65700000000000003}]},{"osd":6,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.68700000000000006,"15min":0.70599999999999996},"min":{"1min":0.47799999999999998,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":1.1739999999999999},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.70399999999999996},"min":{"1min":0.54100000000000004,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.88900000000000001,"5min":0.97799999999999998,"15min":1.0189999999999999},"last":0.63400000000000001}]},{"osd":7,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.68600000000000005,"15min":0.71599999999999997},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":1.1180000000000001},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.68200000000000005,"15min":0.70399999999999996},"min":{"1min":0.53600000000000003,"5min":0.46899999999999997,"15min":0.45100000000000001},"max":{"1min":0.90600000000000003,"5min":0.93799999999999994,"15min":1.0349999999999999},"last":0.61899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-19T14:20:33.826 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-19T14:20:33.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.961+0000 7fb61f640640 1 -- 172.21.15.57:0/874833634 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb61815b2a0 msgr2=0x7fb6180d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:33.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.961+0000 7fb61f640640 1 --2- 172.21.15.57:0/874833634 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb61815b2a0 0x7fb6180d1910 secure :-1 s=READY pgs=2420 cs=0 l=1 rev1=1 crypto rx=0x7fb60c009830 tx=0x7fb60c02bd60 comp rx=0 tx=0).stop 2023-10-19T14:20:33.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.961+0000 7fb61f640640 1 -- 172.21.15.57:0/874833634 shutdown_connections 2023-10-19T14:20:33.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.961+0000 7fb61f640640 1 --2- 172.21.15.57:0/874833634 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb61815b2a0 0x7fb6180d1910 unknown :-1 s=CLOSED pgs=2420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:33.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.961+0000 7fb61f640640 1 -- 172.21.15.57:0/874833634 >> 172.21.15.57:0/874833634 conn(0x7fb6180d0d60 msgr2=0x7fb6180d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:33.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.961+0000 7fb61f640640 1 -- 172.21.15.57:0/874833634 shutdown_connections 2023-10-19T14:20:33.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.961+0000 7fb61f640640 1 -- 172.21.15.57:0/874833634 wait complete. 2023-10-19T14:20:33.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.962+0000 7fb61f640640 1 Processor -- start 2023-10-19T14:20:33.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.962+0000 7fb61f640640 1 -- start start 2023-10-19T14:20:33.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.963+0000 7fb61f640640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb6180ceb40 0x7fb6181f26a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:33.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.963+0000 7fb61f640640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb6180cf340 0x7fb618158620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:33.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.963+0000 7fb61f640640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb61815b2a0 0x7fb6180c9c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:33.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.963+0000 7fb61f640640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb618158b60 con 0x7fb61815b2a0 2023-10-19T14:20:33.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.963+0000 7fb61f640640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6181f18a0 con 0x7fb6180cf340 2023-10-19T14:20:33.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.963+0000 7fb61f640640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6181f1b80 con 0x7fb6180ceb40 2023-10-19T14:20:33.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.963+0000 7fb61dbdd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb61815b2a0 0x7fb6180c9c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:33.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.963+0000 7fb61dbdd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb61815b2a0 0x7fb6180c9c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:58904/0 (socket says 172.21.15.57:58904) 2023-10-19T14:20:33.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.963+0000 7fb61dbdd640 1 -- 172.21.15.57:0/1328875360 learned_addr learned my addr 172.21.15.57:0/1328875360 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:33.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.963+0000 7fb61cbdb640 1 --2- 172.21.15.57:0/1328875360 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb6180cf340 0x7fb618158620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:33.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.963+0000 7fb61d3dc640 1 --2- 172.21.15.57:0/1328875360 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb6180ceb40 0x7fb6181f26a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:33.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.964+0000 7fb61dbdd640 1 -- 172.21.15.57:0/1328875360 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb6180ceb40 msgr2=0x7fb6181f26a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:33.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.964+0000 7fb61dbdd640 1 --2- 172.21.15.57:0/1328875360 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb6180ceb40 0x7fb6181f26a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:33.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.964+0000 7fb61dbdd640 1 -- 172.21.15.57:0/1328875360 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb6180cf340 msgr2=0x7fb618158620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:33.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.964+0000 7fb61dbdd640 1 --2- 172.21.15.57:0/1328875360 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb6180cf340 0x7fb618158620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:33.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.964+0000 7fb61dbdd640 1 -- 172.21.15.57:0/1328875360 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb60c009490 con 0x7fb61815b2a0 2023-10-19T14:20:33.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.964+0000 7fb61cbdb640 1 --2- 172.21.15.57:0/1328875360 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb6180cf340 0x7fb618158620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:20:33.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.964+0000 7fb61dbdd640 1 --2- 172.21.15.57:0/1328875360 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb61815b2a0 0x7fb6180c9c40 secure :-1 s=READY pgs=2421 cs=0 l=1 rev1=1 crypto rx=0x7fb61000c430 tx=0x7fb61000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:33.973 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.964+0000 7fb615ffb640 1 -- 172.21.15.57:0/1328875360 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6100035f0 con 0x7fb61815b2a0 2023-10-19T14:20:33.973 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.964+0000 7fb615ffb640 1 -- 172.21.15.57:0/1328875360 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fb610003770 con 0x7fb61815b2a0 2023-10-19T14:20:33.973 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.964+0000 7fb61f640640 1 -- 172.21.15.57:0/1328875360 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6181f1e60 con 0x7fb61815b2a0 2023-10-19T14:20:33.973 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.964+0000 7fb615ffb640 1 -- 172.21.15.57:0/1328875360 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb610003b80 con 0x7fb61815b2a0 2023-10-19T14:20:33.974 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.964+0000 7fb61f640640 1 -- 172.21.15.57:0/1328875360 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6181f2420 con 0x7fb61815b2a0 2023-10-19T14:20:33.974 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.965+0000 7fb61f640640 1 -- 172.21.15.57:0/1328875360 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6180690e0 con 0x7fb61815b2a0 2023-10-19T14:20:33.974 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.966+0000 7fb615ffb640 1 -- 172.21.15.57:0/1328875360 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb610003ce0 con 0x7fb61815b2a0 2023-10-19T14:20:33.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.966+0000 7fb615ffb640 1 --2- 172.21.15.57:0/1328875360 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fb60406af60 0x7fb60406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:33.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.966+0000 7fb615ffb640 1 -- 172.21.15.57:0/1328875360 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 6921+0+0 (secure 0 0 0) 0x7fb610089430 con 0x7fb61815b2a0 2023-10-19T14:20:33.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.966+0000 7fb61d3dc640 1 --2- 172.21.15.57:0/1328875360 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fb60406af60 0x7fb60406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:33.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.967+0000 7fb61d3dc640 1 --2- 172.21.15.57:0/1328875360 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fb60406af60 0x7fb60406d420 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb61815d5d0 tx=0x7fb608009040 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:33.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:33.970+0000 7fb615ffb640 1 -- 172.21.15.57:0/1328875360 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb610053f10 con 0x7fb61815b2a0 2023-10-19T14:20:34.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:34.141+0000 7fb61f640640 1 -- 172.21.15.57:0/1328875360 --> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb61815d640 con 0x7fb60406af60 2023-10-19T14:20:34.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:34.154+0000 7fb615ffb640 1 -- 172.21.15.57:0/1328875360 <== mgr.9107 v2:172.21.15.179:6832/3232482678 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+528945 (secure 0 0 0) 0x7fb61815d640 con 0x7fb60406af60 2023-10-19T14:20:34.163 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:20:34.167 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2023-10-19T14:20:34.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:34.160+0000 7fb61f640640 1 -- 172.21.15.57:0/1328875360 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fb60406af60 msgr2=0x7fb60406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:34.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:34.160+0000 7fb61f640640 1 --2- 172.21.15.57:0/1328875360 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fb60406af60 0x7fb60406d420 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb61815d5d0 tx=0x7fb608009040 comp rx=0 tx=0).stop 2023-10-19T14:20:34.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:34.160+0000 7fb61f640640 1 -- 172.21.15.57:0/1328875360 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb61815b2a0 msgr2=0x7fb6180c9c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:34.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:34.160+0000 7fb61f640640 1 --2- 172.21.15.57:0/1328875360 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb61815b2a0 0x7fb6180c9c40 secure :-1 s=READY pgs=2421 cs=0 l=1 rev1=1 crypto rx=0x7fb61000c430 tx=0x7fb61000c8f0 comp rx=0 tx=0).stop 2023-10-19T14:20:34.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:34.160+0000 7fb61f640640 1 -- 172.21.15.57:0/1328875360 shutdown_connections 2023-10-19T14:20:34.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:34.160+0000 7fb61f640640 1 --2- 172.21.15.57:0/1328875360 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fb60406af60 0x7fb60406d420 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:34.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:34.160+0000 7fb61f640640 1 --2- 172.21.15.57:0/1328875360 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb61815b2a0 0x7fb6180c9c40 unknown :-1 s=CLOSED pgs=2421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:34.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:34.160+0000 7fb61f640640 1 --2- 172.21.15.57:0/1328875360 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb6180cf340 0x7fb618158620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:34.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:34.160+0000 7fb61f640640 1 --2- 172.21.15.57:0/1328875360 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb6180ceb40 0x7fb6181f26a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:34.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:34.160+0000 7fb61f640640 1 -- 172.21.15.57:0/1328875360 >> 172.21.15.57:0/1328875360 conn(0x7fb6180d0d60 msgr2=0x7fb6180ad8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:34.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:34.160+0000 7fb61f640640 1 -- 172.21.15.57:0/1328875360 shutdown_connections 2023-10-19T14:20:34.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:34.160+0000 7fb61f640640 1 -- 172.21.15.57:0/1328875360 wait complete. 2023-10-19T14:20:34.173 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":33,"stamp":"2023-10-19T14:20:33.992509+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596444,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":2210,"num_read_kb":4624,"num_write":680,"num_write_kb":2446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3265,"ondisk_log_size":3265,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":328,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":411736,"kb_used_data":23460,"kb_used_omap":12,"kb_used_meta":388147,"kb_avail":754562984,"statfs":{"total":773094113280,"available":772672495616,"internally_reserved":0,"allocated":24023040,"data_stored":12849901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":397463097},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3225,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.218852"},"pg_stats":[{"pgid":"30.1f","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.784610+0000","last_change":"2023-10-19T14:20:30.803400+0000","last_active":"2023-10-19T14:20:31.784610+0000","last_peered":"2023-10-19T14:20:31.784610+0000","last_clean":"2023-10-19T14:20:31.784610+0000","last_became_active":"2023-10-19T14:20:30.803141+0000","last_became_peered":"2023-10-19T14:20:30.803141+0000","last_unstale":"2023-10-19T14:20:31.784610+0000","last_undegraded":"2023-10-19T14:20:31.784610+0000","last_fullsized":"2023-10-19T14:20:31.784610+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:44:52.163309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"211'3186","reported_seq":4086,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.786003+0000","last_change":"2023-10-19T12:44:08.636613+0000","last_active":"2023-10-19T14:20:29.786003+0000","last_peered":"2023-10-19T14:20:29.786003+0000","last_clean":"2023-10-19T14:20:29.786003+0000","last_became_active":"2023-10-19T12:44:08.635697+0000","last_became_peered":"2023-10-19T12:44:08.635697+0000","last_unstale":"2023-10-19T14:20:29.786003+0000","last_undegraded":"2023-10-19T14:20:29.786003+0000","last_fullsized":"2023-10-19T14:20:29.786003+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_clean_scrub_stamp":"2023-10-19T12:44:07.399738+0000","objects_scrubbed":0,"log_size":3186,"log_dups_size":0,"ondisk_log_size":3186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:15:23.685485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2210,"num_read_kb":4624,"num_write":589,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1a","version":"222'1","reported_seq":17,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.786649+0000","last_change":"2023-10-19T14:20:18.723791+0000","last_active":"2023-10-19T14:20:30.786649+0000","last_peered":"2023-10-19T14:20:30.786649+0000","last_clean":"2023-10-19T14:20:30.786649+0000","last_became_active":"2023-10-19T14:20:18.723476+0000","last_became_peered":"2023-10-19T14:20:18.723476+0000","last_unstale":"2023-10-19T14:20:30.786649+0000","last_undegraded":"2023-10-19T14:20:30.786649+0000","last_fullsized":"2023-10-19T14:20:30.786649+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:07:17.723824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1d","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.782044+0000","last_change":"2023-10-19T14:20:19.350783+0000","last_active":"2023-10-19T14:20:30.782044+0000","last_peered":"2023-10-19T14:20:30.782044+0000","last_clean":"2023-10-19T14:20:30.782044+0000","last_became_active":"2023-10-19T14:20:19.350441+0000","last_became_peered":"2023-10-19T14:20:19.350441+0000","last_unstale":"2023-10-19T14:20:30.782044+0000","last_undegraded":"2023-10-19T14:20:30.782044+0000","last_fullsized":"2023-10-19T14:20:30.782044+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:47:00.028101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.1c","version":"226'2","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.812900+0000","last_change":"2023-10-19T14:20:29.796419+0000","last_active":"2023-10-19T14:20:30.812900+0000","last_peered":"2023-10-19T14:20:30.812900+0000","last_clean":"2023-10-19T14:20:30.812900+0000","last_became_active":"2023-10-19T14:20:29.796167+0000","last_became_peered":"2023-10-19T14:20:29.796167+0000","last_unstale":"2023-10-19T14:20:30.812900+0000","last_undegraded":"2023-10-19T14:20:30.812900+0000","last_fullsized":"2023-10-19T14:20:30.812900+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_clean_scrub_stamp":"2023-10-19T14:20:28.763908+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:47:00.349856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.5","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.804883+0000","last_change":"2023-10-19T14:20:30.798451+0000","last_active":"2023-10-19T14:20:30.804883+0000","last_peered":"2023-10-19T14:20:30.804883+0000","last_clean":"2023-10-19T14:20:30.804883+0000","last_became_active":"2023-10-19T14:20:30.796684+0000","last_became_peered":"2023-10-19T14:20:30.796684+0000","last_unstale":"2023-10-19T14:20:30.804883+0000","last_undegraded":"2023-10-19T14:20:30.804883+0000","last_fullsized":"2023-10-19T14:20:30.804883+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:10:41.246540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":15,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.782348+0000","last_change":"2023-10-19T14:20:18.722143+0000","last_active":"2023-10-19T14:20:29.782348+0000","last_peered":"2023-10-19T14:20:29.782348+0000","last_clean":"2023-10-19T14:20:29.782348+0000","last_became_active":"2023-10-19T14:20:18.721853+0000","last_became_peered":"2023-10-19T14:20:18.721853+0000","last_unstale":"2023-10-19T14:20:29.782348+0000","last_undegraded":"2023-10-19T14:20:29.782348+0000","last_fullsized":"2023-10-19T14:20:29.782348+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:01:31.494525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.7","version":"0'0","reported_seq":14,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.788060+0000","last_change":"2023-10-19T14:20:19.347562+0000","last_active":"2023-10-19T14:20:29.788060+0000","last_peered":"2023-10-19T14:20:29.788060+0000","last_clean":"2023-10-19T14:20:29.788060+0000","last_became_active":"2023-10-19T14:20:19.347286+0000","last_became_peered":"2023-10-19T14:20:19.347286+0000","last_unstale":"2023-10-19T14:20:29.788060+0000","last_undegraded":"2023-10-19T14:20:29.788060+0000","last_fullsized":"2023-10-19T14:20:29.788060+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:06:12.825567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.6","version":"0'0","reported_seq":11,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.788851+0000","last_change":"2023-10-19T14:20:29.796583+0000","last_active":"2023-10-19T14:20:30.788851+0000","last_peered":"2023-10-19T14:20:30.788851+0000","last_clean":"2023-10-19T14:20:30.788851+0000","last_became_active":"2023-10-19T14:20:29.796107+0000","last_became_peered":"2023-10-19T14:20:29.796107+0000","last_unstale":"2023-10-19T14:20:30.788851+0000","last_undegraded":"2023-10-19T14:20:30.788851+0000","last_fullsized":"2023-10-19T14:20:30.788851+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_clean_scrub_stamp":"2023-10-19T14:20:28.763908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:26:10.775357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.4","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.785627+0000","last_change":"2023-10-19T14:20:30.794356+0000","last_active":"2023-10-19T14:20:31.785627+0000","last_peered":"2023-10-19T14:20:31.785627+0000","last_clean":"2023-10-19T14:20:31.785627+0000","last_became_active":"2023-10-19T14:20:30.794101+0000","last_became_peered":"2023-10-19T14:20:30.794101+0000","last_unstale":"2023-10-19T14:20:31.785627+0000","last_undegraded":"2023-10-19T14:20:31.785627+0000","last_fullsized":"2023-10-19T14:20:31.785627+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:13:36.493553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":16,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.782732+0000","last_change":"2023-10-19T14:20:18.723906+0000","last_active":"2023-10-19T14:20:30.782732+0000","last_peered":"2023-10-19T14:20:30.782732+0000","last_clean":"2023-10-19T14:20:30.782732+0000","last_became_active":"2023-10-19T14:20:18.723585+0000","last_became_peered":"2023-10-19T14:20:18.723585+0000","last_unstale":"2023-10-19T14:20:30.782732+0000","last_undegraded":"2023-10-19T14:20:30.782732+0000","last_fullsized":"2023-10-19T14:20:30.782732+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:28:48.261352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":14,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.787461+0000","last_change":"2023-10-19T14:20:19.345421+0000","last_active":"2023-10-19T14:20:29.787461+0000","last_peered":"2023-10-19T14:20:29.787461+0000","last_clean":"2023-10-19T14:20:29.787461+0000","last_became_active":"2023-10-19T14:20:19.345091+0000","last_became_peered":"2023-10-19T14:20:19.345091+0000","last_unstale":"2023-10-19T14:20:29.787461+0000","last_undegraded":"2023-10-19T14:20:29.787461+0000","last_fullsized":"2023-10-19T14:20:29.787461+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:49:04.080016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.7","version":"226'2","reported_seq":14,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.814420+0000","last_change":"2023-10-19T14:20:29.800281+0000","last_active":"2023-10-19T14:20:31.814420+0000","last_peered":"2023-10-19T14:20:31.814420+0000","last_clean":"2023-10-19T14:20:31.814420+0000","last_became_active":"2023-10-19T14:20:29.799527+0000","last_became_peered":"2023-10-19T14:20:29.799527+0000","last_unstale":"2023-10-19T14:20:31.814420+0000","last_undegraded":"2023-10-19T14:20:31.814420+0000","last_fullsized":"2023-10-19T14:20:31.814420+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_clean_scrub_stamp":"2023-10-19T14:20:28.763908+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:32:29.418946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.7","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.794051+0000","last_change":"2023-10-19T14:20:30.792541+0000","last_active":"2023-10-19T14:20:30.794051+0000","last_peered":"2023-10-19T14:20:30.794051+0000","last_clean":"2023-10-19T14:20:30.794051+0000","last_became_active":"2023-10-19T14:20:30.792275+0000","last_became_peered":"2023-10-19T14:20:30.792275+0000","last_unstale":"2023-10-19T14:20:30.794051+0000","last_undegraded":"2023-10-19T14:20:30.794051+0000","last_fullsized":"2023-10-19T14:20:30.794051+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:34:03.301288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":16,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.786736+0000","last_change":"2023-10-19T14:20:18.722453+0000","last_active":"2023-10-19T14:20:30.786736+0000","last_peered":"2023-10-19T14:20:30.786736+0000","last_clean":"2023-10-19T14:20:30.786736+0000","last_became_active":"2023-10-19T14:20:18.722162+0000","last_became_peered":"2023-10-19T14:20:18.722162+0000","last_unstale":"2023-10-19T14:20:30.786736+0000","last_undegraded":"2023-10-19T14:20:30.786736+0000","last_fullsized":"2023-10-19T14:20:30.786736+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:49:03.077978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":14,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.787669+0000","last_change":"2023-10-19T14:20:19.344708+0000","last_active":"2023-10-19T14:20:29.787669+0000","last_peered":"2023-10-19T14:20:29.787669+0000","last_clean":"2023-10-19T14:20:29.787669+0000","last_became_active":"2023-10-19T14:20:19.344380+0000","last_became_peered":"2023-10-19T14:20:19.344380+0000","last_unstale":"2023-10-19T14:20:29.787669+0000","last_undegraded":"2023-10-19T14:20:29.787669+0000","last_fullsized":"2023-10-19T14:20:29.787669+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:37:55.455379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.4","version":"0'0","reported_seq":12,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.814069+0000","last_change":"2023-10-19T14:20:29.798930+0000","last_active":"2023-10-19T14:20:31.814069+0000","last_peered":"2023-10-19T14:20:31.814069+0000","last_clean":"2023-10-19T14:20:31.814069+0000","last_became_active":"2023-10-19T14:20:29.798658+0000","last_became_peered":"2023-10-19T14:20:29.798658+0000","last_unstale":"2023-10-19T14:20:31.814069+0000","last_undegraded":"2023-10-19T14:20:31.814069+0000","last_fullsized":"2023-10-19T14:20:31.814069+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_clean_scrub_stamp":"2023-10-19T14:20:28.763908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:36:37.487929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.6","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.782950+0000","last_change":"2023-10-19T14:20:30.801925+0000","last_active":"2023-10-19T14:20:31.782950+0000","last_peered":"2023-10-19T14:20:31.782950+0000","last_clean":"2023-10-19T14:20:31.782950+0000","last_became_active":"2023-10-19T14:20:30.801564+0000","last_became_peered":"2023-10-19T14:20:30.801564+0000","last_unstale":"2023-10-19T14:20:31.782950+0000","last_undegraded":"2023-10-19T14:20:31.782950+0000","last_fullsized":"2023-10-19T14:20:31.782950+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:00:34.306103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":16,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.800098+0000","last_change":"2023-10-19T14:20:18.723898+0000","last_active":"2023-10-19T14:20:30.800098+0000","last_peered":"2023-10-19T14:20:30.800098+0000","last_clean":"2023-10-19T14:20:30.800098+0000","last_became_active":"2023-10-19T14:20:18.722703+0000","last_became_peered":"2023-10-19T14:20:18.722703+0000","last_unstale":"2023-10-19T14:20:30.800098+0000","last_undegraded":"2023-10-19T14:20:30.800098+0000","last_fullsized":"2023-10-19T14:20:30.800098+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:25:51.334964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.797684+0000","last_change":"2023-10-19T14:20:19.345692+0000","last_active":"2023-10-19T14:20:30.797684+0000","last_peered":"2023-10-19T14:20:30.797684+0000","last_clean":"2023-10-19T14:20:30.797684+0000","last_became_active":"2023-10-19T14:20:19.344861+0000","last_became_peered":"2023-10-19T14:20:19.344861+0000","last_unstale":"2023-10-19T14:20:30.797684+0000","last_undegraded":"2023-10-19T14:20:30.797684+0000","last_fullsized":"2023-10-19T14:20:30.797684+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:43:57.179927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.5","version":"0'0","reported_seq":11,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.791341+0000","last_change":"2023-10-19T14:20:29.793288+0000","last_active":"2023-10-19T14:20:30.791341+0000","last_peered":"2023-10-19T14:20:30.791341+0000","last_clean":"2023-10-19T14:20:30.791341+0000","last_became_active":"2023-10-19T14:20:29.792978+0000","last_became_peered":"2023-10-19T14:20:29.792978+0000","last_unstale":"2023-10-19T14:20:30.791341+0000","last_undegraded":"2023-10-19T14:20:30.791341+0000","last_fullsized":"2023-10-19T14:20:30.791341+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_clean_scrub_stamp":"2023-10-19T14:20:28.763908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:21:28.975578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.1","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.784439+0000","last_change":"2023-10-19T14:20:30.799002+0000","last_active":"2023-10-19T14:20:31.784439+0000","last_peered":"2023-10-19T14:20:31.784439+0000","last_clean":"2023-10-19T14:20:31.784439+0000","last_became_active":"2023-10-19T14:20:30.798128+0000","last_became_peered":"2023-10-19T14:20:30.798128+0000","last_unstale":"2023-10-19T14:20:31.784439+0000","last_undegraded":"2023-10-19T14:20:31.784439+0000","last_fullsized":"2023-10-19T14:20:31.784439+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:18:44.195844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":16,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.782167+0000","last_change":"2023-10-19T14:20:18.718760+0000","last_active":"2023-10-19T14:20:30.782167+0000","last_peered":"2023-10-19T14:20:30.782167+0000","last_clean":"2023-10-19T14:20:30.782167+0000","last_became_active":"2023-10-19T14:20:18.718459+0000","last_became_peered":"2023-10-19T14:20:18.718459+0000","last_unstale":"2023-10-19T14:20:30.782167+0000","last_undegraded":"2023-10-19T14:20:30.782167+0000","last_fullsized":"2023-10-19T14:20:30.782167+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:32:18.381107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.782276+0000","last_change":"2023-10-19T14:20:19.352214+0000","last_active":"2023-10-19T14:20:30.782276+0000","last_peered":"2023-10-19T14:20:30.782276+0000","last_clean":"2023-10-19T14:20:30.782276+0000","last_became_active":"2023-10-19T14:20:19.351789+0000","last_became_peered":"2023-10-19T14:20:19.351789+0000","last_unstale":"2023-10-19T14:20:30.782276+0000","last_undegraded":"2023-10-19T14:20:30.782276+0000","last_fullsized":"2023-10-19T14:20:30.782276+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:05:58.978059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.2","version":"0'0","reported_seq":11,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.786923+0000","last_change":"2023-10-19T14:20:29.792399+0000","last_active":"2023-10-19T14:20:30.786923+0000","last_peered":"2023-10-19T14:20:30.786923+0000","last_clean":"2023-10-19T14:20:30.786923+0000","last_became_active":"2023-10-19T14:20:29.792120+0000","last_became_peered":"2023-10-19T14:20:29.792120+0000","last_unstale":"2023-10-19T14:20:30.786923+0000","last_undegraded":"2023-10-19T14:20:30.786923+0000","last_fullsized":"2023-10-19T14:20:30.786923+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_clean_scrub_stamp":"2023-10-19T14:20:28.763908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:26:03.175641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.0","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.784658+0000","last_change":"2023-10-19T14:20:30.803671+0000","last_active":"2023-10-19T14:20:31.784658+0000","last_peered":"2023-10-19T14:20:31.784658+0000","last_clean":"2023-10-19T14:20:31.784658+0000","last_became_active":"2023-10-19T14:20:30.803352+0000","last_became_peered":"2023-10-19T14:20:30.803352+0000","last_unstale":"2023-10-19T14:20:31.784658+0000","last_undegraded":"2023-10-19T14:20:31.784658+0000","last_fullsized":"2023-10-19T14:20:31.784658+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:59:44.340796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":15,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.787522+0000","last_change":"2023-10-19T14:20:18.724127+0000","last_active":"2023-10-19T14:20:29.787522+0000","last_peered":"2023-10-19T14:20:29.787522+0000","last_clean":"2023-10-19T14:20:29.787522+0000","last_became_active":"2023-10-19T14:20:18.723847+0000","last_became_peered":"2023-10-19T14:20:18.723847+0000","last_unstale":"2023-10-19T14:20:29.787522+0000","last_undegraded":"2023-10-19T14:20:29.787522+0000","last_fullsized":"2023-10-19T14:20:29.787522+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:36:35.167691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.786444+0000","last_change":"2023-10-19T14:20:19.340725+0000","last_active":"2023-10-19T14:20:30.786444+0000","last_peered":"2023-10-19T14:20:30.786444+0000","last_clean":"2023-10-19T14:20:30.786444+0000","last_became_active":"2023-10-19T14:20:19.340284+0000","last_became_peered":"2023-10-19T14:20:19.340284+0000","last_unstale":"2023-10-19T14:20:30.786444+0000","last_undegraded":"2023-10-19T14:20:30.786444+0000","last_fullsized":"2023-10-19T14:20:30.786444+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:16:20.216590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.3","version":"0'0","reported_seq":12,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.813807+0000","last_change":"2023-10-19T14:20:29.802371+0000","last_active":"2023-10-19T14:20:31.813807+0000","last_peered":"2023-10-19T14:20:31.813807+0000","last_clean":"2023-10-19T14:20:31.813807+0000","last_became_active":"2023-10-19T14:20:29.800479+0000","last_became_peered":"2023-10-19T14:20:29.800479+0000","last_unstale":"2023-10-19T14:20:31.813807+0000","last_undegraded":"2023-10-19T14:20:31.813807+0000","last_fullsized":"2023-10-19T14:20:31.813807+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_clean_scrub_stamp":"2023-10-19T14:20:28.763908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:24:02.062518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.814024+0000","last_change":"2023-10-19T14:20:30.801362+0000","last_active":"2023-10-19T14:20:31.814024+0000","last_peered":"2023-10-19T14:20:31.814024+0000","last_clean":"2023-10-19T14:20:31.814024+0000","last_became_active":"2023-10-19T14:20:30.801064+0000","last_became_peered":"2023-10-19T14:20:30.801064+0000","last_unstale":"2023-10-19T14:20:31.814024+0000","last_undegraded":"2023-10-19T14:20:31.814024+0000","last_fullsized":"2023-10-19T14:20:31.814024+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:28:21.802555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.6","version":"223'1","reported_seq":17,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.797899+0000","last_change":"2023-10-19T14:20:18.722733+0000","last_active":"2023-10-19T14:20:30.797899+0000","last_peered":"2023-10-19T14:20:30.797899+0000","last_clean":"2023-10-19T14:20:30.797899+0000","last_became_active":"2023-10-19T14:20:18.722447+0000","last_became_peered":"2023-10-19T14:20:18.722447+0000","last_unstale":"2023-10-19T14:20:30.797899+0000","last_undegraded":"2023-10-19T14:20:30.797899+0000","last_fullsized":"2023-10-19T14:20:30.797899+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:28:23.768143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1","version":"0'0","reported_seq":14,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.784233+0000","last_change":"2023-10-19T14:20:19.344287+0000","last_active":"2023-10-19T14:20:29.784233+0000","last_peered":"2023-10-19T14:20:29.784233+0000","last_clean":"2023-10-19T14:20:29.784233+0000","last_became_active":"2023-10-19T14:20:19.344030+0000","last_became_peered":"2023-10-19T14:20:19.344030+0000","last_unstale":"2023-10-19T14:20:29.784233+0000","last_undegraded":"2023-10-19T14:20:29.784233+0000","last_fullsized":"2023-10-19T14:20:29.784233+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:35:58.143784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.0","version":"0'0","reported_seq":12,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.813395+0000","last_change":"2023-10-19T14:20:29.797496+0000","last_active":"2023-10-19T14:20:31.813395+0000","last_peered":"2023-10-19T14:20:31.813395+0000","last_clean":"2023-10-19T14:20:31.813395+0000","last_became_active":"2023-10-19T14:20:29.797194+0000","last_became_peered":"2023-10-19T14:20:29.797194+0000","last_unstale":"2023-10-19T14:20:31.813395+0000","last_undegraded":"2023-10-19T14:20:31.813395+0000","last_fullsized":"2023-10-19T14:20:31.813395+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_clean_scrub_stamp":"2023-10-19T14:20:28.763908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:42:03.650411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.794174+0000","last_change":"2023-10-19T14:20:30.788501+0000","last_active":"2023-10-19T14:20:30.794174+0000","last_peered":"2023-10-19T14:20:30.794174+0000","last_clean":"2023-10-19T14:20:30.794174+0000","last_became_active":"2023-10-19T14:20:30.788225+0000","last_became_peered":"2023-10-19T14:20:30.788225+0000","last_unstale":"2023-10-19T14:20:30.794174+0000","last_undegraded":"2023-10-19T14:20:30.794174+0000","last_fullsized":"2023-10-19T14:20:30.794174+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T02:03:58.474932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.7","version":"222'2","reported_seq":18,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.788923+0000","last_change":"2023-10-19T14:20:18.724203+0000","last_active":"2023-10-19T14:20:30.788923+0000","last_peered":"2023-10-19T14:20:30.788923+0000","last_clean":"2023-10-19T14:20:30.788923+0000","last_became_active":"2023-10-19T14:20:18.723883+0000","last_became_peered":"2023-10-19T14:20:18.723883+0000","last_unstale":"2023-10-19T14:20:30.788923+0000","last_undegraded":"2023-10-19T14:20:30.788923+0000","last_fullsized":"2023-10-19T14:20:30.788923+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:38:31.255506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.800298+0000","last_change":"2023-10-19T14:20:19.349129+0000","last_active":"2023-10-19T14:20:30.800298+0000","last_peered":"2023-10-19T14:20:30.800298+0000","last_clean":"2023-10-19T14:20:30.800298+0000","last_became_active":"2023-10-19T14:20:19.348846+0000","last_became_peered":"2023-10-19T14:20:19.348846+0000","last_unstale":"2023-10-19T14:20:30.800298+0000","last_undegraded":"2023-10-19T14:20:30.800298+0000","last_fullsized":"2023-10-19T14:20:30.800298+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:04:34.009285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1","version":"0'0","reported_seq":11,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.787548+0000","last_change":"2023-10-19T14:20:29.796999+0000","last_active":"2023-10-19T14:20:30.787548+0000","last_peered":"2023-10-19T14:20:30.787548+0000","last_clean":"2023-10-19T14:20:30.787548+0000","last_became_active":"2023-10-19T14:20:29.796769+0000","last_became_peered":"2023-10-19T14:20:29.796769+0000","last_unstale":"2023-10-19T14:20:30.787548+0000","last_undegraded":"2023-10-19T14:20:30.787548+0000","last_fullsized":"2023-10-19T14:20:30.787548+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_clean_scrub_stamp":"2023-10-19T14:20:28.763908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:23:35.556147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.d","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.804445+0000","last_change":"2023-10-19T14:20:30.797600+0000","last_active":"2023-10-19T14:20:30.804445+0000","last_peered":"2023-10-19T14:20:30.804445+0000","last_clean":"2023-10-19T14:20:30.804445+0000","last_became_active":"2023-10-19T14:20:30.797358+0000","last_became_peered":"2023-10-19T14:20:30.797358+0000","last_unstale":"2023-10-19T14:20:30.804445+0000","last_undegraded":"2023-10-19T14:20:30.804445+0000","last_fullsized":"2023-10-19T14:20:30.804445+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:32:10.739567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":16,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.782953+0000","last_change":"2023-10-19T14:20:18.715329+0000","last_active":"2023-10-19T14:20:30.782953+0000","last_peered":"2023-10-19T14:20:30.782953+0000","last_clean":"2023-10-19T14:20:30.782953+0000","last_became_active":"2023-10-19T14:20:18.715068+0000","last_became_peered":"2023-10-19T14:20:18.715068+0000","last_unstale":"2023-10-19T14:20:30.782953+0000","last_undegraded":"2023-10-19T14:20:30.782953+0000","last_fullsized":"2023-10-19T14:20:30.782953+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:35:10.527276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.f","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.797934+0000","last_change":"2023-10-19T14:20:19.346087+0000","last_active":"2023-10-19T14:20:30.797934+0000","last_peered":"2023-10-19T14:20:30.797934+0000","last_clean":"2023-10-19T14:20:30.797934+0000","last_became_active":"2023-10-19T14:20:19.345704+0000","last_became_peered":"2023-10-19T14:20:19.345704+0000","last_unstale":"2023-10-19T14:20:30.797934+0000","last_undegraded":"2023-10-19T14:20:30.797934+0000","last_fullsized":"2023-10-19T14:20:30.797934+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:53:30.184157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.e","version":"226'2","reported_seq":13,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.810714+0000","last_change":"2023-10-19T14:20:29.792465+0000","last_active":"2023-10-19T14:20:30.810714+0000","last_peered":"2023-10-19T14:20:30.810714+0000","last_clean":"2023-10-19T14:20:30.810714+0000","last_became_active":"2023-10-19T14:20:29.792174+0000","last_became_peered":"2023-10-19T14:20:29.792174+0000","last_unstale":"2023-10-19T14:20:30.810714+0000","last_undegraded":"2023-10-19T14:20:30.810714+0000","last_fullsized":"2023-10-19T14:20:30.810714+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_clean_scrub_stamp":"2023-10-19T14:20:28.763908+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:36:24.939997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.c","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.785757+0000","last_change":"2023-10-19T14:20:30.802369+0000","last_active":"2023-10-19T14:20:31.785757+0000","last_peered":"2023-10-19T14:20:31.785757+0000","last_clean":"2023-10-19T14:20:31.785757+0000","last_became_active":"2023-10-19T14:20:30.802102+0000","last_became_peered":"2023-10-19T14:20:30.802102+0000","last_unstale":"2023-10-19T14:20:31.785757+0000","last_undegraded":"2023-10-19T14:20:31.785757+0000","last_fullsized":"2023-10-19T14:20:31.785757+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:02:08.150394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":15,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.785119+0000","last_change":"2023-10-19T14:20:18.727394+0000","last_active":"2023-10-19T14:20:29.785119+0000","last_peered":"2023-10-19T14:20:29.785119+0000","last_clean":"2023-10-19T14:20:29.785119+0000","last_became_active":"2023-10-19T14:20:18.727050+0000","last_became_peered":"2023-10-19T14:20:18.727050+0000","last_unstale":"2023-10-19T14:20:29.785119+0000","last_undegraded":"2023-10-19T14:20:29.785119+0000","last_fullsized":"2023-10-19T14:20:29.785119+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:13:48.534905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.e","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.787156+0000","last_change":"2023-10-19T14:20:19.345329+0000","last_active":"2023-10-19T14:20:30.787156+0000","last_peered":"2023-10-19T14:20:30.787156+0000","last_clean":"2023-10-19T14:20:30.787156+0000","last_became_active":"2023-10-19T14:20:19.344983+0000","last_became_peered":"2023-10-19T14:20:19.344983+0000","last_unstale":"2023-10-19T14:20:30.787156+0000","last_undegraded":"2023-10-19T14:20:30.787156+0000","last_fullsized":"2023-10-19T14:20:30.787156+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:17:53.794653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.f","version":"226'1","reported_seq":13,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.785140+0000","last_change":"2023-10-19T14:20:29.795271+0000","last_active":"2023-10-19T14:20:31.785140+0000","last_peered":"2023-10-19T14:20:31.785140+0000","last_clean":"2023-10-19T14:20:31.785140+0000","last_became_active":"2023-10-19T14:20:29.794970+0000","last_became_peered":"2023-10-19T14:20:29.794970+0000","last_unstale":"2023-10-19T14:20:31.785140+0000","last_undegraded":"2023-10-19T14:20:31.785140+0000","last_fullsized":"2023-10-19T14:20:31.785140+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_clean_scrub_stamp":"2023-10-19T14:20:28.763908+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:52:55.038174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.f","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.814575+0000","last_change":"2023-10-19T14:20:30.801356+0000","last_active":"2023-10-19T14:20:31.814575+0000","last_peered":"2023-10-19T14:20:31.814575+0000","last_clean":"2023-10-19T14:20:31.814575+0000","last_became_active":"2023-10-19T14:20:30.801095+0000","last_became_peered":"2023-10-19T14:20:30.801095+0000","last_unstale":"2023-10-19T14:20:31.814575+0000","last_undegraded":"2023-10-19T14:20:31.814575+0000","last_fullsized":"2023-10-19T14:20:31.814575+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:08:12.090774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":16,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.783128+0000","last_change":"2023-10-19T14:20:18.711157+0000","last_active":"2023-10-19T14:20:30.783128+0000","last_peered":"2023-10-19T14:20:30.783128+0000","last_clean":"2023-10-19T14:20:30.783128+0000","last_became_active":"2023-10-19T14:20:18.710747+0000","last_became_peered":"2023-10-19T14:20:18.710747+0000","last_unstale":"2023-10-19T14:20:30.783128+0000","last_undegraded":"2023-10-19T14:20:30.783128+0000","last_fullsized":"2023-10-19T14:20:30.783128+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:46:05.844692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.d","version":"0'0","reported_seq":14,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.787526+0000","last_change":"2023-10-19T14:20:19.350548+0000","last_active":"2023-10-19T14:20:29.787526+0000","last_peered":"2023-10-19T14:20:29.787526+0000","last_clean":"2023-10-19T14:20:29.787526+0000","last_became_active":"2023-10-19T14:20:19.349339+0000","last_became_peered":"2023-10-19T14:20:19.349339+0000","last_unstale":"2023-10-19T14:20:29.787526+0000","last_undegraded":"2023-10-19T14:20:29.787526+0000","last_fullsized":"2023-10-19T14:20:29.787526+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:13:54.112869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.c","version":"0'0","reported_seq":12,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.813971+0000","last_change":"2023-10-19T14:20:29.798678+0000","last_active":"2023-10-19T14:20:31.813971+0000","last_peered":"2023-10-19T14:20:31.813971+0000","last_clean":"2023-10-19T14:20:31.813971+0000","last_became_active":"2023-10-19T14:20:29.798386+0000","last_became_peered":"2023-10-19T14:20:29.798386+0000","last_unstale":"2023-10-19T14:20:31.813971+0000","last_undegraded":"2023-10-19T14:20:31.813971+0000","last_fullsized":"2023-10-19T14:20:31.813971+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_clean_scrub_stamp":"2023-10-19T14:20:28.763908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:08:50.934693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.e","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.785275+0000","last_change":"2023-10-19T14:20:30.800273+0000","last_active":"2023-10-19T14:20:31.785275+0000","last_peered":"2023-10-19T14:20:31.785275+0000","last_clean":"2023-10-19T14:20:31.785275+0000","last_became_active":"2023-10-19T14:20:30.799631+0000","last_became_peered":"2023-10-19T14:20:30.799631+0000","last_unstale":"2023-10-19T14:20:31.785275+0000","last_undegraded":"2023-10-19T14:20:31.785275+0000","last_fullsized":"2023-10-19T14:20:31.785275+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:30:11.739707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.b","version":"222'2","reported_seq":17,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.782562+0000","last_change":"2023-10-19T14:20:18.719825+0000","last_active":"2023-10-19T14:20:29.782562+0000","last_peered":"2023-10-19T14:20:29.782562+0000","last_clean":"2023-10-19T14:20:29.782562+0000","last_became_active":"2023-10-19T14:20:18.719549+0000","last_became_peered":"2023-10-19T14:20:18.719549+0000","last_unstale":"2023-10-19T14:20:29.782562+0000","last_undegraded":"2023-10-19T14:20:29.782562+0000","last_fullsized":"2023-10-19T14:20:29.782562+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T02:18:13.024084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.783414+0000","last_change":"2023-10-19T14:20:19.346026+0000","last_active":"2023-10-19T14:20:30.783414+0000","last_peered":"2023-10-19T14:20:30.783414+0000","last_clean":"2023-10-19T14:20:30.783414+0000","last_became_active":"2023-10-19T14:20:19.345762+0000","last_became_peered":"2023-10-19T14:20:19.345762+0000","last_unstale":"2023-10-19T14:20:30.783414+0000","last_undegraded":"2023-10-19T14:20:30.783414+0000","last_fullsized":"2023-10-19T14:20:30.783414+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:28:21.209211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.d","version":"226'1","reported_seq":13,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.814086+0000","last_change":"2023-10-19T14:20:29.800987+0000","last_active":"2023-10-19T14:20:31.814086+0000","last_peered":"2023-10-19T14:20:31.814086+0000","last_clean":"2023-10-19T14:20:31.814086+0000","last_became_active":"2023-10-19T14:20:29.800691+0000","last_became_peered":"2023-10-19T14:20:29.800691+0000","last_unstale":"2023-10-19T14:20:31.814086+0000","last_undegraded":"2023-10-19T14:20:31.814086+0000","last_fullsized":"2023-10-19T14:20:31.814086+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_clean_scrub_stamp":"2023-10-19T14:20:28.763908+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:50:50.574035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.9","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.804333+0000","last_change":"2023-10-19T14:20:30.796772+0000","last_active":"2023-10-19T14:20:30.804333+0000","last_peered":"2023-10-19T14:20:30.804333+0000","last_clean":"2023-10-19T14:20:30.804333+0000","last_became_active":"2023-10-19T14:20:30.796517+0000","last_became_peered":"2023-10-19T14:20:30.796517+0000","last_unstale":"2023-10-19T14:20:30.804333+0000","last_undegraded":"2023-10-19T14:20:30.804333+0000","last_fullsized":"2023-10-19T14:20:30.804333+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:02:53.373912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":15,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.781908+0000","last_change":"2023-10-19T14:20:18.717037+0000","last_active":"2023-10-19T14:20:29.781908+0000","last_peered":"2023-10-19T14:20:29.781908+0000","last_clean":"2023-10-19T14:20:29.781908+0000","last_became_active":"2023-10-19T14:20:18.716731+0000","last_became_peered":"2023-10-19T14:20:18.716731+0000","last_unstale":"2023-10-19T14:20:29.781908+0000","last_undegraded":"2023-10-19T14:20:29.781908+0000","last_fullsized":"2023-10-19T14:20:29.781908+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:50:59.017027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.b","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.800006+0000","last_change":"2023-10-19T14:20:19.352403+0000","last_active":"2023-10-19T14:20:30.800006+0000","last_peered":"2023-10-19T14:20:30.800006+0000","last_clean":"2023-10-19T14:20:30.800006+0000","last_became_active":"2023-10-19T14:20:19.351892+0000","last_became_peered":"2023-10-19T14:20:19.351892+0000","last_unstale":"2023-10-19T14:20:30.800006+0000","last_undegraded":"2023-10-19T14:20:30.800006+0000","last_fullsized":"2023-10-19T14:20:30.800006+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:33:52.226441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.a","version":"0'0","reported_seq":11,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.787934+0000","last_change":"2023-10-19T14:20:29.788981+0000","last_active":"2023-10-19T14:20:30.787934+0000","last_peered":"2023-10-19T14:20:30.787934+0000","last_clean":"2023-10-19T14:20:30.787934+0000","last_became_active":"2023-10-19T14:20:29.788699+0000","last_became_peered":"2023-10-19T14:20:29.788699+0000","last_unstale":"2023-10-19T14:20:30.787934+0000","last_undegraded":"2023-10-19T14:20:30.787934+0000","last_fullsized":"2023-10-19T14:20:30.787934+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_clean_scrub_stamp":"2023-10-19T14:20:28.763908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:49:14.148270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.8","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.813308+0000","last_change":"2023-10-19T14:20:30.800475+0000","last_active":"2023-10-19T14:20:31.813308+0000","last_peered":"2023-10-19T14:20:31.813308+0000","last_clean":"2023-10-19T14:20:31.813308+0000","last_became_active":"2023-10-19T14:20:30.797151+0000","last_became_peered":"2023-10-19T14:20:30.797151+0000","last_unstale":"2023-10-19T14:20:31.813308+0000","last_undegraded":"2023-10-19T14:20:31.813308+0000","last_fullsized":"2023-10-19T14:20:31.813308+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:41:36.778953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.d","version":"222'1","reported_seq":17,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.788115+0000","last_change":"2023-10-19T14:20:18.719857+0000","last_active":"2023-10-19T14:20:30.788115+0000","last_peered":"2023-10-19T14:20:30.788115+0000","last_clean":"2023-10-19T14:20:30.788115+0000","last_became_active":"2023-10-19T14:20:18.719586+0000","last_became_peered":"2023-10-19T14:20:18.719586+0000","last_unstale":"2023-10-19T14:20:30.788115+0000","last_undegraded":"2023-10-19T14:20:30.788115+0000","last_fullsized":"2023-10-19T14:20:30.788115+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:12:37.668857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.782665+0000","last_change":"2023-10-19T14:20:19.332710+0000","last_active":"2023-10-19T14:20:30.782665+0000","last_peered":"2023-10-19T14:20:30.782665+0000","last_clean":"2023-10-19T14:20:30.782665+0000","last_became_active":"2023-10-19T14:20:19.332132+0000","last_became_peered":"2023-10-19T14:20:19.332132+0000","last_unstale":"2023-10-19T14:20:30.782665+0000","last_undegraded":"2023-10-19T14:20:30.782665+0000","last_fullsized":"2023-10-19T14:20:30.782665+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:36:15.681007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.b","version":"226'2","reported_seq":14,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.813825+0000","last_change":"2023-10-19T14:20:29.801095+0000","last_active":"2023-10-19T14:20:31.813825+0000","last_peered":"2023-10-19T14:20:31.813825+0000","last_clean":"2023-10-19T14:20:31.813825+0000","last_became_active":"2023-10-19T14:20:29.799487+0000","last_became_peered":"2023-10-19T14:20:29.799487+0000","last_unstale":"2023-10-19T14:20:31.813825+0000","last_undegraded":"2023-10-19T14:20:31.813825+0000","last_fullsized":"2023-10-19T14:20:31.813825+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_clean_scrub_stamp":"2023-10-19T14:20:28.763908+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:48:54.823238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.b","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.784907+0000","last_change":"2023-10-19T14:20:30.804612+0000","last_active":"2023-10-19T14:20:31.784907+0000","last_peered":"2023-10-19T14:20:31.784907+0000","last_clean":"2023-10-19T14:20:31.784907+0000","last_became_active":"2023-10-19T14:20:30.804342+0000","last_became_peered":"2023-10-19T14:20:30.804342+0000","last_unstale":"2023-10-19T14:20:31.784907+0000","last_undegraded":"2023-10-19T14:20:31.784907+0000","last_fullsized":"2023-10-19T14:20:31.784907+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:36:59.617065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.e","version":"222'2","reported_seq":18,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.781973+0000","last_change":"2023-10-19T14:20:18.721114+0000","last_active":"2023-10-19T14:20:30.781973+0000","last_peered":"2023-10-19T14:20:30.781973+0000","last_clean":"2023-10-19T14:20:30.781973+0000","last_became_active":"2023-10-19T14:20:18.720840+0000","last_became_peered":"2023-10-19T14:20:18.720840+0000","last_unstale":"2023-10-19T14:20:30.781973+0000","last_undegraded":"2023-10-19T14:20:30.781973+0000","last_fullsized":"2023-10-19T14:20:30.781973+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:44:50.967568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.782688+0000","last_change":"2023-10-19T14:20:19.348569+0000","last_active":"2023-10-19T14:20:30.782688+0000","last_peered":"2023-10-19T14:20:30.782688+0000","last_clean":"2023-10-19T14:20:30.782688+0000","last_became_active":"2023-10-19T14:20:19.348268+0000","last_became_peered":"2023-10-19T14:20:19.348268+0000","last_unstale":"2023-10-19T14:20:30.782688+0000","last_undegraded":"2023-10-19T14:20:30.782688+0000","last_fullsized":"2023-10-19T14:20:30.782688+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:54:41.219828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.8","version":"0'0","reported_seq":11,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.780209+0000","last_change":"2023-10-19T14:20:29.792764+0000","last_active":"2023-10-19T14:20:30.780209+0000","last_peered":"2023-10-19T14:20:30.780209+0000","last_clean":"2023-10-19T14:20:30.780209+0000","last_became_active":"2023-10-19T14:20:29.792199+0000","last_became_peered":"2023-10-19T14:20:29.792199+0000","last_unstale":"2023-10-19T14:20:30.780209+0000","last_undegraded":"2023-10-19T14:20:30.780209+0000","last_fullsized":"2023-10-19T14:20:30.780209+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_clean_scrub_stamp":"2023-10-19T14:20:28.763908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:13:59.605197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.a","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.785546+0000","last_change":"2023-10-19T14:20:30.803004+0000","last_active":"2023-10-19T14:20:31.785546+0000","last_peered":"2023-10-19T14:20:31.785546+0000","last_clean":"2023-10-19T14:20:31.785546+0000","last_became_active":"2023-10-19T14:20:30.802745+0000","last_became_peered":"2023-10-19T14:20:30.802745+0000","last_unstale":"2023-10-19T14:20:31.785546+0000","last_undegraded":"2023-10-19T14:20:31.785546+0000","last_fullsized":"2023-10-19T14:20:31.785546+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:54:08.144842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.f","version":"222'1","reported_seq":17,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.781971+0000","last_change":"2023-10-19T14:20:18.719346+0000","last_active":"2023-10-19T14:20:30.781971+0000","last_peered":"2023-10-19T14:20:30.781971+0000","last_clean":"2023-10-19T14:20:30.781971+0000","last_became_active":"2023-10-19T14:20:18.719101+0000","last_became_peered":"2023-10-19T14:20:18.719101+0000","last_unstale":"2023-10-19T14:20:30.781971+0000","last_undegraded":"2023-10-19T14:20:30.781971+0000","last_fullsized":"2023-10-19T14:20:30.781971+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:01:11.976346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.800218+0000","last_change":"2023-10-19T14:20:19.349188+0000","last_active":"2023-10-19T14:20:30.800218+0000","last_peered":"2023-10-19T14:20:30.800218+0000","last_clean":"2023-10-19T14:20:30.800218+0000","last_became_active":"2023-10-19T14:20:19.348905+0000","last_became_peered":"2023-10-19T14:20:19.348905+0000","last_unstale":"2023-10-19T14:20:30.800218+0000","last_undegraded":"2023-10-19T14:20:30.800218+0000","last_fullsized":"2023-10-19T14:20:30.800218+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:36:36.169729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.9","version":"0'0","reported_seq":11,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.788328+0000","last_change":"2023-10-19T14:20:29.797428+0000","last_active":"2023-10-19T14:20:30.788328+0000","last_peered":"2023-10-19T14:20:30.788328+0000","last_clean":"2023-10-19T14:20:30.788328+0000","last_became_active":"2023-10-19T14:20:29.797186+0000","last_became_peered":"2023-10-19T14:20:29.797186+0000","last_unstale":"2023-10-19T14:20:30.788328+0000","last_undegraded":"2023-10-19T14:20:30.788328+0000","last_fullsized":"2023-10-19T14:20:30.788328+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_clean_scrub_stamp":"2023-10-19T14:20:28.763908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:18:59.806364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.15","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.803321+0000","last_change":"2023-10-19T14:20:30.797685+0000","last_active":"2023-10-19T14:20:30.803321+0000","last_peered":"2023-10-19T14:20:30.803321+0000","last_clean":"2023-10-19T14:20:30.803321+0000","last_became_active":"2023-10-19T14:20:30.797418+0000","last_became_peered":"2023-10-19T14:20:30.797418+0000","last_unstale":"2023-10-19T14:20:30.803321+0000","last_undegraded":"2023-10-19T14:20:30.803321+0000","last_fullsized":"2023-10-19T14:20:30.803321+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:53:03.050006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.10","version":"222'8","reported_seq":23,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.782257+0000","last_change":"2023-10-19T14:20:18.722891+0000","last_active":"2023-10-19T14:20:29.782257+0000","last_peered":"2023-10-19T14:20:29.782257+0000","last_clean":"2023-10-19T14:20:29.782257+0000","last_became_active":"2023-10-19T14:20:18.722633+0000","last_became_peered":"2023-10-19T14:20:18.722633+0000","last_unstale":"2023-10-19T14:20:29.782257+0000","last_undegraded":"2023-10-19T14:20:29.782257+0000","last_fullsized":"2023-10-19T14:20:29.782257+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:32:21.255137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":14,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.788128+0000","last_change":"2023-10-19T14:20:19.353248+0000","last_active":"2023-10-19T14:20:29.788128+0000","last_peered":"2023-10-19T14:20:29.788128+0000","last_clean":"2023-10-19T14:20:29.788128+0000","last_became_active":"2023-10-19T14:20:19.352973+0000","last_became_peered":"2023-10-19T14:20:19.352973+0000","last_unstale":"2023-10-19T14:20:29.788128+0000","last_undegraded":"2023-10-19T14:20:29.788128+0000","last_fullsized":"2023-10-19T14:20:29.788128+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:07:18.725863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.16","version":"0'0","reported_seq":11,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.789745+0000","last_change":"2023-10-19T14:20:29.796603+0000","last_active":"2023-10-19T14:20:30.789745+0000","last_peered":"2023-10-19T14:20:30.789745+0000","last_clean":"2023-10-19T14:20:30.789745+0000","last_became_active":"2023-10-19T14:20:29.796056+0000","last_became_peered":"2023-10-19T14:20:29.796056+0000","last_unstale":"2023-10-19T14:20:30.789745+0000","last_undegraded":"2023-10-19T14:20:30.789745+0000","last_fullsized":"2023-10-19T14:20:30.789745+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_clean_scrub_stamp":"2023-10-19T14:20:28.763908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:18:58.754296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.14","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.806260+0000","last_change":"2023-10-19T14:20:30.800049+0000","last_active":"2023-10-19T14:20:30.806260+0000","last_peered":"2023-10-19T14:20:30.806260+0000","last_clean":"2023-10-19T14:20:30.806260+0000","last_became_active":"2023-10-19T14:20:30.799751+0000","last_became_peered":"2023-10-19T14:20:30.799751+0000","last_unstale":"2023-10-19T14:20:30.806260+0000","last_undegraded":"2023-10-19T14:20:30.806260+0000","last_fullsized":"2023-10-19T14:20:30.806260+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T02:16:51.635514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":16,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.786371+0000","last_change":"2023-10-19T14:20:18.717768+0000","last_active":"2023-10-19T14:20:30.786371+0000","last_peered":"2023-10-19T14:20:30.786371+0000","last_clean":"2023-10-19T14:20:30.786371+0000","last_became_active":"2023-10-19T14:20:18.717375+0000","last_became_peered":"2023-10-19T14:20:18.717375+0000","last_unstale":"2023-10-19T14:20:30.786371+0000","last_undegraded":"2023-10-19T14:20:30.786371+0000","last_fullsized":"2023-10-19T14:20:30.786371+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:44:02.586786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.787066+0000","last_change":"2023-10-19T14:20:19.343343+0000","last_active":"2023-10-19T14:20:30.787066+0000","last_peered":"2023-10-19T14:20:30.787066+0000","last_clean":"2023-10-19T14:20:30.787066+0000","last_became_active":"2023-10-19T14:20:19.343059+0000","last_became_peered":"2023-10-19T14:20:19.343059+0000","last_unstale":"2023-10-19T14:20:30.787066+0000","last_undegraded":"2023-10-19T14:20:30.787066+0000","last_fullsized":"2023-10-19T14:20:30.787066+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:31:59.664720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.17","version":"0'0","reported_seq":11,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.786321+0000","last_change":"2023-10-19T14:20:29.799948+0000","last_active":"2023-10-19T14:20:30.786321+0000","last_peered":"2023-10-19T14:20:30.786321+0000","last_clean":"2023-10-19T14:20:30.786321+0000","last_became_active":"2023-10-19T14:20:29.799698+0000","last_became_peered":"2023-10-19T14:20:29.799698+0000","last_unstale":"2023-10-19T14:20:30.786321+0000","last_undegraded":"2023-10-19T14:20:30.786321+0000","last_fullsized":"2023-10-19T14:20:30.786321+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_clean_scrub_stamp":"2023-10-19T14:20:28.763908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:18:03.862907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.17","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.795489+0000","last_change":"2023-10-19T14:20:30.794377+0000","last_active":"2023-10-19T14:20:30.795489+0000","last_peered":"2023-10-19T14:20:30.795489+0000","last_clean":"2023-10-19T14:20:30.795489+0000","last_became_active":"2023-10-19T14:20:30.794122+0000","last_became_peered":"2023-10-19T14:20:30.794122+0000","last_unstale":"2023-10-19T14:20:30.795489+0000","last_undegraded":"2023-10-19T14:20:30.795489+0000","last_fullsized":"2023-10-19T14:20:30.795489+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:23:07.047251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":15,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.787462+0000","last_change":"2023-10-19T14:20:18.709786+0000","last_active":"2023-10-19T14:20:29.787462+0000","last_peered":"2023-10-19T14:20:29.787462+0000","last_clean":"2023-10-19T14:20:29.787462+0000","last_became_active":"2023-10-19T14:20:18.709496+0000","last_became_peered":"2023-10-19T14:20:18.709496+0000","last_unstale":"2023-10-19T14:20:29.787462+0000","last_undegraded":"2023-10-19T14:20:29.787462+0000","last_fullsized":"2023-10-19T14:20:29.787462+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:46:59.026062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.15","version":"0'0","reported_seq":14,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.790018+0000","last_change":"2023-10-19T14:20:19.349089+0000","last_active":"2023-10-19T14:20:29.790018+0000","last_peered":"2023-10-19T14:20:29.790018+0000","last_clean":"2023-10-19T14:20:29.790018+0000","last_became_active":"2023-10-19T14:20:19.348826+0000","last_became_peered":"2023-10-19T14:20:19.348826+0000","last_unstale":"2023-10-19T14:20:29.790018+0000","last_undegraded":"2023-10-19T14:20:29.790018+0000","last_fullsized":"2023-10-19T14:20:29.790018+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:10:55.372033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.14","version":"226'6","reported_seq":18,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.782786+0000","last_change":"2023-10-19T14:20:29.795087+0000","last_active":"2023-10-19T14:20:31.782786+0000","last_peered":"2023-10-19T14:20:31.782786+0000","last_clean":"2023-10-19T14:20:31.782786+0000","last_became_active":"2023-10-19T14:20:29.794764+0000","last_became_peered":"2023-10-19T14:20:29.794764+0000","last_unstale":"2023-10-19T14:20:31.782786+0000","last_undegraded":"2023-10-19T14:20:31.782786+0000","last_fullsized":"2023-10-19T14:20:31.782786+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_clean_scrub_stamp":"2023-10-19T14:20:28.763908+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T02:16:50.628921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.16","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.785386+0000","last_change":"2023-10-19T14:20:30.802097+0000","last_active":"2023-10-19T14:20:31.785386+0000","last_peered":"2023-10-19T14:20:31.785386+0000","last_clean":"2023-10-19T14:20:31.785386+0000","last_became_active":"2023-10-19T14:20:30.801764+0000","last_became_peered":"2023-10-19T14:20:30.801764+0000","last_unstale":"2023-10-19T14:20:31.785386+0000","last_undegraded":"2023-10-19T14:20:31.785386+0000","last_fullsized":"2023-10-19T14:20:31.785386+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:17:27.514421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.13","version":"222'2","reported_seq":18,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.783682+0000","last_change":"2023-10-19T14:20:18.713594+0000","last_active":"2023-10-19T14:20:30.783682+0000","last_peered":"2023-10-19T14:20:30.783682+0000","last_clean":"2023-10-19T14:20:30.783682+0000","last_became_active":"2023-10-19T14:20:18.713263+0000","last_became_peered":"2023-10-19T14:20:18.713263+0000","last_unstale":"2023-10-19T14:20:30.783682+0000","last_undegraded":"2023-10-19T14:20:30.783682+0000","last_fullsized":"2023-10-19T14:20:30.783682+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:32:02.753539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.14","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.783314+0000","last_change":"2023-10-19T14:20:19.340471+0000","last_active":"2023-10-19T14:20:30.783314+0000","last_peered":"2023-10-19T14:20:30.783314+0000","last_clean":"2023-10-19T14:20:30.783314+0000","last_became_active":"2023-10-19T14:20:19.340187+0000","last_became_peered":"2023-10-19T14:20:19.340187+0000","last_unstale":"2023-10-19T14:20:30.783314+0000","last_undegraded":"2023-10-19T14:20:30.783314+0000","last_fullsized":"2023-10-19T14:20:30.783314+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:53:57.462566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.15","version":"226'3","reported_seq":14,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.812302+0000","last_change":"2023-10-19T14:20:29.796471+0000","last_active":"2023-10-19T14:20:30.812302+0000","last_peered":"2023-10-19T14:20:30.812302+0000","last_clean":"2023-10-19T14:20:30.812302+0000","last_became_active":"2023-10-19T14:20:29.796232+0000","last_became_peered":"2023-10-19T14:20:29.796232+0000","last_unstale":"2023-10-19T14:20:30.812302+0000","last_undegraded":"2023-10-19T14:20:30.812302+0000","last_fullsized":"2023-10-19T14:20:30.812302+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_clean_scrub_stamp":"2023-10-19T14:20:28.763908+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:27:15.682334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.11","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.813383+0000","last_change":"2023-10-19T14:20:30.800576+0000","last_active":"2023-10-19T14:20:31.813383+0000","last_peered":"2023-10-19T14:20:31.813383+0000","last_clean":"2023-10-19T14:20:31.813383+0000","last_became_active":"2023-10-19T14:20:30.800156+0000","last_became_peered":"2023-10-19T14:20:30.800156+0000","last_unstale":"2023-10-19T14:20:31.813383+0000","last_undegraded":"2023-10-19T14:20:31.813383+0000","last_fullsized":"2023-10-19T14:20:31.813383+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:02:04.231680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.14","version":"225'8","reported_seq":23,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.967684+0000","last_change":"2023-10-19T14:20:18.727225+0000","last_active":"2023-10-19T14:20:29.967684+0000","last_peered":"2023-10-19T14:20:29.967684+0000","last_clean":"2023-10-19T14:20:29.967684+0000","last_became_active":"2023-10-19T14:20:18.726839+0000","last_became_peered":"2023-10-19T14:20:18.726839+0000","last_unstale":"2023-10-19T14:20:29.967684+0000","last_undegraded":"2023-10-19T14:20:29.967684+0000","last_fullsized":"2023-10-19T14:20:29.967684+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:10:54.369994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2868,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.13","version":"0'0","reported_seq":14,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.782468+0000","last_change":"2023-10-19T14:20:19.350427+0000","last_active":"2023-10-19T14:20:29.782468+0000","last_peered":"2023-10-19T14:20:29.782468+0000","last_clean":"2023-10-19T14:20:29.782468+0000","last_became_active":"2023-10-19T14:20:19.350112+0000","last_became_peered":"2023-10-19T14:20:19.350112+0000","last_unstale":"2023-10-19T14:20:29.782468+0000","last_undegraded":"2023-10-19T14:20:29.782468+0000","last_fullsized":"2023-10-19T14:20:29.782468+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:01:38.993506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.12","version":"0'0","reported_seq":12,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.785538+0000","last_change":"2023-10-19T14:20:29.784629+0000","last_active":"2023-10-19T14:20:31.785538+0000","last_peered":"2023-10-19T14:20:31.785538+0000","last_clean":"2023-10-19T14:20:31.785538+0000","last_became_active":"2023-10-19T14:20:29.784364+0000","last_became_peered":"2023-10-19T14:20:29.784364+0000","last_unstale":"2023-10-19T14:20:31.785538+0000","last_undegraded":"2023-10-19T14:20:31.785538+0000","last_fullsized":"2023-10-19T14:20:31.785538+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_clean_scrub_stamp":"2023-10-19T14:20:28.763908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:07:07.108020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.10","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.785348+0000","last_change":"2023-10-19T14:20:30.795623+0000","last_active":"2023-10-19T14:20:31.785348+0000","last_peered":"2023-10-19T14:20:31.785348+0000","last_clean":"2023-10-19T14:20:31.785348+0000","last_became_active":"2023-10-19T14:20:30.795347+0000","last_became_peered":"2023-10-19T14:20:30.795347+0000","last_unstale":"2023-10-19T14:20:31.785348+0000","last_undegraded":"2023-10-19T14:20:31.785348+0000","last_fullsized":"2023-10-19T14:20:31.785348+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:44:04.797792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.15","version":"222'3","reported_seq":19,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.789401+0000","last_change":"2023-10-19T14:20:18.723409+0000","last_active":"2023-10-19T14:20:30.789401+0000","last_peered":"2023-10-19T14:20:30.789401+0000","last_clean":"2023-10-19T14:20:30.789401+0000","last_became_active":"2023-10-19T14:20:18.723148+0000","last_became_peered":"2023-10-19T14:20:18.723148+0000","last_unstale":"2023-10-19T14:20:30.789401+0000","last_undegraded":"2023-10-19T14:20:30.789401+0000","last_fullsized":"2023-10-19T14:20:30.789401+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:13:53.110830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.786554+0000","last_change":"2023-10-19T14:20:19.334253+0000","last_active":"2023-10-19T14:20:30.786554+0000","last_peered":"2023-10-19T14:20:30.786554+0000","last_clean":"2023-10-19T14:20:30.786554+0000","last_became_active":"2023-10-19T14:20:19.333981+0000","last_became_peered":"2023-10-19T14:20:19.333981+0000","last_unstale":"2023-10-19T14:20:30.786554+0000","last_undegraded":"2023-10-19T14:20:30.786554+0000","last_fullsized":"2023-10-19T14:20:30.786554+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:48:00.239750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.13","version":"226'2","reported_seq":14,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.813695+0000","last_change":"2023-10-19T14:20:29.800227+0000","last_active":"2023-10-19T14:20:31.813695+0000","last_peered":"2023-10-19T14:20:31.813695+0000","last_clean":"2023-10-19T14:20:31.813695+0000","last_became_active":"2023-10-19T14:20:29.799696+0000","last_became_peered":"2023-10-19T14:20:29.799696+0000","last_unstale":"2023-10-19T14:20:31.813695+0000","last_undegraded":"2023-10-19T14:20:31.813695+0000","last_fullsized":"2023-10-19T14:20:31.813695+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_clean_scrub_stamp":"2023-10-19T14:20:28.763908+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:37:21.704109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.13","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.797344+0000","last_change":"2023-10-19T14:20:30.796333+0000","last_active":"2023-10-19T14:20:30.797344+0000","last_peered":"2023-10-19T14:20:30.797344+0000","last_clean":"2023-10-19T14:20:30.797344+0000","last_became_active":"2023-10-19T14:20:30.796046+0000","last_became_peered":"2023-10-19T14:20:30.796046+0000","last_unstale":"2023-10-19T14:20:30.797344+0000","last_undegraded":"2023-10-19T14:20:30.797344+0000","last_fullsized":"2023-10-19T14:20:30.797344+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:14:05.187716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":15,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.787532+0000","last_change":"2023-10-19T14:20:18.721992+0000","last_active":"2023-10-19T14:20:29.787532+0000","last_peered":"2023-10-19T14:20:29.787532+0000","last_clean":"2023-10-19T14:20:29.787532+0000","last_became_active":"2023-10-19T14:20:18.721742+0000","last_became_peered":"2023-10-19T14:20:18.721742+0000","last_unstale":"2023-10-19T14:20:29.787532+0000","last_undegraded":"2023-10-19T14:20:29.787532+0000","last_fullsized":"2023-10-19T14:20:29.787532+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:16:37.567429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":14,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.784142+0000","last_change":"2023-10-19T14:20:19.339048+0000","last_active":"2023-10-19T14:20:29.784142+0000","last_peered":"2023-10-19T14:20:29.784142+0000","last_clean":"2023-10-19T14:20:29.784142+0000","last_became_active":"2023-10-19T14:20:19.338786+0000","last_became_peered":"2023-10-19T14:20:19.338786+0000","last_unstale":"2023-10-19T14:20:29.784142+0000","last_undegraded":"2023-10-19T14:20:29.784142+0000","last_fullsized":"2023-10-19T14:20:29.784142+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:16:46.603067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.10","version":"226'8","reported_seq":20,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.813335+0000","last_change":"2023-10-19T14:20:29.798135+0000","last_active":"2023-10-19T14:20:31.813335+0000","last_peered":"2023-10-19T14:20:31.813335+0000","last_clean":"2023-10-19T14:20:31.813335+0000","last_became_active":"2023-10-19T14:20:29.797692+0000","last_became_peered":"2023-10-19T14:20:29.797692+0000","last_unstale":"2023-10-19T14:20:31.813335+0000","last_undegraded":"2023-10-19T14:20:31.813335+0000","last_fullsized":"2023-10-19T14:20:31.813335+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_clean_scrub_stamp":"2023-10-19T14:20:28.763908+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:38:29.957301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.12","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.784527+0000","last_change":"2023-10-19T14:20:30.790680+0000","last_active":"2023-10-19T14:20:31.784527+0000","last_peered":"2023-10-19T14:20:31.784527+0000","last_clean":"2023-10-19T14:20:31.784527+0000","last_became_active":"2023-10-19T14:20:30.790400+0000","last_became_peered":"2023-10-19T14:20:30.790400+0000","last_unstale":"2023-10-19T14:20:31.784527+0000","last_undegraded":"2023-10-19T14:20:31.784527+0000","last_fullsized":"2023-10-19T14:20:31.784527+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:26:05.573883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":15,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.782121+0000","last_change":"2023-10-19T14:20:18.722070+0000","last_active":"2023-10-19T14:20:29.782121+0000","last_peered":"2023-10-19T14:20:29.782121+0000","last_clean":"2023-10-19T14:20:29.782121+0000","last_became_active":"2023-10-19T14:20:18.721791+0000","last_became_peered":"2023-10-19T14:20:18.721791+0000","last_unstale":"2023-10-19T14:20:29.782121+0000","last_undegraded":"2023-10-19T14:20:29.782121+0000","last_fullsized":"2023-10-19T14:20:29.782121+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:17:55.048333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.10","version":"0'0","reported_seq":14,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.784494+0000","last_change":"2023-10-19T14:20:19.351668+0000","last_active":"2023-10-19T14:20:29.784494+0000","last_peered":"2023-10-19T14:20:29.784494+0000","last_clean":"2023-10-19T14:20:29.784494+0000","last_became_active":"2023-10-19T14:20:19.351298+0000","last_became_peered":"2023-10-19T14:20:19.351298+0000","last_unstale":"2023-10-19T14:20:29.784494+0000","last_undegraded":"2023-10-19T14:20:29.784494+0000","last_fullsized":"2023-10-19T14:20:29.784494+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:36:14.871743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.11","version":"0'0","reported_seq":12,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.781994+0000","last_change":"2023-10-19T14:20:29.790327+0000","last_active":"2023-10-19T14:20:31.781994+0000","last_peered":"2023-10-19T14:20:31.781994+0000","last_clean":"2023-10-19T14:20:31.781994+0000","last_became_active":"2023-10-19T14:20:29.790038+0000","last_became_peered":"2023-10-19T14:20:29.790038+0000","last_unstale":"2023-10-19T14:20:31.781994+0000","last_undegraded":"2023-10-19T14:20:31.781994+0000","last_fullsized":"2023-10-19T14:20:31.781994+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_clean_scrub_stamp":"2023-10-19T14:20:28.763908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:40:43.580514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.1d","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.784800+0000","last_change":"2023-10-19T14:20:30.801236+0000","last_active":"2023-10-19T14:20:31.784800+0000","last_peered":"2023-10-19T14:20:31.784800+0000","last_clean":"2023-10-19T14:20:31.784800+0000","last_became_active":"2023-10-19T14:20:30.800928+0000","last_became_peered":"2023-10-19T14:20:30.800928+0000","last_unstale":"2023-10-19T14:20:31.784800+0000","last_undegraded":"2023-10-19T14:20:31.784800+0000","last_fullsized":"2023-10-19T14:20:31.784800+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:52:54.676865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.1e","version":"226'1","reported_seq":13,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.814413+0000","last_change":"2023-10-19T14:20:29.798101+0000","last_active":"2023-10-19T14:20:31.814413+0000","last_peered":"2023-10-19T14:20:31.814413+0000","last_clean":"2023-10-19T14:20:31.814413+0000","last_became_active":"2023-10-19T14:20:29.797834+0000","last_became_peered":"2023-10-19T14:20:29.797834+0000","last_unstale":"2023-10-19T14:20:31.814413+0000","last_undegraded":"2023-10-19T14:20:31.814413+0000","last_fullsized":"2023-10-19T14:20:31.814413+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_clean_scrub_stamp":"2023-10-19T14:20:28.763908+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:50:11.018716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":15,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.784192+0000","last_change":"2023-10-19T14:20:18.718913+0000","last_active":"2023-10-19T14:20:29.784192+0000","last_peered":"2023-10-19T14:20:29.784192+0000","last_clean":"2023-10-19T14:20:29.784192+0000","last_became_active":"2023-10-19T14:20:18.718641+0000","last_became_peered":"2023-10-19T14:20:18.718641+0000","last_unstale":"2023-10-19T14:20:29.784192+0000","last_undegraded":"2023-10-19T14:20:29.784192+0000","last_fullsized":"2023-10-19T14:20:29.784192+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:54:40.217789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1f","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.788798+0000","last_change":"2023-10-19T14:20:19.346582+0000","last_active":"2023-10-19T14:20:30.788798+0000","last_peered":"2023-10-19T14:20:30.788798+0000","last_clean":"2023-10-19T14:20:30.788798+0000","last_became_active":"2023-10-19T14:20:19.346246+0000","last_became_peered":"2023-10-19T14:20:19.346246+0000","last_unstale":"2023-10-19T14:20:30.788798+0000","last_undegraded":"2023-10-19T14:20:30.788798+0000","last_fullsized":"2023-10-19T14:20:30.788798+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:37:58.222092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.1c","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.784738+0000","last_change":"2023-10-19T14:20:30.788933+0000","last_active":"2023-10-19T14:20:31.784738+0000","last_peered":"2023-10-19T14:20:31.784738+0000","last_clean":"2023-10-19T14:20:31.784738+0000","last_became_active":"2023-10-19T14:20:30.788378+0000","last_became_peered":"2023-10-19T14:20:30.788378+0000","last_unstale":"2023-10-19T14:20:31.784738+0000","last_undegraded":"2023-10-19T14:20:31.784738+0000","last_fullsized":"2023-10-19T14:20:31.784738+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:41:13.232644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.1f","version":"226'4","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.812922+0000","last_change":"2023-10-19T14:20:29.787116+0000","last_active":"2023-10-19T14:20:30.812922+0000","last_peered":"2023-10-19T14:20:30.812922+0000","last_clean":"2023-10-19T14:20:30.812922+0000","last_became_active":"2023-10-19T14:20:29.786827+0000","last_became_peered":"2023-10-19T14:20:29.786827+0000","last_unstale":"2023-10-19T14:20:30.812922+0000","last_undegraded":"2023-10-19T14:20:30.812922+0000","last_fullsized":"2023-10-19T14:20:30.812922+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_clean_scrub_stamp":"2023-10-19T14:20:28.763908+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:44:07.248181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":15,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.786373+0000","last_change":"2023-10-19T14:20:18.721875+0000","last_active":"2023-10-19T14:20:29.786373+0000","last_peered":"2023-10-19T14:20:29.786373+0000","last_clean":"2023-10-19T14:20:29.786373+0000","last_became_active":"2023-10-19T14:20:18.721611+0000","last_became_peered":"2023-10-19T14:20:18.721611+0000","last_unstale":"2023-10-19T14:20:29.786373+0000","last_undegraded":"2023-10-19T14:20:29.786373+0000","last_fullsized":"2023-10-19T14:20:29.786373+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:46:19.718145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1e","version":"0'0","reported_seq":14,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.786008+0000","last_change":"2023-10-19T14:20:19.343379+0000","last_active":"2023-10-19T14:20:29.786008+0000","last_peered":"2023-10-19T14:20:29.786008+0000","last_clean":"2023-10-19T14:20:29.786008+0000","last_became_active":"2023-10-19T14:20:19.343117+0000","last_became_peered":"2023-10-19T14:20:19.343117+0000","last_unstale":"2023-10-19T14:20:29.786008+0000","last_undegraded":"2023-10-19T14:20:29.786008+0000","last_fullsized":"2023-10-19T14:20:29.786008+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:33:52.226441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1e","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.814298+0000","last_change":"2023-10-19T14:20:30.802110+0000","last_active":"2023-10-19T14:20:31.814298+0000","last_peered":"2023-10-19T14:20:31.814298+0000","last_clean":"2023-10-19T14:20:31.814298+0000","last_became_active":"2023-10-19T14:20:30.801833+0000","last_became_peered":"2023-10-19T14:20:30.801833+0000","last_unstale":"2023-10-19T14:20:31.814298+0000","last_undegraded":"2023-10-19T14:20:31.814298+0000","last_fullsized":"2023-10-19T14:20:31.814298+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:50:03.887637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1b","version":"222'1","reported_seq":16,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.786436+0000","last_change":"2023-10-19T14:20:18.722476+0000","last_active":"2023-10-19T14:20:29.786436+0000","last_peered":"2023-10-19T14:20:29.786436+0000","last_clean":"2023-10-19T14:20:29.786436+0000","last_became_active":"2023-10-19T14:20:18.722195+0000","last_became_peered":"2023-10-19T14:20:18.722195+0000","last_unstale":"2023-10-19T14:20:29.786436+0000","last_undegraded":"2023-10-19T14:20:29.786436+0000","last_fullsized":"2023-10-19T14:20:29.786436+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:38:13.906613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1c","version":"0'0","reported_seq":14,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.786205+0000","last_change":"2023-10-19T14:20:19.349815+0000","last_active":"2023-10-19T14:20:29.786205+0000","last_peered":"2023-10-19T14:20:29.786205+0000","last_clean":"2023-10-19T14:20:29.786205+0000","last_became_active":"2023-10-19T14:20:19.349511+0000","last_became_peered":"2023-10-19T14:20:19.349511+0000","last_unstale":"2023-10-19T14:20:29.786205+0000","last_undegraded":"2023-10-19T14:20:29.786205+0000","last_fullsized":"2023-10-19T14:20:29.786205+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:32:39.312387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.1d","version":"226'2","reported_seq":14,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.782733+0000","last_change":"2023-10-19T14:20:29.802016+0000","last_active":"2023-10-19T14:20:31.782733+0000","last_peered":"2023-10-19T14:20:31.782733+0000","last_clean":"2023-10-19T14:20:31.782733+0000","last_became_active":"2023-10-19T14:20:29.801737+0000","last_became_peered":"2023-10-19T14:20:29.801737+0000","last_unstale":"2023-10-19T14:20:31.782733+0000","last_undegraded":"2023-10-19T14:20:31.782733+0000","last_fullsized":"2023-10-19T14:20:31.782733+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_clean_scrub_stamp":"2023-10-19T14:20:28.763908+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:01:55.267605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.19","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.813383+0000","last_change":"2023-10-19T14:20:30.802232+0000","last_active":"2023-10-19T14:20:31.813383+0000","last_peered":"2023-10-19T14:20:31.813383+0000","last_clean":"2023-10-19T14:20:31.813383+0000","last_became_active":"2023-10-19T14:20:30.801966+0000","last_became_peered":"2023-10-19T14:20:30.801966+0000","last_unstale":"2023-10-19T14:20:31.813383+0000","last_undegraded":"2023-10-19T14:20:31.813383+0000","last_fullsized":"2023-10-19T14:20:31.813383+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:44:41.933734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1c","version":"222'2","reported_seq":18,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.786824+0000","last_change":"2023-10-19T14:20:18.722673+0000","last_active":"2023-10-19T14:20:30.786824+0000","last_peered":"2023-10-19T14:20:30.786824+0000","last_clean":"2023-10-19T14:20:30.786824+0000","last_became_active":"2023-10-19T14:20:18.722418+0000","last_became_peered":"2023-10-19T14:20:18.722418+0000","last_unstale":"2023-10-19T14:20:30.786824+0000","last_undegraded":"2023-10-19T14:20:30.786824+0000","last_fullsized":"2023-10-19T14:20:30.786824+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:40:32.510221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1b","version":"0'0","reported_seq":14,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.782383+0000","last_change":"2023-10-19T14:20:19.348254+0000","last_active":"2023-10-19T14:20:29.782383+0000","last_peered":"2023-10-19T14:20:29.782383+0000","last_clean":"2023-10-19T14:20:29.782383+0000","last_became_active":"2023-10-19T14:20:19.347974+0000","last_became_peered":"2023-10-19T14:20:19.347974+0000","last_unstale":"2023-10-19T14:20:29.782383+0000","last_undegraded":"2023-10-19T14:20:29.782383+0000","last_fullsized":"2023-10-19T14:20:29.782383+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:32:42.913042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.1a","version":"226'1","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.813571+0000","last_change":"2023-10-19T14:20:29.795524+0000","last_active":"2023-10-19T14:20:30.813571+0000","last_peered":"2023-10-19T14:20:30.813571+0000","last_clean":"2023-10-19T14:20:30.813571+0000","last_became_active":"2023-10-19T14:20:29.795205+0000","last_became_peered":"2023-10-19T14:20:29.795205+0000","last_unstale":"2023-10-19T14:20:30.813571+0000","last_undegraded":"2023-10-19T14:20:30.813571+0000","last_fullsized":"2023-10-19T14:20:30.813571+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_clean_scrub_stamp":"2023-10-19T14:20:28.763908+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:16:30.284844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.18","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.784275+0000","last_change":"2023-10-19T14:20:30.791002+0000","last_active":"2023-10-19T14:20:31.784275+0000","last_peered":"2023-10-19T14:20:31.784275+0000","last_clean":"2023-10-19T14:20:31.784275+0000","last_became_active":"2023-10-19T14:20:30.790717+0000","last_became_peered":"2023-10-19T14:20:30.790717+0000","last_unstale":"2023-10-19T14:20:31.784275+0000","last_undegraded":"2023-10-19T14:20:31.784275+0000","last_fullsized":"2023-10-19T14:20:31.784275+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:13:13.631186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.1d","version":"222'2","reported_seq":18,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.787815+0000","last_change":"2023-10-19T14:20:18.717342+0000","last_active":"2023-10-19T14:20:30.787815+0000","last_peered":"2023-10-19T14:20:30.787815+0000","last_clean":"2023-10-19T14:20:30.787815+0000","last_became_active":"2023-10-19T14:20:18.717047+0000","last_became_peered":"2023-10-19T14:20:18.717047+0000","last_unstale":"2023-10-19T14:20:30.787815+0000","last_undegraded":"2023-10-19T14:20:30.787815+0000","last_fullsized":"2023-10-19T14:20:30.787815+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:28:55.961320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.1a","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.799358+0000","last_change":"2023-10-19T14:20:19.346318+0000","last_active":"2023-10-19T14:20:30.799358+0000","last_peered":"2023-10-19T14:20:30.799358+0000","last_clean":"2023-10-19T14:20:30.799358+0000","last_became_active":"2023-10-19T14:20:19.346014+0000","last_became_peered":"2023-10-19T14:20:19.346014+0000","last_unstale":"2023-10-19T14:20:30.799358+0000","last_undegraded":"2023-10-19T14:20:30.799358+0000","last_fullsized":"2023-10-19T14:20:30.799358+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:20:35.750886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1b","version":"226'1","reported_seq":12,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.813306+0000","last_change":"2023-10-19T14:20:29.796824+0000","last_active":"2023-10-19T14:20:30.813306+0000","last_peered":"2023-10-19T14:20:30.813306+0000","last_clean":"2023-10-19T14:20:30.813306+0000","last_became_active":"2023-10-19T14:20:29.796469+0000","last_became_peered":"2023-10-19T14:20:29.796469+0000","last_unstale":"2023-10-19T14:20:30.813306+0000","last_undegraded":"2023-10-19T14:20:30.813306+0000","last_fullsized":"2023-10-19T14:20:30.813306+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_clean_scrub_stamp":"2023-10-19T14:20:28.763908+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:40:43.580514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.1b","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.790310+0000","last_change":"2023-10-19T14:20:30.787300+0000","last_active":"2023-10-19T14:20:30.790310+0000","last_peered":"2023-10-19T14:20:30.790310+0000","last_clean":"2023-10-19T14:20:30.790310+0000","last_became_active":"2023-10-19T14:20:30.786994+0000","last_became_peered":"2023-10-19T14:20:30.786994+0000","last_unstale":"2023-10-19T14:20:30.790310+0000","last_undegraded":"2023-10-19T14:20:30.790310+0000","last_fullsized":"2023-10-19T14:20:30.790310+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:18:21.133124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1e","version":"222'1","reported_seq":17,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.781879+0000","last_change":"2023-10-19T14:20:18.719165+0000","last_active":"2023-10-19T14:20:30.781879+0000","last_peered":"2023-10-19T14:20:30.781879+0000","last_clean":"2023-10-19T14:20:30.781879+0000","last_became_active":"2023-10-19T14:20:18.717146+0000","last_became_peered":"2023-10-19T14:20:18.717146+0000","last_unstale":"2023-10-19T14:20:30.781879+0000","last_undegraded":"2023-10-19T14:20:30.781879+0000","last_fullsized":"2023-10-19T14:20:30.781879+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:57:14.320702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.782644+0000","last_change":"2023-10-19T14:20:19.343393+0000","last_active":"2023-10-19T14:20:30.782644+0000","last_peered":"2023-10-19T14:20:30.782644+0000","last_clean":"2023-10-19T14:20:30.782644+0000","last_became_active":"2023-10-19T14:20:19.343011+0000","last_became_peered":"2023-10-19T14:20:19.343011+0000","last_unstale":"2023-10-19T14:20:30.782644+0000","last_undegraded":"2023-10-19T14:20:30.782644+0000","last_fullsized":"2023-10-19T14:20:30.782644+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:22:47.864868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.18","version":"0'0","reported_seq":12,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.784505+0000","last_change":"2023-10-19T14:20:29.794071+0000","last_active":"2023-10-19T14:20:31.784505+0000","last_peered":"2023-10-19T14:20:31.784505+0000","last_clean":"2023-10-19T14:20:31.784505+0000","last_became_active":"2023-10-19T14:20:29.793702+0000","last_became_peered":"2023-10-19T14:20:29.793702+0000","last_unstale":"2023-10-19T14:20:31.784505+0000","last_undegraded":"2023-10-19T14:20:31.784505+0000","last_fullsized":"2023-10-19T14:20:31.784505+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_clean_scrub_stamp":"2023-10-19T14:20:28.763908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:20:52.649351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.1a","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.797372+0000","last_change":"2023-10-19T14:20:30.795544+0000","last_active":"2023-10-19T14:20:30.797372+0000","last_peered":"2023-10-19T14:20:30.797372+0000","last_clean":"2023-10-19T14:20:30.797372+0000","last_became_active":"2023-10-19T14:20:30.795273+0000","last_became_peered":"2023-10-19T14:20:30.795273+0000","last_unstale":"2023-10-19T14:20:30.797372+0000","last_undegraded":"2023-10-19T14:20:30.797372+0000","last_fullsized":"2023-10-19T14:20:30.797372+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:07:29.800710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.1f","version":"222'4","reported_seq":19,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.786147+0000","last_change":"2023-10-19T14:20:18.725431+0000","last_active":"2023-10-19T14:20:29.786147+0000","last_peered":"2023-10-19T14:20:29.786147+0000","last_clean":"2023-10-19T14:20:29.786147+0000","last_became_active":"2023-10-19T14:20:18.725142+0000","last_became_peered":"2023-10-19T14:20:18.725142+0000","last_unstale":"2023-10-19T14:20:29.786147+0000","last_undegraded":"2023-10-19T14:20:29.786147+0000","last_fullsized":"2023-10-19T14:20:29.786147+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":220,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:17.693615+0000","last_clean_scrub_stamp":"2023-10-19T14:20:17.693615+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T02:03:46.398046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.785197+0000","last_change":"2023-10-19T14:20:19.333259+0000","last_active":"2023-10-19T14:20:30.785197+0000","last_peered":"2023-10-19T14:20:30.785197+0000","last_clean":"2023-10-19T14:20:30.785197+0000","last_became_active":"2023-10-19T14:20:19.332735+0000","last_became_peered":"2023-10-19T14:20:19.332735+0000","last_unstale":"2023-10-19T14:20:30.785197+0000","last_undegraded":"2023-10-19T14:20:30.785197+0000","last_fullsized":"2023-10-19T14:20:30.785197+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:46:26.663119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.19","version":"0'0","reported_seq":11,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.788500+0000","last_change":"2023-10-19T14:20:29.796237+0000","last_active":"2023-10-19T14:20:30.788500+0000","last_peered":"2023-10-19T14:20:30.788500+0000","last_clean":"2023-10-19T14:20:30.788500+0000","last_became_active":"2023-10-19T14:20:29.795731+0000","last_became_peered":"2023-10-19T14:20:29.795731+0000","last_unstale":"2023-10-19T14:20:30.788500+0000","last_undegraded":"2023-10-19T14:20:30.788500+0000","last_fullsized":"2023-10-19T14:20:30.788500+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:28.763908+0000","last_clean_scrub_stamp":"2023-10-19T14:20:28.763908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:38:36.765526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.22","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.782363+0000","last_change":"2023-10-19T14:20:30.803463+0000","last_active":"2023-10-19T14:20:31.782363+0000","last_peered":"2023-10-19T14:20:31.782363+0000","last_clean":"2023-10-19T14:20:31.782363+0000","last_became_active":"2023-10-19T14:20:30.803169+0000","last_became_peered":"2023-10-19T14:20:30.803169+0000","last_unstale":"2023-10-19T14:20:31.782363+0000","last_undegraded":"2023-10-19T14:20:31.782363+0000","last_fullsized":"2023-10-19T14:20:31.782363+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:19:00.812957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.20","version":"0'0","reported_seq":14,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.785838+0000","last_change":"2023-10-19T14:20:19.348899+0000","last_active":"2023-10-19T14:20:29.785838+0000","last_peered":"2023-10-19T14:20:29.785838+0000","last_clean":"2023-10-19T14:20:29.785838+0000","last_became_active":"2023-10-19T14:20:19.348453+0000","last_became_peered":"2023-10-19T14:20:19.348453+0000","last_unstale":"2023-10-19T14:20:29.785838+0000","last_undegraded":"2023-10-19T14:20:29.785838+0000","last_fullsized":"2023-10-19T14:20:29.785838+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:01:53.425378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.23","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.785706+0000","last_change":"2023-10-19T14:20:30.790486+0000","last_active":"2023-10-19T14:20:31.785706+0000","last_peered":"2023-10-19T14:20:31.785706+0000","last_clean":"2023-10-19T14:20:31.785706+0000","last_became_active":"2023-10-19T14:20:30.790164+0000","last_became_peered":"2023-10-19T14:20:30.790164+0000","last_unstale":"2023-10-19T14:20:31.785706+0000","last_undegraded":"2023-10-19T14:20:31.785706+0000","last_fullsized":"2023-10-19T14:20:31.785706+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:12:50.182135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.21","version":"0'0","reported_seq":14,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.785838+0000","last_change":"2023-10-19T14:20:19.349579+0000","last_active":"2023-10-19T14:20:29.785838+0000","last_peered":"2023-10-19T14:20:29.785838+0000","last_clean":"2023-10-19T14:20:29.785838+0000","last_became_active":"2023-10-19T14:20:19.349321+0000","last_became_peered":"2023-10-19T14:20:19.349321+0000","last_unstale":"2023-10-19T14:20:29.785838+0000","last_undegraded":"2023-10-19T14:20:29.785838+0000","last_fullsized":"2023-10-19T14:20:29.785838+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:44:03.588824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.20","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.782192+0000","last_change":"2023-10-19T14:20:30.799253+0000","last_active":"2023-10-19T14:20:31.782192+0000","last_peered":"2023-10-19T14:20:31.782192+0000","last_clean":"2023-10-19T14:20:31.782192+0000","last_became_active":"2023-10-19T14:20:30.799000+0000","last_became_peered":"2023-10-19T14:20:30.799000+0000","last_unstale":"2023-10-19T14:20:31.782192+0000","last_undegraded":"2023-10-19T14:20:31.782192+0000","last_fullsized":"2023-10-19T14:20:31.782192+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:28:29.776053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.22","version":"0'0","reported_seq":14,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.784544+0000","last_change":"2023-10-19T14:20:19.342297+0000","last_active":"2023-10-19T14:20:29.784544+0000","last_peered":"2023-10-19T14:20:29.784544+0000","last_clean":"2023-10-19T14:20:29.784544+0000","last_became_active":"2023-10-19T14:20:19.342026+0000","last_became_peered":"2023-10-19T14:20:19.342026+0000","last_unstale":"2023-10-19T14:20:29.784544+0000","last_undegraded":"2023-10-19T14:20:29.784544+0000","last_fullsized":"2023-10-19T14:20:29.784544+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:43:57.179927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.21","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.793992+0000","last_change":"2023-10-19T14:20:30.791876+0000","last_active":"2023-10-19T14:20:30.793992+0000","last_peered":"2023-10-19T14:20:30.793992+0000","last_clean":"2023-10-19T14:20:30.793992+0000","last_became_active":"2023-10-19T14:20:30.791563+0000","last_became_peered":"2023-10-19T14:20:30.791563+0000","last_unstale":"2023-10-19T14:20:30.793992+0000","last_undegraded":"2023-10-19T14:20:30.793992+0000","last_fullsized":"2023-10-19T14:20:30.793992+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:04:45.084132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.23","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.786920+0000","last_change":"2023-10-19T14:20:19.342642+0000","last_active":"2023-10-19T14:20:30.786920+0000","last_peered":"2023-10-19T14:20:30.786920+0000","last_clean":"2023-10-19T14:20:30.786920+0000","last_became_active":"2023-10-19T14:20:19.342379+0000","last_became_peered":"2023-10-19T14:20:19.342379+0000","last_unstale":"2023-10-19T14:20:30.786920+0000","last_undegraded":"2023-10-19T14:20:30.786920+0000","last_fullsized":"2023-10-19T14:20:30.786920+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:33:34.581616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.26","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.785208+0000","last_change":"2023-10-19T14:20:30.807457+0000","last_active":"2023-10-19T14:20:31.785208+0000","last_peered":"2023-10-19T14:20:31.785208+0000","last_clean":"2023-10-19T14:20:31.785208+0000","last_became_active":"2023-10-19T14:20:30.807150+0000","last_became_peered":"2023-10-19T14:20:30.807150+0000","last_unstale":"2023-10-19T14:20:31.785208+0000","last_undegraded":"2023-10-19T14:20:31.785208+0000","last_fullsized":"2023-10-19T14:20:31.785208+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:13:06.353666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.24","version":"0'0","reported_seq":14,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.781754+0000","last_change":"2023-10-19T14:20:19.345949+0000","last_active":"2023-10-19T14:20:29.781754+0000","last_peered":"2023-10-19T14:20:29.781754+0000","last_clean":"2023-10-19T14:20:29.781754+0000","last_became_active":"2023-10-19T14:20:19.345664+0000","last_became_peered":"2023-10-19T14:20:19.345664+0000","last_unstale":"2023-10-19T14:20:29.781754+0000","last_undegraded":"2023-10-19T14:20:29.781754+0000","last_fullsized":"2023-10-19T14:20:29.781754+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:52:23.062217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.27","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.804953+0000","last_change":"2023-10-19T14:20:30.798702+0000","last_active":"2023-10-19T14:20:30.804953+0000","last_peered":"2023-10-19T14:20:30.804953+0000","last_clean":"2023-10-19T14:20:30.804953+0000","last_became_active":"2023-10-19T14:20:30.798425+0000","last_became_peered":"2023-10-19T14:20:30.798425+0000","last_unstale":"2023-10-19T14:20:30.804953+0000","last_undegraded":"2023-10-19T14:20:30.804953+0000","last_fullsized":"2023-10-19T14:20:30.804953+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:36:40.012568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.25","version":"0'0","reported_seq":14,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.788699+0000","last_change":"2023-10-19T14:20:19.347669+0000","last_active":"2023-10-19T14:20:29.788699+0000","last_peered":"2023-10-19T14:20:29.788699+0000","last_clean":"2023-10-19T14:20:29.788699+0000","last_became_active":"2023-10-19T14:20:19.347397+0000","last_became_peered":"2023-10-19T14:20:19.347397+0000","last_unstale":"2023-10-19T14:20:29.788699+0000","last_undegraded":"2023-10-19T14:20:29.788699+0000","last_fullsized":"2023-10-19T14:20:29.788699+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:47:46.746053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.24","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.813987+0000","last_change":"2023-10-19T14:20:30.803549+0000","last_active":"2023-10-19T14:20:31.813987+0000","last_peered":"2023-10-19T14:20:31.813987+0000","last_clean":"2023-10-19T14:20:31.813987+0000","last_became_active":"2023-10-19T14:20:30.803130+0000","last_became_peered":"2023-10-19T14:20:30.803130+0000","last_unstale":"2023-10-19T14:20:31.813987+0000","last_undegraded":"2023-10-19T14:20:31.813987+0000","last_fullsized":"2023-10-19T14:20:31.813987+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:20:12.233298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.26","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.781764+0000","last_change":"2023-10-19T14:20:19.348136+0000","last_active":"2023-10-19T14:20:30.781764+0000","last_peered":"2023-10-19T14:20:30.781764+0000","last_clean":"2023-10-19T14:20:30.781764+0000","last_became_active":"2023-10-19T14:20:19.347832+0000","last_became_peered":"2023-10-19T14:20:19.347832+0000","last_unstale":"2023-10-19T14:20:30.781764+0000","last_undegraded":"2023-10-19T14:20:30.781764+0000","last_fullsized":"2023-10-19T14:20:30.781764+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:16:46.603067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.25","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.801797+0000","last_change":"2023-10-19T14:20:30.799093+0000","last_active":"2023-10-19T14:20:30.801797+0000","last_peered":"2023-10-19T14:20:30.801797+0000","last_clean":"2023-10-19T14:20:30.801797+0000","last_became_active":"2023-10-19T14:20:30.798726+0000","last_became_peered":"2023-10-19T14:20:30.798726+0000","last_unstale":"2023-10-19T14:20:30.801797+0000","last_undegraded":"2023-10-19T14:20:30.801797+0000","last_fullsized":"2023-10-19T14:20:30.801797+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T02:08:44.803755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.27","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.788702+0000","last_change":"2023-10-19T14:20:19.339558+0000","last_active":"2023-10-19T14:20:30.788702+0000","last_peered":"2023-10-19T14:20:30.788702+0000","last_clean":"2023-10-19T14:20:30.788702+0000","last_became_active":"2023-10-19T14:20:19.339298+0000","last_became_peered":"2023-10-19T14:20:19.339298+0000","last_unstale":"2023-10-19T14:20:30.788702+0000","last_undegraded":"2023-10-19T14:20:30.788702+0000","last_fullsized":"2023-10-19T14:20:30.788702+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:27:05.614080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.2a","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.798377+0000","last_change":"2023-10-19T14:20:30.797190+0000","last_active":"2023-10-19T14:20:30.798377+0000","last_peered":"2023-10-19T14:20:30.798377+0000","last_clean":"2023-10-19T14:20:30.798377+0000","last_became_active":"2023-10-19T14:20:30.796902+0000","last_became_peered":"2023-10-19T14:20:30.796902+0000","last_unstale":"2023-10-19T14:20:30.798377+0000","last_undegraded":"2023-10-19T14:20:30.798377+0000","last_fullsized":"2023-10-19T14:20:30.798377+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:38:06.530226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.28","version":"0'0","reported_seq":14,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.781321+0000","last_change":"2023-10-19T14:20:19.341549+0000","last_active":"2023-10-19T14:20:29.781321+0000","last_peered":"2023-10-19T14:20:29.781321+0000","last_clean":"2023-10-19T14:20:29.781321+0000","last_became_active":"2023-10-19T14:20:19.341276+0000","last_became_peered":"2023-10-19T14:20:19.341276+0000","last_unstale":"2023-10-19T14:20:29.781321+0000","last_undegraded":"2023-10-19T14:20:29.781321+0000","last_fullsized":"2023-10-19T14:20:29.781321+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:23:15.830657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.2b","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.784598+0000","last_change":"2023-10-19T14:20:30.791962+0000","last_active":"2023-10-19T14:20:31.784598+0000","last_peered":"2023-10-19T14:20:31.784598+0000","last_clean":"2023-10-19T14:20:31.784598+0000","last_became_active":"2023-10-19T14:20:30.791621+0000","last_became_peered":"2023-10-19T14:20:30.791621+0000","last_unstale":"2023-10-19T14:20:31.784598+0000","last_undegraded":"2023-10-19T14:20:31.784598+0000","last_fullsized":"2023-10-19T14:20:31.784598+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:32:05.192356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.29","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.782836+0000","last_change":"2023-10-19T14:20:19.333467+0000","last_active":"2023-10-19T14:20:30.782836+0000","last_peered":"2023-10-19T14:20:30.782836+0000","last_clean":"2023-10-19T14:20:30.782836+0000","last_became_active":"2023-10-19T14:20:19.333176+0000","last_became_peered":"2023-10-19T14:20:19.333176+0000","last_unstale":"2023-10-19T14:20:30.782836+0000","last_undegraded":"2023-10-19T14:20:30.782836+0000","last_fullsized":"2023-10-19T14:20:30.782836+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:39:10.256824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.28","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.795475+0000","last_change":"2023-10-19T14:20:30.793171+0000","last_active":"2023-10-19T14:20:30.795475+0000","last_peered":"2023-10-19T14:20:30.795475+0000","last_clean":"2023-10-19T14:20:30.795475+0000","last_became_active":"2023-10-19T14:20:30.792382+0000","last_became_peered":"2023-10-19T14:20:30.792382+0000","last_unstale":"2023-10-19T14:20:30.795475+0000","last_undegraded":"2023-10-19T14:20:30.795475+0000","last_fullsized":"2023-10-19T14:20:30.795475+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:48:11.314597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.2a","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.799275+0000","last_change":"2023-10-19T14:20:19.345667+0000","last_active":"2023-10-19T14:20:30.799275+0000","last_peered":"2023-10-19T14:20:30.799275+0000","last_clean":"2023-10-19T14:20:30.799275+0000","last_became_active":"2023-10-19T14:20:19.345343+0000","last_became_peered":"2023-10-19T14:20:19.345343+0000","last_unstale":"2023-10-19T14:20:30.799275+0000","last_undegraded":"2023-10-19T14:20:30.799275+0000","last_fullsized":"2023-10-19T14:20:30.799275+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:36:14.871743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.29","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.800683+0000","last_change":"2023-10-19T14:20:30.799002+0000","last_active":"2023-10-19T14:20:30.800683+0000","last_peered":"2023-10-19T14:20:30.800683+0000","last_clean":"2023-10-19T14:20:30.800683+0000","last_became_active":"2023-10-19T14:20:30.798780+0000","last_became_peered":"2023-10-19T14:20:30.798780+0000","last_unstale":"2023-10-19T14:20:30.800683+0000","last_undegraded":"2023-10-19T14:20:30.800683+0000","last_fullsized":"2023-10-19T14:20:30.800683+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:36:47.244576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.2b","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.800012+0000","last_change":"2023-10-19T14:20:19.351751+0000","last_active":"2023-10-19T14:20:30.800012+0000","last_peered":"2023-10-19T14:20:30.800012+0000","last_clean":"2023-10-19T14:20:30.800012+0000","last_became_active":"2023-10-19T14:20:19.350197+0000","last_became_peered":"2023-10-19T14:20:19.350197+0000","last_unstale":"2023-10-19T14:20:30.800012+0000","last_undegraded":"2023-10-19T14:20:30.800012+0000","last_fullsized":"2023-10-19T14:20:30.800012+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:18:10.058277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2e","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.785156+0000","last_change":"2023-10-19T14:20:30.800432+0000","last_active":"2023-10-19T14:20:31.785156+0000","last_peered":"2023-10-19T14:20:31.785156+0000","last_clean":"2023-10-19T14:20:31.785156+0000","last_became_active":"2023-10-19T14:20:30.799997+0000","last_became_peered":"2023-10-19T14:20:30.799997+0000","last_unstale":"2023-10-19T14:20:31.785156+0000","last_undegraded":"2023-10-19T14:20:31.785156+0000","last_fullsized":"2023-10-19T14:20:31.785156+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:06:24.375786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.2c","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.797597+0000","last_change":"2023-10-19T14:20:19.343670+0000","last_active":"2023-10-19T14:20:30.797597+0000","last_peered":"2023-10-19T14:20:30.797597+0000","last_clean":"2023-10-19T14:20:30.797597+0000","last_became_active":"2023-10-19T14:20:19.342491+0000","last_became_peered":"2023-10-19T14:20:19.342491+0000","last_unstale":"2023-10-19T14:20:30.797597+0000","last_undegraded":"2023-10-19T14:20:30.797597+0000","last_fullsized":"2023-10-19T14:20:30.797597+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:32:39.312387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2f","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.814351+0000","last_change":"2023-10-19T14:20:30.801539+0000","last_active":"2023-10-19T14:20:31.814351+0000","last_peered":"2023-10-19T14:20:31.814351+0000","last_clean":"2023-10-19T14:20:31.814351+0000","last_became_active":"2023-10-19T14:20:30.801273+0000","last_became_peered":"2023-10-19T14:20:30.801273+0000","last_unstale":"2023-10-19T14:20:31.814351+0000","last_undegraded":"2023-10-19T14:20:31.814351+0000","last_fullsized":"2023-10-19T14:20:31.814351+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:50:44.452283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2d","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.799030+0000","last_change":"2023-10-19T14:20:19.352942+0000","last_active":"2023-10-19T14:20:30.799030+0000","last_peered":"2023-10-19T14:20:30.799030+0000","last_clean":"2023-10-19T14:20:30.799030+0000","last_became_active":"2023-10-19T14:20:19.352678+0000","last_became_peered":"2023-10-19T14:20:19.352678+0000","last_unstale":"2023-10-19T14:20:30.799030+0000","last_undegraded":"2023-10-19T14:20:30.799030+0000","last_fullsized":"2023-10-19T14:20:30.799030+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:19:02.500176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2c","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.782792+0000","last_change":"2023-10-19T14:20:30.799925+0000","last_active":"2023-10-19T14:20:31.782792+0000","last_peered":"2023-10-19T14:20:31.782792+0000","last_clean":"2023-10-19T14:20:31.782792+0000","last_became_active":"2023-10-19T14:20:30.799680+0000","last_became_peered":"2023-10-19T14:20:30.799680+0000","last_unstale":"2023-10-19T14:20:31.782792+0000","last_undegraded":"2023-10-19T14:20:31.782792+0000","last_fullsized":"2023-10-19T14:20:31.782792+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:21:29.982170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.2e","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.786969+0000","last_change":"2023-10-19T14:20:19.341197+0000","last_active":"2023-10-19T14:20:30.786969+0000","last_peered":"2023-10-19T14:20:30.786969+0000","last_clean":"2023-10-19T14:20:30.786969+0000","last_became_active":"2023-10-19T14:20:19.340929+0000","last_became_peered":"2023-10-19T14:20:19.340929+0000","last_unstale":"2023-10-19T14:20:30.786969+0000","last_undegraded":"2023-10-19T14:20:30.786969+0000","last_fullsized":"2023-10-19T14:20:30.786969+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:44:28.147164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.2d","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.804426+0000","last_change":"2023-10-19T14:20:30.797134+0000","last_active":"2023-10-19T14:20:30.804426+0000","last_peered":"2023-10-19T14:20:30.804426+0000","last_clean":"2023-10-19T14:20:30.804426+0000","last_became_active":"2023-10-19T14:20:30.796900+0000","last_became_peered":"2023-10-19T14:20:30.796900+0000","last_unstale":"2023-10-19T14:20:30.804426+0000","last_undegraded":"2023-10-19T14:20:30.804426+0000","last_fullsized":"2023-10-19T14:20:30.804426+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:38:09.296939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.2f","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.788609+0000","last_change":"2023-10-19T14:20:19.344630+0000","last_active":"2023-10-19T14:20:30.788609+0000","last_peered":"2023-10-19T14:20:30.788609+0000","last_clean":"2023-10-19T14:20:30.788609+0000","last_became_active":"2023-10-19T14:20:19.344354+0000","last_became_peered":"2023-10-19T14:20:19.344354+0000","last_unstale":"2023-10-19T14:20:30.788609+0000","last_undegraded":"2023-10-19T14:20:30.788609+0000","last_fullsized":"2023-10-19T14:20:30.788609+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:10:30.171693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.32","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.813589+0000","last_change":"2023-10-19T14:20:30.810659+0000","last_active":"2023-10-19T14:20:31.813589+0000","last_peered":"2023-10-19T14:20:31.813589+0000","last_clean":"2023-10-19T14:20:31.813589+0000","last_became_active":"2023-10-19T14:20:30.810384+0000","last_became_peered":"2023-10-19T14:20:30.810384+0000","last_unstale":"2023-10-19T14:20:31.813589+0000","last_undegraded":"2023-10-19T14:20:31.813589+0000","last_fullsized":"2023-10-19T14:20:31.813589+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:29:51.812187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.30","version":"0'0","reported_seq":14,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.784095+0000","last_change":"2023-10-19T14:20:19.338108+0000","last_active":"2023-10-19T14:20:29.784095+0000","last_peered":"2023-10-19T14:20:29.784095+0000","last_clean":"2023-10-19T14:20:29.784095+0000","last_became_active":"2023-10-19T14:20:19.337834+0000","last_became_peered":"2023-10-19T14:20:19.337834+0000","last_unstale":"2023-10-19T14:20:29.784095+0000","last_undegraded":"2023-10-19T14:20:29.784095+0000","last_fullsized":"2023-10-19T14:20:29.784095+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:15:21.019725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.33","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.782574+0000","last_change":"2023-10-19T14:20:30.791799+0000","last_active":"2023-10-19T14:20:31.782574+0000","last_peered":"2023-10-19T14:20:31.782574+0000","last_clean":"2023-10-19T14:20:31.782574+0000","last_became_active":"2023-10-19T14:20:30.791484+0000","last_became_peered":"2023-10-19T14:20:30.791484+0000","last_unstale":"2023-10-19T14:20:31.782574+0000","last_undegraded":"2023-10-19T14:20:31.782574+0000","last_fullsized":"2023-10-19T14:20:31.782574+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:53:03.050006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.31","version":"0'0","reported_seq":14,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.784960+0000","last_change":"2023-10-19T14:20:19.349749+0000","last_active":"2023-10-19T14:20:29.784960+0000","last_peered":"2023-10-19T14:20:29.784960+0000","last_clean":"2023-10-19T14:20:29.784960+0000","last_became_active":"2023-10-19T14:20:19.349481+0000","last_became_peered":"2023-10-19T14:20:19.349481+0000","last_unstale":"2023-10-19T14:20:29.784960+0000","last_undegraded":"2023-10-19T14:20:29.784960+0000","last_fullsized":"2023-10-19T14:20:29.784960+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:20:35.750886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.30","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.781976+0000","last_change":"2023-10-19T14:20:30.790210+0000","last_active":"2023-10-19T14:20:31.781976+0000","last_peered":"2023-10-19T14:20:31.781976+0000","last_clean":"2023-10-19T14:20:31.781976+0000","last_became_active":"2023-10-19T14:20:30.789906+0000","last_became_peered":"2023-10-19T14:20:30.789906+0000","last_unstale":"2023-10-19T14:20:31.781976+0000","last_undegraded":"2023-10-19T14:20:31.781976+0000","last_fullsized":"2023-10-19T14:20:31.781976+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:49:15.154863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.32","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.799189+0000","last_change":"2023-10-19T14:20:19.346548+0000","last_active":"2023-10-19T14:20:30.799189+0000","last_peered":"2023-10-19T14:20:30.799189+0000","last_clean":"2023-10-19T14:20:30.799189+0000","last_became_active":"2023-10-19T14:20:19.345001+0000","last_became_peered":"2023-10-19T14:20:19.345001+0000","last_unstale":"2023-10-19T14:20:30.799189+0000","last_undegraded":"2023-10-19T14:20:30.799189+0000","last_fullsized":"2023-10-19T14:20:30.799189+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T02:03:47.400085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.31","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.806474+0000","last_change":"2023-10-19T14:20:30.803998+0000","last_active":"2023-10-19T14:20:30.806474+0000","last_peered":"2023-10-19T14:20:30.806474+0000","last_clean":"2023-10-19T14:20:30.806474+0000","last_became_active":"2023-10-19T14:20:30.803734+0000","last_became_peered":"2023-10-19T14:20:30.803734+0000","last_unstale":"2023-10-19T14:20:30.806474+0000","last_undegraded":"2023-10-19T14:20:30.806474+0000","last_fullsized":"2023-10-19T14:20:30.806474+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:28:29.776053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.33","version":"0'0","reported_seq":14,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.786335+0000","last_change":"2023-10-19T14:20:19.351639+0000","last_active":"2023-10-19T14:20:29.786335+0000","last_peered":"2023-10-19T14:20:29.786335+0000","last_clean":"2023-10-19T14:20:29.786335+0000","last_became_active":"2023-10-19T14:20:19.350821+0000","last_became_peered":"2023-10-19T14:20:19.350821+0000","last_unstale":"2023-10-19T14:20:29.786335+0000","last_undegraded":"2023-10-19T14:20:29.786335+0000","last_fullsized":"2023-10-19T14:20:29.786335+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T02:16:40.560667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.36","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.785082+0000","last_change":"2023-10-19T14:20:30.789975+0000","last_active":"2023-10-19T14:20:31.785082+0000","last_peered":"2023-10-19T14:20:31.785082+0000","last_clean":"2023-10-19T14:20:31.785082+0000","last_became_active":"2023-10-19T14:20:30.789652+0000","last_became_peered":"2023-10-19T14:20:30.789652+0000","last_unstale":"2023-10-19T14:20:31.785082+0000","last_undegraded":"2023-10-19T14:20:31.785082+0000","last_fullsized":"2023-10-19T14:20:31.785082+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:37:17.323203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.34","version":"0'0","reported_seq":14,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.781664+0000","last_change":"2023-10-19T14:20:19.350634+0000","last_active":"2023-10-19T14:20:29.781664+0000","last_peered":"2023-10-19T14:20:29.781664+0000","last_clean":"2023-10-19T14:20:29.781664+0000","last_became_active":"2023-10-19T14:20:19.350362+0000","last_became_peered":"2023-10-19T14:20:19.350362+0000","last_unstale":"2023-10-19T14:20:29.781664+0000","last_undegraded":"2023-10-19T14:20:29.781664+0000","last_fullsized":"2023-10-19T14:20:29.781664+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:30:33.652808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.37","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.799021+0000","last_change":"2023-10-19T14:20:30.798066+0000","last_active":"2023-10-19T14:20:30.799021+0000","last_peered":"2023-10-19T14:20:30.799021+0000","last_clean":"2023-10-19T14:20:30.799021+0000","last_became_active":"2023-10-19T14:20:30.797781+0000","last_became_peered":"2023-10-19T14:20:30.797781+0000","last_unstale":"2023-10-19T14:20:30.799021+0000","last_undegraded":"2023-10-19T14:20:30.799021+0000","last_fullsized":"2023-10-19T14:20:30.799021+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:20:46.825732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.35","version":"0'0","reported_seq":14,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.788609+0000","last_change":"2023-10-19T14:20:19.347686+0000","last_active":"2023-10-19T14:20:29.788609+0000","last_peered":"2023-10-19T14:20:29.788609+0000","last_clean":"2023-10-19T14:20:29.788609+0000","last_became_active":"2023-10-19T14:20:19.347432+0000","last_became_peered":"2023-10-19T14:20:19.347432+0000","last_unstale":"2023-10-19T14:20:29.788609+0000","last_undegraded":"2023-10-19T14:20:29.788609+0000","last_fullsized":"2023-10-19T14:20:29.788609+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:46:20.720183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.34","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.813864+0000","last_change":"2023-10-19T14:20:30.803838+0000","last_active":"2023-10-19T14:20:31.813864+0000","last_peered":"2023-10-19T14:20:31.813864+0000","last_clean":"2023-10-19T14:20:31.813864+0000","last_became_active":"2023-10-19T14:20:30.803512+0000","last_became_peered":"2023-10-19T14:20:30.803512+0000","last_unstale":"2023-10-19T14:20:31.813864+0000","last_undegraded":"2023-10-19T14:20:31.813864+0000","last_fullsized":"2023-10-19T14:20:31.813864+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:50:16.345467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.36","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.797815+0000","last_change":"2023-10-19T14:20:19.350276+0000","last_active":"2023-10-19T14:20:30.797815+0000","last_peered":"2023-10-19T14:20:30.797815+0000","last_clean":"2023-10-19T14:20:30.797815+0000","last_became_active":"2023-10-19T14:20:19.349877+0000","last_became_peered":"2023-10-19T14:20:19.349877+0000","last_unstale":"2023-10-19T14:20:30.797815+0000","last_undegraded":"2023-10-19T14:20:30.797815+0000","last_fullsized":"2023-10-19T14:20:30.797815+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:25:10.980817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.35","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.800120+0000","last_change":"2023-10-19T14:20:30.795581+0000","last_active":"2023-10-19T14:20:30.800120+0000","last_peered":"2023-10-19T14:20:30.800120+0000","last_clean":"2023-10-19T14:20:30.800120+0000","last_became_active":"2023-10-19T14:20:30.795262+0000","last_became_peered":"2023-10-19T14:20:30.795262+0000","last_unstale":"2023-10-19T14:20:30.800120+0000","last_undegraded":"2023-10-19T14:20:30.800120+0000","last_fullsized":"2023-10-19T14:20:30.800120+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:04:38.334593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.37","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.787262+0000","last_change":"2023-10-19T14:20:19.335860+0000","last_active":"2023-10-19T14:20:30.787262+0000","last_peered":"2023-10-19T14:20:30.787262+0000","last_clean":"2023-10-19T14:20:30.787262+0000","last_became_active":"2023-10-19T14:20:19.335522+0000","last_became_peered":"2023-10-19T14:20:19.335522+0000","last_unstale":"2023-10-19T14:20:30.787262+0000","last_undegraded":"2023-10-19T14:20:30.787262+0000","last_fullsized":"2023-10-19T14:20:30.787262+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:36:28.937721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.3a","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.782414+0000","last_change":"2023-10-19T14:20:30.788800+0000","last_active":"2023-10-19T14:20:31.782414+0000","last_peered":"2023-10-19T14:20:31.782414+0000","last_clean":"2023-10-19T14:20:31.782414+0000","last_became_active":"2023-10-19T14:20:30.788544+0000","last_became_peered":"2023-10-19T14:20:30.788544+0000","last_unstale":"2023-10-19T14:20:31.782414+0000","last_undegraded":"2023-10-19T14:20:31.782414+0000","last_fullsized":"2023-10-19T14:20:31.782414+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:26:11.781950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.38","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.800130+0000","last_change":"2023-10-19T14:20:19.347387+0000","last_active":"2023-10-19T14:20:30.800130+0000","last_peered":"2023-10-19T14:20:30.800130+0000","last_clean":"2023-10-19T14:20:30.800130+0000","last_became_active":"2023-10-19T14:20:19.347104+0000","last_became_peered":"2023-10-19T14:20:19.347104+0000","last_unstale":"2023-10-19T14:20:30.800130+0000","last_undegraded":"2023-10-19T14:20:30.800130+0000","last_fullsized":"2023-10-19T14:20:30.800130+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:13:49.536943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3b","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.798107+0000","last_change":"2023-10-19T14:20:30.795221+0000","last_active":"2023-10-19T14:20:30.798107+0000","last_peered":"2023-10-19T14:20:30.798107+0000","last_clean":"2023-10-19T14:20:30.798107+0000","last_became_active":"2023-10-19T14:20:30.794905+0000","last_became_peered":"2023-10-19T14:20:30.794905+0000","last_unstale":"2023-10-19T14:20:30.798107+0000","last_undegraded":"2023-10-19T14:20:30.798107+0000","last_fullsized":"2023-10-19T14:20:30.798107+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:33:45.656463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.39","version":"0'0","reported_seq":14,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.781557+0000","last_change":"2023-10-19T14:20:19.343249+0000","last_active":"2023-10-19T14:20:29.781557+0000","last_peered":"2023-10-19T14:20:29.781557+0000","last_clean":"2023-10-19T14:20:29.781557+0000","last_became_active":"2023-10-19T14:20:19.342989+0000","last_became_peered":"2023-10-19T14:20:19.342989+0000","last_unstale":"2023-10-19T14:20:29.781557+0000","last_undegraded":"2023-10-19T14:20:29.781557+0000","last_fullsized":"2023-10-19T14:20:29.781557+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:54:18.133180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.38","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.781967+0000","last_change":"2023-10-19T14:20:30.792568+0000","last_active":"2023-10-19T14:20:31.781967+0000","last_peered":"2023-10-19T14:20:31.781967+0000","last_clean":"2023-10-19T14:20:31.781967+0000","last_became_active":"2023-10-19T14:20:30.792070+0000","last_became_peered":"2023-10-19T14:20:30.792070+0000","last_unstale":"2023-10-19T14:20:31.781967+0000","last_undegraded":"2023-10-19T14:20:31.781967+0000","last_fullsized":"2023-10-19T14:20:31.781967+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:38:06.530226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3a","version":"0'0","reported_seq":14,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.786856+0000","last_change":"2023-10-19T14:20:19.344720+0000","last_active":"2023-10-19T14:20:29.786856+0000","last_peered":"2023-10-19T14:20:29.786856+0000","last_clean":"2023-10-19T14:20:29.786856+0000","last_became_active":"2023-10-19T14:20:19.344415+0000","last_became_peered":"2023-10-19T14:20:19.344415+0000","last_unstale":"2023-10-19T14:20:29.786856+0000","last_undegraded":"2023-10-19T14:20:29.786856+0000","last_fullsized":"2023-10-19T14:20:29.786856+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:01:45.199351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.39","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.784816+0000","last_change":"2023-10-19T14:20:30.792844+0000","last_active":"2023-10-19T14:20:31.784816+0000","last_peered":"2023-10-19T14:20:31.784816+0000","last_clean":"2023-10-19T14:20:31.784816+0000","last_became_active":"2023-10-19T14:20:30.792560+0000","last_became_peered":"2023-10-19T14:20:30.792560+0000","last_unstale":"2023-10-19T14:20:31.784816+0000","last_undegraded":"2023-10-19T14:20:31.784816+0000","last_fullsized":"2023-10-19T14:20:31.784816+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:36:43.318354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.3b","version":"0'0","reported_seq":14,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.782298+0000","last_change":"2023-10-19T14:20:19.344979+0000","last_active":"2023-10-19T14:20:29.782298+0000","last_peered":"2023-10-19T14:20:29.782298+0000","last_clean":"2023-10-19T14:20:29.782298+0000","last_became_active":"2023-10-19T14:20:19.344717+0000","last_became_peered":"2023-10-19T14:20:19.344717+0000","last_unstale":"2023-10-19T14:20:29.782298+0000","last_undegraded":"2023-10-19T14:20:29.782298+0000","last_fullsized":"2023-10-19T14:20:29.782298+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:14:31.826091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.3e","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.785876+0000","last_change":"2023-10-19T14:20:30.797158+0000","last_active":"2023-10-19T14:20:31.785876+0000","last_peered":"2023-10-19T14:20:31.785876+0000","last_clean":"2023-10-19T14:20:31.785876+0000","last_became_active":"2023-10-19T14:20:30.796872+0000","last_became_peered":"2023-10-19T14:20:30.796872+0000","last_unstale":"2023-10-19T14:20:31.785876+0000","last_undegraded":"2023-10-19T14:20:31.785876+0000","last_fullsized":"2023-10-19T14:20:31.785876+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T02:12:07.536009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.3c","version":"0'0","reported_seq":14,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.787311+0000","last_change":"2023-10-19T14:20:19.351919+0000","last_active":"2023-10-19T14:20:29.787311+0000","last_peered":"2023-10-19T14:20:29.787311+0000","last_clean":"2023-10-19T14:20:29.787311+0000","last_became_active":"2023-10-19T14:20:19.351634+0000","last_became_peered":"2023-10-19T14:20:19.351634+0000","last_unstale":"2023-10-19T14:20:29.787311+0000","last_undegraded":"2023-10-19T14:20:29.787311+0000","last_fullsized":"2023-10-19T14:20:29.787311+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:40:33.512260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.3f","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.793889+0000","last_change":"2023-10-19T14:20:30.791868+0000","last_active":"2023-10-19T14:20:30.793889+0000","last_peered":"2023-10-19T14:20:30.793889+0000","last_clean":"2023-10-19T14:20:30.793889+0000","last_became_active":"2023-10-19T14:20:30.791557+0000","last_became_peered":"2023-10-19T14:20:30.791557+0000","last_unstale":"2023-10-19T14:20:30.793889+0000","last_undegraded":"2023-10-19T14:20:30.793889+0000","last_fullsized":"2023-10-19T14:20:30.793889+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:44:39.222010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.3d","version":"0'0","reported_seq":14,"reported_epoch":225,"state":"active+clean","last_fresh":"2023-10-19T14:20:29.788515+0000","last_change":"2023-10-19T14:20:19.349659+0000","last_active":"2023-10-19T14:20:29.788515+0000","last_peered":"2023-10-19T14:20:29.788515+0000","last_clean":"2023-10-19T14:20:29.788515+0000","last_became_active":"2023-10-19T14:20:19.349362+0000","last_became_peered":"2023-10-19T14:20:19.349362+0000","last_unstale":"2023-10-19T14:20:29.788515+0000","last_undegraded":"2023-10-19T14:20:29.788515+0000","last_fullsized":"2023-10-19T14:20:29.788515+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:23:13.051759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.3c","version":"0'0","reported_seq":9,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.788732+0000","last_change":"2023-10-19T14:20:30.787587+0000","last_active":"2023-10-19T14:20:30.788732+0000","last_peered":"2023-10-19T14:20:30.788732+0000","last_clean":"2023-10-19T14:20:30.788732+0000","last_became_active":"2023-10-19T14:20:30.787318+0000","last_became_peered":"2023-10-19T14:20:30.787318+0000","last_unstale":"2023-10-19T14:20:30.788732+0000","last_undegraded":"2023-10-19T14:20:30.788732+0000","last_fullsized":"2023-10-19T14:20:30.788732+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:38:03.480323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.3e","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.785209+0000","last_change":"2023-10-19T14:20:19.346014+0000","last_active":"2023-10-19T14:20:30.785209+0000","last_peered":"2023-10-19T14:20:30.785209+0000","last_clean":"2023-10-19T14:20:30.785209+0000","last_became_active":"2023-10-19T14:20:19.345686+0000","last_became_peered":"2023-10-19T14:20:19.345686+0000","last_unstale":"2023-10-19T14:20:30.785209+0000","last_undegraded":"2023-10-19T14:20:30.785209+0000","last_fullsized":"2023-10-19T14:20:30.785209+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:29:26.735942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.3d","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2023-10-19T14:20:31.784998+0000","last_change":"2023-10-19T14:20:30.794749+0000","last_active":"2023-10-19T14:20:31.784998+0000","last_peered":"2023-10-19T14:20:31.784998+0000","last_clean":"2023-10-19T14:20:31.784998+0000","last_became_active":"2023-10-19T14:20:30.794474+0000","last_became_peered":"2023-10-19T14:20:30.794474+0000","last_unstale":"2023-10-19T14:20:31.784998+0000","last_undegraded":"2023-10-19T14:20:31.784998+0000","last_fullsized":"2023-10-19T14:20:31.784998+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:29.770501+0000","last_clean_scrub_stamp":"2023-10-19T14:20:29.770501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:36:27.153729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.3f","version":"0'0","reported_seq":15,"reported_epoch":226,"state":"active+clean","last_fresh":"2023-10-19T14:20:30.781865+0000","last_change":"2023-10-19T14:20:19.348564+0000","last_active":"2023-10-19T14:20:30.781865+0000","last_peered":"2023-10-19T14:20:30.781865+0000","last_clean":"2023-10-19T14:20:30.781865+0000","last_became_active":"2023-10-19T14:20:19.348281+0000","last_became_peered":"2023-10-19T14:20:19.348281+0000","last_unstale":"2023-10-19T14:20:30.781865+0000","last_undegraded":"2023-10-19T14:20:30.781865+0000","last_fullsized":"2023-10-19T14:20:30.781865+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":221,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:20:18.695654+0000","last_clean_scrub_stamp":"2023-10-19T14:20:18.695654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:15:21.019725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":30,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":29,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":28,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":27,"num_pg":32,"stat_sum":{"num_bytes":3666,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2210,"num_read_kb":4624,"num_write":589,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3186,"ondisk_log_size":3186,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576004,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48448,"kb_used_data":3376,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323392,"statfs":{"total":96636764160,"available":96587153408,"internally_reserved":0,"allocated":3457024,"data_stored":2051182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":46135755},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:20:09 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.56999999999999995,"15min":0.58499999999999996},"min":{"1min":0.35799999999999998,"5min":0.34100000000000003,"15min":0.32200000000000001},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.86199999999999999},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.60199999999999998,"15min":0.60999999999999999},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.312},"max":{"1min":0.77600000000000002,"5min":0.81899999999999995,"15min":0.86599999999999999},"last":0.50700000000000001}]},{"osd":1,"last update":"Thu Oct 19 14:20:09 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.59799999999999998,"15min":0.59999999999999998},"min":{"1min":0.46500000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.76800000000000002,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.59499999999999997,"15min":0.59799999999999998},"min":{"1min":0.373,"5min":0.373,"15min":0.33500000000000002},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.443}]},{"osd":2,"last update":"Thu Oct 19 14:20:09 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64900000000000002,"15min":0.64800000000000002},"min":{"1min":0.50900000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.85499999999999998},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.66600000000000004,"15min":0.66200000000000003},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.40100000000000002},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.46000000000000002}]},{"osd":3,"last update":"Thu Oct 19 14:20:09 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.63300000000000001,"15min":0.63700000000000001},"min":{"1min":0.51100000000000001,"5min":0.42099999999999999,"15min":0.36399999999999999},"max":{"1min":0.82699999999999996,"5min":0.84799999999999998,"15min":0.873},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.68799999999999994,"15min":0.68700000000000006},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":0.873,"5min":0.876,"15min":0.89600000000000002},"last":0.497}]},{"osd":4,"last update":"Thu Oct 19 14:20:09 2023","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.50600000000000001,"15min":0.49299999999999999},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.62,"5min":0.78100000000000003,"15min":0.83799999999999997},"last":0.311},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.51800000000000002,"15min":0.50600000000000001},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.71899999999999997,"5min":0.80200000000000005,"15min":0.81699999999999995},"last":0.36299999999999999}]},{"osd":5,"last update":"Thu Oct 19 14:20:09 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51400000000000001,"15min":0.50800000000000001},"min":{"1min":0.36799999999999999,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.65600000000000003,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.55600000000000005,"15min":0.54900000000000004},"min":{"1min":0.377,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.70999999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.42599999999999999}]},{"osd":6,"last update":"Thu Oct 19 14:20:09 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57699999999999996,"15min":0.57399999999999995},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.35999999999999999},"max":{"1min":0.73199999999999998,"5min":0.82999999999999996,"15min":0.85099999999999998},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.58099999999999996,"15min":0.56799999999999995},"min":{"1min":0.38800000000000001,"5min":0.35599999999999998,"15min":0.33500000000000002},"max":{"1min":0.78200000000000003,"5min":0.85199999999999998,"15min":0.85999999999999999},"last":0.45000000000000001}]}]},{"osd":6,"up_from":13,"seq":55834576004,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38152,"kb_used_data":2812,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333688,"statfs":{"total":96636764160,"available":96597696512,"internally_reserved":0,"allocated":2879488,"data_stored":1462392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:20:00 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.60799999999999998,"15min":0.61499999999999999},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.28799999999999998},"max":{"1min":0.81200000000000006,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.59399999999999997,"15min":0.61499999999999999},"min":{"1min":0.40699999999999997,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.436}]},{"osd":1,"last update":"Thu Oct 19 14:20:00 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.621,"15min":0.621},"min":{"1min":0.52300000000000002,"5min":0.38200000000000001,"15min":0.35799999999999998},"max":{"1min":0.79500000000000004,"5min":0.80200000000000005,"15min":1.165},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.624,"15min":0.63500000000000001},"min":{"1min":0.503,"5min":0.439,"15min":0.36899999999999999},"max":{"1min":0.81599999999999995,"5min":0.83699999999999997,"15min":0.91300000000000003},"last":0.33700000000000002}]},{"osd":2,"last update":"Thu Oct 19 14:20:00 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.60099999999999998,"15min":0.60899999999999999},"min":{"1min":0.442,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.873},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.626,"15min":0.629},"min":{"1min":0.45000000000000001,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.93100000000000005},"last":0.56399999999999995}]},{"osd":3,"last update":"Thu Oct 19 14:20:00 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.65500000000000003,"15min":0.65800000000000003},"min":{"1min":0.53000000000000003,"5min":0.372,"15min":0.34300000000000003},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.65600000000000003,"15min":0.66100000000000003},"min":{"1min":0.48299999999999998,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.84399999999999997,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.622}]},{"osd":4,"last update":"Thu Oct 19 14:20:00 2023","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.47999999999999998,"15min":0.48199999999999998},"min":{"1min":0.34499999999999997,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.79600000000000004},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.502,"15min":0.50800000000000001},"min":{"1min":0.35399999999999998,"5min":0.34300000000000003,"15min":0.30399999999999999},"max":{"1min":0.74199999999999999,"5min":0.82199999999999995,"15min":0.85699999999999998},"last":0.69999999999999996}]},{"osd":5,"last update":"Thu Oct 19 14:20:00 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.5,"15min":0.5},"min":{"1min":0.35599999999999998,"5min":0.32700000000000001,"15min":0.32500000000000001},"max":{"1min":0.747,"5min":1.115,"15min":1.115},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.55100000000000005,"15min":0.54300000000000004},"min":{"1min":0.40000000000000002,"5min":0.35799999999999998,"15min":0.32400000000000001},"max":{"1min":0.78700000000000003,"5min":0.86499999999999999,"15min":0.878},"last":0.749}]},{"osd":7,"last update":"Thu Oct 19 14:20:00 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.55000000000000004,"15min":0.54600000000000004},"min":{"1min":0.39900000000000002,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.77600000000000002,"5min":0.86399999999999999,"15min":0.90100000000000002},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.55700000000000005,"15min":0.55200000000000005},"min":{"1min":0.39400000000000002,"5min":0.38200000000000001,"15min":0.36199999999999999},"max":{"1min":0.85099999999999998,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.66600000000000004}]}]},{"osd":5,"up_from":13,"seq":55834576004,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47864,"kb_used_data":2796,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323976,"statfs":{"total":96636764160,"available":96587751424,"internally_reserved":0,"allocated":2863104,"data_stored":1458942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:19:46 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.59899999999999998,"15min":0.621},"min":{"1min":0.44900000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.79700000000000004,"5min":0.82599999999999996,"15min":0.86399999999999999},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.63200000000000001,"15min":0.64500000000000002},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.82499999999999996,"5min":0.85899999999999999,"15min":0.85999999999999999},"last":0.34599999999999997}]},{"osd":1,"last update":"Thu Oct 19 14:19:46 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.61799999999999999,"15min":0.63500000000000001},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.78800000000000003,"5min":0.83699999999999997,"15min":0.877},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.64100000000000001,"15min":0.64500000000000002},"min":{"1min":0.47999999999999998,"5min":0.44700000000000001,"15min":0.41199999999999998},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.88500000000000001},"last":0.504}]},{"osd":2,"last update":"Thu Oct 19 14:19:46 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.61499999999999999,"15min":0.627},"min":{"1min":0.44800000000000001,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.81299999999999994,"5min":0.83799999999999997,"15min":0.85099999999999998},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.65200000000000002,"15min":0.66100000000000003},"min":{"1min":0.52000000000000002,"5min":0.49399999999999999,"15min":0.47099999999999997},"max":{"1min":0.79500000000000004,"5min":0.83299999999999996,"15min":0.90800000000000003},"last":0.45000000000000001}]},{"osd":3,"last update":"Thu Oct 19 14:19:46 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.63900000000000001,"15min":0.64700000000000002},"min":{"1min":0.51100000000000001,"5min":0.443,"15min":0.443},"max":{"1min":0.84399999999999997,"5min":0.86099999999999999,"15min":0.90100000000000002},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.66500000000000004,"15min":0.67400000000000004},"min":{"1min":0.53300000000000003,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.84899999999999998,"5min":0.876,"15min":0.90900000000000003},"last":0.436}]},{"osd":4,"last update":"Thu Oct 19 14:19:46 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.497,"15min":0.48699999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.32500000000000001},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.371},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50900000000000001,"15min":0.50600000000000001},"min":{"1min":0.35499999999999998,"5min":0.318,"15min":0.28599999999999998},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.82899999999999996},"last":0.55600000000000005}]},{"osd":6,"last update":"Thu Oct 19 14:19:46 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51200000000000001,"15min":0.497},"min":{"1min":0.37,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.88800000000000001},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.54100000000000004,"15min":0.53900000000000003},"min":{"1min":0.39100000000000001,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.77900000000000003,"5min":0.78000000000000003,"15min":0.88100000000000001},"last":0.65100000000000002}]},{"osd":7,"last update":"Thu Oct 19 14:19:46 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.51500000000000001,"15min":0.52000000000000002},"min":{"1min":0.41799999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.749,"5min":0.749,"15min":0.82899999999999996},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.58999999999999997,"15min":0.57499999999999996},"min":{"1min":0.47099999999999997,"5min":0.38400000000000001,"15min":0.34300000000000003},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.61799999999999999}]}]},{"osd":4,"up_from":13,"seq":55834576004,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47856,"kb_used_data":2780,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323984,"statfs":{"total":96636764160,"available":96587759616,"internally_reserved":0,"allocated":2846720,"data_stored":1457876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.55500000000000005,"15min":0.57599999999999996},"min":{"1min":0.35499999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.80800000000000005,"5min":0.81799999999999995,"15min":0.879},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.58499999999999996,"15min":0.59799999999999998},"min":{"1min":0.499,"5min":0.31,"15min":0.31},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.873},"last":0.51000000000000001}]},{"osd":1,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.61499999999999999,"15min":0.64100000000000001},"min":{"1min":0.504,"5min":0.372,"15min":0.32600000000000001},"max":{"1min":0.80400000000000005,"5min":0.80700000000000005,"15min":0.92700000000000005},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.623,"15min":0.64400000000000002},"min":{"1min":0.443,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":1.0940000000000001},"last":0.56599999999999995}]},{"osd":2,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.63400000000000001,"15min":0.65000000000000002},"min":{"1min":0.46400000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.64800000000000002,"15min":0.66900000000000004},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":1.008},"last":0.48499999999999999}]},{"osd":3,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.61299999999999999,"15min":0.63400000000000001},"min":{"1min":0.47399999999999998,"5min":0.37,"15min":0.37},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.91900000000000004},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.66600000000000004,"15min":0.66700000000000004},"min":{"1min":0.54000000000000004,"5min":0.45400000000000001,"15min":0.437},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.57899999999999996}]},{"osd":5,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.48299999999999998,"15min":0.47099999999999997},"min":{"1min":0.34000000000000002,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.504,"15min":0.49199999999999999},"min":{"1min":0.34300000000000003,"5min":0.33700000000000002,"15min":0.29199999999999998},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.38700000000000001}]},{"osd":6,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.496,"15min":0.48599999999999999},"min":{"1min":0.34799999999999998,"5min":0.33900000000000002,"15min":0.28199999999999997},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.496},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.53700000000000003},"min":{"1min":0.39100000000000001,"5min":0.375,"15min":0.33300000000000002},"max":{"1min":0.81299999999999994,"5min":0.84799999999999998,"15min":1.2969999999999999},"last":0.52900000000000003}]},{"osd":7,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.55100000000000005,"15min":0.54100000000000004},"min":{"1min":0.36499999999999999,"5min":0.35699999999999998,"15min":0.32600000000000001},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60499999999999998,"15min":0.59199999999999997},"min":{"1min":0.47099999999999997,"5min":0.32100000000000001,"15min":0.318},"max":{"1min":0.80400000000000005,"5min":0.88600000000000001,"15min":1.032},"last":0.60199999999999998}]}]},{"osd":3,"up_from":13,"seq":55834576003,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56180,"kb_used_data":2788,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315660,"statfs":{"total":96636764160,"available":96579235840,"internally_reserved":0,"allocated":2854912,"data_stored":1457352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54655432},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:19:48 2023","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42299999999999999},"min":{"1min":0.316,"5min":0.253,"15min":0.253},"max":{"1min":0.60599999999999998,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.42599999999999999,"15min":0.432},"min":{"1min":0.29399999999999998,"5min":0.219,"15min":0.219},"max":{"1min":0.57599999999999996,"5min":0.752,"15min":0.752},"last":0.32800000000000001}]},{"osd":1,"last update":"Thu Oct 19 14:19:48 2023","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.439,"15min":0.433},"min":{"1min":0.29599999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.56599999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.374},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.30099999999999999,"5min":0.29399999999999998,"15min":0.26400000000000001},"max":{"1min":0.56599999999999995,"5min":0.73099999999999998,"15min":0.81799999999999995},"last":0.39000000000000001}]},{"osd":2,"last update":"Thu Oct 19 14:19:48 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.47699999999999998,"15min":0.48599999999999999},"min":{"1min":0.39600000000000002,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.64300000000000002,"5min":0.64600000000000002,"15min":0.73099999999999998},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.499,"15min":0.50800000000000001},"min":{"1min":0.33500000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.66800000000000004,"5min":0.67000000000000004,"15min":0.83699999999999997},"last":0.36099999999999999}]},{"osd":4,"last update":"Thu Oct 19 14:19:48 2023","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.71099999999999997,"15min":0.70899999999999996},"min":{"1min":0.45000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.96099999999999997,"5min":1.018,"15min":1.157},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.57999999999999996,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.93100000000000005,"5min":0.996,"15min":1.173},"last":0.56799999999999995}]},{"osd":5,"last update":"Thu Oct 19 14:19:48 2023","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.69499999999999995,"15min":0.69199999999999995},"min":{"1min":0.55000000000000004,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.93100000000000005,"5min":0.997,"15min":1.036},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.69799999999999995,"15min":0.69299999999999995},"min":{"1min":0.47399999999999998,"5min":0.42999999999999999,"15min":0.42599999999999999},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":1.105},"last":0.502}]},{"osd":6,"last update":"Thu Oct 19 14:19:48 2023","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.68500000000000005,"15min":0.69499999999999995},"min":{"1min":0.51900000000000002,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.99099999999999999,"5min":1.018,"15min":1.0669999999999999},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.55800000000000005,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.88800000000000001,"5min":0.91500000000000004,"15min":1.079},"last":0.53500000000000003}]},{"osd":7,"last update":"Thu Oct 19 14:19:48 2023","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.70799999999999996,"15min":0.70499999999999996},"min":{"1min":0.51600000000000001,"5min":0.434,"15min":0.434},"max":{"1min":0.94899999999999995,"5min":0.96899999999999997,"15min":1.085},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.73199999999999998,"15min":0.73599999999999999},"min":{"1min":0.57099999999999995,"5min":0.51900000000000002,"15min":0.51300000000000001},"max":{"1min":0.96499999999999997,"5min":0.97199999999999998,"15min":1.1739999999999999},"last":0.51700000000000002}]}]},{"osd":2,"up_from":13,"seq":55834576005,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64492,"kb_used_data":2784,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94307348,"statfs":{"total":96636764160,"available":96570724352,"internally_reserved":0,"allocated":2850816,"data_stored":1457910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":63175110},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.39900000000000002,"15min":0.40200000000000002},"min":{"1min":0.28199999999999997,"5min":0.218,"15min":0.218},"max":{"1min":0.623,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.254},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.44500000000000001,"15min":0.441},"min":{"1min":0.34100000000000003,"5min":0.249,"15min":0.223},"max":{"1min":0.628,"5min":0.71399999999999997,"15min":0.73599999999999999},"last":0.38300000000000001}]},{"osd":1,"last update":"Thu Oct 19 14:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.40500000000000003,"15min":0.41599999999999998},"min":{"1min":0.28999999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.48999999999999999,"5min":0.61699999999999999,"15min":0.65400000000000003},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.47999999999999998,"15min":0.47099999999999997},"min":{"1min":0.33000000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":1.742,"5min":1.742,"15min":1.742},"last":0.32900000000000001}]},{"osd":3,"last update":"Thu Oct 19 14:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.48199999999999998,"15min":0.48399999999999999},"min":{"1min":0.313,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.56999999999999995,"5min":0.69299999999999995,"15min":0.74399999999999999},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.499,"15min":0.51100000000000001},"min":{"1min":0.38200000000000001,"5min":0.32000000000000001,"15min":0.315},"max":{"1min":0.60599999999999998,"5min":0.67600000000000005,"15min":0.83599999999999997},"last":0.36499999999999999}]},{"osd":4,"last update":"Thu Oct 19 14:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.66900000000000004,"15min":0.67000000000000004},"min":{"1min":0.44900000000000001,"5min":0.39600000000000002,"15min":0.39500000000000002},"max":{"1min":0.88600000000000001,"5min":0.93899999999999995,"15min":0.998},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.67300000000000004,"15min":0.68200000000000005},"min":{"1min":0.51700000000000002,"5min":0.38900000000000001,"15min":0.35199999999999998},"max":{"1min":0.79500000000000004,"5min":0.97099999999999997,"15min":1.0289999999999999},"last":0.65900000000000003}]},{"osd":5,"last update":"Thu Oct 19 14:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.70499999999999996,"15min":0.70199999999999996},"min":{"1min":0.53300000000000003,"5min":0.45100000000000001,"15min":0.40600000000000003},"max":{"1min":1.4330000000000001,"5min":1.4330000000000001,"15min":1.4330000000000001},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.68400000000000005,"15min":0.68899999999999995},"min":{"1min":0.48199999999999998,"5min":0.45100000000000001,"15min":0.39300000000000002},"max":{"1min":1.595,"5min":1.595,"15min":1.595},"last":0.72499999999999998}]},{"osd":6,"last update":"Thu Oct 19 14:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.70299999999999996,"15min":0.71199999999999997},"min":{"1min":0.46500000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.85999999999999999,"5min":0.92800000000000005,"15min":1.0189999999999999},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.72899999999999998,"15min":0.72599999999999998},"min":{"1min":0.47899999999999998,"5min":0.433,"15min":0.40600000000000003},"max":{"1min":1.4930000000000001,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":0.63900000000000001}]},{"osd":7,"last update":"Thu Oct 19 14:19:36 2023","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.71999999999999997,"15min":0.72899999999999998},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.41799999999999998},"max":{"1min":0.97999999999999998,"5min":1.0129999999999999,"15min":1.0289999999999999},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.73899999999999999,"15min":0.73499999999999999},"min":{"1min":0.503,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.53,"5min":1.53,"15min":1.53},"last":0.91100000000000003}]}]},{"osd":1,"up_from":13,"seq":55834576004,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47844,"kb_used_data":2772,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323996,"statfs":{"total":96636764160,"available":96587771904,"internally_reserved":0,"allocated":2838528,"data_stored":1457204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:19:35 2023","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.40300000000000002,"15min":0.40699999999999997},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.23100000000000001},"max":{"1min":0.48799999999999999,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.45400000000000001,"15min":0.45700000000000002},"min":{"1min":0.28299999999999997,"5min":0.27500000000000002,"15min":0.251},"max":{"1min":0.61399999999999999,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.26800000000000002}]},{"osd":2,"last update":"Thu Oct 19 14:19:35 2023","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.44500000000000001,"15min":0.44600000000000001},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.58599999999999997,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.251},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.495,"15min":0.49199999999999999},"min":{"1min":0.34399999999999997,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.64400000000000002,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.33700000000000002}]},{"osd":3,"last update":"Thu Oct 19 14:19:35 2023","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.51600000000000001,"15min":0.50800000000000001},"min":{"1min":0.38400000000000001,"5min":0.36299999999999999,"15min":0.29499999999999998},"max":{"1min":0.64400000000000002,"5min":0.76000000000000001,"15min":0.78900000000000003},"last":0.31},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.44600000000000001,"15min":0.44900000000000001},"min":{"1min":0.32000000000000001,"5min":0.315,"15min":0.26100000000000001},"max":{"1min":0.65200000000000002,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.38}]},{"osd":4,"last update":"Thu Oct 19 14:19:35 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.67000000000000004,"15min":0.67100000000000004},"min":{"1min":0.39900000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.84399999999999997,"5min":1.042,"15min":1.042},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"min":{"1min":0.45100000000000001,"5min":0.41299999999999998,"15min":0.32300000000000001},"max":{"1min":0.88800000000000001,"5min":0.93500000000000005,"15min":1.1659999999999999},"last":0.747}]},{"osd":5,"last update":"Thu Oct 19 14:19:35 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.69199999999999995,"15min":0.68700000000000006},"min":{"1min":0.45900000000000002,"5min":0.42799999999999999,"15min":0.42499999999999999},"max":{"1min":0.83099999999999996,"5min":1.0589999999999999,"15min":1.3500000000000001},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.71199999999999997,"15min":0.70799999999999996},"min":{"1min":0.44,"5min":0.44,"15min":0.40500000000000003},"max":{"1min":0.78600000000000003,"5min":1.0149999999999999,"15min":1.121},"last":0.61699999999999999}]},{"osd":6,"last update":"Thu Oct 19 14:19:35 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.67300000000000004,"15min":0.68300000000000005},"min":{"1min":0.42499999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.77200000000000002,"5min":1.0129999999999999,"15min":1.2310000000000001},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.71999999999999997,"15min":0.71499999999999997},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.86399999999999999,"5min":1.107,"15min":1.274},"last":0.67900000000000005}]},{"osd":7,"last update":"Thu Oct 19 14:19:35 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.72799999999999998,"15min":0.72399999999999998},"min":{"1min":0.46999999999999997,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.78600000000000003,"5min":1.075,"15min":1.075},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.47199999999999998,"5min":0.46100000000000002,"15min":0.44400000000000001},"max":{"1min":0.79600000000000004,"5min":1.0169999999999999,"15min":1.117},"last":0.72799999999999998}]}]},{"osd":0,"up_from":13,"seq":55834576005,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60900,"kb_used_data":3352,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310940,"statfs":{"total":96636764160,"available":96574402560,"internally_reserved":0,"allocated":3432448,"data_stored":2047043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.40699999999999997,"15min":0.41299999999999998},"min":{"1min":0.29599999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.56299999999999994,"5min":0.61499999999999999,"15min":0.624},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.441,"15min":0.46100000000000002},"min":{"1min":0.26800000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.56799999999999995,"5min":0.70899999999999996,"15min":0.81100000000000005},"last":0.33000000000000002}]},{"osd":2,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.496,"15min":0.48399999999999999},"min":{"1min":0.39300000000000002,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.71499999999999997,"5min":0.754,"15min":0.754},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.44800000000000001},"min":{"1min":0.32700000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.60199999999999998,"5min":0.63100000000000001,"15min":0.69099999999999995},"last":0.38800000000000001}]},{"osd":3,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.432,"15min":0.44},"min":{"1min":0.26900000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.58799999999999997,"5min":0.61099999999999999,"15min":0.66800000000000004},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.501,"5min":0.5,"15min":0.51200000000000001},"min":{"1min":0.35199999999999998,"5min":0.254,"15min":0.254},"max":{"1min":0.78100000000000003,"5min":0.79800000000000004,"15min":0.84599999999999997},"last":0.37}]},{"osd":4,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.64800000000000002,"15min":0.67700000000000005},"min":{"1min":0.48799999999999999,"5min":0.44600000000000001,"15min":0.44},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":1.1890000000000001},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.65800000000000003,"15min":0.68200000000000005},"min":{"1min":0.47499999999999998,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.751,"5min":0.95399999999999996,"15min":1.2150000000000001},"last":0.57399999999999995}]},{"osd":5,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.68999999999999995,"15min":0.72199999999999998},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":1.1579999999999999},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.69499999999999995,"15min":0.71999999999999997},"min":{"1min":0.57799999999999996,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.85399999999999998,"5min":0.98299999999999998,"15min":1.2310000000000001},"last":0.65700000000000003}]},{"osd":6,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.68700000000000006,"15min":0.70599999999999996},"min":{"1min":0.47799999999999998,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":1.1739999999999999},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.70399999999999996},"min":{"1min":0.54100000000000004,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.88900000000000001,"5min":0.97799999999999998,"15min":1.0189999999999999},"last":0.63400000000000001}]},{"osd":7,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.68600000000000005,"15min":0.71599999999999997},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":1.1180000000000001},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.68200000000000005,"15min":0.70399999999999996},"min":{"1min":0.53600000000000003,"5min":0.46899999999999997,"15min":0.45100000000000001},"max":{"1min":0.90600000000000003,"5min":0.93799999999999994,"15min":1.0349999999999999},"last":0.61899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-19T14:20:34.183 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-19T14:20:34.184 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T14:20:34.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:34.319+0000 7f33f319e640 1 -- 172.21.15.57:0/458108572 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f33d4003b30 msgr2=0x7f33d4003f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:34.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:34.319+0000 7f33f319e640 1 --2- 172.21.15.57:0/458108572 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f33d4003b30 0x7f33d4003f30 secure :-1 s=READY pgs=2423 cs=0 l=1 rev1=1 crypto rx=0x7f33e4001800 tx=0x7f33d40057f0 comp rx=0 tx=0).stop 2023-10-19T14:20:34.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:34.319+0000 7f33f299d640 1 -- 172.21.15.57:0/458108572 reap_dead start 2023-10-19T14:20:34.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:34.319+0000 7f33f319e640 1 -- 172.21.15.57:0/458108572 shutdown_connections 2023-10-19T14:20:34.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:34.319+0000 7f33f319e640 1 -- 172.21.15.57:0/458108572 >> 172.21.15.57:0/458108572 conn(0x7f33ec0d0c20 msgr2=0x7f33ec0d1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:34.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:34.319+0000 7f33f319e640 1 -- 172.21.15.57:0/458108572 shutdown_connections 2023-10-19T14:20:34.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:34.319+0000 7f33f319e640 1 -- 172.21.15.57:0/458108572 wait complete. 2023-10-19T14:20:34.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:34.321+0000 7f33f319e640 1 Processor -- start 2023-10-19T14:20:34.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:34.321+0000 7f33f319e640 1 -- start start 2023-10-19T14:20:34.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:34.321+0000 7f33f319e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f33d4003b30 0x7f33ec0ca2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:34.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:34.321+0000 7f33f319e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f33ec1e4d90 0x7f33ec154aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:34.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:34.321+0000 7f33f319e640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f33ec1e5210 0x7f33ec157310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:34.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:34.321+0000 7f33f319e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33ec0ca830 con 0x7f33ec1e4d90 2023-10-19T14:20:34.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:34.321+0000 7f33f319e640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33ec1554f0 con 0x7f33d4003b30 2023-10-19T14:20:34.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:34.321+0000 7f33f319e640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33ec1557d0 con 0x7f33ec1e5210 2023-10-19T14:20:34.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:34.321+0000 7f33f199b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f33ec1e4d90 0x7f33ec154aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:34.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:34.321+0000 7f33f199b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f33ec1e4d90 0x7f33ec154aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:58924/0 (socket says 172.21.15.57:58924) 2023-10-19T14:20:34.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:34.321+0000 7f33f199b640 1 -- 172.21.15.57:0/742854709 learned_addr learned my addr 172.21.15.57:0/742854709 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:34.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:34.321+0000 7f33f219c640 1 --2- 172.21.15.57:0/742854709 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f33d4003b30 0x7f33ec0ca2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:34.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:34.321+0000 7f33f299d640 1 --2- 172.21.15.57:0/742854709 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f33ec1e5210 0x7f33ec157310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:34.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:34.321+0000 7f33f199b640 1 -- 172.21.15.57:0/742854709 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f33ec1e5210 msgr2=0x7f33ec157310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:34.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:34.321+0000 7f33f199b640 1 --2- 172.21.15.57:0/742854709 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f33ec1e5210 0x7f33ec157310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:34.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:34.321+0000 7f33f199b640 1 -- 172.21.15.57:0/742854709 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f33d4003b30 msgr2=0x7f33ec0ca2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:34.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:34.321+0000 7f33f199b640 1 --2- 172.21.15.57:0/742854709 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f33d4003b30 0x7f33ec0ca2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:34.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:34.321+0000 7f33f199b640 1 -- 172.21.15.57:0/742854709 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33ec155ab0 con 0x7f33ec1e4d90 2023-10-19T14:20:34.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:34.322+0000 7f33f219c640 1 --2- 172.21.15.57:0/742854709 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f33d4003b30 0x7f33ec0ca2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:20:34.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:34.322+0000 7f33f199b640 1 --2- 172.21.15.57:0/742854709 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f33ec1e4d90 0x7f33ec154aa0 secure :-1 s=READY pgs=2424 cs=0 l=1 rev1=1 crypto rx=0x7f33e0006680 tx=0x7f33e00066b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:34.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:34.322+0000 7f33eaffd640 1 -- 172.21.15.57:0/742854709 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33e0011900 con 0x7f33ec1e4d90 2023-10-19T14:20:34.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:34.322+0000 7f33eaffd640 1 -- 172.21.15.57:0/742854709 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f33e0011a60 con 0x7f33ec1e4d90 2023-10-19T14:20:34.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:34.322+0000 7f33eaffd640 1 -- 172.21.15.57:0/742854709 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33e0011e30 con 0x7f33ec1e4d90 2023-10-19T14:20:34.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:34.322+0000 7f33f319e640 1 -- 172.21.15.57:0/742854709 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33ec155d90 con 0x7f33ec1e4d90 2023-10-19T14:20:34.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:34.322+0000 7f33f319e640 1 -- 172.21.15.57:0/742854709 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33ec154710 con 0x7f33ec1e4d90 2023-10-19T14:20:34.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:34.323+0000 7f33f319e640 1 -- 172.21.15.57:0/742854709 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33ec066960 con 0x7f33ec1e4d90 2023-10-19T14:20:34.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:34.323+0000 7f33eaffd640 1 -- 172.21.15.57:0/742854709 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f33e0010420 con 0x7f33ec1e4d90 2023-10-19T14:20:34.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:34.324+0000 7f33eaffd640 1 --2- 172.21.15.57:0/742854709 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f33d806af60 0x7f33d806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:34.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:34.324+0000 7f33eaffd640 1 -- 172.21.15.57:0/742854709 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 6921+0+0 (secure 0 0 0) 0x7f33e008af60 con 0x7f33ec1e4d90 2023-10-19T14:20:34.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:34.324+0000 7f33f219c640 1 --2- 172.21.15.57:0/742854709 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f33d806af60 0x7f33d806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:34.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:34.324+0000 7f33f219c640 1 --2- 172.21.15.57:0/742854709 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f33d806af60 0x7f33d806d420 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f33e4034000 tx=0x7f33e4039040 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:34.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:34.329+0000 7f33eaffd640 1 -- 172.21.15.57:0/742854709 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f33e0055930 con 0x7f33ec1e4d90 2023-10-19T14:20:34.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:34.543+0000 7f33f319e640 1 -- 172.21.15.57:0/742854709 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f33ec1562b0 con 0x7f33ec1e4d90 2023-10-19T14:20:34.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:34.544+0000 7f33eaffd640 1 -- 172.21.15.57:0/742854709 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+6152 (secure 0 0 0) 0x7f33ec1562b0 con 0x7f33ec1e4d90 2023-10-19T14:20:34.546 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:20:34.547 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":94,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9248,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/1249807745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":1249807745},{"type":"v1","addr":"172.21.15.179:6836","nonce":1249807745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87},{"gid":10513,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/1406825818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":1406825818},{"type":"v1","addr":"172.21.15.179:6837","nonce":1406825818}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":86}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T14:20:19.077396+0000","modified":"2023-10-19T14:20:21.072483+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11718},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11718":{"gid":11718,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.57:6835/1529368280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1529368280},{"type":"v1","addr":"172.21.15.57:6835","nonce":1529368280}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13},{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T14:20:30.145135+0000","modified":"2023-10-19T14:20:32.141937+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11715},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11715":{"gid":11715,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":5,"addr":"172.21.15.57:6834/2467807803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2467807803},{"type":"v1","addr":"172.21.15.57:6834","nonce":2467807803}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2023-10-19T14:20:34.548 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 94 2023-10-19T14:20:34.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:34.545+0000 7f33f319e640 1 -- 172.21.15.57:0/742854709 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f33d806af60 msgr2=0x7f33d806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:34.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:34.545+0000 7f33f319e640 1 --2- 172.21.15.57:0/742854709 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f33d806af60 0x7f33d806d420 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f33e4034000 tx=0x7f33e4039040 comp rx=0 tx=0).stop 2023-10-19T14:20:34.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:34.545+0000 7f33f319e640 1 -- 172.21.15.57:0/742854709 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f33ec1e4d90 msgr2=0x7f33ec154aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:34.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:34.545+0000 7f33f319e640 1 --2- 172.21.15.57:0/742854709 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f33ec1e4d90 0x7f33ec154aa0 secure :-1 s=READY pgs=2424 cs=0 l=1 rev1=1 crypto rx=0x7f33e0006680 tx=0x7f33e00066b0 comp rx=0 tx=0).stop 2023-10-19T14:20:34.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:34.546+0000 7f33f319e640 1 -- 172.21.15.57:0/742854709 shutdown_connections 2023-10-19T14:20:34.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:34.546+0000 7f33f319e640 1 --2- 172.21.15.57:0/742854709 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f33d806af60 0x7f33d806d420 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:34.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:34.546+0000 7f33f319e640 1 --2- 172.21.15.57:0/742854709 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f33ec1e5210 0x7f33ec157310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:34.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:34.546+0000 7f33f319e640 1 --2- 172.21.15.57:0/742854709 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f33ec1e4d90 0x7f33ec154aa0 unknown :-1 s=CLOSED pgs=2424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:34.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:34.546+0000 7f33f319e640 1 --2- 172.21.15.57:0/742854709 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f33d4003b30 0x7f33ec0ca2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:34.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:34.546+0000 7f33f319e640 1 -- 172.21.15.57:0/742854709 >> 172.21.15.57:0/742854709 conn(0x7f33ec0d0c20 msgr2=0x7f33ec0c5010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:34.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:34.546+0000 7f33f319e640 1 -- 172.21.15.57:0/742854709 shutdown_connections 2023-10-19T14:20:34.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:34.546+0000 7f33f319e640 1 -- 172.21.15.57:0/742854709 wait complete. 2023-10-19T14:20:34.961 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 94, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-19T14:20:30.145135+0000', 'modified': '2023-10-19T14:20:32.141937+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 11715}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_11715': {'gid': 11715, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.57:6834/2467807803', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 2467807803}, {'type': 'v1', 'addr': '172.21.15.57:6834', 'nonce': 2467807803}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [30], 'metadata_pool': 29, 'enabled': True, 'fs_name': 'backup_fs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-19T14:20:34.961 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-19T14:20:34.962 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-10-19T14:20:35.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.074+0000 7ff8a4ad7640 1 -- 172.21.15.57:0/1657136540 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff8a0070350 msgr2=0x7ff8a0070780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:35.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.074+0000 7ff8a4ad7640 1 --2- 172.21.15.57:0/1657136540 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff8a0070350 0x7ff8a0070780 secure :-1 s=READY pgs=2426 cs=0 l=1 rev1=1 crypto rx=0x7ff8a00cc560 tx=0x7ff89002c2f0 comp rx=0 tx=0).stop 2023-10-19T14:20:35.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.074+0000 7ff8a4ad7640 1 -- 172.21.15.57:0/1657136540 shutdown_connections 2023-10-19T14:20:35.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.074+0000 7ff8a4ad7640 1 --2- 172.21.15.57:0/1657136540 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff8a0070350 0x7ff8a0070780 unknown :-1 s=CLOSED pgs=2426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:35.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.074+0000 7ff8a4ad7640 1 --2- 172.21.15.57:0/1657136540 >> v2:172.21.15.179:3300/0 conn(0x7ff8a00cb980 0x7ff8a013db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:35.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.074+0000 7ff8a4ad7640 1 -- 172.21.15.57:0/1657136540 >> 172.21.15.57:0/1657136540 conn(0x7ff8a0069930 msgr2=0x7ff8a0069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:35.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.074+0000 7ff8a4ad7640 1 -- 172.21.15.57:0/1657136540 shutdown_connections 2023-10-19T14:20:35.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.074+0000 7ff8a4ad7640 1 -- 172.21.15.57:0/1657136540 wait complete. 2023-10-19T14:20:35.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.075+0000 7ff8a4ad7640 1 Processor -- start 2023-10-19T14:20:35.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.075+0000 7ff8a4ad7640 1 -- start start 2023-10-19T14:20:35.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.076+0000 7ff8a4ad7640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff8a0070350 0x7ff8a013bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:35.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.076+0000 7ff8a4ad7640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff8a00cb980 0x7ff8a013cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:35.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.076+0000 7ff8a4ad7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff8a01f1d90 0x7ff8a01363a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:35.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.076+0000 7ff8a4ad7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a0133a30 con 0x7ff8a01f1d90 2023-10-19T14:20:35.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.076+0000 7ff8a4ad7640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a01f2220 con 0x7ff8a0070350 2023-10-19T14:20:35.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.076+0000 7ff8a4ad7640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a0133750 con 0x7ff8a00cb980 2023-10-19T14:20:35.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.076+0000 7ff89ed9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff8a01f1d90 0x7ff8a01363a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:35.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.076+0000 7ff89ed9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff8a01f1d90 0x7ff8a01363a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:58942/0 (socket says 172.21.15.57:58942) 2023-10-19T14:20:35.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.076+0000 7ff89ed9d640 1 -- 172.21.15.57:0/1442699596 learned_addr learned my addr 172.21.15.57:0/1442699596 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:35.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.076+0000 7ff89e59c640 1 --2- 172.21.15.57:0/1442699596 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff8a0070350 0x7ff8a013bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:35.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.076+0000 7ff89dd9b640 1 --2- 172.21.15.57:0/1442699596 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff8a00cb980 0x7ff8a013cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:35.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.076+0000 7ff89ed9d640 1 -- 172.21.15.57:0/1442699596 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff8a00cb980 msgr2=0x7ff8a013cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:35.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.076+0000 7ff89ed9d640 1 --2- 172.21.15.57:0/1442699596 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff8a00cb980 0x7ff8a013cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:35.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.076+0000 7ff89ed9d640 1 -- 172.21.15.57:0/1442699596 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff8a0070350 msgr2=0x7ff8a013bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:35.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.076+0000 7ff89ed9d640 1 --2- 172.21.15.57:0/1442699596 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff8a0070350 0x7ff8a013bad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:35.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.076+0000 7ff89ed9d640 1 -- 172.21.15.57:0/1442699596 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff890009900 con 0x7ff8a01f1d90 2023-10-19T14:20:35.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.076+0000 7ff89dd9b640 1 --2- 172.21.15.57:0/1442699596 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff8a00cb980 0x7ff8a013cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:20:35.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.077+0000 7ff89ed9d640 1 --2- 172.21.15.57:0/1442699596 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff8a01f1d90 0x7ff8a01363a0 secure :-1 s=READY pgs=2427 cs=0 l=1 rev1=1 crypto rx=0x7ff890037040 tx=0x7ff890035e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:35.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.077+0000 7ff896ffd640 1 -- 172.21.15.57:0/1442699596 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff89002cea0 con 0x7ff8a01f1d90 2023-10-19T14:20:35.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.077+0000 7ff896ffd640 1 -- 172.21.15.57:0/1442699596 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7ff890004250 con 0x7ff8a01f1d90 2023-10-19T14:20:35.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.077+0000 7ff8a4ad7640 1 -- 172.21.15.57:0/1442699596 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8a013aff0 con 0x7ff8a01f1d90 2023-10-19T14:20:35.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.077+0000 7ff896ffd640 1 -- 172.21.15.57:0/1442699596 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff890004640 con 0x7ff8a01f1d90 2023-10-19T14:20:35.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.077+0000 7ff8a4ad7640 1 -- 172.21.15.57:0/1442699596 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8a013b5b0 con 0x7ff8a01f1d90 2023-10-19T14:20:35.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.079+0000 7ff896ffd640 1 -- 172.21.15.57:0/1442699596 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff89003b040 con 0x7ff8a01f1d90 2023-10-19T14:20:35.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.079+0000 7ff896ffd640 1 --2- 172.21.15.57:0/1442699596 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7ff88006af90 0x7ff88006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:35.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.079+0000 7ff896ffd640 1 -- 172.21.15.57:0/1442699596 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 6921+0+0 (secure 0 0 0) 0x7ff8900ac730 con 0x7ff8a01f1d90 2023-10-19T14:20:35.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.079+0000 7ff89e59c640 1 --2- 172.21.15.57:0/1442699596 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7ff88006af90 0x7ff88006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:35.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.080+0000 7ff8a7669740 1 Processor -- start 2023-10-19T14:20:35.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.080+0000 7ff89e59c640 1 --2- 172.21.15.57:0/1442699596 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7ff88006af90 0x7ff88006d450 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff898005fd0 tx=0x7ff898005e40 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:35.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.080+0000 7ff8a7669740 1 -- start start 2023-10-19T14:20:35.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.081+0000 7ff8a7669740 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55b48a3376c0 0x55b48a335280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:35.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.081+0000 7ff8a7669740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b48a316f00 0x55b48a2eebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:35.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.081+0000 7ff8a7669740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b48a2ef130 0x55b48a3198d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:35.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.081+0000 7ff8a7669740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b48a1fb730 con 0x55b48a316f00 2023-10-19T14:20:35.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.081+0000 7ff8a7669740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b48a25c780 con 0x55b48a2ef130 2023-10-19T14:20:35.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.081+0000 7ff8a7669740 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b48a259e70 con 0x55b48a3376c0 2023-10-19T14:20:35.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.081+0000 7ff89dd9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b48a316f00 0x55b48a2eebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:35.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.081+0000 7ff89dd9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b48a316f00 0x55b48a2eebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:58952/0 (socket says 172.21.15.57:58952) 2023-10-19T14:20:35.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.081+0000 7ff89dd9b640 1 -- 172.21.15.57:0/2175937256 learned_addr learned my addr 172.21.15.57:0/2175937256 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:35.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.081+0000 7ff89ed9d640 1 --2- 172.21.15.57:0/2175937256 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b48a2ef130 0x55b48a3198d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:35.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.081+0000 7ff89e59c640 1 --2- 172.21.15.57:0/2175937256 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55b48a3376c0 0x55b48a335280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:35.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.081+0000 7ff89dd9b640 1 -- 172.21.15.57:0/2175937256 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55b48a3376c0 msgr2=0x55b48a335280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T14:20:35.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.081+0000 7ff89dd9b640 1 --2- 172.21.15.57:0/2175937256 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55b48a3376c0 0x55b48a335280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:35.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.081+0000 7ff89dd9b640 1 -- 172.21.15.57:0/2175937256 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b48a2ef130 msgr2=0x55b48a3198d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T14:20:35.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.081+0000 7ff89dd9b640 1 --2- 172.21.15.57:0/2175937256 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b48a2ef130 0x55b48a3198d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:35.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.081+0000 7ff89dd9b640 1 -- 172.21.15.57:0/2175937256 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b48a25a3d0 con 0x55b48a316f00 2023-10-19T14:20:35.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.081+0000 7ff89e59c640 1 --2- 172.21.15.57:0/2175937256 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55b48a3376c0 0x55b48a335280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:20:35.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.081+0000 7ff89dd9b640 1 --2- 172.21.15.57:0/2175937256 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b48a316f00 0x55b48a2eebf0 secure :-1 s=READY pgs=2428 cs=0 l=1 rev1=1 crypto rx=0x7ff888001510 tx=0x7ff8880019d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb7a826342261d4b server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:35.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.082+0000 7ff87ffff640 1 -- 172.21.15.57:0/2175937256 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff88800d890 con 0x55b48a316f00 2023-10-19T14:20:35.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.082+0000 7ff87ffff640 1 -- 172.21.15.57:0/2175937256 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7ff88800dae0 con 0x55b48a316f00 2023-10-19T14:20:35.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.082+0000 7ff87ffff640 1 -- 172.21.15.57:0/2175937256 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8880123f0 con 0x55b48a316f00 2023-10-19T14:20:35.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.082+0000 7ff8a7669740 1 -- 172.21.15.57:0/2175937256 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b48a316f00 msgr2=0x55b48a2eebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:35.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.082+0000 7ff8a7669740 1 --2- 172.21.15.57:0/2175937256 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b48a316f00 0x55b48a2eebf0 secure :-1 s=READY pgs=2428 cs=0 l=1 rev1=1 crypto rx=0x7ff888001510 tx=0x7ff8880019d0 comp rx=0 tx=0).stop 2023-10-19T14:20:35.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.082+0000 7ff8a7669740 1 -- 172.21.15.57:0/2175937256 shutdown_connections 2023-10-19T14:20:35.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.082+0000 7ff8a7669740 1 --2- 172.21.15.57:0/2175937256 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b48a2ef130 0x55b48a3198d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:35.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.082+0000 7ff8a7669740 1 --2- 172.21.15.57:0/2175937256 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b48a316f00 0x55b48a2eebf0 unknown :-1 s=CLOSED pgs=2428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:35.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.082+0000 7ff8a7669740 1 --2- 172.21.15.57:0/2175937256 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55b48a3376c0 0x55b48a335280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:35.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.082+0000 7ff8a7669740 1 -- 172.21.15.57:0/2175937256 >> 172.21.15.57:0/2175937256 conn(0x55b48a2ebd50 msgr2=0x55b48a3348d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:35.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.082+0000 7ff8a7669740 1 -- 172.21.15.57:0/2175937256 shutdown_connections 2023-10-19T14:20:35.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.082+0000 7ff8a7669740 1 -- 172.21.15.57:0/2175937256 wait complete. 2023-10-19T14:20:35.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.083+0000 7ff8a7669740 1 -- start start 2023-10-19T14:20:35.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.083+0000 7ff8a7669740 1 Processor -- start 2023-10-19T14:20:35.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.084+0000 7ff8a7669740 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55b48a316f00 0x55b48a334bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:35.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.084+0000 7ff8a7669740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b48a3376c0 0x55b48a330800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:35.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.084+0000 7ff8a7669740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b48a330d40 0x55b48a303990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:35.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.084+0000 7ff8a7669740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b48a255760 con 0x55b48a3376c0 2023-10-19T14:20:35.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.084+0000 7ff8a7669740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b48a2108f0 con 0x55b48a330d40 2023-10-19T14:20:35.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.084+0000 7ff8a7669740 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b48a210590 con 0x55b48a316f00 2023-10-19T14:20:35.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.084+0000 7ff8a7669740 10 client.0 _ll_register_callbacks cb 0x55b48a24cee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-19T14:20:35.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.084+0000 7ff89dd9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b48a3376c0 0x55b48a330800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:35.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.084+0000 7ff89dd9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b48a3376c0 0x55b48a330800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:58962/0 (socket says 172.21.15.57:58962) 2023-10-19T14:20:35.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.084+0000 7ff89dd9b640 1 -- 172.21.15.57:0/3903227605 learned_addr learned my addr 172.21.15.57:0/3903227605 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:35.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.084+0000 7ff89ed9d640 1 --2- 172.21.15.57:0/3903227605 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b48a330d40 0x55b48a303990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:35.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.084+0000 7ff89e59c640 1 --2- 172.21.15.57:0/3903227605 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55b48a316f00 0x55b48a334bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:35.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.084+0000 7ff89dd9b640 1 -- 172.21.15.57:0/3903227605 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55b48a316f00 msgr2=0x55b48a334bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T14:20:35.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.084+0000 7ff89dd9b640 1 --2- 172.21.15.57:0/3903227605 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55b48a316f00 0x55b48a334bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:35.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.084+0000 7ff89dd9b640 1 -- 172.21.15.57:0/3903227605 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b48a330d40 msgr2=0x55b48a303990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T14:20:35.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.084+0000 7ff89dd9b640 1 --2- 172.21.15.57:0/3903227605 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b48a330d40 0x55b48a303990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:35.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.084+0000 7ff89dd9b640 1 -- 172.21.15.57:0/3903227605 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff88c003520 con 0x55b48a3376c0 2023-10-19T14:20:35.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.085+0000 7ff89dd9b640 1 --2- 172.21.15.57:0/3903227605 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b48a3376c0 0x55b48a330800 secure :-1 s=READY pgs=2429 cs=0 l=1 rev1=1 crypto rx=0x7ff888014000 tx=0x7ff88800dfc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bac54760708fd02a server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:35.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.085+0000 7ff87d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2023-10-19T14:20:35.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.085+0000 7ff87d7fa640 1 -- 172.21.15.57:0/3903227605 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff88801b970 con 0x55b48a3376c0 2023-10-19T14:20:35.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.085+0000 7ff8a7669740 1 -- 172.21.15.57:0/3903227605 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b48a24e740 con 0x55b48a3376c0 2023-10-19T14:20:35.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.085+0000 7ff87d7fa640 1 -- 172.21.15.57:0/3903227605 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(227..227 src has 1..227) v4 ==== 6921+0+0 (secure 0 0 0) 0x7ff88801bad0 con 0x55b48a3376c0 2023-10-19T14:20:35.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.085+0000 7ff87d7fa640 1 -- 172.21.15.57:0/3903227605 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=94) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff88801bc90 con 0x55b48a3376c0 2023-10-19T14:20:35.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.085+0000 7ff8a7669740 10 client.12018 fetch_fsmap learned FSMap version 94 2023-10-19T14:20:35.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.085+0000 7ff8a7669740 1 -- 172.21.15.57:0/3903227605 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=94,monmap=2+}) v3 -- 0x55b48a253de0 con 0x55b48a3376c0 2023-10-19T14:20:35.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.085+0000 7ff87d7fa640 1 -- 172.21.15.57:0/3903227605 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x55b48a24e740 con 0x55b48a3376c0 2023-10-19T14:20:35.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.085+0000 7ff87d7fa640 1 -- 172.21.15.57:0/3903227605 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 94) v1 ==== 3891+0+0 (secure 0 0 0) 0x7ff8880035f0 con 0x55b48a3376c0 2023-10-19T14:20:35.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.085+0000 7ff8a7669740 10 client.12018 fetch_fsmap finished waiting for FSMap version 94 2023-10-19T14:20:35.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.085+0000 7ff8a7669740 10 client.12018 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-19T14:20:35.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.085+0000 7ff8a7669740 20 client.12018 populate_metadata read hostname 'smithi057' 2023-10-19T14:20:35.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.085+0000 7ff8a7669740 1 --2- 172.21.15.57:0/3903227605 >> [v2:172.21.15.57:6832/2467807803,v1:172.21.15.57:6834/2467807803] conn(0x55b48a34c080 0x55b48a34c460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:35.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.085+0000 7ff8a7669740 4 client.12018 mds_command: new command op to 11715 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-19T14:20:35.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.085+0000 7ff8a7669740 1 -- 172.21.15.57:0/3903227605 --> [v2:172.21.15.57:6832/2467807803,v1:172.21.15.57:6834/2467807803] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b48a24e460 con 0x55b48a34c080 2023-10-19T14:20:35.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.085+0000 7ff89e59c640 1 --2- 172.21.15.57:0/3903227605 >> [v2:172.21.15.57:6832/2467807803,v1:172.21.15.57:6834/2467807803] conn(0x55b48a34c080 0x55b48a34c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:35.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.086+0000 7ff87d7fa640 10 client.12018 ms_handle_connect on v2:172.21.15.57:6832/2467807803 2023-10-19T14:20:35.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.086+0000 7ff89e59c640 1 --2- 172.21.15.57:0/3903227605 >> [v2:172.21.15.57:6832/2467807803,v1:172.21.15.57:6834/2467807803] conn(0x55b48a34c080 0x55b48a34c460 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a1268ab34e20633b server_cookie=93fbf674b5f53e9 in_seq=0 out_seq=0 2023-10-19T14:20:35.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.086+0000 7ff87d7fa640 1 -- 172.21.15.57:0/3903227605 <== mds.0 v2:172.21.15.57:6832/2467807803 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55b48a24e460 con 0x55b48a34c080 2023-10-19T14:20:35.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.086+0000 7ff87d7fa640 10 client.12018 handle_command_reply: tid=0 2023-10-19T14:20:35.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.086+0000 7ff8a7669740 1 client.12018 shutdown 2023-10-19T14:20:35.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.086+0000 7ff8a7669740 1 -- 172.21.15.57:0/3903227605 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b48a3376c0 msgr2=0x55b48a330800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:35.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.086+0000 7ff8a7669740 1 --2- 172.21.15.57:0/3903227605 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b48a3376c0 0x55b48a330800 secure :-1 s=READY pgs=2429 cs=0 l=1 rev1=1 crypto rx=0x7ff888014000 tx=0x7ff88800dfc0 comp rx=0 tx=0).stop 2023-10-19T14:20:35.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.087+0000 7ff8a7669740 1 -- 172.21.15.57:0/3903227605 shutdown_connections 2023-10-19T14:20:35.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.087+0000 7ff8a7669740 1 --2- 172.21.15.57:0/3903227605 >> [v2:172.21.15.57:6832/2467807803,v1:172.21.15.57:6834/2467807803] conn(0x55b48a34c080 0x55b48a34c460 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:35.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.087+0000 7ff8a7669740 1 --2- 172.21.15.57:0/3903227605 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b48a330d40 0x55b48a303990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:35.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.087+0000 7ff8a7669740 1 --2- 172.21.15.57:0/3903227605 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b48a3376c0 0x55b48a330800 unknown :-1 s=CLOSED pgs=2429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:35.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.087+0000 7ff87d7fa640 0 client.12018 ms_handle_reset on v2:172.21.15.57:6832/2467807803 2023-10-19T14:20:35.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.087+0000 7ff8a7669740 1 --2- 172.21.15.57:0/3903227605 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55b48a316f00 0x55b48a334bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:35.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.087+0000 7ff8a7669740 1 -- 172.21.15.57:0/3903227605 >> 172.21.15.57:0/3903227605 conn(0x55b48a2ef130 msgr2=0x55b48a2e8ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:35.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.087+0000 7ff8a7669740 1 -- 172.21.15.57:0/3903227605 shutdown_connections 2023-10-19T14:20:35.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.087+0000 7ff8a7669740 1 -- 172.21.15.57:0/3903227605 wait complete. 2023-10-19T14:20:35.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.087+0000 7ff8a7669740 20 client.12018 trim_cache size 0 max 16384 2023-10-19T14:20:35.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.102+0000 7ff8a7669740 1 Processor -- start 2023-10-19T14:20:35.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.102+0000 7ff8a7669740 1 -- start start 2023-10-19T14:20:35.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.103+0000 7ff8a7669740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b48a316f00 0x55b48a330110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:35.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.103+0000 7ff8a7669740 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55b48a330d40 0x55b48a330650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:35.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.103+0000 7ff8a7669740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b48a3376c0 0x55b48a2e8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:35.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.103+0000 7ff8a7669740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b48a1d6270 con 0x55b48a3376c0 2023-10-19T14:20:35.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.103+0000 7ff8a7669740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b48a303bb0 con 0x55b48a316f00 2023-10-19T14:20:35.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.103+0000 7ff8a7669740 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b48a2eb9a0 con 0x55b48a330d40 2023-10-19T14:20:35.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.103+0000 7ff89ed9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b48a3376c0 0x55b48a2e8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:35.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.103+0000 7ff89ed9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b48a3376c0 0x55b48a2e8ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:58972/0 (socket says 172.21.15.57:58972) 2023-10-19T14:20:35.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.103+0000 7ff89ed9d640 1 -- 172.21.15.57:0/19990969 learned_addr learned my addr 172.21.15.57:0/19990969 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:35.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.103+0000 7ff89dd9b640 1 --2- 172.21.15.57:0/19990969 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55b48a330d40 0x55b48a330650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:35.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.103+0000 7ff89e59c640 1 --2- 172.21.15.57:0/19990969 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b48a316f00 0x55b48a330110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:35.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.103+0000 7ff89ed9d640 1 -- 172.21.15.57:0/19990969 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55b48a330d40 msgr2=0x55b48a330650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T14:20:35.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.103+0000 7ff89ed9d640 1 --2- 172.21.15.57:0/19990969 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55b48a330d40 0x55b48a330650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:35.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.103+0000 7ff89ed9d640 1 -- 172.21.15.57:0/19990969 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b48a316f00 msgr2=0x55b48a330110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T14:20:35.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.103+0000 7ff89ed9d640 1 --2- 172.21.15.57:0/19990969 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b48a316f00 0x55b48a330110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:35.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.103+0000 7ff89ed9d640 1 -- 172.21.15.57:0/19990969 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b48a3322c0 con 0x55b48a3376c0 2023-10-19T14:20:35.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.103+0000 7ff89dd9b640 1 --2- 172.21.15.57:0/19990969 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55b48a330d40 0x55b48a330650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:20:35.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.103+0000 7ff89e59c640 1 --2- 172.21.15.57:0/19990969 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b48a316f00 0x55b48a330110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:20:35.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.103+0000 7ff89ed9d640 1 --2- 172.21.15.57:0/19990969 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b48a3376c0 0x55b48a2e8ae0 secure :-1 s=READY pgs=2430 cs=0 l=1 rev1=1 crypto rx=0x7ff890009d60 tx=0x7ff8900346c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=671b3bcc3d7a5ef9 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:35.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.103+0000 7ff894ff9640 1 -- 172.21.15.57:0/19990969 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff89007f0b0 con 0x55b48a3376c0 2023-10-19T14:20:35.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.103+0000 7ff894ff9640 1 -- 172.21.15.57:0/19990969 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7ff8900047a0 con 0x55b48a3376c0 2023-10-19T14:20:35.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.103+0000 7ff894ff9640 1 -- 172.21.15.57:0/19990969 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8900810b0 con 0x55b48a3376c0 2023-10-19T14:20:35.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.104+0000 7ff8a7669740 1 -- 172.21.15.57:0/19990969 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b48a3376c0 msgr2=0x55b48a2e8ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:35.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.104+0000 7ff8a7669740 1 --2- 172.21.15.57:0/19990969 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b48a3376c0 0x55b48a2e8ae0 secure :-1 s=READY pgs=2430 cs=0 l=1 rev1=1 crypto rx=0x7ff890009d60 tx=0x7ff8900346c0 comp rx=0 tx=0).stop 2023-10-19T14:20:35.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.104+0000 7ff8a7669740 1 -- 172.21.15.57:0/19990969 shutdown_connections 2023-10-19T14:20:35.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.104+0000 7ff8a7669740 1 --2- 172.21.15.57:0/19990969 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b48a3376c0 0x55b48a2e8ae0 unknown :-1 s=CLOSED pgs=2430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:35.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.104+0000 7ff8a7669740 1 --2- 172.21.15.57:0/19990969 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55b48a330d40 0x55b48a330650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:35.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.104+0000 7ff8a7669740 1 --2- 172.21.15.57:0/19990969 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b48a316f00 0x55b48a330110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:35.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.104+0000 7ff8a7669740 1 -- 172.21.15.57:0/19990969 >> 172.21.15.57:0/19990969 conn(0x55b48a2ef130 msgr2=0x55b48a2e9bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:35.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.104+0000 7ff8a7669740 1 -- 172.21.15.57:0/19990969 shutdown_connections 2023-10-19T14:20:35.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.104+0000 7ff8a7669740 1 -- 172.21.15.57:0/19990969 wait complete. 2023-10-19T14:20:35.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.104+0000 7ff8a7669740 1 -- start start 2023-10-19T14:20:35.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.105+0000 7ff8a7669740 1 Processor -- start 2023-10-19T14:20:35.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.105+0000 7ff8a7669740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b48a316f00 0x55b48a3a12a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:35.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.105+0000 7ff8a7669740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b48a330d40 0x55b48a311e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:35.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.105+0000 7ff8a7669740 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55b48a3376c0 0x55b48a34b200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:35.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.105+0000 7ff8a7669740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b48a2ee950 con 0x55b48a316f00 2023-10-19T14:20:35.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.105+0000 7ff8a7669740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b48a2ee640 con 0x55b48a330d40 2023-10-19T14:20:35.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.105+0000 7ff8a7669740 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b48a3125e0 con 0x55b48a3376c0 2023-10-19T14:20:35.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.105+0000 7ff8a7669740 10 client.0 _ll_register_callbacks cb 0x7ff888007cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-19T14:20:35.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.106+0000 7ff89e59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b48a316f00 0x55b48a3a12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:35.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.106+0000 7ff89e59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b48a316f00 0x55b48a3a12a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:58976/0 (socket says 172.21.15.57:58976) 2023-10-19T14:20:35.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.106+0000 7ff89e59c640 1 -- 172.21.15.57:0/3853777698 learned_addr learned my addr 172.21.15.57:0/3853777698 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:35.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.106+0000 7ff89ed9d640 1 --2- 172.21.15.57:0/3853777698 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55b48a3376c0 0x55b48a34b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:35.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.106+0000 7ff89dd9b640 1 --2- 172.21.15.57:0/3853777698 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b48a330d40 0x55b48a311e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:35.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.106+0000 7ff89e59c640 1 -- 172.21.15.57:0/3853777698 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55b48a3376c0 msgr2=0x55b48a34b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T14:20:35.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.106+0000 7ff89e59c640 1 --2- 172.21.15.57:0/3853777698 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55b48a3376c0 0x55b48a34b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:35.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.106+0000 7ff89e59c640 1 -- 172.21.15.57:0/3853777698 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b48a330d40 msgr2=0x55b48a311e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T14:20:35.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.106+0000 7ff89e59c640 1 --2- 172.21.15.57:0/3853777698 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b48a330d40 0x55b48a311e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:35.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.106+0000 7ff89e59c640 1 -- 172.21.15.57:0/3853777698 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8a013b5b0 con 0x55b48a316f00 2023-10-19T14:20:35.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.106+0000 7ff89ed9d640 1 --2- 172.21.15.57:0/3853777698 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55b48a3376c0 0x55b48a34b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:20:35.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.106+0000 7ff89e59c640 1 --2- 172.21.15.57:0/3853777698 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b48a316f00 0x55b48a3a12a0 secure :-1 s=READY pgs=2431 cs=0 l=1 rev1=1 crypto rx=0x55b48a32f780 tx=0x7ff898004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fcf448b41ffa24b server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:35.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.106+0000 7ff87d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2023-10-19T14:20:35.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.106+0000 7ff87d7fa640 1 -- 172.21.15.57:0/3853777698 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff898010800 con 0x55b48a316f00 2023-10-19T14:20:35.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.106+0000 7ff87d7fa640 1 -- 172.21.15.57:0/3853777698 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(227..227 src has 1..227) v4 ==== 6921+0+0 (secure 0 0 0) 0x7ff898024050 con 0x55b48a316f00 2023-10-19T14:20:35.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.106+0000 7ff8a7669740 1 -- 172.21.15.57:0/3853777698 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff88800dae0 con 0x55b48a316f00 2023-10-19T14:20:35.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.106+0000 7ff87d7fa640 1 -- 172.21.15.57:0/3853777698 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=94) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff898010960 con 0x55b48a316f00 2023-10-19T14:20:35.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.106+0000 7ff8a7669740 10 client.12024 fetch_fsmap learned FSMap version 94 2023-10-19T14:20:35.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.106+0000 7ff8a7669740 1 -- 172.21.15.57:0/3853777698 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=94,monmap=2+}) v3 -- 0x55b48a1f2d00 con 0x55b48a316f00 2023-10-19T14:20:35.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.106+0000 7ff87d7fa640 1 -- 172.21.15.57:0/3853777698 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7ff88800dae0 con 0x55b48a316f00 2023-10-19T14:20:35.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.107+0000 7ff87d7fa640 1 -- 172.21.15.57:0/3853777698 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 94) v1 ==== 3891+0+0 (secure 0 0 0) 0x7ff898010bb0 con 0x55b48a316f00 2023-10-19T14:20:35.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.107+0000 7ff8a7669740 10 client.12024 fetch_fsmap finished waiting for FSMap version 94 2023-10-19T14:20:35.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.107+0000 7ff8a7669740 10 client.12024 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-19T14:20:35.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.107+0000 7ff8a7669740 20 client.12024 populate_metadata read hostname 'smithi057' 2023-10-19T14:20:35.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.107+0000 7ff8a7669740 1 --2- 172.21.15.57:0/3853777698 >> [v2:172.21.15.57:6832/2467807803,v1:172.21.15.57:6834/2467807803] conn(0x55b48a2e8cd0 0x55b48a2e90b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:35.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.107+0000 7ff8a7669740 4 client.12024 mds_command: new command op to 11715 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-19T14:20:35.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.107+0000 7ff8a7669740 1 -- 172.21.15.57:0/3853777698 --> [v2:172.21.15.57:6832/2467807803,v1:172.21.15.57:6834/2467807803] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55b489f576c0 con 0x55b48a2e8cd0 2023-10-19T14:20:35.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.107+0000 7ff89dd9b640 1 --2- 172.21.15.57:0/3853777698 >> [v2:172.21.15.57:6832/2467807803,v1:172.21.15.57:6834/2467807803] conn(0x55b48a2e8cd0 0x55b48a2e90b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:35.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.107+0000 7ff87d7fa640 10 client.12024 ms_handle_connect on v2:172.21.15.57:6832/2467807803 2023-10-19T14:20:35.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.107+0000 7ff89dd9b640 1 --2- 172.21.15.57:0/3853777698 >> [v2:172.21.15.57:6832/2467807803,v1:172.21.15.57:6834/2467807803] conn(0x55b48a2e8cd0 0x55b48a2e90b0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=632881fbe43af327 server_cookie=959c5de54341f9d9 in_seq=0 out_seq=0 2023-10-19T14:20:35.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.107+0000 7ff87d7fa640 1 -- 172.21.15.57:0/3853777698 <== mds.0 v2:172.21.15.57:6832/2467807803 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+337 (crc 0 0 0) 0x55b489f576c0 con 0x55b48a2e8cd0 2023-10-19T14:20:35.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.107+0000 7ff87d7fa640 10 client.12024 handle_command_reply: tid=0 2023-10-19T14:20:35.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.107+0000 7ff8a7669740 1 client.12024 shutdown 2023-10-19T14:20:35.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.107+0000 7ff8a7669740 1 -- 172.21.15.57:0/3853777698 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b48a316f00 msgr2=0x55b48a3a12a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:35.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.107+0000 7ff8a7669740 1 --2- 172.21.15.57:0/3853777698 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b48a316f00 0x55b48a3a12a0 secure :-1 s=READY pgs=2431 cs=0 l=1 rev1=1 crypto rx=0x55b48a32f780 tx=0x7ff898004730 comp rx=0 tx=0).stop 2023-10-19T14:20:35.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.107+0000 7ff8a7669740 1 -- 172.21.15.57:0/3853777698 shutdown_connections 2023-10-19T14:20:35.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.107+0000 7ff8a7669740 1 --2- 172.21.15.57:0/3853777698 >> [v2:172.21.15.57:6832/2467807803,v1:172.21.15.57:6834/2467807803] conn(0x55b48a2e8cd0 0x55b48a2e90b0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:35.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.107+0000 7ff8a7669740 1 --2- 172.21.15.57:0/3853777698 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x55b48a3376c0 0x55b48a34b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:35.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.107+0000 7ff8a7669740 1 --2- 172.21.15.57:0/3853777698 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b48a330d40 0x55b48a311e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:35.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.107+0000 7ff87d7fa640 0 client.12024 ms_handle_reset on v2:172.21.15.57:6832/2467807803 2023-10-19T14:20:35.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.107+0000 7ff8a7669740 1 --2- 172.21.15.57:0/3853777698 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b48a316f00 0x55b48a3a12a0 unknown :-1 s=CLOSED pgs=2431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:35.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.107+0000 7ff8a7669740 1 -- 172.21.15.57:0/3853777698 >> 172.21.15.57:0/3853777698 conn(0x55b48a2ef130 msgr2=0x55b48a2e5200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:35.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.108+0000 7ff8a7669740 1 -- 172.21.15.57:0/3853777698 shutdown_connections 2023-10-19T14:20:35.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.108+0000 7ff8a7669740 1 -- 172.21.15.57:0/3853777698 wait complete. 2023-10-19T14:20:35.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.108+0000 7ff8a7669740 20 client.12024 trim_cache size 0 max 16384 2023-10-19T14:20:35.130 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2023-10-19T14:20:35.130 INFO:teuthology.orchestra.run.smithi057.stdout: "cluster_fsid": "1e9beb19-3260-4dd7-85ef-5b399ce774ac", 2023-10-19T14:20:35.131 INFO:teuthology.orchestra.run.smithi057.stdout: "whoami": 0, 2023-10-19T14:20:35.131 INFO:teuthology.orchestra.run.smithi057.stdout: "id": 11715, 2023-10-19T14:20:35.131 INFO:teuthology.orchestra.run.smithi057.stdout: "want_state": "up:active", 2023-10-19T14:20:35.131 INFO:teuthology.orchestra.run.smithi057.stdout: "state": "up:active", 2023-10-19T14:20:35.131 INFO:teuthology.orchestra.run.smithi057.stdout: "fs_name": "backup_fs", 2023-10-19T14:20:35.131 INFO:teuthology.orchestra.run.smithi057.stdout: "rank_uptime": 4.3131627139999997, 2023-10-19T14:20:35.132 INFO:teuthology.orchestra.run.smithi057.stdout: "mdsmap_epoch": 94, 2023-10-19T14:20:35.132 INFO:teuthology.orchestra.run.smithi057.stdout: "osdmap_epoch": 227, 2023-10-19T14:20:35.132 INFO:teuthology.orchestra.run.smithi057.stdout: "osdmap_epoch_barrier": 227, 2023-10-19T14:20:35.132 INFO:teuthology.orchestra.run.smithi057.stdout: "uptime": 20.144404892000001 2023-10-19T14:20:35.132 INFO:teuthology.orchestra.run.smithi057.stdout:} 2023-10-19T14:20:35.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.108+0000 7ff8a4ad7640 1 -- 172.21.15.57:0/1442699596 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7ff88006af90 msgr2=0x7ff88006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:35.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.108+0000 7ff8a4ad7640 1 --2- 172.21.15.57:0/1442699596 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7ff88006af90 0x7ff88006d450 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff898005fd0 tx=0x7ff898005e40 comp rx=0 tx=0).stop 2023-10-19T14:20:35.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.108+0000 7ff8a4ad7640 1 -- 172.21.15.57:0/1442699596 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff8a01f1d90 msgr2=0x7ff8a01363a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:35.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.108+0000 7ff8a4ad7640 1 --2- 172.21.15.57:0/1442699596 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff8a01f1d90 0x7ff8a01363a0 secure :-1 s=READY pgs=2427 cs=0 l=1 rev1=1 crypto rx=0x7ff890037040 tx=0x7ff890035e50 comp rx=0 tx=0).stop 2023-10-19T14:20:35.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.108+0000 7ff8a4ad7640 1 -- 172.21.15.57:0/1442699596 shutdown_connections 2023-10-19T14:20:35.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.108+0000 7ff8a4ad7640 1 --2- 172.21.15.57:0/1442699596 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7ff88006af90 0x7ff88006d450 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:35.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.108+0000 7ff8a4ad7640 1 --2- 172.21.15.57:0/1442699596 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff8a01f1d90 0x7ff8a01363a0 unknown :-1 s=CLOSED pgs=2427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:35.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.108+0000 7ff8a4ad7640 1 --2- 172.21.15.57:0/1442699596 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff8a00cb980 0x7ff8a013cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:35.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.108+0000 7ff8a4ad7640 1 --2- 172.21.15.57:0/1442699596 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff8a0070350 0x7ff8a013bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:35.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.108+0000 7ff8a4ad7640 1 -- 172.21.15.57:0/1442699596 >> 172.21.15.57:0/1442699596 conn(0x7ff8a0069930 msgr2=0x7ff8a0156660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:35.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.108+0000 7ff8a4ad7640 1 -- 172.21.15.57:0/1442699596 shutdown_connections 2023-10-19T14:20:35.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.108+0000 7ff8a4ad7640 1 -- 172.21.15.57:0/1442699596 wait complete. 2023-10-19T14:20:35.136 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-19T14:20:35.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.245+0000 7f854c6fc640 1 -- 172.21.15.57:0/1113367253 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85440c9dc0 msgr2=0x7f8544154af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:35.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.245+0000 7f854c6fc640 1 --2- 172.21.15.57:0/1113367253 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85440c9dc0 0x7f8544154af0 secure :-1 s=READY pgs=2432 cs=0 l=1 rev1=1 crypto rx=0x7f853c005fd0 tx=0x7f853c02c2e0 comp rx=0 tx=0).stop 2023-10-19T14:20:35.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.245+0000 7f854c6fc640 1 -- 172.21.15.57:0/1113367253 shutdown_connections 2023-10-19T14:20:35.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.245+0000 7f854c6fc640 1 --2- 172.21.15.57:0/1113367253 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85440c9dc0 0x7f8544154af0 unknown :-1 s=CLOSED pgs=2432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:35.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.245+0000 7f854c6fc640 1 --2- 172.21.15.57:0/1113367253 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f85440b1920 0x7f854406fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:35.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.245+0000 7f854c6fc640 1 -- 172.21.15.57:0/1113367253 >> 172.21.15.57:0/1113367253 conn(0x7f8544069930 msgr2=0x7f8544069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:35.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.245+0000 7f854c6fc640 1 -- 172.21.15.57:0/1113367253 shutdown_connections 2023-10-19T14:20:35.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.245+0000 7f854c6fc640 1 -- 172.21.15.57:0/1113367253 wait complete. 2023-10-19T14:20:35.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.246+0000 7f854c6fc640 1 Processor -- start 2023-10-19T14:20:35.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.246+0000 7f854c6fc640 1 -- start start 2023-10-19T14:20:35.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.247+0000 7f854c6fc640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f854406ffb0 0x7f8544155520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:35.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.247+0000 7f854c6fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85440b1920 0x7f85440b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:35.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.247+0000 7f854c6fc640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f85440c9dc0 0x7f8544148e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:35.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.247+0000 7f854c6fc640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85441f1790 con 0x7f85440b1920 2023-10-19T14:20:35.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.247+0000 7f854c6fc640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85441f2590 con 0x7f854406ffb0 2023-10-19T14:20:35.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.247+0000 7f854c6fc640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85441f1980 con 0x7f85440c9dc0 2023-10-19T14:20:35.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.247+0000 7f8549c97640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85440b1920 0x7f85440b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:35.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.247+0000 7f8549c97640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85440b1920 0x7f85440b03e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:58988/0 (socket says 172.21.15.57:58988) 2023-10-19T14:20:35.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.247+0000 7f8549c97640 1 -- 172.21.15.57:0/2365657005 learned_addr learned my addr 172.21.15.57:0/2365657005 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:35.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.247+0000 7f854ac99640 1 --2- 172.21.15.57:0/2365657005 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f85440c9dc0 0x7f8544148e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:35.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.247+0000 7f854a498640 1 --2- 172.21.15.57:0/2365657005 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f854406ffb0 0x7f8544155520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:35.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.247+0000 7f8549c97640 1 -- 172.21.15.57:0/2365657005 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f85440c9dc0 msgr2=0x7f8544148e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:35.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.247+0000 7f8549c97640 1 --2- 172.21.15.57:0/2365657005 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f85440c9dc0 0x7f8544148e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:35.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.247+0000 7f8549c97640 1 -- 172.21.15.57:0/2365657005 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f854406ffb0 msgr2=0x7f8544155520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:35.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.247+0000 7f8549c97640 1 --2- 172.21.15.57:0/2365657005 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f854406ffb0 0x7f8544155520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:35.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.247+0000 7f8549c97640 1 -- 172.21.15.57:0/2365657005 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f853c009900 con 0x7f85440b1920 2023-10-19T14:20:35.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.247+0000 7f854ac99640 1 --2- 172.21.15.57:0/2365657005 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f85440c9dc0 0x7f8544148e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:20:35.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.247+0000 7f854a498640 1 --2- 172.21.15.57:0/2365657005 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f854406ffb0 0x7f8544155520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:20:35.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.247+0000 7f8549c97640 1 --2- 172.21.15.57:0/2365657005 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85440b1920 0x7f85440b03e0 secure :-1 s=READY pgs=2433 cs=0 l=1 rev1=1 crypto rx=0x7f853400c450 tx=0x7f853400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:35.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.247+0000 7f853affd640 1 -- 172.21.15.57:0/2365657005 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85340052d0 con 0x7f85440b1920 2023-10-19T14:20:35.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.247+0000 7f853affd640 1 -- 172.21.15.57:0/2365657005 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f8534005430 con 0x7f85440b1920 2023-10-19T14:20:35.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.247+0000 7f854c6fc640 1 -- 172.21.15.57:0/2365657005 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85441f1c60 con 0x7f85440b1920 2023-10-19T14:20:35.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.247+0000 7f853affd640 1 -- 172.21.15.57:0/2365657005 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8534012890 con 0x7f85440b1920 2023-10-19T14:20:35.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.248+0000 7f854c6fc640 1 -- 172.21.15.57:0/2365657005 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85441f2220 con 0x7f85440b1920 2023-10-19T14:20:35.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.248+0000 7f854c6fc640 1 -- 172.21.15.57:0/2365657005 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8544070ae0 con 0x7f85440b1920 2023-10-19T14:20:35.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.249+0000 7f853affd640 1 -- 172.21.15.57:0/2365657005 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f85340035f0 con 0x7f85440b1920 2023-10-19T14:20:35.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.249+0000 7f853affd640 1 --2- 172.21.15.57:0/2365657005 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f852806af60 0x7f852806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:35.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.249+0000 7f853affd640 1 -- 172.21.15.57:0/2365657005 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 6921+0+0 (secure 0 0 0) 0x7f85340899a0 con 0x7f85440b1920 2023-10-19T14:20:35.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.250+0000 7f854a498640 1 --2- 172.21.15.57:0/2365657005 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f852806af60 0x7f852806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:35.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.250+0000 7f854a498640 1 --2- 172.21.15.57:0/2365657005 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f852806af60 0x7f852806d420 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f85400058c0 tx=0x7f8540005850 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:35.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.254+0000 7f853affd640 1 -- 172.21.15.57:0/2365657005 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8534054400 con 0x7f85440b1920 2023-10-19T14:20:35.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.457+0000 7f854c6fc640 1 -- 172.21.15.57:0/2365657005 --> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f85441f23b0 con 0x7f852806af60 2023-10-19T14:20:35.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.459+0000 7f853affd640 1 -- 172.21.15.57:0/2365657005 <== mgr.9107 v2:172.21.15.179:6832/3232482678 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+82 (secure 0 0 0) 0x7f85441f23b0 con 0x7f852806af60 2023-10-19T14:20:35.461 INFO:teuthology.orchestra.run.smithi057.stdout:[ 2023-10-19T14:20:35.461 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:20:35.461 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "cephfs" 2023-10-19T14:20:35.461 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:20:35.462 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:20:35.462 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "backup_fs" 2023-10-19T14:20:35.462 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:20:35.462 INFO:teuthology.orchestra.run.smithi057.stdout:] 2023-10-19T14:20:35.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.461+0000 7f854c6fc640 1 -- 172.21.15.57:0/2365657005 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f852806af60 msgr2=0x7f852806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:35.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.461+0000 7f854c6fc640 1 --2- 172.21.15.57:0/2365657005 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f852806af60 0x7f852806d420 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f85400058c0 tx=0x7f8540005850 comp rx=0 tx=0).stop 2023-10-19T14:20:35.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.461+0000 7f854c6fc640 1 -- 172.21.15.57:0/2365657005 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85440b1920 msgr2=0x7f85440b03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:35.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.461+0000 7f854c6fc640 1 --2- 172.21.15.57:0/2365657005 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85440b1920 0x7f85440b03e0 secure :-1 s=READY pgs=2433 cs=0 l=1 rev1=1 crypto rx=0x7f853400c450 tx=0x7f853400c910 comp rx=0 tx=0).stop 2023-10-19T14:20:35.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.461+0000 7f854c6fc640 1 -- 172.21.15.57:0/2365657005 shutdown_connections 2023-10-19T14:20:35.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.461+0000 7f854c6fc640 1 --2- 172.21.15.57:0/2365657005 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f852806af60 0x7f852806d420 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:35.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.461+0000 7f854c6fc640 1 --2- 172.21.15.57:0/2365657005 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f85440c9dc0 0x7f8544148e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:35.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.461+0000 7f854c6fc640 1 --2- 172.21.15.57:0/2365657005 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85440b1920 0x7f85440b03e0 unknown :-1 s=CLOSED pgs=2433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:35.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.461+0000 7f854c6fc640 1 --2- 172.21.15.57:0/2365657005 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f854406ffb0 0x7f8544155520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:35.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.461+0000 7f854c6fc640 1 -- 172.21.15.57:0/2365657005 >> 172.21.15.57:0/2365657005 conn(0x7f8544069930 msgr2=0x7f8544153580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:35.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.461+0000 7f854c6fc640 1 -- 172.21.15.57:0/2365657005 shutdown_connections 2023-10-19T14:20:35.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.461+0000 7f854c6fc640 1 -- 172.21.15.57:0/2365657005 wait complete. 2023-10-19T14:20:35.473 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-19T14:20:35.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.582+0000 7f444f333640 1 -- 172.21.15.57:0/2173145622 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f44480b0c40 msgr2=0x7f4448158300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:35.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.582+0000 7f444f333640 1 --2- 172.21.15.57:0/2173145622 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f44480b0c40 0x7f4448158300 secure :-1 s=READY pgs=2435 cs=0 l=1 rev1=1 crypto rx=0x7f4438009830 tx=0x7f443802bd60 comp rx=0 tx=0).stop 2023-10-19T14:20:35.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.582+0000 7f444f333640 1 -- 172.21.15.57:0/2173145622 shutdown_connections 2023-10-19T14:20:35.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.582+0000 7f444f333640 1 --2- 172.21.15.57:0/2173145622 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f44480b0c40 0x7f4448158300 unknown :-1 s=CLOSED pgs=2435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:35.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.582+0000 7f444f333640 1 -- 172.21.15.57:0/2173145622 >> 172.21.15.57:0/2173145622 conn(0x7f4448069930 msgr2=0x7f4448069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:35.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.582+0000 7f444f333640 1 -- 172.21.15.57:0/2173145622 shutdown_connections 2023-10-19T14:20:35.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.582+0000 7f444f333640 1 -- 172.21.15.57:0/2173145622 wait complete. 2023-10-19T14:20:35.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.583+0000 7f444f333640 1 Processor -- start 2023-10-19T14:20:35.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.583+0000 7f444f333640 1 -- start start 2023-10-19T14:20:35.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.584+0000 7f444f333640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f44480afa20 0x7f44481463f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:35.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.584+0000 7f444f333640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f44480b0c40 0x7f44480ca870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:35.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.584+0000 7f444f333640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f44480cbd20 0x7f44480cadb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:35.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.584+0000 7f444f333640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44481ef8a0 con 0x7f44480b0c40 2023-10-19T14:20:35.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.584+0000 7f444f333640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44481478b0 con 0x7f44480afa20 2023-10-19T14:20:35.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.584+0000 7f444f333640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4448147b90 con 0x7f44480cbd20 2023-10-19T14:20:35.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.584+0000 7f444c8ce640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f44480b0c40 0x7f44480ca870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:35.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.584+0000 7f444c8ce640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f44480b0c40 0x7f44480ca870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59016/0 (socket says 172.21.15.57:59016) 2023-10-19T14:20:35.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.584+0000 7f444c8ce640 1 -- 172.21.15.57:0/2294691135 learned_addr learned my addr 172.21.15.57:0/2294691135 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:35.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.584+0000 7f444c8ce640 1 -- 172.21.15.57:0/2294691135 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f44480cbd20 msgr2=0x7f44480cadb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:35.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.584+0000 7f444c8ce640 1 --2- 172.21.15.57:0/2294691135 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f44480cbd20 0x7f44480cadb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:35.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.584+0000 7f444c8ce640 1 -- 172.21.15.57:0/2294691135 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f44480afa20 msgr2=0x7f44481463f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:35.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.584+0000 7f444c8ce640 1 --2- 172.21.15.57:0/2294691135 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f44480afa20 0x7f44481463f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:35.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.584+0000 7f444c8ce640 1 -- 172.21.15.57:0/2294691135 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4438009490 con 0x7f44480b0c40 2023-10-19T14:20:35.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.585+0000 7f444c8ce640 1 --2- 172.21.15.57:0/2294691135 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f44480b0c40 0x7f44480ca870 secure :-1 s=READY pgs=2436 cs=0 l=1 rev1=1 crypto rx=0x7f443c00c410 tx=0x7f443c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:35.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.585+0000 7f4445ffb640 1 -- 172.21.15.57:0/2294691135 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f443c0035f0 con 0x7f44480b0c40 2023-10-19T14:20:35.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.585+0000 7f4445ffb640 1 -- 172.21.15.57:0/2294691135 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f443c003750 con 0x7f44480b0c40 2023-10-19T14:20:35.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.585+0000 7f444f333640 1 -- 172.21.15.57:0/2294691135 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4448147e70 con 0x7f44480b0c40 2023-10-19T14:20:35.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.585+0000 7f4445ffb640 1 -- 172.21.15.57:0/2294691135 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f443c003ba0 con 0x7f44480b0c40 2023-10-19T14:20:35.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.585+0000 7f444f333640 1 -- 172.21.15.57:0/2294691135 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44481ef4a0 con 0x7f44480b0c40 2023-10-19T14:20:35.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.586+0000 7f444f333640 1 -- 172.21.15.57:0/2294691135 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44480690e0 con 0x7f44480b0c40 2023-10-19T14:20:35.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.586+0000 7f4445ffb640 1 -- 172.21.15.57:0/2294691135 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 128) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f443c003d00 con 0x7f44480b0c40 2023-10-19T14:20:35.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.587+0000 7f4445ffb640 1 --2- 172.21.15.57:0/2294691135 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f443406af60 0x7f443406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:35.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.587+0000 7f4445ffb640 1 -- 172.21.15.57:0/2294691135 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 6921+0+0 (secure 0 0 0) 0x7f443c089440 con 0x7f44480b0c40 2023-10-19T14:20:35.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.587+0000 7f444d0cf640 1 --2- 172.21.15.57:0/2294691135 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f443406af60 0x7f443406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:35.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.588+0000 7f444d0cf640 1 --2- 172.21.15.57:0/2294691135 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f443406af60 0x7f443406d420 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4440001c60 tx=0x7f4440007cb0 comp rx=0 tx=0).ready entity=mgr.9107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:35.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.591+0000 7f4445ffb640 1 -- 172.21.15.57:0/2294691135 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f443c053f20 con 0x7f44480b0c40 2023-10-19T14:20:35.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:35.815+0000 7f444f333640 1 -- 172.21.15.57:0/2294691135 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f4448157ff0 con 0x7f44480b0c40 2023-10-19T14:20:36.207 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:20:36.209 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:20:36.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.347+0000 7f4445ffb640 1 -- 172.21.15.57:0/2294691135 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v129) v1 ==== 92+0+0 (secure 0 0 0) 0x7f443c0170a0 con 0x7f44480b0c40 2023-10-19T14:20:36.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.350+0000 7f444f333640 1 -- 172.21.15.57:0/2294691135 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f443406af60 msgr2=0x7f443406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:36.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.350+0000 7f444f333640 1 --2- 172.21.15.57:0/2294691135 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f443406af60 0x7f443406d420 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4440001c60 tx=0x7f4440007cb0 comp rx=0 tx=0).stop 2023-10-19T14:20:36.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.350+0000 7f444f333640 1 -- 172.21.15.57:0/2294691135 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f44480b0c40 msgr2=0x7f44480ca870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:36.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.350+0000 7f444f333640 1 --2- 172.21.15.57:0/2294691135 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f44480b0c40 0x7f44480ca870 secure :-1 s=READY pgs=2436 cs=0 l=1 rev1=1 crypto rx=0x7f443c00c410 tx=0x7f443c00c8d0 comp rx=0 tx=0).stop 2023-10-19T14:20:36.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.350+0000 7f444f333640 1 -- 172.21.15.57:0/2294691135 shutdown_connections 2023-10-19T14:20:36.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.350+0000 7f444f333640 1 --2- 172.21.15.57:0/2294691135 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f443406af60 0x7f443406d420 secure :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4440001c60 tx=0x7f4440007cb0 comp rx=0 tx=0).stop 2023-10-19T14:20:36.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.350+0000 7f444f333640 1 --2- 172.21.15.57:0/2294691135 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f44480cbd20 0x7f44480cadb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:36.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.350+0000 7f444f333640 1 --2- 172.21.15.57:0/2294691135 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f44480b0c40 0x7f44480ca870 unknown :-1 s=CLOSED pgs=2436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:36.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.350+0000 7f444f333640 1 --2- 172.21.15.57:0/2294691135 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f44480afa20 0x7f44481463f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:36.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.350+0000 7f444f333640 1 -- 172.21.15.57:0/2294691135 >> 172.21.15.57:0/2294691135 conn(0x7f4448069930 msgr2=0x7f4448069d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:36.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.350+0000 7f444f333640 1 -- 172.21.15.57:0/2294691135 shutdown_connections 2023-10-19T14:20:36.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.350+0000 7f444f333640 1 -- 172.21.15.57:0/2294691135 wait complete. 2023-10-19T14:20:36.364 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-19T14:20:36.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.479+0000 7f3e8d0d7640 1 -- 172.21.15.57:0/3350692194 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3e8815b640 msgr2=0x7f3e880cbf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:36.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.479+0000 7f3e8d0d7640 1 --2- 172.21.15.57:0/3350692194 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3e8815b640 0x7f3e880cbf20 secure :-1 s=READY pgs=2440 cs=0 l=1 rev1=1 crypto rx=0x7f3e80005fd0 tx=0x7f3e8002c2e0 comp rx=0 tx=0).stop 2023-10-19T14:20:36.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.479+0000 7f3e8d0d7640 1 -- 172.21.15.57:0/3350692194 shutdown_connections 2023-10-19T14:20:36.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.479+0000 7f3e8d0d7640 1 --2- 172.21.15.57:0/3350692194 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3e8815b640 0x7f3e880cbf20 unknown :-1 s=CLOSED pgs=2440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:36.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.479+0000 7f3e8d0d7640 1 --2- 172.21.15.57:0/3350692194 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3e880cfa20 0x7f3e880cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:36.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.479+0000 7f3e8d0d7640 1 -- 172.21.15.57:0/3350692194 >> 172.21.15.57:0/3350692194 conn(0x7f3e8809a030 msgr2=0x7f3e880d0d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:36.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.479+0000 7f3e8d0d7640 1 -- 172.21.15.57:0/3350692194 shutdown_connections 2023-10-19T14:20:36.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.479+0000 7f3e8d0d7640 1 -- 172.21.15.57:0/3350692194 wait complete. 2023-10-19T14:20:36.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.480+0000 7f3e8d0d7640 1 Processor -- start 2023-10-19T14:20:36.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.481+0000 7f3e8d0d7640 1 -- start start 2023-10-19T14:20:36.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.481+0000 7f3e8d0d7640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3e880cfa20 0x7f3e881e5a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:36.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.481+0000 7f3e8d0d7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3e8815b260 0x7f3e880b57e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:36.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.481+0000 7f3e8d0d7640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3e8815b640 0x7f3e880b6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:36.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.481+0000 7f3e8d0d7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e880b39e0 con 0x7f3e8815b260 2023-10-19T14:20:36.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.481+0000 7f3e8d0d7640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e880b5d20 con 0x7f3e880cfa20 2023-10-19T14:20:36.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.481+0000 7f3e8d0d7640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e881e6060 con 0x7f3e8815b640 2023-10-19T14:20:36.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.481+0000 7f3e8659c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3e8815b260 0x7f3e880b57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:36.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.481+0000 7f3e8659c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3e8815b260 0x7f3e880b57e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59044/0 (socket says 172.21.15.57:59044) 2023-10-19T14:20:36.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.481+0000 7f3e8659c640 1 -- 172.21.15.57:0/1340870767 learned_addr learned my addr 172.21.15.57:0/1340870767 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:36.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.481+0000 7f3e8759e640 1 --2- 172.21.15.57:0/1340870767 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3e8815b640 0x7f3e880b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:36.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.481+0000 7f3e86d9d640 1 --2- 172.21.15.57:0/1340870767 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3e880cfa20 0x7f3e881e5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:36.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.482+0000 7f3e8659c640 1 -- 172.21.15.57:0/1340870767 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3e8815b640 msgr2=0x7f3e880b6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:36.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.482+0000 7f3e8659c640 1 --2- 172.21.15.57:0/1340870767 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3e8815b640 0x7f3e880b6120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:36.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.482+0000 7f3e8659c640 1 -- 172.21.15.57:0/1340870767 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3e880cfa20 msgr2=0x7f3e881e5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:36.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.482+0000 7f3e8659c640 1 --2- 172.21.15.57:0/1340870767 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3e880cfa20 0x7f3e881e5a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:36.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.482+0000 7f3e8659c640 1 -- 172.21.15.57:0/1340870767 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e80009900 con 0x7f3e8815b260 2023-10-19T14:20:36.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.482+0000 7f3e8659c640 1 --2- 172.21.15.57:0/1340870767 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3e8815b260 0x7f3e880b57e0 secure :-1 s=READY pgs=2441 cs=0 l=1 rev1=1 crypto rx=0x7f3e7800b4a0 tx=0x7f3e7800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:36.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.482+0000 7f3e6f7fe640 1 -- 172.21.15.57:0/1340870767 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e78010610 con 0x7f3e8815b260 2023-10-19T14:20:36.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.482+0000 7f3e6f7fe640 1 -- 172.21.15.57:0/1340870767 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f3e78010860 con 0x7f3e8815b260 2023-10-19T14:20:36.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.482+0000 7f3e8d0d7640 1 -- 172.21.15.57:0/1340870767 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e881e6340 con 0x7f3e8815b260 2023-10-19T14:20:36.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.482+0000 7f3e6f7fe640 1 -- 172.21.15.57:0/1340870767 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e78010c70 con 0x7f3e8815b260 2023-10-19T14:20:36.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.482+0000 7f3e8d0d7640 1 -- 172.21.15.57:0/1340870767 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e880b6660 con 0x7f3e8815b260 2023-10-19T14:20:36.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.483+0000 7f3e8d0d7640 1 -- 172.21.15.57:0/1340870767 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e880675f0 con 0x7f3e8815b260 2023-10-19T14:20:36.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.484+0000 7f3e6f7fe640 1 -- 172.21.15.57:0/1340870767 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 129) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f3e78017360 con 0x7f3e8815b260 2023-10-19T14:20:36.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.484+0000 7f3e6f7fe640 1 --2- 172.21.15.57:0/1340870767 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f3e7406afb0 0x7f3e7406d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:36.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.484+0000 7f3e86d9d640 1 -- 172.21.15.57:0/1340870767 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f3e7406afb0 msgr2=0x7f3e7406d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/3232482678 2023-10-19T14:20:36.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.485+0000 7f3e6f7fe640 1 -- 172.21.15.57:0/1340870767 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 6921+0+0 (secure 0 0 0) 0x7f3e78089ab0 con 0x7f3e8815b260 2023-10-19T14:20:36.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.485+0000 7f3e86d9d640 1 --2- 172.21.15.57:0/1340870767 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f3e7406afb0 0x7f3e7406d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T14:20:36.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.489+0000 7f3e6f7fe640 1 -- 172.21.15.57:0/1340870767 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3e78054640 con 0x7f3e8815b260 2023-10-19T14:20:36.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.669+0000 7f3e8d0d7640 1 -- 172.21.15.57:0/1340870767 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f3e880d0610 con 0x7f3e8815b260 2023-10-19T14:20:36.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.673+0000 7f3e6f7fe640 1 -- 172.21.15.57:0/1340870767 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v76)=0 v76) v1 ==== 138+0+0 (secure 0 0 0) 0x7f3e78057c60 con 0x7f3e8815b260 2023-10-19T14:20:36.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.675+0000 7f3e8d0d7640 1 -- 172.21.15.57:0/1340870767 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f3e7406afb0 msgr2=0x7f3e7406d470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T14:20:36.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.675+0000 7f3e8d0d7640 1 --2- 172.21.15.57:0/1340870767 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f3e7406afb0 0x7f3e7406d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:36.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.675+0000 7f3e8d0d7640 1 -- 172.21.15.57:0/1340870767 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3e8815b260 msgr2=0x7f3e880b57e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:36.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.675+0000 7f3e8d0d7640 1 --2- 172.21.15.57:0/1340870767 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3e8815b260 0x7f3e880b57e0 secure :-1 s=READY pgs=2441 cs=0 l=1 rev1=1 crypto rx=0x7f3e7800b4a0 tx=0x7f3e7800b960 comp rx=0 tx=0).stop 2023-10-19T14:20:36.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.675+0000 7f3e8d0d7640 1 -- 172.21.15.57:0/1340870767 shutdown_connections 2023-10-19T14:20:36.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.675+0000 7f3e8d0d7640 1 --2- 172.21.15.57:0/1340870767 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f3e7406afb0 0x7f3e7406d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:36.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.675+0000 7f3e8d0d7640 1 --2- 172.21.15.57:0/1340870767 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3e8815b640 0x7f3e880b6120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:36.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.675+0000 7f3e8d0d7640 1 --2- 172.21.15.57:0/1340870767 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3e8815b260 0x7f3e880b57e0 unknown :-1 s=CLOSED pgs=2441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:36.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.675+0000 7f3e8d0d7640 1 --2- 172.21.15.57:0/1340870767 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3e880cfa20 0x7f3e881e5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:36.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.675+0000 7f3e8d0d7640 1 -- 172.21.15.57:0/1340870767 >> 172.21.15.57:0/1340870767 conn(0x7f3e8809a030 msgr2=0x7f3e880cb9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:36.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.675+0000 7f3e8d0d7640 1 -- 172.21.15.57:0/1340870767 shutdown_connections 2023-10-19T14:20:36.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.675+0000 7f3e8d0d7640 1 -- 172.21.15.57:0/1340870767 wait complete. 2023-10-19T14:20:36.689 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-19T14:20:36.731 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:36.729+0000 7fc253e0f040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-19T14:20:36.752 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:20:36.750+0000 7f6056a37040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-19T14:20:36.775 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:36.773+0000 7fc253e0f040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-19T14:20:36.800 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:20:36.799+0000 7f6056a37040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-19T14:20:36.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.800+0000 7f349c297640 1 -- 172.21.15.57:0/942446497 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f34940cf340 msgr2=0x7f34940d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:36.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.800+0000 7f349c297640 1 --2- 172.21.15.57:0/942446497 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f34940cf340 0x7f34940d1910 secure :-1 s=READY pgs=2442 cs=0 l=1 rev1=1 crypto rx=0x7f349408a120 tx=0x7f349002bdb0 comp rx=0 tx=0).stop 2023-10-19T14:20:36.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.800+0000 7f349c297640 1 -- 172.21.15.57:0/942446497 shutdown_connections 2023-10-19T14:20:36.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.800+0000 7f349c297640 1 --2- 172.21.15.57:0/942446497 >> v2:172.21.15.179:3300/0 conn(0x7f349415b2a0 0x7f34940cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:36.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.800+0000 7f349c297640 1 --2- 172.21.15.57:0/942446497 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f34940cf340 0x7f34940d1910 unknown :-1 s=CLOSED pgs=2442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:36.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.800+0000 7f349c297640 1 -- 172.21.15.57:0/942446497 >> 172.21.15.57:0/942446497 conn(0x7f34940d0d60 msgr2=0x7f34940d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:36.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.800+0000 7f349c297640 1 -- 172.21.15.57:0/942446497 shutdown_connections 2023-10-19T14:20:36.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.800+0000 7f349c297640 1 -- 172.21.15.57:0/942446497 wait complete. 2023-10-19T14:20:36.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.801+0000 7f349c297640 1 Processor -- start 2023-10-19T14:20:36.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.801+0000 7f349c297640 1 -- start start 2023-10-19T14:20:36.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.802+0000 7f349c297640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f34940ceb40 0x7f34940c7e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:36.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.802+0000 7f349c297640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f34940cf340 0x7f349406b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:36.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.802+0000 7f349c297640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f349415b2a0 0x7f34940c9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:36.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.802+0000 7f349c297640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f349415de50 con 0x7f34940ceb40 2023-10-19T14:20:36.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.802+0000 7f349c297640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34940c8510 con 0x7f349415b2a0 2023-10-19T14:20:36.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.802+0000 7f349c297640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34940c87f0 con 0x7f34940cf340 2023-10-19T14:20:36.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.802+0000 7f349a033640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f34940ceb40 0x7f34940c7e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:36.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.802+0000 7f349a033640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f34940ceb40 0x7f34940c7e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42256/0 (socket says 172.21.15.57:42256) 2023-10-19T14:20:36.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.802+0000 7f349a033640 1 -- 172.21.15.57:0/1180969455 learned_addr learned my addr 172.21.15.57:0/1180969455 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:36.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.802+0000 7f3499832640 1 --2- 172.21.15.57:0/1180969455 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f34940cf340 0x7f349406b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:36.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.802+0000 7f349a834640 1 --2- 172.21.15.57:0/1180969455 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f349415b2a0 0x7f34940c9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:36.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.802+0000 7f349a033640 1 -- 172.21.15.57:0/1180969455 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f34940cf340 msgr2=0x7f349406b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:36.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.802+0000 7f349a033640 1 --2- 172.21.15.57:0/1180969455 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f34940cf340 0x7f349406b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:36.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.802+0000 7f349a033640 1 -- 172.21.15.57:0/1180969455 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f349415b2a0 msgr2=0x7f34940c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:36.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.802+0000 7f349a033640 1 --2- 172.21.15.57:0/1180969455 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f349415b2a0 0x7f34940c9bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:36.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.802+0000 7f349a033640 1 -- 172.21.15.57:0/1180969455 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3490009560 con 0x7f34940ceb40 2023-10-19T14:20:36.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.802+0000 7f349a033640 1 --2- 172.21.15.57:0/1180969455 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f34940ceb40 0x7f34940c7e30 secure :-1 s=READY pgs=2443 cs=0 l=1 rev1=1 crypto rx=0x7f3490009850 tx=0x7f349002da80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:36.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.803+0000 7f348affd640 1 -- 172.21.15.57:0/1180969455 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3490003800 con 0x7f34940ceb40 2023-10-19T14:20:36.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.803+0000 7f348affd640 1 -- 172.21.15.57:0/1180969455 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f3490003a50 con 0x7f34940ceb40 2023-10-19T14:20:36.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.803+0000 7f348affd640 1 -- 172.21.15.57:0/1180969455 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3490003e20 con 0x7f34940ceb40 2023-10-19T14:20:36.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.803+0000 7f349c297640 1 -- 172.21.15.57:0/1180969455 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34940c8ad0 con 0x7f34940ceb40 2023-10-19T14:20:36.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.803+0000 7f349c297640 1 -- 172.21.15.57:0/1180969455 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34940c9090 con 0x7f34940ceb40 2023-10-19T14:20:36.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.804+0000 7f349c297640 1 -- 172.21.15.57:0/1180969455 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34940690e0 con 0x7f34940ceb40 2023-10-19T14:20:36.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.805+0000 7f348affd640 1 -- 172.21.15.57:0/1180969455 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 129) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f349003e450 con 0x7f34940ceb40 2023-10-19T14:20:36.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.805+0000 7f348affd640 1 --2- 172.21.15.57:0/1180969455 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f347806afb0 0x7f347806d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:36.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.805+0000 7f348affd640 1 -- 172.21.15.57:0/1180969455 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 6921+0+0 (secure 0 0 0) 0x7f34900ac600 con 0x7f34940ceb40 2023-10-19T14:20:36.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.805+0000 7f3499832640 1 -- 172.21.15.57:0/1180969455 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f347806afb0 msgr2=0x7f347806d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/3232482678 2023-10-19T14:20:36.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.805+0000 7f3499832640 1 --2- 172.21.15.57:0/1180969455 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f347806afb0 0x7f347806d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T14:20:36.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.809+0000 7f348affd640 1 -- 172.21.15.57:0/1180969455 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3490077220 con 0x7f34940ceb40 2023-10-19T14:20:36.990 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.988+0000 7f349c297640 1 -- 172.21.15.57:0/1180969455 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f34940cb9d0 con 0x7f34940ceb40 2023-10-19T14:20:36.998 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.995+0000 7f348affd640 1 -- 172.21.15.57:0/1180969455 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v77)=0 v77) v1 ==== 133+0+0 (secure 0 0 0) 0x7f349007a840 con 0x7f34940ceb40 2023-10-19T14:20:37.000 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.997+0000 7f349c297640 1 -- 172.21.15.57:0/1180969455 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f347806afb0 msgr2=0x7f347806d470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T14:20:37.001 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.997+0000 7f349c297640 1 --2- 172.21.15.57:0/1180969455 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f347806afb0 0x7f347806d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:37.001 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.997+0000 7f349c297640 1 -- 172.21.15.57:0/1180969455 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f34940ceb40 msgr2=0x7f34940c7e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:37.001 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.997+0000 7f349c297640 1 --2- 172.21.15.57:0/1180969455 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f34940ceb40 0x7f34940c7e30 secure :-1 s=READY pgs=2443 cs=0 l=1 rev1=1 crypto rx=0x7f3490009850 tx=0x7f349002da80 comp rx=0 tx=0).stop 2023-10-19T14:20:37.001 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.997+0000 7f349c297640 1 -- 172.21.15.57:0/1180969455 shutdown_connections 2023-10-19T14:20:37.001 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.997+0000 7f349c297640 1 --2- 172.21.15.57:0/1180969455 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7f347806afb0 0x7f347806d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:37.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.997+0000 7f349c297640 1 --2- 172.21.15.57:0/1180969455 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f349415b2a0 0x7f34940c9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:37.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.997+0000 7f349c297640 1 --2- 172.21.15.57:0/1180969455 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f34940cf340 0x7f349406b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:37.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.997+0000 7f349c297640 1 --2- 172.21.15.57:0/1180969455 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f34940ceb40 0x7f34940c7e30 unknown :-1 s=CLOSED pgs=2443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:37.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.997+0000 7f349c297640 1 -- 172.21.15.57:0/1180969455 >> 172.21.15.57:0/1180969455 conn(0x7f34940d0d60 msgr2=0x7f34940b5960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:37.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.997+0000 7f349c297640 1 -- 172.21.15.57:0/1180969455 shutdown_connections 2023-10-19T14:20:37.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:36.997+0000 7f349c297640 1 -- 172.21.15.57:0/1180969455 wait complete. 2023-10-19T14:20:37.011 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:20:37.012 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p mandatory_fs_argument_test_dir) 2023-10-19T14:20:37.031 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add mandatory_fs_argument_test_dir --fs backup_fs 1M 2023-10-19T14:20:37.166 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:37.163+0000 7fc253e0f040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-19T14:20:37.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:37.184+0000 7fef7fa94640 1 -- 172.21.15.57:0/3666615327 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fef7815b260 msgr2=0x7fef780cf310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:37.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:37.184+0000 7fef7fa94640 1 --2- 172.21.15.57:0/3666615327 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fef7815b260 0x7fef780cf310 secure :-1 s=READY pgs=2445 cs=0 l=1 rev1=1 crypto rx=0x7fef68009830 tx=0x7fef6802bdf0 comp rx=0 tx=0).stop 2023-10-19T14:20:37.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:37.184+0000 7fef7fa94640 1 -- 172.21.15.57:0/3666615327 shutdown_connections 2023-10-19T14:20:37.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:37.184+0000 7fef7fa94640 1 --2- 172.21.15.57:0/3666615327 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fef7815b260 0x7fef780cf310 unknown :-1 s=CLOSED pgs=2445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:37.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:37.184+0000 7fef7fa94640 1 -- 172.21.15.57:0/3666615327 >> 172.21.15.57:0/3666615327 conn(0x7fef7809a030 msgr2=0x7fef780d0d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:37.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:37.184+0000 7fef7fa94640 1 -- 172.21.15.57:0/3666615327 shutdown_connections 2023-10-19T14:20:37.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:37.184+0000 7fef7fa94640 1 -- 172.21.15.57:0/3666615327 wait complete. 2023-10-19T14:20:37.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:37.185+0000 7fef7fa94640 1 Processor -- start 2023-10-19T14:20:37.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:37.185+0000 7fef7fa94640 1 -- start start 2023-10-19T14:20:37.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:37.186+0000 7fef7fa94640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fef780cf850 0x7fef780b53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:37.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:37.186+0000 7fef7fa94640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fef7815b260 0x7fef78153500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:37.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:37.186+0000 7fef7fa94640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fef7815b640 0x7fef780b4370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:37.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:37.186+0000 7fef7fa94640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef78153a40 con 0x7fef780cf850 2023-10-19T14:20:37.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:37.186+0000 7fef7fa94640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef780b48b0 con 0x7fef7815b260 2023-10-19T14:20:37.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:37.186+0000 7fef7fa94640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef780b65d0 con 0x7fef7815b640 2023-10-19T14:20:37.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:37.186+0000 7fef7d830640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fef780cf850 0x7fef780b53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:37.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:37.186+0000 7fef7d830640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fef780cf850 0x7fef780b53c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42274/0 (socket says 172.21.15.57:42274) 2023-10-19T14:20:37.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:37.186+0000 7fef7d830640 1 -- 172.21.15.57:0/4176013031 learned_addr learned my addr 172.21.15.57:0/4176013031 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:37.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:37.186+0000 7fef7e031640 1 --2- 172.21.15.57:0/4176013031 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fef7815b640 0x7fef780b4370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:37.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:37.186+0000 7fef7d02f640 1 --2- 172.21.15.57:0/4176013031 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fef7815b260 0x7fef78153500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:37.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:37.186+0000 7fef7d830640 1 -- 172.21.15.57:0/4176013031 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fef7815b640 msgr2=0x7fef780b4370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:37.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:37.186+0000 7fef7d830640 1 --2- 172.21.15.57:0/4176013031 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fef7815b640 0x7fef780b4370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:37.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:37.186+0000 7fef7d830640 1 -- 172.21.15.57:0/4176013031 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fef7815b260 msgr2=0x7fef78153500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:37.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:37.186+0000 7fef7d830640 1 --2- 172.21.15.57:0/4176013031 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fef7815b260 0x7fef78153500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:37.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:37.186+0000 7fef7d830640 1 -- 172.21.15.57:0/4176013031 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef68009490 con 0x7fef780cf850 2023-10-19T14:20:37.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:37.186+0000 7fef7d02f640 1 --2- 172.21.15.57:0/4176013031 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fef7815b260 0x7fef78153500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:20:37.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:37.187+0000 7fef7d830640 1 --2- 172.21.15.57:0/4176013031 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fef780cf850 0x7fef780b53c0 secure :-1 s=READY pgs=2446 cs=0 l=1 rev1=1 crypto rx=0x7fef68009780 tx=0x7fef680038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:37.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:37.187+0000 7fef6e7fc640 1 -- 172.21.15.57:0/4176013031 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef68003ea0 con 0x7fef780cf850 2023-10-19T14:20:37.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:37.187+0000 7fef6e7fc640 1 -- 172.21.15.57:0/4176013031 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fef68034550 con 0x7fef780cf850 2023-10-19T14:20:37.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:37.187+0000 7fef7fa94640 1 -- 172.21.15.57:0/4176013031 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef780b5a50 con 0x7fef780cf850 2023-10-19T14:20:37.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:37.187+0000 7fef6e7fc640 1 -- 172.21.15.57:0/4176013031 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef68034960 con 0x7fef780cf850 2023-10-19T14:20:37.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:37.187+0000 7fef7fa94640 1 -- 172.21.15.57:0/4176013031 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef780b6010 con 0x7fef780cf850 2023-10-19T14:20:37.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:37.188+0000 7fef7fa94640 1 -- 172.21.15.57:0/4176013031 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef780c7e80 con 0x7fef780cf850 2023-10-19T14:20:37.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:37.188+0000 7fef6e7fc640 1 -- 172.21.15.57:0/4176013031 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 129) v1 ==== 89348+0+0 (secure 0 0 0) 0x7fef6803b040 con 0x7fef780cf850 2023-10-19T14:20:37.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:37.189+0000 7fef6e7fc640 1 --2- 172.21.15.57:0/4176013031 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fef5c06afb0 0x7fef5c06d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:37.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:37.189+0000 7fef6e7fc640 1 -- 172.21.15.57:0/4176013031 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 6921+0+0 (secure 0 0 0) 0x7fef680ac250 con 0x7fef780cf850 2023-10-19T14:20:37.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:37.189+0000 7fef7d02f640 1 -- 172.21.15.57:0/4176013031 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fef5c06afb0 msgr2=0x7fef5c06d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/3232482678 2023-10-19T14:20:37.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:37.189+0000 7fef7d02f640 1 --2- 172.21.15.57:0/4176013031 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fef5c06afb0 0x7fef5c06d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T14:20:37.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:37.197+0000 7fef6e7fc640 1 -- 172.21.15.57:0/4176013031 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fef780c7e80 con 0x7fef780cf850 2023-10-19T14:20:37.210 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:37.208+0000 7fc253e0f040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-19T14:20:37.223 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:20:37.221+0000 7f6056a37040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-19T14:20:37.252 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:37.249+0000 7fc253e0f040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-19T14:20:37.270 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:20:37.267+0000 7f6056a37040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-19T14:20:37.303 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:37.300+0000 7fc253e0f040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-19T14:20:37.315 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:20:37.312+0000 7f6056a37040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-19T14:20:37.343 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:37.340+0000 7fc253e0f040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-19T14:20:37.371 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:20:37.368+0000 7f6056a37040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-19T14:20:37.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:37.390+0000 7fef7d02f640 1 -- 172.21.15.57:0/4176013031 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fef5c06afb0 msgr2=0x7fef5c06d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/3232482678 2023-10-19T14:20:37.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:37.390+0000 7fef7d02f640 1 --2- 172.21.15.57:0/4176013031 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fef5c06afb0 0x7fef5c06d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-19T14:20:37.400 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:37.397+0000 7fef7fa94640 1 -- 172.21.15.57:0/4176013031 --> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fef780c7e80 con 0x7fef5c06afb0 2023-10-19T14:20:37.414 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:20:37.412+0000 7f6056a37040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-19T14:20:37.441 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:37.439+0000 7fc253e0f040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-19T14:20:37.494 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:37.491+0000 7fc253e0f040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-19T14:20:37.517 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:20:37.514+0000 7f6056a37040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-19T14:20:37.535 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:37.532+0000 7fc253e0f040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-19T14:20:37.572 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:20:37.570+0000 7f6056a37040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-19T14:20:37.583 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:37.580+0000 7fc253e0f040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-19T14:20:37.616 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:20:37.613+0000 7f6056a37040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-19T14:20:37.625 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:37.622+0000 7fc253e0f040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-19T14:20:37.667 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:20:37.663+0000 7f6056a37040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-19T14:20:37.711 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:20:37.708+0000 7f6056a37040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-19T14:20:37.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:37.791+0000 7fef7d02f640 1 -- 172.21.15.57:0/4176013031 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fef5c06afb0 msgr2=0x7fef5c06d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/3232482678 2023-10-19T14:20:37.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:37.791+0000 7fef7d02f640 1 --2- 172.21.15.57:0/4176013031 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fef5c06afb0 0x7fef5c06d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-19T14:20:37.940 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:37.938+0000 7fc253e0f040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-19T14:20:38.043 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:20:38.040+0000 7f6056a37040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-19T14:20:38.072 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:38.069+0000 7fc253e0f040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-19T14:20:38.161 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:38.159+0000 7fc253e0f040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-19T14:20:38.181 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:20:38.178+0000 7f6056a37040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-19T14:20:38.203 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:38.200+0000 7fc253e0f040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-19T14:20:38.277 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:20:38.274+0000 7f6056a37040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-19T14:20:38.296 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:38.294+0000 7fc253e0f040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-19T14:20:38.321 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:20:38.319+0000 7f6056a37040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-19T14:20:38.386 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:38.384+0000 7fc253e0f040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-19T14:20:38.422 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:20:38.419+0000 7f6056a37040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-19T14:20:38.517 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:20:38.514+0000 7f6056a37040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-19T14:20:38.521 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:38.518+0000 7fc253e0f040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-19T14:20:38.562 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:38.559+0000 7fc253e0f040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-19T14:20:38.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:38.592+0000 7fef7d02f640 1 -- 172.21.15.57:0/4176013031 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fef5c06afb0 msgr2=0x7fef5c06d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/3232482678 2023-10-19T14:20:38.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:38.592+0000 7fef7d02f640 1 --2- 172.21.15.57:0/4176013031 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fef5c06afb0 0x7fef5c06d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-19T14:20:38.660 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:20:38.657+0000 7f6056a37040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-19T14:20:38.665 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:38.662+0000 7fc253e0f040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-19T14:20:38.703 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:20:38.700+0000 7f6056a37040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-19T14:20:38.814 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:20:38.812+0000 7f6056a37040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-19T14:20:39.286 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:39.282+0000 7fc253e0f040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-19T14:20:39.463 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:20:39.460+0000 7f6056a37040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-19T14:20:39.544 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:39.542+0000 7fc253e0f040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-19T14:20:39.592 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:39.589+0000 7fc253e0f040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-19T14:20:39.690 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:39.688+0000 7fc253e0f040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-19T14:20:39.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:39.709+0000 7fef6e7fc640 1 -- 172.21.15.57:0/4176013031 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mgrmap(e 130) v1 ==== 88957+0+0 (secure 0 0 0) 0x7fef6802cd50 con 0x7fef780cf850 2023-10-19T14:20:39.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:39.710+0000 7fef6e7fc640 1 -- 172.21.15.57:0/4176013031 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fef5c06afb0 msgr2=0x7fef5c06d470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T14:20:39.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:39.710+0000 7fef6e7fc640 1 --2- 172.21.15.57:0/4176013031 >> [v2:172.21.15.179:6832/3232482678,v1:172.21.15.179:6833/3232482678] conn(0x7fef5c06afb0 0x7fef5c06d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:39.740 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:20:39.737+0000 7f6056a37040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-19T14:20:39.796 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:20:39.794+0000 7f6056a37040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-19T14:20:39.852 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:39.849+0000 7fc1d2ca0640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:20:39.852 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:39.849+0000 7fc1d2ca0640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:20:39.852 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:39.849+0000 7fc1d2ca0640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:20:39.852 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:39.849+0000 7fc1d2ca0640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:20:39.853 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:39.849+0000 7fc1d2ca0640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:20:39.853 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:39.849+0000 7fc1cfc9a640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:20:39.853 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:39.849+0000 7fc1cfc9a640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:20:39.853 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:39.849+0000 7fc1cfc9a640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:20:39.853 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:39.849+0000 7fc1cfc9a640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:20:39.854 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:39.849+0000 7fc1cfc9a640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:20:39.861 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:39.858+0000 7fc1d2ca0640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:20:39.861 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:39.858+0000 7fc1d2ca0640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:20:39.862 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:39.858+0000 7fc1d2ca0640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:20:39.862 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:39.858+0000 7fc1d2ca0640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:20:39.862 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:39.858+0000 7fc1d2ca0640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:20:39.862 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:39.859+0000 7fc1cfc9a640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:20:39.862 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:39.859+0000 7fc1cfc9a640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:20:39.863 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:39.859+0000 7fc1cfc9a640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:20:39.863 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:39.859+0000 7fc1cfc9a640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:20:39.863 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:39.859+0000 7fc1cfc9a640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:20:39.908 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:20:39.905+0000 7f6056a37040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-19T14:20:40.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:40.714+0000 7fef6e7fc640 1 -- 172.21.15.57:0/4176013031 <== mon.0 v2:172.21.15.57:3300/0 8 ==== mgrmap(e 131) v1 ==== 89084+0+0 (secure 0 0 0) 0x7fef68077190 con 0x7fef780cf850 2023-10-19T14:20:40.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:40.714+0000 7fef6e7fc640 1 --2- 172.21.15.57:0/4176013031 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7fef5c035850 0x7fef5c077a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:40.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:40.714+0000 7fef6e7fc640 1 -- 172.21.15.57:0/4176013031 --> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fef780c7e80 con 0x7fef5c035850 2023-10-19T14:20:40.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:40.716+0000 7fef7d02f640 1 --2- 172.21.15.57:0/4176013031 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7fef5c035850 0x7fef5c077a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:40.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:40.718+0000 7fef7d02f640 1 --2- 172.21.15.57:0/4176013031 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7fef5c035850 0x7fef5c077a10 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fef780d0a80 tx=0x7fef74001100 comp rx=0 tx=0).ready entity=mgr.9422 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:40.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:40.771+0000 7fef6e7fc640 1 -- 172.21.15.57:0/4176013031 <== mgr.9422 v2:172.21.15.179:6832/4242302683 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+53 (secure 0 0 0) 0x7fef780c7e80 con 0x7fef5c035850 2023-10-19T14:20:40.775 INFO:teuthology.orchestra.run.smithi057.stdout:Schedule set for path mandatory_fs_argument_test_dir 2023-10-19T14:20:40.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:40.774+0000 7fef7fa94640 1 -- 172.21.15.57:0/4176013031 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7fef5c035850 msgr2=0x7fef5c077a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:40.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:40.774+0000 7fef7fa94640 1 --2- 172.21.15.57:0/4176013031 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7fef5c035850 0x7fef5c077a10 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fef780d0a80 tx=0x7fef74001100 comp rx=0 tx=0).stop 2023-10-19T14:20:40.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:40.774+0000 7fef7fa94640 1 -- 172.21.15.57:0/4176013031 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fef780cf850 msgr2=0x7fef780b53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:40.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:40.774+0000 7fef7fa94640 1 --2- 172.21.15.57:0/4176013031 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fef780cf850 0x7fef780b53c0 secure :-1 s=READY pgs=2446 cs=0 l=1 rev1=1 crypto rx=0x7fef68009780 tx=0x7fef680038a0 comp rx=0 tx=0).stop 2023-10-19T14:20:40.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:40.774+0000 7fef7e031640 1 -- 172.21.15.57:0/4176013031 reap_dead start 2023-10-19T14:20:40.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:40.774+0000 7fef7fa94640 1 -- 172.21.15.57:0/4176013031 shutdown_connections 2023-10-19T14:20:40.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:40.774+0000 7fef7fa94640 1 -- 172.21.15.57:0/4176013031 >> 172.21.15.57:0/4176013031 conn(0x7fef7809a030 msgr2=0x7fef78069f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:40.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:40.774+0000 7fef7fa94640 1 -- 172.21.15.57:0/4176013031 shutdown_connections 2023-10-19T14:20:40.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:40.774+0000 7fef7fa94640 1 -- 172.21.15.57:0/4176013031 wait complete. 2023-10-19T14:20:40.789 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path mandatory_fs_argument_test_dir 2023-10-19T14:20:40.789 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate mandatory_fs_argument_test_dir --fs backup_fs 1M 2023-10-19T14:20:40.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:40.903+0000 7fd598ef2640 1 -- 172.21.15.57:0/2784552081 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd59406d2d0 msgr2=0x7fd59406d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:40.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:40.903+0000 7fd598ef2640 1 --2- 172.21.15.57:0/2784552081 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd59406d2d0 0x7fd59406d6b0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7fd58c009830 tx=0x7fd58c02bd60 comp rx=0 tx=0).stop 2023-10-19T14:20:40.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:40.904+0000 7fd598ef2640 1 -- 172.21.15.57:0/2784552081 shutdown_connections 2023-10-19T14:20:40.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:40.904+0000 7fd598ef2640 1 --2- 172.21.15.57:0/2784552081 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd5940ad370 0x7fd5940ad800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:40.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:40.904+0000 7fd598ef2640 1 --2- 172.21.15.57:0/2784552081 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd59406d2d0 0x7fd59406d6b0 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:40.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:40.904+0000 7fd598ef2640 1 -- 172.21.15.57:0/2784552081 >> 172.21.15.57:0/2784552081 conn(0x7fd5940b0670 msgr2=0x7fd5940b0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:40.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:40.904+0000 7fd598ef2640 1 -- 172.21.15.57:0/2784552081 shutdown_connections 2023-10-19T14:20:40.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:40.904+0000 7fd598ef2640 1 -- 172.21.15.57:0/2784552081 wait complete. 2023-10-19T14:20:40.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:40.905+0000 7fd598ef2640 1 Processor -- start 2023-10-19T14:20:40.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:40.905+0000 7fd598ef2640 1 -- start start 2023-10-19T14:20:40.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:40.906+0000 7fd598ef2640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd59406d2d0 0x7fd59406bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:40.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:40.906+0000 7fd598ef2640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd59406f0a0 0x7fd594150940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:40.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:40.906+0000 7fd598ef2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5940ad370 0x7fd594150e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:40.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:40.906+0000 7fd598ef2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd59406a830 con 0x7fd5940ad370 2023-10-19T14:20:40.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:40.906+0000 7fd598ef2640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5941f1ab0 con 0x7fd59406f0a0 2023-10-19T14:20:40.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:40.906+0000 7fd598ef2640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd59406aa40 con 0x7fd59406d2d0 2023-10-19T14:20:40.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:40.906+0000 7fd592d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5940ad370 0x7fd594150e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:40.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:40.906+0000 7fd592d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5940ad370 0x7fd594150e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42276/0 (socket says 172.21.15.57:42276) 2023-10-19T14:20:40.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:40.906+0000 7fd592d9d640 1 -- 172.21.15.57:0/588892508 learned_addr learned my addr 172.21.15.57:0/588892508 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:40.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:40.906+0000 7fd591d9b640 1 --2- 172.21.15.57:0/588892508 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd59406f0a0 0x7fd594150940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:40.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:40.906+0000 7fd59259c640 1 --2- 172.21.15.57:0/588892508 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd59406d2d0 0x7fd59406bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:40.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:40.906+0000 7fd592d9d640 1 -- 172.21.15.57:0/588892508 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd59406d2d0 msgr2=0x7fd59406bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:40.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:40.906+0000 7fd592d9d640 1 --2- 172.21.15.57:0/588892508 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd59406d2d0 0x7fd59406bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:40.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:40.906+0000 7fd592d9d640 1 -- 172.21.15.57:0/588892508 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd59406f0a0 msgr2=0x7fd594150940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:40.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:40.906+0000 7fd592d9d640 1 --2- 172.21.15.57:0/588892508 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd59406f0a0 0x7fd594150940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:40.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:40.906+0000 7fd592d9d640 1 -- 172.21.15.57:0/588892508 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd58c009490 con 0x7fd5940ad370 2023-10-19T14:20:40.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:40.906+0000 7fd592d9d640 1 --2- 172.21.15.57:0/588892508 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5940ad370 0x7fd594150e80 secure :-1 s=READY pgs=2449 cs=0 l=1 rev1=1 crypto rx=0x7fd58400ea80 tx=0x7fd58400ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:40.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:40.906+0000 7fd58affd640 1 -- 172.21.15.57:0/588892508 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd58400cea0 con 0x7fd5940ad370 2023-10-19T14:20:40.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:40.906+0000 7fd598ef2640 1 -- 172.21.15.57:0/588892508 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd59406ad20 con 0x7fd5940ad370 2023-10-19T14:20:40.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:40.906+0000 7fd58affd640 1 -- 172.21.15.57:0/588892508 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fd584004b20 con 0x7fd5940ad370 2023-10-19T14:20:40.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:40.907+0000 7fd58affd640 1 -- 172.21.15.57:0/588892508 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd584005020 con 0x7fd5940ad370 2023-10-19T14:20:40.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:40.907+0000 7fd598ef2640 1 -- 172.21.15.57:0/588892508 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd59406b2e0 con 0x7fd5940ad370 2023-10-19T14:20:40.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:40.907+0000 7fd598ef2640 1 -- 172.21.15.57:0/588892508 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5940ce1d0 con 0x7fd5940ad370 2023-10-19T14:20:40.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:40.908+0000 7fd58affd640 1 -- 172.21.15.57:0/588892508 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 131) v1 ==== 89084+0+0 (secure 0 0 0) 0x7fd584005180 con 0x7fd5940ad370 2023-10-19T14:20:40.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:40.908+0000 7fd58affd640 1 --2- 172.21.15.57:0/588892508 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7fd57406ace0 0x7fd57406d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:40.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:40.908+0000 7fd58affd640 1 -- 172.21.15.57:0/588892508 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(228..228 src has 1..228) v4 ==== 7265+0+0 (secure 0 0 0) 0x7fd584014070 con 0x7fd5940ad370 2023-10-19T14:20:40.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:40.908+0000 7fd59259c640 1 --2- 172.21.15.57:0/588892508 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7fd57406ace0 0x7fd57406d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:40.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:40.909+0000 7fd59259c640 1 --2- 172.21.15.57:0/588892508 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7fd57406ace0 0x7fd57406d1a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fd58c036000 tx=0x7fd58c004730 comp rx=0 tx=0).ready entity=mgr.9422 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:40.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:40.913+0000 7fd58affd640 1 -- 172.21.15.57:0/588892508 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd584055380 con 0x7fd5940ad370 2023-10-19T14:20:41.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.110+0000 7fd598ef2640 1 -- 172.21.15.57:0/588892508 --> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fd5940c2630 con 0x7fd57406ace0 2023-10-19T14:20:41.150 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.147+0000 7fd58affd640 1 -- 172.21.15.57:0/588892508 <== mgr.9422 v2:172.21.15.179:6832/4242302683 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7fd58c0040c0 con 0x7fd57406ace0 2023-10-19T14:20:41.150 INFO:teuthology.orchestra.run.smithi057.stdout:Schedule activated for path mandatory_fs_argument_test_dir 2023-10-19T14:20:41.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.149+0000 7fd598ef2640 1 -- 172.21.15.57:0/588892508 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7fd57406ace0 msgr2=0x7fd57406d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:41.152 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.149+0000 7fd598ef2640 1 --2- 172.21.15.57:0/588892508 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7fd57406ace0 0x7fd57406d1a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fd58c036000 tx=0x7fd58c004730 comp rx=0 tx=0).stop 2023-10-19T14:20:41.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.149+0000 7fd598ef2640 1 -- 172.21.15.57:0/588892508 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5940ad370 msgr2=0x7fd594150e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:41.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.149+0000 7fd598ef2640 1 --2- 172.21.15.57:0/588892508 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5940ad370 0x7fd594150e80 secure :-1 s=READY pgs=2449 cs=0 l=1 rev1=1 crypto rx=0x7fd58400ea80 tx=0x7fd58400ef40 comp rx=0 tx=0).stop 2023-10-19T14:20:41.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.149+0000 7fd598ef2640 1 -- 172.21.15.57:0/588892508 shutdown_connections 2023-10-19T14:20:41.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.149+0000 7fd598ef2640 1 --2- 172.21.15.57:0/588892508 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7fd57406ace0 0x7fd57406d1a0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:41.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.149+0000 7fd598ef2640 1 --2- 172.21.15.57:0/588892508 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5940ad370 0x7fd594150e80 unknown :-1 s=CLOSED pgs=2449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:41.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.149+0000 7fd598ef2640 1 --2- 172.21.15.57:0/588892508 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd59406f0a0 0x7fd594150940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:41.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.149+0000 7fd598ef2640 1 --2- 172.21.15.57:0/588892508 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd59406d2d0 0x7fd59406bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:41.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.149+0000 7fd598ef2640 1 -- 172.21.15.57:0/588892508 >> 172.21.15.57:0/588892508 conn(0x7fd5940b0670 msgr2=0x7fd594158d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:41.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.149+0000 7fd598ef2640 1 -- 172.21.15.57:0/588892508 shutdown_connections 2023-10-19T14:20:41.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.149+0000 7fd598ef2640 1 -- 172.21.15.57:0/588892508 wait complete. 2023-10-19T14:20:41.163 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path mandatory_fs_argument_test_dir 2023-10-19T14:20:41.163 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention add mandatory_fs_argument_test_dir --fs backup_fs 1M 2023-10-19T14:20:41.279 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.276+0000 7f96a66b0640 1 -- 172.21.15.57:0/2321534790 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f96880047c0 msgr2=0x7f9688004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:41.280 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.276+0000 7f96a66b0640 1 --2- 172.21.15.57:0/2321534790 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f96880047c0 0x7f9688004c40 secure :-1 s=READY pgs=2451 cs=0 l=1 rev1=1 crypto rx=0x7f969000bb20 tx=0x7f96880057f0 comp rx=0 tx=0).stop 2023-10-19T14:20:41.280 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.276+0000 7f96a4c4d640 1 -- 172.21.15.57:0/2321534790 reap_dead start 2023-10-19T14:20:41.280 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.276+0000 7f96a66b0640 1 -- 172.21.15.57:0/2321534790 shutdown_connections 2023-10-19T14:20:41.280 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.276+0000 7f96a66b0640 1 -- 172.21.15.57:0/2321534790 >> 172.21.15.57:0/2321534790 conn(0x7f96a006fae0 msgr2=0x7f96a006fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:41.280 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.276+0000 7f96a66b0640 1 -- 172.21.15.57:0/2321534790 shutdown_connections 2023-10-19T14:20:41.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.276+0000 7f96a66b0640 1 -- 172.21.15.57:0/2321534790 wait complete. 2023-10-19T14:20:41.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.277+0000 7f96a66b0640 1 Processor -- start 2023-10-19T14:20:41.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.278+0000 7f96a66b0640 1 -- start start 2023-10-19T14:20:41.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.278+0000 7f96a66b0640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f96880047c0 0x7f96a0148d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:41.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.278+0000 7f96a66b0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f96a014d210 0x7f96a014d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:41.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.278+0000 7f96a66b0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f96a00adf00 0x7f96a00ae380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:41.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.278+0000 7f96a66b0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96a00ae8c0 con 0x7f96a00adf00 2023-10-19T14:20:41.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.278+0000 7f96a66b0640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96a014dd10 con 0x7f96a014d210 2023-10-19T14:20:41.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.278+0000 7f96a66b0640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96a014dff0 con 0x7f96880047c0 2023-10-19T14:20:41.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.278+0000 7f96a4c4d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f96a00adf00 0x7f96a00ae380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:41.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.278+0000 7f96a4c4d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f96a00adf00 0x7f96a00ae380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42302/0 (socket says 172.21.15.57:42302) 2023-10-19T14:20:41.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.278+0000 7f96a4c4d640 1 -- 172.21.15.57:0/3164785654 learned_addr learned my addr 172.21.15.57:0/3164785654 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:41.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.279+0000 7f969ffff640 1 --2- 172.21.15.57:0/3164785654 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f96880047c0 0x7f96a0148d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:41.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.279+0000 7f969f7fe640 1 --2- 172.21.15.57:0/3164785654 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f96a014d210 0x7f96a014d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:41.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.279+0000 7f96a4c4d640 1 -- 172.21.15.57:0/3164785654 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f96880047c0 msgr2=0x7f96a0148d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:41.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.279+0000 7f96a4c4d640 1 --2- 172.21.15.57:0/3164785654 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f96880047c0 0x7f96a0148d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:41.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.279+0000 7f96a4c4d640 1 -- 172.21.15.57:0/3164785654 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f96a014d210 msgr2=0x7f96a014d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:41.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.279+0000 7f96a4c4d640 1 --2- 172.21.15.57:0/3164785654 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f96a014d210 0x7f96a014d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:41.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.279+0000 7f96a4c4d640 1 -- 172.21.15.57:0/3164785654 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96a00b2330 con 0x7f96a00adf00 2023-10-19T14:20:41.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.279+0000 7f969ffff640 1 --2- 172.21.15.57:0/3164785654 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f96880047c0 0x7f96a0148d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:20:41.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.279+0000 7f969f7fe640 1 --2- 172.21.15.57:0/3164785654 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f96a014d210 0x7f96a014d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:20:41.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.279+0000 7f96a4c4d640 1 --2- 172.21.15.57:0/3164785654 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f96a00adf00 0x7f96a00ae380 secure :-1 s=READY pgs=2452 cs=0 l=1 rev1=1 crypto rx=0x7f96a00acd30 tx=0x7f9698006720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:41.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.279+0000 7f969cff9640 1 -- 172.21.15.57:0/3164785654 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f969800fc10 con 0x7f96a00adf00 2023-10-19T14:20:41.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.279+0000 7f969cff9640 1 -- 172.21.15.57:0/3164785654 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f969800fd90 con 0x7f96a00adf00 2023-10-19T14:20:41.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.279+0000 7f96a66b0640 1 -- 172.21.15.57:0/3164785654 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96a014e2d0 con 0x7f96a00adf00 2023-10-19T14:20:41.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.279+0000 7f969cff9640 1 -- 172.21.15.57:0/3164785654 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9698011690 con 0x7f96a00adf00 2023-10-19T14:20:41.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.279+0000 7f96a66b0640 1 -- 172.21.15.57:0/3164785654 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96a014f570 con 0x7f96a00adf00 2023-10-19T14:20:41.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.280+0000 7f96a66b0640 1 -- 172.21.15.57:0/3164785654 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96a00690e0 con 0x7f96a00adf00 2023-10-19T14:20:41.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.281+0000 7f969cff9640 1 -- 172.21.15.57:0/3164785654 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 131) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f96980117f0 con 0x7f96a00adf00 2023-10-19T14:20:41.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.281+0000 7f969cff9640 1 --2- 172.21.15.57:0/3164785654 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7f968c06ad10 0x7f968c06d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:41.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.281+0000 7f969cff9640 1 -- 172.21.15.57:0/3164785654 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(228..228 src has 1..228) v4 ==== 7265+0+0 (secure 0 0 0) 0x7f969808a2f0 con 0x7f96a00adf00 2023-10-19T14:20:41.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.281+0000 7f969ffff640 1 --2- 172.21.15.57:0/3164785654 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7f968c06ad10 0x7f968c06d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:41.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.282+0000 7f969ffff640 1 --2- 172.21.15.57:0/3164785654 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7f968c06ad10 0x7f968c06d1d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f9694006570 tx=0x7f9694006b30 comp rx=0 tx=0).ready entity=mgr.9422 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:41.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.285+0000 7f969cff9640 1 -- 172.21.15.57:0/3164785654 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f96980550c0 con 0x7f96a00adf00 2023-10-19T14:20:41.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.483+0000 7f96a66b0640 1 -- 172.21.15.57:0/3164785654 --> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "retention_spec_or_period": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f96a015adc0 con 0x7f968c06ad10 2023-10-19T14:20:41.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.532+0000 7f969cff9640 1 -- 172.21.15.57:0/3164785654 <== mgr.9422 v2:172.21.15.179:6832/4242302683 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+55 (secure 0 0 0) 0x7f96a015adc0 con 0x7f968c06ad10 2023-10-19T14:20:41.535 INFO:teuthology.orchestra.run.smithi057.stdout:Retention added to path mandatory_fs_argument_test_dir 2023-10-19T14:20:41.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.533+0000 7f96a66b0640 1 -- 172.21.15.57:0/3164785654 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7f968c06ad10 msgr2=0x7f968c06d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:41.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.533+0000 7f96a66b0640 1 --2- 172.21.15.57:0/3164785654 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7f968c06ad10 0x7f968c06d1d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f9694006570 tx=0x7f9694006b30 comp rx=0 tx=0).stop 2023-10-19T14:20:41.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.533+0000 7f96a66b0640 1 -- 172.21.15.57:0/3164785654 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f96a00adf00 msgr2=0x7f96a00ae380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:41.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.533+0000 7f96a66b0640 1 --2- 172.21.15.57:0/3164785654 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f96a00adf00 0x7f96a00ae380 secure :-1 s=READY pgs=2452 cs=0 l=1 rev1=1 crypto rx=0x7f96a00acd30 tx=0x7f9698006720 comp rx=0 tx=0).stop 2023-10-19T14:20:41.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.534+0000 7f96a66b0640 1 -- 172.21.15.57:0/3164785654 shutdown_connections 2023-10-19T14:20:41.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.534+0000 7f96a66b0640 1 --2- 172.21.15.57:0/3164785654 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7f968c06ad10 0x7f968c06d1d0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:41.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.534+0000 7f96a66b0640 1 --2- 172.21.15.57:0/3164785654 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f96a00adf00 0x7f96a00ae380 unknown :-1 s=CLOSED pgs=2452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:41.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.534+0000 7f96a66b0640 1 --2- 172.21.15.57:0/3164785654 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f96a014d210 0x7f96a014d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:41.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.534+0000 7f96a66b0640 1 --2- 172.21.15.57:0/3164785654 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f96880047c0 0x7f96a0148d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:41.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.534+0000 7f96a66b0640 1 -- 172.21.15.57:0/3164785654 >> 172.21.15.57:0/3164785654 conn(0x7f96a006fae0 msgr2=0x7f96a0159620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:41.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.534+0000 7f96a66b0640 1 -- 172.21.15.57:0/3164785654 shutdown_connections 2023-10-19T14:20:41.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.534+0000 7f96a66b0640 1 -- 172.21.15.57:0/3164785654 wait complete. 2023-10-19T14:20:41.547 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention added to path mandatory_fs_argument_test_dir 2023-10-19T14:20:41.548 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list mandatory_fs_argument_test_dir --fs backup_fs --format json 2023-10-19T14:20:41.661 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.658+0000 7f5f8f168640 1 -- 172.21.15.57:0/622494789 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5f880cb980 msgr2=0x7f5f8813db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:41.661 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.658+0000 7f5f8f168640 1 --2- 172.21.15.57:0/622494789 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5f880cb980 0x7f5f8813db80 secure :-1 s=READY pgs=2453 cs=0 l=1 rev1=1 crypto rx=0x7f5f8806fa40 tx=0x7f5f7c02bd90 comp rx=0 tx=0).stop 2023-10-19T14:20:41.661 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.658+0000 7f5f8f168640 1 -- 172.21.15.57:0/622494789 shutdown_connections 2023-10-19T14:20:41.662 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.658+0000 7f5f8f168640 1 --2- 172.21.15.57:0/622494789 >> v2:172.21.15.179:3300/0 conn(0x7f5f880c96b0 0x7f5f880c9b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:41.662 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.658+0000 7f5f8f168640 1 --2- 172.21.15.57:0/622494789 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5f8813d350 0x7f5f880cbd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:41.662 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.658+0000 7f5f8f168640 1 --2- 172.21.15.57:0/622494789 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5f880cb980 0x7f5f8813db80 unknown :-1 s=CLOSED pgs=2453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:41.662 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.658+0000 7f5f8f168640 1 -- 172.21.15.57:0/622494789 >> 172.21.15.57:0/622494789 conn(0x7f5f88069930 msgr2=0x7f5f88069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:41.662 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.658+0000 7f5f8f168640 1 -- 172.21.15.57:0/622494789 shutdown_connections 2023-10-19T14:20:41.663 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.658+0000 7f5f8f168640 1 -- 172.21.15.57:0/622494789 wait complete. 2023-10-19T14:20:41.665 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.659+0000 7f5f8f168640 1 Processor -- start 2023-10-19T14:20:41.665 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.659+0000 7f5f8f168640 1 -- start start 2023-10-19T14:20:41.665 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.660+0000 7f5f8f168640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5f880c96b0 0x7f5f881538d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:41.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.660+0000 7f5f8f168640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5f880cb980 0x7f5f88139410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:41.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.660+0000 7f5f8f168640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5f88139950 0x7f5f881efb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:41.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.660+0000 7f5f8f168640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f88136b30 con 0x7f5f880cb980 2023-10-19T14:20:41.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.660+0000 7f5f8f168640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f88139dc0 con 0x7f5f880c96b0 2023-10-19T14:20:41.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.660+0000 7f5f8f168640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f881541e0 con 0x7f5f88139950 2023-10-19T14:20:41.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.660+0000 7f5f87fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5f880cb980 0x7f5f88139410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:41.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.660+0000 7f5f87fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5f880cb980 0x7f5f88139410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42314/0 (socket says 172.21.15.57:42314) 2023-10-19T14:20:41.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.660+0000 7f5f87fff640 1 -- 172.21.15.57:0/157716128 learned_addr learned my addr 172.21.15.57:0/157716128 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:41.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.660+0000 7f5f87fff640 1 -- 172.21.15.57:0/157716128 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5f88139950 msgr2=0x7f5f881efb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:41.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.660+0000 7f5f87fff640 1 --2- 172.21.15.57:0/157716128 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5f88139950 0x7f5f881efb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:41.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.660+0000 7f5f87fff640 1 -- 172.21.15.57:0/157716128 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5f880c96b0 msgr2=0x7f5f881538d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:41.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.660+0000 7f5f8cf04640 1 --2- 172.21.15.57:0/157716128 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5f880c96b0 0x7f5f881538d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:41.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.660+0000 7f5f87fff640 1 --2- 172.21.15.57:0/157716128 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5f880c96b0 0x7f5f881538d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:41.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.660+0000 7f5f87fff640 1 -- 172.21.15.57:0/157716128 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f7c009560 con 0x7f5f880cb980 2023-10-19T14:20:41.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.660+0000 7f5f8cf04640 1 --2- 172.21.15.57:0/157716128 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5f880c96b0 0x7f5f881538d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:20:41.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.660+0000 7f5f87fff640 1 --2- 172.21.15.57:0/157716128 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5f880cb980 0x7f5f88139410 secure :-1 s=READY pgs=2454 cs=0 l=1 rev1=1 crypto rx=0x7f5f7400c9a0 tx=0x7f5f7400ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:41.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.661+0000 7f5f857fa640 1 -- 172.21.15.57:0/157716128 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f74007bb0 con 0x7f5f880cb980 2023-10-19T14:20:41.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.661+0000 7f5f857fa640 1 -- 172.21.15.57:0/157716128 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f5f74007e20 con 0x7f5f880cb980 2023-10-19T14:20:41.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.661+0000 7f5f857fa640 1 -- 172.21.15.57:0/157716128 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f74005280 con 0x7f5f880cb980 2023-10-19T14:20:41.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.661+0000 7f5f8f168640 1 -- 172.21.15.57:0/157716128 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f881544c0 con 0x7f5f880cb980 2023-10-19T14:20:41.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.661+0000 7f5f8f168640 1 -- 172.21.15.57:0/157716128 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f88154a80 con 0x7f5f880cb980 2023-10-19T14:20:41.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.661+0000 7f5f8f168640 1 -- 172.21.15.57:0/157716128 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f880690e0 con 0x7f5f880cb980 2023-10-19T14:20:41.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.662+0000 7f5f857fa640 1 -- 172.21.15.57:0/157716128 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 131) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f5f740053e0 con 0x7f5f880cb980 2023-10-19T14:20:41.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.662+0000 7f5f857fa640 1 --2- 172.21.15.57:0/157716128 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7f5f6c06ace0 0x7f5f6c06d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:41.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.662+0000 7f5f857fa640 1 -- 172.21.15.57:0/157716128 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(228..228 src has 1..228) v4 ==== 7265+0+0 (secure 0 0 0) 0x7f5f74089dc0 con 0x7f5f880cb980 2023-10-19T14:20:41.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.663+0000 7f5f8cf04640 1 --2- 172.21.15.57:0/157716128 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7f5f6c06ace0 0x7f5f6c06d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:41.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.663+0000 7f5f8cf04640 1 --2- 172.21.15.57:0/157716128 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7f5f6c06ace0 0x7f5f6c06d1a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f5f7c035000 tx=0x7f5f7c004730 comp rx=0 tx=0).ready entity=mgr.9422 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:41.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.666+0000 7f5f857fa640 1 -- 172.21.15.57:0/157716128 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f5f74054b90 con 0x7f5f880cb980 2023-10-19T14:20:41.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.863+0000 7f5f8f168640 1 -- 172.21.15.57:0/157716128 --> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5f8815c1e0 con 0x7f5f6c06ace0 2023-10-19T14:20:41.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.870+0000 7f5f857fa640 1 -- 172.21.15.57:0/157716128 <== mgr.9422 v2:172.21.15.179:6832/4242302683 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+88 (secure 0 0 0) 0x7f5f7c004360 con 0x7f5f6c06ace0 2023-10-19T14:20:41.873 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:20:41.873 INFO:teuthology.orchestra.run.smithi057.stdout:{"path": "mandatory_fs_argument_test_dir", "schedule": ["1M"], "retention": [{"M": 1}]} 2023-10-19T14:20:41.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.871+0000 7f5f8f168640 1 -- 172.21.15.57:0/157716128 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7f5f6c06ace0 msgr2=0x7f5f6c06d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:41.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.871+0000 7f5f8f168640 1 --2- 172.21.15.57:0/157716128 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7f5f6c06ace0 0x7f5f6c06d1a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f5f7c035000 tx=0x7f5f7c004730 comp rx=0 tx=0).stop 2023-10-19T14:20:41.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.871+0000 7f5f8f168640 1 -- 172.21.15.57:0/157716128 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5f880cb980 msgr2=0x7f5f88139410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:41.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.871+0000 7f5f8f168640 1 --2- 172.21.15.57:0/157716128 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5f880cb980 0x7f5f88139410 secure :-1 s=READY pgs=2454 cs=0 l=1 rev1=1 crypto rx=0x7f5f7400c9a0 tx=0x7f5f7400ce60 comp rx=0 tx=0).stop 2023-10-19T14:20:41.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.871+0000 7f5f8f168640 1 -- 172.21.15.57:0/157716128 shutdown_connections 2023-10-19T14:20:41.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.871+0000 7f5f8f168640 1 --2- 172.21.15.57:0/157716128 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7f5f6c06ace0 0x7f5f6c06d1a0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:41.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.871+0000 7f5f8f168640 1 --2- 172.21.15.57:0/157716128 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5f88139950 0x7f5f881efb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:41.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.871+0000 7f5f8f168640 1 --2- 172.21.15.57:0/157716128 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5f880cb980 0x7f5f88139410 unknown :-1 s=CLOSED pgs=2454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:41.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.871+0000 7f5f8f168640 1 --2- 172.21.15.57:0/157716128 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5f880c96b0 0x7f5f881538d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:41.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.871+0000 7f5f8f168640 1 -- 172.21.15.57:0/157716128 >> 172.21.15.57:0/157716128 conn(0x7f5f88069930 msgr2=0x7f5f881584e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:41.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.872+0000 7f5f8f168640 1 -- 172.21.15.57:0/157716128 shutdown_connections 2023-10-19T14:20:41.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.872+0000 7f5f8f168640 1 -- 172.21.15.57:0/157716128 wait complete. 2023-10-19T14:20:41.885 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "mandatory_fs_argument_test_dir", "schedule": ["1M"], "retention": [{"M": 1}]} 2023-10-19T14:20:41.885 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status mandatory_fs_argument_test_dir --fs backup_fs --format json 2023-10-19T14:20:41.999 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.996+0000 7fa49e41a640 1 -- 172.21.15.57:0/1829424305 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa498156790 msgr2=0x7fa4980cc990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:41.999 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.996+0000 7fa49e41a640 1 --2- 172.21.15.57:0/1829424305 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa498156790 0x7fa4980cc990 secure :-1 s=READY pgs=2456 cs=0 l=1 rev1=1 crypto rx=0x7fa48c009830 tx=0x7fa48c02bdf0 comp rx=0 tx=0).stop 2023-10-19T14:20:41.999 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.996+0000 7fa49e41a640 1 -- 172.21.15.57:0/1829424305 shutdown_connections 2023-10-19T14:20:42.000 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.996+0000 7fa49e41a640 1 --2- 172.21.15.57:0/1829424305 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa498156790 0x7fa4980cc990 unknown :-1 s=CLOSED pgs=2456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:42.000 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.996+0000 7fa49e41a640 1 -- 172.21.15.57:0/1829424305 >> 172.21.15.57:0/1829424305 conn(0x7fa49806fae0 msgr2=0x7fa49806fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:42.000 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.996+0000 7fa49e41a640 1 -- 172.21.15.57:0/1829424305 shutdown_connections 2023-10-19T14:20:42.000 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.996+0000 7fa49e41a640 1 -- 172.21.15.57:0/1829424305 wait complete. 2023-10-19T14:20:42.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.997+0000 7fa49e41a640 1 Processor -- start 2023-10-19T14:20:42.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.997+0000 7fa49e41a640 1 -- start start 2023-10-19T14:20:42.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.998+0000 7fa49e41a640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa49806a730 0x7fa4981e4900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:42.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.998+0000 7fa49e41a640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa4980ac870 0x7fa49813bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:42.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.998+0000 7fa49e41a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa498156790 0x7fa49813c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:42.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.998+0000 7fa49e41a640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa49814d5f0 con 0x7fa498156790 2023-10-19T14:20:42.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.998+0000 7fa49e41a640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa49814de00 con 0x7fa4980ac870 2023-10-19T14:20:42.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.998+0000 7fa49e41a640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa49814d840 con 0x7fa49806a730 2023-10-19T14:20:42.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.998+0000 7fa49c9b7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa498156790 0x7fa49813c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:42.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.998+0000 7fa49c9b7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa498156790 0x7fa49813c1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42336/0 (socket says 172.21.15.57:42336) 2023-10-19T14:20:42.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.998+0000 7fa49c9b7640 1 -- 172.21.15.57:0/1884025649 learned_addr learned my addr 172.21.15.57:0/1884025649 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:42.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.998+0000 7fa4977fe640 1 --2- 172.21.15.57:0/1884025649 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa4980ac870 0x7fa49813bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:42.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.998+0000 7fa497fff640 1 --2- 172.21.15.57:0/1884025649 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa49806a730 0x7fa4981e4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:42.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.998+0000 7fa49c9b7640 1 -- 172.21.15.57:0/1884025649 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa49806a730 msgr2=0x7fa4981e4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:42.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.998+0000 7fa49c9b7640 1 --2- 172.21.15.57:0/1884025649 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa49806a730 0x7fa4981e4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:42.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.998+0000 7fa49c9b7640 1 -- 172.21.15.57:0/1884025649 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa4980ac870 msgr2=0x7fa49813bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:42.006 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.998+0000 7fa49c9b7640 1 --2- 172.21.15.57:0/1884025649 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa4980ac870 0x7fa49813bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:42.006 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.998+0000 7fa49c9b7640 1 -- 172.21.15.57:0/1884025649 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa48c009490 con 0x7fa498156790 2023-10-19T14:20:42.006 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.998+0000 7fa4977fe640 1 --2- 172.21.15.57:0/1884025649 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa4980ac870 0x7fa49813bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:20:42.006 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.998+0000 7fa49c9b7640 1 --2- 172.21.15.57:0/1884025649 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa498156790 0x7fa49813c1c0 secure :-1 s=READY pgs=2457 cs=0 l=1 rev1=1 crypto rx=0x7fa49000c7b0 tx=0x7fa49000cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:42.006 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.998+0000 7fa494ff9640 1 -- 172.21.15.57:0/1884025649 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa490003ba0 con 0x7fa498156790 2023-10-19T14:20:42.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.998+0000 7fa494ff9640 1 -- 172.21.15.57:0/1884025649 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fa490003d00 con 0x7fa498156790 2023-10-19T14:20:42.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.999+0000 7fa494ff9640 1 -- 172.21.15.57:0/1884025649 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4900052d0 con 0x7fa498156790 2023-10-19T14:20:42.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.999+0000 7fa49e41a640 1 -- 172.21.15.57:0/1884025649 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa49814db20 con 0x7fa498156790 2023-10-19T14:20:42.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.999+0000 7fa49e41a640 1 -- 172.21.15.57:0/1884025649 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4981e43e0 con 0x7fa498156790 2023-10-19T14:20:42.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:41.999+0000 7fa49e41a640 1 -- 172.21.15.57:0/1884025649 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4981e4600 con 0x7fa498156790 2023-10-19T14:20:42.008 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.000+0000 7fa494ff9640 1 -- 172.21.15.57:0/1884025649 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 131) v1 ==== 89084+0+0 (secure 0 0 0) 0x7fa490005430 con 0x7fa498156790 2023-10-19T14:20:42.008 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.000+0000 7fa494ff9640 1 --2- 172.21.15.57:0/1884025649 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7fa48406ace0 0x7fa48406d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:42.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.000+0000 7fa494ff9640 1 -- 172.21.15.57:0/1884025649 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(228..228 src has 1..228) v4 ==== 7265+0+0 (secure 0 0 0) 0x7fa490089820 con 0x7fa498156790 2023-10-19T14:20:42.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.000+0000 7fa497fff640 1 --2- 172.21.15.57:0/1884025649 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7fa48406ace0 0x7fa48406d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:42.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.001+0000 7fa497fff640 1 --2- 172.21.15.57:0/1884025649 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7fa48406ace0 0x7fa48406d1a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa48c036000 tx=0x7fa48c004730 comp rx=0 tx=0).ready entity=mgr.9422 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:42.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.005+0000 7fa494ff9640 1 -- 172.21.15.57:0/1884025649 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa4900545f0 con 0x7fa498156790 2023-10-19T14:20:42.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.200+0000 7fa49e41a640 1 -- 172.21.15.57:0/1884025649 --> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa49806abc0 con 0x7fa48406ace0 2023-10-19T14:20:42.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.208+0000 7fa494ff9640 1 -- 172.21.15.57:0/1884025649 <== mgr.9422 v2:172.21.15.179:6832/4242302683 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+338 (secure 0 0 0) 0x7fa48c004140 con 0x7fa48406ace0 2023-10-19T14:20:42.211 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:20:42.211 INFO:teuthology.orchestra.run.smithi057.stdout:[{"fs": "backup_fs", "subvol": null, "path": "mandatory_fs_argument_test_dir", "rel_path": "mandatory_fs_argument_test_dir", "schedule": "1M", "retention": {"M": 1}, "start": "2023-10-19T00:00:00", "created": "2023-10-19T14:20:40", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true}] 2023-10-19T14:20:42.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.210+0000 7fa49e41a640 1 -- 172.21.15.57:0/1884025649 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7fa48406ace0 msgr2=0x7fa48406d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:42.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.210+0000 7fa49e41a640 1 --2- 172.21.15.57:0/1884025649 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7fa48406ace0 0x7fa48406d1a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa48c036000 tx=0x7fa48c004730 comp rx=0 tx=0).stop 2023-10-19T14:20:42.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.210+0000 7fa49e41a640 1 -- 172.21.15.57:0/1884025649 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa498156790 msgr2=0x7fa49813c1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:42.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.210+0000 7fa49e41a640 1 --2- 172.21.15.57:0/1884025649 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa498156790 0x7fa49813c1c0 secure :-1 s=READY pgs=2457 cs=0 l=1 rev1=1 crypto rx=0x7fa49000c7b0 tx=0x7fa49000cbb0 comp rx=0 tx=0).stop 2023-10-19T14:20:42.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.210+0000 7fa49e41a640 1 -- 172.21.15.57:0/1884025649 shutdown_connections 2023-10-19T14:20:42.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.210+0000 7fa49e41a640 1 --2- 172.21.15.57:0/1884025649 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7fa48406ace0 0x7fa48406d1a0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:42.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.210+0000 7fa49e41a640 1 --2- 172.21.15.57:0/1884025649 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa498156790 0x7fa49813c1c0 unknown :-1 s=CLOSED pgs=2457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:42.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.210+0000 7fa49e41a640 1 --2- 172.21.15.57:0/1884025649 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa4980ac870 0x7fa49813bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:42.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.210+0000 7fa49e41a640 1 --2- 172.21.15.57:0/1884025649 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa49806a730 0x7fa4981e4900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:42.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.210+0000 7fa49e41a640 1 -- 172.21.15.57:0/1884025649 >> 172.21.15.57:0/1884025649 conn(0x7fa49806fae0 msgr2=0x7fa498157f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:42.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.210+0000 7fa49e41a640 1 -- 172.21.15.57:0/1884025649 shutdown_connections 2023-10-19T14:20:42.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.210+0000 7fa49e41a640 1 -- 172.21.15.57:0/1884025649 wait complete. 2023-10-19T14:20:42.223 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "backup_fs", "subvol": null, "path": "mandatory_fs_argument_test_dir", "rel_path": "mandatory_fs_argument_test_dir", "schedule": "1M", "retention": {"M": 1}, "start": "2023-10-19T00:00:00", "created": "2023-10-19T14:20:40", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true}] 2023-10-19T14:20:42.224 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention remove mandatory_fs_argument_test_dir --fs backup_fs 1M 2023-10-19T14:20:42.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.336+0000 7f0d242bb640 1 -- 172.21.15.57:0/264273003 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0d1c0b0530 msgr2=0x7f0d1c0b09c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:42.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.336+0000 7f0d242bb640 1 --2- 172.21.15.57:0/264273003 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0d1c0b0530 0x7f0d1c0b09c0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c005fd0 tx=0x7f0d0c02c0e0 comp rx=0 tx=0).stop 2023-10-19T14:20:42.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.336+0000 7f0d242bb640 1 -- 172.21.15.57:0/264273003 shutdown_connections 2023-10-19T14:20:42.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.336+0000 7f0d242bb640 1 --2- 172.21.15.57:0/264273003 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0d1c0b0530 0x7f0d1c0b09c0 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:42.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.336+0000 7f0d242bb640 1 --2- 172.21.15.57:0/264273003 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0d1c0afa20 0x7f0d1c158300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:42.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.336+0000 7f0d242bb640 1 -- 172.21.15.57:0/264273003 >> 172.21.15.57:0/264273003 conn(0x7f0d1c069930 msgr2=0x7f0d1c069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:42.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.336+0000 7f0d242bb640 1 -- 172.21.15.57:0/264273003 shutdown_connections 2023-10-19T14:20:42.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.336+0000 7f0d242bb640 1 -- 172.21.15.57:0/264273003 wait complete. 2023-10-19T14:20:42.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.337+0000 7f0d242bb640 1 Processor -- start 2023-10-19T14:20:42.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.337+0000 7f0d242bb640 1 -- start start 2023-10-19T14:20:42.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.338+0000 7f0d242bb640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0d1c0afa20 0x7f0d1c13e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:42.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.338+0000 7f0d242bb640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0d1c0b0530 0x7f0d1c136fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:42.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.338+0000 7f0d242bb640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0d1c0cbd20 0x7f0d1c146190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:42.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.338+0000 7f0d242bb640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d1c0ad430 con 0x7f0d1c0b0530 2023-10-19T14:20:42.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.338+0000 7f0d242bb640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d1c13ee70 con 0x7f0d1c0cbd20 2023-10-19T14:20:42.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.338+0000 7f0d242bb640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d1c13bd70 con 0x7f0d1c0afa20 2023-10-19T14:20:42.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.338+0000 7f0d21856640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0d1c0b0530 0x7f0d1c136fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:42.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.338+0000 7f0d21856640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0d1c0b0530 0x7f0d1c136fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42348/0 (socket says 172.21.15.57:42348) 2023-10-19T14:20:42.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.338+0000 7f0d21856640 1 -- 172.21.15.57:0/1624345462 learned_addr learned my addr 172.21.15.57:0/1624345462 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:42.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.338+0000 7f0d22858640 1 --2- 172.21.15.57:0/1624345462 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0d1c0cbd20 0x7f0d1c146190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:42.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.338+0000 7f0d22057640 1 --2- 172.21.15.57:0/1624345462 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0d1c0afa20 0x7f0d1c13e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:42.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.339+0000 7f0d21856640 1 -- 172.21.15.57:0/1624345462 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0d1c0afa20 msgr2=0x7f0d1c13e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:42.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.339+0000 7f0d21856640 1 --2- 172.21.15.57:0/1624345462 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0d1c0afa20 0x7f0d1c13e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:42.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.339+0000 7f0d21856640 1 -- 172.21.15.57:0/1624345462 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0d1c0cbd20 msgr2=0x7f0d1c146190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:42.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.339+0000 7f0d21856640 1 --2- 172.21.15.57:0/1624345462 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0d1c0cbd20 0x7f0d1c146190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:42.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.339+0000 7f0d21856640 1 -- 172.21.15.57:0/1624345462 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d10009490 con 0x7f0d1c0b0530 2023-10-19T14:20:42.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.339+0000 7f0d21856640 1 --2- 172.21.15.57:0/1624345462 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0d1c0b0530 0x7f0d1c136fd0 secure :-1 s=READY pgs=2458 cs=0 l=1 rev1=1 crypto rx=0x7f0d1000ea60 tx=0x7f0d1000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:42.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.339+0000 7f0d1affd640 1 -- 172.21.15.57:0/1624345462 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d1000cde0 con 0x7f0d1c0b0530 2023-10-19T14:20:42.347 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.339+0000 7f0d1affd640 1 -- 172.21.15.57:0/1624345462 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f0d10004020 con 0x7f0d1c0b0530 2023-10-19T14:20:42.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.339+0000 7f0d1affd640 1 -- 172.21.15.57:0/1624345462 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d10004410 con 0x7f0d1c0b0530 2023-10-19T14:20:42.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.339+0000 7f0d242bb640 1 -- 172.21.15.57:0/1624345462 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d0c009840 con 0x7f0d1c0b0530 2023-10-19T14:20:42.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.339+0000 7f0d242bb640 1 -- 172.21.15.57:0/1624345462 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d1c13c330 con 0x7f0d1c0b0530 2023-10-19T14:20:42.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.340+0000 7f0d242bb640 1 -- 172.21.15.57:0/1624345462 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d1c0690e0 con 0x7f0d1c0b0530 2023-10-19T14:20:42.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.341+0000 7f0d1affd640 1 -- 172.21.15.57:0/1624345462 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 131) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f0d100035f0 con 0x7f0d1c0b0530 2023-10-19T14:20:42.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.341+0000 7f0d1affd640 1 --2- 172.21.15.57:0/1624345462 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7f0d0806ace0 0x7f0d0806d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:42.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.341+0000 7f0d1affd640 1 -- 172.21.15.57:0/1624345462 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(228..228 src has 1..228) v4 ==== 7265+0+0 (secure 0 0 0) 0x7f0d10014070 con 0x7f0d1c0b0530 2023-10-19T14:20:42.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.341+0000 7f0d22057640 1 --2- 172.21.15.57:0/1624345462 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7f0d0806ace0 0x7f0d0806d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:42.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.342+0000 7f0d22057640 1 --2- 172.21.15.57:0/1624345462 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7f0d0806ace0 0x7f0d0806d1a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f0d1c13e790 tx=0x7f0d14007e50 comp rx=0 tx=0).ready entity=mgr.9422 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:42.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.345+0000 7f0d1affd640 1 -- 172.21.15.57:0/1624345462 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0d10055210 con 0x7f0d1c0b0530 2023-10-19T14:20:42.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.544+0000 7f0d242bb640 1 -- 172.21.15.57:0/1624345462 --> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention remove", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "retention_spec_or_period": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f0d1c0cc100 con 0x7f0d0806ace0 2023-10-19T14:20:42.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.603+0000 7f0d1affd640 1 -- 172.21.15.57:0/1624345462 <== mgr.9422 v2:172.21.15.179:6832/4242302683 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f0d14005360 con 0x7f0d0806ace0 2023-10-19T14:20:42.607 INFO:teuthology.orchestra.run.smithi057.stdout:Retention removed from path mandatory_fs_argument_test_dir 2023-10-19T14:20:42.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.605+0000 7f0d242bb640 1 -- 172.21.15.57:0/1624345462 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7f0d0806ace0 msgr2=0x7f0d0806d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:42.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.605+0000 7f0d242bb640 1 --2- 172.21.15.57:0/1624345462 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7f0d0806ace0 0x7f0d0806d1a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f0d1c13e790 tx=0x7f0d14007e50 comp rx=0 tx=0).stop 2023-10-19T14:20:42.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.605+0000 7f0d242bb640 1 -- 172.21.15.57:0/1624345462 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0d1c0b0530 msgr2=0x7f0d1c136fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:42.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.605+0000 7f0d242bb640 1 --2- 172.21.15.57:0/1624345462 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0d1c0b0530 0x7f0d1c136fd0 secure :-1 s=READY pgs=2458 cs=0 l=1 rev1=1 crypto rx=0x7f0d1000ea60 tx=0x7f0d1000ef20 comp rx=0 tx=0).stop 2023-10-19T14:20:42.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.605+0000 7f0d242bb640 1 -- 172.21.15.57:0/1624345462 shutdown_connections 2023-10-19T14:20:42.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.605+0000 7f0d242bb640 1 --2- 172.21.15.57:0/1624345462 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7f0d0806ace0 0x7f0d0806d1a0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:42.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.605+0000 7f0d242bb640 1 --2- 172.21.15.57:0/1624345462 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0d1c0cbd20 0x7f0d1c146190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:42.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.605+0000 7f0d242bb640 1 --2- 172.21.15.57:0/1624345462 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0d1c0b0530 0x7f0d1c136fd0 unknown :-1 s=CLOSED pgs=2458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:42.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.606+0000 7f0d242bb640 1 --2- 172.21.15.57:0/1624345462 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0d1c0afa20 0x7f0d1c13e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:42.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.606+0000 7f0d242bb640 1 -- 172.21.15.57:0/1624345462 >> 172.21.15.57:0/1624345462 conn(0x7f0d1c069930 msgr2=0x7f0d1c14a110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:42.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.606+0000 7f0d242bb640 1 -- 172.21.15.57:0/1624345462 shutdown_connections 2023-10-19T14:20:42.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.606+0000 7f0d242bb640 1 -- 172.21.15.57:0/1624345462 wait complete. 2023-10-19T14:20:42.619 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention removed from path mandatory_fs_argument_test_dir 2023-10-19T14:20:42.619 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate mandatory_fs_argument_test_dir --fs backup_fs 1M 2023-10-19T14:20:42.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.734+0000 7f872d38a640 1 -- 172.21.15.57:0/684694346 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f872815b2a0 msgr2=0x7f87280cf720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:42.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.734+0000 7f872d38a640 1 --2- 172.21.15.57:0/684694346 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f872815b2a0 0x7f87280cf720 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f8710009830 tx=0x7f871002bcd0 comp rx=0 tx=0).stop 2023-10-19T14:20:42.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.735+0000 7f872d38a640 1 -- 172.21.15.57:0/684694346 shutdown_connections 2023-10-19T14:20:42.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.735+0000 7f872d38a640 1 --2- 172.21.15.57:0/684694346 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f872815b2a0 0x7f87280cf720 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:42.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.735+0000 7f872d38a640 1 --2- 172.21.15.57:0/684694346 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f87280cf340 0x7f87280d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:42.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.735+0000 7f872d38a640 1 -- 172.21.15.57:0/684694346 >> 172.21.15.57:0/684694346 conn(0x7f87280d0d60 msgr2=0x7f87280d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:42.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.735+0000 7f872d38a640 1 -- 172.21.15.57:0/684694346 shutdown_connections 2023-10-19T14:20:42.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.735+0000 7f872d38a640 1 -- 172.21.15.57:0/684694346 wait complete. 2023-10-19T14:20:42.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.736+0000 7f872d38a640 1 Processor -- start 2023-10-19T14:20:42.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.736+0000 7f872d38a640 1 -- start start 2023-10-19T14:20:42.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.737+0000 7f872d38a640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f87280ceb40 0x7f87280c8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:42.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.737+0000 7f872d38a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f87280cf340 0x7f87280b4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:42.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.737+0000 7f872d38a640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f872815b2a0 0x7f87280b5400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:42.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.737+0000 7f872d38a640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87280c9ab0 con 0x7f87280cf340 2023-10-19T14:20:42.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.737+0000 7f872d38a640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87280c94f0 con 0x7f87280ceb40 2023-10-19T14:20:42.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.737+0000 7f872d38a640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87280c97d0 con 0x7f872815b2a0 2023-10-19T14:20:42.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.737+0000 7f87267fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f87280cf340 0x7f87280b4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:42.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.737+0000 7f87267fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f87280cf340 0x7f87280b4ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42352/0 (socket says 172.21.15.57:42352) 2023-10-19T14:20:42.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.737+0000 7f87267fc640 1 -- 172.21.15.57:0/769645982 learned_addr learned my addr 172.21.15.57:0/769645982 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:42.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.737+0000 7f87277fe640 1 --2- 172.21.15.57:0/769645982 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f872815b2a0 0x7f87280b5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:42.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.737+0000 7f8726ffd640 1 --2- 172.21.15.57:0/769645982 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f87280ceb40 0x7f87280c8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:42.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.737+0000 7f87267fc640 1 -- 172.21.15.57:0/769645982 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f872815b2a0 msgr2=0x7f87280b5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:42.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.737+0000 7f87267fc640 1 --2- 172.21.15.57:0/769645982 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f872815b2a0 0x7f87280b5400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:42.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.737+0000 7f87267fc640 1 -- 172.21.15.57:0/769645982 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f87280ceb40 msgr2=0x7f87280c8aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:42.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.737+0000 7f87267fc640 1 --2- 172.21.15.57:0/769645982 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f87280ceb40 0x7f87280c8aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:42.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.737+0000 7f87267fc640 1 -- 172.21.15.57:0/769645982 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8720009490 con 0x7f87280cf340 2023-10-19T14:20:42.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.737+0000 7f87267fc640 1 --2- 172.21.15.57:0/769645982 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f87280cf340 0x7f87280b4ec0 secure :-1 s=READY pgs=2460 cs=0 l=1 rev1=1 crypto rx=0x7f8710000c00 tx=0x7f8710003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:42.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.737+0000 7f871f7fe640 1 -- 172.21.15.57:0/769645982 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8710003e20 con 0x7f87280cf340 2023-10-19T14:20:42.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.737+0000 7f871f7fe640 1 -- 172.21.15.57:0/769645982 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f8710034440 con 0x7f87280cf340 2023-10-19T14:20:42.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.737+0000 7f872d38a640 1 -- 172.21.15.57:0/769645982 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8710009490 con 0x7f87280cf340 2023-10-19T14:20:42.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.737+0000 7f871f7fe640 1 -- 172.21.15.57:0/769645982 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8710034850 con 0x7f87280cf340 2023-10-19T14:20:42.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.738+0000 7f872d38a640 1 -- 172.21.15.57:0/769645982 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87281e3c60 con 0x7f87280cf340 2023-10-19T14:20:42.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.738+0000 7f872d38a640 1 -- 172.21.15.57:0/769645982 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87280690e0 con 0x7f87280cf340 2023-10-19T14:20:42.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.739+0000 7f871f7fe640 1 -- 172.21.15.57:0/769645982 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 132) v1 ==== 89451+0+0 (secure 0 0 0) 0x7f87100349b0 con 0x7f87280cf340 2023-10-19T14:20:42.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.739+0000 7f871f7fe640 1 --2- 172.21.15.57:0/769645982 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7f870806b0d0 0x7f870806d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:42.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.739+0000 7f871f7fe640 1 -- 172.21.15.57:0/769645982 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(228..228 src has 1..228) v4 ==== 7265+0+0 (secure 0 0 0) 0x7f87100ac8b0 con 0x7f87280cf340 2023-10-19T14:20:42.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.739+0000 7f8726ffd640 1 --2- 172.21.15.57:0/769645982 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7f870806b0d0 0x7f870806d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:42.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.740+0000 7f8726ffd640 1 --2- 172.21.15.57:0/769645982 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7f870806b0d0 0x7f870806d590 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f87280c7870 tx=0x7f8720005f90 comp rx=0 tx=0).ready entity=mgr.9422 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:42.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.743+0000 7f871f7fe640 1 -- 172.21.15.57:0/769645982 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8710076fd0 con 0x7f87280cf340 2023-10-19T14:20:42.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:42.942+0000 7f872d38a640 1 -- 172.21.15.57:0/769645982 --> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f87280cc170 con 0x7f870806b0d0 2023-10-19T14:20:43.006 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.003+0000 7f871f7fe640 1 -- 172.21.15.57:0/769645982 <== mgr.9422 v2:172.21.15.179:6832/4242302683 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+61 (secure 0 0 0) 0x7f87200098a0 con 0x7f870806b0d0 2023-10-19T14:20:43.006 INFO:teuthology.orchestra.run.smithi057.stdout:Schedule deactivated for path mandatory_fs_argument_test_dir 2023-10-19T14:20:43.008 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.005+0000 7f872d38a640 1 -- 172.21.15.57:0/769645982 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7f870806b0d0 msgr2=0x7f870806d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:43.008 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.005+0000 7f872d38a640 1 --2- 172.21.15.57:0/769645982 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7f870806b0d0 0x7f870806d590 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f87280c7870 tx=0x7f8720005f90 comp rx=0 tx=0).stop 2023-10-19T14:20:43.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.005+0000 7f872d38a640 1 -- 172.21.15.57:0/769645982 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f87280cf340 msgr2=0x7f87280b4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:43.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.005+0000 7f872d38a640 1 --2- 172.21.15.57:0/769645982 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f87280cf340 0x7f87280b4ec0 secure :-1 s=READY pgs=2460 cs=0 l=1 rev1=1 crypto rx=0x7f8710000c00 tx=0x7f8710003820 comp rx=0 tx=0).stop 2023-10-19T14:20:43.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.005+0000 7f872d38a640 1 -- 172.21.15.57:0/769645982 shutdown_connections 2023-10-19T14:20:43.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.005+0000 7f872d38a640 1 --2- 172.21.15.57:0/769645982 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7f870806b0d0 0x7f870806d590 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:43.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.005+0000 7f872d38a640 1 --2- 172.21.15.57:0/769645982 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f872815b2a0 0x7f87280b5400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:43.010 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.005+0000 7f872d38a640 1 --2- 172.21.15.57:0/769645982 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f87280cf340 0x7f87280b4ec0 unknown :-1 s=CLOSED pgs=2460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:43.010 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.005+0000 7f872d38a640 1 --2- 172.21.15.57:0/769645982 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f87280ceb40 0x7f87280c8aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:43.010 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.005+0000 7f872d38a640 1 -- 172.21.15.57:0/769645982 >> 172.21.15.57:0/769645982 conn(0x7f87280d0d60 msgr2=0x7f87280adf30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:43.010 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.005+0000 7f872d38a640 1 -- 172.21.15.57:0/769645982 shutdown_connections 2023-10-19T14:20:43.010 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.005+0000 7f872d38a640 1 -- 172.21.15.57:0/769645982 wait complete. 2023-10-19T14:20:43.019 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path mandatory_fs_argument_test_dir 2023-10-19T14:20:43.019 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove mandatory_fs_argument_test_dir --fs backup_fs 1M 2023-10-19T14:20:43.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.130+0000 7f936751d640 1 -- 172.21.15.57:0/424965458 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f93600cf340 msgr2=0x7f93600d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:43.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.130+0000 7f936751d640 1 --2- 172.21.15.57:0/424965458 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f93600cf340 0x7f93600d1910 secure :-1 s=READY pgs=2462 cs=0 l=1 rev1=1 crypto rx=0x7f936008a120 tx=0x7f935c02bcf0 comp rx=0 tx=0).stop 2023-10-19T14:20:43.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.130+0000 7f936751d640 1 -- 172.21.15.57:0/424965458 shutdown_connections 2023-10-19T14:20:43.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.130+0000 7f936751d640 1 --2- 172.21.15.57:0/424965458 >> v2:172.21.15.179:3300/0 conn(0x7f936015b2a0 0x7f93600cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:43.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.130+0000 7f936751d640 1 --2- 172.21.15.57:0/424965458 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f93600cf340 0x7f93600d1910 unknown :-1 s=CLOSED pgs=2462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:43.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.130+0000 7f936751d640 1 -- 172.21.15.57:0/424965458 >> 172.21.15.57:0/424965458 conn(0x7f93600d0d60 msgr2=0x7f93600d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:43.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.130+0000 7f936751d640 1 -- 172.21.15.57:0/424965458 shutdown_connections 2023-10-19T14:20:43.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.130+0000 7f936751d640 1 -- 172.21.15.57:0/424965458 wait complete. 2023-10-19T14:20:43.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.132+0000 7f936751d640 1 Processor -- start 2023-10-19T14:20:43.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.132+0000 7f936751d640 1 -- start start 2023-10-19T14:20:43.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.132+0000 7f936751d640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f93600ceb40 0x7f93601552b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:43.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.132+0000 7f936751d640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f93600cf340 0x7f93600b5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:43.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.132+0000 7f936751d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f936015b2a0 0x7f93600b5840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:43.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.132+0000 7f936751d640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93600af910 con 0x7f936015b2a0 2023-10-19T14:20:43.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.132+0000 7f936751d640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93601557f0 con 0x7f93600ceb40 2023-10-19T14:20:43.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.132+0000 7f936751d640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93600b4560 con 0x7f93600cf340 2023-10-19T14:20:43.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.132+0000 7f9365aba640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f936015b2a0 0x7f93600b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:43.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.132+0000 7f9365aba640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f936015b2a0 0x7f93600b5840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42370/0 (socket says 172.21.15.57:42370) 2023-10-19T14:20:43.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.132+0000 7f9365aba640 1 -- 172.21.15.57:0/2652948377 learned_addr learned my addr 172.21.15.57:0/2652948377 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:43.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.133+0000 7f93652b9640 1 --2- 172.21.15.57:0/2652948377 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f93600ceb40 0x7f93601552b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:43.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.133+0000 7f9364ab8640 1 --2- 172.21.15.57:0/2652948377 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f93600cf340 0x7f93600b5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:43.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.133+0000 7f9365aba640 1 -- 172.21.15.57:0/2652948377 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f93600cf340 msgr2=0x7f93600b5300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:43.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.133+0000 7f9365aba640 1 --2- 172.21.15.57:0/2652948377 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f93600cf340 0x7f93600b5300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:43.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.133+0000 7f9365aba640 1 -- 172.21.15.57:0/2652948377 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f93600ceb40 msgr2=0x7f93601552b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:43.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.133+0000 7f9365aba640 1 --2- 172.21.15.57:0/2652948377 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f93600ceb40 0x7f93601552b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:43.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.133+0000 7f9365aba640 1 -- 172.21.15.57:0/2652948377 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f935c009490 con 0x7f936015b2a0 2023-10-19T14:20:43.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.133+0000 7f9364ab8640 1 --2- 172.21.15.57:0/2652948377 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f93600cf340 0x7f93600b5300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:20:43.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.133+0000 7f9365aba640 1 --2- 172.21.15.57:0/2652948377 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f936015b2a0 0x7f93600b5840 secure :-1 s=READY pgs=2463 cs=0 l=1 rev1=1 crypto rx=0x7f935800cc10 tx=0x7f935800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:43.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.133+0000 7f9355ffb640 1 -- 172.21.15.57:0/2652948377 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f935800ad40 con 0x7f936015b2a0 2023-10-19T14:20:43.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.133+0000 7f9355ffb640 1 -- 172.21.15.57:0/2652948377 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f9358012420 con 0x7f936015b2a0 2023-10-19T14:20:43.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.133+0000 7f936751d640 1 -- 172.21.15.57:0/2652948377 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93600b4840 con 0x7f936015b2a0 2023-10-19T14:20:43.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.133+0000 7f9355ffb640 1 -- 172.21.15.57:0/2652948377 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9358012830 con 0x7f936015b2a0 2023-10-19T14:20:43.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.133+0000 7f936751d640 1 -- 172.21.15.57:0/2652948377 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93600b4e00 con 0x7f936015b2a0 2023-10-19T14:20:43.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.134+0000 7f936751d640 1 -- 172.21.15.57:0/2652948377 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93600b4ff0 con 0x7f936015b2a0 2023-10-19T14:20:43.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.135+0000 7f9355ffb640 1 -- 172.21.15.57:0/2652948377 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 132) v1 ==== 89451+0+0 (secure 0 0 0) 0x7f935801a3b0 con 0x7f936015b2a0 2023-10-19T14:20:43.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.135+0000 7f9355ffb640 1 --2- 172.21.15.57:0/2652948377 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7f934406b0d0 0x7f934406d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:43.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.135+0000 7f9355ffb640 1 -- 172.21.15.57:0/2652948377 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(228..228 src has 1..228) v4 ==== 7265+0+0 (secure 0 0 0) 0x7f935808a880 con 0x7f936015b2a0 2023-10-19T14:20:43.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.135+0000 7f93652b9640 1 --2- 172.21.15.57:0/2652948377 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7f934406b0d0 0x7f934406d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:43.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.136+0000 7f93652b9640 1 --2- 172.21.15.57:0/2652948377 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7f934406b0d0 0x7f934406d590 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f935c036000 tx=0x7f935c004730 comp rx=0 tx=0).ready entity=mgr.9422 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:43.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.139+0000 7f9355ffb640 1 -- 172.21.15.57:0/2652948377 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9358054fb0 con 0x7f936015b2a0 2023-10-19T14:20:43.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.337+0000 7f936751d640 1 -- 172.21.15.57:0/2652948377 --> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f93600690e0 con 0x7f934406b0d0 2023-10-19T14:20:43.399 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.397+0000 7f9355ffb640 1 -- 172.21.15.57:0/2652948377 <== mgr.9422 v2:172.21.15.179:6832/4242302683 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+57 (secure 0 0 0) 0x7f93600690e0 con 0x7f934406b0d0 2023-10-19T14:20:43.400 INFO:teuthology.orchestra.run.smithi057.stdout:Schedule removed for path mandatory_fs_argument_test_dir 2023-10-19T14:20:43.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.399+0000 7f936751d640 1 -- 172.21.15.57:0/2652948377 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7f934406b0d0 msgr2=0x7f934406d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:43.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.399+0000 7f936751d640 1 --2- 172.21.15.57:0/2652948377 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7f934406b0d0 0x7f934406d590 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f935c036000 tx=0x7f935c004730 comp rx=0 tx=0).stop 2023-10-19T14:20:43.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.399+0000 7f936751d640 1 -- 172.21.15.57:0/2652948377 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f936015b2a0 msgr2=0x7f93600b5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:43.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.399+0000 7f936751d640 1 --2- 172.21.15.57:0/2652948377 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f936015b2a0 0x7f93600b5840 secure :-1 s=READY pgs=2463 cs=0 l=1 rev1=1 crypto rx=0x7f935800cc10 tx=0x7f935800a430 comp rx=0 tx=0).stop 2023-10-19T14:20:43.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.399+0000 7f936751d640 1 -- 172.21.15.57:0/2652948377 shutdown_connections 2023-10-19T14:20:43.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.399+0000 7f936751d640 1 --2- 172.21.15.57:0/2652948377 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7f934406b0d0 0x7f934406d590 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:43.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.399+0000 7f936751d640 1 --2- 172.21.15.57:0/2652948377 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f936015b2a0 0x7f93600b5840 unknown :-1 s=CLOSED pgs=2463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:43.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.399+0000 7f936751d640 1 --2- 172.21.15.57:0/2652948377 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f93600cf340 0x7f93600b5300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:43.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.399+0000 7f936751d640 1 --2- 172.21.15.57:0/2652948377 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f93600ceb40 0x7f93601552b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:43.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.399+0000 7f936751d640 1 -- 172.21.15.57:0/2652948377 >> 172.21.15.57:0/2652948377 conn(0x7f93600d0d60 msgr2=0x7f9360069f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:43.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.399+0000 7f936751d640 1 -- 172.21.15.57:0/2652948377 shutdown_connections 2023-10-19T14:20:43.404 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.399+0000 7f936751d640 1 -- 172.21.15.57:0/2652948377 wait complete. 2023-10-19T14:20:43.412 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path mandatory_fs_argument_test_dir 2023-10-19T14:20:43.413 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:20:43.413 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir mandatory_fs_argument_test_dir) 2023-10-19T14:20:43.432 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-19T14:20:43.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.582+0000 7fc67e1c9640 1 -- 172.21.15.57:0/1820998434 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc6600039a0 msgr2=0x7fc660003d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:43.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.582+0000 7fc67e1c9640 1 --2- 172.21.15.57:0/1820998434 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc6600039a0 0x7fc660003d80 secure :-1 s=READY pgs=2465 cs=0 l=1 rev1=1 crypto rx=0x7fc668001790 tx=0x7fc6600057f0 comp rx=0 tx=0).stop 2023-10-19T14:20:43.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.582+0000 7fc677fff640 1 -- 172.21.15.57:0/1820998434 reap_dead start 2023-10-19T14:20:43.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.582+0000 7fc67e1c9640 1 -- 172.21.15.57:0/1820998434 shutdown_connections 2023-10-19T14:20:43.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.582+0000 7fc67e1c9640 1 -- 172.21.15.57:0/1820998434 >> 172.21.15.57:0/1820998434 conn(0x7fc6780d0d60 msgr2=0x7fc6780d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:43.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.582+0000 7fc67e1c9640 1 -- 172.21.15.57:0/1820998434 shutdown_connections 2023-10-19T14:20:43.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.582+0000 7fc67e1c9640 1 -- 172.21.15.57:0/1820998434 wait complete. 2023-10-19T14:20:43.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.583+0000 7fc67e1c9640 1 Processor -- start 2023-10-19T14:20:43.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.583+0000 7fc67e1c9640 1 -- start start 2023-10-19T14:20:43.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.584+0000 7fc67e1c9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc6600039a0 0x7fc67815cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:43.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.584+0000 7fc67e1c9640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc6780b1330 0x7fc6780b4c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:43.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.584+0000 7fc67e1c9640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc6780b17b0 0x7fc6780b5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:43.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.584+0000 7fc67e1c9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6780b3cc0 con 0x7fc6600039a0 2023-10-19T14:20:43.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.584+0000 7fc67e1c9640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6780b3fa0 con 0x7fc6780b17b0 2023-10-19T14:20:43.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.584+0000 7fc67e1c9640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6780b4280 con 0x7fc6780b1330 2023-10-19T14:20:43.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.584+0000 7fc6777fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc6600039a0 0x7fc67815cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:43.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.584+0000 7fc6777fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc6600039a0 0x7fc67815cdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42384/0 (socket says 172.21.15.57:42384) 2023-10-19T14:20:43.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.584+0000 7fc6777fe640 1 -- 172.21.15.57:0/4140370613 learned_addr learned my addr 172.21.15.57:0/4140370613 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:43.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.584+0000 7fc676ffd640 1 --2- 172.21.15.57:0/4140370613 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc6780b1330 0x7fc6780b4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:43.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.584+0000 7fc677fff640 1 --2- 172.21.15.57:0/4140370613 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc6780b17b0 0x7fc6780b5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:43.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.584+0000 7fc6777fe640 1 -- 172.21.15.57:0/4140370613 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc6780b1330 msgr2=0x7fc6780b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:43.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.584+0000 7fc6777fe640 1 --2- 172.21.15.57:0/4140370613 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc6780b1330 0x7fc6780b4c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:43.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.584+0000 7fc6777fe640 1 -- 172.21.15.57:0/4140370613 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc6780b17b0 msgr2=0x7fc6780b5140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:43.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.584+0000 7fc6777fe640 1 --2- 172.21.15.57:0/4140370613 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc6780b17b0 0x7fc6780b5140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:43.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.584+0000 7fc6777fe640 1 -- 172.21.15.57:0/4140370613 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6780b4560 con 0x7fc6600039a0 2023-10-19T14:20:43.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.585+0000 7fc677fff640 1 --2- 172.21.15.57:0/4140370613 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc6780b17b0 0x7fc6780b5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:20:43.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.585+0000 7fc6777fe640 1 --2- 172.21.15.57:0/4140370613 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc6600039a0 0x7fc67815cdb0 secure :-1 s=READY pgs=2466 cs=0 l=1 rev1=1 crypto rx=0x7fc66802c1e0 tx=0x7fc66802cda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:43.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.585+0000 7fc65ffff640 1 -- 172.21.15.57:0/4140370613 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6680037f0 con 0x7fc6600039a0 2023-10-19T14:20:43.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.585+0000 7fc65ffff640 1 -- 172.21.15.57:0/4140370613 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fc668003a40 con 0x7fc6600039a0 2023-10-19T14:20:43.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.585+0000 7fc67e1c9640 1 -- 172.21.15.57:0/4140370613 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc67815b770 con 0x7fc6600039a0 2023-10-19T14:20:43.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.585+0000 7fc65ffff640 1 -- 172.21.15.57:0/4140370613 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc668003e10 con 0x7fc6600039a0 2023-10-19T14:20:43.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.585+0000 7fc67e1c9640 1 -- 172.21.15.57:0/4140370613 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc67815d2f0 con 0x7fc6600039a0 2023-10-19T14:20:43.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.586+0000 7fc67e1c9640 1 -- 172.21.15.57:0/4140370613 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc67806bc10 con 0x7fc6600039a0 2023-10-19T14:20:43.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.586+0000 7fc65ffff640 1 -- 172.21.15.57:0/4140370613 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 132) v1 ==== 89451+0+0 (secure 0 0 0) 0x7fc668032070 con 0x7fc6600039a0 2023-10-19T14:20:43.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.587+0000 7fc65ffff640 1 --2- 172.21.15.57:0/4140370613 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7fc66406b100 0x7fc66406d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:43.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.587+0000 7fc65ffff640 1 -- 172.21.15.57:0/4140370613 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(228..228 src has 1..228) v4 ==== 7265+0+0 (secure 0 0 0) 0x7fc66803c070 con 0x7fc6600039a0 2023-10-19T14:20:43.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.587+0000 7fc676ffd640 1 --2- 172.21.15.57:0/4140370613 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7fc66406b100 0x7fc66406d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:43.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.588+0000 7fc676ffd640 1 --2- 172.21.15.57:0/4140370613 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7fc66406b100 0x7fc66406d5c0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc6780b2790 tx=0x7fc66c000ed0 comp rx=0 tx=0).ready entity=mgr.9422 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:43.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.591+0000 7fc65ffff640 1 -- 172.21.15.57:0/4140370613 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc668076310 con 0x7fc6600039a0 2023-10-19T14:20:43.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:43.815+0000 7fc67e1c9640 1 -- 172.21.15.57:0/4140370613 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7fc6780690e0 con 0x7fc6600039a0 2023-10-19T14:20:44.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:44.744+0000 7fc65ffff640 1 -- 172.21.15.57:0/4140370613 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v133) v1 ==== 93+0+0 (secure 0 0 0) 0x7fc6680460a0 con 0x7fc6600039a0 2023-10-19T14:20:44.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:44.748+0000 7fc67e1c9640 1 -- 172.21.15.57:0/4140370613 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7fc66406b100 msgr2=0x7fc66406d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:44.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:44.748+0000 7fc67e1c9640 1 --2- 172.21.15.57:0/4140370613 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7fc66406b100 0x7fc66406d5c0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc6780b2790 tx=0x7fc66c000ed0 comp rx=0 tx=0).stop 2023-10-19T14:20:44.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:44.748+0000 7fc67e1c9640 1 -- 172.21.15.57:0/4140370613 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc6600039a0 msgr2=0x7fc67815cdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:44.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:44.748+0000 7fc67e1c9640 1 --2- 172.21.15.57:0/4140370613 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc6600039a0 0x7fc67815cdb0 secure :-1 s=READY pgs=2466 cs=0 l=1 rev1=1 crypto rx=0x7fc66802c1e0 tx=0x7fc66802cda0 comp rx=0 tx=0).stop 2023-10-19T14:20:44.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:44.748+0000 7fc67e1c9640 1 -- 172.21.15.57:0/4140370613 shutdown_connections 2023-10-19T14:20:44.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:44.748+0000 7fc67e1c9640 1 --2- 172.21.15.57:0/4140370613 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7fc66406b100 0x7fc66406d5c0 secure :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc6780b2790 tx=0x7fc66c000ed0 comp rx=0 tx=0).stop 2023-10-19T14:20:44.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:44.748+0000 7fc67e1c9640 1 --2- 172.21.15.57:0/4140370613 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc6780b17b0 0x7fc6780b5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:44.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:44.748+0000 7fc67e1c9640 1 --2- 172.21.15.57:0/4140370613 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc6780b1330 0x7fc6780b4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:44.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:44.748+0000 7fc67e1c9640 1 --2- 172.21.15.57:0/4140370613 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc6600039a0 0x7fc67815cdb0 secure :-1 s=CLOSED pgs=2466 cs=0 l=1 rev1=1 crypto rx=0x7fc66802c1e0 tx=0x7fc66802cda0 comp rx=0 tx=0).stop 2023-10-19T14:20:44.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:44.748+0000 7fc67e1c9640 1 -- 172.21.15.57:0/4140370613 >> 172.21.15.57:0/4140370613 conn(0x7fc6780d0d60 msgr2=0x7fc67806b500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:44.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:44.748+0000 7fc67e1c9640 1 -- 172.21.15.57:0/4140370613 shutdown_connections 2023-10-19T14:20:44.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:44.748+0000 7fc67e1c9640 1 -- 172.21.15.57:0/4140370613 wait complete. 2023-10-19T14:20:44.767 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-19T14:20:44.767 DEBUG:teuthology.orchestra.run.smithi057:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-19T14:20:44.818 DEBUG:teuthology.orchestra.run.smithi179:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-19T14:20:44.864 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-19T14:20:44.864 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T14:20:44.865 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T14:20:44.881 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T14:20:45.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.028+0000 7fbc8c6b4640 1 -- 172.21.15.57:0/118598003 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc840cf340 msgr2=0x7fbc840d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:45.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.028+0000 7fbc8c6b4640 1 --2- 172.21.15.57:0/118598003 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc840cf340 0x7fbc840d1910 secure :-1 s=READY pgs=2470 cs=0 l=1 rev1=1 crypto rx=0x7fbc8408a120 tx=0x7fbc7402bcf0 comp rx=0 tx=0).stop 2023-10-19T14:20:45.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.028+0000 7fbc8c6b4640 1 -- 172.21.15.57:0/118598003 shutdown_connections 2023-10-19T14:20:45.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.028+0000 7fbc8c6b4640 1 --2- 172.21.15.57:0/118598003 >> v2:172.21.15.179:3300/0 conn(0x7fbc8415b2a0 0x7fbc840cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:45.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.028+0000 7fbc8c6b4640 1 --2- 172.21.15.57:0/118598003 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc840cf340 0x7fbc840d1910 unknown :-1 s=CLOSED pgs=2470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:45.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.028+0000 7fbc8c6b4640 1 -- 172.21.15.57:0/118598003 >> 172.21.15.57:0/118598003 conn(0x7fbc840d0d60 msgr2=0x7fbc840d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:45.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.028+0000 7fbc8c6b4640 1 -- 172.21.15.57:0/118598003 shutdown_connections 2023-10-19T14:20:45.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.028+0000 7fbc8c6b4640 1 -- 172.21.15.57:0/118598003 wait complete. 2023-10-19T14:20:45.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.029+0000 7fbc8c6b4640 1 Processor -- start 2023-10-19T14:20:45.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.029+0000 7fbc8c6b4640 1 -- start start 2023-10-19T14:20:45.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.030+0000 7fbc8c6b4640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbc840ceb40 0x7fbc84150280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:45.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.030+0000 7fbc8c6b4640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbc840cf340 0x7fbc840b6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:45.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.030+0000 7fbc8c6b4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc8415b2a0 0x7fbc840c9a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:45.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.030+0000 7fbc8c6b4640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc841507c0 con 0x7fbc8415b2a0 2023-10-19T14:20:45.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.030+0000 7fbc8c6b4640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc8415d680 con 0x7fbc840cf340 2023-10-19T14:20:45.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.030+0000 7fbc8c6b4640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc8415d960 con 0x7fbc840ceb40 2023-10-19T14:20:45.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.030+0000 7fbc8ac51640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc8415b2a0 0x7fbc840c9a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:45.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.030+0000 7fbc8ac51640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc8415b2a0 0x7fbc840c9a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42418/0 (socket says 172.21.15.57:42418) 2023-10-19T14:20:45.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.030+0000 7fbc8ac51640 1 -- 172.21.15.57:0/481695055 learned_addr learned my addr 172.21.15.57:0/481695055 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:45.038 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.030+0000 7fbc89c4f640 1 --2- 172.21.15.57:0/481695055 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbc840cf340 0x7fbc840b6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:45.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.030+0000 7fbc8a450640 1 --2- 172.21.15.57:0/481695055 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbc840ceb40 0x7fbc84150280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:45.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.031+0000 7fbc8ac51640 1 -- 172.21.15.57:0/481695055 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbc840ceb40 msgr2=0x7fbc84150280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:45.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.031+0000 7fbc8ac51640 1 --2- 172.21.15.57:0/481695055 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbc840ceb40 0x7fbc84150280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:45.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.031+0000 7fbc8ac51640 1 -- 172.21.15.57:0/481695055 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbc840cf340 msgr2=0x7fbc840b6190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:45.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.031+0000 7fbc8ac51640 1 --2- 172.21.15.57:0/481695055 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbc840cf340 0x7fbc840b6190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:45.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.031+0000 7fbc8ac51640 1 -- 172.21.15.57:0/481695055 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc74009490 con 0x7fbc8415b2a0 2023-10-19T14:20:45.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.031+0000 7fbc89c4f640 1 --2- 172.21.15.57:0/481695055 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbc840cf340 0x7fbc840b6190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T14:20:45.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.031+0000 7fbc8ac51640 1 --2- 172.21.15.57:0/481695055 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc8415b2a0 0x7fbc840c9a00 secure :-1 s=READY pgs=2471 cs=0 l=1 rev1=1 crypto rx=0x7fbc7c00cbc0 tx=0x7fbc7c00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:45.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.031+0000 7fbc7affd640 1 -- 172.21.15.57:0/481695055 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc7c00aca0 con 0x7fbc8415b2a0 2023-10-19T14:20:45.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.031+0000 7fbc7affd640 1 -- 172.21.15.57:0/481695055 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fbc7c012420 con 0x7fbc8415b2a0 2023-10-19T14:20:45.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.031+0000 7fbc8c6b4640 1 -- 172.21.15.57:0/481695055 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc8415dc40 con 0x7fbc8415b2a0 2023-10-19T14:20:45.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.031+0000 7fbc7affd640 1 -- 172.21.15.57:0/481695055 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc7c012830 con 0x7fbc8415b2a0 2023-10-19T14:20:45.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.031+0000 7fbc8c6b4640 1 -- 172.21.15.57:0/481695055 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc840b66d0 con 0x7fbc8415b2a0 2023-10-19T14:20:45.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.032+0000 7fbc8c6b4640 1 -- 172.21.15.57:0/481695055 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc8415ae00 con 0x7fbc8415b2a0 2023-10-19T14:20:45.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.033+0000 7fbc7affd640 1 -- 172.21.15.57:0/481695055 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 133) v1 ==== 89490+0+0 (secure 0 0 0) 0x7fbc7c01b400 con 0x7fbc8415b2a0 2023-10-19T14:20:45.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.033+0000 7fbc7affd640 1 --2- 172.21.15.57:0/481695055 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7fbc6806b110 0x7fbc6806d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:45.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.033+0000 7fbc7affd640 1 -- 172.21.15.57:0/481695055 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(228..228 src has 1..228) v4 ==== 7265+0+0 (secure 0 0 0) 0x7fbc7c08a720 con 0x7fbc8415b2a0 2023-10-19T14:20:45.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.033+0000 7fbc8a450640 1 -- 172.21.15.57:0/481695055 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7fbc6806b110 msgr2=0x7fbc6806d5d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/4242302683 2023-10-19T14:20:45.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.033+0000 7fbc8a450640 1 --2- 172.21.15.57:0/481695055 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7fbc6806b110 0x7fbc6806d5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T14:20:45.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.037+0000 7fbc7affd640 1 -- 172.21.15.57:0/481695055 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fbc7c054f10 con 0x7fbc8415b2a0 2023-10-19T14:20:45.139 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:45.137+0000 7f4d48c11040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-19T14:20:45.156 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:20:45.153+0000 7f10e7e3e040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-19T14:20:45.184 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:45.182+0000 7f4d48c11040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-19T14:20:45.203 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:20:45.201+0000 7f10e7e3e040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-19T14:20:45.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.215+0000 7fbc8c6b4640 1 -- 172.21.15.57:0/481695055 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbc840cb980 con 0x7fbc8415b2a0 2023-10-19T14:20:45.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.217+0000 7fbc7affd640 1 -- 172.21.15.57:0/481695055 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v228) v1 ==== 74+0+21003 (secure 0 0 0) 0x7fbc7c058530 con 0x7fbc8415b2a0 2023-10-19T14:20:45.220 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:20:45.220 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":228,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T14:20:39.702674+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2023-10-19T14:20:17.357433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"222","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2023-10-19T14:20:18.049516+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"223","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_metadata","create_time":"2023-10-19T14:20:28.667664+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"226","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"backup_fs_data","create_time":"2023-10-19T14:20:29.122584+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"227","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.179:0/1194984394":"2023-10-20T14:20:39.702634+0000","172.21.15.179:6833/3232482678":"2023-10-20T14:20:39.702634+0000","172.21.15.179:6832/3232482678":"2023-10-20T14:20:39.702634+0000","172.21.15.179:0/2872579179":"2023-10-20T14:20:39.702634+0000","172.21.15.179:0/3910604357":"2023-10-20T14:20:39.702634+0000","172.21.15.179:0/536160557":"2023-10-20T14:20:39.702634+0000","172.21.15.179:0/3239062050":"2023-10-20T14:20:39.702634+0000","172.21.15.179:0/1948632334":"2023-10-20T14:20:39.702634+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T14:20:45.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.219+0000 7fbc8c6b4640 1 -- 172.21.15.57:0/481695055 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7fbc6806b110 msgr2=0x7fbc6806d5d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T14:20:45.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.219+0000 7fbc8c6b4640 1 --2- 172.21.15.57:0/481695055 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7fbc6806b110 0x7fbc6806d5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:45.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.219+0000 7fbc8c6b4640 1 -- 172.21.15.57:0/481695055 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc8415b2a0 msgr2=0x7fbc840c9a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:45.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.219+0000 7fbc8c6b4640 1 --2- 172.21.15.57:0/481695055 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc8415b2a0 0x7fbc840c9a00 secure :-1 s=READY pgs=2471 cs=0 l=1 rev1=1 crypto rx=0x7fbc7c00cbc0 tx=0x7fbc7c00a060 comp rx=0 tx=0).stop 2023-10-19T14:20:45.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.219+0000 7fbc8c6b4640 1 -- 172.21.15.57:0/481695055 shutdown_connections 2023-10-19T14:20:45.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.219+0000 7fbc8c6b4640 1 --2- 172.21.15.57:0/481695055 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7fbc6806b110 0x7fbc6806d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:45.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.219+0000 7fbc8c6b4640 1 --2- 172.21.15.57:0/481695055 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc8415b2a0 0x7fbc840c9a00 unknown :-1 s=CLOSED pgs=2471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:45.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.219+0000 7fbc8c6b4640 1 --2- 172.21.15.57:0/481695055 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbc840cf340 0x7fbc840b6190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:45.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.219+0000 7fbc8c6b4640 1 --2- 172.21.15.57:0/481695055 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbc840ceb40 0x7fbc84150280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:45.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.219+0000 7fbc8c6b4640 1 -- 172.21.15.57:0/481695055 >> 172.21.15.57:0/481695055 conn(0x7fbc840d0d60 msgr2=0x7fbc840cc880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:45.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.219+0000 7fbc8c6b4640 1 -- 172.21.15.57:0/481695055 shutdown_connections 2023-10-19T14:20:45.224 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.219+0000 7fbc8c6b4640 1 -- 172.21.15.57:0/481695055 wait complete. 2023-10-19T14:20:45.235 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-19T12:44:07.371186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 27, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-19T14:20:17.357433+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '222', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 28, 'pool_name': 'cephfs_data', 'create_time': '2023-10-19T14:20:18.049516+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '223', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 29, 'pool_name': 'backup_fs_metadata', 'create_time': '2023-10-19T14:20:28.667664+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '226', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 30, 'pool_name': 'backup_fs_data', 'create_time': '2023-10-19T14:20:29.122584+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '227', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-19T14:20:45.236 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-19T14:20:45.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.347+0000 7facac3b0640 1 -- 172.21.15.57:0/2182997205 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faca415b2a0 msgr2=0x7faca40d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:45.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.347+0000 7facac3b0640 1 --2- 172.21.15.57:0/2182997205 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faca415b2a0 0x7faca40d1910 secure :-1 s=READY pgs=2473 cs=0 l=1 rev1=1 crypto rx=0x7faca415b680 tx=0x7fac9402bcf0 comp rx=0 tx=0).stop 2023-10-19T14:20:45.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.347+0000 7facac3b0640 1 -- 172.21.15.57:0/2182997205 shutdown_connections 2023-10-19T14:20:45.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.347+0000 7facac3b0640 1 --2- 172.21.15.57:0/2182997205 >> v2:172.21.15.179:3300/0 conn(0x7faca40ceb40 0x7faca40b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:45.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.347+0000 7facac3b0640 1 --2- 172.21.15.57:0/2182997205 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faca415b2a0 0x7faca40d1910 unknown :-1 s=CLOSED pgs=2473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:45.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.347+0000 7facac3b0640 1 -- 172.21.15.57:0/2182997205 >> 172.21.15.57:0/2182997205 conn(0x7faca40d0d60 msgr2=0x7faca40d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:45.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.347+0000 7facac3b0640 1 -- 172.21.15.57:0/2182997205 shutdown_connections 2023-10-19T14:20:45.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.347+0000 7facac3b0640 1 -- 172.21.15.57:0/2182997205 wait complete. 2023-10-19T14:20:45.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.348+0000 7facac3b0640 1 Processor -- start 2023-10-19T14:20:45.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.348+0000 7facac3b0640 1 -- start start 2023-10-19T14:20:45.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.349+0000 7facac3b0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faca40ceb40 0x7faca4157e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:45.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.349+0000 7facac3b0640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7faca40cf340 0x7faca41f1610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:45.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.349+0000 7facac3b0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faca415b2a0 0x7faca40b5e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:45.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.349+0000 7facac3b0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faca415b900 con 0x7faca415b2a0 2023-10-19T14:20:45.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.349+0000 7facac3b0640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faca4158360 con 0x7faca40ceb40 2023-10-19T14:20:45.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.349+0000 7facac3b0640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faca41f1f80 con 0x7faca40cf340 2023-10-19T14:20:45.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.349+0000 7facaa94d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faca415b2a0 0x7faca40b5e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:45.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.349+0000 7facaa94d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faca415b2a0 0x7faca40b5e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42432/0 (socket says 172.21.15.57:42432) 2023-10-19T14:20:45.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.349+0000 7facaa94d640 1 -- 172.21.15.57:0/4082903438 learned_addr learned my addr 172.21.15.57:0/4082903438 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:45.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.349+0000 7facaa14c640 1 --2- 172.21.15.57:0/4082903438 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faca40ceb40 0x7faca4157e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:45.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.349+0000 7faca994b640 1 --2- 172.21.15.57:0/4082903438 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7faca40cf340 0x7faca41f1610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:45.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.349+0000 7facaa94d640 1 -- 172.21.15.57:0/4082903438 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7faca40cf340 msgr2=0x7faca41f1610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:45.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.349+0000 7facaa94d640 1 --2- 172.21.15.57:0/4082903438 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7faca40cf340 0x7faca41f1610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:45.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.349+0000 7facaa94d640 1 -- 172.21.15.57:0/4082903438 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faca40ceb40 msgr2=0x7faca4157e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:45.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.349+0000 7facaa94d640 1 --2- 172.21.15.57:0/4082903438 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faca40ceb40 0x7faca4157e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:45.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.349+0000 7facaa94d640 1 -- 172.21.15.57:0/4082903438 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac94009490 con 0x7faca415b2a0 2023-10-19T14:20:45.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.349+0000 7facaa94d640 1 --2- 172.21.15.57:0/4082903438 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faca415b2a0 0x7faca40b5e30 secure :-1 s=READY pgs=2474 cs=0 l=1 rev1=1 crypto rx=0x7fac9c00eb10 tx=0x7fac9c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:45.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.350+0000 7fac9affd640 1 -- 172.21.15.57:0/4082903438 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac9c004b20 con 0x7faca415b2a0 2023-10-19T14:20:45.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.350+0000 7fac9affd640 1 -- 172.21.15.57:0/4082903438 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fac9c004ca0 con 0x7faca415b2a0 2023-10-19T14:20:45.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.350+0000 7facac3b0640 1 -- 172.21.15.57:0/4082903438 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faca41f2260 con 0x7faca415b2a0 2023-10-19T14:20:45.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.350+0000 7fac9affd640 1 -- 172.21.15.57:0/4082903438 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac9c005280 con 0x7faca415b2a0 2023-10-19T14:20:45.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.350+0000 7facac3b0640 1 -- 172.21.15.57:0/4082903438 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faca41f1b50 con 0x7faca415b2a0 2023-10-19T14:20:45.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.351+0000 7facac3b0640 1 -- 172.21.15.57:0/4082903438 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faca41f2780 con 0x7faca415b2a0 2023-10-19T14:20:45.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.351+0000 7fac9affd640 1 -- 172.21.15.57:0/4082903438 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 133) v1 ==== 89490+0+0 (secure 0 0 0) 0x7fac9c0053e0 con 0x7faca415b2a0 2023-10-19T14:20:45.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.352+0000 7fac9affd640 1 --2- 172.21.15.57:0/4082903438 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7fac8806b110 0x7fac8806d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:45.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.352+0000 7facaa14c640 1 -- 172.21.15.57:0/4082903438 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7fac8806b110 msgr2=0x7fac8806d5d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/4242302683 2023-10-19T14:20:45.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.352+0000 7fac9affd640 1 -- 172.21.15.57:0/4082903438 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(228..228 src has 1..228) v4 ==== 7265+0+0 (secure 0 0 0) 0x7fac9c014070 con 0x7faca415b2a0 2023-10-19T14:20:45.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.352+0000 7facaa14c640 1 --2- 172.21.15.57:0/4082903438 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7fac8806b110 0x7fac8806d5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T14:20:45.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.356+0000 7fac9affd640 1 -- 172.21.15.57:0/4082903438 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7faca41f2780 con 0x7faca415b2a0 2023-10-19T14:20:45.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.533+0000 7facac3b0640 1 -- 172.21.15.57:0/4082903438 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7faca415b6d0 con 0x7faca415b2a0 2023-10-19T14:20:45.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.533+0000 7fac9affd640 1 -- 172.21.15.57:0/4082903438 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v228) v1 ==== 93+0+10 (secure 0 0 0) 0x7faca415b6d0 con 0x7faca415b2a0 2023-10-19T14:20:45.536 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-19T14:20:45.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.535+0000 7facac3b0640 1 -- 172.21.15.57:0/4082903438 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7fac8806b110 msgr2=0x7fac8806d5d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T14:20:45.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.535+0000 7facac3b0640 1 --2- 172.21.15.57:0/4082903438 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7fac8806b110 0x7fac8806d5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:45.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.535+0000 7facac3b0640 1 -- 172.21.15.57:0/4082903438 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faca415b2a0 msgr2=0x7faca40b5e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:45.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.535+0000 7facac3b0640 1 --2- 172.21.15.57:0/4082903438 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faca415b2a0 0x7faca40b5e30 secure :-1 s=READY pgs=2474 cs=0 l=1 rev1=1 crypto rx=0x7fac9c00eb10 tx=0x7fac9c00c2d0 comp rx=0 tx=0).stop 2023-10-19T14:20:45.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.535+0000 7facac3b0640 1 -- 172.21.15.57:0/4082903438 shutdown_connections 2023-10-19T14:20:45.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.535+0000 7facac3b0640 1 --2- 172.21.15.57:0/4082903438 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7fac8806b110 0x7fac8806d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:45.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.535+0000 7facac3b0640 1 --2- 172.21.15.57:0/4082903438 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faca415b2a0 0x7faca40b5e30 unknown :-1 s=CLOSED pgs=2474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:45.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.535+0000 7facac3b0640 1 --2- 172.21.15.57:0/4082903438 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7faca40cf340 0x7faca41f1610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:45.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.535+0000 7facac3b0640 1 --2- 172.21.15.57:0/4082903438 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faca40ceb40 0x7faca4157e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:45.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.535+0000 7facac3b0640 1 -- 172.21.15.57:0/4082903438 >> 172.21.15.57:0/4082903438 conn(0x7faca40d0d60 msgr2=0x7faca40ae7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:45.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.535+0000 7facac3b0640 1 -- 172.21.15.57:0/4082903438 shutdown_connections 2023-10-19T14:20:45.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.535+0000 7facac3b0640 1 -- 172.21.15.57:0/4082903438 wait complete. 2023-10-19T14:20:45.549 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-19T14:20:45.573 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:45.570+0000 7f4d48c11040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-19T14:20:45.617 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:45.614+0000 7f4d48c11040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-19T14:20:45.626 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:20:45.622+0000 7f10e7e3e040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-19T14:20:45.659 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:45.656+0000 7f4d48c11040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-19T14:20:45.664 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.661+0000 7f32ef687640 1 -- 172.21.15.57:0/2785634029 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f32e80cf340 msgr2=0x7f32e80d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:45.664 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.661+0000 7f32ef687640 1 --2- 172.21.15.57:0/2785634029 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f32e80cf340 0x7f32e80d1910 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f32e4009830 tx=0x7f32e402bac0 comp rx=0 tx=0).stop 2023-10-19T14:20:45.664 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.661+0000 7f32ef687640 1 -- 172.21.15.57:0/2785634029 shutdown_connections 2023-10-19T14:20:45.665 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.661+0000 7f32ef687640 1 --2- 172.21.15.57:0/2785634029 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f32e80ceb40 0x7f32e80b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:45.665 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.661+0000 7f32ef687640 1 --2- 172.21.15.57:0/2785634029 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f32e80cf340 0x7f32e80d1910 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:45.665 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.661+0000 7f32ef687640 1 -- 172.21.15.57:0/2785634029 >> 172.21.15.57:0/2785634029 conn(0x7f32e80d0d60 msgr2=0x7f32e80d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:45.665 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.661+0000 7f32ef687640 1 -- 172.21.15.57:0/2785634029 shutdown_connections 2023-10-19T14:20:45.665 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.661+0000 7f32ef687640 1 -- 172.21.15.57:0/2785634029 wait complete. 2023-10-19T14:20:45.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.662+0000 7f32ef687640 1 Processor -- start 2023-10-19T14:20:45.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.662+0000 7f32ef687640 1 -- start start 2023-10-19T14:20:45.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.663+0000 7f32ef687640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f32e80ceb40 0x7f32e81f12a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:45.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.663+0000 7f32ef687640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f32e80cf340 0x7f32e8159320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:45.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.663+0000 7f32ef687640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f32e815b2a0 0x7f32e8159860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:45.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.663+0000 7f32ef687640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32e81f2340 con 0x7f32e80cf340 2023-10-19T14:20:45.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.663+0000 7f32ef687640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32e81f1d80 con 0x7f32e815b2a0 2023-10-19T14:20:45.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.663+0000 7f32ef687640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32e81f2060 con 0x7f32e80ceb40 2023-10-19T14:20:45.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.663+0000 7f32ecc22640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f32e80cf340 0x7f32e8159320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:45.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.663+0000 7f32ecc22640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f32e80cf340 0x7f32e8159320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42440/0 (socket says 172.21.15.57:42440) 2023-10-19T14:20:45.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.663+0000 7f32ecc22640 1 -- 172.21.15.57:0/172124947 learned_addr learned my addr 172.21.15.57:0/172124947 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:45.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.663+0000 7f32ed423640 1 --2- 172.21.15.57:0/172124947 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f32e80ceb40 0x7f32e81f12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:45.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.663+0000 7f32edc24640 1 --2- 172.21.15.57:0/172124947 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f32e815b2a0 0x7f32e8159860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:45.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.663+0000 7f32ecc22640 1 -- 172.21.15.57:0/172124947 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f32e80ceb40 msgr2=0x7f32e81f12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:45.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.663+0000 7f32ecc22640 1 --2- 172.21.15.57:0/172124947 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f32e80ceb40 0x7f32e81f12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:45.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.663+0000 7f32ecc22640 1 -- 172.21.15.57:0/172124947 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f32e815b2a0 msgr2=0x7f32e8159860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:45.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.663+0000 7f32ecc22640 1 --2- 172.21.15.57:0/172124947 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f32e815b2a0 0x7f32e8159860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:45.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.663+0000 7f32ecc22640 1 -- 172.21.15.57:0/172124947 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32e0009840 con 0x7f32e80cf340 2023-10-19T14:20:45.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.664+0000 7f32ecc22640 1 --2- 172.21.15.57:0/172124947 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f32e80cf340 0x7f32e8159320 secure :-1 s=READY pgs=2476 cs=0 l=1 rev1=1 crypto rx=0x7f32d800c710 tx=0x7f32d800cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:45.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.664+0000 7f32ddffb640 1 -- 172.21.15.57:0/172124947 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32d8005270 con 0x7f32e80cf340 2023-10-19T14:20:45.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.664+0000 7f32ddffb640 1 -- 172.21.15.57:0/172124947 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f32d80053d0 con 0x7f32e80cf340 2023-10-19T14:20:45.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.664+0000 7f32ef687640 1 -- 172.21.15.57:0/172124947 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32e4009490 con 0x7f32e80cf340 2023-10-19T14:20:45.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.664+0000 7f32ddffb640 1 -- 172.21.15.57:0/172124947 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32d800a8a0 con 0x7f32e80cf340 2023-10-19T14:20:45.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.664+0000 7f32ef687640 1 -- 172.21.15.57:0/172124947 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32e81f2920 con 0x7f32e80cf340 2023-10-19T14:20:45.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.665+0000 7f32ef687640 1 -- 172.21.15.57:0/172124947 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32e81f2b40 con 0x7f32e80cf340 2023-10-19T14:20:45.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.666+0000 7f32ddffb640 1 -- 172.21.15.57:0/172124947 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 133) v1 ==== 89490+0+0 (secure 0 0 0) 0x7f32d800aa00 con 0x7f32e80cf340 2023-10-19T14:20:45.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.666+0000 7f32ddffb640 1 --2- 172.21.15.57:0/172124947 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7f32cc06b110 0x7f32cc06d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:45.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.666+0000 7f32ddffb640 1 -- 172.21.15.57:0/172124947 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(228..228 src has 1..228) v4 ==== 7265+0+0 (secure 0 0 0) 0x7f32d8089f60 con 0x7f32e80cf340 2023-10-19T14:20:45.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.666+0000 7f32ed423640 1 -- 172.21.15.57:0/172124947 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7f32cc06b110 msgr2=0x7f32cc06d5d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/4242302683 2023-10-19T14:20:45.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.666+0000 7f32ed423640 1 --2- 172.21.15.57:0/172124947 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7f32cc06b110 0x7f32cc06d5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T14:20:45.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.670+0000 7f32ddffb640 1 -- 172.21.15.57:0/172124947 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f32d8058130 con 0x7f32e80cf340 2023-10-19T14:20:45.675 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:20:45.670+0000 7f10e7e3e040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-19T14:20:45.711 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:45.709+0000 7f4d48c11040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-19T14:20:45.718 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:20:45.715+0000 7f10e7e3e040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-19T14:20:45.754 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:45.751+0000 7f4d48c11040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-19T14:20:45.773 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:20:45.771+0000 7f10e7e3e040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-19T14:20:45.818 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:20:45.815+0000 7f10e7e3e040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-19T14:20:45.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.849+0000 7f32ef687640 1 -- 172.21.15.57:0/172124947 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f32e80b0cd0 con 0x7f32e80cf340 2023-10-19T14:20:45.852 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.849+0000 7f32ddffb640 1 -- 172.21.15.57:0/172124947 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v228) v1 ==== 104+0+11 (secure 0 0 0) 0x7f32d8057cc0 con 0x7f32e80cf340 2023-10-19T14:20:45.853 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2023-10-19T14:20:45.853 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:45.851+0000 7f4d48c11040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-19T14:20:45.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.851+0000 7f32ef687640 1 -- 172.21.15.57:0/172124947 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7f32cc06b110 msgr2=0x7f32cc06d5d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T14:20:45.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.851+0000 7f32ef687640 1 --2- 172.21.15.57:0/172124947 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7f32cc06b110 0x7f32cc06d5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:45.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.851+0000 7f32ef687640 1 -- 172.21.15.57:0/172124947 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f32e80cf340 msgr2=0x7f32e8159320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:45.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.851+0000 7f32ef687640 1 --2- 172.21.15.57:0/172124947 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f32e80cf340 0x7f32e8159320 secure :-1 s=READY pgs=2476 cs=0 l=1 rev1=1 crypto rx=0x7f32d800c710 tx=0x7f32d800cb10 comp rx=0 tx=0).stop 2023-10-19T14:20:45.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.852+0000 7f32ef687640 1 -- 172.21.15.57:0/172124947 shutdown_connections 2023-10-19T14:20:45.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.852+0000 7f32ef687640 1 --2- 172.21.15.57:0/172124947 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7f32cc06b110 0x7f32cc06d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:45.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.852+0000 7f32ef687640 1 --2- 172.21.15.57:0/172124947 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f32e815b2a0 0x7f32e8159860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:45.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.852+0000 7f32ef687640 1 --2- 172.21.15.57:0/172124947 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f32e80cf340 0x7f32e8159320 unknown :-1 s=CLOSED pgs=2476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:45.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.852+0000 7f32ef687640 1 --2- 172.21.15.57:0/172124947 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f32e80ceb40 0x7f32e81f12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:45.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.852+0000 7f32ef687640 1 -- 172.21.15.57:0/172124947 >> 172.21.15.57:0/172124947 conn(0x7f32e80d0d60 msgr2=0x7f32e815a560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:45.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.852+0000 7f32ef687640 1 -- 172.21.15.57:0/172124947 shutdown_connections 2023-10-19T14:20:45.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.852+0000 7f32ef687640 1 -- 172.21.15.57:0/172124947 wait complete. 2023-10-19T14:20:45.866 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-19T14:20:45.906 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:45.903+0000 7f4d48c11040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-19T14:20:45.923 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:20:45.920+0000 7f10e7e3e040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-19T14:20:45.947 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:45.945+0000 7f4d48c11040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-19T14:20:45.979 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.976+0000 7f0ce7af2640 1 -- 172.21.15.57:0/2606258078 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0ce00cf340 msgr2=0x7f0ce00d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:45.979 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.976+0000 7f0ce7af2640 1 --2- 172.21.15.57:0/2606258078 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0ce00cf340 0x7f0ce00d1910 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f0cdc009860 tx=0x7f0cdc02bd60 comp rx=0 tx=0).stop 2023-10-19T14:20:45.979 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.976+0000 7f0ce7af2640 1 -- 172.21.15.57:0/2606258078 shutdown_connections 2023-10-19T14:20:45.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.976+0000 7f0ce7af2640 1 --2- 172.21.15.57:0/2606258078 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0ce00cf340 0x7f0ce00d1910 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:45.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.976+0000 7f0ce7af2640 1 -- 172.21.15.57:0/2606258078 >> 172.21.15.57:0/2606258078 conn(0x7f0ce00d0d60 msgr2=0x7f0ce00d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:45.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.976+0000 7f0ce7af2640 1 -- 172.21.15.57:0/2606258078 shutdown_connections 2023-10-19T14:20:45.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.976+0000 7f0ce7af2640 1 -- 172.21.15.57:0/2606258078 wait complete. 2023-10-19T14:20:45.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.977+0000 7f0ce7af2640 1 Processor -- start 2023-10-19T14:20:45.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.977+0000 7f0ce7af2640 1 -- start start 2023-10-19T14:20:45.982 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:20:45.978+0000 7f10e7e3e040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-19T14:20:45.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.978+0000 7f0ce7af2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ce00ceb40 0x7f0ce01511e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:45.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.978+0000 7f0ce7af2640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0ce00cf340 0x7f0ce0151ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:45.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.978+0000 7f0ce7af2640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0ce015b2a0 0x7f0ce0152420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:45.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.978+0000 7f0ce7af2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ce0154d30 con 0x7f0ce00ceb40 2023-10-19T14:20:45.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.978+0000 7f0ce7af2640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ce015cd70 con 0x7f0ce015b2a0 2023-10-19T14:20:45.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.978+0000 7f0ce7af2640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ce0152aa0 con 0x7f0ce00cf340 2023-10-19T14:20:45.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.978+0000 7f0ce588e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ce00ceb40 0x7f0ce01511e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:45.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.978+0000 7f0ce588e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ce00ceb40 0x7f0ce01511e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42444/0 (socket says 172.21.15.57:42444) 2023-10-19T14:20:45.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.978+0000 7f0ce588e640 1 -- 172.21.15.57:0/2434549330 learned_addr learned my addr 172.21.15.57:0/2434549330 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:45.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.978+0000 7f0ce508d640 1 --2- 172.21.15.57:0/2434549330 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0ce00cf340 0x7f0ce0151ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:45.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.978+0000 7f0ce608f640 1 --2- 172.21.15.57:0/2434549330 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0ce015b2a0 0x7f0ce0152420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:45.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.978+0000 7f0ce588e640 1 -- 172.21.15.57:0/2434549330 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0ce00cf340 msgr2=0x7f0ce0151ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:45.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.978+0000 7f0ce588e640 1 --2- 172.21.15.57:0/2434549330 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0ce00cf340 0x7f0ce0151ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:45.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.978+0000 7f0ce588e640 1 -- 172.21.15.57:0/2434549330 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0ce015b2a0 msgr2=0x7f0ce0152420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:45.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.978+0000 7f0ce588e640 1 --2- 172.21.15.57:0/2434549330 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0ce015b2a0 0x7f0ce0152420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:45.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.978+0000 7f0ce588e640 1 -- 172.21.15.57:0/2434549330 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cdc009490 con 0x7f0ce00ceb40 2023-10-19T14:20:45.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.979+0000 7f0ce588e640 1 --2- 172.21.15.57:0/2434549330 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ce00ceb40 0x7f0ce01511e0 secure :-1 s=READY pgs=2478 cs=0 l=1 rev1=1 crypto rx=0x7f0cdc000c00 tx=0x7f0cdc0038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:45.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.979+0000 7f0cd67fc640 1 -- 172.21.15.57:0/2434549330 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cdc003ea0 con 0x7f0ce00ceb40 2023-10-19T14:20:45.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.979+0000 7f0cd67fc640 1 -- 172.21.15.57:0/2434549330 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f0cdc034440 con 0x7f0ce00ceb40 2023-10-19T14:20:45.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.979+0000 7f0ce7af2640 1 -- 172.21.15.57:0/2434549330 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ce0152d80 con 0x7f0ce00ceb40 2023-10-19T14:20:45.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.979+0000 7f0cd67fc640 1 -- 172.21.15.57:0/2434549330 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cdc034850 con 0x7f0ce00ceb40 2023-10-19T14:20:45.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.979+0000 7f0ce7af2640 1 -- 172.21.15.57:0/2434549330 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ce0151720 con 0x7f0ce00ceb40 2023-10-19T14:20:45.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.980+0000 7f0ce7af2640 1 -- 172.21.15.57:0/2434549330 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ce015cf40 con 0x7f0ce00ceb40 2023-10-19T14:20:45.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.980+0000 7f0cd67fc640 1 -- 172.21.15.57:0/2434549330 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 133) v1 ==== 89490+0+0 (secure 0 0 0) 0x7f0cdc0349b0 con 0x7f0ce00ceb40 2023-10-19T14:20:45.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.981+0000 7f0cd67fc640 1 --2- 172.21.15.57:0/2434549330 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7f0cc406b110 0x7f0cc406d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:45.989 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.981+0000 7f0ce508d640 1 -- 172.21.15.57:0/2434549330 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7f0cc406b110 msgr2=0x7f0cc406d5d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/4242302683 2023-10-19T14:20:45.989 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.981+0000 7f0cd67fc640 1 -- 172.21.15.57:0/2434549330 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(228..228 src has 1..228) v4 ==== 7265+0+0 (secure 0 0 0) 0x7f0cdc0ac7c0 con 0x7f0ce00ceb40 2023-10-19T14:20:45.989 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.981+0000 7f0ce508d640 1 --2- 172.21.15.57:0/2434549330 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7f0cc406b110 0x7f0cc406d5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T14:20:45.990 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:45.985+0000 7f0cd67fc640 1 -- 172.21.15.57:0/2434549330 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0cdc076fb0 con 0x7f0ce00ceb40 2023-10-19T14:20:45.996 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:45.993+0000 7f4d48c11040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-19T14:20:46.026 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:20:46.023+0000 7f10e7e3e040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-19T14:20:46.038 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:46.036+0000 7f4d48c11040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-19T14:20:46.077 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:20:46.074+0000 7f10e7e3e040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-19T14:20:46.122 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:20:46.119+0000 7f10e7e3e040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-19T14:20:46.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.162+0000 7f0ce7af2640 1 -- 172.21.15.57:0/2434549330 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0ce00690e0 con 0x7f0ce00ceb40 2023-10-19T14:20:46.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.163+0000 7f0cd67fc640 1 -- 172.21.15.57:0/2434549330 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v228) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0cdc07a5d0 con 0x7f0ce00ceb40 2023-10-19T14:20:46.166 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 64 2023-10-19T14:20:46.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.165+0000 7f0ce7af2640 1 -- 172.21.15.57:0/2434549330 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7f0cc406b110 msgr2=0x7f0cc406d5d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T14:20:46.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.165+0000 7f0ce7af2640 1 --2- 172.21.15.57:0/2434549330 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7f0cc406b110 0x7f0cc406d5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:46.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.165+0000 7f0ce7af2640 1 -- 172.21.15.57:0/2434549330 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ce00ceb40 msgr2=0x7f0ce01511e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:46.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.165+0000 7f0ce7af2640 1 --2- 172.21.15.57:0/2434549330 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ce00ceb40 0x7f0ce01511e0 secure :-1 s=READY pgs=2478 cs=0 l=1 rev1=1 crypto rx=0x7f0cdc000c00 tx=0x7f0cdc0038a0 comp rx=0 tx=0).stop 2023-10-19T14:20:46.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.165+0000 7f0ce7af2640 1 -- 172.21.15.57:0/2434549330 shutdown_connections 2023-10-19T14:20:46.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.165+0000 7f0ce7af2640 1 --2- 172.21.15.57:0/2434549330 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7f0cc406b110 0x7f0cc406d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:46.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.165+0000 7f0ce7af2640 1 --2- 172.21.15.57:0/2434549330 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0ce015b2a0 0x7f0ce0152420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:46.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.165+0000 7f0ce7af2640 1 --2- 172.21.15.57:0/2434549330 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0ce00cf340 0x7f0ce0151ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:46.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.165+0000 7f0ce7af2640 1 --2- 172.21.15.57:0/2434549330 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ce00ceb40 0x7f0ce01511e0 unknown :-1 s=CLOSED pgs=2478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:46.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.165+0000 7f0ce7af2640 1 -- 172.21.15.57:0/2434549330 >> 172.21.15.57:0/2434549330 conn(0x7f0ce00d0d60 msgr2=0x7f0ce00b0ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:46.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.165+0000 7f0ce7af2640 1 -- 172.21.15.57:0/2434549330 shutdown_connections 2023-10-19T14:20:46.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.165+0000 7f0ce7af2640 1 -- 172.21.15.57:0/2434549330 wait complete. 2023-10-19T14:20:46.179 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2023-10-19T14:20:46.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.289+0000 7fde6f933640 1 -- 172.21.15.57:0/1590210608 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde680cf340 msgr2=0x7fde680d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:46.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.289+0000 7fde6f933640 1 --2- 172.21.15.57:0/1590210608 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde680cf340 0x7fde680d1910 secure :-1 s=READY pgs=2480 cs=0 l=1 rev1=1 crypto rx=0x7fde64009830 tx=0x7fde6402bd60 comp rx=0 tx=0).stop 2023-10-19T14:20:46.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.289+0000 7fde6f933640 1 -- 172.21.15.57:0/1590210608 shutdown_connections 2023-10-19T14:20:46.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.289+0000 7fde6f933640 1 --2- 172.21.15.57:0/1590210608 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde680cf340 0x7fde680d1910 unknown :-1 s=CLOSED pgs=2480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:46.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.289+0000 7fde6f933640 1 -- 172.21.15.57:0/1590210608 >> 172.21.15.57:0/1590210608 conn(0x7fde680d0d60 msgr2=0x7fde680d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:46.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.289+0000 7fde6f933640 1 -- 172.21.15.57:0/1590210608 shutdown_connections 2023-10-19T14:20:46.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.289+0000 7fde6f933640 1 -- 172.21.15.57:0/1590210608 wait complete. 2023-10-19T14:20:46.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.290+0000 7fde6f933640 1 Processor -- start 2023-10-19T14:20:46.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.290+0000 7fde6f933640 1 -- start start 2023-10-19T14:20:46.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.291+0000 7fde6f933640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fde680ceb40 0x7fde680b6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:46.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.291+0000 7fde6f933640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde680cf340 0x7fde680b6890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:46.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.291+0000 7fde6f933640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fde6815b2a0 0x7fde68149a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:46.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.291+0000 7fde6f933640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde6814a870 con 0x7fde680cf340 2023-10-19T14:20:46.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.291+0000 7fde6f933640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde6815c980 con 0x7fde680ceb40 2023-10-19T14:20:46.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.291+0000 7fde6f933640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde6815cc60 con 0x7fde6815b2a0 2023-10-19T14:20:46.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.291+0000 7fde6cece640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde680cf340 0x7fde680b6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:46.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.291+0000 7fde6cece640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde680cf340 0x7fde680b6890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42470/0 (socket says 172.21.15.57:42470) 2023-10-19T14:20:46.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.291+0000 7fde6cece640 1 -- 172.21.15.57:0/1714369057 learned_addr learned my addr 172.21.15.57:0/1714369057 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:46.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.291+0000 7fde6d6cf640 1 --2- 172.21.15.57:0/1714369057 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fde680ceb40 0x7fde680b6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:46.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.291+0000 7fde6ded0640 1 --2- 172.21.15.57:0/1714369057 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fde6815b2a0 0x7fde68149a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:46.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.291+0000 7fde6cece640 1 -- 172.21.15.57:0/1714369057 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fde6815b2a0 msgr2=0x7fde68149a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:46.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.291+0000 7fde6cece640 1 --2- 172.21.15.57:0/1714369057 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fde6815b2a0 0x7fde68149a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:46.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.291+0000 7fde6cece640 1 -- 172.21.15.57:0/1714369057 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fde680ceb40 msgr2=0x7fde680b6350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:46.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.291+0000 7fde6cece640 1 --2- 172.21.15.57:0/1714369057 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fde680ceb40 0x7fde680b6350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:46.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.291+0000 7fde6cece640 1 -- 172.21.15.57:0/1714369057 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde64009490 con 0x7fde680cf340 2023-10-19T14:20:46.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.291+0000 7fde6cece640 1 --2- 172.21.15.57:0/1714369057 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde680cf340 0x7fde680b6890 secure :-1 s=READY pgs=2481 cs=0 l=1 rev1=1 crypto rx=0x7fde5800c3c0 tx=0x7fde5800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:46.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.292+0000 7fde5dffb640 1 -- 172.21.15.57:0/1714369057 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde58011790 con 0x7fde680cf340 2023-10-19T14:20:46.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.292+0000 7fde5dffb640 1 -- 172.21.15.57:0/1714369057 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fde580119e0 con 0x7fde680cf340 2023-10-19T14:20:46.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.292+0000 7fde6f933640 1 -- 172.21.15.57:0/1714369057 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde6815cf40 con 0x7fde680cf340 2023-10-19T14:20:46.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.292+0000 7fde5dffb640 1 -- 172.21.15.57:0/1714369057 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde58011e10 con 0x7fde680cf340 2023-10-19T14:20:46.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.292+0000 7fde6f933640 1 -- 172.21.15.57:0/1714369057 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde6815d500 con 0x7fde680cf340 2023-10-19T14:20:46.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.293+0000 7fde6f933640 1 -- 172.21.15.57:0/1714369057 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde6815d720 con 0x7fde680cf340 2023-10-19T14:20:46.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.293+0000 7fde5dffb640 1 -- 172.21.15.57:0/1714369057 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 133) v1 ==== 89490+0+0 (secure 0 0 0) 0x7fde5801c460 con 0x7fde680cf340 2023-10-19T14:20:46.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.294+0000 7fde5dffb640 1 --2- 172.21.15.57:0/1714369057 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7fde4c06b110 0x7fde4c06d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:46.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.294+0000 7fde6d6cf640 1 -- 172.21.15.57:0/1714369057 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7fde4c06b110 msgr2=0x7fde4c06d5d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/4242302683 2023-10-19T14:20:46.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.294+0000 7fde5dffb640 1 -- 172.21.15.57:0/1714369057 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(228..228 src has 1..228) v4 ==== 7265+0+0 (secure 0 0 0) 0x7fde58089ef0 con 0x7fde680cf340 2023-10-19T14:20:46.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.294+0000 7fde6d6cf640 1 --2- 172.21.15.57:0/1714369057 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7fde4c06b110 0x7fde4c06d5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T14:20:46.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.298+0000 7fde5dffb640 1 -- 172.21.15.57:0/1714369057 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fde58054770 con 0x7fde680cf340 2023-10-19T14:20:46.354 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:46.351+0000 7f4d48c11040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-19T14:20:46.461 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:20:46.459+0000 7f10e7e3e040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-19T14:20:46.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.474+0000 7fde6f933640 1 -- 172.21.15.57:0/1714369057 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fde680690e0 con 0x7fde680cf340 2023-10-19T14:20:46.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.475+0000 7fde5dffb640 1 -- 172.21.15.57:0/1714369057 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v228) v1 ==== 107+0+11 (secure 0 0 0) 0x7fde58057d90 con 0x7fde680cf340 2023-10-19T14:20:46.478 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2023-10-19T14:20:46.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.477+0000 7fde6f933640 1 -- 172.21.15.57:0/1714369057 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7fde4c06b110 msgr2=0x7fde4c06d5d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T14:20:46.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.477+0000 7fde6f933640 1 --2- 172.21.15.57:0/1714369057 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7fde4c06b110 0x7fde4c06d5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:46.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.477+0000 7fde6f933640 1 -- 172.21.15.57:0/1714369057 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde680cf340 msgr2=0x7fde680b6890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:46.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.477+0000 7fde6f933640 1 --2- 172.21.15.57:0/1714369057 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde680cf340 0x7fde680b6890 secure :-1 s=READY pgs=2481 cs=0 l=1 rev1=1 crypto rx=0x7fde5800c3c0 tx=0x7fde5800c880 comp rx=0 tx=0).stop 2023-10-19T14:20:46.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.477+0000 7fde6f933640 1 -- 172.21.15.57:0/1714369057 shutdown_connections 2023-10-19T14:20:46.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.477+0000 7fde6f933640 1 --2- 172.21.15.57:0/1714369057 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7fde4c06b110 0x7fde4c06d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:46.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.477+0000 7fde6f933640 1 --2- 172.21.15.57:0/1714369057 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fde6815b2a0 0x7fde68149a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:46.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.477+0000 7fde6f933640 1 --2- 172.21.15.57:0/1714369057 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde680cf340 0x7fde680b6890 unknown :-1 s=CLOSED pgs=2481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:46.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.477+0000 7fde6f933640 1 --2- 172.21.15.57:0/1714369057 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fde680ceb40 0x7fde680b6350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:46.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.477+0000 7fde6f933640 1 -- 172.21.15.57:0/1714369057 >> 172.21.15.57:0/1714369057 conn(0x7fde680d0d60 msgr2=0x7fde680b0ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:46.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.477+0000 7fde6f933640 1 -- 172.21.15.57:0/1714369057 shutdown_connections 2023-10-19T14:20:46.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.477+0000 7fde6f933640 1 -- 172.21.15.57:0/1714369057 wait complete. 2023-10-19T14:20:46.485 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:46.483+0000 7f4d48c11040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-19T14:20:46.491 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2023-10-19T14:20:46.577 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:46.574+0000 7f4d48c11040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-19T14:20:46.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.602+0000 7fc77af39640 1 -- 172.21.15.57:0/4100138993 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc7740cf340 msgr2=0x7fc7740d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:46.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.602+0000 7fc77af39640 1 --2- 172.21.15.57:0/4100138993 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc7740cf340 0x7fc7740d1910 secure :-1 s=READY pgs=2483 cs=0 l=1 rev1=1 crypto rx=0x7fc768005fd0 tx=0x7fc76802be20 comp rx=0 tx=0).stop 2023-10-19T14:20:46.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.602+0000 7fc77af39640 1 -- 172.21.15.57:0/4100138993 shutdown_connections 2023-10-19T14:20:46.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.602+0000 7fc77af39640 1 --2- 172.21.15.57:0/4100138993 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc7740cf340 0x7fc7740d1910 unknown :-1 s=CLOSED pgs=2483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:46.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.602+0000 7fc77af39640 1 -- 172.21.15.57:0/4100138993 >> 172.21.15.57:0/4100138993 conn(0x7fc7740d0d60 msgr2=0x7fc7740d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:46.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.602+0000 7fc77af39640 1 -- 172.21.15.57:0/4100138993 shutdown_connections 2023-10-19T14:20:46.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.602+0000 7fc77af39640 1 -- 172.21.15.57:0/4100138993 wait complete. 2023-10-19T14:20:46.607 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:20:46.602+0000 7f10e7e3e040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-19T14:20:46.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.603+0000 7fc77af39640 1 Processor -- start 2023-10-19T14:20:46.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.603+0000 7fc77af39640 1 -- start start 2023-10-19T14:20:46.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.604+0000 7fc77af39640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc7740ceb40 0x7fc7740c8ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:46.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.604+0000 7fc77af39640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc7740cf340 0x7fc77415e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:46.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.604+0000 7fc77af39640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc77415b2a0 0x7fc77414c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:46.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.604+0000 7fc77af39640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc77415e680 con 0x7fc77415b2a0 2023-10-19T14:20:46.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.604+0000 7fc77af39640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc77414c6f0 con 0x7fc7740ceb40 2023-10-19T14:20:46.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.604+0000 7fc77af39640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7740c95f0 con 0x7fc7740cf340 2023-10-19T14:20:46.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.604+0000 7fc7794d6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc77415b2a0 0x7fc77414c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:46.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.604+0000 7fc7794d6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc77415b2a0 0x7fc77414c1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42492/0 (socket says 172.21.15.57:42492) 2023-10-19T14:20:46.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.604+0000 7fc7794d6640 1 -- 172.21.15.57:0/376785645 learned_addr learned my addr 172.21.15.57:0/376785645 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:46.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.604+0000 7fc778cd5640 1 --2- 172.21.15.57:0/376785645 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc7740ceb40 0x7fc7740c8ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:46.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.604+0000 7fc773fff640 1 --2- 172.21.15.57:0/376785645 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc7740cf340 0x7fc77415e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:46.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.604+0000 7fc7794d6640 1 -- 172.21.15.57:0/376785645 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc7740cf340 msgr2=0x7fc77415e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:46.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.604+0000 7fc7794d6640 1 --2- 172.21.15.57:0/376785645 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc7740cf340 0x7fc77415e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:46.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.604+0000 7fc7794d6640 1 -- 172.21.15.57:0/376785645 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc7740ceb40 msgr2=0x7fc7740c8ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:46.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.604+0000 7fc7794d6640 1 --2- 172.21.15.57:0/376785645 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc7740ceb40 0x7fc7740c8ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:46.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.604+0000 7fc7794d6640 1 -- 172.21.15.57:0/376785645 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc768009560 con 0x7fc77415b2a0 2023-10-19T14:20:46.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.605+0000 7fc7794d6640 1 --2- 172.21.15.57:0/376785645 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc77415b2a0 0x7fc77414c1b0 secure :-1 s=READY pgs=2484 cs=0 l=1 rev1=1 crypto rx=0x7fc76c00c7a0 tx=0x7fc76c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:46.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.605+0000 7fc7717fa640 1 -- 172.21.15.57:0/376785645 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc76c003b70 con 0x7fc77415b2a0 2023-10-19T14:20:46.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.605+0000 7fc7717fa640 1 -- 172.21.15.57:0/376785645 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fc76c003cf0 con 0x7fc77415b2a0 2023-10-19T14:20:46.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.605+0000 7fc7717fa640 1 -- 172.21.15.57:0/376785645 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc76c003ea0 con 0x7fc77415b2a0 2023-10-19T14:20:46.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.605+0000 7fc77af39640 1 -- 172.21.15.57:0/376785645 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7740c98d0 con 0x7fc77415b2a0 2023-10-19T14:20:46.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.605+0000 7fc77af39640 1 -- 172.21.15.57:0/376785645 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7740c9e90 con 0x7fc77415b2a0 2023-10-19T14:20:46.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.606+0000 7fc77af39640 1 -- 172.21.15.57:0/376785645 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7740d04f0 con 0x7fc77415b2a0 2023-10-19T14:20:46.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.606+0000 7fc7717fa640 1 -- 172.21.15.57:0/376785645 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 133) v1 ==== 89490+0+0 (secure 0 0 0) 0x7fc76c00aab0 con 0x7fc77415b2a0 2023-10-19T14:20:46.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.607+0000 7fc7717fa640 1 --2- 172.21.15.57:0/376785645 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7fc75806b110 0x7fc75806d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:46.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.607+0000 7fc778cd5640 1 -- 172.21.15.57:0/376785645 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7fc75806b110 msgr2=0x7fc75806d5d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/4242302683 2023-10-19T14:20:46.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.607+0000 7fc7717fa640 1 -- 172.21.15.57:0/376785645 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(228..228 src has 1..228) v4 ==== 7265+0+0 (secure 0 0 0) 0x7fc76c089f90 con 0x7fc77415b2a0 2023-10-19T14:20:46.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.607+0000 7fc778cd5640 1 --2- 172.21.15.57:0/376785645 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7fc75806b110 0x7fc75806d5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T14:20:46.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.615+0000 7fc7717fa640 1 -- 172.21.15.57:0/376785645 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc76c054700 con 0x7fc77415b2a0 2023-10-19T14:20:46.620 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:46.617+0000 7f4d48c11040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-19T14:20:46.706 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:20:46.703+0000 7f10e7e3e040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-19T14:20:46.715 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:46.713+0000 7f4d48c11040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-19T14:20:46.753 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:20:46.750+0000 7f10e7e3e040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-19T14:20:46.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.796+0000 7fc77af39640 1 -- 172.21.15.57:0/376785645 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) v1 -- 0x7fc7740690e0 con 0x7fc77415b2a0 2023-10-19T14:20:46.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.797+0000 7fc7717fa640 1 -- 172.21.15.57:0/376785645 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v228) v1 ==== 103+0+11 (secure 0 0 0) 0x7fc76c057d20 con 0x7fc77415b2a0 2023-10-19T14:20:46.800 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 64 2023-10-19T14:20:46.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.799+0000 7fc77af39640 1 -- 172.21.15.57:0/376785645 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7fc75806b110 msgr2=0x7fc75806d5d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T14:20:46.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.799+0000 7fc77af39640 1 --2- 172.21.15.57:0/376785645 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7fc75806b110 0x7fc75806d5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:46.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.799+0000 7fc77af39640 1 -- 172.21.15.57:0/376785645 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc77415b2a0 msgr2=0x7fc77414c1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:46.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.799+0000 7fc77af39640 1 --2- 172.21.15.57:0/376785645 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc77415b2a0 0x7fc77414c1b0 secure :-1 s=READY pgs=2484 cs=0 l=1 rev1=1 crypto rx=0x7fc76c00c7a0 tx=0x7fc76c00cc60 comp rx=0 tx=0).stop 2023-10-19T14:20:46.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.799+0000 7fc77af39640 1 -- 172.21.15.57:0/376785645 shutdown_connections 2023-10-19T14:20:46.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.799+0000 7fc77af39640 1 --2- 172.21.15.57:0/376785645 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7fc75806b110 0x7fc75806d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:46.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.799+0000 7fc77af39640 1 --2- 172.21.15.57:0/376785645 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc77415b2a0 0x7fc77414c1b0 unknown :-1 s=CLOSED pgs=2484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:46.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.799+0000 7fc77af39640 1 --2- 172.21.15.57:0/376785645 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc7740cf340 0x7fc77415e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:46.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.799+0000 7fc77af39640 1 --2- 172.21.15.57:0/376785645 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc7740ceb40 0x7fc7740c8ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:46.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.799+0000 7fc77af39640 1 -- 172.21.15.57:0/376785645 >> 172.21.15.57:0/376785645 conn(0x7fc7740d0d60 msgr2=0x7fc7740cf720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:46.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.799+0000 7fc77af39640 1 -- 172.21.15.57:0/376785645 shutdown_connections 2023-10-19T14:20:46.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.799+0000 7fc77af39640 1 -- 172.21.15.57:0/376785645 wait complete. 2023-10-19T14:20:46.812 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:46.809+0000 7f4d48c11040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-19T14:20:46.813 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-19T14:20:46.859 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:20:46.856+0000 7f10e7e3e040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-19T14:20:46.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.922+0000 7f8e6ac4b640 1 -- 172.21.15.57:0/689455161 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e64159060 msgr2=0x7f8e6406d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:46.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.922+0000 7f8e6ac4b640 1 --2- 172.21.15.57:0/689455161 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e64159060 0x7f8e6406d260 secure :-1 s=READY pgs=2486 cs=0 l=1 rev1=1 crypto rx=0x7f8e58009830 tx=0x7f8e5802bcd0 comp rx=0 tx=0).stop 2023-10-19T14:20:46.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.922+0000 7f8e6ac4b640 1 -- 172.21.15.57:0/689455161 shutdown_connections 2023-10-19T14:20:46.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.922+0000 7f8e6ac4b640 1 --2- 172.21.15.57:0/689455161 >> v2:172.21.15.179:3300/0 conn(0x7f8e6415d400 0x7f8e640cdfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:46.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.922+0000 7f8e6ac4b640 1 --2- 172.21.15.57:0/689455161 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e64159060 0x7f8e6406d260 unknown :-1 s=CLOSED pgs=2486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:46.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.922+0000 7f8e6ac4b640 1 -- 172.21.15.57:0/689455161 >> 172.21.15.57:0/689455161 conn(0x7f8e640b0670 msgr2=0x7f8e640b0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:46.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.922+0000 7f8e6ac4b640 1 -- 172.21.15.57:0/689455161 shutdown_connections 2023-10-19T14:20:46.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.922+0000 7f8e6ac4b640 1 -- 172.21.15.57:0/689455161 wait complete. 2023-10-19T14:20:46.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.924+0000 7f8e6ac4b640 1 Processor -- start 2023-10-19T14:20:46.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.924+0000 7f8e6ac4b640 1 -- start start 2023-10-19T14:20:46.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.924+0000 7f8e6ac4b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e640cdb30 0x7f8e641560a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:46.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.925+0000 7f8e6ac4b640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8e64159060 0x7f8e640c45c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:46.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.925+0000 7f8e6ac4b640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8e6415d400 0x7f8e640cca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:46.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.925+0000 7f8e6ac4b640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e641565e0 con 0x7f8e640cdb30 2023-10-19T14:20:46.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.925+0000 7f8e6ac4b640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e640b24f0 con 0x7f8e64159060 2023-10-19T14:20:46.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.925+0000 7f8e6ac4b640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e640b18e0 con 0x7f8e6415d400 2023-10-19T14:20:46.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.925+0000 7f8e689e7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e640cdb30 0x7f8e641560a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:46.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.925+0000 7f8e689e7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e640cdb30 0x7f8e641560a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55872/0 (socket says 172.21.15.57:55872) 2023-10-19T14:20:46.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.925+0000 7f8e689e7640 1 -- 172.21.15.57:0/1397211310 learned_addr learned my addr 172.21.15.57:0/1397211310 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:46.932 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.925+0000 7f8e63fff640 1 --2- 172.21.15.57:0/1397211310 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8e64159060 0x7f8e640c45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:46.932 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.925+0000 7f8e691e8640 1 --2- 172.21.15.57:0/1397211310 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8e6415d400 0x7f8e640cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:46.932 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.925+0000 7f8e689e7640 1 -- 172.21.15.57:0/1397211310 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8e6415d400 msgr2=0x7f8e640cca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:46.932 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.925+0000 7f8e689e7640 1 --2- 172.21.15.57:0/1397211310 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8e6415d400 0x7f8e640cca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:46.932 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.925+0000 7f8e689e7640 1 -- 172.21.15.57:0/1397211310 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8e64159060 msgr2=0x7f8e640c45c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:46.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.925+0000 7f8e689e7640 1 --2- 172.21.15.57:0/1397211310 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8e64159060 0x7f8e640c45c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:46.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.925+0000 7f8e689e7640 1 -- 172.21.15.57:0/1397211310 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e58009490 con 0x7f8e640cdb30 2023-10-19T14:20:46.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.925+0000 7f8e63fff640 1 --2- 172.21.15.57:0/1397211310 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8e64159060 0x7f8e640c45c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:20:46.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.925+0000 7f8e691e8640 1 --2- 172.21.15.57:0/1397211310 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8e6415d400 0x7f8e640cca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:20:46.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.925+0000 7f8e689e7640 1 --2- 172.21.15.57:0/1397211310 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e640cdb30 0x7f8e641560a0 secure :-1 s=READY pgs=2487 cs=0 l=1 rev1=1 crypto rx=0x7f8e58000c00 tx=0x7f8e580038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:46.934 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.925+0000 7f8e617fa640 1 -- 172.21.15.57:0/1397211310 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e58003ea0 con 0x7f8e640cdb30 2023-10-19T14:20:46.934 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.925+0000 7f8e617fa640 1 -- 172.21.15.57:0/1397211310 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f8e58034440 con 0x7f8e640cdb30 2023-10-19T14:20:46.934 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.925+0000 7f8e6ac4b640 1 -- 172.21.15.57:0/1397211310 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e640b1bc0 con 0x7f8e640cdb30 2023-10-19T14:20:46.934 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.925+0000 7f8e617fa640 1 -- 172.21.15.57:0/1397211310 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e58034850 con 0x7f8e640cdb30 2023-10-19T14:20:46.934 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.925+0000 7f8e6ac4b640 1 -- 172.21.15.57:0/1397211310 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e640b2180 con 0x7f8e640cdb30 2023-10-19T14:20:46.935 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.926+0000 7f8e6ac4b640 1 -- 172.21.15.57:0/1397211310 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e640675f0 con 0x7f8e640cdb30 2023-10-19T14:20:46.936 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.927+0000 7f8e617fa640 1 -- 172.21.15.57:0/1397211310 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 133) v1 ==== 89490+0+0 (secure 0 0 0) 0x7f8e580349b0 con 0x7f8e640cdb30 2023-10-19T14:20:46.936 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.928+0000 7f8e617fa640 1 --2- 172.21.15.57:0/1397211310 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7f8e4806b110 0x7f8e4806d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:46.936 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.928+0000 7f8e63fff640 1 -- 172.21.15.57:0/1397211310 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7f8e4806b110 msgr2=0x7f8e4806d5d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/4242302683 2023-10-19T14:20:46.936 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.928+0000 7f8e617fa640 1 -- 172.21.15.57:0/1397211310 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(228..228 src has 1..228) v4 ==== 7265+0+0 (secure 0 0 0) 0x7f8e580ac830 con 0x7f8e640cdb30 2023-10-19T14:20:46.936 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.928+0000 7f8e63fff640 1 --2- 172.21.15.57:0/1397211310 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7f8e4806b110 0x7f8e4806d5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T14:20:46.937 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:46.932+0000 7f8e617fa640 1 -- 172.21.15.57:0/1397211310 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8e58077000 con 0x7f8e640cdb30 2023-10-19T14:20:46.950 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:46.948+0000 7f4d48c11040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-19T14:20:46.958 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:20:46.956+0000 7f10e7e3e040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-19T14:20:46.992 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:46.989+0000 7f4d48c11040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-19T14:20:47.096 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:47.094+0000 7f4d48c11040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-19T14:20:47.103 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:20:47.100+0000 7f10e7e3e040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-19T14:20:47.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:47.122+0000 7f8e6ac4b640 1 -- 172.21.15.57:0/1397211310 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f8e640b4aa0 con 0x7f8e640cdb30 2023-10-19T14:20:47.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:47.122+0000 7f8e617fa640 1 -- 172.21.15.57:0/1397211310 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v228) v1 ==== 80+0+477 (secure 0 0 0) 0x7f8e5807a620 con 0x7f8e640cdb30 2023-10-19T14:20:47.126 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/1194984394 2023-10-20T14:20:39.702634+0000 2023-10-19T14:20:47.126 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:6833/3232482678 2023-10-20T14:20:39.702634+0000 2023-10-19T14:20:47.126 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:6832/3232482678 2023-10-20T14:20:39.702634+0000 2023-10-19T14:20:47.127 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/2872579179 2023-10-20T14:20:39.702634+0000 2023-10-19T14:20:47.127 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/3910604357 2023-10-20T14:20:39.702634+0000 2023-10-19T14:20:47.127 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/536160557 2023-10-20T14:20:39.702634+0000 2023-10-19T14:20:47.127 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/3239062050 2023-10-20T14:20:39.702634+0000 2023-10-19T14:20:47.127 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/1948632334 2023-10-20T14:20:39.702634+0000 2023-10-19T14:20:47.128 INFO:teuthology.orchestra.run.smithi057.stderr:listed 8 entries 2023-10-19T14:20:47.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:47.125+0000 7f8e6ac4b640 1 -- 172.21.15.57:0/1397211310 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7f8e4806b110 msgr2=0x7f8e4806d5d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T14:20:47.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:47.125+0000 7f8e6ac4b640 1 --2- 172.21.15.57:0/1397211310 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7f8e4806b110 0x7f8e4806d5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:47.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:47.125+0000 7f8e6ac4b640 1 -- 172.21.15.57:0/1397211310 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e640cdb30 msgr2=0x7f8e641560a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:47.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:47.125+0000 7f8e6ac4b640 1 --2- 172.21.15.57:0/1397211310 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e640cdb30 0x7f8e641560a0 secure :-1 s=READY pgs=2487 cs=0 l=1 rev1=1 crypto rx=0x7f8e58000c00 tx=0x7f8e580038a0 comp rx=0 tx=0).stop 2023-10-19T14:20:47.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:47.125+0000 7f8e6ac4b640 1 -- 172.21.15.57:0/1397211310 shutdown_connections 2023-10-19T14:20:47.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:47.125+0000 7f8e6ac4b640 1 --2- 172.21.15.57:0/1397211310 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7f8e4806b110 0x7f8e4806d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:47.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:47.125+0000 7f8e6ac4b640 1 --2- 172.21.15.57:0/1397211310 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8e6415d400 0x7f8e640cca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:47.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:47.125+0000 7f8e6ac4b640 1 --2- 172.21.15.57:0/1397211310 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8e64159060 0x7f8e640c45c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:47.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:47.125+0000 7f8e6ac4b640 1 --2- 172.21.15.57:0/1397211310 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e640cdb30 0x7f8e641560a0 unknown :-1 s=CLOSED pgs=2487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:47.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:47.125+0000 7f8e6ac4b640 1 -- 172.21.15.57:0/1397211310 >> 172.21.15.57:0/1397211310 conn(0x7f8e640b0670 msgr2=0x7f8e640ac3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:47.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:47.125+0000 7f8e6ac4b640 1 -- 172.21.15.57:0/1397211310 shutdown_connections 2023-10-19T14:20:47.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:47.125+0000 7f8e6ac4b640 1 -- 172.21.15.57:0/1397211310 wait complete. 2023-10-19T14:20:47.138 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi057.front.sepia.ceph.com... 2023-10-19T14:20:47.139 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:20:47.139 DEBUG:teuthology.orchestra.run.smithi057:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-10-19T14:20:47.148 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:20:47.146+0000 7f10e7e3e040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-19T14:20:47.168 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi057.stderr:ceph-fuse[153701]: fuse finished with error 0 and tester_r 0 2023-10-19T14:20:47.170 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi057.front.sepia.ceph.com 2023-10-19T14:20:47.171 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:20:47.171 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-19T14:20:47.260 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:20:47.258+0000 7f10e7e3e040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-19T14:20:47.336 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-19T14:20:47.336 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T14:20:47.351 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054833,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=284 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3247632k,nr_inodes=811908,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-19T14:20:47.352 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi179.front.sepia.ceph.com 2023-10-19T14:20:47.352 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:20:47.353 DEBUG:teuthology.orchestra.run.smithi179:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-10-19T14:20:47.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-19T14:20:47.406 INFO:teuthology.orchestra.run.smithi179.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2023-10-19T14:20:47.406 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T14:20:47.407 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T14:20:47.422 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055614,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=358 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248260k,nr_inodes=812065,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-19T14:20:47.423 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi057.front.sepia.ceph.com 2023-10-19T14:20:47.423 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:20:47.424 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-10-19T14:20:47.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-19T14:20:47.478 INFO:teuthology.orchestra.run.smithi057.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2023-10-19T14:20:47.479 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-19T14:20:47.479 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T14:20:47.494 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054833,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=284 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3247632k,nr_inodes=811908,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-19T14:20:47.494 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi179.front.sepia.ceph.com 2023-10-19T14:20:47.495 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:20:47.495 DEBUG:teuthology.orchestra.run.smithi179:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-10-19T14:20:47.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-19T14:20:47.549 INFO:teuthology.orchestra.run.smithi179.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2023-10-19T14:20:47.549 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T14:20:47.660 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:47.657+0000 7fec31fa9640 1 -- 172.21.15.57:0/837770875 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec2c0704e0 msgr2=0x7fec2c0cc910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:47.660 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:47.657+0000 7fec31fa9640 1 --2- 172.21.15.57:0/837770875 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec2c0704e0 0x7fec2c0cc910 secure :-1 s=READY pgs=2489 cs=0 l=1 rev1=1 crypto rx=0x7fec24005fd0 tx=0x7fec2402c370 comp rx=0 tx=0).stop 2023-10-19T14:20:47.660 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:47.657+0000 7fec31fa9640 1 -- 172.21.15.57:0/837770875 shutdown_connections 2023-10-19T14:20:47.661 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:47.657+0000 7fec31fa9640 1 --2- 172.21.15.57:0/837770875 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec2c0704e0 0x7fec2c0cc910 unknown :-1 s=CLOSED pgs=2489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:47.661 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:47.657+0000 7fec31fa9640 1 -- 172.21.15.57:0/837770875 >> 172.21.15.57:0/837770875 conn(0x7fec2c0af730 msgr2=0x7fec2c0afb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:47.661 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:47.657+0000 7fec31fa9640 1 -- 172.21.15.57:0/837770875 shutdown_connections 2023-10-19T14:20:47.661 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:47.657+0000 7fec31fa9640 1 -- 172.21.15.57:0/837770875 wait complete. 2023-10-19T14:20:47.664 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:47.658+0000 7fec31fa9640 1 Processor -- start 2023-10-19T14:20:47.664 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:47.658+0000 7fec31fa9640 1 -- start start 2023-10-19T14:20:47.664 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:47.659+0000 7fec31fa9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec2c070080 0x7fec2c136f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:47.664 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:47.659+0000 7fec31fa9640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fec2c0704e0 0x7fec2c13c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:47.664 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:47.659+0000 7fec31fa9640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fec2c1416f0 0x7fec2c13cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:47.665 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:47.659+0000 7fec31fa9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec2c06cee0 con 0x7fec2c070080 2023-10-19T14:20:47.665 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:47.659+0000 7fec31fa9640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec2c159c60 con 0x7fec2c1416f0 2023-10-19T14:20:47.665 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:47.659+0000 7fec31fa9640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec2c159f40 con 0x7fec2c0704e0 2023-10-19T14:20:47.665 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:47.659+0000 7fec2b7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec2c070080 0x7fec2c136f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:47.665 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:47.659+0000 7fec2b7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec2c070080 0x7fec2c136f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55890/0 (socket says 172.21.15.57:55890) 2023-10-19T14:20:47.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:47.659+0000 7fec2b7fe640 1 -- 172.21.15.57:0/2425673004 learned_addr learned my addr 172.21.15.57:0/2425673004 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:47.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:47.659+0000 7fec2bfff640 1 --2- 172.21.15.57:0/2425673004 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fec2c1416f0 0x7fec2c13cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:47.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:47.659+0000 7fec2affd640 1 --2- 172.21.15.57:0/2425673004 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fec2c0704e0 0x7fec2c13c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:47.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:47.659+0000 7fec2b7fe640 1 -- 172.21.15.57:0/2425673004 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fec2c0704e0 msgr2=0x7fec2c13c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:47.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:47.659+0000 7fec2b7fe640 1 --2- 172.21.15.57:0/2425673004 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fec2c0704e0 0x7fec2c13c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:47.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:47.659+0000 7fec2b7fe640 1 -- 172.21.15.57:0/2425673004 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fec2c1416f0 msgr2=0x7fec2c13cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:47.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:47.659+0000 7fec2b7fe640 1 --2- 172.21.15.57:0/2425673004 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fec2c1416f0 0x7fec2c13cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:47.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:47.660+0000 7fec2b7fe640 1 -- 172.21.15.57:0/2425673004 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec24009900 con 0x7fec2c070080 2023-10-19T14:20:47.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:47.660+0000 7fec2b7fe640 1 --2- 172.21.15.57:0/2425673004 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec2c070080 0x7fec2c136f70 secure :-1 s=READY pgs=2490 cs=0 l=1 rev1=1 crypto rx=0x7fec1c00c400 tx=0x7fec1c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:47.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:47.660+0000 7fec23fff640 1 -- 172.21.15.57:0/2425673004 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec1c0052d0 con 0x7fec2c070080 2023-10-19T14:20:47.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:47.660+0000 7fec23fff640 1 -- 172.21.15.57:0/2425673004 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fec1c0035f0 con 0x7fec2c070080 2023-10-19T14:20:47.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:47.660+0000 7fec31fa9640 1 -- 172.21.15.57:0/2425673004 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec2c0b1cd0 con 0x7fec2c070080 2023-10-19T14:20:47.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:47.660+0000 7fec23fff640 1 -- 172.21.15.57:0/2425673004 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec1c003a00 con 0x7fec2c070080 2023-10-19T14:20:47.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:47.660+0000 7fec31fa9640 1 -- 172.21.15.57:0/2425673004 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec2c15a500 con 0x7fec2c070080 2023-10-19T14:20:47.668 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:47.661+0000 7fec31fa9640 1 -- 172.21.15.57:0/2425673004 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec2c0675f0 con 0x7fec2c070080 2023-10-19T14:20:47.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:47.661+0000 7fec23fff640 1 -- 172.21.15.57:0/2425673004 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 133) v1 ==== 89490+0+0 (secure 0 0 0) 0x7fec1c003b60 con 0x7fec2c070080 2023-10-19T14:20:47.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:47.662+0000 7fec23fff640 1 --2- 172.21.15.57:0/2425673004 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7fec0c06b110 0x7fec0c06d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:47.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:47.662+0000 7fec23fff640 1 -- 172.21.15.57:0/2425673004 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(228..228 src has 1..228) v4 ==== 7265+0+0 (secure 0 0 0) 0x7fec1c089750 con 0x7fec2c070080 2023-10-19T14:20:47.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:47.662+0000 7fec2affd640 1 -- 172.21.15.57:0/2425673004 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7fec0c06b110 msgr2=0x7fec0c06d5d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/4242302683 2023-10-19T14:20:47.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:47.662+0000 7fec2affd640 1 --2- 172.21.15.57:0/2425673004 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7fec0c06b110 0x7fec0c06d5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T14:20:47.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:47.666+0000 7fec23fff640 1 -- 172.21.15.57:0/2425673004 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fec1c053fd0 con 0x7fec2c070080 2023-10-19T14:20:47.724 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:47.721+0000 7f4d48c11040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-19T14:20:47.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:47.863+0000 7fec2affd640 1 -- 172.21.15.57:0/2425673004 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7fec0c06b110 msgr2=0x7fec0c06d5d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/4242302683 2023-10-19T14:20:47.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:47.863+0000 7fec2affd640 1 --2- 172.21.15.57:0/2425673004 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7fec0c06b110 0x7fec0c06d5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-19T14:20:47.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:47.881+0000 7fec31fa9640 1 -- 172.21.15.57:0/2425673004 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fec2c0b2150 con 0x7fec2c070080 2023-10-19T14:20:47.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:47.881+0000 7fec23fff640 1 -- 172.21.15.57:0/2425673004 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+6152 (secure 0 0 0) 0x7fec1c0575f0 con 0x7fec2c070080 2023-10-19T14:20:47.885 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:20:47.885 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":94,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9248,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/1249807745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":1249807745},{"type":"v1","addr":"172.21.15.179:6836","nonce":1249807745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87},{"gid":10513,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/1406825818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":1406825818},{"type":"v1","addr":"172.21.15.179:6837","nonce":1406825818}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":86}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T14:20:19.077396+0000","modified":"2023-10-19T14:20:21.072483+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11718},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11718":{"gid":11718,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.57:6835/1529368280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1529368280},{"type":"v1","addr":"172.21.15.57:6835","nonce":1529368280}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13},{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T14:20:30.145135+0000","modified":"2023-10-19T14:20:32.141937+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11715},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11715":{"gid":11715,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":5,"addr":"172.21.15.57:6834/2467807803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2467807803},{"type":"v1","addr":"172.21.15.57:6834","nonce":2467807803}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2023-10-19T14:20:47.886 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 94 2023-10-19T14:20:47.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:47.883+0000 7fec31fa9640 1 -- 172.21.15.57:0/2425673004 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7fec0c06b110 msgr2=0x7fec0c06d5d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T14:20:47.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:47.883+0000 7fec31fa9640 1 --2- 172.21.15.57:0/2425673004 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7fec0c06b110 0x7fec0c06d5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:47.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:47.883+0000 7fec31fa9640 1 -- 172.21.15.57:0/2425673004 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec2c070080 msgr2=0x7fec2c136f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:47.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:47.883+0000 7fec31fa9640 1 --2- 172.21.15.57:0/2425673004 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec2c070080 0x7fec2c136f70 secure :-1 s=READY pgs=2490 cs=0 l=1 rev1=1 crypto rx=0x7fec1c00c400 tx=0x7fec1c00c8c0 comp rx=0 tx=0).stop 2023-10-19T14:20:47.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:47.883+0000 7fec31fa9640 1 -- 172.21.15.57:0/2425673004 shutdown_connections 2023-10-19T14:20:47.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:47.883+0000 7fec31fa9640 1 --2- 172.21.15.57:0/2425673004 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7fec0c06b110 0x7fec0c06d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:47.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:47.883+0000 7fec31fa9640 1 --2- 172.21.15.57:0/2425673004 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fec2c1416f0 0x7fec2c13cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:47.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:47.883+0000 7fec31fa9640 1 --2- 172.21.15.57:0/2425673004 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fec2c0704e0 0x7fec2c13c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:47.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:47.883+0000 7fec31fa9640 1 --2- 172.21.15.57:0/2425673004 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec2c070080 0x7fec2c136f70 unknown :-1 s=CLOSED pgs=2490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:47.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:47.883+0000 7fec31fa9640 1 -- 172.21.15.57:0/2425673004 >> 172.21.15.57:0/2425673004 conn(0x7fec2c0af730 msgr2=0x7fec2c0b13e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:47.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:47.884+0000 7fec31fa9640 1 -- 172.21.15.57:0/2425673004 shutdown_connections 2023-10-19T14:20:47.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:47.884+0000 7fec31fa9640 1 -- 172.21.15.57:0/2425673004 wait complete. 2023-10-19T14:20:47.898 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T14:20:47.932 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:20:47.928+0000 7f10e7e3e040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-19T14:20:47.987 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:47.985+0000 7f4d48c11040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-19T14:20:48.008 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.005+0000 7f36e736c640 1 -- 172.21.15.57:0/1061919840 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f36e00ceb40 msgr2=0x7f36e00b0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:48.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.005+0000 7f36e736c640 1 --2- 172.21.15.57:0/1061919840 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f36e00ceb40 0x7f36e00b0650 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f36d8005fd0 tx=0x7f36d802c050 comp rx=0 tx=0).stop 2023-10-19T14:20:48.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.005+0000 7f36e736c640 1 -- 172.21.15.57:0/1061919840 shutdown_connections 2023-10-19T14:20:48.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.005+0000 7f36e736c640 1 --2- 172.21.15.57:0/1061919840 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f36e00ceb40 0x7f36e00b0650 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:48.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.005+0000 7f36e736c640 1 --2- 172.21.15.57:0/1061919840 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f36e00cf340 0x7f36e00d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:48.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.005+0000 7f36e736c640 1 -- 172.21.15.57:0/1061919840 >> 172.21.15.57:0/1061919840 conn(0x7f36e00d0d60 msgr2=0x7f36e00d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:48.010 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.005+0000 7f36e736c640 1 -- 172.21.15.57:0/1061919840 shutdown_connections 2023-10-19T14:20:48.010 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.005+0000 7f36e736c640 1 -- 172.21.15.57:0/1061919840 wait complete. 2023-10-19T14:20:48.012 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.006+0000 7f36e736c640 1 Processor -- start 2023-10-19T14:20:48.012 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.006+0000 7f36e736c640 1 -- start start 2023-10-19T14:20:48.012 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.007+0000 7f36e736c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f36e00ceb40 0x7f36e00c8160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:48.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.007+0000 7f36e736c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f36e00cf340 0x7f36e01e6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:48.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.007+0000 7f36e736c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f36e015b2a0 0x7f36e00b4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:48.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.007+0000 7f36e736c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36e01e66c0 con 0x7f36e015b2a0 2023-10-19T14:20:48.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.007+0000 7f36e736c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36e00c9350 con 0x7f36e00ceb40 2023-10-19T14:20:48.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.007+0000 7f36e736c640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36e00c8740 con 0x7f36e00cf340 2023-10-19T14:20:48.014 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.007+0000 7f36e5909640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f36e015b2a0 0x7f36e00b4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:48.014 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.007+0000 7f36e5909640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f36e015b2a0 0x7f36e00b4fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55894/0 (socket says 172.21.15.57:55894) 2023-10-19T14:20:48.014 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.007+0000 7f36e5909640 1 -- 172.21.15.57:0/954423940 learned_addr learned my addr 172.21.15.57:0/954423940 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:48.014 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.007+0000 7f36e5108640 1 --2- 172.21.15.57:0/954423940 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f36e00ceb40 0x7f36e00c8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:48.014 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.007+0000 7f36e4907640 1 --2- 172.21.15.57:0/954423940 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f36e00cf340 0x7f36e01e6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:48.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.008+0000 7f36e5909640 1 -- 172.21.15.57:0/954423940 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f36e00cf340 msgr2=0x7f36e01e6180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:48.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.008+0000 7f36e5909640 1 --2- 172.21.15.57:0/954423940 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f36e00cf340 0x7f36e01e6180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:48.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.008+0000 7f36e5909640 1 -- 172.21.15.57:0/954423940 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f36e00ceb40 msgr2=0x7f36e00c8160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:48.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.008+0000 7f36e5909640 1 --2- 172.21.15.57:0/954423940 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f36e00ceb40 0x7f36e00c8160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:48.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.008+0000 7f36e5909640 1 -- 172.21.15.57:0/954423940 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36d0009490 con 0x7f36e015b2a0 2023-10-19T14:20:48.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.008+0000 7f36e5909640 1 --2- 172.21.15.57:0/954423940 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f36e015b2a0 0x7f36e00b4fe0 secure :-1 s=READY pgs=2492 cs=0 l=1 rev1=1 crypto rx=0x7f36d8009b30 tx=0x7f36d80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:48.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.008+0000 7f36d5ffb640 1 -- 172.21.15.57:0/954423940 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36d8003e60 con 0x7f36e015b2a0 2023-10-19T14:20:48.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.008+0000 7f36d5ffb640 1 -- 172.21.15.57:0/954423940 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f36d803e460 con 0x7f36e015b2a0 2023-10-19T14:20:48.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.008+0000 7f36e736c640 1 -- 172.21.15.57:0/954423940 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36d8009840 con 0x7f36e015b2a0 2023-10-19T14:20:48.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.008+0000 7f36d5ffb640 1 -- 172.21.15.57:0/954423940 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36d803e810 con 0x7f36e015b2a0 2023-10-19T14:20:48.016 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.008+0000 7f36e736c640 1 -- 172.21.15.57:0/954423940 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36e00c8d00 con 0x7f36e015b2a0 2023-10-19T14:20:48.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.009+0000 7f36e736c640 1 -- 172.21.15.57:0/954423940 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36e00c9500 con 0x7f36e015b2a0 2023-10-19T14:20:48.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.010+0000 7f36d5ffb640 1 -- 172.21.15.57:0/954423940 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 133) v1 ==== 89490+0+0 (secure 0 0 0) 0x7f36d803e970 con 0x7f36e015b2a0 2023-10-19T14:20:48.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.010+0000 7f36d5ffb640 1 --2- 172.21.15.57:0/954423940 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7f36c406b110 0x7f36c406d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:48.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.010+0000 7f36d5ffb640 1 -- 172.21.15.57:0/954423940 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(228..228 src has 1..228) v4 ==== 7265+0+0 (secure 0 0 0) 0x7f36d80ace50 con 0x7f36e015b2a0 2023-10-19T14:20:48.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.010+0000 7f36e5108640 1 -- 172.21.15.57:0/954423940 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7f36c406b110 msgr2=0x7f36c406d5d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/4242302683 2023-10-19T14:20:48.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.010+0000 7f36e5108640 1 --2- 172.21.15.57:0/954423940 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7f36c406b110 0x7f36c406d5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T14:20:48.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.014+0000 7f36d5ffb640 1 -- 172.21.15.57:0/954423940 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f36e00c9500 con 0x7f36e015b2a0 2023-10-19T14:20:48.037 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:48.035+0000 7f4d48c11040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-19T14:20:48.136 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:48.134+0000 7f4d48c11040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-19T14:20:48.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.157+0000 7f36d5ffb640 1 -- 172.21.15.57:0/954423940 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mgrmap(e 134) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f36d80779d0 con 0x7f36e015b2a0 2023-10-19T14:20:48.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.158+0000 7f36d5ffb640 1 -- 172.21.15.57:0/954423940 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7f36c406b110 msgr2=0x7f36c406d5d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T14:20:48.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.158+0000 7f36d5ffb640 1 --2- 172.21.15.57:0/954423940 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7f36c406b110 0x7f36c406d5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:48.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.196+0000 7f36e736c640 1 -- 172.21.15.57:0/954423940 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f36e00cb9b0 con 0x7f36e015b2a0 2023-10-19T14:20:48.201 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.198+0000 7f36d5ffb640 1 -- 172.21.15.57:0/954423940 <== mon.0 v2:172.21.15.57:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v229) v1 ==== 74+0+21701 (secure 0 0 0) 0x7f36d80a5e70 con 0x7f36e015b2a0 2023-10-19T14:20:48.201 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:20:48.201 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":229,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T14:20:48.148672+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2023-10-19T14:20:17.357433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"222","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2023-10-19T14:20:18.049516+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"223","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_metadata","create_time":"2023-10-19T14:20:28.667664+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"226","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"backup_fs_data","create_time":"2023-10-19T14:20:29.122584+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"227","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.179:0/1513076424":"2023-10-20T14:20:48.148610+0000","172.21.15.179:0/1464950687":"2023-10-20T14:20:48.148610+0000","172.21.15.179:0/2900872065":"2023-10-20T14:20:48.148610+0000","172.21.15.179:0/2140451434":"2023-10-20T14:20:48.148610+0000","172.21.15.179:0/2605624080":"2023-10-20T14:20:48.148610+0000","172.21.15.179:0/1194984394":"2023-10-20T14:20:39.702634+0000","172.21.15.179:6833/3232482678":"2023-10-20T14:20:39.702634+0000","172.21.15.179:6832/4242302683":"2023-10-20T14:20:48.148610+0000","172.21.15.179:6832/3232482678":"2023-10-20T14:20:39.702634+0000","172.21.15.179:0/2872579179":"2023-10-20T14:20:39.702634+0000","172.21.15.179:0/373323540":"2023-10-20T14:20:48.148610+0000","172.21.15.179:0/3910604357":"2023-10-20T14:20:39.702634+0000","172.21.15.179:0/4185855767":"2023-10-20T14:20:48.148610+0000","172.21.15.179:0/536160557":"2023-10-20T14:20:39.702634+0000","172.21.15.179:0/3239062050":"2023-10-20T14:20:39.702634+0000","172.21.15.179:6833/4242302683":"2023-10-20T14:20:48.148610+0000","172.21.15.179:0/3672528294":"2023-10-20T14:20:48.148610+0000","172.21.15.179:0/2003826253":"2023-10-20T14:20:48.148610+0000","172.21.15.179:0/1948632334":"2023-10-20T14:20:39.702634+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T14:20:48.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.200+0000 7f36e736c640 1 -- 172.21.15.57:0/954423940 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f36e015b2a0 msgr2=0x7f36e00b4fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:48.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.200+0000 7f36e736c640 1 --2- 172.21.15.57:0/954423940 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f36e015b2a0 0x7f36e00b4fe0 secure :-1 s=READY pgs=2492 cs=0 l=1 rev1=1 crypto rx=0x7f36d8009b30 tx=0x7f36d80042f0 comp rx=0 tx=0).stop 2023-10-19T14:20:48.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.200+0000 7f36e736c640 1 -- 172.21.15.57:0/954423940 shutdown_connections 2023-10-19T14:20:48.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.200+0000 7f36e736c640 1 --2- 172.21.15.57:0/954423940 >> [v2:172.21.15.179:6832/4242302683,v1:172.21.15.179:6833/4242302683] conn(0x7f36c406b110 0x7f36c406d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:48.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.200+0000 7f36e736c640 1 --2- 172.21.15.57:0/954423940 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f36e015b2a0 0x7f36e00b4fe0 unknown :-1 s=CLOSED pgs=2492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:48.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.200+0000 7f36e736c640 1 --2- 172.21.15.57:0/954423940 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f36e00cf340 0x7f36e01e6180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:48.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.200+0000 7f36e736c640 1 --2- 172.21.15.57:0/954423940 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f36e00ceb40 0x7f36e00c8160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:48.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.200+0000 7f36e736c640 1 -- 172.21.15.57:0/954423940 >> 172.21.15.57:0/954423940 conn(0x7f36e00d0d60 msgr2=0x7f36e00ae660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:48.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.200+0000 7f36e736c640 1 -- 172.21.15.57:0/954423940 shutdown_connections 2023-10-19T14:20:48.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.200+0000 7f36e736c640 1 -- 172.21.15.57:0/954423940 wait complete. 2023-10-19T14:20:48.214 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-19T12:44:07.371186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 27, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-19T14:20:17.357433+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '222', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 28, 'pool_name': 'cephfs_data', 'create_time': '2023-10-19T14:20:18.049516+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '223', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 29, 'pool_name': 'backup_fs_metadata', 'create_time': '2023-10-19T14:20:28.667664+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '226', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 30, 'pool_name': 'backup_fs_data', 'create_time': '2023-10-19T14:20:29.122584+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '227', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-19T14:20:48.215 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-19T14:20:48.216 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:20:48.213+0000 7f10e7e3e040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-19T14:20:48.218 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:48.215+0000 7f4ccaaba640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:20:48.218 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:48.215+0000 7f4ccaaba640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:20:48.218 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:48.215+0000 7f4ccaaba640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:20:48.218 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:48.215+0000 7f4ccaaba640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:20:48.219 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:48.215+0000 7f4ccaaba640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:20:48.219 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:48.215+0000 7f4ccdac0640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:20:48.219 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:48.215+0000 7f4ccdac0640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:20:48.219 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:48.215+0000 7f4ccdac0640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:20:48.219 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:48.215+0000 7f4ccdac0640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:20:48.219 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:48.215+0000 7f4ccdac0640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:20:48.226 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:48.224+0000 7f4ccdac0640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:20:48.227 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:48.224+0000 7f4ccdac0640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:20:48.227 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:48.224+0000 7f4ccdac0640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:20:48.227 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:48.224+0000 7f4ccdac0640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:20:48.227 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:48.224+0000 7f4ccdac0640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:20:48.228 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:48.224+0000 7f4ccaaba640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:20:48.228 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:48.224+0000 7f4ccaaba640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:20:48.228 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:48.224+0000 7f4ccaaba640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:20:48.228 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:48.224+0000 7f4ccaaba640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:20:48.229 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:20:48.224+0000 7f4ccaaba640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:20:48.270 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:20:48.268+0000 7f10e7e3e040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-19T14:20:48.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.327+0000 7faa36f00640 1 -- 172.21.15.57:0/1716372042 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa300704e0 msgr2=0x7faa300cc910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:48.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.327+0000 7faa36f00640 1 --2- 172.21.15.57:0/1716372042 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa300704e0 0x7faa300cc910 secure :-1 s=READY pgs=2495 cs=0 l=1 rev1=1 crypto rx=0x7faa28005fd0 tx=0x7faa2802c370 comp rx=0 tx=0).stop 2023-10-19T14:20:48.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.327+0000 7faa36f00640 1 -- 172.21.15.57:0/1716372042 shutdown_connections 2023-10-19T14:20:48.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.327+0000 7faa36f00640 1 --2- 172.21.15.57:0/1716372042 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa300704e0 0x7faa300cc910 unknown :-1 s=CLOSED pgs=2495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:48.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.327+0000 7faa36f00640 1 -- 172.21.15.57:0/1716372042 >> 172.21.15.57:0/1716372042 conn(0x7faa300af730 msgr2=0x7faa300afb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:48.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.327+0000 7faa36f00640 1 -- 172.21.15.57:0/1716372042 shutdown_connections 2023-10-19T14:20:48.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.327+0000 7faa36f00640 1 -- 172.21.15.57:0/1716372042 wait complete. 2023-10-19T14:20:48.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.328+0000 7faa36f00640 1 Processor -- start 2023-10-19T14:20:48.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.328+0000 7faa36f00640 1 -- start start 2023-10-19T14:20:48.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.329+0000 7faa36f00640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa30070080 0x7faa30137810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:48.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.329+0000 7faa36f00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faa300704e0 0x7faa3013c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:48.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.329+0000 7faa36f00640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7faa301416f0 0x7faa3013cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:48.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.329+0000 7faa36f00640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa301e18a0 con 0x7faa30070080 2023-10-19T14:20:48.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.329+0000 7faa36f00640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa301f55e0 con 0x7faa300704e0 2023-10-19T14:20:48.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.329+0000 7faa36f00640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa30137d50 con 0x7faa301416f0 2023-10-19T14:20:48.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.329+0000 7faa34c9c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa30070080 0x7faa30137810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:48.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.329+0000 7faa34c9c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa30070080 0x7faa30137810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55918/0 (socket says 172.21.15.57:55918) 2023-10-19T14:20:48.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.329+0000 7faa34c9c640 1 -- 172.21.15.57:0/528074577 learned_addr learned my addr 172.21.15.57:0/528074577 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:48.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.329+0000 7faa2ffff640 1 --2- 172.21.15.57:0/528074577 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faa300704e0 0x7faa3013c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:48.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.329+0000 7faa3549d640 1 --2- 172.21.15.57:0/528074577 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7faa301416f0 0x7faa3013cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:48.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.329+0000 7faa34c9c640 1 -- 172.21.15.57:0/528074577 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7faa301416f0 msgr2=0x7faa3013cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:48.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.329+0000 7faa34c9c640 1 --2- 172.21.15.57:0/528074577 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7faa301416f0 0x7faa3013cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:48.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.329+0000 7faa34c9c640 1 -- 172.21.15.57:0/528074577 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faa300704e0 msgr2=0x7faa3013c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:48.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.329+0000 7faa34c9c640 1 --2- 172.21.15.57:0/528074577 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faa300704e0 0x7faa3013c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:48.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.329+0000 7faa34c9c640 1 -- 172.21.15.57:0/528074577 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa28009900 con 0x7faa30070080 2023-10-19T14:20:48.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.329+0000 7faa34c9c640 1 --2- 172.21.15.57:0/528074577 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa30070080 0x7faa30137810 secure :-1 s=READY pgs=2496 cs=0 l=1 rev1=1 crypto rx=0x7faa2400c430 tx=0x7faa2400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:48.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.330+0000 7faa2d7fa640 1 -- 172.21.15.57:0/528074577 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa240052d0 con 0x7faa30070080 2023-10-19T14:20:48.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.330+0000 7faa2d7fa640 1 -- 172.21.15.57:0/528074577 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7faa240035f0 con 0x7faa30070080 2023-10-19T14:20:48.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.330+0000 7faa36f00640 1 -- 172.21.15.57:0/528074577 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa300b1cd0 con 0x7faa30070080 2023-10-19T14:20:48.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.330+0000 7faa2d7fa640 1 -- 172.21.15.57:0/528074577 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa24003a00 con 0x7faa30070080 2023-10-19T14:20:48.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.330+0000 7faa36f00640 1 -- 172.21.15.57:0/528074577 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa3006d180 con 0x7faa30070080 2023-10-19T14:20:48.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.331+0000 7faa36f00640 1 -- 172.21.15.57:0/528074577 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa300675f0 con 0x7faa30070080 2023-10-19T14:20:48.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.331+0000 7faa2d7fa640 1 -- 172.21.15.57:0/528074577 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 134) v1 ==== 88940+0+0 (secure 0 0 0) 0x7faa24003b60 con 0x7faa30070080 2023-10-19T14:20:48.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.332+0000 7faa2d7fa640 1 -- 172.21.15.57:0/528074577 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 7738+0+0 (secure 0 0 0) 0x7faa24089110 con 0x7faa30070080 2023-10-19T14:20:48.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.336+0000 7faa2d7fa640 1 -- 172.21.15.57:0/528074577 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7faa24053da0 con 0x7faa30070080 2023-10-19T14:20:48.378 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:20:48.376+0000 7f10e7e3e040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-19T14:20:48.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.515+0000 7faa36f00640 1 -- 172.21.15.57:0/528074577 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7faa300b2150 con 0x7faa30070080 2023-10-19T14:20:48.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.515+0000 7faa2d7fa640 1 -- 172.21.15.57:0/528074577 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v229) v1 ==== 93+0+10 (secure 0 0 0) 0x7faa240573c0 con 0x7faa30070080 2023-10-19T14:20:48.519 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-19T14:20:48.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.517+0000 7faa36f00640 1 -- 172.21.15.57:0/528074577 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa30070080 msgr2=0x7faa30137810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:48.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.517+0000 7faa36f00640 1 --2- 172.21.15.57:0/528074577 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa30070080 0x7faa30137810 secure :-1 s=READY pgs=2496 cs=0 l=1 rev1=1 crypto rx=0x7faa2400c430 tx=0x7faa2400c8f0 comp rx=0 tx=0).stop 2023-10-19T14:20:48.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.517+0000 7faa36f00640 1 -- 172.21.15.57:0/528074577 shutdown_connections 2023-10-19T14:20:48.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.517+0000 7faa36f00640 1 --2- 172.21.15.57:0/528074577 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7faa301416f0 0x7faa3013cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:48.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.517+0000 7faa36f00640 1 --2- 172.21.15.57:0/528074577 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faa300704e0 0x7faa3013c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:48.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.517+0000 7faa36f00640 1 --2- 172.21.15.57:0/528074577 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa30070080 0x7faa30137810 unknown :-1 s=CLOSED pgs=2496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:48.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.517+0000 7faa36f00640 1 -- 172.21.15.57:0/528074577 >> 172.21.15.57:0/528074577 conn(0x7faa300af730 msgr2=0x7faa30155600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:48.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.517+0000 7faa36f00640 1 -- 172.21.15.57:0/528074577 shutdown_connections 2023-10-19T14:20:48.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.517+0000 7faa36f00640 1 -- 172.21.15.57:0/528074577 wait complete. 2023-10-19T14:20:48.532 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-19T14:20:48.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.640+0000 7f0188ce1640 1 -- 172.21.15.57:0/3661728274 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f018415ccb0 msgr2=0x7f01840c9750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:48.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.640+0000 7f0188ce1640 1 --2- 172.21.15.57:0/3661728274 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f018415ccb0 0x7f01840c9750 secure :-1 s=READY pgs=2497 cs=0 l=1 rev1=1 crypto rx=0x7f0184071350 tx=0x7f017402c1d0 comp rx=0 tx=0).stop 2023-10-19T14:20:48.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.640+0000 7f0188ce1640 1 -- 172.21.15.57:0/3661728274 shutdown_connections 2023-10-19T14:20:48.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.640+0000 7f0188ce1640 1 --2- 172.21.15.57:0/3661728274 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f018415ccb0 0x7f01840c9750 unknown :-1 s=CLOSED pgs=2497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:48.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.640+0000 7f0188ce1640 1 --2- 172.21.15.57:0/3661728274 >> v2:172.21.15.179:3300/0 conn(0x7f0184159bd0 0x7f018415c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:48.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.640+0000 7f0188ce1640 1 --2- 172.21.15.57:0/3661728274 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f018406b030 0x7f018406b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:48.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.640+0000 7f0188ce1640 1 -- 172.21.15.57:0/3661728274 >> 172.21.15.57:0/3661728274 conn(0x7f0184069930 msgr2=0x7f0184069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:48.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.640+0000 7f0188ce1640 1 -- 172.21.15.57:0/3661728274 shutdown_connections 2023-10-19T14:20:48.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.640+0000 7f0188ce1640 1 -- 172.21.15.57:0/3661728274 wait complete. 2023-10-19T14:20:48.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.642+0000 7f0188ce1640 1 Processor -- start 2023-10-19T14:20:48.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.642+0000 7f0188ce1640 1 -- start start 2023-10-19T14:20:48.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.642+0000 7f0188ce1640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f018406b030 0x7f01840a96c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:48.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.642+0000 7f0188ce1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0184159bd0 0x7f01840a7440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:48.648 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.642+0000 7f0188ce1640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f018415ccb0 0x7f01840a7980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:48.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.642+0000 7f0188ce1640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01841e57a0 con 0x7f0184159bd0 2023-10-19T14:20:48.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.642+0000 7f0188ce1640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01840a8070 con 0x7f018415ccb0 2023-10-19T14:20:48.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.642+0000 7f0188ce1640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01840a8350 con 0x7f018406b030 2023-10-19T14:20:48.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.642+0000 7f0181d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0184159bd0 0x7f01840a7440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:48.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.643+0000 7f0181d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0184159bd0 0x7f01840a7440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55930/0 (socket says 172.21.15.57:55930) 2023-10-19T14:20:48.650 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.643+0000 7f0181d9b640 1 -- 172.21.15.57:0/4294144694 learned_addr learned my addr 172.21.15.57:0/4294144694 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:48.650 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.643+0000 7f018259c640 1 --2- 172.21.15.57:0/4294144694 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f018406b030 0x7f01840a96c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:48.650 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.643+0000 7f0182d9d640 1 --2- 172.21.15.57:0/4294144694 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f018415ccb0 0x7f01840a7980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:48.650 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.643+0000 7f0181d9b640 1 -- 172.21.15.57:0/4294144694 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f018406b030 msgr2=0x7f01840a96c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:48.650 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.643+0000 7f0181d9b640 1 --2- 172.21.15.57:0/4294144694 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f018406b030 0x7f01840a96c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:48.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.643+0000 7f0181d9b640 1 -- 172.21.15.57:0/4294144694 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f018415ccb0 msgr2=0x7f01840a7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:48.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.643+0000 7f0181d9b640 1 --2- 172.21.15.57:0/4294144694 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f018415ccb0 0x7f01840a7980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:48.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.643+0000 7f0181d9b640 1 -- 172.21.15.57:0/4294144694 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0174009900 con 0x7f0184159bd0 2023-10-19T14:20:48.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.643+0000 7f0181d9b640 1 --2- 172.21.15.57:0/4294144694 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0184159bd0 0x7f01840a7440 secure :-1 s=READY pgs=2498 cs=0 l=1 rev1=1 crypto rx=0x7f016c00e9f0 tx=0x7f016c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:48.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.643+0000 7f017affd640 1 -- 172.21.15.57:0/4294144694 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f016c009700 con 0x7f0184159bd0 2023-10-19T14:20:48.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.643+0000 7f017affd640 1 -- 172.21.15.57:0/4294144694 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f016c00ce70 con 0x7f0184159bd0 2023-10-19T14:20:48.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.643+0000 7f0188ce1640 1 -- 172.21.15.57:0/4294144694 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01840a8630 con 0x7f0184159bd0 2023-10-19T14:20:48.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.643+0000 7f017affd640 1 -- 172.21.15.57:0/4294144694 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f016c010670 con 0x7f0184159bd0 2023-10-19T14:20:48.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.643+0000 7f0188ce1640 1 -- 172.21.15.57:0/4294144694 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01840a8bf0 con 0x7f0184159bd0 2023-10-19T14:20:48.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.644+0000 7f0188ce1640 1 -- 172.21.15.57:0/4294144694 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f018415a2d0 con 0x7f0184159bd0 2023-10-19T14:20:48.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.645+0000 7f017affd640 1 -- 172.21.15.57:0/4294144694 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 134) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f016c0107d0 con 0x7f0184159bd0 2023-10-19T14:20:48.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.645+0000 7f017affd640 1 -- 172.21.15.57:0/4294144694 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 7738+0+0 (secure 0 0 0) 0x7f016c014070 con 0x7f0184159bd0 2023-10-19T14:20:48.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.649+0000 7f017affd640 1 -- 172.21.15.57:0/4294144694 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f016c051ca0 con 0x7f0184159bd0 2023-10-19T14:20:48.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.824+0000 7f0188ce1640 1 -- 172.21.15.57:0/4294144694 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f018415b920 con 0x7f0184159bd0 2023-10-19T14:20:48.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.825+0000 7f017affd640 1 -- 172.21.15.57:0/4294144694 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v229) v1 ==== 104+0+11 (secure 0 0 0) 0x7f016c0551b0 con 0x7f0184159bd0 2023-10-19T14:20:48.828 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2023-10-19T14:20:48.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.827+0000 7f0188ce1640 1 -- 172.21.15.57:0/4294144694 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0184159bd0 msgr2=0x7f01840a7440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:48.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.827+0000 7f0188ce1640 1 --2- 172.21.15.57:0/4294144694 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0184159bd0 0x7f01840a7440 secure :-1 s=READY pgs=2498 cs=0 l=1 rev1=1 crypto rx=0x7f016c00e9f0 tx=0x7f016c00eeb0 comp rx=0 tx=0).stop 2023-10-19T14:20:48.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.827+0000 7f0188ce1640 1 -- 172.21.15.57:0/4294144694 shutdown_connections 2023-10-19T14:20:48.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.827+0000 7f0188ce1640 1 --2- 172.21.15.57:0/4294144694 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f018415ccb0 0x7f01840a7980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:48.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.827+0000 7f0188ce1640 1 --2- 172.21.15.57:0/4294144694 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0184159bd0 0x7f01840a7440 unknown :-1 s=CLOSED pgs=2498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:48.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.827+0000 7f0188ce1640 1 --2- 172.21.15.57:0/4294144694 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f018406b030 0x7f01840a96c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:48.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.827+0000 7f0188ce1640 1 -- 172.21.15.57:0/4294144694 >> 172.21.15.57:0/4294144694 conn(0x7f0184069930 msgr2=0x7f01840ab620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:48.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.827+0000 7f0188ce1640 1 -- 172.21.15.57:0/4294144694 shutdown_connections 2023-10-19T14:20:48.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.827+0000 7f0188ce1640 1 -- 172.21.15.57:0/4294144694 wait complete. 2023-10-19T14:20:48.841 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-19T14:20:48.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.950+0000 7f8edfade640 1 -- 172.21.15.57:0/1176368129 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8ed80c9d00 msgr2=0x7f8ed806b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:48.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.950+0000 7f8edfade640 1 --2- 172.21.15.57:0/1176368129 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8ed80c9d00 0x7f8ed806b910 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f8ed0005fd0 tx=0x7f8ed002c0e0 comp rx=0 tx=0).stop 2023-10-19T14:20:48.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.950+0000 7f8edfade640 1 -- 172.21.15.57:0/1176368129 shutdown_connections 2023-10-19T14:20:48.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.950+0000 7f8edfade640 1 --2- 172.21.15.57:0/1176368129 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8ed80c9d00 0x7f8ed806b910 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:48.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.950+0000 7f8edfade640 1 --2- 172.21.15.57:0/1176368129 >> v2:172.21.15.179:3300/0 conn(0x7f8ed8140660 0x7f8ed80c97c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:48.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.950+0000 7f8edfade640 1 -- 172.21.15.57:0/1176368129 >> 172.21.15.57:0/1176368129 conn(0x7f8ed8069930 msgr2=0x7f8ed8069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:48.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.950+0000 7f8edfade640 1 -- 172.21.15.57:0/1176368129 shutdown_connections 2023-10-19T14:20:48.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.950+0000 7f8edfade640 1 -- 172.21.15.57:0/1176368129 wait complete. 2023-10-19T14:20:48.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.951+0000 7f8edfade640 1 Processor -- start 2023-10-19T14:20:48.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.951+0000 7f8edfade640 1 -- start start 2023-10-19T14:20:48.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.952+0000 7f8edfade640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ed80c9d00 0x7f8ed813b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:48.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.952+0000 7f8edfade640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8ed813d360 0x7f8ed8156900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:48.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.952+0000 7f8edfade640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8ed8140660 0x7f8ed8155810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:48.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.952+0000 7f8edfade640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed8156e40 con 0x7f8ed80c9d00 2023-10-19T14:20:48.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.952+0000 7f8edfade640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed8155380 con 0x7f8ed8140660 2023-10-19T14:20:48.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.952+0000 7f8edfade640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed81399c0 con 0x7f8ed813d360 2023-10-19T14:20:48.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.952+0000 7f8edd87a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ed80c9d00 0x7f8ed813b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:48.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.952+0000 7f8edd87a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ed80c9d00 0x7f8ed813b830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55946/0 (socket says 172.21.15.57:55946) 2023-10-19T14:20:48.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.952+0000 7f8edd87a640 1 -- 172.21.15.57:0/810477708 learned_addr learned my addr 172.21.15.57:0/810477708 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:48.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.952+0000 7f8edd079640 1 --2- 172.21.15.57:0/810477708 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8ed813d360 0x7f8ed8156900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:48.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.952+0000 7f8ede07b640 1 --2- 172.21.15.57:0/810477708 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8ed8140660 0x7f8ed8155810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:48.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.952+0000 7f8edd87a640 1 -- 172.21.15.57:0/810477708 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8ed813d360 msgr2=0x7f8ed8156900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:48.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.952+0000 7f8edd87a640 1 --2- 172.21.15.57:0/810477708 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8ed813d360 0x7f8ed8156900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:48.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.952+0000 7f8edd87a640 1 -- 172.21.15.57:0/810477708 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8ed8140660 msgr2=0x7f8ed8155810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:48.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.952+0000 7f8edd87a640 1 --2- 172.21.15.57:0/810477708 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8ed8140660 0x7f8ed8155810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:48.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.952+0000 7f8edd87a640 1 -- 172.21.15.57:0/810477708 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ec8009560 con 0x7f8ed80c9d00 2023-10-19T14:20:48.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.952+0000 7f8edd87a640 1 --2- 172.21.15.57:0/810477708 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ed80c9d00 0x7f8ed813b830 secure :-1 s=READY pgs=2499 cs=0 l=1 rev1=1 crypto rx=0x7f8ed400c450 tx=0x7f8ed400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:48.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.953+0000 7f8ece7fc640 1 -- 172.21.15.57:0/810477708 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ed40052d0 con 0x7f8ed80c9d00 2023-10-19T14:20:48.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.953+0000 7f8ece7fc640 1 -- 172.21.15.57:0/810477708 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f8ed4005430 con 0x7f8ed80c9d00 2023-10-19T14:20:48.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.953+0000 7f8edfade640 1 -- 172.21.15.57:0/810477708 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ed0009840 con 0x7f8ed80c9d00 2023-10-19T14:20:48.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.953+0000 7f8ece7fc640 1 -- 172.21.15.57:0/810477708 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ed4012890 con 0x7f8ed80c9d00 2023-10-19T14:20:48.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.953+0000 7f8edfade640 1 -- 172.21.15.57:0/810477708 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ed8139d50 con 0x7f8ed80c9d00 2023-10-19T14:20:48.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.954+0000 7f8edfade640 1 -- 172.21.15.57:0/810477708 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ed80690e0 con 0x7f8ed80c9d00 2023-10-19T14:20:48.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.954+0000 7f8ece7fc640 1 -- 172.21.15.57:0/810477708 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 134) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f8ed40035f0 con 0x7f8ed80c9d00 2023-10-19T14:20:48.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.955+0000 7f8ece7fc640 1 -- 172.21.15.57:0/810477708 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 7738+0+0 (secure 0 0 0) 0x7f8ed4089400 con 0x7f8ed80c9d00 2023-10-19T14:20:48.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:48.959+0000 7f8ece7fc640 1 -- 172.21.15.57:0/810477708 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8ed40541c0 con 0x7f8ed80c9d00 2023-10-19T14:20:49.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.134+0000 7f8edfade640 1 -- 172.21.15.57:0/810477708 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f8ed815a5c0 con 0x7f8ed80c9d00 2023-10-19T14:20:49.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.134+0000 7f8ece7fc640 1 -- 172.21.15.57:0/810477708 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v229) v1 ==== 100+0+11 (secure 0 0 0) 0x7f8ed40577e0 con 0x7f8ed80c9d00 2023-10-19T14:20:49.137 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 64 2023-10-19T14:20:49.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.136+0000 7f8edfade640 1 -- 172.21.15.57:0/810477708 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ed80c9d00 msgr2=0x7f8ed813b830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:49.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.136+0000 7f8edfade640 1 --2- 172.21.15.57:0/810477708 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ed80c9d00 0x7f8ed813b830 secure :-1 s=READY pgs=2499 cs=0 l=1 rev1=1 crypto rx=0x7f8ed400c450 tx=0x7f8ed400c910 comp rx=0 tx=0).stop 2023-10-19T14:20:49.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.136+0000 7f8edfade640 1 -- 172.21.15.57:0/810477708 shutdown_connections 2023-10-19T14:20:49.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.136+0000 7f8edfade640 1 --2- 172.21.15.57:0/810477708 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8ed8140660 0x7f8ed8155810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:49.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.136+0000 7f8edfade640 1 --2- 172.21.15.57:0/810477708 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8ed813d360 0x7f8ed8156900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:49.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.136+0000 7f8edfade640 1 --2- 172.21.15.57:0/810477708 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ed80c9d00 0x7f8ed813b830 unknown :-1 s=CLOSED pgs=2499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:49.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.136+0000 7f8edfade640 1 -- 172.21.15.57:0/810477708 >> 172.21.15.57:0/810477708 conn(0x7f8ed8069930 msgr2=0x7f8ed81576f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:49.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.136+0000 7f8edfade640 1 -- 172.21.15.57:0/810477708 shutdown_connections 2023-10-19T14:20:49.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.136+0000 7f8edfade640 1 -- 172.21.15.57:0/810477708 wait complete. 2023-10-19T14:20:49.149 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2023-10-19T14:20:49.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.260+0000 7f3dde2b4640 1 -- 172.21.15.57:0/3361754358 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3dd80ac870 msgr2=0x7f3dd806b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:49.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.261+0000 7f3dde2b4640 1 --2- 172.21.15.57:0/3361754358 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3dd80ac870 0x7f3dd806b980 secure :-1 s=READY pgs=2501 cs=0 l=1 rev1=1 crypto rx=0x7f3dc0009860 tx=0x7f3dc002bd60 comp rx=0 tx=0).stop 2023-10-19T14:20:49.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.261+0000 7f3dde2b4640 1 -- 172.21.15.57:0/3361754358 shutdown_connections 2023-10-19T14:20:49.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.261+0000 7f3dde2b4640 1 --2- 172.21.15.57:0/3361754358 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3dd806bec0 0x7f3dd806a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:49.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.261+0000 7f3dde2b4640 1 --2- 172.21.15.57:0/3361754358 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3dd80ac870 0x7f3dd806b980 unknown :-1 s=CLOSED pgs=2501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:49.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.261+0000 7f3dde2b4640 1 -- 172.21.15.57:0/3361754358 >> 172.21.15.57:0/3361754358 conn(0x7f3dd806fae0 msgr2=0x7f3dd806fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:49.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.261+0000 7f3dde2b4640 1 -- 172.21.15.57:0/3361754358 shutdown_connections 2023-10-19T14:20:49.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.261+0000 7f3dde2b4640 1 -- 172.21.15.57:0/3361754358 wait complete. 2023-10-19T14:20:49.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.262+0000 7f3dde2b4640 1 Processor -- start 2023-10-19T14:20:49.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.262+0000 7f3dde2b4640 1 -- start start 2023-10-19T14:20:49.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.263+0000 7f3dde2b4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3dd806bec0 0x7f3dd8147dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:49.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.263+0000 7f3dde2b4640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3dd80ac870 0x7f3dd80aa8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:49.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.263+0000 7f3dde2b4640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3dd8156790 0x7f3dd81577a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:49.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.263+0000 7f3dde2b4640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dd80ab2d0 con 0x7f3dd806bec0 2023-10-19T14:20:49.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.263+0000 7f3dde2b4640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dd80b17e0 con 0x7f3dd8156790 2023-10-19T14:20:49.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.263+0000 7f3dde2b4640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3dd80b1ac0 con 0x7f3dd80ac870 2023-10-19T14:20:49.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.263+0000 7f3dd7fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3dd806bec0 0x7f3dd8147dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:49.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.263+0000 7f3dd7fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3dd806bec0 0x7f3dd8147dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55956/0 (socket says 172.21.15.57:55956) 2023-10-19T14:20:49.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.263+0000 7f3dd7fff640 1 -- 172.21.15.57:0/3757845587 learned_addr learned my addr 172.21.15.57:0/3757845587 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:49.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.263+0000 7f3dd77fe640 1 --2- 172.21.15.57:0/3757845587 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3dd80ac870 0x7f3dd80aa8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:49.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.263+0000 7f3ddc851640 1 --2- 172.21.15.57:0/3757845587 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3dd8156790 0x7f3dd81577a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:49.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.263+0000 7f3dd7fff640 1 -- 172.21.15.57:0/3757845587 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3dd80ac870 msgr2=0x7f3dd80aa8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:49.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.263+0000 7f3dd7fff640 1 --2- 172.21.15.57:0/3757845587 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3dd80ac870 0x7f3dd80aa8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:49.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.263+0000 7f3dd7fff640 1 -- 172.21.15.57:0/3757845587 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3dd8156790 msgr2=0x7f3dd81577a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:49.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.263+0000 7f3dd7fff640 1 --2- 172.21.15.57:0/3757845587 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3dd8156790 0x7f3dd81577a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:49.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.263+0000 7f3dd7fff640 1 -- 172.21.15.57:0/3757845587 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dc0009490 con 0x7f3dd806bec0 2023-10-19T14:20:49.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.263+0000 7f3dd77fe640 1 --2- 172.21.15.57:0/3757845587 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3dd80ac870 0x7f3dd80aa8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:20:49.272 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.263+0000 7f3dd7fff640 1 --2- 172.21.15.57:0/3757845587 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3dd806bec0 0x7f3dd8147dd0 secure :-1 s=READY pgs=2502 cs=0 l=1 rev1=1 crypto rx=0x7f3dc800c4b0 tx=0x7f3dc800c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:49.272 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.263+0000 7f3dd4ff9640 1 -- 172.21.15.57:0/3757845587 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dc8005480 con 0x7f3dd806bec0 2023-10-19T14:20:49.272 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.263+0000 7f3dd4ff9640 1 -- 172.21.15.57:0/3757845587 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f3dc80036e0 con 0x7f3dd806bec0 2023-10-19T14:20:49.272 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.263+0000 7f3dde2b4640 1 -- 172.21.15.57:0/3757845587 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3dd80b1da0 con 0x7f3dd806bec0 2023-10-19T14:20:49.272 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.263+0000 7f3dd4ff9640 1 -- 172.21.15.57:0/3757845587 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dc8003b30 con 0x7f3dd806bec0 2023-10-19T14:20:49.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.263+0000 7f3dde2b4640 1 -- 172.21.15.57:0/3757845587 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3dd81e3c70 con 0x7f3dd806bec0 2023-10-19T14:20:49.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.264+0000 7f3dde2b4640 1 -- 172.21.15.57:0/3757845587 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3dd80b1f90 con 0x7f3dd806bec0 2023-10-19T14:20:49.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.265+0000 7f3dd4ff9640 1 -- 172.21.15.57:0/3757845587 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 135) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f3dc8003c90 con 0x7f3dd806bec0 2023-10-19T14:20:49.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.265+0000 7f3dd4ff9640 1 --2- 172.21.15.57:0/3757845587 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f3db806acc0 0x7f3db806d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:49.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.265+0000 7f3dd4ff9640 1 -- 172.21.15.57:0/3757845587 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 7738+0+0 (secure 0 0 0) 0x7f3dc8089200 con 0x7f3dd806bec0 2023-10-19T14:20:49.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.266+0000 7f3dd77fe640 1 --2- 172.21.15.57:0/3757845587 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f3db806acc0 0x7f3db806d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:49.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.266+0000 7f3dd77fe640 1 --2- 172.21.15.57:0/3757845587 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f3db806acc0 0x7f3db806d180 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f3dc0036000 tx=0x7f3dc0004730 comp rx=0 tx=0).ready entity=mgr.9530 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:49.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.270+0000 7f3dd4ff9640 1 -- 172.21.15.57:0/3757845587 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3dc8053e00 con 0x7f3dd806bec0 2023-10-19T14:20:49.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.448+0000 7f3dde2b4640 1 -- 172.21.15.57:0/3757845587 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f3dd80690e0 con 0x7f3dd806bec0 2023-10-19T14:20:49.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.448+0000 7f3dd4ff9640 1 -- 172.21.15.57:0/3757845587 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v229) v1 ==== 107+0+11 (secure 0 0 0) 0x7f3dc8017070 con 0x7f3dd806bec0 2023-10-19T14:20:49.451 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2023-10-19T14:20:49.453 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.450+0000 7f3dde2b4640 1 -- 172.21.15.57:0/3757845587 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f3db806acc0 msgr2=0x7f3db806d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:49.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.450+0000 7f3dde2b4640 1 --2- 172.21.15.57:0/3757845587 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f3db806acc0 0x7f3db806d180 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f3dc0036000 tx=0x7f3dc0004730 comp rx=0 tx=0).stop 2023-10-19T14:20:49.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.450+0000 7f3dde2b4640 1 -- 172.21.15.57:0/3757845587 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3dd806bec0 msgr2=0x7f3dd8147dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:49.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.450+0000 7f3dde2b4640 1 --2- 172.21.15.57:0/3757845587 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3dd806bec0 0x7f3dd8147dd0 secure :-1 s=READY pgs=2502 cs=0 l=1 rev1=1 crypto rx=0x7f3dc800c4b0 tx=0x7f3dc800c970 comp rx=0 tx=0).stop 2023-10-19T14:20:49.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.450+0000 7f3dde2b4640 1 -- 172.21.15.57:0/3757845587 shutdown_connections 2023-10-19T14:20:49.454 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.450+0000 7f3dde2b4640 1 --2- 172.21.15.57:0/3757845587 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f3db806acc0 0x7f3db806d180 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:49.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.450+0000 7f3dde2b4640 1 --2- 172.21.15.57:0/3757845587 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3dd8156790 0x7f3dd81577a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:49.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.450+0000 7f3dde2b4640 1 --2- 172.21.15.57:0/3757845587 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3dd80ac870 0x7f3dd80aa8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:49.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.450+0000 7f3dde2b4640 1 --2- 172.21.15.57:0/3757845587 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3dd806bec0 0x7f3dd8147dd0 unknown :-1 s=CLOSED pgs=2502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:49.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.450+0000 7f3dde2b4640 1 -- 172.21.15.57:0/3757845587 >> 172.21.15.57:0/3757845587 conn(0x7f3dd806fae0 msgr2=0x7f3dd8155dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:49.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.450+0000 7f3dde2b4640 1 -- 172.21.15.57:0/3757845587 shutdown_connections 2023-10-19T14:20:49.456 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.450+0000 7f3dde2b4640 1 -- 172.21.15.57:0/3757845587 wait complete. 2023-10-19T14:20:49.464 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2023-10-19T14:20:49.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.574+0000 7f1b75e90640 1 -- 172.21.15.57:0/3472008339 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1b700c96b0 msgr2=0x7f1b700c9b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:49.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.574+0000 7f1b75e90640 1 --2- 172.21.15.57:0/3472008339 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1b700c96b0 0x7f1b700c9b30 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f1b64009860 tx=0x7f1b6402bd60 comp rx=0 tx=0).stop 2023-10-19T14:20:49.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.574+0000 7f1b75e90640 1 -- 172.21.15.57:0/3472008339 shutdown_connections 2023-10-19T14:20:49.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.574+0000 7f1b75e90640 1 --2- 172.21.15.57:0/3472008339 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1b700c96b0 0x7f1b700c9b30 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:49.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.574+0000 7f1b75e90640 1 --2- 172.21.15.57:0/3472008339 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1b7006af00 0x7f1b7006b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:49.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.574+0000 7f1b75e90640 1 -- 172.21.15.57:0/3472008339 >> 172.21.15.57:0/3472008339 conn(0x7f1b70069930 msgr2=0x7f1b70069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:49.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.574+0000 7f1b75e90640 1 -- 172.21.15.57:0/3472008339 shutdown_connections 2023-10-19T14:20:49.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.574+0000 7f1b75e90640 1 -- 172.21.15.57:0/3472008339 wait complete. 2023-10-19T14:20:49.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.575+0000 7f1b75e90640 1 Processor -- start 2023-10-19T14:20:49.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.575+0000 7f1b75e90640 1 -- start start 2023-10-19T14:20:49.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.576+0000 7f1b75e90640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1b7006af00 0x7f1b7015bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:49.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.576+0000 7f1b75e90640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1b700ab500 0x7f1b70139b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:49.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.576+0000 7f1b75e90640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1b700c96b0 0x7f1b7013a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:49.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.576+0000 7f1b75e90640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b701f1770 con 0x7f1b700c96b0 2023-10-19T14:20:49.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.576+0000 7f1b75e90640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b701f1a50 con 0x7f1b700ab500 2023-10-19T14:20:49.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.576+0000 7f1b75e90640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b701f1d30 con 0x7f1b7006af00 2023-10-19T14:20:49.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.576+0000 7f1b6ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1b700c96b0 0x7f1b7013a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:49.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.576+0000 7f1b6ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1b700c96b0 0x7f1b7013a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55964/0 (socket says 172.21.15.57:55964) 2023-10-19T14:20:49.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.576+0000 7f1b6ffff640 1 -- 172.21.15.57:0/2882521369 learned_addr learned my addr 172.21.15.57:0/2882521369 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:49.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.576+0000 7f1b6f7fe640 1 --2- 172.21.15.57:0/2882521369 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1b7006af00 0x7f1b7015bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:49.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.576+0000 7f1b6effd640 1 --2- 172.21.15.57:0/2882521369 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1b700ab500 0x7f1b70139b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:49.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.576+0000 7f1b6ffff640 1 -- 172.21.15.57:0/2882521369 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1b7006af00 msgr2=0x7f1b7015bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:49.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.576+0000 7f1b6ffff640 1 --2- 172.21.15.57:0/2882521369 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1b7006af00 0x7f1b7015bdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:49.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.576+0000 7f1b6ffff640 1 -- 172.21.15.57:0/2882521369 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1b700ab500 msgr2=0x7f1b70139b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:49.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.576+0000 7f1b6ffff640 1 --2- 172.21.15.57:0/2882521369 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1b700ab500 0x7f1b70139b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:49.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.576+0000 7f1b6ffff640 1 -- 172.21.15.57:0/2882521369 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b64009490 con 0x7f1b700c96b0 2023-10-19T14:20:49.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.577+0000 7f1b6ffff640 1 --2- 172.21.15.57:0/2882521369 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1b700c96b0 0x7f1b7013a0b0 secure :-1 s=READY pgs=2503 cs=0 l=1 rev1=1 crypto rx=0x7f1b6800cb40 tx=0x7f1b6800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:49.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.577+0000 7f1b57fff640 1 -- 172.21.15.57:0/2882521369 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b6800aca0 con 0x7f1b700c96b0 2023-10-19T14:20:49.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.577+0000 7f1b57fff640 1 -- 172.21.15.57:0/2882521369 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f1b68012420 con 0x7f1b700c96b0 2023-10-19T14:20:49.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.577+0000 7f1b75e90640 1 -- 172.21.15.57:0/2882521369 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b701f2010 con 0x7f1b700c96b0 2023-10-19T14:20:49.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.577+0000 7f1b57fff640 1 -- 172.21.15.57:0/2882521369 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b68012850 con 0x7f1b700c96b0 2023-10-19T14:20:49.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.577+0000 7f1b75e90640 1 -- 172.21.15.57:0/2882521369 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b701f22f0 con 0x7f1b700c96b0 2023-10-19T14:20:49.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.578+0000 7f1b75e90640 1 -- 172.21.15.57:0/2882521369 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b700690e0 con 0x7f1b700c96b0 2023-10-19T14:20:49.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.578+0000 7f1b57fff640 1 -- 172.21.15.57:0/2882521369 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 135) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f1b6801a3b0 con 0x7f1b700c96b0 2023-10-19T14:20:49.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.579+0000 7f1b57fff640 1 --2- 172.21.15.57:0/2882521369 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f1b5c06ac90 0x7f1b5c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:49.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.579+0000 7f1b57fff640 1 -- 172.21.15.57:0/2882521369 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 7738+0+0 (secure 0 0 0) 0x7f1b6808a120 con 0x7f1b700c96b0 2023-10-19T14:20:49.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.579+0000 7f1b6f7fe640 1 --2- 172.21.15.57:0/2882521369 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f1b5c06ac90 0x7f1b5c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:49.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.580+0000 7f1b6f7fe640 1 --2- 172.21.15.57:0/2882521369 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f1b5c06ac90 0x7f1b5c06d150 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f1b700ab4d0 tx=0x7f1b60005f90 comp rx=0 tx=0).ready entity=mgr.9530 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:49.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.583+0000 7f1b57fff640 1 -- 172.21.15.57:0/2882521369 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f1b68054d50 con 0x7f1b700c96b0 2023-10-19T14:20:49.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.757+0000 7f1b75e90640 1 -- 172.21.15.57:0/2882521369 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) v1 -- 0x7f1b701f4a50 con 0x7f1b700c96b0 2023-10-19T14:20:49.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.759+0000 7f1b57fff640 1 -- 172.21.15.57:0/2882521369 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v229) v1 ==== 103+0+11 (secure 0 0 0) 0x7f1b68058370 con 0x7f1b700c96b0 2023-10-19T14:20:49.762 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 64 2023-10-19T14:20:49.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.761+0000 7f1b75e90640 1 -- 172.21.15.57:0/2882521369 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f1b5c06ac90 msgr2=0x7f1b5c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:49.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.761+0000 7f1b75e90640 1 --2- 172.21.15.57:0/2882521369 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f1b5c06ac90 0x7f1b5c06d150 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f1b700ab4d0 tx=0x7f1b60005f90 comp rx=0 tx=0).stop 2023-10-19T14:20:49.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.761+0000 7f1b75e90640 1 -- 172.21.15.57:0/2882521369 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1b700c96b0 msgr2=0x7f1b7013a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:49.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.761+0000 7f1b75e90640 1 --2- 172.21.15.57:0/2882521369 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1b700c96b0 0x7f1b7013a0b0 secure :-1 s=READY pgs=2503 cs=0 l=1 rev1=1 crypto rx=0x7f1b6800cb40 tx=0x7f1b6800a060 comp rx=0 tx=0).stop 2023-10-19T14:20:49.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.761+0000 7f1b75e90640 1 -- 172.21.15.57:0/2882521369 shutdown_connections 2023-10-19T14:20:49.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.761+0000 7f1b75e90640 1 --2- 172.21.15.57:0/2882521369 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f1b5c06ac90 0x7f1b5c06d150 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:49.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.761+0000 7f1b75e90640 1 --2- 172.21.15.57:0/2882521369 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1b700c96b0 0x7f1b7013a0b0 unknown :-1 s=CLOSED pgs=2503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:49.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.761+0000 7f1b75e90640 1 --2- 172.21.15.57:0/2882521369 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1b700ab500 0x7f1b70139b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:49.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.761+0000 7f1b75e90640 1 --2- 172.21.15.57:0/2882521369 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1b7006af00 0x7f1b7015bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:49.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.761+0000 7f1b75e90640 1 -- 172.21.15.57:0/2882521369 >> 172.21.15.57:0/2882521369 conn(0x7f1b70069930 msgr2=0x7f1b700adbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:49.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.761+0000 7f1b75e90640 1 -- 172.21.15.57:0/2882521369 shutdown_connections 2023-10-19T14:20:49.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.761+0000 7f1b75e90640 1 -- 172.21.15.57:0/2882521369 wait complete. 2023-10-19T14:20:49.775 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T14:20:49.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.881+0000 7fa56cbd7640 1 -- 172.21.15.57:0/1569404759 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa554004d00 msgr2=0x7fa554005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:49.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.881+0000 7fa56cbd7640 1 --2- 172.21.15.57:0/1569404759 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa554004d00 0x7fa554005180 secure :-1 s=READY pgs=2505 cs=0 l=1 rev1=1 crypto rx=0x7fa558009f00 tx=0x7fa5540057f0 comp rx=0 tx=0).stop 2023-10-19T14:20:49.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.881+0000 7fa566d9d640 1 -- 172.21.15.57:0/1569404759 reap_dead start 2023-10-19T14:20:49.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.882+0000 7fa56cbd7640 1 -- 172.21.15.57:0/1569404759 shutdown_connections 2023-10-19T14:20:49.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.882+0000 7fa56cbd7640 1 -- 172.21.15.57:0/1569404759 >> 172.21.15.57:0/1569404759 conn(0x7fa568069930 msgr2=0x7fa568069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:49.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.882+0000 7fa56cbd7640 1 -- 172.21.15.57:0/1569404759 shutdown_connections 2023-10-19T14:20:49.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.882+0000 7fa56cbd7640 1 -- 172.21.15.57:0/1569404759 wait complete. 2023-10-19T14:20:49.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.883+0000 7fa56cbd7640 1 Processor -- start 2023-10-19T14:20:49.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.883+0000 7fa56cbd7640 1 -- start start 2023-10-19T14:20:49.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.884+0000 7fa56cbd7640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa554004d00 0x7fa56813adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:49.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.884+0000 7fa56cbd7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa5681f32d0 0x7fa5681db4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:49.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.884+0000 7fa56cbd7640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa5681dba20 0x7fa56813b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:49.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.884+0000 7fa56cbd7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5681d9b20 con 0x7fa5681f32d0 2023-10-19T14:20:49.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.884+0000 7fa56cbd7640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5681eefa0 con 0x7fa5681dba20 2023-10-19T14:20:49.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.884+0000 7fa56cbd7640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5681d8ea0 con 0x7fa554004d00 2023-10-19T14:20:49.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.884+0000 7fa565d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa5681f32d0 0x7fa5681db4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:49.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.884+0000 7fa565d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa5681f32d0 0x7fa5681db4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55988/0 (socket says 172.21.15.57:55988) 2023-10-19T14:20:49.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.884+0000 7fa565d9b640 1 -- 172.21.15.57:0/2292230122 learned_addr learned my addr 172.21.15.57:0/2292230122 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:49.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.884+0000 7fa56659c640 1 --2- 172.21.15.57:0/2292230122 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa554004d00 0x7fa56813adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:49.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.884+0000 7fa566d9d640 1 --2- 172.21.15.57:0/2292230122 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa5681dba20 0x7fa56813b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:49.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.884+0000 7fa565d9b640 1 -- 172.21.15.57:0/2292230122 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa554004d00 msgr2=0x7fa56813adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:49.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.884+0000 7fa565d9b640 1 --2- 172.21.15.57:0/2292230122 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa554004d00 0x7fa56813adf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:49.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.884+0000 7fa565d9b640 1 -- 172.21.15.57:0/2292230122 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa5681dba20 msgr2=0x7fa56813b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:49.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.884+0000 7fa565d9b640 1 --2- 172.21.15.57:0/2292230122 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa5681dba20 0x7fa56813b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:49.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.884+0000 7fa565d9b640 1 -- 172.21.15.57:0/2292230122 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5681d9180 con 0x7fa5681f32d0 2023-10-19T14:20:49.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.884+0000 7fa565d9b640 1 --2- 172.21.15.57:0/2292230122 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa5681f32d0 0x7fa5681db4e0 secure :-1 s=READY pgs=2506 cs=0 l=1 rev1=1 crypto rx=0x7fa550011a80 tx=0x7fa550011f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:49.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.884+0000 7fa55effd640 1 -- 172.21.15.57:0/2292230122 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa55000fd50 con 0x7fa5681f32d0 2023-10-19T14:20:49.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.884+0000 7fa55effd640 1 -- 172.21.15.57:0/2292230122 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fa550012450 con 0x7fa5681f32d0 2023-10-19T14:20:49.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.884+0000 7fa55effd640 1 -- 172.21.15.57:0/2292230122 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa550012840 con 0x7fa5681f32d0 2023-10-19T14:20:49.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.884+0000 7fa56cbd7640 1 -- 172.21.15.57:0/2292230122 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5681d9460 con 0x7fa5681f32d0 2023-10-19T14:20:49.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.885+0000 7fa56cbd7640 1 -- 172.21.15.57:0/2292230122 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5681d9f40 con 0x7fa5681f32d0 2023-10-19T14:20:49.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.885+0000 7fa56cbd7640 1 -- 172.21.15.57:0/2292230122 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5680690e0 con 0x7fa5681f32d0 2023-10-19T14:20:49.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.886+0000 7fa55effd640 1 -- 172.21.15.57:0/2292230122 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 135) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fa5500129a0 con 0x7fa5681f32d0 2023-10-19T14:20:49.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.886+0000 7fa55effd640 1 --2- 172.21.15.57:0/2292230122 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fa54806ac90 0x7fa54806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:49.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.886+0000 7fa55effd640 1 -- 172.21.15.57:0/2292230122 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 7738+0+0 (secure 0 0 0) 0x7fa55008a8e0 con 0x7fa5681f32d0 2023-10-19T14:20:49.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.887+0000 7fa56659c640 1 --2- 172.21.15.57:0/2292230122 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fa54806ac90 0x7fa54806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:49.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.887+0000 7fa56659c640 1 --2- 172.21.15.57:0/2292230122 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fa54806ac90 0x7fa54806d150 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa5600017a0 tx=0x7fa560004dd0 comp rx=0 tx=0).ready entity=mgr.9530 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:49.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:49.890+0000 7fa55effd640 1 -- 172.21.15.57:0/2292230122 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa5500554e0 con 0x7fa5681f32d0 2023-10-19T14:20:50.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.103+0000 7fa56cbd7640 1 -- 172.21.15.57:0/2292230122 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa56806d300 con 0x7fa5681f32d0 2023-10-19T14:20:50.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.104+0000 7fa55effd640 1 -- 172.21.15.57:0/2292230122 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+6152 (secure 0 0 0) 0x7fa550058b00 con 0x7fa5681f32d0 2023-10-19T14:20:50.107 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:20:50.108 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":94,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9248,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/1249807745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":1249807745},{"type":"v1","addr":"172.21.15.179:6836","nonce":1249807745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87},{"gid":10513,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/1406825818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":1406825818},{"type":"v1","addr":"172.21.15.179:6837","nonce":1406825818}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":86}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T14:20:19.077396+0000","modified":"2023-10-19T14:20:21.072483+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11718},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11718":{"gid":11718,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.57:6835/1529368280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1529368280},{"type":"v1","addr":"172.21.15.57:6835","nonce":1529368280}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13},{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T14:20:30.145135+0000","modified":"2023-10-19T14:20:32.141937+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11715},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11715":{"gid":11715,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":5,"addr":"172.21.15.57:6834/2467807803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2467807803},{"type":"v1","addr":"172.21.15.57:6834","nonce":2467807803}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2023-10-19T14:20:50.108 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 94 2023-10-19T14:20:50.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.106+0000 7fa56cbd7640 1 -- 172.21.15.57:0/2292230122 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fa54806ac90 msgr2=0x7fa54806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:50.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.106+0000 7fa56cbd7640 1 --2- 172.21.15.57:0/2292230122 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fa54806ac90 0x7fa54806d150 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa5600017a0 tx=0x7fa560004dd0 comp rx=0 tx=0).stop 2023-10-19T14:20:50.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.106+0000 7fa56cbd7640 1 -- 172.21.15.57:0/2292230122 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa5681f32d0 msgr2=0x7fa5681db4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:50.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.106+0000 7fa56cbd7640 1 --2- 172.21.15.57:0/2292230122 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa5681f32d0 0x7fa5681db4e0 secure :-1 s=READY pgs=2506 cs=0 l=1 rev1=1 crypto rx=0x7fa550011a80 tx=0x7fa550011f40 comp rx=0 tx=0).stop 2023-10-19T14:20:50.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.106+0000 7fa56cbd7640 1 -- 172.21.15.57:0/2292230122 shutdown_connections 2023-10-19T14:20:50.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.106+0000 7fa56cbd7640 1 --2- 172.21.15.57:0/2292230122 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fa54806ac90 0x7fa54806d150 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:50.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.106+0000 7fa56cbd7640 1 --2- 172.21.15.57:0/2292230122 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa5681dba20 0x7fa56813b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:50.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.106+0000 7fa56cbd7640 1 --2- 172.21.15.57:0/2292230122 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa5681f32d0 0x7fa5681db4e0 unknown :-1 s=CLOSED pgs=2506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:50.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.106+0000 7fa56cbd7640 1 --2- 172.21.15.57:0/2292230122 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa554004d00 0x7fa56813adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:50.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.106+0000 7fa56cbd7640 1 -- 172.21.15.57:0/2292230122 >> 172.21.15.57:0/2292230122 conn(0x7fa568069930 msgr2=0x7fa5681df900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:50.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.106+0000 7fa56cbd7640 1 -- 172.21.15.57:0/2292230122 shutdown_connections 2023-10-19T14:20:50.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.106+0000 7fa56cbd7640 1 -- 172.21.15.57:0/2292230122 wait complete. 2023-10-19T14:20:50.119 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T14:20:50.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.227+0000 7ff1ab39e640 1 -- 172.21.15.57:0/1976436103 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff1a4142850 msgr2=0x7ff1a4159440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:50.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.227+0000 7ff1ab39e640 1 --2- 172.21.15.57:0/1976436103 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff1a4142850 0x7ff1a4159440 secure :-1 s=READY pgs=2508 cs=0 l=1 rev1=1 crypto rx=0x7ff1a40700c0 tx=0x7ff1a002bcf0 comp rx=0 tx=0).stop 2023-10-19T14:20:50.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.227+0000 7ff1ab39e640 1 -- 172.21.15.57:0/1976436103 shutdown_connections 2023-10-19T14:20:50.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.227+0000 7ff1ab39e640 1 --2- 172.21.15.57:0/1976436103 >> v2:172.21.15.179:3300/0 conn(0x7ff1a4069e30 0x7ff1a40cdb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:50.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.227+0000 7ff1ab39e640 1 --2- 172.21.15.57:0/1976436103 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff1a4142850 0x7ff1a4159440 unknown :-1 s=CLOSED pgs=2508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:50.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.227+0000 7ff1ab39e640 1 -- 172.21.15.57:0/1976436103 >> 172.21.15.57:0/1976436103 conn(0x7ff1a40af7a0 msgr2=0x7ff1a40afb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:50.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.227+0000 7ff1ab39e640 1 -- 172.21.15.57:0/1976436103 shutdown_connections 2023-10-19T14:20:50.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.227+0000 7ff1ab39e640 1 -- 172.21.15.57:0/1976436103 wait complete. 2023-10-19T14:20:50.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.229+0000 7ff1ab39e640 1 Processor -- start 2023-10-19T14:20:50.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.229+0000 7ff1ab39e640 1 -- start start 2023-10-19T14:20:50.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.229+0000 7ff1ab39e640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff1a4069e30 0x7ff1a4158740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:50.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.229+0000 7ff1ab39e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff1a40cd6d0 0x7ff1a40aaad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:50.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.229+0000 7ff1ab39e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff1a4142850 0x7ff1a40ab010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:50.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.229+0000 7ff1ab39e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1a40ac7d0 con 0x7ff1a40cd6d0 2023-10-19T14:20:50.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.229+0000 7ff1ab39e640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1a40ad420 con 0x7ff1a4142850 2023-10-19T14:20:50.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.229+0000 7ff1ab39e640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1a40abb30 con 0x7ff1a4069e30 2023-10-19T14:20:50.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.229+0000 7ff1a8939640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff1a40cd6d0 0x7ff1a40aaad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:50.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.229+0000 7ff1a8939640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff1a40cd6d0 0x7ff1a40aaad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56006/0 (socket says 172.21.15.57:56006) 2023-10-19T14:20:50.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.229+0000 7ff1a8939640 1 -- 172.21.15.57:0/4019185193 learned_addr learned my addr 172.21.15.57:0/4019185193 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:50.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.229+0000 7ff1a913a640 1 --2- 172.21.15.57:0/4019185193 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff1a4069e30 0x7ff1a4158740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:50.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.230+0000 7ff1a993b640 1 --2- 172.21.15.57:0/4019185193 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff1a4142850 0x7ff1a40ab010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:50.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.230+0000 7ff1a8939640 1 -- 172.21.15.57:0/4019185193 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff1a4069e30 msgr2=0x7ff1a4158740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:50.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.230+0000 7ff1a8939640 1 --2- 172.21.15.57:0/4019185193 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff1a4069e30 0x7ff1a4158740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:50.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.230+0000 7ff1a8939640 1 -- 172.21.15.57:0/4019185193 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff1a4142850 msgr2=0x7ff1a40ab010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:50.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.230+0000 7ff1a8939640 1 --2- 172.21.15.57:0/4019185193 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff1a4142850 0x7ff1a40ab010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:50.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.230+0000 7ff1a8939640 1 -- 172.21.15.57:0/4019185193 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1a0009490 con 0x7ff1a40cd6d0 2023-10-19T14:20:50.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.230+0000 7ff1a8939640 1 --2- 172.21.15.57:0/4019185193 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff1a40cd6d0 0x7ff1a40aaad0 secure :-1 s=READY pgs=2509 cs=0 l=1 rev1=1 crypto rx=0x7ff19400c500 tx=0x7ff19400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:50.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.230+0000 7ff199ffb640 1 -- 172.21.15.57:0/4019185193 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff194012830 con 0x7ff1a40cd6d0 2023-10-19T14:20:50.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.230+0000 7ff199ffb640 1 -- 172.21.15.57:0/4019185193 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7ff194012a80 con 0x7ff1a40cd6d0 2023-10-19T14:20:50.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.230+0000 7ff1ab39e640 1 -- 172.21.15.57:0/4019185193 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1a40ab850 con 0x7ff1a40cd6d0 2023-10-19T14:20:50.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.230+0000 7ff199ffb640 1 -- 172.21.15.57:0/4019185193 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff194012e90 con 0x7ff1a40cd6d0 2023-10-19T14:20:50.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.230+0000 7ff1ab39e640 1 -- 172.21.15.57:0/4019185193 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1a413bbc0 con 0x7ff1a40cd6d0 2023-10-19T14:20:50.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.231+0000 7ff1ab39e640 1 -- 172.21.15.57:0/4019185193 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1a413b740 con 0x7ff1a40cd6d0 2023-10-19T14:20:50.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.232+0000 7ff199ffb640 1 -- 172.21.15.57:0/4019185193 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 135) v1 ==== 89067+0+0 (secure 0 0 0) 0x7ff19401b400 con 0x7ff1a40cd6d0 2023-10-19T14:20:50.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.232+0000 7ff199ffb640 1 --2- 172.21.15.57:0/4019185193 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7ff18806ac90 0x7ff18806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:50.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.232+0000 7ff199ffb640 1 -- 172.21.15.57:0/4019185193 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 7738+0+0 (secure 0 0 0) 0x7ff194089b10 con 0x7ff1a40cd6d0 2023-10-19T14:20:50.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.233+0000 7ff1a913a640 1 --2- 172.21.15.57:0/4019185193 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7ff18806ac90 0x7ff18806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:50.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.233+0000 7ff1a913a640 1 --2- 172.21.15.57:0/4019185193 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7ff18806ac90 0x7ff18806d150 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7ff1a0036000 tx=0x7ff1a0004730 comp rx=0 tx=0).ready entity=mgr.9530 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:50.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.236+0000 7ff199ffb640 1 -- 172.21.15.57:0/4019185193 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff1a413b740 con 0x7ff1a40cd6d0 2023-10-19T14:20:50.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.414+0000 7ff1ab39e640 1 -- 172.21.15.57:0/4019185193 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff1a413b740 con 0x7ff1a40cd6d0 2023-10-19T14:20:50.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.415+0000 7ff199ffb640 1 -- 172.21.15.57:0/4019185193 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v229) v1 ==== 74+0+21701 (secure 0 0 0) 0x7ff1a413b740 con 0x7ff1a40cd6d0 2023-10-19T14:20:50.421 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:20:50.421 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":229,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T14:20:48.148672+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2023-10-19T14:20:17.357433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"222","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2023-10-19T14:20:18.049516+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"223","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_metadata","create_time":"2023-10-19T14:20:28.667664+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"226","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"backup_fs_data","create_time":"2023-10-19T14:20:29.122584+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"227","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.179:0/1513076424":"2023-10-20T14:20:48.148610+0000","172.21.15.179:0/1464950687":"2023-10-20T14:20:48.148610+0000","172.21.15.179:0/2900872065":"2023-10-20T14:20:48.148610+0000","172.21.15.179:0/2140451434":"2023-10-20T14:20:48.148610+0000","172.21.15.179:0/2605624080":"2023-10-20T14:20:48.148610+0000","172.21.15.179:0/1194984394":"2023-10-20T14:20:39.702634+0000","172.21.15.179:6833/3232482678":"2023-10-20T14:20:39.702634+0000","172.21.15.179:6832/4242302683":"2023-10-20T14:20:48.148610+0000","172.21.15.179:6832/3232482678":"2023-10-20T14:20:39.702634+0000","172.21.15.179:0/2872579179":"2023-10-20T14:20:39.702634+0000","172.21.15.179:0/373323540":"2023-10-20T14:20:48.148610+0000","172.21.15.179:0/3910604357":"2023-10-20T14:20:39.702634+0000","172.21.15.179:0/4185855767":"2023-10-20T14:20:48.148610+0000","172.21.15.179:0/536160557":"2023-10-20T14:20:39.702634+0000","172.21.15.179:0/3239062050":"2023-10-20T14:20:39.702634+0000","172.21.15.179:6833/4242302683":"2023-10-20T14:20:48.148610+0000","172.21.15.179:0/3672528294":"2023-10-20T14:20:48.148610+0000","172.21.15.179:0/2003826253":"2023-10-20T14:20:48.148610+0000","172.21.15.179:0/1948632334":"2023-10-20T14:20:39.702634+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T14:20:50.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.418+0000 7ff1ab39e640 1 -- 172.21.15.57:0/4019185193 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7ff18806ac90 msgr2=0x7ff18806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:50.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.418+0000 7ff1ab39e640 1 --2- 172.21.15.57:0/4019185193 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7ff18806ac90 0x7ff18806d150 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7ff1a0036000 tx=0x7ff1a0004730 comp rx=0 tx=0).stop 2023-10-19T14:20:50.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.418+0000 7ff1ab39e640 1 -- 172.21.15.57:0/4019185193 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff1a40cd6d0 msgr2=0x7ff1a40aaad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:50.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.418+0000 7ff1ab39e640 1 --2- 172.21.15.57:0/4019185193 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff1a40cd6d0 0x7ff1a40aaad0 secure :-1 s=READY pgs=2509 cs=0 l=1 rev1=1 crypto rx=0x7ff19400c500 tx=0x7ff19400c9c0 comp rx=0 tx=0).stop 2023-10-19T14:20:50.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.418+0000 7ff1ab39e640 1 -- 172.21.15.57:0/4019185193 shutdown_connections 2023-10-19T14:20:50.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.418+0000 7ff1ab39e640 1 --2- 172.21.15.57:0/4019185193 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7ff18806ac90 0x7ff18806d150 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:50.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.418+0000 7ff1ab39e640 1 --2- 172.21.15.57:0/4019185193 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff1a4142850 0x7ff1a40ab010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:50.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.418+0000 7ff1ab39e640 1 --2- 172.21.15.57:0/4019185193 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff1a40cd6d0 0x7ff1a40aaad0 unknown :-1 s=CLOSED pgs=2509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:50.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.418+0000 7ff1ab39e640 1 --2- 172.21.15.57:0/4019185193 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff1a4069e30 0x7ff1a4158740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:50.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.418+0000 7ff1ab39e640 1 -- 172.21.15.57:0/4019185193 >> 172.21.15.57:0/4019185193 conn(0x7ff1a40af7a0 msgr2=0x7ff1a40ae7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:50.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.418+0000 7ff1ab39e640 1 -- 172.21.15.57:0/4019185193 shutdown_connections 2023-10-19T14:20:50.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.418+0000 7ff1ab39e640 1 -- 172.21.15.57:0/4019185193 wait complete. 2023-10-19T14:20:50.431 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-19T14:20:50.432 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T14:20:50.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.538+0000 7f2a0b94c640 1 -- 172.21.15.57:0/993669956 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2a04156790 msgr2=0x7f2a040cc990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:50.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.538+0000 7f2a0b94c640 1 --2- 172.21.15.57:0/993669956 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2a04156790 0x7f2a040cc990 secure :-1 s=READY pgs=2510 cs=0 l=1 rev1=1 crypto rx=0x7f2a04061510 tx=0x7f29fc02bda0 comp rx=0 tx=0).stop 2023-10-19T14:20:50.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.538+0000 7f2a0b94c640 1 -- 172.21.15.57:0/993669956 shutdown_connections 2023-10-19T14:20:50.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.538+0000 7f2a0b94c640 1 --2- 172.21.15.57:0/993669956 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2a0406bec0 0x7f2a0406a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:50.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.538+0000 7f2a0b94c640 1 --2- 172.21.15.57:0/993669956 >> v2:172.21.15.179:3300/0 conn(0x7f2a040ac870 0x7f2a0406b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:50.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.538+0000 7f2a0b94c640 1 --2- 172.21.15.57:0/993669956 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2a04156790 0x7f2a040cc990 unknown :-1 s=CLOSED pgs=2510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:50.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.538+0000 7f2a0b94c640 1 -- 172.21.15.57:0/993669956 >> 172.21.15.57:0/993669956 conn(0x7f2a0406fae0 msgr2=0x7f2a0406fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:50.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.538+0000 7f2a0b94c640 1 -- 172.21.15.57:0/993669956 shutdown_connections 2023-10-19T14:20:50.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.538+0000 7f2a0b94c640 1 -- 172.21.15.57:0/993669956 wait complete. 2023-10-19T14:20:50.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.539+0000 7f2a0b94c640 1 Processor -- start 2023-10-19T14:20:50.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.539+0000 7f2a0b94c640 1 -- start start 2023-10-19T14:20:50.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.540+0000 7f2a0b94c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2a0406bec0 0x7f2a0414f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:50.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.540+0000 7f2a0b94c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2a040ac870 0x7f2a04150890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:50.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.540+0000 7f2a0b94c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2a04156790 0x7f2a040b0e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:50.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.540+0000 7f2a0b94c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a04151cf0 con 0x7f2a040ac870 2023-10-19T14:20:50.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.540+0000 7f2a0b94c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a0414f8e0 con 0x7f2a04156790 2023-10-19T14:20:50.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.540+0000 7f2a0b94c640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a0414fbc0 con 0x7f2a0406bec0 2023-10-19T14:20:50.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.540+0000 7f2a08ee7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2a040ac870 0x7f2a04150890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:50.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.540+0000 7f2a08ee7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2a040ac870 0x7f2a04150890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56030/0 (socket says 172.21.15.57:56030) 2023-10-19T14:20:50.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.540+0000 7f2a08ee7640 1 -- 172.21.15.57:0/3180161040 learned_addr learned my addr 172.21.15.57:0/3180161040 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:50.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.540+0000 7f2a09ee9640 1 --2- 172.21.15.57:0/3180161040 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2a04156790 0x7f2a040b0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:50.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.540+0000 7f2a096e8640 1 --2- 172.21.15.57:0/3180161040 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2a0406bec0 0x7f2a0414f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:50.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.540+0000 7f2a08ee7640 1 -- 172.21.15.57:0/3180161040 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2a0406bec0 msgr2=0x7f2a0414f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:50.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.541+0000 7f2a08ee7640 1 --2- 172.21.15.57:0/3180161040 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2a0406bec0 0x7f2a0414f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:50.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.541+0000 7f2a08ee7640 1 -- 172.21.15.57:0/3180161040 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2a04156790 msgr2=0x7f2a040b0e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:50.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.541+0000 7f2a08ee7640 1 --2- 172.21.15.57:0/3180161040 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2a04156790 0x7f2a040b0e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:50.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.541+0000 7f2a08ee7640 1 -- 172.21.15.57:0/3180161040 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29fc009560 con 0x7f2a040ac870 2023-10-19T14:20:50.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.541+0000 7f2a096e8640 1 --2- 172.21.15.57:0/3180161040 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2a0406bec0 0x7f2a0414f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:20:50.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.541+0000 7f2a08ee7640 1 --2- 172.21.15.57:0/3180161040 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2a040ac870 0x7f2a04150890 secure :-1 s=READY pgs=2511 cs=0 l=1 rev1=1 crypto rx=0x7f29f800d950 tx=0x7f29f800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:50.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.541+0000 7f2a01ffb640 1 -- 172.21.15.57:0/3180161040 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29f8009c30 con 0x7f2a040ac870 2023-10-19T14:20:50.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.541+0000 7f2a01ffb640 1 -- 172.21.15.57:0/3180161040 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f29f8009e80 con 0x7f2a040ac870 2023-10-19T14:20:50.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.541+0000 7f2a0b94c640 1 -- 172.21.15.57:0/3180161040 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a040b0620 con 0x7f2a040ac870 2023-10-19T14:20:50.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.541+0000 7f2a01ffb640 1 -- 172.21.15.57:0/3180161040 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29f80106c0 con 0x7f2a040ac870 2023-10-19T14:20:50.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.541+0000 7f2a0b94c640 1 -- 172.21.15.57:0/3180161040 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a0414f5f0 con 0x7f2a040ac870 2023-10-19T14:20:50.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.542+0000 7f2a0b94c640 1 -- 172.21.15.57:0/3180161040 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a040690e0 con 0x7f2a040ac870 2023-10-19T14:20:50.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.543+0000 7f2a01ffb640 1 -- 172.21.15.57:0/3180161040 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 135) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f29f8010820 con 0x7f2a040ac870 2023-10-19T14:20:50.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.543+0000 7f2a01ffb640 1 --2- 172.21.15.57:0/3180161040 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f29f006ac90 0x7f29f006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:50.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.543+0000 7f2a01ffb640 1 -- 172.21.15.57:0/3180161040 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 7738+0+0 (secure 0 0 0) 0x7f29f8089f10 con 0x7f2a040ac870 2023-10-19T14:20:50.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.543+0000 7f2a096e8640 1 --2- 172.21.15.57:0/3180161040 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f29f006ac90 0x7f29f006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:50.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.544+0000 7f2a096e8640 1 --2- 172.21.15.57:0/3180161040 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f29f006ac90 0x7f29f006d150 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f29fc035000 tx=0x7f29fc004730 comp rx=0 tx=0).ready entity=mgr.9530 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:50.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.547+0000 7f2a01ffb640 1 -- 172.21.15.57:0/3180161040 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f29f8054b10 con 0x7f2a040ac870 2023-10-19T14:20:50.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.761+0000 7f2a0b94c640 1 -- 172.21.15.57:0/3180161040 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2a0414ee90 con 0x7f2a040ac870 2023-10-19T14:20:50.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.762+0000 7f2a01ffb640 1 -- 172.21.15.57:0/3180161040 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+6152 (secure 0 0 0) 0x7f29f8017090 con 0x7f2a040ac870 2023-10-19T14:20:50.766 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:20:50.766 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":94,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9248,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/1249807745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":1249807745},{"type":"v1","addr":"172.21.15.179:6836","nonce":1249807745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87},{"gid":10513,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/1406825818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":1406825818},{"type":"v1","addr":"172.21.15.179:6837","nonce":1406825818}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":86}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T14:20:19.077396+0000","modified":"2023-10-19T14:20:21.072483+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11718},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11718":{"gid":11718,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.57:6835/1529368280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1529368280},{"type":"v1","addr":"172.21.15.57:6835","nonce":1529368280}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13},{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T14:20:30.145135+0000","modified":"2023-10-19T14:20:32.141937+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11715},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11715":{"gid":11715,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":5,"addr":"172.21.15.57:6834/2467807803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2467807803},{"type":"v1","addr":"172.21.15.57:6834","nonce":2467807803}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2023-10-19T14:20:50.767 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 94 2023-10-19T14:20:50.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.764+0000 7f2a0b94c640 1 -- 172.21.15.57:0/3180161040 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f29f006ac90 msgr2=0x7f29f006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:50.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.764+0000 7f2a0b94c640 1 --2- 172.21.15.57:0/3180161040 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f29f006ac90 0x7f29f006d150 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f29fc035000 tx=0x7f29fc004730 comp rx=0 tx=0).stop 2023-10-19T14:20:50.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.764+0000 7f2a0b94c640 1 -- 172.21.15.57:0/3180161040 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2a040ac870 msgr2=0x7f2a04150890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:50.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.764+0000 7f2a0b94c640 1 --2- 172.21.15.57:0/3180161040 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2a040ac870 0x7f2a04150890 secure :-1 s=READY pgs=2511 cs=0 l=1 rev1=1 crypto rx=0x7f29f800d950 tx=0x7f29f800de10 comp rx=0 tx=0).stop 2023-10-19T14:20:50.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.764+0000 7f2a0b94c640 1 -- 172.21.15.57:0/3180161040 shutdown_connections 2023-10-19T14:20:50.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.764+0000 7f2a0b94c640 1 --2- 172.21.15.57:0/3180161040 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f29f006ac90 0x7f29f006d150 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:50.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.764+0000 7f2a0b94c640 1 --2- 172.21.15.57:0/3180161040 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2a04156790 0x7f2a040b0e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:50.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.764+0000 7f2a0b94c640 1 --2- 172.21.15.57:0/3180161040 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2a040ac870 0x7f2a04150890 unknown :-1 s=CLOSED pgs=2511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:50.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.764+0000 7f2a0b94c640 1 --2- 172.21.15.57:0/3180161040 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2a0406bec0 0x7f2a0414f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:50.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.764+0000 7f2a0b94c640 1 -- 172.21.15.57:0/3180161040 >> 172.21.15.57:0/3180161040 conn(0x7f2a0406fae0 msgr2=0x7f2a04155c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:50.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.764+0000 7f2a0b94c640 1 -- 172.21.15.57:0/3180161040 shutdown_connections 2023-10-19T14:20:50.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.764+0000 7f2a0b94c640 1 -- 172.21.15.57:0/3180161040 wait complete. 2023-10-19T14:20:50.778 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T14:20:50.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.885+0000 7f2c88541640 1 -- 172.21.15.57:0/1328365167 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2c6c004d00 msgr2=0x7f2c6c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:50.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.885+0000 7f2c88541640 1 --2- 172.21.15.57:0/1328365167 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2c6c004d00 0x7f2c6c005180 secure :-1 s=READY pgs=2513 cs=0 l=1 rev1=1 crypto rx=0x7f2c78009ed0 tx=0x7f2c6c0057f0 comp rx=0 tx=0).stop 2023-10-19T14:20:50.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.885+0000 7f2c86ade640 1 -- 172.21.15.57:0/1328365167 reap_dead start 2023-10-19T14:20:50.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.885+0000 7f2c88541640 1 -- 172.21.15.57:0/1328365167 shutdown_connections 2023-10-19T14:20:50.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.885+0000 7f2c88541640 1 -- 172.21.15.57:0/1328365167 >> 172.21.15.57:0/1328365167 conn(0x7f2c8006fae0 msgr2=0x7f2c8006fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:50.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.885+0000 7f2c88541640 1 -- 172.21.15.57:0/1328365167 shutdown_connections 2023-10-19T14:20:50.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.885+0000 7f2c88541640 1 -- 172.21.15.57:0/1328365167 wait complete. 2023-10-19T14:20:50.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.886+0000 7f2c88541640 1 Processor -- start 2023-10-19T14:20:50.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.886+0000 7f2c88541640 1 -- start start 2023-10-19T14:20:50.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.887+0000 7f2c88541640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2c6c004d00 0x7f2c8006ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:50.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.887+0000 7f2c88541640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2c8015a310 0x7f2c8014a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:50.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.887+0000 7f2c88541640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2c8015a790 0x7f2c800adb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:50.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.887+0000 7f2c88541640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c801508a0 con 0x7f2c8015a310 2023-10-19T14:20:50.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.887+0000 7f2c88541640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c8014f9d0 con 0x7f2c6c004d00 2023-10-19T14:20:50.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.887+0000 7f2c88541640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c8014fc20 con 0x7f2c8015a790 2023-10-19T14:20:50.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.887+0000 7f2c85adc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2c8015a310 0x7f2c8014a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:50.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.887+0000 7f2c85adc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2c8015a310 0x7f2c8014a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56046/0 (socket says 172.21.15.57:56046) 2023-10-19T14:20:50.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.887+0000 7f2c85adc640 1 -- 172.21.15.57:0/1486135096 learned_addr learned my addr 172.21.15.57:0/1486135096 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:50.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.887+0000 7f2c862dd640 1 --2- 172.21.15.57:0/1486135096 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2c6c004d00 0x7f2c8006ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:50.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.887+0000 7f2c86ade640 1 --2- 172.21.15.57:0/1486135096 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2c8015a790 0x7f2c800adb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:50.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.888+0000 7f2c85adc640 1 -- 172.21.15.57:0/1486135096 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2c8015a790 msgr2=0x7f2c800adb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:50.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.888+0000 7f2c85adc640 1 --2- 172.21.15.57:0/1486135096 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2c8015a790 0x7f2c800adb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:50.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.888+0000 7f2c85adc640 1 -- 172.21.15.57:0/1486135096 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2c6c004d00 msgr2=0x7f2c8006ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:50.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.888+0000 7f2c85adc640 1 --2- 172.21.15.57:0/1486135096 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2c6c004d00 0x7f2c8006ce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:50.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.888+0000 7f2c85adc640 1 -- 172.21.15.57:0/1486135096 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c8014ff00 con 0x7f2c8015a310 2023-10-19T14:20:50.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.888+0000 7f2c862dd640 1 --2- 172.21.15.57:0/1486135096 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2c6c004d00 0x7f2c8006ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:20:50.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.888+0000 7f2c86ade640 1 --2- 172.21.15.57:0/1486135096 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2c8015a790 0x7f2c800adb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:20:50.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.888+0000 7f2c85adc640 1 --2- 172.21.15.57:0/1486135096 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2c8015a310 0x7f2c8014a710 secure :-1 s=READY pgs=2514 cs=0 l=1 rev1=1 crypto rx=0x7f2c7000e700 tx=0x7f2c7000ebc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:50.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.888+0000 7f2c76ffd640 1 -- 172.21.15.57:0/1486135096 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c70004250 con 0x7f2c8015a310 2023-10-19T14:20:50.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.888+0000 7f2c76ffd640 1 -- 172.21.15.57:0/1486135096 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f2c700044c0 con 0x7f2c8015a310 2023-10-19T14:20:50.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.888+0000 7f2c88541640 1 -- 172.21.15.57:0/1486135096 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c801501e0 con 0x7f2c8015a310 2023-10-19T14:20:50.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.888+0000 7f2c76ffd640 1 -- 172.21.15.57:0/1486135096 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c7000caa0 con 0x7f2c8015a310 2023-10-19T14:20:50.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.888+0000 7f2c88541640 1 -- 172.21.15.57:0/1486135096 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c8006d340 con 0x7f2c8015a310 2023-10-19T14:20:50.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.889+0000 7f2c88541640 1 -- 172.21.15.57:0/1486135096 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c800690e0 con 0x7f2c8015a310 2023-10-19T14:20:50.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.890+0000 7f2c76ffd640 1 -- 172.21.15.57:0/1486135096 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 135) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f2c7000cc00 con 0x7f2c8015a310 2023-10-19T14:20:50.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.890+0000 7f2c76ffd640 1 --2- 172.21.15.57:0/1486135096 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f2c6406ac90 0x7f2c6406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:50.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.890+0000 7f2c76ffd640 1 -- 172.21.15.57:0/1486135096 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 7738+0+0 (secure 0 0 0) 0x7f2c7008a140 con 0x7f2c8015a310 2023-10-19T14:20:50.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.890+0000 7f2c862dd640 1 --2- 172.21.15.57:0/1486135096 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f2c6406ac90 0x7f2c6406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:50.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.891+0000 7f2c862dd640 1 --2- 172.21.15.57:0/1486135096 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f2c6406ac90 0x7f2c6406d150 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2c7c009be0 tx=0x7f2c7c005f90 comp rx=0 tx=0).ready entity=mgr.9530 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:50.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:50.894+0000 7f2c76ffd640 1 -- 172.21.15.57:0/1486135096 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2c70054d40 con 0x7f2c8015a310 2023-10-19T14:20:51.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.104+0000 7f2c88541640 1 -- 172.21.15.57:0/1486135096 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2c800cb310 con 0x7f2c8015a310 2023-10-19T14:20:51.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.105+0000 7f2c76ffd640 1 -- 172.21.15.57:0/1486135096 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+6152 (secure 0 0 0) 0x7f2c70015020 con 0x7f2c8015a310 2023-10-19T14:20:51.108 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:20:51.109 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":94,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9248,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/1249807745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":1249807745},{"type":"v1","addr":"172.21.15.179:6836","nonce":1249807745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87},{"gid":10513,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/1406825818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":1406825818},{"type":"v1","addr":"172.21.15.179:6837","nonce":1406825818}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":86}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T14:20:19.077396+0000","modified":"2023-10-19T14:20:21.072483+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11718},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11718":{"gid":11718,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.57:6835/1529368280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":1529368280},{"type":"v1","addr":"172.21.15.57:6835","nonce":1529368280}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13},{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T14:20:30.145135+0000","modified":"2023-10-19T14:20:32.141937+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11715},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11715":{"gid":11715,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":5,"addr":"172.21.15.57:6834/2467807803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2467807803},{"type":"v1","addr":"172.21.15.57:6834","nonce":2467807803}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2023-10-19T14:20:51.109 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 94 2023-10-19T14:20:51.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.107+0000 7f2c88541640 1 -- 172.21.15.57:0/1486135096 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f2c6406ac90 msgr2=0x7f2c6406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:51.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.107+0000 7f2c88541640 1 --2- 172.21.15.57:0/1486135096 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f2c6406ac90 0x7f2c6406d150 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2c7c009be0 tx=0x7f2c7c005f90 comp rx=0 tx=0).stop 2023-10-19T14:20:51.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.107+0000 7f2c88541640 1 -- 172.21.15.57:0/1486135096 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2c8015a310 msgr2=0x7f2c8014a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:51.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.107+0000 7f2c88541640 1 --2- 172.21.15.57:0/1486135096 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2c8015a310 0x7f2c8014a710 secure :-1 s=READY pgs=2514 cs=0 l=1 rev1=1 crypto rx=0x7f2c7000e700 tx=0x7f2c7000ebc0 comp rx=0 tx=0).stop 2023-10-19T14:20:51.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.107+0000 7f2c88541640 1 -- 172.21.15.57:0/1486135096 shutdown_connections 2023-10-19T14:20:51.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.107+0000 7f2c88541640 1 --2- 172.21.15.57:0/1486135096 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f2c6406ac90 0x7f2c6406d150 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:51.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.107+0000 7f2c88541640 1 --2- 172.21.15.57:0/1486135096 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2c8015a790 0x7f2c800adb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:51.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.107+0000 7f2c88541640 1 --2- 172.21.15.57:0/1486135096 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2c8015a310 0x7f2c8014a710 unknown :-1 s=CLOSED pgs=2514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:51.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.107+0000 7f2c88541640 1 --2- 172.21.15.57:0/1486135096 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2c6c004d00 0x7f2c8006ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:51.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.107+0000 7f2c88541640 1 -- 172.21.15.57:0/1486135096 >> 172.21.15.57:0/1486135096 conn(0x7f2c8006fae0 msgr2=0x7f2c8014f0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:51.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.107+0000 7f2c88541640 1 -- 172.21.15.57:0/1486135096 shutdown_connections 2023-10-19T14:20:51.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.107+0000 7f2c88541640 1 -- 172.21.15.57:0/1486135096 wait complete. 2023-10-19T14:20:51.120 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T14:20:51.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.228+0000 7fbe9d44b640 1 -- 172.21.15.57:0/3134847924 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbe980c9dc0 msgr2=0x7fbe9806b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:51.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.228+0000 7fbe9d44b640 1 --2- 172.21.15.57:0/3134847924 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbe980c9dc0 0x7fbe9806b2e0 secure :-1 s=READY pgs=2516 cs=0 l=1 rev1=1 crypto rx=0x7fbe80005fd0 tx=0x7fbe8002bed0 comp rx=0 tx=0).stop 2023-10-19T14:20:51.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.228+0000 7fbe9d44b640 1 -- 172.21.15.57:0/3134847924 shutdown_connections 2023-10-19T14:20:51.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.228+0000 7fbe9d44b640 1 --2- 172.21.15.57:0/3134847924 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbe980c9dc0 0x7fbe9806b2e0 unknown :-1 s=CLOSED pgs=2516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:51.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.228+0000 7fbe9d44b640 1 -- 172.21.15.57:0/3134847924 >> 172.21.15.57:0/3134847924 conn(0x7fbe98069930 msgr2=0x7fbe98069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:51.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.228+0000 7fbe9d44b640 1 -- 172.21.15.57:0/3134847924 shutdown_connections 2023-10-19T14:20:51.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.228+0000 7fbe9d44b640 1 -- 172.21.15.57:0/3134847924 wait complete. 2023-10-19T14:20:51.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.229+0000 7fbe9d44b640 1 Processor -- start 2023-10-19T14:20:51.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.229+0000 7fbe9d44b640 1 -- start start 2023-10-19T14:20:51.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.230+0000 7fbe9d44b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbe9806af00 0x7fbe9812ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:51.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.230+0000 7fbe9d44b640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbe980ab500 0x7fbe9815aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:51.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.230+0000 7fbe9d44b640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbe980c9dc0 0x7fbe980ac7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:51.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.230+0000 7fbe9d44b640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe9812f750 con 0x7fbe9806af00 2023-10-19T14:20:51.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.230+0000 7fbe9d44b640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe980b1a80 con 0x7fbe980c9dc0 2023-10-19T14:20:51.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.230+0000 7fbe9d44b640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe980b1d60 con 0x7fbe980ab500 2023-10-19T14:20:51.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.230+0000 7fbe96ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbe9806af00 0x7fbe9812ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:51.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.230+0000 7fbe96ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbe9806af00 0x7fbe9812ff30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56070/0 (socket says 172.21.15.57:56070) 2023-10-19T14:20:51.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.230+0000 7fbe96ffd640 1 -- 172.21.15.57:0/2245591147 learned_addr learned my addr 172.21.15.57:0/2245591147 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:51.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.230+0000 7fbe967fc640 1 --2- 172.21.15.57:0/2245591147 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbe980ab500 0x7fbe9815aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:51.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.230+0000 7fbe977fe640 1 --2- 172.21.15.57:0/2245591147 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbe980c9dc0 0x7fbe980ac7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:51.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.230+0000 7fbe96ffd640 1 -- 172.21.15.57:0/2245591147 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbe980ab500 msgr2=0x7fbe9815aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:51.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.230+0000 7fbe96ffd640 1 --2- 172.21.15.57:0/2245591147 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbe980ab500 0x7fbe9815aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:51.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.230+0000 7fbe96ffd640 1 -- 172.21.15.57:0/2245591147 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbe980c9dc0 msgr2=0x7fbe980ac7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:51.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.230+0000 7fbe96ffd640 1 --2- 172.21.15.57:0/2245591147 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbe980c9dc0 0x7fbe980ac7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:51.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.230+0000 7fbe96ffd640 1 -- 172.21.15.57:0/2245591147 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe80009560 con 0x7fbe9806af00 2023-10-19T14:20:51.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.231+0000 7fbe96ffd640 1 --2- 172.21.15.57:0/2245591147 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbe9806af00 0x7fbe9812ff30 secure :-1 s=READY pgs=2517 cs=0 l=1 rev1=1 crypto rx=0x7fbe8800c450 tx=0x7fbe8800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:51.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.231+0000 7fbe8f7fe640 1 -- 172.21.15.57:0/2245591147 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe880052d0 con 0x7fbe9806af00 2023-10-19T14:20:51.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.231+0000 7fbe8f7fe640 1 -- 172.21.15.57:0/2245591147 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fbe88005430 con 0x7fbe9806af00 2023-10-19T14:20:51.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.231+0000 7fbe9d44b640 1 -- 172.21.15.57:0/2245591147 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe9812fa10 con 0x7fbe9806af00 2023-10-19T14:20:51.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.231+0000 7fbe8f7fe640 1 -- 172.21.15.57:0/2245591147 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe88012890 con 0x7fbe9806af00 2023-10-19T14:20:51.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.231+0000 7fbe9d44b640 1 -- 172.21.15.57:0/2245591147 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe9815af60 con 0x7fbe9806af00 2023-10-19T14:20:51.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.232+0000 7fbe9d44b640 1 -- 172.21.15.57:0/2245591147 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe9812fc00 con 0x7fbe9806af00 2023-10-19T14:20:51.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.232+0000 7fbe8f7fe640 1 -- 172.21.15.57:0/2245591147 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 136) v1 ==== 89377+0+0 (secure 0 0 0) 0x7fbe880035f0 con 0x7fbe9806af00 2023-10-19T14:20:51.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.233+0000 7fbe8f7fe640 1 --2- 172.21.15.57:0/2245591147 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fbe7806b020 0x7fbe7806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:51.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.233+0000 7fbe8f7fe640 1 -- 172.21.15.57:0/2245591147 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 7738+0+0 (secure 0 0 0) 0x7fbe88089ce0 con 0x7fbe9806af00 2023-10-19T14:20:51.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.233+0000 7fbe967fc640 1 --2- 172.21.15.57:0/2245591147 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fbe7806b020 0x7fbe7806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:51.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.233+0000 7fbe967fc640 1 --2- 172.21.15.57:0/2245591147 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fbe7806b020 0x7fbe7806d4e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fbe80035000 tx=0x7fbe8002e040 comp rx=0 tx=0).ready entity=mgr.9530 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:51.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.237+0000 7fbe8f7fe640 1 -- 172.21.15.57:0/2245591147 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fbe88054330 con 0x7fbe9806af00 2023-10-19T14:20:51.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.416+0000 7fbe9d44b640 1 -- 172.21.15.57:0/2245591147 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbe98159a00 con 0x7fbe9806af00 2023-10-19T14:20:51.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.418+0000 7fbe8f7fe640 1 -- 172.21.15.57:0/2245591147 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v229) v1 ==== 74+0+21701 (secure 0 0 0) 0x7fbe88057950 con 0x7fbe9806af00 2023-10-19T14:20:51.421 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:20:51.421 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":229,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T14:20:48.148672+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2023-10-19T14:20:17.357433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"222","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2023-10-19T14:20:18.049516+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"223","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_metadata","create_time":"2023-10-19T14:20:28.667664+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"226","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"backup_fs_data","create_time":"2023-10-19T14:20:29.122584+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"227","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.179:0/1513076424":"2023-10-20T14:20:48.148610+0000","172.21.15.179:0/1464950687":"2023-10-20T14:20:48.148610+0000","172.21.15.179:0/2900872065":"2023-10-20T14:20:48.148610+0000","172.21.15.179:0/2140451434":"2023-10-20T14:20:48.148610+0000","172.21.15.179:0/2605624080":"2023-10-20T14:20:48.148610+0000","172.21.15.179:0/1194984394":"2023-10-20T14:20:39.702634+0000","172.21.15.179:6833/3232482678":"2023-10-20T14:20:39.702634+0000","172.21.15.179:6832/4242302683":"2023-10-20T14:20:48.148610+0000","172.21.15.179:6832/3232482678":"2023-10-20T14:20:39.702634+0000","172.21.15.179:0/2872579179":"2023-10-20T14:20:39.702634+0000","172.21.15.179:0/373323540":"2023-10-20T14:20:48.148610+0000","172.21.15.179:0/3910604357":"2023-10-20T14:20:39.702634+0000","172.21.15.179:0/4185855767":"2023-10-20T14:20:48.148610+0000","172.21.15.179:0/536160557":"2023-10-20T14:20:39.702634+0000","172.21.15.179:0/3239062050":"2023-10-20T14:20:39.702634+0000","172.21.15.179:6833/4242302683":"2023-10-20T14:20:48.148610+0000","172.21.15.179:0/3672528294":"2023-10-20T14:20:48.148610+0000","172.21.15.179:0/2003826253":"2023-10-20T14:20:48.148610+0000","172.21.15.179:0/1948632334":"2023-10-20T14:20:39.702634+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T14:20:51.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.419+0000 7fbe9d44b640 1 -- 172.21.15.57:0/2245591147 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fbe7806b020 msgr2=0x7fbe7806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:51.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.419+0000 7fbe9d44b640 1 --2- 172.21.15.57:0/2245591147 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fbe7806b020 0x7fbe7806d4e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fbe80035000 tx=0x7fbe8002e040 comp rx=0 tx=0).stop 2023-10-19T14:20:51.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.419+0000 7fbe9d44b640 1 -- 172.21.15.57:0/2245591147 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbe9806af00 msgr2=0x7fbe9812ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:51.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.419+0000 7fbe9d44b640 1 --2- 172.21.15.57:0/2245591147 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbe9806af00 0x7fbe9812ff30 secure :-1 s=READY pgs=2517 cs=0 l=1 rev1=1 crypto rx=0x7fbe8800c450 tx=0x7fbe8800c910 comp rx=0 tx=0).stop 2023-10-19T14:20:51.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.419+0000 7fbe9d44b640 1 -- 172.21.15.57:0/2245591147 shutdown_connections 2023-10-19T14:20:51.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.419+0000 7fbe9d44b640 1 --2- 172.21.15.57:0/2245591147 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fbe7806b020 0x7fbe7806d4e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:51.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.419+0000 7fbe9d44b640 1 --2- 172.21.15.57:0/2245591147 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbe980c9dc0 0x7fbe980ac7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:51.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.420+0000 7fbe9d44b640 1 --2- 172.21.15.57:0/2245591147 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbe980ab500 0x7fbe9815aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:51.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.420+0000 7fbe9d44b640 1 --2- 172.21.15.57:0/2245591147 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbe9806af00 0x7fbe9812ff30 unknown :-1 s=CLOSED pgs=2517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:51.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.420+0000 7fbe9d44b640 1 -- 172.21.15.57:0/2245591147 >> 172.21.15.57:0/2245591147 conn(0x7fbe98069930 msgr2=0x7fbe9814fab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:51.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.420+0000 7fbe9d44b640 1 -- 172.21.15.57:0/2245591147 shutdown_connections 2023-10-19T14:20:51.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.420+0000 7fbe9d44b640 1 -- 172.21.15.57:0/2245591147 wait complete. 2023-10-19T14:20:51.433 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-19T14:20:51.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.540+0000 7fbe24607640 1 -- 172.21.15.57:0/2869078970 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbe1c06af00 msgr2=0x7fbe1c06b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:51.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.540+0000 7fbe24607640 1 --2- 172.21.15.57:0/2869078970 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbe1c06af00 0x7fbe1c06b2e0 secure :-1 s=READY pgs=2519 cs=0 l=1 rev1=1 crypto rx=0x7fbe14009860 tx=0x7fbe1402bd60 comp rx=0 tx=0).stop 2023-10-19T14:20:51.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.540+0000 7fbe24607640 1 -- 172.21.15.57:0/2869078970 shutdown_connections 2023-10-19T14:20:51.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.540+0000 7fbe24607640 1 --2- 172.21.15.57:0/2869078970 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbe1c0c96b0 0x7fbe1c0c9b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:51.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.540+0000 7fbe24607640 1 --2- 172.21.15.57:0/2869078970 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbe1c06af00 0x7fbe1c06b2e0 unknown :-1 s=CLOSED pgs=2519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:51.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.540+0000 7fbe24607640 1 -- 172.21.15.57:0/2869078970 >> 172.21.15.57:0/2869078970 conn(0x7fbe1c069930 msgr2=0x7fbe1c069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:51.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.540+0000 7fbe24607640 1 -- 172.21.15.57:0/2869078970 shutdown_connections 2023-10-19T14:20:51.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.540+0000 7fbe24607640 1 -- 172.21.15.57:0/2869078970 wait complete. 2023-10-19T14:20:51.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.541+0000 7fbe24607640 1 Processor -- start 2023-10-19T14:20:51.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.541+0000 7fbe24607640 1 -- start start 2023-10-19T14:20:51.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.542+0000 7fbe24607640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbe1c06af00 0x7fbe1c152b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:51.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.542+0000 7fbe24607640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbe1c0ab500 0x7fbe1c151da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:51.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.542+0000 7fbe24607640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbe1c0c96b0 0x7fbe1c0aa7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:51.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.542+0000 7fbe24607640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe1c0a7eb0 con 0x7fbe1c06af00 2023-10-19T14:20:51.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.542+0000 7fbe24607640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe1c139290 con 0x7fbe1c0c96b0 2023-10-19T14:20:51.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.542+0000 7fbe24607640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe1c1530c0 con 0x7fbe1c0ab500 2023-10-19T14:20:51.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.542+0000 7fbe223a3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbe1c06af00 0x7fbe1c152b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:51.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.542+0000 7fbe223a3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbe1c06af00 0x7fbe1c152b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56088/0 (socket says 172.21.15.57:56088) 2023-10-19T14:20:51.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.542+0000 7fbe223a3640 1 -- 172.21.15.57:0/4190534227 learned_addr learned my addr 172.21.15.57:0/4190534227 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:51.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.542+0000 7fbe22ba4640 1 --2- 172.21.15.57:0/4190534227 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbe1c0c96b0 0x7fbe1c0aa7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:51.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.542+0000 7fbe21ba2640 1 --2- 172.21.15.57:0/4190534227 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbe1c0ab500 0x7fbe1c151da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:51.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.542+0000 7fbe223a3640 1 -- 172.21.15.57:0/4190534227 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbe1c0ab500 msgr2=0x7fbe1c151da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:51.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.542+0000 7fbe223a3640 1 --2- 172.21.15.57:0/4190534227 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbe1c0ab500 0x7fbe1c151da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:51.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.542+0000 7fbe223a3640 1 -- 172.21.15.57:0/4190534227 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbe1c0c96b0 msgr2=0x7fbe1c0aa7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:51.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.542+0000 7fbe223a3640 1 --2- 172.21.15.57:0/4190534227 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbe1c0c96b0 0x7fbe1c0aa7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:51.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.542+0000 7fbe223a3640 1 -- 172.21.15.57:0/4190534227 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe14009490 con 0x7fbe1c06af00 2023-10-19T14:20:51.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.542+0000 7fbe22ba4640 1 --2- 172.21.15.57:0/4190534227 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbe1c0c96b0 0x7fbe1c0aa7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:20:51.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.543+0000 7fbe223a3640 1 --2- 172.21.15.57:0/4190534227 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbe1c06af00 0x7fbe1c152b80 secure :-1 s=READY pgs=2520 cs=0 l=1 rev1=1 crypto rx=0x7fbe14009980 tx=0x7fbe140038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:51.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.543+0000 7fbe1affd640 1 -- 172.21.15.57:0/4190534227 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe14003ea0 con 0x7fbe1c06af00 2023-10-19T14:20:51.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.543+0000 7fbe1affd640 1 -- 172.21.15.57:0/4190534227 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fbe14005020 con 0x7fbe1c06af00 2023-10-19T14:20:51.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.543+0000 7fbe1affd640 1 -- 172.21.15.57:0/4190534227 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe14005410 con 0x7fbe1c06af00 2023-10-19T14:20:51.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.543+0000 7fbe24607640 1 -- 172.21.15.57:0/4190534227 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe1c0a72a0 con 0x7fbe1c06af00 2023-10-19T14:20:51.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.543+0000 7fbe24607640 1 -- 172.21.15.57:0/4190534227 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe1c0a7860 con 0x7fbe1c06af00 2023-10-19T14:20:51.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.544+0000 7fbe24607640 1 -- 172.21.15.57:0/4190534227 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe1c06c7e0 con 0x7fbe1c06af00 2023-10-19T14:20:51.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.544+0000 7fbe1affd640 1 -- 172.21.15.57:0/4190534227 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 136) v1 ==== 89377+0+0 (secure 0 0 0) 0x7fbe140047a0 con 0x7fbe1c06af00 2023-10-19T14:20:51.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.545+0000 7fbe1affd640 1 --2- 172.21.15.57:0/4190534227 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fbe0806aff0 0x7fbe0806d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:51.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.545+0000 7fbe1affd640 1 -- 172.21.15.57:0/4190534227 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 7738+0+0 (secure 0 0 0) 0x7fbe140aca40 con 0x7fbe1c06af00 2023-10-19T14:20:51.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.545+0000 7fbe21ba2640 1 --2- 172.21.15.57:0/4190534227 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fbe0806aff0 0x7fbe0806d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:51.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.545+0000 7fbe21ba2640 1 --2- 172.21.15.57:0/4190534227 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fbe0806aff0 0x7fbe0806d4b0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fbe100058c0 tx=0x7fbe10005850 comp rx=0 tx=0).ready entity=mgr.9530 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:51.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.549+0000 7fbe1affd640 1 -- 172.21.15.57:0/4190534227 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fbe140aee60 con 0x7fbe1c06af00 2023-10-19T14:20:51.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.760+0000 7fbe24607640 1 -- 172.21.15.57:0/4190534227 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fbe1c13bad0 con 0x7fbe1c06af00 2023-10-19T14:20:51.764 INFO:tasks.ceph.mon.a.smithi057.stderr:2023-10-19T14:20:51.761+0000 7f1253ff3640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-19T14:20:51.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.763+0000 7fbe1affd640 1 -- 172.21.15.57:0/4190534227 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v95) v1 ==== 161+0+0 (secure 0 0 0) 0x7fbe14039070 con 0x7fbe1c06af00 2023-10-19T14:20:51.766 INFO:teuthology.orchestra.run.smithi057.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-19T14:20:51.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.765+0000 7fbe24607640 1 -- 172.21.15.57:0/4190534227 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fbe0806aff0 msgr2=0x7fbe0806d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:51.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.765+0000 7fbe24607640 1 --2- 172.21.15.57:0/4190534227 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fbe0806aff0 0x7fbe0806d4b0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fbe100058c0 tx=0x7fbe10005850 comp rx=0 tx=0).stop 2023-10-19T14:20:51.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.765+0000 7fbe24607640 1 -- 172.21.15.57:0/4190534227 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbe1c06af00 msgr2=0x7fbe1c152b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:51.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.765+0000 7fbe24607640 1 --2- 172.21.15.57:0/4190534227 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbe1c06af00 0x7fbe1c152b80 secure :-1 s=READY pgs=2520 cs=0 l=1 rev1=1 crypto rx=0x7fbe14009980 tx=0x7fbe140038a0 comp rx=0 tx=0).stop 2023-10-19T14:20:51.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.765+0000 7fbe24607640 1 -- 172.21.15.57:0/4190534227 shutdown_connections 2023-10-19T14:20:51.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.765+0000 7fbe24607640 1 --2- 172.21.15.57:0/4190534227 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fbe0806aff0 0x7fbe0806d4b0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:51.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.765+0000 7fbe24607640 1 --2- 172.21.15.57:0/4190534227 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbe1c0c96b0 0x7fbe1c0aa7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:51.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.765+0000 7fbe24607640 1 --2- 172.21.15.57:0/4190534227 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbe1c0ab500 0x7fbe1c151da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:51.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.765+0000 7fbe24607640 1 --2- 172.21.15.57:0/4190534227 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbe1c06af00 0x7fbe1c152b80 unknown :-1 s=CLOSED pgs=2520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:51.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.765+0000 7fbe24607640 1 -- 172.21.15.57:0/4190534227 >> 172.21.15.57:0/4190534227 conn(0x7fbe1c069930 msgr2=0x7fbe1c0ab990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:51.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.765+0000 7fbe24607640 1 -- 172.21.15.57:0/4190534227 shutdown_connections 2023-10-19T14:20:51.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.765+0000 7fbe24607640 1 -- 172.21.15.57:0/4190534227 wait complete. 2023-10-19T14:20:51.770 INFO:tasks.ceph.mds.a.smithi057.stderr: -3413> 2023-10-19T14:20:21.670+0000 7f2368f2e640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-19T14:20:51.780 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-19T14:20:51.800 INFO:tasks.ceph.mds.a.smithi057.stdout:starting mds.a at 2023-10-19T14:20:51.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.891+0000 7f6df7576640 1 -- 172.21.15.57:0/2756656560 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6dd80047c0 msgr2=0x7f6dd8004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:51.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.891+0000 7f6df7576640 1 --2- 172.21.15.57:0/2756656560 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6dd80047c0 0x7f6dd8004c40 secure :-1 s=READY pgs=2525 cs=0 l=1 rev1=1 crypto rx=0x7f6de4009b80 tx=0x7f6dd80057f0 comp rx=0 tx=0).stop 2023-10-19T14:20:51.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.891+0000 7f6df5b13640 1 -- 172.21.15.57:0/2756656560 reap_dead start 2023-10-19T14:20:51.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.891+0000 7f6df7576640 1 -- 172.21.15.57:0/2756656560 shutdown_connections 2023-10-19T14:20:51.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.891+0000 7f6df7576640 1 -- 172.21.15.57:0/2756656560 >> 172.21.15.57:0/2756656560 conn(0x7f6df00d0d60 msgr2=0x7f6df00d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:51.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.891+0000 7f6df7576640 1 -- 172.21.15.57:0/2756656560 shutdown_connections 2023-10-19T14:20:51.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.891+0000 7f6df7576640 1 -- 172.21.15.57:0/2756656560 wait complete. 2023-10-19T14:20:51.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.892+0000 7f6df7576640 1 Processor -- start 2023-10-19T14:20:51.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.892+0000 7f6df7576640 1 -- start start 2023-10-19T14:20:51.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.893+0000 7f6df7576640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6dd80047c0 0x7f6df0157510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:51.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.893+0000 7f6df7576640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6df01f2580 0x7f6df00c92f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:51.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.893+0000 7f6df7576640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6df00cc340 0x7f6df00cc7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:51.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.893+0000 7f6df7576640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6df00ccd00 con 0x7f6dd80047c0 2023-10-19T14:20:51.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.893+0000 7f6df7576640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6df00c9830 con 0x7f6df00cc340 2023-10-19T14:20:51.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.893+0000 7f6df7576640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6df00b4300 con 0x7f6df01f2580 2023-10-19T14:20:51.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.893+0000 7f6df5312640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6dd80047c0 0x7f6df0157510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:51.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.893+0000 7f6df5312640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6dd80047c0 0x7f6df0157510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56132/0 (socket says 172.21.15.57:56132) 2023-10-19T14:20:51.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.893+0000 7f6df5312640 1 -- 172.21.15.57:0/3626329594 learned_addr learned my addr 172.21.15.57:0/3626329594 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:51.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.893+0000 7f6df4b11640 1 --2- 172.21.15.57:0/3626329594 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6df01f2580 0x7f6df00c92f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:51.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.893+0000 7f6df5b13640 1 --2- 172.21.15.57:0/3626329594 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6df00cc340 0x7f6df00cc7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:51.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.893+0000 7f6df5312640 1 -- 172.21.15.57:0/3626329594 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6df01f2580 msgr2=0x7f6df00c92f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:51.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.893+0000 7f6df5312640 1 --2- 172.21.15.57:0/3626329594 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6df01f2580 0x7f6df00c92f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:51.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.893+0000 7f6df5312640 1 -- 172.21.15.57:0/3626329594 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6df00cc340 msgr2=0x7f6df00cc7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:51.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.893+0000 7f6df5312640 1 --2- 172.21.15.57:0/3626329594 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6df00cc340 0x7f6df00cc7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:51.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.893+0000 7f6df5312640 1 -- 172.21.15.57:0/3626329594 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6df00b45e0 con 0x7f6dd80047c0 2023-10-19T14:20:51.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.893+0000 7f6df5312640 1 --2- 172.21.15.57:0/3626329594 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6dd80047c0 0x7f6df0157510 secure :-1 s=READY pgs=2526 cs=0 l=1 rev1=1 crypto rx=0x7f6de800e590 tx=0x7f6de800ea50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:51.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.894+0000 7f6dedffb640 1 -- 172.21.15.57:0/3626329594 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6de8010890 con 0x7f6dd80047c0 2023-10-19T14:20:51.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.894+0000 7f6dedffb640 1 -- 172.21.15.57:0/3626329594 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f6de8010ae0 con 0x7f6dd80047c0 2023-10-19T14:20:51.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.894+0000 7f6df7576640 1 -- 172.21.15.57:0/3626329594 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6df00b48c0 con 0x7f6dd80047c0 2023-10-19T14:20:51.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.894+0000 7f6dedffb640 1 -- 172.21.15.57:0/3626329594 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6de8011460 con 0x7f6dd80047c0 2023-10-19T14:20:51.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.894+0000 7f6df7576640 1 -- 172.21.15.57:0/3626329594 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6df00b4e80 con 0x7f6dd80047c0 2023-10-19T14:20:51.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.895+0000 7f6df7576640 1 -- 172.21.15.57:0/3626329594 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6df00b50a0 con 0x7f6dd80047c0 2023-10-19T14:20:51.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.895+0000 7f6dedffb640 1 -- 172.21.15.57:0/3626329594 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 136) v1 ==== 89377+0+0 (secure 0 0 0) 0x7f6de80115c0 con 0x7f6dd80047c0 2023-10-19T14:20:51.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.895+0000 7f6dedffb640 1 --2- 172.21.15.57:0/3626329594 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f6ddc06b020 0x7f6ddc06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:51.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.896+0000 7f6dedffb640 1 -- 172.21.15.57:0/3626329594 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(230..230 src has 1..230) v4 ==== 7824+0+0 (secure 0 0 0) 0x7f6de808a9a0 con 0x7f6dd80047c0 2023-10-19T14:20:51.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.896+0000 7f6df4b11640 1 --2- 172.21.15.57:0/3626329594 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f6ddc06b020 0x7f6ddc06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:51.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.896+0000 7f6df4b11640 1 --2- 172.21.15.57:0/3626329594 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f6ddc06b020 0x7f6ddc06d4e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f6de4036000 tx=0x7f6de4035040 comp rx=0 tx=0).ready entity=mgr.9530 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:51.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:51.900+0000 7f6dedffb640 1 -- 172.21.15.57:0/3626329594 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6de8054f10 con 0x7f6dd80047c0 2023-10-19T14:20:52.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:52.115+0000 7f6df7576640 1 -- 172.21.15.57:0/3626329594 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f6df00d04f0 con 0x7f6dd80047c0 2023-10-19T14:20:52.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:52.775+0000 7f6dedffb640 1 -- 172.21.15.57:0/3626329594 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v96) v1 ==== 104+0+0 (secure 0 0 0) 0x7f6de801f020 con 0x7f6dd80047c0 2023-10-19T14:20:52.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:52.778+0000 7f6df7576640 1 -- 172.21.15.57:0/3626329594 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f6ddc06b020 msgr2=0x7f6ddc06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:52.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:52.778+0000 7f6df7576640 1 --2- 172.21.15.57:0/3626329594 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f6ddc06b020 0x7f6ddc06d4e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f6de4036000 tx=0x7f6de4035040 comp rx=0 tx=0).stop 2023-10-19T14:20:52.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:52.778+0000 7f6df7576640 1 -- 172.21.15.57:0/3626329594 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6dd80047c0 msgr2=0x7f6df0157510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:52.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:52.778+0000 7f6df7576640 1 --2- 172.21.15.57:0/3626329594 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6dd80047c0 0x7f6df0157510 secure :-1 s=READY pgs=2526 cs=0 l=1 rev1=1 crypto rx=0x7f6de800e590 tx=0x7f6de800ea50 comp rx=0 tx=0).stop 2023-10-19T14:20:52.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:52.778+0000 7f6df7576640 1 -- 172.21.15.57:0/3626329594 shutdown_connections 2023-10-19T14:20:52.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:52.778+0000 7f6df7576640 1 --2- 172.21.15.57:0/3626329594 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f6ddc06b020 0x7f6ddc06d4e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:52.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:52.778+0000 7f6df7576640 1 --2- 172.21.15.57:0/3626329594 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6df00cc340 0x7f6df00cc7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:52.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:52.778+0000 7f6df7576640 1 --2- 172.21.15.57:0/3626329594 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6df01f2580 0x7f6df00c92f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:52.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:52.778+0000 7f6df7576640 1 --2- 172.21.15.57:0/3626329594 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6dd80047c0 0x7f6df0157510 unknown :-1 s=CLOSED pgs=2526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:52.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:52.778+0000 7f6df7576640 1 -- 172.21.15.57:0/3626329594 >> 172.21.15.57:0/3626329594 conn(0x7f6df00d0d60 msgr2=0x7f6df00ae1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:52.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:52.778+0000 7f6df7576640 1 -- 172.21.15.57:0/3626329594 shutdown_connections 2023-10-19T14:20:52.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:52.778+0000 7f6df7576640 1 -- 172.21.15.57:0/3626329594 wait complete. 2023-10-19T14:20:52.796 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-19T14:20:52.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:52.953+0000 7f202ccc2640 1 -- 172.21.15.57:0/1065129348 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f20140047c0 msgr2=0x7f2014004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:52.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:52.953+0000 7f202ccc2640 1 --2- 172.21.15.57:0/1065129348 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f20140047c0 0x7f2014004c40 secure :-1 s=READY pgs=2528 cs=0 l=1 rev1=1 crypto rx=0x7f2010001680 tx=0x7f20140057f0 comp rx=0 tx=0).stop 2023-10-19T14:20:52.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:52.953+0000 7f202ccc2640 1 -- 172.21.15.57:0/1065129348 shutdown_connections 2023-10-19T14:20:52.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:52.953+0000 7f202ccc2640 1 --2- 172.21.15.57:0/1065129348 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f20140047c0 0x7f2014004c40 unknown :-1 s=CLOSED pgs=2528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:52.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:52.953+0000 7f202ccc2640 1 -- 172.21.15.57:0/1065129348 >> 172.21.15.57:0/1065129348 conn(0x7f20280af7a0 msgr2=0x7f20280afb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:52.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:52.953+0000 7f202ccc2640 1 -- 172.21.15.57:0/1065129348 shutdown_connections 2023-10-19T14:20:52.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:52.953+0000 7f202ccc2640 1 -- 172.21.15.57:0/1065129348 wait complete. 2023-10-19T14:20:52.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:52.955+0000 7f202ccc2640 1 Processor -- start 2023-10-19T14:20:52.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:52.955+0000 7f202ccc2640 1 -- start start 2023-10-19T14:20:52.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:52.955+0000 7f202ccc2640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f20140047c0 0x7f20280b2470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:52.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:52.955+0000 7f202ccc2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2028156ea0 0x7f2028157320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:52.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:52.955+0000 7f202ccc2640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2028144dc0 0x7f2028145240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:52.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:52.955+0000 7f202ccc2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2028157860 con 0x7f2028156ea0 2023-10-19T14:20:52.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:52.955+0000 7f202ccc2640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2028070f60 con 0x7f20140047c0 2023-10-19T14:20:52.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:52.955+0000 7f202ccc2640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20281f4bf0 con 0x7f2028144dc0 2023-10-19T14:20:52.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:52.956+0000 7f201ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2028156ea0 0x7f2028157320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:52.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:52.956+0000 7f201ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2028156ea0 0x7f2028157320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56144/0 (socket says 172.21.15.57:56144) 2023-10-19T14:20:52.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:52.956+0000 7f201ffff640 1 -- 172.21.15.57:0/1917515292 learned_addr learned my addr 172.21.15.57:0/1917515292 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:52.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:52.956+0000 7f2026d9d640 1 --2- 172.21.15.57:0/1917515292 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2028144dc0 0x7f2028145240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:52.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:52.956+0000 7f202659c640 1 --2- 172.21.15.57:0/1917515292 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f20140047c0 0x7f20280b2470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:52.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:52.956+0000 7f201ffff640 1 -- 172.21.15.57:0/1917515292 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2028144dc0 msgr2=0x7f2028145240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:52.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:52.956+0000 7f201ffff640 1 --2- 172.21.15.57:0/1917515292 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2028144dc0 0x7f2028145240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:52.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:52.956+0000 7f201ffff640 1 -- 172.21.15.57:0/1917515292 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f20140047c0 msgr2=0x7f20280b2470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:52.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:52.956+0000 7f201ffff640 1 --2- 172.21.15.57:0/1917515292 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f20140047c0 0x7f20280b2470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:52.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:52.956+0000 7f201ffff640 1 -- 172.21.15.57:0/1917515292 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20281f4ed0 con 0x7f2028156ea0 2023-10-19T14:20:52.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:52.956+0000 7f2026d9d640 1 --2- 172.21.15.57:0/1917515292 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2028144dc0 0x7f2028145240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:20:52.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:52.956+0000 7f201ffff640 1 --2- 172.21.15.57:0/1917515292 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2028156ea0 0x7f2028157320 secure :-1 s=READY pgs=2529 cs=0 l=1 rev1=1 crypto rx=0x7f201002c0d0 tx=0x7f2010006360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:52.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:52.956+0000 7f201effd640 1 -- 172.21.15.57:0/1917515292 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2010006540 con 0x7f2028156ea0 2023-10-19T14:20:52.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:52.956+0000 7f201effd640 1 -- 172.21.15.57:0/1917515292 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f2010003c70 con 0x7f2028156ea0 2023-10-19T14:20:52.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:52.956+0000 7f202ccc2640 1 -- 172.21.15.57:0/1917515292 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20281f51b0 con 0x7f2028156ea0 2023-10-19T14:20:52.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:52.956+0000 7f201effd640 1 -- 172.21.15.57:0/1917515292 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f201002c790 con 0x7f2028156ea0 2023-10-19T14:20:52.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:52.956+0000 7f202ccc2640 1 -- 172.21.15.57:0/1917515292 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20281f5770 con 0x7f2028156ea0 2023-10-19T14:20:52.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:52.957+0000 7f202ccc2640 1 -- 172.21.15.57:0/1917515292 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20281f5900 con 0x7f2028156ea0 2023-10-19T14:20:52.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:52.958+0000 7f201effd640 1 -- 172.21.15.57:0/1917515292 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f201002c8f0 con 0x7f2028156ea0 2023-10-19T14:20:52.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:52.958+0000 7f201effd640 1 --2- 172.21.15.57:0/1917515292 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f200806b080 0x7f200806d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:52.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:52.958+0000 7f201effd640 1 -- 172.21.15.57:0/1917515292 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(230..230 src has 1..230) v4 ==== 7824+0+0 (secure 0 0 0) 0x7f20100ab380 con 0x7f2028156ea0 2023-10-19T14:20:52.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:52.958+0000 7f202659c640 1 --2- 172.21.15.57:0/1917515292 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f200806b080 0x7f200806d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:52.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:52.959+0000 7f202659c640 1 --2- 172.21.15.57:0/1917515292 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f200806b080 0x7f200806d540 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f20280709e0 tx=0x7f2018005f90 comp rx=0 tx=0).ready entity=mgr.9530 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:52.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:52.962+0000 7f201effd640 1 -- 172.21.15.57:0/1917515292 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2010075970 con 0x7f2028156ea0 2023-10-19T14:20:53.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:53.141+0000 7f202ccc2640 1 -- 172.21.15.57:0/1917515292 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f20280690e0 con 0x7f2028156ea0 2023-10-19T14:20:53.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:53.776+0000 7f201effd640 1 -- 172.21.15.57:0/1917515292 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v231) v1 ==== 181+0+0 (secure 0 0 0) 0x7f2010044070 con 0x7f2028156ea0 2023-10-19T14:20:53.780 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'cephfs_metadata' removed 2023-10-19T14:20:53.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:53.779+0000 7f202ccc2640 1 -- 172.21.15.57:0/1917515292 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f200806b080 msgr2=0x7f200806d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:53.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:53.779+0000 7f202ccc2640 1 --2- 172.21.15.57:0/1917515292 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f200806b080 0x7f200806d540 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f20280709e0 tx=0x7f2018005f90 comp rx=0 tx=0).stop 2023-10-19T14:20:53.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:53.779+0000 7f202ccc2640 1 -- 172.21.15.57:0/1917515292 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2028156ea0 msgr2=0x7f2028157320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:53.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:53.779+0000 7f202ccc2640 1 --2- 172.21.15.57:0/1917515292 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2028156ea0 0x7f2028157320 secure :-1 s=READY pgs=2529 cs=0 l=1 rev1=1 crypto rx=0x7f201002c0d0 tx=0x7f2010006360 comp rx=0 tx=0).stop 2023-10-19T14:20:53.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:53.782+0000 7f202ccc2640 1 -- 172.21.15.57:0/1917515292 shutdown_connections 2023-10-19T14:20:53.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:53.782+0000 7f202ccc2640 1 --2- 172.21.15.57:0/1917515292 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f200806b080 0x7f200806d540 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:53.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:53.782+0000 7f202ccc2640 1 --2- 172.21.15.57:0/1917515292 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2028144dc0 0x7f2028145240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:53.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:53.782+0000 7f202ccc2640 1 --2- 172.21.15.57:0/1917515292 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2028156ea0 0x7f2028157320 unknown :-1 s=CLOSED pgs=2529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:53.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:53.782+0000 7f202ccc2640 1 --2- 172.21.15.57:0/1917515292 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f20140047c0 0x7f20280b2470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:53.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:53.782+0000 7f202ccc2640 1 -- 172.21.15.57:0/1917515292 >> 172.21.15.57:0/1917515292 conn(0x7f20280af7a0 msgr2=0x7f2028155400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:53.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:53.783+0000 7f202ccc2640 1 -- 172.21.15.57:0/1917515292 shutdown_connections 2023-10-19T14:20:53.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:53.783+0000 7f202ccc2640 1 -- 172.21.15.57:0/1917515292 wait complete. 2023-10-19T14:20:53.801 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-19T14:20:53.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:53.916+0000 7fc224f97640 1 -- 172.21.15.57:0/1386795944 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc220142850 msgr2=0x7fc220159440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:53.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:53.916+0000 7fc224f97640 1 --2- 172.21.15.57:0/1386795944 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc220142850 0x7fc220159440 secure :-1 s=READY pgs=2530 cs=0 l=1 rev1=1 crypto rx=0x7fc218005fd0 tx=0x7fc21802bcf0 comp rx=0 tx=0).stop 2023-10-19T14:20:53.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:53.916+0000 7fc224f97640 1 -- 172.21.15.57:0/1386795944 shutdown_connections 2023-10-19T14:20:53.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:53.916+0000 7fc224f97640 1 --2- 172.21.15.57:0/1386795944 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc220069e30 0x7fc2200cdb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:53.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:53.916+0000 7fc224f97640 1 --2- 172.21.15.57:0/1386795944 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc220142850 0x7fc220159440 unknown :-1 s=CLOSED pgs=2530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:53.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:53.916+0000 7fc224f97640 1 -- 172.21.15.57:0/1386795944 >> 172.21.15.57:0/1386795944 conn(0x7fc2200af7a0 msgr2=0x7fc2200afb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:53.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:53.916+0000 7fc224f97640 1 -- 172.21.15.57:0/1386795944 shutdown_connections 2023-10-19T14:20:53.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:53.916+0000 7fc224f97640 1 -- 172.21.15.57:0/1386795944 wait complete. 2023-10-19T14:20:53.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:53.918+0000 7fc224f97640 1 Processor -- start 2023-10-19T14:20:53.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:53.918+0000 7fc224f97640 1 -- start start 2023-10-19T14:20:53.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:53.918+0000 7fc224f97640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc220069e30 0x7fc220138310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:53.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:53.918+0000 7fc224f97640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc2200cd6d0 0x7fc22014bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:53.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:53.918+0000 7fc224f97640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc220142850 0x7fc22014c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:53.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:53.918+0000 7fc224f97640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc22014a430 con 0x7fc2200cd6d0 2023-10-19T14:20:53.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:53.918+0000 7fc224f97640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc22013afd0 con 0x7fc220142850 2023-10-19T14:20:53.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:53.918+0000 7fc224f97640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc22013b2b0 con 0x7fc220069e30 2023-10-19T14:20:53.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:53.918+0000 7fc21dd9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc2200cd6d0 0x7fc22014bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:53.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:53.919+0000 7fc21dd9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc2200cd6d0 0x7fc22014bfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56154/0 (socket says 172.21.15.57:56154) 2023-10-19T14:20:53.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:53.919+0000 7fc21dd9b640 1 -- 172.21.15.57:0/3283141805 learned_addr learned my addr 172.21.15.57:0/3283141805 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:53.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:53.919+0000 7fc21e59c640 1 --2- 172.21.15.57:0/3283141805 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc220069e30 0x7fc220138310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:53.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:53.919+0000 7fc21ed9d640 1 --2- 172.21.15.57:0/3283141805 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc220142850 0x7fc22014c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:53.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:53.919+0000 7fc21dd9b640 1 -- 172.21.15.57:0/3283141805 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc220069e30 msgr2=0x7fc220138310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:53.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:53.919+0000 7fc21dd9b640 1 --2- 172.21.15.57:0/3283141805 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc220069e30 0x7fc220138310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:53.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:53.919+0000 7fc21dd9b640 1 -- 172.21.15.57:0/3283141805 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc220142850 msgr2=0x7fc22014c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:53.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:53.919+0000 7fc21dd9b640 1 --2- 172.21.15.57:0/3283141805 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc220142850 0x7fc22014c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:53.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:53.919+0000 7fc21dd9b640 1 -- 172.21.15.57:0/3283141805 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc218009560 con 0x7fc2200cd6d0 2023-10-19T14:20:53.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:53.919+0000 7fc21e59c640 1 --2- 172.21.15.57:0/3283141805 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc220069e30 0x7fc220138310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:20:53.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:53.919+0000 7fc21ed9d640 1 --2- 172.21.15.57:0/3283141805 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc220142850 0x7fc22014c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:20:53.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:53.919+0000 7fc21dd9b640 1 --2- 172.21.15.57:0/3283141805 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc2200cd6d0 0x7fc22014bfc0 secure :-1 s=READY pgs=2531 cs=0 l=1 rev1=1 crypto rx=0x7fc20800c410 tx=0x7fc20800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:53.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:53.919+0000 7fc216ffd640 1 -- 172.21.15.57:0/3283141805 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2080035f0 con 0x7fc2200cd6d0 2023-10-19T14:20:53.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:53.919+0000 7fc216ffd640 1 -- 172.21.15.57:0/3283141805 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fc208003750 con 0x7fc2200cd6d0 2023-10-19T14:20:53.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:53.919+0000 7fc224f97640 1 -- 172.21.15.57:0/3283141805 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc220159af0 con 0x7fc2200cd6d0 2023-10-19T14:20:53.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:53.919+0000 7fc216ffd640 1 -- 172.21.15.57:0/3283141805 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc208003ba0 con 0x7fc2200cd6d0 2023-10-19T14:20:53.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:53.919+0000 7fc224f97640 1 -- 172.21.15.57:0/3283141805 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc22014a0c0 con 0x7fc2200cd6d0 2023-10-19T14:20:53.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:53.920+0000 7fc224f97640 1 -- 172.21.15.57:0/3283141805 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2200690e0 con 0x7fc2200cd6d0 2023-10-19T14:20:53.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:53.921+0000 7fc216ffd640 1 -- 172.21.15.57:0/3283141805 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fc208003d00 con 0x7fc2200cd6d0 2023-10-19T14:20:53.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:53.921+0000 7fc216ffd640 1 --2- 172.21.15.57:0/3283141805 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fc20006b080 0x7fc20006d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:53.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:53.921+0000 7fc216ffd640 1 -- 172.21.15.57:0/3283141805 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(231..231 src has 1..231) v4 ==== 7386+0+0 (secure 0 0 0) 0x7fc208089960 con 0x7fc2200cd6d0 2023-10-19T14:20:53.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:53.921+0000 7fc21e59c640 1 --2- 172.21.15.57:0/3283141805 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fc20006b080 0x7fc20006d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:53.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:53.922+0000 7fc21e59c640 1 --2- 172.21.15.57:0/3283141805 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fc20006b080 0x7fc20006d540 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fc218035000 tx=0x7fc218004730 comp rx=0 tx=0).ready entity=mgr.9530 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:53.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:53.925+0000 7fc216ffd640 1 -- 172.21.15.57:0/3283141805 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc208053f90 con 0x7fc2200cd6d0 2023-10-19T14:20:54.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:54.099+0000 7fc224f97640 1 -- 172.21.15.57:0/3283141805 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc22006e640 con 0x7fc2200cd6d0 2023-10-19T14:20:54.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:54.788+0000 7fc216ffd640 1 -- 172.21.15.57:0/3283141805 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v232) v1 ==== 169+0+0 (secure 0 0 0) 0x7fc208017070 con 0x7fc2200cd6d0 2023-10-19T14:20:54.791 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'cephfs_data' removed 2023-10-19T14:20:54.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:54.790+0000 7fc224f97640 1 -- 172.21.15.57:0/3283141805 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fc20006b080 msgr2=0x7fc20006d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:54.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:54.790+0000 7fc224f97640 1 --2- 172.21.15.57:0/3283141805 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fc20006b080 0x7fc20006d540 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fc218035000 tx=0x7fc218004730 comp rx=0 tx=0).stop 2023-10-19T14:20:54.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:54.790+0000 7fc224f97640 1 -- 172.21.15.57:0/3283141805 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc2200cd6d0 msgr2=0x7fc22014bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:54.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:54.790+0000 7fc224f97640 1 --2- 172.21.15.57:0/3283141805 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc2200cd6d0 0x7fc22014bfc0 secure :-1 s=READY pgs=2531 cs=0 l=1 rev1=1 crypto rx=0x7fc20800c410 tx=0x7fc20800c8d0 comp rx=0 tx=0).stop 2023-10-19T14:20:54.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:54.790+0000 7fc224f97640 1 -- 172.21.15.57:0/3283141805 shutdown_connections 2023-10-19T14:20:54.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:54.790+0000 7fc224f97640 1 --2- 172.21.15.57:0/3283141805 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fc20006b080 0x7fc20006d540 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:54.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:54.790+0000 7fc224f97640 1 --2- 172.21.15.57:0/3283141805 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc220142850 0x7fc22014c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:54.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:54.790+0000 7fc224f97640 1 --2- 172.21.15.57:0/3283141805 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc2200cd6d0 0x7fc22014bfc0 unknown :-1 s=CLOSED pgs=2531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:54.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:54.790+0000 7fc224f97640 1 --2- 172.21.15.57:0/3283141805 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc220069e30 0x7fc220138310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:54.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:54.790+0000 7fc224f97640 1 -- 172.21.15.57:0/3283141805 >> 172.21.15.57:0/3283141805 conn(0x7fc2200af7a0 msgr2=0x7fc22013fd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:54.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:54.790+0000 7fc224f97640 1 -- 172.21.15.57:0/3283141805 shutdown_connections 2023-10-19T14:20:54.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:54.791+0000 7fc224f97640 1 -- 172.21.15.57:0/3283141805 wait complete. 2023-10-19T14:20:54.817 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T14:20:54.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:54.930+0000 7fe2cd905640 1 -- 172.21.15.57:0/168332410 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe2b40047c0 msgr2=0x7fe2b4004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:54.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:54.930+0000 7fe2cd905640 1 --2- 172.21.15.57:0/168332410 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe2b40047c0 0x7fe2b4004c40 secure :-1 s=READY pgs=2533 cs=0 l=1 rev1=1 crypto rx=0x7fe2b0009b80 tx=0x7fe2b40057f0 comp rx=0 tx=0).stop 2023-10-19T14:20:54.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:54.930+0000 7fe2c77fe640 1 -- 172.21.15.57:0/168332410 reap_dead start 2023-10-19T14:20:54.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:54.930+0000 7fe2cd905640 1 -- 172.21.15.57:0/168332410 shutdown_connections 2023-10-19T14:20:54.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:54.930+0000 7fe2cd905640 1 -- 172.21.15.57:0/168332410 >> 172.21.15.57:0/168332410 conn(0x7fe2c80d0d60 msgr2=0x7fe2c80d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:54.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:54.930+0000 7fe2cd905640 1 -- 172.21.15.57:0/168332410 shutdown_connections 2023-10-19T14:20:54.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:54.930+0000 7fe2cd905640 1 -- 172.21.15.57:0/168332410 wait complete. 2023-10-19T14:20:54.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:54.931+0000 7fe2cd905640 1 Processor -- start 2023-10-19T14:20:54.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:54.931+0000 7fe2cd905640 1 -- start start 2023-10-19T14:20:54.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:54.932+0000 7fe2cd905640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe2b40047c0 0x7fe2c8145980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:54.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:54.932+0000 7fe2cd905640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe2c80ca0b0 0x7fe2c81f4c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:54.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:54.932+0000 7fe2cd905640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe2c80ca530 0x7fe2c81f5180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:54.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:54.932+0000 7fe2cd905640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c8145ec0 con 0x7fe2c80ca0b0 2023-10-19T14:20:54.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:54.932+0000 7fe2cd905640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c815ba70 con 0x7fe2c80ca530 2023-10-19T14:20:54.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:54.932+0000 7fe2cd905640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c815bd50 con 0x7fe2b40047c0 2023-10-19T14:20:54.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:54.932+0000 7fe2c67fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe2c80ca0b0 0x7fe2c81f4c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:54.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:54.932+0000 7fe2c67fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe2c80ca0b0 0x7fe2c81f4c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56172/0 (socket says 172.21.15.57:56172) 2023-10-19T14:20:54.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:54.932+0000 7fe2c67fc640 1 -- 172.21.15.57:0/297020862 learned_addr learned my addr 172.21.15.57:0/297020862 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:54.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:54.932+0000 7fe2c77fe640 1 --2- 172.21.15.57:0/297020862 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe2c80ca530 0x7fe2c81f5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:54.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:54.932+0000 7fe2c67fc640 1 -- 172.21.15.57:0/297020862 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe2b40047c0 msgr2=0x7fe2c8145980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:54.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:54.932+0000 7fe2c6ffd640 1 --2- 172.21.15.57:0/297020862 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe2b40047c0 0x7fe2c8145980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:54.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:54.932+0000 7fe2c67fc640 1 --2- 172.21.15.57:0/297020862 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe2b40047c0 0x7fe2c8145980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:54.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:54.932+0000 7fe2c67fc640 1 -- 172.21.15.57:0/297020862 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe2c80ca530 msgr2=0x7fe2c81f5180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:54.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:54.932+0000 7fe2c67fc640 1 --2- 172.21.15.57:0/297020862 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe2c80ca530 0x7fe2c81f5180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:54.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:54.932+0000 7fe2c67fc640 1 -- 172.21.15.57:0/297020862 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2c815c030 con 0x7fe2c80ca0b0 2023-10-19T14:20:54.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:54.933+0000 7fe2c67fc640 1 --2- 172.21.15.57:0/297020862 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe2c80ca0b0 0x7fe2c81f4c40 secure :-1 s=READY pgs=2534 cs=0 l=1 rev1=1 crypto rx=0x7fe2b0009a70 tx=0x7fe2b0033cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:54.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:54.933+0000 7fe2bffff640 1 -- 172.21.15.57:0/297020862 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b003d820 con 0x7fe2c80ca0b0 2023-10-19T14:20:54.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:54.933+0000 7fe2bffff640 1 -- 172.21.15.57:0/297020862 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fe2b003d980 con 0x7fe2c80ca0b0 2023-10-19T14:20:54.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:54.933+0000 7fe2cd905640 1 -- 172.21.15.57:0/297020862 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2c815c310 con 0x7fe2c80ca0b0 2023-10-19T14:20:54.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:54.933+0000 7fe2bffff640 1 -- 172.21.15.57:0/297020862 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b003dd50 con 0x7fe2c80ca0b0 2023-10-19T14:20:54.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:54.933+0000 7fe2cd905640 1 -- 172.21.15.57:0/297020862 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2c80b4bb0 con 0x7fe2c80ca0b0 2023-10-19T14:20:54.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:54.934+0000 7fe2cd905640 1 -- 172.21.15.57:0/297020862 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2c815be90 con 0x7fe2c80ca0b0 2023-10-19T14:20:54.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:54.934+0000 7fe2bffff640 1 -- 172.21.15.57:0/297020862 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fe2b00323a0 con 0x7fe2c80ca0b0 2023-10-19T14:20:54.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:54.935+0000 7fe2bffff640 1 --2- 172.21.15.57:0/297020862 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fe2a806b080 0x7fe2a806d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:54.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:54.935+0000 7fe2bffff640 1 -- 172.21.15.57:0/297020862 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 6972+0+0 (secure 0 0 0) 0x7fe2b00acd30 con 0x7fe2c80ca0b0 2023-10-19T14:20:54.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:54.935+0000 7fe2c6ffd640 1 --2- 172.21.15.57:0/297020862 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fe2a806b080 0x7fe2a806d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:54.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:54.936+0000 7fe2c6ffd640 1 --2- 172.21.15.57:0/297020862 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fe2a806b080 0x7fe2a806d540 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe2c0009ac0 tx=0x7fe2c0005f90 comp rx=0 tx=0).ready entity=mgr.9530 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:54.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:54.939+0000 7fe2bffff640 1 -- 172.21.15.57:0/297020862 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe2c815be90 con 0x7fe2c80ca0b0 2023-10-19T14:20:55.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.114+0000 7fe2cd905640 1 -- 172.21.15.57:0/297020862 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe2c815be90 con 0x7fe2c80ca0b0 2023-10-19T14:20:55.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.116+0000 7fe2bffff640 1 -- 172.21.15.57:0/297020862 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v232) v1 ==== 74+0+18119 (secure 0 0 0) 0x7fe2c815be90 con 0x7fe2c80ca0b0 2023-10-19T14:20:55.119 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:20:55.119 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":232,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T14:20:54.777893+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_metadata","create_time":"2023-10-19T14:20:28.667664+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"226","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"backup_fs_data","create_time":"2023-10-19T14:20:29.122584+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"227","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.57:6835/1529368280":"2023-10-20T14:20:51.762660+0000","172.21.15.179:0/1513076424":"2023-10-20T14:20:48.148610+0000","172.21.15.179:0/1464950687":"2023-10-20T14:20:48.148610+0000","172.21.15.179:0/2900872065":"2023-10-20T14:20:48.148610+0000","172.21.15.57:6833/1529368280":"2023-10-20T14:20:51.762660+0000","172.21.15.179:0/2140451434":"2023-10-20T14:20:48.148610+0000","172.21.15.179:0/2605624080":"2023-10-20T14:20:48.148610+0000","172.21.15.179:0/1194984394":"2023-10-20T14:20:39.702634+0000","172.21.15.179:6833/3232482678":"2023-10-20T14:20:39.702634+0000","172.21.15.179:6832/4242302683":"2023-10-20T14:20:48.148610+0000","172.21.15.179:6832/3232482678":"2023-10-20T14:20:39.702634+0000","172.21.15.179:0/2872579179":"2023-10-20T14:20:39.702634+0000","172.21.15.179:0/373323540":"2023-10-20T14:20:48.148610+0000","172.21.15.179:0/3910604357":"2023-10-20T14:20:39.702634+0000","172.21.15.179:0/4185855767":"2023-10-20T14:20:48.148610+0000","172.21.15.179:0/536160557":"2023-10-20T14:20:39.702634+0000","172.21.15.179:0/3239062050":"2023-10-20T14:20:39.702634+0000","172.21.15.179:6833/4242302683":"2023-10-20T14:20:48.148610+0000","172.21.15.179:0/3672528294":"2023-10-20T14:20:48.148610+0000","172.21.15.179:0/2003826253":"2023-10-20T14:20:48.148610+0000","172.21.15.179:0/1948632334":"2023-10-20T14:20:39.702634+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T14:20:55.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.117+0000 7fe2cd905640 1 -- 172.21.15.57:0/297020862 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fe2a806b080 msgr2=0x7fe2a806d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:55.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.117+0000 7fe2cd905640 1 --2- 172.21.15.57:0/297020862 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fe2a806b080 0x7fe2a806d540 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe2c0009ac0 tx=0x7fe2c0005f90 comp rx=0 tx=0).stop 2023-10-19T14:20:55.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.117+0000 7fe2cd905640 1 -- 172.21.15.57:0/297020862 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe2c80ca0b0 msgr2=0x7fe2c81f4c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:55.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.117+0000 7fe2cd905640 1 --2- 172.21.15.57:0/297020862 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe2c80ca0b0 0x7fe2c81f4c40 secure :-1 s=READY pgs=2534 cs=0 l=1 rev1=1 crypto rx=0x7fe2b0009a70 tx=0x7fe2b0033cd0 comp rx=0 tx=0).stop 2023-10-19T14:20:55.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.117+0000 7fe2cd905640 1 -- 172.21.15.57:0/297020862 shutdown_connections 2023-10-19T14:20:55.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.117+0000 7fe2cd905640 1 --2- 172.21.15.57:0/297020862 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fe2a806b080 0x7fe2a806d540 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:55.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.117+0000 7fe2cd905640 1 --2- 172.21.15.57:0/297020862 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe2c80ca530 0x7fe2c81f5180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:55.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.117+0000 7fe2cd905640 1 --2- 172.21.15.57:0/297020862 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe2c80ca0b0 0x7fe2c81f4c40 unknown :-1 s=CLOSED pgs=2534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:55.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.117+0000 7fe2cd905640 1 --2- 172.21.15.57:0/297020862 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe2b40047c0 0x7fe2c8145980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:55.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.118+0000 7fe2cd905640 1 -- 172.21.15.57:0/297020862 >> 172.21.15.57:0/297020862 conn(0x7fe2c80d0d60 msgr2=0x7fe2c80ad8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:55.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.118+0000 7fe2cd905640 1 -- 172.21.15.57:0/297020862 shutdown_connections 2023-10-19T14:20:55.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.118+0000 7fe2cd905640 1 -- 172.21.15.57:0/297020862 wait complete. 2023-10-19T14:20:55.132 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-19T12:44:07.371186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 29, 'pool_name': 'backup_fs_metadata', 'create_time': '2023-10-19T14:20:28.667664+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '226', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 30, 'pool_name': 'backup_fs_data', 'create_time': '2023-10-19T14:20:29.122584+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '227', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-19T14:20:55.133 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-19T14:20:55.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.244+0000 7f82c687c640 1 -- 172.21.15.57:0/226316478 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f82a8005380 msgr2=0x7f82a800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:55.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.244+0000 7f82c687c640 1 --2- 172.21.15.57:0/226316478 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f82a8005380 0x7f82a800b980 secure :-1 s=READY pgs=2536 cs=0 l=1 rev1=1 crypto rx=0x7f82b800c640 tx=0x7f82a80057f0 comp rx=0 tx=0).stop 2023-10-19T14:20:55.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.244+0000 7f82c687c640 1 -- 172.21.15.57:0/226316478 shutdown_connections 2023-10-19T14:20:55.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.244+0000 7f82c687c640 1 --2- 172.21.15.57:0/226316478 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f82a8005380 0x7f82a800b980 unknown :-1 s=CLOSED pgs=2536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:55.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.244+0000 7f82c687c640 1 -- 172.21.15.57:0/226316478 >> 172.21.15.57:0/226316478 conn(0x7f82c00d0d60 msgr2=0x7f82c00d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:55.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.244+0000 7f82c687c640 1 -- 172.21.15.57:0/226316478 shutdown_connections 2023-10-19T14:20:55.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.244+0000 7f82c687c640 1 -- 172.21.15.57:0/226316478 wait complete. 2023-10-19T14:20:55.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.245+0000 7f82c687c640 1 Processor -- start 2023-10-19T14:20:55.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.245+0000 7f82c687c640 1 -- start start 2023-10-19T14:20:55.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.246+0000 7f82c687c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f82a8005380 0x7f82c0153de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:55.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.246+0000 7f82c687c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f82c00c8d70 0x7f82c01549e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:55.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.246+0000 7f82c687c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f82c00ae3c0 0x7f82c00ae840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:55.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.246+0000 7f82c687c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82c00aed80 con 0x7f82a8005380 2023-10-19T14:20:55.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.246+0000 7f82c687c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82c015c860 con 0x7f82c00ae3c0 2023-10-19T14:20:55.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.246+0000 7f82c687c640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82c015d6c0 con 0x7f82c00c8d70 2023-10-19T14:20:55.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.246+0000 7f82bffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f82a8005380 0x7f82c0153de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:55.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.246+0000 7f82bffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f82a8005380 0x7f82c0153de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56180/0 (socket says 172.21.15.57:56180) 2023-10-19T14:20:55.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.246+0000 7f82bffff640 1 -- 172.21.15.57:0/975743476 learned_addr learned my addr 172.21.15.57:0/975743476 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:55.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.246+0000 7f82c4e19640 1 --2- 172.21.15.57:0/975743476 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f82c00ae3c0 0x7f82c00ae840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:55.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.246+0000 7f82bf7fe640 1 --2- 172.21.15.57:0/975743476 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f82c00c8d70 0x7f82c01549e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:55.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.246+0000 7f82bffff640 1 -- 172.21.15.57:0/975743476 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f82c00c8d70 msgr2=0x7f82c01549e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:55.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.246+0000 7f82bffff640 1 --2- 172.21.15.57:0/975743476 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f82c00c8d70 0x7f82c01549e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:55.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.246+0000 7f82bffff640 1 -- 172.21.15.57:0/975743476 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f82c00ae3c0 msgr2=0x7f82c00ae840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:55.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.246+0000 7f82bffff640 1 --2- 172.21.15.57:0/975743476 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f82c00ae3c0 0x7f82c00ae840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:55.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.246+0000 7f82bffff640 1 -- 172.21.15.57:0/975743476 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82c0154320 con 0x7f82a8005380 2023-10-19T14:20:55.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.246+0000 7f82bffff640 1 --2- 172.21.15.57:0/975743476 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f82a8005380 0x7f82c0153de0 secure :-1 s=READY pgs=2537 cs=0 l=1 rev1=1 crypto rx=0x7f82b400c400 tx=0x7f82b400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:55.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.247+0000 7f82bcff9640 1 -- 172.21.15.57:0/975743476 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82b40052d0 con 0x7f82a8005380 2023-10-19T14:20:55.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.247+0000 7f82bcff9640 1 -- 172.21.15.57:0/975743476 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f82b40035f0 con 0x7f82a8005380 2023-10-19T14:20:55.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.247+0000 7f82c687c640 1 -- 172.21.15.57:0/975743476 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82c015cab0 con 0x7f82a8005380 2023-10-19T14:20:55.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.247+0000 7f82bcff9640 1 -- 172.21.15.57:0/975743476 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82b4003a00 con 0x7f82a8005380 2023-10-19T14:20:55.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.247+0000 7f82c687c640 1 -- 172.21.15.57:0/975743476 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82c015d070 con 0x7f82a8005380 2023-10-19T14:20:55.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.248+0000 7f82c687c640 1 -- 172.21.15.57:0/975743476 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82c00690e0 con 0x7f82a8005380 2023-10-19T14:20:55.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.248+0000 7f82bcff9640 1 -- 172.21.15.57:0/975743476 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f82b4003b60 con 0x7f82a8005380 2023-10-19T14:20:55.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.248+0000 7f82bcff9640 1 --2- 172.21.15.57:0/975743476 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f82ac06b080 0x7f82ac06d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:55.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.249+0000 7f82bcff9640 1 -- 172.21.15.57:0/975743476 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 6972+0+0 (secure 0 0 0) 0x7f82b40896d0 con 0x7f82a8005380 2023-10-19T14:20:55.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.249+0000 7f82bf7fe640 1 --2- 172.21.15.57:0/975743476 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f82ac06b080 0x7f82ac06d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:55.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.249+0000 7f82bf7fe640 1 --2- 172.21.15.57:0/975743476 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f82ac06b080 0x7f82ac06d540 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f82c0157ca0 tx=0x7f82b0008040 comp rx=0 tx=0).ready entity=mgr.9530 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:55.262 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.253+0000 7f82bcff9640 1 -- 172.21.15.57:0/975743476 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f82b4054000 con 0x7f82a8005380 2023-10-19T14:20:55.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.432+0000 7f82c687c640 1 -- 172.21.15.57:0/975743476 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f82c00d0b40 con 0x7f82a8005380 2023-10-19T14:20:55.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.432+0000 7f82bcff9640 1 -- 172.21.15.57:0/975743476 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v232) v1 ==== 93+0+10 (secure 0 0 0) 0x7f82b4057620 con 0x7f82a8005380 2023-10-19T14:20:55.435 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-19T14:20:55.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.434+0000 7f82c687c640 1 -- 172.21.15.57:0/975743476 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f82ac06b080 msgr2=0x7f82ac06d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:55.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.434+0000 7f82c687c640 1 --2- 172.21.15.57:0/975743476 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f82ac06b080 0x7f82ac06d540 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f82c0157ca0 tx=0x7f82b0008040 comp rx=0 tx=0).stop 2023-10-19T14:20:55.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.434+0000 7f82c687c640 1 -- 172.21.15.57:0/975743476 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f82a8005380 msgr2=0x7f82c0153de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:55.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.434+0000 7f82c687c640 1 --2- 172.21.15.57:0/975743476 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f82a8005380 0x7f82c0153de0 secure :-1 s=READY pgs=2537 cs=0 l=1 rev1=1 crypto rx=0x7f82b400c400 tx=0x7f82b400c8c0 comp rx=0 tx=0).stop 2023-10-19T14:20:55.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.434+0000 7f82c687c640 1 -- 172.21.15.57:0/975743476 shutdown_connections 2023-10-19T14:20:55.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.434+0000 7f82c687c640 1 --2- 172.21.15.57:0/975743476 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f82ac06b080 0x7f82ac06d540 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:55.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.434+0000 7f82c687c640 1 --2- 172.21.15.57:0/975743476 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f82c00ae3c0 0x7f82c00ae840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:55.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.434+0000 7f82c687c640 1 --2- 172.21.15.57:0/975743476 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f82c00c8d70 0x7f82c01549e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:55.439 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.434+0000 7f82c687c640 1 --2- 172.21.15.57:0/975743476 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f82a8005380 0x7f82c0153de0 unknown :-1 s=CLOSED pgs=2537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:55.439 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.434+0000 7f82c687c640 1 -- 172.21.15.57:0/975743476 >> 172.21.15.57:0/975743476 conn(0x7f82c00d0d60 msgr2=0x7f82c00ad8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:55.439 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.434+0000 7f82c687c640 1 -- 172.21.15.57:0/975743476 shutdown_connections 2023-10-19T14:20:55.439 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.434+0000 7f82c687c640 1 -- 172.21.15.57:0/975743476 wait complete. 2023-10-19T14:20:55.447 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2023-10-19T14:20:55.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.556+0000 7f39461e7640 1 -- 172.21.15.57:0/3905939672 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f394015b2a0 msgr2=0x7f39400d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:55.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.556+0000 7f39461e7640 1 --2- 172.21.15.57:0/3905939672 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f394015b2a0 0x7f39400d1910 secure :-1 s=READY pgs=2539 cs=0 l=1 rev1=1 crypto rx=0x7f3934009830 tx=0x7f393402bcd0 comp rx=0 tx=0).stop 2023-10-19T14:20:55.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.556+0000 7f39461e7640 1 -- 172.21.15.57:0/3905939672 shutdown_connections 2023-10-19T14:20:55.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.556+0000 7f39461e7640 1 --2- 172.21.15.57:0/3905939672 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f394015b2a0 0x7f39400d1910 unknown :-1 s=CLOSED pgs=2539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:55.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.556+0000 7f39461e7640 1 -- 172.21.15.57:0/3905939672 >> 172.21.15.57:0/3905939672 conn(0x7f39400d0d60 msgr2=0x7f39400d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:55.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.556+0000 7f39461e7640 1 -- 172.21.15.57:0/3905939672 shutdown_connections 2023-10-19T14:20:55.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.556+0000 7f39461e7640 1 -- 172.21.15.57:0/3905939672 wait complete. 2023-10-19T14:20:55.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.557+0000 7f39461e7640 1 Processor -- start 2023-10-19T14:20:55.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.557+0000 7f39461e7640 1 -- start start 2023-10-19T14:20:55.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.558+0000 7f39461e7640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f39400ceb40 0x7f394014f250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:55.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.558+0000 7f39461e7640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f39400cf340 0x7f39400c8520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:55.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.558+0000 7f39461e7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f394015b2a0 0x7f39400ae3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:55.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.558+0000 7f39461e7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f394014f790 con 0x7f394015b2a0 2023-10-19T14:20:55.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.558+0000 7f39461e7640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f394015ba20 con 0x7f39400cf340 2023-10-19T14:20:55.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.558+0000 7f39461e7640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f394015bd00 con 0x7f39400ceb40 2023-10-19T14:20:55.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.558+0000 7f393ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f394015b2a0 0x7f39400ae3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:55.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.558+0000 7f393ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f394015b2a0 0x7f39400ae3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56204/0 (socket says 172.21.15.57:56204) 2023-10-19T14:20:55.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.558+0000 7f393ffff640 1 -- 172.21.15.57:0/2247739246 learned_addr learned my addr 172.21.15.57:0/2247739246 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:55.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.558+0000 7f393ffff640 1 -- 172.21.15.57:0/2247739246 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f39400ceb40 msgr2=0x7f394014f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:55.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.558+0000 7f393ffff640 1 --2- 172.21.15.57:0/2247739246 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f39400ceb40 0x7f394014f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:55.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.558+0000 7f393ffff640 1 -- 172.21.15.57:0/2247739246 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f39400cf340 msgr2=0x7f39400c8520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:55.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.558+0000 7f393ffff640 1 --2- 172.21.15.57:0/2247739246 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f39400cf340 0x7f39400c8520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:55.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.558+0000 7f393ffff640 1 -- 172.21.15.57:0/2247739246 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3934009490 con 0x7f394015b2a0 2023-10-19T14:20:55.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.559+0000 7f393ffff640 1 --2- 172.21.15.57:0/2247739246 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f394015b2a0 0x7f39400ae3c0 secure :-1 s=READY pgs=2540 cs=0 l=1 rev1=1 crypto rx=0x7f393800c780 tx=0x7f393800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:55.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.559+0000 7f3927fff640 1 -- 172.21.15.57:0/2247739246 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3938004b60 con 0x7f394015b2a0 2023-10-19T14:20:55.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.559+0000 7f3927fff640 1 -- 172.21.15.57:0/2247739246 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f393800ce90 con 0x7f394015b2a0 2023-10-19T14:20:55.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.559+0000 7f39461e7640 1 -- 172.21.15.57:0/2247739246 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f394015bfe0 con 0x7f394015b2a0 2023-10-19T14:20:55.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.559+0000 7f3927fff640 1 -- 172.21.15.57:0/2247739246 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f393800aab0 con 0x7f394015b2a0 2023-10-19T14:20:55.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.559+0000 7f39461e7640 1 -- 172.21.15.57:0/2247739246 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f394015c5a0 con 0x7f394015b2a0 2023-10-19T14:20:55.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.560+0000 7f39461e7640 1 -- 172.21.15.57:0/2247739246 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39400690e0 con 0x7f394015b2a0 2023-10-19T14:20:55.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.560+0000 7f3927fff640 1 -- 172.21.15.57:0/2247739246 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f393800ac10 con 0x7f394015b2a0 2023-10-19T14:20:55.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.561+0000 7f3927fff640 1 --2- 172.21.15.57:0/2247739246 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f392c06b080 0x7f392c06d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:55.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.561+0000 7f3927fff640 1 -- 172.21.15.57:0/2247739246 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 6972+0+0 (secure 0 0 0) 0x7f393808a2b0 con 0x7f394015b2a0 2023-10-19T14:20:55.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.561+0000 7f393f7fe640 1 --2- 172.21.15.57:0/2247739246 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f392c06b080 0x7f392c06d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:55.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.562+0000 7f393f7fe640 1 --2- 172.21.15.57:0/2247739246 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f392c06b080 0x7f392c06d540 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f39400c8180 tx=0x7f3930007910 comp rx=0 tx=0).ready entity=mgr.9530 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:55.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.565+0000 7f3927fff640 1 -- 172.21.15.57:0/2247739246 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3938054ba0 con 0x7f394015b2a0 2023-10-19T14:20:55.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.741+0000 7f39461e7640 1 -- 172.21.15.57:0/2247739246 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f3940157c00 con 0x7f394015b2a0 2023-10-19T14:20:55.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.741+0000 7f3927fff640 1 -- 172.21.15.57:0/2247739246 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v232) v1 ==== 107+0+11 (secure 0 0 0) 0x7f39380581c0 con 0x7f394015b2a0 2023-10-19T14:20:55.745 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2023-10-19T14:20:55.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.743+0000 7f39461e7640 1 -- 172.21.15.57:0/2247739246 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f392c06b080 msgr2=0x7f392c06d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:55.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.743+0000 7f39461e7640 1 --2- 172.21.15.57:0/2247739246 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f392c06b080 0x7f392c06d540 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f39400c8180 tx=0x7f3930007910 comp rx=0 tx=0).stop 2023-10-19T14:20:55.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.743+0000 7f39461e7640 1 -- 172.21.15.57:0/2247739246 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f394015b2a0 msgr2=0x7f39400ae3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:55.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.743+0000 7f39461e7640 1 --2- 172.21.15.57:0/2247739246 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f394015b2a0 0x7f39400ae3c0 secure :-1 s=READY pgs=2540 cs=0 l=1 rev1=1 crypto rx=0x7f393800c780 tx=0x7f393800cc40 comp rx=0 tx=0).stop 2023-10-19T14:20:55.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.743+0000 7f39461e7640 1 -- 172.21.15.57:0/2247739246 shutdown_connections 2023-10-19T14:20:55.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.743+0000 7f39461e7640 1 --2- 172.21.15.57:0/2247739246 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f392c06b080 0x7f392c06d540 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:55.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.743+0000 7f39461e7640 1 --2- 172.21.15.57:0/2247739246 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f394015b2a0 0x7f39400ae3c0 unknown :-1 s=CLOSED pgs=2540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:55.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.743+0000 7f39461e7640 1 --2- 172.21.15.57:0/2247739246 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f39400cf340 0x7f39400c8520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:55.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.743+0000 7f39461e7640 1 --2- 172.21.15.57:0/2247739246 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f39400ceb40 0x7f394014f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:55.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.743+0000 7f39461e7640 1 -- 172.21.15.57:0/2247739246 >> 172.21.15.57:0/2247739246 conn(0x7f39400d0d60 msgr2=0x7f394006a8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:55.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.743+0000 7f39461e7640 1 -- 172.21.15.57:0/2247739246 shutdown_connections 2023-10-19T14:20:55.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.743+0000 7f39461e7640 1 -- 172.21.15.57:0/2247739246 wait complete. 2023-10-19T14:20:55.757 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2023-10-19T14:20:55.869 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.866+0000 7f5514d39640 1 -- 172.21.15.57:0/336444968 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f55101579f0 msgr2=0x7f55100b0e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:55.869 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.866+0000 7f5514d39640 1 --2- 172.21.15.57:0/336444968 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f55101579f0 0x7f55100b0e90 secure :-1 s=READY pgs=2542 cs=0 l=1 rev1=1 crypto rx=0x7f551006b930 tx=0x7f550002bd70 comp rx=0 tx=0).stop 2023-10-19T14:20:55.869 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.866+0000 7f5514d39640 1 -- 172.21.15.57:0/336444968 shutdown_connections 2023-10-19T14:20:55.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.866+0000 7f5514d39640 1 --2- 172.21.15.57:0/336444968 >> v2:172.21.15.179:3300/0 conn(0x7f551015c1f0 0x7f551006a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:55.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.866+0000 7f5514d39640 1 --2- 172.21.15.57:0/336444968 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f55101579f0 0x7f55100b0e90 unknown :-1 s=CLOSED pgs=2542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:55.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.866+0000 7f5514d39640 1 -- 172.21.15.57:0/336444968 >> 172.21.15.57:0/336444968 conn(0x7f551006fae0 msgr2=0x7f551006fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:55.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.866+0000 7f5514d39640 1 -- 172.21.15.57:0/336444968 shutdown_connections 2023-10-19T14:20:55.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.866+0000 7f5514d39640 1 -- 172.21.15.57:0/336444968 wait complete. 2023-10-19T14:20:55.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.867+0000 7f5514d39640 1 Processor -- start 2023-10-19T14:20:55.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.867+0000 7f5514d39640 1 -- start start 2023-10-19T14:20:55.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.868+0000 7f5514d39640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f551006ac70 0x7f551014adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:55.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.868+0000 7f5514d39640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f55101579f0 0x7f551014b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:55.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.868+0000 7f5514d39640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f551015c1f0 0x7f5510147910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:55.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.868+0000 7f5514d39640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f551014bc60 con 0x7f551015c1f0 2023-10-19T14:20:55.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.868+0000 7f5514d39640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5510150ec0 con 0x7f55101579f0 2023-10-19T14:20:55.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.868+0000 7f5514d39640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5510151180 con 0x7f551006ac70 2023-10-19T14:20:55.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.868+0000 7f550ed9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f551015c1f0 0x7f5510147910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:55.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.868+0000 7f550ed9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f551015c1f0 0x7f5510147910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56220/0 (socket says 172.21.15.57:56220) 2023-10-19T14:20:55.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.868+0000 7f550ed9d640 1 -- 172.21.15.57:0/1864151393 learned_addr learned my addr 172.21.15.57:0/1864151393 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:55.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.868+0000 7f5507fff640 1 --2- 172.21.15.57:0/1864151393 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f55101579f0 0x7f551014b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:55.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.868+0000 7f550e59c640 1 --2- 172.21.15.57:0/1864151393 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f551006ac70 0x7f551014adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:55.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.868+0000 7f550ed9d640 1 -- 172.21.15.57:0/1864151393 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f551006ac70 msgr2=0x7f551014adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:55.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.868+0000 7f550ed9d640 1 --2- 172.21.15.57:0/1864151393 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f551006ac70 0x7f551014adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:55.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.868+0000 7f550ed9d640 1 -- 172.21.15.57:0/1864151393 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f55101579f0 msgr2=0x7f551014b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:55.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.868+0000 7f550ed9d640 1 --2- 172.21.15.57:0/1864151393 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f55101579f0 0x7f551014b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:55.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.868+0000 7f550ed9d640 1 -- 172.21.15.57:0/1864151393 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5500009490 con 0x7f551015c1f0 2023-10-19T14:20:55.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.868+0000 7f550e59c640 1 --2- 172.21.15.57:0/1864151393 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f551006ac70 0x7f551014adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:20:55.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.868+0000 7f550ed9d640 1 --2- 172.21.15.57:0/1864151393 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f551015c1f0 0x7f5510147910 secure :-1 s=READY pgs=2543 cs=0 l=1 rev1=1 crypto rx=0x7f550800cb00 tx=0x7f550800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:55.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.868+0000 7f5506ffd640 1 -- 172.21.15.57:0/1864151393 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f550800aca0 con 0x7f551015c1f0 2023-10-19T14:20:55.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.868+0000 7f5506ffd640 1 -- 172.21.15.57:0/1864151393 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f5508003970 con 0x7f551015c1f0 2023-10-19T14:20:55.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.868+0000 7f5514d39640 1 -- 172.21.15.57:0/1864151393 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5510151460 con 0x7f551015c1f0 2023-10-19T14:20:55.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.868+0000 7f5506ffd640 1 -- 172.21.15.57:0/1864151393 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5508003d80 con 0x7f551015c1f0 2023-10-19T14:20:55.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.869+0000 7f5514d39640 1 -- 172.21.15.57:0/1864151393 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5510147e50 con 0x7f551015c1f0 2023-10-19T14:20:55.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.870+0000 7f5514d39640 1 -- 172.21.15.57:0/1864151393 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f551014e330 con 0x7f551015c1f0 2023-10-19T14:20:55.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.870+0000 7f5506ffd640 1 -- 172.21.15.57:0/1864151393 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f5508004030 con 0x7f551015c1f0 2023-10-19T14:20:55.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.870+0000 7f5506ffd640 1 --2- 172.21.15.57:0/1864151393 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f54f006b080 0x7f54f006d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:55.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.870+0000 7f5506ffd640 1 -- 172.21.15.57:0/1864151393 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 6972+0+0 (secure 0 0 0) 0x7f550808a8f0 con 0x7f551015c1f0 2023-10-19T14:20:55.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.870+0000 7f550e59c640 1 --2- 172.21.15.57:0/1864151393 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f54f006b080 0x7f54f006d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:55.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.871+0000 7f550e59c640 1 --2- 172.21.15.57:0/1864151393 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f54f006b080 0x7f54f006d540 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5500036000 tx=0x7f5500004730 comp rx=0 tx=0).ready entity=mgr.9530 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:55.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:55.874+0000 7f5506ffd640 1 -- 172.21.15.57:0/1864151393 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f551014e330 con 0x7f551015c1f0 2023-10-19T14:20:56.055 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.052+0000 7f5514d39640 1 -- 172.21.15.57:0/1864151393 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) v1 -- 0x7f5510147fe0 con 0x7f551015c1f0 2023-10-19T14:20:56.056 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.053+0000 7f5506ffd640 1 -- 172.21.15.57:0/1864151393 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v232) v1 ==== 103+0+11 (secure 0 0 0) 0x7f5510147fe0 con 0x7f551015c1f0 2023-10-19T14:20:56.056 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 64 2023-10-19T14:20:56.058 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.055+0000 7f5514d39640 1 -- 172.21.15.57:0/1864151393 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f54f006b080 msgr2=0x7f54f006d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:56.058 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.055+0000 7f5514d39640 1 --2- 172.21.15.57:0/1864151393 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f54f006b080 0x7f54f006d540 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5500036000 tx=0x7f5500004730 comp rx=0 tx=0).stop 2023-10-19T14:20:56.058 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.055+0000 7f5514d39640 1 -- 172.21.15.57:0/1864151393 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f551015c1f0 msgr2=0x7f5510147910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:56.058 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.055+0000 7f5514d39640 1 --2- 172.21.15.57:0/1864151393 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f551015c1f0 0x7f5510147910 secure :-1 s=READY pgs=2543 cs=0 l=1 rev1=1 crypto rx=0x7f550800cb00 tx=0x7f550800a060 comp rx=0 tx=0).stop 2023-10-19T14:20:56.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.055+0000 7f5514d39640 1 -- 172.21.15.57:0/1864151393 shutdown_connections 2023-10-19T14:20:56.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.055+0000 7f5514d39640 1 --2- 172.21.15.57:0/1864151393 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f54f006b080 0x7f54f006d540 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:56.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.055+0000 7f5514d39640 1 --2- 172.21.15.57:0/1864151393 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f551015c1f0 0x7f5510147910 unknown :-1 s=CLOSED pgs=2543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:56.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.055+0000 7f5514d39640 1 --2- 172.21.15.57:0/1864151393 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f55101579f0 0x7f551014b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:56.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.055+0000 7f5514d39640 1 --2- 172.21.15.57:0/1864151393 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f551006ac70 0x7f551014adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:56.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.055+0000 7f5514d39640 1 -- 172.21.15.57:0/1864151393 >> 172.21.15.57:0/1864151393 conn(0x7f551006fae0 msgr2=0x7f55100b0250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:56.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.055+0000 7f5514d39640 1 -- 172.21.15.57:0/1864151393 shutdown_connections 2023-10-19T14:20:56.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.055+0000 7f5514d39640 1 -- 172.21.15.57:0/1864151393 wait complete. 2023-10-19T14:20:56.069 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T14:20:56.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.174+0000 7fa944d0c640 1 -- 172.21.15.57:0/3604173051 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa9400cbd20 msgr2=0x7fa940158300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:56.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.174+0000 7fa944d0c640 1 --2- 172.21.15.57:0/3604173051 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa9400cbd20 0x7fa940158300 secure :-1 s=READY pgs=2544 cs=0 l=1 rev1=1 crypto rx=0x7fa9400cc490 tx=0x7fa93002bd90 comp rx=0 tx=0).stop 2023-10-19T14:20:56.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.174+0000 7fa944d0c640 1 -- 172.21.15.57:0/3604173051 shutdown_connections 2023-10-19T14:20:56.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.174+0000 7fa944d0c640 1 --2- 172.21.15.57:0/3604173051 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa9400b0530 0x7fa9400b09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:56.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.174+0000 7fa944d0c640 1 --2- 172.21.15.57:0/3604173051 >> v2:172.21.15.179:3300/0 conn(0x7fa9400afa20 0x7fa9400705a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:56.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.174+0000 7fa944d0c640 1 --2- 172.21.15.57:0/3604173051 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa9400cbd20 0x7fa940158300 unknown :-1 s=CLOSED pgs=2544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:56.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.174+0000 7fa944d0c640 1 -- 172.21.15.57:0/3604173051 >> 172.21.15.57:0/3604173051 conn(0x7fa940069930 msgr2=0x7fa940069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:56.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.174+0000 7fa944d0c640 1 -- 172.21.15.57:0/3604173051 shutdown_connections 2023-10-19T14:20:56.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.174+0000 7fa944d0c640 1 -- 172.21.15.57:0/3604173051 wait complete. 2023-10-19T14:20:56.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.176+0000 7fa944d0c640 1 Processor -- start 2023-10-19T14:20:56.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.176+0000 7fa944d0c640 1 -- start start 2023-10-19T14:20:56.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.176+0000 7fa944d0c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa9400afa20 0x7fa9400adbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:56.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.176+0000 7fa944d0c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa9400b0530 0x7fa940142430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:56.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.176+0000 7fa944d0c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa9400cbd20 0x7fa94014ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:56.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.176+0000 7fa944d0c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9400ad7d0 con 0x7fa9400b0530 2023-10-19T14:20:56.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.176+0000 7fa944d0c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa940142970 con 0x7fa9400afa20 2023-10-19T14:20:56.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.176+0000 7fa944d0c640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9400acb50 con 0x7fa9400cbd20 2023-10-19T14:20:56.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.177+0000 7fa93dd9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa9400b0530 0x7fa940142430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:56.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.177+0000 7fa93e59c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa9400afa20 0x7fa9400adbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:56.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.177+0000 7fa93dd9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa9400b0530 0x7fa940142430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56248/0 (socket says 172.21.15.57:56248) 2023-10-19T14:20:56.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.177+0000 7fa93dd9b640 1 -- 172.21.15.57:0/388039476 learned_addr learned my addr 172.21.15.57:0/388039476 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:56.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.177+0000 7fa93ed9d640 1 --2- 172.21.15.57:0/388039476 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa9400cbd20 0x7fa94014ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:56.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.177+0000 7fa93dd9b640 1 -- 172.21.15.57:0/388039476 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa9400cbd20 msgr2=0x7fa94014ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:56.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.177+0000 7fa93dd9b640 1 --2- 172.21.15.57:0/388039476 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa9400cbd20 0x7fa94014ada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:56.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.177+0000 7fa93dd9b640 1 -- 172.21.15.57:0/388039476 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa9400afa20 msgr2=0x7fa9400adbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:56.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.177+0000 7fa93dd9b640 1 --2- 172.21.15.57:0/388039476 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa9400afa20 0x7fa9400adbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:56.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.177+0000 7fa93dd9b640 1 -- 172.21.15.57:0/388039476 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa930009560 con 0x7fa9400b0530 2023-10-19T14:20:56.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.177+0000 7fa93dd9b640 1 --2- 172.21.15.57:0/388039476 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa9400b0530 0x7fa940142430 secure :-1 s=READY pgs=2545 cs=0 l=1 rev1=1 crypto rx=0x7fa93400d770 tx=0x7fa93400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:56.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.177+0000 7fa92effd640 1 -- 172.21.15.57:0/388039476 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa934009c30 con 0x7fa9400b0530 2023-10-19T14:20:56.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.177+0000 7fa92effd640 1 -- 172.21.15.57:0/388039476 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fa934009e80 con 0x7fa9400b0530 2023-10-19T14:20:56.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.177+0000 7fa944d0c640 1 -- 172.21.15.57:0/388039476 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9400ace30 con 0x7fa9400b0530 2023-10-19T14:20:56.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.177+0000 7fa92effd640 1 -- 172.21.15.57:0/388039476 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9340106a0 con 0x7fa9400b0530 2023-10-19T14:20:56.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.177+0000 7fa944d0c640 1 -- 172.21.15.57:0/388039476 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9400ad3f0 con 0x7fa9400b0530 2023-10-19T14:20:56.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.178+0000 7fa944d0c640 1 -- 172.21.15.57:0/388039476 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa94015abb0 con 0x7fa9400b0530 2023-10-19T14:20:56.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.179+0000 7fa92effd640 1 -- 172.21.15.57:0/388039476 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fa934010800 con 0x7fa9400b0530 2023-10-19T14:20:56.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.179+0000 7fa92effd640 1 --2- 172.21.15.57:0/388039476 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fa92006b080 0x7fa92006d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:56.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.179+0000 7fa92effd640 1 -- 172.21.15.57:0/388039476 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 6972+0+0 (secure 0 0 0) 0x7fa93408a460 con 0x7fa9400b0530 2023-10-19T14:20:56.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.180+0000 7fa93e59c640 1 --2- 172.21.15.57:0/388039476 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fa92006b080 0x7fa92006d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:56.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.180+0000 7fa93e59c640 1 --2- 172.21.15.57:0/388039476 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fa92006b080 0x7fa92006d540 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa930035000 tx=0x7fa930004730 comp rx=0 tx=0).ready entity=mgr.9530 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:56.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.184+0000 7fa92effd640 1 -- 172.21.15.57:0/388039476 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa934054c70 con 0x7fa9400b0530 2023-10-19T14:20:56.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.399+0000 7fa944d0c640 1 -- 172.21.15.57:0/388039476 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa9400c96b0 con 0x7fa9400b0530 2023-10-19T14:20:56.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.400+0000 7fa92effd640 1 -- 172.21.15.57:0/388039476 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+4931 (secure 0 0 0) 0x7fa934058290 con 0x7fa9400b0530 2023-10-19T14:20:56.403 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:20:56.404 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":96,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9248,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/1249807745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":1249807745},{"type":"v1","addr":"172.21.15.179:6836","nonce":1249807745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87},{"gid":10513,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/1406825818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":1406825818},{"type":"v1","addr":"172.21.15.179:6837","nonce":1406825818}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":86},{"gid":12375,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/2367396561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2367396561},{"type":"v1","addr":"172.21.15.57:6835","nonce":2367396561}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T14:20:30.145135+0000","modified":"2023-10-19T14:20:32.141937+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11715},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11715":{"gid":11715,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":5,"addr":"172.21.15.57:6834/2467807803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2467807803},{"type":"v1","addr":"172.21.15.57:6834","nonce":2467807803}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2023-10-19T14:20:56.404 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 96 2023-10-19T14:20:56.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.402+0000 7fa944d0c640 1 -- 172.21.15.57:0/388039476 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fa92006b080 msgr2=0x7fa92006d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:56.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.402+0000 7fa944d0c640 1 --2- 172.21.15.57:0/388039476 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fa92006b080 0x7fa92006d540 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa930035000 tx=0x7fa930004730 comp rx=0 tx=0).stop 2023-10-19T14:20:56.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.402+0000 7fa944d0c640 1 -- 172.21.15.57:0/388039476 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa9400b0530 msgr2=0x7fa940142430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:56.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.402+0000 7fa944d0c640 1 --2- 172.21.15.57:0/388039476 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa9400b0530 0x7fa940142430 secure :-1 s=READY pgs=2545 cs=0 l=1 rev1=1 crypto rx=0x7fa93400d770 tx=0x7fa93400dc30 comp rx=0 tx=0).stop 2023-10-19T14:20:56.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.402+0000 7fa944d0c640 1 -- 172.21.15.57:0/388039476 shutdown_connections 2023-10-19T14:20:56.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.402+0000 7fa944d0c640 1 --2- 172.21.15.57:0/388039476 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fa92006b080 0x7fa92006d540 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:56.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.402+0000 7fa944d0c640 1 --2- 172.21.15.57:0/388039476 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa9400cbd20 0x7fa94014ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:56.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.402+0000 7fa944d0c640 1 --2- 172.21.15.57:0/388039476 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa9400b0530 0x7fa940142430 unknown :-1 s=CLOSED pgs=2545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:56.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.402+0000 7fa944d0c640 1 --2- 172.21.15.57:0/388039476 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa9400afa20 0x7fa9400adbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:56.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.402+0000 7fa944d0c640 1 -- 172.21.15.57:0/388039476 >> 172.21.15.57:0/388039476 conn(0x7fa940069930 msgr2=0x7fa94006ba50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:56.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.402+0000 7fa944d0c640 1 -- 172.21.15.57:0/388039476 shutdown_connections 2023-10-19T14:20:56.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.402+0000 7fa944d0c640 1 -- 172.21.15.57:0/388039476 wait complete. 2023-10-19T14:20:56.416 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T14:20:56.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.523+0000 7fdfb08e0640 1 -- 172.21.15.57:0/777452894 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdfac0cbd20 msgr2=0x7fdfac158300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:56.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.523+0000 7fdfb08e0640 1 --2- 172.21.15.57:0/777452894 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdfac0cbd20 0x7fdfac158300 secure :-1 s=READY pgs=2547 cs=0 l=1 rev1=1 crypto rx=0x7fdfac0cc490 tx=0x7fdf9c02bd70 comp rx=0 tx=0).stop 2023-10-19T14:20:56.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.523+0000 7fdfb08e0640 1 -- 172.21.15.57:0/777452894 shutdown_connections 2023-10-19T14:20:56.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.523+0000 7fdfb08e0640 1 --2- 172.21.15.57:0/777452894 >> v2:172.21.15.179:3300/0 conn(0x7fdfac0afa20 0x7fdfac0705a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:56.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.523+0000 7fdfb08e0640 1 --2- 172.21.15.57:0/777452894 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdfac0cbd20 0x7fdfac158300 unknown :-1 s=CLOSED pgs=2547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:56.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.523+0000 7fdfb08e0640 1 -- 172.21.15.57:0/777452894 >> 172.21.15.57:0/777452894 conn(0x7fdfac069930 msgr2=0x7fdfac069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:56.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.523+0000 7fdfb08e0640 1 -- 172.21.15.57:0/777452894 shutdown_connections 2023-10-19T14:20:56.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.523+0000 7fdfb08e0640 1 -- 172.21.15.57:0/777452894 wait complete. 2023-10-19T14:20:56.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.524+0000 7fdfb08e0640 1 Processor -- start 2023-10-19T14:20:56.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.524+0000 7fdfb08e0640 1 -- start start 2023-10-19T14:20:56.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.525+0000 7fdfb08e0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdfac0afa20 0x7fdfac135b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:56.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.525+0000 7fdfb08e0640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdfac0b0530 0x7fdfac146980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:56.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.525+0000 7fdfb08e0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdfac0cbd20 0x7fdfac0adb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:56.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.525+0000 7fdfb08e0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfac147d30 con 0x7fdfac0afa20 2023-10-19T14:20:56.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.525+0000 7fdfb08e0640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfac1455b0 con 0x7fdfac0cbd20 2023-10-19T14:20:56.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.525+0000 7fdfb08e0640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfac146ec0 con 0x7fdfac0b0530 2023-10-19T14:20:56.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.525+0000 7fdfaa59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdfac0afa20 0x7fdfac135b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:56.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.525+0000 7fdfaa59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdfac0afa20 0x7fdfac135b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56262/0 (socket says 172.21.15.57:56262) 2023-10-19T14:20:56.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.525+0000 7fdfaa59c640 1 -- 172.21.15.57:0/3520717472 learned_addr learned my addr 172.21.15.57:0/3520717472 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:56.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.525+0000 7fdfaa59c640 1 -- 172.21.15.57:0/3520717472 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdfac0b0530 msgr2=0x7fdfac146980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:56.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.525+0000 7fdfaa59c640 1 --2- 172.21.15.57:0/3520717472 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdfac0b0530 0x7fdfac146980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:56.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.525+0000 7fdfaa59c640 1 -- 172.21.15.57:0/3520717472 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdfac0cbd20 msgr2=0x7fdfac0adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:56.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.525+0000 7fdfaad9d640 1 --2- 172.21.15.57:0/3520717472 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdfac0cbd20 0x7fdfac0adb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:56.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.525+0000 7fdfaa59c640 1 --2- 172.21.15.57:0/3520717472 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdfac0cbd20 0x7fdfac0adb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:56.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.525+0000 7fdfaa59c640 1 -- 172.21.15.57:0/3520717472 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf9c009490 con 0x7fdfac0afa20 2023-10-19T14:20:56.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.525+0000 7fdfaad9d640 1 --2- 172.21.15.57:0/3520717472 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdfac0cbd20 0x7fdfac0adb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:20:56.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.526+0000 7fdfaa59c640 1 --2- 172.21.15.57:0/3520717472 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdfac0afa20 0x7fdfac135b70 secure :-1 s=READY pgs=2548 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c000c00 tx=0x7fdf9c0038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:56.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.526+0000 7fdfa2ffd640 1 -- 172.21.15.57:0/3520717472 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf9c003ea0 con 0x7fdfac0afa20 2023-10-19T14:20:56.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.526+0000 7fdfa2ffd640 1 -- 172.21.15.57:0/3520717472 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fdf9c034440 con 0x7fdfac0afa20 2023-10-19T14:20:56.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.526+0000 7fdfb08e0640 1 -- 172.21.15.57:0/3520717472 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfac070cf0 con 0x7fdfac0afa20 2023-10-19T14:20:56.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.526+0000 7fdfa2ffd640 1 -- 172.21.15.57:0/3520717472 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf9c034850 con 0x7fdfac0afa20 2023-10-19T14:20:56.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.526+0000 7fdfb08e0640 1 -- 172.21.15.57:0/3520717472 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfac147390 con 0x7fdfac0afa20 2023-10-19T14:20:56.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.527+0000 7fdfb08e0640 1 -- 172.21.15.57:0/3520717472 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfac0690e0 con 0x7fdfac0afa20 2023-10-19T14:20:56.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.527+0000 7fdfa2ffd640 1 -- 172.21.15.57:0/3520717472 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fdf9c03b040 con 0x7fdfac0afa20 2023-10-19T14:20:56.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.528+0000 7fdfa2ffd640 1 --2- 172.21.15.57:0/3520717472 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fdf8c06b080 0x7fdf8c06d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:56.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.528+0000 7fdfa2ffd640 1 -- 172.21.15.57:0/3520717472 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 6972+0+0 (secure 0 0 0) 0x7fdf9c0ac330 con 0x7fdfac0afa20 2023-10-19T14:20:56.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.528+0000 7fdfa3fff640 1 --2- 172.21.15.57:0/3520717472 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fdf8c06b080 0x7fdf8c06d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:56.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.528+0000 7fdfa3fff640 1 --2- 172.21.15.57:0/3520717472 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fdf8c06b080 0x7fdf8c06d540 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fdf940058c0 tx=0x7fdf94005850 comp rx=0 tx=0).ready entity=mgr.9530 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:56.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.532+0000 7fdfa2ffd640 1 -- 172.21.15.57:0/3520717472 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fdf9c076b80 con 0x7fdfac0afa20 2023-10-19T14:20:56.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.708+0000 7fdfb08e0640 1 -- 172.21.15.57:0/3520717472 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdfac0cc1e0 con 0x7fdfac0afa20 2023-10-19T14:20:56.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.710+0000 7fdfa2ffd640 1 -- 172.21.15.57:0/3520717472 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v232) v1 ==== 74+0+18119 (secure 0 0 0) 0x7fdf9c07a1a0 con 0x7fdfac0afa20 2023-10-19T14:20:56.715 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:20:56.715 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":232,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T14:20:54.777893+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_metadata","create_time":"2023-10-19T14:20:28.667664+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"226","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"backup_fs_data","create_time":"2023-10-19T14:20:29.122584+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"227","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.57:6835/1529368280":"2023-10-20T14:20:51.762660+0000","172.21.15.179:0/1513076424":"2023-10-20T14:20:48.148610+0000","172.21.15.179:0/1464950687":"2023-10-20T14:20:48.148610+0000","172.21.15.179:0/2900872065":"2023-10-20T14:20:48.148610+0000","172.21.15.57:6833/1529368280":"2023-10-20T14:20:51.762660+0000","172.21.15.179:0/2140451434":"2023-10-20T14:20:48.148610+0000","172.21.15.179:0/2605624080":"2023-10-20T14:20:48.148610+0000","172.21.15.179:0/1194984394":"2023-10-20T14:20:39.702634+0000","172.21.15.179:6833/3232482678":"2023-10-20T14:20:39.702634+0000","172.21.15.179:6832/4242302683":"2023-10-20T14:20:48.148610+0000","172.21.15.179:6832/3232482678":"2023-10-20T14:20:39.702634+0000","172.21.15.179:0/2872579179":"2023-10-20T14:20:39.702634+0000","172.21.15.179:0/373323540":"2023-10-20T14:20:48.148610+0000","172.21.15.179:0/3910604357":"2023-10-20T14:20:39.702634+0000","172.21.15.179:0/4185855767":"2023-10-20T14:20:48.148610+0000","172.21.15.179:0/536160557":"2023-10-20T14:20:39.702634+0000","172.21.15.179:0/3239062050":"2023-10-20T14:20:39.702634+0000","172.21.15.179:6833/4242302683":"2023-10-20T14:20:48.148610+0000","172.21.15.179:0/3672528294":"2023-10-20T14:20:48.148610+0000","172.21.15.179:0/2003826253":"2023-10-20T14:20:48.148610+0000","172.21.15.179:0/1948632334":"2023-10-20T14:20:39.702634+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T14:20:56.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.711+0000 7fdfb08e0640 1 -- 172.21.15.57:0/3520717472 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fdf8c06b080 msgr2=0x7fdf8c06d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:56.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.711+0000 7fdfb08e0640 1 --2- 172.21.15.57:0/3520717472 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fdf8c06b080 0x7fdf8c06d540 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fdf940058c0 tx=0x7fdf94005850 comp rx=0 tx=0).stop 2023-10-19T14:20:56.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.711+0000 7fdfb08e0640 1 -- 172.21.15.57:0/3520717472 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdfac0afa20 msgr2=0x7fdfac135b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:56.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.711+0000 7fdfb08e0640 1 --2- 172.21.15.57:0/3520717472 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdfac0afa20 0x7fdfac135b70 secure :-1 s=READY pgs=2548 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c000c00 tx=0x7fdf9c0038a0 comp rx=0 tx=0).stop 2023-10-19T14:20:56.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.711+0000 7fdfb08e0640 1 -- 172.21.15.57:0/3520717472 shutdown_connections 2023-10-19T14:20:56.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.711+0000 7fdfb08e0640 1 --2- 172.21.15.57:0/3520717472 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fdf8c06b080 0x7fdf8c06d540 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:56.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.711+0000 7fdfb08e0640 1 --2- 172.21.15.57:0/3520717472 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdfac0cbd20 0x7fdfac0adb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:56.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.711+0000 7fdfb08e0640 1 --2- 172.21.15.57:0/3520717472 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdfac0b0530 0x7fdfac146980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:56.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.711+0000 7fdfb08e0640 1 --2- 172.21.15.57:0/3520717472 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdfac0afa20 0x7fdfac135b70 unknown :-1 s=CLOSED pgs=2548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:56.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.711+0000 7fdfb08e0640 1 -- 172.21.15.57:0/3520717472 >> 172.21.15.57:0/3520717472 conn(0x7fdfac069930 msgr2=0x7fdfac0ca870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:56.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.712+0000 7fdfb08e0640 1 -- 172.21.15.57:0/3520717472 shutdown_connections 2023-10-19T14:20:56.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.712+0000 7fdfb08e0640 1 -- 172.21.15.57:0/3520717472 wait complete. 2023-10-19T14:20:56.725 INFO:tasks.cephfs.filesystem:Destroying file system backup_fs and related pools 2023-10-19T14:20:56.726 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T14:20:56.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.833+0000 7f46bf3e7640 1 -- 172.21.15.57:0/27848259 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f46b80afa20 msgr2=0x7f46b80705a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:56.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.833+0000 7f46bf3e7640 1 --2- 172.21.15.57:0/27848259 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f46b80afa20 0x7f46b80705a0 secure :-1 s=READY pgs=2549 cs=0 l=1 rev1=1 crypto rx=0x7f46b80cc7e0 tx=0x7f46a802bda0 comp rx=0 tx=0).stop 2023-10-19T14:20:56.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.833+0000 7f46bf3e7640 1 -- 172.21.15.57:0/27848259 shutdown_connections 2023-10-19T14:20:56.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.833+0000 7f46bf3e7640 1 --2- 172.21.15.57:0/27848259 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f46b80b0530 0x7f46b80b09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:56.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.833+0000 7f46bf3e7640 1 --2- 172.21.15.57:0/27848259 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f46b80afa20 0x7f46b80705a0 unknown :-1 s=CLOSED pgs=2549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:56.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.833+0000 7f46bf3e7640 1 --2- 172.21.15.57:0/27848259 >> v2:172.21.15.179:3300/0 conn(0x7f46b80cbd20 0x7f46b8158300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:56.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.833+0000 7f46bf3e7640 1 -- 172.21.15.57:0/27848259 >> 172.21.15.57:0/27848259 conn(0x7f46b8069930 msgr2=0x7f46b8069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:56.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.833+0000 7f46bf3e7640 1 -- 172.21.15.57:0/27848259 shutdown_connections 2023-10-19T14:20:56.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.833+0000 7f46bf3e7640 1 -- 172.21.15.57:0/27848259 wait complete. 2023-10-19T14:20:56.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.835+0000 7f46bf3e7640 1 Processor -- start 2023-10-19T14:20:56.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.835+0000 7f46bf3e7640 1 -- start start 2023-10-19T14:20:56.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.835+0000 7f46bf3e7640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f46b80afa20 0x7f46b814a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:56.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.835+0000 7f46bf3e7640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f46b80b0530 0x7f46b814a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:56.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.835+0000 7f46bf3e7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f46b80cbd20 0x7f46b8141020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:56.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.835+0000 7f46bf3e7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46b815a5e0 con 0x7f46b80cbd20 2023-10-19T14:20:56.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.835+0000 7f46bf3e7640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46b814ae50 con 0x7f46b80b0530 2023-10-19T14:20:56.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.835+0000 7f46bf3e7640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46b815a020 con 0x7f46b80afa20 2023-10-19T14:20:56.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.835+0000 7f46bd984640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f46b80cbd20 0x7f46b8141020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:56.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.836+0000 7f46bd984640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f46b80cbd20 0x7f46b8141020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38392/0 (socket says 172.21.15.57:38392) 2023-10-19T14:20:56.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.836+0000 7f46bd984640 1 -- 172.21.15.57:0/291609441 learned_addr learned my addr 172.21.15.57:0/291609441 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:56.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.836+0000 7f46bd183640 1 --2- 172.21.15.57:0/291609441 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f46b80afa20 0x7f46b814a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:56.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.836+0000 7f46bc982640 1 --2- 172.21.15.57:0/291609441 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f46b80b0530 0x7f46b814a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:56.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.836+0000 7f46bd984640 1 -- 172.21.15.57:0/291609441 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f46b80afa20 msgr2=0x7f46b814a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:56.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.836+0000 7f46bd984640 1 --2- 172.21.15.57:0/291609441 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f46b80afa20 0x7f46b814a3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:56.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.836+0000 7f46bd984640 1 -- 172.21.15.57:0/291609441 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f46b80b0530 msgr2=0x7f46b814a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:56.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.836+0000 7f46bd984640 1 --2- 172.21.15.57:0/291609441 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f46b80b0530 0x7f46b814a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:56.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.836+0000 7f46bd984640 1 -- 172.21.15.57:0/291609441 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46a8009560 con 0x7f46b80cbd20 2023-10-19T14:20:56.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.836+0000 7f46bd984640 1 --2- 172.21.15.57:0/291609441 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f46b80cbd20 0x7f46b8141020 secure :-1 s=READY pgs=2550 cs=0 l=1 rev1=1 crypto rx=0x7f46b000e9b0 tx=0x7f46b000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:56.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.836+0000 7f46adffb640 1 -- 172.21.15.57:0/291609441 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b0004d20 con 0x7f46b80cbd20 2023-10-19T14:20:56.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.836+0000 7f46adffb640 1 -- 172.21.15.57:0/291609441 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f46b000cb20 con 0x7f46b80cbd20 2023-10-19T14:20:56.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.836+0000 7f46bf3e7640 1 -- 172.21.15.57:0/291609441 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46b815a300 con 0x7f46b80cbd20 2023-10-19T14:20:56.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.836+0000 7f46adffb640 1 -- 172.21.15.57:0/291609441 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b0005020 con 0x7f46b80cbd20 2023-10-19T14:20:56.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.836+0000 7f46bf3e7640 1 -- 172.21.15.57:0/291609441 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46b8148850 con 0x7f46b80cbd20 2023-10-19T14:20:56.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.837+0000 7f46bf3e7640 1 -- 172.21.15.57:0/291609441 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46b8067930 con 0x7f46b80cbd20 2023-10-19T14:20:56.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.838+0000 7f46adffb640 1 -- 172.21.15.57:0/291609441 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f46b0005180 con 0x7f46b80cbd20 2023-10-19T14:20:56.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.838+0000 7f46adffb640 1 --2- 172.21.15.57:0/291609441 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f469c06b080 0x7f469c06d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:56.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.838+0000 7f46adffb640 1 -- 172.21.15.57:0/291609441 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 6972+0+0 (secure 0 0 0) 0x7f46b008a900 con 0x7f46b80cbd20 2023-10-19T14:20:56.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.838+0000 7f46bd183640 1 --2- 172.21.15.57:0/291609441 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f469c06b080 0x7f469c06d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:56.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.839+0000 7f46bd183640 1 --2- 172.21.15.57:0/291609441 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f469c06b080 0x7f469c06d540 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f46b814be70 tx=0x7f46b400b2e0 comp rx=0 tx=0).ready entity=mgr.9530 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:56.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:56.842+0000 7f46adffb640 1 -- 172.21.15.57:0/291609441 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f46b0055100 con 0x7f46b80cbd20 2023-10-19T14:20:57.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.057+0000 7f46bf3e7640 1 -- 172.21.15.57:0/291609441 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f46b8158090 con 0x7f46b80cbd20 2023-10-19T14:20:57.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.057+0000 7f46adffb640 1 -- 172.21.15.57:0/291609441 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+4931 (secure 0 0 0) 0x7f46b001f090 con 0x7f46b80cbd20 2023-10-19T14:20:57.060 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:20:57.061 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":96,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9248,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/1249807745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":1249807745},{"type":"v1","addr":"172.21.15.179:6836","nonce":1249807745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87},{"gid":10513,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/1406825818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":1406825818},{"type":"v1","addr":"172.21.15.179:6837","nonce":1406825818}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":86},{"gid":12375,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/2367396561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2367396561},{"type":"v1","addr":"172.21.15.57:6835","nonce":2367396561}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T14:20:30.145135+0000","modified":"2023-10-19T14:20:32.141937+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11715},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11715":{"gid":11715,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":5,"addr":"172.21.15.57:6834/2467807803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2467807803},{"type":"v1","addr":"172.21.15.57:6834","nonce":2467807803}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2023-10-19T14:20:57.061 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 96 2023-10-19T14:20:57.062 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.059+0000 7f46bf3e7640 1 -- 172.21.15.57:0/291609441 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f469c06b080 msgr2=0x7f469c06d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:57.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.059+0000 7f46bf3e7640 1 --2- 172.21.15.57:0/291609441 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f469c06b080 0x7f469c06d540 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f46b814be70 tx=0x7f46b400b2e0 comp rx=0 tx=0).stop 2023-10-19T14:20:57.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.059+0000 7f46bf3e7640 1 -- 172.21.15.57:0/291609441 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f46b80cbd20 msgr2=0x7f46b8141020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:57.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.059+0000 7f46bf3e7640 1 --2- 172.21.15.57:0/291609441 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f46b80cbd20 0x7f46b8141020 secure :-1 s=READY pgs=2550 cs=0 l=1 rev1=1 crypto rx=0x7f46b000e9b0 tx=0x7f46b000ee70 comp rx=0 tx=0).stop 2023-10-19T14:20:57.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.059+0000 7f46bf3e7640 1 -- 172.21.15.57:0/291609441 shutdown_connections 2023-10-19T14:20:57.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.059+0000 7f46bf3e7640 1 --2- 172.21.15.57:0/291609441 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f469c06b080 0x7f469c06d540 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:57.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.059+0000 7f46bf3e7640 1 --2- 172.21.15.57:0/291609441 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f46b80cbd20 0x7f46b8141020 unknown :-1 s=CLOSED pgs=2550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:57.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.059+0000 7f46bf3e7640 1 --2- 172.21.15.57:0/291609441 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f46b80b0530 0x7f46b814a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:57.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.059+0000 7f46bf3e7640 1 --2- 172.21.15.57:0/291609441 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f46b80afa20 0x7f46b814a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:57.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.059+0000 7f46bf3e7640 1 -- 172.21.15.57:0/291609441 >> 172.21.15.57:0/291609441 conn(0x7f46b8069930 msgr2=0x7f46b8069d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:57.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.059+0000 7f46bf3e7640 1 -- 172.21.15.57:0/291609441 shutdown_connections 2023-10-19T14:20:57.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.059+0000 7f46bf3e7640 1 -- 172.21.15.57:0/291609441 wait complete. 2023-10-19T14:20:57.073 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T14:20:57.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.180+0000 7ff6c4b25640 1 -- 172.21.15.57:0/2205556678 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff6c00c96b0 msgr2=0x7ff6c00c9b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:57.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.180+0000 7ff6c4b25640 1 --2- 172.21.15.57:0/2205556678 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff6c00c96b0 0x7ff6c00c9b30 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7ff6a8009830 tx=0x7ff6a802bc20 comp rx=0 tx=0).stop 2023-10-19T14:20:57.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.180+0000 7ff6c4b25640 1 -- 172.21.15.57:0/2205556678 shutdown_connections 2023-10-19T14:20:57.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.180+0000 7ff6c4b25640 1 --2- 172.21.15.57:0/2205556678 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff6c00c96b0 0x7ff6c00c9b30 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:57.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.180+0000 7ff6c4b25640 1 --2- 172.21.15.57:0/2205556678 >> v2:172.21.15.179:3300/0 conn(0x7ff6c006af00 0x7ff6c006b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:57.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.180+0000 7ff6c4b25640 1 -- 172.21.15.57:0/2205556678 >> 172.21.15.57:0/2205556678 conn(0x7ff6c0069930 msgr2=0x7ff6c0069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:57.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.180+0000 7ff6c4b25640 1 -- 172.21.15.57:0/2205556678 shutdown_connections 2023-10-19T14:20:57.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.180+0000 7ff6c4b25640 1 -- 172.21.15.57:0/2205556678 wait complete. 2023-10-19T14:20:57.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.181+0000 7ff6c4b25640 1 Processor -- start 2023-10-19T14:20:57.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.181+0000 7ff6c4b25640 1 -- start start 2023-10-19T14:20:57.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.182+0000 7ff6c4b25640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff6c006af00 0x7ff6c013a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:57.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.182+0000 7ff6c4b25640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff6c00ab500 0x7ff6c00a7fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:57.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.182+0000 7ff6c4b25640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff6c00c96b0 0x7ff6c00a9850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:57.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.182+0000 7ff6c4b25640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6c00a9430 con 0x7ff6c006af00 2023-10-19T14:20:57.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.182+0000 7ff6c4b25640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6c00a8520 con 0x7ff6c00ab500 2023-10-19T14:20:57.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.182+0000 7ff6c4b25640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6c00a87d0 con 0x7ff6c00c96b0 2023-10-19T14:20:57.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.182+0000 7ff6be59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff6c006af00 0x7ff6c013a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:57.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.182+0000 7ff6be59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff6c006af00 0x7ff6c013a560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38406/0 (socket says 172.21.15.57:38406) 2023-10-19T14:20:57.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.182+0000 7ff6be59c640 1 -- 172.21.15.57:0/2121582021 learned_addr learned my addr 172.21.15.57:0/2121582021 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:57.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.182+0000 7ff6bed9d640 1 --2- 172.21.15.57:0/2121582021 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff6c00c96b0 0x7ff6c00a9850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:57.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.182+0000 7ff6b7fff640 1 --2- 172.21.15.57:0/2121582021 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff6c00ab500 0x7ff6c00a7fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:57.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.182+0000 7ff6be59c640 1 -- 172.21.15.57:0/2121582021 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff6c00c96b0 msgr2=0x7ff6c00a9850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:57.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.182+0000 7ff6be59c640 1 --2- 172.21.15.57:0/2121582021 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff6c00c96b0 0x7ff6c00a9850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:57.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.182+0000 7ff6be59c640 1 -- 172.21.15.57:0/2121582021 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff6c00ab500 msgr2=0x7ff6c00a7fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:57.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.182+0000 7ff6be59c640 1 --2- 172.21.15.57:0/2121582021 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff6c00ab500 0x7ff6c00a7fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:57.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.182+0000 7ff6be59c640 1 -- 172.21.15.57:0/2121582021 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6a8009490 con 0x7ff6c006af00 2023-10-19T14:20:57.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.183+0000 7ff6be59c640 1 --2- 172.21.15.57:0/2121582021 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff6c006af00 0x7ff6c013a560 secure :-1 s=READY pgs=2552 cs=0 l=1 rev1=1 crypto rx=0x7ff6b000e9f0 tx=0x7ff6b000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:57.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.183+0000 7ff6b6ffd640 1 -- 172.21.15.57:0/2121582021 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6b0009700 con 0x7ff6c006af00 2023-10-19T14:20:57.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.183+0000 7ff6b6ffd640 1 -- 172.21.15.57:0/2121582021 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7ff6b000ce70 con 0x7ff6c006af00 2023-10-19T14:20:57.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.183+0000 7ff6c4b25640 1 -- 172.21.15.57:0/2121582021 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6c00aaba0 con 0x7ff6c006af00 2023-10-19T14:20:57.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.183+0000 7ff6b6ffd640 1 -- 172.21.15.57:0/2121582021 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6b0010670 con 0x7ff6c006af00 2023-10-19T14:20:57.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.183+0000 7ff6c4b25640 1 -- 172.21.15.57:0/2121582021 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6c00a8d70 con 0x7ff6c006af00 2023-10-19T14:20:57.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.184+0000 7ff6c4b25640 1 -- 172.21.15.57:0/2121582021 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6c00a96a0 con 0x7ff6c006af00 2023-10-19T14:20:57.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.185+0000 7ff6b6ffd640 1 -- 172.21.15.57:0/2121582021 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7ff6b00107d0 con 0x7ff6c006af00 2023-10-19T14:20:57.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.185+0000 7ff6b6ffd640 1 --2- 172.21.15.57:0/2121582021 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7ff6a006b080 0x7ff6a006d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:57.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.185+0000 7ff6b6ffd640 1 -- 172.21.15.57:0/2121582021 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 6972+0+0 (secure 0 0 0) 0x7ff6b0014070 con 0x7ff6c006af00 2023-10-19T14:20:57.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.185+0000 7ff6b7fff640 1 --2- 172.21.15.57:0/2121582021 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7ff6a006b080 0x7ff6a006d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:57.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.186+0000 7ff6b7fff640 1 --2- 172.21.15.57:0/2121582021 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7ff6a006b080 0x7ff6a006d540 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff6a8009830 tx=0x7ff6a8004730 comp rx=0 tx=0).ready entity=mgr.9530 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:57.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.189+0000 7ff6b6ffd640 1 -- 172.21.15.57:0/2121582021 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff6b0051ca0 con 0x7ff6c006af00 2023-10-19T14:20:57.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.400+0000 7ff6c4b25640 1 -- 172.21.15.57:0/2121582021 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff6c00690e0 con 0x7ff6c006af00 2023-10-19T14:20:57.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.400+0000 7ff6b6ffd640 1 -- 172.21.15.57:0/2121582021 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+4931 (secure 0 0 0) 0x7ff6b0058ea0 con 0x7ff6c006af00 2023-10-19T14:20:57.404 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:20:57.404 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":96,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9248,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/1249807745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":1249807745},{"type":"v1","addr":"172.21.15.179:6836","nonce":1249807745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87},{"gid":10513,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/1406825818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":1406825818},{"type":"v1","addr":"172.21.15.179:6837","nonce":1406825818}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":86},{"gid":12375,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/2367396561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2367396561},{"type":"v1","addr":"172.21.15.57:6835","nonce":2367396561}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T14:20:30.145135+0000","modified":"2023-10-19T14:20:32.141937+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11715},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11715":{"gid":11715,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":5,"addr":"172.21.15.57:6834/2467807803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":2467807803},{"type":"v1","addr":"172.21.15.57:6834","nonce":2467807803}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2023-10-19T14:20:57.405 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 96 2023-10-19T14:20:57.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.402+0000 7ff6c4b25640 1 -- 172.21.15.57:0/2121582021 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7ff6a006b080 msgr2=0x7ff6a006d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:57.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.403+0000 7ff6c4b25640 1 --2- 172.21.15.57:0/2121582021 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7ff6a006b080 0x7ff6a006d540 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff6a8009830 tx=0x7ff6a8004730 comp rx=0 tx=0).stop 2023-10-19T14:20:57.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.403+0000 7ff6c4b25640 1 -- 172.21.15.57:0/2121582021 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff6c006af00 msgr2=0x7ff6c013a560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:57.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.403+0000 7ff6c4b25640 1 --2- 172.21.15.57:0/2121582021 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff6c006af00 0x7ff6c013a560 secure :-1 s=READY pgs=2552 cs=0 l=1 rev1=1 crypto rx=0x7ff6b000e9f0 tx=0x7ff6b000eeb0 comp rx=0 tx=0).stop 2023-10-19T14:20:57.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.403+0000 7ff6c4b25640 1 -- 172.21.15.57:0/2121582021 shutdown_connections 2023-10-19T14:20:57.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.403+0000 7ff6c4b25640 1 --2- 172.21.15.57:0/2121582021 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7ff6a006b080 0x7ff6a006d540 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:57.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.403+0000 7ff6c4b25640 1 --2- 172.21.15.57:0/2121582021 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff6c00c96b0 0x7ff6c00a9850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:57.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.403+0000 7ff6c4b25640 1 --2- 172.21.15.57:0/2121582021 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff6c00ab500 0x7ff6c00a7fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:57.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.403+0000 7ff6c4b25640 1 --2- 172.21.15.57:0/2121582021 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff6c006af00 0x7ff6c013a560 unknown :-1 s=CLOSED pgs=2552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:57.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.403+0000 7ff6c4b25640 1 -- 172.21.15.57:0/2121582021 >> 172.21.15.57:0/2121582021 conn(0x7ff6c0069930 msgr2=0x7ff6c00b0710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:57.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.403+0000 7ff6c4b25640 1 -- 172.21.15.57:0/2121582021 shutdown_connections 2023-10-19T14:20:57.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.403+0000 7ff6c4b25640 1 -- 172.21.15.57:0/2121582021 wait complete. 2023-10-19T14:20:57.416 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T14:20:57.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.523+0000 7f1fef626640 1 -- 172.21.15.57:0/2263872706 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1fd4003b30 msgr2=0x7f1fd4003f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:57.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.523+0000 7f1fef626640 1 --2- 172.21.15.57:0/2263872706 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1fd4003b30 0x7f1fd4003f30 secure :-1 s=READY pgs=2554 cs=0 l=1 rev1=1 crypto rx=0x7f1fe4001800 tx=0x7f1fd40057f0 comp rx=0 tx=0).stop 2023-10-19T14:20:57.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.523+0000 7f1fedbc3640 1 -- 172.21.15.57:0/2263872706 reap_dead start 2023-10-19T14:20:57.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.523+0000 7f1fef626640 1 -- 172.21.15.57:0/2263872706 shutdown_connections 2023-10-19T14:20:57.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.523+0000 7f1fef626640 1 -- 172.21.15.57:0/2263872706 >> 172.21.15.57:0/2263872706 conn(0x7f1fe80af7a0 msgr2=0x7f1fe80afb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:57.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.523+0000 7f1fef626640 1 -- 172.21.15.57:0/2263872706 shutdown_connections 2023-10-19T14:20:57.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.523+0000 7f1fef626640 1 -- 172.21.15.57:0/2263872706 wait complete. 2023-10-19T14:20:57.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.524+0000 7f1fef626640 1 Processor -- start 2023-10-19T14:20:57.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.524+0000 7f1fef626640 1 -- start start 2023-10-19T14:20:57.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.525+0000 7f1fef626640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1fd4003b30 0x7f1fe814fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:57.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.525+0000 7f1fef626640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1fe81545b0 0x7f1fe8069720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:57.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.525+0000 7f1fef626640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1fe8154a30 0x7f1fe8069c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:57.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.525+0000 7f1fef626640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe80c5eb0 con 0x7f1fe81545b0 2023-10-19T14:20:57.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.525+0000 7f1fef626640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe80c52a0 con 0x7f1fd4003b30 2023-10-19T14:20:57.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.525+0000 7f1fef626640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe80c5580 con 0x7f1fe8154a30 2023-10-19T14:20:57.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.525+0000 7f1fecbc1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1fe81545b0 0x7f1fe8069720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:57.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.525+0000 7f1fecbc1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1fe81545b0 0x7f1fe8069720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38418/0 (socket says 172.21.15.57:38418) 2023-10-19T14:20:57.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.525+0000 7f1fecbc1640 1 -- 172.21.15.57:0/2344175755 learned_addr learned my addr 172.21.15.57:0/2344175755 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:57.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.526+0000 7f1fecbc1640 1 -- 172.21.15.57:0/2344175755 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1fe8154a30 msgr2=0x7f1fe8069c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:57.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.526+0000 7f1fedbc3640 1 --2- 172.21.15.57:0/2344175755 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1fe8154a30 0x7f1fe8069c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:57.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.526+0000 7f1fecbc1640 1 --2- 172.21.15.57:0/2344175755 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1fe8154a30 0x7f1fe8069c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:57.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.526+0000 7f1fed3c2640 1 --2- 172.21.15.57:0/2344175755 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1fd4003b30 0x7f1fe814fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:57.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.526+0000 7f1fecbc1640 1 -- 172.21.15.57:0/2344175755 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1fd4003b30 msgr2=0x7f1fe814fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:57.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.526+0000 7f1fecbc1640 1 --2- 172.21.15.57:0/2344175755 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1fd4003b30 0x7f1fe814fac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:57.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.526+0000 7f1fecbc1640 1 -- 172.21.15.57:0/2344175755 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fe80c5860 con 0x7f1fe81545b0 2023-10-19T14:20:57.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.526+0000 7f1fecbc1640 1 --2- 172.21.15.57:0/2344175755 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1fe81545b0 0x7f1fe8069720 secure :-1 s=READY pgs=2555 cs=0 l=1 rev1=1 crypto rx=0x7f1fd8006e80 tx=0x7f1fd80049d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:57.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.526+0000 7f1fddffb640 1 -- 172.21.15.57:0/2344175755 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd800e9b0 con 0x7f1fe81545b0 2023-10-19T14:20:57.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.526+0000 7f1fddffb640 1 -- 172.21.15.57:0/2344175755 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f1fd800ec20 con 0x7f1fe81545b0 2023-10-19T14:20:57.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.526+0000 7f1fef626640 1 -- 172.21.15.57:0/2344175755 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fe80c5b40 con 0x7f1fe81545b0 2023-10-19T14:20:57.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.526+0000 7f1fddffb640 1 -- 172.21.15.57:0/2344175755 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd8011460 con 0x7f1fe81545b0 2023-10-19T14:20:57.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.526+0000 7f1fef626640 1 -- 172.21.15.57:0/2344175755 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fe80c4e60 con 0x7f1fe81545b0 2023-10-19T14:20:57.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.527+0000 7f1fef626640 1 -- 172.21.15.57:0/2344175755 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fe80c6020 con 0x7f1fe81545b0 2023-10-19T14:20:57.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.528+0000 7f1fddffb640 1 -- 172.21.15.57:0/2344175755 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f1fd80103c0 con 0x7f1fe81545b0 2023-10-19T14:20:57.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.528+0000 7f1fddffb640 1 --2- 172.21.15.57:0/2344175755 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f1fcc06b0b0 0x7f1fcc06d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:57.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.528+0000 7f1fddffb640 1 -- 172.21.15.57:0/2344175755 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 6972+0+0 (secure 0 0 0) 0x7f1fd808a930 con 0x7f1fe81545b0 2023-10-19T14:20:57.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.528+0000 7f1fed3c2640 1 --2- 172.21.15.57:0/2344175755 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f1fcc06b0b0 0x7f1fcc06d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:57.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.529+0000 7f1fed3c2640 1 --2- 172.21.15.57:0/2344175755 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f1fcc06b0b0 0x7f1fcc06d570 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1fe4034000 tx=0x7f1fe402e070 comp rx=0 tx=0).ready entity=mgr.9530 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:57.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.532+0000 7f1fddffb640 1 -- 172.21.15.57:0/2344175755 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f1fd8055110 con 0x7f1fe81545b0 2023-10-19T14:20:57.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.709+0000 7f1fef626640 1 -- 172.21.15.57:0/2344175755 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1fe815c7f0 con 0x7f1fe81545b0 2023-10-19T14:20:57.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.710+0000 7f1fddffb640 1 -- 172.21.15.57:0/2344175755 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v232) v1 ==== 74+0+18119 (secure 0 0 0) 0x7f1fe815c7f0 con 0x7f1fe81545b0 2023-10-19T14:20:57.713 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:20:57.713 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":232,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T14:20:54.777893+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_metadata","create_time":"2023-10-19T14:20:28.667664+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"226","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"backup_fs_data","create_time":"2023-10-19T14:20:29.122584+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"227","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.57:6835/1529368280":"2023-10-20T14:20:51.762660+0000","172.21.15.179:0/1513076424":"2023-10-20T14:20:48.148610+0000","172.21.15.179:0/1464950687":"2023-10-20T14:20:48.148610+0000","172.21.15.179:0/2900872065":"2023-10-20T14:20:48.148610+0000","172.21.15.57:6833/1529368280":"2023-10-20T14:20:51.762660+0000","172.21.15.179:0/2140451434":"2023-10-20T14:20:48.148610+0000","172.21.15.179:0/2605624080":"2023-10-20T14:20:48.148610+0000","172.21.15.179:0/1194984394":"2023-10-20T14:20:39.702634+0000","172.21.15.179:6833/3232482678":"2023-10-20T14:20:39.702634+0000","172.21.15.179:6832/4242302683":"2023-10-20T14:20:48.148610+0000","172.21.15.179:6832/3232482678":"2023-10-20T14:20:39.702634+0000","172.21.15.179:0/2872579179":"2023-10-20T14:20:39.702634+0000","172.21.15.179:0/373323540":"2023-10-20T14:20:48.148610+0000","172.21.15.179:0/3910604357":"2023-10-20T14:20:39.702634+0000","172.21.15.179:0/4185855767":"2023-10-20T14:20:48.148610+0000","172.21.15.179:0/536160557":"2023-10-20T14:20:39.702634+0000","172.21.15.179:0/3239062050":"2023-10-20T14:20:39.702634+0000","172.21.15.179:6833/4242302683":"2023-10-20T14:20:48.148610+0000","172.21.15.179:0/3672528294":"2023-10-20T14:20:48.148610+0000","172.21.15.179:0/2003826253":"2023-10-20T14:20:48.148610+0000","172.21.15.179:0/1948632334":"2023-10-20T14:20:39.702634+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T14:20:57.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.712+0000 7f1fef626640 1 -- 172.21.15.57:0/2344175755 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f1fcc06b0b0 msgr2=0x7f1fcc06d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:57.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.712+0000 7f1fef626640 1 --2- 172.21.15.57:0/2344175755 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f1fcc06b0b0 0x7f1fcc06d570 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1fe4034000 tx=0x7f1fe402e070 comp rx=0 tx=0).stop 2023-10-19T14:20:57.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.712+0000 7f1fef626640 1 -- 172.21.15.57:0/2344175755 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1fe81545b0 msgr2=0x7f1fe8069720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:57.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.712+0000 7f1fef626640 1 --2- 172.21.15.57:0/2344175755 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1fe81545b0 0x7f1fe8069720 secure :-1 s=READY pgs=2555 cs=0 l=1 rev1=1 crypto rx=0x7f1fd8006e80 tx=0x7f1fd80049d0 comp rx=0 tx=0).stop 2023-10-19T14:20:57.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.712+0000 7f1fef626640 1 -- 172.21.15.57:0/2344175755 shutdown_connections 2023-10-19T14:20:57.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.712+0000 7f1fef626640 1 --2- 172.21.15.57:0/2344175755 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f1fcc06b0b0 0x7f1fcc06d570 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:57.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.712+0000 7f1fef626640 1 --2- 172.21.15.57:0/2344175755 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1fe8154a30 0x7f1fe8069c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:57.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.712+0000 7f1fef626640 1 --2- 172.21.15.57:0/2344175755 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1fe81545b0 0x7f1fe8069720 unknown :-1 s=CLOSED pgs=2555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:57.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.712+0000 7f1fef626640 1 --2- 172.21.15.57:0/2344175755 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1fd4003b30 0x7f1fe814fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:57.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.712+0000 7f1fef626640 1 -- 172.21.15.57:0/2344175755 >> 172.21.15.57:0/2344175755 conn(0x7f1fe80af7a0 msgr2=0x7f1fe81597b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:57.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.712+0000 7f1fef626640 1 -- 172.21.15.57:0/2344175755 shutdown_connections 2023-10-19T14:20:57.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.712+0000 7f1fef626640 1 -- 172.21.15.57:0/2344175755 wait complete. 2023-10-19T14:20:57.725 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail backup_fs 2023-10-19T14:20:57.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.833+0000 7f4cfad8a640 1 -- 172.21.15.57:0/3399212174 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4cdc003970 msgr2=0x7f4cdc003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:57.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.833+0000 7f4cfad8a640 1 --2- 172.21.15.57:0/3399212174 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4cdc003970 0x7f4cdc003d90 secure :-1 s=READY pgs=2557 cs=0 l=1 rev1=1 crypto rx=0x7f4cec009ed0 tx=0x7f4cdc0057f0 comp rx=0 tx=0).stop 2023-10-19T14:20:57.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.834+0000 7f4cfad8a640 1 -- 172.21.15.57:0/3399212174 shutdown_connections 2023-10-19T14:20:57.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.834+0000 7f4cfad8a640 1 --2- 172.21.15.57:0/3399212174 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4cdc003970 0x7f4cdc003d90 unknown :-1 s=CLOSED pgs=2557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:57.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.834+0000 7f4cfad8a640 1 -- 172.21.15.57:0/3399212174 >> 172.21.15.57:0/3399212174 conn(0x7f4cf4069930 msgr2=0x7f4cf4069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:57.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.834+0000 7f4cfad8a640 1 -- 172.21.15.57:0/3399212174 shutdown_connections 2023-10-19T14:20:57.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.834+0000 7f4cfad8a640 1 -- 172.21.15.57:0/3399212174 wait complete. 2023-10-19T14:20:57.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.835+0000 7f4cfad8a640 1 Processor -- start 2023-10-19T14:20:57.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.835+0000 7f4cfad8a640 1 -- start start 2023-10-19T14:20:57.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.836+0000 7f4cfad8a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4cdc003970 0x7f4cf4136240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:57.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.836+0000 7f4cfad8a640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4cf41387d0 0x7f4cf4138c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:57.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.836+0000 7f4cfad8a640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4cf40ae9a0 0x7f4cf40aee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:57.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.836+0000 7f4cfad8a640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cf4139190 con 0x7f4cdc003970 2023-10-19T14:20:57.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.836+0000 7f4cfad8a640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cf4152ec0 con 0x7f4cf41387d0 2023-10-19T14:20:57.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.836+0000 7f4cfad8a640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cf4136780 con 0x7f4cf40ae9a0 2023-10-19T14:20:57.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.836+0000 7f4cf8b26640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4cdc003970 0x7f4cf4136240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:57.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.836+0000 7f4cf8b26640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4cdc003970 0x7f4cf4136240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38446/0 (socket says 172.21.15.57:38446) 2023-10-19T14:20:57.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.836+0000 7f4cf8b26640 1 -- 172.21.15.57:0/1835806886 learned_addr learned my addr 172.21.15.57:0/1835806886 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:57.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.836+0000 7f4cf9327640 1 --2- 172.21.15.57:0/1835806886 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4cf40ae9a0 0x7f4cf40aee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:57.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.836+0000 7f4cf3fff640 1 --2- 172.21.15.57:0/1835806886 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4cf41387d0 0x7f4cf4138c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:57.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.836+0000 7f4cf8b26640 1 -- 172.21.15.57:0/1835806886 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4cf40ae9a0 msgr2=0x7f4cf40aee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:57.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.836+0000 7f4cf8b26640 1 --2- 172.21.15.57:0/1835806886 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4cf40ae9a0 0x7f4cf40aee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:57.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.836+0000 7f4cf8b26640 1 -- 172.21.15.57:0/1835806886 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4cf41387d0 msgr2=0x7f4cf4138c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:57.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.836+0000 7f4cf8b26640 1 --2- 172.21.15.57:0/1835806886 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4cf41387d0 0x7f4cf4138c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:57.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.836+0000 7f4cf8b26640 1 -- 172.21.15.57:0/1835806886 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cf40af710 con 0x7f4cdc003970 2023-10-19T14:20:57.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.836+0000 7f4cf8b26640 1 --2- 172.21.15.57:0/1835806886 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4cdc003970 0x7f4cf4136240 secure :-1 s=READY pgs=2558 cs=0 l=1 rev1=1 crypto rx=0x7f4cec02cef0 tx=0x7f4cec031770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:57.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.836+0000 7f4cf17fa640 1 -- 172.21.15.57:0/1835806886 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cec03f070 con 0x7f4cdc003970 2023-10-19T14:20:57.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.836+0000 7f4cf17fa640 1 -- 172.21.15.57:0/1835806886 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f4cec03a760 con 0x7f4cdc003970 2023-10-19T14:20:57.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.836+0000 7f4cfad8a640 1 -- 172.21.15.57:0/1835806886 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cf40af9f0 con 0x7f4cdc003970 2023-10-19T14:20:57.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.836+0000 7f4cf17fa640 1 -- 172.21.15.57:0/1835806886 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cec03aad0 con 0x7f4cdc003970 2023-10-19T14:20:57.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.836+0000 7f4cfad8a640 1 -- 172.21.15.57:0/1835806886 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cf40affb0 con 0x7f4cdc003970 2023-10-19T14:20:57.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.837+0000 7f4cfad8a640 1 -- 172.21.15.57:0/1835806886 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cf40b01a0 con 0x7f4cdc003970 2023-10-19T14:20:57.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.838+0000 7f4cf17fa640 1 -- 172.21.15.57:0/1835806886 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f4cec03ac30 con 0x7f4cdc003970 2023-10-19T14:20:57.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.838+0000 7f4cf17fa640 1 --2- 172.21.15.57:0/1835806886 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f4ce006b080 0x7f4ce006d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:57.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.838+0000 7f4cf17fa640 1 -- 172.21.15.57:0/1835806886 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 6972+0+0 (secure 0 0 0) 0x7f4cec0accb0 con 0x7f4cdc003970 2023-10-19T14:20:57.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.839+0000 7f4cf3fff640 1 --2- 172.21.15.57:0/1835806886 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f4ce006b080 0x7f4ce006d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:57.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.839+0000 7f4cf3fff640 1 --2- 172.21.15.57:0/1835806886 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f4ce006b080 0x7f4ce006d540 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f4ce8001200 tx=0x7f4ce8001110 comp rx=0 tx=0).ready entity=mgr.9530 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:57.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:57.843+0000 7f4cf17fa640 1 -- 172.21.15.57:0/1835806886 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4cec0773f0 con 0x7f4cdc003970 2023-10-19T14:20:58.058 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:58.056+0000 7f4cfad8a640 1 -- 172.21.15.57:0/1835806886 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "backup_fs"} v 0) v1 -- 0x7f4cf40690e0 con 0x7f4cdc003970 2023-10-19T14:20:58.059 INFO:tasks.ceph.mon.a.smithi057.stderr:2023-10-19T14:20:58.056+0000 7f1253ff3640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-19T14:20:58.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:58.062+0000 7f4cf17fa640 1 -- 172.21.15.57:0/1835806886 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "backup_fs"}]=0 backup_fs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v97) v1 ==== 167+0+0 (secure 0 0 0) 0x7f4cec049090 con 0x7f4cdc003970 2023-10-19T14:20:58.065 INFO:teuthology.orchestra.run.smithi057.stderr:backup_fs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-19T14:20:58.067 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:58.064+0000 7f4cfad8a640 1 -- 172.21.15.57:0/1835806886 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f4ce006b080 msgr2=0x7f4ce006d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:58.067 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:58.064+0000 7f4cfad8a640 1 --2- 172.21.15.57:0/1835806886 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f4ce006b080 0x7f4ce006d540 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f4ce8001200 tx=0x7f4ce8001110 comp rx=0 tx=0).stop 2023-10-19T14:20:58.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:58.064+0000 7f4cfad8a640 1 -- 172.21.15.57:0/1835806886 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4cdc003970 msgr2=0x7f4cf4136240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:58.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:58.064+0000 7f4cfad8a640 1 --2- 172.21.15.57:0/1835806886 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4cdc003970 0x7f4cf4136240 secure :-1 s=READY pgs=2558 cs=0 l=1 rev1=1 crypto rx=0x7f4cec02cef0 tx=0x7f4cec031770 comp rx=0 tx=0).stop 2023-10-19T14:20:58.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:58.064+0000 7f4cfad8a640 1 -- 172.21.15.57:0/1835806886 shutdown_connections 2023-10-19T14:20:58.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:58.064+0000 7f4cfad8a640 1 --2- 172.21.15.57:0/1835806886 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f4ce006b080 0x7f4ce006d540 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:58.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:58.064+0000 7f4cfad8a640 1 --2- 172.21.15.57:0/1835806886 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4cf40ae9a0 0x7f4cf40aee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:58.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:58.064+0000 7f4cfad8a640 1 --2- 172.21.15.57:0/1835806886 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4cf41387d0 0x7f4cf4138c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:58.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:58.064+0000 7f4cfad8a640 1 --2- 172.21.15.57:0/1835806886 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4cdc003970 0x7f4cf4136240 unknown :-1 s=CLOSED pgs=2558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:58.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:58.064+0000 7f4cfad8a640 1 -- 172.21.15.57:0/1835806886 >> 172.21.15.57:0/1835806886 conn(0x7f4cf4069930 msgr2=0x7f4cf4158100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:58.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:58.064+0000 7f4cfad8a640 1 -- 172.21.15.57:0/1835806886 shutdown_connections 2023-10-19T14:20:58.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:58.065+0000 7f4cfad8a640 1 -- 172.21.15.57:0/1835806886 wait complete. 2023-10-19T14:20:58.079 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm backup_fs --yes-i-really-mean-it 2023-10-19T14:20:58.095 INFO:tasks.ceph.mds.c.smithi057.stdout:starting mds.c at 2023-10-19T14:20:58.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:58.188+0000 7febb2ec9640 1 -- 172.21.15.57:0/3892480559 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7febac1579f0 msgr2=0x7febac0b0e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:58.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:58.188+0000 7febb2ec9640 1 --2- 172.21.15.57:0/3892480559 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7febac1579f0 0x7febac0b0e90 secure :-1 s=READY pgs=2561 cs=0 l=1 rev1=1 crypto rx=0x7feba8009830 tx=0x7feba802bd60 comp rx=0 tx=0).stop 2023-10-19T14:20:58.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:58.188+0000 7febb2ec9640 1 -- 172.21.15.57:0/3892480559 shutdown_connections 2023-10-19T14:20:58.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:58.188+0000 7febb2ec9640 1 --2- 172.21.15.57:0/3892480559 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7febac06ae40 0x7febac157dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:58.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:58.188+0000 7febb2ec9640 1 --2- 172.21.15.57:0/3892480559 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7febac1579f0 0x7febac0b0e90 unknown :-1 s=CLOSED pgs=2561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:58.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:58.188+0000 7febb2ec9640 1 -- 172.21.15.57:0/3892480559 >> 172.21.15.57:0/3892480559 conn(0x7febac06fae0 msgr2=0x7febac06fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:58.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:58.188+0000 7febb2ec9640 1 -- 172.21.15.57:0/3892480559 shutdown_connections 2023-10-19T14:20:58.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:58.188+0000 7febb2ec9640 1 -- 172.21.15.57:0/3892480559 wait complete. 2023-10-19T14:20:58.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:58.189+0000 7febb2ec9640 1 Processor -- start 2023-10-19T14:20:58.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:58.189+0000 7febb2ec9640 1 -- start start 2023-10-19T14:20:58.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:58.190+0000 7febb2ec9640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7febac06ae40 0x7febac1eebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:58.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:58.190+0000 7febb2ec9640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7febac1579f0 0x7febac14aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:58.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:58.190+0000 7febb2ec9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7febac15c1f0 0x7febac14c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:58.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:58.190+0000 7febb2ec9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febac13f7b0 con 0x7febac15c1f0 2023-10-19T14:20:58.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:58.190+0000 7febb2ec9640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febac153eb0 con 0x7febac06ae40 2023-10-19T14:20:58.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:58.190+0000 7febb2ec9640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febac1ef0f0 con 0x7febac1579f0 2023-10-19T14:20:58.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:58.190+0000 7febb1466640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7febac15c1f0 0x7febac14c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:58.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:58.190+0000 7febb1466640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7febac15c1f0 0x7febac14c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38468/0 (socket says 172.21.15.57:38468) 2023-10-19T14:20:58.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:58.190+0000 7febb1466640 1 -- 172.21.15.57:0/3456245017 learned_addr learned my addr 172.21.15.57:0/3456245017 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:58.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:58.190+0000 7feba3fff640 1 --2- 172.21.15.57:0/3456245017 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7febac1579f0 0x7febac14aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:58.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:58.190+0000 7febb0c65640 1 --2- 172.21.15.57:0/3456245017 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7febac06ae40 0x7febac1eebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:58.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:58.190+0000 7febb1466640 1 -- 172.21.15.57:0/3456245017 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7febac1579f0 msgr2=0x7febac14aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:58.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:58.190+0000 7febb1466640 1 --2- 172.21.15.57:0/3456245017 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7febac1579f0 0x7febac14aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:58.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:58.190+0000 7febb1466640 1 -- 172.21.15.57:0/3456245017 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7febac06ae40 msgr2=0x7febac1eebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:58.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:58.190+0000 7febb1466640 1 --2- 172.21.15.57:0/3456245017 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7febac06ae40 0x7febac1eebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:58.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:58.190+0000 7febb1466640 1 -- 172.21.15.57:0/3456245017 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feba8009490 con 0x7febac15c1f0 2023-10-19T14:20:58.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:58.190+0000 7febb1466640 1 --2- 172.21.15.57:0/3456245017 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7febac15c1f0 0x7febac14c670 secure :-1 s=READY pgs=2562 cs=0 l=1 rev1=1 crypto rx=0x7feba4009fd0 tx=0x7feba400ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:58.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:58.190+0000 7feba17fa640 1 -- 172.21.15.57:0/3456245017 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feba4004d20 con 0x7febac15c1f0 2023-10-19T14:20:58.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:58.190+0000 7feba17fa640 1 -- 172.21.15.57:0/3456245017 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7feba400cb20 con 0x7febac15c1f0 2023-10-19T14:20:58.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:58.190+0000 7febb2ec9640 1 -- 172.21.15.57:0/3456245017 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febac14b480 con 0x7febac15c1f0 2023-10-19T14:20:58.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:58.190+0000 7feba17fa640 1 -- 172.21.15.57:0/3456245017 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feba4005020 con 0x7febac15c1f0 2023-10-19T14:20:58.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:58.190+0000 7febb2ec9640 1 -- 172.21.15.57:0/3456245017 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febac14ba40 con 0x7febac15c1f0 2023-10-19T14:20:58.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:58.191+0000 7febb2ec9640 1 -- 172.21.15.57:0/3456245017 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febac154020 con 0x7febac15c1f0 2023-10-19T14:20:58.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:58.192+0000 7feba17fa640 1 -- 172.21.15.57:0/3456245017 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7feba4005180 con 0x7febac15c1f0 2023-10-19T14:20:58.201 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:58.192+0000 7feba17fa640 1 --2- 172.21.15.57:0/3456245017 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7feb9806b080 0x7feb9806d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:58.201 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:58.193+0000 7feba17fa640 1 -- 172.21.15.57:0/3456245017 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(233..233 src has 1..233) v4 ==== 7058+0+0 (secure 0 0 0) 0x7feba408aac0 con 0x7febac15c1f0 2023-10-19T14:20:58.201 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:58.193+0000 7febb0c65640 1 --2- 172.21.15.57:0/3456245017 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7feb9806b080 0x7feb9806d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:58.201 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:58.193+0000 7febb0c65640 1 --2- 172.21.15.57:0/3456245017 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7feb9806b080 0x7feb9806d540 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7feba8036000 tx=0x7feba8004730 comp rx=0 tx=0).ready entity=mgr.9530 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:58.201 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:58.197+0000 7feba17fa640 1 -- 172.21.15.57:0/3456245017 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7febac154020 con 0x7febac15c1f0 2023-10-19T14:20:58.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:58.408+0000 7febb2ec9640 1 -- 172.21.15.57:0/3456245017 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "backup_fs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7febac154020 con 0x7febac15c1f0 2023-10-19T14:20:59.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:59.068+0000 7feba17fa640 1 -- 172.21.15.57:0/3456245017 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "backup_fs", "yes_i_really_mean_it": true}]=0 v98) v1 ==== 107+0+0 (secure 0 0 0) 0x7febac154020 con 0x7febac15c1f0 2023-10-19T14:20:59.073 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:59.070+0000 7febb2ec9640 1 -- 172.21.15.57:0/3456245017 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7feb9806b080 msgr2=0x7feb9806d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:59.074 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:59.070+0000 7febb2ec9640 1 --2- 172.21.15.57:0/3456245017 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7feb9806b080 0x7feb9806d540 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7feba8036000 tx=0x7feba8004730 comp rx=0 tx=0).stop 2023-10-19T14:20:59.074 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:59.070+0000 7febb2ec9640 1 -- 172.21.15.57:0/3456245017 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7febac15c1f0 msgr2=0x7febac14c670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:59.074 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:59.070+0000 7febb2ec9640 1 --2- 172.21.15.57:0/3456245017 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7febac15c1f0 0x7febac14c670 secure :-1 s=READY pgs=2562 cs=0 l=1 rev1=1 crypto rx=0x7feba4009fd0 tx=0x7feba400ee60 comp rx=0 tx=0).stop 2023-10-19T14:20:59.074 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:59.071+0000 7febb2ec9640 1 -- 172.21.15.57:0/3456245017 shutdown_connections 2023-10-19T14:20:59.074 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:59.071+0000 7febb2ec9640 1 --2- 172.21.15.57:0/3456245017 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7feb9806b080 0x7feb9806d540 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:59.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:59.071+0000 7febb2ec9640 1 --2- 172.21.15.57:0/3456245017 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7febac15c1f0 0x7febac14c670 unknown :-1 s=CLOSED pgs=2562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:59.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:59.071+0000 7febb2ec9640 1 --2- 172.21.15.57:0/3456245017 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7febac1579f0 0x7febac14aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:59.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:59.071+0000 7febb2ec9640 1 --2- 172.21.15.57:0/3456245017 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7febac06ae40 0x7febac1eebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:59.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:59.071+0000 7febb2ec9640 1 -- 172.21.15.57:0/3456245017 >> 172.21.15.57:0/3456245017 conn(0x7febac06fae0 msgr2=0x7febac06b980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:59.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:59.071+0000 7febb2ec9640 1 -- 172.21.15.57:0/3456245017 shutdown_connections 2023-10-19T14:20:59.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:59.071+0000 7febb2ec9640 1 -- 172.21.15.57:0/3456245017 wait complete. 2023-10-19T14:20:59.087 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm backup_fs_metadata backup_fs_metadata --yes-i-really-really-mean-it 2023-10-19T14:20:59.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:59.245+0000 7f4fe2694640 1 -- 172.21.15.57:0/3503151232 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4fdc070350 msgr2=0x7f4fdc070780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:59.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:59.245+0000 7f4fe2694640 1 --2- 172.21.15.57:0/3503151232 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4fdc070350 0x7f4fdc070780 secure :-1 s=READY pgs=2564 cs=0 l=1 rev1=1 crypto rx=0x7f4fdc0cc560 tx=0x7f4fd402c110 comp rx=0 tx=0).stop 2023-10-19T14:20:59.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:59.245+0000 7f4fe2694640 1 -- 172.21.15.57:0/3503151232 shutdown_connections 2023-10-19T14:20:59.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:59.245+0000 7f4fe2694640 1 --2- 172.21.15.57:0/3503151232 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4fdc070350 0x7f4fdc070780 unknown :-1 s=CLOSED pgs=2564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:59.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:59.245+0000 7f4fe2694640 1 --2- 172.21.15.57:0/3503151232 >> v2:172.21.15.179:3300/0 conn(0x7f4fdc0cb980 0x7f4fdc13db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:59.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:59.245+0000 7f4fe2694640 1 -- 172.21.15.57:0/3503151232 >> 172.21.15.57:0/3503151232 conn(0x7f4fdc069930 msgr2=0x7f4fdc069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:20:59.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:59.245+0000 7f4fe2694640 1 -- 172.21.15.57:0/3503151232 shutdown_connections 2023-10-19T14:20:59.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:59.245+0000 7f4fe2694640 1 -- 172.21.15.57:0/3503151232 wait complete. 2023-10-19T14:20:59.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:59.246+0000 7f4fe2694640 1 Processor -- start 2023-10-19T14:20:59.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:59.246+0000 7f4fe2694640 1 -- start start 2023-10-19T14:20:59.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:59.247+0000 7f4fe2694640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4fdc070350 0x7f4fdc1f2130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:59.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:59.247+0000 7f4fe2694640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4fdc0cb980 0x7f4fdc0ad5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:59.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:59.247+0000 7f4fe2694640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4fdc0adb20 0x7f4fdc138720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:59.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:59.247+0000 7f4fe2694640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fdc0adfb0 con 0x7f4fdc070350 2023-10-19T14:20:59.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:59.247+0000 7f4fe2694640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fdc0b1560 con 0x7f4fdc0cb980 2023-10-19T14:20:59.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:59.247+0000 7f4fe2694640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fdc139370 con 0x7f4fdc0adb20 2023-10-19T14:20:59.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:59.247+0000 7f4fdbfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4fdc070350 0x7f4fdc1f2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:59.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:59.247+0000 7f4fdbfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4fdc070350 0x7f4fdc1f2130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38490/0 (socket says 172.21.15.57:38490) 2023-10-19T14:20:59.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:59.247+0000 7f4fdbfff640 1 -- 172.21.15.57:0/2236881366 learned_addr learned my addr 172.21.15.57:0/2236881366 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:20:59.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:59.247+0000 7f4fe0c31640 1 --2- 172.21.15.57:0/2236881366 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4fdc0adb20 0x7f4fdc138720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:59.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:59.247+0000 7f4fdb7fe640 1 --2- 172.21.15.57:0/2236881366 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4fdc0cb980 0x7f4fdc0ad5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:59.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:59.247+0000 7f4fdbfff640 1 -- 172.21.15.57:0/2236881366 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4fdc0adb20 msgr2=0x7f4fdc138720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:59.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:59.247+0000 7f4fdbfff640 1 --2- 172.21.15.57:0/2236881366 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4fdc0adb20 0x7f4fdc138720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:59.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:59.247+0000 7f4fdbfff640 1 -- 172.21.15.57:0/2236881366 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4fdc0cb980 msgr2=0x7f4fdc0ad5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:20:59.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:59.247+0000 7f4fdbfff640 1 --2- 172.21.15.57:0/2236881366 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4fdc0cb980 0x7f4fdc0ad5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:20:59.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:59.247+0000 7f4fdbfff640 1 -- 172.21.15.57:0/2236881366 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fd4009840 con 0x7f4fdc070350 2023-10-19T14:20:59.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:59.247+0000 7f4fe0c31640 1 --2- 172.21.15.57:0/2236881366 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4fdc0adb20 0x7f4fdc138720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:20:59.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:59.247+0000 7f4fdb7fe640 1 --2- 172.21.15.57:0/2236881366 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4fdc0cb980 0x7f4fdc0ad5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:20:59.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:59.248+0000 7f4fdbfff640 1 --2- 172.21.15.57:0/2236881366 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4fdc070350 0x7f4fdc1f2130 secure :-1 s=READY pgs=2565 cs=0 l=1 rev1=1 crypto rx=0x7f4fcc00b4a0 tx=0x7f4fcc00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:59.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:59.248+0000 7f4fd8ff9640 1 -- 172.21.15.57:0/2236881366 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fcc010610 con 0x7f4fdc070350 2023-10-19T14:20:59.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:59.248+0000 7f4fd8ff9640 1 -- 172.21.15.57:0/2236881366 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f4fcc010860 con 0x7f4fdc070350 2023-10-19T14:20:59.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:59.248+0000 7f4fe2694640 1 -- 172.21.15.57:0/2236881366 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fdc139650 con 0x7f4fdc070350 2023-10-19T14:20:59.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:59.248+0000 7f4fd8ff9640 1 -- 172.21.15.57:0/2236881366 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fcc010c70 con 0x7f4fdc070350 2023-10-19T14:20:59.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:59.248+0000 7f4fe2694640 1 -- 172.21.15.57:0/2236881366 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fdc139c10 con 0x7f4fdc070350 2023-10-19T14:20:59.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:59.249+0000 7f4fe2694640 1 -- 172.21.15.57:0/2236881366 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fdc070c40 con 0x7f4fdc070350 2023-10-19T14:20:59.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:59.249+0000 7f4fd8ff9640 1 -- 172.21.15.57:0/2236881366 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f4fcc017360 con 0x7f4fdc070350 2023-10-19T14:20:59.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:59.250+0000 7f4fd8ff9640 1 --2- 172.21.15.57:0/2236881366 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f4fc806b080 0x7f4fc806d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:20:59.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:59.250+0000 7f4fd8ff9640 1 -- 172.21.15.57:0/2236881366 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(233..233 src has 1..233) v4 ==== 7058+0+0 (secure 0 0 0) 0x7f4fcc089e30 con 0x7f4fdc070350 2023-10-19T14:20:59.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:59.250+0000 7f4fdb7fe640 1 --2- 172.21.15.57:0/2236881366 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f4fc806b080 0x7f4fc806d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:20:59.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:59.251+0000 7f4fdb7fe640 1 --2- 172.21.15.57:0/2236881366 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f4fc806b080 0x7f4fc806d540 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4fdc0b1740 tx=0x7f4fd0009040 comp rx=0 tx=0).ready entity=mgr.9530 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:20:59.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:59.254+0000 7f4fd8ff9640 1 -- 172.21.15.57:0/2236881366 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4fcc0580c0 con 0x7f4fdc070350 2023-10-19T14:20:59.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:20:59.431+0000 7f4fe2694640 1 -- 172.21.15.57:0/2236881366 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "backup_fs_metadata", "pool2": "backup_fs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f4fdc0690e0 con 0x7f4fdc070350 2023-10-19T14:21:00.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:00.337+0000 7f4fd8ff9640 1 -- 172.21.15.57:0/2236881366 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "backup_fs_metadata", "pool2": "backup_fs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'backup_fs_metadata' removed v234) v1 ==== 190+0+0 (secure 0 0 0) 0x7f4fcc0140e0 con 0x7f4fdc070350 2023-10-19T14:21:00.341 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'backup_fs_metadata' removed 2023-10-19T14:21:00.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:00.340+0000 7f4fe2694640 1 -- 172.21.15.57:0/2236881366 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f4fc806b080 msgr2=0x7f4fc806d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:00.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:00.340+0000 7f4fe2694640 1 --2- 172.21.15.57:0/2236881366 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f4fc806b080 0x7f4fc806d540 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4fdc0b1740 tx=0x7f4fd0009040 comp rx=0 tx=0).stop 2023-10-19T14:21:00.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:00.340+0000 7f4fe2694640 1 -- 172.21.15.57:0/2236881366 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4fdc070350 msgr2=0x7f4fdc1f2130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:00.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:00.340+0000 7f4fe2694640 1 --2- 172.21.15.57:0/2236881366 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4fdc070350 0x7f4fdc1f2130 secure :-1 s=READY pgs=2565 cs=0 l=1 rev1=1 crypto rx=0x7f4fcc00b4a0 tx=0x7f4fcc00b960 comp rx=0 tx=0).stop 2023-10-19T14:21:00.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:00.340+0000 7f4fe2694640 1 -- 172.21.15.57:0/2236881366 shutdown_connections 2023-10-19T14:21:00.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:00.340+0000 7f4fe2694640 1 --2- 172.21.15.57:0/2236881366 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f4fc806b080 0x7f4fc806d540 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:00.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:00.340+0000 7f4fe2694640 1 --2- 172.21.15.57:0/2236881366 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4fdc0adb20 0x7f4fdc138720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:00.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:00.340+0000 7f4fe2694640 1 --2- 172.21.15.57:0/2236881366 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4fdc0cb980 0x7f4fdc0ad5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:00.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:00.340+0000 7f4fe2694640 1 --2- 172.21.15.57:0/2236881366 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4fdc070350 0x7f4fdc1f2130 unknown :-1 s=CLOSED pgs=2565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:00.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:00.340+0000 7f4fe2694640 1 -- 172.21.15.57:0/2236881366 >> 172.21.15.57:0/2236881366 conn(0x7f4fdc069930 msgr2=0x7f4fdc156560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:00.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:00.341+0000 7f4fe2694640 1 -- 172.21.15.57:0/2236881366 shutdown_connections 2023-10-19T14:21:00.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:00.341+0000 7f4fe2694640 1 -- 172.21.15.57:0/2236881366 wait complete. 2023-10-19T14:21:00.360 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm backup_fs_data backup_fs_data --yes-i-really-really-mean-it 2023-10-19T14:21:00.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:00.477+0000 7f5ea2226640 1 -- 172.21.15.57:0/1749703296 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5e9c15c1f0 msgr2=0x7f5e9c06a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:00.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:00.477+0000 7f5ea2226640 1 --2- 172.21.15.57:0/1749703296 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5e9c15c1f0 0x7f5e9c06a730 secure :-1 s=READY pgs=2566 cs=0 l=1 rev1=1 crypto rx=0x7f5e9c0662b0 tx=0x7f5e9002bdb0 comp rx=0 tx=0).stop 2023-10-19T14:21:00.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:00.477+0000 7f5ea2226640 1 -- 172.21.15.57:0/1749703296 shutdown_connections 2023-10-19T14:21:00.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:00.477+0000 7f5ea2226640 1 --2- 172.21.15.57:0/1749703296 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5e9c15c1f0 0x7f5e9c06a730 unknown :-1 s=CLOSED pgs=2566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:00.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:00.477+0000 7f5ea2226640 1 --2- 172.21.15.57:0/1749703296 >> v2:172.21.15.179:3300/0 conn(0x7f5e9c1579f0 0x7f5e9c0b0e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:00.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:00.477+0000 7f5ea2226640 1 -- 172.21.15.57:0/1749703296 >> 172.21.15.57:0/1749703296 conn(0x7f5e9c06fae0 msgr2=0x7f5e9c06fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:00.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:00.477+0000 7f5ea2226640 1 -- 172.21.15.57:0/1749703296 shutdown_connections 2023-10-19T14:21:00.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:00.477+0000 7f5ea2226640 1 -- 172.21.15.57:0/1749703296 wait complete. 2023-10-19T14:21:00.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:00.478+0000 7f5ea2226640 1 Processor -- start 2023-10-19T14:21:00.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:00.478+0000 7f5ea2226640 1 -- start start 2023-10-19T14:21:00.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:00.479+0000 7f5ea2226640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5e9c06ac70 0x7f5e9c14d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:00.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:00.479+0000 7f5ea2226640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5e9c1579f0 0x7f5e9c0ad3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:00.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:00.479+0000 7f5ea2226640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5e9c15c1f0 0x7f5e9c1483b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:00.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:00.479+0000 7f5ea2226640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e9c0ad910 con 0x7f5e9c15c1f0 2023-10-19T14:21:00.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:00.479+0000 7f5ea2226640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e9c0adb20 con 0x7f5e9c06ac70 2023-10-19T14:21:00.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:00.479+0000 7f5ea2226640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e9c0ade00 con 0x7f5e9c1579f0 2023-10-19T14:21:00.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:00.479+0000 7f5e9bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5e9c15c1f0 0x7f5e9c1483b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:00.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:00.479+0000 7f5e9bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5e9c15c1f0 0x7f5e9c1483b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38500/0 (socket says 172.21.15.57:38500) 2023-10-19T14:21:00.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:00.479+0000 7f5e9bfff640 1 -- 172.21.15.57:0/2595407832 learned_addr learned my addr 172.21.15.57:0/2595407832 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:00.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:00.479+0000 7f5e9bfff640 1 -- 172.21.15.57:0/2595407832 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5e9c1579f0 msgr2=0x7f5e9c0ad3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:00.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:00.479+0000 7f5e9bfff640 1 --2- 172.21.15.57:0/2595407832 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5e9c1579f0 0x7f5e9c0ad3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:00.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:00.479+0000 7f5e9bfff640 1 -- 172.21.15.57:0/2595407832 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5e9c06ac70 msgr2=0x7f5e9c14d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:00.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:00.479+0000 7f5e9b7fe640 1 --2- 172.21.15.57:0/2595407832 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5e9c06ac70 0x7f5e9c14d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:00.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:00.479+0000 7f5e9bfff640 1 --2- 172.21.15.57:0/2595407832 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5e9c06ac70 0x7f5e9c14d4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:00.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:00.479+0000 7f5e9bfff640 1 -- 172.21.15.57:0/2595407832 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e90009560 con 0x7f5e9c15c1f0 2023-10-19T14:21:00.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:00.480+0000 7f5e9bfff640 1 --2- 172.21.15.57:0/2595407832 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5e9c15c1f0 0x7f5e9c1483b0 secure :-1 s=READY pgs=2567 cs=0 l=1 rev1=1 crypto rx=0x7f5e9400c780 tx=0x7f5e9400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:00.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:00.480+0000 7f5e83fff640 1 -- 172.21.15.57:0/2595407832 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e94004b60 con 0x7f5e9c15c1f0 2023-10-19T14:21:00.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:00.480+0000 7f5e83fff640 1 -- 172.21.15.57:0/2595407832 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f5e9400ce90 con 0x7f5e9c15c1f0 2023-10-19T14:21:00.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:00.480+0000 7f5e83fff640 1 -- 172.21.15.57:0/2595407832 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e9400aab0 con 0x7f5e9c15c1f0 2023-10-19T14:21:00.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:00.480+0000 7f5ea2226640 1 -- 172.21.15.57:0/2595407832 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e9c0ae0e0 con 0x7f5e9c15c1f0 2023-10-19T14:21:00.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:00.480+0000 7f5ea2226640 1 -- 172.21.15.57:0/2595407832 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e9c0ae6a0 con 0x7f5e9c15c1f0 2023-10-19T14:21:00.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:00.481+0000 7f5ea2226640 1 -- 172.21.15.57:0/2595407832 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e9c0690e0 con 0x7f5e9c15c1f0 2023-10-19T14:21:00.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:00.481+0000 7f5e83fff640 1 -- 172.21.15.57:0/2595407832 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f5e9400ac10 con 0x7f5e9c15c1f0 2023-10-19T14:21:00.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:00.482+0000 7f5e83fff640 1 --2- 172.21.15.57:0/2595407832 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f5e8806b0b0 0x7f5e8806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:00.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:00.482+0000 7f5e83fff640 1 -- 172.21.15.57:0/2595407832 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 6614+0+0 (secure 0 0 0) 0x7f5e9408a190 con 0x7f5e9c15c1f0 2023-10-19T14:21:00.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:00.482+0000 7f5e9b7fe640 1 --2- 172.21.15.57:0/2595407832 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f5e8806b0b0 0x7f5e8806d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:00.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:00.483+0000 7f5e9b7fe640 1 --2- 172.21.15.57:0/2595407832 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f5e8806b0b0 0x7f5e8806d570 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5e8c005fd0 tx=0x7f5e8c005850 comp rx=0 tx=0).ready entity=mgr.9530 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:00.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:00.488+0000 7f5e83fff640 1 -- 172.21.15.57:0/2595407832 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f5e94054b50 con 0x7f5e9c15c1f0 2023-10-19T14:21:00.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:00.666+0000 7f5ea2226640 1 -- 172.21.15.57:0/2595407832 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "backup_fs_data", "pool2": "backup_fs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f5e9c06a560 con 0x7f5e9c15c1f0 2023-10-19T14:21:01.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.343+0000 7f5e83fff640 1 -- 172.21.15.57:0/2595407832 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "backup_fs_data", "pool2": "backup_fs_data", "yes_i_really_really_mean_it": true}]=0 pool 'backup_fs_data' removed v235) v1 ==== 178+0+0 (secure 0 0 0) 0x7f5e94058170 con 0x7f5e9c15c1f0 2023-10-19T14:21:01.346 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'backup_fs_data' removed 2023-10-19T14:21:01.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.345+0000 7f5ea2226640 1 -- 172.21.15.57:0/2595407832 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f5e8806b0b0 msgr2=0x7f5e8806d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:01.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.345+0000 7f5ea2226640 1 --2- 172.21.15.57:0/2595407832 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f5e8806b0b0 0x7f5e8806d570 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5e8c005fd0 tx=0x7f5e8c005850 comp rx=0 tx=0).stop 2023-10-19T14:21:01.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.346+0000 7f5ea2226640 1 -- 172.21.15.57:0/2595407832 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5e9c15c1f0 msgr2=0x7f5e9c1483b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:01.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.346+0000 7f5ea2226640 1 --2- 172.21.15.57:0/2595407832 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5e9c15c1f0 0x7f5e9c1483b0 secure :-1 s=READY pgs=2567 cs=0 l=1 rev1=1 crypto rx=0x7f5e9400c780 tx=0x7f5e9400cc40 comp rx=0 tx=0).stop 2023-10-19T14:21:01.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.346+0000 7f5ea2226640 1 -- 172.21.15.57:0/2595407832 shutdown_connections 2023-10-19T14:21:01.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.346+0000 7f5ea2226640 1 --2- 172.21.15.57:0/2595407832 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f5e8806b0b0 0x7f5e8806d570 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:01.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.346+0000 7f5ea2226640 1 --2- 172.21.15.57:0/2595407832 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5e9c15c1f0 0x7f5e9c1483b0 unknown :-1 s=CLOSED pgs=2567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:01.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.346+0000 7f5ea2226640 1 --2- 172.21.15.57:0/2595407832 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5e9c1579f0 0x7f5e9c0ad3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:01.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.346+0000 7f5ea2226640 1 --2- 172.21.15.57:0/2595407832 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5e9c06ac70 0x7f5e9c14d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:01.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.346+0000 7f5ea2226640 1 -- 172.21.15.57:0/2595407832 >> 172.21.15.57:0/2595407832 conn(0x7f5e9c06fae0 msgr2=0x7f5e9c159120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:01.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.346+0000 7f5ea2226640 1 -- 172.21.15.57:0/2595407832 shutdown_connections 2023-10-19T14:21:01.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.346+0000 7f5ea2226640 1 -- 172.21.15.57:0/2595407832 wait complete. 2023-10-19T14:21:01.368 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 30 2023-10-19T14:21:01.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.482+0000 7f13704c9640 1 -- 172.21.15.57:0/570921857 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1368070af0 msgr2=0x7f13680affe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:01.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.482+0000 7f13704c9640 1 --2- 172.21.15.57:0/570921857 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1368070af0 0x7f13680affe0 secure :-1 s=READY pgs=2569 cs=0 l=1 rev1=1 crypto rx=0x7f1360005fd0 tx=0x7f136002c050 comp rx=0 tx=0).stop 2023-10-19T14:21:01.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.482+0000 7f13704c9640 1 -- 172.21.15.57:0/570921857 shutdown_connections 2023-10-19T14:21:01.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.482+0000 7f13704c9640 1 --2- 172.21.15.57:0/570921857 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1368070af0 0x7f13680affe0 unknown :-1 s=CLOSED pgs=2569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:01.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.482+0000 7f13704c9640 1 --2- 172.21.15.57:0/570921857 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f136813e520 0x7f136806bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:01.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.482+0000 7f13704c9640 1 -- 172.21.15.57:0/570921857 >> 172.21.15.57:0/570921857 conn(0x7f1368069930 msgr2=0x7f1368069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:01.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.482+0000 7f13704c9640 1 -- 172.21.15.57:0/570921857 shutdown_connections 2023-10-19T14:21:01.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.482+0000 7f13704c9640 1 -- 172.21.15.57:0/570921857 wait complete. 2023-10-19T14:21:01.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.483+0000 7f13704c9640 1 Processor -- start 2023-10-19T14:21:01.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.483+0000 7f13704c9640 1 -- start start 2023-10-19T14:21:01.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.484+0000 7f13704c9640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1368070af0 0x7f13680a6ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:01.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.484+0000 7f13704c9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f136813e520 0x7f13681e2c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:01.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.484+0000 7f13704c9640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1368141690 0x7f136813ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:01.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.484+0000 7f13704c9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13681e3150 con 0x7f136813e520 2023-10-19T14:21:01.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.484+0000 7f13704c9640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13680a7560 con 0x7f1368070af0 2023-10-19T14:21:01.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.484+0000 7f13704c9640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13680a7840 con 0x7f1368141690 2023-10-19T14:21:01.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.484+0000 7f136da64640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f136813e520 0x7f13681e2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:01.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.484+0000 7f136da64640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f136813e520 0x7f13681e2c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38518/0 (socket says 172.21.15.57:38518) 2023-10-19T14:21:01.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.484+0000 7f136da64640 1 -- 172.21.15.57:0/2042399466 learned_addr learned my addr 172.21.15.57:0/2042399466 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:01.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.484+0000 7f136e265640 1 --2- 172.21.15.57:0/2042399466 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1368070af0 0x7f13680a6ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:01.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.484+0000 7f136ea66640 1 --2- 172.21.15.57:0/2042399466 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1368141690 0x7f136813ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:01.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.484+0000 7f136da64640 1 -- 172.21.15.57:0/2042399466 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1368141690 msgr2=0x7f136813ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:01.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.484+0000 7f136da64640 1 --2- 172.21.15.57:0/2042399466 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1368141690 0x7f136813ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:01.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.484+0000 7f136da64640 1 -- 172.21.15.57:0/2042399466 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1368070af0 msgr2=0x7f13680a6ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:01.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.484+0000 7f136da64640 1 --2- 172.21.15.57:0/2042399466 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1368070af0 0x7f13680a6ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:01.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.484+0000 7f136da64640 1 -- 172.21.15.57:0/2042399466 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1360009840 con 0x7f136813e520 2023-10-19T14:21:01.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.484+0000 7f136e265640 1 --2- 172.21.15.57:0/2042399466 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1368070af0 0x7f13680a6ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:21:01.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.484+0000 7f136da64640 1 --2- 172.21.15.57:0/2042399466 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f136813e520 0x7f13681e2c10 secure :-1 s=READY pgs=2570 cs=0 l=1 rev1=1 crypto rx=0x7f135800c430 tx=0x7f135800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:01.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.484+0000 7f135effd640 1 -- 172.21.15.57:0/2042399466 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1358011790 con 0x7f136813e520 2023-10-19T14:21:01.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.484+0000 7f135effd640 1 -- 172.21.15.57:0/2042399466 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f1358011a00 con 0x7f136813e520 2023-10-19T14:21:01.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.484+0000 7f135effd640 1 -- 172.21.15.57:0/2042399466 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1358011e10 con 0x7f136813e520 2023-10-19T14:21:01.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.485+0000 7f13704c9640 1 -- 172.21.15.57:0/2042399466 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13680a7b20 con 0x7f136813e520 2023-10-19T14:21:01.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.485+0000 7f13704c9640 1 -- 172.21.15.57:0/2042399466 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13680a80e0 con 0x7f136813e520 2023-10-19T14:21:01.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.486+0000 7f13704c9640 1 -- 172.21.15.57:0/2042399466 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13680690e0 con 0x7f136813e520 2023-10-19T14:21:01.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.486+0000 7f135effd640 1 -- 172.21.15.57:0/2042399466 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f135801c460 con 0x7f136813e520 2023-10-19T14:21:01.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.486+0000 7f135effd640 1 --2- 172.21.15.57:0/2042399466 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f134c06b080 0x7f134c06d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:01.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.487+0000 7f135effd640 1 -- 172.21.15.57:0/2042399466 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 6194+0+0 (secure 0 0 0) 0x7f1358089dc0 con 0x7f136813e520 2023-10-19T14:21:01.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.487+0000 7f136e265640 1 --2- 172.21.15.57:0/2042399466 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f134c06b080 0x7f134c06d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:01.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.487+0000 7f136e265640 1 --2- 172.21.15.57:0/2042399466 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f134c06b080 0x7f134c06d540 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f13680b1a90 tx=0x7f1364005f90 comp rx=0 tx=0).ready entity=mgr.9530 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:01.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.491+0000 7f135effd640 1 -- 172.21.15.57:0/2042399466 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f1358054950 con 0x7f136813e520 2023-10-19T14:21:01.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.667+0000 7f13704c9640 1 -- 172.21.15.57:0/2042399466 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "config reset", "num": 30} v 0) v1 -- 0x7f13680af750 con 0x7f136813e520 2023-10-19T14:21:01.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.671+0000 7f135effd640 1 -- 172.21.15.57:0/2042399466 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 30}]=0 v78) v1 ==== 71+0+0 (secure 0 0 0) 0x7f1358057f70 con 0x7f136813e520 2023-10-19T14:21:01.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.673+0000 7f13704c9640 1 -- 172.21.15.57:0/2042399466 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f134c06b080 msgr2=0x7f134c06d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:01.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.673+0000 7f13704c9640 1 --2- 172.21.15.57:0/2042399466 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f134c06b080 0x7f134c06d540 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f13680b1a90 tx=0x7f1364005f90 comp rx=0 tx=0).stop 2023-10-19T14:21:01.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.673+0000 7f13704c9640 1 -- 172.21.15.57:0/2042399466 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f136813e520 msgr2=0x7f13681e2c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:01.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.673+0000 7f13704c9640 1 --2- 172.21.15.57:0/2042399466 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f136813e520 0x7f13681e2c10 secure :-1 s=READY pgs=2570 cs=0 l=1 rev1=1 crypto rx=0x7f135800c430 tx=0x7f135800c8f0 comp rx=0 tx=0).stop 2023-10-19T14:21:01.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.673+0000 7f13704c9640 1 -- 172.21.15.57:0/2042399466 shutdown_connections 2023-10-19T14:21:01.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.673+0000 7f13704c9640 1 --2- 172.21.15.57:0/2042399466 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f134c06b080 0x7f134c06d540 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:01.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.673+0000 7f13704c9640 1 --2- 172.21.15.57:0/2042399466 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1368141690 0x7f136813ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:01.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.673+0000 7f13704c9640 1 --2- 172.21.15.57:0/2042399466 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f136813e520 0x7f13681e2c10 unknown :-1 s=CLOSED pgs=2570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:01.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.673+0000 7f13704c9640 1 --2- 172.21.15.57:0/2042399466 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1368070af0 0x7f13680a6ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:01.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.673+0000 7f13704c9640 1 -- 172.21.15.57:0/2042399466 >> 172.21.15.57:0/2042399466 conn(0x7f1368069930 msgr2=0x7f136813fba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:01.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.673+0000 7f13704c9640 1 -- 172.21.15.57:0/2042399466 shutdown_connections 2023-10-19T14:21:01.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.673+0000 7f13704c9640 1 -- 172.21.15.57:0/2042399466 wait complete. 2023-10-19T14:21:01.687 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs' 2023-10-19T14:21:01.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.793+0000 7fb2d1ba0640 1 -- 172.21.15.57:0/1739111700 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb2cc06e310 msgr2=0x7fb2cc0ca4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:01.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.793+0000 7fb2d1ba0640 1 --2- 172.21.15.57:0/1739111700 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb2cc06e310 0x7fb2cc0ca4a0 secure :-1 s=READY pgs=2572 cs=0 l=1 rev1=1 crypto rx=0x7fb2bc009830 tx=0x7fb2bc02bcd0 comp rx=0 tx=0).stop 2023-10-19T14:21:01.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.793+0000 7fb2d1ba0640 1 -- 172.21.15.57:0/1739111700 shutdown_connections 2023-10-19T14:21:01.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.793+0000 7fb2d1ba0640 1 --2- 172.21.15.57:0/1739111700 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb2cc1413b0 0x7fb2cc06c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:01.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.793+0000 7fb2d1ba0640 1 --2- 172.21.15.57:0/1739111700 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb2cc06e310 0x7fb2cc0ca4a0 unknown :-1 s=CLOSED pgs=2572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:01.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.793+0000 7fb2d1ba0640 1 -- 172.21.15.57:0/1739111700 >> 172.21.15.57:0/1739111700 conn(0x7fb2cc070080 msgr2=0x7fb2cc070490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:01.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.793+0000 7fb2d1ba0640 1 -- 172.21.15.57:0/1739111700 shutdown_connections 2023-10-19T14:21:01.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.793+0000 7fb2d1ba0640 1 -- 172.21.15.57:0/1739111700 wait complete. 2023-10-19T14:21:01.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.794+0000 7fb2d1ba0640 1 Processor -- start 2023-10-19T14:21:01.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.794+0000 7fb2d1ba0640 1 -- start start 2023-10-19T14:21:01.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.795+0000 7fb2d1ba0640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb2cc0b1b40 0x7fb2cc0af150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:01.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.795+0000 7fb2d1ba0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb2cc1413b0 0x7fb2cc1400a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:01.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.795+0000 7fb2d1ba0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb2cc1f0ae0 0x7fb2cc0ab910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:01.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.795+0000 7fb2d1ba0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2cc13faf0 con 0x7fb2cc1f0ae0 2023-10-19T14:21:01.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.795+0000 7fb2d1ba0640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2cc0ad640 con 0x7fb2cc1413b0 2023-10-19T14:21:01.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.795+0000 7fb2d1ba0640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2cc0ad8e0 con 0x7fb2cc0b1b40 2023-10-19T14:21:01.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.795+0000 7fb2d139f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb2cc1f0ae0 0x7fb2cc0ab910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:01.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.795+0000 7fb2d139f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb2cc1f0ae0 0x7fb2cc0ab910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38546/0 (socket says 172.21.15.57:38546) 2023-10-19T14:21:01.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.795+0000 7fb2d139f640 1 -- 172.21.15.57:0/3556333105 learned_addr learned my addr 172.21.15.57:0/3556333105 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:01.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.795+0000 7fb2cbfff640 1 --2- 172.21.15.57:0/3556333105 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb2cc1413b0 0x7fb2cc1400a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:01.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.795+0000 7fb2d0b9e640 1 --2- 172.21.15.57:0/3556333105 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb2cc0b1b40 0x7fb2cc0af150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:01.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.796+0000 7fb2d139f640 1 -- 172.21.15.57:0/3556333105 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb2cc0b1b40 msgr2=0x7fb2cc0af150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:01.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.796+0000 7fb2d139f640 1 --2- 172.21.15.57:0/3556333105 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb2cc0b1b40 0x7fb2cc0af150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:01.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.796+0000 7fb2d139f640 1 -- 172.21.15.57:0/3556333105 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb2cc1413b0 msgr2=0x7fb2cc1400a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:01.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.796+0000 7fb2d139f640 1 --2- 172.21.15.57:0/3556333105 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb2cc1413b0 0x7fb2cc1400a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:01.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.796+0000 7fb2d139f640 1 -- 172.21.15.57:0/3556333105 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2bc009490 con 0x7fb2cc1f0ae0 2023-10-19T14:21:01.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.796+0000 7fb2d139f640 1 --2- 172.21.15.57:0/3556333105 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb2cc1f0ae0 0x7fb2cc0ab910 secure :-1 s=READY pgs=2573 cs=0 l=1 rev1=1 crypto rx=0x7fb2b400d660 tx=0x7fb2b400db20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:01.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.796+0000 7fb2c97fa640 1 -- 172.21.15.57:0/3556333105 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2b4005220 con 0x7fb2cc1f0ae0 2023-10-19T14:21:01.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.796+0000 7fb2c97fa640 1 -- 172.21.15.57:0/3556333105 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fb2b4005470 con 0x7fb2cc1f0ae0 2023-10-19T14:21:01.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.796+0000 7fb2d1ba0640 1 -- 172.21.15.57:0/3556333105 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2cc0adbc0 con 0x7fb2cc1f0ae0 2023-10-19T14:21:01.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.796+0000 7fb2c97fa640 1 -- 172.21.15.57:0/3556333105 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2b4009a80 con 0x7fb2cc1f0ae0 2023-10-19T14:21:01.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.796+0000 7fb2d1ba0640 1 -- 172.21.15.57:0/3556333105 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2cc1407a0 con 0x7fb2cc1f0ae0 2023-10-19T14:21:01.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.797+0000 7fb2d1ba0640 1 -- 172.21.15.57:0/3556333105 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2cc04da70 con 0x7fb2cc1f0ae0 2023-10-19T14:21:01.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.798+0000 7fb2c97fa640 1 -- 172.21.15.57:0/3556333105 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fb2b4009be0 con 0x7fb2cc1f0ae0 2023-10-19T14:21:01.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.798+0000 7fb2c97fa640 1 --2- 172.21.15.57:0/3556333105 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fb2ac06b0b0 0x7fb2ac06d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:01.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.798+0000 7fb2c97fa640 1 -- 172.21.15.57:0/3556333105 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 6194+0+0 (secure 0 0 0) 0x7fb2b4089df0 con 0x7fb2cc1f0ae0 2023-10-19T14:21:01.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.798+0000 7fb2d0b9e640 1 --2- 172.21.15.57:0/3556333105 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fb2ac06b0b0 0x7fb2ac06d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:01.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.799+0000 7fb2d0b9e640 1 --2- 172.21.15.57:0/3556333105 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fb2ac06b0b0 0x7fb2ac06d570 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb2c4004b20 tx=0x7fb2c4001a50 comp rx=0 tx=0).ready entity=mgr.9530 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:01.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:01.802+0000 7fb2c97fa640 1 -- 172.21.15.57:0/3556333105 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb2b4054870 con 0x7fb2cc1f0ae0 2023-10-19T14:21:02.049 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.046+0000 7fb2d1ba0640 1 -- 172.21.15.57:0/3556333105 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs"]} v 0) v1 -- 0x7fb2cc13d050 con 0x7fb2cc1f0ae0 2023-10-19T14:21:02.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.356+0000 7fb2c97fa640 1 -- 172.21.15.57:0/3556333105 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs"]}]=0 v3167) v1 ==== 186+0+0 (secure 0 0 0) 0x7fb2b4017100 con 0x7fb2cc1f0ae0 2023-10-19T14:21:02.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.358+0000 7fb2d1ba0640 1 -- 172.21.15.57:0/3556333105 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fb2ac06b0b0 msgr2=0x7fb2ac06d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:02.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.358+0000 7fb2d1ba0640 1 --2- 172.21.15.57:0/3556333105 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fb2ac06b0b0 0x7fb2ac06d570 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb2c4004b20 tx=0x7fb2c4001a50 comp rx=0 tx=0).stop 2023-10-19T14:21:02.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.358+0000 7fb2d1ba0640 1 -- 172.21.15.57:0/3556333105 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb2cc1f0ae0 msgr2=0x7fb2cc0ab910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:02.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.358+0000 7fb2d1ba0640 1 --2- 172.21.15.57:0/3556333105 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb2cc1f0ae0 0x7fb2cc0ab910 secure :-1 s=READY pgs=2573 cs=0 l=1 rev1=1 crypto rx=0x7fb2b400d660 tx=0x7fb2b400db20 comp rx=0 tx=0).stop 2023-10-19T14:21:02.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.358+0000 7fb2d1ba0640 1 -- 172.21.15.57:0/3556333105 shutdown_connections 2023-10-19T14:21:02.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.358+0000 7fb2d1ba0640 1 --2- 172.21.15.57:0/3556333105 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fb2ac06b0b0 0x7fb2ac06d570 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:02.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.358+0000 7fb2d1ba0640 1 --2- 172.21.15.57:0/3556333105 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb2cc1f0ae0 0x7fb2cc0ab910 unknown :-1 s=CLOSED pgs=2573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:02.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.358+0000 7fb2d1ba0640 1 --2- 172.21.15.57:0/3556333105 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb2cc1413b0 0x7fb2cc1400a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:02.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.358+0000 7fb2d1ba0640 1 --2- 172.21.15.57:0/3556333105 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb2cc0b1b40 0x7fb2cc0af150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:02.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.358+0000 7fb2d1ba0640 1 -- 172.21.15.57:0/3556333105 >> 172.21.15.57:0/3556333105 conn(0x7fb2cc070080 msgr2=0x7fb2cc0b1f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:02.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.358+0000 7fb2d1ba0640 1 -- 172.21.15.57:0/3556333105 shutdown_connections 2023-10-19T14:21:02.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.358+0000 7fb2d1ba0640 1 -- 172.21.15.57:0/3556333105 wait complete. 2023-10-19T14:21:02.373 INFO:tasks.cephfs_test_runner:Test command succes with --fs argument for non-default fs ... ok 2023-10-19T14:21:02.374 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_without_fs_argument (tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument) Test command fails without --fs argument in presence of multiple fs 2023-10-19T14:21:02.374 INFO:tasks.cephfs_test_runner:test_snap_schedule_without_fs_argument (tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument) 2023-10-19T14:21:02.375 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T14:21:02.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.483+0000 7fd19c694640 1 -- 172.21.15.57:0/4135287273 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd180003970 msgr2=0x7fd180003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:02.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.483+0000 7fd19c694640 1 --2- 172.21.15.57:0/4135287273 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd180003970 0x7fd180003d90 secure :-1 s=READY pgs=2575 cs=0 l=1 rev1=1 crypto rx=0x7fd190009b50 tx=0x7fd1800057f0 comp rx=0 tx=0).stop 2023-10-19T14:21:02.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.483+0000 7fd19c694640 1 -- 172.21.15.57:0/4135287273 shutdown_connections 2023-10-19T14:21:02.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.483+0000 7fd19c694640 1 --2- 172.21.15.57:0/4135287273 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd180003970 0x7fd180003d90 unknown :-1 s=CLOSED pgs=2575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:02.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.483+0000 7fd19c694640 1 -- 172.21.15.57:0/4135287273 >> 172.21.15.57:0/4135287273 conn(0x7fd194069930 msgr2=0x7fd194069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:02.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.483+0000 7fd19c694640 1 -- 172.21.15.57:0/4135287273 shutdown_connections 2023-10-19T14:21:02.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.483+0000 7fd19c694640 1 -- 172.21.15.57:0/4135287273 wait complete. 2023-10-19T14:21:02.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.485+0000 7fd19c694640 1 Processor -- start 2023-10-19T14:21:02.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.485+0000 7fd19c694640 1 -- start start 2023-10-19T14:21:02.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.485+0000 7fd19c694640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd180003970 0x7fd194136db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:02.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.485+0000 7fd19c694640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd1941e3e80 0x7fd19406f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:02.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.485+0000 7fd19c694640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd194157ce0 0x7fd194158160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:02.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.485+0000 7fd19c694640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1941586a0 con 0x7fd180003970 2023-10-19T14:21:02.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.485+0000 7fd19c694640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1941397d0 con 0x7fd194157ce0 2023-10-19T14:21:02.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.485+0000 7fd19c694640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd19406f960 con 0x7fd1941e3e80 2023-10-19T14:21:02.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.485+0000 7fd19a430640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd180003970 0x7fd194136db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:02.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.485+0000 7fd19a430640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd180003970 0x7fd194136db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38568/0 (socket says 172.21.15.57:38568) 2023-10-19T14:21:02.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.485+0000 7fd19a430640 1 -- 172.21.15.57:0/1596892731 learned_addr learned my addr 172.21.15.57:0/1596892731 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:02.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.486+0000 7fd19ac31640 1 --2- 172.21.15.57:0/1596892731 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd194157ce0 0x7fd194158160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:02.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.486+0000 7fd199c2f640 1 --2- 172.21.15.57:0/1596892731 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd1941e3e80 0x7fd19406f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:02.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.486+0000 7fd19a430640 1 -- 172.21.15.57:0/1596892731 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd1941e3e80 msgr2=0x7fd19406f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:02.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.486+0000 7fd19a430640 1 --2- 172.21.15.57:0/1596892731 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd1941e3e80 0x7fd19406f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:02.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.486+0000 7fd19a430640 1 -- 172.21.15.57:0/1596892731 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd194157ce0 msgr2=0x7fd194158160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:02.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.486+0000 7fd19a430640 1 --2- 172.21.15.57:0/1596892731 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd194157ce0 0x7fd194158160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:02.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.486+0000 7fd19a430640 1 -- 172.21.15.57:0/1596892731 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd19406fc40 con 0x7fd180003970 2023-10-19T14:21:02.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.486+0000 7fd19ac31640 1 --2- 172.21.15.57:0/1596892731 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd194157ce0 0x7fd194158160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:21:02.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.486+0000 7fd19a430640 1 --2- 172.21.15.57:0/1596892731 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd180003970 0x7fd194136db0 secure :-1 s=READY pgs=2576 cs=0 l=1 rev1=1 crypto rx=0x7fd190009b50 tx=0x7fd190033c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:02.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.486+0000 7fd18affd640 1 -- 172.21.15.57:0/1596892731 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd190004440 con 0x7fd180003970 2023-10-19T14:21:02.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.486+0000 7fd18affd640 1 -- 172.21.15.57:0/1596892731 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fd1900035f0 con 0x7fd180003970 2023-10-19T14:21:02.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.486+0000 7fd18affd640 1 -- 172.21.15.57:0/1596892731 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1900039e0 con 0x7fd180003970 2023-10-19T14:21:02.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.486+0000 7fd19c694640 1 -- 172.21.15.57:0/1596892731 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd19406ff20 con 0x7fd180003970 2023-10-19T14:21:02.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.486+0000 7fd19c694640 1 -- 172.21.15.57:0/1596892731 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1940704e0 con 0x7fd180003970 2023-10-19T14:21:02.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.487+0000 7fd19c694640 1 -- 172.21.15.57:0/1596892731 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd194069e10 con 0x7fd180003970 2023-10-19T14:21:02.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.488+0000 7fd18affd640 1 -- 172.21.15.57:0/1596892731 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fd190003b40 con 0x7fd180003970 2023-10-19T14:21:02.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.488+0000 7fd18affd640 1 --2- 172.21.15.57:0/1596892731 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fd17806b0b0 0x7fd17806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:02.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.488+0000 7fd18affd640 1 -- 172.21.15.57:0/1596892731 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 6194+0+0 (secure 0 0 0) 0x7fd1900ac930 con 0x7fd180003970 2023-10-19T14:21:02.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.488+0000 7fd199c2f640 1 --2- 172.21.15.57:0/1596892731 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fd17806b0b0 0x7fd17806d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:02.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.489+0000 7fd199c2f640 1 --2- 172.21.15.57:0/1596892731 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fd17806b0b0 0x7fd17806d570 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd184001050 tx=0x7fd184001620 comp rx=0 tx=0).ready entity=mgr.9530 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:02.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.492+0000 7fd18affd640 1 -- 172.21.15.57:0/1596892731 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd190077450 con 0x7fd180003970 2023-10-19T14:21:02.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.668+0000 7fd19c694640 1 -- 172.21.15.57:0/1596892731 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd1940690e0 con 0x7fd180003970 2023-10-19T14:21:02.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.669+0000 7fd18affd640 1 -- 172.21.15.57:0/1596892731 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v235) v1 ==== 74+0+14593 (secure 0 0 0) 0x7fd19004b090 con 0x7fd180003970 2023-10-19T14:21:02.672 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:21:02.672 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":235,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T14:21:01.338818+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.57:6835/1529368280":"2023-10-20T14:20:51.762660+0000","172.21.15.179:0/1513076424":"2023-10-20T14:20:48.148610+0000","172.21.15.179:0/1464950687":"2023-10-20T14:20:48.148610+0000","172.21.15.179:0/2900872065":"2023-10-20T14:20:48.148610+0000","172.21.15.57:6832/2467807803":"2023-10-20T14:20:58.057530+0000","172.21.15.57:6833/1529368280":"2023-10-20T14:20:51.762660+0000","172.21.15.179:0/2140451434":"2023-10-20T14:20:48.148610+0000","172.21.15.179:0/2605624080":"2023-10-20T14:20:48.148610+0000","172.21.15.179:0/1194984394":"2023-10-20T14:20:39.702634+0000","172.21.15.179:6833/3232482678":"2023-10-20T14:20:39.702634+0000","172.21.15.179:6832/4242302683":"2023-10-20T14:20:48.148610+0000","172.21.15.179:6832/3232482678":"2023-10-20T14:20:39.702634+0000","172.21.15.179:0/2872579179":"2023-10-20T14:20:39.702634+0000","172.21.15.179:0/373323540":"2023-10-20T14:20:48.148610+0000","172.21.15.179:0/3910604357":"2023-10-20T14:20:39.702634+0000","172.21.15.179:0/4185855767":"2023-10-20T14:20:48.148610+0000","172.21.15.179:0/536160557":"2023-10-20T14:20:39.702634+0000","172.21.15.57:6834/2467807803":"2023-10-20T14:20:58.057530+0000","172.21.15.179:0/3239062050":"2023-10-20T14:20:39.702634+0000","172.21.15.179:6833/4242302683":"2023-10-20T14:20:48.148610+0000","172.21.15.179:0/3672528294":"2023-10-20T14:20:48.148610+0000","172.21.15.179:0/2003826253":"2023-10-20T14:20:48.148610+0000","172.21.15.179:0/1948632334":"2023-10-20T14:20:39.702634+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T14:21:02.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.671+0000 7fd19c694640 1 -- 172.21.15.57:0/1596892731 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fd17806b0b0 msgr2=0x7fd17806d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:02.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.671+0000 7fd19c694640 1 --2- 172.21.15.57:0/1596892731 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fd17806b0b0 0x7fd17806d570 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd184001050 tx=0x7fd184001620 comp rx=0 tx=0).stop 2023-10-19T14:21:02.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.671+0000 7fd19c694640 1 -- 172.21.15.57:0/1596892731 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd180003970 msgr2=0x7fd194136db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:02.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.671+0000 7fd19c694640 1 --2- 172.21.15.57:0/1596892731 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd180003970 0x7fd194136db0 secure :-1 s=READY pgs=2576 cs=0 l=1 rev1=1 crypto rx=0x7fd190009b50 tx=0x7fd190033c10 comp rx=0 tx=0).stop 2023-10-19T14:21:02.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.671+0000 7fd19c694640 1 -- 172.21.15.57:0/1596892731 shutdown_connections 2023-10-19T14:21:02.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.671+0000 7fd19c694640 1 --2- 172.21.15.57:0/1596892731 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fd17806b0b0 0x7fd17806d570 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:02.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.671+0000 7fd19c694640 1 --2- 172.21.15.57:0/1596892731 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd194157ce0 0x7fd194158160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:02.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.671+0000 7fd19c694640 1 --2- 172.21.15.57:0/1596892731 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd1941e3e80 0x7fd19406f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:02.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.671+0000 7fd19c694640 1 --2- 172.21.15.57:0/1596892731 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd180003970 0x7fd194136db0 unknown :-1 s=CLOSED pgs=2576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:02.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.671+0000 7fd19c694640 1 -- 172.21.15.57:0/1596892731 >> 172.21.15.57:0/1596892731 conn(0x7fd194069930 msgr2=0x7fd1940ca870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:02.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.671+0000 7fd19c694640 1 -- 172.21.15.57:0/1596892731 shutdown_connections 2023-10-19T14:21:02.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.671+0000 7fd19c694640 1 -- 172.21.15.57:0/1596892731 wait complete. 2023-10-19T14:21:02.685 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-19T12:44:07.371186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-19T14:21:02.686 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-19T14:21:02.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.796+0000 7f8022056640 1 -- 172.21.15.57:0/4290646053 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8004004d00 msgr2=0x7f8004005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:02.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.796+0000 7f8022056640 1 --2- 172.21.15.57:0/4290646053 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8004004d00 0x7f8004005180 secure :-1 s=READY pgs=2578 cs=0 l=1 rev1=1 crypto rx=0x7f801400c560 tx=0x7f80040057f0 comp rx=0 tx=0).stop 2023-10-19T14:21:02.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.796+0000 7f801bfff640 1 -- 172.21.15.57:0/4290646053 reap_dead start 2023-10-19T14:21:02.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.796+0000 7f8022056640 1 -- 172.21.15.57:0/4290646053 shutdown_connections 2023-10-19T14:21:02.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.796+0000 7f8022056640 1 -- 172.21.15.57:0/4290646053 >> 172.21.15.57:0/4290646053 conn(0x7f801c0d0d60 msgr2=0x7f801c0d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:02.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.796+0000 7f8022056640 1 -- 172.21.15.57:0/4290646053 shutdown_connections 2023-10-19T14:21:02.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.796+0000 7f8022056640 1 -- 172.21.15.57:0/4290646053 wait complete. 2023-10-19T14:21:02.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.797+0000 7f8022056640 1 Processor -- start 2023-10-19T14:21:02.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.797+0000 7f8022056640 1 -- start start 2023-10-19T14:21:02.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.798+0000 7f8022056640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8004004d00 0x7f801c1598d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:02.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.798+0000 7f8022056640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f801c0c8f30 0x7f801c0c9390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:02.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.798+0000 7f8022056640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f801c0b5a20 0x7f801c0b5ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:02.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.798+0000 7f8022056640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f801c0b63e0 con 0x7f801c0c8f30 2023-10-19T14:21:02.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.798+0000 7f8022056640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f801c0c8aa0 con 0x7f801c0b5a20 2023-10-19T14:21:02.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.798+0000 7f8022056640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f801c159e10 con 0x7f8004004d00 2023-10-19T14:21:02.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.798+0000 7f801affd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f801c0c8f30 0x7f801c0c9390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:02.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.798+0000 7f801affd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f801c0c8f30 0x7f801c0c9390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38586/0 (socket says 172.21.15.57:38586) 2023-10-19T14:21:02.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.798+0000 7f801affd640 1 -- 172.21.15.57:0/1173111026 learned_addr learned my addr 172.21.15.57:0/1173111026 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:02.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.798+0000 7f801b7fe640 1 --2- 172.21.15.57:0/1173111026 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8004004d00 0x7f801c1598d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:02.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.798+0000 7f801bfff640 1 --2- 172.21.15.57:0/1173111026 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f801c0b5a20 0x7f801c0b5ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:02.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.798+0000 7f801affd640 1 -- 172.21.15.57:0/1173111026 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8004004d00 msgr2=0x7f801c1598d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:02.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.798+0000 7f801affd640 1 --2- 172.21.15.57:0/1173111026 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8004004d00 0x7f801c1598d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:02.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.798+0000 7f801affd640 1 -- 172.21.15.57:0/1173111026 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f801c0b5a20 msgr2=0x7f801c0b5ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:02.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.798+0000 7f801affd640 1 --2- 172.21.15.57:0/1173111026 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f801c0b5a20 0x7f801c0b5ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:02.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.798+0000 7f801affd640 1 -- 172.21.15.57:0/1173111026 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f801c0c7ed0 con 0x7f801c0c8f30 2023-10-19T14:21:02.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.799+0000 7f801affd640 1 --2- 172.21.15.57:0/1173111026 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f801c0c8f30 0x7f801c0c9390 secure :-1 s=READY pgs=2579 cs=0 l=1 rev1=1 crypto rx=0x7f8010003e40 tx=0x7f8010001770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:02.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.799+0000 7f8003fff640 1 -- 172.21.15.57:0/1173111026 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8010011840 con 0x7f801c0c8f30 2023-10-19T14:21:02.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.799+0000 7f8003fff640 1 -- 172.21.15.57:0/1173111026 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f80100119a0 con 0x7f801c0c8f30 2023-10-19T14:21:02.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.799+0000 7f8003fff640 1 -- 172.21.15.57:0/1173111026 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8010011db0 con 0x7f801c0c8f30 2023-10-19T14:21:02.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.799+0000 7f8022056640 1 -- 172.21.15.57:0/1173111026 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f801c0c8170 con 0x7f801c0c8f30 2023-10-19T14:21:02.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.799+0000 7f8022056640 1 -- 172.21.15.57:0/1173111026 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f801c0c8730 con 0x7f801c0c8f30 2023-10-19T14:21:02.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.800+0000 7f8022056640 1 -- 172.21.15.57:0/1173111026 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f801c0c8c30 con 0x7f801c0c8f30 2023-10-19T14:21:02.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.800+0000 7f8003fff640 1 -- 172.21.15.57:0/1173111026 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f80100103c0 con 0x7f801c0c8f30 2023-10-19T14:21:02.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.801+0000 7f8003fff640 1 --2- 172.21.15.57:0/1173111026 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f800806b080 0x7f800806d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:02.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.801+0000 7f8003fff640 1 -- 172.21.15.57:0/1173111026 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 6194+0+0 (secure 0 0 0) 0x7f801008ad20 con 0x7f801c0c8f30 2023-10-19T14:21:02.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.801+0000 7f801b7fe640 1 --2- 172.21.15.57:0/1173111026 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f800806b080 0x7f800806d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:02.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.802+0000 7f801b7fe640 1 --2- 172.21.15.57:0/1173111026 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f800806b080 0x7f800806d540 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f800c009b90 tx=0x7f800c005f90 comp rx=0 tx=0).ready entity=mgr.9530 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:02.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.805+0000 7f8003fff640 1 -- 172.21.15.57:0/1173111026 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f80100558d0 con 0x7f801c0c8f30 2023-10-19T14:21:02.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.979+0000 7f8022056640 1 -- 172.21.15.57:0/1173111026 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f801c0690e0 con 0x7f801c0c8f30 2023-10-19T14:21:02.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.980+0000 7f8003fff640 1 -- 172.21.15.57:0/1173111026 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v235) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8010015070 con 0x7f801c0c8f30 2023-10-19T14:21:02.983 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-19T14:21:02.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.981+0000 7f8022056640 1 -- 172.21.15.57:0/1173111026 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f800806b080 msgr2=0x7f800806d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:02.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.981+0000 7f8022056640 1 --2- 172.21.15.57:0/1173111026 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f800806b080 0x7f800806d540 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f800c009b90 tx=0x7f800c005f90 comp rx=0 tx=0).stop 2023-10-19T14:21:02.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.982+0000 7f8022056640 1 -- 172.21.15.57:0/1173111026 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f801c0c8f30 msgr2=0x7f801c0c9390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:02.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.982+0000 7f8022056640 1 --2- 172.21.15.57:0/1173111026 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f801c0c8f30 0x7f801c0c9390 secure :-1 s=READY pgs=2579 cs=0 l=1 rev1=1 crypto rx=0x7f8010003e40 tx=0x7f8010001770 comp rx=0 tx=0).stop 2023-10-19T14:21:02.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.982+0000 7f8022056640 1 -- 172.21.15.57:0/1173111026 shutdown_connections 2023-10-19T14:21:02.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.982+0000 7f8022056640 1 --2- 172.21.15.57:0/1173111026 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f800806b080 0x7f800806d540 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:02.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.982+0000 7f8022056640 1 --2- 172.21.15.57:0/1173111026 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f801c0b5a20 0x7f801c0b5ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:02.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.982+0000 7f8022056640 1 --2- 172.21.15.57:0/1173111026 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f801c0c8f30 0x7f801c0c9390 unknown :-1 s=CLOSED pgs=2579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:02.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.982+0000 7f8022056640 1 --2- 172.21.15.57:0/1173111026 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8004004d00 0x7f801c1598d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:02.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.982+0000 7f8022056640 1 -- 172.21.15.57:0/1173111026 >> 172.21.15.57:0/1173111026 conn(0x7f801c0d0d60 msgr2=0x7f801c15a0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:02.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.982+0000 7f8022056640 1 -- 172.21.15.57:0/1173111026 shutdown_connections 2023-10-19T14:21:02.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:02.982+0000 7f8022056640 1 -- 172.21.15.57:0/1173111026 wait complete. 2023-10-19T14:21:02.995 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument' 2023-10-19T14:21:03.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:03.101+0000 7f78b08b4640 1 -- 172.21.15.57:0/3998545033 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f78ac13d350 msgr2=0x7f78ac0cbd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:03.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:03.101+0000 7f78b08b4640 1 --2- 172.21.15.57:0/3998545033 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f78ac13d350 0x7f78ac0cbd60 secure :-1 s=READY pgs=2580 cs=0 l=1 rev1=1 crypto rx=0x7f78ac06d4f0 tx=0x7f78a002be40 comp rx=0 tx=0).stop 2023-10-19T14:21:03.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:03.101+0000 7f78b08b4640 1 -- 172.21.15.57:0/3998545033 shutdown_connections 2023-10-19T14:21:03.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:03.101+0000 7f78b08b4640 1 --2- 172.21.15.57:0/3998545033 >> v2:172.21.15.179:3300/0 conn(0x7f78ac070350 0x7f78ac070780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:03.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:03.101+0000 7f78b08b4640 1 --2- 172.21.15.57:0/3998545033 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f78ac13d350 0x7f78ac0cbd60 unknown :-1 s=CLOSED pgs=2580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:03.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:03.101+0000 7f78b08b4640 1 -- 172.21.15.57:0/3998545033 >> 172.21.15.57:0/3998545033 conn(0x7f78ac069930 msgr2=0x7f78ac069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:03.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:03.101+0000 7f78b08b4640 1 -- 172.21.15.57:0/3998545033 shutdown_connections 2023-10-19T14:21:03.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:03.101+0000 7f78b08b4640 1 -- 172.21.15.57:0/3998545033 wait complete. 2023-10-19T14:21:03.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:03.103+0000 7f78b08b4640 1 Processor -- start 2023-10-19T14:21:03.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:03.103+0000 7f78b08b4640 1 -- start start 2023-10-19T14:21:03.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:03.103+0000 7f78b08b4640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f78ac070350 0x7f78ac0adc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:03.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:03.103+0000 7f78b08b4640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f78ac0cb980 0x7f78ac15a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:03.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:03.103+0000 7f78b08b4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f78ac138330 0x7f78ac1387c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:03.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:03.103+0000 7f78b08b4640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78ac159fe0 con 0x7f78ac138330 2023-10-19T14:21:03.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:03.103+0000 7f78b08b4640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78ac138d00 con 0x7f78ac070350 2023-10-19T14:21:03.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:03.103+0000 7f78b08b4640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78ac1f4910 con 0x7f78ac0cb980 2023-10-19T14:21:03.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:03.104+0000 7f78aad9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f78ac138330 0x7f78ac1387c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:03.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:03.104+0000 7f78aad9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f78ac138330 0x7f78ac1387c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38606/0 (socket says 172.21.15.57:38606) 2023-10-19T14:21:03.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:03.104+0000 7f78aad9d640 1 -- 172.21.15.57:0/1791058353 learned_addr learned my addr 172.21.15.57:0/1791058353 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:03.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:03.104+0000 7f78a9d9b640 1 --2- 172.21.15.57:0/1791058353 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f78ac0cb980 0x7f78ac15a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:03.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:03.104+0000 7f78aa59c640 1 --2- 172.21.15.57:0/1791058353 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f78ac070350 0x7f78ac0adc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:03.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:03.104+0000 7f78aad9d640 1 -- 172.21.15.57:0/1791058353 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f78ac0cb980 msgr2=0x7f78ac15a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:03.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:03.104+0000 7f78aad9d640 1 --2- 172.21.15.57:0/1791058353 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f78ac0cb980 0x7f78ac15a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:03.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:03.104+0000 7f78aad9d640 1 -- 172.21.15.57:0/1791058353 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f78ac070350 msgr2=0x7f78ac0adc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:03.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:03.104+0000 7f78aad9d640 1 --2- 172.21.15.57:0/1791058353 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f78ac070350 0x7f78ac0adc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:03.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:03.104+0000 7f78aad9d640 1 -- 172.21.15.57:0/1791058353 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78a0009560 con 0x7f78ac138330 2023-10-19T14:21:03.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:03.104+0000 7f78aa59c640 1 --2- 172.21.15.57:0/1791058353 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f78ac070350 0x7f78ac0adc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:21:03.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:03.104+0000 7f78a9d9b640 1 --2- 172.21.15.57:0/1791058353 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f78ac0cb980 0x7f78ac15a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:21:03.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:03.104+0000 7f78aad9d640 1 --2- 172.21.15.57:0/1791058353 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f78ac138330 0x7f78ac1387c0 secure :-1 s=READY pgs=2581 cs=0 l=1 rev1=1 crypto rx=0x7f78a400eaa0 tx=0x7f78a400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:03.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:03.104+0000 7f789affd640 1 -- 172.21.15.57:0/1791058353 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78a400cdb0 con 0x7f78ac138330 2023-10-19T14:21:03.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:03.104+0000 7f789affd640 1 -- 172.21.15.57:0/1791058353 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f78a4004b20 con 0x7f78ac138330 2023-10-19T14:21:03.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:03.104+0000 7f78b08b4640 1 -- 172.21.15.57:0/1791058353 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78ac1593d0 con 0x7f78ac138330 2023-10-19T14:21:03.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:03.104+0000 7f789affd640 1 -- 172.21.15.57:0/1791058353 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78a4005020 con 0x7f78ac138330 2023-10-19T14:21:03.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:03.104+0000 7f78b08b4640 1 -- 172.21.15.57:0/1791058353 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78ac159990 con 0x7f78ac138330 2023-10-19T14:21:03.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:03.105+0000 7f78b08b4640 1 -- 172.21.15.57:0/1791058353 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78ac15a150 con 0x7f78ac138330 2023-10-19T14:21:03.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:03.106+0000 7f789affd640 1 -- 172.21.15.57:0/1791058353 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f78a4005180 con 0x7f78ac138330 2023-10-19T14:21:03.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:03.106+0000 7f789affd640 1 --2- 172.21.15.57:0/1791058353 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f788c06b080 0x7f788c06d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:03.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:03.106+0000 7f789affd640 1 -- 172.21.15.57:0/1791058353 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 6194+0+0 (secure 0 0 0) 0x7f78a408a670 con 0x7f78ac138330 2023-10-19T14:21:03.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:03.106+0000 7f78aa59c640 1 --2- 172.21.15.57:0/1791058353 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f788c06b080 0x7f788c06d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:03.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:03.107+0000 7f78aa59c640 1 --2- 172.21.15.57:0/1791058353 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f788c06b080 0x7f788c06d540 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f789c000f60 tx=0x7f789c009040 comp rx=0 tx=0).ready entity=mgr.9530 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:03.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:03.110+0000 7f789affd640 1 -- 172.21.15.57:0/1791058353 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f78a4055150 con 0x7f78ac138330 2023-10-19T14:21:03.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:03.356+0000 7f78b08b4640 1 -- 172.21.15.57:0/1791058353 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument"]} v 0) v1 -- 0x7f78ac0690e0 con 0x7f78ac138330 2023-10-19T14:21:04.364 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.361+0000 7f789affd640 1 -- 172.21.15.57:0/1791058353 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument"]}]=0 v3169) v1 ==== 190+0+0 (secure 0 0 0) 0x7f78a4058770 con 0x7f78ac138330 2023-10-19T14:21:04.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.364+0000 7f78b08b4640 1 -- 172.21.15.57:0/1791058353 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f788c06b080 msgr2=0x7f788c06d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:04.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.364+0000 7f78b08b4640 1 --2- 172.21.15.57:0/1791058353 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f788c06b080 0x7f788c06d540 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f789c000f60 tx=0x7f789c009040 comp rx=0 tx=0).stop 2023-10-19T14:21:04.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.364+0000 7f78b08b4640 1 -- 172.21.15.57:0/1791058353 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f78ac138330 msgr2=0x7f78ac1387c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:04.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.364+0000 7f78b08b4640 1 --2- 172.21.15.57:0/1791058353 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f78ac138330 0x7f78ac1387c0 secure :-1 s=READY pgs=2581 cs=0 l=1 rev1=1 crypto rx=0x7f78a400eaa0 tx=0x7f78a400ef60 comp rx=0 tx=0).stop 2023-10-19T14:21:04.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.364+0000 7f78b08b4640 1 -- 172.21.15.57:0/1791058353 shutdown_connections 2023-10-19T14:21:04.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.364+0000 7f78b08b4640 1 --2- 172.21.15.57:0/1791058353 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f788c06b080 0x7f788c06d540 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:04.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.364+0000 7f78b08b4640 1 --2- 172.21.15.57:0/1791058353 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f78ac138330 0x7f78ac1387c0 unknown :-1 s=CLOSED pgs=2581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:04.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.364+0000 7f78b08b4640 1 --2- 172.21.15.57:0/1791058353 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f78ac0cb980 0x7f78ac15a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:04.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.364+0000 7f78b08b4640 1 --2- 172.21.15.57:0/1791058353 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f78ac070350 0x7f78ac0adc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:04.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.364+0000 7f78b08b4640 1 -- 172.21.15.57:0/1791058353 >> 172.21.15.57:0/1791058353 conn(0x7f78ac069930 msgr2=0x7f78ac0ab090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:04.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.364+0000 7f78b08b4640 1 -- 172.21.15.57:0/1791058353 shutdown_connections 2023-10-19T14:21:04.370 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.364+0000 7f78b08b4640 1 -- 172.21.15.57:0/1791058353 wait complete. 2023-10-19T14:21:04.380 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2023-10-19T14:21:04.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.493+0000 7f95579f5640 1 -- 172.21.15.57:0/2210804951 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9538004d00 msgr2=0x7f9538005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:04.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.493+0000 7f95579f5640 1 --2- 172.21.15.57:0/2210804951 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9538004d00 0x7f9538005180 secure :-1 s=READY pgs=2583 cs=0 l=1 rev1=1 crypto rx=0x7f9548009b50 tx=0x7f95380057f0 comp rx=0 tx=0).stop 2023-10-19T14:21:04.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.493+0000 7f9555f92640 1 -- 172.21.15.57:0/2210804951 reap_dead start 2023-10-19T14:21:04.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.493+0000 7f95579f5640 1 -- 172.21.15.57:0/2210804951 shutdown_connections 2023-10-19T14:21:04.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.494+0000 7f95579f5640 1 -- 172.21.15.57:0/2210804951 >> 172.21.15.57:0/2210804951 conn(0x7f9550069930 msgr2=0x7f9550069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:04.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.494+0000 7f95579f5640 1 -- 172.21.15.57:0/2210804951 shutdown_connections 2023-10-19T14:21:04.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.494+0000 7f95579f5640 1 -- 172.21.15.57:0/2210804951 wait complete. 2023-10-19T14:21:04.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.495+0000 7f95579f5640 1 Processor -- start 2023-10-19T14:21:04.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.495+0000 7f95579f5640 1 -- start start 2023-10-19T14:21:04.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.496+0000 7f95579f5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9538004d00 0x7f955014e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:04.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.496+0000 7f95579f5640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9550151eb0 0x7f9550141170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:04.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.496+0000 7f95579f5640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9550150590 0x7f9550150a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:04.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.496+0000 7f95579f5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9550150f80 con 0x7f9538004d00 2023-10-19T14:21:04.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.496+0000 7f95579f5640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f955014eba0 con 0x7f9550150590 2023-10-19T14:21:04.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.496+0000 7f95579f5640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f955014f030 con 0x7f9550151eb0 2023-10-19T14:21:04.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.496+0000 7f9555791640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9538004d00 0x7f955014e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:04.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.496+0000 7f9555791640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9538004d00 0x7f955014e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38620/0 (socket says 172.21.15.57:38620) 2023-10-19T14:21:04.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.496+0000 7f9555791640 1 -- 172.21.15.57:0/4292972513 learned_addr learned my addr 172.21.15.57:0/4292972513 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:04.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.496+0000 7f9554f90640 1 --2- 172.21.15.57:0/4292972513 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9550151eb0 0x7f9550141170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:04.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.496+0000 7f9555f92640 1 --2- 172.21.15.57:0/4292972513 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9550150590 0x7f9550150a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:04.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.496+0000 7f9555791640 1 -- 172.21.15.57:0/4292972513 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9550151eb0 msgr2=0x7f9550141170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:04.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.496+0000 7f9555791640 1 --2- 172.21.15.57:0/4292972513 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9550151eb0 0x7f9550141170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:04.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.496+0000 7f9555791640 1 -- 172.21.15.57:0/4292972513 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9550150590 msgr2=0x7f9550150a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:04.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.496+0000 7f9555791640 1 --2- 172.21.15.57:0/4292972513 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9550150590 0x7f9550150a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:04.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.496+0000 7f9555791640 1 -- 172.21.15.57:0/4292972513 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95501523b0 con 0x7f9538004d00 2023-10-19T14:21:04.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.496+0000 7f9555791640 1 --2- 172.21.15.57:0/4292972513 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9538004d00 0x7f955014e660 secure :-1 s=READY pgs=2584 cs=0 l=1 rev1=1 crypto rx=0x7f954000ed60 tx=0x7f954000d7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:04.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.496+0000 7f954dffb640 1 -- 172.21.15.57:0/4292972513 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9540014070 con 0x7f9538004d00 2023-10-19T14:21:04.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.496+0000 7f954dffb640 1 -- 172.21.15.57:0/4292972513 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f9540003b80 con 0x7f9538004d00 2023-10-19T14:21:04.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.496+0000 7f954dffb640 1 -- 172.21.15.57:0/4292972513 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9540004050 con 0x7f9538004d00 2023-10-19T14:21:04.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.496+0000 7f95579f5640 1 -- 172.21.15.57:0/4292972513 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9550152690 con 0x7f9538004d00 2023-10-19T14:21:04.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.497+0000 7f95579f5640 1 -- 172.21.15.57:0/4292972513 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9550152c50 con 0x7f9538004d00 2023-10-19T14:21:04.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.498+0000 7f95579f5640 1 -- 172.21.15.57:0/4292972513 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95500b09c0 con 0x7f9538004d00 2023-10-19T14:21:04.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.498+0000 7f954dffb640 1 -- 172.21.15.57:0/4292972513 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f95400041b0 con 0x7f9538004d00 2023-10-19T14:21:04.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.498+0000 7f954dffb640 1 --2- 172.21.15.57:0/4292972513 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f953c06b0b0 0x7f953c06d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:04.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.498+0000 7f954dffb640 1 -- 172.21.15.57:0/4292972513 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 6194+0+0 (secure 0 0 0) 0x7f954008abf0 con 0x7f9538004d00 2023-10-19T14:21:04.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.499+0000 7f9554f90640 1 --2- 172.21.15.57:0/4292972513 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f953c06b0b0 0x7f953c06d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:04.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.499+0000 7f9554f90640 1 --2- 172.21.15.57:0/4292972513 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f953c06b0b0 0x7f953c06d570 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f95500ae0f0 tx=0x7f9544006040 comp rx=0 tx=0).ready entity=mgr.9530 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:04.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.503+0000 7f954dffb640 1 -- 172.21.15.57:0/4292972513 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f95400555a0 con 0x7f9538004d00 2023-10-19T14:21:04.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.679+0000 7f95579f5640 1 -- 172.21.15.57:0/4292972513 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "config dump"} v 0) v1 -- 0x7f95500690e0 con 0x7f9538004d00 2023-10-19T14:21:04.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.680+0000 7f954dffb640 1 -- 172.21.15.57:0/4292972513 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v78) v1 ==== 59+0+2220 (secure 0 0 0) 0x7f9540058bc0 con 0x7f9538004d00 2023-10-19T14:21:04.683 INFO:teuthology.orchestra.run.smithi057.stdout:WHO MASK LEVEL OPTION VALUE RO 2023-10-19T14:21:04.683 INFO:teuthology.orchestra.run.smithi057.stdout:mon advanced mon_op_complaint_time 120 2023-10-19T14:21:04.684 INFO:teuthology.orchestra.run.smithi057.stdout:osd advanced osd_op_complaint_time 180.000000 2023-10-19T14:21:04.684 INFO:teuthology.orchestra.run.smithi057.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 4775.995373 2023-10-19T14:21:04.684 INFO:teuthology.orchestra.run.smithi057.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 4208.118410 2023-10-19T14:21:04.684 INFO:teuthology.orchestra.run.smithi057.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 4541.123030 2023-10-19T14:21:04.685 INFO:teuthology.orchestra.run.smithi057.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 4877.592206 2023-10-19T14:21:04.685 INFO:teuthology.orchestra.run.smithi057.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 5019.748032 2023-10-19T14:21:04.685 INFO:teuthology.orchestra.run.smithi057.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 5000.074376 2023-10-19T14:21:04.685 INFO:teuthology.orchestra.run.smithi057.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 5153.479439 2023-10-19T14:21:04.685 INFO:teuthology.orchestra.run.smithi057.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 4977.363845 2023-10-19T14:21:04.686 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced debug_mds 20/20 2023-10-19T14:21:04.686 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced debug_mds_balancer 20/20 2023-10-19T14:21:04.686 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced debug_ms 1/1 2023-10-19T14:21:04.687 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced mds_bal_fragment_size_max 10000 2023-10-19T14:21:04.687 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced mds_bal_merge_size 5 2023-10-19T14:21:04.687 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced mds_bal_split_bits 3 2023-10-19T14:21:04.687 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced mds_bal_split_size 100 2023-10-19T14:21:04.687 INFO:teuthology.orchestra.run.smithi057.stdout:mds dev mds_debug_frag true 2023-10-19T14:21:04.688 INFO:teuthology.orchestra.run.smithi057.stdout:mds dev mds_debug_scatterstat true 2023-10-19T14:21:04.688 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced mds_op_complaint_time 180.000000 2023-10-19T14:21:04.688 INFO:teuthology.orchestra.run.smithi057.stdout:mds dev mds_verify_scatter true 2023-10-19T14:21:04.688 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced osd_op_complaint_time 180.000000 2023-10-19T14:21:04.689 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced rados_mon_op_timeout 900 2023-10-19T14:21:04.689 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced rados_osd_op_timeout 900 2023-10-19T14:21:04.689 INFO:teuthology.orchestra.run.smithi057.stdout:client advanced client_mount_timeout 600 2023-10-19T14:21:04.689 INFO:teuthology.orchestra.run.smithi057.stdout:client advanced debug_client 20/20 2023-10-19T14:21:04.690 INFO:teuthology.orchestra.run.smithi057.stdout:client advanced debug_ms 1/1 2023-10-19T14:21:04.690 INFO:teuthology.orchestra.run.smithi057.stdout:client advanced rados_mon_op_timeout 900 2023-10-19T14:21:04.690 INFO:teuthology.orchestra.run.smithi057.stdout:client advanced rados_osd_op_timeout 900 2023-10-19T14:21:04.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.682+0000 7f95579f5640 1 -- 172.21.15.57:0/4292972513 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f953c06b0b0 msgr2=0x7f953c06d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:04.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.682+0000 7f95579f5640 1 --2- 172.21.15.57:0/4292972513 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f953c06b0b0 0x7f953c06d570 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f95500ae0f0 tx=0x7f9544006040 comp rx=0 tx=0).stop 2023-10-19T14:21:04.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.682+0000 7f95579f5640 1 -- 172.21.15.57:0/4292972513 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9538004d00 msgr2=0x7f955014e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:04.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.682+0000 7f95579f5640 1 --2- 172.21.15.57:0/4292972513 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9538004d00 0x7f955014e660 secure :-1 s=READY pgs=2584 cs=0 l=1 rev1=1 crypto rx=0x7f954000ed60 tx=0x7f954000d7b0 comp rx=0 tx=0).stop 2023-10-19T14:21:04.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.682+0000 7f95579f5640 1 -- 172.21.15.57:0/4292972513 shutdown_connections 2023-10-19T14:21:04.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.682+0000 7f95579f5640 1 --2- 172.21.15.57:0/4292972513 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f953c06b0b0 0x7f953c06d570 secure :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f95500ae0f0 tx=0x7f9544006040 comp rx=0 tx=0).stop 2023-10-19T14:21:04.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.682+0000 7f95579f5640 1 --2- 172.21.15.57:0/4292972513 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9550150590 0x7f9550150a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:04.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.682+0000 7f95579f5640 1 --2- 172.21.15.57:0/4292972513 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9550151eb0 0x7f9550141170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:04.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.682+0000 7f95579f5640 1 --2- 172.21.15.57:0/4292972513 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9538004d00 0x7f955014e660 unknown :-1 s=CLOSED pgs=2584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:04.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.682+0000 7f95579f5640 1 -- 172.21.15.57:0/4292972513 >> 172.21.15.57:0/4292972513 conn(0x7f9550069930 msgr2=0x7f955006baa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:04.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.682+0000 7f95579f5640 1 -- 172.21.15.57:0/4292972513 shutdown_connections 2023-10-19T14:21:04.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.682+0000 7f95579f5640 1 -- 172.21.15.57:0/4292972513 wait complete. 2023-10-19T14:21:04.695 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-19T14:21:04.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.805+0000 7f319d539640 1 -- 172.21.15.57:0/3858468069 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f31980cf340 msgr2=0x7f31980d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:04.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.805+0000 7f319d539640 1 --2- 172.21.15.57:0/3858468069 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f31980cf340 0x7f31980d1910 secure :-1 s=READY pgs=2586 cs=0 l=1 rev1=1 crypto rx=0x7f319815b680 tx=0x7f319002bcf0 comp rx=0 tx=0).stop 2023-10-19T14:21:04.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.805+0000 7f319d539640 1 -- 172.21.15.57:0/3858468069 shutdown_connections 2023-10-19T14:21:04.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.805+0000 7f319d539640 1 --2- 172.21.15.57:0/3858468069 >> v2:172.21.15.179:3300/0 conn(0x7f31980ceb40 0x7f31980b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:04.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.805+0000 7f319d539640 1 --2- 172.21.15.57:0/3858468069 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f31980cf340 0x7f31980d1910 unknown :-1 s=CLOSED pgs=2586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:04.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.805+0000 7f319d539640 1 -- 172.21.15.57:0/3858468069 >> 172.21.15.57:0/3858468069 conn(0x7f31980d0d60 msgr2=0x7f31980d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:04.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.805+0000 7f319d539640 1 -- 172.21.15.57:0/3858468069 shutdown_connections 2023-10-19T14:21:04.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.805+0000 7f319d539640 1 -- 172.21.15.57:0/3858468069 wait complete. 2023-10-19T14:21:04.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.806+0000 7f319d539640 1 Processor -- start 2023-10-19T14:21:04.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.806+0000 7f319d539640 1 -- start start 2023-10-19T14:21:04.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.807+0000 7f319d539640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f31980ceb40 0x7f3198156f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:04.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.807+0000 7f319d539640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f31980cf340 0x7f31980ca6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:04.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.807+0000 7f319d539640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f319815b2a0 0x7f3198158fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:04.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.807+0000 7f319d539640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f319815d1d0 con 0x7f31980ceb40 2023-10-19T14:21:04.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.807+0000 7f319d539640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31980c96d0 con 0x7f319815b2a0 2023-10-19T14:21:04.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.807+0000 7f319d539640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31980c9920 con 0x7f31980cf340 2023-10-19T14:21:04.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.807+0000 7f3196ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f31980ceb40 0x7f3198156f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:04.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.807+0000 7f3196ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f31980ceb40 0x7f3198156f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38642/0 (socket says 172.21.15.57:38642) 2023-10-19T14:21:04.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.807+0000 7f3196ffd640 1 -- 172.21.15.57:0/882938169 learned_addr learned my addr 172.21.15.57:0/882938169 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:04.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.807+0000 7f31977fe640 1 --2- 172.21.15.57:0/882938169 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f319815b2a0 0x7f3198158fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:04.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.807+0000 7f3196ffd640 1 -- 172.21.15.57:0/882938169 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f31980cf340 msgr2=0x7f31980ca6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:04.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.807+0000 7f31967fc640 1 --2- 172.21.15.57:0/882938169 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f31980cf340 0x7f31980ca6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:04.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.807+0000 7f3196ffd640 1 --2- 172.21.15.57:0/882938169 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f31980cf340 0x7f31980ca6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:04.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.807+0000 7f3196ffd640 1 -- 172.21.15.57:0/882938169 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f319815b2a0 msgr2=0x7f3198158fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:04.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.807+0000 7f3196ffd640 1 --2- 172.21.15.57:0/882938169 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f319815b2a0 0x7f3198158fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:04.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.807+0000 7f3196ffd640 1 -- 172.21.15.57:0/882938169 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3190009490 con 0x7f31980ceb40 2023-10-19T14:21:04.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.807+0000 7f31977fe640 1 --2- 172.21.15.57:0/882938169 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f319815b2a0 0x7f3198158fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:21:04.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.807+0000 7f31967fc640 1 --2- 172.21.15.57:0/882938169 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f31980cf340 0x7f31980ca6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:21:04.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.807+0000 7f3196ffd640 1 --2- 172.21.15.57:0/882938169 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f31980ceb40 0x7f3198156f10 secure :-1 s=READY pgs=2587 cs=0 l=1 rev1=1 crypto rx=0x7f3190000c00 tx=0x7f3190004410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:04.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.808+0000 7f318f7fe640 1 -- 172.21.15.57:0/882938169 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3190003a00 con 0x7f31980ceb40 2023-10-19T14:21:04.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.808+0000 7f319d539640 1 -- 172.21.15.57:0/882938169 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31980c9c00 con 0x7f31980ceb40 2023-10-19T14:21:04.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.808+0000 7f318f7fe640 1 -- 172.21.15.57:0/882938169 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f3190003c70 con 0x7f31980ceb40 2023-10-19T14:21:04.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.808+0000 7f318f7fe640 1 -- 172.21.15.57:0/882938169 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3190034440 con 0x7f31980ceb40 2023-10-19T14:21:04.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.808+0000 7f319d539640 1 -- 172.21.15.57:0/882938169 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31980ca1c0 con 0x7f31980ceb40 2023-10-19T14:21:04.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.809+0000 7f319d539640 1 -- 172.21.15.57:0/882938169 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31980690e0 con 0x7f31980ceb40 2023-10-19T14:21:04.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.809+0000 7f318f7fe640 1 -- 172.21.15.57:0/882938169 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f319003b040 con 0x7f31980ceb40 2023-10-19T14:21:04.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.810+0000 7f318f7fe640 1 --2- 172.21.15.57:0/882938169 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f317806b080 0x7f317806d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:04.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.810+0000 7f318f7fe640 1 -- 172.21.15.57:0/882938169 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 6194+0+0 (secure 0 0 0) 0x7f31900abdb0 con 0x7f31980ceb40 2023-10-19T14:21:04.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.810+0000 7f31967fc640 1 --2- 172.21.15.57:0/882938169 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f317806b080 0x7f317806d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:04.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.810+0000 7f31967fc640 1 --2- 172.21.15.57:0/882938169 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f317806b080 0x7f317806d540 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f31980afb30 tx=0x7f3180009040 comp rx=0 tx=0).ready entity=mgr.9530 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:04.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.814+0000 7f318f7fe640 1 -- 172.21.15.57:0/882938169 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3190076920 con 0x7f31980ceb40 2023-10-19T14:21:04.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.988+0000 7f319d539640 1 -- 172.21.15.57:0/882938169 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f31980b0d00 con 0x7f31980ceb40 2023-10-19T14:21:04.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.992+0000 7f318f7fe640 1 -- 172.21.15.57:0/882938169 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v79)=0 v79) v1 ==== 122+0+0 (secure 0 0 0) 0x7f3190079f40 con 0x7f31980ceb40 2023-10-19T14:21:04.997 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.994+0000 7f319d539640 1 -- 172.21.15.57:0/882938169 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f317806b080 msgr2=0x7f317806d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:04.998 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.994+0000 7f319d539640 1 --2- 172.21.15.57:0/882938169 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f317806b080 0x7f317806d540 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f31980afb30 tx=0x7f3180009040 comp rx=0 tx=0).stop 2023-10-19T14:21:04.998 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.994+0000 7f319d539640 1 -- 172.21.15.57:0/882938169 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f31980ceb40 msgr2=0x7f3198156f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:04.998 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.994+0000 7f319d539640 1 --2- 172.21.15.57:0/882938169 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f31980ceb40 0x7f3198156f10 secure :-1 s=READY pgs=2587 cs=0 l=1 rev1=1 crypto rx=0x7f3190000c00 tx=0x7f3190004410 comp rx=0 tx=0).stop 2023-10-19T14:21:04.998 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.994+0000 7f319d539640 1 -- 172.21.15.57:0/882938169 shutdown_connections 2023-10-19T14:21:04.998 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.994+0000 7f319d539640 1 --2- 172.21.15.57:0/882938169 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f317806b080 0x7f317806d540 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:04.999 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.994+0000 7f319d539640 1 --2- 172.21.15.57:0/882938169 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f319815b2a0 0x7f3198158fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:04.999 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.994+0000 7f319d539640 1 --2- 172.21.15.57:0/882938169 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f31980cf340 0x7f31980ca6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:04.999 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.994+0000 7f319d539640 1 --2- 172.21.15.57:0/882938169 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f31980ceb40 0x7f3198156f10 unknown :-1 s=CLOSED pgs=2587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:05.000 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.994+0000 7f319d539640 1 -- 172.21.15.57:0/882938169 >> 172.21.15.57:0/882938169 conn(0x7f31980d0d60 msgr2=0x7f319806b680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:05.000 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.994+0000 7f319d539640 1 -- 172.21.15.57:0/882938169 shutdown_connections 2023-10-19T14:21:05.001 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:04.994+0000 7f319d539640 1 -- 172.21.15.57:0/882938169 wait complete. 2023-10-19T14:21:05.008 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-19T14:21:05.009 DEBUG:teuthology.orchestra.run.smithi057:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-19T14:21:05.057 DEBUG:teuthology.orchestra.run.smithi179:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-19T14:21:05.110 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-19T14:21:05.111 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T14:21:05.111 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T14:21:05.128 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055614,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=358 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248260k,nr_inodes=812065,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-19T14:21:05.129 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-19T14:21:05.129 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T14:21:05.150 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054833,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=284 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3247632k,nr_inodes=811908,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-19T14:21:05.151 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T14:21:05.152 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T14:21:05.184 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055614,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=358 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248260k,nr_inodes=812065,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-19T14:21:05.184 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-19T14:21:05.185 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T14:21:05.207 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054833,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=284 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3247632k,nr_inodes=811908,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-19T14:21:05.208 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-19T14:21:05.208 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T14:21:05.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.330+0000 7fb160e90640 1 -- 172.21.15.57:0/2903162415 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb15c0cd9e0 msgr2=0x7fb15c0cde50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:05.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.330+0000 7fb160e90640 1 --2- 172.21.15.57:0/2903162415 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb15c0cd9e0 0x7fb15c0cde50 secure :-1 s=READY pgs=2588 cs=0 l=1 rev1=1 crypto rx=0x7fb15c073460 tx=0x7fb15402c2f0 comp rx=0 tx=0).stop 2023-10-19T14:21:05.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.330+0000 7fb160e90640 1 -- 172.21.15.57:0/2903162415 shutdown_connections 2023-10-19T14:21:05.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.330+0000 7fb160e90640 1 --2- 172.21.15.57:0/2903162415 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb15c0cd9e0 0x7fb15c0cde50 unknown :-1 s=CLOSED pgs=2588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:05.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.330+0000 7fb160e90640 1 --2- 172.21.15.57:0/2903162415 >> v2:172.21.15.179:3300/0 conn(0x7fb15c158910 0x7fb15c158cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:05.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.330+0000 7fb160e90640 1 -- 172.21.15.57:0/2903162415 >> 172.21.15.57:0/2903162415 conn(0x7fb15c0af7a0 msgr2=0x7fb15c0afb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:05.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.330+0000 7fb160e90640 1 -- 172.21.15.57:0/2903162415 shutdown_connections 2023-10-19T14:21:05.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.330+0000 7fb160e90640 1 -- 172.21.15.57:0/2903162415 wait complete. 2023-10-19T14:21:05.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.331+0000 7fb160e90640 1 Processor -- start 2023-10-19T14:21:05.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.331+0000 7fb160e90640 1 -- start start 2023-10-19T14:21:05.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.332+0000 7fb160e90640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb15c0cd1c0 0x7fb15c155100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:05.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.332+0000 7fb160e90640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb15c0cd9e0 0x7fb15c1f2550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:05.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.332+0000 7fb160e90640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb15c158910 0x7fb15c142d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:05.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.332+0000 7fb160e90640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb15c155640 con 0x7fb15c0cd9e0 2023-10-19T14:21:05.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.332+0000 7fb160e90640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb15c1f2a90 con 0x7fb15c0cd1c0 2023-10-19T14:21:05.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.332+0000 7fb160e90640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb15c1f1a80 con 0x7fb15c158910 2023-10-19T14:21:05.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.332+0000 7fb151d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb15c0cd9e0 0x7fb15c1f2550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:05.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.332+0000 7fb151d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb15c0cd9e0 0x7fb15c1f2550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38662/0 (socket says 172.21.15.57:38662) 2023-10-19T14:21:05.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.332+0000 7fb151d9b640 1 -- 172.21.15.57:0/938351219 learned_addr learned my addr 172.21.15.57:0/938351219 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:05.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.332+0000 7fb15a59c640 1 --2- 172.21.15.57:0/938351219 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb15c0cd1c0 0x7fb15c155100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:05.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.332+0000 7fb15ad9d640 1 --2- 172.21.15.57:0/938351219 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb15c158910 0x7fb15c142d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:05.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.333+0000 7fb151d9b640 1 -- 172.21.15.57:0/938351219 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb15c158910 msgr2=0x7fb15c142d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:05.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.333+0000 7fb151d9b640 1 --2- 172.21.15.57:0/938351219 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb15c158910 0x7fb15c142d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:05.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.333+0000 7fb151d9b640 1 -- 172.21.15.57:0/938351219 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb15c0cd1c0 msgr2=0x7fb15c155100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:05.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.333+0000 7fb151d9b640 1 --2- 172.21.15.57:0/938351219 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb15c0cd1c0 0x7fb15c155100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:05.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.333+0000 7fb151d9b640 1 -- 172.21.15.57:0/938351219 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb154009900 con 0x7fb15c0cd9e0 2023-10-19T14:21:05.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.333+0000 7fb151d9b640 1 --2- 172.21.15.57:0/938351219 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb15c0cd9e0 0x7fb15c1f2550 secure :-1 s=READY pgs=2589 cs=0 l=1 rev1=1 crypto rx=0x7fb14400c400 tx=0x7fb14400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:05.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.333+0000 7fb1537fe640 1 -- 172.21.15.57:0/938351219 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1440052d0 con 0x7fb15c0cd9e0 2023-10-19T14:21:05.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.333+0000 7fb160e90640 1 -- 172.21.15.57:0/938351219 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb15c143940 con 0x7fb15c0cd9e0 2023-10-19T14:21:05.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.333+0000 7fb1537fe640 1 -- 172.21.15.57:0/938351219 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fb144007020 con 0x7fb15c0cd9e0 2023-10-19T14:21:05.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.333+0000 7fb1537fe640 1 -- 172.21.15.57:0/938351219 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb144007470 con 0x7fb15c0cd9e0 2023-10-19T14:21:05.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.333+0000 7fb160e90640 1 -- 172.21.15.57:0/938351219 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb15c1f1d50 con 0x7fb15c0cd9e0 2023-10-19T14:21:05.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.334+0000 7fb160e90640 1 -- 172.21.15.57:0/938351219 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb15c06d5f0 con 0x7fb15c0cd9e0 2023-10-19T14:21:05.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.335+0000 7fb1537fe640 1 -- 172.21.15.57:0/938351219 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fb1440067d0 con 0x7fb15c0cd9e0 2023-10-19T14:21:05.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.335+0000 7fb1537fe640 1 --2- 172.21.15.57:0/938351219 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fb13c06b080 0x7fb13c06d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:05.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.335+0000 7fb1537fe640 1 -- 172.21.15.57:0/938351219 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 6194+0+0 (secure 0 0 0) 0x7fb144089050 con 0x7fb15c0cd9e0 2023-10-19T14:21:05.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.335+0000 7fb15a59c640 1 --2- 172.21.15.57:0/938351219 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fb13c06b080 0x7fb13c06d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:05.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.336+0000 7fb15a59c640 1 --2- 172.21.15.57:0/938351219 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fb13c06b080 0x7fb13c06d540 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb15c15a420 tx=0x7fb14c00b2e0 comp rx=0 tx=0).ready entity=mgr.9530 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:05.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.339+0000 7fb1537fe640 1 -- 172.21.15.57:0/938351219 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb144053cb0 con 0x7fb15c0cd9e0 2023-10-19T14:21:05.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.552+0000 7fb160e90640 1 -- 172.21.15.57:0/938351219 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb15c15c7f0 con 0x7fb15c0cd9e0 2023-10-19T14:21:05.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.553+0000 7fb1537fe640 1 -- 172.21.15.57:0/938351219 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+3705 (secure 0 0 0) 0x7fb15c15c7f0 con 0x7fb15c0cd9e0 2023-10-19T14:21:05.556 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:21:05.557 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":98,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9248,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/1249807745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":1249807745},{"type":"v1","addr":"172.21.15.179:6836","nonce":1249807745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87},{"gid":10513,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/1406825818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":1406825818},{"type":"v1","addr":"172.21.15.179:6837","nonce":1406825818}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":86},{"gid":12375,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/2367396561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":2367396561},{"type":"v1","addr":"172.21.15.57:6835","nonce":2367396561}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":96},{"gid":12465,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/986128566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":986128566},{"type":"v1","addr":"172.21.15.57:6834","nonce":986128566}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98}],"filesystems":[]} 2023-10-19T14:21:05.557 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 98 2023-10-19T14:21:05.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.555+0000 7fb160e90640 1 -- 172.21.15.57:0/938351219 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fb13c06b080 msgr2=0x7fb13c06d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:05.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.555+0000 7fb160e90640 1 --2- 172.21.15.57:0/938351219 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fb13c06b080 0x7fb13c06d540 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb15c15a420 tx=0x7fb14c00b2e0 comp rx=0 tx=0).stop 2023-10-19T14:21:05.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.555+0000 7fb160e90640 1 -- 172.21.15.57:0/938351219 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb15c0cd9e0 msgr2=0x7fb15c1f2550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:05.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.555+0000 7fb160e90640 1 --2- 172.21.15.57:0/938351219 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb15c0cd9e0 0x7fb15c1f2550 secure :-1 s=READY pgs=2589 cs=0 l=1 rev1=1 crypto rx=0x7fb14400c400 tx=0x7fb14400c8c0 comp rx=0 tx=0).stop 2023-10-19T14:21:05.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.555+0000 7fb160e90640 1 -- 172.21.15.57:0/938351219 shutdown_connections 2023-10-19T14:21:05.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.555+0000 7fb160e90640 1 --2- 172.21.15.57:0/938351219 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fb13c06b080 0x7fb13c06d540 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:05.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.555+0000 7fb160e90640 1 --2- 172.21.15.57:0/938351219 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb15c158910 0x7fb15c142d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:05.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.555+0000 7fb160e90640 1 --2- 172.21.15.57:0/938351219 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb15c0cd9e0 0x7fb15c1f2550 unknown :-1 s=CLOSED pgs=2589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:05.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.555+0000 7fb160e90640 1 --2- 172.21.15.57:0/938351219 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb15c0cd1c0 0x7fb15c155100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:05.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.555+0000 7fb160e90640 1 -- 172.21.15.57:0/938351219 >> 172.21.15.57:0/938351219 conn(0x7fb15c0af7a0 msgr2=0x7fb15c152100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:05.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.555+0000 7fb160e90640 1 -- 172.21.15.57:0/938351219 shutdown_connections 2023-10-19T14:21:05.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.555+0000 7fb160e90640 1 -- 172.21.15.57:0/938351219 wait complete. 2023-10-19T14:21:05.569 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-19T14:21:05.569 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-19T14:21:05.570 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-19T14:21:05.570 INFO:teuthology.orchestra.run:waiting for 300 2023-10-19T14:21:05.570 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-19T14:21:05.571 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-19T14:21:05.571 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-19T14:21:05.571 INFO:teuthology.orchestra.run:waiting for 300 2023-10-19T14:21:05.571 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-19T14:21:05.572 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-19T14:21:05.572 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-19T14:21:05.573 INFO:teuthology.orchestra.run:waiting for 300 2023-10-19T14:21:05.573 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-19T14:21:05.573 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-19T14:21:05.574 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-19T14:21:05.574 INFO:teuthology.orchestra.run:waiting for 300 2023-10-19T14:21:05.579 INFO:tasks.ceph.mds.a:Stopped 2023-10-19T14:21:05.579 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-19T14:21:05.580 INFO:tasks.ceph.mds.c:Stopped 2023-10-19T14:21:05.581 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-19T14:21:05.584 INFO:tasks.ceph.mds.b:Stopped 2023-10-19T14:21:05.584 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-19T14:21:05.585 INFO:tasks.ceph.mds.d:Stopped 2023-10-19T14:21:05.585 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-19T14:21:05.623 INFO:tasks.ceph.mds.a:Started 2023-10-19T14:21:05.624 DEBUG:teuthology.parallel:result is None 2023-10-19T14:21:05.625 INFO:tasks.ceph.mds.c:Started 2023-10-19T14:21:05.625 DEBUG:teuthology.parallel:result is None 2023-10-19T14:21:05.628 INFO:tasks.ceph.mds.b:Started 2023-10-19T14:21:05.629 DEBUG:teuthology.parallel:result is None 2023-10-19T14:21:05.629 INFO:tasks.ceph.mds.d:Started 2023-10-19T14:21:05.630 DEBUG:teuthology.parallel:result is None 2023-10-19T14:21:05.630 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-19T14:21:05.702 INFO:tasks.ceph.mds.a.smithi057.stdout:starting mds.a at 2023-10-19T14:21:05.704 INFO:tasks.ceph.mds.c.smithi057.stdout:starting mds.c at 2023-10-19T14:21:05.709 INFO:tasks.ceph.mds.b.smithi179.stdout:starting mds.b at 2023-10-19T14:21:05.709 INFO:tasks.ceph.mds.d.smithi179.stdout:starting mds.d at 2023-10-19T14:21:05.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.742+0000 7fc890da9640 1 -- 172.21.15.57:0/388152299 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc88c0d1910 msgr2=0x7fc88c0b0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:05.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.742+0000 7fc890da9640 1 --2- 172.21.15.57:0/388152299 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc88c0d1910 0x7fc88c0b0650 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fc884009be0 tx=0x7fc88402bf90 comp rx=0 tx=0).stop 2023-10-19T14:21:05.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.742+0000 7fc890da9640 1 -- 172.21.15.57:0/388152299 shutdown_connections 2023-10-19T14:21:05.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.742+0000 7fc890da9640 1 --2- 172.21.15.57:0/388152299 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc88c0d1910 0x7fc88c0b0650 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:05.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.742+0000 7fc890da9640 1 -- 172.21.15.57:0/388152299 >> 172.21.15.57:0/388152299 conn(0x7fc88c0d0d60 msgr2=0x7fc88c0d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:05.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.742+0000 7fc890da9640 1 -- 172.21.15.57:0/388152299 shutdown_connections 2023-10-19T14:21:05.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.742+0000 7fc890da9640 1 -- 172.21.15.57:0/388152299 wait complete. 2023-10-19T14:21:05.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.743+0000 7fc890da9640 1 Processor -- start 2023-10-19T14:21:05.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.743+0000 7fc890da9640 1 -- start start 2023-10-19T14:21:05.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.744+0000 7fc890da9640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc88c0cf340 0x7fc88c158dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:05.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.744+0000 7fc890da9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc88c0d1910 0x7fc88c159300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:05.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.744+0000 7fc890da9640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc88c15b2a0 0x7fc88c0c8530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:05.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.744+0000 7fc890da9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc88c0cae10 con 0x7fc88c0d1910 2023-10-19T14:21:05.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.744+0000 7fc890da9640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc88c09bac0 con 0x7fc88c15b2a0 2023-10-19T14:21:05.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.744+0000 7fc890da9640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc88c0b47f0 con 0x7fc88c0cf340 2023-10-19T14:21:05.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.744+0000 7fc881d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc88c0d1910 0x7fc88c159300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:05.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.744+0000 7fc881d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc88c0d1910 0x7fc88c159300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38692/0 (socket says 172.21.15.57:38692) 2023-10-19T14:21:05.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.744+0000 7fc881d9b640 1 -- 172.21.15.57:0/3944517013 learned_addr learned my addr 172.21.15.57:0/3944517013 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:05.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.744+0000 7fc88a59c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc88c0cf340 0x7fc88c158dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:05.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.745+0000 7fc88ad9d640 1 --2- 172.21.15.57:0/3944517013 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc88c15b2a0 0x7fc88c0c8530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:05.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.745+0000 7fc881d9b640 1 -- 172.21.15.57:0/3944517013 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc88c0cf340 msgr2=0x7fc88c158dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:05.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.745+0000 7fc881d9b640 1 --2- 172.21.15.57:0/3944517013 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc88c0cf340 0x7fc88c158dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:05.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.745+0000 7fc881d9b640 1 -- 172.21.15.57:0/3944517013 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc88c15b2a0 msgr2=0x7fc88c0c8530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:05.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.745+0000 7fc881d9b640 1 --2- 172.21.15.57:0/3944517013 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc88c15b2a0 0x7fc88c0c8530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:05.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.745+0000 7fc881d9b640 1 -- 172.21.15.57:0/3944517013 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc884009840 con 0x7fc88c0d1910 2023-10-19T14:21:05.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.745+0000 7fc881d9b640 1 --2- 172.21.15.57:0/3944517013 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc88c0d1910 0x7fc88c159300 secure :-1 s=READY pgs=2597 cs=0 l=1 rev1=1 crypto rx=0x7fc87400c430 tx=0x7fc87400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:05.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.745+0000 7fc8837fe640 1 -- 172.21.15.57:0/3944517013 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8740052d0 con 0x7fc88c0d1910 2023-10-19T14:21:05.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.745+0000 7fc8837fe640 1 -- 172.21.15.57:0/3944517013 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fc8740035f0 con 0x7fc88c0d1910 2023-10-19T14:21:05.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.745+0000 7fc8837fe640 1 -- 172.21.15.57:0/3944517013 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc874003a00 con 0x7fc88c0d1910 2023-10-19T14:21:05.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.745+0000 7fc890da9640 1 -- 172.21.15.57:0/3944517013 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc88c0b4ad0 con 0x7fc88c0d1910 2023-10-19T14:21:05.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.745+0000 7fc890da9640 1 -- 172.21.15.57:0/3944517013 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc88c0b5090 con 0x7fc88c0d1910 2023-10-19T14:21:05.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.746+0000 7fc890da9640 1 -- 172.21.15.57:0/3944517013 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc88c09bd10 con 0x7fc88c0d1910 2023-10-19T14:21:05.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.747+0000 7fc8837fe640 1 -- 172.21.15.57:0/3944517013 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fc874003b60 con 0x7fc88c0d1910 2023-10-19T14:21:05.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.747+0000 7fc8837fe640 1 --2- 172.21.15.57:0/3944517013 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fc86c06b080 0x7fc86c06d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:05.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.747+0000 7fc8837fe640 1 -- 172.21.15.57:0/3944517013 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 6194+0+0 (secure 0 0 0) 0x7fc874089370 con 0x7fc88c0d1910 2023-10-19T14:21:05.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.747+0000 7fc88a59c640 1 --2- 172.21.15.57:0/3944517013 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fc86c06b080 0x7fc86c06d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:05.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.748+0000 7fc88a59c640 1 --2- 172.21.15.57:0/3944517013 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fc86c06b080 0x7fc86c06d540 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc87c001eb0 tx=0x7fc87c001430 comp rx=0 tx=0).ready entity=mgr.9530 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:05.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.751+0000 7fc8837fe640 1 -- 172.21.15.57:0/3944517013 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc88c09bd10 con 0x7fc88c0d1910 2023-10-19T14:21:05.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:05.939+0000 7fc890da9640 1 -- 172.21.15.57:0/3944517013 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fc88c09bd10 con 0x7fc88c0d1910 2023-10-19T14:21:06.329 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:21:06.332 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:21:06.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.374+0000 7fc8837fe640 1 -- 172.21.15.57:0/3944517013 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v237) v1 ==== 97+0+0 (secure 0 0 0) 0x7fc88c09bd10 con 0x7fc88c0d1910 2023-10-19T14:21:06.378 INFO:teuthology.orchestra.run.smithi057.stderr: removed all blocklist entries 2023-10-19T14:21:06.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.376+0000 7fc890da9640 1 -- 172.21.15.57:0/3944517013 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fc86c06b080 msgr2=0x7fc86c06d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:06.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.376+0000 7fc890da9640 1 --2- 172.21.15.57:0/3944517013 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fc86c06b080 0x7fc86c06d540 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc87c001eb0 tx=0x7fc87c001430 comp rx=0 tx=0).stop 2023-10-19T14:21:06.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.376+0000 7fc890da9640 1 -- 172.21.15.57:0/3944517013 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc88c0d1910 msgr2=0x7fc88c159300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:06.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.376+0000 7fc890da9640 1 --2- 172.21.15.57:0/3944517013 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc88c0d1910 0x7fc88c159300 secure :-1 s=READY pgs=2597 cs=0 l=1 rev1=1 crypto rx=0x7fc87400c430 tx=0x7fc87400c8f0 comp rx=0 tx=0).stop 2023-10-19T14:21:06.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.376+0000 7fc890da9640 1 -- 172.21.15.57:0/3944517013 shutdown_connections 2023-10-19T14:21:06.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.377+0000 7fc890da9640 1 --2- 172.21.15.57:0/3944517013 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fc86c06b080 0x7fc86c06d540 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:06.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.377+0000 7fc890da9640 1 --2- 172.21.15.57:0/3944517013 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc88c15b2a0 0x7fc88c0c8530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:06.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.377+0000 7fc890da9640 1 --2- 172.21.15.57:0/3944517013 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc88c0d1910 0x7fc88c159300 unknown :-1 s=CLOSED pgs=2597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:06.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.377+0000 7fc890da9640 1 --2- 172.21.15.57:0/3944517013 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc88c0cf340 0x7fc88c158dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:06.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.377+0000 7fc890da9640 1 -- 172.21.15.57:0/3944517013 >> 172.21.15.57:0/3944517013 conn(0x7fc88c0d0d60 msgr2=0x7fc88c0ad8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:06.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.377+0000 7fc890da9640 1 -- 172.21.15.57:0/3944517013 shutdown_connections 2023-10-19T14:21:06.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.377+0000 7fc890da9640 1 -- 172.21.15.57:0/3944517013 wait complete. 2023-10-19T14:21:06.392 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-19T14:21:06.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.540+0000 7fc42d3d0640 1 -- 172.21.15.57:0/104829697 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4280afa20 msgr2=0x7fc428158300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:06.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.540+0000 7fc42d3d0640 1 --2- 172.21.15.57:0/104829697 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4280afa20 0x7fc428158300 secure :-1 s=READY pgs=2598 cs=0 l=1 rev1=1 crypto rx=0x7fc4280b10f0 tx=0x7fc41002bdb0 comp rx=0 tx=0).stop 2023-10-19T14:21:06.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.540+0000 7fc42d3d0640 1 -- 172.21.15.57:0/104829697 shutdown_connections 2023-10-19T14:21:06.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.540+0000 7fc42d3d0640 1 --2- 172.21.15.57:0/104829697 >> v2:172.21.15.179:3300/0 conn(0x7fc4280b0530 0x7fc4280b09a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:06.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.540+0000 7fc42d3d0640 1 --2- 172.21.15.57:0/104829697 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4280afa20 0x7fc428158300 unknown :-1 s=CLOSED pgs=2598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:06.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.540+0000 7fc42d3d0640 1 -- 172.21.15.57:0/104829697 >> 172.21.15.57:0/104829697 conn(0x7fc428069930 msgr2=0x7fc428069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:06.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.540+0000 7fc42d3d0640 1 -- 172.21.15.57:0/104829697 shutdown_connections 2023-10-19T14:21:06.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.540+0000 7fc42d3d0640 1 -- 172.21.15.57:0/104829697 wait complete. 2023-10-19T14:21:06.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.541+0000 7fc42d3d0640 1 Processor -- start 2023-10-19T14:21:06.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.541+0000 7fc42d3d0640 1 -- start start 2023-10-19T14:21:06.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.542+0000 7fc42d3d0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4280afa20 0x7fc428151eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:06.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.542+0000 7fc42d3d0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc4280b0530 0x7fc42814cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:06.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.542+0000 7fc42d3d0640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc4280cbd20 0x7fc428138df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:06.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.542+0000 7fc42d3d0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4281537a0 con 0x7fc4280afa20 2023-10-19T14:21:06.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.542+0000 7fc42d3d0640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc428152700 con 0x7fc4280b0530 2023-10-19T14:21:06.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.542+0000 7fc42d3d0640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4281529e0 con 0x7fc4280cbd20 2023-10-19T14:21:06.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.542+0000 7fc426ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4280afa20 0x7fc428151eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:06.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.542+0000 7fc426ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4280afa20 0x7fc428151eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38712/0 (socket says 172.21.15.57:38712) 2023-10-19T14:21:06.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.542+0000 7fc426ffd640 1 -- 172.21.15.57:0/3148387260 learned_addr learned my addr 172.21.15.57:0/3148387260 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:06.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.542+0000 7fc41ffff640 1 --2- 172.21.15.57:0/3148387260 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc4280b0530 0x7fc42814cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:06.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.542+0000 7fc4277fe640 1 --2- 172.21.15.57:0/3148387260 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc4280cbd20 0x7fc428138df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:06.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.542+0000 7fc426ffd640 1 -- 172.21.15.57:0/3148387260 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc4280cbd20 msgr2=0x7fc428138df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:06.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.542+0000 7fc426ffd640 1 --2- 172.21.15.57:0/3148387260 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc4280cbd20 0x7fc428138df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:06.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.542+0000 7fc426ffd640 1 -- 172.21.15.57:0/3148387260 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc4280b0530 msgr2=0x7fc42814cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:06.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.542+0000 7fc426ffd640 1 --2- 172.21.15.57:0/3148387260 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc4280b0530 0x7fc42814cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:06.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.542+0000 7fc426ffd640 1 -- 172.21.15.57:0/3148387260 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc410009560 con 0x7fc4280afa20 2023-10-19T14:21:06.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.542+0000 7fc41ffff640 1 --2- 172.21.15.57:0/3148387260 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc4280b0530 0x7fc42814cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:21:06.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.542+0000 7fc4277fe640 1 --2- 172.21.15.57:0/3148387260 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc4280cbd20 0x7fc428138df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:21:06.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.542+0000 7fc426ffd640 1 --2- 172.21.15.57:0/3148387260 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4280afa20 0x7fc428151eb0 secure :-1 s=READY pgs=2599 cs=0 l=1 rev1=1 crypto rx=0x7fc41800b3e0 tx=0x7fc41800b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:06.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.542+0000 7fc41f7fe640 1 -- 172.21.15.57:0/3148387260 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc418011830 con 0x7fc4280afa20 2023-10-19T14:21:06.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.542+0000 7fc41f7fe640 1 -- 172.21.15.57:0/3148387260 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fc418011aa0 con 0x7fc4280afa20 2023-10-19T14:21:06.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.542+0000 7fc42d3d0640 1 -- 172.21.15.57:0/3148387260 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc428152cc0 con 0x7fc4280afa20 2023-10-19T14:21:06.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.542+0000 7fc41f7fe640 1 -- 172.21.15.57:0/3148387260 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc418010380 con 0x7fc4280afa20 2023-10-19T14:21:06.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.542+0000 7fc42d3d0640 1 -- 172.21.15.57:0/3148387260 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc428153280 con 0x7fc4280afa20 2023-10-19T14:21:06.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.543+0000 7fc42d3d0640 1 -- 172.21.15.57:0/3148387260 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4280690e0 con 0x7fc4280afa20 2023-10-19T14:21:06.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.544+0000 7fc41f7fe640 1 -- 172.21.15.57:0/3148387260 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fc4180104e0 con 0x7fc4280afa20 2023-10-19T14:21:06.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.544+0000 7fc41f7fe640 1 --2- 172.21.15.57:0/3148387260 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fc40806b080 0x7fc40806d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:06.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.545+0000 7fc41f7fe640 1 -- 172.21.15.57:0/3148387260 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(239..239 src has 1..239) v4 ==== 5205+0+0 (secure 0 0 0) 0x7fc418089ba0 con 0x7fc4280afa20 2023-10-19T14:21:06.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.545+0000 7fc41ffff640 1 --2- 172.21.15.57:0/3148387260 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fc40806b080 0x7fc40806d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:06.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.545+0000 7fc41ffff640 1 --2- 172.21.15.57:0/3148387260 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fc40806b080 0x7fc40806d540 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc410035000 tx=0x7fc410004730 comp rx=0 tx=0).ready entity=mgr.9530 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:06.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.549+0000 7fc41f7fe640 1 -- 172.21.15.57:0/3148387260 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc418054b70 con 0x7fc4280afa20 2023-10-19T14:21:06.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.796+0000 7fc42d3d0640 1 -- 172.21.15.57:0/3148387260 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fc42806f700 con 0x7fc4280afa20 2023-10-19T14:21:06.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.796+0000 7fc41f7fe640 1 -- 172.21.15.57:0/3148387260 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v60) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fc418015090 con 0x7fc4280afa20 2023-10-19T14:21:06.800 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:21:06.800 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2023-10-19T14:21:06.800 INFO:teuthology.orchestra.run.smithi057.stdout: "auth_dump": [ 2023-10-19T14:21:06.800 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:21:06.801 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "mds.a", 2023-10-19T14:21:06.801 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBjJDFlmbVANhAAP/ynHlU/OxaRq5lQoAa8Ng==", 2023-10-19T14:21:06.801 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:21:06.801 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T14:21:06.801 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow *", 2023-10-19T14:21:06.802 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow *", 2023-10-19T14:21:06.802 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:21:06.802 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:21:06.802 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:21:06.802 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:21:06.803 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "mds.b", 2023-10-19T14:21:06.803 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFl9l7RBxAAC0geI2tJ1SIZZr1++kS/BA==", 2023-10-19T14:21:06.803 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:21:06.803 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T14:21:06.803 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow *", 2023-10-19T14:21:06.804 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow *", 2023-10-19T14:21:06.804 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:21:06.804 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:21:06.804 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:21:06.804 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:21:06.804 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "mds.c", 2023-10-19T14:21:06.805 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFlFkaoAhAA/PBizkJ4bm958oP4CmPuIQ==", 2023-10-19T14:21:06.805 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:21:06.805 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T14:21:06.805 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow *", 2023-10-19T14:21:06.805 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow *", 2023-10-19T14:21:06.806 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:21:06.806 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:21:06.806 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:21:06.806 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:21:06.806 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "mds.d", 2023-10-19T14:21:06.807 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFlx8skDxAAAfCHTYtYJgJik0WHSWeQfQ==", 2023-10-19T14:21:06.807 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:21:06.807 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T14:21:06.807 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow *", 2023-10-19T14:21:06.807 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow *", 2023-10-19T14:21:06.808 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:21:06.808 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:21:06.808 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:21:06.808 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:21:06.808 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.0", 2023-10-19T14:21:06.809 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBrJDFlCo9sBxAAQN2cID4nCQgRwGaGFEV8LQ==", 2023-10-19T14:21:06.809 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:21:06.809 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T14:21:06.809 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T14:21:06.809 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:21:06.810 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:21:06.810 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:21:06.810 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:21:06.810 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.1", 2023-10-19T14:21:06.811 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBuJDFl4fsLCBAAp3Pi80yZtc5sju0GisEuVA==", 2023-10-19T14:21:06.811 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:21:06.811 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T14:21:06.811 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T14:21:06.811 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:21:06.812 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:21:06.812 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:21:06.812 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:21:06.812 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.2", 2023-10-19T14:21:06.812 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBxJDFl+KAKCBAAd4N9cluARxPVfBLU+7Hd7A==", 2023-10-19T14:21:06.812 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:21:06.813 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T14:21:06.813 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T14:21:06.813 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:21:06.813 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:21:06.813 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:21:06.814 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:21:06.814 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.3", 2023-10-19T14:21:06.814 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQB0JDFlU/toCBAAP0Ou1zcHKNx5SfbybJkh4Q==", 2023-10-19T14:21:06.814 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:21:06.814 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T14:21:06.815 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T14:21:06.815 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:21:06.815 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:21:06.815 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:21:06.815 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:21:06.816 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.4", 2023-10-19T14:21:06.816 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQB6JDFlc6stEBAALkgNyxoiCTzZnKpBSPxI6w==", 2023-10-19T14:21:06.816 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:21:06.816 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T14:21:06.816 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T14:21:06.817 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:21:06.817 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:21:06.817 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:21:06.817 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:21:06.817 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.5", 2023-10-19T14:21:06.818 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQB9JDFlI3cNDhAAuG3tHJ6RvPwAzPE76wTIVQ==", 2023-10-19T14:21:06.818 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:21:06.818 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T14:21:06.818 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T14:21:06.818 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:21:06.819 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:21:06.819 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:21:06.819 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:21:06.819 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.6", 2023-10-19T14:21:06.819 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQCAJDFlLPumCxAA/iy08t/2N9rXfn71KtbSQw==", 2023-10-19T14:21:06.820 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:21:06.820 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T14:21:06.820 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T14:21:06.820 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:21:06.820 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:21:06.820 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:21:06.821 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:21:06.821 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.7", 2023-10-19T14:21:06.821 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQCDJDFlHmrbCRAA+hpWygGIJydfU6cHWlfx/Q==", 2023-10-19T14:21:06.821 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:21:06.821 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T14:21:06.822 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T14:21:06.822 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:21:06.822 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:21:06.822 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:21:06.822 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:21:06.823 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "client.0", 2023-10-19T14:21:06.823 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFlg9yuExAATxiWDfS+p07G1F0pORbJEg==", 2023-10-19T14:21:06.823 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:21:06.823 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T14:21:06.823 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow r", 2023-10-19T14:21:06.824 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-19T14:21:06.824 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:21:06.824 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:21:06.824 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:21:06.824 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "client.1", 2023-10-19T14:21:06.824 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFl9IwYGxAAHPituiJSPkHDuD0QyG4Hww==", 2023-10-19T14:21:06.825 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:21:06.825 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T14:21:06.825 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow r", 2023-10-19T14:21:06.825 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow rw", 2023-10-19T14:21:06.825 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow rwx" 2023-10-19T14:21:06.826 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:21:06.826 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:21:06.826 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:21:06.826 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "client.2", 2023-10-19T14:21:06.826 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFlMd+AFxAAS/UTESDt76GmrhhN2o/Fdw==", 2023-10-19T14:21:06.827 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:21:06.827 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T14:21:06.827 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow r", 2023-10-19T14:21:06.827 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow rw", 2023-10-19T14:21:06.827 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow rwx" 2023-10-19T14:21:06.828 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:21:06.828 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:21:06.828 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:21:06.828 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "client.3", 2023-10-19T14:21:06.828 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFlHZ3fIBAA42B7n8opLS5smXIwJyVYzw==", 2023-10-19T14:21:06.828 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:21:06.829 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T14:21:06.829 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow r", 2023-10-19T14:21:06.829 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow rw", 2023-10-19T14:21:06.829 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow rwx" 2023-10-19T14:21:06.829 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:21:06.830 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:21:06.830 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:21:06.830 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "client.admin", 2023-10-19T14:21:06.830 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBjJDFlBgasFxAAMA24j8KszcXMEpYRQDXj4g==", 2023-10-19T14:21:06.830 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:21:06.831 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow *", 2023-10-19T14:21:06.831 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow *", 2023-10-19T14:21:06.831 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow *", 2023-10-19T14:21:06.831 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:21:06.831 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:21:06.832 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:21:06.832 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:21:06.832 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "mgr.x", 2023-10-19T14:21:06.832 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBjJDFllFBfMhAA5+KPXWmYZqcESmLObaUbBA==", 2023-10-19T14:21:06.832 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:21:06.832 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow *", 2023-10-19T14:21:06.833 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile mgr", 2023-10-19T14:21:06.833 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:21:06.833 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:21:06.833 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:21:06.833 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:21:06.834 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "mgr.y", 2023-10-19T14:21:06.834 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBjJDFlxRzJLRAAGjG58o7qXAHvIEIRYwl6QA==", 2023-10-19T14:21:06.834 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:21:06.834 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow *", 2023-10-19T14:21:06.834 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile mgr", 2023-10-19T14:21:06.835 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:21:06.835 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:21:06.835 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:21:06.835 INFO:teuthology.orchestra.run.smithi057.stdout: ] 2023-10-19T14:21:06.835 INFO:teuthology.orchestra.run.smithi057.stdout:} 2023-10-19T14:21:06.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.799+0000 7fc42d3d0640 1 -- 172.21.15.57:0/3148387260 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fc40806b080 msgr2=0x7fc40806d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:06.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.799+0000 7fc42d3d0640 1 --2- 172.21.15.57:0/3148387260 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fc40806b080 0x7fc40806d540 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc410035000 tx=0x7fc410004730 comp rx=0 tx=0).stop 2023-10-19T14:21:06.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.799+0000 7fc42d3d0640 1 -- 172.21.15.57:0/3148387260 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4280afa20 msgr2=0x7fc428151eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:06.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.799+0000 7fc42d3d0640 1 --2- 172.21.15.57:0/3148387260 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4280afa20 0x7fc428151eb0 secure :-1 s=READY pgs=2599 cs=0 l=1 rev1=1 crypto rx=0x7fc41800b3e0 tx=0x7fc41800b8a0 comp rx=0 tx=0).stop 2023-10-19T14:21:06.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.799+0000 7fc42d3d0640 1 -- 172.21.15.57:0/3148387260 shutdown_connections 2023-10-19T14:21:06.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.799+0000 7fc42d3d0640 1 --2- 172.21.15.57:0/3148387260 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fc40806b080 0x7fc40806d540 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:06.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.799+0000 7fc42d3d0640 1 --2- 172.21.15.57:0/3148387260 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc4280cbd20 0x7fc428138df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:06.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.799+0000 7fc42d3d0640 1 --2- 172.21.15.57:0/3148387260 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc4280b0530 0x7fc42814cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:06.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.799+0000 7fc42d3d0640 1 --2- 172.21.15.57:0/3148387260 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4280afa20 0x7fc428151eb0 unknown :-1 s=CLOSED pgs=2599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:06.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.799+0000 7fc42d3d0640 1 -- 172.21.15.57:0/3148387260 >> 172.21.15.57:0/3148387260 conn(0x7fc428069930 msgr2=0x7fc4280aef90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:06.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.799+0000 7fc42d3d0640 1 -- 172.21.15.57:0/3148387260 shutdown_connections 2023-10-19T14:21:06.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.799+0000 7fc42d3d0640 1 -- 172.21.15.57:0/3148387260 wait complete. 2023-10-19T14:21:06.840 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T14:21:06.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.947+0000 7fc080f2c640 1 -- 172.21.15.57:0/2693911184 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc07c13e520 msgr2=0x7fc07c06bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:06.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.947+0000 7fc080f2c640 1 --2- 172.21.15.57:0/2693911184 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc07c13e520 0x7fc07c06bb60 secure :-1 s=READY pgs=2601 cs=0 l=1 rev1=1 crypto rx=0x7fc06c009830 tx=0x7fc06c02bc30 comp rx=0 tx=0).stop 2023-10-19T14:21:06.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.947+0000 7fc080f2c640 1 -- 172.21.15.57:0/2693911184 shutdown_connections 2023-10-19T14:21:06.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.947+0000 7fc080f2c640 1 --2- 172.21.15.57:0/2693911184 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc07c141690 0x7fc07c0703e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:06.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.947+0000 7fc080f2c640 1 --2- 172.21.15.57:0/2693911184 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc07c13e520 0x7fc07c06bb60 unknown :-1 s=CLOSED pgs=2601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:06.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.947+0000 7fc080f2c640 1 -- 172.21.15.57:0/2693911184 >> 172.21.15.57:0/2693911184 conn(0x7fc07c069930 msgr2=0x7fc07c069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:06.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.947+0000 7fc080f2c640 1 -- 172.21.15.57:0/2693911184 shutdown_connections 2023-10-19T14:21:06.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.947+0000 7fc080f2c640 1 -- 172.21.15.57:0/2693911184 wait complete. 2023-10-19T14:21:06.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.948+0000 7fc080f2c640 1 Processor -- start 2023-10-19T14:21:06.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.948+0000 7fc080f2c640 1 -- start start 2023-10-19T14:21:06.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.949+0000 7fc080f2c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc07c06c0a0 0x7fc07c1e2ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:06.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.949+0000 7fc080f2c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc07c13e520 0x7fc07c1e38f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:06.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.949+0000 7fc080f2c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc07c141690 0x7fc07c13b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:06.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.949+0000 7fc080f2c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc07c1e33b0 con 0x7fc07c06c0a0 2023-10-19T14:21:06.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.949+0000 7fc080f2c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc07c1e30d0 con 0x7fc07c141690 2023-10-19T14:21:06.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.949+0000 7fc080f2c640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc07c0a6650 con 0x7fc07c13e520 2023-10-19T14:21:06.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.949+0000 7fc07a59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc07c06c0a0 0x7fc07c1e2ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:06.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.949+0000 7fc07a59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc07c06c0a0 0x7fc07c1e2ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51458/0 (socket says 172.21.15.57:51458) 2023-10-19T14:21:06.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.949+0000 7fc07a59c640 1 -- 172.21.15.57:0/1754587791 learned_addr learned my addr 172.21.15.57:0/1754587791 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:06.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.949+0000 7fc073fff640 1 --2- 172.21.15.57:0/1754587791 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc07c13e520 0x7fc07c1e38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:06.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.949+0000 7fc07a59c640 1 -- 172.21.15.57:0/1754587791 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc07c13e520 msgr2=0x7fc07c1e38f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:06.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.949+0000 7fc07a59c640 1 --2- 172.21.15.57:0/1754587791 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc07c13e520 0x7fc07c1e38f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:06.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.949+0000 7fc07a59c640 1 -- 172.21.15.57:0/1754587791 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc07c141690 msgr2=0x7fc07c13b010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:06.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.949+0000 7fc07a59c640 1 --2- 172.21.15.57:0/1754587791 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc07c141690 0x7fc07c13b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:06.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.949+0000 7fc07a59c640 1 -- 172.21.15.57:0/1754587791 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc06c009490 con 0x7fc07c06c0a0 2023-10-19T14:21:06.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.949+0000 7fc073fff640 1 --2- 172.21.15.57:0/1754587791 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc07c13e520 0x7fc07c1e38f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:21:06.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.950+0000 7fc07a59c640 1 --2- 172.21.15.57:0/1754587791 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc07c06c0a0 0x7fc07c1e2ad0 secure :-1 s=READY pgs=2602 cs=0 l=1 rev1=1 crypto rx=0x7fc06c000c00 tx=0x7fc06c004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:06.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.950+0000 7fc072ffd640 1 -- 172.21.15.57:0/1754587791 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc06c005220 con 0x7fc07c06c0a0 2023-10-19T14:21:06.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.950+0000 7fc072ffd640 1 -- 172.21.15.57:0/1754587791 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fc06c005470 con 0x7fc07c06c0a0 2023-10-19T14:21:06.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.950+0000 7fc072ffd640 1 -- 172.21.15.57:0/1754587791 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc06c003800 con 0x7fc07c06c0a0 2023-10-19T14:21:06.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.950+0000 7fc080f2c640 1 -- 172.21.15.57:0/1754587791 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc07c0a6930 con 0x7fc07c06c0a0 2023-10-19T14:21:06.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.950+0000 7fc080f2c640 1 -- 172.21.15.57:0/1754587791 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc07c1e3e30 con 0x7fc07c06c0a0 2023-10-19T14:21:06.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.951+0000 7fc080f2c640 1 -- 172.21.15.57:0/1754587791 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc07c0a64b0 con 0x7fc07c06c0a0 2023-10-19T14:21:06.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.951+0000 7fc072ffd640 1 -- 172.21.15.57:0/1754587791 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fc06c003960 con 0x7fc07c06c0a0 2023-10-19T14:21:06.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.952+0000 7fc072ffd640 1 --2- 172.21.15.57:0/1754587791 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fc05c06b080 0x7fc05c06d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:06.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.952+0000 7fc072ffd640 1 -- 172.21.15.57:0/1754587791 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(239..239 src has 1..239) v4 ==== 5205+0+0 (secure 0 0 0) 0x7fc06c0abdc0 con 0x7fc07c06c0a0 2023-10-19T14:21:06.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.952+0000 7fc073fff640 1 --2- 172.21.15.57:0/1754587791 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fc05c06b080 0x7fc05c06d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:06.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.952+0000 7fc073fff640 1 --2- 172.21.15.57:0/1754587791 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fc05c06b080 0x7fc05c06d540 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc07c0cb6a0 tx=0x7fc064005db0 comp rx=0 tx=0).ready entity=mgr.9530 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:06.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:06.956+0000 7fc072ffd640 1 -- 172.21.15.57:0/1754587791 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc07c0a64b0 con 0x7fc07c06c0a0 2023-10-19T14:21:07.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.133+0000 7fc080f2c640 1 -- 172.21.15.57:0/1754587791 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc07c0a64b0 con 0x7fc07c06c0a0 2023-10-19T14:21:07.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.134+0000 7fc072ffd640 1 -- 172.21.15.57:0/1754587791 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v239) v1 ==== 74+0+13127 (secure 0 0 0) 0x7fc07c0a64b0 con 0x7fc07c06c0a0 2023-10-19T14:21:07.139 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:21:07.139 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":239,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T14:21:06.382869+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T14:21:07.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.136+0000 7fc080f2c640 1 -- 172.21.15.57:0/1754587791 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fc05c06b080 msgr2=0x7fc05c06d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:07.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.136+0000 7fc080f2c640 1 --2- 172.21.15.57:0/1754587791 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fc05c06b080 0x7fc05c06d540 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc07c0cb6a0 tx=0x7fc064005db0 comp rx=0 tx=0).stop 2023-10-19T14:21:07.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.136+0000 7fc080f2c640 1 -- 172.21.15.57:0/1754587791 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc07c06c0a0 msgr2=0x7fc07c1e2ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:07.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.136+0000 7fc080f2c640 1 --2- 172.21.15.57:0/1754587791 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc07c06c0a0 0x7fc07c1e2ad0 secure :-1 s=READY pgs=2602 cs=0 l=1 rev1=1 crypto rx=0x7fc06c000c00 tx=0x7fc06c004360 comp rx=0 tx=0).stop 2023-10-19T14:21:07.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.136+0000 7fc080f2c640 1 -- 172.21.15.57:0/1754587791 shutdown_connections 2023-10-19T14:21:07.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.136+0000 7fc080f2c640 1 --2- 172.21.15.57:0/1754587791 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fc05c06b080 0x7fc05c06d540 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:07.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.136+0000 7fc080f2c640 1 --2- 172.21.15.57:0/1754587791 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc07c141690 0x7fc07c13b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:07.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.136+0000 7fc080f2c640 1 --2- 172.21.15.57:0/1754587791 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc07c13e520 0x7fc07c1e38f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:07.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.136+0000 7fc080f2c640 1 --2- 172.21.15.57:0/1754587791 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc07c06c0a0 0x7fc07c1e2ad0 unknown :-1 s=CLOSED pgs=2602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:07.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.136+0000 7fc080f2c640 1 -- 172.21.15.57:0/1754587791 >> 172.21.15.57:0/1754587791 conn(0x7fc07c069930 msgr2=0x7fc07c06f080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:07.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.136+0000 7fc080f2c640 1 -- 172.21.15.57:0/1754587791 shutdown_connections 2023-10-19T14:21:07.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.136+0000 7fc080f2c640 1 -- 172.21.15.57:0/1754587791 wait complete. 2023-10-19T14:21:07.150 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-19T12:44:07.371186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-19T14:21:07.151 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-19T14:21:07.263 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.260+0000 7f529d397640 1 -- 172.21.15.57:0/1564617815 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f52980b0c40 msgr2=0x7f5298158300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:07.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.260+0000 7f529d397640 1 --2- 172.21.15.57:0/1564617815 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f52980b0c40 0x7f5298158300 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f5288005fd0 tx=0x7f528802beb0 comp rx=0 tx=0).stop 2023-10-19T14:21:07.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.260+0000 7f529d397640 1 -- 172.21.15.57:0/1564617815 shutdown_connections 2023-10-19T14:21:07.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.260+0000 7f529d397640 1 --2- 172.21.15.57:0/1564617815 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f52980b0c40 0x7f5298158300 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:07.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.260+0000 7f529d397640 1 -- 172.21.15.57:0/1564617815 >> 172.21.15.57:0/1564617815 conn(0x7f5298069930 msgr2=0x7f5298069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:07.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.260+0000 7f529d397640 1 -- 172.21.15.57:0/1564617815 shutdown_connections 2023-10-19T14:21:07.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.260+0000 7f529d397640 1 -- 172.21.15.57:0/1564617815 wait complete. 2023-10-19T14:21:07.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.261+0000 7f529d397640 1 Processor -- start 2023-10-19T14:21:07.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.261+0000 7f529d397640 1 -- start start 2023-10-19T14:21:07.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.262+0000 7f529d397640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f52980afa20 0x7f52981405a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:07.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.262+0000 7f529d397640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f52980b0c40 0x7f52980ca870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:07.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.262+0000 7f529d397640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f52980cbd20 0x7f52980cadb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:07.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.262+0000 7f529d397640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f529812cd60 con 0x7f52980b0c40 2023-10-19T14:21:07.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.262+0000 7f529d397640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5298145740 con 0x7f52980cbd20 2023-10-19T14:21:07.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.262+0000 7f529d397640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f529812d000 con 0x7f52980afa20 2023-10-19T14:21:07.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.262+0000 7f52967fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f52980b0c40 0x7f52980ca870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:07.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.262+0000 7f52967fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f52980b0c40 0x7f52980ca870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51462/0 (socket says 172.21.15.57:51462) 2023-10-19T14:21:07.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.262+0000 7f52967fc640 1 -- 172.21.15.57:0/1940262969 learned_addr learned my addr 172.21.15.57:0/1940262969 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:07.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.262+0000 7f52977fe640 1 --2- 172.21.15.57:0/1940262969 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f52980cbd20 0x7f52980cadb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:07.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.262+0000 7f52967fc640 1 -- 172.21.15.57:0/1940262969 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f52980afa20 msgr2=0x7f52981405a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:07.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.262+0000 7f5296ffd640 1 --2- 172.21.15.57:0/1940262969 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f52980afa20 0x7f52981405a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:07.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.263+0000 7f52967fc640 1 --2- 172.21.15.57:0/1940262969 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f52980afa20 0x7f52981405a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:07.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.263+0000 7f52967fc640 1 -- 172.21.15.57:0/1940262969 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f52980cbd20 msgr2=0x7f52980cadb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:07.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.263+0000 7f52967fc640 1 --2- 172.21.15.57:0/1940262969 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f52980cbd20 0x7f52980cadb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:07.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.263+0000 7f52967fc640 1 -- 172.21.15.57:0/1940262969 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5288009840 con 0x7f52980b0c40 2023-10-19T14:21:07.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.263+0000 7f52977fe640 1 --2- 172.21.15.57:0/1940262969 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f52980cbd20 0x7f52980cadb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:21:07.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.263+0000 7f5296ffd640 1 --2- 172.21.15.57:0/1940262969 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f52980afa20 0x7f52981405a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:21:07.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.263+0000 7f52967fc640 1 --2- 172.21.15.57:0/1940262969 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f52980b0c40 0x7f52980ca870 secure :-1 s=READY pgs=2604 cs=0 l=1 rev1=1 crypto rx=0x7f528000c750 tx=0x7f528000cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:07.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.263+0000 7f528f7fe640 1 -- 172.21.15.57:0/1940262969 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5280005270 con 0x7f52980b0c40 2023-10-19T14:21:07.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.263+0000 7f529d397640 1 -- 172.21.15.57:0/1940262969 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f529812d2e0 con 0x7f52980b0c40 2023-10-19T14:21:07.271 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.263+0000 7f528f7fe640 1 -- 172.21.15.57:0/1940262969 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f528000a640 con 0x7f52980b0c40 2023-10-19T14:21:07.272 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.263+0000 7f528f7fe640 1 -- 172.21.15.57:0/1940262969 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f528000aa90 con 0x7f52980b0c40 2023-10-19T14:21:07.272 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.263+0000 7f529d397640 1 -- 172.21.15.57:0/1940262969 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5298140ae0 con 0x7f52980b0c40 2023-10-19T14:21:07.272 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.264+0000 7f529d397640 1 -- 172.21.15.57:0/1940262969 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f529812d520 con 0x7f52980b0c40 2023-10-19T14:21:07.272 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.264+0000 7f528f7fe640 1 -- 172.21.15.57:0/1940262969 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f528000abf0 con 0x7f52980b0c40 2023-10-19T14:21:07.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.265+0000 7f528f7fe640 1 --2- 172.21.15.57:0/1940262969 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f527806b0b0 0x7f527806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:07.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.265+0000 7f528f7fe640 1 -- 172.21.15.57:0/1940262969 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(239..239 src has 1..239) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f52800892b0 con 0x7f52980b0c40 2023-10-19T14:21:07.273 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.265+0000 7f5296ffd640 1 --2- 172.21.15.57:0/1940262969 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f527806b0b0 0x7f527806d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:07.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.265+0000 7f5296ffd640 1 --2- 172.21.15.57:0/1940262969 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f527806b0b0 0x7f527806d570 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5290007fd0 tx=0x7f5290009040 comp rx=0 tx=0).ready entity=mgr.9530 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:07.274 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.269+0000 7f528f7fe640 1 -- 172.21.15.57:0/1940262969 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f529812d520 con 0x7f52980b0c40 2023-10-19T14:21:07.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.444+0000 7f529d397640 1 -- 172.21.15.57:0/1940262969 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f529812d520 con 0x7f52980b0c40 2023-10-19T14:21:07.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.445+0000 7f528f7fe640 1 -- 172.21.15.57:0/1940262969 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v239) v1 ==== 93+0+10 (secure 0 0 0) 0x7f529812d520 con 0x7f52980b0c40 2023-10-19T14:21:07.448 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-19T14:21:07.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.446+0000 7f529d397640 1 -- 172.21.15.57:0/1940262969 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f527806b0b0 msgr2=0x7f527806d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:07.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.446+0000 7f529d397640 1 --2- 172.21.15.57:0/1940262969 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f527806b0b0 0x7f527806d570 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5290007fd0 tx=0x7f5290009040 comp rx=0 tx=0).stop 2023-10-19T14:21:07.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.446+0000 7f529d397640 1 -- 172.21.15.57:0/1940262969 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f52980b0c40 msgr2=0x7f52980ca870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:07.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.446+0000 7f529d397640 1 --2- 172.21.15.57:0/1940262969 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f52980b0c40 0x7f52980ca870 secure :-1 s=READY pgs=2604 cs=0 l=1 rev1=1 crypto rx=0x7f528000c750 tx=0x7f528000cc10 comp rx=0 tx=0).stop 2023-10-19T14:21:07.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.446+0000 7f529d397640 1 -- 172.21.15.57:0/1940262969 shutdown_connections 2023-10-19T14:21:07.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.446+0000 7f529d397640 1 --2- 172.21.15.57:0/1940262969 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f527806b0b0 0x7f527806d570 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:07.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.446+0000 7f529d397640 1 --2- 172.21.15.57:0/1940262969 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f52980cbd20 0x7f52980cadb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:07.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.446+0000 7f529d397640 1 --2- 172.21.15.57:0/1940262969 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f52980b0c40 0x7f52980ca870 unknown :-1 s=CLOSED pgs=2604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:07.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.446+0000 7f529d397640 1 --2- 172.21.15.57:0/1940262969 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f52980afa20 0x7f52981405a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:07.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.446+0000 7f529d397640 1 -- 172.21.15.57:0/1940262969 >> 172.21.15.57:0/1940262969 conn(0x7f5298069930 msgr2=0x7f5298069d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:07.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.447+0000 7f529d397640 1 -- 172.21.15.57:0/1940262969 shutdown_connections 2023-10-19T14:21:07.452 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.447+0000 7f529d397640 1 -- 172.21.15.57:0/1940262969 wait complete. 2023-10-19T14:21:07.460 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-19T14:21:07.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.567+0000 7f84a4dc7640 1 -- 172.21.15.57:0/2887012263 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f84a00c9dc0 msgr2=0x7f84a006b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:07.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.567+0000 7f84a4dc7640 1 --2- 172.21.15.57:0/2887012263 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f84a00c9dc0 0x7f84a006b2e0 secure :-1 s=READY pgs=2606 cs=0 l=1 rev1=1 crypto rx=0x7f8488009830 tx=0x7f848802bd60 comp rx=0 tx=0).stop 2023-10-19T14:21:07.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.567+0000 7f84a4dc7640 1 -- 172.21.15.57:0/2887012263 shutdown_connections 2023-10-19T14:21:07.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.567+0000 7f84a4dc7640 1 --2- 172.21.15.57:0/2887012263 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f84a00c9dc0 0x7f84a006b2e0 unknown :-1 s=CLOSED pgs=2606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:07.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.567+0000 7f84a4dc7640 1 -- 172.21.15.57:0/2887012263 >> 172.21.15.57:0/2887012263 conn(0x7f84a0069930 msgr2=0x7f84a0069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:07.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.567+0000 7f84a4dc7640 1 -- 172.21.15.57:0/2887012263 shutdown_connections 2023-10-19T14:21:07.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.567+0000 7f84a4dc7640 1 -- 172.21.15.57:0/2887012263 wait complete. 2023-10-19T14:21:07.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.568+0000 7f84a4dc7640 1 Processor -- start 2023-10-19T14:21:07.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.568+0000 7f84a4dc7640 1 -- start start 2023-10-19T14:21:07.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.569+0000 7f84a4dc7640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f84a006af00 0x7f84a013a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:07.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.569+0000 7f84a4dc7640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f84a00ab500 0x7f84a01522c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:07.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.569+0000 7f84a4dc7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f84a00c9dc0 0x7f84a01f0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:07.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.569+0000 7f84a4dc7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84a01512d0 con 0x7f84a00c9dc0 2023-10-19T14:21:07.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.569+0000 7f84a4dc7640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84a01f1400 con 0x7f84a006af00 2023-10-19T14:21:07.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.569+0000 7f84a4dc7640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84a013aea0 con 0x7f84a00ab500 2023-10-19T14:21:07.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.569+0000 7f849ed9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f84a00c9dc0 0x7f84a01f0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:07.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.569+0000 7f849ed9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f84a00c9dc0 0x7f84a01f0ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51482/0 (socket says 172.21.15.57:51482) 2023-10-19T14:21:07.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.569+0000 7f849ed9d640 1 -- 172.21.15.57:0/4028442571 learned_addr learned my addr 172.21.15.57:0/4028442571 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:07.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.569+0000 7f8495d9b640 1 --2- 172.21.15.57:0/4028442571 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f84a00ab500 0x7f84a01522c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:07.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.569+0000 7f849e59c640 1 --2- 172.21.15.57:0/4028442571 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f84a006af00 0x7f84a013a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:07.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.569+0000 7f849ed9d640 1 -- 172.21.15.57:0/4028442571 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f84a00ab500 msgr2=0x7f84a01522c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:07.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.569+0000 7f849ed9d640 1 --2- 172.21.15.57:0/4028442571 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f84a00ab500 0x7f84a01522c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:07.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.569+0000 7f849ed9d640 1 -- 172.21.15.57:0/4028442571 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f84a006af00 msgr2=0x7f84a013a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:07.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.569+0000 7f849ed9d640 1 --2- 172.21.15.57:0/4028442571 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f84a006af00 0x7f84a013a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:07.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.569+0000 7f849ed9d640 1 -- 172.21.15.57:0/4028442571 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8488009490 con 0x7f84a00c9dc0 2023-10-19T14:21:07.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.569+0000 7f849e59c640 1 --2- 172.21.15.57:0/4028442571 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f84a006af00 0x7f84a013a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:21:07.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.569+0000 7f8495d9b640 1 --2- 172.21.15.57:0/4028442571 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f84a00ab500 0x7f84a01522c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:21:07.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.569+0000 7f849ed9d640 1 --2- 172.21.15.57:0/4028442571 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f84a00c9dc0 0x7f84a01f0ec0 secure :-1 s=READY pgs=2607 cs=0 l=1 rev1=1 crypto rx=0x7f849800c780 tx=0x7f849800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:07.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.570+0000 7f84977fe640 1 -- 172.21.15.57:0/4028442571 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8498004b60 con 0x7f84a00c9dc0 2023-10-19T14:21:07.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.570+0000 7f84977fe640 1 -- 172.21.15.57:0/4028442571 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f849800ce90 con 0x7f84a00c9dc0 2023-10-19T14:21:07.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.570+0000 7f84a4dc7640 1 -- 172.21.15.57:0/4028442571 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84a0159a20 con 0x7f84a00c9dc0 2023-10-19T14:21:07.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.570+0000 7f84977fe640 1 -- 172.21.15.57:0/4028442571 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f849800a850 con 0x7f84a00c9dc0 2023-10-19T14:21:07.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.570+0000 7f84a4dc7640 1 -- 172.21.15.57:0/4028442571 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84a013d040 con 0x7f84a00c9dc0 2023-10-19T14:21:07.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.571+0000 7f84a4dc7640 1 -- 172.21.15.57:0/4028442571 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84a00690e0 con 0x7f84a00c9dc0 2023-10-19T14:21:07.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.571+0000 7f84977fe640 1 -- 172.21.15.57:0/4028442571 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f849800a9b0 con 0x7f84a00c9dc0 2023-10-19T14:21:07.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.571+0000 7f84977fe640 1 --2- 172.21.15.57:0/4028442571 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f848006b080 0x7f848006d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:07.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.572+0000 7f84977fe640 1 -- 172.21.15.57:0/4028442571 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(239..239 src has 1..239) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f8498089c80 con 0x7f84a00c9dc0 2023-10-19T14:21:07.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.572+0000 7f849e59c640 1 --2- 172.21.15.57:0/4028442571 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f848006b080 0x7f848006d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:07.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.572+0000 7f849e59c640 1 --2- 172.21.15.57:0/4028442571 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f848006b080 0x7f848006d540 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f8490000f60 tx=0x7f8490009040 comp rx=0 tx=0).ready entity=mgr.9530 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:07.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.576+0000 7f84977fe640 1 -- 172.21.15.57:0/4028442571 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8498054bc0 con 0x7f84a00c9dc0 2023-10-19T14:21:07.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.754+0000 7f84a4dc7640 1 -- 172.21.15.57:0/4028442571 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f84a00c9b70 con 0x7f84a00c9dc0 2023-10-19T14:21:07.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.754+0000 7f84977fe640 1 -- 172.21.15.57:0/4028442571 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v239) v1 ==== 84+0+65 (secure 0 0 0) 0x7f84980581e0 con 0x7f84a00c9dc0 2023-10-19T14:21:07.758 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:21:07.758 INFO:teuthology.orchestra.run.smithi057.stdout:[ 2023-10-19T14:21:07.758 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:21:07.758 INFO:teuthology.orchestra.run.smithi057.stdout: "poolnum": 1, 2023-10-19T14:21:07.759 INFO:teuthology.orchestra.run.smithi057.stdout: "poolname": ".mgr" 2023-10-19T14:21:07.759 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:21:07.759 INFO:teuthology.orchestra.run.smithi057.stdout:] 2023-10-19T14:21:07.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.757+0000 7f84a4dc7640 1 -- 172.21.15.57:0/4028442571 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f848006b080 msgr2=0x7f848006d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:07.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.757+0000 7f84a4dc7640 1 --2- 172.21.15.57:0/4028442571 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f848006b080 0x7f848006d540 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f8490000f60 tx=0x7f8490009040 comp rx=0 tx=0).stop 2023-10-19T14:21:07.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.757+0000 7f84a4dc7640 1 -- 172.21.15.57:0/4028442571 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f84a00c9dc0 msgr2=0x7f84a01f0ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:07.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.757+0000 7f84a4dc7640 1 --2- 172.21.15.57:0/4028442571 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f84a00c9dc0 0x7f84a01f0ec0 secure :-1 s=READY pgs=2607 cs=0 l=1 rev1=1 crypto rx=0x7f849800c780 tx=0x7f849800cc40 comp rx=0 tx=0).stop 2023-10-19T14:21:07.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.757+0000 7f84a4dc7640 1 -- 172.21.15.57:0/4028442571 shutdown_connections 2023-10-19T14:21:07.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.757+0000 7f84a4dc7640 1 --2- 172.21.15.57:0/4028442571 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f848006b080 0x7f848006d540 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:07.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.757+0000 7f84a4dc7640 1 --2- 172.21.15.57:0/4028442571 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f84a00c9dc0 0x7f84a01f0ec0 unknown :-1 s=CLOSED pgs=2607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:07.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.757+0000 7f84a4dc7640 1 --2- 172.21.15.57:0/4028442571 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f84a00ab500 0x7f84a01522c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:07.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.757+0000 7f84a4dc7640 1 --2- 172.21.15.57:0/4028442571 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f84a006af00 0x7f84a013a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:07.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.757+0000 7f84a4dc7640 1 -- 172.21.15.57:0/4028442571 >> 172.21.15.57:0/4028442571 conn(0x7f84a0069930 msgr2=0x7f84a00ac320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:07.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.757+0000 7f84a4dc7640 1 -- 172.21.15.57:0/4028442571 shutdown_connections 2023-10-19T14:21:07.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.757+0000 7f84a4dc7640 1 -- 172.21.15.57:0/4028442571 wait complete. 2023-10-19T14:21:07.770 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-19T14:21:07.771 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-19T14:21:07.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.881+0000 7f256c8c9640 1 -- 172.21.15.57:0/1713198462 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25680cf340 msgr2=0x7f25680d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:07.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.881+0000 7f256c8c9640 1 --2- 172.21.15.57:0/1713198462 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25680cf340 0x7f25680d1910 secure :-1 s=READY pgs=2608 cs=0 l=1 rev1=1 crypto rx=0x7f256815b680 tx=0x7f255802bd10 comp rx=0 tx=0).stop 2023-10-19T14:21:07.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.881+0000 7f256c8c9640 1 -- 172.21.15.57:0/1713198462 shutdown_connections 2023-10-19T14:21:07.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.881+0000 7f256c8c9640 1 --2- 172.21.15.57:0/1713198462 >> v2:172.21.15.179:3300/0 conn(0x7f25680ceb40 0x7f25680b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:07.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.881+0000 7f256c8c9640 1 --2- 172.21.15.57:0/1713198462 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25680cf340 0x7f25680d1910 unknown :-1 s=CLOSED pgs=2608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:07.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.881+0000 7f256c8c9640 1 -- 172.21.15.57:0/1713198462 >> 172.21.15.57:0/1713198462 conn(0x7f25680d0d60 msgr2=0x7f25680d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:07.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.881+0000 7f256c8c9640 1 -- 172.21.15.57:0/1713198462 shutdown_connections 2023-10-19T14:21:07.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.881+0000 7f256c8c9640 1 -- 172.21.15.57:0/1713198462 wait complete. 2023-10-19T14:21:07.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.883+0000 7f256c8c9640 1 Processor -- start 2023-10-19T14:21:07.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.883+0000 7f256c8c9640 1 -- start start 2023-10-19T14:21:07.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.883+0000 7f256c8c9640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f25680ceb40 0x7f25680cca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:07.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.883+0000 7f256c8c9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25680cf340 0x7f2568157280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:07.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.883+0000 7f256c8c9640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f256815b2a0 0x7f256815d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:07.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.883+0000 7f256c8c9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2568157c80 con 0x7f25680cf340 2023-10-19T14:21:07.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.883+0000 7f256c8c9640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2568157f60 con 0x7f256815b2a0 2023-10-19T14:21:07.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.883+0000 7f256c8c9640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2568158240 con 0x7f25680ceb40 2023-10-19T14:21:07.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.883+0000 7f2565d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25680cf340 0x7f2568157280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:07.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.884+0000 7f2565d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25680cf340 0x7f2568157280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51500/0 (socket says 172.21.15.57:51500) 2023-10-19T14:21:07.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.884+0000 7f2565d9b640 1 -- 172.21.15.57:0/2807818054 learned_addr learned my addr 172.21.15.57:0/2807818054 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:07.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.884+0000 7f256659c640 1 --2- 172.21.15.57:0/2807818054 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f25680ceb40 0x7f25680cca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:07.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.884+0000 7f2566d9d640 1 --2- 172.21.15.57:0/2807818054 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f256815b2a0 0x7f256815d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:07.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.884+0000 7f2565d9b640 1 -- 172.21.15.57:0/2807818054 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f25680ceb40 msgr2=0x7f25680cca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:07.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.884+0000 7f2565d9b640 1 --2- 172.21.15.57:0/2807818054 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f25680ceb40 0x7f25680cca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:07.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.884+0000 7f2565d9b640 1 -- 172.21.15.57:0/2807818054 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f256815b2a0 msgr2=0x7f256815d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:07.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.884+0000 7f2565d9b640 1 --2- 172.21.15.57:0/2807818054 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f256815b2a0 0x7f256815d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:07.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.884+0000 7f2565d9b640 1 -- 172.21.15.57:0/2807818054 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2558009560 con 0x7f25680cf340 2023-10-19T14:21:07.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.884+0000 7f2565d9b640 1 --2- 172.21.15.57:0/2807818054 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25680cf340 0x7f2568157280 secure :-1 s=READY pgs=2609 cs=0 l=1 rev1=1 crypto rx=0x7f255c00b3f0 tx=0x7f255c00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:07.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.884+0000 7f2556ffd640 1 -- 172.21.15.57:0/2807818054 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f255c011830 con 0x7f25680cf340 2023-10-19T14:21:07.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.884+0000 7f2556ffd640 1 -- 172.21.15.57:0/2807818054 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f255c011a80 con 0x7f25680cf340 2023-10-19T14:21:07.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.884+0000 7f256c8c9640 1 -- 172.21.15.57:0/2807818054 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2568158520 con 0x7f25680cf340 2023-10-19T14:21:07.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.884+0000 7f2556ffd640 1 -- 172.21.15.57:0/2807818054 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f255c011e90 con 0x7f25680cf340 2023-10-19T14:21:07.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.884+0000 7f256c8c9640 1 -- 172.21.15.57:0/2807818054 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25680ccf40 con 0x7f25680cf340 2023-10-19T14:21:07.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.885+0000 7f256c8c9640 1 -- 172.21.15.57:0/2807818054 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25681e3b50 con 0x7f25680cf340 2023-10-19T14:21:07.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.886+0000 7f2556ffd640 1 -- 172.21.15.57:0/2807818054 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f255c01a400 con 0x7f25680cf340 2023-10-19T14:21:07.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.886+0000 7f2556ffd640 1 --2- 172.21.15.57:0/2807818054 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f254806b080 0x7f254806d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:07.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.886+0000 7f2556ffd640 1 -- 172.21.15.57:0/2807818054 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(239..239 src has 1..239) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f255c089730 con 0x7f25680cf340 2023-10-19T14:21:07.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.886+0000 7f256659c640 1 --2- 172.21.15.57:0/2807818054 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f254806b080 0x7f254806d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:07.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.887+0000 7f256659c640 1 --2- 172.21.15.57:0/2807818054 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f254806b080 0x7f254806d540 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2558035000 tx=0x7f2558004730 comp rx=0 tx=0).ready entity=mgr.9530 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:07.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:07.890+0000 7f2556ffd640 1 -- 172.21.15.57:0/2807818054 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f25681e3b50 con 0x7f25680cf340 2023-10-19T14:21:08.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:08.066+0000 7f256c8c9640 1 -- 172.21.15.57:0/2807818054 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f25681e3b50 con 0x7f25680cf340 2023-10-19T14:21:08.391 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:08.388+0000 7f2556ffd640 1 -- 172.21.15.57:0/2807818054 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v240) v1 ==== 138+0+0 (secure 0 0 0) 0x7f25681e3b50 con 0x7f25680cf340 2023-10-19T14:21:08.391 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'cephfs_metadata' created 2023-10-19T14:21:08.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:08.390+0000 7f256c8c9640 1 -- 172.21.15.57:0/2807818054 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f254806b080 msgr2=0x7f254806d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:08.394 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:08.390+0000 7f256c8c9640 1 --2- 172.21.15.57:0/2807818054 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f254806b080 0x7f254806d540 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2558035000 tx=0x7f2558004730 comp rx=0 tx=0).stop 2023-10-19T14:21:08.394 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:08.390+0000 7f256c8c9640 1 -- 172.21.15.57:0/2807818054 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25680cf340 msgr2=0x7f2568157280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:08.394 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:08.390+0000 7f256c8c9640 1 --2- 172.21.15.57:0/2807818054 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25680cf340 0x7f2568157280 secure :-1 s=READY pgs=2609 cs=0 l=1 rev1=1 crypto rx=0x7f255c00b3f0 tx=0x7f255c00b8b0 comp rx=0 tx=0).stop 2023-10-19T14:21:08.394 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:08.390+0000 7f256c8c9640 1 -- 172.21.15.57:0/2807818054 shutdown_connections 2023-10-19T14:21:08.394 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:08.390+0000 7f256c8c9640 1 --2- 172.21.15.57:0/2807818054 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f254806b080 0x7f254806d540 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:08.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:08.390+0000 7f256c8c9640 1 --2- 172.21.15.57:0/2807818054 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f256815b2a0 0x7f256815d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:08.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:08.390+0000 7f256c8c9640 1 --2- 172.21.15.57:0/2807818054 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25680cf340 0x7f2568157280 unknown :-1 s=CLOSED pgs=2609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:08.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:08.390+0000 7f256c8c9640 1 --2- 172.21.15.57:0/2807818054 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f25680ceb40 0x7f25680cca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:08.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:08.390+0000 7f256c8c9640 1 -- 172.21.15.57:0/2807818054 >> 172.21.15.57:0/2807818054 conn(0x7f25680d0d60 msgr2=0x7f256815a3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:08.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:08.390+0000 7f256c8c9640 1 -- 172.21.15.57:0/2807818054 shutdown_connections 2023-10-19T14:21:08.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:08.390+0000 7f256c8c9640 1 -- 172.21.15.57:0/2807818054 wait complete. 2023-10-19T14:21:08.409 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-19T14:21:08.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:08.525+0000 7fb505f73640 1 -- 172.21.15.57:0/800394109 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb50006af00 msgr2=0x7fb50006b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:08.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:08.525+0000 7fb505f73640 1 --2- 172.21.15.57:0/800394109 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb50006af00 0x7fb50006b2e0 secure :-1 s=READY pgs=2610 cs=0 l=1 rev1=1 crypto rx=0x7fb4f0009ca0 tx=0x7fb4f002c2e0 comp rx=0 tx=0).stop 2023-10-19T14:21:08.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:08.525+0000 7fb505f73640 1 -- 172.21.15.57:0/800394109 shutdown_connections 2023-10-19T14:21:08.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:08.525+0000 7fb505f73640 1 --2- 172.21.15.57:0/800394109 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb5000abc10 0x7fb5000c9b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:08.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:08.525+0000 7fb505f73640 1 --2- 172.21.15.57:0/800394109 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb50006af00 0x7fb50006b2e0 unknown :-1 s=CLOSED pgs=2610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:08.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:08.525+0000 7fb505f73640 1 -- 172.21.15.57:0/800394109 >> 172.21.15.57:0/800394109 conn(0x7fb500069930 msgr2=0x7fb500069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:08.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:08.525+0000 7fb505f73640 1 -- 172.21.15.57:0/800394109 shutdown_connections 2023-10-19T14:21:08.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:08.525+0000 7fb505f73640 1 -- 172.21.15.57:0/800394109 wait complete. 2023-10-19T14:21:08.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:08.526+0000 7fb505f73640 1 Processor -- start 2023-10-19T14:21:08.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:08.526+0000 7fb505f73640 1 -- start start 2023-10-19T14:21:08.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:08.527+0000 7fb505f73640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb50006af00 0x7fb5000af750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:08.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:08.527+0000 7fb505f73640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb5000abc10 0x7fb5001f11e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:08.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:08.527+0000 7fb505f73640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb5000c96b0 0x7fb500070930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:08.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:08.527+0000 7fb505f73640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb500137f20 con 0x7fb50006af00 2023-10-19T14:21:08.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:08.527+0000 7fb505f73640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb500138200 con 0x7fb5000c96b0 2023-10-19T14:21:08.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:08.527+0000 7fb505f73640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5001384e0 con 0x7fb5000abc10 2023-10-19T14:21:08.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:08.527+0000 7fb4ff7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb50006af00 0x7fb5000af750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:08.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:08.527+0000 7fb4ff7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb50006af00 0x7fb5000af750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51516/0 (socket says 172.21.15.57:51516) 2023-10-19T14:21:08.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:08.527+0000 7fb4ff7fe640 1 -- 172.21.15.57:0/2319968709 learned_addr learned my addr 172.21.15.57:0/2319968709 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:08.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:08.527+0000 7fb4fffff640 1 --2- 172.21.15.57:0/2319968709 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb5000c96b0 0x7fb500070930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:08.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:08.527+0000 7fb4feffd640 1 --2- 172.21.15.57:0/2319968709 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb5000abc10 0x7fb5001f11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:08.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:08.527+0000 7fb4ff7fe640 1 -- 172.21.15.57:0/2319968709 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb5000abc10 msgr2=0x7fb5001f11e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:08.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:08.527+0000 7fb4ff7fe640 1 --2- 172.21.15.57:0/2319968709 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb5000abc10 0x7fb5001f11e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:08.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:08.527+0000 7fb4ff7fe640 1 -- 172.21.15.57:0/2319968709 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb5000c96b0 msgr2=0x7fb500070930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:08.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:08.527+0000 7fb4ff7fe640 1 --2- 172.21.15.57:0/2319968709 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb5000c96b0 0x7fb500070930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:08.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:08.527+0000 7fb4ff7fe640 1 -- 172.21.15.57:0/2319968709 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4f0009900 con 0x7fb50006af00 2023-10-19T14:21:08.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:08.527+0000 7fb4fffff640 1 --2- 172.21.15.57:0/2319968709 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb5000c96b0 0x7fb500070930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:21:08.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:08.527+0000 7fb4feffd640 1 --2- 172.21.15.57:0/2319968709 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb5000abc10 0x7fb5001f11e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:21:08.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:08.528+0000 7fb4ff7fe640 1 --2- 172.21.15.57:0/2319968709 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb50006af00 0x7fb5000af750 secure :-1 s=READY pgs=2611 cs=0 l=1 rev1=1 crypto rx=0x7fb4f0037040 tx=0x7fb4f0035b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:08.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:08.528+0000 7fb4f7fff640 1 -- 172.21.15.57:0/2319968709 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4f0004240 con 0x7fb50006af00 2023-10-19T14:21:08.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:08.528+0000 7fb4f7fff640 1 -- 172.21.15.57:0/2319968709 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fb4f0004490 con 0x7fb50006af00 2023-10-19T14:21:08.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:08.528+0000 7fb505f73640 1 -- 172.21.15.57:0/2319968709 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5001387c0 con 0x7fb50006af00 2023-10-19T14:21:08.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:08.528+0000 7fb4f7fff640 1 -- 172.21.15.57:0/2319968709 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4f0004860 con 0x7fb50006af00 2023-10-19T14:21:08.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:08.528+0000 7fb505f73640 1 -- 172.21.15.57:0/2319968709 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5001f56e0 con 0x7fb50006af00 2023-10-19T14:21:08.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:08.529+0000 7fb505f73640 1 -- 172.21.15.57:0/2319968709 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5000690e0 con 0x7fb50006af00 2023-10-19T14:21:08.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:08.529+0000 7fb4f7fff640 1 -- 172.21.15.57:0/2319968709 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fb4f003b040 con 0x7fb50006af00 2023-10-19T14:21:08.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:08.530+0000 7fb4f7fff640 1 --2- 172.21.15.57:0/2319968709 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fb4e006b080 0x7fb4e006d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:08.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:08.530+0000 7fb4f7fff640 1 -- 172.21.15.57:0/2319968709 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(240..240 src has 1..240) v4 ==== 5575+0+0 (secure 0 0 0) 0x7fb4f00aca50 con 0x7fb50006af00 2023-10-19T14:21:08.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:08.530+0000 7fb4feffd640 1 --2- 172.21.15.57:0/2319968709 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fb4e006b080 0x7fb4e006d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:08.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:08.531+0000 7fb4feffd640 1 --2- 172.21.15.57:0/2319968709 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fb4e006b080 0x7fb4e006d540 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb4e8007fd0 tx=0x7fb4e8009040 comp rx=0 tx=0).ready entity=mgr.9530 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:08.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:08.534+0000 7fb4f7fff640 1 -- 172.21.15.57:0/2319968709 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb4f0077c60 con 0x7fb50006af00 2023-10-19T14:21:08.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:08.711+0000 7fb505f73640 1 -- 172.21.15.57:0/2319968709 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fb500138060 con 0x7fb50006af00 2023-10-19T14:21:09.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:09.390+0000 7fb4f7fff640 1 -- 172.21.15.57:0/2319968709 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v241) v1 ==== 170+0+0 (secure 0 0 0) 0x7fb500138060 con 0x7fb50006af00 2023-10-19T14:21:09.393 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'cephfs_data' created 2023-10-19T14:21:09.397 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:09.393+0000 7fb505f73640 1 -- 172.21.15.57:0/2319968709 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fb4e006b080 msgr2=0x7fb4e006d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:09.397 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:09.393+0000 7fb505f73640 1 --2- 172.21.15.57:0/2319968709 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fb4e006b080 0x7fb4e006d540 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb4e8007fd0 tx=0x7fb4e8009040 comp rx=0 tx=0).stop 2023-10-19T14:21:09.397 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:09.393+0000 7fb505f73640 1 -- 172.21.15.57:0/2319968709 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb50006af00 msgr2=0x7fb5000af750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:09.398 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:09.393+0000 7fb505f73640 1 --2- 172.21.15.57:0/2319968709 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb50006af00 0x7fb5000af750 secure :-1 s=READY pgs=2611 cs=0 l=1 rev1=1 crypto rx=0x7fb4f0037040 tx=0x7fb4f0035b50 comp rx=0 tx=0).stop 2023-10-19T14:21:09.398 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:09.394+0000 7fb505f73640 1 -- 172.21.15.57:0/2319968709 shutdown_connections 2023-10-19T14:21:09.398 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:09.394+0000 7fb505f73640 1 --2- 172.21.15.57:0/2319968709 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fb4e006b080 0x7fb4e006d540 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:09.398 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:09.394+0000 7fb505f73640 1 --2- 172.21.15.57:0/2319968709 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb5000c96b0 0x7fb500070930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:09.399 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:09.394+0000 7fb505f73640 1 --2- 172.21.15.57:0/2319968709 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb5000abc10 0x7fb5001f11e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:09.399 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:09.394+0000 7fb505f73640 1 --2- 172.21.15.57:0/2319968709 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb50006af00 0x7fb5000af750 unknown :-1 s=CLOSED pgs=2611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:09.399 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:09.394+0000 7fb505f73640 1 -- 172.21.15.57:0/2319968709 >> 172.21.15.57:0/2319968709 conn(0x7fb500069930 msgr2=0x7fb5000cafc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:09.399 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:09.394+0000 7fb505f73640 1 -- 172.21.15.57:0/2319968709 shutdown_connections 2023-10-19T14:21:09.400 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:09.395+0000 7fb505f73640 1 -- 172.21.15.57:0/2319968709 wait complete. 2023-10-19T14:21:09.422 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-19T14:21:09.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:09.536+0000 7f776744b640 1 -- 172.21.15.57:0/1837936812 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f77600ce420 msgr2=0x7f77600cd6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:09.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:09.536+0000 7f776744b640 1 --2- 172.21.15.57:0/1837936812 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f77600ce420 0x7f77600cd6b0 secure :-1 s=READY pgs=2612 cs=0 l=1 rev1=1 crypto rx=0x7f7760089890 tx=0x7f775402be20 comp rx=0 tx=0).stop 2023-10-19T14:21:09.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:09.536+0000 7f776744b640 1 -- 172.21.15.57:0/1837936812 shutdown_connections 2023-10-19T14:21:09.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:09.536+0000 7f776744b640 1 --2- 172.21.15.57:0/1837936812 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f77600cdbf0 0x7f77600cb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:09.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:09.536+0000 7f776744b640 1 --2- 172.21.15.57:0/1837936812 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f77600ce420 0x7f77600cd6b0 unknown :-1 s=CLOSED pgs=2612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:09.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:09.536+0000 7f776744b640 1 --2- 172.21.15.57:0/1837936812 >> v2:172.21.15.179:3300/0 conn(0x7f776015b180 0x7f77600cf350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:09.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:09.536+0000 7f776744b640 1 -- 172.21.15.57:0/1837936812 >> 172.21.15.57:0/1837936812 conn(0x7f7760099ff0 msgr2=0x7f77600d0d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:09.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:09.536+0000 7f776744b640 1 -- 172.21.15.57:0/1837936812 shutdown_connections 2023-10-19T14:21:09.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:09.536+0000 7f776744b640 1 -- 172.21.15.57:0/1837936812 wait complete. 2023-10-19T14:21:09.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:09.538+0000 7f776744b640 1 Processor -- start 2023-10-19T14:21:09.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:09.538+0000 7f776744b640 1 -- start start 2023-10-19T14:21:09.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:09.538+0000 7f776744b640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f77600cdbf0 0x7f7760158a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:09.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:09.538+0000 7f776744b640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f77600ce420 0x7f776006b110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:09.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:09.538+0000 7f776744b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f776015b180 0x7f776006b650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:09.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:09.538+0000 7f776744b640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77600c8f30 con 0x7f776015b180 2023-10-19T14:21:09.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:09.538+0000 7f776744b640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77600c9210 con 0x7f77600cdbf0 2023-10-19T14:21:09.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:09.538+0000 7f776744b640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77600c94f0 con 0x7f77600ce420 2023-10-19T14:21:09.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:09.539+0000 7f77659e8640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f776015b180 0x7f776006b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:09.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:09.539+0000 7f77659e8640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f776015b180 0x7f776006b650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51542/0 (socket says 172.21.15.57:51542) 2023-10-19T14:21:09.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:09.539+0000 7f77659e8640 1 -- 172.21.15.57:0/577818524 learned_addr learned my addr 172.21.15.57:0/577818524 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:09.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:09.539+0000 7f77651e7640 1 --2- 172.21.15.57:0/577818524 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f77600cdbf0 0x7f7760158a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:09.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:09.539+0000 7f77649e6640 1 --2- 172.21.15.57:0/577818524 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f77600ce420 0x7f776006b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:09.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:09.539+0000 7f77659e8640 1 -- 172.21.15.57:0/577818524 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f77600ce420 msgr2=0x7f776006b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:09.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:09.539+0000 7f77659e8640 1 --2- 172.21.15.57:0/577818524 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f77600ce420 0x7f776006b110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:09.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:09.539+0000 7f77659e8640 1 -- 172.21.15.57:0/577818524 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f77600cdbf0 msgr2=0x7f7760158a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:09.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:09.539+0000 7f77659e8640 1 --2- 172.21.15.57:0/577818524 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f77600cdbf0 0x7f7760158a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:09.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:09.539+0000 7f77659e8640 1 -- 172.21.15.57:0/577818524 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7754009560 con 0x7f776015b180 2023-10-19T14:21:09.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:09.539+0000 7f77659e8640 1 --2- 172.21.15.57:0/577818524 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f776015b180 0x7f776006b650 secure :-1 s=READY pgs=2613 cs=0 l=1 rev1=1 crypto rx=0x7f775000e980 tx=0x7f775000ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:09.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:09.539+0000 7f775dffb640 1 -- 172.21.15.57:0/577818524 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7750004d20 con 0x7f776015b180 2023-10-19T14:21:09.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:09.539+0000 7f775dffb640 1 -- 172.21.15.57:0/577818524 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f775000cb20 con 0x7f776015b180 2023-10-19T14:21:09.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:09.539+0000 7f776744b640 1 -- 172.21.15.57:0/577818524 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77600c97d0 con 0x7f776015b180 2023-10-19T14:21:09.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:09.539+0000 7f775dffb640 1 -- 172.21.15.57:0/577818524 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7750005020 con 0x7f776015b180 2023-10-19T14:21:09.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:09.539+0000 7f776744b640 1 -- 172.21.15.57:0/577818524 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77600c57b0 con 0x7f776015b180 2023-10-19T14:21:09.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:09.540+0000 7f776744b640 1 -- 172.21.15.57:0/577818524 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7760066960 con 0x7f776015b180 2023-10-19T14:21:09.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:09.541+0000 7f775dffb640 1 -- 172.21.15.57:0/577818524 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f7750005180 con 0x7f776015b180 2023-10-19T14:21:09.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:09.541+0000 7f775dffb640 1 --2- 172.21.15.57:0/577818524 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f774c06b080 0x7f774c06d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:09.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:09.541+0000 7f775dffb640 1 -- 172.21.15.57:0/577818524 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(241..241 src has 1..241) v4 ==== 5957+0+0 (secure 0 0 0) 0x7f775008a7a0 con 0x7f776015b180 2023-10-19T14:21:09.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:09.542+0000 7f77651e7640 1 --2- 172.21.15.57:0/577818524 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f774c06b080 0x7f774c06d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:09.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:09.542+0000 7f77651e7640 1 --2- 172.21.15.57:0/577818524 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f774c06b080 0x7f774c06d540 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f775800bb50 tx=0x7f775800b2e0 comp rx=0 tx=0).ready entity=mgr.9530 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:09.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:09.546+0000 7f775dffb640 1 -- 172.21.15.57:0/577818524 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7750055420 con 0x7f776015b180 2023-10-19T14:21:09.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:09.758+0000 7f776744b640 1 -- 172.21.15.57:0/577818524 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f77600ce1e0 con 0x7f776015b180 2023-10-19T14:21:10.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.420+0000 7f775dffb640 1 -- 172.21.15.57:0/577818524 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 31 and data pool 32 v101) v1 ==== 174+0+0 (secure 0 0 0) 0x7f775001f0c0 con 0x7f776015b180 2023-10-19T14:21:10.424 INFO:teuthology.orchestra.run.smithi057.stderr:new fs with metadata pool 31 and data pool 32 2023-10-19T14:21:10.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.431+0000 7f776744b640 1 -- 172.21.15.57:0/577818524 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f774c06b080 msgr2=0x7f774c06d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:10.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.431+0000 7f776744b640 1 --2- 172.21.15.57:0/577818524 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f774c06b080 0x7f774c06d540 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f775800bb50 tx=0x7f775800b2e0 comp rx=0 tx=0).stop 2023-10-19T14:21:10.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.431+0000 7f776744b640 1 -- 172.21.15.57:0/577818524 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f776015b180 msgr2=0x7f776006b650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:10.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.431+0000 7f776744b640 1 --2- 172.21.15.57:0/577818524 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f776015b180 0x7f776006b650 secure :-1 s=READY pgs=2613 cs=0 l=1 rev1=1 crypto rx=0x7f775000e980 tx=0x7f775000ee40 comp rx=0 tx=0).stop 2023-10-19T14:21:10.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.432+0000 7f776744b640 1 -- 172.21.15.57:0/577818524 shutdown_connections 2023-10-19T14:21:10.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.432+0000 7f776744b640 1 --2- 172.21.15.57:0/577818524 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f774c06b080 0x7f774c06d540 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:10.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.432+0000 7f776744b640 1 --2- 172.21.15.57:0/577818524 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f776015b180 0x7f776006b650 unknown :-1 s=CLOSED pgs=2613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:10.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.432+0000 7f776744b640 1 --2- 172.21.15.57:0/577818524 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f77600ce420 0x7f776006b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:10.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.432+0000 7f776744b640 1 --2- 172.21.15.57:0/577818524 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f77600cdbf0 0x7f7760158a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:10.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.432+0000 7f776744b640 1 -- 172.21.15.57:0/577818524 >> 172.21.15.57:0/577818524 conn(0x7f7760099ff0 msgr2=0x7f776015c240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:10.439 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.437+0000 7f776744b640 1 -- 172.21.15.57:0/577818524 shutdown_connections 2023-10-19T14:21:10.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.437+0000 7f776744b640 1 -- 172.21.15.57:0/577818524 wait complete. 2023-10-19T14:21:10.462 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T14:21:10.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.573+0000 7f2b64026640 1 -- 172.21.15.57:0/1157056117 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2b5c0cf340 msgr2=0x7f2b5c0d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:10.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.573+0000 7f2b64026640 1 --2- 172.21.15.57:0/1157056117 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2b5c0cf340 0x7f2b5c0d1910 secure :-1 s=READY pgs=2614 cs=0 l=1 rev1=1 crypto rx=0x7f2b58005fd0 tx=0x7f2b5802bcf0 comp rx=0 tx=0).stop 2023-10-19T14:21:10.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.573+0000 7f2b64026640 1 -- 172.21.15.57:0/1157056117 shutdown_connections 2023-10-19T14:21:10.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.573+0000 7f2b64026640 1 --2- 172.21.15.57:0/1157056117 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2b5c0ceb40 0x7f2b5c0b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:10.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.573+0000 7f2b64026640 1 --2- 172.21.15.57:0/1157056117 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2b5c0cf340 0x7f2b5c0d1910 unknown :-1 s=CLOSED pgs=2614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:10.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.573+0000 7f2b64026640 1 -- 172.21.15.57:0/1157056117 >> 172.21.15.57:0/1157056117 conn(0x7f2b5c0d0d60 msgr2=0x7f2b5c0d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:10.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.573+0000 7f2b64026640 1 -- 172.21.15.57:0/1157056117 shutdown_connections 2023-10-19T14:21:10.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.573+0000 7f2b64026640 1 -- 172.21.15.57:0/1157056117 wait complete. 2023-10-19T14:21:10.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.575+0000 7f2b64026640 1 Processor -- start 2023-10-19T14:21:10.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.575+0000 7f2b64026640 1 -- start start 2023-10-19T14:21:10.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.575+0000 7f2b64026640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2b5c0ceb40 0x7f2b5c1e5d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:10.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.575+0000 7f2b64026640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2b5c0cf340 0x7f2b5c1e4930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:10.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.575+0000 7f2b64026640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2b5c15b2a0 0x7f2b5c1e4e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:10.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.575+0000 7f2b64026640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b5c0caef0 con 0x7f2b5c0cf340 2023-10-19T14:21:10.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.575+0000 7f2b64026640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b5c0cac10 con 0x7f2b5c15b2a0 2023-10-19T14:21:10.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.575+0000 7f2b64026640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b5c15b990 con 0x7f2b5c0ceb40 2023-10-19T14:21:10.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.575+0000 7f2b615c1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2b5c0cf340 0x7f2b5c1e4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:10.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.576+0000 7f2b61dc2640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2b5c0ceb40 0x7f2b5c1e5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:10.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.576+0000 7f2b615c1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2b5c0cf340 0x7f2b5c1e4930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51554/0 (socket says 172.21.15.57:51554) 2023-10-19T14:21:10.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.576+0000 7f2b615c1640 1 -- 172.21.15.57:0/2133381312 learned_addr learned my addr 172.21.15.57:0/2133381312 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:10.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.576+0000 7f2b625c3640 1 --2- 172.21.15.57:0/2133381312 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2b5c15b2a0 0x7f2b5c1e4e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:10.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.576+0000 7f2b615c1640 1 -- 172.21.15.57:0/2133381312 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2b5c0ceb40 msgr2=0x7f2b5c1e5d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:10.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.576+0000 7f2b615c1640 1 --2- 172.21.15.57:0/2133381312 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2b5c0ceb40 0x7f2b5c1e5d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:10.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.576+0000 7f2b615c1640 1 -- 172.21.15.57:0/2133381312 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2b5c15b2a0 msgr2=0x7f2b5c1e4e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:10.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.576+0000 7f2b615c1640 1 --2- 172.21.15.57:0/2133381312 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2b5c15b2a0 0x7f2b5c1e4e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:10.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.576+0000 7f2b615c1640 1 -- 172.21.15.57:0/2133381312 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b58009560 con 0x7f2b5c0cf340 2023-10-19T14:21:10.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.576+0000 7f2b615c1640 1 --2- 172.21.15.57:0/2133381312 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2b5c0cf340 0x7f2b5c1e4930 secure :-1 s=READY pgs=2615 cs=0 l=1 rev1=1 crypto rx=0x7f2b4c00b3f0 tx=0x7f2b4c00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:10.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.576+0000 7f2b527fc640 1 -- 172.21.15.57:0/2133381312 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b4c011830 con 0x7f2b5c0cf340 2023-10-19T14:21:10.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.576+0000 7f2b527fc640 1 -- 172.21.15.57:0/2133381312 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f2b4c011a80 con 0x7f2b5c0cf340 2023-10-19T14:21:10.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.576+0000 7f2b64026640 1 -- 172.21.15.57:0/2133381312 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b5c0b0d50 con 0x7f2b5c0cf340 2023-10-19T14:21:10.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.576+0000 7f2b527fc640 1 -- 172.21.15.57:0/2133381312 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b4c011e90 con 0x7f2b5c0cf340 2023-10-19T14:21:10.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.576+0000 7f2b64026640 1 -- 172.21.15.57:0/2133381312 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b5c15bf50 con 0x7f2b5c0cf340 2023-10-19T14:21:10.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.577+0000 7f2b64026640 1 -- 172.21.15.57:0/2133381312 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b5c15b7f0 con 0x7f2b5c0cf340 2023-10-19T14:21:10.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.578+0000 7f2b527fc640 1 -- 172.21.15.57:0/2133381312 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f2b4c01a400 con 0x7f2b5c0cf340 2023-10-19T14:21:10.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.578+0000 7f2b527fc640 1 --2- 172.21.15.57:0/2133381312 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f2b4006b080 0x7f2b4006d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:10.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.578+0000 7f2b527fc640 1 -- 172.21.15.57:0/2133381312 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(242..242 src has 1..242) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f2b4c089e70 con 0x7f2b5c0cf340 2023-10-19T14:21:10.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.578+0000 7f2b61dc2640 1 --2- 172.21.15.57:0/2133381312 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f2b4006b080 0x7f2b4006d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:10.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.579+0000 7f2b61dc2640 1 --2- 172.21.15.57:0/2133381312 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f2b4006b080 0x7f2b4006d540 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f2b58035000 tx=0x7f2b58004730 comp rx=0 tx=0).ready entity=mgr.9530 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:10.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.582+0000 7f2b527fc640 1 -- 172.21.15.57:0/2133381312 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2b5c15b7f0 con 0x7f2b5c0cf340 2023-10-19T14:21:10.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.761+0000 7f2b64026640 1 -- 172.21.15.57:0/2133381312 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2b5c15b7f0 con 0x7f2b5c0cf340 2023-10-19T14:21:10.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.762+0000 7f2b527fc640 1 -- 172.21.15.57:0/2133381312 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v242) v1 ==== 74+0+16852 (secure 0 0 0) 0x7f2b5c15b7f0 con 0x7f2b5c0cf340 2023-10-19T14:21:10.766 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:21:10.766 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":242,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T14:21:09.759627+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":32,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2023-10-19T14:21:08.101023+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"242","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2023-10-19T14:21:08.745817+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"242","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":241,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":241,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":241,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":241,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":241,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":241,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":241,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":241,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T14:21:10.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.764+0000 7f2b64026640 1 -- 172.21.15.57:0/2133381312 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f2b4006b080 msgr2=0x7f2b4006d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:10.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.764+0000 7f2b64026640 1 --2- 172.21.15.57:0/2133381312 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f2b4006b080 0x7f2b4006d540 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f2b58035000 tx=0x7f2b58004730 comp rx=0 tx=0).stop 2023-10-19T14:21:10.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.764+0000 7f2b64026640 1 -- 172.21.15.57:0/2133381312 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2b5c0cf340 msgr2=0x7f2b5c1e4930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:10.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.764+0000 7f2b64026640 1 --2- 172.21.15.57:0/2133381312 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2b5c0cf340 0x7f2b5c1e4930 secure :-1 s=READY pgs=2615 cs=0 l=1 rev1=1 crypto rx=0x7f2b4c00b3f0 tx=0x7f2b4c00b8b0 comp rx=0 tx=0).stop 2023-10-19T14:21:10.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.764+0000 7f2b64026640 1 -- 172.21.15.57:0/2133381312 shutdown_connections 2023-10-19T14:21:10.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.764+0000 7f2b64026640 1 --2- 172.21.15.57:0/2133381312 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f2b4006b080 0x7f2b4006d540 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:10.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.764+0000 7f2b64026640 1 --2- 172.21.15.57:0/2133381312 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2b5c15b2a0 0x7f2b5c1e4e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:10.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.764+0000 7f2b64026640 1 --2- 172.21.15.57:0/2133381312 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2b5c0cf340 0x7f2b5c1e4930 unknown :-1 s=CLOSED pgs=2615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:10.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.764+0000 7f2b64026640 1 --2- 172.21.15.57:0/2133381312 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2b5c0ceb40 0x7f2b5c1e5d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:10.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.764+0000 7f2b64026640 1 -- 172.21.15.57:0/2133381312 >> 172.21.15.57:0/2133381312 conn(0x7f2b5c0d0d60 msgr2=0x7f2b5c0ad8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:10.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.765+0000 7f2b64026640 1 -- 172.21.15.57:0/2133381312 shutdown_connections 2023-10-19T14:21:10.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.765+0000 7f2b64026640 1 -- 172.21.15.57:0/2133381312 wait complete. 2023-10-19T14:21:10.778 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T14:21:10.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.885+0000 7f3da7355640 1 -- 172.21.15.57:0/81755689 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3da00cb980 msgr2=0x7f3da013db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:10.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.885+0000 7f3da7355640 1 --2- 172.21.15.57:0/81755689 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3da00cb980 0x7f3da013db80 secure :-1 s=READY pgs=2616 cs=0 l=1 rev1=1 crypto rx=0x7f3da006fa40 tx=0x7f3d9802bda0 comp rx=0 tx=0).stop 2023-10-19T14:21:10.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.885+0000 7f3da7355640 1 -- 172.21.15.57:0/81755689 shutdown_connections 2023-10-19T14:21:10.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.885+0000 7f3da7355640 1 --2- 172.21.15.57:0/81755689 >> v2:172.21.15.179:3300/0 conn(0x7f3da00c96b0 0x7f3da00c9b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:10.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.885+0000 7f3da7355640 1 --2- 172.21.15.57:0/81755689 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3da013d350 0x7f3da00cbd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:10.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.885+0000 7f3da7355640 1 --2- 172.21.15.57:0/81755689 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3da00cb980 0x7f3da013db80 unknown :-1 s=CLOSED pgs=2616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:10.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.885+0000 7f3da7355640 1 -- 172.21.15.57:0/81755689 >> 172.21.15.57:0/81755689 conn(0x7f3da0069930 msgr2=0x7f3da0069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:10.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.885+0000 7f3da7355640 1 -- 172.21.15.57:0/81755689 shutdown_connections 2023-10-19T14:21:10.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.885+0000 7f3da7355640 1 -- 172.21.15.57:0/81755689 wait complete. 2023-10-19T14:21:10.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.886+0000 7f3da7355640 1 Processor -- start 2023-10-19T14:21:10.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.886+0000 7f3da7355640 1 -- start start 2023-10-19T14:21:10.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.887+0000 7f3da7355640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3da00c96b0 0x7f3da0139640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:10.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.887+0000 7f3da7355640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3da00cb980 0x7f3da0136110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:10.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.887+0000 7f3da7355640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3da0136650 0x7f3da0150440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:10.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.887+0000 7f3da7355640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3da0151b70 con 0x7f3da00cb980 2023-10-19T14:21:10.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.887+0000 7f3da7355640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3da0136ac0 con 0x7f3da0136650 2023-10-19T14:21:10.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.887+0000 7f3da7355640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3da014f300 con 0x7f3da00c96b0 2023-10-19T14:21:10.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.887+0000 7f3da48f0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3da00cb980 0x7f3da0136110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:10.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.887+0000 7f3da48f0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3da00cb980 0x7f3da0136110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51566/0 (socket says 172.21.15.57:51566) 2023-10-19T14:21:10.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.887+0000 7f3da48f0640 1 -- 172.21.15.57:0/4139648481 learned_addr learned my addr 172.21.15.57:0/4139648481 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:10.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.887+0000 7f3da48f0640 1 -- 172.21.15.57:0/4139648481 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3da00c96b0 msgr2=0x7f3da0139640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:10.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.887+0000 7f3da48f0640 1 --2- 172.21.15.57:0/4139648481 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3da00c96b0 0x7f3da0139640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:10.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.887+0000 7f3da48f0640 1 -- 172.21.15.57:0/4139648481 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3da0136650 msgr2=0x7f3da0150440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:10.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.887+0000 7f3da48f0640 1 --2- 172.21.15.57:0/4139648481 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3da0136650 0x7f3da0150440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:10.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.887+0000 7f3da48f0640 1 -- 172.21.15.57:0/4139648481 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d98009560 con 0x7f3da00cb980 2023-10-19T14:21:10.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.888+0000 7f3da48f0640 1 --2- 172.21.15.57:0/4139648481 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3da00cb980 0x7f3da0136110 secure :-1 s=READY pgs=2617 cs=0 l=1 rev1=1 crypto rx=0x7f3d9400d6d0 tx=0x7f3d9400db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:10.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.888+0000 7f3d9dffb640 1 -- 172.21.15.57:0/4139648481 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d94005220 con 0x7f3da00cb980 2023-10-19T14:21:10.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.888+0000 7f3d9dffb640 1 -- 172.21.15.57:0/4139648481 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f3d94005490 con 0x7f3da00cb980 2023-10-19T14:21:10.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.888+0000 7f3da7355640 1 -- 172.21.15.57:0/4139648481 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3da014f5e0 con 0x7f3da00cb980 2023-10-19T14:21:10.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.888+0000 7f3d9dffb640 1 -- 172.21.15.57:0/4139648481 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d94009a80 con 0x7f3da00cb980 2023-10-19T14:21:10.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.888+0000 7f3da7355640 1 -- 172.21.15.57:0/4139648481 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3da014fe80 con 0x7f3da00cb980 2023-10-19T14:21:10.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.889+0000 7f3da7355640 1 -- 172.21.15.57:0/4139648481 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3da006b1b0 con 0x7f3da00cb980 2023-10-19T14:21:10.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.889+0000 7f3d9dffb640 1 -- 172.21.15.57:0/4139648481 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f3d940047a0 con 0x7f3da00cb980 2023-10-19T14:21:10.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.890+0000 7f3d9dffb640 1 --2- 172.21.15.57:0/4139648481 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f3d8c06b0b0 0x7f3d8c06d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:10.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.890+0000 7f3d9dffb640 1 -- 172.21.15.57:0/4139648481 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(242..242 src has 1..242) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f3d94089f80 con 0x7f3da00cb980 2023-10-19T14:21:10.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.890+0000 7f3da50f1640 1 --2- 172.21.15.57:0/4139648481 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f3d8c06b0b0 0x7f3d8c06d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:10.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.890+0000 7f3da50f1640 1 --2- 172.21.15.57:0/4139648481 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f3d8c06b0b0 0x7f3d8c06d570 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3d98035000 tx=0x7f3d980046c0 comp rx=0 tx=0).ready entity=mgr.9530 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:10.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:10.894+0000 7f3d9dffb640 1 -- 172.21.15.57:0/4139648481 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3d940bcca0 con 0x7f3da00cb980 2023-10-19T14:21:11.074 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.071+0000 7f3da7355640 1 -- 172.21.15.57:0/4139648481 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3da00690e0 con 0x7f3da00cb980 2023-10-19T14:21:11.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.073+0000 7f3d9dffb640 1 -- 172.21.15.57:0/4139648481 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v242) v1 ==== 74+0+16852 (secure 0 0 0) 0x7f3d94017020 con 0x7f3da00cb980 2023-10-19T14:21:11.076 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:21:11.076 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":242,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T14:21:09.759627+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":32,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2023-10-19T14:21:08.101023+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"242","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2023-10-19T14:21:08.745817+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"242","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":241,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":241,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":241,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":241,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":241,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":241,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":241,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":241,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T14:21:11.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.074+0000 7f3da7355640 1 -- 172.21.15.57:0/4139648481 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f3d8c06b0b0 msgr2=0x7f3d8c06d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:11.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.075+0000 7f3da7355640 1 --2- 172.21.15.57:0/4139648481 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f3d8c06b0b0 0x7f3d8c06d570 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3d98035000 tx=0x7f3d980046c0 comp rx=0 tx=0).stop 2023-10-19T14:21:11.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.075+0000 7f3da7355640 1 -- 172.21.15.57:0/4139648481 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3da00cb980 msgr2=0x7f3da0136110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:11.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.075+0000 7f3da7355640 1 --2- 172.21.15.57:0/4139648481 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3da00cb980 0x7f3da0136110 secure :-1 s=READY pgs=2617 cs=0 l=1 rev1=1 crypto rx=0x7f3d9400d6d0 tx=0x7f3d9400db90 comp rx=0 tx=0).stop 2023-10-19T14:21:11.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.075+0000 7f3da7355640 1 -- 172.21.15.57:0/4139648481 shutdown_connections 2023-10-19T14:21:11.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.075+0000 7f3da7355640 1 --2- 172.21.15.57:0/4139648481 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f3d8c06b0b0 0x7f3d8c06d570 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:11.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.075+0000 7f3da7355640 1 --2- 172.21.15.57:0/4139648481 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3da0136650 0x7f3da0150440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:11.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.075+0000 7f3da7355640 1 --2- 172.21.15.57:0/4139648481 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3da00cb980 0x7f3da0136110 unknown :-1 s=CLOSED pgs=2617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:11.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.075+0000 7f3da7355640 1 --2- 172.21.15.57:0/4139648481 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3da00c96b0 0x7f3da0139640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:11.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.075+0000 7f3da7355640 1 -- 172.21.15.57:0/4139648481 >> 172.21.15.57:0/4139648481 conn(0x7f3da0069930 msgr2=0x7f3da015a480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:11.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.075+0000 7f3da7355640 1 -- 172.21.15.57:0/4139648481 shutdown_connections 2023-10-19T14:21:11.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.075+0000 7f3da7355640 1 -- 172.21.15.57:0/4139648481 wait complete. 2023-10-19T14:21:11.089 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-19T14:21:11.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.197+0000 7fa0c89e7640 1 -- 172.21.15.57:0/2845084432 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0c40afa20 msgr2=0x7fa0c4158300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:11.200 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.197+0000 7fa0c89e7640 1 --2- 172.21.15.57:0/2845084432 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0c40afa20 0x7fa0c4158300 secure :-1 s=READY pgs=2618 cs=0 l=1 rev1=1 crypto rx=0x7fa0ac005fd0 tx=0x7fa0ac02d0d0 comp rx=0 tx=0).stop 2023-10-19T14:21:11.201 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.197+0000 7fa0c89e7640 1 -- 172.21.15.57:0/2845084432 shutdown_connections 2023-10-19T14:21:11.201 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.197+0000 7fa0c89e7640 1 --2- 172.21.15.57:0/2845084432 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa0c40b0530 0x7fa0c40b09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:11.201 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.197+0000 7fa0c89e7640 1 --2- 172.21.15.57:0/2845084432 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0c40afa20 0x7fa0c4158300 unknown :-1 s=CLOSED pgs=2618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:11.201 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.197+0000 7fa0c89e7640 1 -- 172.21.15.57:0/2845084432 >> 172.21.15.57:0/2845084432 conn(0x7fa0c4069930 msgr2=0x7fa0c4069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:11.201 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.197+0000 7fa0c89e7640 1 -- 172.21.15.57:0/2845084432 shutdown_connections 2023-10-19T14:21:11.202 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.197+0000 7fa0c89e7640 1 -- 172.21.15.57:0/2845084432 wait complete. 2023-10-19T14:21:11.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.198+0000 7fa0c89e7640 1 Processor -- start 2023-10-19T14:21:11.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.198+0000 7fa0c89e7640 1 -- start start 2023-10-19T14:21:11.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.199+0000 7fa0c89e7640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa0c40afa20 0x7fa0c414aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:11.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.199+0000 7fa0c89e7640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa0c40b0530 0x7fa0c4148bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:11.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.199+0000 7fa0c89e7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0c40cbd20 0x7fa0c413fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:11.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.199+0000 7fa0c89e7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c40ac770 con 0x7fa0c40cbd20 2023-10-19T14:21:11.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.199+0000 7fa0c89e7640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c40ac9c0 con 0x7fa0c40b0530 2023-10-19T14:21:11.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.199+0000 7fa0c89e7640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c40acca0 con 0x7fa0c40afa20 2023-10-19T14:21:11.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.199+0000 7fa0c2d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0c40cbd20 0x7fa0c413fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:11.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.199+0000 7fa0c2d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0c40cbd20 0x7fa0c413fcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51588/0 (socket says 172.21.15.57:51588) 2023-10-19T14:21:11.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.199+0000 7fa0c2d9d640 1 -- 172.21.15.57:0/767467458 learned_addr learned my addr 172.21.15.57:0/767467458 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:11.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.199+0000 7fa0c259c640 1 --2- 172.21.15.57:0/767467458 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa0c40afa20 0x7fa0c414aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:11.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.199+0000 7fa0c1d9b640 1 --2- 172.21.15.57:0/767467458 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa0c40b0530 0x7fa0c4148bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:11.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.200+0000 7fa0c2d9d640 1 -- 172.21.15.57:0/767467458 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa0c40afa20 msgr2=0x7fa0c414aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:11.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.200+0000 7fa0c2d9d640 1 --2- 172.21.15.57:0/767467458 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa0c40afa20 0x7fa0c414aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:11.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.200+0000 7fa0c2d9d640 1 -- 172.21.15.57:0/767467458 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa0c40b0530 msgr2=0x7fa0c4148bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:11.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.200+0000 7fa0c2d9d640 1 --2- 172.21.15.57:0/767467458 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa0c40b0530 0x7fa0c4148bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:11.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.200+0000 7fa0c2d9d640 1 -- 172.21.15.57:0/767467458 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0ac00a670 con 0x7fa0c40cbd20 2023-10-19T14:21:11.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.200+0000 7fa0c2d9d640 1 --2- 172.21.15.57:0/767467458 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0c40cbd20 0x7fa0c413fcd0 secure :-1 s=READY pgs=2619 cs=0 l=1 rev1=1 crypto rx=0x7fa0b400cce0 tx=0x7fa0b400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:11.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.200+0000 7fa0baffd640 1 -- 172.21.15.57:0/767467458 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0b400f450 con 0x7fa0c40cbd20 2023-10-19T14:21:11.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.200+0000 7fa0baffd640 1 -- 172.21.15.57:0/767467458 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fa0b400f6c0 con 0x7fa0c40cbd20 2023-10-19T14:21:11.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.200+0000 7fa0c89e7640 1 -- 172.21.15.57:0/767467458 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0c40acf80 con 0x7fa0c40cbd20 2023-10-19T14:21:11.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.200+0000 7fa0baffd640 1 -- 172.21.15.57:0/767467458 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0b400fad0 con 0x7fa0c40cbd20 2023-10-19T14:21:11.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.200+0000 7fa0c89e7640 1 -- 172.21.15.57:0/767467458 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0c414b000 con 0x7fa0c40cbd20 2023-10-19T14:21:11.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.201+0000 7fa0c89e7640 1 -- 172.21.15.57:0/767467458 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0c40ad4a0 con 0x7fa0c40cbd20 2023-10-19T14:21:11.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.202+0000 7fa0baffd640 1 -- 172.21.15.57:0/767467458 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fa0b400fc30 con 0x7fa0c40cbd20 2023-10-19T14:21:11.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.202+0000 7fa0baffd640 1 --2- 172.21.15.57:0/767467458 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fa0a406b080 0x7fa0a406d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:11.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.202+0000 7fa0baffd640 1 -- 172.21.15.57:0/767467458 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(242..242 src has 1..242) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fa0b408a760 con 0x7fa0c40cbd20 2023-10-19T14:21:11.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.202+0000 7fa0c259c640 1 --2- 172.21.15.57:0/767467458 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fa0a406b080 0x7fa0a406d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:11.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.203+0000 7fa0c259c640 1 --2- 172.21.15.57:0/767467458 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fa0a406b080 0x7fa0a406d540 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc006fd0 tx=0x7fa0bc009060 comp rx=0 tx=0).ready entity=mgr.9530 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:11.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.206+0000 7fa0baffd640 1 -- 172.21.15.57:0/767467458 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa0c40ad4a0 con 0x7fa0c40cbd20 2023-10-19T14:21:11.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.418+0000 7fa0c89e7640 1 -- 172.21.15.57:0/767467458 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fa0c40ad4a0 con 0x7fa0c40cbd20 2023-10-19T14:21:11.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.420+0000 7fa0baffd640 1 -- 172.21.15.57:0/767467458 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v102) v1 ==== 118+0+0 (secure 0 0 0) 0x7fa0c40ad4a0 con 0x7fa0c40cbd20 2023-10-19T14:21:11.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.422+0000 7fa0c89e7640 1 -- 172.21.15.57:0/767467458 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fa0a406b080 msgr2=0x7fa0a406d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:11.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.422+0000 7fa0c89e7640 1 --2- 172.21.15.57:0/767467458 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fa0a406b080 0x7fa0a406d540 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc006fd0 tx=0x7fa0bc009060 comp rx=0 tx=0).stop 2023-10-19T14:21:11.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.422+0000 7fa0c89e7640 1 -- 172.21.15.57:0/767467458 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0c40cbd20 msgr2=0x7fa0c413fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:11.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.422+0000 7fa0c89e7640 1 --2- 172.21.15.57:0/767467458 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0c40cbd20 0x7fa0c413fcd0 secure :-1 s=READY pgs=2619 cs=0 l=1 rev1=1 crypto rx=0x7fa0b400cce0 tx=0x7fa0b400a620 comp rx=0 tx=0).stop 2023-10-19T14:21:11.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.422+0000 7fa0c89e7640 1 -- 172.21.15.57:0/767467458 shutdown_connections 2023-10-19T14:21:11.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.422+0000 7fa0c89e7640 1 --2- 172.21.15.57:0/767467458 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fa0a406b080 0x7fa0a406d540 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:11.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.422+0000 7fa0c89e7640 1 --2- 172.21.15.57:0/767467458 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0c40cbd20 0x7fa0c413fcd0 unknown :-1 s=CLOSED pgs=2619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:11.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.422+0000 7fa0c89e7640 1 --2- 172.21.15.57:0/767467458 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa0c40b0530 0x7fa0c4148bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:11.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.422+0000 7fa0c89e7640 1 --2- 172.21.15.57:0/767467458 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa0c40afa20 0x7fa0c414aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:11.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.422+0000 7fa0c89e7640 1 -- 172.21.15.57:0/767467458 >> 172.21.15.57:0/767467458 conn(0x7fa0c4069930 msgr2=0x7fa0c406bc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:11.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.422+0000 7fa0c89e7640 1 -- 172.21.15.57:0/767467458 shutdown_connections 2023-10-19T14:21:11.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.422+0000 7fa0c89e7640 1 -- 172.21.15.57:0/767467458 wait complete. 2023-10-19T14:21:11.436 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-19T14:21:11.436 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-19T14:21:11.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.546+0000 7fdd8da90640 1 -- 172.21.15.57:0/1214015718 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd700047c0 msgr2=0x7fdd70004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:11.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.546+0000 7fdd8da90640 1 --2- 172.21.15.57:0/1214015718 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd700047c0 0x7fdd70004c40 secure :-1 s=READY pgs=2621 cs=0 l=1 rev1=1 crypto rx=0x7fdd7c009b50 tx=0x7fdd700057f0 comp rx=0 tx=0).stop 2023-10-19T14:21:11.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.546+0000 7fdd8da90640 1 -- 172.21.15.57:0/1214015718 shutdown_connections 2023-10-19T14:21:11.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.546+0000 7fdd8da90640 1 --2- 172.21.15.57:0/1214015718 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd700047c0 0x7fdd70004c40 unknown :-1 s=CLOSED pgs=2621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:11.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.546+0000 7fdd8da90640 1 -- 172.21.15.57:0/1214015718 >> 172.21.15.57:0/1214015718 conn(0x7fdd88069930 msgr2=0x7fdd88069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:11.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.546+0000 7fdd8da90640 1 -- 172.21.15.57:0/1214015718 shutdown_connections 2023-10-19T14:21:11.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.546+0000 7fdd8da90640 1 -- 172.21.15.57:0/1214015718 wait complete. 2023-10-19T14:21:11.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.547+0000 7fdd8da90640 1 Processor -- start 2023-10-19T14:21:11.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.547+0000 7fdd8da90640 1 -- start start 2023-10-19T14:21:11.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.548+0000 7fdd8da90640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdd700047c0 0x7fdd88152900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:11.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.548+0000 7fdd8da90640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd88155ea0 0x7fdd88156320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:11.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.548+0000 7fdd8da90640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdd8815cd30 0x7fdd8815d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:11.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.548+0000 7fdd8da90640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd88156860 con 0x7fdd88155ea0 2023-10-19T14:21:11.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.548+0000 7fdd8da90640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd8815d6d0 con 0x7fdd8815cd30 2023-10-19T14:21:11.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.548+0000 7fdd8da90640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd881533a0 con 0x7fdd700047c0 2023-10-19T14:21:11.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.548+0000 7fdd86ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd88155ea0 0x7fdd88156320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:11.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.548+0000 7fdd86ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd88155ea0 0x7fdd88156320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51606/0 (socket says 172.21.15.57:51606) 2023-10-19T14:21:11.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.548+0000 7fdd86ffd640 1 -- 172.21.15.57:0/2658614725 learned_addr learned my addr 172.21.15.57:0/2658614725 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:11.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.548+0000 7fdd87fff640 1 --2- 172.21.15.57:0/2658614725 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdd8815cd30 0x7fdd8815d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:11.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.548+0000 7fdd877fe640 1 --2- 172.21.15.57:0/2658614725 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdd700047c0 0x7fdd88152900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:11.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.548+0000 7fdd86ffd640 1 -- 172.21.15.57:0/2658614725 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdd700047c0 msgr2=0x7fdd88152900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:11.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.548+0000 7fdd86ffd640 1 --2- 172.21.15.57:0/2658614725 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdd700047c0 0x7fdd88152900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:11.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.548+0000 7fdd86ffd640 1 -- 172.21.15.57:0/2658614725 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdd8815cd30 msgr2=0x7fdd8815d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:11.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.548+0000 7fdd86ffd640 1 --2- 172.21.15.57:0/2658614725 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdd8815cd30 0x7fdd8815d190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:11.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.548+0000 7fdd86ffd640 1 -- 172.21.15.57:0/2658614725 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd88152e40 con 0x7fdd88155ea0 2023-10-19T14:21:11.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.548+0000 7fdd877fe640 1 --2- 172.21.15.57:0/2658614725 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdd700047c0 0x7fdd88152900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:21:11.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.549+0000 7fdd86ffd640 1 --2- 172.21.15.57:0/2658614725 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd88155ea0 0x7fdd88156320 secure :-1 s=READY pgs=2622 cs=0 l=1 rev1=1 crypto rx=0x7fdd7c02cb70 tx=0x7fdd7c033c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:11.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.549+0000 7fdd6ffff640 1 -- 172.21.15.57:0/2658614725 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd7c004460 con 0x7fdd88155ea0 2023-10-19T14:21:11.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.549+0000 7fdd6ffff640 1 -- 172.21.15.57:0/2658614725 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fdd7c0035f0 con 0x7fdd88155ea0 2023-10-19T14:21:11.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.549+0000 7fdd8da90640 1 -- 172.21.15.57:0/2658614725 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd8815d910 con 0x7fdd88155ea0 2023-10-19T14:21:11.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.549+0000 7fdd6ffff640 1 -- 172.21.15.57:0/2658614725 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd7c0039e0 con 0x7fdd88155ea0 2023-10-19T14:21:11.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.549+0000 7fdd8da90640 1 -- 172.21.15.57:0/2658614725 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd8815ded0 con 0x7fdd88155ea0 2023-10-19T14:21:11.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.550+0000 7fdd8da90640 1 -- 172.21.15.57:0/2658614725 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd880690e0 con 0x7fdd88155ea0 2023-10-19T14:21:11.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.550+0000 7fdd6ffff640 1 -- 172.21.15.57:0/2658614725 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fdd7c003b40 con 0x7fdd88155ea0 2023-10-19T14:21:11.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.551+0000 7fdd6ffff640 1 --2- 172.21.15.57:0/2658614725 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fdd7406b080 0x7fdd7406d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:11.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.551+0000 7fdd6ffff640 1 -- 172.21.15.57:0/2658614725 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(243..243 src has 1..243) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fdd7c0acc60 con 0x7fdd88155ea0 2023-10-19T14:21:11.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.551+0000 7fdd877fe640 1 --2- 172.21.15.57:0/2658614725 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fdd7406b080 0x7fdd7406d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:11.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.552+0000 7fdd877fe640 1 --2- 172.21.15.57:0/2658614725 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fdd7406b080 0x7fdd7406d540 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fdd78001050 tx=0x7fdd78001620 comp rx=0 tx=0).ready entity=mgr.9530 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:11.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.555+0000 7fdd6ffff640 1 -- 172.21.15.57:0/2658614725 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fdd7c077800 con 0x7fdd88155ea0 2023-10-19T14:21:11.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:11.767+0000 7fdd8da90640 1 -- 172.21.15.57:0/2658614725 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fdd881580f0 con 0x7fdd88155ea0 2023-10-19T14:21:12.381 INFO:tasks.ceph.mds.c.smithi057.stderr:2023-10-19T14:21:12.379+0000 7f7968d74640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-19T14:21:12.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.440+0000 7fdd6ffff640 1 -- 172.21.15.57:0/2658614725 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v103) v1 ==== 122+0+0 (secure 0 0 0) 0x7fdd7c07ae20 con 0x7fdd88155ea0 2023-10-19T14:21:12.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.443+0000 7fdd8da90640 1 -- 172.21.15.57:0/2658614725 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fdd7406b080 msgr2=0x7fdd7406d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:12.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.443+0000 7fdd8da90640 1 --2- 172.21.15.57:0/2658614725 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fdd7406b080 0x7fdd7406d540 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fdd78001050 tx=0x7fdd78001620 comp rx=0 tx=0).stop 2023-10-19T14:21:12.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.443+0000 7fdd8da90640 1 -- 172.21.15.57:0/2658614725 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd88155ea0 msgr2=0x7fdd88156320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:12.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.443+0000 7fdd8da90640 1 --2- 172.21.15.57:0/2658614725 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd88155ea0 0x7fdd88156320 secure :-1 s=READY pgs=2622 cs=0 l=1 rev1=1 crypto rx=0x7fdd7c02cb70 tx=0x7fdd7c033c10 comp rx=0 tx=0).stop 2023-10-19T14:21:12.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.443+0000 7fdd8da90640 1 -- 172.21.15.57:0/2658614725 shutdown_connections 2023-10-19T14:21:12.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.443+0000 7fdd8da90640 1 --2- 172.21.15.57:0/2658614725 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fdd7406b080 0x7fdd7406d540 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:12.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.443+0000 7fdd8da90640 1 --2- 172.21.15.57:0/2658614725 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdd8815cd30 0x7fdd8815d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:12.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.443+0000 7fdd8da90640 1 --2- 172.21.15.57:0/2658614725 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd88155ea0 0x7fdd88156320 unknown :-1 s=CLOSED pgs=2622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:12.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.443+0000 7fdd8da90640 1 --2- 172.21.15.57:0/2658614725 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdd700047c0 0x7fdd88152900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:12.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.443+0000 7fdd8da90640 1 -- 172.21.15.57:0/2658614725 >> 172.21.15.57:0/2658614725 conn(0x7fdd88069930 msgr2=0x7fdd88159030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:12.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.443+0000 7fdd8da90640 1 -- 172.21.15.57:0/2658614725 shutdown_connections 2023-10-19T14:21:12.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.443+0000 7fdd8da90640 1 -- 172.21.15.57:0/2658614725 wait complete. 2023-10-19T14:21:12.462 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-19T14:21:12.463 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T14:21:12.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.575+0000 7f528e864640 1 -- 172.21.15.57:0/140783167 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f52880ac870 msgr2=0x7f528806b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:12.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.575+0000 7f528e864640 1 --2- 172.21.15.57:0/140783167 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f52880ac870 0x7f528806b980 secure :-1 s=READY pgs=2624 cs=0 l=1 rev1=1 crypto rx=0x7f5270009830 tx=0x7f527002bdf0 comp rx=0 tx=0).stop 2023-10-19T14:21:12.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.575+0000 7f528e864640 1 -- 172.21.15.57:0/140783167 shutdown_connections 2023-10-19T14:21:12.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.575+0000 7f528e864640 1 --2- 172.21.15.57:0/140783167 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f52880ac870 0x7f528806b980 unknown :-1 s=CLOSED pgs=2624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:12.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.575+0000 7f528e864640 1 -- 172.21.15.57:0/140783167 >> 172.21.15.57:0/140783167 conn(0x7f528806fae0 msgr2=0x7f528806fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:12.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.575+0000 7f528e864640 1 -- 172.21.15.57:0/140783167 shutdown_connections 2023-10-19T14:21:12.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.575+0000 7f528e864640 1 -- 172.21.15.57:0/140783167 wait complete. 2023-10-19T14:21:12.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.576+0000 7f528e864640 1 Processor -- start 2023-10-19T14:21:12.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.577+0000 7f528e864640 1 -- start start 2023-10-19T14:21:12.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.577+0000 7f528e864640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f528806bec0 0x7f528814de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:12.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.577+0000 7f528e864640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f52880ac870 0x7f52880b1b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:12.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.577+0000 7f528e864640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5288156790 0x7f52880aa8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:12.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.577+0000 7f528e864640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52881f49f0 con 0x7f5288156790 2023-10-19T14:21:12.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.577+0000 7f528e864640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52881f4cd0 con 0x7f528806bec0 2023-10-19T14:21:12.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.577+0000 7f528e864640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52881f4fb0 con 0x7f52880ac870 2023-10-19T14:21:12.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.578+0000 7f528ce01640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5288156790 0x7f52880aa8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:12.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.578+0000 7f528ce01640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5288156790 0x7f52880aa8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51618/0 (socket says 172.21.15.57:51618) 2023-10-19T14:21:12.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.578+0000 7f528ce01640 1 -- 172.21.15.57:0/3087658393 learned_addr learned my addr 172.21.15.57:0/3087658393 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:12.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.578+0000 7f5287fff640 1 --2- 172.21.15.57:0/3087658393 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f528806bec0 0x7f528814de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:12.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.578+0000 7f52877fe640 1 --2- 172.21.15.57:0/3087658393 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f52880ac870 0x7f52880b1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:12.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.578+0000 7f528ce01640 1 -- 172.21.15.57:0/3087658393 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f52880ac870 msgr2=0x7f52880b1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:12.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.578+0000 7f528ce01640 1 --2- 172.21.15.57:0/3087658393 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f52880ac870 0x7f52880b1b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:12.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.578+0000 7f528ce01640 1 -- 172.21.15.57:0/3087658393 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f528806bec0 msgr2=0x7f528814de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:12.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.578+0000 7f528ce01640 1 --2- 172.21.15.57:0/3087658393 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f528806bec0 0x7f528814de30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:12.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.578+0000 7f528ce01640 1 -- 172.21.15.57:0/3087658393 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5270009490 con 0x7f5288156790 2023-10-19T14:21:12.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.578+0000 7f52877fe640 1 --2- 172.21.15.57:0/3087658393 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f52880ac870 0x7f52880b1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:21:12.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.578+0000 7f528ce01640 1 --2- 172.21.15.57:0/3087658393 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5288156790 0x7f52880aa8c0 secure :-1 s=READY pgs=2625 cs=0 l=1 rev1=1 crypto rx=0x7f528000c680 tx=0x7f528000cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:12.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.578+0000 7f5284ff9640 1 -- 172.21.15.57:0/3087658393 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5280003b70 con 0x7f5288156790 2023-10-19T14:21:12.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.578+0000 7f5284ff9640 1 -- 172.21.15.57:0/3087658393 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f5280003cd0 con 0x7f5288156790 2023-10-19T14:21:12.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.578+0000 7f5284ff9640 1 -- 172.21.15.57:0/3087658393 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5280005330 con 0x7f5288156790 2023-10-19T14:21:12.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.578+0000 7f528e864640 1 -- 172.21.15.57:0/3087658393 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52881f5290 con 0x7f5288156790 2023-10-19T14:21:12.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.578+0000 7f528e864640 1 -- 172.21.15.57:0/3087658393 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52880b2050 con 0x7f5288156790 2023-10-19T14:21:12.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.580+0000 7f528e864640 1 -- 172.21.15.57:0/3087658393 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52881f57b0 con 0x7f5288156790 2023-10-19T14:21:12.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.580+0000 7f5284ff9640 1 -- 172.21.15.57:0/3087658393 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f5280005490 con 0x7f5288156790 2023-10-19T14:21:12.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.580+0000 7f5284ff9640 1 --2- 172.21.15.57:0/3087658393 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f526806b080 0x7f526806d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:12.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.580+0000 7f5284ff9640 1 -- 172.21.15.57:0/3087658393 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(243..243 src has 1..243) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f5280089950 con 0x7f5288156790 2023-10-19T14:21:12.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.581+0000 7f5287fff640 1 --2- 172.21.15.57:0/3087658393 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f526806b080 0x7f526806d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:12.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.581+0000 7f5287fff640 1 --2- 172.21.15.57:0/3087658393 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f526806b080 0x7f526806d540 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f52780046b0 tx=0x7f5278007eb0 comp rx=0 tx=0).ready entity=mgr.9530 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:12.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.585+0000 7f5284ff9640 1 -- 172.21.15.57:0/3087658393 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f52881f57b0 con 0x7f5288156790 2023-10-19T14:21:12.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.794+0000 7f528e864640 1 -- 172.21.15.57:0/3087658393 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f52881f57b0 con 0x7f5288156790 2023-10-19T14:21:12.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.794+0000 7f5284ff9640 1 -- 172.21.15.57:0/3087658393 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+4914 (secure 0 0 0) 0x7f52881f57b0 con 0x7f5288156790 2023-10-19T14:21:12.798 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:21:12.798 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":103,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9761,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/325888554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":325888554},{"type":"v1","addr":"172.21.15.179:6836","nonce":325888554}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":100},{"gid":9764,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/8038232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":8038232},{"type":"v1","addr":"172.21.15.179:6837","nonce":8038232}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":100},{"gid":12555,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6833/4256498777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":4256498777},{"type":"v1","addr":"172.21.15.57:6833","nonce":4256498777}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":99}],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T14:21:09.759935+0000","modified":"2023-10-19T14:21:11.768676+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":12564},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12564":{"gid":12564,"name":"c","rank":0,"incarnation":101,"state":"up:active","state_seq":3,"addr":"172.21.15.57:6835/18187246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":18187246},{"type":"v1","addr":"172.21.15.57:6835","nonce":18187246}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15}]} 2023-10-19T14:21:12.798 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 103 2023-10-19T14:21:12.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.796+0000 7f528e864640 1 -- 172.21.15.57:0/3087658393 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f526806b080 msgr2=0x7f526806d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:12.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.796+0000 7f528e864640 1 --2- 172.21.15.57:0/3087658393 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f526806b080 0x7f526806d540 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f52780046b0 tx=0x7f5278007eb0 comp rx=0 tx=0).stop 2023-10-19T14:21:12.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.796+0000 7f528e864640 1 -- 172.21.15.57:0/3087658393 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5288156790 msgr2=0x7f52880aa8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:12.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.796+0000 7f528e864640 1 --2- 172.21.15.57:0/3087658393 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5288156790 0x7f52880aa8c0 secure :-1 s=READY pgs=2625 cs=0 l=1 rev1=1 crypto rx=0x7f528000c680 tx=0x7f528000cb40 comp rx=0 tx=0).stop 2023-10-19T14:21:12.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.796+0000 7f528e864640 1 -- 172.21.15.57:0/3087658393 shutdown_connections 2023-10-19T14:21:12.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.796+0000 7f528e864640 1 --2- 172.21.15.57:0/3087658393 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f526806b080 0x7f526806d540 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:12.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.796+0000 7f528e864640 1 --2- 172.21.15.57:0/3087658393 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5288156790 0x7f52880aa8c0 unknown :-1 s=CLOSED pgs=2625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:12.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.796+0000 7f528e864640 1 --2- 172.21.15.57:0/3087658393 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f52880ac870 0x7f52880b1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:12.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.796+0000 7f528e864640 1 --2- 172.21.15.57:0/3087658393 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f528806bec0 0x7f528814de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:12.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.796+0000 7f528e864640 1 -- 172.21.15.57:0/3087658393 >> 172.21.15.57:0/3087658393 conn(0x7f528806fae0 msgr2=0x7f528806c2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:12.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.796+0000 7f528e864640 1 -- 172.21.15.57:0/3087658393 shutdown_connections 2023-10-19T14:21:12.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.796+0000 7f528e864640 1 -- 172.21.15.57:0/3087658393 wait complete. 2023-10-19T14:21:12.810 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T14:21:12.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.916+0000 7f3e5a7e2640 1 -- 172.21.15.57:0/1142414792 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3e540ceb40 msgr2=0x7f3e540b0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:12.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.916+0000 7f3e5a7e2640 1 --2- 172.21.15.57:0/1142414792 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3e540ceb40 0x7f3e540b0650 secure :-1 s=READY pgs=2627 cs=0 l=1 rev1=1 crypto rx=0x7f3e5408a120 tx=0x7f3e4c02c080 comp rx=0 tx=0).stop 2023-10-19T14:21:12.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.916+0000 7f3e5a7e2640 1 -- 172.21.15.57:0/1142414792 shutdown_connections 2023-10-19T14:21:12.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.916+0000 7f3e5a7e2640 1 --2- 172.21.15.57:0/1142414792 >> v2:172.21.15.179:3300/0 conn(0x7f3e5415b2a0 0x7f3e540d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:12.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.916+0000 7f3e5a7e2640 1 --2- 172.21.15.57:0/1142414792 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3e540ceb40 0x7f3e540b0650 unknown :-1 s=CLOSED pgs=2627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:12.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.916+0000 7f3e5a7e2640 1 -- 172.21.15.57:0/1142414792 >> 172.21.15.57:0/1142414792 conn(0x7f3e540d0d60 msgr2=0x7f3e540d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:12.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.916+0000 7f3e5a7e2640 1 -- 172.21.15.57:0/1142414792 shutdown_connections 2023-10-19T14:21:12.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.916+0000 7f3e5a7e2640 1 -- 172.21.15.57:0/1142414792 wait complete. 2023-10-19T14:21:12.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.917+0000 7f3e5a7e2640 1 Processor -- start 2023-10-19T14:21:12.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.918+0000 7f3e5a7e2640 1 -- start start 2023-10-19T14:21:12.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.918+0000 7f3e5a7e2640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3e540ceb40 0x7f3e541f23d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:12.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.918+0000 7f3e5a7e2640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3e540cf340 0x7f3e540c9bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:12.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.918+0000 7f3e5a7e2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3e5415b2a0 0x7f3e540b58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:12.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.918+0000 7f3e5a7e2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e541f1f40 con 0x7f3e5415b2a0 2023-10-19T14:21:12.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.918+0000 7f3e5a7e2640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e540ca5b0 con 0x7f3e540cf340 2023-10-19T14:21:12.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.918+0000 7f3e5a7e2640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e540ca890 con 0x7f3e540ceb40 2023-10-19T14:21:12.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.918+0000 7f3e58d7f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3e5415b2a0 0x7f3e540b58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:12.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.918+0000 7f3e58d7f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3e5415b2a0 0x7f3e540b58f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51626/0 (socket says 172.21.15.57:51626) 2023-10-19T14:21:12.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.918+0000 7f3e58d7f640 1 -- 172.21.15.57:0/104800921 learned_addr learned my addr 172.21.15.57:0/104800921 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:12.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.919+0000 7f3e537fe640 1 --2- 172.21.15.57:0/104800921 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3e540cf340 0x7f3e540c9bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:12.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.919+0000 7f3e53fff640 1 --2- 172.21.15.57:0/104800921 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3e540ceb40 0x7f3e541f23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:12.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.919+0000 7f3e58d7f640 1 -- 172.21.15.57:0/104800921 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3e540ceb40 msgr2=0x7f3e541f23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:12.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.919+0000 7f3e58d7f640 1 --2- 172.21.15.57:0/104800921 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3e540ceb40 0x7f3e541f23d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:12.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.919+0000 7f3e58d7f640 1 -- 172.21.15.57:0/104800921 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3e540cf340 msgr2=0x7f3e540c9bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:12.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.919+0000 7f3e58d7f640 1 --2- 172.21.15.57:0/104800921 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3e540cf340 0x7f3e540c9bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:12.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.919+0000 7f3e58d7f640 1 -- 172.21.15.57:0/104800921 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e4c009840 con 0x7f3e5415b2a0 2023-10-19T14:21:12.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.919+0000 7f3e53fff640 1 --2- 172.21.15.57:0/104800921 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3e540ceb40 0x7f3e541f23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:21:12.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.919+0000 7f3e537fe640 1 --2- 172.21.15.57:0/104800921 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3e540cf340 0x7f3e540c9bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:21:12.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.919+0000 7f3e58d7f640 1 --2- 172.21.15.57:0/104800921 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3e5415b2a0 0x7f3e540b58f0 secure :-1 s=READY pgs=2628 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c009ca0 tx=0x7f3e4c004340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:12.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.919+0000 7f3e50ff9640 1 -- 172.21.15.57:0/104800921 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e4c03d430 con 0x7f3e5415b2a0 2023-10-19T14:21:12.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.919+0000 7f3e50ff9640 1 -- 172.21.15.57:0/104800921 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f3e4c03d680 con 0x7f3e5415b2a0 2023-10-19T14:21:12.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.919+0000 7f3e5a7e2640 1 -- 172.21.15.57:0/104800921 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e541f1bd0 con 0x7f3e5415b2a0 2023-10-19T14:21:12.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.919+0000 7f3e50ff9640 1 -- 172.21.15.57:0/104800921 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e4c03da50 con 0x7f3e5415b2a0 2023-10-19T14:21:12.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.919+0000 7f3e5a7e2640 1 -- 172.21.15.57:0/104800921 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e5415e3f0 con 0x7f3e5415b2a0 2023-10-19T14:21:12.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.920+0000 7f3e5a7e2640 1 -- 172.21.15.57:0/104800921 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e5415e5e0 con 0x7f3e5415b2a0 2023-10-19T14:21:12.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.921+0000 7f3e50ff9640 1 -- 172.21.15.57:0/104800921 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f3e4c03dbb0 con 0x7f3e5415b2a0 2023-10-19T14:21:12.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.921+0000 7f3e50ff9640 1 --2- 172.21.15.57:0/104800921 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f3e4006b080 0x7f3e4006d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:12.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.921+0000 7f3e50ff9640 1 -- 172.21.15.57:0/104800921 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(243..243 src has 1..243) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f3e4c0acbd0 con 0x7f3e5415b2a0 2023-10-19T14:21:12.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.921+0000 7f3e53fff640 1 --2- 172.21.15.57:0/104800921 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f3e4006b080 0x7f3e4006d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:12.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.922+0000 7f3e53fff640 1 --2- 172.21.15.57:0/104800921 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f3e4006b080 0x7f3e4006d540 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3e48000f60 tx=0x7f3e48009040 comp rx=0 tx=0).ready entity=mgr.9530 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:12.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:12.925+0000 7f3e50ff9640 1 -- 172.21.15.57:0/104800921 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3e4c0777c0 con 0x7f3e5415b2a0 2023-10-19T14:21:13.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.103+0000 7f3e5a7e2640 1 -- 172.21.15.57:0/104800921 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3e540b0d00 con 0x7f3e5415b2a0 2023-10-19T14:21:13.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.104+0000 7f3e50ff9640 1 -- 172.21.15.57:0/104800921 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v243) v1 ==== 74+0+16839 (secure 0 0 0) 0x7f3e4c04b090 con 0x7f3e5415b2a0 2023-10-19T14:21:13.107 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:21:13.108 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":243,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T14:21:11.406848+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":32,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2023-10-19T14:21:08.101023+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"242","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2023-10-19T14:21:08.745817+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"243","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":241,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":241,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":241,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":241,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":241,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":241,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":241,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":241,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T14:21:13.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.106+0000 7f3e5a7e2640 1 -- 172.21.15.57:0/104800921 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f3e4006b080 msgr2=0x7f3e4006d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:13.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.106+0000 7f3e5a7e2640 1 --2- 172.21.15.57:0/104800921 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f3e4006b080 0x7f3e4006d540 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3e48000f60 tx=0x7f3e48009040 comp rx=0 tx=0).stop 2023-10-19T14:21:13.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.106+0000 7f3e5a7e2640 1 -- 172.21.15.57:0/104800921 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3e5415b2a0 msgr2=0x7f3e540b58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:13.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.106+0000 7f3e5a7e2640 1 --2- 172.21.15.57:0/104800921 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3e5415b2a0 0x7f3e540b58f0 secure :-1 s=READY pgs=2628 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c009ca0 tx=0x7f3e4c004340 comp rx=0 tx=0).stop 2023-10-19T14:21:13.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.106+0000 7f3e5a7e2640 1 -- 172.21.15.57:0/104800921 shutdown_connections 2023-10-19T14:21:13.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.106+0000 7f3e5a7e2640 1 --2- 172.21.15.57:0/104800921 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f3e4006b080 0x7f3e4006d540 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:13.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.106+0000 7f3e5a7e2640 1 --2- 172.21.15.57:0/104800921 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3e5415b2a0 0x7f3e540b58f0 unknown :-1 s=CLOSED pgs=2628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:13.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.106+0000 7f3e5a7e2640 1 --2- 172.21.15.57:0/104800921 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3e540cf340 0x7f3e540c9bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:13.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.106+0000 7f3e5a7e2640 1 --2- 172.21.15.57:0/104800921 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3e540ceb40 0x7f3e541f23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:13.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.106+0000 7f3e5a7e2640 1 -- 172.21.15.57:0/104800921 >> 172.21.15.57:0/104800921 conn(0x7f3e540d0d60 msgr2=0x7f3e540ad8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:13.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.106+0000 7f3e5a7e2640 1 -- 172.21.15.57:0/104800921 shutdown_connections 2023-10-19T14:21:13.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.106+0000 7f3e5a7e2640 1 -- 172.21.15.57:0/104800921 wait complete. 2023-10-19T14:21:13.120 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-19T14:21:13.121 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-19T14:21:13.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.227+0000 7f5eb11c9640 1 -- 172.21.15.57:0/609273360 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5e98004d00 msgr2=0x7f5e98005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:13.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.227+0000 7f5eb11c9640 1 --2- 172.21.15.57:0/609273360 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5e98004d00 0x7f5e98005180 secure :-1 s=READY pgs=2630 cs=0 l=1 rev1=1 crypto rx=0x7f5eac15d100 tx=0x7f5e980057f0 comp rx=0 tx=0).stop 2023-10-19T14:21:13.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.227+0000 7f5eab59e640 1 -- 172.21.15.57:0/609273360 reap_dead start 2023-10-19T14:21:13.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.227+0000 7f5eb11c9640 1 -- 172.21.15.57:0/609273360 shutdown_connections 2023-10-19T14:21:13.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.227+0000 7f5eb11c9640 1 -- 172.21.15.57:0/609273360 >> 172.21.15.57:0/609273360 conn(0x7f5eac069930 msgr2=0x7f5eac069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:13.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.227+0000 7f5eb11c9640 1 -- 172.21.15.57:0/609273360 shutdown_connections 2023-10-19T14:21:13.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.227+0000 7f5eb11c9640 1 -- 172.21.15.57:0/609273360 wait complete. 2023-10-19T14:21:13.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.229+0000 7f5eb11c9640 1 Processor -- start 2023-10-19T14:21:13.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.229+0000 7f5eb11c9640 1 -- start start 2023-10-19T14:21:13.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.229+0000 7f5eb11c9640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5e98004d00 0x7f5eac0a9920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:13.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.229+0000 7f5eb11c9640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5eac0a8180 0x7f5eac0aaec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:13.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.229+0000 7f5eb11c9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5eac1ef480 0x7f5eac1ef900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:13.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.229+0000 7f5eb11c9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eac0ab400 con 0x7f5eac1ef480 2023-10-19T14:21:13.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.229+0000 7f5eb11c9640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eac0a9490 con 0x7f5eac0a8180 2023-10-19T14:21:13.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.229+0000 7f5eb11c9640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5eac0a8600 con 0x7f5e98004d00 2023-10-19T14:21:13.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.230+0000 7f5eab59e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5eac1ef480 0x7f5eac1ef900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:13.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.230+0000 7f5eab59e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5eac1ef480 0x7f5eac1ef900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51636/0 (socket says 172.21.15.57:51636) 2023-10-19T14:21:13.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.230+0000 7f5eab59e640 1 -- 172.21.15.57:0/1427606936 learned_addr learned my addr 172.21.15.57:0/1427606936 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:13.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.230+0000 7f5eaad9d640 1 --2- 172.21.15.57:0/1427606936 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5e98004d00 0x7f5eac0a9920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:13.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.230+0000 7f5ea3fff640 1 --2- 172.21.15.57:0/1427606936 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5eac0a8180 0x7f5eac0aaec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:13.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.230+0000 7f5eab59e640 1 -- 172.21.15.57:0/1427606936 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5e98004d00 msgr2=0x7f5eac0a9920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:13.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.230+0000 7f5eab59e640 1 --2- 172.21.15.57:0/1427606936 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5e98004d00 0x7f5eac0a9920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:13.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.230+0000 7f5eab59e640 1 -- 172.21.15.57:0/1427606936 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5eac0a8180 msgr2=0x7f5eac0aaec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:13.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.230+0000 7f5eab59e640 1 --2- 172.21.15.57:0/1427606936 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5eac0a8180 0x7f5eac0aaec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:13.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.230+0000 7f5eab59e640 1 -- 172.21.15.57:0/1427606936 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eac0a8880 con 0x7f5eac1ef480 2023-10-19T14:21:13.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.230+0000 7f5eab59e640 1 --2- 172.21.15.57:0/1427606936 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5eac1ef480 0x7f5eac1ef900 secure :-1 s=READY pgs=2631 cs=0 l=1 rev1=1 crypto rx=0x7f5ea400c700 tx=0x7f5ea4002070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:13.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.230+0000 7f5ea37fe640 1 -- 172.21.15.57:0/1427606936 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ea4001970 con 0x7f5eac1ef480 2023-10-19T14:21:13.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.230+0000 7f5ea37fe640 1 -- 172.21.15.57:0/1427606936 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f5ea4001ad0 con 0x7f5eac1ef480 2023-10-19T14:21:13.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.230+0000 7f5eb11c9640 1 -- 172.21.15.57:0/1427606936 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5eac0a8b60 con 0x7f5eac1ef480 2023-10-19T14:21:13.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.230+0000 7f5ea37fe640 1 -- 172.21.15.57:0/1427606936 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ea4003bb0 con 0x7f5eac1ef480 2023-10-19T14:21:13.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.230+0000 7f5eb11c9640 1 -- 172.21.15.57:0/1427606936 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5eac0a9120 con 0x7f5eac1ef480 2023-10-19T14:21:13.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.231+0000 7f5eb11c9640 1 -- 172.21.15.57:0/1427606936 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5eac0a9620 con 0x7f5eac1ef480 2023-10-19T14:21:13.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.232+0000 7f5ea37fe640 1 -- 172.21.15.57:0/1427606936 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f5ea4031030 con 0x7f5eac1ef480 2023-10-19T14:21:13.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.232+0000 7f5ea37fe640 1 --2- 172.21.15.57:0/1427606936 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f5e8c06b0b0 0x7f5e8c06d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:13.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.232+0000 7f5ea37fe640 1 -- 172.21.15.57:0/1427606936 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(243..243 src has 1..243) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f5ea40aac80 con 0x7f5eac1ef480 2023-10-19T14:21:13.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.232+0000 7f5eaad9d640 1 --2- 172.21.15.57:0/1427606936 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f5e8c06b0b0 0x7f5e8c06d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:13.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.233+0000 7f5eaad9d640 1 --2- 172.21.15.57:0/1427606936 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f5e8c06b0b0 0x7f5e8c06d570 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5eac1f29a0 tx=0x7f5e9c006b30 comp rx=0 tx=0).ready entity=mgr.9530 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:13.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.236+0000 7f5ea37fe640 1 -- 172.21.15.57:0/1427606936 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f5ea4075950 con 0x7f5eac1ef480 2023-10-19T14:21:13.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.411+0000 7f5eb11c9640 1 -- 172.21.15.57:0/1427606936 --> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5eac0ca070 con 0x7f5e8c06b0b0 2023-10-19T14:21:13.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.419+0000 7f5ea37fe640 1 -- 172.21.15.57:0/1427606936 <== mgr.9530 v2:172.21.15.179:6832/3547895240 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+277694 (secure 0 0 0) 0x7f5e9c0125e0 con 0x7f5e8c06b0b0 2023-10-19T14:21:13.424 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:21:13.426 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2023-10-19T14:21:13.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.422+0000 7f5eb11c9640 1 -- 172.21.15.57:0/1427606936 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f5e8c06b0b0 msgr2=0x7f5e8c06d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:13.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.422+0000 7f5eb11c9640 1 --2- 172.21.15.57:0/1427606936 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f5e8c06b0b0 0x7f5e8c06d570 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5eac1f29a0 tx=0x7f5e9c006b30 comp rx=0 tx=0).stop 2023-10-19T14:21:13.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.422+0000 7f5eb11c9640 1 -- 172.21.15.57:0/1427606936 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5eac1ef480 msgr2=0x7f5eac1ef900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:13.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.422+0000 7f5eb11c9640 1 --2- 172.21.15.57:0/1427606936 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5eac1ef480 0x7f5eac1ef900 secure :-1 s=READY pgs=2631 cs=0 l=1 rev1=1 crypto rx=0x7f5ea400c700 tx=0x7f5ea4002070 comp rx=0 tx=0).stop 2023-10-19T14:21:13.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.422+0000 7f5eb11c9640 1 -- 172.21.15.57:0/1427606936 shutdown_connections 2023-10-19T14:21:13.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.422+0000 7f5eb11c9640 1 --2- 172.21.15.57:0/1427606936 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f5e8c06b0b0 0x7f5e8c06d570 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:13.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.422+0000 7f5eb11c9640 1 --2- 172.21.15.57:0/1427606936 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5eac1ef480 0x7f5eac1ef900 unknown :-1 s=CLOSED pgs=2631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:13.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.422+0000 7f5eb11c9640 1 --2- 172.21.15.57:0/1427606936 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5eac0a8180 0x7f5eac0aaec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:13.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.422+0000 7f5eb11c9640 1 --2- 172.21.15.57:0/1427606936 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5e98004d00 0x7f5eac0a9920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:13.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.422+0000 7f5eb11c9640 1 -- 172.21.15.57:0/1427606936 >> 172.21.15.57:0/1427606936 conn(0x7f5eac069930 msgr2=0x7f5eac0b14c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:13.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.422+0000 7f5eb11c9640 1 -- 172.21.15.57:0/1427606936 shutdown_connections 2023-10-19T14:21:13.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.422+0000 7f5eb11c9640 1 -- 172.21.15.57:0/1427606936 wait complete. 2023-10-19T14:21:13.435 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":29,"stamp":"2023-10-19T14:21:12.172150+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2402,"num_read_kb":5036,"num_write":629,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3208,"ondisk_log_size":3208,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":449160,"kb_used_data":24768,"kb_used_omap":12,"kb_used_meta":424243,"kb_avail":754525560,"statfs":{"total":773094113280,"available":772634173440,"internally_reserved":0,"allocated":25362432,"data_stored":13607110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":434425401},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000817"},"pg_stats":[{"pgid":"32.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.1f","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.395395+0000","last_change":"2023-10-19T14:21:08.395395+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.395395+0000","last_undegraded":"2023-10-19T14:21:08.395395+0000","last_fullsized":"2023-10-19T14:21:08.395395+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.1d","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.393449+0000","last_change":"2023-10-19T14:21:08.393449+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.393449+0000","last_undegraded":"2023-10-19T14:21:08.393449+0000","last_fullsized":"2023-10-19T14:21:08.393449+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.1c","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.399714+0000","last_change":"2023-10-19T14:21:08.399714+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.399714+0000","last_undegraded":"2023-10-19T14:21:08.399714+0000","last_fullsized":"2023-10-19T14:21:08.399714+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.1b","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.397712+0000","last_change":"2023-10-19T14:21:08.397712+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.397712+0000","last_undegraded":"2023-10-19T14:21:08.397712+0000","last_fullsized":"2023-10-19T14:21:08.397712+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.1a","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.399541+0000","last_change":"2023-10-19T14:21:08.399541+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.399541+0000","last_undegraded":"2023-10-19T14:21:08.399541+0000","last_fullsized":"2023-10-19T14:21:08.399541+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.401761+0000","last_change":"2023-10-19T14:21:08.401761+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.401761+0000","last_undegraded":"2023-10-19T14:21:08.401761+0000","last_fullsized":"2023-10-19T14:21:08.401761+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.395206+0000","last_change":"2023-10-19T14:21:08.395206+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.395206+0000","last_undegraded":"2023-10-19T14:21:08.395206+0000","last_fullsized":"2023-10-19T14:21:08.395206+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.399532+0000","last_change":"2023-10-19T14:21:08.399532+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.399532+0000","last_undegraded":"2023-10-19T14:21:08.399532+0000","last_fullsized":"2023-10-19T14:21:08.399532+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.401377+0000","last_change":"2023-10-19T14:21:08.401377+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.401377+0000","last_undegraded":"2023-10-19T14:21:08.401377+0000","last_fullsized":"2023-10-19T14:21:08.401377+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.15","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.397203+0000","last_change":"2023-10-19T14:21:08.397203+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.397203+0000","last_undegraded":"2023-10-19T14:21:08.397203+0000","last_fullsized":"2023-10-19T14:21:08.397203+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.14","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.394976+0000","last_change":"2023-10-19T14:21:08.394976+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.394976+0000","last_undegraded":"2023-10-19T14:21:08.394976+0000","last_fullsized":"2023-10-19T14:21:08.394976+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.13","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.397244+0000","last_change":"2023-10-19T14:21:08.397244+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.397244+0000","last_undegraded":"2023-10-19T14:21:08.397244+0000","last_fullsized":"2023-10-19T14:21:08.397244+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.397963+0000","last_change":"2023-10-19T14:21:08.397963+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.397963+0000","last_undegraded":"2023-10-19T14:21:08.397963+0000","last_fullsized":"2023-10-19T14:21:08.397963+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.397350+0000","last_change":"2023-10-19T14:21:08.397350+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.397350+0000","last_undegraded":"2023-10-19T14:21:08.397350+0000","last_fullsized":"2023-10-19T14:21:08.397350+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.10","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.398284+0000","last_change":"2023-10-19T14:21:08.398284+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.398284+0000","last_undegraded":"2023-10-19T14:21:08.398284+0000","last_fullsized":"2023-10-19T14:21:08.398284+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.f","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.394918+0000","last_change":"2023-10-19T14:21:08.394918+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.394918+0000","last_undegraded":"2023-10-19T14:21:08.394918+0000","last_fullsized":"2023-10-19T14:21:08.394918+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.e","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.398215+0000","last_change":"2023-10-19T14:21:08.398215+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.398215+0000","last_undegraded":"2023-10-19T14:21:08.398215+0000","last_fullsized":"2023-10-19T14:21:08.398215+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.d","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.392973+0000","last_change":"2023-10-19T14:21:08.392973+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.392973+0000","last_undegraded":"2023-10-19T14:21:08.392973+0000","last_fullsized":"2023-10-19T14:21:08.392973+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.397335+0000","last_change":"2023-10-19T14:21:08.397335+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.397335+0000","last_undegraded":"2023-10-19T14:21:08.397335+0000","last_fullsized":"2023-10-19T14:21:08.397335+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.b","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.401221+0000","last_change":"2023-10-19T14:21:08.401221+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.401221+0000","last_undegraded":"2023-10-19T14:21:08.401221+0000","last_fullsized":"2023-10-19T14:21:08.401221+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.392818+0000","last_change":"2023-10-19T14:21:08.392818+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.392818+0000","last_undegraded":"2023-10-19T14:21:08.392818+0000","last_fullsized":"2023-10-19T14:21:08.392818+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.392864+0000","last_change":"2023-10-19T14:21:08.392864+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.392864+0000","last_undegraded":"2023-10-19T14:21:08.392864+0000","last_fullsized":"2023-10-19T14:21:08.392864+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.399409+0000","last_change":"2023-10-19T14:21:08.399409+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.399409+0000","last_undegraded":"2023-10-19T14:21:08.399409+0000","last_fullsized":"2023-10-19T14:21:08.399409+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.7","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.401244+0000","last_change":"2023-10-19T14:21:08.401244+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.401244+0000","last_undegraded":"2023-10-19T14:21:08.401244+0000","last_fullsized":"2023-10-19T14:21:08.401244+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.6","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.397194+0000","last_change":"2023-10-19T14:21:08.397194+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.397194+0000","last_undegraded":"2023-10-19T14:21:08.397194+0000","last_fullsized":"2023-10-19T14:21:08.397194+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.398883+0000","last_change":"2023-10-19T14:21:08.398883+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.398883+0000","last_undegraded":"2023-10-19T14:21:08.398883+0000","last_fullsized":"2023-10-19T14:21:08.398883+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.398251+0000","last_change":"2023-10-19T14:21:08.398251+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.398251+0000","last_undegraded":"2023-10-19T14:21:08.398251+0000","last_fullsized":"2023-10-19T14:21:08.398251+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.392809+0000","last_change":"2023-10-19T14:21:08.392809+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.392809+0000","last_undegraded":"2023-10-19T14:21:08.392809+0000","last_fullsized":"2023-10-19T14:21:08.392809+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.397367+0000","last_change":"2023-10-19T14:21:08.397367+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.397367+0000","last_undegraded":"2023-10-19T14:21:08.397367+0000","last_fullsized":"2023-10-19T14:21:08.397367+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.401262+0000","last_change":"2023-10-19T14:21:08.401262+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.401262+0000","last_undegraded":"2023-10-19T14:21:08.401262+0000","last_fullsized":"2023-10-19T14:21:08.401262+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.392705+0000","last_change":"2023-10-19T14:21:08.392705+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.392705+0000","last_undegraded":"2023-10-19T14:21:08.392705+0000","last_fullsized":"2023-10-19T14:21:08.392705+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.1e","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.392309+0000","last_change":"2023-10-19T14:21:08.392309+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.392309+0000","last_undegraded":"2023-10-19T14:21:08.392309+0000","last_fullsized":"2023-10-19T14:21:08.392309+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"229'3208","reported_seq":4177,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-19T14:21:06.428133+0000","last_change":"2023-10-19T12:44:08.636613+0000","last_active":"2023-10-19T14:21:06.428133+0000","last_peered":"2023-10-19T14:21:06.428133+0000","last_clean":"2023-10-19T14:21:06.428133+0000","last_became_active":"2023-10-19T12:44:08.635697+0000","last_became_peered":"2023-10-19T12:44:08.635697+0000","last_unstale":"2023-10-19T14:21:06.428133+0000","last_undegraded":"2023-10-19T14:21:06.428133+0000","last_fullsized":"2023-10-19T14:21:06.428133+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_clean_scrub_stamp":"2023-10-19T12:44:07.399738+0000","objects_scrubbed":0,"log_size":3208,"log_dups_size":0,"ondisk_log_size":3208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:15:23.685485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2402,"num_read_kb":5036,"num_write":629,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":32,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":31,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2402,"num_read_kb":5036,"num_write":629,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3208,"ondisk_log_size":3208,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576010,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66652,"kb_used_data":3536,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305188,"statfs":{"total":96636764160,"available":96568512512,"internally_reserved":0,"allocated":3620864,"data_stored":2146054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":64616907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:20:09 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.56999999999999995,"15min":0.58499999999999996},"min":{"1min":0.35799999999999998,"5min":0.34100000000000003,"15min":0.32200000000000001},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.86199999999999999},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.60199999999999998,"15min":0.60999999999999999},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.312},"max":{"1min":0.77600000000000002,"5min":0.81899999999999995,"15min":0.86599999999999999},"last":0.58699999999999997}]},{"osd":1,"last update":"Thu Oct 19 14:20:09 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.59799999999999998,"15min":0.59999999999999998},"min":{"1min":0.46500000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.76800000000000002,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.59499999999999997,"15min":0.59799999999999998},"min":{"1min":0.373,"5min":0.373,"15min":0.33500000000000002},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.45000000000000001}]},{"osd":2,"last update":"Thu Oct 19 14:20:09 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64900000000000002,"15min":0.64800000000000002},"min":{"1min":0.50900000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.85499999999999998},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.66600000000000004,"15min":0.66200000000000003},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.40100000000000002},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.66900000000000004}]},{"osd":3,"last update":"Thu Oct 19 14:20:09 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.63300000000000001,"15min":0.63700000000000001},"min":{"1min":0.51100000000000001,"5min":0.42099999999999999,"15min":0.36399999999999999},"max":{"1min":0.82699999999999996,"5min":0.84799999999999998,"15min":0.873},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.68799999999999994,"15min":0.68700000000000006},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":0.873,"5min":0.876,"15min":0.89600000000000002},"last":0.71199999999999997}]},{"osd":4,"last update":"Thu Oct 19 14:20:09 2023","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.50600000000000001,"15min":0.49299999999999999},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.62,"5min":0.78100000000000003,"15min":0.83799999999999997},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.51800000000000002,"15min":0.50600000000000001},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.71899999999999997,"5min":0.80200000000000005,"15min":0.81699999999999995},"last":0.67900000000000005}]},{"osd":5,"last update":"Thu Oct 19 14:20:09 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51400000000000001,"15min":0.50800000000000001},"min":{"1min":0.36799999999999999,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.65600000000000003,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.628},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.55600000000000005,"15min":0.54900000000000004},"min":{"1min":0.377,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.70999999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.72699999999999998}]},{"osd":6,"last update":"Thu Oct 19 14:20:09 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57699999999999996,"15min":0.57399999999999995},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.35999999999999999},"max":{"1min":0.73199999999999998,"5min":0.82999999999999996,"15min":0.85099999999999998},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.58099999999999996,"15min":0.56799999999999995},"min":{"1min":0.38800000000000001,"5min":0.35599999999999998,"15min":0.33500000000000002},"max":{"1min":0.78200000000000003,"5min":0.85199999999999998,"15min":0.85999999999999999},"last":0.75600000000000001}]}]},{"osd":6,"up_from":13,"seq":55834576010,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38288,"kb_used_data":2944,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333552,"statfs":{"total":96636764160,"available":96597557248,"internally_reserved":0,"allocated":3014656,"data_stored":1549521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:21:02 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.59199999999999997,"15min":0.61099999999999999},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.78400000000000003,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.56299999999999994,"15min":0.60599999999999998},"min":{"1min":0.36699999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.71799999999999997,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.59999999999999998}]},{"osd":1,"last update":"Thu Oct 19 14:21:02 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.60099999999999998,"15min":0.61799999999999999},"min":{"1min":0.373,"5min":0.373,"15min":0.35799999999999998},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":1.165},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.60199999999999998,"15min":0.626},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.77400000000000002,"5min":0.83699999999999997,"15min":0.91300000000000003},"last":0.51400000000000001}]},{"osd":2,"last update":"Thu Oct 19 14:21:02 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.57599999999999996,"15min":0.60199999999999998},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.69099999999999995,"5min":0.80100000000000005,"15min":0.873},"last":0.497},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.60599999999999998,"15min":0.624},"min":{"1min":0.40899999999999997,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.74199999999999999,"5min":0.83499999999999996,"15min":0.93100000000000005},"last":0.48499999999999999}]},{"osd":3,"last update":"Thu Oct 19 14:21:02 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.629,"15min":0.64900000000000002},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.66700000000000004,"5min":0.875,"15min":0.875},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.63800000000000001,"15min":0.65500000000000003},"min":{"1min":0.45400000000000001,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.92200000000000004,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.54500000000000004}]},{"osd":4,"last update":"Thu Oct 19 14:21:02 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.48899999999999999,"15min":0.48399999999999999},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.377},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.51000000000000001,"15min":0.50800000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.30399999999999999},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.85699999999999998},"last":0.40500000000000003}]},{"osd":5,"last update":"Thu Oct 19 14:21:02 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.50700000000000001,"15min":0.502},"min":{"1min":0.34799999999999998,"5min":0.32700000000000001,"15min":0.32500000000000001},"max":{"1min":0.79900000000000004,"5min":1.115,"15min":1.115},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.56599999999999995,"15min":0.54500000000000004},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.33900000000000002}]},{"osd":7,"last update":"Thu Oct 19 14:21:02 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.56000000000000005,"15min":0.54700000000000004},"min":{"1min":0.36799999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.90100000000000002},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.56699999999999995,"15min":0.55400000000000005},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.98399999999999999,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.38900000000000001}]}]},{"osd":5,"up_from":13,"seq":55834576010,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66072,"kb_used_data":2956,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305768,"statfs":{"total":96636764160,"available":96569106432,"internally_reserved":0,"allocated":3026944,"data_stored":1555686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:20:46 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.58499999999999996,"15min":0.61699999999999999},"min":{"1min":0.33200000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.78700000000000003,"5min":0.79700000000000004,"15min":0.86399999999999999},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.60999999999999999,"15min":0.63800000000000001},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.77400000000000002,"5min":0.85899999999999999,"15min":0.85999999999999999},"last":0.68700000000000006}]},{"osd":1,"last update":"Thu Oct 19 14:20:46 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.59499999999999997,"15min":0.628},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.749,"5min":0.80300000000000005,"15min":0.877},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.626,"15min":0.64000000000000001},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.81599999999999995,"5min":0.86699999999999999,"15min":0.88500000000000001},"last":0.59899999999999998}]},{"osd":2,"last update":"Thu Oct 19 14:20:46 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.59999999999999998,"15min":0.622},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.75900000000000001,"5min":0.83799999999999997,"15min":0.85099999999999998},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.63300000000000001,"15min":0.65500000000000003},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.81299999999999994,"5min":0.83299999999999996,"15min":0.90800000000000003},"last":0.629}]},{"osd":3,"last update":"Thu Oct 19 14:20:46 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.61399999999999999,"15min":0.64100000000000001},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.77100000000000002,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.64000000000000001,"15min":0.66700000000000004},"min":{"1min":0.436,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.73899999999999999,"5min":0.876,"15min":0.90900000000000003},"last":0.58699999999999997}]},{"osd":4,"last update":"Thu Oct 19 14:20:46 2023","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.48499999999999999,"15min":0.48299999999999998},"min":{"1min":0.35499999999999998,"5min":0.34999999999999998,"15min":0.32500000000000001},"max":{"1min":0.56200000000000006,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.437,"5min":0.499,"15min":0.501},"min":{"1min":0.308,"5min":0.308,"15min":0.28599999999999998},"max":{"1min":0.57299999999999995,"5min":0.77500000000000002,"15min":0.82099999999999995},"last":0.61299999999999999}]},{"osd":6,"last update":"Thu Oct 19 14:20:46 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.51000000000000001,"15min":0.497},"min":{"1min":0.32200000000000001,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.72999999999999998,"5min":0.80800000000000005,"15min":0.88800000000000001},"last":0.435},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.53300000000000003,"15min":0.53600000000000003},"min":{"1min":0.39900000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.70399999999999996,"5min":0.78000000000000003,"15min":0.88100000000000001},"last":0.54700000000000004}]},{"osd":7,"last update":"Thu Oct 19 14:20:46 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.50800000000000001,"15min":0.51500000000000001},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.625,"5min":0.749,"15min":0.82899999999999996},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.57399999999999995,"15min":0.56999999999999995},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.67300000000000004,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.64400000000000002}]}]},{"osd":4,"up_from":13,"seq":55834576010,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48016,"kb_used_data":2944,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323824,"statfs":{"total":96636764160,"available":96587595776,"internally_reserved":0,"allocated":3014656,"data_stored":1549521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.55500000000000005,"15min":0.57599999999999996},"min":{"1min":0.35499999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.80800000000000005,"5min":0.81799999999999995,"15min":0.879},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.58499999999999996,"15min":0.59799999999999998},"min":{"1min":0.499,"5min":0.31,"15min":0.31},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.873},"last":0.53100000000000003}]},{"osd":1,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.61499999999999999,"15min":0.64100000000000001},"min":{"1min":0.504,"5min":0.372,"15min":0.32600000000000001},"max":{"1min":0.80400000000000005,"5min":0.80700000000000005,"15min":0.92700000000000005},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.623,"15min":0.64400000000000002},"min":{"1min":0.443,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":1.0940000000000001},"last":0.55300000000000005}]},{"osd":2,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.63400000000000001,"15min":0.65000000000000002},"min":{"1min":0.46400000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.745},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.64800000000000002,"15min":0.66900000000000004},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":1.008},"last":0.70599999999999996}]},{"osd":3,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.61299999999999999,"15min":0.63400000000000001},"min":{"1min":0.47399999999999998,"5min":0.37,"15min":0.37},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.91900000000000004},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.66600000000000004,"15min":0.66700000000000004},"min":{"1min":0.54000000000000004,"5min":0.45400000000000001,"15min":0.437},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.76300000000000001}]},{"osd":5,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.48299999999999998,"15min":0.47099999999999997},"min":{"1min":0.34000000000000002,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.504,"15min":0.49199999999999999},"min":{"1min":0.34300000000000003,"5min":0.33700000000000002,"15min":0.29199999999999998},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.45100000000000001}]},{"osd":6,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.496,"15min":0.48599999999999999},"min":{"1min":0.34799999999999998,"5min":0.33900000000000002,"15min":0.28199999999999997},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.53700000000000003},"min":{"1min":0.39100000000000001,"5min":0.375,"15min":0.33300000000000002},"max":{"1min":0.81299999999999994,"5min":0.84799999999999998,"15min":1.2969999999999999},"last":0.501}]},{"osd":7,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.55100000000000005,"15min":0.54100000000000004},"min":{"1min":0.36499999999999999,"5min":0.35699999999999998,"15min":0.32600000000000001},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60499999999999998,"15min":0.59199999999999997},"min":{"1min":0.47099999999999997,"5min":0.32100000000000001,"15min":0.318},"max":{"1min":0.80400000000000005,"5min":0.88600000000000001,"15min":1.032},"last":0.51900000000000002}]}]},{"osd":3,"up_from":13,"seq":55834576009,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56356,"kb_used_data":2956,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315484,"statfs":{"total":96636764160,"available":96579055616,"internally_reserved":0,"allocated":3026944,"data_stored":1555686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54655432},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:20:49 2023","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.41099999999999998,"15min":0.41799999999999998},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.51800000000000002,"5min":0.61499999999999999,"15min":0.70699999999999996},"last":0.23899999999999999},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.40799999999999997,"15min":0.42799999999999999},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.752},"last":0.29799999999999999}]},{"osd":1,"last update":"Thu Oct 19 14:20:49 2023","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.42199999999999999,"15min":0.43099999999999999},"min":{"1min":0.25,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.54200000000000004,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.432,"5min":0.46400000000000002,"15min":0.47199999999999998},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.26400000000000001},"max":{"1min":0.67900000000000005,"5min":0.73099999999999998,"15min":0.81799999999999995},"last":0.35099999999999998}]},{"osd":2,"last update":"Thu Oct 19 14:20:49 2023","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.46899999999999997,"15min":0.48299999999999998},"min":{"1min":0.33100000000000002,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.625,"5min":0.64600000000000002,"15min":0.73099999999999998},"last":0.375},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.49199999999999999,"15min":0.504},"min":{"1min":0.31900000000000001,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.83699999999999997},"last":0.39300000000000002}]},{"osd":4,"last update":"Thu Oct 19 14:20:49 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.69199999999999995,"15min":0.70299999999999996},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.871,"5min":1.018,"15min":1.157},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.71099999999999997,"15min":0.72899999999999998},"min":{"1min":0.47399999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.97699999999999998,"5min":0.996,"15min":1.173},"last":0.56000000000000005}]},{"osd":5,"last update":"Thu Oct 19 14:20:49 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.68300000000000005,"15min":0.68500000000000005},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.41999999999999998},"max":{"1min":0.83399999999999996,"5min":0.97299999999999998,"15min":1.0149999999999999},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.67500000000000004,"15min":0.68600000000000005},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.84999999999999998,"5min":0.99399999999999999,"15min":1.105},"last":0.48899999999999999}]},{"osd":6,"last update":"Thu Oct 19 14:20:49 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.65900000000000003,"15min":0.68700000000000006},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.70499999999999996,"5min":1.0009999999999999,"15min":1.0669999999999999},"last":0.502},{"interface":"front","average":{"1min":0.621,"5min":0.69599999999999995,"15min":0.70999999999999996},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.85599999999999998,"5min":0.91500000000000004,"15min":1.079},"last":0.59399999999999997}]},{"osd":7,"last update":"Thu Oct 19 14:20:49 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.68799999999999994,"15min":0.69899999999999995},"min":{"1min":0.436,"5min":0.434,"15min":0.434},"max":{"1min":0.78800000000000003,"5min":0.96899999999999997,"15min":1.085},"last":0.621},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.70899999999999996,"15min":0.72899999999999998},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.79400000000000004,"5min":0.97199999999999998,"15min":1.1739999999999999},"last":0.60799999999999998}]}]},{"osd":2,"up_from":13,"seq":55834576011,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64656,"kb_used_data":2944,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94307184,"statfs":{"total":96636764160,"available":96570556416,"internally_reserved":0,"allocated":3014656,"data_stored":1549521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":63175110},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:20:36 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.38800000000000001,"15min":0.39700000000000002},"min":{"1min":0.254,"5min":0.218,"15min":0.218},"max":{"1min":0.55300000000000005,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.435,"15min":0.436},"min":{"1min":0.28199999999999997,"5min":0.249,"15min":0.223},"max":{"1min":0.54100000000000004,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.315}]},{"osd":1,"last update":"Thu Oct 19 14:20:36 2023","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.39600000000000002,"15min":0.41199999999999998},"min":{"1min":0.27500000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.47799999999999998,"5min":0.61699999999999999,"15min":0.65400000000000003},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.46700000000000003,"15min":0.46500000000000002},"min":{"1min":0.28499999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.60499999999999998,"5min":1.742,"15min":1.742},"last":0.39900000000000002}]},{"osd":3,"last update":"Thu Oct 19 14:20:36 2023","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.47099999999999997,"15min":0.47999999999999998},"min":{"1min":0.32100000000000001,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.63700000000000001,"5min":0.69299999999999995,"15min":0.74399999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.48799999999999999,"15min":0.50600000000000001},"min":{"1min":0.34499999999999997,"5min":0.32000000000000001,"15min":0.315},"max":{"1min":0.56999999999999995,"5min":0.67600000000000005,"15min":0.83599999999999997},"last":0.37}]},{"osd":4,"last update":"Thu Oct 19 14:20:36 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.65700000000000003,"15min":0.66900000000000004},"min":{"1min":0.47099999999999997,"5min":0.39600000000000002,"15min":0.39500000000000002},"max":{"1min":0.85599999999999998,"5min":0.93300000000000005,"15min":0.998},"last":0.44},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.67000000000000004,"15min":0.67900000000000005},"min":{"1min":0.46500000000000002,"5min":0.38900000000000001,"15min":0.35199999999999998},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.54500000000000004}]},{"osd":5,"last update":"Thu Oct 19 14:20:36 2023","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.69699999999999995,"15min":0.69599999999999995},"min":{"1min":0.52700000000000002,"5min":0.45100000000000001,"15min":0.40600000000000003},"max":{"1min":0.80000000000000004,"5min":1.4330000000000001,"15min":1.4330000000000001},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.68100000000000005,"15min":0.68700000000000006},"min":{"1min":0.52700000000000002,"5min":0.45100000000000001,"15min":0.39300000000000002},"max":{"1min":0.93300000000000005,"5min":1.595,"15min":1.595},"last":0.48399999999999999}]},{"osd":6,"last update":"Thu Oct 19 14:20:36 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.69299999999999995,"15min":0.70799999999999996},"min":{"1min":0.48799999999999999,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.996,"5min":0.996,"15min":1.0189999999999999},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.71899999999999997,"15min":0.72099999999999997},"min":{"1min":0.48899999999999999,"5min":0.433,"15min":0.40600000000000003},"max":{"1min":0.93899999999999995,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":0.53100000000000003}]},{"osd":7,"last update":"Thu Oct 19 14:20:36 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.70199999999999996,"15min":0.71999999999999997},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.41799999999999998},"max":{"1min":0.88700000000000001,"5min":1.0129999999999999,"15min":1.0289999999999999},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.73599999999999999,"15min":0.73099999999999998},"min":{"1min":0.51900000000000002,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.97299999999999998,"5min":1.53,"15min":1.53},"last":0.498}]}]},{"osd":1,"up_from":13,"seq":55834576010,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48036,"kb_used_data":2956,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323804,"statfs":{"total":96636764160,"available":96587575296,"internally_reserved":0,"allocated":3026944,"data_stored":1555686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.39600000000000002,"15min":0.40400000000000003},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23100000000000001},"max":{"1min":0.57299999999999995,"5min":0.61599999999999999,"15min":0.67400000000000004},"last":0.19900000000000001},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.435,"15min":0.45400000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.54200000000000004,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.32800000000000001}]},{"osd":2,"last update":"Thu Oct 19 14:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.43099999999999999,"15min":0.442},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.56899999999999995,"5min":0.63100000000000001,"15min":0.77500000000000002},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.48899999999999999,"15min":0.48999999999999999},"min":{"1min":0.32400000000000001,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.67300000000000004,"5min":0.68700000000000006,"15min":0.82799999999999996},"last":0.28100000000000003}]},{"osd":3,"last update":"Thu Oct 19 14:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.499,"15min":0.505},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.29499999999999998},"max":{"1min":0.64300000000000002,"5min":0.76000000000000001,"15min":0.78900000000000003},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.436,"15min":0.44600000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.26100000000000001},"max":{"1min":0.60799999999999998,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.30299999999999999}]},{"osd":4,"last update":"Thu Oct 19 14:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.66400000000000003,"15min":0.67100000000000004},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.92800000000000005,"5min":0.98299999999999998,"15min":1.042},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.67100000000000004,"15min":0.66900000000000004},"min":{"1min":0.44500000000000001,"5min":0.41299999999999998,"15min":0.32300000000000001},"max":{"1min":0.89400000000000002,"5min":0.93500000000000005,"15min":1.1659999999999999},"last":0.48499999999999999}]},{"osd":5,"last update":"Thu Oct 19 14:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.68799999999999994,"15min":0.68600000000000005},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.82099999999999995,"5min":1.002,"15min":1.3500000000000001},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.70599999999999996,"15min":0.70799999999999996},"min":{"1min":0.46700000000000003,"5min":0.44,"15min":0.40500000000000003},"max":{"1min":1.0009999999999999,"5min":1.0149999999999999,"15min":1.121},"last":0.442}]},{"osd":6,"last update":"Thu Oct 19 14:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.67800000000000005,"15min":0.68200000000000005},"min":{"1min":0.51000000000000001,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.95499999999999996,"5min":1.0129999999999999,"15min":1.2310000000000001},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.71899999999999997,"15min":0.71599999999999997},"min":{"1min":0.48199999999999998,"5min":0.434,"15min":0.434},"max":{"1min":0.93100000000000005,"5min":1.0109999999999999,"15min":1.274},"last":0.52600000000000002}]},{"osd":7,"last update":"Thu Oct 19 14:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.71899999999999997,"15min":0.72399999999999998},"min":{"1min":0.498,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.96799999999999997,"5min":0.99099999999999999,"15min":1.075},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.70099999999999996,"15min":0.70199999999999996},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.44400000000000001},"max":{"1min":0.91200000000000003,"5min":1.0169999999999999,"15min":1.117},"last":0.505}]}]},{"osd":0,"up_from":13,"seq":55834576011,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61084,"kb_used_data":3532,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310756,"statfs":{"total":96636764160,"available":96574214144,"internally_reserved":0,"allocated":3616768,"data_stored":2145435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.40699999999999997,"15min":0.41299999999999998},"min":{"1min":0.29599999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.56299999999999994,"5min":0.61499999999999999,"15min":0.624},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.441,"15min":0.46100000000000002},"min":{"1min":0.26800000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.56799999999999995,"5min":0.70899999999999996,"15min":0.81100000000000005},"last":0.27600000000000002}]},{"osd":2,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.496,"15min":0.48399999999999999},"min":{"1min":0.39300000000000002,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.71499999999999997,"5min":0.754,"15min":0.754},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.44800000000000001},"min":{"1min":0.32700000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.60199999999999998,"5min":0.63100000000000001,"15min":0.69099999999999995},"last":0.375}]},{"osd":3,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.432,"15min":0.44},"min":{"1min":0.26900000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.58799999999999997,"5min":0.61099999999999999,"15min":0.66800000000000004},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.501,"5min":0.5,"15min":0.51200000000000001},"min":{"1min":0.35199999999999998,"5min":0.254,"15min":0.254},"max":{"1min":0.78100000000000003,"5min":0.79800000000000004,"15min":0.84599999999999997},"last":0.46700000000000003}]},{"osd":4,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.64800000000000002,"15min":0.67700000000000005},"min":{"1min":0.48799999999999999,"5min":0.44600000000000001,"15min":0.44},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":1.1890000000000001},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.65800000000000003,"15min":0.68200000000000005},"min":{"1min":0.47499999999999998,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.751,"5min":0.95399999999999996,"15min":1.2150000000000001},"last":0.51600000000000001}]},{"osd":5,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.68999999999999995,"15min":0.72199999999999998},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":1.1579999999999999},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.69499999999999995,"15min":0.71999999999999997},"min":{"1min":0.57799999999999996,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.85399999999999998,"5min":0.98299999999999998,"15min":1.2310000000000001},"last":0.56299999999999994}]},{"osd":6,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.68700000000000006,"15min":0.70599999999999996},"min":{"1min":0.47799999999999998,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":1.1739999999999999},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.70399999999999996},"min":{"1min":0.54100000000000004,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.88900000000000001,"5min":0.97799999999999998,"15min":1.0189999999999999},"last":0.441}]},{"osd":7,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.68600000000000005,"15min":0.71599999999999997},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":1.1180000000000001},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.68200000000000005,"15min":0.70399999999999996},"min":{"1min":0.53600000000000003,"5min":0.46899999999999997,"15min":0.45100000000000001},"max":{"1min":0.90600000000000003,"5min":0.93799999999999994,"15min":1.0349999999999999},"last":0.625}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-19T14:21:13.442 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-19T14:21:13.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.583+0000 7fd9f6410640 1 -- 172.21.15.57:0/2916201754 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd9f00b0530 msgr2=0x7fd9f00b09a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:13.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.583+0000 7fd9f6410640 1 --2- 172.21.15.57:0/2916201754 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd9f00b0530 0x7fd9f00b09a0 secure :-1 s=READY pgs=2632 cs=0 l=1 rev1=1 crypto rx=0x7fd9f00cc7e0 tx=0x7fd9e802d150 comp rx=0 tx=0).stop 2023-10-19T14:21:13.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.583+0000 7fd9f6410640 1 -- 172.21.15.57:0/2916201754 shutdown_connections 2023-10-19T14:21:13.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.583+0000 7fd9f6410640 1 --2- 172.21.15.57:0/2916201754 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd9f00b0530 0x7fd9f00b09a0 unknown :-1 s=CLOSED pgs=2632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:13.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.583+0000 7fd9f6410640 1 --2- 172.21.15.57:0/2916201754 >> v2:172.21.15.179:3300/0 conn(0x7fd9f00cbd20 0x7fd9f0158300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:13.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.583+0000 7fd9f6410640 1 -- 172.21.15.57:0/2916201754 >> 172.21.15.57:0/2916201754 conn(0x7fd9f0069930 msgr2=0x7fd9f0069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:13.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.583+0000 7fd9f6410640 1 -- 172.21.15.57:0/2916201754 shutdown_connections 2023-10-19T14:21:13.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.583+0000 7fd9f6410640 1 -- 172.21.15.57:0/2916201754 wait complete. 2023-10-19T14:21:13.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.584+0000 7fd9f6410640 1 Processor -- start 2023-10-19T14:21:13.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.584+0000 7fd9f6410640 1 -- start start 2023-10-19T14:21:13.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.585+0000 7fd9f6410640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd9f00afa20 0x7fd9f01e1410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:13.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.585+0000 7fd9f6410640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd9f00b0530 0x7fd9f014a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:13.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.585+0000 7fd9f6410640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd9f00cbd20 0x7fd9f014a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:13.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.585+0000 7fd9f6410640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9f0158d40 con 0x7fd9f00b0530 2023-10-19T14:21:13.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.585+0000 7fd9f6410640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9f0158980 con 0x7fd9f00cbd20 2023-10-19T14:21:13.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.585+0000 7fd9f6410640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9f014ae90 con 0x7fd9f00afa20 2023-10-19T14:21:13.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.585+0000 7fd9ef7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd9f00b0530 0x7fd9f014a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:13.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.585+0000 7fd9ef7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd9f00b0530 0x7fd9f014a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51654/0 (socket says 172.21.15.57:51654) 2023-10-19T14:21:13.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.585+0000 7fd9ef7fe640 1 -- 172.21.15.57:0/3495359865 learned_addr learned my addr 172.21.15.57:0/3495359865 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:13.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.585+0000 7fd9effff640 1 --2- 172.21.15.57:0/3495359865 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd9f00afa20 0x7fd9f01e1410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:13.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.585+0000 7fd9f49ad640 1 --2- 172.21.15.57:0/3495359865 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd9f00cbd20 0x7fd9f014a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:13.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.585+0000 7fd9ef7fe640 1 -- 172.21.15.57:0/3495359865 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd9f00afa20 msgr2=0x7fd9f01e1410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:13.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.585+0000 7fd9ef7fe640 1 --2- 172.21.15.57:0/3495359865 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd9f00afa20 0x7fd9f01e1410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:13.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.585+0000 7fd9ef7fe640 1 -- 172.21.15.57:0/3495359865 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd9f00cbd20 msgr2=0x7fd9f014a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:13.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.585+0000 7fd9ef7fe640 1 --2- 172.21.15.57:0/3495359865 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd9f00cbd20 0x7fd9f014a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:13.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.585+0000 7fd9ef7fe640 1 -- 172.21.15.57:0/3495359865 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9e800aa10 con 0x7fd9f00b0530 2023-10-19T14:21:13.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.585+0000 7fd9f49ad640 1 --2- 172.21.15.57:0/3495359865 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd9f00cbd20 0x7fd9f014a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:21:13.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.585+0000 7fd9ef7fe640 1 --2- 172.21.15.57:0/3495359865 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd9f00b0530 0x7fd9f014a410 secure :-1 s=READY pgs=2633 cs=0 l=1 rev1=1 crypto rx=0x7fd9e000c430 tx=0x7fd9e000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:13.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.586+0000 7fd9ecff9640 1 -- 172.21.15.57:0/3495359865 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9e0011790 con 0x7fd9f00b0530 2023-10-19T14:21:13.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.586+0000 7fd9ecff9640 1 -- 172.21.15.57:0/3495359865 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fd9e00119e0 con 0x7fd9f00b0530 2023-10-19T14:21:13.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.586+0000 7fd9f6410640 1 -- 172.21.15.57:0/3495359865 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9f014b160 con 0x7fd9f00b0530 2023-10-19T14:21:13.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.586+0000 7fd9ecff9640 1 -- 172.21.15.57:0/3495359865 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9e0011df0 con 0x7fd9f00b0530 2023-10-19T14:21:13.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.586+0000 7fd9f6410640 1 -- 172.21.15.57:0/3495359865 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9f014b720 con 0x7fd9f00b0530 2023-10-19T14:21:13.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.587+0000 7fd9f6410640 1 -- 172.21.15.57:0/3495359865 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9f00690e0 con 0x7fd9f00b0530 2023-10-19T14:21:13.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.587+0000 7fd9ecff9640 1 -- 172.21.15.57:0/3495359865 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fd9e001c460 con 0x7fd9f00b0530 2023-10-19T14:21:13.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.587+0000 7fd9ecff9640 1 --2- 172.21.15.57:0/3495359865 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fd9dc06b080 0x7fd9dc06d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:13.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.587+0000 7fd9ecff9640 1 -- 172.21.15.57:0/3495359865 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(243..243 src has 1..243) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fd9e0089c80 con 0x7fd9f00b0530 2023-10-19T14:21:13.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.588+0000 7fd9effff640 1 --2- 172.21.15.57:0/3495359865 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fd9dc06b080 0x7fd9dc06d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:13.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.588+0000 7fd9effff640 1 --2- 172.21.15.57:0/3495359865 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fd9dc06b080 0x7fd9dc06d540 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd9e4005fd0 tx=0x7fd9e4005850 comp rx=0 tx=0).ready entity=mgr.9530 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:13.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.592+0000 7fd9ecff9640 1 -- 172.21.15.57:0/3495359865 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd9e00549d0 con 0x7fd9f00b0530 2023-10-19T14:21:13.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.764+0000 7fd9f6410640 1 -- 172.21.15.57:0/3495359865 --> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd9f00cc100 con 0x7fd9dc06b080 2023-10-19T14:21:13.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.771+0000 7fd9ecff9640 1 -- 172.21.15.57:0/3495359865 <== mgr.9530 v2:172.21.15.179:6832/3547895240 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+277694 (secure 0 0 0) 0x7fd9e400b7f0 con 0x7fd9dc06b080 2023-10-19T14:21:13.776 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:21:13.777 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2023-10-19T14:21:13.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.774+0000 7fd9f6410640 1 -- 172.21.15.57:0/3495359865 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fd9dc06b080 msgr2=0x7fd9dc06d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:13.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.774+0000 7fd9f6410640 1 --2- 172.21.15.57:0/3495359865 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fd9dc06b080 0x7fd9dc06d540 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd9e4005fd0 tx=0x7fd9e4005850 comp rx=0 tx=0).stop 2023-10-19T14:21:13.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.774+0000 7fd9f6410640 1 -- 172.21.15.57:0/3495359865 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd9f00b0530 msgr2=0x7fd9f014a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:13.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.774+0000 7fd9f6410640 1 --2- 172.21.15.57:0/3495359865 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd9f00b0530 0x7fd9f014a410 secure :-1 s=READY pgs=2633 cs=0 l=1 rev1=1 crypto rx=0x7fd9e000c430 tx=0x7fd9e000c8f0 comp rx=0 tx=0).stop 2023-10-19T14:21:13.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.774+0000 7fd9f6410640 1 -- 172.21.15.57:0/3495359865 shutdown_connections 2023-10-19T14:21:13.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.774+0000 7fd9f6410640 1 --2- 172.21.15.57:0/3495359865 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fd9dc06b080 0x7fd9dc06d540 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:13.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.774+0000 7fd9f6410640 1 --2- 172.21.15.57:0/3495359865 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd9f00cbd20 0x7fd9f014a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:13.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.774+0000 7fd9f6410640 1 --2- 172.21.15.57:0/3495359865 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd9f00b0530 0x7fd9f014a410 unknown :-1 s=CLOSED pgs=2633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:13.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.774+0000 7fd9f6410640 1 --2- 172.21.15.57:0/3495359865 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd9f00afa20 0x7fd9f01e1410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:13.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.774+0000 7fd9f6410640 1 -- 172.21.15.57:0/3495359865 >> 172.21.15.57:0/3495359865 conn(0x7fd9f0069930 msgr2=0x7fd9f0069d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:13.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.774+0000 7fd9f6410640 1 -- 172.21.15.57:0/3495359865 shutdown_connections 2023-10-19T14:21:13.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.774+0000 7fd9f6410640 1 -- 172.21.15.57:0/3495359865 wait complete. 2023-10-19T14:21:13.787 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":29,"stamp":"2023-10-19T14:21:12.172150+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2402,"num_read_kb":5036,"num_write":629,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3208,"ondisk_log_size":3208,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":449160,"kb_used_data":24768,"kb_used_omap":12,"kb_used_meta":424243,"kb_avail":754525560,"statfs":{"total":773094113280,"available":772634173440,"internally_reserved":0,"allocated":25362432,"data_stored":13607110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":434425401},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000817"},"pg_stats":[{"pgid":"32.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.1f","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.395395+0000","last_change":"2023-10-19T14:21:08.395395+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.395395+0000","last_undegraded":"2023-10-19T14:21:08.395395+0000","last_fullsized":"2023-10-19T14:21:08.395395+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.1d","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.393449+0000","last_change":"2023-10-19T14:21:08.393449+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.393449+0000","last_undegraded":"2023-10-19T14:21:08.393449+0000","last_fullsized":"2023-10-19T14:21:08.393449+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.1c","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.399714+0000","last_change":"2023-10-19T14:21:08.399714+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.399714+0000","last_undegraded":"2023-10-19T14:21:08.399714+0000","last_fullsized":"2023-10-19T14:21:08.399714+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.1b","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.397712+0000","last_change":"2023-10-19T14:21:08.397712+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.397712+0000","last_undegraded":"2023-10-19T14:21:08.397712+0000","last_fullsized":"2023-10-19T14:21:08.397712+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.1a","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.399541+0000","last_change":"2023-10-19T14:21:08.399541+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.399541+0000","last_undegraded":"2023-10-19T14:21:08.399541+0000","last_fullsized":"2023-10-19T14:21:08.399541+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.401761+0000","last_change":"2023-10-19T14:21:08.401761+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.401761+0000","last_undegraded":"2023-10-19T14:21:08.401761+0000","last_fullsized":"2023-10-19T14:21:08.401761+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.395206+0000","last_change":"2023-10-19T14:21:08.395206+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.395206+0000","last_undegraded":"2023-10-19T14:21:08.395206+0000","last_fullsized":"2023-10-19T14:21:08.395206+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.399532+0000","last_change":"2023-10-19T14:21:08.399532+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.399532+0000","last_undegraded":"2023-10-19T14:21:08.399532+0000","last_fullsized":"2023-10-19T14:21:08.399532+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.401377+0000","last_change":"2023-10-19T14:21:08.401377+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.401377+0000","last_undegraded":"2023-10-19T14:21:08.401377+0000","last_fullsized":"2023-10-19T14:21:08.401377+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.15","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.397203+0000","last_change":"2023-10-19T14:21:08.397203+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.397203+0000","last_undegraded":"2023-10-19T14:21:08.397203+0000","last_fullsized":"2023-10-19T14:21:08.397203+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.14","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.394976+0000","last_change":"2023-10-19T14:21:08.394976+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.394976+0000","last_undegraded":"2023-10-19T14:21:08.394976+0000","last_fullsized":"2023-10-19T14:21:08.394976+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.13","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.397244+0000","last_change":"2023-10-19T14:21:08.397244+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.397244+0000","last_undegraded":"2023-10-19T14:21:08.397244+0000","last_fullsized":"2023-10-19T14:21:08.397244+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.397963+0000","last_change":"2023-10-19T14:21:08.397963+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.397963+0000","last_undegraded":"2023-10-19T14:21:08.397963+0000","last_fullsized":"2023-10-19T14:21:08.397963+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.397350+0000","last_change":"2023-10-19T14:21:08.397350+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.397350+0000","last_undegraded":"2023-10-19T14:21:08.397350+0000","last_fullsized":"2023-10-19T14:21:08.397350+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.10","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.398284+0000","last_change":"2023-10-19T14:21:08.398284+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.398284+0000","last_undegraded":"2023-10-19T14:21:08.398284+0000","last_fullsized":"2023-10-19T14:21:08.398284+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.f","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.394918+0000","last_change":"2023-10-19T14:21:08.394918+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.394918+0000","last_undegraded":"2023-10-19T14:21:08.394918+0000","last_fullsized":"2023-10-19T14:21:08.394918+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.e","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.398215+0000","last_change":"2023-10-19T14:21:08.398215+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.398215+0000","last_undegraded":"2023-10-19T14:21:08.398215+0000","last_fullsized":"2023-10-19T14:21:08.398215+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.d","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.392973+0000","last_change":"2023-10-19T14:21:08.392973+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.392973+0000","last_undegraded":"2023-10-19T14:21:08.392973+0000","last_fullsized":"2023-10-19T14:21:08.392973+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.397335+0000","last_change":"2023-10-19T14:21:08.397335+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.397335+0000","last_undegraded":"2023-10-19T14:21:08.397335+0000","last_fullsized":"2023-10-19T14:21:08.397335+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.b","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.401221+0000","last_change":"2023-10-19T14:21:08.401221+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.401221+0000","last_undegraded":"2023-10-19T14:21:08.401221+0000","last_fullsized":"2023-10-19T14:21:08.401221+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.392818+0000","last_change":"2023-10-19T14:21:08.392818+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.392818+0000","last_undegraded":"2023-10-19T14:21:08.392818+0000","last_fullsized":"2023-10-19T14:21:08.392818+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.392864+0000","last_change":"2023-10-19T14:21:08.392864+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.392864+0000","last_undegraded":"2023-10-19T14:21:08.392864+0000","last_fullsized":"2023-10-19T14:21:08.392864+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.399409+0000","last_change":"2023-10-19T14:21:08.399409+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.399409+0000","last_undegraded":"2023-10-19T14:21:08.399409+0000","last_fullsized":"2023-10-19T14:21:08.399409+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.7","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.401244+0000","last_change":"2023-10-19T14:21:08.401244+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.401244+0000","last_undegraded":"2023-10-19T14:21:08.401244+0000","last_fullsized":"2023-10-19T14:21:08.401244+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.6","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.397194+0000","last_change":"2023-10-19T14:21:08.397194+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.397194+0000","last_undegraded":"2023-10-19T14:21:08.397194+0000","last_fullsized":"2023-10-19T14:21:08.397194+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.398883+0000","last_change":"2023-10-19T14:21:08.398883+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.398883+0000","last_undegraded":"2023-10-19T14:21:08.398883+0000","last_fullsized":"2023-10-19T14:21:08.398883+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.398251+0000","last_change":"2023-10-19T14:21:08.398251+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.398251+0000","last_undegraded":"2023-10-19T14:21:08.398251+0000","last_fullsized":"2023-10-19T14:21:08.398251+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.392809+0000","last_change":"2023-10-19T14:21:08.392809+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.392809+0000","last_undegraded":"2023-10-19T14:21:08.392809+0000","last_fullsized":"2023-10-19T14:21:08.392809+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.397367+0000","last_change":"2023-10-19T14:21:08.397367+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.397367+0000","last_undegraded":"2023-10-19T14:21:08.397367+0000","last_fullsized":"2023-10-19T14:21:08.397367+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.401262+0000","last_change":"2023-10-19T14:21:08.401262+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.401262+0000","last_undegraded":"2023-10-19T14:21:08.401262+0000","last_fullsized":"2023-10-19T14:21:08.401262+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.392705+0000","last_change":"2023-10-19T14:21:08.392705+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.392705+0000","last_undegraded":"2023-10-19T14:21:08.392705+0000","last_fullsized":"2023-10-19T14:21:08.392705+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.1e","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.392309+0000","last_change":"2023-10-19T14:21:08.392309+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.392309+0000","last_undegraded":"2023-10-19T14:21:08.392309+0000","last_fullsized":"2023-10-19T14:21:08.392309+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"229'3208","reported_seq":4177,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-19T14:21:06.428133+0000","last_change":"2023-10-19T12:44:08.636613+0000","last_active":"2023-10-19T14:21:06.428133+0000","last_peered":"2023-10-19T14:21:06.428133+0000","last_clean":"2023-10-19T14:21:06.428133+0000","last_became_active":"2023-10-19T12:44:08.635697+0000","last_became_peered":"2023-10-19T12:44:08.635697+0000","last_unstale":"2023-10-19T14:21:06.428133+0000","last_undegraded":"2023-10-19T14:21:06.428133+0000","last_fullsized":"2023-10-19T14:21:06.428133+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_clean_scrub_stamp":"2023-10-19T12:44:07.399738+0000","objects_scrubbed":0,"log_size":3208,"log_dups_size":0,"ondisk_log_size":3208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:15:23.685485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2402,"num_read_kb":5036,"num_write":629,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":32,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":31,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2402,"num_read_kb":5036,"num_write":629,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3208,"ondisk_log_size":3208,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576010,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66652,"kb_used_data":3536,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305188,"statfs":{"total":96636764160,"available":96568512512,"internally_reserved":0,"allocated":3620864,"data_stored":2146054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":64616907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:20:09 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.56999999999999995,"15min":0.58499999999999996},"min":{"1min":0.35799999999999998,"5min":0.34100000000000003,"15min":0.32200000000000001},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.86199999999999999},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.60199999999999998,"15min":0.60999999999999999},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.312},"max":{"1min":0.77600000000000002,"5min":0.81899999999999995,"15min":0.86599999999999999},"last":0.58699999999999997}]},{"osd":1,"last update":"Thu Oct 19 14:20:09 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.59799999999999998,"15min":0.59999999999999998},"min":{"1min":0.46500000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.76800000000000002,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.59499999999999997,"15min":0.59799999999999998},"min":{"1min":0.373,"5min":0.373,"15min":0.33500000000000002},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.45000000000000001}]},{"osd":2,"last update":"Thu Oct 19 14:20:09 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64900000000000002,"15min":0.64800000000000002},"min":{"1min":0.50900000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.85499999999999998},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.66600000000000004,"15min":0.66200000000000003},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.40100000000000002},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.66900000000000004}]},{"osd":3,"last update":"Thu Oct 19 14:20:09 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.63300000000000001,"15min":0.63700000000000001},"min":{"1min":0.51100000000000001,"5min":0.42099999999999999,"15min":0.36399999999999999},"max":{"1min":0.82699999999999996,"5min":0.84799999999999998,"15min":0.873},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.68799999999999994,"15min":0.68700000000000006},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":0.873,"5min":0.876,"15min":0.89600000000000002},"last":0.71199999999999997}]},{"osd":4,"last update":"Thu Oct 19 14:20:09 2023","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.50600000000000001,"15min":0.49299999999999999},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.62,"5min":0.78100000000000003,"15min":0.83799999999999997},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.51800000000000002,"15min":0.50600000000000001},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.71899999999999997,"5min":0.80200000000000005,"15min":0.81699999999999995},"last":0.67900000000000005}]},{"osd":5,"last update":"Thu Oct 19 14:20:09 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51400000000000001,"15min":0.50800000000000001},"min":{"1min":0.36799999999999999,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.65600000000000003,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.628},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.55600000000000005,"15min":0.54900000000000004},"min":{"1min":0.377,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.70999999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.72699999999999998}]},{"osd":6,"last update":"Thu Oct 19 14:20:09 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57699999999999996,"15min":0.57399999999999995},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.35999999999999999},"max":{"1min":0.73199999999999998,"5min":0.82999999999999996,"15min":0.85099999999999998},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.58099999999999996,"15min":0.56799999999999995},"min":{"1min":0.38800000000000001,"5min":0.35599999999999998,"15min":0.33500000000000002},"max":{"1min":0.78200000000000003,"5min":0.85199999999999998,"15min":0.85999999999999999},"last":0.75600000000000001}]}]},{"osd":6,"up_from":13,"seq":55834576010,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38288,"kb_used_data":2944,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333552,"statfs":{"total":96636764160,"available":96597557248,"internally_reserved":0,"allocated":3014656,"data_stored":1549521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:21:02 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.59199999999999997,"15min":0.61099999999999999},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.78400000000000003,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.56299999999999994,"15min":0.60599999999999998},"min":{"1min":0.36699999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.71799999999999997,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.59999999999999998}]},{"osd":1,"last update":"Thu Oct 19 14:21:02 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.60099999999999998,"15min":0.61799999999999999},"min":{"1min":0.373,"5min":0.373,"15min":0.35799999999999998},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":1.165},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.60199999999999998,"15min":0.626},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.77400000000000002,"5min":0.83699999999999997,"15min":0.91300000000000003},"last":0.51400000000000001}]},{"osd":2,"last update":"Thu Oct 19 14:21:02 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.57599999999999996,"15min":0.60199999999999998},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.69099999999999995,"5min":0.80100000000000005,"15min":0.873},"last":0.497},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.60599999999999998,"15min":0.624},"min":{"1min":0.40899999999999997,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.74199999999999999,"5min":0.83499999999999996,"15min":0.93100000000000005},"last":0.48499999999999999}]},{"osd":3,"last update":"Thu Oct 19 14:21:02 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.629,"15min":0.64900000000000002},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.66700000000000004,"5min":0.875,"15min":0.875},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.63800000000000001,"15min":0.65500000000000003},"min":{"1min":0.45400000000000001,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.92200000000000004,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.54500000000000004}]},{"osd":4,"last update":"Thu Oct 19 14:21:02 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.48899999999999999,"15min":0.48399999999999999},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.377},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.51000000000000001,"15min":0.50800000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.30399999999999999},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.85699999999999998},"last":0.40500000000000003}]},{"osd":5,"last update":"Thu Oct 19 14:21:02 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.50700000000000001,"15min":0.502},"min":{"1min":0.34799999999999998,"5min":0.32700000000000001,"15min":0.32500000000000001},"max":{"1min":0.79900000000000004,"5min":1.115,"15min":1.115},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.56599999999999995,"15min":0.54500000000000004},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.33900000000000002}]},{"osd":7,"last update":"Thu Oct 19 14:21:02 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.56000000000000005,"15min":0.54700000000000004},"min":{"1min":0.36799999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.90100000000000002},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.56699999999999995,"15min":0.55400000000000005},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.98399999999999999,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.38900000000000001}]}]},{"osd":5,"up_from":13,"seq":55834576010,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66072,"kb_used_data":2956,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305768,"statfs":{"total":96636764160,"available":96569106432,"internally_reserved":0,"allocated":3026944,"data_stored":1555686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:20:46 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.58499999999999996,"15min":0.61699999999999999},"min":{"1min":0.33200000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.78700000000000003,"5min":0.79700000000000004,"15min":0.86399999999999999},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.60999999999999999,"15min":0.63800000000000001},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.77400000000000002,"5min":0.85899999999999999,"15min":0.85999999999999999},"last":0.68700000000000006}]},{"osd":1,"last update":"Thu Oct 19 14:20:46 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.59499999999999997,"15min":0.628},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.749,"5min":0.80300000000000005,"15min":0.877},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.626,"15min":0.64000000000000001},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.81599999999999995,"5min":0.86699999999999999,"15min":0.88500000000000001},"last":0.59899999999999998}]},{"osd":2,"last update":"Thu Oct 19 14:20:46 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.59999999999999998,"15min":0.622},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.75900000000000001,"5min":0.83799999999999997,"15min":0.85099999999999998},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.63300000000000001,"15min":0.65500000000000003},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.81299999999999994,"5min":0.83299999999999996,"15min":0.90800000000000003},"last":0.629}]},{"osd":3,"last update":"Thu Oct 19 14:20:46 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.61399999999999999,"15min":0.64100000000000001},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.77100000000000002,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.64000000000000001,"15min":0.66700000000000004},"min":{"1min":0.436,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.73899999999999999,"5min":0.876,"15min":0.90900000000000003},"last":0.58699999999999997}]},{"osd":4,"last update":"Thu Oct 19 14:20:46 2023","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.48499999999999999,"15min":0.48299999999999998},"min":{"1min":0.35499999999999998,"5min":0.34999999999999998,"15min":0.32500000000000001},"max":{"1min":0.56200000000000006,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.437,"5min":0.499,"15min":0.501},"min":{"1min":0.308,"5min":0.308,"15min":0.28599999999999998},"max":{"1min":0.57299999999999995,"5min":0.77500000000000002,"15min":0.82099999999999995},"last":0.61299999999999999}]},{"osd":6,"last update":"Thu Oct 19 14:20:46 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.51000000000000001,"15min":0.497},"min":{"1min":0.32200000000000001,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.72999999999999998,"5min":0.80800000000000005,"15min":0.88800000000000001},"last":0.435},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.53300000000000003,"15min":0.53600000000000003},"min":{"1min":0.39900000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.70399999999999996,"5min":0.78000000000000003,"15min":0.88100000000000001},"last":0.54700000000000004}]},{"osd":7,"last update":"Thu Oct 19 14:20:46 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.50800000000000001,"15min":0.51500000000000001},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.625,"5min":0.749,"15min":0.82899999999999996},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.57399999999999995,"15min":0.56999999999999995},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.67300000000000004,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.64400000000000002}]}]},{"osd":4,"up_from":13,"seq":55834576010,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48016,"kb_used_data":2944,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323824,"statfs":{"total":96636764160,"available":96587595776,"internally_reserved":0,"allocated":3014656,"data_stored":1549521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.55500000000000005,"15min":0.57599999999999996},"min":{"1min":0.35499999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.80800000000000005,"5min":0.81799999999999995,"15min":0.879},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.58499999999999996,"15min":0.59799999999999998},"min":{"1min":0.499,"5min":0.31,"15min":0.31},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.873},"last":0.53100000000000003}]},{"osd":1,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.61499999999999999,"15min":0.64100000000000001},"min":{"1min":0.504,"5min":0.372,"15min":0.32600000000000001},"max":{"1min":0.80400000000000005,"5min":0.80700000000000005,"15min":0.92700000000000005},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.623,"15min":0.64400000000000002},"min":{"1min":0.443,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":1.0940000000000001},"last":0.55300000000000005}]},{"osd":2,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.63400000000000001,"15min":0.65000000000000002},"min":{"1min":0.46400000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.745},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.64800000000000002,"15min":0.66900000000000004},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":1.008},"last":0.70599999999999996}]},{"osd":3,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.61299999999999999,"15min":0.63400000000000001},"min":{"1min":0.47399999999999998,"5min":0.37,"15min":0.37},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.91900000000000004},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.66600000000000004,"15min":0.66700000000000004},"min":{"1min":0.54000000000000004,"5min":0.45400000000000001,"15min":0.437},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.76300000000000001}]},{"osd":5,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.48299999999999998,"15min":0.47099999999999997},"min":{"1min":0.34000000000000002,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.504,"15min":0.49199999999999999},"min":{"1min":0.34300000000000003,"5min":0.33700000000000002,"15min":0.29199999999999998},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.45100000000000001}]},{"osd":6,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.496,"15min":0.48599999999999999},"min":{"1min":0.34799999999999998,"5min":0.33900000000000002,"15min":0.28199999999999997},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.53700000000000003},"min":{"1min":0.39100000000000001,"5min":0.375,"15min":0.33300000000000002},"max":{"1min":0.81299999999999994,"5min":0.84799999999999998,"15min":1.2969999999999999},"last":0.501}]},{"osd":7,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.55100000000000005,"15min":0.54100000000000004},"min":{"1min":0.36499999999999999,"5min":0.35699999999999998,"15min":0.32600000000000001},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60499999999999998,"15min":0.59199999999999997},"min":{"1min":0.47099999999999997,"5min":0.32100000000000001,"15min":0.318},"max":{"1min":0.80400000000000005,"5min":0.88600000000000001,"15min":1.032},"last":0.51900000000000002}]}]},{"osd":3,"up_from":13,"seq":55834576009,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56356,"kb_used_data":2956,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315484,"statfs":{"total":96636764160,"available":96579055616,"internally_reserved":0,"allocated":3026944,"data_stored":1555686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54655432},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:20:49 2023","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.41099999999999998,"15min":0.41799999999999998},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.51800000000000002,"5min":0.61499999999999999,"15min":0.70699999999999996},"last":0.23899999999999999},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.40799999999999997,"15min":0.42799999999999999},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.752},"last":0.29799999999999999}]},{"osd":1,"last update":"Thu Oct 19 14:20:49 2023","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.42199999999999999,"15min":0.43099999999999999},"min":{"1min":0.25,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.54200000000000004,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.432,"5min":0.46400000000000002,"15min":0.47199999999999998},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.26400000000000001},"max":{"1min":0.67900000000000005,"5min":0.73099999999999998,"15min":0.81799999999999995},"last":0.35099999999999998}]},{"osd":2,"last update":"Thu Oct 19 14:20:49 2023","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.46899999999999997,"15min":0.48299999999999998},"min":{"1min":0.33100000000000002,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.625,"5min":0.64600000000000002,"15min":0.73099999999999998},"last":0.375},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.49199999999999999,"15min":0.504},"min":{"1min":0.31900000000000001,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.83699999999999997},"last":0.39300000000000002}]},{"osd":4,"last update":"Thu Oct 19 14:20:49 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.69199999999999995,"15min":0.70299999999999996},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.871,"5min":1.018,"15min":1.157},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.71099999999999997,"15min":0.72899999999999998},"min":{"1min":0.47399999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.97699999999999998,"5min":0.996,"15min":1.173},"last":0.56000000000000005}]},{"osd":5,"last update":"Thu Oct 19 14:20:49 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.68300000000000005,"15min":0.68500000000000005},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.41999999999999998},"max":{"1min":0.83399999999999996,"5min":0.97299999999999998,"15min":1.0149999999999999},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.67500000000000004,"15min":0.68600000000000005},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.84999999999999998,"5min":0.99399999999999999,"15min":1.105},"last":0.48899999999999999}]},{"osd":6,"last update":"Thu Oct 19 14:20:49 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.65900000000000003,"15min":0.68700000000000006},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.70499999999999996,"5min":1.0009999999999999,"15min":1.0669999999999999},"last":0.502},{"interface":"front","average":{"1min":0.621,"5min":0.69599999999999995,"15min":0.70999999999999996},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.85599999999999998,"5min":0.91500000000000004,"15min":1.079},"last":0.59399999999999997}]},{"osd":7,"last update":"Thu Oct 19 14:20:49 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.68799999999999994,"15min":0.69899999999999995},"min":{"1min":0.436,"5min":0.434,"15min":0.434},"max":{"1min":0.78800000000000003,"5min":0.96899999999999997,"15min":1.085},"last":0.621},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.70899999999999996,"15min":0.72899999999999998},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.79400000000000004,"5min":0.97199999999999998,"15min":1.1739999999999999},"last":0.60799999999999998}]}]},{"osd":2,"up_from":13,"seq":55834576011,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64656,"kb_used_data":2944,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94307184,"statfs":{"total":96636764160,"available":96570556416,"internally_reserved":0,"allocated":3014656,"data_stored":1549521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":63175110},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:20:36 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.38800000000000001,"15min":0.39700000000000002},"min":{"1min":0.254,"5min":0.218,"15min":0.218},"max":{"1min":0.55300000000000005,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.435,"15min":0.436},"min":{"1min":0.28199999999999997,"5min":0.249,"15min":0.223},"max":{"1min":0.54100000000000004,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.315}]},{"osd":1,"last update":"Thu Oct 19 14:20:36 2023","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.39600000000000002,"15min":0.41199999999999998},"min":{"1min":0.27500000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.47799999999999998,"5min":0.61699999999999999,"15min":0.65400000000000003},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.46700000000000003,"15min":0.46500000000000002},"min":{"1min":0.28499999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.60499999999999998,"5min":1.742,"15min":1.742},"last":0.39900000000000002}]},{"osd":3,"last update":"Thu Oct 19 14:20:36 2023","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.47099999999999997,"15min":0.47999999999999998},"min":{"1min":0.32100000000000001,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.63700000000000001,"5min":0.69299999999999995,"15min":0.74399999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.48799999999999999,"15min":0.50600000000000001},"min":{"1min":0.34499999999999997,"5min":0.32000000000000001,"15min":0.315},"max":{"1min":0.56999999999999995,"5min":0.67600000000000005,"15min":0.83599999999999997},"last":0.37}]},{"osd":4,"last update":"Thu Oct 19 14:20:36 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.65700000000000003,"15min":0.66900000000000004},"min":{"1min":0.47099999999999997,"5min":0.39600000000000002,"15min":0.39500000000000002},"max":{"1min":0.85599999999999998,"5min":0.93300000000000005,"15min":0.998},"last":0.44},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.67000000000000004,"15min":0.67900000000000005},"min":{"1min":0.46500000000000002,"5min":0.38900000000000001,"15min":0.35199999999999998},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.54500000000000004}]},{"osd":5,"last update":"Thu Oct 19 14:20:36 2023","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.69699999999999995,"15min":0.69599999999999995},"min":{"1min":0.52700000000000002,"5min":0.45100000000000001,"15min":0.40600000000000003},"max":{"1min":0.80000000000000004,"5min":1.4330000000000001,"15min":1.4330000000000001},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.68100000000000005,"15min":0.68700000000000006},"min":{"1min":0.52700000000000002,"5min":0.45100000000000001,"15min":0.39300000000000002},"max":{"1min":0.93300000000000005,"5min":1.595,"15min":1.595},"last":0.48399999999999999}]},{"osd":6,"last update":"Thu Oct 19 14:20:36 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.69299999999999995,"15min":0.70799999999999996},"min":{"1min":0.48799999999999999,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.996,"5min":0.996,"15min":1.0189999999999999},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.71899999999999997,"15min":0.72099999999999997},"min":{"1min":0.48899999999999999,"5min":0.433,"15min":0.40600000000000003},"max":{"1min":0.93899999999999995,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":0.53100000000000003}]},{"osd":7,"last update":"Thu Oct 19 14:20:36 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.70199999999999996,"15min":0.71999999999999997},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.41799999999999998},"max":{"1min":0.88700000000000001,"5min":1.0129999999999999,"15min":1.0289999999999999},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.73599999999999999,"15min":0.73099999999999998},"min":{"1min":0.51900000000000002,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.97299999999999998,"5min":1.53,"15min":1.53},"last":0.498}]}]},{"osd":1,"up_from":13,"seq":55834576010,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48036,"kb_used_data":2956,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323804,"statfs":{"total":96636764160,"available":96587575296,"internally_reserved":0,"allocated":3026944,"data_stored":1555686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.39600000000000002,"15min":0.40400000000000003},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23100000000000001},"max":{"1min":0.57299999999999995,"5min":0.61599999999999999,"15min":0.67400000000000004},"last":0.19900000000000001},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.435,"15min":0.45400000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.54200000000000004,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.32800000000000001}]},{"osd":2,"last update":"Thu Oct 19 14:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.43099999999999999,"15min":0.442},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.56899999999999995,"5min":0.63100000000000001,"15min":0.77500000000000002},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.48899999999999999,"15min":0.48999999999999999},"min":{"1min":0.32400000000000001,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.67300000000000004,"5min":0.68700000000000006,"15min":0.82799999999999996},"last":0.28100000000000003}]},{"osd":3,"last update":"Thu Oct 19 14:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.499,"15min":0.505},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.29499999999999998},"max":{"1min":0.64300000000000002,"5min":0.76000000000000001,"15min":0.78900000000000003},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.436,"15min":0.44600000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.26100000000000001},"max":{"1min":0.60799999999999998,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.30299999999999999}]},{"osd":4,"last update":"Thu Oct 19 14:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.66400000000000003,"15min":0.67100000000000004},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.92800000000000005,"5min":0.98299999999999998,"15min":1.042},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.67100000000000004,"15min":0.66900000000000004},"min":{"1min":0.44500000000000001,"5min":0.41299999999999998,"15min":0.32300000000000001},"max":{"1min":0.89400000000000002,"5min":0.93500000000000005,"15min":1.1659999999999999},"last":0.48499999999999999}]},{"osd":5,"last update":"Thu Oct 19 14:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.68799999999999994,"15min":0.68600000000000005},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.82099999999999995,"5min":1.002,"15min":1.3500000000000001},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.70599999999999996,"15min":0.70799999999999996},"min":{"1min":0.46700000000000003,"5min":0.44,"15min":0.40500000000000003},"max":{"1min":1.0009999999999999,"5min":1.0149999999999999,"15min":1.121},"last":0.442}]},{"osd":6,"last update":"Thu Oct 19 14:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.67800000000000005,"15min":0.68200000000000005},"min":{"1min":0.51000000000000001,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.95499999999999996,"5min":1.0129999999999999,"15min":1.2310000000000001},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.71899999999999997,"15min":0.71599999999999997},"min":{"1min":0.48199999999999998,"5min":0.434,"15min":0.434},"max":{"1min":0.93100000000000005,"5min":1.0109999999999999,"15min":1.274},"last":0.52600000000000002}]},{"osd":7,"last update":"Thu Oct 19 14:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.71899999999999997,"15min":0.72399999999999998},"min":{"1min":0.498,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.96799999999999997,"5min":0.99099999999999999,"15min":1.075},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.70099999999999996,"15min":0.70199999999999996},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.44400000000000001},"max":{"1min":0.91200000000000003,"5min":1.0169999999999999,"15min":1.117},"last":0.505}]}]},{"osd":0,"up_from":13,"seq":55834576011,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61084,"kb_used_data":3532,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310756,"statfs":{"total":96636764160,"available":96574214144,"internally_reserved":0,"allocated":3616768,"data_stored":2145435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.40699999999999997,"15min":0.41299999999999998},"min":{"1min":0.29599999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.56299999999999994,"5min":0.61499999999999999,"15min":0.624},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.441,"15min":0.46100000000000002},"min":{"1min":0.26800000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.56799999999999995,"5min":0.70899999999999996,"15min":0.81100000000000005},"last":0.27600000000000002}]},{"osd":2,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.496,"15min":0.48399999999999999},"min":{"1min":0.39300000000000002,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.71499999999999997,"5min":0.754,"15min":0.754},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.44800000000000001},"min":{"1min":0.32700000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.60199999999999998,"5min":0.63100000000000001,"15min":0.69099999999999995},"last":0.375}]},{"osd":3,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.432,"15min":0.44},"min":{"1min":0.26900000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.58799999999999997,"5min":0.61099999999999999,"15min":0.66800000000000004},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.501,"5min":0.5,"15min":0.51200000000000001},"min":{"1min":0.35199999999999998,"5min":0.254,"15min":0.254},"max":{"1min":0.78100000000000003,"5min":0.79800000000000004,"15min":0.84599999999999997},"last":0.46700000000000003}]},{"osd":4,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.64800000000000002,"15min":0.67700000000000005},"min":{"1min":0.48799999999999999,"5min":0.44600000000000001,"15min":0.44},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":1.1890000000000001},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.65800000000000003,"15min":0.68200000000000005},"min":{"1min":0.47499999999999998,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.751,"5min":0.95399999999999996,"15min":1.2150000000000001},"last":0.51600000000000001}]},{"osd":5,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.68999999999999995,"15min":0.72199999999999998},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":1.1579999999999999},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.69499999999999995,"15min":0.71999999999999997},"min":{"1min":0.57799999999999996,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.85399999999999998,"5min":0.98299999999999998,"15min":1.2310000000000001},"last":0.56299999999999994}]},{"osd":6,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.68700000000000006,"15min":0.70599999999999996},"min":{"1min":0.47799999999999998,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":1.1739999999999999},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.70399999999999996},"min":{"1min":0.54100000000000004,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.88900000000000001,"5min":0.97799999999999998,"15min":1.0189999999999999},"last":0.441}]},{"osd":7,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.68600000000000005,"15min":0.71599999999999997},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":1.1180000000000001},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.68200000000000005,"15min":0.70399999999999996},"min":{"1min":0.53600000000000003,"5min":0.46899999999999997,"15min":0.45100000000000001},"max":{"1min":0.90600000000000003,"5min":0.93799999999999994,"15min":1.0349999999999999},"last":0.625}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-19T14:21:13.793 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-19T14:21:13.937 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.933+0000 7eff8dbf0640 1 -- 172.21.15.57:0/167471813 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7eff8806ac70 msgr2=0x7eff8806ca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:13.937 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.933+0000 7eff8dbf0640 1 --2- 172.21.15.57:0/167471813 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7eff8806ac70 0x7eff8806ca90 secure :-1 s=READY pgs=2634 cs=0 l=1 rev1=1 crypto rx=0x7eff880662b0 tx=0x7eff8002c260 comp rx=0 tx=0).stop 2023-10-19T14:21:13.937 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.933+0000 7eff8dbf0640 1 -- 172.21.15.57:0/167471813 shutdown_connections 2023-10-19T14:21:13.937 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.934+0000 7eff8dbf0640 1 --2- 172.21.15.57:0/167471813 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7eff8806ac70 0x7eff8806ca90 unknown :-1 s=CLOSED pgs=2634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:13.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.934+0000 7eff8dbf0640 1 --2- 172.21.15.57:0/167471813 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7eff8815c1f0 0x7eff8806a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:13.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.934+0000 7eff8dbf0640 1 --2- 172.21.15.57:0/167471813 >> v2:172.21.15.179:3300/0 conn(0x7eff881579f0 0x7eff880b0e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:13.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.934+0000 7eff8dbf0640 1 -- 172.21.15.57:0/167471813 >> 172.21.15.57:0/167471813 conn(0x7eff8806fae0 msgr2=0x7eff8806fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:13.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.934+0000 7eff8dbf0640 1 -- 172.21.15.57:0/167471813 shutdown_connections 2023-10-19T14:21:13.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.934+0000 7eff8dbf0640 1 -- 172.21.15.57:0/167471813 wait complete. 2023-10-19T14:21:13.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.935+0000 7eff8dbf0640 1 Processor -- start 2023-10-19T14:21:13.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.935+0000 7eff8dbf0640 1 -- start start 2023-10-19T14:21:13.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.936+0000 7eff8dbf0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7eff8806ac70 0x7eff8814a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:13.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.936+0000 7eff8dbf0640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7eff881579f0 0x7eff88145b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:13.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.936+0000 7eff8dbf0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7eff8815c1f0 0x7eff8814fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:13.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.936+0000 7eff8dbf0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff8814af30 con 0x7eff8815c1f0 2023-10-19T14:21:13.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.936+0000 7eff8dbf0640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff881460c0 con 0x7eff8806ac70 2023-10-19T14:21:13.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.936+0000 7eff8dbf0640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff8814b7a0 con 0x7eff881579f0 2023-10-19T14:21:13.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.936+0000 7eff87fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7eff8815c1f0 0x7eff8814fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:13.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.936+0000 7eff87fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7eff8815c1f0 0x7eff8814fa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51680/0 (socket says 172.21.15.57:51680) 2023-10-19T14:21:13.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.936+0000 7eff87fff640 1 -- 172.21.15.57:0/1999767063 learned_addr learned my addr 172.21.15.57:0/1999767063 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:13.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.936+0000 7eff877fe640 1 --2- 172.21.15.57:0/1999767063 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7eff8806ac70 0x7eff8814a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:13.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.936+0000 7eff86ffd640 1 --2- 172.21.15.57:0/1999767063 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7eff881579f0 0x7eff88145b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:13.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.936+0000 7eff87fff640 1 -- 172.21.15.57:0/1999767063 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7eff881579f0 msgr2=0x7eff88145b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:13.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.936+0000 7eff87fff640 1 --2- 172.21.15.57:0/1999767063 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7eff881579f0 0x7eff88145b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:13.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.936+0000 7eff87fff640 1 -- 172.21.15.57:0/1999767063 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7eff8806ac70 msgr2=0x7eff8814a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:13.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.936+0000 7eff87fff640 1 --2- 172.21.15.57:0/1999767063 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7eff8806ac70 0x7eff8814a9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:13.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.936+0000 7eff87fff640 1 -- 172.21.15.57:0/1999767063 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff80009900 con 0x7eff8815c1f0 2023-10-19T14:21:13.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.936+0000 7eff877fe640 1 --2- 172.21.15.57:0/1999767063 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7eff8806ac70 0x7eff8814a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:21:13.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.936+0000 7eff87fff640 1 --2- 172.21.15.57:0/1999767063 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7eff8815c1f0 0x7eff8814fa70 secure :-1 s=READY pgs=2635 cs=0 l=1 rev1=1 crypto rx=0x7eff80009d60 tx=0x7eff800093f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:13.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.936+0000 7eff6ffff640 1 -- 172.21.15.57:0/1999767063 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff80004240 con 0x7eff8815c1f0 2023-10-19T14:21:13.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.936+0000 7eff6ffff640 1 -- 172.21.15.57:0/1999767063 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7eff80004490 con 0x7eff8815c1f0 2023-10-19T14:21:13.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.936+0000 7eff8dbf0640 1 -- 172.21.15.57:0/1999767063 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff8814ba80 con 0x7eff8815c1f0 2023-10-19T14:21:13.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.936+0000 7eff6ffff640 1 -- 172.21.15.57:0/1999767063 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff800048a0 con 0x7eff8815c1f0 2023-10-19T14:21:13.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.936+0000 7eff8dbf0640 1 -- 172.21.15.57:0/1999767063 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff8814c040 con 0x7eff8815c1f0 2023-10-19T14:21:13.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.938+0000 7eff8dbf0640 1 -- 172.21.15.57:0/1999767063 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff880690e0 con 0x7eff8815c1f0 2023-10-19T14:21:13.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.938+0000 7eff6ffff640 1 -- 172.21.15.57:0/1999767063 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7eff8000a040 con 0x7eff8815c1f0 2023-10-19T14:21:13.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.938+0000 7eff6ffff640 1 --2- 172.21.15.57:0/1999767063 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7eff7406b080 0x7eff7406d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:13.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.939+0000 7eff6ffff640 1 -- 172.21.15.57:0/1999767063 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(243..243 src has 1..243) v4 ==== 6057+0+0 (secure 0 0 0) 0x7eff800aca80 con 0x7eff8815c1f0 2023-10-19T14:21:13.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.939+0000 7eff877fe640 1 --2- 172.21.15.57:0/1999767063 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7eff7406b080 0x7eff7406d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:13.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.939+0000 7eff877fe640 1 --2- 172.21.15.57:0/1999767063 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7eff7406b080 0x7eff7406d540 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7eff88149a70 tx=0x7eff78005db0 comp rx=0 tx=0).ready entity=mgr.9530 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:13.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:13.943+0000 7eff6ffff640 1 -- 172.21.15.57:0/1999767063 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7eff800775d0 con 0x7eff8815c1f0 2023-10-19T14:21:14.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.117+0000 7eff8dbf0640 1 -- 172.21.15.57:0/1999767063 --> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7eff8806cfd0 con 0x7eff7406b080 2023-10-19T14:21:14.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.124+0000 7eff6ffff640 1 -- 172.21.15.57:0/1999767063 <== mgr.9530 v2:172.21.15.179:6832/3547895240 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+277694 (secure 0 0 0) 0x7eff7800b6b0 con 0x7eff7406b080 2023-10-19T14:21:14.129 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:21:14.131 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2023-10-19T14:21:14.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.127+0000 7eff8dbf0640 1 -- 172.21.15.57:0/1999767063 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7eff7406b080 msgr2=0x7eff7406d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:14.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.127+0000 7eff8dbf0640 1 --2- 172.21.15.57:0/1999767063 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7eff7406b080 0x7eff7406d540 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7eff88149a70 tx=0x7eff78005db0 comp rx=0 tx=0).stop 2023-10-19T14:21:14.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.127+0000 7eff8dbf0640 1 -- 172.21.15.57:0/1999767063 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7eff8815c1f0 msgr2=0x7eff8814fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:14.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.127+0000 7eff8dbf0640 1 --2- 172.21.15.57:0/1999767063 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7eff8815c1f0 0x7eff8814fa70 secure :-1 s=READY pgs=2635 cs=0 l=1 rev1=1 crypto rx=0x7eff80009d60 tx=0x7eff800093f0 comp rx=0 tx=0).stop 2023-10-19T14:21:14.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.127+0000 7eff8dbf0640 1 -- 172.21.15.57:0/1999767063 shutdown_connections 2023-10-19T14:21:14.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.127+0000 7eff8dbf0640 1 --2- 172.21.15.57:0/1999767063 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7eff7406b080 0x7eff7406d540 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:14.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.127+0000 7eff8dbf0640 1 --2- 172.21.15.57:0/1999767063 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7eff8815c1f0 0x7eff8814fa70 unknown :-1 s=CLOSED pgs=2635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:14.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.127+0000 7eff8dbf0640 1 --2- 172.21.15.57:0/1999767063 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7eff881579f0 0x7eff88145b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:14.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.127+0000 7eff8dbf0640 1 --2- 172.21.15.57:0/1999767063 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7eff8806ac70 0x7eff8814a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:14.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.127+0000 7eff8dbf0640 1 -- 172.21.15.57:0/1999767063 >> 172.21.15.57:0/1999767063 conn(0x7eff8806fae0 msgr2=0x7eff88150b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:14.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.127+0000 7eff8dbf0640 1 -- 172.21.15.57:0/1999767063 shutdown_connections 2023-10-19T14:21:14.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.127+0000 7eff8dbf0640 1 -- 172.21.15.57:0/1999767063 wait complete. 2023-10-19T14:21:14.141 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":29,"stamp":"2023-10-19T14:21:12.172150+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2402,"num_read_kb":5036,"num_write":629,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3208,"ondisk_log_size":3208,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":449160,"kb_used_data":24768,"kb_used_omap":12,"kb_used_meta":424243,"kb_avail":754525560,"statfs":{"total":773094113280,"available":772634173440,"internally_reserved":0,"allocated":25362432,"data_stored":13607110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":434425401},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000817"},"pg_stats":[{"pgid":"32.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.1f","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.395395+0000","last_change":"2023-10-19T14:21:08.395395+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.395395+0000","last_undegraded":"2023-10-19T14:21:08.395395+0000","last_fullsized":"2023-10-19T14:21:08.395395+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.1d","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.393449+0000","last_change":"2023-10-19T14:21:08.393449+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.393449+0000","last_undegraded":"2023-10-19T14:21:08.393449+0000","last_fullsized":"2023-10-19T14:21:08.393449+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.1c","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.399714+0000","last_change":"2023-10-19T14:21:08.399714+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.399714+0000","last_undegraded":"2023-10-19T14:21:08.399714+0000","last_fullsized":"2023-10-19T14:21:08.399714+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.1b","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.397712+0000","last_change":"2023-10-19T14:21:08.397712+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.397712+0000","last_undegraded":"2023-10-19T14:21:08.397712+0000","last_fullsized":"2023-10-19T14:21:08.397712+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.1a","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.399541+0000","last_change":"2023-10-19T14:21:08.399541+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.399541+0000","last_undegraded":"2023-10-19T14:21:08.399541+0000","last_fullsized":"2023-10-19T14:21:08.399541+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.401761+0000","last_change":"2023-10-19T14:21:08.401761+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.401761+0000","last_undegraded":"2023-10-19T14:21:08.401761+0000","last_fullsized":"2023-10-19T14:21:08.401761+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.395206+0000","last_change":"2023-10-19T14:21:08.395206+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.395206+0000","last_undegraded":"2023-10-19T14:21:08.395206+0000","last_fullsized":"2023-10-19T14:21:08.395206+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.399532+0000","last_change":"2023-10-19T14:21:08.399532+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.399532+0000","last_undegraded":"2023-10-19T14:21:08.399532+0000","last_fullsized":"2023-10-19T14:21:08.399532+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.401377+0000","last_change":"2023-10-19T14:21:08.401377+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.401377+0000","last_undegraded":"2023-10-19T14:21:08.401377+0000","last_fullsized":"2023-10-19T14:21:08.401377+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.15","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.397203+0000","last_change":"2023-10-19T14:21:08.397203+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.397203+0000","last_undegraded":"2023-10-19T14:21:08.397203+0000","last_fullsized":"2023-10-19T14:21:08.397203+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.14","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.394976+0000","last_change":"2023-10-19T14:21:08.394976+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.394976+0000","last_undegraded":"2023-10-19T14:21:08.394976+0000","last_fullsized":"2023-10-19T14:21:08.394976+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.13","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.397244+0000","last_change":"2023-10-19T14:21:08.397244+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.397244+0000","last_undegraded":"2023-10-19T14:21:08.397244+0000","last_fullsized":"2023-10-19T14:21:08.397244+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.397963+0000","last_change":"2023-10-19T14:21:08.397963+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.397963+0000","last_undegraded":"2023-10-19T14:21:08.397963+0000","last_fullsized":"2023-10-19T14:21:08.397963+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.397350+0000","last_change":"2023-10-19T14:21:08.397350+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.397350+0000","last_undegraded":"2023-10-19T14:21:08.397350+0000","last_fullsized":"2023-10-19T14:21:08.397350+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.10","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.398284+0000","last_change":"2023-10-19T14:21:08.398284+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.398284+0000","last_undegraded":"2023-10-19T14:21:08.398284+0000","last_fullsized":"2023-10-19T14:21:08.398284+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.f","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.394918+0000","last_change":"2023-10-19T14:21:08.394918+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.394918+0000","last_undegraded":"2023-10-19T14:21:08.394918+0000","last_fullsized":"2023-10-19T14:21:08.394918+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.e","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.398215+0000","last_change":"2023-10-19T14:21:08.398215+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.398215+0000","last_undegraded":"2023-10-19T14:21:08.398215+0000","last_fullsized":"2023-10-19T14:21:08.398215+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.d","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.392973+0000","last_change":"2023-10-19T14:21:08.392973+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.392973+0000","last_undegraded":"2023-10-19T14:21:08.392973+0000","last_fullsized":"2023-10-19T14:21:08.392973+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.397335+0000","last_change":"2023-10-19T14:21:08.397335+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.397335+0000","last_undegraded":"2023-10-19T14:21:08.397335+0000","last_fullsized":"2023-10-19T14:21:08.397335+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.b","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.401221+0000","last_change":"2023-10-19T14:21:08.401221+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.401221+0000","last_undegraded":"2023-10-19T14:21:08.401221+0000","last_fullsized":"2023-10-19T14:21:08.401221+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.392818+0000","last_change":"2023-10-19T14:21:08.392818+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.392818+0000","last_undegraded":"2023-10-19T14:21:08.392818+0000","last_fullsized":"2023-10-19T14:21:08.392818+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.392864+0000","last_change":"2023-10-19T14:21:08.392864+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.392864+0000","last_undegraded":"2023-10-19T14:21:08.392864+0000","last_fullsized":"2023-10-19T14:21:08.392864+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.399409+0000","last_change":"2023-10-19T14:21:08.399409+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.399409+0000","last_undegraded":"2023-10-19T14:21:08.399409+0000","last_fullsized":"2023-10-19T14:21:08.399409+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.7","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.401244+0000","last_change":"2023-10-19T14:21:08.401244+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.401244+0000","last_undegraded":"2023-10-19T14:21:08.401244+0000","last_fullsized":"2023-10-19T14:21:08.401244+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.6","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.397194+0000","last_change":"2023-10-19T14:21:08.397194+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.397194+0000","last_undegraded":"2023-10-19T14:21:08.397194+0000","last_fullsized":"2023-10-19T14:21:08.397194+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.398883+0000","last_change":"2023-10-19T14:21:08.398883+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.398883+0000","last_undegraded":"2023-10-19T14:21:08.398883+0000","last_fullsized":"2023-10-19T14:21:08.398883+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.398251+0000","last_change":"2023-10-19T14:21:08.398251+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.398251+0000","last_undegraded":"2023-10-19T14:21:08.398251+0000","last_fullsized":"2023-10-19T14:21:08.398251+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.392809+0000","last_change":"2023-10-19T14:21:08.392809+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.392809+0000","last_undegraded":"2023-10-19T14:21:08.392809+0000","last_fullsized":"2023-10-19T14:21:08.392809+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.397367+0000","last_change":"2023-10-19T14:21:08.397367+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.397367+0000","last_undegraded":"2023-10-19T14:21:08.397367+0000","last_fullsized":"2023-10-19T14:21:08.397367+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.401262+0000","last_change":"2023-10-19T14:21:08.401262+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.401262+0000","last_undegraded":"2023-10-19T14:21:08.401262+0000","last_fullsized":"2023-10-19T14:21:08.401262+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.392705+0000","last_change":"2023-10-19T14:21:08.392705+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.392705+0000","last_undegraded":"2023-10-19T14:21:08.392705+0000","last_fullsized":"2023-10-19T14:21:08.392705+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-19T14:21:09.388927+0000","last_change":"2023-10-19T14:21:09.388927+0000","last_active":"2023-10-19T14:21:09.388927+0000","last_peered":"2023-10-19T14:21:09.388927+0000","last_clean":"2023-10-19T14:21:09.388927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:09.388927+0000","last_undegraded":"2023-10-19T14:21:09.388927+0000","last_fullsized":"2023-10-19T14:21:09.388927+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.1e","version":"0'0","reported_seq":1,"reported_epoch":240,"state":"creating+peering","last_fresh":"2023-10-19T14:21:08.392309+0000","last_change":"2023-10-19T14:21:08.392309+0000","last_active":"2023-10-19T14:21:08.378327+0000","last_peered":"2023-10-19T14:21:08.378327+0000","last_clean":"2023-10-19T14:21:08.378327+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:21:08.392309+0000","last_undegraded":"2023-10-19T14:21:08.392309+0000","last_fullsized":"2023-10-19T14:21:08.392309+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"229'3208","reported_seq":4177,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-19T14:21:06.428133+0000","last_change":"2023-10-19T12:44:08.636613+0000","last_active":"2023-10-19T14:21:06.428133+0000","last_peered":"2023-10-19T14:21:06.428133+0000","last_clean":"2023-10-19T14:21:06.428133+0000","last_became_active":"2023-10-19T12:44:08.635697+0000","last_became_peered":"2023-10-19T12:44:08.635697+0000","last_unstale":"2023-10-19T14:21:06.428133+0000","last_undegraded":"2023-10-19T14:21:06.428133+0000","last_fullsized":"2023-10-19T14:21:06.428133+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_clean_scrub_stamp":"2023-10-19T12:44:07.399738+0000","objects_scrubbed":0,"log_size":3208,"log_dups_size":0,"ondisk_log_size":3208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:15:23.685485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2402,"num_read_kb":5036,"num_write":629,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":32,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":31,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2402,"num_read_kb":5036,"num_write":629,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3208,"ondisk_log_size":3208,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576010,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66652,"kb_used_data":3536,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305188,"statfs":{"total":96636764160,"available":96568512512,"internally_reserved":0,"allocated":3620864,"data_stored":2146054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":64616907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:20:09 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.56999999999999995,"15min":0.58499999999999996},"min":{"1min":0.35799999999999998,"5min":0.34100000000000003,"15min":0.32200000000000001},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.86199999999999999},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.60199999999999998,"15min":0.60999999999999999},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.312},"max":{"1min":0.77600000000000002,"5min":0.81899999999999995,"15min":0.86599999999999999},"last":0.58699999999999997}]},{"osd":1,"last update":"Thu Oct 19 14:20:09 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.59799999999999998,"15min":0.59999999999999998},"min":{"1min":0.46500000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.76800000000000002,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.59499999999999997,"15min":0.59799999999999998},"min":{"1min":0.373,"5min":0.373,"15min":0.33500000000000002},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.45000000000000001}]},{"osd":2,"last update":"Thu Oct 19 14:20:09 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64900000000000002,"15min":0.64800000000000002},"min":{"1min":0.50900000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.85499999999999998},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.66600000000000004,"15min":0.66200000000000003},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.40100000000000002},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.66900000000000004}]},{"osd":3,"last update":"Thu Oct 19 14:20:09 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.63300000000000001,"15min":0.63700000000000001},"min":{"1min":0.51100000000000001,"5min":0.42099999999999999,"15min":0.36399999999999999},"max":{"1min":0.82699999999999996,"5min":0.84799999999999998,"15min":0.873},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.68799999999999994,"15min":0.68700000000000006},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":0.873,"5min":0.876,"15min":0.89600000000000002},"last":0.71199999999999997}]},{"osd":4,"last update":"Thu Oct 19 14:20:09 2023","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.50600000000000001,"15min":0.49299999999999999},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.62,"5min":0.78100000000000003,"15min":0.83799999999999997},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.51800000000000002,"15min":0.50600000000000001},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.71899999999999997,"5min":0.80200000000000005,"15min":0.81699999999999995},"last":0.67900000000000005}]},{"osd":5,"last update":"Thu Oct 19 14:20:09 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51400000000000001,"15min":0.50800000000000001},"min":{"1min":0.36799999999999999,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.65600000000000003,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.628},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.55600000000000005,"15min":0.54900000000000004},"min":{"1min":0.377,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.70999999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.72699999999999998}]},{"osd":6,"last update":"Thu Oct 19 14:20:09 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57699999999999996,"15min":0.57399999999999995},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.35999999999999999},"max":{"1min":0.73199999999999998,"5min":0.82999999999999996,"15min":0.85099999999999998},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.58099999999999996,"15min":0.56799999999999995},"min":{"1min":0.38800000000000001,"5min":0.35599999999999998,"15min":0.33500000000000002},"max":{"1min":0.78200000000000003,"5min":0.85199999999999998,"15min":0.85999999999999999},"last":0.75600000000000001}]}]},{"osd":6,"up_from":13,"seq":55834576010,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38288,"kb_used_data":2944,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333552,"statfs":{"total":96636764160,"available":96597557248,"internally_reserved":0,"allocated":3014656,"data_stored":1549521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:21:02 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.59199999999999997,"15min":0.61099999999999999},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.78400000000000003,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.56299999999999994,"15min":0.60599999999999998},"min":{"1min":0.36699999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.71799999999999997,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.59999999999999998}]},{"osd":1,"last update":"Thu Oct 19 14:21:02 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.60099999999999998,"15min":0.61799999999999999},"min":{"1min":0.373,"5min":0.373,"15min":0.35799999999999998},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":1.165},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.60199999999999998,"15min":0.626},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.77400000000000002,"5min":0.83699999999999997,"15min":0.91300000000000003},"last":0.51400000000000001}]},{"osd":2,"last update":"Thu Oct 19 14:21:02 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.57599999999999996,"15min":0.60199999999999998},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.69099999999999995,"5min":0.80100000000000005,"15min":0.873},"last":0.497},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.60599999999999998,"15min":0.624},"min":{"1min":0.40899999999999997,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.74199999999999999,"5min":0.83499999999999996,"15min":0.93100000000000005},"last":0.48499999999999999}]},{"osd":3,"last update":"Thu Oct 19 14:21:02 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.629,"15min":0.64900000000000002},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.66700000000000004,"5min":0.875,"15min":0.875},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.63800000000000001,"15min":0.65500000000000003},"min":{"1min":0.45400000000000001,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.92200000000000004,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.54500000000000004}]},{"osd":4,"last update":"Thu Oct 19 14:21:02 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.48899999999999999,"15min":0.48399999999999999},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.377},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.51000000000000001,"15min":0.50800000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.30399999999999999},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.85699999999999998},"last":0.40500000000000003}]},{"osd":5,"last update":"Thu Oct 19 14:21:02 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.50700000000000001,"15min":0.502},"min":{"1min":0.34799999999999998,"5min":0.32700000000000001,"15min":0.32500000000000001},"max":{"1min":0.79900000000000004,"5min":1.115,"15min":1.115},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.56599999999999995,"15min":0.54500000000000004},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.33900000000000002}]},{"osd":7,"last update":"Thu Oct 19 14:21:02 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.56000000000000005,"15min":0.54700000000000004},"min":{"1min":0.36799999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.90100000000000002},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.56699999999999995,"15min":0.55400000000000005},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.98399999999999999,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.38900000000000001}]}]},{"osd":5,"up_from":13,"seq":55834576010,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66072,"kb_used_data":2956,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305768,"statfs":{"total":96636764160,"available":96569106432,"internally_reserved":0,"allocated":3026944,"data_stored":1555686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:20:46 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.58499999999999996,"15min":0.61699999999999999},"min":{"1min":0.33200000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.78700000000000003,"5min":0.79700000000000004,"15min":0.86399999999999999},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.60999999999999999,"15min":0.63800000000000001},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.77400000000000002,"5min":0.85899999999999999,"15min":0.85999999999999999},"last":0.68700000000000006}]},{"osd":1,"last update":"Thu Oct 19 14:20:46 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.59499999999999997,"15min":0.628},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.749,"5min":0.80300000000000005,"15min":0.877},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.626,"15min":0.64000000000000001},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.81599999999999995,"5min":0.86699999999999999,"15min":0.88500000000000001},"last":0.59899999999999998}]},{"osd":2,"last update":"Thu Oct 19 14:20:46 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.59999999999999998,"15min":0.622},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.75900000000000001,"5min":0.83799999999999997,"15min":0.85099999999999998},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.63300000000000001,"15min":0.65500000000000003},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.81299999999999994,"5min":0.83299999999999996,"15min":0.90800000000000003},"last":0.629}]},{"osd":3,"last update":"Thu Oct 19 14:20:46 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.61399999999999999,"15min":0.64100000000000001},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.77100000000000002,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.64000000000000001,"15min":0.66700000000000004},"min":{"1min":0.436,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.73899999999999999,"5min":0.876,"15min":0.90900000000000003},"last":0.58699999999999997}]},{"osd":4,"last update":"Thu Oct 19 14:20:46 2023","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.48499999999999999,"15min":0.48299999999999998},"min":{"1min":0.35499999999999998,"5min":0.34999999999999998,"15min":0.32500000000000001},"max":{"1min":0.56200000000000006,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.437,"5min":0.499,"15min":0.501},"min":{"1min":0.308,"5min":0.308,"15min":0.28599999999999998},"max":{"1min":0.57299999999999995,"5min":0.77500000000000002,"15min":0.82099999999999995},"last":0.61299999999999999}]},{"osd":6,"last update":"Thu Oct 19 14:20:46 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.51000000000000001,"15min":0.497},"min":{"1min":0.32200000000000001,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.72999999999999998,"5min":0.80800000000000005,"15min":0.88800000000000001},"last":0.435},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.53300000000000003,"15min":0.53600000000000003},"min":{"1min":0.39900000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.70399999999999996,"5min":0.78000000000000003,"15min":0.88100000000000001},"last":0.54700000000000004}]},{"osd":7,"last update":"Thu Oct 19 14:20:46 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.50800000000000001,"15min":0.51500000000000001},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.625,"5min":0.749,"15min":0.82899999999999996},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.57399999999999995,"15min":0.56999999999999995},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.67300000000000004,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.64400000000000002}]}]},{"osd":4,"up_from":13,"seq":55834576010,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48016,"kb_used_data":2944,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323824,"statfs":{"total":96636764160,"available":96587595776,"internally_reserved":0,"allocated":3014656,"data_stored":1549521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.55500000000000005,"15min":0.57599999999999996},"min":{"1min":0.35499999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.80800000000000005,"5min":0.81799999999999995,"15min":0.879},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.58499999999999996,"15min":0.59799999999999998},"min":{"1min":0.499,"5min":0.31,"15min":0.31},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.873},"last":0.53100000000000003}]},{"osd":1,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.61499999999999999,"15min":0.64100000000000001},"min":{"1min":0.504,"5min":0.372,"15min":0.32600000000000001},"max":{"1min":0.80400000000000005,"5min":0.80700000000000005,"15min":0.92700000000000005},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.623,"15min":0.64400000000000002},"min":{"1min":0.443,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":1.0940000000000001},"last":0.55300000000000005}]},{"osd":2,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.63400000000000001,"15min":0.65000000000000002},"min":{"1min":0.46400000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.745},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.64800000000000002,"15min":0.66900000000000004},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":1.008},"last":0.70599999999999996}]},{"osd":3,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.61299999999999999,"15min":0.63400000000000001},"min":{"1min":0.47399999999999998,"5min":0.37,"15min":0.37},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.91900000000000004},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.66600000000000004,"15min":0.66700000000000004},"min":{"1min":0.54000000000000004,"5min":0.45400000000000001,"15min":0.437},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.76300000000000001}]},{"osd":5,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.48299999999999998,"15min":0.47099999999999997},"min":{"1min":0.34000000000000002,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.504,"15min":0.49199999999999999},"min":{"1min":0.34300000000000003,"5min":0.33700000000000002,"15min":0.29199999999999998},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.45100000000000001}]},{"osd":6,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.496,"15min":0.48599999999999999},"min":{"1min":0.34799999999999998,"5min":0.33900000000000002,"15min":0.28199999999999997},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.53700000000000003},"min":{"1min":0.39100000000000001,"5min":0.375,"15min":0.33300000000000002},"max":{"1min":0.81299999999999994,"5min":0.84799999999999998,"15min":1.2969999999999999},"last":0.501}]},{"osd":7,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.55100000000000005,"15min":0.54100000000000004},"min":{"1min":0.36499999999999999,"5min":0.35699999999999998,"15min":0.32600000000000001},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60499999999999998,"15min":0.59199999999999997},"min":{"1min":0.47099999999999997,"5min":0.32100000000000001,"15min":0.318},"max":{"1min":0.80400000000000005,"5min":0.88600000000000001,"15min":1.032},"last":0.51900000000000002}]}]},{"osd":3,"up_from":13,"seq":55834576009,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56356,"kb_used_data":2956,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315484,"statfs":{"total":96636764160,"available":96579055616,"internally_reserved":0,"allocated":3026944,"data_stored":1555686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54655432},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:20:49 2023","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.41099999999999998,"15min":0.41799999999999998},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.51800000000000002,"5min":0.61499999999999999,"15min":0.70699999999999996},"last":0.23899999999999999},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.40799999999999997,"15min":0.42799999999999999},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.752},"last":0.29799999999999999}]},{"osd":1,"last update":"Thu Oct 19 14:20:49 2023","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.42199999999999999,"15min":0.43099999999999999},"min":{"1min":0.25,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.54200000000000004,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.432,"5min":0.46400000000000002,"15min":0.47199999999999998},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.26400000000000001},"max":{"1min":0.67900000000000005,"5min":0.73099999999999998,"15min":0.81799999999999995},"last":0.35099999999999998}]},{"osd":2,"last update":"Thu Oct 19 14:20:49 2023","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.46899999999999997,"15min":0.48299999999999998},"min":{"1min":0.33100000000000002,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.625,"5min":0.64600000000000002,"15min":0.73099999999999998},"last":0.375},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.49199999999999999,"15min":0.504},"min":{"1min":0.31900000000000001,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.83699999999999997},"last":0.39300000000000002}]},{"osd":4,"last update":"Thu Oct 19 14:20:49 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.69199999999999995,"15min":0.70299999999999996},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.871,"5min":1.018,"15min":1.157},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.71099999999999997,"15min":0.72899999999999998},"min":{"1min":0.47399999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.97699999999999998,"5min":0.996,"15min":1.173},"last":0.56000000000000005}]},{"osd":5,"last update":"Thu Oct 19 14:20:49 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.68300000000000005,"15min":0.68500000000000005},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.41999999999999998},"max":{"1min":0.83399999999999996,"5min":0.97299999999999998,"15min":1.0149999999999999},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.67500000000000004,"15min":0.68600000000000005},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.84999999999999998,"5min":0.99399999999999999,"15min":1.105},"last":0.48899999999999999}]},{"osd":6,"last update":"Thu Oct 19 14:20:49 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.65900000000000003,"15min":0.68700000000000006},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.70499999999999996,"5min":1.0009999999999999,"15min":1.0669999999999999},"last":0.502},{"interface":"front","average":{"1min":0.621,"5min":0.69599999999999995,"15min":0.70999999999999996},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.85599999999999998,"5min":0.91500000000000004,"15min":1.079},"last":0.59399999999999997}]},{"osd":7,"last update":"Thu Oct 19 14:20:49 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.68799999999999994,"15min":0.69899999999999995},"min":{"1min":0.436,"5min":0.434,"15min":0.434},"max":{"1min":0.78800000000000003,"5min":0.96899999999999997,"15min":1.085},"last":0.621},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.70899999999999996,"15min":0.72899999999999998},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.79400000000000004,"5min":0.97199999999999998,"15min":1.1739999999999999},"last":0.60799999999999998}]}]},{"osd":2,"up_from":13,"seq":55834576011,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64656,"kb_used_data":2944,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94307184,"statfs":{"total":96636764160,"available":96570556416,"internally_reserved":0,"allocated":3014656,"data_stored":1549521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":63175110},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:20:36 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.38800000000000001,"15min":0.39700000000000002},"min":{"1min":0.254,"5min":0.218,"15min":0.218},"max":{"1min":0.55300000000000005,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.435,"15min":0.436},"min":{"1min":0.28199999999999997,"5min":0.249,"15min":0.223},"max":{"1min":0.54100000000000004,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.315}]},{"osd":1,"last update":"Thu Oct 19 14:20:36 2023","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.39600000000000002,"15min":0.41199999999999998},"min":{"1min":0.27500000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.47799999999999998,"5min":0.61699999999999999,"15min":0.65400000000000003},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.46700000000000003,"15min":0.46500000000000002},"min":{"1min":0.28499999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.60499999999999998,"5min":1.742,"15min":1.742},"last":0.39900000000000002}]},{"osd":3,"last update":"Thu Oct 19 14:20:36 2023","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.47099999999999997,"15min":0.47999999999999998},"min":{"1min":0.32100000000000001,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.63700000000000001,"5min":0.69299999999999995,"15min":0.74399999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.48799999999999999,"15min":0.50600000000000001},"min":{"1min":0.34499999999999997,"5min":0.32000000000000001,"15min":0.315},"max":{"1min":0.56999999999999995,"5min":0.67600000000000005,"15min":0.83599999999999997},"last":0.37}]},{"osd":4,"last update":"Thu Oct 19 14:20:36 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.65700000000000003,"15min":0.66900000000000004},"min":{"1min":0.47099999999999997,"5min":0.39600000000000002,"15min":0.39500000000000002},"max":{"1min":0.85599999999999998,"5min":0.93300000000000005,"15min":0.998},"last":0.44},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.67000000000000004,"15min":0.67900000000000005},"min":{"1min":0.46500000000000002,"5min":0.38900000000000001,"15min":0.35199999999999998},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.54500000000000004}]},{"osd":5,"last update":"Thu Oct 19 14:20:36 2023","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.69699999999999995,"15min":0.69599999999999995},"min":{"1min":0.52700000000000002,"5min":0.45100000000000001,"15min":0.40600000000000003},"max":{"1min":0.80000000000000004,"5min":1.4330000000000001,"15min":1.4330000000000001},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.68100000000000005,"15min":0.68700000000000006},"min":{"1min":0.52700000000000002,"5min":0.45100000000000001,"15min":0.39300000000000002},"max":{"1min":0.93300000000000005,"5min":1.595,"15min":1.595},"last":0.48399999999999999}]},{"osd":6,"last update":"Thu Oct 19 14:20:36 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.69299999999999995,"15min":0.70799999999999996},"min":{"1min":0.48799999999999999,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.996,"5min":0.996,"15min":1.0189999999999999},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.71899999999999997,"15min":0.72099999999999997},"min":{"1min":0.48899999999999999,"5min":0.433,"15min":0.40600000000000003},"max":{"1min":0.93899999999999995,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":0.53100000000000003}]},{"osd":7,"last update":"Thu Oct 19 14:20:36 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.70199999999999996,"15min":0.71999999999999997},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.41799999999999998},"max":{"1min":0.88700000000000001,"5min":1.0129999999999999,"15min":1.0289999999999999},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.73599999999999999,"15min":0.73099999999999998},"min":{"1min":0.51900000000000002,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.97299999999999998,"5min":1.53,"15min":1.53},"last":0.498}]}]},{"osd":1,"up_from":13,"seq":55834576010,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48036,"kb_used_data":2956,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323804,"statfs":{"total":96636764160,"available":96587575296,"internally_reserved":0,"allocated":3026944,"data_stored":1555686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.39600000000000002,"15min":0.40400000000000003},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23100000000000001},"max":{"1min":0.57299999999999995,"5min":0.61599999999999999,"15min":0.67400000000000004},"last":0.19900000000000001},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.435,"15min":0.45400000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.54200000000000004,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.32800000000000001}]},{"osd":2,"last update":"Thu Oct 19 14:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.43099999999999999,"15min":0.442},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.56899999999999995,"5min":0.63100000000000001,"15min":0.77500000000000002},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.48899999999999999,"15min":0.48999999999999999},"min":{"1min":0.32400000000000001,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.67300000000000004,"5min":0.68700000000000006,"15min":0.82799999999999996},"last":0.28100000000000003}]},{"osd":3,"last update":"Thu Oct 19 14:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.499,"15min":0.505},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.29499999999999998},"max":{"1min":0.64300000000000002,"5min":0.76000000000000001,"15min":0.78900000000000003},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.436,"15min":0.44600000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.26100000000000001},"max":{"1min":0.60799999999999998,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.30299999999999999}]},{"osd":4,"last update":"Thu Oct 19 14:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.66400000000000003,"15min":0.67100000000000004},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.92800000000000005,"5min":0.98299999999999998,"15min":1.042},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.67100000000000004,"15min":0.66900000000000004},"min":{"1min":0.44500000000000001,"5min":0.41299999999999998,"15min":0.32300000000000001},"max":{"1min":0.89400000000000002,"5min":0.93500000000000005,"15min":1.1659999999999999},"last":0.48499999999999999}]},{"osd":5,"last update":"Thu Oct 19 14:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.68799999999999994,"15min":0.68600000000000005},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.82099999999999995,"5min":1.002,"15min":1.3500000000000001},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.70599999999999996,"15min":0.70799999999999996},"min":{"1min":0.46700000000000003,"5min":0.44,"15min":0.40500000000000003},"max":{"1min":1.0009999999999999,"5min":1.0149999999999999,"15min":1.121},"last":0.442}]},{"osd":6,"last update":"Thu Oct 19 14:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.67800000000000005,"15min":0.68200000000000005},"min":{"1min":0.51000000000000001,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.95499999999999996,"5min":1.0129999999999999,"15min":1.2310000000000001},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.71899999999999997,"15min":0.71599999999999997},"min":{"1min":0.48199999999999998,"5min":0.434,"15min":0.434},"max":{"1min":0.93100000000000005,"5min":1.0109999999999999,"15min":1.274},"last":0.52600000000000002}]},{"osd":7,"last update":"Thu Oct 19 14:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.71899999999999997,"15min":0.72399999999999998},"min":{"1min":0.498,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.96799999999999997,"5min":0.99099999999999999,"15min":1.075},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.70099999999999996,"15min":0.70199999999999996},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.44400000000000001},"max":{"1min":0.91200000000000003,"5min":1.0169999999999999,"15min":1.117},"last":0.505}]}]},{"osd":0,"up_from":13,"seq":55834576011,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61084,"kb_used_data":3532,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310756,"statfs":{"total":96636764160,"available":96574214144,"internally_reserved":0,"allocated":3616768,"data_stored":2145435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.40699999999999997,"15min":0.41299999999999998},"min":{"1min":0.29599999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.56299999999999994,"5min":0.61499999999999999,"15min":0.624},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.441,"15min":0.46100000000000002},"min":{"1min":0.26800000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.56799999999999995,"5min":0.70899999999999996,"15min":0.81100000000000005},"last":0.27600000000000002}]},{"osd":2,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.496,"15min":0.48399999999999999},"min":{"1min":0.39300000000000002,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.71499999999999997,"5min":0.754,"15min":0.754},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.44800000000000001},"min":{"1min":0.32700000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.60199999999999998,"5min":0.63100000000000001,"15min":0.69099999999999995},"last":0.375}]},{"osd":3,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.432,"15min":0.44},"min":{"1min":0.26900000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.58799999999999997,"5min":0.61099999999999999,"15min":0.66800000000000004},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.501,"5min":0.5,"15min":0.51200000000000001},"min":{"1min":0.35199999999999998,"5min":0.254,"15min":0.254},"max":{"1min":0.78100000000000003,"5min":0.79800000000000004,"15min":0.84599999999999997},"last":0.46700000000000003}]},{"osd":4,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.64800000000000002,"15min":0.67700000000000005},"min":{"1min":0.48799999999999999,"5min":0.44600000000000001,"15min":0.44},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":1.1890000000000001},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.65800000000000003,"15min":0.68200000000000005},"min":{"1min":0.47499999999999998,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.751,"5min":0.95399999999999996,"15min":1.2150000000000001},"last":0.51600000000000001}]},{"osd":5,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.68999999999999995,"15min":0.72199999999999998},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":1.1579999999999999},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.69499999999999995,"15min":0.71999999999999997},"min":{"1min":0.57799999999999996,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.85399999999999998,"5min":0.98299999999999998,"15min":1.2310000000000001},"last":0.56299999999999994}]},{"osd":6,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.68700000000000006,"15min":0.70599999999999996},"min":{"1min":0.47799999999999998,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":1.1739999999999999},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.70399999999999996},"min":{"1min":0.54100000000000004,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.88900000000000001,"5min":0.97799999999999998,"15min":1.0189999999999999},"last":0.441}]},{"osd":7,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.68600000000000005,"15min":0.71599999999999997},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":1.1180000000000001},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.68200000000000005,"15min":0.70399999999999996},"min":{"1min":0.53600000000000003,"5min":0.46899999999999997,"15min":0.45100000000000001},"max":{"1min":0.90600000000000003,"5min":0.93799999999999994,"15min":1.0349999999999999},"last":0.625}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-19T14:21:14.147 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.1f is not active+clean 2023-10-19T14:21:14.147 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.1f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T14:21:09.388927+0000', 'last_change': '2023-10-19T14:21:09.388927+0000', 'last_active': '2023-10-19T14:21:09.388927+0000', 'last_peered': '2023-10-19T14:21:09.388927+0000', 'last_clean': '2023-10-19T14:21:09.388927+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:09.388927+0000', 'last_undegraded': '2023-10-19T14:21:09.388927+0000', 'last_fullsized': '2023-10-19T14:21:09.388927+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T14:21:14.148 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.1e is not active+clean 2023-10-19T14:21:14.148 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.1e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T14:21:09.388927+0000', 'last_change': '2023-10-19T14:21:09.388927+0000', 'last_active': '2023-10-19T14:21:09.388927+0000', 'last_peered': '2023-10-19T14:21:09.388927+0000', 'last_clean': '2023-10-19T14:21:09.388927+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:09.388927+0000', 'last_undegraded': '2023-10-19T14:21:09.388927+0000', 'last_fullsized': '2023-10-19T14:21:09.388927+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T14:21:14.148 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.1d is not active+clean 2023-10-19T14:21:14.149 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.1d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T14:21:09.388927+0000', 'last_change': '2023-10-19T14:21:09.388927+0000', 'last_active': '2023-10-19T14:21:09.388927+0000', 'last_peered': '2023-10-19T14:21:09.388927+0000', 'last_clean': '2023-10-19T14:21:09.388927+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:09.388927+0000', 'last_undegraded': '2023-10-19T14:21:09.388927+0000', 'last_fullsized': '2023-10-19T14:21:09.388927+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T14:21:14.149 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.1c is not active+clean 2023-10-19T14:21:14.150 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.1c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T14:21:09.388927+0000', 'last_change': '2023-10-19T14:21:09.388927+0000', 'last_active': '2023-10-19T14:21:09.388927+0000', 'last_peered': '2023-10-19T14:21:09.388927+0000', 'last_clean': '2023-10-19T14:21:09.388927+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:09.388927+0000', 'last_undegraded': '2023-10-19T14:21:09.388927+0000', 'last_fullsized': '2023-10-19T14:21:09.388927+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T14:21:14.150 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.1b is not active+clean 2023-10-19T14:21:14.151 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.1b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T14:21:09.388927+0000', 'last_change': '2023-10-19T14:21:09.388927+0000', 'last_active': '2023-10-19T14:21:09.388927+0000', 'last_peered': '2023-10-19T14:21:09.388927+0000', 'last_clean': '2023-10-19T14:21:09.388927+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:09.388927+0000', 'last_undegraded': '2023-10-19T14:21:09.388927+0000', 'last_fullsized': '2023-10-19T14:21:09.388927+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T14:21:14.151 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.1a is not active+clean 2023-10-19T14:21:14.151 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.1a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T14:21:09.388927+0000', 'last_change': '2023-10-19T14:21:09.388927+0000', 'last_active': '2023-10-19T14:21:09.388927+0000', 'last_peered': '2023-10-19T14:21:09.388927+0000', 'last_clean': '2023-10-19T14:21:09.388927+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:09.388927+0000', 'last_undegraded': '2023-10-19T14:21:09.388927+0000', 'last_fullsized': '2023-10-19T14:21:09.388927+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T14:21:14.152 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.19 is not active+clean 2023-10-19T14:21:14.152 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.19', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T14:21:09.388927+0000', 'last_change': '2023-10-19T14:21:09.388927+0000', 'last_active': '2023-10-19T14:21:09.388927+0000', 'last_peered': '2023-10-19T14:21:09.388927+0000', 'last_clean': '2023-10-19T14:21:09.388927+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:09.388927+0000', 'last_undegraded': '2023-10-19T14:21:09.388927+0000', 'last_fullsized': '2023-10-19T14:21:09.388927+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T14:21:14.153 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.18 is not active+clean 2023-10-19T14:21:14.153 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.18', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T14:21:09.388927+0000', 'last_change': '2023-10-19T14:21:09.388927+0000', 'last_active': '2023-10-19T14:21:09.388927+0000', 'last_peered': '2023-10-19T14:21:09.388927+0000', 'last_clean': '2023-10-19T14:21:09.388927+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:09.388927+0000', 'last_undegraded': '2023-10-19T14:21:09.388927+0000', 'last_fullsized': '2023-10-19T14:21:09.388927+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T14:21:14.153 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.17 is not active+clean 2023-10-19T14:21:14.154 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.17', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T14:21:09.388927+0000', 'last_change': '2023-10-19T14:21:09.388927+0000', 'last_active': '2023-10-19T14:21:09.388927+0000', 'last_peered': '2023-10-19T14:21:09.388927+0000', 'last_clean': '2023-10-19T14:21:09.388927+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:09.388927+0000', 'last_undegraded': '2023-10-19T14:21:09.388927+0000', 'last_fullsized': '2023-10-19T14:21:09.388927+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T14:21:14.154 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.16 is not active+clean 2023-10-19T14:21:14.155 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.16', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T14:21:09.388927+0000', 'last_change': '2023-10-19T14:21:09.388927+0000', 'last_active': '2023-10-19T14:21:09.388927+0000', 'last_peered': '2023-10-19T14:21:09.388927+0000', 'last_clean': '2023-10-19T14:21:09.388927+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:09.388927+0000', 'last_undegraded': '2023-10-19T14:21:09.388927+0000', 'last_fullsized': '2023-10-19T14:21:09.388927+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T14:21:14.155 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.15 is not active+clean 2023-10-19T14:21:14.155 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.15', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T14:21:09.388927+0000', 'last_change': '2023-10-19T14:21:09.388927+0000', 'last_active': '2023-10-19T14:21:09.388927+0000', 'last_peered': '2023-10-19T14:21:09.388927+0000', 'last_clean': '2023-10-19T14:21:09.388927+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:09.388927+0000', 'last_undegraded': '2023-10-19T14:21:09.388927+0000', 'last_fullsized': '2023-10-19T14:21:09.388927+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T14:21:14.156 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.14 is not active+clean 2023-10-19T14:21:14.156 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.14', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T14:21:09.388927+0000', 'last_change': '2023-10-19T14:21:09.388927+0000', 'last_active': '2023-10-19T14:21:09.388927+0000', 'last_peered': '2023-10-19T14:21:09.388927+0000', 'last_clean': '2023-10-19T14:21:09.388927+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:09.388927+0000', 'last_undegraded': '2023-10-19T14:21:09.388927+0000', 'last_fullsized': '2023-10-19T14:21:09.388927+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T14:21:14.157 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.13 is not active+clean 2023-10-19T14:21:14.157 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.13', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T14:21:09.388927+0000', 'last_change': '2023-10-19T14:21:09.388927+0000', 'last_active': '2023-10-19T14:21:09.388927+0000', 'last_peered': '2023-10-19T14:21:09.388927+0000', 'last_clean': '2023-10-19T14:21:09.388927+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:09.388927+0000', 'last_undegraded': '2023-10-19T14:21:09.388927+0000', 'last_fullsized': '2023-10-19T14:21:09.388927+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T14:21:14.157 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.12 is not active+clean 2023-10-19T14:21:14.158 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.12', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T14:21:09.388927+0000', 'last_change': '2023-10-19T14:21:09.388927+0000', 'last_active': '2023-10-19T14:21:09.388927+0000', 'last_peered': '2023-10-19T14:21:09.388927+0000', 'last_clean': '2023-10-19T14:21:09.388927+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:09.388927+0000', 'last_undegraded': '2023-10-19T14:21:09.388927+0000', 'last_fullsized': '2023-10-19T14:21:09.388927+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T14:21:14.158 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.11 is not active+clean 2023-10-19T14:21:14.158 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.11', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T14:21:09.388927+0000', 'last_change': '2023-10-19T14:21:09.388927+0000', 'last_active': '2023-10-19T14:21:09.388927+0000', 'last_peered': '2023-10-19T14:21:09.388927+0000', 'last_clean': '2023-10-19T14:21:09.388927+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:09.388927+0000', 'last_undegraded': '2023-10-19T14:21:09.388927+0000', 'last_fullsized': '2023-10-19T14:21:09.388927+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T14:21:14.159 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.10 is not active+clean 2023-10-19T14:21:14.159 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.10', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T14:21:09.388927+0000', 'last_change': '2023-10-19T14:21:09.388927+0000', 'last_active': '2023-10-19T14:21:09.388927+0000', 'last_peered': '2023-10-19T14:21:09.388927+0000', 'last_clean': '2023-10-19T14:21:09.388927+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:09.388927+0000', 'last_undegraded': '2023-10-19T14:21:09.388927+0000', 'last_fullsized': '2023-10-19T14:21:09.388927+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T14:21:14.160 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.f is not active+clean 2023-10-19T14:21:14.160 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T14:21:09.388927+0000', 'last_change': '2023-10-19T14:21:09.388927+0000', 'last_active': '2023-10-19T14:21:09.388927+0000', 'last_peered': '2023-10-19T14:21:09.388927+0000', 'last_clean': '2023-10-19T14:21:09.388927+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:09.388927+0000', 'last_undegraded': '2023-10-19T14:21:09.388927+0000', 'last_fullsized': '2023-10-19T14:21:09.388927+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T14:21:14.160 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.e is not active+clean 2023-10-19T14:21:14.161 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T14:21:09.388927+0000', 'last_change': '2023-10-19T14:21:09.388927+0000', 'last_active': '2023-10-19T14:21:09.388927+0000', 'last_peered': '2023-10-19T14:21:09.388927+0000', 'last_clean': '2023-10-19T14:21:09.388927+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:09.388927+0000', 'last_undegraded': '2023-10-19T14:21:09.388927+0000', 'last_fullsized': '2023-10-19T14:21:09.388927+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T14:21:14.161 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.d is not active+clean 2023-10-19T14:21:14.162 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T14:21:09.388927+0000', 'last_change': '2023-10-19T14:21:09.388927+0000', 'last_active': '2023-10-19T14:21:09.388927+0000', 'last_peered': '2023-10-19T14:21:09.388927+0000', 'last_clean': '2023-10-19T14:21:09.388927+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:09.388927+0000', 'last_undegraded': '2023-10-19T14:21:09.388927+0000', 'last_fullsized': '2023-10-19T14:21:09.388927+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T14:21:14.162 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.c is not active+clean 2023-10-19T14:21:14.162 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T14:21:09.388927+0000', 'last_change': '2023-10-19T14:21:09.388927+0000', 'last_active': '2023-10-19T14:21:09.388927+0000', 'last_peered': '2023-10-19T14:21:09.388927+0000', 'last_clean': '2023-10-19T14:21:09.388927+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:09.388927+0000', 'last_undegraded': '2023-10-19T14:21:09.388927+0000', 'last_fullsized': '2023-10-19T14:21:09.388927+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T14:21:14.163 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.b is not active+clean 2023-10-19T14:21:14.163 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T14:21:09.388927+0000', 'last_change': '2023-10-19T14:21:09.388927+0000', 'last_active': '2023-10-19T14:21:09.388927+0000', 'last_peered': '2023-10-19T14:21:09.388927+0000', 'last_clean': '2023-10-19T14:21:09.388927+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:09.388927+0000', 'last_undegraded': '2023-10-19T14:21:09.388927+0000', 'last_fullsized': '2023-10-19T14:21:09.388927+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T14:21:14.164 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.a is not active+clean 2023-10-19T14:21:14.164 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T14:21:09.388927+0000', 'last_change': '2023-10-19T14:21:09.388927+0000', 'last_active': '2023-10-19T14:21:09.388927+0000', 'last_peered': '2023-10-19T14:21:09.388927+0000', 'last_clean': '2023-10-19T14:21:09.388927+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:09.388927+0000', 'last_undegraded': '2023-10-19T14:21:09.388927+0000', 'last_fullsized': '2023-10-19T14:21:09.388927+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T14:21:14.165 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.9 is not active+clean 2023-10-19T14:21:14.165 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.9', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T14:21:09.388927+0000', 'last_change': '2023-10-19T14:21:09.388927+0000', 'last_active': '2023-10-19T14:21:09.388927+0000', 'last_peered': '2023-10-19T14:21:09.388927+0000', 'last_clean': '2023-10-19T14:21:09.388927+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:09.388927+0000', 'last_undegraded': '2023-10-19T14:21:09.388927+0000', 'last_fullsized': '2023-10-19T14:21:09.388927+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T14:21:14.165 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.8 is not active+clean 2023-10-19T14:21:14.166 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.8', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T14:21:09.388927+0000', 'last_change': '2023-10-19T14:21:09.388927+0000', 'last_active': '2023-10-19T14:21:09.388927+0000', 'last_peered': '2023-10-19T14:21:09.388927+0000', 'last_clean': '2023-10-19T14:21:09.388927+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:09.388927+0000', 'last_undegraded': '2023-10-19T14:21:09.388927+0000', 'last_fullsized': '2023-10-19T14:21:09.388927+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T14:21:14.166 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.7 is not active+clean 2023-10-19T14:21:14.166 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.7', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T14:21:09.388927+0000', 'last_change': '2023-10-19T14:21:09.388927+0000', 'last_active': '2023-10-19T14:21:09.388927+0000', 'last_peered': '2023-10-19T14:21:09.388927+0000', 'last_clean': '2023-10-19T14:21:09.388927+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:09.388927+0000', 'last_undegraded': '2023-10-19T14:21:09.388927+0000', 'last_fullsized': '2023-10-19T14:21:09.388927+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T14:21:14.167 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.6 is not active+clean 2023-10-19T14:21:14.167 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.6', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T14:21:09.388927+0000', 'last_change': '2023-10-19T14:21:09.388927+0000', 'last_active': '2023-10-19T14:21:09.388927+0000', 'last_peered': '2023-10-19T14:21:09.388927+0000', 'last_clean': '2023-10-19T14:21:09.388927+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:09.388927+0000', 'last_undegraded': '2023-10-19T14:21:09.388927+0000', 'last_fullsized': '2023-10-19T14:21:09.388927+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T14:21:14.168 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.5 is not active+clean 2023-10-19T14:21:14.168 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.5', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T14:21:09.388927+0000', 'last_change': '2023-10-19T14:21:09.388927+0000', 'last_active': '2023-10-19T14:21:09.388927+0000', 'last_peered': '2023-10-19T14:21:09.388927+0000', 'last_clean': '2023-10-19T14:21:09.388927+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:09.388927+0000', 'last_undegraded': '2023-10-19T14:21:09.388927+0000', 'last_fullsized': '2023-10-19T14:21:09.388927+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T14:21:14.168 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.4 is not active+clean 2023-10-19T14:21:14.169 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.4', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T14:21:09.388927+0000', 'last_change': '2023-10-19T14:21:09.388927+0000', 'last_active': '2023-10-19T14:21:09.388927+0000', 'last_peered': '2023-10-19T14:21:09.388927+0000', 'last_clean': '2023-10-19T14:21:09.388927+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:09.388927+0000', 'last_undegraded': '2023-10-19T14:21:09.388927+0000', 'last_fullsized': '2023-10-19T14:21:09.388927+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T14:21:14.169 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.3 is not active+clean 2023-10-19T14:21:14.169 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.3', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T14:21:09.388927+0000', 'last_change': '2023-10-19T14:21:09.388927+0000', 'last_active': '2023-10-19T14:21:09.388927+0000', 'last_peered': '2023-10-19T14:21:09.388927+0000', 'last_clean': '2023-10-19T14:21:09.388927+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:09.388927+0000', 'last_undegraded': '2023-10-19T14:21:09.388927+0000', 'last_fullsized': '2023-10-19T14:21:09.388927+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T14:21:14.170 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.2 is not active+clean 2023-10-19T14:21:14.170 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.2', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T14:21:09.388927+0000', 'last_change': '2023-10-19T14:21:09.388927+0000', 'last_active': '2023-10-19T14:21:09.388927+0000', 'last_peered': '2023-10-19T14:21:09.388927+0000', 'last_clean': '2023-10-19T14:21:09.388927+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:09.388927+0000', 'last_undegraded': '2023-10-19T14:21:09.388927+0000', 'last_fullsized': '2023-10-19T14:21:09.388927+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T14:21:14.170 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.1 is not active+clean 2023-10-19T14:21:14.171 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.1', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T14:21:09.388927+0000', 'last_change': '2023-10-19T14:21:09.388927+0000', 'last_active': '2023-10-19T14:21:09.388927+0000', 'last_peered': '2023-10-19T14:21:09.388927+0000', 'last_clean': '2023-10-19T14:21:09.388927+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:09.388927+0000', 'last_undegraded': '2023-10-19T14:21:09.388927+0000', 'last_fullsized': '2023-10-19T14:21:09.388927+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T14:21:14.171 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.0 is not active+clean 2023-10-19T14:21:14.172 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.0', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T14:21:09.388927+0000', 'last_change': '2023-10-19T14:21:09.388927+0000', 'last_active': '2023-10-19T14:21:09.388927+0000', 'last_peered': '2023-10-19T14:21:09.388927+0000', 'last_clean': '2023-10-19T14:21:09.388927+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:09.388927+0000', 'last_undegraded': '2023-10-19T14:21:09.388927+0000', 'last_fullsized': '2023-10-19T14:21:09.388927+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T14:21:14.172 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.20 is not active+clean 2023-10-19T14:21:14.172 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.20', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T14:21:09.388927+0000', 'last_change': '2023-10-19T14:21:09.388927+0000', 'last_active': '2023-10-19T14:21:09.388927+0000', 'last_peered': '2023-10-19T14:21:09.388927+0000', 'last_clean': '2023-10-19T14:21:09.388927+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:09.388927+0000', 'last_undegraded': '2023-10-19T14:21:09.388927+0000', 'last_fullsized': '2023-10-19T14:21:09.388927+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T14:21:14.173 INFO:tasks.cephfs.filesystem.ceph_manager:PG 31.1f is not active+clean 2023-10-19T14:21:14.173 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '31.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 240, 'state': 'creating+peering', 'last_fresh': '2023-10-19T14:21:08.395395+0000', 'last_change': '2023-10-19T14:21:08.395395+0000', 'last_active': '2023-10-19T14:21:08.378327+0000', 'last_peered': '2023-10-19T14:21:08.378327+0000', 'last_clean': '2023-10-19T14:21:08.378327+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:08.395395+0000', 'last_undegraded': '2023-10-19T14:21:08.395395+0000', 'last_fullsized': '2023-10-19T14:21:08.395395+0000', 'mapping_epoch': 240, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 240, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-10-19T14:21:14.174 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.22 is not active+clean 2023-10-19T14:21:14.174 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.22', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T14:21:09.388927+0000', 'last_change': '2023-10-19T14:21:09.388927+0000', 'last_active': '2023-10-19T14:21:09.388927+0000', 'last_peered': '2023-10-19T14:21:09.388927+0000', 'last_clean': '2023-10-19T14:21:09.388927+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:09.388927+0000', 'last_undegraded': '2023-10-19T14:21:09.388927+0000', 'last_fullsized': '2023-10-19T14:21:09.388927+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T14:21:14.174 INFO:tasks.cephfs.filesystem.ceph_manager:PG 31.1d is not active+clean 2023-10-19T14:21:14.175 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '31.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 240, 'state': 'creating+peering', 'last_fresh': '2023-10-19T14:21:08.393449+0000', 'last_change': '2023-10-19T14:21:08.393449+0000', 'last_active': '2023-10-19T14:21:08.378327+0000', 'last_peered': '2023-10-19T14:21:08.378327+0000', 'last_clean': '2023-10-19T14:21:08.378327+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:08.393449+0000', 'last_undegraded': '2023-10-19T14:21:08.393449+0000', 'last_fullsized': '2023-10-19T14:21:08.393449+0000', 'mapping_epoch': 240, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 240, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-10-19T14:21:14.175 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.23 is not active+clean 2023-10-19T14:21:14.175 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.23', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T14:21:09.388927+0000', 'last_change': '2023-10-19T14:21:09.388927+0000', 'last_active': '2023-10-19T14:21:09.388927+0000', 'last_peered': '2023-10-19T14:21:09.388927+0000', 'last_clean': '2023-10-19T14:21:09.388927+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:09.388927+0000', 'last_undegraded': '2023-10-19T14:21:09.388927+0000', 'last_fullsized': '2023-10-19T14:21:09.388927+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T14:21:14.176 INFO:tasks.cephfs.filesystem.ceph_manager:PG 31.1c is not active+clean 2023-10-19T14:21:14.176 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '31.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 240, 'state': 'creating+peering', 'last_fresh': '2023-10-19T14:21:08.399714+0000', 'last_change': '2023-10-19T14:21:08.399714+0000', 'last_active': '2023-10-19T14:21:08.378327+0000', 'last_peered': '2023-10-19T14:21:08.378327+0000', 'last_clean': '2023-10-19T14:21:08.378327+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:08.399714+0000', 'last_undegraded': '2023-10-19T14:21:08.399714+0000', 'last_fullsized': '2023-10-19T14:21:08.399714+0000', 'mapping_epoch': 240, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 240, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-10-19T14:21:14.176 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.24 is not active+clean 2023-10-19T14:21:14.177 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.24', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T14:21:09.388927+0000', 'last_change': '2023-10-19T14:21:09.388927+0000', 'last_active': '2023-10-19T14:21:09.388927+0000', 'last_peered': '2023-10-19T14:21:09.388927+0000', 'last_clean': '2023-10-19T14:21:09.388927+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:09.388927+0000', 'last_undegraded': '2023-10-19T14:21:09.388927+0000', 'last_fullsized': '2023-10-19T14:21:09.388927+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T14:21:14.177 INFO:tasks.cephfs.filesystem.ceph_manager:PG 31.1b is not active+clean 2023-10-19T14:21:14.178 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '31.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 240, 'state': 'creating+peering', 'last_fresh': '2023-10-19T14:21:08.397712+0000', 'last_change': '2023-10-19T14:21:08.397712+0000', 'last_active': '2023-10-19T14:21:08.378327+0000', 'last_peered': '2023-10-19T14:21:08.378327+0000', 'last_clean': '2023-10-19T14:21:08.378327+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:08.397712+0000', 'last_undegraded': '2023-10-19T14:21:08.397712+0000', 'last_fullsized': '2023-10-19T14:21:08.397712+0000', 'mapping_epoch': 240, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 240, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-10-19T14:21:14.178 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.25 is not active+clean 2023-10-19T14:21:14.178 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.25', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T14:21:09.388927+0000', 'last_change': '2023-10-19T14:21:09.388927+0000', 'last_active': '2023-10-19T14:21:09.388927+0000', 'last_peered': '2023-10-19T14:21:09.388927+0000', 'last_clean': '2023-10-19T14:21:09.388927+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:09.388927+0000', 'last_undegraded': '2023-10-19T14:21:09.388927+0000', 'last_fullsized': '2023-10-19T14:21:09.388927+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T14:21:14.179 INFO:tasks.cephfs.filesystem.ceph_manager:PG 31.1a is not active+clean 2023-10-19T14:21:14.179 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '31.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 240, 'state': 'creating+peering', 'last_fresh': '2023-10-19T14:21:08.399541+0000', 'last_change': '2023-10-19T14:21:08.399541+0000', 'last_active': '2023-10-19T14:21:08.378327+0000', 'last_peered': '2023-10-19T14:21:08.378327+0000', 'last_clean': '2023-10-19T14:21:08.378327+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:08.399541+0000', 'last_undegraded': '2023-10-19T14:21:08.399541+0000', 'last_fullsized': '2023-10-19T14:21:08.399541+0000', 'mapping_epoch': 240, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 240, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-10-19T14:21:14.180 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.26 is not active+clean 2023-10-19T14:21:14.180 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.26', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T14:21:09.388927+0000', 'last_change': '2023-10-19T14:21:09.388927+0000', 'last_active': '2023-10-19T14:21:09.388927+0000', 'last_peered': '2023-10-19T14:21:09.388927+0000', 'last_clean': '2023-10-19T14:21:09.388927+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:09.388927+0000', 'last_undegraded': '2023-10-19T14:21:09.388927+0000', 'last_fullsized': '2023-10-19T14:21:09.388927+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T14:21:14.180 INFO:tasks.cephfs.filesystem.ceph_manager:PG 31.19 is not active+clean 2023-10-19T14:21:14.181 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '31.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 240, 'state': 'creating+peering', 'last_fresh': '2023-10-19T14:21:08.401761+0000', 'last_change': '2023-10-19T14:21:08.401761+0000', 'last_active': '2023-10-19T14:21:08.378327+0000', 'last_peered': '2023-10-19T14:21:08.378327+0000', 'last_clean': '2023-10-19T14:21:08.378327+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:08.401761+0000', 'last_undegraded': '2023-10-19T14:21:08.401761+0000', 'last_fullsized': '2023-10-19T14:21:08.401761+0000', 'mapping_epoch': 240, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 240, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 7], 'acting': [1, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-19T14:21:14.181 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.27 is not active+clean 2023-10-19T14:21:14.181 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.27', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T14:21:09.388927+0000', 'last_change': '2023-10-19T14:21:09.388927+0000', 'last_active': '2023-10-19T14:21:09.388927+0000', 'last_peered': '2023-10-19T14:21:09.388927+0000', 'last_clean': '2023-10-19T14:21:09.388927+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:09.388927+0000', 'last_undegraded': '2023-10-19T14:21:09.388927+0000', 'last_fullsized': '2023-10-19T14:21:09.388927+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T14:21:14.182 INFO:tasks.cephfs.filesystem.ceph_manager:PG 31.18 is not active+clean 2023-10-19T14:21:14.182 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '31.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 240, 'state': 'creating+peering', 'last_fresh': '2023-10-19T14:21:08.395206+0000', 'last_change': '2023-10-19T14:21:08.395206+0000', 'last_active': '2023-10-19T14:21:08.378327+0000', 'last_peered': '2023-10-19T14:21:08.378327+0000', 'last_clean': '2023-10-19T14:21:08.378327+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:08.395206+0000', 'last_undegraded': '2023-10-19T14:21:08.395206+0000', 'last_fullsized': '2023-10-19T14:21:08.395206+0000', 'mapping_epoch': 240, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 240, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-10-19T14:21:14.183 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.28 is not active+clean 2023-10-19T14:21:14.183 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.28', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T14:21:09.388927+0000', 'last_change': '2023-10-19T14:21:09.388927+0000', 'last_active': '2023-10-19T14:21:09.388927+0000', 'last_peered': '2023-10-19T14:21:09.388927+0000', 'last_clean': '2023-10-19T14:21:09.388927+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:09.388927+0000', 'last_undegraded': '2023-10-19T14:21:09.388927+0000', 'last_fullsized': '2023-10-19T14:21:09.388927+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T14:21:14.183 INFO:tasks.cephfs.filesystem.ceph_manager:PG 31.17 is not active+clean 2023-10-19T14:21:14.184 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '31.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 240, 'state': 'creating+peering', 'last_fresh': '2023-10-19T14:21:08.399532+0000', 'last_change': '2023-10-19T14:21:08.399532+0000', 'last_active': '2023-10-19T14:21:08.378327+0000', 'last_peered': '2023-10-19T14:21:08.378327+0000', 'last_clean': '2023-10-19T14:21:08.378327+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:08.399532+0000', 'last_undegraded': '2023-10-19T14:21:08.399532+0000', 'last_fullsized': '2023-10-19T14:21:08.399532+0000', 'mapping_epoch': 240, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 240, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-10-19T14:21:14.184 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.29 is not active+clean 2023-10-19T14:21:14.185 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.29', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T14:21:09.388927+0000', 'last_change': '2023-10-19T14:21:09.388927+0000', 'last_active': '2023-10-19T14:21:09.388927+0000', 'last_peered': '2023-10-19T14:21:09.388927+0000', 'last_clean': '2023-10-19T14:21:09.388927+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:09.388927+0000', 'last_undegraded': '2023-10-19T14:21:09.388927+0000', 'last_fullsized': '2023-10-19T14:21:09.388927+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T14:21:14.185 INFO:tasks.cephfs.filesystem.ceph_manager:PG 31.16 is not active+clean 2023-10-19T14:21:14.185 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '31.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 240, 'state': 'creating+peering', 'last_fresh': '2023-10-19T14:21:08.401377+0000', 'last_change': '2023-10-19T14:21:08.401377+0000', 'last_active': '2023-10-19T14:21:08.378327+0000', 'last_peered': '2023-10-19T14:21:08.378327+0000', 'last_clean': '2023-10-19T14:21:08.378327+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:08.401377+0000', 'last_undegraded': '2023-10-19T14:21:08.401377+0000', 'last_fullsized': '2023-10-19T14:21:08.401377+0000', 'mapping_epoch': 240, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 240, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-19T14:21:14.186 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.2a is not active+clean 2023-10-19T14:21:14.186 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.2a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T14:21:09.388927+0000', 'last_change': '2023-10-19T14:21:09.388927+0000', 'last_active': '2023-10-19T14:21:09.388927+0000', 'last_peered': '2023-10-19T14:21:09.388927+0000', 'last_clean': '2023-10-19T14:21:09.388927+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:09.388927+0000', 'last_undegraded': '2023-10-19T14:21:09.388927+0000', 'last_fullsized': '2023-10-19T14:21:09.388927+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T14:21:14.187 INFO:tasks.cephfs.filesystem.ceph_manager:PG 31.15 is not active+clean 2023-10-19T14:21:14.187 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '31.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 240, 'state': 'creating+peering', 'last_fresh': '2023-10-19T14:21:08.397203+0000', 'last_change': '2023-10-19T14:21:08.397203+0000', 'last_active': '2023-10-19T14:21:08.378327+0000', 'last_peered': '2023-10-19T14:21:08.378327+0000', 'last_clean': '2023-10-19T14:21:08.378327+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:08.397203+0000', 'last_undegraded': '2023-10-19T14:21:08.397203+0000', 'last_fullsized': '2023-10-19T14:21:08.397203+0000', 'mapping_epoch': 240, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 240, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-10-19T14:21:14.187 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.2b is not active+clean 2023-10-19T14:21:14.188 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.2b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T14:21:09.388927+0000', 'last_change': '2023-10-19T14:21:09.388927+0000', 'last_active': '2023-10-19T14:21:09.388927+0000', 'last_peered': '2023-10-19T14:21:09.388927+0000', 'last_clean': '2023-10-19T14:21:09.388927+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:09.388927+0000', 'last_undegraded': '2023-10-19T14:21:09.388927+0000', 'last_fullsized': '2023-10-19T14:21:09.388927+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T14:21:14.188 INFO:tasks.cephfs.filesystem.ceph_manager:PG 31.14 is not active+clean 2023-10-19T14:21:14.188 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '31.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 240, 'state': 'creating+peering', 'last_fresh': '2023-10-19T14:21:08.394976+0000', 'last_change': '2023-10-19T14:21:08.394976+0000', 'last_active': '2023-10-19T14:21:08.378327+0000', 'last_peered': '2023-10-19T14:21:08.378327+0000', 'last_clean': '2023-10-19T14:21:08.378327+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:08.394976+0000', 'last_undegraded': '2023-10-19T14:21:08.394976+0000', 'last_fullsized': '2023-10-19T14:21:08.394976+0000', 'mapping_epoch': 240, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 240, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-10-19T14:21:14.189 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.2c is not active+clean 2023-10-19T14:21:14.189 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.2c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T14:21:09.388927+0000', 'last_change': '2023-10-19T14:21:09.388927+0000', 'last_active': '2023-10-19T14:21:09.388927+0000', 'last_peered': '2023-10-19T14:21:09.388927+0000', 'last_clean': '2023-10-19T14:21:09.388927+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:09.388927+0000', 'last_undegraded': '2023-10-19T14:21:09.388927+0000', 'last_fullsized': '2023-10-19T14:21:09.388927+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T14:21:14.190 INFO:tasks.cephfs.filesystem.ceph_manager:PG 31.13 is not active+clean 2023-10-19T14:21:14.190 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '31.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 240, 'state': 'creating+peering', 'last_fresh': '2023-10-19T14:21:08.397244+0000', 'last_change': '2023-10-19T14:21:08.397244+0000', 'last_active': '2023-10-19T14:21:08.378327+0000', 'last_peered': '2023-10-19T14:21:08.378327+0000', 'last_clean': '2023-10-19T14:21:08.378327+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:08.397244+0000', 'last_undegraded': '2023-10-19T14:21:08.397244+0000', 'last_fullsized': '2023-10-19T14:21:08.397244+0000', 'mapping_epoch': 240, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 240, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-10-19T14:21:14.190 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.2d is not active+clean 2023-10-19T14:21:14.191 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.2d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T14:21:09.388927+0000', 'last_change': '2023-10-19T14:21:09.388927+0000', 'last_active': '2023-10-19T14:21:09.388927+0000', 'last_peered': '2023-10-19T14:21:09.388927+0000', 'last_clean': '2023-10-19T14:21:09.388927+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:09.388927+0000', 'last_undegraded': '2023-10-19T14:21:09.388927+0000', 'last_fullsized': '2023-10-19T14:21:09.388927+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T14:21:14.191 INFO:tasks.cephfs.filesystem.ceph_manager:PG 31.12 is not active+clean 2023-10-19T14:21:14.192 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '31.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 240, 'state': 'creating+peering', 'last_fresh': '2023-10-19T14:21:08.397963+0000', 'last_change': '2023-10-19T14:21:08.397963+0000', 'last_active': '2023-10-19T14:21:08.378327+0000', 'last_peered': '2023-10-19T14:21:08.378327+0000', 'last_clean': '2023-10-19T14:21:08.378327+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:08.397963+0000', 'last_undegraded': '2023-10-19T14:21:08.397963+0000', 'last_fullsized': '2023-10-19T14:21:08.397963+0000', 'mapping_epoch': 240, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 240, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-10-19T14:21:14.192 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.2e is not active+clean 2023-10-19T14:21:14.192 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.2e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T14:21:09.388927+0000', 'last_change': '2023-10-19T14:21:09.388927+0000', 'last_active': '2023-10-19T14:21:09.388927+0000', 'last_peered': '2023-10-19T14:21:09.388927+0000', 'last_clean': '2023-10-19T14:21:09.388927+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:09.388927+0000', 'last_undegraded': '2023-10-19T14:21:09.388927+0000', 'last_fullsized': '2023-10-19T14:21:09.388927+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T14:21:14.193 INFO:tasks.cephfs.filesystem.ceph_manager:PG 31.11 is not active+clean 2023-10-19T14:21:14.193 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '31.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 240, 'state': 'creating+peering', 'last_fresh': '2023-10-19T14:21:08.397350+0000', 'last_change': '2023-10-19T14:21:08.397350+0000', 'last_active': '2023-10-19T14:21:08.378327+0000', 'last_peered': '2023-10-19T14:21:08.378327+0000', 'last_clean': '2023-10-19T14:21:08.378327+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:08.397350+0000', 'last_undegraded': '2023-10-19T14:21:08.397350+0000', 'last_fullsized': '2023-10-19T14:21:08.397350+0000', 'mapping_epoch': 240, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 240, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-10-19T14:21:14.194 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.2f is not active+clean 2023-10-19T14:21:14.194 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.2f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T14:21:09.388927+0000', 'last_change': '2023-10-19T14:21:09.388927+0000', 'last_active': '2023-10-19T14:21:09.388927+0000', 'last_peered': '2023-10-19T14:21:09.388927+0000', 'last_clean': '2023-10-19T14:21:09.388927+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:09.388927+0000', 'last_undegraded': '2023-10-19T14:21:09.388927+0000', 'last_fullsized': '2023-10-19T14:21:09.388927+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T14:21:14.194 INFO:tasks.cephfs.filesystem.ceph_manager:PG 31.10 is not active+clean 2023-10-19T14:21:14.195 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '31.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 240, 'state': 'creating+peering', 'last_fresh': '2023-10-19T14:21:08.398284+0000', 'last_change': '2023-10-19T14:21:08.398284+0000', 'last_active': '2023-10-19T14:21:08.378327+0000', 'last_peered': '2023-10-19T14:21:08.378327+0000', 'last_clean': '2023-10-19T14:21:08.378327+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:08.398284+0000', 'last_undegraded': '2023-10-19T14:21:08.398284+0000', 'last_fullsized': '2023-10-19T14:21:08.398284+0000', 'mapping_epoch': 240, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 240, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-10-19T14:21:14.195 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.30 is not active+clean 2023-10-19T14:21:14.196 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.30', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T14:21:09.388927+0000', 'last_change': '2023-10-19T14:21:09.388927+0000', 'last_active': '2023-10-19T14:21:09.388927+0000', 'last_peered': '2023-10-19T14:21:09.388927+0000', 'last_clean': '2023-10-19T14:21:09.388927+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:09.388927+0000', 'last_undegraded': '2023-10-19T14:21:09.388927+0000', 'last_fullsized': '2023-10-19T14:21:09.388927+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T14:21:14.196 INFO:tasks.cephfs.filesystem.ceph_manager:PG 31.f is not active+clean 2023-10-19T14:21:14.196 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '31.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 240, 'state': 'creating+peering', 'last_fresh': '2023-10-19T14:21:08.394918+0000', 'last_change': '2023-10-19T14:21:08.394918+0000', 'last_active': '2023-10-19T14:21:08.378327+0000', 'last_peered': '2023-10-19T14:21:08.378327+0000', 'last_clean': '2023-10-19T14:21:08.378327+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:08.394918+0000', 'last_undegraded': '2023-10-19T14:21:08.394918+0000', 'last_fullsized': '2023-10-19T14:21:08.394918+0000', 'mapping_epoch': 240, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 240, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-10-19T14:21:14.197 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.31 is not active+clean 2023-10-19T14:21:14.197 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.31', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T14:21:09.388927+0000', 'last_change': '2023-10-19T14:21:09.388927+0000', 'last_active': '2023-10-19T14:21:09.388927+0000', 'last_peered': '2023-10-19T14:21:09.388927+0000', 'last_clean': '2023-10-19T14:21:09.388927+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:09.388927+0000', 'last_undegraded': '2023-10-19T14:21:09.388927+0000', 'last_fullsized': '2023-10-19T14:21:09.388927+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T14:21:14.197 INFO:tasks.cephfs.filesystem.ceph_manager:PG 31.e is not active+clean 2023-10-19T14:21:14.198 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '31.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 240, 'state': 'creating+peering', 'last_fresh': '2023-10-19T14:21:08.398215+0000', 'last_change': '2023-10-19T14:21:08.398215+0000', 'last_active': '2023-10-19T14:21:08.378327+0000', 'last_peered': '2023-10-19T14:21:08.378327+0000', 'last_clean': '2023-10-19T14:21:08.378327+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:08.398215+0000', 'last_undegraded': '2023-10-19T14:21:08.398215+0000', 'last_fullsized': '2023-10-19T14:21:08.398215+0000', 'mapping_epoch': 240, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 240, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-10-19T14:21:14.198 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.32 is not active+clean 2023-10-19T14:21:14.199 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.32', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T14:21:09.388927+0000', 'last_change': '2023-10-19T14:21:09.388927+0000', 'last_active': '2023-10-19T14:21:09.388927+0000', 'last_peered': '2023-10-19T14:21:09.388927+0000', 'last_clean': '2023-10-19T14:21:09.388927+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:09.388927+0000', 'last_undegraded': '2023-10-19T14:21:09.388927+0000', 'last_fullsized': '2023-10-19T14:21:09.388927+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T14:21:14.199 INFO:tasks.cephfs.filesystem.ceph_manager:PG 31.d is not active+clean 2023-10-19T14:21:14.199 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '31.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 240, 'state': 'creating+peering', 'last_fresh': '2023-10-19T14:21:08.392973+0000', 'last_change': '2023-10-19T14:21:08.392973+0000', 'last_active': '2023-10-19T14:21:08.378327+0000', 'last_peered': '2023-10-19T14:21:08.378327+0000', 'last_clean': '2023-10-19T14:21:08.378327+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:08.392973+0000', 'last_undegraded': '2023-10-19T14:21:08.392973+0000', 'last_fullsized': '2023-10-19T14:21:08.392973+0000', 'mapping_epoch': 240, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 240, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-10-19T14:21:14.200 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.33 is not active+clean 2023-10-19T14:21:14.200 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.33', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T14:21:09.388927+0000', 'last_change': '2023-10-19T14:21:09.388927+0000', 'last_active': '2023-10-19T14:21:09.388927+0000', 'last_peered': '2023-10-19T14:21:09.388927+0000', 'last_clean': '2023-10-19T14:21:09.388927+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:09.388927+0000', 'last_undegraded': '2023-10-19T14:21:09.388927+0000', 'last_fullsized': '2023-10-19T14:21:09.388927+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T14:21:14.201 INFO:tasks.cephfs.filesystem.ceph_manager:PG 31.c is not active+clean 2023-10-19T14:21:14.201 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '31.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 240, 'state': 'creating+peering', 'last_fresh': '2023-10-19T14:21:08.397335+0000', 'last_change': '2023-10-19T14:21:08.397335+0000', 'last_active': '2023-10-19T14:21:08.378327+0000', 'last_peered': '2023-10-19T14:21:08.378327+0000', 'last_clean': '2023-10-19T14:21:08.378327+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:08.397335+0000', 'last_undegraded': '2023-10-19T14:21:08.397335+0000', 'last_fullsized': '2023-10-19T14:21:08.397335+0000', 'mapping_epoch': 240, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 240, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-10-19T14:21:14.202 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.34 is not active+clean 2023-10-19T14:21:14.202 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.34', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T14:21:09.388927+0000', 'last_change': '2023-10-19T14:21:09.388927+0000', 'last_active': '2023-10-19T14:21:09.388927+0000', 'last_peered': '2023-10-19T14:21:09.388927+0000', 'last_clean': '2023-10-19T14:21:09.388927+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:09.388927+0000', 'last_undegraded': '2023-10-19T14:21:09.388927+0000', 'last_fullsized': '2023-10-19T14:21:09.388927+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T14:21:14.202 INFO:tasks.cephfs.filesystem.ceph_manager:PG 31.b is not active+clean 2023-10-19T14:21:14.203 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '31.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 240, 'state': 'creating+peering', 'last_fresh': '2023-10-19T14:21:08.401221+0000', 'last_change': '2023-10-19T14:21:08.401221+0000', 'last_active': '2023-10-19T14:21:08.378327+0000', 'last_peered': '2023-10-19T14:21:08.378327+0000', 'last_clean': '2023-10-19T14:21:08.378327+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:08.401221+0000', 'last_undegraded': '2023-10-19T14:21:08.401221+0000', 'last_fullsized': '2023-10-19T14:21:08.401221+0000', 'mapping_epoch': 240, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 240, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-19T14:21:14.203 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.35 is not active+clean 2023-10-19T14:21:14.204 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.35', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T14:21:09.388927+0000', 'last_change': '2023-10-19T14:21:09.388927+0000', 'last_active': '2023-10-19T14:21:09.388927+0000', 'last_peered': '2023-10-19T14:21:09.388927+0000', 'last_clean': '2023-10-19T14:21:09.388927+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:09.388927+0000', 'last_undegraded': '2023-10-19T14:21:09.388927+0000', 'last_fullsized': '2023-10-19T14:21:09.388927+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T14:21:14.204 INFO:tasks.cephfs.filesystem.ceph_manager:PG 31.a is not active+clean 2023-10-19T14:21:14.204 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '31.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 240, 'state': 'creating+peering', 'last_fresh': '2023-10-19T14:21:08.392818+0000', 'last_change': '2023-10-19T14:21:08.392818+0000', 'last_active': '2023-10-19T14:21:08.378327+0000', 'last_peered': '2023-10-19T14:21:08.378327+0000', 'last_clean': '2023-10-19T14:21:08.378327+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:08.392818+0000', 'last_undegraded': '2023-10-19T14:21:08.392818+0000', 'last_fullsized': '2023-10-19T14:21:08.392818+0000', 'mapping_epoch': 240, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 240, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-10-19T14:21:14.205 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.36 is not active+clean 2023-10-19T14:21:14.205 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.36', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T14:21:09.388927+0000', 'last_change': '2023-10-19T14:21:09.388927+0000', 'last_active': '2023-10-19T14:21:09.388927+0000', 'last_peered': '2023-10-19T14:21:09.388927+0000', 'last_clean': '2023-10-19T14:21:09.388927+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:09.388927+0000', 'last_undegraded': '2023-10-19T14:21:09.388927+0000', 'last_fullsized': '2023-10-19T14:21:09.388927+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T14:21:14.206 INFO:tasks.cephfs.filesystem.ceph_manager:PG 31.9 is not active+clean 2023-10-19T14:21:14.206 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '31.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 240, 'state': 'creating+peering', 'last_fresh': '2023-10-19T14:21:08.392864+0000', 'last_change': '2023-10-19T14:21:08.392864+0000', 'last_active': '2023-10-19T14:21:08.378327+0000', 'last_peered': '2023-10-19T14:21:08.378327+0000', 'last_clean': '2023-10-19T14:21:08.378327+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:08.392864+0000', 'last_undegraded': '2023-10-19T14:21:08.392864+0000', 'last_fullsized': '2023-10-19T14:21:08.392864+0000', 'mapping_epoch': 240, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 240, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-10-19T14:21:14.206 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.37 is not active+clean 2023-10-19T14:21:14.207 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.37', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T14:21:09.388927+0000', 'last_change': '2023-10-19T14:21:09.388927+0000', 'last_active': '2023-10-19T14:21:09.388927+0000', 'last_peered': '2023-10-19T14:21:09.388927+0000', 'last_clean': '2023-10-19T14:21:09.388927+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:09.388927+0000', 'last_undegraded': '2023-10-19T14:21:09.388927+0000', 'last_fullsized': '2023-10-19T14:21:09.388927+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T14:21:14.207 INFO:tasks.cephfs.filesystem.ceph_manager:PG 31.8 is not active+clean 2023-10-19T14:21:14.208 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '31.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 240, 'state': 'creating+peering', 'last_fresh': '2023-10-19T14:21:08.399409+0000', 'last_change': '2023-10-19T14:21:08.399409+0000', 'last_active': '2023-10-19T14:21:08.378327+0000', 'last_peered': '2023-10-19T14:21:08.378327+0000', 'last_clean': '2023-10-19T14:21:08.378327+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:08.399409+0000', 'last_undegraded': '2023-10-19T14:21:08.399409+0000', 'last_fullsized': '2023-10-19T14:21:08.399409+0000', 'mapping_epoch': 240, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 240, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-10-19T14:21:14.208 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.38 is not active+clean 2023-10-19T14:21:14.208 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.38', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T14:21:09.388927+0000', 'last_change': '2023-10-19T14:21:09.388927+0000', 'last_active': '2023-10-19T14:21:09.388927+0000', 'last_peered': '2023-10-19T14:21:09.388927+0000', 'last_clean': '2023-10-19T14:21:09.388927+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:09.388927+0000', 'last_undegraded': '2023-10-19T14:21:09.388927+0000', 'last_fullsized': '2023-10-19T14:21:09.388927+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T14:21:14.209 INFO:tasks.cephfs.filesystem.ceph_manager:PG 31.7 is not active+clean 2023-10-19T14:21:14.209 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '31.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 240, 'state': 'creating+peering', 'last_fresh': '2023-10-19T14:21:08.401244+0000', 'last_change': '2023-10-19T14:21:08.401244+0000', 'last_active': '2023-10-19T14:21:08.378327+0000', 'last_peered': '2023-10-19T14:21:08.378327+0000', 'last_clean': '2023-10-19T14:21:08.378327+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:08.401244+0000', 'last_undegraded': '2023-10-19T14:21:08.401244+0000', 'last_fullsized': '2023-10-19T14:21:08.401244+0000', 'mapping_epoch': 240, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 240, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [1, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-19T14:21:14.209 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.39 is not active+clean 2023-10-19T14:21:14.210 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.39', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T14:21:09.388927+0000', 'last_change': '2023-10-19T14:21:09.388927+0000', 'last_active': '2023-10-19T14:21:09.388927+0000', 'last_peered': '2023-10-19T14:21:09.388927+0000', 'last_clean': '2023-10-19T14:21:09.388927+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:09.388927+0000', 'last_undegraded': '2023-10-19T14:21:09.388927+0000', 'last_fullsized': '2023-10-19T14:21:09.388927+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T14:21:14.210 INFO:tasks.cephfs.filesystem.ceph_manager:PG 31.6 is not active+clean 2023-10-19T14:21:14.211 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '31.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 240, 'state': 'creating+peering', 'last_fresh': '2023-10-19T14:21:08.397194+0000', 'last_change': '2023-10-19T14:21:08.397194+0000', 'last_active': '2023-10-19T14:21:08.378327+0000', 'last_peered': '2023-10-19T14:21:08.378327+0000', 'last_clean': '2023-10-19T14:21:08.378327+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:08.397194+0000', 'last_undegraded': '2023-10-19T14:21:08.397194+0000', 'last_fullsized': '2023-10-19T14:21:08.397194+0000', 'mapping_epoch': 240, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 240, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 5], 'acting': [6, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-10-19T14:21:14.211 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.3a is not active+clean 2023-10-19T14:21:14.211 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.3a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T14:21:09.388927+0000', 'last_change': '2023-10-19T14:21:09.388927+0000', 'last_active': '2023-10-19T14:21:09.388927+0000', 'last_peered': '2023-10-19T14:21:09.388927+0000', 'last_clean': '2023-10-19T14:21:09.388927+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:09.388927+0000', 'last_undegraded': '2023-10-19T14:21:09.388927+0000', 'last_fullsized': '2023-10-19T14:21:09.388927+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T14:21:14.212 INFO:tasks.cephfs.filesystem.ceph_manager:PG 31.5 is not active+clean 2023-10-19T14:21:14.212 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '31.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 240, 'state': 'creating+peering', 'last_fresh': '2023-10-19T14:21:08.398883+0000', 'last_change': '2023-10-19T14:21:08.398883+0000', 'last_active': '2023-10-19T14:21:08.378327+0000', 'last_peered': '2023-10-19T14:21:08.378327+0000', 'last_clean': '2023-10-19T14:21:08.378327+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:08.398883+0000', 'last_undegraded': '2023-10-19T14:21:08.398883+0000', 'last_fullsized': '2023-10-19T14:21:08.398883+0000', 'mapping_epoch': 240, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 240, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-10-19T14:21:14.212 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.3b is not active+clean 2023-10-19T14:21:14.213 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.3b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T14:21:09.388927+0000', 'last_change': '2023-10-19T14:21:09.388927+0000', 'last_active': '2023-10-19T14:21:09.388927+0000', 'last_peered': '2023-10-19T14:21:09.388927+0000', 'last_clean': '2023-10-19T14:21:09.388927+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:09.388927+0000', 'last_undegraded': '2023-10-19T14:21:09.388927+0000', 'last_fullsized': '2023-10-19T14:21:09.388927+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T14:21:14.213 INFO:tasks.cephfs.filesystem.ceph_manager:PG 31.4 is not active+clean 2023-10-19T14:21:14.214 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '31.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 240, 'state': 'creating+peering', 'last_fresh': '2023-10-19T14:21:08.398251+0000', 'last_change': '2023-10-19T14:21:08.398251+0000', 'last_active': '2023-10-19T14:21:08.378327+0000', 'last_peered': '2023-10-19T14:21:08.378327+0000', 'last_clean': '2023-10-19T14:21:08.378327+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:08.398251+0000', 'last_undegraded': '2023-10-19T14:21:08.398251+0000', 'last_fullsized': '2023-10-19T14:21:08.398251+0000', 'mapping_epoch': 240, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 240, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-10-19T14:21:14.214 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.3c is not active+clean 2023-10-19T14:21:14.214 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.3c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T14:21:09.388927+0000', 'last_change': '2023-10-19T14:21:09.388927+0000', 'last_active': '2023-10-19T14:21:09.388927+0000', 'last_peered': '2023-10-19T14:21:09.388927+0000', 'last_clean': '2023-10-19T14:21:09.388927+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:09.388927+0000', 'last_undegraded': '2023-10-19T14:21:09.388927+0000', 'last_fullsized': '2023-10-19T14:21:09.388927+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T14:21:14.215 INFO:tasks.cephfs.filesystem.ceph_manager:PG 31.3 is not active+clean 2023-10-19T14:21:14.215 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '31.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 240, 'state': 'creating+peering', 'last_fresh': '2023-10-19T14:21:08.392809+0000', 'last_change': '2023-10-19T14:21:08.392809+0000', 'last_active': '2023-10-19T14:21:08.378327+0000', 'last_peered': '2023-10-19T14:21:08.378327+0000', 'last_clean': '2023-10-19T14:21:08.378327+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:08.392809+0000', 'last_undegraded': '2023-10-19T14:21:08.392809+0000', 'last_fullsized': '2023-10-19T14:21:08.392809+0000', 'mapping_epoch': 240, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 240, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-10-19T14:21:14.216 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.3d is not active+clean 2023-10-19T14:21:14.216 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.3d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T14:21:09.388927+0000', 'last_change': '2023-10-19T14:21:09.388927+0000', 'last_active': '2023-10-19T14:21:09.388927+0000', 'last_peered': '2023-10-19T14:21:09.388927+0000', 'last_clean': '2023-10-19T14:21:09.388927+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:09.388927+0000', 'last_undegraded': '2023-10-19T14:21:09.388927+0000', 'last_fullsized': '2023-10-19T14:21:09.388927+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T14:21:14.216 INFO:tasks.cephfs.filesystem.ceph_manager:PG 31.2 is not active+clean 2023-10-19T14:21:14.217 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '31.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 240, 'state': 'creating+peering', 'last_fresh': '2023-10-19T14:21:08.397367+0000', 'last_change': '2023-10-19T14:21:08.397367+0000', 'last_active': '2023-10-19T14:21:08.378327+0000', 'last_peered': '2023-10-19T14:21:08.378327+0000', 'last_clean': '2023-10-19T14:21:08.378327+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:08.397367+0000', 'last_undegraded': '2023-10-19T14:21:08.397367+0000', 'last_fullsized': '2023-10-19T14:21:08.397367+0000', 'mapping_epoch': 240, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 240, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-10-19T14:21:14.217 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.3e is not active+clean 2023-10-19T14:21:14.218 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.3e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T14:21:09.388927+0000', 'last_change': '2023-10-19T14:21:09.388927+0000', 'last_active': '2023-10-19T14:21:09.388927+0000', 'last_peered': '2023-10-19T14:21:09.388927+0000', 'last_clean': '2023-10-19T14:21:09.388927+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:09.388927+0000', 'last_undegraded': '2023-10-19T14:21:09.388927+0000', 'last_fullsized': '2023-10-19T14:21:09.388927+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T14:21:14.218 INFO:tasks.cephfs.filesystem.ceph_manager:PG 31.1 is not active+clean 2023-10-19T14:21:14.218 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '31.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 240, 'state': 'creating+peering', 'last_fresh': '2023-10-19T14:21:08.401262+0000', 'last_change': '2023-10-19T14:21:08.401262+0000', 'last_active': '2023-10-19T14:21:08.378327+0000', 'last_peered': '2023-10-19T14:21:08.378327+0000', 'last_clean': '2023-10-19T14:21:08.378327+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:08.401262+0000', 'last_undegraded': '2023-10-19T14:21:08.401262+0000', 'last_fullsized': '2023-10-19T14:21:08.401262+0000', 'mapping_epoch': 240, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 240, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-19T14:21:14.219 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.3f is not active+clean 2023-10-19T14:21:14.219 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.3f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T14:21:09.388927+0000', 'last_change': '2023-10-19T14:21:09.388927+0000', 'last_active': '2023-10-19T14:21:09.388927+0000', 'last_peered': '2023-10-19T14:21:09.388927+0000', 'last_clean': '2023-10-19T14:21:09.388927+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:09.388927+0000', 'last_undegraded': '2023-10-19T14:21:09.388927+0000', 'last_fullsized': '2023-10-19T14:21:09.388927+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T14:21:14.220 INFO:tasks.cephfs.filesystem.ceph_manager:PG 31.0 is not active+clean 2023-10-19T14:21:14.220 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '31.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 240, 'state': 'creating+peering', 'last_fresh': '2023-10-19T14:21:08.392705+0000', 'last_change': '2023-10-19T14:21:08.392705+0000', 'last_active': '2023-10-19T14:21:08.378327+0000', 'last_peered': '2023-10-19T14:21:08.378327+0000', 'last_clean': '2023-10-19T14:21:08.378327+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:08.392705+0000', 'last_undegraded': '2023-10-19T14:21:08.392705+0000', 'last_fullsized': '2023-10-19T14:21:08.392705+0000', 'mapping_epoch': 240, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 240, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-10-19T14:21:14.220 INFO:tasks.cephfs.filesystem.ceph_manager:PG 32.21 is not active+clean 2023-10-19T14:21:14.221 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '32.21', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-19T14:21:09.388927+0000', 'last_change': '2023-10-19T14:21:09.388927+0000', 'last_active': '2023-10-19T14:21:09.388927+0000', 'last_peered': '2023-10-19T14:21:09.388927+0000', 'last_clean': '2023-10-19T14:21:09.388927+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:09.388927+0000', 'last_undegraded': '2023-10-19T14:21:09.388927+0000', 'last_fullsized': '2023-10-19T14:21:09.388927+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:09.388927+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-19T14:21:14.221 INFO:tasks.cephfs.filesystem.ceph_manager:PG 31.1e is not active+clean 2023-10-19T14:21:14.222 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '31.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 240, 'state': 'creating+peering', 'last_fresh': '2023-10-19T14:21:08.392309+0000', 'last_change': '2023-10-19T14:21:08.392309+0000', 'last_active': '2023-10-19T14:21:08.378327+0000', 'last_peered': '2023-10-19T14:21:08.378327+0000', 'last_clean': '2023-10-19T14:21:08.378327+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:21:08.392309+0000', 'last_undegraded': '2023-10-19T14:21:08.392309+0000', 'last_fullsized': '2023-10-19T14:21:08.392309+0000', 'mapping_epoch': 240, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 240, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'last_clean_scrub_stamp': '2023-10-19T14:21:08.378327+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-10-19T14:21:14.222 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2023-10-19T14:21:14.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.329+0000 7f4555c4f640 1 -- 172.21.15.57:0/2446762028 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f45380047c0 msgr2=0x7f4538004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:14.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.329+0000 7f4555c4f640 1 --2- 172.21.15.57:0/2446762028 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f45380047c0 0x7f4538004c40 secure :-1 s=READY pgs=2637 cs=0 l=1 rev1=1 crypto rx=0x7f454400c260 tx=0x7f45380057f0 comp rx=0 tx=0).stop 2023-10-19T14:21:14.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.329+0000 7f454ffff640 1 -- 172.21.15.57:0/2446762028 reap_dead start 2023-10-19T14:21:14.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.329+0000 7f4555c4f640 1 -- 172.21.15.57:0/2446762028 shutdown_connections 2023-10-19T14:21:14.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.329+0000 7f4555c4f640 1 -- 172.21.15.57:0/2446762028 >> 172.21.15.57:0/2446762028 conn(0x7f4550069930 msgr2=0x7f4550069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:14.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.329+0000 7f4555c4f640 1 -- 172.21.15.57:0/2446762028 shutdown_connections 2023-10-19T14:21:14.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.329+0000 7f4555c4f640 1 -- 172.21.15.57:0/2446762028 wait complete. 2023-10-19T14:21:14.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.330+0000 7f4555c4f640 1 Processor -- start 2023-10-19T14:21:14.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.330+0000 7f4555c4f640 1 -- start start 2023-10-19T14:21:14.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.331+0000 7f4555c4f640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f45380047c0 0x7f455014f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:14.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.331+0000 7f4555c4f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f45501efbe0 0x7f45501f0040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:14.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.331+0000 7f4555c4f640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f45500ac720 0x7f45500acba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:14.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.331+0000 7f4555c4f640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45501f3810 con 0x7f45501efbe0 2023-10-19T14:21:14.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.331+0000 7f4555c4f640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45500ad0e0 con 0x7f45380047c0 2023-10-19T14:21:14.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.331+0000 7f4555c4f640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45501f2b90 con 0x7f45500ac720 2023-10-19T14:21:14.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.331+0000 7f454effd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f45501efbe0 0x7f45501f0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:14.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.331+0000 7f454effd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f45501efbe0 0x7f45501f0040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51702/0 (socket says 172.21.15.57:51702) 2023-10-19T14:21:14.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.331+0000 7f454effd640 1 -- 172.21.15.57:0/2126657932 learned_addr learned my addr 172.21.15.57:0/2126657932 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:14.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.331+0000 7f454ffff640 1 --2- 172.21.15.57:0/2126657932 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f45500ac720 0x7f45500acba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:14.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.331+0000 7f454f7fe640 1 --2- 172.21.15.57:0/2126657932 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f45380047c0 0x7f455014f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:14.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.331+0000 7f454effd640 1 -- 172.21.15.57:0/2126657932 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f45500ac720 msgr2=0x7f45500acba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:14.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.331+0000 7f454effd640 1 --2- 172.21.15.57:0/2126657932 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f45500ac720 0x7f45500acba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:14.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.331+0000 7f454effd640 1 -- 172.21.15.57:0/2126657932 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f45380047c0 msgr2=0x7f455014f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:14.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.331+0000 7f454effd640 1 --2- 172.21.15.57:0/2126657932 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f45380047c0 0x7f455014f9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:14.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.331+0000 7f454effd640 1 -- 172.21.15.57:0/2126657932 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45501f2e70 con 0x7f45501efbe0 2023-10-19T14:21:14.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.331+0000 7f454ffff640 1 --2- 172.21.15.57:0/2126657932 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f45500ac720 0x7f45500acba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:21:14.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.332+0000 7f454effd640 1 --2- 172.21.15.57:0/2126657932 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f45501efbe0 0x7f45501f0040 secure :-1 s=READY pgs=2638 cs=0 l=1 rev1=1 crypto rx=0x7f454400c380 tx=0x7f45440016d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:14.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.332+0000 7f4537fff640 1 -- 172.21.15.57:0/2126657932 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4544006230 con 0x7f45501efbe0 2023-10-19T14:21:14.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.332+0000 7f4537fff640 1 -- 172.21.15.57:0/2126657932 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f4544006390 con 0x7f45501efbe0 2023-10-19T14:21:14.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.332+0000 7f4555c4f640 1 -- 172.21.15.57:0/2126657932 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45501f3150 con 0x7f45501efbe0 2023-10-19T14:21:14.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.332+0000 7f4537fff640 1 -- 172.21.15.57:0/2126657932 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45440035f0 con 0x7f45501efbe0 2023-10-19T14:21:14.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.332+0000 7f4555c4f640 1 -- 172.21.15.57:0/2126657932 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45501ef770 con 0x7f45501efbe0 2023-10-19T14:21:14.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.333+0000 7f4555c4f640 1 -- 172.21.15.57:0/2126657932 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45501ef900 con 0x7f45501efbe0 2023-10-19T14:21:14.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.333+0000 7f4537fff640 1 -- 172.21.15.57:0/2126657932 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f4544031030 con 0x7f45501efbe0 2023-10-19T14:21:14.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.334+0000 7f4537fff640 1 --2- 172.21.15.57:0/2126657932 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f453c06b080 0x7f453c06d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:14.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.334+0000 7f4537fff640 1 -- 172.21.15.57:0/2126657932 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(243..243 src has 1..243) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f45440aa7d0 con 0x7f45501efbe0 2023-10-19T14:21:14.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.334+0000 7f454f7fe640 1 --2- 172.21.15.57:0/2126657932 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f453c06b080 0x7f453c06d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:14.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.334+0000 7f454f7fe640 1 --2- 172.21.15.57:0/2126657932 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f453c06b080 0x7f453c06d540 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4540009940 tx=0x7f4540005f90 comp rx=0 tx=0).ready entity=mgr.9530 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:14.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.338+0000 7f4537fff640 1 -- 172.21.15.57:0/2126657932 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f45501ef900 con 0x7f45501efbe0 2023-10-19T14:21:14.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.581+0000 7f4555c4f640 1 -- 172.21.15.57:0/2126657932 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f45500ca0e0 con 0x7f45501efbe0 2023-10-19T14:21:14.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.581+0000 7f4537fff640 1 -- 172.21.15.57:0/2126657932 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+3214 (secure 0 0 0) 0x7f4544044ca0 con 0x7f45501efbe0 2023-10-19T14:21:14.585 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:21:14.585 INFO:teuthology.orchestra.run.smithi057.stdout:{"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":5841,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":243,"num_osds":8,"num_up_osds":8,"osd_up_since":1697719445,"num_in_osds":8,"osd_in_since":1697719436,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":23,"data_bytes":592778,"bytes_used":443281408,"bytes_avail":772650831872,"bytes_total":773094113280,"write_bytes_sec":2479,"read_op_per_sec":0,"write_op_per_sec":7},"fsmap":{"epoch":103,"id":15,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":15,"rank":0,"name":"c","status":"up:active","gid":12564}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":17,"modified":"2023-10-19T14:20:58.168763+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"mgr":{"daemons":{"summary":"","x":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"y":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"mon":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"4":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"5":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"6":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"7":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2023-10-19T14:21:14.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.583+0000 7f4555c4f640 1 -- 172.21.15.57:0/2126657932 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f453c06b080 msgr2=0x7f453c06d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:14.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.583+0000 7f4555c4f640 1 --2- 172.21.15.57:0/2126657932 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f453c06b080 0x7f453c06d540 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4540009940 tx=0x7f4540005f90 comp rx=0 tx=0).stop 2023-10-19T14:21:14.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.583+0000 7f4555c4f640 1 -- 172.21.15.57:0/2126657932 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f45501efbe0 msgr2=0x7f45501f0040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:14.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.583+0000 7f4555c4f640 1 --2- 172.21.15.57:0/2126657932 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f45501efbe0 0x7f45501f0040 secure :-1 s=READY pgs=2638 cs=0 l=1 rev1=1 crypto rx=0x7f454400c380 tx=0x7f45440016d0 comp rx=0 tx=0).stop 2023-10-19T14:21:14.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.583+0000 7f4555c4f640 1 -- 172.21.15.57:0/2126657932 shutdown_connections 2023-10-19T14:21:14.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.583+0000 7f4555c4f640 1 --2- 172.21.15.57:0/2126657932 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f453c06b080 0x7f453c06d540 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:14.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.583+0000 7f4555c4f640 1 --2- 172.21.15.57:0/2126657932 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f45500ac720 0x7f45500acba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:14.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.583+0000 7f4555c4f640 1 --2- 172.21.15.57:0/2126657932 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f45501efbe0 0x7f45501f0040 unknown :-1 s=CLOSED pgs=2638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:14.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.583+0000 7f4555c4f640 1 --2- 172.21.15.57:0/2126657932 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f45380047c0 0x7f455014f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:14.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.583+0000 7f4555c4f640 1 -- 172.21.15.57:0/2126657932 >> 172.21.15.57:0/2126657932 conn(0x7f4550069930 msgr2=0x7f455013a440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:14.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.583+0000 7f4555c4f640 1 -- 172.21.15.57:0/2126657932 shutdown_connections 2023-10-19T14:21:14.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.583+0000 7f4555c4f640 1 -- 172.21.15.57:0/2126657932 wait complete. 2023-10-19T14:21:14.597 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2023-10-19T14:21:14.597 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-19T14:21:14.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.705+0000 7f7f6b2c9640 1 -- 172.21.15.57:0/2848039392 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7f4c003b30 msgr2=0x7f7f4c003f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:14.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.705+0000 7f7f6b2c9640 1 --2- 172.21.15.57:0/2848039392 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7f4c003b30 0x7f7f4c003f30 secure :-1 s=READY pgs=2640 cs=0 l=1 rev1=1 crypto rx=0x7f7f5c006f80 tx=0x7f7f4c0057f0 comp rx=0 tx=0).stop 2023-10-19T14:21:14.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.705+0000 7f7f69866640 1 -- 172.21.15.57:0/2848039392 reap_dead start 2023-10-19T14:21:14.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.705+0000 7f7f6b2c9640 1 -- 172.21.15.57:0/2848039392 shutdown_connections 2023-10-19T14:21:14.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.705+0000 7f7f6b2c9640 1 -- 172.21.15.57:0/2848039392 >> 172.21.15.57:0/2848039392 conn(0x7f7f640d0d60 msgr2=0x7f7f640d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:14.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.705+0000 7f7f6b2c9640 1 -- 172.21.15.57:0/2848039392 shutdown_connections 2023-10-19T14:21:14.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.705+0000 7f7f6b2c9640 1 -- 172.21.15.57:0/2848039392 wait complete. 2023-10-19T14:21:14.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.706+0000 7f7f6b2c9640 1 Processor -- start 2023-10-19T14:21:14.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.706+0000 7f7f6b2c9640 1 -- start start 2023-10-19T14:21:14.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.707+0000 7f7f6b2c9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7f4c003b30 0x7f7f6414fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:14.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.707+0000 7f7f6b2c9640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7f641f1680 0x7f7f640b47b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:14.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.707+0000 7f7f6b2c9640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7f640b3970 0x7f7f640b57c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:14.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.707+0000 7f7f6b2c9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f640c9dc0 con 0x7f7f4c003b30 2023-10-19T14:21:14.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.707+0000 7f7f6b2c9640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f640c91b0 con 0x7f7f640b3970 2023-10-19T14:21:14.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.707+0000 7f7f6b2c9640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f640c9490 con 0x7f7f641f1680 2023-10-19T14:21:14.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.707+0000 7f7f69065640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7f4c003b30 0x7f7f6414fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:14.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.707+0000 7f7f69065640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7f4c003b30 0x7f7f6414fcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51712/0 (socket says 172.21.15.57:51712) 2023-10-19T14:21:14.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.707+0000 7f7f69065640 1 -- 172.21.15.57:0/3946714235 learned_addr learned my addr 172.21.15.57:0/3946714235 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:14.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.707+0000 7f7f69065640 1 -- 172.21.15.57:0/3946714235 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7f641f1680 msgr2=0x7f7f640b47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:14.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.707+0000 7f7f69065640 1 --2- 172.21.15.57:0/3946714235 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7f641f1680 0x7f7f640b47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:14.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.707+0000 7f7f69065640 1 -- 172.21.15.57:0/3946714235 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7f640b3970 msgr2=0x7f7f640b57c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:14.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.707+0000 7f7f69065640 1 --2- 172.21.15.57:0/3946714235 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7f640b3970 0x7f7f640b57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:14.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.707+0000 7f7f69065640 1 -- 172.21.15.57:0/3946714235 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f640c9770 con 0x7f7f4c003b30 2023-10-19T14:21:14.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.707+0000 7f7f69065640 1 --2- 172.21.15.57:0/3946714235 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7f4c003b30 0x7f7f6414fcf0 secure :-1 s=READY pgs=2641 cs=0 l=1 rev1=1 crypto rx=0x7f7f5c02c880 tx=0x7f7f5c0065f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:14.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.707+0000 7f7f61ffb640 1 -- 172.21.15.57:0/3946714235 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f5c003a00 con 0x7f7f4c003b30 2023-10-19T14:21:14.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.707+0000 7f7f61ffb640 1 -- 172.21.15.57:0/3946714235 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f7f5c003c50 con 0x7f7f4c003b30 2023-10-19T14:21:14.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.708+0000 7f7f6b2c9640 1 -- 172.21.15.57:0/3946714235 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f640c9a50 con 0x7f7f4c003b30 2023-10-19T14:21:14.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.708+0000 7f7f61ffb640 1 -- 172.21.15.57:0/3946714235 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f5c035470 con 0x7f7f4c003b30 2023-10-19T14:21:14.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.708+0000 7f7f6b2c9640 1 -- 172.21.15.57:0/3946714235 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f640b5d00 con 0x7f7f4c003b30 2023-10-19T14:21:14.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.709+0000 7f7f6b2c9640 1 -- 172.21.15.57:0/3946714235 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f640690e0 con 0x7f7f4c003b30 2023-10-19T14:21:14.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.709+0000 7f7f61ffb640 1 -- 172.21.15.57:0/3946714235 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f7f5c032070 con 0x7f7f4c003b30 2023-10-19T14:21:14.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.709+0000 7f7f61ffb640 1 --2- 172.21.15.57:0/3946714235 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f7f5006b0b0 0x7f7f5006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:14.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.710+0000 7f7f61ffb640 1 -- 172.21.15.57:0/3946714235 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(243..243 src has 1..243) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f7f5c0ab2e0 con 0x7f7f4c003b30 2023-10-19T14:21:14.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.710+0000 7f7f68864640 1 --2- 172.21.15.57:0/3946714235 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f7f5006b0b0 0x7f7f5006d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:14.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.710+0000 7f7f68864640 1 --2- 172.21.15.57:0/3946714235 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f7f5006b0b0 0x7f7f5006d570 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f7f6406b8d0 tx=0x7f7f58005d50 comp rx=0 tx=0).ready entity=mgr.9530 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:14.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.714+0000 7f7f61ffb640 1 -- 172.21.15.57:0/3946714235 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7f5c076130 con 0x7f7f4c003b30 2023-10-19T14:21:14.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.885+0000 7f7f6b2c9640 1 -- 172.21.15.57:0/3946714235 --> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7f640c9600 con 0x7f7f5006b0b0 2023-10-19T14:21:14.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.893+0000 7f7f61ffb640 1 -- 172.21.15.57:0/3946714235 <== mgr.9530 v2:172.21.15.179:6832/3547895240 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289995 (secure 0 0 0) 0x7f7f640c9600 con 0x7f7f5006b0b0 2023-10-19T14:21:14.898 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:21:14.900 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2023-10-19T14:21:14.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.896+0000 7f7f6b2c9640 1 -- 172.21.15.57:0/3946714235 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f7f5006b0b0 msgr2=0x7f7f5006d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:14.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.896+0000 7f7f6b2c9640 1 --2- 172.21.15.57:0/3946714235 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f7f5006b0b0 0x7f7f5006d570 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f7f6406b8d0 tx=0x7f7f58005d50 comp rx=0 tx=0).stop 2023-10-19T14:21:14.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.896+0000 7f7f6b2c9640 1 -- 172.21.15.57:0/3946714235 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7f4c003b30 msgr2=0x7f7f6414fcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:14.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.896+0000 7f7f6b2c9640 1 --2- 172.21.15.57:0/3946714235 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7f4c003b30 0x7f7f6414fcf0 secure :-1 s=READY pgs=2641 cs=0 l=1 rev1=1 crypto rx=0x7f7f5c02c880 tx=0x7f7f5c0065f0 comp rx=0 tx=0).stop 2023-10-19T14:21:14.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.896+0000 7f7f6b2c9640 1 -- 172.21.15.57:0/3946714235 shutdown_connections 2023-10-19T14:21:14.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.896+0000 7f7f6b2c9640 1 --2- 172.21.15.57:0/3946714235 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f7f5006b0b0 0x7f7f5006d570 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:14.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.896+0000 7f7f6b2c9640 1 --2- 172.21.15.57:0/3946714235 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7f640b3970 0x7f7f640b57c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:14.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.896+0000 7f7f6b2c9640 1 --2- 172.21.15.57:0/3946714235 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7f641f1680 0x7f7f640b47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:14.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.896+0000 7f7f6b2c9640 1 --2- 172.21.15.57:0/3946714235 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7f4c003b30 0x7f7f6414fcf0 unknown :-1 s=CLOSED pgs=2641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:14.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.896+0000 7f7f6b2c9640 1 -- 172.21.15.57:0/3946714235 >> 172.21.15.57:0/3946714235 conn(0x7f7f640d0d60 msgr2=0x7f7f640b0ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:14.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.896+0000 7f7f6b2c9640 1 -- 172.21.15.57:0/3946714235 shutdown_connections 2023-10-19T14:21:14.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:14.896+0000 7f7f6b2c9640 1 -- 172.21.15.57:0/3946714235 wait complete. 2023-10-19T14:21:14.910 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":30,"stamp":"2023-10-19T14:21:14.172655+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2402,"num_read_kb":5036,"num_write":673,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3246,"ondisk_log_size":3246,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":432892,"kb_used_data":25156,"kb_used_omap":12,"kb_used_meta":407603,"kb_avail":754541828,"statfs":{"total":773094113280,"available":772650831872,"internally_reserved":0,"allocated":25759744,"data_stored":13793577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":417386041},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.782707"},"pg_stats":[{"pgid":"32.1f","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:11.413799+0000","last_change":"2023-10-19T14:21:10.436485+0000","last_active":"2023-10-19T14:21:11.413799+0000","last_peered":"2023-10-19T14:21:11.413799+0000","last_clean":"2023-10-19T14:21:11.413799+0000","last_became_active":"2023-10-19T14:21:10.436174+0000","last_became_peered":"2023-10-19T14:21:10.436174+0000","last_unstale":"2023-10-19T14:21:11.413799+0000","last_undegraded":"2023-10-19T14:21:11.413799+0000","last_fullsized":"2023-10-19T14:21:11.413799+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:01:21.155069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.1e","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:11.415074+0000","last_change":"2023-10-19T14:21:10.439371+0000","last_active":"2023-10-19T14:21:11.415074+0000","last_peered":"2023-10-19T14:21:11.415074+0000","last_clean":"2023-10-19T14:21:11.415074+0000","last_became_active":"2023-10-19T14:21:10.439072+0000","last_became_peered":"2023-10-19T14:21:10.439072+0000","last_unstale":"2023-10-19T14:21:11.415074+0000","last_undegraded":"2023-10-19T14:21:11.415074+0000","last_fullsized":"2023-10-19T14:21:11.415074+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:45:21.552160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1d","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:12.059002+0000","last_change":"2023-10-19T14:21:10.422867+0000","last_active":"2023-10-19T14:21:12.059002+0000","last_peered":"2023-10-19T14:21:12.059002+0000","last_clean":"2023-10-19T14:21:12.059002+0000","last_became_active":"2023-10-19T14:21:10.422558+0000","last_became_peered":"2023-10-19T14:21:10.422558+0000","last_unstale":"2023-10-19T14:21:12.059002+0000","last_undegraded":"2023-10-19T14:21:12.059002+0000","last_fullsized":"2023-10-19T14:21:12.059002+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:46:57.540004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1c","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:12.058917+0000","last_change":"2023-10-19T14:21:10.426687+0000","last_active":"2023-10-19T14:21:12.058917+0000","last_peered":"2023-10-19T14:21:12.058917+0000","last_clean":"2023-10-19T14:21:12.058917+0000","last_became_active":"2023-10-19T14:21:10.426422+0000","last_became_peered":"2023-10-19T14:21:10.426422+0000","last_unstale":"2023-10-19T14:21:12.058917+0000","last_undegraded":"2023-10-19T14:21:12.058917+0000","last_fullsized":"2023-10-19T14:21:12.058917+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:13:29.364169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1b","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:11.522064+0000","last_change":"2023-10-19T14:21:10.427647+0000","last_active":"2023-10-19T14:21:11.522064+0000","last_peered":"2023-10-19T14:21:11.522064+0000","last_clean":"2023-10-19T14:21:11.522064+0000","last_became_active":"2023-10-19T14:21:10.427412+0000","last_became_peered":"2023-10-19T14:21:10.427412+0000","last_unstale":"2023-10-19T14:21:11.522064+0000","last_undegraded":"2023-10-19T14:21:11.522064+0000","last_fullsized":"2023-10-19T14:21:11.522064+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T02:17:31.253940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1a","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:11.413640+0000","last_change":"2023-10-19T14:21:10.435192+0000","last_active":"2023-10-19T14:21:11.413640+0000","last_peered":"2023-10-19T14:21:11.413640+0000","last_clean":"2023-10-19T14:21:11.413640+0000","last_became_active":"2023-10-19T14:21:10.434868+0000","last_became_peered":"2023-10-19T14:21:10.434868+0000","last_unstale":"2023-10-19T14:21:11.413640+0000","last_undegraded":"2023-10-19T14:21:11.413640+0000","last_fullsized":"2023-10-19T14:21:11.413640+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:40:01.514130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.19","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:11.410788+0000","last_change":"2023-10-19T14:21:10.431422+0000","last_active":"2023-10-19T14:21:11.410788+0000","last_peered":"2023-10-19T14:21:11.410788+0000","last_clean":"2023-10-19T14:21:11.410788+0000","last_became_active":"2023-10-19T14:21:10.431131+0000","last_became_peered":"2023-10-19T14:21:10.431131+0000","last_unstale":"2023-10-19T14:21:11.410788+0000","last_undegraded":"2023-10-19T14:21:11.410788+0000","last_fullsized":"2023-10-19T14:21:11.410788+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:14:16.111979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.18","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:11.521898+0000","last_change":"2023-10-19T14:21:10.435886+0000","last_active":"2023-10-19T14:21:11.521898+0000","last_peered":"2023-10-19T14:21:11.521898+0000","last_clean":"2023-10-19T14:21:11.521898+0000","last_became_active":"2023-10-19T14:21:10.435541+0000","last_became_peered":"2023-10-19T14:21:10.435541+0000","last_unstale":"2023-10-19T14:21:11.521898+0000","last_undegraded":"2023-10-19T14:21:11.521898+0000","last_fullsized":"2023-10-19T14:21:11.521898+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:02:44.118651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.17","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:11.413917+0000","last_change":"2023-10-19T14:21:10.436436+0000","last_active":"2023-10-19T14:21:11.413917+0000","last_peered":"2023-10-19T14:21:11.413917+0000","last_clean":"2023-10-19T14:21:11.413917+0000","last_became_active":"2023-10-19T14:21:10.435013+0000","last_became_peered":"2023-10-19T14:21:10.435013+0000","last_unstale":"2023-10-19T14:21:11.413917+0000","last_undegraded":"2023-10-19T14:21:11.413917+0000","last_fullsized":"2023-10-19T14:21:11.413917+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:16:49.709525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.16","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:12.059121+0000","last_change":"2023-10-19T14:21:10.430483+0000","last_active":"2023-10-19T14:21:12.059121+0000","last_peered":"2023-10-19T14:21:12.059121+0000","last_clean":"2023-10-19T14:21:12.059121+0000","last_became_active":"2023-10-19T14:21:10.430168+0000","last_became_peered":"2023-10-19T14:21:10.430168+0000","last_unstale":"2023-10-19T14:21:12.059121+0000","last_undegraded":"2023-10-19T14:21:12.059121+0000","last_fullsized":"2023-10-19T14:21:12.059121+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:40:00.950097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.15","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:12.059091+0000","last_change":"2023-10-19T14:21:10.431222+0000","last_active":"2023-10-19T14:21:12.059091+0000","last_peered":"2023-10-19T14:21:12.059091+0000","last_clean":"2023-10-19T14:21:12.059091+0000","last_became_active":"2023-10-19T14:21:10.430868+0000","last_became_peered":"2023-10-19T14:21:10.430868+0000","last_unstale":"2023-10-19T14:21:12.059091+0000","last_undegraded":"2023-10-19T14:21:12.059091+0000","last_fullsized":"2023-10-19T14:21:12.059091+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:32:54.448851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.14","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:11.413522+0000","last_change":"2023-10-19T14:21:10.425461+0000","last_active":"2023-10-19T14:21:11.413522+0000","last_peered":"2023-10-19T14:21:11.413522+0000","last_clean":"2023-10-19T14:21:11.413522+0000","last_became_active":"2023-10-19T14:21:10.425149+0000","last_became_peered":"2023-10-19T14:21:10.425149+0000","last_unstale":"2023-10-19T14:21:11.413522+0000","last_undegraded":"2023-10-19T14:21:11.413522+0000","last_fullsized":"2023-10-19T14:21:11.413522+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:30:59.739271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.13","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:12.058937+0000","last_change":"2023-10-19T14:21:10.424757+0000","last_active":"2023-10-19T14:21:12.058937+0000","last_peered":"2023-10-19T14:21:12.058937+0000","last_clean":"2023-10-19T14:21:12.058937+0000","last_became_active":"2023-10-19T14:21:10.424482+0000","last_became_peered":"2023-10-19T14:21:10.424482+0000","last_unstale":"2023-10-19T14:21:12.058937+0000","last_undegraded":"2023-10-19T14:21:12.058937+0000","last_fullsized":"2023-10-19T14:21:12.058937+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:43:36.429903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.12","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:12.058368+0000","last_change":"2023-10-19T14:21:10.423736+0000","last_active":"2023-10-19T14:21:12.058368+0000","last_peered":"2023-10-19T14:21:12.058368+0000","last_clean":"2023-10-19T14:21:12.058368+0000","last_became_active":"2023-10-19T14:21:10.423448+0000","last_became_peered":"2023-10-19T14:21:10.423448+0000","last_unstale":"2023-10-19T14:21:12.058368+0000","last_undegraded":"2023-10-19T14:21:12.058368+0000","last_fullsized":"2023-10-19T14:21:12.058368+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:42:04.331509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.11","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:11.628397+0000","last_change":"2023-10-19T14:21:10.430551+0000","last_active":"2023-10-19T14:21:11.628397+0000","last_peered":"2023-10-19T14:21:11.628397+0000","last_clean":"2023-10-19T14:21:11.628397+0000","last_became_active":"2023-10-19T14:21:10.429509+0000","last_became_peered":"2023-10-19T14:21:10.429509+0000","last_unstale":"2023-10-19T14:21:11.628397+0000","last_undegraded":"2023-10-19T14:21:11.628397+0000","last_fullsized":"2023-10-19T14:21:11.628397+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:58:08.366505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.10","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:12.058596+0000","last_change":"2023-10-19T14:21:10.437276+0000","last_active":"2023-10-19T14:21:12.058596+0000","last_peered":"2023-10-19T14:21:12.058596+0000","last_clean":"2023-10-19T14:21:12.058596+0000","last_became_active":"2023-10-19T14:21:10.437013+0000","last_became_peered":"2023-10-19T14:21:10.437013+0000","last_unstale":"2023-10-19T14:21:12.058596+0000","last_undegraded":"2023-10-19T14:21:12.058596+0000","last_fullsized":"2023-10-19T14:21:12.058596+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:06:58.837634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.f","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:12.059177+0000","last_change":"2023-10-19T14:21:10.429097+0000","last_active":"2023-10-19T14:21:12.059177+0000","last_peered":"2023-10-19T14:21:12.059177+0000","last_clean":"2023-10-19T14:21:12.059177+0000","last_became_active":"2023-10-19T14:21:10.428825+0000","last_became_peered":"2023-10-19T14:21:10.428825+0000","last_unstale":"2023-10-19T14:21:12.059177+0000","last_undegraded":"2023-10-19T14:21:12.059177+0000","last_fullsized":"2023-10-19T14:21:12.059177+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:33:10.076418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.e","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:12.059192+0000","last_change":"2023-10-19T14:21:10.425661+0000","last_active":"2023-10-19T14:21:12.059192+0000","last_peered":"2023-10-19T14:21:12.059192+0000","last_clean":"2023-10-19T14:21:12.059192+0000","last_became_active":"2023-10-19T14:21:10.425369+0000","last_became_peered":"2023-10-19T14:21:10.425369+0000","last_unstale":"2023-10-19T14:21:12.059192+0000","last_undegraded":"2023-10-19T14:21:12.059192+0000","last_fullsized":"2023-10-19T14:21:12.059192+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:54:48.155840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.d","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:11.628476+0000","last_change":"2023-10-19T14:21:10.423050+0000","last_active":"2023-10-19T14:21:11.628476+0000","last_peered":"2023-10-19T14:21:11.628476+0000","last_clean":"2023-10-19T14:21:11.628476+0000","last_became_active":"2023-10-19T14:21:10.422727+0000","last_became_peered":"2023-10-19T14:21:10.422727+0000","last_unstale":"2023-10-19T14:21:11.628476+0000","last_undegraded":"2023-10-19T14:21:11.628476+0000","last_fullsized":"2023-10-19T14:21:11.628476+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:21:43.393777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.c","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:11.414823+0000","last_change":"2023-10-19T14:21:10.432921+0000","last_active":"2023-10-19T14:21:11.414823+0000","last_peered":"2023-10-19T14:21:11.414823+0000","last_clean":"2023-10-19T14:21:11.414823+0000","last_became_active":"2023-10-19T14:21:10.432654+0000","last_became_peered":"2023-10-19T14:21:10.432654+0000","last_unstale":"2023-10-19T14:21:11.414823+0000","last_undegraded":"2023-10-19T14:21:11.414823+0000","last_fullsized":"2023-10-19T14:21:11.414823+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:42:44.275430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.b","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:11.410901+0000","last_change":"2023-10-19T14:21:10.422339+0000","last_active":"2023-10-19T14:21:11.410901+0000","last_peered":"2023-10-19T14:21:11.410901+0000","last_clean":"2023-10-19T14:21:11.410901+0000","last_became_active":"2023-10-19T14:21:10.422033+0000","last_became_peered":"2023-10-19T14:21:10.422033+0000","last_unstale":"2023-10-19T14:21:11.410901+0000","last_undegraded":"2023-10-19T14:21:11.410901+0000","last_fullsized":"2023-10-19T14:21:11.410901+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:19:11.812010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.a","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:12.058969+0000","last_change":"2023-10-19T14:21:10.438315+0000","last_active":"2023-10-19T14:21:12.058969+0000","last_peered":"2023-10-19T14:21:12.058969+0000","last_clean":"2023-10-19T14:21:12.058969+0000","last_became_active":"2023-10-19T14:21:10.438026+0000","last_became_peered":"2023-10-19T14:21:10.438026+0000","last_unstale":"2023-10-19T14:21:12.058969+0000","last_undegraded":"2023-10-19T14:21:12.058969+0000","last_fullsized":"2023-10-19T14:21:12.058969+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:36:02.222588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.9","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:11.628505+0000","last_change":"2023-10-19T14:21:10.430361+0000","last_active":"2023-10-19T14:21:11.628505+0000","last_peered":"2023-10-19T14:21:11.628505+0000","last_clean":"2023-10-19T14:21:11.628505+0000","last_became_active":"2023-10-19T14:21:10.429861+0000","last_became_peered":"2023-10-19T14:21:10.429861+0000","last_unstale":"2023-10-19T14:21:11.628505+0000","last_undegraded":"2023-10-19T14:21:11.628505+0000","last_fullsized":"2023-10-19T14:21:11.628505+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:50:34.147669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.8","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:12.058654+0000","last_change":"2023-10-19T14:21:10.413051+0000","last_active":"2023-10-19T14:21:12.058654+0000","last_peered":"2023-10-19T14:21:12.058654+0000","last_clean":"2023-10-19T14:21:12.058654+0000","last_became_active":"2023-10-19T14:21:10.412765+0000","last_became_peered":"2023-10-19T14:21:10.412765+0000","last_unstale":"2023-10-19T14:21:12.058654+0000","last_undegraded":"2023-10-19T14:21:12.058654+0000","last_fullsized":"2023-10-19T14:21:12.058654+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T02:02:33.398058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.7","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:11.415075+0000","last_change":"2023-10-19T14:21:10.430882+0000","last_active":"2023-10-19T14:21:11.415075+0000","last_peered":"2023-10-19T14:21:11.415075+0000","last_clean":"2023-10-19T14:21:11.415075+0000","last_became_active":"2023-10-19T14:21:10.430631+0000","last_became_peered":"2023-10-19T14:21:10.430631+0000","last_unstale":"2023-10-19T14:21:11.415075+0000","last_undegraded":"2023-10-19T14:21:11.415075+0000","last_fullsized":"2023-10-19T14:21:11.415075+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:02:43.850106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.6","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:11.411120+0000","last_change":"2023-10-19T14:21:10.428612+0000","last_active":"2023-10-19T14:21:11.411120+0000","last_peered":"2023-10-19T14:21:11.411120+0000","last_clean":"2023-10-19T14:21:11.411120+0000","last_became_active":"2023-10-19T14:21:10.428248+0000","last_became_peered":"2023-10-19T14:21:10.428248+0000","last_unstale":"2023-10-19T14:21:11.411120+0000","last_undegraded":"2023-10-19T14:21:11.411120+0000","last_fullsized":"2023-10-19T14:21:11.411120+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:13:47.366105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.5","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:11.521896+0000","last_change":"2023-10-19T14:21:10.424660+0000","last_active":"2023-10-19T14:21:11.521896+0000","last_peered":"2023-10-19T14:21:11.521896+0000","last_clean":"2023-10-19T14:21:11.521896+0000","last_became_active":"2023-10-19T14:21:10.423849+0000","last_became_peered":"2023-10-19T14:21:10.423849+0000","last_unstale":"2023-10-19T14:21:11.521896+0000","last_undegraded":"2023-10-19T14:21:11.521896+0000","last_fullsized":"2023-10-19T14:21:11.521896+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:44:54.282098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.4","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:11.413652+0000","last_change":"2023-10-19T14:21:10.432004+0000","last_active":"2023-10-19T14:21:11.413652+0000","last_peered":"2023-10-19T14:21:11.413652+0000","last_clean":"2023-10-19T14:21:11.413652+0000","last_became_active":"2023-10-19T14:21:10.431687+0000","last_became_peered":"2023-10-19T14:21:10.431687+0000","last_unstale":"2023-10-19T14:21:11.413652+0000","last_undegraded":"2023-10-19T14:21:11.413652+0000","last_fullsized":"2023-10-19T14:21:11.413652+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:31:38.548044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.3","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:12.058450+0000","last_change":"2023-10-19T14:21:10.419200+0000","last_active":"2023-10-19T14:21:12.058450+0000","last_peered":"2023-10-19T14:21:12.058450+0000","last_clean":"2023-10-19T14:21:12.058450+0000","last_became_active":"2023-10-19T14:21:10.418921+0000","last_became_peered":"2023-10-19T14:21:10.418921+0000","last_unstale":"2023-10-19T14:21:12.058450+0000","last_undegraded":"2023-10-19T14:21:12.058450+0000","last_fullsized":"2023-10-19T14:21:12.058450+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:18:49.408210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.2","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:12.058344+0000","last_change":"2023-10-19T14:21:10.426386+0000","last_active":"2023-10-19T14:21:12.058344+0000","last_peered":"2023-10-19T14:21:12.058344+0000","last_clean":"2023-10-19T14:21:12.058344+0000","last_became_active":"2023-10-19T14:21:10.426117+0000","last_became_peered":"2023-10-19T14:21:10.426117+0000","last_unstale":"2023-10-19T14:21:12.058344+0000","last_undegraded":"2023-10-19T14:21:12.058344+0000","last_fullsized":"2023-10-19T14:21:12.058344+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:46:24.186941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.1","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:11.414632+0000","last_change":"2023-10-19T14:21:10.429319+0000","last_active":"2023-10-19T14:21:11.414632+0000","last_peered":"2023-10-19T14:21:11.414632+0000","last_clean":"2023-10-19T14:21:11.414632+0000","last_became_active":"2023-10-19T14:21:10.429037+0000","last_became_peered":"2023-10-19T14:21:10.429037+0000","last_unstale":"2023-10-19T14:21:11.414632+0000","last_undegraded":"2023-10-19T14:21:11.414632+0000","last_fullsized":"2023-10-19T14:21:11.414632+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:38:02.329128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.0","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:11.413350+0000","last_change":"2023-10-19T14:21:10.432051+0000","last_active":"2023-10-19T14:21:11.413350+0000","last_peered":"2023-10-19T14:21:11.413350+0000","last_clean":"2023-10-19T14:21:11.413350+0000","last_became_active":"2023-10-19T14:21:10.431732+0000","last_became_peered":"2023-10-19T14:21:10.431732+0000","last_unstale":"2023-10-19T14:21:11.413350+0000","last_undegraded":"2023-10-19T14:21:11.413350+0000","last_fullsized":"2023-10-19T14:21:11.413350+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:19:07.965561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.20","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:11.414580+0000","last_change":"2023-10-19T14:21:10.438066+0000","last_active":"2023-10-19T14:21:11.414580+0000","last_peered":"2023-10-19T14:21:11.414580+0000","last_clean":"2023-10-19T14:21:11.414580+0000","last_became_active":"2023-10-19T14:21:10.437789+0000","last_became_peered":"2023-10-19T14:21:10.437789+0000","last_unstale":"2023-10-19T14:21:11.414580+0000","last_undegraded":"2023-10-19T14:21:11.414580+0000","last_fullsized":"2023-10-19T14:21:11.414580+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:51:31.199054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1f","version":"242'4","reported_seq":16,"reported_epoch":242,"state":"active+clean","last_fresh":"2023-10-19T14:21:10.451688+0000","last_change":"2023-10-19T14:21:09.412253+0000","last_active":"2023-10-19T14:21:10.451688+0000","last_peered":"2023-10-19T14:21:10.451688+0000","last_clean":"2023-10-19T14:21:10.451688+0000","last_became_active":"2023-10-19T14:21:09.411979+0000","last_became_peered":"2023-10-19T14:21:09.411979+0000","last_unstale":"2023-10-19T14:21:10.451688+0000","last_undegraded":"2023-10-19T14:21:10.451688+0000","last_fullsized":"2023-10-19T14:21:10.451688+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:21:16.759464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.22","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:11.628416+0000","last_change":"2023-10-19T14:21:10.426693+0000","last_active":"2023-10-19T14:21:11.628416+0000","last_peered":"2023-10-19T14:21:11.628416+0000","last_clean":"2023-10-19T14:21:11.628416+0000","last_became_active":"2023-10-19T14:21:10.426439+0000","last_became_peered":"2023-10-19T14:21:10.426439+0000","last_unstale":"2023-10-19T14:21:11.628416+0000","last_undegraded":"2023-10-19T14:21:11.628416+0000","last_fullsized":"2023-10-19T14:21:11.628416+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:04:41.865811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.1d","version":"242'2","reported_seq":14,"reported_epoch":242,"state":"active+clean","last_fresh":"2023-10-19T14:21:10.440881+0000","last_change":"2023-10-19T14:21:09.414225+0000","last_active":"2023-10-19T14:21:10.440881+0000","last_peered":"2023-10-19T14:21:10.440881+0000","last_clean":"2023-10-19T14:21:10.440881+0000","last_became_active":"2023-10-19T14:21:09.413679+0000","last_became_peered":"2023-10-19T14:21:09.413679+0000","last_unstale":"2023-10-19T14:21:10.440881+0000","last_undegraded":"2023-10-19T14:21:10.440881+0000","last_fullsized":"2023-10-19T14:21:10.440881+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:30:30.420013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.23","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:11.413538+0000","last_change":"2023-10-19T14:21:10.426221+0000","last_active":"2023-10-19T14:21:11.413538+0000","last_peered":"2023-10-19T14:21:11.413538+0000","last_clean":"2023-10-19T14:21:11.413538+0000","last_became_active":"2023-10-19T14:21:10.425945+0000","last_became_peered":"2023-10-19T14:21:10.425945+0000","last_unstale":"2023-10-19T14:21:11.413538+0000","last_undegraded":"2023-10-19T14:21:11.413538+0000","last_fullsized":"2023-10-19T14:21:11.413538+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:34:44.606413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1c","version":"242'2","reported_seq":14,"reported_epoch":242,"state":"active+clean","last_fresh":"2023-10-19T14:21:10.443282+0000","last_change":"2023-10-19T14:21:09.412165+0000","last_active":"2023-10-19T14:21:10.443282+0000","last_peered":"2023-10-19T14:21:10.443282+0000","last_clean":"2023-10-19T14:21:10.443282+0000","last_became_active":"2023-10-19T14:21:09.411874+0000","last_became_peered":"2023-10-19T14:21:09.411874+0000","last_unstale":"2023-10-19T14:21:10.443282+0000","last_undegraded":"2023-10-19T14:21:10.443282+0000","last_fullsized":"2023-10-19T14:21:10.443282+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:14:02.794831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.24","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:12.058497+0000","last_change":"2023-10-19T14:21:10.430265+0000","last_active":"2023-10-19T14:21:12.058497+0000","last_peered":"2023-10-19T14:21:12.058497+0000","last_clean":"2023-10-19T14:21:12.058497+0000","last_became_active":"2023-10-19T14:21:10.429918+0000","last_became_peered":"2023-10-19T14:21:10.429918+0000","last_unstale":"2023-10-19T14:21:12.058497+0000","last_undegraded":"2023-10-19T14:21:12.058497+0000","last_fullsized":"2023-10-19T14:21:12.058497+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:02:23.852783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.1b","version":"242'1","reported_seq":13,"reported_epoch":242,"state":"active+clean","last_fresh":"2023-10-19T14:21:10.438885+0000","last_change":"2023-10-19T14:21:09.411062+0000","last_active":"2023-10-19T14:21:10.438885+0000","last_peered":"2023-10-19T14:21:10.438885+0000","last_clean":"2023-10-19T14:21:10.438885+0000","last_became_active":"2023-10-19T14:21:09.410715+0000","last_became_peered":"2023-10-19T14:21:09.410715+0000","last_unstale":"2023-10-19T14:21:10.438885+0000","last_undegraded":"2023-10-19T14:21:10.438885+0000","last_fullsized":"2023-10-19T14:21:10.438885+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:46:56.529404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.25","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:11.410903+0000","last_change":"2023-10-19T14:21:10.422856+0000","last_active":"2023-10-19T14:21:11.410903+0000","last_peered":"2023-10-19T14:21:11.410903+0000","last_clean":"2023-10-19T14:21:11.410903+0000","last_became_active":"2023-10-19T14:21:10.422047+0000","last_became_peered":"2023-10-19T14:21:10.422047+0000","last_unstale":"2023-10-19T14:21:11.410903+0000","last_undegraded":"2023-10-19T14:21:11.410903+0000","last_fullsized":"2023-10-19T14:21:11.410903+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:35:21.048326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.1a","version":"242'1","reported_seq":13,"reported_epoch":242,"state":"active+clean","last_fresh":"2023-10-19T14:21:10.439565+0000","last_change":"2023-10-19T14:21:09.408272+0000","last_active":"2023-10-19T14:21:10.439565+0000","last_peered":"2023-10-19T14:21:10.439565+0000","last_clean":"2023-10-19T14:21:10.439565+0000","last_became_active":"2023-10-19T14:21:09.407997+0000","last_became_peered":"2023-10-19T14:21:09.407997+0000","last_unstale":"2023-10-19T14:21:10.439565+0000","last_undegraded":"2023-10-19T14:21:10.439565+0000","last_fullsized":"2023-10-19T14:21:10.439565+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:32:11.516962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.26","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:12.059150+0000","last_change":"2023-10-19T14:21:10.427832+0000","last_active":"2023-10-19T14:21:12.059150+0000","last_peered":"2023-10-19T14:21:12.059150+0000","last_clean":"2023-10-19T14:21:12.059150+0000","last_became_active":"2023-10-19T14:21:10.427540+0000","last_became_peered":"2023-10-19T14:21:10.427540+0000","last_unstale":"2023-10-19T14:21:12.059150+0000","last_undegraded":"2023-10-19T14:21:12.059150+0000","last_fullsized":"2023-10-19T14:21:12.059150+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:47:17.356392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2023-10-19T14:21:10.422345+0000","last_change":"2023-10-19T14:21:09.408331+0000","last_active":"2023-10-19T14:21:10.422345+0000","last_peered":"2023-10-19T14:21:10.422345+0000","last_clean":"2023-10-19T14:21:10.422345+0000","last_became_active":"2023-10-19T14:21:09.407722+0000","last_became_peered":"2023-10-19T14:21:09.407722+0000","last_unstale":"2023-10-19T14:21:10.422345+0000","last_undegraded":"2023-10-19T14:21:10.422345+0000","last_fullsized":"2023-10-19T14:21:10.422345+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:06:49.485559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.27","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:11.522145+0000","last_change":"2023-10-19T14:21:10.425541+0000","last_active":"2023-10-19T14:21:11.522145+0000","last_peered":"2023-10-19T14:21:11.522145+0000","last_clean":"2023-10-19T14:21:11.522145+0000","last_became_active":"2023-10-19T14:21:10.423860+0000","last_became_peered":"2023-10-19T14:21:10.423860+0000","last_unstale":"2023-10-19T14:21:11.522145+0000","last_undegraded":"2023-10-19T14:21:11.522145+0000","last_fullsized":"2023-10-19T14:21:11.522145+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:41:24.205533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2023-10-19T14:21:10.417790+0000","last_change":"2023-10-19T14:21:09.411121+0000","last_active":"2023-10-19T14:21:10.417790+0000","last_peered":"2023-10-19T14:21:10.417790+0000","last_clean":"2023-10-19T14:21:10.417790+0000","last_became_active":"2023-10-19T14:21:09.409416+0000","last_became_peered":"2023-10-19T14:21:09.409416+0000","last_unstale":"2023-10-19T14:21:10.417790+0000","last_undegraded":"2023-10-19T14:21:10.417790+0000","last_fullsized":"2023-10-19T14:21:10.417790+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:09:01.088634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.28","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:11.410895+0000","last_change":"2023-10-19T14:21:10.422391+0000","last_active":"2023-10-19T14:21:11.410895+0000","last_peered":"2023-10-19T14:21:11.410895+0000","last_clean":"2023-10-19T14:21:11.410895+0000","last_became_active":"2023-10-19T14:21:10.422118+0000","last_became_peered":"2023-10-19T14:21:10.422118+0000","last_unstale":"2023-10-19T14:21:11.410895+0000","last_undegraded":"2023-10-19T14:21:11.410895+0000","last_fullsized":"2023-10-19T14:21:11.410895+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T02:12:47.154435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2023-10-19T14:21:10.414279+0000","last_change":"2023-10-19T14:21:09.412983+0000","last_active":"2023-10-19T14:21:10.414279+0000","last_peered":"2023-10-19T14:21:10.414279+0000","last_clean":"2023-10-19T14:21:10.414279+0000","last_became_active":"2023-10-19T14:21:09.412690+0000","last_became_peered":"2023-10-19T14:21:09.412690+0000","last_unstale":"2023-10-19T14:21:10.414279+0000","last_undegraded":"2023-10-19T14:21:10.414279+0000","last_fullsized":"2023-10-19T14:21:10.414279+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:21:42.383177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.29","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:12.058829+0000","last_change":"2023-10-19T14:21:10.427650+0000","last_active":"2023-10-19T14:21:12.058829+0000","last_peered":"2023-10-19T14:21:12.058829+0000","last_clean":"2023-10-19T14:21:12.058829+0000","last_became_active":"2023-10-19T14:21:10.427367+0000","last_became_peered":"2023-10-19T14:21:10.427367+0000","last_unstale":"2023-10-19T14:21:12.058829+0000","last_undegraded":"2023-10-19T14:21:12.058829+0000","last_fullsized":"2023-10-19T14:21:12.058829+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:54:17.097512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2023-10-19T14:21:10.421178+0000","last_change":"2023-10-19T14:21:09.408912+0000","last_active":"2023-10-19T14:21:10.421178+0000","last_peered":"2023-10-19T14:21:10.421178+0000","last_clean":"2023-10-19T14:21:10.421178+0000","last_became_active":"2023-10-19T14:21:09.408661+0000","last_became_peered":"2023-10-19T14:21:09.408661+0000","last_unstale":"2023-10-19T14:21:10.421178+0000","last_undegraded":"2023-10-19T14:21:10.421178+0000","last_fullsized":"2023-10-19T14:21:10.421178+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:55:16.857948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.2a","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:12.058773+0000","last_change":"2023-10-19T14:21:10.430153+0000","last_active":"2023-10-19T14:21:12.058773+0000","last_peered":"2023-10-19T14:21:12.058773+0000","last_clean":"2023-10-19T14:21:12.058773+0000","last_became_active":"2023-10-19T14:21:10.429868+0000","last_became_peered":"2023-10-19T14:21:10.429868+0000","last_unstale":"2023-10-19T14:21:12.058773+0000","last_undegraded":"2023-10-19T14:21:12.058773+0000","last_fullsized":"2023-10-19T14:21:12.058773+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:55:49.837750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.15","version":"242'3","reported_seq":15,"reported_epoch":242,"state":"active+clean","last_fresh":"2023-10-19T14:21:10.439239+0000","last_change":"2023-10-19T14:21:09.414620+0000","last_active":"2023-10-19T14:21:10.439239+0000","last_peered":"2023-10-19T14:21:10.439239+0000","last_clean":"2023-10-19T14:21:10.439239+0000","last_became_active":"2023-10-19T14:21:09.414378+0000","last_became_peered":"2023-10-19T14:21:09.414378+0000","last_unstale":"2023-10-19T14:21:10.439239+0000","last_undegraded":"2023-10-19T14:21:10.439239+0000","last_fullsized":"2023-10-19T14:21:10.439239+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:16:10.702399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2b","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:11.410878+0000","last_change":"2023-10-19T14:21:10.436778+0000","last_active":"2023-10-19T14:21:11.410878+0000","last_peered":"2023-10-19T14:21:11.410878+0000","last_clean":"2023-10-19T14:21:11.410878+0000","last_became_active":"2023-10-19T14:21:10.436506+0000","last_became_peered":"2023-10-19T14:21:10.436506+0000","last_unstale":"2023-10-19T14:21:11.410878+0000","last_undegraded":"2023-10-19T14:21:11.410878+0000","last_fullsized":"2023-10-19T14:21:11.410878+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:02:47.768820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.14","version":"242'6","reported_seq":18,"reported_epoch":242,"state":"active+clean","last_fresh":"2023-10-19T14:21:10.450638+0000","last_change":"2023-10-19T14:21:09.411345+0000","last_active":"2023-10-19T14:21:10.450638+0000","last_peered":"2023-10-19T14:21:10.450638+0000","last_clean":"2023-10-19T14:21:10.450638+0000","last_became_active":"2023-10-19T14:21:09.410711+0000","last_became_peered":"2023-10-19T14:21:09.410711+0000","last_unstale":"2023-10-19T14:21:10.450638+0000","last_undegraded":"2023-10-19T14:21:10.450638+0000","last_fullsized":"2023-10-19T14:21:10.450638+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:35:21.400082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.2c","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:11.414772+0000","last_change":"2023-10-19T14:21:10.435528+0000","last_active":"2023-10-19T14:21:11.414772+0000","last_peered":"2023-10-19T14:21:11.414772+0000","last_clean":"2023-10-19T14:21:11.414772+0000","last_became_active":"2023-10-19T14:21:10.435235+0000","last_became_peered":"2023-10-19T14:21:10.435235+0000","last_unstale":"2023-10-19T14:21:11.414772+0000","last_undegraded":"2023-10-19T14:21:11.414772+0000","last_fullsized":"2023-10-19T14:21:11.414772+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:24:42.687537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.13","version":"242'2","reported_seq":14,"reported_epoch":242,"state":"active+clean","last_fresh":"2023-10-19T14:21:10.451084+0000","last_change":"2023-10-19T14:21:09.413386+0000","last_active":"2023-10-19T14:21:10.451084+0000","last_peered":"2023-10-19T14:21:10.451084+0000","last_clean":"2023-10-19T14:21:10.451084+0000","last_became_active":"2023-10-19T14:21:09.413119+0000","last_became_peered":"2023-10-19T14:21:09.413119+0000","last_unstale":"2023-10-19T14:21:10.451084+0000","last_undegraded":"2023-10-19T14:21:10.451084+0000","last_fullsized":"2023-10-19T14:21:10.451084+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:32:53.438251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.2d","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:11.410836+0000","last_change":"2023-10-19T14:21:10.417244+0000","last_active":"2023-10-19T14:21:11.410836+0000","last_peered":"2023-10-19T14:21:11.410836+0000","last_clean":"2023-10-19T14:21:11.410836+0000","last_became_active":"2023-10-19T14:21:10.416948+0000","last_became_peered":"2023-10-19T14:21:10.416948+0000","last_unstale":"2023-10-19T14:21:11.410836+0000","last_undegraded":"2023-10-19T14:21:11.410836+0000","last_fullsized":"2023-10-19T14:21:11.410836+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:13:29.800561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2023-10-19T14:21:10.418418+0000","last_change":"2023-10-19T14:21:09.417189+0000","last_active":"2023-10-19T14:21:10.418418+0000","last_peered":"2023-10-19T14:21:10.418418+0000","last_clean":"2023-10-19T14:21:10.418418+0000","last_became_active":"2023-10-19T14:21:09.416920+0000","last_became_peered":"2023-10-19T14:21:09.416920+0000","last_unstale":"2023-10-19T14:21:10.418418+0000","last_undegraded":"2023-10-19T14:21:10.418418+0000","last_fullsized":"2023-10-19T14:21:10.418418+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:17:36.285741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2e","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:11.628407+0000","last_change":"2023-10-19T14:21:10.422922+0000","last_active":"2023-10-19T14:21:11.628407+0000","last_peered":"2023-10-19T14:21:11.628407+0000","last_clean":"2023-10-19T14:21:11.628407+0000","last_became_active":"2023-10-19T14:21:10.422656+0000","last_became_peered":"2023-10-19T14:21:10.422656+0000","last_unstale":"2023-10-19T14:21:11.628407+0000","last_undegraded":"2023-10-19T14:21:11.628407+0000","last_fullsized":"2023-10-19T14:21:11.628407+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:16:55.773348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2023-10-19T14:21:10.418781+0000","last_change":"2023-10-19T14:21:09.413176+0000","last_active":"2023-10-19T14:21:10.418781+0000","last_peered":"2023-10-19T14:21:10.418781+0000","last_clean":"2023-10-19T14:21:10.418781+0000","last_became_active":"2023-10-19T14:21:09.412906+0000","last_became_peered":"2023-10-19T14:21:09.412906+0000","last_unstale":"2023-10-19T14:21:10.418781+0000","last_undegraded":"2023-10-19T14:21:10.418781+0000","last_fullsized":"2023-10-19T14:21:10.418781+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:07:02.508240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2f","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:11.522117+0000","last_change":"2023-10-19T14:21:10.431716+0000","last_active":"2023-10-19T14:21:11.522117+0000","last_peered":"2023-10-19T14:21:11.522117+0000","last_clean":"2023-10-19T14:21:11.522117+0000","last_became_active":"2023-10-19T14:21:10.430784+0000","last_became_peered":"2023-10-19T14:21:10.430784+0000","last_unstale":"2023-10-19T14:21:11.522117+0000","last_undegraded":"2023-10-19T14:21:11.522117+0000","last_fullsized":"2023-10-19T14:21:11.522117+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:14:44.806142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.10","version":"242'8","reported_seq":20,"reported_epoch":242,"state":"active+clean","last_fresh":"2023-10-19T14:21:10.451476+0000","last_change":"2023-10-19T14:21:09.415001+0000","last_active":"2023-10-19T14:21:10.451476+0000","last_peered":"2023-10-19T14:21:10.451476+0000","last_clean":"2023-10-19T14:21:10.451476+0000","last_became_active":"2023-10-19T14:21:09.414727+0000","last_became_peered":"2023-10-19T14:21:09.414727+0000","last_unstale":"2023-10-19T14:21:10.451476+0000","last_undegraded":"2023-10-19T14:21:10.451476+0000","last_fullsized":"2023-10-19T14:21:10.451476+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:54:37.202858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.30","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:11.521882+0000","last_change":"2023-10-19T14:21:10.437415+0000","last_active":"2023-10-19T14:21:11.521882+0000","last_peered":"2023-10-19T14:21:11.521882+0000","last_clean":"2023-10-19T14:21:11.521882+0000","last_became_active":"2023-10-19T14:21:10.437137+0000","last_became_peered":"2023-10-19T14:21:10.437137+0000","last_unstale":"2023-10-19T14:21:11.521882+0000","last_undegraded":"2023-10-19T14:21:11.521882+0000","last_fullsized":"2023-10-19T14:21:11.521882+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:39:56.303696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.f","version":"242'1","reported_seq":13,"reported_epoch":242,"state":"active+clean","last_fresh":"2023-10-19T14:21:10.445085+0000","last_change":"2023-10-19T14:21:09.417455+0000","last_active":"2023-10-19T14:21:10.445085+0000","last_peered":"2023-10-19T14:21:10.445085+0000","last_clean":"2023-10-19T14:21:10.445085+0000","last_became_active":"2023-10-19T14:21:09.417228+0000","last_became_peered":"2023-10-19T14:21:09.417228+0000","last_unstale":"2023-10-19T14:21:10.445085+0000","last_undegraded":"2023-10-19T14:21:10.445085+0000","last_fullsized":"2023-10-19T14:21:10.445085+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:37:45.851912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.31","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:11.521633+0000","last_change":"2023-10-19T14:21:10.423316+0000","last_active":"2023-10-19T14:21:11.521633+0000","last_peered":"2023-10-19T14:21:11.521633+0000","last_clean":"2023-10-19T14:21:11.521633+0000","last_became_active":"2023-10-19T14:21:10.423028+0000","last_became_peered":"2023-10-19T14:21:10.423028+0000","last_unstale":"2023-10-19T14:21:11.521633+0000","last_undegraded":"2023-10-19T14:21:11.521633+0000","last_fullsized":"2023-10-19T14:21:11.521633+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:39:05.601924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.e","version":"242'2","reported_seq":14,"reported_epoch":242,"state":"active+clean","last_fresh":"2023-10-19T14:21:10.450825+0000","last_change":"2023-10-19T14:21:09.409551+0000","last_active":"2023-10-19T14:21:10.450825+0000","last_peered":"2023-10-19T14:21:10.450825+0000","last_clean":"2023-10-19T14:21:10.450825+0000","last_became_active":"2023-10-19T14:21:09.409238+0000","last_became_peered":"2023-10-19T14:21:09.409238+0000","last_unstale":"2023-10-19T14:21:10.450825+0000","last_undegraded":"2023-10-19T14:21:10.450825+0000","last_fullsized":"2023-10-19T14:21:10.450825+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:38:52.516509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.32","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:11.414657+0000","last_change":"2023-10-19T14:21:10.433187+0000","last_active":"2023-10-19T14:21:11.414657+0000","last_peered":"2023-10-19T14:21:11.414657+0000","last_clean":"2023-10-19T14:21:11.414657+0000","last_became_active":"2023-10-19T14:21:10.432876+0000","last_became_peered":"2023-10-19T14:21:10.432876+0000","last_unstale":"2023-10-19T14:21:11.414657+0000","last_undegraded":"2023-10-19T14:21:11.414657+0000","last_fullsized":"2023-10-19T14:21:11.414657+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:49:35.448257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.d","version":"242'1","reported_seq":13,"reported_epoch":242,"state":"active+clean","last_fresh":"2023-10-19T14:21:10.440231+0000","last_change":"2023-10-19T14:21:09.413076+0000","last_active":"2023-10-19T14:21:10.440231+0000","last_peered":"2023-10-19T14:21:10.440231+0000","last_clean":"2023-10-19T14:21:10.440231+0000","last_became_active":"2023-10-19T14:21:09.412330+0000","last_became_peered":"2023-10-19T14:21:09.412330+0000","last_unstale":"2023-10-19T14:21:10.440231+0000","last_undegraded":"2023-10-19T14:21:10.440231+0000","last_fullsized":"2023-10-19T14:21:10.440231+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:20:50.841124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.33","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:12.058817+0000","last_change":"2023-10-19T14:21:10.429531+0000","last_active":"2023-10-19T14:21:12.058817+0000","last_peered":"2023-10-19T14:21:12.058817+0000","last_clean":"2023-10-19T14:21:12.058817+0000","last_became_active":"2023-10-19T14:21:10.429261+0000","last_became_peered":"2023-10-19T14:21:10.429261+0000","last_unstale":"2023-10-19T14:21:12.058817+0000","last_undegraded":"2023-10-19T14:21:12.058817+0000","last_fullsized":"2023-10-19T14:21:12.058817+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:29:47.656632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2023-10-19T14:21:10.417666+0000","last_change":"2023-10-19T14:21:09.409030+0000","last_active":"2023-10-19T14:21:10.417666+0000","last_peered":"2023-10-19T14:21:10.417666+0000","last_clean":"2023-10-19T14:21:10.417666+0000","last_became_active":"2023-10-19T14:21:09.408717+0000","last_became_peered":"2023-10-19T14:21:09.408717+0000","last_unstale":"2023-10-19T14:21:10.417666+0000","last_undegraded":"2023-10-19T14:21:10.417666+0000","last_fullsized":"2023-10-19T14:21:10.417666+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:04:32.926013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.34","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:12.058794+0000","last_change":"2023-10-19T14:21:10.437932+0000","last_active":"2023-10-19T14:21:12.058794+0000","last_peered":"2023-10-19T14:21:12.058794+0000","last_clean":"2023-10-19T14:21:12.058794+0000","last_became_active":"2023-10-19T14:21:10.437668+0000","last_became_peered":"2023-10-19T14:21:10.437668+0000","last_unstale":"2023-10-19T14:21:12.058794+0000","last_undegraded":"2023-10-19T14:21:12.058794+0000","last_fullsized":"2023-10-19T14:21:12.058794+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:04:33.936613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.b","version":"242'2","reported_seq":14,"reported_epoch":242,"state":"active+clean","last_fresh":"2023-10-19T14:21:10.447660+0000","last_change":"2023-10-19T14:21:09.415147+0000","last_active":"2023-10-19T14:21:10.447660+0000","last_peered":"2023-10-19T14:21:10.447660+0000","last_clean":"2023-10-19T14:21:10.447660+0000","last_became_active":"2023-10-19T14:21:09.414852+0000","last_became_peered":"2023-10-19T14:21:09.414852+0000","last_unstale":"2023-10-19T14:21:10.447660+0000","last_undegraded":"2023-10-19T14:21:10.447660+0000","last_fullsized":"2023-10-19T14:21:10.447660+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:53:33.599617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.35","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:11.414824+0000","last_change":"2023-10-19T14:21:10.439769+0000","last_active":"2023-10-19T14:21:11.414824+0000","last_peered":"2023-10-19T14:21:11.414824+0000","last_clean":"2023-10-19T14:21:11.414824+0000","last_became_active":"2023-10-19T14:21:10.439056+0000","last_became_peered":"2023-10-19T14:21:10.439056+0000","last_unstale":"2023-10-19T14:21:11.414824+0000","last_undegraded":"2023-10-19T14:21:11.414824+0000","last_fullsized":"2023-10-19T14:21:11.414824+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:18:35.286014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2023-10-19T14:21:10.411249+0000","last_change":"2023-10-19T14:21:09.411613+0000","last_active":"2023-10-19T14:21:10.411249+0000","last_peered":"2023-10-19T14:21:10.411249+0000","last_clean":"2023-10-19T14:21:10.411249+0000","last_became_active":"2023-10-19T14:21:09.411365+0000","last_became_peered":"2023-10-19T14:21:09.411365+0000","last_unstale":"2023-10-19T14:21:10.411249+0000","last_undegraded":"2023-10-19T14:21:10.411249+0000","last_fullsized":"2023-10-19T14:21:10.411249+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:22:34.404197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.36","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:12.059108+0000","last_change":"2023-10-19T14:21:10.424290+0000","last_active":"2023-10-19T14:21:12.059108+0000","last_peered":"2023-10-19T14:21:12.059108+0000","last_clean":"2023-10-19T14:21:12.059108+0000","last_became_active":"2023-10-19T14:21:10.423951+0000","last_became_peered":"2023-10-19T14:21:10.423951+0000","last_unstale":"2023-10-19T14:21:12.059108+0000","last_undegraded":"2023-10-19T14:21:12.059108+0000","last_fullsized":"2023-10-19T14:21:12.059108+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:43:48.977855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2023-10-19T14:21:10.410485+0000","last_change":"2023-10-19T14:21:09.410869+0000","last_active":"2023-10-19T14:21:10.410485+0000","last_peered":"2023-10-19T14:21:10.410485+0000","last_clean":"2023-10-19T14:21:10.410485+0000","last_became_active":"2023-10-19T14:21:09.410586+0000","last_became_peered":"2023-10-19T14:21:09.410586+0000","last_unstale":"2023-10-19T14:21:10.410485+0000","last_undegraded":"2023-10-19T14:21:10.410485+0000","last_fullsized":"2023-10-19T14:21:10.410485+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:02:42.839507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.37","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:12.059139+0000","last_change":"2023-10-19T14:21:10.422504+0000","last_active":"2023-10-19T14:21:12.059139+0000","last_peered":"2023-10-19T14:21:12.059139+0000","last_clean":"2023-10-19T14:21:12.059139+0000","last_became_active":"2023-10-19T14:21:10.422196+0000","last_became_peered":"2023-10-19T14:21:10.422196+0000","last_unstale":"2023-10-19T14:21:12.059139+0000","last_undegraded":"2023-10-19T14:21:12.059139+0000","last_fullsized":"2023-10-19T14:21:12.059139+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:53:46.360770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2023-10-19T14:21:10.413826+0000","last_change":"2023-10-19T14:21:09.415870+0000","last_active":"2023-10-19T14:21:10.413826+0000","last_peered":"2023-10-19T14:21:10.413826+0000","last_clean":"2023-10-19T14:21:10.413826+0000","last_became_active":"2023-10-19T14:21:09.415609+0000","last_became_peered":"2023-10-19T14:21:09.415609+0000","last_unstale":"2023-10-19T14:21:10.413826+0000","last_undegraded":"2023-10-19T14:21:10.413826+0000","last_fullsized":"2023-10-19T14:21:10.413826+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:29:10.891884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.38","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:11.410640+0000","last_change":"2023-10-19T14:21:10.428770+0000","last_active":"2023-10-19T14:21:11.410640+0000","last_peered":"2023-10-19T14:21:11.410640+0000","last_clean":"2023-10-19T14:21:11.410640+0000","last_became_active":"2023-10-19T14:21:10.428494+0000","last_became_peered":"2023-10-19T14:21:10.428494+0000","last_unstale":"2023-10-19T14:21:11.410640+0000","last_undegraded":"2023-10-19T14:21:11.410640+0000","last_fullsized":"2023-10-19T14:21:11.410640+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:54:47.763268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.7","version":"242'2","reported_seq":14,"reported_epoch":242,"state":"active+clean","last_fresh":"2023-10-19T14:21:10.447794+0000","last_change":"2023-10-19T14:21:09.416206+0000","last_active":"2023-10-19T14:21:10.447794+0000","last_peered":"2023-10-19T14:21:10.447794+0000","last_clean":"2023-10-19T14:21:10.447794+0000","last_became_active":"2023-10-19T14:21:09.415929+0000","last_became_peered":"2023-10-19T14:21:09.415929+0000","last_unstale":"2023-10-19T14:21:10.447794+0000","last_undegraded":"2023-10-19T14:21:10.447794+0000","last_fullsized":"2023-10-19T14:21:10.447794+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:41:27.899602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.39","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:11.628206+0000","last_change":"2023-10-19T14:21:10.431238+0000","last_active":"2023-10-19T14:21:11.628206+0000","last_peered":"2023-10-19T14:21:11.628206+0000","last_clean":"2023-10-19T14:21:11.628206+0000","last_became_active":"2023-10-19T14:21:10.430970+0000","last_became_peered":"2023-10-19T14:21:10.430970+0000","last_unstale":"2023-10-19T14:21:11.628206+0000","last_undegraded":"2023-10-19T14:21:11.628206+0000","last_fullsized":"2023-10-19T14:21:11.628206+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:29:11.902484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.6","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2023-10-19T14:21:10.418749+0000","last_change":"2023-10-19T14:21:09.415597+0000","last_active":"2023-10-19T14:21:10.418749+0000","last_peered":"2023-10-19T14:21:10.418749+0000","last_clean":"2023-10-19T14:21:10.418749+0000","last_became_active":"2023-10-19T14:21:09.415325+0000","last_became_peered":"2023-10-19T14:21:09.415325+0000","last_unstale":"2023-10-19T14:21:10.418749+0000","last_undegraded":"2023-10-19T14:21:10.418749+0000","last_fullsized":"2023-10-19T14:21:10.418749+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:47:10.402857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.3a","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:12.058632+0000","last_change":"2023-10-19T14:21:10.430013+0000","last_active":"2023-10-19T14:21:12.058632+0000","last_peered":"2023-10-19T14:21:12.058632+0000","last_clean":"2023-10-19T14:21:12.058632+0000","last_became_active":"2023-10-19T14:21:10.429744+0000","last_became_peered":"2023-10-19T14:21:10.429744+0000","last_unstale":"2023-10-19T14:21:12.058632+0000","last_undegraded":"2023-10-19T14:21:12.058632+0000","last_fullsized":"2023-10-19T14:21:12.058632+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:23:00.738206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2023-10-19T14:21:10.414280+0000","last_change":"2023-10-19T14:21:09.412015+0000","last_active":"2023-10-19T14:21:10.414280+0000","last_peered":"2023-10-19T14:21:10.414280+0000","last_clean":"2023-10-19T14:21:10.414280+0000","last_became_active":"2023-10-19T14:21:09.411729+0000","last_became_peered":"2023-10-19T14:21:09.411729+0000","last_unstale":"2023-10-19T14:21:10.414280+0000","last_undegraded":"2023-10-19T14:21:10.414280+0000","last_fullsized":"2023-10-19T14:21:10.414280+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:50:33.137069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.3b","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:11.521905+0000","last_change":"2023-10-19T14:21:10.430397+0000","last_active":"2023-10-19T14:21:11.521905+0000","last_peered":"2023-10-19T14:21:11.521905+0000","last_clean":"2023-10-19T14:21:11.521905+0000","last_became_active":"2023-10-19T14:21:10.430094+0000","last_became_peered":"2023-10-19T14:21:10.430094+0000","last_unstale":"2023-10-19T14:21:11.521905+0000","last_undegraded":"2023-10-19T14:21:11.521905+0000","last_fullsized":"2023-10-19T14:21:11.521905+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:02:35.892624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2023-10-19T14:21:10.414646+0000","last_change":"2023-10-19T14:21:09.414910+0000","last_active":"2023-10-19T14:21:10.414646+0000","last_peered":"2023-10-19T14:21:10.414646+0000","last_clean":"2023-10-19T14:21:10.414646+0000","last_became_active":"2023-10-19T14:21:09.414646+0000","last_became_peered":"2023-10-19T14:21:09.414646+0000","last_unstale":"2023-10-19T14:21:10.414646+0000","last_undegraded":"2023-10-19T14:21:10.414646+0000","last_fullsized":"2023-10-19T14:21:10.414646+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:49:25.343751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.3c","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:12.058375+0000","last_change":"2023-10-19T14:21:10.418399+0000","last_active":"2023-10-19T14:21:12.058375+0000","last_peered":"2023-10-19T14:21:12.058375+0000","last_clean":"2023-10-19T14:21:12.058375+0000","last_became_active":"2023-10-19T14:21:10.416779+0000","last_became_peered":"2023-10-19T14:21:10.416779+0000","last_unstale":"2023-10-19T14:21:12.058375+0000","last_undegraded":"2023-10-19T14:21:12.058375+0000","last_fullsized":"2023-10-19T14:21:12.058375+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:50:13.187179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2023-10-19T14:21:10.411090+0000","last_change":"2023-10-19T14:21:09.412344+0000","last_active":"2023-10-19T14:21:10.411090+0000","last_peered":"2023-10-19T14:21:10.411090+0000","last_clean":"2023-10-19T14:21:10.411090+0000","last_became_active":"2023-10-19T14:21:09.412101+0000","last_became_peered":"2023-10-19T14:21:09.412101+0000","last_unstale":"2023-10-19T14:21:10.411090+0000","last_undegraded":"2023-10-19T14:21:10.411090+0000","last_fullsized":"2023-10-19T14:21:10.411090+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:51:23.060110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.3d","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:11.413772+0000","last_change":"2023-10-19T14:21:10.435707+0000","last_active":"2023-10-19T14:21:11.413772+0000","last_peered":"2023-10-19T14:21:11.413772+0000","last_clean":"2023-10-19T14:21:11.413772+0000","last_became_active":"2023-10-19T14:21:10.435426+0000","last_became_peered":"2023-10-19T14:21:10.435426+0000","last_unstale":"2023-10-19T14:21:11.413772+0000","last_undegraded":"2023-10-19T14:21:11.413772+0000","last_fullsized":"2023-10-19T14:21:11.413772+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:21:50.005281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2023-10-19T14:21:10.418546+0000","last_change":"2023-10-19T14:21:09.415572+0000","last_active":"2023-10-19T14:21:10.418546+0000","last_peered":"2023-10-19T14:21:10.418546+0000","last_clean":"2023-10-19T14:21:10.418546+0000","last_became_active":"2023-10-19T14:21:09.415271+0000","last_became_peered":"2023-10-19T14:21:09.415271+0000","last_unstale":"2023-10-19T14:21:10.418546+0000","last_undegraded":"2023-10-19T14:21:10.418546+0000","last_fullsized":"2023-10-19T14:21:10.418546+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:24:02.734433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.3e","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:11.414939+0000","last_change":"2023-10-19T14:21:10.439966+0000","last_active":"2023-10-19T14:21:11.414939+0000","last_peered":"2023-10-19T14:21:11.414939+0000","last_clean":"2023-10-19T14:21:11.414939+0000","last_became_active":"2023-10-19T14:21:10.438773+0000","last_became_peered":"2023-10-19T14:21:10.438773+0000","last_unstale":"2023-10-19T14:21:11.414939+0000","last_undegraded":"2023-10-19T14:21:11.414939+0000","last_fullsized":"2023-10-19T14:21:11.414939+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:42:16.397379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2023-10-19T14:21:10.422469+0000","last_change":"2023-10-19T14:21:09.409229+0000","last_active":"2023-10-19T14:21:10.422469+0000","last_peered":"2023-10-19T14:21:10.422469+0000","last_clean":"2023-10-19T14:21:10.422469+0000","last_became_active":"2023-10-19T14:21:09.408885+0000","last_became_peered":"2023-10-19T14:21:09.408885+0000","last_unstale":"2023-10-19T14:21:10.422469+0000","last_undegraded":"2023-10-19T14:21:10.422469+0000","last_fullsized":"2023-10-19T14:21:10.422469+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:05:37.351111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.3f","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:12.059061+0000","last_change":"2023-10-19T14:21:10.437868+0000","last_active":"2023-10-19T14:21:12.059061+0000","last_peered":"2023-10-19T14:21:12.059061+0000","last_clean":"2023-10-19T14:21:12.059061+0000","last_became_active":"2023-10-19T14:21:10.437535+0000","last_became_peered":"2023-10-19T14:21:10.437535+0000","last_unstale":"2023-10-19T14:21:12.059061+0000","last_undegraded":"2023-10-19T14:21:12.059061+0000","last_fullsized":"2023-10-19T14:21:12.059061+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:44:47.905654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2023-10-19T14:21:10.410436+0000","last_change":"2023-10-19T14:21:09.407963+0000","last_active":"2023-10-19T14:21:10.410436+0000","last_peered":"2023-10-19T14:21:10.410436+0000","last_clean":"2023-10-19T14:21:10.410436+0000","last_became_active":"2023-10-19T14:21:09.407684+0000","last_became_peered":"2023-10-19T14:21:09.407684+0000","last_unstale":"2023-10-19T14:21:10.410436+0000","last_undegraded":"2023-10-19T14:21:10.410436+0000","last_fullsized":"2023-10-19T14:21:10.410436+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:45:20.541560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.21","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:11.413672+0000","last_change":"2023-10-19T14:21:10.434210+0000","last_active":"2023-10-19T14:21:11.413672+0000","last_peered":"2023-10-19T14:21:11.413672+0000","last_clean":"2023-10-19T14:21:11.413672+0000","last_became_active":"2023-10-19T14:21:10.433919+0000","last_became_peered":"2023-10-19T14:21:10.433919+0000","last_unstale":"2023-10-19T14:21:11.413672+0000","last_undegraded":"2023-10-19T14:21:11.413672+0000","last_fullsized":"2023-10-19T14:21:11.413672+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:56:57.148327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1e","version":"242'1","reported_seq":13,"reported_epoch":242,"state":"active+clean","last_fresh":"2023-10-19T14:21:10.440196+0000","last_change":"2023-10-19T14:21:09.411855+0000","last_active":"2023-10-19T14:21:10.440196+0000","last_peered":"2023-10-19T14:21:10.440196+0000","last_clean":"2023-10-19T14:21:10.440196+0000","last_became_active":"2023-10-19T14:21:09.411510+0000","last_became_peered":"2023-10-19T14:21:09.411510+0000","last_unstale":"2023-10-19T14:21:10.440196+0000","last_undegraded":"2023-10-19T14:21:10.440196+0000","last_fullsized":"2023-10-19T14:21:10.440196+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:44:43.405619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"229'3208","reported_seq":4180,"reported_epoch":242,"state":"active+clean","last_fresh":"2023-10-19T14:21:10.418626+0000","last_change":"2023-10-19T12:44:08.636613+0000","last_active":"2023-10-19T14:21:10.418626+0000","last_peered":"2023-10-19T14:21:10.418626+0000","last_clean":"2023-10-19T14:21:10.418626+0000","last_became_active":"2023-10-19T12:44:08.635697+0000","last_became_peered":"2023-10-19T12:44:08.635697+0000","last_unstale":"2023-10-19T14:21:10.418626+0000","last_undegraded":"2023-10-19T14:21:10.418626+0000","last_fullsized":"2023-10-19T14:21:10.418626+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_clean_scrub_stamp":"2023-10-19T12:44:07.399738+0000","objects_scrubbed":0,"log_size":3208,"log_dups_size":0,"ondisk_log_size":3208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:15:23.685485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2402,"num_read_kb":5036,"num_write":629,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":32,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":31,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2402,"num_read_kb":5036,"num_write":629,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3208,"ondisk_log_size":3208,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576011,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66696,"kb_used_data":3580,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305144,"statfs":{"total":96636764160,"available":96568467456,"internally_reserved":0,"allocated":3665920,"data_stored":2166572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":64616907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:21:10 2023","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.55100000000000005,"15min":0.57999999999999996},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.32200000000000001},"max":{"1min":0.754,"5min":0.80900000000000005,"15min":0.86199999999999999},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.503,"5min":0.57599999999999996,"15min":0.59899999999999998},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.312},"max":{"1min":0.66500000000000004,"5min":0.81899999999999995,"15min":0.86599999999999999},"last":0.504}]},{"osd":1,"last update":"Thu Oct 19 14:21:10 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.58199999999999996,"15min":0.59499999999999997},"min":{"1min":0.45200000000000001,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.72199999999999998,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.56899999999999995,"15min":0.59299999999999997},"min":{"1min":0.39300000000000002,"5min":0.373,"15min":0.36699999999999999},"max":{"1min":0.78500000000000003,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.58599999999999997}]},{"osd":2,"last update":"Thu Oct 19 14:21:10 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.622,"15min":0.64200000000000002},"min":{"1min":0.47299999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.70499999999999996,"5min":0.84799999999999998,"15min":0.85499999999999998},"last":0.628},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.63900000000000001,"15min":0.65700000000000003},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.71799999999999997}]},{"osd":3,"last update":"Thu Oct 19 14:21:10 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.60799999999999998,"15min":0.63100000000000001},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.628,"5min":0.67100000000000004,"15min":0.68300000000000005},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.83299999999999996,"5min":0.876,"15min":0.89600000000000002},"last":0.70499999999999996}]},{"osd":4,"last update":"Thu Oct 19 14:21:10 2023","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.50600000000000001,"15min":0.495},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.53000000000000003,"15min":0.51100000000000001},"min":{"1min":0.34999999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.73599999999999999}]},{"osd":5,"last update":"Thu Oct 19 14:21:10 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52300000000000002,"15min":0.51100000000000001},"min":{"1min":0.33700000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.57799999999999996,"15min":0.55400000000000005},"min":{"1min":0.40699999999999997,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.054,"5min":1.054,"15min":1.054},"last":0.76600000000000001}]},{"osd":6,"last update":"Thu Oct 19 14:21:10 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.58999999999999997,"15min":0.57599999999999996},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.35999999999999999},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.86299999999999999},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.59299999999999997,"15min":0.57299999999999995},"min":{"1min":0.42999999999999999,"5min":0.35599999999999998,"15min":0.33500000000000002},"max":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.84999999999999998}]}]},{"osd":6,"up_from":13,"seq":55834576011,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38344,"kb_used_data":3000,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333496,"statfs":{"total":96636764160,"available":96597499904,"internally_reserved":0,"allocated":3072000,"data_stored":1576192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:21:02 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.59199999999999997,"15min":0.61099999999999999},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.78400000000000003,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.56299999999999994,"15min":0.60599999999999998},"min":{"1min":0.36699999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.71799999999999997,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.51600000000000001}]},{"osd":1,"last update":"Thu Oct 19 14:21:02 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.60099999999999998,"15min":0.61799999999999999},"min":{"1min":0.373,"5min":0.373,"15min":0.35799999999999998},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":1.165},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.60199999999999998,"15min":0.626},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.77400000000000002,"5min":0.83699999999999997,"15min":0.91300000000000003},"last":0.58099999999999996}]},{"osd":2,"last update":"Thu Oct 19 14:21:02 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.57599999999999996,"15min":0.60199999999999998},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.69099999999999995,"5min":0.80100000000000005,"15min":0.873},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.60599999999999998,"15min":0.624},"min":{"1min":0.40899999999999997,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.74199999999999999,"5min":0.83499999999999996,"15min":0.93100000000000005},"last":0.745}]},{"osd":3,"last update":"Thu Oct 19 14:21:02 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.629,"15min":0.64900000000000002},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.66700000000000004,"5min":0.875,"15min":0.875},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.63800000000000001,"15min":0.65500000000000003},"min":{"1min":0.45400000000000001,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.92200000000000004,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.72799999999999998}]},{"osd":4,"last update":"Thu Oct 19 14:21:02 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.48899999999999999,"15min":0.48399999999999999},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.51000000000000001,"15min":0.50800000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.30399999999999999},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.85699999999999998},"last":0.45900000000000002}]},{"osd":5,"last update":"Thu Oct 19 14:21:02 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.50700000000000001,"15min":0.502},"min":{"1min":0.34799999999999998,"5min":0.32700000000000001,"15min":0.32500000000000001},"max":{"1min":0.79900000000000004,"5min":1.115,"15min":1.115},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.56599999999999995,"15min":0.54500000000000004},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.67900000000000005}]},{"osd":7,"last update":"Thu Oct 19 14:21:02 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.56000000000000005,"15min":0.54700000000000004},"min":{"1min":0.36799999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.90100000000000002},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.56699999999999995,"15min":0.55400000000000005},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.98399999999999999,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.76500000000000001}]}]},{"osd":5,"up_from":13,"seq":55834576011,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66108,"kb_used_data":2992,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305732,"statfs":{"total":96636764160,"available":96569069568,"internally_reserved":0,"allocated":3063808,"data_stored":1576080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:20:46 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.58499999999999996,"15min":0.61699999999999999},"min":{"1min":0.33200000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.78700000000000003,"5min":0.79700000000000004,"15min":0.86399999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.60999999999999999,"15min":0.63800000000000001},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.77400000000000002,"5min":0.85899999999999999,"15min":0.85999999999999999},"last":0.70499999999999996}]},{"osd":1,"last update":"Thu Oct 19 14:20:46 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.59499999999999997,"15min":0.628},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.749,"5min":0.80300000000000005,"15min":0.877},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.626,"15min":0.64000000000000001},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.81599999999999995,"5min":0.86699999999999999,"15min":0.88500000000000001},"last":0.877}]},{"osd":2,"last update":"Thu Oct 19 14:20:46 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.59999999999999998,"15min":0.622},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.75900000000000001,"5min":0.83799999999999997,"15min":0.85099999999999998},"last":0.85799999999999998},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.63300000000000001,"15min":0.65500000000000003},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.81299999999999994,"5min":0.83299999999999996,"15min":0.90800000000000003},"last":0.77300000000000002}]},{"osd":3,"last update":"Thu Oct 19 14:20:46 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.61399999999999999,"15min":0.64100000000000001},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.77100000000000002,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.64000000000000001,"15min":0.66700000000000004},"min":{"1min":0.436,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.73899999999999999,"5min":0.876,"15min":0.90900000000000003},"last":0.89900000000000002}]},{"osd":4,"last update":"Thu Oct 19 14:20:46 2023","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.48499999999999999,"15min":0.48299999999999998},"min":{"1min":0.35499999999999998,"5min":0.34999999999999998,"15min":0.32500000000000001},"max":{"1min":0.56200000000000006,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.437,"5min":0.499,"15min":0.501},"min":{"1min":0.308,"5min":0.308,"15min":0.28599999999999998},"max":{"1min":0.57299999999999995,"5min":0.77500000000000002,"15min":0.82099999999999995},"last":0.79500000000000004}]},{"osd":6,"last update":"Thu Oct 19 14:20:46 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.51000000000000001,"15min":0.497},"min":{"1min":0.32200000000000001,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.72999999999999998,"5min":0.80800000000000005,"15min":0.88800000000000001},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.53300000000000003,"15min":0.53600000000000003},"min":{"1min":0.39900000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.70399999999999996,"5min":0.78000000000000003,"15min":0.88100000000000001},"last":0.65200000000000002}]},{"osd":7,"last update":"Thu Oct 19 14:20:46 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.50800000000000001,"15min":0.51500000000000001},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.625,"5min":0.749,"15min":0.82899999999999996},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.57399999999999995,"15min":0.56999999999999995},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.67300000000000004,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.74299999999999999}]}]},{"osd":4,"up_from":13,"seq":55834576011,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48068,"kb_used_data":2996,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323772,"statfs":{"total":96636764160,"available":96587542528,"internally_reserved":0,"allocated":3067904,"data_stored":1576150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.55500000000000005,"15min":0.57599999999999996},"min":{"1min":0.35499999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.80800000000000005,"5min":0.81799999999999995,"15min":0.879},"last":0.439},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.58499999999999996,"15min":0.59799999999999998},"min":{"1min":0.499,"5min":0.31,"15min":0.31},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.873},"last":0.54300000000000004}]},{"osd":1,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.61499999999999999,"15min":0.64100000000000001},"min":{"1min":0.504,"5min":0.372,"15min":0.32600000000000001},"max":{"1min":0.80400000000000005,"5min":0.80700000000000005,"15min":0.92700000000000005},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.623,"15min":0.64400000000000002},"min":{"1min":0.443,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":1.0940000000000001},"last":0.497}]},{"osd":2,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.63400000000000001,"15min":0.65000000000000002},"min":{"1min":0.46400000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.64800000000000002,"15min":0.66900000000000004},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":1.008},"last":0.69499999999999995}]},{"osd":3,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.61299999999999999,"15min":0.63400000000000001},"min":{"1min":0.47399999999999998,"5min":0.37,"15min":0.37},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.91900000000000004},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.66600000000000004,"15min":0.66700000000000004},"min":{"1min":0.54000000000000004,"5min":0.45400000000000001,"15min":0.437},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.71899999999999997}]},{"osd":5,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.48299999999999998,"15min":0.47099999999999997},"min":{"1min":0.34000000000000002,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.504,"15min":0.49199999999999999},"min":{"1min":0.34300000000000003,"5min":0.33700000000000002,"15min":0.29199999999999998},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.65400000000000003}]},{"osd":6,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.496,"15min":0.48599999999999999},"min":{"1min":0.34799999999999998,"5min":0.33900000000000002,"15min":0.28199999999999997},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.53700000000000003},"min":{"1min":0.39100000000000001,"5min":0.375,"15min":0.33300000000000002},"max":{"1min":0.81299999999999994,"5min":0.84799999999999998,"15min":1.2969999999999999},"last":0.749}]},{"osd":7,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.55100000000000005,"15min":0.54100000000000004},"min":{"1min":0.36499999999999999,"5min":0.35699999999999998,"15min":0.32600000000000001},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.625},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60499999999999998,"15min":0.59199999999999997},"min":{"1min":0.47099999999999997,"5min":0.32100000000000001,"15min":0.318},"max":{"1min":0.80400000000000005,"5min":0.88600000000000001,"15min":1.032},"last":0.76600000000000001}]}]},{"osd":3,"up_from":13,"seq":55834576010,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56412,"kb_used_data":3016,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315428,"statfs":{"total":96636764160,"available":96578998272,"internally_reserved":0,"allocated":3088384,"data_stored":1578330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54655432},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:20:49 2023","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.41099999999999998,"15min":0.41799999999999998},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.51800000000000002,"5min":0.61499999999999999,"15min":0.70699999999999996},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.40799999999999997,"15min":0.42799999999999999},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.752},"last":0.28899999999999998}]},{"osd":1,"last update":"Thu Oct 19 14:20:49 2023","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.42199999999999999,"15min":0.43099999999999999},"min":{"1min":0.25,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.54200000000000004,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.432,"5min":0.46400000000000002,"15min":0.47199999999999998},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.26400000000000001},"max":{"1min":0.67900000000000005,"5min":0.73099999999999998,"15min":0.81799999999999995},"last":0.40000000000000002}]},{"osd":2,"last update":"Thu Oct 19 14:20:49 2023","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.46899999999999997,"15min":0.48299999999999998},"min":{"1min":0.33100000000000002,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.625,"5min":0.64600000000000002,"15min":0.73099999999999998},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.49199999999999999,"15min":0.504},"min":{"1min":0.31900000000000001,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.83699999999999997},"last":0.30599999999999999}]},{"osd":4,"last update":"Thu Oct 19 14:20:49 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.69199999999999995,"15min":0.70299999999999996},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.871,"5min":1.018,"15min":1.157},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.71099999999999997,"15min":0.72899999999999998},"min":{"1min":0.47399999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.97699999999999998,"5min":0.996,"15min":1.173},"last":0.58299999999999996}]},{"osd":5,"last update":"Thu Oct 19 14:20:49 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.68300000000000005,"15min":0.68500000000000005},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.41999999999999998},"max":{"1min":0.83399999999999996,"5min":0.97299999999999998,"15min":1.0149999999999999},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.67500000000000004,"15min":0.68600000000000005},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.84999999999999998,"5min":0.99399999999999999,"15min":1.105},"last":0.54400000000000004}]},{"osd":6,"last update":"Thu Oct 19 14:20:49 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.65900000000000003,"15min":0.68700000000000006},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.70499999999999996,"5min":1.0009999999999999,"15min":1.0669999999999999},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.621,"5min":0.69599999999999995,"15min":0.70999999999999996},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.85599999999999998,"5min":0.91500000000000004,"15min":1.079},"last":0.59999999999999998}]},{"osd":7,"last update":"Thu Oct 19 14:20:49 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.68799999999999994,"15min":0.69899999999999995},"min":{"1min":0.436,"5min":0.434,"15min":0.434},"max":{"1min":0.78800000000000003,"5min":0.96899999999999997,"15min":1.085},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.70899999999999996,"15min":0.72899999999999998},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.79400000000000004,"5min":0.97199999999999998,"15min":1.1739999999999999},"last":0.66700000000000004}]}]},{"osd":2,"up_from":13,"seq":55834576012,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48072,"kb_used_data":3000,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323768,"statfs":{"total":96636764160,"available":96587538432,"internally_reserved":0,"allocated":3072000,"data_stored":1576732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:20:36 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.38800000000000001,"15min":0.39700000000000002},"min":{"1min":0.254,"5min":0.218,"15min":0.218},"max":{"1min":0.55300000000000005,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.435,"15min":0.436},"min":{"1min":0.28199999999999997,"5min":0.249,"15min":0.223},"max":{"1min":0.54100000000000004,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.29999999999999999}]},{"osd":1,"last update":"Thu Oct 19 14:20:36 2023","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.39600000000000002,"15min":0.41199999999999998},"min":{"1min":0.27500000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.47799999999999998,"5min":0.61699999999999999,"15min":0.65400000000000003},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.46700000000000003,"15min":0.46500000000000002},"min":{"1min":0.28499999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.60499999999999998,"5min":1.742,"15min":1.742},"last":0.39200000000000002}]},{"osd":3,"last update":"Thu Oct 19 14:20:36 2023","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.47099999999999997,"15min":0.47999999999999998},"min":{"1min":0.32100000000000001,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.63700000000000001,"5min":0.69299999999999995,"15min":0.74399999999999999},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.48799999999999999,"15min":0.50600000000000001},"min":{"1min":0.34499999999999997,"5min":0.32000000000000001,"15min":0.315},"max":{"1min":0.56999999999999995,"5min":0.67600000000000005,"15min":0.83599999999999997},"last":0.38100000000000001}]},{"osd":4,"last update":"Thu Oct 19 14:20:36 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.65700000000000003,"15min":0.66900000000000004},"min":{"1min":0.47099999999999997,"5min":0.39600000000000002,"15min":0.39500000000000002},"max":{"1min":0.85599999999999998,"5min":0.93300000000000005,"15min":0.998},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.67000000000000004,"15min":0.67900000000000005},"min":{"1min":0.46500000000000002,"5min":0.38900000000000001,"15min":0.35199999999999998},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.61499999999999999}]},{"osd":5,"last update":"Thu Oct 19 14:20:36 2023","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.69699999999999995,"15min":0.69599999999999995},"min":{"1min":0.52700000000000002,"5min":0.45100000000000001,"15min":0.40600000000000003},"max":{"1min":0.80000000000000004,"5min":1.4330000000000001,"15min":1.4330000000000001},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.68100000000000005,"15min":0.68700000000000006},"min":{"1min":0.52700000000000002,"5min":0.45100000000000001,"15min":0.39300000000000002},"max":{"1min":0.93300000000000005,"5min":1.595,"15min":1.595},"last":0.623}]},{"osd":6,"last update":"Thu Oct 19 14:20:36 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.69299999999999995,"15min":0.70799999999999996},"min":{"1min":0.48799999999999999,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.996,"5min":0.996,"15min":1.0189999999999999},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.71899999999999997,"15min":0.72099999999999997},"min":{"1min":0.48899999999999999,"5min":0.433,"15min":0.40600000000000003},"max":{"1min":0.93899999999999995,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":0.63300000000000001}]},{"osd":7,"last update":"Thu Oct 19 14:20:36 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.70199999999999996,"15min":0.71999999999999997},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.41799999999999998},"max":{"1min":0.88700000000000001,"5min":1.0129999999999999,"15min":1.0289999999999999},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.73599999999999999,"15min":0.73099999999999998},"min":{"1min":0.51900000000000002,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.97299999999999998,"5min":1.53,"15min":1.53},"last":0.65500000000000003}]}]},{"osd":1,"up_from":13,"seq":55834576011,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48060,"kb_used_data":2992,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323780,"statfs":{"total":96636764160,"available":96587550720,"internally_reserved":0,"allocated":3063808,"data_stored":1576080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.39600000000000002,"15min":0.40400000000000003},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23100000000000001},"max":{"1min":0.57299999999999995,"5min":0.61599999999999999,"15min":0.67400000000000004},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.435,"15min":0.45400000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.54200000000000004,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.442}]},{"osd":2,"last update":"Thu Oct 19 14:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.43099999999999999,"15min":0.442},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.56899999999999995,"5min":0.63100000000000001,"15min":0.77500000000000002},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.48899999999999999,"15min":0.48999999999999999},"min":{"1min":0.32400000000000001,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.67300000000000004,"5min":0.68700000000000006,"15min":0.82799999999999996},"last":0.378}]},{"osd":3,"last update":"Thu Oct 19 14:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.499,"15min":0.505},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.29499999999999998},"max":{"1min":0.64300000000000002,"5min":0.76000000000000001,"15min":0.78900000000000003},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.436,"15min":0.44600000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.26100000000000001},"max":{"1min":0.60799999999999998,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.42399999999999999}]},{"osd":4,"last update":"Thu Oct 19 14:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.66400000000000003,"15min":0.67100000000000004},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.92800000000000005,"5min":0.98299999999999998,"15min":1.042},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.67100000000000004,"15min":0.66900000000000004},"min":{"1min":0.44500000000000001,"5min":0.41299999999999998,"15min":0.32300000000000001},"max":{"1min":0.89400000000000002,"5min":0.93500000000000005,"15min":1.1659999999999999},"last":0.63600000000000001}]},{"osd":5,"last update":"Thu Oct 19 14:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.68799999999999994,"15min":0.68600000000000005},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.82099999999999995,"5min":1.002,"15min":1.3500000000000001},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.70599999999999996,"15min":0.70799999999999996},"min":{"1min":0.46700000000000003,"5min":0.44,"15min":0.40500000000000003},"max":{"1min":1.0009999999999999,"5min":1.0149999999999999,"15min":1.121},"last":0.72199999999999998}]},{"osd":6,"last update":"Thu Oct 19 14:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.67800000000000005,"15min":0.68200000000000005},"min":{"1min":0.51000000000000001,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.95499999999999996,"5min":1.0129999999999999,"15min":1.2310000000000001},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.71899999999999997,"15min":0.71599999999999997},"min":{"1min":0.48199999999999998,"5min":0.434,"15min":0.434},"max":{"1min":0.93100000000000005,"5min":1.0109999999999999,"15min":1.274},"last":0.66900000000000004}]},{"osd":7,"last update":"Thu Oct 19 14:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.71899999999999997,"15min":0.72399999999999998},"min":{"1min":0.498,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.96799999999999997,"5min":0.99099999999999999,"15min":1.075},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.70099999999999996,"15min":0.70199999999999996},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.44400000000000001},"max":{"1min":0.91200000000000003,"5min":1.0169999999999999,"15min":1.117},"last":0.61199999999999999}]}]},{"osd":0,"up_from":13,"seq":55834576012,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61132,"kb_used_data":3580,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310708,"statfs":{"total":96636764160,"available":96574164992,"internally_reserved":0,"allocated":3665920,"data_stored":2167441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Oct 19 14:21:13 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.40100000000000002,"15min":0.41099999999999998},"min":{"1min":0.26300000000000001,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.47799999999999998,"5min":0.61499999999999999,"15min":0.624},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.41699999999999998,"15min":0.45500000000000002},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.49199999999999999,"5min":0.70899999999999996,"15min":0.81100000000000005},"last":0.38300000000000001}]},{"osd":2,"last update":"Thu Oct 19 14:21:13 2023","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.34100000000000003,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.54400000000000004,"5min":0.754,"15min":0.754},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.377,"5min":0.42399999999999999,"15min":0.44400000000000001},"min":{"1min":0.28100000000000003,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.46899999999999997,"5min":0.60199999999999998,"15min":0.69099999999999995},"last":0.40300000000000002}]},{"osd":3,"last update":"Thu Oct 19 14:21:13 2023","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.436},"min":{"1min":0.27200000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.50800000000000001,"5min":0.58799999999999997,"15min":0.66800000000000004},"last":0.314},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.48199999999999998,"15min":0.50700000000000001},"min":{"1min":0.29299999999999998,"5min":0.254,"15min":0.254},"max":{"1min":0.72399999999999998,"5min":0.78100000000000003,"15min":0.84599999999999997},"last":0.45000000000000001}]},{"osd":4,"last update":"Thu Oct 19 14:21:13 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.63,"15min":0.67000000000000004},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.80300000000000005,"5min":0.93100000000000005,"15min":1.1890000000000001},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.628,"15min":0.67400000000000004},"min":{"1min":0.45900000000000002,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.75600000000000001,"5min":0.93400000000000005,"15min":1.2150000000000001},"last":0.47299999999999998}]},{"osd":5,"last update":"Thu Oct 19 14:21:13 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.67200000000000004,"15min":0.71399999999999997},"min":{"1min":0.48599999999999999,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.83099999999999996,"5min":0.98499999999999999,"15min":1.1579999999999999},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.67500000000000004,"15min":0.71399999999999997},"min":{"1min":0.499,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.80000000000000004,"5min":0.98299999999999998,"15min":1.2310000000000001},"last":0.51900000000000002}]},{"osd":6,"last update":"Thu Oct 19 14:21:13 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.66300000000000003,"15min":0.69899999999999995},"min":{"1min":0.46100000000000002,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.85499999999999998,"5min":0.97799999999999998,"15min":1.1739999999999999},"last":0.495},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.65300000000000002,"15min":0.69599999999999995},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.70299999999999996,"5min":0.97799999999999998,"15min":1.0189999999999999},"last":0.55000000000000004}]},{"osd":7,"last update":"Thu Oct 19 14:21:13 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.66600000000000004,"15min":0.70999999999999996},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.79900000000000004,"5min":0.93400000000000005,"15min":1.1180000000000001},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.66100000000000003,"15min":0.69599999999999995},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.84499999999999997,"5min":0.93799999999999994,"15min":1.0349999999999999},"last":0.53200000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-19T14:21:17.918 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-19T14:21:18.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.037+0000 7f3b4ac26640 1 -- 172.21.15.57:0/1517444866 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3b440cf340 msgr2=0x7f3b440d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:18.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.038+0000 7f3b4ac26640 1 --2- 172.21.15.57:0/1517444866 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3b440cf340 0x7f3b440d1910 secure :-1 s=READY pgs=2642 cs=0 l=1 rev1=1 crypto rx=0x7f3b2c005fd0 tx=0x7f3b2c02bd90 comp rx=0 tx=0).stop 2023-10-19T14:21:18.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.038+0000 7f3b4ac26640 1 -- 172.21.15.57:0/1517444866 shutdown_connections 2023-10-19T14:21:18.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.038+0000 7f3b4ac26640 1 --2- 172.21.15.57:0/1517444866 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3b4415b2a0 0x7f3b440cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:18.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.038+0000 7f3b4ac26640 1 --2- 172.21.15.57:0/1517444866 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3b440cf340 0x7f3b440d1910 unknown :-1 s=CLOSED pgs=2642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:18.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.038+0000 7f3b4ac26640 1 -- 172.21.15.57:0/1517444866 >> 172.21.15.57:0/1517444866 conn(0x7f3b440d0d60 msgr2=0x7f3b440d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:18.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.038+0000 7f3b4ac26640 1 -- 172.21.15.57:0/1517444866 shutdown_connections 2023-10-19T14:21:18.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.038+0000 7f3b4ac26640 1 -- 172.21.15.57:0/1517444866 wait complete. 2023-10-19T14:21:18.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.039+0000 7f3b4ac26640 1 Processor -- start 2023-10-19T14:21:18.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.039+0000 7f3b4ac26640 1 -- start start 2023-10-19T14:21:18.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.040+0000 7f3b4ac26640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3b440ceb40 0x7f3b4415c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:18.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.040+0000 7f3b4ac26640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3b440cf340 0x7f3b4415de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:18.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.040+0000 7f3b4ac26640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3b4415b2a0 0x7f3b440b0ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:18.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.040+0000 7f3b4ac26640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b440c8d20 con 0x7f3b4415b2a0 2023-10-19T14:21:18.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.040+0000 7f3b4ac26640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b440c7e30 con 0x7f3b440ceb40 2023-10-19T14:21:18.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.040+0000 7f3b4ac26640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b440c8110 con 0x7f3b440cf340 2023-10-19T14:21:18.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.040+0000 7f3b491c3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3b4415b2a0 0x7f3b440b0ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:18.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.040+0000 7f3b491c3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3b4415b2a0 0x7f3b440b0ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57436/0 (socket says 172.21.15.57:57436) 2023-10-19T14:21:18.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.040+0000 7f3b491c3640 1 -- 172.21.15.57:0/759129218 learned_addr learned my addr 172.21.15.57:0/759129218 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:18.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.040+0000 7f3b43fff640 1 --2- 172.21.15.57:0/759129218 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3b440cf340 0x7f3b4415de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:18.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.040+0000 7f3b489c2640 1 --2- 172.21.15.57:0/759129218 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3b440ceb40 0x7f3b4415c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:18.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.040+0000 7f3b491c3640 1 -- 172.21.15.57:0/759129218 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3b440cf340 msgr2=0x7f3b4415de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:18.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.040+0000 7f3b491c3640 1 --2- 172.21.15.57:0/759129218 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3b440cf340 0x7f3b4415de30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:18.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.040+0000 7f3b491c3640 1 -- 172.21.15.57:0/759129218 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3b440ceb40 msgr2=0x7f3b4415c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:18.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.040+0000 7f3b491c3640 1 --2- 172.21.15.57:0/759129218 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3b440ceb40 0x7f3b4415c0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:18.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.040+0000 7f3b491c3640 1 -- 172.21.15.57:0/759129218 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b2c009560 con 0x7f3b4415b2a0 2023-10-19T14:21:18.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.041+0000 7f3b491c3640 1 --2- 172.21.15.57:0/759129218 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3b4415b2a0 0x7f3b440b0ea0 secure :-1 s=READY pgs=2643 cs=0 l=1 rev1=1 crypto rx=0x7f3b3c00c710 tx=0x7f3b3c00cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:18.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.041+0000 7f3b417fa640 1 -- 172.21.15.57:0/759129218 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b3c006d50 con 0x7f3b4415b2a0 2023-10-19T14:21:18.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.041+0000 7f3b417fa640 1 -- 172.21.15.57:0/759129218 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f3b3c00ce20 con 0x7f3b4415b2a0 2023-10-19T14:21:18.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.041+0000 7f3b4ac26640 1 -- 172.21.15.57:0/759129218 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b440c83f0 con 0x7f3b4415b2a0 2023-10-19T14:21:18.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.041+0000 7f3b417fa640 1 -- 172.21.15.57:0/759129218 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b3c003bd0 con 0x7f3b4415b2a0 2023-10-19T14:21:18.049 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.041+0000 7f3b4ac26640 1 -- 172.21.15.57:0/759129218 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b440c89b0 con 0x7f3b4415b2a0 2023-10-19T14:21:18.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.042+0000 7f3b4ac26640 1 -- 172.21.15.57:0/759129218 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b440c8ed0 con 0x7f3b4415b2a0 2023-10-19T14:21:18.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.042+0000 7f3b417fa640 1 -- 172.21.15.57:0/759129218 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f3b3c003d30 con 0x7f3b4415b2a0 2023-10-19T14:21:18.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.043+0000 7f3b417fa640 1 --2- 172.21.15.57:0/759129218 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f3b2406b080 0x7f3b2406d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:18.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.043+0000 7f3b417fa640 1 -- 172.21.15.57:0/759129218 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(243..243 src has 1..243) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f3b3c089950 con 0x7f3b4415b2a0 2023-10-19T14:21:18.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.043+0000 7f3b489c2640 1 --2- 172.21.15.57:0/759129218 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f3b2406b080 0x7f3b2406d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:18.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.043+0000 7f3b489c2640 1 --2- 172.21.15.57:0/759129218 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f3b2406b080 0x7f3b2406d540 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f3b2c035000 tx=0x7f3b2c004730 comp rx=0 tx=0).ready entity=mgr.9530 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:18.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.047+0000 7f3b417fa640 1 -- 172.21.15.57:0/759129218 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3b3c0545d0 con 0x7f3b4415b2a0 2023-10-19T14:21:18.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.219+0000 7f3b4ac26640 1 -- 172.21.15.57:0/759129218 --> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3b440cc170 con 0x7f3b2406b080 2023-10-19T14:21:18.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.228+0000 7f3b417fa640 1 -- 172.21.15.57:0/759129218 <== mgr.9530 v2:172.21.15.179:6832/3547895240 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289995 (secure 0 0 0) 0x7f3b2c0042f0 con 0x7f3b2406b080 2023-10-19T14:21:18.234 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:21:18.235 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2023-10-19T14:21:18.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.230+0000 7f3b4ac26640 1 -- 172.21.15.57:0/759129218 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f3b2406b080 msgr2=0x7f3b2406d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:18.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.230+0000 7f3b4ac26640 1 --2- 172.21.15.57:0/759129218 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f3b2406b080 0x7f3b2406d540 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f3b2c035000 tx=0x7f3b2c004730 comp rx=0 tx=0).stop 2023-10-19T14:21:18.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.230+0000 7f3b4ac26640 1 -- 172.21.15.57:0/759129218 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3b4415b2a0 msgr2=0x7f3b440b0ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:18.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.230+0000 7f3b4ac26640 1 --2- 172.21.15.57:0/759129218 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3b4415b2a0 0x7f3b440b0ea0 secure :-1 s=READY pgs=2643 cs=0 l=1 rev1=1 crypto rx=0x7f3b3c00c710 tx=0x7f3b3c00cbd0 comp rx=0 tx=0).stop 2023-10-19T14:21:18.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.231+0000 7f3b4ac26640 1 -- 172.21.15.57:0/759129218 shutdown_connections 2023-10-19T14:21:18.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.231+0000 7f3b4ac26640 1 --2- 172.21.15.57:0/759129218 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f3b2406b080 0x7f3b2406d540 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:18.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.231+0000 7f3b4ac26640 1 --2- 172.21.15.57:0/759129218 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3b4415b2a0 0x7f3b440b0ea0 unknown :-1 s=CLOSED pgs=2643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:18.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.231+0000 7f3b4ac26640 1 --2- 172.21.15.57:0/759129218 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3b440cf340 0x7f3b4415de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:18.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.231+0000 7f3b4ac26640 1 --2- 172.21.15.57:0/759129218 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3b440ceb40 0x7f3b4415c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:18.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.231+0000 7f3b4ac26640 1 -- 172.21.15.57:0/759129218 >> 172.21.15.57:0/759129218 conn(0x7f3b440d0d60 msgr2=0x7f3b4415a440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:18.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.231+0000 7f3b4ac26640 1 -- 172.21.15.57:0/759129218 shutdown_connections 2023-10-19T14:21:18.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.231+0000 7f3b4ac26640 1 -- 172.21.15.57:0/759129218 wait complete. 2023-10-19T14:21:18.244 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":32,"stamp":"2023-10-19T14:21:18.173838+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2402,"num_read_kb":5036,"num_write":673,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3246,"ondisk_log_size":3246,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":432892,"kb_used_data":25156,"kb_used_omap":12,"kb_used_meta":407603,"kb_avail":754541828,"statfs":{"total":773094113280,"available":772650831872,"internally_reserved":0,"allocated":25759744,"data_stored":13793577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":417386041},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002132"},"pg_stats":[{"pgid":"32.1f","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:11.413799+0000","last_change":"2023-10-19T14:21:10.436485+0000","last_active":"2023-10-19T14:21:11.413799+0000","last_peered":"2023-10-19T14:21:11.413799+0000","last_clean":"2023-10-19T14:21:11.413799+0000","last_became_active":"2023-10-19T14:21:10.436174+0000","last_became_peered":"2023-10-19T14:21:10.436174+0000","last_unstale":"2023-10-19T14:21:11.413799+0000","last_undegraded":"2023-10-19T14:21:11.413799+0000","last_fullsized":"2023-10-19T14:21:11.413799+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:01:21.155069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.1e","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:11.415074+0000","last_change":"2023-10-19T14:21:10.439371+0000","last_active":"2023-10-19T14:21:11.415074+0000","last_peered":"2023-10-19T14:21:11.415074+0000","last_clean":"2023-10-19T14:21:11.415074+0000","last_became_active":"2023-10-19T14:21:10.439072+0000","last_became_peered":"2023-10-19T14:21:10.439072+0000","last_unstale":"2023-10-19T14:21:11.415074+0000","last_undegraded":"2023-10-19T14:21:11.415074+0000","last_fullsized":"2023-10-19T14:21:11.415074+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:45:21.552160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1d","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:12.059002+0000","last_change":"2023-10-19T14:21:10.422867+0000","last_active":"2023-10-19T14:21:12.059002+0000","last_peered":"2023-10-19T14:21:12.059002+0000","last_clean":"2023-10-19T14:21:12.059002+0000","last_became_active":"2023-10-19T14:21:10.422558+0000","last_became_peered":"2023-10-19T14:21:10.422558+0000","last_unstale":"2023-10-19T14:21:12.059002+0000","last_undegraded":"2023-10-19T14:21:12.059002+0000","last_fullsized":"2023-10-19T14:21:12.059002+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:46:57.540004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1c","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:12.058917+0000","last_change":"2023-10-19T14:21:10.426687+0000","last_active":"2023-10-19T14:21:12.058917+0000","last_peered":"2023-10-19T14:21:12.058917+0000","last_clean":"2023-10-19T14:21:12.058917+0000","last_became_active":"2023-10-19T14:21:10.426422+0000","last_became_peered":"2023-10-19T14:21:10.426422+0000","last_unstale":"2023-10-19T14:21:12.058917+0000","last_undegraded":"2023-10-19T14:21:12.058917+0000","last_fullsized":"2023-10-19T14:21:12.058917+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:13:29.364169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1b","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:11.522064+0000","last_change":"2023-10-19T14:21:10.427647+0000","last_active":"2023-10-19T14:21:11.522064+0000","last_peered":"2023-10-19T14:21:11.522064+0000","last_clean":"2023-10-19T14:21:11.522064+0000","last_became_active":"2023-10-19T14:21:10.427412+0000","last_became_peered":"2023-10-19T14:21:10.427412+0000","last_unstale":"2023-10-19T14:21:11.522064+0000","last_undegraded":"2023-10-19T14:21:11.522064+0000","last_fullsized":"2023-10-19T14:21:11.522064+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T02:17:31.253940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1a","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:11.413640+0000","last_change":"2023-10-19T14:21:10.435192+0000","last_active":"2023-10-19T14:21:11.413640+0000","last_peered":"2023-10-19T14:21:11.413640+0000","last_clean":"2023-10-19T14:21:11.413640+0000","last_became_active":"2023-10-19T14:21:10.434868+0000","last_became_peered":"2023-10-19T14:21:10.434868+0000","last_unstale":"2023-10-19T14:21:11.413640+0000","last_undegraded":"2023-10-19T14:21:11.413640+0000","last_fullsized":"2023-10-19T14:21:11.413640+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:40:01.514130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.19","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:11.410788+0000","last_change":"2023-10-19T14:21:10.431422+0000","last_active":"2023-10-19T14:21:11.410788+0000","last_peered":"2023-10-19T14:21:11.410788+0000","last_clean":"2023-10-19T14:21:11.410788+0000","last_became_active":"2023-10-19T14:21:10.431131+0000","last_became_peered":"2023-10-19T14:21:10.431131+0000","last_unstale":"2023-10-19T14:21:11.410788+0000","last_undegraded":"2023-10-19T14:21:11.410788+0000","last_fullsized":"2023-10-19T14:21:11.410788+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:14:16.111979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.18","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:11.521898+0000","last_change":"2023-10-19T14:21:10.435886+0000","last_active":"2023-10-19T14:21:11.521898+0000","last_peered":"2023-10-19T14:21:11.521898+0000","last_clean":"2023-10-19T14:21:11.521898+0000","last_became_active":"2023-10-19T14:21:10.435541+0000","last_became_peered":"2023-10-19T14:21:10.435541+0000","last_unstale":"2023-10-19T14:21:11.521898+0000","last_undegraded":"2023-10-19T14:21:11.521898+0000","last_fullsized":"2023-10-19T14:21:11.521898+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:02:44.118651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.17","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:11.413917+0000","last_change":"2023-10-19T14:21:10.436436+0000","last_active":"2023-10-19T14:21:11.413917+0000","last_peered":"2023-10-19T14:21:11.413917+0000","last_clean":"2023-10-19T14:21:11.413917+0000","last_became_active":"2023-10-19T14:21:10.435013+0000","last_became_peered":"2023-10-19T14:21:10.435013+0000","last_unstale":"2023-10-19T14:21:11.413917+0000","last_undegraded":"2023-10-19T14:21:11.413917+0000","last_fullsized":"2023-10-19T14:21:11.413917+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:16:49.709525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.16","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:12.059121+0000","last_change":"2023-10-19T14:21:10.430483+0000","last_active":"2023-10-19T14:21:12.059121+0000","last_peered":"2023-10-19T14:21:12.059121+0000","last_clean":"2023-10-19T14:21:12.059121+0000","last_became_active":"2023-10-19T14:21:10.430168+0000","last_became_peered":"2023-10-19T14:21:10.430168+0000","last_unstale":"2023-10-19T14:21:12.059121+0000","last_undegraded":"2023-10-19T14:21:12.059121+0000","last_fullsized":"2023-10-19T14:21:12.059121+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:40:00.950097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.15","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:12.059091+0000","last_change":"2023-10-19T14:21:10.431222+0000","last_active":"2023-10-19T14:21:12.059091+0000","last_peered":"2023-10-19T14:21:12.059091+0000","last_clean":"2023-10-19T14:21:12.059091+0000","last_became_active":"2023-10-19T14:21:10.430868+0000","last_became_peered":"2023-10-19T14:21:10.430868+0000","last_unstale":"2023-10-19T14:21:12.059091+0000","last_undegraded":"2023-10-19T14:21:12.059091+0000","last_fullsized":"2023-10-19T14:21:12.059091+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:32:54.448851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.14","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:11.413522+0000","last_change":"2023-10-19T14:21:10.425461+0000","last_active":"2023-10-19T14:21:11.413522+0000","last_peered":"2023-10-19T14:21:11.413522+0000","last_clean":"2023-10-19T14:21:11.413522+0000","last_became_active":"2023-10-19T14:21:10.425149+0000","last_became_peered":"2023-10-19T14:21:10.425149+0000","last_unstale":"2023-10-19T14:21:11.413522+0000","last_undegraded":"2023-10-19T14:21:11.413522+0000","last_fullsized":"2023-10-19T14:21:11.413522+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:30:59.739271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.13","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:12.058937+0000","last_change":"2023-10-19T14:21:10.424757+0000","last_active":"2023-10-19T14:21:12.058937+0000","last_peered":"2023-10-19T14:21:12.058937+0000","last_clean":"2023-10-19T14:21:12.058937+0000","last_became_active":"2023-10-19T14:21:10.424482+0000","last_became_peered":"2023-10-19T14:21:10.424482+0000","last_unstale":"2023-10-19T14:21:12.058937+0000","last_undegraded":"2023-10-19T14:21:12.058937+0000","last_fullsized":"2023-10-19T14:21:12.058937+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:43:36.429903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.12","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:12.058368+0000","last_change":"2023-10-19T14:21:10.423736+0000","last_active":"2023-10-19T14:21:12.058368+0000","last_peered":"2023-10-19T14:21:12.058368+0000","last_clean":"2023-10-19T14:21:12.058368+0000","last_became_active":"2023-10-19T14:21:10.423448+0000","last_became_peered":"2023-10-19T14:21:10.423448+0000","last_unstale":"2023-10-19T14:21:12.058368+0000","last_undegraded":"2023-10-19T14:21:12.058368+0000","last_fullsized":"2023-10-19T14:21:12.058368+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:42:04.331509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.11","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:11.628397+0000","last_change":"2023-10-19T14:21:10.430551+0000","last_active":"2023-10-19T14:21:11.628397+0000","last_peered":"2023-10-19T14:21:11.628397+0000","last_clean":"2023-10-19T14:21:11.628397+0000","last_became_active":"2023-10-19T14:21:10.429509+0000","last_became_peered":"2023-10-19T14:21:10.429509+0000","last_unstale":"2023-10-19T14:21:11.628397+0000","last_undegraded":"2023-10-19T14:21:11.628397+0000","last_fullsized":"2023-10-19T14:21:11.628397+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:58:08.366505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.10","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:12.058596+0000","last_change":"2023-10-19T14:21:10.437276+0000","last_active":"2023-10-19T14:21:12.058596+0000","last_peered":"2023-10-19T14:21:12.058596+0000","last_clean":"2023-10-19T14:21:12.058596+0000","last_became_active":"2023-10-19T14:21:10.437013+0000","last_became_peered":"2023-10-19T14:21:10.437013+0000","last_unstale":"2023-10-19T14:21:12.058596+0000","last_undegraded":"2023-10-19T14:21:12.058596+0000","last_fullsized":"2023-10-19T14:21:12.058596+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:06:58.837634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.f","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:12.059177+0000","last_change":"2023-10-19T14:21:10.429097+0000","last_active":"2023-10-19T14:21:12.059177+0000","last_peered":"2023-10-19T14:21:12.059177+0000","last_clean":"2023-10-19T14:21:12.059177+0000","last_became_active":"2023-10-19T14:21:10.428825+0000","last_became_peered":"2023-10-19T14:21:10.428825+0000","last_unstale":"2023-10-19T14:21:12.059177+0000","last_undegraded":"2023-10-19T14:21:12.059177+0000","last_fullsized":"2023-10-19T14:21:12.059177+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:33:10.076418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.e","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:12.059192+0000","last_change":"2023-10-19T14:21:10.425661+0000","last_active":"2023-10-19T14:21:12.059192+0000","last_peered":"2023-10-19T14:21:12.059192+0000","last_clean":"2023-10-19T14:21:12.059192+0000","last_became_active":"2023-10-19T14:21:10.425369+0000","last_became_peered":"2023-10-19T14:21:10.425369+0000","last_unstale":"2023-10-19T14:21:12.059192+0000","last_undegraded":"2023-10-19T14:21:12.059192+0000","last_fullsized":"2023-10-19T14:21:12.059192+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:54:48.155840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.d","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:11.628476+0000","last_change":"2023-10-19T14:21:10.423050+0000","last_active":"2023-10-19T14:21:11.628476+0000","last_peered":"2023-10-19T14:21:11.628476+0000","last_clean":"2023-10-19T14:21:11.628476+0000","last_became_active":"2023-10-19T14:21:10.422727+0000","last_became_peered":"2023-10-19T14:21:10.422727+0000","last_unstale":"2023-10-19T14:21:11.628476+0000","last_undegraded":"2023-10-19T14:21:11.628476+0000","last_fullsized":"2023-10-19T14:21:11.628476+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:21:43.393777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.c","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:11.414823+0000","last_change":"2023-10-19T14:21:10.432921+0000","last_active":"2023-10-19T14:21:11.414823+0000","last_peered":"2023-10-19T14:21:11.414823+0000","last_clean":"2023-10-19T14:21:11.414823+0000","last_became_active":"2023-10-19T14:21:10.432654+0000","last_became_peered":"2023-10-19T14:21:10.432654+0000","last_unstale":"2023-10-19T14:21:11.414823+0000","last_undegraded":"2023-10-19T14:21:11.414823+0000","last_fullsized":"2023-10-19T14:21:11.414823+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:42:44.275430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.b","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:11.410901+0000","last_change":"2023-10-19T14:21:10.422339+0000","last_active":"2023-10-19T14:21:11.410901+0000","last_peered":"2023-10-19T14:21:11.410901+0000","last_clean":"2023-10-19T14:21:11.410901+0000","last_became_active":"2023-10-19T14:21:10.422033+0000","last_became_peered":"2023-10-19T14:21:10.422033+0000","last_unstale":"2023-10-19T14:21:11.410901+0000","last_undegraded":"2023-10-19T14:21:11.410901+0000","last_fullsized":"2023-10-19T14:21:11.410901+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:19:11.812010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.a","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:12.058969+0000","last_change":"2023-10-19T14:21:10.438315+0000","last_active":"2023-10-19T14:21:12.058969+0000","last_peered":"2023-10-19T14:21:12.058969+0000","last_clean":"2023-10-19T14:21:12.058969+0000","last_became_active":"2023-10-19T14:21:10.438026+0000","last_became_peered":"2023-10-19T14:21:10.438026+0000","last_unstale":"2023-10-19T14:21:12.058969+0000","last_undegraded":"2023-10-19T14:21:12.058969+0000","last_fullsized":"2023-10-19T14:21:12.058969+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:36:02.222588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.9","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:11.628505+0000","last_change":"2023-10-19T14:21:10.430361+0000","last_active":"2023-10-19T14:21:11.628505+0000","last_peered":"2023-10-19T14:21:11.628505+0000","last_clean":"2023-10-19T14:21:11.628505+0000","last_became_active":"2023-10-19T14:21:10.429861+0000","last_became_peered":"2023-10-19T14:21:10.429861+0000","last_unstale":"2023-10-19T14:21:11.628505+0000","last_undegraded":"2023-10-19T14:21:11.628505+0000","last_fullsized":"2023-10-19T14:21:11.628505+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:50:34.147669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.8","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:12.058654+0000","last_change":"2023-10-19T14:21:10.413051+0000","last_active":"2023-10-19T14:21:12.058654+0000","last_peered":"2023-10-19T14:21:12.058654+0000","last_clean":"2023-10-19T14:21:12.058654+0000","last_became_active":"2023-10-19T14:21:10.412765+0000","last_became_peered":"2023-10-19T14:21:10.412765+0000","last_unstale":"2023-10-19T14:21:12.058654+0000","last_undegraded":"2023-10-19T14:21:12.058654+0000","last_fullsized":"2023-10-19T14:21:12.058654+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T02:02:33.398058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.7","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:11.415075+0000","last_change":"2023-10-19T14:21:10.430882+0000","last_active":"2023-10-19T14:21:11.415075+0000","last_peered":"2023-10-19T14:21:11.415075+0000","last_clean":"2023-10-19T14:21:11.415075+0000","last_became_active":"2023-10-19T14:21:10.430631+0000","last_became_peered":"2023-10-19T14:21:10.430631+0000","last_unstale":"2023-10-19T14:21:11.415075+0000","last_undegraded":"2023-10-19T14:21:11.415075+0000","last_fullsized":"2023-10-19T14:21:11.415075+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:02:43.850106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.6","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:11.411120+0000","last_change":"2023-10-19T14:21:10.428612+0000","last_active":"2023-10-19T14:21:11.411120+0000","last_peered":"2023-10-19T14:21:11.411120+0000","last_clean":"2023-10-19T14:21:11.411120+0000","last_became_active":"2023-10-19T14:21:10.428248+0000","last_became_peered":"2023-10-19T14:21:10.428248+0000","last_unstale":"2023-10-19T14:21:11.411120+0000","last_undegraded":"2023-10-19T14:21:11.411120+0000","last_fullsized":"2023-10-19T14:21:11.411120+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:13:47.366105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.5","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:11.521896+0000","last_change":"2023-10-19T14:21:10.424660+0000","last_active":"2023-10-19T14:21:11.521896+0000","last_peered":"2023-10-19T14:21:11.521896+0000","last_clean":"2023-10-19T14:21:11.521896+0000","last_became_active":"2023-10-19T14:21:10.423849+0000","last_became_peered":"2023-10-19T14:21:10.423849+0000","last_unstale":"2023-10-19T14:21:11.521896+0000","last_undegraded":"2023-10-19T14:21:11.521896+0000","last_fullsized":"2023-10-19T14:21:11.521896+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:44:54.282098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.4","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:11.413652+0000","last_change":"2023-10-19T14:21:10.432004+0000","last_active":"2023-10-19T14:21:11.413652+0000","last_peered":"2023-10-19T14:21:11.413652+0000","last_clean":"2023-10-19T14:21:11.413652+0000","last_became_active":"2023-10-19T14:21:10.431687+0000","last_became_peered":"2023-10-19T14:21:10.431687+0000","last_unstale":"2023-10-19T14:21:11.413652+0000","last_undegraded":"2023-10-19T14:21:11.413652+0000","last_fullsized":"2023-10-19T14:21:11.413652+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:31:38.548044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.3","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:12.058450+0000","last_change":"2023-10-19T14:21:10.419200+0000","last_active":"2023-10-19T14:21:12.058450+0000","last_peered":"2023-10-19T14:21:12.058450+0000","last_clean":"2023-10-19T14:21:12.058450+0000","last_became_active":"2023-10-19T14:21:10.418921+0000","last_became_peered":"2023-10-19T14:21:10.418921+0000","last_unstale":"2023-10-19T14:21:12.058450+0000","last_undegraded":"2023-10-19T14:21:12.058450+0000","last_fullsized":"2023-10-19T14:21:12.058450+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:18:49.408210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.2","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:12.058344+0000","last_change":"2023-10-19T14:21:10.426386+0000","last_active":"2023-10-19T14:21:12.058344+0000","last_peered":"2023-10-19T14:21:12.058344+0000","last_clean":"2023-10-19T14:21:12.058344+0000","last_became_active":"2023-10-19T14:21:10.426117+0000","last_became_peered":"2023-10-19T14:21:10.426117+0000","last_unstale":"2023-10-19T14:21:12.058344+0000","last_undegraded":"2023-10-19T14:21:12.058344+0000","last_fullsized":"2023-10-19T14:21:12.058344+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:46:24.186941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.1","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:11.414632+0000","last_change":"2023-10-19T14:21:10.429319+0000","last_active":"2023-10-19T14:21:11.414632+0000","last_peered":"2023-10-19T14:21:11.414632+0000","last_clean":"2023-10-19T14:21:11.414632+0000","last_became_active":"2023-10-19T14:21:10.429037+0000","last_became_peered":"2023-10-19T14:21:10.429037+0000","last_unstale":"2023-10-19T14:21:11.414632+0000","last_undegraded":"2023-10-19T14:21:11.414632+0000","last_fullsized":"2023-10-19T14:21:11.414632+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:38:02.329128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.0","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:11.413350+0000","last_change":"2023-10-19T14:21:10.432051+0000","last_active":"2023-10-19T14:21:11.413350+0000","last_peered":"2023-10-19T14:21:11.413350+0000","last_clean":"2023-10-19T14:21:11.413350+0000","last_became_active":"2023-10-19T14:21:10.431732+0000","last_became_peered":"2023-10-19T14:21:10.431732+0000","last_unstale":"2023-10-19T14:21:11.413350+0000","last_undegraded":"2023-10-19T14:21:11.413350+0000","last_fullsized":"2023-10-19T14:21:11.413350+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:19:07.965561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.20","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:11.414580+0000","last_change":"2023-10-19T14:21:10.438066+0000","last_active":"2023-10-19T14:21:11.414580+0000","last_peered":"2023-10-19T14:21:11.414580+0000","last_clean":"2023-10-19T14:21:11.414580+0000","last_became_active":"2023-10-19T14:21:10.437789+0000","last_became_peered":"2023-10-19T14:21:10.437789+0000","last_unstale":"2023-10-19T14:21:11.414580+0000","last_undegraded":"2023-10-19T14:21:11.414580+0000","last_fullsized":"2023-10-19T14:21:11.414580+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:51:31.199054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1f","version":"242'4","reported_seq":16,"reported_epoch":242,"state":"active+clean","last_fresh":"2023-10-19T14:21:10.451688+0000","last_change":"2023-10-19T14:21:09.412253+0000","last_active":"2023-10-19T14:21:10.451688+0000","last_peered":"2023-10-19T14:21:10.451688+0000","last_clean":"2023-10-19T14:21:10.451688+0000","last_became_active":"2023-10-19T14:21:09.411979+0000","last_became_peered":"2023-10-19T14:21:09.411979+0000","last_unstale":"2023-10-19T14:21:10.451688+0000","last_undegraded":"2023-10-19T14:21:10.451688+0000","last_fullsized":"2023-10-19T14:21:10.451688+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:21:16.759464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.22","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:11.628416+0000","last_change":"2023-10-19T14:21:10.426693+0000","last_active":"2023-10-19T14:21:11.628416+0000","last_peered":"2023-10-19T14:21:11.628416+0000","last_clean":"2023-10-19T14:21:11.628416+0000","last_became_active":"2023-10-19T14:21:10.426439+0000","last_became_peered":"2023-10-19T14:21:10.426439+0000","last_unstale":"2023-10-19T14:21:11.628416+0000","last_undegraded":"2023-10-19T14:21:11.628416+0000","last_fullsized":"2023-10-19T14:21:11.628416+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:04:41.865811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.1d","version":"242'2","reported_seq":14,"reported_epoch":242,"state":"active+clean","last_fresh":"2023-10-19T14:21:10.440881+0000","last_change":"2023-10-19T14:21:09.414225+0000","last_active":"2023-10-19T14:21:10.440881+0000","last_peered":"2023-10-19T14:21:10.440881+0000","last_clean":"2023-10-19T14:21:10.440881+0000","last_became_active":"2023-10-19T14:21:09.413679+0000","last_became_peered":"2023-10-19T14:21:09.413679+0000","last_unstale":"2023-10-19T14:21:10.440881+0000","last_undegraded":"2023-10-19T14:21:10.440881+0000","last_fullsized":"2023-10-19T14:21:10.440881+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:30:30.420013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.23","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:11.413538+0000","last_change":"2023-10-19T14:21:10.426221+0000","last_active":"2023-10-19T14:21:11.413538+0000","last_peered":"2023-10-19T14:21:11.413538+0000","last_clean":"2023-10-19T14:21:11.413538+0000","last_became_active":"2023-10-19T14:21:10.425945+0000","last_became_peered":"2023-10-19T14:21:10.425945+0000","last_unstale":"2023-10-19T14:21:11.413538+0000","last_undegraded":"2023-10-19T14:21:11.413538+0000","last_fullsized":"2023-10-19T14:21:11.413538+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:34:44.606413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1c","version":"242'2","reported_seq":14,"reported_epoch":242,"state":"active+clean","last_fresh":"2023-10-19T14:21:10.443282+0000","last_change":"2023-10-19T14:21:09.412165+0000","last_active":"2023-10-19T14:21:10.443282+0000","last_peered":"2023-10-19T14:21:10.443282+0000","last_clean":"2023-10-19T14:21:10.443282+0000","last_became_active":"2023-10-19T14:21:09.411874+0000","last_became_peered":"2023-10-19T14:21:09.411874+0000","last_unstale":"2023-10-19T14:21:10.443282+0000","last_undegraded":"2023-10-19T14:21:10.443282+0000","last_fullsized":"2023-10-19T14:21:10.443282+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:14:02.794831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.24","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:12.058497+0000","last_change":"2023-10-19T14:21:10.430265+0000","last_active":"2023-10-19T14:21:12.058497+0000","last_peered":"2023-10-19T14:21:12.058497+0000","last_clean":"2023-10-19T14:21:12.058497+0000","last_became_active":"2023-10-19T14:21:10.429918+0000","last_became_peered":"2023-10-19T14:21:10.429918+0000","last_unstale":"2023-10-19T14:21:12.058497+0000","last_undegraded":"2023-10-19T14:21:12.058497+0000","last_fullsized":"2023-10-19T14:21:12.058497+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:02:23.852783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.1b","version":"242'1","reported_seq":13,"reported_epoch":242,"state":"active+clean","last_fresh":"2023-10-19T14:21:10.438885+0000","last_change":"2023-10-19T14:21:09.411062+0000","last_active":"2023-10-19T14:21:10.438885+0000","last_peered":"2023-10-19T14:21:10.438885+0000","last_clean":"2023-10-19T14:21:10.438885+0000","last_became_active":"2023-10-19T14:21:09.410715+0000","last_became_peered":"2023-10-19T14:21:09.410715+0000","last_unstale":"2023-10-19T14:21:10.438885+0000","last_undegraded":"2023-10-19T14:21:10.438885+0000","last_fullsized":"2023-10-19T14:21:10.438885+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:46:56.529404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.25","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:11.410903+0000","last_change":"2023-10-19T14:21:10.422856+0000","last_active":"2023-10-19T14:21:11.410903+0000","last_peered":"2023-10-19T14:21:11.410903+0000","last_clean":"2023-10-19T14:21:11.410903+0000","last_became_active":"2023-10-19T14:21:10.422047+0000","last_became_peered":"2023-10-19T14:21:10.422047+0000","last_unstale":"2023-10-19T14:21:11.410903+0000","last_undegraded":"2023-10-19T14:21:11.410903+0000","last_fullsized":"2023-10-19T14:21:11.410903+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:35:21.048326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.1a","version":"242'1","reported_seq":13,"reported_epoch":242,"state":"active+clean","last_fresh":"2023-10-19T14:21:10.439565+0000","last_change":"2023-10-19T14:21:09.408272+0000","last_active":"2023-10-19T14:21:10.439565+0000","last_peered":"2023-10-19T14:21:10.439565+0000","last_clean":"2023-10-19T14:21:10.439565+0000","last_became_active":"2023-10-19T14:21:09.407997+0000","last_became_peered":"2023-10-19T14:21:09.407997+0000","last_unstale":"2023-10-19T14:21:10.439565+0000","last_undegraded":"2023-10-19T14:21:10.439565+0000","last_fullsized":"2023-10-19T14:21:10.439565+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:32:11.516962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.26","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:12.059150+0000","last_change":"2023-10-19T14:21:10.427832+0000","last_active":"2023-10-19T14:21:12.059150+0000","last_peered":"2023-10-19T14:21:12.059150+0000","last_clean":"2023-10-19T14:21:12.059150+0000","last_became_active":"2023-10-19T14:21:10.427540+0000","last_became_peered":"2023-10-19T14:21:10.427540+0000","last_unstale":"2023-10-19T14:21:12.059150+0000","last_undegraded":"2023-10-19T14:21:12.059150+0000","last_fullsized":"2023-10-19T14:21:12.059150+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:47:17.356392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2023-10-19T14:21:10.422345+0000","last_change":"2023-10-19T14:21:09.408331+0000","last_active":"2023-10-19T14:21:10.422345+0000","last_peered":"2023-10-19T14:21:10.422345+0000","last_clean":"2023-10-19T14:21:10.422345+0000","last_became_active":"2023-10-19T14:21:09.407722+0000","last_became_peered":"2023-10-19T14:21:09.407722+0000","last_unstale":"2023-10-19T14:21:10.422345+0000","last_undegraded":"2023-10-19T14:21:10.422345+0000","last_fullsized":"2023-10-19T14:21:10.422345+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:06:49.485559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.27","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:11.522145+0000","last_change":"2023-10-19T14:21:10.425541+0000","last_active":"2023-10-19T14:21:11.522145+0000","last_peered":"2023-10-19T14:21:11.522145+0000","last_clean":"2023-10-19T14:21:11.522145+0000","last_became_active":"2023-10-19T14:21:10.423860+0000","last_became_peered":"2023-10-19T14:21:10.423860+0000","last_unstale":"2023-10-19T14:21:11.522145+0000","last_undegraded":"2023-10-19T14:21:11.522145+0000","last_fullsized":"2023-10-19T14:21:11.522145+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:41:24.205533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2023-10-19T14:21:10.417790+0000","last_change":"2023-10-19T14:21:09.411121+0000","last_active":"2023-10-19T14:21:10.417790+0000","last_peered":"2023-10-19T14:21:10.417790+0000","last_clean":"2023-10-19T14:21:10.417790+0000","last_became_active":"2023-10-19T14:21:09.409416+0000","last_became_peered":"2023-10-19T14:21:09.409416+0000","last_unstale":"2023-10-19T14:21:10.417790+0000","last_undegraded":"2023-10-19T14:21:10.417790+0000","last_fullsized":"2023-10-19T14:21:10.417790+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:09:01.088634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.28","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:11.410895+0000","last_change":"2023-10-19T14:21:10.422391+0000","last_active":"2023-10-19T14:21:11.410895+0000","last_peered":"2023-10-19T14:21:11.410895+0000","last_clean":"2023-10-19T14:21:11.410895+0000","last_became_active":"2023-10-19T14:21:10.422118+0000","last_became_peered":"2023-10-19T14:21:10.422118+0000","last_unstale":"2023-10-19T14:21:11.410895+0000","last_undegraded":"2023-10-19T14:21:11.410895+0000","last_fullsized":"2023-10-19T14:21:11.410895+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T02:12:47.154435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2023-10-19T14:21:10.414279+0000","last_change":"2023-10-19T14:21:09.412983+0000","last_active":"2023-10-19T14:21:10.414279+0000","last_peered":"2023-10-19T14:21:10.414279+0000","last_clean":"2023-10-19T14:21:10.414279+0000","last_became_active":"2023-10-19T14:21:09.412690+0000","last_became_peered":"2023-10-19T14:21:09.412690+0000","last_unstale":"2023-10-19T14:21:10.414279+0000","last_undegraded":"2023-10-19T14:21:10.414279+0000","last_fullsized":"2023-10-19T14:21:10.414279+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:21:42.383177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.29","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:12.058829+0000","last_change":"2023-10-19T14:21:10.427650+0000","last_active":"2023-10-19T14:21:12.058829+0000","last_peered":"2023-10-19T14:21:12.058829+0000","last_clean":"2023-10-19T14:21:12.058829+0000","last_became_active":"2023-10-19T14:21:10.427367+0000","last_became_peered":"2023-10-19T14:21:10.427367+0000","last_unstale":"2023-10-19T14:21:12.058829+0000","last_undegraded":"2023-10-19T14:21:12.058829+0000","last_fullsized":"2023-10-19T14:21:12.058829+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:54:17.097512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2023-10-19T14:21:10.421178+0000","last_change":"2023-10-19T14:21:09.408912+0000","last_active":"2023-10-19T14:21:10.421178+0000","last_peered":"2023-10-19T14:21:10.421178+0000","last_clean":"2023-10-19T14:21:10.421178+0000","last_became_active":"2023-10-19T14:21:09.408661+0000","last_became_peered":"2023-10-19T14:21:09.408661+0000","last_unstale":"2023-10-19T14:21:10.421178+0000","last_undegraded":"2023-10-19T14:21:10.421178+0000","last_fullsized":"2023-10-19T14:21:10.421178+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:55:16.857948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.2a","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:12.058773+0000","last_change":"2023-10-19T14:21:10.430153+0000","last_active":"2023-10-19T14:21:12.058773+0000","last_peered":"2023-10-19T14:21:12.058773+0000","last_clean":"2023-10-19T14:21:12.058773+0000","last_became_active":"2023-10-19T14:21:10.429868+0000","last_became_peered":"2023-10-19T14:21:10.429868+0000","last_unstale":"2023-10-19T14:21:12.058773+0000","last_undegraded":"2023-10-19T14:21:12.058773+0000","last_fullsized":"2023-10-19T14:21:12.058773+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:55:49.837750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.15","version":"242'3","reported_seq":15,"reported_epoch":242,"state":"active+clean","last_fresh":"2023-10-19T14:21:10.439239+0000","last_change":"2023-10-19T14:21:09.414620+0000","last_active":"2023-10-19T14:21:10.439239+0000","last_peered":"2023-10-19T14:21:10.439239+0000","last_clean":"2023-10-19T14:21:10.439239+0000","last_became_active":"2023-10-19T14:21:09.414378+0000","last_became_peered":"2023-10-19T14:21:09.414378+0000","last_unstale":"2023-10-19T14:21:10.439239+0000","last_undegraded":"2023-10-19T14:21:10.439239+0000","last_fullsized":"2023-10-19T14:21:10.439239+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:16:10.702399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2b","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:11.410878+0000","last_change":"2023-10-19T14:21:10.436778+0000","last_active":"2023-10-19T14:21:11.410878+0000","last_peered":"2023-10-19T14:21:11.410878+0000","last_clean":"2023-10-19T14:21:11.410878+0000","last_became_active":"2023-10-19T14:21:10.436506+0000","last_became_peered":"2023-10-19T14:21:10.436506+0000","last_unstale":"2023-10-19T14:21:11.410878+0000","last_undegraded":"2023-10-19T14:21:11.410878+0000","last_fullsized":"2023-10-19T14:21:11.410878+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:02:47.768820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.14","version":"242'6","reported_seq":18,"reported_epoch":242,"state":"active+clean","last_fresh":"2023-10-19T14:21:10.450638+0000","last_change":"2023-10-19T14:21:09.411345+0000","last_active":"2023-10-19T14:21:10.450638+0000","last_peered":"2023-10-19T14:21:10.450638+0000","last_clean":"2023-10-19T14:21:10.450638+0000","last_became_active":"2023-10-19T14:21:09.410711+0000","last_became_peered":"2023-10-19T14:21:09.410711+0000","last_unstale":"2023-10-19T14:21:10.450638+0000","last_undegraded":"2023-10-19T14:21:10.450638+0000","last_fullsized":"2023-10-19T14:21:10.450638+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:35:21.400082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.2c","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:11.414772+0000","last_change":"2023-10-19T14:21:10.435528+0000","last_active":"2023-10-19T14:21:11.414772+0000","last_peered":"2023-10-19T14:21:11.414772+0000","last_clean":"2023-10-19T14:21:11.414772+0000","last_became_active":"2023-10-19T14:21:10.435235+0000","last_became_peered":"2023-10-19T14:21:10.435235+0000","last_unstale":"2023-10-19T14:21:11.414772+0000","last_undegraded":"2023-10-19T14:21:11.414772+0000","last_fullsized":"2023-10-19T14:21:11.414772+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:24:42.687537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.13","version":"242'2","reported_seq":14,"reported_epoch":242,"state":"active+clean","last_fresh":"2023-10-19T14:21:10.451084+0000","last_change":"2023-10-19T14:21:09.413386+0000","last_active":"2023-10-19T14:21:10.451084+0000","last_peered":"2023-10-19T14:21:10.451084+0000","last_clean":"2023-10-19T14:21:10.451084+0000","last_became_active":"2023-10-19T14:21:09.413119+0000","last_became_peered":"2023-10-19T14:21:09.413119+0000","last_unstale":"2023-10-19T14:21:10.451084+0000","last_undegraded":"2023-10-19T14:21:10.451084+0000","last_fullsized":"2023-10-19T14:21:10.451084+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:32:53.438251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.2d","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:11.410836+0000","last_change":"2023-10-19T14:21:10.417244+0000","last_active":"2023-10-19T14:21:11.410836+0000","last_peered":"2023-10-19T14:21:11.410836+0000","last_clean":"2023-10-19T14:21:11.410836+0000","last_became_active":"2023-10-19T14:21:10.416948+0000","last_became_peered":"2023-10-19T14:21:10.416948+0000","last_unstale":"2023-10-19T14:21:11.410836+0000","last_undegraded":"2023-10-19T14:21:11.410836+0000","last_fullsized":"2023-10-19T14:21:11.410836+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:13:29.800561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2023-10-19T14:21:10.418418+0000","last_change":"2023-10-19T14:21:09.417189+0000","last_active":"2023-10-19T14:21:10.418418+0000","last_peered":"2023-10-19T14:21:10.418418+0000","last_clean":"2023-10-19T14:21:10.418418+0000","last_became_active":"2023-10-19T14:21:09.416920+0000","last_became_peered":"2023-10-19T14:21:09.416920+0000","last_unstale":"2023-10-19T14:21:10.418418+0000","last_undegraded":"2023-10-19T14:21:10.418418+0000","last_fullsized":"2023-10-19T14:21:10.418418+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:17:36.285741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2e","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:11.628407+0000","last_change":"2023-10-19T14:21:10.422922+0000","last_active":"2023-10-19T14:21:11.628407+0000","last_peered":"2023-10-19T14:21:11.628407+0000","last_clean":"2023-10-19T14:21:11.628407+0000","last_became_active":"2023-10-19T14:21:10.422656+0000","last_became_peered":"2023-10-19T14:21:10.422656+0000","last_unstale":"2023-10-19T14:21:11.628407+0000","last_undegraded":"2023-10-19T14:21:11.628407+0000","last_fullsized":"2023-10-19T14:21:11.628407+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:16:55.773348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2023-10-19T14:21:10.418781+0000","last_change":"2023-10-19T14:21:09.413176+0000","last_active":"2023-10-19T14:21:10.418781+0000","last_peered":"2023-10-19T14:21:10.418781+0000","last_clean":"2023-10-19T14:21:10.418781+0000","last_became_active":"2023-10-19T14:21:09.412906+0000","last_became_peered":"2023-10-19T14:21:09.412906+0000","last_unstale":"2023-10-19T14:21:10.418781+0000","last_undegraded":"2023-10-19T14:21:10.418781+0000","last_fullsized":"2023-10-19T14:21:10.418781+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:07:02.508240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2f","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:11.522117+0000","last_change":"2023-10-19T14:21:10.431716+0000","last_active":"2023-10-19T14:21:11.522117+0000","last_peered":"2023-10-19T14:21:11.522117+0000","last_clean":"2023-10-19T14:21:11.522117+0000","last_became_active":"2023-10-19T14:21:10.430784+0000","last_became_peered":"2023-10-19T14:21:10.430784+0000","last_unstale":"2023-10-19T14:21:11.522117+0000","last_undegraded":"2023-10-19T14:21:11.522117+0000","last_fullsized":"2023-10-19T14:21:11.522117+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:14:44.806142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.10","version":"242'8","reported_seq":20,"reported_epoch":242,"state":"active+clean","last_fresh":"2023-10-19T14:21:10.451476+0000","last_change":"2023-10-19T14:21:09.415001+0000","last_active":"2023-10-19T14:21:10.451476+0000","last_peered":"2023-10-19T14:21:10.451476+0000","last_clean":"2023-10-19T14:21:10.451476+0000","last_became_active":"2023-10-19T14:21:09.414727+0000","last_became_peered":"2023-10-19T14:21:09.414727+0000","last_unstale":"2023-10-19T14:21:10.451476+0000","last_undegraded":"2023-10-19T14:21:10.451476+0000","last_fullsized":"2023-10-19T14:21:10.451476+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:54:37.202858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.30","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:11.521882+0000","last_change":"2023-10-19T14:21:10.437415+0000","last_active":"2023-10-19T14:21:11.521882+0000","last_peered":"2023-10-19T14:21:11.521882+0000","last_clean":"2023-10-19T14:21:11.521882+0000","last_became_active":"2023-10-19T14:21:10.437137+0000","last_became_peered":"2023-10-19T14:21:10.437137+0000","last_unstale":"2023-10-19T14:21:11.521882+0000","last_undegraded":"2023-10-19T14:21:11.521882+0000","last_fullsized":"2023-10-19T14:21:11.521882+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:39:56.303696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.f","version":"242'1","reported_seq":13,"reported_epoch":242,"state":"active+clean","last_fresh":"2023-10-19T14:21:10.445085+0000","last_change":"2023-10-19T14:21:09.417455+0000","last_active":"2023-10-19T14:21:10.445085+0000","last_peered":"2023-10-19T14:21:10.445085+0000","last_clean":"2023-10-19T14:21:10.445085+0000","last_became_active":"2023-10-19T14:21:09.417228+0000","last_became_peered":"2023-10-19T14:21:09.417228+0000","last_unstale":"2023-10-19T14:21:10.445085+0000","last_undegraded":"2023-10-19T14:21:10.445085+0000","last_fullsized":"2023-10-19T14:21:10.445085+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:37:45.851912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.31","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:11.521633+0000","last_change":"2023-10-19T14:21:10.423316+0000","last_active":"2023-10-19T14:21:11.521633+0000","last_peered":"2023-10-19T14:21:11.521633+0000","last_clean":"2023-10-19T14:21:11.521633+0000","last_became_active":"2023-10-19T14:21:10.423028+0000","last_became_peered":"2023-10-19T14:21:10.423028+0000","last_unstale":"2023-10-19T14:21:11.521633+0000","last_undegraded":"2023-10-19T14:21:11.521633+0000","last_fullsized":"2023-10-19T14:21:11.521633+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:39:05.601924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.e","version":"242'2","reported_seq":14,"reported_epoch":242,"state":"active+clean","last_fresh":"2023-10-19T14:21:10.450825+0000","last_change":"2023-10-19T14:21:09.409551+0000","last_active":"2023-10-19T14:21:10.450825+0000","last_peered":"2023-10-19T14:21:10.450825+0000","last_clean":"2023-10-19T14:21:10.450825+0000","last_became_active":"2023-10-19T14:21:09.409238+0000","last_became_peered":"2023-10-19T14:21:09.409238+0000","last_unstale":"2023-10-19T14:21:10.450825+0000","last_undegraded":"2023-10-19T14:21:10.450825+0000","last_fullsized":"2023-10-19T14:21:10.450825+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:38:52.516509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.32","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:11.414657+0000","last_change":"2023-10-19T14:21:10.433187+0000","last_active":"2023-10-19T14:21:11.414657+0000","last_peered":"2023-10-19T14:21:11.414657+0000","last_clean":"2023-10-19T14:21:11.414657+0000","last_became_active":"2023-10-19T14:21:10.432876+0000","last_became_peered":"2023-10-19T14:21:10.432876+0000","last_unstale":"2023-10-19T14:21:11.414657+0000","last_undegraded":"2023-10-19T14:21:11.414657+0000","last_fullsized":"2023-10-19T14:21:11.414657+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:49:35.448257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.d","version":"242'1","reported_seq":13,"reported_epoch":242,"state":"active+clean","last_fresh":"2023-10-19T14:21:10.440231+0000","last_change":"2023-10-19T14:21:09.413076+0000","last_active":"2023-10-19T14:21:10.440231+0000","last_peered":"2023-10-19T14:21:10.440231+0000","last_clean":"2023-10-19T14:21:10.440231+0000","last_became_active":"2023-10-19T14:21:09.412330+0000","last_became_peered":"2023-10-19T14:21:09.412330+0000","last_unstale":"2023-10-19T14:21:10.440231+0000","last_undegraded":"2023-10-19T14:21:10.440231+0000","last_fullsized":"2023-10-19T14:21:10.440231+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:20:50.841124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.33","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:12.058817+0000","last_change":"2023-10-19T14:21:10.429531+0000","last_active":"2023-10-19T14:21:12.058817+0000","last_peered":"2023-10-19T14:21:12.058817+0000","last_clean":"2023-10-19T14:21:12.058817+0000","last_became_active":"2023-10-19T14:21:10.429261+0000","last_became_peered":"2023-10-19T14:21:10.429261+0000","last_unstale":"2023-10-19T14:21:12.058817+0000","last_undegraded":"2023-10-19T14:21:12.058817+0000","last_fullsized":"2023-10-19T14:21:12.058817+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:29:47.656632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2023-10-19T14:21:10.417666+0000","last_change":"2023-10-19T14:21:09.409030+0000","last_active":"2023-10-19T14:21:10.417666+0000","last_peered":"2023-10-19T14:21:10.417666+0000","last_clean":"2023-10-19T14:21:10.417666+0000","last_became_active":"2023-10-19T14:21:09.408717+0000","last_became_peered":"2023-10-19T14:21:09.408717+0000","last_unstale":"2023-10-19T14:21:10.417666+0000","last_undegraded":"2023-10-19T14:21:10.417666+0000","last_fullsized":"2023-10-19T14:21:10.417666+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:04:32.926013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.34","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:12.058794+0000","last_change":"2023-10-19T14:21:10.437932+0000","last_active":"2023-10-19T14:21:12.058794+0000","last_peered":"2023-10-19T14:21:12.058794+0000","last_clean":"2023-10-19T14:21:12.058794+0000","last_became_active":"2023-10-19T14:21:10.437668+0000","last_became_peered":"2023-10-19T14:21:10.437668+0000","last_unstale":"2023-10-19T14:21:12.058794+0000","last_undegraded":"2023-10-19T14:21:12.058794+0000","last_fullsized":"2023-10-19T14:21:12.058794+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:04:33.936613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.b","version":"242'2","reported_seq":14,"reported_epoch":242,"state":"active+clean","last_fresh":"2023-10-19T14:21:10.447660+0000","last_change":"2023-10-19T14:21:09.415147+0000","last_active":"2023-10-19T14:21:10.447660+0000","last_peered":"2023-10-19T14:21:10.447660+0000","last_clean":"2023-10-19T14:21:10.447660+0000","last_became_active":"2023-10-19T14:21:09.414852+0000","last_became_peered":"2023-10-19T14:21:09.414852+0000","last_unstale":"2023-10-19T14:21:10.447660+0000","last_undegraded":"2023-10-19T14:21:10.447660+0000","last_fullsized":"2023-10-19T14:21:10.447660+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:53:33.599617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.35","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:11.414824+0000","last_change":"2023-10-19T14:21:10.439769+0000","last_active":"2023-10-19T14:21:11.414824+0000","last_peered":"2023-10-19T14:21:11.414824+0000","last_clean":"2023-10-19T14:21:11.414824+0000","last_became_active":"2023-10-19T14:21:10.439056+0000","last_became_peered":"2023-10-19T14:21:10.439056+0000","last_unstale":"2023-10-19T14:21:11.414824+0000","last_undegraded":"2023-10-19T14:21:11.414824+0000","last_fullsized":"2023-10-19T14:21:11.414824+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:18:35.286014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2023-10-19T14:21:10.411249+0000","last_change":"2023-10-19T14:21:09.411613+0000","last_active":"2023-10-19T14:21:10.411249+0000","last_peered":"2023-10-19T14:21:10.411249+0000","last_clean":"2023-10-19T14:21:10.411249+0000","last_became_active":"2023-10-19T14:21:09.411365+0000","last_became_peered":"2023-10-19T14:21:09.411365+0000","last_unstale":"2023-10-19T14:21:10.411249+0000","last_undegraded":"2023-10-19T14:21:10.411249+0000","last_fullsized":"2023-10-19T14:21:10.411249+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:22:34.404197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.36","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:12.059108+0000","last_change":"2023-10-19T14:21:10.424290+0000","last_active":"2023-10-19T14:21:12.059108+0000","last_peered":"2023-10-19T14:21:12.059108+0000","last_clean":"2023-10-19T14:21:12.059108+0000","last_became_active":"2023-10-19T14:21:10.423951+0000","last_became_peered":"2023-10-19T14:21:10.423951+0000","last_unstale":"2023-10-19T14:21:12.059108+0000","last_undegraded":"2023-10-19T14:21:12.059108+0000","last_fullsized":"2023-10-19T14:21:12.059108+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:43:48.977855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2023-10-19T14:21:10.410485+0000","last_change":"2023-10-19T14:21:09.410869+0000","last_active":"2023-10-19T14:21:10.410485+0000","last_peered":"2023-10-19T14:21:10.410485+0000","last_clean":"2023-10-19T14:21:10.410485+0000","last_became_active":"2023-10-19T14:21:09.410586+0000","last_became_peered":"2023-10-19T14:21:09.410586+0000","last_unstale":"2023-10-19T14:21:10.410485+0000","last_undegraded":"2023-10-19T14:21:10.410485+0000","last_fullsized":"2023-10-19T14:21:10.410485+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:02:42.839507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.37","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:12.059139+0000","last_change":"2023-10-19T14:21:10.422504+0000","last_active":"2023-10-19T14:21:12.059139+0000","last_peered":"2023-10-19T14:21:12.059139+0000","last_clean":"2023-10-19T14:21:12.059139+0000","last_became_active":"2023-10-19T14:21:10.422196+0000","last_became_peered":"2023-10-19T14:21:10.422196+0000","last_unstale":"2023-10-19T14:21:12.059139+0000","last_undegraded":"2023-10-19T14:21:12.059139+0000","last_fullsized":"2023-10-19T14:21:12.059139+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:53:46.360770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2023-10-19T14:21:10.413826+0000","last_change":"2023-10-19T14:21:09.415870+0000","last_active":"2023-10-19T14:21:10.413826+0000","last_peered":"2023-10-19T14:21:10.413826+0000","last_clean":"2023-10-19T14:21:10.413826+0000","last_became_active":"2023-10-19T14:21:09.415609+0000","last_became_peered":"2023-10-19T14:21:09.415609+0000","last_unstale":"2023-10-19T14:21:10.413826+0000","last_undegraded":"2023-10-19T14:21:10.413826+0000","last_fullsized":"2023-10-19T14:21:10.413826+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:29:10.891884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.38","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:11.410640+0000","last_change":"2023-10-19T14:21:10.428770+0000","last_active":"2023-10-19T14:21:11.410640+0000","last_peered":"2023-10-19T14:21:11.410640+0000","last_clean":"2023-10-19T14:21:11.410640+0000","last_became_active":"2023-10-19T14:21:10.428494+0000","last_became_peered":"2023-10-19T14:21:10.428494+0000","last_unstale":"2023-10-19T14:21:11.410640+0000","last_undegraded":"2023-10-19T14:21:11.410640+0000","last_fullsized":"2023-10-19T14:21:11.410640+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:54:47.763268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.7","version":"242'2","reported_seq":14,"reported_epoch":242,"state":"active+clean","last_fresh":"2023-10-19T14:21:10.447794+0000","last_change":"2023-10-19T14:21:09.416206+0000","last_active":"2023-10-19T14:21:10.447794+0000","last_peered":"2023-10-19T14:21:10.447794+0000","last_clean":"2023-10-19T14:21:10.447794+0000","last_became_active":"2023-10-19T14:21:09.415929+0000","last_became_peered":"2023-10-19T14:21:09.415929+0000","last_unstale":"2023-10-19T14:21:10.447794+0000","last_undegraded":"2023-10-19T14:21:10.447794+0000","last_fullsized":"2023-10-19T14:21:10.447794+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:41:27.899602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.39","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:11.628206+0000","last_change":"2023-10-19T14:21:10.431238+0000","last_active":"2023-10-19T14:21:11.628206+0000","last_peered":"2023-10-19T14:21:11.628206+0000","last_clean":"2023-10-19T14:21:11.628206+0000","last_became_active":"2023-10-19T14:21:10.430970+0000","last_became_peered":"2023-10-19T14:21:10.430970+0000","last_unstale":"2023-10-19T14:21:11.628206+0000","last_undegraded":"2023-10-19T14:21:11.628206+0000","last_fullsized":"2023-10-19T14:21:11.628206+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:29:11.902484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.6","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2023-10-19T14:21:10.418749+0000","last_change":"2023-10-19T14:21:09.415597+0000","last_active":"2023-10-19T14:21:10.418749+0000","last_peered":"2023-10-19T14:21:10.418749+0000","last_clean":"2023-10-19T14:21:10.418749+0000","last_became_active":"2023-10-19T14:21:09.415325+0000","last_became_peered":"2023-10-19T14:21:09.415325+0000","last_unstale":"2023-10-19T14:21:10.418749+0000","last_undegraded":"2023-10-19T14:21:10.418749+0000","last_fullsized":"2023-10-19T14:21:10.418749+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:47:10.402857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.3a","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:12.058632+0000","last_change":"2023-10-19T14:21:10.430013+0000","last_active":"2023-10-19T14:21:12.058632+0000","last_peered":"2023-10-19T14:21:12.058632+0000","last_clean":"2023-10-19T14:21:12.058632+0000","last_became_active":"2023-10-19T14:21:10.429744+0000","last_became_peered":"2023-10-19T14:21:10.429744+0000","last_unstale":"2023-10-19T14:21:12.058632+0000","last_undegraded":"2023-10-19T14:21:12.058632+0000","last_fullsized":"2023-10-19T14:21:12.058632+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:23:00.738206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2023-10-19T14:21:10.414280+0000","last_change":"2023-10-19T14:21:09.412015+0000","last_active":"2023-10-19T14:21:10.414280+0000","last_peered":"2023-10-19T14:21:10.414280+0000","last_clean":"2023-10-19T14:21:10.414280+0000","last_became_active":"2023-10-19T14:21:09.411729+0000","last_became_peered":"2023-10-19T14:21:09.411729+0000","last_unstale":"2023-10-19T14:21:10.414280+0000","last_undegraded":"2023-10-19T14:21:10.414280+0000","last_fullsized":"2023-10-19T14:21:10.414280+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:50:33.137069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.3b","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:11.521905+0000","last_change":"2023-10-19T14:21:10.430397+0000","last_active":"2023-10-19T14:21:11.521905+0000","last_peered":"2023-10-19T14:21:11.521905+0000","last_clean":"2023-10-19T14:21:11.521905+0000","last_became_active":"2023-10-19T14:21:10.430094+0000","last_became_peered":"2023-10-19T14:21:10.430094+0000","last_unstale":"2023-10-19T14:21:11.521905+0000","last_undegraded":"2023-10-19T14:21:11.521905+0000","last_fullsized":"2023-10-19T14:21:11.521905+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:02:35.892624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2023-10-19T14:21:10.414646+0000","last_change":"2023-10-19T14:21:09.414910+0000","last_active":"2023-10-19T14:21:10.414646+0000","last_peered":"2023-10-19T14:21:10.414646+0000","last_clean":"2023-10-19T14:21:10.414646+0000","last_became_active":"2023-10-19T14:21:09.414646+0000","last_became_peered":"2023-10-19T14:21:09.414646+0000","last_unstale":"2023-10-19T14:21:10.414646+0000","last_undegraded":"2023-10-19T14:21:10.414646+0000","last_fullsized":"2023-10-19T14:21:10.414646+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:49:25.343751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.3c","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:12.058375+0000","last_change":"2023-10-19T14:21:10.418399+0000","last_active":"2023-10-19T14:21:12.058375+0000","last_peered":"2023-10-19T14:21:12.058375+0000","last_clean":"2023-10-19T14:21:12.058375+0000","last_became_active":"2023-10-19T14:21:10.416779+0000","last_became_peered":"2023-10-19T14:21:10.416779+0000","last_unstale":"2023-10-19T14:21:12.058375+0000","last_undegraded":"2023-10-19T14:21:12.058375+0000","last_fullsized":"2023-10-19T14:21:12.058375+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:50:13.187179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2023-10-19T14:21:10.411090+0000","last_change":"2023-10-19T14:21:09.412344+0000","last_active":"2023-10-19T14:21:10.411090+0000","last_peered":"2023-10-19T14:21:10.411090+0000","last_clean":"2023-10-19T14:21:10.411090+0000","last_became_active":"2023-10-19T14:21:09.412101+0000","last_became_peered":"2023-10-19T14:21:09.412101+0000","last_unstale":"2023-10-19T14:21:10.411090+0000","last_undegraded":"2023-10-19T14:21:10.411090+0000","last_fullsized":"2023-10-19T14:21:10.411090+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:51:23.060110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.3d","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:11.413772+0000","last_change":"2023-10-19T14:21:10.435707+0000","last_active":"2023-10-19T14:21:11.413772+0000","last_peered":"2023-10-19T14:21:11.413772+0000","last_clean":"2023-10-19T14:21:11.413772+0000","last_became_active":"2023-10-19T14:21:10.435426+0000","last_became_peered":"2023-10-19T14:21:10.435426+0000","last_unstale":"2023-10-19T14:21:11.413772+0000","last_undegraded":"2023-10-19T14:21:11.413772+0000","last_fullsized":"2023-10-19T14:21:11.413772+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:21:50.005281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2023-10-19T14:21:10.418546+0000","last_change":"2023-10-19T14:21:09.415572+0000","last_active":"2023-10-19T14:21:10.418546+0000","last_peered":"2023-10-19T14:21:10.418546+0000","last_clean":"2023-10-19T14:21:10.418546+0000","last_became_active":"2023-10-19T14:21:09.415271+0000","last_became_peered":"2023-10-19T14:21:09.415271+0000","last_unstale":"2023-10-19T14:21:10.418546+0000","last_undegraded":"2023-10-19T14:21:10.418546+0000","last_fullsized":"2023-10-19T14:21:10.418546+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:24:02.734433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.3e","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:11.414939+0000","last_change":"2023-10-19T14:21:10.439966+0000","last_active":"2023-10-19T14:21:11.414939+0000","last_peered":"2023-10-19T14:21:11.414939+0000","last_clean":"2023-10-19T14:21:11.414939+0000","last_became_active":"2023-10-19T14:21:10.438773+0000","last_became_peered":"2023-10-19T14:21:10.438773+0000","last_unstale":"2023-10-19T14:21:11.414939+0000","last_undegraded":"2023-10-19T14:21:11.414939+0000","last_fullsized":"2023-10-19T14:21:11.414939+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:42:16.397379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2023-10-19T14:21:10.422469+0000","last_change":"2023-10-19T14:21:09.409229+0000","last_active":"2023-10-19T14:21:10.422469+0000","last_peered":"2023-10-19T14:21:10.422469+0000","last_clean":"2023-10-19T14:21:10.422469+0000","last_became_active":"2023-10-19T14:21:09.408885+0000","last_became_peered":"2023-10-19T14:21:09.408885+0000","last_unstale":"2023-10-19T14:21:10.422469+0000","last_undegraded":"2023-10-19T14:21:10.422469+0000","last_fullsized":"2023-10-19T14:21:10.422469+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:05:37.351111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.3f","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:12.059061+0000","last_change":"2023-10-19T14:21:10.437868+0000","last_active":"2023-10-19T14:21:12.059061+0000","last_peered":"2023-10-19T14:21:12.059061+0000","last_clean":"2023-10-19T14:21:12.059061+0000","last_became_active":"2023-10-19T14:21:10.437535+0000","last_became_peered":"2023-10-19T14:21:10.437535+0000","last_unstale":"2023-10-19T14:21:12.059061+0000","last_undegraded":"2023-10-19T14:21:12.059061+0000","last_fullsized":"2023-10-19T14:21:12.059061+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:44:47.905654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":12,"reported_epoch":242,"state":"active+clean","last_fresh":"2023-10-19T14:21:10.410436+0000","last_change":"2023-10-19T14:21:09.407963+0000","last_active":"2023-10-19T14:21:10.410436+0000","last_peered":"2023-10-19T14:21:10.410436+0000","last_clean":"2023-10-19T14:21:10.410436+0000","last_became_active":"2023-10-19T14:21:09.407684+0000","last_became_peered":"2023-10-19T14:21:09.407684+0000","last_unstale":"2023-10-19T14:21:10.410436+0000","last_undegraded":"2023-10-19T14:21:10.410436+0000","last_fullsized":"2023-10-19T14:21:10.410436+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:45:20.541560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.21","version":"0'0","reported_seq":11,"reported_epoch":243,"state":"active+clean","last_fresh":"2023-10-19T14:21:11.413672+0000","last_change":"2023-10-19T14:21:10.434210+0000","last_active":"2023-10-19T14:21:11.413672+0000","last_peered":"2023-10-19T14:21:11.413672+0000","last_clean":"2023-10-19T14:21:11.413672+0000","last_became_active":"2023-10-19T14:21:10.433919+0000","last_became_peered":"2023-10-19T14:21:10.433919+0000","last_unstale":"2023-10-19T14:21:11.413672+0000","last_undegraded":"2023-10-19T14:21:11.413672+0000","last_fullsized":"2023-10-19T14:21:11.413672+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:56:57.148327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1e","version":"242'1","reported_seq":13,"reported_epoch":242,"state":"active+clean","last_fresh":"2023-10-19T14:21:10.440196+0000","last_change":"2023-10-19T14:21:09.411855+0000","last_active":"2023-10-19T14:21:10.440196+0000","last_peered":"2023-10-19T14:21:10.440196+0000","last_clean":"2023-10-19T14:21:10.440196+0000","last_became_active":"2023-10-19T14:21:09.411510+0000","last_became_peered":"2023-10-19T14:21:09.411510+0000","last_unstale":"2023-10-19T14:21:10.440196+0000","last_undegraded":"2023-10-19T14:21:10.440196+0000","last_fullsized":"2023-10-19T14:21:10.440196+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:44:43.405619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"229'3208","reported_seq":4180,"reported_epoch":242,"state":"active+clean","last_fresh":"2023-10-19T14:21:10.418626+0000","last_change":"2023-10-19T12:44:08.636613+0000","last_active":"2023-10-19T14:21:10.418626+0000","last_peered":"2023-10-19T14:21:10.418626+0000","last_clean":"2023-10-19T14:21:10.418626+0000","last_became_active":"2023-10-19T12:44:08.635697+0000","last_became_peered":"2023-10-19T12:44:08.635697+0000","last_unstale":"2023-10-19T14:21:10.418626+0000","last_undegraded":"2023-10-19T14:21:10.418626+0000","last_fullsized":"2023-10-19T14:21:10.418626+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_clean_scrub_stamp":"2023-10-19T12:44:07.399738+0000","objects_scrubbed":0,"log_size":3208,"log_dups_size":0,"ondisk_log_size":3208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:15:23.685485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2402,"num_read_kb":5036,"num_write":629,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":32,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":31,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2402,"num_read_kb":5036,"num_write":629,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3208,"ondisk_log_size":3208,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576011,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66696,"kb_used_data":3580,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305144,"statfs":{"total":96636764160,"available":96568467456,"internally_reserved":0,"allocated":3665920,"data_stored":2166572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":64616907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:21:10 2023","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.55100000000000005,"15min":0.57999999999999996},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.32200000000000001},"max":{"1min":0.754,"5min":0.80900000000000005,"15min":0.86199999999999999},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.503,"5min":0.57599999999999996,"15min":0.59899999999999998},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.312},"max":{"1min":0.66500000000000004,"5min":0.81899999999999995,"15min":0.86599999999999999},"last":0.504}]},{"osd":1,"last update":"Thu Oct 19 14:21:10 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.58199999999999996,"15min":0.59499999999999997},"min":{"1min":0.45200000000000001,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.72199999999999998,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.56899999999999995,"15min":0.59299999999999997},"min":{"1min":0.39300000000000002,"5min":0.373,"15min":0.36699999999999999},"max":{"1min":0.78500000000000003,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.58599999999999997}]},{"osd":2,"last update":"Thu Oct 19 14:21:10 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.622,"15min":0.64200000000000002},"min":{"1min":0.47299999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.70499999999999996,"5min":0.84799999999999998,"15min":0.85499999999999998},"last":0.628},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.63900000000000001,"15min":0.65700000000000003},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.71799999999999997}]},{"osd":3,"last update":"Thu Oct 19 14:21:10 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.60799999999999998,"15min":0.63100000000000001},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.628,"5min":0.67100000000000004,"15min":0.68300000000000005},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.83299999999999996,"5min":0.876,"15min":0.89600000000000002},"last":0.70499999999999996}]},{"osd":4,"last update":"Thu Oct 19 14:21:10 2023","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.50600000000000001,"15min":0.495},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.53000000000000003,"15min":0.51100000000000001},"min":{"1min":0.34999999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.73599999999999999}]},{"osd":5,"last update":"Thu Oct 19 14:21:10 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52300000000000002,"15min":0.51100000000000001},"min":{"1min":0.33700000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.57799999999999996,"15min":0.55400000000000005},"min":{"1min":0.40699999999999997,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.054,"5min":1.054,"15min":1.054},"last":0.76600000000000001}]},{"osd":6,"last update":"Thu Oct 19 14:21:10 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.58999999999999997,"15min":0.57599999999999996},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.35999999999999999},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.86299999999999999},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.59299999999999997,"15min":0.57299999999999995},"min":{"1min":0.42999999999999999,"5min":0.35599999999999998,"15min":0.33500000000000002},"max":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.84999999999999998}]}]},{"osd":6,"up_from":13,"seq":55834576011,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38344,"kb_used_data":3000,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333496,"statfs":{"total":96636764160,"available":96597499904,"internally_reserved":0,"allocated":3072000,"data_stored":1576192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:21:02 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.59199999999999997,"15min":0.61099999999999999},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.78400000000000003,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.56299999999999994,"15min":0.60599999999999998},"min":{"1min":0.36699999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.71799999999999997,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.51600000000000001}]},{"osd":1,"last update":"Thu Oct 19 14:21:02 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.60099999999999998,"15min":0.61799999999999999},"min":{"1min":0.373,"5min":0.373,"15min":0.35799999999999998},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":1.165},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.60199999999999998,"15min":0.626},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.77400000000000002,"5min":0.83699999999999997,"15min":0.91300000000000003},"last":0.58099999999999996}]},{"osd":2,"last update":"Thu Oct 19 14:21:02 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.57599999999999996,"15min":0.60199999999999998},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.69099999999999995,"5min":0.80100000000000005,"15min":0.873},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.60599999999999998,"15min":0.624},"min":{"1min":0.40899999999999997,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.74199999999999999,"5min":0.83499999999999996,"15min":0.93100000000000005},"last":0.745}]},{"osd":3,"last update":"Thu Oct 19 14:21:02 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.629,"15min":0.64900000000000002},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.66700000000000004,"5min":0.875,"15min":0.875},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.63800000000000001,"15min":0.65500000000000003},"min":{"1min":0.45400000000000001,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.92200000000000004,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.72799999999999998}]},{"osd":4,"last update":"Thu Oct 19 14:21:02 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.48899999999999999,"15min":0.48399999999999999},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.51000000000000001,"15min":0.50800000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.30399999999999999},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.85699999999999998},"last":0.45900000000000002}]},{"osd":5,"last update":"Thu Oct 19 14:21:02 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.50700000000000001,"15min":0.502},"min":{"1min":0.34799999999999998,"5min":0.32700000000000001,"15min":0.32500000000000001},"max":{"1min":0.79900000000000004,"5min":1.115,"15min":1.115},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.56599999999999995,"15min":0.54500000000000004},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.67900000000000005}]},{"osd":7,"last update":"Thu Oct 19 14:21:02 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.56000000000000005,"15min":0.54700000000000004},"min":{"1min":0.36799999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.90100000000000002},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.56699999999999995,"15min":0.55400000000000005},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.98399999999999999,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.76500000000000001}]}]},{"osd":5,"up_from":13,"seq":55834576011,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66108,"kb_used_data":2992,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305732,"statfs":{"total":96636764160,"available":96569069568,"internally_reserved":0,"allocated":3063808,"data_stored":1576080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:20:46 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.58499999999999996,"15min":0.61699999999999999},"min":{"1min":0.33200000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.78700000000000003,"5min":0.79700000000000004,"15min":0.86399999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.60999999999999999,"15min":0.63800000000000001},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.77400000000000002,"5min":0.85899999999999999,"15min":0.85999999999999999},"last":0.70499999999999996}]},{"osd":1,"last update":"Thu Oct 19 14:20:46 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.59499999999999997,"15min":0.628},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.749,"5min":0.80300000000000005,"15min":0.877},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.626,"15min":0.64000000000000001},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.81599999999999995,"5min":0.86699999999999999,"15min":0.88500000000000001},"last":0.877}]},{"osd":2,"last update":"Thu Oct 19 14:20:46 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.59999999999999998,"15min":0.622},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.75900000000000001,"5min":0.83799999999999997,"15min":0.85099999999999998},"last":0.85799999999999998},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.63300000000000001,"15min":0.65500000000000003},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.81299999999999994,"5min":0.83299999999999996,"15min":0.90800000000000003},"last":0.77300000000000002}]},{"osd":3,"last update":"Thu Oct 19 14:20:46 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.61399999999999999,"15min":0.64100000000000001},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.77100000000000002,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.64000000000000001,"15min":0.66700000000000004},"min":{"1min":0.436,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.73899999999999999,"5min":0.876,"15min":0.90900000000000003},"last":0.89900000000000002}]},{"osd":4,"last update":"Thu Oct 19 14:20:46 2023","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.48499999999999999,"15min":0.48299999999999998},"min":{"1min":0.35499999999999998,"5min":0.34999999999999998,"15min":0.32500000000000001},"max":{"1min":0.56200000000000006,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.437,"5min":0.499,"15min":0.501},"min":{"1min":0.308,"5min":0.308,"15min":0.28599999999999998},"max":{"1min":0.57299999999999995,"5min":0.77500000000000002,"15min":0.82099999999999995},"last":0.79500000000000004}]},{"osd":6,"last update":"Thu Oct 19 14:20:46 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.51000000000000001,"15min":0.497},"min":{"1min":0.32200000000000001,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.72999999999999998,"5min":0.80800000000000005,"15min":0.88800000000000001},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.53300000000000003,"15min":0.53600000000000003},"min":{"1min":0.39900000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.70399999999999996,"5min":0.78000000000000003,"15min":0.88100000000000001},"last":0.65200000000000002}]},{"osd":7,"last update":"Thu Oct 19 14:20:46 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.50800000000000001,"15min":0.51500000000000001},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.625,"5min":0.749,"15min":0.82899999999999996},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.57399999999999995,"15min":0.56999999999999995},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.67300000000000004,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.74299999999999999}]}]},{"osd":4,"up_from":13,"seq":55834576011,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48068,"kb_used_data":2996,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323772,"statfs":{"total":96636764160,"available":96587542528,"internally_reserved":0,"allocated":3067904,"data_stored":1576150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.55500000000000005,"15min":0.57599999999999996},"min":{"1min":0.35499999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.80800000000000005,"5min":0.81799999999999995,"15min":0.879},"last":0.439},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.58499999999999996,"15min":0.59799999999999998},"min":{"1min":0.499,"5min":0.31,"15min":0.31},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.873},"last":0.54300000000000004}]},{"osd":1,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.61499999999999999,"15min":0.64100000000000001},"min":{"1min":0.504,"5min":0.372,"15min":0.32600000000000001},"max":{"1min":0.80400000000000005,"5min":0.80700000000000005,"15min":0.92700000000000005},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.623,"15min":0.64400000000000002},"min":{"1min":0.443,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":1.0940000000000001},"last":0.497}]},{"osd":2,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.63400000000000001,"15min":0.65000000000000002},"min":{"1min":0.46400000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.64800000000000002,"15min":0.66900000000000004},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":1.008},"last":0.69499999999999995}]},{"osd":3,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.61299999999999999,"15min":0.63400000000000001},"min":{"1min":0.47399999999999998,"5min":0.37,"15min":0.37},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.91900000000000004},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.66600000000000004,"15min":0.66700000000000004},"min":{"1min":0.54000000000000004,"5min":0.45400000000000001,"15min":0.437},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.71899999999999997}]},{"osd":5,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.48299999999999998,"15min":0.47099999999999997},"min":{"1min":0.34000000000000002,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.504,"15min":0.49199999999999999},"min":{"1min":0.34300000000000003,"5min":0.33700000000000002,"15min":0.29199999999999998},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.65400000000000003}]},{"osd":6,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.496,"15min":0.48599999999999999},"min":{"1min":0.34799999999999998,"5min":0.33900000000000002,"15min":0.28199999999999997},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.53700000000000003},"min":{"1min":0.39100000000000001,"5min":0.375,"15min":0.33300000000000002},"max":{"1min":0.81299999999999994,"5min":0.84799999999999998,"15min":1.2969999999999999},"last":0.749}]},{"osd":7,"last update":"Thu Oct 19 14:20:12 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.55100000000000005,"15min":0.54100000000000004},"min":{"1min":0.36499999999999999,"5min":0.35699999999999998,"15min":0.32600000000000001},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.625},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60499999999999998,"15min":0.59199999999999997},"min":{"1min":0.47099999999999997,"5min":0.32100000000000001,"15min":0.318},"max":{"1min":0.80400000000000005,"5min":0.88600000000000001,"15min":1.032},"last":0.76600000000000001}]}]},{"osd":3,"up_from":13,"seq":55834576010,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56412,"kb_used_data":3016,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315428,"statfs":{"total":96636764160,"available":96578998272,"internally_reserved":0,"allocated":3088384,"data_stored":1578330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54655432},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:20:49 2023","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.41099999999999998,"15min":0.41799999999999998},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.51800000000000002,"5min":0.61499999999999999,"15min":0.70699999999999996},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.40799999999999997,"15min":0.42799999999999999},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.752},"last":0.28899999999999998}]},{"osd":1,"last update":"Thu Oct 19 14:20:49 2023","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.42199999999999999,"15min":0.43099999999999999},"min":{"1min":0.25,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.54200000000000004,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.432,"5min":0.46400000000000002,"15min":0.47199999999999998},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.26400000000000001},"max":{"1min":0.67900000000000005,"5min":0.73099999999999998,"15min":0.81799999999999995},"last":0.40000000000000002}]},{"osd":2,"last update":"Thu Oct 19 14:20:49 2023","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.46899999999999997,"15min":0.48299999999999998},"min":{"1min":0.33100000000000002,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.625,"5min":0.64600000000000002,"15min":0.73099999999999998},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.49199999999999999,"15min":0.504},"min":{"1min":0.31900000000000001,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.83699999999999997},"last":0.30599999999999999}]},{"osd":4,"last update":"Thu Oct 19 14:20:49 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.69199999999999995,"15min":0.70299999999999996},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.871,"5min":1.018,"15min":1.157},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.71099999999999997,"15min":0.72899999999999998},"min":{"1min":0.47399999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.97699999999999998,"5min":0.996,"15min":1.173},"last":0.58299999999999996}]},{"osd":5,"last update":"Thu Oct 19 14:20:49 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.68300000000000005,"15min":0.68500000000000005},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.41999999999999998},"max":{"1min":0.83399999999999996,"5min":0.97299999999999998,"15min":1.0149999999999999},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.67500000000000004,"15min":0.68600000000000005},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.84999999999999998,"5min":0.99399999999999999,"15min":1.105},"last":0.54400000000000004}]},{"osd":6,"last update":"Thu Oct 19 14:20:49 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.65900000000000003,"15min":0.68700000000000006},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.70499999999999996,"5min":1.0009999999999999,"15min":1.0669999999999999},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.621,"5min":0.69599999999999995,"15min":0.70999999999999996},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.85599999999999998,"5min":0.91500000000000004,"15min":1.079},"last":0.59999999999999998}]},{"osd":7,"last update":"Thu Oct 19 14:20:49 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.68799999999999994,"15min":0.69899999999999995},"min":{"1min":0.436,"5min":0.434,"15min":0.434},"max":{"1min":0.78800000000000003,"5min":0.96899999999999997,"15min":1.085},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.70899999999999996,"15min":0.72899999999999998},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.79400000000000004,"5min":0.97199999999999998,"15min":1.1739999999999999},"last":0.66700000000000004}]}]},{"osd":2,"up_from":13,"seq":55834576012,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48072,"kb_used_data":3000,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323768,"statfs":{"total":96636764160,"available":96587538432,"internally_reserved":0,"allocated":3072000,"data_stored":1576732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:20:36 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.38800000000000001,"15min":0.39700000000000002},"min":{"1min":0.254,"5min":0.218,"15min":0.218},"max":{"1min":0.55300000000000005,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.435,"15min":0.436},"min":{"1min":0.28199999999999997,"5min":0.249,"15min":0.223},"max":{"1min":0.54100000000000004,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.29999999999999999}]},{"osd":1,"last update":"Thu Oct 19 14:20:36 2023","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.39600000000000002,"15min":0.41199999999999998},"min":{"1min":0.27500000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.47799999999999998,"5min":0.61699999999999999,"15min":0.65400000000000003},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.46700000000000003,"15min":0.46500000000000002},"min":{"1min":0.28499999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.60499999999999998,"5min":1.742,"15min":1.742},"last":0.39200000000000002}]},{"osd":3,"last update":"Thu Oct 19 14:20:36 2023","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.47099999999999997,"15min":0.47999999999999998},"min":{"1min":0.32100000000000001,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.63700000000000001,"5min":0.69299999999999995,"15min":0.74399999999999999},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.48799999999999999,"15min":0.50600000000000001},"min":{"1min":0.34499999999999997,"5min":0.32000000000000001,"15min":0.315},"max":{"1min":0.56999999999999995,"5min":0.67600000000000005,"15min":0.83599999999999997},"last":0.38100000000000001}]},{"osd":4,"last update":"Thu Oct 19 14:20:36 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.65700000000000003,"15min":0.66900000000000004},"min":{"1min":0.47099999999999997,"5min":0.39600000000000002,"15min":0.39500000000000002},"max":{"1min":0.85599999999999998,"5min":0.93300000000000005,"15min":0.998},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.67000000000000004,"15min":0.67900000000000005},"min":{"1min":0.46500000000000002,"5min":0.38900000000000001,"15min":0.35199999999999998},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.61499999999999999}]},{"osd":5,"last update":"Thu Oct 19 14:20:36 2023","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.69699999999999995,"15min":0.69599999999999995},"min":{"1min":0.52700000000000002,"5min":0.45100000000000001,"15min":0.40600000000000003},"max":{"1min":0.80000000000000004,"5min":1.4330000000000001,"15min":1.4330000000000001},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.68100000000000005,"15min":0.68700000000000006},"min":{"1min":0.52700000000000002,"5min":0.45100000000000001,"15min":0.39300000000000002},"max":{"1min":0.93300000000000005,"5min":1.595,"15min":1.595},"last":0.623}]},{"osd":6,"last update":"Thu Oct 19 14:20:36 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.69299999999999995,"15min":0.70799999999999996},"min":{"1min":0.48799999999999999,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.996,"5min":0.996,"15min":1.0189999999999999},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.71899999999999997,"15min":0.72099999999999997},"min":{"1min":0.48899999999999999,"5min":0.433,"15min":0.40600000000000003},"max":{"1min":0.93899999999999995,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":0.63300000000000001}]},{"osd":7,"last update":"Thu Oct 19 14:20:36 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.70199999999999996,"15min":0.71999999999999997},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.41799999999999998},"max":{"1min":0.88700000000000001,"5min":1.0129999999999999,"15min":1.0289999999999999},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.73599999999999999,"15min":0.73099999999999998},"min":{"1min":0.51900000000000002,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.97299999999999998,"5min":1.53,"15min":1.53},"last":0.65500000000000003}]}]},{"osd":1,"up_from":13,"seq":55834576011,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48060,"kb_used_data":2992,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323780,"statfs":{"total":96636764160,"available":96587550720,"internally_reserved":0,"allocated":3063808,"data_stored":1576080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.39600000000000002,"15min":0.40400000000000003},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23100000000000001},"max":{"1min":0.57299999999999995,"5min":0.61599999999999999,"15min":0.67400000000000004},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.435,"15min":0.45400000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.54200000000000004,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.442}]},{"osd":2,"last update":"Thu Oct 19 14:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.43099999999999999,"15min":0.442},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.56899999999999995,"5min":0.63100000000000001,"15min":0.77500000000000002},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.48899999999999999,"15min":0.48999999999999999},"min":{"1min":0.32400000000000001,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.67300000000000004,"5min":0.68700000000000006,"15min":0.82799999999999996},"last":0.378}]},{"osd":3,"last update":"Thu Oct 19 14:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.499,"15min":0.505},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.29499999999999998},"max":{"1min":0.64300000000000002,"5min":0.76000000000000001,"15min":0.78900000000000003},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.436,"15min":0.44600000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.26100000000000001},"max":{"1min":0.60799999999999998,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.42399999999999999}]},{"osd":4,"last update":"Thu Oct 19 14:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.66400000000000003,"15min":0.67100000000000004},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.92800000000000005,"5min":0.98299999999999998,"15min":1.042},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.67100000000000004,"15min":0.66900000000000004},"min":{"1min":0.44500000000000001,"5min":0.41299999999999998,"15min":0.32300000000000001},"max":{"1min":0.89400000000000002,"5min":0.93500000000000005,"15min":1.1659999999999999},"last":0.63600000000000001}]},{"osd":5,"last update":"Thu Oct 19 14:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.68799999999999994,"15min":0.68600000000000005},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.82099999999999995,"5min":1.002,"15min":1.3500000000000001},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.70599999999999996,"15min":0.70799999999999996},"min":{"1min":0.46700000000000003,"5min":0.44,"15min":0.40500000000000003},"max":{"1min":1.0009999999999999,"5min":1.0149999999999999,"15min":1.121},"last":0.72199999999999998}]},{"osd":6,"last update":"Thu Oct 19 14:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.67800000000000005,"15min":0.68200000000000005},"min":{"1min":0.51000000000000001,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.95499999999999996,"5min":1.0129999999999999,"15min":1.2310000000000001},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.71899999999999997,"15min":0.71599999999999997},"min":{"1min":0.48199999999999998,"5min":0.434,"15min":0.434},"max":{"1min":0.93100000000000005,"5min":1.0109999999999999,"15min":1.274},"last":0.66900000000000004}]},{"osd":7,"last update":"Thu Oct 19 14:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.71899999999999997,"15min":0.72399999999999998},"min":{"1min":0.498,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.96799999999999997,"5min":0.99099999999999999,"15min":1.075},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.70099999999999996,"15min":0.70199999999999996},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.44400000000000001},"max":{"1min":0.91200000000000003,"5min":1.0169999999999999,"15min":1.117},"last":0.61199999999999999}]}]},{"osd":0,"up_from":13,"seq":55834576012,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61132,"kb_used_data":3580,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310708,"statfs":{"total":96636764160,"available":96574164992,"internally_reserved":0,"allocated":3665920,"data_stored":2167441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Oct 19 14:21:13 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.40100000000000002,"15min":0.41099999999999998},"min":{"1min":0.26300000000000001,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.47799999999999998,"5min":0.61499999999999999,"15min":0.624},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.41699999999999998,"15min":0.45500000000000002},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.49199999999999999,"5min":0.70899999999999996,"15min":0.81100000000000005},"last":0.38300000000000001}]},{"osd":2,"last update":"Thu Oct 19 14:21:13 2023","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.34100000000000003,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.54400000000000004,"5min":0.754,"15min":0.754},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.377,"5min":0.42399999999999999,"15min":0.44400000000000001},"min":{"1min":0.28100000000000003,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.46899999999999997,"5min":0.60199999999999998,"15min":0.69099999999999995},"last":0.40300000000000002}]},{"osd":3,"last update":"Thu Oct 19 14:21:13 2023","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.436},"min":{"1min":0.27200000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.50800000000000001,"5min":0.58799999999999997,"15min":0.66800000000000004},"last":0.314},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.48199999999999998,"15min":0.50700000000000001},"min":{"1min":0.29299999999999998,"5min":0.254,"15min":0.254},"max":{"1min":0.72399999999999998,"5min":0.78100000000000003,"15min":0.84599999999999997},"last":0.45000000000000001}]},{"osd":4,"last update":"Thu Oct 19 14:21:13 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.63,"15min":0.67000000000000004},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.80300000000000005,"5min":0.93100000000000005,"15min":1.1890000000000001},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.628,"15min":0.67400000000000004},"min":{"1min":0.45900000000000002,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.75600000000000001,"5min":0.93400000000000005,"15min":1.2150000000000001},"last":0.47299999999999998}]},{"osd":5,"last update":"Thu Oct 19 14:21:13 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.67200000000000004,"15min":0.71399999999999997},"min":{"1min":0.48599999999999999,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.83099999999999996,"5min":0.98499999999999999,"15min":1.1579999999999999},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.67500000000000004,"15min":0.71399999999999997},"min":{"1min":0.499,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.80000000000000004,"5min":0.98299999999999998,"15min":1.2310000000000001},"last":0.51900000000000002}]},{"osd":6,"last update":"Thu Oct 19 14:21:13 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.66300000000000003,"15min":0.69899999999999995},"min":{"1min":0.46100000000000002,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.85499999999999998,"5min":0.97799999999999998,"15min":1.1739999999999999},"last":0.495},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.65300000000000002,"15min":0.69599999999999995},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.70299999999999996,"5min":0.97799999999999998,"15min":1.0189999999999999},"last":0.55000000000000004}]},{"osd":7,"last update":"Thu Oct 19 14:21:13 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.66600000000000004,"15min":0.70999999999999996},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.79900000000000004,"5min":0.93400000000000005,"15min":1.1180000000000001},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.66100000000000003,"15min":0.69599999999999995},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.84499999999999997,"5min":0.93799999999999994,"15min":1.0349999999999999},"last":0.53200000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-19T14:21:18.250 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-19T14:21:18.251 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-19T14:21:18.401 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.397+0000 7fe1a5e75640 1 -- 172.21.15.57:0/1294850684 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe1a006f0a0 msgr2=0x7fe1a006ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:18.401 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.397+0000 7fe1a5e75640 1 --2- 172.21.15.57:0/1294850684 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe1a006f0a0 0x7fe1a006ddc0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fe194005fd0 tx=0x7fe19402be10 comp rx=0 tx=0).stop 2023-10-19T14:21:18.401 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.397+0000 7fe1a5e75640 1 -- 172.21.15.57:0/1294850684 shutdown_connections 2023-10-19T14:21:18.401 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.397+0000 7fe1a5e75640 1 --2- 172.21.15.57:0/1294850684 >> v2:172.21.15.179:3300/0 conn(0x7fe1a00ad370 0x7fe1a00ad800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:18.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.397+0000 7fe1a5e75640 1 --2- 172.21.15.57:0/1294850684 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe1a006f0a0 0x7fe1a006ddc0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:18.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.397+0000 7fe1a5e75640 1 -- 172.21.15.57:0/1294850684 >> 172.21.15.57:0/1294850684 conn(0x7fe1a00b0670 msgr2=0x7fe1a00b0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:18.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.397+0000 7fe1a5e75640 1 -- 172.21.15.57:0/1294850684 shutdown_connections 2023-10-19T14:21:18.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.397+0000 7fe1a5e75640 1 -- 172.21.15.57:0/1294850684 wait complete. 2023-10-19T14:21:18.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.399+0000 7fe1a5e75640 1 Processor -- start 2023-10-19T14:21:18.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.399+0000 7fe1a5e75640 1 -- start start 2023-10-19T14:21:18.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.399+0000 7fe1a5e75640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe1a006d2d0 0x7fe1a014fe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:18.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.399+0000 7fe1a5e75640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe1a006f0a0 0x7fe1a00cb2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:18.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.399+0000 7fe1a5e75640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1a00ad370 0x7fe1a00cb7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:18.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.399+0000 7fe1a5e75640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a00cae30 con 0x7fe1a00ad370 2023-10-19T14:21:18.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.399+0000 7fe1a5e75640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a01f36a0 con 0x7fe1a006d2d0 2023-10-19T14:21:18.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.399+0000 7fe1a5e75640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a0150380 con 0x7fe1a006f0a0 2023-10-19T14:21:18.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.400+0000 7fe19ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1a00ad370 0x7fe1a00cb7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:18.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.400+0000 7fe19ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1a00ad370 0x7fe1a00cb7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57440/0 (socket says 172.21.15.57:57440) 2023-10-19T14:21:18.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.400+0000 7fe19ffff640 1 -- 172.21.15.57:0/836993827 learned_addr learned my addr 172.21.15.57:0/836993827 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:18.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.400+0000 7fe19effd640 1 --2- 172.21.15.57:0/836993827 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe1a006f0a0 0x7fe1a00cb2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:18.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.400+0000 7fe19f7fe640 1 --2- 172.21.15.57:0/836993827 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe1a006d2d0 0x7fe1a014fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:18.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.400+0000 7fe19ffff640 1 -- 172.21.15.57:0/836993827 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe1a006f0a0 msgr2=0x7fe1a00cb2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:18.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.400+0000 7fe19ffff640 1 --2- 172.21.15.57:0/836993827 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe1a006f0a0 0x7fe1a00cb2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:18.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.400+0000 7fe19ffff640 1 -- 172.21.15.57:0/836993827 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe1a006d2d0 msgr2=0x7fe1a014fe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:18.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.400+0000 7fe19ffff640 1 --2- 172.21.15.57:0/836993827 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe1a006d2d0 0x7fe1a014fe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:18.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.400+0000 7fe19ffff640 1 -- 172.21.15.57:0/836993827 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe194009560 con 0x7fe1a00ad370 2023-10-19T14:21:18.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.400+0000 7fe19ffff640 1 --2- 172.21.15.57:0/836993827 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1a00ad370 0x7fe1a00cb7f0 secure :-1 s=READY pgs=2644 cs=0 l=1 rev1=1 crypto rx=0x7fe19800ea80 tx=0x7fe19800ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:18.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.400+0000 7fe187fff640 1 -- 172.21.15.57:0/836993827 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe19800cea0 con 0x7fe1a00ad370 2023-10-19T14:21:18.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.400+0000 7fe187fff640 1 -- 172.21.15.57:0/836993827 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fe198004b20 con 0x7fe1a00ad370 2023-10-19T14:21:18.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.400+0000 7fe1a5e75640 1 -- 172.21.15.57:0/836993827 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1a01f33c0 con 0x7fe1a00ad370 2023-10-19T14:21:18.409 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.400+0000 7fe187fff640 1 -- 172.21.15.57:0/836993827 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe198005020 con 0x7fe1a00ad370 2023-10-19T14:21:18.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.400+0000 7fe1a5e75640 1 -- 172.21.15.57:0/836993827 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1a0150d50 con 0x7fe1a00ad370 2023-10-19T14:21:18.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.401+0000 7fe1a5e75640 1 -- 172.21.15.57:0/836993827 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1a00ce1d0 con 0x7fe1a00ad370 2023-10-19T14:21:18.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.402+0000 7fe187fff640 1 -- 172.21.15.57:0/836993827 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fe198005180 con 0x7fe1a00ad370 2023-10-19T14:21:18.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.402+0000 7fe187fff640 1 --2- 172.21.15.57:0/836993827 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fe18c06b080 0x7fe18c06d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:18.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.402+0000 7fe187fff640 1 -- 172.21.15.57:0/836993827 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(243..243 src has 1..243) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fe198014070 con 0x7fe1a00ad370 2023-10-19T14:21:18.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.403+0000 7fe19f7fe640 1 --2- 172.21.15.57:0/836993827 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fe18c06b080 0x7fe18c06d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:18.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.403+0000 7fe19f7fe640 1 --2- 172.21.15.57:0/836993827 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fe18c06b080 0x7fe18c06d540 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe190001780 tx=0x7fe190009040 comp rx=0 tx=0).ready entity=mgr.9530 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:18.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.407+0000 7fe187fff640 1 -- 172.21.15.57:0/836993827 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe1980553e0 con 0x7fe1a00ad370 2023-10-19T14:21:18.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.653+0000 7fe1a5e75640 1 -- 172.21.15.57:0/836993827 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fe1a00690e0 con 0x7fe1a00ad370 2023-10-19T14:21:18.658 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.656+0000 7fe187fff640 1 -- 172.21.15.57:0/836993827 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v61) v1 ==== 190+0+147 (secure 0 0 0) 0x7fe198058a00 con 0x7fe1a00ad370 2023-10-19T14:21:18.659 INFO:teuthology.orchestra.run.smithi057.stdout:[client.0] 2023-10-19T14:21:18.659 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQBkJDFlg9yuExAATxiWDfS+p07G1F0pORbJEg== 2023-10-19T14:21:18.659 INFO:teuthology.orchestra.run.smithi057.stdout: caps mds = "allow" 2023-10-19T14:21:18.660 INFO:teuthology.orchestra.run.smithi057.stdout: caps mon = "allow r" 2023-10-19T14:21:18.660 INFO:teuthology.orchestra.run.smithi057.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-19T14:21:18.661 INFO:teuthology.orchestra.run.smithi057.stderr:updated caps for client.0 2023-10-19T14:21:18.661 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.658+0000 7fe1a5e75640 1 -- 172.21.15.57:0/836993827 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fe18c06b080 msgr2=0x7fe18c06d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:18.661 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.658+0000 7fe1a5e75640 1 --2- 172.21.15.57:0/836993827 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fe18c06b080 0x7fe18c06d540 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe190001780 tx=0x7fe190009040 comp rx=0 tx=0).stop 2023-10-19T14:21:18.661 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.658+0000 7fe1a5e75640 1 -- 172.21.15.57:0/836993827 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1a00ad370 msgr2=0x7fe1a00cb7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:18.662 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.658+0000 7fe1a5e75640 1 --2- 172.21.15.57:0/836993827 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1a00ad370 0x7fe1a00cb7f0 secure :-1 s=READY pgs=2644 cs=0 l=1 rev1=1 crypto rx=0x7fe19800ea80 tx=0x7fe19800ef40 comp rx=0 tx=0).stop 2023-10-19T14:21:18.662 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.658+0000 7fe1a5e75640 1 -- 172.21.15.57:0/836993827 shutdown_connections 2023-10-19T14:21:18.662 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.658+0000 7fe1a5e75640 1 --2- 172.21.15.57:0/836993827 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fe18c06b080 0x7fe18c06d540 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:18.662 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.658+0000 7fe1a5e75640 1 --2- 172.21.15.57:0/836993827 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1a00ad370 0x7fe1a00cb7f0 unknown :-1 s=CLOSED pgs=2644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:18.663 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.658+0000 7fe1a5e75640 1 --2- 172.21.15.57:0/836993827 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe1a006f0a0 0x7fe1a00cb2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:18.663 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.658+0000 7fe1a5e75640 1 --2- 172.21.15.57:0/836993827 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe1a006d2d0 0x7fe1a014fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:18.663 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.658+0000 7fe1a5e75640 1 -- 172.21.15.57:0/836993827 >> 172.21.15.57:0/836993827 conn(0x7fe1a00b0670 msgr2=0x7fe1a015ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:18.663 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.658+0000 7fe1a5e75640 1 -- 172.21.15.57:0/836993827 shutdown_connections 2023-10-19T14:21:18.663 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.658+0000 7fe1a5e75640 1 -- 172.21.15.57:0/836993827 wait complete. 2023-10-19T14:21:18.672 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T14:21:18.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.779+0000 7fce728bb640 1 -- 172.21.15.57:0/2068117420 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fce54005180 msgr2=0x7fce5400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:18.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.779+0000 7fce728bb640 1 --2- 172.21.15.57:0/2068117420 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fce54005180 0x7fce5400b980 secure :-1 s=READY pgs=2646 cs=0 l=1 rev1=1 crypto rx=0x7fce64001d80 tx=0x7fce540057f0 comp rx=0 tx=0).stop 2023-10-19T14:21:18.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.779+0000 7fce728bb640 1 -- 172.21.15.57:0/2068117420 shutdown_connections 2023-10-19T14:21:18.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.779+0000 7fce728bb640 1 --2- 172.21.15.57:0/2068117420 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fce54005180 0x7fce5400b980 unknown :-1 s=CLOSED pgs=2646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:18.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.779+0000 7fce728bb640 1 -- 172.21.15.57:0/2068117420 >> 172.21.15.57:0/2068117420 conn(0x7fce6c069930 msgr2=0x7fce6c069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:18.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.779+0000 7fce728bb640 1 -- 172.21.15.57:0/2068117420 shutdown_connections 2023-10-19T14:21:18.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.779+0000 7fce728bb640 1 -- 172.21.15.57:0/2068117420 wait complete. 2023-10-19T14:21:18.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.780+0000 7fce728bb640 1 Processor -- start 2023-10-19T14:21:18.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.780+0000 7fce728bb640 1 -- start start 2023-10-19T14:21:18.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.781+0000 7fce728bb640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fce54005180 0x7fce6c151f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:18.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.781+0000 7fce728bb640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fce6c1e3e40 0x7fce6c1e42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:18.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.781+0000 7fce728bb640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fce6c0ae8c0 0x7fce6c0aed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:18.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.781+0000 7fce728bb640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce6c0b02b0 con 0x7fce6c1e3e40 2023-10-19T14:21:18.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.781+0000 7fce728bb640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce6c1e4800 con 0x7fce6c0ae8c0 2023-10-19T14:21:18.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.781+0000 7fce728bb640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce6c0af630 con 0x7fce54005180 2023-10-19T14:21:18.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.781+0000 7fce6b7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fce6c1e3e40 0x7fce6c1e42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:18.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.781+0000 7fce6b7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fce6c1e3e40 0x7fce6c1e42c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57450/0 (socket says 172.21.15.57:57450) 2023-10-19T14:21:18.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.781+0000 7fce6b7fe640 1 -- 172.21.15.57:0/1367825279 learned_addr learned my addr 172.21.15.57:0/1367825279 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:18.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.781+0000 7fce6b7fe640 1 -- 172.21.15.57:0/1367825279 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fce54005180 msgr2=0x7fce6c151f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:18.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.781+0000 7fce6b7fe640 1 --2- 172.21.15.57:0/1367825279 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fce54005180 0x7fce6c151f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:18.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.781+0000 7fce6b7fe640 1 -- 172.21.15.57:0/1367825279 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fce6c0ae8c0 msgr2=0x7fce6c0aed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:18.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.781+0000 7fce6b7fe640 1 --2- 172.21.15.57:0/1367825279 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fce6c0ae8c0 0x7fce6c0aed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:18.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.781+0000 7fce6b7fe640 1 -- 172.21.15.57:0/1367825279 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce6c0af910 con 0x7fce6c1e3e40 2023-10-19T14:21:18.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.781+0000 7fce6b7fe640 1 --2- 172.21.15.57:0/1367825279 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fce6c1e3e40 0x7fce6c1e42c0 secure :-1 s=READY pgs=2647 cs=0 l=1 rev1=1 crypto rx=0x7fce5c011820 tx=0x7fce5c011ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:18.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.782+0000 7fce68ff9640 1 -- 172.21.15.57:0/1367825279 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce5c00fd50 con 0x7fce6c1e3e40 2023-10-19T14:21:18.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.782+0000 7fce68ff9640 1 -- 172.21.15.57:0/1367825279 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fce5c012450 con 0x7fce6c1e3e40 2023-10-19T14:21:18.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.782+0000 7fce728bb640 1 -- 172.21.15.57:0/1367825279 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce6c0afbf0 con 0x7fce6c1e3e40 2023-10-19T14:21:18.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.782+0000 7fce68ff9640 1 -- 172.21.15.57:0/1367825279 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce5c012860 con 0x7fce6c1e3e40 2023-10-19T14:21:18.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.782+0000 7fce728bb640 1 -- 172.21.15.57:0/1367825279 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce6c152470 con 0x7fce6c1e3e40 2023-10-19T14:21:18.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.783+0000 7fce728bb640 1 -- 172.21.15.57:0/1367825279 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce6c06bf40 con 0x7fce6c1e3e40 2023-10-19T14:21:18.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.783+0000 7fce68ff9640 1 -- 172.21.15.57:0/1367825279 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fce5c0129c0 con 0x7fce6c1e3e40 2023-10-19T14:21:18.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.783+0000 7fce68ff9640 1 --2- 172.21.15.57:0/1367825279 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fce5806b080 0x7fce5806d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:18.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.783+0000 7fce68ff9640 1 -- 172.21.15.57:0/1367825279 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(243..243 src has 1..243) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fce5c08aa70 con 0x7fce6c1e3e40 2023-10-19T14:21:18.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.784+0000 7fce6bfff640 1 --2- 172.21.15.57:0/1367825279 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fce5806b080 0x7fce5806d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:18.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.784+0000 7fce6bfff640 1 --2- 172.21.15.57:0/1367825279 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fce5806b080 0x7fce5806d540 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fce60001200 tx=0x7fce60001170 comp rx=0 tx=0).ready entity=mgr.9530 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:18.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.787+0000 7fce68ff9640 1 -- 172.21.15.57:0/1367825279 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fce5c0555d0 con 0x7fce6c1e3e40 2023-10-19T14:21:19.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:18.999+0000 7fce728bb640 1 -- 172.21.15.57:0/1367825279 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fce6c0690e0 con 0x7fce6c1e3e40 2023-10-19T14:21:19.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.000+0000 7fce68ff9640 1 -- 172.21.15.57:0/1367825279 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+4914 (secure 0 0 0) 0x7fce5c058bf0 con 0x7fce6c1e3e40 2023-10-19T14:21:19.003 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:21:19.003 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":103,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9761,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/325888554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":325888554},{"type":"v1","addr":"172.21.15.179:6836","nonce":325888554}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":100},{"gid":9764,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/8038232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":8038232},{"type":"v1","addr":"172.21.15.179:6837","nonce":8038232}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":100},{"gid":12555,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6833/4256498777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":4256498777},{"type":"v1","addr":"172.21.15.57:6833","nonce":4256498777}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":99}],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T14:21:09.759935+0000","modified":"2023-10-19T14:21:11.768676+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":12564},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12564":{"gid":12564,"name":"c","rank":0,"incarnation":101,"state":"up:active","state_seq":3,"addr":"172.21.15.57:6835/18187246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":18187246},{"type":"v1","addr":"172.21.15.57:6835","nonce":18187246}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15}]} 2023-10-19T14:21:19.004 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 103 2023-10-19T14:21:19.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.002+0000 7fce728bb640 1 -- 172.21.15.57:0/1367825279 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fce5806b080 msgr2=0x7fce5806d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:19.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.002+0000 7fce728bb640 1 --2- 172.21.15.57:0/1367825279 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fce5806b080 0x7fce5806d540 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fce60001200 tx=0x7fce60001170 comp rx=0 tx=0).stop 2023-10-19T14:21:19.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.002+0000 7fce728bb640 1 -- 172.21.15.57:0/1367825279 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fce6c1e3e40 msgr2=0x7fce6c1e42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:19.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.002+0000 7fce728bb640 1 --2- 172.21.15.57:0/1367825279 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fce6c1e3e40 0x7fce6c1e42c0 secure :-1 s=READY pgs=2647 cs=0 l=1 rev1=1 crypto rx=0x7fce5c011820 tx=0x7fce5c011ce0 comp rx=0 tx=0).stop 2023-10-19T14:21:19.006 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.002+0000 7fce728bb640 1 -- 172.21.15.57:0/1367825279 shutdown_connections 2023-10-19T14:21:19.006 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.002+0000 7fce728bb640 1 --2- 172.21.15.57:0/1367825279 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fce5806b080 0x7fce5806d540 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:19.006 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.002+0000 7fce728bb640 1 --2- 172.21.15.57:0/1367825279 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fce6c0ae8c0 0x7fce6c0aed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:19.006 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.002+0000 7fce728bb640 1 --2- 172.21.15.57:0/1367825279 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fce6c1e3e40 0x7fce6c1e42c0 unknown :-1 s=CLOSED pgs=2647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:19.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.002+0000 7fce728bb640 1 --2- 172.21.15.57:0/1367825279 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fce54005180 0x7fce6c151f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:19.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.002+0000 7fce728bb640 1 -- 172.21.15.57:0/1367825279 >> 172.21.15.57:0/1367825279 conn(0x7fce6c069930 msgr2=0x7fce6c0ac690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:19.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.002+0000 7fce728bb640 1 -- 172.21.15.57:0/1367825279 shutdown_connections 2023-10-19T14:21:19.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.002+0000 7fce728bb640 1 -- 172.21.15.57:0/1367825279 wait complete. 2023-10-19T14:21:19.418 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 103, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-19T14:21:09.759935+0000', 'modified': '2023-10-19T14:21:11.768676+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 12564}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_12564': {'gid': 12564, 'name': 'c', 'rank': 0, 'incarnation': 101, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.57:6835/18187246', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 18187246}, {'type': 'v1', 'addr': '172.21.15.57:6835', 'nonce': 18187246}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [32], 'metadata_pool': 31, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-19T14:21:19.418 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-19T14:21:19.419 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-10-19T14:21:19.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.534+0000 7f9e53b3e640 1 -- 172.21.15.57:0/222205504 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9e34005180 msgr2=0x7f9e3400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:19.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.534+0000 7f9e53b3e640 1 --2- 172.21.15.57:0/222205504 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9e34005180 0x7f9e3400b980 secure :-1 s=READY pgs=2649 cs=0 l=1 rev1=1 crypto rx=0x7f9e3c00c610 tx=0x7f9e340057f0 comp rx=0 tx=0).stop 2023-10-19T14:21:19.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.534+0000 7f9e53b3e640 1 -- 172.21.15.57:0/222205504 shutdown_connections 2023-10-19T14:21:19.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.534+0000 7f9e53b3e640 1 --2- 172.21.15.57:0/222205504 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9e34005180 0x7f9e3400b980 unknown :-1 s=CLOSED pgs=2649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:19.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.534+0000 7f9e53b3e640 1 -- 172.21.15.57:0/222205504 >> 172.21.15.57:0/222205504 conn(0x7f9e4c0b3f00 msgr2=0x7f9e4c0b4310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:19.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.534+0000 7f9e53b3e640 1 -- 172.21.15.57:0/222205504 shutdown_connections 2023-10-19T14:21:19.538 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.534+0000 7f9e53b3e640 1 -- 172.21.15.57:0/222205504 wait complete. 2023-10-19T14:21:19.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.535+0000 7f9e53b3e640 1 Processor -- start 2023-10-19T14:21:19.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.535+0000 7f9e53b3e640 1 -- start start 2023-10-19T14:21:19.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.536+0000 7f9e53b3e640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9e34005180 0x7f9e4c13cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:19.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.536+0000 7f9e53b3e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9e4c1526c0 0x7f9e4c13c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:19.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.536+0000 7f9e53b3e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9e4c13fa20 0x7f9e4c15bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:19.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.536+0000 7f9e53b3e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e4c13d500 con 0x7f9e4c1526c0 2023-10-19T14:21:19.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.536+0000 7f9e53b3e640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e4c13ccd0 con 0x7f9e4c13fa20 2023-10-19T14:21:19.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.536+0000 7f9e53b3e640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e4c14e910 con 0x7f9e34005180 2023-10-19T14:21:19.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.536+0000 7f9e510d9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9e4c1526c0 0x7f9e4c13c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:19.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.536+0000 7f9e510d9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9e4c1526c0 0x7f9e4c13c790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57466/0 (socket says 172.21.15.57:57466) 2023-10-19T14:21:19.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.536+0000 7f9e510d9640 1 -- 172.21.15.57:0/1953775925 learned_addr learned my addr 172.21.15.57:0/1953775925 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:19.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.536+0000 7f9e510d9640 1 -- 172.21.15.57:0/1953775925 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9e34005180 msgr2=0x7f9e4c13cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:19.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.536+0000 7f9e520db640 1 --2- 172.21.15.57:0/1953775925 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9e4c13fa20 0x7f9e4c15bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:19.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.536+0000 7f9e518da640 1 --2- 172.21.15.57:0/1953775925 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9e34005180 0x7f9e4c13cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:19.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.536+0000 7f9e510d9640 1 --2- 172.21.15.57:0/1953775925 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9e34005180 0x7f9e4c13cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:19.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.536+0000 7f9e510d9640 1 -- 172.21.15.57:0/1953775925 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9e4c13fa20 msgr2=0x7f9e4c15bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:19.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.536+0000 7f9e510d9640 1 --2- 172.21.15.57:0/1953775925 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9e4c13fa20 0x7f9e4c15bf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:19.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.536+0000 7f9e510d9640 1 -- 172.21.15.57:0/1953775925 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e4c152bb0 con 0x7f9e4c1526c0 2023-10-19T14:21:19.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.536+0000 7f9e520db640 1 --2- 172.21.15.57:0/1953775925 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9e4c13fa20 0x7f9e4c15bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:21:19.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.536+0000 7f9e518da640 1 --2- 172.21.15.57:0/1953775925 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9e34005180 0x7f9e4c13cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:21:19.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.536+0000 7f9e510d9640 1 --2- 172.21.15.57:0/1953775925 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9e4c1526c0 0x7f9e4c13c790 secure :-1 s=READY pgs=2650 cs=0 l=1 rev1=1 crypto rx=0x7f9e40004d20 tx=0x7f9e40001630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:19.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.536+0000 7f9e4a7fc640 1 -- 172.21.15.57:0/1953775925 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e40016030 con 0x7f9e4c1526c0 2023-10-19T14:21:19.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.536+0000 7f9e4a7fc640 1 -- 172.21.15.57:0/1953775925 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f9e400108c0 con 0x7f9e4c1526c0 2023-10-19T14:21:19.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.536+0000 7f9e4a7fc640 1 -- 172.21.15.57:0/1953775925 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e40010cb0 con 0x7f9e4c1526c0 2023-10-19T14:21:19.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.536+0000 7f9e53b3e640 1 -- 172.21.15.57:0/1953775925 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e4c0af740 con 0x7f9e4c1526c0 2023-10-19T14:21:19.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.537+0000 7f9e53b3e640 1 -- 172.21.15.57:0/1953775925 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e4c0afd00 con 0x7f9e4c1526c0 2023-10-19T14:21:19.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.538+0000 7f9e4a7fc640 1 -- 172.21.15.57:0/1953775925 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f9e40019070 con 0x7f9e4c1526c0 2023-10-19T14:21:19.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.538+0000 7f9e4a7fc640 1 --2- 172.21.15.57:0/1953775925 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f9e3806b080 0x7f9e3806d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:19.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.538+0000 7f9e4a7fc640 1 -- 172.21.15.57:0/1953775925 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(243..243 src has 1..243) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f9e40017070 con 0x7f9e4c1526c0 2023-10-19T14:21:19.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.539+0000 7f9e518da640 1 --2- 172.21.15.57:0/1953775925 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f9e3806b080 0x7f9e3806d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:19.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.539+0000 7f9e518da640 1 --2- 172.21.15.57:0/1953775925 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f9e3806b080 0x7f9e3806d540 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9e44009460 tx=0x7f9e44005f90 comp rx=0 tx=0).ready entity=mgr.9530 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:19.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.539+0000 7f9e567a8740 1 Processor -- start 2023-10-19T14:21:19.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.540+0000 7f9e567a8740 1 -- start start 2023-10-19T14:21:19.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.540+0000 7f9e567a8740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5654a3299b40 0x5654a326cae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:19.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.540+0000 7f9e567a8740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5654a3299f20 0x5654a329c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:19.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.540+0000 7f9e567a8740 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x5654a326d020 0x5654a329ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:19.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.540+0000 7f9e567a8740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654a317e730 con 0x5654a3299b40 2023-10-19T14:21:19.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.540+0000 7f9e567a8740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654a31df780 con 0x5654a3299f20 2023-10-19T14:21:19.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.540+0000 7f9e567a8740 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x5654a31dce70 con 0x5654a326d020 2023-10-19T14:21:19.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.540+0000 7f9e518da640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5654a3299b40 0x5654a326cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:19.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.540+0000 7f9e518da640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5654a3299b40 0x5654a326cae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57468/0 (socket says 172.21.15.57:57468) 2023-10-19T14:21:19.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.540+0000 7f9e518da640 1 -- 172.21.15.57:0/535818912 learned_addr learned my addr 172.21.15.57:0/535818912 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:19.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.541+0000 7f9e510d9640 1 --2- 172.21.15.57:0/535818912 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5654a3299f20 0x5654a329c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:19.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.541+0000 7f9e520db640 1 --2- 172.21.15.57:0/535818912 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x5654a326d020 0x5654a329ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:19.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.541+0000 7f9e518da640 1 -- 172.21.15.57:0/535818912 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x5654a326d020 msgr2=0x5654a329ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T14:21:19.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.541+0000 7f9e518da640 1 --2- 172.21.15.57:0/535818912 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x5654a326d020 0x5654a329ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:19.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.541+0000 7f9e518da640 1 -- 172.21.15.57:0/535818912 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5654a3299f20 msgr2=0x5654a329c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T14:21:19.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.541+0000 7f9e518da640 1 --2- 172.21.15.57:0/535818912 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5654a3299f20 0x5654a329c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:19.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.541+0000 7f9e518da640 1 -- 172.21.15.57:0/535818912 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5654a31dd3d0 con 0x5654a3299b40 2023-10-19T14:21:19.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.541+0000 7f9e510d9640 1 --2- 172.21.15.57:0/535818912 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5654a3299f20 0x5654a329c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:21:19.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.541+0000 7f9e518da640 1 --2- 172.21.15.57:0/535818912 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5654a3299b40 0x5654a326cae0 secure :-1 s=READY pgs=2651 cs=0 l=1 rev1=1 crypto rx=0x7f9e44011470 tx=0x7f9e44016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af5c0c6e4afae482 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:19.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.541+0000 7f9e337fe640 1 -- 172.21.15.57:0/535818912 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e4401c680 con 0x5654a3299b40 2023-10-19T14:21:19.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.541+0000 7f9e337fe640 1 -- 172.21.15.57:0/535818912 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f9e4401c7e0 con 0x5654a3299b40 2023-10-19T14:21:19.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.541+0000 7f9e337fe640 1 -- 172.21.15.57:0/535818912 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e4401cb90 con 0x5654a3299b40 2023-10-19T14:21:19.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.542+0000 7f9e567a8740 1 -- 172.21.15.57:0/535818912 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5654a3299b40 msgr2=0x5654a326cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:19.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.542+0000 7f9e567a8740 1 --2- 172.21.15.57:0/535818912 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5654a3299b40 0x5654a326cae0 secure :-1 s=READY pgs=2651 cs=0 l=1 rev1=1 crypto rx=0x7f9e44011470 tx=0x7f9e44016000 comp rx=0 tx=0).stop 2023-10-19T14:21:19.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.542+0000 7f9e567a8740 1 -- 172.21.15.57:0/535818912 shutdown_connections 2023-10-19T14:21:19.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.542+0000 7f9e567a8740 1 --2- 172.21.15.57:0/535818912 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x5654a326d020 0x5654a329ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:19.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.542+0000 7f9e567a8740 1 --2- 172.21.15.57:0/535818912 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5654a3299f20 0x5654a329c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:19.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.542+0000 7f9e567a8740 1 --2- 172.21.15.57:0/535818912 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5654a3299b40 0x5654a326cae0 unknown :-1 s=CLOSED pgs=2651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:19.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.542+0000 7f9e567a8740 1 -- 172.21.15.57:0/535818912 >> 172.21.15.57:0/535818912 conn(0x5654a326ed50 msgr2=0x5654a32b78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:19.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.542+0000 7f9e567a8740 1 -- 172.21.15.57:0/535818912 shutdown_connections 2023-10-19T14:21:19.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.542+0000 7f9e567a8740 1 -- 172.21.15.57:0/535818912 wait complete. 2023-10-19T14:21:19.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.542+0000 7f9e567a8740 1 -- start start 2023-10-19T14:21:19.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.542+0000 7f9e567a8740 1 Processor -- start 2023-10-19T14:21:19.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.543+0000 7f9e567a8740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5654a3299b40 0x5654a329f360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:19.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.543+0000 7f9e567a8740 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x5654a3299f20 0x5654a32c7cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:19.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.543+0000 7f9e567a8740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5654a32b24c0 0x5654a3287450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:19.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.543+0000 7f9e567a8740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654a31d8760 con 0x5654a3299b40 2023-10-19T14:21:19.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.543+0000 7f9e567a8740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654a31938f0 con 0x5654a32b24c0 2023-10-19T14:21:19.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.543+0000 7f9e567a8740 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x5654a3193590 con 0x5654a3299f20 2023-10-19T14:21:19.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.543+0000 7f9e567a8740 10 client.0 _ll_register_callbacks cb 0x5654a31cfee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-19T14:21:19.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.543+0000 7f9e518da640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5654a3299b40 0x5654a329f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:19.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.543+0000 7f9e518da640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5654a3299b40 0x5654a329f360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57480/0 (socket says 172.21.15.57:57480) 2023-10-19T14:21:19.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.543+0000 7f9e518da640 1 -- 172.21.15.57:0/276470299 learned_addr learned my addr 172.21.15.57:0/276470299 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:19.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.543+0000 7f9e510d9640 1 --2- 172.21.15.57:0/276470299 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x5654a3299f20 0x5654a32c7cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:19.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.543+0000 7f9e520db640 1 --2- 172.21.15.57:0/276470299 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5654a32b24c0 0x5654a3287450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:19.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.544+0000 7f9e518da640 1 -- 172.21.15.57:0/276470299 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x5654a3299f20 msgr2=0x5654a32c7cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T14:21:19.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.544+0000 7f9e518da640 1 --2- 172.21.15.57:0/276470299 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x5654a3299f20 0x5654a32c7cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:19.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.544+0000 7f9e518da640 1 -- 172.21.15.57:0/276470299 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5654a32b24c0 msgr2=0x5654a3287450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T14:21:19.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.544+0000 7f9e518da640 1 --2- 172.21.15.57:0/276470299 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5654a32b24c0 0x5654a3287450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:19.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.544+0000 7f9e518da640 1 -- 172.21.15.57:0/276470299 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e44013c60 con 0x5654a3299b40 2023-10-19T14:21:19.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.544+0000 7f9e518da640 1 --2- 172.21.15.57:0/276470299 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5654a3299b40 0x5654a329f360 secure :-1 s=READY pgs=2652 cs=0 l=1 rev1=1 crypto rx=0x7f9e440098b0 tx=0x7f9e4401cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1997c1380dbdde17 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:19.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.544+0000 7f9e30ff9640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2023-10-19T14:21:19.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.544+0000 7f9e30ff9640 1 -- 172.21.15.57:0/276470299 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e44027640 con 0x5654a3299b40 2023-10-19T14:21:19.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.544+0000 7f9e30ff9640 1 -- 172.21.15.57:0/276470299 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(243..243 src has 1..243) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f9e440277a0 con 0x5654a3299b40 2023-10-19T14:21:19.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.544+0000 7f9e567a8740 1 -- 172.21.15.57:0/276470299 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5654a31d1740 con 0x5654a3299b40 2023-10-19T14:21:19.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.544+0000 7f9e30ff9640 1 -- 172.21.15.57:0/276470299 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=103) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9e4401ced0 con 0x5654a3299b40 2023-10-19T14:21:19.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.544+0000 7f9e567a8740 10 client.12723 fetch_fsmap learned FSMap version 103 2023-10-19T14:21:19.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.544+0000 7f9e567a8740 1 -- 172.21.15.57:0/276470299 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=103,monmap=2+}) v3 -- 0x5654a31d6de0 con 0x5654a3299b40 2023-10-19T14:21:19.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.544+0000 7f9e30ff9640 1 -- 172.21.15.57:0/276470299 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x5654a31d1740 con 0x5654a3299b40 2023-10-19T14:21:19.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.545+0000 7f9e30ff9640 1 -- 172.21.15.57:0/276470299 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 103) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f9e44027a90 con 0x5654a3299b40 2023-10-19T14:21:19.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.545+0000 7f9e567a8740 10 client.12723 fetch_fsmap finished waiting for FSMap version 103 2023-10-19T14:21:19.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.545+0000 7f9e567a8740 10 client.12723 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-19T14:21:19.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.545+0000 7f9e567a8740 20 client.12723 populate_metadata read hostname 'smithi057' 2023-10-19T14:21:19.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.545+0000 7f9e567a8740 1 --2- 172.21.15.57:0/276470299 >> [v2:172.21.15.57:6834/18187246,v1:172.21.15.57:6835/18187246] conn(0x5654a328a260 0x5654a328a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:19.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.545+0000 7f9e567a8740 4 client.12723 mds_command: new command op to 12564 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-19T14:21:19.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.545+0000 7f9e567a8740 1 -- 172.21.15.57:0/276470299 --> [v2:172.21.15.57:6834/18187246,v1:172.21.15.57:6835/18187246] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5654a31d1460 con 0x5654a328a260 2023-10-19T14:21:19.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.545+0000 7f9e510d9640 1 --2- 172.21.15.57:0/276470299 >> [v2:172.21.15.57:6834/18187246,v1:172.21.15.57:6835/18187246] conn(0x5654a328a260 0x5654a328a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:19.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.545+0000 7f9e30ff9640 10 client.12723 ms_handle_connect on v2:172.21.15.57:6834/18187246 2023-10-19T14:21:19.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.545+0000 7f9e510d9640 1 --2- 172.21.15.57:0/276470299 >> [v2:172.21.15.57:6834/18187246,v1:172.21.15.57:6835/18187246] conn(0x5654a328a260 0x5654a328a640 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1445008c2b3a22e9 server_cookie=c76f7e56b3e42590 in_seq=0 out_seq=0 2023-10-19T14:21:19.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.546+0000 7f9e30ff9640 1 -- 172.21.15.57:0/276470299 <== mds.0 v2:172.21.15.57:6834/18187246 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5654a31d1460 con 0x5654a328a260 2023-10-19T14:21:19.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.546+0000 7f9e30ff9640 10 client.12723 handle_command_reply: tid=0 2023-10-19T14:21:19.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.546+0000 7f9e567a8740 1 client.12723 shutdown 2023-10-19T14:21:19.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.546+0000 7f9e567a8740 1 -- 172.21.15.57:0/276470299 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5654a3299b40 msgr2=0x5654a329f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:19.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.546+0000 7f9e567a8740 1 --2- 172.21.15.57:0/276470299 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5654a3299b40 0x5654a329f360 secure :-1 s=READY pgs=2652 cs=0 l=1 rev1=1 crypto rx=0x7f9e440098b0 tx=0x7f9e4401cca0 comp rx=0 tx=0).stop 2023-10-19T14:21:19.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.546+0000 7f9e567a8740 1 -- 172.21.15.57:0/276470299 shutdown_connections 2023-10-19T14:21:19.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.546+0000 7f9e567a8740 1 --2- 172.21.15.57:0/276470299 >> [v2:172.21.15.57:6834/18187246,v1:172.21.15.57:6835/18187246] conn(0x5654a328a260 0x5654a328a640 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:19.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.546+0000 7f9e567a8740 1 --2- 172.21.15.57:0/276470299 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5654a32b24c0 0x5654a3287450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:19.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.546+0000 7f9e567a8740 1 --2- 172.21.15.57:0/276470299 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x5654a3299f20 0x5654a32c7cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:19.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.546+0000 7f9e567a8740 1 --2- 172.21.15.57:0/276470299 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5654a3299b40 0x5654a329f360 unknown :-1 s=CLOSED pgs=2652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:19.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.546+0000 7f9e30ff9640 0 client.12723 ms_handle_reset on v2:172.21.15.57:6834/18187246 2023-10-19T14:21:19.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.546+0000 7f9e567a8740 1 -- 172.21.15.57:0/276470299 >> 172.21.15.57:0/276470299 conn(0x5654a326d020 msgr2=0x5654a32708f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:19.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.546+0000 7f9e567a8740 1 -- 172.21.15.57:0/276470299 shutdown_connections 2023-10-19T14:21:19.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.546+0000 7f9e567a8740 1 -- 172.21.15.57:0/276470299 wait complete. 2023-10-19T14:21:19.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.546+0000 7f9e567a8740 20 client.12723 trim_cache size 0 max 16384 2023-10-19T14:21:19.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.561+0000 7f9e567a8740 1 Processor -- start 2023-10-19T14:21:19.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.562+0000 7f9e567a8740 1 -- start start 2023-10-19T14:21:19.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.562+0000 7f9e567a8740 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x5654a3299b40 0x5654a326dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:19.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.562+0000 7f9e567a8740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5654a3299f20 0x5654a32b1030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:19.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.562+0000 7f9e567a8740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5654a32b24c0 0x5654a329ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:19.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.562+0000 7f9e567a8740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654a3159270 con 0x5654a3299f20 2023-10-19T14:21:19.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.562+0000 7f9e567a8740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654a32b35e0 con 0x5654a32b24c0 2023-10-19T14:21:19.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.562+0000 7f9e567a8740 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x5654a32ca8b0 con 0x5654a3299b40 2023-10-19T14:21:19.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.562+0000 7f9e510d9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5654a3299f20 0x5654a32b1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:19.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.562+0000 7f9e510d9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5654a3299f20 0x5654a32b1030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57490/0 (socket says 172.21.15.57:57490) 2023-10-19T14:21:19.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.562+0000 7f9e510d9640 1 -- 172.21.15.57:0/1187138151 learned_addr learned my addr 172.21.15.57:0/1187138151 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:19.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.562+0000 7f9e520db640 1 --2- 172.21.15.57:0/1187138151 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5654a32b24c0 0x5654a329ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:19.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.563+0000 7f9e518da640 1 --2- 172.21.15.57:0/1187138151 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x5654a3299b40 0x5654a326dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:19.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.563+0000 7f9e510d9640 1 -- 172.21.15.57:0/1187138151 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x5654a3299b40 msgr2=0x5654a326dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T14:21:19.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.563+0000 7f9e510d9640 1 --2- 172.21.15.57:0/1187138151 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x5654a3299b40 0x5654a326dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:19.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.563+0000 7f9e510d9640 1 -- 172.21.15.57:0/1187138151 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5654a32b24c0 msgr2=0x5654a329ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T14:21:19.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.563+0000 7f9e510d9640 1 --2- 172.21.15.57:0/1187138151 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5654a32b24c0 0x5654a329ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:19.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.563+0000 7f9e510d9640 1 -- 172.21.15.57:0/1187138151 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5654a3175d00 con 0x5654a3299f20 2023-10-19T14:21:19.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.563+0000 7f9e520db640 1 --2- 172.21.15.57:0/1187138151 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5654a32b24c0 0x5654a329ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:21:19.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.563+0000 7f9e510d9640 1 --2- 172.21.15.57:0/1187138151 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5654a3299f20 0x5654a32b1030 secure :-1 s=READY pgs=2653 cs=0 l=1 rev1=1 crypto rx=0x7f9e40054fa0 tx=0x7f9e400585e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7ec9d475ee42915 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:19.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.563+0000 7f9e33fff640 1 -- 172.21.15.57:0/1187138151 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e4008a0c0 con 0x5654a3299f20 2023-10-19T14:21:19.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.563+0000 7f9e33fff640 1 -- 172.21.15.57:0/1187138151 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f9e40004e10 con 0x5654a3299f20 2023-10-19T14:21:19.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.563+0000 7f9e33fff640 1 -- 172.21.15.57:0/1187138151 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e40010e10 con 0x5654a3299f20 2023-10-19T14:21:19.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.563+0000 7f9e567a8740 1 -- 172.21.15.57:0/1187138151 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5654a3299f20 msgr2=0x5654a32b1030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:19.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.563+0000 7f9e567a8740 1 --2- 172.21.15.57:0/1187138151 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5654a3299f20 0x5654a32b1030 secure :-1 s=READY pgs=2653 cs=0 l=1 rev1=1 crypto rx=0x7f9e40054fa0 tx=0x7f9e400585e0 comp rx=0 tx=0).stop 2023-10-19T14:21:19.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.564+0000 7f9e567a8740 1 -- 172.21.15.57:0/1187138151 shutdown_connections 2023-10-19T14:21:19.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.564+0000 7f9e567a8740 1 --2- 172.21.15.57:0/1187138151 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5654a32b24c0 0x5654a329ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:19.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.564+0000 7f9e567a8740 1 --2- 172.21.15.57:0/1187138151 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5654a3299f20 0x5654a32b1030 unknown :-1 s=CLOSED pgs=2653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:19.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.564+0000 7f9e567a8740 1 --2- 172.21.15.57:0/1187138151 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x5654a3299b40 0x5654a326dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:19.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.564+0000 7f9e567a8740 1 -- 172.21.15.57:0/1187138151 >> 172.21.15.57:0/1187138151 conn(0x5654a326d020 msgr2=0x5654a32c7170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:19.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.564+0000 7f9e567a8740 1 -- 172.21.15.57:0/1187138151 shutdown_connections 2023-10-19T14:21:19.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.564+0000 7f9e567a8740 1 -- 172.21.15.57:0/1187138151 wait complete. 2023-10-19T14:21:19.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.564+0000 7f9e567a8740 1 -- start start 2023-10-19T14:21:19.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.564+0000 7f9e567a8740 1 Processor -- start 2023-10-19T14:21:19.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.565+0000 7f9e567a8740 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x5654a3299b40 0x5654a3298790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:19.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.565+0000 7f9e567a8740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5654a3299f20 0x5654a32718e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:19.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.565+0000 7f9e567a8740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5654a32b24c0 0x5654a3270640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:19.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.565+0000 7f9e567a8740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654a329c560 con 0x5654a3299f20 2023-10-19T14:21:19.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.565+0000 7f9e567a8740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654a3289400 con 0x5654a32b24c0 2023-10-19T14:21:19.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.565+0000 7f9e567a8740 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x5654a32b2050 con 0x5654a3299b40 2023-10-19T14:21:19.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.565+0000 7f9e567a8740 10 client.0 _ll_register_callbacks cb 0x7f9e440097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-19T14:21:19.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.565+0000 7f9e510d9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5654a3299f20 0x5654a32718e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:19.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.565+0000 7f9e510d9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5654a3299f20 0x5654a32718e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57496/0 (socket says 172.21.15.57:57496) 2023-10-19T14:21:19.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.565+0000 7f9e510d9640 1 -- 172.21.15.57:0/3741541818 learned_addr learned my addr 172.21.15.57:0/3741541818 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:19.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.565+0000 7f9e518da640 1 --2- 172.21.15.57:0/3741541818 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x5654a3299b40 0x5654a3298790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:19.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.565+0000 7f9e520db640 1 --2- 172.21.15.57:0/3741541818 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5654a32b24c0 0x5654a3270640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:19.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.565+0000 7f9e510d9640 1 -- 172.21.15.57:0/3741541818 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x5654a3299b40 msgr2=0x5654a3298790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T14:21:19.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.565+0000 7f9e510d9640 1 --2- 172.21.15.57:0/3741541818 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x5654a3299b40 0x5654a3298790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:19.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.565+0000 7f9e510d9640 1 -- 172.21.15.57:0/3741541818 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5654a32b24c0 msgr2=0x5654a3270640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T14:21:19.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.565+0000 7f9e510d9640 1 --2- 172.21.15.57:0/3741541818 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5654a32b24c0 0x5654a3270640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:19.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.565+0000 7f9e510d9640 1 -- 172.21.15.57:0/3741541818 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e4c0afd00 con 0x5654a3299f20 2023-10-19T14:21:19.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.566+0000 7f9e510d9640 1 --2- 172.21.15.57:0/3741541818 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5654a3299f20 0x5654a32718e0 secure :-1 s=READY pgs=2654 cs=0 l=1 rev1=1 crypto rx=0x7f9e40004cf0 tx=0x7f9e40019320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a5c1292bc21015f server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:19.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.566+0000 7f9e30ff9640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2023-10-19T14:21:19.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.566+0000 7f9e30ff9640 1 -- 172.21.15.57:0/3741541818 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e40010e10 con 0x5654a3299f20 2023-10-19T14:21:19.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.566+0000 7f9e30ff9640 1 -- 172.21.15.57:0/3741541818 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(243..243 src has 1..243) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f9e400551b0 con 0x5654a3299f20 2023-10-19T14:21:19.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.566+0000 7f9e567a8740 1 -- 172.21.15.57:0/3741541818 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9e4401c7e0 con 0x5654a3299f20 2023-10-19T14:21:19.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.566+0000 7f9e30ff9640 1 -- 172.21.15.57:0/3741541818 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=103) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9e40055370 con 0x5654a3299f20 2023-10-19T14:21:19.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.566+0000 7f9e567a8740 10 client.12729 fetch_fsmap learned FSMap version 103 2023-10-19T14:21:19.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.566+0000 7f9e567a8740 1 -- 172.21.15.57:0/3741541818 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=103,monmap=2+}) v3 -- 0x5654a319b4f0 con 0x5654a3299f20 2023-10-19T14:21:19.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.566+0000 7f9e30ff9640 1 -- 172.21.15.57:0/3741541818 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f9e4401c7e0 con 0x5654a3299f20 2023-10-19T14:21:19.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.566+0000 7f9e30ff9640 1 -- 172.21.15.57:0/3741541818 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 103) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f9e40070200 con 0x5654a3299f20 2023-10-19T14:21:19.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.566+0000 7f9e567a8740 10 client.12729 fetch_fsmap finished waiting for FSMap version 103 2023-10-19T14:21:19.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.566+0000 7f9e567a8740 10 client.12729 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-19T14:21:19.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.566+0000 7f9e567a8740 20 client.12729 populate_metadata read hostname 'smithi057' 2023-10-19T14:21:19.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.566+0000 7f9e567a8740 1 --2- 172.21.15.57:0/3741541818 >> [v2:172.21.15.57:6834/18187246,v1:172.21.15.57:6835/18187246] conn(0x5654a3286570 0x5654a3270b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:19.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.566+0000 7f9e567a8740 4 client.12729 mds_command: new command op to 12564 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-19T14:21:19.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.566+0000 7f9e567a8740 1 -- 172.21.15.57:0/3741541818 --> [v2:172.21.15.57:6834/18187246,v1:172.21.15.57:6835/18187246] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5654a2eda6c0 con 0x5654a3286570 2023-10-19T14:21:19.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.566+0000 7f9e518da640 1 --2- 172.21.15.57:0/3741541818 >> [v2:172.21.15.57:6834/18187246,v1:172.21.15.57:6835/18187246] conn(0x5654a3286570 0x5654a3270b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:19.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.567+0000 7f9e30ff9640 10 client.12729 ms_handle_connect on v2:172.21.15.57:6834/18187246 2023-10-19T14:21:19.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.567+0000 7f9e518da640 1 --2- 172.21.15.57:0/3741541818 >> [v2:172.21.15.57:6834/18187246,v1:172.21.15.57:6835/18187246] conn(0x5654a3286570 0x5654a3270b80 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a2ec953e28c7a61 server_cookie=a1b5c4adc23255b7 in_seq=0 out_seq=0 2023-10-19T14:21:19.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.567+0000 7f9e30ff9640 1 -- 172.21.15.57:0/3741541818 <== mds.0 v2:172.21.15.57:6834/18187246 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+328 (crc 0 0 0) 0x5654a2eda6c0 con 0x5654a3286570 2023-10-19T14:21:19.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.567+0000 7f9e30ff9640 10 client.12729 handle_command_reply: tid=0 2023-10-19T14:21:19.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.567+0000 7f9e567a8740 1 client.12729 shutdown 2023-10-19T14:21:19.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.567+0000 7f9e567a8740 1 -- 172.21.15.57:0/3741541818 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5654a3299f20 msgr2=0x5654a32718e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:19.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.567+0000 7f9e567a8740 1 --2- 172.21.15.57:0/3741541818 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5654a3299f20 0x5654a32718e0 secure :-1 s=READY pgs=2654 cs=0 l=1 rev1=1 crypto rx=0x7f9e40004cf0 tx=0x7f9e40019320 comp rx=0 tx=0).stop 2023-10-19T14:21:19.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.567+0000 7f9e567a8740 1 -- 172.21.15.57:0/3741541818 shutdown_connections 2023-10-19T14:21:19.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.567+0000 7f9e567a8740 1 --2- 172.21.15.57:0/3741541818 >> [v2:172.21.15.57:6834/18187246,v1:172.21.15.57:6835/18187246] conn(0x5654a3286570 0x5654a3270b80 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:19.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.567+0000 7f9e567a8740 1 --2- 172.21.15.57:0/3741541818 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5654a32b24c0 0x5654a3270640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:19.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.567+0000 7f9e567a8740 1 --2- 172.21.15.57:0/3741541818 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5654a3299f20 0x5654a32718e0 unknown :-1 s=CLOSED pgs=2654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:19.596 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.567+0000 7f9e567a8740 1 --2- 172.21.15.57:0/3741541818 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x5654a3299b40 0x5654a3298790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:19.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.567+0000 7f9e567a8740 1 -- 172.21.15.57:0/3741541818 >> 172.21.15.57:0/3741541818 conn(0x5654a326d020 msgr2=0x5654a32c7090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:19.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.567+0000 7f9e30ff9640 0 client.12729 ms_handle_reset on v2:172.21.15.57:6834/18187246 2023-10-19T14:21:19.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.567+0000 7f9e567a8740 1 -- 172.21.15.57:0/3741541818 shutdown_connections 2023-10-19T14:21:19.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.567+0000 7f9e567a8740 1 -- 172.21.15.57:0/3741541818 wait complete. 2023-10-19T14:21:19.597 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.567+0000 7f9e567a8740 20 client.12729 trim_cache size 0 max 16384 2023-10-19T14:21:19.598 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2023-10-19T14:21:19.598 INFO:teuthology.orchestra.run.smithi057.stdout: "cluster_fsid": "1e9beb19-3260-4dd7-85ef-5b399ce774ac", 2023-10-19T14:21:19.598 INFO:teuthology.orchestra.run.smithi057.stdout: "whoami": 0, 2023-10-19T14:21:19.599 INFO:teuthology.orchestra.run.smithi057.stdout: "id": 12564, 2023-10-19T14:21:19.599 INFO:teuthology.orchestra.run.smithi057.stdout: "want_state": "up:active", 2023-10-19T14:21:19.599 INFO:teuthology.orchestra.run.smithi057.stdout: "state": "up:active", 2023-10-19T14:21:19.599 INFO:teuthology.orchestra.run.smithi057.stdout: "fs_name": "cephfs", 2023-10-19T14:21:19.599 INFO:teuthology.orchestra.run.smithi057.stdout: "rank_uptime": 9.1469728220000004, 2023-10-19T14:21:19.599 INFO:teuthology.orchestra.run.smithi057.stdout: "mdsmap_epoch": 103, 2023-10-19T14:21:19.600 INFO:teuthology.orchestra.run.smithi057.stdout: "osdmap_epoch": 243, 2023-10-19T14:21:19.600 INFO:teuthology.orchestra.run.smithi057.stdout: "osdmap_epoch_barrier": 243, 2023-10-19T14:21:19.600 INFO:teuthology.orchestra.run.smithi057.stdout: "uptime": 13.86372691 2023-10-19T14:21:19.600 INFO:teuthology.orchestra.run.smithi057.stdout:} 2023-10-19T14:21:19.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.568+0000 7f9e53b3e640 1 -- 172.21.15.57:0/1953775925 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f9e3806b080 msgr2=0x7f9e3806d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:19.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.568+0000 7f9e53b3e640 1 --2- 172.21.15.57:0/1953775925 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f9e3806b080 0x7f9e3806d540 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9e44009460 tx=0x7f9e44005f90 comp rx=0 tx=0).stop 2023-10-19T14:21:19.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.568+0000 7f9e53b3e640 1 -- 172.21.15.57:0/1953775925 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9e4c1526c0 msgr2=0x7f9e4c13c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:19.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.568+0000 7f9e53b3e640 1 --2- 172.21.15.57:0/1953775925 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9e4c1526c0 0x7f9e4c13c790 secure :-1 s=READY pgs=2650 cs=0 l=1 rev1=1 crypto rx=0x7f9e40004d20 tx=0x7f9e40001630 comp rx=0 tx=0).stop 2023-10-19T14:21:19.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.568+0000 7f9e53b3e640 1 -- 172.21.15.57:0/1953775925 shutdown_connections 2023-10-19T14:21:19.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.568+0000 7f9e53b3e640 1 --2- 172.21.15.57:0/1953775925 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f9e3806b080 0x7f9e3806d540 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:19.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.568+0000 7f9e53b3e640 1 --2- 172.21.15.57:0/1953775925 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9e4c13fa20 0x7f9e4c15bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:19.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.568+0000 7f9e53b3e640 1 --2- 172.21.15.57:0/1953775925 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9e4c1526c0 0x7f9e4c13c790 unknown :-1 s=CLOSED pgs=2650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:19.603 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.568+0000 7f9e53b3e640 1 --2- 172.21.15.57:0/1953775925 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9e34005180 0x7f9e4c13cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:19.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.568+0000 7f9e53b3e640 1 -- 172.21.15.57:0/1953775925 >> 172.21.15.57:0/1953775925 conn(0x7f9e4c0b3f00 msgr2=0x7f9e4c0ab9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:19.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.568+0000 7f9e53b3e640 1 -- 172.21.15.57:0/1953775925 shutdown_connections 2023-10-19T14:21:19.604 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.568+0000 7f9e53b3e640 1 -- 172.21.15.57:0/1953775925 wait complete. 2023-10-19T14:21:19.604 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-19T14:21:19.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.711+0000 7f8ee0173640 1 -- 172.21.15.57:0/2935049359 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ed80cf340 msgr2=0x7f8ed80d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:19.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.711+0000 7f8ee0173640 1 --2- 172.21.15.57:0/2935049359 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ed80cf340 0x7f8ed80d1910 secure :-1 s=READY pgs=2656 cs=0 l=1 rev1=1 crypto rx=0x7f8ed808a120 tx=0x7f8ed402bcf0 comp rx=0 tx=0).stop 2023-10-19T14:21:19.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.711+0000 7f8ee0173640 1 -- 172.21.15.57:0/2935049359 shutdown_connections 2023-10-19T14:21:19.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.711+0000 7f8ee0173640 1 --2- 172.21.15.57:0/2935049359 >> v2:172.21.15.179:3300/0 conn(0x7f8ed815b2a0 0x7f8ed80cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:19.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.711+0000 7f8ee0173640 1 --2- 172.21.15.57:0/2935049359 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ed80cf340 0x7f8ed80d1910 unknown :-1 s=CLOSED pgs=2656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:19.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.711+0000 7f8ee0173640 1 -- 172.21.15.57:0/2935049359 >> 172.21.15.57:0/2935049359 conn(0x7f8ed80d0d60 msgr2=0x7f8ed80d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:19.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.711+0000 7f8ee0173640 1 -- 172.21.15.57:0/2935049359 shutdown_connections 2023-10-19T14:21:19.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.711+0000 7f8ee0173640 1 -- 172.21.15.57:0/2935049359 wait complete. 2023-10-19T14:21:19.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.713+0000 7f8ee0173640 1 Processor -- start 2023-10-19T14:21:19.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.713+0000 7f8ee0173640 1 -- start start 2023-10-19T14:21:19.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.713+0000 7f8ee0173640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8ed80ceb40 0x7f8ed80ae5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:19.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.713+0000 7f8ee0173640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ed80cf340 0x7f8ed80b41d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:19.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.713+0000 7f8ee0173640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8ed815b2a0 0x7f8ed8156ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:19.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.713+0000 7f8ee0173640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed815bac0 con 0x7f8ed80cf340 2023-10-19T14:21:19.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.713+0000 7f8ee0173640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed809e4a0 con 0x7f8ed80ceb40 2023-10-19T14:21:19.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.713+0000 7f8ee0173640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed809e740 con 0x7f8ed815b2a0 2023-10-19T14:21:19.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.713+0000 7f8edd70e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ed80cf340 0x7f8ed80b41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:19.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.713+0000 7f8edd70e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ed80cf340 0x7f8ed80b41d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57512/0 (socket says 172.21.15.57:57512) 2023-10-19T14:21:19.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.713+0000 7f8edd70e640 1 -- 172.21.15.57:0/2788947285 learned_addr learned my addr 172.21.15.57:0/2788947285 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:19.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.714+0000 7f8eddf0f640 1 --2- 172.21.15.57:0/2788947285 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8ed80ceb40 0x7f8ed80ae5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:19.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.714+0000 7f8ede710640 1 --2- 172.21.15.57:0/2788947285 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8ed815b2a0 0x7f8ed8156ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:19.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.714+0000 7f8edd70e640 1 -- 172.21.15.57:0/2788947285 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8ed815b2a0 msgr2=0x7f8ed8156ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:19.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.714+0000 7f8edd70e640 1 --2- 172.21.15.57:0/2788947285 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8ed815b2a0 0x7f8ed8156ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:19.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.714+0000 7f8edd70e640 1 -- 172.21.15.57:0/2788947285 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8ed80ceb40 msgr2=0x7f8ed80ae5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:19.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.714+0000 7f8edd70e640 1 --2- 172.21.15.57:0/2788947285 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8ed80ceb40 0x7f8ed80ae5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:19.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.714+0000 7f8edd70e640 1 -- 172.21.15.57:0/2788947285 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ed4009490 con 0x7f8ed80cf340 2023-10-19T14:21:19.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.714+0000 7f8ede710640 1 --2- 172.21.15.57:0/2788947285 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8ed815b2a0 0x7f8ed8156ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:21:19.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.714+0000 7f8edd70e640 1 --2- 172.21.15.57:0/2788947285 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ed80cf340 0x7f8ed80b41d0 secure :-1 s=READY pgs=2657 cs=0 l=1 rev1=1 crypto rx=0x7f8ec800e790 tx=0x7f8ec800ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:19.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.714+0000 7f8ece7fc640 1 -- 172.21.15.57:0/2788947285 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ec8009580 con 0x7f8ed80cf340 2023-10-19T14:21:19.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.714+0000 7f8ece7fc640 1 -- 172.21.15.57:0/2788947285 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f8ec800cdb0 con 0x7f8ed80cf340 2023-10-19T14:21:19.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.714+0000 7f8ee0173640 1 -- 172.21.15.57:0/2788947285 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ed809ea20 con 0x7f8ed80cf340 2023-10-19T14:21:19.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.714+0000 7f8ece7fc640 1 -- 172.21.15.57:0/2788947285 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ec8010690 con 0x7f8ed80cf340 2023-10-19T14:21:19.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.714+0000 7f8ee0173640 1 -- 172.21.15.57:0/2788947285 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ed80b6b20 con 0x7f8ed80cf340 2023-10-19T14:21:19.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.715+0000 7f8ee0173640 1 -- 172.21.15.57:0/2788947285 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ed815bc30 con 0x7f8ed80cf340 2023-10-19T14:21:19.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.716+0000 7f8ece7fc640 1 -- 172.21.15.57:0/2788947285 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f8ec80107f0 con 0x7f8ed80cf340 2023-10-19T14:21:19.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.716+0000 7f8ece7fc640 1 --2- 172.21.15.57:0/2788947285 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f8ebc06b0b0 0x7f8ebc06d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:19.724 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.716+0000 7f8ece7fc640 1 -- 172.21.15.57:0/2788947285 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(243..243 src has 1..243) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f8ec8014070 con 0x7f8ed80cf340 2023-10-19T14:21:19.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.716+0000 7f8eddf0f640 1 --2- 172.21.15.57:0/2788947285 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f8ebc06b0b0 0x7f8ebc06d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:19.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.717+0000 7f8eddf0f640 1 --2- 172.21.15.57:0/2788947285 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f8ebc06b0b0 0x7f8ebc06d570 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8ed4036000 tx=0x7f8ed4004730 comp rx=0 tx=0).ready entity=mgr.9530 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:19.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.720+0000 7f8ece7fc640 1 -- 172.21.15.57:0/2788947285 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8ec8051ca0 con 0x7f8ed80cf340 2023-10-19T14:21:19.935 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.932+0000 7f8ee0173640 1 -- 172.21.15.57:0/2788947285 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f8ed80690e0 con 0x7f8ed80cf340 2023-10-19T14:21:19.935 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.932+0000 7f8ece7fc640 1 -- 172.21.15.57:0/2788947285 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v103) v1 ==== 53+0+73 (secure 0 0 0) 0x7f8ec8055230 con 0x7f8ed80cf340 2023-10-19T14:21:19.936 INFO:teuthology.orchestra.run.smithi057.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-19T14:21:19.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.935+0000 7f8ee0173640 1 -- 172.21.15.57:0/2788947285 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f8ebc06b0b0 msgr2=0x7f8ebc06d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:19.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.935+0000 7f8ee0173640 1 --2- 172.21.15.57:0/2788947285 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f8ebc06b0b0 0x7f8ebc06d570 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8ed4036000 tx=0x7f8ed4004730 comp rx=0 tx=0).stop 2023-10-19T14:21:19.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.935+0000 7f8ee0173640 1 -- 172.21.15.57:0/2788947285 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ed80cf340 msgr2=0x7f8ed80b41d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:19.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.935+0000 7f8ee0173640 1 --2- 172.21.15.57:0/2788947285 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ed80cf340 0x7f8ed80b41d0 secure :-1 s=READY pgs=2657 cs=0 l=1 rev1=1 crypto rx=0x7f8ec800e790 tx=0x7f8ec800ec50 comp rx=0 tx=0).stop 2023-10-19T14:21:19.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.935+0000 7f8ee0173640 1 -- 172.21.15.57:0/2788947285 shutdown_connections 2023-10-19T14:21:19.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.935+0000 7f8ee0173640 1 --2- 172.21.15.57:0/2788947285 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f8ebc06b0b0 0x7f8ebc06d570 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:19.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.935+0000 7f8ee0173640 1 --2- 172.21.15.57:0/2788947285 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8ed815b2a0 0x7f8ed8156ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:19.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.935+0000 7f8ee0173640 1 --2- 172.21.15.57:0/2788947285 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ed80cf340 0x7f8ed80b41d0 unknown :-1 s=CLOSED pgs=2657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:19.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.935+0000 7f8ee0173640 1 --2- 172.21.15.57:0/2788947285 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8ed80ceb40 0x7f8ed80ae5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:19.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.935+0000 7f8ee0173640 1 -- 172.21.15.57:0/2788947285 >> 172.21.15.57:0/2788947285 conn(0x7f8ed80d0d60 msgr2=0x7f8ed80ad8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:19.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.935+0000 7f8ee0173640 1 -- 172.21.15.57:0/2788947285 shutdown_connections 2023-10-19T14:21:19.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:19.935+0000 7f8ee0173640 1 -- 172.21.15.57:0/2788947285 wait complete. 2023-10-19T14:21:19.948 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-10-19T14:21:19.948 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-19T14:21:19.949 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi057.front.sepia.ceph.com 2023-10-19T14:21:19.949 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-19T14:21:19.950 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-19T14:21:19.950 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-10-19T14:21:19.950 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-10-19T14:21:19.951 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-19T14:21:19.951 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-19T14:21:19.951 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:21:19.952 DEBUG:teuthology.orchestra.run.smithi057:> ip addr 2023-10-19T14:21:19.968 INFO:teuthology.orchestra.run.smithi057.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-19T14:21:19.968 INFO:teuthology.orchestra.run.smithi057.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-19T14:21:19.968 INFO:teuthology.orchestra.run.smithi057.stdout: inet 127.0.0.1/8 scope host lo 2023-10-19T14:21:19.969 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T14:21:19.969 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 ::1/128 scope host 2023-10-19T14:21:19.969 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T14:21:19.969 INFO:teuthology.orchestra.run.smithi057.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-19T14:21:19.969 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:6b:ff:86 brd ff:ff:ff:ff:ff:ff 2023-10-19T14:21:19.970 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp7s0f0 2023-10-19T14:21:19.970 INFO:teuthology.orchestra.run.smithi057.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-19T14:21:19.970 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:6b:ff:87 brd ff:ff:ff:ff:ff:ff 2023-10-19T14:21:19.970 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp7s0f1 2023-10-19T14:21:19.971 INFO:teuthology.orchestra.run.smithi057.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-19T14:21:19.971 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:bd:15:62 brd ff:ff:ff:ff:ff:ff 2023-10-19T14:21:19.971 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp1s0f0 2023-10-19T14:21:19.971 INFO:teuthology.orchestra.run.smithi057.stdout: inet 172.21.15.57/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-19T14:21:19.971 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft 36831sec preferred_lft 36831sec 2023-10-19T14:21:19.972 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 fe80::ec4:7aff:febd:1562/64 scope link noprefixroute 2023-10-19T14:21:19.972 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T14:21:19.972 INFO:teuthology.orchestra.run.smithi057.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-19T14:21:19.972 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:bd:15:63 brd ff:ff:ff:ff:ff:ff 2023-10-19T14:21:19.972 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp1s0f1 2023-10-19T14:21:19.972 INFO:teuthology.orchestra.run.smithi057.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-19T14:21:19.973 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0e:fe:f1:b5:41:59 brd ff:ff:ff:ff:ff:ff 2023-10-19T14:21:19.973 INFO:teuthology.orchestra.run.smithi057.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-19T14:21:19.973 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T14:21:19.973 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 fe80::64b1:a2ff:feb3:b08/64 scope link 2023-10-19T14:21:19.973 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T14:21:19.974 INFO:teuthology.orchestra.run.smithi057.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-19T14:21:19.974 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0e:fe:f1:b5:41:59 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T14:21:19.974 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 fe80::cfe:f1ff:feb5:4159/64 scope link 2023-10-19T14:21:19.974 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T14:21:19.975 INFO:teuthology.orchestra.run.smithi057.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-19T14:21:19.975 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 62:53:72:9d:52:71 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-19T14:21:19.975 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 fe80::6053:72ff:fe9d:5271/64 scope link 2023-10-19T14:21:19.975 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T14:21:19.975 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:21:19.976 DEBUG:teuthology.orchestra.run.smithi057:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-19T14:21:20.037 INFO:teuthology.orchestra.run.smithi057.stdout:1 2023-10-19T14:21:20.038 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:21:20.039 DEBUG:teuthology.orchestra.run.smithi057:> route 2023-10-19T14:21:20.094 INFO:teuthology.orchestra.run.smithi057.stdout:Kernel IP routing table 2023-10-19T14:21:20.095 INFO:teuthology.orchestra.run.smithi057.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-19T14:21:20.095 INFO:teuthology.orchestra.run.smithi057.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-19T14:21:20.095 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-19T14:21:20.095 INFO:teuthology.orchestra.run.smithi057.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-19T14:21:20.096 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:21:20.096 DEBUG:teuthology.orchestra.run.smithi057:> (cd / && exec bash -c ' 2023-10-19T14:21:20.097 DEBUG:teuthology.orchestra.run.smithi057:> set -e 2023-10-19T14:21:20.097 DEBUG:teuthology.orchestra.run.smithi057:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-19T14:21:20.097 DEBUG:teuthology.orchestra.run.smithi057:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-19T14:21:20.098 DEBUG:teuthology.orchestra.run.smithi057:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-19T14:21:20.098 DEBUG:teuthology.orchestra.run.smithi057:> ') 2023-10-19T14:21:20.200 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:21:20.201 DEBUG:teuthology.orchestra.run.smithi057:> ip netns list 2023-10-19T14:21:20.216 INFO:teuthology.orchestra.run.smithi057.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-19T14:21:20.217 INFO:teuthology.orchestra.run.smithi057.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-19T14:21:20.217 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-19T14:21:20.217 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:21:20.218 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip link set brx.0 up 2023-10-19T14:21:20.287 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-10-19T14:21:20.287 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-19T14:21:20.288 DEBUG:teuthology.orchestra.run.smithi057:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-19T14:21:20.343 INFO:teuthology.orchestra.run.smithi057.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-19T14:21:20.343 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-19T14:21:20.344 DEBUG:teuthology.orchestra.run.smithi057:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-19T14:21:20.398 DEBUG:teuthology.orchestra.run.smithi057:> sudo modprobe fuse 2023-10-19T14:21:20.468 DEBUG:teuthology.orchestra.run.smithi057:> cat /proc/self/mounts | awk '{print $2}' 2023-10-19T14:21:20.525 INFO:teuthology.orchestra.run.smithi057.stdout:/proc 2023-10-19T14:21:20.525 INFO:teuthology.orchestra.run.smithi057.stdout:/sys 2023-10-19T14:21:20.525 INFO:teuthology.orchestra.run.smithi057.stdout:/dev 2023-10-19T14:21:20.526 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/security 2023-10-19T14:21:20.526 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/shm 2023-10-19T14:21:20.526 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/pts 2023-10-19T14:21:20.526 INFO:teuthology.orchestra.run.smithi057.stdout:/run 2023-10-19T14:21:20.527 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/cgroup 2023-10-19T14:21:20.527 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/pstore 2023-10-19T14:21:20.527 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/bpf 2023-10-19T14:21:20.527 INFO:teuthology.orchestra.run.smithi057.stdout:/ 2023-10-19T14:21:20.527 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/selinux 2023-10-19T14:21:20.528 INFO:teuthology.orchestra.run.smithi057.stdout:/proc/sys/fs/binfmt_misc 2023-10-19T14:21:20.528 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/hugepages 2023-10-19T14:21:20.528 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/mqueue 2023-10-19T14:21:20.528 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/debug 2023-10-19T14:21:20.528 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/tracing 2023-10-19T14:21:20.529 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/fuse/connections 2023-10-19T14:21:20.529 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/config 2023-10-19T14:21:20.529 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-sysctl.service 2023-10-19T14:21:20.530 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-10-19T14:21:20.530 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-10-19T14:21:20.530 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/nfs/rpc_pipefs 2023-10-19T14:21:20.531 INFO:teuthology.orchestra.run.smithi057.stdout:/run/user/1000 2023-10-19T14:21:20.531 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph 2023-10-19T14:21:20.531 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-0 2023-10-19T14:21:20.531 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-1 2023-10-19T14:21:20.531 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-2 2023-10-19T14:21:20.532 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-3 2023-10-19T14:21:20.532 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns 2023-10-19T14:21:20.532 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T14:21:20.533 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T14:21:20.533 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-19T14:21:20.533 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-19T14:21:20.533 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:21:20.534 DEBUG:teuthology.orchestra.run.smithi057:> ls /sys/fs/fuse/connections 2023-10-19T14:21:20.580 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-10-19T14:21:20.581 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-10-19T14:21:20.623 DEBUG:teuthology.orchestra.run.smithi057:> sudo modprobe fuse 2023-10-19T14:21:20.653 DEBUG:teuthology.orchestra.run.smithi057:> cat /proc/self/mounts | awk '{print $2}' 2023-10-19T14:21:20.709 INFO:teuthology.orchestra.run.smithi057.stdout:/proc 2023-10-19T14:21:20.709 INFO:teuthology.orchestra.run.smithi057.stdout:/sys 2023-10-19T14:21:20.710 INFO:teuthology.orchestra.run.smithi057.stdout:/dev 2023-10-19T14:21:20.710 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/security 2023-10-19T14:21:20.710 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/shm 2023-10-19T14:21:20.710 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/pts 2023-10-19T14:21:20.710 INFO:teuthology.orchestra.run.smithi057.stdout:/run 2023-10-19T14:21:20.711 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/cgroup 2023-10-19T14:21:20.711 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/pstore 2023-10-19T14:21:20.711 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/bpf 2023-10-19T14:21:20.711 INFO:teuthology.orchestra.run.smithi057.stdout:/ 2023-10-19T14:21:20.711 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/selinux 2023-10-19T14:21:20.712 INFO:teuthology.orchestra.run.smithi057.stdout:/proc/sys/fs/binfmt_misc 2023-10-19T14:21:20.712 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/hugepages 2023-10-19T14:21:20.712 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/mqueue 2023-10-19T14:21:20.712 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/debug 2023-10-19T14:21:20.712 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/tracing 2023-10-19T14:21:20.713 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/fuse/connections 2023-10-19T14:21:20.713 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/config 2023-10-19T14:21:20.713 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-sysctl.service 2023-10-19T14:21:20.713 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-10-19T14:21:20.713 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-10-19T14:21:20.714 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/nfs/rpc_pipefs 2023-10-19T14:21:20.714 INFO:teuthology.orchestra.run.smithi057.stdout:/run/user/1000 2023-10-19T14:21:20.714 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph 2023-10-19T14:21:20.714 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-0 2023-10-19T14:21:20.714 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-1 2023-10-19T14:21:20.714 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-2 2023-10-19T14:21:20.715 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-3 2023-10-19T14:21:20.715 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns 2023-10-19T14:21:20.715 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T14:21:20.715 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T14:21:20.716 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-19T14:21:20.716 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-19T14:21:20.716 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:21:20.716 DEBUG:teuthology.orchestra.run.smithi057:> ls /sys/fs/fuse/connections 2023-10-19T14:21:20.718 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi057.stderr:2023-10-19T14:21:20.713+0000 7f89cc04c480 -1 init, newargv = 0x55a027ff1ca0 newargc=15 2023-10-19T14:21:20.718 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi057.stderr:ceph-fuse[158769]: starting ceph client 2023-10-19T14:21:20.720 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi057.stderr:ceph-fuse[158769]: starting fuse 2023-10-19T14:21:20.733 INFO:teuthology.orchestra.run.smithi057.stdout:43 2023-10-19T14:21:20.733 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2023-10-19T14:21:20.733 DEBUG:teuthology.orchestra.run.smithi057:> sudo stdin-killer -- python3 -c ' 2023-10-19T14:21:20.734 DEBUG:teuthology.orchestra.run.smithi057:> import glob 2023-10-19T14:21:20.734 DEBUG:teuthology.orchestra.run.smithi057:> import re 2023-10-19T14:21:20.735 DEBUG:teuthology.orchestra.run.smithi057:> import os 2023-10-19T14:21:20.735 DEBUG:teuthology.orchestra.run.smithi057:> import subprocess 2023-10-19T14:21:20.735 DEBUG:teuthology.orchestra.run.smithi057:> 2023-10-19T14:21:20.736 DEBUG:teuthology.orchestra.run.smithi057:> def _find_admin_socket(client_name): 2023-10-19T14:21:20.736 DEBUG:teuthology.orchestra.run.smithi057:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-10-19T14:21:20.736 DEBUG:teuthology.orchestra.run.smithi057:> files = glob.glob(asok_path) 2023-10-19T14:21:20.737 DEBUG:teuthology.orchestra.run.smithi057:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-10-19T14:21:20.737 DEBUG:teuthology.orchestra.run.smithi057:> 2023-10-19T14:21:20.738 DEBUG:teuthology.orchestra.run.smithi057:> # Given a non-glob path, it better be there 2023-10-19T14:21:20.738 DEBUG:teuthology.orchestra.run.smithi057:> if "*" not in asok_path: 2023-10-19T14:21:20.738 DEBUG:teuthology.orchestra.run.smithi057:> assert(len(files) == 1) 2023-10-19T14:21:20.739 DEBUG:teuthology.orchestra.run.smithi057:> return files[0] 2023-10-19T14:21:20.739 DEBUG:teuthology.orchestra.run.smithi057:> 2023-10-19T14:21:20.739 DEBUG:teuthology.orchestra.run.smithi057:> for f in files: 2023-10-19T14:21:20.740 DEBUG:teuthology.orchestra.run.smithi057:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-10-19T14:21:20.740 DEBUG:teuthology.orchestra.run.smithi057:> if os.path.exists("/proc/{0}".format(pid)): 2023-10-19T14:21:20.741 DEBUG:teuthology.orchestra.run.smithi057:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-10-19T14:21:20.741 DEBUG:teuthology.orchestra.run.smithi057:> contents = proc_f.read() 2023-10-19T14:21:20.741 DEBUG:teuthology.orchestra.run.smithi057:> if mountpoint in contents: 2023-10-19T14:21:20.742 DEBUG:teuthology.orchestra.run.smithi057:> return f 2023-10-19T14:21:20.742 DEBUG:teuthology.orchestra.run.smithi057:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-10-19T14:21:20.742 DEBUG:teuthology.orchestra.run.smithi057:> 2023-10-19T14:21:20.743 DEBUG:teuthology.orchestra.run.smithi057:> print(_find_admin_socket("client.0")) 2023-10-19T14:21:20.743 DEBUG:teuthology.orchestra.run.smithi057:> ' 2023-10-19T14:21:20.857 INFO:teuthology.orchestra.run.smithi057.stdout:/var/run/ceph/ceph-client.0.158769.asok 2023-10-19T14:21:20.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2023-10-19T14:21:20.867 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.158769.asok 2023-10-19T14:21:20.869 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:21:20.870 DEBUG:teuthology.orchestra.run.smithi057:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.158769.asok status 2023-10-19T14:21:21.009 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2023-10-19T14:21:21.009 INFO:teuthology.orchestra.run.smithi057.stdout: "metadata": { 2023-10-19T14:21:21.009 INFO:teuthology.orchestra.run.smithi057.stdout: "ceph_sha1": "f7420d7f8e55ceea786dcbf536c338ce920c21eb", 2023-10-19T14:21:21.010 INFO:teuthology.orchestra.run.smithi057.stdout: "ceph_version": "ceph version 18.0.0-6786-gf7420d7f (f7420d7f8e55ceea786dcbf536c338ce920c21eb) reef (dev)", 2023-10-19T14:21:21.010 INFO:teuthology.orchestra.run.smithi057.stdout: "entity_id": "0", 2023-10-19T14:21:21.010 INFO:teuthology.orchestra.run.smithi057.stdout: "hostname": "smithi057", 2023-10-19T14:21:21.010 INFO:teuthology.orchestra.run.smithi057.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-10-19T14:21:21.010 INFO:teuthology.orchestra.run.smithi057.stdout: "pid": "158769", 2023-10-19T14:21:21.011 INFO:teuthology.orchestra.run.smithi057.stdout: "root": "/" 2023-10-19T14:21:21.011 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:21:21.011 INFO:teuthology.orchestra.run.smithi057.stdout: "dentry_count": 0, 2023-10-19T14:21:21.011 INFO:teuthology.orchestra.run.smithi057.stdout: "dentry_pinned_count": 0, 2023-10-19T14:21:21.011 INFO:teuthology.orchestra.run.smithi057.stdout: "id": 12741, 2023-10-19T14:21:21.012 INFO:teuthology.orchestra.run.smithi057.stdout: "inst": { 2023-10-19T14:21:21.012 INFO:teuthology.orchestra.run.smithi057.stdout: "name": { 2023-10-19T14:21:21.012 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "client", 2023-10-19T14:21:21.012 INFO:teuthology.orchestra.run.smithi057.stdout: "num": 12741 2023-10-19T14:21:21.012 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:21:21.013 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": { 2023-10-19T14:21:21.013 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "v1", 2023-10-19T14:21:21.013 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": "192.168.0.1:0", 2023-10-19T14:21:21.013 INFO:teuthology.orchestra.run.smithi057.stdout: "nonce": 1820585998 2023-10-19T14:21:21.013 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:21:21.013 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:21:21.014 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": { 2023-10-19T14:21:21.014 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "v1", 2023-10-19T14:21:21.014 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": "192.168.0.1:0", 2023-10-19T14:21:21.014 INFO:teuthology.orchestra.run.smithi057.stdout: "nonce": 1820585998 2023-10-19T14:21:21.015 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:21:21.015 INFO:teuthology.orchestra.run.smithi057.stdout: "inst_str": "client.12741 192.168.0.1:0/1820585998", 2023-10-19T14:21:21.015 INFO:teuthology.orchestra.run.smithi057.stdout: "addr_str": "192.168.0.1:0/1820585998", 2023-10-19T14:21:21.015 INFO:teuthology.orchestra.run.smithi057.stdout: "inode_count": 1, 2023-10-19T14:21:21.015 INFO:teuthology.orchestra.run.smithi057.stdout: "mds_epoch": 103, 2023-10-19T14:21:21.016 INFO:teuthology.orchestra.run.smithi057.stdout: "osd_epoch": 243, 2023-10-19T14:21:21.016 INFO:teuthology.orchestra.run.smithi057.stdout: "osd_epoch_barrier": 0, 2023-10-19T14:21:21.016 INFO:teuthology.orchestra.run.smithi057.stdout: "blocklisted": false, 2023-10-19T14:21:21.016 INFO:teuthology.orchestra.run.smithi057.stdout: "fs_name": "cephfs" 2023-10-19T14:21:21.016 INFO:teuthology.orchestra.run.smithi057.stdout:} 2023-10-19T14:21:21.018 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:21:21.018 DEBUG:teuthology.orchestra.run.smithi057:> stat --file-system '--printf=%T 2023-10-19T14:21:21.018 DEBUG:teuthology.orchestra.run.smithi057:> ' -- /home/ubuntu/cephtest/mnt.0 2023-10-19T14:21:21.034 INFO:teuthology.orchestra.run.smithi057.stdout:fuseblk 2023-10-19T14:21:21.034 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-10-19T14:21:21.035 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:21:21.035 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-19T14:21:21.105 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2023-10-19T14:21:21.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:21.255+0000 7f1705fa9640 1 -- 172.21.15.57:0/3950946581 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f170015b000 msgr2=0x7f17000d1810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:21.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:21.255+0000 7f1705fa9640 1 --2- 172.21.15.57:0/3950946581 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f170015b000 0x7f17000d1810 secure :-1 s=READY pgs=2659 cs=0 l=1 rev1=1 crypto rx=0x7f16f0005fd0 tx=0x7f16f002be10 comp rx=0 tx=0).stop 2023-10-19T14:21:21.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:21.255+0000 7f1705fa9640 1 -- 172.21.15.57:0/3950946581 shutdown_connections 2023-10-19T14:21:21.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:21.255+0000 7f1705fa9640 1 --2- 172.21.15.57:0/3950946581 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f17000cf9e0 0x7f17000b0c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:21.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:21.255+0000 7f1705fa9640 1 --2- 172.21.15.57:0/3950946581 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f170015b000 0x7f17000d1810 unknown :-1 s=CLOSED pgs=2659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:21.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:21.255+0000 7f1705fa9640 1 -- 172.21.15.57:0/3950946581 >> 172.21.15.57:0/3950946581 conn(0x7f17000d0cf0 msgr2=0x7f17000d1100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:21.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:21.255+0000 7f1705fa9640 1 -- 172.21.15.57:0/3950946581 shutdown_connections 2023-10-19T14:21:21.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:21.255+0000 7f1705fa9640 1 -- 172.21.15.57:0/3950946581 wait complete. 2023-10-19T14:21:21.262 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:21.257+0000 7f1705fa9640 1 Processor -- start 2023-10-19T14:21:21.262 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:21.257+0000 7f1705fa9640 1 -- start start 2023-10-19T14:21:21.262 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:21.257+0000 7f1705fa9640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f17000cf9e0 0x7f170014f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:21.263 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:21.257+0000 7f1705fa9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f170015b000 0x7f17000c8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:21.263 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:21.257+0000 7f1705fa9640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f170015b3e0 0x7f17000c9220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:21.263 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:21.257+0000 7f1705fa9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17000c4820 con 0x7f170015b000 2023-10-19T14:21:21.263 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:21.257+0000 7f1705fa9640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17000c5360 con 0x7f170015b3e0 2023-10-19T14:21:21.263 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:21.257+0000 7f1705fa9640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17000c4a30 con 0x7f17000cf9e0 2023-10-19T14:21:21.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:21.258+0000 7f16feffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f170015b000 0x7f17000c8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:21.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:21.258+0000 7f16feffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f170015b000 0x7f17000c8ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57542/0 (socket says 172.21.15.57:57542) 2023-10-19T14:21:21.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:21.258+0000 7f16feffd640 1 -- 172.21.15.57:0/1719402764 learned_addr learned my addr 172.21.15.57:0/1719402764 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:21.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:21.258+0000 7f16ff7fe640 1 --2- 172.21.15.57:0/1719402764 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f17000cf9e0 0x7f170014f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:21.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:21.258+0000 7f16fffff640 1 --2- 172.21.15.57:0/1719402764 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f170015b3e0 0x7f17000c9220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:21.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:21.258+0000 7f16feffd640 1 -- 172.21.15.57:0/1719402764 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f17000cf9e0 msgr2=0x7f170014f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:21.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:21.258+0000 7f16feffd640 1 --2- 172.21.15.57:0/1719402764 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f17000cf9e0 0x7f170014f200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:21.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:21.258+0000 7f16feffd640 1 -- 172.21.15.57:0/1719402764 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f170015b3e0 msgr2=0x7f17000c9220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:21.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:21.258+0000 7f16feffd640 1 --2- 172.21.15.57:0/1719402764 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f170015b3e0 0x7f17000c9220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:21.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:21.258+0000 7f16feffd640 1 -- 172.21.15.57:0/1719402764 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16f0009560 con 0x7f170015b000 2023-10-19T14:21:21.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:21.258+0000 7f16feffd640 1 --2- 172.21.15.57:0/1719402764 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f170015b000 0x7f17000c8ce0 secure :-1 s=READY pgs=2660 cs=0 l=1 rev1=1 crypto rx=0x7f16f400c400 tx=0x7f16f400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:21.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:21.258+0000 7f16e7fff640 1 -- 172.21.15.57:0/1719402764 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f40052d0 con 0x7f170015b000 2023-10-19T14:21:21.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:21.258+0000 7f1705fa9640 1 -- 172.21.15.57:0/1719402764 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17000c4d10 con 0x7f170015b000 2023-10-19T14:21:21.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:21.258+0000 7f16e7fff640 1 -- 172.21.15.57:0/1719402764 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f16f40035f0 con 0x7f170015b000 2023-10-19T14:21:21.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:21.258+0000 7f16e7fff640 1 -- 172.21.15.57:0/1719402764 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f4003a00 con 0x7f170015b000 2023-10-19T14:21:21.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:21.258+0000 7f1705fa9640 1 -- 172.21.15.57:0/1719402764 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17000c8080 con 0x7f170015b000 2023-10-19T14:21:21.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:21.259+0000 7f1705fa9640 1 -- 172.21.15.57:0/1719402764 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17000c54f0 con 0x7f170015b000 2023-10-19T14:21:21.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:21.260+0000 7f16e7fff640 1 -- 172.21.15.57:0/1719402764 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f16f4003b60 con 0x7f170015b000 2023-10-19T14:21:21.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:21.260+0000 7f16e7fff640 1 --2- 172.21.15.57:0/1719402764 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f16ec06b080 0x7f16ec06d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:21.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:21.260+0000 7f16e7fff640 1 -- 172.21.15.57:0/1719402764 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(243..243 src has 1..243) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f16f40893a0 con 0x7f170015b000 2023-10-19T14:21:21.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:21.261+0000 7f16ff7fe640 1 --2- 172.21.15.57:0/1719402764 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f16ec06b080 0x7f16ec06d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:21.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:21.261+0000 7f16ff7fe640 1 --2- 172.21.15.57:0/1719402764 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f16ec06b080 0x7f16ec06d540 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f16f0035000 tx=0x7f16f0004730 comp rx=0 tx=0).ready entity=mgr.9530 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:21.270 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:21.264+0000 7f16e7fff640 1 -- 172.21.15.57:0/1719402764 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f16f4053fe0 con 0x7f170015b000 2023-10-19T14:21:21.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:21.479+0000 7f1705fa9640 1 -- 172.21.15.57:0/1719402764 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f17000690e0 con 0x7f170015b000 2023-10-19T14:21:22.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.476+0000 7f16e7fff640 1 -- 172.21.15.57:0/1719402764 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v104) v1 ==== 136+0+0 (secure 0 0 0) 0x7f16f4057600 con 0x7f170015b000 2023-10-19T14:21:22.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.479+0000 7f1705fa9640 1 -- 172.21.15.57:0/1719402764 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f16ec06b080 msgr2=0x7f16ec06d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:22.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.479+0000 7f1705fa9640 1 --2- 172.21.15.57:0/1719402764 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f16ec06b080 0x7f16ec06d540 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f16f0035000 tx=0x7f16f0004730 comp rx=0 tx=0).stop 2023-10-19T14:21:22.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.479+0000 7f1705fa9640 1 -- 172.21.15.57:0/1719402764 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f170015b000 msgr2=0x7f17000c8ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:22.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.479+0000 7f1705fa9640 1 --2- 172.21.15.57:0/1719402764 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f170015b000 0x7f17000c8ce0 secure :-1 s=READY pgs=2660 cs=0 l=1 rev1=1 crypto rx=0x7f16f400c400 tx=0x7f16f400c8c0 comp rx=0 tx=0).stop 2023-10-19T14:21:22.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.479+0000 7f1705fa9640 1 -- 172.21.15.57:0/1719402764 shutdown_connections 2023-10-19T14:21:22.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.479+0000 7f1705fa9640 1 --2- 172.21.15.57:0/1719402764 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f16ec06b080 0x7f16ec06d540 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:22.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.479+0000 7f1705fa9640 1 --2- 172.21.15.57:0/1719402764 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f170015b3e0 0x7f17000c9220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:22.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.479+0000 7f1705fa9640 1 --2- 172.21.15.57:0/1719402764 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f170015b000 0x7f17000c8ce0 unknown :-1 s=CLOSED pgs=2660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:22.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.479+0000 7f1705fa9640 1 --2- 172.21.15.57:0/1719402764 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f17000cf9e0 0x7f170014f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:22.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.479+0000 7f1705fa9640 1 -- 172.21.15.57:0/1719402764 >> 172.21.15.57:0/1719402764 conn(0x7f17000d0cf0 msgr2=0x7f170006b670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:22.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.479+0000 7f1705fa9640 1 -- 172.21.15.57:0/1719402764 shutdown_connections 2023-10-19T14:21:22.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.479+0000 7f1705fa9640 1 -- 172.21.15.57:0/1719402764 wait complete. 2023-10-19T14:21:22.496 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T14:21:22.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.608+0000 7f0c51df0640 1 -- 172.21.15.57:0/2030364691 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0c4c0c99e0 msgr2=0x7f0c4c152c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:22.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.608+0000 7f0c51df0640 1 --2- 172.21.15.57:0/2030364691 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0c4c0c99e0 0x7f0c4c152c90 secure :-1 s=READY pgs=2662 cs=0 l=1 rev1=1 crypto rx=0x7f0c4c071370 tx=0x7f0c3c02be40 comp rx=0 tx=0).stop 2023-10-19T14:21:22.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.608+0000 7f0c51df0640 1 -- 172.21.15.57:0/2030364691 shutdown_connections 2023-10-19T14:21:22.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.608+0000 7f0c51df0640 1 --2- 172.21.15.57:0/2030364691 >> v2:172.21.15.179:3300/0 conn(0x7f0c4c0abc10 0x7f0c4c1531d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:22.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.608+0000 7f0c51df0640 1 --2- 172.21.15.57:0/2030364691 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0c4c0c99e0 0x7f0c4c152c90 unknown :-1 s=CLOSED pgs=2662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:22.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.608+0000 7f0c51df0640 1 -- 172.21.15.57:0/2030364691 >> 172.21.15.57:0/2030364691 conn(0x7f0c4c069930 msgr2=0x7f0c4c069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:22.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.608+0000 7f0c51df0640 1 -- 172.21.15.57:0/2030364691 shutdown_connections 2023-10-19T14:21:22.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.608+0000 7f0c51df0640 1 -- 172.21.15.57:0/2030364691 wait complete. 2023-10-19T14:21:22.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.609+0000 7f0c51df0640 1 Processor -- start 2023-10-19T14:21:22.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.609+0000 7f0c51df0640 1 -- start start 2023-10-19T14:21:22.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.610+0000 7f0c51df0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0c4c0abc10 0x7f0c4c149180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:22.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.610+0000 7f0c51df0640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0c4c0c99e0 0x7f0c4c156df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:22.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.610+0000 7f0c51df0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0c4c0c9dc0 0x7f0c4c157330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:22.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.610+0000 7f0c51df0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c4c156160 con 0x7f0c4c0abc10 2023-10-19T14:21:22.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.610+0000 7f0c51df0640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c4c0ab0f0 con 0x7f0c4c0c9dc0 2023-10-19T14:21:22.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.610+0000 7f0c51df0640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c4c0a6650 con 0x7f0c4c0c99e0 2023-10-19T14:21:22.618 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.610+0000 7f0c4b7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0c4c0abc10 0x7f0c4c149180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:22.618 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.610+0000 7f0c4b7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0c4c0abc10 0x7f0c4c149180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57562/0 (socket says 172.21.15.57:57562) 2023-10-19T14:21:22.618 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.610+0000 7f0c4b7fe640 1 -- 172.21.15.57:0/4209971742 learned_addr learned my addr 172.21.15.57:0/4209971742 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:22.618 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.610+0000 7f0c4affd640 1 --2- 172.21.15.57:0/4209971742 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0c4c0c99e0 0x7f0c4c156df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:22.618 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.610+0000 7f0c4bfff640 1 --2- 172.21.15.57:0/4209971742 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0c4c0c9dc0 0x7f0c4c157330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:22.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.611+0000 7f0c4b7fe640 1 -- 172.21.15.57:0/4209971742 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0c4c0c99e0 msgr2=0x7f0c4c156df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:22.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.611+0000 7f0c4b7fe640 1 --2- 172.21.15.57:0/4209971742 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0c4c0c99e0 0x7f0c4c156df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:22.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.611+0000 7f0c4b7fe640 1 -- 172.21.15.57:0/4209971742 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0c4c0c9dc0 msgr2=0x7f0c4c157330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:22.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.611+0000 7f0c4b7fe640 1 --2- 172.21.15.57:0/4209971742 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0c4c0c9dc0 0x7f0c4c157330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:22.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.611+0000 7f0c4b7fe640 1 -- 172.21.15.57:0/4209971742 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c3c009560 con 0x7f0c4c0abc10 2023-10-19T14:21:22.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.611+0000 7f0c4bfff640 1 --2- 172.21.15.57:0/4209971742 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0c4c0c9dc0 0x7f0c4c157330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:21:22.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.611+0000 7f0c4b7fe640 1 --2- 172.21.15.57:0/4209971742 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0c4c0abc10 0x7f0c4c149180 secure :-1 s=READY pgs=2663 cs=0 l=1 rev1=1 crypto rx=0x7f0c3c009850 tx=0x7f0c3c02de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:22.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.611+0000 7f0c33fff640 1 -- 172.21.15.57:0/4209971742 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c3c005650 con 0x7f0c4c0abc10 2023-10-19T14:21:22.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.611+0000 7f0c33fff640 1 -- 172.21.15.57:0/4209971742 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f0c3c0038f0 con 0x7f0c4c0abc10 2023-10-19T14:21:22.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.611+0000 7f0c51df0640 1 -- 172.21.15.57:0/4209971742 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c4c0a6930 con 0x7f0c4c0abc10 2023-10-19T14:21:22.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.611+0000 7f0c33fff640 1 -- 172.21.15.57:0/4209971742 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c3c003ce0 con 0x7f0c4c0abc10 2023-10-19T14:21:22.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.611+0000 7f0c51df0640 1 -- 172.21.15.57:0/4209971742 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c4c146900 con 0x7f0c4c0abc10 2023-10-19T14:21:22.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.612+0000 7f0c51df0640 1 -- 172.21.15.57:0/4209971742 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c4c0a64b0 con 0x7f0c4c0abc10 2023-10-19T14:21:22.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.612+0000 7f0c33fff640 1 -- 172.21.15.57:0/4209971742 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f0c3c03e450 con 0x7f0c4c0abc10 2023-10-19T14:21:22.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.613+0000 7f0c33fff640 1 --2- 172.21.15.57:0/4209971742 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f0c3806b080 0x7f0c3806d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:22.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.613+0000 7f0c33fff640 1 -- 172.21.15.57:0/4209971742 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(243..243 src has 1..243) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f0c3c0ac5f0 con 0x7f0c4c0abc10 2023-10-19T14:21:22.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.613+0000 7f0c4affd640 1 --2- 172.21.15.57:0/4209971742 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f0c3806b080 0x7f0c3806d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:22.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.614+0000 7f0c4affd640 1 --2- 172.21.15.57:0/4209971742 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f0c3806b080 0x7f0c3806d540 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f0c4c0b2130 tx=0x7f0c40009040 comp rx=0 tx=0).ready entity=mgr.9530 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:22.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.617+0000 7f0c33fff640 1 -- 172.21.15.57:0/4209971742 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0c4c0a64b0 con 0x7f0c4c0abc10 2023-10-19T14:21:22.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.793+0000 7f0c51df0640 1 -- 172.21.15.57:0/4209971742 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0c4c0a64b0 con 0x7f0c4c0abc10 2023-10-19T14:21:22.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.794+0000 7f0c33fff640 1 -- 172.21.15.57:0/4209971742 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v243) v1 ==== 74+0+16839 (secure 0 0 0) 0x7f0c4c0a64b0 con 0x7f0c4c0abc10 2023-10-19T14:21:22.797 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:21:22.797 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":243,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T14:21:11.406848+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":32,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2023-10-19T14:21:08.101023+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"242","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2023-10-19T14:21:08.745817+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"243","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":241,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":241,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":241,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":241,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":241,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":241,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":241,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":241,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T14:21:22.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.796+0000 7f0c51df0640 1 -- 172.21.15.57:0/4209971742 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f0c3806b080 msgr2=0x7f0c3806d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:22.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.796+0000 7f0c51df0640 1 --2- 172.21.15.57:0/4209971742 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f0c3806b080 0x7f0c3806d540 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f0c4c0b2130 tx=0x7f0c40009040 comp rx=0 tx=0).stop 2023-10-19T14:21:22.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.796+0000 7f0c51df0640 1 -- 172.21.15.57:0/4209971742 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0c4c0abc10 msgr2=0x7f0c4c149180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:22.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.796+0000 7f0c51df0640 1 --2- 172.21.15.57:0/4209971742 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0c4c0abc10 0x7f0c4c149180 secure :-1 s=READY pgs=2663 cs=0 l=1 rev1=1 crypto rx=0x7f0c3c009850 tx=0x7f0c3c02de10 comp rx=0 tx=0).stop 2023-10-19T14:21:22.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.796+0000 7f0c51df0640 1 -- 172.21.15.57:0/4209971742 shutdown_connections 2023-10-19T14:21:22.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.796+0000 7f0c51df0640 1 --2- 172.21.15.57:0/4209971742 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f0c3806b080 0x7f0c3806d540 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:22.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.796+0000 7f0c51df0640 1 --2- 172.21.15.57:0/4209971742 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0c4c0c9dc0 0x7f0c4c157330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:22.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.796+0000 7f0c51df0640 1 --2- 172.21.15.57:0/4209971742 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0c4c0c99e0 0x7f0c4c156df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:22.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.796+0000 7f0c51df0640 1 --2- 172.21.15.57:0/4209971742 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0c4c0abc10 0x7f0c4c149180 unknown :-1 s=CLOSED pgs=2663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:22.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.796+0000 7f0c51df0640 1 -- 172.21.15.57:0/4209971742 >> 172.21.15.57:0/4209971742 conn(0x7f0c4c069930 msgr2=0x7f0c4c0aee70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:22.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.796+0000 7f0c51df0640 1 -- 172.21.15.57:0/4209971742 shutdown_connections 2023-10-19T14:21:22.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.796+0000 7f0c51df0640 1 -- 172.21.15.57:0/4209971742 wait complete. 2023-10-19T14:21:22.809 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-19T12:44:07.371186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 31, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-19T14:21:08.101023+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '242', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 32, 'pool_name': 'cephfs_data', 'create_time': '2023-10-19T14:21:08.745817+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '243', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-19T14:21:22.810 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-19T14:21:22.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.920+0000 7f70c74b4640 1 -- 172.21.15.57:0/254625660 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f70ac009d00 msgr2=0x7f70ac00c170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:22.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.920+0000 7f70c74b4640 1 --2- 172.21.15.57:0/254625660 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f70ac009d00 0x7f70ac00c170 secure :-1 s=READY pgs=2665 cs=0 l=1 rev1=1 crypto rx=0x7f70b8005fd0 tx=0x7f70ac0037d0 comp rx=0 tx=0).stop 2023-10-19T14:21:22.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.920+0000 7f70c5a51640 1 -- 172.21.15.57:0/254625660 reap_dead start 2023-10-19T14:21:22.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.920+0000 7f70c74b4640 1 -- 172.21.15.57:0/254625660 shutdown_connections 2023-10-19T14:21:22.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.920+0000 7f70c74b4640 1 -- 172.21.15.57:0/254625660 >> 172.21.15.57:0/254625660 conn(0x7f70c0069930 msgr2=0x7f70c0069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:22.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.920+0000 7f70c74b4640 1 -- 172.21.15.57:0/254625660 shutdown_connections 2023-10-19T14:21:22.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.920+0000 7f70c74b4640 1 -- 172.21.15.57:0/254625660 wait complete. 2023-10-19T14:21:22.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.921+0000 7f70c74b4640 1 Processor -- start 2023-10-19T14:21:22.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.921+0000 7f70c74b4640 1 -- start start 2023-10-19T14:21:22.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.922+0000 7f70c74b4640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f70c01f0280 0x7f70c01391d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:22.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.922+0000 7f70c74b4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f70c0139710 0x7f70c0157590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:22.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.922+0000 7f70c74b4640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f70c0157ad0 0x7f70c0159690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:22.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.922+0000 7f70c74b4640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70c01581f0 con 0x7f70c0139710 2023-10-19T14:21:22.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.922+0000 7f70c74b4640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70c013a910 con 0x7f70c0157ad0 2023-10-19T14:21:22.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.922+0000 7f70c74b4640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70c0157f80 con 0x7f70c01f0280 2023-10-19T14:21:22.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.922+0000 7f70c4a4f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f70c0139710 0x7f70c0157590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:22.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.922+0000 7f70c4a4f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f70c0139710 0x7f70c0157590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57590/0 (socket says 172.21.15.57:57590) 2023-10-19T14:21:22.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.922+0000 7f70c4a4f640 1 -- 172.21.15.57:0/2807555192 learned_addr learned my addr 172.21.15.57:0/2807555192 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:22.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.922+0000 7f70c5250640 1 --2- 172.21.15.57:0/2807555192 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f70c01f0280 0x7f70c01391d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:22.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.922+0000 7f70c4a4f640 1 -- 172.21.15.57:0/2807555192 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f70c01f0280 msgr2=0x7f70c01391d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:22.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.922+0000 7f70c4a4f640 1 --2- 172.21.15.57:0/2807555192 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f70c01f0280 0x7f70c01391d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:22.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.922+0000 7f70c4a4f640 1 -- 172.21.15.57:0/2807555192 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f70c0157ad0 msgr2=0x7f70c0159690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:22.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.922+0000 7f70c5a51640 1 --2- 172.21.15.57:0/2807555192 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f70c0157ad0 0x7f70c0159690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:22.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.922+0000 7f70c4a4f640 1 --2- 172.21.15.57:0/2807555192 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f70c0157ad0 0x7f70c0159690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:22.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.922+0000 7f70c4a4f640 1 -- 172.21.15.57:0/2807555192 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70c0139c90 con 0x7f70c0139710 2023-10-19T14:21:22.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.922+0000 7f70c4a4f640 1 --2- 172.21.15.57:0/2807555192 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f70c0139710 0x7f70c0157590 secure :-1 s=READY pgs=2666 cs=0 l=1 rev1=1 crypto rx=0x7f70b0000fb0 tx=0x7f70b0003aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:22.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.922+0000 7f70b5ffb640 1 -- 172.21.15.57:0/2807555192 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70b000f920 con 0x7f70c0139710 2023-10-19T14:21:22.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.922+0000 7f70b5ffb640 1 -- 172.21.15.57:0/2807555192 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f70b000faa0 con 0x7f70c0139710 2023-10-19T14:21:22.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.922+0000 7f70c74b4640 1 -- 172.21.15.57:0/2807555192 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70c0139f70 con 0x7f70c0139710 2023-10-19T14:21:22.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.922+0000 7f70b5ffb640 1 -- 172.21.15.57:0/2807555192 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70b000feb0 con 0x7f70c0139710 2023-10-19T14:21:22.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.923+0000 7f70c74b4640 1 -- 172.21.15.57:0/2807555192 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70c013a530 con 0x7f70c0139710 2023-10-19T14:21:22.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.924+0000 7f70c74b4640 1 -- 172.21.15.57:0/2807555192 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70c013aa50 con 0x7f70c0139710 2023-10-19T14:21:22.932 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.924+0000 7f70b5ffb640 1 -- 172.21.15.57:0/2807555192 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f70b0011430 con 0x7f70c0139710 2023-10-19T14:21:22.932 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.925+0000 7f70b5ffb640 1 --2- 172.21.15.57:0/2807555192 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f70a406b0b0 0x7f70a406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:22.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.925+0000 7f70b5ffb640 1 -- 172.21.15.57:0/2807555192 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(243..243 src has 1..243) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f70b008aab0 con 0x7f70c0139710 2023-10-19T14:21:22.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.925+0000 7f70c5250640 1 --2- 172.21.15.57:0/2807555192 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f70a406b0b0 0x7f70a406d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:22.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.925+0000 7f70c5250640 1 --2- 172.21.15.57:0/2807555192 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f70a406b0b0 0x7f70a406d570 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f70c013e8c0 tx=0x7f70bc005f90 comp rx=0 tx=0).ready entity=mgr.9530 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:22.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:22.929+0000 7f70b5ffb640 1 -- 172.21.15.57:0/2807555192 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f70c013aa50 con 0x7f70c0139710 2023-10-19T14:21:23.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.105+0000 7f70c74b4640 1 -- 172.21.15.57:0/2807555192 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f70c013aa50 con 0x7f70c0139710 2023-10-19T14:21:23.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.105+0000 7f70b5ffb640 1 -- 172.21.15.57:0/2807555192 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v243) v1 ==== 93+0+10 (secure 0 0 0) 0x7f70b0015020 con 0x7f70c0139710 2023-10-19T14:21:23.109 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-19T14:21:23.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.107+0000 7f70c74b4640 1 -- 172.21.15.57:0/2807555192 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f70a406b0b0 msgr2=0x7f70a406d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:23.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.107+0000 7f70c74b4640 1 --2- 172.21.15.57:0/2807555192 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f70a406b0b0 0x7f70a406d570 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f70c013e8c0 tx=0x7f70bc005f90 comp rx=0 tx=0).stop 2023-10-19T14:21:23.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.107+0000 7f70c74b4640 1 -- 172.21.15.57:0/2807555192 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f70c0139710 msgr2=0x7f70c0157590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:23.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.107+0000 7f70c74b4640 1 --2- 172.21.15.57:0/2807555192 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f70c0139710 0x7f70c0157590 secure :-1 s=READY pgs=2666 cs=0 l=1 rev1=1 crypto rx=0x7f70b0000fb0 tx=0x7f70b0003aa0 comp rx=0 tx=0).stop 2023-10-19T14:21:23.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.107+0000 7f70c74b4640 1 -- 172.21.15.57:0/2807555192 shutdown_connections 2023-10-19T14:21:23.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.107+0000 7f70c74b4640 1 --2- 172.21.15.57:0/2807555192 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f70a406b0b0 0x7f70a406d570 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:23.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.107+0000 7f70c74b4640 1 --2- 172.21.15.57:0/2807555192 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f70c0157ad0 0x7f70c0159690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:23.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.107+0000 7f70c74b4640 1 --2- 172.21.15.57:0/2807555192 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f70c0139710 0x7f70c0157590 unknown :-1 s=CLOSED pgs=2666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:23.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.107+0000 7f70c74b4640 1 --2- 172.21.15.57:0/2807555192 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f70c01f0280 0x7f70c01391d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:23.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.107+0000 7f70c74b4640 1 -- 172.21.15.57:0/2807555192 >> 172.21.15.57:0/2807555192 conn(0x7f70c0069930 msgr2=0x7f70c00b01d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:23.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.108+0000 7f70c74b4640 1 -- 172.21.15.57:0/2807555192 shutdown_connections 2023-10-19T14:21:23.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.108+0000 7f70c74b4640 1 -- 172.21.15.57:0/2807555192 wait complete. 2023-10-19T14:21:23.121 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-19T14:21:23.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.230+0000 7f181905d640 1 -- 172.21.15.57:0/3481687694 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1800003ef0 msgr2=0x7f18000047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:23.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.230+0000 7f181905d640 1 --2- 172.21.15.57:0/3481687694 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1800003ef0 0x7f18000047c0 secure :-1 s=READY pgs=2668 cs=0 l=1 rev1=1 crypto rx=0x7f17fc009b50 tx=0x7f18000057f0 comp rx=0 tx=0).stop 2023-10-19T14:21:23.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.230+0000 7f181359e640 1 -- 172.21.15.57:0/3481687694 reap_dead start 2023-10-19T14:21:23.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.230+0000 7f181905d640 1 -- 172.21.15.57:0/3481687694 shutdown_connections 2023-10-19T14:21:23.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.230+0000 7f181905d640 1 -- 172.21.15.57:0/3481687694 >> 172.21.15.57:0/3481687694 conn(0x7f1814069930 msgr2=0x7f1814069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:23.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.230+0000 7f181905d640 1 -- 172.21.15.57:0/3481687694 shutdown_connections 2023-10-19T14:21:23.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.230+0000 7f181905d640 1 -- 172.21.15.57:0/3481687694 wait complete. 2023-10-19T14:21:23.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.232+0000 7f181905d640 1 Processor -- start 2023-10-19T14:21:23.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.232+0000 7f181905d640 1 -- start start 2023-10-19T14:21:23.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.233+0000 7f181905d640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1800003ef0 0x7f18141e14d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:23.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.233+0000 7f181905d640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f18140ae6d0 0x7f181406f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:23.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.233+0000 7f181905d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f181413ada0 0x7f181413b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:23.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.233+0000 7f181905d640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18140ae2b0 con 0x7f181413ada0 2023-10-19T14:21:23.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.233+0000 7f181905d640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f181413b760 con 0x7f18140ae6d0 2023-10-19T14:21:23.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.233+0000 7f181905d640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18141e1a10 con 0x7f1800003ef0 2023-10-19T14:21:23.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.233+0000 7f181359e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f181413ada0 0x7f181413b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:23.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.233+0000 7f181359e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f181413ada0 0x7f181413b220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57612/0 (socket says 172.21.15.57:57612) 2023-10-19T14:21:23.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.233+0000 7f181359e640 1 -- 172.21.15.57:0/3736811363 learned_addr learned my addr 172.21.15.57:0/3736811363 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:23.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.233+0000 7f181259c640 1 --2- 172.21.15.57:0/3736811363 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f18140ae6d0 0x7f181406f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:23.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.233+0000 7f1812d9d640 1 --2- 172.21.15.57:0/3736811363 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1800003ef0 0x7f18141e14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:23.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.233+0000 7f181359e640 1 -- 172.21.15.57:0/3736811363 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1800003ef0 msgr2=0x7f18141e14d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:23.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.233+0000 7f181359e640 1 --2- 172.21.15.57:0/3736811363 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1800003ef0 0x7f18141e14d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:23.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.233+0000 7f181359e640 1 -- 172.21.15.57:0/3736811363 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f18140ae6d0 msgr2=0x7f181406f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:23.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.233+0000 7f181359e640 1 --2- 172.21.15.57:0/3736811363 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f18140ae6d0 0x7f181406f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:23.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.233+0000 7f181359e640 1 -- 172.21.15.57:0/3736811363 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18140ac2c0 con 0x7f181413ada0 2023-10-19T14:21:23.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.233+0000 7f181359e640 1 --2- 172.21.15.57:0/3736811363 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f181413ada0 0x7f181413b220 secure :-1 s=READY pgs=2669 cs=0 l=1 rev1=1 crypto rx=0x7f181406b5f0 tx=0x7f180400ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:23.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.233+0000 7f180b7fe640 1 -- 172.21.15.57:0/3736811363 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f180400c9a0 con 0x7f181413ada0 2023-10-19T14:21:23.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.233+0000 7f180b7fe640 1 -- 172.21.15.57:0/3736811363 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f180400cbf0 con 0x7f181413ada0 2023-10-19T14:21:23.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.233+0000 7f181905d640 1 -- 172.21.15.57:0/3736811363 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18140abfe0 con 0x7f181413ada0 2023-10-19T14:21:23.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.233+0000 7f180b7fe640 1 -- 172.21.15.57:0/3736811363 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1804011460 con 0x7f181413ada0 2023-10-19T14:21:23.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.234+0000 7f181905d640 1 -- 172.21.15.57:0/3736811363 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18140aded0 con 0x7f181413ada0 2023-10-19T14:21:23.242 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.235+0000 7f181905d640 1 -- 172.21.15.57:0/3736811363 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18140ae3f0 con 0x7f181413ada0 2023-10-19T14:21:23.243 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.235+0000 7f180b7fe640 1 -- 172.21.15.57:0/3736811363 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f18040115c0 con 0x7f181413ada0 2023-10-19T14:21:23.243 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.236+0000 7f180b7fe640 1 --2- 172.21.15.57:0/3736811363 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f17f406b080 0x7f17f406d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:23.243 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.236+0000 7f180b7fe640 1 -- 172.21.15.57:0/3736811363 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(243..243 src has 1..243) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f180408a520 con 0x7f181413ada0 2023-10-19T14:21:23.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.236+0000 7f1812d9d640 1 --2- 172.21.15.57:0/3736811363 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f17f406b080 0x7f17f406d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:23.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.236+0000 7f1812d9d640 1 --2- 172.21.15.57:0/3736811363 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f17f406b080 0x7f17f406d540 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f180c0099b0 tx=0x7f180c005f90 comp rx=0 tx=0).ready entity=mgr.9530 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:23.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.240+0000 7f180b7fe640 1 -- 172.21.15.57:0/3736811363 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f18140ae3f0 con 0x7f181413ada0 2023-10-19T14:21:23.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.416+0000 7f181905d640 1 -- 172.21.15.57:0/3736811363 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f18140ae3f0 con 0x7f181413ada0 2023-10-19T14:21:23.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.416+0000 7f180b7fe640 1 -- 172.21.15.57:0/3736811363 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v243) v1 ==== 104+0+11 (secure 0 0 0) 0x7f18140ae3f0 con 0x7f181413ada0 2023-10-19T14:21:23.420 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2023-10-19T14:21:23.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.418+0000 7f181905d640 1 -- 172.21.15.57:0/3736811363 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f17f406b080 msgr2=0x7f17f406d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:23.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.418+0000 7f181905d640 1 --2- 172.21.15.57:0/3736811363 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f17f406b080 0x7f17f406d540 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f180c0099b0 tx=0x7f180c005f90 comp rx=0 tx=0).stop 2023-10-19T14:21:23.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.418+0000 7f181905d640 1 -- 172.21.15.57:0/3736811363 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f181413ada0 msgr2=0x7f181413b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:23.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.418+0000 7f181905d640 1 --2- 172.21.15.57:0/3736811363 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f181413ada0 0x7f181413b220 secure :-1 s=READY pgs=2669 cs=0 l=1 rev1=1 crypto rx=0x7f181406b5f0 tx=0x7f180400ed10 comp rx=0 tx=0).stop 2023-10-19T14:21:23.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.418+0000 7f181905d640 1 -- 172.21.15.57:0/3736811363 shutdown_connections 2023-10-19T14:21:23.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.418+0000 7f181905d640 1 --2- 172.21.15.57:0/3736811363 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f17f406b080 0x7f17f406d540 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:23.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.418+0000 7f181905d640 1 --2- 172.21.15.57:0/3736811363 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f181413ada0 0x7f181413b220 unknown :-1 s=CLOSED pgs=2669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:23.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.418+0000 7f181905d640 1 --2- 172.21.15.57:0/3736811363 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f18140ae6d0 0x7f181406f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:23.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.418+0000 7f181905d640 1 --2- 172.21.15.57:0/3736811363 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1800003ef0 0x7f18141e14d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:23.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.418+0000 7f181905d640 1 -- 172.21.15.57:0/3736811363 >> 172.21.15.57:0/3736811363 conn(0x7f1814069930 msgr2=0x7f18141e0b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:23.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.418+0000 7f181905d640 1 -- 172.21.15.57:0/3736811363 shutdown_connections 2023-10-19T14:21:23.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.418+0000 7f181905d640 1 -- 172.21.15.57:0/3736811363 wait complete. 2023-10-19T14:21:23.432 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-19T14:21:23.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.543+0000 7f09757b0640 1 -- 172.21.15.57:0/3406868439 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f095c004d00 msgr2=0x7f095c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:23.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.543+0000 7f09757b0640 1 --2- 172.21.15.57:0/3406868439 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f095c004d00 0x7f095c005180 secure :-1 s=READY pgs=2671 cs=0 l=1 rev1=1 crypto rx=0x7f0968009e20 tx=0x7f095c0057f0 comp rx=0 tx=0).stop 2023-10-19T14:21:23.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.543+0000 7f096f7fe640 1 -- 172.21.15.57:0/3406868439 reap_dead start 2023-10-19T14:21:23.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.543+0000 7f09757b0640 1 -- 172.21.15.57:0/3406868439 shutdown_connections 2023-10-19T14:21:23.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.543+0000 7f09757b0640 1 -- 172.21.15.57:0/3406868439 >> 172.21.15.57:0/3406868439 conn(0x7f097006fae0 msgr2=0x7f097006fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:23.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.543+0000 7f09757b0640 1 -- 172.21.15.57:0/3406868439 shutdown_connections 2023-10-19T14:21:23.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.543+0000 7f09757b0640 1 -- 172.21.15.57:0/3406868439 wait complete. 2023-10-19T14:21:23.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.544+0000 7f09757b0640 1 Processor -- start 2023-10-19T14:21:23.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.545+0000 7f09757b0640 1 -- start start 2023-10-19T14:21:23.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.545+0000 7f09757b0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f097014a710 0x7f0970152eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:23.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.545+0000 7f09757b0640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f097015c070 0x7f09701573d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:23.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.545+0000 7f09757b0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0970157910 0x7f09701f4410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:23.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.545+0000 7f09757b0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0970157d90 con 0x7f097014a710 2023-10-19T14:21:23.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.545+0000 7f09757b0640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f097014b1f0 con 0x7f0970157910 2023-10-19T14:21:23.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.545+0000 7f09757b0640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09700b08d0 con 0x7f097015c070 2023-10-19T14:21:23.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.545+0000 7f096effd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f097014a710 0x7f0970152eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:23.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.545+0000 7f096effd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f097014a710 0x7f0970152eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57642/0 (socket says 172.21.15.57:57642) 2023-10-19T14:21:23.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.545+0000 7f096effd640 1 -- 172.21.15.57:0/558064309 learned_addr learned my addr 172.21.15.57:0/558064309 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:23.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.546+0000 7f09667fc640 1 --2- 172.21.15.57:0/558064309 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f097015c070 0x7f09701573d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:23.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.546+0000 7f096f7fe640 1 --2- 172.21.15.57:0/558064309 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0970157910 0x7f09701f4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:23.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.546+0000 7f096effd640 1 -- 172.21.15.57:0/558064309 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f097015c070 msgr2=0x7f09701573d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:23.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.546+0000 7f096effd640 1 --2- 172.21.15.57:0/558064309 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f097015c070 0x7f09701573d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:23.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.546+0000 7f096effd640 1 -- 172.21.15.57:0/558064309 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0970157910 msgr2=0x7f09701f4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:23.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.546+0000 7f096effd640 1 --2- 172.21.15.57:0/558064309 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0970157910 0x7f09701f4410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:23.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.546+0000 7f096effd640 1 -- 172.21.15.57:0/558064309 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f097015c4b0 con 0x7f097014a710 2023-10-19T14:21:23.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.546+0000 7f096effd640 1 --2- 172.21.15.57:0/558064309 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f097014a710 0x7f0970152eb0 secure :-1 s=READY pgs=2672 cs=0 l=1 rev1=1 crypto rx=0x7f096000b630 tx=0x7f096000baf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:23.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.546+0000 7f0967fff640 1 -- 172.21.15.57:0/558064309 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0960010ab0 con 0x7f097014a710 2023-10-19T14:21:23.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.546+0000 7f0967fff640 1 -- 172.21.15.57:0/558064309 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f0960010c10 con 0x7f097014a710 2023-10-19T14:21:23.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.546+0000 7f09757b0640 1 -- 172.21.15.57:0/558064309 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09700b0580 con 0x7f097014a710 2023-10-19T14:21:23.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.546+0000 7f0967fff640 1 -- 172.21.15.57:0/558064309 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f096001a470 con 0x7f097014a710 2023-10-19T14:21:23.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.546+0000 7f09757b0640 1 -- 172.21.15.57:0/558064309 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f097015c9e0 con 0x7f097014a710 2023-10-19T14:21:23.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.547+0000 7f09757b0640 1 -- 172.21.15.57:0/558064309 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09700690e0 con 0x7f097014a710 2023-10-19T14:21:23.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.547+0000 7f0967fff640 1 -- 172.21.15.57:0/558064309 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f0960017070 con 0x7f097014a710 2023-10-19T14:21:23.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.548+0000 7f0967fff640 1 --2- 172.21.15.57:0/558064309 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f095006b080 0x7f095006d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:23.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.548+0000 7f0967fff640 1 -- 172.21.15.57:0/558064309 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(243..243 src has 1..243) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f096008a860 con 0x7f097014a710 2023-10-19T14:21:23.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.548+0000 7f09667fc640 1 --2- 172.21.15.57:0/558064309 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f095006b080 0x7f095006d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:23.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.549+0000 7f09667fc640 1 --2- 172.21.15.57:0/558064309 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f095006b080 0x7f095006d540 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f0958003670 tx=0x7f09580015a0 comp rx=0 tx=0).ready entity=mgr.9530 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:23.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.552+0000 7f0967fff640 1 -- 172.21.15.57:0/558064309 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0960058ef0 con 0x7f097014a710 2023-10-19T14:21:23.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.729+0000 7f09757b0640 1 -- 172.21.15.57:0/558064309 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f09700b0710 con 0x7f097014a710 2023-10-19T14:21:23.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.729+0000 7f0967fff640 1 -- 172.21.15.57:0/558064309 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v243) v1 ==== 100+0+11 (secure 0 0 0) 0x7f09700b0710 con 0x7f097014a710 2023-10-19T14:21:23.732 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 64 2023-10-19T14:21:23.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.731+0000 7f09757b0640 1 -- 172.21.15.57:0/558064309 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f095006b080 msgr2=0x7f095006d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:23.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.731+0000 7f09757b0640 1 --2- 172.21.15.57:0/558064309 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f095006b080 0x7f095006d540 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f0958003670 tx=0x7f09580015a0 comp rx=0 tx=0).stop 2023-10-19T14:21:23.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.731+0000 7f09757b0640 1 -- 172.21.15.57:0/558064309 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f097014a710 msgr2=0x7f0970152eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:23.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.731+0000 7f09757b0640 1 --2- 172.21.15.57:0/558064309 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f097014a710 0x7f0970152eb0 secure :-1 s=READY pgs=2672 cs=0 l=1 rev1=1 crypto rx=0x7f096000b630 tx=0x7f096000baf0 comp rx=0 tx=0).stop 2023-10-19T14:21:23.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.731+0000 7f09757b0640 1 -- 172.21.15.57:0/558064309 shutdown_connections 2023-10-19T14:21:23.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.731+0000 7f09757b0640 1 --2- 172.21.15.57:0/558064309 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f095006b080 0x7f095006d540 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:23.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.731+0000 7f09757b0640 1 --2- 172.21.15.57:0/558064309 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0970157910 0x7f09701f4410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:23.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.731+0000 7f09757b0640 1 --2- 172.21.15.57:0/558064309 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f097015c070 0x7f09701573d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:23.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.731+0000 7f09757b0640 1 --2- 172.21.15.57:0/558064309 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f097014a710 0x7f0970152eb0 unknown :-1 s=CLOSED pgs=2672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:23.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.731+0000 7f09757b0640 1 -- 172.21.15.57:0/558064309 >> 172.21.15.57:0/558064309 conn(0x7f097006fae0 msgr2=0x7f09700ae920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:23.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.731+0000 7f09757b0640 1 -- 172.21.15.57:0/558064309 shutdown_connections 2023-10-19T14:21:23.736 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.731+0000 7f09757b0640 1 -- 172.21.15.57:0/558064309 wait complete. 2023-10-19T14:21:23.744 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-19T14:21:23.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.851+0000 7f8d9c444640 1 -- 172.21.15.57:0/2719472010 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8d80004d00 msgr2=0x7f8d80005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:23.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.851+0000 7f8d9c444640 1 --2- 172.21.15.57:0/2719472010 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8d80004d00 0x7f8d80005180 secure :-1 s=READY pgs=2674 cs=0 l=1 rev1=1 crypto rx=0x7f8d90009f00 tx=0x7f8d800057f0 comp rx=0 tx=0).stop 2023-10-19T14:21:23.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.851+0000 7f8d9a9e1640 1 -- 172.21.15.57:0/2719472010 reap_dead start 2023-10-19T14:21:23.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.851+0000 7f8d9c444640 1 -- 172.21.15.57:0/2719472010 shutdown_connections 2023-10-19T14:21:23.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.851+0000 7f8d9c444640 1 -- 172.21.15.57:0/2719472010 >> 172.21.15.57:0/2719472010 conn(0x7f8d94069930 msgr2=0x7f8d94069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:23.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.851+0000 7f8d9c444640 1 -- 172.21.15.57:0/2719472010 shutdown_connections 2023-10-19T14:21:23.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.851+0000 7f8d9c444640 1 -- 172.21.15.57:0/2719472010 wait complete. 2023-10-19T14:21:23.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.852+0000 7f8d9c444640 1 Processor -- start 2023-10-19T14:21:23.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.852+0000 7f8d9c444640 1 -- start start 2023-10-19T14:21:23.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.853+0000 7f8d9c444640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8d80004d00 0x7f8d9413d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:23.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.853+0000 7f8d9c444640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8d94134600 0x7f8d94151cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:23.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.853+0000 7f8d9c444640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8d9415b3c0 0x7f8d9415b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:23.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.853+0000 7f8d9c444640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d9415bd80 con 0x7f8d9415b3c0 2023-10-19T14:21:23.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.853+0000 7f8d9c444640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d9413e960 con 0x7f8d94134600 2023-10-19T14:21:23.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.853+0000 7f8d9c444640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d9413dd50 con 0x7f8d80004d00 2023-10-19T14:21:23.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.853+0000 7f8d9a9e1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8d9415b3c0 0x7f8d9415b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:23.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.853+0000 7f8d9a9e1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8d9415b3c0 0x7f8d9415b840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57664/0 (socket says 172.21.15.57:57664) 2023-10-19T14:21:23.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.853+0000 7f8d9a9e1640 1 -- 172.21.15.57:0/3828518150 learned_addr learned my addr 172.21.15.57:0/3828518150 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:23.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.853+0000 7f8d9a1e0640 1 --2- 172.21.15.57:0/3828518150 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8d80004d00 0x7f8d9413d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:23.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.853+0000 7f8d999df640 1 --2- 172.21.15.57:0/3828518150 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8d94134600 0x7f8d94151cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:23.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.853+0000 7f8d9a9e1640 1 -- 172.21.15.57:0/3828518150 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8d80004d00 msgr2=0x7f8d9413d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:23.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.853+0000 7f8d9a9e1640 1 --2- 172.21.15.57:0/3828518150 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8d80004d00 0x7f8d9413d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:23.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.853+0000 7f8d9a9e1640 1 -- 172.21.15.57:0/3828518150 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8d94134600 msgr2=0x7f8d94151cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:23.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.853+0000 7f8d9a9e1640 1 --2- 172.21.15.57:0/3828518150 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8d94134600 0x7f8d94151cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:23.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.853+0000 7f8d9a9e1640 1 -- 172.21.15.57:0/3828518150 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d9413e030 con 0x7f8d9415b3c0 2023-10-19T14:21:23.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.853+0000 7f8d999df640 1 --2- 172.21.15.57:0/3828518150 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8d94134600 0x7f8d94151cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:21:23.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.853+0000 7f8d9a9e1640 1 --2- 172.21.15.57:0/3828518150 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8d9415b3c0 0x7f8d9415b840 secure :-1 s=READY pgs=2675 cs=0 l=1 rev1=1 crypto rx=0x7f8d90009f00 tx=0x7f8d90031760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:23.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.853+0000 7f8d8affd640 1 -- 172.21.15.57:0/3828518150 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d9003f070 con 0x7f8d9415b3c0 2023-10-19T14:21:23.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.854+0000 7f8d8affd640 1 -- 172.21.15.57:0/3828518150 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f8d90031c10 con 0x7f8d9415b3c0 2023-10-19T14:21:23.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.854+0000 7f8d9c444640 1 -- 172.21.15.57:0/3828518150 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d9413e310 con 0x7f8d9415b3c0 2023-10-19T14:21:23.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.854+0000 7f8d8affd640 1 -- 172.21.15.57:0/3828518150 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d90003970 con 0x7f8d9415b3c0 2023-10-19T14:21:23.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.854+0000 7f8d9c444640 1 -- 172.21.15.57:0/3828518150 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d941372d0 con 0x7f8d9415b3c0 2023-10-19T14:21:23.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.855+0000 7f8d9c444640 1 -- 172.21.15.57:0/3828518150 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d9413eaf0 con 0x7f8d9415b3c0 2023-10-19T14:21:23.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.855+0000 7f8d8affd640 1 -- 172.21.15.57:0/3828518150 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f8d90003ad0 con 0x7f8d9415b3c0 2023-10-19T14:21:23.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.855+0000 7f8d8affd640 1 --2- 172.21.15.57:0/3828518150 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f8d7806b0b0 0x7f8d7806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:23.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.856+0000 7f8d8affd640 1 -- 172.21.15.57:0/3828518150 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(243..243 src has 1..243) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f8d900ac430 con 0x7f8d9415b3c0 2023-10-19T14:21:23.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.856+0000 7f8d9a1e0640 1 --2- 172.21.15.57:0/3828518150 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f8d7806b0b0 0x7f8d7806d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:23.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.856+0000 7f8d9a1e0640 1 --2- 172.21.15.57:0/3828518150 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f8d7806b0b0 0x7f8d7806d570 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c000c00 tx=0x7f8d8c001e70 comp rx=0 tx=0).ready entity=mgr.9530 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:23.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:23.860+0000 7f8d8affd640 1 -- 172.21.15.57:0/3828518150 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8d90077120 con 0x7f8d9415b3c0 2023-10-19T14:21:24.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:24.038+0000 7f8d9c444640 1 -- 172.21.15.57:0/3828518150 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f8d9415bd80 con 0x7f8d9415b3c0 2023-10-19T14:21:24.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:24.038+0000 7f8d8affd640 1 -- 172.21.15.57:0/3828518150 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v243) v1 ==== 84+0+209 (secure 0 0 0) 0x7f8d9415bd80 con 0x7f8d9415b3c0 2023-10-19T14:21:24.042 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:21:24.042 INFO:teuthology.orchestra.run.smithi057.stdout:[ 2023-10-19T14:21:24.042 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:21:24.043 INFO:teuthology.orchestra.run.smithi057.stdout: "poolnum": 1, 2023-10-19T14:21:24.043 INFO:teuthology.orchestra.run.smithi057.stdout: "poolname": ".mgr" 2023-10-19T14:21:24.043 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:21:24.043 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:21:24.043 INFO:teuthology.orchestra.run.smithi057.stdout: "poolnum": 31, 2023-10-19T14:21:24.043 INFO:teuthology.orchestra.run.smithi057.stdout: "poolname": "cephfs_metadata" 2023-10-19T14:21:24.044 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:21:24.044 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:21:24.044 INFO:teuthology.orchestra.run.smithi057.stdout: "poolnum": 32, 2023-10-19T14:21:24.044 INFO:teuthology.orchestra.run.smithi057.stdout: "poolname": "cephfs_data" 2023-10-19T14:21:24.045 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:21:24.045 INFO:teuthology.orchestra.run.smithi057.stdout:] 2023-10-19T14:21:24.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:24.040+0000 7f8d9c444640 1 -- 172.21.15.57:0/3828518150 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f8d7806b0b0 msgr2=0x7f8d7806d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:24.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:24.040+0000 7f8d9c444640 1 --2- 172.21.15.57:0/3828518150 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f8d7806b0b0 0x7f8d7806d570 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c000c00 tx=0x7f8d8c001e70 comp rx=0 tx=0).stop 2023-10-19T14:21:24.046 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:24.040+0000 7f8d9c444640 1 -- 172.21.15.57:0/3828518150 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8d9415b3c0 msgr2=0x7f8d9415b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:24.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:24.040+0000 7f8d9c444640 1 --2- 172.21.15.57:0/3828518150 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8d9415b3c0 0x7f8d9415b840 secure :-1 s=READY pgs=2675 cs=0 l=1 rev1=1 crypto rx=0x7f8d90009f00 tx=0x7f8d90031760 comp rx=0 tx=0).stop 2023-10-19T14:21:24.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:24.040+0000 7f8d9c444640 1 -- 172.21.15.57:0/3828518150 shutdown_connections 2023-10-19T14:21:24.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:24.040+0000 7f8d9c444640 1 --2- 172.21.15.57:0/3828518150 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f8d7806b0b0 0x7f8d7806d570 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:24.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:24.040+0000 7f8d9c444640 1 --2- 172.21.15.57:0/3828518150 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8d9415b3c0 0x7f8d9415b840 unknown :-1 s=CLOSED pgs=2675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:24.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:24.040+0000 7f8d9c444640 1 --2- 172.21.15.57:0/3828518150 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8d94134600 0x7f8d94151cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:24.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:24.040+0000 7f8d9c444640 1 --2- 172.21.15.57:0/3828518150 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8d80004d00 0x7f8d9413d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:24.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:24.040+0000 7f8d9c444640 1 -- 172.21.15.57:0/3828518150 >> 172.21.15.57:0/3828518150 conn(0x7f8d94069930 msgr2=0x7f8d940b19d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:24.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:24.040+0000 7f8d9c444640 1 -- 172.21.15.57:0/3828518150 shutdown_connections 2023-10-19T14:21:24.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:24.040+0000 7f8d9c444640 1 -- 172.21.15.57:0/3828518150 wait complete. 2023-10-19T14:21:24.053 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'backup_fs' 2023-10-19T14:21:24.054 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create backup_fs_metadata --pg_num_min 64 2023-10-19T14:21:24.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:24.165+0000 7f67f03e9640 1 -- 172.21.15.57:0/1355369984 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f67d0004d00 msgr2=0x7f67d0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:24.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:24.165+0000 7f67f03e9640 1 --2- 172.21.15.57:0/1355369984 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f67d0004d00 0x7f67d0005180 secure :-1 s=READY pgs=2677 cs=0 l=1 rev1=1 crypto rx=0x7f67d8009b50 tx=0x7f67d00057f0 comp rx=0 tx=0).stop 2023-10-19T14:21:24.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:24.165+0000 7f67ee986640 1 -- 172.21.15.57:0/1355369984 reap_dead start 2023-10-19T14:21:24.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:24.165+0000 7f67f03e9640 1 -- 172.21.15.57:0/1355369984 shutdown_connections 2023-10-19T14:21:24.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:24.165+0000 7f67f03e9640 1 -- 172.21.15.57:0/1355369984 >> 172.21.15.57:0/1355369984 conn(0x7f67e80d0d60 msgr2=0x7f67e80d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:24.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:24.165+0000 7f67f03e9640 1 -- 172.21.15.57:0/1355369984 shutdown_connections 2023-10-19T14:21:24.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:24.165+0000 7f67f03e9640 1 -- 172.21.15.57:0/1355369984 wait complete. 2023-10-19T14:21:24.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:24.166+0000 7f67f03e9640 1 Processor -- start 2023-10-19T14:21:24.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:24.166+0000 7f67f03e9640 1 -- start start 2023-10-19T14:21:24.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:24.167+0000 7f67f03e9640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f67d0004d00 0x7f67e815d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:24.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:24.167+0000 7f67f03e9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f67e8155a80 0x7f67e80caca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:24.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:24.167+0000 7f67f03e9640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f67e806b110 0x7f67e806b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:24.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:24.167+0000 7f67f03e9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67e80c8b50 con 0x7f67e8155a80 2023-10-19T14:21:24.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:24.167+0000 7f67f03e9640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67e806bad0 con 0x7f67d0004d00 2023-10-19T14:21:24.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:24.167+0000 7f67f03e9640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67e80cb1e0 con 0x7f67e806b110 2023-10-19T14:21:24.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:24.167+0000 7f67ed984640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f67e8155a80 0x7f67e80caca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:24.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:24.167+0000 7f67ed984640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f67e8155a80 0x7f67e80caca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57690/0 (socket says 172.21.15.57:57690) 2023-10-19T14:21:24.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:24.167+0000 7f67ed984640 1 -- 172.21.15.57:0/1263396145 learned_addr learned my addr 172.21.15.57:0/1263396145 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:24.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:24.167+0000 7f67ee185640 1 --2- 172.21.15.57:0/1263396145 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f67d0004d00 0x7f67e815d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:24.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:24.167+0000 7f67ee986640 1 --2- 172.21.15.57:0/1263396145 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f67e806b110 0x7f67e806b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:24.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:24.167+0000 7f67ed984640 1 -- 172.21.15.57:0/1263396145 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f67e806b110 msgr2=0x7f67e806b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:24.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:24.167+0000 7f67ed984640 1 --2- 172.21.15.57:0/1263396145 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f67e806b110 0x7f67e806b590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:24.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:24.167+0000 7f67ed984640 1 -- 172.21.15.57:0/1263396145 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f67d0004d00 msgr2=0x7f67e815d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:24.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:24.167+0000 7f67ed984640 1 --2- 172.21.15.57:0/1263396145 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f67d0004d00 0x7f67e815d1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:24.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:24.167+0000 7f67ed984640 1 -- 172.21.15.57:0/1263396145 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67e8156a90 con 0x7f67e8155a80 2023-10-19T14:21:24.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:24.167+0000 7f67ee986640 1 --2- 172.21.15.57:0/1263396145 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f67e806b110 0x7f67e806b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:21:24.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:24.167+0000 7f67ed984640 1 --2- 172.21.15.57:0/1263396145 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f67e8155a80 0x7f67e80caca0 secure :-1 s=READY pgs=2678 cs=0 l=1 rev1=1 crypto rx=0x7f67dc011a50 tx=0x7f67dc011f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:24.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:24.168+0000 7f67e6ffd640 1 -- 172.21.15.57:0/1263396145 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67dc00fd90 con 0x7f67e8155a80 2023-10-19T14:21:24.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:24.168+0000 7f67e6ffd640 1 -- 172.21.15.57:0/1263396145 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f67dc012450 con 0x7f67e8155a80 2023-10-19T14:21:24.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:24.168+0000 7f67f03e9640 1 -- 172.21.15.57:0/1263396145 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67e81561f0 con 0x7f67e8155a80 2023-10-19T14:21:24.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:24.168+0000 7f67e6ffd640 1 -- 172.21.15.57:0/1263396145 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67dc012800 con 0x7f67e8155a80 2023-10-19T14:21:24.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:24.168+0000 7f67f03e9640 1 -- 172.21.15.57:0/1263396145 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67e81567b0 con 0x7f67e8155a80 2023-10-19T14:21:24.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:24.169+0000 7f67f03e9640 1 -- 172.21.15.57:0/1263396145 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67e80c88b0 con 0x7f67e8155a80 2023-10-19T14:21:24.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:24.169+0000 7f67e6ffd640 1 -- 172.21.15.57:0/1263396145 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f67dc012960 con 0x7f67e8155a80 2023-10-19T14:21:24.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:24.170+0000 7f67e6ffd640 1 --2- 172.21.15.57:0/1263396145 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f67d406b0b0 0x7f67d406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:24.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:24.170+0000 7f67e6ffd640 1 -- 172.21.15.57:0/1263396145 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(243..243 src has 1..243) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f67dc08aab0 con 0x7f67e8155a80 2023-10-19T14:21:24.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:24.170+0000 7f67ee185640 1 --2- 172.21.15.57:0/1263396145 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f67d406b0b0 0x7f67d406d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:24.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:24.170+0000 7f67ee185640 1 --2- 172.21.15.57:0/1263396145 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f67d406b0b0 0x7f67d406d570 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f67e00013b0 tx=0x7f67e0001980 comp rx=0 tx=0).ready entity=mgr.9530 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:24.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:24.174+0000 7f67e6ffd640 1 -- 172.21.15.57:0/1263396145 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f67e80c88b0 con 0x7f67e8155a80 2023-10-19T14:21:24.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:24.349+0000 7f67f03e9640 1 -- 172.21.15.57:0/1263396145 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "backup_fs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f67e80c88b0 con 0x7f67e8155a80 2023-10-19T14:21:25.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:25.346+0000 7f67e6ffd640 1 -- 172.21.15.57:0/1263396145 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "backup_fs_metadata", "pg_num_min": 64}]=0 pool 'backup_fs_metadata' created v244) v1 ==== 144+0+0 (secure 0 0 0) 0x7f67e80c88b0 con 0x7f67e8155a80 2023-10-19T14:21:25.349 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'backup_fs_metadata' created 2023-10-19T14:21:25.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:25.348+0000 7f67f03e9640 1 -- 172.21.15.57:0/1263396145 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f67d406b0b0 msgr2=0x7f67d406d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:25.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:25.348+0000 7f67f03e9640 1 --2- 172.21.15.57:0/1263396145 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f67d406b0b0 0x7f67d406d570 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f67e00013b0 tx=0x7f67e0001980 comp rx=0 tx=0).stop 2023-10-19T14:21:25.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:25.348+0000 7f67f03e9640 1 -- 172.21.15.57:0/1263396145 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f67e8155a80 msgr2=0x7f67e80caca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:25.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:25.348+0000 7f67f03e9640 1 --2- 172.21.15.57:0/1263396145 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f67e8155a80 0x7f67e80caca0 secure :-1 s=READY pgs=2678 cs=0 l=1 rev1=1 crypto rx=0x7f67dc011a50 tx=0x7f67dc011f10 comp rx=0 tx=0).stop 2023-10-19T14:21:25.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:25.349+0000 7f67f03e9640 1 -- 172.21.15.57:0/1263396145 shutdown_connections 2023-10-19T14:21:25.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:25.349+0000 7f67f03e9640 1 --2- 172.21.15.57:0/1263396145 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f67d406b0b0 0x7f67d406d570 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:25.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:25.349+0000 7f67f03e9640 1 --2- 172.21.15.57:0/1263396145 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f67e806b110 0x7f67e806b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:25.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:25.349+0000 7f67f03e9640 1 --2- 172.21.15.57:0/1263396145 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f67e8155a80 0x7f67e80caca0 unknown :-1 s=CLOSED pgs=2678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:25.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:25.349+0000 7f67f03e9640 1 --2- 172.21.15.57:0/1263396145 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f67d0004d00 0x7f67e815d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:25.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:25.349+0000 7f67f03e9640 1 -- 172.21.15.57:0/1263396145 >> 172.21.15.57:0/1263396145 conn(0x7f67e80d0d60 msgr2=0x7f67e80b2850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:25.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:25.349+0000 7f67f03e9640 1 -- 172.21.15.57:0/1263396145 shutdown_connections 2023-10-19T14:21:25.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:25.350+0000 7f67f03e9640 1 -- 172.21.15.57:0/1263396145 wait complete. 2023-10-19T14:21:25.370 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create backup_fs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-19T14:21:25.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:25.496+0000 7f1981c56640 1 -- 172.21.15.57:0/2608633179 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f197c06a730 msgr2=0x7f197c06abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:25.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:25.496+0000 7f1981c56640 1 --2- 172.21.15.57:0/2608633179 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f197c06a730 0x7f197c06abc0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f1974009be0 tx=0x7f197402c160 comp rx=0 tx=0).stop 2023-10-19T14:21:25.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:25.496+0000 7f1981c56640 1 -- 172.21.15.57:0/2608633179 shutdown_connections 2023-10-19T14:21:25.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:25.496+0000 7f1981c56640 1 --2- 172.21.15.57:0/2608633179 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f197c06a730 0x7f197c06abc0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:25.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:25.496+0000 7f1981c56640 1 -- 172.21.15.57:0/2608633179 >> 172.21.15.57:0/2608633179 conn(0x7f197c06fae0 msgr2=0x7f197c06fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:25.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:25.496+0000 7f1981c56640 1 -- 172.21.15.57:0/2608633179 shutdown_connections 2023-10-19T14:21:25.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:25.496+0000 7f1981c56640 1 -- 172.21.15.57:0/2608633179 wait complete. 2023-10-19T14:21:25.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:25.497+0000 7f1981c56640 1 Processor -- start 2023-10-19T14:21:25.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:25.497+0000 7f1981c56640 1 -- start start 2023-10-19T14:21:25.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:25.498+0000 7f1981c56640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f197c06a730 0x7f197c0a8470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:25.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:25.498+0000 7f1981c56640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f197c0ac870 0x7f197c147650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:25.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:25.498+0000 7f1981c56640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f197c156790 0x7f197c0aae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:25.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:25.498+0000 7f1981c56640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f197c146000 con 0x7f197c0ac870 2023-10-19T14:21:25.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:25.498+0000 7f1981c56640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f197c1453f0 con 0x7f197c156790 2023-10-19T14:21:25.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:25.498+0000 7f1981c56640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f197c1456d0 con 0x7f197c06a730 2023-10-19T14:21:25.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:25.498+0000 7f197affd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f197c0ac870 0x7f197c147650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:25.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:25.498+0000 7f197affd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f197c0ac870 0x7f197c147650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57696/0 (socket says 172.21.15.57:57696) 2023-10-19T14:21:25.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:25.498+0000 7f197affd640 1 -- 172.21.15.57:0/1286886052 learned_addr learned my addr 172.21.15.57:0/1286886052 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:25.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:25.498+0000 7f197b7fe640 1 --2- 172.21.15.57:0/1286886052 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f197c06a730 0x7f197c0a8470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:25.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:25.499+0000 7f197bfff640 1 --2- 172.21.15.57:0/1286886052 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f197c156790 0x7f197c0aae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:25.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:25.499+0000 7f197affd640 1 -- 172.21.15.57:0/1286886052 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f197c06a730 msgr2=0x7f197c0a8470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:25.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:25.499+0000 7f197affd640 1 --2- 172.21.15.57:0/1286886052 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f197c06a730 0x7f197c0a8470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:25.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:25.499+0000 7f197affd640 1 -- 172.21.15.57:0/1286886052 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f197c156790 msgr2=0x7f197c0aae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:25.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:25.499+0000 7f197affd640 1 --2- 172.21.15.57:0/1286886052 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f197c156790 0x7f197c0aae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:25.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:25.499+0000 7f197affd640 1 -- 172.21.15.57:0/1286886052 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1974009840 con 0x7f197c0ac870 2023-10-19T14:21:25.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:25.499+0000 7f197affd640 1 --2- 172.21.15.57:0/1286886052 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f197c0ac870 0x7f197c147650 secure :-1 s=READY pgs=2680 cs=0 l=1 rev1=1 crypto rx=0x7f197000c490 tx=0x7f197000c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:25.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:25.499+0000 7f1963fff640 1 -- 172.21.15.57:0/1286886052 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1970012830 con 0x7f197c0ac870 2023-10-19T14:21:25.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:25.499+0000 7f1963fff640 1 -- 172.21.15.57:0/1286886052 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f1970012a80 con 0x7f197c0ac870 2023-10-19T14:21:25.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:25.499+0000 7f1981c56640 1 -- 172.21.15.57:0/1286886052 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f197c1459b0 con 0x7f197c0ac870 2023-10-19T14:21:25.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:25.499+0000 7f1963fff640 1 -- 172.21.15.57:0/1286886052 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1970011380 con 0x7f197c0ac870 2023-10-19T14:21:25.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:25.499+0000 7f1981c56640 1 -- 172.21.15.57:0/1286886052 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f197c147b90 con 0x7f197c0ac870 2023-10-19T14:21:25.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:25.500+0000 7f1981c56640 1 -- 172.21.15.57:0/1286886052 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f197c0690e0 con 0x7f197c0ac870 2023-10-19T14:21:25.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:25.501+0000 7f1963fff640 1 -- 172.21.15.57:0/1286886052 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f19700114e0 con 0x7f197c0ac870 2023-10-19T14:21:25.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:25.501+0000 7f1963fff640 1 --2- 172.21.15.57:0/1286886052 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f196806b080 0x7f196806d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:25.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:25.501+0000 7f1963fff640 1 -- 172.21.15.57:0/1286886052 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(244..244 src has 1..244) v4 ==== 6430+0+0 (secure 0 0 0) 0x7f1970089dc0 con 0x7f197c0ac870 2023-10-19T14:21:25.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:25.501+0000 7f197b7fe640 1 --2- 172.21.15.57:0/1286886052 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f196806b080 0x7f196806d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:25.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:25.502+0000 7f197b7fe640 1 --2- 172.21.15.57:0/1286886052 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f196806b080 0x7f196806d540 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f196c000f60 tx=0x7f196c009040 comp rx=0 tx=0).ready entity=mgr.9530 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:25.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:25.506+0000 7f1963fff640 1 -- 172.21.15.57:0/1286886052 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f19700583a0 con 0x7f197c0ac870 2023-10-19T14:21:25.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:25.680+0000 7f1981c56640 1 -- 172.21.15.57:0/1286886052 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "backup_fs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f197c065630 con 0x7f197c0ac870 2023-10-19T14:21:26.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:26.348+0000 7f1963fff640 1 -- 172.21.15.57:0/1286886052 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "backup_fs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'backup_fs_data' created v245) v1 ==== 176+0+0 (secure 0 0 0) 0x7f19700160d0 con 0x7f197c0ac870 2023-10-19T14:21:26.351 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'backup_fs_data' created 2023-10-19T14:21:26.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:26.352+0000 7f1981c56640 1 -- 172.21.15.57:0/1286886052 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f196806b080 msgr2=0x7f196806d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:26.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:26.352+0000 7f1981c56640 1 --2- 172.21.15.57:0/1286886052 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f196806b080 0x7f196806d540 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f196c000f60 tx=0x7f196c009040 comp rx=0 tx=0).stop 2023-10-19T14:21:26.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:26.352+0000 7f1981c56640 1 -- 172.21.15.57:0/1286886052 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f197c0ac870 msgr2=0x7f197c147650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:26.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:26.352+0000 7f1981c56640 1 --2- 172.21.15.57:0/1286886052 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f197c0ac870 0x7f197c147650 secure :-1 s=READY pgs=2680 cs=0 l=1 rev1=1 crypto rx=0x7f197000c490 tx=0x7f197000c950 comp rx=0 tx=0).stop 2023-10-19T14:21:26.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:26.353+0000 7f1981c56640 1 -- 172.21.15.57:0/1286886052 shutdown_connections 2023-10-19T14:21:26.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:26.353+0000 7f1981c56640 1 --2- 172.21.15.57:0/1286886052 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f196806b080 0x7f196806d540 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:26.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:26.353+0000 7f1981c56640 1 --2- 172.21.15.57:0/1286886052 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f197c156790 0x7f197c0aae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:26.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:26.353+0000 7f1981c56640 1 --2- 172.21.15.57:0/1286886052 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f197c0ac870 0x7f197c147650 unknown :-1 s=CLOSED pgs=2680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:26.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:26.353+0000 7f1981c56640 1 --2- 172.21.15.57:0/1286886052 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f197c06a730 0x7f197c0a8470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:26.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:26.353+0000 7f1981c56640 1 -- 172.21.15.57:0/1286886052 >> 172.21.15.57:0/1286886052 conn(0x7f197c06fae0 msgr2=0x7f197c0c9670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:26.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:26.353+0000 7f1981c56640 1 -- 172.21.15.57:0/1286886052 shutdown_connections 2023-10-19T14:21:26.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:26.354+0000 7f1981c56640 1 -- 172.21.15.57:0/1286886052 wait complete. 2023-10-19T14:21:26.383 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new backup_fs backup_fs_metadata backup_fs_data 2023-10-19T14:21:26.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:26.499+0000 7feeda6cf640 1 -- 172.21.15.57:0/1626141125 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feed40cd9e0 msgr2=0x7feed40cde50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:26.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:26.499+0000 7feeda6cf640 1 --2- 172.21.15.57:0/1626141125 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feed40cd9e0 0x7feed40cde50 secure :-1 s=READY pgs=2682 cs=0 l=1 rev1=1 crypto rx=0x7feed4073460 tx=0x7feecc02c110 comp rx=0 tx=0).stop 2023-10-19T14:21:26.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:26.499+0000 7feeda6cf640 1 -- 172.21.15.57:0/1626141125 shutdown_connections 2023-10-19T14:21:26.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:26.499+0000 7feeda6cf640 1 --2- 172.21.15.57:0/1626141125 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feed40cd9e0 0x7feed40cde50 unknown :-1 s=CLOSED pgs=2682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:26.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:26.499+0000 7feeda6cf640 1 --2- 172.21.15.57:0/1626141125 >> v2:172.21.15.179:3300/0 conn(0x7feed4158910 0x7feed4158cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:26.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:26.499+0000 7feeda6cf640 1 -- 172.21.15.57:0/1626141125 >> 172.21.15.57:0/1626141125 conn(0x7feed40af7a0 msgr2=0x7feed40afb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:26.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:26.499+0000 7feeda6cf640 1 -- 172.21.15.57:0/1626141125 shutdown_connections 2023-10-19T14:21:26.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:26.499+0000 7feeda6cf640 1 -- 172.21.15.57:0/1626141125 wait complete. 2023-10-19T14:21:26.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:26.500+0000 7feeda6cf640 1 Processor -- start 2023-10-19T14:21:26.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:26.501+0000 7feeda6cf640 1 -- start start 2023-10-19T14:21:26.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:26.501+0000 7feeda6cf640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feed40cd1c0 0x7feed414a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:26.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:26.501+0000 7feeda6cf640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7feed40cd9e0 0x7feed40b0890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:26.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:26.501+0000 7feeda6cf640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feed4158910 0x7feed414bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:26.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:26.501+0000 7feeda6cf640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feed4152a10 con 0x7feed4158910 2023-10-19T14:21:26.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:26.501+0000 7feeda6cf640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feed4153540 con 0x7feed40cd1c0 2023-10-19T14:21:26.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:26.501+0000 7feeda6cf640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feed414c360 con 0x7feed40cd9e0 2023-10-19T14:21:26.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:26.501+0000 7feed8c6c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feed4158910 0x7feed414bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:26.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:26.502+0000 7feed8c6c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feed4158910 0x7feed414bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57706/0 (socket says 172.21.15.57:57706) 2023-10-19T14:21:26.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:26.502+0000 7feed8c6c640 1 -- 172.21.15.57:0/4058252120 learned_addr learned my addr 172.21.15.57:0/4058252120 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:26.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:26.502+0000 7feed3fff640 1 --2- 172.21.15.57:0/4058252120 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feed40cd1c0 0x7feed414a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:26.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:26.502+0000 7feed37fe640 1 --2- 172.21.15.57:0/4058252120 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7feed40cd9e0 0x7feed40b0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:26.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:26.502+0000 7feed8c6c640 1 -- 172.21.15.57:0/4058252120 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7feed40cd9e0 msgr2=0x7feed40b0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:26.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:26.502+0000 7feed8c6c640 1 --2- 172.21.15.57:0/4058252120 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7feed40cd9e0 0x7feed40b0890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:26.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:26.502+0000 7feed8c6c640 1 -- 172.21.15.57:0/4058252120 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feed40cd1c0 msgr2=0x7feed414a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:26.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:26.502+0000 7feed8c6c640 1 --2- 172.21.15.57:0/4058252120 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feed40cd1c0 0x7feed414a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:26.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:26.502+0000 7feed8c6c640 1 -- 172.21.15.57:0/4058252120 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feecc009840 con 0x7feed4158910 2023-10-19T14:21:26.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:26.502+0000 7feed8c6c640 1 --2- 172.21.15.57:0/4058252120 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feed4158910 0x7feed414bb30 secure :-1 s=READY pgs=2683 cs=0 l=1 rev1=1 crypto rx=0x7feecc009ca0 tx=0x7feecc004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:26.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:26.504+0000 7feed0ff9640 1 -- 172.21.15.57:0/4058252120 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feecc004670 con 0x7feed4158910 2023-10-19T14:21:26.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:26.504+0000 7feed0ff9640 1 -- 172.21.15.57:0/4058252120 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7feecc0047f0 con 0x7feed4158910 2023-10-19T14:21:26.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:26.504+0000 7feeda6cf640 1 -- 172.21.15.57:0/4058252120 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feed414c640 con 0x7feed4158910 2023-10-19T14:21:26.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:26.504+0000 7feed0ff9640 1 -- 172.21.15.57:0/4058252120 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feecc034800 con 0x7feed4158910 2023-10-19T14:21:26.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:26.504+0000 7feeda6cf640 1 -- 172.21.15.57:0/4058252120 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feed41537c0 con 0x7feed4158910 2023-10-19T14:21:26.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:26.505+0000 7feeda6cf640 1 -- 172.21.15.57:0/4058252120 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feed414cb60 con 0x7feed4158910 2023-10-19T14:21:26.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:26.505+0000 7feed0ff9640 1 -- 172.21.15.57:0/4058252120 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7feecc034960 con 0x7feed4158910 2023-10-19T14:21:26.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:26.506+0000 7feed0ff9640 1 --2- 172.21.15.57:0/4058252120 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7feec006b0b0 0x7feec006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:26.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:26.506+0000 7feed0ff9640 1 -- 172.21.15.57:0/4058252120 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(245..245 src has 1..245) v4 ==== 6815+0+0 (secure 0 0 0) 0x7feecc0ad270 con 0x7feed4158910 2023-10-19T14:21:26.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:26.506+0000 7feed3fff640 1 --2- 172.21.15.57:0/4058252120 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7feec006b0b0 0x7feec006d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:26.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:26.506+0000 7feed3fff640 1 --2- 172.21.15.57:0/4058252120 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7feec006b0b0 0x7feec006d570 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7feed4070c20 tx=0x7feec8005f90 comp rx=0 tx=0).ready entity=mgr.9530 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:26.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:26.510+0000 7feed0ff9640 1 -- 172.21.15.57:0/4058252120 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7feed414cb60 con 0x7feed4158910 2023-10-19T14:21:26.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:26.723+0000 7feeda6cf640 1 -- 172.21.15.57:0/4058252120 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "backup_fs", "metadata": "backup_fs_metadata", "data": "backup_fs_data"} v 0) v1 -- 0x7feed414cb60 con 0x7feed4158910 2023-10-19T14:21:27.524 INFO:teuthology.orchestra.run.smithi057.stderr:new fs with metadata pool 33 and data pool 34 2023-10-19T14:21:27.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.518+0000 7feed0ff9640 1 -- 172.21.15.57:0/4058252120 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "backup_fs", "metadata": "backup_fs_metadata", "data": "backup_fs_data"}]=0 new fs with metadata pool 33 and data pool 34 v105) v1 ==== 183+0+0 (secure 0 0 0) 0x7feed414cb60 con 0x7feed4158910 2023-10-19T14:21:27.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.532+0000 7feeda6cf640 1 -- 172.21.15.57:0/4058252120 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7feec006b0b0 msgr2=0x7feec006d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:27.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.532+0000 7feeda6cf640 1 --2- 172.21.15.57:0/4058252120 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7feec006b0b0 0x7feec006d570 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7feed4070c20 tx=0x7feec8005f90 comp rx=0 tx=0).stop 2023-10-19T14:21:27.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.532+0000 7feeda6cf640 1 -- 172.21.15.57:0/4058252120 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feed4158910 msgr2=0x7feed414bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:27.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.532+0000 7feeda6cf640 1 --2- 172.21.15.57:0/4058252120 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feed4158910 0x7feed414bb30 secure :-1 s=READY pgs=2683 cs=0 l=1 rev1=1 crypto rx=0x7feecc009ca0 tx=0x7feecc004270 comp rx=0 tx=0).stop 2023-10-19T14:21:27.539 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.535+0000 7feeda6cf640 1 -- 172.21.15.57:0/4058252120 shutdown_connections 2023-10-19T14:21:27.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.535+0000 7feeda6cf640 1 --2- 172.21.15.57:0/4058252120 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7feec006b0b0 0x7feec006d570 secure :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7feed4070c20 tx=0x7feec8005f90 comp rx=0 tx=0).stop 2023-10-19T14:21:27.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.535+0000 7feeda6cf640 1 --2- 172.21.15.57:0/4058252120 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feed4158910 0x7feed414bb30 unknown :-1 s=CLOSED pgs=2683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:27.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.535+0000 7feeda6cf640 1 --2- 172.21.15.57:0/4058252120 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7feed40cd9e0 0x7feed40b0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:27.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.535+0000 7feeda6cf640 1 --2- 172.21.15.57:0/4058252120 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feed40cd1c0 0x7feed414a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:27.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.535+0000 7feeda6cf640 1 -- 172.21.15.57:0/4058252120 >> 172.21.15.57:0/4058252120 conn(0x7feed40af7a0 msgr2=0x7feed4155f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:27.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.537+0000 7feeda6cf640 1 -- 172.21.15.57:0/4058252120 shutdown_connections 2023-10-19T14:21:27.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.537+0000 7feeda6cf640 1 -- 172.21.15.57:0/4058252120 wait complete. 2023-10-19T14:21:27.559 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T14:21:27.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.670+0000 7fc185705640 1 -- 172.21.15.57:0/148531353 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc18006ae40 msgr2=0x7fc1800b0e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:27.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.670+0000 7fc185705640 1 --2- 172.21.15.57:0/148531353 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc18006ae40 0x7fc1800b0e90 secure :-1 s=READY pgs=2684 cs=0 l=1 rev1=1 crypto rx=0x7fc1800acd30 tx=0x7fc16802bdb0 comp rx=0 tx=0).stop 2023-10-19T14:21:27.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.670+0000 7fc185705640 1 -- 172.21.15.57:0/148531353 shutdown_connections 2023-10-19T14:21:27.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.670+0000 7fc185705640 1 --2- 172.21.15.57:0/148531353 >> v2:172.21.15.179:3300/0 conn(0x7fc18015c1f0 0x7fc180157dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:27.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.670+0000 7fc185705640 1 --2- 172.21.15.57:0/148531353 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc18006ae40 0x7fc1800b0e90 unknown :-1 s=CLOSED pgs=2684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:27.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.670+0000 7fc185705640 1 -- 172.21.15.57:0/148531353 >> 172.21.15.57:0/148531353 conn(0x7fc18006fae0 msgr2=0x7fc18006fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:27.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.670+0000 7fc185705640 1 -- 172.21.15.57:0/148531353 shutdown_connections 2023-10-19T14:21:27.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.670+0000 7fc185705640 1 -- 172.21.15.57:0/148531353 wait complete. 2023-10-19T14:21:27.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.671+0000 7fc185705640 1 Processor -- start 2023-10-19T14:21:27.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.672+0000 7fc185705640 1 -- start start 2023-10-19T14:21:27.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.672+0000 7fc185705640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc18006ae40 0x7fc1801568e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:27.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.672+0000 7fc185705640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc1801579f0 0x7fc1801f1040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:27.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.672+0000 7fc185705640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc18015c1f0 0x7fc1801f4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:27.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.672+0000 7fc185705640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1801f5c20 con 0x7fc18006ae40 2023-10-19T14:21:27.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.672+0000 7fc185705640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1801540b0 con 0x7fc18015c1f0 2023-10-19T14:21:27.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.672+0000 7fc185705640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc180156e20 con 0x7fc1801579f0 2023-10-19T14:21:27.678 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.672+0000 7fc17effd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc18006ae40 0x7fc1801568e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:27.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.672+0000 7fc17effd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc18006ae40 0x7fc1801568e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37086/0 (socket says 172.21.15.57:37086) 2023-10-19T14:21:27.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.672+0000 7fc17effd640 1 -- 172.21.15.57:0/2327923589 learned_addr learned my addr 172.21.15.57:0/2327923589 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:27.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.673+0000 7fc17f7fe640 1 --2- 172.21.15.57:0/2327923589 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc18015c1f0 0x7fc1801f4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:27.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.673+0000 7fc1767fc640 1 --2- 172.21.15.57:0/2327923589 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc1801579f0 0x7fc1801f1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:27.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.673+0000 7fc17effd640 1 -- 172.21.15.57:0/2327923589 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc1801579f0 msgr2=0x7fc1801f1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:27.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.673+0000 7fc17effd640 1 --2- 172.21.15.57:0/2327923589 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc1801579f0 0x7fc1801f1040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:27.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.673+0000 7fc17effd640 1 -- 172.21.15.57:0/2327923589 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc18015c1f0 msgr2=0x7fc1801f4dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:27.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.673+0000 7fc17effd640 1 --2- 172.21.15.57:0/2327923589 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc18015c1f0 0x7fc1801f4dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:27.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.673+0000 7fc17effd640 1 -- 172.21.15.57:0/2327923589 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc168009560 con 0x7fc18006ae40 2023-10-19T14:21:27.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.673+0000 7fc17effd640 1 --2- 172.21.15.57:0/2327923589 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc18006ae40 0x7fc1801568e0 secure :-1 s=READY pgs=2685 cs=0 l=1 rev1=1 crypto rx=0x7fc17000c430 tx=0x7fc17000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:27.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.673+0000 7fc177fff640 1 -- 172.21.15.57:0/2327923589 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc170011790 con 0x7fc18006ae40 2023-10-19T14:21:27.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.673+0000 7fc177fff640 1 -- 172.21.15.57:0/2327923589 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fc1700119e0 con 0x7fc18006ae40 2023-10-19T14:21:27.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.673+0000 7fc185705640 1 -- 172.21.15.57:0/2327923589 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1800b1de0 con 0x7fc18006ae40 2023-10-19T14:21:27.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.673+0000 7fc177fff640 1 -- 172.21.15.57:0/2327923589 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc170011e30 con 0x7fc18006ae40 2023-10-19T14:21:27.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.673+0000 7fc185705640 1 -- 172.21.15.57:0/2327923589 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc180154630 con 0x7fc18006ae40 2023-10-19T14:21:27.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.674+0000 7fc185705640 1 -- 172.21.15.57:0/2327923589 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1800b11f0 con 0x7fc18006ae40 2023-10-19T14:21:27.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.675+0000 7fc177fff640 1 -- 172.21.15.57:0/2327923589 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fc17001c460 con 0x7fc18006ae40 2023-10-19T14:21:27.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.675+0000 7fc177fff640 1 --2- 172.21.15.57:0/2327923589 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fc16006b080 0x7fc16006d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:27.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.675+0000 7fc177fff640 1 -- 172.21.15.57:0/2327923589 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(246..246 src has 1..246) v4 ==== 6921+0+0 (secure 0 0 0) 0x7fc170089f60 con 0x7fc18006ae40 2023-10-19T14:21:27.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.675+0000 7fc1767fc640 1 --2- 172.21.15.57:0/2327923589 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fc16006b080 0x7fc16006d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:27.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.676+0000 7fc1767fc640 1 --2- 172.21.15.57:0/2327923589 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fc16006b080 0x7fc16006d540 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc168035000 tx=0x7fc168004730 comp rx=0 tx=0).ready entity=mgr.9530 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:27.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.679+0000 7fc177fff640 1 -- 172.21.15.57:0/2327923589 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc170054970 con 0x7fc18006ae40 2023-10-19T14:21:27.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.857+0000 7fc185705640 1 -- 172.21.15.57:0/2327923589 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc1800690e0 con 0x7fc18006ae40 2023-10-19T14:21:27.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.858+0000 7fc177fff640 1 -- 172.21.15.57:0/2327923589 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v246) v1 ==== 74+0+20512 (secure 0 0 0) 0x7fc170057f90 con 0x7fc18006ae40 2023-10-19T14:21:27.862 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:21:27.862 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":246,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T14:21:26.724414+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2023-10-19T14:21:08.101023+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"242","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2023-10-19T14:21:08.745817+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"243","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_metadata","create_time":"2023-10-19T14:21:24.384693+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"246","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"backup_fs_data","create_time":"2023-10-19T14:21:25.717524+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"246","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T14:21:27.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.860+0000 7fc185705640 1 -- 172.21.15.57:0/2327923589 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fc16006b080 msgr2=0x7fc16006d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:27.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.860+0000 7fc185705640 1 --2- 172.21.15.57:0/2327923589 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fc16006b080 0x7fc16006d540 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc168035000 tx=0x7fc168004730 comp rx=0 tx=0).stop 2023-10-19T14:21:27.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.860+0000 7fc185705640 1 -- 172.21.15.57:0/2327923589 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc18006ae40 msgr2=0x7fc1801568e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:27.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.860+0000 7fc185705640 1 --2- 172.21.15.57:0/2327923589 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc18006ae40 0x7fc1801568e0 secure :-1 s=READY pgs=2685 cs=0 l=1 rev1=1 crypto rx=0x7fc17000c430 tx=0x7fc17000c8f0 comp rx=0 tx=0).stop 2023-10-19T14:21:27.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.860+0000 7fc185705640 1 -- 172.21.15.57:0/2327923589 shutdown_connections 2023-10-19T14:21:27.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.860+0000 7fc185705640 1 --2- 172.21.15.57:0/2327923589 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fc16006b080 0x7fc16006d540 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:27.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.860+0000 7fc185705640 1 --2- 172.21.15.57:0/2327923589 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc18015c1f0 0x7fc1801f4dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:27.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.860+0000 7fc185705640 1 --2- 172.21.15.57:0/2327923589 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc1801579f0 0x7fc1801f1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:27.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.860+0000 7fc185705640 1 --2- 172.21.15.57:0/2327923589 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc18006ae40 0x7fc1801568e0 unknown :-1 s=CLOSED pgs=2685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:27.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.860+0000 7fc185705640 1 -- 172.21.15.57:0/2327923589 >> 172.21.15.57:0/2327923589 conn(0x7fc18006fae0 msgr2=0x7fc18006a5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:27.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.861+0000 7fc185705640 1 -- 172.21.15.57:0/2327923589 shutdown_connections 2023-10-19T14:21:27.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.861+0000 7fc185705640 1 -- 172.21.15.57:0/2327923589 wait complete. 2023-10-19T14:21:27.874 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T14:21:27.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.981+0000 7fb239aa9640 1 -- 172.21.15.57:0/2998265940 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb21c0052a0 msgr2=0x7fb21c00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:27.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.981+0000 7fb239aa9640 1 --2- 172.21.15.57:0/2998265940 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb21c0052a0 0x7fb21c00b980 secure :-1 s=READY pgs=2687 cs=0 l=1 rev1=1 crypto rx=0x7fb22c005fd0 tx=0x7fb21c0057f0 comp rx=0 tx=0).stop 2023-10-19T14:21:27.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.981+0000 7fb239aa9640 1 -- 172.21.15.57:0/2998265940 shutdown_connections 2023-10-19T14:21:27.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.981+0000 7fb239aa9640 1 --2- 172.21.15.57:0/2998265940 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb21c0052a0 0x7fb21c00b980 unknown :-1 s=CLOSED pgs=2687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:27.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.981+0000 7fb239aa9640 1 -- 172.21.15.57:0/2998265940 >> 172.21.15.57:0/2998265940 conn(0x7fb2340d0d60 msgr2=0x7fb2340d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:27.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.981+0000 7fb239aa9640 1 -- 172.21.15.57:0/2998265940 shutdown_connections 2023-10-19T14:21:27.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.981+0000 7fb239aa9640 1 -- 172.21.15.57:0/2998265940 wait complete. 2023-10-19T14:21:27.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.982+0000 7fb239aa9640 1 Processor -- start 2023-10-19T14:21:27.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.982+0000 7fb239aa9640 1 -- start start 2023-10-19T14:21:27.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.983+0000 7fb239aa9640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb21c0052a0 0x7fb234158300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:27.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.983+0000 7fb239aa9640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb2340b38e0 0x7fb2340b51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:27.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.983+0000 7fb239aa9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb23406afe0 0x7fb23406b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:27.989 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.983+0000 7fb239aa9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb23406b9d0 con 0x7fb23406afe0 2023-10-19T14:21:27.989 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.983+0000 7fb239aa9640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2340c7e30 con 0x7fb2340b38e0 2023-10-19T14:21:27.989 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.983+0000 7fb239aa9640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2340b4990 con 0x7fb21c0052a0 2023-10-19T14:21:27.989 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.983+0000 7fb233fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb23406afe0 0x7fb23406b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:27.989 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.983+0000 7fb233fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb23406afe0 0x7fb23406b490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37112/0 (socket says 172.21.15.57:37112) 2023-10-19T14:21:27.990 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.983+0000 7fb233fff640 1 -- 172.21.15.57:0/233825587 learned_addr learned my addr 172.21.15.57:0/233825587 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:27.990 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.983+0000 7fb233fff640 1 -- 172.21.15.57:0/233825587 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb21c0052a0 msgr2=0x7fb234158300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:27.990 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.983+0000 7fb233fff640 1 --2- 172.21.15.57:0/233825587 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb21c0052a0 0x7fb234158300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:27.990 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.983+0000 7fb233fff640 1 -- 172.21.15.57:0/233825587 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb2340b38e0 msgr2=0x7fb2340b51c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:27.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.983+0000 7fb233fff640 1 --2- 172.21.15.57:0/233825587 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb2340b38e0 0x7fb2340b51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:27.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.983+0000 7fb233fff640 1 -- 172.21.15.57:0/233825587 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2340b3e10 con 0x7fb23406afe0 2023-10-19T14:21:27.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.984+0000 7fb233fff640 1 --2- 172.21.15.57:0/233825587 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb23406afe0 0x7fb23406b490 secure :-1 s=READY pgs=2688 cs=0 l=1 rev1=1 crypto rx=0x7fb22c009f10 tx=0x7fb22c005340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:27.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.984+0000 7fb21bfff640 1 -- 172.21.15.57:0/233825587 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb22c005940 con 0x7fb23406afe0 2023-10-19T14:21:27.991 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.984+0000 7fb21bfff640 1 -- 172.21.15.57:0/233825587 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fb22c03d550 con 0x7fb23406afe0 2023-10-19T14:21:27.992 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.984+0000 7fb21bfff640 1 -- 172.21.15.57:0/233825587 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb22c03d960 con 0x7fb23406afe0 2023-10-19T14:21:27.992 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.984+0000 7fb239aa9640 1 -- 172.21.15.57:0/233825587 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2340b40f0 con 0x7fb23406afe0 2023-10-19T14:21:27.992 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.984+0000 7fb239aa9640 1 -- 172.21.15.57:0/233825587 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2340b46b0 con 0x7fb23406afe0 2023-10-19T14:21:27.992 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.985+0000 7fb239aa9640 1 -- 172.21.15.57:0/233825587 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2340c7fc0 con 0x7fb23406afe0 2023-10-19T14:21:27.993 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.985+0000 7fb21bfff640 1 -- 172.21.15.57:0/233825587 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fb22c03dac0 con 0x7fb23406afe0 2023-10-19T14:21:27.993 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.986+0000 7fb21bfff640 1 --2- 172.21.15.57:0/233825587 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fb22006b0b0 0x7fb22006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:27.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.986+0000 7fb21bfff640 1 -- 172.21.15.57:0/233825587 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(246..246 src has 1..246) v4 ==== 6921+0+0 (secure 0 0 0) 0x7fb22c0ac940 con 0x7fb23406afe0 2023-10-19T14:21:27.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.986+0000 7fb2337fe640 1 --2- 172.21.15.57:0/233825587 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fb22006b0b0 0x7fb22006d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:27.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.986+0000 7fb2337fe640 1 --2- 172.21.15.57:0/233825587 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fb22006b0b0 0x7fb22006d570 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb224000fd0 tx=0x7fb224000f40 comp rx=0 tx=0).ready entity=mgr.9530 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:27.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:27.990+0000 7fb21bfff640 1 -- 172.21.15.57:0/233825587 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb2340c7fc0 con 0x7fb23406afe0 2023-10-19T14:21:28.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.166+0000 7fb239aa9640 1 -- 172.21.15.57:0/233825587 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb2340c7fc0 con 0x7fb23406afe0 2023-10-19T14:21:28.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.167+0000 7fb21bfff640 1 -- 172.21.15.57:0/233825587 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v246) v1 ==== 74+0+20512 (secure 0 0 0) 0x7fb2340c7fc0 con 0x7fb23406afe0 2023-10-19T14:21:28.171 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:21:28.171 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":246,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T14:21:26.724414+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2023-10-19T14:21:08.101023+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"242","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2023-10-19T14:21:08.745817+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"243","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_metadata","create_time":"2023-10-19T14:21:24.384693+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"246","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"backup_fs_data","create_time":"2023-10-19T14:21:25.717524+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"246","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T14:21:28.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.169+0000 7fb239aa9640 1 -- 172.21.15.57:0/233825587 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fb22006b0b0 msgr2=0x7fb22006d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:28.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.169+0000 7fb239aa9640 1 --2- 172.21.15.57:0/233825587 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fb22006b0b0 0x7fb22006d570 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb224000fd0 tx=0x7fb224000f40 comp rx=0 tx=0).stop 2023-10-19T14:21:28.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.169+0000 7fb239aa9640 1 -- 172.21.15.57:0/233825587 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb23406afe0 msgr2=0x7fb23406b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:28.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.169+0000 7fb239aa9640 1 --2- 172.21.15.57:0/233825587 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb23406afe0 0x7fb23406b490 secure :-1 s=READY pgs=2688 cs=0 l=1 rev1=1 crypto rx=0x7fb22c009f10 tx=0x7fb22c005340 comp rx=0 tx=0).stop 2023-10-19T14:21:28.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.169+0000 7fb239aa9640 1 -- 172.21.15.57:0/233825587 shutdown_connections 2023-10-19T14:21:28.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.169+0000 7fb239aa9640 1 --2- 172.21.15.57:0/233825587 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fb22006b0b0 0x7fb22006d570 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:28.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.169+0000 7fb239aa9640 1 --2- 172.21.15.57:0/233825587 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb23406afe0 0x7fb23406b490 unknown :-1 s=CLOSED pgs=2688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:28.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.169+0000 7fb239aa9640 1 --2- 172.21.15.57:0/233825587 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb2340b38e0 0x7fb2340b51c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:28.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.169+0000 7fb239aa9640 1 --2- 172.21.15.57:0/233825587 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb21c0052a0 0x7fb234158300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:28.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.169+0000 7fb239aa9640 1 -- 172.21.15.57:0/233825587 >> 172.21.15.57:0/233825587 conn(0x7fb2340d0d60 msgr2=0x7fb2340cc970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:28.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.169+0000 7fb239aa9640 1 -- 172.21.15.57:0/233825587 shutdown_connections 2023-10-19T14:21:28.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.169+0000 7fb239aa9640 1 -- 172.21.15.57:0/233825587 wait complete. 2023-10-19T14:21:28.183 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set backup_fs standby_count_wanted 0 2023-10-19T14:21:28.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.292+0000 7f3690ae5640 1 -- 172.21.15.57:0/1857351834 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f368c06af00 msgr2=0x7f368c06b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:28.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.292+0000 7f3690ae5640 1 --2- 172.21.15.57:0/1857351834 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f368c06af00 0x7f368c06b2e0 secure :-1 s=READY pgs=2689 cs=0 l=1 rev1=1 crypto rx=0x7f3684005fd0 tx=0x7f368402bd10 comp rx=0 tx=0).stop 2023-10-19T14:21:28.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.292+0000 7f3690ae5640 1 -- 172.21.15.57:0/1857351834 shutdown_connections 2023-10-19T14:21:28.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.292+0000 7f3690ae5640 1 --2- 172.21.15.57:0/1857351834 >> v2:172.21.15.179:3300/0 conn(0x7f368c0c96b0 0x7f368c0c9b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:28.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.292+0000 7f3690ae5640 1 --2- 172.21.15.57:0/1857351834 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f368c06af00 0x7f368c06b2e0 unknown :-1 s=CLOSED pgs=2689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:28.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.292+0000 7f3690ae5640 1 -- 172.21.15.57:0/1857351834 >> 172.21.15.57:0/1857351834 conn(0x7f368c069930 msgr2=0x7f368c069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:28.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.292+0000 7f3690ae5640 1 -- 172.21.15.57:0/1857351834 shutdown_connections 2023-10-19T14:21:28.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.292+0000 7f3690ae5640 1 -- 172.21.15.57:0/1857351834 wait complete. 2023-10-19T14:21:28.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.293+0000 7f3690ae5640 1 Processor -- start 2023-10-19T14:21:28.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.293+0000 7f3690ae5640 1 -- start start 2023-10-19T14:21:28.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.294+0000 7f3690ae5640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f368c06af00 0x7f368c12cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:28.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.294+0000 7f3690ae5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f368c0ab500 0x7f368c1eede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:28.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.294+0000 7f3690ae5640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f368c0c96b0 0x7f368c0ad6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:28.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.294+0000 7f3690ae5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f368c12d560 con 0x7f368c0ab500 2023-10-19T14:21:28.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.294+0000 7f3690ae5640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f368c12d840 con 0x7f368c0c96b0 2023-10-19T14:21:28.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.294+0000 7f3690ae5640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f368c12db20 con 0x7f368c06af00 2023-10-19T14:21:28.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.294+0000 7f3689d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f368c0ab500 0x7f368c1eede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:28.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.294+0000 7f3689d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f368c0ab500 0x7f368c1eede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37144/0 (socket says 172.21.15.57:37144) 2023-10-19T14:21:28.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.294+0000 7f3689d9b640 1 -- 172.21.15.57:0/1390382551 learned_addr learned my addr 172.21.15.57:0/1390382551 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:28.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.295+0000 7f368a59c640 1 --2- 172.21.15.57:0/1390382551 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f368c06af00 0x7f368c12cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:28.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.295+0000 7f368ad9d640 1 --2- 172.21.15.57:0/1390382551 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f368c0c96b0 0x7f368c0ad6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:28.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.295+0000 7f3689d9b640 1 -- 172.21.15.57:0/1390382551 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f368c06af00 msgr2=0x7f368c12cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:28.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.295+0000 7f3689d9b640 1 --2- 172.21.15.57:0/1390382551 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f368c06af00 0x7f368c12cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:28.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.295+0000 7f3689d9b640 1 -- 172.21.15.57:0/1390382551 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f368c0c96b0 msgr2=0x7f368c0ad6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:28.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.295+0000 7f3689d9b640 1 --2- 172.21.15.57:0/1390382551 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f368c0c96b0 0x7f368c0ad6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:28.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.295+0000 7f3689d9b640 1 -- 172.21.15.57:0/1390382551 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3684009560 con 0x7f368c0ab500 2023-10-19T14:21:28.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.295+0000 7f3689d9b640 1 --2- 172.21.15.57:0/1390382551 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f368c0ab500 0x7f368c1eede0 secure :-1 s=READY pgs=2690 cs=0 l=1 rev1=1 crypto rx=0x7f368000b410 tx=0x7f368000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:28.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.295+0000 7f367affd640 1 -- 172.21.15.57:0/1390382551 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3680011830 con 0x7f368c0ab500 2023-10-19T14:21:28.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.295+0000 7f367affd640 1 -- 172.21.15.57:0/1390382551 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f3680011aa0 con 0x7f368c0ab500 2023-10-19T14:21:28.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.295+0000 7f3690ae5640 1 -- 172.21.15.57:0/1390382551 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f368c12aca0 con 0x7f368c0ab500 2023-10-19T14:21:28.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.295+0000 7f367affd640 1 -- 172.21.15.57:0/1390382551 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3680010380 con 0x7f368c0ab500 2023-10-19T14:21:28.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.295+0000 7f3690ae5640 1 -- 172.21.15.57:0/1390382551 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f368c12d270 con 0x7f368c0ab500 2023-10-19T14:21:28.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.296+0000 7f3690ae5640 1 -- 172.21.15.57:0/1390382551 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f368c0690e0 con 0x7f368c0ab500 2023-10-19T14:21:28.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.297+0000 7f367affd640 1 -- 172.21.15.57:0/1390382551 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f36800104e0 con 0x7f368c0ab500 2023-10-19T14:21:28.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.297+0000 7f367affd640 1 --2- 172.21.15.57:0/1390382551 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f366c06b080 0x7f366c06d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:28.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.297+0000 7f367affd640 1 -- 172.21.15.57:0/1390382551 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(246..246 src has 1..246) v4 ==== 6921+0+0 (secure 0 0 0) 0x7f368008a3e0 con 0x7f368c0ab500 2023-10-19T14:21:28.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.297+0000 7f368a59c640 1 --2- 172.21.15.57:0/1390382551 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f366c06b080 0x7f366c06d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:28.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.298+0000 7f368a59c640 1 --2- 172.21.15.57:0/1390382551 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f366c06b080 0x7f366c06d540 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3684035000 tx=0x7f3684004730 comp rx=0 tx=0).ready entity=mgr.9530 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:28.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.301+0000 7f367affd640 1 -- 172.21.15.57:0/1390382551 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3680054bf0 con 0x7f368c0ab500 2023-10-19T14:21:28.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.512+0000 7f3690ae5640 1 -- 172.21.15.57:0/1390382551 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "backup_fs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f368c0c8dc0 con 0x7f368c0ab500 2023-10-19T14:21:28.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.520+0000 7f367affd640 1 -- 172.21.15.57:0/1390382551 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "backup_fs", "var": "standby_count_wanted", "val": "0"}]=0 v106) v1 ==== 121+0+0 (secure 0 0 0) 0x7f3680058210 con 0x7f368c0ab500 2023-10-19T14:21:28.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.524+0000 7f3690ae5640 1 -- 172.21.15.57:0/1390382551 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f366c06b080 msgr2=0x7f366c06d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:28.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.524+0000 7f3690ae5640 1 --2- 172.21.15.57:0/1390382551 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f366c06b080 0x7f366c06d540 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3684035000 tx=0x7f3684004730 comp rx=0 tx=0).stop 2023-10-19T14:21:28.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.524+0000 7f3690ae5640 1 -- 172.21.15.57:0/1390382551 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f368c0ab500 msgr2=0x7f368c1eede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:28.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.524+0000 7f3690ae5640 1 --2- 172.21.15.57:0/1390382551 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f368c0ab500 0x7f368c1eede0 secure :-1 s=READY pgs=2690 cs=0 l=1 rev1=1 crypto rx=0x7f368000b410 tx=0x7f368000b8d0 comp rx=0 tx=0).stop 2023-10-19T14:21:28.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.524+0000 7f3690ae5640 1 -- 172.21.15.57:0/1390382551 shutdown_connections 2023-10-19T14:21:28.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.524+0000 7f3690ae5640 1 --2- 172.21.15.57:0/1390382551 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f366c06b080 0x7f366c06d540 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:28.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.524+0000 7f3690ae5640 1 --2- 172.21.15.57:0/1390382551 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f368c0c96b0 0x7f368c0ad6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:28.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.524+0000 7f3690ae5640 1 --2- 172.21.15.57:0/1390382551 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f368c0ab500 0x7f368c1eede0 unknown :-1 s=CLOSED pgs=2690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:28.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.524+0000 7f3690ae5640 1 --2- 172.21.15.57:0/1390382551 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f368c06af00 0x7f368c12cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:28.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.524+0000 7f3690ae5640 1 -- 172.21.15.57:0/1390382551 >> 172.21.15.57:0/1390382551 conn(0x7f368c069930 msgr2=0x7f368c069d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:28.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.524+0000 7f3690ae5640 1 -- 172.21.15.57:0/1390382551 shutdown_connections 2023-10-19T14:21:28.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.524+0000 7f3690ae5640 1 -- 172.21.15.57:0/1390382551 wait complete. 2023-10-19T14:21:28.538 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-19T14:21:28.539 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set backup_fs allow_standby_replay false 2023-10-19T14:21:28.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.650+0000 7f5f7da64640 1 -- 172.21.15.57:0/3462003547 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5f780c96b0 msgr2=0x7f5f780c9b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:28.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.650+0000 7f5f7da64640 1 --2- 172.21.15.57:0/3462003547 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5f780c96b0 0x7f5f780c9b20 secure :-1 s=READY pgs=2691 cs=0 l=1 rev1=1 crypto rx=0x7f5f780cc590 tx=0x7f5f7002c250 comp rx=0 tx=0).stop 2023-10-19T14:21:28.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.650+0000 7f5f7da64640 1 -- 172.21.15.57:0/3462003547 shutdown_connections 2023-10-19T14:21:28.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.650+0000 7f5f7da64640 1 --2- 172.21.15.57:0/3462003547 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5f780c96b0 0x7f5f780c9b20 unknown :-1 s=CLOSED pgs=2691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:28.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.650+0000 7f5f7da64640 1 --2- 172.21.15.57:0/3462003547 >> v2:172.21.15.179:3300/0 conn(0x7f5f7813d350 0x7f5f780cbd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:28.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.650+0000 7f5f7da64640 1 --2- 172.21.15.57:0/3462003547 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5f780cb980 0x7f5f7813db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:28.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.650+0000 7f5f7da64640 1 -- 172.21.15.57:0/3462003547 >> 172.21.15.57:0/3462003547 conn(0x7f5f78069930 msgr2=0x7f5f78069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:28.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.650+0000 7f5f7da64640 1 -- 172.21.15.57:0/3462003547 shutdown_connections 2023-10-19T14:21:28.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.650+0000 7f5f7da64640 1 -- 172.21.15.57:0/3462003547 wait complete. 2023-10-19T14:21:28.658 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.651+0000 7f5f7da64640 1 Processor -- start 2023-10-19T14:21:28.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.652+0000 7f5f7da64640 1 -- start start 2023-10-19T14:21:28.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.652+0000 7f5f7da64640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5f780c96b0 0x7f5f781360e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:28.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.652+0000 7f5f7da64640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5f780cb980 0x7f5f78136870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:28.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.652+0000 7f5f7da64640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5f781589e0 0x7f5f780b0c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:28.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.652+0000 7f5f7da64640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f78141080 con 0x7f5f781589e0 2023-10-19T14:21:28.660 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.652+0000 7f5f7da64640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f78157920 con 0x7f5f780c96b0 2023-10-19T14:21:28.660 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.652+0000 7f5f7da64640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f78136db0 con 0x7f5f780cb980 2023-10-19T14:21:28.660 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.653+0000 7f5f77fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5f781589e0 0x7f5f780b0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:28.660 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.653+0000 7f5f77fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5f781589e0 0x7f5f780b0c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37162/0 (socket says 172.21.15.57:37162) 2023-10-19T14:21:28.660 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.653+0000 7f5f77fff640 1 -- 172.21.15.57:0/1082372558 learned_addr learned my addr 172.21.15.57:0/1082372558 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:28.660 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.653+0000 7f5f76ffd640 1 --2- 172.21.15.57:0/1082372558 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5f780cb980 0x7f5f78136870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:28.661 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.653+0000 7f5f777fe640 1 --2- 172.21.15.57:0/1082372558 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5f780c96b0 0x7f5f781360e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:28.661 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.653+0000 7f5f77fff640 1 -- 172.21.15.57:0/1082372558 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5f780cb980 msgr2=0x7f5f78136870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:28.661 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.653+0000 7f5f77fff640 1 --2- 172.21.15.57:0/1082372558 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5f780cb980 0x7f5f78136870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:28.661 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.653+0000 7f5f77fff640 1 -- 172.21.15.57:0/1082372558 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5f780c96b0 msgr2=0x7f5f781360e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:28.661 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.653+0000 7f5f77fff640 1 --2- 172.21.15.57:0/1082372558 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5f780c96b0 0x7f5f781360e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:28.662 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.653+0000 7f5f77fff640 1 -- 172.21.15.57:0/1082372558 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f70009900 con 0x7f5f781589e0 2023-10-19T14:21:28.662 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.653+0000 7f5f77fff640 1 --2- 172.21.15.57:0/1082372558 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5f781589e0 0x7f5f780b0c10 secure :-1 s=READY pgs=2692 cs=0 l=1 rev1=1 crypto rx=0x7f5f70037040 tx=0x7f5f70035db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:28.662 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.653+0000 7f5f5ffff640 1 -- 172.21.15.57:0/1082372558 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f70034440 con 0x7f5f781589e0 2023-10-19T14:21:28.662 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.653+0000 7f5f5ffff640 1 -- 172.21.15.57:0/1082372558 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f5f70004250 con 0x7f5f781589e0 2023-10-19T14:21:28.663 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.653+0000 7f5f5ffff640 1 -- 172.21.15.57:0/1082372558 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f70004640 con 0x7f5f781589e0 2023-10-19T14:21:28.663 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.653+0000 7f5f7da64640 1 -- 172.21.15.57:0/1082372558 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f7806c830 con 0x7f5f781589e0 2023-10-19T14:21:28.663 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.653+0000 7f5f7da64640 1 -- 172.21.15.57:0/1082372558 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f78140a30 con 0x7f5f781589e0 2023-10-19T14:21:28.663 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.654+0000 7f5f7da64640 1 -- 172.21.15.57:0/1082372558 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f780690e0 con 0x7f5f781589e0 2023-10-19T14:21:28.663 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.655+0000 7f5f5ffff640 1 -- 172.21.15.57:0/1082372558 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f5f7003b040 con 0x7f5f781589e0 2023-10-19T14:21:28.663 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.655+0000 7f5f5ffff640 1 --2- 172.21.15.57:0/1082372558 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f5f6406b080 0x7f5f6406d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:28.664 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.655+0000 7f5f5ffff640 1 -- 172.21.15.57:0/1082372558 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(247..247 src has 1..247) v4 ==== 6921+0+0 (secure 0 0 0) 0x7f5f700acaf0 con 0x7f5f781589e0 2023-10-19T14:21:28.664 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.656+0000 7f5f777fe640 1 --2- 172.21.15.57:0/1082372558 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f5f6406b080 0x7f5f6406d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:28.664 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.656+0000 7f5f777fe640 1 --2- 172.21.15.57:0/1082372558 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f5f6406b080 0x7f5f6406d540 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f5f68005910 tx=0x7f5f680058a0 comp rx=0 tx=0).ready entity=mgr.9530 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:28.665 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.660+0000 7f5f5ffff640 1 -- 172.21.15.57:0/1082372558 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f5f70077390 con 0x7f5f781589e0 2023-10-19T14:21:28.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:28.871+0000 7f5f7da64640 1 -- 172.21.15.57:0/1082372558 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "backup_fs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f5f7806ff80 con 0x7f5f781589e0 2023-10-19T14:21:29.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:29.522+0000 7f5f5ffff640 1 -- 172.21.15.57:0/1082372558 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "backup_fs", "var": "allow_standby_replay", "val": "false"}]=0 v107) v1 ==== 125+0+0 (secure 0 0 0) 0x7f5f7007a9b0 con 0x7f5f781589e0 2023-10-19T14:21:29.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:29.526+0000 7f5f7da64640 1 -- 172.21.15.57:0/1082372558 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f5f6406b080 msgr2=0x7f5f6406d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:29.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:29.526+0000 7f5f7da64640 1 --2- 172.21.15.57:0/1082372558 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f5f6406b080 0x7f5f6406d540 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f5f68005910 tx=0x7f5f680058a0 comp rx=0 tx=0).stop 2023-10-19T14:21:29.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:29.526+0000 7f5f7da64640 1 -- 172.21.15.57:0/1082372558 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5f781589e0 msgr2=0x7f5f780b0c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:29.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:29.526+0000 7f5f7da64640 1 --2- 172.21.15.57:0/1082372558 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5f781589e0 0x7f5f780b0c10 secure :-1 s=READY pgs=2692 cs=0 l=1 rev1=1 crypto rx=0x7f5f70037040 tx=0x7f5f70035db0 comp rx=0 tx=0).stop 2023-10-19T14:21:29.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:29.526+0000 7f5f7da64640 1 -- 172.21.15.57:0/1082372558 shutdown_connections 2023-10-19T14:21:29.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:29.526+0000 7f5f7da64640 1 --2- 172.21.15.57:0/1082372558 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f5f6406b080 0x7f5f6406d540 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:29.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:29.526+0000 7f5f7da64640 1 --2- 172.21.15.57:0/1082372558 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5f781589e0 0x7f5f780b0c10 unknown :-1 s=CLOSED pgs=2692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:29.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:29.526+0000 7f5f7da64640 1 --2- 172.21.15.57:0/1082372558 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5f780cb980 0x7f5f78136870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:29.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:29.526+0000 7f5f7da64640 1 --2- 172.21.15.57:0/1082372558 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5f780c96b0 0x7f5f781360e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:29.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:29.526+0000 7f5f7da64640 1 -- 172.21.15.57:0/1082372558 >> 172.21.15.57:0/1082372558 conn(0x7f5f78069930 msgr2=0x7f5f7806be60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:29.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:29.526+0000 7f5f7da64640 1 -- 172.21.15.57:0/1082372558 shutdown_connections 2023-10-19T14:21:29.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:29.526+0000 7f5f7da64640 1 -- 172.21.15.57:0/1082372558 wait complete. 2023-10-19T14:21:29.543 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'backup_fs' 2023-10-19T14:21:29.544 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T14:21:29.662 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:29.658+0000 7fa9cba56640 1 -- 172.21.15.57:0/2251144865 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa9c40cf340 msgr2=0x7fa9c40d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:29.662 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:29.658+0000 7fa9cba56640 1 --2- 172.21.15.57:0/2251144865 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa9c40cf340 0x7fa9c40d1910 secure :-1 s=READY pgs=2694 cs=0 l=1 rev1=1 crypto rx=0x7fa9c0009830 tx=0x7fa9c002bd60 comp rx=0 tx=0).stop 2023-10-19T14:21:29.662 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:29.658+0000 7fa9cba56640 1 -- 172.21.15.57:0/2251144865 shutdown_connections 2023-10-19T14:21:29.662 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:29.658+0000 7fa9cba56640 1 --2- 172.21.15.57:0/2251144865 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa9c40cf340 0x7fa9c40d1910 unknown :-1 s=CLOSED pgs=2694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:29.663 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:29.658+0000 7fa9cba56640 1 -- 172.21.15.57:0/2251144865 >> 172.21.15.57:0/2251144865 conn(0x7fa9c40d0d60 msgr2=0x7fa9c40d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:29.663 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:29.658+0000 7fa9cba56640 1 -- 172.21.15.57:0/2251144865 shutdown_connections 2023-10-19T14:21:29.663 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:29.659+0000 7fa9cba56640 1 -- 172.21.15.57:0/2251144865 wait complete. 2023-10-19T14:21:29.665 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:29.660+0000 7fa9cba56640 1 Processor -- start 2023-10-19T14:21:29.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:29.660+0000 7fa9cba56640 1 -- start start 2023-10-19T14:21:29.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:29.660+0000 7fa9cba56640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa9c40ceb40 0x7fa9c41573f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:29.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:29.660+0000 7fa9cba56640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa9c40cf340 0x7fa9c41f1420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:29.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:29.660+0000 7fa9cba56640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa9c415b2a0 0x7fa9c40c86d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:29.666 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:29.660+0000 7fa9cba56640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9c41f1d90 con 0x7fa9c415b2a0 2023-10-19T14:21:29.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:29.660+0000 7fa9cba56640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9c41f2070 con 0x7fa9c40cf340 2023-10-19T14:21:29.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:29.660+0000 7fa9cba56640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9c41f2350 con 0x7fa9c40ceb40 2023-10-19T14:21:29.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:29.661+0000 7fa9c9ff3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa9c415b2a0 0x7fa9c40c86d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:29.667 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:29.661+0000 7fa9c9ff3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa9c415b2a0 0x7fa9c40c86d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37188/0 (socket says 172.21.15.57:37188) 2023-10-19T14:21:29.669 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:29.661+0000 7fa9c9ff3640 1 -- 172.21.15.57:0/1105543773 learned_addr learned my addr 172.21.15.57:0/1105543773 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:29.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:29.661+0000 7fa9c8ff1640 1 --2- 172.21.15.57:0/1105543773 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa9c40cf340 0x7fa9c41f1420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:29.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:29.661+0000 7fa9c97f2640 1 --2- 172.21.15.57:0/1105543773 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa9c40ceb40 0x7fa9c41573f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:29.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:29.661+0000 7fa9c9ff3640 1 -- 172.21.15.57:0/1105543773 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa9c40ceb40 msgr2=0x7fa9c41573f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:29.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:29.661+0000 7fa9c9ff3640 1 --2- 172.21.15.57:0/1105543773 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa9c40ceb40 0x7fa9c41573f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:29.670 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:29.661+0000 7fa9c9ff3640 1 -- 172.21.15.57:0/1105543773 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa9c40cf340 msgr2=0x7fa9c41f1420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:29.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:29.661+0000 7fa9c9ff3640 1 --2- 172.21.15.57:0/1105543773 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa9c40cf340 0x7fa9c41f1420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:29.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:29.661+0000 7fa9c9ff3640 1 -- 172.21.15.57:0/1105543773 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9c0009490 con 0x7fa9c415b2a0 2023-10-19T14:21:29.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:29.661+0000 7fa9c97f2640 1 --2- 172.21.15.57:0/1105543773 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa9c40ceb40 0x7fa9c41573f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:21:29.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:29.661+0000 7fa9c9ff3640 1 --2- 172.21.15.57:0/1105543773 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa9c415b2a0 0x7fa9c40c86d0 secure :-1 s=READY pgs=2695 cs=0 l=1 rev1=1 crypto rx=0x7fa9bc00c7a0 tx=0x7fa9bc00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:29.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:29.661+0000 7fa9b9ffb640 1 -- 172.21.15.57:0/1105543773 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9bc003b70 con 0x7fa9c415b2a0 2023-10-19T14:21:29.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:29.661+0000 7fa9b9ffb640 1 -- 172.21.15.57:0/1105543773 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fa9bc003cf0 con 0x7fa9c415b2a0 2023-10-19T14:21:29.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:29.661+0000 7fa9b9ffb640 1 -- 172.21.15.57:0/1105543773 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9bc00aab0 con 0x7fa9c415b2a0 2023-10-19T14:21:29.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:29.661+0000 7fa9cba56640 1 -- 172.21.15.57:0/1105543773 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9c40c7ed0 con 0x7fa9c415b2a0 2023-10-19T14:21:29.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:29.662+0000 7fa9cba56640 1 -- 172.21.15.57:0/1105543773 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9c4157930 con 0x7fa9c415b2a0 2023-10-19T14:21:29.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:29.663+0000 7fa9cba56640 1 -- 172.21.15.57:0/1105543773 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9c40c83f0 con 0x7fa9c415b2a0 2023-10-19T14:21:29.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:29.663+0000 7fa9b9ffb640 1 -- 172.21.15.57:0/1105543773 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fa9bc00ac10 con 0x7fa9c415b2a0 2023-10-19T14:21:29.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:29.663+0000 7fa9b9ffb640 1 --2- 172.21.15.57:0/1105543773 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fa9a806b080 0x7fa9a806d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:29.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:29.663+0000 7fa9b9ffb640 1 -- 172.21.15.57:0/1105543773 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(247..247 src has 1..247) v4 ==== 6921+0+0 (secure 0 0 0) 0x7fa9bc089ec0 con 0x7fa9c415b2a0 2023-10-19T14:21:29.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:29.664+0000 7fa9c97f2640 1 --2- 172.21.15.57:0/1105543773 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fa9a806b080 0x7fa9a806d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:29.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:29.664+0000 7fa9c97f2640 1 --2- 172.21.15.57:0/1105543773 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fa9a806b080 0x7fa9a806d540 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa9c0036000 tx=0x7fa9c0004730 comp rx=0 tx=0).ready entity=mgr.9530 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:29.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:29.668+0000 7fa9b9ffb640 1 -- 172.21.15.57:0/1105543773 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa9c40c83f0 con 0x7fa9c415b2a0 2023-10-19T14:21:29.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:29.879+0000 7fa9cba56640 1 -- 172.21.15.57:0/1105543773 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa9c40c83f0 con 0x7fa9c415b2a0 2023-10-19T14:21:29.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:29.880+0000 7fa9b9ffb640 1 -- 172.21.15.57:0/1105543773 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+6140 (secure 0 0 0) 0x7fa9c40c83f0 con 0x7fa9c415b2a0 2023-10-19T14:21:29.883 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:21:29.883 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":107,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9761,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/325888554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":325888554},{"type":"v1","addr":"172.21.15.179:6836","nonce":325888554}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":100},{"gid":9764,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/8038232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":8038232},{"type":"v1","addr":"172.21.15.179:6837","nonce":8038232}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":100}],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T14:21:09.759935+0000","modified":"2023-10-19T14:21:11.768676+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":12564},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12564":{"gid":12564,"name":"c","rank":0,"incarnation":101,"state":"up:active","state_seq":3,"addr":"172.21.15.57:6835/18187246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":18187246},{"type":"v1","addr":"172.21.15.57:6835","nonce":18187246}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T14:21:26.724728+0000","modified":"2023-10-19T14:21:28.872856+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":12555},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12555":{"gid":12555,"name":"a","rank":0,"incarnation":105,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6833/4256498777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":4256498777},{"type":"v1","addr":"172.21.15.57:6833","nonce":4256498777}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2023-10-19T14:21:29.884 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 107 2023-10-19T14:21:29.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:29.882+0000 7fa9cba56640 1 -- 172.21.15.57:0/1105543773 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fa9a806b080 msgr2=0x7fa9a806d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:29.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:29.882+0000 7fa9cba56640 1 --2- 172.21.15.57:0/1105543773 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fa9a806b080 0x7fa9a806d540 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa9c0036000 tx=0x7fa9c0004730 comp rx=0 tx=0).stop 2023-10-19T14:21:29.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:29.882+0000 7fa9cba56640 1 -- 172.21.15.57:0/1105543773 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa9c415b2a0 msgr2=0x7fa9c40c86d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:29.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:29.882+0000 7fa9cba56640 1 --2- 172.21.15.57:0/1105543773 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa9c415b2a0 0x7fa9c40c86d0 secure :-1 s=READY pgs=2695 cs=0 l=1 rev1=1 crypto rx=0x7fa9bc00c7a0 tx=0x7fa9bc00cc60 comp rx=0 tx=0).stop 2023-10-19T14:21:29.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:29.882+0000 7fa9cba56640 1 -- 172.21.15.57:0/1105543773 shutdown_connections 2023-10-19T14:21:29.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:29.882+0000 7fa9cba56640 1 --2- 172.21.15.57:0/1105543773 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fa9a806b080 0x7fa9a806d540 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:29.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:29.882+0000 7fa9cba56640 1 --2- 172.21.15.57:0/1105543773 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa9c415b2a0 0x7fa9c40c86d0 unknown :-1 s=CLOSED pgs=2695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:29.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:29.882+0000 7fa9cba56640 1 --2- 172.21.15.57:0/1105543773 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa9c40cf340 0x7fa9c41f1420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:29.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:29.882+0000 7fa9cba56640 1 --2- 172.21.15.57:0/1105543773 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa9c40ceb40 0x7fa9c41573f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:29.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:29.882+0000 7fa9cba56640 1 -- 172.21.15.57:0/1105543773 >> 172.21.15.57:0/1105543773 conn(0x7fa9c40d0d60 msgr2=0x7fa9c406b700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:29.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:29.882+0000 7fa9cba56640 1 -- 172.21.15.57:0/1105543773 shutdown_connections 2023-10-19T14:21:29.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:29.882+0000 7fa9cba56640 1 -- 172.21.15.57:0/1105543773 wait complete. 2023-10-19T14:21:29.895 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T14:21:30.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.002+0000 7fc659fc0640 1 -- 172.21.15.57:0/1444504588 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc65406ae40 msgr2=0x7fc654157dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:30.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.002+0000 7fc659fc0640 1 --2- 172.21.15.57:0/1444504588 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc65406ae40 0x7fc654157dd0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fc64c009830 tx=0x7fc64c02bd60 comp rx=0 tx=0).stop 2023-10-19T14:21:30.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.002+0000 7fc659fc0640 1 -- 172.21.15.57:0/1444504588 shutdown_connections 2023-10-19T14:21:30.006 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.002+0000 7fc659fc0640 1 --2- 172.21.15.57:0/1444504588 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc65406ae40 0x7fc654157dd0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:30.006 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.002+0000 7fc659fc0640 1 --2- 172.21.15.57:0/1444504588 >> v2:172.21.15.179:3300/0 conn(0x7fc6541579f0 0x7fc6540b0e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:30.006 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.002+0000 7fc659fc0640 1 -- 172.21.15.57:0/1444504588 >> 172.21.15.57:0/1444504588 conn(0x7fc65406fae0 msgr2=0x7fc65406fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:30.006 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.002+0000 7fc659fc0640 1 -- 172.21.15.57:0/1444504588 shutdown_connections 2023-10-19T14:21:30.006 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.002+0000 7fc659fc0640 1 -- 172.21.15.57:0/1444504588 wait complete. 2023-10-19T14:21:30.008 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.003+0000 7fc659fc0640 1 Processor -- start 2023-10-19T14:21:30.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.003+0000 7fc659fc0640 1 -- start start 2023-10-19T14:21:30.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.004+0000 7fc659fc0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc65406ae40 0x7fc65413bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:30.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.004+0000 7fc659fc0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc6541579f0 0x7fc6541517b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:30.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.004+0000 7fc659fc0640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc65415c1f0 0x7fc654151cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:30.010 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.004+0000 7fc659fc0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc65414f040 con 0x7fc6541579f0 2023-10-19T14:21:30.010 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.004+0000 7fc659fc0640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc65413c250 con 0x7fc65406ae40 2023-10-19T14:21:30.010 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.004+0000 7fc659fc0640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc65414be50 con 0x7fc65415c1f0 2023-10-19T14:21:30.010 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.004+0000 7fc652ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc6541579f0 0x7fc6541517b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:30.010 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.004+0000 7fc652ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc6541579f0 0x7fc6541517b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37204/0 (socket says 172.21.15.57:37204) 2023-10-19T14:21:30.011 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.004+0000 7fc652ffd640 1 -- 172.21.15.57:0/154383686 learned_addr learned my addr 172.21.15.57:0/154383686 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:30.011 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.004+0000 7fc6537fe640 1 --2- 172.21.15.57:0/154383686 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc65406ae40 0x7fc65413bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:30.011 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.004+0000 7fc653fff640 1 --2- 172.21.15.57:0/154383686 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc65415c1f0 0x7fc654151cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:30.011 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.004+0000 7fc652ffd640 1 -- 172.21.15.57:0/154383686 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc65415c1f0 msgr2=0x7fc654151cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:30.011 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.004+0000 7fc652ffd640 1 --2- 172.21.15.57:0/154383686 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc65415c1f0 0x7fc654151cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:30.012 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.004+0000 7fc652ffd640 1 -- 172.21.15.57:0/154383686 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc65406ae40 msgr2=0x7fc65413bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:30.012 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.004+0000 7fc652ffd640 1 --2- 172.21.15.57:0/154383686 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc65406ae40 0x7fc65413bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:30.012 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.004+0000 7fc652ffd640 1 -- 172.21.15.57:0/154383686 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc64c009490 con 0x7fc6541579f0 2023-10-19T14:21:30.012 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.005+0000 7fc652ffd640 1 --2- 172.21.15.57:0/154383686 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc6541579f0 0x7fc6541517b0 secure :-1 s=READY pgs=2696 cs=0 l=1 rev1=1 crypto rx=0x7fc64800c430 tx=0x7fc64800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:30.012 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.005+0000 7fc63bfff640 1 -- 172.21.15.57:0/154383686 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6480052d0 con 0x7fc6541579f0 2023-10-19T14:21:30.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.005+0000 7fc659fc0640 1 -- 172.21.15.57:0/154383686 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6541567a0 con 0x7fc6541579f0 2023-10-19T14:21:30.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.005+0000 7fc63bfff640 1 -- 172.21.15.57:0/154383686 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fc6480035f0 con 0x7fc6541579f0 2023-10-19T14:21:30.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.005+0000 7fc63bfff640 1 -- 172.21.15.57:0/154383686 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc648003a00 con 0x7fc6541579f0 2023-10-19T14:21:30.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.005+0000 7fc659fc0640 1 -- 172.21.15.57:0/154383686 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc65414c0a0 con 0x7fc6541579f0 2023-10-19T14:21:30.013 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.006+0000 7fc659fc0640 1 -- 172.21.15.57:0/154383686 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6540690e0 con 0x7fc6541579f0 2023-10-19T14:21:30.014 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.006+0000 7fc63bfff640 1 -- 172.21.15.57:0/154383686 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fc648003b60 con 0x7fc6541579f0 2023-10-19T14:21:30.014 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.007+0000 7fc63bfff640 1 --2- 172.21.15.57:0/154383686 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fc64006b080 0x7fc64006d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:30.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.007+0000 7fc63bfff640 1 -- 172.21.15.57:0/154383686 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(247..247 src has 1..247) v4 ==== 6921+0+0 (secure 0 0 0) 0x7fc648089600 con 0x7fc6541579f0 2023-10-19T14:21:30.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.007+0000 7fc6537fe640 1 --2- 172.21.15.57:0/154383686 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fc64006b080 0x7fc64006d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:30.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.007+0000 7fc6537fe640 1 --2- 172.21.15.57:0/154383686 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fc64006b080 0x7fc64006d540 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc644005fd0 tx=0x7fc644010040 comp rx=0 tx=0).ready entity=mgr.9530 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:30.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.011+0000 7fc63bfff640 1 -- 172.21.15.57:0/154383686 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc648054020 con 0x7fc6541579f0 2023-10-19T14:21:30.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.188+0000 7fc659fc0640 1 -- 172.21.15.57:0/154383686 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc65406a8d0 con 0x7fc6541579f0 2023-10-19T14:21:30.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.189+0000 7fc63bfff640 1 -- 172.21.15.57:0/154383686 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v247) v1 ==== 74+0+20499 (secure 0 0 0) 0x7fc648057640 con 0x7fc6541579f0 2023-10-19T14:21:30.192 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:21:30.193 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":247,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T14:21:28.518288+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2023-10-19T14:21:08.101023+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"242","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2023-10-19T14:21:08.745817+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"243","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_metadata","create_time":"2023-10-19T14:21:24.384693+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"246","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"backup_fs_data","create_time":"2023-10-19T14:21:25.717524+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"247","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T14:21:30.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.191+0000 7fc659fc0640 1 -- 172.21.15.57:0/154383686 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fc64006b080 msgr2=0x7fc64006d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:30.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.191+0000 7fc659fc0640 1 --2- 172.21.15.57:0/154383686 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fc64006b080 0x7fc64006d540 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc644005fd0 tx=0x7fc644010040 comp rx=0 tx=0).stop 2023-10-19T14:21:30.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.191+0000 7fc659fc0640 1 -- 172.21.15.57:0/154383686 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc6541579f0 msgr2=0x7fc6541517b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:30.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.191+0000 7fc659fc0640 1 --2- 172.21.15.57:0/154383686 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc6541579f0 0x7fc6541517b0 secure :-1 s=READY pgs=2696 cs=0 l=1 rev1=1 crypto rx=0x7fc64800c430 tx=0x7fc64800c8f0 comp rx=0 tx=0).stop 2023-10-19T14:21:30.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.191+0000 7fc659fc0640 1 -- 172.21.15.57:0/154383686 shutdown_connections 2023-10-19T14:21:30.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.191+0000 7fc659fc0640 1 --2- 172.21.15.57:0/154383686 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fc64006b080 0x7fc64006d540 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:30.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.191+0000 7fc659fc0640 1 --2- 172.21.15.57:0/154383686 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc65415c1f0 0x7fc654151cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:30.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.191+0000 7fc659fc0640 1 --2- 172.21.15.57:0/154383686 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc6541579f0 0x7fc6541517b0 unknown :-1 s=CLOSED pgs=2696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:30.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.191+0000 7fc659fc0640 1 --2- 172.21.15.57:0/154383686 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc65406ae40 0x7fc65413bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:30.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.191+0000 7fc659fc0640 1 -- 172.21.15.57:0/154383686 >> 172.21.15.57:0/154383686 conn(0x7fc65406fae0 msgr2=0x7fc65406c210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:30.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.192+0000 7fc659fc0640 1 -- 172.21.15.57:0/154383686 shutdown_connections 2023-10-19T14:21:30.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.192+0000 7fc659fc0640 1 -- 172.21.15.57:0/154383686 wait complete. 2023-10-19T14:21:30.205 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-19T14:21:30.206 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-19T14:21:30.315 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.312+0000 7fbaa24c9640 1 -- 172.21.15.57:0/1333048533 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fba9c06f0a0 msgr2=0x7fba9c06ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:30.316 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.312+0000 7fbaa24c9640 1 --2- 172.21.15.57:0/1333048533 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fba9c06f0a0 0x7fba9c06ddc0 secure :-1 s=READY pgs=2698 cs=0 l=1 rev1=1 crypto rx=0x7fba8c009830 tx=0x7fba8c02bd60 comp rx=0 tx=0).stop 2023-10-19T14:21:30.316 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.312+0000 7fbaa24c9640 1 -- 172.21.15.57:0/1333048533 shutdown_connections 2023-10-19T14:21:30.316 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.312+0000 7fbaa24c9640 1 --2- 172.21.15.57:0/1333048533 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fba9c06f0a0 0x7fba9c06ddc0 unknown :-1 s=CLOSED pgs=2698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:30.316 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.312+0000 7fbaa24c9640 1 -- 172.21.15.57:0/1333048533 >> 172.21.15.57:0/1333048533 conn(0x7fba9c0b0670 msgr2=0x7fba9c0b0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:30.316 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.312+0000 7fbaa24c9640 1 -- 172.21.15.57:0/1333048533 shutdown_connections 2023-10-19T14:21:30.317 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.312+0000 7fbaa24c9640 1 -- 172.21.15.57:0/1333048533 wait complete. 2023-10-19T14:21:30.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.313+0000 7fbaa24c9640 1 Processor -- start 2023-10-19T14:21:30.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.313+0000 7fbaa24c9640 1 -- start start 2023-10-19T14:21:30.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.314+0000 7fbaa24c9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fba9c06d2d0 0x7fba9c158780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:30.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.314+0000 7fbaa24c9640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fba9c06f0a0 0x7fba9c098ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:30.320 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.314+0000 7fbaa24c9640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fba9c0ad370 0x7fba9c0c7170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:30.320 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.314+0000 7fbaa24c9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba9c0cad30 con 0x7fba9c06d2d0 2023-10-19T14:21:30.320 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.314+0000 7fbaa24c9640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba9c0caff0 con 0x7fba9c06f0a0 2023-10-19T14:21:30.320 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.314+0000 7fbaa24c9640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba9c0cb2d0 con 0x7fba9c0ad370 2023-10-19T14:21:30.320 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.314+0000 7fba9bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fba9c06d2d0 0x7fba9c158780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:30.321 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.314+0000 7fba9bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fba9c06d2d0 0x7fba9c158780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37222/0 (socket says 172.21.15.57:37222) 2023-10-19T14:21:30.321 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.314+0000 7fba9bfff640 1 -- 172.21.15.57:0/1600840514 learned_addr learned my addr 172.21.15.57:0/1600840514 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:30.321 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.314+0000 7fbaa0a66640 1 --2- 172.21.15.57:0/1600840514 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fba9c0ad370 0x7fba9c0c7170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:30.321 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.314+0000 7fba9b7fe640 1 --2- 172.21.15.57:0/1600840514 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fba9c06f0a0 0x7fba9c098ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:30.321 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.315+0000 7fba9bfff640 1 -- 172.21.15.57:0/1600840514 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fba9c0ad370 msgr2=0x7fba9c0c7170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:30.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.315+0000 7fba9bfff640 1 --2- 172.21.15.57:0/1600840514 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fba9c0ad370 0x7fba9c0c7170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:30.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.315+0000 7fba9bfff640 1 -- 172.21.15.57:0/1600840514 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fba9c06f0a0 msgr2=0x7fba9c098ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:30.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.315+0000 7fba9bfff640 1 --2- 172.21.15.57:0/1600840514 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fba9c06f0a0 0x7fba9c098ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:30.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.315+0000 7fba9bfff640 1 -- 172.21.15.57:0/1600840514 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba8c009490 con 0x7fba9c06d2d0 2023-10-19T14:21:30.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.315+0000 7fba9b7fe640 1 --2- 172.21.15.57:0/1600840514 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fba9c06f0a0 0x7fba9c098ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:21:30.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.315+0000 7fba9bfff640 1 --2- 172.21.15.57:0/1600840514 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fba9c06d2d0 0x7fba9c158780 secure :-1 s=READY pgs=2699 cs=0 l=1 rev1=1 crypto rx=0x7fba9000c400 tx=0x7fba9000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:30.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.315+0000 7fba98ff9640 1 -- 172.21.15.57:0/1600840514 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba90004230 con 0x7fba9c06d2d0 2023-10-19T14:21:30.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.315+0000 7fba98ff9640 1 -- 172.21.15.57:0/1600840514 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fba900044a0 con 0x7fba9c06d2d0 2023-10-19T14:21:30.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.315+0000 7fbaa24c9640 1 -- 172.21.15.57:0/1600840514 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba9c0cb5b0 con 0x7fba9c06d2d0 2023-10-19T14:21:30.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.315+0000 7fba98ff9640 1 -- 172.21.15.57:0/1600840514 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba90003850 con 0x7fba9c06d2d0 2023-10-19T14:21:30.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.315+0000 7fbaa24c9640 1 -- 172.21.15.57:0/1600840514 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba9c158cc0 con 0x7fba9c06d2d0 2023-10-19T14:21:30.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.316+0000 7fbaa24c9640 1 -- 172.21.15.57:0/1600840514 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba9c0ce1d0 con 0x7fba9c06d2d0 2023-10-19T14:21:30.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.317+0000 7fba98ff9640 1 -- 172.21.15.57:0/1600840514 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fba900039b0 con 0x7fba9c06d2d0 2023-10-19T14:21:30.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.317+0000 7fba98ff9640 1 --2- 172.21.15.57:0/1600840514 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fba8806b080 0x7fba8806d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:30.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.317+0000 7fba98ff9640 1 -- 172.21.15.57:0/1600840514 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(247..247 src has 1..247) v4 ==== 6921+0+0 (secure 0 0 0) 0x7fba90089510 con 0x7fba9c06d2d0 2023-10-19T14:21:30.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.317+0000 7fba9b7fe640 1 --2- 172.21.15.57:0/1600840514 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fba8806b080 0x7fba8806d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:30.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.318+0000 7fba9b7fe640 1 --2- 172.21.15.57:0/1600840514 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fba8806b080 0x7fba8806d540 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fba8c036000 tx=0x7fba8c004730 comp rx=0 tx=0).ready entity=mgr.9530 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:30.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.321+0000 7fba98ff9640 1 -- 172.21.15.57:0/1600840514 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fba90053fb0 con 0x7fba9c06d2d0 2023-10-19T14:21:30.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.493+0000 7fbaa24c9640 1 -- 172.21.15.57:0/1600840514 --> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fba9c0c2630 con 0x7fba8806b080 2023-10-19T14:21:30.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.507+0000 7fba98ff9640 1 -- 172.21.15.57:0/1600840514 <== mgr.9530 v2:172.21.15.179:6832/3547895240 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+529049 (secure 0 0 0) 0x7fba8c004260 con 0x7fba8806b080 2023-10-19T14:21:30.515 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:21:30.519 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2023-10-19T14:21:30.520 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.511+0000 7fbaa24c9640 1 -- 172.21.15.57:0/1600840514 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fba8806b080 msgr2=0x7fba8806d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:30.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.511+0000 7fbaa24c9640 1 --2- 172.21.15.57:0/1600840514 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fba8806b080 0x7fba8806d540 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fba8c036000 tx=0x7fba8c004730 comp rx=0 tx=0).stop 2023-10-19T14:21:30.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.511+0000 7fbaa24c9640 1 -- 172.21.15.57:0/1600840514 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fba9c06d2d0 msgr2=0x7fba9c158780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:30.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.511+0000 7fbaa24c9640 1 --2- 172.21.15.57:0/1600840514 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fba9c06d2d0 0x7fba9c158780 secure :-1 s=READY pgs=2699 cs=0 l=1 rev1=1 crypto rx=0x7fba9000c400 tx=0x7fba9000c8c0 comp rx=0 tx=0).stop 2023-10-19T14:21:30.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.511+0000 7fbaa24c9640 1 -- 172.21.15.57:0/1600840514 shutdown_connections 2023-10-19T14:21:30.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.511+0000 7fbaa24c9640 1 --2- 172.21.15.57:0/1600840514 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fba8806b080 0x7fba8806d540 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:30.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.511+0000 7fbaa24c9640 1 --2- 172.21.15.57:0/1600840514 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fba9c0ad370 0x7fba9c0c7170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:30.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.511+0000 7fbaa24c9640 1 --2- 172.21.15.57:0/1600840514 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fba9c06f0a0 0x7fba9c098ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:30.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.511+0000 7fbaa24c9640 1 --2- 172.21.15.57:0/1600840514 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fba9c06d2d0 0x7fba9c158780 unknown :-1 s=CLOSED pgs=2699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:30.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.511+0000 7fbaa24c9640 1 -- 172.21.15.57:0/1600840514 >> 172.21.15.57:0/1600840514 conn(0x7fba9c0b0670 msgr2=0x7fba9c0b0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:30.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.511+0000 7fbaa24c9640 1 -- 172.21.15.57:0/1600840514 shutdown_connections 2023-10-19T14:21:30.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.511+0000 7fbaa24c9640 1 -- 172.21.15.57:0/1600840514 wait complete. 2023-10-19T14:21:30.525 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":42,"stamp":"2023-10-19T14:21:30.176706+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596444,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":2402,"num_read_kb":5036,"num_write":721,"num_write_kb":2447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3288,"ondisk_log_size":3288,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":347,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":451388,"kb_used_data":25592,"kb_used_omap":12,"kb_used_meta":425651,"kb_avail":754523332,"statfs":{"total":773094113280,"available":772631891968,"internally_reserved":0,"allocated":26206208,"data_stored":14033514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":435867193},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3225,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":46,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.828890"},"pg_stats":[{"pgid":"34.1d","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.590689+0000","last_change":"2023-10-19T14:21:27.533913+0000","last_active":"2023-10-19T14:21:28.590689+0000","last_peered":"2023-10-19T14:21:28.590689+0000","last_clean":"2023-10-19T14:21:28.590689+0000","last_became_active":"2023-10-19T14:21:27.533603+0000","last_became_peered":"2023-10-19T14:21:27.533603+0000","last_unstale":"2023-10-19T14:21:28.590689+0000","last_undegraded":"2023-10-19T14:21:28.590689+0000","last_fullsized":"2023-10-19T14:21:28.590689+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:45:59.620638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1e","version":"246'1","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.592522+0000","last_change":"2023-10-19T14:21:26.366411+0000","last_active":"2023-10-19T14:21:28.592522+0000","last_peered":"2023-10-19T14:21:28.592522+0000","last_clean":"2023-10-19T14:21:28.592522+0000","last_became_active":"2023-10-19T14:21:26.366143+0000","last_became_peered":"2023-10-19T14:21:26.366143+0000","last_unstale":"2023-10-19T14:21:28.592522+0000","last_undegraded":"2023-10-19T14:21:28.592522+0000","last_fullsized":"2023-10-19T14:21:28.592522+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_clean_scrub_stamp":"2023-10-19T14:21:25.343524+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:11:36.819564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1f","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.515056+0000","last_change":"2023-10-19T14:21:10.436485+0000","last_active":"2023-10-19T14:21:27.515056+0000","last_peered":"2023-10-19T14:21:27.515056+0000","last_clean":"2023-10-19T14:21:27.515056+0000","last_became_active":"2023-10-19T14:21:10.436174+0000","last_became_peered":"2023-10-19T14:21:10.436174+0000","last_unstale":"2023-10-19T14:21:27.515056+0000","last_undegraded":"2023-10-19T14:21:27.515056+0000","last_fullsized":"2023-10-19T14:21:27.515056+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:01:21.155069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.1c","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.591906+0000","last_change":"2023-10-19T14:21:27.525352+0000","last_active":"2023-10-19T14:21:28.591906+0000","last_peered":"2023-10-19T14:21:28.591906+0000","last_clean":"2023-10-19T14:21:28.591906+0000","last_became_active":"2023-10-19T14:21:27.525095+0000","last_became_peered":"2023-10-19T14:21:27.525095+0000","last_unstale":"2023-10-19T14:21:28.591906+0000","last_undegraded":"2023-10-19T14:21:28.591906+0000","last_fullsized":"2023-10-19T14:21:28.591906+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:24:33.138924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.1f","version":"246'4","reported_seq":16,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.591196+0000","last_change":"2023-10-19T14:21:26.372524+0000","last_active":"2023-10-19T14:21:28.591196+0000","last_peered":"2023-10-19T14:21:28.591196+0000","last_clean":"2023-10-19T14:21:28.591196+0000","last_became_active":"2023-10-19T14:21:26.372245+0000","last_became_peered":"2023-10-19T14:21:26.372245+0000","last_unstale":"2023-10-19T14:21:28.591196+0000","last_undegraded":"2023-10-19T14:21:28.591196+0000","last_fullsized":"2023-10-19T14:21:28.591196+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_clean_scrub_stamp":"2023-10-19T14:21:25.343524+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:02:19.626255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1e","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.523249+0000","last_change":"2023-10-19T14:21:10.439371+0000","last_active":"2023-10-19T14:21:27.523249+0000","last_peered":"2023-10-19T14:21:27.523249+0000","last_clean":"2023-10-19T14:21:27.523249+0000","last_became_active":"2023-10-19T14:21:10.439072+0000","last_became_peered":"2023-10-19T14:21:10.439072+0000","last_unstale":"2023-10-19T14:21:27.523249+0000","last_undegraded":"2023-10-19T14:21:27.523249+0000","last_fullsized":"2023-10-19T14:21:27.523249+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:45:21.552160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.1f","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.525255+0000","last_change":"2023-10-19T14:21:27.533201+0000","last_active":"2023-10-19T14:21:28.525255+0000","last_peered":"2023-10-19T14:21:28.525255+0000","last_clean":"2023-10-19T14:21:28.525255+0000","last_became_active":"2023-10-19T14:21:27.532933+0000","last_became_peered":"2023-10-19T14:21:27.532933+0000","last_unstale":"2023-10-19T14:21:28.525255+0000","last_undegraded":"2023-10-19T14:21:28.525255+0000","last_fullsized":"2023-10-19T14:21:28.525255+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:03:43.253629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1c","version":"246'2","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.591688+0000","last_change":"2023-10-19T14:21:26.369943+0000","last_active":"2023-10-19T14:21:28.591688+0000","last_peered":"2023-10-19T14:21:28.591688+0000","last_clean":"2023-10-19T14:21:28.591688+0000","last_became_active":"2023-10-19T14:21:26.369675+0000","last_became_peered":"2023-10-19T14:21:26.369675+0000","last_unstale":"2023-10-19T14:21:28.591688+0000","last_undegraded":"2023-10-19T14:21:28.591688+0000","last_fullsized":"2023-10-19T14:21:28.591688+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_clean_scrub_stamp":"2023-10-19T14:21:25.343524+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:33:06.312590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1d","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.511874+0000","last_change":"2023-10-19T14:21:10.422867+0000","last_active":"2023-10-19T14:21:27.511874+0000","last_peered":"2023-10-19T14:21:27.511874+0000","last_clean":"2023-10-19T14:21:27.511874+0000","last_became_active":"2023-10-19T14:21:10.422558+0000","last_became_peered":"2023-10-19T14:21:10.422558+0000","last_unstale":"2023-10-19T14:21:27.511874+0000","last_undegraded":"2023-10-19T14:21:27.511874+0000","last_fullsized":"2023-10-19T14:21:27.511874+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:46:57.540004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.1e","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.591126+0000","last_change":"2023-10-19T14:21:27.540496+0000","last_active":"2023-10-19T14:21:28.591126+0000","last_peered":"2023-10-19T14:21:28.591126+0000","last_clean":"2023-10-19T14:21:28.591126+0000","last_became_active":"2023-10-19T14:21:27.540219+0000","last_became_peered":"2023-10-19T14:21:27.540219+0000","last_unstale":"2023-10-19T14:21:28.591126+0000","last_undegraded":"2023-10-19T14:21:28.591126+0000","last_fullsized":"2023-10-19T14:21:28.591126+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:55:48.870859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1d","version":"246'2","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.592308+0000","last_change":"2023-10-19T14:21:26.362677+0000","last_active":"2023-10-19T14:21:28.592308+0000","last_peered":"2023-10-19T14:21:28.592308+0000","last_clean":"2023-10-19T14:21:28.592308+0000","last_became_active":"2023-10-19T14:21:26.362337+0000","last_became_peered":"2023-10-19T14:21:26.362337+0000","last_unstale":"2023-10-19T14:21:28.592308+0000","last_undegraded":"2023-10-19T14:21:28.592308+0000","last_fullsized":"2023-10-19T14:21:28.592308+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_clean_scrub_stamp":"2023-10-19T14:21:25.343524+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:47:27.368054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.1c","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.511719+0000","last_change":"2023-10-19T14:21:10.426687+0000","last_active":"2023-10-19T14:21:27.511719+0000","last_peered":"2023-10-19T14:21:27.511719+0000","last_clean":"2023-10-19T14:21:27.511719+0000","last_became_active":"2023-10-19T14:21:10.426422+0000","last_became_peered":"2023-10-19T14:21:10.426422+0000","last_unstale":"2023-10-19T14:21:27.511719+0000","last_undegraded":"2023-10-19T14:21:27.511719+0000","last_fullsized":"2023-10-19T14:21:27.511719+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:13:29.364169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.19","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.525000+0000","last_change":"2023-10-19T14:21:27.537576+0000","last_active":"2023-10-19T14:21:28.525000+0000","last_peered":"2023-10-19T14:21:28.525000+0000","last_clean":"2023-10-19T14:21:28.525000+0000","last_became_active":"2023-10-19T14:21:27.537268+0000","last_became_peered":"2023-10-19T14:21:27.537268+0000","last_unstale":"2023-10-19T14:21:28.525000+0000","last_undegraded":"2023-10-19T14:21:28.525000+0000","last_fullsized":"2023-10-19T14:21:28.525000+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T02:10:36.623775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1a","version":"246'1","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.524632+0000","last_change":"2023-10-19T14:21:26.376713+0000","last_active":"2023-10-19T14:21:28.524632+0000","last_peered":"2023-10-19T14:21:28.524632+0000","last_clean":"2023-10-19T14:21:28.524632+0000","last_became_active":"2023-10-19T14:21:26.376424+0000","last_became_peered":"2023-10-19T14:21:26.376424+0000","last_unstale":"2023-10-19T14:21:28.524632+0000","last_undegraded":"2023-10-19T14:21:28.524632+0000","last_fullsized":"2023-10-19T14:21:28.524632+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_clean_scrub_stamp":"2023-10-19T14:21:25.343524+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:42:18.892113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.1b","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.519477+0000","last_change":"2023-10-19T14:21:10.427647+0000","last_active":"2023-10-19T14:21:27.519477+0000","last_peered":"2023-10-19T14:21:27.519477+0000","last_clean":"2023-10-19T14:21:27.519477+0000","last_became_active":"2023-10-19T14:21:10.427412+0000","last_became_peered":"2023-10-19T14:21:10.427412+0000","last_unstale":"2023-10-19T14:21:27.519477+0000","last_undegraded":"2023-10-19T14:21:27.519477+0000","last_fullsized":"2023-10-19T14:21:27.519477+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T02:17:31.253940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.18","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.524262+0000","last_change":"2023-10-19T14:21:27.528624+0000","last_active":"2023-10-19T14:21:28.524262+0000","last_peered":"2023-10-19T14:21:28.524262+0000","last_clean":"2023-10-19T14:21:28.524262+0000","last_became_active":"2023-10-19T14:21:27.528220+0000","last_became_peered":"2023-10-19T14:21:27.528220+0000","last_unstale":"2023-10-19T14:21:28.524262+0000","last_undegraded":"2023-10-19T14:21:28.524262+0000","last_fullsized":"2023-10-19T14:21:28.524262+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:03:14.837836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1b","version":"246'1","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.590799+0000","last_change":"2023-10-19T14:21:26.364898+0000","last_active":"2023-10-19T14:21:28.590799+0000","last_peered":"2023-10-19T14:21:28.590799+0000","last_clean":"2023-10-19T14:21:28.590799+0000","last_became_active":"2023-10-19T14:21:26.364604+0000","last_became_peered":"2023-10-19T14:21:26.364604+0000","last_unstale":"2023-10-19T14:21:28.590799+0000","last_undegraded":"2023-10-19T14:21:28.590799+0000","last_fullsized":"2023-10-19T14:21:28.590799+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_clean_scrub_stamp":"2023-10-19T14:21:25.343524+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:04:46.470039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1a","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.515605+0000","last_change":"2023-10-19T14:21:10.435192+0000","last_active":"2023-10-19T14:21:27.515605+0000","last_peered":"2023-10-19T14:21:27.515605+0000","last_clean":"2023-10-19T14:21:27.515605+0000","last_became_active":"2023-10-19T14:21:10.434868+0000","last_became_peered":"2023-10-19T14:21:10.434868+0000","last_unstale":"2023-10-19T14:21:27.515605+0000","last_undegraded":"2023-10-19T14:21:27.515605+0000","last_fullsized":"2023-10-19T14:21:27.515605+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:40:01.514130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.1b","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.526932+0000","last_change":"2023-10-19T14:21:27.538695+0000","last_active":"2023-10-19T14:21:28.526932+0000","last_peered":"2023-10-19T14:21:28.526932+0000","last_clean":"2023-10-19T14:21:28.526932+0000","last_became_active":"2023-10-19T14:21:27.537546+0000","last_became_peered":"2023-10-19T14:21:27.537546+0000","last_unstale":"2023-10-19T14:21:28.526932+0000","last_undegraded":"2023-10-19T14:21:28.526932+0000","last_fullsized":"2023-10-19T14:21:28.526932+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:44:00.953524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.18","version":"0'0","reported_seq":12,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.589875+0000","last_change":"2023-10-19T14:21:26.365537+0000","last_active":"2023-10-19T14:21:28.589875+0000","last_peered":"2023-10-19T14:21:28.589875+0000","last_clean":"2023-10-19T14:21:28.589875+0000","last_became_active":"2023-10-19T14:21:26.365272+0000","last_became_peered":"2023-10-19T14:21:26.365272+0000","last_unstale":"2023-10-19T14:21:28.589875+0000","last_undegraded":"2023-10-19T14:21:28.589875+0000","last_fullsized":"2023-10-19T14:21:28.589875+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_clean_scrub_stamp":"2023-10-19T14:21:25.343524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:32:28.482159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.19","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.512713+0000","last_change":"2023-10-19T14:21:10.431422+0000","last_active":"2023-10-19T14:21:27.512713+0000","last_peered":"2023-10-19T14:21:27.512713+0000","last_clean":"2023-10-19T14:21:27.512713+0000","last_became_active":"2023-10-19T14:21:10.431131+0000","last_became_peered":"2023-10-19T14:21:10.431131+0000","last_unstale":"2023-10-19T14:21:27.512713+0000","last_undegraded":"2023-10-19T14:21:27.512713+0000","last_fullsized":"2023-10-19T14:21:27.512713+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:14:16.111979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.1a","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.527155+0000","last_change":"2023-10-19T14:21:27.538410+0000","last_active":"2023-10-19T14:21:28.527155+0000","last_peered":"2023-10-19T14:21:28.527155+0000","last_clean":"2023-10-19T14:21:28.527155+0000","last_became_active":"2023-10-19T14:21:27.538183+0000","last_became_peered":"2023-10-19T14:21:27.538183+0000","last_unstale":"2023-10-19T14:21:28.527155+0000","last_undegraded":"2023-10-19T14:21:28.527155+0000","last_fullsized":"2023-10-19T14:21:28.527155+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:17:12.731105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.19","version":"0'0","reported_seq":12,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.591540+0000","last_change":"2023-10-19T14:21:26.365920+0000","last_active":"2023-10-19T14:21:28.591540+0000","last_peered":"2023-10-19T14:21:28.591540+0000","last_clean":"2023-10-19T14:21:28.591540+0000","last_became_active":"2023-10-19T14:21:26.365644+0000","last_became_peered":"2023-10-19T14:21:26.365644+0000","last_unstale":"2023-10-19T14:21:28.591540+0000","last_undegraded":"2023-10-19T14:21:28.591540+0000","last_fullsized":"2023-10-19T14:21:28.591540+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_clean_scrub_stamp":"2023-10-19T14:21:25.343524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:37:35.585591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.18","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.518114+0000","last_change":"2023-10-19T14:21:10.435886+0000","last_active":"2023-10-19T14:21:27.518114+0000","last_peered":"2023-10-19T14:21:27.518114+0000","last_clean":"2023-10-19T14:21:27.518114+0000","last_became_active":"2023-10-19T14:21:10.435541+0000","last_became_peered":"2023-10-19T14:21:10.435541+0000","last_unstale":"2023-10-19T14:21:27.518114+0000","last_undegraded":"2023-10-19T14:21:27.518114+0000","last_fullsized":"2023-10-19T14:21:27.518114+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:02:44.118651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.15","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.590812+0000","last_change":"2023-10-19T14:21:27.530614+0000","last_active":"2023-10-19T14:21:28.590812+0000","last_peered":"2023-10-19T14:21:28.590812+0000","last_clean":"2023-10-19T14:21:28.590812+0000","last_became_active":"2023-10-19T14:21:27.530322+0000","last_became_peered":"2023-10-19T14:21:27.530322+0000","last_unstale":"2023-10-19T14:21:28.590812+0000","last_undegraded":"2023-10-19T14:21:28.590812+0000","last_fullsized":"2023-10-19T14:21:28.590812+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:23:55.515899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.16","version":"0'0","reported_seq":12,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.592624+0000","last_change":"2023-10-19T14:21:26.366314+0000","last_active":"2023-10-19T14:21:28.592624+0000","last_peered":"2023-10-19T14:21:28.592624+0000","last_clean":"2023-10-19T14:21:28.592624+0000","last_became_active":"2023-10-19T14:21:26.366024+0000","last_became_peered":"2023-10-19T14:21:26.366024+0000","last_unstale":"2023-10-19T14:21:28.592624+0000","last_undegraded":"2023-10-19T14:21:28.592624+0000","last_fullsized":"2023-10-19T14:21:28.592624+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_clean_scrub_stamp":"2023-10-19T14:21:25.343524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:28:12.261950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.17","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.515141+0000","last_change":"2023-10-19T14:21:10.436436+0000","last_active":"2023-10-19T14:21:27.515141+0000","last_peered":"2023-10-19T14:21:27.515141+0000","last_clean":"2023-10-19T14:21:27.515141+0000","last_became_active":"2023-10-19T14:21:10.435013+0000","last_became_peered":"2023-10-19T14:21:10.435013+0000","last_unstale":"2023-10-19T14:21:27.515141+0000","last_undegraded":"2023-10-19T14:21:27.515141+0000","last_fullsized":"2023-10-19T14:21:27.515141+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:16:49.709525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.14","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.527254+0000","last_change":"2023-10-19T14:21:27.526052+0000","last_active":"2023-10-19T14:21:28.527254+0000","last_peered":"2023-10-19T14:21:28.527254+0000","last_clean":"2023-10-19T14:21:28.527254+0000","last_became_active":"2023-10-19T14:21:27.524077+0000","last_became_peered":"2023-10-19T14:21:27.524077+0000","last_unstale":"2023-10-19T14:21:28.527254+0000","last_undegraded":"2023-10-19T14:21:28.527254+0000","last_fullsized":"2023-10-19T14:21:28.527254+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:38:13.814928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.17","version":"0'0","reported_seq":12,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.527081+0000","last_change":"2023-10-19T14:21:26.376096+0000","last_active":"2023-10-19T14:21:28.527081+0000","last_peered":"2023-10-19T14:21:28.527081+0000","last_clean":"2023-10-19T14:21:28.527081+0000","last_became_active":"2023-10-19T14:21:26.375073+0000","last_became_peered":"2023-10-19T14:21:26.375073+0000","last_unstale":"2023-10-19T14:21:28.527081+0000","last_undegraded":"2023-10-19T14:21:28.527081+0000","last_fullsized":"2023-10-19T14:21:28.527081+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_clean_scrub_stamp":"2023-10-19T14:21:25.343524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:00:02.644609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.16","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.512103+0000","last_change":"2023-10-19T14:21:10.430483+0000","last_active":"2023-10-19T14:21:27.512103+0000","last_peered":"2023-10-19T14:21:27.512103+0000","last_clean":"2023-10-19T14:21:27.512103+0000","last_became_active":"2023-10-19T14:21:10.430168+0000","last_became_peered":"2023-10-19T14:21:10.430168+0000","last_unstale":"2023-10-19T14:21:27.512103+0000","last_undegraded":"2023-10-19T14:21:27.512103+0000","last_fullsized":"2023-10-19T14:21:27.512103+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:40:00.950097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.17","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.593164+0000","last_change":"2023-10-19T14:21:27.534781+0000","last_active":"2023-10-19T14:21:28.593164+0000","last_peered":"2023-10-19T14:21:28.593164+0000","last_clean":"2023-10-19T14:21:28.593164+0000","last_became_active":"2023-10-19T14:21:27.532881+0000","last_became_peered":"2023-10-19T14:21:27.532881+0000","last_unstale":"2023-10-19T14:21:28.593164+0000","last_undegraded":"2023-10-19T14:21:28.593164+0000","last_fullsized":"2023-10-19T14:21:28.593164+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:15:01.763900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.14","version":"246'6","reported_seq":18,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.590915+0000","last_change":"2023-10-19T14:21:26.371959+0000","last_active":"2023-10-19T14:21:28.590915+0000","last_peered":"2023-10-19T14:21:28.590915+0000","last_clean":"2023-10-19T14:21:28.590915+0000","last_became_active":"2023-10-19T14:21:26.371488+0000","last_became_peered":"2023-10-19T14:21:26.371488+0000","last_unstale":"2023-10-19T14:21:28.590915+0000","last_undegraded":"2023-10-19T14:21:28.590915+0000","last_fullsized":"2023-10-19T14:21:28.590915+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_clean_scrub_stamp":"2023-10-19T14:21:25.343524+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:54:02.315367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.15","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.511882+0000","last_change":"2023-10-19T14:21:10.431222+0000","last_active":"2023-10-19T14:21:27.511882+0000","last_peered":"2023-10-19T14:21:27.511882+0000","last_clean":"2023-10-19T14:21:27.511882+0000","last_became_active":"2023-10-19T14:21:10.430868+0000","last_became_peered":"2023-10-19T14:21:10.430868+0000","last_unstale":"2023-10-19T14:21:27.511882+0000","last_undegraded":"2023-10-19T14:21:27.511882+0000","last_fullsized":"2023-10-19T14:21:27.511882+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:32:54.448851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.16","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.592000+0000","last_change":"2023-10-19T14:21:27.530838+0000","last_active":"2023-10-19T14:21:28.592000+0000","last_peered":"2023-10-19T14:21:28.592000+0000","last_clean":"2023-10-19T14:21:28.592000+0000","last_became_active":"2023-10-19T14:21:27.530570+0000","last_became_peered":"2023-10-19T14:21:27.530570+0000","last_unstale":"2023-10-19T14:21:28.592000+0000","last_undegraded":"2023-10-19T14:21:28.592000+0000","last_fullsized":"2023-10-19T14:21:28.592000+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:42:09.808829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.15","version":"246'3","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.526791+0000","last_change":"2023-10-19T14:21:26.378051+0000","last_active":"2023-10-19T14:21:28.526791+0000","last_peered":"2023-10-19T14:21:28.526791+0000","last_clean":"2023-10-19T14:21:28.526791+0000","last_became_active":"2023-10-19T14:21:26.377730+0000","last_became_peered":"2023-10-19T14:21:26.377730+0000","last_unstale":"2023-10-19T14:21:28.526791+0000","last_undegraded":"2023-10-19T14:21:28.526791+0000","last_fullsized":"2023-10-19T14:21:28.526791+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_clean_scrub_stamp":"2023-10-19T14:21:25.343524+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:53:13.687057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.14","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.515702+0000","last_change":"2023-10-19T14:21:10.425461+0000","last_active":"2023-10-19T14:21:27.515702+0000","last_peered":"2023-10-19T14:21:27.515702+0000","last_clean":"2023-10-19T14:21:27.515702+0000","last_became_active":"2023-10-19T14:21:10.425149+0000","last_became_peered":"2023-10-19T14:21:10.425149+0000","last_unstale":"2023-10-19T14:21:27.515702+0000","last_undegraded":"2023-10-19T14:21:27.515702+0000","last_fullsized":"2023-10-19T14:21:27.515702+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:30:59.739271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.11","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.590695+0000","last_change":"2023-10-19T14:21:27.533305+0000","last_active":"2023-10-19T14:21:28.590695+0000","last_peered":"2023-10-19T14:21:28.590695+0000","last_clean":"2023-10-19T14:21:28.590695+0000","last_became_active":"2023-10-19T14:21:27.533034+0000","last_became_peered":"2023-10-19T14:21:27.533034+0000","last_unstale":"2023-10-19T14:21:28.590695+0000","last_undegraded":"2023-10-19T14:21:28.590695+0000","last_fullsized":"2023-10-19T14:21:28.590695+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:06:14.851117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.12","version":"0'0","reported_seq":12,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.526980+0000","last_change":"2023-10-19T14:21:26.375752+0000","last_active":"2023-10-19T14:21:28.526980+0000","last_peered":"2023-10-19T14:21:28.526980+0000","last_clean":"2023-10-19T14:21:28.526980+0000","last_became_active":"2023-10-19T14:21:26.375426+0000","last_became_peered":"2023-10-19T14:21:26.375426+0000","last_unstale":"2023-10-19T14:21:28.526980+0000","last_undegraded":"2023-10-19T14:21:28.526980+0000","last_fullsized":"2023-10-19T14:21:28.526980+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_clean_scrub_stamp":"2023-10-19T14:21:25.343524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:55:15.110443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.13","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.520916+0000","last_change":"2023-10-19T14:21:10.424757+0000","last_active":"2023-10-19T14:21:27.520916+0000","last_peered":"2023-10-19T14:21:27.520916+0000","last_clean":"2023-10-19T14:21:27.520916+0000","last_became_active":"2023-10-19T14:21:10.424482+0000","last_became_peered":"2023-10-19T14:21:10.424482+0000","last_unstale":"2023-10-19T14:21:27.520916+0000","last_undegraded":"2023-10-19T14:21:27.520916+0000","last_fullsized":"2023-10-19T14:21:27.520916+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:43:36.429903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.10","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.589844+0000","last_change":"2023-10-19T14:21:27.527982+0000","last_active":"2023-10-19T14:21:28.589844+0000","last_peered":"2023-10-19T14:21:28.589844+0000","last_clean":"2023-10-19T14:21:28.589844+0000","last_became_active":"2023-10-19T14:21:27.527692+0000","last_became_peered":"2023-10-19T14:21:27.527692+0000","last_unstale":"2023-10-19T14:21:28.589844+0000","last_undegraded":"2023-10-19T14:21:28.589844+0000","last_fullsized":"2023-10-19T14:21:28.589844+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:10:28.073265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.13","version":"246'2","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.590605+0000","last_change":"2023-10-19T14:21:26.369644+0000","last_active":"2023-10-19T14:21:28.590605+0000","last_peered":"2023-10-19T14:21:28.590605+0000","last_clean":"2023-10-19T14:21:28.590605+0000","last_became_active":"2023-10-19T14:21:26.369176+0000","last_became_peered":"2023-10-19T14:21:26.369176+0000","last_unstale":"2023-10-19T14:21:28.590605+0000","last_undegraded":"2023-10-19T14:21:28.590605+0000","last_fullsized":"2023-10-19T14:21:28.590605+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_clean_scrub_stamp":"2023-10-19T14:21:25.343524+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:19:05.362807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.12","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.536049+0000","last_change":"2023-10-19T14:21:10.423736+0000","last_active":"2023-10-19T14:21:27.536049+0000","last_peered":"2023-10-19T14:21:27.536049+0000","last_clean":"2023-10-19T14:21:27.536049+0000","last_became_active":"2023-10-19T14:21:10.423448+0000","last_became_peered":"2023-10-19T14:21:10.423448+0000","last_unstale":"2023-10-19T14:21:27.536049+0000","last_undegraded":"2023-10-19T14:21:27.536049+0000","last_fullsized":"2023-10-19T14:21:27.536049+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:42:04.331509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.13","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.590122+0000","last_change":"2023-10-19T14:21:27.529495+0000","last_active":"2023-10-19T14:21:28.590122+0000","last_peered":"2023-10-19T14:21:28.590122+0000","last_clean":"2023-10-19T14:21:28.590122+0000","last_became_active":"2023-10-19T14:21:27.529200+0000","last_became_peered":"2023-10-19T14:21:27.529200+0000","last_unstale":"2023-10-19T14:21:28.590122+0000","last_undegraded":"2023-10-19T14:21:28.590122+0000","last_fullsized":"2023-10-19T14:21:28.590122+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:21:50.232128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.10","version":"246'8","reported_seq":20,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.526325+0000","last_change":"2023-10-19T14:21:26.376659+0000","last_active":"2023-10-19T14:21:28.526325+0000","last_peered":"2023-10-19T14:21:28.526325+0000","last_clean":"2023-10-19T14:21:28.526325+0000","last_became_active":"2023-10-19T14:21:26.376196+0000","last_became_peered":"2023-10-19T14:21:26.376196+0000","last_unstale":"2023-10-19T14:21:28.526325+0000","last_undegraded":"2023-10-19T14:21:28.526325+0000","last_fullsized":"2023-10-19T14:21:28.526325+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_clean_scrub_stamp":"2023-10-19T14:21:25.343524+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T02:12:05.812260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.11","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.516210+0000","last_change":"2023-10-19T14:21:10.430551+0000","last_active":"2023-10-19T14:21:27.516210+0000","last_peered":"2023-10-19T14:21:27.516210+0000","last_clean":"2023-10-19T14:21:27.516210+0000","last_became_active":"2023-10-19T14:21:10.429509+0000","last_became_peered":"2023-10-19T14:21:10.429509+0000","last_unstale":"2023-10-19T14:21:27.516210+0000","last_undegraded":"2023-10-19T14:21:27.516210+0000","last_fullsized":"2023-10-19T14:21:27.516210+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:58:08.366505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.12","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.590510+0000","last_change":"2023-10-19T14:21:27.535132+0000","last_active":"2023-10-19T14:21:28.590510+0000","last_peered":"2023-10-19T14:21:28.590510+0000","last_clean":"2023-10-19T14:21:28.590510+0000","last_became_active":"2023-10-19T14:21:27.534830+0000","last_became_peered":"2023-10-19T14:21:27.534830+0000","last_unstale":"2023-10-19T14:21:28.590510+0000","last_undegraded":"2023-10-19T14:21:28.590510+0000","last_fullsized":"2023-10-19T14:21:28.590510+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:53:52.620951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.11","version":"0'0","reported_seq":12,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.526503+0000","last_change":"2023-10-19T14:21:26.374288+0000","last_active":"2023-10-19T14:21:28.526503+0000","last_peered":"2023-10-19T14:21:28.526503+0000","last_clean":"2023-10-19T14:21:28.526503+0000","last_became_active":"2023-10-19T14:21:26.374018+0000","last_became_peered":"2023-10-19T14:21:26.374018+0000","last_unstale":"2023-10-19T14:21:28.526503+0000","last_undegraded":"2023-10-19T14:21:28.526503+0000","last_fullsized":"2023-10-19T14:21:28.526503+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_clean_scrub_stamp":"2023-10-19T14:21:25.343524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:38:55.132511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.10","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.510813+0000","last_change":"2023-10-19T14:21:10.437276+0000","last_active":"2023-10-19T14:21:27.510813+0000","last_peered":"2023-10-19T14:21:27.510813+0000","last_clean":"2023-10-19T14:21:27.510813+0000","last_became_active":"2023-10-19T14:21:10.437013+0000","last_became_peered":"2023-10-19T14:21:10.437013+0000","last_unstale":"2023-10-19T14:21:27.510813+0000","last_undegraded":"2023-10-19T14:21:27.510813+0000","last_fullsized":"2023-10-19T14:21:27.510813+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:06:58.837634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.d","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.592119+0000","last_change":"2023-10-19T14:21:27.528223+0000","last_active":"2023-10-19T14:21:28.592119+0000","last_peered":"2023-10-19T14:21:28.592119+0000","last_clean":"2023-10-19T14:21:28.592119+0000","last_became_active":"2023-10-19T14:21:27.527684+0000","last_became_peered":"2023-10-19T14:21:27.527684+0000","last_unstale":"2023-10-19T14:21:28.592119+0000","last_undegraded":"2023-10-19T14:21:28.592119+0000","last_fullsized":"2023-10-19T14:21:28.592119+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:40:13.261454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.e","version":"246'2","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.591227+0000","last_change":"2023-10-19T14:21:26.365681+0000","last_active":"2023-10-19T14:21:28.591227+0000","last_peered":"2023-10-19T14:21:28.591227+0000","last_clean":"2023-10-19T14:21:28.591227+0000","last_became_active":"2023-10-19T14:21:26.365428+0000","last_became_peered":"2023-10-19T14:21:26.365428+0000","last_unstale":"2023-10-19T14:21:28.591227+0000","last_undegraded":"2023-10-19T14:21:28.591227+0000","last_fullsized":"2023-10-19T14:21:28.591227+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_clean_scrub_stamp":"2023-10-19T14:21:25.343524+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:45:26.686054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.f","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.525813+0000","last_change":"2023-10-19T14:21:10.429097+0000","last_active":"2023-10-19T14:21:27.525813+0000","last_peered":"2023-10-19T14:21:27.525813+0000","last_clean":"2023-10-19T14:21:27.525813+0000","last_became_active":"2023-10-19T14:21:10.428825+0000","last_became_peered":"2023-10-19T14:21:10.428825+0000","last_unstale":"2023-10-19T14:21:27.525813+0000","last_undegraded":"2023-10-19T14:21:27.525813+0000","last_fullsized":"2023-10-19T14:21:27.525813+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:33:10.076418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.c","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.592421+0000","last_change":"2023-10-19T14:21:27.531130+0000","last_active":"2023-10-19T14:21:28.592421+0000","last_peered":"2023-10-19T14:21:28.592421+0000","last_clean":"2023-10-19T14:21:28.592421+0000","last_became_active":"2023-10-19T14:21:27.529377+0000","last_became_peered":"2023-10-19T14:21:27.529377+0000","last_unstale":"2023-10-19T14:21:28.592421+0000","last_undegraded":"2023-10-19T14:21:28.592421+0000","last_fullsized":"2023-10-19T14:21:28.592421+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:45:11.239855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.f","version":"246'1","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.591966+0000","last_change":"2023-10-19T14:21:26.372895+0000","last_active":"2023-10-19T14:21:28.591966+0000","last_peered":"2023-10-19T14:21:28.591966+0000","last_clean":"2023-10-19T14:21:28.591966+0000","last_became_active":"2023-10-19T14:21:26.372573+0000","last_became_peered":"2023-10-19T14:21:26.372573+0000","last_unstale":"2023-10-19T14:21:28.591966+0000","last_undegraded":"2023-10-19T14:21:28.591966+0000","last_fullsized":"2023-10-19T14:21:28.591966+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_clean_scrub_stamp":"2023-10-19T14:21:25.343524+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:10:27.070104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.e","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.512194+0000","last_change":"2023-10-19T14:21:10.425661+0000","last_active":"2023-10-19T14:21:27.512194+0000","last_peered":"2023-10-19T14:21:27.512194+0000","last_clean":"2023-10-19T14:21:27.512194+0000","last_became_active":"2023-10-19T14:21:10.425369+0000","last_became_peered":"2023-10-19T14:21:10.425369+0000","last_unstale":"2023-10-19T14:21:27.512194+0000","last_undegraded":"2023-10-19T14:21:27.512194+0000","last_fullsized":"2023-10-19T14:21:27.512194+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:54:48.155840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.f","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.593177+0000","last_change":"2023-10-19T14:21:27.534960+0000","last_active":"2023-10-19T14:21:28.593177+0000","last_peered":"2023-10-19T14:21:28.593177+0000","last_clean":"2023-10-19T14:21:28.593177+0000","last_became_active":"2023-10-19T14:21:27.533567+0000","last_became_peered":"2023-10-19T14:21:27.533567+0000","last_unstale":"2023-10-19T14:21:28.593177+0000","last_undegraded":"2023-10-19T14:21:28.593177+0000","last_fullsized":"2023-10-19T14:21:28.593177+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:41:41.163291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.c","version":"0'0","reported_seq":12,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.590925+0000","last_change":"2023-10-19T14:21:26.372370+0000","last_active":"2023-10-19T14:21:28.590925+0000","last_peered":"2023-10-19T14:21:28.590925+0000","last_clean":"2023-10-19T14:21:28.590925+0000","last_became_active":"2023-10-19T14:21:26.371478+0000","last_became_peered":"2023-10-19T14:21:26.371478+0000","last_unstale":"2023-10-19T14:21:28.590925+0000","last_undegraded":"2023-10-19T14:21:28.590925+0000","last_fullsized":"2023-10-19T14:21:28.590925+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_clean_scrub_stamp":"2023-10-19T14:21:25.343524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:00:13.656453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.d","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.517542+0000","last_change":"2023-10-19T14:21:10.423050+0000","last_active":"2023-10-19T14:21:27.517542+0000","last_peered":"2023-10-19T14:21:27.517542+0000","last_clean":"2023-10-19T14:21:27.517542+0000","last_became_active":"2023-10-19T14:21:10.422727+0000","last_became_peered":"2023-10-19T14:21:10.422727+0000","last_unstale":"2023-10-19T14:21:27.517542+0000","last_undegraded":"2023-10-19T14:21:27.517542+0000","last_fullsized":"2023-10-19T14:21:27.517542+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:21:43.393777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.e","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.527011+0000","last_change":"2023-10-19T14:21:27.533236+0000","last_active":"2023-10-19T14:21:28.527011+0000","last_peered":"2023-10-19T14:21:28.527011+0000","last_clean":"2023-10-19T14:21:28.527011+0000","last_became_active":"2023-10-19T14:21:27.532931+0000","last_became_peered":"2023-10-19T14:21:27.532931+0000","last_unstale":"2023-10-19T14:21:28.527011+0000","last_undegraded":"2023-10-19T14:21:28.527011+0000","last_fullsized":"2023-10-19T14:21:28.527011+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:49:00.966124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.d","version":"246'1","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.527454+0000","last_change":"2023-10-19T14:21:26.364297+0000","last_active":"2023-10-19T14:21:28.527454+0000","last_peered":"2023-10-19T14:21:28.527454+0000","last_clean":"2023-10-19T14:21:28.527454+0000","last_became_active":"2023-10-19T14:21:26.364036+0000","last_became_peered":"2023-10-19T14:21:26.364036+0000","last_unstale":"2023-10-19T14:21:28.527454+0000","last_undegraded":"2023-10-19T14:21:28.527454+0000","last_fullsized":"2023-10-19T14:21:28.527454+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_clean_scrub_stamp":"2023-10-19T14:21:25.343524+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:30:33.383813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.c","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.522998+0000","last_change":"2023-10-19T14:21:10.432921+0000","last_active":"2023-10-19T14:21:27.522998+0000","last_peered":"2023-10-19T14:21:27.522998+0000","last_clean":"2023-10-19T14:21:27.522998+0000","last_became_active":"2023-10-19T14:21:10.432654+0000","last_became_peered":"2023-10-19T14:21:10.432654+0000","last_unstale":"2023-10-19T14:21:27.522998+0000","last_undegraded":"2023-10-19T14:21:27.522998+0000","last_fullsized":"2023-10-19T14:21:27.522998+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:42:44.275430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.9","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.590841+0000","last_change":"2023-10-19T14:21:27.531873+0000","last_active":"2023-10-19T14:21:28.590841+0000","last_peered":"2023-10-19T14:21:28.590841+0000","last_clean":"2023-10-19T14:21:28.590841+0000","last_became_active":"2023-10-19T14:21:27.531581+0000","last_became_peered":"2023-10-19T14:21:27.531581+0000","last_unstale":"2023-10-19T14:21:28.590841+0000","last_undegraded":"2023-10-19T14:21:28.590841+0000","last_fullsized":"2023-10-19T14:21:28.590841+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:17:46.220498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.a","version":"0'0","reported_seq":12,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.590098+0000","last_change":"2023-10-19T14:21:26.374177+0000","last_active":"2023-10-19T14:21:28.590098+0000","last_peered":"2023-10-19T14:21:28.590098+0000","last_clean":"2023-10-19T14:21:28.590098+0000","last_became_active":"2023-10-19T14:21:26.373883+0000","last_became_peered":"2023-10-19T14:21:26.373883+0000","last_unstale":"2023-10-19T14:21:28.590098+0000","last_undegraded":"2023-10-19T14:21:28.590098+0000","last_fullsized":"2023-10-19T14:21:28.590098+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_clean_scrub_stamp":"2023-10-19T14:21:25.343524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:12:22.183091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.b","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.512502+0000","last_change":"2023-10-19T14:21:10.422339+0000","last_active":"2023-10-19T14:21:27.512502+0000","last_peered":"2023-10-19T14:21:27.512502+0000","last_clean":"2023-10-19T14:21:27.512502+0000","last_became_active":"2023-10-19T14:21:10.422033+0000","last_became_peered":"2023-10-19T14:21:10.422033+0000","last_unstale":"2023-10-19T14:21:27.512502+0000","last_undegraded":"2023-10-19T14:21:27.512502+0000","last_fullsized":"2023-10-19T14:21:27.512502+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:19:11.812010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.8","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.524439+0000","last_change":"2023-10-19T14:21:27.532550+0000","last_active":"2023-10-19T14:21:28.524439+0000","last_peered":"2023-10-19T14:21:28.524439+0000","last_clean":"2023-10-19T14:21:28.524439+0000","last_became_active":"2023-10-19T14:21:27.532289+0000","last_became_peered":"2023-10-19T14:21:27.532289+0000","last_unstale":"2023-10-19T14:21:28.524439+0000","last_undegraded":"2023-10-19T14:21:28.524439+0000","last_fullsized":"2023-10-19T14:21:28.524439+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:54:21.231604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.b","version":"246'2","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.590650+0000","last_change":"2023-10-19T14:21:26.375601+0000","last_active":"2023-10-19T14:21:28.590650+0000","last_peered":"2023-10-19T14:21:28.590650+0000","last_clean":"2023-10-19T14:21:28.590650+0000","last_became_active":"2023-10-19T14:21:26.375309+0000","last_became_peered":"2023-10-19T14:21:26.375309+0000","last_unstale":"2023-10-19T14:21:28.590650+0000","last_undegraded":"2023-10-19T14:21:28.590650+0000","last_fullsized":"2023-10-19T14:21:28.590650+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_clean_scrub_stamp":"2023-10-19T14:21:25.343524+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:02:39.807380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.a","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.521896+0000","last_change":"2023-10-19T14:21:10.438315+0000","last_active":"2023-10-19T14:21:27.521896+0000","last_peered":"2023-10-19T14:21:27.521896+0000","last_clean":"2023-10-19T14:21:27.521896+0000","last_became_active":"2023-10-19T14:21:10.438026+0000","last_became_peered":"2023-10-19T14:21:10.438026+0000","last_unstale":"2023-10-19T14:21:27.521896+0000","last_undegraded":"2023-10-19T14:21:27.521896+0000","last_fullsized":"2023-10-19T14:21:27.521896+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:36:02.222588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.b","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.524601+0000","last_change":"2023-10-19T14:21:27.542409+0000","last_active":"2023-10-19T14:21:28.524601+0000","last_peered":"2023-10-19T14:21:28.524601+0000","last_clean":"2023-10-19T14:21:28.524601+0000","last_became_active":"2023-10-19T14:21:27.542134+0000","last_became_peered":"2023-10-19T14:21:27.542134+0000","last_unstale":"2023-10-19T14:21:28.524601+0000","last_undegraded":"2023-10-19T14:21:28.524601+0000","last_fullsized":"2023-10-19T14:21:28.524601+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:43:22.958184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.8","version":"0'0","reported_seq":12,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.526628+0000","last_change":"2023-10-19T14:21:26.376475+0000","last_active":"2023-10-19T14:21:28.526628+0000","last_peered":"2023-10-19T14:21:28.526628+0000","last_clean":"2023-10-19T14:21:28.526628+0000","last_became_active":"2023-10-19T14:21:26.375820+0000","last_became_peered":"2023-10-19T14:21:26.375820+0000","last_unstale":"2023-10-19T14:21:28.526628+0000","last_undegraded":"2023-10-19T14:21:28.526628+0000","last_fullsized":"2023-10-19T14:21:28.526628+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_clean_scrub_stamp":"2023-10-19T14:21:25.343524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:09:51.132064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.9","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.516297+0000","last_change":"2023-10-19T14:21:10.430361+0000","last_active":"2023-10-19T14:21:27.516297+0000","last_peered":"2023-10-19T14:21:27.516297+0000","last_clean":"2023-10-19T14:21:27.516297+0000","last_became_active":"2023-10-19T14:21:10.429861+0000","last_became_peered":"2023-10-19T14:21:10.429861+0000","last_unstale":"2023-10-19T14:21:27.516297+0000","last_undegraded":"2023-10-19T14:21:27.516297+0000","last_fullsized":"2023-10-19T14:21:27.516297+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:50:34.147669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.a","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.526568+0000","last_change":"2023-10-19T14:21:27.518932+0000","last_active":"2023-10-19T14:21:28.526568+0000","last_peered":"2023-10-19T14:21:28.526568+0000","last_clean":"2023-10-19T14:21:28.526568+0000","last_became_active":"2023-10-19T14:21:27.518518+0000","last_became_peered":"2023-10-19T14:21:27.518518+0000","last_unstale":"2023-10-19T14:21:28.526568+0000","last_undegraded":"2023-10-19T14:21:28.526568+0000","last_fullsized":"2023-10-19T14:21:28.526568+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:50:03.901367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.9","version":"0'0","reported_seq":12,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.524769+0000","last_change":"2023-10-19T14:21:26.364933+0000","last_active":"2023-10-19T14:21:28.524769+0000","last_peered":"2023-10-19T14:21:28.524769+0000","last_clean":"2023-10-19T14:21:28.524769+0000","last_became_active":"2023-10-19T14:21:26.364682+0000","last_became_peered":"2023-10-19T14:21:26.364682+0000","last_unstale":"2023-10-19T14:21:28.524769+0000","last_undegraded":"2023-10-19T14:21:28.524769+0000","last_fullsized":"2023-10-19T14:21:28.524769+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_clean_scrub_stamp":"2023-10-19T14:21:25.343524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:18:23.087445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.8","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.523749+0000","last_change":"2023-10-19T14:21:10.413051+0000","last_active":"2023-10-19T14:21:27.523749+0000","last_peered":"2023-10-19T14:21:27.523749+0000","last_clean":"2023-10-19T14:21:27.523749+0000","last_became_active":"2023-10-19T14:21:10.412765+0000","last_became_peered":"2023-10-19T14:21:10.412765+0000","last_unstale":"2023-10-19T14:21:27.523749+0000","last_undegraded":"2023-10-19T14:21:27.523749+0000","last_fullsized":"2023-10-19T14:21:27.523749+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T02:02:33.398058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.5","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.590793+0000","last_change":"2023-10-19T14:21:27.534347+0000","last_active":"2023-10-19T14:21:28.590793+0000","last_peered":"2023-10-19T14:21:28.590793+0000","last_clean":"2023-10-19T14:21:28.590793+0000","last_became_active":"2023-10-19T14:21:27.534058+0000","last_became_peered":"2023-10-19T14:21:27.534058+0000","last_unstale":"2023-10-19T14:21:28.590793+0000","last_undegraded":"2023-10-19T14:21:28.590793+0000","last_fullsized":"2023-10-19T14:21:28.590793+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:15:17.090163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.6","version":"0'0","reported_seq":12,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.591314+0000","last_change":"2023-10-19T14:21:26.363091+0000","last_active":"2023-10-19T14:21:28.591314+0000","last_peered":"2023-10-19T14:21:28.591314+0000","last_clean":"2023-10-19T14:21:28.591314+0000","last_became_active":"2023-10-19T14:21:26.362822+0000","last_became_peered":"2023-10-19T14:21:26.362822+0000","last_unstale":"2023-10-19T14:21:28.591314+0000","last_undegraded":"2023-10-19T14:21:28.591314+0000","last_fullsized":"2023-10-19T14:21:28.591314+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_clean_scrub_stamp":"2023-10-19T14:21:25.343524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:29:55.911261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.7","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.524677+0000","last_change":"2023-10-19T14:21:10.430882+0000","last_active":"2023-10-19T14:21:27.524677+0000","last_peered":"2023-10-19T14:21:27.524677+0000","last_clean":"2023-10-19T14:21:27.524677+0000","last_became_active":"2023-10-19T14:21:10.430631+0000","last_became_peered":"2023-10-19T14:21:10.430631+0000","last_unstale":"2023-10-19T14:21:27.524677+0000","last_undegraded":"2023-10-19T14:21:27.524677+0000","last_fullsized":"2023-10-19T14:21:27.524677+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:02:43.850106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.4","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.592010+0000","last_change":"2023-10-19T14:21:27.528861+0000","last_active":"2023-10-19T14:21:28.592010+0000","last_peered":"2023-10-19T14:21:28.592010+0000","last_clean":"2023-10-19T14:21:28.592010+0000","last_became_active":"2023-10-19T14:21:27.528600+0000","last_became_peered":"2023-10-19T14:21:27.528600+0000","last_unstale":"2023-10-19T14:21:28.592010+0000","last_undegraded":"2023-10-19T14:21:28.592010+0000","last_fullsized":"2023-10-19T14:21:28.592010+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:53:51.253050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.7","version":"246'2","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.527121+0000","last_change":"2023-10-19T14:21:26.375412+0000","last_active":"2023-10-19T14:21:28.527121+0000","last_peered":"2023-10-19T14:21:28.527121+0000","last_clean":"2023-10-19T14:21:28.527121+0000","last_became_active":"2023-10-19T14:21:26.375133+0000","last_became_peered":"2023-10-19T14:21:26.375133+0000","last_unstale":"2023-10-19T14:21:28.527121+0000","last_undegraded":"2023-10-19T14:21:28.527121+0000","last_fullsized":"2023-10-19T14:21:28.527121+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_clean_scrub_stamp":"2023-10-19T14:21:25.343524+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:13:37.862094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.6","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.512746+0000","last_change":"2023-10-19T14:21:10.428612+0000","last_active":"2023-10-19T14:21:27.512746+0000","last_peered":"2023-10-19T14:21:27.512746+0000","last_clean":"2023-10-19T14:21:27.512746+0000","last_became_active":"2023-10-19T14:21:10.428248+0000","last_became_peered":"2023-10-19T14:21:10.428248+0000","last_unstale":"2023-10-19T14:21:27.512746+0000","last_undegraded":"2023-10-19T14:21:27.512746+0000","last_fullsized":"2023-10-19T14:21:27.512746+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:13:47.366105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.7","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.591911+0000","last_change":"2023-10-19T14:21:27.532673+0000","last_active":"2023-10-19T14:21:28.591911+0000","last_peered":"2023-10-19T14:21:28.591911+0000","last_clean":"2023-10-19T14:21:28.591911+0000","last_became_active":"2023-10-19T14:21:27.532411+0000","last_became_peered":"2023-10-19T14:21:27.532411+0000","last_unstale":"2023-10-19T14:21:28.591911+0000","last_undegraded":"2023-10-19T14:21:28.591911+0000","last_fullsized":"2023-10-19T14:21:28.591911+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:00:40.916980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.4","version":"0'0","reported_seq":12,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.527041+0000","last_change":"2023-10-19T14:21:26.364085+0000","last_active":"2023-10-19T14:21:28.527041+0000","last_peered":"2023-10-19T14:21:28.527041+0000","last_clean":"2023-10-19T14:21:28.527041+0000","last_became_active":"2023-10-19T14:21:26.363821+0000","last_became_peered":"2023-10-19T14:21:26.363821+0000","last_unstale":"2023-10-19T14:21:28.527041+0000","last_undegraded":"2023-10-19T14:21:28.527041+0000","last_fullsized":"2023-10-19T14:21:28.527041+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_clean_scrub_stamp":"2023-10-19T14:21:25.343524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:47:33.310989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.5","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.517643+0000","last_change":"2023-10-19T14:21:10.424660+0000","last_active":"2023-10-19T14:21:27.517643+0000","last_peered":"2023-10-19T14:21:27.517643+0000","last_clean":"2023-10-19T14:21:27.517643+0000","last_became_active":"2023-10-19T14:21:10.423849+0000","last_became_peered":"2023-10-19T14:21:10.423849+0000","last_unstale":"2023-10-19T14:21:27.517643+0000","last_undegraded":"2023-10-19T14:21:27.517643+0000","last_fullsized":"2023-10-19T14:21:27.517643+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:44:54.282098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.6","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.527407+0000","last_change":"2023-10-19T14:21:27.519397+0000","last_active":"2023-10-19T14:21:28.527407+0000","last_peered":"2023-10-19T14:21:28.527407+0000","last_clean":"2023-10-19T14:21:28.527407+0000","last_became_active":"2023-10-19T14:21:27.519113+0000","last_became_peered":"2023-10-19T14:21:27.519113+0000","last_unstale":"2023-10-19T14:21:28.527407+0000","last_undegraded":"2023-10-19T14:21:28.527407+0000","last_fullsized":"2023-10-19T14:21:28.527407+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:04:58.823569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.5","version":"0'0","reported_seq":12,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.527138+0000","last_change":"2023-10-19T14:21:26.363474+0000","last_active":"2023-10-19T14:21:28.527138+0000","last_peered":"2023-10-19T14:21:28.527138+0000","last_clean":"2023-10-19T14:21:28.527138+0000","last_became_active":"2023-10-19T14:21:26.363190+0000","last_became_peered":"2023-10-19T14:21:26.363190+0000","last_unstale":"2023-10-19T14:21:28.527138+0000","last_undegraded":"2023-10-19T14:21:28.527138+0000","last_fullsized":"2023-10-19T14:21:28.527138+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_clean_scrub_stamp":"2023-10-19T14:21:25.343524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:33:10.403448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.4","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.515692+0000","last_change":"2023-10-19T14:21:10.432004+0000","last_active":"2023-10-19T14:21:27.515692+0000","last_peered":"2023-10-19T14:21:27.515692+0000","last_clean":"2023-10-19T14:21:27.515692+0000","last_became_active":"2023-10-19T14:21:10.431687+0000","last_became_peered":"2023-10-19T14:21:10.431687+0000","last_unstale":"2023-10-19T14:21:27.515692+0000","last_undegraded":"2023-10-19T14:21:27.515692+0000","last_fullsized":"2023-10-19T14:21:27.515692+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:31:38.548044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.1","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.524606+0000","last_change":"2023-10-19T14:21:27.540440+0000","last_active":"2023-10-19T14:21:28.524606+0000","last_peered":"2023-10-19T14:21:28.524606+0000","last_clean":"2023-10-19T14:21:28.524606+0000","last_became_active":"2023-10-19T14:21:27.540079+0000","last_became_peered":"2023-10-19T14:21:27.540079+0000","last_unstale":"2023-10-19T14:21:28.524606+0000","last_undegraded":"2023-10-19T14:21:28.524606+0000","last_fullsized":"2023-10-19T14:21:28.524606+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:09:52.510151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.2","version":"0'0","reported_seq":12,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.590089+0000","last_change":"2023-10-19T14:21:26.376809+0000","last_active":"2023-10-19T14:21:28.590089+0000","last_peered":"2023-10-19T14:21:28.590089+0000","last_clean":"2023-10-19T14:21:28.590089+0000","last_became_active":"2023-10-19T14:21:26.376555+0000","last_became_peered":"2023-10-19T14:21:26.376555+0000","last_unstale":"2023-10-19T14:21:28.590089+0000","last_undegraded":"2023-10-19T14:21:28.590089+0000","last_fullsized":"2023-10-19T14:21:28.590089+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_clean_scrub_stamp":"2023-10-19T14:21:25.343524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:12:03.508740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.3","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.536100+0000","last_change":"2023-10-19T14:21:10.419200+0000","last_active":"2023-10-19T14:21:27.536100+0000","last_peered":"2023-10-19T14:21:27.536100+0000","last_clean":"2023-10-19T14:21:27.536100+0000","last_became_active":"2023-10-19T14:21:10.418921+0000","last_became_peered":"2023-10-19T14:21:10.418921+0000","last_unstale":"2023-10-19T14:21:27.536100+0000","last_undegraded":"2023-10-19T14:21:27.536100+0000","last_fullsized":"2023-10-19T14:21:27.536100+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:18:49.408210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.0","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.526340+0000","last_change":"2023-10-19T14:21:27.520641+0000","last_active":"2023-10-19T14:21:28.526340+0000","last_peered":"2023-10-19T14:21:28.526340+0000","last_clean":"2023-10-19T14:21:28.526340+0000","last_became_active":"2023-10-19T14:21:27.520348+0000","last_became_peered":"2023-10-19T14:21:27.520348+0000","last_unstale":"2023-10-19T14:21:28.526340+0000","last_undegraded":"2023-10-19T14:21:28.526340+0000","last_fullsized":"2023-10-19T14:21:28.526340+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:49:26.624655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.3","version":"0'0","reported_seq":12,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.590541+0000","last_change":"2023-10-19T14:21:26.368505+0000","last_active":"2023-10-19T14:21:28.590541+0000","last_peered":"2023-10-19T14:21:28.590541+0000","last_clean":"2023-10-19T14:21:28.590541+0000","last_became_active":"2023-10-19T14:21:26.368200+0000","last_became_peered":"2023-10-19T14:21:26.368200+0000","last_unstale":"2023-10-19T14:21:28.590541+0000","last_undegraded":"2023-10-19T14:21:28.590541+0000","last_fullsized":"2023-10-19T14:21:28.590541+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_clean_scrub_stamp":"2023-10-19T14:21:25.343524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:42:25.318847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.2","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.536075+0000","last_change":"2023-10-19T14:21:10.426386+0000","last_active":"2023-10-19T14:21:27.536075+0000","last_peered":"2023-10-19T14:21:27.536075+0000","last_clean":"2023-10-19T14:21:27.536075+0000","last_became_active":"2023-10-19T14:21:10.426117+0000","last_became_peered":"2023-10-19T14:21:10.426117+0000","last_unstale":"2023-10-19T14:21:27.536075+0000","last_undegraded":"2023-10-19T14:21:27.536075+0000","last_fullsized":"2023-10-19T14:21:27.536075+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:46:24.186941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.3","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.592690+0000","last_change":"2023-10-19T14:21:27.531185+0000","last_active":"2023-10-19T14:21:28.592690+0000","last_peered":"2023-10-19T14:21:28.592690+0000","last_clean":"2023-10-19T14:21:28.592690+0000","last_became_active":"2023-10-19T14:21:27.530896+0000","last_became_peered":"2023-10-19T14:21:27.530896+0000","last_unstale":"2023-10-19T14:21:28.592690+0000","last_undegraded":"2023-10-19T14:21:28.592690+0000","last_fullsized":"2023-10-19T14:21:28.592690+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:43:20.053694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.0","version":"0'0","reported_seq":12,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.590464+0000","last_change":"2023-10-19T14:21:26.364519+0000","last_active":"2023-10-19T14:21:28.590464+0000","last_peered":"2023-10-19T14:21:28.590464+0000","last_clean":"2023-10-19T14:21:28.590464+0000","last_became_active":"2023-10-19T14:21:26.364257+0000","last_became_peered":"2023-10-19T14:21:26.364257+0000","last_unstale":"2023-10-19T14:21:28.590464+0000","last_undegraded":"2023-10-19T14:21:28.590464+0000","last_fullsized":"2023-10-19T14:21:28.590464+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_clean_scrub_stamp":"2023-10-19T14:21:25.343524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:33:26.031016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.521704+0000","last_change":"2023-10-19T14:21:10.429319+0000","last_active":"2023-10-19T14:21:27.521704+0000","last_peered":"2023-10-19T14:21:27.521704+0000","last_clean":"2023-10-19T14:21:27.521704+0000","last_became_active":"2023-10-19T14:21:10.429037+0000","last_became_peered":"2023-10-19T14:21:10.429037+0000","last_unstale":"2023-10-19T14:21:27.521704+0000","last_undegraded":"2023-10-19T14:21:27.521704+0000","last_fullsized":"2023-10-19T14:21:27.521704+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:38:02.329128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.2","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.524445+0000","last_change":"2023-10-19T14:21:27.541136+0000","last_active":"2023-10-19T14:21:28.524445+0000","last_peered":"2023-10-19T14:21:28.524445+0000","last_clean":"2023-10-19T14:21:28.524445+0000","last_became_active":"2023-10-19T14:21:27.540862+0000","last_became_peered":"2023-10-19T14:21:27.540862+0000","last_unstale":"2023-10-19T14:21:28.524445+0000","last_undegraded":"2023-10-19T14:21:28.524445+0000","last_fullsized":"2023-10-19T14:21:28.524445+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:44:29.301692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1","version":"0'0","reported_seq":12,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.526874+0000","last_change":"2023-10-19T14:21:26.364816+0000","last_active":"2023-10-19T14:21:28.526874+0000","last_peered":"2023-10-19T14:21:28.526874+0000","last_clean":"2023-10-19T14:21:28.526874+0000","last_became_active":"2023-10-19T14:21:26.364542+0000","last_became_peered":"2023-10-19T14:21:26.364542+0000","last_unstale":"2023-10-19T14:21:28.526874+0000","last_undegraded":"2023-10-19T14:21:28.526874+0000","last_fullsized":"2023-10-19T14:21:28.526874+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_clean_scrub_stamp":"2023-10-19T14:21:25.343524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:29:27.857081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.0","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.515824+0000","last_change":"2023-10-19T14:21:10.432051+0000","last_active":"2023-10-19T14:21:27.515824+0000","last_peered":"2023-10-19T14:21:27.515824+0000","last_clean":"2023-10-19T14:21:27.515824+0000","last_became_active":"2023-10-19T14:21:10.431732+0000","last_became_peered":"2023-10-19T14:21:10.431732+0000","last_unstale":"2023-10-19T14:21:27.515824+0000","last_undegraded":"2023-10-19T14:21:27.515824+0000","last_fullsized":"2023-10-19T14:21:27.515824+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:19:07.965561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.22","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.591443+0000","last_change":"2023-10-19T14:21:27.530847+0000","last_active":"2023-10-19T14:21:28.591443+0000","last_peered":"2023-10-19T14:21:28.591443+0000","last_clean":"2023-10-19T14:21:28.591443+0000","last_became_active":"2023-10-19T14:21:27.530561+0000","last_became_peered":"2023-10-19T14:21:27.530561+0000","last_unstale":"2023-10-19T14:21:28.591443+0000","last_undegraded":"2023-10-19T14:21:28.591443+0000","last_fullsized":"2023-10-19T14:21:28.591443+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:14:10.207370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.20","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.522619+0000","last_change":"2023-10-19T14:21:10.438066+0000","last_active":"2023-10-19T14:21:27.522619+0000","last_peered":"2023-10-19T14:21:27.522619+0000","last_clean":"2023-10-19T14:21:27.522619+0000","last_became_active":"2023-10-19T14:21:10.437789+0000","last_became_peered":"2023-10-19T14:21:10.437789+0000","last_unstale":"2023-10-19T14:21:27.522619+0000","last_undegraded":"2023-10-19T14:21:27.522619+0000","last_fullsized":"2023-10-19T14:21:27.522619+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:51:31.199054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1f","version":"242'4","reported_seq":20,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.515118+0000","last_change":"2023-10-19T14:21:09.412253+0000","last_active":"2023-10-19T14:21:27.515118+0000","last_peered":"2023-10-19T14:21:27.515118+0000","last_clean":"2023-10-19T14:21:27.515118+0000","last_became_active":"2023-10-19T14:21:09.411979+0000","last_became_peered":"2023-10-19T14:21:09.411979+0000","last_unstale":"2023-10-19T14:21:27.515118+0000","last_undegraded":"2023-10-19T14:21:27.515118+0000","last_fullsized":"2023-10-19T14:21:27.515118+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:21:16.759464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.20","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.524261+0000","last_change":"2023-10-19T14:21:27.546161+0000","last_active":"2023-10-19T14:21:28.524261+0000","last_peered":"2023-10-19T14:21:28.524261+0000","last_clean":"2023-10-19T14:21:28.524261+0000","last_became_active":"2023-10-19T14:21:27.545854+0000","last_became_peered":"2023-10-19T14:21:27.545854+0000","last_unstale":"2023-10-19T14:21:28.524261+0000","last_undegraded":"2023-10-19T14:21:28.524261+0000","last_fullsized":"2023-10-19T14:21:28.524261+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:43:38.888599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.22","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.516831+0000","last_change":"2023-10-19T14:21:10.426693+0000","last_active":"2023-10-19T14:21:27.516831+0000","last_peered":"2023-10-19T14:21:27.516831+0000","last_clean":"2023-10-19T14:21:27.516831+0000","last_became_active":"2023-10-19T14:21:10.426439+0000","last_became_peered":"2023-10-19T14:21:10.426439+0000","last_unstale":"2023-10-19T14:21:27.516831+0000","last_undegraded":"2023-10-19T14:21:27.516831+0000","last_fullsized":"2023-10-19T14:21:27.516831+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:04:41.865811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.1d","version":"242'2","reported_seq":18,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.522334+0000","last_change":"2023-10-19T14:21:09.414225+0000","last_active":"2023-10-19T14:21:27.522334+0000","last_peered":"2023-10-19T14:21:27.522334+0000","last_clean":"2023-10-19T14:21:27.522334+0000","last_became_active":"2023-10-19T14:21:09.413679+0000","last_became_peered":"2023-10-19T14:21:09.413679+0000","last_unstale":"2023-10-19T14:21:27.522334+0000","last_undegraded":"2023-10-19T14:21:27.522334+0000","last_fullsized":"2023-10-19T14:21:27.522334+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:30:30.420013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.21","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.524463+0000","last_change":"2023-10-19T14:21:27.541516+0000","last_active":"2023-10-19T14:21:28.524463+0000","last_peered":"2023-10-19T14:21:28.524463+0000","last_clean":"2023-10-19T14:21:28.524463+0000","last_became_active":"2023-10-19T14:21:27.541266+0000","last_became_peered":"2023-10-19T14:21:27.541266+0000","last_unstale":"2023-10-19T14:21:28.524463+0000","last_undegraded":"2023-10-19T14:21:28.524463+0000","last_fullsized":"2023-10-19T14:21:28.524463+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:10:12.738332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.23","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.516714+0000","last_change":"2023-10-19T14:21:10.426221+0000","last_active":"2023-10-19T14:21:27.516714+0000","last_peered":"2023-10-19T14:21:27.516714+0000","last_clean":"2023-10-19T14:21:27.516714+0000","last_became_active":"2023-10-19T14:21:10.425945+0000","last_became_peered":"2023-10-19T14:21:10.425945+0000","last_unstale":"2023-10-19T14:21:27.516714+0000","last_undegraded":"2023-10-19T14:21:27.516714+0000","last_fullsized":"2023-10-19T14:21:27.516714+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:34:44.606413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1c","version":"242'2","reported_seq":18,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.535579+0000","last_change":"2023-10-19T14:21:09.412165+0000","last_active":"2023-10-19T14:21:27.535579+0000","last_peered":"2023-10-19T14:21:27.535579+0000","last_clean":"2023-10-19T14:21:27.535579+0000","last_became_active":"2023-10-19T14:21:09.411874+0000","last_became_peered":"2023-10-19T14:21:09.411874+0000","last_unstale":"2023-10-19T14:21:27.535579+0000","last_undegraded":"2023-10-19T14:21:27.535579+0000","last_fullsized":"2023-10-19T14:21:27.535579+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:14:02.794831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.26","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.525340+0000","last_change":"2023-10-19T14:21:27.536718+0000","last_active":"2023-10-19T14:21:28.525340+0000","last_peered":"2023-10-19T14:21:28.525340+0000","last_clean":"2023-10-19T14:21:28.525340+0000","last_became_active":"2023-10-19T14:21:27.536417+0000","last_became_peered":"2023-10-19T14:21:27.536417+0000","last_unstale":"2023-10-19T14:21:28.525340+0000","last_undegraded":"2023-10-19T14:21:28.525340+0000","last_fullsized":"2023-10-19T14:21:28.525340+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:03:20.484441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.24","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.535488+0000","last_change":"2023-10-19T14:21:10.430265+0000","last_active":"2023-10-19T14:21:27.535488+0000","last_peered":"2023-10-19T14:21:27.535488+0000","last_clean":"2023-10-19T14:21:27.535488+0000","last_became_active":"2023-10-19T14:21:10.429918+0000","last_became_peered":"2023-10-19T14:21:10.429918+0000","last_unstale":"2023-10-19T14:21:27.535488+0000","last_undegraded":"2023-10-19T14:21:27.535488+0000","last_fullsized":"2023-10-19T14:21:27.535488+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:02:23.852783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.1b","version":"242'1","reported_seq":17,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.517545+0000","last_change":"2023-10-19T14:21:09.411062+0000","last_active":"2023-10-19T14:21:27.517545+0000","last_peered":"2023-10-19T14:21:27.517545+0000","last_clean":"2023-10-19T14:21:27.517545+0000","last_became_active":"2023-10-19T14:21:09.410715+0000","last_became_peered":"2023-10-19T14:21:09.410715+0000","last_unstale":"2023-10-19T14:21:27.517545+0000","last_undegraded":"2023-10-19T14:21:27.517545+0000","last_fullsized":"2023-10-19T14:21:27.517545+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:46:56.529404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.27","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.527448+0000","last_change":"2023-10-19T14:21:27.522301+0000","last_active":"2023-10-19T14:21:28.527448+0000","last_peered":"2023-10-19T14:21:28.527448+0000","last_clean":"2023-10-19T14:21:28.527448+0000","last_became_active":"2023-10-19T14:21:27.520013+0000","last_became_peered":"2023-10-19T14:21:27.520013+0000","last_unstale":"2023-10-19T14:21:28.527448+0000","last_undegraded":"2023-10-19T14:21:28.527448+0000","last_fullsized":"2023-10-19T14:21:28.527448+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:22:00.351535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.25","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.512641+0000","last_change":"2023-10-19T14:21:10.422856+0000","last_active":"2023-10-19T14:21:27.512641+0000","last_peered":"2023-10-19T14:21:27.512641+0000","last_clean":"2023-10-19T14:21:27.512641+0000","last_became_active":"2023-10-19T14:21:10.422047+0000","last_became_peered":"2023-10-19T14:21:10.422047+0000","last_unstale":"2023-10-19T14:21:27.512641+0000","last_undegraded":"2023-10-19T14:21:27.512641+0000","last_fullsized":"2023-10-19T14:21:27.512641+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:35:21.048326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.1a","version":"242'1","reported_seq":17,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.536148+0000","last_change":"2023-10-19T14:21:09.408272+0000","last_active":"2023-10-19T14:21:27.536148+0000","last_peered":"2023-10-19T14:21:27.536148+0000","last_clean":"2023-10-19T14:21:27.536148+0000","last_became_active":"2023-10-19T14:21:09.407997+0000","last_became_peered":"2023-10-19T14:21:09.407997+0000","last_unstale":"2023-10-19T14:21:27.536148+0000","last_undegraded":"2023-10-19T14:21:27.536148+0000","last_fullsized":"2023-10-19T14:21:27.536148+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:32:11.516962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.24","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.592671+0000","last_change":"2023-10-19T14:21:27.532654+0000","last_active":"2023-10-19T14:21:28.592671+0000","last_peered":"2023-10-19T14:21:28.592671+0000","last_clean":"2023-10-19T14:21:28.592671+0000","last_became_active":"2023-10-19T14:21:27.532435+0000","last_became_peered":"2023-10-19T14:21:27.532435+0000","last_unstale":"2023-10-19T14:21:28.592671+0000","last_undegraded":"2023-10-19T14:21:28.592671+0000","last_fullsized":"2023-10-19T14:21:28.592671+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:03:01.076409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.26","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.512001+0000","last_change":"2023-10-19T14:21:10.427832+0000","last_active":"2023-10-19T14:21:27.512001+0000","last_peered":"2023-10-19T14:21:27.512001+0000","last_clean":"2023-10-19T14:21:27.512001+0000","last_became_active":"2023-10-19T14:21:10.427540+0000","last_became_peered":"2023-10-19T14:21:10.427540+0000","last_unstale":"2023-10-19T14:21:27.512001+0000","last_undegraded":"2023-10-19T14:21:27.512001+0000","last_fullsized":"2023-10-19T14:21:27.512001+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:47:17.356392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":16,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.510726+0000","last_change":"2023-10-19T14:21:09.408331+0000","last_active":"2023-10-19T14:21:27.510726+0000","last_peered":"2023-10-19T14:21:27.510726+0000","last_clean":"2023-10-19T14:21:27.510726+0000","last_became_active":"2023-10-19T14:21:09.407722+0000","last_became_peered":"2023-10-19T14:21:09.407722+0000","last_unstale":"2023-10-19T14:21:27.510726+0000","last_undegraded":"2023-10-19T14:21:27.510726+0000","last_fullsized":"2023-10-19T14:21:27.510726+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:06:49.485559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.25","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.591086+0000","last_change":"2023-10-19T14:21:27.531503+0000","last_active":"2023-10-19T14:21:28.591086+0000","last_peered":"2023-10-19T14:21:28.591086+0000","last_clean":"2023-10-19T14:21:28.591086+0000","last_became_active":"2023-10-19T14:21:27.530519+0000","last_became_peered":"2023-10-19T14:21:27.530519+0000","last_unstale":"2023-10-19T14:21:28.591086+0000","last_undegraded":"2023-10-19T14:21:28.591086+0000","last_fullsized":"2023-10-19T14:21:28.591086+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:39:22.559682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.27","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.517103+0000","last_change":"2023-10-19T14:21:10.425541+0000","last_active":"2023-10-19T14:21:27.517103+0000","last_peered":"2023-10-19T14:21:27.517103+0000","last_clean":"2023-10-19T14:21:27.517103+0000","last_became_active":"2023-10-19T14:21:10.423860+0000","last_became_peered":"2023-10-19T14:21:10.423860+0000","last_unstale":"2023-10-19T14:21:27.517103+0000","last_undegraded":"2023-10-19T14:21:27.517103+0000","last_fullsized":"2023-10-19T14:21:27.517103+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:41:24.205533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":16,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.515914+0000","last_change":"2023-10-19T14:21:09.411121+0000","last_active":"2023-10-19T14:21:27.515914+0000","last_peered":"2023-10-19T14:21:27.515914+0000","last_clean":"2023-10-19T14:21:27.515914+0000","last_became_active":"2023-10-19T14:21:09.409416+0000","last_became_peered":"2023-10-19T14:21:09.409416+0000","last_unstale":"2023-10-19T14:21:27.515914+0000","last_undegraded":"2023-10-19T14:21:27.515914+0000","last_fullsized":"2023-10-19T14:21:27.515914+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:09:01.088634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.2a","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.592875+0000","last_change":"2023-10-19T14:21:27.542992+0000","last_active":"2023-10-19T14:21:28.592875+0000","last_peered":"2023-10-19T14:21:28.592875+0000","last_clean":"2023-10-19T14:21:28.592875+0000","last_became_active":"2023-10-19T14:21:27.542737+0000","last_became_peered":"2023-10-19T14:21:27.542737+0000","last_unstale":"2023-10-19T14:21:28.592875+0000","last_undegraded":"2023-10-19T14:21:28.592875+0000","last_fullsized":"2023-10-19T14:21:28.592875+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:02:52.850382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.28","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.512160+0000","last_change":"2023-10-19T14:21:10.422391+0000","last_active":"2023-10-19T14:21:27.512160+0000","last_peered":"2023-10-19T14:21:27.512160+0000","last_clean":"2023-10-19T14:21:27.512160+0000","last_became_active":"2023-10-19T14:21:10.422118+0000","last_became_peered":"2023-10-19T14:21:10.422118+0000","last_unstale":"2023-10-19T14:21:27.512160+0000","last_undegraded":"2023-10-19T14:21:27.512160+0000","last_fullsized":"2023-10-19T14:21:27.512160+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T02:12:47.154435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":16,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.537368+0000","last_change":"2023-10-19T14:21:09.412983+0000","last_active":"2023-10-19T14:21:27.537368+0000","last_peered":"2023-10-19T14:21:27.537368+0000","last_clean":"2023-10-19T14:21:27.537368+0000","last_became_active":"2023-10-19T14:21:09.412690+0000","last_became_peered":"2023-10-19T14:21:09.412690+0000","last_unstale":"2023-10-19T14:21:27.537368+0000","last_undegraded":"2023-10-19T14:21:27.537368+0000","last_fullsized":"2023-10-19T14:21:27.537368+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:21:42.383177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.2b","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.589985+0000","last_change":"2023-10-19T14:21:27.527226+0000","last_active":"2023-10-19T14:21:28.589985+0000","last_peered":"2023-10-19T14:21:28.589985+0000","last_clean":"2023-10-19T14:21:28.589985+0000","last_became_active":"2023-10-19T14:21:27.526880+0000","last_became_peered":"2023-10-19T14:21:27.526880+0000","last_unstale":"2023-10-19T14:21:28.589985+0000","last_undegraded":"2023-10-19T14:21:28.589985+0000","last_fullsized":"2023-10-19T14:21:28.589985+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:06:38.822458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.29","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.516657+0000","last_change":"2023-10-19T14:21:10.427650+0000","last_active":"2023-10-19T14:21:27.516657+0000","last_peered":"2023-10-19T14:21:27.516657+0000","last_clean":"2023-10-19T14:21:27.516657+0000","last_became_active":"2023-10-19T14:21:10.427367+0000","last_became_peered":"2023-10-19T14:21:10.427367+0000","last_unstale":"2023-10-19T14:21:27.516657+0000","last_undegraded":"2023-10-19T14:21:27.516657+0000","last_fullsized":"2023-10-19T14:21:27.516657+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:54:17.097512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":16,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.512311+0000","last_change":"2023-10-19T14:21:09.408912+0000","last_active":"2023-10-19T14:21:27.512311+0000","last_peered":"2023-10-19T14:21:27.512311+0000","last_clean":"2023-10-19T14:21:27.512311+0000","last_became_active":"2023-10-19T14:21:09.408661+0000","last_became_peered":"2023-10-19T14:21:09.408661+0000","last_unstale":"2023-10-19T14:21:27.512311+0000","last_undegraded":"2023-10-19T14:21:27.512311+0000","last_fullsized":"2023-10-19T14:21:27.512311+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:55:16.857948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.28","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.590553+0000","last_change":"2023-10-19T14:21:27.529521+0000","last_active":"2023-10-19T14:21:28.590553+0000","last_peered":"2023-10-19T14:21:28.590553+0000","last_clean":"2023-10-19T14:21:28.590553+0000","last_became_active":"2023-10-19T14:21:27.529236+0000","last_became_peered":"2023-10-19T14:21:27.529236+0000","last_unstale":"2023-10-19T14:21:28.590553+0000","last_undegraded":"2023-10-19T14:21:28.590553+0000","last_fullsized":"2023-10-19T14:21:28.590553+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:48:54.397084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.2a","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.511256+0000","last_change":"2023-10-19T14:21:10.430153+0000","last_active":"2023-10-19T14:21:27.511256+0000","last_peered":"2023-10-19T14:21:27.511256+0000","last_clean":"2023-10-19T14:21:27.511256+0000","last_became_active":"2023-10-19T14:21:10.429868+0000","last_became_peered":"2023-10-19T14:21:10.429868+0000","last_unstale":"2023-10-19T14:21:27.511256+0000","last_undegraded":"2023-10-19T14:21:27.511256+0000","last_fullsized":"2023-10-19T14:21:27.511256+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:55:49.837750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.15","version":"242'3","reported_seq":19,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.517746+0000","last_change":"2023-10-19T14:21:09.414620+0000","last_active":"2023-10-19T14:21:27.517746+0000","last_peered":"2023-10-19T14:21:27.517746+0000","last_clean":"2023-10-19T14:21:27.517746+0000","last_became_active":"2023-10-19T14:21:09.414378+0000","last_became_peered":"2023-10-19T14:21:09.414378+0000","last_unstale":"2023-10-19T14:21:27.517746+0000","last_undegraded":"2023-10-19T14:21:27.517746+0000","last_fullsized":"2023-10-19T14:21:27.517746+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:16:10.702399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.29","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.524309+0000","last_change":"2023-10-19T14:21:27.536121+0000","last_active":"2023-10-19T14:21:28.524309+0000","last_peered":"2023-10-19T14:21:28.524309+0000","last_clean":"2023-10-19T14:21:28.524309+0000","last_became_active":"2023-10-19T14:21:27.535825+0000","last_became_peered":"2023-10-19T14:21:27.535825+0000","last_unstale":"2023-10-19T14:21:28.524309+0000","last_undegraded":"2023-10-19T14:21:28.524309+0000","last_fullsized":"2023-10-19T14:21:28.524309+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:51:08.953710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.2b","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.512405+0000","last_change":"2023-10-19T14:21:10.436778+0000","last_active":"2023-10-19T14:21:27.512405+0000","last_peered":"2023-10-19T14:21:27.512405+0000","last_clean":"2023-10-19T14:21:27.512405+0000","last_became_active":"2023-10-19T14:21:10.436506+0000","last_became_peered":"2023-10-19T14:21:10.436506+0000","last_unstale":"2023-10-19T14:21:27.512405+0000","last_undegraded":"2023-10-19T14:21:27.512405+0000","last_fullsized":"2023-10-19T14:21:27.512405+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:02:47.768820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.14","version":"244'9","reported_seq":25,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.515817+0000","last_change":"2023-10-19T14:21:09.411345+0000","last_active":"2023-10-19T14:21:27.515817+0000","last_peered":"2023-10-19T14:21:27.515817+0000","last_clean":"2023-10-19T14:21:27.515817+0000","last_became_active":"2023-10-19T14:21:09.410711+0000","last_became_peered":"2023-10-19T14:21:09.410711+0000","last_unstale":"2023-10-19T14:21:27.515817+0000","last_undegraded":"2023-10-19T14:21:27.515817+0000","last_fullsized":"2023-10-19T14:21:27.515817+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:35:21.400082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2868,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.2e","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.591360+0000","last_change":"2023-10-19T14:21:27.529607+0000","last_active":"2023-10-19T14:21:28.591360+0000","last_peered":"2023-10-19T14:21:28.591360+0000","last_clean":"2023-10-19T14:21:28.591360+0000","last_became_active":"2023-10-19T14:21:27.529260+0000","last_became_peered":"2023-10-19T14:21:27.529260+0000","last_unstale":"2023-10-19T14:21:28.591360+0000","last_undegraded":"2023-10-19T14:21:28.591360+0000","last_fullsized":"2023-10-19T14:21:28.591360+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:07:06.629090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.2c","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.521816+0000","last_change":"2023-10-19T14:21:10.435528+0000","last_active":"2023-10-19T14:21:27.521816+0000","last_peered":"2023-10-19T14:21:27.521816+0000","last_clean":"2023-10-19T14:21:27.521816+0000","last_became_active":"2023-10-19T14:21:10.435235+0000","last_became_peered":"2023-10-19T14:21:10.435235+0000","last_unstale":"2023-10-19T14:21:27.521816+0000","last_undegraded":"2023-10-19T14:21:27.521816+0000","last_fullsized":"2023-10-19T14:21:27.521816+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:24:42.687537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.13","version":"242'2","reported_seq":18,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.518287+0000","last_change":"2023-10-19T14:21:09.413386+0000","last_active":"2023-10-19T14:21:27.518287+0000","last_peered":"2023-10-19T14:21:27.518287+0000","last_clean":"2023-10-19T14:21:27.518287+0000","last_became_active":"2023-10-19T14:21:09.413119+0000","last_became_peered":"2023-10-19T14:21:09.413119+0000","last_unstale":"2023-10-19T14:21:27.518287+0000","last_undegraded":"2023-10-19T14:21:27.518287+0000","last_fullsized":"2023-10-19T14:21:27.518287+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:32:53.438251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.2f","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.592793+0000","last_change":"2023-10-19T14:21:27.531259+0000","last_active":"2023-10-19T14:21:28.592793+0000","last_peered":"2023-10-19T14:21:28.592793+0000","last_clean":"2023-10-19T14:21:28.592793+0000","last_became_active":"2023-10-19T14:21:27.530949+0000","last_became_peered":"2023-10-19T14:21:27.530949+0000","last_unstale":"2023-10-19T14:21:28.592793+0000","last_undegraded":"2023-10-19T14:21:28.592793+0000","last_fullsized":"2023-10-19T14:21:28.592793+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:11:42.825804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2d","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.512418+0000","last_change":"2023-10-19T14:21:10.417244+0000","last_active":"2023-10-19T14:21:27.512418+0000","last_peered":"2023-10-19T14:21:27.512418+0000","last_clean":"2023-10-19T14:21:27.512418+0000","last_became_active":"2023-10-19T14:21:10.416948+0000","last_became_peered":"2023-10-19T14:21:10.416948+0000","last_unstale":"2023-10-19T14:21:27.512418+0000","last_undegraded":"2023-10-19T14:21:27.512418+0000","last_fullsized":"2023-10-19T14:21:27.512418+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:13:29.800561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":16,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.516499+0000","last_change":"2023-10-19T14:21:09.417189+0000","last_active":"2023-10-19T14:21:27.516499+0000","last_peered":"2023-10-19T14:21:27.516499+0000","last_clean":"2023-10-19T14:21:27.516499+0000","last_became_active":"2023-10-19T14:21:09.416920+0000","last_became_peered":"2023-10-19T14:21:09.416920+0000","last_unstale":"2023-10-19T14:21:27.516499+0000","last_undegraded":"2023-10-19T14:21:27.516499+0000","last_fullsized":"2023-10-19T14:21:27.516499+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:17:36.285741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.2c","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.591881+0000","last_change":"2023-10-19T14:21:27.526720+0000","last_active":"2023-10-19T14:21:28.591881+0000","last_peered":"2023-10-19T14:21:28.591881+0000","last_clean":"2023-10-19T14:21:28.591881+0000","last_became_active":"2023-10-19T14:21:27.526430+0000","last_became_peered":"2023-10-19T14:21:27.526430+0000","last_unstale":"2023-10-19T14:21:28.591881+0000","last_undegraded":"2023-10-19T14:21:28.591881+0000","last_fullsized":"2023-10-19T14:21:28.591881+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:45:48.739493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.2e","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.518681+0000","last_change":"2023-10-19T14:21:10.422922+0000","last_active":"2023-10-19T14:21:27.518681+0000","last_peered":"2023-10-19T14:21:27.518681+0000","last_clean":"2023-10-19T14:21:27.518681+0000","last_became_active":"2023-10-19T14:21:10.422656+0000","last_became_peered":"2023-10-19T14:21:10.422656+0000","last_unstale":"2023-10-19T14:21:27.518681+0000","last_undegraded":"2023-10-19T14:21:27.518681+0000","last_fullsized":"2023-10-19T14:21:27.518681+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:16:55.773348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":16,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.517654+0000","last_change":"2023-10-19T14:21:09.413176+0000","last_active":"2023-10-19T14:21:27.517654+0000","last_peered":"2023-10-19T14:21:27.517654+0000","last_clean":"2023-10-19T14:21:27.517654+0000","last_became_active":"2023-10-19T14:21:09.412906+0000","last_became_peered":"2023-10-19T14:21:09.412906+0000","last_unstale":"2023-10-19T14:21:27.517654+0000","last_undegraded":"2023-10-19T14:21:27.517654+0000","last_fullsized":"2023-10-19T14:21:27.517654+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:07:02.508240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.2d","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.592222+0000","last_change":"2023-10-19T14:21:27.534816+0000","last_active":"2023-10-19T14:21:28.592222+0000","last_peered":"2023-10-19T14:21:28.592222+0000","last_clean":"2023-10-19T14:21:28.592222+0000","last_became_active":"2023-10-19T14:21:27.534490+0000","last_became_peered":"2023-10-19T14:21:27.534490+0000","last_unstale":"2023-10-19T14:21:28.592222+0000","last_undegraded":"2023-10-19T14:21:28.592222+0000","last_fullsized":"2023-10-19T14:21:28.592222+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:47:57.932633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2f","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.517011+0000","last_change":"2023-10-19T14:21:10.431716+0000","last_active":"2023-10-19T14:21:27.517011+0000","last_peered":"2023-10-19T14:21:27.517011+0000","last_clean":"2023-10-19T14:21:27.517011+0000","last_became_active":"2023-10-19T14:21:10.430784+0000","last_became_peered":"2023-10-19T14:21:10.430784+0000","last_unstale":"2023-10-19T14:21:27.517011+0000","last_undegraded":"2023-10-19T14:21:27.517011+0000","last_fullsized":"2023-10-19T14:21:27.517011+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:14:44.806142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.10","version":"242'8","reported_seq":24,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.522719+0000","last_change":"2023-10-19T14:21:09.415001+0000","last_active":"2023-10-19T14:21:27.522719+0000","last_peered":"2023-10-19T14:21:27.522719+0000","last_clean":"2023-10-19T14:21:27.522719+0000","last_became_active":"2023-10-19T14:21:09.414727+0000","last_became_peered":"2023-10-19T14:21:09.414727+0000","last_unstale":"2023-10-19T14:21:27.522719+0000","last_undegraded":"2023-10-19T14:21:27.522719+0000","last_fullsized":"2023-10-19T14:21:27.522719+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:54:37.202858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.32","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.524440+0000","last_change":"2023-10-19T14:21:27.541645+0000","last_active":"2023-10-19T14:21:28.524440+0000","last_peered":"2023-10-19T14:21:28.524440+0000","last_clean":"2023-10-19T14:21:28.524440+0000","last_became_active":"2023-10-19T14:21:27.541380+0000","last_became_peered":"2023-10-19T14:21:27.541380+0000","last_unstale":"2023-10-19T14:21:28.524440+0000","last_undegraded":"2023-10-19T14:21:28.524440+0000","last_fullsized":"2023-10-19T14:21:28.524440+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:24:40.034377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.30","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.518001+0000","last_change":"2023-10-19T14:21:10.437415+0000","last_active":"2023-10-19T14:21:27.518001+0000","last_peered":"2023-10-19T14:21:27.518001+0000","last_clean":"2023-10-19T14:21:27.518001+0000","last_became_active":"2023-10-19T14:21:10.437137+0000","last_became_peered":"2023-10-19T14:21:10.437137+0000","last_unstale":"2023-10-19T14:21:27.518001+0000","last_undegraded":"2023-10-19T14:21:27.518001+0000","last_fullsized":"2023-10-19T14:21:27.518001+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:39:56.303696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.f","version":"242'1","reported_seq":17,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.515277+0000","last_change":"2023-10-19T14:21:09.417455+0000","last_active":"2023-10-19T14:21:27.515277+0000","last_peered":"2023-10-19T14:21:27.515277+0000","last_clean":"2023-10-19T14:21:27.515277+0000","last_became_active":"2023-10-19T14:21:09.417228+0000","last_became_peered":"2023-10-19T14:21:09.417228+0000","last_unstale":"2023-10-19T14:21:27.515277+0000","last_undegraded":"2023-10-19T14:21:27.515277+0000","last_fullsized":"2023-10-19T14:21:27.515277+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:37:45.851912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.33","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.526774+0000","last_change":"2023-10-19T14:21:27.540879+0000","last_active":"2023-10-19T14:21:28.526774+0000","last_peered":"2023-10-19T14:21:28.526774+0000","last_clean":"2023-10-19T14:21:28.526774+0000","last_became_active":"2023-10-19T14:21:27.539190+0000","last_became_peered":"2023-10-19T14:21:27.539190+0000","last_unstale":"2023-10-19T14:21:28.526774+0000","last_undegraded":"2023-10-19T14:21:28.526774+0000","last_fullsized":"2023-10-19T14:21:28.526774+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:36:31.003136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.31","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.517323+0000","last_change":"2023-10-19T14:21:10.423316+0000","last_active":"2023-10-19T14:21:27.517323+0000","last_peered":"2023-10-19T14:21:27.517323+0000","last_clean":"2023-10-19T14:21:27.517323+0000","last_became_active":"2023-10-19T14:21:10.423028+0000","last_became_peered":"2023-10-19T14:21:10.423028+0000","last_unstale":"2023-10-19T14:21:27.517323+0000","last_undegraded":"2023-10-19T14:21:27.517323+0000","last_fullsized":"2023-10-19T14:21:27.517323+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:39:05.601924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.e","version":"242'2","reported_seq":18,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.523435+0000","last_change":"2023-10-19T14:21:09.409551+0000","last_active":"2023-10-19T14:21:27.523435+0000","last_peered":"2023-10-19T14:21:27.523435+0000","last_clean":"2023-10-19T14:21:27.523435+0000","last_became_active":"2023-10-19T14:21:09.409238+0000","last_became_peered":"2023-10-19T14:21:09.409238+0000","last_unstale":"2023-10-19T14:21:27.523435+0000","last_undegraded":"2023-10-19T14:21:27.523435+0000","last_fullsized":"2023-10-19T14:21:27.523435+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:38:52.516509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.30","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.591552+0000","last_change":"2023-10-19T14:21:27.535573+0000","last_active":"2023-10-19T14:21:28.591552+0000","last_peered":"2023-10-19T14:21:28.591552+0000","last_clean":"2023-10-19T14:21:28.591552+0000","last_became_active":"2023-10-19T14:21:27.534545+0000","last_became_peered":"2023-10-19T14:21:27.534545+0000","last_unstale":"2023-10-19T14:21:28.591552+0000","last_undegraded":"2023-10-19T14:21:28.591552+0000","last_fullsized":"2023-10-19T14:21:28.591552+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:03:49.950096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.32","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.522504+0000","last_change":"2023-10-19T14:21:10.433187+0000","last_active":"2023-10-19T14:21:27.522504+0000","last_peered":"2023-10-19T14:21:27.522504+0000","last_clean":"2023-10-19T14:21:27.522504+0000","last_became_active":"2023-10-19T14:21:10.432876+0000","last_became_peered":"2023-10-19T14:21:10.432876+0000","last_unstale":"2023-10-19T14:21:27.522504+0000","last_undegraded":"2023-10-19T14:21:27.522504+0000","last_fullsized":"2023-10-19T14:21:27.522504+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:49:35.448257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.d","version":"242'1","reported_seq":17,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.522407+0000","last_change":"2023-10-19T14:21:09.413076+0000","last_active":"2023-10-19T14:21:27.522407+0000","last_peered":"2023-10-19T14:21:27.522407+0000","last_clean":"2023-10-19T14:21:27.522407+0000","last_became_active":"2023-10-19T14:21:09.412330+0000","last_became_peered":"2023-10-19T14:21:09.412330+0000","last_unstale":"2023-10-19T14:21:27.522407+0000","last_undegraded":"2023-10-19T14:21:27.522407+0000","last_fullsized":"2023-10-19T14:21:27.522407+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:20:50.841124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.31","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.524325+0000","last_change":"2023-10-19T14:21:27.534582+0000","last_active":"2023-10-19T14:21:28.524325+0000","last_peered":"2023-10-19T14:21:28.524325+0000","last_clean":"2023-10-19T14:21:28.524325+0000","last_became_active":"2023-10-19T14:21:27.534261+0000","last_became_peered":"2023-10-19T14:21:27.534261+0000","last_unstale":"2023-10-19T14:21:28.524325+0000","last_undegraded":"2023-10-19T14:21:28.524325+0000","last_fullsized":"2023-10-19T14:21:28.524325+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:41:18.907883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.33","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.512259+0000","last_change":"2023-10-19T14:21:10.429531+0000","last_active":"2023-10-19T14:21:27.512259+0000","last_peered":"2023-10-19T14:21:27.512259+0000","last_clean":"2023-10-19T14:21:27.512259+0000","last_became_active":"2023-10-19T14:21:10.429261+0000","last_became_peered":"2023-10-19T14:21:10.429261+0000","last_unstale":"2023-10-19T14:21:27.512259+0000","last_undegraded":"2023-10-19T14:21:27.512259+0000","last_fullsized":"2023-10-19T14:21:27.512259+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:29:47.656632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":16,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.523428+0000","last_change":"2023-10-19T14:21:09.409030+0000","last_active":"2023-10-19T14:21:27.523428+0000","last_peered":"2023-10-19T14:21:27.523428+0000","last_clean":"2023-10-19T14:21:27.523428+0000","last_became_active":"2023-10-19T14:21:09.408717+0000","last_became_peered":"2023-10-19T14:21:09.408717+0000","last_unstale":"2023-10-19T14:21:27.523428+0000","last_undegraded":"2023-10-19T14:21:27.523428+0000","last_fullsized":"2023-10-19T14:21:27.523428+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:04:32.926013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.36","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.592580+0000","last_change":"2023-10-19T14:21:27.521181+0000","last_active":"2023-10-19T14:21:28.592580+0000","last_peered":"2023-10-19T14:21:28.592580+0000","last_clean":"2023-10-19T14:21:28.592580+0000","last_became_active":"2023-10-19T14:21:27.520913+0000","last_became_peered":"2023-10-19T14:21:27.520913+0000","last_unstale":"2023-10-19T14:21:28.592580+0000","last_undegraded":"2023-10-19T14:21:28.592580+0000","last_fullsized":"2023-10-19T14:21:28.592580+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:21:46.163073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.34","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.511624+0000","last_change":"2023-10-19T14:21:10.437932+0000","last_active":"2023-10-19T14:21:27.511624+0000","last_peered":"2023-10-19T14:21:27.511624+0000","last_clean":"2023-10-19T14:21:27.511624+0000","last_became_active":"2023-10-19T14:21:10.437668+0000","last_became_peered":"2023-10-19T14:21:10.437668+0000","last_unstale":"2023-10-19T14:21:27.511624+0000","last_undegraded":"2023-10-19T14:21:27.511624+0000","last_fullsized":"2023-10-19T14:21:27.511624+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:04:33.936613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.b","version":"242'2","reported_seq":18,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.512364+0000","last_change":"2023-10-19T14:21:09.415147+0000","last_active":"2023-10-19T14:21:27.512364+0000","last_peered":"2023-10-19T14:21:27.512364+0000","last_clean":"2023-10-19T14:21:27.512364+0000","last_became_active":"2023-10-19T14:21:09.414852+0000","last_became_peered":"2023-10-19T14:21:09.414852+0000","last_unstale":"2023-10-19T14:21:27.512364+0000","last_undegraded":"2023-10-19T14:21:27.512364+0000","last_fullsized":"2023-10-19T14:21:27.512364+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:53:33.599617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.37","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.591838+0000","last_change":"2023-10-19T14:21:27.530671+0000","last_active":"2023-10-19T14:21:28.591838+0000","last_peered":"2023-10-19T14:21:28.591838+0000","last_clean":"2023-10-19T14:21:28.591838+0000","last_became_active":"2023-10-19T14:21:27.530249+0000","last_became_peered":"2023-10-19T14:21:27.530249+0000","last_unstale":"2023-10-19T14:21:28.591838+0000","last_undegraded":"2023-10-19T14:21:28.591838+0000","last_fullsized":"2023-10-19T14:21:28.591838+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:33:01.768540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.35","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.521989+0000","last_change":"2023-10-19T14:21:10.439769+0000","last_active":"2023-10-19T14:21:27.521989+0000","last_peered":"2023-10-19T14:21:27.521989+0000","last_clean":"2023-10-19T14:21:27.521989+0000","last_became_active":"2023-10-19T14:21:10.439056+0000","last_became_peered":"2023-10-19T14:21:10.439056+0000","last_unstale":"2023-10-19T14:21:27.521989+0000","last_undegraded":"2023-10-19T14:21:27.521989+0000","last_fullsized":"2023-10-19T14:21:27.521989+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:18:35.286014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":16,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.521980+0000","last_change":"2023-10-19T14:21:09.411613+0000","last_active":"2023-10-19T14:21:27.521980+0000","last_peered":"2023-10-19T14:21:27.521980+0000","last_clean":"2023-10-19T14:21:27.521980+0000","last_became_active":"2023-10-19T14:21:09.411365+0000","last_became_peered":"2023-10-19T14:21:09.411365+0000","last_unstale":"2023-10-19T14:21:27.521980+0000","last_undegraded":"2023-10-19T14:21:27.521980+0000","last_fullsized":"2023-10-19T14:21:27.521980+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:22:34.404197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.34","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.591670+0000","last_change":"2023-10-19T14:21:27.524569+0000","last_active":"2023-10-19T14:21:28.591670+0000","last_peered":"2023-10-19T14:21:28.591670+0000","last_clean":"2023-10-19T14:21:28.591670+0000","last_became_active":"2023-10-19T14:21:27.524247+0000","last_became_peered":"2023-10-19T14:21:27.524247+0000","last_unstale":"2023-10-19T14:21:28.591670+0000","last_undegraded":"2023-10-19T14:21:28.591670+0000","last_fullsized":"2023-10-19T14:21:28.591670+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:19:40.772028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.36","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.524514+0000","last_change":"2023-10-19T14:21:10.424290+0000","last_active":"2023-10-19T14:21:27.524514+0000","last_peered":"2023-10-19T14:21:27.524514+0000","last_clean":"2023-10-19T14:21:27.524514+0000","last_became_active":"2023-10-19T14:21:10.423951+0000","last_became_peered":"2023-10-19T14:21:10.423951+0000","last_unstale":"2023-10-19T14:21:27.524514+0000","last_undegraded":"2023-10-19T14:21:27.524514+0000","last_fullsized":"2023-10-19T14:21:27.524514+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:43:48.977855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":16,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.516770+0000","last_change":"2023-10-19T14:21:09.410869+0000","last_active":"2023-10-19T14:21:27.516770+0000","last_peered":"2023-10-19T14:21:27.516770+0000","last_clean":"2023-10-19T14:21:27.516770+0000","last_became_active":"2023-10-19T14:21:09.410586+0000","last_became_peered":"2023-10-19T14:21:09.410586+0000","last_unstale":"2023-10-19T14:21:27.516770+0000","last_undegraded":"2023-10-19T14:21:27.516770+0000","last_fullsized":"2023-10-19T14:21:27.516770+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:02:42.839507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.35","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.590903+0000","last_change":"2023-10-19T14:21:27.528697+0000","last_active":"2023-10-19T14:21:28.590903+0000","last_peered":"2023-10-19T14:21:28.590903+0000","last_clean":"2023-10-19T14:21:28.590903+0000","last_became_active":"2023-10-19T14:21:27.528441+0000","last_became_peered":"2023-10-19T14:21:27.528441+0000","last_unstale":"2023-10-19T14:21:28.590903+0000","last_undegraded":"2023-10-19T14:21:28.590903+0000","last_fullsized":"2023-10-19T14:21:28.590903+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:30:18.406194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.37","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.525586+0000","last_change":"2023-10-19T14:21:10.422504+0000","last_active":"2023-10-19T14:21:27.525586+0000","last_peered":"2023-10-19T14:21:27.525586+0000","last_clean":"2023-10-19T14:21:27.525586+0000","last_became_active":"2023-10-19T14:21:10.422196+0000","last_became_peered":"2023-10-19T14:21:10.422196+0000","last_unstale":"2023-10-19T14:21:27.525586+0000","last_undegraded":"2023-10-19T14:21:27.525586+0000","last_fullsized":"2023-10-19T14:21:27.525586+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:53:46.360770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":16,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.526319+0000","last_change":"2023-10-19T14:21:09.415870+0000","last_active":"2023-10-19T14:21:27.526319+0000","last_peered":"2023-10-19T14:21:27.526319+0000","last_clean":"2023-10-19T14:21:27.526319+0000","last_became_active":"2023-10-19T14:21:09.415609+0000","last_became_peered":"2023-10-19T14:21:09.415609+0000","last_unstale":"2023-10-19T14:21:27.526319+0000","last_undegraded":"2023-10-19T14:21:27.526319+0000","last_fullsized":"2023-10-19T14:21:27.526319+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:29:10.891884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.3a","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.591778+0000","last_change":"2023-10-19T14:21:27.521038+0000","last_active":"2023-10-19T14:21:28.591778+0000","last_peered":"2023-10-19T14:21:28.591778+0000","last_clean":"2023-10-19T14:21:28.591778+0000","last_became_active":"2023-10-19T14:21:27.519463+0000","last_became_peered":"2023-10-19T14:21:27.519463+0000","last_unstale":"2023-10-19T14:21:28.591778+0000","last_undegraded":"2023-10-19T14:21:28.591778+0000","last_fullsized":"2023-10-19T14:21:28.591778+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:27:02.150067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.38","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.512047+0000","last_change":"2023-10-19T14:21:10.428770+0000","last_active":"2023-10-19T14:21:27.512047+0000","last_peered":"2023-10-19T14:21:27.512047+0000","last_clean":"2023-10-19T14:21:27.512047+0000","last_became_active":"2023-10-19T14:21:10.428494+0000","last_became_peered":"2023-10-19T14:21:10.428494+0000","last_unstale":"2023-10-19T14:21:27.512047+0000","last_undegraded":"2023-10-19T14:21:27.512047+0000","last_fullsized":"2023-10-19T14:21:27.512047+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:54:47.763268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.7","version":"242'2","reported_seq":18,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.511934+0000","last_change":"2023-10-19T14:21:09.416206+0000","last_active":"2023-10-19T14:21:27.511934+0000","last_peered":"2023-10-19T14:21:27.511934+0000","last_clean":"2023-10-19T14:21:27.511934+0000","last_became_active":"2023-10-19T14:21:09.415929+0000","last_became_peered":"2023-10-19T14:21:09.415929+0000","last_unstale":"2023-10-19T14:21:27.511934+0000","last_undegraded":"2023-10-19T14:21:27.511934+0000","last_fullsized":"2023-10-19T14:21:27.511934+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:41:27.899602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.3b","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.590724+0000","last_change":"2023-10-19T14:21:27.521444+0000","last_active":"2023-10-19T14:21:28.590724+0000","last_peered":"2023-10-19T14:21:28.590724+0000","last_clean":"2023-10-19T14:21:28.590724+0000","last_became_active":"2023-10-19T14:21:27.521158+0000","last_became_peered":"2023-10-19T14:21:27.521158+0000","last_unstale":"2023-10-19T14:21:28.590724+0000","last_undegraded":"2023-10-19T14:21:28.590724+0000","last_fullsized":"2023-10-19T14:21:28.590724+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:15:17.090163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.39","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.516124+0000","last_change":"2023-10-19T14:21:10.431238+0000","last_active":"2023-10-19T14:21:27.516124+0000","last_peered":"2023-10-19T14:21:27.516124+0000","last_clean":"2023-10-19T14:21:27.516124+0000","last_became_active":"2023-10-19T14:21:10.430970+0000","last_became_peered":"2023-10-19T14:21:10.430970+0000","last_unstale":"2023-10-19T14:21:27.516124+0000","last_undegraded":"2023-10-19T14:21:27.516124+0000","last_fullsized":"2023-10-19T14:21:27.516124+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:29:11.902484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.6","version":"243'1","reported_seq":17,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.517630+0000","last_change":"2023-10-19T14:21:09.415597+0000","last_active":"2023-10-19T14:21:27.517630+0000","last_peered":"2023-10-19T14:21:27.517630+0000","last_clean":"2023-10-19T14:21:27.517630+0000","last_became_active":"2023-10-19T14:21:09.415325+0000","last_became_peered":"2023-10-19T14:21:09.415325+0000","last_unstale":"2023-10-19T14:21:27.517630+0000","last_undegraded":"2023-10-19T14:21:27.517630+0000","last_fullsized":"2023-10-19T14:21:27.517630+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:47:10.402857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.38","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.526470+0000","last_change":"2023-10-19T14:21:27.537623+0000","last_active":"2023-10-19T14:21:28.526470+0000","last_peered":"2023-10-19T14:21:28.526470+0000","last_clean":"2023-10-19T14:21:28.526470+0000","last_became_active":"2023-10-19T14:21:27.537350+0000","last_became_peered":"2023-10-19T14:21:27.537350+0000","last_unstale":"2023-10-19T14:21:28.526470+0000","last_undegraded":"2023-10-19T14:21:28.526470+0000","last_fullsized":"2023-10-19T14:21:28.526470+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:12:47.431775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.3a","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.511130+0000","last_change":"2023-10-19T14:21:10.430013+0000","last_active":"2023-10-19T14:21:27.511130+0000","last_peered":"2023-10-19T14:21:27.511130+0000","last_clean":"2023-10-19T14:21:27.511130+0000","last_became_active":"2023-10-19T14:21:10.429744+0000","last_became_peered":"2023-10-19T14:21:10.429744+0000","last_unstale":"2023-10-19T14:21:27.511130+0000","last_undegraded":"2023-10-19T14:21:27.511130+0000","last_fullsized":"2023-10-19T14:21:27.511130+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:23:00.738206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":16,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.536369+0000","last_change":"2023-10-19T14:21:09.412015+0000","last_active":"2023-10-19T14:21:27.536369+0000","last_peered":"2023-10-19T14:21:27.536369+0000","last_clean":"2023-10-19T14:21:27.536369+0000","last_became_active":"2023-10-19T14:21:09.411729+0000","last_became_peered":"2023-10-19T14:21:09.411729+0000","last_unstale":"2023-10-19T14:21:27.536369+0000","last_undegraded":"2023-10-19T14:21:27.536369+0000","last_fullsized":"2023-10-19T14:21:27.536369+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:50:33.137069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.39","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.526763+0000","last_change":"2023-10-19T14:21:27.526140+0000","last_active":"2023-10-19T14:21:28.526763+0000","last_peered":"2023-10-19T14:21:28.526763+0000","last_clean":"2023-10-19T14:21:28.526763+0000","last_became_active":"2023-10-19T14:21:27.525851+0000","last_became_peered":"2023-10-19T14:21:27.525851+0000","last_unstale":"2023-10-19T14:21:28.526763+0000","last_undegraded":"2023-10-19T14:21:28.526763+0000","last_fullsized":"2023-10-19T14:21:28.526763+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:58:25.324263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.3b","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.519383+0000","last_change":"2023-10-19T14:21:10.430397+0000","last_active":"2023-10-19T14:21:27.519383+0000","last_peered":"2023-10-19T14:21:27.519383+0000","last_clean":"2023-10-19T14:21:27.519383+0000","last_became_active":"2023-10-19T14:21:10.430094+0000","last_became_peered":"2023-10-19T14:21:10.430094+0000","last_unstale":"2023-10-19T14:21:27.519383+0000","last_undegraded":"2023-10-19T14:21:27.519383+0000","last_fullsized":"2023-10-19T14:21:27.519383+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:02:35.892624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":16,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.535666+0000","last_change":"2023-10-19T14:21:09.414910+0000","last_active":"2023-10-19T14:21:27.535666+0000","last_peered":"2023-10-19T14:21:27.535666+0000","last_clean":"2023-10-19T14:21:27.535666+0000","last_became_active":"2023-10-19T14:21:09.414646+0000","last_became_peered":"2023-10-19T14:21:09.414646+0000","last_unstale":"2023-10-19T14:21:27.535666+0000","last_undegraded":"2023-10-19T14:21:27.535666+0000","last_fullsized":"2023-10-19T14:21:27.535666+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:49:25.343751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.3e","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.525220+0000","last_change":"2023-10-19T14:21:27.547075+0000","last_active":"2023-10-19T14:21:28.525220+0000","last_peered":"2023-10-19T14:21:28.525220+0000","last_clean":"2023-10-19T14:21:28.525220+0000","last_became_active":"2023-10-19T14:21:27.546658+0000","last_became_peered":"2023-10-19T14:21:27.546658+0000","last_unstale":"2023-10-19T14:21:28.525220+0000","last_undegraded":"2023-10-19T14:21:28.525220+0000","last_fullsized":"2023-10-19T14:21:28.525220+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:15:49.232430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.3c","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.535394+0000","last_change":"2023-10-19T14:21:10.418399+0000","last_active":"2023-10-19T14:21:27.535394+0000","last_peered":"2023-10-19T14:21:27.535394+0000","last_clean":"2023-10-19T14:21:27.535394+0000","last_became_active":"2023-10-19T14:21:10.416779+0000","last_became_peered":"2023-10-19T14:21:10.416779+0000","last_unstale":"2023-10-19T14:21:27.535394+0000","last_undegraded":"2023-10-19T14:21:27.535394+0000","last_fullsized":"2023-10-19T14:21:27.535394+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:50:13.187179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":16,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.521467+0000","last_change":"2023-10-19T14:21:09.412344+0000","last_active":"2023-10-19T14:21:27.521467+0000","last_peered":"2023-10-19T14:21:27.521467+0000","last_clean":"2023-10-19T14:21:27.521467+0000","last_became_active":"2023-10-19T14:21:09.412101+0000","last_became_peered":"2023-10-19T14:21:09.412101+0000","last_unstale":"2023-10-19T14:21:27.521467+0000","last_undegraded":"2023-10-19T14:21:27.521467+0000","last_fullsized":"2023-10-19T14:21:27.521467+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:51:23.060110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.3f","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.593033+0000","last_change":"2023-10-19T14:21:27.534871+0000","last_active":"2023-10-19T14:21:28.593033+0000","last_peered":"2023-10-19T14:21:28.593033+0000","last_clean":"2023-10-19T14:21:28.593033+0000","last_became_active":"2023-10-19T14:21:27.533520+0000","last_became_peered":"2023-10-19T14:21:27.533520+0000","last_unstale":"2023-10-19T14:21:28.593033+0000","last_undegraded":"2023-10-19T14:21:28.593033+0000","last_fullsized":"2023-10-19T14:21:28.593033+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T02:17:48.211698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.3d","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.514961+0000","last_change":"2023-10-19T14:21:10.435707+0000","last_active":"2023-10-19T14:21:27.514961+0000","last_peered":"2023-10-19T14:21:27.514961+0000","last_clean":"2023-10-19T14:21:27.514961+0000","last_became_active":"2023-10-19T14:21:10.435426+0000","last_became_peered":"2023-10-19T14:21:10.435426+0000","last_unstale":"2023-10-19T14:21:27.514961+0000","last_undegraded":"2023-10-19T14:21:27.514961+0000","last_fullsized":"2023-10-19T14:21:27.514961+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:21:50.005281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":16,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.516581+0000","last_change":"2023-10-19T14:21:09.415572+0000","last_active":"2023-10-19T14:21:27.516581+0000","last_peered":"2023-10-19T14:21:27.516581+0000","last_clean":"2023-10-19T14:21:27.516581+0000","last_became_active":"2023-10-19T14:21:09.415271+0000","last_became_peered":"2023-10-19T14:21:09.415271+0000","last_unstale":"2023-10-19T14:21:27.516581+0000","last_undegraded":"2023-10-19T14:21:27.516581+0000","last_fullsized":"2023-10-19T14:21:27.516581+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:24:02.734433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.3c","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.526681+0000","last_change":"2023-10-19T14:21:27.528251+0000","last_active":"2023-10-19T14:21:28.526681+0000","last_peered":"2023-10-19T14:21:28.526681+0000","last_clean":"2023-10-19T14:21:28.526681+0000","last_became_active":"2023-10-19T14:21:27.527967+0000","last_became_peered":"2023-10-19T14:21:27.527967+0000","last_unstale":"2023-10-19T14:21:28.526681+0000","last_undegraded":"2023-10-19T14:21:28.526681+0000","last_fullsized":"2023-10-19T14:21:28.526681+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:45:06.945233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.3e","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.523341+0000","last_change":"2023-10-19T14:21:10.439966+0000","last_active":"2023-10-19T14:21:27.523341+0000","last_peered":"2023-10-19T14:21:27.523341+0000","last_clean":"2023-10-19T14:21:27.523341+0000","last_became_active":"2023-10-19T14:21:10.438773+0000","last_became_peered":"2023-10-19T14:21:10.438773+0000","last_unstale":"2023-10-19T14:21:27.523341+0000","last_undegraded":"2023-10-19T14:21:27.523341+0000","last_fullsized":"2023-10-19T14:21:27.523341+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:42:16.397379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":16,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.510814+0000","last_change":"2023-10-19T14:21:09.409229+0000","last_active":"2023-10-19T14:21:27.510814+0000","last_peered":"2023-10-19T14:21:27.510814+0000","last_clean":"2023-10-19T14:21:27.510814+0000","last_became_active":"2023-10-19T14:21:09.408885+0000","last_became_peered":"2023-10-19T14:21:09.408885+0000","last_unstale":"2023-10-19T14:21:27.510814+0000","last_undegraded":"2023-10-19T14:21:27.510814+0000","last_fullsized":"2023-10-19T14:21:27.510814+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:05:37.351111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.3d","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.527622+0000","last_change":"2023-10-19T14:21:27.534679+0000","last_active":"2023-10-19T14:21:28.527622+0000","last_peered":"2023-10-19T14:21:28.527622+0000","last_clean":"2023-10-19T14:21:28.527622+0000","last_became_active":"2023-10-19T14:21:27.534418+0000","last_became_peered":"2023-10-19T14:21:27.534418+0000","last_unstale":"2023-10-19T14:21:28.527622+0000","last_undegraded":"2023-10-19T14:21:28.527622+0000","last_fullsized":"2023-10-19T14:21:28.527622+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:50:51.105426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.3f","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.525499+0000","last_change":"2023-10-19T14:21:10.437868+0000","last_active":"2023-10-19T14:21:27.525499+0000","last_peered":"2023-10-19T14:21:27.525499+0000","last_clean":"2023-10-19T14:21:27.525499+0000","last_became_active":"2023-10-19T14:21:10.437535+0000","last_became_peered":"2023-10-19T14:21:10.437535+0000","last_unstale":"2023-10-19T14:21:27.525499+0000","last_undegraded":"2023-10-19T14:21:27.525499+0000","last_fullsized":"2023-10-19T14:21:27.525499+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:44:47.905654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":16,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.523786+0000","last_change":"2023-10-19T14:21:09.407963+0000","last_active":"2023-10-19T14:21:27.523786+0000","last_peered":"2023-10-19T14:21:27.523786+0000","last_clean":"2023-10-19T14:21:27.523786+0000","last_became_active":"2023-10-19T14:21:09.407684+0000","last_became_peered":"2023-10-19T14:21:09.407684+0000","last_unstale":"2023-10-19T14:21:27.523786+0000","last_undegraded":"2023-10-19T14:21:27.523786+0000","last_fullsized":"2023-10-19T14:21:27.523786+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:45:20.541560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.23","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.526849+0000","last_change":"2023-10-19T14:21:27.538875+0000","last_active":"2023-10-19T14:21:28.526849+0000","last_peered":"2023-10-19T14:21:28.526849+0000","last_clean":"2023-10-19T14:21:28.526849+0000","last_became_active":"2023-10-19T14:21:27.537248+0000","last_became_peered":"2023-10-19T14:21:27.537248+0000","last_unstale":"2023-10-19T14:21:28.526849+0000","last_undegraded":"2023-10-19T14:21:28.526849+0000","last_fullsized":"2023-10-19T14:21:28.526849+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:03:10.552408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.21","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.512445+0000","last_change":"2023-10-19T14:21:10.434210+0000","last_active":"2023-10-19T14:21:27.512445+0000","last_peered":"2023-10-19T14:21:27.512445+0000","last_clean":"2023-10-19T14:21:27.512445+0000","last_became_active":"2023-10-19T14:21:10.433919+0000","last_became_peered":"2023-10-19T14:21:10.433919+0000","last_unstale":"2023-10-19T14:21:27.512445+0000","last_undegraded":"2023-10-19T14:21:27.512445+0000","last_fullsized":"2023-10-19T14:21:27.512445+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:56:57.148327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1e","version":"242'1","reported_seq":17,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.512784+0000","last_change":"2023-10-19T14:21:09.411855+0000","last_active":"2023-10-19T14:21:27.512784+0000","last_peered":"2023-10-19T14:21:27.512784+0000","last_clean":"2023-10-19T14:21:27.512784+0000","last_became_active":"2023-10-19T14:21:09.411510+0000","last_became_peered":"2023-10-19T14:21:09.411510+0000","last_unstale":"2023-10-19T14:21:27.512784+0000","last_undegraded":"2023-10-19T14:21:27.512784+0000","last_fullsized":"2023-10-19T14:21:27.512784+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:44:43.405619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"229'3208","reported_seq":4184,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.516605+0000","last_change":"2023-10-19T12:44:08.636613+0000","last_active":"2023-10-19T14:21:27.516605+0000","last_peered":"2023-10-19T14:21:27.516605+0000","last_clean":"2023-10-19T14:21:27.516605+0000","last_became_active":"2023-10-19T12:44:08.635697+0000","last_became_peered":"2023-10-19T12:44:08.635697+0000","last_unstale":"2023-10-19T14:21:27.516605+0000","last_undegraded":"2023-10-19T14:21:27.516605+0000","last_fullsized":"2023-10-19T14:21:27.516605+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_clean_scrub_stamp":"2023-10-19T12:44:07.399738+0000","objects_scrubbed":0,"log_size":3208,"log_dups_size":0,"ondisk_log_size":3208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:15:23.685485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2402,"num_read_kb":5036,"num_write":629,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":34,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":33,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":32,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":31,"num_pg":32,"stat_sum":{"num_bytes":3666,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2402,"num_read_kb":5036,"num_write":629,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3208,"ondisk_log_size":3208,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576014,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66760,"kb_used_data":3636,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305080,"statfs":{"total":96636764160,"available":96568401920,"internally_reserved":0,"allocated":3723264,"data_stored":2197191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":64616907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:21:10 2023","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.55100000000000005,"15min":0.57999999999999996},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.32200000000000001},"max":{"1min":0.754,"5min":0.80900000000000005,"15min":0.86199999999999999},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.503,"5min":0.57599999999999996,"15min":0.59899999999999998},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.312},"max":{"1min":0.66500000000000004,"5min":0.81899999999999995,"15min":0.86599999999999999},"last":0.56599999999999995}]},{"osd":1,"last update":"Thu Oct 19 14:21:10 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.58199999999999996,"15min":0.59499999999999997},"min":{"1min":0.45200000000000001,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.72199999999999998,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.56899999999999995,"15min":0.59299999999999997},"min":{"1min":0.39300000000000002,"5min":0.373,"15min":0.36699999999999999},"max":{"1min":0.78500000000000003,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.55700000000000005}]},{"osd":2,"last update":"Thu Oct 19 14:21:10 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.622,"15min":0.64200000000000002},"min":{"1min":0.47299999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.70499999999999996,"5min":0.84799999999999998,"15min":0.85499999999999998},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.63900000000000001,"15min":0.65700000000000003},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.622}]},{"osd":3,"last update":"Thu Oct 19 14:21:10 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.60799999999999998,"15min":0.63100000000000001},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.496},{"interface":"front","average":{"1min":0.628,"5min":0.67100000000000004,"15min":0.68300000000000005},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.83299999999999996,"5min":0.876,"15min":0.89600000000000002},"last":0.60999999999999999}]},{"osd":4,"last update":"Thu Oct 19 14:21:10 2023","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.50600000000000001,"15min":0.495},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.377},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.53000000000000003,"15min":0.51100000000000001},"min":{"1min":0.34999999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.41699999999999998}]},{"osd":5,"last update":"Thu Oct 19 14:21:10 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52300000000000002,"15min":0.51100000000000001},"min":{"1min":0.33700000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.57799999999999996,"15min":0.55400000000000005},"min":{"1min":0.40699999999999997,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.054,"5min":1.054,"15min":1.054},"last":0.51300000000000001}]},{"osd":6,"last update":"Thu Oct 19 14:21:10 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.58999999999999997,"15min":0.57599999999999996},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.35999999999999999},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.59299999999999997,"15min":0.57299999999999995},"min":{"1min":0.42999999999999999,"5min":0.35599999999999998,"15min":0.33500000000000002},"max":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.54400000000000004}]}]},{"osd":6,"up_from":13,"seq":55834576014,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56440,"kb_used_data":3052,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315400,"statfs":{"total":96636764160,"available":96578969600,"internally_reserved":0,"allocated":3125248,"data_stored":1606241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:21:02 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.59199999999999997,"15min":0.61099999999999999},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.78400000000000003,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.56299999999999994,"15min":0.60599999999999998},"min":{"1min":0.36699999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.71799999999999997,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.41499999999999998}]},{"osd":1,"last update":"Thu Oct 19 14:21:02 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.60099999999999998,"15min":0.61799999999999999},"min":{"1min":0.373,"5min":0.373,"15min":0.35799999999999998},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":1.165},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.60199999999999998,"15min":0.626},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.77400000000000002,"5min":0.83699999999999997,"15min":0.91300000000000003},"last":0.56899999999999995}]},{"osd":2,"last update":"Thu Oct 19 14:21:02 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.57599999999999996,"15min":0.60199999999999998},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.69099999999999995,"5min":0.80100000000000005,"15min":0.873},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.60599999999999998,"15min":0.624},"min":{"1min":0.40899999999999997,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.74199999999999999,"5min":0.83499999999999996,"15min":0.93100000000000005},"last":0.52500000000000002}]},{"osd":3,"last update":"Thu Oct 19 14:21:02 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.629,"15min":0.64900000000000002},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.66700000000000004,"5min":0.875,"15min":0.875},"last":0.621},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.63800000000000001,"15min":0.65500000000000003},"min":{"1min":0.45400000000000001,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.92200000000000004,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.60499999999999998}]},{"osd":4,"last update":"Thu Oct 19 14:21:02 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.48899999999999999,"15min":0.48399999999999999},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.51000000000000001,"15min":0.50800000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.30399999999999999},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.85699999999999998},"last":0.50900000000000001}]},{"osd":5,"last update":"Thu Oct 19 14:21:02 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.50700000000000001,"15min":0.502},"min":{"1min":0.34799999999999998,"5min":0.32700000000000001,"15min":0.32500000000000001},"max":{"1min":0.79900000000000004,"5min":1.115,"15min":1.115},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.56599999999999995,"15min":0.54500000000000004},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.58599999999999997}]},{"osd":7,"last update":"Thu Oct 19 14:21:02 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.56000000000000005,"15min":0.54700000000000004},"min":{"1min":0.36799999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.90100000000000002},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.56699999999999995,"15min":0.55400000000000005},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.98399999999999999,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.65500000000000003}]}]},{"osd":5,"up_from":13,"seq":55834576014,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66168,"kb_used_data":3044,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305672,"statfs":{"total":96636764160,"available":96569008128,"internally_reserved":0,"allocated":3117056,"data_stored":1606165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:20:46 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.58499999999999996,"15min":0.61699999999999999},"min":{"1min":0.33200000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.78700000000000003,"5min":0.79700000000000004,"15min":0.86399999999999999},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.60999999999999999,"15min":0.63800000000000001},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.77400000000000002,"5min":0.85899999999999999,"15min":0.85999999999999999},"last":0.627}]},{"osd":1,"last update":"Thu Oct 19 14:20:46 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.59499999999999997,"15min":0.628},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.749,"5min":0.80300000000000005,"15min":0.877},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.626,"15min":0.64000000000000001},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.81599999999999995,"5min":0.86699999999999999,"15min":0.88500000000000001},"last":0.68000000000000005}]},{"osd":2,"last update":"Thu Oct 19 14:20:46 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.59999999999999998,"15min":0.622},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.75900000000000001,"5min":0.83799999999999997,"15min":0.85099999999999998},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.63300000000000001,"15min":0.65500000000000003},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.81299999999999994,"5min":0.83299999999999996,"15min":0.90800000000000003},"last":0.51100000000000001}]},{"osd":3,"last update":"Thu Oct 19 14:20:46 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.61399999999999999,"15min":0.64100000000000001},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.77100000000000002,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.64000000000000001,"15min":0.66700000000000004},"min":{"1min":0.436,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.73899999999999999,"5min":0.876,"15min":0.90900000000000003},"last":0.59299999999999997}]},{"osd":4,"last update":"Thu Oct 19 14:20:46 2023","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.48499999999999999,"15min":0.48299999999999998},"min":{"1min":0.35499999999999998,"5min":0.34999999999999998,"15min":0.32500000000000001},"max":{"1min":0.56200000000000006,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.437,"5min":0.499,"15min":0.501},"min":{"1min":0.308,"5min":0.308,"15min":0.28599999999999998},"max":{"1min":0.57299999999999995,"5min":0.77500000000000002,"15min":0.82099999999999995},"last":0.64000000000000001}]},{"osd":6,"last update":"Thu Oct 19 14:20:46 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.51000000000000001,"15min":0.497},"min":{"1min":0.32200000000000001,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.72999999999999998,"5min":0.80800000000000005,"15min":0.88800000000000001},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.53300000000000003,"15min":0.53600000000000003},"min":{"1min":0.39900000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.70399999999999996,"5min":0.78000000000000003,"15min":0.88100000000000001},"last":0.71199999999999997}]},{"osd":7,"last update":"Thu Oct 19 14:20:46 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.50800000000000001,"15min":0.51500000000000001},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.625,"5min":0.749,"15min":0.82899999999999996},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.57399999999999995,"15min":0.56999999999999995},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.67300000000000004,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.66800000000000004}]}]},{"osd":4,"up_from":13,"seq":55834576014,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48120,"kb_used_data":3044,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323720,"statfs":{"total":96636764160,"available":96587489280,"internally_reserved":0,"allocated":3117056,"data_stored":1600262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:21:16 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.54700000000000004,"15min":0.57099999999999995},"min":{"1min":0.30299999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.879},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.56899999999999995,"15min":0.59299999999999997},"min":{"1min":0.35299999999999998,"5min":0.31,"15min":0.31},"max":{"1min":0.72699999999999998,"5min":0.82599999999999996,"15min":0.873},"last":0.498}]},{"osd":1,"last update":"Thu Oct 19 14:21:16 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.59699999999999998,"15min":0.63500000000000001},"min":{"1min":0.40300000000000002,"5min":0.372,"15min":0.32600000000000001},"max":{"1min":0.74099999999999999,"5min":0.80400000000000005,"15min":0.92700000000000005},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.60899999999999999,"15min":0.63900000000000001},"min":{"1min":0.40500000000000003,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.755,"5min":0.90000000000000002,"15min":1.0940000000000001},"last":0.60099999999999998}]},{"osd":2,"last update":"Thu Oct 19 14:21:16 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.626,"15min":0.64700000000000002},"min":{"1min":0.38900000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.80300000000000005,"5min":1.008,"15min":1.008},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.64200000000000002,"15min":0.66600000000000004},"min":{"1min":0.47199999999999998,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.81799999999999995,"5min":0.98799999999999999,"15min":1.008},"last":0.63}]},{"osd":3,"last update":"Thu Oct 19 14:21:16 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.59499999999999997,"15min":0.627},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.68600000000000005,"5min":0.83399999999999996,"15min":0.91900000000000004},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.65600000000000003,"15min":0.66600000000000004},"min":{"1min":0.48199999999999998,"5min":0.45400000000000001,"15min":0.437},"max":{"1min":0.85999999999999999,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.56299999999999994}]},{"osd":5,"last update":"Thu Oct 19 14:21:16 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.497,"15min":0.47799999999999998},"min":{"1min":0.33800000000000002,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.79700000000000004,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.50800000000000001,"15min":0.498},"min":{"1min":0.36499999999999999,"5min":0.33700000000000002,"15min":0.29199999999999998},"max":{"1min":0.81399999999999995,"5min":0.873,"15min":0.873},"last":0.40200000000000002}]},{"osd":6,"last update":"Thu Oct 19 14:21:16 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.50700000000000001,"15min":0.49399999999999999},"min":{"1min":0.38500000000000001,"5min":0.34799999999999998,"15min":0.28199999999999997},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.54900000000000004,"15min":0.54300000000000004},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.88,"5min":0.88,"15min":1.2969999999999999},"last":0.41899999999999998}]},{"osd":7,"last update":"Thu Oct 19 14:21:16 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55100000000000005,"15min":0.54500000000000004},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.32600000000000001},"max":{"1min":0.78200000000000003,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.61199999999999999,"15min":0.59899999999999998},"min":{"1min":0.371,"5min":0.32100000000000001,"15min":0.318},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":1.032},"last":0.46000000000000002}]}]},{"osd":3,"up_from":13,"seq":55834576013,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56460,"kb_used_data":3068,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315380,"statfs":{"total":96636764160,"available":96578949120,"internally_reserved":0,"allocated":3141632,"data_stored":1609691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54655432},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:20:49 2023","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.41099999999999998,"15min":0.41799999999999998},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.51800000000000002,"5min":0.61499999999999999,"15min":0.70699999999999996},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.40799999999999997,"15min":0.42799999999999999},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.752},"last":0.38800000000000001}]},{"osd":1,"last update":"Thu Oct 19 14:20:49 2023","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.42199999999999999,"15min":0.43099999999999999},"min":{"1min":0.25,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.54200000000000004,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.432,"5min":0.46400000000000002,"15min":0.47199999999999998},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.26400000000000001},"max":{"1min":0.67900000000000005,"5min":0.73099999999999998,"15min":0.81799999999999995},"last":0.48199999999999998}]},{"osd":2,"last update":"Thu Oct 19 14:20:49 2023","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.46899999999999997,"15min":0.48299999999999998},"min":{"1min":0.33100000000000002,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.625,"5min":0.64600000000000002,"15min":0.73099999999999998},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.49199999999999999,"15min":0.504},"min":{"1min":0.31900000000000001,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.83699999999999997},"last":0.41799999999999998}]},{"osd":4,"last update":"Thu Oct 19 14:20:49 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.69199999999999995,"15min":0.70299999999999996},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.871,"5min":1.018,"15min":1.157},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.71099999999999997,"15min":0.72899999999999998},"min":{"1min":0.47399999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.97699999999999998,"5min":0.996,"15min":1.173},"last":0.83599999999999997}]},{"osd":5,"last update":"Thu Oct 19 14:20:49 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.68300000000000005,"15min":0.68500000000000005},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.41999999999999998},"max":{"1min":0.83399999999999996,"5min":0.97299999999999998,"15min":1.0149999999999999},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.67500000000000004,"15min":0.68600000000000005},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.84999999999999998,"5min":0.99399999999999999,"15min":1.105},"last":0.85199999999999998}]},{"osd":6,"last update":"Thu Oct 19 14:20:49 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.65900000000000003,"15min":0.68700000000000006},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.70499999999999996,"5min":1.0009999999999999,"15min":1.0669999999999999},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.621,"5min":0.69599999999999995,"15min":0.70999999999999996},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.85599999999999998,"5min":0.91500000000000004,"15min":1.079},"last":0.69399999999999995}]},{"osd":7,"last update":"Thu Oct 19 14:20:49 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.68799999999999994,"15min":0.69899999999999995},"min":{"1min":0.436,"5min":0.434,"15min":0.434},"max":{"1min":0.78800000000000003,"5min":0.96899999999999997,"15min":1.085},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.70899999999999996,"15min":0.72899999999999998},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.79400000000000004,"5min":0.97199999999999998,"15min":1.1739999999999999},"last":0.875}]}]},{"osd":2,"up_from":13,"seq":55834576015,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48140,"kb_used_data":3068,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323700,"statfs":{"total":96636764160,"available":96587468800,"internally_reserved":0,"allocated":3141632,"data_stored":1608963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:20:36 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.38800000000000001,"15min":0.39700000000000002},"min":{"1min":0.254,"5min":0.218,"15min":0.218},"max":{"1min":0.55300000000000005,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.435,"15min":0.436},"min":{"1min":0.28199999999999997,"5min":0.249,"15min":0.223},"max":{"1min":0.54100000000000004,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.53100000000000003}]},{"osd":1,"last update":"Thu Oct 19 14:20:36 2023","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.39600000000000002,"15min":0.41199999999999998},"min":{"1min":0.27500000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.47799999999999998,"5min":0.61699999999999999,"15min":0.65400000000000003},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.46700000000000003,"15min":0.46500000000000002},"min":{"1min":0.28499999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.60499999999999998,"5min":1.742,"15min":1.742},"last":0.28699999999999998}]},{"osd":3,"last update":"Thu Oct 19 14:20:36 2023","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.47099999999999997,"15min":0.47999999999999998},"min":{"1min":0.32100000000000001,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.63700000000000001,"5min":0.69299999999999995,"15min":0.74399999999999999},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.48799999999999999,"15min":0.50600000000000001},"min":{"1min":0.34499999999999997,"5min":0.32000000000000001,"15min":0.315},"max":{"1min":0.56999999999999995,"5min":0.67600000000000005,"15min":0.83599999999999997},"last":0.67700000000000005}]},{"osd":4,"last update":"Thu Oct 19 14:20:36 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.65700000000000003,"15min":0.66900000000000004},"min":{"1min":0.47099999999999997,"5min":0.39600000000000002,"15min":0.39500000000000002},"max":{"1min":0.85599999999999998,"5min":0.93300000000000005,"15min":0.998},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.67000000000000004,"15min":0.67900000000000005},"min":{"1min":0.46500000000000002,"5min":0.38900000000000001,"15min":0.35199999999999998},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.43099999999999999}]},{"osd":5,"last update":"Thu Oct 19 14:20:36 2023","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.69699999999999995,"15min":0.69599999999999995},"min":{"1min":0.52700000000000002,"5min":0.45100000000000001,"15min":0.40600000000000003},"max":{"1min":0.80000000000000004,"5min":1.4330000000000001,"15min":1.4330000000000001},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.68100000000000005,"15min":0.68700000000000006},"min":{"1min":0.52700000000000002,"5min":0.45100000000000001,"15min":0.39300000000000002},"max":{"1min":0.93300000000000005,"5min":1.595,"15min":1.595},"last":0.61499999999999999}]},{"osd":6,"last update":"Thu Oct 19 14:20:36 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.69299999999999995,"15min":0.70799999999999996},"min":{"1min":0.48799999999999999,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.996,"5min":0.996,"15min":1.0189999999999999},"last":0.503},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.71899999999999997,"15min":0.72099999999999997},"min":{"1min":0.48899999999999999,"5min":0.433,"15min":0.40600000000000003},"max":{"1min":0.93899999999999995,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":0.83399999999999996}]},{"osd":7,"last update":"Thu Oct 19 14:20:36 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.70199999999999996,"15min":0.71999999999999997},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.41799999999999998},"max":{"1min":0.88700000000000001,"5min":1.0129999999999999,"15min":1.0289999999999999},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.73599999999999999,"15min":0.73099999999999998},"min":{"1min":0.51900000000000002,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.97299999999999998,"5min":1.53,"15min":1.53},"last":0.86699999999999999}]}]},{"osd":1,"up_from":13,"seq":55834576014,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48120,"kb_used_data":3048,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323720,"statfs":{"total":96636764160,"available":96587489280,"internally_reserved":0,"allocated":3121152,"data_stored":1606219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.39600000000000002,"15min":0.40400000000000003},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23100000000000001},"max":{"1min":0.57299999999999995,"5min":0.61599999999999999,"15min":0.67400000000000004},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.435,"15min":0.45400000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.54200000000000004,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.33100000000000002}]},{"osd":2,"last update":"Thu Oct 19 14:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.43099999999999999,"15min":0.442},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.56899999999999995,"5min":0.63100000000000001,"15min":0.77500000000000002},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.48899999999999999,"15min":0.48999999999999999},"min":{"1min":0.32400000000000001,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.67300000000000004,"5min":0.68700000000000006,"15min":0.82799999999999996},"last":0.44600000000000001}]},{"osd":3,"last update":"Thu Oct 19 14:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.499,"15min":0.505},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.29499999999999998},"max":{"1min":0.64300000000000002,"5min":0.76000000000000001,"15min":0.78900000000000003},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.436,"15min":0.44600000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.26100000000000001},"max":{"1min":0.60799999999999998,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.377}]},{"osd":4,"last update":"Thu Oct 19 14:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.66400000000000003,"15min":0.67100000000000004},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.92800000000000005,"5min":0.98299999999999998,"15min":1.042},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.67100000000000004,"15min":0.66900000000000004},"min":{"1min":0.44500000000000001,"5min":0.41299999999999998,"15min":0.32300000000000001},"max":{"1min":0.89400000000000002,"5min":0.93500000000000005,"15min":1.1659999999999999},"last":0.65300000000000002}]},{"osd":5,"last update":"Thu Oct 19 14:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.68799999999999994,"15min":0.68600000000000005},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.82099999999999995,"5min":1.002,"15min":1.3500000000000001},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.70599999999999996,"15min":0.70799999999999996},"min":{"1min":0.46700000000000003,"5min":0.44,"15min":0.40500000000000003},"max":{"1min":1.0009999999999999,"5min":1.0149999999999999,"15min":1.121},"last":0.47799999999999998}]},{"osd":6,"last update":"Thu Oct 19 14:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.67800000000000005,"15min":0.68200000000000005},"min":{"1min":0.51000000000000001,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.95499999999999996,"5min":1.0129999999999999,"15min":1.2310000000000001},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.71899999999999997,"15min":0.71599999999999997},"min":{"1min":0.48199999999999998,"5min":0.434,"15min":0.434},"max":{"1min":0.93100000000000005,"5min":1.0109999999999999,"15min":1.274},"last":0.69699999999999995}]},{"osd":7,"last update":"Thu Oct 19 14:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.71899999999999997,"15min":0.72399999999999998},"min":{"1min":0.498,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.96799999999999997,"5min":0.99099999999999999,"15min":1.075},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.70099999999999996,"15min":0.70199999999999996},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.44400000000000001},"max":{"1min":0.91200000000000003,"5min":1.0169999999999999,"15min":1.117},"last":0.60999999999999999}]}]},{"osd":0,"up_from":13,"seq":55834576015,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61180,"kb_used_data":3632,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310660,"statfs":{"total":96636764160,"available":96574115840,"internally_reserved":0,"allocated":3719168,"data_stored":2198782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Oct 19 14:21:13 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.40100000000000002,"15min":0.41099999999999998},"min":{"1min":0.26300000000000001,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.47799999999999998,"5min":0.61499999999999999,"15min":0.624},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.41699999999999998,"15min":0.45500000000000002},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.49199999999999999,"5min":0.70899999999999996,"15min":0.81100000000000005},"last":0.71199999999999997}]},{"osd":2,"last update":"Thu Oct 19 14:21:13 2023","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.34100000000000003,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.54400000000000004,"5min":0.754,"15min":0.754},"last":0.434},{"interface":"front","average":{"1min":0.377,"5min":0.42399999999999999,"15min":0.44400000000000001},"min":{"1min":0.28100000000000003,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.46899999999999997,"5min":0.60199999999999998,"15min":0.69099999999999995},"last":0.38900000000000001}]},{"osd":3,"last update":"Thu Oct 19 14:21:13 2023","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.436},"min":{"1min":0.27200000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.50800000000000001,"5min":0.58799999999999997,"15min":0.66800000000000004},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.48199999999999998,"15min":0.50700000000000001},"min":{"1min":0.29299999999999998,"5min":0.254,"15min":0.254},"max":{"1min":0.72399999999999998,"5min":0.78100000000000003,"15min":0.84599999999999997},"last":0.55600000000000005}]},{"osd":4,"last update":"Thu Oct 19 14:21:13 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.63,"15min":0.67000000000000004},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.80300000000000005,"5min":0.93100000000000005,"15min":1.1890000000000001},"last":0.497},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.628,"15min":0.67400000000000004},"min":{"1min":0.45900000000000002,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.75600000000000001,"5min":0.93400000000000005,"15min":1.2150000000000001},"last":0.51600000000000001}]},{"osd":5,"last update":"Thu Oct 19 14:21:13 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.67200000000000004,"15min":0.71399999999999997},"min":{"1min":0.48599999999999999,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.83099999999999996,"5min":0.98499999999999999,"15min":1.1579999999999999},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.67500000000000004,"15min":0.71399999999999997},"min":{"1min":0.499,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.80000000000000004,"5min":0.98299999999999998,"15min":1.2310000000000001},"last":0.64200000000000002}]},{"osd":6,"last update":"Thu Oct 19 14:21:13 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.66300000000000003,"15min":0.69899999999999995},"min":{"1min":0.46100000000000002,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.85499999999999998,"5min":0.97799999999999998,"15min":1.1739999999999999},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.65300000000000002,"15min":0.69599999999999995},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.70299999999999996,"5min":0.97799999999999998,"15min":1.0189999999999999},"last":0.61599999999999999}]},{"osd":7,"last update":"Thu Oct 19 14:21:13 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.66600000000000004,"15min":0.70999999999999996},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.79900000000000004,"5min":0.93400000000000005,"15min":1.1180000000000001},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.66100000000000003,"15min":0.69599999999999995},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.84499999999999997,"5min":0.93799999999999994,"15min":1.0349999999999999},"last":0.53600000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-19T14:21:30.537 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-19T14:21:30.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.672+0000 7f2924458640 1 -- 172.21.15.57:0/3210285051 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f291c141690 msgr2=0x7f291c0703e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:30.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.672+0000 7f2924458640 1 --2- 172.21.15.57:0/3210285051 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f291c141690 0x7f291c0703e0 secure :-1 s=READY pgs=2701 cs=0 l=1 rev1=1 crypto rx=0x7f2910009830 tx=0x7f291002bcd0 comp rx=0 tx=0).stop 2023-10-19T14:21:30.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.672+0000 7f2924458640 1 -- 172.21.15.57:0/3210285051 shutdown_connections 2023-10-19T14:21:30.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.672+0000 7f2924458640 1 --2- 172.21.15.57:0/3210285051 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f291c070920 0x7f291c0affe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:30.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.672+0000 7f2924458640 1 --2- 172.21.15.57:0/3210285051 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f291c141690 0x7f291c0703e0 unknown :-1 s=CLOSED pgs=2701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:30.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.672+0000 7f2924458640 1 -- 172.21.15.57:0/3210285051 >> 172.21.15.57:0/3210285051 conn(0x7f291c069930 msgr2=0x7f291c069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:30.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.672+0000 7f2924458640 1 -- 172.21.15.57:0/3210285051 shutdown_connections 2023-10-19T14:21:30.677 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.672+0000 7f2924458640 1 -- 172.21.15.57:0/3210285051 wait complete. 2023-10-19T14:21:30.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.673+0000 7f2924458640 1 Processor -- start 2023-10-19T14:21:30.679 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.673+0000 7f2924458640 1 -- start start 2023-10-19T14:21:30.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.674+0000 7f2924458640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f291c070920 0x7f291c0ca870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:30.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.674+0000 7f2924458640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f291c13e520 0x7f291c0cadb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:30.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.674+0000 7f2924458640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f291c141690 0x7f291c1e5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:30.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.674+0000 7f2924458640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f291c0c8750 con 0x7f291c070920 2023-10-19T14:21:30.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.674+0000 7f2924458640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f291c1e14a0 con 0x7f291c141690 2023-10-19T14:21:30.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.674+0000 7f2924458640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f291c1e5fb0 con 0x7f291c13e520 2023-10-19T14:21:30.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.674+0000 7f29221f4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f291c070920 0x7f291c0ca870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:30.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.674+0000 7f29221f4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f291c070920 0x7f291c0ca870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37228/0 (socket says 172.21.15.57:37228) 2023-10-19T14:21:30.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.674+0000 7f29221f4640 1 -- 172.21.15.57:0/2119770659 learned_addr learned my addr 172.21.15.57:0/2119770659 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:30.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.674+0000 7f29219f3640 1 --2- 172.21.15.57:0/2119770659 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f291c13e520 0x7f291c0cadb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:30.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.674+0000 7f29229f5640 1 --2- 172.21.15.57:0/2119770659 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f291c141690 0x7f291c1e5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:30.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.674+0000 7f29221f4640 1 -- 172.21.15.57:0/2119770659 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f291c13e520 msgr2=0x7f291c0cadb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:30.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.674+0000 7f29221f4640 1 --2- 172.21.15.57:0/2119770659 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f291c13e520 0x7f291c0cadb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:30.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.674+0000 7f29221f4640 1 -- 172.21.15.57:0/2119770659 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f291c141690 msgr2=0x7f291c1e5360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:30.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.674+0000 7f29221f4640 1 --2- 172.21.15.57:0/2119770659 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f291c141690 0x7f291c1e5360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:30.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.674+0000 7f29221f4640 1 -- 172.21.15.57:0/2119770659 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2910009490 con 0x7f291c070920 2023-10-19T14:21:30.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.674+0000 7f29229f5640 1 --2- 172.21.15.57:0/2119770659 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f291c141690 0x7f291c1e5360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:21:30.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.674+0000 7f29221f4640 1 --2- 172.21.15.57:0/2119770659 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f291c070920 0x7f291c0ca870 secure :-1 s=READY pgs=2702 cs=0 l=1 rev1=1 crypto rx=0x7f290c00cbc0 tx=0x7f290c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:30.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.675+0000 7f291affd640 1 -- 172.21.15.57:0/2119770659 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f290c00ace0 con 0x7f291c070920 2023-10-19T14:21:30.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.675+0000 7f291affd640 1 -- 172.21.15.57:0/2119770659 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f290c003970 con 0x7f291c070920 2023-10-19T14:21:30.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.675+0000 7f291affd640 1 -- 172.21.15.57:0/2119770659 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f290c003d60 con 0x7f291c070920 2023-10-19T14:21:30.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.675+0000 7f2924458640 1 -- 172.21.15.57:0/2119770659 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f291c1df390 con 0x7f291c070920 2023-10-19T14:21:30.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.675+0000 7f2924458640 1 -- 172.21.15.57:0/2119770659 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f291c1df930 con 0x7f291c070920 2023-10-19T14:21:30.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.676+0000 7f2924458640 1 -- 172.21.15.57:0/2119770659 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f291c13e0d0 con 0x7f291c070920 2023-10-19T14:21:30.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.676+0000 7f291affd640 1 -- 172.21.15.57:0/2119770659 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f290c004030 con 0x7f291c070920 2023-10-19T14:21:30.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.677+0000 7f291affd640 1 --2- 172.21.15.57:0/2119770659 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f290806b080 0x7f290806d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:30.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.677+0000 7f291affd640 1 -- 172.21.15.57:0/2119770659 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(247..247 src has 1..247) v4 ==== 6921+0+0 (secure 0 0 0) 0x7f290c08a9e0 con 0x7f291c070920 2023-10-19T14:21:30.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.677+0000 7f29219f3640 1 --2- 172.21.15.57:0/2119770659 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f290806b080 0x7f290806d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:30.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.677+0000 7f29219f3640 1 --2- 172.21.15.57:0/2119770659 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f290806b080 0x7f290806d540 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2910036000 tx=0x7f2910004730 comp rx=0 tx=0).ready entity=mgr.9530 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:30.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.681+0000 7f291affd640 1 -- 172.21.15.57:0/2119770659 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f290c055270 con 0x7f291c070920 2023-10-19T14:21:30.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.854+0000 7f2924458640 1 -- 172.21.15.57:0/2119770659 --> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f291c0690e0 con 0x7f290806b080 2023-10-19T14:21:30.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.867+0000 7f291affd640 1 -- 172.21.15.57:0/2119770659 <== mgr.9530 v2:172.21.15.179:6832/3547895240 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+529049 (secure 0 0 0) 0x7f291c0690e0 con 0x7f290806b080 2023-10-19T14:21:30.875 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:21:30.879 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2023-10-19T14:21:30.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.872+0000 7f2924458640 1 -- 172.21.15.57:0/2119770659 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f290806b080 msgr2=0x7f290806d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:30.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.872+0000 7f2924458640 1 --2- 172.21.15.57:0/2119770659 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f290806b080 0x7f290806d540 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2910036000 tx=0x7f2910004730 comp rx=0 tx=0).stop 2023-10-19T14:21:30.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.872+0000 7f2924458640 1 -- 172.21.15.57:0/2119770659 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f291c070920 msgr2=0x7f291c0ca870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:30.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.872+0000 7f2924458640 1 --2- 172.21.15.57:0/2119770659 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f291c070920 0x7f291c0ca870 secure :-1 s=READY pgs=2702 cs=0 l=1 rev1=1 crypto rx=0x7f290c00cbc0 tx=0x7f290c00a430 comp rx=0 tx=0).stop 2023-10-19T14:21:30.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.872+0000 7f2924458640 1 -- 172.21.15.57:0/2119770659 shutdown_connections 2023-10-19T14:21:30.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.872+0000 7f2924458640 1 --2- 172.21.15.57:0/2119770659 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f290806b080 0x7f290806d540 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:30.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.872+0000 7f2924458640 1 --2- 172.21.15.57:0/2119770659 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f291c141690 0x7f291c1e5360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:30.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.872+0000 7f2924458640 1 --2- 172.21.15.57:0/2119770659 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f291c13e520 0x7f291c0cadb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:30.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.872+0000 7f2924458640 1 --2- 172.21.15.57:0/2119770659 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f291c070920 0x7f291c0ca870 unknown :-1 s=CLOSED pgs=2702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:30.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.872+0000 7f2924458640 1 -- 172.21.15.57:0/2119770659 >> 172.21.15.57:0/2119770659 conn(0x7f291c069930 msgr2=0x7f291c1e2ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:30.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.872+0000 7f2924458640 1 -- 172.21.15.57:0/2119770659 shutdown_connections 2023-10-19T14:21:30.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:30.872+0000 7f2924458640 1 -- 172.21.15.57:0/2119770659 wait complete. 2023-10-19T14:21:30.885 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":42,"stamp":"2023-10-19T14:21:30.176706+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596444,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":2402,"num_read_kb":5036,"num_write":721,"num_write_kb":2447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3288,"ondisk_log_size":3288,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":347,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":451388,"kb_used_data":25592,"kb_used_omap":12,"kb_used_meta":425651,"kb_avail":754523332,"statfs":{"total":773094113280,"available":772631891968,"internally_reserved":0,"allocated":26206208,"data_stored":14033514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":435867193},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3225,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":46,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.828890"},"pg_stats":[{"pgid":"34.1d","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.590689+0000","last_change":"2023-10-19T14:21:27.533913+0000","last_active":"2023-10-19T14:21:28.590689+0000","last_peered":"2023-10-19T14:21:28.590689+0000","last_clean":"2023-10-19T14:21:28.590689+0000","last_became_active":"2023-10-19T14:21:27.533603+0000","last_became_peered":"2023-10-19T14:21:27.533603+0000","last_unstale":"2023-10-19T14:21:28.590689+0000","last_undegraded":"2023-10-19T14:21:28.590689+0000","last_fullsized":"2023-10-19T14:21:28.590689+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:45:59.620638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1e","version":"246'1","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.592522+0000","last_change":"2023-10-19T14:21:26.366411+0000","last_active":"2023-10-19T14:21:28.592522+0000","last_peered":"2023-10-19T14:21:28.592522+0000","last_clean":"2023-10-19T14:21:28.592522+0000","last_became_active":"2023-10-19T14:21:26.366143+0000","last_became_peered":"2023-10-19T14:21:26.366143+0000","last_unstale":"2023-10-19T14:21:28.592522+0000","last_undegraded":"2023-10-19T14:21:28.592522+0000","last_fullsized":"2023-10-19T14:21:28.592522+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_clean_scrub_stamp":"2023-10-19T14:21:25.343524+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:11:36.819564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1f","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.515056+0000","last_change":"2023-10-19T14:21:10.436485+0000","last_active":"2023-10-19T14:21:27.515056+0000","last_peered":"2023-10-19T14:21:27.515056+0000","last_clean":"2023-10-19T14:21:27.515056+0000","last_became_active":"2023-10-19T14:21:10.436174+0000","last_became_peered":"2023-10-19T14:21:10.436174+0000","last_unstale":"2023-10-19T14:21:27.515056+0000","last_undegraded":"2023-10-19T14:21:27.515056+0000","last_fullsized":"2023-10-19T14:21:27.515056+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:01:21.155069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.1c","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.591906+0000","last_change":"2023-10-19T14:21:27.525352+0000","last_active":"2023-10-19T14:21:28.591906+0000","last_peered":"2023-10-19T14:21:28.591906+0000","last_clean":"2023-10-19T14:21:28.591906+0000","last_became_active":"2023-10-19T14:21:27.525095+0000","last_became_peered":"2023-10-19T14:21:27.525095+0000","last_unstale":"2023-10-19T14:21:28.591906+0000","last_undegraded":"2023-10-19T14:21:28.591906+0000","last_fullsized":"2023-10-19T14:21:28.591906+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:24:33.138924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.1f","version":"246'4","reported_seq":16,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.591196+0000","last_change":"2023-10-19T14:21:26.372524+0000","last_active":"2023-10-19T14:21:28.591196+0000","last_peered":"2023-10-19T14:21:28.591196+0000","last_clean":"2023-10-19T14:21:28.591196+0000","last_became_active":"2023-10-19T14:21:26.372245+0000","last_became_peered":"2023-10-19T14:21:26.372245+0000","last_unstale":"2023-10-19T14:21:28.591196+0000","last_undegraded":"2023-10-19T14:21:28.591196+0000","last_fullsized":"2023-10-19T14:21:28.591196+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_clean_scrub_stamp":"2023-10-19T14:21:25.343524+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:02:19.626255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1e","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.523249+0000","last_change":"2023-10-19T14:21:10.439371+0000","last_active":"2023-10-19T14:21:27.523249+0000","last_peered":"2023-10-19T14:21:27.523249+0000","last_clean":"2023-10-19T14:21:27.523249+0000","last_became_active":"2023-10-19T14:21:10.439072+0000","last_became_peered":"2023-10-19T14:21:10.439072+0000","last_unstale":"2023-10-19T14:21:27.523249+0000","last_undegraded":"2023-10-19T14:21:27.523249+0000","last_fullsized":"2023-10-19T14:21:27.523249+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:45:21.552160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.1f","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.525255+0000","last_change":"2023-10-19T14:21:27.533201+0000","last_active":"2023-10-19T14:21:28.525255+0000","last_peered":"2023-10-19T14:21:28.525255+0000","last_clean":"2023-10-19T14:21:28.525255+0000","last_became_active":"2023-10-19T14:21:27.532933+0000","last_became_peered":"2023-10-19T14:21:27.532933+0000","last_unstale":"2023-10-19T14:21:28.525255+0000","last_undegraded":"2023-10-19T14:21:28.525255+0000","last_fullsized":"2023-10-19T14:21:28.525255+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:03:43.253629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1c","version":"246'2","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.591688+0000","last_change":"2023-10-19T14:21:26.369943+0000","last_active":"2023-10-19T14:21:28.591688+0000","last_peered":"2023-10-19T14:21:28.591688+0000","last_clean":"2023-10-19T14:21:28.591688+0000","last_became_active":"2023-10-19T14:21:26.369675+0000","last_became_peered":"2023-10-19T14:21:26.369675+0000","last_unstale":"2023-10-19T14:21:28.591688+0000","last_undegraded":"2023-10-19T14:21:28.591688+0000","last_fullsized":"2023-10-19T14:21:28.591688+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_clean_scrub_stamp":"2023-10-19T14:21:25.343524+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:33:06.312590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1d","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.511874+0000","last_change":"2023-10-19T14:21:10.422867+0000","last_active":"2023-10-19T14:21:27.511874+0000","last_peered":"2023-10-19T14:21:27.511874+0000","last_clean":"2023-10-19T14:21:27.511874+0000","last_became_active":"2023-10-19T14:21:10.422558+0000","last_became_peered":"2023-10-19T14:21:10.422558+0000","last_unstale":"2023-10-19T14:21:27.511874+0000","last_undegraded":"2023-10-19T14:21:27.511874+0000","last_fullsized":"2023-10-19T14:21:27.511874+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:46:57.540004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.1e","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.591126+0000","last_change":"2023-10-19T14:21:27.540496+0000","last_active":"2023-10-19T14:21:28.591126+0000","last_peered":"2023-10-19T14:21:28.591126+0000","last_clean":"2023-10-19T14:21:28.591126+0000","last_became_active":"2023-10-19T14:21:27.540219+0000","last_became_peered":"2023-10-19T14:21:27.540219+0000","last_unstale":"2023-10-19T14:21:28.591126+0000","last_undegraded":"2023-10-19T14:21:28.591126+0000","last_fullsized":"2023-10-19T14:21:28.591126+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:55:48.870859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1d","version":"246'2","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.592308+0000","last_change":"2023-10-19T14:21:26.362677+0000","last_active":"2023-10-19T14:21:28.592308+0000","last_peered":"2023-10-19T14:21:28.592308+0000","last_clean":"2023-10-19T14:21:28.592308+0000","last_became_active":"2023-10-19T14:21:26.362337+0000","last_became_peered":"2023-10-19T14:21:26.362337+0000","last_unstale":"2023-10-19T14:21:28.592308+0000","last_undegraded":"2023-10-19T14:21:28.592308+0000","last_fullsized":"2023-10-19T14:21:28.592308+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_clean_scrub_stamp":"2023-10-19T14:21:25.343524+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:47:27.368054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.1c","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.511719+0000","last_change":"2023-10-19T14:21:10.426687+0000","last_active":"2023-10-19T14:21:27.511719+0000","last_peered":"2023-10-19T14:21:27.511719+0000","last_clean":"2023-10-19T14:21:27.511719+0000","last_became_active":"2023-10-19T14:21:10.426422+0000","last_became_peered":"2023-10-19T14:21:10.426422+0000","last_unstale":"2023-10-19T14:21:27.511719+0000","last_undegraded":"2023-10-19T14:21:27.511719+0000","last_fullsized":"2023-10-19T14:21:27.511719+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:13:29.364169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.19","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.525000+0000","last_change":"2023-10-19T14:21:27.537576+0000","last_active":"2023-10-19T14:21:28.525000+0000","last_peered":"2023-10-19T14:21:28.525000+0000","last_clean":"2023-10-19T14:21:28.525000+0000","last_became_active":"2023-10-19T14:21:27.537268+0000","last_became_peered":"2023-10-19T14:21:27.537268+0000","last_unstale":"2023-10-19T14:21:28.525000+0000","last_undegraded":"2023-10-19T14:21:28.525000+0000","last_fullsized":"2023-10-19T14:21:28.525000+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T02:10:36.623775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1a","version":"246'1","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.524632+0000","last_change":"2023-10-19T14:21:26.376713+0000","last_active":"2023-10-19T14:21:28.524632+0000","last_peered":"2023-10-19T14:21:28.524632+0000","last_clean":"2023-10-19T14:21:28.524632+0000","last_became_active":"2023-10-19T14:21:26.376424+0000","last_became_peered":"2023-10-19T14:21:26.376424+0000","last_unstale":"2023-10-19T14:21:28.524632+0000","last_undegraded":"2023-10-19T14:21:28.524632+0000","last_fullsized":"2023-10-19T14:21:28.524632+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_clean_scrub_stamp":"2023-10-19T14:21:25.343524+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:42:18.892113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.1b","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.519477+0000","last_change":"2023-10-19T14:21:10.427647+0000","last_active":"2023-10-19T14:21:27.519477+0000","last_peered":"2023-10-19T14:21:27.519477+0000","last_clean":"2023-10-19T14:21:27.519477+0000","last_became_active":"2023-10-19T14:21:10.427412+0000","last_became_peered":"2023-10-19T14:21:10.427412+0000","last_unstale":"2023-10-19T14:21:27.519477+0000","last_undegraded":"2023-10-19T14:21:27.519477+0000","last_fullsized":"2023-10-19T14:21:27.519477+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T02:17:31.253940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.18","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.524262+0000","last_change":"2023-10-19T14:21:27.528624+0000","last_active":"2023-10-19T14:21:28.524262+0000","last_peered":"2023-10-19T14:21:28.524262+0000","last_clean":"2023-10-19T14:21:28.524262+0000","last_became_active":"2023-10-19T14:21:27.528220+0000","last_became_peered":"2023-10-19T14:21:27.528220+0000","last_unstale":"2023-10-19T14:21:28.524262+0000","last_undegraded":"2023-10-19T14:21:28.524262+0000","last_fullsized":"2023-10-19T14:21:28.524262+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:03:14.837836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1b","version":"246'1","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.590799+0000","last_change":"2023-10-19T14:21:26.364898+0000","last_active":"2023-10-19T14:21:28.590799+0000","last_peered":"2023-10-19T14:21:28.590799+0000","last_clean":"2023-10-19T14:21:28.590799+0000","last_became_active":"2023-10-19T14:21:26.364604+0000","last_became_peered":"2023-10-19T14:21:26.364604+0000","last_unstale":"2023-10-19T14:21:28.590799+0000","last_undegraded":"2023-10-19T14:21:28.590799+0000","last_fullsized":"2023-10-19T14:21:28.590799+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_clean_scrub_stamp":"2023-10-19T14:21:25.343524+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:04:46.470039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1a","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.515605+0000","last_change":"2023-10-19T14:21:10.435192+0000","last_active":"2023-10-19T14:21:27.515605+0000","last_peered":"2023-10-19T14:21:27.515605+0000","last_clean":"2023-10-19T14:21:27.515605+0000","last_became_active":"2023-10-19T14:21:10.434868+0000","last_became_peered":"2023-10-19T14:21:10.434868+0000","last_unstale":"2023-10-19T14:21:27.515605+0000","last_undegraded":"2023-10-19T14:21:27.515605+0000","last_fullsized":"2023-10-19T14:21:27.515605+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:40:01.514130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.1b","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.526932+0000","last_change":"2023-10-19T14:21:27.538695+0000","last_active":"2023-10-19T14:21:28.526932+0000","last_peered":"2023-10-19T14:21:28.526932+0000","last_clean":"2023-10-19T14:21:28.526932+0000","last_became_active":"2023-10-19T14:21:27.537546+0000","last_became_peered":"2023-10-19T14:21:27.537546+0000","last_unstale":"2023-10-19T14:21:28.526932+0000","last_undegraded":"2023-10-19T14:21:28.526932+0000","last_fullsized":"2023-10-19T14:21:28.526932+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:44:00.953524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.18","version":"0'0","reported_seq":12,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.589875+0000","last_change":"2023-10-19T14:21:26.365537+0000","last_active":"2023-10-19T14:21:28.589875+0000","last_peered":"2023-10-19T14:21:28.589875+0000","last_clean":"2023-10-19T14:21:28.589875+0000","last_became_active":"2023-10-19T14:21:26.365272+0000","last_became_peered":"2023-10-19T14:21:26.365272+0000","last_unstale":"2023-10-19T14:21:28.589875+0000","last_undegraded":"2023-10-19T14:21:28.589875+0000","last_fullsized":"2023-10-19T14:21:28.589875+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_clean_scrub_stamp":"2023-10-19T14:21:25.343524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:32:28.482159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.19","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.512713+0000","last_change":"2023-10-19T14:21:10.431422+0000","last_active":"2023-10-19T14:21:27.512713+0000","last_peered":"2023-10-19T14:21:27.512713+0000","last_clean":"2023-10-19T14:21:27.512713+0000","last_became_active":"2023-10-19T14:21:10.431131+0000","last_became_peered":"2023-10-19T14:21:10.431131+0000","last_unstale":"2023-10-19T14:21:27.512713+0000","last_undegraded":"2023-10-19T14:21:27.512713+0000","last_fullsized":"2023-10-19T14:21:27.512713+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:14:16.111979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.1a","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.527155+0000","last_change":"2023-10-19T14:21:27.538410+0000","last_active":"2023-10-19T14:21:28.527155+0000","last_peered":"2023-10-19T14:21:28.527155+0000","last_clean":"2023-10-19T14:21:28.527155+0000","last_became_active":"2023-10-19T14:21:27.538183+0000","last_became_peered":"2023-10-19T14:21:27.538183+0000","last_unstale":"2023-10-19T14:21:28.527155+0000","last_undegraded":"2023-10-19T14:21:28.527155+0000","last_fullsized":"2023-10-19T14:21:28.527155+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:17:12.731105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.19","version":"0'0","reported_seq":12,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.591540+0000","last_change":"2023-10-19T14:21:26.365920+0000","last_active":"2023-10-19T14:21:28.591540+0000","last_peered":"2023-10-19T14:21:28.591540+0000","last_clean":"2023-10-19T14:21:28.591540+0000","last_became_active":"2023-10-19T14:21:26.365644+0000","last_became_peered":"2023-10-19T14:21:26.365644+0000","last_unstale":"2023-10-19T14:21:28.591540+0000","last_undegraded":"2023-10-19T14:21:28.591540+0000","last_fullsized":"2023-10-19T14:21:28.591540+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_clean_scrub_stamp":"2023-10-19T14:21:25.343524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:37:35.585591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.18","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.518114+0000","last_change":"2023-10-19T14:21:10.435886+0000","last_active":"2023-10-19T14:21:27.518114+0000","last_peered":"2023-10-19T14:21:27.518114+0000","last_clean":"2023-10-19T14:21:27.518114+0000","last_became_active":"2023-10-19T14:21:10.435541+0000","last_became_peered":"2023-10-19T14:21:10.435541+0000","last_unstale":"2023-10-19T14:21:27.518114+0000","last_undegraded":"2023-10-19T14:21:27.518114+0000","last_fullsized":"2023-10-19T14:21:27.518114+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:02:44.118651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.15","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.590812+0000","last_change":"2023-10-19T14:21:27.530614+0000","last_active":"2023-10-19T14:21:28.590812+0000","last_peered":"2023-10-19T14:21:28.590812+0000","last_clean":"2023-10-19T14:21:28.590812+0000","last_became_active":"2023-10-19T14:21:27.530322+0000","last_became_peered":"2023-10-19T14:21:27.530322+0000","last_unstale":"2023-10-19T14:21:28.590812+0000","last_undegraded":"2023-10-19T14:21:28.590812+0000","last_fullsized":"2023-10-19T14:21:28.590812+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:23:55.515899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.16","version":"0'0","reported_seq":12,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.592624+0000","last_change":"2023-10-19T14:21:26.366314+0000","last_active":"2023-10-19T14:21:28.592624+0000","last_peered":"2023-10-19T14:21:28.592624+0000","last_clean":"2023-10-19T14:21:28.592624+0000","last_became_active":"2023-10-19T14:21:26.366024+0000","last_became_peered":"2023-10-19T14:21:26.366024+0000","last_unstale":"2023-10-19T14:21:28.592624+0000","last_undegraded":"2023-10-19T14:21:28.592624+0000","last_fullsized":"2023-10-19T14:21:28.592624+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_clean_scrub_stamp":"2023-10-19T14:21:25.343524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:28:12.261950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.17","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.515141+0000","last_change":"2023-10-19T14:21:10.436436+0000","last_active":"2023-10-19T14:21:27.515141+0000","last_peered":"2023-10-19T14:21:27.515141+0000","last_clean":"2023-10-19T14:21:27.515141+0000","last_became_active":"2023-10-19T14:21:10.435013+0000","last_became_peered":"2023-10-19T14:21:10.435013+0000","last_unstale":"2023-10-19T14:21:27.515141+0000","last_undegraded":"2023-10-19T14:21:27.515141+0000","last_fullsized":"2023-10-19T14:21:27.515141+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:16:49.709525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.14","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.527254+0000","last_change":"2023-10-19T14:21:27.526052+0000","last_active":"2023-10-19T14:21:28.527254+0000","last_peered":"2023-10-19T14:21:28.527254+0000","last_clean":"2023-10-19T14:21:28.527254+0000","last_became_active":"2023-10-19T14:21:27.524077+0000","last_became_peered":"2023-10-19T14:21:27.524077+0000","last_unstale":"2023-10-19T14:21:28.527254+0000","last_undegraded":"2023-10-19T14:21:28.527254+0000","last_fullsized":"2023-10-19T14:21:28.527254+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:38:13.814928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.17","version":"0'0","reported_seq":12,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.527081+0000","last_change":"2023-10-19T14:21:26.376096+0000","last_active":"2023-10-19T14:21:28.527081+0000","last_peered":"2023-10-19T14:21:28.527081+0000","last_clean":"2023-10-19T14:21:28.527081+0000","last_became_active":"2023-10-19T14:21:26.375073+0000","last_became_peered":"2023-10-19T14:21:26.375073+0000","last_unstale":"2023-10-19T14:21:28.527081+0000","last_undegraded":"2023-10-19T14:21:28.527081+0000","last_fullsized":"2023-10-19T14:21:28.527081+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_clean_scrub_stamp":"2023-10-19T14:21:25.343524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:00:02.644609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.16","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.512103+0000","last_change":"2023-10-19T14:21:10.430483+0000","last_active":"2023-10-19T14:21:27.512103+0000","last_peered":"2023-10-19T14:21:27.512103+0000","last_clean":"2023-10-19T14:21:27.512103+0000","last_became_active":"2023-10-19T14:21:10.430168+0000","last_became_peered":"2023-10-19T14:21:10.430168+0000","last_unstale":"2023-10-19T14:21:27.512103+0000","last_undegraded":"2023-10-19T14:21:27.512103+0000","last_fullsized":"2023-10-19T14:21:27.512103+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:40:00.950097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.17","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.593164+0000","last_change":"2023-10-19T14:21:27.534781+0000","last_active":"2023-10-19T14:21:28.593164+0000","last_peered":"2023-10-19T14:21:28.593164+0000","last_clean":"2023-10-19T14:21:28.593164+0000","last_became_active":"2023-10-19T14:21:27.532881+0000","last_became_peered":"2023-10-19T14:21:27.532881+0000","last_unstale":"2023-10-19T14:21:28.593164+0000","last_undegraded":"2023-10-19T14:21:28.593164+0000","last_fullsized":"2023-10-19T14:21:28.593164+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:15:01.763900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.14","version":"246'6","reported_seq":18,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.590915+0000","last_change":"2023-10-19T14:21:26.371959+0000","last_active":"2023-10-19T14:21:28.590915+0000","last_peered":"2023-10-19T14:21:28.590915+0000","last_clean":"2023-10-19T14:21:28.590915+0000","last_became_active":"2023-10-19T14:21:26.371488+0000","last_became_peered":"2023-10-19T14:21:26.371488+0000","last_unstale":"2023-10-19T14:21:28.590915+0000","last_undegraded":"2023-10-19T14:21:28.590915+0000","last_fullsized":"2023-10-19T14:21:28.590915+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_clean_scrub_stamp":"2023-10-19T14:21:25.343524+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:54:02.315367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.15","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.511882+0000","last_change":"2023-10-19T14:21:10.431222+0000","last_active":"2023-10-19T14:21:27.511882+0000","last_peered":"2023-10-19T14:21:27.511882+0000","last_clean":"2023-10-19T14:21:27.511882+0000","last_became_active":"2023-10-19T14:21:10.430868+0000","last_became_peered":"2023-10-19T14:21:10.430868+0000","last_unstale":"2023-10-19T14:21:27.511882+0000","last_undegraded":"2023-10-19T14:21:27.511882+0000","last_fullsized":"2023-10-19T14:21:27.511882+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:32:54.448851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.16","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.592000+0000","last_change":"2023-10-19T14:21:27.530838+0000","last_active":"2023-10-19T14:21:28.592000+0000","last_peered":"2023-10-19T14:21:28.592000+0000","last_clean":"2023-10-19T14:21:28.592000+0000","last_became_active":"2023-10-19T14:21:27.530570+0000","last_became_peered":"2023-10-19T14:21:27.530570+0000","last_unstale":"2023-10-19T14:21:28.592000+0000","last_undegraded":"2023-10-19T14:21:28.592000+0000","last_fullsized":"2023-10-19T14:21:28.592000+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:42:09.808829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.15","version":"246'3","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.526791+0000","last_change":"2023-10-19T14:21:26.378051+0000","last_active":"2023-10-19T14:21:28.526791+0000","last_peered":"2023-10-19T14:21:28.526791+0000","last_clean":"2023-10-19T14:21:28.526791+0000","last_became_active":"2023-10-19T14:21:26.377730+0000","last_became_peered":"2023-10-19T14:21:26.377730+0000","last_unstale":"2023-10-19T14:21:28.526791+0000","last_undegraded":"2023-10-19T14:21:28.526791+0000","last_fullsized":"2023-10-19T14:21:28.526791+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_clean_scrub_stamp":"2023-10-19T14:21:25.343524+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:53:13.687057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.14","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.515702+0000","last_change":"2023-10-19T14:21:10.425461+0000","last_active":"2023-10-19T14:21:27.515702+0000","last_peered":"2023-10-19T14:21:27.515702+0000","last_clean":"2023-10-19T14:21:27.515702+0000","last_became_active":"2023-10-19T14:21:10.425149+0000","last_became_peered":"2023-10-19T14:21:10.425149+0000","last_unstale":"2023-10-19T14:21:27.515702+0000","last_undegraded":"2023-10-19T14:21:27.515702+0000","last_fullsized":"2023-10-19T14:21:27.515702+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:30:59.739271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.11","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.590695+0000","last_change":"2023-10-19T14:21:27.533305+0000","last_active":"2023-10-19T14:21:28.590695+0000","last_peered":"2023-10-19T14:21:28.590695+0000","last_clean":"2023-10-19T14:21:28.590695+0000","last_became_active":"2023-10-19T14:21:27.533034+0000","last_became_peered":"2023-10-19T14:21:27.533034+0000","last_unstale":"2023-10-19T14:21:28.590695+0000","last_undegraded":"2023-10-19T14:21:28.590695+0000","last_fullsized":"2023-10-19T14:21:28.590695+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:06:14.851117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.12","version":"0'0","reported_seq":12,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.526980+0000","last_change":"2023-10-19T14:21:26.375752+0000","last_active":"2023-10-19T14:21:28.526980+0000","last_peered":"2023-10-19T14:21:28.526980+0000","last_clean":"2023-10-19T14:21:28.526980+0000","last_became_active":"2023-10-19T14:21:26.375426+0000","last_became_peered":"2023-10-19T14:21:26.375426+0000","last_unstale":"2023-10-19T14:21:28.526980+0000","last_undegraded":"2023-10-19T14:21:28.526980+0000","last_fullsized":"2023-10-19T14:21:28.526980+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_clean_scrub_stamp":"2023-10-19T14:21:25.343524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:55:15.110443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.13","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.520916+0000","last_change":"2023-10-19T14:21:10.424757+0000","last_active":"2023-10-19T14:21:27.520916+0000","last_peered":"2023-10-19T14:21:27.520916+0000","last_clean":"2023-10-19T14:21:27.520916+0000","last_became_active":"2023-10-19T14:21:10.424482+0000","last_became_peered":"2023-10-19T14:21:10.424482+0000","last_unstale":"2023-10-19T14:21:27.520916+0000","last_undegraded":"2023-10-19T14:21:27.520916+0000","last_fullsized":"2023-10-19T14:21:27.520916+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:43:36.429903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.10","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.589844+0000","last_change":"2023-10-19T14:21:27.527982+0000","last_active":"2023-10-19T14:21:28.589844+0000","last_peered":"2023-10-19T14:21:28.589844+0000","last_clean":"2023-10-19T14:21:28.589844+0000","last_became_active":"2023-10-19T14:21:27.527692+0000","last_became_peered":"2023-10-19T14:21:27.527692+0000","last_unstale":"2023-10-19T14:21:28.589844+0000","last_undegraded":"2023-10-19T14:21:28.589844+0000","last_fullsized":"2023-10-19T14:21:28.589844+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:10:28.073265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.13","version":"246'2","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.590605+0000","last_change":"2023-10-19T14:21:26.369644+0000","last_active":"2023-10-19T14:21:28.590605+0000","last_peered":"2023-10-19T14:21:28.590605+0000","last_clean":"2023-10-19T14:21:28.590605+0000","last_became_active":"2023-10-19T14:21:26.369176+0000","last_became_peered":"2023-10-19T14:21:26.369176+0000","last_unstale":"2023-10-19T14:21:28.590605+0000","last_undegraded":"2023-10-19T14:21:28.590605+0000","last_fullsized":"2023-10-19T14:21:28.590605+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_clean_scrub_stamp":"2023-10-19T14:21:25.343524+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:19:05.362807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.12","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.536049+0000","last_change":"2023-10-19T14:21:10.423736+0000","last_active":"2023-10-19T14:21:27.536049+0000","last_peered":"2023-10-19T14:21:27.536049+0000","last_clean":"2023-10-19T14:21:27.536049+0000","last_became_active":"2023-10-19T14:21:10.423448+0000","last_became_peered":"2023-10-19T14:21:10.423448+0000","last_unstale":"2023-10-19T14:21:27.536049+0000","last_undegraded":"2023-10-19T14:21:27.536049+0000","last_fullsized":"2023-10-19T14:21:27.536049+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:42:04.331509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.13","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.590122+0000","last_change":"2023-10-19T14:21:27.529495+0000","last_active":"2023-10-19T14:21:28.590122+0000","last_peered":"2023-10-19T14:21:28.590122+0000","last_clean":"2023-10-19T14:21:28.590122+0000","last_became_active":"2023-10-19T14:21:27.529200+0000","last_became_peered":"2023-10-19T14:21:27.529200+0000","last_unstale":"2023-10-19T14:21:28.590122+0000","last_undegraded":"2023-10-19T14:21:28.590122+0000","last_fullsized":"2023-10-19T14:21:28.590122+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:21:50.232128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.10","version":"246'8","reported_seq":20,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.526325+0000","last_change":"2023-10-19T14:21:26.376659+0000","last_active":"2023-10-19T14:21:28.526325+0000","last_peered":"2023-10-19T14:21:28.526325+0000","last_clean":"2023-10-19T14:21:28.526325+0000","last_became_active":"2023-10-19T14:21:26.376196+0000","last_became_peered":"2023-10-19T14:21:26.376196+0000","last_unstale":"2023-10-19T14:21:28.526325+0000","last_undegraded":"2023-10-19T14:21:28.526325+0000","last_fullsized":"2023-10-19T14:21:28.526325+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_clean_scrub_stamp":"2023-10-19T14:21:25.343524+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T02:12:05.812260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.11","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.516210+0000","last_change":"2023-10-19T14:21:10.430551+0000","last_active":"2023-10-19T14:21:27.516210+0000","last_peered":"2023-10-19T14:21:27.516210+0000","last_clean":"2023-10-19T14:21:27.516210+0000","last_became_active":"2023-10-19T14:21:10.429509+0000","last_became_peered":"2023-10-19T14:21:10.429509+0000","last_unstale":"2023-10-19T14:21:27.516210+0000","last_undegraded":"2023-10-19T14:21:27.516210+0000","last_fullsized":"2023-10-19T14:21:27.516210+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:58:08.366505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.12","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.590510+0000","last_change":"2023-10-19T14:21:27.535132+0000","last_active":"2023-10-19T14:21:28.590510+0000","last_peered":"2023-10-19T14:21:28.590510+0000","last_clean":"2023-10-19T14:21:28.590510+0000","last_became_active":"2023-10-19T14:21:27.534830+0000","last_became_peered":"2023-10-19T14:21:27.534830+0000","last_unstale":"2023-10-19T14:21:28.590510+0000","last_undegraded":"2023-10-19T14:21:28.590510+0000","last_fullsized":"2023-10-19T14:21:28.590510+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:53:52.620951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.11","version":"0'0","reported_seq":12,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.526503+0000","last_change":"2023-10-19T14:21:26.374288+0000","last_active":"2023-10-19T14:21:28.526503+0000","last_peered":"2023-10-19T14:21:28.526503+0000","last_clean":"2023-10-19T14:21:28.526503+0000","last_became_active":"2023-10-19T14:21:26.374018+0000","last_became_peered":"2023-10-19T14:21:26.374018+0000","last_unstale":"2023-10-19T14:21:28.526503+0000","last_undegraded":"2023-10-19T14:21:28.526503+0000","last_fullsized":"2023-10-19T14:21:28.526503+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_clean_scrub_stamp":"2023-10-19T14:21:25.343524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:38:55.132511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.10","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.510813+0000","last_change":"2023-10-19T14:21:10.437276+0000","last_active":"2023-10-19T14:21:27.510813+0000","last_peered":"2023-10-19T14:21:27.510813+0000","last_clean":"2023-10-19T14:21:27.510813+0000","last_became_active":"2023-10-19T14:21:10.437013+0000","last_became_peered":"2023-10-19T14:21:10.437013+0000","last_unstale":"2023-10-19T14:21:27.510813+0000","last_undegraded":"2023-10-19T14:21:27.510813+0000","last_fullsized":"2023-10-19T14:21:27.510813+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:06:58.837634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.d","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.592119+0000","last_change":"2023-10-19T14:21:27.528223+0000","last_active":"2023-10-19T14:21:28.592119+0000","last_peered":"2023-10-19T14:21:28.592119+0000","last_clean":"2023-10-19T14:21:28.592119+0000","last_became_active":"2023-10-19T14:21:27.527684+0000","last_became_peered":"2023-10-19T14:21:27.527684+0000","last_unstale":"2023-10-19T14:21:28.592119+0000","last_undegraded":"2023-10-19T14:21:28.592119+0000","last_fullsized":"2023-10-19T14:21:28.592119+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:40:13.261454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.e","version":"246'2","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.591227+0000","last_change":"2023-10-19T14:21:26.365681+0000","last_active":"2023-10-19T14:21:28.591227+0000","last_peered":"2023-10-19T14:21:28.591227+0000","last_clean":"2023-10-19T14:21:28.591227+0000","last_became_active":"2023-10-19T14:21:26.365428+0000","last_became_peered":"2023-10-19T14:21:26.365428+0000","last_unstale":"2023-10-19T14:21:28.591227+0000","last_undegraded":"2023-10-19T14:21:28.591227+0000","last_fullsized":"2023-10-19T14:21:28.591227+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_clean_scrub_stamp":"2023-10-19T14:21:25.343524+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:45:26.686054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.f","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.525813+0000","last_change":"2023-10-19T14:21:10.429097+0000","last_active":"2023-10-19T14:21:27.525813+0000","last_peered":"2023-10-19T14:21:27.525813+0000","last_clean":"2023-10-19T14:21:27.525813+0000","last_became_active":"2023-10-19T14:21:10.428825+0000","last_became_peered":"2023-10-19T14:21:10.428825+0000","last_unstale":"2023-10-19T14:21:27.525813+0000","last_undegraded":"2023-10-19T14:21:27.525813+0000","last_fullsized":"2023-10-19T14:21:27.525813+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:33:10.076418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.c","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.592421+0000","last_change":"2023-10-19T14:21:27.531130+0000","last_active":"2023-10-19T14:21:28.592421+0000","last_peered":"2023-10-19T14:21:28.592421+0000","last_clean":"2023-10-19T14:21:28.592421+0000","last_became_active":"2023-10-19T14:21:27.529377+0000","last_became_peered":"2023-10-19T14:21:27.529377+0000","last_unstale":"2023-10-19T14:21:28.592421+0000","last_undegraded":"2023-10-19T14:21:28.592421+0000","last_fullsized":"2023-10-19T14:21:28.592421+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:45:11.239855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.f","version":"246'1","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.591966+0000","last_change":"2023-10-19T14:21:26.372895+0000","last_active":"2023-10-19T14:21:28.591966+0000","last_peered":"2023-10-19T14:21:28.591966+0000","last_clean":"2023-10-19T14:21:28.591966+0000","last_became_active":"2023-10-19T14:21:26.372573+0000","last_became_peered":"2023-10-19T14:21:26.372573+0000","last_unstale":"2023-10-19T14:21:28.591966+0000","last_undegraded":"2023-10-19T14:21:28.591966+0000","last_fullsized":"2023-10-19T14:21:28.591966+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_clean_scrub_stamp":"2023-10-19T14:21:25.343524+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:10:27.070104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.e","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.512194+0000","last_change":"2023-10-19T14:21:10.425661+0000","last_active":"2023-10-19T14:21:27.512194+0000","last_peered":"2023-10-19T14:21:27.512194+0000","last_clean":"2023-10-19T14:21:27.512194+0000","last_became_active":"2023-10-19T14:21:10.425369+0000","last_became_peered":"2023-10-19T14:21:10.425369+0000","last_unstale":"2023-10-19T14:21:27.512194+0000","last_undegraded":"2023-10-19T14:21:27.512194+0000","last_fullsized":"2023-10-19T14:21:27.512194+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:54:48.155840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.f","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.593177+0000","last_change":"2023-10-19T14:21:27.534960+0000","last_active":"2023-10-19T14:21:28.593177+0000","last_peered":"2023-10-19T14:21:28.593177+0000","last_clean":"2023-10-19T14:21:28.593177+0000","last_became_active":"2023-10-19T14:21:27.533567+0000","last_became_peered":"2023-10-19T14:21:27.533567+0000","last_unstale":"2023-10-19T14:21:28.593177+0000","last_undegraded":"2023-10-19T14:21:28.593177+0000","last_fullsized":"2023-10-19T14:21:28.593177+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:41:41.163291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.c","version":"0'0","reported_seq":12,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.590925+0000","last_change":"2023-10-19T14:21:26.372370+0000","last_active":"2023-10-19T14:21:28.590925+0000","last_peered":"2023-10-19T14:21:28.590925+0000","last_clean":"2023-10-19T14:21:28.590925+0000","last_became_active":"2023-10-19T14:21:26.371478+0000","last_became_peered":"2023-10-19T14:21:26.371478+0000","last_unstale":"2023-10-19T14:21:28.590925+0000","last_undegraded":"2023-10-19T14:21:28.590925+0000","last_fullsized":"2023-10-19T14:21:28.590925+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_clean_scrub_stamp":"2023-10-19T14:21:25.343524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:00:13.656453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.d","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.517542+0000","last_change":"2023-10-19T14:21:10.423050+0000","last_active":"2023-10-19T14:21:27.517542+0000","last_peered":"2023-10-19T14:21:27.517542+0000","last_clean":"2023-10-19T14:21:27.517542+0000","last_became_active":"2023-10-19T14:21:10.422727+0000","last_became_peered":"2023-10-19T14:21:10.422727+0000","last_unstale":"2023-10-19T14:21:27.517542+0000","last_undegraded":"2023-10-19T14:21:27.517542+0000","last_fullsized":"2023-10-19T14:21:27.517542+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:21:43.393777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.e","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.527011+0000","last_change":"2023-10-19T14:21:27.533236+0000","last_active":"2023-10-19T14:21:28.527011+0000","last_peered":"2023-10-19T14:21:28.527011+0000","last_clean":"2023-10-19T14:21:28.527011+0000","last_became_active":"2023-10-19T14:21:27.532931+0000","last_became_peered":"2023-10-19T14:21:27.532931+0000","last_unstale":"2023-10-19T14:21:28.527011+0000","last_undegraded":"2023-10-19T14:21:28.527011+0000","last_fullsized":"2023-10-19T14:21:28.527011+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:49:00.966124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.d","version":"246'1","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.527454+0000","last_change":"2023-10-19T14:21:26.364297+0000","last_active":"2023-10-19T14:21:28.527454+0000","last_peered":"2023-10-19T14:21:28.527454+0000","last_clean":"2023-10-19T14:21:28.527454+0000","last_became_active":"2023-10-19T14:21:26.364036+0000","last_became_peered":"2023-10-19T14:21:26.364036+0000","last_unstale":"2023-10-19T14:21:28.527454+0000","last_undegraded":"2023-10-19T14:21:28.527454+0000","last_fullsized":"2023-10-19T14:21:28.527454+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_clean_scrub_stamp":"2023-10-19T14:21:25.343524+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:30:33.383813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.c","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.522998+0000","last_change":"2023-10-19T14:21:10.432921+0000","last_active":"2023-10-19T14:21:27.522998+0000","last_peered":"2023-10-19T14:21:27.522998+0000","last_clean":"2023-10-19T14:21:27.522998+0000","last_became_active":"2023-10-19T14:21:10.432654+0000","last_became_peered":"2023-10-19T14:21:10.432654+0000","last_unstale":"2023-10-19T14:21:27.522998+0000","last_undegraded":"2023-10-19T14:21:27.522998+0000","last_fullsized":"2023-10-19T14:21:27.522998+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:42:44.275430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.9","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.590841+0000","last_change":"2023-10-19T14:21:27.531873+0000","last_active":"2023-10-19T14:21:28.590841+0000","last_peered":"2023-10-19T14:21:28.590841+0000","last_clean":"2023-10-19T14:21:28.590841+0000","last_became_active":"2023-10-19T14:21:27.531581+0000","last_became_peered":"2023-10-19T14:21:27.531581+0000","last_unstale":"2023-10-19T14:21:28.590841+0000","last_undegraded":"2023-10-19T14:21:28.590841+0000","last_fullsized":"2023-10-19T14:21:28.590841+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:17:46.220498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.a","version":"0'0","reported_seq":12,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.590098+0000","last_change":"2023-10-19T14:21:26.374177+0000","last_active":"2023-10-19T14:21:28.590098+0000","last_peered":"2023-10-19T14:21:28.590098+0000","last_clean":"2023-10-19T14:21:28.590098+0000","last_became_active":"2023-10-19T14:21:26.373883+0000","last_became_peered":"2023-10-19T14:21:26.373883+0000","last_unstale":"2023-10-19T14:21:28.590098+0000","last_undegraded":"2023-10-19T14:21:28.590098+0000","last_fullsized":"2023-10-19T14:21:28.590098+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_clean_scrub_stamp":"2023-10-19T14:21:25.343524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:12:22.183091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.b","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.512502+0000","last_change":"2023-10-19T14:21:10.422339+0000","last_active":"2023-10-19T14:21:27.512502+0000","last_peered":"2023-10-19T14:21:27.512502+0000","last_clean":"2023-10-19T14:21:27.512502+0000","last_became_active":"2023-10-19T14:21:10.422033+0000","last_became_peered":"2023-10-19T14:21:10.422033+0000","last_unstale":"2023-10-19T14:21:27.512502+0000","last_undegraded":"2023-10-19T14:21:27.512502+0000","last_fullsized":"2023-10-19T14:21:27.512502+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:19:11.812010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.8","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.524439+0000","last_change":"2023-10-19T14:21:27.532550+0000","last_active":"2023-10-19T14:21:28.524439+0000","last_peered":"2023-10-19T14:21:28.524439+0000","last_clean":"2023-10-19T14:21:28.524439+0000","last_became_active":"2023-10-19T14:21:27.532289+0000","last_became_peered":"2023-10-19T14:21:27.532289+0000","last_unstale":"2023-10-19T14:21:28.524439+0000","last_undegraded":"2023-10-19T14:21:28.524439+0000","last_fullsized":"2023-10-19T14:21:28.524439+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:54:21.231604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.b","version":"246'2","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.590650+0000","last_change":"2023-10-19T14:21:26.375601+0000","last_active":"2023-10-19T14:21:28.590650+0000","last_peered":"2023-10-19T14:21:28.590650+0000","last_clean":"2023-10-19T14:21:28.590650+0000","last_became_active":"2023-10-19T14:21:26.375309+0000","last_became_peered":"2023-10-19T14:21:26.375309+0000","last_unstale":"2023-10-19T14:21:28.590650+0000","last_undegraded":"2023-10-19T14:21:28.590650+0000","last_fullsized":"2023-10-19T14:21:28.590650+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_clean_scrub_stamp":"2023-10-19T14:21:25.343524+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:02:39.807380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.a","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.521896+0000","last_change":"2023-10-19T14:21:10.438315+0000","last_active":"2023-10-19T14:21:27.521896+0000","last_peered":"2023-10-19T14:21:27.521896+0000","last_clean":"2023-10-19T14:21:27.521896+0000","last_became_active":"2023-10-19T14:21:10.438026+0000","last_became_peered":"2023-10-19T14:21:10.438026+0000","last_unstale":"2023-10-19T14:21:27.521896+0000","last_undegraded":"2023-10-19T14:21:27.521896+0000","last_fullsized":"2023-10-19T14:21:27.521896+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:36:02.222588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.b","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.524601+0000","last_change":"2023-10-19T14:21:27.542409+0000","last_active":"2023-10-19T14:21:28.524601+0000","last_peered":"2023-10-19T14:21:28.524601+0000","last_clean":"2023-10-19T14:21:28.524601+0000","last_became_active":"2023-10-19T14:21:27.542134+0000","last_became_peered":"2023-10-19T14:21:27.542134+0000","last_unstale":"2023-10-19T14:21:28.524601+0000","last_undegraded":"2023-10-19T14:21:28.524601+0000","last_fullsized":"2023-10-19T14:21:28.524601+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:43:22.958184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.8","version":"0'0","reported_seq":12,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.526628+0000","last_change":"2023-10-19T14:21:26.376475+0000","last_active":"2023-10-19T14:21:28.526628+0000","last_peered":"2023-10-19T14:21:28.526628+0000","last_clean":"2023-10-19T14:21:28.526628+0000","last_became_active":"2023-10-19T14:21:26.375820+0000","last_became_peered":"2023-10-19T14:21:26.375820+0000","last_unstale":"2023-10-19T14:21:28.526628+0000","last_undegraded":"2023-10-19T14:21:28.526628+0000","last_fullsized":"2023-10-19T14:21:28.526628+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_clean_scrub_stamp":"2023-10-19T14:21:25.343524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:09:51.132064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.9","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.516297+0000","last_change":"2023-10-19T14:21:10.430361+0000","last_active":"2023-10-19T14:21:27.516297+0000","last_peered":"2023-10-19T14:21:27.516297+0000","last_clean":"2023-10-19T14:21:27.516297+0000","last_became_active":"2023-10-19T14:21:10.429861+0000","last_became_peered":"2023-10-19T14:21:10.429861+0000","last_unstale":"2023-10-19T14:21:27.516297+0000","last_undegraded":"2023-10-19T14:21:27.516297+0000","last_fullsized":"2023-10-19T14:21:27.516297+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:50:34.147669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.a","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.526568+0000","last_change":"2023-10-19T14:21:27.518932+0000","last_active":"2023-10-19T14:21:28.526568+0000","last_peered":"2023-10-19T14:21:28.526568+0000","last_clean":"2023-10-19T14:21:28.526568+0000","last_became_active":"2023-10-19T14:21:27.518518+0000","last_became_peered":"2023-10-19T14:21:27.518518+0000","last_unstale":"2023-10-19T14:21:28.526568+0000","last_undegraded":"2023-10-19T14:21:28.526568+0000","last_fullsized":"2023-10-19T14:21:28.526568+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:50:03.901367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.9","version":"0'0","reported_seq":12,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.524769+0000","last_change":"2023-10-19T14:21:26.364933+0000","last_active":"2023-10-19T14:21:28.524769+0000","last_peered":"2023-10-19T14:21:28.524769+0000","last_clean":"2023-10-19T14:21:28.524769+0000","last_became_active":"2023-10-19T14:21:26.364682+0000","last_became_peered":"2023-10-19T14:21:26.364682+0000","last_unstale":"2023-10-19T14:21:28.524769+0000","last_undegraded":"2023-10-19T14:21:28.524769+0000","last_fullsized":"2023-10-19T14:21:28.524769+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_clean_scrub_stamp":"2023-10-19T14:21:25.343524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:18:23.087445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.8","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.523749+0000","last_change":"2023-10-19T14:21:10.413051+0000","last_active":"2023-10-19T14:21:27.523749+0000","last_peered":"2023-10-19T14:21:27.523749+0000","last_clean":"2023-10-19T14:21:27.523749+0000","last_became_active":"2023-10-19T14:21:10.412765+0000","last_became_peered":"2023-10-19T14:21:10.412765+0000","last_unstale":"2023-10-19T14:21:27.523749+0000","last_undegraded":"2023-10-19T14:21:27.523749+0000","last_fullsized":"2023-10-19T14:21:27.523749+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T02:02:33.398058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.5","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.590793+0000","last_change":"2023-10-19T14:21:27.534347+0000","last_active":"2023-10-19T14:21:28.590793+0000","last_peered":"2023-10-19T14:21:28.590793+0000","last_clean":"2023-10-19T14:21:28.590793+0000","last_became_active":"2023-10-19T14:21:27.534058+0000","last_became_peered":"2023-10-19T14:21:27.534058+0000","last_unstale":"2023-10-19T14:21:28.590793+0000","last_undegraded":"2023-10-19T14:21:28.590793+0000","last_fullsized":"2023-10-19T14:21:28.590793+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:15:17.090163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.6","version":"0'0","reported_seq":12,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.591314+0000","last_change":"2023-10-19T14:21:26.363091+0000","last_active":"2023-10-19T14:21:28.591314+0000","last_peered":"2023-10-19T14:21:28.591314+0000","last_clean":"2023-10-19T14:21:28.591314+0000","last_became_active":"2023-10-19T14:21:26.362822+0000","last_became_peered":"2023-10-19T14:21:26.362822+0000","last_unstale":"2023-10-19T14:21:28.591314+0000","last_undegraded":"2023-10-19T14:21:28.591314+0000","last_fullsized":"2023-10-19T14:21:28.591314+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_clean_scrub_stamp":"2023-10-19T14:21:25.343524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:29:55.911261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.7","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.524677+0000","last_change":"2023-10-19T14:21:10.430882+0000","last_active":"2023-10-19T14:21:27.524677+0000","last_peered":"2023-10-19T14:21:27.524677+0000","last_clean":"2023-10-19T14:21:27.524677+0000","last_became_active":"2023-10-19T14:21:10.430631+0000","last_became_peered":"2023-10-19T14:21:10.430631+0000","last_unstale":"2023-10-19T14:21:27.524677+0000","last_undegraded":"2023-10-19T14:21:27.524677+0000","last_fullsized":"2023-10-19T14:21:27.524677+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:02:43.850106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.4","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.592010+0000","last_change":"2023-10-19T14:21:27.528861+0000","last_active":"2023-10-19T14:21:28.592010+0000","last_peered":"2023-10-19T14:21:28.592010+0000","last_clean":"2023-10-19T14:21:28.592010+0000","last_became_active":"2023-10-19T14:21:27.528600+0000","last_became_peered":"2023-10-19T14:21:27.528600+0000","last_unstale":"2023-10-19T14:21:28.592010+0000","last_undegraded":"2023-10-19T14:21:28.592010+0000","last_fullsized":"2023-10-19T14:21:28.592010+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:53:51.253050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.7","version":"246'2","reported_seq":14,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.527121+0000","last_change":"2023-10-19T14:21:26.375412+0000","last_active":"2023-10-19T14:21:28.527121+0000","last_peered":"2023-10-19T14:21:28.527121+0000","last_clean":"2023-10-19T14:21:28.527121+0000","last_became_active":"2023-10-19T14:21:26.375133+0000","last_became_peered":"2023-10-19T14:21:26.375133+0000","last_unstale":"2023-10-19T14:21:28.527121+0000","last_undegraded":"2023-10-19T14:21:28.527121+0000","last_fullsized":"2023-10-19T14:21:28.527121+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_clean_scrub_stamp":"2023-10-19T14:21:25.343524+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:13:37.862094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.6","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.512746+0000","last_change":"2023-10-19T14:21:10.428612+0000","last_active":"2023-10-19T14:21:27.512746+0000","last_peered":"2023-10-19T14:21:27.512746+0000","last_clean":"2023-10-19T14:21:27.512746+0000","last_became_active":"2023-10-19T14:21:10.428248+0000","last_became_peered":"2023-10-19T14:21:10.428248+0000","last_unstale":"2023-10-19T14:21:27.512746+0000","last_undegraded":"2023-10-19T14:21:27.512746+0000","last_fullsized":"2023-10-19T14:21:27.512746+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:13:47.366105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.7","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.591911+0000","last_change":"2023-10-19T14:21:27.532673+0000","last_active":"2023-10-19T14:21:28.591911+0000","last_peered":"2023-10-19T14:21:28.591911+0000","last_clean":"2023-10-19T14:21:28.591911+0000","last_became_active":"2023-10-19T14:21:27.532411+0000","last_became_peered":"2023-10-19T14:21:27.532411+0000","last_unstale":"2023-10-19T14:21:28.591911+0000","last_undegraded":"2023-10-19T14:21:28.591911+0000","last_fullsized":"2023-10-19T14:21:28.591911+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:00:40.916980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.4","version":"0'0","reported_seq":12,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.527041+0000","last_change":"2023-10-19T14:21:26.364085+0000","last_active":"2023-10-19T14:21:28.527041+0000","last_peered":"2023-10-19T14:21:28.527041+0000","last_clean":"2023-10-19T14:21:28.527041+0000","last_became_active":"2023-10-19T14:21:26.363821+0000","last_became_peered":"2023-10-19T14:21:26.363821+0000","last_unstale":"2023-10-19T14:21:28.527041+0000","last_undegraded":"2023-10-19T14:21:28.527041+0000","last_fullsized":"2023-10-19T14:21:28.527041+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_clean_scrub_stamp":"2023-10-19T14:21:25.343524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:47:33.310989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.5","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.517643+0000","last_change":"2023-10-19T14:21:10.424660+0000","last_active":"2023-10-19T14:21:27.517643+0000","last_peered":"2023-10-19T14:21:27.517643+0000","last_clean":"2023-10-19T14:21:27.517643+0000","last_became_active":"2023-10-19T14:21:10.423849+0000","last_became_peered":"2023-10-19T14:21:10.423849+0000","last_unstale":"2023-10-19T14:21:27.517643+0000","last_undegraded":"2023-10-19T14:21:27.517643+0000","last_fullsized":"2023-10-19T14:21:27.517643+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:44:54.282098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.6","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.527407+0000","last_change":"2023-10-19T14:21:27.519397+0000","last_active":"2023-10-19T14:21:28.527407+0000","last_peered":"2023-10-19T14:21:28.527407+0000","last_clean":"2023-10-19T14:21:28.527407+0000","last_became_active":"2023-10-19T14:21:27.519113+0000","last_became_peered":"2023-10-19T14:21:27.519113+0000","last_unstale":"2023-10-19T14:21:28.527407+0000","last_undegraded":"2023-10-19T14:21:28.527407+0000","last_fullsized":"2023-10-19T14:21:28.527407+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:04:58.823569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.5","version":"0'0","reported_seq":12,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.527138+0000","last_change":"2023-10-19T14:21:26.363474+0000","last_active":"2023-10-19T14:21:28.527138+0000","last_peered":"2023-10-19T14:21:28.527138+0000","last_clean":"2023-10-19T14:21:28.527138+0000","last_became_active":"2023-10-19T14:21:26.363190+0000","last_became_peered":"2023-10-19T14:21:26.363190+0000","last_unstale":"2023-10-19T14:21:28.527138+0000","last_undegraded":"2023-10-19T14:21:28.527138+0000","last_fullsized":"2023-10-19T14:21:28.527138+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_clean_scrub_stamp":"2023-10-19T14:21:25.343524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:33:10.403448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.4","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.515692+0000","last_change":"2023-10-19T14:21:10.432004+0000","last_active":"2023-10-19T14:21:27.515692+0000","last_peered":"2023-10-19T14:21:27.515692+0000","last_clean":"2023-10-19T14:21:27.515692+0000","last_became_active":"2023-10-19T14:21:10.431687+0000","last_became_peered":"2023-10-19T14:21:10.431687+0000","last_unstale":"2023-10-19T14:21:27.515692+0000","last_undegraded":"2023-10-19T14:21:27.515692+0000","last_fullsized":"2023-10-19T14:21:27.515692+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:31:38.548044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.1","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.524606+0000","last_change":"2023-10-19T14:21:27.540440+0000","last_active":"2023-10-19T14:21:28.524606+0000","last_peered":"2023-10-19T14:21:28.524606+0000","last_clean":"2023-10-19T14:21:28.524606+0000","last_became_active":"2023-10-19T14:21:27.540079+0000","last_became_peered":"2023-10-19T14:21:27.540079+0000","last_unstale":"2023-10-19T14:21:28.524606+0000","last_undegraded":"2023-10-19T14:21:28.524606+0000","last_fullsized":"2023-10-19T14:21:28.524606+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:09:52.510151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.2","version":"0'0","reported_seq":12,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.590089+0000","last_change":"2023-10-19T14:21:26.376809+0000","last_active":"2023-10-19T14:21:28.590089+0000","last_peered":"2023-10-19T14:21:28.590089+0000","last_clean":"2023-10-19T14:21:28.590089+0000","last_became_active":"2023-10-19T14:21:26.376555+0000","last_became_peered":"2023-10-19T14:21:26.376555+0000","last_unstale":"2023-10-19T14:21:28.590089+0000","last_undegraded":"2023-10-19T14:21:28.590089+0000","last_fullsized":"2023-10-19T14:21:28.590089+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_clean_scrub_stamp":"2023-10-19T14:21:25.343524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:12:03.508740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.3","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.536100+0000","last_change":"2023-10-19T14:21:10.419200+0000","last_active":"2023-10-19T14:21:27.536100+0000","last_peered":"2023-10-19T14:21:27.536100+0000","last_clean":"2023-10-19T14:21:27.536100+0000","last_became_active":"2023-10-19T14:21:10.418921+0000","last_became_peered":"2023-10-19T14:21:10.418921+0000","last_unstale":"2023-10-19T14:21:27.536100+0000","last_undegraded":"2023-10-19T14:21:27.536100+0000","last_fullsized":"2023-10-19T14:21:27.536100+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:18:49.408210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.0","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.526340+0000","last_change":"2023-10-19T14:21:27.520641+0000","last_active":"2023-10-19T14:21:28.526340+0000","last_peered":"2023-10-19T14:21:28.526340+0000","last_clean":"2023-10-19T14:21:28.526340+0000","last_became_active":"2023-10-19T14:21:27.520348+0000","last_became_peered":"2023-10-19T14:21:27.520348+0000","last_unstale":"2023-10-19T14:21:28.526340+0000","last_undegraded":"2023-10-19T14:21:28.526340+0000","last_fullsized":"2023-10-19T14:21:28.526340+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:49:26.624655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.3","version":"0'0","reported_seq":12,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.590541+0000","last_change":"2023-10-19T14:21:26.368505+0000","last_active":"2023-10-19T14:21:28.590541+0000","last_peered":"2023-10-19T14:21:28.590541+0000","last_clean":"2023-10-19T14:21:28.590541+0000","last_became_active":"2023-10-19T14:21:26.368200+0000","last_became_peered":"2023-10-19T14:21:26.368200+0000","last_unstale":"2023-10-19T14:21:28.590541+0000","last_undegraded":"2023-10-19T14:21:28.590541+0000","last_fullsized":"2023-10-19T14:21:28.590541+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_clean_scrub_stamp":"2023-10-19T14:21:25.343524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:42:25.318847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.2","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.536075+0000","last_change":"2023-10-19T14:21:10.426386+0000","last_active":"2023-10-19T14:21:27.536075+0000","last_peered":"2023-10-19T14:21:27.536075+0000","last_clean":"2023-10-19T14:21:27.536075+0000","last_became_active":"2023-10-19T14:21:10.426117+0000","last_became_peered":"2023-10-19T14:21:10.426117+0000","last_unstale":"2023-10-19T14:21:27.536075+0000","last_undegraded":"2023-10-19T14:21:27.536075+0000","last_fullsized":"2023-10-19T14:21:27.536075+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:46:24.186941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.3","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.592690+0000","last_change":"2023-10-19T14:21:27.531185+0000","last_active":"2023-10-19T14:21:28.592690+0000","last_peered":"2023-10-19T14:21:28.592690+0000","last_clean":"2023-10-19T14:21:28.592690+0000","last_became_active":"2023-10-19T14:21:27.530896+0000","last_became_peered":"2023-10-19T14:21:27.530896+0000","last_unstale":"2023-10-19T14:21:28.592690+0000","last_undegraded":"2023-10-19T14:21:28.592690+0000","last_fullsized":"2023-10-19T14:21:28.592690+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:43:20.053694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.0","version":"0'0","reported_seq":12,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.590464+0000","last_change":"2023-10-19T14:21:26.364519+0000","last_active":"2023-10-19T14:21:28.590464+0000","last_peered":"2023-10-19T14:21:28.590464+0000","last_clean":"2023-10-19T14:21:28.590464+0000","last_became_active":"2023-10-19T14:21:26.364257+0000","last_became_peered":"2023-10-19T14:21:26.364257+0000","last_unstale":"2023-10-19T14:21:28.590464+0000","last_undegraded":"2023-10-19T14:21:28.590464+0000","last_fullsized":"2023-10-19T14:21:28.590464+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_clean_scrub_stamp":"2023-10-19T14:21:25.343524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:33:26.031016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.521704+0000","last_change":"2023-10-19T14:21:10.429319+0000","last_active":"2023-10-19T14:21:27.521704+0000","last_peered":"2023-10-19T14:21:27.521704+0000","last_clean":"2023-10-19T14:21:27.521704+0000","last_became_active":"2023-10-19T14:21:10.429037+0000","last_became_peered":"2023-10-19T14:21:10.429037+0000","last_unstale":"2023-10-19T14:21:27.521704+0000","last_undegraded":"2023-10-19T14:21:27.521704+0000","last_fullsized":"2023-10-19T14:21:27.521704+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:38:02.329128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.2","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.524445+0000","last_change":"2023-10-19T14:21:27.541136+0000","last_active":"2023-10-19T14:21:28.524445+0000","last_peered":"2023-10-19T14:21:28.524445+0000","last_clean":"2023-10-19T14:21:28.524445+0000","last_became_active":"2023-10-19T14:21:27.540862+0000","last_became_peered":"2023-10-19T14:21:27.540862+0000","last_unstale":"2023-10-19T14:21:28.524445+0000","last_undegraded":"2023-10-19T14:21:28.524445+0000","last_fullsized":"2023-10-19T14:21:28.524445+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:44:29.301692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1","version":"0'0","reported_seq":12,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.526874+0000","last_change":"2023-10-19T14:21:26.364816+0000","last_active":"2023-10-19T14:21:28.526874+0000","last_peered":"2023-10-19T14:21:28.526874+0000","last_clean":"2023-10-19T14:21:28.526874+0000","last_became_active":"2023-10-19T14:21:26.364542+0000","last_became_peered":"2023-10-19T14:21:26.364542+0000","last_unstale":"2023-10-19T14:21:28.526874+0000","last_undegraded":"2023-10-19T14:21:28.526874+0000","last_fullsized":"2023-10-19T14:21:28.526874+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:25.343524+0000","last_clean_scrub_stamp":"2023-10-19T14:21:25.343524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:29:27.857081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.0","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.515824+0000","last_change":"2023-10-19T14:21:10.432051+0000","last_active":"2023-10-19T14:21:27.515824+0000","last_peered":"2023-10-19T14:21:27.515824+0000","last_clean":"2023-10-19T14:21:27.515824+0000","last_became_active":"2023-10-19T14:21:10.431732+0000","last_became_peered":"2023-10-19T14:21:10.431732+0000","last_unstale":"2023-10-19T14:21:27.515824+0000","last_undegraded":"2023-10-19T14:21:27.515824+0000","last_fullsized":"2023-10-19T14:21:27.515824+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:19:07.965561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.22","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.591443+0000","last_change":"2023-10-19T14:21:27.530847+0000","last_active":"2023-10-19T14:21:28.591443+0000","last_peered":"2023-10-19T14:21:28.591443+0000","last_clean":"2023-10-19T14:21:28.591443+0000","last_became_active":"2023-10-19T14:21:27.530561+0000","last_became_peered":"2023-10-19T14:21:27.530561+0000","last_unstale":"2023-10-19T14:21:28.591443+0000","last_undegraded":"2023-10-19T14:21:28.591443+0000","last_fullsized":"2023-10-19T14:21:28.591443+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:14:10.207370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.20","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.522619+0000","last_change":"2023-10-19T14:21:10.438066+0000","last_active":"2023-10-19T14:21:27.522619+0000","last_peered":"2023-10-19T14:21:27.522619+0000","last_clean":"2023-10-19T14:21:27.522619+0000","last_became_active":"2023-10-19T14:21:10.437789+0000","last_became_peered":"2023-10-19T14:21:10.437789+0000","last_unstale":"2023-10-19T14:21:27.522619+0000","last_undegraded":"2023-10-19T14:21:27.522619+0000","last_fullsized":"2023-10-19T14:21:27.522619+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:51:31.199054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1f","version":"242'4","reported_seq":20,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.515118+0000","last_change":"2023-10-19T14:21:09.412253+0000","last_active":"2023-10-19T14:21:27.515118+0000","last_peered":"2023-10-19T14:21:27.515118+0000","last_clean":"2023-10-19T14:21:27.515118+0000","last_became_active":"2023-10-19T14:21:09.411979+0000","last_became_peered":"2023-10-19T14:21:09.411979+0000","last_unstale":"2023-10-19T14:21:27.515118+0000","last_undegraded":"2023-10-19T14:21:27.515118+0000","last_fullsized":"2023-10-19T14:21:27.515118+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:21:16.759464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.20","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.524261+0000","last_change":"2023-10-19T14:21:27.546161+0000","last_active":"2023-10-19T14:21:28.524261+0000","last_peered":"2023-10-19T14:21:28.524261+0000","last_clean":"2023-10-19T14:21:28.524261+0000","last_became_active":"2023-10-19T14:21:27.545854+0000","last_became_peered":"2023-10-19T14:21:27.545854+0000","last_unstale":"2023-10-19T14:21:28.524261+0000","last_undegraded":"2023-10-19T14:21:28.524261+0000","last_fullsized":"2023-10-19T14:21:28.524261+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:43:38.888599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.22","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.516831+0000","last_change":"2023-10-19T14:21:10.426693+0000","last_active":"2023-10-19T14:21:27.516831+0000","last_peered":"2023-10-19T14:21:27.516831+0000","last_clean":"2023-10-19T14:21:27.516831+0000","last_became_active":"2023-10-19T14:21:10.426439+0000","last_became_peered":"2023-10-19T14:21:10.426439+0000","last_unstale":"2023-10-19T14:21:27.516831+0000","last_undegraded":"2023-10-19T14:21:27.516831+0000","last_fullsized":"2023-10-19T14:21:27.516831+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:04:41.865811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.1d","version":"242'2","reported_seq":18,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.522334+0000","last_change":"2023-10-19T14:21:09.414225+0000","last_active":"2023-10-19T14:21:27.522334+0000","last_peered":"2023-10-19T14:21:27.522334+0000","last_clean":"2023-10-19T14:21:27.522334+0000","last_became_active":"2023-10-19T14:21:09.413679+0000","last_became_peered":"2023-10-19T14:21:09.413679+0000","last_unstale":"2023-10-19T14:21:27.522334+0000","last_undegraded":"2023-10-19T14:21:27.522334+0000","last_fullsized":"2023-10-19T14:21:27.522334+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:30:30.420013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.21","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.524463+0000","last_change":"2023-10-19T14:21:27.541516+0000","last_active":"2023-10-19T14:21:28.524463+0000","last_peered":"2023-10-19T14:21:28.524463+0000","last_clean":"2023-10-19T14:21:28.524463+0000","last_became_active":"2023-10-19T14:21:27.541266+0000","last_became_peered":"2023-10-19T14:21:27.541266+0000","last_unstale":"2023-10-19T14:21:28.524463+0000","last_undegraded":"2023-10-19T14:21:28.524463+0000","last_fullsized":"2023-10-19T14:21:28.524463+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:10:12.738332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.23","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.516714+0000","last_change":"2023-10-19T14:21:10.426221+0000","last_active":"2023-10-19T14:21:27.516714+0000","last_peered":"2023-10-19T14:21:27.516714+0000","last_clean":"2023-10-19T14:21:27.516714+0000","last_became_active":"2023-10-19T14:21:10.425945+0000","last_became_peered":"2023-10-19T14:21:10.425945+0000","last_unstale":"2023-10-19T14:21:27.516714+0000","last_undegraded":"2023-10-19T14:21:27.516714+0000","last_fullsized":"2023-10-19T14:21:27.516714+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:34:44.606413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1c","version":"242'2","reported_seq":18,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.535579+0000","last_change":"2023-10-19T14:21:09.412165+0000","last_active":"2023-10-19T14:21:27.535579+0000","last_peered":"2023-10-19T14:21:27.535579+0000","last_clean":"2023-10-19T14:21:27.535579+0000","last_became_active":"2023-10-19T14:21:09.411874+0000","last_became_peered":"2023-10-19T14:21:09.411874+0000","last_unstale":"2023-10-19T14:21:27.535579+0000","last_undegraded":"2023-10-19T14:21:27.535579+0000","last_fullsized":"2023-10-19T14:21:27.535579+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:14:02.794831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.26","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.525340+0000","last_change":"2023-10-19T14:21:27.536718+0000","last_active":"2023-10-19T14:21:28.525340+0000","last_peered":"2023-10-19T14:21:28.525340+0000","last_clean":"2023-10-19T14:21:28.525340+0000","last_became_active":"2023-10-19T14:21:27.536417+0000","last_became_peered":"2023-10-19T14:21:27.536417+0000","last_unstale":"2023-10-19T14:21:28.525340+0000","last_undegraded":"2023-10-19T14:21:28.525340+0000","last_fullsized":"2023-10-19T14:21:28.525340+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:03:20.484441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.24","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.535488+0000","last_change":"2023-10-19T14:21:10.430265+0000","last_active":"2023-10-19T14:21:27.535488+0000","last_peered":"2023-10-19T14:21:27.535488+0000","last_clean":"2023-10-19T14:21:27.535488+0000","last_became_active":"2023-10-19T14:21:10.429918+0000","last_became_peered":"2023-10-19T14:21:10.429918+0000","last_unstale":"2023-10-19T14:21:27.535488+0000","last_undegraded":"2023-10-19T14:21:27.535488+0000","last_fullsized":"2023-10-19T14:21:27.535488+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:02:23.852783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.1b","version":"242'1","reported_seq":17,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.517545+0000","last_change":"2023-10-19T14:21:09.411062+0000","last_active":"2023-10-19T14:21:27.517545+0000","last_peered":"2023-10-19T14:21:27.517545+0000","last_clean":"2023-10-19T14:21:27.517545+0000","last_became_active":"2023-10-19T14:21:09.410715+0000","last_became_peered":"2023-10-19T14:21:09.410715+0000","last_unstale":"2023-10-19T14:21:27.517545+0000","last_undegraded":"2023-10-19T14:21:27.517545+0000","last_fullsized":"2023-10-19T14:21:27.517545+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:46:56.529404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.27","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.527448+0000","last_change":"2023-10-19T14:21:27.522301+0000","last_active":"2023-10-19T14:21:28.527448+0000","last_peered":"2023-10-19T14:21:28.527448+0000","last_clean":"2023-10-19T14:21:28.527448+0000","last_became_active":"2023-10-19T14:21:27.520013+0000","last_became_peered":"2023-10-19T14:21:27.520013+0000","last_unstale":"2023-10-19T14:21:28.527448+0000","last_undegraded":"2023-10-19T14:21:28.527448+0000","last_fullsized":"2023-10-19T14:21:28.527448+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:22:00.351535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.25","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.512641+0000","last_change":"2023-10-19T14:21:10.422856+0000","last_active":"2023-10-19T14:21:27.512641+0000","last_peered":"2023-10-19T14:21:27.512641+0000","last_clean":"2023-10-19T14:21:27.512641+0000","last_became_active":"2023-10-19T14:21:10.422047+0000","last_became_peered":"2023-10-19T14:21:10.422047+0000","last_unstale":"2023-10-19T14:21:27.512641+0000","last_undegraded":"2023-10-19T14:21:27.512641+0000","last_fullsized":"2023-10-19T14:21:27.512641+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:35:21.048326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.1a","version":"242'1","reported_seq":17,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.536148+0000","last_change":"2023-10-19T14:21:09.408272+0000","last_active":"2023-10-19T14:21:27.536148+0000","last_peered":"2023-10-19T14:21:27.536148+0000","last_clean":"2023-10-19T14:21:27.536148+0000","last_became_active":"2023-10-19T14:21:09.407997+0000","last_became_peered":"2023-10-19T14:21:09.407997+0000","last_unstale":"2023-10-19T14:21:27.536148+0000","last_undegraded":"2023-10-19T14:21:27.536148+0000","last_fullsized":"2023-10-19T14:21:27.536148+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:32:11.516962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.24","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.592671+0000","last_change":"2023-10-19T14:21:27.532654+0000","last_active":"2023-10-19T14:21:28.592671+0000","last_peered":"2023-10-19T14:21:28.592671+0000","last_clean":"2023-10-19T14:21:28.592671+0000","last_became_active":"2023-10-19T14:21:27.532435+0000","last_became_peered":"2023-10-19T14:21:27.532435+0000","last_unstale":"2023-10-19T14:21:28.592671+0000","last_undegraded":"2023-10-19T14:21:28.592671+0000","last_fullsized":"2023-10-19T14:21:28.592671+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:03:01.076409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.26","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.512001+0000","last_change":"2023-10-19T14:21:10.427832+0000","last_active":"2023-10-19T14:21:27.512001+0000","last_peered":"2023-10-19T14:21:27.512001+0000","last_clean":"2023-10-19T14:21:27.512001+0000","last_became_active":"2023-10-19T14:21:10.427540+0000","last_became_peered":"2023-10-19T14:21:10.427540+0000","last_unstale":"2023-10-19T14:21:27.512001+0000","last_undegraded":"2023-10-19T14:21:27.512001+0000","last_fullsized":"2023-10-19T14:21:27.512001+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:47:17.356392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":16,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.510726+0000","last_change":"2023-10-19T14:21:09.408331+0000","last_active":"2023-10-19T14:21:27.510726+0000","last_peered":"2023-10-19T14:21:27.510726+0000","last_clean":"2023-10-19T14:21:27.510726+0000","last_became_active":"2023-10-19T14:21:09.407722+0000","last_became_peered":"2023-10-19T14:21:09.407722+0000","last_unstale":"2023-10-19T14:21:27.510726+0000","last_undegraded":"2023-10-19T14:21:27.510726+0000","last_fullsized":"2023-10-19T14:21:27.510726+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:06:49.485559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.25","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.591086+0000","last_change":"2023-10-19T14:21:27.531503+0000","last_active":"2023-10-19T14:21:28.591086+0000","last_peered":"2023-10-19T14:21:28.591086+0000","last_clean":"2023-10-19T14:21:28.591086+0000","last_became_active":"2023-10-19T14:21:27.530519+0000","last_became_peered":"2023-10-19T14:21:27.530519+0000","last_unstale":"2023-10-19T14:21:28.591086+0000","last_undegraded":"2023-10-19T14:21:28.591086+0000","last_fullsized":"2023-10-19T14:21:28.591086+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:39:22.559682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.27","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.517103+0000","last_change":"2023-10-19T14:21:10.425541+0000","last_active":"2023-10-19T14:21:27.517103+0000","last_peered":"2023-10-19T14:21:27.517103+0000","last_clean":"2023-10-19T14:21:27.517103+0000","last_became_active":"2023-10-19T14:21:10.423860+0000","last_became_peered":"2023-10-19T14:21:10.423860+0000","last_unstale":"2023-10-19T14:21:27.517103+0000","last_undegraded":"2023-10-19T14:21:27.517103+0000","last_fullsized":"2023-10-19T14:21:27.517103+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:41:24.205533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":16,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.515914+0000","last_change":"2023-10-19T14:21:09.411121+0000","last_active":"2023-10-19T14:21:27.515914+0000","last_peered":"2023-10-19T14:21:27.515914+0000","last_clean":"2023-10-19T14:21:27.515914+0000","last_became_active":"2023-10-19T14:21:09.409416+0000","last_became_peered":"2023-10-19T14:21:09.409416+0000","last_unstale":"2023-10-19T14:21:27.515914+0000","last_undegraded":"2023-10-19T14:21:27.515914+0000","last_fullsized":"2023-10-19T14:21:27.515914+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:09:01.088634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.2a","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.592875+0000","last_change":"2023-10-19T14:21:27.542992+0000","last_active":"2023-10-19T14:21:28.592875+0000","last_peered":"2023-10-19T14:21:28.592875+0000","last_clean":"2023-10-19T14:21:28.592875+0000","last_became_active":"2023-10-19T14:21:27.542737+0000","last_became_peered":"2023-10-19T14:21:27.542737+0000","last_unstale":"2023-10-19T14:21:28.592875+0000","last_undegraded":"2023-10-19T14:21:28.592875+0000","last_fullsized":"2023-10-19T14:21:28.592875+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:02:52.850382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.28","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.512160+0000","last_change":"2023-10-19T14:21:10.422391+0000","last_active":"2023-10-19T14:21:27.512160+0000","last_peered":"2023-10-19T14:21:27.512160+0000","last_clean":"2023-10-19T14:21:27.512160+0000","last_became_active":"2023-10-19T14:21:10.422118+0000","last_became_peered":"2023-10-19T14:21:10.422118+0000","last_unstale":"2023-10-19T14:21:27.512160+0000","last_undegraded":"2023-10-19T14:21:27.512160+0000","last_fullsized":"2023-10-19T14:21:27.512160+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T02:12:47.154435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":16,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.537368+0000","last_change":"2023-10-19T14:21:09.412983+0000","last_active":"2023-10-19T14:21:27.537368+0000","last_peered":"2023-10-19T14:21:27.537368+0000","last_clean":"2023-10-19T14:21:27.537368+0000","last_became_active":"2023-10-19T14:21:09.412690+0000","last_became_peered":"2023-10-19T14:21:09.412690+0000","last_unstale":"2023-10-19T14:21:27.537368+0000","last_undegraded":"2023-10-19T14:21:27.537368+0000","last_fullsized":"2023-10-19T14:21:27.537368+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:21:42.383177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.2b","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.589985+0000","last_change":"2023-10-19T14:21:27.527226+0000","last_active":"2023-10-19T14:21:28.589985+0000","last_peered":"2023-10-19T14:21:28.589985+0000","last_clean":"2023-10-19T14:21:28.589985+0000","last_became_active":"2023-10-19T14:21:27.526880+0000","last_became_peered":"2023-10-19T14:21:27.526880+0000","last_unstale":"2023-10-19T14:21:28.589985+0000","last_undegraded":"2023-10-19T14:21:28.589985+0000","last_fullsized":"2023-10-19T14:21:28.589985+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:06:38.822458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.29","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.516657+0000","last_change":"2023-10-19T14:21:10.427650+0000","last_active":"2023-10-19T14:21:27.516657+0000","last_peered":"2023-10-19T14:21:27.516657+0000","last_clean":"2023-10-19T14:21:27.516657+0000","last_became_active":"2023-10-19T14:21:10.427367+0000","last_became_peered":"2023-10-19T14:21:10.427367+0000","last_unstale":"2023-10-19T14:21:27.516657+0000","last_undegraded":"2023-10-19T14:21:27.516657+0000","last_fullsized":"2023-10-19T14:21:27.516657+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:54:17.097512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":16,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.512311+0000","last_change":"2023-10-19T14:21:09.408912+0000","last_active":"2023-10-19T14:21:27.512311+0000","last_peered":"2023-10-19T14:21:27.512311+0000","last_clean":"2023-10-19T14:21:27.512311+0000","last_became_active":"2023-10-19T14:21:09.408661+0000","last_became_peered":"2023-10-19T14:21:09.408661+0000","last_unstale":"2023-10-19T14:21:27.512311+0000","last_undegraded":"2023-10-19T14:21:27.512311+0000","last_fullsized":"2023-10-19T14:21:27.512311+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:55:16.857948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.28","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.590553+0000","last_change":"2023-10-19T14:21:27.529521+0000","last_active":"2023-10-19T14:21:28.590553+0000","last_peered":"2023-10-19T14:21:28.590553+0000","last_clean":"2023-10-19T14:21:28.590553+0000","last_became_active":"2023-10-19T14:21:27.529236+0000","last_became_peered":"2023-10-19T14:21:27.529236+0000","last_unstale":"2023-10-19T14:21:28.590553+0000","last_undegraded":"2023-10-19T14:21:28.590553+0000","last_fullsized":"2023-10-19T14:21:28.590553+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:48:54.397084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.2a","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.511256+0000","last_change":"2023-10-19T14:21:10.430153+0000","last_active":"2023-10-19T14:21:27.511256+0000","last_peered":"2023-10-19T14:21:27.511256+0000","last_clean":"2023-10-19T14:21:27.511256+0000","last_became_active":"2023-10-19T14:21:10.429868+0000","last_became_peered":"2023-10-19T14:21:10.429868+0000","last_unstale":"2023-10-19T14:21:27.511256+0000","last_undegraded":"2023-10-19T14:21:27.511256+0000","last_fullsized":"2023-10-19T14:21:27.511256+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:55:49.837750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.15","version":"242'3","reported_seq":19,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.517746+0000","last_change":"2023-10-19T14:21:09.414620+0000","last_active":"2023-10-19T14:21:27.517746+0000","last_peered":"2023-10-19T14:21:27.517746+0000","last_clean":"2023-10-19T14:21:27.517746+0000","last_became_active":"2023-10-19T14:21:09.414378+0000","last_became_peered":"2023-10-19T14:21:09.414378+0000","last_unstale":"2023-10-19T14:21:27.517746+0000","last_undegraded":"2023-10-19T14:21:27.517746+0000","last_fullsized":"2023-10-19T14:21:27.517746+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:16:10.702399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.29","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.524309+0000","last_change":"2023-10-19T14:21:27.536121+0000","last_active":"2023-10-19T14:21:28.524309+0000","last_peered":"2023-10-19T14:21:28.524309+0000","last_clean":"2023-10-19T14:21:28.524309+0000","last_became_active":"2023-10-19T14:21:27.535825+0000","last_became_peered":"2023-10-19T14:21:27.535825+0000","last_unstale":"2023-10-19T14:21:28.524309+0000","last_undegraded":"2023-10-19T14:21:28.524309+0000","last_fullsized":"2023-10-19T14:21:28.524309+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:51:08.953710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.2b","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.512405+0000","last_change":"2023-10-19T14:21:10.436778+0000","last_active":"2023-10-19T14:21:27.512405+0000","last_peered":"2023-10-19T14:21:27.512405+0000","last_clean":"2023-10-19T14:21:27.512405+0000","last_became_active":"2023-10-19T14:21:10.436506+0000","last_became_peered":"2023-10-19T14:21:10.436506+0000","last_unstale":"2023-10-19T14:21:27.512405+0000","last_undegraded":"2023-10-19T14:21:27.512405+0000","last_fullsized":"2023-10-19T14:21:27.512405+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:02:47.768820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.14","version":"244'9","reported_seq":25,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.515817+0000","last_change":"2023-10-19T14:21:09.411345+0000","last_active":"2023-10-19T14:21:27.515817+0000","last_peered":"2023-10-19T14:21:27.515817+0000","last_clean":"2023-10-19T14:21:27.515817+0000","last_became_active":"2023-10-19T14:21:09.410711+0000","last_became_peered":"2023-10-19T14:21:09.410711+0000","last_unstale":"2023-10-19T14:21:27.515817+0000","last_undegraded":"2023-10-19T14:21:27.515817+0000","last_fullsized":"2023-10-19T14:21:27.515817+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:35:21.400082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2868,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.2e","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.591360+0000","last_change":"2023-10-19T14:21:27.529607+0000","last_active":"2023-10-19T14:21:28.591360+0000","last_peered":"2023-10-19T14:21:28.591360+0000","last_clean":"2023-10-19T14:21:28.591360+0000","last_became_active":"2023-10-19T14:21:27.529260+0000","last_became_peered":"2023-10-19T14:21:27.529260+0000","last_unstale":"2023-10-19T14:21:28.591360+0000","last_undegraded":"2023-10-19T14:21:28.591360+0000","last_fullsized":"2023-10-19T14:21:28.591360+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:07:06.629090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.2c","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.521816+0000","last_change":"2023-10-19T14:21:10.435528+0000","last_active":"2023-10-19T14:21:27.521816+0000","last_peered":"2023-10-19T14:21:27.521816+0000","last_clean":"2023-10-19T14:21:27.521816+0000","last_became_active":"2023-10-19T14:21:10.435235+0000","last_became_peered":"2023-10-19T14:21:10.435235+0000","last_unstale":"2023-10-19T14:21:27.521816+0000","last_undegraded":"2023-10-19T14:21:27.521816+0000","last_fullsized":"2023-10-19T14:21:27.521816+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:24:42.687537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.13","version":"242'2","reported_seq":18,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.518287+0000","last_change":"2023-10-19T14:21:09.413386+0000","last_active":"2023-10-19T14:21:27.518287+0000","last_peered":"2023-10-19T14:21:27.518287+0000","last_clean":"2023-10-19T14:21:27.518287+0000","last_became_active":"2023-10-19T14:21:09.413119+0000","last_became_peered":"2023-10-19T14:21:09.413119+0000","last_unstale":"2023-10-19T14:21:27.518287+0000","last_undegraded":"2023-10-19T14:21:27.518287+0000","last_fullsized":"2023-10-19T14:21:27.518287+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:32:53.438251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.2f","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.592793+0000","last_change":"2023-10-19T14:21:27.531259+0000","last_active":"2023-10-19T14:21:28.592793+0000","last_peered":"2023-10-19T14:21:28.592793+0000","last_clean":"2023-10-19T14:21:28.592793+0000","last_became_active":"2023-10-19T14:21:27.530949+0000","last_became_peered":"2023-10-19T14:21:27.530949+0000","last_unstale":"2023-10-19T14:21:28.592793+0000","last_undegraded":"2023-10-19T14:21:28.592793+0000","last_fullsized":"2023-10-19T14:21:28.592793+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:11:42.825804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2d","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.512418+0000","last_change":"2023-10-19T14:21:10.417244+0000","last_active":"2023-10-19T14:21:27.512418+0000","last_peered":"2023-10-19T14:21:27.512418+0000","last_clean":"2023-10-19T14:21:27.512418+0000","last_became_active":"2023-10-19T14:21:10.416948+0000","last_became_peered":"2023-10-19T14:21:10.416948+0000","last_unstale":"2023-10-19T14:21:27.512418+0000","last_undegraded":"2023-10-19T14:21:27.512418+0000","last_fullsized":"2023-10-19T14:21:27.512418+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:13:29.800561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":16,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.516499+0000","last_change":"2023-10-19T14:21:09.417189+0000","last_active":"2023-10-19T14:21:27.516499+0000","last_peered":"2023-10-19T14:21:27.516499+0000","last_clean":"2023-10-19T14:21:27.516499+0000","last_became_active":"2023-10-19T14:21:09.416920+0000","last_became_peered":"2023-10-19T14:21:09.416920+0000","last_unstale":"2023-10-19T14:21:27.516499+0000","last_undegraded":"2023-10-19T14:21:27.516499+0000","last_fullsized":"2023-10-19T14:21:27.516499+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:17:36.285741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.2c","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.591881+0000","last_change":"2023-10-19T14:21:27.526720+0000","last_active":"2023-10-19T14:21:28.591881+0000","last_peered":"2023-10-19T14:21:28.591881+0000","last_clean":"2023-10-19T14:21:28.591881+0000","last_became_active":"2023-10-19T14:21:27.526430+0000","last_became_peered":"2023-10-19T14:21:27.526430+0000","last_unstale":"2023-10-19T14:21:28.591881+0000","last_undegraded":"2023-10-19T14:21:28.591881+0000","last_fullsized":"2023-10-19T14:21:28.591881+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:45:48.739493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.2e","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.518681+0000","last_change":"2023-10-19T14:21:10.422922+0000","last_active":"2023-10-19T14:21:27.518681+0000","last_peered":"2023-10-19T14:21:27.518681+0000","last_clean":"2023-10-19T14:21:27.518681+0000","last_became_active":"2023-10-19T14:21:10.422656+0000","last_became_peered":"2023-10-19T14:21:10.422656+0000","last_unstale":"2023-10-19T14:21:27.518681+0000","last_undegraded":"2023-10-19T14:21:27.518681+0000","last_fullsized":"2023-10-19T14:21:27.518681+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:16:55.773348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":16,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.517654+0000","last_change":"2023-10-19T14:21:09.413176+0000","last_active":"2023-10-19T14:21:27.517654+0000","last_peered":"2023-10-19T14:21:27.517654+0000","last_clean":"2023-10-19T14:21:27.517654+0000","last_became_active":"2023-10-19T14:21:09.412906+0000","last_became_peered":"2023-10-19T14:21:09.412906+0000","last_unstale":"2023-10-19T14:21:27.517654+0000","last_undegraded":"2023-10-19T14:21:27.517654+0000","last_fullsized":"2023-10-19T14:21:27.517654+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:07:02.508240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.2d","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.592222+0000","last_change":"2023-10-19T14:21:27.534816+0000","last_active":"2023-10-19T14:21:28.592222+0000","last_peered":"2023-10-19T14:21:28.592222+0000","last_clean":"2023-10-19T14:21:28.592222+0000","last_became_active":"2023-10-19T14:21:27.534490+0000","last_became_peered":"2023-10-19T14:21:27.534490+0000","last_unstale":"2023-10-19T14:21:28.592222+0000","last_undegraded":"2023-10-19T14:21:28.592222+0000","last_fullsized":"2023-10-19T14:21:28.592222+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:47:57.932633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2f","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.517011+0000","last_change":"2023-10-19T14:21:10.431716+0000","last_active":"2023-10-19T14:21:27.517011+0000","last_peered":"2023-10-19T14:21:27.517011+0000","last_clean":"2023-10-19T14:21:27.517011+0000","last_became_active":"2023-10-19T14:21:10.430784+0000","last_became_peered":"2023-10-19T14:21:10.430784+0000","last_unstale":"2023-10-19T14:21:27.517011+0000","last_undegraded":"2023-10-19T14:21:27.517011+0000","last_fullsized":"2023-10-19T14:21:27.517011+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:14:44.806142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.10","version":"242'8","reported_seq":24,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.522719+0000","last_change":"2023-10-19T14:21:09.415001+0000","last_active":"2023-10-19T14:21:27.522719+0000","last_peered":"2023-10-19T14:21:27.522719+0000","last_clean":"2023-10-19T14:21:27.522719+0000","last_became_active":"2023-10-19T14:21:09.414727+0000","last_became_peered":"2023-10-19T14:21:09.414727+0000","last_unstale":"2023-10-19T14:21:27.522719+0000","last_undegraded":"2023-10-19T14:21:27.522719+0000","last_fullsized":"2023-10-19T14:21:27.522719+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:54:37.202858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.32","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.524440+0000","last_change":"2023-10-19T14:21:27.541645+0000","last_active":"2023-10-19T14:21:28.524440+0000","last_peered":"2023-10-19T14:21:28.524440+0000","last_clean":"2023-10-19T14:21:28.524440+0000","last_became_active":"2023-10-19T14:21:27.541380+0000","last_became_peered":"2023-10-19T14:21:27.541380+0000","last_unstale":"2023-10-19T14:21:28.524440+0000","last_undegraded":"2023-10-19T14:21:28.524440+0000","last_fullsized":"2023-10-19T14:21:28.524440+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:24:40.034377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.30","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.518001+0000","last_change":"2023-10-19T14:21:10.437415+0000","last_active":"2023-10-19T14:21:27.518001+0000","last_peered":"2023-10-19T14:21:27.518001+0000","last_clean":"2023-10-19T14:21:27.518001+0000","last_became_active":"2023-10-19T14:21:10.437137+0000","last_became_peered":"2023-10-19T14:21:10.437137+0000","last_unstale":"2023-10-19T14:21:27.518001+0000","last_undegraded":"2023-10-19T14:21:27.518001+0000","last_fullsized":"2023-10-19T14:21:27.518001+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:39:56.303696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.f","version":"242'1","reported_seq":17,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.515277+0000","last_change":"2023-10-19T14:21:09.417455+0000","last_active":"2023-10-19T14:21:27.515277+0000","last_peered":"2023-10-19T14:21:27.515277+0000","last_clean":"2023-10-19T14:21:27.515277+0000","last_became_active":"2023-10-19T14:21:09.417228+0000","last_became_peered":"2023-10-19T14:21:09.417228+0000","last_unstale":"2023-10-19T14:21:27.515277+0000","last_undegraded":"2023-10-19T14:21:27.515277+0000","last_fullsized":"2023-10-19T14:21:27.515277+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:37:45.851912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.33","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.526774+0000","last_change":"2023-10-19T14:21:27.540879+0000","last_active":"2023-10-19T14:21:28.526774+0000","last_peered":"2023-10-19T14:21:28.526774+0000","last_clean":"2023-10-19T14:21:28.526774+0000","last_became_active":"2023-10-19T14:21:27.539190+0000","last_became_peered":"2023-10-19T14:21:27.539190+0000","last_unstale":"2023-10-19T14:21:28.526774+0000","last_undegraded":"2023-10-19T14:21:28.526774+0000","last_fullsized":"2023-10-19T14:21:28.526774+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:36:31.003136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.31","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.517323+0000","last_change":"2023-10-19T14:21:10.423316+0000","last_active":"2023-10-19T14:21:27.517323+0000","last_peered":"2023-10-19T14:21:27.517323+0000","last_clean":"2023-10-19T14:21:27.517323+0000","last_became_active":"2023-10-19T14:21:10.423028+0000","last_became_peered":"2023-10-19T14:21:10.423028+0000","last_unstale":"2023-10-19T14:21:27.517323+0000","last_undegraded":"2023-10-19T14:21:27.517323+0000","last_fullsized":"2023-10-19T14:21:27.517323+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:39:05.601924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.e","version":"242'2","reported_seq":18,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.523435+0000","last_change":"2023-10-19T14:21:09.409551+0000","last_active":"2023-10-19T14:21:27.523435+0000","last_peered":"2023-10-19T14:21:27.523435+0000","last_clean":"2023-10-19T14:21:27.523435+0000","last_became_active":"2023-10-19T14:21:09.409238+0000","last_became_peered":"2023-10-19T14:21:09.409238+0000","last_unstale":"2023-10-19T14:21:27.523435+0000","last_undegraded":"2023-10-19T14:21:27.523435+0000","last_fullsized":"2023-10-19T14:21:27.523435+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:38:52.516509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.30","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.591552+0000","last_change":"2023-10-19T14:21:27.535573+0000","last_active":"2023-10-19T14:21:28.591552+0000","last_peered":"2023-10-19T14:21:28.591552+0000","last_clean":"2023-10-19T14:21:28.591552+0000","last_became_active":"2023-10-19T14:21:27.534545+0000","last_became_peered":"2023-10-19T14:21:27.534545+0000","last_unstale":"2023-10-19T14:21:28.591552+0000","last_undegraded":"2023-10-19T14:21:28.591552+0000","last_fullsized":"2023-10-19T14:21:28.591552+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:03:49.950096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.32","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.522504+0000","last_change":"2023-10-19T14:21:10.433187+0000","last_active":"2023-10-19T14:21:27.522504+0000","last_peered":"2023-10-19T14:21:27.522504+0000","last_clean":"2023-10-19T14:21:27.522504+0000","last_became_active":"2023-10-19T14:21:10.432876+0000","last_became_peered":"2023-10-19T14:21:10.432876+0000","last_unstale":"2023-10-19T14:21:27.522504+0000","last_undegraded":"2023-10-19T14:21:27.522504+0000","last_fullsized":"2023-10-19T14:21:27.522504+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:49:35.448257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.d","version":"242'1","reported_seq":17,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.522407+0000","last_change":"2023-10-19T14:21:09.413076+0000","last_active":"2023-10-19T14:21:27.522407+0000","last_peered":"2023-10-19T14:21:27.522407+0000","last_clean":"2023-10-19T14:21:27.522407+0000","last_became_active":"2023-10-19T14:21:09.412330+0000","last_became_peered":"2023-10-19T14:21:09.412330+0000","last_unstale":"2023-10-19T14:21:27.522407+0000","last_undegraded":"2023-10-19T14:21:27.522407+0000","last_fullsized":"2023-10-19T14:21:27.522407+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:20:50.841124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.31","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.524325+0000","last_change":"2023-10-19T14:21:27.534582+0000","last_active":"2023-10-19T14:21:28.524325+0000","last_peered":"2023-10-19T14:21:28.524325+0000","last_clean":"2023-10-19T14:21:28.524325+0000","last_became_active":"2023-10-19T14:21:27.534261+0000","last_became_peered":"2023-10-19T14:21:27.534261+0000","last_unstale":"2023-10-19T14:21:28.524325+0000","last_undegraded":"2023-10-19T14:21:28.524325+0000","last_fullsized":"2023-10-19T14:21:28.524325+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:41:18.907883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.33","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.512259+0000","last_change":"2023-10-19T14:21:10.429531+0000","last_active":"2023-10-19T14:21:27.512259+0000","last_peered":"2023-10-19T14:21:27.512259+0000","last_clean":"2023-10-19T14:21:27.512259+0000","last_became_active":"2023-10-19T14:21:10.429261+0000","last_became_peered":"2023-10-19T14:21:10.429261+0000","last_unstale":"2023-10-19T14:21:27.512259+0000","last_undegraded":"2023-10-19T14:21:27.512259+0000","last_fullsized":"2023-10-19T14:21:27.512259+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:29:47.656632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":16,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.523428+0000","last_change":"2023-10-19T14:21:09.409030+0000","last_active":"2023-10-19T14:21:27.523428+0000","last_peered":"2023-10-19T14:21:27.523428+0000","last_clean":"2023-10-19T14:21:27.523428+0000","last_became_active":"2023-10-19T14:21:09.408717+0000","last_became_peered":"2023-10-19T14:21:09.408717+0000","last_unstale":"2023-10-19T14:21:27.523428+0000","last_undegraded":"2023-10-19T14:21:27.523428+0000","last_fullsized":"2023-10-19T14:21:27.523428+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:04:32.926013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.36","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.592580+0000","last_change":"2023-10-19T14:21:27.521181+0000","last_active":"2023-10-19T14:21:28.592580+0000","last_peered":"2023-10-19T14:21:28.592580+0000","last_clean":"2023-10-19T14:21:28.592580+0000","last_became_active":"2023-10-19T14:21:27.520913+0000","last_became_peered":"2023-10-19T14:21:27.520913+0000","last_unstale":"2023-10-19T14:21:28.592580+0000","last_undegraded":"2023-10-19T14:21:28.592580+0000","last_fullsized":"2023-10-19T14:21:28.592580+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:21:46.163073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.34","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.511624+0000","last_change":"2023-10-19T14:21:10.437932+0000","last_active":"2023-10-19T14:21:27.511624+0000","last_peered":"2023-10-19T14:21:27.511624+0000","last_clean":"2023-10-19T14:21:27.511624+0000","last_became_active":"2023-10-19T14:21:10.437668+0000","last_became_peered":"2023-10-19T14:21:10.437668+0000","last_unstale":"2023-10-19T14:21:27.511624+0000","last_undegraded":"2023-10-19T14:21:27.511624+0000","last_fullsized":"2023-10-19T14:21:27.511624+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:04:33.936613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.b","version":"242'2","reported_seq":18,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.512364+0000","last_change":"2023-10-19T14:21:09.415147+0000","last_active":"2023-10-19T14:21:27.512364+0000","last_peered":"2023-10-19T14:21:27.512364+0000","last_clean":"2023-10-19T14:21:27.512364+0000","last_became_active":"2023-10-19T14:21:09.414852+0000","last_became_peered":"2023-10-19T14:21:09.414852+0000","last_unstale":"2023-10-19T14:21:27.512364+0000","last_undegraded":"2023-10-19T14:21:27.512364+0000","last_fullsized":"2023-10-19T14:21:27.512364+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:53:33.599617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.37","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.591838+0000","last_change":"2023-10-19T14:21:27.530671+0000","last_active":"2023-10-19T14:21:28.591838+0000","last_peered":"2023-10-19T14:21:28.591838+0000","last_clean":"2023-10-19T14:21:28.591838+0000","last_became_active":"2023-10-19T14:21:27.530249+0000","last_became_peered":"2023-10-19T14:21:27.530249+0000","last_unstale":"2023-10-19T14:21:28.591838+0000","last_undegraded":"2023-10-19T14:21:28.591838+0000","last_fullsized":"2023-10-19T14:21:28.591838+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:33:01.768540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.35","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.521989+0000","last_change":"2023-10-19T14:21:10.439769+0000","last_active":"2023-10-19T14:21:27.521989+0000","last_peered":"2023-10-19T14:21:27.521989+0000","last_clean":"2023-10-19T14:21:27.521989+0000","last_became_active":"2023-10-19T14:21:10.439056+0000","last_became_peered":"2023-10-19T14:21:10.439056+0000","last_unstale":"2023-10-19T14:21:27.521989+0000","last_undegraded":"2023-10-19T14:21:27.521989+0000","last_fullsized":"2023-10-19T14:21:27.521989+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:18:35.286014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":16,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.521980+0000","last_change":"2023-10-19T14:21:09.411613+0000","last_active":"2023-10-19T14:21:27.521980+0000","last_peered":"2023-10-19T14:21:27.521980+0000","last_clean":"2023-10-19T14:21:27.521980+0000","last_became_active":"2023-10-19T14:21:09.411365+0000","last_became_peered":"2023-10-19T14:21:09.411365+0000","last_unstale":"2023-10-19T14:21:27.521980+0000","last_undegraded":"2023-10-19T14:21:27.521980+0000","last_fullsized":"2023-10-19T14:21:27.521980+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:22:34.404197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.34","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.591670+0000","last_change":"2023-10-19T14:21:27.524569+0000","last_active":"2023-10-19T14:21:28.591670+0000","last_peered":"2023-10-19T14:21:28.591670+0000","last_clean":"2023-10-19T14:21:28.591670+0000","last_became_active":"2023-10-19T14:21:27.524247+0000","last_became_peered":"2023-10-19T14:21:27.524247+0000","last_unstale":"2023-10-19T14:21:28.591670+0000","last_undegraded":"2023-10-19T14:21:28.591670+0000","last_fullsized":"2023-10-19T14:21:28.591670+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:19:40.772028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.36","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.524514+0000","last_change":"2023-10-19T14:21:10.424290+0000","last_active":"2023-10-19T14:21:27.524514+0000","last_peered":"2023-10-19T14:21:27.524514+0000","last_clean":"2023-10-19T14:21:27.524514+0000","last_became_active":"2023-10-19T14:21:10.423951+0000","last_became_peered":"2023-10-19T14:21:10.423951+0000","last_unstale":"2023-10-19T14:21:27.524514+0000","last_undegraded":"2023-10-19T14:21:27.524514+0000","last_fullsized":"2023-10-19T14:21:27.524514+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:43:48.977855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":16,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.516770+0000","last_change":"2023-10-19T14:21:09.410869+0000","last_active":"2023-10-19T14:21:27.516770+0000","last_peered":"2023-10-19T14:21:27.516770+0000","last_clean":"2023-10-19T14:21:27.516770+0000","last_became_active":"2023-10-19T14:21:09.410586+0000","last_became_peered":"2023-10-19T14:21:09.410586+0000","last_unstale":"2023-10-19T14:21:27.516770+0000","last_undegraded":"2023-10-19T14:21:27.516770+0000","last_fullsized":"2023-10-19T14:21:27.516770+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:02:42.839507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.35","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.590903+0000","last_change":"2023-10-19T14:21:27.528697+0000","last_active":"2023-10-19T14:21:28.590903+0000","last_peered":"2023-10-19T14:21:28.590903+0000","last_clean":"2023-10-19T14:21:28.590903+0000","last_became_active":"2023-10-19T14:21:27.528441+0000","last_became_peered":"2023-10-19T14:21:27.528441+0000","last_unstale":"2023-10-19T14:21:28.590903+0000","last_undegraded":"2023-10-19T14:21:28.590903+0000","last_fullsized":"2023-10-19T14:21:28.590903+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:30:18.406194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.37","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.525586+0000","last_change":"2023-10-19T14:21:10.422504+0000","last_active":"2023-10-19T14:21:27.525586+0000","last_peered":"2023-10-19T14:21:27.525586+0000","last_clean":"2023-10-19T14:21:27.525586+0000","last_became_active":"2023-10-19T14:21:10.422196+0000","last_became_peered":"2023-10-19T14:21:10.422196+0000","last_unstale":"2023-10-19T14:21:27.525586+0000","last_undegraded":"2023-10-19T14:21:27.525586+0000","last_fullsized":"2023-10-19T14:21:27.525586+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:53:46.360770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":16,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.526319+0000","last_change":"2023-10-19T14:21:09.415870+0000","last_active":"2023-10-19T14:21:27.526319+0000","last_peered":"2023-10-19T14:21:27.526319+0000","last_clean":"2023-10-19T14:21:27.526319+0000","last_became_active":"2023-10-19T14:21:09.415609+0000","last_became_peered":"2023-10-19T14:21:09.415609+0000","last_unstale":"2023-10-19T14:21:27.526319+0000","last_undegraded":"2023-10-19T14:21:27.526319+0000","last_fullsized":"2023-10-19T14:21:27.526319+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:29:10.891884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.3a","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.591778+0000","last_change":"2023-10-19T14:21:27.521038+0000","last_active":"2023-10-19T14:21:28.591778+0000","last_peered":"2023-10-19T14:21:28.591778+0000","last_clean":"2023-10-19T14:21:28.591778+0000","last_became_active":"2023-10-19T14:21:27.519463+0000","last_became_peered":"2023-10-19T14:21:27.519463+0000","last_unstale":"2023-10-19T14:21:28.591778+0000","last_undegraded":"2023-10-19T14:21:28.591778+0000","last_fullsized":"2023-10-19T14:21:28.591778+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:27:02.150067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.38","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.512047+0000","last_change":"2023-10-19T14:21:10.428770+0000","last_active":"2023-10-19T14:21:27.512047+0000","last_peered":"2023-10-19T14:21:27.512047+0000","last_clean":"2023-10-19T14:21:27.512047+0000","last_became_active":"2023-10-19T14:21:10.428494+0000","last_became_peered":"2023-10-19T14:21:10.428494+0000","last_unstale":"2023-10-19T14:21:27.512047+0000","last_undegraded":"2023-10-19T14:21:27.512047+0000","last_fullsized":"2023-10-19T14:21:27.512047+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:54:47.763268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.7","version":"242'2","reported_seq":18,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.511934+0000","last_change":"2023-10-19T14:21:09.416206+0000","last_active":"2023-10-19T14:21:27.511934+0000","last_peered":"2023-10-19T14:21:27.511934+0000","last_clean":"2023-10-19T14:21:27.511934+0000","last_became_active":"2023-10-19T14:21:09.415929+0000","last_became_peered":"2023-10-19T14:21:09.415929+0000","last_unstale":"2023-10-19T14:21:27.511934+0000","last_undegraded":"2023-10-19T14:21:27.511934+0000","last_fullsized":"2023-10-19T14:21:27.511934+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:41:27.899602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.3b","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.590724+0000","last_change":"2023-10-19T14:21:27.521444+0000","last_active":"2023-10-19T14:21:28.590724+0000","last_peered":"2023-10-19T14:21:28.590724+0000","last_clean":"2023-10-19T14:21:28.590724+0000","last_became_active":"2023-10-19T14:21:27.521158+0000","last_became_peered":"2023-10-19T14:21:27.521158+0000","last_unstale":"2023-10-19T14:21:28.590724+0000","last_undegraded":"2023-10-19T14:21:28.590724+0000","last_fullsized":"2023-10-19T14:21:28.590724+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:15:17.090163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.39","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.516124+0000","last_change":"2023-10-19T14:21:10.431238+0000","last_active":"2023-10-19T14:21:27.516124+0000","last_peered":"2023-10-19T14:21:27.516124+0000","last_clean":"2023-10-19T14:21:27.516124+0000","last_became_active":"2023-10-19T14:21:10.430970+0000","last_became_peered":"2023-10-19T14:21:10.430970+0000","last_unstale":"2023-10-19T14:21:27.516124+0000","last_undegraded":"2023-10-19T14:21:27.516124+0000","last_fullsized":"2023-10-19T14:21:27.516124+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:29:11.902484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.6","version":"243'1","reported_seq":17,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.517630+0000","last_change":"2023-10-19T14:21:09.415597+0000","last_active":"2023-10-19T14:21:27.517630+0000","last_peered":"2023-10-19T14:21:27.517630+0000","last_clean":"2023-10-19T14:21:27.517630+0000","last_became_active":"2023-10-19T14:21:09.415325+0000","last_became_peered":"2023-10-19T14:21:09.415325+0000","last_unstale":"2023-10-19T14:21:27.517630+0000","last_undegraded":"2023-10-19T14:21:27.517630+0000","last_fullsized":"2023-10-19T14:21:27.517630+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:47:10.402857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.38","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.526470+0000","last_change":"2023-10-19T14:21:27.537623+0000","last_active":"2023-10-19T14:21:28.526470+0000","last_peered":"2023-10-19T14:21:28.526470+0000","last_clean":"2023-10-19T14:21:28.526470+0000","last_became_active":"2023-10-19T14:21:27.537350+0000","last_became_peered":"2023-10-19T14:21:27.537350+0000","last_unstale":"2023-10-19T14:21:28.526470+0000","last_undegraded":"2023-10-19T14:21:28.526470+0000","last_fullsized":"2023-10-19T14:21:28.526470+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:12:47.431775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.3a","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.511130+0000","last_change":"2023-10-19T14:21:10.430013+0000","last_active":"2023-10-19T14:21:27.511130+0000","last_peered":"2023-10-19T14:21:27.511130+0000","last_clean":"2023-10-19T14:21:27.511130+0000","last_became_active":"2023-10-19T14:21:10.429744+0000","last_became_peered":"2023-10-19T14:21:10.429744+0000","last_unstale":"2023-10-19T14:21:27.511130+0000","last_undegraded":"2023-10-19T14:21:27.511130+0000","last_fullsized":"2023-10-19T14:21:27.511130+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:23:00.738206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":16,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.536369+0000","last_change":"2023-10-19T14:21:09.412015+0000","last_active":"2023-10-19T14:21:27.536369+0000","last_peered":"2023-10-19T14:21:27.536369+0000","last_clean":"2023-10-19T14:21:27.536369+0000","last_became_active":"2023-10-19T14:21:09.411729+0000","last_became_peered":"2023-10-19T14:21:09.411729+0000","last_unstale":"2023-10-19T14:21:27.536369+0000","last_undegraded":"2023-10-19T14:21:27.536369+0000","last_fullsized":"2023-10-19T14:21:27.536369+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:50:33.137069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.39","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.526763+0000","last_change":"2023-10-19T14:21:27.526140+0000","last_active":"2023-10-19T14:21:28.526763+0000","last_peered":"2023-10-19T14:21:28.526763+0000","last_clean":"2023-10-19T14:21:28.526763+0000","last_became_active":"2023-10-19T14:21:27.525851+0000","last_became_peered":"2023-10-19T14:21:27.525851+0000","last_unstale":"2023-10-19T14:21:28.526763+0000","last_undegraded":"2023-10-19T14:21:28.526763+0000","last_fullsized":"2023-10-19T14:21:28.526763+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:58:25.324263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.3b","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.519383+0000","last_change":"2023-10-19T14:21:10.430397+0000","last_active":"2023-10-19T14:21:27.519383+0000","last_peered":"2023-10-19T14:21:27.519383+0000","last_clean":"2023-10-19T14:21:27.519383+0000","last_became_active":"2023-10-19T14:21:10.430094+0000","last_became_peered":"2023-10-19T14:21:10.430094+0000","last_unstale":"2023-10-19T14:21:27.519383+0000","last_undegraded":"2023-10-19T14:21:27.519383+0000","last_fullsized":"2023-10-19T14:21:27.519383+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:02:35.892624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":16,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.535666+0000","last_change":"2023-10-19T14:21:09.414910+0000","last_active":"2023-10-19T14:21:27.535666+0000","last_peered":"2023-10-19T14:21:27.535666+0000","last_clean":"2023-10-19T14:21:27.535666+0000","last_became_active":"2023-10-19T14:21:09.414646+0000","last_became_peered":"2023-10-19T14:21:09.414646+0000","last_unstale":"2023-10-19T14:21:27.535666+0000","last_undegraded":"2023-10-19T14:21:27.535666+0000","last_fullsized":"2023-10-19T14:21:27.535666+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:49:25.343751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.3e","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.525220+0000","last_change":"2023-10-19T14:21:27.547075+0000","last_active":"2023-10-19T14:21:28.525220+0000","last_peered":"2023-10-19T14:21:28.525220+0000","last_clean":"2023-10-19T14:21:28.525220+0000","last_became_active":"2023-10-19T14:21:27.546658+0000","last_became_peered":"2023-10-19T14:21:27.546658+0000","last_unstale":"2023-10-19T14:21:28.525220+0000","last_undegraded":"2023-10-19T14:21:28.525220+0000","last_fullsized":"2023-10-19T14:21:28.525220+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:15:49.232430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.3c","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.535394+0000","last_change":"2023-10-19T14:21:10.418399+0000","last_active":"2023-10-19T14:21:27.535394+0000","last_peered":"2023-10-19T14:21:27.535394+0000","last_clean":"2023-10-19T14:21:27.535394+0000","last_became_active":"2023-10-19T14:21:10.416779+0000","last_became_peered":"2023-10-19T14:21:10.416779+0000","last_unstale":"2023-10-19T14:21:27.535394+0000","last_undegraded":"2023-10-19T14:21:27.535394+0000","last_fullsized":"2023-10-19T14:21:27.535394+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:50:13.187179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":16,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.521467+0000","last_change":"2023-10-19T14:21:09.412344+0000","last_active":"2023-10-19T14:21:27.521467+0000","last_peered":"2023-10-19T14:21:27.521467+0000","last_clean":"2023-10-19T14:21:27.521467+0000","last_became_active":"2023-10-19T14:21:09.412101+0000","last_became_peered":"2023-10-19T14:21:09.412101+0000","last_unstale":"2023-10-19T14:21:27.521467+0000","last_undegraded":"2023-10-19T14:21:27.521467+0000","last_fullsized":"2023-10-19T14:21:27.521467+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:51:23.060110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.3f","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.593033+0000","last_change":"2023-10-19T14:21:27.534871+0000","last_active":"2023-10-19T14:21:28.593033+0000","last_peered":"2023-10-19T14:21:28.593033+0000","last_clean":"2023-10-19T14:21:28.593033+0000","last_became_active":"2023-10-19T14:21:27.533520+0000","last_became_peered":"2023-10-19T14:21:27.533520+0000","last_unstale":"2023-10-19T14:21:28.593033+0000","last_undegraded":"2023-10-19T14:21:28.593033+0000","last_fullsized":"2023-10-19T14:21:28.593033+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T02:17:48.211698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.3d","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.514961+0000","last_change":"2023-10-19T14:21:10.435707+0000","last_active":"2023-10-19T14:21:27.514961+0000","last_peered":"2023-10-19T14:21:27.514961+0000","last_clean":"2023-10-19T14:21:27.514961+0000","last_became_active":"2023-10-19T14:21:10.435426+0000","last_became_peered":"2023-10-19T14:21:10.435426+0000","last_unstale":"2023-10-19T14:21:27.514961+0000","last_undegraded":"2023-10-19T14:21:27.514961+0000","last_fullsized":"2023-10-19T14:21:27.514961+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:21:50.005281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":16,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.516581+0000","last_change":"2023-10-19T14:21:09.415572+0000","last_active":"2023-10-19T14:21:27.516581+0000","last_peered":"2023-10-19T14:21:27.516581+0000","last_clean":"2023-10-19T14:21:27.516581+0000","last_became_active":"2023-10-19T14:21:09.415271+0000","last_became_peered":"2023-10-19T14:21:09.415271+0000","last_unstale":"2023-10-19T14:21:27.516581+0000","last_undegraded":"2023-10-19T14:21:27.516581+0000","last_fullsized":"2023-10-19T14:21:27.516581+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:24:02.734433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.3c","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.526681+0000","last_change":"2023-10-19T14:21:27.528251+0000","last_active":"2023-10-19T14:21:28.526681+0000","last_peered":"2023-10-19T14:21:28.526681+0000","last_clean":"2023-10-19T14:21:28.526681+0000","last_became_active":"2023-10-19T14:21:27.527967+0000","last_became_peered":"2023-10-19T14:21:27.527967+0000","last_unstale":"2023-10-19T14:21:28.526681+0000","last_undegraded":"2023-10-19T14:21:28.526681+0000","last_fullsized":"2023-10-19T14:21:28.526681+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:45:06.945233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.3e","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.523341+0000","last_change":"2023-10-19T14:21:10.439966+0000","last_active":"2023-10-19T14:21:27.523341+0000","last_peered":"2023-10-19T14:21:27.523341+0000","last_clean":"2023-10-19T14:21:27.523341+0000","last_became_active":"2023-10-19T14:21:10.438773+0000","last_became_peered":"2023-10-19T14:21:10.438773+0000","last_unstale":"2023-10-19T14:21:27.523341+0000","last_undegraded":"2023-10-19T14:21:27.523341+0000","last_fullsized":"2023-10-19T14:21:27.523341+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:42:16.397379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":16,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.510814+0000","last_change":"2023-10-19T14:21:09.409229+0000","last_active":"2023-10-19T14:21:27.510814+0000","last_peered":"2023-10-19T14:21:27.510814+0000","last_clean":"2023-10-19T14:21:27.510814+0000","last_became_active":"2023-10-19T14:21:09.408885+0000","last_became_peered":"2023-10-19T14:21:09.408885+0000","last_unstale":"2023-10-19T14:21:27.510814+0000","last_undegraded":"2023-10-19T14:21:27.510814+0000","last_fullsized":"2023-10-19T14:21:27.510814+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:05:37.351111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.3d","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.527622+0000","last_change":"2023-10-19T14:21:27.534679+0000","last_active":"2023-10-19T14:21:28.527622+0000","last_peered":"2023-10-19T14:21:28.527622+0000","last_clean":"2023-10-19T14:21:28.527622+0000","last_became_active":"2023-10-19T14:21:27.534418+0000","last_became_peered":"2023-10-19T14:21:27.534418+0000","last_unstale":"2023-10-19T14:21:28.527622+0000","last_undegraded":"2023-10-19T14:21:28.527622+0000","last_fullsized":"2023-10-19T14:21:28.527622+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:50:51.105426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.3f","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.525499+0000","last_change":"2023-10-19T14:21:10.437868+0000","last_active":"2023-10-19T14:21:27.525499+0000","last_peered":"2023-10-19T14:21:27.525499+0000","last_clean":"2023-10-19T14:21:27.525499+0000","last_became_active":"2023-10-19T14:21:10.437535+0000","last_became_peered":"2023-10-19T14:21:10.437535+0000","last_unstale":"2023-10-19T14:21:27.525499+0000","last_undegraded":"2023-10-19T14:21:27.525499+0000","last_fullsized":"2023-10-19T14:21:27.525499+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:44:47.905654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":16,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.523786+0000","last_change":"2023-10-19T14:21:09.407963+0000","last_active":"2023-10-19T14:21:27.523786+0000","last_peered":"2023-10-19T14:21:27.523786+0000","last_clean":"2023-10-19T14:21:27.523786+0000","last_became_active":"2023-10-19T14:21:09.407684+0000","last_became_peered":"2023-10-19T14:21:09.407684+0000","last_unstale":"2023-10-19T14:21:27.523786+0000","last_undegraded":"2023-10-19T14:21:27.523786+0000","last_fullsized":"2023-10-19T14:21:27.523786+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:45:20.541560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.23","version":"0'0","reported_seq":11,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-19T14:21:28.526849+0000","last_change":"2023-10-19T14:21:27.538875+0000","last_active":"2023-10-19T14:21:28.526849+0000","last_peered":"2023-10-19T14:21:28.526849+0000","last_clean":"2023-10-19T14:21:28.526849+0000","last_became_active":"2023-10-19T14:21:27.537248+0000","last_became_peered":"2023-10-19T14:21:27.537248+0000","last_unstale":"2023-10-19T14:21:28.526849+0000","last_undegraded":"2023-10-19T14:21:28.526849+0000","last_fullsized":"2023-10-19T14:21:28.526849+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:26.346685+0000","last_clean_scrub_stamp":"2023-10-19T14:21:26.346685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:03:10.552408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.21","version":"0'0","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.512445+0000","last_change":"2023-10-19T14:21:10.434210+0000","last_active":"2023-10-19T14:21:27.512445+0000","last_peered":"2023-10-19T14:21:27.512445+0000","last_clean":"2023-10-19T14:21:27.512445+0000","last_became_active":"2023-10-19T14:21:10.433919+0000","last_became_peered":"2023-10-19T14:21:10.433919+0000","last_unstale":"2023-10-19T14:21:27.512445+0000","last_undegraded":"2023-10-19T14:21:27.512445+0000","last_fullsized":"2023-10-19T14:21:27.512445+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:09.388927+0000","last_clean_scrub_stamp":"2023-10-19T14:21:09.388927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:56:57.148327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1e","version":"242'1","reported_seq":17,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.512784+0000","last_change":"2023-10-19T14:21:09.411855+0000","last_active":"2023-10-19T14:21:27.512784+0000","last_peered":"2023-10-19T14:21:27.512784+0000","last_clean":"2023-10-19T14:21:27.512784+0000","last_became_active":"2023-10-19T14:21:09.411510+0000","last_became_peered":"2023-10-19T14:21:09.411510+0000","last_unstale":"2023-10-19T14:21:27.512784+0000","last_undegraded":"2023-10-19T14:21:27.512784+0000","last_fullsized":"2023-10-19T14:21:27.512784+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":240,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:21:08.378327+0000","last_clean_scrub_stamp":"2023-10-19T14:21:08.378327+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:44:43.405619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"229'3208","reported_seq":4184,"reported_epoch":246,"state":"active+clean","last_fresh":"2023-10-19T14:21:27.516605+0000","last_change":"2023-10-19T12:44:08.636613+0000","last_active":"2023-10-19T14:21:27.516605+0000","last_peered":"2023-10-19T14:21:27.516605+0000","last_clean":"2023-10-19T14:21:27.516605+0000","last_became_active":"2023-10-19T12:44:08.635697+0000","last_became_peered":"2023-10-19T12:44:08.635697+0000","last_unstale":"2023-10-19T14:21:27.516605+0000","last_undegraded":"2023-10-19T14:21:27.516605+0000","last_fullsized":"2023-10-19T14:21:27.516605+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_clean_scrub_stamp":"2023-10-19T12:44:07.399738+0000","objects_scrubbed":0,"log_size":3208,"log_dups_size":0,"ondisk_log_size":3208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:15:23.685485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2402,"num_read_kb":5036,"num_write":629,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":34,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":33,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":32,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":31,"num_pg":32,"stat_sum":{"num_bytes":3666,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2402,"num_read_kb":5036,"num_write":629,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3208,"ondisk_log_size":3208,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576014,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66760,"kb_used_data":3636,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305080,"statfs":{"total":96636764160,"available":96568401920,"internally_reserved":0,"allocated":3723264,"data_stored":2197191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":64616907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:21:10 2023","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.55100000000000005,"15min":0.57999999999999996},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.32200000000000001},"max":{"1min":0.754,"5min":0.80900000000000005,"15min":0.86199999999999999},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.503,"5min":0.57599999999999996,"15min":0.59899999999999998},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.312},"max":{"1min":0.66500000000000004,"5min":0.81899999999999995,"15min":0.86599999999999999},"last":0.56599999999999995}]},{"osd":1,"last update":"Thu Oct 19 14:21:10 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.58199999999999996,"15min":0.59499999999999997},"min":{"1min":0.45200000000000001,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.72199999999999998,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.56899999999999995,"15min":0.59299999999999997},"min":{"1min":0.39300000000000002,"5min":0.373,"15min":0.36699999999999999},"max":{"1min":0.78500000000000003,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.55700000000000005}]},{"osd":2,"last update":"Thu Oct 19 14:21:10 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.622,"15min":0.64200000000000002},"min":{"1min":0.47299999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.70499999999999996,"5min":0.84799999999999998,"15min":0.85499999999999998},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.63900000000000001,"15min":0.65700000000000003},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.622}]},{"osd":3,"last update":"Thu Oct 19 14:21:10 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.60799999999999998,"15min":0.63100000000000001},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.496},{"interface":"front","average":{"1min":0.628,"5min":0.67100000000000004,"15min":0.68300000000000005},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.83299999999999996,"5min":0.876,"15min":0.89600000000000002},"last":0.60999999999999999}]},{"osd":4,"last update":"Thu Oct 19 14:21:10 2023","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.50600000000000001,"15min":0.495},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.377},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.53000000000000003,"15min":0.51100000000000001},"min":{"1min":0.34999999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.41699999999999998}]},{"osd":5,"last update":"Thu Oct 19 14:21:10 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52300000000000002,"15min":0.51100000000000001},"min":{"1min":0.33700000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.57799999999999996,"15min":0.55400000000000005},"min":{"1min":0.40699999999999997,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.054,"5min":1.054,"15min":1.054},"last":0.51300000000000001}]},{"osd":6,"last update":"Thu Oct 19 14:21:10 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.58999999999999997,"15min":0.57599999999999996},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.35999999999999999},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.59299999999999997,"15min":0.57299999999999995},"min":{"1min":0.42999999999999999,"5min":0.35599999999999998,"15min":0.33500000000000002},"max":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.54400000000000004}]}]},{"osd":6,"up_from":13,"seq":55834576014,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56440,"kb_used_data":3052,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315400,"statfs":{"total":96636764160,"available":96578969600,"internally_reserved":0,"allocated":3125248,"data_stored":1606241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:21:02 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.59199999999999997,"15min":0.61099999999999999},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.78400000000000003,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.56299999999999994,"15min":0.60599999999999998},"min":{"1min":0.36699999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.71799999999999997,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.41499999999999998}]},{"osd":1,"last update":"Thu Oct 19 14:21:02 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.60099999999999998,"15min":0.61799999999999999},"min":{"1min":0.373,"5min":0.373,"15min":0.35799999999999998},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":1.165},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.60199999999999998,"15min":0.626},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.77400000000000002,"5min":0.83699999999999997,"15min":0.91300000000000003},"last":0.56899999999999995}]},{"osd":2,"last update":"Thu Oct 19 14:21:02 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.57599999999999996,"15min":0.60199999999999998},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.69099999999999995,"5min":0.80100000000000005,"15min":0.873},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.60599999999999998,"15min":0.624},"min":{"1min":0.40899999999999997,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.74199999999999999,"5min":0.83499999999999996,"15min":0.93100000000000005},"last":0.52500000000000002}]},{"osd":3,"last update":"Thu Oct 19 14:21:02 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.629,"15min":0.64900000000000002},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.66700000000000004,"5min":0.875,"15min":0.875},"last":0.621},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.63800000000000001,"15min":0.65500000000000003},"min":{"1min":0.45400000000000001,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.92200000000000004,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.60499999999999998}]},{"osd":4,"last update":"Thu Oct 19 14:21:02 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.48899999999999999,"15min":0.48399999999999999},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.51000000000000001,"15min":0.50800000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.30399999999999999},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.85699999999999998},"last":0.50900000000000001}]},{"osd":5,"last update":"Thu Oct 19 14:21:02 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.50700000000000001,"15min":0.502},"min":{"1min":0.34799999999999998,"5min":0.32700000000000001,"15min":0.32500000000000001},"max":{"1min":0.79900000000000004,"5min":1.115,"15min":1.115},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.56599999999999995,"15min":0.54500000000000004},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.58599999999999997}]},{"osd":7,"last update":"Thu Oct 19 14:21:02 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.56000000000000005,"15min":0.54700000000000004},"min":{"1min":0.36799999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.90100000000000002},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.56699999999999995,"15min":0.55400000000000005},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.98399999999999999,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.65500000000000003}]}]},{"osd":5,"up_from":13,"seq":55834576014,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66168,"kb_used_data":3044,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305672,"statfs":{"total":96636764160,"available":96569008128,"internally_reserved":0,"allocated":3117056,"data_stored":1606165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:20:46 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.58499999999999996,"15min":0.61699999999999999},"min":{"1min":0.33200000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.78700000000000003,"5min":0.79700000000000004,"15min":0.86399999999999999},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.60999999999999999,"15min":0.63800000000000001},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.77400000000000002,"5min":0.85899999999999999,"15min":0.85999999999999999},"last":0.627}]},{"osd":1,"last update":"Thu Oct 19 14:20:46 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.59499999999999997,"15min":0.628},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.749,"5min":0.80300000000000005,"15min":0.877},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.626,"15min":0.64000000000000001},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.81599999999999995,"5min":0.86699999999999999,"15min":0.88500000000000001},"last":0.68000000000000005}]},{"osd":2,"last update":"Thu Oct 19 14:20:46 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.59999999999999998,"15min":0.622},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.75900000000000001,"5min":0.83799999999999997,"15min":0.85099999999999998},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.63300000000000001,"15min":0.65500000000000003},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.81299999999999994,"5min":0.83299999999999996,"15min":0.90800000000000003},"last":0.51100000000000001}]},{"osd":3,"last update":"Thu Oct 19 14:20:46 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.61399999999999999,"15min":0.64100000000000001},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.77100000000000002,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.64000000000000001,"15min":0.66700000000000004},"min":{"1min":0.436,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.73899999999999999,"5min":0.876,"15min":0.90900000000000003},"last":0.59299999999999997}]},{"osd":4,"last update":"Thu Oct 19 14:20:46 2023","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.48499999999999999,"15min":0.48299999999999998},"min":{"1min":0.35499999999999998,"5min":0.34999999999999998,"15min":0.32500000000000001},"max":{"1min":0.56200000000000006,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.437,"5min":0.499,"15min":0.501},"min":{"1min":0.308,"5min":0.308,"15min":0.28599999999999998},"max":{"1min":0.57299999999999995,"5min":0.77500000000000002,"15min":0.82099999999999995},"last":0.64000000000000001}]},{"osd":6,"last update":"Thu Oct 19 14:20:46 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.51000000000000001,"15min":0.497},"min":{"1min":0.32200000000000001,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.72999999999999998,"5min":0.80800000000000005,"15min":0.88800000000000001},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.53300000000000003,"15min":0.53600000000000003},"min":{"1min":0.39900000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.70399999999999996,"5min":0.78000000000000003,"15min":0.88100000000000001},"last":0.71199999999999997}]},{"osd":7,"last update":"Thu Oct 19 14:20:46 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.50800000000000001,"15min":0.51500000000000001},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.625,"5min":0.749,"15min":0.82899999999999996},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.57399999999999995,"15min":0.56999999999999995},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.67300000000000004,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.66800000000000004}]}]},{"osd":4,"up_from":13,"seq":55834576014,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48120,"kb_used_data":3044,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323720,"statfs":{"total":96636764160,"available":96587489280,"internally_reserved":0,"allocated":3117056,"data_stored":1600262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:21:16 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.54700000000000004,"15min":0.57099999999999995},"min":{"1min":0.30299999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.879},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.56899999999999995,"15min":0.59299999999999997},"min":{"1min":0.35299999999999998,"5min":0.31,"15min":0.31},"max":{"1min":0.72699999999999998,"5min":0.82599999999999996,"15min":0.873},"last":0.498}]},{"osd":1,"last update":"Thu Oct 19 14:21:16 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.59699999999999998,"15min":0.63500000000000001},"min":{"1min":0.40300000000000002,"5min":0.372,"15min":0.32600000000000001},"max":{"1min":0.74099999999999999,"5min":0.80400000000000005,"15min":0.92700000000000005},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.60899999999999999,"15min":0.63900000000000001},"min":{"1min":0.40500000000000003,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.755,"5min":0.90000000000000002,"15min":1.0940000000000001},"last":0.60099999999999998}]},{"osd":2,"last update":"Thu Oct 19 14:21:16 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.626,"15min":0.64700000000000002},"min":{"1min":0.38900000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.80300000000000005,"5min":1.008,"15min":1.008},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.64200000000000002,"15min":0.66600000000000004},"min":{"1min":0.47199999999999998,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.81799999999999995,"5min":0.98799999999999999,"15min":1.008},"last":0.63}]},{"osd":3,"last update":"Thu Oct 19 14:21:16 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.59499999999999997,"15min":0.627},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.68600000000000005,"5min":0.83399999999999996,"15min":0.91900000000000004},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.65600000000000003,"15min":0.66600000000000004},"min":{"1min":0.48199999999999998,"5min":0.45400000000000001,"15min":0.437},"max":{"1min":0.85999999999999999,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.56299999999999994}]},{"osd":5,"last update":"Thu Oct 19 14:21:16 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.497,"15min":0.47799999999999998},"min":{"1min":0.33800000000000002,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.79700000000000004,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.50800000000000001,"15min":0.498},"min":{"1min":0.36499999999999999,"5min":0.33700000000000002,"15min":0.29199999999999998},"max":{"1min":0.81399999999999995,"5min":0.873,"15min":0.873},"last":0.40200000000000002}]},{"osd":6,"last update":"Thu Oct 19 14:21:16 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.50700000000000001,"15min":0.49399999999999999},"min":{"1min":0.38500000000000001,"5min":0.34799999999999998,"15min":0.28199999999999997},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.54900000000000004,"15min":0.54300000000000004},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.88,"5min":0.88,"15min":1.2969999999999999},"last":0.41899999999999998}]},{"osd":7,"last update":"Thu Oct 19 14:21:16 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55100000000000005,"15min":0.54500000000000004},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.32600000000000001},"max":{"1min":0.78200000000000003,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.61199999999999999,"15min":0.59899999999999998},"min":{"1min":0.371,"5min":0.32100000000000001,"15min":0.318},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":1.032},"last":0.46000000000000002}]}]},{"osd":3,"up_from":13,"seq":55834576013,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56460,"kb_used_data":3068,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315380,"statfs":{"total":96636764160,"available":96578949120,"internally_reserved":0,"allocated":3141632,"data_stored":1609691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54655432},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:20:49 2023","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.41099999999999998,"15min":0.41799999999999998},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.51800000000000002,"5min":0.61499999999999999,"15min":0.70699999999999996},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.40799999999999997,"15min":0.42799999999999999},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.752},"last":0.38800000000000001}]},{"osd":1,"last update":"Thu Oct 19 14:20:49 2023","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.42199999999999999,"15min":0.43099999999999999},"min":{"1min":0.25,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.54200000000000004,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.432,"5min":0.46400000000000002,"15min":0.47199999999999998},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.26400000000000001},"max":{"1min":0.67900000000000005,"5min":0.73099999999999998,"15min":0.81799999999999995},"last":0.48199999999999998}]},{"osd":2,"last update":"Thu Oct 19 14:20:49 2023","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.46899999999999997,"15min":0.48299999999999998},"min":{"1min":0.33100000000000002,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.625,"5min":0.64600000000000002,"15min":0.73099999999999998},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.49199999999999999,"15min":0.504},"min":{"1min":0.31900000000000001,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.83699999999999997},"last":0.41799999999999998}]},{"osd":4,"last update":"Thu Oct 19 14:20:49 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.69199999999999995,"15min":0.70299999999999996},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.871,"5min":1.018,"15min":1.157},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.71099999999999997,"15min":0.72899999999999998},"min":{"1min":0.47399999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.97699999999999998,"5min":0.996,"15min":1.173},"last":0.83599999999999997}]},{"osd":5,"last update":"Thu Oct 19 14:20:49 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.68300000000000005,"15min":0.68500000000000005},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.41999999999999998},"max":{"1min":0.83399999999999996,"5min":0.97299999999999998,"15min":1.0149999999999999},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.67500000000000004,"15min":0.68600000000000005},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.84999999999999998,"5min":0.99399999999999999,"15min":1.105},"last":0.85199999999999998}]},{"osd":6,"last update":"Thu Oct 19 14:20:49 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.65900000000000003,"15min":0.68700000000000006},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.70499999999999996,"5min":1.0009999999999999,"15min":1.0669999999999999},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.621,"5min":0.69599999999999995,"15min":0.70999999999999996},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.85599999999999998,"5min":0.91500000000000004,"15min":1.079},"last":0.69399999999999995}]},{"osd":7,"last update":"Thu Oct 19 14:20:49 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.68799999999999994,"15min":0.69899999999999995},"min":{"1min":0.436,"5min":0.434,"15min":0.434},"max":{"1min":0.78800000000000003,"5min":0.96899999999999997,"15min":1.085},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.70899999999999996,"15min":0.72899999999999998},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.79400000000000004,"5min":0.97199999999999998,"15min":1.1739999999999999},"last":0.875}]}]},{"osd":2,"up_from":13,"seq":55834576015,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48140,"kb_used_data":3068,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323700,"statfs":{"total":96636764160,"available":96587468800,"internally_reserved":0,"allocated":3141632,"data_stored":1608963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:20:36 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.38800000000000001,"15min":0.39700000000000002},"min":{"1min":0.254,"5min":0.218,"15min":0.218},"max":{"1min":0.55300000000000005,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.435,"15min":0.436},"min":{"1min":0.28199999999999997,"5min":0.249,"15min":0.223},"max":{"1min":0.54100000000000004,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.53100000000000003}]},{"osd":1,"last update":"Thu Oct 19 14:20:36 2023","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.39600000000000002,"15min":0.41199999999999998},"min":{"1min":0.27500000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.47799999999999998,"5min":0.61699999999999999,"15min":0.65400000000000003},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.46700000000000003,"15min":0.46500000000000002},"min":{"1min":0.28499999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.60499999999999998,"5min":1.742,"15min":1.742},"last":0.28699999999999998}]},{"osd":3,"last update":"Thu Oct 19 14:20:36 2023","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.47099999999999997,"15min":0.47999999999999998},"min":{"1min":0.32100000000000001,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.63700000000000001,"5min":0.69299999999999995,"15min":0.74399999999999999},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.48799999999999999,"15min":0.50600000000000001},"min":{"1min":0.34499999999999997,"5min":0.32000000000000001,"15min":0.315},"max":{"1min":0.56999999999999995,"5min":0.67600000000000005,"15min":0.83599999999999997},"last":0.67700000000000005}]},{"osd":4,"last update":"Thu Oct 19 14:20:36 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.65700000000000003,"15min":0.66900000000000004},"min":{"1min":0.47099999999999997,"5min":0.39600000000000002,"15min":0.39500000000000002},"max":{"1min":0.85599999999999998,"5min":0.93300000000000005,"15min":0.998},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.67000000000000004,"15min":0.67900000000000005},"min":{"1min":0.46500000000000002,"5min":0.38900000000000001,"15min":0.35199999999999998},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.43099999999999999}]},{"osd":5,"last update":"Thu Oct 19 14:20:36 2023","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.69699999999999995,"15min":0.69599999999999995},"min":{"1min":0.52700000000000002,"5min":0.45100000000000001,"15min":0.40600000000000003},"max":{"1min":0.80000000000000004,"5min":1.4330000000000001,"15min":1.4330000000000001},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.68100000000000005,"15min":0.68700000000000006},"min":{"1min":0.52700000000000002,"5min":0.45100000000000001,"15min":0.39300000000000002},"max":{"1min":0.93300000000000005,"5min":1.595,"15min":1.595},"last":0.61499999999999999}]},{"osd":6,"last update":"Thu Oct 19 14:20:36 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.69299999999999995,"15min":0.70799999999999996},"min":{"1min":0.48799999999999999,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.996,"5min":0.996,"15min":1.0189999999999999},"last":0.503},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.71899999999999997,"15min":0.72099999999999997},"min":{"1min":0.48899999999999999,"5min":0.433,"15min":0.40600000000000003},"max":{"1min":0.93899999999999995,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":0.83399999999999996}]},{"osd":7,"last update":"Thu Oct 19 14:20:36 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.70199999999999996,"15min":0.71999999999999997},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.41799999999999998},"max":{"1min":0.88700000000000001,"5min":1.0129999999999999,"15min":1.0289999999999999},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.73599999999999999,"15min":0.73099999999999998},"min":{"1min":0.51900000000000002,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.97299999999999998,"5min":1.53,"15min":1.53},"last":0.86699999999999999}]}]},{"osd":1,"up_from":13,"seq":55834576014,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48120,"kb_used_data":3048,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323720,"statfs":{"total":96636764160,"available":96587489280,"internally_reserved":0,"allocated":3121152,"data_stored":1606219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.39600000000000002,"15min":0.40400000000000003},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23100000000000001},"max":{"1min":0.57299999999999995,"5min":0.61599999999999999,"15min":0.67400000000000004},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.435,"15min":0.45400000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.54200000000000004,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.33100000000000002}]},{"osd":2,"last update":"Thu Oct 19 14:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.43099999999999999,"15min":0.442},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.56899999999999995,"5min":0.63100000000000001,"15min":0.77500000000000002},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.48899999999999999,"15min":0.48999999999999999},"min":{"1min":0.32400000000000001,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.67300000000000004,"5min":0.68700000000000006,"15min":0.82799999999999996},"last":0.44600000000000001}]},{"osd":3,"last update":"Thu Oct 19 14:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.499,"15min":0.505},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.29499999999999998},"max":{"1min":0.64300000000000002,"5min":0.76000000000000001,"15min":0.78900000000000003},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.436,"15min":0.44600000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.26100000000000001},"max":{"1min":0.60799999999999998,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.377}]},{"osd":4,"last update":"Thu Oct 19 14:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.66400000000000003,"15min":0.67100000000000004},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.92800000000000005,"5min":0.98299999999999998,"15min":1.042},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.67100000000000004,"15min":0.66900000000000004},"min":{"1min":0.44500000000000001,"5min":0.41299999999999998,"15min":0.32300000000000001},"max":{"1min":0.89400000000000002,"5min":0.93500000000000005,"15min":1.1659999999999999},"last":0.65300000000000002}]},{"osd":5,"last update":"Thu Oct 19 14:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.68799999999999994,"15min":0.68600000000000005},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.82099999999999995,"5min":1.002,"15min":1.3500000000000001},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.70599999999999996,"15min":0.70799999999999996},"min":{"1min":0.46700000000000003,"5min":0.44,"15min":0.40500000000000003},"max":{"1min":1.0009999999999999,"5min":1.0149999999999999,"15min":1.121},"last":0.47799999999999998}]},{"osd":6,"last update":"Thu Oct 19 14:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.67800000000000005,"15min":0.68200000000000005},"min":{"1min":0.51000000000000001,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.95499999999999996,"5min":1.0129999999999999,"15min":1.2310000000000001},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.71899999999999997,"15min":0.71599999999999997},"min":{"1min":0.48199999999999998,"5min":0.434,"15min":0.434},"max":{"1min":0.93100000000000005,"5min":1.0109999999999999,"15min":1.274},"last":0.69699999999999995}]},{"osd":7,"last update":"Thu Oct 19 14:20:40 2023","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.71899999999999997,"15min":0.72399999999999998},"min":{"1min":0.498,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.96799999999999997,"5min":0.99099999999999999,"15min":1.075},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.70099999999999996,"15min":0.70199999999999996},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.44400000000000001},"max":{"1min":0.91200000000000003,"5min":1.0169999999999999,"15min":1.117},"last":0.60999999999999999}]}]},{"osd":0,"up_from":13,"seq":55834576015,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61180,"kb_used_data":3632,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310660,"statfs":{"total":96636764160,"available":96574115840,"internally_reserved":0,"allocated":3719168,"data_stored":2198782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Oct 19 14:21:13 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.40100000000000002,"15min":0.41099999999999998},"min":{"1min":0.26300000000000001,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.47799999999999998,"5min":0.61499999999999999,"15min":0.624},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.41699999999999998,"15min":0.45500000000000002},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.49199999999999999,"5min":0.70899999999999996,"15min":0.81100000000000005},"last":0.71199999999999997}]},{"osd":2,"last update":"Thu Oct 19 14:21:13 2023","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.34100000000000003,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.54400000000000004,"5min":0.754,"15min":0.754},"last":0.434},{"interface":"front","average":{"1min":0.377,"5min":0.42399999999999999,"15min":0.44400000000000001},"min":{"1min":0.28100000000000003,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.46899999999999997,"5min":0.60199999999999998,"15min":0.69099999999999995},"last":0.38900000000000001}]},{"osd":3,"last update":"Thu Oct 19 14:21:13 2023","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.436},"min":{"1min":0.27200000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.50800000000000001,"5min":0.58799999999999997,"15min":0.66800000000000004},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.48199999999999998,"15min":0.50700000000000001},"min":{"1min":0.29299999999999998,"5min":0.254,"15min":0.254},"max":{"1min":0.72399999999999998,"5min":0.78100000000000003,"15min":0.84599999999999997},"last":0.55600000000000005}]},{"osd":4,"last update":"Thu Oct 19 14:21:13 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.63,"15min":0.67000000000000004},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.80300000000000005,"5min":0.93100000000000005,"15min":1.1890000000000001},"last":0.497},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.628,"15min":0.67400000000000004},"min":{"1min":0.45900000000000002,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.75600000000000001,"5min":0.93400000000000005,"15min":1.2150000000000001},"last":0.51600000000000001}]},{"osd":5,"last update":"Thu Oct 19 14:21:13 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.67200000000000004,"15min":0.71399999999999997},"min":{"1min":0.48599999999999999,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.83099999999999996,"5min":0.98499999999999999,"15min":1.1579999999999999},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.67500000000000004,"15min":0.71399999999999997},"min":{"1min":0.499,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.80000000000000004,"5min":0.98299999999999998,"15min":1.2310000000000001},"last":0.64200000000000002}]},{"osd":6,"last update":"Thu Oct 19 14:21:13 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.66300000000000003,"15min":0.69899999999999995},"min":{"1min":0.46100000000000002,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.85499999999999998,"5min":0.97799999999999998,"15min":1.1739999999999999},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.65300000000000002,"15min":0.69599999999999995},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.70299999999999996,"5min":0.97799999999999998,"15min":1.0189999999999999},"last":0.61599999999999999}]},{"osd":7,"last update":"Thu Oct 19 14:21:13 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.66600000000000004,"15min":0.70999999999999996},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.79900000000000004,"5min":0.93400000000000005,"15min":1.1180000000000001},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.66100000000000003,"15min":0.69599999999999995},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.84499999999999997,"5min":0.93799999999999994,"15min":1.0349999999999999},"last":0.53600000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-19T14:21:30.895 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-19T14:21:30.895 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T14:21:31.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.031+0000 7f8647864640 1 -- 172.21.15.57:0/613244644 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f86400cf340 msgr2=0x7f86400d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:31.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.031+0000 7f8647864640 1 --2- 172.21.15.57:0/613244644 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f86400cf340 0x7f86400d1910 secure :-1 s=READY pgs=2704 cs=0 l=1 rev1=1 crypto rx=0x7f864008a120 tx=0x7f863c02c260 comp rx=0 tx=0).stop 2023-10-19T14:21:31.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.031+0000 7f8647864640 1 -- 172.21.15.57:0/613244644 shutdown_connections 2023-10-19T14:21:31.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.031+0000 7f8647864640 1 --2- 172.21.15.57:0/613244644 >> v2:172.21.15.179:3300/0 conn(0x7f864015b2a0 0x7f86400cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:31.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.031+0000 7f8647864640 1 --2- 172.21.15.57:0/613244644 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f86400cf340 0x7f86400d1910 unknown :-1 s=CLOSED pgs=2704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:31.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.031+0000 7f8647864640 1 -- 172.21.15.57:0/613244644 >> 172.21.15.57:0/613244644 conn(0x7f86400d0d60 msgr2=0x7f86400d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:31.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.032+0000 7f8647864640 1 -- 172.21.15.57:0/613244644 shutdown_connections 2023-10-19T14:21:31.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.032+0000 7f8647864640 1 -- 172.21.15.57:0/613244644 wait complete. 2023-10-19T14:21:31.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.033+0000 7f8647864640 1 Processor -- start 2023-10-19T14:21:31.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.033+0000 7f8647864640 1 -- start start 2023-10-19T14:21:31.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.033+0000 7f8647864640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f86400ceb40 0x7f8640158350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:31.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.033+0000 7f8647864640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f86400cf340 0x7f86400c9c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:31.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.033+0000 7f8647864640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f864015b2a0 0x7f864006b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:31.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.033+0000 7f8647864640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f864015c480 con 0x7f864015b2a0 2023-10-19T14:21:31.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.034+0000 7f8647864640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f864015b870 con 0x7f86400ceb40 2023-10-19T14:21:31.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.034+0000 7f8647864640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f864015bb50 con 0x7f86400cf340 2023-10-19T14:21:31.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.034+0000 7f8645e01640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f864015b2a0 0x7f864006b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:31.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.034+0000 7f8645e01640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f864015b2a0 0x7f864006b1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37248/0 (socket says 172.21.15.57:37248) 2023-10-19T14:21:31.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.034+0000 7f8645e01640 1 -- 172.21.15.57:0/125398301 learned_addr learned my addr 172.21.15.57:0/125398301 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:31.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.034+0000 7f8644dff640 1 --2- 172.21.15.57:0/125398301 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f86400cf340 0x7f86400c9c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:31.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.034+0000 7f8645600640 1 --2- 172.21.15.57:0/125398301 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f86400ceb40 0x7f8640158350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:31.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.034+0000 7f8645e01640 1 -- 172.21.15.57:0/125398301 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f86400cf340 msgr2=0x7f86400c9c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:31.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.034+0000 7f8645e01640 1 --2- 172.21.15.57:0/125398301 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f86400cf340 0x7f86400c9c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:31.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.034+0000 7f8645e01640 1 -- 172.21.15.57:0/125398301 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f86400ceb40 msgr2=0x7f8640158350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:31.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.034+0000 7f8645e01640 1 --2- 172.21.15.57:0/125398301 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f86400ceb40 0x7f8640158350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:31.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.034+0000 7f8645e01640 1 -- 172.21.15.57:0/125398301 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f863c009900 con 0x7f864015b2a0 2023-10-19T14:21:31.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.034+0000 7f8645e01640 1 --2- 172.21.15.57:0/125398301 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f864015b2a0 0x7f864006b1f0 secure :-1 s=READY pgs=2705 cs=0 l=1 rev1=1 crypto rx=0x7f863800c3c0 tx=0x7f863800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:31.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.034+0000 7f8635ffb640 1 -- 172.21.15.57:0/125398301 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86380052d0 con 0x7f864015b2a0 2023-10-19T14:21:31.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.034+0000 7f8635ffb640 1 -- 172.21.15.57:0/125398301 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f8638003610 con 0x7f864015b2a0 2023-10-19T14:21:31.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.034+0000 7f8635ffb640 1 -- 172.21.15.57:0/125398301 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8638003a20 con 0x7f864015b2a0 2023-10-19T14:21:31.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.034+0000 7f8647864640 1 -- 172.21.15.57:0/125398301 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f864015be30 con 0x7f864015b2a0 2023-10-19T14:21:31.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.034+0000 7f8647864640 1 -- 172.21.15.57:0/125398301 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8640158890 con 0x7f864015b2a0 2023-10-19T14:21:31.043 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.035+0000 7f8647864640 1 -- 172.21.15.57:0/125398301 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f864015c5f0 con 0x7f864015b2a0 2023-10-19T14:21:31.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.036+0000 7f8635ffb640 1 -- 172.21.15.57:0/125398301 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f8638003b80 con 0x7f864015b2a0 2023-10-19T14:21:31.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.036+0000 7f8635ffb640 1 --2- 172.21.15.57:0/125398301 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f862406b080 0x7f862406d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:31.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.036+0000 7f8635ffb640 1 -- 172.21.15.57:0/125398301 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(247..247 src has 1..247) v4 ==== 6921+0+0 (secure 0 0 0) 0x7f86380895c0 con 0x7f864015b2a0 2023-10-19T14:21:31.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.036+0000 7f8645600640 1 --2- 172.21.15.57:0/125398301 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f862406b080 0x7f862406d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:31.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.037+0000 7f8645600640 1 --2- 172.21.15.57:0/125398301 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f862406b080 0x7f862406d540 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f863c036000 tx=0x7f863c004ab0 comp rx=0 tx=0).ready entity=mgr.9530 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:31.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.040+0000 7f8635ffb640 1 -- 172.21.15.57:0/125398301 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f8638053fe0 con 0x7f864015b2a0 2023-10-19T14:21:31.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.250+0000 7f8647864640 1 -- 172.21.15.57:0/125398301 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f86400cb980 con 0x7f864015b2a0 2023-10-19T14:21:31.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.251+0000 7f8635ffb640 1 -- 172.21.15.57:0/125398301 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+6140 (secure 0 0 0) 0x7f8638057600 con 0x7f864015b2a0 2023-10-19T14:21:31.254 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:21:31.255 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":107,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9761,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/325888554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":325888554},{"type":"v1","addr":"172.21.15.179:6836","nonce":325888554}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":100},{"gid":9764,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/8038232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":8038232},{"type":"v1","addr":"172.21.15.179:6837","nonce":8038232}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":100}],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T14:21:09.759935+0000","modified":"2023-10-19T14:21:11.768676+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":12564},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12564":{"gid":12564,"name":"c","rank":0,"incarnation":101,"state":"up:active","state_seq":3,"addr":"172.21.15.57:6835/18187246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":18187246},{"type":"v1","addr":"172.21.15.57:6835","nonce":18187246}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T14:21:26.724728+0000","modified":"2023-10-19T14:21:28.872856+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":12555},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12555":{"gid":12555,"name":"a","rank":0,"incarnation":105,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6833/4256498777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":4256498777},{"type":"v1","addr":"172.21.15.57:6833","nonce":4256498777}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2023-10-19T14:21:31.255 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 107 2023-10-19T14:21:31.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.253+0000 7f8647864640 1 -- 172.21.15.57:0/125398301 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f862406b080 msgr2=0x7f862406d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:31.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.253+0000 7f8647864640 1 --2- 172.21.15.57:0/125398301 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f862406b080 0x7f862406d540 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f863c036000 tx=0x7f863c004ab0 comp rx=0 tx=0).stop 2023-10-19T14:21:31.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.253+0000 7f8647864640 1 -- 172.21.15.57:0/125398301 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f864015b2a0 msgr2=0x7f864006b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:31.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.253+0000 7f8647864640 1 --2- 172.21.15.57:0/125398301 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f864015b2a0 0x7f864006b1f0 secure :-1 s=READY pgs=2705 cs=0 l=1 rev1=1 crypto rx=0x7f863800c3c0 tx=0x7f863800c880 comp rx=0 tx=0).stop 2023-10-19T14:21:31.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.253+0000 7f8647864640 1 -- 172.21.15.57:0/125398301 shutdown_connections 2023-10-19T14:21:31.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.253+0000 7f8647864640 1 --2- 172.21.15.57:0/125398301 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f862406b080 0x7f862406d540 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:31.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.253+0000 7f8647864640 1 --2- 172.21.15.57:0/125398301 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f864015b2a0 0x7f864006b1f0 unknown :-1 s=CLOSED pgs=2705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:31.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.253+0000 7f8647864640 1 --2- 172.21.15.57:0/125398301 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f86400cf340 0x7f86400c9c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:31.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.253+0000 7f8647864640 1 --2- 172.21.15.57:0/125398301 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f86400ceb40 0x7f8640158350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:31.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.253+0000 7f8647864640 1 -- 172.21.15.57:0/125398301 >> 172.21.15.57:0/125398301 conn(0x7f86400d0d60 msgr2=0x7f864015a3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:31.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.253+0000 7f8647864640 1 -- 172.21.15.57:0/125398301 shutdown_connections 2023-10-19T14:21:31.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.253+0000 7f8647864640 1 -- 172.21.15.57:0/125398301 wait complete. 2023-10-19T14:21:31.380 INFO:tasks.ceph.mds.a.smithi057.stderr:2023-10-19T14:21:31.377+0000 7f6f39e5d640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-19T14:21:31.669 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-19T14:21:26.724728+0000', 'modified': '2023-10-19T14:21:28.872856+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 12555}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_12555': {'gid': 12555, 'name': 'a', 'rank': 0, 'incarnation': 105, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6833/4256498777', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 4256498777}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 4256498777}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'backup_fs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-19T14:21:31.670 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-19T14:21:31.670 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2023-10-19T14:21:31.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.780+0000 7f04fc58e640 1 -- 172.21.15.57:0/1111186324 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f04e0003f20 msgr2=0x7f04e00047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:31.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.780+0000 7f04fc58e640 1 --2- 172.21.15.57:0/1111186324 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f04e0003f20 0x7f04e00047c0 secure :-1 s=READY pgs=2707 cs=0 l=1 rev1=1 crypto rx=0x7f04e4009b50 tx=0x7f04e00057f0 comp rx=0 tx=0).stop 2023-10-19T14:21:31.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.780+0000 7f04fab2b640 1 -- 172.21.15.57:0/1111186324 reap_dead start 2023-10-19T14:21:31.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.780+0000 7f04fc58e640 1 -- 172.21.15.57:0/1111186324 shutdown_connections 2023-10-19T14:21:31.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.780+0000 7f04fc58e640 1 -- 172.21.15.57:0/1111186324 >> 172.21.15.57:0/1111186324 conn(0x7f04f406fae0 msgr2=0x7f04f406fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:31.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.780+0000 7f04fc58e640 1 -- 172.21.15.57:0/1111186324 shutdown_connections 2023-10-19T14:21:31.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.780+0000 7f04fc58e640 1 -- 172.21.15.57:0/1111186324 wait complete. 2023-10-19T14:21:31.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.781+0000 7f04fc58e640 1 Processor -- start 2023-10-19T14:21:31.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.782+0000 7f04fc58e640 1 -- start start 2023-10-19T14:21:31.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.782+0000 7f04fc58e640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f04e0003f20 0x7f04f414ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:31.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.782+0000 7f04fc58e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f04f40abf20 0x7f04f41e37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:31.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.782+0000 7f04fc58e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f04f40aa8c0 0x7f04f4145b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:31.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.782+0000 7f04fc58e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04f414d890 con 0x7f04f40abf20 2023-10-19T14:21:31.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.782+0000 7f04fc58e640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04f40aae80 con 0x7f04f40aa8c0 2023-10-19T14:21:31.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.782+0000 7f04fc58e640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04f40ab160 con 0x7f04e0003f20 2023-10-19T14:21:31.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.782+0000 7f04f9b29640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f04f40abf20 0x7f04f41e37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:31.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.782+0000 7f04f9b29640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f04f40abf20 0x7f04f41e37e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37266/0 (socket says 172.21.15.57:37266) 2023-10-19T14:21:31.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.782+0000 7f04f9b29640 1 -- 172.21.15.57:0/3932263000 learned_addr learned my addr 172.21.15.57:0/3932263000 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:31.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.782+0000 7f04fa32a640 1 --2- 172.21.15.57:0/3932263000 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f04e0003f20 0x7f04f414ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:31.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.782+0000 7f04fab2b640 1 --2- 172.21.15.57:0/3932263000 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f04f40aa8c0 0x7f04f4145b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:31.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.783+0000 7f04f9b29640 1 -- 172.21.15.57:0/3932263000 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f04e0003f20 msgr2=0x7f04f414ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:31.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.783+0000 7f04f9b29640 1 --2- 172.21.15.57:0/3932263000 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f04e0003f20 0x7f04f414ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:31.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.783+0000 7f04f9b29640 1 -- 172.21.15.57:0/3932263000 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f04f40aa8c0 msgr2=0x7f04f4145b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:31.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.783+0000 7f04f9b29640 1 --2- 172.21.15.57:0/3932263000 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f04f40aa8c0 0x7f04f4145b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:31.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.783+0000 7f04f9b29640 1 -- 172.21.15.57:0/3932263000 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04f40b2330 con 0x7f04f40abf20 2023-10-19T14:21:31.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.783+0000 7f04fab2b640 1 --2- 172.21.15.57:0/3932263000 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f04f40aa8c0 0x7f04f4145b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:21:31.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.783+0000 7f04f9b29640 1 --2- 172.21.15.57:0/3932263000 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f04f40abf20 0x7f04f41e37e0 secure :-1 s=READY pgs=2708 cs=0 l=1 rev1=1 crypto rx=0x7f04e402cb70 tx=0x7f04e4033c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:31.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.783+0000 7f04eaffd640 1 -- 172.21.15.57:0/3932263000 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04e4004440 con 0x7f04f40abf20 2023-10-19T14:21:31.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.783+0000 7f04eaffd640 1 -- 172.21.15.57:0/3932263000 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f04e4033e40 con 0x7f04f40abf20 2023-10-19T14:21:31.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.783+0000 7f04eaffd640 1 -- 172.21.15.57:0/3932263000 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04e4003850 con 0x7f04f40abf20 2023-10-19T14:21:31.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.783+0000 7f04fc58e640 1 -- 172.21.15.57:0/3932263000 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04f40ab440 con 0x7f04f40abf20 2023-10-19T14:21:31.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.783+0000 7f04fc58e640 1 -- 172.21.15.57:0/3932263000 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04f40aba00 con 0x7f04f40abf20 2023-10-19T14:21:31.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.785+0000 7f04eaffd640 1 -- 172.21.15.57:0/3932263000 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f04e40039b0 con 0x7f04f40abf20 2023-10-19T14:21:31.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.785+0000 7f04eaffd640 1 --2- 172.21.15.57:0/3932263000 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f04d806b0b0 0x7f04d806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:31.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.785+0000 7f04eaffd640 1 -- 172.21.15.57:0/3932263000 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(247..247 src has 1..247) v4 ==== 6921+0+0 (secure 0 0 0) 0x7f04e40acc60 con 0x7f04f40abf20 2023-10-19T14:21:31.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.786+0000 7f04fa32a640 1 --2- 172.21.15.57:0/3932263000 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f04d806b0b0 0x7f04d806d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:31.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.786+0000 7f04ff1fb740 1 Processor -- start 2023-10-19T14:21:31.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.786+0000 7f04fa32a640 1 --2- 172.21.15.57:0/3932263000 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f04d806b0b0 0x7f04d806d570 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f04f0001200 tx=0x7f04f0001110 comp rx=0 tx=0).ready entity=mgr.9530 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:31.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.786+0000 7f04ff1fb740 1 -- start start 2023-10-19T14:21:31.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.787+0000 7f04ff1fb740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556347a3bb40 0x556347a0eae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:31.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.787+0000 7f04ff1fb740 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x556347a3bf20 0x556347a3e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:31.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.787+0000 7f04ff1fb740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556347a0f020 0x556347a3ee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:31.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.787+0000 7f04ff1fb740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x556347920730 con 0x556347a3bb40 2023-10-19T14:21:31.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.787+0000 7f04ff1fb740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x556347981780 con 0x556347a0f020 2023-10-19T14:21:31.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.787+0000 7f04ff1fb740 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x55634797ee70 con 0x556347a3bf20 2023-10-19T14:21:31.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.787+0000 7f04fa32a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556347a3bb40 0x556347a0eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:31.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.787+0000 7f04fa32a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556347a3bb40 0x556347a0eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37280/0 (socket says 172.21.15.57:37280) 2023-10-19T14:21:31.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.787+0000 7f04fa32a640 1 -- 172.21.15.57:0/1231477341 learned_addr learned my addr 172.21.15.57:0/1231477341 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:31.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.787+0000 7f04f9b29640 1 --2- 172.21.15.57:0/1231477341 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x556347a3bf20 0x556347a3e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:31.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.787+0000 7f04fab2b640 1 --2- 172.21.15.57:0/1231477341 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556347a0f020 0x556347a3ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:31.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.787+0000 7f04fa32a640 1 -- 172.21.15.57:0/1231477341 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x556347a3bf20 msgr2=0x556347a3e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T14:21:31.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.787+0000 7f04fa32a640 1 --2- 172.21.15.57:0/1231477341 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x556347a3bf20 0x556347a3e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:31.801 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.787+0000 7f04fa32a640 1 -- 172.21.15.57:0/1231477341 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556347a0f020 msgr2=0x556347a3ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T14:21:31.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.787+0000 7f04fa32a640 1 --2- 172.21.15.57:0/1231477341 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556347a0f020 0x556347a3ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:31.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.787+0000 7f04fa32a640 1 -- 172.21.15.57:0/1231477341 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55634797f3d0 con 0x556347a3bb40 2023-10-19T14:21:31.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.787+0000 7f04f9b29640 1 --2- 172.21.15.57:0/1231477341 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x556347a3bf20 0x556347a3e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:21:31.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.787+0000 7f04fab2b640 1 --2- 172.21.15.57:0/1231477341 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556347a0f020 0x556347a3ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:21:31.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.788+0000 7f04fa32a640 1 --2- 172.21.15.57:0/1231477341 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556347a3bb40 0x556347a0eae0 secure :-1 s=READY pgs=2709 cs=0 l=1 rev1=1 crypto rx=0x7f04f0003940 tx=0x7f04f0014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f052fba56ec636fd server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:31.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.788+0000 7f04d7fff640 1 -- 172.21.15.57:0/1231477341 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04f00127c0 con 0x556347a3bb40 2023-10-19T14:21:31.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.788+0000 7f04d7fff640 1 -- 172.21.15.57:0/1231477341 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f04f0016070 con 0x556347a3bb40 2023-10-19T14:21:31.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.788+0000 7f04d7fff640 1 -- 172.21.15.57:0/1231477341 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04f0012b60 con 0x556347a3bb40 2023-10-19T14:21:31.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.788+0000 7f04ff1fb740 1 -- 172.21.15.57:0/1231477341 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556347a3bb40 msgr2=0x556347a0eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:31.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.788+0000 7f04ff1fb740 1 --2- 172.21.15.57:0/1231477341 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556347a3bb40 0x556347a0eae0 secure :-1 s=READY pgs=2709 cs=0 l=1 rev1=1 crypto rx=0x7f04f0003940 tx=0x7f04f0014000 comp rx=0 tx=0).stop 2023-10-19T14:21:31.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.788+0000 7f04ff1fb740 1 -- 172.21.15.57:0/1231477341 shutdown_connections 2023-10-19T14:21:31.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.788+0000 7f04ff1fb740 1 --2- 172.21.15.57:0/1231477341 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556347a0f020 0x556347a3ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:31.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.788+0000 7f04ff1fb740 1 --2- 172.21.15.57:0/1231477341 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x556347a3bf20 0x556347a3e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:31.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.788+0000 7f04ff1fb740 1 --2- 172.21.15.57:0/1231477341 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556347a3bb40 0x556347a0eae0 unknown :-1 s=CLOSED pgs=2709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:31.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.788+0000 7f04ff1fb740 1 -- 172.21.15.57:0/1231477341 >> 172.21.15.57:0/1231477341 conn(0x556347a10d50 msgr2=0x556347a598d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:31.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.788+0000 7f04ff1fb740 1 -- 172.21.15.57:0/1231477341 shutdown_connections 2023-10-19T14:21:31.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.788+0000 7f04ff1fb740 1 -- 172.21.15.57:0/1231477341 wait complete. 2023-10-19T14:21:31.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.789+0000 7f04ff1fb740 1 -- start start 2023-10-19T14:21:31.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.789+0000 7f04ff1fb740 1 Processor -- start 2023-10-19T14:21:31.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.790+0000 7f04ff1fb740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556347a3bb40 0x556347a25ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:31.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.790+0000 7f04ff1fb740 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x556347a3bf20 0x556347a3d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:31.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.790+0000 7f04ff1fb740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556347a3d870 0x556347a503f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:31.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.790+0000 7f04ff1fb740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x55634797a760 con 0x556347a3bb40 2023-10-19T14:21:31.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.790+0000 7f04ff1fb740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563479358f0 con 0x556347a3d870 2023-10-19T14:21:31.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.790+0000 7f04ff1fb740 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x556347935590 con 0x556347a3bf20 2023-10-19T14:21:31.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.790+0000 7f04ff1fb740 10 client.0 _ll_register_callbacks cb 0x556347971ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-19T14:21:31.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.790+0000 7f04fa32a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556347a3bb40 0x556347a25ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:31.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.790+0000 7f04fa32a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556347a3bb40 0x556347a25ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37284/0 (socket says 172.21.15.57:37284) 2023-10-19T14:21:31.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.790+0000 7f04fa32a640 1 -- 172.21.15.57:0/3721482964 learned_addr learned my addr 172.21.15.57:0/3721482964 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:31.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.790+0000 7f04fab2b640 1 --2- 172.21.15.57:0/3721482964 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556347a3d870 0x556347a503f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:31.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.790+0000 7f04f9b29640 1 --2- 172.21.15.57:0/3721482964 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x556347a3bf20 0x556347a3d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:31.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.790+0000 7f04fa32a640 1 -- 172.21.15.57:0/3721482964 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x556347a3bf20 msgr2=0x556347a3d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T14:21:31.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.790+0000 7f04fa32a640 1 --2- 172.21.15.57:0/3721482964 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x556347a3bf20 0x556347a3d330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:31.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.790+0000 7f04fa32a640 1 -- 172.21.15.57:0/3721482964 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556347a3d870 msgr2=0x556347a503f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T14:21:31.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.790+0000 7f04fa32a640 1 --2- 172.21.15.57:0/3721482964 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556347a3d870 0x556347a503f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:31.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.790+0000 7f04fa32a640 1 -- 172.21.15.57:0/3721482964 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04f40b2560 con 0x556347a3bb40 2023-10-19T14:21:31.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.791+0000 7f04fa32a640 1 --2- 172.21.15.57:0/3721482964 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556347a3bb40 0x556347a25ab0 secure :-1 s=READY pgs=2710 cs=0 l=1 rev1=1 crypto rx=0x7f04f00064f0 tx=0x7f04f0012ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=124b7efa6bcd3854 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:31.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.791+0000 7f04d57fa640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2023-10-19T14:21:31.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.791+0000 7f04d57fa640 1 -- 172.21.15.57:0/3721482964 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04f002d070 con 0x556347a3bb40 2023-10-19T14:21:31.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.791+0000 7f04d57fa640 1 -- 172.21.15.57:0/3721482964 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(247..247 src has 1..247) v4 ==== 6921+0+0 (secure 0 0 0) 0x7f04f0012d10 con 0x556347a3bb40 2023-10-19T14:21:31.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.791+0000 7f04ff1fb740 1 -- 172.21.15.57:0/3721482964 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556347973740 con 0x556347a3bb40 2023-10-19T14:21:31.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.791+0000 7f04d57fa640 1 -- 172.21.15.57:0/3721482964 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=107) v2 ==== 24+0+0 (secure 0 0 0) 0x7f04f0033420 con 0x556347a3bb40 2023-10-19T14:21:31.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.791+0000 7f04ff1fb740 10 client.12876 fetch_fsmap learned FSMap version 107 2023-10-19T14:21:31.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.791+0000 7f04ff1fb740 1 -- 172.21.15.57:0/3721482964 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=107,monmap=2+}) v3 -- 0x556347978de0 con 0x556347a3bb40 2023-10-19T14:21:31.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.791+0000 7f04d57fa640 1 -- 172.21.15.57:0/3721482964 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x556347973740 con 0x556347a3bb40 2023-10-19T14:21:31.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.791+0000 7f04d57fa640 1 -- 172.21.15.57:0/3721482964 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 107) v1 ==== 3891+0+0 (secure 0 0 0) 0x7f04f0033580 con 0x556347a3bb40 2023-10-19T14:21:31.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.791+0000 7f04ff1fb740 10 client.12876 fetch_fsmap finished waiting for FSMap version 107 2023-10-19T14:21:31.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.791+0000 7f04ff1fb740 10 client.12876 resolve_mds: resolved name 'a' to daemon mds.a 2023-10-19T14:21:31.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.791+0000 7f04ff1fb740 20 client.12876 populate_metadata read hostname 'smithi057' 2023-10-19T14:21:31.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.791+0000 7f04ff1fb740 1 --2- 172.21.15.57:0/3721482964 >> [v2:172.21.15.57:6832/4256498777,v1:172.21.15.57:6833/4256498777] conn(0x556347a26f10 0x556347a272f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:31.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.791+0000 7f04ff1fb740 4 client.12876 mds_command: new command op to 12555 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-19T14:21:31.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.791+0000 7f04ff1fb740 1 -- 172.21.15.57:0/3721482964 --> [v2:172.21.15.57:6832/4256498777,v1:172.21.15.57:6833/4256498777] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556347973460 con 0x556347a26f10 2023-10-19T14:21:31.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.791+0000 7f04f9b29640 1 --2- 172.21.15.57:0/3721482964 >> [v2:172.21.15.57:6832/4256498777,v1:172.21.15.57:6833/4256498777] conn(0x556347a26f10 0x556347a272f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:31.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.792+0000 7f04f9b29640 1 --2- 172.21.15.57:0/3721482964 >> [v2:172.21.15.57:6832/4256498777,v1:172.21.15.57:6833/4256498777] conn(0x556347a26f10 0x556347a272f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=989948be8c8aeaa3 server_cookie=ad308eee69301b48 in_seq=0 out_seq=0 2023-10-19T14:21:31.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.792+0000 7f04d57fa640 10 client.12876 ms_handle_connect on v2:172.21.15.57:6832/4256498777 2023-10-19T14:21:31.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.792+0000 7f04d57fa640 1 -- 172.21.15.57:0/3721482964 <== mds.0 v2:172.21.15.57:6832/4256498777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x556347973460 con 0x556347a26f10 2023-10-19T14:21:31.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.792+0000 7f04d57fa640 10 client.12876 handle_command_reply: tid=0 2023-10-19T14:21:31.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.792+0000 7f04ff1fb740 1 client.12876 shutdown 2023-10-19T14:21:31.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.792+0000 7f04ff1fb740 1 -- 172.21.15.57:0/3721482964 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556347a3bb40 msgr2=0x556347a25ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:31.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.792+0000 7f04ff1fb740 1 --2- 172.21.15.57:0/3721482964 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556347a3bb40 0x556347a25ab0 secure :-1 s=READY pgs=2710 cs=0 l=1 rev1=1 crypto rx=0x7f04f00064f0 tx=0x7f04f0012ae0 comp rx=0 tx=0).stop 2023-10-19T14:21:31.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.792+0000 7f04ff1fb740 1 -- 172.21.15.57:0/3721482964 shutdown_connections 2023-10-19T14:21:31.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.792+0000 7f04ff1fb740 1 --2- 172.21.15.57:0/3721482964 >> [v2:172.21.15.57:6832/4256498777,v1:172.21.15.57:6833/4256498777] conn(0x556347a26f10 0x556347a272f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:31.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.792+0000 7f04ff1fb740 1 --2- 172.21.15.57:0/3721482964 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556347a3d870 0x556347a503f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:31.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.792+0000 7f04ff1fb740 1 --2- 172.21.15.57:0/3721482964 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x556347a3bf20 0x556347a3d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:31.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.792+0000 7f04d57fa640 0 client.12876 ms_handle_reset on v2:172.21.15.57:6832/4256498777 2023-10-19T14:21:31.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.792+0000 7f04ff1fb740 1 --2- 172.21.15.57:0/3721482964 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556347a3bb40 0x556347a25ab0 unknown :-1 s=CLOSED pgs=2710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:31.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.792+0000 7f04ff1fb740 1 -- 172.21.15.57:0/3721482964 >> 172.21.15.57:0/3721482964 conn(0x556347a0f020 msgr2=0x556347a128f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:31.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.793+0000 7f04ff1fb740 1 -- 172.21.15.57:0/3721482964 shutdown_connections 2023-10-19T14:21:31.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.793+0000 7f04ff1fb740 1 -- 172.21.15.57:0/3721482964 wait complete. 2023-10-19T14:21:31.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.793+0000 7f04ff1fb740 20 client.12876 trim_cache size 0 max 16384 2023-10-19T14:21:31.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.808+0000 7f04ff1fb740 1 Processor -- start 2023-10-19T14:21:31.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.808+0000 7f04ff1fb740 1 -- start start 2023-10-19T14:21:31.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.809+0000 7f04ff1fb740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556347a3bb40 0x556347a12830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:31.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.809+0000 7f04ff1fb740 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x556347a3bf20 0x556347a72980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:31.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.809+0000 7f04ff1fb740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556347a3d870 0x556347a72ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:31.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.809+0000 7f04ff1fb740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563478fb270 con 0x556347a3bb40 2023-10-19T14:21:31.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.809+0000 7f04ff1fb740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x556347a28170 con 0x556347a3d870 2023-10-19T14:21:31.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.809+0000 7f04ff1fb740 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x556347a25530 con 0x556347a3bf20 2023-10-19T14:21:31.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.809+0000 7f04fa32a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556347a3bb40 0x556347a12830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:31.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.809+0000 7f04fa32a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556347a3bb40 0x556347a12830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37298/0 (socket says 172.21.15.57:37298) 2023-10-19T14:21:31.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.809+0000 7f04fa32a640 1 -- 172.21.15.57:0/73284924 learned_addr learned my addr 172.21.15.57:0/73284924 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:31.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.809+0000 7f04fa32a640 1 -- 172.21.15.57:0/73284924 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x556347a3bf20 msgr2=0x556347a72980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T14:21:31.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.809+0000 7f04fa32a640 1 --2- 172.21.15.57:0/73284924 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x556347a3bf20 0x556347a72980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:31.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.809+0000 7f04fa32a640 1 -- 172.21.15.57:0/73284924 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556347a3d870 msgr2=0x556347a72ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T14:21:31.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.809+0000 7f04fa32a640 1 --2- 172.21.15.57:0/73284924 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556347a3d870 0x556347a72ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:31.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.809+0000 7f04fa32a640 1 -- 172.21.15.57:0/73284924 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556347917d00 con 0x556347a3bb40 2023-10-19T14:21:31.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.809+0000 7f04fa32a640 1 --2- 172.21.15.57:0/73284924 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556347a3bb40 0x556347a12830 secure :-1 s=READY pgs=2711 cs=0 l=1 rev1=1 crypto rx=0x7f04f0012b50 tx=0x7f04f0014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70070adbfc9a37c server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:31.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.810+0000 7f04e8ff9640 1 -- 172.21.15.57:0/73284924 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04f002d040 con 0x556347a3bb40 2023-10-19T14:21:31.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.810+0000 7f04e8ff9640 1 -- 172.21.15.57:0/73284924 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f04f00128d0 con 0x556347a3bb40 2023-10-19T14:21:31.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.810+0000 7f04e8ff9640 1 -- 172.21.15.57:0/73284924 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04f00037e0 con 0x556347a3bb40 2023-10-19T14:21:31.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.810+0000 7f04ff1fb740 1 -- 172.21.15.57:0/73284924 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556347a3bb40 msgr2=0x556347a12830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:31.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.810+0000 7f04ff1fb740 1 --2- 172.21.15.57:0/73284924 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556347a3bb40 0x556347a12830 secure :-1 s=READY pgs=2711 cs=0 l=1 rev1=1 crypto rx=0x7f04f0012b50 tx=0x7f04f0014000 comp rx=0 tx=0).stop 2023-10-19T14:21:31.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.810+0000 7f04ff1fb740 1 -- 172.21.15.57:0/73284924 shutdown_connections 2023-10-19T14:21:31.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.810+0000 7f04ff1fb740 1 --2- 172.21.15.57:0/73284924 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556347a3d870 0x556347a72ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:31.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.810+0000 7f04ff1fb740 1 --2- 172.21.15.57:0/73284924 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x556347a3bf20 0x556347a72980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:31.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.810+0000 7f04ff1fb740 1 --2- 172.21.15.57:0/73284924 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556347a3bb40 0x556347a12830 unknown :-1 s=CLOSED pgs=2711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:31.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.810+0000 7f04ff1fb740 1 -- 172.21.15.57:0/73284924 >> 172.21.15.57:0/73284924 conn(0x556347a0f020 msgr2=0x556347a5c240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:31.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.810+0000 7f04ff1fb740 1 -- 172.21.15.57:0/73284924 shutdown_connections 2023-10-19T14:21:31.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.810+0000 7f04ff1fb740 1 -- 172.21.15.57:0/73284924 wait complete. 2023-10-19T14:21:31.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.811+0000 7f04ff1fb740 1 -- start start 2023-10-19T14:21:31.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.811+0000 7f04ff1fb740 1 Processor -- start 2023-10-19T14:21:31.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.812+0000 7f04ff1fb740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556347a3bb40 0x556347a3cc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:31.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.812+0000 7f04ff1fb740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556347a3bf20 0x556347a137e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:31.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.812+0000 7f04ff1fb740 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x556347a3d870 0x556347a5b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:31.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.812+0000 7f04ff1fb740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x556347a5aec0 con 0x556347a3bf20 2023-10-19T14:21:31.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.812+0000 7f04ff1fb740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x556347a144c0 con 0x556347a3bb40 2023-10-19T14:21:31.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.812+0000 7f04ff1fb740 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x556347a3e2d0 con 0x556347a3d870 2023-10-19T14:21:31.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.812+0000 7f04ff1fb740 10 client.0 _ll_register_callbacks cb 0x7f04f00015f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-19T14:21:31.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.812+0000 7f04f9b29640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556347a3bf20 0x556347a137e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:31.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.812+0000 7f04f9b29640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556347a3bf20 0x556347a137e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37308/0 (socket says 172.21.15.57:37308) 2023-10-19T14:21:31.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.812+0000 7f04f9b29640 1 -- 172.21.15.57:0/3984555419 learned_addr learned my addr 172.21.15.57:0/3984555419 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:31.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.812+0000 7f04fa32a640 1 --2- 172.21.15.57:0/3984555419 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556347a3bb40 0x556347a3cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:31.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.812+0000 7f04fab2b640 1 --2- 172.21.15.57:0/3984555419 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x556347a3d870 0x556347a5b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:31.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.812+0000 7f04f9b29640 1 -- 172.21.15.57:0/3984555419 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x556347a3d870 msgr2=0x556347a5b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T14:21:31.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.812+0000 7f04f9b29640 1 --2- 172.21.15.57:0/3984555419 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x556347a3d870 0x556347a5b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:31.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.812+0000 7f04f9b29640 1 -- 172.21.15.57:0/3984555419 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556347a3bb40 msgr2=0x556347a3cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T14:21:31.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.812+0000 7f04f9b29640 1 --2- 172.21.15.57:0/3984555419 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556347a3bb40 0x556347a3cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:31.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.812+0000 7f04f9b29640 1 -- 172.21.15.57:0/3984555419 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556347978de0 con 0x556347a3bf20 2023-10-19T14:21:31.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.812+0000 7f04f9b29640 1 --2- 172.21.15.57:0/3984555419 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556347a3bf20 0x556347a137e0 secure :-1 s=READY pgs=2712 cs=0 l=1 rev1=1 crypto rx=0x7f04e40aa9d0 tx=0x7f04e407b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4dabbf7ad6a2a0d3 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:31.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.812+0000 7f04d57fa640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2023-10-19T14:21:31.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.812+0000 7f04d57fa640 1 -- 172.21.15.57:0/3984555419 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04e4083470 con 0x556347a3bf20 2023-10-19T14:21:31.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.812+0000 7f04d57fa640 1 -- 172.21.15.57:0/3984555419 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(247..247 src has 1..247) v4 ==== 6921+0+0 (secure 0 0 0) 0x7f04e4077880 con 0x556347a3bf20 2023-10-19T14:21:31.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.812+0000 7f04ff1fb740 1 -- 172.21.15.57:0/3984555419 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55634767c6c0 con 0x556347a3bf20 2023-10-19T14:21:31.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.812+0000 7f04d57fa640 1 -- 172.21.15.57:0/3984555419 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=107) v2 ==== 24+0+0 (secure 0 0 0) 0x7f04e4077a40 con 0x556347a3bf20 2023-10-19T14:21:31.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.812+0000 7f04ff1fb740 10 client.12882 fetch_fsmap learned FSMap version 107 2023-10-19T14:21:31.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.812+0000 7f04ff1fb740 1 -- 172.21.15.57:0/3984555419 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=107,monmap=2+}) v3 -- 0x55634793d4f0 con 0x556347a3bf20 2023-10-19T14:21:31.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.813+0000 7f04d57fa640 1 -- 172.21.15.57:0/3984555419 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x55634767c6c0 con 0x556347a3bf20 2023-10-19T14:21:31.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.813+0000 7f04d57fa640 1 -- 172.21.15.57:0/3984555419 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 107) v1 ==== 3891+0+0 (secure 0 0 0) 0x7f04e403a040 con 0x556347a3bf20 2023-10-19T14:21:31.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.813+0000 7f04ff1fb740 10 client.12882 fetch_fsmap finished waiting for FSMap version 107 2023-10-19T14:21:31.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.813+0000 7f04ff1fb740 10 client.12882 resolve_mds: resolved name 'a' to daemon mds.a 2023-10-19T14:21:31.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.813+0000 7f04ff1fb740 20 client.12882 populate_metadata read hostname 'smithi057' 2023-10-19T14:21:31.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.813+0000 7f04ff1fb740 1 --2- 172.21.15.57:0/3984555419 >> [v2:172.21.15.57:6832/4256498777,v1:172.21.15.57:6833/4256498777] conn(0x556347ac4ed0 0x556347a5ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:31.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.813+0000 7f04ff1fb740 4 client.12882 mds_command: new command op to 12555 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-19T14:21:31.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.813+0000 7f04ff1fb740 1 -- 172.21.15.57:0/3984555419 --> [v2:172.21.15.57:6832/4256498777,v1:172.21.15.57:6833/4256498777] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5563476cfd10 con 0x556347ac4ed0 2023-10-19T14:21:31.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.813+0000 7f04fa32a640 1 --2- 172.21.15.57:0/3984555419 >> [v2:172.21.15.57:6832/4256498777,v1:172.21.15.57:6833/4256498777] conn(0x556347ac4ed0 0x556347a5ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:31.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.813+0000 7f04d57fa640 10 client.12882 ms_handle_connect on v2:172.21.15.57:6832/4256498777 2023-10-19T14:21:31.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.813+0000 7f04fa32a640 1 --2- 172.21.15.57:0/3984555419 >> [v2:172.21.15.57:6832/4256498777,v1:172.21.15.57:6833/4256498777] conn(0x556347ac4ed0 0x556347a5ba00 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c5e8ac8cc591219b server_cookie=106a8a0fc3726b1e in_seq=0 out_seq=0 2023-10-19T14:21:31.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.813+0000 7f04d57fa640 1 -- 172.21.15.57:0/3984555419 <== mds.0 v2:172.21.15.57:6832/4256498777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+338 (crc 0 0 0) 0x5563476cfd10 con 0x556347ac4ed0 2023-10-19T14:21:31.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.813+0000 7f04d57fa640 10 client.12882 handle_command_reply: tid=0 2023-10-19T14:21:31.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.813+0000 7f04ff1fb740 1 client.12882 shutdown 2023-10-19T14:21:31.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.813+0000 7f04ff1fb740 1 -- 172.21.15.57:0/3984555419 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556347a3bf20 msgr2=0x556347a137e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:31.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.813+0000 7f04ff1fb740 1 --2- 172.21.15.57:0/3984555419 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556347a3bf20 0x556347a137e0 secure :-1 s=READY pgs=2712 cs=0 l=1 rev1=1 crypto rx=0x7f04e40aa9d0 tx=0x7f04e407b9a0 comp rx=0 tx=0).stop 2023-10-19T14:21:31.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.814+0000 7f04ff1fb740 1 -- 172.21.15.57:0/3984555419 shutdown_connections 2023-10-19T14:21:31.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.814+0000 7f04ff1fb740 1 --2- 172.21.15.57:0/3984555419 >> [v2:172.21.15.57:6832/4256498777,v1:172.21.15.57:6833/4256498777] conn(0x556347ac4ed0 0x556347a5ba00 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:31.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.814+0000 7f04ff1fb740 1 --2- 172.21.15.57:0/3984555419 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x556347a3d870 0x556347a5b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:31.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.814+0000 7f04d57fa640 0 client.12882 ms_handle_reset on v2:172.21.15.57:6832/4256498777 2023-10-19T14:21:31.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.814+0000 7f04ff1fb740 1 --2- 172.21.15.57:0/3984555419 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556347a3bf20 0x556347a137e0 unknown :-1 s=CLOSED pgs=2712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:31.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.814+0000 7f04ff1fb740 1 --2- 172.21.15.57:0/3984555419 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556347a3bb40 0x556347a3cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:31.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.814+0000 7f04ff1fb740 1 -- 172.21.15.57:0/3984555419 >> 172.21.15.57:0/3984555419 conn(0x556347a0f020 msgr2=0x556347a421b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:31.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.814+0000 7f04ff1fb740 1 -- 172.21.15.57:0/3984555419 shutdown_connections 2023-10-19T14:21:31.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.814+0000 7f04ff1fb740 1 -- 172.21.15.57:0/3984555419 wait complete. 2023-10-19T14:21:31.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.814+0000 7f04ff1fb740 20 client.12882 trim_cache size 0 max 16384 2023-10-19T14:21:31.836 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2023-10-19T14:21:31.837 INFO:teuthology.orchestra.run.smithi057.stdout: "cluster_fsid": "1e9beb19-3260-4dd7-85ef-5b399ce774ac", 2023-10-19T14:21:31.837 INFO:teuthology.orchestra.run.smithi057.stdout: "whoami": 0, 2023-10-19T14:21:31.837 INFO:teuthology.orchestra.run.smithi057.stdout: "id": 12555, 2023-10-19T14:21:31.837 INFO:teuthology.orchestra.run.smithi057.stdout: "want_state": "up:active", 2023-10-19T14:21:31.837 INFO:teuthology.orchestra.run.smithi057.stdout: "state": "up:active", 2023-10-19T14:21:31.838 INFO:teuthology.orchestra.run.smithi057.stdout: "fs_name": "backup_fs", 2023-10-19T14:21:31.838 INFO:teuthology.orchestra.run.smithi057.stdout: "rank_uptime": 4.2939029079999997, 2023-10-19T14:21:31.838 INFO:teuthology.orchestra.run.smithi057.stdout: "mdsmap_epoch": 107, 2023-10-19T14:21:31.838 INFO:teuthology.orchestra.run.smithi057.stdout: "osdmap_epoch": 247, 2023-10-19T14:21:31.838 INFO:teuthology.orchestra.run.smithi057.stdout: "osdmap_epoch_barrier": 247, 2023-10-19T14:21:31.839 INFO:teuthology.orchestra.run.smithi057.stdout: "uptime": 26.111937258000001 2023-10-19T14:21:31.839 INFO:teuthology.orchestra.run.smithi057.stdout:} 2023-10-19T14:21:31.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.814+0000 7f04fc58e640 1 -- 172.21.15.57:0/3932263000 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f04d806b0b0 msgr2=0x7f04d806d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:31.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.814+0000 7f04fc58e640 1 --2- 172.21.15.57:0/3932263000 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f04d806b0b0 0x7f04d806d570 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f04f0001200 tx=0x7f04f0001110 comp rx=0 tx=0).stop 2023-10-19T14:21:31.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.814+0000 7f04fc58e640 1 -- 172.21.15.57:0/3932263000 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f04f40abf20 msgr2=0x7f04f41e37e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:31.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.814+0000 7f04fc58e640 1 --2- 172.21.15.57:0/3932263000 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f04f40abf20 0x7f04f41e37e0 secure :-1 s=READY pgs=2708 cs=0 l=1 rev1=1 crypto rx=0x7f04e402cb70 tx=0x7f04e4033c10 comp rx=0 tx=0).stop 2023-10-19T14:21:31.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.814+0000 7f04fc58e640 1 -- 172.21.15.57:0/3932263000 shutdown_connections 2023-10-19T14:21:31.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.814+0000 7f04fc58e640 1 --2- 172.21.15.57:0/3932263000 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f04d806b0b0 0x7f04d806d570 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:31.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.814+0000 7f04fc58e640 1 --2- 172.21.15.57:0/3932263000 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f04f40aa8c0 0x7f04f4145b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:31.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.814+0000 7f04fc58e640 1 --2- 172.21.15.57:0/3932263000 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f04f40abf20 0x7f04f41e37e0 unknown :-1 s=CLOSED pgs=2708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:31.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.814+0000 7f04fc58e640 1 --2- 172.21.15.57:0/3932263000 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f04e0003f20 0x7f04f414ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:31.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.814+0000 7f04fc58e640 1 -- 172.21.15.57:0/3932263000 >> 172.21.15.57:0/3932263000 conn(0x7f04f406fae0 msgr2=0x7f04f4151010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:31.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.814+0000 7f04fc58e640 1 -- 172.21.15.57:0/3932263000 shutdown_connections 2023-10-19T14:21:31.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.814+0000 7f04fc58e640 1 -- 172.21.15.57:0/3932263000 wait complete. 2023-10-19T14:21:31.843 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-19T14:21:31.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.950+0000 7effe6687640 1 -- 172.21.15.57:0/2014787754 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7effcc0039c0 msgr2=0x7effcc003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:31.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.950+0000 7effe6687640 1 --2- 172.21.15.57:0/2014787754 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7effcc0039c0 0x7effcc003de0 secure :-1 s=READY pgs=2714 cs=0 l=1 rev1=1 crypto rx=0x7effd0009b50 tx=0x7effcc0057f0 comp rx=0 tx=0).stop 2023-10-19T14:21:31.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.950+0000 7effe6687640 1 -- 172.21.15.57:0/2014787754 shutdown_connections 2023-10-19T14:21:31.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.950+0000 7effe6687640 1 --2- 172.21.15.57:0/2014787754 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7effcc0039c0 0x7effcc003de0 unknown :-1 s=CLOSED pgs=2714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:31.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.950+0000 7effe6687640 1 -- 172.21.15.57:0/2014787754 >> 172.21.15.57:0/2014787754 conn(0x7effe006a830 msgr2=0x7effe006ac40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:31.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.950+0000 7effe6687640 1 -- 172.21.15.57:0/2014787754 shutdown_connections 2023-10-19T14:21:31.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.950+0000 7effe6687640 1 -- 172.21.15.57:0/2014787754 wait complete. 2023-10-19T14:21:31.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.951+0000 7effe6687640 1 Processor -- start 2023-10-19T14:21:31.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.951+0000 7effe6687640 1 -- start start 2023-10-19T14:21:31.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.952+0000 7effe6687640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7effcc0039c0 0x7effe0158750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:31.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.952+0000 7effe6687640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7effe0159ef0 0x7effe014dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:31.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.952+0000 7effe6687640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7effe013e2d0 0x7effe014eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:31.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.952+0000 7effe6687640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effe014e4e0 con 0x7effe0159ef0 2023-10-19T14:21:31.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.952+0000 7effe6687640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effe0158e50 con 0x7effe013e2d0 2023-10-19T14:21:31.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.952+0000 7effe6687640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effe0159130 con 0x7effcc0039c0 2023-10-19T14:21:31.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.952+0000 7effdf7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7effe0159ef0 0x7effe014dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:31.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.952+0000 7effdf7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7effe0159ef0 0x7effe014dfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37330/0 (socket says 172.21.15.57:37330) 2023-10-19T14:21:31.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.952+0000 7effdf7fe640 1 -- 172.21.15.57:0/3446528134 learned_addr learned my addr 172.21.15.57:0/3446528134 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:31.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.952+0000 7effdffff640 1 --2- 172.21.15.57:0/3446528134 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7effcc0039c0 0x7effe0158750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:31.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.952+0000 7effe4c24640 1 --2- 172.21.15.57:0/3446528134 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7effe013e2d0 0x7effe014eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:31.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.953+0000 7effdf7fe640 1 -- 172.21.15.57:0/3446528134 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7effcc0039c0 msgr2=0x7effe0158750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:31.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.953+0000 7effdf7fe640 1 --2- 172.21.15.57:0/3446528134 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7effcc0039c0 0x7effe0158750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:31.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.953+0000 7effdf7fe640 1 -- 172.21.15.57:0/3446528134 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7effe013e2d0 msgr2=0x7effe014eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:31.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.953+0000 7effdf7fe640 1 --2- 172.21.15.57:0/3446528134 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7effe013e2d0 0x7effe014eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:31.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.953+0000 7effdf7fe640 1 -- 172.21.15.57:0/3446528134 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effe0159410 con 0x7effe0159ef0 2023-10-19T14:21:31.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.953+0000 7effdffff640 1 --2- 172.21.15.57:0/3446528134 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7effcc0039c0 0x7effe0158750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:21:31.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.953+0000 7effe4c24640 1 --2- 172.21.15.57:0/3446528134 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7effe013e2d0 0x7effe014eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:21:31.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.953+0000 7effdf7fe640 1 --2- 172.21.15.57:0/3446528134 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7effe0159ef0 0x7effe014dfa0 secure :-1 s=READY pgs=2715 cs=0 l=1 rev1=1 crypto rx=0x7effc8001c10 tx=0x7effc800e980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:31.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.953+0000 7effdcff9640 1 -- 172.21.15.57:0/3446528134 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effc8010890 con 0x7effe0159ef0 2023-10-19T14:21:31.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.953+0000 7effdcff9640 1 -- 172.21.15.57:0/3446528134 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7effc8010ae0 con 0x7effe0159ef0 2023-10-19T14:21:31.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.953+0000 7effe6687640 1 -- 172.21.15.57:0/3446528134 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effe01596f0 con 0x7effe0159ef0 2023-10-19T14:21:31.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.953+0000 7effdcff9640 1 -- 172.21.15.57:0/3446528134 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effc8011460 con 0x7effe0159ef0 2023-10-19T14:21:31.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.953+0000 7effe6687640 1 -- 172.21.15.57:0/3446528134 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effe013e750 con 0x7effe0159ef0 2023-10-19T14:21:31.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.954+0000 7effe6687640 1 -- 172.21.15.57:0/3446528134 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effe004dae0 con 0x7effe0159ef0 2023-10-19T14:21:31.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.954+0000 7effdcff9640 1 -- 172.21.15.57:0/3446528134 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7effc80115c0 con 0x7effe0159ef0 2023-10-19T14:21:31.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.955+0000 7effdcff9640 1 --2- 172.21.15.57:0/3446528134 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7effc006b080 0x7effc006d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:31.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.955+0000 7effdcff9640 1 -- 172.21.15.57:0/3446528134 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(247..247 src has 1..247) v4 ==== 6921+0+0 (secure 0 0 0) 0x7effc808a6f0 con 0x7effe0159ef0 2023-10-19T14:21:31.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.955+0000 7effdffff640 1 --2- 172.21.15.57:0/3446528134 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7effc006b080 0x7effc006d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:31.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.956+0000 7effdffff640 1 --2- 172.21.15.57:0/3446528134 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7effc006b080 0x7effc006d540 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7effd0036000 tx=0x7effd00046c0 comp rx=0 tx=0).ready entity=mgr.9530 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:31.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:31.959+0000 7effdcff9640 1 -- 172.21.15.57:0/3446528134 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7effc8054e70 con 0x7effe0159ef0 2023-10-19T14:21:32.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.157+0000 7effe6687640 1 -- 172.21.15.57:0/3446528134 --> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7effe0070eb0 con 0x7effc006b080 2023-10-19T14:21:32.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.159+0000 7effdcff9640 1 -- 172.21.15.57:0/3446528134 <== mgr.9530 v2:172.21.15.179:6832/3547895240 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+82 (secure 0 0 0) 0x7effd0041070 con 0x7effc006b080 2023-10-19T14:21:32.162 INFO:teuthology.orchestra.run.smithi057.stdout:[ 2023-10-19T14:21:32.162 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:21:32.162 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "cephfs" 2023-10-19T14:21:32.162 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:21:32.163 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:21:32.163 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "backup_fs" 2023-10-19T14:21:32.163 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:21:32.163 INFO:teuthology.orchestra.run.smithi057.stdout:] 2023-10-19T14:21:32.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.160+0000 7effe6687640 1 -- 172.21.15.57:0/3446528134 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7effc006b080 msgr2=0x7effc006d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:32.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.160+0000 7effe6687640 1 --2- 172.21.15.57:0/3446528134 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7effc006b080 0x7effc006d540 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7effd0036000 tx=0x7effd00046c0 comp rx=0 tx=0).stop 2023-10-19T14:21:32.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.160+0000 7effe6687640 1 -- 172.21.15.57:0/3446528134 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7effe0159ef0 msgr2=0x7effe014dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:32.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.160+0000 7effe6687640 1 --2- 172.21.15.57:0/3446528134 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7effe0159ef0 0x7effe014dfa0 secure :-1 s=READY pgs=2715 cs=0 l=1 rev1=1 crypto rx=0x7effc8001c10 tx=0x7effc800e980 comp rx=0 tx=0).stop 2023-10-19T14:21:32.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.160+0000 7effe6687640 1 -- 172.21.15.57:0/3446528134 shutdown_connections 2023-10-19T14:21:32.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.160+0000 7effe6687640 1 --2- 172.21.15.57:0/3446528134 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7effc006b080 0x7effc006d540 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:32.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.160+0000 7effe6687640 1 --2- 172.21.15.57:0/3446528134 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7effe013e2d0 0x7effe014eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:32.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.160+0000 7effe6687640 1 --2- 172.21.15.57:0/3446528134 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7effe0159ef0 0x7effe014dfa0 unknown :-1 s=CLOSED pgs=2715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:32.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.160+0000 7effe6687640 1 --2- 172.21.15.57:0/3446528134 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7effcc0039c0 0x7effe0158750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:32.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.161+0000 7effe6687640 1 -- 172.21.15.57:0/3446528134 >> 172.21.15.57:0/3446528134 conn(0x7effe006a830 msgr2=0x7effe00ae070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:32.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.161+0000 7effe6687640 1 -- 172.21.15.57:0/3446528134 shutdown_connections 2023-10-19T14:21:32.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.161+0000 7effe6687640 1 -- 172.21.15.57:0/3446528134 wait complete. 2023-10-19T14:21:32.174 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-19T14:21:32.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.283+0000 7f106f765640 1 -- 172.21.15.57:0/3779899712 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1068070af0 msgr2=0x7f10680affe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:32.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.283+0000 7f106f765640 1 --2- 172.21.15.57:0/3779899712 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1068070af0 0x7f10680affe0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f1060009830 tx=0x7f106002bcd0 comp rx=0 tx=0).stop 2023-10-19T14:21:32.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.283+0000 7f106f765640 1 -- 172.21.15.57:0/3779899712 shutdown_connections 2023-10-19T14:21:32.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.283+0000 7f106f765640 1 --2- 172.21.15.57:0/3779899712 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1068070af0 0x7f10680affe0 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:32.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.283+0000 7f106f765640 1 --2- 172.21.15.57:0/3779899712 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f106813e520 0x7f106806bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:32.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.283+0000 7f106f765640 1 -- 172.21.15.57:0/3779899712 >> 172.21.15.57:0/3779899712 conn(0x7f1068069930 msgr2=0x7f1068069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:32.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.283+0000 7f106f765640 1 -- 172.21.15.57:0/3779899712 shutdown_connections 2023-10-19T14:21:32.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.283+0000 7f106f765640 1 -- 172.21.15.57:0/3779899712 wait complete. 2023-10-19T14:21:32.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.284+0000 7f106f765640 1 Processor -- start 2023-10-19T14:21:32.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.284+0000 7f106f765640 1 -- start start 2023-10-19T14:21:32.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.285+0000 7f106f765640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1068070af0 0x7f1068131110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:32.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.285+0000 7f106f765640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f106813e520 0x7f10681e1560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:32.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.285+0000 7f106f765640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1068141690 0x7f10681e07e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:32.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.285+0000 7f106f765640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f106813cb10 con 0x7f1068070af0 2023-10-19T14:21:32.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.285+0000 7f106f765640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f106813bf00 con 0x7f106813e520 2023-10-19T14:21:32.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.285+0000 7f106f765640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f106813c1e0 con 0x7f1068141690 2023-10-19T14:21:32.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.285+0000 7f106d501640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1068070af0 0x7f1068131110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:32.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.285+0000 7f106d501640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1068070af0 0x7f1068131110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37336/0 (socket says 172.21.15.57:37336) 2023-10-19T14:21:32.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.285+0000 7f106d501640 1 -- 172.21.15.57:0/2999722098 learned_addr learned my addr 172.21.15.57:0/2999722098 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:32.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.285+0000 7f106dd02640 1 --2- 172.21.15.57:0/2999722098 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1068141690 0x7f10681e07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:32.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.285+0000 7f106cd00640 1 --2- 172.21.15.57:0/2999722098 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f106813e520 0x7f10681e1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:32.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.285+0000 7f106d501640 1 -- 172.21.15.57:0/2999722098 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1068141690 msgr2=0x7f10681e07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:32.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.285+0000 7f106d501640 1 --2- 172.21.15.57:0/2999722098 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1068141690 0x7f10681e07e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:32.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.285+0000 7f106d501640 1 -- 172.21.15.57:0/2999722098 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f106813e520 msgr2=0x7f10681e1560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:32.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.285+0000 7f106d501640 1 --2- 172.21.15.57:0/2999722098 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f106813e520 0x7f10681e1560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:32.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.285+0000 7f106d501640 1 -- 172.21.15.57:0/2999722098 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1060009490 con 0x7f1068070af0 2023-10-19T14:21:32.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.285+0000 7f106dd02640 1 --2- 172.21.15.57:0/2999722098 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1068141690 0x7f10681e07e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:21:32.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.286+0000 7f106d501640 1 --2- 172.21.15.57:0/2999722098 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1068070af0 0x7f1068131110 secure :-1 s=READY pgs=2716 cs=0 l=1 rev1=1 crypto rx=0x7f106400ebc0 tx=0x7f106400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:32.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.286+0000 7f105dffb640 1 -- 172.21.15.57:0/2999722098 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10640099b0 con 0x7f1068070af0 2023-10-19T14:21:32.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.286+0000 7f105dffb640 1 -- 172.21.15.57:0/2999722098 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f1064010720 con 0x7f1068070af0 2023-10-19T14:21:32.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.286+0000 7f106f765640 1 -- 172.21.15.57:0/2999722098 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f106813c4c0 con 0x7f1068070af0 2023-10-19T14:21:32.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.286+0000 7f105dffb640 1 -- 172.21.15.57:0/2999722098 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1064010b30 con 0x7f1068070af0 2023-10-19T14:21:32.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.286+0000 7f106f765640 1 -- 172.21.15.57:0/2999722098 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10681e1aa0 con 0x7f1068070af0 2023-10-19T14:21:32.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.287+0000 7f106f765640 1 -- 172.21.15.57:0/2999722098 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10681e1c90 con 0x7f1068070af0 2023-10-19T14:21:32.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.288+0000 7f105dffb640 1 -- 172.21.15.57:0/2999722098 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 137) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f1064010c90 con 0x7f1068070af0 2023-10-19T14:21:32.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.288+0000 7f105dffb640 1 --2- 172.21.15.57:0/2999722098 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f104c06b080 0x7f104c06d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:32.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.288+0000 7f105dffb640 1 -- 172.21.15.57:0/2999722098 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(247..247 src has 1..247) v4 ==== 6921+0+0 (secure 0 0 0) 0x7f1064014070 con 0x7f1068070af0 2023-10-19T14:21:32.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.288+0000 7f106cd00640 1 --2- 172.21.15.57:0/2999722098 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f104c06b080 0x7f104c06d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:32.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.289+0000 7f106cd00640 1 --2- 172.21.15.57:0/2999722098 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f104c06b080 0x7f104c06d540 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f1058007fd0 tx=0x7f1058009040 comp rx=0 tx=0).ready entity=mgr.9530 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:32.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.292+0000 7f105dffb640 1 -- 172.21.15.57:0/2999722098 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f1064051ca0 con 0x7f1068070af0 2023-10-19T14:21:32.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.514+0000 7f106f765640 1 -- 172.21.15.57:0/2999722098 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f10680690e0 con 0x7f1068070af0 2023-10-19T14:21:32.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.539+0000 7f105dffb640 1 -- 172.21.15.57:0/2999722098 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v138) v1 ==== 92+0+0 (secure 0 0 0) 0x7f106408e030 con 0x7f1068070af0 2023-10-19T14:21:32.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.542+0000 7f106f765640 1 -- 172.21.15.57:0/2999722098 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f104c06b080 msgr2=0x7f104c06d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:32.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.542+0000 7f106f765640 1 --2- 172.21.15.57:0/2999722098 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f104c06b080 0x7f104c06d540 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f1058007fd0 tx=0x7f1058009040 comp rx=0 tx=0).stop 2023-10-19T14:21:32.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.542+0000 7f106f765640 1 -- 172.21.15.57:0/2999722098 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1068070af0 msgr2=0x7f1068131110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:32.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.542+0000 7f106f765640 1 --2- 172.21.15.57:0/2999722098 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1068070af0 0x7f1068131110 secure :-1 s=READY pgs=2716 cs=0 l=1 rev1=1 crypto rx=0x7f106400ebc0 tx=0x7f106400c6a0 comp rx=0 tx=0).stop 2023-10-19T14:21:32.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.542+0000 7f106f765640 1 -- 172.21.15.57:0/2999722098 shutdown_connections 2023-10-19T14:21:32.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.542+0000 7f106f765640 1 --2- 172.21.15.57:0/2999722098 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7f104c06b080 0x7f104c06d540 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:32.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.542+0000 7f106f765640 1 --2- 172.21.15.57:0/2999722098 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f1068141690 0x7f10681e07e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:32.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.542+0000 7f106f765640 1 --2- 172.21.15.57:0/2999722098 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f106813e520 0x7f10681e1560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:32.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.542+0000 7f106f765640 1 --2- 172.21.15.57:0/2999722098 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1068070af0 0x7f1068131110 unknown :-1 s=CLOSED pgs=2716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:32.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.542+0000 7f106f765640 1 -- 172.21.15.57:0/2999722098 >> 172.21.15.57:0/2999722098 conn(0x7f1068069930 msgr2=0x7f106813f400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:32.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.542+0000 7f106f765640 1 -- 172.21.15.57:0/2999722098 shutdown_connections 2023-10-19T14:21:32.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.542+0000 7f106f765640 1 -- 172.21.15.57:0/2999722098 wait complete. 2023-10-19T14:21:32.557 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-19T14:21:32.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.670+0000 7fd4f92f2640 1 -- 172.21.15.57:0/942069015 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd4dc003970 msgr2=0x7fd4dc003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:32.674 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.670+0000 7fd4f92f2640 1 --2- 172.21.15.57:0/942069015 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd4dc003970 0x7fd4dc003d90 secure :-1 s=READY pgs=2720 cs=0 l=1 rev1=1 crypto rx=0x7fd4e800c290 tx=0x7fd4dc0057f0 comp rx=0 tx=0).stop 2023-10-19T14:21:32.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.670+0000 7fd4f37fe640 1 -- 172.21.15.57:0/942069015 reap_dead start 2023-10-19T14:21:32.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.670+0000 7fd4f92f2640 1 -- 172.21.15.57:0/942069015 shutdown_connections 2023-10-19T14:21:32.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.670+0000 7fd4f92f2640 1 -- 172.21.15.57:0/942069015 >> 172.21.15.57:0/942069015 conn(0x7fd4f40d0c50 msgr2=0x7fd4f40d1060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:32.675 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.671+0000 7fd4f92f2640 1 -- 172.21.15.57:0/942069015 shutdown_connections 2023-10-19T14:21:32.676 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.671+0000 7fd4f92f2640 1 -- 172.21.15.57:0/942069015 wait complete. 2023-10-19T14:21:32.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.672+0000 7fd4f92f2640 1 Processor -- start 2023-10-19T14:21:32.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.672+0000 7fd4f92f2640 1 -- start start 2023-10-19T14:21:32.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.673+0000 7fd4f92f2640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd4dc003970 0x7fd4f41e54b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:32.680 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.673+0000 7fd4f92f2640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd4f41e59f0 0x7fd4f4155fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:32.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.673+0000 7fd4f92f2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd4f4156500 0x7fd4f4145a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:32.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.673+0000 7fd4f92f2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4f4156980 con 0x7fd4f4156500 2023-10-19T14:21:32.681 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.673+0000 7fd4f92f2640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4f40c8a00 con 0x7fd4dc003970 2023-10-19T14:21:32.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.673+0000 7fd4f92f2640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4f41e5f00 con 0x7fd4f41e59f0 2023-10-19T14:21:32.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.673+0000 7fd4f37fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd4f4156500 0x7fd4f4145a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:32.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.673+0000 7fd4f37fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd4f4156500 0x7fd4f4145a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37368/0 (socket says 172.21.15.57:37368) 2023-10-19T14:21:32.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.673+0000 7fd4f37fe640 1 -- 172.21.15.57:0/686755258 learned_addr learned my addr 172.21.15.57:0/686755258 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:32.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.673+0000 7fd4f27fc640 1 --2- 172.21.15.57:0/686755258 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd4f41e59f0 0x7fd4f4155fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:32.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.673+0000 7fd4f2ffd640 1 --2- 172.21.15.57:0/686755258 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd4dc003970 0x7fd4f41e54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:32.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.673+0000 7fd4f37fe640 1 -- 172.21.15.57:0/686755258 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd4f41e59f0 msgr2=0x7fd4f4155fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:32.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.673+0000 7fd4f37fe640 1 --2- 172.21.15.57:0/686755258 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd4f41e59f0 0x7fd4f4155fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:32.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.673+0000 7fd4f37fe640 1 -- 172.21.15.57:0/686755258 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd4dc003970 msgr2=0x7fd4f41e54b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:32.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.673+0000 7fd4f37fe640 1 --2- 172.21.15.57:0/686755258 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd4dc003970 0x7fd4f41e54b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:32.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.673+0000 7fd4f37fe640 1 -- 172.21.15.57:0/686755258 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4f41e61e0 con 0x7fd4f4156500 2023-10-19T14:21:32.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.673+0000 7fd4f37fe640 1 --2- 172.21.15.57:0/686755258 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd4f4156500 0x7fd4f4145a80 secure :-1 s=READY pgs=2721 cs=0 l=1 rev1=1 crypto rx=0x7fd4f408a900 tx=0x7fd4ec00f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:32.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.674+0000 7fd4db7fe640 1 -- 172.21.15.57:0/686755258 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4ec003d80 con 0x7fd4f4156500 2023-10-19T14:21:32.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.674+0000 7fd4db7fe640 1 -- 172.21.15.57:0/686755258 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fd4ec00fe50 con 0x7fd4f4156500 2023-10-19T14:21:32.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.674+0000 7fd4f92f2640 1 -- 172.21.15.57:0/686755258 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4f40c8690 con 0x7fd4f4156500 2023-10-19T14:21:32.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.674+0000 7fd4db7fe640 1 -- 172.21.15.57:0/686755258 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4ec004d80 con 0x7fd4f4156500 2023-10-19T14:21:32.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.674+0000 7fd4f92f2640 1 -- 172.21.15.57:0/686755258 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4f4145fc0 con 0x7fd4f4156500 2023-10-19T14:21:32.686 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.675+0000 7fd4f92f2640 1 -- 172.21.15.57:0/686755258 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4f40c84d0 con 0x7fd4f4156500 2023-10-19T14:21:32.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.675+0000 7fd4db7fe640 1 -- 172.21.15.57:0/686755258 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 138) v1 ==== 89450+0+0 (secure 0 0 0) 0x7fd4ec004030 con 0x7fd4f4156500 2023-10-19T14:21:32.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.675+0000 7fd4db7fe640 1 --2- 172.21.15.57:0/686755258 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fd4e006b0d0 0x7fd4e006d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:32.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.676+0000 7fd4f2ffd640 1 -- 172.21.15.57:0/686755258 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fd4e006b0d0 msgr2=0x7fd4e006d590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/3547895240 2023-10-19T14:21:32.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.676+0000 7fd4db7fe640 1 -- 172.21.15.57:0/686755258 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(247..247 src has 1..247) v4 ==== 6921+0+0 (secure 0 0 0) 0x7fd4ec08aaa0 con 0x7fd4f4156500 2023-10-19T14:21:32.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.676+0000 7fd4f2ffd640 1 --2- 172.21.15.57:0/686755258 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fd4e006b0d0 0x7fd4e006d590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T14:21:32.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.684+0000 7fd4db7fe640 1 -- 172.21.15.57:0/686755258 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd4ec0551c0 con 0x7fd4f4156500 2023-10-19T14:21:32.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.864+0000 7fd4f92f2640 1 -- 172.21.15.57:0/686755258 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7fd4f415eb50 con 0x7fd4f4156500 2023-10-19T14:21:32.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.867+0000 7fd4db7fe640 1 -- 172.21.15.57:0/686755258 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v80)=0 v80) v1 ==== 138+0+0 (secure 0 0 0) 0x7fd4ec0587e0 con 0x7fd4f4156500 2023-10-19T14:21:32.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.869+0000 7fd4f92f2640 1 -- 172.21.15.57:0/686755258 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fd4e006b0d0 msgr2=0x7fd4e006d590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T14:21:32.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.869+0000 7fd4f92f2640 1 --2- 172.21.15.57:0/686755258 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fd4e006b0d0 0x7fd4e006d590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:32.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.869+0000 7fd4f92f2640 1 -- 172.21.15.57:0/686755258 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd4f4156500 msgr2=0x7fd4f4145a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:32.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.870+0000 7fd4f92f2640 1 --2- 172.21.15.57:0/686755258 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd4f4156500 0x7fd4f4145a80 secure :-1 s=READY pgs=2721 cs=0 l=1 rev1=1 crypto rx=0x7fd4f408a900 tx=0x7fd4ec00f790 comp rx=0 tx=0).stop 2023-10-19T14:21:32.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.870+0000 7fd4f92f2640 1 -- 172.21.15.57:0/686755258 shutdown_connections 2023-10-19T14:21:32.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.870+0000 7fd4f92f2640 1 --2- 172.21.15.57:0/686755258 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fd4e006b0d0 0x7fd4e006d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:32.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.870+0000 7fd4f92f2640 1 --2- 172.21.15.57:0/686755258 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd4f4156500 0x7fd4f4145a80 unknown :-1 s=CLOSED pgs=2721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:32.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.870+0000 7fd4f92f2640 1 --2- 172.21.15.57:0/686755258 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd4f41e59f0 0x7fd4f4155fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:32.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.870+0000 7fd4f92f2640 1 --2- 172.21.15.57:0/686755258 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd4dc003970 0x7fd4f41e54b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:32.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.870+0000 7fd4f92f2640 1 -- 172.21.15.57:0/686755258 >> 172.21.15.57:0/686755258 conn(0x7fd4f40d0c50 msgr2=0x7fd4f40b5160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:32.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.870+0000 7fd4f92f2640 1 -- 172.21.15.57:0/686755258 shutdown_connections 2023-10-19T14:21:32.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.870+0000 7fd4f92f2640 1 -- 172.21.15.57:0/686755258 wait complete. 2023-10-19T14:21:32.884 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-19T14:21:32.929 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:32.927+0000 7f1a4fe0f040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-19T14:21:32.946 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:21:32.943+0000 7f37e1a2d040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-19T14:21:32.974 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:32.971+0000 7f1a4fe0f040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-19T14:21:32.994 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:21:32.991+0000 7f37e1a2d040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-19T14:21:32.997 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.994+0000 7ff30a21a640 1 -- 172.21.15.57:0/2481066254 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff30415b2a0 msgr2=0x7ff3040cf720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:32.997 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.994+0000 7ff30a21a640 1 --2- 172.21.15.57:0/2481066254 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff30415b2a0 0x7ff3040cf720 secure :-1 s=READY pgs=2723 cs=0 l=1 rev1=1 crypto rx=0x7ff30408a810 tx=0x7ff2f802bcf0 comp rx=0 tx=0).stop 2023-10-19T14:21:32.998 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.994+0000 7ff30a21a640 1 -- 172.21.15.57:0/2481066254 shutdown_connections 2023-10-19T14:21:32.998 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.994+0000 7ff30a21a640 1 --2- 172.21.15.57:0/2481066254 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff30415b2a0 0x7ff3040cf720 unknown :-1 s=CLOSED pgs=2723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:32.998 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.994+0000 7ff30a21a640 1 --2- 172.21.15.57:0/2481066254 >> v2:172.21.15.179:3300/0 conn(0x7ff3040cf340 0x7ff3040d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:32.998 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.994+0000 7ff30a21a640 1 -- 172.21.15.57:0/2481066254 >> 172.21.15.57:0/2481066254 conn(0x7ff3040d0d60 msgr2=0x7ff3040d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:32.999 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.994+0000 7ff30a21a640 1 -- 172.21.15.57:0/2481066254 shutdown_connections 2023-10-19T14:21:32.999 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.994+0000 7ff30a21a640 1 -- 172.21.15.57:0/2481066254 wait complete. 2023-10-19T14:21:33.001 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.995+0000 7ff30a21a640 1 Processor -- start 2023-10-19T14:21:33.001 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.996+0000 7ff30a21a640 1 -- start start 2023-10-19T14:21:33.001 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.996+0000 7ff30a21a640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff3040ceb40 0x7ff3040cadc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:33.001 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.996+0000 7ff30a21a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff3040cf340 0x7ff304150380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:33.001 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.996+0000 7ff30a21a640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff30415b2a0 0x7ff3040b3db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:33.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.996+0000 7ff30a21a640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3041f1aa0 con 0x7ff3040cf340 2023-10-19T14:21:33.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.996+0000 7ff30a21a640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3040b4ea0 con 0x7ff30415b2a0 2023-10-19T14:21:33.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.996+0000 7ff30a21a640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3040b4600 con 0x7ff3040ceb40 2023-10-19T14:21:33.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.997+0000 7ff302ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff3040cf340 0x7ff304150380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:33.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.997+0000 7ff302ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff3040cf340 0x7ff304150380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37394/0 (socket says 172.21.15.57:37394) 2023-10-19T14:21:33.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.997+0000 7ff302ffd640 1 -- 172.21.15.57:0/819990076 learned_addr learned my addr 172.21.15.57:0/819990076 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:33.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.997+0000 7ff303fff640 1 --2- 172.21.15.57:0/819990076 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff30415b2a0 0x7ff3040b3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:33.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.997+0000 7ff3037fe640 1 --2- 172.21.15.57:0/819990076 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff3040ceb40 0x7ff3040cadc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:33.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.997+0000 7ff302ffd640 1 -- 172.21.15.57:0/819990076 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff3040ceb40 msgr2=0x7ff3040cadc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:33.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.997+0000 7ff302ffd640 1 --2- 172.21.15.57:0/819990076 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff3040ceb40 0x7ff3040cadc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:33.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.997+0000 7ff302ffd640 1 -- 172.21.15.57:0/819990076 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff30415b2a0 msgr2=0x7ff3040b3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:33.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.997+0000 7ff302ffd640 1 --2- 172.21.15.57:0/819990076 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff30415b2a0 0x7ff3040b3db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:33.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.997+0000 7ff302ffd640 1 -- 172.21.15.57:0/819990076 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2f8009490 con 0x7ff3040cf340 2023-10-19T14:21:33.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.997+0000 7ff302ffd640 1 --2- 172.21.15.57:0/819990076 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff3040cf340 0x7ff304150380 secure :-1 s=READY pgs=2724 cs=0 l=1 rev1=1 crypto rx=0x7ff2f8009780 tx=0x7ff2f80038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:33.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.997+0000 7ff2ebfff640 1 -- 172.21.15.57:0/819990076 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2f8003ea0 con 0x7ff3040cf340 2023-10-19T14:21:33.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.997+0000 7ff2ebfff640 1 -- 172.21.15.57:0/819990076 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7ff2f8005020 con 0x7ff3040cf340 2023-10-19T14:21:33.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.997+0000 7ff30a21a640 1 -- 172.21.15.57:0/819990076 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3040b48e0 con 0x7ff3040cf340 2023-10-19T14:21:33.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.997+0000 7ff2ebfff640 1 -- 172.21.15.57:0/819990076 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2f8005410 con 0x7ff3040cf340 2023-10-19T14:21:33.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.997+0000 7ff30a21a640 1 -- 172.21.15.57:0/819990076 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3040b42f0 con 0x7ff3040cf340 2023-10-19T14:21:33.006 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.998+0000 7ff30a21a640 1 -- 172.21.15.57:0/819990076 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3041f18b0 con 0x7ff3040cf340 2023-10-19T14:21:33.006 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.999+0000 7ff2ebfff640 1 -- 172.21.15.57:0/819990076 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 138) v1 ==== 89450+0+0 (secure 0 0 0) 0x7ff2f80047a0 con 0x7ff3040cf340 2023-10-19T14:21:33.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.999+0000 7ff2ebfff640 1 --2- 172.21.15.57:0/819990076 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7ff2f006b100 0x7ff2f006d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:33.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.999+0000 7ff2ebfff640 1 -- 172.21.15.57:0/819990076 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(247..247 src has 1..247) v4 ==== 6921+0+0 (secure 0 0 0) 0x7ff2f80ac240 con 0x7ff3040cf340 2023-10-19T14:21:33.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.999+0000 7ff3037fe640 1 -- 172.21.15.57:0/819990076 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7ff2f006b100 msgr2=0x7ff2f006d5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/3547895240 2023-10-19T14:21:33.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:32.999+0000 7ff3037fe640 1 --2- 172.21.15.57:0/819990076 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7ff2f006b100 0x7ff2f006d5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T14:21:33.010 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:33.008+0000 7ff2ebfff640 1 -- 172.21.15.57:0/819990076 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff2f8076b00 con 0x7ff3040cf340 2023-10-19T14:21:33.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:33.187+0000 7ff30a21a640 1 -- 172.21.15.57:0/819990076 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7ff3040cb980 con 0x7ff3040cf340 2023-10-19T14:21:33.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:33.192+0000 7ff2ebfff640 1 -- 172.21.15.57:0/819990076 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v81)=0 v81) v1 ==== 133+0+0 (secure 0 0 0) 0x7ff2f807a120 con 0x7ff3040cf340 2023-10-19T14:21:33.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:33.194+0000 7ff30a21a640 1 -- 172.21.15.57:0/819990076 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7ff2f006b100 msgr2=0x7ff2f006d5c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T14:21:33.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:33.194+0000 7ff30a21a640 1 --2- 172.21.15.57:0/819990076 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7ff2f006b100 0x7ff2f006d5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:33.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:33.194+0000 7ff30a21a640 1 -- 172.21.15.57:0/819990076 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff3040cf340 msgr2=0x7ff304150380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:33.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:33.194+0000 7ff30a21a640 1 --2- 172.21.15.57:0/819990076 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff3040cf340 0x7ff304150380 secure :-1 s=READY pgs=2724 cs=0 l=1 rev1=1 crypto rx=0x7ff2f8009780 tx=0x7ff2f80038a0 comp rx=0 tx=0).stop 2023-10-19T14:21:33.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:33.194+0000 7ff30a21a640 1 -- 172.21.15.57:0/819990076 shutdown_connections 2023-10-19T14:21:33.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:33.194+0000 7ff30a21a640 1 --2- 172.21.15.57:0/819990076 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7ff2f006b100 0x7ff2f006d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:33.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:33.194+0000 7ff30a21a640 1 --2- 172.21.15.57:0/819990076 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff30415b2a0 0x7ff3040b3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:33.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:33.194+0000 7ff30a21a640 1 --2- 172.21.15.57:0/819990076 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff3040cf340 0x7ff304150380 unknown :-1 s=CLOSED pgs=2724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:33.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:33.194+0000 7ff30a21a640 1 --2- 172.21.15.57:0/819990076 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff3040ceb40 0x7ff3040cadc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:33.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:33.194+0000 7ff30a21a640 1 -- 172.21.15.57:0/819990076 >> 172.21.15.57:0/819990076 conn(0x7ff3040d0d60 msgr2=0x7ff30406b490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:33.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:33.194+0000 7ff30a21a640 1 -- 172.21.15.57:0/819990076 shutdown_connections 2023-10-19T14:21:33.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:33.194+0000 7ff30a21a640 1 -- 172.21.15.57:0/819990076 wait complete. 2023-10-19T14:21:33.208 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:21:33.208 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p mandatory_fs_argument_test_dir) 2023-10-19T14:21:33.228 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add mandatory_fs_argument_test_dir 1M 2023-10-19T14:21:33.368 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:33.366+0000 7f1a4fe0f040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-19T14:21:33.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:33.378+0000 7fe880f4c640 1 -- 172.21.15.57:0/3178495910 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe87c15b2a0 msgr2=0x7fe87c0d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:33.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:33.378+0000 7fe880f4c640 1 --2- 172.21.15.57:0/3178495910 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe87c15b2a0 0x7fe87c0d1910 secure :-1 s=READY pgs=2726 cs=0 l=1 rev1=1 crypto rx=0x7fe86c009830 tx=0x7fe86c02bd60 comp rx=0 tx=0).stop 2023-10-19T14:21:33.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:33.378+0000 7fe880f4c640 1 -- 172.21.15.57:0/3178495910 shutdown_connections 2023-10-19T14:21:33.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:33.378+0000 7fe880f4c640 1 --2- 172.21.15.57:0/3178495910 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe87c15b2a0 0x7fe87c0d1910 unknown :-1 s=CLOSED pgs=2726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:33.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:33.378+0000 7fe880f4c640 1 -- 172.21.15.57:0/3178495910 >> 172.21.15.57:0/3178495910 conn(0x7fe87c0d0d60 msgr2=0x7fe87c0d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:33.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:33.378+0000 7fe880f4c640 1 -- 172.21.15.57:0/3178495910 shutdown_connections 2023-10-19T14:21:33.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:33.378+0000 7fe880f4c640 1 -- 172.21.15.57:0/3178495910 wait complete. 2023-10-19T14:21:33.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:33.379+0000 7fe880f4c640 1 Processor -- start 2023-10-19T14:21:33.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:33.380+0000 7fe880f4c640 1 -- start start 2023-10-19T14:21:33.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:33.380+0000 7fe880f4c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe87c0ceb40 0x7fe87c0c9520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:33.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:33.380+0000 7fe880f4c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe87c0cf340 0x7fe87c0c7e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:33.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:33.380+0000 7fe880f4c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe87c15b2a0 0x7fe87c158080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:33.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:33.380+0000 7fe880f4c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe87c157630 con 0x7fe87c0ceb40 2023-10-19T14:21:33.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:33.380+0000 7fe880f4c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe87c0c9dc0 con 0x7fe87c15b2a0 2023-10-19T14:21:33.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:33.380+0000 7fe880f4c640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe87c0ca0a0 con 0x7fe87c0cf340 2023-10-19T14:21:33.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:33.380+0000 7fe87a59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe87c0ceb40 0x7fe87c0c9520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:33.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:33.380+0000 7fe87a59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe87c0ceb40 0x7fe87c0c9520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37408/0 (socket says 172.21.15.57:37408) 2023-10-19T14:21:33.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:33.380+0000 7fe87a59c640 1 -- 172.21.15.57:0/174934418 learned_addr learned my addr 172.21.15.57:0/174934418 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:33.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:33.380+0000 7fe87ad9d640 1 --2- 172.21.15.57:0/174934418 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe87c15b2a0 0x7fe87c158080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:33.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:33.380+0000 7fe879d9b640 1 --2- 172.21.15.57:0/174934418 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe87c0cf340 0x7fe87c0c7e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:33.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:33.381+0000 7fe87a59c640 1 -- 172.21.15.57:0/174934418 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe87c0cf340 msgr2=0x7fe87c0c7e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:33.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:33.381+0000 7fe87a59c640 1 --2- 172.21.15.57:0/174934418 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe87c0cf340 0x7fe87c0c7e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:33.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:33.381+0000 7fe87a59c640 1 -- 172.21.15.57:0/174934418 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe87c15b2a0 msgr2=0x7fe87c158080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:33.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:33.381+0000 7fe87a59c640 1 --2- 172.21.15.57:0/174934418 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe87c15b2a0 0x7fe87c158080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:33.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:33.381+0000 7fe87a59c640 1 -- 172.21.15.57:0/174934418 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe86c009490 con 0x7fe87c0ceb40 2023-10-19T14:21:33.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:33.381+0000 7fe87a59c640 1 --2- 172.21.15.57:0/174934418 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe87c0ceb40 0x7fe87c0c9520 secure :-1 s=READY pgs=2727 cs=0 l=1 rev1=1 crypto rx=0x7fe87000c450 tx=0x7fe87000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:33.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:33.381+0000 7fe86affd640 1 -- 172.21.15.57:0/174934418 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8700052d0 con 0x7fe87c0ceb40 2023-10-19T14:21:33.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:33.381+0000 7fe86affd640 1 -- 172.21.15.57:0/174934418 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fe870005430 con 0x7fe87c0ceb40 2023-10-19T14:21:33.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:33.381+0000 7fe86affd640 1 -- 172.21.15.57:0/174934418 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe870012890 con 0x7fe87c0ceb40 2023-10-19T14:21:33.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:33.381+0000 7fe880f4c640 1 -- 172.21.15.57:0/174934418 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe87c157880 con 0x7fe87c0ceb40 2023-10-19T14:21:33.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:33.381+0000 7fe880f4c640 1 -- 172.21.15.57:0/174934418 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe87c0c9a60 con 0x7fe87c0ceb40 2023-10-19T14:21:33.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:33.382+0000 7fe880f4c640 1 -- 172.21.15.57:0/174934418 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe87c0af9d0 con 0x7fe87c0ceb40 2023-10-19T14:21:33.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:33.383+0000 7fe86affd640 1 -- 172.21.15.57:0/174934418 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 138) v1 ==== 89450+0+0 (secure 0 0 0) 0x7fe8700035f0 con 0x7fe87c0ceb40 2023-10-19T14:21:33.391 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:33.383+0000 7fe86affd640 1 --2- 172.21.15.57:0/174934418 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fe85c06b0d0 0x7fe85c06d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:33.392 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:33.383+0000 7fe879d9b640 1 -- 172.21.15.57:0/174934418 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fe85c06b0d0 msgr2=0x7fe85c06d590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/3547895240 2023-10-19T14:21:33.392 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:33.383+0000 7fe86affd640 1 -- 172.21.15.57:0/174934418 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(247..247 src has 1..247) v4 ==== 6921+0+0 (secure 0 0 0) 0x7fe870089a30 con 0x7fe87c0ceb40 2023-10-19T14:21:33.392 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:33.383+0000 7fe879d9b640 1 --2- 172.21.15.57:0/174934418 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fe85c06b0d0 0x7fe85c06d590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T14:21:33.392 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:33.387+0000 7fe86affd640 1 -- 172.21.15.57:0/174934418 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe870054400 con 0x7fe87c0ceb40 2023-10-19T14:21:33.414 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:33.411+0000 7f1a4fe0f040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-19T14:21:33.416 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:21:33.414+0000 7f37e1a2d040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-19T14:21:33.458 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:33.455+0000 7f1a4fe0f040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-19T14:21:33.464 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:21:33.461+0000 7f37e1a2d040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-19T14:21:33.509 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:33.506+0000 7f1a4fe0f040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-19T14:21:33.509 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:21:33.506+0000 7f37e1a2d040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-19T14:21:33.552 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:33.550+0000 7f1a4fe0f040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-19T14:21:33.564 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:21:33.561+0000 7f37e1a2d040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-19T14:21:33.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:33.584+0000 7fe879d9b640 1 -- 172.21.15.57:0/174934418 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fe85c06b0d0 msgr2=0x7fe85c06d590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/3547895240 2023-10-19T14:21:33.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:33.584+0000 7fe879d9b640 1 --2- 172.21.15.57:0/174934418 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fe85c06b0d0 0x7fe85c06d590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-19T14:21:33.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:33.588+0000 7fe880f4c640 1 -- 172.21.15.57:0/174934418 --> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "mandatory_fs_argument_test_dir", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fe87c0690e0 con 0x7fe85c06b0d0 2023-10-19T14:21:33.608 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:21:33.605+0000 7f37e1a2d040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-19T14:21:33.650 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:33.647+0000 7f1a4fe0f040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-19T14:21:33.702 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:33.700+0000 7f1a4fe0f040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-19T14:21:33.711 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:21:33.708+0000 7f37e1a2d040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-19T14:21:33.744 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:33.742+0000 7f1a4fe0f040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-19T14:21:33.767 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:21:33.764+0000 7f37e1a2d040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-19T14:21:33.791 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:33.789+0000 7f1a4fe0f040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-19T14:21:33.810 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:21:33.807+0000 7f37e1a2d040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-19T14:21:33.834 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:33.831+0000 7f1a4fe0f040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-19T14:21:33.861 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:21:33.858+0000 7f37e1a2d040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-19T14:21:33.906 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:21:33.903+0000 7f37e1a2d040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-19T14:21:33.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:33.985+0000 7fe879d9b640 1 -- 172.21.15.57:0/174934418 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fe85c06b0d0 msgr2=0x7fe85c06d590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/3547895240 2023-10-19T14:21:33.988 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:33.985+0000 7fe879d9b640 1 --2- 172.21.15.57:0/174934418 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fe85c06b0d0 0x7fe85c06d590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-19T14:21:34.150 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:34.147+0000 7f1a4fe0f040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-19T14:21:34.239 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:21:34.236+0000 7f37e1a2d040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-19T14:21:34.278 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:34.276+0000 7f1a4fe0f040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-19T14:21:34.372 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:34.369+0000 7f1a4fe0f040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-19T14:21:34.377 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:21:34.375+0000 7f37e1a2d040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-19T14:21:34.415 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:34.413+0000 7f1a4fe0f040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-19T14:21:34.476 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:21:34.473+0000 7f37e1a2d040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-19T14:21:34.513 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:34.510+0000 7f1a4fe0f040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-19T14:21:34.520 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:21:34.518+0000 7f37e1a2d040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-19T14:21:34.603 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:34.600+0000 7f1a4fe0f040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-19T14:21:34.620 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:21:34.617+0000 7f37e1a2d040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-19T14:21:34.715 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:21:34.712+0000 7f37e1a2d040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-19T14:21:34.737 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:34.735+0000 7f1a4fe0f040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-19T14:21:34.779 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:34.777+0000 7f1a4fe0f040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-19T14:21:34.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:34.786+0000 7fe879d9b640 1 -- 172.21.15.57:0/174934418 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fe85c06b0d0 msgr2=0x7fe85c06d590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/3547895240 2023-10-19T14:21:34.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:34.786+0000 7fe879d9b640 1 --2- 172.21.15.57:0/174934418 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fe85c06b0d0 0x7fe85c06d590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-19T14:21:34.858 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:21:34.855+0000 7f37e1a2d040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-19T14:21:34.884 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:34.882+0000 7f1a4fe0f040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-19T14:21:34.902 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:21:34.899+0000 7f37e1a2d040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-19T14:21:35.011 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:21:35.008+0000 7f37e1a2d040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-19T14:21:35.504 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:35.502+0000 7f1a4fe0f040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-19T14:21:35.662 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:21:35.660+0000 7f37e1a2d040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-19T14:21:35.769 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:35.766+0000 7f1a4fe0f040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-19T14:21:35.816 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:35.814+0000 7f1a4fe0f040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-19T14:21:35.915 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:35.913+0000 7f1a4fe0f040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-19T14:21:35.937 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:35.934+0000 7fe86affd640 1 -- 172.21.15.57:0/174934418 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mgrmap(e 139) v1 ==== 88957+0+0 (secure 0 0 0) 0x7fe8700547d0 con 0x7fe87c0ceb40 2023-10-19T14:21:35.937 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:35.934+0000 7fe86affd640 1 -- 172.21.15.57:0/174934418 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fe85c06b0d0 msgr2=0x7fe85c06d590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T14:21:35.937 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:35.934+0000 7fe86affd640 1 --2- 172.21.15.57:0/174934418 >> [v2:172.21.15.179:6832/3547895240,v1:172.21.15.179:6833/3547895240] conn(0x7fe85c06b0d0 0x7fe85c06d590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:35.939 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:21:35.936+0000 7f37e1a2d040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-19T14:21:35.994 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:21:35.991+0000 7f37e1a2d040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-19T14:21:36.089 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:36.086+0000 7f19cbc9a640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:21:36.089 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:36.086+0000 7f19cbc9a640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:21:36.089 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:36.086+0000 7f19cbc9a640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:21:36.090 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:36.086+0000 7f19cbc9a640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:21:36.090 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:36.086+0000 7f19cbc9a640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:21:36.090 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:36.086+0000 7f19ceca0640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:21:36.090 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:36.086+0000 7f19ceca0640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:21:36.090 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:36.086+0000 7f19ceca0640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:21:36.090 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:36.086+0000 7f19ceca0640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:21:36.091 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:36.086+0000 7f19ceca0640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:21:36.098 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:36.096+0000 7f19cbc9a640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:21:36.098 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:36.096+0000 7f19cbc9a640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:21:36.098 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:36.096+0000 7f19cbc9a640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:21:36.099 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:36.096+0000 7f19cbc9a640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:21:36.099 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:36.096+0000 7f19cbc9a640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:21:36.099 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:36.096+0000 7f19ceca0640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:21:36.100 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:36.096+0000 7f19ceca0640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:21:36.100 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:36.096+0000 7f19ceca0640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:21:36.100 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:36.096+0000 7f19ceca0640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:21:36.100 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:36.096+0000 7f19ceca0640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:21:36.107 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:21:36.105+0000 7f37e1a2d040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-19T14:21:36.386 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:21:36.389 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:21:36.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:36.938+0000 7fe86affd640 1 -- 172.21.15.57:0/174934418 <== mon.0 v2:172.21.15.57:3300/0 8 ==== mgrmap(e 140) v1 ==== 89084+0+0 (secure 0 0 0) 0x7fe87005e0f0 con 0x7fe87c0ceb40 2023-10-19T14:21:36.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:36.938+0000 7fe86affd640 1 --2- 172.21.15.57:0/174934418 >> [v2:172.21.15.179:6832/609071913,v1:172.21.15.179:6833/609071913] conn(0x7fe85c035690 0x7fe85c077bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:36.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:36.938+0000 7fe86affd640 1 -- 172.21.15.57:0/174934418 --> [v2:172.21.15.179:6832/609071913,v1:172.21.15.179:6833/609071913] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "mandatory_fs_argument_test_dir", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fe87c0690e0 con 0x7fe85c035690 2023-10-19T14:21:36.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:36.942+0000 7fe879d9b640 1 --2- 172.21.15.57:0/174934418 >> [v2:172.21.15.179:6832/609071913,v1:172.21.15.179:6833/609071913] conn(0x7fe85c035690 0x7fe85c077bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:36.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:36.943+0000 7fe879d9b640 1 --2- 172.21.15.57:0/174934418 >> [v2:172.21.15.179:6832/609071913,v1:172.21.15.179:6833/609071913] conn(0x7fe85c035690 0x7fe85c077bb0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe86c036000 tx=0x7fe86c004730 comp rx=0 tx=0).ready entity=mgr.9950 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:36.947 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:36.945+0000 7f19d5cee640 -1 mgr.server reply reply (22) Invalid argument filesystem argument is required when there is more than one file system 2023-10-19T14:21:36.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:36.945+0000 7fe86affd640 1 -- 172.21.15.57:0/174934418 <== mgr.9950 v2:172.21.15.179:6832/609071913 1 ==== mgr_command_reply(tid 0: -22 filesystem argument is required when there is more than one file system) v1 ==== 79+0+0 (secure 0 0 0) 0x7fe87c0690e0 con 0x7fe85c035690 2023-10-19T14:21:36.948 INFO:teuthology.orchestra.run.smithi057.stderr:Error EINVAL: filesystem argument is required when there is more than one file system 2023-10-19T14:21:36.954 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-10-19T14:21:36.955 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:21:36.956 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir mandatory_fs_argument_test_dir) 2023-10-19T14:21:36.976 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-19T14:21:37.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:37.133+0000 7fc004e64640 1 -- 172.21.15.57:0/1251525324 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc000141690 msgr2=0x7fc0000703e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:37.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:37.133+0000 7fc004e64640 1 --2- 172.21.15.57:0/1251525324 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc000141690 0x7fc0000703e0 secure :-1 s=READY pgs=2729 cs=0 l=1 rev1=1 crypto rx=0x7fbfe8005fd0 tx=0x7fbfe802bcf0 comp rx=0 tx=0).stop 2023-10-19T14:21:37.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:37.133+0000 7fc004e64640 1 -- 172.21.15.57:0/1251525324 shutdown_connections 2023-10-19T14:21:37.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:37.133+0000 7fc004e64640 1 --2- 172.21.15.57:0/1251525324 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc000070920 0x7fc0000affe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:37.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:37.133+0000 7fc004e64640 1 --2- 172.21.15.57:0/1251525324 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc000141690 0x7fc0000703e0 unknown :-1 s=CLOSED pgs=2729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:37.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:37.133+0000 7fc004e64640 1 -- 172.21.15.57:0/1251525324 >> 172.21.15.57:0/1251525324 conn(0x7fc000069930 msgr2=0x7fc000069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:37.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:37.133+0000 7fc004e64640 1 -- 172.21.15.57:0/1251525324 shutdown_connections 2023-10-19T14:21:37.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:37.133+0000 7fc004e64640 1 -- 172.21.15.57:0/1251525324 wait complete. 2023-10-19T14:21:37.139 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:37.134+0000 7fc004e64640 1 Processor -- start 2023-10-19T14:21:37.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:37.134+0000 7fc004e64640 1 -- start start 2023-10-19T14:21:37.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:37.135+0000 7fc004e64640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc000070920 0x7fc00013b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:37.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:37.135+0000 7fc004e64640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc00013e520 0x7fc0001ceea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:37.140 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:37.135+0000 7fc004e64640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc000141690 0x7fc00013d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:37.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:37.135+0000 7fc004e64640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc00013b760 con 0x7fc00013e520 2023-10-19T14:21:37.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:37.135+0000 7fc004e64640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0001cf8f0 con 0x7fc000141690 2023-10-19T14:21:37.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:37.135+0000 7fc004e64640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0001cfbd0 con 0x7fc000070920 2023-10-19T14:21:37.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:37.135+0000 7fbff7fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc00013e520 0x7fc0001ceea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:37.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:37.135+0000 7fbff7fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc00013e520 0x7fc0001ceea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43842/0 (socket says 172.21.15.57:43842) 2023-10-19T14:21:37.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:37.135+0000 7fbff7fff640 1 -- 172.21.15.57:0/209999936 learned_addr learned my addr 172.21.15.57:0/209999936 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:37.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:37.135+0000 7fbffe59c640 1 --2- 172.21.15.57:0/209999936 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc000070920 0x7fc00013b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:37.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:37.135+0000 7fbff7fff640 1 -- 172.21.15.57:0/209999936 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc000070920 msgr2=0x7fc00013b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:37.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:37.135+0000 7fbffed9d640 1 --2- 172.21.15.57:0/209999936 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc000141690 0x7fc00013d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:37.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:37.135+0000 7fbff7fff640 1 --2- 172.21.15.57:0/209999936 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc000070920 0x7fc00013b220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:37.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:37.135+0000 7fbff7fff640 1 -- 172.21.15.57:0/209999936 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc000141690 msgr2=0x7fc00013d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:37.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:37.135+0000 7fbff7fff640 1 --2- 172.21.15.57:0/209999936 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc000141690 0x7fc00013d3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:37.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:37.135+0000 7fbff7fff640 1 -- 172.21.15.57:0/209999936 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfe8009560 con 0x7fc00013e520 2023-10-19T14:21:37.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:37.136+0000 7fbff7fff640 1 --2- 172.21.15.57:0/209999936 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc00013e520 0x7fc0001ceea0 secure :-1 s=READY pgs=2730 cs=0 l=1 rev1=1 crypto rx=0x7fbfe8009850 tx=0x7fbfe802d9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:37.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:37.136+0000 7fbff6ffd640 1 -- 172.21.15.57:0/209999936 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfe802ddc0 con 0x7fc00013e520 2023-10-19T14:21:37.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:37.136+0000 7fbff6ffd640 1 -- 172.21.15.57:0/209999936 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fbfe80036e0 con 0x7fc00013e520 2023-10-19T14:21:37.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:37.136+0000 7fc004e64640 1 -- 172.21.15.57:0/209999936 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0001cfeb0 con 0x7fc00013e520 2023-10-19T14:21:37.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:37.136+0000 7fbff6ffd640 1 -- 172.21.15.57:0/209999936 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfe8003a90 con 0x7fc00013e520 2023-10-19T14:21:37.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:37.136+0000 7fc004e64640 1 -- 172.21.15.57:0/209999936 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0001d0470 con 0x7fc00013e520 2023-10-19T14:21:37.144 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:37.137+0000 7fc004e64640 1 -- 172.21.15.57:0/209999936 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc00013e0d0 con 0x7fc00013e520 2023-10-19T14:21:37.145 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:37.137+0000 7fbff6ffd640 1 -- 172.21.15.57:0/209999936 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 140) v1 ==== 89084+0+0 (secure 0 0 0) 0x7fbfe8003bf0 con 0x7fc00013e520 2023-10-19T14:21:37.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:37.138+0000 7fbff6ffd640 1 --2- 172.21.15.57:0/209999936 >> [v2:172.21.15.179:6832/609071913,v1:172.21.15.179:6833/609071913] conn(0x7fbfe006ace0 0x7fbfe006d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:37.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:37.138+0000 7fbff6ffd640 1 -- 172.21.15.57:0/209999936 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 7351+0+0 (secure 0 0 0) 0x7fbfe80ac530 con 0x7fc00013e520 2023-10-19T14:21:37.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:37.138+0000 7fbffe59c640 1 --2- 172.21.15.57:0/209999936 >> [v2:172.21.15.179:6832/609071913,v1:172.21.15.179:6833/609071913] conn(0x7fbfe006ace0 0x7fbfe006d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:37.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:37.139+0000 7fbffe59c640 1 --2- 172.21.15.57:0/209999936 >> [v2:172.21.15.179:6832/609071913,v1:172.21.15.179:6833/609071913] conn(0x7fbfe006ace0 0x7fbfe006d1a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc00006fee0 tx=0x7fbff0009040 comp rx=0 tx=0).ready entity=mgr.9950 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:37.146 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:37.142+0000 7fbff6ffd640 1 -- 172.21.15.57:0/209999936 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fbfe8077280 con 0x7fc00013e520 2023-10-19T14:21:37.369 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:37.367+0000 7fc004e64640 1 -- 172.21.15.57:0/209999936 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7fc0000af770 con 0x7fc00013e520 2023-10-19T14:21:37.989 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:37.987+0000 7fbff6ffd640 1 -- 172.21.15.57:0/209999936 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v141) v1 ==== 93+0+0 (secure 0 0 0) 0x7fbfe807a8a0 con 0x7fc00013e520 2023-10-19T14:21:37.993 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:37.991+0000 7fc004e64640 1 -- 172.21.15.57:0/209999936 >> [v2:172.21.15.179:6832/609071913,v1:172.21.15.179:6833/609071913] conn(0x7fbfe006ace0 msgr2=0x7fbfe006d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:37.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:37.991+0000 7fc004e64640 1 --2- 172.21.15.57:0/209999936 >> [v2:172.21.15.179:6832/609071913,v1:172.21.15.179:6833/609071913] conn(0x7fbfe006ace0 0x7fbfe006d1a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc00006fee0 tx=0x7fbff0009040 comp rx=0 tx=0).stop 2023-10-19T14:21:37.994 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:37.992+0000 7fc004e64640 1 -- 172.21.15.57:0/209999936 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc00013e520 msgr2=0x7fc0001ceea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:37.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:37.992+0000 7fbff6ffd640 1 -- 172.21.15.57:0/209999936 <== mon.0 v2:172.21.15.57:3300/0 8 ==== mgrmap(e 141) v1 ==== 89434+0+0 (secure 0 0 0) 0x7fbfe8077650 con 0x7fc00013e520 2023-10-19T14:21:37.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:37.992+0000 7fc004e64640 1 --2- 172.21.15.57:0/209999936 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc00013e520 0x7fc0001ceea0 secure :-1 s=READY pgs=2730 cs=0 l=1 rev1=1 crypto rx=0x7fbfe8009850 tx=0x7fbfe802d9c0 comp rx=0 tx=0).stop 2023-10-19T14:21:37.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:37.992+0000 7fc004e64640 1 -- 172.21.15.57:0/209999936 shutdown_connections 2023-10-19T14:21:37.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:37.992+0000 7fc004e64640 1 --2- 172.21.15.57:0/209999936 >> [v2:172.21.15.179:6832/609071913,v1:172.21.15.179:6833/609071913] conn(0x7fbfe006ace0 0x7fbfe006d1a0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:37.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:37.992+0000 7fc004e64640 1 --2- 172.21.15.57:0/209999936 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc000141690 0x7fc00013d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:37.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:37.992+0000 7fc004e64640 1 --2- 172.21.15.57:0/209999936 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc00013e520 0x7fc0001ceea0 unknown :-1 s=CLOSED pgs=2730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:37.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:37.992+0000 7fc004e64640 1 --2- 172.21.15.57:0/209999936 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc000070920 0x7fc00013b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:37.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:37.992+0000 7fc004e64640 1 -- 172.21.15.57:0/209999936 >> 172.21.15.57:0/209999936 conn(0x7fc000069930 msgr2=0x7fc000069d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:37.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:37.993+0000 7fc004e64640 1 -- 172.21.15.57:0/209999936 shutdown_connections 2023-10-19T14:21:37.997 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:37.993+0000 7fc004e64640 1 -- 172.21.15.57:0/209999936 wait complete. 2023-10-19T14:21:38.009 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-19T14:21:38.009 DEBUG:teuthology.orchestra.run.smithi057:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-19T14:21:38.070 DEBUG:teuthology.orchestra.run.smithi179:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-19T14:21:38.109 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-19T14:21:38.110 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T14:21:38.110 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T14:21:38.126 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T14:21:38.278 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.276+0000 7f642cf05640 1 -- 172.21.15.57:0/444479896 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6410004090 msgr2=0x7f64100047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:38.278 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.276+0000 7f642cf05640 1 --2- 172.21.15.57:0/444479896 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6410004090 0x7f64100047c0 secure :-1 s=READY pgs=2735 cs=0 l=1 rev1=1 crypto rx=0x7f6418006f60 tx=0x7f64100057f0 comp rx=0 tx=0).stop 2023-10-19T14:21:38.279 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.276+0000 7f6426d9d640 1 -- 172.21.15.57:0/444479896 reap_dead start 2023-10-19T14:21:38.279 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.276+0000 7f642cf05640 1 -- 172.21.15.57:0/444479896 shutdown_connections 2023-10-19T14:21:38.279 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.276+0000 7f642cf05640 1 -- 172.21.15.57:0/444479896 >> 172.21.15.57:0/444479896 conn(0x7f64280d0d60 msgr2=0x7f64280d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:38.279 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.276+0000 7f642cf05640 1 -- 172.21.15.57:0/444479896 shutdown_connections 2023-10-19T14:21:38.279 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.276+0000 7f642cf05640 1 -- 172.21.15.57:0/444479896 wait complete. 2023-10-19T14:21:38.282 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.277+0000 7f642cf05640 1 Processor -- start 2023-10-19T14:21:38.282 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.277+0000 7f642cf05640 1 -- start start 2023-10-19T14:21:38.282 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.278+0000 7f642cf05640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6410004090 0x7f642815b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:38.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.278+0000 7f642cf05640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f64280c7270 0x7f6428144c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:38.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.278+0000 7f642cf05640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f64280c9be0 0x7f64280ca090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:38.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.278+0000 7f642cf05640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64280c97b0 con 0x7f6410004090 2023-10-19T14:21:38.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.278+0000 7f642cf05640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6428145150 con 0x7f64280c9be0 2023-10-19T14:21:38.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.278+0000 7f642cf05640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f642815e630 con 0x7f64280c7270 2023-10-19T14:21:38.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.278+0000 7f642659c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6410004090 0x7f642815b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:38.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.278+0000 7f642659c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6410004090 0x7f642815b6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43878/0 (socket says 172.21.15.57:43878) 2023-10-19T14:21:38.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.278+0000 7f642659c640 1 -- 172.21.15.57:0/3205469230 learned_addr learned my addr 172.21.15.57:0/3205469230 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:38.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.278+0000 7f6426d9d640 1 --2- 172.21.15.57:0/3205469230 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f64280c9be0 0x7f64280ca090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:38.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.278+0000 7f6425d9b640 1 --2- 172.21.15.57:0/3205469230 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f64280c7270 0x7f6428144c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:38.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.278+0000 7f642659c640 1 -- 172.21.15.57:0/3205469230 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f64280c7270 msgr2=0x7f6428144c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:38.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.278+0000 7f642659c640 1 --2- 172.21.15.57:0/3205469230 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f64280c7270 0x7f6428144c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:38.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.278+0000 7f642659c640 1 -- 172.21.15.57:0/3205469230 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f64280c9be0 msgr2=0x7f64280ca090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:38.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.278+0000 7f642659c640 1 --2- 172.21.15.57:0/3205469230 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f64280c9be0 0x7f64280ca090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:38.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.278+0000 7f642659c640 1 -- 172.21.15.57:0/3205469230 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64280c76f0 con 0x7f6410004090 2023-10-19T14:21:38.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.278+0000 7f642659c640 1 --2- 172.21.15.57:0/3205469230 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6410004090 0x7f642815b6d0 secure :-1 s=READY pgs=2736 cs=0 l=1 rev1=1 crypto rx=0x7f641c003e40 tx=0x7f641c001770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:38.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.279+0000 7f6416ffd640 1 -- 172.21.15.57:0/3205469230 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f641c011840 con 0x7f6410004090 2023-10-19T14:21:38.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.279+0000 7f6416ffd640 1 -- 172.21.15.57:0/3205469230 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f641c0119a0 con 0x7f6410004090 2023-10-19T14:21:38.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.279+0000 7f6416ffd640 1 -- 172.21.15.57:0/3205469230 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f641c011db0 con 0x7f6410004090 2023-10-19T14:21:38.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.279+0000 7f642cf05640 1 -- 172.21.15.57:0/3205469230 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6428156550 con 0x7f6410004090 2023-10-19T14:21:38.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.279+0000 7f642cf05640 1 -- 172.21.15.57:0/3205469230 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f642815e350 con 0x7f6410004090 2023-10-19T14:21:38.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.280+0000 7f642cf05640 1 -- 172.21.15.57:0/3205469230 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64280c9900 con 0x7f6410004090 2023-10-19T14:21:38.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.280+0000 7f6416ffd640 1 -- 172.21.15.57:0/3205469230 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 141) v1 ==== 89434+0+0 (secure 0 0 0) 0x7f641c0103c0 con 0x7f6410004090 2023-10-19T14:21:38.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.281+0000 7f6416ffd640 1 --2- 172.21.15.57:0/3205469230 >> [v2:172.21.15.179:6832/609071913,v1:172.21.15.179:6833/609071913] conn(0x7f640806b080 0x7f640806d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:38.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.281+0000 7f6425d9b640 1 -- 172.21.15.57:0/3205469230 >> [v2:172.21.15.179:6832/609071913,v1:172.21.15.179:6833/609071913] conn(0x7f640806b080 msgr2=0x7f640806d540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/609071913 2023-10-19T14:21:38.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.281+0000 7f6416ffd640 1 -- 172.21.15.57:0/3205469230 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 7351+0+0 (secure 0 0 0) 0x7f641c08b0c0 con 0x7f6410004090 2023-10-19T14:21:38.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.281+0000 7f6425d9b640 1 --2- 172.21.15.57:0/3205469230 >> [v2:172.21.15.179:6832/609071913,v1:172.21.15.179:6833/609071913] conn(0x7f640806b080 0x7f640806d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T14:21:38.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.289+0000 7f6416ffd640 1 -- 172.21.15.57:0/3205469230 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f64280c9900 con 0x7f6410004090 2023-10-19T14:21:38.382 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:38.379+0000 7fa0a5617040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-19T14:21:38.401 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:21:38.399+0000 7f0f9ee3e040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-19T14:21:38.428 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:38.426+0000 7fa0a5617040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-19T14:21:38.448 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:21:38.447+0000 7f0f9ee3e040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-19T14:21:38.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.467+0000 7f642cf05640 1 -- 172.21.15.57:0/3205469230 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f64280c9900 con 0x7f6410004090 2023-10-19T14:21:38.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.469+0000 7f6416ffd640 1 -- 172.21.15.57:0/3205469230 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v248) v1 ==== 74+0+21132 (secure 0 0 0) 0x7f64280c9900 con 0x7f6410004090 2023-10-19T14:21:38.471 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:21:38.471 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":248,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T14:21:35.927462+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2023-10-19T14:21:08.101023+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"242","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2023-10-19T14:21:08.745817+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"243","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_metadata","create_time":"2023-10-19T14:21:24.384693+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"246","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"backup_fs_data","create_time":"2023-10-19T14:21:25.717524+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"247","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.179:0/2525508070":"2023-10-20T14:21:35.927420+0000","172.21.15.179:0/720769249":"2023-10-20T14:21:35.927420+0000","172.21.15.179:6833/3547895240":"2023-10-20T14:21:35.927420+0000","172.21.15.179:6832/3547895240":"2023-10-20T14:21:35.927420+0000","172.21.15.179:0/177563067":"2023-10-20T14:21:35.927420+0000","172.21.15.179:0/1877726906":"2023-10-20T14:21:35.927420+0000","172.21.15.179:0/3471744208":"2023-10-20T14:21:35.927420+0000","172.21.15.179:0/2477290155":"2023-10-20T14:21:35.927420+0000","172.21.15.179:0/1240052109":"2023-10-20T14:21:35.927420+0000","172.21.15.179:0/4042925688":"2023-10-20T14:21:35.927420+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T14:21:38.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.471+0000 7f642cf05640 1 -- 172.21.15.57:0/3205469230 >> [v2:172.21.15.179:6832/609071913,v1:172.21.15.179:6833/609071913] conn(0x7f640806b080 msgr2=0x7f640806d540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T14:21:38.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.471+0000 7f642cf05640 1 --2- 172.21.15.57:0/3205469230 >> [v2:172.21.15.179:6832/609071913,v1:172.21.15.179:6833/609071913] conn(0x7f640806b080 0x7f640806d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:38.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.471+0000 7f642cf05640 1 -- 172.21.15.57:0/3205469230 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6410004090 msgr2=0x7f642815b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:38.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.471+0000 7f642cf05640 1 --2- 172.21.15.57:0/3205469230 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6410004090 0x7f642815b6d0 secure :-1 s=READY pgs=2736 cs=0 l=1 rev1=1 crypto rx=0x7f641c003e40 tx=0x7f641c001770 comp rx=0 tx=0).stop 2023-10-19T14:21:38.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.471+0000 7f642cf05640 1 -- 172.21.15.57:0/3205469230 shutdown_connections 2023-10-19T14:21:38.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.471+0000 7f642cf05640 1 --2- 172.21.15.57:0/3205469230 >> [v2:172.21.15.179:6832/609071913,v1:172.21.15.179:6833/609071913] conn(0x7f640806b080 0x7f640806d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:38.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.471+0000 7f642cf05640 1 --2- 172.21.15.57:0/3205469230 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f64280c9be0 0x7f64280ca090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:38.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.471+0000 7f642cf05640 1 --2- 172.21.15.57:0/3205469230 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f64280c7270 0x7f6428144c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:38.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.471+0000 7f642cf05640 1 --2- 172.21.15.57:0/3205469230 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6410004090 0x7f642815b6d0 unknown :-1 s=CLOSED pgs=2736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:38.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.471+0000 7f642cf05640 1 -- 172.21.15.57:0/3205469230 >> 172.21.15.57:0/3205469230 conn(0x7f64280d0d60 msgr2=0x7f64280aed40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:38.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.471+0000 7f642cf05640 1 -- 172.21.15.57:0/3205469230 shutdown_connections 2023-10-19T14:21:38.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.471+0000 7f642cf05640 1 -- 172.21.15.57:0/3205469230 wait complete. 2023-10-19T14:21:38.485 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-19T12:44:07.371186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 31, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-19T14:21:08.101023+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '242', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 32, 'pool_name': 'cephfs_data', 'create_time': '2023-10-19T14:21:08.745817+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '243', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 33, 'pool_name': 'backup_fs_metadata', 'create_time': '2023-10-19T14:21:24.384693+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '246', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 34, 'pool_name': 'backup_fs_data', 'create_time': '2023-10-19T14:21:25.717524+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '247', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-19T14:21:38.486 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-19T14:21:38.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.598+0000 7fafd19c9640 1 -- 172.21.15.57:0/491102240 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fafcc15b000 msgr2=0x7fafcc0d1810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:38.600 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.598+0000 7fafd19c9640 1 --2- 172.21.15.57:0/491102240 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fafcc15b000 0x7fafcc0d1810 secure :-1 s=READY pgs=2737 cs=0 l=1 rev1=1 crypto rx=0x7fafbc005fd0 tx=0x7fafbc02be10 comp rx=0 tx=0).stop 2023-10-19T14:21:38.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.598+0000 7fafd19c9640 1 -- 172.21.15.57:0/491102240 shutdown_connections 2023-10-19T14:21:38.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.598+0000 7fafd19c9640 1 --2- 172.21.15.57:0/491102240 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fafcc0cf9e0 0x7fafcc0b0c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:38.601 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.598+0000 7fafd19c9640 1 --2- 172.21.15.57:0/491102240 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fafcc15b000 0x7fafcc0d1810 unknown :-1 s=CLOSED pgs=2737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:38.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.598+0000 7fafd19c9640 1 -- 172.21.15.57:0/491102240 >> 172.21.15.57:0/491102240 conn(0x7fafcc0d0cf0 msgr2=0x7fafcc0d1100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:38.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.598+0000 7fafd19c9640 1 -- 172.21.15.57:0/491102240 shutdown_connections 2023-10-19T14:21:38.602 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.598+0000 7fafd19c9640 1 -- 172.21.15.57:0/491102240 wait complete. 2023-10-19T14:21:38.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.599+0000 7fafd19c9640 1 Processor -- start 2023-10-19T14:21:38.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.599+0000 7fafd19c9640 1 -- start start 2023-10-19T14:21:38.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.600+0000 7fafd19c9640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fafcc0cf9e0 0x7fafcc0c7070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:38.605 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.600+0000 7fafd19c9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fafcc15b000 0x7fafcc1e3030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:38.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.600+0000 7fafd19c9640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fafcc15b3e0 0x7fafcc0ae2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:38.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.600+0000 7fafd19c9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafcc1e3570 con 0x7fafcc15b000 2023-10-19T14:21:38.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.600+0000 7fafd19c9640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafcc0c5fd0 con 0x7fafcc15b3e0 2023-10-19T14:21:38.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.600+0000 7fafd19c9640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafcc0c62b0 con 0x7fafcc0cf9e0 2023-10-19T14:21:38.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.600+0000 7fafc3fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fafcc15b000 0x7fafcc1e3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:38.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.600+0000 7fafc3fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fafcc15b000 0x7fafcc1e3030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43894/0 (socket says 172.21.15.57:43894) 2023-10-19T14:21:38.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.600+0000 7fafc3fff640 1 -- 172.21.15.57:0/3651193676 learned_addr learned my addr 172.21.15.57:0/3651193676 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:38.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.600+0000 7fafcaffd640 1 --2- 172.21.15.57:0/3651193676 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fafcc0cf9e0 0x7fafcc0c7070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:38.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.600+0000 7fafcb7fe640 1 --2- 172.21.15.57:0/3651193676 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fafcc15b3e0 0x7fafcc0ae2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:38.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.600+0000 7fafc3fff640 1 -- 172.21.15.57:0/3651193676 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fafcc0cf9e0 msgr2=0x7fafcc0c7070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:38.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.600+0000 7fafc3fff640 1 --2- 172.21.15.57:0/3651193676 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fafcc0cf9e0 0x7fafcc0c7070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:38.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.600+0000 7fafc3fff640 1 -- 172.21.15.57:0/3651193676 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fafcc15b3e0 msgr2=0x7fafcc0ae2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:38.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.600+0000 7fafc3fff640 1 --2- 172.21.15.57:0/3651193676 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fafcc15b3e0 0x7fafcc0ae2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:38.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.600+0000 7fafc3fff640 1 -- 172.21.15.57:0/3651193676 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafbc009560 con 0x7fafcc15b000 2023-10-19T14:21:38.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.600+0000 7fafc3fff640 1 --2- 172.21.15.57:0/3651193676 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fafcc15b000 0x7fafcc1e3030 secure :-1 s=READY pgs=2738 cs=0 l=1 rev1=1 crypto rx=0x7fafb400c400 tx=0x7fafb400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:38.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.601+0000 7fafc8ff9640 1 -- 172.21.15.57:0/3651193676 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb40052d0 con 0x7fafcc15b000 2023-10-19T14:21:38.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.601+0000 7fafc8ff9640 1 -- 172.21.15.57:0/3651193676 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fafb40035f0 con 0x7fafcc15b000 2023-10-19T14:21:38.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.601+0000 7fafd19c9640 1 -- 172.21.15.57:0/3651193676 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafcc0c6590 con 0x7fafcc15b000 2023-10-19T14:21:38.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.601+0000 7fafc8ff9640 1 -- 172.21.15.57:0/3651193676 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb4003a00 con 0x7fafcc15b000 2023-10-19T14:21:38.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.601+0000 7fafd19c9640 1 -- 172.21.15.57:0/3651193676 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafcc0c6b50 con 0x7fafcc15b000 2023-10-19T14:21:38.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.602+0000 7fafd19c9640 1 -- 172.21.15.57:0/3651193676 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafcc0690e0 con 0x7fafcc15b000 2023-10-19T14:21:38.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.602+0000 7fafc8ff9640 1 -- 172.21.15.57:0/3651193676 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 141) v1 ==== 89434+0+0 (secure 0 0 0) 0x7fafb4003b60 con 0x7fafcc15b000 2023-10-19T14:21:38.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.603+0000 7fafc8ff9640 1 --2- 172.21.15.57:0/3651193676 >> [v2:172.21.15.179:6832/609071913,v1:172.21.15.179:6833/609071913] conn(0x7fafac06b080 0x7fafac06d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:38.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.603+0000 7fafcaffd640 1 -- 172.21.15.57:0/3651193676 >> [v2:172.21.15.179:6832/609071913,v1:172.21.15.179:6833/609071913] conn(0x7fafac06b080 msgr2=0x7fafac06d540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/609071913 2023-10-19T14:21:38.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.603+0000 7fafc8ff9640 1 -- 172.21.15.57:0/3651193676 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 7351+0+0 (secure 0 0 0) 0x7fafb4089750 con 0x7fafcc15b000 2023-10-19T14:21:38.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.603+0000 7fafcaffd640 1 --2- 172.21.15.57:0/3651193676 >> [v2:172.21.15.179:6832/609071913,v1:172.21.15.179:6833/609071913] conn(0x7fafac06b080 0x7fafac06d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T14:21:38.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.607+0000 7fafc8ff9640 1 -- 172.21.15.57:0/3651193676 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fafb4053fa0 con 0x7fafcc15b000 2023-10-19T14:21:38.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.783+0000 7fafd19c9640 1 -- 172.21.15.57:0/3651193676 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fafcc0c7650 con 0x7fafcc15b000 2023-10-19T14:21:38.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.784+0000 7fafc8ff9640 1 -- 172.21.15.57:0/3651193676 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v248) v1 ==== 93+0+10 (secure 0 0 0) 0x7fafb40575c0 con 0x7fafcc15b000 2023-10-19T14:21:38.786 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-19T14:21:38.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.786+0000 7fafd19c9640 1 -- 172.21.15.57:0/3651193676 >> [v2:172.21.15.179:6832/609071913,v1:172.21.15.179:6833/609071913] conn(0x7fafac06b080 msgr2=0x7fafac06d540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T14:21:38.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.786+0000 7fafd19c9640 1 --2- 172.21.15.57:0/3651193676 >> [v2:172.21.15.179:6832/609071913,v1:172.21.15.179:6833/609071913] conn(0x7fafac06b080 0x7fafac06d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:38.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.786+0000 7fafd19c9640 1 -- 172.21.15.57:0/3651193676 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fafcc15b000 msgr2=0x7fafcc1e3030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:38.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.786+0000 7fafd19c9640 1 --2- 172.21.15.57:0/3651193676 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fafcc15b000 0x7fafcc1e3030 secure :-1 s=READY pgs=2738 cs=0 l=1 rev1=1 crypto rx=0x7fafb400c400 tx=0x7fafb400c8c0 comp rx=0 tx=0).stop 2023-10-19T14:21:38.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.786+0000 7fafd19c9640 1 -- 172.21.15.57:0/3651193676 shutdown_connections 2023-10-19T14:21:38.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.786+0000 7fafd19c9640 1 --2- 172.21.15.57:0/3651193676 >> [v2:172.21.15.179:6832/609071913,v1:172.21.15.179:6833/609071913] conn(0x7fafac06b080 0x7fafac06d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:38.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.786+0000 7fafd19c9640 1 --2- 172.21.15.57:0/3651193676 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fafcc15b3e0 0x7fafcc0ae2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:38.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.786+0000 7fafd19c9640 1 --2- 172.21.15.57:0/3651193676 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fafcc15b000 0x7fafcc1e3030 unknown :-1 s=CLOSED pgs=2738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:38.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.786+0000 7fafd19c9640 1 --2- 172.21.15.57:0/3651193676 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fafcc0cf9e0 0x7fafcc0c7070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:38.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.786+0000 7fafd19c9640 1 -- 172.21.15.57:0/3651193676 >> 172.21.15.57:0/3651193676 conn(0x7fafcc0d0cf0 msgr2=0x7fafcc159ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:38.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.786+0000 7fafd19c9640 1 -- 172.21.15.57:0/3651193676 shutdown_connections 2023-10-19T14:21:38.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.786+0000 7fafd19c9640 1 -- 172.21.15.57:0/3651193676 wait complete. 2023-10-19T14:21:38.799 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-19T14:21:38.820 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:38.818+0000 7fa0a5617040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-19T14:21:38.864 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:38.862+0000 7fa0a5617040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-19T14:21:38.872 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:21:38.871+0000 7f0f9ee3e040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-19T14:21:38.907 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:38.904+0000 7fa0a5617040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-19T14:21:38.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.911+0000 7fe8685a5640 1 -- 172.21.15.57:0/4167843819 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe86015b640 msgr2=0x7fe8600cbf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:38.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.911+0000 7fe8685a5640 1 --2- 172.21.15.57:0/4167843819 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe86015b640 0x7fe8600cbf20 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7fe858005fd0 tx=0x7fe85802c0e0 comp rx=0 tx=0).stop 2023-10-19T14:21:38.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.911+0000 7fe8685a5640 1 -- 172.21.15.57:0/4167843819 shutdown_connections 2023-10-19T14:21:38.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.911+0000 7fe8685a5640 1 --2- 172.21.15.57:0/4167843819 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe86015b640 0x7fe8600cbf20 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:38.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.911+0000 7fe8685a5640 1 --2- 172.21.15.57:0/4167843819 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe8600cfa20 0x7fe8600cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:38.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.911+0000 7fe8685a5640 1 -- 172.21.15.57:0/4167843819 >> 172.21.15.57:0/4167843819 conn(0x7fe86009a030 msgr2=0x7fe8600d0d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:38.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.911+0000 7fe8685a5640 1 -- 172.21.15.57:0/4167843819 shutdown_connections 2023-10-19T14:21:38.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.911+0000 7fe8685a5640 1 -- 172.21.15.57:0/4167843819 wait complete. 2023-10-19T14:21:38.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.912+0000 7fe8685a5640 1 Processor -- start 2023-10-19T14:21:38.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.912+0000 7fe8685a5640 1 -- start start 2023-10-19T14:21:38.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.913+0000 7fe8685a5640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe8600cfa20 0x7fe860154e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:38.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.913+0000 7fe8685a5640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe86015b260 0x7fe860155660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:38.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.913+0000 7fe8685a5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe86015b640 0x7fe8600b1e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:38.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.913+0000 7fe8685a5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8600b3940 con 0x7fe86015b640 2023-10-19T14:21:38.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.913+0000 7fe8685a5640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8600b3c00 con 0x7fe86015b260 2023-10-19T14:21:38.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.913+0000 7fe8685a5640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8600b3ee0 con 0x7fe8600cfa20 2023-10-19T14:21:38.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.913+0000 7fe866b42640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe86015b640 0x7fe8600b1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:38.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.913+0000 7fe866b42640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe86015b640 0x7fe8600b1e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43906/0 (socket says 172.21.15.57:43906) 2023-10-19T14:21:38.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.913+0000 7fe866b42640 1 -- 172.21.15.57:0/1437160858 learned_addr learned my addr 172.21.15.57:0/1437160858 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:38.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.913+0000 7fe865b40640 1 --2- 172.21.15.57:0/1437160858 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe86015b260 0x7fe860155660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:38.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.913+0000 7fe866341640 1 --2- 172.21.15.57:0/1437160858 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe8600cfa20 0x7fe860154e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:38.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.913+0000 7fe866b42640 1 -- 172.21.15.57:0/1437160858 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe8600cfa20 msgr2=0x7fe860154e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:38.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.913+0000 7fe866b42640 1 --2- 172.21.15.57:0/1437160858 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe8600cfa20 0x7fe860154e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:38.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.913+0000 7fe866b42640 1 -- 172.21.15.57:0/1437160858 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe86015b260 msgr2=0x7fe860155660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:38.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.913+0000 7fe866b42640 1 --2- 172.21.15.57:0/1437160858 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe86015b260 0x7fe860155660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:38.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.913+0000 7fe866b42640 1 -- 172.21.15.57:0/1437160858 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe85c009490 con 0x7fe86015b640 2023-10-19T14:21:38.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.913+0000 7fe865b40640 1 --2- 172.21.15.57:0/1437160858 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe86015b260 0x7fe860155660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T14:21:38.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.913+0000 7fe866341640 1 --2- 172.21.15.57:0/1437160858 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe8600cfa20 0x7fe860154e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T14:21:38.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.913+0000 7fe866b42640 1 --2- 172.21.15.57:0/1437160858 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe86015b640 0x7fe8600b1e70 secure :-1 s=READY pgs=2739 cs=0 l=1 rev1=1 crypto rx=0x7fe858036040 tx=0x7fe858005c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:38.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.914+0000 7fe852ffd640 1 -- 172.21.15.57:0/1437160858 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe858003e60 con 0x7fe86015b640 2023-10-19T14:21:38.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.914+0000 7fe852ffd640 1 -- 172.21.15.57:0/1437160858 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fe858004790 con 0x7fe86015b640 2023-10-19T14:21:38.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.914+0000 7fe8685a5640 1 -- 172.21.15.57:0/1437160858 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe858009840 con 0x7fe86015b640 2023-10-19T14:21:38.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.914+0000 7fe852ffd640 1 -- 172.21.15.57:0/1437160858 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe858034800 con 0x7fe86015b640 2023-10-19T14:21:38.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.914+0000 7fe8685a5640 1 -- 172.21.15.57:0/1437160858 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8600b44a0 con 0x7fe86015b640 2023-10-19T14:21:38.923 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.915+0000 7fe8685a5640 1 -- 172.21.15.57:0/1437160858 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8600b4690 con 0x7fe86015b640 2023-10-19T14:21:38.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.916+0000 7fe852ffd640 1 -- 172.21.15.57:0/1437160858 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 141) v1 ==== 89434+0+0 (secure 0 0 0) 0x7fe858034960 con 0x7fe86015b640 2023-10-19T14:21:38.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.916+0000 7fe852ffd640 1 --2- 172.21.15.57:0/1437160858 >> [v2:172.21.15.179:6832/609071913,v1:172.21.15.179:6833/609071913] conn(0x7fe84406b080 0x7fe84406d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:38.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.916+0000 7fe852ffd640 1 -- 172.21.15.57:0/1437160858 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 7351+0+0 (secure 0 0 0) 0x7fe8580acfc0 con 0x7fe86015b640 2023-10-19T14:21:38.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.916+0000 7fe866341640 1 -- 172.21.15.57:0/1437160858 >> [v2:172.21.15.179:6832/609071913,v1:172.21.15.179:6833/609071913] conn(0x7fe84406b080 msgr2=0x7fe84406d540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/609071913 2023-10-19T14:21:38.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.916+0000 7fe866341640 1 --2- 172.21.15.57:0/1437160858 >> [v2:172.21.15.179:6832/609071913,v1:172.21.15.179:6833/609071913] conn(0x7fe84406b080 0x7fe84406d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T14:21:38.925 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:21:38.920+0000 7f0f9ee3e040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-19T14:21:38.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:38.920+0000 7fe852ffd640 1 -- 172.21.15.57:0/1437160858 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe8580dfca0 con 0x7fe86015b640 2023-10-19T14:21:38.961 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:38.958+0000 7fa0a5617040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-19T14:21:38.967 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:21:38.965+0000 7f0f9ee3e040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-19T14:21:39.003 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:39.000+0000 7fa0a5617040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-19T14:21:39.022 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:21:39.020+0000 7f0f9ee3e040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-19T14:21:39.066 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:21:39.064+0000 7f0f9ee3e040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-19T14:21:39.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.097+0000 7fe8685a5640 1 -- 172.21.15.57:0/1437160858 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe8600675f0 con 0x7fe86015b640 2023-10-19T14:21:39.102 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:39.097+0000 7fa0a5617040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-19T14:21:39.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.097+0000 7fe852ffd640 1 -- 172.21.15.57:0/1437160858 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v248) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe858043020 con 0x7fe86015b640 2023-10-19T14:21:39.103 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2023-10-19T14:21:39.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.099+0000 7fe8685a5640 1 -- 172.21.15.57:0/1437160858 >> [v2:172.21.15.179:6832/609071913,v1:172.21.15.179:6833/609071913] conn(0x7fe84406b080 msgr2=0x7fe84406d540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T14:21:39.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.099+0000 7fe8685a5640 1 --2- 172.21.15.57:0/1437160858 >> [v2:172.21.15.179:6832/609071913,v1:172.21.15.179:6833/609071913] conn(0x7fe84406b080 0x7fe84406d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:39.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.099+0000 7fe8685a5640 1 -- 172.21.15.57:0/1437160858 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe86015b640 msgr2=0x7fe8600b1e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:39.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.099+0000 7fe8685a5640 1 --2- 172.21.15.57:0/1437160858 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe86015b640 0x7fe8600b1e70 secure :-1 s=READY pgs=2739 cs=0 l=1 rev1=1 crypto rx=0x7fe858036040 tx=0x7fe858005c80 comp rx=0 tx=0).stop 2023-10-19T14:21:39.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.099+0000 7fe8685a5640 1 -- 172.21.15.57:0/1437160858 shutdown_connections 2023-10-19T14:21:39.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.099+0000 7fe8685a5640 1 --2- 172.21.15.57:0/1437160858 >> [v2:172.21.15.179:6832/609071913,v1:172.21.15.179:6833/609071913] conn(0x7fe84406b080 0x7fe84406d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:39.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.099+0000 7fe8685a5640 1 --2- 172.21.15.57:0/1437160858 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe86015b640 0x7fe8600b1e70 unknown :-1 s=CLOSED pgs=2739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:39.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.099+0000 7fe8685a5640 1 --2- 172.21.15.57:0/1437160858 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe86015b260 0x7fe860155660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:39.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.099+0000 7fe8685a5640 1 --2- 172.21.15.57:0/1437160858 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe8600cfa20 0x7fe860154e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:39.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.099+0000 7fe8685a5640 1 -- 172.21.15.57:0/1437160858 >> 172.21.15.57:0/1437160858 conn(0x7fe86009a030 msgr2=0x7fe8600b1310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:39.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.099+0000 7fe8685a5640 1 -- 172.21.15.57:0/1437160858 shutdown_connections 2023-10-19T14:21:39.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.099+0000 7fe8685a5640 1 -- 172.21.15.57:0/1437160858 wait complete. 2023-10-19T14:21:39.112 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-19T14:21:39.151 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:39.149+0000 7fa0a5617040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-19T14:21:39.171 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:21:39.169+0000 7f0f9ee3e040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-19T14:21:39.192 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:39.190+0000 7fa0a5617040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-19T14:21:39.225 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.223+0000 7fbc7a7e9640 1 -- 172.21.15.57:0/3729437904 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc740d1910 msgr2=0x7fbc740b0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:39.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.223+0000 7fbc7a7e9640 1 --2- 172.21.15.57:0/3729437904 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc740d1910 0x7fbc740b0650 secure :-1 s=READY pgs=2741 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c005fd0 tx=0x7fbc6c02c0e0 comp rx=0 tx=0).stop 2023-10-19T14:21:39.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.223+0000 7fbc7a7e9640 1 -- 172.21.15.57:0/3729437904 shutdown_connections 2023-10-19T14:21:39.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.223+0000 7fbc7a7e9640 1 --2- 172.21.15.57:0/3729437904 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc740d1910 0x7fbc740b0650 unknown :-1 s=CLOSED pgs=2741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:39.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.223+0000 7fbc7a7e9640 1 -- 172.21.15.57:0/3729437904 >> 172.21.15.57:0/3729437904 conn(0x7fbc740d0d60 msgr2=0x7fbc740d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:39.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.223+0000 7fbc7a7e9640 1 -- 172.21.15.57:0/3729437904 shutdown_connections 2023-10-19T14:21:39.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.223+0000 7fbc7a7e9640 1 -- 172.21.15.57:0/3729437904 wait complete. 2023-10-19T14:21:39.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.225+0000 7fbc7a7e9640 1 Processor -- start 2023-10-19T14:21:39.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.225+0000 7fbc7a7e9640 1 -- start start 2023-10-19T14:21:39.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.225+0000 7fbc7a7e9640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbc740cf340 0x7fbc74158040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:39.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.225+0000 7fbc7a7e9640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbc740d1910 0x7fbc740b5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:39.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.225+0000 7fbc7a7e9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc7415b2a0 0x7fbc740b59d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:39.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.225+0000 7fbc7a7e9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc740c87f0 con 0x7fbc7415b2a0 2023-10-19T14:21:39.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.226+0000 7fbc7a7e9640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc74158580 con 0x7fbc740cf340 2023-10-19T14:21:39.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.226+0000 7fbc7a7e9640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc740c89e0 con 0x7fbc740d1910 2023-10-19T14:21:39.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.226+0000 7fbc78d86640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc7415b2a0 0x7fbc740b59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:39.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.226+0000 7fbc78d86640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc7415b2a0 0x7fbc740b59d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43922/0 (socket says 172.21.15.57:43922) 2023-10-19T14:21:39.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.226+0000 7fbc78d86640 1 -- 172.21.15.57:0/4227545426 learned_addr learned my addr 172.21.15.57:0/4227545426 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:39.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.226+0000 7fbc73fff640 1 --2- 172.21.15.57:0/4227545426 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbc740cf340 0x7fbc74158040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:39.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.226+0000 7fbc737fe640 1 --2- 172.21.15.57:0/4227545426 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbc740d1910 0x7fbc740b5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:39.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.226+0000 7fbc78d86640 1 -- 172.21.15.57:0/4227545426 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbc740d1910 msgr2=0x7fbc740b5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:39.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.226+0000 7fbc78d86640 1 --2- 172.21.15.57:0/4227545426 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbc740d1910 0x7fbc740b5490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:39.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.226+0000 7fbc78d86640 1 -- 172.21.15.57:0/4227545426 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbc740cf340 msgr2=0x7fbc74158040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:39.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.226+0000 7fbc78d86640 1 --2- 172.21.15.57:0/4227545426 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbc740cf340 0x7fbc74158040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:39.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.226+0000 7fbc78d86640 1 -- 172.21.15.57:0/4227545426 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc6c009840 con 0x7fbc7415b2a0 2023-10-19T14:21:39.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.226+0000 7fbc78d86640 1 --2- 172.21.15.57:0/4227545426 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc7415b2a0 0x7fbc740b59d0 secure :-1 s=READY pgs=2742 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c009cd0 tx=0x7fbc6c004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:39.233 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:21:39.226+0000 7f0f9ee3e040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-19T14:21:39.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.226+0000 7fbc70ff9640 1 -- 172.21.15.57:0/4227545426 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc6c004760 con 0x7fbc7415b2a0 2023-10-19T14:21:39.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.226+0000 7fbc70ff9640 1 -- 172.21.15.57:0/4227545426 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fbc6c0345a0 con 0x7fbc7415b2a0 2023-10-19T14:21:39.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.226+0000 7fbc7a7e9640 1 -- 172.21.15.57:0/4227545426 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc740c8cc0 con 0x7fbc7415b2a0 2023-10-19T14:21:39.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.226+0000 7fbc70ff9640 1 -- 172.21.15.57:0/4227545426 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc6c034990 con 0x7fbc7415b2a0 2023-10-19T14:21:39.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.226+0000 7fbc7a7e9640 1 -- 172.21.15.57:0/4227545426 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc740c9280 con 0x7fbc7415b2a0 2023-10-19T14:21:39.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.227+0000 7fbc7a7e9640 1 -- 172.21.15.57:0/4227545426 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc740690e0 con 0x7fbc7415b2a0 2023-10-19T14:21:39.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.228+0000 7fbc70ff9640 1 -- 172.21.15.57:0/4227545426 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 141) v1 ==== 89434+0+0 (secure 0 0 0) 0x7fbc6c034af0 con 0x7fbc7415b2a0 2023-10-19T14:21:39.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.228+0000 7fbc70ff9640 1 --2- 172.21.15.57:0/4227545426 >> [v2:172.21.15.179:6832/609071913,v1:172.21.15.179:6833/609071913] conn(0x7fbc5406b080 0x7fbc5406d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:39.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.228+0000 7fbc73fff640 1 -- 172.21.15.57:0/4227545426 >> [v2:172.21.15.179:6832/609071913,v1:172.21.15.179:6833/609071913] conn(0x7fbc5406b080 msgr2=0x7fbc5406d540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/609071913 2023-10-19T14:21:39.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.228+0000 7fbc70ff9640 1 -- 172.21.15.57:0/4227545426 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 7351+0+0 (secure 0 0 0) 0x7fbc6c0ad240 con 0x7fbc7415b2a0 2023-10-19T14:21:39.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.228+0000 7fbc73fff640 1 --2- 172.21.15.57:0/4227545426 >> [v2:172.21.15.179:6832/609071913,v1:172.21.15.179:6833/609071913] conn(0x7fbc5406b080 0x7fbc5406d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T14:21:39.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.236+0000 7fbc70ff9640 1 -- 172.21.15.57:0/4227545426 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fbc6c0779d0 con 0x7fbc7415b2a0 2023-10-19T14:21:39.240 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:39.238+0000 7fa0a5617040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-19T14:21:39.276 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:21:39.274+0000 7f0f9ee3e040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-19T14:21:39.283 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:39.280+0000 7fa0a5617040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-19T14:21:39.327 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:21:39.325+0000 7f0f9ee3e040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-19T14:21:39.374 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:21:39.372+0000 7f0f9ee3e040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-19T14:21:39.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.413+0000 7fbc7a7e9640 1 -- 172.21.15.57:0/4227545426 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fbc740d0b40 con 0x7fbc7415b2a0 2023-10-19T14:21:39.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.413+0000 7fbc70ff9640 1 -- 172.21.15.57:0/4227545426 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v248) v1 ==== 100+0+11 (secure 0 0 0) 0x7fbc6c07aff0 con 0x7fbc7415b2a0 2023-10-19T14:21:39.416 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 64 2023-10-19T14:21:39.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.415+0000 7fbc7a7e9640 1 -- 172.21.15.57:0/4227545426 >> [v2:172.21.15.179:6832/609071913,v1:172.21.15.179:6833/609071913] conn(0x7fbc5406b080 msgr2=0x7fbc5406d540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T14:21:39.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.415+0000 7fbc7a7e9640 1 --2- 172.21.15.57:0/4227545426 >> [v2:172.21.15.179:6832/609071913,v1:172.21.15.179:6833/609071913] conn(0x7fbc5406b080 0x7fbc5406d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:39.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.415+0000 7fbc7a7e9640 1 -- 172.21.15.57:0/4227545426 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc7415b2a0 msgr2=0x7fbc740b59d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:39.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.415+0000 7fbc7a7e9640 1 --2- 172.21.15.57:0/4227545426 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc7415b2a0 0x7fbc740b59d0 secure :-1 s=READY pgs=2742 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c009cd0 tx=0x7fbc6c004360 comp rx=0 tx=0).stop 2023-10-19T14:21:39.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.415+0000 7fbc7a7e9640 1 -- 172.21.15.57:0/4227545426 shutdown_connections 2023-10-19T14:21:39.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.415+0000 7fbc7a7e9640 1 --2- 172.21.15.57:0/4227545426 >> [v2:172.21.15.179:6832/609071913,v1:172.21.15.179:6833/609071913] conn(0x7fbc5406b080 0x7fbc5406d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:39.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.415+0000 7fbc7a7e9640 1 --2- 172.21.15.57:0/4227545426 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc7415b2a0 0x7fbc740b59d0 unknown :-1 s=CLOSED pgs=2742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:39.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.415+0000 7fbc7a7e9640 1 --2- 172.21.15.57:0/4227545426 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbc740d1910 0x7fbc740b5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:39.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.415+0000 7fbc7a7e9640 1 --2- 172.21.15.57:0/4227545426 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbc740cf340 0x7fbc74158040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:39.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.415+0000 7fbc7a7e9640 1 -- 172.21.15.57:0/4227545426 >> 172.21.15.57:0/4227545426 conn(0x7fbc740d0d60 msgr2=0x7fbc740b0da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:39.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.416+0000 7fbc7a7e9640 1 -- 172.21.15.57:0/4227545426 shutdown_connections 2023-10-19T14:21:39.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.416+0000 7fbc7a7e9640 1 -- 172.21.15.57:0/4227545426 wait complete. 2023-10-19T14:21:39.429 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2023-10-19T14:21:39.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.540+0000 7f879a505640 1 -- 172.21.15.57:0/3906128425 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f879415eba0 msgr2=0x7f87940b00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:39.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.540+0000 7f879a505640 1 --2- 172.21.15.57:0/3906128425 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f879415eba0 0x7f87940b00f0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f8784009830 tx=0x7f878402bde0 comp rx=0 tx=0).stop 2023-10-19T14:21:39.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.540+0000 7f879a505640 1 -- 172.21.15.57:0/3906128425 shutdown_connections 2023-10-19T14:21:39.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.540+0000 7f879a505640 1 --2- 172.21.15.57:0/3906128425 >> v2:172.21.15.179:3300/0 conn(0x7f87940cf2a0 0x7f87940cf730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:39.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.540+0000 7f879a505640 1 --2- 172.21.15.57:0/3906128425 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f879415eba0 0x7f87940b00f0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:39.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.540+0000 7f879a505640 1 -- 172.21.15.57:0/3906128425 >> 172.21.15.57:0/3906128425 conn(0x7f879408a650 msgr2=0x7f8794099f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:39.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.540+0000 7f879a505640 1 -- 172.21.15.57:0/3906128425 shutdown_connections 2023-10-19T14:21:39.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.540+0000 7f879a505640 1 -- 172.21.15.57:0/3906128425 wait complete. 2023-10-19T14:21:39.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.541+0000 7f879a505640 1 Processor -- start 2023-10-19T14:21:39.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.541+0000 7f879a505640 1 -- start start 2023-10-19T14:21:39.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.542+0000 7f879a505640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f87940cf2a0 0x7f87940c47b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:39.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.542+0000 7f879a505640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f879415b200 0x7f87941f4470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:39.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.542+0000 7f879a505640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f879415eba0 0x7f87940aec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:39.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.542+0000 7f879a505640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87940af550 con 0x7f879415b200 2023-10-19T14:21:39.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.542+0000 7f879a505640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87940af830 con 0x7f87940cf2a0 2023-10-19T14:21:39.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.542+0000 7f879a505640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f879415c8e0 con 0x7f879415eba0 2023-10-19T14:21:39.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.542+0000 7f87937fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f879415b200 0x7f87941f4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:39.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.542+0000 7f87937fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f879415b200 0x7f87941f4470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43924/0 (socket says 172.21.15.57:43924) 2023-10-19T14:21:39.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.542+0000 7f87937fe640 1 -- 172.21.15.57:0/1280097913 learned_addr learned my addr 172.21.15.57:0/1280097913 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:39.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.542+0000 7f8793fff640 1 --2- 172.21.15.57:0/1280097913 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f87940cf2a0 0x7f87940c47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:39.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.542+0000 7f8798aa2640 1 --2- 172.21.15.57:0/1280097913 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f879415eba0 0x7f87940aec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:39.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.542+0000 7f8798aa2640 1 -- 172.21.15.57:0/1280097913 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f87940cf2a0 msgr2=0x7f87940c47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:39.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.542+0000 7f8798aa2640 1 --2- 172.21.15.57:0/1280097913 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f87940cf2a0 0x7f87940c47b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:39.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.543+0000 7f8798aa2640 1 -- 172.21.15.57:0/1280097913 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f879415b200 msgr2=0x7f87941f4470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:39.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.543+0000 7f8798aa2640 1 --2- 172.21.15.57:0/1280097913 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f879415b200 0x7f87941f4470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:39.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.543+0000 7f8798aa2640 1 -- 172.21.15.57:0/1280097913 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8784009490 con 0x7f879415eba0 2023-10-19T14:21:39.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.543+0000 7f8793fff640 1 --2- 172.21.15.57:0/1280097913 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f87940cf2a0 0x7f87940c47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T14:21:39.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.543+0000 7f87937fe640 1 --2- 172.21.15.57:0/1280097913 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f879415b200 0x7f87941f4470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T14:21:39.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.543+0000 7f8798aa2640 1 --2- 172.21.15.57:0/1280097913 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f879415eba0 0x7f87940aec70 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f878c009fd0 tx=0x7f878c00ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:39.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.543+0000 7f8790ff9640 1 -- 172.21.15.57:0/1280097913 <== mon.2 v2:172.21.15.179:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f878c00ce20 con 0x7f879415eba0 2023-10-19T14:21:39.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.543+0000 7f8790ff9640 1 -- 172.21.15.57:0/1280097913 <== mon.2 v2:172.21.15.179:3301/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f878c004c10 con 0x7f879415eba0 2023-10-19T14:21:39.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.543+0000 7f8790ff9640 1 -- 172.21.15.57:0/1280097913 <== mon.2 v2:172.21.15.179:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f878c005020 con 0x7f879415eba0 2023-10-19T14:21:39.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.543+0000 7f879a505640 1 -- 172.21.15.57:0/1280097913 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f879415cbc0 con 0x7f879415eba0 2023-10-19T14:21:39.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.543+0000 7f879a505640 1 -- 172.21.15.57:0/1280097913 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87940af1b0 con 0x7f879415eba0 2023-10-19T14:21:39.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.544+0000 7f879a505640 1 -- 172.21.15.57:0/1280097913 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8794066930 con 0x7f879415eba0 2023-10-19T14:21:39.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.545+0000 7f8790ff9640 1 -- 172.21.15.57:0/1280097913 <== mon.2 v2:172.21.15.179:3301/0 4 ==== mgrmap(e 141) v1 ==== 89434+0+0 (secure 0 0 0) 0x7f878c005180 con 0x7f879415eba0 2023-10-19T14:21:39.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.545+0000 7f8790ff9640 1 --2- 172.21.15.57:0/1280097913 >> [v2:172.21.15.179:6832/609071913,v1:172.21.15.179:6833/609071913] conn(0x7f877406b080 0x7f877406d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:39.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.545+0000 7f8793fff640 1 -- 172.21.15.57:0/1280097913 >> [v2:172.21.15.179:6832/609071913,v1:172.21.15.179:6833/609071913] conn(0x7f877406b080 msgr2=0x7f877406d540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/609071913 2023-10-19T14:21:39.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.545+0000 7f8790ff9640 1 -- 172.21.15.57:0/1280097913 <== mon.2 v2:172.21.15.179:3301/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 7351+0+0 (secure 0 0 0) 0x7f878c014070 con 0x7f879415eba0 2023-10-19T14:21:39.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.545+0000 7f8793fff640 1 --2- 172.21.15.57:0/1280097913 >> [v2:172.21.15.179:6832/609071913,v1:172.21.15.179:6833/609071913] conn(0x7f877406b080 0x7f877406d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T14:21:39.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.549+0000 7f8790ff9640 1 -- 172.21.15.57:0/1280097913 <== mon.2 v2:172.21.15.179:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f878c0555e0 con 0x7f879415eba0 2023-10-19T14:21:39.610 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:39.607+0000 7fa0a5617040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-19T14:21:39.715 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:21:39.713+0000 7f0f9ee3e040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-19T14:21:39.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.727+0000 7f879a505640 1 -- 172.21.15.57:0/1280097913 --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f879415ca20 con 0x7f879415eba0 2023-10-19T14:21:39.729 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.728+0000 7f8790ff9640 1 -- 172.21.15.57:0/1280097913 <== mon.2 v2:172.21.15.179:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v248) v1 ==== 107+0+11 (secure 0 0 0) 0x7f879415ca20 con 0x7f879415eba0 2023-10-19T14:21:39.730 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2023-10-19T14:21:39.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.730+0000 7f879a505640 1 -- 172.21.15.57:0/1280097913 >> [v2:172.21.15.179:6832/609071913,v1:172.21.15.179:6833/609071913] conn(0x7f877406b080 msgr2=0x7f877406d540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T14:21:39.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.730+0000 7f879a505640 1 --2- 172.21.15.57:0/1280097913 >> [v2:172.21.15.179:6832/609071913,v1:172.21.15.179:6833/609071913] conn(0x7f877406b080 0x7f877406d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:39.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.730+0000 7f879a505640 1 -- 172.21.15.57:0/1280097913 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f879415eba0 msgr2=0x7f87940aec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:39.732 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.730+0000 7f879a505640 1 --2- 172.21.15.57:0/1280097913 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f879415eba0 0x7f87940aec70 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f878c009fd0 tx=0x7f878c00ef10 comp rx=0 tx=0).stop 2023-10-19T14:21:39.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.730+0000 7f879a505640 1 -- 172.21.15.57:0/1280097913 shutdown_connections 2023-10-19T14:21:39.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.730+0000 7f879a505640 1 --2- 172.21.15.57:0/1280097913 >> [v2:172.21.15.179:6832/609071913,v1:172.21.15.179:6833/609071913] conn(0x7f877406b080 0x7f877406d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:39.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.730+0000 7f879a505640 1 --2- 172.21.15.57:0/1280097913 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f879415eba0 0x7f87940aec70 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:39.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.730+0000 7f879a505640 1 --2- 172.21.15.57:0/1280097913 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f879415b200 0x7f87941f4470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:39.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.730+0000 7f879a505640 1 --2- 172.21.15.57:0/1280097913 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f87940cf2a0 0x7f87940c47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:39.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.730+0000 7f879a505640 1 -- 172.21.15.57:0/1280097913 >> 172.21.15.57:0/1280097913 conn(0x7f879408a650 msgr2=0x7f87940cc0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:39.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.730+0000 7f879a505640 1 -- 172.21.15.57:0/1280097913 shutdown_connections 2023-10-19T14:21:39.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.730+0000 7f879a505640 1 -- 172.21.15.57:0/1280097913 wait complete. 2023-10-19T14:21:39.739 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:39.737+0000 7fa0a5617040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-19T14:21:39.742 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2023-10-19T14:21:39.829 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:39.827+0000 7fa0a5617040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-19T14:21:39.855 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.853+0000 7f85d9ada640 1 -- 172.21.15.57:0/2835399938 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85d40d1910 msgr2=0x7f85d40b0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:39.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.853+0000 7f85d9ada640 1 --2- 172.21.15.57:0/2835399938 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85d40d1910 0x7f85d40b0650 secure :-1 s=READY pgs=2745 cs=0 l=1 rev1=1 crypto rx=0x7f85cc005fd0 tx=0x7f85cc02bf30 comp rx=0 tx=0).stop 2023-10-19T14:21:39.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.853+0000 7f85d9ada640 1 -- 172.21.15.57:0/2835399938 shutdown_connections 2023-10-19T14:21:39.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.853+0000 7f85d9ada640 1 --2- 172.21.15.57:0/2835399938 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85d40d1910 0x7f85d40b0650 unknown :-1 s=CLOSED pgs=2745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:39.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.853+0000 7f85d9ada640 1 -- 172.21.15.57:0/2835399938 >> 172.21.15.57:0/2835399938 conn(0x7f85d40d0d60 msgr2=0x7f85d40d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:39.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.854+0000 7f85d9ada640 1 -- 172.21.15.57:0/2835399938 shutdown_connections 2023-10-19T14:21:39.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.854+0000 7f85d9ada640 1 -- 172.21.15.57:0/2835399938 wait complete. 2023-10-19T14:21:39.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.855+0000 7f85d9ada640 1 Processor -- start 2023-10-19T14:21:39.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.855+0000 7f85d9ada640 1 -- start start 2023-10-19T14:21:39.858 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:21:39.855+0000 7f0f9ee3e040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-19T14:21:39.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.856+0000 7f85d9ada640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f85d40cf340 0x7f85d40caa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:39.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.856+0000 7f85d9ada640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85d40d1910 0x7f85d406b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:39.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.856+0000 7f85d9ada640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f85d415b2a0 0x7f85d40b6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:39.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.856+0000 7f85d9ada640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85d40b50f0 con 0x7f85d40d1910 2023-10-19T14:21:39.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.856+0000 7f85d9ada640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85d40b53d0 con 0x7f85d40cf340 2023-10-19T14:21:39.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.856+0000 7f85d9ada640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85d40b56b0 con 0x7f85d415b2a0 2023-10-19T14:21:39.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.856+0000 7f85caffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85d40d1910 0x7f85d406b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:39.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.856+0000 7f85caffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85d40d1910 0x7f85d406b250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43938/0 (socket says 172.21.15.57:43938) 2023-10-19T14:21:39.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.856+0000 7f85caffd640 1 -- 172.21.15.57:0/3563587324 learned_addr learned my addr 172.21.15.57:0/3563587324 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:39.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.856+0000 7f85d37fe640 1 --2- 172.21.15.57:0/3563587324 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f85d40cf340 0x7f85d40caa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:39.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.856+0000 7f85d3fff640 1 --2- 172.21.15.57:0/3563587324 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f85d415b2a0 0x7f85d40b6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:39.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.856+0000 7f85caffd640 1 -- 172.21.15.57:0/3563587324 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f85d415b2a0 msgr2=0x7f85d40b6190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:39.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.856+0000 7f85caffd640 1 --2- 172.21.15.57:0/3563587324 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f85d415b2a0 0x7f85d40b6190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:39.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.856+0000 7f85caffd640 1 -- 172.21.15.57:0/3563587324 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f85d40cf340 msgr2=0x7f85d40caa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:39.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.856+0000 7f85caffd640 1 --2- 172.21.15.57:0/3563587324 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f85d40cf340 0x7f85d40caa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:39.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.856+0000 7f85caffd640 1 -- 172.21.15.57:0/3563587324 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85cc009840 con 0x7f85d40d1910 2023-10-19T14:21:39.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.856+0000 7f85caffd640 1 --2- 172.21.15.57:0/3563587324 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85d40d1910 0x7f85d406b250 secure :-1 s=READY pgs=2746 cs=0 l=1 rev1=1 crypto rx=0x7f85bc00c550 tx=0x7f85bc00ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:39.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.856+0000 7f85d0ff9640 1 -- 172.21.15.57:0/3563587324 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85bc012830 con 0x7f85d40d1910 2023-10-19T14:21:39.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.856+0000 7f85d0ff9640 1 -- 172.21.15.57:0/3563587324 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f85bc012aa0 con 0x7f85d40d1910 2023-10-19T14:21:39.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.857+0000 7f85d9ada640 1 -- 172.21.15.57:0/3563587324 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85d40b5990 con 0x7f85d40d1910 2023-10-19T14:21:39.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.857+0000 7f85d0ff9640 1 -- 172.21.15.57:0/3563587324 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85bc011380 con 0x7f85d40d1910 2023-10-19T14:21:39.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.857+0000 7f85d9ada640 1 -- 172.21.15.57:0/3563587324 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85d40b4d50 con 0x7f85d40d1910 2023-10-19T14:21:39.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.858+0000 7f85d9ada640 1 -- 172.21.15.57:0/3563587324 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85d40b5eb0 con 0x7f85d40d1910 2023-10-19T14:21:39.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.858+0000 7f85d0ff9640 1 -- 172.21.15.57:0/3563587324 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 141) v1 ==== 89434+0+0 (secure 0 0 0) 0x7f85bc0114e0 con 0x7f85d40d1910 2023-10-19T14:21:39.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.858+0000 7f85d0ff9640 1 --2- 172.21.15.57:0/3563587324 >> [v2:172.21.15.179:6832/609071913,v1:172.21.15.179:6833/609071913] conn(0x7f85b406b0b0 0x7f85b406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:39.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.859+0000 7f85d0ff9640 1 -- 172.21.15.57:0/3563587324 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 7351+0+0 (secure 0 0 0) 0x7f85bc08a290 con 0x7f85d40d1910 2023-10-19T14:21:39.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.859+0000 7f85d37fe640 1 -- 172.21.15.57:0/3563587324 >> [v2:172.21.15.179:6832/609071913,v1:172.21.15.179:6833/609071913] conn(0x7f85b406b0b0 msgr2=0x7f85b406d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/609071913 2023-10-19T14:21:39.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.859+0000 7f85d37fe640 1 --2- 172.21.15.57:0/3563587324 >> [v2:172.21.15.179:6832/609071913,v1:172.21.15.179:6833/609071913] conn(0x7f85b406b0b0 0x7f85b406d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T14:21:39.868 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:39.867+0000 7f85d0ff9640 1 -- 172.21.15.57:0/3563587324 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f85d40b5eb0 con 0x7f85d40d1910 2023-10-19T14:21:39.872 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:39.870+0000 7fa0a5617040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-19T14:21:39.958 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:21:39.957+0000 7f0f9ee3e040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-19T14:21:39.968 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:39.966+0000 7fa0a5617040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-19T14:21:40.004 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:21:40.002+0000 7f0f9ee3e040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-19T14:21:40.044 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.042+0000 7f85d9ada640 1 -- 172.21.15.57:0/3563587324 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) v1 -- 0x7f85d40b5eb0 con 0x7f85d40d1910 2023-10-19T14:21:40.045 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.043+0000 7f85d0ff9640 1 -- 172.21.15.57:0/3563587324 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v248) v1 ==== 103+0+11 (secure 0 0 0) 0x7f85d40b5eb0 con 0x7f85d40d1910 2023-10-19T14:21:40.045 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 64 2023-10-19T14:21:40.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.045+0000 7f85d9ada640 1 -- 172.21.15.57:0/3563587324 >> [v2:172.21.15.179:6832/609071913,v1:172.21.15.179:6833/609071913] conn(0x7f85b406b0b0 msgr2=0x7f85b406d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T14:21:40.047 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.045+0000 7f85d9ada640 1 --2- 172.21.15.57:0/3563587324 >> [v2:172.21.15.179:6832/609071913,v1:172.21.15.179:6833/609071913] conn(0x7f85b406b0b0 0x7f85b406d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:40.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.045+0000 7f85d9ada640 1 -- 172.21.15.57:0/3563587324 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85d40d1910 msgr2=0x7f85d406b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:40.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.045+0000 7f85d9ada640 1 --2- 172.21.15.57:0/3563587324 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85d40d1910 0x7f85d406b250 secure :-1 s=READY pgs=2746 cs=0 l=1 rev1=1 crypto rx=0x7f85bc00c550 tx=0x7f85bc00ca10 comp rx=0 tx=0).stop 2023-10-19T14:21:40.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.045+0000 7f85d9ada640 1 -- 172.21.15.57:0/3563587324 shutdown_connections 2023-10-19T14:21:40.048 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.045+0000 7f85d9ada640 1 --2- 172.21.15.57:0/3563587324 >> [v2:172.21.15.179:6832/609071913,v1:172.21.15.179:6833/609071913] conn(0x7f85b406b0b0 0x7f85b406d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:40.049 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.045+0000 7f85d9ada640 1 --2- 172.21.15.57:0/3563587324 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f85d415b2a0 0x7f85d40b6190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:40.049 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.045+0000 7f85d9ada640 1 --2- 172.21.15.57:0/3563587324 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85d40d1910 0x7f85d406b250 unknown :-1 s=CLOSED pgs=2746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:40.049 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.045+0000 7f85d9ada640 1 --2- 172.21.15.57:0/3563587324 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f85d40cf340 0x7f85d40caa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:40.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.045+0000 7f85d9ada640 1 -- 172.21.15.57:0/3563587324 >> 172.21.15.57:0/3563587324 conn(0x7f85d40d0d60 msgr2=0x7f85d41e3870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:40.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.045+0000 7f85d9ada640 1 -- 172.21.15.57:0/3563587324 shutdown_connections 2023-10-19T14:21:40.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.045+0000 7f85d9ada640 1 -- 172.21.15.57:0/3563587324 wait complete. 2023-10-19T14:21:40.058 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-19T14:21:40.061 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:40.058+0000 7fa0a5617040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-19T14:21:40.106 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:21:40.104+0000 7f0f9ee3e040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-19T14:21:40.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.167+0000 7f524aac9640 1 -- 172.21.15.57:0/1526175722 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f524415b2a0 msgr2=0x7f52440d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:40.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.167+0000 7f524aac9640 1 --2- 172.21.15.57:0/1526175722 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f524415b2a0 0x7f52440d1910 secure :-1 s=READY pgs=2748 cs=0 l=1 rev1=1 crypto rx=0x7f5230009830 tx=0x7f523002bd60 comp rx=0 tx=0).stop 2023-10-19T14:21:40.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.167+0000 7f524aac9640 1 -- 172.21.15.57:0/1526175722 shutdown_connections 2023-10-19T14:21:40.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.167+0000 7f524aac9640 1 --2- 172.21.15.57:0/1526175722 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f524415b2a0 0x7f52440d1910 unknown :-1 s=CLOSED pgs=2748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:40.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.167+0000 7f524aac9640 1 -- 172.21.15.57:0/1526175722 >> 172.21.15.57:0/1526175722 conn(0x7f52440d0d60 msgr2=0x7f52440d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:40.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.167+0000 7f524aac9640 1 -- 172.21.15.57:0/1526175722 shutdown_connections 2023-10-19T14:21:40.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.167+0000 7f524aac9640 1 -- 172.21.15.57:0/1526175722 wait complete. 2023-10-19T14:21:40.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.168+0000 7f524aac9640 1 Processor -- start 2023-10-19T14:21:40.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.168+0000 7f524aac9640 1 -- start start 2023-10-19T14:21:40.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.169+0000 7f524aac9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f52440ceb40 0x7f5244151200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:40.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.169+0000 7f524aac9640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f52440cf340 0x7f52440b38e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:40.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.169+0000 7f524aac9640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f524415b2a0 0x7f52440b0ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:40.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.169+0000 7f524aac9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5244151740 con 0x7f52440ceb40 2023-10-19T14:21:40.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.169+0000 7f524aac9640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f524415d370 con 0x7f524415b2a0 2023-10-19T14:21:40.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.169+0000 7f524aac9640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f524415c760 con 0x7f52440cf340 2023-10-19T14:21:40.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.169+0000 7f5248865640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f52440ceb40 0x7f5244151200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:40.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.169+0000 7f5248865640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f52440ceb40 0x7f5244151200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43944/0 (socket says 172.21.15.57:43944) 2023-10-19T14:21:40.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.169+0000 7f5248865640 1 -- 172.21.15.57:0/1796131535 learned_addr learned my addr 172.21.15.57:0/1796131535 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:40.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.169+0000 7f5243fff640 1 --2- 172.21.15.57:0/1796131535 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f52440cf340 0x7f52440b38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:40.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.169+0000 7f5249066640 1 --2- 172.21.15.57:0/1796131535 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f524415b2a0 0x7f52440b0ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:40.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.169+0000 7f5248865640 1 -- 172.21.15.57:0/1796131535 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f52440cf340 msgr2=0x7f52440b38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:40.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.169+0000 7f5248865640 1 --2- 172.21.15.57:0/1796131535 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f52440cf340 0x7f52440b38e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:40.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.169+0000 7f5248865640 1 -- 172.21.15.57:0/1796131535 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f524415b2a0 msgr2=0x7f52440b0ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:40.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.169+0000 7f5248865640 1 --2- 172.21.15.57:0/1796131535 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f524415b2a0 0x7f52440b0ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:40.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.169+0000 7f5248865640 1 -- 172.21.15.57:0/1796131535 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5230009490 con 0x7f52440ceb40 2023-10-19T14:21:40.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.170+0000 7f5248865640 1 --2- 172.21.15.57:0/1796131535 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f52440ceb40 0x7f5244151200 secure :-1 s=READY pgs=2749 cs=0 l=1 rev1=1 crypto rx=0x7f523800c500 tx=0x7f523800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:40.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.170+0000 7f52417fa640 1 -- 172.21.15.57:0/1796131535 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5238012830 con 0x7f52440ceb40 2023-10-19T14:21:40.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.170+0000 7f52417fa640 1 -- 172.21.15.57:0/1796131535 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f5238012a80 con 0x7f52440ceb40 2023-10-19T14:21:40.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.170+0000 7f524aac9640 1 -- 172.21.15.57:0/1796131535 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f524415ca40 con 0x7f52440ceb40 2023-10-19T14:21:40.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.170+0000 7f52417fa640 1 -- 172.21.15.57:0/1796131535 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5238012e50 con 0x7f52440ceb40 2023-10-19T14:21:40.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.170+0000 7f524aac9640 1 -- 172.21.15.57:0/1796131535 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f524415d000 con 0x7f52440ceb40 2023-10-19T14:21:40.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.171+0000 7f524aac9640 1 -- 172.21.15.57:0/1796131535 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52440cca00 con 0x7f52440ceb40 2023-10-19T14:21:40.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.171+0000 7f52417fa640 1 -- 172.21.15.57:0/1796131535 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 141) v1 ==== 89434+0+0 (secure 0 0 0) 0x7f523801b400 con 0x7f52440ceb40 2023-10-19T14:21:40.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.172+0000 7f52417fa640 1 --2- 172.21.15.57:0/1796131535 >> [v2:172.21.15.179:6832/609071913,v1:172.21.15.179:6833/609071913] conn(0x7f522806b080 0x7f522806d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:40.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.172+0000 7f52417fa640 1 -- 172.21.15.57:0/1796131535 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 7351+0+0 (secure 0 0 0) 0x7f523808a050 con 0x7f52440ceb40 2023-10-19T14:21:40.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.172+0000 7f5243fff640 1 -- 172.21.15.57:0/1796131535 >> [v2:172.21.15.179:6832/609071913,v1:172.21.15.179:6833/609071913] conn(0x7f522806b080 msgr2=0x7f522806d540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/609071913 2023-10-19T14:21:40.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.172+0000 7f5243fff640 1 --2- 172.21.15.57:0/1796131535 >> [v2:172.21.15.179:6832/609071913,v1:172.21.15.179:6833/609071913] conn(0x7f522806b080 0x7f522806d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T14:21:40.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.180+0000 7f52417fa640 1 -- 172.21.15.57:0/1796131535 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f52380548c0 con 0x7f52440ceb40 2023-10-19T14:21:40.197 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:40.195+0000 7fa0a5617040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-19T14:21:40.208 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:21:40.206+0000 7f0f9ee3e040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-19T14:21:40.238 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:40.236+0000 7fa0a5617040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-19T14:21:40.344 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:40.341+0000 7fa0a5617040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-19T14:21:40.353 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:21:40.351+0000 7f0f9ee3e040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-19T14:21:40.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.371+0000 7f524aac9640 1 -- 172.21.15.57:0/1796131535 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f52440690e0 con 0x7f52440ceb40 2023-10-19T14:21:40.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.372+0000 7f52417fa640 1 -- 172.21.15.57:0/1796131535 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v248) v1 ==== 81+0+594 (secure 0 0 0) 0x7f5238016270 con 0x7f52440ceb40 2023-10-19T14:21:40.374 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/2525508070 2023-10-20T14:21:35.927420+0000 2023-10-19T14:21:40.374 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/720769249 2023-10-20T14:21:35.927420+0000 2023-10-19T14:21:40.375 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:6833/3547895240 2023-10-20T14:21:35.927420+0000 2023-10-19T14:21:40.375 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:6832/3547895240 2023-10-20T14:21:35.927420+0000 2023-10-19T14:21:40.375 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/177563067 2023-10-20T14:21:35.927420+0000 2023-10-19T14:21:40.375 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/1877726906 2023-10-20T14:21:35.927420+0000 2023-10-19T14:21:40.375 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/3471744208 2023-10-20T14:21:35.927420+0000 2023-10-19T14:21:40.375 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/2477290155 2023-10-20T14:21:35.927420+0000 2023-10-19T14:21:40.376 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/1240052109 2023-10-20T14:21:35.927420+0000 2023-10-19T14:21:40.376 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/4042925688 2023-10-20T14:21:35.927420+0000 2023-10-19T14:21:40.377 INFO:teuthology.orchestra.run.smithi057.stderr:listed 10 entries 2023-10-19T14:21:40.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.373+0000 7f5243fff640 1 -- 172.21.15.57:0/1796131535 >> [v2:172.21.15.179:6832/609071913,v1:172.21.15.179:6833/609071913] conn(0x7f522806b080 msgr2=0x7f522806d540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/609071913 2023-10-19T14:21:40.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.373+0000 7f5243fff640 1 --2- 172.21.15.57:0/1796131535 >> [v2:172.21.15.179:6832/609071913,v1:172.21.15.179:6833/609071913] conn(0x7f522806b080 0x7f522806d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-19T14:21:40.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.374+0000 7f524aac9640 1 -- 172.21.15.57:0/1796131535 >> [v2:172.21.15.179:6832/609071913,v1:172.21.15.179:6833/609071913] conn(0x7f522806b080 msgr2=0x7f522806d540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T14:21:40.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.374+0000 7f524aac9640 1 --2- 172.21.15.57:0/1796131535 >> [v2:172.21.15.179:6832/609071913,v1:172.21.15.179:6833/609071913] conn(0x7f522806b080 0x7f522806d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:40.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.374+0000 7f524aac9640 1 -- 172.21.15.57:0/1796131535 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f52440ceb40 msgr2=0x7f5244151200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:40.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.374+0000 7f524aac9640 1 --2- 172.21.15.57:0/1796131535 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f52440ceb40 0x7f5244151200 secure :-1 s=READY pgs=2749 cs=0 l=1 rev1=1 crypto rx=0x7f523800c500 tx=0x7f523800c9c0 comp rx=0 tx=0).stop 2023-10-19T14:21:40.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.374+0000 7f524aac9640 1 -- 172.21.15.57:0/1796131535 shutdown_connections 2023-10-19T14:21:40.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.374+0000 7f524aac9640 1 --2- 172.21.15.57:0/1796131535 >> [v2:172.21.15.179:6832/609071913,v1:172.21.15.179:6833/609071913] conn(0x7f522806b080 0x7f522806d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:40.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.374+0000 7f524aac9640 1 --2- 172.21.15.57:0/1796131535 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f524415b2a0 0x7f52440b0ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:40.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.374+0000 7f524aac9640 1 --2- 172.21.15.57:0/1796131535 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f52440cf340 0x7f52440b38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:40.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.374+0000 7f524aac9640 1 --2- 172.21.15.57:0/1796131535 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f52440ceb40 0x7f5244151200 unknown :-1 s=CLOSED pgs=2749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:40.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.374+0000 7f524aac9640 1 -- 172.21.15.57:0/1796131535 >> 172.21.15.57:0/1796131535 conn(0x7f52440d0d60 msgr2=0x7f52440b0630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:40.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.374+0000 7f524aac9640 1 -- 172.21.15.57:0/1796131535 shutdown_connections 2023-10-19T14:21:40.380 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.374+0000 7f524aac9640 1 -- 172.21.15.57:0/1796131535 wait complete. 2023-10-19T14:21:40.387 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi057.front.sepia.ceph.com... 2023-10-19T14:21:40.388 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:21:40.388 DEBUG:teuthology.orchestra.run.smithi057:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-10-19T14:21:40.398 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:21:40.397+0000 7f0f9ee3e040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-19T14:21:40.417 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi057.stderr:ceph-fuse[158769]: fuse finished with error 0 and tester_r 0 2023-10-19T14:21:40.419 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi057.front.sepia.ceph.com 2023-10-19T14:21:40.420 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:21:40.420 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-19T14:21:40.513 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:21:40.510+0000 7f0f9ee3e040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-19T14:21:40.532 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-19T14:21:40.532 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T14:21:40.547 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054833,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=284 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3247632k,nr_inodes=811908,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-19T14:21:40.548 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi179.front.sepia.ceph.com 2023-10-19T14:21:40.548 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:21:40.549 DEBUG:teuthology.orchestra.run.smithi179:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-10-19T14:21:40.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-19T14:21:40.602 INFO:teuthology.orchestra.run.smithi179.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2023-10-19T14:21:40.603 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T14:21:40.603 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T14:21:40.619 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055614,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=358 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248260k,nr_inodes=812065,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-19T14:21:40.619 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi057.front.sepia.ceph.com 2023-10-19T14:21:40.619 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:21:40.620 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-10-19T14:21:40.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-19T14:21:40.674 INFO:teuthology.orchestra.run.smithi057.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2023-10-19T14:21:40.674 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-19T14:21:40.675 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T14:21:40.690 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054833,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=284 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3247632k,nr_inodes=811908,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-19T14:21:40.691 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi179.front.sepia.ceph.com 2023-10-19T14:21:40.691 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:21:40.691 DEBUG:teuthology.orchestra.run.smithi179:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-10-19T14:21:40.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-19T14:21:40.746 INFO:teuthology.orchestra.run.smithi179.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2023-10-19T14:21:40.746 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T14:21:40.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.853+0000 7fbf87226640 1 -- 172.21.15.57:0/411441587 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbf800b0c40 msgr2=0x7fbf80158300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:40.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.853+0000 7fbf87226640 1 --2- 172.21.15.57:0/411441587 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbf800b0c40 0x7fbf80158300 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fbf78005fd0 tx=0x7fbf7802c170 comp rx=0 tx=0).stop 2023-10-19T14:21:40.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.853+0000 7fbf87226640 1 -- 172.21.15.57:0/411441587 shutdown_connections 2023-10-19T14:21:40.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.853+0000 7fbf87226640 1 --2- 172.21.15.57:0/411441587 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbf800b0c40 0x7fbf80158300 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:40.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.853+0000 7fbf87226640 1 -- 172.21.15.57:0/411441587 >> 172.21.15.57:0/411441587 conn(0x7fbf80069930 msgr2=0x7fbf80069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:40.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.853+0000 7fbf87226640 1 -- 172.21.15.57:0/411441587 shutdown_connections 2023-10-19T14:21:40.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.853+0000 7fbf87226640 1 -- 172.21.15.57:0/411441587 wait complete. 2023-10-19T14:21:40.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.854+0000 7fbf87226640 1 Processor -- start 2023-10-19T14:21:40.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.854+0000 7fbf87226640 1 -- start start 2023-10-19T14:21:40.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.855+0000 7fbf87226640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbf800afa20 0x7fbf8014be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:40.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.855+0000 7fbf87226640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbf800b0c40 0x7fbf8013c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:40.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.855+0000 7fbf87226640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbf800cbd20 0x7fbf80139320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:40.860 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.855+0000 7fbf87226640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf801e2dd0 con 0x7fbf800cbd20 2023-10-19T14:21:40.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.855+0000 7fbf87226640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf8013ce70 con 0x7fbf800afa20 2023-10-19T14:21:40.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.855+0000 7fbf87226640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf80139860 con 0x7fbf800b0c40 2023-10-19T14:21:40.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.855+0000 7fbf857c3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbf800cbd20 0x7fbf80139320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:40.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.855+0000 7fbf857c3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbf800cbd20 0x7fbf80139320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43958/0 (socket says 172.21.15.57:43958) 2023-10-19T14:21:40.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.855+0000 7fbf857c3640 1 -- 172.21.15.57:0/855471922 learned_addr learned my addr 172.21.15.57:0/855471922 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:40.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.855+0000 7fbf7ffff640 1 --2- 172.21.15.57:0/855471922 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbf800b0c40 0x7fbf8013c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:40.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.855+0000 7fbf84fc2640 1 --2- 172.21.15.57:0/855471922 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbf800afa20 0x7fbf8014be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:40.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.856+0000 7fbf857c3640 1 -- 172.21.15.57:0/855471922 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbf800b0c40 msgr2=0x7fbf8013c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:40.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.856+0000 7fbf857c3640 1 --2- 172.21.15.57:0/855471922 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbf800b0c40 0x7fbf8013c930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:40.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.856+0000 7fbf857c3640 1 -- 172.21.15.57:0/855471922 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbf800afa20 msgr2=0x7fbf8014be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:40.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.856+0000 7fbf857c3640 1 --2- 172.21.15.57:0/855471922 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbf800afa20 0x7fbf8014be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:40.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.856+0000 7fbf857c3640 1 -- 172.21.15.57:0/855471922 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf78009840 con 0x7fbf800cbd20 2023-10-19T14:21:40.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.856+0000 7fbf857c3640 1 --2- 172.21.15.57:0/855471922 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbf800cbd20 0x7fbf80139320 secure :-1 s=READY pgs=2751 cs=0 l=1 rev1=1 crypto rx=0x7fbf78005fd0 tx=0x7fbf780042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:40.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.856+0000 7fbf7d7fa640 1 -- 172.21.15.57:0/855471922 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf78003e80 con 0x7fbf800cbd20 2023-10-19T14:21:40.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.856+0000 7fbf7d7fa640 1 -- 172.21.15.57:0/855471922 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fbf780047e0 con 0x7fbf800cbd20 2023-10-19T14:21:40.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.856+0000 7fbf87226640 1 -- 172.21.15.57:0/855471922 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf8014c530 con 0x7fbf800cbd20 2023-10-19T14:21:40.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.856+0000 7fbf7d7fa640 1 -- 172.21.15.57:0/855471922 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf78034800 con 0x7fbf800cbd20 2023-10-19T14:21:40.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.856+0000 7fbf87226640 1 -- 172.21.15.57:0/855471922 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf8014caf0 con 0x7fbf800cbd20 2023-10-19T14:21:40.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.857+0000 7fbf87226640 1 -- 172.21.15.57:0/855471922 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf800690e0 con 0x7fbf800cbd20 2023-10-19T14:21:40.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.858+0000 7fbf7d7fa640 1 -- 172.21.15.57:0/855471922 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 141) v1 ==== 89434+0+0 (secure 0 0 0) 0x7fbf78034960 con 0x7fbf800cbd20 2023-10-19T14:21:40.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.858+0000 7fbf7d7fa640 1 --2- 172.21.15.57:0/855471922 >> [v2:172.21.15.179:6832/609071913,v1:172.21.15.179:6833/609071913] conn(0x7fbf6406b080 0x7fbf6406d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:40.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.858+0000 7fbf7d7fa640 1 -- 172.21.15.57:0/855471922 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 7351+0+0 (secure 0 0 0) 0x7fbf780ad0b0 con 0x7fbf800cbd20 2023-10-19T14:21:40.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.858+0000 7fbf84fc2640 1 -- 172.21.15.57:0/855471922 >> [v2:172.21.15.179:6832/609071913,v1:172.21.15.179:6833/609071913] conn(0x7fbf6406b080 msgr2=0x7fbf6406d540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/609071913 2023-10-19T14:21:40.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.858+0000 7fbf84fc2640 1 --2- 172.21.15.57:0/855471922 >> [v2:172.21.15.179:6832/609071913,v1:172.21.15.179:6833/609071913] conn(0x7fbf6406b080 0x7fbf6406d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T14:21:40.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:40.863+0000 7fbf7d7fa640 1 -- 172.21.15.57:0/855471922 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fbf780778d0 con 0x7fbf800cbd20 2023-10-19T14:21:40.976 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:40.973+0000 7fa0a5617040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-19T14:21:41.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.059+0000 7fbf84fc2640 1 -- 172.21.15.57:0/855471922 >> [v2:172.21.15.179:6832/609071913,v1:172.21.15.179:6833/609071913] conn(0x7fbf6406b080 msgr2=0x7fbf6406d540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/609071913 2023-10-19T14:21:41.062 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.059+0000 7fbf84fc2640 1 --2- 172.21.15.57:0/855471922 >> [v2:172.21.15.179:6832/609071913,v1:172.21.15.179:6833/609071913] conn(0x7fbf6406b080 0x7fbf6406d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-19T14:21:41.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.076+0000 7fbf87226640 1 -- 172.21.15.57:0/855471922 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbf8006f700 con 0x7fbf800cbd20 2023-10-19T14:21:41.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.077+0000 7fbf7d7fa640 1 -- 172.21.15.57:0/855471922 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+6140 (secure 0 0 0) 0x7fbf7807aef0 con 0x7fbf800cbd20 2023-10-19T14:21:41.080 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:21:41.080 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":107,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9761,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/325888554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":325888554},{"type":"v1","addr":"172.21.15.179:6836","nonce":325888554}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":100},{"gid":9764,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/8038232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":8038232},{"type":"v1","addr":"172.21.15.179:6837","nonce":8038232}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":100}],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T14:21:09.759935+0000","modified":"2023-10-19T14:21:11.768676+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":12564},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12564":{"gid":12564,"name":"c","rank":0,"incarnation":101,"state":"up:active","state_seq":3,"addr":"172.21.15.57:6835/18187246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":18187246},{"type":"v1","addr":"172.21.15.57:6835","nonce":18187246}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T14:21:26.724728+0000","modified":"2023-10-19T14:21:28.872856+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":12555},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12555":{"gid":12555,"name":"a","rank":0,"incarnation":105,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6833/4256498777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":4256498777},{"type":"v1","addr":"172.21.15.57:6833","nonce":4256498777}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2023-10-19T14:21:41.080 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 107 2023-10-19T14:21:41.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.079+0000 7fbf87226640 1 -- 172.21.15.57:0/855471922 >> [v2:172.21.15.179:6832/609071913,v1:172.21.15.179:6833/609071913] conn(0x7fbf6406b080 msgr2=0x7fbf6406d540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T14:21:41.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.079+0000 7fbf87226640 1 --2- 172.21.15.57:0/855471922 >> [v2:172.21.15.179:6832/609071913,v1:172.21.15.179:6833/609071913] conn(0x7fbf6406b080 0x7fbf6406d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:41.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.079+0000 7fbf87226640 1 -- 172.21.15.57:0/855471922 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbf800cbd20 msgr2=0x7fbf80139320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:41.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.079+0000 7fbf87226640 1 --2- 172.21.15.57:0/855471922 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbf800cbd20 0x7fbf80139320 secure :-1 s=READY pgs=2751 cs=0 l=1 rev1=1 crypto rx=0x7fbf78005fd0 tx=0x7fbf780042f0 comp rx=0 tx=0).stop 2023-10-19T14:21:41.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.079+0000 7fbf87226640 1 -- 172.21.15.57:0/855471922 shutdown_connections 2023-10-19T14:21:41.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.079+0000 7fbf87226640 1 --2- 172.21.15.57:0/855471922 >> [v2:172.21.15.179:6832/609071913,v1:172.21.15.179:6833/609071913] conn(0x7fbf6406b080 0x7fbf6406d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:41.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.079+0000 7fbf87226640 1 --2- 172.21.15.57:0/855471922 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbf800cbd20 0x7fbf80139320 unknown :-1 s=CLOSED pgs=2751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:41.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.079+0000 7fbf87226640 1 --2- 172.21.15.57:0/855471922 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbf800b0c40 0x7fbf8013c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:41.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.079+0000 7fbf87226640 1 --2- 172.21.15.57:0/855471922 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbf800afa20 0x7fbf8014be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:41.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.079+0000 7fbf87226640 1 -- 172.21.15.57:0/855471922 >> 172.21.15.57:0/855471922 conn(0x7fbf80069930 msgr2=0x7fbf800ae700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:41.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.079+0000 7fbf87226640 1 -- 172.21.15.57:0/855471922 shutdown_connections 2023-10-19T14:21:41.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.079+0000 7fbf87226640 1 -- 172.21.15.57:0/855471922 wait complete. 2023-10-19T14:21:41.093 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T14:21:41.189 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:21:41.187+0000 7f0f9ee3e040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-19T14:21:41.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.200+0000 7fd5ef6ec640 1 -- 172.21.15.57:0/875876000 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5e80cdb30 msgr2=0x7fd5e80cdfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:41.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.200+0000 7fd5ef6ec640 1 --2- 172.21.15.57:0/875876000 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5e80cdb30 0x7fd5e80cdfb0 secure :-1 s=READY pgs=2752 cs=0 l=1 rev1=1 crypto rx=0x7fd5e80ceab0 tx=0x7fd5d802bd10 comp rx=0 tx=0).stop 2023-10-19T14:21:41.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.200+0000 7fd5ef6ec640 1 -- 172.21.15.57:0/875876000 shutdown_connections 2023-10-19T14:21:41.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.200+0000 7fd5ef6ec640 1 --2- 172.21.15.57:0/875876000 >> v2:172.21.15.179:3300/0 conn(0x7fd5e815ccf0 0x7fd5e815d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:41.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.200+0000 7fd5ef6ec640 1 --2- 172.21.15.57:0/875876000 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5e80cdb30 0x7fd5e80cdfb0 unknown :-1 s=CLOSED pgs=2752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:41.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.200+0000 7fd5ef6ec640 1 --2- 172.21.15.57:0/875876000 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd5e8159060 0x7fd5e806d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:41.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.200+0000 7fd5ef6ec640 1 -- 172.21.15.57:0/875876000 >> 172.21.15.57:0/875876000 conn(0x7fd5e80b0670 msgr2=0x7fd5e80b0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:41.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.200+0000 7fd5ef6ec640 1 -- 172.21.15.57:0/875876000 shutdown_connections 2023-10-19T14:21:41.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.200+0000 7fd5ef6ec640 1 -- 172.21.15.57:0/875876000 wait complete. 2023-10-19T14:21:41.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.201+0000 7fd5ef6ec640 1 Processor -- start 2023-10-19T14:21:41.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.201+0000 7fd5ef6ec640 1 -- start start 2023-10-19T14:21:41.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.202+0000 7fd5ef6ec640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd5e80cdb30 0x7fd5e80c5bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:41.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.202+0000 7fd5ef6ec640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd5e8159060 0x7fd5e80c4490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:41.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.202+0000 7fd5ef6ec640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5e815ccf0 0x7fd5e8150490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:41.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.202+0000 7fd5ef6ec640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5e80c49d0 con 0x7fd5e815ccf0 2023-10-19T14:21:41.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.202+0000 7fd5ef6ec640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5e80c2610 con 0x7fd5e80cdb30 2023-10-19T14:21:41.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.202+0000 7fd5ef6ec640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5e81509d0 con 0x7fd5e8159060 2023-10-19T14:21:41.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.202+0000 7fd5edc89640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5e815ccf0 0x7fd5e8150490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:41.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.202+0000 7fd5ed488640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd5e80cdb30 0x7fd5e80c5bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:41.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.202+0000 7fd5edc89640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5e815ccf0 0x7fd5e8150490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43978/0 (socket says 172.21.15.57:43978) 2023-10-19T14:21:41.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.202+0000 7fd5ecc87640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd5e8159060 0x7fd5e80c4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:41.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.202+0000 7fd5edc89640 1 -- 172.21.15.57:0/3801232150 learned_addr learned my addr 172.21.15.57:0/3801232150 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:41.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.202+0000 7fd5edc89640 1 -- 172.21.15.57:0/3801232150 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd5e8159060 msgr2=0x7fd5e80c4490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:41.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.202+0000 7fd5edc89640 1 --2- 172.21.15.57:0/3801232150 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd5e8159060 0x7fd5e80c4490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:41.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.202+0000 7fd5edc89640 1 -- 172.21.15.57:0/3801232150 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd5e80cdb30 msgr2=0x7fd5e80c5bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:41.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.202+0000 7fd5edc89640 1 --2- 172.21.15.57:0/3801232150 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd5e80cdb30 0x7fd5e80c5bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:41.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.202+0000 7fd5edc89640 1 -- 172.21.15.57:0/3801232150 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5d8009560 con 0x7fd5e815ccf0 2023-10-19T14:21:41.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.203+0000 7fd5edc89640 1 --2- 172.21.15.57:0/3801232150 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5e815ccf0 0x7fd5e8150490 secure :-1 s=READY pgs=2753 cs=0 l=1 rev1=1 crypto rx=0x7fd5e000eab0 tx=0x7fd5e000ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:41.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.203+0000 7fd5ddffb640 1 -- 172.21.15.57:0/3801232150 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5e000cea0 con 0x7fd5e815ccf0 2023-10-19T14:21:41.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.203+0000 7fd5ddffb640 1 -- 172.21.15.57:0/3801232150 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fd5e0004b20 con 0x7fd5e815ccf0 2023-10-19T14:21:41.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.203+0000 7fd5ef6ec640 1 -- 172.21.15.57:0/3801232150 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5e80c5000 con 0x7fd5e815ccf0 2023-10-19T14:21:41.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.203+0000 7fd5ddffb640 1 -- 172.21.15.57:0/3801232150 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5e0005020 con 0x7fd5e815ccf0 2023-10-19T14:21:41.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.203+0000 7fd5ef6ec640 1 -- 172.21.15.57:0/3801232150 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5e80afc20 con 0x7fd5e815ccf0 2023-10-19T14:21:41.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.204+0000 7fd5ef6ec640 1 -- 172.21.15.57:0/3801232150 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5e80c27c0 con 0x7fd5e815ccf0 2023-10-19T14:21:41.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.204+0000 7fd5ddffb640 1 -- 172.21.15.57:0/3801232150 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 141) v1 ==== 89434+0+0 (secure 0 0 0) 0x7fd5e0005180 con 0x7fd5e815ccf0 2023-10-19T14:21:41.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.205+0000 7fd5ddffb640 1 --2- 172.21.15.57:0/3801232150 >> [v2:172.21.15.179:6832/609071913,v1:172.21.15.179:6833/609071913] conn(0x7fd5cc06b080 0x7fd5cc06d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:41.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.205+0000 7fd5ddffb640 1 -- 172.21.15.57:0/3801232150 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 7351+0+0 (secure 0 0 0) 0x7fd5e0014070 con 0x7fd5e815ccf0 2023-10-19T14:21:41.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.205+0000 7fd5ed488640 1 -- 172.21.15.57:0/3801232150 >> [v2:172.21.15.179:6832/609071913,v1:172.21.15.179:6833/609071913] conn(0x7fd5cc06b080 msgr2=0x7fd5cc06d540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/609071913 2023-10-19T14:21:41.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.205+0000 7fd5ed488640 1 --2- 172.21.15.57:0/3801232150 >> [v2:172.21.15.179:6832/609071913,v1:172.21.15.179:6833/609071913] conn(0x7fd5cc06b080 0x7fd5cc06d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T14:21:41.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.209+0000 7fd5ddffb640 1 -- 172.21.15.57:0/3801232150 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd5e80c27c0 con 0x7fd5e815ccf0 2023-10-19T14:21:41.236 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:41.234+0000 7fa0a5617040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-19T14:21:41.284 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:41.282+0000 7fa0a5617040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-19T14:21:41.383 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:41.380+0000 7fa0a5617040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-19T14:21:41.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.388+0000 7fd5ef6ec640 1 -- 172.21.15.57:0/3801232150 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd5e80c27c0 con 0x7fd5e815ccf0 2023-10-19T14:21:41.392 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.389+0000 7fd5ddffb640 1 -- 172.21.15.57:0/3801232150 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v248) v1 ==== 74+0+21132 (secure 0 0 0) 0x7fd5e80c27c0 con 0x7fd5e815ccf0 2023-10-19T14:21:41.392 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:21:41.393 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":248,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T14:21:35.927462+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2023-10-19T14:21:08.101023+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"242","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2023-10-19T14:21:08.745817+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"243","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_metadata","create_time":"2023-10-19T14:21:24.384693+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"246","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"backup_fs_data","create_time":"2023-10-19T14:21:25.717524+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"247","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.179:0/2525508070":"2023-10-20T14:21:35.927420+0000","172.21.15.179:0/720769249":"2023-10-20T14:21:35.927420+0000","172.21.15.179:6833/3547895240":"2023-10-20T14:21:35.927420+0000","172.21.15.179:6832/3547895240":"2023-10-20T14:21:35.927420+0000","172.21.15.179:0/177563067":"2023-10-20T14:21:35.927420+0000","172.21.15.179:0/1877726906":"2023-10-20T14:21:35.927420+0000","172.21.15.179:0/3471744208":"2023-10-20T14:21:35.927420+0000","172.21.15.179:0/2477290155":"2023-10-20T14:21:35.927420+0000","172.21.15.179:0/1240052109":"2023-10-20T14:21:35.927420+0000","172.21.15.179:0/4042925688":"2023-10-20T14:21:35.927420+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T14:21:41.394 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.391+0000 7fd5ef6ec640 1 -- 172.21.15.57:0/3801232150 >> [v2:172.21.15.179:6832/609071913,v1:172.21.15.179:6833/609071913] conn(0x7fd5cc06b080 msgr2=0x7fd5cc06d540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T14:21:41.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.391+0000 7fd5ef6ec640 1 --2- 172.21.15.57:0/3801232150 >> [v2:172.21.15.179:6832/609071913,v1:172.21.15.179:6833/609071913] conn(0x7fd5cc06b080 0x7fd5cc06d540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:41.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.391+0000 7fd5ef6ec640 1 -- 172.21.15.57:0/3801232150 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5e815ccf0 msgr2=0x7fd5e8150490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:41.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.391+0000 7fd5ef6ec640 1 --2- 172.21.15.57:0/3801232150 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5e815ccf0 0x7fd5e8150490 secure :-1 s=READY pgs=2753 cs=0 l=1 rev1=1 crypto rx=0x7fd5e000eab0 tx=0x7fd5e000ef70 comp rx=0 tx=0).stop 2023-10-19T14:21:41.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.391+0000 7fd5ef6ec640 1 -- 172.21.15.57:0/3801232150 shutdown_connections 2023-10-19T14:21:41.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.391+0000 7fd5ef6ec640 1 --2- 172.21.15.57:0/3801232150 >> [v2:172.21.15.179:6832/609071913,v1:172.21.15.179:6833/609071913] conn(0x7fd5cc06b080 0x7fd5cc06d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:41.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.391+0000 7fd5ef6ec640 1 --2- 172.21.15.57:0/3801232150 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5e815ccf0 0x7fd5e8150490 unknown :-1 s=CLOSED pgs=2753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:41.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.391+0000 7fd5ef6ec640 1 --2- 172.21.15.57:0/3801232150 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd5e8159060 0x7fd5e80c4490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:41.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.391+0000 7fd5ef6ec640 1 --2- 172.21.15.57:0/3801232150 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd5e80cdb30 0x7fd5e80c5bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:41.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.391+0000 7fd5ef6ec640 1 -- 172.21.15.57:0/3801232150 >> 172.21.15.57:0/3801232150 conn(0x7fd5e80b0670 msgr2=0x7fd5e80ab950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:41.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.391+0000 7fd5ef6ec640 1 -- 172.21.15.57:0/3801232150 shutdown_connections 2023-10-19T14:21:41.397 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.391+0000 7fd5ef6ec640 1 -- 172.21.15.57:0/3801232150 wait complete. 2023-10-19T14:21:41.408 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-19T12:44:07.371186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 31, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-19T14:21:08.101023+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '242', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 32, 'pool_name': 'cephfs_data', 'create_time': '2023-10-19T14:21:08.745817+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '243', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 33, 'pool_name': 'backup_fs_metadata', 'create_time': '2023-10-19T14:21:24.384693+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '246', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 34, 'pool_name': 'backup_fs_data', 'create_time': '2023-10-19T14:21:25.717524+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '247', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-19T14:21:41.409 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-19T14:21:41.467 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:41.465+0000 7fa027618640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:21:41.467 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:41.465+0000 7fa027618640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:21:41.468 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:41.465+0000 7fa027618640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:21:41.468 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:41.465+0000 7fa027618640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:21:41.468 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:41.465+0000 7fa027618640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:21:41.469 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:41.466+0000 7fa02a61e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:21:41.469 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:41.466+0000 7fa02a61e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:21:41.469 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:41.466+0000 7fa02a61e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:21:41.469 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:41.466+0000 7fa02a61e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:21:41.470 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:41.466+0000 7fa02a61e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:21:41.477 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:41.474+0000 7fa027618640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:21:41.477 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:41.474+0000 7fa027618640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:21:41.477 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:41.474+0000 7fa027618640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:21:41.478 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:41.474+0000 7fa027618640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:21:41.478 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:41.474+0000 7fa027618640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:21:41.478 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:41.474+0000 7fa02a61e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:21:41.478 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:41.474+0000 7fa02a61e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:21:41.478 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:41.474+0000 7fa02a61e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:21:41.478 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:41.474+0000 7fa02a61e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:21:41.479 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:21:41.474+0000 7fa02a61e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:21:41.483 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:21:41.480+0000 7f0f9ee3e040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-19T14:21:41.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.523+0000 7ff53fb97640 1 -- 172.21.15.57:0/3320710373 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff5380c96b0 msgr2=0x7ff5380c9b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:41.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.523+0000 7ff53fb97640 1 --2- 172.21.15.57:0/3320710373 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff5380c96b0 0x7ff5380c9b60 secure :-1 s=READY pgs=2755 cs=0 l=1 rev1=1 crypto rx=0x7ff52c009830 tx=0x7ff52c02bdf0 comp rx=0 tx=0).stop 2023-10-19T14:21:41.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.523+0000 7ff53fb97640 1 -- 172.21.15.57:0/3320710373 shutdown_connections 2023-10-19T14:21:41.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.523+0000 7ff53fb97640 1 --2- 172.21.15.57:0/3320710373 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff5380c96b0 0x7ff5380c9b60 unknown :-1 s=CLOSED pgs=2755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:41.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.523+0000 7ff53fb97640 1 -- 172.21.15.57:0/3320710373 >> 172.21.15.57:0/3320710373 conn(0x7ff538069930 msgr2=0x7ff538069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:41.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.523+0000 7ff53fb97640 1 -- 172.21.15.57:0/3320710373 shutdown_connections 2023-10-19T14:21:41.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.523+0000 7ff53fb97640 1 -- 172.21.15.57:0/3320710373 wait complete. 2023-10-19T14:21:41.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.525+0000 7ff53fb97640 1 Processor -- start 2023-10-19T14:21:41.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.525+0000 7ff53fb97640 1 -- start start 2023-10-19T14:21:41.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.525+0000 7ff53fb97640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff5380b1920 0x7ff53814cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:41.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.525+0000 7ff53fb97640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff5380c96b0 0x7ff53815dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:41.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.525+0000 7ff53fb97640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff538154af0 0x7ff53814d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:41.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.525+0000 7ff53fb97640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff538158860 con 0x7ff5380c96b0 2023-10-19T14:21:41.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.525+0000 7ff53fb97640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff53814e620 con 0x7ff5380b1920 2023-10-19T14:21:41.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.525+0000 7ff53fb97640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff53814dba0 con 0x7ff538154af0 2023-10-19T14:21:41.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.525+0000 7ff53d132640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff5380c96b0 0x7ff53815dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:41.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.525+0000 7ff53d132640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff5380c96b0 0x7ff53815dcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43992/0 (socket says 172.21.15.57:43992) 2023-10-19T14:21:41.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.525+0000 7ff53d132640 1 -- 172.21.15.57:0/466795974 learned_addr learned my addr 172.21.15.57:0/466795974 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:41.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.526+0000 7ff53d933640 1 --2- 172.21.15.57:0/466795974 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff5380b1920 0x7ff53814cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:41.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.526+0000 7ff53e134640 1 --2- 172.21.15.57:0/466795974 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff538154af0 0x7ff53814d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:41.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.526+0000 7ff53d132640 1 -- 172.21.15.57:0/466795974 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff538154af0 msgr2=0x7ff53814d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:41.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.526+0000 7ff53d132640 1 --2- 172.21.15.57:0/466795974 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff538154af0 0x7ff53814d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:41.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.526+0000 7ff53d132640 1 -- 172.21.15.57:0/466795974 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff5380b1920 msgr2=0x7ff53814cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:41.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.526+0000 7ff53d132640 1 --2- 172.21.15.57:0/466795974 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff5380b1920 0x7ff53814cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:41.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.526+0000 7ff53d132640 1 -- 172.21.15.57:0/466795974 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff52c009490 con 0x7ff5380c96b0 2023-10-19T14:21:41.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.526+0000 7ff53d933640 1 --2- 172.21.15.57:0/466795974 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff5380b1920 0x7ff53814cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:21:41.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.526+0000 7ff53e134640 1 --2- 172.21.15.57:0/466795974 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff538154af0 0x7ff53814d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:21:41.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.526+0000 7ff53d132640 1 --2- 172.21.15.57:0/466795974 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff5380c96b0 0x7ff53815dcc0 secure :-1 s=READY pgs=2756 cs=0 l=1 rev1=1 crypto rx=0x7ff52c02c400 tx=0x7ff52c003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:41.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.526+0000 7ff5367fc640 1 -- 172.21.15.57:0/466795974 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff52c003e20 con 0x7ff5380c96b0 2023-10-19T14:21:41.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.526+0000 7ff5367fc640 1 -- 172.21.15.57:0/466795974 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7ff52c005110 con 0x7ff5380c96b0 2023-10-19T14:21:41.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.526+0000 7ff5367fc640 1 -- 172.21.15.57:0/466795974 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff52c005520 con 0x7ff5380c96b0 2023-10-19T14:21:41.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.526+0000 7ff53fb97640 1 -- 172.21.15.57:0/466795974 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff538070300 con 0x7ff5380c96b0 2023-10-19T14:21:41.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.526+0000 7ff53fb97640 1 -- 172.21.15.57:0/466795974 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff53815e310 con 0x7ff5380c96b0 2023-10-19T14:21:41.535 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.527+0000 7ff53fb97640 1 -- 172.21.15.57:0/466795974 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5380690e0 con 0x7ff5380c96b0 2023-10-19T14:21:41.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.528+0000 7ff5367fc640 1 -- 172.21.15.57:0/466795974 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 142) v1 ==== 88940+0+0 (secure 0 0 0) 0x7ff52c0047a0 con 0x7ff5380c96b0 2023-10-19T14:21:41.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.528+0000 7ff5367fc640 1 -- 172.21.15.57:0/466795974 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 7781+0+0 (secure 0 0 0) 0x7ff52c0ac280 con 0x7ff5380c96b0 2023-10-19T14:21:41.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.532+0000 7ff5367fc640 1 -- 172.21.15.57:0/466795974 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff52c076e70 con 0x7ff5380c96b0 2023-10-19T14:21:41.537 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:21:41.534+0000 7f0f9ee3e040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-19T14:21:41.643 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:21:41.640+0000 7f0f9ee3e040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-19T14:21:41.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.712+0000 7ff53fb97640 1 -- 172.21.15.57:0/466795974 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff5380af290 con 0x7ff5380c96b0 2023-10-19T14:21:41.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.712+0000 7ff5367fc640 1 -- 172.21.15.57:0/466795974 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v249) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff52c04b0a0 con 0x7ff5380c96b0 2023-10-19T14:21:41.716 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-19T14:21:41.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.715+0000 7ff53fb97640 1 -- 172.21.15.57:0/466795974 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff5380c96b0 msgr2=0x7ff53815dcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:41.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.715+0000 7ff53fb97640 1 --2- 172.21.15.57:0/466795974 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff5380c96b0 0x7ff53815dcc0 secure :-1 s=READY pgs=2756 cs=0 l=1 rev1=1 crypto rx=0x7ff52c02c400 tx=0x7ff52c003820 comp rx=0 tx=0).stop 2023-10-19T14:21:41.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.715+0000 7ff53fb97640 1 -- 172.21.15.57:0/466795974 shutdown_connections 2023-10-19T14:21:41.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.715+0000 7ff53fb97640 1 --2- 172.21.15.57:0/466795974 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff538154af0 0x7ff53814d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:41.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.715+0000 7ff53fb97640 1 --2- 172.21.15.57:0/466795974 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff5380c96b0 0x7ff53815dcc0 unknown :-1 s=CLOSED pgs=2756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:41.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.715+0000 7ff53fb97640 1 --2- 172.21.15.57:0/466795974 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff5380b1920 0x7ff53814cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:41.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.715+0000 7ff53fb97640 1 -- 172.21.15.57:0/466795974 >> 172.21.15.57:0/466795974 conn(0x7ff538069930 msgr2=0x7ff53815aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:41.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.715+0000 7ff53fb97640 1 -- 172.21.15.57:0/466795974 shutdown_connections 2023-10-19T14:21:41.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.715+0000 7ff53fb97640 1 -- 172.21.15.57:0/466795974 wait complete. 2023-10-19T14:21:41.728 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-19T14:21:41.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.837+0000 7fa814494640 1 -- 172.21.15.57:0/1533609911 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa80c0cf340 msgr2=0x7fa80c0d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:41.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.837+0000 7fa814494640 1 --2- 172.21.15.57:0/1533609911 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa80c0cf340 0x7fa80c0d1910 secure :-1 s=READY pgs=2757 cs=0 l=1 rev1=1 crypto rx=0x7fa80c08a120 tx=0x7fa80402bd10 comp rx=0 tx=0).stop 2023-10-19T14:21:41.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.837+0000 7fa814494640 1 -- 172.21.15.57:0/1533609911 shutdown_connections 2023-10-19T14:21:41.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.837+0000 7fa814494640 1 --2- 172.21.15.57:0/1533609911 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa80c0ceb40 0x7fa80c0b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:41.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.837+0000 7fa814494640 1 --2- 172.21.15.57:0/1533609911 >> v2:172.21.15.179:3300/0 conn(0x7fa80c15b2a0 0x7fa80c0cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:41.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.837+0000 7fa814494640 1 --2- 172.21.15.57:0/1533609911 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa80c0cf340 0x7fa80c0d1910 unknown :-1 s=CLOSED pgs=2757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:41.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.837+0000 7fa814494640 1 -- 172.21.15.57:0/1533609911 >> 172.21.15.57:0/1533609911 conn(0x7fa80c0d0d60 msgr2=0x7fa80c0d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:41.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.837+0000 7fa814494640 1 -- 172.21.15.57:0/1533609911 shutdown_connections 2023-10-19T14:21:41.842 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.837+0000 7fa814494640 1 -- 172.21.15.57:0/1533609911 wait complete. 2023-10-19T14:21:41.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.838+0000 7fa814494640 1 Processor -- start 2023-10-19T14:21:41.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.838+0000 7fa814494640 1 -- start start 2023-10-19T14:21:41.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.839+0000 7fa814494640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa80c0ceb40 0x7fa80c15d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:41.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.839+0000 7fa814494640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa80c0cf340 0x7fa80c0ca930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:41.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.839+0000 7fa814494640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa80c15b2a0 0x7fa80c0cae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:41.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.839+0000 7fa814494640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa80c1f24c0 con 0x7fa80c15b2a0 2023-10-19T14:21:41.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.839+0000 7fa814494640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa80c1f18b0 con 0x7fa80c0ceb40 2023-10-19T14:21:41.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.839+0000 7fa814494640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa80c1f1b90 con 0x7fa80c0cf340 2023-10-19T14:21:41.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.839+0000 7fa812a31640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa80c15b2a0 0x7fa80c0cae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:41.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.839+0000 7fa812a31640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa80c15b2a0 0x7fa80c0cae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44008/0 (socket says 172.21.15.57:44008) 2023-10-19T14:21:41.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.839+0000 7fa812a31640 1 -- 172.21.15.57:0/2886597542 learned_addr learned my addr 172.21.15.57:0/2886597542 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:41.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.839+0000 7fa812230640 1 --2- 172.21.15.57:0/2886597542 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa80c0ceb40 0x7fa80c15d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:41.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.839+0000 7fa811a2f640 1 --2- 172.21.15.57:0/2886597542 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa80c0cf340 0x7fa80c0ca930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:41.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.839+0000 7fa812a31640 1 -- 172.21.15.57:0/2886597542 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa80c0cf340 msgr2=0x7fa80c0ca930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:41.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.839+0000 7fa812a31640 1 --2- 172.21.15.57:0/2886597542 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa80c0cf340 0x7fa80c0ca930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:41.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.839+0000 7fa812a31640 1 -- 172.21.15.57:0/2886597542 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa80c0ceb40 msgr2=0x7fa80c15d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:41.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.839+0000 7fa812a31640 1 --2- 172.21.15.57:0/2886597542 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa80c0ceb40 0x7fa80c15d950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:41.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.839+0000 7fa812a31640 1 -- 172.21.15.57:0/2886597542 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa804009560 con 0x7fa80c15b2a0 2023-10-19T14:21:41.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.840+0000 7fa812a31640 1 --2- 172.21.15.57:0/2886597542 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa80c15b2a0 0x7fa80c0cae70 secure :-1 s=READY pgs=2758 cs=0 l=1 rev1=1 crypto rx=0x7fa7fc00ca40 tx=0x7fa7fc00cf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:41.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.840+0000 7fa80affd640 1 -- 172.21.15.57:0/2886597542 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7fc00adc0 con 0x7fa80c15b2a0 2023-10-19T14:21:41.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.840+0000 7fa80affd640 1 -- 172.21.15.57:0/2886597542 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fa7fc004c50 con 0x7fa80c15b2a0 2023-10-19T14:21:41.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.840+0000 7fa814494640 1 -- 172.21.15.57:0/2886597542 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa80c1f1e70 con 0x7fa80c15b2a0 2023-10-19T14:21:41.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.840+0000 7fa80affd640 1 -- 172.21.15.57:0/2886597542 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7fc005280 con 0x7fa80c15b2a0 2023-10-19T14:21:41.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.840+0000 7fa814494640 1 -- 172.21.15.57:0/2886597542 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa80c1f15b0 con 0x7fa80c15b2a0 2023-10-19T14:21:41.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.841+0000 7fa814494640 1 -- 172.21.15.57:0/2886597542 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa80c1f2670 con 0x7fa80c15b2a0 2023-10-19T14:21:41.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.841+0000 7fa80affd640 1 -- 172.21.15.57:0/2886597542 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 142) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fa7fc0053e0 con 0x7fa80c15b2a0 2023-10-19T14:21:41.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.842+0000 7fa80affd640 1 -- 172.21.15.57:0/2886597542 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 7781+0+0 (secure 0 0 0) 0x7fa7fc08a1c0 con 0x7fa80c15b2a0 2023-10-19T14:21:41.850 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:41.846+0000 7fa80affd640 1 -- 172.21.15.57:0/2886597542 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa80c1f2670 con 0x7fa80c15b2a0 2023-10-19T14:21:42.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.023+0000 7fa814494640 1 -- 172.21.15.57:0/2886597542 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa80c0690e0 con 0x7fa80c15b2a0 2023-10-19T14:21:42.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.023+0000 7fa80affd640 1 -- 172.21.15.57:0/2886597542 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v249) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa7fc01d020 con 0x7fa80c15b2a0 2023-10-19T14:21:42.027 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2023-10-19T14:21:42.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.026+0000 7fa814494640 1 -- 172.21.15.57:0/2886597542 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa80c15b2a0 msgr2=0x7fa80c0cae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:42.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.026+0000 7fa814494640 1 --2- 172.21.15.57:0/2886597542 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa80c15b2a0 0x7fa80c0cae70 secure :-1 s=READY pgs=2758 cs=0 l=1 rev1=1 crypto rx=0x7fa7fc00ca40 tx=0x7fa7fc00cf00 comp rx=0 tx=0).stop 2023-10-19T14:21:42.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.026+0000 7fa814494640 1 -- 172.21.15.57:0/2886597542 shutdown_connections 2023-10-19T14:21:42.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.026+0000 7fa814494640 1 --2- 172.21.15.57:0/2886597542 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa80c15b2a0 0x7fa80c0cae70 unknown :-1 s=CLOSED pgs=2758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:42.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.026+0000 7fa814494640 1 --2- 172.21.15.57:0/2886597542 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa80c0cf340 0x7fa80c0ca930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:42.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.026+0000 7fa814494640 1 --2- 172.21.15.57:0/2886597542 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa80c0ceb40 0x7fa80c15d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:42.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.026+0000 7fa814494640 1 -- 172.21.15.57:0/2886597542 >> 172.21.15.57:0/2886597542 conn(0x7fa80c0d0d60 msgr2=0x7fa80c0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:42.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.026+0000 7fa814494640 1 -- 172.21.15.57:0/2886597542 shutdown_connections 2023-10-19T14:21:42.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.026+0000 7fa814494640 1 -- 172.21.15.57:0/2886597542 wait complete. 2023-10-19T14:21:42.039 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-19T14:21:42.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.149+0000 7f28ac1a9640 1 -- 172.21.15.57:0/1889923061 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f288c0047c0 msgr2=0x7f288c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:42.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.149+0000 7f28ac1a9640 1 --2- 172.21.15.57:0/1889923061 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f288c0047c0 0x7f288c004c40 secure :-1 s=READY pgs=2760 cs=0 l=1 rev1=1 crypto rx=0x7f2898006f30 tx=0x7f288c0057f0 comp rx=0 tx=0).stop 2023-10-19T14:21:42.153 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.149+0000 7f28aa746640 1 -- 172.21.15.57:0/1889923061 reap_dead start 2023-10-19T14:21:42.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.149+0000 7f28ac1a9640 1 -- 172.21.15.57:0/1889923061 shutdown_connections 2023-10-19T14:21:42.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.149+0000 7f28ac1a9640 1 -- 172.21.15.57:0/1889923061 >> 172.21.15.57:0/1889923061 conn(0x7f28a4069930 msgr2=0x7f28a4069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:42.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.150+0000 7f28ac1a9640 1 -- 172.21.15.57:0/1889923061 shutdown_connections 2023-10-19T14:21:42.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.150+0000 7f28ac1a9640 1 -- 172.21.15.57:0/1889923061 wait complete. 2023-10-19T14:21:42.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.151+0000 7f28ac1a9640 1 Processor -- start 2023-10-19T14:21:42.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.151+0000 7f28ac1a9640 1 -- start start 2023-10-19T14:21:42.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.151+0000 7f28ac1a9640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f288c0047c0 0x7f28a413a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:42.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.151+0000 7f28ac1a9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f28a4070cb0 0x7f28a406f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:42.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.151+0000 7f28ac1a9640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f28a40aa8c0 0x7f28a40aad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:42.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.152+0000 7f28ac1a9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28a4071110 con 0x7f28a4070cb0 2023-10-19T14:21:42.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.152+0000 7f28ac1a9640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28a40ab280 con 0x7f28a40aa8c0 2023-10-19T14:21:42.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.152+0000 7f28ac1a9640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28a414af00 con 0x7f288c0047c0 2023-10-19T14:21:42.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.152+0000 7f28a9744640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f28a4070cb0 0x7f28a406f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:42.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.152+0000 7f28a9744640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f28a4070cb0 0x7f28a406f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44014/0 (socket says 172.21.15.57:44014) 2023-10-19T14:21:42.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.152+0000 7f28a9744640 1 -- 172.21.15.57:0/1878350729 learned_addr learned my addr 172.21.15.57:0/1878350729 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:42.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.152+0000 7f28a9f45640 1 --2- 172.21.15.57:0/1878350729 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f288c0047c0 0x7f28a413a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:42.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.152+0000 7f28aa746640 1 --2- 172.21.15.57:0/1878350729 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f28a40aa8c0 0x7f28a40aad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:42.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.152+0000 7f28a9744640 1 -- 172.21.15.57:0/1878350729 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f288c0047c0 msgr2=0x7f28a413a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:42.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.152+0000 7f28a9744640 1 --2- 172.21.15.57:0/1878350729 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f288c0047c0 0x7f28a413a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:42.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.152+0000 7f28a9744640 1 -- 172.21.15.57:0/1878350729 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f28a40aa8c0 msgr2=0x7f28a40aad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:42.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.152+0000 7f28a9744640 1 --2- 172.21.15.57:0/1878350729 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f28a40aa8c0 0x7f28a40aad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:42.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.152+0000 7f28a9744640 1 -- 172.21.15.57:0/1878350729 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28a414a960 con 0x7f28a4070cb0 2023-10-19T14:21:42.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.152+0000 7f28aa746640 1 --2- 172.21.15.57:0/1878350729 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f28a40aa8c0 0x7f28a40aad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:21:42.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.152+0000 7f28a9744640 1 --2- 172.21.15.57:0/1878350729 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f28a4070cb0 0x7f28a406f710 secure :-1 s=READY pgs=2761 cs=0 l=1 rev1=1 crypto rx=0x7f289802c820 tx=0x7f2898006510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:42.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.152+0000 7f28a27fc640 1 -- 172.21.15.57:0/1878350729 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2898003a20 con 0x7f28a4070cb0 2023-10-19T14:21:42.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.152+0000 7f28a27fc640 1 -- 172.21.15.57:0/1878350729 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f2898003c70 con 0x7f28a4070cb0 2023-10-19T14:21:42.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.152+0000 7f28ac1a9640 1 -- 172.21.15.57:0/1878350729 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28a406fc50 con 0x7f28a4070cb0 2023-10-19T14:21:42.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.152+0000 7f28a27fc640 1 -- 172.21.15.57:0/1878350729 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2898035470 con 0x7f28a4070cb0 2023-10-19T14:21:42.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.152+0000 7f28ac1a9640 1 -- 172.21.15.57:0/1878350729 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28a40701d0 con 0x7f28a4070cb0 2023-10-19T14:21:42.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.153+0000 7f28ac1a9640 1 -- 172.21.15.57:0/1878350729 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28a40690e0 con 0x7f28a4070cb0 2023-10-19T14:21:42.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.154+0000 7f28a27fc640 1 -- 172.21.15.57:0/1878350729 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 142) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f2898032070 con 0x7f28a4070cb0 2023-10-19T14:21:42.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.154+0000 7f28a27fc640 1 -- 172.21.15.57:0/1878350729 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 7781+0+0 (secure 0 0 0) 0x7f28980ab0e0 con 0x7f28a4070cb0 2023-10-19T14:21:42.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.158+0000 7f28a27fc640 1 -- 172.21.15.57:0/1878350729 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2898075e60 con 0x7f28a4070cb0 2023-10-19T14:21:42.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.332+0000 7f28ac1a9640 1 -- 172.21.15.57:0/1878350729 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f28a40ca0e0 con 0x7f28a4070cb0 2023-10-19T14:21:42.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.333+0000 7f28a27fc640 1 -- 172.21.15.57:0/1878350729 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v249) v1 ==== 100+0+11 (secure 0 0 0) 0x7f28a40ca0e0 con 0x7f28a4070cb0 2023-10-19T14:21:42.336 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 64 2023-10-19T14:21:42.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.335+0000 7f28ac1a9640 1 -- 172.21.15.57:0/1878350729 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f28a4070cb0 msgr2=0x7f28a406f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:42.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.335+0000 7f28ac1a9640 1 --2- 172.21.15.57:0/1878350729 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f28a4070cb0 0x7f28a406f710 secure :-1 s=READY pgs=2761 cs=0 l=1 rev1=1 crypto rx=0x7f289802c820 tx=0x7f2898006510 comp rx=0 tx=0).stop 2023-10-19T14:21:42.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.335+0000 7f28ac1a9640 1 -- 172.21.15.57:0/1878350729 shutdown_connections 2023-10-19T14:21:42.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.335+0000 7f28ac1a9640 1 --2- 172.21.15.57:0/1878350729 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f28a40aa8c0 0x7f28a40aad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:42.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.335+0000 7f28ac1a9640 1 --2- 172.21.15.57:0/1878350729 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f28a4070cb0 0x7f28a406f710 unknown :-1 s=CLOSED pgs=2761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:42.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.335+0000 7f28ac1a9640 1 --2- 172.21.15.57:0/1878350729 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f288c0047c0 0x7f28a413a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:42.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.335+0000 7f28ac1a9640 1 -- 172.21.15.57:0/1878350729 >> 172.21.15.57:0/1878350729 conn(0x7f28a4069930 msgr2=0x7f28a40ca870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:42.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.335+0000 7f28ac1a9640 1 -- 172.21.15.57:0/1878350729 shutdown_connections 2023-10-19T14:21:42.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.335+0000 7f28ac1a9640 1 -- 172.21.15.57:0/1878350729 wait complete. 2023-10-19T14:21:42.348 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2023-10-19T14:21:42.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.464+0000 7fde6e7d7640 1 -- 172.21.15.57:0/2796685846 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde6806a730 msgr2=0x7fde6806abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:42.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.464+0000 7fde6e7d7640 1 --2- 172.21.15.57:0/2796685846 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde6806a730 0x7fde6806abc0 secure :-1 s=READY pgs=2762 cs=0 l=1 rev1=1 crypto rx=0x7fde68072e80 tx=0x7fde6002c250 comp rx=0 tx=0).stop 2023-10-19T14:21:42.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.464+0000 7fde6e7d7640 1 -- 172.21.15.57:0/2796685846 shutdown_connections 2023-10-19T14:21:42.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.464+0000 7fde6e7d7640 1 --2- 172.21.15.57:0/2796685846 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde6806a730 0x7fde6806abc0 unknown :-1 s=CLOSED pgs=2762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:42.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.464+0000 7fde6e7d7640 1 --2- 172.21.15.57:0/2796685846 >> v2:172.21.15.179:3300/0 conn(0x7fde68156790 0x7fde680cc990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:42.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.464+0000 7fde6e7d7640 1 -- 172.21.15.57:0/2796685846 >> 172.21.15.57:0/2796685846 conn(0x7fde6806fae0 msgr2=0x7fde6806fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:42.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.464+0000 7fde6e7d7640 1 -- 172.21.15.57:0/2796685846 shutdown_connections 2023-10-19T14:21:42.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.464+0000 7fde6e7d7640 1 -- 172.21.15.57:0/2796685846 wait complete. 2023-10-19T14:21:42.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.465+0000 7fde6e7d7640 1 Processor -- start 2023-10-19T14:21:42.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.465+0000 7fde6e7d7640 1 -- start start 2023-10-19T14:21:42.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.466+0000 7fde6e7d7640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fde6806a730 0x7fde68141ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:42.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.466+0000 7fde6e7d7640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fde680ac870 0x7fde680a8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:42.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.466+0000 7fde6e7d7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde68156790 0x7fde680a9150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:42.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.466+0000 7fde6e7d7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde68145510 con 0x7fde68156790 2023-10-19T14:21:42.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.466+0000 7fde6e7d7640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde68142220 con 0x7fde6806a730 2023-10-19T14:21:42.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.466+0000 7fde6e7d7640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde68144990 con 0x7fde680ac870 2023-10-19T14:21:42.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.466+0000 7fde6cd74640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde68156790 0x7fde680a9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:42.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.466+0000 7fde6cd74640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde68156790 0x7fde680a9150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44026/0 (socket says 172.21.15.57:44026) 2023-10-19T14:21:42.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.466+0000 7fde6cd74640 1 -- 172.21.15.57:0/3721632237 learned_addr learned my addr 172.21.15.57:0/3721632237 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:42.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.467+0000 7fde67fff640 1 --2- 172.21.15.57:0/3721632237 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fde6806a730 0x7fde68141ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:42.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.467+0000 7fde677fe640 1 --2- 172.21.15.57:0/3721632237 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fde680ac870 0x7fde680a8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:42.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.467+0000 7fde6cd74640 1 -- 172.21.15.57:0/3721632237 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fde680ac870 msgr2=0x7fde680a8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:42.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.467+0000 7fde6cd74640 1 --2- 172.21.15.57:0/3721632237 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fde680ac870 0x7fde680a8c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:42.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.467+0000 7fde6cd74640 1 -- 172.21.15.57:0/3721632237 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fde6806a730 msgr2=0x7fde68141ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:42.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.467+0000 7fde6cd74640 1 --2- 172.21.15.57:0/3721632237 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fde6806a730 0x7fde68141ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:42.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.467+0000 7fde6cd74640 1 -- 172.21.15.57:0/3721632237 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde60009900 con 0x7fde68156790 2023-10-19T14:21:42.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.467+0000 7fde6cd74640 1 --2- 172.21.15.57:0/3721632237 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde68156790 0x7fde680a9150 secure :-1 s=READY pgs=2763 cs=0 l=1 rev1=1 crypto rx=0x7fde60037040 tx=0x7fde60035db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:42.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.467+0000 7fde64ff9640 1 -- 172.21.15.57:0/3721632237 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde60034440 con 0x7fde68156790 2023-10-19T14:21:42.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.467+0000 7fde64ff9640 1 -- 172.21.15.57:0/3721632237 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fde60004250 con 0x7fde68156790 2023-10-19T14:21:42.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.467+0000 7fde6e7d7640 1 -- 172.21.15.57:0/3721632237 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde68144c70 con 0x7fde68156790 2023-10-19T14:21:42.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.467+0000 7fde64ff9640 1 -- 172.21.15.57:0/3721632237 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde60004600 con 0x7fde68156790 2023-10-19T14:21:42.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.467+0000 7fde6e7d7640 1 -- 172.21.15.57:0/3721632237 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde68145230 con 0x7fde68156790 2023-10-19T14:21:42.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.468+0000 7fde6e7d7640 1 -- 172.21.15.57:0/3721632237 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde68142590 con 0x7fde68156790 2023-10-19T14:21:42.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.469+0000 7fde64ff9640 1 -- 172.21.15.57:0/3721632237 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 143) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fde6003b040 con 0x7fde68156790 2023-10-19T14:21:42.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.469+0000 7fde64ff9640 1 --2- 172.21.15.57:0/3721632237 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fde5406ac90 0x7fde5406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:42.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.469+0000 7fde64ff9640 1 -- 172.21.15.57:0/3721632237 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 7781+0+0 (secure 0 0 0) 0x7fde600ac830 con 0x7fde68156790 2023-10-19T14:21:42.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.469+0000 7fde67fff640 1 --2- 172.21.15.57:0/3721632237 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fde5406ac90 0x7fde5406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:42.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.470+0000 7fde67fff640 1 --2- 172.21.15.57:0/3721632237 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fde5406ac90 0x7fde5406d150 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fde5c0058c0 tx=0x7fde5c00a4c0 comp rx=0 tx=0).ready entity=mgr.11446 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:42.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.473+0000 7fde64ff9640 1 -- 172.21.15.57:0/3721632237 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fde68142590 con 0x7fde68156790 2023-10-19T14:21:42.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.650+0000 7fde6e7d7640 1 -- 172.21.15.57:0/3721632237 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fde68142590 con 0x7fde68156790 2023-10-19T14:21:42.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.651+0000 7fde64ff9640 1 -- 172.21.15.57:0/3721632237 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v249) v1 ==== 107+0+11 (secure 0 0 0) 0x7fde60039020 con 0x7fde68156790 2023-10-19T14:21:42.654 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2023-10-19T14:21:42.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.653+0000 7fde6e7d7640 1 -- 172.21.15.57:0/3721632237 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fde5406ac90 msgr2=0x7fde5406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:42.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.653+0000 7fde6e7d7640 1 --2- 172.21.15.57:0/3721632237 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fde5406ac90 0x7fde5406d150 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fde5c0058c0 tx=0x7fde5c00a4c0 comp rx=0 tx=0).stop 2023-10-19T14:21:42.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.653+0000 7fde6e7d7640 1 -- 172.21.15.57:0/3721632237 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde68156790 msgr2=0x7fde680a9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:42.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.653+0000 7fde6e7d7640 1 --2- 172.21.15.57:0/3721632237 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde68156790 0x7fde680a9150 secure :-1 s=READY pgs=2763 cs=0 l=1 rev1=1 crypto rx=0x7fde60037040 tx=0x7fde60035db0 comp rx=0 tx=0).stop 2023-10-19T14:21:42.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.653+0000 7fde6e7d7640 1 -- 172.21.15.57:0/3721632237 shutdown_connections 2023-10-19T14:21:42.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.653+0000 7fde6e7d7640 1 --2- 172.21.15.57:0/3721632237 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fde5406ac90 0x7fde5406d150 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:42.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.653+0000 7fde6e7d7640 1 --2- 172.21.15.57:0/3721632237 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde68156790 0x7fde680a9150 unknown :-1 s=CLOSED pgs=2763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:42.657 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.653+0000 7fde6e7d7640 1 --2- 172.21.15.57:0/3721632237 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fde680ac870 0x7fde680a8c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:42.658 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.653+0000 7fde6e7d7640 1 --2- 172.21.15.57:0/3721632237 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fde6806a730 0x7fde68141ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:42.658 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.653+0000 7fde6e7d7640 1 -- 172.21.15.57:0/3721632237 >> 172.21.15.57:0/3721632237 conn(0x7fde6806fae0 msgr2=0x7fde68155e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:42.658 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.653+0000 7fde6e7d7640 1 -- 172.21.15.57:0/3721632237 shutdown_connections 2023-10-19T14:21:42.658 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.653+0000 7fde6e7d7640 1 -- 172.21.15.57:0/3721632237 wait complete. 2023-10-19T14:21:42.667 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2023-10-19T14:21:42.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.775+0000 7f3d50e87640 1 -- 172.21.15.57:0/1089633315 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3d4c0ceb40 msgr2=0x7f3d4c0b0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:42.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.775+0000 7f3d50e87640 1 --2- 172.21.15.57:0/1089633315 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3d4c0ceb40 0x7f3d4c0b0650 secure :-1 s=READY pgs=2765 cs=0 l=1 rev1=1 crypto rx=0x7f3d4c08a120 tx=0x7f3d4402bcf0 comp rx=0 tx=0).stop 2023-10-19T14:21:42.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.776+0000 7f3d50e87640 1 -- 172.21.15.57:0/1089633315 shutdown_connections 2023-10-19T14:21:42.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.776+0000 7f3d50e87640 1 --2- 172.21.15.57:0/1089633315 >> v2:172.21.15.179:3300/0 conn(0x7f3d4c15b2a0 0x7f3d4c0d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:42.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.776+0000 7f3d50e87640 1 --2- 172.21.15.57:0/1089633315 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3d4c0ceb40 0x7f3d4c0b0650 unknown :-1 s=CLOSED pgs=2765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:42.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.776+0000 7f3d50e87640 1 -- 172.21.15.57:0/1089633315 >> 172.21.15.57:0/1089633315 conn(0x7f3d4c0d0d60 msgr2=0x7f3d4c0d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:42.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.776+0000 7f3d50e87640 1 -- 172.21.15.57:0/1089633315 shutdown_connections 2023-10-19T14:21:42.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.776+0000 7f3d50e87640 1 -- 172.21.15.57:0/1089633315 wait complete. 2023-10-19T14:21:42.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.777+0000 7f3d50e87640 1 Processor -- start 2023-10-19T14:21:42.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.777+0000 7f3d50e87640 1 -- start start 2023-10-19T14:21:42.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.778+0000 7f3d50e87640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3d4c0ceb40 0x7f3d4c0b5510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:42.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.778+0000 7f3d50e87640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3d4c0cf340 0x7f3d4c15de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:42.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.778+0000 7f3d50e87640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3d4c15b2a0 0x7f3d4c158590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:42.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.778+0000 7f3d50e87640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d4c0b4470 con 0x7f3d4c0ceb40 2023-10-19T14:21:42.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.778+0000 7f3d50e87640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d4c0b4750 con 0x7f3d4c0cf340 2023-10-19T14:21:42.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.778+0000 7f3d50e87640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d4c0b4a30 con 0x7f3d4c15b2a0 2023-10-19T14:21:42.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.778+0000 7f3d4a59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3d4c0ceb40 0x7f3d4c0b5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:42.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.778+0000 7f3d4a59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3d4c0ceb40 0x7f3d4c0b5510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44052/0 (socket says 172.21.15.57:44052) 2023-10-19T14:21:42.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.778+0000 7f3d4a59c640 1 -- 172.21.15.57:0/52836779 learned_addr learned my addr 172.21.15.57:0/52836779 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:42.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.778+0000 7f3d4a59c640 1 -- 172.21.15.57:0/52836779 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3d4c15b2a0 msgr2=0x7f3d4c158590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:42.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.778+0000 7f3d4a59c640 1 --2- 172.21.15.57:0/52836779 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3d4c15b2a0 0x7f3d4c158590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:42.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.778+0000 7f3d4a59c640 1 -- 172.21.15.57:0/52836779 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3d4c0cf340 msgr2=0x7f3d4c15de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:42.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.778+0000 7f3d4a59c640 1 --2- 172.21.15.57:0/52836779 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3d4c0cf340 0x7f3d4c15de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:42.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.778+0000 7f3d4a59c640 1 -- 172.21.15.57:0/52836779 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d44009490 con 0x7f3d4c0ceb40 2023-10-19T14:21:42.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.778+0000 7f3d4a59c640 1 --2- 172.21.15.57:0/52836779 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3d4c0ceb40 0x7f3d4c0b5510 secure :-1 s=READY pgs=2766 cs=0 l=1 rev1=1 crypto rx=0x7f3d3c00c890 tx=0x7f3d3c00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:42.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.778+0000 7f3d42ffd640 1 -- 172.21.15.57:0/52836779 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d3c003da0 con 0x7f3d4c0ceb40 2023-10-19T14:21:42.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.778+0000 7f3d42ffd640 1 -- 172.21.15.57:0/52836779 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f3d3c005070 con 0x7f3d4c0ceb40 2023-10-19T14:21:42.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.778+0000 7f3d42ffd640 1 -- 172.21.15.57:0/52836779 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d3c005460 con 0x7f3d4c0ceb40 2023-10-19T14:21:42.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.778+0000 7f3d50e87640 1 -- 172.21.15.57:0/52836779 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d4c0b4d10 con 0x7f3d4c0ceb40 2023-10-19T14:21:42.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.778+0000 7f3d50e87640 1 -- 172.21.15.57:0/52836779 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d4c15e390 con 0x7f3d4c0ceb40 2023-10-19T14:21:42.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.779+0000 7f3d50e87640 1 -- 172.21.15.57:0/52836779 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d4c0b5230 con 0x7f3d4c0ceb40 2023-10-19T14:21:42.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.780+0000 7f3d42ffd640 1 -- 172.21.15.57:0/52836779 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 143) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f3d3c0055c0 con 0x7f3d4c0ceb40 2023-10-19T14:21:42.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.780+0000 7f3d42ffd640 1 --2- 172.21.15.57:0/52836779 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f3d2c06ac90 0x7f3d2c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:42.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.780+0000 7f3d42ffd640 1 -- 172.21.15.57:0/52836779 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 7781+0+0 (secure 0 0 0) 0x7f3d3c0896f0 con 0x7f3d4c0ceb40 2023-10-19T14:21:42.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.781+0000 7f3d49d9b640 1 --2- 172.21.15.57:0/52836779 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f3d2c06ac90 0x7f3d2c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:42.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.781+0000 7f3d49d9b640 1 --2- 172.21.15.57:0/52836779 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f3d2c06ac90 0x7f3d2c06d150 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f3d340058c0 tx=0x7f3d34005850 comp rx=0 tx=0).ready entity=mgr.11446 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:42.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.784+0000 7f3d42ffd640 1 -- 172.21.15.57:0/52836779 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3d3c0541f0 con 0x7f3d4c0ceb40 2023-10-19T14:21:42.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.958+0000 7f3d50e87640 1 -- 172.21.15.57:0/52836779 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) v1 -- 0x7f3d4c0b0d00 con 0x7f3d4c0ceb40 2023-10-19T14:21:42.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.959+0000 7f3d42ffd640 1 -- 172.21.15.57:0/52836779 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v249) v1 ==== 103+0+11 (secure 0 0 0) 0x7f3d3c017070 con 0x7f3d4c0ceb40 2023-10-19T14:21:42.962 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 64 2023-10-19T14:21:42.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.961+0000 7f3d50e87640 1 -- 172.21.15.57:0/52836779 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f3d2c06ac90 msgr2=0x7f3d2c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:42.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.961+0000 7f3d50e87640 1 --2- 172.21.15.57:0/52836779 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f3d2c06ac90 0x7f3d2c06d150 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f3d340058c0 tx=0x7f3d34005850 comp rx=0 tx=0).stop 2023-10-19T14:21:42.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.961+0000 7f3d50e87640 1 -- 172.21.15.57:0/52836779 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3d4c0ceb40 msgr2=0x7f3d4c0b5510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:42.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.961+0000 7f3d50e87640 1 --2- 172.21.15.57:0/52836779 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3d4c0ceb40 0x7f3d4c0b5510 secure :-1 s=READY pgs=2766 cs=0 l=1 rev1=1 crypto rx=0x7f3d3c00c890 tx=0x7f3d3c00cd50 comp rx=0 tx=0).stop 2023-10-19T14:21:42.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.961+0000 7f3d50e87640 1 -- 172.21.15.57:0/52836779 shutdown_connections 2023-10-19T14:21:42.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.961+0000 7f3d50e87640 1 --2- 172.21.15.57:0/52836779 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f3d2c06ac90 0x7f3d2c06d150 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:42.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.961+0000 7f3d50e87640 1 --2- 172.21.15.57:0/52836779 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3d4c15b2a0 0x7f3d4c158590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:42.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.961+0000 7f3d50e87640 1 --2- 172.21.15.57:0/52836779 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3d4c0cf340 0x7f3d4c15de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:42.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.961+0000 7f3d50e87640 1 --2- 172.21.15.57:0/52836779 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3d4c0ceb40 0x7f3d4c0b5510 unknown :-1 s=CLOSED pgs=2766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:42.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.961+0000 7f3d50e87640 1 -- 172.21.15.57:0/52836779 >> 172.21.15.57:0/52836779 conn(0x7f3d4c0d0d60 msgr2=0x7f3d4c0b0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:42.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.961+0000 7f3d50e87640 1 -- 172.21.15.57:0/52836779 shutdown_connections 2023-10-19T14:21:42.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:42.961+0000 7f3d50e87640 1 -- 172.21.15.57:0/52836779 wait complete. 2023-10-19T14:21:42.975 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T14:21:43.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.082+0000 7f12cda61640 1 -- 172.21.15.57:0/2873856734 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f12b4004d00 msgr2=0x7f12b4005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:43.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.082+0000 7f12cda61640 1 --2- 172.21.15.57:0/2873856734 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f12b4004d00 0x7f12b4005180 secure :-1 s=READY pgs=2768 cs=0 l=1 rev1=1 crypto rx=0x7f12c0001d80 tx=0x7f12b40057f0 comp rx=0 tx=0).stop 2023-10-19T14:21:43.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.082+0000 7f12c77fe640 1 -- 172.21.15.57:0/2873856734 reap_dead start 2023-10-19T14:21:43.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.082+0000 7f12cda61640 1 -- 172.21.15.57:0/2873856734 shutdown_connections 2023-10-19T14:21:43.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.082+0000 7f12cda61640 1 -- 172.21.15.57:0/2873856734 >> 172.21.15.57:0/2873856734 conn(0x7f12c8069930 msgr2=0x7f12c8069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:43.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.082+0000 7f12cda61640 1 -- 172.21.15.57:0/2873856734 shutdown_connections 2023-10-19T14:21:43.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.082+0000 7f12cda61640 1 -- 172.21.15.57:0/2873856734 wait complete. 2023-10-19T14:21:43.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.083+0000 7f12cda61640 1 Processor -- start 2023-10-19T14:21:43.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.084+0000 7f12cda61640 1 -- start start 2023-10-19T14:21:43.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.084+0000 7f12cda61640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f12c8138380 0x7f12c80b0d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:43.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.084+0000 7f12cda61640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f12c8070180 0x7f12c80aa8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:43.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.084+0000 7f12cda61640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f12c80b12b0 0x7f12c80aae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:43.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.084+0000 7f12cda61640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12c81512d0 con 0x7f12c80b12b0 2023-10-19T14:21:43.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.084+0000 7f12cda61640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12c8151e50 con 0x7f12c8070180 2023-10-19T14:21:43.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.084+0000 7f12cda61640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12c8151520 con 0x7f12c8138380 2023-10-19T14:21:43.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.084+0000 7f12c77fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f12c80b12b0 0x7f12c80aae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:43.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.085+0000 7f12c77fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f12c80b12b0 0x7f12c80aae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44064/0 (socket says 172.21.15.57:44064) 2023-10-19T14:21:43.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.085+0000 7f12c77fe640 1 -- 172.21.15.57:0/629722818 learned_addr learned my addr 172.21.15.57:0/629722818 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:43.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.085+0000 7f12be7fc640 1 --2- 172.21.15.57:0/629722818 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f12c8070180 0x7f12c80aa8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:43.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.085+0000 7f12c6ffd640 1 --2- 172.21.15.57:0/629722818 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f12c8138380 0x7f12c80b0d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:43.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.085+0000 7f12c77fe640 1 -- 172.21.15.57:0/629722818 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f12c8138380 msgr2=0x7f12c80b0d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:43.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.085+0000 7f12c77fe640 1 --2- 172.21.15.57:0/629722818 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f12c8138380 0x7f12c80b0d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:43.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.085+0000 7f12c77fe640 1 -- 172.21.15.57:0/629722818 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f12c8070180 msgr2=0x7f12c80aa8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:43.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.085+0000 7f12c77fe640 1 --2- 172.21.15.57:0/629722818 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f12c8070180 0x7f12c80aa8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:43.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.085+0000 7f12c77fe640 1 -- 172.21.15.57:0/629722818 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12c8151800 con 0x7f12c80b12b0 2023-10-19T14:21:43.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.085+0000 7f12c77fe640 1 --2- 172.21.15.57:0/629722818 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f12c80b12b0 0x7f12c80aae00 secure :-1 s=READY pgs=2769 cs=0 l=1 rev1=1 crypto rx=0x7f12c0001ea0 tx=0x7f12c0003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:43.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.085+0000 7f12c4ff9640 1 -- 172.21.15.57:0/629722818 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12c002ccd0 con 0x7f12c80b12b0 2023-10-19T14:21:43.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.085+0000 7f12c4ff9640 1 -- 172.21.15.57:0/629722818 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f12c0003e40 con 0x7f12c80b12b0 2023-10-19T14:21:43.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.085+0000 7f12cda61640 1 -- 172.21.15.57:0/629722818 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12c8151ae0 con 0x7f12c80b12b0 2023-10-19T14:21:43.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.085+0000 7f12c4ff9640 1 -- 172.21.15.57:0/629722818 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12c0006310 con 0x7f12c80b12b0 2023-10-19T14:21:43.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.085+0000 7f12cda61640 1 -- 172.21.15.57:0/629722818 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12c8138940 con 0x7f12c80b12b0 2023-10-19T14:21:43.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.086+0000 7f12cda61640 1 -- 172.21.15.57:0/629722818 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12c8152000 con 0x7f12c80b12b0 2023-10-19T14:21:43.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.087+0000 7f12c4ff9640 1 -- 172.21.15.57:0/629722818 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 143) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f12c0006470 con 0x7f12c80b12b0 2023-10-19T14:21:43.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.087+0000 7f12c4ff9640 1 --2- 172.21.15.57:0/629722818 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f12a806ac90 0x7f12a806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:43.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.087+0000 7f12c4ff9640 1 -- 172.21.15.57:0/629722818 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 7781+0+0 (secure 0 0 0) 0x7f12c00aa900 con 0x7f12c80b12b0 2023-10-19T14:21:43.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.087+0000 7f12c6ffd640 1 --2- 172.21.15.57:0/629722818 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f12a806ac90 0x7f12a806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:43.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.088+0000 7f12c6ffd640 1 --2- 172.21.15.57:0/629722818 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f12a806ac90 0x7f12a806d150 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f12c806a0e0 tx=0x7f12b8005f90 comp rx=0 tx=0).ready entity=mgr.11446 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:43.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.091+0000 7f12c4ff9640 1 -- 172.21.15.57:0/629722818 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f12c8152000 con 0x7f12c80b12b0 2023-10-19T14:21:43.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.303+0000 7f12cda61640 1 -- 172.21.15.57:0/629722818 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f12c8151fc0 con 0x7f12c80b12b0 2023-10-19T14:21:43.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.304+0000 7f12c4ff9640 1 -- 172.21.15.57:0/629722818 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+6140 (secure 0 0 0) 0x7f12c0044030 con 0x7f12c80b12b0 2023-10-19T14:21:43.307 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:21:43.307 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":107,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9761,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/325888554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":325888554},{"type":"v1","addr":"172.21.15.179:6836","nonce":325888554}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":100},{"gid":9764,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/8038232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":8038232},{"type":"v1","addr":"172.21.15.179:6837","nonce":8038232}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":100}],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T14:21:09.759935+0000","modified":"2023-10-19T14:21:11.768676+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":12564},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12564":{"gid":12564,"name":"c","rank":0,"incarnation":101,"state":"up:active","state_seq":3,"addr":"172.21.15.57:6835/18187246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":18187246},{"type":"v1","addr":"172.21.15.57:6835","nonce":18187246}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T14:21:26.724728+0000","modified":"2023-10-19T14:21:28.872856+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":12555},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12555":{"gid":12555,"name":"a","rank":0,"incarnation":105,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6833/4256498777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":4256498777},{"type":"v1","addr":"172.21.15.57:6833","nonce":4256498777}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2023-10-19T14:21:43.308 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 107 2023-10-19T14:21:43.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.306+0000 7f12cda61640 1 -- 172.21.15.57:0/629722818 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f12a806ac90 msgr2=0x7f12a806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:43.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.306+0000 7f12cda61640 1 --2- 172.21.15.57:0/629722818 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f12a806ac90 0x7f12a806d150 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f12c806a0e0 tx=0x7f12b8005f90 comp rx=0 tx=0).stop 2023-10-19T14:21:43.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.306+0000 7f12cda61640 1 -- 172.21.15.57:0/629722818 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f12c80b12b0 msgr2=0x7f12c80aae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:43.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.306+0000 7f12cda61640 1 --2- 172.21.15.57:0/629722818 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f12c80b12b0 0x7f12c80aae00 secure :-1 s=READY pgs=2769 cs=0 l=1 rev1=1 crypto rx=0x7f12c0001ea0 tx=0x7f12c0003ba0 comp rx=0 tx=0).stop 2023-10-19T14:21:43.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.306+0000 7f12cda61640 1 -- 172.21.15.57:0/629722818 shutdown_connections 2023-10-19T14:21:43.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.306+0000 7f12cda61640 1 --2- 172.21.15.57:0/629722818 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f12a806ac90 0x7f12a806d150 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:43.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.306+0000 7f12cda61640 1 --2- 172.21.15.57:0/629722818 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f12c80b12b0 0x7f12c80aae00 unknown :-1 s=CLOSED pgs=2769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:43.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.306+0000 7f12cda61640 1 --2- 172.21.15.57:0/629722818 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f12c8070180 0x7f12c80aa8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:43.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.306+0000 7f12cda61640 1 --2- 172.21.15.57:0/629722818 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f12c8138380 0x7f12c80b0d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:43.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.306+0000 7f12cda61640 1 -- 172.21.15.57:0/629722818 >> 172.21.15.57:0/629722818 conn(0x7f12c8069930 msgr2=0x7f12c80b18b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:43.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.306+0000 7f12cda61640 1 -- 172.21.15.57:0/629722818 shutdown_connections 2023-10-19T14:21:43.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.306+0000 7f12cda61640 1 -- 172.21.15.57:0/629722818 wait complete. 2023-10-19T14:21:43.319 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T14:21:43.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.426+0000 7f5be734c640 1 -- 172.21.15.57:0/2073814403 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5be00ac870 msgr2=0x7f5be006b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:43.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.426+0000 7f5be734c640 1 --2- 172.21.15.57:0/2073814403 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5be00ac870 0x7f5be006b980 secure :-1 s=READY pgs=2771 cs=0 l=1 rev1=1 crypto rx=0x7f5be00ac610 tx=0x7f5bd002bd60 comp rx=0 tx=0).stop 2023-10-19T14:21:43.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.426+0000 7f5be734c640 1 -- 172.21.15.57:0/2073814403 shutdown_connections 2023-10-19T14:21:43.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.426+0000 7f5be734c640 1 --2- 172.21.15.57:0/2073814403 >> v2:172.21.15.179:3300/0 conn(0x7f5be006bec0 0x7f5be006a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:43.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.426+0000 7f5be734c640 1 --2- 172.21.15.57:0/2073814403 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5be00ac870 0x7f5be006b980 unknown :-1 s=CLOSED pgs=2771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:43.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.426+0000 7f5be734c640 1 -- 172.21.15.57:0/2073814403 >> 172.21.15.57:0/2073814403 conn(0x7f5be006fae0 msgr2=0x7f5be006fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:43.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.426+0000 7f5be734c640 1 -- 172.21.15.57:0/2073814403 shutdown_connections 2023-10-19T14:21:43.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.426+0000 7f5be734c640 1 -- 172.21.15.57:0/2073814403 wait complete. 2023-10-19T14:21:43.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.427+0000 7f5be734c640 1 Processor -- start 2023-10-19T14:21:43.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.427+0000 7f5be734c640 1 -- start start 2023-10-19T14:21:43.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.428+0000 7f5be734c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5be006bec0 0x7f5be014d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:43.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.428+0000 7f5be734c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5be00ac870 0x7f5be014dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:43.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.428+0000 7f5be734c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5be0156790 0x7f5be0152200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:43.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.428+0000 7f5be734c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5be0150240 con 0x7f5be0156790 2023-10-19T14:21:43.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.428+0000 7f5be734c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5be0150a10 con 0x7f5be00ac870 2023-10-19T14:21:43.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.428+0000 7f5be734c640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5be0151c50 con 0x7f5be006bec0 2023-10-19T14:21:43.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.428+0000 7f5be58e9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5be0156790 0x7f5be0152200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:43.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.428+0000 7f5be58e9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5be0156790 0x7f5be0152200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44084/0 (socket says 172.21.15.57:44084) 2023-10-19T14:21:43.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.428+0000 7f5be58e9640 1 -- 172.21.15.57:0/705762546 learned_addr learned my addr 172.21.15.57:0/705762546 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:43.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.428+0000 7f5be48e7640 1 --2- 172.21.15.57:0/705762546 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5be00ac870 0x7f5be014dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:43.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.428+0000 7f5be50e8640 1 --2- 172.21.15.57:0/705762546 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5be006bec0 0x7f5be014d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:43.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.428+0000 7f5be58e9640 1 -- 172.21.15.57:0/705762546 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5be006bec0 msgr2=0x7f5be014d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:43.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.428+0000 7f5be58e9640 1 --2- 172.21.15.57:0/705762546 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5be006bec0 0x7f5be014d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:43.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.428+0000 7f5be58e9640 1 -- 172.21.15.57:0/705762546 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5be00ac870 msgr2=0x7f5be014dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:43.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.428+0000 7f5be58e9640 1 --2- 172.21.15.57:0/705762546 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5be00ac870 0x7f5be014dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:43.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.428+0000 7f5be58e9640 1 -- 172.21.15.57:0/705762546 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bd0009490 con 0x7f5be0156790 2023-10-19T14:21:43.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.428+0000 7f5be48e7640 1 --2- 172.21.15.57:0/705762546 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5be00ac870 0x7f5be014dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:21:43.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.428+0000 7f5be50e8640 1 --2- 172.21.15.57:0/705762546 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5be006bec0 0x7f5be014d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:21:43.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.428+0000 7f5be58e9640 1 --2- 172.21.15.57:0/705762546 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5be0156790 0x7f5be0152200 secure :-1 s=READY pgs=2772 cs=0 l=1 rev1=1 crypto rx=0x7f5bd800e990 tx=0x7f5bd800ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:43.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.428+0000 7f5bd5ffb640 1 -- 172.21.15.57:0/705762546 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bd8004d20 con 0x7f5be0156790 2023-10-19T14:21:43.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.428+0000 7f5bd5ffb640 1 -- 172.21.15.57:0/705762546 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f5bd800caa0 con 0x7f5be0156790 2023-10-19T14:21:43.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.428+0000 7f5bd5ffb640 1 -- 172.21.15.57:0/705762546 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bd8005020 con 0x7f5be0156790 2023-10-19T14:21:43.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.428+0000 7f5be734c640 1 -- 172.21.15.57:0/705762546 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5be0150450 con 0x7f5be0156790 2023-10-19T14:21:43.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.429+0000 7f5be734c640 1 -- 172.21.15.57:0/705762546 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5be0151870 con 0x7f5be0156790 2023-10-19T14:21:43.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.430+0000 7f5be734c640 1 -- 172.21.15.57:0/705762546 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5be01e3a20 con 0x7f5be0156790 2023-10-19T14:21:43.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.430+0000 7f5bd5ffb640 1 -- 172.21.15.57:0/705762546 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 143) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f5bd8005180 con 0x7f5be0156790 2023-10-19T14:21:43.439 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.430+0000 7f5bd5ffb640 1 --2- 172.21.15.57:0/705762546 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f5bc406ac90 0x7f5bc406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:43.439 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.431+0000 7f5bd5ffb640 1 -- 172.21.15.57:0/705762546 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 7781+0+0 (secure 0 0 0) 0x7f5bd808a340 con 0x7f5be0156790 2023-10-19T14:21:43.439 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.431+0000 7f5be50e8640 1 --2- 172.21.15.57:0/705762546 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f5bc406ac90 0x7f5bc406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:43.439 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.431+0000 7f5be50e8640 1 --2- 172.21.15.57:0/705762546 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f5bc406ac90 0x7f5bc406d150 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5bdc000f60 tx=0x7f5bdc009040 comp rx=0 tx=0).ready entity=mgr.11446 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:43.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.435+0000 7f5bd5ffb640 1 -- 172.21.15.57:0/705762546 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f5bd8054f10 con 0x7f5be0156790 2023-10-19T14:21:43.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.619+0000 7f5be734c640 1 -- 172.21.15.57:0/705762546 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5be00690e0 con 0x7f5be0156790 2023-10-19T14:21:43.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.621+0000 7f5bd5ffb640 1 -- 172.21.15.57:0/705762546 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v249) v1 ==== 74+0+21762 (secure 0 0 0) 0x7f5bd801f030 con 0x7f5be0156790 2023-10-19T14:21:43.624 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:21:43.624 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":249,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T14:21:41.395765+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2023-10-19T14:21:08.101023+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"242","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2023-10-19T14:21:08.745817+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"243","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_metadata","create_time":"2023-10-19T14:21:24.384693+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"246","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"backup_fs_data","create_time":"2023-10-19T14:21:25.717524+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"247","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.179:0/3646431735":"2023-10-20T14:21:41.395725+0000","172.21.15.179:0/2986772961":"2023-10-20T14:21:41.395725+0000","172.21.15.179:0/837564123":"2023-10-20T14:21:41.395725+0000","172.21.15.179:0/1917522846":"2023-10-20T14:21:41.395725+0000","172.21.15.179:0/3420876591":"2023-10-20T14:21:41.395725+0000","172.21.15.179:6833/609071913":"2023-10-20T14:21:41.395725+0000","172.21.15.179:6832/609071913":"2023-10-20T14:21:41.395725+0000","172.21.15.179:0/2525508070":"2023-10-20T14:21:35.927420+0000","172.21.15.179:0/75322738":"2023-10-20T14:21:41.395725+0000","172.21.15.179:0/720769249":"2023-10-20T14:21:35.927420+0000","172.21.15.179:6833/3547895240":"2023-10-20T14:21:35.927420+0000","172.21.15.179:6832/3547895240":"2023-10-20T14:21:35.927420+0000","172.21.15.179:0/177563067":"2023-10-20T14:21:35.927420+0000","172.21.15.179:0/1656904674":"2023-10-20T14:21:41.395725+0000","172.21.15.179:0/647327435":"2023-10-20T14:21:41.395725+0000","172.21.15.179:0/1877726906":"2023-10-20T14:21:35.927420+0000","172.21.15.179:0/3471744208":"2023-10-20T14:21:35.927420+0000","172.21.15.179:0/2477290155":"2023-10-20T14:21:35.927420+0000","172.21.15.179:0/1240052109":"2023-10-20T14:21:35.927420+0000","172.21.15.179:0/4042925688":"2023-10-20T14:21:35.927420+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T14:21:43.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.623+0000 7f5be734c640 1 -- 172.21.15.57:0/705762546 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f5bc406ac90 msgr2=0x7f5bc406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:43.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.623+0000 7f5be734c640 1 --2- 172.21.15.57:0/705762546 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f5bc406ac90 0x7f5bc406d150 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5bdc000f60 tx=0x7f5bdc009040 comp rx=0 tx=0).stop 2023-10-19T14:21:43.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.623+0000 7f5be734c640 1 -- 172.21.15.57:0/705762546 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5be0156790 msgr2=0x7f5be0152200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:43.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.623+0000 7f5be734c640 1 --2- 172.21.15.57:0/705762546 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5be0156790 0x7f5be0152200 secure :-1 s=READY pgs=2772 cs=0 l=1 rev1=1 crypto rx=0x7f5bd800e990 tx=0x7f5bd800ee50 comp rx=0 tx=0).stop 2023-10-19T14:21:43.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.623+0000 7f5be734c640 1 -- 172.21.15.57:0/705762546 shutdown_connections 2023-10-19T14:21:43.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.623+0000 7f5be734c640 1 --2- 172.21.15.57:0/705762546 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f5bc406ac90 0x7f5bc406d150 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:43.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.623+0000 7f5be734c640 1 --2- 172.21.15.57:0/705762546 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5be0156790 0x7f5be0152200 unknown :-1 s=CLOSED pgs=2772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:43.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.623+0000 7f5be734c640 1 --2- 172.21.15.57:0/705762546 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5be00ac870 0x7f5be014dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:43.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.623+0000 7f5be734c640 1 --2- 172.21.15.57:0/705762546 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5be006bec0 0x7f5be014d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:43.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.623+0000 7f5be734c640 1 -- 172.21.15.57:0/705762546 >> 172.21.15.57:0/705762546 conn(0x7f5be006fae0 msgr2=0x7f5be00656a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:43.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.623+0000 7f5be734c640 1 -- 172.21.15.57:0/705762546 shutdown_connections 2023-10-19T14:21:43.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.623+0000 7f5be734c640 1 -- 172.21.15.57:0/705762546 wait complete. 2023-10-19T14:21:43.637 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-19T14:21:43.637 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T14:21:43.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.744+0000 7f1041500640 1 -- 172.21.15.57:0/3620289484 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f103c141690 msgr2=0x7f103c0703e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:43.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.744+0000 7f1041500640 1 --2- 172.21.15.57:0/3620289484 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f103c141690 0x7f103c0703e0 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f1024009830 tx=0x7f102402bcd0 comp rx=0 tx=0).stop 2023-10-19T14:21:43.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.744+0000 7f1041500640 1 -- 172.21.15.57:0/3620289484 shutdown_connections 2023-10-19T14:21:43.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.744+0000 7f1041500640 1 --2- 172.21.15.57:0/3620289484 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f103c070920 0x7f103c0affe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:43.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.744+0000 7f1041500640 1 --2- 172.21.15.57:0/3620289484 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f103c141690 0x7f103c0703e0 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:43.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.744+0000 7f1041500640 1 -- 172.21.15.57:0/3620289484 >> 172.21.15.57:0/3620289484 conn(0x7f103c069930 msgr2=0x7f103c069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:43.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.744+0000 7f1041500640 1 -- 172.21.15.57:0/3620289484 shutdown_connections 2023-10-19T14:21:43.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.744+0000 7f1041500640 1 -- 172.21.15.57:0/3620289484 wait complete. 2023-10-19T14:21:43.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.745+0000 7f1041500640 1 Processor -- start 2023-10-19T14:21:43.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.745+0000 7f1041500640 1 -- start start 2023-10-19T14:21:43.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.746+0000 7f1041500640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f103c070920 0x7f103c13b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:43.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.746+0000 7f1041500640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f103c13e520 0x7f103c0ca870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:43.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.746+0000 7f1041500640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f103c141690 0x7f103c0cadb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:43.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.746+0000 7f1041500640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f103c1cfb60 con 0x7f103c13e520 2023-10-19T14:21:43.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.746+0000 7f1041500640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f103c1cfe40 con 0x7f103c141690 2023-10-19T14:21:43.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.746+0000 7f1041500640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f103c1d0120 con 0x7f103c070920 2023-10-19T14:21:43.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.746+0000 7f103a7fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f103c13e520 0x7f103c0ca870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:43.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.746+0000 7f103a7fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f103c13e520 0x7f103c0ca870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44100/0 (socket says 172.21.15.57:44100) 2023-10-19T14:21:43.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.746+0000 7f103a7fc640 1 -- 172.21.15.57:0/1471997069 learned_addr learned my addr 172.21.15.57:0/1471997069 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:43.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.746+0000 7f103affd640 1 --2- 172.21.15.57:0/1471997069 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f103c070920 0x7f103c13b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:43.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.746+0000 7f103b7fe640 1 --2- 172.21.15.57:0/1471997069 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f103c141690 0x7f103c0cadb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:43.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.747+0000 7f103a7fc640 1 -- 172.21.15.57:0/1471997069 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f103c070920 msgr2=0x7f103c13b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:43.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.747+0000 7f103a7fc640 1 --2- 172.21.15.57:0/1471997069 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f103c070920 0x7f103c13b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:43.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.747+0000 7f103a7fc640 1 -- 172.21.15.57:0/1471997069 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f103c141690 msgr2=0x7f103c0cadb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:43.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.747+0000 7f103a7fc640 1 --2- 172.21.15.57:0/1471997069 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f103c141690 0x7f103c0cadb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:43.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.747+0000 7f103a7fc640 1 -- 172.21.15.57:0/1471997069 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f102c009900 con 0x7f103c13e520 2023-10-19T14:21:43.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.747+0000 7f103a7fc640 1 --2- 172.21.15.57:0/1471997069 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f103c13e520 0x7f103c0ca870 secure :-1 s=READY pgs=2773 cs=0 l=1 rev1=1 crypto rx=0x7f1024000c00 tx=0x7f10240038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:43.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.747+0000 7f10337fe640 1 -- 172.21.15.57:0/1471997069 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1024003ea0 con 0x7f103c13e520 2023-10-19T14:21:43.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.747+0000 7f10337fe640 1 -- 172.21.15.57:0/1471997069 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f1024034440 con 0x7f103c13e520 2023-10-19T14:21:43.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.747+0000 7f1041500640 1 -- 172.21.15.57:0/1471997069 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1024009490 con 0x7f103c13e520 2023-10-19T14:21:43.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.747+0000 7f10337fe640 1 -- 172.21.15.57:0/1471997069 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1024034850 con 0x7f103c13e520 2023-10-19T14:21:43.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.747+0000 7f1041500640 1 -- 172.21.15.57:0/1471997069 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f103c1d06e0 con 0x7f103c13e520 2023-10-19T14:21:43.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.748+0000 7f1041500640 1 -- 172.21.15.57:0/1471997069 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f103c0b1bd0 con 0x7f103c13e520 2023-10-19T14:21:43.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.749+0000 7f10337fe640 1 -- 172.21.15.57:0/1471997069 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 143) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f10240349b0 con 0x7f103c13e520 2023-10-19T14:21:43.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.749+0000 7f10337fe640 1 --2- 172.21.15.57:0/1471997069 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f101c06ac90 0x7f101c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:43.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.749+0000 7f10337fe640 1 -- 172.21.15.57:0/1471997069 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 7781+0+0 (secure 0 0 0) 0x7f10240ac1d0 con 0x7f103c13e520 2023-10-19T14:21:43.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.749+0000 7f103affd640 1 --2- 172.21.15.57:0/1471997069 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f101c06ac90 0x7f101c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:43.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.750+0000 7f103affd640 1 --2- 172.21.15.57:0/1471997069 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f101c06ac90 0x7f101c06d150 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1034001780 tx=0x7f1034009040 comp rx=0 tx=0).ready entity=mgr.11446 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:43.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.753+0000 7f10337fe640 1 -- 172.21.15.57:0/1471997069 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f1024076e50 con 0x7f103c13e520 2023-10-19T14:21:43.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.965+0000 7f1041500640 1 -- 172.21.15.57:0/1471997069 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f103c1d08d0 con 0x7f103c13e520 2023-10-19T14:21:43.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.966+0000 7f10337fe640 1 -- 172.21.15.57:0/1471997069 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+6140 (secure 0 0 0) 0x7f1024042090 con 0x7f103c13e520 2023-10-19T14:21:43.969 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:21:43.969 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":107,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9761,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/325888554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":325888554},{"type":"v1","addr":"172.21.15.179:6836","nonce":325888554}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":100},{"gid":9764,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/8038232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":8038232},{"type":"v1","addr":"172.21.15.179:6837","nonce":8038232}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":100}],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T14:21:09.759935+0000","modified":"2023-10-19T14:21:11.768676+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":12564},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12564":{"gid":12564,"name":"c","rank":0,"incarnation":101,"state":"up:active","state_seq":3,"addr":"172.21.15.57:6835/18187246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":18187246},{"type":"v1","addr":"172.21.15.57:6835","nonce":18187246}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T14:21:26.724728+0000","modified":"2023-10-19T14:21:28.872856+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":12555},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12555":{"gid":12555,"name":"a","rank":0,"incarnation":105,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6833/4256498777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":4256498777},{"type":"v1","addr":"172.21.15.57:6833","nonce":4256498777}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2023-10-19T14:21:43.970 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 107 2023-10-19T14:21:43.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.967+0000 7f1041500640 1 -- 172.21.15.57:0/1471997069 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f101c06ac90 msgr2=0x7f101c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:43.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.967+0000 7f1041500640 1 --2- 172.21.15.57:0/1471997069 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f101c06ac90 0x7f101c06d150 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1034001780 tx=0x7f1034009040 comp rx=0 tx=0).stop 2023-10-19T14:21:43.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.967+0000 7f1041500640 1 -- 172.21.15.57:0/1471997069 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f103c13e520 msgr2=0x7f103c0ca870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:43.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.967+0000 7f1041500640 1 --2- 172.21.15.57:0/1471997069 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f103c13e520 0x7f103c0ca870 secure :-1 s=READY pgs=2773 cs=0 l=1 rev1=1 crypto rx=0x7f1024000c00 tx=0x7f10240038a0 comp rx=0 tx=0).stop 2023-10-19T14:21:43.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.968+0000 7f1041500640 1 -- 172.21.15.57:0/1471997069 shutdown_connections 2023-10-19T14:21:43.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.968+0000 7f1041500640 1 --2- 172.21.15.57:0/1471997069 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f101c06ac90 0x7f101c06d150 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:43.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.968+0000 7f1041500640 1 --2- 172.21.15.57:0/1471997069 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f103c141690 0x7f103c0cadb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:43.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.968+0000 7f1041500640 1 --2- 172.21.15.57:0/1471997069 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f103c13e520 0x7f103c0ca870 unknown :-1 s=CLOSED pgs=2773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:43.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.968+0000 7f1041500640 1 --2- 172.21.15.57:0/1471997069 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f103c070920 0x7f103c13b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:43.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.968+0000 7f1041500640 1 -- 172.21.15.57:0/1471997069 >> 172.21.15.57:0/1471997069 conn(0x7f103c069930 msgr2=0x7f103c06f990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:43.973 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.968+0000 7f1041500640 1 -- 172.21.15.57:0/1471997069 shutdown_connections 2023-10-19T14:21:43.973 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:43.968+0000 7f1041500640 1 -- 172.21.15.57:0/1471997069 wait complete. 2023-10-19T14:21:43.981 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T14:21:44.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.088+0000 7f6b1075e640 1 -- 172.21.15.57:0/2508966345 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6b0806af00 msgr2=0x7f6b0806b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:44.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.088+0000 7f6b1075e640 1 --2- 172.21.15.57:0/2508966345 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6b0806af00 0x7f6b0806b2e0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f6b04005fd0 tx=0x7f6b0402be10 comp rx=0 tx=0).stop 2023-10-19T14:21:44.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.088+0000 7f6b1075e640 1 -- 172.21.15.57:0/2508966345 shutdown_connections 2023-10-19T14:21:44.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.088+0000 7f6b1075e640 1 --2- 172.21.15.57:0/2508966345 >> v2:172.21.15.179:3300/0 conn(0x7f6b080c96b0 0x7f6b080c9b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:44.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.088+0000 7f6b1075e640 1 --2- 172.21.15.57:0/2508966345 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6b0806af00 0x7f6b0806b2e0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:44.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.088+0000 7f6b1075e640 1 -- 172.21.15.57:0/2508966345 >> 172.21.15.57:0/2508966345 conn(0x7f6b08069930 msgr2=0x7f6b08069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:44.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.088+0000 7f6b1075e640 1 -- 172.21.15.57:0/2508966345 shutdown_connections 2023-10-19T14:21:44.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.088+0000 7f6b1075e640 1 -- 172.21.15.57:0/2508966345 wait complete. 2023-10-19T14:21:44.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.089+0000 7f6b1075e640 1 Processor -- start 2023-10-19T14:21:44.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.089+0000 7f6b1075e640 1 -- start start 2023-10-19T14:21:44.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.090+0000 7f6b1075e640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6b0806af00 0x7f6b0813a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:44.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.090+0000 7f6b1075e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6b080ab500 0x7f6b08158820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:44.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.090+0000 7f6b1075e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6b080c96b0 0x7f6b080aa8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:44.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.090+0000 7f6b1075e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b0813ab90 con 0x7f6b080ab500 2023-10-19T14:21:44.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.090+0000 7f6b1075e640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b0806f770 con 0x7f6b080c96b0 2023-10-19T14:21:44.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.090+0000 7f6b1075e640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b081546a0 con 0x7f6b0806af00 2023-10-19T14:21:44.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.090+0000 7f6b0dcf9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6b080ab500 0x7f6b08158820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:44.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.090+0000 7f6b0dcf9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6b080ab500 0x7f6b08158820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44110/0 (socket says 172.21.15.57:44110) 2023-10-19T14:21:44.096 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.090+0000 7f6b0dcf9640 1 -- 172.21.15.57:0/3557063594 learned_addr learned my addr 172.21.15.57:0/3557063594 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:44.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.090+0000 7f6b0e4fa640 1 --2- 172.21.15.57:0/3557063594 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6b0806af00 0x7f6b0813a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:44.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.090+0000 7f6b0ecfb640 1 --2- 172.21.15.57:0/3557063594 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6b080c96b0 0x7f6b080aa8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:44.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.090+0000 7f6b0dcf9640 1 -- 172.21.15.57:0/3557063594 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6b0806af00 msgr2=0x7f6b0813a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:44.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.090+0000 7f6b0dcf9640 1 --2- 172.21.15.57:0/3557063594 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6b0806af00 0x7f6b0813a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:44.097 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.090+0000 7f6b0dcf9640 1 -- 172.21.15.57:0/3557063594 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6b080c96b0 msgr2=0x7f6b080aa8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:44.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.090+0000 7f6b0dcf9640 1 --2- 172.21.15.57:0/3557063594 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6b080c96b0 0x7f6b080aa8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:44.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.090+0000 7f6b0dcf9640 1 -- 172.21.15.57:0/3557063594 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b04009560 con 0x7f6b080ab500 2023-10-19T14:21:44.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.091+0000 7f6b0dcf9640 1 --2- 172.21.15.57:0/3557063594 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6b080ab500 0x7f6b08158820 secure :-1 s=READY pgs=2774 cs=0 l=1 rev1=1 crypto rx=0x7f6af800e790 tx=0x7f6af800ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:44.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.091+0000 7f6afeffd640 1 -- 172.21.15.57:0/3557063594 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6af8009580 con 0x7f6b080ab500 2023-10-19T14:21:44.098 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.091+0000 7f6afeffd640 1 -- 172.21.15.57:0/3557063594 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f6af800cdb0 con 0x7f6b080ab500 2023-10-19T14:21:44.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.091+0000 7f6b1075e640 1 -- 172.21.15.57:0/3557063594 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b0806f120 con 0x7f6b080ab500 2023-10-19T14:21:44.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.091+0000 7f6afeffd640 1 -- 172.21.15.57:0/3557063594 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6af8010690 con 0x7f6b080ab500 2023-10-19T14:21:44.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.091+0000 7f6b1075e640 1 -- 172.21.15.57:0/3557063594 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b08154980 con 0x7f6b080ab500 2023-10-19T14:21:44.099 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.092+0000 7f6b1075e640 1 -- 172.21.15.57:0/3557063594 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b0806bf40 con 0x7f6b080ab500 2023-10-19T14:21:44.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.092+0000 7f6afeffd640 1 -- 172.21.15.57:0/3557063594 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 143) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f6af80107f0 con 0x7f6b080ab500 2023-10-19T14:21:44.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.093+0000 7f6afeffd640 1 --2- 172.21.15.57:0/3557063594 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f6aec06ac90 0x7f6aec06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:44.100 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.093+0000 7f6afeffd640 1 -- 172.21.15.57:0/3557063594 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 7781+0+0 (secure 0 0 0) 0x7f6af8014070 con 0x7f6b080ab500 2023-10-19T14:21:44.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.093+0000 7f6b0e4fa640 1 --2- 172.21.15.57:0/3557063594 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f6aec06ac90 0x7f6aec06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:44.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.094+0000 7f6b0e4fa640 1 --2- 172.21.15.57:0/3557063594 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f6aec06ac90 0x7f6aec06d150 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6b04005fd0 tx=0x7f6b04004730 comp rx=0 tx=0).ready entity=mgr.11446 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:44.101 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.097+0000 7f6afeffd640 1 -- 172.21.15.57:0/3557063594 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6af8051ca0 con 0x7f6b080ab500 2023-10-19T14:21:44.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.307+0000 7f6b1075e640 1 -- 172.21.15.57:0/3557063594 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6b080690e0 con 0x7f6b080ab500 2023-10-19T14:21:44.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.308+0000 7f6afeffd640 1 -- 172.21.15.57:0/3557063594 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+6140 (secure 0 0 0) 0x7f6af80550d0 con 0x7f6b080ab500 2023-10-19T14:21:44.311 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:21:44.311 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":107,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9761,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/325888554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":325888554},{"type":"v1","addr":"172.21.15.179:6836","nonce":325888554}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":100},{"gid":9764,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/8038232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":8038232},{"type":"v1","addr":"172.21.15.179:6837","nonce":8038232}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":100}],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T14:21:09.759935+0000","modified":"2023-10-19T14:21:11.768676+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":12564},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12564":{"gid":12564,"name":"c","rank":0,"incarnation":101,"state":"up:active","state_seq":3,"addr":"172.21.15.57:6835/18187246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":18187246},{"type":"v1","addr":"172.21.15.57:6835","nonce":18187246}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T14:21:26.724728+0000","modified":"2023-10-19T14:21:28.872856+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":12555},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12555":{"gid":12555,"name":"a","rank":0,"incarnation":105,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6833/4256498777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":4256498777},{"type":"v1","addr":"172.21.15.57:6833","nonce":4256498777}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2023-10-19T14:21:44.313 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 107 2023-10-19T14:21:44.313 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.309+0000 7f6b1075e640 1 -- 172.21.15.57:0/3557063594 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f6aec06ac90 msgr2=0x7f6aec06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:44.313 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.309+0000 7f6b1075e640 1 --2- 172.21.15.57:0/3557063594 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f6aec06ac90 0x7f6aec06d150 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6b04005fd0 tx=0x7f6b04004730 comp rx=0 tx=0).stop 2023-10-19T14:21:44.313 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.309+0000 7f6b1075e640 1 -- 172.21.15.57:0/3557063594 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6b080ab500 msgr2=0x7f6b08158820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:44.314 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.309+0000 7f6b1075e640 1 --2- 172.21.15.57:0/3557063594 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6b080ab500 0x7f6b08158820 secure :-1 s=READY pgs=2774 cs=0 l=1 rev1=1 crypto rx=0x7f6af800e790 tx=0x7f6af800ec50 comp rx=0 tx=0).stop 2023-10-19T14:21:44.314 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.310+0000 7f6b1075e640 1 -- 172.21.15.57:0/3557063594 shutdown_connections 2023-10-19T14:21:44.315 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.310+0000 7f6b1075e640 1 --2- 172.21.15.57:0/3557063594 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f6aec06ac90 0x7f6aec06d150 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:44.315 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.310+0000 7f6b1075e640 1 --2- 172.21.15.57:0/3557063594 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6b080c96b0 0x7f6b080aa8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:44.315 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.310+0000 7f6b1075e640 1 --2- 172.21.15.57:0/3557063594 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6b080ab500 0x7f6b08158820 unknown :-1 s=CLOSED pgs=2774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:44.315 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.310+0000 7f6b1075e640 1 --2- 172.21.15.57:0/3557063594 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6b0806af00 0x7f6b0813a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:44.315 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.310+0000 7f6b1075e640 1 -- 172.21.15.57:0/3557063594 >> 172.21.15.57:0/3557063594 conn(0x7f6b08069930 msgr2=0x7f6b08069d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:44.316 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.310+0000 7f6b1075e640 1 -- 172.21.15.57:0/3557063594 shutdown_connections 2023-10-19T14:21:44.316 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.310+0000 7f6b1075e640 1 -- 172.21.15.57:0/3557063594 wait complete. 2023-10-19T14:21:44.323 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T14:21:44.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.433+0000 7f28bc0c9640 1 -- 172.21.15.57:0/3502728710 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f28a0004d00 msgr2=0x7f28a0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:44.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.433+0000 7f28bc0c9640 1 --2- 172.21.15.57:0/3502728710 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f28a0004d00 0x7f28a0005180 secure :-1 s=READY pgs=2776 cs=0 l=1 rev1=1 crypto rx=0x7f28ac001d80 tx=0x7f28a00057f0 comp rx=0 tx=0).stop 2023-10-19T14:21:44.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.433+0000 7f28ba666640 1 -- 172.21.15.57:0/3502728710 reap_dead start 2023-10-19T14:21:44.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.433+0000 7f28bc0c9640 1 -- 172.21.15.57:0/3502728710 shutdown_connections 2023-10-19T14:21:44.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.433+0000 7f28bc0c9640 1 -- 172.21.15.57:0/3502728710 >> 172.21.15.57:0/3502728710 conn(0x7f28b4069930 msgr2=0x7f28b4069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:44.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.433+0000 7f28bc0c9640 1 -- 172.21.15.57:0/3502728710 shutdown_connections 2023-10-19T14:21:44.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.433+0000 7f28bc0c9640 1 -- 172.21.15.57:0/3502728710 wait complete. 2023-10-19T14:21:44.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.434+0000 7f28bc0c9640 1 Processor -- start 2023-10-19T14:21:44.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.435+0000 7f28bc0c9640 1 -- start start 2023-10-19T14:21:44.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.435+0000 7f28bc0c9640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f28a0004d00 0x7f28b413a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:44.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.435+0000 7f28bc0c9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f28b413ace0 0x7f28b41529c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:44.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.435+0000 7f28bc0c9640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f28b40b07b0 0x7f28b40b0c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:44.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.435+0000 7f28bc0c9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28b40b1170 con 0x7f28b413ace0 2023-10-19T14:21:44.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.435+0000 7f28bc0c9640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28b4152530 con 0x7f28a0004d00 2023-10-19T14:21:44.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.435+0000 7f28bc0c9640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28b4151920 con 0x7f28b40b07b0 2023-10-19T14:21:44.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.435+0000 7f28b9664640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f28b413ace0 0x7f28b41529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:44.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.435+0000 7f28b9664640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f28b413ace0 0x7f28b41529c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44142/0 (socket says 172.21.15.57:44142) 2023-10-19T14:21:44.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.435+0000 7f28b9664640 1 -- 172.21.15.57:0/2988562702 learned_addr learned my addr 172.21.15.57:0/2988562702 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:44.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.435+0000 7f28b9e65640 1 --2- 172.21.15.57:0/2988562702 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f28a0004d00 0x7f28b413a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:44.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.435+0000 7f28ba666640 1 --2- 172.21.15.57:0/2988562702 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f28b40b07b0 0x7f28b40b0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:44.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.436+0000 7f28b9664640 1 -- 172.21.15.57:0/2988562702 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f28b40b07b0 msgr2=0x7f28b40b0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:44.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.436+0000 7f28b9664640 1 --2- 172.21.15.57:0/2988562702 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f28b40b07b0 0x7f28b40b0c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:44.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.436+0000 7f28b9664640 1 -- 172.21.15.57:0/2988562702 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f28a0004d00 msgr2=0x7f28b413a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:44.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.436+0000 7f28b9664640 1 --2- 172.21.15.57:0/2988562702 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f28a0004d00 0x7f28b413a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:44.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.436+0000 7f28b9664640 1 -- 172.21.15.57:0/2988562702 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28b4151c00 con 0x7f28b413ace0 2023-10-19T14:21:44.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.436+0000 7f28b9664640 1 --2- 172.21.15.57:0/2988562702 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f28b413ace0 0x7f28b41529c0 secure :-1 s=READY pgs=2777 cs=0 l=1 rev1=1 crypto rx=0x7f28a4004650 tx=0x7f28a400ba40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:44.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.436+0000 7f28aa7fc640 1 -- 172.21.15.57:0/2988562702 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28a40041d0 con 0x7f28b413ace0 2023-10-19T14:21:44.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.436+0000 7f28aa7fc640 1 -- 172.21.15.57:0/2988562702 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f28a4015070 con 0x7f28b413ace0 2023-10-19T14:21:44.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.436+0000 7f28aa7fc640 1 -- 172.21.15.57:0/2988562702 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28a4010ad0 con 0x7f28b413ace0 2023-10-19T14:21:44.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.436+0000 7f28bc0c9640 1 -- 172.21.15.57:0/2988562702 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28b4151ee0 con 0x7f28b413ace0 2023-10-19T14:21:44.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.436+0000 7f28bc0c9640 1 -- 172.21.15.57:0/2988562702 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28b413b4c0 con 0x7f28b413ace0 2023-10-19T14:21:44.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.437+0000 7f28bc0c9640 1 -- 172.21.15.57:0/2988562702 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28b40690e0 con 0x7f28b413ace0 2023-10-19T14:21:44.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.437+0000 7f28aa7fc640 1 -- 172.21.15.57:0/2988562702 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 144) v1 ==== 89377+0+0 (secure 0 0 0) 0x7f28a4017030 con 0x7f28b413ace0 2023-10-19T14:21:44.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.438+0000 7f28aa7fc640 1 --2- 172.21.15.57:0/2988562702 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f289806aff0 0x7f289806d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:44.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.438+0000 7f28aa7fc640 1 -- 172.21.15.57:0/2988562702 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 7781+0+0 (secure 0 0 0) 0x7f28a408a9a0 con 0x7f28b413ace0 2023-10-19T14:21:44.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.438+0000 7f28b9e65640 1 --2- 172.21.15.57:0/2988562702 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f289806aff0 0x7f289806d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:44.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.439+0000 7f28b9e65640 1 --2- 172.21.15.57:0/2988562702 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f289806aff0 0x7f289806d4b0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f28b0001050 tx=0x7f28b0000fa0 comp rx=0 tx=0).ready entity=mgr.11446 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:44.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.442+0000 7f28aa7fc640 1 -- 172.21.15.57:0/2988562702 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f28a4055070 con 0x7f28b413ace0 2023-10-19T14:21:44.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.620+0000 7f28bc0c9640 1 -- 172.21.15.57:0/2988562702 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f28b40cc420 con 0x7f28b413ace0 2023-10-19T14:21:44.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.621+0000 7f28aa7fc640 1 -- 172.21.15.57:0/2988562702 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v249) v1 ==== 74+0+21762 (secure 0 0 0) 0x7f28a4058690 con 0x7f28b413ace0 2023-10-19T14:21:44.627 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:21:44.627 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":249,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T14:21:41.395765+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2023-10-19T14:21:08.101023+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"242","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2023-10-19T14:21:08.745817+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"243","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_metadata","create_time":"2023-10-19T14:21:24.384693+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"246","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"backup_fs_data","create_time":"2023-10-19T14:21:25.717524+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"247","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.179:0/3646431735":"2023-10-20T14:21:41.395725+0000","172.21.15.179:0/2986772961":"2023-10-20T14:21:41.395725+0000","172.21.15.179:0/837564123":"2023-10-20T14:21:41.395725+0000","172.21.15.179:0/1917522846":"2023-10-20T14:21:41.395725+0000","172.21.15.179:0/3420876591":"2023-10-20T14:21:41.395725+0000","172.21.15.179:6833/609071913":"2023-10-20T14:21:41.395725+0000","172.21.15.179:6832/609071913":"2023-10-20T14:21:41.395725+0000","172.21.15.179:0/2525508070":"2023-10-20T14:21:35.927420+0000","172.21.15.179:0/75322738":"2023-10-20T14:21:41.395725+0000","172.21.15.179:0/720769249":"2023-10-20T14:21:35.927420+0000","172.21.15.179:6833/3547895240":"2023-10-20T14:21:35.927420+0000","172.21.15.179:6832/3547895240":"2023-10-20T14:21:35.927420+0000","172.21.15.179:0/177563067":"2023-10-20T14:21:35.927420+0000","172.21.15.179:0/1656904674":"2023-10-20T14:21:41.395725+0000","172.21.15.179:0/647327435":"2023-10-20T14:21:41.395725+0000","172.21.15.179:0/1877726906":"2023-10-20T14:21:35.927420+0000","172.21.15.179:0/3471744208":"2023-10-20T14:21:35.927420+0000","172.21.15.179:0/2477290155":"2023-10-20T14:21:35.927420+0000","172.21.15.179:0/1240052109":"2023-10-20T14:21:35.927420+0000","172.21.15.179:0/4042925688":"2023-10-20T14:21:35.927420+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T14:21:44.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.623+0000 7f28bc0c9640 1 -- 172.21.15.57:0/2988562702 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f289806aff0 msgr2=0x7f289806d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:44.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.623+0000 7f28bc0c9640 1 --2- 172.21.15.57:0/2988562702 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f289806aff0 0x7f289806d4b0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f28b0001050 tx=0x7f28b0000fa0 comp rx=0 tx=0).stop 2023-10-19T14:21:44.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.623+0000 7f28bc0c9640 1 -- 172.21.15.57:0/2988562702 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f28b413ace0 msgr2=0x7f28b41529c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:44.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.623+0000 7f28bc0c9640 1 --2- 172.21.15.57:0/2988562702 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f28b413ace0 0x7f28b41529c0 secure :-1 s=READY pgs=2777 cs=0 l=1 rev1=1 crypto rx=0x7f28a4004650 tx=0x7f28a400ba40 comp rx=0 tx=0).stop 2023-10-19T14:21:44.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.623+0000 7f28bc0c9640 1 -- 172.21.15.57:0/2988562702 shutdown_connections 2023-10-19T14:21:44.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.623+0000 7f28bc0c9640 1 --2- 172.21.15.57:0/2988562702 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f289806aff0 0x7f289806d4b0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:44.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.623+0000 7f28bc0c9640 1 --2- 172.21.15.57:0/2988562702 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f28b40b07b0 0x7f28b40b0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:44.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.623+0000 7f28bc0c9640 1 --2- 172.21.15.57:0/2988562702 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f28b413ace0 0x7f28b41529c0 unknown :-1 s=CLOSED pgs=2777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:44.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.623+0000 7f28bc0c9640 1 --2- 172.21.15.57:0/2988562702 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f28a0004d00 0x7f28b413a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:44.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.623+0000 7f28bc0c9640 1 -- 172.21.15.57:0/2988562702 >> 172.21.15.57:0/2988562702 conn(0x7f28b4069930 msgr2=0x7f28b415a3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:44.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.623+0000 7f28bc0c9640 1 -- 172.21.15.57:0/2988562702 shutdown_connections 2023-10-19T14:21:44.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.623+0000 7f28bc0c9640 1 -- 172.21.15.57:0/2988562702 wait complete. 2023-10-19T14:21:44.637 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-19T14:21:44.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.745+0000 7f755e753640 1 -- 172.21.15.57:0/3522724335 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7544005040 msgr2=0x7f754400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:44.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.745+0000 7f755e753640 1 --2- 172.21.15.57:0/3522724335 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7544005040 0x7f754400b980 secure :-1 s=READY pgs=2779 cs=0 l=1 rev1=1 crypto rx=0x7f7550001e90 tx=0x7f75440057f0 comp rx=0 tx=0).stop 2023-10-19T14:21:44.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.745+0000 7f755e753640 1 -- 172.21.15.57:0/3522724335 shutdown_connections 2023-10-19T14:21:44.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.745+0000 7f755e753640 1 --2- 172.21.15.57:0/3522724335 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7544005040 0x7f754400b980 unknown :-1 s=CLOSED pgs=2779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:44.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.745+0000 7f755e753640 1 -- 172.21.15.57:0/3522724335 >> 172.21.15.57:0/3522724335 conn(0x7f75580af7a0 msgr2=0x7f75580afb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:44.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.745+0000 7f755e753640 1 -- 172.21.15.57:0/3522724335 shutdown_connections 2023-10-19T14:21:44.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.745+0000 7f755e753640 1 -- 172.21.15.57:0/3522724335 wait complete. 2023-10-19T14:21:44.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.746+0000 7f755e753640 1 Processor -- start 2023-10-19T14:21:44.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.746+0000 7f755e753640 1 -- start start 2023-10-19T14:21:44.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.747+0000 7f755e753640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7544005040 0x7f7558145710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:44.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.747+0000 7f755e753640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f75581f22c0 0x7f75581ef720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:44.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.747+0000 7f755e753640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f755814ac60 0x7f755814b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:44.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.747+0000 7f755e753640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75581effa0 con 0x7f75581f22c0 2023-10-19T14:21:44.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.747+0000 7f755e753640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f755814b620 con 0x7f7544005040 2023-10-19T14:21:44.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.747+0000 7f755e753640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75581efc60 con 0x7f755814ac60 2023-10-19T14:21:44.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.747+0000 7f75577fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f75581f22c0 0x7f75581ef720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:44.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.747+0000 7f75577fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f75581f22c0 0x7f75581ef720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44162/0 (socket says 172.21.15.57:44162) 2023-10-19T14:21:44.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.747+0000 7f75577fe640 1 -- 172.21.15.57:0/133732421 learned_addr learned my addr 172.21.15.57:0/133732421 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:44.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.747+0000 7f7557fff640 1 --2- 172.21.15.57:0/133732421 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7544005040 0x7f7558145710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:44.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.747+0000 7f755ccf0640 1 --2- 172.21.15.57:0/133732421 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f755814ac60 0x7f755814b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:44.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.747+0000 7f75577fe640 1 -- 172.21.15.57:0/133732421 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f755814ac60 msgr2=0x7f755814b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:44.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.747+0000 7f75577fe640 1 --2- 172.21.15.57:0/133732421 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f755814ac60 0x7f755814b0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:44.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.747+0000 7f75577fe640 1 -- 172.21.15.57:0/133732421 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7544005040 msgr2=0x7f7558145710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:44.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.747+0000 7f75577fe640 1 --2- 172.21.15.57:0/133732421 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7544005040 0x7f7558145710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:44.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.747+0000 7f75577fe640 1 -- 172.21.15.57:0/133732421 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7558149860 con 0x7f75581f22c0 2023-10-19T14:21:44.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.747+0000 7f75577fe640 1 --2- 172.21.15.57:0/133732421 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f75581f22c0 0x7f75581ef720 secure :-1 s=READY pgs=2780 cs=0 l=1 rev1=1 crypto rx=0x7f7548011ab0 tx=0x7f7548011f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:44.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.748+0000 7f7554ff9640 1 -- 172.21.15.57:0/133732421 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f754800fe70 con 0x7f75581f22c0 2023-10-19T14:21:44.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.748+0000 7f7554ff9640 1 -- 172.21.15.57:0/133732421 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f7548004050 con 0x7f75581f22c0 2023-10-19T14:21:44.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.748+0000 7f755e753640 1 -- 172.21.15.57:0/133732421 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75581f2830 con 0x7f75581f22c0 2023-10-19T14:21:44.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.748+0000 7f7554ff9640 1 -- 172.21.15.57:0/133732421 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7548004440 con 0x7f75581f22c0 2023-10-19T14:21:44.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.748+0000 7f755e753640 1 -- 172.21.15.57:0/133732421 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75581f2df0 con 0x7f75581f22c0 2023-10-19T14:21:44.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.749+0000 7f755e753640 1 -- 172.21.15.57:0/133732421 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75580690e0 con 0x7f75581f22c0 2023-10-19T14:21:44.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.749+0000 7f7554ff9640 1 -- 172.21.15.57:0/133732421 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 144) v1 ==== 89377+0+0 (secure 0 0 0) 0x7f75480035f0 con 0x7f75581f22c0 2023-10-19T14:21:44.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.750+0000 7f7554ff9640 1 --2- 172.21.15.57:0/133732421 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f753806b020 0x7f753806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:44.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.750+0000 7f7554ff9640 1 -- 172.21.15.57:0/133732421 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 7781+0+0 (secure 0 0 0) 0x7f754808ae80 con 0x7f75581f22c0 2023-10-19T14:21:44.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.750+0000 7f7557fff640 1 --2- 172.21.15.57:0/133732421 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f753806b020 0x7f753806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:44.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.751+0000 7f7557fff640 1 --2- 172.21.15.57:0/133732421 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f753806b020 0x7f753806d4e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f7540001050 tx=0x7f7540001110 comp rx=0 tx=0).ready entity=mgr.11446 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:44.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.754+0000 7f7554ff9640 1 -- 172.21.15.57:0/133732421 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f75480554c0 con 0x7f75581f22c0 2023-10-19T14:21:44.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.966+0000 7f755e753640 1 -- 172.21.15.57:0/133732421 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f75580ce5e0 con 0x7f75581f22c0 2023-10-19T14:21:44.969 INFO:tasks.ceph.mon.a.smithi057.stderr:2023-10-19T14:21:44.967+0000 7f1253ff3640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-19T14:21:44.974 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.972+0000 7f7554ff9640 1 -- 172.21.15.57:0/133732421 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v108) v1 ==== 161+0+0 (secure 0 0 0) 0x7f7548058ae0 con 0x7f75581f22c0 2023-10-19T14:21:44.975 INFO:teuthology.orchestra.run.smithi057.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-19T14:21:44.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.974+0000 7f755e753640 1 -- 172.21.15.57:0/133732421 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f753806b020 msgr2=0x7f753806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:44.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.974+0000 7f755e753640 1 --2- 172.21.15.57:0/133732421 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f753806b020 0x7f753806d4e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f7540001050 tx=0x7f7540001110 comp rx=0 tx=0).stop 2023-10-19T14:21:44.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.974+0000 7f755e753640 1 -- 172.21.15.57:0/133732421 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f75581f22c0 msgr2=0x7f75581ef720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:44.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.974+0000 7f755e753640 1 --2- 172.21.15.57:0/133732421 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f75581f22c0 0x7f75581ef720 secure :-1 s=READY pgs=2780 cs=0 l=1 rev1=1 crypto rx=0x7f7548011ab0 tx=0x7f7548011f70 comp rx=0 tx=0).stop 2023-10-19T14:21:44.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.974+0000 7f755e753640 1 -- 172.21.15.57:0/133732421 shutdown_connections 2023-10-19T14:21:44.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.974+0000 7f755e753640 1 --2- 172.21.15.57:0/133732421 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f753806b020 0x7f753806d4e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:44.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.974+0000 7f755e753640 1 --2- 172.21.15.57:0/133732421 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f755814ac60 0x7f755814b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:44.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.974+0000 7f755e753640 1 --2- 172.21.15.57:0/133732421 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f75581f22c0 0x7f75581ef720 unknown :-1 s=CLOSED pgs=2780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:44.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.974+0000 7f755e753640 1 --2- 172.21.15.57:0/133732421 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7544005040 0x7f7558145710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:44.979 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.974+0000 7f755e753640 1 -- 172.21.15.57:0/133732421 >> 172.21.15.57:0/133732421 conn(0x7f75580af7a0 msgr2=0x7f755815a2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:44.979 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.975+0000 7f755e753640 1 -- 172.21.15.57:0/133732421 shutdown_connections 2023-10-19T14:21:44.979 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:44.975+0000 7f755e753640 1 -- 172.21.15.57:0/133732421 wait complete. 2023-10-19T14:21:44.980 INFO:tasks.ceph.mds.c.smithi057.stderr: -3239> 2023-10-19T14:21:12.379+0000 7f7968d74640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-19T14:21:44.990 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-19T14:21:45.008 INFO:tasks.ceph.mds.c.smithi057.stdout:starting mds.c at 2023-10-19T14:21:45.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:45.100+0000 7f0bb1407640 1 -- 172.21.15.57:0/1277770712 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0bac13e520 msgr2=0x7f0bac06bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:45.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:45.100+0000 7f0bb1407640 1 --2- 172.21.15.57:0/1277770712 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0bac13e520 0x7f0bac06bb60 secure :-1 s=READY pgs=2783 cs=0 l=1 rev1=1 crypto rx=0x7f0ba4009830 tx=0x7f0ba402bc30 comp rx=0 tx=0).stop 2023-10-19T14:21:45.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:45.100+0000 7f0bb1407640 1 -- 172.21.15.57:0/1277770712 shutdown_connections 2023-10-19T14:21:45.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:45.100+0000 7f0bb1407640 1 --2- 172.21.15.57:0/1277770712 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0bac141690 0x7f0bac0703e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:45.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:45.100+0000 7f0bb1407640 1 --2- 172.21.15.57:0/1277770712 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0bac13e520 0x7f0bac06bb60 unknown :-1 s=CLOSED pgs=2783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:45.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:45.100+0000 7f0bb1407640 1 -- 172.21.15.57:0/1277770712 >> 172.21.15.57:0/1277770712 conn(0x7f0bac069930 msgr2=0x7f0bac069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:45.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:45.100+0000 7f0bb1407640 1 -- 172.21.15.57:0/1277770712 shutdown_connections 2023-10-19T14:21:45.105 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:45.100+0000 7f0bb1407640 1 -- 172.21.15.57:0/1277770712 wait complete. 2023-10-19T14:21:45.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:45.101+0000 7f0bb1407640 1 Processor -- start 2023-10-19T14:21:45.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:45.101+0000 7f0bb1407640 1 -- start start 2023-10-19T14:21:45.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:45.102+0000 7f0bb1407640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0bac06c0a0 0x7f0bac13ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:45.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:45.102+0000 7f0bb1407640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0bac13e520 0x7f0bac1d8d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:45.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:45.102+0000 7f0bb1407640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0bac141690 0x7f0bac0ca870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:45.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:45.102+0000 7f0bb1407640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bac1d4fe0 con 0x7f0bac141690 2023-10-19T14:21:45.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:45.102+0000 7f0bb1407640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bac1d5880 con 0x7f0bac06c0a0 2023-10-19T14:21:45.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:45.102+0000 7f0bb1407640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bac1d9810 con 0x7f0bac13e520 2023-10-19T14:21:45.109 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:45.102+0000 7f0bab7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0bac141690 0x7f0bac0ca870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:45.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:45.102+0000 7f0bab7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0bac141690 0x7f0bac0ca870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44202/0 (socket says 172.21.15.57:44202) 2023-10-19T14:21:45.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:45.102+0000 7f0bab7fe640 1 -- 172.21.15.57:0/425324557 learned_addr learned my addr 172.21.15.57:0/425324557 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:45.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:45.102+0000 7f0baa7fc640 1 --2- 172.21.15.57:0/425324557 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0bac13e520 0x7f0bac1d8d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:45.110 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:45.102+0000 7f0bab7fe640 1 -- 172.21.15.57:0/425324557 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0bac13e520 msgr2=0x7f0bac1d8d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:45.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:45.102+0000 7f0bab7fe640 1 --2- 172.21.15.57:0/425324557 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0bac13e520 0x7f0bac1d8d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:45.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:45.102+0000 7f0baaffd640 1 --2- 172.21.15.57:0/425324557 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0bac06c0a0 0x7f0bac13ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:45.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:45.102+0000 7f0bab7fe640 1 -- 172.21.15.57:0/425324557 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0bac06c0a0 msgr2=0x7f0bac13ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:45.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:45.102+0000 7f0bab7fe640 1 --2- 172.21.15.57:0/425324557 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0bac06c0a0 0x7f0bac13ae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:45.111 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:45.102+0000 7f0bab7fe640 1 -- 172.21.15.57:0/425324557 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ba4009490 con 0x7f0bac141690 2023-10-19T14:21:45.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:45.102+0000 7f0bab7fe640 1 --2- 172.21.15.57:0/425324557 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0bac141690 0x7f0bac0ca870 secure :-1 s=READY pgs=2784 cs=0 l=1 rev1=1 crypto rx=0x7f0b9c00c760 tx=0x7f0b9c00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:45.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:45.103+0000 7f0ba37fe640 1 -- 172.21.15.57:0/425324557 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b9c003da0 con 0x7f0bac141690 2023-10-19T14:21:45.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:45.103+0000 7f0ba37fe640 1 -- 172.21.15.57:0/425324557 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f0b9c00ce70 con 0x7f0bac141690 2023-10-19T14:21:45.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:45.103+0000 7f0bb1407640 1 -- 172.21.15.57:0/425324557 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bac1d5230 con 0x7f0bac141690 2023-10-19T14:21:45.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:45.103+0000 7f0ba37fe640 1 -- 172.21.15.57:0/425324557 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b9c0052b0 con 0x7f0bac141690 2023-10-19T14:21:45.112 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:45.103+0000 7f0bb1407640 1 -- 172.21.15.57:0/425324557 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bac1d7ec0 con 0x7f0bac141690 2023-10-19T14:21:45.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:45.104+0000 7f0bb1407640 1 -- 172.21.15.57:0/425324557 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bac1d59f0 con 0x7f0bac141690 2023-10-19T14:21:45.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:45.104+0000 7f0ba37fe640 1 -- 172.21.15.57:0/425324557 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 144) v1 ==== 89377+0+0 (secure 0 0 0) 0x7f0b9c005410 con 0x7f0bac141690 2023-10-19T14:21:45.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:45.105+0000 7f0ba37fe640 1 --2- 172.21.15.57:0/425324557 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f0b8c06aff0 0x7f0b8c06d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:45.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:45.105+0000 7f0ba37fe640 1 -- 172.21.15.57:0/425324557 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(250..250 src has 1..250) v4 ==== 7867+0+0 (secure 0 0 0) 0x7f0b9c08a1c0 con 0x7f0bac141690 2023-10-19T14:21:45.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:45.105+0000 7f0baaffd640 1 --2- 172.21.15.57:0/425324557 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f0b8c06aff0 0x7f0b8c06d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:45.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:45.105+0000 7f0baaffd640 1 --2- 172.21.15.57:0/425324557 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f0b8c06aff0 0x7f0b8c06d4b0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0ba4036000 tx=0x7f0ba4004730 comp rx=0 tx=0).ready entity=mgr.11446 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:45.114 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:45.109+0000 7f0ba37fe640 1 -- 172.21.15.57:0/425324557 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0b9c054780 con 0x7f0bac141690 2023-10-19T14:21:45.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:45.321+0000 7f0bb1407640 1 -- 172.21.15.57:0/425324557 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f0bac0690e0 con 0x7f0bac141690 2023-10-19T14:21:46.014 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:46.011+0000 7f0ba37fe640 1 -- 172.21.15.57:0/425324557 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v109) v1 ==== 104+0+0 (secure 0 0 0) 0x7f0b9c057da0 con 0x7f0bac141690 2023-10-19T14:21:46.017 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:46.014+0000 7f0bb1407640 1 -- 172.21.15.57:0/425324557 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f0b8c06aff0 msgr2=0x7f0b8c06d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:46.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:46.014+0000 7f0bb1407640 1 --2- 172.21.15.57:0/425324557 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f0b8c06aff0 0x7f0b8c06d4b0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0ba4036000 tx=0x7f0ba4004730 comp rx=0 tx=0).stop 2023-10-19T14:21:46.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:46.014+0000 7f0bb1407640 1 -- 172.21.15.57:0/425324557 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0bac141690 msgr2=0x7f0bac0ca870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:46.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:46.014+0000 7f0bb1407640 1 --2- 172.21.15.57:0/425324557 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0bac141690 0x7f0bac0ca870 secure :-1 s=READY pgs=2784 cs=0 l=1 rev1=1 crypto rx=0x7f0b9c00c760 tx=0x7f0b9c00cc20 comp rx=0 tx=0).stop 2023-10-19T14:21:46.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:46.015+0000 7f0bb1407640 1 -- 172.21.15.57:0/425324557 shutdown_connections 2023-10-19T14:21:46.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:46.015+0000 7f0bb1407640 1 --2- 172.21.15.57:0/425324557 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f0b8c06aff0 0x7f0b8c06d4b0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:46.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:46.015+0000 7f0bb1407640 1 --2- 172.21.15.57:0/425324557 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0bac141690 0x7f0bac0ca870 unknown :-1 s=CLOSED pgs=2784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:46.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:46.015+0000 7f0bb1407640 1 --2- 172.21.15.57:0/425324557 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f0bac13e520 0x7f0bac1d8d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:46.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:46.015+0000 7f0bb1407640 1 --2- 172.21.15.57:0/425324557 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0bac06c0a0 0x7f0bac13ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:46.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:46.015+0000 7f0bb1407640 1 -- 172.21.15.57:0/425324557 >> 172.21.15.57:0/425324557 conn(0x7f0bac069930 msgr2=0x7f0bac06f080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:46.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:46.015+0000 7f0bb1407640 1 -- 172.21.15.57:0/425324557 shutdown_connections 2023-10-19T14:21:46.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:46.015+0000 7f0bb1407640 1 -- 172.21.15.57:0/425324557 wait complete. 2023-10-19T14:21:46.033 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-19T14:21:46.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:46.194+0000 7fe316eda640 1 -- 172.21.15.57:0/426320998 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe310156790 msgr2=0x7fe3100cc990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:46.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:46.194+0000 7fe316eda640 1 --2- 172.21.15.57:0/426320998 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe310156790 0x7fe3100cc990 secure :-1 s=READY pgs=2785 cs=0 l=1 rev1=1 crypto rx=0x7fe30c005fd0 tx=0x7fe30c02be10 comp rx=0 tx=0).stop 2023-10-19T14:21:46.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:46.194+0000 7fe316eda640 1 -- 172.21.15.57:0/426320998 shutdown_connections 2023-10-19T14:21:46.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:46.194+0000 7fe316eda640 1 --2- 172.21.15.57:0/426320998 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe31006a730 0x7fe31006abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:46.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:46.194+0000 7fe316eda640 1 --2- 172.21.15.57:0/426320998 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe310156790 0x7fe3100cc990 unknown :-1 s=CLOSED pgs=2785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:46.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:46.194+0000 7fe316eda640 1 -- 172.21.15.57:0/426320998 >> 172.21.15.57:0/426320998 conn(0x7fe31006fae0 msgr2=0x7fe31006fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:46.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:46.194+0000 7fe316eda640 1 -- 172.21.15.57:0/426320998 shutdown_connections 2023-10-19T14:21:46.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:46.194+0000 7fe316eda640 1 -- 172.21.15.57:0/426320998 wait complete. 2023-10-19T14:21:46.201 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:46.195+0000 7fe316eda640 1 Processor -- start 2023-10-19T14:21:46.201 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:46.195+0000 7fe316eda640 1 -- start start 2023-10-19T14:21:46.201 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:46.196+0000 7fe316eda640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe31006a730 0x7fe3100c9050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:46.201 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:46.196+0000 7fe316eda640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe3100ac870 0x7fe3100addb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:46.202 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:46.196+0000 7fe316eda640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe310156790 0x7fe3100aa8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:46.202 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:46.196+0000 7fe316eda640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3100c9590 con 0x7fe310156790 2023-10-19T14:21:46.202 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:46.196+0000 7fe316eda640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe31014bee0 con 0x7fe3100ac870 2023-10-19T14:21:46.202 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:46.196+0000 7fe316eda640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3100b0620 con 0x7fe31006a730 2023-10-19T14:21:46.202 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:46.196+0000 7fe315477640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe310156790 0x7fe3100aa8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:46.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:46.196+0000 7fe315477640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe310156790 0x7fe3100aa8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44212/0 (socket says 172.21.15.57:44212) 2023-10-19T14:21:46.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:46.196+0000 7fe315477640 1 -- 172.21.15.57:0/1113827161 learned_addr learned my addr 172.21.15.57:0/1113827161 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:46.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:46.196+0000 7fe314c76640 1 --2- 172.21.15.57:0/1113827161 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe31006a730 0x7fe3100c9050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:46.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:46.196+0000 7fe307fff640 1 --2- 172.21.15.57:0/1113827161 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe3100ac870 0x7fe3100addb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:46.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:46.197+0000 7fe315477640 1 -- 172.21.15.57:0/1113827161 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe31006a730 msgr2=0x7fe3100c9050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:46.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:46.197+0000 7fe315477640 1 --2- 172.21.15.57:0/1113827161 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe31006a730 0x7fe3100c9050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:46.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:46.197+0000 7fe315477640 1 -- 172.21.15.57:0/1113827161 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe3100ac870 msgr2=0x7fe3100addb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:46.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:46.197+0000 7fe315477640 1 --2- 172.21.15.57:0/1113827161 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe3100ac870 0x7fe3100addb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:46.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:46.197+0000 7fe315477640 1 -- 172.21.15.57:0/1113827161 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe30c009560 con 0x7fe310156790 2023-10-19T14:21:46.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:46.197+0000 7fe315477640 1 --2- 172.21.15.57:0/1113827161 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe310156790 0x7fe3100aa8c0 secure :-1 s=READY pgs=2786 cs=0 l=1 rev1=1 crypto rx=0x7fe30800e960 tx=0x7fe30800ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:46.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:46.197+0000 7fe3057fa640 1 -- 172.21.15.57:0/1113827161 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe308004d20 con 0x7fe310156790 2023-10-19T14:21:46.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:46.197+0000 7fe3057fa640 1 -- 172.21.15.57:0/1113827161 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fe30800cb20 con 0x7fe310156790 2023-10-19T14:21:46.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:46.197+0000 7fe316eda640 1 -- 172.21.15.57:0/1113827161 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3100c2f10 con 0x7fe310156790 2023-10-19T14:21:46.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:46.197+0000 7fe3057fa640 1 -- 172.21.15.57:0/1113827161 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe308005020 con 0x7fe310156790 2023-10-19T14:21:46.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:46.197+0000 7fe316eda640 1 -- 172.21.15.57:0/1113827161 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3100b08c0 con 0x7fe310156790 2023-10-19T14:21:46.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:46.198+0000 7fe316eda640 1 -- 172.21.15.57:0/1113827161 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3100b0ae0 con 0x7fe310156790 2023-10-19T14:21:46.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:46.199+0000 7fe3057fa640 1 -- 172.21.15.57:0/1113827161 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 145) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fe308005180 con 0x7fe310156790 2023-10-19T14:21:46.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:46.199+0000 7fe3057fa640 1 --2- 172.21.15.57:0/1113827161 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fe2fc06b080 0x7fe2fc06d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:46.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:46.199+0000 7fe3057fa640 1 -- 172.21.15.57:0/1113827161 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(250..250 src has 1..250) v4 ==== 7867+0+0 (secure 0 0 0) 0x7fe30808ac80 con 0x7fe310156790 2023-10-19T14:21:46.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:46.199+0000 7fe314c76640 1 --2- 172.21.15.57:0/1113827161 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fe2fc06b080 0x7fe2fc06d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:46.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:46.200+0000 7fe314c76640 1 --2- 172.21.15.57:0/1113827161 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fe2fc06b080 0x7fe2fc06d540 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fe30c035000 tx=0x7fe30c004730 comp rx=0 tx=0).ready entity=mgr.11446 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:46.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:46.203+0000 7fe3057fa640 1 -- 172.21.15.57:0/1113827161 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe308055100 con 0x7fe310156790 2023-10-19T14:21:46.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:46.379+0000 7fe316eda640 1 -- 172.21.15.57:0/1113827161 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe3100690e0 con 0x7fe310156790 2023-10-19T14:21:47.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:47.025+0000 7fe3057fa640 1 -- 172.21.15.57:0/1113827161 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v251) v1 ==== 181+0+0 (secure 0 0 0) 0x7fe30801f070 con 0x7fe310156790 2023-10-19T14:21:47.028 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'cephfs_metadata' removed 2023-10-19T14:21:47.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:47.028+0000 7fe316eda640 1 -- 172.21.15.57:0/1113827161 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fe2fc06b080 msgr2=0x7fe2fc06d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:47.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:47.028+0000 7fe316eda640 1 --2- 172.21.15.57:0/1113827161 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fe2fc06b080 0x7fe2fc06d540 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fe30c035000 tx=0x7fe30c004730 comp rx=0 tx=0).stop 2023-10-19T14:21:47.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:47.028+0000 7fe316eda640 1 -- 172.21.15.57:0/1113827161 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe310156790 msgr2=0x7fe3100aa8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:47.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:47.028+0000 7fe316eda640 1 --2- 172.21.15.57:0/1113827161 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe310156790 0x7fe3100aa8c0 secure :-1 s=READY pgs=2786 cs=0 l=1 rev1=1 crypto rx=0x7fe30800e960 tx=0x7fe30800ee20 comp rx=0 tx=0).stop 2023-10-19T14:21:47.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:47.028+0000 7fe316eda640 1 -- 172.21.15.57:0/1113827161 shutdown_connections 2023-10-19T14:21:47.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:47.028+0000 7fe316eda640 1 --2- 172.21.15.57:0/1113827161 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fe2fc06b080 0x7fe2fc06d540 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:47.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:47.028+0000 7fe316eda640 1 --2- 172.21.15.57:0/1113827161 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe310156790 0x7fe3100aa8c0 unknown :-1 s=CLOSED pgs=2786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:47.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:47.028+0000 7fe316eda640 1 --2- 172.21.15.57:0/1113827161 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe3100ac870 0x7fe3100addb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:47.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:47.028+0000 7fe316eda640 1 --2- 172.21.15.57:0/1113827161 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe31006a730 0x7fe3100c9050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:47.033 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:47.028+0000 7fe316eda640 1 -- 172.21.15.57:0/1113827161 >> 172.21.15.57:0/1113827161 conn(0x7fe31006fae0 msgr2=0x7fe3101581b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:47.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:47.029+0000 7fe316eda640 1 -- 172.21.15.57:0/1113827161 shutdown_connections 2023-10-19T14:21:47.034 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:47.029+0000 7fe316eda640 1 -- 172.21.15.57:0/1113827161 wait complete. 2023-10-19T14:21:47.048 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-19T14:21:47.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:47.163+0000 7fdc438e0640 1 -- 172.21.15.57:0/33186522 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdc3c157c00 msgr2=0x7fdc3c0c98b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:47.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:47.163+0000 7fdc438e0640 1 --2- 172.21.15.57:0/33186522 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdc3c157c00 0x7fdc3c0c98b0 secure :-1 s=READY pgs=2787 cs=0 l=1 rev1=1 crypto rx=0x7fdc3c071940 tx=0x7fdc3802bc80 comp rx=0 tx=0).stop 2023-10-19T14:21:47.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:47.163+0000 7fdc438e0640 1 -- 172.21.15.57:0/33186522 shutdown_connections 2023-10-19T14:21:47.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:47.163+0000 7fdc438e0640 1 --2- 172.21.15.57:0/33186522 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdc3c070a60 0x7fdc3c15b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:47.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:47.163+0000 7fdc438e0640 1 --2- 172.21.15.57:0/33186522 >> v2:172.21.15.179:3300/0 conn(0x7fdc3c0cb2f0 0x7fdc3c070520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:47.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:47.163+0000 7fdc438e0640 1 --2- 172.21.15.57:0/33186522 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdc3c157c00 0x7fdc3c0c98b0 unknown :-1 s=CLOSED pgs=2787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:47.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:47.163+0000 7fdc438e0640 1 -- 172.21.15.57:0/33186522 >> 172.21.15.57:0/33186522 conn(0x7fdc3c0aec40 msgr2=0x7fdc3c0af050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:47.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:47.163+0000 7fdc438e0640 1 -- 172.21.15.57:0/33186522 shutdown_connections 2023-10-19T14:21:47.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:47.163+0000 7fdc438e0640 1 -- 172.21.15.57:0/33186522 wait complete. 2023-10-19T14:21:47.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:47.165+0000 7fdc438e0640 1 Processor -- start 2023-10-19T14:21:47.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:47.165+0000 7fdc438e0640 1 -- start start 2023-10-19T14:21:47.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:47.165+0000 7fdc438e0640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdc3c070a60 0x7fdc3c153030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:47.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:47.165+0000 7fdc438e0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdc3c0cb2f0 0x7fdc3c14fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:47.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:47.165+0000 7fdc438e0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdc3c157c00 0x7fdc3c140640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:47.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:47.165+0000 7fdc438e0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc3c151460 con 0x7fdc3c157c00 2023-10-19T14:21:47.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:47.165+0000 7fdc438e0640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc3c150850 con 0x7fdc3c0cb2f0 2023-10-19T14:21:47.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:47.165+0000 7fdc438e0640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc3c150b30 con 0x7fdc3c070a60 2023-10-19T14:21:47.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:47.165+0000 7fdc41e7d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdc3c157c00 0x7fdc3c140640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:47.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:47.165+0000 7fdc41e7d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdc3c157c00 0x7fdc3c140640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33974/0 (socket says 172.21.15.57:33974) 2023-10-19T14:21:47.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:47.165+0000 7fdc41e7d640 1 -- 172.21.15.57:0/2393822853 learned_addr learned my addr 172.21.15.57:0/2393822853 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:47.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:47.165+0000 7fdc40e7b640 1 --2- 172.21.15.57:0/2393822853 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdc3c0cb2f0 0x7fdc3c14fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:47.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:47.165+0000 7fdc4167c640 1 --2- 172.21.15.57:0/2393822853 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdc3c070a60 0x7fdc3c153030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:47.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:47.166+0000 7fdc41e7d640 1 -- 172.21.15.57:0/2393822853 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdc3c070a60 msgr2=0x7fdc3c153030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:47.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:47.166+0000 7fdc41e7d640 1 --2- 172.21.15.57:0/2393822853 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdc3c070a60 0x7fdc3c153030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:47.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:47.166+0000 7fdc41e7d640 1 -- 172.21.15.57:0/2393822853 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdc3c0cb2f0 msgr2=0x7fdc3c14fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:47.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:47.166+0000 7fdc41e7d640 1 --2- 172.21.15.57:0/2393822853 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdc3c0cb2f0 0x7fdc3c14fa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:47.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:47.166+0000 7fdc41e7d640 1 -- 172.21.15.57:0/2393822853 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc38009560 con 0x7fdc3c157c00 2023-10-19T14:21:47.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:47.166+0000 7fdc41e7d640 1 --2- 172.21.15.57:0/2393822853 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdc3c157c00 0x7fdc3c140640 secure :-1 s=READY pgs=2788 cs=0 l=1 rev1=1 crypto rx=0x7fdc3400e980 tx=0x7fdc3400ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:47.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:47.166+0000 7fdc31ffb640 1 -- 172.21.15.57:0/2393822853 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc34004d20 con 0x7fdc3c157c00 2023-10-19T14:21:47.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:47.166+0000 7fdc31ffb640 1 -- 172.21.15.57:0/2393822853 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fdc3400cb20 con 0x7fdc3c157c00 2023-10-19T14:21:47.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:47.166+0000 7fdc438e0640 1 -- 172.21.15.57:0/2393822853 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc3c150e10 con 0x7fdc3c157c00 2023-10-19T14:21:47.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:47.166+0000 7fdc31ffb640 1 -- 172.21.15.57:0/2393822853 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc34005020 con 0x7fdc3c157c00 2023-10-19T14:21:47.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:47.166+0000 7fdc438e0640 1 -- 172.21.15.57:0/2393822853 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc3c153570 con 0x7fdc3c157c00 2023-10-19T14:21:47.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:47.167+0000 7fdc438e0640 1 -- 172.21.15.57:0/2393822853 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc3c151610 con 0x7fdc3c157c00 2023-10-19T14:21:47.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:47.168+0000 7fdc31ffb640 1 -- 172.21.15.57:0/2393822853 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 145) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fdc34005180 con 0x7fdc3c157c00 2023-10-19T14:21:47.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:47.168+0000 7fdc31ffb640 1 --2- 172.21.15.57:0/2393822853 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fdc2006b080 0x7fdc2006d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:47.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:47.168+0000 7fdc31ffb640 1 -- 172.21.15.57:0/2393822853 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(251..251 src has 1..251) v4 ==== 7429+0+0 (secure 0 0 0) 0x7fdc3408aa80 con 0x7fdc3c157c00 2023-10-19T14:21:47.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:47.168+0000 7fdc4167c640 1 --2- 172.21.15.57:0/2393822853 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fdc2006b080 0x7fdc2006d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:47.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:47.169+0000 7fdc4167c640 1 --2- 172.21.15.57:0/2393822853 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fdc2006b080 0x7fdc2006d540 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fdc38035000 tx=0x7fdc38004730 comp rx=0 tx=0).ready entity=mgr.11446 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:47.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:47.172+0000 7fdc31ffb640 1 -- 172.21.15.57:0/2393822853 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fdc3c151610 con 0x7fdc3c157c00 2023-10-19T14:21:47.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:47.348+0000 7fdc438e0640 1 -- 172.21.15.57:0/2393822853 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fdc3c0690e0 con 0x7fdc3c157c00 2023-10-19T14:21:48.032 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.029+0000 7fdc31ffb640 1 -- 172.21.15.57:0/2393822853 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v252) v1 ==== 169+0+0 (secure 0 0 0) 0x7fdc3401f0d0 con 0x7fdc3c157c00 2023-10-19T14:21:48.033 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'cephfs_data' removed 2023-10-19T14:21:48.036 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.033+0000 7fdc438e0640 1 -- 172.21.15.57:0/2393822853 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fdc2006b080 msgr2=0x7fdc2006d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:48.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.033+0000 7fdc438e0640 1 --2- 172.21.15.57:0/2393822853 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fdc2006b080 0x7fdc2006d540 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fdc38035000 tx=0x7fdc38004730 comp rx=0 tx=0).stop 2023-10-19T14:21:48.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.034+0000 7fdc438e0640 1 -- 172.21.15.57:0/2393822853 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdc3c157c00 msgr2=0x7fdc3c140640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:48.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.034+0000 7fdc438e0640 1 --2- 172.21.15.57:0/2393822853 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdc3c157c00 0x7fdc3c140640 secure :-1 s=READY pgs=2788 cs=0 l=1 rev1=1 crypto rx=0x7fdc3400e980 tx=0x7fdc3400ee40 comp rx=0 tx=0).stop 2023-10-19T14:21:48.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.037+0000 7fdc438e0640 1 -- 172.21.15.57:0/2393822853 shutdown_connections 2023-10-19T14:21:48.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.037+0000 7fdc438e0640 1 --2- 172.21.15.57:0/2393822853 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fdc2006b080 0x7fdc2006d540 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:48.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.037+0000 7fdc438e0640 1 --2- 172.21.15.57:0/2393822853 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdc3c157c00 0x7fdc3c140640 unknown :-1 s=CLOSED pgs=2788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:48.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.037+0000 7fdc438e0640 1 --2- 172.21.15.57:0/2393822853 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdc3c0cb2f0 0x7fdc3c14fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:48.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.037+0000 7fdc438e0640 1 --2- 172.21.15.57:0/2393822853 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdc3c070a60 0x7fdc3c153030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:48.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.037+0000 7fdc438e0640 1 -- 172.21.15.57:0/2393822853 >> 172.21.15.57:0/2393822853 conn(0x7fdc3c0aec40 msgr2=0x7fdc3c0b0040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:48.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.037+0000 7fdc438e0640 1 -- 172.21.15.57:0/2393822853 shutdown_connections 2023-10-19T14:21:48.042 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.038+0000 7fdc438e0640 1 -- 172.21.15.57:0/2393822853 wait complete. 2023-10-19T14:21:48.058 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T14:21:48.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.170+0000 7f74bf67c640 1 -- 172.21.15.57:0/4233540730 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f74b8140660 msgr2=0x7f74b80c97c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:48.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.170+0000 7f74bf67c640 1 --2- 172.21.15.57:0/4233540730 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f74b8140660 0x7f74b80c97c0 secure :-1 s=READY pgs=2789 cs=0 l=1 rev1=1 crypto rx=0x7f74b80cbd90 tx=0x7f74a802ca10 comp rx=0 tx=0).stop 2023-10-19T14:21:48.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.170+0000 7f74bf67c640 1 -- 172.21.15.57:0/4233540730 shutdown_connections 2023-10-19T14:21:48.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.170+0000 7f74bf67c640 1 --2- 172.21.15.57:0/4233540730 >> v2:172.21.15.179:3300/0 conn(0x7f74b80c9d00 0x7f74b80ca870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:48.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.170+0000 7f74bf67c640 1 --2- 172.21.15.57:0/4233540730 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f74b8140660 0x7f74b80c97c0 unknown :-1 s=CLOSED pgs=2789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:48.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.170+0000 7f74bf67c640 1 --2- 172.21.15.57:0/4233540730 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f74b813d360 0x7f74b815a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:48.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.170+0000 7f74bf67c640 1 -- 172.21.15.57:0/4233540730 >> 172.21.15.57:0/4233540730 conn(0x7f74b8069930 msgr2=0x7f74b8069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:48.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.170+0000 7f74bf67c640 1 -- 172.21.15.57:0/4233540730 shutdown_connections 2023-10-19T14:21:48.175 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.170+0000 7f74bf67c640 1 -- 172.21.15.57:0/4233540730 wait complete. 2023-10-19T14:21:48.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.171+0000 7f74bf67c640 1 Processor -- start 2023-10-19T14:21:48.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.171+0000 7f74bf67c640 1 -- start start 2023-10-19T14:21:48.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.172+0000 7f74bf67c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f74b80c9d00 0x7f74b8139ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:48.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.172+0000 7f74bf67c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f74b813d360 0x7f74b8136f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:48.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.172+0000 7f74bf67c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f74b8140660 0x7f74b8157470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:48.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.172+0000 7f74bf67c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74b813a530 con 0x7f74b80c9d00 2023-10-19T14:21:48.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.172+0000 7f74bf67c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74b8139550 con 0x7f74b8140660 2023-10-19T14:21:48.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.172+0000 7f74bf67c640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74b81374a0 con 0x7f74b813d360 2023-10-19T14:21:48.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.172+0000 7f74bd418640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f74b80c9d00 0x7f74b8139ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:48.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.172+0000 7f74bd418640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f74b80c9d00 0x7f74b8139ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34002/0 (socket says 172.21.15.57:34002) 2023-10-19T14:21:48.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.172+0000 7f74bd418640 1 -- 172.21.15.57:0/585411764 learned_addr learned my addr 172.21.15.57:0/585411764 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:48.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.172+0000 7f74bcc17640 1 --2- 172.21.15.57:0/585411764 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f74b813d360 0x7f74b8136f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:48.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.172+0000 7f74bdc19640 1 --2- 172.21.15.57:0/585411764 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f74b8140660 0x7f74b8157470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:48.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.172+0000 7f74bd418640 1 -- 172.21.15.57:0/585411764 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f74b813d360 msgr2=0x7f74b8136f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:48.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.172+0000 7f74bd418640 1 --2- 172.21.15.57:0/585411764 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f74b813d360 0x7f74b8136f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:48.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.172+0000 7f74bd418640 1 -- 172.21.15.57:0/585411764 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f74b8140660 msgr2=0x7f74b8157470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:48.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.172+0000 7f74bd418640 1 --2- 172.21.15.57:0/585411764 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f74b8140660 0x7f74b8157470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:48.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.172+0000 7f74bd418640 1 -- 172.21.15.57:0/585411764 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74a80071c0 con 0x7f74b80c9d00 2023-10-19T14:21:48.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.172+0000 7f74bcc17640 1 --2- 172.21.15.57:0/585411764 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f74b813d360 0x7f74b8136f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:21:48.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.172+0000 7f74bdc19640 1 --2- 172.21.15.57:0/585411764 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f74b8140660 0x7f74b8157470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:21:48.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.173+0000 7f74bd418640 1 --2- 172.21.15.57:0/585411764 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f74b80c9d00 0x7f74b8139ff0 secure :-1 s=READY pgs=2790 cs=0 l=1 rev1=1 crypto rx=0x7f74b400c970 tx=0x7f74b400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:48.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.173+0000 7f74adffb640 1 -- 172.21.15.57:0/585411764 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74b4007bb0 con 0x7f74b80c9d00 2023-10-19T14:21:48.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.173+0000 7f74adffb640 1 -- 172.21.15.57:0/585411764 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f74b4007e20 con 0x7f74b80c9d00 2023-10-19T14:21:48.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.173+0000 7f74bf67c640 1 -- 172.21.15.57:0/585411764 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74b8157f30 con 0x7f74b80c9d00 2023-10-19T14:21:48.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.173+0000 7f74adffb640 1 -- 172.21.15.57:0/585411764 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74b4005280 con 0x7f74b80c9d00 2023-10-19T14:21:48.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.173+0000 7f74bf67c640 1 -- 172.21.15.57:0/585411764 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74b81397f0 con 0x7f74b80c9d00 2023-10-19T14:21:48.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.174+0000 7f74bf67c640 1 -- 172.21.15.57:0/585411764 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74b80690e0 con 0x7f74b80c9d00 2023-10-19T14:21:48.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.174+0000 7f74adffb640 1 -- 172.21.15.57:0/585411764 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 145) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f74b40053e0 con 0x7f74b80c9d00 2023-10-19T14:21:48.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.175+0000 7f74adffb640 1 --2- 172.21.15.57:0/585411764 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f749c06b080 0x7f749c06d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:48.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.175+0000 7f74adffb640 1 -- 172.21.15.57:0/585411764 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 7015+0+0 (secure 0 0 0) 0x7f74b408a6f0 con 0x7f74b80c9d00 2023-10-19T14:21:48.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.175+0000 7f74bcc17640 1 --2- 172.21.15.57:0/585411764 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f749c06b080 0x7f749c06d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:48.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.176+0000 7f74bcc17640 1 --2- 172.21.15.57:0/585411764 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f749c06b080 0x7f749c06d540 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f74a8035000 tx=0x7f74a8004730 comp rx=0 tx=0).ready entity=mgr.11446 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:48.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.179+0000 7f74adffb640 1 -- 172.21.15.57:0/585411764 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f74b4054d90 con 0x7f74b80c9d00 2023-10-19T14:21:48.361 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.359+0000 7f74bf67c640 1 -- 172.21.15.57:0/585411764 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f74b815b940 con 0x7f74b80c9d00 2023-10-19T14:21:48.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.360+0000 7f74adffb640 1 -- 172.21.15.57:0/585411764 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v252) v1 ==== 74+0+18176 (secure 0 0 0) 0x7f74b40583b0 con 0x7f74b80c9d00 2023-10-19T14:21:48.363 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:21:48.363 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":252,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T14:21:48.026331+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_metadata","create_time":"2023-10-19T14:21:24.384693+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"246","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"backup_fs_data","create_time":"2023-10-19T14:21:25.717524+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"247","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.57:6834/18187246":"2023-10-20T14:21:44.967770+0000","172.21.15.179:0/3646431735":"2023-10-20T14:21:41.395725+0000","172.21.15.179:0/2986772961":"2023-10-20T14:21:41.395725+0000","172.21.15.179:0/837564123":"2023-10-20T14:21:41.395725+0000","172.21.15.179:0/1917522846":"2023-10-20T14:21:41.395725+0000","172.21.15.179:0/3420876591":"2023-10-20T14:21:41.395725+0000","172.21.15.179:6833/609071913":"2023-10-20T14:21:41.395725+0000","172.21.15.179:6832/609071913":"2023-10-20T14:21:41.395725+0000","172.21.15.179:0/2525508070":"2023-10-20T14:21:35.927420+0000","172.21.15.179:0/75322738":"2023-10-20T14:21:41.395725+0000","172.21.15.179:0/720769249":"2023-10-20T14:21:35.927420+0000","172.21.15.57:6835/18187246":"2023-10-20T14:21:44.967770+0000","172.21.15.179:6833/3547895240":"2023-10-20T14:21:35.927420+0000","172.21.15.179:6832/3547895240":"2023-10-20T14:21:35.927420+0000","172.21.15.179:0/177563067":"2023-10-20T14:21:35.927420+0000","172.21.15.179:0/1656904674":"2023-10-20T14:21:41.395725+0000","172.21.15.179:0/647327435":"2023-10-20T14:21:41.395725+0000","172.21.15.179:0/1877726906":"2023-10-20T14:21:35.927420+0000","172.21.15.179:0/3471744208":"2023-10-20T14:21:35.927420+0000","172.21.15.179:0/2477290155":"2023-10-20T14:21:35.927420+0000","172.21.15.179:0/1240052109":"2023-10-20T14:21:35.927420+0000","172.21.15.179:0/4042925688":"2023-10-20T14:21:35.927420+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T14:21:48.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.362+0000 7f74bf67c640 1 -- 172.21.15.57:0/585411764 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f749c06b080 msgr2=0x7f749c06d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:48.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.362+0000 7f74bf67c640 1 --2- 172.21.15.57:0/585411764 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f749c06b080 0x7f749c06d540 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f74a8035000 tx=0x7f74a8004730 comp rx=0 tx=0).stop 2023-10-19T14:21:48.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.362+0000 7f74bf67c640 1 -- 172.21.15.57:0/585411764 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f74b80c9d00 msgr2=0x7f74b8139ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:48.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.362+0000 7f74bf67c640 1 --2- 172.21.15.57:0/585411764 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f74b80c9d00 0x7f74b8139ff0 secure :-1 s=READY pgs=2790 cs=0 l=1 rev1=1 crypto rx=0x7f74b400c970 tx=0x7f74b400ce30 comp rx=0 tx=0).stop 2023-10-19T14:21:48.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.362+0000 7f74bf67c640 1 -- 172.21.15.57:0/585411764 shutdown_connections 2023-10-19T14:21:48.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.362+0000 7f74bf67c640 1 --2- 172.21.15.57:0/585411764 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f749c06b080 0x7f749c06d540 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:48.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.362+0000 7f74bf67c640 1 --2- 172.21.15.57:0/585411764 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f74b8140660 0x7f74b8157470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:48.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.362+0000 7f74bf67c640 1 --2- 172.21.15.57:0/585411764 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f74b813d360 0x7f74b8136f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:48.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.362+0000 7f74bf67c640 1 --2- 172.21.15.57:0/585411764 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f74b80c9d00 0x7f74b8139ff0 unknown :-1 s=CLOSED pgs=2790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:48.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.362+0000 7f74bf67c640 1 -- 172.21.15.57:0/585411764 >> 172.21.15.57:0/585411764 conn(0x7f74b8069930 msgr2=0x7f74b8069d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:48.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.362+0000 7f74bf67c640 1 -- 172.21.15.57:0/585411764 shutdown_connections 2023-10-19T14:21:48.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.362+0000 7f74bf67c640 1 -- 172.21.15.57:0/585411764 wait complete. 2023-10-19T14:21:48.376 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-19T12:44:07.371186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 33, 'pool_name': 'backup_fs_metadata', 'create_time': '2023-10-19T14:21:24.384693+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '246', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 34, 'pool_name': 'backup_fs_data', 'create_time': '2023-10-19T14:21:25.717524+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '247', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-19T14:21:48.377 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-19T14:21:48.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.486+0000 7f6aa591d640 1 -- 172.21.15.57:0/1678939800 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6aa006ffb0 msgr2=0x7f6aa00c96b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:48.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.486+0000 7f6aa591d640 1 --2- 172.21.15.57:0/1678939800 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6aa006ffb0 0x7f6aa00c96b0 secure :-1 s=READY pgs=2791 cs=0 l=1 rev1=1 crypto rx=0x7f6aa00ca460 tx=0x7f6a9002bda0 comp rx=0 tx=0).stop 2023-10-19T14:21:48.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.486+0000 7f6aa591d640 1 -- 172.21.15.57:0/1678939800 shutdown_connections 2023-10-19T14:21:48.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.486+0000 7f6aa591d640 1 --2- 172.21.15.57:0/1678939800 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6aa00c9bf0 0x7f6aa0154af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:48.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.486+0000 7f6aa591d640 1 --2- 172.21.15.57:0/1678939800 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6aa006ffb0 0x7f6aa00c96b0 unknown :-1 s=CLOSED pgs=2791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:48.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.486+0000 7f6aa591d640 1 --2- 172.21.15.57:0/1678939800 >> v2:172.21.15.179:3300/0 conn(0x7f6aa00b1920 0x7f6aa006fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:48.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.486+0000 7f6aa591d640 1 -- 172.21.15.57:0/1678939800 >> 172.21.15.57:0/1678939800 conn(0x7f6aa0069930 msgr2=0x7f6aa0069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:48.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.486+0000 7f6aa591d640 1 -- 172.21.15.57:0/1678939800 shutdown_connections 2023-10-19T14:21:48.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.486+0000 7f6aa591d640 1 -- 172.21.15.57:0/1678939800 wait complete. 2023-10-19T14:21:48.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.487+0000 7f6aa591d640 1 Processor -- start 2023-10-19T14:21:48.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.488+0000 7f6aa591d640 1 -- start start 2023-10-19T14:21:48.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.488+0000 7f6aa591d640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6aa006ffb0 0x7f6aa015c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:48.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.488+0000 7f6aa591d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6aa00b1920 0x7f6aa00ac460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:48.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.488+0000 7f6aa591d640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6aa00c9bf0 0x7f6aa0148530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:48.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.488+0000 7f6aa591d640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6aa00ac9a0 con 0x7f6aa00b1920 2023-10-19T14:21:48.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.488+0000 7f6aa591d640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6aa00acc20 con 0x7f6aa00c9bf0 2023-10-19T14:21:48.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.488+0000 7f6aa591d640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6aa00acf00 con 0x7f6aa006ffb0 2023-10-19T14:21:48.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.489+0000 7f6a9e7fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6aa00b1920 0x7f6aa00ac460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:48.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.489+0000 7f6a9e7fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6aa00b1920 0x7f6aa00ac460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34014/0 (socket says 172.21.15.57:34014) 2023-10-19T14:21:48.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.489+0000 7f6a9e7fc640 1 -- 172.21.15.57:0/2032786842 learned_addr learned my addr 172.21.15.57:0/2032786842 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:48.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.489+0000 7f6a9effd640 1 --2- 172.21.15.57:0/2032786842 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6aa006ffb0 0x7f6aa015c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:48.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.489+0000 7f6a9f7fe640 1 --2- 172.21.15.57:0/2032786842 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6aa00c9bf0 0x7f6aa0148530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:48.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.489+0000 7f6a9e7fc640 1 -- 172.21.15.57:0/2032786842 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6aa006ffb0 msgr2=0x7f6aa015c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:48.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.489+0000 7f6a9e7fc640 1 --2- 172.21.15.57:0/2032786842 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6aa006ffb0 0x7f6aa015c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:48.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.489+0000 7f6a9e7fc640 1 -- 172.21.15.57:0/2032786842 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6aa00c9bf0 msgr2=0x7f6aa0148530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:48.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.489+0000 7f6a9e7fc640 1 --2- 172.21.15.57:0/2032786842 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6aa00c9bf0 0x7f6aa0148530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:48.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.489+0000 7f6a9e7fc640 1 -- 172.21.15.57:0/2032786842 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a90009560 con 0x7f6aa00b1920 2023-10-19T14:21:48.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.489+0000 7f6a9e7fc640 1 --2- 172.21.15.57:0/2032786842 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6aa00b1920 0x7f6aa00ac460 secure :-1 s=READY pgs=2792 cs=0 l=1 rev1=1 crypto rx=0x7f6a90009850 tx=0x7f6a9002da70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:48.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.489+0000 7f6a87fff640 1 -- 172.21.15.57:0/2032786842 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a90003800 con 0x7f6aa00b1920 2023-10-19T14:21:48.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.490+0000 7f6aa591d640 1 -- 172.21.15.57:0/2032786842 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6aa00ad1e0 con 0x7f6aa00b1920 2023-10-19T14:21:48.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.490+0000 7f6a87fff640 1 -- 172.21.15.57:0/2032786842 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f6a90003a50 con 0x7f6aa00b1920 2023-10-19T14:21:48.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.490+0000 7f6a87fff640 1 -- 172.21.15.57:0/2032786842 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a90003e20 con 0x7f6aa00b1920 2023-10-19T14:21:48.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.490+0000 7f6aa591d640 1 -- 172.21.15.57:0/2032786842 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6aa014fb30 con 0x7f6aa00b1920 2023-10-19T14:21:48.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.491+0000 7f6aa591d640 1 -- 172.21.15.57:0/2032786842 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6aa0070dc0 con 0x7f6aa00b1920 2023-10-19T14:21:48.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.493+0000 7f6a87fff640 1 -- 172.21.15.57:0/2032786842 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 145) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f6a9003e450 con 0x7f6aa00b1920 2023-10-19T14:21:48.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.494+0000 7f6a87fff640 1 --2- 172.21.15.57:0/2032786842 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f6a8c06b080 0x7f6a8c06d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:48.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.494+0000 7f6a87fff640 1 -- 172.21.15.57:0/2032786842 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 7015+0+0 (secure 0 0 0) 0x7f6a900acab0 con 0x7f6aa00b1920 2023-10-19T14:21:48.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.495+0000 7f6a9effd640 1 --2- 172.21.15.57:0/2032786842 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f6a8c06b080 0x7f6a8c06d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:48.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.498+0000 7f6a9effd640 1 --2- 172.21.15.57:0/2032786842 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f6a8c06b080 0x7f6a8c06d540 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6a94009b50 tx=0x7f6a94005f90 comp rx=0 tx=0).ready entity=mgr.11446 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:48.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.500+0000 7f6a87fff640 1 -- 172.21.15.57:0/2032786842 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6a900771e0 con 0x7f6aa00b1920 2023-10-19T14:21:48.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.681+0000 7f6aa591d640 1 -- 172.21.15.57:0/2032786842 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6aa014fd20 con 0x7f6aa00b1920 2023-10-19T14:21:48.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.681+0000 7f6a87fff640 1 -- 172.21.15.57:0/2032786842 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v252) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6a90042020 con 0x7f6aa00b1920 2023-10-19T14:21:48.685 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-19T14:21:48.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.684+0000 7f6aa591d640 1 -- 172.21.15.57:0/2032786842 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f6a8c06b080 msgr2=0x7f6a8c06d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:48.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.684+0000 7f6aa591d640 1 --2- 172.21.15.57:0/2032786842 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f6a8c06b080 0x7f6a8c06d540 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6a94009b50 tx=0x7f6a94005f90 comp rx=0 tx=0).stop 2023-10-19T14:21:48.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.684+0000 7f6aa591d640 1 -- 172.21.15.57:0/2032786842 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6aa00b1920 msgr2=0x7f6aa00ac460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:48.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.684+0000 7f6aa591d640 1 --2- 172.21.15.57:0/2032786842 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6aa00b1920 0x7f6aa00ac460 secure :-1 s=READY pgs=2792 cs=0 l=1 rev1=1 crypto rx=0x7f6a90009850 tx=0x7f6a9002da70 comp rx=0 tx=0).stop 2023-10-19T14:21:48.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.684+0000 7f6aa591d640 1 -- 172.21.15.57:0/2032786842 shutdown_connections 2023-10-19T14:21:48.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.684+0000 7f6aa591d640 1 --2- 172.21.15.57:0/2032786842 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f6a8c06b080 0x7f6a8c06d540 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:48.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.684+0000 7f6aa591d640 1 --2- 172.21.15.57:0/2032786842 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6aa00c9bf0 0x7f6aa0148530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:48.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.684+0000 7f6aa591d640 1 --2- 172.21.15.57:0/2032786842 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6aa00b1920 0x7f6aa00ac460 unknown :-1 s=CLOSED pgs=2792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:48.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.684+0000 7f6aa591d640 1 --2- 172.21.15.57:0/2032786842 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6aa006ffb0 0x7f6aa015c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:48.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.684+0000 7f6aa591d640 1 -- 172.21.15.57:0/2032786842 >> 172.21.15.57:0/2032786842 conn(0x7f6aa0069930 msgr2=0x7f6aa00cc1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:48.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.684+0000 7f6aa591d640 1 -- 172.21.15.57:0/2032786842 shutdown_connections 2023-10-19T14:21:48.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.684+0000 7f6aa591d640 1 -- 172.21.15.57:0/2032786842 wait complete. 2023-10-19T14:21:48.697 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2023-10-19T14:21:48.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.806+0000 7f5d50f25640 1 -- 172.21.15.57:0/3772882731 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5d4c0ceb40 msgr2=0x7f5d4c0b0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:48.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.806+0000 7f5d50f25640 1 --2- 172.21.15.57:0/3772882731 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5d4c0ceb40 0x7f5d4c0b0650 secure :-1 s=READY pgs=2793 cs=0 l=1 rev1=1 crypto rx=0x7f5d44005fd0 tx=0x7f5d4402c1b0 comp rx=0 tx=0).stop 2023-10-19T14:21:48.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.806+0000 7f5d50f25640 1 -- 172.21.15.57:0/3772882731 shutdown_connections 2023-10-19T14:21:48.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.806+0000 7f5d50f25640 1 --2- 172.21.15.57:0/3772882731 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5d4c0ceb40 0x7f5d4c0b0650 unknown :-1 s=CLOSED pgs=2793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:48.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.806+0000 7f5d50f25640 1 --2- 172.21.15.57:0/3772882731 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5d4c0cf340 0x7f5d4c0d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:48.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.806+0000 7f5d50f25640 1 -- 172.21.15.57:0/3772882731 >> 172.21.15.57:0/3772882731 conn(0x7f5d4c0d0d60 msgr2=0x7f5d4c0d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:48.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.806+0000 7f5d50f25640 1 -- 172.21.15.57:0/3772882731 shutdown_connections 2023-10-19T14:21:48.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.806+0000 7f5d50f25640 1 -- 172.21.15.57:0/3772882731 wait complete. 2023-10-19T14:21:48.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.808+0000 7f5d50f25640 1 Processor -- start 2023-10-19T14:21:48.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.808+0000 7f5d50f25640 1 -- start start 2023-10-19T14:21:48.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.808+0000 7f5d50f25640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5d4c0ceb40 0x7f5d4c0b4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:48.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.808+0000 7f5d50f25640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5d4c0cf340 0x7f5d4c157e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:48.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.808+0000 7f5d50f25640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5d4c15b2a0 0x7f5d4c15e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:48.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.808+0000 7f5d50f25640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d4c15e590 con 0x7f5d4c0cf340 2023-10-19T14:21:48.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.808+0000 7f5d50f25640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d4c09aad0 con 0x7f5d4c0ceb40 2023-10-19T14:21:48.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.808+0000 7f5d50f25640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d4c0b5550 con 0x7f5d4c15b2a0 2023-10-19T14:21:48.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.809+0000 7f5d43fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5d4c0cf340 0x7f5d4c157e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:48.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.809+0000 7f5d43fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5d4c0cf340 0x7f5d4c157e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34038/0 (socket says 172.21.15.57:34038) 2023-10-19T14:21:48.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.809+0000 7f5d43fff640 1 -- 172.21.15.57:0/2635503698 learned_addr learned my addr 172.21.15.57:0/2635503698 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:48.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.809+0000 7f5d4ad9d640 1 --2- 172.21.15.57:0/2635503698 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5d4c15b2a0 0x7f5d4c15e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:48.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.809+0000 7f5d4a59c640 1 --2- 172.21.15.57:0/2635503698 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5d4c0ceb40 0x7f5d4c0b4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:48.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.809+0000 7f5d43fff640 1 -- 172.21.15.57:0/2635503698 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5d4c15b2a0 msgr2=0x7f5d4c15e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:48.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.809+0000 7f5d43fff640 1 --2- 172.21.15.57:0/2635503698 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5d4c15b2a0 0x7f5d4c15e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:48.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.809+0000 7f5d43fff640 1 -- 172.21.15.57:0/2635503698 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5d4c0ceb40 msgr2=0x7f5d4c0b4360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:48.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.809+0000 7f5d43fff640 1 --2- 172.21.15.57:0/2635503698 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5d4c0ceb40 0x7f5d4c0b4360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:48.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.809+0000 7f5d43fff640 1 -- 172.21.15.57:0/2635503698 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d44009900 con 0x7f5d4c0cf340 2023-10-19T14:21:48.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.809+0000 7f5d4a59c640 1 --2- 172.21.15.57:0/2635503698 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5d4c0ceb40 0x7f5d4c0b4360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:21:48.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.809+0000 7f5d43fff640 1 --2- 172.21.15.57:0/2635503698 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5d4c0cf340 0x7f5d4c157e20 secure :-1 s=READY pgs=2794 cs=0 l=1 rev1=1 crypto rx=0x7f5d3400b3c0 tx=0x7f5d3400b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:48.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.809+0000 7f5d42ffd640 1 -- 172.21.15.57:0/2635503698 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d34011830 con 0x7f5d4c0cf340 2023-10-19T14:21:48.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.809+0000 7f5d42ffd640 1 -- 172.21.15.57:0/2635503698 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f5d34011a80 con 0x7f5d4c0cf340 2023-10-19T14:21:48.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.809+0000 7f5d50f25640 1 -- 172.21.15.57:0/2635503698 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d4c0b4940 con 0x7f5d4c0cf340 2023-10-19T14:21:48.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.809+0000 7f5d42ffd640 1 -- 172.21.15.57:0/2635503698 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d34010380 con 0x7f5d4c0cf340 2023-10-19T14:21:48.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.809+0000 7f5d50f25640 1 -- 172.21.15.57:0/2635503698 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d4c0b4f00 con 0x7f5d4c0cf340 2023-10-19T14:21:48.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.810+0000 7f5d50f25640 1 -- 172.21.15.57:0/2635503698 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d4c0b5700 con 0x7f5d4c0cf340 2023-10-19T14:21:48.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.811+0000 7f5d42ffd640 1 -- 172.21.15.57:0/2635503698 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 145) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f5d340104e0 con 0x7f5d4c0cf340 2023-10-19T14:21:48.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.811+0000 7f5d42ffd640 1 --2- 172.21.15.57:0/2635503698 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f5d2c06b080 0x7f5d2c06d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:48.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.811+0000 7f5d42ffd640 1 -- 172.21.15.57:0/2635503698 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 7015+0+0 (secure 0 0 0) 0x7f5d34089cd0 con 0x7f5d4c0cf340 2023-10-19T14:21:48.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.811+0000 7f5d4a59c640 1 --2- 172.21.15.57:0/2635503698 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f5d2c06b080 0x7f5d2c06d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:48.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.812+0000 7f5d4a59c640 1 --2- 172.21.15.57:0/2635503698 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f5d2c06b080 0x7f5d2c06d540 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5d3c0058c0 tx=0x7f5d3c005850 comp rx=0 tx=0).ready entity=mgr.11446 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:48.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.815+0000 7f5d42ffd640 1 -- 172.21.15.57:0/2635503698 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f5d4c0b5700 con 0x7f5d4c0cf340 2023-10-19T14:21:48.993 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.990+0000 7f5d50f25640 1 -- 172.21.15.57:0/2635503698 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5d4c0cba00 con 0x7f5d4c0cf340 2023-10-19T14:21:48.993 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.991+0000 7f5d42ffd640 1 -- 172.21.15.57:0/2635503698 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v252) v1 ==== 107+0+11 (secure 0 0 0) 0x7f5d34054500 con 0x7f5d4c0cf340 2023-10-19T14:21:48.994 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2023-10-19T14:21:48.995 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.992+0000 7f5d50f25640 1 -- 172.21.15.57:0/2635503698 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f5d2c06b080 msgr2=0x7f5d2c06d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:48.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.992+0000 7f5d50f25640 1 --2- 172.21.15.57:0/2635503698 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f5d2c06b080 0x7f5d2c06d540 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5d3c0058c0 tx=0x7f5d3c005850 comp rx=0 tx=0).stop 2023-10-19T14:21:48.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.992+0000 7f5d50f25640 1 -- 172.21.15.57:0/2635503698 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5d4c0cf340 msgr2=0x7f5d4c157e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:48.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.992+0000 7f5d50f25640 1 --2- 172.21.15.57:0/2635503698 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5d4c0cf340 0x7f5d4c157e20 secure :-1 s=READY pgs=2794 cs=0 l=1 rev1=1 crypto rx=0x7f5d3400b3c0 tx=0x7f5d3400b880 comp rx=0 tx=0).stop 2023-10-19T14:21:48.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.992+0000 7f5d50f25640 1 -- 172.21.15.57:0/2635503698 shutdown_connections 2023-10-19T14:21:48.996 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.992+0000 7f5d50f25640 1 --2- 172.21.15.57:0/2635503698 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f5d2c06b080 0x7f5d2c06d540 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:48.997 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.992+0000 7f5d50f25640 1 --2- 172.21.15.57:0/2635503698 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5d4c15b2a0 0x7f5d4c15e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:48.997 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.992+0000 7f5d50f25640 1 --2- 172.21.15.57:0/2635503698 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5d4c0cf340 0x7f5d4c157e20 unknown :-1 s=CLOSED pgs=2794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:48.997 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.992+0000 7f5d50f25640 1 --2- 172.21.15.57:0/2635503698 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5d4c0ceb40 0x7f5d4c0b4360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:48.997 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.992+0000 7f5d50f25640 1 -- 172.21.15.57:0/2635503698 >> 172.21.15.57:0/2635503698 conn(0x7f5d4c0d0d60 msgr2=0x7f5d4c06a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:48.997 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.992+0000 7f5d50f25640 1 -- 172.21.15.57:0/2635503698 shutdown_connections 2023-10-19T14:21:48.998 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:48.992+0000 7f5d50f25640 1 -- 172.21.15.57:0/2635503698 wait complete. 2023-10-19T14:21:49.006 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2023-10-19T14:21:49.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.116+0000 7f7420173640 1 -- 172.21.15.57:0/176538006 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f74180ab500 msgr2=0x7f74180ab930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:49.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.116+0000 7f7420173640 1 --2- 172.21.15.57:0/176538006 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f74180ab500 0x7f74180ab930 secure :-1 s=READY pgs=2795 cs=0 l=1 rev1=1 crypto rx=0x7f7418071310 tx=0x7f740802c250 comp rx=0 tx=0).stop 2023-10-19T14:21:49.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.116+0000 7f7420173640 1 -- 172.21.15.57:0/176538006 shutdown_connections 2023-10-19T14:21:49.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.116+0000 7f7420173640 1 --2- 172.21.15.57:0/176538006 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f74180ab500 0x7f74180ab930 unknown :-1 s=CLOSED pgs=2795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:49.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.116+0000 7f7420173640 1 --2- 172.21.15.57:0/176538006 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f74180c9dc0 0x7f74181531d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:49.121 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.116+0000 7f7420173640 1 --2- 172.21.15.57:0/176538006 >> v2:172.21.15.179:3300/0 conn(0x7f74180c99e0 0x7f7418152c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:49.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.116+0000 7f7420173640 1 -- 172.21.15.57:0/176538006 >> 172.21.15.57:0/176538006 conn(0x7f7418069930 msgr2=0x7f7418069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:49.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.116+0000 7f7420173640 1 -- 172.21.15.57:0/176538006 shutdown_connections 2023-10-19T14:21:49.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.116+0000 7f7420173640 1 -- 172.21.15.57:0/176538006 wait complete. 2023-10-19T14:21:49.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.117+0000 7f7420173640 1 Processor -- start 2023-10-19T14:21:49.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.117+0000 7f7420173640 1 -- start start 2023-10-19T14:21:49.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.118+0000 7f7420173640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f74180ab500 0x7f74181394d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:49.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.118+0000 7f7420173640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f74180c99e0 0x7f74181f2220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:49.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.118+0000 7f7420173640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f74180c9dc0 0x7f74180aa8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:49.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.118+0000 7f7420173640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7418139a10 con 0x7f74180ab500 2023-10-19T14:21:49.129 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.118+0000 7f7420173640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74181f2760 con 0x7f74180c9dc0 2023-10-19T14:21:49.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.118+0000 7f7420173640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f741806f200 con 0x7f74180c99e0 2023-10-19T14:21:49.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.118+0000 7f741df0f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f74180ab500 0x7f74181394d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:49.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.118+0000 7f741df0f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f74180ab500 0x7f74181394d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34064/0 (socket says 172.21.15.57:34064) 2023-10-19T14:21:49.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.118+0000 7f741df0f640 1 -- 172.21.15.57:0/2989486365 learned_addr learned my addr 172.21.15.57:0/2989486365 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:49.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.118+0000 7f741d70e640 1 --2- 172.21.15.57:0/2989486365 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f74180c99e0 0x7f74181f2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:49.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.118+0000 7f741e710640 1 --2- 172.21.15.57:0/2989486365 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f74180c9dc0 0x7f74180aa8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:49.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.119+0000 7f741df0f640 1 -- 172.21.15.57:0/2989486365 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f74180c99e0 msgr2=0x7f74181f2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:49.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.119+0000 7f741df0f640 1 --2- 172.21.15.57:0/2989486365 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f74180c99e0 0x7f74181f2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:49.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.119+0000 7f741df0f640 1 -- 172.21.15.57:0/2989486365 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f74180c9dc0 msgr2=0x7f74180aa8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:49.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.119+0000 7f741df0f640 1 --2- 172.21.15.57:0/2989486365 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f74180c9dc0 0x7f74180aa8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:49.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.119+0000 7f741df0f640 1 -- 172.21.15.57:0/2989486365 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7408009900 con 0x7f74180ab500 2023-10-19T14:21:49.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.119+0000 7f741df0f640 1 --2- 172.21.15.57:0/2989486365 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f74180ab500 0x7f74181394d0 secure :-1 s=READY pgs=2796 cs=0 l=1 rev1=1 crypto rx=0x7f741000c890 tx=0x7f741000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:49.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.119+0000 7f74167fc640 1 -- 172.21.15.57:0/2989486365 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7410004220 con 0x7f74180ab500 2023-10-19T14:21:49.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.119+0000 7f74167fc640 1 -- 172.21.15.57:0/2989486365 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f7410004470 con 0x7f74180ab500 2023-10-19T14:21:49.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.119+0000 7f7420173640 1 -- 172.21.15.57:0/2989486365 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74180abff0 con 0x7f74180ab500 2023-10-19T14:21:49.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.119+0000 7f74167fc640 1 -- 172.21.15.57:0/2989486365 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7410007c10 con 0x7f74180ab500 2023-10-19T14:21:49.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.119+0000 7f7420173640 1 -- 172.21.15.57:0/2989486365 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f741806f7c0 con 0x7f74180ab500 2023-10-19T14:21:49.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.120+0000 7f7420173640 1 -- 172.21.15.57:0/2989486365 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7418067930 con 0x7f74180ab500 2023-10-19T14:21:49.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.120+0000 7f74167fc640 1 -- 172.21.15.57:0/2989486365 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 145) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f7410005020 con 0x7f74180ab500 2023-10-19T14:21:49.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.121+0000 7f74167fc640 1 --2- 172.21.15.57:0/2989486365 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f740406b080 0x7f740406d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:49.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.121+0000 7f74167fc640 1 -- 172.21.15.57:0/2989486365 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 7015+0+0 (secure 0 0 0) 0x7f741008a5d0 con 0x7f74180ab500 2023-10-19T14:21:49.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.121+0000 7f741d70e640 1 --2- 172.21.15.57:0/2989486365 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f740406b080 0x7f740406d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:49.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.122+0000 7f741d70e640 1 --2- 172.21.15.57:0/2989486365 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f740406b080 0x7f740406d540 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f740c0045b0 tx=0x7f740c00b2e0 comp rx=0 tx=0).ready entity=mgr.11446 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:49.138 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.125+0000 7f74167fc640 1 -- 172.21.15.57:0/2989486365 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7410054da0 con 0x7f74180ab500 2023-10-19T14:21:49.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.299+0000 7f7420173640 1 -- 172.21.15.57:0/2989486365 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) v1 -- 0x7f741806c850 con 0x7f74180ab500 2023-10-19T14:21:49.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.299+0000 7f74167fc640 1 -- 172.21.15.57:0/2989486365 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v252) v1 ==== 103+0+11 (secure 0 0 0) 0x7f741001d0a0 con 0x7f74180ab500 2023-10-19T14:21:49.303 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 64 2023-10-19T14:21:49.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.301+0000 7f7420173640 1 -- 172.21.15.57:0/2989486365 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f740406b080 msgr2=0x7f740406d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:49.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.301+0000 7f7420173640 1 --2- 172.21.15.57:0/2989486365 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f740406b080 0x7f740406d540 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f740c0045b0 tx=0x7f740c00b2e0 comp rx=0 tx=0).stop 2023-10-19T14:21:49.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.301+0000 7f7420173640 1 -- 172.21.15.57:0/2989486365 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f74180ab500 msgr2=0x7f74181394d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:49.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.301+0000 7f7420173640 1 --2- 172.21.15.57:0/2989486365 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f74180ab500 0x7f74181394d0 secure :-1 s=READY pgs=2796 cs=0 l=1 rev1=1 crypto rx=0x7f741000c890 tx=0x7f741000cd50 comp rx=0 tx=0).stop 2023-10-19T14:21:49.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.301+0000 7f7420173640 1 -- 172.21.15.57:0/2989486365 shutdown_connections 2023-10-19T14:21:49.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.301+0000 7f7420173640 1 --2- 172.21.15.57:0/2989486365 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f740406b080 0x7f740406d540 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:49.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.301+0000 7f7420173640 1 --2- 172.21.15.57:0/2989486365 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f74180c9dc0 0x7f74180aa8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:49.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.301+0000 7f7420173640 1 --2- 172.21.15.57:0/2989486365 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f74180c99e0 0x7f74181f2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:49.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.301+0000 7f7420173640 1 --2- 172.21.15.57:0/2989486365 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f74180ab500 0x7f74181394d0 unknown :-1 s=CLOSED pgs=2796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:49.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.301+0000 7f7420173640 1 -- 172.21.15.57:0/2989486365 >> 172.21.15.57:0/2989486365 conn(0x7f7418069930 msgr2=0x7f74180ac3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:49.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.301+0000 7f7420173640 1 -- 172.21.15.57:0/2989486365 shutdown_connections 2023-10-19T14:21:49.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.301+0000 7f7420173640 1 -- 172.21.15.57:0/2989486365 wait complete. 2023-10-19T14:21:49.315 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T14:21:49.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.422+0000 7f143f2d7640 1 -- 172.21.15.57:0/3597129034 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1424004d00 msgr2=0x7f1424005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:49.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.422+0000 7f143f2d7640 1 --2- 172.21.15.57:0/3597129034 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1424004d00 0x7f1424005180 secure :-1 s=READY pgs=2798 cs=0 l=1 rev1=1 crypto rx=0x7f1430009ed0 tx=0x7f14240057f0 comp rx=0 tx=0).stop 2023-10-19T14:21:49.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.422+0000 7f143d874640 1 -- 172.21.15.57:0/3597129034 reap_dead start 2023-10-19T14:21:49.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.422+0000 7f143f2d7640 1 -- 172.21.15.57:0/3597129034 shutdown_connections 2023-10-19T14:21:49.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.422+0000 7f143f2d7640 1 -- 172.21.15.57:0/3597129034 >> 172.21.15.57:0/3597129034 conn(0x7f14380d0d60 msgr2=0x7f14380d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:49.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.422+0000 7f143f2d7640 1 -- 172.21.15.57:0/3597129034 shutdown_connections 2023-10-19T14:21:49.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.422+0000 7f143f2d7640 1 -- 172.21.15.57:0/3597129034 wait complete. 2023-10-19T14:21:49.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.423+0000 7f143f2d7640 1 Processor -- start 2023-10-19T14:21:49.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.423+0000 7f143f2d7640 1 -- start start 2023-10-19T14:21:49.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.424+0000 7f143f2d7640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f14381524d0 0x7f143815c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:49.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.424+0000 7f143f2d7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f14381528b0 0x7f14381e4410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:49.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.424+0000 7f143f2d7640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f14381e5c90 0x7f14381e7030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:49.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.424+0000 7f143f2d7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14381e5870 con 0x7f14381528b0 2023-10-19T14:21:49.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.424+0000 7f143f2d7640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f143815c830 con 0x7f14381524d0 2023-10-19T14:21:49.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.424+0000 7f143f2d7640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14381e4950 con 0x7f14381e5c90 2023-10-19T14:21:49.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.424+0000 7f143c872640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f14381528b0 0x7f14381e4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:49.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.424+0000 7f143c872640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f14381528b0 0x7f14381e4410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34088/0 (socket says 172.21.15.57:34088) 2023-10-19T14:21:49.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.424+0000 7f143c872640 1 -- 172.21.15.57:0/3056817093 learned_addr learned my addr 172.21.15.57:0/3056817093 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:49.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.424+0000 7f143d874640 1 --2- 172.21.15.57:0/3056817093 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f14381e5c90 0x7f14381e7030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:49.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.424+0000 7f143d073640 1 --2- 172.21.15.57:0/3056817093 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f14381524d0 0x7f143815c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:49.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.424+0000 7f143c872640 1 -- 172.21.15.57:0/3056817093 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f14381e5c90 msgr2=0x7f14381e7030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:49.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.424+0000 7f143c872640 1 --2- 172.21.15.57:0/3056817093 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f14381e5c90 0x7f14381e7030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:49.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.424+0000 7f143c872640 1 -- 172.21.15.57:0/3056817093 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f14381524d0 msgr2=0x7f143815c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:49.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.424+0000 7f143c872640 1 --2- 172.21.15.57:0/3056817093 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f14381524d0 0x7f143815c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:49.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.424+0000 7f143c872640 1 -- 172.21.15.57:0/3056817093 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14381e4bf0 con 0x7f14381528b0 2023-10-19T14:21:49.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.424+0000 7f143c872640 1 --2- 172.21.15.57:0/3056817093 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f14381528b0 0x7f14381e4410 secure :-1 s=READY pgs=2799 cs=0 l=1 rev1=1 crypto rx=0x7f142800e6d0 tx=0x7f142800eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:49.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.424+0000 7f142dffb640 1 -- 172.21.15.57:0/3056817093 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f142800c9e0 con 0x7f14381528b0 2023-10-19T14:21:49.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.425+0000 7f142dffb640 1 -- 172.21.15.57:0/3056817093 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f142800cc30 con 0x7f14381528b0 2023-10-19T14:21:49.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.425+0000 7f142dffb640 1 -- 172.21.15.57:0/3056817093 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1428011460 con 0x7f14381528b0 2023-10-19T14:21:49.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.425+0000 7f143f2d7640 1 -- 172.21.15.57:0/3056817093 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14381e4ed0 con 0x7f14381528b0 2023-10-19T14:21:49.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.425+0000 7f143f2d7640 1 -- 172.21.15.57:0/3056817093 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14381e5490 con 0x7f14381528b0 2023-10-19T14:21:49.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.426+0000 7f143f2d7640 1 -- 172.21.15.57:0/3056817093 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14381e59b0 con 0x7f14381528b0 2023-10-19T14:21:49.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.426+0000 7f142dffb640 1 -- 172.21.15.57:0/3056817093 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 145) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f14280117d0 con 0x7f14381528b0 2023-10-19T14:21:49.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.426+0000 7f142dffb640 1 --2- 172.21.15.57:0/3056817093 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f141c06b080 0x7f141c06d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:49.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.427+0000 7f142dffb640 1 -- 172.21.15.57:0/3056817093 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 7015+0+0 (secure 0 0 0) 0x7f142808ab80 con 0x7f14381528b0 2023-10-19T14:21:49.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.427+0000 7f143d073640 1 --2- 172.21.15.57:0/3056817093 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f141c06b080 0x7f141c06d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:49.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.427+0000 7f143d073640 1 --2- 172.21.15.57:0/3056817093 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f141c06b080 0x7f141c06d540 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f1434009b10 tx=0x7f1434005f90 comp rx=0 tx=0).ready entity=mgr.11446 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:49.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.431+0000 7f142dffb640 1 -- 172.21.15.57:0/3056817093 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f14381e59b0 con 0x7f14381528b0 2023-10-19T14:21:49.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.646+0000 7f143f2d7640 1 -- 172.21.15.57:0/3056817093 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f14381e59b0 con 0x7f14381528b0 2023-10-19T14:21:49.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.646+0000 7f142dffb640 1 -- 172.21.15.57:0/3056817093 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 109 v109) v1 ==== 95+0+4915 (secure 0 0 0) 0x7f14381e59b0 con 0x7f14381528b0 2023-10-19T14:21:49.650 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:21:49.650 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":109,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9761,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/325888554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":325888554},{"type":"v1","addr":"172.21.15.179:6836","nonce":325888554}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":100},{"gid":9764,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/8038232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":8038232},{"type":"v1","addr":"172.21.15.179:6837","nonce":8038232}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":100},{"gid":13170,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/4130227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":4130227},{"type":"v1","addr":"172.21.15.57:6835","nonce":4130227}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":109}],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T14:21:26.724728+0000","modified":"2023-10-19T14:21:28.872856+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":12555},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12555":{"gid":12555,"name":"a","rank":0,"incarnation":105,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6833/4256498777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":4256498777},{"type":"v1","addr":"172.21.15.57:6833","nonce":4256498777}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2023-10-19T14:21:49.650 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 109 2023-10-19T14:21:49.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.648+0000 7f143f2d7640 1 -- 172.21.15.57:0/3056817093 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f141c06b080 msgr2=0x7f141c06d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:49.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.649+0000 7f143f2d7640 1 --2- 172.21.15.57:0/3056817093 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f141c06b080 0x7f141c06d540 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f1434009b10 tx=0x7f1434005f90 comp rx=0 tx=0).stop 2023-10-19T14:21:49.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.649+0000 7f143f2d7640 1 -- 172.21.15.57:0/3056817093 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f14381528b0 msgr2=0x7f14381e4410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:49.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.649+0000 7f143f2d7640 1 --2- 172.21.15.57:0/3056817093 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f14381528b0 0x7f14381e4410 secure :-1 s=READY pgs=2799 cs=0 l=1 rev1=1 crypto rx=0x7f142800e6d0 tx=0x7f142800eb90 comp rx=0 tx=0).stop 2023-10-19T14:21:49.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.649+0000 7f143f2d7640 1 -- 172.21.15.57:0/3056817093 shutdown_connections 2023-10-19T14:21:49.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.649+0000 7f143f2d7640 1 --2- 172.21.15.57:0/3056817093 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f141c06b080 0x7f141c06d540 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:49.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.649+0000 7f143f2d7640 1 --2- 172.21.15.57:0/3056817093 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f14381e5c90 0x7f14381e7030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:49.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.649+0000 7f143f2d7640 1 --2- 172.21.15.57:0/3056817093 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f14381528b0 0x7f14381e4410 unknown :-1 s=CLOSED pgs=2799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:49.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.649+0000 7f143f2d7640 1 --2- 172.21.15.57:0/3056817093 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f14381524d0 0x7f143815c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:49.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.649+0000 7f143f2d7640 1 -- 172.21.15.57:0/3056817093 >> 172.21.15.57:0/3056817093 conn(0x7f14380d0d60 msgr2=0x7f143806a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:49.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.649+0000 7f143f2d7640 1 -- 172.21.15.57:0/3056817093 shutdown_connections 2023-10-19T14:21:49.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.649+0000 7f143f2d7640 1 -- 172.21.15.57:0/3056817093 wait complete. 2023-10-19T14:21:49.663 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T14:21:49.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.769+0000 7f6c4f0d7640 1 -- 172.21.15.57:0/265354680 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6c4815b2a0 msgr2=0x7f6c480d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:49.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.769+0000 7f6c4f0d7640 1 --2- 172.21.15.57:0/265354680 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6c4815b2a0 0x7f6c480d1910 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f6c34005fd0 tx=0x7f6c3402bd90 comp rx=0 tx=0).stop 2023-10-19T14:21:49.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.769+0000 7f6c4f0d7640 1 -- 172.21.15.57:0/265354680 shutdown_connections 2023-10-19T14:21:49.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.769+0000 7f6c4f0d7640 1 --2- 172.21.15.57:0/265354680 >> v2:172.21.15.179:3300/0 conn(0x7f6c480ceb40 0x7f6c480b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:49.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.769+0000 7f6c4f0d7640 1 --2- 172.21.15.57:0/265354680 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6c4815b2a0 0x7f6c480d1910 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:49.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.769+0000 7f6c4f0d7640 1 -- 172.21.15.57:0/265354680 >> 172.21.15.57:0/265354680 conn(0x7f6c480d0d60 msgr2=0x7f6c480d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:49.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.769+0000 7f6c4f0d7640 1 -- 172.21.15.57:0/265354680 shutdown_connections 2023-10-19T14:21:49.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.769+0000 7f6c4f0d7640 1 -- 172.21.15.57:0/265354680 wait complete. 2023-10-19T14:21:49.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.770+0000 7f6c4f0d7640 1 Processor -- start 2023-10-19T14:21:49.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.770+0000 7f6c4f0d7640 1 -- start start 2023-10-19T14:21:49.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.771+0000 7f6c4f0d7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6c480ceb40 0x7f6c4814f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:49.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.771+0000 7f6c4f0d7640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6c480cf340 0x7f6c480b6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:49.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.771+0000 7f6c4f0d7640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6c4815b2a0 0x7f6c481e3e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:49.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.771+0000 7f6c4f0d7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c480cad00 con 0x7f6c480ceb40 2023-10-19T14:21:49.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.771+0000 7f6c4f0d7640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c480ca080 con 0x7f6c480cf340 2023-10-19T14:21:49.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.771+0000 7f6c4f0d7640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c480ca360 con 0x7f6c4815b2a0 2023-10-19T14:21:49.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.771+0000 7f6c4ce73640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6c480ceb40 0x7f6c4814f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:49.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.771+0000 7f6c4ce73640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6c480ceb40 0x7f6c4814f600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34094/0 (socket says 172.21.15.57:34094) 2023-10-19T14:21:49.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.771+0000 7f6c4ce73640 1 -- 172.21.15.57:0/1453657542 learned_addr learned my addr 172.21.15.57:0/1453657542 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:49.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.771+0000 7f6c47fff640 1 --2- 172.21.15.57:0/1453657542 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6c480cf340 0x7f6c480b6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:49.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.771+0000 7f6c4d674640 1 --2- 172.21.15.57:0/1453657542 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6c4815b2a0 0x7f6c481e3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:49.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.771+0000 7f6c4ce73640 1 -- 172.21.15.57:0/1453657542 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6c4815b2a0 msgr2=0x7f6c481e3e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:49.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.771+0000 7f6c4ce73640 1 --2- 172.21.15.57:0/1453657542 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6c4815b2a0 0x7f6c481e3e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:49.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.771+0000 7f6c4ce73640 1 -- 172.21.15.57:0/1453657542 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6c480cf340 msgr2=0x7f6c480b6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:49.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.771+0000 7f6c4ce73640 1 --2- 172.21.15.57:0/1453657542 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6c480cf340 0x7f6c480b6240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:49.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.771+0000 7f6c4ce73640 1 -- 172.21.15.57:0/1453657542 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c40009840 con 0x7f6c480ceb40 2023-10-19T14:21:49.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.771+0000 7f6c4ce73640 1 --2- 172.21.15.57:0/1453657542 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6c480ceb40 0x7f6c4814f600 secure :-1 s=READY pgs=2800 cs=0 l=1 rev1=1 crypto rx=0x7f6c3c00c450 tx=0x7f6c3c00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:49.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.772+0000 7f6c457fa640 1 -- 172.21.15.57:0/1453657542 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c3c0052d0 con 0x7f6c480ceb40 2023-10-19T14:21:49.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.772+0000 7f6c457fa640 1 -- 172.21.15.57:0/1453657542 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f6c3c0035f0 con 0x7f6c480ceb40 2023-10-19T14:21:49.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.772+0000 7f6c4f0d7640 1 -- 172.21.15.57:0/1453657542 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c34009560 con 0x7f6c480ceb40 2023-10-19T14:21:49.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.772+0000 7f6c457fa640 1 -- 172.21.15.57:0/1453657542 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c3c003a00 con 0x7f6c480ceb40 2023-10-19T14:21:49.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.772+0000 7f6c4f0d7640 1 -- 172.21.15.57:0/1453657542 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c480b0d50 con 0x7f6c480ceb40 2023-10-19T14:21:49.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.773+0000 7f6c4f0d7640 1 -- 172.21.15.57:0/1453657542 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c4808a4b0 con 0x7f6c480ceb40 2023-10-19T14:21:49.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.773+0000 7f6c457fa640 1 -- 172.21.15.57:0/1453657542 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 145) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f6c3c003b60 con 0x7f6c480ceb40 2023-10-19T14:21:49.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.774+0000 7f6c457fa640 1 --2- 172.21.15.57:0/1453657542 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f6c2c06b080 0x7f6c2c06d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:49.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.774+0000 7f6c457fa640 1 -- 172.21.15.57:0/1453657542 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 7015+0+0 (secure 0 0 0) 0x7f6c3c089530 con 0x7f6c480ceb40 2023-10-19T14:21:49.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.774+0000 7f6c47fff640 1 --2- 172.21.15.57:0/1453657542 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f6c2c06b080 0x7f6c2c06d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:49.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.775+0000 7f6c47fff640 1 --2- 172.21.15.57:0/1453657542 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f6c2c06b080 0x7f6c2c06d540 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6c34035000 tx=0x7f6c34004730 comp rx=0 tx=0).ready entity=mgr.11446 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:49.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.778+0000 7f6c457fa640 1 -- 172.21.15.57:0/1453657542 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6c3c053ed0 con 0x7f6c480ceb40 2023-10-19T14:21:49.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.954+0000 7f6c4f0d7640 1 -- 172.21.15.57:0/1453657542 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6c481e3ce0 con 0x7f6c480ceb40 2023-10-19T14:21:49.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.955+0000 7f6c457fa640 1 -- 172.21.15.57:0/1453657542 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v252) v1 ==== 74+0+18176 (secure 0 0 0) 0x7f6c3c0574f0 con 0x7f6c480ceb40 2023-10-19T14:21:49.958 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:21:49.958 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":252,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T14:21:48.026331+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_metadata","create_time":"2023-10-19T14:21:24.384693+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"246","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"backup_fs_data","create_time":"2023-10-19T14:21:25.717524+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"247","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.57:6834/18187246":"2023-10-20T14:21:44.967770+0000","172.21.15.179:0/3646431735":"2023-10-20T14:21:41.395725+0000","172.21.15.179:0/2986772961":"2023-10-20T14:21:41.395725+0000","172.21.15.179:0/837564123":"2023-10-20T14:21:41.395725+0000","172.21.15.179:0/1917522846":"2023-10-20T14:21:41.395725+0000","172.21.15.179:0/3420876591":"2023-10-20T14:21:41.395725+0000","172.21.15.179:6833/609071913":"2023-10-20T14:21:41.395725+0000","172.21.15.179:6832/609071913":"2023-10-20T14:21:41.395725+0000","172.21.15.179:0/2525508070":"2023-10-20T14:21:35.927420+0000","172.21.15.179:0/75322738":"2023-10-20T14:21:41.395725+0000","172.21.15.179:0/720769249":"2023-10-20T14:21:35.927420+0000","172.21.15.57:6835/18187246":"2023-10-20T14:21:44.967770+0000","172.21.15.179:6833/3547895240":"2023-10-20T14:21:35.927420+0000","172.21.15.179:6832/3547895240":"2023-10-20T14:21:35.927420+0000","172.21.15.179:0/177563067":"2023-10-20T14:21:35.927420+0000","172.21.15.179:0/1656904674":"2023-10-20T14:21:41.395725+0000","172.21.15.179:0/647327435":"2023-10-20T14:21:41.395725+0000","172.21.15.179:0/1877726906":"2023-10-20T14:21:35.927420+0000","172.21.15.179:0/3471744208":"2023-10-20T14:21:35.927420+0000","172.21.15.179:0/2477290155":"2023-10-20T14:21:35.927420+0000","172.21.15.179:0/1240052109":"2023-10-20T14:21:35.927420+0000","172.21.15.179:0/4042925688":"2023-10-20T14:21:35.927420+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T14:21:49.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.957+0000 7f6c4f0d7640 1 -- 172.21.15.57:0/1453657542 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f6c2c06b080 msgr2=0x7f6c2c06d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:49.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.957+0000 7f6c4f0d7640 1 --2- 172.21.15.57:0/1453657542 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f6c2c06b080 0x7f6c2c06d540 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6c34035000 tx=0x7f6c34004730 comp rx=0 tx=0).stop 2023-10-19T14:21:49.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.957+0000 7f6c4f0d7640 1 -- 172.21.15.57:0/1453657542 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6c480ceb40 msgr2=0x7f6c4814f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:49.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.957+0000 7f6c4f0d7640 1 --2- 172.21.15.57:0/1453657542 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6c480ceb40 0x7f6c4814f600 secure :-1 s=READY pgs=2800 cs=0 l=1 rev1=1 crypto rx=0x7f6c3c00c450 tx=0x7f6c3c00c850 comp rx=0 tx=0).stop 2023-10-19T14:21:49.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.957+0000 7f6c4f0d7640 1 -- 172.21.15.57:0/1453657542 shutdown_connections 2023-10-19T14:21:49.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.957+0000 7f6c4f0d7640 1 --2- 172.21.15.57:0/1453657542 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f6c2c06b080 0x7f6c2c06d540 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:49.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.957+0000 7f6c4f0d7640 1 --2- 172.21.15.57:0/1453657542 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6c4815b2a0 0x7f6c481e3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:49.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.957+0000 7f6c4f0d7640 1 --2- 172.21.15.57:0/1453657542 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6c480cf340 0x7f6c480b6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:49.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.957+0000 7f6c4f0d7640 1 --2- 172.21.15.57:0/1453657542 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6c480ceb40 0x7f6c4814f600 unknown :-1 s=CLOSED pgs=2800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:49.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.957+0000 7f6c4f0d7640 1 -- 172.21.15.57:0/1453657542 >> 172.21.15.57:0/1453657542 conn(0x7f6c480d0d60 msgr2=0x7f6c4806b3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:49.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.957+0000 7f6c4f0d7640 1 -- 172.21.15.57:0/1453657542 shutdown_connections 2023-10-19T14:21:49.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:49.957+0000 7f6c4f0d7640 1 -- 172.21.15.57:0/1453657542 wait complete. 2023-10-19T14:21:49.971 INFO:tasks.cephfs.filesystem:Destroying file system backup_fs and related pools 2023-10-19T14:21:49.971 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T14:21:50.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.078+0000 7f4b1ba1a640 1 -- 172.21.15.57:0/681201415 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4b140ac870 msgr2=0x7f4b1406b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:50.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.078+0000 7f4b1ba1a640 1 --2- 172.21.15.57:0/681201415 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4b140ac870 0x7f4b1406b980 secure :-1 s=READY pgs=2802 cs=0 l=1 rev1=1 crypto rx=0x7f4b140ac610 tx=0x7f4b0402be20 comp rx=0 tx=0).stop 2023-10-19T14:21:50.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.078+0000 7f4b1ba1a640 1 -- 172.21.15.57:0/681201415 shutdown_connections 2023-10-19T14:21:50.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.078+0000 7f4b1ba1a640 1 --2- 172.21.15.57:0/681201415 >> v2:172.21.15.179:3300/0 conn(0x7f4b1406bec0 0x7f4b1406a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:50.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.078+0000 7f4b1ba1a640 1 --2- 172.21.15.57:0/681201415 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4b140ac870 0x7f4b1406b980 unknown :-1 s=CLOSED pgs=2802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:50.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.078+0000 7f4b1ba1a640 1 -- 172.21.15.57:0/681201415 >> 172.21.15.57:0/681201415 conn(0x7f4b1406fae0 msgr2=0x7f4b1406fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:50.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.078+0000 7f4b1ba1a640 1 -- 172.21.15.57:0/681201415 shutdown_connections 2023-10-19T14:21:50.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.078+0000 7f4b1ba1a640 1 -- 172.21.15.57:0/681201415 wait complete. 2023-10-19T14:21:50.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.079+0000 7f4b1ba1a640 1 Processor -- start 2023-10-19T14:21:50.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.079+0000 7f4b1ba1a640 1 -- start start 2023-10-19T14:21:50.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.080+0000 7f4b1ba1a640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4b1406bec0 0x7f4b1414b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:50.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.080+0000 7f4b1ba1a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4b140ac870 0x7f4b14151390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:50.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.080+0000 7f4b1ba1a640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4b14156790 0x7f4b141518d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:50.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.080+0000 7f4b1ba1a640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b14150670 con 0x7f4b140ac870 2023-10-19T14:21:50.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.080+0000 7f4b1ba1a640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b1415a3e0 con 0x7f4b1406bec0 2023-10-19T14:21:50.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.080+0000 7f4b1ba1a640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b1415a6c0 con 0x7f4b14156790 2023-10-19T14:21:50.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.080+0000 7f4b18fb5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4b140ac870 0x7f4b14151390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:50.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.080+0000 7f4b18fb5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4b140ac870 0x7f4b14151390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34116/0 (socket says 172.21.15.57:34116) 2023-10-19T14:21:50.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.080+0000 7f4b18fb5640 1 -- 172.21.15.57:0/4091347674 learned_addr learned my addr 172.21.15.57:0/4091347674 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:50.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.080+0000 7f4b18fb5640 1 -- 172.21.15.57:0/4091347674 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4b14156790 msgr2=0x7f4b141518d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:50.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.080+0000 7f4b18fb5640 1 --2- 172.21.15.57:0/4091347674 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4b14156790 0x7f4b141518d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:50.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.080+0000 7f4b18fb5640 1 -- 172.21.15.57:0/4091347674 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4b1406bec0 msgr2=0x7f4b1414b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:50.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.080+0000 7f4b18fb5640 1 --2- 172.21.15.57:0/4091347674 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4b1406bec0 0x7f4b1414b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:50.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.080+0000 7f4b18fb5640 1 -- 172.21.15.57:0/4091347674 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b04009560 con 0x7f4b140ac870 2023-10-19T14:21:50.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.080+0000 7f4b18fb5640 1 --2- 172.21.15.57:0/4091347674 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4b140ac870 0x7f4b14151390 secure :-1 s=READY pgs=2803 cs=0 l=1 rev1=1 crypto rx=0x7f4b04036040 tx=0x7f4b0402dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:50.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.080+0000 7f4b09ffb640 1 -- 172.21.15.57:0/4091347674 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b040035f0 con 0x7f4b140ac870 2023-10-19T14:21:50.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.080+0000 7f4b09ffb640 1 -- 172.21.15.57:0/4091347674 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f4b04003840 con 0x7f4b140ac870 2023-10-19T14:21:50.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.080+0000 7f4b1ba1a640 1 -- 172.21.15.57:0/4091347674 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b1415a9a0 con 0x7f4b140ac870 2023-10-19T14:21:50.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.080+0000 7f4b09ffb640 1 -- 172.21.15.57:0/4091347674 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b04003bf0 con 0x7f4b140ac870 2023-10-19T14:21:50.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.081+0000 7f4b1ba1a640 1 -- 172.21.15.57:0/4091347674 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b1414f8d0 con 0x7f4b140ac870 2023-10-19T14:21:50.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.082+0000 7f4b1ba1a640 1 -- 172.21.15.57:0/4091347674 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b1415a800 con 0x7f4b140ac870 2023-10-19T14:21:50.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.082+0000 7f4b09ffb640 1 -- 172.21.15.57:0/4091347674 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 145) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f4b04003d50 con 0x7f4b140ac870 2023-10-19T14:21:50.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.083+0000 7f4b09ffb640 1 --2- 172.21.15.57:0/4091347674 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f4af806b080 0x7f4af806d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:50.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.083+0000 7f4b09ffb640 1 -- 172.21.15.57:0/4091347674 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 7015+0+0 (secure 0 0 0) 0x7f4b040acbb0 con 0x7f4b140ac870 2023-10-19T14:21:50.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.083+0000 7f4b197b6640 1 --2- 172.21.15.57:0/4091347674 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f4af806b080 0x7f4af806d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:50.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.084+0000 7f4b197b6640 1 --2- 172.21.15.57:0/4091347674 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f4af806b080 0x7f4af806d540 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f4b10000f60 tx=0x7f4b100079b0 comp rx=0 tx=0).ready entity=mgr.11446 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:50.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.087+0000 7f4b09ffb640 1 -- 172.21.15.57:0/4091347674 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4b1415a800 con 0x7f4b140ac870 2023-10-19T14:21:50.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.297+0000 7f4b1ba1a640 1 -- 172.21.15.57:0/4091347674 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4b1415a800 con 0x7f4b140ac870 2023-10-19T14:21:50.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.297+0000 7f4b09ffb640 1 -- 172.21.15.57:0/4091347674 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 109 v109) v1 ==== 95+0+4915 (secure 0 0 0) 0x7f4b1415a800 con 0x7f4b140ac870 2023-10-19T14:21:50.301 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:21:50.301 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":109,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9761,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/325888554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":325888554},{"type":"v1","addr":"172.21.15.179:6836","nonce":325888554}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":100},{"gid":9764,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/8038232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":8038232},{"type":"v1","addr":"172.21.15.179:6837","nonce":8038232}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":100},{"gid":13170,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/4130227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":4130227},{"type":"v1","addr":"172.21.15.57:6835","nonce":4130227}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":109}],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T14:21:26.724728+0000","modified":"2023-10-19T14:21:28.872856+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":12555},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12555":{"gid":12555,"name":"a","rank":0,"incarnation":105,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6833/4256498777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":4256498777},{"type":"v1","addr":"172.21.15.57:6833","nonce":4256498777}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2023-10-19T14:21:50.301 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 109 2023-10-19T14:21:50.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.299+0000 7f4b1ba1a640 1 -- 172.21.15.57:0/4091347674 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f4af806b080 msgr2=0x7f4af806d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:50.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.299+0000 7f4b1ba1a640 1 --2- 172.21.15.57:0/4091347674 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f4af806b080 0x7f4af806d540 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f4b10000f60 tx=0x7f4b100079b0 comp rx=0 tx=0).stop 2023-10-19T14:21:50.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.299+0000 7f4b1ba1a640 1 -- 172.21.15.57:0/4091347674 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4b140ac870 msgr2=0x7f4b14151390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:50.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.299+0000 7f4b1ba1a640 1 --2- 172.21.15.57:0/4091347674 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4b140ac870 0x7f4b14151390 secure :-1 s=READY pgs=2803 cs=0 l=1 rev1=1 crypto rx=0x7f4b04036040 tx=0x7f4b0402dbc0 comp rx=0 tx=0).stop 2023-10-19T14:21:50.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.299+0000 7f4b1ba1a640 1 -- 172.21.15.57:0/4091347674 shutdown_connections 2023-10-19T14:21:50.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.299+0000 7f4b1ba1a640 1 --2- 172.21.15.57:0/4091347674 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f4af806b080 0x7f4af806d540 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:50.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.299+0000 7f4b1ba1a640 1 --2- 172.21.15.57:0/4091347674 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4b14156790 0x7f4b141518d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:50.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.299+0000 7f4b1ba1a640 1 --2- 172.21.15.57:0/4091347674 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4b140ac870 0x7f4b14151390 unknown :-1 s=CLOSED pgs=2803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:50.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.299+0000 7f4b1ba1a640 1 --2- 172.21.15.57:0/4091347674 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4b1406bec0 0x7f4b1414b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:50.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.299+0000 7f4b1ba1a640 1 -- 172.21.15.57:0/4091347674 >> 172.21.15.57:0/4091347674 conn(0x7f4b1406fae0 msgr2=0x7f4b140c9670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:50.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.299+0000 7f4b1ba1a640 1 -- 172.21.15.57:0/4091347674 shutdown_connections 2023-10-19T14:21:50.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.299+0000 7f4b1ba1a640 1 -- 172.21.15.57:0/4091347674 wait complete. 2023-10-19T14:21:50.313 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T14:21:50.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.422+0000 7f470afb6640 1 -- 172.21.15.57:0/2801559297 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f47040ceb40 msgr2=0x7f47040b0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:50.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.422+0000 7f470afb6640 1 --2- 172.21.15.57:0/2801559297 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f47040ceb40 0x7f47040b0650 secure :-1 s=READY pgs=2805 cs=0 l=1 rev1=1 crypto rx=0x7f46f4009830 tx=0x7f46f402bcd0 comp rx=0 tx=0).stop 2023-10-19T14:21:50.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.422+0000 7f470afb6640 1 -- 172.21.15.57:0/2801559297 shutdown_connections 2023-10-19T14:21:50.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.422+0000 7f470afb6640 1 --2- 172.21.15.57:0/2801559297 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f470415b2a0 0x7f47040d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:50.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.422+0000 7f470afb6640 1 --2- 172.21.15.57:0/2801559297 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f47040ceb40 0x7f47040b0650 unknown :-1 s=CLOSED pgs=2805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:50.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.422+0000 7f470afb6640 1 -- 172.21.15.57:0/2801559297 >> 172.21.15.57:0/2801559297 conn(0x7f47040d0d60 msgr2=0x7f47040d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:50.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.422+0000 7f470afb6640 1 -- 172.21.15.57:0/2801559297 shutdown_connections 2023-10-19T14:21:50.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.422+0000 7f470afb6640 1 -- 172.21.15.57:0/2801559297 wait complete. 2023-10-19T14:21:50.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.423+0000 7f470afb6640 1 Processor -- start 2023-10-19T14:21:50.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.423+0000 7f470afb6640 1 -- start start 2023-10-19T14:21:50.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.424+0000 7f470afb6640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f47040ceb40 0x7f47041f4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:50.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.424+0000 7f470afb6640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f47040cf340 0x7f47040b5cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:50.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.424+0000 7f470afb6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f470415b2a0 0x7f470415dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:50.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.424+0000 7f470afb6640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f470415b790 con 0x7f470415b2a0 2023-10-19T14:21:50.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.424+0000 7f470afb6640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4704156060 con 0x7f47040ceb40 2023-10-19T14:21:50.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.424+0000 7f470afb6640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47041f4be0 con 0x7f47040cf340 2023-10-19T14:21:50.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.424+0000 7f4709553640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f470415b2a0 0x7f470415dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:50.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.424+0000 7f4709553640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f470415b2a0 0x7f470415dd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34134/0 (socket says 172.21.15.57:34134) 2023-10-19T14:21:50.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.424+0000 7f4709553640 1 -- 172.21.15.57:0/2729163802 learned_addr learned my addr 172.21.15.57:0/2729163802 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:50.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.424+0000 7f4703fff640 1 --2- 172.21.15.57:0/2729163802 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f47040cf340 0x7f47040b5cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:50.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.424+0000 7f4708d52640 1 --2- 172.21.15.57:0/2729163802 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f47040ceb40 0x7f47041f4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:50.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.425+0000 7f4709553640 1 -- 172.21.15.57:0/2729163802 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f47040cf340 msgr2=0x7f47040b5cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:50.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.425+0000 7f4709553640 1 --2- 172.21.15.57:0/2729163802 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f47040cf340 0x7f47040b5cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:50.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.425+0000 7f4709553640 1 -- 172.21.15.57:0/2729163802 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f47040ceb40 msgr2=0x7f47041f4600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:50.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.425+0000 7f4709553640 1 --2- 172.21.15.57:0/2729163802 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f47040ceb40 0x7f47041f4600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:50.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.425+0000 7f4709553640 1 -- 172.21.15.57:0/2729163802 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46f4009490 con 0x7f470415b2a0 2023-10-19T14:21:50.432 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.425+0000 7f4703fff640 1 --2- 172.21.15.57:0/2729163802 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f47040cf340 0x7f47040b5cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:21:50.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.425+0000 7f4709553640 1 --2- 172.21.15.57:0/2729163802 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f470415b2a0 0x7f470415dd50 secure :-1 s=READY pgs=2806 cs=0 l=1 rev1=1 crypto rx=0x7f46f4009780 tx=0x7f46f4003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:50.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.425+0000 7f47017fa640 1 -- 172.21.15.57:0/2729163802 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46f4003e20 con 0x7f470415b2a0 2023-10-19T14:21:50.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.425+0000 7f47017fa640 1 -- 172.21.15.57:0/2729163802 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f46f4034530 con 0x7f470415b2a0 2023-10-19T14:21:50.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.425+0000 7f470afb6640 1 -- 172.21.15.57:0/2729163802 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47041f4ec0 con 0x7f470415b2a0 2023-10-19T14:21:50.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.425+0000 7f47017fa640 1 -- 172.21.15.57:0/2729163802 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46f4034940 con 0x7f470415b2a0 2023-10-19T14:21:50.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.425+0000 7f470afb6640 1 -- 172.21.15.57:0/2729163802 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47041f5480 con 0x7f470415b2a0 2023-10-19T14:21:50.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.426+0000 7f470afb6640 1 -- 172.21.15.57:0/2729163802 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47040690e0 con 0x7f470415b2a0 2023-10-19T14:21:50.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.426+0000 7f47017fa640 1 -- 172.21.15.57:0/2729163802 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 145) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f46f403b040 con 0x7f470415b2a0 2023-10-19T14:21:50.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.427+0000 7f47017fa640 1 --2- 172.21.15.57:0/2729163802 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f46f006b080 0x7f46f006d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:50.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.427+0000 7f47017fa640 1 -- 172.21.15.57:0/2729163802 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 7015+0+0 (secure 0 0 0) 0x7f46f40ac490 con 0x7f470415b2a0 2023-10-19T14:21:50.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.427+0000 7f4708d52640 1 --2- 172.21.15.57:0/2729163802 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f46f006b080 0x7f46f006d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:50.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.428+0000 7f4708d52640 1 --2- 172.21.15.57:0/2729163802 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f46f006b080 0x7f46f006d540 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f46fc007fd0 tx=0x7f46fc009040 comp rx=0 tx=0).ready entity=mgr.11446 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:50.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.431+0000 7f47017fa640 1 -- 172.21.15.57:0/2729163802 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f46f4076cd0 con 0x7f470415b2a0 2023-10-19T14:21:50.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.644+0000 7f470afb6640 1 -- 172.21.15.57:0/2729163802 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f47040cb980 con 0x7f470415b2a0 2023-10-19T14:21:50.647 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.644+0000 7f47017fa640 1 -- 172.21.15.57:0/2729163802 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 109 v109) v1 ==== 95+0+4915 (secure 0 0 0) 0x7f46f407a2f0 con 0x7f470415b2a0 2023-10-19T14:21:50.648 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:21:50.648 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":109,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9761,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/325888554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":325888554},{"type":"v1","addr":"172.21.15.179:6836","nonce":325888554}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":100},{"gid":9764,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/8038232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":8038232},{"type":"v1","addr":"172.21.15.179:6837","nonce":8038232}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":100},{"gid":13170,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/4130227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":4130227},{"type":"v1","addr":"172.21.15.57:6835","nonce":4130227}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":109}],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T14:21:26.724728+0000","modified":"2023-10-19T14:21:28.872856+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":12555},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12555":{"gid":12555,"name":"a","rank":0,"incarnation":105,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6833/4256498777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":4256498777},{"type":"v1","addr":"172.21.15.57:6833","nonce":4256498777}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2023-10-19T14:21:50.648 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 109 2023-10-19T14:21:50.649 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.646+0000 7f470afb6640 1 -- 172.21.15.57:0/2729163802 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f46f006b080 msgr2=0x7f46f006d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:50.650 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.646+0000 7f470afb6640 1 --2- 172.21.15.57:0/2729163802 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f46f006b080 0x7f46f006d540 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f46fc007fd0 tx=0x7f46fc009040 comp rx=0 tx=0).stop 2023-10-19T14:21:50.650 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.646+0000 7f470afb6640 1 -- 172.21.15.57:0/2729163802 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f470415b2a0 msgr2=0x7f470415dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:50.650 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.646+0000 7f470afb6640 1 --2- 172.21.15.57:0/2729163802 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f470415b2a0 0x7f470415dd50 secure :-1 s=READY pgs=2806 cs=0 l=1 rev1=1 crypto rx=0x7f46f4009780 tx=0x7f46f4003820 comp rx=0 tx=0).stop 2023-10-19T14:21:50.650 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.647+0000 7f470afb6640 1 -- 172.21.15.57:0/2729163802 shutdown_connections 2023-10-19T14:21:50.650 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.647+0000 7f470afb6640 1 --2- 172.21.15.57:0/2729163802 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f46f006b080 0x7f46f006d540 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:50.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.647+0000 7f470afb6640 1 --2- 172.21.15.57:0/2729163802 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f470415b2a0 0x7f470415dd50 unknown :-1 s=CLOSED pgs=2806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:50.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.647+0000 7f470afb6640 1 --2- 172.21.15.57:0/2729163802 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f47040cf340 0x7f47040b5cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:50.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.647+0000 7f470afb6640 1 --2- 172.21.15.57:0/2729163802 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f47040ceb40 0x7f47041f4600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:50.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.647+0000 7f470afb6640 1 -- 172.21.15.57:0/2729163802 >> 172.21.15.57:0/2729163802 conn(0x7f47040d0d60 msgr2=0x7f4704159ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:50.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.647+0000 7f470afb6640 1 -- 172.21.15.57:0/2729163802 shutdown_connections 2023-10-19T14:21:50.652 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.647+0000 7f470afb6640 1 -- 172.21.15.57:0/2729163802 wait complete. 2023-10-19T14:21:50.660 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T14:21:50.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.767+0000 7ff242675640 1 -- 172.21.15.57:0/2415112471 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff228004d00 msgr2=0x7ff228005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:50.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.767+0000 7ff242675640 1 --2- 172.21.15.57:0/2415112471 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff228004d00 0x7ff228005180 secure :-1 s=READY pgs=2808 cs=0 l=1 rev1=1 crypto rx=0x7ff234001d80 tx=0x7ff2280057f0 comp rx=0 tx=0).stop 2023-10-19T14:21:50.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.767+0000 7ff240c12640 1 -- 172.21.15.57:0/2415112471 reap_dead start 2023-10-19T14:21:50.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.767+0000 7ff242675640 1 -- 172.21.15.57:0/2415112471 shutdown_connections 2023-10-19T14:21:50.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.767+0000 7ff242675640 1 -- 172.21.15.57:0/2415112471 >> 172.21.15.57:0/2415112471 conn(0x7ff23c069930 msgr2=0x7ff23c069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:50.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.767+0000 7ff242675640 1 -- 172.21.15.57:0/2415112471 shutdown_connections 2023-10-19T14:21:50.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.767+0000 7ff242675640 1 -- 172.21.15.57:0/2415112471 wait complete. 2023-10-19T14:21:50.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.769+0000 7ff242675640 1 Processor -- start 2023-10-19T14:21:50.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.769+0000 7ff242675640 1 -- start start 2023-10-19T14:21:50.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.769+0000 7ff242675640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff23c0caae0 0x7ff23c1f13e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:50.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.769+0000 7ff242675640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff23c0caec0 0x7ff23c0c7bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:50.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.769+0000 7ff242675640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff23c0c80f0 0x7ff23c13ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:50.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.769+0000 7ff242675640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff23c0c61b0 con 0x7ff23c0c80f0 2023-10-19T14:21:50.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.769+0000 7ff242675640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff23c0c8600 con 0x7ff23c0caae0 2023-10-19T14:21:50.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.769+0000 7ff242675640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff23c0c88e0 con 0x7ff23c0caec0 2023-10-19T14:21:50.776 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.770+0000 7ff240c12640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff23c0c80f0 0x7ff23c13ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:50.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.770+0000 7ff240c12640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff23c0c80f0 0x7ff23c13ae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34152/0 (socket says 172.21.15.57:34152) 2023-10-19T14:21:50.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.770+0000 7ff240c12640 1 -- 172.21.15.57:0/592678002 learned_addr learned my addr 172.21.15.57:0/592678002 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:50.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.770+0000 7ff23b7fe640 1 --2- 172.21.15.57:0/592678002 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff23c0caec0 0x7ff23c0c7bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:50.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.770+0000 7ff23bfff640 1 --2- 172.21.15.57:0/592678002 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff23c0caae0 0x7ff23c1f13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:50.777 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.770+0000 7ff240c12640 1 -- 172.21.15.57:0/592678002 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff23c0caec0 msgr2=0x7ff23c0c7bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:50.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.770+0000 7ff240c12640 1 --2- 172.21.15.57:0/592678002 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff23c0caec0 0x7ff23c0c7bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:50.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.770+0000 7ff240c12640 1 -- 172.21.15.57:0/592678002 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff23c0caae0 msgr2=0x7ff23c1f13e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:50.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.770+0000 7ff240c12640 1 --2- 172.21.15.57:0/592678002 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff23c0caae0 0x7ff23c1f13e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:50.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.770+0000 7ff240c12640 1 -- 172.21.15.57:0/592678002 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff23c0c8bc0 con 0x7ff23c0c80f0 2023-10-19T14:21:50.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.770+0000 7ff23bfff640 1 --2- 172.21.15.57:0/592678002 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff23c0caae0 0x7ff23c1f13e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:21:50.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.770+0000 7ff240c12640 1 --2- 172.21.15.57:0/592678002 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff23c0c80f0 0x7ff23c13ae10 secure :-1 s=READY pgs=2809 cs=0 l=1 rev1=1 crypto rx=0x7ff23402c530 tx=0x7ff234003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:50.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.770+0000 7ff238ff9640 1 -- 172.21.15.57:0/592678002 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff23402ccd0 con 0x7ff23c0c80f0 2023-10-19T14:21:50.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.770+0000 7ff238ff9640 1 -- 172.21.15.57:0/592678002 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7ff2340060b0 con 0x7ff23c0c80f0 2023-10-19T14:21:50.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.770+0000 7ff242675640 1 -- 172.21.15.57:0/592678002 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff23c0c5e40 con 0x7ff23c0c80f0 2023-10-19T14:21:50.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.770+0000 7ff238ff9640 1 -- 172.21.15.57:0/592678002 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2340064a0 con 0x7ff23c0c80f0 2023-10-19T14:21:50.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.770+0000 7ff242675640 1 -- 172.21.15.57:0/592678002 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff23c1f1920 con 0x7ff23c0c80f0 2023-10-19T14:21:50.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.771+0000 7ff242675640 1 -- 172.21.15.57:0/592678002 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff23c1f1b40 con 0x7ff23c0c80f0 2023-10-19T14:21:50.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.772+0000 7ff238ff9640 1 -- 172.21.15.57:0/592678002 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 145) v1 ==== 89433+0+0 (secure 0 0 0) 0x7ff234006600 con 0x7ff23c0c80f0 2023-10-19T14:21:50.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.772+0000 7ff238ff9640 1 --2- 172.21.15.57:0/592678002 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7ff21c06b080 0x7ff21c06d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:50.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.772+0000 7ff238ff9640 1 -- 172.21.15.57:0/592678002 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 7015+0+0 (secure 0 0 0) 0x7ff2340aae60 con 0x7ff23c0c80f0 2023-10-19T14:21:50.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.772+0000 7ff23bfff640 1 --2- 172.21.15.57:0/592678002 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7ff21c06b080 0x7ff21c06d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:50.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.773+0000 7ff23bfff640 1 --2- 172.21.15.57:0/592678002 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7ff21c06b080 0x7ff21c06d540 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff224001200 tx=0x7ff224001150 comp rx=0 tx=0).ready entity=mgr.11446 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:50.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.776+0000 7ff238ff9640 1 -- 172.21.15.57:0/592678002 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff234075740 con 0x7ff23c0c80f0 2023-10-19T14:21:50.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.952+0000 7ff242675640 1 -- 172.21.15.57:0/592678002 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff23c0690e0 con 0x7ff23c0c80f0 2023-10-19T14:21:50.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.954+0000 7ff238ff9640 1 -- 172.21.15.57:0/592678002 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v252) v1 ==== 74+0+18176 (secure 0 0 0) 0x7ff234044070 con 0x7ff23c0c80f0 2023-10-19T14:21:50.957 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:21:50.958 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":252,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T14:21:48.026331+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_metadata","create_time":"2023-10-19T14:21:24.384693+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"246","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"backup_fs_data","create_time":"2023-10-19T14:21:25.717524+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"247","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.57:6834/18187246":"2023-10-20T14:21:44.967770+0000","172.21.15.179:0/3646431735":"2023-10-20T14:21:41.395725+0000","172.21.15.179:0/2986772961":"2023-10-20T14:21:41.395725+0000","172.21.15.179:0/837564123":"2023-10-20T14:21:41.395725+0000","172.21.15.179:0/1917522846":"2023-10-20T14:21:41.395725+0000","172.21.15.179:0/3420876591":"2023-10-20T14:21:41.395725+0000","172.21.15.179:6833/609071913":"2023-10-20T14:21:41.395725+0000","172.21.15.179:6832/609071913":"2023-10-20T14:21:41.395725+0000","172.21.15.179:0/2525508070":"2023-10-20T14:21:35.927420+0000","172.21.15.179:0/75322738":"2023-10-20T14:21:41.395725+0000","172.21.15.179:0/720769249":"2023-10-20T14:21:35.927420+0000","172.21.15.57:6835/18187246":"2023-10-20T14:21:44.967770+0000","172.21.15.179:6833/3547895240":"2023-10-20T14:21:35.927420+0000","172.21.15.179:6832/3547895240":"2023-10-20T14:21:35.927420+0000","172.21.15.179:0/177563067":"2023-10-20T14:21:35.927420+0000","172.21.15.179:0/1656904674":"2023-10-20T14:21:41.395725+0000","172.21.15.179:0/647327435":"2023-10-20T14:21:41.395725+0000","172.21.15.179:0/1877726906":"2023-10-20T14:21:35.927420+0000","172.21.15.179:0/3471744208":"2023-10-20T14:21:35.927420+0000","172.21.15.179:0/2477290155":"2023-10-20T14:21:35.927420+0000","172.21.15.179:0/1240052109":"2023-10-20T14:21:35.927420+0000","172.21.15.179:0/4042925688":"2023-10-20T14:21:35.927420+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T14:21:50.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.956+0000 7ff242675640 1 -- 172.21.15.57:0/592678002 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7ff21c06b080 msgr2=0x7ff21c06d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:50.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.956+0000 7ff242675640 1 --2- 172.21.15.57:0/592678002 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7ff21c06b080 0x7ff21c06d540 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff224001200 tx=0x7ff224001150 comp rx=0 tx=0).stop 2023-10-19T14:21:50.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.956+0000 7ff242675640 1 -- 172.21.15.57:0/592678002 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff23c0c80f0 msgr2=0x7ff23c13ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:50.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.956+0000 7ff242675640 1 --2- 172.21.15.57:0/592678002 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff23c0c80f0 0x7ff23c13ae10 secure :-1 s=READY pgs=2809 cs=0 l=1 rev1=1 crypto rx=0x7ff23402c530 tx=0x7ff234003ba0 comp rx=0 tx=0).stop 2023-10-19T14:21:50.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.956+0000 7ff242675640 1 -- 172.21.15.57:0/592678002 shutdown_connections 2023-10-19T14:21:50.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.956+0000 7ff242675640 1 --2- 172.21.15.57:0/592678002 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7ff21c06b080 0x7ff21c06d540 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:50.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.956+0000 7ff242675640 1 --2- 172.21.15.57:0/592678002 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff23c0c80f0 0x7ff23c13ae10 unknown :-1 s=CLOSED pgs=2809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:50.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.956+0000 7ff242675640 1 --2- 172.21.15.57:0/592678002 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff23c0caec0 0x7ff23c0c7bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:50.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.956+0000 7ff242675640 1 --2- 172.21.15.57:0/592678002 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff23c0caae0 0x7ff23c1f13e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:50.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.956+0000 7ff242675640 1 -- 172.21.15.57:0/592678002 >> 172.21.15.57:0/592678002 conn(0x7ff23c069930 msgr2=0x7ff23c0affe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:50.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.956+0000 7ff242675640 1 -- 172.21.15.57:0/592678002 shutdown_connections 2023-10-19T14:21:50.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:50.956+0000 7ff242675640 1 -- 172.21.15.57:0/592678002 wait complete. 2023-10-19T14:21:50.970 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail backup_fs 2023-10-19T14:21:51.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.078+0000 7fbafa621640 1 -- 172.21.15.57:0/4051488484 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbadc0039a0 msgr2=0x7fbadc003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:51.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.078+0000 7fbafa621640 1 --2- 172.21.15.57:0/4051488484 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbadc0039a0 0x7fbadc003dc0 secure :-1 s=READY pgs=2811 cs=0 l=1 rev1=1 crypto rx=0x7fbae800c260 tx=0x7fbadc0057f0 comp rx=0 tx=0).stop 2023-10-19T14:21:51.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.078+0000 7fbaf8bbe640 1 -- 172.21.15.57:0/4051488484 reap_dead start 2023-10-19T14:21:51.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.078+0000 7fbafa621640 1 -- 172.21.15.57:0/4051488484 shutdown_connections 2023-10-19T14:21:51.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.078+0000 7fbafa621640 1 -- 172.21.15.57:0/4051488484 >> 172.21.15.57:0/4051488484 conn(0x7fbaf4069930 msgr2=0x7fbaf4069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:51.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.078+0000 7fbafa621640 1 -- 172.21.15.57:0/4051488484 shutdown_connections 2023-10-19T14:21:51.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.078+0000 7fbafa621640 1 -- 172.21.15.57:0/4051488484 wait complete. 2023-10-19T14:21:51.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.079+0000 7fbafa621640 1 Processor -- start 2023-10-19T14:21:51.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.080+0000 7fbafa621640 1 -- start start 2023-10-19T14:21:51.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.080+0000 7fbafa621640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbadc0039a0 0x7fbaf4136b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:51.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.080+0000 7fbafa621640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbaf41586a0 0x7fbaf4158b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:51.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.080+0000 7fbafa621640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbaf4153f10 0x7fbaf41543a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:51.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.080+0000 7fbafa621640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf40aac70 con 0x7fbaf41586a0 2023-10-19T14:21:51.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.080+0000 7fbafa621640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf4159060 con 0x7fbaf4153f10 2023-10-19T14:21:51.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.080+0000 7fbafa621640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf41f1820 con 0x7fbadc0039a0 2023-10-19T14:21:51.086 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.081+0000 7fbaf37fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbaf41586a0 0x7fbaf4158b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:51.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.081+0000 7fbaf37fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbaf41586a0 0x7fbaf4158b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34178/0 (socket says 172.21.15.57:34178) 2023-10-19T14:21:51.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.081+0000 7fbaf37fe640 1 -- 172.21.15.57:0/643956128 learned_addr learned my addr 172.21.15.57:0/643956128 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:51.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.081+0000 7fbaf3fff640 1 --2- 172.21.15.57:0/643956128 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbadc0039a0 0x7fbaf4136b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:51.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.081+0000 7fbaf8bbe640 1 --2- 172.21.15.57:0/643956128 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbaf4153f10 0x7fbaf41543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:51.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.081+0000 7fbaf37fe640 1 -- 172.21.15.57:0/643956128 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbadc0039a0 msgr2=0x7fbaf4136b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:51.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.081+0000 7fbaf37fe640 1 --2- 172.21.15.57:0/643956128 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbadc0039a0 0x7fbaf4136b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:51.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.081+0000 7fbaf37fe640 1 -- 172.21.15.57:0/643956128 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbaf4153f10 msgr2=0x7fbaf41543a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:51.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.081+0000 7fbaf37fe640 1 --2- 172.21.15.57:0/643956128 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbaf4153f10 0x7fbaf41543a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:51.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.081+0000 7fbaf37fe640 1 -- 172.21.15.57:0/643956128 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaf41557a0 con 0x7fbaf41586a0 2023-10-19T14:21:51.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.081+0000 7fbaf37fe640 1 --2- 172.21.15.57:0/643956128 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbaf41586a0 0x7fbaf4158b20 secure :-1 s=READY pgs=2812 cs=0 l=1 rev1=1 crypto rx=0x7fbae4004ed0 tx=0x7fbae4001630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:51.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.081+0000 7fbaf0ff9640 1 -- 172.21.15.57:0/643956128 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbae4011870 con 0x7fbaf41586a0 2023-10-19T14:21:51.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.081+0000 7fbaf0ff9640 1 -- 172.21.15.57:0/643956128 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fbae4011ac0 con 0x7fbaf41586a0 2023-10-19T14:21:51.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.081+0000 7fbafa621640 1 -- 172.21.15.57:0/643956128 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbaf4154c20 con 0x7fbaf41586a0 2023-10-19T14:21:51.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.081+0000 7fbaf0ff9640 1 -- 172.21.15.57:0/643956128 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbae4011e70 con 0x7fbaf41586a0 2023-10-19T14:21:51.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.081+0000 7fbafa621640 1 -- 172.21.15.57:0/643956128 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbaf41551e0 con 0x7fbaf41586a0 2023-10-19T14:21:51.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.082+0000 7fbafa621640 1 -- 172.21.15.57:0/643956128 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbaf40690e0 con 0x7fbaf41586a0 2023-10-19T14:21:51.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.083+0000 7fbaf0ff9640 1 -- 172.21.15.57:0/643956128 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 145) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fbae401a400 con 0x7fbaf41586a0 2023-10-19T14:21:51.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.083+0000 7fbaf0ff9640 1 --2- 172.21.15.57:0/643956128 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fbae006b0b0 0x7fbae006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:51.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.083+0000 7fbaf0ff9640 1 -- 172.21.15.57:0/643956128 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 7015+0+0 (secure 0 0 0) 0x7fbae408acb0 con 0x7fbaf41586a0 2023-10-19T14:21:51.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.083+0000 7fbaf3fff640 1 --2- 172.21.15.57:0/643956128 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fbae006b0b0 0x7fbae006d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:51.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.084+0000 7fbaf3fff640 1 --2- 172.21.15.57:0/643956128 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fbae006b0b0 0x7fbae006d570 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fbae8034000 tx=0x7fbae802e000 comp rx=0 tx=0).ready entity=mgr.11446 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:51.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.087+0000 7fbaf0ff9640 1 -- 172.21.15.57:0/643956128 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fbae40555d0 con 0x7fbaf41586a0 2023-10-19T14:21:51.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.300+0000 7fbafa621640 1 -- 172.21.15.57:0/643956128 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "backup_fs"} v 0) v1 -- 0x7fbaf4160800 con 0x7fbaf41586a0 2023-10-19T14:21:51.304 INFO:tasks.ceph.mon.a.smithi057.stderr:2023-10-19T14:21:51.301+0000 7f1253ff3640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-19T14:21:51.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.306+0000 7fbaf0ff9640 1 -- 172.21.15.57:0/643956128 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "backup_fs"}]=0 backup_fs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v110) v1 ==== 167+0+0 (secure 0 0 0) 0x7fbae4058bf0 con 0x7fbaf41586a0 2023-10-19T14:21:51.310 INFO:teuthology.orchestra.run.smithi057.stderr:backup_fs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-19T14:21:51.312 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.309+0000 7fbafa621640 1 -- 172.21.15.57:0/643956128 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fbae006b0b0 msgr2=0x7fbae006d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:51.312 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.309+0000 7fbafa621640 1 --2- 172.21.15.57:0/643956128 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fbae006b0b0 0x7fbae006d570 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fbae8034000 tx=0x7fbae802e000 comp rx=0 tx=0).stop 2023-10-19T14:21:51.312 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.309+0000 7fbafa621640 1 -- 172.21.15.57:0/643956128 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbaf41586a0 msgr2=0x7fbaf4158b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:51.312 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.309+0000 7fbafa621640 1 --2- 172.21.15.57:0/643956128 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbaf41586a0 0x7fbaf4158b20 secure :-1 s=READY pgs=2812 cs=0 l=1 rev1=1 crypto rx=0x7fbae4004ed0 tx=0x7fbae4001630 comp rx=0 tx=0).stop 2023-10-19T14:21:51.313 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.309+0000 7fbafa621640 1 -- 172.21.15.57:0/643956128 shutdown_connections 2023-10-19T14:21:51.313 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.309+0000 7fbafa621640 1 --2- 172.21.15.57:0/643956128 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fbae006b0b0 0x7fbae006d570 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:51.313 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.309+0000 7fbafa621640 1 --2- 172.21.15.57:0/643956128 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbaf4153f10 0x7fbaf41543a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:51.313 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.309+0000 7fbafa621640 1 --2- 172.21.15.57:0/643956128 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbaf41586a0 0x7fbaf4158b20 unknown :-1 s=CLOSED pgs=2812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:51.313 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.309+0000 7fbafa621640 1 --2- 172.21.15.57:0/643956128 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbadc0039a0 0x7fbaf4136b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:51.314 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.309+0000 7fbafa621640 1 -- 172.21.15.57:0/643956128 >> 172.21.15.57:0/643956128 conn(0x7fbaf4069930 msgr2=0x7fbaf41570f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:51.314 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.309+0000 7fbafa621640 1 -- 172.21.15.57:0/643956128 shutdown_connections 2023-10-19T14:21:51.314 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.309+0000 7fbafa621640 1 -- 172.21.15.57:0/643956128 wait complete. 2023-10-19T14:21:51.314 INFO:tasks.ceph.mds.a.smithi057.stderr: -1431> 2023-10-19T14:21:31.377+0000 7f6f39e5d640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-19T14:21:51.324 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm backup_fs --yes-i-really-mean-it 2023-10-19T14:21:51.338 INFO:tasks.ceph.mds.a.smithi057.stdout:starting mds.a at 2023-10-19T14:21:51.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.434+0000 7f19fb984640 1 -- 172.21.15.57:0/2654428774 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f19f40d1910 msgr2=0x7f19f40b0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:51.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.434+0000 7f19fb984640 1 --2- 172.21.15.57:0/2654428774 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f19f40d1910 0x7f19f40b0650 secure :-1 s=READY pgs=2817 cs=0 l=1 rev1=1 crypto rx=0x7f19ec005fd0 tx=0x7f19ec02c0e0 comp rx=0 tx=0).stop 2023-10-19T14:21:51.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.434+0000 7f19fb984640 1 -- 172.21.15.57:0/2654428774 shutdown_connections 2023-10-19T14:21:51.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.434+0000 7f19fb984640 1 --2- 172.21.15.57:0/2654428774 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f19f40d1910 0x7f19f40b0650 unknown :-1 s=CLOSED pgs=2817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:51.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.434+0000 7f19fb984640 1 -- 172.21.15.57:0/2654428774 >> 172.21.15.57:0/2654428774 conn(0x7f19f40d0d60 msgr2=0x7f19f40d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:51.439 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.434+0000 7f19fb984640 1 -- 172.21.15.57:0/2654428774 shutdown_connections 2023-10-19T14:21:51.439 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.434+0000 7f19fb984640 1 -- 172.21.15.57:0/2654428774 wait complete. 2023-10-19T14:21:51.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.436+0000 7f19fb984640 1 Processor -- start 2023-10-19T14:21:51.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.436+0000 7f19fb984640 1 -- start start 2023-10-19T14:21:51.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.436+0000 7f19fb984640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f19f40cf340 0x7f19f40b1010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:51.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.436+0000 7f19fb984640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f19f40d1910 0x7f19f40b5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:51.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.436+0000 7f19fb984640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f19f415b2a0 0x7f19f40b5cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:51.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.436+0000 7f19fb984640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19f40c84d0 con 0x7f19f415b2a0 2023-10-19T14:21:51.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.436+0000 7f19fb984640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19f40b6210 con 0x7f19f40d1910 2023-10-19T14:21:51.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.436+0000 7f19fb984640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19f40c7e30 con 0x7f19f40cf340 2023-10-19T14:21:51.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.436+0000 7f19f9f21640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f19f415b2a0 0x7f19f40b5cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:51.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.436+0000 7f19f9f21640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f19f415b2a0 0x7f19f40b5cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34218/0 (socket says 172.21.15.57:34218) 2023-10-19T14:21:51.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.436+0000 7f19f9f21640 1 -- 172.21.15.57:0/3721159212 learned_addr learned my addr 172.21.15.57:0/3721159212 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:51.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.437+0000 7f19f8f1f640 1 --2- 172.21.15.57:0/3721159212 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f19f40d1910 0x7f19f40b5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:51.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.437+0000 7f19f9720640 1 --2- 172.21.15.57:0/3721159212 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f19f40cf340 0x7f19f40b1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:51.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.437+0000 7f19f9f21640 1 -- 172.21.15.57:0/3721159212 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f19f40cf340 msgr2=0x7f19f40b1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:51.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.437+0000 7f19f9f21640 1 --2- 172.21.15.57:0/3721159212 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f19f40cf340 0x7f19f40b1010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:51.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.437+0000 7f19f9f21640 1 -- 172.21.15.57:0/3721159212 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f19f40d1910 msgr2=0x7f19f40b5790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:51.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.437+0000 7f19f9f21640 1 --2- 172.21.15.57:0/3721159212 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f19f40d1910 0x7f19f40b5790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:51.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.437+0000 7f19f9f21640 1 -- 172.21.15.57:0/3721159212 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19ec009840 con 0x7f19f415b2a0 2023-10-19T14:21:51.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.437+0000 7f19f9f21640 1 --2- 172.21.15.57:0/3721159212 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f19f415b2a0 0x7f19f40b5cd0 secure :-1 s=READY pgs=2818 cs=0 l=1 rev1=1 crypto rx=0x7f19ec005fd0 tx=0x7f19ec004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:51.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.437+0000 7f19e9ffb640 1 -- 172.21.15.57:0/3721159212 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19ec004820 con 0x7f19f415b2a0 2023-10-19T14:21:51.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.437+0000 7f19e9ffb640 1 -- 172.21.15.57:0/3721159212 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f19ec0346b0 con 0x7f19f415b2a0 2023-10-19T14:21:51.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.437+0000 7f19fb984640 1 -- 172.21.15.57:0/3721159212 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19f40b64c0 con 0x7f19f415b2a0 2023-10-19T14:21:51.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.437+0000 7f19e9ffb640 1 -- 172.21.15.57:0/3721159212 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19ec034ac0 con 0x7f19f415b2a0 2023-10-19T14:21:51.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.437+0000 7f19fb984640 1 -- 172.21.15.57:0/3721159212 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19f40c80f0 con 0x7f19f415b2a0 2023-10-19T14:21:51.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.438+0000 7f19fb984640 1 -- 172.21.15.57:0/3721159212 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19f40c8610 con 0x7f19f415b2a0 2023-10-19T14:21:51.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.439+0000 7f19e9ffb640 1 -- 172.21.15.57:0/3721159212 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 145) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f19ec034c20 con 0x7f19f415b2a0 2023-10-19T14:21:51.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.439+0000 7f19e9ffb640 1 --2- 172.21.15.57:0/3721159212 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f19d806b080 0x7f19d806d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:51.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.439+0000 7f19e9ffb640 1 -- 172.21.15.57:0/3721159212 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(253..253 src has 1..253) v4 ==== 7101+0+0 (secure 0 0 0) 0x7f19ec0ad150 con 0x7f19f415b2a0 2023-10-19T14:21:51.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.439+0000 7f19f9720640 1 --2- 172.21.15.57:0/3721159212 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f19d806b080 0x7f19d806d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:51.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.440+0000 7f19f9720640 1 --2- 172.21.15.57:0/3721159212 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f19d806b080 0x7f19d806d540 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f19f0007fd0 tx=0x7f19f00013f0 comp rx=0 tx=0).ready entity=mgr.11446 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:51.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.444+0000 7f19e9ffb640 1 -- 172.21.15.57:0/3721159212 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f19f40c8610 con 0x7f19f415b2a0 2023-10-19T14:21:51.662 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:51.659+0000 7f19fb984640 1 -- 172.21.15.57:0/3721159212 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "backup_fs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f19f40c8610 con 0x7f19f415b2a0 2023-10-19T14:21:52.321 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:52.318+0000 7f19e9ffb640 1 -- 172.21.15.57:0/3721159212 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "backup_fs", "yes_i_really_mean_it": true}]=0 v111) v1 ==== 107+0+0 (secure 0 0 0) 0x7f19f40c8610 con 0x7f19f415b2a0 2023-10-19T14:21:52.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:52.321+0000 7f19fb984640 1 -- 172.21.15.57:0/3721159212 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f19d806b080 msgr2=0x7f19d806d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:52.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:52.321+0000 7f19fb984640 1 --2- 172.21.15.57:0/3721159212 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f19d806b080 0x7f19d806d540 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f19f0007fd0 tx=0x7f19f00013f0 comp rx=0 tx=0).stop 2023-10-19T14:21:52.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:52.321+0000 7f19fb984640 1 -- 172.21.15.57:0/3721159212 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f19f415b2a0 msgr2=0x7f19f40b5cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:52.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:52.321+0000 7f19fb984640 1 --2- 172.21.15.57:0/3721159212 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f19f415b2a0 0x7f19f40b5cd0 secure :-1 s=READY pgs=2818 cs=0 l=1 rev1=1 crypto rx=0x7f19ec005fd0 tx=0x7f19ec004360 comp rx=0 tx=0).stop 2023-10-19T14:21:52.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:52.321+0000 7f19fb984640 1 -- 172.21.15.57:0/3721159212 shutdown_connections 2023-10-19T14:21:52.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:52.321+0000 7f19fb984640 1 --2- 172.21.15.57:0/3721159212 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f19d806b080 0x7f19d806d540 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:52.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:52.321+0000 7f19fb984640 1 --2- 172.21.15.57:0/3721159212 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f19f415b2a0 0x7f19f40b5cd0 unknown :-1 s=CLOSED pgs=2818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:52.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:52.321+0000 7f19fb984640 1 --2- 172.21.15.57:0/3721159212 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f19f40d1910 0x7f19f40b5790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:52.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:52.321+0000 7f19fb984640 1 --2- 172.21.15.57:0/3721159212 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f19f40cf340 0x7f19f40b1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:52.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:52.321+0000 7f19fb984640 1 -- 172.21.15.57:0/3721159212 >> 172.21.15.57:0/3721159212 conn(0x7f19f40d0d60 msgr2=0x7f19f40ad8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:52.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:52.321+0000 7f19fb984640 1 -- 172.21.15.57:0/3721159212 shutdown_connections 2023-10-19T14:21:52.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:52.321+0000 7f19fb984640 1 -- 172.21.15.57:0/3721159212 wait complete. 2023-10-19T14:21:52.338 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm backup_fs_metadata backup_fs_metadata --yes-i-really-really-mean-it 2023-10-19T14:21:52.499 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:52.496+0000 7faa6ac07640 1 -- 172.21.15.57:0/1307895400 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa6415b2a0 msgr2=0x7faa640d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:52.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:52.496+0000 7faa6ac07640 1 --2- 172.21.15.57:0/1307895400 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa6415b2a0 0x7faa640d1910 secure :-1 s=READY pgs=2819 cs=0 l=1 rev1=1 crypto rx=0x7faa4c005fd0 tx=0x7faa4c02bd90 comp rx=0 tx=0).stop 2023-10-19T14:21:52.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:52.496+0000 7faa6ac07640 1 -- 172.21.15.57:0/1307895400 shutdown_connections 2023-10-19T14:21:52.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:52.496+0000 7faa6ac07640 1 --2- 172.21.15.57:0/1307895400 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7faa640ceb40 0x7faa640b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:52.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:52.496+0000 7faa6ac07640 1 --2- 172.21.15.57:0/1307895400 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa6415b2a0 0x7faa640d1910 unknown :-1 s=CLOSED pgs=2819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:52.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:52.496+0000 7faa6ac07640 1 -- 172.21.15.57:0/1307895400 >> 172.21.15.57:0/1307895400 conn(0x7faa640d0d60 msgr2=0x7faa640d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:52.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:52.496+0000 7faa6ac07640 1 -- 172.21.15.57:0/1307895400 shutdown_connections 2023-10-19T14:21:52.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:52.496+0000 7faa6ac07640 1 -- 172.21.15.57:0/1307895400 wait complete. 2023-10-19T14:21:52.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:52.497+0000 7faa6ac07640 1 Processor -- start 2023-10-19T14:21:52.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:52.497+0000 7faa6ac07640 1 -- start start 2023-10-19T14:21:52.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:52.498+0000 7faa6ac07640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faa640ceb40 0x7faa6414fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:52.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:52.498+0000 7faa6ac07640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa640cf340 0x7faa640caa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:52.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:52.498+0000 7faa6ac07640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7faa6415b2a0 0x7faa6415d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:52.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:52.498+0000 7faa6ac07640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa640b6500 con 0x7faa640cf340 2023-10-19T14:21:52.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:52.498+0000 7faa6ac07640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa641500b0 con 0x7faa640ceb40 2023-10-19T14:21:52.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:52.498+0000 7faa6ac07640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa640caf60 con 0x7faa6415b2a0 2023-10-19T14:21:52.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:52.498+0000 7faa63fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa640cf340 0x7faa640caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:52.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:52.498+0000 7faa63fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa640cf340 0x7faa640caa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34240/0 (socket says 172.21.15.57:34240) 2023-10-19T14:21:52.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:52.498+0000 7faa63fff640 1 -- 172.21.15.57:0/1038914613 learned_addr learned my addr 172.21.15.57:0/1038914613 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:52.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:52.499+0000 7faa689a3640 1 --2- 172.21.15.57:0/1038914613 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faa640ceb40 0x7faa6414fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:52.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:52.499+0000 7faa691a4640 1 --2- 172.21.15.57:0/1038914613 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7faa6415b2a0 0x7faa6415d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:52.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:52.499+0000 7faa63fff640 1 -- 172.21.15.57:0/1038914613 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7faa6415b2a0 msgr2=0x7faa6415d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:52.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:52.499+0000 7faa63fff640 1 --2- 172.21.15.57:0/1038914613 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7faa6415b2a0 0x7faa6415d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:52.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:52.499+0000 7faa63fff640 1 -- 172.21.15.57:0/1038914613 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faa640ceb40 msgr2=0x7faa6414fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:52.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:52.499+0000 7faa63fff640 1 --2- 172.21.15.57:0/1038914613 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faa640ceb40 0x7faa6414fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:52.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:52.499+0000 7faa63fff640 1 -- 172.21.15.57:0/1038914613 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa4c009560 con 0x7faa640cf340 2023-10-19T14:21:52.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:52.499+0000 7faa63fff640 1 --2- 172.21.15.57:0/1038914613 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa640cf340 0x7faa640caa20 secure :-1 s=READY pgs=2820 cs=0 l=1 rev1=1 crypto rx=0x7faa4c0099f0 tx=0x7faa4c02db30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:52.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:52.499+0000 7faa617fa640 1 -- 172.21.15.57:0/1038914613 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa4c003800 con 0x7faa640cf340 2023-10-19T14:21:52.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:52.499+0000 7faa617fa640 1 -- 172.21.15.57:0/1038914613 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7faa4c003a50 con 0x7faa640cf340 2023-10-19T14:21:52.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:52.499+0000 7faa6ac07640 1 -- 172.21.15.57:0/1038914613 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa640b0d50 con 0x7faa640cf340 2023-10-19T14:21:52.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:52.499+0000 7faa617fa640 1 -- 172.21.15.57:0/1038914613 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa4c003e60 con 0x7faa640cf340 2023-10-19T14:21:52.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:52.499+0000 7faa6ac07640 1 -- 172.21.15.57:0/1038914613 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa640c9e70 con 0x7faa640cf340 2023-10-19T14:21:52.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:52.500+0000 7faa6ac07640 1 -- 172.21.15.57:0/1038914613 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa640b38e0 con 0x7faa640cf340 2023-10-19T14:21:52.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:52.501+0000 7faa617fa640 1 -- 172.21.15.57:0/1038914613 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 145) v1 ==== 89433+0+0 (secure 0 0 0) 0x7faa4c03e450 con 0x7faa640cf340 2023-10-19T14:21:52.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:52.501+0000 7faa617fa640 1 --2- 172.21.15.57:0/1038914613 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7faa4406b080 0x7faa4406d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:52.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:52.501+0000 7faa617fa640 1 -- 172.21.15.57:0/1038914613 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(253..253 src has 1..253) v4 ==== 7101+0+0 (secure 0 0 0) 0x7faa4c0ac8c0 con 0x7faa640cf340 2023-10-19T14:21:52.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:52.501+0000 7faa689a3640 1 --2- 172.21.15.57:0/1038914613 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7faa4406b080 0x7faa4406d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:52.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:52.502+0000 7faa689a3640 1 --2- 172.21.15.57:0/1038914613 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7faa4406b080 0x7faa4406d540 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7faa5c001f10 tx=0x7faa5c009040 comp rx=0 tx=0).ready entity=mgr.11446 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:52.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:52.505+0000 7faa617fa640 1 -- 172.21.15.57:0/1038914613 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7faa640b38e0 con 0x7faa640cf340 2023-10-19T14:21:52.685 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:52.682+0000 7faa6ac07640 1 -- 172.21.15.57:0/1038914613 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "backup_fs_metadata", "pool2": "backup_fs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7faa640b38e0 con 0x7faa640cf340 2023-10-19T14:21:53.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:53.320+0000 7faa617fa640 1 -- 172.21.15.57:0/1038914613 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "backup_fs_metadata", "pool2": "backup_fs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'backup_fs_metadata' removed v254) v1 ==== 190+0+0 (secure 0 0 0) 0x7faa640b38e0 con 0x7faa640cf340 2023-10-19T14:21:53.324 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'backup_fs_metadata' removed 2023-10-19T14:21:53.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:53.323+0000 7faa6ac07640 1 -- 172.21.15.57:0/1038914613 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7faa4406b080 msgr2=0x7faa4406d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:53.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:53.323+0000 7faa6ac07640 1 --2- 172.21.15.57:0/1038914613 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7faa4406b080 0x7faa4406d540 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7faa5c001f10 tx=0x7faa5c009040 comp rx=0 tx=0).stop 2023-10-19T14:21:53.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:53.323+0000 7faa6ac07640 1 -- 172.21.15.57:0/1038914613 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa640cf340 msgr2=0x7faa640caa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:53.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:53.323+0000 7faa6ac07640 1 --2- 172.21.15.57:0/1038914613 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa640cf340 0x7faa640caa20 secure :-1 s=READY pgs=2820 cs=0 l=1 rev1=1 crypto rx=0x7faa4c0099f0 tx=0x7faa4c02db30 comp rx=0 tx=0).stop 2023-10-19T14:21:53.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:53.323+0000 7faa6ac07640 1 -- 172.21.15.57:0/1038914613 shutdown_connections 2023-10-19T14:21:53.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:53.323+0000 7faa6ac07640 1 --2- 172.21.15.57:0/1038914613 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7faa4406b080 0x7faa4406d540 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:53.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:53.323+0000 7faa6ac07640 1 --2- 172.21.15.57:0/1038914613 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7faa6415b2a0 0x7faa6415d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:53.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:53.323+0000 7faa6ac07640 1 --2- 172.21.15.57:0/1038914613 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa640cf340 0x7faa640caa20 unknown :-1 s=CLOSED pgs=2820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:53.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:53.323+0000 7faa6ac07640 1 --2- 172.21.15.57:0/1038914613 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faa640ceb40 0x7faa6414fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:53.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:53.323+0000 7faa6ac07640 1 -- 172.21.15.57:0/1038914613 >> 172.21.15.57:0/1038914613 conn(0x7faa640d0d60 msgr2=0x7faa640cca00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:53.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:53.323+0000 7faa6ac07640 1 -- 172.21.15.57:0/1038914613 shutdown_connections 2023-10-19T14:21:53.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:53.323+0000 7faa6ac07640 1 -- 172.21.15.57:0/1038914613 wait complete. 2023-10-19T14:21:53.342 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm backup_fs_data backup_fs_data --yes-i-really-really-mean-it 2023-10-19T14:21:53.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:53.464+0000 7f2ec0a07640 1 -- 172.21.15.57:0/885754410 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2ea8003ef0 msgr2=0x7f2ea80047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:53.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:53.464+0000 7f2ec0a07640 1 --2- 172.21.15.57:0/885754410 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2ea8003ef0 0x7f2ea80047c0 secure :-1 s=READY pgs=2822 cs=0 l=1 rev1=1 crypto rx=0x7f2ea4009b50 tx=0x7f2ea80057f0 comp rx=0 tx=0).stop 2023-10-19T14:21:53.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:53.464+0000 7f2ebad9d640 1 -- 172.21.15.57:0/885754410 reap_dead start 2023-10-19T14:21:53.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:53.464+0000 7f2ec0a07640 1 -- 172.21.15.57:0/885754410 shutdown_connections 2023-10-19T14:21:53.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:53.464+0000 7f2ec0a07640 1 -- 172.21.15.57:0/885754410 >> 172.21.15.57:0/885754410 conn(0x7f2ebc0d0cf0 msgr2=0x7f2ebc0d1100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:53.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:53.464+0000 7f2ec0a07640 1 -- 172.21.15.57:0/885754410 shutdown_connections 2023-10-19T14:21:53.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:53.464+0000 7f2ec0a07640 1 -- 172.21.15.57:0/885754410 wait complete. 2023-10-19T14:21:53.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:53.465+0000 7f2ec0a07640 1 Processor -- start 2023-10-19T14:21:53.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:53.465+0000 7f2ec0a07640 1 -- start start 2023-10-19T14:21:53.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:53.466+0000 7f2ec0a07640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2ea8003ef0 0x7f2ebc157db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:53.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:53.466+0000 7f2ec0a07640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2ebc15b890 0x7f2ebc0cf260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:53.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:53.466+0000 7f2ec0a07640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2ebc0cf7a0 0x7f2ebc0c9240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:53.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:53.466+0000 7f2ec0a07640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ebc0cfc20 con 0x7f2ebc0cf7a0 2023-10-19T14:21:53.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:53.466+0000 7f2ec0a07640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ebc0b50f0 con 0x7f2ea8003ef0 2023-10-19T14:21:53.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:53.466+0000 7f2ec0a07640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ebc0b5340 con 0x7f2ebc15b890 2023-10-19T14:21:53.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:53.466+0000 7f2ebad9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2ebc0cf7a0 0x7f2ebc0c9240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:53.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:53.466+0000 7f2ebad9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2ebc0cf7a0 0x7f2ebc0c9240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34260/0 (socket says 172.21.15.57:34260) 2023-10-19T14:21:53.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:53.466+0000 7f2ebad9d640 1 -- 172.21.15.57:0/3956762078 learned_addr learned my addr 172.21.15.57:0/3956762078 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:53.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:53.466+0000 7f2eb1d9b640 1 --2- 172.21.15.57:0/3956762078 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2ebc15b890 0x7f2ebc0cf260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:53.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:53.466+0000 7f2eba59c640 1 --2- 172.21.15.57:0/3956762078 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2ea8003ef0 0x7f2ebc157db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:53.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:53.467+0000 7f2ebad9d640 1 -- 172.21.15.57:0/3956762078 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2ebc15b890 msgr2=0x7f2ebc0cf260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:53.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:53.467+0000 7f2ebad9d640 1 --2- 172.21.15.57:0/3956762078 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2ebc15b890 0x7f2ebc0cf260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:53.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:53.467+0000 7f2ebad9d640 1 -- 172.21.15.57:0/3956762078 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2ea8003ef0 msgr2=0x7f2ebc157db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:53.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:53.467+0000 7f2ebad9d640 1 --2- 172.21.15.57:0/3956762078 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2ea8003ef0 0x7f2ebc157db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:53.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:53.467+0000 7f2ebad9d640 1 -- 172.21.15.57:0/3956762078 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ebc0b5620 con 0x7f2ebc0cf7a0 2023-10-19T14:21:53.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:53.467+0000 7f2ebad9d640 1 --2- 172.21.15.57:0/3956762078 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2ebc0cf7a0 0x7f2ebc0c9240 secure :-1 s=READY pgs=2823 cs=0 l=1 rev1=1 crypto rx=0x7f2eb4000f80 tx=0x7f2eb400ebc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:53.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:53.467+0000 7f2eb37fe640 1 -- 172.21.15.57:0/3956762078 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eb400c990 con 0x7f2ebc0cf7a0 2023-10-19T14:21:53.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:53.467+0000 7f2eb37fe640 1 -- 172.21.15.57:0/3956762078 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f2eb400cc00 con 0x7f2ebc0cf7a0 2023-10-19T14:21:53.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:53.467+0000 7f2ec0a07640 1 -- 172.21.15.57:0/3956762078 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ebc0b5900 con 0x7f2ebc0cf7a0 2023-10-19T14:21:53.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:53.467+0000 7f2eb37fe640 1 -- 172.21.15.57:0/3956762078 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eb4011460 con 0x7f2ebc0cf7a0 2023-10-19T14:21:53.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:53.467+0000 7f2ec0a07640 1 -- 172.21.15.57:0/3956762078 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ebc15bd10 con 0x7f2ebc0cf7a0 2023-10-19T14:21:53.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:53.468+0000 7f2ec0a07640 1 -- 172.21.15.57:0/3956762078 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ebc0c7e50 con 0x7f2ebc0cf7a0 2023-10-19T14:21:53.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:53.469+0000 7f2eb37fe640 1 -- 172.21.15.57:0/3956762078 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 145) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f2eb40115c0 con 0x7f2ebc0cf7a0 2023-10-19T14:21:53.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:53.469+0000 7f2eb37fe640 1 --2- 172.21.15.57:0/3956762078 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f2e9c06b080 0x7f2e9c06d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:53.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:53.470+0000 7f2eb37fe640 1 -- 172.21.15.57:0/3956762078 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(254..254 src has 1..254) v4 ==== 6657+0+0 (secure 0 0 0) 0x7f2eb408a630 con 0x7f2ebc0cf7a0 2023-10-19T14:21:53.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:53.470+0000 7f2eba59c640 1 --2- 172.21.15.57:0/3956762078 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f2e9c06b080 0x7f2e9c06d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:53.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:53.470+0000 7f2eba59c640 1 --2- 172.21.15.57:0/3956762078 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f2e9c06b080 0x7f2e9c06d540 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2eac009cd0 tx=0x7f2eac005f90 comp rx=0 tx=0).ready entity=mgr.11446 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:53.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:53.474+0000 7f2eb37fe640 1 -- 172.21.15.57:0/3956762078 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2ebc0c7e50 con 0x7f2ebc0cf7a0 2023-10-19T14:21:53.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:53.651+0000 7f2ec0a07640 1 -- 172.21.15.57:0/3956762078 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "backup_fs_data", "pool2": "backup_fs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f2ebc0c7e50 con 0x7f2ebc0cf7a0 2023-10-19T14:21:54.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.322+0000 7f2eb37fe640 1 -- 172.21.15.57:0/3956762078 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "backup_fs_data", "pool2": "backup_fs_data", "yes_i_really_really_mean_it": true}]=0 pool 'backup_fs_data' removed v255) v1 ==== 178+0+0 (secure 0 0 0) 0x7f2ebc0c7e50 con 0x7f2ebc0cf7a0 2023-10-19T14:21:54.325 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'backup_fs_data' removed 2023-10-19T14:21:54.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.324+0000 7f2ec0a07640 1 -- 172.21.15.57:0/3956762078 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f2e9c06b080 msgr2=0x7f2e9c06d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:54.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.324+0000 7f2ec0a07640 1 --2- 172.21.15.57:0/3956762078 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f2e9c06b080 0x7f2e9c06d540 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2eac009cd0 tx=0x7f2eac005f90 comp rx=0 tx=0).stop 2023-10-19T14:21:54.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.324+0000 7f2ec0a07640 1 -- 172.21.15.57:0/3956762078 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2ebc0cf7a0 msgr2=0x7f2ebc0c9240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:54.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.324+0000 7f2ec0a07640 1 --2- 172.21.15.57:0/3956762078 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2ebc0cf7a0 0x7f2ebc0c9240 secure :-1 s=READY pgs=2823 cs=0 l=1 rev1=1 crypto rx=0x7f2eb4000f80 tx=0x7f2eb400ebc0 comp rx=0 tx=0).stop 2023-10-19T14:21:54.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.324+0000 7f2ec0a07640 1 -- 172.21.15.57:0/3956762078 shutdown_connections 2023-10-19T14:21:54.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.324+0000 7f2ec0a07640 1 --2- 172.21.15.57:0/3956762078 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f2e9c06b080 0x7f2e9c06d540 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:54.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.324+0000 7f2ec0a07640 1 --2- 172.21.15.57:0/3956762078 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2ebc0cf7a0 0x7f2ebc0c9240 unknown :-1 s=CLOSED pgs=2823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:54.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.324+0000 7f2ec0a07640 1 --2- 172.21.15.57:0/3956762078 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2ebc15b890 0x7f2ebc0cf260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:54.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.324+0000 7f2ec0a07640 1 --2- 172.21.15.57:0/3956762078 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2ea8003ef0 0x7f2ebc157db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:54.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.324+0000 7f2ec0a07640 1 -- 172.21.15.57:0/3956762078 >> 172.21.15.57:0/3956762078 conn(0x7f2ebc0d0cf0 msgr2=0x7f2ebc0c66f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:54.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.324+0000 7f2ec0a07640 1 -- 172.21.15.57:0/3956762078 shutdown_connections 2023-10-19T14:21:54.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.324+0000 7f2ec0a07640 1 -- 172.21.15.57:0/3956762078 wait complete. 2023-10-19T14:21:54.347 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 30 2023-10-19T14:21:54.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.460+0000 7f50729c9640 1 -- 172.21.15.57:0/2100241182 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f506c1579f0 msgr2=0x7f506c0b0e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:54.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.460+0000 7f50729c9640 1 --2- 172.21.15.57:0/2100241182 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f506c1579f0 0x7f506c0b0e90 secure :-1 s=READY pgs=2824 cs=0 l=1 rev1=1 crypto rx=0x7f506c06b930 tx=0x7f505c02bda0 comp rx=0 tx=0).stop 2023-10-19T14:21:54.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.460+0000 7f50729c9640 1 -- 172.21.15.57:0/2100241182 shutdown_connections 2023-10-19T14:21:54.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.460+0000 7f50729c9640 1 --2- 172.21.15.57:0/2100241182 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f506c06ac70 0x7f506c06ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:54.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.460+0000 7f50729c9640 1 --2- 172.21.15.57:0/2100241182 >> v2:172.21.15.179:3300/0 conn(0x7f506c15c1f0 0x7f506c06a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:54.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.460+0000 7f50729c9640 1 --2- 172.21.15.57:0/2100241182 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f506c1579f0 0x7f506c0b0e90 unknown :-1 s=CLOSED pgs=2824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:54.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.460+0000 7f50729c9640 1 -- 172.21.15.57:0/2100241182 >> 172.21.15.57:0/2100241182 conn(0x7f506c06fae0 msgr2=0x7f506c06fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:54.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.460+0000 7f50729c9640 1 -- 172.21.15.57:0/2100241182 shutdown_connections 2023-10-19T14:21:54.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.460+0000 7f50729c9640 1 -- 172.21.15.57:0/2100241182 wait complete. 2023-10-19T14:21:54.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.461+0000 7f50729c9640 1 Processor -- start 2023-10-19T14:21:54.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.461+0000 7f50729c9640 1 -- start start 2023-10-19T14:21:54.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.462+0000 7f50729c9640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f506c06ac70 0x7f506c156960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:54.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.462+0000 7f50729c9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f506c1579f0 0x7f506c13f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:54.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.462+0000 7f50729c9640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f506c15c1f0 0x7f506c153120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:54.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.462+0000 7f50729c9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f506c0ae700 con 0x7f506c1579f0 2023-10-19T14:21:54.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.462+0000 7f50729c9640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f506c153fe0 con 0x7f506c06ac70 2023-10-19T14:21:54.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.462+0000 7f50729c9640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f506c154280 con 0x7f506c15c1f0 2023-10-19T14:21:54.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.462+0000 7f506b7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f506c1579f0 0x7f506c13f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:54.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.462+0000 7f506b7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f506c1579f0 0x7f506c13f3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34268/0 (socket says 172.21.15.57:34268) 2023-10-19T14:21:54.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.462+0000 7f506b7fe640 1 -- 172.21.15.57:0/3652287455 learned_addr learned my addr 172.21.15.57:0/3652287455 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:54.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.462+0000 7f506bfff640 1 --2- 172.21.15.57:0/3652287455 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f506c06ac70 0x7f506c156960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:54.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.462+0000 7f5070f66640 1 --2- 172.21.15.57:0/3652287455 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f506c15c1f0 0x7f506c153120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:54.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.463+0000 7f506b7fe640 1 -- 172.21.15.57:0/3652287455 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f506c15c1f0 msgr2=0x7f506c153120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:54.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.463+0000 7f506b7fe640 1 --2- 172.21.15.57:0/3652287455 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f506c15c1f0 0x7f506c153120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:54.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.463+0000 7f506b7fe640 1 -- 172.21.15.57:0/3652287455 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f506c06ac70 msgr2=0x7f506c156960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:54.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.463+0000 7f506b7fe640 1 --2- 172.21.15.57:0/3652287455 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f506c06ac70 0x7f506c156960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:54.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.463+0000 7f506b7fe640 1 -- 172.21.15.57:0/3652287455 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f505c009560 con 0x7f506c1579f0 2023-10-19T14:21:54.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.463+0000 7f506bfff640 1 --2- 172.21.15.57:0/3652287455 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f506c06ac70 0x7f506c156960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:21:54.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.463+0000 7f5070f66640 1 --2- 172.21.15.57:0/3652287455 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f506c15c1f0 0x7f506c153120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:21:54.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.463+0000 7f506b7fe640 1 --2- 172.21.15.57:0/3652287455 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f506c1579f0 0x7f506c13f3b0 secure :-1 s=READY pgs=2825 cs=0 l=1 rev1=1 crypto rx=0x7f506000e790 tx=0x7f506000ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:54.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.463+0000 7f5068ff9640 1 -- 172.21.15.57:0/3652287455 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5060009580 con 0x7f506c1579f0 2023-10-19T14:21:54.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.463+0000 7f5068ff9640 1 -- 172.21.15.57:0/3652287455 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f506000cdb0 con 0x7f506c1579f0 2023-10-19T14:21:54.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.463+0000 7f50729c9640 1 -- 172.21.15.57:0/3652287455 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f506c154560 con 0x7f506c1579f0 2023-10-19T14:21:54.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.463+0000 7f5068ff9640 1 -- 172.21.15.57:0/3652287455 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5060010690 con 0x7f506c1579f0 2023-10-19T14:21:54.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.463+0000 7f50729c9640 1 -- 172.21.15.57:0/3652287455 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f506c13f8f0 con 0x7f506c1579f0 2023-10-19T14:21:54.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.464+0000 7f50729c9640 1 -- 172.21.15.57:0/3652287455 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f506c0690e0 con 0x7f506c1579f0 2023-10-19T14:21:54.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.464+0000 7f5068ff9640 1 -- 172.21.15.57:0/3652287455 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 145) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f50600107f0 con 0x7f506c1579f0 2023-10-19T14:21:54.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.465+0000 7f5068ff9640 1 --2- 172.21.15.57:0/3652287455 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f505806b080 0x7f505806d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:54.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.465+0000 7f5068ff9640 1 -- 172.21.15.57:0/3652287455 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(255..255 src has 1..255) v4 ==== 6237+0+0 (secure 0 0 0) 0x7f5060014070 con 0x7f506c1579f0 2023-10-19T14:21:54.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.465+0000 7f506bfff640 1 --2- 172.21.15.57:0/3652287455 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f505806b080 0x7f505806d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:54.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.466+0000 7f506bfff640 1 --2- 172.21.15.57:0/3652287455 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f505806b080 0x7f505806d540 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f505c035000 tx=0x7f505c004730 comp rx=0 tx=0).ready entity=mgr.11446 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:54.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.469+0000 7f5068ff9640 1 -- 172.21.15.57:0/3652287455 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f5060051ca0 con 0x7f506c1579f0 2023-10-19T14:21:54.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.648+0000 7f50729c9640 1 -- 172.21.15.57:0/3652287455 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "config reset", "num": 30} v 0) v1 -- 0x7f506c070d70 con 0x7f506c1579f0 2023-10-19T14:21:54.658 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.656+0000 7f5068ff9640 1 -- 172.21.15.57:0/3652287455 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 30}]=0 v82) v1 ==== 71+0+0 (secure 0 0 0) 0x7f50600551a0 con 0x7f506c1579f0 2023-10-19T14:21:54.661 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.658+0000 7f50729c9640 1 -- 172.21.15.57:0/3652287455 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f505806b080 msgr2=0x7f505806d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:54.661 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.658+0000 7f50729c9640 1 --2- 172.21.15.57:0/3652287455 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f505806b080 0x7f505806d540 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f505c035000 tx=0x7f505c004730 comp rx=0 tx=0).stop 2023-10-19T14:21:54.662 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.658+0000 7f50729c9640 1 -- 172.21.15.57:0/3652287455 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f506c1579f0 msgr2=0x7f506c13f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:54.662 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.658+0000 7f50729c9640 1 --2- 172.21.15.57:0/3652287455 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f506c1579f0 0x7f506c13f3b0 secure :-1 s=READY pgs=2825 cs=0 l=1 rev1=1 crypto rx=0x7f506000e790 tx=0x7f506000ec50 comp rx=0 tx=0).stop 2023-10-19T14:21:54.662 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.658+0000 7f50729c9640 1 -- 172.21.15.57:0/3652287455 shutdown_connections 2023-10-19T14:21:54.662 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.658+0000 7f50729c9640 1 --2- 172.21.15.57:0/3652287455 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f505806b080 0x7f505806d540 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:54.662 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.658+0000 7f50729c9640 1 --2- 172.21.15.57:0/3652287455 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f506c15c1f0 0x7f506c153120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:54.663 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.658+0000 7f50729c9640 1 --2- 172.21.15.57:0/3652287455 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f506c1579f0 0x7f506c13f3b0 unknown :-1 s=CLOSED pgs=2825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:54.663 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.658+0000 7f50729c9640 1 --2- 172.21.15.57:0/3652287455 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f506c06ac70 0x7f506c156960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:54.663 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.658+0000 7f50729c9640 1 -- 172.21.15.57:0/3652287455 >> 172.21.15.57:0/3652287455 conn(0x7f506c06fae0 msgr2=0x7f506c14c1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:54.663 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.658+0000 7f50729c9640 1 -- 172.21.15.57:0/3652287455 shutdown_connections 2023-10-19T14:21:54.663 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.658+0000 7f50729c9640 1 -- 172.21.15.57:0/3652287455 wait complete. 2023-10-19T14:21:54.671 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument' 2023-10-19T14:21:54.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.778+0000 7fd5dabf0640 1 -- 172.21.15.57:0/1488407355 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5d40ab500 msgr2=0x7fd5d40ab990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:54.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.778+0000 7fd5dabf0640 1 --2- 172.21.15.57:0/1488407355 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5d40ab500 0x7fd5d40ab990 secure :-1 s=READY pgs=2826 cs=0 l=1 rev1=1 crypto rx=0x7fd5d40cbac0 tx=0x7fd5cc02c250 comp rx=0 tx=0).stop 2023-10-19T14:21:54.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.778+0000 7fd5dabf0640 1 -- 172.21.15.57:0/1488407355 shutdown_connections 2023-10-19T14:21:54.781 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.778+0000 7fd5dabf0640 1 --2- 172.21.15.57:0/1488407355 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5d40ab500 0x7fd5d40ab990 unknown :-1 s=CLOSED pgs=2826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:54.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.778+0000 7fd5dabf0640 1 --2- 172.21.15.57:0/1488407355 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd5d40c96b0 0x7fd5d40c9b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:54.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.778+0000 7fd5dabf0640 1 --2- 172.21.15.57:0/1488407355 >> v2:172.21.15.179:3300/0 conn(0x7fd5d406af00 0x7fd5d406b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:54.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.778+0000 7fd5dabf0640 1 -- 172.21.15.57:0/1488407355 >> 172.21.15.57:0/1488407355 conn(0x7fd5d4069930 msgr2=0x7fd5d4069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:54.782 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.778+0000 7fd5dabf0640 1 -- 172.21.15.57:0/1488407355 shutdown_connections 2023-10-19T14:21:54.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.778+0000 7fd5dabf0640 1 -- 172.21.15.57:0/1488407355 wait complete. 2023-10-19T14:21:54.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.779+0000 7fd5dabf0640 1 Processor -- start 2023-10-19T14:21:54.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.779+0000 7fd5dabf0640 1 -- start start 2023-10-19T14:21:54.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.780+0000 7fd5dabf0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5d406af00 0x7fd5d4151d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:54.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.780+0000 7fd5dabf0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd5d40ab500 0x7fd5d40b0950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:54.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.780+0000 7fd5dabf0640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd5d40c96b0 0x7fd5d40b0e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:54.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.780+0000 7fd5dabf0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5d40b0070 con 0x7fd5d406af00 2023-10-19T14:21:54.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.780+0000 7fd5dabf0640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5d413aac0 con 0x7fd5d40ab500 2023-10-19T14:21:54.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.780+0000 7fd5dabf0640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5d40af4f0 con 0x7fd5d40c96b0 2023-10-19T14:21:54.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.780+0000 7fd5d898c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5d406af00 0x7fd5d4151d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:54.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.780+0000 7fd5d898c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5d406af00 0x7fd5d4151d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34278/0 (socket says 172.21.15.57:34278) 2023-10-19T14:21:54.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.780+0000 7fd5d898c640 1 -- 172.21.15.57:0/3218219582 learned_addr learned my addr 172.21.15.57:0/3218219582 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:54.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.780+0000 7fd5d918d640 1 --2- 172.21.15.57:0/3218219582 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd5d40c96b0 0x7fd5d40b0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:54.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.780+0000 7fd5cbfff640 1 --2- 172.21.15.57:0/3218219582 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd5d40ab500 0x7fd5d40b0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:54.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.780+0000 7fd5d898c640 1 -- 172.21.15.57:0/3218219582 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd5d40c96b0 msgr2=0x7fd5d40b0e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:54.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.780+0000 7fd5d898c640 1 --2- 172.21.15.57:0/3218219582 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd5d40c96b0 0x7fd5d40b0e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:54.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.780+0000 7fd5d898c640 1 -- 172.21.15.57:0/3218219582 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd5d40ab500 msgr2=0x7fd5d40b0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:54.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.780+0000 7fd5d898c640 1 --2- 172.21.15.57:0/3218219582 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd5d40ab500 0x7fd5d40b0950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:54.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.780+0000 7fd5d898c640 1 -- 172.21.15.57:0/3218219582 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5cc009900 con 0x7fd5d406af00 2023-10-19T14:21:54.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.780+0000 7fd5d918d640 1 --2- 172.21.15.57:0/3218219582 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd5d40c96b0 0x7fd5d40b0e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:21:54.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.780+0000 7fd5d898c640 1 --2- 172.21.15.57:0/3218219582 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5d406af00 0x7fd5d4151d00 secure :-1 s=READY pgs=2827 cs=0 l=1 rev1=1 crypto rx=0x7fd5d000c890 tx=0x7fd5d000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:54.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.781+0000 7fd5c97fa640 1 -- 172.21.15.57:0/3218219582 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5d0004220 con 0x7fd5d406af00 2023-10-19T14:21:54.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.781+0000 7fd5c97fa640 1 -- 172.21.15.57:0/3218219582 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fd5d0004470 con 0x7fd5d406af00 2023-10-19T14:21:54.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.781+0000 7fd5dabf0640 1 -- 172.21.15.57:0/3218219582 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5d40af7d0 con 0x7fd5d406af00 2023-10-19T14:21:54.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.781+0000 7fd5c97fa640 1 -- 172.21.15.57:0/3218219582 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5d0007c10 con 0x7fd5d406af00 2023-10-19T14:21:54.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.781+0000 7fd5dabf0640 1 -- 172.21.15.57:0/3218219582 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5d40afd90 con 0x7fd5d406af00 2023-10-19T14:21:54.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.782+0000 7fd5dabf0640 1 -- 172.21.15.57:0/3218219582 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5d4153920 con 0x7fd5d406af00 2023-10-19T14:21:54.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.782+0000 7fd5c97fa640 1 -- 172.21.15.57:0/3218219582 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 145) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fd5d0005020 con 0x7fd5d406af00 2023-10-19T14:21:54.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.783+0000 7fd5c97fa640 1 --2- 172.21.15.57:0/3218219582 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fd5c006b0b0 0x7fd5c006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:54.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.783+0000 7fd5c97fa640 1 -- 172.21.15.57:0/3218219582 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(255..255 src has 1..255) v4 ==== 6237+0+0 (secure 0 0 0) 0x7fd5d008a290 con 0x7fd5d406af00 2023-10-19T14:21:54.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.783+0000 7fd5cbfff640 1 --2- 172.21.15.57:0/3218219582 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fd5c006b0b0 0x7fd5c006d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:54.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.784+0000 7fd5cbfff640 1 --2- 172.21.15.57:0/3218219582 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fd5c006b0b0 0x7fd5c006d570 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd5d4134ea0 tx=0x7fd5c4008040 comp rx=0 tx=0).ready entity=mgr.11446 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:54.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:54.789+0000 7fd5c97fa640 1 -- 172.21.15.57:0/3218219582 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd5d0054cc0 con 0x7fd5d406af00 2023-10-19T14:21:55.035 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.033+0000 7fd5dabf0640 1 -- 172.21.15.57:0/3218219582 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument"]} v 0) v1 -- 0x7fd5d40690e0 con 0x7fd5d406af00 2023-10-19T14:21:55.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.338+0000 7fd5c97fa640 1 -- 172.21.15.57:0/3218219582 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument"]}]=0 v3217) v1 ==== 187+0+0 (secure 0 0 0) 0x7fd5d001d100 con 0x7fd5d406af00 2023-10-19T14:21:55.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.339+0000 7fd5dabf0640 1 -- 172.21.15.57:0/3218219582 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fd5c006b0b0 msgr2=0x7fd5c006d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:55.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.339+0000 7fd5dabf0640 1 --2- 172.21.15.57:0/3218219582 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fd5c006b0b0 0x7fd5c006d570 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd5d4134ea0 tx=0x7fd5c4008040 comp rx=0 tx=0).stop 2023-10-19T14:21:55.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.339+0000 7fd5dabf0640 1 -- 172.21.15.57:0/3218219582 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5d406af00 msgr2=0x7fd5d4151d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:55.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.339+0000 7fd5dabf0640 1 --2- 172.21.15.57:0/3218219582 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5d406af00 0x7fd5d4151d00 secure :-1 s=READY pgs=2827 cs=0 l=1 rev1=1 crypto rx=0x7fd5d000c890 tx=0x7fd5d000cd50 comp rx=0 tx=0).stop 2023-10-19T14:21:55.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.340+0000 7fd5dabf0640 1 -- 172.21.15.57:0/3218219582 shutdown_connections 2023-10-19T14:21:55.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.340+0000 7fd5dabf0640 1 --2- 172.21.15.57:0/3218219582 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fd5c006b0b0 0x7fd5c006d570 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:55.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.340+0000 7fd5dabf0640 1 --2- 172.21.15.57:0/3218219582 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd5d40c96b0 0x7fd5d40b0e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:55.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.340+0000 7fd5dabf0640 1 --2- 172.21.15.57:0/3218219582 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd5d40ab500 0x7fd5d40b0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:55.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.340+0000 7fd5dabf0640 1 --2- 172.21.15.57:0/3218219582 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5d406af00 0x7fd5d4151d00 unknown :-1 s=CLOSED pgs=2827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:55.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.340+0000 7fd5dabf0640 1 -- 172.21.15.57:0/3218219582 >> 172.21.15.57:0/3218219582 conn(0x7fd5d4069930 msgr2=0x7fd5d4069d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:55.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.340+0000 7fd5dabf0640 1 -- 172.21.15.57:0/3218219582 shutdown_connections 2023-10-19T14:21:55.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.340+0000 7fd5dabf0640 1 -- 172.21.15.57:0/3218219582 wait complete. 2023-10-19T14:21:55.354 INFO:tasks.cephfs_test_runner:Test command fails without --fs argument in presence of multiple fs ... ok 2023-10-19T14:21:55.355 INFO:tasks.cephfs_test_runner:Starting test: test_snap_dir_name (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir) Test the correctness of snap directory name 2023-10-19T14:21:55.355 INFO:tasks.cephfs_test_runner:test_snap_dir_name (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir) 2023-10-19T14:21:55.356 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T14:21:55.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.464+0000 7fd9cdda9640 1 -- 172.21.15.57:0/1885412878 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd9c815ccb0 msgr2=0x7fd9c80c9750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:55.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.464+0000 7fd9cdda9640 1 --2- 172.21.15.57:0/1885412878 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd9c815ccb0 0x7fd9c80c9750 secure :-1 s=READY pgs=2828 cs=0 l=1 rev1=1 crypto rx=0x7fd9c80cc8a0 tx=0x7fd9c002c1d0 comp rx=0 tx=0).stop 2023-10-19T14:21:55.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.464+0000 7fd9cdda9640 1 -- 172.21.15.57:0/1885412878 shutdown_connections 2023-10-19T14:21:55.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.464+0000 7fd9cdda9640 1 --2- 172.21.15.57:0/1885412878 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd9c815ccb0 0x7fd9c80c9750 unknown :-1 s=CLOSED pgs=2828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:55.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.464+0000 7fd9cdda9640 1 --2- 172.21.15.57:0/1885412878 >> v2:172.21.15.179:3300/0 conn(0x7fd9c806b410 0x7fd9c815c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:55.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.464+0000 7fd9cdda9640 1 -- 172.21.15.57:0/1885412878 >> 172.21.15.57:0/1885412878 conn(0x7fd9c8069930 msgr2=0x7fd9c8069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:55.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.464+0000 7fd9cdda9640 1 -- 172.21.15.57:0/1885412878 shutdown_connections 2023-10-19T14:21:55.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.464+0000 7fd9cdda9640 1 -- 172.21.15.57:0/1885412878 wait complete. 2023-10-19T14:21:55.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.465+0000 7fd9cdda9640 1 Processor -- start 2023-10-19T14:21:55.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.465+0000 7fd9cdda9640 1 -- start start 2023-10-19T14:21:55.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.466+0000 7fd9cdda9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd9c806b030 0x7fd9c813d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:55.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.466+0000 7fd9cdda9640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd9c806b410 0x7fd9c813a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:55.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.466+0000 7fd9cdda9640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd9c815ccb0 0x7fd9c80b0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:55.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.466+0000 7fd9cdda9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9c80b1110 con 0x7fd9c806b030 2023-10-19T14:21:55.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.466+0000 7fd9cdda9640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9c8070130 con 0x7fd9c806b410 2023-10-19T14:21:55.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.466+0000 7fd9cdda9640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9c8070410 con 0x7fd9c815ccb0 2023-10-19T14:21:55.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.466+0000 7fd9c77fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd9c806b030 0x7fd9c813d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:55.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.466+0000 7fd9c77fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd9c806b030 0x7fd9c813d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34292/0 (socket says 172.21.15.57:34292) 2023-10-19T14:21:55.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.466+0000 7fd9c77fe640 1 -- 172.21.15.57:0/3475107307 learned_addr learned my addr 172.21.15.57:0/3475107307 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:55.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.466+0000 7fd9c7fff640 1 --2- 172.21.15.57:0/3475107307 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd9c815ccb0 0x7fd9c80b0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:55.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.466+0000 7fd9c6ffd640 1 --2- 172.21.15.57:0/3475107307 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd9c806b410 0x7fd9c813a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:55.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.466+0000 7fd9c77fe640 1 -- 172.21.15.57:0/3475107307 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd9c815ccb0 msgr2=0x7fd9c80b0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:55.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.466+0000 7fd9c77fe640 1 --2- 172.21.15.57:0/3475107307 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd9c815ccb0 0x7fd9c80b0bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:55.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.466+0000 7fd9c77fe640 1 -- 172.21.15.57:0/3475107307 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd9c806b410 msgr2=0x7fd9c813a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:55.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.466+0000 7fd9c77fe640 1 --2- 172.21.15.57:0/3475107307 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd9c806b410 0x7fd9c813a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:55.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.466+0000 7fd9c77fe640 1 -- 172.21.15.57:0/3475107307 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9c0009900 con 0x7fd9c806b030 2023-10-19T14:21:55.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.466+0000 7fd9c77fe640 1 --2- 172.21.15.57:0/3475107307 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd9c806b030 0x7fd9c813d3d0 secure :-1 s=READY pgs=2829 cs=0 l=1 rev1=1 crypto rx=0x7fd9b800b3f0 tx=0x7fd9b800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:55.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.467+0000 7fd9bffff640 1 -- 172.21.15.57:0/3475107307 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9b8011830 con 0x7fd9c806b030 2023-10-19T14:21:55.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.467+0000 7fd9bffff640 1 -- 172.21.15.57:0/3475107307 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fd9b8011a80 con 0x7fd9c806b030 2023-10-19T14:21:55.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.467+0000 7fd9cdda9640 1 -- 172.21.15.57:0/3475107307 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9c80706f0 con 0x7fd9c806b030 2023-10-19T14:21:55.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.467+0000 7fd9bffff640 1 -- 172.21.15.57:0/3475107307 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9b8011e90 con 0x7fd9c806b030 2023-10-19T14:21:55.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.467+0000 7fd9cdda9640 1 -- 172.21.15.57:0/3475107307 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9c8070cb0 con 0x7fd9c806b030 2023-10-19T14:21:55.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.468+0000 7fd9cdda9640 1 -- 172.21.15.57:0/3475107307 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9c80690e0 con 0x7fd9c806b030 2023-10-19T14:21:55.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.468+0000 7fd9bffff640 1 -- 172.21.15.57:0/3475107307 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 145) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fd9b801a400 con 0x7fd9c806b030 2023-10-19T14:21:55.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.468+0000 7fd9bffff640 1 --2- 172.21.15.57:0/3475107307 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fd9a806b080 0x7fd9a806d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:55.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.468+0000 7fd9bffff640 1 -- 172.21.15.57:0/3475107307 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(255..255 src has 1..255) v4 ==== 6237+0+0 (secure 0 0 0) 0x7fd9b8089bd0 con 0x7fd9c806b030 2023-10-19T14:21:55.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.469+0000 7fd9c6ffd640 1 --2- 172.21.15.57:0/3475107307 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fd9a806b080 0x7fd9a806d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:55.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.469+0000 7fd9c6ffd640 1 --2- 172.21.15.57:0/3475107307 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fd9a806b080 0x7fd9a806d540 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd9b0005fd0 tx=0x7fd9b000b390 comp rx=0 tx=0).ready entity=mgr.11446 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:55.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.473+0000 7fd9bffff640 1 -- 172.21.15.57:0/3475107307 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd9b80547c0 con 0x7fd9c806b030 2023-10-19T14:21:55.650 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.647+0000 7fd9cdda9640 1 -- 172.21.15.57:0/3475107307 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd9c80cbae0 con 0x7fd9c806b030 2023-10-19T14:21:55.651 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.648+0000 7fd9bffff640 1 -- 172.21.15.57:0/3475107307 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v255) v1 ==== 74+0+14646 (secure 0 0 0) 0x7fd9b8015090 con 0x7fd9c806b030 2023-10-19T14:21:55.651 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:21:55.652 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":255,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T14:21:54.321536+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.57:6834/18187246":"2023-10-20T14:21:44.967770+0000","172.21.15.179:0/3646431735":"2023-10-20T14:21:41.395725+0000","172.21.15.179:0/2986772961":"2023-10-20T14:21:41.395725+0000","172.21.15.179:0/837564123":"2023-10-20T14:21:41.395725+0000","172.21.15.57:6833/4256498777":"2023-10-20T14:21:51.302022+0000","172.21.15.179:0/1917522846":"2023-10-20T14:21:41.395725+0000","172.21.15.179:0/3420876591":"2023-10-20T14:21:41.395725+0000","172.21.15.179:6833/609071913":"2023-10-20T14:21:41.395725+0000","172.21.15.57:6832/4256498777":"2023-10-20T14:21:51.302022+0000","172.21.15.179:6832/609071913":"2023-10-20T14:21:41.395725+0000","172.21.15.179:0/2525508070":"2023-10-20T14:21:35.927420+0000","172.21.15.179:0/75322738":"2023-10-20T14:21:41.395725+0000","172.21.15.179:0/720769249":"2023-10-20T14:21:35.927420+0000","172.21.15.57:6835/18187246":"2023-10-20T14:21:44.967770+0000","172.21.15.179:6833/3547895240":"2023-10-20T14:21:35.927420+0000","172.21.15.179:6832/3547895240":"2023-10-20T14:21:35.927420+0000","172.21.15.179:0/177563067":"2023-10-20T14:21:35.927420+0000","172.21.15.179:0/1656904674":"2023-10-20T14:21:41.395725+0000","172.21.15.179:0/647327435":"2023-10-20T14:21:41.395725+0000","172.21.15.179:0/1877726906":"2023-10-20T14:21:35.927420+0000","172.21.15.179:0/3471744208":"2023-10-20T14:21:35.927420+0000","172.21.15.179:0/2477290155":"2023-10-20T14:21:35.927420+0000","172.21.15.179:0/1240052109":"2023-10-20T14:21:35.927420+0000","172.21.15.179:0/4042925688":"2023-10-20T14:21:35.927420+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T14:21:55.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.650+0000 7fd9cdda9640 1 -- 172.21.15.57:0/3475107307 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fd9a806b080 msgr2=0x7fd9a806d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:55.653 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.650+0000 7fd9cdda9640 1 --2- 172.21.15.57:0/3475107307 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fd9a806b080 0x7fd9a806d540 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd9b0005fd0 tx=0x7fd9b000b390 comp rx=0 tx=0).stop 2023-10-19T14:21:55.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.650+0000 7fd9cdda9640 1 -- 172.21.15.57:0/3475107307 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd9c806b030 msgr2=0x7fd9c813d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:55.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.650+0000 7fd9cdda9640 1 --2- 172.21.15.57:0/3475107307 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd9c806b030 0x7fd9c813d3d0 secure :-1 s=READY pgs=2829 cs=0 l=1 rev1=1 crypto rx=0x7fd9b800b3f0 tx=0x7fd9b800b8b0 comp rx=0 tx=0).stop 2023-10-19T14:21:55.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.650+0000 7fd9cdda9640 1 -- 172.21.15.57:0/3475107307 shutdown_connections 2023-10-19T14:21:55.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.650+0000 7fd9cdda9640 1 --2- 172.21.15.57:0/3475107307 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fd9a806b080 0x7fd9a806d540 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:55.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.650+0000 7fd9cdda9640 1 --2- 172.21.15.57:0/3475107307 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd9c815ccb0 0x7fd9c80b0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:55.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.650+0000 7fd9cdda9640 1 --2- 172.21.15.57:0/3475107307 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd9c806b410 0x7fd9c813a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:55.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.650+0000 7fd9cdda9640 1 --2- 172.21.15.57:0/3475107307 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd9c806b030 0x7fd9c813d3d0 unknown :-1 s=CLOSED pgs=2829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:55.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.650+0000 7fd9cdda9640 1 -- 172.21.15.57:0/3475107307 >> 172.21.15.57:0/3475107307 conn(0x7fd9c8069930 msgr2=0x7fd9c815bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:55.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.650+0000 7fd9cdda9640 1 -- 172.21.15.57:0/3475107307 shutdown_connections 2023-10-19T14:21:55.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.650+0000 7fd9cdda9640 1 -- 172.21.15.57:0/3475107307 wait complete. 2023-10-19T14:21:55.664 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-19T12:44:07.371186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-19T14:21:55.665 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-19T14:21:55.778 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.775+0000 7faf24783640 1 -- 172.21.15.57:0/3979996659 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faf08004a80 msgr2=0x7faf08004ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:55.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.775+0000 7faf24783640 1 --2- 172.21.15.57:0/3979996659 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faf08004a80 0x7faf08004ea0 secure :-1 s=READY pgs=2831 cs=0 l=1 rev1=1 crypto rx=0x7faf1800bb50 tx=0x7faf08006900 comp rx=0 tx=0).stop 2023-10-19T14:21:55.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.775+0000 7faf22d20640 1 -- 172.21.15.57:0/3979996659 reap_dead start 2023-10-19T14:21:55.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.775+0000 7faf24783640 1 -- 172.21.15.57:0/3979996659 shutdown_connections 2023-10-19T14:21:55.779 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.775+0000 7faf24783640 1 -- 172.21.15.57:0/3979996659 >> 172.21.15.57:0/3979996659 conn(0x7faf1c069930 msgr2=0x7faf1c069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:55.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.775+0000 7faf24783640 1 -- 172.21.15.57:0/3979996659 shutdown_connections 2023-10-19T14:21:55.780 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.775+0000 7faf24783640 1 -- 172.21.15.57:0/3979996659 wait complete. 2023-10-19T14:21:55.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.776+0000 7faf24783640 1 Processor -- start 2023-10-19T14:21:55.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.776+0000 7faf24783640 1 -- start start 2023-10-19T14:21:55.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.777+0000 7faf24783640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7faf08004a80 0x7faf1c0abd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:55.783 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.777+0000 7faf24783640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faf1c1e50a0 0x7faf1c1e5520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:55.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.777+0000 7faf24783640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faf1c153ec0 0x7faf1c154340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:55.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.777+0000 7faf24783640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf1c1f17f0 con 0x7faf1c153ec0 2023-10-19T14:21:55.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.777+0000 7faf24783640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf1c156cc0 con 0x7faf1c1e50a0 2023-10-19T14:21:55.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.777+0000 7faf24783640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf1c1e5a60 con 0x7faf08004a80 2023-10-19T14:21:55.784 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.777+0000 7faf22d20640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faf1c153ec0 0x7faf1c154340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:55.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.777+0000 7faf22d20640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faf1c153ec0 0x7faf1c154340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34318/0 (socket says 172.21.15.57:34318) 2023-10-19T14:21:55.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.777+0000 7faf22d20640 1 -- 172.21.15.57:0/1705154327 learned_addr learned my addr 172.21.15.57:0/1705154327 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:55.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.777+0000 7faf21d1e640 1 --2- 172.21.15.57:0/1705154327 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faf1c1e50a0 0x7faf1c1e5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:55.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.777+0000 7faf2251f640 1 --2- 172.21.15.57:0/1705154327 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7faf08004a80 0x7faf1c0abd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:55.785 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.778+0000 7faf22d20640 1 -- 172.21.15.57:0/1705154327 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7faf08004a80 msgr2=0x7faf1c0abd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:55.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.778+0000 7faf22d20640 1 --2- 172.21.15.57:0/1705154327 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7faf08004a80 0x7faf1c0abd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:55.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.778+0000 7faf22d20640 1 -- 172.21.15.57:0/1705154327 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faf1c1e50a0 msgr2=0x7faf1c1e5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:55.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.778+0000 7faf22d20640 1 --2- 172.21.15.57:0/1705154327 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faf1c1e50a0 0x7faf1c1e5520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:55.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.778+0000 7faf22d20640 1 -- 172.21.15.57:0/1705154327 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf1c154b90 con 0x7faf1c153ec0 2023-10-19T14:21:55.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.778+0000 7faf2251f640 1 --2- 172.21.15.57:0/1705154327 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7faf08004a80 0x7faf1c0abd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:21:55.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.778+0000 7faf21d1e640 1 --2- 172.21.15.57:0/1705154327 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faf1c1e50a0 0x7faf1c1e5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:21:55.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.778+0000 7faf22d20640 1 --2- 172.21.15.57:0/1705154327 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faf1c153ec0 0x7faf1c154340 secure :-1 s=READY pgs=2832 cs=0 l=1 rev1=1 crypto rx=0x7faf14001330 tx=0x7faf1400ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:55.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.778+0000 7faf12ffd640 1 -- 172.21.15.57:0/1705154327 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf1400cba0 con 0x7faf1c153ec0 2023-10-19T14:21:55.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.778+0000 7faf12ffd640 1 -- 172.21.15.57:0/1705154327 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7faf1400ce10 con 0x7faf1c153ec0 2023-10-19T14:21:55.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.778+0000 7faf24783640 1 -- 172.21.15.57:0/1705154327 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf1c154e70 con 0x7faf1c153ec0 2023-10-19T14:21:55.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.778+0000 7faf12ffd640 1 -- 172.21.15.57:0/1705154327 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf140116c0 con 0x7faf1c153ec0 2023-10-19T14:21:55.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.778+0000 7faf24783640 1 -- 172.21.15.57:0/1705154327 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf1c155430 con 0x7faf1c153ec0 2023-10-19T14:21:55.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.779+0000 7faf24783640 1 -- 172.21.15.57:0/1705154327 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf1c1f1930 con 0x7faf1c153ec0 2023-10-19T14:21:55.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.780+0000 7faf12ffd640 1 -- 172.21.15.57:0/1705154327 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 145) v1 ==== 89433+0+0 (secure 0 0 0) 0x7faf14011820 con 0x7faf1c153ec0 2023-10-19T14:21:55.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.780+0000 7faf12ffd640 1 --2- 172.21.15.57:0/1705154327 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7faf0006b080 0x7faf0006d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:55.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.780+0000 7faf12ffd640 1 -- 172.21.15.57:0/1705154327 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(255..255 src has 1..255) v4 ==== 6237+0+0 (secure 0 0 0) 0x7faf1408a550 con 0x7faf1c153ec0 2023-10-19T14:21:55.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.780+0000 7faf2251f640 1 --2- 172.21.15.57:0/1705154327 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7faf0006b080 0x7faf0006d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:55.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.781+0000 7faf2251f640 1 --2- 172.21.15.57:0/1705154327 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7faf0006b080 0x7faf0006d540 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7faf18037000 tx=0x7faf1802e040 comp rx=0 tx=0).ready entity=mgr.11446 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:55.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.784+0000 7faf12ffd640 1 -- 172.21.15.57:0/1705154327 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7faf1c1f1930 con 0x7faf1c153ec0 2023-10-19T14:21:55.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.958+0000 7faf24783640 1 -- 172.21.15.57:0/1705154327 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7faf1c1f1930 con 0x7faf1c153ec0 2023-10-19T14:21:55.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.959+0000 7faf12ffd640 1 -- 172.21.15.57:0/1705154327 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v255) v1 ==== 93+0+10 (secure 0 0 0) 0x7faf1c1f1930 con 0x7faf1c153ec0 2023-10-19T14:21:55.962 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-19T14:21:55.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.960+0000 7faf24783640 1 -- 172.21.15.57:0/1705154327 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7faf0006b080 msgr2=0x7faf0006d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:55.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.960+0000 7faf24783640 1 --2- 172.21.15.57:0/1705154327 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7faf0006b080 0x7faf0006d540 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7faf18037000 tx=0x7faf1802e040 comp rx=0 tx=0).stop 2023-10-19T14:21:55.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.961+0000 7faf24783640 1 -- 172.21.15.57:0/1705154327 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faf1c153ec0 msgr2=0x7faf1c154340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:55.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.961+0000 7faf24783640 1 --2- 172.21.15.57:0/1705154327 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faf1c153ec0 0x7faf1c154340 secure :-1 s=READY pgs=2832 cs=0 l=1 rev1=1 crypto rx=0x7faf14001330 tx=0x7faf1400ec70 comp rx=0 tx=0).stop 2023-10-19T14:21:55.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.961+0000 7faf24783640 1 -- 172.21.15.57:0/1705154327 shutdown_connections 2023-10-19T14:21:55.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.961+0000 7faf24783640 1 --2- 172.21.15.57:0/1705154327 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7faf0006b080 0x7faf0006d540 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:55.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.961+0000 7faf24783640 1 --2- 172.21.15.57:0/1705154327 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faf1c153ec0 0x7faf1c154340 unknown :-1 s=CLOSED pgs=2832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:55.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.961+0000 7faf24783640 1 --2- 172.21.15.57:0/1705154327 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faf1c1e50a0 0x7faf1c1e5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:55.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.961+0000 7faf24783640 1 --2- 172.21.15.57:0/1705154327 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7faf08004a80 0x7faf1c0abd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:55.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.961+0000 7faf24783640 1 -- 172.21.15.57:0/1705154327 >> 172.21.15.57:0/1705154327 conn(0x7faf1c069930 msgr2=0x7faf1c0ae9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:55.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.961+0000 7faf24783640 1 -- 172.21.15.57:0/1705154327 shutdown_connections 2023-10-19T14:21:55.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:55.961+0000 7faf24783640 1 -- 172.21.15.57:0/1705154327 wait complete. 2023-10-19T14:21:55.974 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name' 2023-10-19T14:21:56.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.081+0000 7f20daed7640 1 -- 172.21.15.57:0/228471899 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f20d4070920 msgr2=0x7f20d40affe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:56.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.081+0000 7f20daed7640 1 --2- 172.21.15.57:0/228471899 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f20d4070920 0x7f20d40affe0 secure :-1 s=READY pgs=2834 cs=0 l=1 rev1=1 crypto rx=0x7f20cc005fd0 tx=0x7f20cc02c050 comp rx=0 tx=0).stop 2023-10-19T14:21:56.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.081+0000 7f20daed7640 1 -- 172.21.15.57:0/228471899 shutdown_connections 2023-10-19T14:21:56.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.081+0000 7f20daed7640 1 --2- 172.21.15.57:0/228471899 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f20d4141690 0x7f20d40703e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:56.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.081+0000 7f20daed7640 1 --2- 172.21.15.57:0/228471899 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f20d4070920 0x7f20d40affe0 unknown :-1 s=CLOSED pgs=2834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:56.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.081+0000 7f20daed7640 1 -- 172.21.15.57:0/228471899 >> 172.21.15.57:0/228471899 conn(0x7f20d4069930 msgr2=0x7f20d4069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:56.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.081+0000 7f20daed7640 1 -- 172.21.15.57:0/228471899 shutdown_connections 2023-10-19T14:21:56.085 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.081+0000 7f20daed7640 1 -- 172.21.15.57:0/228471899 wait complete. 2023-10-19T14:21:56.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.082+0000 7f20daed7640 1 Processor -- start 2023-10-19T14:21:56.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.082+0000 7f20daed7640 1 -- start start 2023-10-19T14:21:56.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.083+0000 7f20daed7640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f20d4070920 0x7f20d41dd540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:56.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.083+0000 7f20daed7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f20d413e520 0x7f20d413a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:56.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.083+0000 7f20daed7640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f20d4141690 0x7f20d41dbb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:56.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.083+0000 7f20daed7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20d41dd0b0 con 0x7f20d413e520 2023-10-19T14:21:56.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.083+0000 7f20daed7640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20d41dda80 con 0x7f20d4141690 2023-10-19T14:21:56.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.083+0000 7f20daed7640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20d413acc0 con 0x7f20d4070920 2023-10-19T14:21:56.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.083+0000 7f20d3fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f20d413e520 0x7f20d413a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:56.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.083+0000 7f20d3fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f20d413e520 0x7f20d413a780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34344/0 (socket says 172.21.15.57:34344) 2023-10-19T14:21:56.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.083+0000 7f20d3fff640 1 -- 172.21.15.57:0/2792486521 learned_addr learned my addr 172.21.15.57:0/2792486521 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:56.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.083+0000 7f20d3fff640 1 -- 172.21.15.57:0/2792486521 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f20d4070920 msgr2=0x7f20d41dd540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:56.090 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.083+0000 7f20d3fff640 1 --2- 172.21.15.57:0/2792486521 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f20d4070920 0x7f20d41dd540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:56.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.083+0000 7f20d3fff640 1 -- 172.21.15.57:0/2792486521 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f20d4141690 msgr2=0x7f20d41dbb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:56.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.083+0000 7f20d3fff640 1 --2- 172.21.15.57:0/2792486521 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f20d4141690 0x7f20d41dbb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:56.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.083+0000 7f20d3fff640 1 -- 172.21.15.57:0/2792486521 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20cc009840 con 0x7f20d413e520 2023-10-19T14:21:56.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.083+0000 7f20d3fff640 1 --2- 172.21.15.57:0/2792486521 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f20d413e520 0x7f20d413a780 secure :-1 s=READY pgs=2835 cs=0 l=1 rev1=1 crypto rx=0x7f20c000e7c0 tx=0x7f20c000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:56.091 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.083+0000 7f20d17fa640 1 -- 172.21.15.57:0/2792486521 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20c00095a0 con 0x7f20d413e520 2023-10-19T14:21:56.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.084+0000 7f20d17fa640 1 -- 172.21.15.57:0/2792486521 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f20c000eed0 con 0x7f20d413e520 2023-10-19T14:21:56.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.084+0000 7f20daed7640 1 -- 172.21.15.57:0/2792486521 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20d41dc4a0 con 0x7f20d413e520 2023-10-19T14:21:56.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.084+0000 7f20d17fa640 1 -- 172.21.15.57:0/2792486521 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20c0005020 con 0x7f20d413e520 2023-10-19T14:21:56.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.084+0000 7f20daed7640 1 -- 172.21.15.57:0/2792486521 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20d41dca60 con 0x7f20d413e520 2023-10-19T14:21:56.092 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.085+0000 7f20daed7640 1 -- 172.21.15.57:0/2792486521 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20d40690e0 con 0x7f20d413e520 2023-10-19T14:21:56.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.085+0000 7f20d17fa640 1 -- 172.21.15.57:0/2792486521 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 145) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f20c0005180 con 0x7f20d413e520 2023-10-19T14:21:56.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.085+0000 7f20d17fa640 1 --2- 172.21.15.57:0/2792486521 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f20b806b080 0x7f20b806d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:56.093 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.086+0000 7f20d17fa640 1 -- 172.21.15.57:0/2792486521 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(255..255 src has 1..255) v4 ==== 6237+0+0 (secure 0 0 0) 0x7f20c0014070 con 0x7f20d413e520 2023-10-19T14:21:56.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.086+0000 7f20d8c73640 1 --2- 172.21.15.57:0/2792486521 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f20b806b080 0x7f20b806d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:56.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.086+0000 7f20d8c73640 1 --2- 172.21.15.57:0/2792486521 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f20b806b080 0x7f20b806d540 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f20d4070540 tx=0x7f20c8007910 comp rx=0 tx=0).ready entity=mgr.11446 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:56.094 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.090+0000 7f20d17fa640 1 -- 172.21.15.57:0/2792486521 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f20c0058c00 con 0x7f20d413e520 2023-10-19T14:21:56.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.330+0000 7f20daed7640 1 -- 172.21.15.57:0/2792486521 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name"]} v 0) v1 -- 0x7f20d41dd1f0 con 0x7f20d413e520 2023-10-19T14:21:56.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.349+0000 7f20d17fa640 1 -- 172.21.15.57:0/2792486521 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name"]}]=0 v3218) v1 ==== 158+0+0 (secure 0 0 0) 0x7f20c0058790 con 0x7f20d413e520 2023-10-19T14:21:56.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.351+0000 7f20daed7640 1 -- 172.21.15.57:0/2792486521 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f20b806b080 msgr2=0x7f20b806d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:56.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.351+0000 7f20daed7640 1 --2- 172.21.15.57:0/2792486521 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f20b806b080 0x7f20b806d540 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f20d4070540 tx=0x7f20c8007910 comp rx=0 tx=0).stop 2023-10-19T14:21:56.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.351+0000 7f20daed7640 1 -- 172.21.15.57:0/2792486521 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f20d413e520 msgr2=0x7f20d413a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:56.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.351+0000 7f20daed7640 1 --2- 172.21.15.57:0/2792486521 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f20d413e520 0x7f20d413a780 secure :-1 s=READY pgs=2835 cs=0 l=1 rev1=1 crypto rx=0x7f20c000e7c0 tx=0x7f20c000ec80 comp rx=0 tx=0).stop 2023-10-19T14:21:56.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.351+0000 7f20daed7640 1 -- 172.21.15.57:0/2792486521 shutdown_connections 2023-10-19T14:21:56.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.351+0000 7f20daed7640 1 --2- 172.21.15.57:0/2792486521 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f20b806b080 0x7f20b806d540 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:56.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.351+0000 7f20daed7640 1 --2- 172.21.15.57:0/2792486521 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f20d4141690 0x7f20d41dbb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:56.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.351+0000 7f20daed7640 1 --2- 172.21.15.57:0/2792486521 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f20d413e520 0x7f20d413a780 unknown :-1 s=CLOSED pgs=2835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:56.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.351+0000 7f20daed7640 1 --2- 172.21.15.57:0/2792486521 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f20d4070920 0x7f20d41dd540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:56.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.351+0000 7f20daed7640 1 -- 172.21.15.57:0/2792486521 >> 172.21.15.57:0/2792486521 conn(0x7f20d4069930 msgr2=0x7f20d41e41a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:56.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.351+0000 7f20daed7640 1 -- 172.21.15.57:0/2792486521 shutdown_connections 2023-10-19T14:21:56.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.351+0000 7f20daed7640 1 -- 172.21.15.57:0/2792486521 wait complete. 2023-10-19T14:21:56.364 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2023-10-19T14:21:56.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.471+0000 7f696aee5640 1 -- 172.21.15.57:0/1904033310 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f69640cf340 msgr2=0x7f69640d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:56.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.471+0000 7f696aee5640 1 --2- 172.21.15.57:0/1904033310 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f69640cf340 0x7f69640d1910 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f695c005fd0 tx=0x7f695c02c240 comp rx=0 tx=0).stop 2023-10-19T14:21:56.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.471+0000 7f696aee5640 1 -- 172.21.15.57:0/1904033310 shutdown_connections 2023-10-19T14:21:56.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.471+0000 7f696aee5640 1 --2- 172.21.15.57:0/1904033310 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f696415b2a0 0x7f69640cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:56.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.471+0000 7f696aee5640 1 --2- 172.21.15.57:0/1904033310 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f69640cf340 0x7f69640d1910 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:56.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.471+0000 7f696aee5640 1 -- 172.21.15.57:0/1904033310 >> 172.21.15.57:0/1904033310 conn(0x7f69640d0d60 msgr2=0x7f69640d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:56.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.471+0000 7f696aee5640 1 -- 172.21.15.57:0/1904033310 shutdown_connections 2023-10-19T14:21:56.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.471+0000 7f696aee5640 1 -- 172.21.15.57:0/1904033310 wait complete. 2023-10-19T14:21:56.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.473+0000 7f696aee5640 1 Processor -- start 2023-10-19T14:21:56.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.473+0000 7f696aee5640 1 -- start start 2023-10-19T14:21:56.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.473+0000 7f696aee5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f69640ceb40 0x7f69640b5110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:56.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.473+0000 7f696aee5640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f69640cf340 0x7f696409e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:56.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.473+0000 7f696aee5640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f696415b2a0 0x7f6964158370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:56.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.473+0000 7f696aee5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f696409eaf0 con 0x7f69640ceb40 2023-10-19T14:21:56.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.473+0000 7f696aee5640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69640b4c80 con 0x7f696415b2a0 2023-10-19T14:21:56.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.473+0000 7f696aee5640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69640b4390 con 0x7f69640cf340 2023-10-19T14:21:56.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.474+0000 7f6968c81640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f69640ceb40 0x7f69640b5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:56.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.474+0000 7f6968c81640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f69640ceb40 0x7f69640b5110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34356/0 (socket says 172.21.15.57:34356) 2023-10-19T14:21:56.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.474+0000 7f6968c81640 1 -- 172.21.15.57:0/298486600 learned_addr learned my addr 172.21.15.57:0/298486600 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:56.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.474+0000 7f6963fff640 1 --2- 172.21.15.57:0/298486600 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f69640cf340 0x7f696409e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:56.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.474+0000 7f6969482640 1 --2- 172.21.15.57:0/298486600 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f696415b2a0 0x7f6964158370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:56.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.474+0000 7f6968c81640 1 -- 172.21.15.57:0/298486600 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f69640cf340 msgr2=0x7f696409e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:56.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.474+0000 7f6968c81640 1 --2- 172.21.15.57:0/298486600 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f69640cf340 0x7f696409e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:56.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.474+0000 7f6968c81640 1 -- 172.21.15.57:0/298486600 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f696415b2a0 msgr2=0x7f6964158370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:56.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.474+0000 7f6968c81640 1 --2- 172.21.15.57:0/298486600 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f696415b2a0 0x7f6964158370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:56.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.474+0000 7f6968c81640 1 -- 172.21.15.57:0/298486600 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6954009490 con 0x7f69640ceb40 2023-10-19T14:21:56.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.474+0000 7f6968c81640 1 --2- 172.21.15.57:0/298486600 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f69640ceb40 0x7f69640b5110 secure :-1 s=READY pgs=2836 cs=0 l=1 rev1=1 crypto rx=0x7f695c006010 tx=0x7f695c035aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:56.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.474+0000 7f69617fa640 1 -- 172.21.15.57:0/298486600 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f695c034440 con 0x7f69640ceb40 2023-10-19T14:21:56.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.474+0000 7f69617fa640 1 -- 172.21.15.57:0/298486600 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f695c004330 con 0x7f69640ceb40 2023-10-19T14:21:56.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.474+0000 7f696aee5640 1 -- 172.21.15.57:0/298486600 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f695c009900 con 0x7f69640ceb40 2023-10-19T14:21:56.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.474+0000 7f69617fa640 1 -- 172.21.15.57:0/298486600 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f695c035ea0 con 0x7f69640ceb40 2023-10-19T14:21:56.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.474+0000 7f696aee5640 1 -- 172.21.15.57:0/298486600 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69640b4910 con 0x7f69640ceb40 2023-10-19T14:21:56.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.475+0000 7f696aee5640 1 -- 172.21.15.57:0/298486600 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69640690e0 con 0x7f69640ceb40 2023-10-19T14:21:56.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.476+0000 7f69617fa640 1 -- 172.21.15.57:0/298486600 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 145) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f695c03b040 con 0x7f69640ceb40 2023-10-19T14:21:56.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.476+0000 7f69617fa640 1 --2- 172.21.15.57:0/298486600 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f694406b080 0x7f694406d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:56.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.476+0000 7f69617fa640 1 -- 172.21.15.57:0/298486600 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(255..255 src has 1..255) v4 ==== 6237+0+0 (secure 0 0 0) 0x7f695c0ac660 con 0x7f69640ceb40 2023-10-19T14:21:56.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.476+0000 7f6963fff640 1 --2- 172.21.15.57:0/298486600 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f694406b080 0x7f694406d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:56.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.477+0000 7f6963fff640 1 --2- 172.21.15.57:0/298486600 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f694406b080 0x7f694406d540 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f696415acb0 tx=0x7f6954005f90 comp rx=0 tx=0).ready entity=mgr.11446 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:56.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.481+0000 7f69617fa640 1 -- 172.21.15.57:0/298486600 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f695c077240 con 0x7f69640ceb40 2023-10-19T14:21:56.663 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.660+0000 7f696aee5640 1 -- 172.21.15.57:0/298486600 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "config dump"} v 0) v1 -- 0x7f69640d0b40 con 0x7f69640ceb40 2023-10-19T14:21:56.664 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.661+0000 7f69617fa640 1 -- 172.21.15.57:0/298486600 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v82) v1 ==== 59+0+2220 (secure 0 0 0) 0x7f695c07a860 con 0x7f69640ceb40 2023-10-19T14:21:56.664 INFO:teuthology.orchestra.run.smithi057.stdout:WHO MASK LEVEL OPTION VALUE RO 2023-10-19T14:21:56.664 INFO:teuthology.orchestra.run.smithi057.stdout:mon advanced mon_op_complaint_time 120 2023-10-19T14:21:56.665 INFO:teuthology.orchestra.run.smithi057.stdout:osd advanced osd_op_complaint_time 180.000000 2023-10-19T14:21:56.665 INFO:teuthology.orchestra.run.smithi057.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 4775.995373 2023-10-19T14:21:56.665 INFO:teuthology.orchestra.run.smithi057.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 4208.118410 2023-10-19T14:21:56.665 INFO:teuthology.orchestra.run.smithi057.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 4541.123030 2023-10-19T14:21:56.665 INFO:teuthology.orchestra.run.smithi057.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 4877.592206 2023-10-19T14:21:56.666 INFO:teuthology.orchestra.run.smithi057.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 5019.748032 2023-10-19T14:21:56.666 INFO:teuthology.orchestra.run.smithi057.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 5000.074376 2023-10-19T14:21:56.666 INFO:teuthology.orchestra.run.smithi057.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 5153.479439 2023-10-19T14:21:56.666 INFO:teuthology.orchestra.run.smithi057.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 4977.363845 2023-10-19T14:21:56.666 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced debug_mds 20/20 2023-10-19T14:21:56.667 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced debug_mds_balancer 20/20 2023-10-19T14:21:56.667 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced debug_ms 1/1 2023-10-19T14:21:56.667 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced mds_bal_fragment_size_max 10000 2023-10-19T14:21:56.667 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced mds_bal_merge_size 5 2023-10-19T14:21:56.667 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced mds_bal_split_bits 3 2023-10-19T14:21:56.667 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced mds_bal_split_size 100 2023-10-19T14:21:56.668 INFO:teuthology.orchestra.run.smithi057.stdout:mds dev mds_debug_frag true 2023-10-19T14:21:56.668 INFO:teuthology.orchestra.run.smithi057.stdout:mds dev mds_debug_scatterstat true 2023-10-19T14:21:56.668 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced mds_op_complaint_time 180.000000 2023-10-19T14:21:56.668 INFO:teuthology.orchestra.run.smithi057.stdout:mds dev mds_verify_scatter true 2023-10-19T14:21:56.668 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced osd_op_complaint_time 180.000000 2023-10-19T14:21:56.669 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced rados_mon_op_timeout 900 2023-10-19T14:21:56.669 INFO:teuthology.orchestra.run.smithi057.stdout:mds advanced rados_osd_op_timeout 900 2023-10-19T14:21:56.669 INFO:teuthology.orchestra.run.smithi057.stdout:client advanced client_mount_timeout 600 2023-10-19T14:21:56.669 INFO:teuthology.orchestra.run.smithi057.stdout:client advanced debug_client 20/20 2023-10-19T14:21:56.669 INFO:teuthology.orchestra.run.smithi057.stdout:client advanced debug_ms 1/1 2023-10-19T14:21:56.669 INFO:teuthology.orchestra.run.smithi057.stdout:client advanced rados_mon_op_timeout 900 2023-10-19T14:21:56.670 INFO:teuthology.orchestra.run.smithi057.stdout:client advanced rados_osd_op_timeout 900 2023-10-19T14:21:56.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.663+0000 7f696aee5640 1 -- 172.21.15.57:0/298486600 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f694406b080 msgr2=0x7f694406d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:56.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.663+0000 7f696aee5640 1 --2- 172.21.15.57:0/298486600 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f694406b080 0x7f694406d540 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f696415acb0 tx=0x7f6954005f90 comp rx=0 tx=0).stop 2023-10-19T14:21:56.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.663+0000 7f696aee5640 1 -- 172.21.15.57:0/298486600 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f69640ceb40 msgr2=0x7f69640b5110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:56.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.663+0000 7f696aee5640 1 --2- 172.21.15.57:0/298486600 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f69640ceb40 0x7f69640b5110 secure :-1 s=READY pgs=2836 cs=0 l=1 rev1=1 crypto rx=0x7f695c006010 tx=0x7f695c035aa0 comp rx=0 tx=0).stop 2023-10-19T14:21:56.671 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.663+0000 7f696aee5640 1 -- 172.21.15.57:0/298486600 shutdown_connections 2023-10-19T14:21:56.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.663+0000 7f696aee5640 1 --2- 172.21.15.57:0/298486600 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f694406b080 0x7f694406d540 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:56.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.663+0000 7f696aee5640 1 --2- 172.21.15.57:0/298486600 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f696415b2a0 0x7f6964158370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:56.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.663+0000 7f696aee5640 1 --2- 172.21.15.57:0/298486600 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f69640cf340 0x7f696409e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:56.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.663+0000 7f696aee5640 1 --2- 172.21.15.57:0/298486600 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f69640ceb40 0x7f69640b5110 unknown :-1 s=CLOSED pgs=2836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:56.672 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.663+0000 7f696aee5640 1 -- 172.21.15.57:0/298486600 >> 172.21.15.57:0/298486600 conn(0x7f69640d0d60 msgr2=0x7f696406b810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:56.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.663+0000 7f696aee5640 1 -- 172.21.15.57:0/298486600 shutdown_connections 2023-10-19T14:21:56.673 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.663+0000 7f696aee5640 1 -- 172.21.15.57:0/298486600 wait complete. 2023-10-19T14:21:56.677 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-19T14:21:56.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.786+0000 7f3b2bc75640 1 -- 172.21.15.57:0/3626439350 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3b240c9750 msgr2=0x7f3b240c9b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:56.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.786+0000 7f3b2bc75640 1 --2- 172.21.15.57:0/3626439350 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3b240c9750 0x7f3b240c9b70 secure :-1 s=READY pgs=2837 cs=0 l=1 rev1=1 crypto rx=0x7f3b240cbd30 tx=0x7f3b1c02c1d0 comp rx=0 tx=0).stop 2023-10-19T14:21:56.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.786+0000 7f3b2bc75640 1 -- 172.21.15.57:0/3626439350 shutdown_connections 2023-10-19T14:21:56.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.786+0000 7f3b2bc75640 1 --2- 172.21.15.57:0/3626439350 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3b240c9750 0x7f3b240c9b70 unknown :-1 s=CLOSED pgs=2837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:56.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.786+0000 7f3b2bc75640 1 --2- 172.21.15.57:0/3626439350 >> v2:172.21.15.179:3300/0 conn(0x7f3b2406b030 0x7f3b2406b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:56.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.786+0000 7f3b2bc75640 1 -- 172.21.15.57:0/3626439350 >> 172.21.15.57:0/3626439350 conn(0x7f3b24069930 msgr2=0x7f3b24069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:56.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.786+0000 7f3b2bc75640 1 -- 172.21.15.57:0/3626439350 shutdown_connections 2023-10-19T14:21:56.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.786+0000 7f3b2bc75640 1 -- 172.21.15.57:0/3626439350 wait complete. 2023-10-19T14:21:56.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.787+0000 7f3b2bc75640 1 Processor -- start 2023-10-19T14:21:56.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.787+0000 7f3b2bc75640 1 -- start start 2023-10-19T14:21:56.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.788+0000 7f3b2bc75640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3b2406b030 0x7f3b2413a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:56.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.788+0000 7f3b2bc75640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3b240c9750 0x7f3b240a84c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:56.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.788+0000 7f3b2bc75640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3b24159bd0 0x7f3b2415a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:56.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.788+0000 7f3b2bc75640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b240a9a90 con 0x7f3b240c9750 2023-10-19T14:21:56.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.788+0000 7f3b2bc75640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b240a8f10 con 0x7f3b24159bd0 2023-10-19T14:21:56.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.788+0000 7f3b2bc75640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b240a91f0 con 0x7f3b2406b030 2023-10-19T14:21:56.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.788+0000 7f3b29210640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3b240c9750 0x7f3b240a84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:56.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.788+0000 7f3b29210640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3b240c9750 0x7f3b240a84c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51124/0 (socket says 172.21.15.57:51124) 2023-10-19T14:21:56.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.788+0000 7f3b29210640 1 -- 172.21.15.57:0/3076601666 learned_addr learned my addr 172.21.15.57:0/3076601666 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:56.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.788+0000 7f3b29210640 1 -- 172.21.15.57:0/3076601666 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3b2406b030 msgr2=0x7f3b2413a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:56.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.788+0000 7f3b29210640 1 --2- 172.21.15.57:0/3076601666 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3b2406b030 0x7f3b2413a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:56.795 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.788+0000 7f3b29210640 1 -- 172.21.15.57:0/3076601666 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3b24159bd0 msgr2=0x7f3b2415a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:56.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.788+0000 7f3b29210640 1 --2- 172.21.15.57:0/3076601666 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3b24159bd0 0x7f3b2415a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:56.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.788+0000 7f3b29210640 1 -- 172.21.15.57:0/3076601666 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b1c009900 con 0x7f3b240c9750 2023-10-19T14:21:56.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.788+0000 7f3b29210640 1 --2- 172.21.15.57:0/3076601666 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3b240c9750 0x7f3b240a84c0 secure :-1 s=READY pgs=2838 cs=0 l=1 rev1=1 crypto rx=0x7f3b1400b3f0 tx=0x7f3b1400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:56.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.788+0000 7f3b1a7fc640 1 -- 172.21.15.57:0/3076601666 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b14011830 con 0x7f3b240c9750 2023-10-19T14:21:56.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.788+0000 7f3b1a7fc640 1 -- 172.21.15.57:0/3076601666 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f3b14011a80 con 0x7f3b240c9750 2023-10-19T14:21:56.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.788+0000 7f3b1a7fc640 1 -- 172.21.15.57:0/3076601666 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b14011e90 con 0x7f3b240c9750 2023-10-19T14:21:56.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.788+0000 7f3b2bc75640 1 -- 172.21.15.57:0/3076601666 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b240a94d0 con 0x7f3b240c9750 2023-10-19T14:21:56.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.788+0000 7f3b2bc75640 1 -- 172.21.15.57:0/3076601666 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b2413a7e0 con 0x7f3b240c9750 2023-10-19T14:21:56.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.789+0000 7f3b2bc75640 1 -- 172.21.15.57:0/3076601666 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b2406c290 con 0x7f3b240c9750 2023-10-19T14:21:56.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.790+0000 7f3b1a7fc640 1 -- 172.21.15.57:0/3076601666 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 145) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f3b1401a400 con 0x7f3b240c9750 2023-10-19T14:21:56.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.790+0000 7f3b1a7fc640 1 --2- 172.21.15.57:0/3076601666 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f3b0806b080 0x7f3b0806d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:56.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.790+0000 7f3b1a7fc640 1 -- 172.21.15.57:0/3076601666 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(255..255 src has 1..255) v4 ==== 6237+0+0 (secure 0 0 0) 0x7f3b14089d60 con 0x7f3b240c9750 2023-10-19T14:21:56.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.790+0000 7f3b29a11640 1 --2- 172.21.15.57:0/3076601666 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f3b0806b080 0x7f3b0806d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:56.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.791+0000 7f3b29a11640 1 --2- 172.21.15.57:0/3076601666 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f3b0806b080 0x7f3b0806d540 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3b24155e20 tx=0x7f3b20005c70 comp rx=0 tx=0).ready entity=mgr.11446 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:56.799 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.794+0000 7f3b1a7fc640 1 -- 172.21.15.57:0/3076601666 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3b140548a0 con 0x7f3b240c9750 2023-10-19T14:21:56.974 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.971+0000 7f3b2bc75640 1 -- 172.21.15.57:0/3076601666 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f3b2406ae00 con 0x7f3b240c9750 2023-10-19T14:21:56.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.976+0000 7f3b1a7fc640 1 -- 172.21.15.57:0/3076601666 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v83)=0 v83) v1 ==== 122+0+0 (secure 0 0 0) 0x7f3b140150c0 con 0x7f3b240c9750 2023-10-19T14:21:56.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.978+0000 7f3b2bc75640 1 -- 172.21.15.57:0/3076601666 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f3b0806b080 msgr2=0x7f3b0806d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:56.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.978+0000 7f3b2bc75640 1 --2- 172.21.15.57:0/3076601666 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f3b0806b080 0x7f3b0806d540 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3b24155e20 tx=0x7f3b20005c70 comp rx=0 tx=0).stop 2023-10-19T14:21:56.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.978+0000 7f3b2bc75640 1 -- 172.21.15.57:0/3076601666 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3b240c9750 msgr2=0x7f3b240a84c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:56.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.978+0000 7f3b2bc75640 1 --2- 172.21.15.57:0/3076601666 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3b240c9750 0x7f3b240a84c0 secure :-1 s=READY pgs=2838 cs=0 l=1 rev1=1 crypto rx=0x7f3b1400b3f0 tx=0x7f3b1400b8b0 comp rx=0 tx=0).stop 2023-10-19T14:21:56.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.978+0000 7f3b2bc75640 1 -- 172.21.15.57:0/3076601666 shutdown_connections 2023-10-19T14:21:56.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.978+0000 7f3b2bc75640 1 --2- 172.21.15.57:0/3076601666 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f3b0806b080 0x7f3b0806d540 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:56.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.978+0000 7f3b2bc75640 1 --2- 172.21.15.57:0/3076601666 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3b24159bd0 0x7f3b2415a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:56.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.978+0000 7f3b2bc75640 1 --2- 172.21.15.57:0/3076601666 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3b240c9750 0x7f3b240a84c0 unknown :-1 s=CLOSED pgs=2838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:56.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.978+0000 7f3b2bc75640 1 --2- 172.21.15.57:0/3076601666 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3b2406b030 0x7f3b2413a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:56.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.978+0000 7f3b2bc75640 1 -- 172.21.15.57:0/3076601666 >> 172.21.15.57:0/3076601666 conn(0x7f3b24069930 msgr2=0x7f3b2415baf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:56.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.978+0000 7f3b2bc75640 1 -- 172.21.15.57:0/3076601666 shutdown_connections 2023-10-19T14:21:56.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:56.978+0000 7f3b2bc75640 1 -- 172.21.15.57:0/3076601666 wait complete. 2023-10-19T14:21:56.991 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-19T14:21:56.991 DEBUG:teuthology.orchestra.run.smithi057:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-19T14:21:57.031 DEBUG:teuthology.orchestra.run.smithi179:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-19T14:21:57.079 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-19T14:21:57.080 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T14:21:57.080 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T14:21:57.096 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055614,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=358 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248260k,nr_inodes=812065,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-19T14:21:57.097 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-19T14:21:57.097 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T14:21:57.114 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054833,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=284 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3247632k,nr_inodes=811908,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-19T14:21:57.115 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T14:21:57.115 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T14:21:57.152 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055614,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=358 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248260k,nr_inodes=812065,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-19T14:21:57.152 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-19T14:21:57.153 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T14:21:57.171 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054833,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=284 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3247632k,nr_inodes=811908,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-19T14:21:57.172 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-19T14:21:57.172 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T14:21:57.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.298+0000 7f1628e2b640 1 -- 172.21.15.57:0/3617935182 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1610004d00 msgr2=0x7f1610005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:57.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.298+0000 7f1628e2b640 1 --2- 172.21.15.57:0/3617935182 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1610004d00 0x7f1610005180 secure :-1 s=READY pgs=2840 cs=0 l=1 rev1=1 crypto rx=0x7f161c00c640 tx=0x7f16100057f0 comp rx=0 tx=0).stop 2023-10-19T14:21:57.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.298+0000 7f1622d9d640 1 -- 172.21.15.57:0/3617935182 reap_dead start 2023-10-19T14:21:57.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.298+0000 7f1628e2b640 1 -- 172.21.15.57:0/3617935182 shutdown_connections 2023-10-19T14:21:57.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.298+0000 7f1628e2b640 1 -- 172.21.15.57:0/3617935182 >> 172.21.15.57:0/3617935182 conn(0x7f1624069930 msgr2=0x7f1624069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:57.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.298+0000 7f1628e2b640 1 -- 172.21.15.57:0/3617935182 shutdown_connections 2023-10-19T14:21:57.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.298+0000 7f1628e2b640 1 -- 172.21.15.57:0/3617935182 wait complete. 2023-10-19T14:21:57.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.300+0000 7f1628e2b640 1 Processor -- start 2023-10-19T14:21:57.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.300+0000 7f1628e2b640 1 -- start start 2023-10-19T14:21:57.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.300+0000 7f1628e2b640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1610004d00 0x7f1624133d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:57.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.300+0000 7f1628e2b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1624136f60 0x7f16241373e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:57.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.300+0000 7f1628e2b640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f162415c9f0 0x7f162415ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:57.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.300+0000 7f1628e2b640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f162415d3b0 con 0x7f1624136f60 2023-10-19T14:21:57.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.300+0000 7f1628e2b640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f162413ce90 con 0x7f1610004d00 2023-10-19T14:21:57.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.300+0000 7f1628e2b640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16241342c0 con 0x7f162415c9f0 2023-10-19T14:21:57.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.300+0000 7f161bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1624136f60 0x7f16241373e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:57.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.300+0000 7f161bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1624136f60 0x7f16241373e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51146/0 (socket says 172.21.15.57:51146) 2023-10-19T14:21:57.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.300+0000 7f161bfff640 1 -- 172.21.15.57:0/3455387554 learned_addr learned my addr 172.21.15.57:0/3455387554 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:57.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.301+0000 7f162259c640 1 --2- 172.21.15.57:0/3455387554 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1610004d00 0x7f1624133d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:57.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.301+0000 7f1622d9d640 1 --2- 172.21.15.57:0/3455387554 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f162415c9f0 0x7f162415ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:57.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.301+0000 7f161bfff640 1 -- 172.21.15.57:0/3455387554 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f162415c9f0 msgr2=0x7f162415ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:57.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.301+0000 7f161bfff640 1 --2- 172.21.15.57:0/3455387554 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f162415c9f0 0x7f162415ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:57.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.301+0000 7f161bfff640 1 -- 172.21.15.57:0/3455387554 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1610004d00 msgr2=0x7f1624133d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:57.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.301+0000 7f161bfff640 1 --2- 172.21.15.57:0/3455387554 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1610004d00 0x7f1624133d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:57.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.301+0000 7f161bfff640 1 -- 172.21.15.57:0/3455387554 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1624152260 con 0x7f1624136f60 2023-10-19T14:21:57.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.301+0000 7f161bfff640 1 --2- 172.21.15.57:0/3455387554 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1624136f60 0x7f16241373e0 secure :-1 s=READY pgs=2841 cs=0 l=1 rev1=1 crypto rx=0x7f160c011ae0 tx=0x7f160c011ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:57.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.301+0000 7f161affd640 1 -- 172.21.15.57:0/3455387554 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f160c00fe70 con 0x7f1624136f60 2023-10-19T14:21:57.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.301+0000 7f161affd640 1 -- 172.21.15.57:0/3455387554 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f160c004050 con 0x7f1624136f60 2023-10-19T14:21:57.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.301+0000 7f1628e2b640 1 -- 172.21.15.57:0/3455387554 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f162413c280 con 0x7f1624136f60 2023-10-19T14:21:57.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.301+0000 7f161affd640 1 -- 172.21.15.57:0/3455387554 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f160c004460 con 0x7f1624136f60 2023-10-19T14:21:57.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.301+0000 7f1628e2b640 1 -- 172.21.15.57:0/3455387554 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f162413c840 con 0x7f1624136f60 2023-10-19T14:21:57.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.302+0000 7f1628e2b640 1 -- 172.21.15.57:0/3455387554 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1624152e30 con 0x7f1624136f60 2023-10-19T14:21:57.311 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.303+0000 7f161affd640 1 -- 172.21.15.57:0/3455387554 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 145) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f160c0035f0 con 0x7f1624136f60 2023-10-19T14:21:57.312 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.303+0000 7f161affd640 1 --2- 172.21.15.57:0/3455387554 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f160406b080 0x7f160406d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:57.312 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.303+0000 7f161affd640 1 -- 172.21.15.57:0/3455387554 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(255..255 src has 1..255) v4 ==== 6237+0+0 (secure 0 0 0) 0x7f160c08a960 con 0x7f1624136f60 2023-10-19T14:21:57.312 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.303+0000 7f162259c640 1 --2- 172.21.15.57:0/3455387554 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f160406b080 0x7f160406d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:57.313 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.304+0000 7f162259c640 1 --2- 172.21.15.57:0/3455387554 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f160406b080 0x7f160406d540 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1614009cd0 tx=0x7f1614005f90 comp rx=0 tx=0).ready entity=mgr.11446 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:57.313 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.307+0000 7f161affd640 1 -- 172.21.15.57:0/3455387554 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f160c055500 con 0x7f1624136f60 2023-10-19T14:21:57.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.520+0000 7f1628e2b640 1 -- 172.21.15.57:0/3455387554 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1624159a00 con 0x7f1624136f60 2023-10-19T14:21:57.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.521+0000 7f161affd640 1 -- 172.21.15.57:0/3455387554 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 111 v111) v1 ==== 95+0+3688 (secure 0 0 0) 0x7f160c058b20 con 0x7f1624136f60 2023-10-19T14:21:57.525 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:21:57.525 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":111,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9761,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/325888554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":325888554},{"type":"v1","addr":"172.21.15.179:6836","nonce":325888554}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":100},{"gid":9764,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/8038232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":8038232},{"type":"v1","addr":"172.21.15.179:6837","nonce":8038232}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":100},{"gid":13170,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/4130227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":4130227},{"type":"v1","addr":"172.21.15.57:6835","nonce":4130227}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":109},{"gid":13263,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6833/444914605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":444914605},{"type":"v1","addr":"172.21.15.57:6833","nonce":444914605}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[]} 2023-10-19T14:21:57.526 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 111 2023-10-19T14:21:57.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.523+0000 7f1628e2b640 1 -- 172.21.15.57:0/3455387554 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f160406b080 msgr2=0x7f160406d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:57.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.523+0000 7f1628e2b640 1 --2- 172.21.15.57:0/3455387554 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f160406b080 0x7f160406d540 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1614009cd0 tx=0x7f1614005f90 comp rx=0 tx=0).stop 2023-10-19T14:21:57.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.523+0000 7f1628e2b640 1 -- 172.21.15.57:0/3455387554 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1624136f60 msgr2=0x7f16241373e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:57.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.523+0000 7f1628e2b640 1 --2- 172.21.15.57:0/3455387554 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1624136f60 0x7f16241373e0 secure :-1 s=READY pgs=2841 cs=0 l=1 rev1=1 crypto rx=0x7f160c011ae0 tx=0x7f160c011ee0 comp rx=0 tx=0).stop 2023-10-19T14:21:57.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.523+0000 7f1628e2b640 1 -- 172.21.15.57:0/3455387554 shutdown_connections 2023-10-19T14:21:57.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.523+0000 7f1628e2b640 1 --2- 172.21.15.57:0/3455387554 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f160406b080 0x7f160406d540 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:57.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.523+0000 7f1628e2b640 1 --2- 172.21.15.57:0/3455387554 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f162415c9f0 0x7f162415ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:57.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.523+0000 7f1628e2b640 1 --2- 172.21.15.57:0/3455387554 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1624136f60 0x7f16241373e0 unknown :-1 s=CLOSED pgs=2841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:57.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.523+0000 7f1628e2b640 1 --2- 172.21.15.57:0/3455387554 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1610004d00 0x7f1624133d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:57.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.523+0000 7f1628e2b640 1 -- 172.21.15.57:0/3455387554 >> 172.21.15.57:0/3455387554 conn(0x7f1624069930 msgr2=0x7f1624152720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:57.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.523+0000 7f1628e2b640 1 -- 172.21.15.57:0/3455387554 shutdown_connections 2023-10-19T14:21:57.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.523+0000 7f1628e2b640 1 -- 172.21.15.57:0/3455387554 wait complete. 2023-10-19T14:21:57.537 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-19T14:21:57.537 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-19T14:21:57.537 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-19T14:21:57.538 INFO:teuthology.orchestra.run:waiting for 300 2023-10-19T14:21:57.538 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-19T14:21:57.538 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-19T14:21:57.539 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-19T14:21:57.539 INFO:teuthology.orchestra.run:waiting for 300 2023-10-19T14:21:57.539 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-19T14:21:57.539 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-19T14:21:57.540 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-19T14:21:57.540 INFO:teuthology.orchestra.run:waiting for 300 2023-10-19T14:21:57.540 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-19T14:21:57.540 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-19T14:21:57.541 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-19T14:21:57.541 INFO:teuthology.orchestra.run:waiting for 300 2023-10-19T14:21:57.546 INFO:tasks.ceph.mds.a:Stopped 2023-10-19T14:21:57.546 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-19T14:21:57.548 INFO:tasks.ceph.mds.c:Stopped 2023-10-19T14:21:57.549 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-19T14:21:57.554 INFO:tasks.ceph.mds.b:Stopped 2023-10-19T14:21:57.555 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-19T14:21:57.555 INFO:tasks.ceph.mds.d:Stopped 2023-10-19T14:21:57.556 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-19T14:21:57.590 INFO:tasks.ceph.mds.a:Started 2023-10-19T14:21:57.591 DEBUG:teuthology.parallel:result is None 2023-10-19T14:21:57.592 INFO:tasks.ceph.mds.c:Started 2023-10-19T14:21:57.592 DEBUG:teuthology.parallel:result is None 2023-10-19T14:21:57.597 INFO:tasks.ceph.mds.b:Started 2023-10-19T14:21:57.597 DEBUG:teuthology.parallel:result is None 2023-10-19T14:21:57.598 INFO:tasks.ceph.mds.d:Started 2023-10-19T14:21:57.599 DEBUG:teuthology.parallel:result is None 2023-10-19T14:21:57.599 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-19T14:21:57.671 INFO:tasks.ceph.mds.a.smithi057.stdout:starting mds.a at 2023-10-19T14:21:57.672 INFO:tasks.ceph.mds.c.smithi057.stdout:starting mds.c at 2023-10-19T14:21:57.679 INFO:tasks.ceph.mds.b.smithi179.stdout:starting mds.b at 2023-10-19T14:21:57.680 INFO:tasks.ceph.mds.d.smithi179.stdout:starting mds.d at 2023-10-19T14:21:57.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.709+0000 7ffa50226640 1 -- 172.21.15.57:0/2620282348 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffa480cf340 msgr2=0x7ffa480d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:57.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.709+0000 7ffa50226640 1 --2- 172.21.15.57:0/2620282348 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffa480cf340 0x7ffa480d1910 secure :-1 s=READY pgs=2848 cs=0 l=1 rev1=1 crypto rx=0x7ffa40005fd0 tx=0x7ffa4002bd90 comp rx=0 tx=0).stop 2023-10-19T14:21:57.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.709+0000 7ffa50226640 1 -- 172.21.15.57:0/2620282348 shutdown_connections 2023-10-19T14:21:57.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.709+0000 7ffa50226640 1 --2- 172.21.15.57:0/2620282348 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffa480cf340 0x7ffa480d1910 unknown :-1 s=CLOSED pgs=2848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:57.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.709+0000 7ffa50226640 1 -- 172.21.15.57:0/2620282348 >> 172.21.15.57:0/2620282348 conn(0x7ffa480d0d60 msgr2=0x7ffa480d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:57.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.709+0000 7ffa50226640 1 -- 172.21.15.57:0/2620282348 shutdown_connections 2023-10-19T14:21:57.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.709+0000 7ffa50226640 1 -- 172.21.15.57:0/2620282348 wait complete. 2023-10-19T14:21:57.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.710+0000 7ffa50226640 1 Processor -- start 2023-10-19T14:21:57.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.710+0000 7ffa50226640 1 -- start start 2023-10-19T14:21:57.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.711+0000 7ffa50226640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffa480ceb40 0x7ffa480b4850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:57.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.711+0000 7ffa50226640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffa480cf340 0x7ffa481f2920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:57.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.711+0000 7ffa50226640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ffa4815b2a0 0x7ffa480ca0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:57.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.711+0000 7ffa50226640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa480b5770 con 0x7ffa480cf340 2023-10-19T14:21:57.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.711+0000 7ffa50226640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa480b5460 con 0x7ffa480ceb40 2023-10-19T14:21:57.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.711+0000 7ffa50226640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa481f1fb0 con 0x7ffa4815b2a0 2023-10-19T14:21:57.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.711+0000 7ffa4d7c1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffa480cf340 0x7ffa481f2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:57.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.711+0000 7ffa4d7c1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffa480cf340 0x7ffa481f2920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51196/0 (socket says 172.21.15.57:51196) 2023-10-19T14:21:57.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.711+0000 7ffa4d7c1640 1 -- 172.21.15.57:0/3678237973 learned_addr learned my addr 172.21.15.57:0/3678237973 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:57.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.712+0000 7ffa4dfc2640 1 --2- 172.21.15.57:0/3678237973 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffa480ceb40 0x7ffa480b4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:57.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.712+0000 7ffa4e7c3640 1 --2- 172.21.15.57:0/3678237973 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ffa4815b2a0 0x7ffa480ca0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:57.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.712+0000 7ffa4d7c1640 1 -- 172.21.15.57:0/3678237973 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ffa4815b2a0 msgr2=0x7ffa480ca0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:57.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.712+0000 7ffa4d7c1640 1 --2- 172.21.15.57:0/3678237973 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ffa4815b2a0 0x7ffa480ca0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:57.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.712+0000 7ffa4d7c1640 1 -- 172.21.15.57:0/3678237973 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffa480ceb40 msgr2=0x7ffa480b4850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:57.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.712+0000 7ffa4d7c1640 1 --2- 172.21.15.57:0/3678237973 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffa480ceb40 0x7ffa480b4850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:57.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.712+0000 7ffa4d7c1640 1 -- 172.21.15.57:0/3678237973 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa40009560 con 0x7ffa480cf340 2023-10-19T14:21:57.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.712+0000 7ffa4dfc2640 1 --2- 172.21.15.57:0/3678237973 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffa480ceb40 0x7ffa480b4850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:21:57.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.712+0000 7ffa4d7c1640 1 --2- 172.21.15.57:0/3678237973 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffa480cf340 0x7ffa481f2920 secure :-1 s=READY pgs=2849 cs=0 l=1 rev1=1 crypto rx=0x7ffa3800b3e0 tx=0x7ffa3800b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:57.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.712+0000 7ffa3e7fc640 1 -- 172.21.15.57:0/3678237973 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa38011830 con 0x7ffa480cf340 2023-10-19T14:21:57.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.712+0000 7ffa3e7fc640 1 -- 172.21.15.57:0/3678237973 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7ffa38011aa0 con 0x7ffa480cf340 2023-10-19T14:21:57.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.712+0000 7ffa50226640 1 -- 172.21.15.57:0/3678237973 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa481f2290 con 0x7ffa480cf340 2023-10-19T14:21:57.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.712+0000 7ffa3e7fc640 1 -- 172.21.15.57:0/3678237973 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa38010380 con 0x7ffa480cf340 2023-10-19T14:21:57.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.712+0000 7ffa50226640 1 -- 172.21.15.57:0/3678237973 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa480ca5f0 con 0x7ffa480cf340 2023-10-19T14:21:57.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.713+0000 7ffa50226640 1 -- 172.21.15.57:0/3678237973 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa480ca810 con 0x7ffa480cf340 2023-10-19T14:21:57.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.714+0000 7ffa3e7fc640 1 -- 172.21.15.57:0/3678237973 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 145) v1 ==== 89433+0+0 (secure 0 0 0) 0x7ffa380104e0 con 0x7ffa480cf340 2023-10-19T14:21:57.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.714+0000 7ffa3e7fc640 1 --2- 172.21.15.57:0/3678237973 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7ffa2c06b080 0x7ffa2c06d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:57.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.714+0000 7ffa3e7fc640 1 -- 172.21.15.57:0/3678237973 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(255..255 src has 1..255) v4 ==== 6237+0+0 (secure 0 0 0) 0x7ffa38089f40 con 0x7ffa480cf340 2023-10-19T14:21:57.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.714+0000 7ffa4dfc2640 1 --2- 172.21.15.57:0/3678237973 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7ffa2c06b080 0x7ffa2c06d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:57.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.715+0000 7ffa4dfc2640 1 --2- 172.21.15.57:0/3678237973 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7ffa2c06b080 0x7ffa2c06d540 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ffa40035000 tx=0x7ffa40004730 comp rx=0 tx=0).ready entity=mgr.11446 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:57.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.718+0000 7ffa3e7fc640 1 -- 172.21.15.57:0/3678237973 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ffa38054b10 con 0x7ffa480cf340 2023-10-19T14:21:57.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:57.907+0000 7ffa50226640 1 -- 172.21.15.57:0/3678237973 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7ffa480690e0 con 0x7ffa480cf340 2023-10-19T14:21:58.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.365+0000 7ffa3e7fc640 1 -- 172.21.15.57:0/3678237973 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v257) v1 ==== 97+0+0 (secure 0 0 0) 0x7ffa38015080 con 0x7ffa480cf340 2023-10-19T14:21:58.368 INFO:teuthology.orchestra.run.smithi057.stderr: removed all blocklist entries 2023-10-19T14:21:58.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.368+0000 7ffa50226640 1 -- 172.21.15.57:0/3678237973 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7ffa2c06b080 msgr2=0x7ffa2c06d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:58.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.368+0000 7ffa50226640 1 --2- 172.21.15.57:0/3678237973 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7ffa2c06b080 0x7ffa2c06d540 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ffa40035000 tx=0x7ffa40004730 comp rx=0 tx=0).stop 2023-10-19T14:21:58.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.368+0000 7ffa50226640 1 -- 172.21.15.57:0/3678237973 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffa480cf340 msgr2=0x7ffa481f2920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:58.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.368+0000 7ffa50226640 1 --2- 172.21.15.57:0/3678237973 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffa480cf340 0x7ffa481f2920 secure :-1 s=READY pgs=2849 cs=0 l=1 rev1=1 crypto rx=0x7ffa3800b3e0 tx=0x7ffa3800b8a0 comp rx=0 tx=0).stop 2023-10-19T14:21:58.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.368+0000 7ffa50226640 1 -- 172.21.15.57:0/3678237973 shutdown_connections 2023-10-19T14:21:58.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.368+0000 7ffa50226640 1 --2- 172.21.15.57:0/3678237973 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7ffa2c06b080 0x7ffa2c06d540 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:58.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.368+0000 7ffa50226640 1 --2- 172.21.15.57:0/3678237973 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ffa4815b2a0 0x7ffa480ca0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:58.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.368+0000 7ffa50226640 1 --2- 172.21.15.57:0/3678237973 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffa480cf340 0x7ffa481f2920 unknown :-1 s=CLOSED pgs=2849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:58.372 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.368+0000 7ffa50226640 1 --2- 172.21.15.57:0/3678237973 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffa480ceb40 0x7ffa480b4850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:58.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.368+0000 7ffa50226640 1 -- 172.21.15.57:0/3678237973 >> 172.21.15.57:0/3678237973 conn(0x7ffa480d0d60 msgr2=0x7ffa4806b610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:58.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.368+0000 7ffa50226640 1 -- 172.21.15.57:0/3678237973 shutdown_connections 2023-10-19T14:21:58.373 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.368+0000 7ffa50226640 1 -- 172.21.15.57:0/3678237973 wait complete. 2023-10-19T14:21:58.383 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-19T14:21:58.496 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.493+0000 7efdc737d640 1 -- 172.21.15.57:0/2341205059 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efdac004d00 msgr2=0x7efdac005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:58.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.493+0000 7efdc737d640 1 --2- 172.21.15.57:0/2341205059 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efdac004d00 0x7efdac005180 secure :-1 s=READY pgs=2851 cs=0 l=1 rev1=1 crypto rx=0x7efdb8001d50 tx=0x7efdac0057f0 comp rx=0 tx=0).stop 2023-10-19T14:21:58.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.493+0000 7efdc591a640 1 -- 172.21.15.57:0/2341205059 reap_dead start 2023-10-19T14:21:58.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.493+0000 7efdc737d640 1 -- 172.21.15.57:0/2341205059 shutdown_connections 2023-10-19T14:21:58.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.493+0000 7efdc737d640 1 -- 172.21.15.57:0/2341205059 >> 172.21.15.57:0/2341205059 conn(0x7efdc0069930 msgr2=0x7efdc0069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:58.497 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.493+0000 7efdc737d640 1 -- 172.21.15.57:0/2341205059 shutdown_connections 2023-10-19T14:21:58.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.493+0000 7efdc737d640 1 -- 172.21.15.57:0/2341205059 wait complete. 2023-10-19T14:21:58.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.494+0000 7efdc737d640 1 Processor -- start 2023-10-19T14:21:58.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.494+0000 7efdc737d640 1 -- start start 2023-10-19T14:21:58.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.495+0000 7efdc737d640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efdc015a5e0 0x7efdc015b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:58.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.495+0000 7efdc737d640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7efdc01483f0 0x7efdc01427c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:58.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.495+0000 7efdc737d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efdc0142d00 0x7efdc0143190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:58.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.495+0000 7efdc737d640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdc00ac040 con 0x7efdc0142d00 2023-10-19T14:21:58.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.495+0000 7efdc737d640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdc015b470 con 0x7efdc015a5e0 2023-10-19T14:21:58.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.495+0000 7efdc737d640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdc01566b0 con 0x7efdc01483f0 2023-10-19T14:21:58.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.495+0000 7efdc591a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efdc0142d00 0x7efdc0143190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:58.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.495+0000 7efdc591a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efdc0142d00 0x7efdc0143190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51222/0 (socket says 172.21.15.57:51222) 2023-10-19T14:21:58.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.495+0000 7efdc591a640 1 -- 172.21.15.57:0/1328601392 learned_addr learned my addr 172.21.15.57:0/1328601392 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:58.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.496+0000 7efdc5119640 1 --2- 172.21.15.57:0/1328601392 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efdc015a5e0 0x7efdc015b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:58.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.496+0000 7efdc4918640 1 --2- 172.21.15.57:0/1328601392 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7efdc01483f0 0x7efdc01427c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:58.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.496+0000 7efdc591a640 1 -- 172.21.15.57:0/1328601392 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7efdc01483f0 msgr2=0x7efdc01427c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:58.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.496+0000 7efdc591a640 1 --2- 172.21.15.57:0/1328601392 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7efdc01483f0 0x7efdc01427c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:58.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.496+0000 7efdc591a640 1 -- 172.21.15.57:0/1328601392 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efdc015a5e0 msgr2=0x7efdc015b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:58.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.496+0000 7efdc591a640 1 --2- 172.21.15.57:0/1328601392 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efdc015a5e0 0x7efdc015b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:58.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.496+0000 7efdc591a640 1 -- 172.21.15.57:0/1328601392 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdc01563d0 con 0x7efdc0142d00 2023-10-19T14:21:58.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.496+0000 7efdc5119640 1 --2- 172.21.15.57:0/1328601392 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efdc015a5e0 0x7efdc015b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:21:58.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.496+0000 7efdc4918640 1 --2- 172.21.15.57:0/1328601392 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7efdc01483f0 0x7efdc01427c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:21:58.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.496+0000 7efdc591a640 1 --2- 172.21.15.57:0/1328601392 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efdc0142d00 0x7efdc0143190 secure :-1 s=READY pgs=2852 cs=0 l=1 rev1=1 crypto rx=0x7efdb8001e70 tx=0x7efdb8003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:58.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.496+0000 7efdb5ffb640 1 -- 172.21.15.57:0/1328601392 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb802ccb0 con 0x7efdc0142d00 2023-10-19T14:21:58.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.496+0000 7efdb5ffb640 1 -- 172.21.15.57:0/1328601392 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7efdb8003e40 con 0x7efdc0142d00 2023-10-19T14:21:58.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.496+0000 7efdc737d640 1 -- 172.21.15.57:0/1328601392 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdc015ae20 con 0x7efdc0142d00 2023-10-19T14:21:58.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.496+0000 7efdb5ffb640 1 -- 172.21.15.57:0/1328601392 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb8006310 con 0x7efdc0142d00 2023-10-19T14:21:58.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.496+0000 7efdc737d640 1 -- 172.21.15.57:0/1328601392 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdc0156000 con 0x7efdc0142d00 2023-10-19T14:21:58.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.497+0000 7efdc737d640 1 -- 172.21.15.57:0/1328601392 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdc015b620 con 0x7efdc0142d00 2023-10-19T14:21:58.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.498+0000 7efdb5ffb640 1 -- 172.21.15.57:0/1328601392 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 145) v1 ==== 89433+0+0 (secure 0 0 0) 0x7efdb8006470 con 0x7efdc0142d00 2023-10-19T14:21:58.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.498+0000 7efdb5ffb640 1 --2- 172.21.15.57:0/1328601392 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7efda406b080 0x7efda406d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:58.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.498+0000 7efdb5ffb640 1 -- 172.21.15.57:0/1328601392 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(259..259 src has 1..259) v4 ==== 5205+0+0 (secure 0 0 0) 0x7efdb80aa7e0 con 0x7efdc0142d00 2023-10-19T14:21:58.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.498+0000 7efdc5119640 1 --2- 172.21.15.57:0/1328601392 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7efda406b080 0x7efda406d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:58.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.499+0000 7efdc5119640 1 --2- 172.21.15.57:0/1328601392 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7efda406b080 0x7efda406d540 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7efdbc009b40 tx=0x7efdbc005f90 comp rx=0 tx=0).ready entity=mgr.11446 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:58.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.502+0000 7efdb5ffb640 1 -- 172.21.15.57:0/1328601392 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7efdc015b620 con 0x7efdc0142d00 2023-10-19T14:21:58.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.745+0000 7efdc737d640 1 -- 172.21.15.57:0/1328601392 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7efdc015b5e0 con 0x7efdc0142d00 2023-10-19T14:21:58.749 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.746+0000 7efdb5ffb640 1 -- 172.21.15.57:0/1328601392 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v61) v1 ==== 80+0+5304 (secure 0 0 0) 0x7efdb8044070 con 0x7efdc0142d00 2023-10-19T14:21:58.749 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:21:58.749 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2023-10-19T14:21:58.750 INFO:teuthology.orchestra.run.smithi057.stdout: "auth_dump": [ 2023-10-19T14:21:58.750 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:21:58.750 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "mds.a", 2023-10-19T14:21:58.750 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBjJDFlmbVANhAAP/ynHlU/OxaRq5lQoAa8Ng==", 2023-10-19T14:21:58.751 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:21:58.751 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T14:21:58.751 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow *", 2023-10-19T14:21:58.751 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow *", 2023-10-19T14:21:58.751 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:21:58.751 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:21:58.752 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:21:58.752 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:21:58.752 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "mds.b", 2023-10-19T14:21:58.752 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFl9l7RBxAAC0geI2tJ1SIZZr1++kS/BA==", 2023-10-19T14:21:58.752 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:21:58.753 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T14:21:58.753 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow *", 2023-10-19T14:21:58.753 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow *", 2023-10-19T14:21:58.753 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:21:58.753 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:21:58.754 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:21:58.754 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:21:58.754 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "mds.c", 2023-10-19T14:21:58.754 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFlFkaoAhAA/PBizkJ4bm958oP4CmPuIQ==", 2023-10-19T14:21:58.754 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:21:58.755 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T14:21:58.755 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow *", 2023-10-19T14:21:58.755 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow *", 2023-10-19T14:21:58.755 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:21:58.755 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:21:58.756 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:21:58.756 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:21:58.756 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "mds.d", 2023-10-19T14:21:58.756 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFlx8skDxAAAfCHTYtYJgJik0WHSWeQfQ==", 2023-10-19T14:21:58.756 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:21:58.757 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T14:21:58.757 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow *", 2023-10-19T14:21:58.757 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow *", 2023-10-19T14:21:58.757 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:21:58.757 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:21:58.757 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:21:58.758 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:21:58.758 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.0", 2023-10-19T14:21:58.758 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBrJDFlCo9sBxAAQN2cID4nCQgRwGaGFEV8LQ==", 2023-10-19T14:21:58.758 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:21:58.758 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T14:21:58.759 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T14:21:58.759 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:21:58.759 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:21:58.759 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:21:58.759 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:21:58.760 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.1", 2023-10-19T14:21:58.760 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBuJDFl4fsLCBAAp3Pi80yZtc5sju0GisEuVA==", 2023-10-19T14:21:58.760 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:21:58.760 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T14:21:58.760 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T14:21:58.761 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:21:58.761 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:21:58.761 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:21:58.761 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:21:58.761 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.2", 2023-10-19T14:21:58.761 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBxJDFl+KAKCBAAd4N9cluARxPVfBLU+7Hd7A==", 2023-10-19T14:21:58.762 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:21:58.762 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T14:21:58.762 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T14:21:58.762 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:21:58.763 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:21:58.763 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:21:58.763 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:21:58.763 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.3", 2023-10-19T14:21:58.764 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQB0JDFlU/toCBAAP0Ou1zcHKNx5SfbybJkh4Q==", 2023-10-19T14:21:58.764 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:21:58.764 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T14:21:58.764 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T14:21:58.764 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:21:58.765 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:21:58.765 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:21:58.765 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:21:58.765 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.4", 2023-10-19T14:21:58.765 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQB6JDFlc6stEBAALkgNyxoiCTzZnKpBSPxI6w==", 2023-10-19T14:21:58.766 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:21:58.766 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T14:21:58.766 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T14:21:58.766 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:21:58.766 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:21:58.767 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:21:58.767 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:21:58.767 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.5", 2023-10-19T14:21:58.767 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQB9JDFlI3cNDhAAuG3tHJ6RvPwAzPE76wTIVQ==", 2023-10-19T14:21:58.767 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:21:58.768 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T14:21:58.768 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T14:21:58.768 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:21:58.768 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:21:58.768 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:21:58.768 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:21:58.769 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.6", 2023-10-19T14:21:58.769 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQCAJDFlLPumCxAA/iy08t/2N9rXfn71KtbSQw==", 2023-10-19T14:21:58.769 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:21:58.769 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T14:21:58.769 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T14:21:58.770 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:21:58.770 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:21:58.770 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:21:58.770 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:21:58.770 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "osd.7", 2023-10-19T14:21:58.771 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQCDJDFlHmrbCRAA+hpWygGIJydfU6cHWlfx/Q==", 2023-10-19T14:21:58.771 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:21:58.771 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow profile osd", 2023-10-19T14:21:58.771 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile osd", 2023-10-19T14:21:58.771 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:21:58.772 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:21:58.772 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:21:58.772 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:21:58.772 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "client.0", 2023-10-19T14:21:58.772 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFlg9yuExAATxiWDfS+p07G1F0pORbJEg==", 2023-10-19T14:21:58.773 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:21:58.773 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T14:21:58.773 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow r", 2023-10-19T14:21:58.773 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-19T14:21:58.773 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:21:58.774 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:21:58.774 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:21:58.774 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "client.1", 2023-10-19T14:21:58.774 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFl9IwYGxAAHPituiJSPkHDuD0QyG4Hww==", 2023-10-19T14:21:58.774 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:21:58.775 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T14:21:58.775 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow r", 2023-10-19T14:21:58.775 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow rw", 2023-10-19T14:21:58.775 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow rwx" 2023-10-19T14:21:58.775 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:21:58.776 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:21:58.776 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:21:58.776 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "client.2", 2023-10-19T14:21:58.776 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFlMd+AFxAAS/UTESDt76GmrhhN2o/Fdw==", 2023-10-19T14:21:58.776 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:21:58.776 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T14:21:58.777 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow r", 2023-10-19T14:21:58.777 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow rw", 2023-10-19T14:21:58.777 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow rwx" 2023-10-19T14:21:58.777 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:21:58.777 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:21:58.778 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:21:58.778 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "client.3", 2023-10-19T14:21:58.778 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBkJDFlHZ3fIBAA42B7n8opLS5smXIwJyVYzw==", 2023-10-19T14:21:58.778 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:21:58.778 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow", 2023-10-19T14:21:58.779 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow r", 2023-10-19T14:21:58.779 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow rw", 2023-10-19T14:21:58.779 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow rwx" 2023-10-19T14:21:58.779 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:21:58.779 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:21:58.780 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:21:58.780 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "client.admin", 2023-10-19T14:21:58.780 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBjJDFlBgasFxAAMA24j8KszcXMEpYRQDXj4g==", 2023-10-19T14:21:58.780 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:21:58.780 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow *", 2023-10-19T14:21:58.780 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr": "allow *", 2023-10-19T14:21:58.781 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow *", 2023-10-19T14:21:58.781 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:21:58.781 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:21:58.781 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:21:58.781 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:21:58.781 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "mgr.x", 2023-10-19T14:21:58.782 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBjJDFllFBfMhAA5+KPXWmYZqcESmLObaUbBA==", 2023-10-19T14:21:58.782 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:21:58.782 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow *", 2023-10-19T14:21:58.782 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile mgr", 2023-10-19T14:21:58.782 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:21:58.783 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:21:58.783 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:21:58.783 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:21:58.783 INFO:teuthology.orchestra.run.smithi057.stdout: "entity": "mgr.y", 2023-10-19T14:21:58.783 INFO:teuthology.orchestra.run.smithi057.stdout: "key": "AQBjJDFlxRzJLRAAGjG58o7qXAHvIEIRYwl6QA==", 2023-10-19T14:21:58.784 INFO:teuthology.orchestra.run.smithi057.stdout: "caps": { 2023-10-19T14:21:58.784 INFO:teuthology.orchestra.run.smithi057.stdout: "mds": "allow *", 2023-10-19T14:21:58.784 INFO:teuthology.orchestra.run.smithi057.stdout: "mon": "allow profile mgr", 2023-10-19T14:21:58.784 INFO:teuthology.orchestra.run.smithi057.stdout: "osd": "allow *" 2023-10-19T14:21:58.784 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:21:58.784 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:21:58.785 INFO:teuthology.orchestra.run.smithi057.stdout: ] 2023-10-19T14:21:58.785 INFO:teuthology.orchestra.run.smithi057.stdout:} 2023-10-19T14:21:58.786 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.748+0000 7efdc737d640 1 -- 172.21.15.57:0/1328601392 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7efda406b080 msgr2=0x7efda406d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:58.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.748+0000 7efdc737d640 1 --2- 172.21.15.57:0/1328601392 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7efda406b080 0x7efda406d540 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7efdbc009b40 tx=0x7efdbc005f90 comp rx=0 tx=0).stop 2023-10-19T14:21:58.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.748+0000 7efdc737d640 1 -- 172.21.15.57:0/1328601392 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efdc0142d00 msgr2=0x7efdc0143190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:58.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.748+0000 7efdc737d640 1 --2- 172.21.15.57:0/1328601392 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efdc0142d00 0x7efdc0143190 secure :-1 s=READY pgs=2852 cs=0 l=1 rev1=1 crypto rx=0x7efdb8001e70 tx=0x7efdb8003ba0 comp rx=0 tx=0).stop 2023-10-19T14:21:58.787 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.748+0000 7efdc737d640 1 -- 172.21.15.57:0/1328601392 shutdown_connections 2023-10-19T14:21:58.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.748+0000 7efdc737d640 1 --2- 172.21.15.57:0/1328601392 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7efda406b080 0x7efda406d540 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:58.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.748+0000 7efdc737d640 1 --2- 172.21.15.57:0/1328601392 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efdc0142d00 0x7efdc0143190 unknown :-1 s=CLOSED pgs=2852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:58.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.748+0000 7efdc737d640 1 --2- 172.21.15.57:0/1328601392 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7efdc01483f0 0x7efdc01427c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:58.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.748+0000 7efdc737d640 1 --2- 172.21.15.57:0/1328601392 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efdc015a5e0 0x7efdc015b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:58.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.748+0000 7efdc737d640 1 -- 172.21.15.57:0/1328601392 >> 172.21.15.57:0/1328601392 conn(0x7efdc0069930 msgr2=0x7efdc01583c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:58.788 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.748+0000 7efdc737d640 1 -- 172.21.15.57:0/1328601392 shutdown_connections 2023-10-19T14:21:58.789 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.748+0000 7efdc737d640 1 -- 172.21.15.57:0/1328601392 wait complete. 2023-10-19T14:21:58.789 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T14:21:58.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.896+0000 7f33b3d5e640 1 -- 172.21.15.57:0/1345259288 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f33ac141690 msgr2=0x7f33ac0703e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:58.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.896+0000 7f33b3d5e640 1 --2- 172.21.15.57:0/1345259288 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f33ac141690 0x7f33ac0703e0 secure :-1 s=READY pgs=2853 cs=0 l=1 rev1=1 crypto rx=0x7f33ac0b20f0 tx=0x7f339c02bd30 comp rx=0 tx=0).stop 2023-10-19T14:21:58.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.896+0000 7f33b3d5e640 1 -- 172.21.15.57:0/1345259288 shutdown_connections 2023-10-19T14:21:58.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.896+0000 7f33b3d5e640 1 --2- 172.21.15.57:0/1345259288 >> v2:172.21.15.179:3300/0 conn(0x7f33ac070920 0x7f33ac0affe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:58.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.896+0000 7f33b3d5e640 1 --2- 172.21.15.57:0/1345259288 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f33ac141690 0x7f33ac0703e0 unknown :-1 s=CLOSED pgs=2853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:58.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.896+0000 7f33b3d5e640 1 -- 172.21.15.57:0/1345259288 >> 172.21.15.57:0/1345259288 conn(0x7f33ac069930 msgr2=0x7f33ac069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:58.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.896+0000 7f33b3d5e640 1 -- 172.21.15.57:0/1345259288 shutdown_connections 2023-10-19T14:21:58.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.896+0000 7f33b3d5e640 1 -- 172.21.15.57:0/1345259288 wait complete. 2023-10-19T14:21:58.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.897+0000 7f33b3d5e640 1 Processor -- start 2023-10-19T14:21:58.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.897+0000 7f33b3d5e640 1 -- start start 2023-10-19T14:21:58.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.898+0000 7f33b3d5e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f33ac070920 0x7f33ac13b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:58.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.898+0000 7f33b3d5e640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f33ac13e520 0x7f33ac0c67b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:58.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.898+0000 7f33b3d5e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f33ac141690 0x7f33ac1e4740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:58.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.898+0000 7f33b3d5e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33ac0c7200 con 0x7f33ac070920 2023-10-19T14:21:58.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.898+0000 7f33b3d5e640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33ac0c74e0 con 0x7f33ac141690 2023-10-19T14:21:58.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.898+0000 7f33b3d5e640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33ac1e29e0 con 0x7f33ac13e520 2023-10-19T14:21:58.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.898+0000 7f33b1afa640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f33ac070920 0x7f33ac13b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:58.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.898+0000 7f33b1afa640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f33ac070920 0x7f33ac13b1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51234/0 (socket says 172.21.15.57:51234) 2023-10-19T14:21:58.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.898+0000 7f33b1afa640 1 -- 172.21.15.57:0/978959939 learned_addr learned my addr 172.21.15.57:0/978959939 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:58.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.898+0000 7f33b22fb640 1 --2- 172.21.15.57:0/978959939 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f33ac141690 0x7f33ac1e4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:58.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.898+0000 7f33b12f9640 1 --2- 172.21.15.57:0/978959939 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f33ac13e520 0x7f33ac0c67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:58.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.898+0000 7f33b1afa640 1 -- 172.21.15.57:0/978959939 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f33ac13e520 msgr2=0x7f33ac0c67b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:58.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.898+0000 7f33b1afa640 1 --2- 172.21.15.57:0/978959939 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f33ac13e520 0x7f33ac0c67b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:58.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.898+0000 7f33b1afa640 1 -- 172.21.15.57:0/978959939 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f33ac141690 msgr2=0x7f33ac1e4740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:58.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.898+0000 7f33b1afa640 1 --2- 172.21.15.57:0/978959939 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f33ac141690 0x7f33ac1e4740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:58.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.898+0000 7f33b1afa640 1 -- 172.21.15.57:0/978959939 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f339c009560 con 0x7f33ac070920 2023-10-19T14:21:58.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.898+0000 7f33b22fb640 1 --2- 172.21.15.57:0/978959939 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f33ac141690 0x7f33ac1e4740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:21:58.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.898+0000 7f33b1afa640 1 --2- 172.21.15.57:0/978959939 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f33ac070920 0x7f33ac13b1d0 secure :-1 s=READY pgs=2854 cs=0 l=1 rev1=1 crypto rx=0x7f33a800c430 tx=0x7f33a800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:58.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.899+0000 7f33a27fc640 1 -- 172.21.15.57:0/978959939 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33a8011790 con 0x7f33ac070920 2023-10-19T14:21:58.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.899+0000 7f33a27fc640 1 -- 172.21.15.57:0/978959939 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f33a80119e0 con 0x7f33ac070920 2023-10-19T14:21:58.908 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.899+0000 7f33b3d5e640 1 -- 172.21.15.57:0/978959939 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33ac1e2cc0 con 0x7f33ac070920 2023-10-19T14:21:58.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.899+0000 7f33a27fc640 1 -- 172.21.15.57:0/978959939 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33a8011e30 con 0x7f33ac070920 2023-10-19T14:21:58.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.899+0000 7f33b3d5e640 1 -- 172.21.15.57:0/978959939 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33ac1d6af0 con 0x7f33ac070920 2023-10-19T14:21:58.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.900+0000 7f33b3d5e640 1 -- 172.21.15.57:0/978959939 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33ac1d6c80 con 0x7f33ac070920 2023-10-19T14:21:58.909 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.900+0000 7f33a27fc640 1 -- 172.21.15.57:0/978959939 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 145) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f33a801c460 con 0x7f33ac070920 2023-10-19T14:21:58.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.900+0000 7f33a27fc640 1 --2- 172.21.15.57:0/978959939 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f339006b080 0x7f339006d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:58.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.900+0000 7f33a27fc640 1 -- 172.21.15.57:0/978959939 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(259..259 src has 1..259) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f33a8089990 con 0x7f33ac070920 2023-10-19T14:21:58.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.901+0000 7f33b12f9640 1 --2- 172.21.15.57:0/978959939 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f339006b080 0x7f339006d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:58.910 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.901+0000 7f33b12f9640 1 --2- 172.21.15.57:0/978959939 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f339006b080 0x7f339006d540 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f339c035000 tx=0x7f339c004730 comp rx=0 tx=0).ready entity=mgr.11446 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:58.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:58.905+0000 7f33a27fc640 1 -- 172.21.15.57:0/978959939 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f33a80549b0 con 0x7f33ac070920 2023-10-19T14:21:59.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.081+0000 7f33b3d5e640 1 -- 172.21.15.57:0/978959939 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f33ac0690e0 con 0x7f33ac070920 2023-10-19T14:21:59.084 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.082+0000 7f33a27fc640 1 -- 172.21.15.57:0/978959939 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v259) v1 ==== 74+0+13127 (secure 0 0 0) 0x7f33a8057fd0 con 0x7f33ac070920 2023-10-19T14:21:59.085 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:21:59.085 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":259,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T14:21:58.382238+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T14:21:59.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.083+0000 7f33b3d5e640 1 -- 172.21.15.57:0/978959939 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f339006b080 msgr2=0x7f339006d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:59.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.083+0000 7f33b3d5e640 1 --2- 172.21.15.57:0/978959939 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f339006b080 0x7f339006d540 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f339c035000 tx=0x7f339c004730 comp rx=0 tx=0).stop 2023-10-19T14:21:59.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.083+0000 7f33b3d5e640 1 -- 172.21.15.57:0/978959939 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f33ac070920 msgr2=0x7f33ac13b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:59.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.083+0000 7f33b3d5e640 1 --2- 172.21.15.57:0/978959939 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f33ac070920 0x7f33ac13b1d0 secure :-1 s=READY pgs=2854 cs=0 l=1 rev1=1 crypto rx=0x7f33a800c430 tx=0x7f33a800c8f0 comp rx=0 tx=0).stop 2023-10-19T14:21:59.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.083+0000 7f33b3d5e640 1 -- 172.21.15.57:0/978959939 shutdown_connections 2023-10-19T14:21:59.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.084+0000 7f33b3d5e640 1 --2- 172.21.15.57:0/978959939 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f339006b080 0x7f339006d540 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:59.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.084+0000 7f33b3d5e640 1 --2- 172.21.15.57:0/978959939 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f33ac141690 0x7f33ac1e4740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:59.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.084+0000 7f33b3d5e640 1 --2- 172.21.15.57:0/978959939 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f33ac13e520 0x7f33ac0c67b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:59.088 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.084+0000 7f33b3d5e640 1 --2- 172.21.15.57:0/978959939 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f33ac070920 0x7f33ac13b1d0 unknown :-1 s=CLOSED pgs=2854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:59.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.084+0000 7f33b3d5e640 1 -- 172.21.15.57:0/978959939 >> 172.21.15.57:0/978959939 conn(0x7f33ac069930 msgr2=0x7f33ac069d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:59.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.084+0000 7f33b3d5e640 1 -- 172.21.15.57:0/978959939 shutdown_connections 2023-10-19T14:21:59.089 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.084+0000 7f33b3d5e640 1 -- 172.21.15.57:0/978959939 wait complete. 2023-10-19T14:21:59.097 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-19T12:44:07.371186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-19T14:21:59.098 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-19T14:21:59.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.208+0000 7fa8242c2640 1 -- 172.21.15.57:0/3354324330 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa804003970 msgr2=0x7fa804003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:59.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.208+0000 7fa8242c2640 1 --2- 172.21.15.57:0/3354324330 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa804003970 0x7fa804003d90 secure :-1 s=READY pgs=2856 cs=0 l=1 rev1=1 crypto rx=0x7fa80c00e400 tx=0x7fa8040057f0 comp rx=0 tx=0).stop 2023-10-19T14:21:59.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.208+0000 7fa82285f640 1 -- 172.21.15.57:0/3354324330 reap_dead start 2023-10-19T14:21:59.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.208+0000 7fa8242c2640 1 -- 172.21.15.57:0/3354324330 shutdown_connections 2023-10-19T14:21:59.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.208+0000 7fa8242c2640 1 -- 172.21.15.57:0/3354324330 >> 172.21.15.57:0/3354324330 conn(0x7fa81c069930 msgr2=0x7fa81c069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:59.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.208+0000 7fa8242c2640 1 -- 172.21.15.57:0/3354324330 shutdown_connections 2023-10-19T14:21:59.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.208+0000 7fa8242c2640 1 -- 172.21.15.57:0/3354324330 wait complete. 2023-10-19T14:21:59.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.209+0000 7fa8242c2640 1 Processor -- start 2023-10-19T14:21:59.216 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.209+0000 7fa8242c2640 1 -- start start 2023-10-19T14:21:59.216 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.210+0000 7fa8242c2640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa804003970 0x7fa81c1f1110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:59.216 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.210+0000 7fa8242c2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa81c0cb200 0x7fa81c151ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:59.216 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.210+0000 7fa8242c2640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa81c1522c0 0x7fa81c0ad8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:59.217 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.210+0000 7fa8242c2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa81c1f1650 con 0x7fa81c0cb200 2023-10-19T14:21:59.217 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.210+0000 7fa8242c2640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa81c0b02a0 con 0x7fa804003970 2023-10-19T14:21:59.217 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.210+0000 7fa8242c2640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa81c1520e0 con 0x7fa81c1522c0 2023-10-19T14:21:59.217 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.210+0000 7fa82185d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa81c0cb200 0x7fa81c151ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:59.217 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.210+0000 7fa82185d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa81c0cb200 0x7fa81c151ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51252/0 (socket says 172.21.15.57:51252) 2023-10-19T14:21:59.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.210+0000 7fa82185d640 1 -- 172.21.15.57:0/3923563666 learned_addr learned my addr 172.21.15.57:0/3923563666 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:59.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.210+0000 7fa82205e640 1 --2- 172.21.15.57:0/3923563666 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa804003970 0x7fa81c1f1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:59.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.210+0000 7fa82285f640 1 --2- 172.21.15.57:0/3923563666 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa81c1522c0 0x7fa81c0ad8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:59.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.210+0000 7fa82185d640 1 -- 172.21.15.57:0/3923563666 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa81c1522c0 msgr2=0x7fa81c0ad8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:59.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.210+0000 7fa82185d640 1 --2- 172.21.15.57:0/3923563666 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa81c1522c0 0x7fa81c0ad8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:59.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.210+0000 7fa82185d640 1 -- 172.21.15.57:0/3923563666 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa804003970 msgr2=0x7fa81c1f1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:59.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.210+0000 7fa82185d640 1 --2- 172.21.15.57:0/3923563666 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa804003970 0x7fa81c1f1110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:59.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.210+0000 7fa82185d640 1 -- 172.21.15.57:0/3923563666 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa81c0cb700 con 0x7fa81c0cb200 2023-10-19T14:21:59.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.210+0000 7fa82205e640 1 --2- 172.21.15.57:0/3923563666 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa804003970 0x7fa81c1f1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:21:59.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.210+0000 7fa82185d640 1 --2- 172.21.15.57:0/3923563666 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa81c0cb200 0x7fa81c151ba0 secure :-1 s=READY pgs=2857 cs=0 l=1 rev1=1 crypto rx=0x7fa810000ff0 tx=0x7fa81000ea00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:59.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.210+0000 7fa81affd640 1 -- 172.21.15.57:0/3923563666 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa810010890 con 0x7fa81c0cb200 2023-10-19T14:21:59.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.211+0000 7fa81affd640 1 -- 172.21.15.57:0/3923563666 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fa810010ae0 con 0x7fa81c0cb200 2023-10-19T14:21:59.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.211+0000 7fa8242c2640 1 -- 172.21.15.57:0/3923563666 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa81c155b20 con 0x7fa81c0cb200 2023-10-19T14:21:59.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.211+0000 7fa81affd640 1 -- 172.21.15.57:0/3923563666 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa810010e90 con 0x7fa81c0cb200 2023-10-19T14:21:59.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.211+0000 7fa8242c2640 1 -- 172.21.15.57:0/3923563666 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa81c1560e0 con 0x7fa81c0cb200 2023-10-19T14:21:59.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.211+0000 7fa8242c2640 1 -- 172.21.15.57:0/3923563666 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa81c0b0450 con 0x7fa81c0cb200 2023-10-19T14:21:59.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.212+0000 7fa81affd640 1 -- 172.21.15.57:0/3923563666 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 145) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fa810011460 con 0x7fa81c0cb200 2023-10-19T14:21:59.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.212+0000 7fa81affd640 1 --2- 172.21.15.57:0/3923563666 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fa80806b0b0 0x7fa80806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:59.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.212+0000 7fa81affd640 1 -- 172.21.15.57:0/3923563666 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(259..259 src has 1..259) v4 ==== 5205+0+0 (secure 0 0 0) 0x7fa81008a050 con 0x7fa81c0cb200 2023-10-19T14:21:59.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.213+0000 7fa82205e640 1 --2- 172.21.15.57:0/3923563666 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fa80806b0b0 0x7fa80806d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:59.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.213+0000 7fa82205e640 1 --2- 172.21.15.57:0/3923563666 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fa80806b0b0 0x7fa80806d570 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa80c037000 tx=0x7fa80c02f040 comp rx=0 tx=0).ready entity=mgr.11446 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:59.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.217+0000 7fa81affd640 1 -- 172.21.15.57:0/3923563666 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa81c0b0450 con 0x7fa81c0cb200 2023-10-19T14:21:59.394 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.391+0000 7fa8242c2640 1 -- 172.21.15.57:0/3923563666 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa81c0b0410 con 0x7fa81c0cb200 2023-10-19T14:21:59.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.392+0000 7fa81affd640 1 -- 172.21.15.57:0/3923563666 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v259) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa8100150a0 con 0x7fa81c0cb200 2023-10-19T14:21:59.395 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-19T14:21:59.397 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.394+0000 7fa8242c2640 1 -- 172.21.15.57:0/3923563666 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fa80806b0b0 msgr2=0x7fa80806d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:59.398 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.394+0000 7fa8242c2640 1 --2- 172.21.15.57:0/3923563666 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fa80806b0b0 0x7fa80806d570 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa80c037000 tx=0x7fa80c02f040 comp rx=0 tx=0).stop 2023-10-19T14:21:59.398 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.394+0000 7fa8242c2640 1 -- 172.21.15.57:0/3923563666 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa81c0cb200 msgr2=0x7fa81c151ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:59.398 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.394+0000 7fa8242c2640 1 --2- 172.21.15.57:0/3923563666 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa81c0cb200 0x7fa81c151ba0 secure :-1 s=READY pgs=2857 cs=0 l=1 rev1=1 crypto rx=0x7fa810000ff0 tx=0x7fa81000ea00 comp rx=0 tx=0).stop 2023-10-19T14:21:59.398 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.394+0000 7fa8242c2640 1 -- 172.21.15.57:0/3923563666 shutdown_connections 2023-10-19T14:21:59.398 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.394+0000 7fa8242c2640 1 --2- 172.21.15.57:0/3923563666 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fa80806b0b0 0x7fa80806d570 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:59.399 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.394+0000 7fa8242c2640 1 --2- 172.21.15.57:0/3923563666 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa81c1522c0 0x7fa81c0ad8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:59.399 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.394+0000 7fa8242c2640 1 --2- 172.21.15.57:0/3923563666 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa81c0cb200 0x7fa81c151ba0 unknown :-1 s=CLOSED pgs=2857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:59.399 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.394+0000 7fa8242c2640 1 --2- 172.21.15.57:0/3923563666 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa804003970 0x7fa81c1f1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:59.399 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.394+0000 7fa8242c2640 1 -- 172.21.15.57:0/3923563666 >> 172.21.15.57:0/3923563666 conn(0x7fa81c069930 msgr2=0x7fa81c153340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:59.399 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.394+0000 7fa8242c2640 1 -- 172.21.15.57:0/3923563666 shutdown_connections 2023-10-19T14:21:59.400 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.394+0000 7fa8242c2640 1 -- 172.21.15.57:0/3923563666 wait complete. 2023-10-19T14:21:59.408 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-19T14:21:59.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.515+0000 7f3381133640 1 -- 172.21.15.57:0/1726997230 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3368003970 msgr2=0x7f3368003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:59.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.515+0000 7f3381133640 1 --2- 172.21.15.57:0/1726997230 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3368003970 0x7f3368003d90 secure :-1 s=READY pgs=2859 cs=0 l=1 rev1=1 crypto rx=0x7f337400c260 tx=0x7f33680057f0 comp rx=0 tx=0).stop 2023-10-19T14:21:59.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.515+0000 7f337b59e640 1 -- 172.21.15.57:0/1726997230 reap_dead start 2023-10-19T14:21:59.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.515+0000 7f3381133640 1 -- 172.21.15.57:0/1726997230 shutdown_connections 2023-10-19T14:21:59.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.515+0000 7f3381133640 1 -- 172.21.15.57:0/1726997230 >> 172.21.15.57:0/1726997230 conn(0x7f337c069930 msgr2=0x7f337c069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:59.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.515+0000 7f3381133640 1 -- 172.21.15.57:0/1726997230 shutdown_connections 2023-10-19T14:21:59.519 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.515+0000 7f3381133640 1 -- 172.21.15.57:0/1726997230 wait complete. 2023-10-19T14:21:59.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.516+0000 7f3381133640 1 Processor -- start 2023-10-19T14:21:59.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.516+0000 7f3381133640 1 -- start start 2023-10-19T14:21:59.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.517+0000 7f3381133640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3368003970 0x7f337c156360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:59.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.517+0000 7f3381133640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f337c1390b0 0x7f337c158de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:59.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.517+0000 7f3381133640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f337c13bd30 0x7f337c13c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:59.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.517+0000 7f3381133640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f337c13c6f0 con 0x7f337c1390b0 2023-10-19T14:21:59.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.517+0000 7f3381133640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f337c159750 con 0x7f337c13bd30 2023-10-19T14:21:59.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.517+0000 7f3381133640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f337c0ac710 con 0x7f3368003970 2023-10-19T14:21:59.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.517+0000 7f337a59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f337c1390b0 0x7f337c158de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:59.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.517+0000 7f337a59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f337c1390b0 0x7f337c158de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51266/0 (socket says 172.21.15.57:51266) 2023-10-19T14:21:59.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.517+0000 7f337a59c640 1 -- 172.21.15.57:0/2297807660 learned_addr learned my addr 172.21.15.57:0/2297807660 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:59.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.517+0000 7f337ad9d640 1 --2- 172.21.15.57:0/2297807660 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3368003970 0x7f337c156360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:59.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.517+0000 7f337b59e640 1 --2- 172.21.15.57:0/2297807660 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f337c13bd30 0x7f337c13c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:59.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.517+0000 7f337a59c640 1 -- 172.21.15.57:0/2297807660 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3368003970 msgr2=0x7f337c156360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:59.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.517+0000 7f337a59c640 1 --2- 172.21.15.57:0/2297807660 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3368003970 0x7f337c156360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:59.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.517+0000 7f337a59c640 1 -- 172.21.15.57:0/2297807660 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f337c13bd30 msgr2=0x7f337c13c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:59.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.517+0000 7f337a59c640 1 --2- 172.21.15.57:0/2297807660 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f337c13bd30 0x7f337c13c1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:59.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.517+0000 7f337a59c640 1 -- 172.21.15.57:0/2297807660 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f337c0ac9f0 con 0x7f337c1390b0 2023-10-19T14:21:59.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.517+0000 7f337ad9d640 1 --2- 172.21.15.57:0/2297807660 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3368003970 0x7f337c156360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:21:59.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.517+0000 7f337b59e640 1 --2- 172.21.15.57:0/2297807660 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f337c13bd30 0x7f337c13c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:21:59.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.518+0000 7f337a59c640 1 --2- 172.21.15.57:0/2297807660 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f337c1390b0 0x7f337c158de0 secure :-1 s=READY pgs=2860 cs=0 l=1 rev1=1 crypto rx=0x7f3364011ab0 tx=0x7f3364011f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:59.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.518+0000 7f33737fe640 1 -- 172.21.15.57:0/2297807660 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f336400fe70 con 0x7f337c1390b0 2023-10-19T14:21:59.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.518+0000 7f33737fe640 1 -- 172.21.15.57:0/2297807660 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f3364004050 con 0x7f337c1390b0 2023-10-19T14:21:59.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.518+0000 7f3381133640 1 -- 172.21.15.57:0/2297807660 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f337c0accd0 con 0x7f337c1390b0 2023-10-19T14:21:59.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.518+0000 7f33737fe640 1 -- 172.21.15.57:0/2297807660 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3364004440 con 0x7f337c1390b0 2023-10-19T14:21:59.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.518+0000 7f3381133640 1 -- 172.21.15.57:0/2297807660 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f337c159500 con 0x7f337c1390b0 2023-10-19T14:21:59.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.519+0000 7f3381133640 1 -- 172.21.15.57:0/2297807660 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f337c0ace60 con 0x7f337c1390b0 2023-10-19T14:21:59.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.519+0000 7f33737fe640 1 -- 172.21.15.57:0/2297807660 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 145) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f33640035f0 con 0x7f337c1390b0 2023-10-19T14:21:59.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.520+0000 7f33737fe640 1 --2- 172.21.15.57:0/2297807660 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f335c06b0b0 0x7f335c06d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:59.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.520+0000 7f33737fe640 1 -- 172.21.15.57:0/2297807660 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(259..259 src has 1..259) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f336408a630 con 0x7f337c1390b0 2023-10-19T14:21:59.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.520+0000 7f337ad9d640 1 --2- 172.21.15.57:0/2297807660 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f335c06b0b0 0x7f335c06d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:59.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.520+0000 7f337ad9d640 1 --2- 172.21.15.57:0/2297807660 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f335c06b0b0 0x7f335c06d570 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3374034000 tx=0x7f3374001a70 comp rx=0 tx=0).ready entity=mgr.11446 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:59.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.524+0000 7f33737fe640 1 -- 172.21.15.57:0/2297807660 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f3364055510 con 0x7f337c1390b0 2023-10-19T14:21:59.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.701+0000 7f3381133640 1 -- 172.21.15.57:0/2297807660 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f337c0ac510 con 0x7f337c1390b0 2023-10-19T14:21:59.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.701+0000 7f33737fe640 1 -- 172.21.15.57:0/2297807660 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v259) v1 ==== 84+0+65 (secure 0 0 0) 0x7f336408a090 con 0x7f337c1390b0 2023-10-19T14:21:59.704 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:21:59.704 INFO:teuthology.orchestra.run.smithi057.stdout:[ 2023-10-19T14:21:59.705 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:21:59.705 INFO:teuthology.orchestra.run.smithi057.stdout: "poolnum": 1, 2023-10-19T14:21:59.705 INFO:teuthology.orchestra.run.smithi057.stdout: "poolname": ".mgr" 2023-10-19T14:21:59.705 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:21:59.705 INFO:teuthology.orchestra.run.smithi057.stdout:] 2023-10-19T14:21:59.706 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.703+0000 7f3381133640 1 -- 172.21.15.57:0/2297807660 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f335c06b0b0 msgr2=0x7f335c06d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:59.707 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.703+0000 7f3381133640 1 --2- 172.21.15.57:0/2297807660 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f335c06b0b0 0x7f335c06d570 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3374034000 tx=0x7f3374001a70 comp rx=0 tx=0).stop 2023-10-19T14:21:59.707 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.703+0000 7f3381133640 1 -- 172.21.15.57:0/2297807660 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f337c1390b0 msgr2=0x7f337c158de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:59.707 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.703+0000 7f3381133640 1 --2- 172.21.15.57:0/2297807660 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f337c1390b0 0x7f337c158de0 secure :-1 s=READY pgs=2860 cs=0 l=1 rev1=1 crypto rx=0x7f3364011ab0 tx=0x7f3364011f70 comp rx=0 tx=0).stop 2023-10-19T14:21:59.707 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.703+0000 7f3381133640 1 -- 172.21.15.57:0/2297807660 shutdown_connections 2023-10-19T14:21:59.707 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.703+0000 7f3381133640 1 --2- 172.21.15.57:0/2297807660 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f335c06b0b0 0x7f335c06d570 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:59.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.703+0000 7f3381133640 1 --2- 172.21.15.57:0/2297807660 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f337c13bd30 0x7f337c13c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:59.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.703+0000 7f3381133640 1 --2- 172.21.15.57:0/2297807660 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f337c1390b0 0x7f337c158de0 unknown :-1 s=CLOSED pgs=2860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:59.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.703+0000 7f3381133640 1 --2- 172.21.15.57:0/2297807660 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3368003970 0x7f337c156360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:59.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.703+0000 7f3381133640 1 -- 172.21.15.57:0/2297807660 >> 172.21.15.57:0/2297807660 conn(0x7f337c069930 msgr2=0x7f337c0b1430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:59.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.703+0000 7f3381133640 1 -- 172.21.15.57:0/2297807660 shutdown_connections 2023-10-19T14:21:59.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.703+0000 7f3381133640 1 -- 172.21.15.57:0/2297807660 wait complete. 2023-10-19T14:21:59.716 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-19T14:21:59.717 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-19T14:21:59.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.828+0000 7f7eef305640 1 -- 172.21.15.57:0/1144490961 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7ee813d360 msgr2=0x7f7ee815a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:59.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.828+0000 7f7eef305640 1 --2- 172.21.15.57:0/1144490961 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7ee813d360 0x7f7ee815a9f0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f7ee0005fd0 tx=0x7f7ee002c2e0 comp rx=0 tx=0).stop 2023-10-19T14:21:59.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.828+0000 7f7eef305640 1 -- 172.21.15.57:0/1144490961 shutdown_connections 2023-10-19T14:21:59.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.828+0000 7f7eef305640 1 --2- 172.21.15.57:0/1144490961 >> v2:172.21.15.179:3300/0 conn(0x7f7ee8140660 0x7f7ee80c97c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:59.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.828+0000 7f7eef305640 1 --2- 172.21.15.57:0/1144490961 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7ee813d360 0x7f7ee815a9f0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:59.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.828+0000 7f7eef305640 1 -- 172.21.15.57:0/1144490961 >> 172.21.15.57:0/1144490961 conn(0x7f7ee8069930 msgr2=0x7f7ee8069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:21:59.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.828+0000 7f7eef305640 1 -- 172.21.15.57:0/1144490961 shutdown_connections 2023-10-19T14:21:59.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.828+0000 7f7eef305640 1 -- 172.21.15.57:0/1144490961 wait complete. 2023-10-19T14:21:59.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.829+0000 7f7eef305640 1 Processor -- start 2023-10-19T14:21:59.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.829+0000 7f7eef305640 1 -- start start 2023-10-19T14:21:59.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.830+0000 7f7eef305640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ee80c9d00 0x7f7ee8156580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:59.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.830+0000 7f7eef305640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7ee813d360 0x7f7ee80b0060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:59.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.830+0000 7f7eef305640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7ee8140660 0x7f7ee81e0f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:59.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.830+0000 7f7eef305640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee80b0ab0 con 0x7f7ee80c9d00 2023-10-19T14:21:59.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.830+0000 7f7eef305640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee80b0d90 con 0x7f7ee8140660 2023-10-19T14:21:59.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.830+0000 7f7eef305640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee80b1070 con 0x7f7ee813d360 2023-10-19T14:21:59.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.830+0000 7f7eed0a1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ee80c9d00 0x7f7ee8156580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:59.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.830+0000 7f7eed0a1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ee80c9d00 0x7f7ee8156580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51280/0 (socket says 172.21.15.57:51280) 2023-10-19T14:21:59.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.830+0000 7f7eed0a1640 1 -- 172.21.15.57:0/3869929332 learned_addr learned my addr 172.21.15.57:0/3869929332 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:21:59.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.830+0000 7f7eec8a0640 1 --2- 172.21.15.57:0/3869929332 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7ee813d360 0x7f7ee80b0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:59.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.830+0000 7f7eed8a2640 1 --2- 172.21.15.57:0/3869929332 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7ee8140660 0x7f7ee81e0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:59.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.830+0000 7f7eed0a1640 1 -- 172.21.15.57:0/3869929332 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7ee813d360 msgr2=0x7f7ee80b0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:59.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.830+0000 7f7eed0a1640 1 --2- 172.21.15.57:0/3869929332 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7ee813d360 0x7f7ee80b0060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:59.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.830+0000 7f7eed0a1640 1 -- 172.21.15.57:0/3869929332 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7ee8140660 msgr2=0x7f7ee81e0f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:21:59.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.830+0000 7f7eed0a1640 1 --2- 172.21.15.57:0/3869929332 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7ee8140660 0x7f7ee81e0f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:21:59.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.830+0000 7f7eed0a1640 1 -- 172.21.15.57:0/3869929332 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ee0009900 con 0x7f7ee80c9d00 2023-10-19T14:21:59.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.830+0000 7f7eed0a1640 1 --2- 172.21.15.57:0/3869929332 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ee80c9d00 0x7f7ee8156580 secure :-1 s=READY pgs=2861 cs=0 l=1 rev1=1 crypto rx=0x7f7ee0006010 tx=0x7f7ee0035b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:59.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.830+0000 7f7eddffb640 1 -- 172.21.15.57:0/3869929332 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ee0004240 con 0x7f7ee80c9d00 2023-10-19T14:21:59.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.831+0000 7f7eddffb640 1 -- 172.21.15.57:0/3869929332 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f7ee0004490 con 0x7f7ee80c9d00 2023-10-19T14:21:59.838 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.831+0000 7f7eef305640 1 -- 172.21.15.57:0/3869929332 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ee80b1350 con 0x7f7ee80c9d00 2023-10-19T14:21:59.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.831+0000 7f7eddffb640 1 -- 172.21.15.57:0/3869929332 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ee0004840 con 0x7f7ee80c9d00 2023-10-19T14:21:59.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.831+0000 7f7eef305640 1 -- 172.21.15.57:0/3869929332 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ee8156ac0 con 0x7f7ee80c9d00 2023-10-19T14:21:59.839 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.832+0000 7f7eef305640 1 -- 172.21.15.57:0/3869929332 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ee81e2fd0 con 0x7f7ee80c9d00 2023-10-19T14:21:59.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.832+0000 7f7eddffb640 1 -- 172.21.15.57:0/3869929332 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 145) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f7ee003b050 con 0x7f7ee80c9d00 2023-10-19T14:21:59.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.832+0000 7f7eddffb640 1 --2- 172.21.15.57:0/3869929332 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f7ecc06b0b0 0x7f7ecc06d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:21:59.840 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.833+0000 7f7eddffb640 1 -- 172.21.15.57:0/3869929332 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(259..259 src has 1..259) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f7ee00ac100 con 0x7f7ee80c9d00 2023-10-19T14:21:59.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.833+0000 7f7eec8a0640 1 --2- 172.21.15.57:0/3869929332 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f7ecc06b0b0 0x7f7ecc06d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:21:59.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.833+0000 7f7eec8a0640 1 --2- 172.21.15.57:0/3869929332 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f7ecc06b0b0 0x7f7ecc06d570 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f7ee81f22e0 tx=0x7f7ed8010040 comp rx=0 tx=0).ready entity=mgr.11446 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:21:59.841 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:21:59.837+0000 7f7eddffb640 1 -- 172.21.15.57:0/3869929332 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7ee81e2fd0 con 0x7f7ee80c9d00 2023-10-19T14:22:00.015 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:00.013+0000 7f7eef305640 1 -- 172.21.15.57:0/3869929332 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f7ee81e33b0 con 0x7f7ee80c9d00 2023-10-19T14:22:00.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:00.385+0000 7f7eddffb640 1 -- 172.21.15.57:0/3869929332 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v260) v1 ==== 138+0+0 (secure 0 0 0) 0x7f7ee81e33b0 con 0x7f7ee80c9d00 2023-10-19T14:22:00.388 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'cephfs_metadata' created 2023-10-19T14:22:00.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:00.387+0000 7f7eef305640 1 -- 172.21.15.57:0/3869929332 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f7ecc06b0b0 msgr2=0x7f7ecc06d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:00.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:00.387+0000 7f7eef305640 1 --2- 172.21.15.57:0/3869929332 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f7ecc06b0b0 0x7f7ecc06d570 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f7ee81f22e0 tx=0x7f7ed8010040 comp rx=0 tx=0).stop 2023-10-19T14:22:00.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:00.387+0000 7f7eef305640 1 -- 172.21.15.57:0/3869929332 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ee80c9d00 msgr2=0x7f7ee8156580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:00.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:00.387+0000 7f7eef305640 1 --2- 172.21.15.57:0/3869929332 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ee80c9d00 0x7f7ee8156580 secure :-1 s=READY pgs=2861 cs=0 l=1 rev1=1 crypto rx=0x7f7ee0006010 tx=0x7f7ee0035b50 comp rx=0 tx=0).stop 2023-10-19T14:22:00.391 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:00.387+0000 7f7eef305640 1 -- 172.21.15.57:0/3869929332 shutdown_connections 2023-10-19T14:22:00.391 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:00.387+0000 7f7eef305640 1 --2- 172.21.15.57:0/3869929332 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f7ecc06b0b0 0x7f7ecc06d570 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:00.391 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:00.387+0000 7f7eef305640 1 --2- 172.21.15.57:0/3869929332 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7ee8140660 0x7f7ee81e0f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:00.391 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:00.387+0000 7f7eef305640 1 --2- 172.21.15.57:0/3869929332 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7ee813d360 0x7f7ee80b0060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:00.391 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:00.387+0000 7f7eef305640 1 --2- 172.21.15.57:0/3869929332 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ee80c9d00 0x7f7ee8156580 unknown :-1 s=CLOSED pgs=2861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:00.392 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:00.387+0000 7f7eef305640 1 -- 172.21.15.57:0/3869929332 >> 172.21.15.57:0/3869929332 conn(0x7f7ee8069930 msgr2=0x7f7ee813f1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:00.392 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:00.387+0000 7f7eef305640 1 -- 172.21.15.57:0/3869929332 shutdown_connections 2023-10-19T14:22:00.392 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:00.387+0000 7f7eef305640 1 -- 172.21.15.57:0/3869929332 wait complete. 2023-10-19T14:22:00.404 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-19T14:22:00.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:00.519+0000 7f5fa6dc0640 1 -- 172.21.15.57:0/533506961 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5f90005e10 msgr2=0x7f5f90006290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:00.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:00.519+0000 7f5fa6dc0640 1 --2- 172.21.15.57:0/533506961 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5f90005e10 0x7f5f90006290 secure :-1 s=READY pgs=2863 cs=0 l=1 rev1=1 crypto rx=0x7f5f9800c610 tx=0x7f5f90006900 comp rx=0 tx=0).stop 2023-10-19T14:22:00.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:00.519+0000 7f5fa535d640 1 -- 172.21.15.57:0/533506961 reap_dead start 2023-10-19T14:22:00.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:00.519+0000 7f5fa6dc0640 1 -- 172.21.15.57:0/533506961 shutdown_connections 2023-10-19T14:22:00.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:00.519+0000 7f5fa6dc0640 1 -- 172.21.15.57:0/533506961 >> 172.21.15.57:0/533506961 conn(0x7f5fa0069930 msgr2=0x7f5fa0069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:00.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:00.519+0000 7f5fa6dc0640 1 -- 172.21.15.57:0/533506961 shutdown_connections 2023-10-19T14:22:00.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:00.519+0000 7f5fa6dc0640 1 -- 172.21.15.57:0/533506961 wait complete. 2023-10-19T14:22:00.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:00.520+0000 7f5fa6dc0640 1 Processor -- start 2023-10-19T14:22:00.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:00.520+0000 7f5fa6dc0640 1 -- start start 2023-10-19T14:22:00.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:00.521+0000 7f5fa6dc0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5f90005e10 0x7f5fa0155c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:00.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:00.521+0000 7f5fa6dc0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5fa014f4c0 0x7f5fa014e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:00.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:00.521+0000 7f5fa6dc0640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5fa0154840 0x7f5fa01e50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:00.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:00.521+0000 7f5fa6dc0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa0156a20 con 0x7f5f90005e10 2023-10-19T14:22:00.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:00.521+0000 7f5fa6dc0640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa0156460 con 0x7f5fa014f4c0 2023-10-19T14:22:00.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:00.521+0000 7f5fa6dc0640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa0156740 con 0x7f5fa0154840 2023-10-19T14:22:00.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:00.521+0000 7f5fa4b5c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5f90005e10 0x7f5fa0155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:00.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:00.521+0000 7f5fa4b5c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5f90005e10 0x7f5fa0155c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51288/0 (socket says 172.21.15.57:51288) 2023-10-19T14:22:00.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:00.521+0000 7f5fa4b5c640 1 -- 172.21.15.57:0/3443801003 learned_addr learned my addr 172.21.15.57:0/3443801003 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:22:00.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:00.522+0000 7f5fa535d640 1 --2- 172.21.15.57:0/3443801003 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5fa0154840 0x7f5fa01e50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:00.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:00.522+0000 7f5f9ffff640 1 --2- 172.21.15.57:0/3443801003 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5fa014f4c0 0x7f5fa014e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:00.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:00.522+0000 7f5fa4b5c640 1 -- 172.21.15.57:0/3443801003 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5fa0154840 msgr2=0x7f5fa01e50a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:00.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:00.522+0000 7f5fa4b5c640 1 --2- 172.21.15.57:0/3443801003 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5fa0154840 0x7f5fa01e50a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:00.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:00.522+0000 7f5fa4b5c640 1 -- 172.21.15.57:0/3443801003 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5fa014f4c0 msgr2=0x7f5fa014e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:00.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:00.522+0000 7f5fa4b5c640 1 --2- 172.21.15.57:0/3443801003 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5fa014f4c0 0x7f5fa014e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:00.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:00.522+0000 7f5fa4b5c640 1 -- 172.21.15.57:0/3443801003 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fa0154040 con 0x7f5f90005e10 2023-10-19T14:22:00.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:00.522+0000 7f5fa535d640 1 --2- 172.21.15.57:0/3443801003 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5fa0154840 0x7f5fa01e50a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:22:00.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:00.522+0000 7f5f9ffff640 1 --2- 172.21.15.57:0/3443801003 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5fa014f4c0 0x7f5fa014e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:22:00.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:00.522+0000 7f5fa4b5c640 1 --2- 172.21.15.57:0/3443801003 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5f90005e10 0x7f5fa0155c10 secure :-1 s=READY pgs=2864 cs=0 l=1 rev1=1 crypto rx=0x7f5f94004650 tx=0x7f5f9400ba40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:00.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:00.522+0000 7f5f9d7fa640 1 -- 172.21.15.57:0/3443801003 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f940041d0 con 0x7f5f90005e10 2023-10-19T14:22:00.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:00.522+0000 7f5f9d7fa640 1 -- 172.21.15.57:0/3443801003 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f5f94015070 con 0x7f5f90005e10 2023-10-19T14:22:00.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:00.522+0000 7f5fa6dc0640 1 -- 172.21.15.57:0/3443801003 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fa0154320 con 0x7f5f90005e10 2023-10-19T14:22:00.532 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:00.522+0000 7f5f9d7fa640 1 -- 172.21.15.57:0/3443801003 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f94010ad0 con 0x7f5f90005e10 2023-10-19T14:22:00.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:00.522+0000 7f5fa6dc0640 1 -- 172.21.15.57:0/3443801003 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fa014e680 con 0x7f5f90005e10 2023-10-19T14:22:00.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:00.523+0000 7f5fa6dc0640 1 -- 172.21.15.57:0/3443801003 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fa00690e0 con 0x7f5f90005e10 2023-10-19T14:22:00.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:00.524+0000 7f5f9d7fa640 1 -- 172.21.15.57:0/3443801003 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 145) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f5f94017030 con 0x7f5f90005e10 2023-10-19T14:22:00.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:00.525+0000 7f5f9d7fa640 1 --2- 172.21.15.57:0/3443801003 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f5f8406b080 0x7f5f8406d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:00.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:00.525+0000 7f5f9d7fa640 1 -- 172.21.15.57:0/3443801003 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(260..260 src has 1..260) v4 ==== 5575+0+0 (secure 0 0 0) 0x7f5f9408a850 con 0x7f5f90005e10 2023-10-19T14:22:00.533 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:00.525+0000 7f5f9ffff640 1 --2- 172.21.15.57:0/3443801003 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f5f8406b080 0x7f5f8406d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:00.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:00.526+0000 7f5f9ffff640 1 --2- 172.21.15.57:0/3443801003 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f5f8406b080 0x7f5f8406d540 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f5fa01e5c40 tx=0x7f5f8c007040 comp rx=0 tx=0).ready entity=mgr.11446 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:00.534 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:00.529+0000 7f5f9d7fa640 1 -- 172.21.15.57:0/3443801003 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f5f94055610 con 0x7f5f90005e10 2023-10-19T14:22:00.707 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:00.705+0000 7f5fa6dc0640 1 -- 172.21.15.57:0/3443801003 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f5fa006f700 con 0x7f5f90005e10 2023-10-19T14:22:01.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:01.393+0000 7f5f9d7fa640 1 -- 172.21.15.57:0/3443801003 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v261) v1 ==== 170+0+0 (secure 0 0 0) 0x7f5f94058c30 con 0x7f5f90005e10 2023-10-19T14:22:01.396 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'cephfs_data' created 2023-10-19T14:22:01.399 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:01.396+0000 7f5fa6dc0640 1 -- 172.21.15.57:0/3443801003 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f5f8406b080 msgr2=0x7f5f8406d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:01.399 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:01.396+0000 7f5fa6dc0640 1 --2- 172.21.15.57:0/3443801003 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f5f8406b080 0x7f5f8406d540 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f5fa01e5c40 tx=0x7f5f8c007040 comp rx=0 tx=0).stop 2023-10-19T14:22:01.399 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:01.396+0000 7f5fa6dc0640 1 -- 172.21.15.57:0/3443801003 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5f90005e10 msgr2=0x7f5fa0155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:01.400 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:01.396+0000 7f5fa6dc0640 1 --2- 172.21.15.57:0/3443801003 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5f90005e10 0x7f5fa0155c10 secure :-1 s=READY pgs=2864 cs=0 l=1 rev1=1 crypto rx=0x7f5f94004650 tx=0x7f5f9400ba40 comp rx=0 tx=0).stop 2023-10-19T14:22:01.400 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:01.396+0000 7f5fa6dc0640 1 -- 172.21.15.57:0/3443801003 shutdown_connections 2023-10-19T14:22:01.400 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:01.396+0000 7f5fa6dc0640 1 --2- 172.21.15.57:0/3443801003 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f5f8406b080 0x7f5f8406d540 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:01.400 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:01.396+0000 7f5fa6dc0640 1 --2- 172.21.15.57:0/3443801003 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f5fa0154840 0x7f5fa01e50a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:01.400 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:01.396+0000 7f5fa6dc0640 1 --2- 172.21.15.57:0/3443801003 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5fa014f4c0 0x7f5fa014e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:01.401 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:01.396+0000 7f5fa6dc0640 1 --2- 172.21.15.57:0/3443801003 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5f90005e10 0x7f5fa0155c10 unknown :-1 s=CLOSED pgs=2864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:01.401 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:01.396+0000 7f5fa6dc0640 1 -- 172.21.15.57:0/3443801003 >> 172.21.15.57:0/3443801003 conn(0x7f5fa0069930 msgr2=0x7f5fa01578d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:01.401 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:01.396+0000 7f5fa6dc0640 1 -- 172.21.15.57:0/3443801003 shutdown_connections 2023-10-19T14:22:01.401 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:01.396+0000 7f5fa6dc0640 1 -- 172.21.15.57:0/3443801003 wait complete. 2023-10-19T14:22:01.425 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-19T14:22:01.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:01.539+0000 7f49501c7640 1 -- 172.21.15.57:0/2299502155 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4948069e30 msgr2=0x7f49480b1620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:01.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:01.539+0000 7f49501c7640 1 --2- 172.21.15.57:0/2299502155 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4948069e30 0x7f49480b1620 secure :-1 s=READY pgs=2866 cs=0 l=1 rev1=1 crypto rx=0x7f49480cb530 tx=0x7f494002c080 comp rx=0 tx=0).stop 2023-10-19T14:22:01.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:01.539+0000 7f49501c7640 1 -- 172.21.15.57:0/2299502155 shutdown_connections 2023-10-19T14:22:01.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:01.539+0000 7f49501c7640 1 --2- 172.21.15.57:0/2299502155 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4948069e30 0x7f49480b1620 unknown :-1 s=CLOSED pgs=2866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:01.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:01.539+0000 7f49501c7640 1 --2- 172.21.15.57:0/2299502155 >> v2:172.21.15.179:3300/0 conn(0x7f4948158d70 0x7f49480ce090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:01.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:01.540+0000 7f49501c7640 1 -- 172.21.15.57:0/2299502155 >> 172.21.15.57:0/2299502155 conn(0x7f49480af7a0 msgr2=0x7f49480afb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:01.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:01.540+0000 7f49501c7640 1 -- 172.21.15.57:0/2299502155 shutdown_connections 2023-10-19T14:22:01.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:01.540+0000 7f49501c7640 1 -- 172.21.15.57:0/2299502155 wait complete. 2023-10-19T14:22:01.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:01.541+0000 7f49501c7640 1 Processor -- start 2023-10-19T14:22:01.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:01.541+0000 7f49501c7640 1 -- start start 2023-10-19T14:22:01.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:01.541+0000 7f49501c7640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4948069e30 0x7f49481f4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:01.547 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:01.541+0000 7f49501c7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49480ce5d0 0x7f494806c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:01.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:01.541+0000 7f49501c7640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4948158d70 0x7f49480acb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:01.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:01.542+0000 7f49501c7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49481f1730 con 0x7f49480ce5d0 2023-10-19T14:22:01.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:01.542+0000 7f49501c7640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49481f51d0 con 0x7f4948158d70 2023-10-19T14:22:01.548 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:01.542+0000 7f49501c7640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49481f54b0 con 0x7f4948069e30 2023-10-19T14:22:01.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:01.542+0000 7f494d762640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49480ce5d0 0x7f494806c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:01.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:01.542+0000 7f494d762640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49480ce5d0 0x7f494806c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51310/0 (socket says 172.21.15.57:51310) 2023-10-19T14:22:01.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:01.542+0000 7f494d762640 1 -- 172.21.15.57:0/297370063 learned_addr learned my addr 172.21.15.57:0/297370063 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:22:01.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:01.542+0000 7f494e764640 1 --2- 172.21.15.57:0/297370063 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4948158d70 0x7f49480acb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:01.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:01.542+0000 7f494df63640 1 --2- 172.21.15.57:0/297370063 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4948069e30 0x7f49481f4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:01.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:01.542+0000 7f494d762640 1 -- 172.21.15.57:0/297370063 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4948069e30 msgr2=0x7f49481f4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:01.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:01.542+0000 7f494d762640 1 --2- 172.21.15.57:0/297370063 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4948069e30 0x7f49481f4bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:01.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:01.542+0000 7f494d762640 1 -- 172.21.15.57:0/297370063 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4948158d70 msgr2=0x7f49480acb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:01.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:01.542+0000 7f494d762640 1 --2- 172.21.15.57:0/297370063 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4948158d70 0x7f49480acb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:01.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:01.542+0000 7f494d762640 1 -- 172.21.15.57:0/297370063 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4940009840 con 0x7f49480ce5d0 2023-10-19T14:22:01.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:01.542+0000 7f494df63640 1 --2- 172.21.15.57:0/297370063 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4948069e30 0x7f49481f4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:22:01.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:01.542+0000 7f494e764640 1 --2- 172.21.15.57:0/297370063 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4948158d70 0x7f49480acb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:22:01.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:01.542+0000 7f494d762640 1 --2- 172.21.15.57:0/297370063 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49480ce5d0 0x7f494806c8c0 secure :-1 s=READY pgs=2867 cs=0 l=1 rev1=1 crypto rx=0x7f493800c440 tx=0x7f4938001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:01.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:01.542+0000 7f493e7fc640 1 -- 172.21.15.57:0/297370063 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4938004230 con 0x7f49480ce5d0 2023-10-19T14:22:01.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:01.542+0000 7f49501c7640 1 -- 172.21.15.57:0/297370063 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49481f5790 con 0x7f49480ce5d0 2023-10-19T14:22:01.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:01.542+0000 7f493e7fc640 1 -- 172.21.15.57:0/297370063 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f4938004480 con 0x7f49480ce5d0 2023-10-19T14:22:01.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:01.542+0000 7f493e7fc640 1 -- 172.21.15.57:0/297370063 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4938003850 con 0x7f49480ce5d0 2023-10-19T14:22:01.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:01.542+0000 7f49501c7640 1 -- 172.21.15.57:0/297370063 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49480ad050 con 0x7f49480ce5d0 2023-10-19T14:22:01.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:01.543+0000 7f49501c7640 1 -- 172.21.15.57:0/297370063 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49480690e0 con 0x7f49480ce5d0 2023-10-19T14:22:01.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:01.544+0000 7f493e7fc640 1 -- 172.21.15.57:0/297370063 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 145) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f49380039b0 con 0x7f49480ce5d0 2023-10-19T14:22:01.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:01.545+0000 7f493e7fc640 1 --2- 172.21.15.57:0/297370063 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f492c06b080 0x7f492c06d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:01.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:01.545+0000 7f493e7fc640 1 -- 172.21.15.57:0/297370063 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(261..261 src has 1..261) v4 ==== 5957+0+0 (secure 0 0 0) 0x7f4938089370 con 0x7f49480ce5d0 2023-10-19T14:22:01.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:01.545+0000 7f494df63640 1 --2- 172.21.15.57:0/297370063 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f492c06b080 0x7f492c06d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:01.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:01.545+0000 7f494df63640 1 --2- 172.21.15.57:0/297370063 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f492c06b080 0x7f492c06d540 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4944005fd0 tx=0x7f4944005e20 comp rx=0 tx=0).ready entity=mgr.11446 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:01.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:01.549+0000 7f493e7fc640 1 -- 172.21.15.57:0/297370063 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4938053f70 con 0x7f49480ce5d0 2023-10-19T14:22:01.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:01.761+0000 7f49501c7640 1 -- 172.21.15.57:0/297370063 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f494814fb20 con 0x7f49480ce5d0 2023-10-19T14:22:02.397 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.394+0000 7f493e7fc640 1 -- 172.21.15.57:0/297370063 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 35 and data pool 36 v114) v1 ==== 174+0+0 (secure 0 0 0) 0x7f4938057590 con 0x7f49480ce5d0 2023-10-19T14:22:02.398 INFO:teuthology.orchestra.run.smithi057.stderr:new fs with metadata pool 35 and data pool 36 2023-10-19T14:22:02.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.399+0000 7f49501c7640 1 -- 172.21.15.57:0/297370063 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f492c06b080 msgr2=0x7f492c06d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:02.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.399+0000 7f49501c7640 1 --2- 172.21.15.57:0/297370063 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f492c06b080 0x7f492c06d540 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4944005fd0 tx=0x7f4944005e20 comp rx=0 tx=0).stop 2023-10-19T14:22:02.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.399+0000 7f49501c7640 1 -- 172.21.15.57:0/297370063 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49480ce5d0 msgr2=0x7f494806c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:02.403 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.399+0000 7f49501c7640 1 --2- 172.21.15.57:0/297370063 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49480ce5d0 0x7f494806c8c0 secure :-1 s=READY pgs=2867 cs=0 l=1 rev1=1 crypto rx=0x7f493800c440 tx=0x7f4938001e70 comp rx=0 tx=0).stop 2023-10-19T14:22:02.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.400+0000 7f49501c7640 1 -- 172.21.15.57:0/297370063 shutdown_connections 2023-10-19T14:22:02.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.400+0000 7f49501c7640 1 --2- 172.21.15.57:0/297370063 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f492c06b080 0x7f492c06d540 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:02.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.400+0000 7f49501c7640 1 --2- 172.21.15.57:0/297370063 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4948158d70 0x7f49480acb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:02.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.400+0000 7f49501c7640 1 --2- 172.21.15.57:0/297370063 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49480ce5d0 0x7f494806c8c0 unknown :-1 s=CLOSED pgs=2867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:02.405 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.400+0000 7f49501c7640 1 --2- 172.21.15.57:0/297370063 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4948069e30 0x7f49481f4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:02.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.401+0000 7f49501c7640 1 -- 172.21.15.57:0/297370063 >> 172.21.15.57:0/297370063 conn(0x7f49480af7a0 msgr2=0x7f49480aeb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:02.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.401+0000 7f49501c7640 1 -- 172.21.15.57:0/297370063 shutdown_connections 2023-10-19T14:22:02.406 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.401+0000 7f49501c7640 1 -- 172.21.15.57:0/297370063 wait complete. 2023-10-19T14:22:02.436 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T14:22:02.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.550+0000 7fab5ea10640 1 -- 172.21.15.57:0/1812136327 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab5815b2a0 msgr2=0x7fab580cf720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:02.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.550+0000 7fab5ea10640 1 --2- 172.21.15.57:0/1812136327 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab5815b2a0 0x7fab580cf720 secure :-1 s=READY pgs=2868 cs=0 l=1 rev1=1 crypto rx=0x7fab5808a120 tx=0x7fab4802bcf0 comp rx=0 tx=0).stop 2023-10-19T14:22:02.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.550+0000 7fab5ea10640 1 -- 172.21.15.57:0/1812136327 shutdown_connections 2023-10-19T14:22:02.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.550+0000 7fab5ea10640 1 --2- 172.21.15.57:0/1812136327 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab5815b2a0 0x7fab580cf720 unknown :-1 s=CLOSED pgs=2868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:02.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.550+0000 7fab5ea10640 1 --2- 172.21.15.57:0/1812136327 >> v2:172.21.15.179:3300/0 conn(0x7fab580cf340 0x7fab580d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:02.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.550+0000 7fab5ea10640 1 -- 172.21.15.57:0/1812136327 >> 172.21.15.57:0/1812136327 conn(0x7fab580d0d60 msgr2=0x7fab580d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:02.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.550+0000 7fab5ea10640 1 -- 172.21.15.57:0/1812136327 shutdown_connections 2023-10-19T14:22:02.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.550+0000 7fab5ea10640 1 -- 172.21.15.57:0/1812136327 wait complete. 2023-10-19T14:22:02.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.551+0000 7fab5ea10640 1 Processor -- start 2023-10-19T14:22:02.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.552+0000 7fab5ea10640 1 -- start start 2023-10-19T14:22:02.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.552+0000 7fab5ea10640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fab580ceb40 0x7fab58155a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:02.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.552+0000 7fab5ea10640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fab580cf340 0x7fab5806b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:02.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.552+0000 7fab5ea10640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab5815b2a0 0x7fab5806b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:02.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.552+0000 7fab5ea10640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab581f5740 con 0x7fab5815b2a0 2023-10-19T14:22:02.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.552+0000 7fab5ea10640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab58155f80 con 0x7fab580ceb40 2023-10-19T14:22:02.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.552+0000 7fab5ea10640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab581f4b00 con 0x7fab580cf340 2023-10-19T14:22:02.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.552+0000 7fab5cfad640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab5815b2a0 0x7fab5806b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:02.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.552+0000 7fab5cfad640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab5815b2a0 0x7fab5806b720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51330/0 (socket says 172.21.15.57:51330) 2023-10-19T14:22:02.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.552+0000 7fab5cfad640 1 -- 172.21.15.57:0/1911430651 learned_addr learned my addr 172.21.15.57:0/1911430651 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:22:02.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.553+0000 7fab57fff640 1 --2- 172.21.15.57:0/1911430651 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fab580ceb40 0x7fab58155a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:02.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.553+0000 7fab577fe640 1 --2- 172.21.15.57:0/1911430651 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fab580cf340 0x7fab5806b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:02.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.553+0000 7fab5cfad640 1 -- 172.21.15.57:0/1911430651 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fab580cf340 msgr2=0x7fab5806b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:02.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.553+0000 7fab5cfad640 1 --2- 172.21.15.57:0/1911430651 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fab580cf340 0x7fab5806b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:02.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.553+0000 7fab5cfad640 1 -- 172.21.15.57:0/1911430651 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fab580ceb40 msgr2=0x7fab58155a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:02.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.553+0000 7fab5cfad640 1 --2- 172.21.15.57:0/1911430651 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fab580ceb40 0x7fab58155a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:02.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.553+0000 7fab5cfad640 1 -- 172.21.15.57:0/1911430651 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab48009560 con 0x7fab5815b2a0 2023-10-19T14:22:02.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.553+0000 7fab5cfad640 1 --2- 172.21.15.57:0/1911430651 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab5815b2a0 0x7fab5806b720 secure :-1 s=READY pgs=2869 cs=0 l=1 rev1=1 crypto rx=0x7fab5000c780 tx=0x7fab5000cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:02.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.553+0000 7fab54ff9640 1 -- 172.21.15.57:0/1911430651 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab50004b60 con 0x7fab5815b2a0 2023-10-19T14:22:02.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.553+0000 7fab54ff9640 1 -- 172.21.15.57:0/1911430651 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fab50004dd0 con 0x7fab5815b2a0 2023-10-19T14:22:02.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.553+0000 7fab54ff9640 1 -- 172.21.15.57:0/1911430651 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab50005290 con 0x7fab5815b2a0 2023-10-19T14:22:02.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.553+0000 7fab5ea10640 1 -- 172.21.15.57:0/1911430651 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab581f4da0 con 0x7fab5815b2a0 2023-10-19T14:22:02.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.553+0000 7fab5ea10640 1 -- 172.21.15.57:0/1911430651 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab581f5360 con 0x7fab5815b2a0 2023-10-19T14:22:02.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.554+0000 7fab5ea10640 1 -- 172.21.15.57:0/1911430651 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab580690e0 con 0x7fab5815b2a0 2023-10-19T14:22:02.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.555+0000 7fab54ff9640 1 -- 172.21.15.57:0/1911430651 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 145) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fab500053f0 con 0x7fab5815b2a0 2023-10-19T14:22:02.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.555+0000 7fab54ff9640 1 --2- 172.21.15.57:0/1911430651 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fab4406b080 0x7fab4406d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:02.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.555+0000 7fab54ff9640 1 -- 172.21.15.57:0/1911430651 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(262..262 src has 1..262) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fab50089ed0 con 0x7fab5815b2a0 2023-10-19T14:22:02.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.555+0000 7fab57fff640 1 --2- 172.21.15.57:0/1911430651 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fab4406b080 0x7fab4406d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:02.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.556+0000 7fab57fff640 1 --2- 172.21.15.57:0/1911430651 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fab4406b080 0x7fab4406d540 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fab4c00abe0 tx=0x7fab4c00a2d0 comp rx=0 tx=0).ready entity=mgr.11446 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:02.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.560+0000 7fab54ff9640 1 -- 172.21.15.57:0/1911430651 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fab50054ab0 con 0x7fab5815b2a0 2023-10-19T14:22:02.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.738+0000 7fab5ea10640 1 -- 172.21.15.57:0/1911430651 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fab580cb9d0 con 0x7fab5815b2a0 2023-10-19T14:22:02.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.739+0000 7fab54ff9640 1 -- 172.21.15.57:0/1911430651 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v262) v1 ==== 74+0+16788 (secure 0 0 0) 0x7fab500580d0 con 0x7fab5815b2a0 2023-10-19T14:22:02.743 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:22:02.743 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":262,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T14:22:01.762807+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":36,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_metadata","create_time":"2023-10-19T14:22:00.047912+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"262","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_data","create_time":"2023-10-19T14:22:00.739698+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"262","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T14:22:02.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.741+0000 7fab5ea10640 1 -- 172.21.15.57:0/1911430651 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fab4406b080 msgr2=0x7fab4406d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:02.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.741+0000 7fab5ea10640 1 --2- 172.21.15.57:0/1911430651 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fab4406b080 0x7fab4406d540 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fab4c00abe0 tx=0x7fab4c00a2d0 comp rx=0 tx=0).stop 2023-10-19T14:22:02.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.741+0000 7fab5ea10640 1 -- 172.21.15.57:0/1911430651 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab5815b2a0 msgr2=0x7fab5806b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:02.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.741+0000 7fab5ea10640 1 --2- 172.21.15.57:0/1911430651 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab5815b2a0 0x7fab5806b720 secure :-1 s=READY pgs=2869 cs=0 l=1 rev1=1 crypto rx=0x7fab5000c780 tx=0x7fab5000cb80 comp rx=0 tx=0).stop 2023-10-19T14:22:02.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.741+0000 7fab5ea10640 1 -- 172.21.15.57:0/1911430651 shutdown_connections 2023-10-19T14:22:02.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.741+0000 7fab5ea10640 1 --2- 172.21.15.57:0/1911430651 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fab4406b080 0x7fab4406d540 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:02.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.741+0000 7fab5ea10640 1 --2- 172.21.15.57:0/1911430651 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab5815b2a0 0x7fab5806b720 unknown :-1 s=CLOSED pgs=2869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:02.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.741+0000 7fab5ea10640 1 --2- 172.21.15.57:0/1911430651 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fab580cf340 0x7fab5806b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:02.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.741+0000 7fab5ea10640 1 --2- 172.21.15.57:0/1911430651 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fab580ceb40 0x7fab58155a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:02.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.741+0000 7fab5ea10640 1 -- 172.21.15.57:0/1911430651 >> 172.21.15.57:0/1911430651 conn(0x7fab580d0d60 msgr2=0x7fab5815a4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:02.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.741+0000 7fab5ea10640 1 -- 172.21.15.57:0/1911430651 shutdown_connections 2023-10-19T14:22:02.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.742+0000 7fab5ea10640 1 -- 172.21.15.57:0/1911430651 wait complete. 2023-10-19T14:22:02.755 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T14:22:02.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.862+0000 7fc3e7e26640 1 -- 172.21.15.57:0/3900104731 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc3cc004d00 msgr2=0x7fc3cc005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:02.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.862+0000 7fc3e7e26640 1 --2- 172.21.15.57:0/3900104731 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc3cc004d00 0x7fc3cc005180 secure :-1 s=READY pgs=2871 cs=0 l=1 rev1=1 crypto rx=0x7fc3d8009ed0 tx=0x7fc3cc0057f0 comp rx=0 tx=0).stop 2023-10-19T14:22:02.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.862+0000 7fc3e63c3640 1 -- 172.21.15.57:0/3900104731 reap_dead start 2023-10-19T14:22:02.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.862+0000 7fc3e7e26640 1 -- 172.21.15.57:0/3900104731 shutdown_connections 2023-10-19T14:22:02.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.862+0000 7fc3e7e26640 1 -- 172.21.15.57:0/3900104731 >> 172.21.15.57:0/3900104731 conn(0x7fc3e0069930 msgr2=0x7fc3e0069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:02.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.862+0000 7fc3e7e26640 1 -- 172.21.15.57:0/3900104731 shutdown_connections 2023-10-19T14:22:02.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.862+0000 7fc3e7e26640 1 -- 172.21.15.57:0/3900104731 wait complete. 2023-10-19T14:22:02.869 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.864+0000 7fc3e7e26640 1 Processor -- start 2023-10-19T14:22:02.869 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.864+0000 7fc3e7e26640 1 -- start start 2023-10-19T14:22:02.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.864+0000 7fc3e7e26640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc3cc004d00 0x7fc3e013e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:02.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.864+0000 7fc3e7e26640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc3e0153aa0 0x7fc3e013df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:02.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.864+0000 7fc3e7e26640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc3e013fef0 0x7fc3e006bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:02.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.864+0000 7fc3e7e26640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e00a6970 con 0x7fc3e013fef0 2023-10-19T14:22:02.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.864+0000 7fc3e7e26640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e013e480 con 0x7fc3cc004d00 2023-10-19T14:22:02.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.864+0000 7fc3e7e26640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e013ee50 con 0x7fc3e0153aa0 2023-10-19T14:22:02.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.864+0000 7fc3e63c3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc3e013fef0 0x7fc3e006bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:02.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.864+0000 7fc3e63c3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc3e013fef0 0x7fc3e006bc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51352/0 (socket says 172.21.15.57:51352) 2023-10-19T14:22:02.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.864+0000 7fc3e63c3640 1 -- 172.21.15.57:0/3805219133 learned_addr learned my addr 172.21.15.57:0/3805219133 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:22:02.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.865+0000 7fc3e5bc2640 1 --2- 172.21.15.57:0/3805219133 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc3cc004d00 0x7fc3e013e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:02.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.865+0000 7fc3e53c1640 1 --2- 172.21.15.57:0/3805219133 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc3e0153aa0 0x7fc3e013df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:02.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.865+0000 7fc3e63c3640 1 -- 172.21.15.57:0/3805219133 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc3e0153aa0 msgr2=0x7fc3e013df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:02.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.865+0000 7fc3e63c3640 1 --2- 172.21.15.57:0/3805219133 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc3e0153aa0 0x7fc3e013df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:02.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.865+0000 7fc3e63c3640 1 -- 172.21.15.57:0/3805219133 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc3cc004d00 msgr2=0x7fc3e013e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:02.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.865+0000 7fc3e63c3640 1 --2- 172.21.15.57:0/3805219133 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc3cc004d00 0x7fc3e013e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:02.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.865+0000 7fc3e63c3640 1 -- 172.21.15.57:0/3805219133 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3e013f130 con 0x7fc3e013fef0 2023-10-19T14:22:02.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.865+0000 7fc3e5bc2640 1 --2- 172.21.15.57:0/3805219133 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc3cc004d00 0x7fc3e013e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:22:02.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.865+0000 7fc3e63c3640 1 --2- 172.21.15.57:0/3805219133 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc3e013fef0 0x7fc3e006bc50 secure :-1 s=READY pgs=2872 cs=0 l=1 rev1=1 crypto rx=0x7fc3d802c680 tx=0x7fc3d8031770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:02.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.865+0000 7fc3d67fc640 1 -- 172.21.15.57:0/3805219133 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3d803f070 con 0x7fc3e013fef0 2023-10-19T14:22:02.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.865+0000 7fc3d67fc640 1 -- 172.21.15.57:0/3805219133 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fc3d8003970 con 0x7fc3e013fef0 2023-10-19T14:22:02.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.865+0000 7fc3d67fc640 1 -- 172.21.15.57:0/3805219133 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3d8003d40 con 0x7fc3e013fef0 2023-10-19T14:22:02.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.865+0000 7fc3e7e26640 1 -- 172.21.15.57:0/3805219133 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3e013f410 con 0x7fc3e013fef0 2023-10-19T14:22:02.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.865+0000 7fc3e7e26640 1 -- 172.21.15.57:0/3805219133 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3e013f9d0 con 0x7fc3e013fef0 2023-10-19T14:22:02.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.866+0000 7fc3e7e26640 1 -- 172.21.15.57:0/3805219133 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3e013fbc0 con 0x7fc3e013fef0 2023-10-19T14:22:02.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.867+0000 7fc3d67fc640 1 -- 172.21.15.57:0/3805219133 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 145) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fc3d803a660 con 0x7fc3e013fef0 2023-10-19T14:22:02.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.867+0000 7fc3d67fc640 1 --2- 172.21.15.57:0/3805219133 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fc3c406b080 0x7fc3c406d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:02.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.867+0000 7fc3d67fc640 1 -- 172.21.15.57:0/3805219133 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(262..262 src has 1..262) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fc3d80ac6c0 con 0x7fc3e013fef0 2023-10-19T14:22:02.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.867+0000 7fc3e5bc2640 1 --2- 172.21.15.57:0/3805219133 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fc3c406b080 0x7fc3c406d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:02.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.868+0000 7fc3e5bc2640 1 --2- 172.21.15.57:0/3805219133 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fc3c406b080 0x7fc3c406d540 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc009c50 tx=0x7fc3dc005f90 comp rx=0 tx=0).ready entity=mgr.11446 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:02.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:02.871+0000 7fc3d67fc640 1 -- 172.21.15.57:0/3805219133 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc3d80771a0 con 0x7fc3e013fef0 2023-10-19T14:22:03.050 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.047+0000 7fc3e7e26640 1 -- 172.21.15.57:0/3805219133 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc3e00690e0 con 0x7fc3e013fef0 2023-10-19T14:22:03.051 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.048+0000 7fc3d67fc640 1 -- 172.21.15.57:0/3805219133 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v262) v1 ==== 74+0+16788 (secure 0 0 0) 0x7fc3d807a7c0 con 0x7fc3e013fef0 2023-10-19T14:22:03.052 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:22:03.052 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":262,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T14:22:01.762807+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":36,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_metadata","create_time":"2023-10-19T14:22:00.047912+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"262","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_data","create_time":"2023-10-19T14:22:00.739698+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"262","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T14:22:03.053 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.050+0000 7fc3e7e26640 1 -- 172.21.15.57:0/3805219133 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fc3c406b080 msgr2=0x7fc3c406d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:03.054 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.050+0000 7fc3e7e26640 1 --2- 172.21.15.57:0/3805219133 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fc3c406b080 0x7fc3c406d540 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc009c50 tx=0x7fc3dc005f90 comp rx=0 tx=0).stop 2023-10-19T14:22:03.054 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.050+0000 7fc3e7e26640 1 -- 172.21.15.57:0/3805219133 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc3e013fef0 msgr2=0x7fc3e006bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:03.054 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.050+0000 7fc3e7e26640 1 --2- 172.21.15.57:0/3805219133 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc3e013fef0 0x7fc3e006bc50 secure :-1 s=READY pgs=2872 cs=0 l=1 rev1=1 crypto rx=0x7fc3d802c680 tx=0x7fc3d8031770 comp rx=0 tx=0).stop 2023-10-19T14:22:03.054 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.050+0000 7fc3e7e26640 1 -- 172.21.15.57:0/3805219133 shutdown_connections 2023-10-19T14:22:03.055 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.050+0000 7fc3e7e26640 1 --2- 172.21.15.57:0/3805219133 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fc3c406b080 0x7fc3c406d540 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:03.055 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.050+0000 7fc3e7e26640 1 --2- 172.21.15.57:0/3805219133 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc3e013fef0 0x7fc3e006bc50 unknown :-1 s=CLOSED pgs=2872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:03.055 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.050+0000 7fc3e7e26640 1 --2- 172.21.15.57:0/3805219133 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc3e0153aa0 0x7fc3e013df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:03.055 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.050+0000 7fc3e7e26640 1 --2- 172.21.15.57:0/3805219133 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc3cc004d00 0x7fc3e013e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:03.055 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.050+0000 7fc3e7e26640 1 -- 172.21.15.57:0/3805219133 >> 172.21.15.57:0/3805219133 conn(0x7fc3e0069930 msgr2=0x7fc3e006b200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:03.056 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.051+0000 7fc3e7e26640 1 -- 172.21.15.57:0/3805219133 shutdown_connections 2023-10-19T14:22:03.056 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.051+0000 7fc3e7e26640 1 -- 172.21.15.57:0/3805219133 wait complete. 2023-10-19T14:22:03.064 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-19T14:22:03.176 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.173+0000 7f52d6a1a640 1 -- 172.21.15.57:0/2353836917 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f52b8003970 msgr2=0x7f52b8003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:03.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.173+0000 7f52d6a1a640 1 --2- 172.21.15.57:0/2353836917 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f52b8003970 0x7f52b8003d90 secure :-1 s=READY pgs=2874 cs=0 l=1 rev1=1 crypto rx=0x7f52c00096e0 tx=0x7f52b80057f0 comp rx=0 tx=0).stop 2023-10-19T14:22:03.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.173+0000 7f52d4fb7640 1 -- 172.21.15.57:0/2353836917 reap_dead start 2023-10-19T14:22:03.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.173+0000 7f52d6a1a640 1 -- 172.21.15.57:0/2353836917 shutdown_connections 2023-10-19T14:22:03.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.173+0000 7f52d6a1a640 1 -- 172.21.15.57:0/2353836917 >> 172.21.15.57:0/2353836917 conn(0x7f52d00af7a0 msgr2=0x7f52d00afb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:03.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.173+0000 7f52d6a1a640 1 -- 172.21.15.57:0/2353836917 shutdown_connections 2023-10-19T14:22:03.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.173+0000 7f52d6a1a640 1 -- 172.21.15.57:0/2353836917 wait complete. 2023-10-19T14:22:03.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.174+0000 7f52d6a1a640 1 Processor -- start 2023-10-19T14:22:03.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.175+0000 7f52d6a1a640 1 -- start start 2023-10-19T14:22:03.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.175+0000 7f52d6a1a640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f52b8003970 0x7f52d014da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:03.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.175+0000 7f52d6a1a640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f52d0070380 0x7f52d0149f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:03.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.175+0000 7f52d6a1a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f52d0070800 0x7f52d014a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:03.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.175+0000 7f52d6a1a640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52d01f0a30 con 0x7f52d0070800 2023-10-19T14:22:03.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.175+0000 7f52d6a1a640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52d01f0d10 con 0x7f52b8003970 2023-10-19T14:22:03.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.175+0000 7f52d6a1a640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52d01f0ff0 con 0x7f52d0070380 2023-10-19T14:22:03.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.175+0000 7f52d4fb7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f52d0070800 0x7f52d014a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:03.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.175+0000 7f52d4fb7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f52d0070800 0x7f52d014a450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51378/0 (socket says 172.21.15.57:51378) 2023-10-19T14:22:03.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.175+0000 7f52d4fb7640 1 -- 172.21.15.57:0/1664539129 learned_addr learned my addr 172.21.15.57:0/1664539129 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:22:03.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.176+0000 7f52cffff640 1 --2- 172.21.15.57:0/1664539129 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f52b8003970 0x7f52d014da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:03.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.176+0000 7f52cf7fe640 1 --2- 172.21.15.57:0/1664539129 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f52d0070380 0x7f52d0149f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:03.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.176+0000 7f52d4fb7640 1 -- 172.21.15.57:0/1664539129 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f52d0070380 msgr2=0x7f52d0149f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:03.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.176+0000 7f52d4fb7640 1 --2- 172.21.15.57:0/1664539129 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f52d0070380 0x7f52d0149f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:03.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.176+0000 7f52d4fb7640 1 -- 172.21.15.57:0/1664539129 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f52b8003970 msgr2=0x7f52d014da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:03.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.176+0000 7f52d4fb7640 1 --2- 172.21.15.57:0/1664539129 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f52b8003970 0x7f52d014da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:03.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.176+0000 7f52d4fb7640 1 -- 172.21.15.57:0/1664539129 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52d01f12d0 con 0x7f52d0070800 2023-10-19T14:22:03.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.176+0000 7f52d4fb7640 1 --2- 172.21.15.57:0/1664539129 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f52d0070800 0x7f52d014a450 secure :-1 s=READY pgs=2875 cs=0 l=1 rev1=1 crypto rx=0x7f52d0066990 tx=0x7f52c8010f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:03.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.176+0000 7f52ccff9640 1 -- 172.21.15.57:0/1664539129 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52c8003970 con 0x7f52d0070800 2023-10-19T14:22:03.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.176+0000 7f52ccff9640 1 -- 172.21.15.57:0/1664539129 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f52c8003bc0 con 0x7f52d0070800 2023-10-19T14:22:03.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.176+0000 7f52d6a1a640 1 -- 172.21.15.57:0/1664539129 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52d01f15b0 con 0x7f52d0070800 2023-10-19T14:22:03.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.176+0000 7f52ccff9640 1 -- 172.21.15.57:0/1664539129 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52c8004030 con 0x7f52d0070800 2023-10-19T14:22:03.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.176+0000 7f52d6a1a640 1 -- 172.21.15.57:0/1664539129 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52d014df50 con 0x7f52d0070800 2023-10-19T14:22:03.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.177+0000 7f52d6a1a640 1 -- 172.21.15.57:0/1664539129 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52d01f17a0 con 0x7f52d0070800 2023-10-19T14:22:03.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.178+0000 7f52ccff9640 1 -- 172.21.15.57:0/1664539129 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 145) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f52c8019030 con 0x7f52d0070800 2023-10-19T14:22:03.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.178+0000 7f52ccff9640 1 --2- 172.21.15.57:0/1664539129 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f52bc06b080 0x7f52bc06d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:03.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.178+0000 7f52ccff9640 1 -- 172.21.15.57:0/1664539129 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(262..262 src has 1..262) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f52c8017070 con 0x7f52d0070800 2023-10-19T14:22:03.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.178+0000 7f52cffff640 1 --2- 172.21.15.57:0/1664539129 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f52bc06b080 0x7f52bc06d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:03.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.179+0000 7f52cffff640 1 --2- 172.21.15.57:0/1664539129 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f52bc06b080 0x7f52bc06d540 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f52c0039000 tx=0x7f52c0040040 comp rx=0 tx=0).ready entity=mgr.11446 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:03.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.182+0000 7f52ccff9640 1 -- 172.21.15.57:0/1664539129 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f52c8055650 con 0x7f52d0070800 2023-10-19T14:22:03.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.392+0000 7f52d6a1a640 1 -- 172.21.15.57:0/1664539129 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f52d00690e0 con 0x7f52d0070800 2023-10-19T14:22:03.408 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.405+0000 7f52ccff9640 1 -- 172.21.15.57:0/1664539129 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v115) v1 ==== 118+0+0 (secure 0 0 0) 0x7f52c80210b0 con 0x7f52d0070800 2023-10-19T14:22:03.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.407+0000 7f52d6a1a640 1 -- 172.21.15.57:0/1664539129 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f52bc06b080 msgr2=0x7f52bc06d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:03.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.407+0000 7f52d6a1a640 1 --2- 172.21.15.57:0/1664539129 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f52bc06b080 0x7f52bc06d540 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f52c0039000 tx=0x7f52c0040040 comp rx=0 tx=0).stop 2023-10-19T14:22:03.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.407+0000 7f52d6a1a640 1 -- 172.21.15.57:0/1664539129 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f52d0070800 msgr2=0x7f52d014a450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:03.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.407+0000 7f52d6a1a640 1 --2- 172.21.15.57:0/1664539129 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f52d0070800 0x7f52d014a450 secure :-1 s=READY pgs=2875 cs=0 l=1 rev1=1 crypto rx=0x7f52d0066990 tx=0x7f52c8010f40 comp rx=0 tx=0).stop 2023-10-19T14:22:03.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.408+0000 7f52d6a1a640 1 -- 172.21.15.57:0/1664539129 shutdown_connections 2023-10-19T14:22:03.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.408+0000 7f52d6a1a640 1 --2- 172.21.15.57:0/1664539129 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f52bc06b080 0x7f52bc06d540 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:03.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.408+0000 7f52d6a1a640 1 --2- 172.21.15.57:0/1664539129 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f52d0070800 0x7f52d014a450 unknown :-1 s=CLOSED pgs=2875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:03.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.408+0000 7f52d6a1a640 1 --2- 172.21.15.57:0/1664539129 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f52d0070380 0x7f52d0149f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:03.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.408+0000 7f52d6a1a640 1 --2- 172.21.15.57:0/1664539129 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f52b8003970 0x7f52d014da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:03.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.408+0000 7f52d6a1a640 1 -- 172.21.15.57:0/1664539129 >> 172.21.15.57:0/1664539129 conn(0x7f52d00af7a0 msgr2=0x7f52d015a1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:03.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.408+0000 7f52d6a1a640 1 -- 172.21.15.57:0/1664539129 shutdown_connections 2023-10-19T14:22:03.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.408+0000 7f52d6a1a640 1 -- 172.21.15.57:0/1664539129 wait complete. 2023-10-19T14:22:03.422 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-19T14:22:03.423 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-19T14:22:03.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.532+0000 7fc3042f2640 1 -- 172.21.15.57:0/989999338 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc2fc141690 msgr2=0x7fc2fc0703e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:03.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.532+0000 7fc3042f2640 1 --2- 172.21.15.57:0/989999338 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc2fc141690 0x7fc2fc0703e0 secure :-1 s=READY pgs=2876 cs=0 l=1 rev1=1 crypto rx=0x7fc2fc0cbd00 tx=0x7fc2f002bd10 comp rx=0 tx=0).stop 2023-10-19T14:22:03.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.532+0000 7fc3042f2640 1 -- 172.21.15.57:0/989999338 shutdown_connections 2023-10-19T14:22:03.536 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.532+0000 7fc3042f2640 1 --2- 172.21.15.57:0/989999338 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc2fc06c0a0 0x7fc2fc070920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:03.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.532+0000 7fc3042f2640 1 --2- 172.21.15.57:0/989999338 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc2fc141690 0x7fc2fc0703e0 unknown :-1 s=CLOSED pgs=2876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:03.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.532+0000 7fc3042f2640 1 --2- 172.21.15.57:0/989999338 >> v2:172.21.15.179:3300/0 conn(0x7fc2fc13e520 0x7fc2fc06bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:03.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.532+0000 7fc3042f2640 1 -- 172.21.15.57:0/989999338 >> 172.21.15.57:0/989999338 conn(0x7fc2fc069930 msgr2=0x7fc2fc069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:03.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.532+0000 7fc3042f2640 1 -- 172.21.15.57:0/989999338 shutdown_connections 2023-10-19T14:22:03.537 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.532+0000 7fc3042f2640 1 -- 172.21.15.57:0/989999338 wait complete. 2023-10-19T14:22:03.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.533+0000 7fc3042f2640 1 Processor -- start 2023-10-19T14:22:03.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.534+0000 7fc3042f2640 1 -- start start 2023-10-19T14:22:03.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.534+0000 7fc3042f2640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc2fc06c0a0 0x7fc2fc1e30f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:03.540 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.534+0000 7fc3042f2640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc2fc13e520 0x7fc2fc0b0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:03.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.534+0000 7fc3042f2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc2fc141690 0x7fc2fc1e1120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:03.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.534+0000 7fc3042f2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2fc1e3630 con 0x7fc2fc141690 2023-10-19T14:22:03.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.534+0000 7fc3042f2640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2fc0a9710 con 0x7fc2fc13e520 2023-10-19T14:22:03.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.534+0000 7fc3042f2640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2fc0a9960 con 0x7fc2fc06c0a0 2023-10-19T14:22:03.541 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.534+0000 7fc30288f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc2fc141690 0x7fc2fc1e1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:03.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.534+0000 7fc30288f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc2fc141690 0x7fc2fc1e1120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51386/0 (socket says 172.21.15.57:51386) 2023-10-19T14:22:03.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.534+0000 7fc30288f640 1 -- 172.21.15.57:0/3211655502 learned_addr learned my addr 172.21.15.57:0/3211655502 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:22:03.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.534+0000 7fc30188d640 1 --2- 172.21.15.57:0/3211655502 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc2fc13e520 0x7fc2fc0b0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:03.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.535+0000 7fc30208e640 1 --2- 172.21.15.57:0/3211655502 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc2fc06c0a0 0x7fc2fc1e30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:03.542 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.535+0000 7fc30288f640 1 -- 172.21.15.57:0/3211655502 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc2fc06c0a0 msgr2=0x7fc2fc1e30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:03.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.535+0000 7fc30288f640 1 --2- 172.21.15.57:0/3211655502 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc2fc06c0a0 0x7fc2fc1e30f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:03.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.535+0000 7fc30288f640 1 -- 172.21.15.57:0/3211655502 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc2fc13e520 msgr2=0x7fc2fc0b0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:03.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.535+0000 7fc30288f640 1 --2- 172.21.15.57:0/3211655502 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc2fc13e520 0x7fc2fc0b0dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:03.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.535+0000 7fc30288f640 1 -- 172.21.15.57:0/3211655502 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2f0009560 con 0x7fc2fc141690 2023-10-19T14:22:03.543 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.535+0000 7fc30288f640 1 --2- 172.21.15.57:0/3211655502 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc2fc141690 0x7fc2fc1e1120 secure :-1 s=READY pgs=2877 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec00e980 tx=0x7fc2ec00ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:03.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.535+0000 7fc2faffd640 1 -- 172.21.15.57:0/3211655502 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2ec004d20 con 0x7fc2fc141690 2023-10-19T14:22:03.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.535+0000 7fc2faffd640 1 -- 172.21.15.57:0/3211655502 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fc2ec00cb20 con 0x7fc2fc141690 2023-10-19T14:22:03.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.535+0000 7fc2faffd640 1 -- 172.21.15.57:0/3211655502 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2ec005020 con 0x7fc2fc141690 2023-10-19T14:22:03.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.535+0000 7fc3042f2640 1 -- 172.21.15.57:0/3211655502 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2fc0a9c40 con 0x7fc2fc141690 2023-10-19T14:22:03.544 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.535+0000 7fc3042f2640 1 -- 172.21.15.57:0/3211655502 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2fc0aa200 con 0x7fc2fc141690 2023-10-19T14:22:03.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.536+0000 7fc3042f2640 1 -- 172.21.15.57:0/3211655502 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2fc0c9170 con 0x7fc2fc141690 2023-10-19T14:22:03.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.537+0000 7fc2faffd640 1 -- 172.21.15.57:0/3211655502 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 145) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fc2ec005180 con 0x7fc2fc141690 2023-10-19T14:22:03.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.537+0000 7fc2faffd640 1 --2- 172.21.15.57:0/3211655502 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fc2e806b080 0x7fc2e806d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:03.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.537+0000 7fc2faffd640 1 -- 172.21.15.57:0/3211655502 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(263..263 src has 1..263) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fc2ec08a4f0 con 0x7fc2fc141690 2023-10-19T14:22:03.545 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.537+0000 7fc30208e640 1 --2- 172.21.15.57:0/3211655502 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fc2e806b080 0x7fc2e806d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:03.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.538+0000 7fc30208e640 1 --2- 172.21.15.57:0/3211655502 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fc2e806b080 0x7fc2e806d540 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc2fc0c8b80 tx=0x7fc2f400b390 comp rx=0 tx=0).ready entity=mgr.11446 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:03.546 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.541+0000 7fc2faffd640 1 -- 172.21.15.57:0/3211655502 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc2fc0c9170 con 0x7fc2fc141690 2023-10-19T14:22:03.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:03.756+0000 7fc3042f2640 1 -- 172.21.15.57:0/3211655502 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fc2fc0c9170 con 0x7fc2fc141690 2023-10-19T14:22:04.376 INFO:tasks.ceph.mds.d.smithi179.stderr:2023-10-19T14:22:04.373+0000 7f67eaf27640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-19T14:22:04.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.414+0000 7fc2faffd640 1 -- 172.21.15.57:0/3211655502 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v116) v1 ==== 122+0+0 (secure 0 0 0) 0x7fc2fc0c9170 con 0x7fc2fc141690 2023-10-19T14:22:04.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.417+0000 7fc3042f2640 1 -- 172.21.15.57:0/3211655502 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fc2e806b080 msgr2=0x7fc2e806d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:04.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.417+0000 7fc3042f2640 1 --2- 172.21.15.57:0/3211655502 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fc2e806b080 0x7fc2e806d540 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc2fc0c8b80 tx=0x7fc2f400b390 comp rx=0 tx=0).stop 2023-10-19T14:22:04.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.417+0000 7fc3042f2640 1 -- 172.21.15.57:0/3211655502 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc2fc141690 msgr2=0x7fc2fc1e1120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:04.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.417+0000 7fc3042f2640 1 --2- 172.21.15.57:0/3211655502 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc2fc141690 0x7fc2fc1e1120 secure :-1 s=READY pgs=2877 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec00e980 tx=0x7fc2ec00ee40 comp rx=0 tx=0).stop 2023-10-19T14:22:04.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.417+0000 7fc3042f2640 1 -- 172.21.15.57:0/3211655502 shutdown_connections 2023-10-19T14:22:04.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.417+0000 7fc3042f2640 1 --2- 172.21.15.57:0/3211655502 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fc2e806b080 0x7fc2e806d540 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:04.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.417+0000 7fc3042f2640 1 --2- 172.21.15.57:0/3211655502 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc2fc141690 0x7fc2fc1e1120 unknown :-1 s=CLOSED pgs=2877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:04.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.417+0000 7fc3042f2640 1 --2- 172.21.15.57:0/3211655502 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc2fc13e520 0x7fc2fc0b0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:04.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.417+0000 7fc3042f2640 1 --2- 172.21.15.57:0/3211655502 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc2fc06c0a0 0x7fc2fc1e30f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:04.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.417+0000 7fc3042f2640 1 -- 172.21.15.57:0/3211655502 >> 172.21.15.57:0/3211655502 conn(0x7fc2fc069930 msgr2=0x7fc2fc0affe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:04.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.417+0000 7fc3042f2640 1 -- 172.21.15.57:0/3211655502 shutdown_connections 2023-10-19T14:22:04.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.417+0000 7fc3042f2640 1 -- 172.21.15.57:0/3211655502 wait complete. 2023-10-19T14:22:04.433 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-19T14:22:04.433 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T14:22:04.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.546+0000 7fdd5e81a640 1 -- 172.21.15.57:0/2842097188 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd580b0530 msgr2=0x7fdd580b09c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:04.549 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.546+0000 7fdd5e81a640 1 --2- 172.21.15.57:0/2842097188 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd580b0530 0x7fdd580b09c0 secure :-1 s=READY pgs=2879 cs=0 l=1 rev1=1 crypto rx=0x7fdd50005fd0 tx=0x7fdd5002c0e0 comp rx=0 tx=0).stop 2023-10-19T14:22:04.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.546+0000 7fdd5e81a640 1 -- 172.21.15.57:0/2842097188 shutdown_connections 2023-10-19T14:22:04.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.546+0000 7fdd5e81a640 1 --2- 172.21.15.57:0/2842097188 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdd580afa20 0x7fdd58158300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:04.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.546+0000 7fdd5e81a640 1 --2- 172.21.15.57:0/2842097188 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd580b0530 0x7fdd580b09c0 unknown :-1 s=CLOSED pgs=2879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:04.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.546+0000 7fdd5e81a640 1 -- 172.21.15.57:0/2842097188 >> 172.21.15.57:0/2842097188 conn(0x7fdd58069930 msgr2=0x7fdd58069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:04.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.546+0000 7fdd5e81a640 1 -- 172.21.15.57:0/2842097188 shutdown_connections 2023-10-19T14:22:04.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.546+0000 7fdd5e81a640 1 -- 172.21.15.57:0/2842097188 wait complete. 2023-10-19T14:22:04.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.547+0000 7fdd5e81a640 1 Processor -- start 2023-10-19T14:22:04.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.547+0000 7fdd5e81a640 1 -- start start 2023-10-19T14:22:04.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.548+0000 7fdd5e81a640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdd580afa20 0x7fdd5813f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:04.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.548+0000 7fdd5e81a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd580b0530 0x7fdd58145e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:04.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.548+0000 7fdd5e81a640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdd580cbd20 0x7fdd58143b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:04.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.548+0000 7fdd5e81a640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd5814d990 con 0x7fdd580b0530 2023-10-19T14:22:04.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.548+0000 7fdd5e81a640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd58147780 con 0x7fdd580cbd20 2023-10-19T14:22:04.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.548+0000 7fdd5e81a640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd5813fae0 con 0x7fdd580afa20 2023-10-19T14:22:04.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.548+0000 7fdd577fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd580b0530 0x7fdd58145e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:04.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.548+0000 7fdd577fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd580b0530 0x7fdd58145e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51396/0 (socket says 172.21.15.57:51396) 2023-10-19T14:22:04.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.548+0000 7fdd577fe640 1 -- 172.21.15.57:0/238480459 learned_addr learned my addr 172.21.15.57:0/238480459 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:22:04.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.548+0000 7fdd57fff640 1 --2- 172.21.15.57:0/238480459 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdd580afa20 0x7fdd5813f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:04.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.548+0000 7fdd5cdb7640 1 --2- 172.21.15.57:0/238480459 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdd580cbd20 0x7fdd58143b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:04.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.548+0000 7fdd577fe640 1 -- 172.21.15.57:0/238480459 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdd580afa20 msgr2=0x7fdd5813f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:04.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.548+0000 7fdd577fe640 1 --2- 172.21.15.57:0/238480459 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdd580afa20 0x7fdd5813f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:04.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.548+0000 7fdd577fe640 1 -- 172.21.15.57:0/238480459 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdd580cbd20 msgr2=0x7fdd58143b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:04.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.548+0000 7fdd577fe640 1 --2- 172.21.15.57:0/238480459 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdd580cbd20 0x7fdd58143b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:04.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.548+0000 7fdd577fe640 1 -- 172.21.15.57:0/238480459 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd50009840 con 0x7fdd580b0530 2023-10-19T14:22:04.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.548+0000 7fdd5cdb7640 1 --2- 172.21.15.57:0/238480459 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdd580cbd20 0x7fdd58143b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:22:04.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.549+0000 7fdd577fe640 1 --2- 172.21.15.57:0/238480459 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd580b0530 0x7fdd58145e90 secure :-1 s=READY pgs=2880 cs=0 l=1 rev1=1 crypto rx=0x7fdd4800b4a0 tx=0x7fdd4800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:04.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.549+0000 7fdd54ff9640 1 -- 172.21.15.57:0/238480459 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd48010610 con 0x7fdd580b0530 2023-10-19T14:22:04.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.549+0000 7fdd54ff9640 1 -- 172.21.15.57:0/238480459 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fdd48010860 con 0x7fdd580b0530 2023-10-19T14:22:04.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.549+0000 7fdd5e81a640 1 -- 172.21.15.57:0/238480459 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd581466e0 con 0x7fdd580b0530 2023-10-19T14:22:04.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.549+0000 7fdd54ff9640 1 -- 172.21.15.57:0/238480459 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd48010c70 con 0x7fdd580b0530 2023-10-19T14:22:04.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.549+0000 7fdd5e81a640 1 -- 172.21.15.57:0/238480459 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd58146ca0 con 0x7fdd580b0530 2023-10-19T14:22:04.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.550+0000 7fdd5e81a640 1 -- 172.21.15.57:0/238480459 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd580690e0 con 0x7fdd580b0530 2023-10-19T14:22:04.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.550+0000 7fdd54ff9640 1 -- 172.21.15.57:0/238480459 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 145) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fdd48017360 con 0x7fdd580b0530 2023-10-19T14:22:04.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.551+0000 7fdd54ff9640 1 --2- 172.21.15.57:0/238480459 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fdd4406b080 0x7fdd4406d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:04.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.551+0000 7fdd54ff9640 1 -- 172.21.15.57:0/238480459 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(263..263 src has 1..263) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fdd48089a50 con 0x7fdd580b0530 2023-10-19T14:22:04.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.551+0000 7fdd57fff640 1 --2- 172.21.15.57:0/238480459 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fdd4406b080 0x7fdd4406d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:04.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.552+0000 7fdd57fff640 1 --2- 172.21.15.57:0/238480459 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fdd4406b080 0x7fdd4406d540 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fdd4c007fd0 tx=0x7fdd4c009040 comp rx=0 tx=0).ready entity=mgr.11446 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:04.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.555+0000 7fdd54ff9640 1 -- 172.21.15.57:0/238480459 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fdd480580d0 con 0x7fdd580b0530 2023-10-19T14:22:04.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.766+0000 7fdd5e81a640 1 -- 172.21.15.57:0/238480459 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdd5806f700 con 0x7fdd580b0530 2023-10-19T14:22:04.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.766+0000 7fdd54ff9640 1 -- 172.21.15.57:0/238480459 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+4935 (secure 0 0 0) 0x7fdd48014090 con 0x7fdd580b0530 2023-10-19T14:22:04.770 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:22:04.770 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":116,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10280,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/1738328496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":1738328496},{"type":"v1","addr":"172.21.15.179:6837","nonce":1738328496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":113},{"gid":13344,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/3191801579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":3191801579},{"type":"v1","addr":"172.21.15.57:6834","nonce":3191801579}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":113},{"gid":13347,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/3876822488","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":3876822488},{"type":"v1","addr":"172.21.15.57:6835","nonce":3876822488}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T14:22:01.763128+0000","modified":"2023-10-19T14:22:03.758375+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":13359},"failed":[],"damaged":[],"stopped":[],"info":{"gid_13359":{"gid":13359,"name":"d","rank":0,"incarnation":114,"state":"up:active","state_seq":3,"addr":"172.21.15.179:6836/4133411706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":4133411706},{"type":"v1","addr":"172.21.15.179:6836","nonce":4133411706}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[36],"metadata_pool":35,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":17}]} 2023-10-19T14:22:04.771 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 116 2023-10-19T14:22:04.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.768+0000 7fdd5e81a640 1 -- 172.21.15.57:0/238480459 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fdd4406b080 msgr2=0x7fdd4406d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:04.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.768+0000 7fdd5e81a640 1 --2- 172.21.15.57:0/238480459 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fdd4406b080 0x7fdd4406d540 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fdd4c007fd0 tx=0x7fdd4c009040 comp rx=0 tx=0).stop 2023-10-19T14:22:04.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.768+0000 7fdd5e81a640 1 -- 172.21.15.57:0/238480459 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd580b0530 msgr2=0x7fdd58145e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:04.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.768+0000 7fdd5e81a640 1 --2- 172.21.15.57:0/238480459 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd580b0530 0x7fdd58145e90 secure :-1 s=READY pgs=2880 cs=0 l=1 rev1=1 crypto rx=0x7fdd4800b4a0 tx=0x7fdd4800b960 comp rx=0 tx=0).stop 2023-10-19T14:22:04.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.768+0000 7fdd5e81a640 1 -- 172.21.15.57:0/238480459 shutdown_connections 2023-10-19T14:22:04.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.768+0000 7fdd5e81a640 1 --2- 172.21.15.57:0/238480459 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fdd4406b080 0x7fdd4406d540 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:04.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.768+0000 7fdd5e81a640 1 --2- 172.21.15.57:0/238480459 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdd580cbd20 0x7fdd58143b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:04.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.768+0000 7fdd5e81a640 1 --2- 172.21.15.57:0/238480459 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd580b0530 0x7fdd58145e90 unknown :-1 s=CLOSED pgs=2880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:04.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.768+0000 7fdd5e81a640 1 --2- 172.21.15.57:0/238480459 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdd580afa20 0x7fdd5813f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:04.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.768+0000 7fdd5e81a640 1 -- 172.21.15.57:0/238480459 >> 172.21.15.57:0/238480459 conn(0x7fdd58069930 msgr2=0x7fdd5814bc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:04.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.768+0000 7fdd5e81a640 1 -- 172.21.15.57:0/238480459 shutdown_connections 2023-10-19T14:22:04.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.768+0000 7fdd5e81a640 1 -- 172.21.15.57:0/238480459 wait complete. 2023-10-19T14:22:04.782 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T14:22:04.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.888+0000 7efdf5bb6640 1 -- 172.21.15.57:0/3693681055 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efdf01579f0 msgr2=0x7efdf00b0e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:04.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.888+0000 7efdf5bb6640 1 --2- 172.21.15.57:0/3693681055 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efdf01579f0 0x7efdf00b0e90 secure :-1 s=READY pgs=2882 cs=0 l=1 rev1=1 crypto rx=0x7efde0009830 tx=0x7efde002bd70 comp rx=0 tx=0).stop 2023-10-19T14:22:04.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.888+0000 7efdf5bb6640 1 -- 172.21.15.57:0/3693681055 shutdown_connections 2023-10-19T14:22:04.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.888+0000 7efdf5bb6640 1 --2- 172.21.15.57:0/3693681055 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7efdf006ae40 0x7efdf0157dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:04.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.888+0000 7efdf5bb6640 1 --2- 172.21.15.57:0/3693681055 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efdf01579f0 0x7efdf00b0e90 unknown :-1 s=CLOSED pgs=2882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:04.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.888+0000 7efdf5bb6640 1 -- 172.21.15.57:0/3693681055 >> 172.21.15.57:0/3693681055 conn(0x7efdf006fae0 msgr2=0x7efdf006fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:04.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.888+0000 7efdf5bb6640 1 -- 172.21.15.57:0/3693681055 shutdown_connections 2023-10-19T14:22:04.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.888+0000 7efdf5bb6640 1 -- 172.21.15.57:0/3693681055 wait complete. 2023-10-19T14:22:04.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.889+0000 7efdf5bb6640 1 Processor -- start 2023-10-19T14:22:04.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.889+0000 7efdf5bb6640 1 -- start start 2023-10-19T14:22:04.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.890+0000 7efdf5bb6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efdf006ae40 0x7efdf014a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:04.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.890+0000 7efdf5bb6640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efdf01579f0 0x7efdf014f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:04.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.890+0000 7efdf5bb6640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7efdf015c1f0 0x7efdf014f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:04.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.890+0000 7efdf5bb6640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdf014c790 con 0x7efdf006ae40 2023-10-19T14:22:04.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.890+0000 7efdf5bb6640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdf014b9d0 con 0x7efdf01579f0 2023-10-19T14:22:04.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.890+0000 7efdf5bb6640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdf014bcb0 con 0x7efdf015c1f0 2023-10-19T14:22:04.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.890+0000 7efdef7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efdf006ae40 0x7efdf014a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:04.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.890+0000 7efdef7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efdf006ae40 0x7efdf014a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51424/0 (socket says 172.21.15.57:51424) 2023-10-19T14:22:04.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.890+0000 7efdef7fe640 1 -- 172.21.15.57:0/2968142988 learned_addr learned my addr 172.21.15.57:0/2968142988 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:22:04.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.890+0000 7efdef7fe640 1 -- 172.21.15.57:0/2968142988 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7efdf015c1f0 msgr2=0x7efdf014f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:04.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.890+0000 7efdef7fe640 1 --2- 172.21.15.57:0/2968142988 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7efdf015c1f0 0x7efdf014f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:04.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.890+0000 7efdef7fe640 1 -- 172.21.15.57:0/2968142988 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efdf01579f0 msgr2=0x7efdf014f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:04.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.890+0000 7efdef7fe640 1 --2- 172.21.15.57:0/2968142988 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efdf01579f0 0x7efdf014f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:04.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.890+0000 7efdef7fe640 1 -- 172.21.15.57:0/2968142988 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efde0009490 con 0x7efdf006ae40 2023-10-19T14:22:04.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.890+0000 7efdef7fe640 1 --2- 172.21.15.57:0/2968142988 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efdf006ae40 0x7efdf014a340 secure :-1 s=READY pgs=2883 cs=0 l=1 rev1=1 crypto rx=0x7efde0000c00 tx=0x7efde0003880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:04.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.890+0000 7efdd7fff640 1 -- 172.21.15.57:0/2968142988 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde0003e80 con 0x7efdf006ae40 2023-10-19T14:22:04.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.891+0000 7efdd7fff640 1 -- 172.21.15.57:0/2968142988 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7efde0034530 con 0x7efdf006ae40 2023-10-19T14:22:04.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.891+0000 7efdf5bb6640 1 -- 172.21.15.57:0/2968142988 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdf014bf90 con 0x7efdf006ae40 2023-10-19T14:22:04.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.891+0000 7efdd7fff640 1 -- 172.21.15.57:0/2968142988 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde00348e0 con 0x7efdf006ae40 2023-10-19T14:22:04.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.891+0000 7efdf5bb6640 1 -- 172.21.15.57:0/2968142988 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdf01513f0 con 0x7efdf006ae40 2023-10-19T14:22:04.898 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.892+0000 7efdf5bb6640 1 -- 172.21.15.57:0/2968142988 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdf00690e0 con 0x7efdf006ae40 2023-10-19T14:22:04.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.892+0000 7efdd7fff640 1 -- 172.21.15.57:0/2968142988 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 145) v1 ==== 89433+0+0 (secure 0 0 0) 0x7efde0034a40 con 0x7efdf006ae40 2023-10-19T14:22:04.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.892+0000 7efdd7fff640 1 --2- 172.21.15.57:0/2968142988 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7efddc06b080 0x7efddc06d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:04.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.892+0000 7efdd7fff640 1 -- 172.21.15.57:0/2968142988 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(263..263 src has 1..263) v4 ==== 6057+0+0 (secure 0 0 0) 0x7efde00ac550 con 0x7efdf006ae40 2023-10-19T14:22:04.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.893+0000 7efdeeffd640 1 --2- 172.21.15.57:0/2968142988 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7efddc06b080 0x7efddc06d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:04.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.893+0000 7efdeeffd640 1 --2- 172.21.15.57:0/2968142988 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7efddc06b080 0x7efddc06d540 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7efde40046b0 tx=0x7efde4007a70 comp rx=0 tx=0).ready entity=mgr.11446 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:04.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:04.897+0000 7efdd7fff640 1 -- 172.21.15.57:0/2968142988 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7efde0077150 con 0x7efdf006ae40 2023-10-19T14:22:05.074 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.071+0000 7efdf5bb6640 1 -- 172.21.15.57:0/2968142988 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efdf014c450 con 0x7efdf006ae40 2023-10-19T14:22:05.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.072+0000 7efdd7fff640 1 -- 172.21.15.57:0/2968142988 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v263) v1 ==== 74+0+16775 (secure 0 0 0) 0x7efde007a770 con 0x7efdf006ae40 2023-10-19T14:22:05.076 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:22:05.076 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":263,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T14:22:03.405850+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":36,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_metadata","create_time":"2023-10-19T14:22:00.047912+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"262","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_data","create_time":"2023-10-19T14:22:00.739698+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"263","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T14:22:05.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.074+0000 7efdf5bb6640 1 -- 172.21.15.57:0/2968142988 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7efddc06b080 msgr2=0x7efddc06d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:05.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.074+0000 7efdf5bb6640 1 --2- 172.21.15.57:0/2968142988 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7efddc06b080 0x7efddc06d540 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7efde40046b0 tx=0x7efde4007a70 comp rx=0 tx=0).stop 2023-10-19T14:22:05.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.074+0000 7efdf5bb6640 1 -- 172.21.15.57:0/2968142988 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efdf006ae40 msgr2=0x7efdf014a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:05.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.074+0000 7efdf5bb6640 1 --2- 172.21.15.57:0/2968142988 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efdf006ae40 0x7efdf014a340 secure :-1 s=READY pgs=2883 cs=0 l=1 rev1=1 crypto rx=0x7efde0000c00 tx=0x7efde0003880 comp rx=0 tx=0).stop 2023-10-19T14:22:05.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.074+0000 7efdf5bb6640 1 -- 172.21.15.57:0/2968142988 shutdown_connections 2023-10-19T14:22:05.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.074+0000 7efdf5bb6640 1 --2- 172.21.15.57:0/2968142988 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7efddc06b080 0x7efddc06d540 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:05.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.074+0000 7efdf5bb6640 1 --2- 172.21.15.57:0/2968142988 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7efdf015c1f0 0x7efdf014f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:05.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.074+0000 7efdf5bb6640 1 --2- 172.21.15.57:0/2968142988 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efdf01579f0 0x7efdf014f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:05.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.074+0000 7efdf5bb6640 1 --2- 172.21.15.57:0/2968142988 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efdf006ae40 0x7efdf014a340 unknown :-1 s=CLOSED pgs=2883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:05.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.074+0000 7efdf5bb6640 1 -- 172.21.15.57:0/2968142988 >> 172.21.15.57:0/2968142988 conn(0x7efdf006fae0 msgr2=0x7efdf00cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:05.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.075+0000 7efdf5bb6640 1 -- 172.21.15.57:0/2968142988 shutdown_connections 2023-10-19T14:22:05.081 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.075+0000 7efdf5bb6640 1 -- 172.21.15.57:0/2968142988 wait complete. 2023-10-19T14:22:05.089 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-19T14:22:05.091 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-19T14:22:05.202 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.198+0000 7f99834c2640 1 -- 172.21.15.57:0/2811436564 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f997c0b0530 msgr2=0x7f997c0b09c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:05.202 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.198+0000 7f99834c2640 1 --2- 172.21.15.57:0/2811436564 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f997c0b0530 0x7f997c0b09c0 secure :-1 s=READY pgs=2884 cs=0 l=1 rev1=1 crypto rx=0x7f997c0cc7e0 tx=0x7f997402bda0 comp rx=0 tx=0).stop 2023-10-19T14:22:05.202 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.198+0000 7f99834c2640 1 -- 172.21.15.57:0/2811436564 shutdown_connections 2023-10-19T14:22:05.202 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.198+0000 7f99834c2640 1 --2- 172.21.15.57:0/2811436564 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f997c0b0530 0x7f997c0b09c0 unknown :-1 s=CLOSED pgs=2884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:05.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.198+0000 7f99834c2640 1 --2- 172.21.15.57:0/2811436564 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f997c0afa20 0x7f997c0705a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:05.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.198+0000 7f99834c2640 1 --2- 172.21.15.57:0/2811436564 >> v2:172.21.15.179:3300/0 conn(0x7f997c0cbd20 0x7f997c158300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:05.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.198+0000 7f99834c2640 1 -- 172.21.15.57:0/2811436564 >> 172.21.15.57:0/2811436564 conn(0x7f997c069930 msgr2=0x7f997c069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:05.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.198+0000 7f99834c2640 1 -- 172.21.15.57:0/2811436564 shutdown_connections 2023-10-19T14:22:05.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.198+0000 7f99834c2640 1 -- 172.21.15.57:0/2811436564 wait complete. 2023-10-19T14:22:05.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.199+0000 7f99834c2640 1 Processor -- start 2023-10-19T14:22:05.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.199+0000 7f99834c2640 1 -- start start 2023-10-19T14:22:05.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.200+0000 7f99834c2640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f997c0afa20 0x7f997c12d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:05.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.200+0000 7f99834c2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f997c0b0530 0x7f997c159d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:05.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.200+0000 7f99834c2640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f997c0cbd20 0x7f997c1413a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:05.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.200+0000 7f99834c2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f997c12d930 con 0x7f997c0b0530 2023-10-19T14:22:05.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.200+0000 7f99834c2640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f997c142360 con 0x7f997c0afa20 2023-10-19T14:22:05.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.200+0000 7f99834c2640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f997c142640 con 0x7f997c0cbd20 2023-10-19T14:22:05.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.200+0000 7f9980a5d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f997c0b0530 0x7f997c159d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:05.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.200+0000 7f9980a5d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f997c0b0530 0x7f997c159d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51438/0 (socket says 172.21.15.57:51438) 2023-10-19T14:22:05.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.200+0000 7f9980a5d640 1 -- 172.21.15.57:0/636314036 learned_addr learned my addr 172.21.15.57:0/636314036 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:22:05.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.200+0000 7f998125e640 1 --2- 172.21.15.57:0/636314036 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f997c0afa20 0x7f997c12d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:05.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.200+0000 7f9981a5f640 1 --2- 172.21.15.57:0/636314036 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f997c0cbd20 0x7f997c1413a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:05.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.200+0000 7f9980a5d640 1 -- 172.21.15.57:0/636314036 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f997c0cbd20 msgr2=0x7f997c1413a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:05.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.200+0000 7f9980a5d640 1 --2- 172.21.15.57:0/636314036 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f997c0cbd20 0x7f997c1413a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:05.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.200+0000 7f9980a5d640 1 -- 172.21.15.57:0/636314036 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f997c0afa20 msgr2=0x7f997c12d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:05.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.200+0000 7f9980a5d640 1 --2- 172.21.15.57:0/636314036 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f997c0afa20 0x7f997c12d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:05.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.200+0000 7f9980a5d640 1 -- 172.21.15.57:0/636314036 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9974009560 con 0x7f997c0b0530 2023-10-19T14:22:05.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.200+0000 7f9981a5f640 1 --2- 172.21.15.57:0/636314036 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f997c0cbd20 0x7f997c1413a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:22:05.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.200+0000 7f9980a5d640 1 --2- 172.21.15.57:0/636314036 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f997c0b0530 0x7f997c159d70 secure :-1 s=READY pgs=2885 cs=0 l=1 rev1=1 crypto rx=0x7f997000d750 tx=0x7f997000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:05.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.200+0000 7f9979ffb640 1 -- 172.21.15.57:0/636314036 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9970009bd0 con 0x7f997c0b0530 2023-10-19T14:22:05.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.200+0000 7f9979ffb640 1 -- 172.21.15.57:0/636314036 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f9970009e20 con 0x7f997c0b0530 2023-10-19T14:22:05.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.201+0000 7f9979ffb640 1 -- 172.21.15.57:0/636314036 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9970005280 con 0x7f997c0b0530 2023-10-19T14:22:05.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.201+0000 7f99834c2640 1 -- 172.21.15.57:0/636314036 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f997c142920 con 0x7f997c0b0530 2023-10-19T14:22:05.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.201+0000 7f99834c2640 1 -- 172.21.15.57:0/636314036 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f997c142ee0 con 0x7f997c0b0530 2023-10-19T14:22:05.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.202+0000 7f99834c2640 1 -- 172.21.15.57:0/636314036 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f997c143100 con 0x7f997c0b0530 2023-10-19T14:22:05.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.202+0000 7f9979ffb640 1 -- 172.21.15.57:0/636314036 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 145) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f99700053e0 con 0x7f997c0b0530 2023-10-19T14:22:05.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.202+0000 7f9979ffb640 1 --2- 172.21.15.57:0/636314036 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f996806b0b0 0x7f996806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:05.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.202+0000 7f9979ffb640 1 -- 172.21.15.57:0/636314036 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(263..263 src has 1..263) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f9970089f10 con 0x7f997c0b0530 2023-10-19T14:22:05.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.203+0000 7f998125e640 1 --2- 172.21.15.57:0/636314036 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f996806b0b0 0x7f996806d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:05.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.203+0000 7f998125e640 1 --2- 172.21.15.57:0/636314036 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f996806b0b0 0x7f996806d570 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f996c005fd0 tx=0x7f996c0097d0 comp rx=0 tx=0).ready entity=mgr.11446 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:05.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.207+0000 7f9979ffb640 1 -- 172.21.15.57:0/636314036 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9970054960 con 0x7f997c0b0530 2023-10-19T14:22:05.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.378+0000 7f99834c2640 1 -- 172.21.15.57:0/636314036 --> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f997c0c96b0 con 0x7f996806b0b0 2023-10-19T14:22:05.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.387+0000 7f9979ffb640 1 -- 172.21.15.57:0/636314036 <== mgr.11446 v2:172.21.15.179:6832/1573806521 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+288853 (secure 0 0 0) 0x7f997c0c96b0 con 0x7f996806b0b0 2023-10-19T14:22:05.393 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:22:05.395 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2023-10-19T14:22:05.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.390+0000 7f99834c2640 1 -- 172.21.15.57:0/636314036 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f996806b0b0 msgr2=0x7f996806d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:05.397 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.390+0000 7f99834c2640 1 --2- 172.21.15.57:0/636314036 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f996806b0b0 0x7f996806d570 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f996c005fd0 tx=0x7f996c0097d0 comp rx=0 tx=0).stop 2023-10-19T14:22:05.397 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.390+0000 7f99834c2640 1 -- 172.21.15.57:0/636314036 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f997c0b0530 msgr2=0x7f997c159d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:05.397 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.390+0000 7f99834c2640 1 --2- 172.21.15.57:0/636314036 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f997c0b0530 0x7f997c159d70 secure :-1 s=READY pgs=2885 cs=0 l=1 rev1=1 crypto rx=0x7f997000d750 tx=0x7f997000dc10 comp rx=0 tx=0).stop 2023-10-19T14:22:05.397 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.390+0000 7f99834c2640 1 -- 172.21.15.57:0/636314036 shutdown_connections 2023-10-19T14:22:05.398 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.390+0000 7f99834c2640 1 --2- 172.21.15.57:0/636314036 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f996806b0b0 0x7f996806d570 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:05.398 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.390+0000 7f99834c2640 1 --2- 172.21.15.57:0/636314036 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f997c0cbd20 0x7f997c1413a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:05.398 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.390+0000 7f99834c2640 1 --2- 172.21.15.57:0/636314036 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f997c0b0530 0x7f997c159d70 unknown :-1 s=CLOSED pgs=2885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:05.398 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.390+0000 7f99834c2640 1 --2- 172.21.15.57:0/636314036 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f997c0afa20 0x7f997c12d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:05.399 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.390+0000 7f99834c2640 1 -- 172.21.15.57:0/636314036 >> 172.21.15.57:0/636314036 conn(0x7f997c069930 msgr2=0x7f997c06b1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:05.399 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.390+0000 7f99834c2640 1 -- 172.21.15.57:0/636314036 shutdown_connections 2023-10-19T14:22:05.399 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.390+0000 7f99834c2640 1 -- 172.21.15.57:0/636314036 wait complete. 2023-10-19T14:22:05.404 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":28,"stamp":"2023-10-19T14:22:03.419299+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":591838,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":2539,"num_read_kb":5339,"num_write":697,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3257,"ondisk_log_size":3257,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":40,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":452732,"kb_used_data":26948,"kb_used_omap":12,"kb_used_meta":425651,"kb_avail":754521988,"statfs":{"total":773094113280,"available":772630515712,"internally_reserved":0,"allocated":27594752,"data_stored":14813661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":435867193},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1470,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000901"},"pg_stats":[{"pgid":"36.3f","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.407757+0000","last_change":"2023-10-19T14:22:02.407757+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.407757+0000","last_undegraded":"2023-10-19T14:22:02.407757+0000","last_fullsized":"2023-10-19T14:22:02.407757+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:14:21.361763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":["6","3"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.3e","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.406175+0000","last_change":"2023-10-19T14:22:02.406175+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.406175+0000","last_undegraded":"2023-10-19T14:22:02.406175+0000","last_fullsized":"2023-10-19T14:22:02.406175+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:23:52.735800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.3d","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.412915+0000","last_change":"2023-10-19T14:22:02.412915+0000","last_active":"2023-10-19T14:22:02.412915+0000","last_peered":"2023-10-19T14:22:02.412915+0000","last_clean":"2023-10-19T14:22:02.412915+0000","last_became_active":"2023-10-19T14:22:02.412634+0000","last_became_peered":"2023-10-19T14:22:02.412634+0000","last_unstale":"2023-10-19T14:22:02.412915+0000","last_undegraded":"2023-10-19T14:22:02.412915+0000","last_fullsized":"2023-10-19T14:22:02.412915+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:20:32.428977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.3c","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.403761+0000","last_change":"2023-10-19T14:22:02.403761+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.403761+0000","last_undegraded":"2023-10-19T14:22:02.403761+0000","last_fullsized":"2023-10-19T14:22:02.403761+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:50:46.770884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.3b","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.409103+0000","last_change":"2023-10-19T14:22:02.409103+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.409103+0000","last_undegraded":"2023-10-19T14:22:02.409103+0000","last_fullsized":"2023-10-19T14:22:02.409103+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:03:49.877672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.3a","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.408280+0000","last_change":"2023-10-19T14:22:02.408280+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.408280+0000","last_undegraded":"2023-10-19T14:22:02.408280+0000","last_fullsized":"2023-10-19T14:22:02.408280+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:18:59.130442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.39","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.413984+0000","last_change":"2023-10-19T14:22:02.413984+0000","last_active":"2023-10-19T14:22:02.413984+0000","last_peered":"2023-10-19T14:22:02.413984+0000","last_clean":"2023-10-19T14:22:02.413984+0000","last_became_active":"2023-10-19T14:22:02.413714+0000","last_became_peered":"2023-10-19T14:22:02.413714+0000","last_unstale":"2023-10-19T14:22:02.413984+0000","last_undegraded":"2023-10-19T14:22:02.413984+0000","last_fullsized":"2023-10-19T14:22:02.413984+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:03:55.524277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.38","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.403768+0000","last_change":"2023-10-19T14:22:02.403768+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.403768+0000","last_undegraded":"2023-10-19T14:22:02.403768+0000","last_fullsized":"2023-10-19T14:22:02.403768+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:30:01.392073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.37","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.404303+0000","last_change":"2023-10-19T14:22:02.404303+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.404303+0000","last_undegraded":"2023-10-19T14:22:02.404303+0000","last_fullsized":"2023-10-19T14:22:02.404303+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:16:24.272266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.36","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.413432+0000","last_change":"2023-10-19T14:22:02.413432+0000","last_active":"2023-10-19T14:22:02.413432+0000","last_peered":"2023-10-19T14:22:02.413432+0000","last_clean":"2023-10-19T14:22:02.413432+0000","last_became_active":"2023-10-19T14:22:02.412808+0000","last_became_peered":"2023-10-19T14:22:02.412808+0000","last_unstale":"2023-10-19T14:22:02.413432+0000","last_undegraded":"2023-10-19T14:22:02.413432+0000","last_fullsized":"2023-10-19T14:22:02.413432+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T02:11:11.663612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.35","version":"0'0","reported_seq":9,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.418007+0000","last_change":"2023-10-19T14:22:02.413336+0000","last_active":"2023-10-19T14:22:02.418007+0000","last_peered":"2023-10-19T14:22:02.418007+0000","last_clean":"2023-10-19T14:22:02.418007+0000","last_became_active":"2023-10-19T14:22:02.412942+0000","last_became_peered":"2023-10-19T14:22:02.412942+0000","last_unstale":"2023-10-19T14:22:02.418007+0000","last_undegraded":"2023-10-19T14:22:02.418007+0000","last_fullsized":"2023-10-19T14:22:02.418007+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:37:40.908777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.34","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.406632+0000","last_change":"2023-10-19T14:22:02.406632+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.406632+0000","last_undegraded":"2023-10-19T14:22:02.406632+0000","last_fullsized":"2023-10-19T14:22:02.406632+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:30:39.654226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":["6","3"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.33","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.403708+0000","last_change":"2023-10-19T14:22:02.403708+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.403708+0000","last_undegraded":"2023-10-19T14:22:02.403708+0000","last_fullsized":"2023-10-19T14:22:02.403708+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:39:38.146246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.32","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.403725+0000","last_change":"2023-10-19T14:22:02.403725+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.403725+0000","last_undegraded":"2023-10-19T14:22:02.403725+0000","last_fullsized":"2023-10-19T14:22:02.403725+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:10:47.778168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.31","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.403277+0000","last_change":"2023-10-19T14:22:02.403277+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.403277+0000","last_undegraded":"2023-10-19T14:22:02.403277+0000","last_fullsized":"2023-10-19T14:22:02.403277+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:15:36.803736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.30","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.420492+0000","last_change":"2023-10-19T14:22:02.420492+0000","last_active":"2023-10-19T14:22:02.420492+0000","last_peered":"2023-10-19T14:22:02.420492+0000","last_clean":"2023-10-19T14:22:02.420492+0000","last_became_active":"2023-10-19T14:22:02.420210+0000","last_became_peered":"2023-10-19T14:22:02.420210+0000","last_unstale":"2023-10-19T14:22:02.420492+0000","last_undegraded":"2023-10-19T14:22:02.420492+0000","last_fullsized":"2023-10-19T14:22:02.420492+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:38:11.628588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.2f","version":"0'0","reported_seq":9,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.420525+0000","last_change":"2023-10-19T14:22:02.415526+0000","last_active":"2023-10-19T14:22:02.420525+0000","last_peered":"2023-10-19T14:22:02.420525+0000","last_clean":"2023-10-19T14:22:02.420525+0000","last_became_active":"2023-10-19T14:22:02.415243+0000","last_became_peered":"2023-10-19T14:22:02.415243+0000","last_unstale":"2023-10-19T14:22:02.420525+0000","last_undegraded":"2023-10-19T14:22:02.420525+0000","last_fullsized":"2023-10-19T14:22:02.420525+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:25:08.178761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.2e","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.405331+0000","last_change":"2023-10-19T14:22:02.405331+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.405331+0000","last_undegraded":"2023-10-19T14:22:02.405331+0000","last_fullsized":"2023-10-19T14:22:02.405331+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:49:42.930618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.2d","version":"0'0","reported_seq":9,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.420730+0000","last_change":"2023-10-19T14:22:02.418292+0000","last_active":"2023-10-19T14:22:02.420730+0000","last_peered":"2023-10-19T14:22:02.420730+0000","last_clean":"2023-10-19T14:22:02.420730+0000","last_became_active":"2023-10-19T14:22:02.415107+0000","last_became_peered":"2023-10-19T14:22:02.415107+0000","last_unstale":"2023-10-19T14:22:02.420730+0000","last_undegraded":"2023-10-19T14:22:02.420730+0000","last_fullsized":"2023-10-19T14:22:02.420730+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:22:21.202909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.2c","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.421040+0000","last_change":"2023-10-19T14:22:02.421040+0000","last_active":"2023-10-19T14:22:02.421040+0000","last_peered":"2023-10-19T14:22:02.421040+0000","last_clean":"2023-10-19T14:22:02.421040+0000","last_became_active":"2023-10-19T14:22:02.420785+0000","last_became_peered":"2023-10-19T14:22:02.420785+0000","last_unstale":"2023-10-19T14:22:02.421040+0000","last_undegraded":"2023-10-19T14:22:02.421040+0000","last_fullsized":"2023-10-19T14:22:02.421040+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:48:32.972469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.2b","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.418617+0000","last_change":"2023-10-19T14:22:02.418617+0000","last_active":"2023-10-19T14:22:02.418617+0000","last_peered":"2023-10-19T14:22:02.418617+0000","last_clean":"2023-10-19T14:22:02.418617+0000","last_became_active":"2023-10-19T14:22:02.418348+0000","last_became_peered":"2023-10-19T14:22:02.418348+0000","last_unstale":"2023-10-19T14:22:02.418617+0000","last_undegraded":"2023-10-19T14:22:02.418617+0000","last_fullsized":"2023-10-19T14:22:02.418617+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:03:48.649163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.2a","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.399641+0000","last_change":"2023-10-19T14:22:02.399641+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.399641+0000","last_undegraded":"2023-10-19T14:22:02.399641+0000","last_fullsized":"2023-10-19T14:22:02.399641+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:35:08.037582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.29","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.405143+0000","last_change":"2023-10-19T14:22:02.405143+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.405143+0000","last_undegraded":"2023-10-19T14:22:02.405143+0000","last_fullsized":"2023-10-19T14:22:02.405143+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:09:54.096828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.28","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.398717+0000","last_change":"2023-10-19T14:22:02.398717+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.398717+0000","last_undegraded":"2023-10-19T14:22:02.398717+0000","last_fullsized":"2023-10-19T14:22:02.398717+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:35:36.604007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.27","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.417381+0000","last_change":"2023-10-19T14:22:02.417381+0000","last_active":"2023-10-19T14:22:02.417381+0000","last_peered":"2023-10-19T14:22:02.417381+0000","last_clean":"2023-10-19T14:22:02.417381+0000","last_became_active":"2023-10-19T14:22:02.417103+0000","last_became_peered":"2023-10-19T14:22:02.417103+0000","last_unstale":"2023-10-19T14:22:02.417381+0000","last_undegraded":"2023-10-19T14:22:02.417381+0000","last_fullsized":"2023-10-19T14:22:02.417381+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:05:25.934207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.26","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.410522+0000","last_change":"2023-10-19T14:22:02.410522+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.410522+0000","last_undegraded":"2023-10-19T14:22:02.410522+0000","last_fullsized":"2023-10-19T14:22:02.410522+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:41:53.947719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.24","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.407514+0000","last_change":"2023-10-19T14:22:02.407514+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.407514+0000","last_undegraded":"2023-10-19T14:22:02.407514+0000","last_fullsized":"2023-10-19T14:22:02.407514+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:14:37.969686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.23","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.398963+0000","last_change":"2023-10-19T14:22:02.398963+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.398963+0000","last_undegraded":"2023-10-19T14:22:02.398963+0000","last_fullsized":"2023-10-19T14:22:02.398963+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:57:49.145921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.22","version":"0'0","reported_seq":9,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.404880+0000","last_change":"2023-10-19T14:22:02.402715+0000","last_active":"2023-10-19T14:22:02.404880+0000","last_peered":"2023-10-19T14:22:02.404880+0000","last_clean":"2023-10-19T14:22:02.404880+0000","last_became_active":"2023-10-19T14:22:02.402451+0000","last_became_peered":"2023-10-19T14:22:02.402451+0000","last_unstale":"2023-10-19T14:22:02.404880+0000","last_undegraded":"2023-10-19T14:22:02.404880+0000","last_fullsized":"2023-10-19T14:22:02.404880+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:42:44.848665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.21","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.399356+0000","last_change":"2023-10-19T14:22:02.399356+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.399356+0000","last_undegraded":"2023-10-19T14:22:02.399356+0000","last_fullsized":"2023-10-19T14:22:02.399356+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:45:44.957902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.20","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.403114+0000","last_change":"2023-10-19T14:22:02.403114+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.403114+0000","last_undegraded":"2023-10-19T14:22:02.403114+0000","last_fullsized":"2023-10-19T14:22:02.403114+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T02:18:23.251534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.18","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.398614+0000","last_change":"2023-10-19T14:22:02.398614+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.398614+0000","last_undegraded":"2023-10-19T14:22:02.398614+0000","last_fullsized":"2023-10-19T14:22:02.398614+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:31:51.736865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.1f","version":"262'4","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.404439+0000","last_change":"2023-10-19T14:22:01.414254+0000","last_active":"2023-10-19T14:22:02.404439+0000","last_peered":"2023-10-19T14:22:02.404439+0000","last_clean":"2023-10-19T14:22:02.404439+0000","last_became_active":"2023-10-19T14:22:01.413084+0000","last_became_peered":"2023-10-19T14:22:01.413084+0000","last_unstale":"2023-10-19T14:22:02.404439+0000","last_undegraded":"2023-10-19T14:22:02.404439+0000","last_fullsized":"2023-10-19T14:22:02.404439+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:04:17.277980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.19","version":"0'0","reported_seq":9,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.416536+0000","last_change":"2023-10-19T14:22:02.412925+0000","last_active":"2023-10-19T14:22:02.416536+0000","last_peered":"2023-10-19T14:22:02.416536+0000","last_clean":"2023-10-19T14:22:02.416536+0000","last_became_active":"2023-10-19T14:22:02.412639+0000","last_became_peered":"2023-10-19T14:22:02.412639+0000","last_unstale":"2023-10-19T14:22:02.416536+0000","last_undegraded":"2023-10-19T14:22:02.416536+0000","last_fullsized":"2023-10-19T14:22:02.416536+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:14:38.697589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.1e","version":"0'0","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.399308+0000","last_change":"2023-10-19T14:22:01.410546+0000","last_active":"2023-10-19T14:22:02.399308+0000","last_peered":"2023-10-19T14:22:02.399308+0000","last_clean":"2023-10-19T14:22:02.399308+0000","last_became_active":"2023-10-19T14:22:01.410263+0000","last_became_peered":"2023-10-19T14:22:01.410263+0000","last_unstale":"2023-10-19T14:22:02.399308+0000","last_undegraded":"2023-10-19T14:22:02.399308+0000","last_fullsized":"2023-10-19T14:22:02.399308+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:02:12.137179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.1a","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.406083+0000","last_change":"2023-10-19T14:22:02.406083+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.406083+0000","last_undegraded":"2023-10-19T14:22:02.406083+0000","last_fullsized":"2023-10-19T14:22:02.406083+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:07:50.835228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.1d","version":"262'2","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.400756+0000","last_change":"2023-10-19T14:22:01.417975+0000","last_active":"2023-10-19T14:22:02.400756+0000","last_peered":"2023-10-19T14:22:02.400756+0000","last_clean":"2023-10-19T14:22:02.400756+0000","last_became_active":"2023-10-19T14:22:01.417704+0000","last_became_peered":"2023-10-19T14:22:01.417704+0000","last_unstale":"2023-10-19T14:22:02.400756+0000","last_undegraded":"2023-10-19T14:22:02.400756+0000","last_fullsized":"2023-10-19T14:22:02.400756+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:56:00.152228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.1b","version":"0'0","reported_seq":9,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.416951+0000","last_change":"2023-10-19T14:22:02.412219+0000","last_active":"2023-10-19T14:22:02.416951+0000","last_peered":"2023-10-19T14:22:02.416951+0000","last_clean":"2023-10-19T14:22:02.416951+0000","last_became_active":"2023-10-19T14:22:02.411959+0000","last_became_peered":"2023-10-19T14:22:02.411959+0000","last_unstale":"2023-10-19T14:22:02.416951+0000","last_undegraded":"2023-10-19T14:22:02.416951+0000","last_fullsized":"2023-10-19T14:22:02.416951+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:54:56.271440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.1c","version":"262'2","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.400340+0000","last_change":"2023-10-19T14:22:01.419018+0000","last_active":"2023-10-19T14:22:02.400340+0000","last_peered":"2023-10-19T14:22:02.400340+0000","last_clean":"2023-10-19T14:22:02.400340+0000","last_became_active":"2023-10-19T14:22:01.418769+0000","last_became_peered":"2023-10-19T14:22:01.418769+0000","last_unstale":"2023-10-19T14:22:02.400340+0000","last_undegraded":"2023-10-19T14:22:02.400340+0000","last_fullsized":"2023-10-19T14:22:02.400340+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:59:43.263869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.1c","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.402955+0000","last_change":"2023-10-19T14:22:02.402955+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.402955+0000","last_undegraded":"2023-10-19T14:22:02.402955+0000","last_fullsized":"2023-10-19T14:22:02.402955+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:03:27.890218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.1b","version":"262'1","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.409720+0000","last_change":"2023-10-19T14:22:01.413953+0000","last_active":"2023-10-19T14:22:02.409720+0000","last_peered":"2023-10-19T14:22:02.409720+0000","last_clean":"2023-10-19T14:22:02.409720+0000","last_became_active":"2023-10-19T14:22:01.413689+0000","last_became_peered":"2023-10-19T14:22:01.413689+0000","last_unstale":"2023-10-19T14:22:02.409720+0000","last_undegraded":"2023-10-19T14:22:02.409720+0000","last_fullsized":"2023-10-19T14:22:02.409720+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:33:45.430960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.1d","version":"0'0","reported_seq":6,"reported_epoch":262,"state":"active","last_fresh":"2023-10-19T14:22:02.413928+0000","last_change":"2023-10-19T14:22:02.413862+0000","last_active":"2023-10-19T14:22:02.413928+0000","last_peered":"2023-10-19T14:22:02.413928+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"2023-10-19T14:22:02.413862+0000","last_became_peered":"2023-10-19T14:22:02.413862+0000","last_unstale":"2023-10-19T14:22:02.413928+0000","last_undegraded":"2023-10-19T14:22:02.413928+0000","last_fullsized":"2023-10-19T14:22:02.413928+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:20:55.394812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.1a","version":"0'0","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.408577+0000","last_change":"2023-10-19T14:22:01.409991+0000","last_active":"2023-10-19T14:22:02.408577+0000","last_peered":"2023-10-19T14:22:02.408577+0000","last_clean":"2023-10-19T14:22:02.408577+0000","last_became_active":"2023-10-19T14:22:01.409510+0000","last_became_peered":"2023-10-19T14:22:01.409510+0000","last_unstale":"2023-10-19T14:22:02.408577+0000","last_undegraded":"2023-10-19T14:22:02.408577+0000","last_fullsized":"2023-10-19T14:22:02.408577+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:42:53.919625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.1e","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.412327+0000","last_change":"2023-10-19T14:22:02.412327+0000","last_active":"2023-10-19T14:22:02.412327+0000","last_peered":"2023-10-19T14:22:02.412327+0000","last_clean":"2023-10-19T14:22:02.412327+0000","last_became_active":"2023-10-19T14:22:02.412057+0000","last_became_peered":"2023-10-19T14:22:02.412057+0000","last_unstale":"2023-10-19T14:22:02.412327+0000","last_undegraded":"2023-10-19T14:22:02.412327+0000","last_fullsized":"2023-10-19T14:22:02.412327+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:32:30.545638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.19","version":"0'0","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.399288+0000","last_change":"2023-10-19T14:22:01.416901+0000","last_active":"2023-10-19T14:22:02.399288+0000","last_peered":"2023-10-19T14:22:02.399288+0000","last_clean":"2023-10-19T14:22:02.399288+0000","last_became_active":"2023-10-19T14:22:01.416656+0000","last_became_peered":"2023-10-19T14:22:01.416656+0000","last_unstale":"2023-10-19T14:22:02.399288+0000","last_undegraded":"2023-10-19T14:22:02.399288+0000","last_fullsized":"2023-10-19T14:22:02.399288+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:27:36.174418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.1f","version":"0'0","reported_seq":9,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.420545+0000","last_change":"2023-10-19T14:22:02.418425+0000","last_active":"2023-10-19T14:22:02.420545+0000","last_peered":"2023-10-19T14:22:02.420545+0000","last_clean":"2023-10-19T14:22:02.420545+0000","last_became_active":"2023-10-19T14:22:02.417278+0000","last_became_peered":"2023-10-19T14:22:02.417278+0000","last_unstale":"2023-10-19T14:22:02.420545+0000","last_undegraded":"2023-10-19T14:22:02.420545+0000","last_fullsized":"2023-10-19T14:22:02.420545+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:04:24.989932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.18","version":"0'0","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.404196+0000","last_change":"2023-10-19T14:22:01.412557+0000","last_active":"2023-10-19T14:22:02.404196+0000","last_peered":"2023-10-19T14:22:02.404196+0000","last_clean":"2023-10-19T14:22:02.404196+0000","last_became_active":"2023-10-19T14:22:01.412115+0000","last_became_peered":"2023-10-19T14:22:01.412115+0000","last_unstale":"2023-10-19T14:22:02.404196+0000","last_undegraded":"2023-10-19T14:22:02.404196+0000","last_fullsized":"2023-10-19T14:22:02.404196+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:54:33.650542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.10","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.402706+0000","last_change":"2023-10-19T14:22:02.402706+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.402706+0000","last_undegraded":"2023-10-19T14:22:02.402706+0000","last_fullsized":"2023-10-19T14:22:02.402706+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:45:46.279692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.17","version":"0'0","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.415002+0000","last_change":"2023-10-19T14:22:01.412962+0000","last_active":"2023-10-19T14:22:02.415002+0000","last_peered":"2023-10-19T14:22:02.415002+0000","last_clean":"2023-10-19T14:22:02.415002+0000","last_became_active":"2023-10-19T14:22:01.412680+0000","last_became_peered":"2023-10-19T14:22:01.412680+0000","last_unstale":"2023-10-19T14:22:02.415002+0000","last_undegraded":"2023-10-19T14:22:02.415002+0000","last_fullsized":"2023-10-19T14:22:02.415002+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:03:54.917148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.11","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.405750+0000","last_change":"2023-10-19T14:22:02.405750+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.405750+0000","last_undegraded":"2023-10-19T14:22:02.405750+0000","last_fullsized":"2023-10-19T14:22:02.405750+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:06:30.359305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.16","version":"0'0","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.399221+0000","last_change":"2023-10-19T14:22:01.414464+0000","last_active":"2023-10-19T14:22:02.399221+0000","last_peered":"2023-10-19T14:22:02.399221+0000","last_clean":"2023-10-19T14:22:02.399221+0000","last_became_active":"2023-10-19T14:22:01.413414+0000","last_became_peered":"2023-10-19T14:22:01.413414+0000","last_unstale":"2023-10-19T14:22:02.399221+0000","last_undegraded":"2023-10-19T14:22:02.399221+0000","last_fullsized":"2023-10-19T14:22:02.399221+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:46:22.763844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.12","version":"0'0","reported_seq":9,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.413606+0000","last_change":"2023-10-19T14:22:02.410248+0000","last_active":"2023-10-19T14:22:02.413606+0000","last_peered":"2023-10-19T14:22:02.413606+0000","last_clean":"2023-10-19T14:22:02.413606+0000","last_became_active":"2023-10-19T14:22:02.409982+0000","last_became_peered":"2023-10-19T14:22:02.409982+0000","last_unstale":"2023-10-19T14:22:02.413606+0000","last_undegraded":"2023-10-19T14:22:02.413606+0000","last_fullsized":"2023-10-19T14:22:02.413606+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:27:43.397970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.15","version":"262'3","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.414741+0000","last_change":"2023-10-19T14:22:01.409543+0000","last_active":"2023-10-19T14:22:02.414741+0000","last_peered":"2023-10-19T14:22:02.414741+0000","last_clean":"2023-10-19T14:22:02.414741+0000","last_became_active":"2023-10-19T14:22:01.409293+0000","last_became_peered":"2023-10-19T14:22:01.409293+0000","last_unstale":"2023-10-19T14:22:02.414741+0000","last_undegraded":"2023-10-19T14:22:02.414741+0000","last_fullsized":"2023-10-19T14:22:02.414741+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:12:16.850155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.13","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.408938+0000","last_change":"2023-10-19T14:22:02.408938+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.408938+0000","last_undegraded":"2023-10-19T14:22:02.408938+0000","last_fullsized":"2023-10-19T14:22:02.408938+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:22:40.195315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.14","version":"262'5","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.404183+0000","last_change":"2023-10-19T14:22:01.411386+0000","last_active":"2023-10-19T14:22:02.404183+0000","last_peered":"2023-10-19T14:22:02.404183+0000","last_clean":"2023-10-19T14:22:02.404183+0000","last_became_active":"2023-10-19T14:22:01.411111+0000","last_became_peered":"2023-10-19T14:22:01.411111+0000","last_unstale":"2023-10-19T14:22:02.404183+0000","last_undegraded":"2023-10-19T14:22:02.404183+0000","last_fullsized":"2023-10-19T14:22:02.404183+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:10:26.534502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.14","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.416878+0000","last_change":"2023-10-19T14:22:02.416878+0000","last_active":"2023-10-19T14:22:02.416878+0000","last_peered":"2023-10-19T14:22:02.416878+0000","last_clean":"2023-10-19T14:22:02.416878+0000","last_became_active":"2023-10-19T14:22:02.416633+0000","last_became_peered":"2023-10-19T14:22:02.416633+0000","last_unstale":"2023-10-19T14:22:02.416878+0000","last_undegraded":"2023-10-19T14:22:02.416878+0000","last_fullsized":"2023-10-19T14:22:02.416878+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:56:41.835345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.13","version":"262'2","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.399368+0000","last_change":"2023-10-19T14:22:01.410762+0000","last_active":"2023-10-19T14:22:02.399368+0000","last_peered":"2023-10-19T14:22:02.399368+0000","last_clean":"2023-10-19T14:22:02.399368+0000","last_became_active":"2023-10-19T14:22:01.410507+0000","last_became_peered":"2023-10-19T14:22:01.410507+0000","last_unstale":"2023-10-19T14:22:02.399368+0000","last_undegraded":"2023-10-19T14:22:02.399368+0000","last_fullsized":"2023-10-19T14:22:02.399368+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:38:13.918889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.15","version":"0'0","reported_seq":6,"reported_epoch":262,"state":"active","last_fresh":"2023-10-19T14:22:02.414429+0000","last_change":"2023-10-19T14:22:02.414358+0000","last_active":"2023-10-19T14:22:02.414429+0000","last_peered":"2023-10-19T14:22:02.414429+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"2023-10-19T14:22:02.414358+0000","last_became_peered":"2023-10-19T14:22:02.414358+0000","last_unstale":"2023-10-19T14:22:02.414429+0000","last_undegraded":"2023-10-19T14:22:02.414429+0000","last_fullsized":"2023-10-19T14:22:02.414429+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:02:05.922124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.12","version":"0'0","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.409323+0000","last_change":"2023-10-19T14:22:01.413870+0000","last_active":"2023-10-19T14:22:02.409323+0000","last_peered":"2023-10-19T14:22:02.409323+0000","last_clean":"2023-10-19T14:22:02.409323+0000","last_became_active":"2023-10-19T14:22:01.413532+0000","last_became_peered":"2023-10-19T14:22:01.413532+0000","last_unstale":"2023-10-19T14:22:02.409323+0000","last_undegraded":"2023-10-19T14:22:02.409323+0000","last_fullsized":"2023-10-19T14:22:02.409323+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:47:48.522113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.16","version":"0'0","reported_seq":9,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.421387+0000","last_change":"2023-10-19T14:22:02.419152+0000","last_active":"2023-10-19T14:22:02.421387+0000","last_peered":"2023-10-19T14:22:02.421387+0000","last_clean":"2023-10-19T14:22:02.421387+0000","last_became_active":"2023-10-19T14:22:02.417515+0000","last_became_peered":"2023-10-19T14:22:02.417515+0000","last_unstale":"2023-10-19T14:22:02.421387+0000","last_undegraded":"2023-10-19T14:22:02.421387+0000","last_fullsized":"2023-10-19T14:22:02.421387+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:39:40.912960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.11","version":"0'0","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.409983+0000","last_change":"2023-10-19T14:22:01.416474+0000","last_active":"2023-10-19T14:22:02.409983+0000","last_peered":"2023-10-19T14:22:02.409983+0000","last_clean":"2023-10-19T14:22:02.409983+0000","last_became_active":"2023-10-19T14:22:01.416200+0000","last_became_peered":"2023-10-19T14:22:01.416200+0000","last_unstale":"2023-10-19T14:22:02.409983+0000","last_undegraded":"2023-10-19T14:22:02.409983+0000","last_fullsized":"2023-10-19T14:22:02.409983+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:55:39.137949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.17","version":"0'0","reported_seq":9,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.421450+0000","last_change":"2023-10-19T14:22:02.419116+0000","last_active":"2023-10-19T14:22:02.421450+0000","last_peered":"2023-10-19T14:22:02.421450+0000","last_clean":"2023-10-19T14:22:02.421450+0000","last_became_active":"2023-10-19T14:22:02.418655+0000","last_became_peered":"2023-10-19T14:22:02.418655+0000","last_unstale":"2023-10-19T14:22:02.421450+0000","last_undegraded":"2023-10-19T14:22:02.421450+0000","last_fullsized":"2023-10-19T14:22:02.421450+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:28:48.304947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.10","version":"262'8","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.402744+0000","last_change":"2023-10-19T14:22:01.415486+0000","last_active":"2023-10-19T14:22:02.402744+0000","last_peered":"2023-10-19T14:22:02.402744+0000","last_clean":"2023-10-19T14:22:02.402744+0000","last_became_active":"2023-10-19T14:22:01.415192+0000","last_became_peered":"2023-10-19T14:22:01.415192+0000","last_unstale":"2023-10-19T14:22:02.402744+0000","last_undegraded":"2023-10-19T14:22:02.402744+0000","last_fullsized":"2023-10-19T14:22:02.402744+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:45:03.326043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.8","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.407065+0000","last_change":"2023-10-19T14:22:02.407065+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.407065+0000","last_undegraded":"2023-10-19T14:22:02.407065+0000","last_fullsized":"2023-10-19T14:22:02.407065+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:07:55.991807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.f","version":"262'1","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.417942+0000","last_change":"2023-10-19T14:22:01.417521+0000","last_active":"2023-10-19T14:22:02.417942+0000","last_peered":"2023-10-19T14:22:02.417942+0000","last_clean":"2023-10-19T14:22:02.417942+0000","last_became_active":"2023-10-19T14:22:01.417209+0000","last_became_peered":"2023-10-19T14:22:01.417209+0000","last_unstale":"2023-10-19T14:22:02.417942+0000","last_undegraded":"2023-10-19T14:22:02.417942+0000","last_fullsized":"2023-10-19T14:22:02.417942+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:43:26.494066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.9","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.398780+0000","last_change":"2023-10-19T14:22:02.398780+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.398780+0000","last_undegraded":"2023-10-19T14:22:02.398780+0000","last_fullsized":"2023-10-19T14:22:02.398780+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:40:53.511724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.e","version":"0'0","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.411175+0000","last_change":"2023-10-19T14:22:01.412531+0000","last_active":"2023-10-19T14:22:02.411175+0000","last_peered":"2023-10-19T14:22:02.411175+0000","last_clean":"2023-10-19T14:22:02.411175+0000","last_became_active":"2023-10-19T14:22:01.412209+0000","last_became_peered":"2023-10-19T14:22:01.412209+0000","last_unstale":"2023-10-19T14:22:02.411175+0000","last_undegraded":"2023-10-19T14:22:02.411175+0000","last_fullsized":"2023-10-19T14:22:02.411175+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:45:03.326043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.a","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.407858+0000","last_change":"2023-10-19T14:22:02.407858+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.407858+0000","last_undegraded":"2023-10-19T14:22:02.407858+0000","last_fullsized":"2023-10-19T14:22:02.407858+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:31:43.355728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.d","version":"262'1","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.400892+0000","last_change":"2023-10-19T14:22:01.415741+0000","last_active":"2023-10-19T14:22:02.400892+0000","last_peered":"2023-10-19T14:22:02.400892+0000","last_clean":"2023-10-19T14:22:02.400892+0000","last_became_active":"2023-10-19T14:22:01.415385+0000","last_became_peered":"2023-10-19T14:22:01.415385+0000","last_unstale":"2023-10-19T14:22:02.400892+0000","last_undegraded":"2023-10-19T14:22:02.400892+0000","last_fullsized":"2023-10-19T14:22:02.400892+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:34:06.962782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.b","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.413768+0000","last_change":"2023-10-19T14:22:02.413768+0000","last_active":"2023-10-19T14:22:02.413768+0000","last_peered":"2023-10-19T14:22:02.413768+0000","last_clean":"2023-10-19T14:22:02.413768+0000","last_became_active":"2023-10-19T14:22:02.413498+0000","last_became_peered":"2023-10-19T14:22:02.413498+0000","last_unstale":"2023-10-19T14:22:02.413768+0000","last_undegraded":"2023-10-19T14:22:02.413768+0000","last_fullsized":"2023-10-19T14:22:02.413768+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:09:01.416730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.c","version":"0'0","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.399235+0000","last_change":"2023-10-19T14:22:01.411380+0000","last_active":"2023-10-19T14:22:02.399235+0000","last_peered":"2023-10-19T14:22:02.399235+0000","last_clean":"2023-10-19T14:22:02.399235+0000","last_became_active":"2023-10-19T14:22:01.411092+0000","last_became_peered":"2023-10-19T14:22:01.411092+0000","last_unstale":"2023-10-19T14:22:02.399235+0000","last_undegraded":"2023-10-19T14:22:02.399235+0000","last_fullsized":"2023-10-19T14:22:02.399235+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:01:14.941331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.c","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.413516+0000","last_change":"2023-10-19T14:22:02.413516+0000","last_active":"2023-10-19T14:22:02.413516+0000","last_peered":"2023-10-19T14:22:02.413516+0000","last_clean":"2023-10-19T14:22:02.413516+0000","last_became_active":"2023-10-19T14:22:02.413230+0000","last_became_peered":"2023-10-19T14:22:02.413230+0000","last_unstale":"2023-10-19T14:22:02.413516+0000","last_undegraded":"2023-10-19T14:22:02.413516+0000","last_fullsized":"2023-10-19T14:22:02.413516+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:22:42.002875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.b","version":"262'2","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.399530+0000","last_change":"2023-10-19T14:22:01.409927+0000","last_active":"2023-10-19T14:22:02.399530+0000","last_peered":"2023-10-19T14:22:02.399530+0000","last_clean":"2023-10-19T14:22:02.399530+0000","last_became_active":"2023-10-19T14:22:01.409617+0000","last_became_peered":"2023-10-19T14:22:01.409617+0000","last_unstale":"2023-10-19T14:22:02.399530+0000","last_undegraded":"2023-10-19T14:22:02.399530+0000","last_fullsized":"2023-10-19T14:22:02.399530+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:38:30.217600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.d","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.399329+0000","last_change":"2023-10-19T14:22:02.399329+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.399329+0000","last_undegraded":"2023-10-19T14:22:02.399329+0000","last_fullsized":"2023-10-19T14:22:02.399329+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:17:41.707119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.a","version":"0'0","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.404454+0000","last_change":"2023-10-19T14:22:01.417576+0000","last_active":"2023-10-19T14:22:02.404454+0000","last_peered":"2023-10-19T14:22:02.404454+0000","last_clean":"2023-10-19T14:22:02.404454+0000","last_became_active":"2023-10-19T14:22:01.417230+0000","last_became_peered":"2023-10-19T14:22:01.417230+0000","last_unstale":"2023-10-19T14:22:02.404454+0000","last_undegraded":"2023-10-19T14:22:02.404454+0000","last_fullsized":"2023-10-19T14:22:02.404454+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:23:00.582706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.e","version":"0'0","reported_seq":9,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.421407+0000","last_change":"2023-10-19T14:22:02.419344+0000","last_active":"2023-10-19T14:22:02.421407+0000","last_peered":"2023-10-19T14:22:02.421407+0000","last_clean":"2023-10-19T14:22:02.421407+0000","last_became_active":"2023-10-19T14:22:02.417238+0000","last_became_peered":"2023-10-19T14:22:02.417238+0000","last_unstale":"2023-10-19T14:22:02.421407+0000","last_undegraded":"2023-10-19T14:22:02.421407+0000","last_fullsized":"2023-10-19T14:22:02.421407+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:19:36.485520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.9","version":"0'0","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.400018+0000","last_change":"2023-10-19T14:22:01.413457+0000","last_active":"2023-10-19T14:22:02.400018+0000","last_peered":"2023-10-19T14:22:02.400018+0000","last_clean":"2023-10-19T14:22:02.400018+0000","last_became_active":"2023-10-19T14:22:01.413179+0000","last_became_peered":"2023-10-19T14:22:01.413179+0000","last_unstale":"2023-10-19T14:22:02.400018+0000","last_undegraded":"2023-10-19T14:22:02.400018+0000","last_fullsized":"2023-10-19T14:22:02.400018+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:34:30.334044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.f","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.409049+0000","last_change":"2023-10-19T14:22:02.409049+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.409049+0000","last_undegraded":"2023-10-19T14:22:02.409049+0000","last_fullsized":"2023-10-19T14:22:02.409049+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:44:13.928436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.8","version":"0'0","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.399188+0000","last_change":"2023-10-19T14:22:01.413949+0000","last_active":"2023-10-19T14:22:02.399188+0000","last_peered":"2023-10-19T14:22:02.399188+0000","last_clean":"2023-10-19T14:22:02.399188+0000","last_became_active":"2023-10-19T14:22:01.413670+0000","last_became_peered":"2023-10-19T14:22:01.413670+0000","last_unstale":"2023-10-19T14:22:02.399188+0000","last_undegraded":"2023-10-19T14:22:02.399188+0000","last_fullsized":"2023-10-19T14:22:02.399188+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:33:35.792891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.0","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.402684+0000","last_change":"2023-10-19T14:22:02.402684+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.402684+0000","last_undegraded":"2023-10-19T14:22:02.402684+0000","last_fullsized":"2023-10-19T14:22:02.402684+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:40:48.301290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.7","version":"262'2","reported_seq":13,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.413481+0000","last_change":"2023-10-19T14:22:01.412421+0000","last_active":"2023-10-19T14:22:02.413481+0000","last_peered":"2023-10-19T14:22:02.413481+0000","last_clean":"2023-10-19T14:22:02.413481+0000","last_became_active":"2023-10-19T14:22:01.412153+0000","last_became_peered":"2023-10-19T14:22:01.412153+0000","last_unstale":"2023-10-19T14:22:02.413481+0000","last_undegraded":"2023-10-19T14:22:02.413481+0000","last_fullsized":"2023-10-19T14:22:02.413481+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:38:47.923739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.1","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.419843+0000","last_change":"2023-10-19T14:22:02.419843+0000","last_active":"2023-10-19T14:22:02.419843+0000","last_peered":"2023-10-19T14:22:02.419843+0000","last_clean":"2023-10-19T14:22:02.419843+0000","last_became_active":"2023-10-19T14:22:02.417481+0000","last_became_peered":"2023-10-19T14:22:02.417481+0000","last_unstale":"2023-10-19T14:22:02.419843+0000","last_undegraded":"2023-10-19T14:22:02.419843+0000","last_fullsized":"2023-10-19T14:22:02.419843+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:12:12.862561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.6","version":"0'0","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.409952+0000","last_change":"2023-10-19T14:22:01.412088+0000","last_active":"2023-10-19T14:22:02.409952+0000","last_peered":"2023-10-19T14:22:02.409952+0000","last_clean":"2023-10-19T14:22:02.409952+0000","last_became_active":"2023-10-19T14:22:01.411646+0000","last_became_peered":"2023-10-19T14:22:01.411646+0000","last_unstale":"2023-10-19T14:22:02.409952+0000","last_undegraded":"2023-10-19T14:22:02.409952+0000","last_fullsized":"2023-10-19T14:22:02.409952+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:48:08.338501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.2","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.403340+0000","last_change":"2023-10-19T14:22:02.403340+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.403340+0000","last_undegraded":"2023-10-19T14:22:02.403340+0000","last_fullsized":"2023-10-19T14:22:02.403340+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:42:16.203127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.5","version":"0'0","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.401015+0000","last_change":"2023-10-19T14:22:01.417906+0000","last_active":"2023-10-19T14:22:02.401015+0000","last_peered":"2023-10-19T14:22:02.401015+0000","last_clean":"2023-10-19T14:22:02.401015+0000","last_became_active":"2023-10-19T14:22:01.417614+0000","last_became_peered":"2023-10-19T14:22:01.417614+0000","last_unstale":"2023-10-19T14:22:02.401015+0000","last_undegraded":"2023-10-19T14:22:02.401015+0000","last_fullsized":"2023-10-19T14:22:02.401015+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:26:05.681833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.3","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.417066+0000","last_change":"2023-10-19T14:22:02.417066+0000","last_active":"2023-10-19T14:22:02.417066+0000","last_peered":"2023-10-19T14:22:02.417066+0000","last_clean":"2023-10-19T14:22:02.417066+0000","last_became_active":"2023-10-19T14:22:02.416770+0000","last_became_peered":"2023-10-19T14:22:02.416770+0000","last_unstale":"2023-10-19T14:22:02.417066+0000","last_undegraded":"2023-10-19T14:22:02.417066+0000","last_fullsized":"2023-10-19T14:22:02.417066+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:25:15.074214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.4","version":"0'0","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.409766+0000","last_change":"2023-10-19T14:22:01.412056+0000","last_active":"2023-10-19T14:22:02.409766+0000","last_peered":"2023-10-19T14:22:02.409766+0000","last_clean":"2023-10-19T14:22:02.409766+0000","last_became_active":"2023-10-19T14:22:01.411786+0000","last_became_peered":"2023-10-19T14:22:01.411786+0000","last_unstale":"2023-10-19T14:22:02.409766+0000","last_undegraded":"2023-10-19T14:22:02.409766+0000","last_fullsized":"2023-10-19T14:22:02.409766+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:40:51.932206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.4","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.403966+0000","last_change":"2023-10-19T14:22:02.403966+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.403966+0000","last_undegraded":"2023-10-19T14:22:02.403966+0000","last_fullsized":"2023-10-19T14:22:02.403966+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:43:57.998020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.3","version":"0'0","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.399541+0000","last_change":"2023-10-19T14:22:01.413689+0000","last_active":"2023-10-19T14:22:02.399541+0000","last_peered":"2023-10-19T14:22:02.399541+0000","last_clean":"2023-10-19T14:22:02.399541+0000","last_became_active":"2023-10-19T14:22:01.413419+0000","last_became_peered":"2023-10-19T14:22:01.413419+0000","last_unstale":"2023-10-19T14:22:02.399541+0000","last_undegraded":"2023-10-19T14:22:02.399541+0000","last_fullsized":"2023-10-19T14:22:02.399541+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:37:57.754265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.5","version":"0'0","reported_seq":9,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.414555+0000","last_change":"2023-10-19T14:22:02.409126+0000","last_active":"2023-10-19T14:22:02.414555+0000","last_peered":"2023-10-19T14:22:02.414555+0000","last_clean":"2023-10-19T14:22:02.414555+0000","last_became_active":"2023-10-19T14:22:02.408855+0000","last_became_peered":"2023-10-19T14:22:02.408855+0000","last_unstale":"2023-10-19T14:22:02.414555+0000","last_undegraded":"2023-10-19T14:22:02.414555+0000","last_fullsized":"2023-10-19T14:22:02.414555+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:03:36.116245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.2","version":"0'0","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.404648+0000","last_change":"2023-10-19T14:22:01.411905+0000","last_active":"2023-10-19T14:22:02.404648+0000","last_peered":"2023-10-19T14:22:02.404648+0000","last_clean":"2023-10-19T14:22:02.404648+0000","last_became_active":"2023-10-19T14:22:01.411584+0000","last_became_peered":"2023-10-19T14:22:01.411584+0000","last_unstale":"2023-10-19T14:22:02.404648+0000","last_undegraded":"2023-10-19T14:22:02.404648+0000","last_fullsized":"2023-10-19T14:22:02.404648+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:40:13.932927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.6","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.409726+0000","last_change":"2023-10-19T14:22:02.409726+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.409726+0000","last_undegraded":"2023-10-19T14:22:02.409726+0000","last_fullsized":"2023-10-19T14:22:02.409726+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:51:43.993546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.1","version":"0'0","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.403849+0000","last_change":"2023-10-19T14:22:01.412806+0000","last_active":"2023-10-19T14:22:02.403849+0000","last_peered":"2023-10-19T14:22:02.403849+0000","last_clean":"2023-10-19T14:22:02.403849+0000","last_became_active":"2023-10-19T14:22:01.412524+0000","last_became_peered":"2023-10-19T14:22:01.412524+0000","last_unstale":"2023-10-19T14:22:02.403849+0000","last_undegraded":"2023-10-19T14:22:02.403849+0000","last_fullsized":"2023-10-19T14:22:02.403849+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:43:54.078045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.7","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.414678+0000","last_change":"2023-10-19T14:22:02.414678+0000","last_active":"2023-10-19T14:22:02.414678+0000","last_peered":"2023-10-19T14:22:02.414678+0000","last_clean":"2023-10-19T14:22:02.414678+0000","last_became_active":"2023-10-19T14:22:02.414348+0000","last_became_peered":"2023-10-19T14:22:02.414348+0000","last_unstale":"2023-10-19T14:22:02.414678+0000","last_undegraded":"2023-10-19T14:22:02.414678+0000","last_fullsized":"2023-10-19T14:22:02.414678+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:18:02.907457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.0","version":"0'0","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.399367+0000","last_change":"2023-10-19T14:22:01.414455+0000","last_active":"2023-10-19T14:22:02.399367+0000","last_peered":"2023-10-19T14:22:02.399367+0000","last_clean":"2023-10-19T14:22:02.399367+0000","last_became_active":"2023-10-19T14:22:01.414081+0000","last_became_peered":"2023-10-19T14:22:01.414081+0000","last_unstale":"2023-10-19T14:22:02.399367+0000","last_undegraded":"2023-10-19T14:22:02.399367+0000","last_fullsized":"2023-10-19T14:22:02.399367+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:54:25.277401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.25","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.418670+0000","last_change":"2023-10-19T14:22:02.418670+0000","last_active":"2023-10-19T14:22:02.418670+0000","last_peered":"2023-10-19T14:22:02.418670+0000","last_clean":"2023-10-19T14:22:02.418670+0000","last_became_active":"2023-10-19T14:22:02.417862+0000","last_became_peered":"2023-10-19T14:22:02.417862+0000","last_unstale":"2023-10-19T14:22:02.418670+0000","last_undegraded":"2023-10-19T14:22:02.418670+0000","last_fullsized":"2023-10-19T14:22:02.418670+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:20:15.811864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"249'3224","reported_seq":4253,"reported_epoch":261,"state":"active+clean","last_fresh":"2023-10-19T14:22:01.406839+0000","last_change":"2023-10-19T12:44:08.636613+0000","last_active":"2023-10-19T14:22:01.406839+0000","last_peered":"2023-10-19T14:22:01.406839+0000","last_clean":"2023-10-19T14:22:01.406839+0000","last_became_active":"2023-10-19T12:44:08.635697+0000","last_became_peered":"2023-10-19T12:44:08.635697+0000","last_unstale":"2023-10-19T14:22:01.406839+0000","last_undegraded":"2023-10-19T14:22:01.406839+0000","last_fullsized":"2023-10-19T14:22:01.406839+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_clean_scrub_stamp":"2023-10-19T12:44:07.399738+0000","objects_scrubbed":0,"log_size":3224,"log_dups_size":0,"ondisk_log_size":3224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:15:23.685485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2539,"num_read_kb":5339,"num_write":657,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":36,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":35,"num_pg":32,"stat_sum":{"num_bytes":1470,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":33,"ondisk_log_size":33,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2539,"num_read_kb":5339,"num_write":657,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3224,"ondisk_log_size":3224,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576020,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66928,"kb_used_data":3804,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94304912,"statfs":{"total":96636764160,"available":96568229888,"internally_reserved":0,"allocated":3895296,"data_stored":2294561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":64616907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:21:10 2023","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.55100000000000005,"15min":0.57999999999999996},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.32200000000000001},"max":{"1min":0.754,"5min":0.80900000000000005,"15min":0.86199999999999999},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.503,"5min":0.57599999999999996,"15min":0.59899999999999998},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.312},"max":{"1min":0.66500000000000004,"5min":0.81899999999999995,"15min":0.86599999999999999},"last":0.39400000000000002}]},{"osd":1,"last update":"Thu Oct 19 14:21:10 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.58199999999999996,"15min":0.59499999999999997},"min":{"1min":0.45200000000000001,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.72199999999999998,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.56899999999999995,"15min":0.59299999999999997},"min":{"1min":0.39300000000000002,"5min":0.373,"15min":0.36699999999999999},"max":{"1min":0.78500000000000003,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.56200000000000006}]},{"osd":2,"last update":"Thu Oct 19 14:21:10 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.622,"15min":0.64200000000000002},"min":{"1min":0.47299999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.70499999999999996,"5min":0.84799999999999998,"15min":0.85499999999999998},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.63900000000000001,"15min":0.65700000000000003},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.54600000000000004}]},{"osd":3,"last update":"Thu Oct 19 14:21:10 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.60799999999999998,"15min":0.63100000000000001},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.628,"5min":0.67100000000000004,"15min":0.68300000000000005},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.83299999999999996,"5min":0.876,"15min":0.89600000000000002},"last":0.66700000000000004}]},{"osd":4,"last update":"Thu Oct 19 14:21:10 2023","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.50600000000000001,"15min":0.495},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.53000000000000003,"15min":0.51100000000000001},"min":{"1min":0.34999999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.505}]},{"osd":5,"last update":"Thu Oct 19 14:21:10 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52300000000000002,"15min":0.51100000000000001},"min":{"1min":0.33700000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.57799999999999996,"15min":0.55400000000000005},"min":{"1min":0.40699999999999997,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.054,"5min":1.054,"15min":1.054},"last":0.626}]},{"osd":6,"last update":"Thu Oct 19 14:21:10 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.58999999999999997,"15min":0.57599999999999996},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.35999999999999999},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.59299999999999997,"15min":0.57299999999999995},"min":{"1min":0.42999999999999999,"5min":0.35599999999999998,"15min":0.33500000000000002},"max":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.57399999999999995}]}]},{"osd":6,"up_from":13,"seq":55834576020,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56612,"kb_used_data":3224,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315228,"statfs":{"total":96636764160,"available":96578793472,"internally_reserved":0,"allocated":3301376,"data_stored":1704193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:21:02 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.59199999999999997,"15min":0.61099999999999999},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.78400000000000003,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.56299999999999994,"15min":0.60599999999999998},"min":{"1min":0.36699999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.71799999999999997,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.80200000000000005}]},{"osd":1,"last update":"Thu Oct 19 14:21:02 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.60099999999999998,"15min":0.61799999999999999},"min":{"1min":0.373,"5min":0.373,"15min":0.35799999999999998},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":1.165},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.60199999999999998,"15min":0.626},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.77400000000000002,"5min":0.83699999999999997,"15min":0.91300000000000003},"last":0.76600000000000001}]},{"osd":2,"last update":"Thu Oct 19 14:21:02 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.57599999999999996,"15min":0.60199999999999998},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.69099999999999995,"5min":0.80100000000000005,"15min":0.873},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.60599999999999998,"15min":0.624},"min":{"1min":0.40899999999999997,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.74199999999999999,"5min":0.83499999999999996,"15min":0.93100000000000005},"last":0.73599999999999999}]},{"osd":3,"last update":"Thu Oct 19 14:21:02 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.629,"15min":0.64900000000000002},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.66700000000000004,"5min":0.875,"15min":0.875},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.63800000000000001,"15min":0.65500000000000003},"min":{"1min":0.45400000000000001,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.92200000000000004,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.70399999999999996}]},{"osd":4,"last update":"Thu Oct 19 14:21:02 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.48899999999999999,"15min":0.48399999999999999},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.51000000000000001,"15min":0.50800000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.30399999999999999},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.85699999999999998},"last":0.61399999999999999}]},{"osd":5,"last update":"Thu Oct 19 14:21:02 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.50700000000000001,"15min":0.502},"min":{"1min":0.34799999999999998,"5min":0.32700000000000001,"15min":0.32500000000000001},"max":{"1min":0.79900000000000004,"5min":1.115,"15min":1.115},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.56599999999999995,"15min":0.54500000000000004},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.69199999999999995}]},{"osd":7,"last update":"Thu Oct 19 14:21:02 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.56000000000000005,"15min":0.54700000000000004},"min":{"1min":0.36799999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.90100000000000002},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.56699999999999995,"15min":0.55400000000000005},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.98399999999999999,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.67800000000000005}]}]},{"osd":5,"up_from":13,"seq":55834576020,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66348,"kb_used_data":3224,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305492,"statfs":{"total":96636764160,"available":96568823808,"internally_reserved":0,"allocated":3301376,"data_stored":1704193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:21:50 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.57399999999999995,"15min":0.60699999999999998},"min":{"1min":0.38,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.69399999999999995,"5min":0.79700000000000004,"15min":0.86399999999999999},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.59599999999999997,"15min":0.63300000000000001},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.70499999999999996,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.51100000000000001}]},{"osd":1,"last update":"Thu Oct 19 14:21:50 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.59099999999999997,"15min":0.622},"min":{"1min":0.34599999999999997,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.877},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.61199999999999999,"15min":0.63700000000000001},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.877,"5min":0.877,"15min":0.88500000000000001},"last":0.46999999999999997}]},{"osd":2,"last update":"Thu Oct 19 14:21:50 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.58199999999999996,"15min":0.61399999999999999},"min":{"1min":0.378,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.625,"15min":0.65100000000000002},"min":{"1min":0.441,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.78900000000000003,"5min":0.83299999999999996,"15min":0.90800000000000003},"last":0.65600000000000003}]},{"osd":3,"last update":"Thu Oct 19 14:21:50 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.60499999999999998,"15min":0.63600000000000001},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.81100000000000005,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.626,"15min":0.65900000000000003},"min":{"1min":0.40000000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.90900000000000003},"last":0.70099999999999996}]},{"osd":4,"last update":"Thu Oct 19 14:21:50 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.48799999999999999,"15min":0.48699999999999999},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.51800000000000002,"15min":0.505},"min":{"1min":0.309,"5min":0.308,"15min":0.28599999999999998},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.82099999999999995},"last":0.63900000000000001}]},{"osd":6,"last update":"Thu Oct 19 14:21:50 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.51700000000000002,"15min":0.502},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.88800000000000001},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.54100000000000004,"15min":0.53900000000000003},"min":{"1min":0.36899999999999999,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.67700000000000005}]},{"osd":7,"last update":"Thu Oct 19 14:21:50 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.52200000000000002,"15min":0.51800000000000002},"min":{"1min":0.32700000000000001,"5min":0.312,"15min":0.312},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.499},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.57899999999999996,"15min":0.57199999999999995},"min":{"1min":0.38,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.72299999999999998}]}]},{"osd":4,"up_from":13,"seq":55834576020,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48292,"kb_used_data":3224,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323548,"statfs":{"total":96636764160,"available":96587313152,"internally_reserved":0,"allocated":3301376,"data_stored":1704193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:21:16 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.54700000000000004,"15min":0.57099999999999995},"min":{"1min":0.30299999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.879},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.56899999999999995,"15min":0.59299999999999997},"min":{"1min":0.35299999999999998,"5min":0.31,"15min":0.31},"max":{"1min":0.72699999999999998,"5min":0.82599999999999996,"15min":0.873},"last":0.54700000000000004}]},{"osd":1,"last update":"Thu Oct 19 14:21:16 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.59699999999999998,"15min":0.63500000000000001},"min":{"1min":0.40300000000000002,"5min":0.372,"15min":0.32600000000000001},"max":{"1min":0.74099999999999999,"5min":0.80400000000000005,"15min":0.92700000000000005},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.60899999999999999,"15min":0.63900000000000001},"min":{"1min":0.40500000000000003,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.755,"5min":0.90000000000000002,"15min":1.0940000000000001},"last":0.53100000000000003}]},{"osd":2,"last update":"Thu Oct 19 14:21:16 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.626,"15min":0.64700000000000002},"min":{"1min":0.38900000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.80300000000000005,"5min":1.008,"15min":1.008},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.64200000000000002,"15min":0.66600000000000004},"min":{"1min":0.47199999999999998,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.81799999999999995,"5min":0.98799999999999999,"15min":1.008},"last":0.68799999999999994}]},{"osd":3,"last update":"Thu Oct 19 14:21:16 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.59499999999999997,"15min":0.627},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.68600000000000005,"5min":0.83399999999999996,"15min":0.91900000000000004},"last":0.755},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.65600000000000003,"15min":0.66600000000000004},"min":{"1min":0.48199999999999998,"5min":0.45400000000000001,"15min":0.437},"max":{"1min":0.85999999999999999,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.76500000000000001}]},{"osd":5,"last update":"Thu Oct 19 14:21:16 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.497,"15min":0.47799999999999998},"min":{"1min":0.33800000000000002,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.79700000000000004,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.50800000000000001,"15min":0.498},"min":{"1min":0.36499999999999999,"5min":0.33700000000000002,"15min":0.29199999999999998},"max":{"1min":0.81399999999999995,"5min":0.873,"15min":0.873},"last":0.502}]},{"osd":6,"last update":"Thu Oct 19 14:21:16 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.50700000000000001,"15min":0.49399999999999999},"min":{"1min":0.38500000000000001,"5min":0.34799999999999998,"15min":0.28199999999999997},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.54900000000000004,"15min":0.54300000000000004},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.88,"5min":0.88,"15min":1.2969999999999999},"last":0.47999999999999998}]},{"osd":7,"last update":"Thu Oct 19 14:21:16 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55100000000000005,"15min":0.54500000000000004},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.32600000000000001},"max":{"1min":0.78200000000000003,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.438},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.61199999999999999,"15min":0.59899999999999998},"min":{"1min":0.371,"5min":0.32100000000000001,"15min":0.318},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":1.032},"last":0.45600000000000002}]}]},{"osd":3,"up_from":13,"seq":55834576019,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56616,"kb_used_data":3224,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315224,"statfs":{"total":96636764160,"available":96578789376,"internally_reserved":0,"allocated":3301376,"data_stored":1704193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54655432},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:21:53 2023","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.39900000000000002,"15min":0.41699999999999998},"min":{"1min":0.23899999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.54500000000000004,"5min":0.60599999999999998,"15min":0.70699999999999996},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.39400000000000002,"15min":0.42199999999999999},"min":{"1min":0.23899999999999999,"5min":0.217,"15min":0.217},"max":{"1min":0.51900000000000002,"5min":0.61899999999999999,"15min":0.752},"last":0.36699999999999999}]},{"osd":1,"last update":"Thu Oct 19 14:21:53 2023","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.40699999999999997,"15min":0.42799999999999999},"min":{"1min":0.28100000000000003,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.71599999999999997,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.45500000000000002,"15min":0.47199999999999998},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26400000000000001},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.38800000000000001}]},{"osd":2,"last update":"Thu Oct 19 14:21:53 2023","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.46200000000000002,"15min":0.48099999999999998},"min":{"1min":0.34300000000000003,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.73099999999999998},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48499999999999999,"15min":0.504},"min":{"1min":0.30599999999999999,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":1.212,"5min":1.212,"15min":1.212},"last":0.35499999999999998}]},{"osd":4,"last update":"Thu Oct 19 14:21:53 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.67400000000000004,"15min":0.70099999999999996},"min":{"1min":0.44600000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.94199999999999995,"5min":1.018,"15min":1.157},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.629,"5min":0.68700000000000006,"15min":0.72299999999999998},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.89500000000000002,"5min":0.996,"15min":1.173},"last":0.752}]},{"osd":5,"last update":"Thu Oct 19 14:21:53 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.67500000000000004,"15min":0.68600000000000005},"min":{"1min":0.50900000000000001,"5min":0.46600000000000003,"15min":0.41999999999999998},"max":{"1min":0.96799999999999997,"5min":0.97299999999999998,"15min":1.0149999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.66800000000000004,"15min":0.68600000000000005},"min":{"1min":0.48899999999999999,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.97699999999999998,"5min":0.99399999999999999,"15min":1.105},"last":0.80600000000000005}]},{"osd":6,"last update":"Thu Oct 19 14:21:53 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.65000000000000002,"15min":0.68600000000000005},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.93000000000000005,"5min":1.0009999999999999,"15min":1.0669999999999999},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.68899999999999995,"15min":0.70999999999999996},"min":{"1min":0.505,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.079},"last":0.79100000000000004}]},{"osd":7,"last update":"Thu Oct 19 14:21:53 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67500000000000004,"15min":0.69799999999999995},"min":{"1min":0.46600000000000003,"5min":0.434,"15min":0.434},"max":{"1min":0.94999999999999996,"5min":0.96899999999999997,"15min":1.085},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.69799999999999995,"15min":0.72799999999999998},"min":{"1min":0.48799999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.95899999999999996,"5min":0.97199999999999998,"15min":1.1739999999999999},"last":0.60299999999999998}]}]},{"osd":2,"up_from":13,"seq":55834576021,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48292,"kb_used_data":3224,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323548,"statfs":{"total":96636764160,"available":96587313152,"internally_reserved":0,"allocated":3301376,"data_stored":1704193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.374,"15min":0.39600000000000002},"min":{"1min":0.25700000000000001,"5min":0.218,"15min":0.218},"max":{"1min":0.58799999999999997,"5min":0.66600000000000004,"15min":0.69699999999999995},"last":0.27800000000000002},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.41499999999999998,"15min":0.432},"min":{"1min":0.248,"5min":0.248,"15min":0.223},"max":{"1min":0.53100000000000003,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.379}]},{"osd":1,"last update":"Thu Oct 19 14:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.38300000000000001,"15min":0.40600000000000003},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.65400000000000003},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.44400000000000001,"15min":0.46000000000000002},"min":{"1min":0.28699999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.45100000000000001,"5min":1.742,"15min":1.742},"last":0.40100000000000002}]},{"osd":3,"last update":"Thu Oct 19 14:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.45500000000000002,"15min":0.47599999999999998},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.63300000000000001,"5min":0.69299999999999995,"15min":0.74399999999999999},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.47299999999999998,"15min":0.501},"min":{"1min":0.32700000000000001,"5min":0.32000000000000001,"15min":0.315},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.83599999999999997},"last":0.5}]},{"osd":4,"last update":"Thu Oct 19 14:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.63900000000000001,"15min":0.66200000000000003},"min":{"1min":0.40699999999999997,"5min":0.39600000000000002,"15min":0.39500000000000002},"max":{"1min":0.74399999999999999,"5min":0.89200000000000002,"15min":0.998},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.65400000000000003,"15min":0.67600000000000005},"min":{"1min":0.41899999999999998,"5min":0.38900000000000001,"15min":0.35199999999999998},"max":{"1min":0.79000000000000004,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.60599999999999998}]},{"osd":5,"last update":"Thu Oct 19 14:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.67800000000000005,"15min":0.69299999999999995},"min":{"1min":0.443,"5min":0.443,"15min":0.40600000000000003},"max":{"1min":0.79300000000000004,"5min":1.4330000000000001,"15min":1.4330000000000001},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.66900000000000004,"15min":0.68000000000000005},"min":{"1min":0.46899999999999997,"5min":0.45100000000000001,"15min":0.39300000000000002},"max":{"1min":0.77400000000000002,"5min":1.595,"15min":1.595},"last":0.80900000000000005}]},{"osd":6,"last update":"Thu Oct 19 14:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.67700000000000005,"15min":0.70199999999999996},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.80900000000000005,"5min":0.996,"15min":1.0189999999999999},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.629,"5min":0.69499999999999995,"15min":0.71399999999999997},"min":{"1min":0.438,"5min":0.433,"15min":0.40600000000000003},"max":{"1min":0.83399999999999996,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":0.73099999999999998}]},{"osd":7,"last update":"Thu Oct 19 14:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.68400000000000005,"15min":0.71299999999999997},"min":{"1min":0.47199999999999998,"5min":0.44800000000000001,"15min":0.41799999999999998},"max":{"1min":0.78800000000000003,"5min":1.0129999999999999,"15min":1.0289999999999999},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.71199999999999997,"15min":0.72599999999999998},"min":{"1min":0.42199999999999999,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.86699999999999999,"5min":1.53,"15min":1.53},"last":0.64200000000000002}]}]},{"osd":1,"up_from":13,"seq":55834576020,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48296,"kb_used_data":3224,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323544,"statfs":{"total":96636764160,"available":96587309056,"internally_reserved":0,"allocated":3301376,"data_stored":1704193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.38400000000000001,"15min":0.40300000000000002},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.46899999999999997,"5min":0.61599999999999999,"15min":0.67400000000000004},"last":0.22500000000000001},{"interface":"front","average":{"1min":0.378,"5min":0.41999999999999998,"15min":0.45000000000000001},"min":{"1min":0.27700000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.51500000000000001,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.28100000000000003}]},{"osd":2,"last update":"Thu Oct 19 14:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.42399999999999999,"15min":0.438},"min":{"1min":0.26100000000000001,"5min":0.251,"15min":0.251},"max":{"1min":0.505,"5min":0.63100000000000001,"15min":0.77500000000000002},"last":0.318},{"interface":"front","average":{"1min":0.434,"5min":0.47699999999999998,"15min":0.48799999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.56699999999999995,"5min":0.68700000000000006,"15min":0.82799999999999996},"last":0.44400000000000001}]},{"osd":3,"last update":"Thu Oct 19 14:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.495,"15min":0.501},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.29499999999999998},"max":{"1min":0.59399999999999997,"5min":0.76000000000000001,"15min":0.78900000000000003},"last":0.376},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.42899999999999999,"15min":0.44600000000000001},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.26100000000000001},"max":{"1min":0.59899999999999998,"5min":0.65200000000000002,"15min":0.69499999999999995},"last":0.245}]},{"osd":4,"last update":"Thu Oct 19 14:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.64200000000000002,"15min":0.66700000000000004},"min":{"1min":0.42299999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.80400000000000005,"5min":0.95999999999999996,"15min":1.042},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.65400000000000003,"15min":0.66600000000000004},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.32300000000000001},"max":{"1min":0.80500000000000005,"5min":0.90800000000000003,"15min":1.1659999999999999},"last":0.39500000000000002}]},{"osd":5,"last update":"Thu Oct 19 14:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.66700000000000004,"15min":0.68200000000000005},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.81200000000000006,"5min":0.96199999999999997,"15min":1.3500000000000001},"last":0.434},{"interface":"front","average":{"1min":0.62,"5min":0.68899999999999995,"15min":0.70299999999999996},"min":{"1min":0.442,"5min":0.44,"15min":0.40500000000000003},"max":{"1min":0.82899999999999996,"5min":1.0009999999999999,"15min":1.121},"last":0.49399999999999999}]},{"osd":6,"last update":"Thu Oct 19 14:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.65400000000000003,"15min":0.67600000000000005},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.84999999999999998,"5min":1.0129999999999999,"15min":1.2310000000000001},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.69699999999999995,"15min":0.71499999999999997},"min":{"1min":0.46700000000000003,"5min":0.434,"15min":0.434},"max":{"1min":0.85799999999999998,"5min":1.0009999999999999,"15min":1.274},"last":0.47699999999999998}]},{"osd":7,"last update":"Thu Oct 19 14:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.70499999999999996,"15min":0.72199999999999998},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.86899999999999999,"5min":0.99099999999999999,"15min":1.075},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.68200000000000005,"15min":0.69599999999999995},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.84599999999999997,"5min":1.0169999999999999,"15min":1.117},"last":0.46200000000000002}]}]},{"osd":0,"up_from":13,"seq":55834576021,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61348,"kb_used_data":3800,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310492,"statfs":{"total":96636764160,"available":96573943808,"internally_reserved":0,"allocated":3891200,"data_stored":2293942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Oct 19 14:21:13 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.40100000000000002,"15min":0.41099999999999998},"min":{"1min":0.26300000000000001,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.47799999999999998,"5min":0.61499999999999999,"15min":0.624},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.41699999999999998,"15min":0.45500000000000002},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.49199999999999999,"5min":0.70899999999999996,"15min":0.81100000000000005},"last":0.64300000000000002}]},{"osd":2,"last update":"Thu Oct 19 14:21:13 2023","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.34100000000000003,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.54400000000000004,"5min":0.754,"15min":0.754},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.377,"5min":0.42399999999999999,"15min":0.44400000000000001},"min":{"1min":0.28100000000000003,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.46899999999999997,"5min":0.60199999999999998,"15min":0.69099999999999995},"last":0.52400000000000002}]},{"osd":3,"last update":"Thu Oct 19 14:21:13 2023","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.436},"min":{"1min":0.27200000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.50800000000000001,"5min":0.58799999999999997,"15min":0.66800000000000004},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.48199999999999998,"15min":0.50700000000000001},"min":{"1min":0.29299999999999998,"5min":0.254,"15min":0.254},"max":{"1min":0.72399999999999998,"5min":0.78100000000000003,"15min":0.84599999999999997},"last":0.53400000000000003}]},{"osd":4,"last update":"Thu Oct 19 14:21:13 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.63,"15min":0.67000000000000004},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.80300000000000005,"5min":0.93100000000000005,"15min":1.1890000000000001},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.628,"15min":0.67400000000000004},"min":{"1min":0.45900000000000002,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.75600000000000001,"5min":0.93400000000000005,"15min":1.2150000000000001},"last":0.78300000000000003}]},{"osd":5,"last update":"Thu Oct 19 14:21:13 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.67200000000000004,"15min":0.71399999999999997},"min":{"1min":0.48599999999999999,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.83099999999999996,"5min":0.98499999999999999,"15min":1.1579999999999999},"last":0.748},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.67500000000000004,"15min":0.71399999999999997},"min":{"1min":0.499,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.80000000000000004,"5min":0.98299999999999998,"15min":1.2310000000000001},"last":0.73899999999999999}]},{"osd":6,"last update":"Thu Oct 19 14:21:13 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.66300000000000003,"15min":0.69899999999999995},"min":{"1min":0.46100000000000002,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.85499999999999998,"5min":0.97799999999999998,"15min":1.1739999999999999},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.65300000000000002,"15min":0.69599999999999995},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.70299999999999996,"5min":0.97799999999999998,"15min":1.0189999999999999},"last":0.76100000000000001}]},{"osd":7,"last update":"Thu Oct 19 14:21:13 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.66600000000000004,"15min":0.70999999999999996},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.79900000000000004,"5min":0.93400000000000005,"15min":1.1180000000000001},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.66100000000000003,"15min":0.69599999999999995},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.84499999999999997,"5min":0.93799999999999994,"15min":1.0349999999999999},"last":0.77300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-19T14:22:05.411 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-19T14:22:05.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.551+0000 7fca60a90640 1 -- 172.21.15.57:0/1085626956 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fca5c154af0 msgr2=0x7fca5c154f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:05.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.551+0000 7fca60a90640 1 --2- 172.21.15.57:0/1085626956 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fca5c154af0 0x7fca5c154f00 secure :-1 s=READY pgs=2887 cs=0 l=1 rev1=1 crypto rx=0x7fca5c0abdc0 tx=0x7fca5402c2f0 comp rx=0 tx=0).stop 2023-10-19T14:22:05.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.551+0000 7fca60a90640 1 -- 172.21.15.57:0/1085626956 shutdown_connections 2023-10-19T14:22:05.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.551+0000 7fca60a90640 1 --2- 172.21.15.57:0/1085626956 >> v2:172.21.15.179:3300/0 conn(0x7fca5c0c96b0 0x7fca5c0c9b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:05.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.551+0000 7fca60a90640 1 --2- 172.21.15.57:0/1085626956 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fca5c154af0 0x7fca5c154f00 unknown :-1 s=CLOSED pgs=2887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:05.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.551+0000 7fca60a90640 1 -- 172.21.15.57:0/1085626956 >> 172.21.15.57:0/1085626956 conn(0x7fca5c069930 msgr2=0x7fca5c069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:05.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.551+0000 7fca60a90640 1 -- 172.21.15.57:0/1085626956 shutdown_connections 2023-10-19T14:22:05.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.551+0000 7fca60a90640 1 -- 172.21.15.57:0/1085626956 wait complete. 2023-10-19T14:22:05.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.553+0000 7fca60a90640 1 Processor -- start 2023-10-19T14:22:05.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.553+0000 7fca60a90640 1 -- start start 2023-10-19T14:22:05.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.553+0000 7fca60a90640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fca5c0b1920 0x7fca5c150b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:05.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.553+0000 7fca60a90640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fca5c0c96b0 0x7fca5c1f1220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:05.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.553+0000 7fca60a90640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fca5c154af0 0x7fca5c1f1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:05.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.553+0000 7fca60a90640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca5c1e48d0 con 0x7fca5c0b1920 2023-10-19T14:22:05.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.553+0000 7fca60a90640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca5c151370 con 0x7fca5c154af0 2023-10-19T14:22:05.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.553+0000 7fca60a90640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca5c1510d0 con 0x7fca5c0c96b0 2023-10-19T14:22:05.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.553+0000 7fca5a59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fca5c0b1920 0x7fca5c150b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:05.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.554+0000 7fca5a59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fca5c0b1920 0x7fca5c150b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51454/0 (socket says 172.21.15.57:51454) 2023-10-19T14:22:05.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.554+0000 7fca5a59c640 1 -- 172.21.15.57:0/3757154217 learned_addr learned my addr 172.21.15.57:0/3757154217 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:22:05.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.554+0000 7fca5a59c640 1 -- 172.21.15.57:0/3757154217 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fca5c0c96b0 msgr2=0x7fca5c1f1220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:05.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.554+0000 7fca5ad9d640 1 --2- 172.21.15.57:0/3757154217 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fca5c154af0 0x7fca5c1f1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:05.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.554+0000 7fca5a59c640 1 --2- 172.21.15.57:0/3757154217 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fca5c0c96b0 0x7fca5c1f1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:05.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.554+0000 7fca5a59c640 1 -- 172.21.15.57:0/3757154217 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fca5c154af0 msgr2=0x7fca5c1f1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:05.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.554+0000 7fca5a59c640 1 --2- 172.21.15.57:0/3757154217 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fca5c154af0 0x7fca5c1f1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:05.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.554+0000 7fca5a59c640 1 -- 172.21.15.57:0/3757154217 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca54009900 con 0x7fca5c0b1920 2023-10-19T14:22:05.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.554+0000 7fca5ad9d640 1 --2- 172.21.15.57:0/3757154217 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fca5c154af0 0x7fca5c1f1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:22:05.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.554+0000 7fca5a59c640 1 --2- 172.21.15.57:0/3757154217 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fca5c0b1920 0x7fca5c150b90 secure :-1 s=READY pgs=2888 cs=0 l=1 rev1=1 crypto rx=0x7fca4c00c400 tx=0x7fca4c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:05.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.554+0000 7fca52ffd640 1 -- 172.21.15.57:0/3757154217 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca4c0052d0 con 0x7fca5c0b1920 2023-10-19T14:22:05.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.554+0000 7fca52ffd640 1 -- 172.21.15.57:0/3757154217 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fca4c007020 con 0x7fca5c0b1920 2023-10-19T14:22:05.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.554+0000 7fca52ffd640 1 -- 172.21.15.57:0/3757154217 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca4c007470 con 0x7fca5c0b1920 2023-10-19T14:22:05.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.554+0000 7fca60a90640 1 -- 172.21.15.57:0/3757154217 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca5c070300 con 0x7fca5c0b1920 2023-10-19T14:22:05.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.554+0000 7fca60a90640 1 -- 172.21.15.57:0/3757154217 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca5c06b990 con 0x7fca5c0b1920 2023-10-19T14:22:05.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.555+0000 7fca60a90640 1 -- 172.21.15.57:0/3757154217 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca5c0690e0 con 0x7fca5c0b1920 2023-10-19T14:22:05.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.556+0000 7fca52ffd640 1 -- 172.21.15.57:0/3757154217 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 145) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fca4c0067d0 con 0x7fca5c0b1920 2023-10-19T14:22:05.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.556+0000 7fca52ffd640 1 --2- 172.21.15.57:0/3757154217 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fca3c06b080 0x7fca3c06d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:05.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.556+0000 7fca52ffd640 1 -- 172.21.15.57:0/3757154217 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(263..263 src has 1..263) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fca4c089180 con 0x7fca5c0b1920 2023-10-19T14:22:05.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.556+0000 7fca53fff640 1 --2- 172.21.15.57:0/3757154217 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fca3c06b080 0x7fca3c06d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:05.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.557+0000 7fca53fff640 1 --2- 172.21.15.57:0/3757154217 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fca3c06b080 0x7fca3c06d540 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fca44005fd0 tx=0x7fca44005850 comp rx=0 tx=0).ready entity=mgr.11446 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:05.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.560+0000 7fca52ffd640 1 -- 172.21.15.57:0/3757154217 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fca4c053e40 con 0x7fca5c0b1920 2023-10-19T14:22:05.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.731+0000 7fca60a90640 1 -- 172.21.15.57:0/3757154217 --> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fca5c070180 con 0x7fca3c06b080 2023-10-19T14:22:05.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.739+0000 7fca52ffd640 1 -- 172.21.15.57:0/3757154217 <== mgr.11446 v2:172.21.15.179:6832/1573806521 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+288853 (secure 0 0 0) 0x7fca4400b6f0 con 0x7fca3c06b080 2023-10-19T14:22:05.748 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:22:05.749 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2023-10-19T14:22:05.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.743+0000 7fca60a90640 1 -- 172.21.15.57:0/3757154217 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fca3c06b080 msgr2=0x7fca3c06d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:05.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.743+0000 7fca60a90640 1 --2- 172.21.15.57:0/3757154217 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fca3c06b080 0x7fca3c06d540 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fca44005fd0 tx=0x7fca44005850 comp rx=0 tx=0).stop 2023-10-19T14:22:05.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.743+0000 7fca60a90640 1 -- 172.21.15.57:0/3757154217 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fca5c0b1920 msgr2=0x7fca5c150b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:05.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.743+0000 7fca60a90640 1 --2- 172.21.15.57:0/3757154217 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fca5c0b1920 0x7fca5c150b90 secure :-1 s=READY pgs=2888 cs=0 l=1 rev1=1 crypto rx=0x7fca4c00c400 tx=0x7fca4c00c8c0 comp rx=0 tx=0).stop 2023-10-19T14:22:05.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.743+0000 7fca60a90640 1 -- 172.21.15.57:0/3757154217 shutdown_connections 2023-10-19T14:22:05.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.743+0000 7fca60a90640 1 --2- 172.21.15.57:0/3757154217 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fca3c06b080 0x7fca3c06d540 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:05.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.743+0000 7fca60a90640 1 --2- 172.21.15.57:0/3757154217 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fca5c154af0 0x7fca5c1f1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:05.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.743+0000 7fca60a90640 1 --2- 172.21.15.57:0/3757154217 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fca5c0c96b0 0x7fca5c1f1220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:05.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.743+0000 7fca60a90640 1 --2- 172.21.15.57:0/3757154217 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fca5c0b1920 0x7fca5c150b90 unknown :-1 s=CLOSED pgs=2888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:05.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.743+0000 7fca60a90640 1 -- 172.21.15.57:0/3757154217 >> 172.21.15.57:0/3757154217 conn(0x7fca5c069930 msgr2=0x7fca5c0ae700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:05.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.743+0000 7fca60a90640 1 -- 172.21.15.57:0/3757154217 shutdown_connections 2023-10-19T14:22:05.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.743+0000 7fca60a90640 1 -- 172.21.15.57:0/3757154217 wait complete. 2023-10-19T14:22:05.756 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":29,"stamp":"2023-10-19T14:22:05.419632+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":591838,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":2539,"num_read_kb":5339,"num_write":697,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3257,"ondisk_log_size":3257,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":40,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":452732,"kb_used_data":26948,"kb_used_omap":12,"kb_used_meta":425651,"kb_avail":754521988,"statfs":{"total":773094113280,"available":772630515712,"internally_reserved":0,"allocated":27594752,"data_stored":14813661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":435867193},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1470,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.025206"},"pg_stats":[{"pgid":"36.3f","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.407757+0000","last_change":"2023-10-19T14:22:02.407757+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.407757+0000","last_undegraded":"2023-10-19T14:22:02.407757+0000","last_fullsized":"2023-10-19T14:22:02.407757+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:14:21.361763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":["6","3"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.3e","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.406175+0000","last_change":"2023-10-19T14:22:02.406175+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.406175+0000","last_undegraded":"2023-10-19T14:22:02.406175+0000","last_fullsized":"2023-10-19T14:22:02.406175+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:23:52.735800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.3d","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.412915+0000","last_change":"2023-10-19T14:22:02.412915+0000","last_active":"2023-10-19T14:22:02.412915+0000","last_peered":"2023-10-19T14:22:02.412915+0000","last_clean":"2023-10-19T14:22:02.412915+0000","last_became_active":"2023-10-19T14:22:02.412634+0000","last_became_peered":"2023-10-19T14:22:02.412634+0000","last_unstale":"2023-10-19T14:22:02.412915+0000","last_undegraded":"2023-10-19T14:22:02.412915+0000","last_fullsized":"2023-10-19T14:22:02.412915+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:20:32.428977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.3c","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.403761+0000","last_change":"2023-10-19T14:22:02.403761+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.403761+0000","last_undegraded":"2023-10-19T14:22:02.403761+0000","last_fullsized":"2023-10-19T14:22:02.403761+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:50:46.770884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.3b","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.409103+0000","last_change":"2023-10-19T14:22:02.409103+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.409103+0000","last_undegraded":"2023-10-19T14:22:02.409103+0000","last_fullsized":"2023-10-19T14:22:02.409103+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:03:49.877672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.3a","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.408280+0000","last_change":"2023-10-19T14:22:02.408280+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.408280+0000","last_undegraded":"2023-10-19T14:22:02.408280+0000","last_fullsized":"2023-10-19T14:22:02.408280+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:18:59.130442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.39","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.413984+0000","last_change":"2023-10-19T14:22:02.413984+0000","last_active":"2023-10-19T14:22:02.413984+0000","last_peered":"2023-10-19T14:22:02.413984+0000","last_clean":"2023-10-19T14:22:02.413984+0000","last_became_active":"2023-10-19T14:22:02.413714+0000","last_became_peered":"2023-10-19T14:22:02.413714+0000","last_unstale":"2023-10-19T14:22:02.413984+0000","last_undegraded":"2023-10-19T14:22:02.413984+0000","last_fullsized":"2023-10-19T14:22:02.413984+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:03:55.524277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.38","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.403768+0000","last_change":"2023-10-19T14:22:02.403768+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.403768+0000","last_undegraded":"2023-10-19T14:22:02.403768+0000","last_fullsized":"2023-10-19T14:22:02.403768+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:30:01.392073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.37","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.404303+0000","last_change":"2023-10-19T14:22:02.404303+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.404303+0000","last_undegraded":"2023-10-19T14:22:02.404303+0000","last_fullsized":"2023-10-19T14:22:02.404303+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:16:24.272266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.36","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.413432+0000","last_change":"2023-10-19T14:22:02.413432+0000","last_active":"2023-10-19T14:22:02.413432+0000","last_peered":"2023-10-19T14:22:02.413432+0000","last_clean":"2023-10-19T14:22:02.413432+0000","last_became_active":"2023-10-19T14:22:02.412808+0000","last_became_peered":"2023-10-19T14:22:02.412808+0000","last_unstale":"2023-10-19T14:22:02.413432+0000","last_undegraded":"2023-10-19T14:22:02.413432+0000","last_fullsized":"2023-10-19T14:22:02.413432+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T02:11:11.663612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.35","version":"0'0","reported_seq":9,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.418007+0000","last_change":"2023-10-19T14:22:02.413336+0000","last_active":"2023-10-19T14:22:02.418007+0000","last_peered":"2023-10-19T14:22:02.418007+0000","last_clean":"2023-10-19T14:22:02.418007+0000","last_became_active":"2023-10-19T14:22:02.412942+0000","last_became_peered":"2023-10-19T14:22:02.412942+0000","last_unstale":"2023-10-19T14:22:02.418007+0000","last_undegraded":"2023-10-19T14:22:02.418007+0000","last_fullsized":"2023-10-19T14:22:02.418007+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:37:40.908777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.34","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.406632+0000","last_change":"2023-10-19T14:22:02.406632+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.406632+0000","last_undegraded":"2023-10-19T14:22:02.406632+0000","last_fullsized":"2023-10-19T14:22:02.406632+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:30:39.654226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":["6","3"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.33","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.403708+0000","last_change":"2023-10-19T14:22:02.403708+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.403708+0000","last_undegraded":"2023-10-19T14:22:02.403708+0000","last_fullsized":"2023-10-19T14:22:02.403708+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:39:38.146246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.32","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.403725+0000","last_change":"2023-10-19T14:22:02.403725+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.403725+0000","last_undegraded":"2023-10-19T14:22:02.403725+0000","last_fullsized":"2023-10-19T14:22:02.403725+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:10:47.778168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.31","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.403277+0000","last_change":"2023-10-19T14:22:02.403277+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.403277+0000","last_undegraded":"2023-10-19T14:22:02.403277+0000","last_fullsized":"2023-10-19T14:22:02.403277+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:15:36.803736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.30","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.420492+0000","last_change":"2023-10-19T14:22:02.420492+0000","last_active":"2023-10-19T14:22:02.420492+0000","last_peered":"2023-10-19T14:22:02.420492+0000","last_clean":"2023-10-19T14:22:02.420492+0000","last_became_active":"2023-10-19T14:22:02.420210+0000","last_became_peered":"2023-10-19T14:22:02.420210+0000","last_unstale":"2023-10-19T14:22:02.420492+0000","last_undegraded":"2023-10-19T14:22:02.420492+0000","last_fullsized":"2023-10-19T14:22:02.420492+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:38:11.628588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.2f","version":"0'0","reported_seq":9,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.420525+0000","last_change":"2023-10-19T14:22:02.415526+0000","last_active":"2023-10-19T14:22:02.420525+0000","last_peered":"2023-10-19T14:22:02.420525+0000","last_clean":"2023-10-19T14:22:02.420525+0000","last_became_active":"2023-10-19T14:22:02.415243+0000","last_became_peered":"2023-10-19T14:22:02.415243+0000","last_unstale":"2023-10-19T14:22:02.420525+0000","last_undegraded":"2023-10-19T14:22:02.420525+0000","last_fullsized":"2023-10-19T14:22:02.420525+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:25:08.178761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.2e","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.405331+0000","last_change":"2023-10-19T14:22:02.405331+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.405331+0000","last_undegraded":"2023-10-19T14:22:02.405331+0000","last_fullsized":"2023-10-19T14:22:02.405331+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:49:42.930618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.2d","version":"0'0","reported_seq":9,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.420730+0000","last_change":"2023-10-19T14:22:02.418292+0000","last_active":"2023-10-19T14:22:02.420730+0000","last_peered":"2023-10-19T14:22:02.420730+0000","last_clean":"2023-10-19T14:22:02.420730+0000","last_became_active":"2023-10-19T14:22:02.415107+0000","last_became_peered":"2023-10-19T14:22:02.415107+0000","last_unstale":"2023-10-19T14:22:02.420730+0000","last_undegraded":"2023-10-19T14:22:02.420730+0000","last_fullsized":"2023-10-19T14:22:02.420730+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:22:21.202909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.2c","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.421040+0000","last_change":"2023-10-19T14:22:02.421040+0000","last_active":"2023-10-19T14:22:02.421040+0000","last_peered":"2023-10-19T14:22:02.421040+0000","last_clean":"2023-10-19T14:22:02.421040+0000","last_became_active":"2023-10-19T14:22:02.420785+0000","last_became_peered":"2023-10-19T14:22:02.420785+0000","last_unstale":"2023-10-19T14:22:02.421040+0000","last_undegraded":"2023-10-19T14:22:02.421040+0000","last_fullsized":"2023-10-19T14:22:02.421040+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:48:32.972469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.2b","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.418617+0000","last_change":"2023-10-19T14:22:02.418617+0000","last_active":"2023-10-19T14:22:02.418617+0000","last_peered":"2023-10-19T14:22:02.418617+0000","last_clean":"2023-10-19T14:22:02.418617+0000","last_became_active":"2023-10-19T14:22:02.418348+0000","last_became_peered":"2023-10-19T14:22:02.418348+0000","last_unstale":"2023-10-19T14:22:02.418617+0000","last_undegraded":"2023-10-19T14:22:02.418617+0000","last_fullsized":"2023-10-19T14:22:02.418617+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:03:48.649163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.2a","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.399641+0000","last_change":"2023-10-19T14:22:02.399641+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.399641+0000","last_undegraded":"2023-10-19T14:22:02.399641+0000","last_fullsized":"2023-10-19T14:22:02.399641+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:35:08.037582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.29","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.405143+0000","last_change":"2023-10-19T14:22:02.405143+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.405143+0000","last_undegraded":"2023-10-19T14:22:02.405143+0000","last_fullsized":"2023-10-19T14:22:02.405143+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:09:54.096828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.28","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.398717+0000","last_change":"2023-10-19T14:22:02.398717+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.398717+0000","last_undegraded":"2023-10-19T14:22:02.398717+0000","last_fullsized":"2023-10-19T14:22:02.398717+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:35:36.604007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.27","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.417381+0000","last_change":"2023-10-19T14:22:02.417381+0000","last_active":"2023-10-19T14:22:02.417381+0000","last_peered":"2023-10-19T14:22:02.417381+0000","last_clean":"2023-10-19T14:22:02.417381+0000","last_became_active":"2023-10-19T14:22:02.417103+0000","last_became_peered":"2023-10-19T14:22:02.417103+0000","last_unstale":"2023-10-19T14:22:02.417381+0000","last_undegraded":"2023-10-19T14:22:02.417381+0000","last_fullsized":"2023-10-19T14:22:02.417381+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:05:25.934207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.26","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.410522+0000","last_change":"2023-10-19T14:22:02.410522+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.410522+0000","last_undegraded":"2023-10-19T14:22:02.410522+0000","last_fullsized":"2023-10-19T14:22:02.410522+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:41:53.947719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.24","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.407514+0000","last_change":"2023-10-19T14:22:02.407514+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.407514+0000","last_undegraded":"2023-10-19T14:22:02.407514+0000","last_fullsized":"2023-10-19T14:22:02.407514+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:14:37.969686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.23","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.398963+0000","last_change":"2023-10-19T14:22:02.398963+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.398963+0000","last_undegraded":"2023-10-19T14:22:02.398963+0000","last_fullsized":"2023-10-19T14:22:02.398963+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:57:49.145921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.22","version":"0'0","reported_seq":9,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.404880+0000","last_change":"2023-10-19T14:22:02.402715+0000","last_active":"2023-10-19T14:22:02.404880+0000","last_peered":"2023-10-19T14:22:02.404880+0000","last_clean":"2023-10-19T14:22:02.404880+0000","last_became_active":"2023-10-19T14:22:02.402451+0000","last_became_peered":"2023-10-19T14:22:02.402451+0000","last_unstale":"2023-10-19T14:22:02.404880+0000","last_undegraded":"2023-10-19T14:22:02.404880+0000","last_fullsized":"2023-10-19T14:22:02.404880+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:42:44.848665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.21","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.399356+0000","last_change":"2023-10-19T14:22:02.399356+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.399356+0000","last_undegraded":"2023-10-19T14:22:02.399356+0000","last_fullsized":"2023-10-19T14:22:02.399356+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:45:44.957902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.20","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.403114+0000","last_change":"2023-10-19T14:22:02.403114+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.403114+0000","last_undegraded":"2023-10-19T14:22:02.403114+0000","last_fullsized":"2023-10-19T14:22:02.403114+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T02:18:23.251534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.18","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.398614+0000","last_change":"2023-10-19T14:22:02.398614+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.398614+0000","last_undegraded":"2023-10-19T14:22:02.398614+0000","last_fullsized":"2023-10-19T14:22:02.398614+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:31:51.736865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.1f","version":"262'4","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.404439+0000","last_change":"2023-10-19T14:22:01.414254+0000","last_active":"2023-10-19T14:22:02.404439+0000","last_peered":"2023-10-19T14:22:02.404439+0000","last_clean":"2023-10-19T14:22:02.404439+0000","last_became_active":"2023-10-19T14:22:01.413084+0000","last_became_peered":"2023-10-19T14:22:01.413084+0000","last_unstale":"2023-10-19T14:22:02.404439+0000","last_undegraded":"2023-10-19T14:22:02.404439+0000","last_fullsized":"2023-10-19T14:22:02.404439+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:04:17.277980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.19","version":"0'0","reported_seq":9,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.416536+0000","last_change":"2023-10-19T14:22:02.412925+0000","last_active":"2023-10-19T14:22:02.416536+0000","last_peered":"2023-10-19T14:22:02.416536+0000","last_clean":"2023-10-19T14:22:02.416536+0000","last_became_active":"2023-10-19T14:22:02.412639+0000","last_became_peered":"2023-10-19T14:22:02.412639+0000","last_unstale":"2023-10-19T14:22:02.416536+0000","last_undegraded":"2023-10-19T14:22:02.416536+0000","last_fullsized":"2023-10-19T14:22:02.416536+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:14:38.697589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.1e","version":"0'0","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.399308+0000","last_change":"2023-10-19T14:22:01.410546+0000","last_active":"2023-10-19T14:22:02.399308+0000","last_peered":"2023-10-19T14:22:02.399308+0000","last_clean":"2023-10-19T14:22:02.399308+0000","last_became_active":"2023-10-19T14:22:01.410263+0000","last_became_peered":"2023-10-19T14:22:01.410263+0000","last_unstale":"2023-10-19T14:22:02.399308+0000","last_undegraded":"2023-10-19T14:22:02.399308+0000","last_fullsized":"2023-10-19T14:22:02.399308+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:02:12.137179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.1a","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.406083+0000","last_change":"2023-10-19T14:22:02.406083+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.406083+0000","last_undegraded":"2023-10-19T14:22:02.406083+0000","last_fullsized":"2023-10-19T14:22:02.406083+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:07:50.835228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.1d","version":"262'2","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.400756+0000","last_change":"2023-10-19T14:22:01.417975+0000","last_active":"2023-10-19T14:22:02.400756+0000","last_peered":"2023-10-19T14:22:02.400756+0000","last_clean":"2023-10-19T14:22:02.400756+0000","last_became_active":"2023-10-19T14:22:01.417704+0000","last_became_peered":"2023-10-19T14:22:01.417704+0000","last_unstale":"2023-10-19T14:22:02.400756+0000","last_undegraded":"2023-10-19T14:22:02.400756+0000","last_fullsized":"2023-10-19T14:22:02.400756+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:56:00.152228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.1b","version":"0'0","reported_seq":9,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.416951+0000","last_change":"2023-10-19T14:22:02.412219+0000","last_active":"2023-10-19T14:22:02.416951+0000","last_peered":"2023-10-19T14:22:02.416951+0000","last_clean":"2023-10-19T14:22:02.416951+0000","last_became_active":"2023-10-19T14:22:02.411959+0000","last_became_peered":"2023-10-19T14:22:02.411959+0000","last_unstale":"2023-10-19T14:22:02.416951+0000","last_undegraded":"2023-10-19T14:22:02.416951+0000","last_fullsized":"2023-10-19T14:22:02.416951+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:54:56.271440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.1c","version":"262'2","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.400340+0000","last_change":"2023-10-19T14:22:01.419018+0000","last_active":"2023-10-19T14:22:02.400340+0000","last_peered":"2023-10-19T14:22:02.400340+0000","last_clean":"2023-10-19T14:22:02.400340+0000","last_became_active":"2023-10-19T14:22:01.418769+0000","last_became_peered":"2023-10-19T14:22:01.418769+0000","last_unstale":"2023-10-19T14:22:02.400340+0000","last_undegraded":"2023-10-19T14:22:02.400340+0000","last_fullsized":"2023-10-19T14:22:02.400340+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:59:43.263869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.1c","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.402955+0000","last_change":"2023-10-19T14:22:02.402955+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.402955+0000","last_undegraded":"2023-10-19T14:22:02.402955+0000","last_fullsized":"2023-10-19T14:22:02.402955+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:03:27.890218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.1b","version":"262'1","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.409720+0000","last_change":"2023-10-19T14:22:01.413953+0000","last_active":"2023-10-19T14:22:02.409720+0000","last_peered":"2023-10-19T14:22:02.409720+0000","last_clean":"2023-10-19T14:22:02.409720+0000","last_became_active":"2023-10-19T14:22:01.413689+0000","last_became_peered":"2023-10-19T14:22:01.413689+0000","last_unstale":"2023-10-19T14:22:02.409720+0000","last_undegraded":"2023-10-19T14:22:02.409720+0000","last_fullsized":"2023-10-19T14:22:02.409720+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:33:45.430960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.1d","version":"0'0","reported_seq":6,"reported_epoch":262,"state":"active","last_fresh":"2023-10-19T14:22:02.413928+0000","last_change":"2023-10-19T14:22:02.413862+0000","last_active":"2023-10-19T14:22:02.413928+0000","last_peered":"2023-10-19T14:22:02.413928+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"2023-10-19T14:22:02.413862+0000","last_became_peered":"2023-10-19T14:22:02.413862+0000","last_unstale":"2023-10-19T14:22:02.413928+0000","last_undegraded":"2023-10-19T14:22:02.413928+0000","last_fullsized":"2023-10-19T14:22:02.413928+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:20:55.394812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.1a","version":"0'0","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.408577+0000","last_change":"2023-10-19T14:22:01.409991+0000","last_active":"2023-10-19T14:22:02.408577+0000","last_peered":"2023-10-19T14:22:02.408577+0000","last_clean":"2023-10-19T14:22:02.408577+0000","last_became_active":"2023-10-19T14:22:01.409510+0000","last_became_peered":"2023-10-19T14:22:01.409510+0000","last_unstale":"2023-10-19T14:22:02.408577+0000","last_undegraded":"2023-10-19T14:22:02.408577+0000","last_fullsized":"2023-10-19T14:22:02.408577+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:42:53.919625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.1e","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.412327+0000","last_change":"2023-10-19T14:22:02.412327+0000","last_active":"2023-10-19T14:22:02.412327+0000","last_peered":"2023-10-19T14:22:02.412327+0000","last_clean":"2023-10-19T14:22:02.412327+0000","last_became_active":"2023-10-19T14:22:02.412057+0000","last_became_peered":"2023-10-19T14:22:02.412057+0000","last_unstale":"2023-10-19T14:22:02.412327+0000","last_undegraded":"2023-10-19T14:22:02.412327+0000","last_fullsized":"2023-10-19T14:22:02.412327+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:32:30.545638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.19","version":"0'0","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.399288+0000","last_change":"2023-10-19T14:22:01.416901+0000","last_active":"2023-10-19T14:22:02.399288+0000","last_peered":"2023-10-19T14:22:02.399288+0000","last_clean":"2023-10-19T14:22:02.399288+0000","last_became_active":"2023-10-19T14:22:01.416656+0000","last_became_peered":"2023-10-19T14:22:01.416656+0000","last_unstale":"2023-10-19T14:22:02.399288+0000","last_undegraded":"2023-10-19T14:22:02.399288+0000","last_fullsized":"2023-10-19T14:22:02.399288+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:27:36.174418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.1f","version":"0'0","reported_seq":9,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.420545+0000","last_change":"2023-10-19T14:22:02.418425+0000","last_active":"2023-10-19T14:22:02.420545+0000","last_peered":"2023-10-19T14:22:02.420545+0000","last_clean":"2023-10-19T14:22:02.420545+0000","last_became_active":"2023-10-19T14:22:02.417278+0000","last_became_peered":"2023-10-19T14:22:02.417278+0000","last_unstale":"2023-10-19T14:22:02.420545+0000","last_undegraded":"2023-10-19T14:22:02.420545+0000","last_fullsized":"2023-10-19T14:22:02.420545+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:04:24.989932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.18","version":"0'0","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.404196+0000","last_change":"2023-10-19T14:22:01.412557+0000","last_active":"2023-10-19T14:22:02.404196+0000","last_peered":"2023-10-19T14:22:02.404196+0000","last_clean":"2023-10-19T14:22:02.404196+0000","last_became_active":"2023-10-19T14:22:01.412115+0000","last_became_peered":"2023-10-19T14:22:01.412115+0000","last_unstale":"2023-10-19T14:22:02.404196+0000","last_undegraded":"2023-10-19T14:22:02.404196+0000","last_fullsized":"2023-10-19T14:22:02.404196+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:54:33.650542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.10","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.402706+0000","last_change":"2023-10-19T14:22:02.402706+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.402706+0000","last_undegraded":"2023-10-19T14:22:02.402706+0000","last_fullsized":"2023-10-19T14:22:02.402706+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:45:46.279692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.17","version":"0'0","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.415002+0000","last_change":"2023-10-19T14:22:01.412962+0000","last_active":"2023-10-19T14:22:02.415002+0000","last_peered":"2023-10-19T14:22:02.415002+0000","last_clean":"2023-10-19T14:22:02.415002+0000","last_became_active":"2023-10-19T14:22:01.412680+0000","last_became_peered":"2023-10-19T14:22:01.412680+0000","last_unstale":"2023-10-19T14:22:02.415002+0000","last_undegraded":"2023-10-19T14:22:02.415002+0000","last_fullsized":"2023-10-19T14:22:02.415002+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:03:54.917148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.11","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.405750+0000","last_change":"2023-10-19T14:22:02.405750+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.405750+0000","last_undegraded":"2023-10-19T14:22:02.405750+0000","last_fullsized":"2023-10-19T14:22:02.405750+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:06:30.359305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.16","version":"0'0","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.399221+0000","last_change":"2023-10-19T14:22:01.414464+0000","last_active":"2023-10-19T14:22:02.399221+0000","last_peered":"2023-10-19T14:22:02.399221+0000","last_clean":"2023-10-19T14:22:02.399221+0000","last_became_active":"2023-10-19T14:22:01.413414+0000","last_became_peered":"2023-10-19T14:22:01.413414+0000","last_unstale":"2023-10-19T14:22:02.399221+0000","last_undegraded":"2023-10-19T14:22:02.399221+0000","last_fullsized":"2023-10-19T14:22:02.399221+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:46:22.763844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.12","version":"0'0","reported_seq":9,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.413606+0000","last_change":"2023-10-19T14:22:02.410248+0000","last_active":"2023-10-19T14:22:02.413606+0000","last_peered":"2023-10-19T14:22:02.413606+0000","last_clean":"2023-10-19T14:22:02.413606+0000","last_became_active":"2023-10-19T14:22:02.409982+0000","last_became_peered":"2023-10-19T14:22:02.409982+0000","last_unstale":"2023-10-19T14:22:02.413606+0000","last_undegraded":"2023-10-19T14:22:02.413606+0000","last_fullsized":"2023-10-19T14:22:02.413606+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:27:43.397970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.15","version":"262'3","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.414741+0000","last_change":"2023-10-19T14:22:01.409543+0000","last_active":"2023-10-19T14:22:02.414741+0000","last_peered":"2023-10-19T14:22:02.414741+0000","last_clean":"2023-10-19T14:22:02.414741+0000","last_became_active":"2023-10-19T14:22:01.409293+0000","last_became_peered":"2023-10-19T14:22:01.409293+0000","last_unstale":"2023-10-19T14:22:02.414741+0000","last_undegraded":"2023-10-19T14:22:02.414741+0000","last_fullsized":"2023-10-19T14:22:02.414741+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:12:16.850155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.13","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.408938+0000","last_change":"2023-10-19T14:22:02.408938+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.408938+0000","last_undegraded":"2023-10-19T14:22:02.408938+0000","last_fullsized":"2023-10-19T14:22:02.408938+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:22:40.195315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.14","version":"262'5","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.404183+0000","last_change":"2023-10-19T14:22:01.411386+0000","last_active":"2023-10-19T14:22:02.404183+0000","last_peered":"2023-10-19T14:22:02.404183+0000","last_clean":"2023-10-19T14:22:02.404183+0000","last_became_active":"2023-10-19T14:22:01.411111+0000","last_became_peered":"2023-10-19T14:22:01.411111+0000","last_unstale":"2023-10-19T14:22:02.404183+0000","last_undegraded":"2023-10-19T14:22:02.404183+0000","last_fullsized":"2023-10-19T14:22:02.404183+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:10:26.534502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.14","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.416878+0000","last_change":"2023-10-19T14:22:02.416878+0000","last_active":"2023-10-19T14:22:02.416878+0000","last_peered":"2023-10-19T14:22:02.416878+0000","last_clean":"2023-10-19T14:22:02.416878+0000","last_became_active":"2023-10-19T14:22:02.416633+0000","last_became_peered":"2023-10-19T14:22:02.416633+0000","last_unstale":"2023-10-19T14:22:02.416878+0000","last_undegraded":"2023-10-19T14:22:02.416878+0000","last_fullsized":"2023-10-19T14:22:02.416878+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:56:41.835345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.13","version":"262'2","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.399368+0000","last_change":"2023-10-19T14:22:01.410762+0000","last_active":"2023-10-19T14:22:02.399368+0000","last_peered":"2023-10-19T14:22:02.399368+0000","last_clean":"2023-10-19T14:22:02.399368+0000","last_became_active":"2023-10-19T14:22:01.410507+0000","last_became_peered":"2023-10-19T14:22:01.410507+0000","last_unstale":"2023-10-19T14:22:02.399368+0000","last_undegraded":"2023-10-19T14:22:02.399368+0000","last_fullsized":"2023-10-19T14:22:02.399368+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:38:13.918889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.15","version":"0'0","reported_seq":6,"reported_epoch":262,"state":"active","last_fresh":"2023-10-19T14:22:02.414429+0000","last_change":"2023-10-19T14:22:02.414358+0000","last_active":"2023-10-19T14:22:02.414429+0000","last_peered":"2023-10-19T14:22:02.414429+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"2023-10-19T14:22:02.414358+0000","last_became_peered":"2023-10-19T14:22:02.414358+0000","last_unstale":"2023-10-19T14:22:02.414429+0000","last_undegraded":"2023-10-19T14:22:02.414429+0000","last_fullsized":"2023-10-19T14:22:02.414429+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:02:05.922124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.12","version":"0'0","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.409323+0000","last_change":"2023-10-19T14:22:01.413870+0000","last_active":"2023-10-19T14:22:02.409323+0000","last_peered":"2023-10-19T14:22:02.409323+0000","last_clean":"2023-10-19T14:22:02.409323+0000","last_became_active":"2023-10-19T14:22:01.413532+0000","last_became_peered":"2023-10-19T14:22:01.413532+0000","last_unstale":"2023-10-19T14:22:02.409323+0000","last_undegraded":"2023-10-19T14:22:02.409323+0000","last_fullsized":"2023-10-19T14:22:02.409323+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:47:48.522113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.16","version":"0'0","reported_seq":9,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.421387+0000","last_change":"2023-10-19T14:22:02.419152+0000","last_active":"2023-10-19T14:22:02.421387+0000","last_peered":"2023-10-19T14:22:02.421387+0000","last_clean":"2023-10-19T14:22:02.421387+0000","last_became_active":"2023-10-19T14:22:02.417515+0000","last_became_peered":"2023-10-19T14:22:02.417515+0000","last_unstale":"2023-10-19T14:22:02.421387+0000","last_undegraded":"2023-10-19T14:22:02.421387+0000","last_fullsized":"2023-10-19T14:22:02.421387+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:39:40.912960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.11","version":"0'0","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.409983+0000","last_change":"2023-10-19T14:22:01.416474+0000","last_active":"2023-10-19T14:22:02.409983+0000","last_peered":"2023-10-19T14:22:02.409983+0000","last_clean":"2023-10-19T14:22:02.409983+0000","last_became_active":"2023-10-19T14:22:01.416200+0000","last_became_peered":"2023-10-19T14:22:01.416200+0000","last_unstale":"2023-10-19T14:22:02.409983+0000","last_undegraded":"2023-10-19T14:22:02.409983+0000","last_fullsized":"2023-10-19T14:22:02.409983+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:55:39.137949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.17","version":"0'0","reported_seq":9,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.421450+0000","last_change":"2023-10-19T14:22:02.419116+0000","last_active":"2023-10-19T14:22:02.421450+0000","last_peered":"2023-10-19T14:22:02.421450+0000","last_clean":"2023-10-19T14:22:02.421450+0000","last_became_active":"2023-10-19T14:22:02.418655+0000","last_became_peered":"2023-10-19T14:22:02.418655+0000","last_unstale":"2023-10-19T14:22:02.421450+0000","last_undegraded":"2023-10-19T14:22:02.421450+0000","last_fullsized":"2023-10-19T14:22:02.421450+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:28:48.304947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.10","version":"262'8","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.402744+0000","last_change":"2023-10-19T14:22:01.415486+0000","last_active":"2023-10-19T14:22:02.402744+0000","last_peered":"2023-10-19T14:22:02.402744+0000","last_clean":"2023-10-19T14:22:02.402744+0000","last_became_active":"2023-10-19T14:22:01.415192+0000","last_became_peered":"2023-10-19T14:22:01.415192+0000","last_unstale":"2023-10-19T14:22:02.402744+0000","last_undegraded":"2023-10-19T14:22:02.402744+0000","last_fullsized":"2023-10-19T14:22:02.402744+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:45:03.326043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.8","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.407065+0000","last_change":"2023-10-19T14:22:02.407065+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.407065+0000","last_undegraded":"2023-10-19T14:22:02.407065+0000","last_fullsized":"2023-10-19T14:22:02.407065+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:07:55.991807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.f","version":"262'1","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.417942+0000","last_change":"2023-10-19T14:22:01.417521+0000","last_active":"2023-10-19T14:22:02.417942+0000","last_peered":"2023-10-19T14:22:02.417942+0000","last_clean":"2023-10-19T14:22:02.417942+0000","last_became_active":"2023-10-19T14:22:01.417209+0000","last_became_peered":"2023-10-19T14:22:01.417209+0000","last_unstale":"2023-10-19T14:22:02.417942+0000","last_undegraded":"2023-10-19T14:22:02.417942+0000","last_fullsized":"2023-10-19T14:22:02.417942+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:43:26.494066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.9","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.398780+0000","last_change":"2023-10-19T14:22:02.398780+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.398780+0000","last_undegraded":"2023-10-19T14:22:02.398780+0000","last_fullsized":"2023-10-19T14:22:02.398780+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:40:53.511724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.e","version":"0'0","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.411175+0000","last_change":"2023-10-19T14:22:01.412531+0000","last_active":"2023-10-19T14:22:02.411175+0000","last_peered":"2023-10-19T14:22:02.411175+0000","last_clean":"2023-10-19T14:22:02.411175+0000","last_became_active":"2023-10-19T14:22:01.412209+0000","last_became_peered":"2023-10-19T14:22:01.412209+0000","last_unstale":"2023-10-19T14:22:02.411175+0000","last_undegraded":"2023-10-19T14:22:02.411175+0000","last_fullsized":"2023-10-19T14:22:02.411175+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:45:03.326043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.a","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.407858+0000","last_change":"2023-10-19T14:22:02.407858+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.407858+0000","last_undegraded":"2023-10-19T14:22:02.407858+0000","last_fullsized":"2023-10-19T14:22:02.407858+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:31:43.355728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.d","version":"262'1","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.400892+0000","last_change":"2023-10-19T14:22:01.415741+0000","last_active":"2023-10-19T14:22:02.400892+0000","last_peered":"2023-10-19T14:22:02.400892+0000","last_clean":"2023-10-19T14:22:02.400892+0000","last_became_active":"2023-10-19T14:22:01.415385+0000","last_became_peered":"2023-10-19T14:22:01.415385+0000","last_unstale":"2023-10-19T14:22:02.400892+0000","last_undegraded":"2023-10-19T14:22:02.400892+0000","last_fullsized":"2023-10-19T14:22:02.400892+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:34:06.962782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.b","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.413768+0000","last_change":"2023-10-19T14:22:02.413768+0000","last_active":"2023-10-19T14:22:02.413768+0000","last_peered":"2023-10-19T14:22:02.413768+0000","last_clean":"2023-10-19T14:22:02.413768+0000","last_became_active":"2023-10-19T14:22:02.413498+0000","last_became_peered":"2023-10-19T14:22:02.413498+0000","last_unstale":"2023-10-19T14:22:02.413768+0000","last_undegraded":"2023-10-19T14:22:02.413768+0000","last_fullsized":"2023-10-19T14:22:02.413768+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:09:01.416730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.c","version":"0'0","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.399235+0000","last_change":"2023-10-19T14:22:01.411380+0000","last_active":"2023-10-19T14:22:02.399235+0000","last_peered":"2023-10-19T14:22:02.399235+0000","last_clean":"2023-10-19T14:22:02.399235+0000","last_became_active":"2023-10-19T14:22:01.411092+0000","last_became_peered":"2023-10-19T14:22:01.411092+0000","last_unstale":"2023-10-19T14:22:02.399235+0000","last_undegraded":"2023-10-19T14:22:02.399235+0000","last_fullsized":"2023-10-19T14:22:02.399235+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:01:14.941331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.c","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.413516+0000","last_change":"2023-10-19T14:22:02.413516+0000","last_active":"2023-10-19T14:22:02.413516+0000","last_peered":"2023-10-19T14:22:02.413516+0000","last_clean":"2023-10-19T14:22:02.413516+0000","last_became_active":"2023-10-19T14:22:02.413230+0000","last_became_peered":"2023-10-19T14:22:02.413230+0000","last_unstale":"2023-10-19T14:22:02.413516+0000","last_undegraded":"2023-10-19T14:22:02.413516+0000","last_fullsized":"2023-10-19T14:22:02.413516+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:22:42.002875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.b","version":"262'2","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.399530+0000","last_change":"2023-10-19T14:22:01.409927+0000","last_active":"2023-10-19T14:22:02.399530+0000","last_peered":"2023-10-19T14:22:02.399530+0000","last_clean":"2023-10-19T14:22:02.399530+0000","last_became_active":"2023-10-19T14:22:01.409617+0000","last_became_peered":"2023-10-19T14:22:01.409617+0000","last_unstale":"2023-10-19T14:22:02.399530+0000","last_undegraded":"2023-10-19T14:22:02.399530+0000","last_fullsized":"2023-10-19T14:22:02.399530+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:38:30.217600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.d","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.399329+0000","last_change":"2023-10-19T14:22:02.399329+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.399329+0000","last_undegraded":"2023-10-19T14:22:02.399329+0000","last_fullsized":"2023-10-19T14:22:02.399329+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:17:41.707119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.a","version":"0'0","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.404454+0000","last_change":"2023-10-19T14:22:01.417576+0000","last_active":"2023-10-19T14:22:02.404454+0000","last_peered":"2023-10-19T14:22:02.404454+0000","last_clean":"2023-10-19T14:22:02.404454+0000","last_became_active":"2023-10-19T14:22:01.417230+0000","last_became_peered":"2023-10-19T14:22:01.417230+0000","last_unstale":"2023-10-19T14:22:02.404454+0000","last_undegraded":"2023-10-19T14:22:02.404454+0000","last_fullsized":"2023-10-19T14:22:02.404454+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:23:00.582706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.e","version":"0'0","reported_seq":9,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.421407+0000","last_change":"2023-10-19T14:22:02.419344+0000","last_active":"2023-10-19T14:22:02.421407+0000","last_peered":"2023-10-19T14:22:02.421407+0000","last_clean":"2023-10-19T14:22:02.421407+0000","last_became_active":"2023-10-19T14:22:02.417238+0000","last_became_peered":"2023-10-19T14:22:02.417238+0000","last_unstale":"2023-10-19T14:22:02.421407+0000","last_undegraded":"2023-10-19T14:22:02.421407+0000","last_fullsized":"2023-10-19T14:22:02.421407+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:19:36.485520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.9","version":"0'0","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.400018+0000","last_change":"2023-10-19T14:22:01.413457+0000","last_active":"2023-10-19T14:22:02.400018+0000","last_peered":"2023-10-19T14:22:02.400018+0000","last_clean":"2023-10-19T14:22:02.400018+0000","last_became_active":"2023-10-19T14:22:01.413179+0000","last_became_peered":"2023-10-19T14:22:01.413179+0000","last_unstale":"2023-10-19T14:22:02.400018+0000","last_undegraded":"2023-10-19T14:22:02.400018+0000","last_fullsized":"2023-10-19T14:22:02.400018+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:34:30.334044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.f","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.409049+0000","last_change":"2023-10-19T14:22:02.409049+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.409049+0000","last_undegraded":"2023-10-19T14:22:02.409049+0000","last_fullsized":"2023-10-19T14:22:02.409049+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:44:13.928436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.8","version":"0'0","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.399188+0000","last_change":"2023-10-19T14:22:01.413949+0000","last_active":"2023-10-19T14:22:02.399188+0000","last_peered":"2023-10-19T14:22:02.399188+0000","last_clean":"2023-10-19T14:22:02.399188+0000","last_became_active":"2023-10-19T14:22:01.413670+0000","last_became_peered":"2023-10-19T14:22:01.413670+0000","last_unstale":"2023-10-19T14:22:02.399188+0000","last_undegraded":"2023-10-19T14:22:02.399188+0000","last_fullsized":"2023-10-19T14:22:02.399188+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:33:35.792891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.0","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.402684+0000","last_change":"2023-10-19T14:22:02.402684+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.402684+0000","last_undegraded":"2023-10-19T14:22:02.402684+0000","last_fullsized":"2023-10-19T14:22:02.402684+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:40:48.301290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.7","version":"262'2","reported_seq":13,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.413481+0000","last_change":"2023-10-19T14:22:01.412421+0000","last_active":"2023-10-19T14:22:02.413481+0000","last_peered":"2023-10-19T14:22:02.413481+0000","last_clean":"2023-10-19T14:22:02.413481+0000","last_became_active":"2023-10-19T14:22:01.412153+0000","last_became_peered":"2023-10-19T14:22:01.412153+0000","last_unstale":"2023-10-19T14:22:02.413481+0000","last_undegraded":"2023-10-19T14:22:02.413481+0000","last_fullsized":"2023-10-19T14:22:02.413481+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:38:47.923739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.1","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.419843+0000","last_change":"2023-10-19T14:22:02.419843+0000","last_active":"2023-10-19T14:22:02.419843+0000","last_peered":"2023-10-19T14:22:02.419843+0000","last_clean":"2023-10-19T14:22:02.419843+0000","last_became_active":"2023-10-19T14:22:02.417481+0000","last_became_peered":"2023-10-19T14:22:02.417481+0000","last_unstale":"2023-10-19T14:22:02.419843+0000","last_undegraded":"2023-10-19T14:22:02.419843+0000","last_fullsized":"2023-10-19T14:22:02.419843+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:12:12.862561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.6","version":"0'0","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.409952+0000","last_change":"2023-10-19T14:22:01.412088+0000","last_active":"2023-10-19T14:22:02.409952+0000","last_peered":"2023-10-19T14:22:02.409952+0000","last_clean":"2023-10-19T14:22:02.409952+0000","last_became_active":"2023-10-19T14:22:01.411646+0000","last_became_peered":"2023-10-19T14:22:01.411646+0000","last_unstale":"2023-10-19T14:22:02.409952+0000","last_undegraded":"2023-10-19T14:22:02.409952+0000","last_fullsized":"2023-10-19T14:22:02.409952+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:48:08.338501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.2","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.403340+0000","last_change":"2023-10-19T14:22:02.403340+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.403340+0000","last_undegraded":"2023-10-19T14:22:02.403340+0000","last_fullsized":"2023-10-19T14:22:02.403340+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:42:16.203127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.5","version":"0'0","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.401015+0000","last_change":"2023-10-19T14:22:01.417906+0000","last_active":"2023-10-19T14:22:02.401015+0000","last_peered":"2023-10-19T14:22:02.401015+0000","last_clean":"2023-10-19T14:22:02.401015+0000","last_became_active":"2023-10-19T14:22:01.417614+0000","last_became_peered":"2023-10-19T14:22:01.417614+0000","last_unstale":"2023-10-19T14:22:02.401015+0000","last_undegraded":"2023-10-19T14:22:02.401015+0000","last_fullsized":"2023-10-19T14:22:02.401015+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:26:05.681833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.3","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.417066+0000","last_change":"2023-10-19T14:22:02.417066+0000","last_active":"2023-10-19T14:22:02.417066+0000","last_peered":"2023-10-19T14:22:02.417066+0000","last_clean":"2023-10-19T14:22:02.417066+0000","last_became_active":"2023-10-19T14:22:02.416770+0000","last_became_peered":"2023-10-19T14:22:02.416770+0000","last_unstale":"2023-10-19T14:22:02.417066+0000","last_undegraded":"2023-10-19T14:22:02.417066+0000","last_fullsized":"2023-10-19T14:22:02.417066+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:25:15.074214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.4","version":"0'0","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.409766+0000","last_change":"2023-10-19T14:22:01.412056+0000","last_active":"2023-10-19T14:22:02.409766+0000","last_peered":"2023-10-19T14:22:02.409766+0000","last_clean":"2023-10-19T14:22:02.409766+0000","last_became_active":"2023-10-19T14:22:01.411786+0000","last_became_peered":"2023-10-19T14:22:01.411786+0000","last_unstale":"2023-10-19T14:22:02.409766+0000","last_undegraded":"2023-10-19T14:22:02.409766+0000","last_fullsized":"2023-10-19T14:22:02.409766+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:40:51.932206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.4","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.403966+0000","last_change":"2023-10-19T14:22:02.403966+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.403966+0000","last_undegraded":"2023-10-19T14:22:02.403966+0000","last_fullsized":"2023-10-19T14:22:02.403966+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:43:57.998020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.3","version":"0'0","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.399541+0000","last_change":"2023-10-19T14:22:01.413689+0000","last_active":"2023-10-19T14:22:02.399541+0000","last_peered":"2023-10-19T14:22:02.399541+0000","last_clean":"2023-10-19T14:22:02.399541+0000","last_became_active":"2023-10-19T14:22:01.413419+0000","last_became_peered":"2023-10-19T14:22:01.413419+0000","last_unstale":"2023-10-19T14:22:02.399541+0000","last_undegraded":"2023-10-19T14:22:02.399541+0000","last_fullsized":"2023-10-19T14:22:02.399541+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:37:57.754265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.5","version":"0'0","reported_seq":9,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.414555+0000","last_change":"2023-10-19T14:22:02.409126+0000","last_active":"2023-10-19T14:22:02.414555+0000","last_peered":"2023-10-19T14:22:02.414555+0000","last_clean":"2023-10-19T14:22:02.414555+0000","last_became_active":"2023-10-19T14:22:02.408855+0000","last_became_peered":"2023-10-19T14:22:02.408855+0000","last_unstale":"2023-10-19T14:22:02.414555+0000","last_undegraded":"2023-10-19T14:22:02.414555+0000","last_fullsized":"2023-10-19T14:22:02.414555+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:03:36.116245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.2","version":"0'0","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.404648+0000","last_change":"2023-10-19T14:22:01.411905+0000","last_active":"2023-10-19T14:22:02.404648+0000","last_peered":"2023-10-19T14:22:02.404648+0000","last_clean":"2023-10-19T14:22:02.404648+0000","last_became_active":"2023-10-19T14:22:01.411584+0000","last_became_peered":"2023-10-19T14:22:01.411584+0000","last_unstale":"2023-10-19T14:22:02.404648+0000","last_undegraded":"2023-10-19T14:22:02.404648+0000","last_fullsized":"2023-10-19T14:22:02.404648+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:40:13.932927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.6","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.409726+0000","last_change":"2023-10-19T14:22:02.409726+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.409726+0000","last_undegraded":"2023-10-19T14:22:02.409726+0000","last_fullsized":"2023-10-19T14:22:02.409726+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:51:43.993546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.1","version":"0'0","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.403849+0000","last_change":"2023-10-19T14:22:01.412806+0000","last_active":"2023-10-19T14:22:02.403849+0000","last_peered":"2023-10-19T14:22:02.403849+0000","last_clean":"2023-10-19T14:22:02.403849+0000","last_became_active":"2023-10-19T14:22:01.412524+0000","last_became_peered":"2023-10-19T14:22:01.412524+0000","last_unstale":"2023-10-19T14:22:02.403849+0000","last_undegraded":"2023-10-19T14:22:02.403849+0000","last_fullsized":"2023-10-19T14:22:02.403849+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:43:54.078045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.7","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.414678+0000","last_change":"2023-10-19T14:22:02.414678+0000","last_active":"2023-10-19T14:22:02.414678+0000","last_peered":"2023-10-19T14:22:02.414678+0000","last_clean":"2023-10-19T14:22:02.414678+0000","last_became_active":"2023-10-19T14:22:02.414348+0000","last_became_peered":"2023-10-19T14:22:02.414348+0000","last_unstale":"2023-10-19T14:22:02.414678+0000","last_undegraded":"2023-10-19T14:22:02.414678+0000","last_fullsized":"2023-10-19T14:22:02.414678+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:18:02.907457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.0","version":"0'0","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.399367+0000","last_change":"2023-10-19T14:22:01.414455+0000","last_active":"2023-10-19T14:22:02.399367+0000","last_peered":"2023-10-19T14:22:02.399367+0000","last_clean":"2023-10-19T14:22:02.399367+0000","last_became_active":"2023-10-19T14:22:01.414081+0000","last_became_peered":"2023-10-19T14:22:01.414081+0000","last_unstale":"2023-10-19T14:22:02.399367+0000","last_undegraded":"2023-10-19T14:22:02.399367+0000","last_fullsized":"2023-10-19T14:22:02.399367+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:54:25.277401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.25","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.418670+0000","last_change":"2023-10-19T14:22:02.418670+0000","last_active":"2023-10-19T14:22:02.418670+0000","last_peered":"2023-10-19T14:22:02.418670+0000","last_clean":"2023-10-19T14:22:02.418670+0000","last_became_active":"2023-10-19T14:22:02.417862+0000","last_became_peered":"2023-10-19T14:22:02.417862+0000","last_unstale":"2023-10-19T14:22:02.418670+0000","last_undegraded":"2023-10-19T14:22:02.418670+0000","last_fullsized":"2023-10-19T14:22:02.418670+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:20:15.811864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"249'3224","reported_seq":4253,"reported_epoch":261,"state":"active+clean","last_fresh":"2023-10-19T14:22:01.406839+0000","last_change":"2023-10-19T12:44:08.636613+0000","last_active":"2023-10-19T14:22:01.406839+0000","last_peered":"2023-10-19T14:22:01.406839+0000","last_clean":"2023-10-19T14:22:01.406839+0000","last_became_active":"2023-10-19T12:44:08.635697+0000","last_became_peered":"2023-10-19T12:44:08.635697+0000","last_unstale":"2023-10-19T14:22:01.406839+0000","last_undegraded":"2023-10-19T14:22:01.406839+0000","last_fullsized":"2023-10-19T14:22:01.406839+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_clean_scrub_stamp":"2023-10-19T12:44:07.399738+0000","objects_scrubbed":0,"log_size":3224,"log_dups_size":0,"ondisk_log_size":3224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:15:23.685485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2539,"num_read_kb":5339,"num_write":657,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":36,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":35,"num_pg":32,"stat_sum":{"num_bytes":1470,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":33,"ondisk_log_size":33,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2539,"num_read_kb":5339,"num_write":657,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3224,"ondisk_log_size":3224,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576020,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66928,"kb_used_data":3804,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94304912,"statfs":{"total":96636764160,"available":96568229888,"internally_reserved":0,"allocated":3895296,"data_stored":2294561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":64616907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:21:10 2023","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.55100000000000005,"15min":0.57999999999999996},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.32200000000000001},"max":{"1min":0.754,"5min":0.80900000000000005,"15min":0.86199999999999999},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.503,"5min":0.57599999999999996,"15min":0.59899999999999998},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.312},"max":{"1min":0.66500000000000004,"5min":0.81899999999999995,"15min":0.86599999999999999},"last":0.39400000000000002}]},{"osd":1,"last update":"Thu Oct 19 14:21:10 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.58199999999999996,"15min":0.59499999999999997},"min":{"1min":0.45200000000000001,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.72199999999999998,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.56899999999999995,"15min":0.59299999999999997},"min":{"1min":0.39300000000000002,"5min":0.373,"15min":0.36699999999999999},"max":{"1min":0.78500000000000003,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.56200000000000006}]},{"osd":2,"last update":"Thu Oct 19 14:21:10 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.622,"15min":0.64200000000000002},"min":{"1min":0.47299999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.70499999999999996,"5min":0.84799999999999998,"15min":0.85499999999999998},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.63900000000000001,"15min":0.65700000000000003},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.54600000000000004}]},{"osd":3,"last update":"Thu Oct 19 14:21:10 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.60799999999999998,"15min":0.63100000000000001},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.628,"5min":0.67100000000000004,"15min":0.68300000000000005},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.83299999999999996,"5min":0.876,"15min":0.89600000000000002},"last":0.66700000000000004}]},{"osd":4,"last update":"Thu Oct 19 14:21:10 2023","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.50600000000000001,"15min":0.495},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.53000000000000003,"15min":0.51100000000000001},"min":{"1min":0.34999999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.505}]},{"osd":5,"last update":"Thu Oct 19 14:21:10 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52300000000000002,"15min":0.51100000000000001},"min":{"1min":0.33700000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.57799999999999996,"15min":0.55400000000000005},"min":{"1min":0.40699999999999997,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.054,"5min":1.054,"15min":1.054},"last":0.626}]},{"osd":6,"last update":"Thu Oct 19 14:21:10 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.58999999999999997,"15min":0.57599999999999996},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.35999999999999999},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.59299999999999997,"15min":0.57299999999999995},"min":{"1min":0.42999999999999999,"5min":0.35599999999999998,"15min":0.33500000000000002},"max":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.57399999999999995}]}]},{"osd":6,"up_from":13,"seq":55834576020,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56612,"kb_used_data":3224,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315228,"statfs":{"total":96636764160,"available":96578793472,"internally_reserved":0,"allocated":3301376,"data_stored":1704193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:21:02 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.59199999999999997,"15min":0.61099999999999999},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.78400000000000003,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.56299999999999994,"15min":0.60599999999999998},"min":{"1min":0.36699999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.71799999999999997,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.80200000000000005}]},{"osd":1,"last update":"Thu Oct 19 14:21:02 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.60099999999999998,"15min":0.61799999999999999},"min":{"1min":0.373,"5min":0.373,"15min":0.35799999999999998},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":1.165},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.60199999999999998,"15min":0.626},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.77400000000000002,"5min":0.83699999999999997,"15min":0.91300000000000003},"last":0.76600000000000001}]},{"osd":2,"last update":"Thu Oct 19 14:21:02 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.57599999999999996,"15min":0.60199999999999998},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.69099999999999995,"5min":0.80100000000000005,"15min":0.873},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.60599999999999998,"15min":0.624},"min":{"1min":0.40899999999999997,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.74199999999999999,"5min":0.83499999999999996,"15min":0.93100000000000005},"last":0.73599999999999999}]},{"osd":3,"last update":"Thu Oct 19 14:21:02 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.629,"15min":0.64900000000000002},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.66700000000000004,"5min":0.875,"15min":0.875},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.63800000000000001,"15min":0.65500000000000003},"min":{"1min":0.45400000000000001,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.92200000000000004,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.70399999999999996}]},{"osd":4,"last update":"Thu Oct 19 14:21:02 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.48899999999999999,"15min":0.48399999999999999},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.51000000000000001,"15min":0.50800000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.30399999999999999},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.85699999999999998},"last":0.61399999999999999}]},{"osd":5,"last update":"Thu Oct 19 14:21:02 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.50700000000000001,"15min":0.502},"min":{"1min":0.34799999999999998,"5min":0.32700000000000001,"15min":0.32500000000000001},"max":{"1min":0.79900000000000004,"5min":1.115,"15min":1.115},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.56599999999999995,"15min":0.54500000000000004},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.69199999999999995}]},{"osd":7,"last update":"Thu Oct 19 14:21:02 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.56000000000000005,"15min":0.54700000000000004},"min":{"1min":0.36799999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.90100000000000002},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.56699999999999995,"15min":0.55400000000000005},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.98399999999999999,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.67800000000000005}]}]},{"osd":5,"up_from":13,"seq":55834576020,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66348,"kb_used_data":3224,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305492,"statfs":{"total":96636764160,"available":96568823808,"internally_reserved":0,"allocated":3301376,"data_stored":1704193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:21:50 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.57399999999999995,"15min":0.60699999999999998},"min":{"1min":0.38,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.69399999999999995,"5min":0.79700000000000004,"15min":0.86399999999999999},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.59599999999999997,"15min":0.63300000000000001},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.70499999999999996,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.51100000000000001}]},{"osd":1,"last update":"Thu Oct 19 14:21:50 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.59099999999999997,"15min":0.622},"min":{"1min":0.34599999999999997,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.877},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.61199999999999999,"15min":0.63700000000000001},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.877,"5min":0.877,"15min":0.88500000000000001},"last":0.46999999999999997}]},{"osd":2,"last update":"Thu Oct 19 14:21:50 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.58199999999999996,"15min":0.61399999999999999},"min":{"1min":0.378,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.625,"15min":0.65100000000000002},"min":{"1min":0.441,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.78900000000000003,"5min":0.83299999999999996,"15min":0.90800000000000003},"last":0.65600000000000003}]},{"osd":3,"last update":"Thu Oct 19 14:21:50 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.60499999999999998,"15min":0.63600000000000001},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.81100000000000005,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.626,"15min":0.65900000000000003},"min":{"1min":0.40000000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.90900000000000003},"last":0.70099999999999996}]},{"osd":4,"last update":"Thu Oct 19 14:21:50 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.48799999999999999,"15min":0.48699999999999999},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.51800000000000002,"15min":0.505},"min":{"1min":0.309,"5min":0.308,"15min":0.28599999999999998},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.82099999999999995},"last":0.63900000000000001}]},{"osd":6,"last update":"Thu Oct 19 14:21:50 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.51700000000000002,"15min":0.502},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.88800000000000001},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.54100000000000004,"15min":0.53900000000000003},"min":{"1min":0.36899999999999999,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.67700000000000005}]},{"osd":7,"last update":"Thu Oct 19 14:21:50 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.52200000000000002,"15min":0.51800000000000002},"min":{"1min":0.32700000000000001,"5min":0.312,"15min":0.312},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.499},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.57899999999999996,"15min":0.57199999999999995},"min":{"1min":0.38,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.72299999999999998}]}]},{"osd":4,"up_from":13,"seq":55834576020,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48292,"kb_used_data":3224,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323548,"statfs":{"total":96636764160,"available":96587313152,"internally_reserved":0,"allocated":3301376,"data_stored":1704193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:21:16 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.54700000000000004,"15min":0.57099999999999995},"min":{"1min":0.30299999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.879},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.56899999999999995,"15min":0.59299999999999997},"min":{"1min":0.35299999999999998,"5min":0.31,"15min":0.31},"max":{"1min":0.72699999999999998,"5min":0.82599999999999996,"15min":0.873},"last":0.54700000000000004}]},{"osd":1,"last update":"Thu Oct 19 14:21:16 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.59699999999999998,"15min":0.63500000000000001},"min":{"1min":0.40300000000000002,"5min":0.372,"15min":0.32600000000000001},"max":{"1min":0.74099999999999999,"5min":0.80400000000000005,"15min":0.92700000000000005},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.60899999999999999,"15min":0.63900000000000001},"min":{"1min":0.40500000000000003,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.755,"5min":0.90000000000000002,"15min":1.0940000000000001},"last":0.53100000000000003}]},{"osd":2,"last update":"Thu Oct 19 14:21:16 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.626,"15min":0.64700000000000002},"min":{"1min":0.38900000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.80300000000000005,"5min":1.008,"15min":1.008},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.64200000000000002,"15min":0.66600000000000004},"min":{"1min":0.47199999999999998,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.81799999999999995,"5min":0.98799999999999999,"15min":1.008},"last":0.68799999999999994}]},{"osd":3,"last update":"Thu Oct 19 14:21:16 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.59499999999999997,"15min":0.627},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.68600000000000005,"5min":0.83399999999999996,"15min":0.91900000000000004},"last":0.755},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.65600000000000003,"15min":0.66600000000000004},"min":{"1min":0.48199999999999998,"5min":0.45400000000000001,"15min":0.437},"max":{"1min":0.85999999999999999,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.76500000000000001}]},{"osd":5,"last update":"Thu Oct 19 14:21:16 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.497,"15min":0.47799999999999998},"min":{"1min":0.33800000000000002,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.79700000000000004,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.50800000000000001,"15min":0.498},"min":{"1min":0.36499999999999999,"5min":0.33700000000000002,"15min":0.29199999999999998},"max":{"1min":0.81399999999999995,"5min":0.873,"15min":0.873},"last":0.502}]},{"osd":6,"last update":"Thu Oct 19 14:21:16 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.50700000000000001,"15min":0.49399999999999999},"min":{"1min":0.38500000000000001,"5min":0.34799999999999998,"15min":0.28199999999999997},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.54900000000000004,"15min":0.54300000000000004},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.88,"5min":0.88,"15min":1.2969999999999999},"last":0.47999999999999998}]},{"osd":7,"last update":"Thu Oct 19 14:21:16 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55100000000000005,"15min":0.54500000000000004},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.32600000000000001},"max":{"1min":0.78200000000000003,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.438},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.61199999999999999,"15min":0.59899999999999998},"min":{"1min":0.371,"5min":0.32100000000000001,"15min":0.318},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":1.032},"last":0.45600000000000002}]}]},{"osd":3,"up_from":13,"seq":55834576019,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56616,"kb_used_data":3224,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315224,"statfs":{"total":96636764160,"available":96578789376,"internally_reserved":0,"allocated":3301376,"data_stored":1704193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54655432},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:21:53 2023","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.39900000000000002,"15min":0.41699999999999998},"min":{"1min":0.23899999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.54500000000000004,"5min":0.60599999999999998,"15min":0.70699999999999996},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.39400000000000002,"15min":0.42199999999999999},"min":{"1min":0.23899999999999999,"5min":0.217,"15min":0.217},"max":{"1min":0.51900000000000002,"5min":0.61899999999999999,"15min":0.752},"last":0.36699999999999999}]},{"osd":1,"last update":"Thu Oct 19 14:21:53 2023","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.40699999999999997,"15min":0.42799999999999999},"min":{"1min":0.28100000000000003,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.71599999999999997,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.45500000000000002,"15min":0.47199999999999998},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26400000000000001},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.38800000000000001}]},{"osd":2,"last update":"Thu Oct 19 14:21:53 2023","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.46200000000000002,"15min":0.48099999999999998},"min":{"1min":0.34300000000000003,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.73099999999999998},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48499999999999999,"15min":0.504},"min":{"1min":0.30599999999999999,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":1.212,"5min":1.212,"15min":1.212},"last":0.35499999999999998}]},{"osd":4,"last update":"Thu Oct 19 14:21:53 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.67400000000000004,"15min":0.70099999999999996},"min":{"1min":0.44600000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.94199999999999995,"5min":1.018,"15min":1.157},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.629,"5min":0.68700000000000006,"15min":0.72299999999999998},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.89500000000000002,"5min":0.996,"15min":1.173},"last":0.752}]},{"osd":5,"last update":"Thu Oct 19 14:21:53 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.67500000000000004,"15min":0.68600000000000005},"min":{"1min":0.50900000000000001,"5min":0.46600000000000003,"15min":0.41999999999999998},"max":{"1min":0.96799999999999997,"5min":0.97299999999999998,"15min":1.0149999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.66800000000000004,"15min":0.68600000000000005},"min":{"1min":0.48899999999999999,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.97699999999999998,"5min":0.99399999999999999,"15min":1.105},"last":0.80600000000000005}]},{"osd":6,"last update":"Thu Oct 19 14:21:53 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.65000000000000002,"15min":0.68600000000000005},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.93000000000000005,"5min":1.0009999999999999,"15min":1.0669999999999999},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.68899999999999995,"15min":0.70999999999999996},"min":{"1min":0.505,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.079},"last":0.79100000000000004}]},{"osd":7,"last update":"Thu Oct 19 14:21:53 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67500000000000004,"15min":0.69799999999999995},"min":{"1min":0.46600000000000003,"5min":0.434,"15min":0.434},"max":{"1min":0.94999999999999996,"5min":0.96899999999999997,"15min":1.085},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.69799999999999995,"15min":0.72799999999999998},"min":{"1min":0.48799999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.95899999999999996,"5min":0.97199999999999998,"15min":1.1739999999999999},"last":0.60299999999999998}]}]},{"osd":2,"up_from":13,"seq":55834576021,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48292,"kb_used_data":3224,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323548,"statfs":{"total":96636764160,"available":96587313152,"internally_reserved":0,"allocated":3301376,"data_stored":1704193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.374,"15min":0.39600000000000002},"min":{"1min":0.25700000000000001,"5min":0.218,"15min":0.218},"max":{"1min":0.58799999999999997,"5min":0.66600000000000004,"15min":0.69699999999999995},"last":0.27800000000000002},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.41499999999999998,"15min":0.432},"min":{"1min":0.248,"5min":0.248,"15min":0.223},"max":{"1min":0.53100000000000003,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.379}]},{"osd":1,"last update":"Thu Oct 19 14:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.38300000000000001,"15min":0.40600000000000003},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.65400000000000003},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.44400000000000001,"15min":0.46000000000000002},"min":{"1min":0.28699999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.45100000000000001,"5min":1.742,"15min":1.742},"last":0.40100000000000002}]},{"osd":3,"last update":"Thu Oct 19 14:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.45500000000000002,"15min":0.47599999999999998},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.63300000000000001,"5min":0.69299999999999995,"15min":0.74399999999999999},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.47299999999999998,"15min":0.501},"min":{"1min":0.32700000000000001,"5min":0.32000000000000001,"15min":0.315},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.83599999999999997},"last":0.5}]},{"osd":4,"last update":"Thu Oct 19 14:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.63900000000000001,"15min":0.66200000000000003},"min":{"1min":0.40699999999999997,"5min":0.39600000000000002,"15min":0.39500000000000002},"max":{"1min":0.74399999999999999,"5min":0.89200000000000002,"15min":0.998},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.65400000000000003,"15min":0.67600000000000005},"min":{"1min":0.41899999999999998,"5min":0.38900000000000001,"15min":0.35199999999999998},"max":{"1min":0.79000000000000004,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.60599999999999998}]},{"osd":5,"last update":"Thu Oct 19 14:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.67800000000000005,"15min":0.69299999999999995},"min":{"1min":0.443,"5min":0.443,"15min":0.40600000000000003},"max":{"1min":0.79300000000000004,"5min":1.4330000000000001,"15min":1.4330000000000001},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.66900000000000004,"15min":0.68000000000000005},"min":{"1min":0.46899999999999997,"5min":0.45100000000000001,"15min":0.39300000000000002},"max":{"1min":0.77400000000000002,"5min":1.595,"15min":1.595},"last":0.80900000000000005}]},{"osd":6,"last update":"Thu Oct 19 14:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.67700000000000005,"15min":0.70199999999999996},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.80900000000000005,"5min":0.996,"15min":1.0189999999999999},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.629,"5min":0.69499999999999995,"15min":0.71399999999999997},"min":{"1min":0.438,"5min":0.433,"15min":0.40600000000000003},"max":{"1min":0.83399999999999996,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":0.73099999999999998}]},{"osd":7,"last update":"Thu Oct 19 14:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.68400000000000005,"15min":0.71299999999999997},"min":{"1min":0.47199999999999998,"5min":0.44800000000000001,"15min":0.41799999999999998},"max":{"1min":0.78800000000000003,"5min":1.0129999999999999,"15min":1.0289999999999999},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.71199999999999997,"15min":0.72599999999999998},"min":{"1min":0.42199999999999999,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.86699999999999999,"5min":1.53,"15min":1.53},"last":0.64200000000000002}]}]},{"osd":1,"up_from":13,"seq":55834576020,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48296,"kb_used_data":3224,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323544,"statfs":{"total":96636764160,"available":96587309056,"internally_reserved":0,"allocated":3301376,"data_stored":1704193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.38400000000000001,"15min":0.40300000000000002},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.46899999999999997,"5min":0.61599999999999999,"15min":0.67400000000000004},"last":0.22500000000000001},{"interface":"front","average":{"1min":0.378,"5min":0.41999999999999998,"15min":0.45000000000000001},"min":{"1min":0.27700000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.51500000000000001,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.28100000000000003}]},{"osd":2,"last update":"Thu Oct 19 14:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.42399999999999999,"15min":0.438},"min":{"1min":0.26100000000000001,"5min":0.251,"15min":0.251},"max":{"1min":0.505,"5min":0.63100000000000001,"15min":0.77500000000000002},"last":0.318},{"interface":"front","average":{"1min":0.434,"5min":0.47699999999999998,"15min":0.48799999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.56699999999999995,"5min":0.68700000000000006,"15min":0.82799999999999996},"last":0.44400000000000001}]},{"osd":3,"last update":"Thu Oct 19 14:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.495,"15min":0.501},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.29499999999999998},"max":{"1min":0.59399999999999997,"5min":0.76000000000000001,"15min":0.78900000000000003},"last":0.376},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.42899999999999999,"15min":0.44600000000000001},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.26100000000000001},"max":{"1min":0.59899999999999998,"5min":0.65200000000000002,"15min":0.69499999999999995},"last":0.245}]},{"osd":4,"last update":"Thu Oct 19 14:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.64200000000000002,"15min":0.66700000000000004},"min":{"1min":0.42299999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.80400000000000005,"5min":0.95999999999999996,"15min":1.042},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.65400000000000003,"15min":0.66600000000000004},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.32300000000000001},"max":{"1min":0.80500000000000005,"5min":0.90800000000000003,"15min":1.1659999999999999},"last":0.39500000000000002}]},{"osd":5,"last update":"Thu Oct 19 14:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.66700000000000004,"15min":0.68200000000000005},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.81200000000000006,"5min":0.96199999999999997,"15min":1.3500000000000001},"last":0.434},{"interface":"front","average":{"1min":0.62,"5min":0.68899999999999995,"15min":0.70299999999999996},"min":{"1min":0.442,"5min":0.44,"15min":0.40500000000000003},"max":{"1min":0.82899999999999996,"5min":1.0009999999999999,"15min":1.121},"last":0.49399999999999999}]},{"osd":6,"last update":"Thu Oct 19 14:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.65400000000000003,"15min":0.67600000000000005},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.84999999999999998,"5min":1.0129999999999999,"15min":1.2310000000000001},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.69699999999999995,"15min":0.71499999999999997},"min":{"1min":0.46700000000000003,"5min":0.434,"15min":0.434},"max":{"1min":0.85799999999999998,"5min":1.0009999999999999,"15min":1.274},"last":0.47699999999999998}]},{"osd":7,"last update":"Thu Oct 19 14:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.70499999999999996,"15min":0.72199999999999998},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.86899999999999999,"5min":0.99099999999999999,"15min":1.075},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.68200000000000005,"15min":0.69599999999999995},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.84599999999999997,"5min":1.0169999999999999,"15min":1.117},"last":0.46200000000000002}]}]},{"osd":0,"up_from":13,"seq":55834576021,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61348,"kb_used_data":3800,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310492,"statfs":{"total":96636764160,"available":96573943808,"internally_reserved":0,"allocated":3891200,"data_stored":2293942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Oct 19 14:21:13 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.40100000000000002,"15min":0.41099999999999998},"min":{"1min":0.26300000000000001,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.47799999999999998,"5min":0.61499999999999999,"15min":0.624},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.41699999999999998,"15min":0.45500000000000002},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.49199999999999999,"5min":0.70899999999999996,"15min":0.81100000000000005},"last":0.64300000000000002}]},{"osd":2,"last update":"Thu Oct 19 14:21:13 2023","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.34100000000000003,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.54400000000000004,"5min":0.754,"15min":0.754},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.377,"5min":0.42399999999999999,"15min":0.44400000000000001},"min":{"1min":0.28100000000000003,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.46899999999999997,"5min":0.60199999999999998,"15min":0.69099999999999995},"last":0.52400000000000002}]},{"osd":3,"last update":"Thu Oct 19 14:21:13 2023","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.436},"min":{"1min":0.27200000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.50800000000000001,"5min":0.58799999999999997,"15min":0.66800000000000004},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.48199999999999998,"15min":0.50700000000000001},"min":{"1min":0.29299999999999998,"5min":0.254,"15min":0.254},"max":{"1min":0.72399999999999998,"5min":0.78100000000000003,"15min":0.84599999999999997},"last":0.53400000000000003}]},{"osd":4,"last update":"Thu Oct 19 14:21:13 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.63,"15min":0.67000000000000004},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.80300000000000005,"5min":0.93100000000000005,"15min":1.1890000000000001},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.628,"15min":0.67400000000000004},"min":{"1min":0.45900000000000002,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.75600000000000001,"5min":0.93400000000000005,"15min":1.2150000000000001},"last":0.78300000000000003}]},{"osd":5,"last update":"Thu Oct 19 14:21:13 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.67200000000000004,"15min":0.71399999999999997},"min":{"1min":0.48599999999999999,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.83099999999999996,"5min":0.98499999999999999,"15min":1.1579999999999999},"last":0.748},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.67500000000000004,"15min":0.71399999999999997},"min":{"1min":0.499,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.80000000000000004,"5min":0.98299999999999998,"15min":1.2310000000000001},"last":0.73899999999999999}]},{"osd":6,"last update":"Thu Oct 19 14:21:13 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.66300000000000003,"15min":0.69899999999999995},"min":{"1min":0.46100000000000002,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.85499999999999998,"5min":0.97799999999999998,"15min":1.1739999999999999},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.65300000000000002,"15min":0.69599999999999995},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.70299999999999996,"5min":0.97799999999999998,"15min":1.0189999999999999},"last":0.76100000000000001}]},{"osd":7,"last update":"Thu Oct 19 14:21:13 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.66600000000000004,"15min":0.70999999999999996},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.79900000000000004,"5min":0.93400000000000005,"15min":1.1180000000000001},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.66100000000000003,"15min":0.69599999999999995},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.84499999999999997,"5min":0.93799999999999994,"15min":1.0349999999999999},"last":0.77300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-19T14:22:05.762 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-19T14:22:05.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.902+0000 7f174498e640 1 -- 172.21.15.57:0/3705285080 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f17400cbd20 msgr2=0x7f1740158300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:05.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.902+0000 7f174498e640 1 --2- 172.21.15.57:0/3705285080 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f17400cbd20 0x7f1740158300 secure :-1 s=READY pgs=2889 cs=0 l=1 rev1=1 crypto rx=0x7f1730005fd0 tx=0x7f173002bd90 comp rx=0 tx=0).stop 2023-10-19T14:22:05.906 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.903+0000 7f174498e640 1 -- 172.21.15.57:0/3705285080 shutdown_connections 2023-10-19T14:22:05.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.903+0000 7f174498e640 1 --2- 172.21.15.57:0/3705285080 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f17400afa20 0x7f17400705a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:05.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.903+0000 7f174498e640 1 --2- 172.21.15.57:0/3705285080 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f17400cbd20 0x7f1740158300 unknown :-1 s=CLOSED pgs=2889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:05.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.903+0000 7f174498e640 1 -- 172.21.15.57:0/3705285080 >> 172.21.15.57:0/3705285080 conn(0x7f1740069930 msgr2=0x7f1740069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:05.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.903+0000 7f174498e640 1 -- 172.21.15.57:0/3705285080 shutdown_connections 2023-10-19T14:22:05.907 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.903+0000 7f174498e640 1 -- 172.21.15.57:0/3705285080 wait complete. 2023-10-19T14:22:05.911 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.904+0000 7f174498e640 1 Processor -- start 2023-10-19T14:22:05.912 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.904+0000 7f174498e640 1 -- start start 2023-10-19T14:22:05.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.905+0000 7f174498e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f17400afa20 0x7f17401380e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:05.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.905+0000 7f174498e640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f17400b0530 0x7f1740144860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:05.913 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.905+0000 7f174498e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f17400cbd20 0x7f1740144da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:05.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.905+0000 7f174498e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1740145df0 con 0x7f17400afa20 2023-10-19T14:22:05.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.905+0000 7f174498e640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17401452e0 con 0x7f17400cbd20 2023-10-19T14:22:05.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.905+0000 7f174498e640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1740145550 con 0x7f17400b0530 2023-10-19T14:22:05.914 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.905+0000 7f173e59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f17400afa20 0x7f17401380e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:05.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.905+0000 7f173e59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f17400afa20 0x7f17401380e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51480/0 (socket says 172.21.15.57:51480) 2023-10-19T14:22:05.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.905+0000 7f173e59c640 1 -- 172.21.15.57:0/3073121051 learned_addr learned my addr 172.21.15.57:0/3073121051 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:22:05.915 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.905+0000 7f173ed9d640 1 --2- 172.21.15.57:0/3073121051 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f17400cbd20 0x7f1740144da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:05.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.905+0000 7f1735d9b640 1 --2- 172.21.15.57:0/3073121051 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f17400b0530 0x7f1740144860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:05.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.905+0000 7f173e59c640 1 -- 172.21.15.57:0/3073121051 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f17400b0530 msgr2=0x7f1740144860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:05.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.905+0000 7f173e59c640 1 --2- 172.21.15.57:0/3073121051 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f17400b0530 0x7f1740144860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:05.916 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.905+0000 7f173e59c640 1 -- 172.21.15.57:0/3073121051 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f17400cbd20 msgr2=0x7f1740144da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:05.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.905+0000 7f173e59c640 1 --2- 172.21.15.57:0/3073121051 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f17400cbd20 0x7f1740144da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:05.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.905+0000 7f173e59c640 1 -- 172.21.15.57:0/3073121051 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1730009560 con 0x7f17400afa20 2023-10-19T14:22:05.917 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.905+0000 7f173ed9d640 1 --2- 172.21.15.57:0/3073121051 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f17400cbd20 0x7f1740144da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:22:05.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.906+0000 7f173e59c640 1 --2- 172.21.15.57:0/3073121051 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f17400afa20 0x7f17401380e0 secure :-1 s=READY pgs=2890 cs=0 l=1 rev1=1 crypto rx=0x7f17300099f0 tx=0x7f173002dd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:05.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.906+0000 7f17377fe640 1 -- 172.21.15.57:0/3073121051 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1730005650 con 0x7f17400afa20 2023-10-19T14:22:05.918 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.906+0000 7f17377fe640 1 -- 172.21.15.57:0/3073121051 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f17300038f0 con 0x7f17400afa20 2023-10-19T14:22:05.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.906+0000 7f174498e640 1 -- 172.21.15.57:0/3073121051 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1740070cf0 con 0x7f17400afa20 2023-10-19T14:22:05.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.906+0000 7f17377fe640 1 -- 172.21.15.57:0/3073121051 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1730003d00 con 0x7f17400afa20 2023-10-19T14:22:05.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.906+0000 7f174498e640 1 -- 172.21.15.57:0/3073121051 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1740145b10 con 0x7f17400afa20 2023-10-19T14:22:05.919 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.907+0000 7f174498e640 1 -- 172.21.15.57:0/3073121051 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17400690e0 con 0x7f17400afa20 2023-10-19T14:22:05.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.907+0000 7f17377fe640 1 -- 172.21.15.57:0/3073121051 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 145) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f173003e450 con 0x7f17400afa20 2023-10-19T14:22:05.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.908+0000 7f17377fe640 1 --2- 172.21.15.57:0/3073121051 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f172006b080 0x7f172006d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:05.920 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.908+0000 7f17377fe640 1 -- 172.21.15.57:0/3073121051 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(263..263 src has 1..263) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f17300ac4e0 con 0x7f17400afa20 2023-10-19T14:22:05.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.908+0000 7f1735d9b640 1 --2- 172.21.15.57:0/3073121051 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f172006b080 0x7f172006d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:05.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.908+0000 7f1735d9b640 1 --2- 172.21.15.57:0/3073121051 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f172006b080 0x7f172006d540 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f1740149730 tx=0x7f172800a4c0 comp rx=0 tx=0).ready entity=mgr.11446 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:05.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:05.912+0000 7f17377fe640 1 -- 172.21.15.57:0/3073121051 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f1730077220 con 0x7f17400afa20 2023-10-19T14:22:06.087 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.085+0000 7f174498e640 1 -- 172.21.15.57:0/3073121051 --> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f17400cc1e0 con 0x7f172006b080 2023-10-19T14:22:06.095 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.093+0000 7f17377fe640 1 -- 172.21.15.57:0/3073121051 <== mgr.11446 v2:172.21.15.179:6832/1573806521 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+288853 (secure 0 0 0) 0x7f17400cc1e0 con 0x7f172006b080 2023-10-19T14:22:06.098 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:22:06.100 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2023-10-19T14:22:06.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.096+0000 7f174498e640 1 -- 172.21.15.57:0/3073121051 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f172006b080 msgr2=0x7f172006d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:06.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.096+0000 7f174498e640 1 --2- 172.21.15.57:0/3073121051 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f172006b080 0x7f172006d540 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f1740149730 tx=0x7f172800a4c0 comp rx=0 tx=0).stop 2023-10-19T14:22:06.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.096+0000 7f174498e640 1 -- 172.21.15.57:0/3073121051 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f17400afa20 msgr2=0x7f17401380e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:06.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.096+0000 7f174498e640 1 --2- 172.21.15.57:0/3073121051 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f17400afa20 0x7f17401380e0 secure :-1 s=READY pgs=2890 cs=0 l=1 rev1=1 crypto rx=0x7f17300099f0 tx=0x7f173002dd60 comp rx=0 tx=0).stop 2023-10-19T14:22:06.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.096+0000 7f174498e640 1 -- 172.21.15.57:0/3073121051 shutdown_connections 2023-10-19T14:22:06.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.096+0000 7f174498e640 1 --2- 172.21.15.57:0/3073121051 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f172006b080 0x7f172006d540 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:06.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.096+0000 7f174498e640 1 --2- 172.21.15.57:0/3073121051 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f17400cbd20 0x7f1740144da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:06.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.096+0000 7f174498e640 1 --2- 172.21.15.57:0/3073121051 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f17400b0530 0x7f1740144860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:06.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.096+0000 7f174498e640 1 --2- 172.21.15.57:0/3073121051 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f17400afa20 0x7f17401380e0 unknown :-1 s=CLOSED pgs=2890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:06.103 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.096+0000 7f174498e640 1 -- 172.21.15.57:0/3073121051 >> 172.21.15.57:0/3073121051 conn(0x7f1740069930 msgr2=0x7f174014d250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:06.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.096+0000 7f174498e640 1 -- 172.21.15.57:0/3073121051 shutdown_connections 2023-10-19T14:22:06.104 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.096+0000 7f174498e640 1 -- 172.21.15.57:0/3073121051 wait complete. 2023-10-19T14:22:06.110 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":29,"stamp":"2023-10-19T14:22:05.419632+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":591838,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":2539,"num_read_kb":5339,"num_write":697,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3257,"ondisk_log_size":3257,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":40,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":452732,"kb_used_data":26948,"kb_used_omap":12,"kb_used_meta":425651,"kb_avail":754521988,"statfs":{"total":773094113280,"available":772630515712,"internally_reserved":0,"allocated":27594752,"data_stored":14813661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":435867193},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1470,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.025206"},"pg_stats":[{"pgid":"36.3f","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.407757+0000","last_change":"2023-10-19T14:22:02.407757+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.407757+0000","last_undegraded":"2023-10-19T14:22:02.407757+0000","last_fullsized":"2023-10-19T14:22:02.407757+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:14:21.361763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":["6","3"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.3e","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.406175+0000","last_change":"2023-10-19T14:22:02.406175+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.406175+0000","last_undegraded":"2023-10-19T14:22:02.406175+0000","last_fullsized":"2023-10-19T14:22:02.406175+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:23:52.735800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.3d","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.412915+0000","last_change":"2023-10-19T14:22:02.412915+0000","last_active":"2023-10-19T14:22:02.412915+0000","last_peered":"2023-10-19T14:22:02.412915+0000","last_clean":"2023-10-19T14:22:02.412915+0000","last_became_active":"2023-10-19T14:22:02.412634+0000","last_became_peered":"2023-10-19T14:22:02.412634+0000","last_unstale":"2023-10-19T14:22:02.412915+0000","last_undegraded":"2023-10-19T14:22:02.412915+0000","last_fullsized":"2023-10-19T14:22:02.412915+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:20:32.428977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.3c","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.403761+0000","last_change":"2023-10-19T14:22:02.403761+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.403761+0000","last_undegraded":"2023-10-19T14:22:02.403761+0000","last_fullsized":"2023-10-19T14:22:02.403761+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:50:46.770884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.3b","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.409103+0000","last_change":"2023-10-19T14:22:02.409103+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.409103+0000","last_undegraded":"2023-10-19T14:22:02.409103+0000","last_fullsized":"2023-10-19T14:22:02.409103+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:03:49.877672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.3a","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.408280+0000","last_change":"2023-10-19T14:22:02.408280+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.408280+0000","last_undegraded":"2023-10-19T14:22:02.408280+0000","last_fullsized":"2023-10-19T14:22:02.408280+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:18:59.130442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.39","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.413984+0000","last_change":"2023-10-19T14:22:02.413984+0000","last_active":"2023-10-19T14:22:02.413984+0000","last_peered":"2023-10-19T14:22:02.413984+0000","last_clean":"2023-10-19T14:22:02.413984+0000","last_became_active":"2023-10-19T14:22:02.413714+0000","last_became_peered":"2023-10-19T14:22:02.413714+0000","last_unstale":"2023-10-19T14:22:02.413984+0000","last_undegraded":"2023-10-19T14:22:02.413984+0000","last_fullsized":"2023-10-19T14:22:02.413984+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:03:55.524277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.38","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.403768+0000","last_change":"2023-10-19T14:22:02.403768+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.403768+0000","last_undegraded":"2023-10-19T14:22:02.403768+0000","last_fullsized":"2023-10-19T14:22:02.403768+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:30:01.392073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.37","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.404303+0000","last_change":"2023-10-19T14:22:02.404303+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.404303+0000","last_undegraded":"2023-10-19T14:22:02.404303+0000","last_fullsized":"2023-10-19T14:22:02.404303+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:16:24.272266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.36","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.413432+0000","last_change":"2023-10-19T14:22:02.413432+0000","last_active":"2023-10-19T14:22:02.413432+0000","last_peered":"2023-10-19T14:22:02.413432+0000","last_clean":"2023-10-19T14:22:02.413432+0000","last_became_active":"2023-10-19T14:22:02.412808+0000","last_became_peered":"2023-10-19T14:22:02.412808+0000","last_unstale":"2023-10-19T14:22:02.413432+0000","last_undegraded":"2023-10-19T14:22:02.413432+0000","last_fullsized":"2023-10-19T14:22:02.413432+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T02:11:11.663612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.35","version":"0'0","reported_seq":9,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.418007+0000","last_change":"2023-10-19T14:22:02.413336+0000","last_active":"2023-10-19T14:22:02.418007+0000","last_peered":"2023-10-19T14:22:02.418007+0000","last_clean":"2023-10-19T14:22:02.418007+0000","last_became_active":"2023-10-19T14:22:02.412942+0000","last_became_peered":"2023-10-19T14:22:02.412942+0000","last_unstale":"2023-10-19T14:22:02.418007+0000","last_undegraded":"2023-10-19T14:22:02.418007+0000","last_fullsized":"2023-10-19T14:22:02.418007+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:37:40.908777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.34","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.406632+0000","last_change":"2023-10-19T14:22:02.406632+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.406632+0000","last_undegraded":"2023-10-19T14:22:02.406632+0000","last_fullsized":"2023-10-19T14:22:02.406632+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:30:39.654226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":["6","3"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.33","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.403708+0000","last_change":"2023-10-19T14:22:02.403708+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.403708+0000","last_undegraded":"2023-10-19T14:22:02.403708+0000","last_fullsized":"2023-10-19T14:22:02.403708+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:39:38.146246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.32","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.403725+0000","last_change":"2023-10-19T14:22:02.403725+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.403725+0000","last_undegraded":"2023-10-19T14:22:02.403725+0000","last_fullsized":"2023-10-19T14:22:02.403725+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:10:47.778168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.31","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.403277+0000","last_change":"2023-10-19T14:22:02.403277+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.403277+0000","last_undegraded":"2023-10-19T14:22:02.403277+0000","last_fullsized":"2023-10-19T14:22:02.403277+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:15:36.803736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.30","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.420492+0000","last_change":"2023-10-19T14:22:02.420492+0000","last_active":"2023-10-19T14:22:02.420492+0000","last_peered":"2023-10-19T14:22:02.420492+0000","last_clean":"2023-10-19T14:22:02.420492+0000","last_became_active":"2023-10-19T14:22:02.420210+0000","last_became_peered":"2023-10-19T14:22:02.420210+0000","last_unstale":"2023-10-19T14:22:02.420492+0000","last_undegraded":"2023-10-19T14:22:02.420492+0000","last_fullsized":"2023-10-19T14:22:02.420492+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:38:11.628588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.2f","version":"0'0","reported_seq":9,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.420525+0000","last_change":"2023-10-19T14:22:02.415526+0000","last_active":"2023-10-19T14:22:02.420525+0000","last_peered":"2023-10-19T14:22:02.420525+0000","last_clean":"2023-10-19T14:22:02.420525+0000","last_became_active":"2023-10-19T14:22:02.415243+0000","last_became_peered":"2023-10-19T14:22:02.415243+0000","last_unstale":"2023-10-19T14:22:02.420525+0000","last_undegraded":"2023-10-19T14:22:02.420525+0000","last_fullsized":"2023-10-19T14:22:02.420525+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:25:08.178761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.2e","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.405331+0000","last_change":"2023-10-19T14:22:02.405331+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.405331+0000","last_undegraded":"2023-10-19T14:22:02.405331+0000","last_fullsized":"2023-10-19T14:22:02.405331+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:49:42.930618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.2d","version":"0'0","reported_seq":9,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.420730+0000","last_change":"2023-10-19T14:22:02.418292+0000","last_active":"2023-10-19T14:22:02.420730+0000","last_peered":"2023-10-19T14:22:02.420730+0000","last_clean":"2023-10-19T14:22:02.420730+0000","last_became_active":"2023-10-19T14:22:02.415107+0000","last_became_peered":"2023-10-19T14:22:02.415107+0000","last_unstale":"2023-10-19T14:22:02.420730+0000","last_undegraded":"2023-10-19T14:22:02.420730+0000","last_fullsized":"2023-10-19T14:22:02.420730+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:22:21.202909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.2c","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.421040+0000","last_change":"2023-10-19T14:22:02.421040+0000","last_active":"2023-10-19T14:22:02.421040+0000","last_peered":"2023-10-19T14:22:02.421040+0000","last_clean":"2023-10-19T14:22:02.421040+0000","last_became_active":"2023-10-19T14:22:02.420785+0000","last_became_peered":"2023-10-19T14:22:02.420785+0000","last_unstale":"2023-10-19T14:22:02.421040+0000","last_undegraded":"2023-10-19T14:22:02.421040+0000","last_fullsized":"2023-10-19T14:22:02.421040+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:48:32.972469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.2b","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.418617+0000","last_change":"2023-10-19T14:22:02.418617+0000","last_active":"2023-10-19T14:22:02.418617+0000","last_peered":"2023-10-19T14:22:02.418617+0000","last_clean":"2023-10-19T14:22:02.418617+0000","last_became_active":"2023-10-19T14:22:02.418348+0000","last_became_peered":"2023-10-19T14:22:02.418348+0000","last_unstale":"2023-10-19T14:22:02.418617+0000","last_undegraded":"2023-10-19T14:22:02.418617+0000","last_fullsized":"2023-10-19T14:22:02.418617+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:03:48.649163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.2a","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.399641+0000","last_change":"2023-10-19T14:22:02.399641+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.399641+0000","last_undegraded":"2023-10-19T14:22:02.399641+0000","last_fullsized":"2023-10-19T14:22:02.399641+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:35:08.037582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.29","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.405143+0000","last_change":"2023-10-19T14:22:02.405143+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.405143+0000","last_undegraded":"2023-10-19T14:22:02.405143+0000","last_fullsized":"2023-10-19T14:22:02.405143+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:09:54.096828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.28","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.398717+0000","last_change":"2023-10-19T14:22:02.398717+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.398717+0000","last_undegraded":"2023-10-19T14:22:02.398717+0000","last_fullsized":"2023-10-19T14:22:02.398717+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:35:36.604007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.27","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.417381+0000","last_change":"2023-10-19T14:22:02.417381+0000","last_active":"2023-10-19T14:22:02.417381+0000","last_peered":"2023-10-19T14:22:02.417381+0000","last_clean":"2023-10-19T14:22:02.417381+0000","last_became_active":"2023-10-19T14:22:02.417103+0000","last_became_peered":"2023-10-19T14:22:02.417103+0000","last_unstale":"2023-10-19T14:22:02.417381+0000","last_undegraded":"2023-10-19T14:22:02.417381+0000","last_fullsized":"2023-10-19T14:22:02.417381+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:05:25.934207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.26","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.410522+0000","last_change":"2023-10-19T14:22:02.410522+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.410522+0000","last_undegraded":"2023-10-19T14:22:02.410522+0000","last_fullsized":"2023-10-19T14:22:02.410522+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:41:53.947719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.24","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.407514+0000","last_change":"2023-10-19T14:22:02.407514+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.407514+0000","last_undegraded":"2023-10-19T14:22:02.407514+0000","last_fullsized":"2023-10-19T14:22:02.407514+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:14:37.969686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.23","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.398963+0000","last_change":"2023-10-19T14:22:02.398963+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.398963+0000","last_undegraded":"2023-10-19T14:22:02.398963+0000","last_fullsized":"2023-10-19T14:22:02.398963+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:57:49.145921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.22","version":"0'0","reported_seq":9,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.404880+0000","last_change":"2023-10-19T14:22:02.402715+0000","last_active":"2023-10-19T14:22:02.404880+0000","last_peered":"2023-10-19T14:22:02.404880+0000","last_clean":"2023-10-19T14:22:02.404880+0000","last_became_active":"2023-10-19T14:22:02.402451+0000","last_became_peered":"2023-10-19T14:22:02.402451+0000","last_unstale":"2023-10-19T14:22:02.404880+0000","last_undegraded":"2023-10-19T14:22:02.404880+0000","last_fullsized":"2023-10-19T14:22:02.404880+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:42:44.848665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.21","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.399356+0000","last_change":"2023-10-19T14:22:02.399356+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.399356+0000","last_undegraded":"2023-10-19T14:22:02.399356+0000","last_fullsized":"2023-10-19T14:22:02.399356+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:45:44.957902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.20","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.403114+0000","last_change":"2023-10-19T14:22:02.403114+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.403114+0000","last_undegraded":"2023-10-19T14:22:02.403114+0000","last_fullsized":"2023-10-19T14:22:02.403114+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T02:18:23.251534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.18","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.398614+0000","last_change":"2023-10-19T14:22:02.398614+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.398614+0000","last_undegraded":"2023-10-19T14:22:02.398614+0000","last_fullsized":"2023-10-19T14:22:02.398614+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:31:51.736865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.1f","version":"262'4","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.404439+0000","last_change":"2023-10-19T14:22:01.414254+0000","last_active":"2023-10-19T14:22:02.404439+0000","last_peered":"2023-10-19T14:22:02.404439+0000","last_clean":"2023-10-19T14:22:02.404439+0000","last_became_active":"2023-10-19T14:22:01.413084+0000","last_became_peered":"2023-10-19T14:22:01.413084+0000","last_unstale":"2023-10-19T14:22:02.404439+0000","last_undegraded":"2023-10-19T14:22:02.404439+0000","last_fullsized":"2023-10-19T14:22:02.404439+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:04:17.277980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.19","version":"0'0","reported_seq":9,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.416536+0000","last_change":"2023-10-19T14:22:02.412925+0000","last_active":"2023-10-19T14:22:02.416536+0000","last_peered":"2023-10-19T14:22:02.416536+0000","last_clean":"2023-10-19T14:22:02.416536+0000","last_became_active":"2023-10-19T14:22:02.412639+0000","last_became_peered":"2023-10-19T14:22:02.412639+0000","last_unstale":"2023-10-19T14:22:02.416536+0000","last_undegraded":"2023-10-19T14:22:02.416536+0000","last_fullsized":"2023-10-19T14:22:02.416536+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:14:38.697589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.1e","version":"0'0","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.399308+0000","last_change":"2023-10-19T14:22:01.410546+0000","last_active":"2023-10-19T14:22:02.399308+0000","last_peered":"2023-10-19T14:22:02.399308+0000","last_clean":"2023-10-19T14:22:02.399308+0000","last_became_active":"2023-10-19T14:22:01.410263+0000","last_became_peered":"2023-10-19T14:22:01.410263+0000","last_unstale":"2023-10-19T14:22:02.399308+0000","last_undegraded":"2023-10-19T14:22:02.399308+0000","last_fullsized":"2023-10-19T14:22:02.399308+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:02:12.137179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.1a","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.406083+0000","last_change":"2023-10-19T14:22:02.406083+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.406083+0000","last_undegraded":"2023-10-19T14:22:02.406083+0000","last_fullsized":"2023-10-19T14:22:02.406083+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:07:50.835228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.1d","version":"262'2","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.400756+0000","last_change":"2023-10-19T14:22:01.417975+0000","last_active":"2023-10-19T14:22:02.400756+0000","last_peered":"2023-10-19T14:22:02.400756+0000","last_clean":"2023-10-19T14:22:02.400756+0000","last_became_active":"2023-10-19T14:22:01.417704+0000","last_became_peered":"2023-10-19T14:22:01.417704+0000","last_unstale":"2023-10-19T14:22:02.400756+0000","last_undegraded":"2023-10-19T14:22:02.400756+0000","last_fullsized":"2023-10-19T14:22:02.400756+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:56:00.152228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.1b","version":"0'0","reported_seq":9,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.416951+0000","last_change":"2023-10-19T14:22:02.412219+0000","last_active":"2023-10-19T14:22:02.416951+0000","last_peered":"2023-10-19T14:22:02.416951+0000","last_clean":"2023-10-19T14:22:02.416951+0000","last_became_active":"2023-10-19T14:22:02.411959+0000","last_became_peered":"2023-10-19T14:22:02.411959+0000","last_unstale":"2023-10-19T14:22:02.416951+0000","last_undegraded":"2023-10-19T14:22:02.416951+0000","last_fullsized":"2023-10-19T14:22:02.416951+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:54:56.271440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.1c","version":"262'2","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.400340+0000","last_change":"2023-10-19T14:22:01.419018+0000","last_active":"2023-10-19T14:22:02.400340+0000","last_peered":"2023-10-19T14:22:02.400340+0000","last_clean":"2023-10-19T14:22:02.400340+0000","last_became_active":"2023-10-19T14:22:01.418769+0000","last_became_peered":"2023-10-19T14:22:01.418769+0000","last_unstale":"2023-10-19T14:22:02.400340+0000","last_undegraded":"2023-10-19T14:22:02.400340+0000","last_fullsized":"2023-10-19T14:22:02.400340+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:59:43.263869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.1c","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.402955+0000","last_change":"2023-10-19T14:22:02.402955+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.402955+0000","last_undegraded":"2023-10-19T14:22:02.402955+0000","last_fullsized":"2023-10-19T14:22:02.402955+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:03:27.890218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.1b","version":"262'1","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.409720+0000","last_change":"2023-10-19T14:22:01.413953+0000","last_active":"2023-10-19T14:22:02.409720+0000","last_peered":"2023-10-19T14:22:02.409720+0000","last_clean":"2023-10-19T14:22:02.409720+0000","last_became_active":"2023-10-19T14:22:01.413689+0000","last_became_peered":"2023-10-19T14:22:01.413689+0000","last_unstale":"2023-10-19T14:22:02.409720+0000","last_undegraded":"2023-10-19T14:22:02.409720+0000","last_fullsized":"2023-10-19T14:22:02.409720+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:33:45.430960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.1d","version":"0'0","reported_seq":6,"reported_epoch":262,"state":"active","last_fresh":"2023-10-19T14:22:02.413928+0000","last_change":"2023-10-19T14:22:02.413862+0000","last_active":"2023-10-19T14:22:02.413928+0000","last_peered":"2023-10-19T14:22:02.413928+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"2023-10-19T14:22:02.413862+0000","last_became_peered":"2023-10-19T14:22:02.413862+0000","last_unstale":"2023-10-19T14:22:02.413928+0000","last_undegraded":"2023-10-19T14:22:02.413928+0000","last_fullsized":"2023-10-19T14:22:02.413928+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:20:55.394812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.1a","version":"0'0","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.408577+0000","last_change":"2023-10-19T14:22:01.409991+0000","last_active":"2023-10-19T14:22:02.408577+0000","last_peered":"2023-10-19T14:22:02.408577+0000","last_clean":"2023-10-19T14:22:02.408577+0000","last_became_active":"2023-10-19T14:22:01.409510+0000","last_became_peered":"2023-10-19T14:22:01.409510+0000","last_unstale":"2023-10-19T14:22:02.408577+0000","last_undegraded":"2023-10-19T14:22:02.408577+0000","last_fullsized":"2023-10-19T14:22:02.408577+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:42:53.919625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.1e","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.412327+0000","last_change":"2023-10-19T14:22:02.412327+0000","last_active":"2023-10-19T14:22:02.412327+0000","last_peered":"2023-10-19T14:22:02.412327+0000","last_clean":"2023-10-19T14:22:02.412327+0000","last_became_active":"2023-10-19T14:22:02.412057+0000","last_became_peered":"2023-10-19T14:22:02.412057+0000","last_unstale":"2023-10-19T14:22:02.412327+0000","last_undegraded":"2023-10-19T14:22:02.412327+0000","last_fullsized":"2023-10-19T14:22:02.412327+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:32:30.545638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.19","version":"0'0","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.399288+0000","last_change":"2023-10-19T14:22:01.416901+0000","last_active":"2023-10-19T14:22:02.399288+0000","last_peered":"2023-10-19T14:22:02.399288+0000","last_clean":"2023-10-19T14:22:02.399288+0000","last_became_active":"2023-10-19T14:22:01.416656+0000","last_became_peered":"2023-10-19T14:22:01.416656+0000","last_unstale":"2023-10-19T14:22:02.399288+0000","last_undegraded":"2023-10-19T14:22:02.399288+0000","last_fullsized":"2023-10-19T14:22:02.399288+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:27:36.174418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.1f","version":"0'0","reported_seq":9,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.420545+0000","last_change":"2023-10-19T14:22:02.418425+0000","last_active":"2023-10-19T14:22:02.420545+0000","last_peered":"2023-10-19T14:22:02.420545+0000","last_clean":"2023-10-19T14:22:02.420545+0000","last_became_active":"2023-10-19T14:22:02.417278+0000","last_became_peered":"2023-10-19T14:22:02.417278+0000","last_unstale":"2023-10-19T14:22:02.420545+0000","last_undegraded":"2023-10-19T14:22:02.420545+0000","last_fullsized":"2023-10-19T14:22:02.420545+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:04:24.989932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.18","version":"0'0","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.404196+0000","last_change":"2023-10-19T14:22:01.412557+0000","last_active":"2023-10-19T14:22:02.404196+0000","last_peered":"2023-10-19T14:22:02.404196+0000","last_clean":"2023-10-19T14:22:02.404196+0000","last_became_active":"2023-10-19T14:22:01.412115+0000","last_became_peered":"2023-10-19T14:22:01.412115+0000","last_unstale":"2023-10-19T14:22:02.404196+0000","last_undegraded":"2023-10-19T14:22:02.404196+0000","last_fullsized":"2023-10-19T14:22:02.404196+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:54:33.650542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.10","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.402706+0000","last_change":"2023-10-19T14:22:02.402706+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.402706+0000","last_undegraded":"2023-10-19T14:22:02.402706+0000","last_fullsized":"2023-10-19T14:22:02.402706+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:45:46.279692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.17","version":"0'0","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.415002+0000","last_change":"2023-10-19T14:22:01.412962+0000","last_active":"2023-10-19T14:22:02.415002+0000","last_peered":"2023-10-19T14:22:02.415002+0000","last_clean":"2023-10-19T14:22:02.415002+0000","last_became_active":"2023-10-19T14:22:01.412680+0000","last_became_peered":"2023-10-19T14:22:01.412680+0000","last_unstale":"2023-10-19T14:22:02.415002+0000","last_undegraded":"2023-10-19T14:22:02.415002+0000","last_fullsized":"2023-10-19T14:22:02.415002+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:03:54.917148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.11","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.405750+0000","last_change":"2023-10-19T14:22:02.405750+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.405750+0000","last_undegraded":"2023-10-19T14:22:02.405750+0000","last_fullsized":"2023-10-19T14:22:02.405750+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:06:30.359305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.16","version":"0'0","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.399221+0000","last_change":"2023-10-19T14:22:01.414464+0000","last_active":"2023-10-19T14:22:02.399221+0000","last_peered":"2023-10-19T14:22:02.399221+0000","last_clean":"2023-10-19T14:22:02.399221+0000","last_became_active":"2023-10-19T14:22:01.413414+0000","last_became_peered":"2023-10-19T14:22:01.413414+0000","last_unstale":"2023-10-19T14:22:02.399221+0000","last_undegraded":"2023-10-19T14:22:02.399221+0000","last_fullsized":"2023-10-19T14:22:02.399221+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:46:22.763844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.12","version":"0'0","reported_seq":9,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.413606+0000","last_change":"2023-10-19T14:22:02.410248+0000","last_active":"2023-10-19T14:22:02.413606+0000","last_peered":"2023-10-19T14:22:02.413606+0000","last_clean":"2023-10-19T14:22:02.413606+0000","last_became_active":"2023-10-19T14:22:02.409982+0000","last_became_peered":"2023-10-19T14:22:02.409982+0000","last_unstale":"2023-10-19T14:22:02.413606+0000","last_undegraded":"2023-10-19T14:22:02.413606+0000","last_fullsized":"2023-10-19T14:22:02.413606+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:27:43.397970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.15","version":"262'3","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.414741+0000","last_change":"2023-10-19T14:22:01.409543+0000","last_active":"2023-10-19T14:22:02.414741+0000","last_peered":"2023-10-19T14:22:02.414741+0000","last_clean":"2023-10-19T14:22:02.414741+0000","last_became_active":"2023-10-19T14:22:01.409293+0000","last_became_peered":"2023-10-19T14:22:01.409293+0000","last_unstale":"2023-10-19T14:22:02.414741+0000","last_undegraded":"2023-10-19T14:22:02.414741+0000","last_fullsized":"2023-10-19T14:22:02.414741+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:12:16.850155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.13","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.408938+0000","last_change":"2023-10-19T14:22:02.408938+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.408938+0000","last_undegraded":"2023-10-19T14:22:02.408938+0000","last_fullsized":"2023-10-19T14:22:02.408938+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:22:40.195315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.14","version":"262'5","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.404183+0000","last_change":"2023-10-19T14:22:01.411386+0000","last_active":"2023-10-19T14:22:02.404183+0000","last_peered":"2023-10-19T14:22:02.404183+0000","last_clean":"2023-10-19T14:22:02.404183+0000","last_became_active":"2023-10-19T14:22:01.411111+0000","last_became_peered":"2023-10-19T14:22:01.411111+0000","last_unstale":"2023-10-19T14:22:02.404183+0000","last_undegraded":"2023-10-19T14:22:02.404183+0000","last_fullsized":"2023-10-19T14:22:02.404183+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:10:26.534502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.14","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.416878+0000","last_change":"2023-10-19T14:22:02.416878+0000","last_active":"2023-10-19T14:22:02.416878+0000","last_peered":"2023-10-19T14:22:02.416878+0000","last_clean":"2023-10-19T14:22:02.416878+0000","last_became_active":"2023-10-19T14:22:02.416633+0000","last_became_peered":"2023-10-19T14:22:02.416633+0000","last_unstale":"2023-10-19T14:22:02.416878+0000","last_undegraded":"2023-10-19T14:22:02.416878+0000","last_fullsized":"2023-10-19T14:22:02.416878+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:56:41.835345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.13","version":"262'2","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.399368+0000","last_change":"2023-10-19T14:22:01.410762+0000","last_active":"2023-10-19T14:22:02.399368+0000","last_peered":"2023-10-19T14:22:02.399368+0000","last_clean":"2023-10-19T14:22:02.399368+0000","last_became_active":"2023-10-19T14:22:01.410507+0000","last_became_peered":"2023-10-19T14:22:01.410507+0000","last_unstale":"2023-10-19T14:22:02.399368+0000","last_undegraded":"2023-10-19T14:22:02.399368+0000","last_fullsized":"2023-10-19T14:22:02.399368+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:38:13.918889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.15","version":"0'0","reported_seq":6,"reported_epoch":262,"state":"active","last_fresh":"2023-10-19T14:22:02.414429+0000","last_change":"2023-10-19T14:22:02.414358+0000","last_active":"2023-10-19T14:22:02.414429+0000","last_peered":"2023-10-19T14:22:02.414429+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"2023-10-19T14:22:02.414358+0000","last_became_peered":"2023-10-19T14:22:02.414358+0000","last_unstale":"2023-10-19T14:22:02.414429+0000","last_undegraded":"2023-10-19T14:22:02.414429+0000","last_fullsized":"2023-10-19T14:22:02.414429+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:02:05.922124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.12","version":"0'0","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.409323+0000","last_change":"2023-10-19T14:22:01.413870+0000","last_active":"2023-10-19T14:22:02.409323+0000","last_peered":"2023-10-19T14:22:02.409323+0000","last_clean":"2023-10-19T14:22:02.409323+0000","last_became_active":"2023-10-19T14:22:01.413532+0000","last_became_peered":"2023-10-19T14:22:01.413532+0000","last_unstale":"2023-10-19T14:22:02.409323+0000","last_undegraded":"2023-10-19T14:22:02.409323+0000","last_fullsized":"2023-10-19T14:22:02.409323+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:47:48.522113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.16","version":"0'0","reported_seq":9,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.421387+0000","last_change":"2023-10-19T14:22:02.419152+0000","last_active":"2023-10-19T14:22:02.421387+0000","last_peered":"2023-10-19T14:22:02.421387+0000","last_clean":"2023-10-19T14:22:02.421387+0000","last_became_active":"2023-10-19T14:22:02.417515+0000","last_became_peered":"2023-10-19T14:22:02.417515+0000","last_unstale":"2023-10-19T14:22:02.421387+0000","last_undegraded":"2023-10-19T14:22:02.421387+0000","last_fullsized":"2023-10-19T14:22:02.421387+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:39:40.912960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.11","version":"0'0","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.409983+0000","last_change":"2023-10-19T14:22:01.416474+0000","last_active":"2023-10-19T14:22:02.409983+0000","last_peered":"2023-10-19T14:22:02.409983+0000","last_clean":"2023-10-19T14:22:02.409983+0000","last_became_active":"2023-10-19T14:22:01.416200+0000","last_became_peered":"2023-10-19T14:22:01.416200+0000","last_unstale":"2023-10-19T14:22:02.409983+0000","last_undegraded":"2023-10-19T14:22:02.409983+0000","last_fullsized":"2023-10-19T14:22:02.409983+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:55:39.137949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.17","version":"0'0","reported_seq":9,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.421450+0000","last_change":"2023-10-19T14:22:02.419116+0000","last_active":"2023-10-19T14:22:02.421450+0000","last_peered":"2023-10-19T14:22:02.421450+0000","last_clean":"2023-10-19T14:22:02.421450+0000","last_became_active":"2023-10-19T14:22:02.418655+0000","last_became_peered":"2023-10-19T14:22:02.418655+0000","last_unstale":"2023-10-19T14:22:02.421450+0000","last_undegraded":"2023-10-19T14:22:02.421450+0000","last_fullsized":"2023-10-19T14:22:02.421450+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:28:48.304947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.10","version":"262'8","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.402744+0000","last_change":"2023-10-19T14:22:01.415486+0000","last_active":"2023-10-19T14:22:02.402744+0000","last_peered":"2023-10-19T14:22:02.402744+0000","last_clean":"2023-10-19T14:22:02.402744+0000","last_became_active":"2023-10-19T14:22:01.415192+0000","last_became_peered":"2023-10-19T14:22:01.415192+0000","last_unstale":"2023-10-19T14:22:02.402744+0000","last_undegraded":"2023-10-19T14:22:02.402744+0000","last_fullsized":"2023-10-19T14:22:02.402744+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:45:03.326043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.8","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.407065+0000","last_change":"2023-10-19T14:22:02.407065+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.407065+0000","last_undegraded":"2023-10-19T14:22:02.407065+0000","last_fullsized":"2023-10-19T14:22:02.407065+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:07:55.991807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.f","version":"262'1","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.417942+0000","last_change":"2023-10-19T14:22:01.417521+0000","last_active":"2023-10-19T14:22:02.417942+0000","last_peered":"2023-10-19T14:22:02.417942+0000","last_clean":"2023-10-19T14:22:02.417942+0000","last_became_active":"2023-10-19T14:22:01.417209+0000","last_became_peered":"2023-10-19T14:22:01.417209+0000","last_unstale":"2023-10-19T14:22:02.417942+0000","last_undegraded":"2023-10-19T14:22:02.417942+0000","last_fullsized":"2023-10-19T14:22:02.417942+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:43:26.494066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.9","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.398780+0000","last_change":"2023-10-19T14:22:02.398780+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.398780+0000","last_undegraded":"2023-10-19T14:22:02.398780+0000","last_fullsized":"2023-10-19T14:22:02.398780+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:40:53.511724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.e","version":"0'0","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.411175+0000","last_change":"2023-10-19T14:22:01.412531+0000","last_active":"2023-10-19T14:22:02.411175+0000","last_peered":"2023-10-19T14:22:02.411175+0000","last_clean":"2023-10-19T14:22:02.411175+0000","last_became_active":"2023-10-19T14:22:01.412209+0000","last_became_peered":"2023-10-19T14:22:01.412209+0000","last_unstale":"2023-10-19T14:22:02.411175+0000","last_undegraded":"2023-10-19T14:22:02.411175+0000","last_fullsized":"2023-10-19T14:22:02.411175+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:45:03.326043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.a","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.407858+0000","last_change":"2023-10-19T14:22:02.407858+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.407858+0000","last_undegraded":"2023-10-19T14:22:02.407858+0000","last_fullsized":"2023-10-19T14:22:02.407858+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:31:43.355728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.d","version":"262'1","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.400892+0000","last_change":"2023-10-19T14:22:01.415741+0000","last_active":"2023-10-19T14:22:02.400892+0000","last_peered":"2023-10-19T14:22:02.400892+0000","last_clean":"2023-10-19T14:22:02.400892+0000","last_became_active":"2023-10-19T14:22:01.415385+0000","last_became_peered":"2023-10-19T14:22:01.415385+0000","last_unstale":"2023-10-19T14:22:02.400892+0000","last_undegraded":"2023-10-19T14:22:02.400892+0000","last_fullsized":"2023-10-19T14:22:02.400892+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:34:06.962782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.b","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.413768+0000","last_change":"2023-10-19T14:22:02.413768+0000","last_active":"2023-10-19T14:22:02.413768+0000","last_peered":"2023-10-19T14:22:02.413768+0000","last_clean":"2023-10-19T14:22:02.413768+0000","last_became_active":"2023-10-19T14:22:02.413498+0000","last_became_peered":"2023-10-19T14:22:02.413498+0000","last_unstale":"2023-10-19T14:22:02.413768+0000","last_undegraded":"2023-10-19T14:22:02.413768+0000","last_fullsized":"2023-10-19T14:22:02.413768+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:09:01.416730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.c","version":"0'0","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.399235+0000","last_change":"2023-10-19T14:22:01.411380+0000","last_active":"2023-10-19T14:22:02.399235+0000","last_peered":"2023-10-19T14:22:02.399235+0000","last_clean":"2023-10-19T14:22:02.399235+0000","last_became_active":"2023-10-19T14:22:01.411092+0000","last_became_peered":"2023-10-19T14:22:01.411092+0000","last_unstale":"2023-10-19T14:22:02.399235+0000","last_undegraded":"2023-10-19T14:22:02.399235+0000","last_fullsized":"2023-10-19T14:22:02.399235+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:01:14.941331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.c","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.413516+0000","last_change":"2023-10-19T14:22:02.413516+0000","last_active":"2023-10-19T14:22:02.413516+0000","last_peered":"2023-10-19T14:22:02.413516+0000","last_clean":"2023-10-19T14:22:02.413516+0000","last_became_active":"2023-10-19T14:22:02.413230+0000","last_became_peered":"2023-10-19T14:22:02.413230+0000","last_unstale":"2023-10-19T14:22:02.413516+0000","last_undegraded":"2023-10-19T14:22:02.413516+0000","last_fullsized":"2023-10-19T14:22:02.413516+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:22:42.002875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.b","version":"262'2","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.399530+0000","last_change":"2023-10-19T14:22:01.409927+0000","last_active":"2023-10-19T14:22:02.399530+0000","last_peered":"2023-10-19T14:22:02.399530+0000","last_clean":"2023-10-19T14:22:02.399530+0000","last_became_active":"2023-10-19T14:22:01.409617+0000","last_became_peered":"2023-10-19T14:22:01.409617+0000","last_unstale":"2023-10-19T14:22:02.399530+0000","last_undegraded":"2023-10-19T14:22:02.399530+0000","last_fullsized":"2023-10-19T14:22:02.399530+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:38:30.217600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.d","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.399329+0000","last_change":"2023-10-19T14:22:02.399329+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.399329+0000","last_undegraded":"2023-10-19T14:22:02.399329+0000","last_fullsized":"2023-10-19T14:22:02.399329+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:17:41.707119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.a","version":"0'0","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.404454+0000","last_change":"2023-10-19T14:22:01.417576+0000","last_active":"2023-10-19T14:22:02.404454+0000","last_peered":"2023-10-19T14:22:02.404454+0000","last_clean":"2023-10-19T14:22:02.404454+0000","last_became_active":"2023-10-19T14:22:01.417230+0000","last_became_peered":"2023-10-19T14:22:01.417230+0000","last_unstale":"2023-10-19T14:22:02.404454+0000","last_undegraded":"2023-10-19T14:22:02.404454+0000","last_fullsized":"2023-10-19T14:22:02.404454+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:23:00.582706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.e","version":"0'0","reported_seq":9,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.421407+0000","last_change":"2023-10-19T14:22:02.419344+0000","last_active":"2023-10-19T14:22:02.421407+0000","last_peered":"2023-10-19T14:22:02.421407+0000","last_clean":"2023-10-19T14:22:02.421407+0000","last_became_active":"2023-10-19T14:22:02.417238+0000","last_became_peered":"2023-10-19T14:22:02.417238+0000","last_unstale":"2023-10-19T14:22:02.421407+0000","last_undegraded":"2023-10-19T14:22:02.421407+0000","last_fullsized":"2023-10-19T14:22:02.421407+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:19:36.485520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.9","version":"0'0","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.400018+0000","last_change":"2023-10-19T14:22:01.413457+0000","last_active":"2023-10-19T14:22:02.400018+0000","last_peered":"2023-10-19T14:22:02.400018+0000","last_clean":"2023-10-19T14:22:02.400018+0000","last_became_active":"2023-10-19T14:22:01.413179+0000","last_became_peered":"2023-10-19T14:22:01.413179+0000","last_unstale":"2023-10-19T14:22:02.400018+0000","last_undegraded":"2023-10-19T14:22:02.400018+0000","last_fullsized":"2023-10-19T14:22:02.400018+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:34:30.334044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.f","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.409049+0000","last_change":"2023-10-19T14:22:02.409049+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.409049+0000","last_undegraded":"2023-10-19T14:22:02.409049+0000","last_fullsized":"2023-10-19T14:22:02.409049+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:44:13.928436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.8","version":"0'0","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.399188+0000","last_change":"2023-10-19T14:22:01.413949+0000","last_active":"2023-10-19T14:22:02.399188+0000","last_peered":"2023-10-19T14:22:02.399188+0000","last_clean":"2023-10-19T14:22:02.399188+0000","last_became_active":"2023-10-19T14:22:01.413670+0000","last_became_peered":"2023-10-19T14:22:01.413670+0000","last_unstale":"2023-10-19T14:22:02.399188+0000","last_undegraded":"2023-10-19T14:22:02.399188+0000","last_fullsized":"2023-10-19T14:22:02.399188+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:33:35.792891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.0","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.402684+0000","last_change":"2023-10-19T14:22:02.402684+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.402684+0000","last_undegraded":"2023-10-19T14:22:02.402684+0000","last_fullsized":"2023-10-19T14:22:02.402684+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:40:48.301290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.7","version":"262'2","reported_seq":13,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.413481+0000","last_change":"2023-10-19T14:22:01.412421+0000","last_active":"2023-10-19T14:22:02.413481+0000","last_peered":"2023-10-19T14:22:02.413481+0000","last_clean":"2023-10-19T14:22:02.413481+0000","last_became_active":"2023-10-19T14:22:01.412153+0000","last_became_peered":"2023-10-19T14:22:01.412153+0000","last_unstale":"2023-10-19T14:22:02.413481+0000","last_undegraded":"2023-10-19T14:22:02.413481+0000","last_fullsized":"2023-10-19T14:22:02.413481+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:38:47.923739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.1","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.419843+0000","last_change":"2023-10-19T14:22:02.419843+0000","last_active":"2023-10-19T14:22:02.419843+0000","last_peered":"2023-10-19T14:22:02.419843+0000","last_clean":"2023-10-19T14:22:02.419843+0000","last_became_active":"2023-10-19T14:22:02.417481+0000","last_became_peered":"2023-10-19T14:22:02.417481+0000","last_unstale":"2023-10-19T14:22:02.419843+0000","last_undegraded":"2023-10-19T14:22:02.419843+0000","last_fullsized":"2023-10-19T14:22:02.419843+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:12:12.862561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.6","version":"0'0","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.409952+0000","last_change":"2023-10-19T14:22:01.412088+0000","last_active":"2023-10-19T14:22:02.409952+0000","last_peered":"2023-10-19T14:22:02.409952+0000","last_clean":"2023-10-19T14:22:02.409952+0000","last_became_active":"2023-10-19T14:22:01.411646+0000","last_became_peered":"2023-10-19T14:22:01.411646+0000","last_unstale":"2023-10-19T14:22:02.409952+0000","last_undegraded":"2023-10-19T14:22:02.409952+0000","last_fullsized":"2023-10-19T14:22:02.409952+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:48:08.338501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.2","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.403340+0000","last_change":"2023-10-19T14:22:02.403340+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.403340+0000","last_undegraded":"2023-10-19T14:22:02.403340+0000","last_fullsized":"2023-10-19T14:22:02.403340+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:42:16.203127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.5","version":"0'0","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.401015+0000","last_change":"2023-10-19T14:22:01.417906+0000","last_active":"2023-10-19T14:22:02.401015+0000","last_peered":"2023-10-19T14:22:02.401015+0000","last_clean":"2023-10-19T14:22:02.401015+0000","last_became_active":"2023-10-19T14:22:01.417614+0000","last_became_peered":"2023-10-19T14:22:01.417614+0000","last_unstale":"2023-10-19T14:22:02.401015+0000","last_undegraded":"2023-10-19T14:22:02.401015+0000","last_fullsized":"2023-10-19T14:22:02.401015+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:26:05.681833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.3","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.417066+0000","last_change":"2023-10-19T14:22:02.417066+0000","last_active":"2023-10-19T14:22:02.417066+0000","last_peered":"2023-10-19T14:22:02.417066+0000","last_clean":"2023-10-19T14:22:02.417066+0000","last_became_active":"2023-10-19T14:22:02.416770+0000","last_became_peered":"2023-10-19T14:22:02.416770+0000","last_unstale":"2023-10-19T14:22:02.417066+0000","last_undegraded":"2023-10-19T14:22:02.417066+0000","last_fullsized":"2023-10-19T14:22:02.417066+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:25:15.074214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.4","version":"0'0","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.409766+0000","last_change":"2023-10-19T14:22:01.412056+0000","last_active":"2023-10-19T14:22:02.409766+0000","last_peered":"2023-10-19T14:22:02.409766+0000","last_clean":"2023-10-19T14:22:02.409766+0000","last_became_active":"2023-10-19T14:22:01.411786+0000","last_became_peered":"2023-10-19T14:22:01.411786+0000","last_unstale":"2023-10-19T14:22:02.409766+0000","last_undegraded":"2023-10-19T14:22:02.409766+0000","last_fullsized":"2023-10-19T14:22:02.409766+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:40:51.932206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.4","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.403966+0000","last_change":"2023-10-19T14:22:02.403966+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.403966+0000","last_undegraded":"2023-10-19T14:22:02.403966+0000","last_fullsized":"2023-10-19T14:22:02.403966+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:43:57.998020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.3","version":"0'0","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.399541+0000","last_change":"2023-10-19T14:22:01.413689+0000","last_active":"2023-10-19T14:22:02.399541+0000","last_peered":"2023-10-19T14:22:02.399541+0000","last_clean":"2023-10-19T14:22:02.399541+0000","last_became_active":"2023-10-19T14:22:01.413419+0000","last_became_peered":"2023-10-19T14:22:01.413419+0000","last_unstale":"2023-10-19T14:22:02.399541+0000","last_undegraded":"2023-10-19T14:22:02.399541+0000","last_fullsized":"2023-10-19T14:22:02.399541+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:37:57.754265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.5","version":"0'0","reported_seq":9,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.414555+0000","last_change":"2023-10-19T14:22:02.409126+0000","last_active":"2023-10-19T14:22:02.414555+0000","last_peered":"2023-10-19T14:22:02.414555+0000","last_clean":"2023-10-19T14:22:02.414555+0000","last_became_active":"2023-10-19T14:22:02.408855+0000","last_became_peered":"2023-10-19T14:22:02.408855+0000","last_unstale":"2023-10-19T14:22:02.414555+0000","last_undegraded":"2023-10-19T14:22:02.414555+0000","last_fullsized":"2023-10-19T14:22:02.414555+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:03:36.116245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.2","version":"0'0","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.404648+0000","last_change":"2023-10-19T14:22:01.411905+0000","last_active":"2023-10-19T14:22:02.404648+0000","last_peered":"2023-10-19T14:22:02.404648+0000","last_clean":"2023-10-19T14:22:02.404648+0000","last_became_active":"2023-10-19T14:22:01.411584+0000","last_became_peered":"2023-10-19T14:22:01.411584+0000","last_unstale":"2023-10-19T14:22:02.404648+0000","last_undegraded":"2023-10-19T14:22:02.404648+0000","last_fullsized":"2023-10-19T14:22:02.404648+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:40:13.932927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.6","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.409726+0000","last_change":"2023-10-19T14:22:02.409726+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.409726+0000","last_undegraded":"2023-10-19T14:22:02.409726+0000","last_fullsized":"2023-10-19T14:22:02.409726+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:51:43.993546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.1","version":"0'0","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.403849+0000","last_change":"2023-10-19T14:22:01.412806+0000","last_active":"2023-10-19T14:22:02.403849+0000","last_peered":"2023-10-19T14:22:02.403849+0000","last_clean":"2023-10-19T14:22:02.403849+0000","last_became_active":"2023-10-19T14:22:01.412524+0000","last_became_peered":"2023-10-19T14:22:01.412524+0000","last_unstale":"2023-10-19T14:22:02.403849+0000","last_undegraded":"2023-10-19T14:22:02.403849+0000","last_fullsized":"2023-10-19T14:22:02.403849+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:43:54.078045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.7","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.414678+0000","last_change":"2023-10-19T14:22:02.414678+0000","last_active":"2023-10-19T14:22:02.414678+0000","last_peered":"2023-10-19T14:22:02.414678+0000","last_clean":"2023-10-19T14:22:02.414678+0000","last_became_active":"2023-10-19T14:22:02.414348+0000","last_became_peered":"2023-10-19T14:22:02.414348+0000","last_unstale":"2023-10-19T14:22:02.414678+0000","last_undegraded":"2023-10-19T14:22:02.414678+0000","last_fullsized":"2023-10-19T14:22:02.414678+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:18:02.907457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.0","version":"0'0","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.399367+0000","last_change":"2023-10-19T14:22:01.414455+0000","last_active":"2023-10-19T14:22:02.399367+0000","last_peered":"2023-10-19T14:22:02.399367+0000","last_clean":"2023-10-19T14:22:02.399367+0000","last_became_active":"2023-10-19T14:22:01.414081+0000","last_became_peered":"2023-10-19T14:22:01.414081+0000","last_unstale":"2023-10-19T14:22:02.399367+0000","last_undegraded":"2023-10-19T14:22:02.399367+0000","last_fullsized":"2023-10-19T14:22:02.399367+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:54:25.277401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.25","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.418670+0000","last_change":"2023-10-19T14:22:02.418670+0000","last_active":"2023-10-19T14:22:02.418670+0000","last_peered":"2023-10-19T14:22:02.418670+0000","last_clean":"2023-10-19T14:22:02.418670+0000","last_became_active":"2023-10-19T14:22:02.417862+0000","last_became_peered":"2023-10-19T14:22:02.417862+0000","last_unstale":"2023-10-19T14:22:02.418670+0000","last_undegraded":"2023-10-19T14:22:02.418670+0000","last_fullsized":"2023-10-19T14:22:02.418670+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:20:15.811864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"249'3224","reported_seq":4253,"reported_epoch":261,"state":"active+clean","last_fresh":"2023-10-19T14:22:01.406839+0000","last_change":"2023-10-19T12:44:08.636613+0000","last_active":"2023-10-19T14:22:01.406839+0000","last_peered":"2023-10-19T14:22:01.406839+0000","last_clean":"2023-10-19T14:22:01.406839+0000","last_became_active":"2023-10-19T12:44:08.635697+0000","last_became_peered":"2023-10-19T12:44:08.635697+0000","last_unstale":"2023-10-19T14:22:01.406839+0000","last_undegraded":"2023-10-19T14:22:01.406839+0000","last_fullsized":"2023-10-19T14:22:01.406839+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_clean_scrub_stamp":"2023-10-19T12:44:07.399738+0000","objects_scrubbed":0,"log_size":3224,"log_dups_size":0,"ondisk_log_size":3224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:15:23.685485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2539,"num_read_kb":5339,"num_write":657,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":36,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":35,"num_pg":32,"stat_sum":{"num_bytes":1470,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":33,"ondisk_log_size":33,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2539,"num_read_kb":5339,"num_write":657,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3224,"ondisk_log_size":3224,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576020,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66928,"kb_used_data":3804,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94304912,"statfs":{"total":96636764160,"available":96568229888,"internally_reserved":0,"allocated":3895296,"data_stored":2294561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":64616907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:21:10 2023","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.55100000000000005,"15min":0.57999999999999996},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.32200000000000001},"max":{"1min":0.754,"5min":0.80900000000000005,"15min":0.86199999999999999},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.503,"5min":0.57599999999999996,"15min":0.59899999999999998},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.312},"max":{"1min":0.66500000000000004,"5min":0.81899999999999995,"15min":0.86599999999999999},"last":0.39400000000000002}]},{"osd":1,"last update":"Thu Oct 19 14:21:10 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.58199999999999996,"15min":0.59499999999999997},"min":{"1min":0.45200000000000001,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.72199999999999998,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.56899999999999995,"15min":0.59299999999999997},"min":{"1min":0.39300000000000002,"5min":0.373,"15min":0.36699999999999999},"max":{"1min":0.78500000000000003,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.56200000000000006}]},{"osd":2,"last update":"Thu Oct 19 14:21:10 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.622,"15min":0.64200000000000002},"min":{"1min":0.47299999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.70499999999999996,"5min":0.84799999999999998,"15min":0.85499999999999998},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.63900000000000001,"15min":0.65700000000000003},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.54600000000000004}]},{"osd":3,"last update":"Thu Oct 19 14:21:10 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.60799999999999998,"15min":0.63100000000000001},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.628,"5min":0.67100000000000004,"15min":0.68300000000000005},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.83299999999999996,"5min":0.876,"15min":0.89600000000000002},"last":0.66700000000000004}]},{"osd":4,"last update":"Thu Oct 19 14:21:10 2023","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.50600000000000001,"15min":0.495},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.53000000000000003,"15min":0.51100000000000001},"min":{"1min":0.34999999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.505}]},{"osd":5,"last update":"Thu Oct 19 14:21:10 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52300000000000002,"15min":0.51100000000000001},"min":{"1min":0.33700000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.57799999999999996,"15min":0.55400000000000005},"min":{"1min":0.40699999999999997,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.054,"5min":1.054,"15min":1.054},"last":0.626}]},{"osd":6,"last update":"Thu Oct 19 14:21:10 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.58999999999999997,"15min":0.57599999999999996},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.35999999999999999},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.59299999999999997,"15min":0.57299999999999995},"min":{"1min":0.42999999999999999,"5min":0.35599999999999998,"15min":0.33500000000000002},"max":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.57399999999999995}]}]},{"osd":6,"up_from":13,"seq":55834576020,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56612,"kb_used_data":3224,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315228,"statfs":{"total":96636764160,"available":96578793472,"internally_reserved":0,"allocated":3301376,"data_stored":1704193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:21:02 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.59199999999999997,"15min":0.61099999999999999},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.78400000000000003,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.56299999999999994,"15min":0.60599999999999998},"min":{"1min":0.36699999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.71799999999999997,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.80200000000000005}]},{"osd":1,"last update":"Thu Oct 19 14:21:02 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.60099999999999998,"15min":0.61799999999999999},"min":{"1min":0.373,"5min":0.373,"15min":0.35799999999999998},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":1.165},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.60199999999999998,"15min":0.626},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.77400000000000002,"5min":0.83699999999999997,"15min":0.91300000000000003},"last":0.76600000000000001}]},{"osd":2,"last update":"Thu Oct 19 14:21:02 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.57599999999999996,"15min":0.60199999999999998},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.69099999999999995,"5min":0.80100000000000005,"15min":0.873},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.60599999999999998,"15min":0.624},"min":{"1min":0.40899999999999997,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.74199999999999999,"5min":0.83499999999999996,"15min":0.93100000000000005},"last":0.73599999999999999}]},{"osd":3,"last update":"Thu Oct 19 14:21:02 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.629,"15min":0.64900000000000002},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.66700000000000004,"5min":0.875,"15min":0.875},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.63800000000000001,"15min":0.65500000000000003},"min":{"1min":0.45400000000000001,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.92200000000000004,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.70399999999999996}]},{"osd":4,"last update":"Thu Oct 19 14:21:02 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.48899999999999999,"15min":0.48399999999999999},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.51000000000000001,"15min":0.50800000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.30399999999999999},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.85699999999999998},"last":0.61399999999999999}]},{"osd":5,"last update":"Thu Oct 19 14:21:02 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.50700000000000001,"15min":0.502},"min":{"1min":0.34799999999999998,"5min":0.32700000000000001,"15min":0.32500000000000001},"max":{"1min":0.79900000000000004,"5min":1.115,"15min":1.115},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.56599999999999995,"15min":0.54500000000000004},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.69199999999999995}]},{"osd":7,"last update":"Thu Oct 19 14:21:02 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.56000000000000005,"15min":0.54700000000000004},"min":{"1min":0.36799999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.90100000000000002},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.56699999999999995,"15min":0.55400000000000005},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.98399999999999999,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.67800000000000005}]}]},{"osd":5,"up_from":13,"seq":55834576020,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66348,"kb_used_data":3224,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305492,"statfs":{"total":96636764160,"available":96568823808,"internally_reserved":0,"allocated":3301376,"data_stored":1704193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:21:50 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.57399999999999995,"15min":0.60699999999999998},"min":{"1min":0.38,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.69399999999999995,"5min":0.79700000000000004,"15min":0.86399999999999999},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.59599999999999997,"15min":0.63300000000000001},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.70499999999999996,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.51100000000000001}]},{"osd":1,"last update":"Thu Oct 19 14:21:50 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.59099999999999997,"15min":0.622},"min":{"1min":0.34599999999999997,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.877},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.61199999999999999,"15min":0.63700000000000001},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.877,"5min":0.877,"15min":0.88500000000000001},"last":0.46999999999999997}]},{"osd":2,"last update":"Thu Oct 19 14:21:50 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.58199999999999996,"15min":0.61399999999999999},"min":{"1min":0.378,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.625,"15min":0.65100000000000002},"min":{"1min":0.441,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.78900000000000003,"5min":0.83299999999999996,"15min":0.90800000000000003},"last":0.65600000000000003}]},{"osd":3,"last update":"Thu Oct 19 14:21:50 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.60499999999999998,"15min":0.63600000000000001},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.81100000000000005,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.626,"15min":0.65900000000000003},"min":{"1min":0.40000000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.90900000000000003},"last":0.70099999999999996}]},{"osd":4,"last update":"Thu Oct 19 14:21:50 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.48799999999999999,"15min":0.48699999999999999},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.51800000000000002,"15min":0.505},"min":{"1min":0.309,"5min":0.308,"15min":0.28599999999999998},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.82099999999999995},"last":0.63900000000000001}]},{"osd":6,"last update":"Thu Oct 19 14:21:50 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.51700000000000002,"15min":0.502},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.88800000000000001},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.54100000000000004,"15min":0.53900000000000003},"min":{"1min":0.36899999999999999,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.67700000000000005}]},{"osd":7,"last update":"Thu Oct 19 14:21:50 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.52200000000000002,"15min":0.51800000000000002},"min":{"1min":0.32700000000000001,"5min":0.312,"15min":0.312},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.499},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.57899999999999996,"15min":0.57199999999999995},"min":{"1min":0.38,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.72299999999999998}]}]},{"osd":4,"up_from":13,"seq":55834576020,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48292,"kb_used_data":3224,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323548,"statfs":{"total":96636764160,"available":96587313152,"internally_reserved":0,"allocated":3301376,"data_stored":1704193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:21:16 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.54700000000000004,"15min":0.57099999999999995},"min":{"1min":0.30299999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.879},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.56899999999999995,"15min":0.59299999999999997},"min":{"1min":0.35299999999999998,"5min":0.31,"15min":0.31},"max":{"1min":0.72699999999999998,"5min":0.82599999999999996,"15min":0.873},"last":0.54700000000000004}]},{"osd":1,"last update":"Thu Oct 19 14:21:16 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.59699999999999998,"15min":0.63500000000000001},"min":{"1min":0.40300000000000002,"5min":0.372,"15min":0.32600000000000001},"max":{"1min":0.74099999999999999,"5min":0.80400000000000005,"15min":0.92700000000000005},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.60899999999999999,"15min":0.63900000000000001},"min":{"1min":0.40500000000000003,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.755,"5min":0.90000000000000002,"15min":1.0940000000000001},"last":0.53100000000000003}]},{"osd":2,"last update":"Thu Oct 19 14:21:16 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.626,"15min":0.64700000000000002},"min":{"1min":0.38900000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.80300000000000005,"5min":1.008,"15min":1.008},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.64200000000000002,"15min":0.66600000000000004},"min":{"1min":0.47199999999999998,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.81799999999999995,"5min":0.98799999999999999,"15min":1.008},"last":0.68799999999999994}]},{"osd":3,"last update":"Thu Oct 19 14:21:16 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.59499999999999997,"15min":0.627},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.68600000000000005,"5min":0.83399999999999996,"15min":0.91900000000000004},"last":0.755},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.65600000000000003,"15min":0.66600000000000004},"min":{"1min":0.48199999999999998,"5min":0.45400000000000001,"15min":0.437},"max":{"1min":0.85999999999999999,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.76500000000000001}]},{"osd":5,"last update":"Thu Oct 19 14:21:16 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.497,"15min":0.47799999999999998},"min":{"1min":0.33800000000000002,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.79700000000000004,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.50800000000000001,"15min":0.498},"min":{"1min":0.36499999999999999,"5min":0.33700000000000002,"15min":0.29199999999999998},"max":{"1min":0.81399999999999995,"5min":0.873,"15min":0.873},"last":0.502}]},{"osd":6,"last update":"Thu Oct 19 14:21:16 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.50700000000000001,"15min":0.49399999999999999},"min":{"1min":0.38500000000000001,"5min":0.34799999999999998,"15min":0.28199999999999997},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.54900000000000004,"15min":0.54300000000000004},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.88,"5min":0.88,"15min":1.2969999999999999},"last":0.47999999999999998}]},{"osd":7,"last update":"Thu Oct 19 14:21:16 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55100000000000005,"15min":0.54500000000000004},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.32600000000000001},"max":{"1min":0.78200000000000003,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.438},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.61199999999999999,"15min":0.59899999999999998},"min":{"1min":0.371,"5min":0.32100000000000001,"15min":0.318},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":1.032},"last":0.45600000000000002}]}]},{"osd":3,"up_from":13,"seq":55834576019,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56616,"kb_used_data":3224,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315224,"statfs":{"total":96636764160,"available":96578789376,"internally_reserved":0,"allocated":3301376,"data_stored":1704193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54655432},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:21:53 2023","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.39900000000000002,"15min":0.41699999999999998},"min":{"1min":0.23899999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.54500000000000004,"5min":0.60599999999999998,"15min":0.70699999999999996},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.39400000000000002,"15min":0.42199999999999999},"min":{"1min":0.23899999999999999,"5min":0.217,"15min":0.217},"max":{"1min":0.51900000000000002,"5min":0.61899999999999999,"15min":0.752},"last":0.36699999999999999}]},{"osd":1,"last update":"Thu Oct 19 14:21:53 2023","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.40699999999999997,"15min":0.42799999999999999},"min":{"1min":0.28100000000000003,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.71599999999999997,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.45500000000000002,"15min":0.47199999999999998},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26400000000000001},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.38800000000000001}]},{"osd":2,"last update":"Thu Oct 19 14:21:53 2023","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.46200000000000002,"15min":0.48099999999999998},"min":{"1min":0.34300000000000003,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.73099999999999998},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48499999999999999,"15min":0.504},"min":{"1min":0.30599999999999999,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":1.212,"5min":1.212,"15min":1.212},"last":0.35499999999999998}]},{"osd":4,"last update":"Thu Oct 19 14:21:53 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.67400000000000004,"15min":0.70099999999999996},"min":{"1min":0.44600000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.94199999999999995,"5min":1.018,"15min":1.157},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.629,"5min":0.68700000000000006,"15min":0.72299999999999998},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.89500000000000002,"5min":0.996,"15min":1.173},"last":0.752}]},{"osd":5,"last update":"Thu Oct 19 14:21:53 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.67500000000000004,"15min":0.68600000000000005},"min":{"1min":0.50900000000000001,"5min":0.46600000000000003,"15min":0.41999999999999998},"max":{"1min":0.96799999999999997,"5min":0.97299999999999998,"15min":1.0149999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.66800000000000004,"15min":0.68600000000000005},"min":{"1min":0.48899999999999999,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.97699999999999998,"5min":0.99399999999999999,"15min":1.105},"last":0.80600000000000005}]},{"osd":6,"last update":"Thu Oct 19 14:21:53 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.65000000000000002,"15min":0.68600000000000005},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.93000000000000005,"5min":1.0009999999999999,"15min":1.0669999999999999},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.68899999999999995,"15min":0.70999999999999996},"min":{"1min":0.505,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.079},"last":0.79100000000000004}]},{"osd":7,"last update":"Thu Oct 19 14:21:53 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67500000000000004,"15min":0.69799999999999995},"min":{"1min":0.46600000000000003,"5min":0.434,"15min":0.434},"max":{"1min":0.94999999999999996,"5min":0.96899999999999997,"15min":1.085},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.69799999999999995,"15min":0.72799999999999998},"min":{"1min":0.48799999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.95899999999999996,"5min":0.97199999999999998,"15min":1.1739999999999999},"last":0.60299999999999998}]}]},{"osd":2,"up_from":13,"seq":55834576021,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48292,"kb_used_data":3224,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323548,"statfs":{"total":96636764160,"available":96587313152,"internally_reserved":0,"allocated":3301376,"data_stored":1704193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.374,"15min":0.39600000000000002},"min":{"1min":0.25700000000000001,"5min":0.218,"15min":0.218},"max":{"1min":0.58799999999999997,"5min":0.66600000000000004,"15min":0.69699999999999995},"last":0.27800000000000002},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.41499999999999998,"15min":0.432},"min":{"1min":0.248,"5min":0.248,"15min":0.223},"max":{"1min":0.53100000000000003,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.379}]},{"osd":1,"last update":"Thu Oct 19 14:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.38300000000000001,"15min":0.40600000000000003},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.65400000000000003},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.44400000000000001,"15min":0.46000000000000002},"min":{"1min":0.28699999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.45100000000000001,"5min":1.742,"15min":1.742},"last":0.40100000000000002}]},{"osd":3,"last update":"Thu Oct 19 14:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.45500000000000002,"15min":0.47599999999999998},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.63300000000000001,"5min":0.69299999999999995,"15min":0.74399999999999999},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.47299999999999998,"15min":0.501},"min":{"1min":0.32700000000000001,"5min":0.32000000000000001,"15min":0.315},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.83599999999999997},"last":0.5}]},{"osd":4,"last update":"Thu Oct 19 14:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.63900000000000001,"15min":0.66200000000000003},"min":{"1min":0.40699999999999997,"5min":0.39600000000000002,"15min":0.39500000000000002},"max":{"1min":0.74399999999999999,"5min":0.89200000000000002,"15min":0.998},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.65400000000000003,"15min":0.67600000000000005},"min":{"1min":0.41899999999999998,"5min":0.38900000000000001,"15min":0.35199999999999998},"max":{"1min":0.79000000000000004,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.60599999999999998}]},{"osd":5,"last update":"Thu Oct 19 14:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.67800000000000005,"15min":0.69299999999999995},"min":{"1min":0.443,"5min":0.443,"15min":0.40600000000000003},"max":{"1min":0.79300000000000004,"5min":1.4330000000000001,"15min":1.4330000000000001},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.66900000000000004,"15min":0.68000000000000005},"min":{"1min":0.46899999999999997,"5min":0.45100000000000001,"15min":0.39300000000000002},"max":{"1min":0.77400000000000002,"5min":1.595,"15min":1.595},"last":0.80900000000000005}]},{"osd":6,"last update":"Thu Oct 19 14:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.67700000000000005,"15min":0.70199999999999996},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.80900000000000005,"5min":0.996,"15min":1.0189999999999999},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.629,"5min":0.69499999999999995,"15min":0.71399999999999997},"min":{"1min":0.438,"5min":0.433,"15min":0.40600000000000003},"max":{"1min":0.83399999999999996,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":0.73099999999999998}]},{"osd":7,"last update":"Thu Oct 19 14:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.68400000000000005,"15min":0.71299999999999997},"min":{"1min":0.47199999999999998,"5min":0.44800000000000001,"15min":0.41799999999999998},"max":{"1min":0.78800000000000003,"5min":1.0129999999999999,"15min":1.0289999999999999},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.71199999999999997,"15min":0.72599999999999998},"min":{"1min":0.42199999999999999,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.86699999999999999,"5min":1.53,"15min":1.53},"last":0.64200000000000002}]}]},{"osd":1,"up_from":13,"seq":55834576020,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48296,"kb_used_data":3224,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323544,"statfs":{"total":96636764160,"available":96587309056,"internally_reserved":0,"allocated":3301376,"data_stored":1704193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.38400000000000001,"15min":0.40300000000000002},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.46899999999999997,"5min":0.61599999999999999,"15min":0.67400000000000004},"last":0.22500000000000001},{"interface":"front","average":{"1min":0.378,"5min":0.41999999999999998,"15min":0.45000000000000001},"min":{"1min":0.27700000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.51500000000000001,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.28100000000000003}]},{"osd":2,"last update":"Thu Oct 19 14:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.42399999999999999,"15min":0.438},"min":{"1min":0.26100000000000001,"5min":0.251,"15min":0.251},"max":{"1min":0.505,"5min":0.63100000000000001,"15min":0.77500000000000002},"last":0.318},{"interface":"front","average":{"1min":0.434,"5min":0.47699999999999998,"15min":0.48799999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.56699999999999995,"5min":0.68700000000000006,"15min":0.82799999999999996},"last":0.44400000000000001}]},{"osd":3,"last update":"Thu Oct 19 14:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.495,"15min":0.501},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.29499999999999998},"max":{"1min":0.59399999999999997,"5min":0.76000000000000001,"15min":0.78900000000000003},"last":0.376},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.42899999999999999,"15min":0.44600000000000001},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.26100000000000001},"max":{"1min":0.59899999999999998,"5min":0.65200000000000002,"15min":0.69499999999999995},"last":0.245}]},{"osd":4,"last update":"Thu Oct 19 14:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.64200000000000002,"15min":0.66700000000000004},"min":{"1min":0.42299999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.80400000000000005,"5min":0.95999999999999996,"15min":1.042},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.65400000000000003,"15min":0.66600000000000004},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.32300000000000001},"max":{"1min":0.80500000000000005,"5min":0.90800000000000003,"15min":1.1659999999999999},"last":0.39500000000000002}]},{"osd":5,"last update":"Thu Oct 19 14:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.66700000000000004,"15min":0.68200000000000005},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.81200000000000006,"5min":0.96199999999999997,"15min":1.3500000000000001},"last":0.434},{"interface":"front","average":{"1min":0.62,"5min":0.68899999999999995,"15min":0.70299999999999996},"min":{"1min":0.442,"5min":0.44,"15min":0.40500000000000003},"max":{"1min":0.82899999999999996,"5min":1.0009999999999999,"15min":1.121},"last":0.49399999999999999}]},{"osd":6,"last update":"Thu Oct 19 14:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.65400000000000003,"15min":0.67600000000000005},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.84999999999999998,"5min":1.0129999999999999,"15min":1.2310000000000001},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.69699999999999995,"15min":0.71499999999999997},"min":{"1min":0.46700000000000003,"5min":0.434,"15min":0.434},"max":{"1min":0.85799999999999998,"5min":1.0009999999999999,"15min":1.274},"last":0.47699999999999998}]},{"osd":7,"last update":"Thu Oct 19 14:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.70499999999999996,"15min":0.72199999999999998},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.86899999999999999,"5min":0.99099999999999999,"15min":1.075},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.68200000000000005,"15min":0.69599999999999995},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.84599999999999997,"5min":1.0169999999999999,"15min":1.117},"last":0.46200000000000002}]}]},{"osd":0,"up_from":13,"seq":55834576021,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61348,"kb_used_data":3800,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310492,"statfs":{"total":96636764160,"available":96573943808,"internally_reserved":0,"allocated":3891200,"data_stored":2293942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Oct 19 14:21:13 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.40100000000000002,"15min":0.41099999999999998},"min":{"1min":0.26300000000000001,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.47799999999999998,"5min":0.61499999999999999,"15min":0.624},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.41699999999999998,"15min":0.45500000000000002},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.49199999999999999,"5min":0.70899999999999996,"15min":0.81100000000000005},"last":0.64300000000000002}]},{"osd":2,"last update":"Thu Oct 19 14:21:13 2023","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.34100000000000003,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.54400000000000004,"5min":0.754,"15min":0.754},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.377,"5min":0.42399999999999999,"15min":0.44400000000000001},"min":{"1min":0.28100000000000003,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.46899999999999997,"5min":0.60199999999999998,"15min":0.69099999999999995},"last":0.52400000000000002}]},{"osd":3,"last update":"Thu Oct 19 14:21:13 2023","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.436},"min":{"1min":0.27200000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.50800000000000001,"5min":0.58799999999999997,"15min":0.66800000000000004},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.48199999999999998,"15min":0.50700000000000001},"min":{"1min":0.29299999999999998,"5min":0.254,"15min":0.254},"max":{"1min":0.72399999999999998,"5min":0.78100000000000003,"15min":0.84599999999999997},"last":0.53400000000000003}]},{"osd":4,"last update":"Thu Oct 19 14:21:13 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.63,"15min":0.67000000000000004},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.80300000000000005,"5min":0.93100000000000005,"15min":1.1890000000000001},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.628,"15min":0.67400000000000004},"min":{"1min":0.45900000000000002,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.75600000000000001,"5min":0.93400000000000005,"15min":1.2150000000000001},"last":0.78300000000000003}]},{"osd":5,"last update":"Thu Oct 19 14:21:13 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.67200000000000004,"15min":0.71399999999999997},"min":{"1min":0.48599999999999999,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.83099999999999996,"5min":0.98499999999999999,"15min":1.1579999999999999},"last":0.748},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.67500000000000004,"15min":0.71399999999999997},"min":{"1min":0.499,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.80000000000000004,"5min":0.98299999999999998,"15min":1.2310000000000001},"last":0.73899999999999999}]},{"osd":6,"last update":"Thu Oct 19 14:21:13 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.66300000000000003,"15min":0.69899999999999995},"min":{"1min":0.46100000000000002,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.85499999999999998,"5min":0.97799999999999998,"15min":1.1739999999999999},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.65300000000000002,"15min":0.69599999999999995},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.70299999999999996,"5min":0.97799999999999998,"15min":1.0189999999999999},"last":0.76100000000000001}]},{"osd":7,"last update":"Thu Oct 19 14:21:13 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.66600000000000004,"15min":0.70999999999999996},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.79900000000000004,"5min":0.93400000000000005,"15min":1.1180000000000001},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.66100000000000003,"15min":0.69599999999999995},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.84499999999999997,"5min":0.93799999999999994,"15min":1.0349999999999999},"last":0.77300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-19T14:22:06.116 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.3f is not active+clean 2023-10-19T14:22:06.116 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.3f', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 262, 'state': 'creating+activating', 'last_fresh': '2023-10-19T14:22:02.407757+0000', 'last_change': '2023-10-19T14:22:02.407757+0000', 'last_active': '2023-10-19T14:22:01.386521+0000', 'last_peered': '2023-10-19T14:22:01.386521+0000', 'last_clean': '2023-10-19T14:22:01.386521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:22:02.407757+0000', 'last_undegraded': '2023-10-19T14:22:02.407757+0000', 'last_fullsized': '2023-10-19T14:22:02.407757+0000', 'mapping_epoch': 261, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 261, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'last_clean_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-20T17:14:21.361763+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': ['6', '3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-10-19T14:22:06.117 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.3e is not active+clean 2023-10-19T14:22:06.117 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.3e', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 262, 'state': 'creating+activating', 'last_fresh': '2023-10-19T14:22:02.406175+0000', 'last_change': '2023-10-19T14:22:02.406175+0000', 'last_active': '2023-10-19T14:22:01.386521+0000', 'last_peered': '2023-10-19T14:22:01.386521+0000', 'last_clean': '2023-10-19T14:22:01.386521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:22:02.406175+0000', 'last_undegraded': '2023-10-19T14:22:02.406175+0000', 'last_fullsized': '2023-10-19T14:22:02.406175+0000', 'mapping_epoch': 261, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 261, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'last_clean_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-20T18:23:52.735800+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': ['6', '2'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-10-19T14:22:06.117 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.3c is not active+clean 2023-10-19T14:22:06.118 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.3c', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 262, 'state': 'creating+activating', 'last_fresh': '2023-10-19T14:22:02.403761+0000', 'last_change': '2023-10-19T14:22:02.403761+0000', 'last_active': '2023-10-19T14:22:01.386521+0000', 'last_peered': '2023-10-19T14:22:01.386521+0000', 'last_clean': '2023-10-19T14:22:01.386521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:22:02.403761+0000', 'last_undegraded': '2023-10-19T14:22:02.403761+0000', 'last_fullsized': '2023-10-19T14:22:02.403761+0000', 'mapping_epoch': 261, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 261, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'last_clean_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-20T16:50:46.770884+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': ['2', '7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-10-19T14:22:06.118 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.3b is not active+clean 2023-10-19T14:22:06.119 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.3b', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 262, 'state': 'creating+activating', 'last_fresh': '2023-10-19T14:22:02.409103+0000', 'last_change': '2023-10-19T14:22:02.409103+0000', 'last_active': '2023-10-19T14:22:01.386521+0000', 'last_peered': '2023-10-19T14:22:01.386521+0000', 'last_clean': '2023-10-19T14:22:01.386521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:22:02.409103+0000', 'last_undegraded': '2023-10-19T14:22:02.409103+0000', 'last_fullsized': '2023-10-19T14:22:02.409103+0000', 'mapping_epoch': 261, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 261, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'last_clean_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-20T16:03:49.877672+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': ['1', '5'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-19T14:22:06.119 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.3a is not active+clean 2023-10-19T14:22:06.119 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.3a', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 262, 'state': 'creating+activating', 'last_fresh': '2023-10-19T14:22:02.408280+0000', 'last_change': '2023-10-19T14:22:02.408280+0000', 'last_active': '2023-10-19T14:22:01.386521+0000', 'last_peered': '2023-10-19T14:22:01.386521+0000', 'last_clean': '2023-10-19T14:22:01.386521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:22:02.408280+0000', 'last_undegraded': '2023-10-19T14:22:02.408280+0000', 'last_fullsized': '2023-10-19T14:22:02.408280+0000', 'mapping_epoch': 261, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 261, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'last_clean_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-21T00:18:59.130442+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [1, 0], 'avail_no_missing': ['1', '0'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-19T14:22:06.120 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.38 is not active+clean 2023-10-19T14:22:06.120 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.38', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 262, 'state': 'creating+activating', 'last_fresh': '2023-10-19T14:22:02.403768+0000', 'last_change': '2023-10-19T14:22:02.403768+0000', 'last_active': '2023-10-19T14:22:01.386521+0000', 'last_peered': '2023-10-19T14:22:01.386521+0000', 'last_clean': '2023-10-19T14:22:01.386521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:22:02.403768+0000', 'last_undegraded': '2023-10-19T14:22:02.403768+0000', 'last_fullsized': '2023-10-19T14:22:02.403768+0000', 'mapping_epoch': 261, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 261, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'last_clean_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-20T15:30:01.392073+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': ['2', '4'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-10-19T14:22:06.121 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.37 is not active+clean 2023-10-19T14:22:06.121 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.37', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 262, 'state': 'creating+activating', 'last_fresh': '2023-10-19T14:22:02.404303+0000', 'last_change': '2023-10-19T14:22:02.404303+0000', 'last_active': '2023-10-19T14:22:01.386521+0000', 'last_peered': '2023-10-19T14:22:01.386521+0000', 'last_clean': '2023-10-19T14:22:01.386521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:22:02.404303+0000', 'last_undegraded': '2023-10-19T14:22:02.404303+0000', 'last_fullsized': '2023-10-19T14:22:02.404303+0000', 'mapping_epoch': 261, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 261, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'last_clean_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-20T20:16:24.272266+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': ['7', '0'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-10-19T14:22:06.121 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.34 is not active+clean 2023-10-19T14:22:06.122 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.34', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 262, 'state': 'creating+activating', 'last_fresh': '2023-10-19T14:22:02.406632+0000', 'last_change': '2023-10-19T14:22:02.406632+0000', 'last_active': '2023-10-19T14:22:01.386521+0000', 'last_peered': '2023-10-19T14:22:01.386521+0000', 'last_clean': '2023-10-19T14:22:01.386521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:22:02.406632+0000', 'last_undegraded': '2023-10-19T14:22:02.406632+0000', 'last_fullsized': '2023-10-19T14:22:02.406632+0000', 'mapping_epoch': 261, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 261, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'last_clean_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-20T22:30:39.654226+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': ['6', '3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-10-19T14:22:06.122 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.33 is not active+clean 2023-10-19T14:22:06.122 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.33', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 262, 'state': 'creating+activating', 'last_fresh': '2023-10-19T14:22:02.403708+0000', 'last_change': '2023-10-19T14:22:02.403708+0000', 'last_active': '2023-10-19T14:22:01.386521+0000', 'last_peered': '2023-10-19T14:22:01.386521+0000', 'last_clean': '2023-10-19T14:22:01.386521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:22:02.403708+0000', 'last_undegraded': '2023-10-19T14:22:02.403708+0000', 'last_fullsized': '2023-10-19T14:22:02.403708+0000', 'mapping_epoch': 261, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 261, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'last_clean_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-20T16:39:38.146246+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 6], 'acting': [2, 6], 'avail_no_missing': ['2', '6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-10-19T14:22:06.123 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.32 is not active+clean 2023-10-19T14:22:06.123 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.32', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 262, 'state': 'creating+activating', 'last_fresh': '2023-10-19T14:22:02.403725+0000', 'last_change': '2023-10-19T14:22:02.403725+0000', 'last_active': '2023-10-19T14:22:01.386521+0000', 'last_peered': '2023-10-19T14:22:01.386521+0000', 'last_clean': '2023-10-19T14:22:01.386521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:22:02.403725+0000', 'last_undegraded': '2023-10-19T14:22:02.403725+0000', 'last_fullsized': '2023-10-19T14:22:02.403725+0000', 'mapping_epoch': 261, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 261, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'last_clean_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-21T00:10:47.778168+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': ['7', '2'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-10-19T14:22:06.124 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.31 is not active+clean 2023-10-19T14:22:06.124 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.31', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 262, 'state': 'creating+activating', 'last_fresh': '2023-10-19T14:22:02.403277+0000', 'last_change': '2023-10-19T14:22:02.403277+0000', 'last_active': '2023-10-19T14:22:01.386521+0000', 'last_peered': '2023-10-19T14:22:01.386521+0000', 'last_clean': '2023-10-19T14:22:01.386521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:22:02.403277+0000', 'last_undegraded': '2023-10-19T14:22:02.403277+0000', 'last_fullsized': '2023-10-19T14:22:02.403277+0000', 'mapping_epoch': 261, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 261, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'last_clean_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-20T19:15:36.803736+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': ['2', '4'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-10-19T14:22:06.125 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.2e is not active+clean 2023-10-19T14:22:06.125 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.2e', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 262, 'state': 'creating+activating', 'last_fresh': '2023-10-19T14:22:02.405331+0000', 'last_change': '2023-10-19T14:22:02.405331+0000', 'last_active': '2023-10-19T14:22:01.386521+0000', 'last_peered': '2023-10-19T14:22:01.386521+0000', 'last_clean': '2023-10-19T14:22:01.386521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:22:02.405331+0000', 'last_undegraded': '2023-10-19T14:22:02.405331+0000', 'last_fullsized': '2023-10-19T14:22:02.405331+0000', 'mapping_epoch': 261, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 261, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'last_clean_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-20T14:49:42.930618+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': ['2', '0'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-10-19T14:22:06.125 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.2a is not active+clean 2023-10-19T14:22:06.126 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.2a', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 262, 'state': 'creating+activating', 'last_fresh': '2023-10-19T14:22:02.399641+0000', 'last_change': '2023-10-19T14:22:02.399641+0000', 'last_active': '2023-10-19T14:22:01.386521+0000', 'last_peered': '2023-10-19T14:22:01.386521+0000', 'last_clean': '2023-10-19T14:22:01.386521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:22:02.399641+0000', 'last_undegraded': '2023-10-19T14:22:02.399641+0000', 'last_fullsized': '2023-10-19T14:22:02.399641+0000', 'mapping_epoch': 261, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 261, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'last_clean_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-20T22:35:08.037582+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': ['0', '4'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-10-19T14:22:06.126 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.29 is not active+clean 2023-10-19T14:22:06.127 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.29', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 262, 'state': 'creating+activating', 'last_fresh': '2023-10-19T14:22:02.405143+0000', 'last_change': '2023-10-19T14:22:02.405143+0000', 'last_active': '2023-10-19T14:22:01.386521+0000', 'last_peered': '2023-10-19T14:22:01.386521+0000', 'last_clean': '2023-10-19T14:22:01.386521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:22:02.405143+0000', 'last_undegraded': '2023-10-19T14:22:02.405143+0000', 'last_fullsized': '2023-10-19T14:22:02.405143+0000', 'mapping_epoch': 261, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 261, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'last_clean_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-20T17:09:54.096828+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': ['7', '3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-10-19T14:22:06.127 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.28 is not active+clean 2023-10-19T14:22:06.127 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.28', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 262, 'state': 'creating+activating', 'last_fresh': '2023-10-19T14:22:02.398717+0000', 'last_change': '2023-10-19T14:22:02.398717+0000', 'last_active': '2023-10-19T14:22:01.386521+0000', 'last_peered': '2023-10-19T14:22:01.386521+0000', 'last_clean': '2023-10-19T14:22:01.386521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:22:02.398717+0000', 'last_undegraded': '2023-10-19T14:22:02.398717+0000', 'last_fullsized': '2023-10-19T14:22:02.398717+0000', 'mapping_epoch': 261, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 261, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'last_clean_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-20T14:35:36.604007+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': ['0', '6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-10-19T14:22:06.128 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.26 is not active+clean 2023-10-19T14:22:06.128 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.26', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 262, 'state': 'creating+activating', 'last_fresh': '2023-10-19T14:22:02.410522+0000', 'last_change': '2023-10-19T14:22:02.410522+0000', 'last_active': '2023-10-19T14:22:01.386521+0000', 'last_peered': '2023-10-19T14:22:01.386521+0000', 'last_clean': '2023-10-19T14:22:01.386521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:22:02.410522+0000', 'last_undegraded': '2023-10-19T14:22:02.410522+0000', 'last_fullsized': '2023-10-19T14:22:02.410522+0000', 'mapping_epoch': 261, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 261, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'last_clean_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-20T18:41:53.947719+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': ['1', '6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-19T14:22:06.129 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.24 is not active+clean 2023-10-19T14:22:06.129 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.24', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 262, 'state': 'creating+activating', 'last_fresh': '2023-10-19T14:22:02.407514+0000', 'last_change': '2023-10-19T14:22:02.407514+0000', 'last_active': '2023-10-19T14:22:01.386521+0000', 'last_peered': '2023-10-19T14:22:01.386521+0000', 'last_clean': '2023-10-19T14:22:01.386521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:22:02.407514+0000', 'last_undegraded': '2023-10-19T14:22:02.407514+0000', 'last_fullsized': '2023-10-19T14:22:02.407514+0000', 'mapping_epoch': 261, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 261, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'last_clean_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-21T01:14:37.969686+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': ['1', '4'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-19T14:22:06.129 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.23 is not active+clean 2023-10-19T14:22:06.130 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.23', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 262, 'state': 'creating+activating', 'last_fresh': '2023-10-19T14:22:02.398963+0000', 'last_change': '2023-10-19T14:22:02.398963+0000', 'last_active': '2023-10-19T14:22:01.386521+0000', 'last_peered': '2023-10-19T14:22:01.386521+0000', 'last_clean': '2023-10-19T14:22:01.386521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:22:02.398963+0000', 'last_undegraded': '2023-10-19T14:22:02.398963+0000', 'last_fullsized': '2023-10-19T14:22:02.398963+0000', 'mapping_epoch': 261, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 261, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'last_clean_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-20T16:57:49.145921+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': ['0', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-10-19T14:22:06.130 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.21 is not active+clean 2023-10-19T14:22:06.130 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.21', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 262, 'state': 'creating+activating', 'last_fresh': '2023-10-19T14:22:02.399356+0000', 'last_change': '2023-10-19T14:22:02.399356+0000', 'last_active': '2023-10-19T14:22:01.386521+0000', 'last_peered': '2023-10-19T14:22:01.386521+0000', 'last_clean': '2023-10-19T14:22:01.386521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:22:02.399356+0000', 'last_undegraded': '2023-10-19T14:22:02.399356+0000', 'last_fullsized': '2023-10-19T14:22:02.399356+0000', 'mapping_epoch': 261, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 261, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'last_clean_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-21T00:45:44.957902+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': ['0', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-10-19T14:22:06.131 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.20 is not active+clean 2023-10-19T14:22:06.131 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.20', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 262, 'state': 'creating+activating', 'last_fresh': '2023-10-19T14:22:02.403114+0000', 'last_change': '2023-10-19T14:22:02.403114+0000', 'last_active': '2023-10-19T14:22:01.386521+0000', 'last_peered': '2023-10-19T14:22:01.386521+0000', 'last_clean': '2023-10-19T14:22:01.386521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:22:02.403114+0000', 'last_undegraded': '2023-10-19T14:22:02.403114+0000', 'last_fullsized': '2023-10-19T14:22:02.403114+0000', 'mapping_epoch': 261, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 261, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'last_clean_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-21T02:18:23.251534+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-10-19T14:22:06.132 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.18 is not active+clean 2023-10-19T14:22:06.132 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.18', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 262, 'state': 'creating+activating', 'last_fresh': '2023-10-19T14:22:02.398614+0000', 'last_change': '2023-10-19T14:22:02.398614+0000', 'last_active': '2023-10-19T14:22:01.386521+0000', 'last_peered': '2023-10-19T14:22:01.386521+0000', 'last_clean': '2023-10-19T14:22:01.386521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:22:02.398614+0000', 'last_undegraded': '2023-10-19T14:22:02.398614+0000', 'last_fullsized': '2023-10-19T14:22:02.398614+0000', 'mapping_epoch': 261, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 261, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'last_clean_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-20T23:31:51.736865+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': ['0', '6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-10-19T14:22:06.132 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.1a is not active+clean 2023-10-19T14:22:06.133 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.1a', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 262, 'state': 'creating+activating', 'last_fresh': '2023-10-19T14:22:02.406083+0000', 'last_change': '2023-10-19T14:22:02.406083+0000', 'last_active': '2023-10-19T14:22:01.386521+0000', 'last_peered': '2023-10-19T14:22:01.386521+0000', 'last_clean': '2023-10-19T14:22:01.386521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:22:02.406083+0000', 'last_undegraded': '2023-10-19T14:22:02.406083+0000', 'last_fullsized': '2023-10-19T14:22:02.406083+0000', 'mapping_epoch': 261, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 261, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'last_clean_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-20T21:07:50.835228+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': ['6', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-10-19T14:22:06.133 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.1c is not active+clean 2023-10-19T14:22:06.134 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.1c', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 262, 'state': 'creating+activating', 'last_fresh': '2023-10-19T14:22:02.402955+0000', 'last_change': '2023-10-19T14:22:02.402955+0000', 'last_active': '2023-10-19T14:22:01.386521+0000', 'last_peered': '2023-10-19T14:22:01.386521+0000', 'last_clean': '2023-10-19T14:22:01.386521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:22:02.402955+0000', 'last_undegraded': '2023-10-19T14:22:02.402955+0000', 'last_fullsized': '2023-10-19T14:22:02.402955+0000', 'mapping_epoch': 261, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 261, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'last_clean_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-21T01:03:27.890218+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': ['2', '0'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-10-19T14:22:06.134 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.1d is not active+clean 2023-10-19T14:22:06.134 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.1d', 'version': "0'0", 'reported_seq': 6, 'reported_epoch': 262, 'state': 'active', 'last_fresh': '2023-10-19T14:22:02.413928+0000', 'last_change': '2023-10-19T14:22:02.413862+0000', 'last_active': '2023-10-19T14:22:02.413928+0000', 'last_peered': '2023-10-19T14:22:02.413928+0000', 'last_clean': '2023-10-19T14:22:01.386521+0000', 'last_became_active': '2023-10-19T14:22:02.413862+0000', 'last_became_peered': '2023-10-19T14:22:02.413862+0000', 'last_unstale': '2023-10-19T14:22:02.413928+0000', 'last_undegraded': '2023-10-19T14:22:02.413928+0000', 'last_fullsized': '2023-10-19T14:22:02.413928+0000', 'mapping_epoch': 261, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 261, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'last_clean_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-20T22:20:55.394812+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': ['0', '6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-10-19T14:22:06.135 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.10 is not active+clean 2023-10-19T14:22:06.135 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.10', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 262, 'state': 'creating+activating', 'last_fresh': '2023-10-19T14:22:02.402706+0000', 'last_change': '2023-10-19T14:22:02.402706+0000', 'last_active': '2023-10-19T14:22:01.386521+0000', 'last_peered': '2023-10-19T14:22:01.386521+0000', 'last_clean': '2023-10-19T14:22:01.386521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:22:02.402706+0000', 'last_undegraded': '2023-10-19T14:22:02.402706+0000', 'last_fullsized': '2023-10-19T14:22:02.402706+0000', 'mapping_epoch': 261, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 261, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'last_clean_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-20T18:45:46.279692+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 6], 'acting': [2, 6], 'avail_no_missing': ['2', '6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-10-19T14:22:06.136 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.11 is not active+clean 2023-10-19T14:22:06.136 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.11', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 262, 'state': 'creating+activating', 'last_fresh': '2023-10-19T14:22:02.405750+0000', 'last_change': '2023-10-19T14:22:02.405750+0000', 'last_active': '2023-10-19T14:22:01.386521+0000', 'last_peered': '2023-10-19T14:22:01.386521+0000', 'last_clean': '2023-10-19T14:22:01.386521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:22:02.405750+0000', 'last_undegraded': '2023-10-19T14:22:02.405750+0000', 'last_fullsized': '2023-10-19T14:22:02.405750+0000', 'mapping_epoch': 261, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 261, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'last_clean_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-20T19:06:30.359305+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': ['6', '4'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-10-19T14:22:06.136 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.13 is not active+clean 2023-10-19T14:22:06.137 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.13', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 262, 'state': 'creating+activating', 'last_fresh': '2023-10-19T14:22:02.408938+0000', 'last_change': '2023-10-19T14:22:02.408938+0000', 'last_active': '2023-10-19T14:22:01.386521+0000', 'last_peered': '2023-10-19T14:22:01.386521+0000', 'last_clean': '2023-10-19T14:22:01.386521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:22:02.408938+0000', 'last_undegraded': '2023-10-19T14:22:02.408938+0000', 'last_fullsized': '2023-10-19T14:22:02.408938+0000', 'mapping_epoch': 261, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 261, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'last_clean_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-21T01:22:40.195315+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': ['1', '2'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-19T14:22:06.137 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.15 is not active+clean 2023-10-19T14:22:06.138 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.15', 'version': "0'0", 'reported_seq': 6, 'reported_epoch': 262, 'state': 'active', 'last_fresh': '2023-10-19T14:22:02.414429+0000', 'last_change': '2023-10-19T14:22:02.414358+0000', 'last_active': '2023-10-19T14:22:02.414429+0000', 'last_peered': '2023-10-19T14:22:02.414429+0000', 'last_clean': '2023-10-19T14:22:01.386521+0000', 'last_became_active': '2023-10-19T14:22:02.414358+0000', 'last_became_peered': '2023-10-19T14:22:02.414358+0000', 'last_unstale': '2023-10-19T14:22:02.414429+0000', 'last_undegraded': '2023-10-19T14:22:02.414429+0000', 'last_fullsized': '2023-10-19T14:22:02.414429+0000', 'mapping_epoch': 261, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 261, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'last_clean_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-20T20:02:05.922124+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': ['2', '5'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-10-19T14:22:06.138 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.8 is not active+clean 2023-10-19T14:22:06.138 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.8', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 262, 'state': 'creating+activating', 'last_fresh': '2023-10-19T14:22:02.407065+0000', 'last_change': '2023-10-19T14:22:02.407065+0000', 'last_active': '2023-10-19T14:22:01.386521+0000', 'last_peered': '2023-10-19T14:22:01.386521+0000', 'last_clean': '2023-10-19T14:22:01.386521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:22:02.407065+0000', 'last_undegraded': '2023-10-19T14:22:02.407065+0000', 'last_fullsized': '2023-10-19T14:22:02.407065+0000', 'mapping_epoch': 261, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 261, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'last_clean_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-20T16:07:55.991807+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': ['1', '5'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-19T14:22:06.139 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.9 is not active+clean 2023-10-19T14:22:06.139 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.9', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 262, 'state': 'creating+activating', 'last_fresh': '2023-10-19T14:22:02.398780+0000', 'last_change': '2023-10-19T14:22:02.398780+0000', 'last_active': '2023-10-19T14:22:01.386521+0000', 'last_peered': '2023-10-19T14:22:01.386521+0000', 'last_clean': '2023-10-19T14:22:01.386521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:22:02.398780+0000', 'last_undegraded': '2023-10-19T14:22:02.398780+0000', 'last_fullsized': '2023-10-19T14:22:02.398780+0000', 'mapping_epoch': 261, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 261, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'last_clean_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-20T19:40:53.511724+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': ['0', '6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-10-19T14:22:06.140 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.a is not active+clean 2023-10-19T14:22:06.140 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.a', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 262, 'state': 'creating+activating', 'last_fresh': '2023-10-19T14:22:02.407858+0000', 'last_change': '2023-10-19T14:22:02.407858+0000', 'last_active': '2023-10-19T14:22:01.386521+0000', 'last_peered': '2023-10-19T14:22:01.386521+0000', 'last_clean': '2023-10-19T14:22:01.386521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:22:02.407858+0000', 'last_undegraded': '2023-10-19T14:22:02.407858+0000', 'last_fullsized': '2023-10-19T14:22:02.407858+0000', 'mapping_epoch': 261, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 261, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'last_clean_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-21T00:31:43.355728+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': ['1', '5'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-19T14:22:06.140 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.d is not active+clean 2023-10-19T14:22:06.141 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.d', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 262, 'state': 'creating+activating', 'last_fresh': '2023-10-19T14:22:02.399329+0000', 'last_change': '2023-10-19T14:22:02.399329+0000', 'last_active': '2023-10-19T14:22:01.386521+0000', 'last_peered': '2023-10-19T14:22:01.386521+0000', 'last_clean': '2023-10-19T14:22:01.386521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:22:02.399329+0000', 'last_undegraded': '2023-10-19T14:22:02.399329+0000', 'last_fullsized': '2023-10-19T14:22:02.399329+0000', 'mapping_epoch': 261, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 261, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'last_clean_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-20T21:17:41.707119+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': ['0', '4'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-10-19T14:22:06.141 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.f is not active+clean 2023-10-19T14:22:06.141 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.f', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 262, 'state': 'creating+activating', 'last_fresh': '2023-10-19T14:22:02.409049+0000', 'last_change': '2023-10-19T14:22:02.409049+0000', 'last_active': '2023-10-19T14:22:01.386521+0000', 'last_peered': '2023-10-19T14:22:01.386521+0000', 'last_clean': '2023-10-19T14:22:01.386521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:22:02.409049+0000', 'last_undegraded': '2023-10-19T14:22:02.409049+0000', 'last_fullsized': '2023-10-19T14:22:02.409049+0000', 'mapping_epoch': 261, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 261, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'last_clean_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-20T16:44:13.928436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [1, 0], 'avail_no_missing': ['1', '0'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-19T14:22:06.142 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.0 is not active+clean 2023-10-19T14:22:06.142 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.0', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 262, 'state': 'creating+activating', 'last_fresh': '2023-10-19T14:22:02.402684+0000', 'last_change': '2023-10-19T14:22:02.402684+0000', 'last_active': '2023-10-19T14:22:01.386521+0000', 'last_peered': '2023-10-19T14:22:01.386521+0000', 'last_clean': '2023-10-19T14:22:01.386521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:22:02.402684+0000', 'last_undegraded': '2023-10-19T14:22:02.402684+0000', 'last_fullsized': '2023-10-19T14:22:02.402684+0000', 'mapping_epoch': 261, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 261, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'last_clean_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-20T14:40:48.301290+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 6], 'acting': [2, 6], 'avail_no_missing': ['2', '6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-10-19T14:22:06.143 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.2 is not active+clean 2023-10-19T14:22:06.143 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.2', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 262, 'state': 'creating+activating', 'last_fresh': '2023-10-19T14:22:02.403340+0000', 'last_change': '2023-10-19T14:22:02.403340+0000', 'last_active': '2023-10-19T14:22:01.386521+0000', 'last_peered': '2023-10-19T14:22:01.386521+0000', 'last_clean': '2023-10-19T14:22:01.386521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:22:02.403340+0000', 'last_undegraded': '2023-10-19T14:22:02.403340+0000', 'last_fullsized': '2023-10-19T14:22:02.403340+0000', 'mapping_epoch': 261, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 261, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'last_clean_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-20T14:42:16.203127+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 6], 'acting': [2, 6], 'avail_no_missing': ['2', '6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-10-19T14:22:06.143 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.4 is not active+clean 2023-10-19T14:22:06.145 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.4', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 262, 'state': 'creating+activating', 'last_fresh': '2023-10-19T14:22:02.403966+0000', 'last_change': '2023-10-19T14:22:02.403966+0000', 'last_active': '2023-10-19T14:22:01.386521+0000', 'last_peered': '2023-10-19T14:22:01.386521+0000', 'last_clean': '2023-10-19T14:22:01.386521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:22:02.403966+0000', 'last_undegraded': '2023-10-19T14:22:02.403966+0000', 'last_fullsized': '2023-10-19T14:22:02.403966+0000', 'mapping_epoch': 261, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 261, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'last_clean_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-21T01:43:57.998020+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': ['7', '3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-10-19T14:22:06.145 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.6 is not active+clean 2023-10-19T14:22:06.146 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.6', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 262, 'state': 'creating+activating', 'last_fresh': '2023-10-19T14:22:02.409726+0000', 'last_change': '2023-10-19T14:22:02.409726+0000', 'last_active': '2023-10-19T14:22:01.386521+0000', 'last_peered': '2023-10-19T14:22:01.386521+0000', 'last_clean': '2023-10-19T14:22:01.386521+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-19T14:22:02.409726+0000', 'last_undegraded': '2023-10-19T14:22:02.409726+0000', 'last_fullsized': '2023-10-19T14:22:02.409726+0000', 'mapping_epoch': 261, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 261, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'last_clean_scrub_stamp': '2023-10-19T14:22:01.386521+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-20T16:51:43.993546+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 7], 'acting': [1, 7], 'avail_no_missing': ['1', '7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-19T14:22:06.147 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2023-10-19T14:22:06.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.255+0000 7f7dd86de640 1 -- 172.21.15.57:0/3656036473 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7dd006b030 msgr2=0x7f7dd006b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:06.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.255+0000 7f7dd86de640 1 --2- 172.21.15.57:0/3656036473 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7dd006b030 0x7f7dd006b410 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc005fd0 tx=0x7f7dcc02bd90 comp rx=0 tx=0).stop 2023-10-19T14:22:06.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.255+0000 7f7dd86de640 1 -- 172.21.15.57:0/3656036473 shutdown_connections 2023-10-19T14:22:06.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.255+0000 7f7dd86de640 1 --2- 172.21.15.57:0/3656036473 >> v2:172.21.15.179:3300/0 conn(0x7f7dd00c9750 0x7f7dd00c9ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:06.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.255+0000 7f7dd86de640 1 --2- 172.21.15.57:0/3656036473 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7dd006b030 0x7f7dd006b410 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:06.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.255+0000 7f7dd86de640 1 -- 172.21.15.57:0/3656036473 >> 172.21.15.57:0/3656036473 conn(0x7f7dd0069930 msgr2=0x7f7dd0069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:06.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.255+0000 7f7dd86de640 1 -- 172.21.15.57:0/3656036473 shutdown_connections 2023-10-19T14:22:06.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.255+0000 7f7dd86de640 1 -- 172.21.15.57:0/3656036473 wait complete. 2023-10-19T14:22:06.262 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.256+0000 7f7dd86de640 1 Processor -- start 2023-10-19T14:22:06.262 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.256+0000 7f7dd86de640 1 -- start start 2023-10-19T14:22:06.262 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.257+0000 7f7dd86de640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7dd006b030 0x7f7dd014fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:06.262 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.257+0000 7f7dd86de640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7dd00c9750 0x7f7dd0151d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:06.263 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.257+0000 7f7dd86de640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7dd0159bd0 0x7f7dd0152290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:06.263 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.257+0000 7f7dd86de640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd0138980 con 0x7f7dd006b030 2023-10-19T14:22:06.263 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.257+0000 7f7dd86de640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd0137d00 con 0x7f7dd00c9750 2023-10-19T14:22:06.263 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.257+0000 7f7dd86de640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd0137fe0 con 0x7f7dd0159bd0 2023-10-19T14:22:06.263 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.257+0000 7f7dd647a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7dd006b030 0x7f7dd014fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:06.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.257+0000 7f7dd647a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7dd006b030 0x7f7dd014fad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51498/0 (socket says 172.21.15.57:51498) 2023-10-19T14:22:06.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.257+0000 7f7dd647a640 1 -- 172.21.15.57:0/1249630781 learned_addr learned my addr 172.21.15.57:0/1249630781 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:22:06.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.258+0000 7f7dd5c79640 1 --2- 172.21.15.57:0/1249630781 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7dd00c9750 0x7f7dd0151d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:06.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.258+0000 7f7dd6c7b640 1 --2- 172.21.15.57:0/1249630781 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7dd0159bd0 0x7f7dd0152290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:06.264 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.258+0000 7f7dd647a640 1 -- 172.21.15.57:0/1249630781 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7dd0159bd0 msgr2=0x7f7dd0152290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:06.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.258+0000 7f7dd647a640 1 --2- 172.21.15.57:0/1249630781 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7dd0159bd0 0x7f7dd0152290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:06.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.258+0000 7f7dd647a640 1 -- 172.21.15.57:0/1249630781 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7dd00c9750 msgr2=0x7f7dd0151d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:06.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.258+0000 7f7dd647a640 1 --2- 172.21.15.57:0/1249630781 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7dd00c9750 0x7f7dd0151d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:06.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.258+0000 7f7dd647a640 1 -- 172.21.15.57:0/1249630781 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dc8009840 con 0x7f7dd006b030 2023-10-19T14:22:06.265 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.258+0000 7f7dd647a640 1 --2- 172.21.15.57:0/1249630781 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7dd006b030 0x7f7dd014fad0 secure :-1 s=READY pgs=2891 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc036040 tx=0x7f7dcc02d9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:06.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.258+0000 7f7dc6ffd640 1 -- 172.21.15.57:0/1249630781 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dcc02ddf0 con 0x7f7dd006b030 2023-10-19T14:22:06.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.258+0000 7f7dc6ffd640 1 -- 172.21.15.57:0/1249630781 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f7dcc0036e0 con 0x7f7dd006b030 2023-10-19T14:22:06.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.258+0000 7f7dc6ffd640 1 -- 172.21.15.57:0/1249630781 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dcc003af0 con 0x7f7dd006b030 2023-10-19T14:22:06.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.258+0000 7f7dd86de640 1 -- 172.21.15.57:0/1249630781 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dcc009560 con 0x7f7dd006b030 2023-10-19T14:22:06.266 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.258+0000 7f7dd86de640 1 -- 172.21.15.57:0/1249630781 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dd01385a0 con 0x7f7dd006b030 2023-10-19T14:22:06.267 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.259+0000 7f7dd86de640 1 -- 172.21.15.57:0/1249630781 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dd00690e0 con 0x7f7dd006b030 2023-10-19T14:22:06.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.260+0000 7f7dc6ffd640 1 -- 172.21.15.57:0/1249630781 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 145) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f7dcc003c50 con 0x7f7dd006b030 2023-10-19T14:22:06.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.260+0000 7f7dc6ffd640 1 --2- 172.21.15.57:0/1249630781 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f7db406b080 0x7f7db406d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:06.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.260+0000 7f7dc6ffd640 1 -- 172.21.15.57:0/1249630781 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(263..263 src has 1..263) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f7dcc0ac5d0 con 0x7f7dd006b030 2023-10-19T14:22:06.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.260+0000 7f7dd5c79640 1 --2- 172.21.15.57:0/1249630781 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f7db406b080 0x7f7db406d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:06.268 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.261+0000 7f7dd5c79640 1 --2- 172.21.15.57:0/1249630781 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f7db406b080 0x7f7db406d540 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f7dd01f4a20 tx=0x7f7dc0009040 comp rx=0 tx=0).ready entity=mgr.11446 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:06.269 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.264+0000 7f7dc6ffd640 1 -- 172.21.15.57:0/1249630781 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7dcc0771b0 con 0x7f7dd006b030 2023-10-19T14:22:06.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.508+0000 7f7dd86de640 1 -- 172.21.15.57:0/1249630781 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f7dd00cc460 con 0x7f7dd006b030 2023-10-19T14:22:06.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.509+0000 7f7dc6ffd640 1 -- 172.21.15.57:0/1249630781 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+3337 (secure 0 0 0) 0x7f7dcc043d20 con 0x7f7dd006b030 2023-10-19T14:22:06.512 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:22:06.512 INFO:teuthology.orchestra.run.smithi057.stdout:{"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":5893,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":263,"num_osds":8,"num_up_osds":8,"osd_up_since":1697719445,"num_in_osds":8,"osd_in_since":1697719436,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":60},{"state_name":"creating+activating","count":35},{"state_name":"active","count":2}],"num_pgs":97,"num_pools":3,"num_objects":21,"data_bytes":591838,"bytes_used":463597568,"bytes_avail":772630515712,"bytes_total":773094113280,"inactive_pgs_ratio":0.36082473397254944,"write_bytes_sec":2649,"read_op_per_sec":0,"write_op_per_sec":7},"fsmap":{"epoch":116,"id":17,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":17,"rank":0,"name":"d","status":"up:active","gid":13359}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":19,"modified":"2023-10-19T14:21:51.417047+0000","services":{"mds":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"mgr":{"daemons":{"summary":"","x":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"y":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"mon":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"4":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"5":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"6":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"7":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2023-10-19T14:22:06.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.511+0000 7f7dd86de640 1 -- 172.21.15.57:0/1249630781 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f7db406b080 msgr2=0x7f7db406d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:06.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.511+0000 7f7dd86de640 1 --2- 172.21.15.57:0/1249630781 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f7db406b080 0x7f7db406d540 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f7dd01f4a20 tx=0x7f7dc0009040 comp rx=0 tx=0).stop 2023-10-19T14:22:06.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.511+0000 7f7dd86de640 1 -- 172.21.15.57:0/1249630781 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7dd006b030 msgr2=0x7f7dd014fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:06.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.511+0000 7f7dd86de640 1 --2- 172.21.15.57:0/1249630781 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7dd006b030 0x7f7dd014fad0 secure :-1 s=READY pgs=2891 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc036040 tx=0x7f7dcc02d9f0 comp rx=0 tx=0).stop 2023-10-19T14:22:06.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.511+0000 7f7dd86de640 1 -- 172.21.15.57:0/1249630781 shutdown_connections 2023-10-19T14:22:06.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.511+0000 7f7dd86de640 1 --2- 172.21.15.57:0/1249630781 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f7db406b080 0x7f7db406d540 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:06.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.511+0000 7f7dd86de640 1 --2- 172.21.15.57:0/1249630781 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7dd0159bd0 0x7f7dd0152290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:06.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.511+0000 7f7dd86de640 1 --2- 172.21.15.57:0/1249630781 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7dd00c9750 0x7f7dd0151d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:06.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.511+0000 7f7dd86de640 1 --2- 172.21.15.57:0/1249630781 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7dd006b030 0x7f7dd014fad0 unknown :-1 s=CLOSED pgs=2891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:06.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.511+0000 7f7dd86de640 1 -- 172.21.15.57:0/1249630781 >> 172.21.15.57:0/1249630781 conn(0x7f7dd0069930 msgr2=0x7f7dd00ac620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:06.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.511+0000 7f7dd86de640 1 -- 172.21.15.57:0/1249630781 shutdown_connections 2023-10-19T14:22:06.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.511+0000 7f7dd86de640 1 -- 172.21.15.57:0/1249630781 wait complete. 2023-10-19T14:22:06.524 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2023-10-19T14:22:06.525 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-19T14:22:06.525 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:22:06.528 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:22:06.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.632+0000 7fceb5e26640 1 -- 172.21.15.57:0/1038983114 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fceb013d350 msgr2=0x7fceb00cbd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:06.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.632+0000 7fceb5e26640 1 --2- 172.21.15.57:0/1038983114 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fceb013d350 0x7fceb00cbd60 secure :-1 s=READY pgs=2892 cs=0 l=1 rev1=1 crypto rx=0x7fcea4005fd0 tx=0x7fcea402bd90 comp rx=0 tx=0).stop 2023-10-19T14:22:06.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.632+0000 7fceb5e26640 1 -- 172.21.15.57:0/1038983114 shutdown_connections 2023-10-19T14:22:06.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.632+0000 7fceb5e26640 1 --2- 172.21.15.57:0/1038983114 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fceb013d350 0x7fceb00cbd60 unknown :-1 s=CLOSED pgs=2892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:06.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.632+0000 7fceb5e26640 1 --2- 172.21.15.57:0/1038983114 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fceb00cb980 0x7fceb013db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:06.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.632+0000 7fceb5e26640 1 -- 172.21.15.57:0/1038983114 >> 172.21.15.57:0/1038983114 conn(0x7fceb0069930 msgr2=0x7fceb0069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:06.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.632+0000 7fceb5e26640 1 -- 172.21.15.57:0/1038983114 shutdown_connections 2023-10-19T14:22:06.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.632+0000 7fceb5e26640 1 -- 172.21.15.57:0/1038983114 wait complete. 2023-10-19T14:22:06.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.633+0000 7fceb5e26640 1 Processor -- start 2023-10-19T14:22:06.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.633+0000 7fceb5e26640 1 -- start start 2023-10-19T14:22:06.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.634+0000 7fceb5e26640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fceb00c96b0 0x7fceb0134480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:06.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.634+0000 7fceb5e26640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fceb00cb980 0x7fceb01589a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:06.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.634+0000 7fceb5e26640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fceb01f0430 0x7fceb0151360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:06.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.634+0000 7fceb5e26640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fceb0150ed0 con 0x7fceb01f0430 2023-10-19T14:22:06.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.634+0000 7fceb5e26640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fceb01502c0 con 0x7fceb00c96b0 2023-10-19T14:22:06.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.634+0000 7fceb5e26640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fceb01505a0 con 0x7fceb00cb980 2023-10-19T14:22:06.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.634+0000 7fceaffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fceb01f0430 0x7fceb0151360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:06.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.634+0000 7fceaffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fceb01f0430 0x7fceb0151360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:49986/0 (socket says 172.21.15.57:49986) 2023-10-19T14:22:06.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.634+0000 7fceaffff640 1 -- 172.21.15.57:0/2261548784 learned_addr learned my addr 172.21.15.57:0/2261548784 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:22:06.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.634+0000 7fceaeffd640 1 --2- 172.21.15.57:0/2261548784 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fceb00cb980 0x7fceb01589a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:06.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.634+0000 7fceaf7fe640 1 --2- 172.21.15.57:0/2261548784 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fceb00c96b0 0x7fceb0134480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:06.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.635+0000 7fceaffff640 1 -- 172.21.15.57:0/2261548784 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fceb00cb980 msgr2=0x7fceb01589a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:06.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.635+0000 7fceaffff640 1 --2- 172.21.15.57:0/2261548784 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fceb00cb980 0x7fceb01589a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:06.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.635+0000 7fceaffff640 1 -- 172.21.15.57:0/2261548784 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fceb00c96b0 msgr2=0x7fceb0134480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:06.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.635+0000 7fceaffff640 1 --2- 172.21.15.57:0/2261548784 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fceb00c96b0 0x7fceb0134480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:06.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.635+0000 7fceaffff640 1 -- 172.21.15.57:0/2261548784 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcea4009560 con 0x7fceb01f0430 2023-10-19T14:22:06.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.635+0000 7fceaeffd640 1 --2- 172.21.15.57:0/2261548784 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fceb00cb980 0x7fceb01589a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:22:06.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.635+0000 7fceaf7fe640 1 --2- 172.21.15.57:0/2261548784 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fceb00c96b0 0x7fceb0134480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:22:06.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.635+0000 7fceaffff640 1 --2- 172.21.15.57:0/2261548784 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fceb01f0430 0x7fceb0151360 secure :-1 s=READY pgs=2893 cs=0 l=1 rev1=1 crypto rx=0x7fcea800b990 tx=0x7fcea800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:06.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.635+0000 7fce97fff640 1 -- 172.21.15.57:0/2261548784 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea800cba0 con 0x7fceb01f0430 2023-10-19T14:22:06.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.635+0000 7fce97fff640 1 -- 172.21.15.57:0/2261548784 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fcea800cdf0 con 0x7fceb01f0430 2023-10-19T14:22:06.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.635+0000 7fceb5e26640 1 -- 172.21.15.57:0/2261548784 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fceb0150880 con 0x7fceb01f0430 2023-10-19T14:22:06.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.635+0000 7fce97fff640 1 -- 172.21.15.57:0/2261548784 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea80146a0 con 0x7fceb01f0430 2023-10-19T14:22:06.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.635+0000 7fceb5e26640 1 -- 172.21.15.57:0/2261548784 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fceb01349c0 con 0x7fceb01f0430 2023-10-19T14:22:06.644 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.636+0000 7fceb5e26640 1 -- 172.21.15.57:0/2261548784 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fceb0151040 con 0x7fceb01f0430 2023-10-19T14:22:06.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.636+0000 7fce97fff640 1 -- 172.21.15.57:0/2261548784 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 145) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fcea8014800 con 0x7fceb01f0430 2023-10-19T14:22:06.645 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.637+0000 7fce97fff640 1 --2- 172.21.15.57:0/2261548784 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fce9c06b0b0 0x7fce9c06d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:06.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.637+0000 7fce97fff640 1 -- 172.21.15.57:0/2261548784 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(263..263 src has 1..263) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fcea8089d50 con 0x7fceb01f0430 2023-10-19T14:22:06.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.637+0000 7fceaf7fe640 1 --2- 172.21.15.57:0/2261548784 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fce9c06b0b0 0x7fce9c06d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:06.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.638+0000 7fceaf7fe640 1 --2- 172.21.15.57:0/2261548784 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fce9c06b0b0 0x7fce9c06d570 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fceb0158800 tx=0x7fcea0008040 comp rx=0 tx=0).ready entity=mgr.11446 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:06.646 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.641+0000 7fce97fff640 1 -- 172.21.15.57:0/2261548784 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fcea808d030 con 0x7fceb01f0430 2023-10-19T14:22:06.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.813+0000 7fceb5e26640 1 -- 172.21.15.57:0/2261548784 --> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fceb006bb60 con 0x7fce9c06b0b0 2023-10-19T14:22:06.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.820+0000 7fce97fff640 1 -- 172.21.15.57:0/2261548784 <== mgr.11446 v2:172.21.15.179:6832/1573806521 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+288853 (secure 0 0 0) 0x7fcea000d7e0 con 0x7fce9c06b0b0 2023-10-19T14:22:06.826 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:22:06.828 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2023-10-19T14:22:06.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.823+0000 7fceb5e26640 1 -- 172.21.15.57:0/2261548784 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fce9c06b0b0 msgr2=0x7fce9c06d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:06.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.823+0000 7fceb5e26640 1 --2- 172.21.15.57:0/2261548784 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fce9c06b0b0 0x7fce9c06d570 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fceb0158800 tx=0x7fcea0008040 comp rx=0 tx=0).stop 2023-10-19T14:22:06.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.823+0000 7fceb5e26640 1 -- 172.21.15.57:0/2261548784 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fceb01f0430 msgr2=0x7fceb0151360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:06.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.823+0000 7fceb5e26640 1 --2- 172.21.15.57:0/2261548784 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fceb01f0430 0x7fceb0151360 secure :-1 s=READY pgs=2893 cs=0 l=1 rev1=1 crypto rx=0x7fcea800b990 tx=0x7fcea800be50 comp rx=0 tx=0).stop 2023-10-19T14:22:06.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.823+0000 7fceb5e26640 1 -- 172.21.15.57:0/2261548784 shutdown_connections 2023-10-19T14:22:06.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.823+0000 7fceb5e26640 1 --2- 172.21.15.57:0/2261548784 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fce9c06b0b0 0x7fce9c06d570 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:06.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.823+0000 7fceb5e26640 1 --2- 172.21.15.57:0/2261548784 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fceb01f0430 0x7fceb0151360 unknown :-1 s=CLOSED pgs=2893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:06.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.823+0000 7fceb5e26640 1 --2- 172.21.15.57:0/2261548784 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fceb00cb980 0x7fceb01589a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:06.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.823+0000 7fceb5e26640 1 --2- 172.21.15.57:0/2261548784 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fceb00c96b0 0x7fceb0134480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:06.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.823+0000 7fceb5e26640 1 -- 172.21.15.57:0/2261548784 >> 172.21.15.57:0/2261548784 conn(0x7fceb0069930 msgr2=0x7fceb006be80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:06.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.824+0000 7fceb5e26640 1 -- 172.21.15.57:0/2261548784 shutdown_connections 2023-10-19T14:22:06.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:06.824+0000 7fceb5e26640 1 -- 172.21.15.57:0/2261548784 wait complete. 2023-10-19T14:22:06.837 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":29,"stamp":"2023-10-19T14:22:05.419632+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":591838,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":2539,"num_read_kb":5339,"num_write":697,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3257,"ondisk_log_size":3257,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":40,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":452732,"kb_used_data":26948,"kb_used_omap":12,"kb_used_meta":425651,"kb_avail":754521988,"statfs":{"total":773094113280,"available":772630515712,"internally_reserved":0,"allocated":27594752,"data_stored":14813661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":435867193},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1470,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.025206"},"pg_stats":[{"pgid":"36.3f","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.407757+0000","last_change":"2023-10-19T14:22:02.407757+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.407757+0000","last_undegraded":"2023-10-19T14:22:02.407757+0000","last_fullsized":"2023-10-19T14:22:02.407757+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:14:21.361763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":["6","3"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.3e","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.406175+0000","last_change":"2023-10-19T14:22:02.406175+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.406175+0000","last_undegraded":"2023-10-19T14:22:02.406175+0000","last_fullsized":"2023-10-19T14:22:02.406175+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:23:52.735800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.3d","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.412915+0000","last_change":"2023-10-19T14:22:02.412915+0000","last_active":"2023-10-19T14:22:02.412915+0000","last_peered":"2023-10-19T14:22:02.412915+0000","last_clean":"2023-10-19T14:22:02.412915+0000","last_became_active":"2023-10-19T14:22:02.412634+0000","last_became_peered":"2023-10-19T14:22:02.412634+0000","last_unstale":"2023-10-19T14:22:02.412915+0000","last_undegraded":"2023-10-19T14:22:02.412915+0000","last_fullsized":"2023-10-19T14:22:02.412915+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:20:32.428977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.3c","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.403761+0000","last_change":"2023-10-19T14:22:02.403761+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.403761+0000","last_undegraded":"2023-10-19T14:22:02.403761+0000","last_fullsized":"2023-10-19T14:22:02.403761+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:50:46.770884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.3b","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.409103+0000","last_change":"2023-10-19T14:22:02.409103+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.409103+0000","last_undegraded":"2023-10-19T14:22:02.409103+0000","last_fullsized":"2023-10-19T14:22:02.409103+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:03:49.877672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.3a","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.408280+0000","last_change":"2023-10-19T14:22:02.408280+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.408280+0000","last_undegraded":"2023-10-19T14:22:02.408280+0000","last_fullsized":"2023-10-19T14:22:02.408280+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:18:59.130442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.39","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.413984+0000","last_change":"2023-10-19T14:22:02.413984+0000","last_active":"2023-10-19T14:22:02.413984+0000","last_peered":"2023-10-19T14:22:02.413984+0000","last_clean":"2023-10-19T14:22:02.413984+0000","last_became_active":"2023-10-19T14:22:02.413714+0000","last_became_peered":"2023-10-19T14:22:02.413714+0000","last_unstale":"2023-10-19T14:22:02.413984+0000","last_undegraded":"2023-10-19T14:22:02.413984+0000","last_fullsized":"2023-10-19T14:22:02.413984+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:03:55.524277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.38","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.403768+0000","last_change":"2023-10-19T14:22:02.403768+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.403768+0000","last_undegraded":"2023-10-19T14:22:02.403768+0000","last_fullsized":"2023-10-19T14:22:02.403768+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:30:01.392073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.37","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.404303+0000","last_change":"2023-10-19T14:22:02.404303+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.404303+0000","last_undegraded":"2023-10-19T14:22:02.404303+0000","last_fullsized":"2023-10-19T14:22:02.404303+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:16:24.272266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.36","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.413432+0000","last_change":"2023-10-19T14:22:02.413432+0000","last_active":"2023-10-19T14:22:02.413432+0000","last_peered":"2023-10-19T14:22:02.413432+0000","last_clean":"2023-10-19T14:22:02.413432+0000","last_became_active":"2023-10-19T14:22:02.412808+0000","last_became_peered":"2023-10-19T14:22:02.412808+0000","last_unstale":"2023-10-19T14:22:02.413432+0000","last_undegraded":"2023-10-19T14:22:02.413432+0000","last_fullsized":"2023-10-19T14:22:02.413432+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T02:11:11.663612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.35","version":"0'0","reported_seq":9,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.418007+0000","last_change":"2023-10-19T14:22:02.413336+0000","last_active":"2023-10-19T14:22:02.418007+0000","last_peered":"2023-10-19T14:22:02.418007+0000","last_clean":"2023-10-19T14:22:02.418007+0000","last_became_active":"2023-10-19T14:22:02.412942+0000","last_became_peered":"2023-10-19T14:22:02.412942+0000","last_unstale":"2023-10-19T14:22:02.418007+0000","last_undegraded":"2023-10-19T14:22:02.418007+0000","last_fullsized":"2023-10-19T14:22:02.418007+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:37:40.908777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.34","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.406632+0000","last_change":"2023-10-19T14:22:02.406632+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.406632+0000","last_undegraded":"2023-10-19T14:22:02.406632+0000","last_fullsized":"2023-10-19T14:22:02.406632+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:30:39.654226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":["6","3"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.33","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.403708+0000","last_change":"2023-10-19T14:22:02.403708+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.403708+0000","last_undegraded":"2023-10-19T14:22:02.403708+0000","last_fullsized":"2023-10-19T14:22:02.403708+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:39:38.146246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.32","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.403725+0000","last_change":"2023-10-19T14:22:02.403725+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.403725+0000","last_undegraded":"2023-10-19T14:22:02.403725+0000","last_fullsized":"2023-10-19T14:22:02.403725+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:10:47.778168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.31","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.403277+0000","last_change":"2023-10-19T14:22:02.403277+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.403277+0000","last_undegraded":"2023-10-19T14:22:02.403277+0000","last_fullsized":"2023-10-19T14:22:02.403277+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:15:36.803736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.30","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.420492+0000","last_change":"2023-10-19T14:22:02.420492+0000","last_active":"2023-10-19T14:22:02.420492+0000","last_peered":"2023-10-19T14:22:02.420492+0000","last_clean":"2023-10-19T14:22:02.420492+0000","last_became_active":"2023-10-19T14:22:02.420210+0000","last_became_peered":"2023-10-19T14:22:02.420210+0000","last_unstale":"2023-10-19T14:22:02.420492+0000","last_undegraded":"2023-10-19T14:22:02.420492+0000","last_fullsized":"2023-10-19T14:22:02.420492+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:38:11.628588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.2f","version":"0'0","reported_seq":9,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.420525+0000","last_change":"2023-10-19T14:22:02.415526+0000","last_active":"2023-10-19T14:22:02.420525+0000","last_peered":"2023-10-19T14:22:02.420525+0000","last_clean":"2023-10-19T14:22:02.420525+0000","last_became_active":"2023-10-19T14:22:02.415243+0000","last_became_peered":"2023-10-19T14:22:02.415243+0000","last_unstale":"2023-10-19T14:22:02.420525+0000","last_undegraded":"2023-10-19T14:22:02.420525+0000","last_fullsized":"2023-10-19T14:22:02.420525+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:25:08.178761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.2e","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.405331+0000","last_change":"2023-10-19T14:22:02.405331+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.405331+0000","last_undegraded":"2023-10-19T14:22:02.405331+0000","last_fullsized":"2023-10-19T14:22:02.405331+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:49:42.930618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.2d","version":"0'0","reported_seq":9,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.420730+0000","last_change":"2023-10-19T14:22:02.418292+0000","last_active":"2023-10-19T14:22:02.420730+0000","last_peered":"2023-10-19T14:22:02.420730+0000","last_clean":"2023-10-19T14:22:02.420730+0000","last_became_active":"2023-10-19T14:22:02.415107+0000","last_became_peered":"2023-10-19T14:22:02.415107+0000","last_unstale":"2023-10-19T14:22:02.420730+0000","last_undegraded":"2023-10-19T14:22:02.420730+0000","last_fullsized":"2023-10-19T14:22:02.420730+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:22:21.202909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.2c","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.421040+0000","last_change":"2023-10-19T14:22:02.421040+0000","last_active":"2023-10-19T14:22:02.421040+0000","last_peered":"2023-10-19T14:22:02.421040+0000","last_clean":"2023-10-19T14:22:02.421040+0000","last_became_active":"2023-10-19T14:22:02.420785+0000","last_became_peered":"2023-10-19T14:22:02.420785+0000","last_unstale":"2023-10-19T14:22:02.421040+0000","last_undegraded":"2023-10-19T14:22:02.421040+0000","last_fullsized":"2023-10-19T14:22:02.421040+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:48:32.972469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.2b","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.418617+0000","last_change":"2023-10-19T14:22:02.418617+0000","last_active":"2023-10-19T14:22:02.418617+0000","last_peered":"2023-10-19T14:22:02.418617+0000","last_clean":"2023-10-19T14:22:02.418617+0000","last_became_active":"2023-10-19T14:22:02.418348+0000","last_became_peered":"2023-10-19T14:22:02.418348+0000","last_unstale":"2023-10-19T14:22:02.418617+0000","last_undegraded":"2023-10-19T14:22:02.418617+0000","last_fullsized":"2023-10-19T14:22:02.418617+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:03:48.649163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.2a","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.399641+0000","last_change":"2023-10-19T14:22:02.399641+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.399641+0000","last_undegraded":"2023-10-19T14:22:02.399641+0000","last_fullsized":"2023-10-19T14:22:02.399641+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:35:08.037582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.29","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.405143+0000","last_change":"2023-10-19T14:22:02.405143+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.405143+0000","last_undegraded":"2023-10-19T14:22:02.405143+0000","last_fullsized":"2023-10-19T14:22:02.405143+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:09:54.096828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.28","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.398717+0000","last_change":"2023-10-19T14:22:02.398717+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.398717+0000","last_undegraded":"2023-10-19T14:22:02.398717+0000","last_fullsized":"2023-10-19T14:22:02.398717+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:35:36.604007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.27","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.417381+0000","last_change":"2023-10-19T14:22:02.417381+0000","last_active":"2023-10-19T14:22:02.417381+0000","last_peered":"2023-10-19T14:22:02.417381+0000","last_clean":"2023-10-19T14:22:02.417381+0000","last_became_active":"2023-10-19T14:22:02.417103+0000","last_became_peered":"2023-10-19T14:22:02.417103+0000","last_unstale":"2023-10-19T14:22:02.417381+0000","last_undegraded":"2023-10-19T14:22:02.417381+0000","last_fullsized":"2023-10-19T14:22:02.417381+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:05:25.934207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.26","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.410522+0000","last_change":"2023-10-19T14:22:02.410522+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.410522+0000","last_undegraded":"2023-10-19T14:22:02.410522+0000","last_fullsized":"2023-10-19T14:22:02.410522+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:41:53.947719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.24","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.407514+0000","last_change":"2023-10-19T14:22:02.407514+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.407514+0000","last_undegraded":"2023-10-19T14:22:02.407514+0000","last_fullsized":"2023-10-19T14:22:02.407514+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:14:37.969686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.23","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.398963+0000","last_change":"2023-10-19T14:22:02.398963+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.398963+0000","last_undegraded":"2023-10-19T14:22:02.398963+0000","last_fullsized":"2023-10-19T14:22:02.398963+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:57:49.145921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.22","version":"0'0","reported_seq":9,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.404880+0000","last_change":"2023-10-19T14:22:02.402715+0000","last_active":"2023-10-19T14:22:02.404880+0000","last_peered":"2023-10-19T14:22:02.404880+0000","last_clean":"2023-10-19T14:22:02.404880+0000","last_became_active":"2023-10-19T14:22:02.402451+0000","last_became_peered":"2023-10-19T14:22:02.402451+0000","last_unstale":"2023-10-19T14:22:02.404880+0000","last_undegraded":"2023-10-19T14:22:02.404880+0000","last_fullsized":"2023-10-19T14:22:02.404880+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:42:44.848665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.21","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.399356+0000","last_change":"2023-10-19T14:22:02.399356+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.399356+0000","last_undegraded":"2023-10-19T14:22:02.399356+0000","last_fullsized":"2023-10-19T14:22:02.399356+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:45:44.957902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.20","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.403114+0000","last_change":"2023-10-19T14:22:02.403114+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.403114+0000","last_undegraded":"2023-10-19T14:22:02.403114+0000","last_fullsized":"2023-10-19T14:22:02.403114+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T02:18:23.251534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.18","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.398614+0000","last_change":"2023-10-19T14:22:02.398614+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.398614+0000","last_undegraded":"2023-10-19T14:22:02.398614+0000","last_fullsized":"2023-10-19T14:22:02.398614+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:31:51.736865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.1f","version":"262'4","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.404439+0000","last_change":"2023-10-19T14:22:01.414254+0000","last_active":"2023-10-19T14:22:02.404439+0000","last_peered":"2023-10-19T14:22:02.404439+0000","last_clean":"2023-10-19T14:22:02.404439+0000","last_became_active":"2023-10-19T14:22:01.413084+0000","last_became_peered":"2023-10-19T14:22:01.413084+0000","last_unstale":"2023-10-19T14:22:02.404439+0000","last_undegraded":"2023-10-19T14:22:02.404439+0000","last_fullsized":"2023-10-19T14:22:02.404439+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:04:17.277980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.19","version":"0'0","reported_seq":9,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.416536+0000","last_change":"2023-10-19T14:22:02.412925+0000","last_active":"2023-10-19T14:22:02.416536+0000","last_peered":"2023-10-19T14:22:02.416536+0000","last_clean":"2023-10-19T14:22:02.416536+0000","last_became_active":"2023-10-19T14:22:02.412639+0000","last_became_peered":"2023-10-19T14:22:02.412639+0000","last_unstale":"2023-10-19T14:22:02.416536+0000","last_undegraded":"2023-10-19T14:22:02.416536+0000","last_fullsized":"2023-10-19T14:22:02.416536+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:14:38.697589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.1e","version":"0'0","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.399308+0000","last_change":"2023-10-19T14:22:01.410546+0000","last_active":"2023-10-19T14:22:02.399308+0000","last_peered":"2023-10-19T14:22:02.399308+0000","last_clean":"2023-10-19T14:22:02.399308+0000","last_became_active":"2023-10-19T14:22:01.410263+0000","last_became_peered":"2023-10-19T14:22:01.410263+0000","last_unstale":"2023-10-19T14:22:02.399308+0000","last_undegraded":"2023-10-19T14:22:02.399308+0000","last_fullsized":"2023-10-19T14:22:02.399308+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:02:12.137179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.1a","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.406083+0000","last_change":"2023-10-19T14:22:02.406083+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.406083+0000","last_undegraded":"2023-10-19T14:22:02.406083+0000","last_fullsized":"2023-10-19T14:22:02.406083+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:07:50.835228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.1d","version":"262'2","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.400756+0000","last_change":"2023-10-19T14:22:01.417975+0000","last_active":"2023-10-19T14:22:02.400756+0000","last_peered":"2023-10-19T14:22:02.400756+0000","last_clean":"2023-10-19T14:22:02.400756+0000","last_became_active":"2023-10-19T14:22:01.417704+0000","last_became_peered":"2023-10-19T14:22:01.417704+0000","last_unstale":"2023-10-19T14:22:02.400756+0000","last_undegraded":"2023-10-19T14:22:02.400756+0000","last_fullsized":"2023-10-19T14:22:02.400756+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:56:00.152228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.1b","version":"0'0","reported_seq":9,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.416951+0000","last_change":"2023-10-19T14:22:02.412219+0000","last_active":"2023-10-19T14:22:02.416951+0000","last_peered":"2023-10-19T14:22:02.416951+0000","last_clean":"2023-10-19T14:22:02.416951+0000","last_became_active":"2023-10-19T14:22:02.411959+0000","last_became_peered":"2023-10-19T14:22:02.411959+0000","last_unstale":"2023-10-19T14:22:02.416951+0000","last_undegraded":"2023-10-19T14:22:02.416951+0000","last_fullsized":"2023-10-19T14:22:02.416951+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:54:56.271440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.1c","version":"262'2","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.400340+0000","last_change":"2023-10-19T14:22:01.419018+0000","last_active":"2023-10-19T14:22:02.400340+0000","last_peered":"2023-10-19T14:22:02.400340+0000","last_clean":"2023-10-19T14:22:02.400340+0000","last_became_active":"2023-10-19T14:22:01.418769+0000","last_became_peered":"2023-10-19T14:22:01.418769+0000","last_unstale":"2023-10-19T14:22:02.400340+0000","last_undegraded":"2023-10-19T14:22:02.400340+0000","last_fullsized":"2023-10-19T14:22:02.400340+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:59:43.263869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.1c","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.402955+0000","last_change":"2023-10-19T14:22:02.402955+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.402955+0000","last_undegraded":"2023-10-19T14:22:02.402955+0000","last_fullsized":"2023-10-19T14:22:02.402955+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:03:27.890218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.1b","version":"262'1","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.409720+0000","last_change":"2023-10-19T14:22:01.413953+0000","last_active":"2023-10-19T14:22:02.409720+0000","last_peered":"2023-10-19T14:22:02.409720+0000","last_clean":"2023-10-19T14:22:02.409720+0000","last_became_active":"2023-10-19T14:22:01.413689+0000","last_became_peered":"2023-10-19T14:22:01.413689+0000","last_unstale":"2023-10-19T14:22:02.409720+0000","last_undegraded":"2023-10-19T14:22:02.409720+0000","last_fullsized":"2023-10-19T14:22:02.409720+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:33:45.430960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.1d","version":"0'0","reported_seq":6,"reported_epoch":262,"state":"active","last_fresh":"2023-10-19T14:22:02.413928+0000","last_change":"2023-10-19T14:22:02.413862+0000","last_active":"2023-10-19T14:22:02.413928+0000","last_peered":"2023-10-19T14:22:02.413928+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"2023-10-19T14:22:02.413862+0000","last_became_peered":"2023-10-19T14:22:02.413862+0000","last_unstale":"2023-10-19T14:22:02.413928+0000","last_undegraded":"2023-10-19T14:22:02.413928+0000","last_fullsized":"2023-10-19T14:22:02.413928+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:20:55.394812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.1a","version":"0'0","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.408577+0000","last_change":"2023-10-19T14:22:01.409991+0000","last_active":"2023-10-19T14:22:02.408577+0000","last_peered":"2023-10-19T14:22:02.408577+0000","last_clean":"2023-10-19T14:22:02.408577+0000","last_became_active":"2023-10-19T14:22:01.409510+0000","last_became_peered":"2023-10-19T14:22:01.409510+0000","last_unstale":"2023-10-19T14:22:02.408577+0000","last_undegraded":"2023-10-19T14:22:02.408577+0000","last_fullsized":"2023-10-19T14:22:02.408577+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:42:53.919625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.1e","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.412327+0000","last_change":"2023-10-19T14:22:02.412327+0000","last_active":"2023-10-19T14:22:02.412327+0000","last_peered":"2023-10-19T14:22:02.412327+0000","last_clean":"2023-10-19T14:22:02.412327+0000","last_became_active":"2023-10-19T14:22:02.412057+0000","last_became_peered":"2023-10-19T14:22:02.412057+0000","last_unstale":"2023-10-19T14:22:02.412327+0000","last_undegraded":"2023-10-19T14:22:02.412327+0000","last_fullsized":"2023-10-19T14:22:02.412327+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:32:30.545638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.19","version":"0'0","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.399288+0000","last_change":"2023-10-19T14:22:01.416901+0000","last_active":"2023-10-19T14:22:02.399288+0000","last_peered":"2023-10-19T14:22:02.399288+0000","last_clean":"2023-10-19T14:22:02.399288+0000","last_became_active":"2023-10-19T14:22:01.416656+0000","last_became_peered":"2023-10-19T14:22:01.416656+0000","last_unstale":"2023-10-19T14:22:02.399288+0000","last_undegraded":"2023-10-19T14:22:02.399288+0000","last_fullsized":"2023-10-19T14:22:02.399288+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:27:36.174418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.1f","version":"0'0","reported_seq":9,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.420545+0000","last_change":"2023-10-19T14:22:02.418425+0000","last_active":"2023-10-19T14:22:02.420545+0000","last_peered":"2023-10-19T14:22:02.420545+0000","last_clean":"2023-10-19T14:22:02.420545+0000","last_became_active":"2023-10-19T14:22:02.417278+0000","last_became_peered":"2023-10-19T14:22:02.417278+0000","last_unstale":"2023-10-19T14:22:02.420545+0000","last_undegraded":"2023-10-19T14:22:02.420545+0000","last_fullsized":"2023-10-19T14:22:02.420545+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:04:24.989932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.18","version":"0'0","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.404196+0000","last_change":"2023-10-19T14:22:01.412557+0000","last_active":"2023-10-19T14:22:02.404196+0000","last_peered":"2023-10-19T14:22:02.404196+0000","last_clean":"2023-10-19T14:22:02.404196+0000","last_became_active":"2023-10-19T14:22:01.412115+0000","last_became_peered":"2023-10-19T14:22:01.412115+0000","last_unstale":"2023-10-19T14:22:02.404196+0000","last_undegraded":"2023-10-19T14:22:02.404196+0000","last_fullsized":"2023-10-19T14:22:02.404196+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:54:33.650542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.10","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.402706+0000","last_change":"2023-10-19T14:22:02.402706+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.402706+0000","last_undegraded":"2023-10-19T14:22:02.402706+0000","last_fullsized":"2023-10-19T14:22:02.402706+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:45:46.279692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.17","version":"0'0","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.415002+0000","last_change":"2023-10-19T14:22:01.412962+0000","last_active":"2023-10-19T14:22:02.415002+0000","last_peered":"2023-10-19T14:22:02.415002+0000","last_clean":"2023-10-19T14:22:02.415002+0000","last_became_active":"2023-10-19T14:22:01.412680+0000","last_became_peered":"2023-10-19T14:22:01.412680+0000","last_unstale":"2023-10-19T14:22:02.415002+0000","last_undegraded":"2023-10-19T14:22:02.415002+0000","last_fullsized":"2023-10-19T14:22:02.415002+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:03:54.917148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.11","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.405750+0000","last_change":"2023-10-19T14:22:02.405750+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.405750+0000","last_undegraded":"2023-10-19T14:22:02.405750+0000","last_fullsized":"2023-10-19T14:22:02.405750+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:06:30.359305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.16","version":"0'0","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.399221+0000","last_change":"2023-10-19T14:22:01.414464+0000","last_active":"2023-10-19T14:22:02.399221+0000","last_peered":"2023-10-19T14:22:02.399221+0000","last_clean":"2023-10-19T14:22:02.399221+0000","last_became_active":"2023-10-19T14:22:01.413414+0000","last_became_peered":"2023-10-19T14:22:01.413414+0000","last_unstale":"2023-10-19T14:22:02.399221+0000","last_undegraded":"2023-10-19T14:22:02.399221+0000","last_fullsized":"2023-10-19T14:22:02.399221+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:46:22.763844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.12","version":"0'0","reported_seq":9,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.413606+0000","last_change":"2023-10-19T14:22:02.410248+0000","last_active":"2023-10-19T14:22:02.413606+0000","last_peered":"2023-10-19T14:22:02.413606+0000","last_clean":"2023-10-19T14:22:02.413606+0000","last_became_active":"2023-10-19T14:22:02.409982+0000","last_became_peered":"2023-10-19T14:22:02.409982+0000","last_unstale":"2023-10-19T14:22:02.413606+0000","last_undegraded":"2023-10-19T14:22:02.413606+0000","last_fullsized":"2023-10-19T14:22:02.413606+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:27:43.397970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.15","version":"262'3","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.414741+0000","last_change":"2023-10-19T14:22:01.409543+0000","last_active":"2023-10-19T14:22:02.414741+0000","last_peered":"2023-10-19T14:22:02.414741+0000","last_clean":"2023-10-19T14:22:02.414741+0000","last_became_active":"2023-10-19T14:22:01.409293+0000","last_became_peered":"2023-10-19T14:22:01.409293+0000","last_unstale":"2023-10-19T14:22:02.414741+0000","last_undegraded":"2023-10-19T14:22:02.414741+0000","last_fullsized":"2023-10-19T14:22:02.414741+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:12:16.850155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.13","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.408938+0000","last_change":"2023-10-19T14:22:02.408938+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.408938+0000","last_undegraded":"2023-10-19T14:22:02.408938+0000","last_fullsized":"2023-10-19T14:22:02.408938+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:22:40.195315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.14","version":"262'5","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.404183+0000","last_change":"2023-10-19T14:22:01.411386+0000","last_active":"2023-10-19T14:22:02.404183+0000","last_peered":"2023-10-19T14:22:02.404183+0000","last_clean":"2023-10-19T14:22:02.404183+0000","last_became_active":"2023-10-19T14:22:01.411111+0000","last_became_peered":"2023-10-19T14:22:01.411111+0000","last_unstale":"2023-10-19T14:22:02.404183+0000","last_undegraded":"2023-10-19T14:22:02.404183+0000","last_fullsized":"2023-10-19T14:22:02.404183+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:10:26.534502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.14","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.416878+0000","last_change":"2023-10-19T14:22:02.416878+0000","last_active":"2023-10-19T14:22:02.416878+0000","last_peered":"2023-10-19T14:22:02.416878+0000","last_clean":"2023-10-19T14:22:02.416878+0000","last_became_active":"2023-10-19T14:22:02.416633+0000","last_became_peered":"2023-10-19T14:22:02.416633+0000","last_unstale":"2023-10-19T14:22:02.416878+0000","last_undegraded":"2023-10-19T14:22:02.416878+0000","last_fullsized":"2023-10-19T14:22:02.416878+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:56:41.835345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.13","version":"262'2","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.399368+0000","last_change":"2023-10-19T14:22:01.410762+0000","last_active":"2023-10-19T14:22:02.399368+0000","last_peered":"2023-10-19T14:22:02.399368+0000","last_clean":"2023-10-19T14:22:02.399368+0000","last_became_active":"2023-10-19T14:22:01.410507+0000","last_became_peered":"2023-10-19T14:22:01.410507+0000","last_unstale":"2023-10-19T14:22:02.399368+0000","last_undegraded":"2023-10-19T14:22:02.399368+0000","last_fullsized":"2023-10-19T14:22:02.399368+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:38:13.918889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.15","version":"0'0","reported_seq":6,"reported_epoch":262,"state":"active","last_fresh":"2023-10-19T14:22:02.414429+0000","last_change":"2023-10-19T14:22:02.414358+0000","last_active":"2023-10-19T14:22:02.414429+0000","last_peered":"2023-10-19T14:22:02.414429+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"2023-10-19T14:22:02.414358+0000","last_became_peered":"2023-10-19T14:22:02.414358+0000","last_unstale":"2023-10-19T14:22:02.414429+0000","last_undegraded":"2023-10-19T14:22:02.414429+0000","last_fullsized":"2023-10-19T14:22:02.414429+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:02:05.922124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.12","version":"0'0","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.409323+0000","last_change":"2023-10-19T14:22:01.413870+0000","last_active":"2023-10-19T14:22:02.409323+0000","last_peered":"2023-10-19T14:22:02.409323+0000","last_clean":"2023-10-19T14:22:02.409323+0000","last_became_active":"2023-10-19T14:22:01.413532+0000","last_became_peered":"2023-10-19T14:22:01.413532+0000","last_unstale":"2023-10-19T14:22:02.409323+0000","last_undegraded":"2023-10-19T14:22:02.409323+0000","last_fullsized":"2023-10-19T14:22:02.409323+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:47:48.522113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.16","version":"0'0","reported_seq":9,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.421387+0000","last_change":"2023-10-19T14:22:02.419152+0000","last_active":"2023-10-19T14:22:02.421387+0000","last_peered":"2023-10-19T14:22:02.421387+0000","last_clean":"2023-10-19T14:22:02.421387+0000","last_became_active":"2023-10-19T14:22:02.417515+0000","last_became_peered":"2023-10-19T14:22:02.417515+0000","last_unstale":"2023-10-19T14:22:02.421387+0000","last_undegraded":"2023-10-19T14:22:02.421387+0000","last_fullsized":"2023-10-19T14:22:02.421387+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:39:40.912960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.11","version":"0'0","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.409983+0000","last_change":"2023-10-19T14:22:01.416474+0000","last_active":"2023-10-19T14:22:02.409983+0000","last_peered":"2023-10-19T14:22:02.409983+0000","last_clean":"2023-10-19T14:22:02.409983+0000","last_became_active":"2023-10-19T14:22:01.416200+0000","last_became_peered":"2023-10-19T14:22:01.416200+0000","last_unstale":"2023-10-19T14:22:02.409983+0000","last_undegraded":"2023-10-19T14:22:02.409983+0000","last_fullsized":"2023-10-19T14:22:02.409983+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:55:39.137949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.17","version":"0'0","reported_seq":9,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.421450+0000","last_change":"2023-10-19T14:22:02.419116+0000","last_active":"2023-10-19T14:22:02.421450+0000","last_peered":"2023-10-19T14:22:02.421450+0000","last_clean":"2023-10-19T14:22:02.421450+0000","last_became_active":"2023-10-19T14:22:02.418655+0000","last_became_peered":"2023-10-19T14:22:02.418655+0000","last_unstale":"2023-10-19T14:22:02.421450+0000","last_undegraded":"2023-10-19T14:22:02.421450+0000","last_fullsized":"2023-10-19T14:22:02.421450+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:28:48.304947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.10","version":"262'8","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.402744+0000","last_change":"2023-10-19T14:22:01.415486+0000","last_active":"2023-10-19T14:22:02.402744+0000","last_peered":"2023-10-19T14:22:02.402744+0000","last_clean":"2023-10-19T14:22:02.402744+0000","last_became_active":"2023-10-19T14:22:01.415192+0000","last_became_peered":"2023-10-19T14:22:01.415192+0000","last_unstale":"2023-10-19T14:22:02.402744+0000","last_undegraded":"2023-10-19T14:22:02.402744+0000","last_fullsized":"2023-10-19T14:22:02.402744+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:45:03.326043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.8","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.407065+0000","last_change":"2023-10-19T14:22:02.407065+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.407065+0000","last_undegraded":"2023-10-19T14:22:02.407065+0000","last_fullsized":"2023-10-19T14:22:02.407065+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:07:55.991807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.f","version":"262'1","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.417942+0000","last_change":"2023-10-19T14:22:01.417521+0000","last_active":"2023-10-19T14:22:02.417942+0000","last_peered":"2023-10-19T14:22:02.417942+0000","last_clean":"2023-10-19T14:22:02.417942+0000","last_became_active":"2023-10-19T14:22:01.417209+0000","last_became_peered":"2023-10-19T14:22:01.417209+0000","last_unstale":"2023-10-19T14:22:02.417942+0000","last_undegraded":"2023-10-19T14:22:02.417942+0000","last_fullsized":"2023-10-19T14:22:02.417942+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:43:26.494066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.9","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.398780+0000","last_change":"2023-10-19T14:22:02.398780+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.398780+0000","last_undegraded":"2023-10-19T14:22:02.398780+0000","last_fullsized":"2023-10-19T14:22:02.398780+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:40:53.511724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.e","version":"0'0","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.411175+0000","last_change":"2023-10-19T14:22:01.412531+0000","last_active":"2023-10-19T14:22:02.411175+0000","last_peered":"2023-10-19T14:22:02.411175+0000","last_clean":"2023-10-19T14:22:02.411175+0000","last_became_active":"2023-10-19T14:22:01.412209+0000","last_became_peered":"2023-10-19T14:22:01.412209+0000","last_unstale":"2023-10-19T14:22:02.411175+0000","last_undegraded":"2023-10-19T14:22:02.411175+0000","last_fullsized":"2023-10-19T14:22:02.411175+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:45:03.326043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.a","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.407858+0000","last_change":"2023-10-19T14:22:02.407858+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.407858+0000","last_undegraded":"2023-10-19T14:22:02.407858+0000","last_fullsized":"2023-10-19T14:22:02.407858+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:31:43.355728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.d","version":"262'1","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.400892+0000","last_change":"2023-10-19T14:22:01.415741+0000","last_active":"2023-10-19T14:22:02.400892+0000","last_peered":"2023-10-19T14:22:02.400892+0000","last_clean":"2023-10-19T14:22:02.400892+0000","last_became_active":"2023-10-19T14:22:01.415385+0000","last_became_peered":"2023-10-19T14:22:01.415385+0000","last_unstale":"2023-10-19T14:22:02.400892+0000","last_undegraded":"2023-10-19T14:22:02.400892+0000","last_fullsized":"2023-10-19T14:22:02.400892+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:34:06.962782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.b","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.413768+0000","last_change":"2023-10-19T14:22:02.413768+0000","last_active":"2023-10-19T14:22:02.413768+0000","last_peered":"2023-10-19T14:22:02.413768+0000","last_clean":"2023-10-19T14:22:02.413768+0000","last_became_active":"2023-10-19T14:22:02.413498+0000","last_became_peered":"2023-10-19T14:22:02.413498+0000","last_unstale":"2023-10-19T14:22:02.413768+0000","last_undegraded":"2023-10-19T14:22:02.413768+0000","last_fullsized":"2023-10-19T14:22:02.413768+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:09:01.416730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.c","version":"0'0","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.399235+0000","last_change":"2023-10-19T14:22:01.411380+0000","last_active":"2023-10-19T14:22:02.399235+0000","last_peered":"2023-10-19T14:22:02.399235+0000","last_clean":"2023-10-19T14:22:02.399235+0000","last_became_active":"2023-10-19T14:22:01.411092+0000","last_became_peered":"2023-10-19T14:22:01.411092+0000","last_unstale":"2023-10-19T14:22:02.399235+0000","last_undegraded":"2023-10-19T14:22:02.399235+0000","last_fullsized":"2023-10-19T14:22:02.399235+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:01:14.941331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.c","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.413516+0000","last_change":"2023-10-19T14:22:02.413516+0000","last_active":"2023-10-19T14:22:02.413516+0000","last_peered":"2023-10-19T14:22:02.413516+0000","last_clean":"2023-10-19T14:22:02.413516+0000","last_became_active":"2023-10-19T14:22:02.413230+0000","last_became_peered":"2023-10-19T14:22:02.413230+0000","last_unstale":"2023-10-19T14:22:02.413516+0000","last_undegraded":"2023-10-19T14:22:02.413516+0000","last_fullsized":"2023-10-19T14:22:02.413516+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:22:42.002875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.b","version":"262'2","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.399530+0000","last_change":"2023-10-19T14:22:01.409927+0000","last_active":"2023-10-19T14:22:02.399530+0000","last_peered":"2023-10-19T14:22:02.399530+0000","last_clean":"2023-10-19T14:22:02.399530+0000","last_became_active":"2023-10-19T14:22:01.409617+0000","last_became_peered":"2023-10-19T14:22:01.409617+0000","last_unstale":"2023-10-19T14:22:02.399530+0000","last_undegraded":"2023-10-19T14:22:02.399530+0000","last_fullsized":"2023-10-19T14:22:02.399530+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:38:30.217600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.d","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.399329+0000","last_change":"2023-10-19T14:22:02.399329+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.399329+0000","last_undegraded":"2023-10-19T14:22:02.399329+0000","last_fullsized":"2023-10-19T14:22:02.399329+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:17:41.707119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.a","version":"0'0","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.404454+0000","last_change":"2023-10-19T14:22:01.417576+0000","last_active":"2023-10-19T14:22:02.404454+0000","last_peered":"2023-10-19T14:22:02.404454+0000","last_clean":"2023-10-19T14:22:02.404454+0000","last_became_active":"2023-10-19T14:22:01.417230+0000","last_became_peered":"2023-10-19T14:22:01.417230+0000","last_unstale":"2023-10-19T14:22:02.404454+0000","last_undegraded":"2023-10-19T14:22:02.404454+0000","last_fullsized":"2023-10-19T14:22:02.404454+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:23:00.582706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.e","version":"0'0","reported_seq":9,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.421407+0000","last_change":"2023-10-19T14:22:02.419344+0000","last_active":"2023-10-19T14:22:02.421407+0000","last_peered":"2023-10-19T14:22:02.421407+0000","last_clean":"2023-10-19T14:22:02.421407+0000","last_became_active":"2023-10-19T14:22:02.417238+0000","last_became_peered":"2023-10-19T14:22:02.417238+0000","last_unstale":"2023-10-19T14:22:02.421407+0000","last_undegraded":"2023-10-19T14:22:02.421407+0000","last_fullsized":"2023-10-19T14:22:02.421407+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:19:36.485520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.9","version":"0'0","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.400018+0000","last_change":"2023-10-19T14:22:01.413457+0000","last_active":"2023-10-19T14:22:02.400018+0000","last_peered":"2023-10-19T14:22:02.400018+0000","last_clean":"2023-10-19T14:22:02.400018+0000","last_became_active":"2023-10-19T14:22:01.413179+0000","last_became_peered":"2023-10-19T14:22:01.413179+0000","last_unstale":"2023-10-19T14:22:02.400018+0000","last_undegraded":"2023-10-19T14:22:02.400018+0000","last_fullsized":"2023-10-19T14:22:02.400018+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:34:30.334044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.f","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.409049+0000","last_change":"2023-10-19T14:22:02.409049+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.409049+0000","last_undegraded":"2023-10-19T14:22:02.409049+0000","last_fullsized":"2023-10-19T14:22:02.409049+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:44:13.928436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.8","version":"0'0","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.399188+0000","last_change":"2023-10-19T14:22:01.413949+0000","last_active":"2023-10-19T14:22:02.399188+0000","last_peered":"2023-10-19T14:22:02.399188+0000","last_clean":"2023-10-19T14:22:02.399188+0000","last_became_active":"2023-10-19T14:22:01.413670+0000","last_became_peered":"2023-10-19T14:22:01.413670+0000","last_unstale":"2023-10-19T14:22:02.399188+0000","last_undegraded":"2023-10-19T14:22:02.399188+0000","last_fullsized":"2023-10-19T14:22:02.399188+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:33:35.792891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.0","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.402684+0000","last_change":"2023-10-19T14:22:02.402684+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.402684+0000","last_undegraded":"2023-10-19T14:22:02.402684+0000","last_fullsized":"2023-10-19T14:22:02.402684+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:40:48.301290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.7","version":"262'2","reported_seq":13,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.413481+0000","last_change":"2023-10-19T14:22:01.412421+0000","last_active":"2023-10-19T14:22:02.413481+0000","last_peered":"2023-10-19T14:22:02.413481+0000","last_clean":"2023-10-19T14:22:02.413481+0000","last_became_active":"2023-10-19T14:22:01.412153+0000","last_became_peered":"2023-10-19T14:22:01.412153+0000","last_unstale":"2023-10-19T14:22:02.413481+0000","last_undegraded":"2023-10-19T14:22:02.413481+0000","last_fullsized":"2023-10-19T14:22:02.413481+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:38:47.923739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.1","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.419843+0000","last_change":"2023-10-19T14:22:02.419843+0000","last_active":"2023-10-19T14:22:02.419843+0000","last_peered":"2023-10-19T14:22:02.419843+0000","last_clean":"2023-10-19T14:22:02.419843+0000","last_became_active":"2023-10-19T14:22:02.417481+0000","last_became_peered":"2023-10-19T14:22:02.417481+0000","last_unstale":"2023-10-19T14:22:02.419843+0000","last_undegraded":"2023-10-19T14:22:02.419843+0000","last_fullsized":"2023-10-19T14:22:02.419843+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:12:12.862561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.6","version":"0'0","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.409952+0000","last_change":"2023-10-19T14:22:01.412088+0000","last_active":"2023-10-19T14:22:02.409952+0000","last_peered":"2023-10-19T14:22:02.409952+0000","last_clean":"2023-10-19T14:22:02.409952+0000","last_became_active":"2023-10-19T14:22:01.411646+0000","last_became_peered":"2023-10-19T14:22:01.411646+0000","last_unstale":"2023-10-19T14:22:02.409952+0000","last_undegraded":"2023-10-19T14:22:02.409952+0000","last_fullsized":"2023-10-19T14:22:02.409952+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:48:08.338501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.2","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.403340+0000","last_change":"2023-10-19T14:22:02.403340+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.403340+0000","last_undegraded":"2023-10-19T14:22:02.403340+0000","last_fullsized":"2023-10-19T14:22:02.403340+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:42:16.203127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.5","version":"0'0","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.401015+0000","last_change":"2023-10-19T14:22:01.417906+0000","last_active":"2023-10-19T14:22:02.401015+0000","last_peered":"2023-10-19T14:22:02.401015+0000","last_clean":"2023-10-19T14:22:02.401015+0000","last_became_active":"2023-10-19T14:22:01.417614+0000","last_became_peered":"2023-10-19T14:22:01.417614+0000","last_unstale":"2023-10-19T14:22:02.401015+0000","last_undegraded":"2023-10-19T14:22:02.401015+0000","last_fullsized":"2023-10-19T14:22:02.401015+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:26:05.681833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.3","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.417066+0000","last_change":"2023-10-19T14:22:02.417066+0000","last_active":"2023-10-19T14:22:02.417066+0000","last_peered":"2023-10-19T14:22:02.417066+0000","last_clean":"2023-10-19T14:22:02.417066+0000","last_became_active":"2023-10-19T14:22:02.416770+0000","last_became_peered":"2023-10-19T14:22:02.416770+0000","last_unstale":"2023-10-19T14:22:02.417066+0000","last_undegraded":"2023-10-19T14:22:02.417066+0000","last_fullsized":"2023-10-19T14:22:02.417066+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:25:15.074214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.4","version":"0'0","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.409766+0000","last_change":"2023-10-19T14:22:01.412056+0000","last_active":"2023-10-19T14:22:02.409766+0000","last_peered":"2023-10-19T14:22:02.409766+0000","last_clean":"2023-10-19T14:22:02.409766+0000","last_became_active":"2023-10-19T14:22:01.411786+0000","last_became_peered":"2023-10-19T14:22:01.411786+0000","last_unstale":"2023-10-19T14:22:02.409766+0000","last_undegraded":"2023-10-19T14:22:02.409766+0000","last_fullsized":"2023-10-19T14:22:02.409766+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:40:51.932206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.4","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.403966+0000","last_change":"2023-10-19T14:22:02.403966+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.403966+0000","last_undegraded":"2023-10-19T14:22:02.403966+0000","last_fullsized":"2023-10-19T14:22:02.403966+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:43:57.998020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.3","version":"0'0","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.399541+0000","last_change":"2023-10-19T14:22:01.413689+0000","last_active":"2023-10-19T14:22:02.399541+0000","last_peered":"2023-10-19T14:22:02.399541+0000","last_clean":"2023-10-19T14:22:02.399541+0000","last_became_active":"2023-10-19T14:22:01.413419+0000","last_became_peered":"2023-10-19T14:22:01.413419+0000","last_unstale":"2023-10-19T14:22:02.399541+0000","last_undegraded":"2023-10-19T14:22:02.399541+0000","last_fullsized":"2023-10-19T14:22:02.399541+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:37:57.754265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.5","version":"0'0","reported_seq":9,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.414555+0000","last_change":"2023-10-19T14:22:02.409126+0000","last_active":"2023-10-19T14:22:02.414555+0000","last_peered":"2023-10-19T14:22:02.414555+0000","last_clean":"2023-10-19T14:22:02.414555+0000","last_became_active":"2023-10-19T14:22:02.408855+0000","last_became_peered":"2023-10-19T14:22:02.408855+0000","last_unstale":"2023-10-19T14:22:02.414555+0000","last_undegraded":"2023-10-19T14:22:02.414555+0000","last_fullsized":"2023-10-19T14:22:02.414555+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:03:36.116245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.2","version":"0'0","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.404648+0000","last_change":"2023-10-19T14:22:01.411905+0000","last_active":"2023-10-19T14:22:02.404648+0000","last_peered":"2023-10-19T14:22:02.404648+0000","last_clean":"2023-10-19T14:22:02.404648+0000","last_became_active":"2023-10-19T14:22:01.411584+0000","last_became_peered":"2023-10-19T14:22:01.411584+0000","last_unstale":"2023-10-19T14:22:02.404648+0000","last_undegraded":"2023-10-19T14:22:02.404648+0000","last_fullsized":"2023-10-19T14:22:02.404648+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:40:13.932927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.6","version":"0'0","reported_seq":3,"reported_epoch":262,"state":"creating+activating","last_fresh":"2023-10-19T14:22:02.409726+0000","last_change":"2023-10-19T14:22:02.409726+0000","last_active":"2023-10-19T14:22:01.386521+0000","last_peered":"2023-10-19T14:22:01.386521+0000","last_clean":"2023-10-19T14:22:01.386521+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-19T14:22:02.409726+0000","last_undegraded":"2023-10-19T14:22:02.409726+0000","last_fullsized":"2023-10-19T14:22:02.409726+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:51:43.993546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.1","version":"0'0","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.403849+0000","last_change":"2023-10-19T14:22:01.412806+0000","last_active":"2023-10-19T14:22:02.403849+0000","last_peered":"2023-10-19T14:22:02.403849+0000","last_clean":"2023-10-19T14:22:02.403849+0000","last_became_active":"2023-10-19T14:22:01.412524+0000","last_became_peered":"2023-10-19T14:22:01.412524+0000","last_unstale":"2023-10-19T14:22:02.403849+0000","last_undegraded":"2023-10-19T14:22:02.403849+0000","last_fullsized":"2023-10-19T14:22:02.403849+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:43:54.078045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.7","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.414678+0000","last_change":"2023-10-19T14:22:02.414678+0000","last_active":"2023-10-19T14:22:02.414678+0000","last_peered":"2023-10-19T14:22:02.414678+0000","last_clean":"2023-10-19T14:22:02.414678+0000","last_became_active":"2023-10-19T14:22:02.414348+0000","last_became_peered":"2023-10-19T14:22:02.414348+0000","last_unstale":"2023-10-19T14:22:02.414678+0000","last_undegraded":"2023-10-19T14:22:02.414678+0000","last_fullsized":"2023-10-19T14:22:02.414678+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:18:02.907457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.0","version":"0'0","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.399367+0000","last_change":"2023-10-19T14:22:01.414455+0000","last_active":"2023-10-19T14:22:02.399367+0000","last_peered":"2023-10-19T14:22:02.399367+0000","last_clean":"2023-10-19T14:22:02.399367+0000","last_became_active":"2023-10-19T14:22:01.414081+0000","last_became_peered":"2023-10-19T14:22:01.414081+0000","last_unstale":"2023-10-19T14:22:02.399367+0000","last_undegraded":"2023-10-19T14:22:02.399367+0000","last_fullsized":"2023-10-19T14:22:02.399367+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:54:25.277401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.25","version":"0'0","reported_seq":8,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.418670+0000","last_change":"2023-10-19T14:22:02.418670+0000","last_active":"2023-10-19T14:22:02.418670+0000","last_peered":"2023-10-19T14:22:02.418670+0000","last_clean":"2023-10-19T14:22:02.418670+0000","last_became_active":"2023-10-19T14:22:02.417862+0000","last_became_peered":"2023-10-19T14:22:02.417862+0000","last_unstale":"2023-10-19T14:22:02.418670+0000","last_undegraded":"2023-10-19T14:22:02.418670+0000","last_fullsized":"2023-10-19T14:22:02.418670+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:20:15.811864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"249'3224","reported_seq":4253,"reported_epoch":261,"state":"active+clean","last_fresh":"2023-10-19T14:22:01.406839+0000","last_change":"2023-10-19T12:44:08.636613+0000","last_active":"2023-10-19T14:22:01.406839+0000","last_peered":"2023-10-19T14:22:01.406839+0000","last_clean":"2023-10-19T14:22:01.406839+0000","last_became_active":"2023-10-19T12:44:08.635697+0000","last_became_peered":"2023-10-19T12:44:08.635697+0000","last_unstale":"2023-10-19T14:22:01.406839+0000","last_undegraded":"2023-10-19T14:22:01.406839+0000","last_fullsized":"2023-10-19T14:22:01.406839+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_clean_scrub_stamp":"2023-10-19T12:44:07.399738+0000","objects_scrubbed":0,"log_size":3224,"log_dups_size":0,"ondisk_log_size":3224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:15:23.685485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2539,"num_read_kb":5339,"num_write":657,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":36,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":35,"num_pg":32,"stat_sum":{"num_bytes":1470,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":33,"ondisk_log_size":33,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2539,"num_read_kb":5339,"num_write":657,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3224,"ondisk_log_size":3224,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576020,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66928,"kb_used_data":3804,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94304912,"statfs":{"total":96636764160,"available":96568229888,"internally_reserved":0,"allocated":3895296,"data_stored":2294561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":64616907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:21:10 2023","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.55100000000000005,"15min":0.57999999999999996},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.32200000000000001},"max":{"1min":0.754,"5min":0.80900000000000005,"15min":0.86199999999999999},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.503,"5min":0.57599999999999996,"15min":0.59899999999999998},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.312},"max":{"1min":0.66500000000000004,"5min":0.81899999999999995,"15min":0.86599999999999999},"last":0.39400000000000002}]},{"osd":1,"last update":"Thu Oct 19 14:21:10 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.58199999999999996,"15min":0.59499999999999997},"min":{"1min":0.45200000000000001,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.72199999999999998,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.56899999999999995,"15min":0.59299999999999997},"min":{"1min":0.39300000000000002,"5min":0.373,"15min":0.36699999999999999},"max":{"1min":0.78500000000000003,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.56200000000000006}]},{"osd":2,"last update":"Thu Oct 19 14:21:10 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.622,"15min":0.64200000000000002},"min":{"1min":0.47299999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.70499999999999996,"5min":0.84799999999999998,"15min":0.85499999999999998},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.63900000000000001,"15min":0.65700000000000003},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.54600000000000004}]},{"osd":3,"last update":"Thu Oct 19 14:21:10 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.60799999999999998,"15min":0.63100000000000001},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.628,"5min":0.67100000000000004,"15min":0.68300000000000005},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.83299999999999996,"5min":0.876,"15min":0.89600000000000002},"last":0.66700000000000004}]},{"osd":4,"last update":"Thu Oct 19 14:21:10 2023","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.50600000000000001,"15min":0.495},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.53000000000000003,"15min":0.51100000000000001},"min":{"1min":0.34999999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.505}]},{"osd":5,"last update":"Thu Oct 19 14:21:10 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52300000000000002,"15min":0.51100000000000001},"min":{"1min":0.33700000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.57799999999999996,"15min":0.55400000000000005},"min":{"1min":0.40699999999999997,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.054,"5min":1.054,"15min":1.054},"last":0.626}]},{"osd":6,"last update":"Thu Oct 19 14:21:10 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.58999999999999997,"15min":0.57599999999999996},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.35999999999999999},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.59299999999999997,"15min":0.57299999999999995},"min":{"1min":0.42999999999999999,"5min":0.35599999999999998,"15min":0.33500000000000002},"max":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.57399999999999995}]}]},{"osd":6,"up_from":13,"seq":55834576020,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56612,"kb_used_data":3224,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315228,"statfs":{"total":96636764160,"available":96578793472,"internally_reserved":0,"allocated":3301376,"data_stored":1704193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:21:02 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.59199999999999997,"15min":0.61099999999999999},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.78400000000000003,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.56299999999999994,"15min":0.60599999999999998},"min":{"1min":0.36699999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.71799999999999997,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.80200000000000005}]},{"osd":1,"last update":"Thu Oct 19 14:21:02 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.60099999999999998,"15min":0.61799999999999999},"min":{"1min":0.373,"5min":0.373,"15min":0.35799999999999998},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":1.165},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.60199999999999998,"15min":0.626},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.77400000000000002,"5min":0.83699999999999997,"15min":0.91300000000000003},"last":0.76600000000000001}]},{"osd":2,"last update":"Thu Oct 19 14:21:02 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.57599999999999996,"15min":0.60199999999999998},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.69099999999999995,"5min":0.80100000000000005,"15min":0.873},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.60599999999999998,"15min":0.624},"min":{"1min":0.40899999999999997,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.74199999999999999,"5min":0.83499999999999996,"15min":0.93100000000000005},"last":0.73599999999999999}]},{"osd":3,"last update":"Thu Oct 19 14:21:02 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.629,"15min":0.64900000000000002},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.66700000000000004,"5min":0.875,"15min":0.875},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.63800000000000001,"15min":0.65500000000000003},"min":{"1min":0.45400000000000001,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.92200000000000004,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.70399999999999996}]},{"osd":4,"last update":"Thu Oct 19 14:21:02 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.48899999999999999,"15min":0.48399999999999999},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.51000000000000001,"15min":0.50800000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.30399999999999999},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.85699999999999998},"last":0.61399999999999999}]},{"osd":5,"last update":"Thu Oct 19 14:21:02 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.50700000000000001,"15min":0.502},"min":{"1min":0.34799999999999998,"5min":0.32700000000000001,"15min":0.32500000000000001},"max":{"1min":0.79900000000000004,"5min":1.115,"15min":1.115},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.56599999999999995,"15min":0.54500000000000004},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.69199999999999995}]},{"osd":7,"last update":"Thu Oct 19 14:21:02 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.56000000000000005,"15min":0.54700000000000004},"min":{"1min":0.36799999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.90100000000000002},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.56699999999999995,"15min":0.55400000000000005},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.98399999999999999,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.67800000000000005}]}]},{"osd":5,"up_from":13,"seq":55834576020,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66348,"kb_used_data":3224,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305492,"statfs":{"total":96636764160,"available":96568823808,"internally_reserved":0,"allocated":3301376,"data_stored":1704193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:21:50 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.57399999999999995,"15min":0.60699999999999998},"min":{"1min":0.38,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.69399999999999995,"5min":0.79700000000000004,"15min":0.86399999999999999},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.59599999999999997,"15min":0.63300000000000001},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.70499999999999996,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.51100000000000001}]},{"osd":1,"last update":"Thu Oct 19 14:21:50 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.59099999999999997,"15min":0.622},"min":{"1min":0.34599999999999997,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.877},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.61199999999999999,"15min":0.63700000000000001},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.877,"5min":0.877,"15min":0.88500000000000001},"last":0.46999999999999997}]},{"osd":2,"last update":"Thu Oct 19 14:21:50 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.58199999999999996,"15min":0.61399999999999999},"min":{"1min":0.378,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.625,"15min":0.65100000000000002},"min":{"1min":0.441,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.78900000000000003,"5min":0.83299999999999996,"15min":0.90800000000000003},"last":0.65600000000000003}]},{"osd":3,"last update":"Thu Oct 19 14:21:50 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.60499999999999998,"15min":0.63600000000000001},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.81100000000000005,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.626,"15min":0.65900000000000003},"min":{"1min":0.40000000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.90900000000000003},"last":0.70099999999999996}]},{"osd":4,"last update":"Thu Oct 19 14:21:50 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.48799999999999999,"15min":0.48699999999999999},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.51800000000000002,"15min":0.505},"min":{"1min":0.309,"5min":0.308,"15min":0.28599999999999998},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.82099999999999995},"last":0.63900000000000001}]},{"osd":6,"last update":"Thu Oct 19 14:21:50 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.51700000000000002,"15min":0.502},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.88800000000000001},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.54100000000000004,"15min":0.53900000000000003},"min":{"1min":0.36899999999999999,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.67700000000000005}]},{"osd":7,"last update":"Thu Oct 19 14:21:50 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.52200000000000002,"15min":0.51800000000000002},"min":{"1min":0.32700000000000001,"5min":0.312,"15min":0.312},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.499},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.57899999999999996,"15min":0.57199999999999995},"min":{"1min":0.38,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.72299999999999998}]}]},{"osd":4,"up_from":13,"seq":55834576020,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48292,"kb_used_data":3224,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323548,"statfs":{"total":96636764160,"available":96587313152,"internally_reserved":0,"allocated":3301376,"data_stored":1704193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:21:16 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.54700000000000004,"15min":0.57099999999999995},"min":{"1min":0.30299999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.879},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.56899999999999995,"15min":0.59299999999999997},"min":{"1min":0.35299999999999998,"5min":0.31,"15min":0.31},"max":{"1min":0.72699999999999998,"5min":0.82599999999999996,"15min":0.873},"last":0.54700000000000004}]},{"osd":1,"last update":"Thu Oct 19 14:21:16 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.59699999999999998,"15min":0.63500000000000001},"min":{"1min":0.40300000000000002,"5min":0.372,"15min":0.32600000000000001},"max":{"1min":0.74099999999999999,"5min":0.80400000000000005,"15min":0.92700000000000005},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.60899999999999999,"15min":0.63900000000000001},"min":{"1min":0.40500000000000003,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.755,"5min":0.90000000000000002,"15min":1.0940000000000001},"last":0.53100000000000003}]},{"osd":2,"last update":"Thu Oct 19 14:21:16 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.626,"15min":0.64700000000000002},"min":{"1min":0.38900000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.80300000000000005,"5min":1.008,"15min":1.008},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.64200000000000002,"15min":0.66600000000000004},"min":{"1min":0.47199999999999998,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.81799999999999995,"5min":0.98799999999999999,"15min":1.008},"last":0.68799999999999994}]},{"osd":3,"last update":"Thu Oct 19 14:21:16 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.59499999999999997,"15min":0.627},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.68600000000000005,"5min":0.83399999999999996,"15min":0.91900000000000004},"last":0.755},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.65600000000000003,"15min":0.66600000000000004},"min":{"1min":0.48199999999999998,"5min":0.45400000000000001,"15min":0.437},"max":{"1min":0.85999999999999999,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.76500000000000001}]},{"osd":5,"last update":"Thu Oct 19 14:21:16 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.497,"15min":0.47799999999999998},"min":{"1min":0.33800000000000002,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.79700000000000004,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.50800000000000001,"15min":0.498},"min":{"1min":0.36499999999999999,"5min":0.33700000000000002,"15min":0.29199999999999998},"max":{"1min":0.81399999999999995,"5min":0.873,"15min":0.873},"last":0.502}]},{"osd":6,"last update":"Thu Oct 19 14:21:16 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.50700000000000001,"15min":0.49399999999999999},"min":{"1min":0.38500000000000001,"5min":0.34799999999999998,"15min":0.28199999999999997},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.54900000000000004,"15min":0.54300000000000004},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.88,"5min":0.88,"15min":1.2969999999999999},"last":0.47999999999999998}]},{"osd":7,"last update":"Thu Oct 19 14:21:16 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55100000000000005,"15min":0.54500000000000004},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.32600000000000001},"max":{"1min":0.78200000000000003,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.438},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.61199999999999999,"15min":0.59899999999999998},"min":{"1min":0.371,"5min":0.32100000000000001,"15min":0.318},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":1.032},"last":0.45600000000000002}]}]},{"osd":3,"up_from":13,"seq":55834576019,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56616,"kb_used_data":3224,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315224,"statfs":{"total":96636764160,"available":96578789376,"internally_reserved":0,"allocated":3301376,"data_stored":1704193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54655432},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:21:53 2023","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.39900000000000002,"15min":0.41699999999999998},"min":{"1min":0.23899999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.54500000000000004,"5min":0.60599999999999998,"15min":0.70699999999999996},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.39400000000000002,"15min":0.42199999999999999},"min":{"1min":0.23899999999999999,"5min":0.217,"15min":0.217},"max":{"1min":0.51900000000000002,"5min":0.61899999999999999,"15min":0.752},"last":0.36699999999999999}]},{"osd":1,"last update":"Thu Oct 19 14:21:53 2023","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.40699999999999997,"15min":0.42799999999999999},"min":{"1min":0.28100000000000003,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.71599999999999997,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.45500000000000002,"15min":0.47199999999999998},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26400000000000001},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.38800000000000001}]},{"osd":2,"last update":"Thu Oct 19 14:21:53 2023","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.46200000000000002,"15min":0.48099999999999998},"min":{"1min":0.34300000000000003,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.73099999999999998},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48499999999999999,"15min":0.504},"min":{"1min":0.30599999999999999,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":1.212,"5min":1.212,"15min":1.212},"last":0.35499999999999998}]},{"osd":4,"last update":"Thu Oct 19 14:21:53 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.67400000000000004,"15min":0.70099999999999996},"min":{"1min":0.44600000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.94199999999999995,"5min":1.018,"15min":1.157},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.629,"5min":0.68700000000000006,"15min":0.72299999999999998},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.89500000000000002,"5min":0.996,"15min":1.173},"last":0.752}]},{"osd":5,"last update":"Thu Oct 19 14:21:53 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.67500000000000004,"15min":0.68600000000000005},"min":{"1min":0.50900000000000001,"5min":0.46600000000000003,"15min":0.41999999999999998},"max":{"1min":0.96799999999999997,"5min":0.97299999999999998,"15min":1.0149999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.66800000000000004,"15min":0.68600000000000005},"min":{"1min":0.48899999999999999,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.97699999999999998,"5min":0.99399999999999999,"15min":1.105},"last":0.80600000000000005}]},{"osd":6,"last update":"Thu Oct 19 14:21:53 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.65000000000000002,"15min":0.68600000000000005},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.93000000000000005,"5min":1.0009999999999999,"15min":1.0669999999999999},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.68899999999999995,"15min":0.70999999999999996},"min":{"1min":0.505,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.079},"last":0.79100000000000004}]},{"osd":7,"last update":"Thu Oct 19 14:21:53 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67500000000000004,"15min":0.69799999999999995},"min":{"1min":0.46600000000000003,"5min":0.434,"15min":0.434},"max":{"1min":0.94999999999999996,"5min":0.96899999999999997,"15min":1.085},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.69799999999999995,"15min":0.72799999999999998},"min":{"1min":0.48799999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.95899999999999996,"5min":0.97199999999999998,"15min":1.1739999999999999},"last":0.60299999999999998}]}]},{"osd":2,"up_from":13,"seq":55834576021,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48292,"kb_used_data":3224,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323548,"statfs":{"total":96636764160,"available":96587313152,"internally_reserved":0,"allocated":3301376,"data_stored":1704193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.374,"15min":0.39600000000000002},"min":{"1min":0.25700000000000001,"5min":0.218,"15min":0.218},"max":{"1min":0.58799999999999997,"5min":0.66600000000000004,"15min":0.69699999999999995},"last":0.27800000000000002},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.41499999999999998,"15min":0.432},"min":{"1min":0.248,"5min":0.248,"15min":0.223},"max":{"1min":0.53100000000000003,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.379}]},{"osd":1,"last update":"Thu Oct 19 14:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.38300000000000001,"15min":0.40600000000000003},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.65400000000000003},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.44400000000000001,"15min":0.46000000000000002},"min":{"1min":0.28699999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.45100000000000001,"5min":1.742,"15min":1.742},"last":0.40100000000000002}]},{"osd":3,"last update":"Thu Oct 19 14:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.45500000000000002,"15min":0.47599999999999998},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.63300000000000001,"5min":0.69299999999999995,"15min":0.74399999999999999},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.47299999999999998,"15min":0.501},"min":{"1min":0.32700000000000001,"5min":0.32000000000000001,"15min":0.315},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.83599999999999997},"last":0.5}]},{"osd":4,"last update":"Thu Oct 19 14:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.63900000000000001,"15min":0.66200000000000003},"min":{"1min":0.40699999999999997,"5min":0.39600000000000002,"15min":0.39500000000000002},"max":{"1min":0.74399999999999999,"5min":0.89200000000000002,"15min":0.998},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.65400000000000003,"15min":0.67600000000000005},"min":{"1min":0.41899999999999998,"5min":0.38900000000000001,"15min":0.35199999999999998},"max":{"1min":0.79000000000000004,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.60599999999999998}]},{"osd":5,"last update":"Thu Oct 19 14:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.67800000000000005,"15min":0.69299999999999995},"min":{"1min":0.443,"5min":0.443,"15min":0.40600000000000003},"max":{"1min":0.79300000000000004,"5min":1.4330000000000001,"15min":1.4330000000000001},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.66900000000000004,"15min":0.68000000000000005},"min":{"1min":0.46899999999999997,"5min":0.45100000000000001,"15min":0.39300000000000002},"max":{"1min":0.77400000000000002,"5min":1.595,"15min":1.595},"last":0.80900000000000005}]},{"osd":6,"last update":"Thu Oct 19 14:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.67700000000000005,"15min":0.70199999999999996},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.80900000000000005,"5min":0.996,"15min":1.0189999999999999},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.629,"5min":0.69499999999999995,"15min":0.71399999999999997},"min":{"1min":0.438,"5min":0.433,"15min":0.40600000000000003},"max":{"1min":0.83399999999999996,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":0.73099999999999998}]},{"osd":7,"last update":"Thu Oct 19 14:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.68400000000000005,"15min":0.71299999999999997},"min":{"1min":0.47199999999999998,"5min":0.44800000000000001,"15min":0.41799999999999998},"max":{"1min":0.78800000000000003,"5min":1.0129999999999999,"15min":1.0289999999999999},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.71199999999999997,"15min":0.72599999999999998},"min":{"1min":0.42199999999999999,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.86699999999999999,"5min":1.53,"15min":1.53},"last":0.64200000000000002}]}]},{"osd":1,"up_from":13,"seq":55834576020,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48296,"kb_used_data":3224,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323544,"statfs":{"total":96636764160,"available":96587309056,"internally_reserved":0,"allocated":3301376,"data_stored":1704193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.38400000000000001,"15min":0.40300000000000002},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.46899999999999997,"5min":0.61599999999999999,"15min":0.67400000000000004},"last":0.22500000000000001},{"interface":"front","average":{"1min":0.378,"5min":0.41999999999999998,"15min":0.45000000000000001},"min":{"1min":0.27700000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.51500000000000001,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.28100000000000003}]},{"osd":2,"last update":"Thu Oct 19 14:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.42399999999999999,"15min":0.438},"min":{"1min":0.26100000000000001,"5min":0.251,"15min":0.251},"max":{"1min":0.505,"5min":0.63100000000000001,"15min":0.77500000000000002},"last":0.318},{"interface":"front","average":{"1min":0.434,"5min":0.47699999999999998,"15min":0.48799999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.56699999999999995,"5min":0.68700000000000006,"15min":0.82799999999999996},"last":0.44400000000000001}]},{"osd":3,"last update":"Thu Oct 19 14:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.495,"15min":0.501},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.29499999999999998},"max":{"1min":0.59399999999999997,"5min":0.76000000000000001,"15min":0.78900000000000003},"last":0.376},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.42899999999999999,"15min":0.44600000000000001},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.26100000000000001},"max":{"1min":0.59899999999999998,"5min":0.65200000000000002,"15min":0.69499999999999995},"last":0.245}]},{"osd":4,"last update":"Thu Oct 19 14:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.64200000000000002,"15min":0.66700000000000004},"min":{"1min":0.42299999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.80400000000000005,"5min":0.95999999999999996,"15min":1.042},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.65400000000000003,"15min":0.66600000000000004},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.32300000000000001},"max":{"1min":0.80500000000000005,"5min":0.90800000000000003,"15min":1.1659999999999999},"last":0.39500000000000002}]},{"osd":5,"last update":"Thu Oct 19 14:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.66700000000000004,"15min":0.68200000000000005},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.81200000000000006,"5min":0.96199999999999997,"15min":1.3500000000000001},"last":0.434},{"interface":"front","average":{"1min":0.62,"5min":0.68899999999999995,"15min":0.70299999999999996},"min":{"1min":0.442,"5min":0.44,"15min":0.40500000000000003},"max":{"1min":0.82899999999999996,"5min":1.0009999999999999,"15min":1.121},"last":0.49399999999999999}]},{"osd":6,"last update":"Thu Oct 19 14:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.65400000000000003,"15min":0.67600000000000005},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.84999999999999998,"5min":1.0129999999999999,"15min":1.2310000000000001},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.69699999999999995,"15min":0.71499999999999997},"min":{"1min":0.46700000000000003,"5min":0.434,"15min":0.434},"max":{"1min":0.85799999999999998,"5min":1.0009999999999999,"15min":1.274},"last":0.47699999999999998}]},{"osd":7,"last update":"Thu Oct 19 14:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.70499999999999996,"15min":0.72199999999999998},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.86899999999999999,"5min":0.99099999999999999,"15min":1.075},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.68200000000000005,"15min":0.69599999999999995},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.84599999999999997,"5min":1.0169999999999999,"15min":1.117},"last":0.46200000000000002}]}]},{"osd":0,"up_from":13,"seq":55834576021,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61348,"kb_used_data":3800,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310492,"statfs":{"total":96636764160,"available":96573943808,"internally_reserved":0,"allocated":3891200,"data_stored":2293942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Oct 19 14:21:13 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.40100000000000002,"15min":0.41099999999999998},"min":{"1min":0.26300000000000001,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.47799999999999998,"5min":0.61499999999999999,"15min":0.624},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.41699999999999998,"15min":0.45500000000000002},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.49199999999999999,"5min":0.70899999999999996,"15min":0.81100000000000005},"last":0.64300000000000002}]},{"osd":2,"last update":"Thu Oct 19 14:21:13 2023","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.34100000000000003,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.54400000000000004,"5min":0.754,"15min":0.754},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.377,"5min":0.42399999999999999,"15min":0.44400000000000001},"min":{"1min":0.28100000000000003,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.46899999999999997,"5min":0.60199999999999998,"15min":0.69099999999999995},"last":0.52400000000000002}]},{"osd":3,"last update":"Thu Oct 19 14:21:13 2023","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.436},"min":{"1min":0.27200000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.50800000000000001,"5min":0.58799999999999997,"15min":0.66800000000000004},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.48199999999999998,"15min":0.50700000000000001},"min":{"1min":0.29299999999999998,"5min":0.254,"15min":0.254},"max":{"1min":0.72399999999999998,"5min":0.78100000000000003,"15min":0.84599999999999997},"last":0.53400000000000003}]},{"osd":4,"last update":"Thu Oct 19 14:21:13 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.63,"15min":0.67000000000000004},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.80300000000000005,"5min":0.93100000000000005,"15min":1.1890000000000001},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.628,"15min":0.67400000000000004},"min":{"1min":0.45900000000000002,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.75600000000000001,"5min":0.93400000000000005,"15min":1.2150000000000001},"last":0.78300000000000003}]},{"osd":5,"last update":"Thu Oct 19 14:21:13 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.67200000000000004,"15min":0.71399999999999997},"min":{"1min":0.48599999999999999,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.83099999999999996,"5min":0.98499999999999999,"15min":1.1579999999999999},"last":0.748},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.67500000000000004,"15min":0.71399999999999997},"min":{"1min":0.499,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.80000000000000004,"5min":0.98299999999999998,"15min":1.2310000000000001},"last":0.73899999999999999}]},{"osd":6,"last update":"Thu Oct 19 14:21:13 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.66300000000000003,"15min":0.69899999999999995},"min":{"1min":0.46100000000000002,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.85499999999999998,"5min":0.97799999999999998,"15min":1.1739999999999999},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.65300000000000002,"15min":0.69599999999999995},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.70299999999999996,"5min":0.97799999999999998,"15min":1.0189999999999999},"last":0.76100000000000001}]},{"osd":7,"last update":"Thu Oct 19 14:21:13 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.66600000000000004,"15min":0.70999999999999996},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.79900000000000004,"5min":0.93400000000000005,"15min":1.1180000000000001},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.66100000000000003,"15min":0.69599999999999995},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.84499999999999997,"5min":0.93799999999999994,"15min":1.0349999999999999},"last":0.77300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-19T14:22:09.845 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-19T14:22:09.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:09.964+0000 7f00f35a5640 1 -- 172.21.15.57:0/117275087 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f00d8004d00 msgr2=0x7f00d8005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:09.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:09.964+0000 7f00f35a5640 1 --2- 172.21.15.57:0/117275087 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f00d8004d00 0x7f00d8005180 secure :-1 s=READY pgs=2895 cs=0 l=1 rev1=1 crypto rx=0x7f00e4009ed0 tx=0x7f00d80057f0 comp rx=0 tx=0).stop 2023-10-19T14:22:09.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:09.964+0000 7f00f1b42640 1 -- 172.21.15.57:0/117275087 reap_dead start 2023-10-19T14:22:09.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:09.964+0000 7f00f35a5640 1 -- 172.21.15.57:0/117275087 shutdown_connections 2023-10-19T14:22:09.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:09.964+0000 7f00f35a5640 1 -- 172.21.15.57:0/117275087 >> 172.21.15.57:0/117275087 conn(0x7f00ec069930 msgr2=0x7f00ec069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:09.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:09.964+0000 7f00f35a5640 1 -- 172.21.15.57:0/117275087 shutdown_connections 2023-10-19T14:22:09.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:09.964+0000 7f00f35a5640 1 -- 172.21.15.57:0/117275087 wait complete. 2023-10-19T14:22:09.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:09.966+0000 7f00f35a5640 1 Processor -- start 2023-10-19T14:22:09.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:09.966+0000 7f00f35a5640 1 -- start start 2023-10-19T14:22:09.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:09.966+0000 7f00f35a5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f00ec150770 0x7f00ec151ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:09.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:09.966+0000 7f00f35a5640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f00ec1556c0 0x7f00ec0b1160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:09.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:09.966+0000 7f00f35a5640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f00ec0b16a0 0x7f00ec1542f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:09.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:09.966+0000 7f00f35a5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00ec0b1b20 con 0x7f00ec150770 2023-10-19T14:22:09.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:09.966+0000 7f00f35a5640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00ec156ab0 con 0x7f00ec1556c0 2023-10-19T14:22:09.973 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:09.966+0000 7f00f35a5640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00ec155ea0 con 0x7f00ec0b16a0 2023-10-19T14:22:09.973 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:09.967+0000 7f00f1341640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f00ec150770 0x7f00ec151ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:09.973 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:09.967+0000 7f00f1341640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f00ec150770 0x7f00ec151ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50000/0 (socket says 172.21.15.57:50000) 2023-10-19T14:22:09.973 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:09.967+0000 7f00f1341640 1 -- 172.21.15.57:0/2453457628 learned_addr learned my addr 172.21.15.57:0/2453457628 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:22:09.973 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:09.967+0000 7f00f0b40640 1 --2- 172.21.15.57:0/2453457628 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f00ec1556c0 0x7f00ec0b1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:09.974 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:09.967+0000 7f00f1b42640 1 --2- 172.21.15.57:0/2453457628 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f00ec0b16a0 0x7f00ec1542f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:09.974 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:09.967+0000 7f00f1341640 1 -- 172.21.15.57:0/2453457628 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f00ec0b16a0 msgr2=0x7f00ec1542f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:09.974 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:09.967+0000 7f00f1341640 1 --2- 172.21.15.57:0/2453457628 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f00ec0b16a0 0x7f00ec1542f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:09.974 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:09.967+0000 7f00f1341640 1 -- 172.21.15.57:0/2453457628 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f00ec1556c0 msgr2=0x7f00ec0b1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:09.974 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:09.967+0000 7f00f1341640 1 --2- 172.21.15.57:0/2453457628 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f00ec1556c0 0x7f00ec0b1160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:09.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:09.967+0000 7f00f1341640 1 -- 172.21.15.57:0/2453457628 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00ec156180 con 0x7f00ec150770 2023-10-19T14:22:09.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:09.967+0000 7f00f0b40640 1 --2- 172.21.15.57:0/2453457628 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f00ec1556c0 0x7f00ec0b1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:22:09.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:09.967+0000 7f00f1341640 1 --2- 172.21.15.57:0/2453457628 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f00ec150770 0x7f00ec151ce0 secure :-1 s=READY pgs=2896 cs=0 l=1 rev1=1 crypto rx=0x7f00e80066b0 tx=0x7f00e80039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:09.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:09.967+0000 7f00e1ffb640 1 -- 172.21.15.57:0/2453457628 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00e8011900 con 0x7f00ec150770 2023-10-19T14:22:09.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:09.967+0000 7f00e1ffb640 1 -- 172.21.15.57:0/2453457628 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f00e8011a80 con 0x7f00ec150770 2023-10-19T14:22:09.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:09.967+0000 7f00e1ffb640 1 -- 172.21.15.57:0/2453457628 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00e8011e90 con 0x7f00ec150770 2023-10-19T14:22:09.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:09.967+0000 7f00f35a5640 1 -- 172.21.15.57:0/2453457628 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00ec156460 con 0x7f00ec150770 2023-10-19T14:22:09.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:09.968+0000 7f00f35a5640 1 -- 172.21.15.57:0/2453457628 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00ec150b50 con 0x7f00ec150770 2023-10-19T14:22:09.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:09.969+0000 7f00f35a5640 1 -- 172.21.15.57:0/2453457628 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00ec0690e0 con 0x7f00ec150770 2023-10-19T14:22:09.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:09.969+0000 7f00e1ffb640 1 -- 172.21.15.57:0/2453457628 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 145) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f00e800f430 con 0x7f00ec150770 2023-10-19T14:22:09.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:09.969+0000 7f00e1ffb640 1 --2- 172.21.15.57:0/2453457628 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f00d006b080 0x7f00d006d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:09.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:09.969+0000 7f00e1ffb640 1 -- 172.21.15.57:0/2453457628 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(263..263 src has 1..263) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f00e808ab00 con 0x7f00ec150770 2023-10-19T14:22:09.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:09.970+0000 7f00f0b40640 1 --2- 172.21.15.57:0/2453457628 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f00d006b080 0x7f00d006d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:09.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:09.970+0000 7f00f0b40640 1 --2- 172.21.15.57:0/2453457628 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f00d006b080 0x7f00d006d540 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f00ec15a390 tx=0x7f00dc005f90 comp rx=0 tx=0).ready entity=mgr.11446 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:09.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:09.974+0000 7f00e1ffb640 1 -- 172.21.15.57:0/2453457628 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f00e80555f0 con 0x7f00ec150770 2023-10-19T14:22:10.148 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.146+0000 7f00f35a5640 1 -- 172.21.15.57:0/2453457628 --> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f00ec0c97c0 con 0x7f00d006b080 2023-10-19T14:22:10.156 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.153+0000 7f00e1ffb640 1 -- 172.21.15.57:0/2453457628 <== mgr.11446 v2:172.21.15.179:6832/1573806521 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+290041 (secure 0 0 0) 0x7f00dc009c50 con 0x7f00d006b080 2023-10-19T14:22:10.159 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:22:10.160 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2023-10-19T14:22:10.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.156+0000 7f00f35a5640 1 -- 172.21.15.57:0/2453457628 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f00d006b080 msgr2=0x7f00d006d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:10.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.156+0000 7f00f35a5640 1 --2- 172.21.15.57:0/2453457628 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f00d006b080 0x7f00d006d540 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f00ec15a390 tx=0x7f00dc005f90 comp rx=0 tx=0).stop 2023-10-19T14:22:10.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.156+0000 7f00f35a5640 1 -- 172.21.15.57:0/2453457628 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f00ec150770 msgr2=0x7f00ec151ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:10.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.156+0000 7f00f35a5640 1 --2- 172.21.15.57:0/2453457628 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f00ec150770 0x7f00ec151ce0 secure :-1 s=READY pgs=2896 cs=0 l=1 rev1=1 crypto rx=0x7f00e80066b0 tx=0x7f00e80039c0 comp rx=0 tx=0).stop 2023-10-19T14:22:10.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.156+0000 7f00f35a5640 1 -- 172.21.15.57:0/2453457628 shutdown_connections 2023-10-19T14:22:10.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.157+0000 7f00f35a5640 1 --2- 172.21.15.57:0/2453457628 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f00d006b080 0x7f00d006d540 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:10.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.157+0000 7f00f35a5640 1 --2- 172.21.15.57:0/2453457628 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f00ec0b16a0 0x7f00ec1542f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:10.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.157+0000 7f00f35a5640 1 --2- 172.21.15.57:0/2453457628 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f00ec1556c0 0x7f00ec0b1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:10.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.157+0000 7f00f35a5640 1 --2- 172.21.15.57:0/2453457628 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f00ec150770 0x7f00ec151ce0 unknown :-1 s=CLOSED pgs=2896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:10.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.157+0000 7f00f35a5640 1 -- 172.21.15.57:0/2453457628 >> 172.21.15.57:0/2453457628 conn(0x7f00ec069930 msgr2=0x7f00ec0af900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:10.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.157+0000 7f00f35a5640 1 -- 172.21.15.57:0/2453457628 shutdown_connections 2023-10-19T14:22:10.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.157+0000 7f00f35a5640 1 -- 172.21.15.57:0/2453457628 wait complete. 2023-10-19T14:22:10.170 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":31,"stamp":"2023-10-19T14:22:09.420553+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2539,"num_read_kb":5339,"num_write":701,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3262,"ondisk_log_size":3262,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":457156,"kb_used_data":27204,"kb_used_omap":12,"kb_used_meta":429811,"kb_avail":754517564,"statfs":{"total":773094113280,"available":772625985536,"internally_reserved":0,"allocated":27856896,"data_stored":14928649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":440127033},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001705"},"pg_stats":[{"pgid":"36.3f","version":"0'0","reported_seq":11,"reported_epoch":263,"state":"active+clean","last_fresh":"2023-10-19T14:22:03.635796+0000","last_change":"2023-10-19T14:22:02.422114+0000","last_active":"2023-10-19T14:22:03.635796+0000","last_peered":"2023-10-19T14:22:03.635796+0000","last_clean":"2023-10-19T14:22:03.635796+0000","last_became_active":"2023-10-19T14:22:02.421696+0000","last_became_peered":"2023-10-19T14:22:02.421696+0000","last_unstale":"2023-10-19T14:22:03.635796+0000","last_undegraded":"2023-10-19T14:22:03.635796+0000","last_fullsized":"2023-10-19T14:22:03.635796+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:51:26.145263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.3e","version":"0'0","reported_seq":11,"reported_epoch":263,"state":"active+clean","last_fresh":"2023-10-19T14:22:03.635723+0000","last_change":"2023-10-19T14:22:02.425321+0000","last_active":"2023-10-19T14:22:03.635723+0000","last_peered":"2023-10-19T14:22:03.635723+0000","last_clean":"2023-10-19T14:22:03.635723+0000","last_became_active":"2023-10-19T14:22:02.425059+0000","last_became_peered":"2023-10-19T14:22:02.425059+0000","last_unstale":"2023-10-19T14:22:03.635723+0000","last_undegraded":"2023-10-19T14:22:03.635723+0000","last_fullsized":"2023-10-19T14:22:03.635723+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:05:33.863405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.3d","version":"0'0","reported_seq":11,"reported_epoch":263,"state":"active+clean","last_fresh":"2023-10-19T14:22:03.633866+0000","last_change":"2023-10-19T14:22:02.412915+0000","last_active":"2023-10-19T14:22:03.633866+0000","last_peered":"2023-10-19T14:22:03.633866+0000","last_clean":"2023-10-19T14:22:03.633866+0000","last_became_active":"2023-10-19T14:22:02.412634+0000","last_became_peered":"2023-10-19T14:22:02.412634+0000","last_unstale":"2023-10-19T14:22:03.633866+0000","last_undegraded":"2023-10-19T14:22:03.633866+0000","last_fullsized":"2023-10-19T14:22:03.633866+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:04:24.989932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.3c","version":"0'0","reported_seq":11,"reported_epoch":263,"state":"active+clean","last_fresh":"2023-10-19T14:22:03.633827+0000","last_change":"2023-10-19T14:22:02.417897+0000","last_active":"2023-10-19T14:22:03.633827+0000","last_peered":"2023-10-19T14:22:03.633827+0000","last_clean":"2023-10-19T14:22:03.633827+0000","last_became_active":"2023-10-19T14:22:02.417573+0000","last_became_peered":"2023-10-19T14:22:02.417573+0000","last_unstale":"2023-10-19T14:22:03.633827+0000","last_undegraded":"2023-10-19T14:22:03.633827+0000","last_fullsized":"2023-10-19T14:22:03.633827+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:39:40.912960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.3b","version":"0'0","reported_seq":11,"reported_epoch":263,"state":"active+clean","last_fresh":"2023-10-19T14:22:03.415459+0000","last_change":"2023-10-19T14:22:02.417221+0000","last_active":"2023-10-19T14:22:03.415459+0000","last_peered":"2023-10-19T14:22:03.415459+0000","last_clean":"2023-10-19T14:22:03.415459+0000","last_became_active":"2023-10-19T14:22:02.416941+0000","last_became_peered":"2023-10-19T14:22:02.416941+0000","last_unstale":"2023-10-19T14:22:03.415459+0000","last_undegraded":"2023-10-19T14:22:03.415459+0000","last_fullsized":"2023-10-19T14:22:03.415459+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T02:11:11.663612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.3a","version":"0'0","reported_seq":11,"reported_epoch":263,"state":"active+clean","last_fresh":"2023-10-19T14:22:03.415237+0000","last_change":"2023-10-19T14:22:02.426006+0000","last_active":"2023-10-19T14:22:03.415237+0000","last_peered":"2023-10-19T14:22:03.415237+0000","last_clean":"2023-10-19T14:22:03.415237+0000","last_became_active":"2023-10-19T14:22:02.425718+0000","last_became_peered":"2023-10-19T14:22:02.425718+0000","last_unstale":"2023-10-19T14:22:03.415237+0000","last_undegraded":"2023-10-19T14:22:03.415237+0000","last_fullsized":"2023-10-19T14:22:03.415237+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:10:47.778168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.39","version":"0'0","reported_seq":11,"reported_epoch":263,"state":"active+clean","last_fresh":"2023-10-19T14:22:03.634420+0000","last_change":"2023-10-19T14:22:02.413984+0000","last_active":"2023-10-19T14:22:03.634420+0000","last_peered":"2023-10-19T14:22:03.634420+0000","last_clean":"2023-10-19T14:22:03.634420+0000","last_became_active":"2023-10-19T14:22:02.413714+0000","last_became_peered":"2023-10-19T14:22:02.413714+0000","last_unstale":"2023-10-19T14:22:03.634420+0000","last_undegraded":"2023-10-19T14:22:03.634420+0000","last_fullsized":"2023-10-19T14:22:03.634420+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:39:01.090397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.38","version":"0'0","reported_seq":11,"reported_epoch":263,"state":"active+clean","last_fresh":"2023-10-19T14:22:03.633615+0000","last_change":"2023-10-19T14:22:02.419844+0000","last_active":"2023-10-19T14:22:03.633615+0000","last_peered":"2023-10-19T14:22:03.633615+0000","last_clean":"2023-10-19T14:22:03.633615+0000","last_became_active":"2023-10-19T14:22:02.419560+0000","last_became_peered":"2023-10-19T14:22:02.419560+0000","last_unstale":"2023-10-19T14:22:03.633615+0000","last_undegraded":"2023-10-19T14:22:03.633615+0000","last_fullsized":"2023-10-19T14:22:03.633615+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:33:42.355587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.37","version":"0'0","reported_seq":11,"reported_epoch":263,"state":"active+clean","last_fresh":"2023-10-19T14:22:03.634816+0000","last_change":"2023-10-19T14:22:02.423160+0000","last_active":"2023-10-19T14:22:03.634816+0000","last_peered":"2023-10-19T14:22:03.634816+0000","last_clean":"2023-10-19T14:22:03.634816+0000","last_became_active":"2023-10-19T14:22:02.422862+0000","last_became_peered":"2023-10-19T14:22:02.422862+0000","last_unstale":"2023-10-19T14:22:03.634816+0000","last_undegraded":"2023-10-19T14:22:03.634816+0000","last_fullsized":"2023-10-19T14:22:03.634816+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:27:23.379990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.36","version":"0'0","reported_seq":11,"reported_epoch":263,"state":"active+clean","last_fresh":"2023-10-19T14:22:03.635128+0000","last_change":"2023-10-19T14:22:02.413432+0000","last_active":"2023-10-19T14:22:03.635128+0000","last_peered":"2023-10-19T14:22:03.635128+0000","last_clean":"2023-10-19T14:22:03.635128+0000","last_became_active":"2023-10-19T14:22:02.412808+0000","last_became_peered":"2023-10-19T14:22:02.412808+0000","last_unstale":"2023-10-19T14:22:03.635128+0000","last_undegraded":"2023-10-19T14:22:03.635128+0000","last_fullsized":"2023-10-19T14:22:03.635128+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:38:38.860106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.35","version":"0'0","reported_seq":11,"reported_epoch":263,"state":"active+clean","last_fresh":"2023-10-19T14:22:03.414655+0000","last_change":"2023-10-19T14:22:02.413336+0000","last_active":"2023-10-19T14:22:03.414655+0000","last_peered":"2023-10-19T14:22:03.414655+0000","last_clean":"2023-10-19T14:22:03.414655+0000","last_became_active":"2023-10-19T14:22:02.412942+0000","last_became_peered":"2023-10-19T14:22:02.412942+0000","last_unstale":"2023-10-19T14:22:03.414655+0000","last_undegraded":"2023-10-19T14:22:03.414655+0000","last_fullsized":"2023-10-19T14:22:03.414655+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:24:01.445704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.34","version":"0'0","reported_seq":11,"reported_epoch":263,"state":"active+clean","last_fresh":"2023-10-19T14:22:03.634864+0000","last_change":"2023-10-19T14:22:02.421799+0000","last_active":"2023-10-19T14:22:03.634864+0000","last_peered":"2023-10-19T14:22:03.634864+0000","last_clean":"2023-10-19T14:22:03.634864+0000","last_became_active":"2023-10-19T14:22:02.421541+0000","last_became_peered":"2023-10-19T14:22:02.421541+0000","last_unstale":"2023-10-19T14:22:03.634864+0000","last_undegraded":"2023-10-19T14:22:03.634864+0000","last_fullsized":"2023-10-19T14:22:03.634864+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:31:09.426810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.33","version":"0'0","reported_seq":11,"reported_epoch":263,"state":"active+clean","last_fresh":"2023-10-19T14:22:03.633965+0000","last_change":"2023-10-19T14:22:02.419639+0000","last_active":"2023-10-19T14:22:03.633965+0000","last_peered":"2023-10-19T14:22:03.633965+0000","last_clean":"2023-10-19T14:22:03.633965+0000","last_became_active":"2023-10-19T14:22:02.419155+0000","last_became_peered":"2023-10-19T14:22:02.419155+0000","last_unstale":"2023-10-19T14:22:03.633965+0000","last_undegraded":"2023-10-19T14:22:03.633965+0000","last_fullsized":"2023-10-19T14:22:03.633965+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:48:32.972469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.32","version":"0'0","reported_seq":11,"reported_epoch":263,"state":"active+clean","last_fresh":"2023-10-19T14:22:03.634513+0000","last_change":"2023-10-19T14:22:02.424660+0000","last_active":"2023-10-19T14:22:03.634513+0000","last_peered":"2023-10-19T14:22:03.634513+0000","last_clean":"2023-10-19T14:22:03.634513+0000","last_became_active":"2023-10-19T14:22:02.424418+0000","last_became_peered":"2023-10-19T14:22:02.424418+0000","last_unstale":"2023-10-19T14:22:03.634513+0000","last_undegraded":"2023-10-19T14:22:03.634513+0000","last_fullsized":"2023-10-19T14:22:03.634513+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:11:25.996762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.31","version":"0'0","reported_seq":11,"reported_epoch":263,"state":"active+clean","last_fresh":"2023-10-19T14:22:03.633648+0000","last_change":"2023-10-19T14:22:02.424178+0000","last_active":"2023-10-19T14:22:03.633648+0000","last_peered":"2023-10-19T14:22:03.633648+0000","last_clean":"2023-10-19T14:22:03.633648+0000","last_became_active":"2023-10-19T14:22:02.423883+0000","last_became_peered":"2023-10-19T14:22:02.423883+0000","last_unstale":"2023-10-19T14:22:03.633648+0000","last_undegraded":"2023-10-19T14:22:03.633648+0000","last_fullsized":"2023-10-19T14:22:03.633648+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:38:11.628588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.30","version":"0'0","reported_seq":11,"reported_epoch":263,"state":"active+clean","last_fresh":"2023-10-19T14:22:03.635194+0000","last_change":"2023-10-19T14:22:02.420492+0000","last_active":"2023-10-19T14:22:03.635194+0000","last_peered":"2023-10-19T14:22:03.635194+0000","last_clean":"2023-10-19T14:22:03.635194+0000","last_became_active":"2023-10-19T14:22:02.420210+0000","last_became_peered":"2023-10-19T14:22:02.420210+0000","last_unstale":"2023-10-19T14:22:03.635194+0000","last_undegraded":"2023-10-19T14:22:03.635194+0000","last_fullsized":"2023-10-19T14:22:03.635194+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:45:36.413813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.2f","version":"0'0","reported_seq":11,"reported_epoch":263,"state":"active+clean","last_fresh":"2023-10-19T14:22:03.635101+0000","last_change":"2023-10-19T14:22:02.415526+0000","last_active":"2023-10-19T14:22:03.635101+0000","last_peered":"2023-10-19T14:22:03.635101+0000","last_clean":"2023-10-19T14:22:03.635101+0000","last_became_active":"2023-10-19T14:22:02.415243+0000","last_became_peered":"2023-10-19T14:22:02.415243+0000","last_unstale":"2023-10-19T14:22:03.635101+0000","last_undegraded":"2023-10-19T14:22:03.635101+0000","last_fullsized":"2023-10-19T14:22:03.635101+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:57:31.349828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.2e","version":"0'0","reported_seq":11,"reported_epoch":263,"state":"active+clean","last_fresh":"2023-10-19T14:22:03.634104+0000","last_change":"2023-10-19T14:22:02.424901+0000","last_active":"2023-10-19T14:22:03.634104+0000","last_peered":"2023-10-19T14:22:03.634104+0000","last_clean":"2023-10-19T14:22:03.634104+0000","last_became_active":"2023-10-19T14:22:02.424648+0000","last_became_peered":"2023-10-19T14:22:02.424648+0000","last_unstale":"2023-10-19T14:22:03.634104+0000","last_undegraded":"2023-10-19T14:22:03.634104+0000","last_fullsized":"2023-10-19T14:22:03.634104+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:12:17.865640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.2d","version":"0'0","reported_seq":11,"reported_epoch":263,"state":"active+clean","last_fresh":"2023-10-19T14:22:03.635459+0000","last_change":"2023-10-19T14:22:02.418292+0000","last_active":"2023-10-19T14:22:03.635459+0000","last_peered":"2023-10-19T14:22:03.635459+0000","last_clean":"2023-10-19T14:22:03.635459+0000","last_became_active":"2023-10-19T14:22:02.415107+0000","last_became_peered":"2023-10-19T14:22:02.415107+0000","last_unstale":"2023-10-19T14:22:03.635459+0000","last_undegraded":"2023-10-19T14:22:03.635459+0000","last_fullsized":"2023-10-19T14:22:03.635459+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:14:21.798155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.2c","version":"0'0","reported_seq":11,"reported_epoch":263,"state":"active+clean","last_fresh":"2023-10-19T14:22:03.634905+0000","last_change":"2023-10-19T14:22:02.421040+0000","last_active":"2023-10-19T14:22:03.634905+0000","last_peered":"2023-10-19T14:22:03.634905+0000","last_clean":"2023-10-19T14:22:03.634905+0000","last_became_active":"2023-10-19T14:22:02.420785+0000","last_became_peered":"2023-10-19T14:22:02.420785+0000","last_unstale":"2023-10-19T14:22:03.634905+0000","last_undegraded":"2023-10-19T14:22:03.634905+0000","last_fullsized":"2023-10-19T14:22:03.634905+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:08:39.730633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.2b","version":"0'0","reported_seq":11,"reported_epoch":263,"state":"active+clean","last_fresh":"2023-10-19T14:22:03.414972+0000","last_change":"2023-10-19T14:22:02.418617+0000","last_active":"2023-10-19T14:22:03.414972+0000","last_peered":"2023-10-19T14:22:03.414972+0000","last_clean":"2023-10-19T14:22:03.414972+0000","last_became_active":"2023-10-19T14:22:02.418348+0000","last_became_peered":"2023-10-19T14:22:02.418348+0000","last_unstale":"2023-10-19T14:22:03.414972+0000","last_undegraded":"2023-10-19T14:22:03.414972+0000","last_fullsized":"2023-10-19T14:22:03.414972+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:25:29.063337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.2a","version":"0'0","reported_seq":11,"reported_epoch":263,"state":"active+clean","last_fresh":"2023-10-19T14:22:03.586367+0000","last_change":"2023-10-19T14:22:02.419562+0000","last_active":"2023-10-19T14:22:03.586367+0000","last_peered":"2023-10-19T14:22:03.586367+0000","last_clean":"2023-10-19T14:22:03.586367+0000","last_became_active":"2023-10-19T14:22:02.419299+0000","last_became_peered":"2023-10-19T14:22:02.419299+0000","last_unstale":"2023-10-19T14:22:03.586367+0000","last_undegraded":"2023-10-19T14:22:03.586367+0000","last_fullsized":"2023-10-19T14:22:03.586367+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:05:35.764355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.29","version":"0'0","reported_seq":11,"reported_epoch":263,"state":"active+clean","last_fresh":"2023-10-19T14:22:03.634585+0000","last_change":"2023-10-19T14:22:02.423503+0000","last_active":"2023-10-19T14:22:03.634585+0000","last_peered":"2023-10-19T14:22:03.634585+0000","last_clean":"2023-10-19T14:22:03.634585+0000","last_became_active":"2023-10-19T14:22:02.423247+0000","last_became_peered":"2023-10-19T14:22:02.423247+0000","last_unstale":"2023-10-19T14:22:03.634585+0000","last_undegraded":"2023-10-19T14:22:03.634585+0000","last_fullsized":"2023-10-19T14:22:03.634585+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:52:44.254820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.28","version":"0'0","reported_seq":11,"reported_epoch":263,"state":"active+clean","last_fresh":"2023-10-19T14:22:03.586189+0000","last_change":"2023-10-19T14:22:02.416085+0000","last_active":"2023-10-19T14:22:03.586189+0000","last_peered":"2023-10-19T14:22:03.586189+0000","last_clean":"2023-10-19T14:22:03.586189+0000","last_became_active":"2023-10-19T14:22:02.415831+0000","last_became_peered":"2023-10-19T14:22:02.415831+0000","last_unstale":"2023-10-19T14:22:03.586189+0000","last_undegraded":"2023-10-19T14:22:03.586189+0000","last_fullsized":"2023-10-19T14:22:03.586189+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:51:27.534806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.27","version":"0'0","reported_seq":11,"reported_epoch":263,"state":"active+clean","last_fresh":"2023-10-19T14:22:03.635801+0000","last_change":"2023-10-19T14:22:02.417381+0000","last_active":"2023-10-19T14:22:03.635801+0000","last_peered":"2023-10-19T14:22:03.635801+0000","last_clean":"2023-10-19T14:22:03.635801+0000","last_became_active":"2023-10-19T14:22:02.417103+0000","last_became_peered":"2023-10-19T14:22:02.417103+0000","last_unstale":"2023-10-19T14:22:03.635801+0000","last_undegraded":"2023-10-19T14:22:03.635801+0000","last_fullsized":"2023-10-19T14:22:03.635801+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:22:35.391371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.26","version":"0'0","reported_seq":11,"reported_epoch":263,"state":"active+clean","last_fresh":"2023-10-19T14:22:03.415478+0000","last_change":"2023-10-19T14:22:02.426904+0000","last_active":"2023-10-19T14:22:03.415478+0000","last_peered":"2023-10-19T14:22:03.415478+0000","last_clean":"2023-10-19T14:22:03.415478+0000","last_became_active":"2023-10-19T14:22:02.426634+0000","last_became_peered":"2023-10-19T14:22:02.426634+0000","last_unstale":"2023-10-19T14:22:03.415478+0000","last_undegraded":"2023-10-19T14:22:03.415478+0000","last_fullsized":"2023-10-19T14:22:03.415478+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:16:24.272266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.24","version":"0'0","reported_seq":11,"reported_epoch":263,"state":"active+clean","last_fresh":"2023-10-19T14:22:03.415185+0000","last_change":"2023-10-19T14:22:02.421854+0000","last_active":"2023-10-19T14:22:03.415185+0000","last_peered":"2023-10-19T14:22:03.415185+0000","last_clean":"2023-10-19T14:22:03.415185+0000","last_became_active":"2023-10-19T14:22:02.421569+0000","last_became_peered":"2023-10-19T14:22:02.421569+0000","last_unstale":"2023-10-19T14:22:03.415185+0000","last_undegraded":"2023-10-19T14:22:03.415185+0000","last_fullsized":"2023-10-19T14:22:03.415185+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:25:15.074214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.23","version":"0'0","reported_seq":11,"reported_epoch":263,"state":"active+clean","last_fresh":"2023-10-19T14:22:03.586508+0000","last_change":"2023-10-19T14:22:02.425144+0000","last_active":"2023-10-19T14:22:03.586508+0000","last_peered":"2023-10-19T14:22:03.586508+0000","last_clean":"2023-10-19T14:22:03.586508+0000","last_became_active":"2023-10-19T14:22:02.423278+0000","last_became_peered":"2023-10-19T14:22:02.423278+0000","last_unstale":"2023-10-19T14:22:03.586508+0000","last_undegraded":"2023-10-19T14:22:03.586508+0000","last_fullsized":"2023-10-19T14:22:03.586508+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:01:04.549750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.22","version":"0'0","reported_seq":11,"reported_epoch":263,"state":"active+clean","last_fresh":"2023-10-19T14:22:03.413142+0000","last_change":"2023-10-19T14:22:02.402715+0000","last_active":"2023-10-19T14:22:03.413142+0000","last_peered":"2023-10-19T14:22:03.413142+0000","last_clean":"2023-10-19T14:22:03.413142+0000","last_became_active":"2023-10-19T14:22:02.402451+0000","last_became_peered":"2023-10-19T14:22:02.402451+0000","last_unstale":"2023-10-19T14:22:03.413142+0000","last_undegraded":"2023-10-19T14:22:03.413142+0000","last_fullsized":"2023-10-19T14:22:03.413142+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:07:55.991807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.21","version":"0'0","reported_seq":11,"reported_epoch":263,"state":"active+clean","last_fresh":"2023-10-19T14:22:03.586188+0000","last_change":"2023-10-19T14:22:02.420644+0000","last_active":"2023-10-19T14:22:03.586188+0000","last_peered":"2023-10-19T14:22:03.586188+0000","last_clean":"2023-10-19T14:22:03.586188+0000","last_became_active":"2023-10-19T14:22:02.419980+0000","last_became_peered":"2023-10-19T14:22:02.419980+0000","last_unstale":"2023-10-19T14:22:03.586188+0000","last_undegraded":"2023-10-19T14:22:03.586188+0000","last_fullsized":"2023-10-19T14:22:03.586188+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:51:38.465202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.20","version":"0'0","reported_seq":11,"reported_epoch":263,"state":"active+clean","last_fresh":"2023-10-19T14:22:03.633615+0000","last_change":"2023-10-19T14:22:02.419819+0000","last_active":"2023-10-19T14:22:03.633615+0000","last_peered":"2023-10-19T14:22:03.633615+0000","last_clean":"2023-10-19T14:22:03.633615+0000","last_became_active":"2023-10-19T14:22:02.419422+0000","last_became_peered":"2023-10-19T14:22:02.419422+0000","last_unstale":"2023-10-19T14:22:03.633615+0000","last_undegraded":"2023-10-19T14:22:03.633615+0000","last_fullsized":"2023-10-19T14:22:03.633615+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:46:10.838031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.18","version":"0'0","reported_seq":11,"reported_epoch":263,"state":"active+clean","last_fresh":"2023-10-19T14:22:03.586188+0000","last_change":"2023-10-19T14:22:02.415452+0000","last_active":"2023-10-19T14:22:03.586188+0000","last_peered":"2023-10-19T14:22:03.586188+0000","last_clean":"2023-10-19T14:22:03.586188+0000","last_became_active":"2023-10-19T14:22:02.415173+0000","last_became_peered":"2023-10-19T14:22:02.415173+0000","last_unstale":"2023-10-19T14:22:03.586188+0000","last_undegraded":"2023-10-19T14:22:03.586188+0000","last_fullsized":"2023-10-19T14:22:03.586188+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:23:41.626620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.1f","version":"262'4","reported_seq":16,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.427196+0000","last_change":"2023-10-19T14:22:01.414254+0000","last_active":"2023-10-19T14:22:02.427196+0000","last_peered":"2023-10-19T14:22:02.427196+0000","last_clean":"2023-10-19T14:22:02.427196+0000","last_became_active":"2023-10-19T14:22:01.413084+0000","last_became_peered":"2023-10-19T14:22:01.413084+0000","last_unstale":"2023-10-19T14:22:02.427196+0000","last_undegraded":"2023-10-19T14:22:02.427196+0000","last_fullsized":"2023-10-19T14:22:02.427196+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:04:17.277980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.19","version":"0'0","reported_seq":11,"reported_epoch":263,"state":"active+clean","last_fresh":"2023-10-19T14:22:03.634752+0000","last_change":"2023-10-19T14:22:02.412925+0000","last_active":"2023-10-19T14:22:03.634752+0000","last_peered":"2023-10-19T14:22:03.634752+0000","last_clean":"2023-10-19T14:22:03.634752+0000","last_became_active":"2023-10-19T14:22:02.412639+0000","last_became_peered":"2023-10-19T14:22:02.412639+0000","last_unstale":"2023-10-19T14:22:03.634752+0000","last_undegraded":"2023-10-19T14:22:03.634752+0000","last_fullsized":"2023-10-19T14:22:03.634752+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:05:22.214955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.1e","version":"262'1","reported_seq":13,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.426144+0000","last_change":"2023-10-19T14:22:01.410546+0000","last_active":"2023-10-19T14:22:02.426144+0000","last_peered":"2023-10-19T14:22:02.426144+0000","last_clean":"2023-10-19T14:22:02.426144+0000","last_became_active":"2023-10-19T14:22:01.410263+0000","last_became_peered":"2023-10-19T14:22:01.410263+0000","last_unstale":"2023-10-19T14:22:02.426144+0000","last_undegraded":"2023-10-19T14:22:02.426144+0000","last_fullsized":"2023-10-19T14:22:02.426144+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:02:12.137179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.1a","version":"0'0","reported_seq":11,"reported_epoch":263,"state":"active+clean","last_fresh":"2023-10-19T14:22:03.634930+0000","last_change":"2023-10-19T14:22:02.425816+0000","last_active":"2023-10-19T14:22:03.634930+0000","last_peered":"2023-10-19T14:22:03.634930+0000","last_clean":"2023-10-19T14:22:03.634930+0000","last_became_active":"2023-10-19T14:22:02.425592+0000","last_became_peered":"2023-10-19T14:22:02.425592+0000","last_unstale":"2023-10-19T14:22:03.634930+0000","last_undegraded":"2023-10-19T14:22:03.634930+0000","last_fullsized":"2023-10-19T14:22:03.634930+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:30:03.900078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.1d","version":"262'2","reported_seq":14,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.422828+0000","last_change":"2023-10-19T14:22:01.417975+0000","last_active":"2023-10-19T14:22:02.422828+0000","last_peered":"2023-10-19T14:22:02.422828+0000","last_clean":"2023-10-19T14:22:02.422828+0000","last_became_active":"2023-10-19T14:22:01.417704+0000","last_became_peered":"2023-10-19T14:22:01.417704+0000","last_unstale":"2023-10-19T14:22:02.422828+0000","last_undegraded":"2023-10-19T14:22:02.422828+0000","last_fullsized":"2023-10-19T14:22:02.422828+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:56:00.152228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.1b","version":"0'0","reported_seq":11,"reported_epoch":263,"state":"active+clean","last_fresh":"2023-10-19T14:22:03.634471+0000","last_change":"2023-10-19T14:22:02.412219+0000","last_active":"2023-10-19T14:22:03.634471+0000","last_peered":"2023-10-19T14:22:03.634471+0000","last_clean":"2023-10-19T14:22:03.634471+0000","last_became_active":"2023-10-19T14:22:02.411959+0000","last_became_peered":"2023-10-19T14:22:02.411959+0000","last_unstale":"2023-10-19T14:22:03.634471+0000","last_undegraded":"2023-10-19T14:22:03.634471+0000","last_fullsized":"2023-10-19T14:22:03.634471+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:49:08.667102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.1c","version":"262'2","reported_seq":14,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.422154+0000","last_change":"2023-10-19T14:22:01.419018+0000","last_active":"2023-10-19T14:22:02.422154+0000","last_peered":"2023-10-19T14:22:02.422154+0000","last_clean":"2023-10-19T14:22:02.422154+0000","last_became_active":"2023-10-19T14:22:01.418769+0000","last_became_peered":"2023-10-19T14:22:01.418769+0000","last_unstale":"2023-10-19T14:22:02.422154+0000","last_undegraded":"2023-10-19T14:22:02.422154+0000","last_fullsized":"2023-10-19T14:22:02.422154+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:59:43.263869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.1c","version":"0'0","reported_seq":11,"reported_epoch":263,"state":"active+clean","last_fresh":"2023-10-19T14:22:03.634202+0000","last_change":"2023-10-19T14:22:02.415430+0000","last_active":"2023-10-19T14:22:03.634202+0000","last_peered":"2023-10-19T14:22:03.634202+0000","last_clean":"2023-10-19T14:22:03.634202+0000","last_became_active":"2023-10-19T14:22:02.415147+0000","last_became_peered":"2023-10-19T14:22:02.415147+0000","last_unstale":"2023-10-19T14:22:03.634202+0000","last_undegraded":"2023-10-19T14:22:03.634202+0000","last_fullsized":"2023-10-19T14:22:03.634202+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:08:39.730633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.1b","version":"262'1","reported_seq":13,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.426761+0000","last_change":"2023-10-19T14:22:01.413953+0000","last_active":"2023-10-19T14:22:02.426761+0000","last_peered":"2023-10-19T14:22:02.426761+0000","last_clean":"2023-10-19T14:22:02.426761+0000","last_became_active":"2023-10-19T14:22:01.413689+0000","last_became_peered":"2023-10-19T14:22:01.413689+0000","last_unstale":"2023-10-19T14:22:02.426761+0000","last_undegraded":"2023-10-19T14:22:02.426761+0000","last_fullsized":"2023-10-19T14:22:02.426761+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:33:45.430960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.1d","version":"0'0","reported_seq":11,"reported_epoch":263,"state":"active+clean","last_fresh":"2023-10-19T14:22:03.586399+0000","last_change":"2023-10-19T14:22:02.414206+0000","last_active":"2023-10-19T14:22:03.586399+0000","last_peered":"2023-10-19T14:22:03.586399+0000","last_clean":"2023-10-19T14:22:03.586399+0000","last_became_active":"2023-10-19T14:22:02.413862+0000","last_became_peered":"2023-10-19T14:22:02.413862+0000","last_unstale":"2023-10-19T14:22:03.586399+0000","last_undegraded":"2023-10-19T14:22:03.586399+0000","last_fullsized":"2023-10-19T14:22:03.586399+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:44:04.845874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.1a","version":"262'1","reported_seq":13,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.424473+0000","last_change":"2023-10-19T14:22:01.409991+0000","last_active":"2023-10-19T14:22:02.424473+0000","last_peered":"2023-10-19T14:22:02.424473+0000","last_clean":"2023-10-19T14:22:02.424473+0000","last_became_active":"2023-10-19T14:22:01.409510+0000","last_became_peered":"2023-10-19T14:22:01.409510+0000","last_unstale":"2023-10-19T14:22:02.424473+0000","last_undegraded":"2023-10-19T14:22:02.424473+0000","last_fullsized":"2023-10-19T14:22:02.424473+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:42:53.919625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.1e","version":"0'0","reported_seq":11,"reported_epoch":263,"state":"active+clean","last_fresh":"2023-10-19T14:22:03.586572+0000","last_change":"2023-10-19T14:22:02.412327+0000","last_active":"2023-10-19T14:22:03.586572+0000","last_peered":"2023-10-19T14:22:03.586572+0000","last_clean":"2023-10-19T14:22:03.586572+0000","last_became_active":"2023-10-19T14:22:02.412057+0000","last_became_peered":"2023-10-19T14:22:02.412057+0000","last_unstale":"2023-10-19T14:22:03.586572+0000","last_undegraded":"2023-10-19T14:22:03.586572+0000","last_fullsized":"2023-10-19T14:22:03.586572+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:06:38.456069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.19","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.417694+0000","last_change":"2023-10-19T14:22:01.416901+0000","last_active":"2023-10-19T14:22:02.417694+0000","last_peered":"2023-10-19T14:22:02.417694+0000","last_clean":"2023-10-19T14:22:02.417694+0000","last_became_active":"2023-10-19T14:22:01.416656+0000","last_became_peered":"2023-10-19T14:22:01.416656+0000","last_unstale":"2023-10-19T14:22:02.417694+0000","last_undegraded":"2023-10-19T14:22:02.417694+0000","last_fullsized":"2023-10-19T14:22:02.417694+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:27:36.174418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.1f","version":"0'0","reported_seq":11,"reported_epoch":263,"state":"active+clean","last_fresh":"2023-10-19T14:22:03.635265+0000","last_change":"2023-10-19T14:22:02.418425+0000","last_active":"2023-10-19T14:22:03.635265+0000","last_peered":"2023-10-19T14:22:03.635265+0000","last_clean":"2023-10-19T14:22:03.635265+0000","last_became_active":"2023-10-19T14:22:02.417278+0000","last_became_peered":"2023-10-19T14:22:02.417278+0000","last_unstale":"2023-10-19T14:22:03.635265+0000","last_undegraded":"2023-10-19T14:22:03.635265+0000","last_fullsized":"2023-10-19T14:22:03.635265+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:55:39.760862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.18","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.414129+0000","last_change":"2023-10-19T14:22:01.412557+0000","last_active":"2023-10-19T14:22:02.414129+0000","last_peered":"2023-10-19T14:22:02.414129+0000","last_clean":"2023-10-19T14:22:02.414129+0000","last_became_active":"2023-10-19T14:22:01.412115+0000","last_became_peered":"2023-10-19T14:22:01.412115+0000","last_unstale":"2023-10-19T14:22:02.414129+0000","last_undegraded":"2023-10-19T14:22:02.414129+0000","last_fullsized":"2023-10-19T14:22:02.414129+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:54:33.650542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.10","version":"0'0","reported_seq":11,"reported_epoch":263,"state":"active+clean","last_fresh":"2023-10-19T14:22:03.633775+0000","last_change":"2023-10-19T14:22:02.419681+0000","last_active":"2023-10-19T14:22:03.633775+0000","last_peered":"2023-10-19T14:22:03.633775+0000","last_clean":"2023-10-19T14:22:03.633775+0000","last_became_active":"2023-10-19T14:22:02.418981+0000","last_became_peered":"2023-10-19T14:22:02.418981+0000","last_unstale":"2023-10-19T14:22:03.633775+0000","last_undegraded":"2023-10-19T14:22:03.633775+0000","last_fullsized":"2023-10-19T14:22:03.633775+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:28:48.304947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.17","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.421792+0000","last_change":"2023-10-19T14:22:01.412962+0000","last_active":"2023-10-19T14:22:02.421792+0000","last_peered":"2023-10-19T14:22:02.421792+0000","last_clean":"2023-10-19T14:22:02.421792+0000","last_became_active":"2023-10-19T14:22:01.412680+0000","last_became_peered":"2023-10-19T14:22:01.412680+0000","last_unstale":"2023-10-19T14:22:02.421792+0000","last_undegraded":"2023-10-19T14:22:02.421792+0000","last_fullsized":"2023-10-19T14:22:02.421792+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:03:54.917148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.11","version":"0'0","reported_seq":11,"reported_epoch":263,"state":"active+clean","last_fresh":"2023-10-19T14:22:03.635193+0000","last_change":"2023-10-19T14:22:02.424612+0000","last_active":"2023-10-19T14:22:03.635193+0000","last_peered":"2023-10-19T14:22:03.635193+0000","last_clean":"2023-10-19T14:22:03.635193+0000","last_became_active":"2023-10-19T14:22:02.424368+0000","last_became_peered":"2023-10-19T14:22:02.424368+0000","last_unstale":"2023-10-19T14:22:03.635193+0000","last_undegraded":"2023-10-19T14:22:03.635193+0000","last_fullsized":"2023-10-19T14:22:03.635193+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:59:00.364099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.16","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.418788+0000","last_change":"2023-10-19T14:22:01.414464+0000","last_active":"2023-10-19T14:22:02.418788+0000","last_peered":"2023-10-19T14:22:02.418788+0000","last_clean":"2023-10-19T14:22:02.418788+0000","last_became_active":"2023-10-19T14:22:01.413414+0000","last_became_peered":"2023-10-19T14:22:01.413414+0000","last_unstale":"2023-10-19T14:22:02.418788+0000","last_undegraded":"2023-10-19T14:22:02.418788+0000","last_fullsized":"2023-10-19T14:22:02.418788+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:46:22.763844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.12","version":"0'0","reported_seq":11,"reported_epoch":263,"state":"active+clean","last_fresh":"2023-10-19T14:22:03.633739+0000","last_change":"2023-10-19T14:22:02.410248+0000","last_active":"2023-10-19T14:22:03.633739+0000","last_peered":"2023-10-19T14:22:03.633739+0000","last_clean":"2023-10-19T14:22:03.633739+0000","last_became_active":"2023-10-19T14:22:02.409982+0000","last_became_peered":"2023-10-19T14:22:02.409982+0000","last_unstale":"2023-10-19T14:22:03.633739+0000","last_undegraded":"2023-10-19T14:22:03.633739+0000","last_fullsized":"2023-10-19T14:22:03.633739+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:19:36.485520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.15","version":"262'3","reported_seq":15,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.428131+0000","last_change":"2023-10-19T14:22:01.409543+0000","last_active":"2023-10-19T14:22:02.428131+0000","last_peered":"2023-10-19T14:22:02.428131+0000","last_clean":"2023-10-19T14:22:02.428131+0000","last_became_active":"2023-10-19T14:22:01.409293+0000","last_became_peered":"2023-10-19T14:22:01.409293+0000","last_unstale":"2023-10-19T14:22:02.428131+0000","last_undegraded":"2023-10-19T14:22:02.428131+0000","last_fullsized":"2023-10-19T14:22:02.428131+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T18:12:16.850155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.13","version":"0'0","reported_seq":11,"reported_epoch":263,"state":"active+clean","last_fresh":"2023-10-19T14:22:03.415486+0000","last_change":"2023-10-19T14:22:02.419148+0000","last_active":"2023-10-19T14:22:03.415486+0000","last_peered":"2023-10-19T14:22:03.415486+0000","last_clean":"2023-10-19T14:22:03.415486+0000","last_became_active":"2023-10-19T14:22:02.418886+0000","last_became_peered":"2023-10-19T14:22:02.418886+0000","last_unstale":"2023-10-19T14:22:03.415486+0000","last_undegraded":"2023-10-19T14:22:03.415486+0000","last_fullsized":"2023-10-19T14:22:03.415486+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:04:18.293465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.14","version":"262'6","reported_seq":18,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.429104+0000","last_change":"2023-10-19T14:22:01.411386+0000","last_active":"2023-10-19T14:22:02.429104+0000","last_peered":"2023-10-19T14:22:02.429104+0000","last_clean":"2023-10-19T14:22:02.429104+0000","last_became_active":"2023-10-19T14:22:01.411111+0000","last_became_peered":"2023-10-19T14:22:01.411111+0000","last_unstale":"2023-10-19T14:22:02.429104+0000","last_undegraded":"2023-10-19T14:22:02.429104+0000","last_fullsized":"2023-10-19T14:22:02.429104+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:10:26.534502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.14","version":"0'0","reported_seq":11,"reported_epoch":263,"state":"active+clean","last_fresh":"2023-10-19T14:22:03.635523+0000","last_change":"2023-10-19T14:22:02.416878+0000","last_active":"2023-10-19T14:22:03.635523+0000","last_peered":"2023-10-19T14:22:03.635523+0000","last_clean":"2023-10-19T14:22:03.635523+0000","last_became_active":"2023-10-19T14:22:02.416633+0000","last_became_peered":"2023-10-19T14:22:02.416633+0000","last_unstale":"2023-10-19T14:22:03.635523+0000","last_undegraded":"2023-10-19T14:22:03.635523+0000","last_fullsized":"2023-10-19T14:22:03.635523+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:17:47.770942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.13","version":"262'2","reported_seq":14,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.422513+0000","last_change":"2023-10-19T14:22:01.410762+0000","last_active":"2023-10-19T14:22:02.422513+0000","last_peered":"2023-10-19T14:22:02.422513+0000","last_clean":"2023-10-19T14:22:02.422513+0000","last_became_active":"2023-10-19T14:22:01.410507+0000","last_became_peered":"2023-10-19T14:22:01.410507+0000","last_unstale":"2023-10-19T14:22:02.422513+0000","last_undegraded":"2023-10-19T14:22:02.422513+0000","last_fullsized":"2023-10-19T14:22:02.422513+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:38:13.918889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.15","version":"0'0","reported_seq":11,"reported_epoch":263,"state":"active+clean","last_fresh":"2023-10-19T14:22:03.634007+0000","last_change":"2023-10-19T14:22:02.414706+0000","last_active":"2023-10-19T14:22:03.634007+0000","last_peered":"2023-10-19T14:22:03.634007+0000","last_clean":"2023-10-19T14:22:03.634007+0000","last_became_active":"2023-10-19T14:22:02.414358+0000","last_became_peered":"2023-10-19T14:22:02.414358+0000","last_unstale":"2023-10-19T14:22:03.634007+0000","last_undegraded":"2023-10-19T14:22:03.634007+0000","last_fullsized":"2023-10-19T14:22:03.634007+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:22:21.202909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.12","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.419288+0000","last_change":"2023-10-19T14:22:01.413870+0000","last_active":"2023-10-19T14:22:02.419288+0000","last_peered":"2023-10-19T14:22:02.419288+0000","last_clean":"2023-10-19T14:22:02.419288+0000","last_became_active":"2023-10-19T14:22:01.413532+0000","last_became_peered":"2023-10-19T14:22:01.413532+0000","last_unstale":"2023-10-19T14:22:02.419288+0000","last_undegraded":"2023-10-19T14:22:02.419288+0000","last_fullsized":"2023-10-19T14:22:02.419288+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:47:48.522113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.16","version":"0'0","reported_seq":11,"reported_epoch":263,"state":"active+clean","last_fresh":"2023-10-19T14:22:03.635648+0000","last_change":"2023-10-19T14:22:02.419152+0000","last_active":"2023-10-19T14:22:03.635648+0000","last_peered":"2023-10-19T14:22:03.635648+0000","last_clean":"2023-10-19T14:22:03.635648+0000","last_became_active":"2023-10-19T14:22:02.417515+0000","last_became_peered":"2023-10-19T14:22:02.417515+0000","last_unstale":"2023-10-19T14:22:03.635648+0000","last_undegraded":"2023-10-19T14:22:03.635648+0000","last_fullsized":"2023-10-19T14:22:03.635648+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T02:13:39.152029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.11","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.419226+0000","last_change":"2023-10-19T14:22:01.416474+0000","last_active":"2023-10-19T14:22:02.419226+0000","last_peered":"2023-10-19T14:22:02.419226+0000","last_clean":"2023-10-19T14:22:02.419226+0000","last_became_active":"2023-10-19T14:22:01.416200+0000","last_became_peered":"2023-10-19T14:22:01.416200+0000","last_unstale":"2023-10-19T14:22:02.419226+0000","last_undegraded":"2023-10-19T14:22:02.419226+0000","last_fullsized":"2023-10-19T14:22:02.419226+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:55:39.137949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.17","version":"0'0","reported_seq":11,"reported_epoch":263,"state":"active+clean","last_fresh":"2023-10-19T14:22:03.635408+0000","last_change":"2023-10-19T14:22:02.419116+0000","last_active":"2023-10-19T14:22:03.635408+0000","last_peered":"2023-10-19T14:22:03.635408+0000","last_clean":"2023-10-19T14:22:03.635408+0000","last_became_active":"2023-10-19T14:22:02.418655+0000","last_became_peered":"2023-10-19T14:22:02.418655+0000","last_unstale":"2023-10-19T14:22:03.635408+0000","last_undegraded":"2023-10-19T14:22:03.635408+0000","last_fullsized":"2023-10-19T14:22:03.635408+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:15:08.109573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.10","version":"262'8","reported_seq":20,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.427889+0000","last_change":"2023-10-19T14:22:01.415486+0000","last_active":"2023-10-19T14:22:02.427889+0000","last_peered":"2023-10-19T14:22:02.427889+0000","last_clean":"2023-10-19T14:22:02.427889+0000","last_became_active":"2023-10-19T14:22:01.415192+0000","last_became_peered":"2023-10-19T14:22:01.415192+0000","last_unstale":"2023-10-19T14:22:02.427889+0000","last_undegraded":"2023-10-19T14:22:02.427889+0000","last_fullsized":"2023-10-19T14:22:02.427889+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:45:03.326043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.8","version":"0'0","reported_seq":11,"reported_epoch":263,"state":"active+clean","last_fresh":"2023-10-19T14:22:03.415089+0000","last_change":"2023-10-19T14:22:02.417062+0000","last_active":"2023-10-19T14:22:03.415089+0000","last_peered":"2023-10-19T14:22:03.415089+0000","last_clean":"2023-10-19T14:22:03.415089+0000","last_became_active":"2023-10-19T14:22:02.416775+0000","last_became_peered":"2023-10-19T14:22:02.416775+0000","last_unstale":"2023-10-19T14:22:03.415089+0000","last_undegraded":"2023-10-19T14:22:03.415089+0000","last_fullsized":"2023-10-19T14:22:03.415089+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:54:56.271440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.f","version":"262'1","reported_seq":13,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.420761+0000","last_change":"2023-10-19T14:22:01.417521+0000","last_active":"2023-10-19T14:22:02.420761+0000","last_peered":"2023-10-19T14:22:02.420761+0000","last_clean":"2023-10-19T14:22:02.420761+0000","last_became_active":"2023-10-19T14:22:01.417209+0000","last_became_peered":"2023-10-19T14:22:01.417209+0000","last_unstale":"2023-10-19T14:22:02.420761+0000","last_undegraded":"2023-10-19T14:22:02.420761+0000","last_fullsized":"2023-10-19T14:22:02.420761+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:43:26.494066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.9","version":"0'0","reported_seq":11,"reported_epoch":263,"state":"active+clean","last_fresh":"2023-10-19T14:22:03.586187+0000","last_change":"2023-10-19T14:22:02.414803+0000","last_active":"2023-10-19T14:22:03.586187+0000","last_peered":"2023-10-19T14:22:03.586187+0000","last_clean":"2023-10-19T14:22:03.586187+0000","last_became_active":"2023-10-19T14:22:02.414533+0000","last_became_peered":"2023-10-19T14:22:02.414533+0000","last_unstale":"2023-10-19T14:22:03.586187+0000","last_undegraded":"2023-10-19T14:22:03.586187+0000","last_fullsized":"2023-10-19T14:22:03.586187+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:09:21.558278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.e","version":"262'2","reported_seq":14,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.425280+0000","last_change":"2023-10-19T14:22:01.412531+0000","last_active":"2023-10-19T14:22:02.425280+0000","last_peered":"2023-10-19T14:22:02.425280+0000","last_clean":"2023-10-19T14:22:02.425280+0000","last_became_active":"2023-10-19T14:22:01.412209+0000","last_became_peered":"2023-10-19T14:22:01.412209+0000","last_unstale":"2023-10-19T14:22:02.425280+0000","last_undegraded":"2023-10-19T14:22:02.425280+0000","last_fullsized":"2023-10-19T14:22:02.425280+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:45:03.326043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.a","version":"0'0","reported_seq":11,"reported_epoch":263,"state":"active+clean","last_fresh":"2023-10-19T14:22:03.415307+0000","last_change":"2023-10-19T14:22:02.417612+0000","last_active":"2023-10-19T14:22:03.415307+0000","last_peered":"2023-10-19T14:22:03.415307+0000","last_clean":"2023-10-19T14:22:03.415307+0000","last_became_active":"2023-10-19T14:22:02.417352+0000","last_became_peered":"2023-10-19T14:22:02.417352+0000","last_unstale":"2023-10-19T14:22:03.415307+0000","last_undegraded":"2023-10-19T14:22:03.415307+0000","last_fullsized":"2023-10-19T14:22:03.415307+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:43:57.998020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.d","version":"262'1","reported_seq":13,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.428855+0000","last_change":"2023-10-19T14:22:01.415741+0000","last_active":"2023-10-19T14:22:02.428855+0000","last_peered":"2023-10-19T14:22:02.428855+0000","last_clean":"2023-10-19T14:22:02.428855+0000","last_became_active":"2023-10-19T14:22:01.415385+0000","last_became_peered":"2023-10-19T14:22:01.415385+0000","last_unstale":"2023-10-19T14:22:02.428855+0000","last_undegraded":"2023-10-19T14:22:02.428855+0000","last_fullsized":"2023-10-19T14:22:02.428855+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:34:06.962782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.b","version":"0'0","reported_seq":11,"reported_epoch":263,"state":"active+clean","last_fresh":"2023-10-19T14:22:03.634007+0000","last_change":"2023-10-19T14:22:02.413768+0000","last_active":"2023-10-19T14:22:03.634007+0000","last_peered":"2023-10-19T14:22:03.634007+0000","last_clean":"2023-10-19T14:22:03.634007+0000","last_became_active":"2023-10-19T14:22:02.413498+0000","last_became_peered":"2023-10-19T14:22:02.413498+0000","last_unstale":"2023-10-19T14:22:03.634007+0000","last_undegraded":"2023-10-19T14:22:03.634007+0000","last_fullsized":"2023-10-19T14:22:03.634007+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:43:55.093530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.c","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.417784+0000","last_change":"2023-10-19T14:22:01.411380+0000","last_active":"2023-10-19T14:22:02.417784+0000","last_peered":"2023-10-19T14:22:02.417784+0000","last_clean":"2023-10-19T14:22:02.417784+0000","last_became_active":"2023-10-19T14:22:01.411092+0000","last_became_peered":"2023-10-19T14:22:01.411092+0000","last_unstale":"2023-10-19T14:22:02.417784+0000","last_undegraded":"2023-10-19T14:22:02.417784+0000","last_fullsized":"2023-10-19T14:22:02.417784+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:01:14.941331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.c","version":"0'0","reported_seq":11,"reported_epoch":263,"state":"active+clean","last_fresh":"2023-10-19T14:22:03.586407+0000","last_change":"2023-10-19T14:22:02.413516+0000","last_active":"2023-10-19T14:22:03.586407+0000","last_peered":"2023-10-19T14:22:03.586407+0000","last_clean":"2023-10-19T14:22:03.586407+0000","last_became_active":"2023-10-19T14:22:02.413230+0000","last_became_peered":"2023-10-19T14:22:02.413230+0000","last_unstale":"2023-10-19T14:22:03.586407+0000","last_undegraded":"2023-10-19T14:22:03.586407+0000","last_fullsized":"2023-10-19T14:22:03.586407+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:39:31.175508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.b","version":"262'2","reported_seq":14,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.424065+0000","last_change":"2023-10-19T14:22:01.409927+0000","last_active":"2023-10-19T14:22:02.424065+0000","last_peered":"2023-10-19T14:22:02.424065+0000","last_clean":"2023-10-19T14:22:02.424065+0000","last_became_active":"2023-10-19T14:22:01.409617+0000","last_became_peered":"2023-10-19T14:22:01.409617+0000","last_unstale":"2023-10-19T14:22:02.424065+0000","last_undegraded":"2023-10-19T14:22:02.424065+0000","last_fullsized":"2023-10-19T14:22:02.424065+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:38:30.217600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.d","version":"0'0","reported_seq":11,"reported_epoch":263,"state":"active+clean","last_fresh":"2023-10-19T14:22:03.586501+0000","last_change":"2023-10-19T14:22:02.418945+0000","last_active":"2023-10-19T14:22:03.586501+0000","last_peered":"2023-10-19T14:22:03.586501+0000","last_clean":"2023-10-19T14:22:03.586501+0000","last_became_active":"2023-10-19T14:22:02.418667+0000","last_became_peered":"2023-10-19T14:22:02.418667+0000","last_unstale":"2023-10-19T14:22:03.586501+0000","last_undegraded":"2023-10-19T14:22:03.586501+0000","last_fullsized":"2023-10-19T14:22:03.586501+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T14:59:05.323967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.a","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.414263+0000","last_change":"2023-10-19T14:22:01.417576+0000","last_active":"2023-10-19T14:22:02.414263+0000","last_peered":"2023-10-19T14:22:02.414263+0000","last_clean":"2023-10-19T14:22:02.414263+0000","last_became_active":"2023-10-19T14:22:01.417230+0000","last_became_peered":"2023-10-19T14:22:01.417230+0000","last_unstale":"2023-10-19T14:22:02.414263+0000","last_undegraded":"2023-10-19T14:22:02.414263+0000","last_fullsized":"2023-10-19T14:22:02.414263+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:23:00.582706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.e","version":"0'0","reported_seq":11,"reported_epoch":263,"state":"active+clean","last_fresh":"2023-10-19T14:22:03.635676+0000","last_change":"2023-10-19T14:22:02.419344+0000","last_active":"2023-10-19T14:22:03.635676+0000","last_peered":"2023-10-19T14:22:03.635676+0000","last_clean":"2023-10-19T14:22:03.635676+0000","last_became_active":"2023-10-19T14:22:02.417238+0000","last_became_peered":"2023-10-19T14:22:02.417238+0000","last_unstale":"2023-10-19T14:22:03.635676+0000","last_undegraded":"2023-10-19T14:22:03.635676+0000","last_fullsized":"2023-10-19T14:22:03.635676+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:20:03.809604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.9","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.413661+0000","last_change":"2023-10-19T14:22:01.413457+0000","last_active":"2023-10-19T14:22:02.413661+0000","last_peered":"2023-10-19T14:22:02.413661+0000","last_clean":"2023-10-19T14:22:02.413661+0000","last_became_active":"2023-10-19T14:22:01.413179+0000","last_became_peered":"2023-10-19T14:22:01.413179+0000","last_unstale":"2023-10-19T14:22:02.413661+0000","last_undegraded":"2023-10-19T14:22:02.413661+0000","last_fullsized":"2023-10-19T14:22:02.413661+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:34:30.334044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.f","version":"0'0","reported_seq":11,"reported_epoch":263,"state":"active+clean","last_fresh":"2023-10-19T14:22:03.415331+0000","last_change":"2023-10-19T14:22:02.426370+0000","last_active":"2023-10-19T14:22:03.415331+0000","last_peered":"2023-10-19T14:22:03.415331+0000","last_clean":"2023-10-19T14:22:03.415331+0000","last_became_active":"2023-10-19T14:22:02.426109+0000","last_became_peered":"2023-10-19T14:22:02.426109+0000","last_unstale":"2023-10-19T14:22:03.415331+0000","last_undegraded":"2023-10-19T14:22:03.415331+0000","last_fullsized":"2023-10-19T14:22:03.415331+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:10:27.549987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.8","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.417655+0000","last_change":"2023-10-19T14:22:01.413949+0000","last_active":"2023-10-19T14:22:02.417655+0000","last_peered":"2023-10-19T14:22:02.417655+0000","last_clean":"2023-10-19T14:22:02.417655+0000","last_became_active":"2023-10-19T14:22:01.413670+0000","last_became_peered":"2023-10-19T14:22:01.413670+0000","last_unstale":"2023-10-19T14:22:02.417655+0000","last_undegraded":"2023-10-19T14:22:02.417655+0000","last_fullsized":"2023-10-19T14:22:02.417655+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:33:35.792891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.0","version":"0'0","reported_seq":11,"reported_epoch":263,"state":"active+clean","last_fresh":"2023-10-19T14:22:03.633926+0000","last_change":"2023-10-19T14:22:02.418876+0000","last_active":"2023-10-19T14:22:03.633926+0000","last_peered":"2023-10-19T14:22:03.633926+0000","last_clean":"2023-10-19T14:22:03.633926+0000","last_became_active":"2023-10-19T14:22:02.417450+0000","last_became_peered":"2023-10-19T14:22:02.417450+0000","last_unstale":"2023-10-19T14:22:03.633926+0000","last_undegraded":"2023-10-19T14:22:03.633926+0000","last_fullsized":"2023-10-19T14:22:03.633926+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:25:08.178761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.7","version":"262'2","reported_seq":14,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.418809+0000","last_change":"2023-10-19T14:22:01.412421+0000","last_active":"2023-10-19T14:22:02.418809+0000","last_peered":"2023-10-19T14:22:02.418809+0000","last_clean":"2023-10-19T14:22:02.418809+0000","last_became_active":"2023-10-19T14:22:01.412153+0000","last_became_peered":"2023-10-19T14:22:01.412153+0000","last_unstale":"2023-10-19T14:22:02.418809+0000","last_undegraded":"2023-10-19T14:22:02.418809+0000","last_fullsized":"2023-10-19T14:22:02.418809+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:38:47.923739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.1","version":"0'0","reported_seq":11,"reported_epoch":263,"state":"active+clean","last_fresh":"2023-10-19T14:22:03.635530+0000","last_change":"2023-10-19T14:22:02.419843+0000","last_active":"2023-10-19T14:22:03.635530+0000","last_peered":"2023-10-19T14:22:03.635530+0000","last_clean":"2023-10-19T14:22:03.635530+0000","last_became_active":"2023-10-19T14:22:02.417481+0000","last_became_peered":"2023-10-19T14:22:02.417481+0000","last_unstale":"2023-10-19T14:22:03.635530+0000","last_undegraded":"2023-10-19T14:22:03.635530+0000","last_fullsized":"2023-10-19T14:22:03.635530+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T20:03:39.766415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.6","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.419464+0000","last_change":"2023-10-19T14:22:01.412088+0000","last_active":"2023-10-19T14:22:02.419464+0000","last_peered":"2023-10-19T14:22:02.419464+0000","last_clean":"2023-10-19T14:22:02.419464+0000","last_became_active":"2023-10-19T14:22:01.411646+0000","last_became_peered":"2023-10-19T14:22:01.411646+0000","last_unstale":"2023-10-19T14:22:02.419464+0000","last_undegraded":"2023-10-19T14:22:02.419464+0000","last_fullsized":"2023-10-19T14:22:02.419464+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:48:08.338501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.2","version":"0'0","reported_seq":11,"reported_epoch":263,"state":"active+clean","last_fresh":"2023-10-19T14:22:03.633679+0000","last_change":"2023-10-19T14:22:02.417069+0000","last_active":"2023-10-19T14:22:03.633679+0000","last_peered":"2023-10-19T14:22:03.633679+0000","last_clean":"2023-10-19T14:22:03.633679+0000","last_became_active":"2023-10-19T14:22:02.416788+0000","last_became_peered":"2023-10-19T14:22:02.416788+0000","last_unstale":"2023-10-19T14:22:03.633679+0000","last_undegraded":"2023-10-19T14:22:03.633679+0000","last_fullsized":"2023-10-19T14:22:03.633679+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T17:12:12.862561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.5","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.420734+0000","last_change":"2023-10-19T14:22:01.417906+0000","last_active":"2023-10-19T14:22:02.420734+0000","last_peered":"2023-10-19T14:22:02.420734+0000","last_clean":"2023-10-19T14:22:02.420734+0000","last_became_active":"2023-10-19T14:22:01.417614+0000","last_became_peered":"2023-10-19T14:22:01.417614+0000","last_unstale":"2023-10-19T14:22:02.420734+0000","last_undegraded":"2023-10-19T14:22:02.420734+0000","last_fullsized":"2023-10-19T14:22:02.420734+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:26:05.681833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.3","version":"0'0","reported_seq":11,"reported_epoch":263,"state":"active+clean","last_fresh":"2023-10-19T14:22:03.634624+0000","last_change":"2023-10-19T14:22:02.417066+0000","last_active":"2023-10-19T14:22:03.634624+0000","last_peered":"2023-10-19T14:22:03.634624+0000","last_clean":"2023-10-19T14:22:03.634624+0000","last_became_active":"2023-10-19T14:22:02.416770+0000","last_became_peered":"2023-10-19T14:22:02.416770+0000","last_unstale":"2023-10-19T14:22:03.634624+0000","last_undegraded":"2023-10-19T14:22:03.634624+0000","last_fullsized":"2023-10-19T14:22:03.634624+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:36:14.408276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.4","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.419386+0000","last_change":"2023-10-19T14:22:01.412056+0000","last_active":"2023-10-19T14:22:02.419386+0000","last_peered":"2023-10-19T14:22:02.419386+0000","last_clean":"2023-10-19T14:22:02.419386+0000","last_became_active":"2023-10-19T14:22:01.411786+0000","last_became_peered":"2023-10-19T14:22:01.411786+0000","last_unstale":"2023-10-19T14:22:02.419386+0000","last_undegraded":"2023-10-19T14:22:02.419386+0000","last_fullsized":"2023-10-19T14:22:02.419386+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T16:40:51.932206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.4","version":"0'0","reported_seq":11,"reported_epoch":263,"state":"active+clean","last_fresh":"2023-10-19T14:22:03.634931+0000","last_change":"2023-10-19T14:22:02.423498+0000","last_active":"2023-10-19T14:22:03.634931+0000","last_peered":"2023-10-19T14:22:03.634931+0000","last_clean":"2023-10-19T14:22:03.634931+0000","last_became_active":"2023-10-19T14:22:02.423212+0000","last_became_peered":"2023-10-19T14:22:02.423212+0000","last_unstale":"2023-10-19T14:22:03.634931+0000","last_undegraded":"2023-10-19T14:22:03.634931+0000","last_fullsized":"2023-10-19T14:22:03.634931+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:22:09.767658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.3","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.417768+0000","last_change":"2023-10-19T14:22:01.413689+0000","last_active":"2023-10-19T14:22:02.417768+0000","last_peered":"2023-10-19T14:22:02.417768+0000","last_clean":"2023-10-19T14:22:02.417768+0000","last_became_active":"2023-10-19T14:22:01.413419+0000","last_became_peered":"2023-10-19T14:22:01.413419+0000","last_unstale":"2023-10-19T14:22:02.417768+0000","last_undegraded":"2023-10-19T14:22:02.417768+0000","last_fullsized":"2023-10-19T14:22:02.417768+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:37:57.754265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.5","version":"0'0","reported_seq":11,"reported_epoch":263,"state":"active+clean","last_fresh":"2023-10-19T14:22:03.634126+0000","last_change":"2023-10-19T14:22:02.409126+0000","last_active":"2023-10-19T14:22:03.634126+0000","last_peered":"2023-10-19T14:22:03.634126+0000","last_clean":"2023-10-19T14:22:03.634126+0000","last_became_active":"2023-10-19T14:22:02.408855+0000","last_became_peered":"2023-10-19T14:22:02.408855+0000","last_unstale":"2023-10-19T14:22:03.634126+0000","last_undegraded":"2023-10-19T14:22:03.634126+0000","last_fullsized":"2023-10-19T14:22:03.634126+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T00:25:21.597207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.2","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.414293+0000","last_change":"2023-10-19T14:22:01.411905+0000","last_active":"2023-10-19T14:22:02.414293+0000","last_peered":"2023-10-19T14:22:02.414293+0000","last_clean":"2023-10-19T14:22:02.414293+0000","last_became_active":"2023-10-19T14:22:01.411584+0000","last_became_peered":"2023-10-19T14:22:01.411584+0000","last_unstale":"2023-10-19T14:22:02.414293+0000","last_undegraded":"2023-10-19T14:22:02.414293+0000","last_fullsized":"2023-10-19T14:22:02.414293+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T15:40:13.932927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.6","version":"0'0","reported_seq":11,"reported_epoch":263,"state":"active+clean","last_fresh":"2023-10-19T14:22:03.415496+0000","last_change":"2023-10-19T14:22:02.424130+0000","last_active":"2023-10-19T14:22:03.415496+0000","last_peered":"2023-10-19T14:22:03.415496+0000","last_clean":"2023-10-19T14:22:03.415496+0000","last_became_active":"2023-10-19T14:22:02.423859+0000","last_became_peered":"2023-10-19T14:22:02.423859+0000","last_unstale":"2023-10-19T14:22:03.415496+0000","last_undegraded":"2023-10-19T14:22:03.415496+0000","last_fullsized":"2023-10-19T14:22:03.415496+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T19:03:55.524277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.1","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.421657+0000","last_change":"2023-10-19T14:22:01.412806+0000","last_active":"2023-10-19T14:22:02.421657+0000","last_peered":"2023-10-19T14:22:02.421657+0000","last_clean":"2023-10-19T14:22:02.421657+0000","last_became_active":"2023-10-19T14:22:01.412524+0000","last_became_peered":"2023-10-19T14:22:01.412524+0000","last_unstale":"2023-10-19T14:22:02.421657+0000","last_undegraded":"2023-10-19T14:22:02.421657+0000","last_fullsized":"2023-10-19T14:22:02.421657+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:43:54.078045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.7","version":"0'0","reported_seq":11,"reported_epoch":263,"state":"active+clean","last_fresh":"2023-10-19T14:22:03.634102+0000","last_change":"2023-10-19T14:22:02.414678+0000","last_active":"2023-10-19T14:22:03.634102+0000","last_peered":"2023-10-19T14:22:03.634102+0000","last_clean":"2023-10-19T14:22:03.634102+0000","last_became_active":"2023-10-19T14:22:02.414348+0000","last_became_peered":"2023-10-19T14:22:02.414348+0000","last_unstale":"2023-10-19T14:22:03.634102+0000","last_undegraded":"2023-10-19T14:22:03.634102+0000","last_fullsized":"2023-10-19T14:22:03.634102+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:03:55.932633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.0","version":"0'0","reported_seq":12,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.417676+0000","last_change":"2023-10-19T14:22:01.414455+0000","last_active":"2023-10-19T14:22:02.417676+0000","last_peered":"2023-10-19T14:22:02.417676+0000","last_clean":"2023-10-19T14:22:02.417676+0000","last_became_active":"2023-10-19T14:22:01.414081+0000","last_became_peered":"2023-10-19T14:22:01.414081+0000","last_unstale":"2023-10-19T14:22:02.417676+0000","last_undegraded":"2023-10-19T14:22:02.417676+0000","last_fullsized":"2023-10-19T14:22:02.417676+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:00.371036+0000","last_clean_scrub_stamp":"2023-10-19T14:22:00.371036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T23:54:25.277401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.25","version":"0'0","reported_seq":11,"reported_epoch":263,"state":"active+clean","last_fresh":"2023-10-19T14:22:03.413439+0000","last_change":"2023-10-19T14:22:02.418670+0000","last_active":"2023-10-19T14:22:03.413439+0000","last_peered":"2023-10-19T14:22:03.413439+0000","last_clean":"2023-10-19T14:22:03.413439+0000","last_became_active":"2023-10-19T14:22:02.417862+0000","last_became_peered":"2023-10-19T14:22:02.417862+0000","last_unstale":"2023-10-19T14:22:03.413439+0000","last_undegraded":"2023-10-19T14:22:03.413439+0000","last_fullsized":"2023-10-19T14:22:03.413439+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":261,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T14:22:01.386521+0000","last_clean_scrub_stamp":"2023-10-19T14:22:01.386521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-21T01:14:37.969686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"249'3224","reported_seq":4254,"reported_epoch":262,"state":"active+clean","last_fresh":"2023-10-19T14:22:02.402835+0000","last_change":"2023-10-19T12:44:08.636613+0000","last_active":"2023-10-19T14:22:02.402835+0000","last_peered":"2023-10-19T14:22:02.402835+0000","last_clean":"2023-10-19T14:22:02.402835+0000","last_became_active":"2023-10-19T12:44:08.635697+0000","last_became_peered":"2023-10-19T12:44:08.635697+0000","last_unstale":"2023-10-19T14:22:02.402835+0000","last_undegraded":"2023-10-19T14:22:02.402835+0000","last_fullsized":"2023-10-19T14:22:02.402835+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_clean_scrub_stamp":"2023-10-19T12:44:07.399738+0000","objects_scrubbed":0,"log_size":3224,"log_dups_size":0,"ondisk_log_size":3224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:15:23.685485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2539,"num_read_kb":5339,"num_write":657,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":36,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":35,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2539,"num_read_kb":5339,"num_write":657,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3224,"ondisk_log_size":3224,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576021,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66968,"kb_used_data":3848,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94304872,"statfs":{"total":96636764160,"available":96568188928,"internally_reserved":0,"allocated":3940352,"data_stored":2310596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":64616907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:21:10 2023","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.55100000000000005,"15min":0.57999999999999996},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.32200000000000001},"max":{"1min":0.754,"5min":0.80900000000000005,"15min":0.86199999999999999},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.503,"5min":0.57599999999999996,"15min":0.59899999999999998},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.312},"max":{"1min":0.66500000000000004,"5min":0.81899999999999995,"15min":0.86599999999999999},"last":0.52200000000000002}]},{"osd":1,"last update":"Thu Oct 19 14:21:10 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.58199999999999996,"15min":0.59499999999999997},"min":{"1min":0.45200000000000001,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.72199999999999998,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.56899999999999995,"15min":0.59299999999999997},"min":{"1min":0.39300000000000002,"5min":0.373,"15min":0.36699999999999999},"max":{"1min":0.78500000000000003,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.72899999999999998}]},{"osd":2,"last update":"Thu Oct 19 14:21:10 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.622,"15min":0.64200000000000002},"min":{"1min":0.47299999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.70499999999999996,"5min":0.84799999999999998,"15min":0.85499999999999998},"last":0.754},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.63900000000000001,"15min":0.65700000000000003},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.61299999999999999}]},{"osd":3,"last update":"Thu Oct 19 14:21:10 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.60799999999999998,"15min":0.63100000000000001},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.628,"5min":0.67100000000000004,"15min":0.68300000000000005},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.83299999999999996,"5min":0.876,"15min":0.89600000000000002},"last":0.77600000000000002}]},{"osd":4,"last update":"Thu Oct 19 14:21:10 2023","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.50600000000000001,"15min":0.495},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.53000000000000003,"15min":0.51100000000000001},"min":{"1min":0.34999999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.58899999999999997}]},{"osd":5,"last update":"Thu Oct 19 14:21:10 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52300000000000002,"15min":0.51100000000000001},"min":{"1min":0.33700000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.57799999999999996,"15min":0.55400000000000005},"min":{"1min":0.40699999999999997,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.054,"5min":1.054,"15min":1.054},"last":0.69799999999999995}]},{"osd":6,"last update":"Thu Oct 19 14:21:10 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.58999999999999997,"15min":0.57599999999999996},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.35999999999999999},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.59299999999999997,"15min":0.57299999999999995},"min":{"1min":0.42999999999999999,"5min":0.35599999999999998,"15min":0.33500000000000002},"max":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.63400000000000001}]}]},{"osd":6,"up_from":13,"seq":55834576021,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60804,"kb_used_data":3248,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311036,"statfs":{"total":96636764160,"available":96574500864,"internally_reserved":0,"allocated":3325952,"data_stored":1717964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58915271},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:22:04 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.57399999999999995,"15min":0.60299999999999998},"min":{"1min":0.32500000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.70799999999999996,"5min":0.81200000000000006,"15min":0.85799999999999998},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.55700000000000005,"15min":0.60299999999999998},"min":{"1min":0.371,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.80600000000000005,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.67100000000000004}]},{"osd":1,"last update":"Thu Oct 19 14:22:04 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.59099999999999997,"15min":0.61499999999999999},"min":{"1min":0.41899999999999998,"5min":0.373,"15min":0.35799999999999998},"max":{"1min":0.81100000000000005,"5min":0.82799999999999996,"15min":1.165},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.59499999999999997,"15min":0.622},"min":{"1min":0.435,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.83099999999999996,"5min":0.83699999999999997,"15min":0.91300000000000003},"last":0.55100000000000005}]},{"osd":2,"last update":"Thu Oct 19 14:22:04 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.55900000000000005,"15min":0.59699999999999998},"min":{"1min":0.38300000000000001,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.72399999999999998,"5min":0.80100000000000005,"15min":0.873},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59699999999999998,"15min":0.621},"min":{"1min":0.39400000000000002,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.93100000000000005},"last":0.68799999999999994}]},{"osd":3,"last update":"Thu Oct 19 14:22:04 2023","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.622,"15min":0.64800000000000002},"min":{"1min":0.377,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.83799999999999997,"5min":0.875,"15min":0.875},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.63,"15min":0.65300000000000002},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.78100000000000003,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.57799999999999996}]},{"osd":4,"last update":"Thu Oct 19 14:22:04 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.503,"15min":0.48799999999999999},"min":{"1min":0.31,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.89300000000000002,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51600000000000001,"15min":0.50900000000000001},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.78700000000000003,"5min":0.84899999999999998,"15min":0.85699999999999998},"last":0.42799999999999999}]},{"osd":5,"last update":"Thu Oct 19 14:22:04 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.52000000000000002,"15min":0.505},"min":{"1min":0.33200000000000002,"5min":0.32700000000000001,"15min":0.32500000000000001},"max":{"1min":0.86799999999999999,"5min":1.115,"15min":1.115},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.57799999999999996,"15min":0.54800000000000004},"min":{"1min":0.33900000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.56399999999999995}]},{"osd":7,"last update":"Thu Oct 19 14:22:04 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.57299999999999995,"15min":0.55100000000000005},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.82599999999999996,"5min":0.89300000000000002,"15min":0.90100000000000002},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.58199999999999996,"15min":0.55700000000000005},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.82899999999999996,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.50900000000000001}]}]},{"osd":5,"up_from":13,"seq":55834576021,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66376,"kb_used_data":3256,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305464,"statfs":{"total":96636764160,"available":96568795136,"internally_reserved":0,"allocated":3334144,"data_stored":1718020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:21:50 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.57399999999999995,"15min":0.60699999999999998},"min":{"1min":0.38,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.69399999999999995,"5min":0.79700000000000004,"15min":0.86399999999999999},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.59599999999999997,"15min":0.63300000000000001},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.70499999999999996,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.56399999999999995}]},{"osd":1,"last update":"Thu Oct 19 14:21:50 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.59099999999999997,"15min":0.622},"min":{"1min":0.34599999999999997,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.877},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.61199999999999999,"15min":0.63700000000000001},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.877,"5min":0.877,"15min":0.88500000000000001},"last":0.503}]},{"osd":2,"last update":"Thu Oct 19 14:21:50 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.58199999999999996,"15min":0.61399999999999999},"min":{"1min":0.378,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.374},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.625,"15min":0.65100000000000002},"min":{"1min":0.441,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.78900000000000003,"5min":0.83299999999999996,"15min":0.90800000000000003},"last":0.60899999999999999}]},{"osd":3,"last update":"Thu Oct 19 14:21:50 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.60499999999999998,"15min":0.63600000000000001},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.81100000000000005,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.626,"15min":0.65900000000000003},"min":{"1min":0.40000000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.90900000000000003},"last":0.57899999999999996}]},{"osd":4,"last update":"Thu Oct 19 14:21:50 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.48799999999999999,"15min":0.48699999999999999},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.51800000000000002,"15min":0.505},"min":{"1min":0.309,"5min":0.308,"15min":0.28599999999999998},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.82099999999999995},"last":0.48799999999999999}]},{"osd":6,"last update":"Thu Oct 19 14:21:50 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.51700000000000002,"15min":0.502},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.88800000000000001},"last":0.438},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.54100000000000004,"15min":0.53900000000000003},"min":{"1min":0.36899999999999999,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.52400000000000002}]},{"osd":7,"last update":"Thu Oct 19 14:21:50 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.52200000000000002,"15min":0.51800000000000002},"min":{"1min":0.32700000000000001,"5min":0.312,"15min":0.312},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.57899999999999996,"15min":0.57199999999999995},"min":{"1min":0.38,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.42499999999999999}]}]},{"osd":4,"up_from":13,"seq":55834576021,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48340,"kb_used_data":3264,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323500,"statfs":{"total":96636764160,"available":96587264000,"internally_reserved":0,"allocated":3342336,"data_stored":1719666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:21:16 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.54700000000000004,"15min":0.57099999999999995},"min":{"1min":0.30299999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.879},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.56899999999999995,"15min":0.59299999999999997},"min":{"1min":0.35299999999999998,"5min":0.31,"15min":0.31},"max":{"1min":0.72699999999999998,"5min":0.82599999999999996,"15min":0.873},"last":0.38100000000000001}]},{"osd":1,"last update":"Thu Oct 19 14:21:16 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.59699999999999998,"15min":0.63500000000000001},"min":{"1min":0.40300000000000002,"5min":0.372,"15min":0.32600000000000001},"max":{"1min":0.74099999999999999,"5min":0.80400000000000005,"15min":0.92700000000000005},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.60899999999999999,"15min":0.63900000000000001},"min":{"1min":0.40500000000000003,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.755,"5min":0.90000000000000002,"15min":1.0940000000000001},"last":0.54000000000000004}]},{"osd":2,"last update":"Thu Oct 19 14:21:16 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.626,"15min":0.64700000000000002},"min":{"1min":0.38900000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.80300000000000005,"5min":1.008,"15min":1.008},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.64200000000000002,"15min":0.66600000000000004},"min":{"1min":0.47199999999999998,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.81799999999999995,"5min":0.98799999999999999,"15min":1.008},"last":0.624}]},{"osd":3,"last update":"Thu Oct 19 14:21:16 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.59499999999999997,"15min":0.627},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.68600000000000005,"5min":0.83399999999999996,"15min":0.91900000000000004},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.65600000000000003,"15min":0.66600000000000004},"min":{"1min":0.48199999999999998,"5min":0.45400000000000001,"15min":0.437},"max":{"1min":0.85999999999999999,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.57299999999999995}]},{"osd":5,"last update":"Thu Oct 19 14:21:16 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.497,"15min":0.47799999999999998},"min":{"1min":0.33800000000000002,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.79700000000000004,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.50800000000000001,"15min":0.498},"min":{"1min":0.36499999999999999,"5min":0.33700000000000002,"15min":0.29199999999999998},"max":{"1min":0.81399999999999995,"5min":0.873,"15min":0.873},"last":0.50600000000000001}]},{"osd":6,"last update":"Thu Oct 19 14:21:16 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.50700000000000001,"15min":0.49399999999999999},"min":{"1min":0.38500000000000001,"5min":0.34799999999999998,"15min":0.28199999999999997},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.54900000000000004,"15min":0.54300000000000004},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.88,"5min":0.88,"15min":1.2969999999999999},"last":0.42699999999999999}]},{"osd":7,"last update":"Thu Oct 19 14:21:16 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55100000000000005,"15min":0.54500000000000004},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.32600000000000001},"max":{"1min":0.78200000000000003,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.61199999999999999,"15min":0.59899999999999998},"min":{"1min":0.371,"5min":0.32100000000000001,"15min":0.318},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":1.032},"last":0.60899999999999999}]}]},{"osd":3,"up_from":13,"seq":55834576020,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56644,"kb_used_data":3256,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315196,"statfs":{"total":96636764160,"available":96578760704,"internally_reserved":0,"allocated":3334144,"data_stored":1718056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54655432},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:21:53 2023","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.39900000000000002,"15min":0.41699999999999998},"min":{"1min":0.23899999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.54500000000000004,"5min":0.60599999999999998,"15min":0.70699999999999996},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.39400000000000002,"15min":0.42199999999999999},"min":{"1min":0.23899999999999999,"5min":0.217,"15min":0.217},"max":{"1min":0.51900000000000002,"5min":0.61899999999999999,"15min":0.752},"last":0.38}]},{"osd":1,"last update":"Thu Oct 19 14:21:53 2023","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.40699999999999997,"15min":0.42799999999999999},"min":{"1min":0.28100000000000003,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.71599999999999997,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.45500000000000002,"15min":0.47199999999999998},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26400000000000001},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.30099999999999999}]},{"osd":2,"last update":"Thu Oct 19 14:21:53 2023","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.46200000000000002,"15min":0.48099999999999998},"min":{"1min":0.34300000000000003,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.73099999999999998},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48499999999999999,"15min":0.504},"min":{"1min":0.30599999999999999,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":1.212,"5min":1.212,"15min":1.212},"last":0.35099999999999998}]},{"osd":4,"last update":"Thu Oct 19 14:21:53 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.67400000000000004,"15min":0.70099999999999996},"min":{"1min":0.44600000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.94199999999999995,"5min":1.018,"15min":1.157},"last":0.318},{"interface":"front","average":{"1min":0.629,"5min":0.68700000000000006,"15min":0.72299999999999998},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.89500000000000002,"5min":0.996,"15min":1.173},"last":0.41499999999999998}]},{"osd":5,"last update":"Thu Oct 19 14:21:53 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.67500000000000004,"15min":0.68600000000000005},"min":{"1min":0.50900000000000001,"5min":0.46600000000000003,"15min":0.41999999999999998},"max":{"1min":0.96799999999999997,"5min":0.97299999999999998,"15min":1.0149999999999999},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.66800000000000004,"15min":0.68600000000000005},"min":{"1min":0.48899999999999999,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.97699999999999998,"5min":0.99399999999999999,"15min":1.105},"last":0.49299999999999999}]},{"osd":6,"last update":"Thu Oct 19 14:21:53 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.65000000000000002,"15min":0.68600000000000005},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.93000000000000005,"5min":1.0009999999999999,"15min":1.0669999999999999},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.68899999999999995,"15min":0.70999999999999996},"min":{"1min":0.505,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.079},"last":0.434}]},{"osd":7,"last update":"Thu Oct 19 14:21:53 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67500000000000004,"15min":0.69799999999999995},"min":{"1min":0.46600000000000003,"5min":0.434,"15min":0.434},"max":{"1min":0.94999999999999996,"5min":0.96899999999999997,"15min":1.085},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.69799999999999995,"15min":0.72799999999999998},"min":{"1min":0.48799999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.95899999999999996,"5min":0.97199999999999998,"15min":1.1739999999999999},"last":0.51600000000000001}]}]},{"osd":2,"up_from":13,"seq":55834576022,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48316,"kb_used_data":3248,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323524,"statfs":{"total":96636764160,"available":96587288576,"internally_reserved":0,"allocated":3325952,"data_stored":1717964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.374,"15min":0.39600000000000002},"min":{"1min":0.25700000000000001,"5min":0.218,"15min":0.218},"max":{"1min":0.58799999999999997,"5min":0.66600000000000004,"15min":0.69699999999999995},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.41499999999999998,"15min":0.432},"min":{"1min":0.248,"5min":0.248,"15min":0.223},"max":{"1min":0.53100000000000003,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.29399999999999998}]},{"osd":1,"last update":"Thu Oct 19 14:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.38300000000000001,"15min":0.40600000000000003},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.65400000000000003},"last":0.373},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.44400000000000001,"15min":0.46000000000000002},"min":{"1min":0.28699999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.45100000000000001,"5min":1.742,"15min":1.742},"last":0.40200000000000002}]},{"osd":3,"last update":"Thu Oct 19 14:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.45500000000000002,"15min":0.47599999999999998},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.63300000000000001,"5min":0.69299999999999995,"15min":0.74399999999999999},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.47299999999999998,"15min":0.501},"min":{"1min":0.32700000000000001,"5min":0.32000000000000001,"15min":0.315},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.83599999999999997},"last":0.41499999999999998}]},{"osd":4,"last update":"Thu Oct 19 14:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.63900000000000001,"15min":0.66200000000000003},"min":{"1min":0.40699999999999997,"5min":0.39600000000000002,"15min":0.39500000000000002},"max":{"1min":0.74399999999999999,"5min":0.89200000000000002,"15min":0.998},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.65400000000000003,"15min":0.67600000000000005},"min":{"1min":0.41899999999999998,"5min":0.38900000000000001,"15min":0.35199999999999998},"max":{"1min":0.79000000000000004,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.67500000000000004}]},{"osd":5,"last update":"Thu Oct 19 14:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.67800000000000005,"15min":0.69299999999999995},"min":{"1min":0.443,"5min":0.443,"15min":0.40600000000000003},"max":{"1min":0.79300000000000004,"5min":1.4330000000000001,"15min":1.4330000000000001},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.66900000000000004,"15min":0.68000000000000005},"min":{"1min":0.46899999999999997,"5min":0.45100000000000001,"15min":0.39300000000000002},"max":{"1min":0.77400000000000002,"5min":1.595,"15min":1.595},"last":0.70699999999999996}]},{"osd":6,"last update":"Thu Oct 19 14:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.67700000000000005,"15min":0.70199999999999996},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.80900000000000005,"5min":0.996,"15min":1.0189999999999999},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.629,"5min":0.69499999999999995,"15min":0.71399999999999997},"min":{"1min":0.438,"5min":0.433,"15min":0.40600000000000003},"max":{"1min":0.83399999999999996,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":0.56899999999999995}]},{"osd":7,"last update":"Thu Oct 19 14:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.68400000000000005,"15min":0.71299999999999997},"min":{"1min":0.47199999999999998,"5min":0.44800000000000001,"15min":0.41799999999999998},"max":{"1min":0.78800000000000003,"5min":1.0129999999999999,"15min":1.0289999999999999},"last":0.754},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.71199999999999997,"15min":0.72599999999999998},"min":{"1min":0.42199999999999999,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.86699999999999999,"5min":1.53,"15min":1.53},"last":0.59299999999999997}]}]},{"osd":1,"up_from":13,"seq":55834576021,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48320,"kb_used_data":3252,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323520,"statfs":{"total":96636764160,"available":96587284480,"internally_reserved":0,"allocated":3330048,"data_stored":1718054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.38400000000000001,"15min":0.40300000000000002},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.46899999999999997,"5min":0.61599999999999999,"15min":0.67400000000000004},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.378,"5min":0.41999999999999998,"15min":0.45000000000000001},"min":{"1min":0.27700000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.51500000000000001,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.309}]},{"osd":2,"last update":"Thu Oct 19 14:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.42399999999999999,"15min":0.438},"min":{"1min":0.26100000000000001,"5min":0.251,"15min":0.251},"max":{"1min":0.505,"5min":0.63100000000000001,"15min":0.77500000000000002},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.434,"5min":0.47699999999999998,"15min":0.48799999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.56699999999999995,"5min":0.68700000000000006,"15min":0.82799999999999996},"last":0.34499999999999997}]},{"osd":3,"last update":"Thu Oct 19 14:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.495,"15min":0.501},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.29499999999999998},"max":{"1min":0.59399999999999997,"5min":0.76000000000000001,"15min":0.78900000000000003},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.42899999999999999,"15min":0.44600000000000001},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.26100000000000001},"max":{"1min":0.59899999999999998,"5min":0.65200000000000002,"15min":0.69499999999999995},"last":0.32600000000000001}]},{"osd":4,"last update":"Thu Oct 19 14:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.64200000000000002,"15min":0.66700000000000004},"min":{"1min":0.42299999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.80400000000000005,"5min":0.95999999999999996,"15min":1.042},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.65400000000000003,"15min":0.66600000000000004},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.32300000000000001},"max":{"1min":0.80500000000000005,"5min":0.90800000000000003,"15min":1.1659999999999999},"last":0.42799999999999999}]},{"osd":5,"last update":"Thu Oct 19 14:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.66700000000000004,"15min":0.68200000000000005},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.81200000000000006,"5min":0.96199999999999997,"15min":1.3500000000000001},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.62,"5min":0.68899999999999995,"15min":0.70299999999999996},"min":{"1min":0.442,"5min":0.44,"15min":0.40500000000000003},"max":{"1min":0.82899999999999996,"5min":1.0009999999999999,"15min":1.121},"last":0.44600000000000001}]},{"osd":6,"last update":"Thu Oct 19 14:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.65400000000000003,"15min":0.67600000000000005},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.84999999999999998,"5min":1.0129999999999999,"15min":1.2310000000000001},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.69699999999999995,"15min":0.71499999999999997},"min":{"1min":0.46700000000000003,"5min":0.434,"15min":0.434},"max":{"1min":0.85799999999999998,"5min":1.0009999999999999,"15min":1.274},"last":0.5}]},{"osd":7,"last update":"Thu Oct 19 14:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.70499999999999996,"15min":0.72199999999999998},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.86899999999999999,"5min":0.99099999999999999,"15min":1.075},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.68200000000000005,"15min":0.69599999999999995},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.84599999999999997,"5min":1.0169999999999999,"15min":1.117},"last":0.51200000000000001}]}]},{"osd":0,"up_from":13,"seq":55834576022,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61388,"kb_used_data":3832,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310452,"statfs":{"total":96636764160,"available":96573902848,"internally_reserved":0,"allocated":3923968,"data_stored":2308329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Oct 19 14:21:13 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.40100000000000002,"15min":0.41099999999999998},"min":{"1min":0.26300000000000001,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.47799999999999998,"5min":0.61499999999999999,"15min":0.624},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.41699999999999998,"15min":0.45500000000000002},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.49199999999999999,"5min":0.70899999999999996,"15min":0.81100000000000005},"last":0.443}]},{"osd":2,"last update":"Thu Oct 19 14:21:13 2023","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.34100000000000003,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.54400000000000004,"5min":0.754,"15min":0.754},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.377,"5min":0.42399999999999999,"15min":0.44400000000000001},"min":{"1min":0.28100000000000003,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.46899999999999997,"5min":0.60199999999999998,"15min":0.69099999999999995},"last":0.46700000000000003}]},{"osd":3,"last update":"Thu Oct 19 14:21:13 2023","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.436},"min":{"1min":0.27200000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.50800000000000001,"5min":0.58799999999999997,"15min":0.66800000000000004},"last":0.501},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.48199999999999998,"15min":0.50700000000000001},"min":{"1min":0.29299999999999998,"5min":0.254,"15min":0.254},"max":{"1min":0.72399999999999998,"5min":0.78100000000000003,"15min":0.84599999999999997},"last":0.48999999999999999}]},{"osd":4,"last update":"Thu Oct 19 14:21:13 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.63,"15min":0.67000000000000004},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.80300000000000005,"5min":0.93100000000000005,"15min":1.1890000000000001},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.628,"15min":0.67400000000000004},"min":{"1min":0.45900000000000002,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.75600000000000001,"5min":0.93400000000000005,"15min":1.2150000000000001},"last":0.79600000000000004}]},{"osd":5,"last update":"Thu Oct 19 14:21:13 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.67200000000000004,"15min":0.71399999999999997},"min":{"1min":0.48599999999999999,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.83099999999999996,"5min":0.98499999999999999,"15min":1.1579999999999999},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.67500000000000004,"15min":0.71399999999999997},"min":{"1min":0.499,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.80000000000000004,"5min":0.98299999999999998,"15min":1.2310000000000001},"last":0.84199999999999997}]},{"osd":6,"last update":"Thu Oct 19 14:21:13 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.66300000000000003,"15min":0.69899999999999995},"min":{"1min":0.46100000000000002,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.85499999999999998,"5min":0.97799999999999998,"15min":1.1739999999999999},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.65300000000000002,"15min":0.69599999999999995},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.70299999999999996,"5min":0.97799999999999998,"15min":1.0189999999999999},"last":0.751}]},{"osd":7,"last update":"Thu Oct 19 14:21:13 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.66600000000000004,"15min":0.70999999999999996},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.79900000000000004,"5min":0.93400000000000005,"15min":1.1180000000000001},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.66100000000000003,"15min":0.69599999999999995},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.84499999999999997,"5min":0.93799999999999994,"15min":1.0349999999999999},"last":0.85199999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-19T14:22:10.176 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-19T14:22:10.177 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-19T14:22:10.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.324+0000 7fbaf10d7640 1 -- 172.21.15.57:0/3108618106 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbaec13d350 msgr2=0x7fbaec0cbd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:10.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.324+0000 7fbaf10d7640 1 --2- 172.21.15.57:0/3108618106 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbaec13d350 0x7fbaec0cbd60 secure :-1 s=READY pgs=2898 cs=0 l=1 rev1=1 crypto rx=0x7fbad4009830 tx=0x7fbad402bd60 comp rx=0 tx=0).stop 2023-10-19T14:22:10.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.324+0000 7fbaf10d7640 1 -- 172.21.15.57:0/3108618106 shutdown_connections 2023-10-19T14:22:10.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.324+0000 7fbaf10d7640 1 --2- 172.21.15.57:0/3108618106 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbaec13d350 0x7fbaec0cbd60 unknown :-1 s=CLOSED pgs=2898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:10.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.324+0000 7fbaf10d7640 1 -- 172.21.15.57:0/3108618106 >> 172.21.15.57:0/3108618106 conn(0x7fbaec069930 msgr2=0x7fbaec069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:10.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.324+0000 7fbaf10d7640 1 -- 172.21.15.57:0/3108618106 shutdown_connections 2023-10-19T14:22:10.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.324+0000 7fbaf10d7640 1 -- 172.21.15.57:0/3108618106 wait complete. 2023-10-19T14:22:10.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.325+0000 7fbaf10d7640 1 Processor -- start 2023-10-19T14:22:10.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.325+0000 7fbaf10d7640 1 -- start start 2023-10-19T14:22:10.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.326+0000 7fbaf10d7640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbaec070350 0x7fbaec1361d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:10.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.326+0000 7fbaf10d7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbaec0cb980 0x7fbaec06c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:10.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.326+0000 7fbaf10d7640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbaec06caf0 0x7fbaec0aa8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:10.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.326+0000 7fbaf10d7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaec0ac230 con 0x7fbaec0cb980 2023-10-19T14:22:10.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.326+0000 7fbaf10d7640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaec06cfa0 con 0x7fbaec070350 2023-10-19T14:22:10.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.326+0000 7fbaf10d7640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbaec0ab5b0 con 0x7fbaec06caf0 2023-10-19T14:22:10.332 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.326+0000 7fbae3fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbaec0cb980 0x7fbaec06c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:10.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.326+0000 7fbae3fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbaec0cb980 0x7fbaec06c5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50030/0 (socket says 172.21.15.57:50030) 2023-10-19T14:22:10.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.326+0000 7fbae3fff640 1 -- 172.21.15.57:0/593149377 learned_addr learned my addr 172.21.15.57:0/593149377 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:22:10.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.326+0000 7fbae3fff640 1 -- 172.21.15.57:0/593149377 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbaec06caf0 msgr2=0x7fbaec0aa8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:10.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.326+0000 7fbaeb59e640 1 --2- 172.21.15.57:0/593149377 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbaec06caf0 0x7fbaec0aa8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:10.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.326+0000 7fbaead9d640 1 --2- 172.21.15.57:0/593149377 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbaec070350 0x7fbaec1361d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:10.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.326+0000 7fbae3fff640 1 --2- 172.21.15.57:0/593149377 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbaec06caf0 0x7fbaec0aa8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:10.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.326+0000 7fbae3fff640 1 -- 172.21.15.57:0/593149377 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbaec070350 msgr2=0x7fbaec1361d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:10.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.326+0000 7fbae3fff640 1 --2- 172.21.15.57:0/593149377 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbaec070350 0x7fbaec1361d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:10.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.326+0000 7fbae3fff640 1 -- 172.21.15.57:0/593149377 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbad4009490 con 0x7fbaec0cb980 2023-10-19T14:22:10.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.326+0000 7fbae3fff640 1 --2- 172.21.15.57:0/593149377 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbaec0cb980 0x7fbaec06c5b0 secure :-1 s=READY pgs=2899 cs=0 l=1 rev1=1 crypto rx=0x7fbad4009780 tx=0x7fbad40038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:10.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.326+0000 7fbae37fe640 1 -- 172.21.15.57:0/593149377 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbad4003ea0 con 0x7fbaec0cb980 2023-10-19T14:22:10.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.327+0000 7fbae37fe640 1 -- 172.21.15.57:0/593149377 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fbad4034550 con 0x7fbaec0cb980 2023-10-19T14:22:10.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.327+0000 7fbaf10d7640 1 -- 172.21.15.57:0/593149377 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbaec0ab890 con 0x7fbaec0cb980 2023-10-19T14:22:10.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.327+0000 7fbae37fe640 1 -- 172.21.15.57:0/593149377 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbad4034960 con 0x7fbaec0cb980 2023-10-19T14:22:10.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.327+0000 7fbaf10d7640 1 -- 172.21.15.57:0/593149377 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbaec0abe50 con 0x7fbaec0cb980 2023-10-19T14:22:10.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.327+0000 7fbaf10d7640 1 -- 172.21.15.57:0/593149377 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbaec0690e0 con 0x7fbaec0cb980 2023-10-19T14:22:10.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.328+0000 7fbae37fe640 1 -- 172.21.15.57:0/593149377 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 145) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fbad403b040 con 0x7fbaec0cb980 2023-10-19T14:22:10.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.328+0000 7fbae37fe640 1 --2- 172.21.15.57:0/593149377 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fbacc06b080 0x7fbacc06d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:10.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.329+0000 7fbae37fe640 1 -- 172.21.15.57:0/593149377 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(263..263 src has 1..263) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fbad40ac2b0 con 0x7fbaec0cb980 2023-10-19T14:22:10.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.329+0000 7fbaead9d640 1 --2- 172.21.15.57:0/593149377 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fbacc06b080 0x7fbacc06d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:10.337 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.329+0000 7fbaead9d640 1 --2- 172.21.15.57:0/593149377 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fbacc06b080 0x7fbacc06d540 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fbadc0046b0 tx=0x7fbadc009040 comp rx=0 tx=0).ready entity=mgr.11446 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:10.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.333+0000 7fbae37fe640 1 -- 172.21.15.57:0/593149377 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fbad4076e10 con 0x7fbaec0cb980 2023-10-19T14:22:10.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.580+0000 7fbaf10d7640 1 -- 172.21.15.57:0/593149377 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fbaec06ff80 con 0x7fbaec0cb980 2023-10-19T14:22:10.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.586+0000 7fbae37fe640 1 -- 172.21.15.57:0/593149377 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v62) v1 ==== 190+0+147 (secure 0 0 0) 0x7fbad40390e0 con 0x7fbaec0cb980 2023-10-19T14:22:10.589 INFO:teuthology.orchestra.run.smithi057.stdout:[client.0] 2023-10-19T14:22:10.590 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQBkJDFlg9yuExAATxiWDfS+p07G1F0pORbJEg== 2023-10-19T14:22:10.590 INFO:teuthology.orchestra.run.smithi057.stdout: caps mds = "allow" 2023-10-19T14:22:10.590 INFO:teuthology.orchestra.run.smithi057.stdout: caps mon = "allow r" 2023-10-19T14:22:10.590 INFO:teuthology.orchestra.run.smithi057.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-19T14:22:10.591 INFO:teuthology.orchestra.run.smithi057.stderr:updated caps for client.0 2023-10-19T14:22:10.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.588+0000 7fbaf10d7640 1 -- 172.21.15.57:0/593149377 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fbacc06b080 msgr2=0x7fbacc06d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:10.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.588+0000 7fbaf10d7640 1 --2- 172.21.15.57:0/593149377 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fbacc06b080 0x7fbacc06d540 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fbadc0046b0 tx=0x7fbadc009040 comp rx=0 tx=0).stop 2023-10-19T14:22:10.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.588+0000 7fbaf10d7640 1 -- 172.21.15.57:0/593149377 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbaec0cb980 msgr2=0x7fbaec06c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:10.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.588+0000 7fbaf10d7640 1 --2- 172.21.15.57:0/593149377 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbaec0cb980 0x7fbaec06c5b0 secure :-1 s=READY pgs=2899 cs=0 l=1 rev1=1 crypto rx=0x7fbad4009780 tx=0x7fbad40038a0 comp rx=0 tx=0).stop 2023-10-19T14:22:10.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.588+0000 7fbaf10d7640 1 -- 172.21.15.57:0/593149377 shutdown_connections 2023-10-19T14:22:10.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.588+0000 7fbaf10d7640 1 --2- 172.21.15.57:0/593149377 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fbacc06b080 0x7fbacc06d540 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:10.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.588+0000 7fbaf10d7640 1 --2- 172.21.15.57:0/593149377 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fbaec06caf0 0x7fbaec0aa8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:10.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.588+0000 7fbaf10d7640 1 --2- 172.21.15.57:0/593149377 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbaec0cb980 0x7fbaec06c5b0 unknown :-1 s=CLOSED pgs=2899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:10.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.588+0000 7fbaf10d7640 1 --2- 172.21.15.57:0/593149377 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbaec070350 0x7fbaec1361d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:10.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.588+0000 7fbaf10d7640 1 -- 172.21.15.57:0/593149377 >> 172.21.15.57:0/593149377 conn(0x7fbaec069930 msgr2=0x7fbaec159a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:10.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.588+0000 7fbaf10d7640 1 -- 172.21.15.57:0/593149377 shutdown_connections 2023-10-19T14:22:10.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.588+0000 7fbaf10d7640 1 -- 172.21.15.57:0/593149377 wait complete. 2023-10-19T14:22:10.602 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T14:22:10.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.709+0000 7f88969b6640 1 -- 172.21.15.57:0/1523480472 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f88780058d0 msgr2=0x7f8878005d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:10.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.709+0000 7f88969b6640 1 --2- 172.21.15.57:0/1523480472 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f88780058d0 0x7f8878005d50 secure :-1 s=READY pgs=2901 cs=0 l=1 rev1=1 crypto rx=0x7f8884006fd0 tx=0x7f8878006900 comp rx=0 tx=0).stop 2023-10-19T14:22:10.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.709+0000 7f88969b6640 1 -- 172.21.15.57:0/1523480472 shutdown_connections 2023-10-19T14:22:10.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.709+0000 7f88969b6640 1 --2- 172.21.15.57:0/1523480472 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f88780058d0 0x7f8878005d50 unknown :-1 s=CLOSED pgs=2901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:10.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.709+0000 7f88969b6640 1 -- 172.21.15.57:0/1523480472 >> 172.21.15.57:0/1523480472 conn(0x7f8890069930 msgr2=0x7f8890069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:10.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.709+0000 7f88969b6640 1 -- 172.21.15.57:0/1523480472 shutdown_connections 2023-10-19T14:22:10.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.709+0000 7f88969b6640 1 -- 172.21.15.57:0/1523480472 wait complete. 2023-10-19T14:22:10.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.710+0000 7f88969b6640 1 Processor -- start 2023-10-19T14:22:10.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.710+0000 7f88969b6640 1 -- start start 2023-10-19T14:22:10.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.711+0000 7f88969b6640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f88780058d0 0x7f889014ff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:10.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.711+0000 7f88969b6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f889014f260 0x7f889014d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:10.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.711+0000 7f88969b6640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f889014da80 0x7f889013ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:10.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.711+0000 7f88969b6640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f889014df10 con 0x7f889014f260 2023-10-19T14:22:10.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.711+0000 7f88969b6640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f889014e1c0 con 0x7f88780058d0 2023-10-19T14:22:10.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.711+0000 7f88969b6640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f889014e4a0 con 0x7f889014da80 2023-10-19T14:22:10.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.711+0000 7f888f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f889014f260 0x7f889014d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:10.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.711+0000 7f888f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f889014f260 0x7f889014d540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50052/0 (socket says 172.21.15.57:50052) 2023-10-19T14:22:10.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.711+0000 7f888f7fe640 1 -- 172.21.15.57:0/3514540249 learned_addr learned my addr 172.21.15.57:0/3514540249 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:22:10.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.711+0000 7f888ffff640 1 --2- 172.21.15.57:0/3514540249 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f88780058d0 0x7f889014ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:10.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.711+0000 7f8894f53640 1 --2- 172.21.15.57:0/3514540249 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f889014da80 0x7f889013ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:10.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.712+0000 7f888f7fe640 1 -- 172.21.15.57:0/3514540249 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f889014da80 msgr2=0x7f889013ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:10.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.712+0000 7f888f7fe640 1 --2- 172.21.15.57:0/3514540249 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f889014da80 0x7f889013ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:10.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.712+0000 7f888f7fe640 1 -- 172.21.15.57:0/3514540249 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f88780058d0 msgr2=0x7f889014ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:10.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.712+0000 7f888f7fe640 1 --2- 172.21.15.57:0/3514540249 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f88780058d0 0x7f889014ff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:10.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.712+0000 7f888f7fe640 1 -- 172.21.15.57:0/3514540249 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f889014e780 con 0x7f889014f260 2023-10-19T14:22:10.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.712+0000 7f8894f53640 1 --2- 172.21.15.57:0/3514540249 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f889014da80 0x7f889013ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:22:10.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.712+0000 7f888f7fe640 1 --2- 172.21.15.57:0/3514540249 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f889014f260 0x7f889014d540 secure :-1 s=READY pgs=2902 cs=0 l=1 rev1=1 crypto rx=0x7f8884001930 tx=0x7f88840014f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:10.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.712+0000 7f888cff9640 1 -- 172.21.15.57:0/3514540249 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8884006460 con 0x7f889014f260 2023-10-19T14:22:10.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.712+0000 7f888cff9640 1 -- 172.21.15.57:0/3514540249 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f88840035f0 con 0x7f889014f260 2023-10-19T14:22:10.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.712+0000 7f88969b6640 1 -- 172.21.15.57:0/3514540249 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f889014ea60 con 0x7f889014f260 2023-10-19T14:22:10.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.712+0000 7f888cff9640 1 -- 172.21.15.57:0/3514540249 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88840065c0 con 0x7f889014f260 2023-10-19T14:22:10.720 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.712+0000 7f88969b6640 1 -- 172.21.15.57:0/3514540249 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88901504a0 con 0x7f889014f260 2023-10-19T14:22:10.721 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.713+0000 7f88969b6640 1 -- 172.21.15.57:0/3514540249 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8890150630 con 0x7f889014f260 2023-10-19T14:22:10.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.713+0000 7f888cff9640 1 -- 172.21.15.57:0/3514540249 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 145) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f8884031030 con 0x7f889014f260 2023-10-19T14:22:10.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.714+0000 7f888cff9640 1 --2- 172.21.15.57:0/3514540249 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f887c06b080 0x7f887c06d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:10.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.714+0000 7f888cff9640 1 -- 172.21.15.57:0/3514540249 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(263..263 src has 1..263) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f88840ab630 con 0x7f889014f260 2023-10-19T14:22:10.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.714+0000 7f888ffff640 1 --2- 172.21.15.57:0/3514540249 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f887c06b080 0x7f887c06d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:10.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.715+0000 7f888ffff640 1 --2- 172.21.15.57:0/3514540249 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f887c06b080 0x7f887c06d540 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f88901508e0 tx=0x7f8880005f90 comp rx=0 tx=0).ready entity=mgr.11446 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:10.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.718+0000 7f888cff9640 1 -- 172.21.15.57:0/3514540249 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f88840761e0 con 0x7f889014f260 2023-10-19T14:22:10.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.928+0000 7f88969b6640 1 -- 172.21.15.57:0/3514540249 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f88900690e0 con 0x7f889014f260 2023-10-19T14:22:10.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.928+0000 7f888cff9640 1 -- 172.21.15.57:0/3514540249 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+4935 (secure 0 0 0) 0x7f8884079800 con 0x7f889014f260 2023-10-19T14:22:10.932 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:22:10.932 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":116,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10280,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/1738328496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":1738328496},{"type":"v1","addr":"172.21.15.179:6837","nonce":1738328496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":113},{"gid":13344,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/3191801579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":3191801579},{"type":"v1","addr":"172.21.15.57:6834","nonce":3191801579}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":113},{"gid":13347,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/3876822488","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":3876822488},{"type":"v1","addr":"172.21.15.57:6835","nonce":3876822488}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T14:22:01.763128+0000","modified":"2023-10-19T14:22:03.758375+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":13359},"failed":[],"damaged":[],"stopped":[],"info":{"gid_13359":{"gid":13359,"name":"d","rank":0,"incarnation":114,"state":"up:active","state_seq":3,"addr":"172.21.15.179:6836/4133411706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":4133411706},{"type":"v1","addr":"172.21.15.179:6836","nonce":4133411706}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[36],"metadata_pool":35,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":17}]} 2023-10-19T14:22:10.932 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 116 2023-10-19T14:22:10.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.930+0000 7f88969b6640 1 -- 172.21.15.57:0/3514540249 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f887c06b080 msgr2=0x7f887c06d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:10.934 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.930+0000 7f88969b6640 1 --2- 172.21.15.57:0/3514540249 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f887c06b080 0x7f887c06d540 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f88901508e0 tx=0x7f8880005f90 comp rx=0 tx=0).stop 2023-10-19T14:22:10.934 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.930+0000 7f88969b6640 1 -- 172.21.15.57:0/3514540249 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f889014f260 msgr2=0x7f889014d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:10.934 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.930+0000 7f88969b6640 1 --2- 172.21.15.57:0/3514540249 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f889014f260 0x7f889014d540 secure :-1 s=READY pgs=2902 cs=0 l=1 rev1=1 crypto rx=0x7f8884001930 tx=0x7f88840014f0 comp rx=0 tx=0).stop 2023-10-19T14:22:10.934 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.930+0000 7f88969b6640 1 -- 172.21.15.57:0/3514540249 shutdown_connections 2023-10-19T14:22:10.934 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.930+0000 7f88969b6640 1 --2- 172.21.15.57:0/3514540249 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f887c06b080 0x7f887c06d540 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:10.935 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.930+0000 7f88969b6640 1 --2- 172.21.15.57:0/3514540249 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f889014da80 0x7f889013ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:10.935 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.930+0000 7f88969b6640 1 --2- 172.21.15.57:0/3514540249 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f889014f260 0x7f889014d540 unknown :-1 s=CLOSED pgs=2902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:10.935 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.930+0000 7f88969b6640 1 --2- 172.21.15.57:0/3514540249 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f88780058d0 0x7f889014ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:10.935 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.930+0000 7f88969b6640 1 -- 172.21.15.57:0/3514540249 >> 172.21.15.57:0/3514540249 conn(0x7f8890069930 msgr2=0x7f88900af1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:10.935 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.930+0000 7f88969b6640 1 -- 172.21.15.57:0/3514540249 shutdown_connections 2023-10-19T14:22:10.936 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:10.930+0000 7f88969b6640 1 -- 172.21.15.57:0/3514540249 wait complete. 2023-10-19T14:22:11.346 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 116, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-19T14:22:01.763128+0000', 'modified': '2023-10-19T14:22:03.758375+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 13359}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_13359': {'gid': 13359, 'name': 'd', 'rank': 0, 'incarnation': 114, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.179:6836/4133411706', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6834', 'nonce': 4133411706}, {'type': 'v1', 'addr': '172.21.15.179:6836', 'nonce': 4133411706}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [36], 'metadata_pool': 35, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-19T14:22:11.347 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-19T14:22:11.347 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2023-10-19T14:22:11.462 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.459+0000 7fa267133640 1 -- 172.21.15.57:0/3997204074 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa2500039a0 msgr2=0x7fa250003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:11.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.459+0000 7fa267133640 1 --2- 172.21.15.57:0/3997204074 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa2500039a0 0x7fa250003dc0 secure :-1 s=READY pgs=2904 cs=0 l=1 rev1=1 crypto rx=0x7fa254001680 tx=0x7fa2500057f0 comp rx=0 tx=0).stop 2023-10-19T14:22:11.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.459+0000 7fa2656d0640 1 -- 172.21.15.57:0/3997204074 reap_dead start 2023-10-19T14:22:11.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.459+0000 7fa267133640 1 -- 172.21.15.57:0/3997204074 shutdown_connections 2023-10-19T14:22:11.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.459+0000 7fa267133640 1 -- 172.21.15.57:0/3997204074 >> 172.21.15.57:0/3997204074 conn(0x7fa260069930 msgr2=0x7fa260069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:11.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.459+0000 7fa267133640 1 -- 172.21.15.57:0/3997204074 shutdown_connections 2023-10-19T14:22:11.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.459+0000 7fa267133640 1 -- 172.21.15.57:0/3997204074 wait complete. 2023-10-19T14:22:11.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.460+0000 7fa267133640 1 Processor -- start 2023-10-19T14:22:11.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.461+0000 7fa267133640 1 -- start start 2023-10-19T14:22:11.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.461+0000 7fa267133640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa2500039a0 0x7fa26013a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:11.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.461+0000 7fa267133640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa26015ff00 0x7fa26013ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:11.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.461+0000 7fa267133640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa260160380 0x7fa26013b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:11.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.461+0000 7fa267133640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa26013a700 con 0x7fa26015ff00 2023-10-19T14:22:11.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.461+0000 7fa267133640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa26013bb00 con 0x7fa2500039a0 2023-10-19T14:22:11.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.461+0000 7fa267133640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa26013bde0 con 0x7fa260160380 2023-10-19T14:22:11.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.462+0000 7fa25ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa26015ff00 0x7fa26013ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:11.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.462+0000 7fa25ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa26015ff00 0x7fa26013ada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50068/0 (socket says 172.21.15.57:50068) 2023-10-19T14:22:11.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.462+0000 7fa25ffff640 1 -- 172.21.15.57:0/315262566 learned_addr learned my addr 172.21.15.57:0/315262566 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:22:11.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.462+0000 7fa264ecf640 1 --2- 172.21.15.57:0/315262566 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa2500039a0 0x7fa26013a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:11.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.462+0000 7fa2656d0640 1 --2- 172.21.15.57:0/315262566 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa260160380 0x7fa26013b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:11.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.462+0000 7fa25ffff640 1 -- 172.21.15.57:0/315262566 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa260160380 msgr2=0x7fa26013b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:11.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.462+0000 7fa25ffff640 1 --2- 172.21.15.57:0/315262566 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa260160380 0x7fa26013b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:11.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.462+0000 7fa25ffff640 1 -- 172.21.15.57:0/315262566 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa2500039a0 msgr2=0x7fa26013a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:11.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.462+0000 7fa25ffff640 1 --2- 172.21.15.57:0/315262566 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa2500039a0 0x7fa26013a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:11.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.462+0000 7fa25ffff640 1 -- 172.21.15.57:0/315262566 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa26013c0c0 con 0x7fa26015ff00 2023-10-19T14:22:11.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.462+0000 7fa25ffff640 1 --2- 172.21.15.57:0/315262566 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa26015ff00 0x7fa26013ada0 secure :-1 s=READY pgs=2905 cs=0 l=1 rev1=1 crypto rx=0x7fa24c000fb0 tx=0x7fa24c00ea90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:11.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.462+0000 7fa25d7fa640 1 -- 172.21.15.57:0/315262566 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa24c00c910 con 0x7fa26015ff00 2023-10-19T14:22:11.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.462+0000 7fa25d7fa640 1 -- 172.21.15.57:0/315262566 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fa24c00cb60 con 0x7fa26015ff00 2023-10-19T14:22:11.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.462+0000 7fa267133640 1 -- 172.21.15.57:0/315262566 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa26013c3a0 con 0x7fa26015ff00 2023-10-19T14:22:11.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.462+0000 7fa25d7fa640 1 -- 172.21.15.57:0/315262566 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa24c011460 con 0x7fa26015ff00 2023-10-19T14:22:11.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.462+0000 7fa267133640 1 -- 172.21.15.57:0/315262566 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa26013d680 con 0x7fa26015ff00 2023-10-19T14:22:11.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.464+0000 7fa25d7fa640 1 -- 172.21.15.57:0/315262566 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 145) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fa24c0115c0 con 0x7fa26015ff00 2023-10-19T14:22:11.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.464+0000 7fa25d7fa640 1 --2- 172.21.15.57:0/315262566 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fa24406b080 0x7fa24406d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:11.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.464+0000 7fa25d7fa640 1 -- 172.21.15.57:0/315262566 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(263..263 src has 1..263) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fa24c08a3c0 con 0x7fa26015ff00 2023-10-19T14:22:11.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.464+0000 7fa264ecf640 1 --2- 172.21.15.57:0/315262566 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fa24406b080 0x7fa24406d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:11.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.465+0000 7fa264ecf640 1 --2- 172.21.15.57:0/315262566 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fa24406b080 0x7fa24406d540 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa254034000 tx=0x7fa254001190 comp rx=0 tx=0).ready entity=mgr.11446 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:11.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.465+0000 7fa269d9f740 1 Processor -- start 2023-10-19T14:22:11.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.465+0000 7fa269d9f740 1 -- start start 2023-10-19T14:22:11.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.466+0000 7fa269d9f740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56325beaeb40 0x56325be81ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:11.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.466+0000 7fa269d9f740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56325beaef20 0x56325beb18d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:11.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.466+0000 7fa269d9f740 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x56325be82020 0x56325beb1e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:11.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.466+0000 7fa269d9f740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x56325bd93730 con 0x56325beaeb40 2023-10-19T14:22:11.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.466+0000 7fa269d9f740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x56325bdf4780 con 0x56325beaef20 2023-10-19T14:22:11.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.466+0000 7fa269d9f740 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x56325bdf1e70 con 0x56325be82020 2023-10-19T14:22:11.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.466+0000 7fa264ecf640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56325beaeb40 0x56325be81ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:11.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.466+0000 7fa264ecf640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56325beaeb40 0x56325be81ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50080/0 (socket says 172.21.15.57:50080) 2023-10-19T14:22:11.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.466+0000 7fa264ecf640 1 -- 172.21.15.57:0/3399224509 learned_addr learned my addr 172.21.15.57:0/3399224509 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:22:11.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.466+0000 7fa25ffff640 1 --2- 172.21.15.57:0/3399224509 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56325beaef20 0x56325beb18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:11.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.466+0000 7fa2656d0640 1 --2- 172.21.15.57:0/3399224509 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x56325be82020 0x56325beb1e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:11.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.466+0000 7fa264ecf640 1 -- 172.21.15.57:0/3399224509 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x56325be82020 msgr2=0x56325beb1e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T14:22:11.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.466+0000 7fa264ecf640 1 --2- 172.21.15.57:0/3399224509 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x56325be82020 0x56325beb1e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:11.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.466+0000 7fa264ecf640 1 -- 172.21.15.57:0/3399224509 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56325beaef20 msgr2=0x56325beb18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T14:22:11.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.466+0000 7fa264ecf640 1 --2- 172.21.15.57:0/3399224509 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56325beaef20 0x56325beb18d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:11.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.466+0000 7fa264ecf640 1 -- 172.21.15.57:0/3399224509 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56325bdf23d0 con 0x56325beaeb40 2023-10-19T14:22:11.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.466+0000 7fa264ecf640 1 --2- 172.21.15.57:0/3399224509 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56325beaeb40 0x56325be81ae0 secure :-1 s=READY pgs=2906 cs=0 l=1 rev1=1 crypto rx=0x7fa254001770 tx=0x7fa25403d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e88d22e371a9bdb server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:11.482 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.467+0000 7fa24a7fc640 1 -- 172.21.15.57:0/3399224509 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa25402c750 con 0x56325beaeb40 2023-10-19T14:22:11.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.467+0000 7fa24a7fc640 1 -- 172.21.15.57:0/3399224509 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fa25402c9a0 con 0x56325beaeb40 2023-10-19T14:22:11.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.467+0000 7fa24a7fc640 1 -- 172.21.15.57:0/3399224509 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa25402cd70 con 0x56325beaeb40 2023-10-19T14:22:11.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.467+0000 7fa269d9f740 1 -- 172.21.15.57:0/3399224509 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56325beaeb40 msgr2=0x56325be81ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:11.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.467+0000 7fa269d9f740 1 --2- 172.21.15.57:0/3399224509 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56325beaeb40 0x56325be81ae0 secure :-1 s=READY pgs=2906 cs=0 l=1 rev1=1 crypto rx=0x7fa254001770 tx=0x7fa25403d000 comp rx=0 tx=0).stop 2023-10-19T14:22:11.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.467+0000 7fa269d9f740 1 -- 172.21.15.57:0/3399224509 shutdown_connections 2023-10-19T14:22:11.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.467+0000 7fa269d9f740 1 --2- 172.21.15.57:0/3399224509 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x56325be82020 0x56325beb1e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:11.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.467+0000 7fa269d9f740 1 --2- 172.21.15.57:0/3399224509 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56325beaef20 0x56325beb18d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:11.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.467+0000 7fa269d9f740 1 --2- 172.21.15.57:0/3399224509 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56325beaeb40 0x56325be81ae0 unknown :-1 s=CLOSED pgs=2906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:11.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.467+0000 7fa269d9f740 1 -- 172.21.15.57:0/3399224509 >> 172.21.15.57:0/3399224509 conn(0x56325be83d50 msgr2=0x56325becc8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:11.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.467+0000 7fa269d9f740 1 -- 172.21.15.57:0/3399224509 shutdown_connections 2023-10-19T14:22:11.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.467+0000 7fa269d9f740 1 -- 172.21.15.57:0/3399224509 wait complete. 2023-10-19T14:22:11.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.468+0000 7fa269d9f740 1 -- start start 2023-10-19T14:22:11.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.468+0000 7fa269d9f740 1 Processor -- start 2023-10-19T14:22:11.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.469+0000 7fa269d9f740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56325beaeb40 0x56325bec7300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:11.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.469+0000 7fa269d9f740 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x56325beaef20 0x56325beb4780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:11.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.469+0000 7fa269d9f740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56325bedc280 0x56325bedc700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:11.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.469+0000 7fa269d9f740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x56325bded760 con 0x56325bedc280 2023-10-19T14:22:11.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.469+0000 7fa269d9f740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x56325bda88f0 con 0x56325beaeb40 2023-10-19T14:22:11.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.469+0000 7fa269d9f740 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x56325bda8590 con 0x56325beaef20 2023-10-19T14:22:11.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.469+0000 7fa269d9f740 10 client.0 _ll_register_callbacks cb 0x56325bde4ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-19T14:22:11.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.469+0000 7fa2656d0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56325bedc280 0x56325bedc700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:11.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.469+0000 7fa2656d0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56325bedc280 0x56325bedc700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50090/0 (socket says 172.21.15.57:50090) 2023-10-19T14:22:11.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.469+0000 7fa2656d0640 1 -- 172.21.15.57:0/2773166002 learned_addr learned my addr 172.21.15.57:0/2773166002 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:22:11.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.469+0000 7fa264ecf640 1 --2- 172.21.15.57:0/2773166002 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56325beaeb40 0x56325bec7300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:11.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.469+0000 7fa25ffff640 1 --2- 172.21.15.57:0/2773166002 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x56325beaef20 0x56325beb4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:11.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.469+0000 7fa2656d0640 1 -- 172.21.15.57:0/2773166002 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x56325beaef20 msgr2=0x56325beb4780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T14:22:11.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.469+0000 7fa2656d0640 1 --2- 172.21.15.57:0/2773166002 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x56325beaef20 0x56325beb4780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:11.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.469+0000 7fa2656d0640 1 -- 172.21.15.57:0/2773166002 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56325beaeb40 msgr2=0x56325bec7300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T14:22:11.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.469+0000 7fa2656d0640 1 --2- 172.21.15.57:0/2773166002 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56325beaeb40 0x56325bec7300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:11.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.469+0000 7fa2656d0640 1 -- 172.21.15.57:0/2773166002 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa250003520 con 0x56325bedc280 2023-10-19T14:22:11.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.469+0000 7fa264ecf640 1 --2- 172.21.15.57:0/2773166002 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56325beaeb40 0x56325bec7300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T14:22:11.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.470+0000 7fa2656d0640 1 --2- 172.21.15.57:0/2773166002 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56325bedc280 0x56325bedc700 secure :-1 s=READY pgs=2907 cs=0 l=1 rev1=1 crypto rx=0x7fa2600cbac0 tx=0x7fa2580041a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f373f72e37060acd server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:11.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.470+0000 7fa23f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2023-10-19T14:22:11.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.470+0000 7fa23f7fe640 1 -- 172.21.15.57:0/2773166002 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa258017070 con 0x56325bedc280 2023-10-19T14:22:11.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.470+0000 7fa269d9f740 1 -- 172.21.15.57:0/2773166002 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56325bde6740 con 0x56325bedc280 2023-10-19T14:22:11.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.470+0000 7fa23f7fe640 1 -- 172.21.15.57:0/2773166002 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(263..263 src has 1..263) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fa2580105e0 con 0x56325bedc280 2023-10-19T14:22:11.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.470+0000 7fa23f7fe640 1 -- 172.21.15.57:0/2773166002 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=116) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa258012500 con 0x56325bedc280 2023-10-19T14:22:11.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.470+0000 7fa269d9f740 10 client.13515 fetch_fsmap learned FSMap version 116 2023-10-19T14:22:11.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.470+0000 7fa269d9f740 1 -- 172.21.15.57:0/2773166002 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=116,monmap=2+}) v3 -- 0x56325bdebde0 con 0x56325bedc280 2023-10-19T14:22:11.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.470+0000 7fa23f7fe640 1 -- 172.21.15.57:0/2773166002 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x56325bde6740 con 0x56325bedc280 2023-10-19T14:22:11.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.470+0000 7fa23f7fe640 1 -- 172.21.15.57:0/2773166002 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 116) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fa2580107a0 con 0x56325bedc280 2023-10-19T14:22:11.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.470+0000 7fa269d9f740 10 client.13515 fetch_fsmap finished waiting for FSMap version 116 2023-10-19T14:22:11.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.470+0000 7fa269d9f740 10 client.13515 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-19T14:22:11.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.470+0000 7fa269d9f740 20 client.13515 populate_metadata read hostname 'smithi057' 2023-10-19T14:22:11.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.470+0000 7fa269d9f740 1 --2- 172.21.15.57:0/2773166002 >> [v2:172.21.15.179:6834/4133411706,v1:172.21.15.179:6836/4133411706] conn(0x56325bee0440 0x56325bee0820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:11.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.470+0000 7fa269d9f740 4 client.13515 mds_command: new command op to 13359 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-19T14:22:11.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.470+0000 7fa269d9f740 1 -- 172.21.15.57:0/2773166002 --> [v2:172.21.15.179:6834/4133411706,v1:172.21.15.179:6836/4133411706] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56325bde6460 con 0x56325bee0440 2023-10-19T14:22:11.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.470+0000 7fa264ecf640 1 --2- 172.21.15.57:0/2773166002 >> [v2:172.21.15.179:6834/4133411706,v1:172.21.15.179:6836/4133411706] conn(0x56325bee0440 0x56325bee0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:11.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.471+0000 7fa23f7fe640 10 client.13515 ms_handle_connect on v2:172.21.15.179:6834/4133411706 2023-10-19T14:22:11.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.471+0000 7fa264ecf640 1 --2- 172.21.15.57:0/2773166002 >> [v2:172.21.15.179:6834/4133411706,v1:172.21.15.179:6836/4133411706] conn(0x56325bee0440 0x56325bee0820 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2425828ea4a75111 server_cookie=1429ca57369f9a3 in_seq=0 out_seq=0 2023-10-19T14:22:11.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.472+0000 7fa23f7fe640 1 -- 172.21.15.57:0/2773166002 <== mds.0 v2:172.21.15.179:6834/4133411706 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x56325bde6460 con 0x56325bee0440 2023-10-19T14:22:11.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.472+0000 7fa23f7fe640 10 client.13515 handle_command_reply: tid=0 2023-10-19T14:22:11.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.472+0000 7fa269d9f740 1 client.13515 shutdown 2023-10-19T14:22:11.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.472+0000 7fa269d9f740 1 -- 172.21.15.57:0/2773166002 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56325bedc280 msgr2=0x56325bedc700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:11.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.472+0000 7fa269d9f740 1 --2- 172.21.15.57:0/2773166002 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56325bedc280 0x56325bedc700 secure :-1 s=READY pgs=2907 cs=0 l=1 rev1=1 crypto rx=0x7fa2600cbac0 tx=0x7fa2580041a0 comp rx=0 tx=0).stop 2023-10-19T14:22:11.493 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.472+0000 7fa269d9f740 1 -- 172.21.15.57:0/2773166002 shutdown_connections 2023-10-19T14:22:11.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.472+0000 7fa269d9f740 1 --2- 172.21.15.57:0/2773166002 >> [v2:172.21.15.179:6834/4133411706,v1:172.21.15.179:6836/4133411706] conn(0x56325bee0440 0x56325bee0820 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:11.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.472+0000 7fa269d9f740 1 --2- 172.21.15.57:0/2773166002 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56325bedc280 0x56325bedc700 unknown :-1 s=CLOSED pgs=2907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:11.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.472+0000 7fa23f7fe640 0 client.13515 ms_handle_reset on v2:172.21.15.179:6834/4133411706 2023-10-19T14:22:11.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.472+0000 7fa269d9f740 1 --2- 172.21.15.57:0/2773166002 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x56325beaef20 0x56325beb4780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:11.494 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.472+0000 7fa269d9f740 1 --2- 172.21.15.57:0/2773166002 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56325beaeb40 0x56325bec7300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:11.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.472+0000 7fa269d9f740 1 -- 172.21.15.57:0/2773166002 >> 172.21.15.57:0/2773166002 conn(0x56325be82020 msgr2=0x56325beb18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:11.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.472+0000 7fa269d9f740 1 -- 172.21.15.57:0/2773166002 shutdown_connections 2023-10-19T14:22:11.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.472+0000 7fa269d9f740 1 -- 172.21.15.57:0/2773166002 wait complete. 2023-10-19T14:22:11.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.472+0000 7fa269d9f740 20 client.13515 trim_cache size 0 max 16384 2023-10-19T14:22:11.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.487+0000 7fa269d9f740 1 Processor -- start 2023-10-19T14:22:11.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.487+0000 7fa269d9f740 1 -- start start 2023-10-19T14:22:11.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.488+0000 7fa269d9f740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56325beaeb40 0x56325bed0210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:11.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.488+0000 7fa269d9f740 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x56325beaef20 0x56325be85640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:11.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.488+0000 7fa269d9f740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56325bedc280 0x56325be85b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:11.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.488+0000 7fa269d9f740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x56325bd6e270 con 0x56325beaeb40 2023-10-19T14:22:11.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.488+0000 7fa269d9f740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x56325becd630 con 0x56325bedc280 2023-10-19T14:22:11.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.488+0000 7fa269d9f740 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x56325bef0d60 con 0x56325beaef20 2023-10-19T14:22:11.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.488+0000 7fa264ecf640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56325beaeb40 0x56325bed0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:11.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.488+0000 7fa264ecf640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56325beaeb40 0x56325bed0210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50106/0 (socket says 172.21.15.57:50106) 2023-10-19T14:22:11.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.488+0000 7fa264ecf640 1 -- 172.21.15.57:0/4049013716 learned_addr learned my addr 172.21.15.57:0/4049013716 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:22:11.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.488+0000 7fa264ecf640 1 -- 172.21.15.57:0/4049013716 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x56325beaef20 msgr2=0x56325be85640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T14:22:11.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.488+0000 7fa264ecf640 1 --2- 172.21.15.57:0/4049013716 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x56325beaef20 0x56325be85640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:11.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.488+0000 7fa264ecf640 1 -- 172.21.15.57:0/4049013716 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56325bedc280 msgr2=0x56325be85b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T14:22:11.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.488+0000 7fa264ecf640 1 --2- 172.21.15.57:0/4049013716 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56325bedc280 0x56325be85b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:11.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.488+0000 7fa264ecf640 1 -- 172.21.15.57:0/4049013716 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56325bd8ad00 con 0x56325beaeb40 2023-10-19T14:22:11.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.488+0000 7fa264ecf640 1 --2- 172.21.15.57:0/4049013716 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56325beaeb40 0x56325bed0210 secure :-1 s=READY pgs=2908 cs=0 l=1 rev1=1 crypto rx=0x7fa254001650 tx=0x7fa254006b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7c357309417cf34 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:11.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.489+0000 7fa24affd640 1 -- 172.21.15.57:0/4049013716 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa254000ed0 con 0x56325beaeb40 2023-10-19T14:22:11.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.489+0000 7fa24affd640 1 -- 172.21.15.57:0/4049013716 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fa254041540 con 0x56325beaeb40 2023-10-19T14:22:11.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.489+0000 7fa24affd640 1 -- 172.21.15.57:0/4049013716 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2540418b0 con 0x56325beaeb40 2023-10-19T14:22:11.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.489+0000 7fa269d9f740 1 -- 172.21.15.57:0/4049013716 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56325beaeb40 msgr2=0x56325bed0210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:11.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.489+0000 7fa269d9f740 1 --2- 172.21.15.57:0/4049013716 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56325beaeb40 0x56325bed0210 secure :-1 s=READY pgs=2908 cs=0 l=1 rev1=1 crypto rx=0x7fa254001650 tx=0x7fa254006b30 comp rx=0 tx=0).stop 2023-10-19T14:22:11.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.489+0000 7fa269d9f740 1 -- 172.21.15.57:0/4049013716 shutdown_connections 2023-10-19T14:22:11.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.489+0000 7fa269d9f740 1 --2- 172.21.15.57:0/4049013716 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56325bedc280 0x56325be85b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:11.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.489+0000 7fa269d9f740 1 --2- 172.21.15.57:0/4049013716 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x56325beaef20 0x56325be85640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:11.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.489+0000 7fa269d9f740 1 --2- 172.21.15.57:0/4049013716 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56325beaeb40 0x56325bed0210 unknown :-1 s=CLOSED pgs=2908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:11.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.489+0000 7fa269d9f740 1 -- 172.21.15.57:0/4049013716 >> 172.21.15.57:0/4049013716 conn(0x56325be82020 msgr2=0x56325becefa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:11.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.489+0000 7fa269d9f740 1 -- 172.21.15.57:0/4049013716 shutdown_connections 2023-10-19T14:22:11.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.489+0000 7fa269d9f740 1 -- 172.21.15.57:0/4049013716 wait complete. 2023-10-19T14:22:11.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.490+0000 7fa269d9f740 1 -- start start 2023-10-19T14:22:11.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.490+0000 7fa269d9f740 1 Processor -- start 2023-10-19T14:22:11.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.491+0000 7fa269d9f740 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x56325beaeb40 0x56325beb3db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:11.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.491+0000 7fa269d9f740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56325beaef20 0x56325beb42f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:11.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.491+0000 7fa269d9f740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56325bedc280 0x56325bedafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:11.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.491+0000 7fa269d9f740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x56325bedefd0 con 0x56325beaef20 2023-10-19T14:22:11.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.491+0000 7fa269d9f740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x56325bededc0 con 0x56325bedc280 2023-10-19T14:22:11.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.491+0000 7fa269d9f740 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x56325bec6600 con 0x56325beaeb40 2023-10-19T14:22:11.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.491+0000 7fa269d9f740 10 client.0 _ll_register_callbacks cb 0x7fa25000ba60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-19T14:22:11.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.491+0000 7fa25ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56325beaef20 0x56325beb42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:11.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.491+0000 7fa25ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56325beaef20 0x56325beb42f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50114/0 (socket says 172.21.15.57:50114) 2023-10-19T14:22:11.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.491+0000 7fa25ffff640 1 -- 172.21.15.57:0/2493339139 learned_addr learned my addr 172.21.15.57:0/2493339139 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:22:11.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.491+0000 7fa264ecf640 1 --2- 172.21.15.57:0/2493339139 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x56325beaeb40 0x56325beb3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:11.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.491+0000 7fa2656d0640 1 --2- 172.21.15.57:0/2493339139 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56325bedc280 0x56325bedafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:11.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.491+0000 7fa25ffff640 1 -- 172.21.15.57:0/2493339139 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x56325beaeb40 msgr2=0x56325beb3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T14:22:11.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.491+0000 7fa25ffff640 1 --2- 172.21.15.57:0/2493339139 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x56325beaeb40 0x56325beb3db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:11.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.491+0000 7fa25ffff640 1 -- 172.21.15.57:0/2493339139 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56325bedc280 msgr2=0x56325bedafa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-19T14:22:11.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.491+0000 7fa25ffff640 1 --2- 172.21.15.57:0/2493339139 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56325bedc280 0x56325bedafa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:11.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.491+0000 7fa25ffff640 1 -- 172.21.15.57:0/2493339139 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56325bdf23d0 con 0x56325beaef20 2023-10-19T14:22:11.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.491+0000 7fa264ecf640 1 --2- 172.21.15.57:0/2493339139 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x56325beaeb40 0x56325beb3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:22:11.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.491+0000 7fa2656d0640 1 --2- 172.21.15.57:0/2493339139 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56325bedc280 0x56325bedafa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:22:11.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.491+0000 7fa25ffff640 1 --2- 172.21.15.57:0/2493339139 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56325beaef20 0x56325beb42f0 secure :-1 s=READY pgs=2909 cs=0 l=1 rev1=1 crypto rx=0x7fa24c0589a0 tx=0x7fa24c0550e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76e87b22d6384355 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:11.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.491+0000 7fa23f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2023-10-19T14:22:11.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.491+0000 7fa23f7fe640 1 -- 172.21.15.57:0/2493339139 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa24c088120 con 0x56325beaef20 2023-10-19T14:22:11.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.491+0000 7fa23f7fe640 1 -- 172.21.15.57:0/2493339139 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(263..263 src has 1..263) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fa24c060a60 con 0x56325beaef20 2023-10-19T14:22:11.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.491+0000 7fa269d9f740 1 -- 172.21.15.57:0/2493339139 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa25402c9a0 con 0x56325beaef20 2023-10-19T14:22:11.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.492+0000 7fa23f7fe640 1 -- 172.21.15.57:0/2493339139 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=116) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa24c060c40 con 0x56325beaef20 2023-10-19T14:22:11.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.492+0000 7fa269d9f740 10 client.13521 fetch_fsmap learned FSMap version 116 2023-10-19T14:22:11.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.492+0000 7fa269d9f740 1 -- 172.21.15.57:0/2493339139 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=116,monmap=2+}) v3 -- 0x56325bdb04f0 con 0x56325beaef20 2023-10-19T14:22:11.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.492+0000 7fa23f7fe640 1 -- 172.21.15.57:0/2493339139 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fa25402c9a0 con 0x56325beaef20 2023-10-19T14:22:11.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.492+0000 7fa23f7fe640 1 -- 172.21.15.57:0/2493339139 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 116) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fa24c077330 con 0x56325beaef20 2023-10-19T14:22:11.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.492+0000 7fa269d9f740 10 client.13521 fetch_fsmap finished waiting for FSMap version 116 2023-10-19T14:22:11.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.492+0000 7fa269d9f740 10 client.13521 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-19T14:22:11.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.492+0000 7fa269d9f740 20 client.13521 populate_metadata read hostname 'smithi057' 2023-10-19T14:22:11.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.492+0000 7fa269d9f740 1 --2- 172.21.15.57:0/2493339139 >> [v2:172.21.15.179:6834/4133411706,v1:172.21.15.179:6836/4133411706] conn(0x56325bedb930 0x56325beb13a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:11.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.492+0000 7fa269d9f740 4 client.13521 mds_command: new command op to 13359 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-19T14:22:11.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.492+0000 7fa269d9f740 1 -- 172.21.15.57:0/2493339139 --> [v2:172.21.15.179:6834/4133411706,v1:172.21.15.179:6836/4133411706] -- command(tid 0: {"prefix": "status"}) v1 -- 0x56325baef6c0 con 0x56325bedb930 2023-10-19T14:22:11.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.492+0000 7fa264ecf640 1 --2- 172.21.15.57:0/2493339139 >> [v2:172.21.15.179:6834/4133411706,v1:172.21.15.179:6836/4133411706] conn(0x56325bedb930 0x56325beb13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:11.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.493+0000 7fa23f7fe640 10 client.13521 ms_handle_connect on v2:172.21.15.179:6834/4133411706 2023-10-19T14:22:11.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.493+0000 7fa264ecf640 1 --2- 172.21.15.57:0/2493339139 >> [v2:172.21.15.179:6834/4133411706,v1:172.21.15.179:6836/4133411706] conn(0x56325bedb930 0x56325beb13a0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=148de4550cdbd64e server_cookie=5df83eb231d8de16 in_seq=0 out_seq=0 2023-10-19T14:22:11.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.493+0000 7fa23f7fe640 1 -- 172.21.15.57:0/2493339139 <== mds.0 v2:172.21.15.179:6834/4133411706 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+335 (crc 0 0 0) 0x56325baef6c0 con 0x56325bedb930 2023-10-19T14:22:11.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.493+0000 7fa23f7fe640 10 client.13521 handle_command_reply: tid=0 2023-10-19T14:22:11.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.493+0000 7fa269d9f740 1 client.13521 shutdown 2023-10-19T14:22:11.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.493+0000 7fa269d9f740 1 -- 172.21.15.57:0/2493339139 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56325beaef20 msgr2=0x56325beb42f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:11.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.493+0000 7fa269d9f740 1 --2- 172.21.15.57:0/2493339139 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56325beaef20 0x56325beb42f0 secure :-1 s=READY pgs=2909 cs=0 l=1 rev1=1 crypto rx=0x7fa24c0589a0 tx=0x7fa24c0550e0 comp rx=0 tx=0).stop 2023-10-19T14:22:11.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.493+0000 7fa269d9f740 1 -- 172.21.15.57:0/2493339139 shutdown_connections 2023-10-19T14:22:11.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.493+0000 7fa269d9f740 1 --2- 172.21.15.57:0/2493339139 >> [v2:172.21.15.179:6834/4133411706,v1:172.21.15.179:6836/4133411706] conn(0x56325bedb930 0x56325beb13a0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:11.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.493+0000 7fa269d9f740 1 --2- 172.21.15.57:0/2493339139 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56325bedc280 0x56325bedafa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:11.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.493+0000 7fa269d9f740 1 --2- 172.21.15.57:0/2493339139 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56325beaef20 0x56325beb42f0 unknown :-1 s=CLOSED pgs=2909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:11.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.493+0000 7fa269d9f740 1 --2- 172.21.15.57:0/2493339139 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x56325beaeb40 0x56325beb3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:11.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.493+0000 7fa23f7fe640 0 client.13521 ms_handle_reset on v2:172.21.15.179:6834/4133411706 2023-10-19T14:22:11.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.493+0000 7fa269d9f740 1 -- 172.21.15.57:0/2493339139 >> 172.21.15.57:0/2493339139 conn(0x56325be82020 msgr2=0x56325be85540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:11.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.493+0000 7fa269d9f740 1 -- 172.21.15.57:0/2493339139 shutdown_connections 2023-10-19T14:22:11.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.493+0000 7fa269d9f740 1 -- 172.21.15.57:0/2493339139 wait complete. 2023-10-19T14:22:11.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.493+0000 7fa269d9f740 20 client.13521 trim_cache size 0 max 16384 2023-10-19T14:22:11.517 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2023-10-19T14:22:11.518 INFO:teuthology.orchestra.run.smithi057.stdout: "cluster_fsid": "1e9beb19-3260-4dd7-85ef-5b399ce774ac", 2023-10-19T14:22:11.518 INFO:teuthology.orchestra.run.smithi057.stdout: "whoami": 0, 2023-10-19T14:22:11.518 INFO:teuthology.orchestra.run.smithi057.stdout: "id": 13359, 2023-10-19T14:22:11.518 INFO:teuthology.orchestra.run.smithi057.stdout: "want_state": "up:active", 2023-10-19T14:22:11.518 INFO:teuthology.orchestra.run.smithi057.stdout: "state": "up:active", 2023-10-19T14:22:11.519 INFO:teuthology.orchestra.run.smithi057.stdout: "fs_name": "cephfs", 2023-10-19T14:22:11.519 INFO:teuthology.orchestra.run.smithi057.stdout: "rank_uptime": 9.0977332050000008, 2023-10-19T14:22:11.519 INFO:teuthology.orchestra.run.smithi057.stdout: "mdsmap_epoch": 116, 2023-10-19T14:22:11.519 INFO:teuthology.orchestra.run.smithi057.stdout: "osdmap_epoch": 263, 2023-10-19T14:22:11.519 INFO:teuthology.orchestra.run.smithi057.stdout: "osdmap_epoch_barrier": 263, 2023-10-19T14:22:11.520 INFO:teuthology.orchestra.run.smithi057.stdout: "uptime": 13.814491737000001 2023-10-19T14:22:11.520 INFO:teuthology.orchestra.run.smithi057.stdout:} 2023-10-19T14:22:11.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.494+0000 7fa267133640 1 -- 172.21.15.57:0/315262566 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fa24406b080 msgr2=0x7fa24406d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:11.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.494+0000 7fa267133640 1 --2- 172.21.15.57:0/315262566 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fa24406b080 0x7fa24406d540 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa254034000 tx=0x7fa254001190 comp rx=0 tx=0).stop 2023-10-19T14:22:11.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.494+0000 7fa267133640 1 -- 172.21.15.57:0/315262566 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa26015ff00 msgr2=0x7fa26013ada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:11.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.494+0000 7fa267133640 1 --2- 172.21.15.57:0/315262566 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa26015ff00 0x7fa26013ada0 secure :-1 s=READY pgs=2905 cs=0 l=1 rev1=1 crypto rx=0x7fa24c000fb0 tx=0x7fa24c00ea90 comp rx=0 tx=0).stop 2023-10-19T14:22:11.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.494+0000 7fa267133640 1 -- 172.21.15.57:0/315262566 shutdown_connections 2023-10-19T14:22:11.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.494+0000 7fa267133640 1 --2- 172.21.15.57:0/315262566 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fa24406b080 0x7fa24406d540 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:11.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.494+0000 7fa267133640 1 --2- 172.21.15.57:0/315262566 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa260160380 0x7fa26013b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:11.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.494+0000 7fa267133640 1 --2- 172.21.15.57:0/315262566 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa26015ff00 0x7fa26013ada0 unknown :-1 s=CLOSED pgs=2905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:11.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.494+0000 7fa267133640 1 --2- 172.21.15.57:0/315262566 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa2500039a0 0x7fa26013a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:11.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.494+0000 7fa267133640 1 -- 172.21.15.57:0/315262566 >> 172.21.15.57:0/315262566 conn(0x7fa260069930 msgr2=0x7fa2600703e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:11.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.494+0000 7fa267133640 1 -- 172.21.15.57:0/315262566 shutdown_connections 2023-10-19T14:22:11.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.494+0000 7fa267133640 1 -- 172.21.15.57:0/315262566 wait complete. 2023-10-19T14:22:11.524 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-19T14:22:11.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.630+0000 7f2a9569b640 1 -- 172.21.15.57:0/795609067 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2a900c96b0 msgr2=0x7f2a900c9b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:11.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.630+0000 7f2a9569b640 1 --2- 172.21.15.57:0/795609067 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2a900c96b0 0x7f2a900c9b30 secure :-1 s=READY pgs=2911 cs=0 l=1 rev1=1 crypto rx=0x7f2a78009860 tx=0x7f2a7802bd60 comp rx=0 tx=0).stop 2023-10-19T14:22:11.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.630+0000 7f2a9569b640 1 -- 172.21.15.57:0/795609067 shutdown_connections 2023-10-19T14:22:11.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.630+0000 7f2a9569b640 1 --2- 172.21.15.57:0/795609067 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2a900c96b0 0x7f2a900c9b30 unknown :-1 s=CLOSED pgs=2911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:11.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.630+0000 7f2a9569b640 1 --2- 172.21.15.57:0/795609067 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2a9006af00 0x7f2a9006b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:11.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.630+0000 7f2a9569b640 1 -- 172.21.15.57:0/795609067 >> 172.21.15.57:0/795609067 conn(0x7f2a90069930 msgr2=0x7f2a90069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:11.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.630+0000 7f2a9569b640 1 -- 172.21.15.57:0/795609067 shutdown_connections 2023-10-19T14:22:11.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.630+0000 7f2a9569b640 1 -- 172.21.15.57:0/795609067 wait complete. 2023-10-19T14:22:11.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.631+0000 7f2a9569b640 1 Processor -- start 2023-10-19T14:22:11.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.631+0000 7f2a9569b640 1 -- start start 2023-10-19T14:22:11.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.632+0000 7f2a9569b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2a9006af00 0x7f2a90152600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:11.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.632+0000 7f2a9569b640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2a900ab500 0x7f2a90134d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:11.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.632+0000 7f2a9569b640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2a900c96b0 0x7f2a90135280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:11.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.632+0000 7f2a9569b640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a901f03c0 con 0x7f2a9006af00 2023-10-19T14:22:11.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.632+0000 7f2a9569b640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a901efa20 con 0x7f2a900c96b0 2023-10-19T14:22:11.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.632+0000 7f2a9569b640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a901efd00 con 0x7f2a900ab500 2023-10-19T14:22:11.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.632+0000 7f2a8effd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2a9006af00 0x7f2a90152600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:11.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.632+0000 7f2a8effd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2a9006af00 0x7f2a90152600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50124/0 (socket says 172.21.15.57:50124) 2023-10-19T14:22:11.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.632+0000 7f2a8effd640 1 -- 172.21.15.57:0/1050248414 learned_addr learned my addr 172.21.15.57:0/1050248414 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:22:11.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.632+0000 7f2a8effd640 1 -- 172.21.15.57:0/1050248414 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2a900ab500 msgr2=0x7f2a90134d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:11.639 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.632+0000 7f2a8effd640 1 --2- 172.21.15.57:0/1050248414 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2a900ab500 0x7f2a90134d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:11.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.632+0000 7f2a8effd640 1 -- 172.21.15.57:0/1050248414 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2a900c96b0 msgr2=0x7f2a90135280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:11.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.632+0000 7f2a8effd640 1 --2- 172.21.15.57:0/1050248414 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2a900c96b0 0x7f2a90135280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:11.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.632+0000 7f2a8effd640 1 -- 172.21.15.57:0/1050248414 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a78009490 con 0x7f2a9006af00 2023-10-19T14:22:11.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.633+0000 7f2a8effd640 1 --2- 172.21.15.57:0/1050248414 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2a9006af00 0x7f2a90152600 secure :-1 s=READY pgs=2912 cs=0 l=1 rev1=1 crypto rx=0x7f2a8000d950 tx=0x7f2a8000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:11.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.633+0000 7f2a877fe640 1 -- 172.21.15.57:0/1050248414 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a80009c30 con 0x7f2a9006af00 2023-10-19T14:22:11.640 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.633+0000 7f2a877fe640 1 -- 172.21.15.57:0/1050248414 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f2a80009e80 con 0x7f2a9006af00 2023-10-19T14:22:11.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.633+0000 7f2a9569b640 1 -- 172.21.15.57:0/1050248414 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a901effe0 con 0x7f2a9006af00 2023-10-19T14:22:11.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.633+0000 7f2a877fe640 1 -- 172.21.15.57:0/1050248414 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a800106c0 con 0x7f2a9006af00 2023-10-19T14:22:11.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.633+0000 7f2a9569b640 1 -- 172.21.15.57:0/1050248414 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a90152b40 con 0x7f2a9006af00 2023-10-19T14:22:11.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.634+0000 7f2a9569b640 1 -- 172.21.15.57:0/1050248414 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a90152d30 con 0x7f2a9006af00 2023-10-19T14:22:11.641 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.634+0000 7f2a877fe640 1 -- 172.21.15.57:0/1050248414 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 145) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f2a80010820 con 0x7f2a9006af00 2023-10-19T14:22:11.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.635+0000 7f2a877fe640 1 --2- 172.21.15.57:0/1050248414 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f2a7006b080 0x7f2a7006d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:11.642 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.635+0000 7f2a877fe640 1 -- 172.21.15.57:0/1050248414 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(263..263 src has 1..263) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f2a8008a1a0 con 0x7f2a9006af00 2023-10-19T14:22:11.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.635+0000 7f2a87fff640 1 --2- 172.21.15.57:0/1050248414 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f2a7006b080 0x7f2a7006d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:11.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.636+0000 7f2a87fff640 1 --2- 172.21.15.57:0/1050248414 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f2a7006b080 0x7f2a7006d540 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2a78036000 tx=0x7f2a780046c0 comp rx=0 tx=0).ready entity=mgr.11446 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:11.643 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.639+0000 7f2a877fe640 1 -- 172.21.15.57:0/1050248414 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2a80054c80 con 0x7f2a9006af00 2023-10-19T14:22:11.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.851+0000 7f2a9569b640 1 -- 172.21.15.57:0/1050248414 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f2a900690e0 con 0x7f2a9006af00 2023-10-19T14:22:11.854 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.851+0000 7f2a877fe640 1 -- 172.21.15.57:0/1050248414 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v116) v1 ==== 53+0+73 (secure 0 0 0) 0x7f2a800582a0 con 0x7f2a9006af00 2023-10-19T14:22:11.855 INFO:teuthology.orchestra.run.smithi057.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-19T14:22:11.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.853+0000 7f2a9569b640 1 -- 172.21.15.57:0/1050248414 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f2a7006b080 msgr2=0x7f2a7006d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:11.856 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.853+0000 7f2a9569b640 1 --2- 172.21.15.57:0/1050248414 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f2a7006b080 0x7f2a7006d540 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2a78036000 tx=0x7f2a780046c0 comp rx=0 tx=0).stop 2023-10-19T14:22:11.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.853+0000 7f2a9569b640 1 -- 172.21.15.57:0/1050248414 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2a9006af00 msgr2=0x7f2a90152600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:11.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.853+0000 7f2a9569b640 1 --2- 172.21.15.57:0/1050248414 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2a9006af00 0x7f2a90152600 secure :-1 s=READY pgs=2912 cs=0 l=1 rev1=1 crypto rx=0x7f2a8000d950 tx=0x7f2a8000de10 comp rx=0 tx=0).stop 2023-10-19T14:22:11.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.853+0000 7f2a9569b640 1 -- 172.21.15.57:0/1050248414 shutdown_connections 2023-10-19T14:22:11.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.853+0000 7f2a9569b640 1 --2- 172.21.15.57:0/1050248414 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f2a7006b080 0x7f2a7006d540 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:11.857 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.853+0000 7f2a9569b640 1 --2- 172.21.15.57:0/1050248414 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2a900c96b0 0x7f2a90135280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:11.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.853+0000 7f2a9569b640 1 --2- 172.21.15.57:0/1050248414 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2a900ab500 0x7f2a90134d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:11.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.853+0000 7f2a9569b640 1 --2- 172.21.15.57:0/1050248414 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2a9006af00 0x7f2a90152600 unknown :-1 s=CLOSED pgs=2912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:11.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.853+0000 7f2a9569b640 1 -- 172.21.15.57:0/1050248414 >> 172.21.15.57:0/1050248414 conn(0x7f2a90069930 msgr2=0x7f2a9015b410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:11.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.853+0000 7f2a9569b640 1 -- 172.21.15.57:0/1050248414 shutdown_connections 2023-10-19T14:22:11.858 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:11.853+0000 7f2a9569b640 1 -- 172.21.15.57:0/1050248414 wait complete. 2023-10-19T14:22:11.867 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-10-19T14:22:11.867 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-19T14:22:11.867 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi057.front.sepia.ceph.com 2023-10-19T14:22:11.868 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-19T14:22:11.868 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-19T14:22:11.868 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-10-19T14:22:11.869 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-10-19T14:22:11.869 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-19T14:22:11.869 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-19T14:22:11.870 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:22:11.870 DEBUG:teuthology.orchestra.run.smithi057:> ip addr 2023-10-19T14:22:11.886 INFO:teuthology.orchestra.run.smithi057.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-19T14:22:11.886 INFO:teuthology.orchestra.run.smithi057.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-19T14:22:11.886 INFO:teuthology.orchestra.run.smithi057.stdout: inet 127.0.0.1/8 scope host lo 2023-10-19T14:22:11.887 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T14:22:11.887 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 ::1/128 scope host 2023-10-19T14:22:11.887 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T14:22:11.887 INFO:teuthology.orchestra.run.smithi057.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-19T14:22:11.887 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:6b:ff:86 brd ff:ff:ff:ff:ff:ff 2023-10-19T14:22:11.888 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp7s0f0 2023-10-19T14:22:11.888 INFO:teuthology.orchestra.run.smithi057.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-19T14:22:11.888 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:6b:ff:87 brd ff:ff:ff:ff:ff:ff 2023-10-19T14:22:11.888 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp7s0f1 2023-10-19T14:22:11.888 INFO:teuthology.orchestra.run.smithi057.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-19T14:22:11.888 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:bd:15:62 brd ff:ff:ff:ff:ff:ff 2023-10-19T14:22:11.889 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp1s0f0 2023-10-19T14:22:11.889 INFO:teuthology.orchestra.run.smithi057.stdout: inet 172.21.15.57/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-19T14:22:11.889 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft 36779sec preferred_lft 36779sec 2023-10-19T14:22:11.889 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 fe80::ec4:7aff:febd:1562/64 scope link noprefixroute 2023-10-19T14:22:11.889 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T14:22:11.890 INFO:teuthology.orchestra.run.smithi057.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-19T14:22:11.890 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:bd:15:63 brd ff:ff:ff:ff:ff:ff 2023-10-19T14:22:11.890 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp1s0f1 2023-10-19T14:22:11.890 INFO:teuthology.orchestra.run.smithi057.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-19T14:22:11.890 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0e:fe:f1:b5:41:59 brd ff:ff:ff:ff:ff:ff 2023-10-19T14:22:11.891 INFO:teuthology.orchestra.run.smithi057.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-19T14:22:11.891 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T14:22:11.891 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 fe80::64b1:a2ff:feb3:b08/64 scope link 2023-10-19T14:22:11.891 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T14:22:11.891 INFO:teuthology.orchestra.run.smithi057.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-19T14:22:11.891 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0e:fe:f1:b5:41:59 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T14:22:11.892 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 fe80::cfe:f1ff:feb5:4159/64 scope link 2023-10-19T14:22:11.892 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T14:22:11.892 INFO:teuthology.orchestra.run.smithi057.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-19T14:22:11.892 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 62:53:72:9d:52:71 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-19T14:22:11.892 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 fe80::6053:72ff:fe9d:5271/64 scope link 2023-10-19T14:22:11.893 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T14:22:11.893 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:22:11.893 DEBUG:teuthology.orchestra.run.smithi057:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-19T14:22:11.956 INFO:teuthology.orchestra.run.smithi057.stdout:1 2023-10-19T14:22:11.958 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:22:11.958 DEBUG:teuthology.orchestra.run.smithi057:> route 2023-10-19T14:22:12.014 INFO:teuthology.orchestra.run.smithi057.stdout:Kernel IP routing table 2023-10-19T14:22:12.014 INFO:teuthology.orchestra.run.smithi057.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-19T14:22:12.014 INFO:teuthology.orchestra.run.smithi057.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-19T14:22:12.014 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-19T14:22:12.015 INFO:teuthology.orchestra.run.smithi057.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-19T14:22:12.015 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:22:12.015 DEBUG:teuthology.orchestra.run.smithi057:> (cd / && exec bash -c ' 2023-10-19T14:22:12.016 DEBUG:teuthology.orchestra.run.smithi057:> set -e 2023-10-19T14:22:12.016 DEBUG:teuthology.orchestra.run.smithi057:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-19T14:22:12.017 DEBUG:teuthology.orchestra.run.smithi057:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-19T14:22:12.017 DEBUG:teuthology.orchestra.run.smithi057:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-19T14:22:12.017 DEBUG:teuthology.orchestra.run.smithi057:> ') 2023-10-19T14:22:12.119 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:22:12.120 DEBUG:teuthology.orchestra.run.smithi057:> ip netns list 2023-10-19T14:22:12.135 INFO:teuthology.orchestra.run.smithi057.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-19T14:22:12.135 INFO:teuthology.orchestra.run.smithi057.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-19T14:22:12.136 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-19T14:22:12.136 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:22:12.136 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip link set brx.0 up 2023-10-19T14:22:12.206 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-10-19T14:22:12.207 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-19T14:22:12.207 DEBUG:teuthology.orchestra.run.smithi057:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-19T14:22:12.262 INFO:teuthology.orchestra.run.smithi057.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-19T14:22:12.262 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-19T14:22:12.263 DEBUG:teuthology.orchestra.run.smithi057:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-19T14:22:12.318 DEBUG:teuthology.orchestra.run.smithi057:> sudo modprobe fuse 2023-10-19T14:22:12.388 DEBUG:teuthology.orchestra.run.smithi057:> cat /proc/self/mounts | awk '{print $2}' 2023-10-19T14:22:12.445 INFO:teuthology.orchestra.run.smithi057.stdout:/proc 2023-10-19T14:22:12.445 INFO:teuthology.orchestra.run.smithi057.stdout:/sys 2023-10-19T14:22:12.445 INFO:teuthology.orchestra.run.smithi057.stdout:/dev 2023-10-19T14:22:12.446 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/security 2023-10-19T14:22:12.446 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/shm 2023-10-19T14:22:12.446 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/pts 2023-10-19T14:22:12.446 INFO:teuthology.orchestra.run.smithi057.stdout:/run 2023-10-19T14:22:12.446 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/cgroup 2023-10-19T14:22:12.447 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/pstore 2023-10-19T14:22:12.447 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/bpf 2023-10-19T14:22:12.447 INFO:teuthology.orchestra.run.smithi057.stdout:/ 2023-10-19T14:22:12.447 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/selinux 2023-10-19T14:22:12.447 INFO:teuthology.orchestra.run.smithi057.stdout:/proc/sys/fs/binfmt_misc 2023-10-19T14:22:12.448 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/hugepages 2023-10-19T14:22:12.448 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/mqueue 2023-10-19T14:22:12.448 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/debug 2023-10-19T14:22:12.448 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/tracing 2023-10-19T14:22:12.448 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/fuse/connections 2023-10-19T14:22:12.449 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/config 2023-10-19T14:22:12.449 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-sysctl.service 2023-10-19T14:22:12.449 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-10-19T14:22:12.449 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-10-19T14:22:12.450 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/nfs/rpc_pipefs 2023-10-19T14:22:12.450 INFO:teuthology.orchestra.run.smithi057.stdout:/run/user/1000 2023-10-19T14:22:12.450 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph 2023-10-19T14:22:12.450 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-0 2023-10-19T14:22:12.450 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-1 2023-10-19T14:22:12.451 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-2 2023-10-19T14:22:12.451 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-3 2023-10-19T14:22:12.451 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns 2023-10-19T14:22:12.451 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T14:22:12.452 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T14:22:12.452 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-19T14:22:12.452 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-19T14:22:12.452 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:22:12.453 DEBUG:teuthology.orchestra.run.smithi057:> ls /sys/fs/fuse/connections 2023-10-19T14:22:12.501 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-10-19T14:22:12.502 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-10-19T14:22:12.544 DEBUG:teuthology.orchestra.run.smithi057:> sudo modprobe fuse 2023-10-19T14:22:12.575 DEBUG:teuthology.orchestra.run.smithi057:> cat /proc/self/mounts | awk '{print $2}' 2023-10-19T14:22:12.632 INFO:teuthology.orchestra.run.smithi057.stdout:/proc 2023-10-19T14:22:12.632 INFO:teuthology.orchestra.run.smithi057.stdout:/sys 2023-10-19T14:22:12.632 INFO:teuthology.orchestra.run.smithi057.stdout:/dev 2023-10-19T14:22:12.633 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/security 2023-10-19T14:22:12.633 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/shm 2023-10-19T14:22:12.633 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/pts 2023-10-19T14:22:12.633 INFO:teuthology.orchestra.run.smithi057.stdout:/run 2023-10-19T14:22:12.633 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/cgroup 2023-10-19T14:22:12.634 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/pstore 2023-10-19T14:22:12.634 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/bpf 2023-10-19T14:22:12.634 INFO:teuthology.orchestra.run.smithi057.stdout:/ 2023-10-19T14:22:12.635 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/selinux 2023-10-19T14:22:12.635 INFO:teuthology.orchestra.run.smithi057.stdout:/proc/sys/fs/binfmt_misc 2023-10-19T14:22:12.635 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/hugepages 2023-10-19T14:22:12.635 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/mqueue 2023-10-19T14:22:12.635 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/debug 2023-10-19T14:22:12.636 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/tracing 2023-10-19T14:22:12.636 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/fuse/connections 2023-10-19T14:22:12.636 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/config 2023-10-19T14:22:12.636 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-sysctl.service 2023-10-19T14:22:12.636 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-10-19T14:22:12.637 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-10-19T14:22:12.637 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/nfs/rpc_pipefs 2023-10-19T14:22:12.637 INFO:teuthology.orchestra.run.smithi057.stdout:/run/user/1000 2023-10-19T14:22:12.637 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph 2023-10-19T14:22:12.637 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-0 2023-10-19T14:22:12.638 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-1 2023-10-19T14:22:12.638 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-2 2023-10-19T14:22:12.638 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-3 2023-10-19T14:22:12.638 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns 2023-10-19T14:22:12.638 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T14:22:12.639 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T14:22:12.639 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-19T14:22:12.639 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-19T14:22:12.639 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:22:12.640 DEBUG:teuthology.orchestra.run.smithi057:> ls /sys/fs/fuse/connections 2023-10-19T14:22:12.641 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi057.stderr:2023-10-19T14:22:12.634+0000 7fcba67ee480 -1 init, newargv = 0x56382d195c80 newargc=15 2023-10-19T14:22:12.641 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi057.stderr:ceph-fuse[163529]: starting ceph client 2023-10-19T14:22:12.641 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi057.stderr:ceph-fuse[163529]: starting fuse 2023-10-19T14:22:12.665 INFO:teuthology.orchestra.run.smithi057.stdout:43 2023-10-19T14:22:12.669 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2023-10-19T14:22:12.670 DEBUG:teuthology.orchestra.run.smithi057:> sudo stdin-killer -- python3 -c ' 2023-10-19T14:22:12.670 DEBUG:teuthology.orchestra.run.smithi057:> import glob 2023-10-19T14:22:12.671 DEBUG:teuthology.orchestra.run.smithi057:> import re 2023-10-19T14:22:12.671 DEBUG:teuthology.orchestra.run.smithi057:> import os 2023-10-19T14:22:12.671 DEBUG:teuthology.orchestra.run.smithi057:> import subprocess 2023-10-19T14:22:12.672 DEBUG:teuthology.orchestra.run.smithi057:> 2023-10-19T14:22:12.672 DEBUG:teuthology.orchestra.run.smithi057:> def _find_admin_socket(client_name): 2023-10-19T14:22:12.673 DEBUG:teuthology.orchestra.run.smithi057:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-10-19T14:22:12.673 DEBUG:teuthology.orchestra.run.smithi057:> files = glob.glob(asok_path) 2023-10-19T14:22:12.673 DEBUG:teuthology.orchestra.run.smithi057:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-10-19T14:22:12.674 DEBUG:teuthology.orchestra.run.smithi057:> 2023-10-19T14:22:12.674 DEBUG:teuthology.orchestra.run.smithi057:> # Given a non-glob path, it better be there 2023-10-19T14:22:12.675 DEBUG:teuthology.orchestra.run.smithi057:> if "*" not in asok_path: 2023-10-19T14:22:12.675 DEBUG:teuthology.orchestra.run.smithi057:> assert(len(files) == 1) 2023-10-19T14:22:12.675 DEBUG:teuthology.orchestra.run.smithi057:> return files[0] 2023-10-19T14:22:12.676 DEBUG:teuthology.orchestra.run.smithi057:> 2023-10-19T14:22:12.676 DEBUG:teuthology.orchestra.run.smithi057:> for f in files: 2023-10-19T14:22:12.677 DEBUG:teuthology.orchestra.run.smithi057:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-10-19T14:22:12.677 DEBUG:teuthology.orchestra.run.smithi057:> if os.path.exists("/proc/{0}".format(pid)): 2023-10-19T14:22:12.677 DEBUG:teuthology.orchestra.run.smithi057:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-10-19T14:22:12.678 DEBUG:teuthology.orchestra.run.smithi057:> contents = proc_f.read() 2023-10-19T14:22:12.678 DEBUG:teuthology.orchestra.run.smithi057:> if mountpoint in contents: 2023-10-19T14:22:12.678 DEBUG:teuthology.orchestra.run.smithi057:> return f 2023-10-19T14:22:12.679 DEBUG:teuthology.orchestra.run.smithi057:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-10-19T14:22:12.679 DEBUG:teuthology.orchestra.run.smithi057:> 2023-10-19T14:22:12.680 DEBUG:teuthology.orchestra.run.smithi057:> print(_find_admin_socket("client.0")) 2023-10-19T14:22:12.680 DEBUG:teuthology.orchestra.run.smithi057:> ' 2023-10-19T14:22:12.798 INFO:teuthology.orchestra.run.smithi057.stdout:/var/run/ceph/ceph-client.0.163529.asok 2023-10-19T14:22:12.800 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:12 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2023-10-19T14:22:12.808 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.163529.asok 2023-10-19T14:22:12.808 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:22:12.809 DEBUG:teuthology.orchestra.run.smithi057:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.163529.asok status 2023-10-19T14:22:12.954 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2023-10-19T14:22:12.954 INFO:teuthology.orchestra.run.smithi057.stdout: "metadata": { 2023-10-19T14:22:12.954 INFO:teuthology.orchestra.run.smithi057.stdout: "ceph_sha1": "f7420d7f8e55ceea786dcbf536c338ce920c21eb", 2023-10-19T14:22:12.955 INFO:teuthology.orchestra.run.smithi057.stdout: "ceph_version": "ceph version 18.0.0-6786-gf7420d7f (f7420d7f8e55ceea786dcbf536c338ce920c21eb) reef (dev)", 2023-10-19T14:22:12.955 INFO:teuthology.orchestra.run.smithi057.stdout: "entity_id": "0", 2023-10-19T14:22:12.955 INFO:teuthology.orchestra.run.smithi057.stdout: "hostname": "smithi057", 2023-10-19T14:22:12.955 INFO:teuthology.orchestra.run.smithi057.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-10-19T14:22:12.955 INFO:teuthology.orchestra.run.smithi057.stdout: "pid": "163529", 2023-10-19T14:22:12.956 INFO:teuthology.orchestra.run.smithi057.stdout: "root": "/" 2023-10-19T14:22:12.956 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:22:12.956 INFO:teuthology.orchestra.run.smithi057.stdout: "dentry_count": 0, 2023-10-19T14:22:12.956 INFO:teuthology.orchestra.run.smithi057.stdout: "dentry_pinned_count": 0, 2023-10-19T14:22:12.956 INFO:teuthology.orchestra.run.smithi057.stdout: "id": 13533, 2023-10-19T14:22:12.957 INFO:teuthology.orchestra.run.smithi057.stdout: "inst": { 2023-10-19T14:22:12.957 INFO:teuthology.orchestra.run.smithi057.stdout: "name": { 2023-10-19T14:22:12.957 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "client", 2023-10-19T14:22:12.957 INFO:teuthology.orchestra.run.smithi057.stdout: "num": 13533 2023-10-19T14:22:12.958 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:22:12.958 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": { 2023-10-19T14:22:12.958 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "v1", 2023-10-19T14:22:12.958 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": "192.168.0.1:0", 2023-10-19T14:22:12.958 INFO:teuthology.orchestra.run.smithi057.stdout: "nonce": 2928940113 2023-10-19T14:22:12.958 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:22:12.959 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:22:12.959 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": { 2023-10-19T14:22:12.959 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "v1", 2023-10-19T14:22:12.960 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": "192.168.0.1:0", 2023-10-19T14:22:12.960 INFO:teuthology.orchestra.run.smithi057.stdout: "nonce": 2928940113 2023-10-19T14:22:12.960 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:22:12.960 INFO:teuthology.orchestra.run.smithi057.stdout: "inst_str": "client.13533 192.168.0.1:0/2928940113", 2023-10-19T14:22:12.960 INFO:teuthology.orchestra.run.smithi057.stdout: "addr_str": "192.168.0.1:0/2928940113", 2023-10-19T14:22:12.961 INFO:teuthology.orchestra.run.smithi057.stdout: "inode_count": 1, 2023-10-19T14:22:12.961 INFO:teuthology.orchestra.run.smithi057.stdout: "mds_epoch": 116, 2023-10-19T14:22:12.961 INFO:teuthology.orchestra.run.smithi057.stdout: "osd_epoch": 263, 2023-10-19T14:22:12.961 INFO:teuthology.orchestra.run.smithi057.stdout: "osd_epoch_barrier": 0, 2023-10-19T14:22:12.962 INFO:teuthology.orchestra.run.smithi057.stdout: "blocklisted": false, 2023-10-19T14:22:12.962 INFO:teuthology.orchestra.run.smithi057.stdout: "fs_name": "cephfs" 2023-10-19T14:22:12.962 INFO:teuthology.orchestra.run.smithi057.stdout:} 2023-10-19T14:22:12.964 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:22:12.965 DEBUG:teuthology.orchestra.run.smithi057:> stat --file-system '--printf=%T 2023-10-19T14:22:12.965 DEBUG:teuthology.orchestra.run.smithi057:> ' -- /home/ubuntu/cephtest/mnt.0 2023-10-19T14:22:12.982 INFO:teuthology.orchestra.run.smithi057.stdout:fuseblk 2023-10-19T14:22:12.982 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-10-19T14:22:12.983 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:22:12.983 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-19T14:22:13.056 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-19T14:22:13.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.218+0000 7fa6b73a9640 1 -- 172.21.15.57:0/1460516999 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa69c003970 msgr2=0x7fa69c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:13.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.218+0000 7fa6b73a9640 1 --2- 172.21.15.57:0/1460516999 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa69c003970 0x7fa69c003d90 secure :-1 s=READY pgs=2915 cs=0 l=1 rev1=1 crypto rx=0x7fa6ac009b50 tx=0x7fa69c0057f0 comp rx=0 tx=0).stop 2023-10-19T14:22:13.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.218+0000 7fa6b73a9640 1 -- 172.21.15.57:0/1460516999 shutdown_connections 2023-10-19T14:22:13.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.218+0000 7fa6b73a9640 1 --2- 172.21.15.57:0/1460516999 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa69c003970 0x7fa69c003d90 unknown :-1 s=CLOSED pgs=2915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:13.222 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.218+0000 7fa6b73a9640 1 -- 172.21.15.57:0/1460516999 >> 172.21.15.57:0/1460516999 conn(0x7fa6b0069930 msgr2=0x7fa6b0069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:13.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.218+0000 7fa6b73a9640 1 -- 172.21.15.57:0/1460516999 shutdown_connections 2023-10-19T14:22:13.223 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.218+0000 7fa6b73a9640 1 -- 172.21.15.57:0/1460516999 wait complete. 2023-10-19T14:22:13.226 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.219+0000 7fa6b73a9640 1 Processor -- start 2023-10-19T14:22:13.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.219+0000 7fa6b73a9640 1 -- start start 2023-10-19T14:22:13.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.220+0000 7fa6b73a9640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa69c003970 0x7fa6b013a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:13.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.220+0000 7fa6b73a9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa6b01553a0 0x7fa6b00ae700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:13.227 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.220+0000 7fa6b73a9640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa6b0155820 0x7fa6b01524c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:13.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.220+0000 7fa6b73a9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b013ab90 con 0x7fa6b01553a0 2023-10-19T14:22:13.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.220+0000 7fa6b73a9640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b00af150 con 0x7fa6b0155820 2023-10-19T14:22:13.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.220+0000 7fa6b73a9640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b00af430 con 0x7fa69c003970 2023-10-19T14:22:13.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.220+0000 7fa6b4944640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa6b01553a0 0x7fa6b00ae700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:13.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.220+0000 7fa6b4944640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa6b01553a0 0x7fa6b00ae700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50136/0 (socket says 172.21.15.57:50136) 2023-10-19T14:22:13.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.220+0000 7fa6b4944640 1 -- 172.21.15.57:0/572601164 learned_addr learned my addr 172.21.15.57:0/572601164 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:22:13.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.221+0000 7fa6b5145640 1 --2- 172.21.15.57:0/572601164 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa69c003970 0x7fa6b013a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:13.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.221+0000 7fa6b5946640 1 --2- 172.21.15.57:0/572601164 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa6b0155820 0x7fa6b01524c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:13.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.221+0000 7fa6b4944640 1 -- 172.21.15.57:0/572601164 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa69c003970 msgr2=0x7fa6b013a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:13.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.221+0000 7fa6b4944640 1 --2- 172.21.15.57:0/572601164 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa69c003970 0x7fa6b013a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:13.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.221+0000 7fa6b4944640 1 -- 172.21.15.57:0/572601164 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa6b0155820 msgr2=0x7fa6b01524c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:13.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.221+0000 7fa6b4944640 1 --2- 172.21.15.57:0/572601164 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa6b0155820 0x7fa6b01524c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:13.230 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.221+0000 7fa6b4944640 1 -- 172.21.15.57:0/572601164 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6b00af710 con 0x7fa6b01553a0 2023-10-19T14:22:13.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.221+0000 7fa6b4944640 1 --2- 172.21.15.57:0/572601164 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa6b01553a0 0x7fa6b00ae700 secure :-1 s=READY pgs=2916 cs=0 l=1 rev1=1 crypto rx=0x7fa6a0003e90 tx=0x7fa6a0001630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:13.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.221+0000 7fa6a5ffb640 1 -- 172.21.15.57:0/572601164 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6a0011840 con 0x7fa6b01553a0 2023-10-19T14:22:13.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.221+0000 7fa6a5ffb640 1 -- 172.21.15.57:0/572601164 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fa6a00119a0 con 0x7fa6b01553a0 2023-10-19T14:22:13.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.221+0000 7fa6b73a9640 1 -- 172.21.15.57:0/572601164 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6b00af9f0 con 0x7fa6b01553a0 2023-10-19T14:22:13.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.221+0000 7fa6a5ffb640 1 -- 172.21.15.57:0/572601164 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6a0011d90 con 0x7fa6b01553a0 2023-10-19T14:22:13.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.221+0000 7fa6b73a9640 1 -- 172.21.15.57:0/572601164 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6b0153a80 con 0x7fa6b01553a0 2023-10-19T14:22:13.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.222+0000 7fa6b73a9640 1 -- 172.21.15.57:0/572601164 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6b00690e0 con 0x7fa6b01553a0 2023-10-19T14:22:13.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.223+0000 7fa6a5ffb640 1 -- 172.21.15.57:0/572601164 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 145) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fa6a00103c0 con 0x7fa6b01553a0 2023-10-19T14:22:13.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.223+0000 7fa6a5ffb640 1 --2- 172.21.15.57:0/572601164 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fa69406b080 0x7fa69406d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:13.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.223+0000 7fa6a5ffb640 1 -- 172.21.15.57:0/572601164 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(263..263 src has 1..263) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fa6a008abb0 con 0x7fa6b01553a0 2023-10-19T14:22:13.233 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.223+0000 7fa6b5145640 1 --2- 172.21.15.57:0/572601164 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fa69406b080 0x7fa69406d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:13.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.224+0000 7fa6b5145640 1 --2- 172.21.15.57:0/572601164 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fa69406b080 0x7fa69406d540 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa6ac036000 tx=0x7fa6ac035040 comp rx=0 tx=0).ready entity=mgr.11446 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:13.234 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.227+0000 7fa6a5ffb640 1 -- 172.21.15.57:0/572601164 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa6a0055720 con 0x7fa6b01553a0 2023-10-19T14:22:13.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.424+0000 7fa6b73a9640 1 -- 172.21.15.57:0/572601164 --> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fa6b015a670 con 0x7fa69406b080 2023-10-19T14:22:13.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.426+0000 7fa6a5ffb640 1 -- 172.21.15.57:0/572601164 <== mgr.11446 v2:172.21.15.179:6832/1573806521 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fa6b015a670 con 0x7fa69406b080 2023-10-19T14:22:13.429 INFO:teuthology.orchestra.run.smithi057.stdout:[ 2023-10-19T14:22:13.430 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-10-19T14:22:13.430 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "cephfs" 2023-10-19T14:22:13.431 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:22:13.431 INFO:teuthology.orchestra.run.smithi057.stdout:] 2023-10-19T14:22:13.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.427+0000 7fa6b73a9640 1 -- 172.21.15.57:0/572601164 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fa69406b080 msgr2=0x7fa69406d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:13.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.427+0000 7fa6b73a9640 1 --2- 172.21.15.57:0/572601164 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fa69406b080 0x7fa69406d540 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa6ac036000 tx=0x7fa6ac035040 comp rx=0 tx=0).stop 2023-10-19T14:22:13.433 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.427+0000 7fa6b73a9640 1 -- 172.21.15.57:0/572601164 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa6b01553a0 msgr2=0x7fa6b00ae700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:13.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.427+0000 7fa6b73a9640 1 --2- 172.21.15.57:0/572601164 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa6b01553a0 0x7fa6b00ae700 secure :-1 s=READY pgs=2916 cs=0 l=1 rev1=1 crypto rx=0x7fa6a0003e90 tx=0x7fa6a0001630 comp rx=0 tx=0).stop 2023-10-19T14:22:13.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.427+0000 7fa6b73a9640 1 -- 172.21.15.57:0/572601164 shutdown_connections 2023-10-19T14:22:13.434 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.427+0000 7fa6b73a9640 1 --2- 172.21.15.57:0/572601164 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fa69406b080 0x7fa69406d540 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:13.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.427+0000 7fa6b73a9640 1 --2- 172.21.15.57:0/572601164 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa6b0155820 0x7fa6b01524c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:13.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.427+0000 7fa6b73a9640 1 --2- 172.21.15.57:0/572601164 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa6b01553a0 0x7fa6b00ae700 unknown :-1 s=CLOSED pgs=2916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:13.435 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.427+0000 7fa6b73a9640 1 --2- 172.21.15.57:0/572601164 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa69c003970 0x7fa6b013a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:13.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.427+0000 7fa6b73a9640 1 -- 172.21.15.57:0/572601164 >> 172.21.15.57:0/572601164 conn(0x7fa6b0069930 msgr2=0x7fa6b013ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:13.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.427+0000 7fa6b73a9640 1 -- 172.21.15.57:0/572601164 shutdown_connections 2023-10-19T14:22:13.436 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.428+0000 7fa6b73a9640 1 -- 172.21.15.57:0/572601164 wait complete. 2023-10-19T14:22:13.442 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-19T14:22:13.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.554+0000 7f4a44b8e640 1 -- 172.21.15.57:0/1066522139 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4a40141690 msgr2=0x7f4a400703e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:13.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.554+0000 7f4a44b8e640 1 --2- 172.21.15.57:0/1066522139 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4a40141690 0x7f4a400703e0 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f4a28009830 tx=0x7f4a2802bc30 comp rx=0 tx=0).stop 2023-10-19T14:22:13.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.554+0000 7f4a44b8e640 1 -- 172.21.15.57:0/1066522139 shutdown_connections 2023-10-19T14:22:13.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.554+0000 7f4a44b8e640 1 --2- 172.21.15.57:0/1066522139 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4a40141690 0x7f4a400703e0 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:13.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.554+0000 7f4a44b8e640 1 --2- 172.21.15.57:0/1066522139 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4a4013e520 0x7f4a4006bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:13.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.554+0000 7f4a44b8e640 1 -- 172.21.15.57:0/1066522139 >> 172.21.15.57:0/1066522139 conn(0x7f4a40069930 msgr2=0x7f4a40069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:13.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.554+0000 7f4a44b8e640 1 -- 172.21.15.57:0/1066522139 shutdown_connections 2023-10-19T14:22:13.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.554+0000 7f4a44b8e640 1 -- 172.21.15.57:0/1066522139 wait complete. 2023-10-19T14:22:13.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.555+0000 7f4a44b8e640 1 Processor -- start 2023-10-19T14:22:13.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.555+0000 7f4a44b8e640 1 -- start start 2023-10-19T14:22:13.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.556+0000 7f4a44b8e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4a4006c0a0 0x7f4a4013aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:13.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.556+0000 7f4a44b8e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a4013e520 0x7f4a4013d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:13.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.556+0000 7f4a44b8e640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4a40141690 0x7f4a4013d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:13.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.556+0000 7f4a44b8e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a401e0430 con 0x7f4a4013e520 2023-10-19T14:22:13.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.556+0000 7f4a44b8e640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a401e0710 con 0x7f4a4006c0a0 2023-10-19T14:22:13.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.556+0000 7f4a44b8e640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a401e09f0 con 0x7f4a40141690 2023-10-19T14:22:13.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.556+0000 7f4a35d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a4013e520 0x7f4a4013d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:13.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.556+0000 7f4a35d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a4013e520 0x7f4a4013d1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50140/0 (socket says 172.21.15.57:50140) 2023-10-19T14:22:13.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.556+0000 7f4a35d9b640 1 -- 172.21.15.57:0/3627538840 learned_addr learned my addr 172.21.15.57:0/3627538840 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:22:13.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.556+0000 7f4a3e59c640 1 --2- 172.21.15.57:0/3627538840 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4a4006c0a0 0x7f4a4013aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:13.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.556+0000 7f4a3ed9d640 1 --2- 172.21.15.57:0/3627538840 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4a40141690 0x7f4a4013d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:13.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.556+0000 7f4a35d9b640 1 -- 172.21.15.57:0/3627538840 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4a40141690 msgr2=0x7f4a4013d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:13.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.556+0000 7f4a35d9b640 1 --2- 172.21.15.57:0/3627538840 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4a40141690 0x7f4a4013d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:13.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.556+0000 7f4a35d9b640 1 -- 172.21.15.57:0/3627538840 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4a4006c0a0 msgr2=0x7f4a4013aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:13.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.556+0000 7f4a35d9b640 1 --2- 172.21.15.57:0/3627538840 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4a4006c0a0 0x7f4a4013aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:13.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.556+0000 7f4a35d9b640 1 -- 172.21.15.57:0/3627538840 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a30009560 con 0x7f4a4013e520 2023-10-19T14:22:13.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.557+0000 7f4a35d9b640 1 --2- 172.21.15.57:0/3627538840 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a4013e520 0x7f4a4013d1c0 secure :-1 s=READY pgs=2917 cs=0 l=1 rev1=1 crypto rx=0x7f4a28009780 tx=0x7f4a28004340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:13.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.557+0000 7f4a377fe640 1 -- 172.21.15.57:0/3627538840 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a28003a00 con 0x7f4a4013e520 2023-10-19T14:22:13.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.557+0000 7f4a377fe640 1 -- 172.21.15.57:0/3627538840 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f4a28003c70 con 0x7f4a4013e520 2023-10-19T14:22:13.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.557+0000 7f4a44b8e640 1 -- 172.21.15.57:0/3627538840 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a28009490 con 0x7f4a4013e520 2023-10-19T14:22:13.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.557+0000 7f4a377fe640 1 -- 172.21.15.57:0/3627538840 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a28034440 con 0x7f4a4013e520 2023-10-19T14:22:13.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.557+0000 7f4a44b8e640 1 -- 172.21.15.57:0/3627538840 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a400b1320 con 0x7f4a4013e520 2023-10-19T14:22:13.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.558+0000 7f4a44b8e640 1 -- 172.21.15.57:0/3627538840 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a401e0eb0 con 0x7f4a4013e520 2023-10-19T14:22:13.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.558+0000 7f4a377fe640 1 -- 172.21.15.57:0/3627538840 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 145) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f4a2803b040 con 0x7f4a4013e520 2023-10-19T14:22:13.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.559+0000 7f4a377fe640 1 --2- 172.21.15.57:0/3627538840 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f4a2006b080 0x7f4a2006d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:13.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.559+0000 7f4a377fe640 1 -- 172.21.15.57:0/3627538840 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(263..263 src has 1..263) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f4a280abcd0 con 0x7f4a4013e520 2023-10-19T14:22:13.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.559+0000 7f4a3e59c640 1 --2- 172.21.15.57:0/3627538840 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f4a2006b080 0x7f4a2006d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:13.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.559+0000 7f4a3e59c640 1 --2- 172.21.15.57:0/3627538840 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f4a2006b080 0x7f4a2006d540 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f4a400ca2f0 tx=0x7f4a30009260 comp rx=0 tx=0).ready entity=mgr.11446 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:13.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.563+0000 7f4a377fe640 1 -- 172.21.15.57:0/3627538840 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4a28076830 con 0x7f4a4013e520 2023-10-19T14:22:13.792 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:13.789+0000 7f4a44b8e640 1 -- 172.21.15.57:0/3627538840 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f4a401e0570 con 0x7f4a4013e520 2023-10-19T14:22:14.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.482+0000 7f4a377fe640 1 -- 172.21.15.57:0/3627538840 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v146) v1 ==== 92+0+0 (secure 0 0 0) 0x7f4a401e0570 con 0x7f4a4013e520 2023-10-19T14:22:14.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.486+0000 7f4a44b8e640 1 -- 172.21.15.57:0/3627538840 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f4a2006b080 msgr2=0x7f4a2006d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:14.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.486+0000 7f4a44b8e640 1 --2- 172.21.15.57:0/3627538840 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f4a2006b080 0x7f4a2006d540 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f4a400ca2f0 tx=0x7f4a30009260 comp rx=0 tx=0).stop 2023-10-19T14:22:14.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.486+0000 7f4a44b8e640 1 -- 172.21.15.57:0/3627538840 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a4013e520 msgr2=0x7f4a4013d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:14.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.486+0000 7f4a44b8e640 1 --2- 172.21.15.57:0/3627538840 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a4013e520 0x7f4a4013d1c0 secure :-1 s=READY pgs=2917 cs=0 l=1 rev1=1 crypto rx=0x7f4a28009780 tx=0x7f4a28004340 comp rx=0 tx=0).stop 2023-10-19T14:22:14.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.486+0000 7f4a44b8e640 1 -- 172.21.15.57:0/3627538840 shutdown_connections 2023-10-19T14:22:14.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.486+0000 7f4a44b8e640 1 --2- 172.21.15.57:0/3627538840 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f4a2006b080 0x7f4a2006d540 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:14.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.486+0000 7f4a44b8e640 1 --2- 172.21.15.57:0/3627538840 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4a40141690 0x7f4a4013d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:14.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.486+0000 7f4a44b8e640 1 --2- 172.21.15.57:0/3627538840 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a4013e520 0x7f4a4013d1c0 unknown :-1 s=CLOSED pgs=2917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:14.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.486+0000 7f4a44b8e640 1 --2- 172.21.15.57:0/3627538840 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4a4006c0a0 0x7f4a4013aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:14.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.486+0000 7f4a44b8e640 1 -- 172.21.15.57:0/3627538840 >> 172.21.15.57:0/3627538840 conn(0x7f4a40069930 msgr2=0x7f4a40069d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:14.491 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.487+0000 7f4a44b8e640 1 -- 172.21.15.57:0/3627538840 shutdown_connections 2023-10-19T14:22:14.492 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.487+0000 7f4a44b8e640 1 -- 172.21.15.57:0/3627538840 wait complete. 2023-10-19T14:22:14.507 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-19T14:22:14.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.625+0000 7fdf9b610640 1 -- 172.21.15.57:0/2975764027 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdf940cf340 msgr2=0x7fdf940d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:14.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.625+0000 7fdf9b610640 1 --2- 172.21.15.57:0/2975764027 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdf940cf340 0x7fdf940d1910 secure :-1 s=READY pgs=2920 cs=0 l=1 rev1=1 crypto rx=0x7fdf84005fd0 tx=0x7fdf8402be20 comp rx=0 tx=0).stop 2023-10-19T14:22:14.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.625+0000 7fdf9b610640 1 -- 172.21.15.57:0/2975764027 shutdown_connections 2023-10-19T14:22:14.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.625+0000 7fdf9b610640 1 --2- 172.21.15.57:0/2975764027 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdf940cf340 0x7fdf940d1910 unknown :-1 s=CLOSED pgs=2920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:14.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.625+0000 7fdf9b610640 1 -- 172.21.15.57:0/2975764027 >> 172.21.15.57:0/2975764027 conn(0x7fdf940d0d60 msgr2=0x7fdf940d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:14.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.625+0000 7fdf9b610640 1 -- 172.21.15.57:0/2975764027 shutdown_connections 2023-10-19T14:22:14.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.625+0000 7fdf9b610640 1 -- 172.21.15.57:0/2975764027 wait complete. 2023-10-19T14:22:14.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.626+0000 7fdf9b610640 1 Processor -- start 2023-10-19T14:22:14.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.626+0000 7fdf9b610640 1 -- start start 2023-10-19T14:22:14.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.627+0000 7fdf9b610640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdf940ceb40 0x7fdf94159630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:14.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.627+0000 7fdf9b610640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdf940cf340 0x7fdf94156ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:14.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.627+0000 7fdf9b610640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdf9415b2a0 0x7fdf940b2850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:14.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.627+0000 7fdf9b610640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf94158900 con 0x7fdf940cf340 2023-10-19T14:22:14.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.627+0000 7fdf9b610640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf941591a0 con 0x7fdf940ceb40 2023-10-19T14:22:14.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.627+0000 7fdf9b610640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf940c8cf0 con 0x7fdf9415b2a0 2023-10-19T14:22:14.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.627+0000 7fdf98bab640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdf940cf340 0x7fdf94156ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:14.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.627+0000 7fdf98bab640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdf940cf340 0x7fdf94156ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50182/0 (socket says 172.21.15.57:50182) 2023-10-19T14:22:14.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.627+0000 7fdf98bab640 1 -- 172.21.15.57:0/1759326077 learned_addr learned my addr 172.21.15.57:0/1759326077 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:22:14.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.627+0000 7fdf993ac640 1 --2- 172.21.15.57:0/1759326077 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdf940ceb40 0x7fdf94159630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:14.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.627+0000 7fdf99bad640 1 --2- 172.21.15.57:0/1759326077 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdf9415b2a0 0x7fdf940b2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:14.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.627+0000 7fdf98bab640 1 -- 172.21.15.57:0/1759326077 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdf9415b2a0 msgr2=0x7fdf940b2850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:14.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.627+0000 7fdf98bab640 1 --2- 172.21.15.57:0/1759326077 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdf9415b2a0 0x7fdf940b2850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:14.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.627+0000 7fdf98bab640 1 -- 172.21.15.57:0/1759326077 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdf940ceb40 msgr2=0x7fdf94159630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:14.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.627+0000 7fdf98bab640 1 --2- 172.21.15.57:0/1759326077 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdf940ceb40 0x7fdf94159630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:14.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.627+0000 7fdf98bab640 1 -- 172.21.15.57:0/1759326077 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf84009560 con 0x7fdf940cf340 2023-10-19T14:22:14.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.627+0000 7fdf98bab640 1 --2- 172.21.15.57:0/1759326077 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdf940cf340 0x7fdf94156ba0 secure :-1 s=READY pgs=2921 cs=0 l=1 rev1=1 crypto rx=0x7fdf9000c3f0 tx=0x7fdf9000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:14.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.628+0000 7fdf89ffb640 1 -- 172.21.15.57:0/1759326077 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf900052d0 con 0x7fdf940cf340 2023-10-19T14:22:14.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.628+0000 7fdf89ffb640 1 -- 172.21.15.57:0/1759326077 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fdf90007020 con 0x7fdf940cf340 2023-10-19T14:22:14.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.628+0000 7fdf9b610640 1 -- 172.21.15.57:0/1759326077 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf94158b50 con 0x7fdf940cf340 2023-10-19T14:22:14.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.628+0000 7fdf89ffb640 1 -- 172.21.15.57:0/1759326077 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf90007430 con 0x7fdf940cf340 2023-10-19T14:22:14.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.628+0000 7fdf9b610640 1 -- 172.21.15.57:0/1759326077 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf941570e0 con 0x7fdf940cf340 2023-10-19T14:22:14.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.629+0000 7fdf9b610640 1 -- 172.21.15.57:0/1759326077 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf940690e0 con 0x7fdf940cf340 2023-10-19T14:22:14.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.629+0000 7fdf89ffb640 1 -- 172.21.15.57:0/1759326077 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 146) v1 ==== 89450+0+0 (secure 0 0 0) 0x7fdf90007590 con 0x7fdf940cf340 2023-10-19T14:22:14.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.630+0000 7fdf89ffb640 1 --2- 172.21.15.57:0/1759326077 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fdf7806b0d0 0x7fdf7806d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:14.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.630+0000 7fdf89ffb640 1 -- 172.21.15.57:0/1759326077 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(263..263 src has 1..263) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fdf900894f0 con 0x7fdf940cf340 2023-10-19T14:22:14.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.630+0000 7fdf993ac640 1 -- 172.21.15.57:0/1759326077 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fdf7806b0d0 msgr2=0x7fdf7806d590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/1573806521 2023-10-19T14:22:14.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.630+0000 7fdf993ac640 1 --2- 172.21.15.57:0/1759326077 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fdf7806b0d0 0x7fdf7806d590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T14:22:14.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.634+0000 7fdf89ffb640 1 -- 172.21.15.57:0/1759326077 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fdf90054340 con 0x7fdf940cf340 2023-10-19T14:22:14.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.814+0000 7fdf9b610640 1 -- 172.21.15.57:0/1759326077 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7fdf940d0b40 con 0x7fdf940cf340 2023-10-19T14:22:14.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.821+0000 7fdf89ffb640 1 -- 172.21.15.57:0/1759326077 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v84)=0 v84) v1 ==== 138+0+0 (secure 0 0 0) 0x7fdf90057960 con 0x7fdf940cf340 2023-10-19T14:22:14.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.823+0000 7fdf9b610640 1 -- 172.21.15.57:0/1759326077 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fdf7806b0d0 msgr2=0x7fdf7806d590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T14:22:14.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.823+0000 7fdf9b610640 1 --2- 172.21.15.57:0/1759326077 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fdf7806b0d0 0x7fdf7806d590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:14.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.823+0000 7fdf9b610640 1 -- 172.21.15.57:0/1759326077 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdf940cf340 msgr2=0x7fdf94156ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:14.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.823+0000 7fdf9b610640 1 --2- 172.21.15.57:0/1759326077 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdf940cf340 0x7fdf94156ba0 secure :-1 s=READY pgs=2921 cs=0 l=1 rev1=1 crypto rx=0x7fdf9000c3f0 tx=0x7fdf9000c8b0 comp rx=0 tx=0).stop 2023-10-19T14:22:14.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.823+0000 7fdf9b610640 1 -- 172.21.15.57:0/1759326077 shutdown_connections 2023-10-19T14:22:14.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.823+0000 7fdf9b610640 1 --2- 172.21.15.57:0/1759326077 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7fdf7806b0d0 0x7fdf7806d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:14.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.823+0000 7fdf9b610640 1 --2- 172.21.15.57:0/1759326077 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdf9415b2a0 0x7fdf940b2850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:14.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.823+0000 7fdf9b610640 1 --2- 172.21.15.57:0/1759326077 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdf940cf340 0x7fdf94156ba0 unknown :-1 s=CLOSED pgs=2921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:14.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.823+0000 7fdf9b610640 1 --2- 172.21.15.57:0/1759326077 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdf940ceb40 0x7fdf94159630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:14.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.823+0000 7fdf9b610640 1 -- 172.21.15.57:0/1759326077 >> 172.21.15.57:0/1759326077 conn(0x7fdf940d0d60 msgr2=0x7fdf940b1d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:14.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.823+0000 7fdf9b610640 1 -- 172.21.15.57:0/1759326077 shutdown_connections 2023-10-19T14:22:14.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.823+0000 7fdf9b610640 1 -- 172.21.15.57:0/1759326077 wait complete. 2023-10-19T14:22:14.837 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-19T14:22:14.875 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:22:14.872+0000 7f519c42c040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-19T14:22:14.907 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:22:14.904+0000 7f5b0620e040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-19T14:22:14.919 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:22:14.917+0000 7f519c42c040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-19T14:22:14.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.947+0000 7f9467883640 1 -- 172.21.15.57:0/289376129 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f946015b640 msgr2=0x7f94600cbf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:14.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.947+0000 7f9467883640 1 --2- 172.21.15.57:0/289376129 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f946015b640 0x7f94600cbf20 secure :-1 s=READY pgs=2923 cs=0 l=1 rev1=1 crypto rx=0x7f94600895b0 tx=0x7f945802c2f0 comp rx=0 tx=0).stop 2023-10-19T14:22:14.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.947+0000 7f9467883640 1 -- 172.21.15.57:0/289376129 shutdown_connections 2023-10-19T14:22:14.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.947+0000 7f9467883640 1 --2- 172.21.15.57:0/289376129 >> v2:172.21.15.179:3300/0 conn(0x7f94600cfa20 0x7f94600cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:14.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.947+0000 7f9467883640 1 --2- 172.21.15.57:0/289376129 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f946015b640 0x7f94600cbf20 unknown :-1 s=CLOSED pgs=2923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:14.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.947+0000 7f9467883640 1 -- 172.21.15.57:0/289376129 >> 172.21.15.57:0/289376129 conn(0x7f946009a030 msgr2=0x7f94600d0d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:14.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.947+0000 7f9467883640 1 -- 172.21.15.57:0/289376129 shutdown_connections 2023-10-19T14:22:14.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.947+0000 7f9467883640 1 -- 172.21.15.57:0/289376129 wait complete. 2023-10-19T14:22:14.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.949+0000 7f9467883640 1 Processor -- start 2023-10-19T14:22:14.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.949+0000 7f9467883640 1 -- start start 2023-10-19T14:22:14.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.949+0000 7f9467883640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f94600cfa20 0x7f94600b47a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:14.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.949+0000 7f9467883640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f946015b260 0x7f9460152460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:14.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.949+0000 7f9467883640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f946015b640 0x7f946015df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:14.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.949+0000 7f9467883640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f946015ba20 con 0x7f946015b260 2023-10-19T14:22:14.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.949+0000 7f9467883640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f946015c5a0 con 0x7f946015b640 2023-10-19T14:22:14.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.949+0000 7f9467883640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f946015bc70 con 0x7f94600cfa20 2023-10-19T14:22:14.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.949+0000 7f9464e1e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f946015b260 0x7f9460152460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:14.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.950+0000 7f9464e1e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f946015b260 0x7f9460152460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50206/0 (socket says 172.21.15.57:50206) 2023-10-19T14:22:14.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.950+0000 7f9465e20640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f946015b640 0x7f946015df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:14.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.950+0000 7f9464e1e640 1 -- 172.21.15.57:0/3505860001 learned_addr learned my addr 172.21.15.57:0/3505860001 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:22:14.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.950+0000 7f946561f640 1 --2- 172.21.15.57:0/3505860001 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f94600cfa20 0x7f94600b47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:14.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.950+0000 7f9464e1e640 1 -- 172.21.15.57:0/3505860001 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f94600cfa20 msgr2=0x7f94600b47a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:14.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.950+0000 7f9464e1e640 1 --2- 172.21.15.57:0/3505860001 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f94600cfa20 0x7f94600b47a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:14.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.950+0000 7f9464e1e640 1 -- 172.21.15.57:0/3505860001 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f946015b640 msgr2=0x7f946015df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:14.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.950+0000 7f9464e1e640 1 --2- 172.21.15.57:0/3505860001 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f946015b640 0x7f946015df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:14.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.950+0000 7f9464e1e640 1 -- 172.21.15.57:0/3505860001 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9458009900 con 0x7f946015b260 2023-10-19T14:22:14.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.950+0000 7f9465e20640 1 --2- 172.21.15.57:0/3505860001 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f946015b640 0x7f946015df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T14:22:14.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.950+0000 7f9464e1e640 1 --2- 172.21.15.57:0/3505860001 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f946015b260 0x7f9460152460 secure :-1 s=READY pgs=2924 cs=0 l=1 rev1=1 crypto rx=0x7f945c00ea20 tx=0x7f945c00eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:14.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.950+0000 7f9455ffb640 1 -- 172.21.15.57:0/3505860001 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f945c009700 con 0x7f946015b260 2023-10-19T14:22:14.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.950+0000 7f9455ffb640 1 -- 172.21.15.57:0/3505860001 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f945c00ce70 con 0x7f946015b260 2023-10-19T14:22:14.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.950+0000 7f9455ffb640 1 -- 172.21.15.57:0/3505860001 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f945c010690 con 0x7f946015b260 2023-10-19T14:22:14.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.950+0000 7f9467883640 1 -- 172.21.15.57:0/3505860001 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f946015bf50 con 0x7f946015b260 2023-10-19T14:22:14.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.951+0000 7f9467883640 1 -- 172.21.15.57:0/3505860001 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94600b4ce0 con 0x7f946015b260 2023-10-19T14:22:14.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.952+0000 7f9467883640 1 -- 172.21.15.57:0/3505860001 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f946015adc0 con 0x7f946015b260 2023-10-19T14:22:14.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.952+0000 7f9455ffb640 1 -- 172.21.15.57:0/3505860001 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 146) v1 ==== 89450+0+0 (secure 0 0 0) 0x7f945c0107f0 con 0x7f946015b260 2023-10-19T14:22:14.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.952+0000 7f9455ffb640 1 --2- 172.21.15.57:0/3505860001 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f944406b100 0x7f944406d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:14.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.953+0000 7f946561f640 1 -- 172.21.15.57:0/3505860001 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f944406b100 msgr2=0x7f944406d5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/1573806521 2023-10-19T14:22:14.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.953+0000 7f9455ffb640 1 -- 172.21.15.57:0/3505860001 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(263..263 src has 1..263) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f945c014070 con 0x7f946015b260 2023-10-19T14:22:14.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.953+0000 7f946561f640 1 --2- 172.21.15.57:0/3505860001 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f944406b100 0x7f944406d5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T14:22:14.962 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:22:14.955+0000 7f5b0620e040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-19T14:22:14.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:14.956+0000 7f9455ffb640 1 -- 172.21.15.57:0/3505860001 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f945c051ca0 con 0x7f946015b260 2023-10-19T14:22:15.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:15.154+0000 7f946561f640 1 -- 172.21.15.57:0/3505860001 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f944406b100 msgr2=0x7f944406d5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/1573806521 2023-10-19T14:22:15.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:15.154+0000 7f946561f640 1 --2- 172.21.15.57:0/3505860001 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f944406b100 0x7f944406d5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-19T14:22:15.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:15.154+0000 7f9467883640 1 -- 172.21.15.57:0/3505860001 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f94600675f0 con 0x7f946015b260 2023-10-19T14:22:15.160 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:15.157+0000 7f9455ffb640 1 -- 172.21.15.57:0/3505860001 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v85)=0 v85) v1 ==== 133+0+0 (secure 0 0 0) 0x7f945c0553a0 con 0x7f946015b260 2023-10-19T14:22:15.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:15.160+0000 7f9467883640 1 -- 172.21.15.57:0/3505860001 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f944406b100 msgr2=0x7f944406d5c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T14:22:15.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:15.160+0000 7f9467883640 1 --2- 172.21.15.57:0/3505860001 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f944406b100 0x7f944406d5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:15.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:15.160+0000 7f9467883640 1 -- 172.21.15.57:0/3505860001 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f946015b260 msgr2=0x7f9460152460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:15.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:15.160+0000 7f9467883640 1 --2- 172.21.15.57:0/3505860001 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f946015b260 0x7f9460152460 secure :-1 s=READY pgs=2924 cs=0 l=1 rev1=1 crypto rx=0x7f945c00ea20 tx=0x7f945c00eee0 comp rx=0 tx=0).stop 2023-10-19T14:22:15.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:15.160+0000 7f9467883640 1 -- 172.21.15.57:0/3505860001 shutdown_connections 2023-10-19T14:22:15.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:15.160+0000 7f9467883640 1 --2- 172.21.15.57:0/3505860001 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f944406b100 0x7f944406d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:15.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:15.160+0000 7f9467883640 1 --2- 172.21.15.57:0/3505860001 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f946015b640 0x7f946015df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:15.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:15.160+0000 7f9467883640 1 --2- 172.21.15.57:0/3505860001 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f946015b260 0x7f9460152460 unknown :-1 s=CLOSED pgs=2924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:15.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:15.160+0000 7f9467883640 1 --2- 172.21.15.57:0/3505860001 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f94600cfa20 0x7f94600b47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:15.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:15.160+0000 7f9467883640 1 -- 172.21.15.57:0/3505860001 >> 172.21.15.57:0/3505860001 conn(0x7f946009a030 msgr2=0x7f94600ad990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:15.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:15.160+0000 7f9467883640 1 -- 172.21.15.57:0/3505860001 shutdown_connections 2023-10-19T14:22:15.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:15.160+0000 7f9467883640 1 -- 172.21.15.57:0/3505860001 wait complete. 2023-10-19T14:22:15.174 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:22:15.174 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-10-19T14:22:15.195 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 1m 2023-10-19T14:22:15.312 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:22:15.310+0000 7f519c42c040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-19T14:22:15.348 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:15.345+0000 7f909431d640 1 -- 172.21.15.57:0/1088223852 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f908c0ce420 msgr2=0x7f908c0cd6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:15.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:15.345+0000 7f909431d640 1 --2- 172.21.15.57:0/1088223852 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f908c0ce420 0x7f908c0cd6b0 secure :-1 s=READY pgs=2926 cs=0 l=1 rev1=1 crypto rx=0x7f9088009930 tx=0x7f908802be80 comp rx=0 tx=0).stop 2023-10-19T14:22:15.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:15.345+0000 7f909431d640 1 -- 172.21.15.57:0/1088223852 shutdown_connections 2023-10-19T14:22:15.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:15.345+0000 7f909431d640 1 --2- 172.21.15.57:0/1088223852 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f908c0ce420 0x7f908c0cd6b0 unknown :-1 s=CLOSED pgs=2926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:15.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:15.345+0000 7f909431d640 1 --2- 172.21.15.57:0/1088223852 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f908c15b180 0x7f908c0cf350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:15.349 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:15.345+0000 7f909431d640 1 -- 172.21.15.57:0/1088223852 >> 172.21.15.57:0/1088223852 conn(0x7f908c099ff0 msgr2=0x7f908c0d0d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:15.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:15.345+0000 7f909431d640 1 -- 172.21.15.57:0/1088223852 shutdown_connections 2023-10-19T14:22:15.350 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:15.345+0000 7f909431d640 1 -- 172.21.15.57:0/1088223852 wait complete. 2023-10-19T14:22:15.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:15.346+0000 7f909431d640 1 Processor -- start 2023-10-19T14:22:15.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:15.346+0000 7f909431d640 1 -- start start 2023-10-19T14:22:15.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:15.347+0000 7f909431d640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f908c0cdbf0 0x7f908c0b43a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:15.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:15.347+0000 7f909431d640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f908c0ce420 0x7f908c150e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:15.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:15.347+0000 7f909431d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f908c15b180 0x7f908c0b16b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:15.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:15.347+0000 7f909431d640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f908c155040 con 0x7f908c15b180 2023-10-19T14:22:15.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:15.347+0000 7f909431d640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f908c0b4ca0 con 0x7f908c0cdbf0 2023-10-19T14:22:15.353 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:15.347+0000 7f909431d640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f908c0c9250 con 0x7f908c0ce420 2023-10-19T14:22:15.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:15.347+0000 7f90928ba640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f908c15b180 0x7f908c0b16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:15.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:15.347+0000 7f90928ba640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f908c15b180 0x7f908c0b16b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50212/0 (socket says 172.21.15.57:50212) 2023-10-19T14:22:15.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:15.347+0000 7f90928ba640 1 -- 172.21.15.57:0/2685503104 learned_addr learned my addr 172.21.15.57:0/2685503104 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:22:15.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:15.347+0000 7f90918b8640 1 --2- 172.21.15.57:0/2685503104 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f908c0ce420 0x7f908c150e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:15.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:15.347+0000 7f90920b9640 1 --2- 172.21.15.57:0/2685503104 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f908c0cdbf0 0x7f908c0b43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:15.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:15.348+0000 7f90928ba640 1 -- 172.21.15.57:0/2685503104 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f908c0ce420 msgr2=0x7f908c150e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:15.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:15.348+0000 7f90928ba640 1 --2- 172.21.15.57:0/2685503104 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f908c0ce420 0x7f908c150e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:15.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:15.348+0000 7f90928ba640 1 -- 172.21.15.57:0/2685503104 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f908c0cdbf0 msgr2=0x7f908c0b43a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:15.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:15.348+0000 7f90928ba640 1 --2- 172.21.15.57:0/2685503104 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f908c0cdbf0 0x7f908c0b43a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:15.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:15.348+0000 7f90928ba640 1 -- 172.21.15.57:0/2685503104 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9088009560 con 0x7f908c15b180 2023-10-19T14:22:15.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:15.348+0000 7f90928ba640 1 --2- 172.21.15.57:0/2685503104 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f908c15b180 0x7f908c0b16b0 secure :-1 s=READY pgs=2927 cs=0 l=1 rev1=1 crypto rx=0x7f9084001560 tx=0x7f908400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:15.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:15.348+0000 7f9082ffd640 1 -- 172.21.15.57:0/2685503104 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9084003da0 con 0x7f908c15b180 2023-10-19T14:22:15.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:15.348+0000 7f9082ffd640 1 -- 172.21.15.57:0/2685503104 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f9084005070 con 0x7f908c15b180 2023-10-19T14:22:15.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:15.348+0000 7f909431d640 1 -- 172.21.15.57:0/2685503104 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f908c0c9530 con 0x7f908c15b180 2023-10-19T14:22:15.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:15.348+0000 7f9082ffd640 1 -- 172.21.15.57:0/2685503104 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9084005460 con 0x7f908c15b180 2023-10-19T14:22:15.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:15.348+0000 7f909431d640 1 -- 172.21.15.57:0/2685503104 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f908c0b4f80 con 0x7f908c15b180 2023-10-19T14:22:15.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:15.349+0000 7f909431d640 1 -- 172.21.15.57:0/2685503104 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f908c066960 con 0x7f908c15b180 2023-10-19T14:22:15.357 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:22:15.354+0000 7f519c42c040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-19T14:22:15.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:15.350+0000 7f9082ffd640 1 -- 172.21.15.57:0/2685503104 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 146) v1 ==== 89450+0+0 (secure 0 0 0) 0x7f90840055c0 con 0x7f908c15b180 2023-10-19T14:22:15.358 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:15.350+0000 7f9082ffd640 1 --2- 172.21.15.57:0/2685503104 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f907006b100 0x7f907006d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:15.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:15.350+0000 7f90920b9640 1 -- 172.21.15.57:0/2685503104 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f907006b100 msgr2=0x7f907006d5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/1573806521 2023-10-19T14:22:15.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:15.350+0000 7f9082ffd640 1 -- 172.21.15.57:0/2685503104 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(263..263 src has 1..263) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f9084089620 con 0x7f908c15b180 2023-10-19T14:22:15.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:15.350+0000 7f90920b9640 1 --2- 172.21.15.57:0/2685503104 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f907006b100 0x7f907006d5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T14:22:15.359 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:15.354+0000 7f9082ffd640 1 -- 172.21.15.57:0/2685503104 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f90840542d0 con 0x7f908c15b180 2023-10-19T14:22:15.397 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:22:15.394+0000 7f5b0620e040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-19T14:22:15.400 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:22:15.397+0000 7f519c42c040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-19T14:22:15.444 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:22:15.442+0000 7f5b0620e040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-19T14:22:15.452 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:22:15.450+0000 7f519c42c040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-19T14:22:15.490 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:22:15.488+0000 7f5b0620e040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-19T14:22:15.495 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:22:15.493+0000 7f519c42c040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-19T14:22:15.546 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:22:15.543+0000 7f5b0620e040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-19T14:22:15.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:15.551+0000 7f90920b9640 1 -- 172.21.15.57:0/2685503104 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f907006b100 msgr2=0x7f907006d5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/1573806521 2023-10-19T14:22:15.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:15.551+0000 7f90920b9640 1 --2- 172.21.15.57:0/2685503104 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f907006b100 0x7f907006d5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-19T14:22:15.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:15.553+0000 7f909431d640 1 -- 172.21.15.57:0/2685503104 --> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f908c0690e0 con 0x7f907006b100 2023-10-19T14:22:15.590 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:22:15.587+0000 7f5b0620e040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-19T14:22:15.593 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:22:15.591+0000 7f519c42c040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-19T14:22:15.646 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:22:15.644+0000 7f519c42c040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-19T14:22:15.688 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:22:15.685+0000 7f519c42c040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-19T14:22:15.695 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:22:15.692+0000 7f5b0620e040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-19T14:22:15.735 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:22:15.733+0000 7f519c42c040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-19T14:22:15.751 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:22:15.748+0000 7f5b0620e040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-19T14:22:15.777 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:22:15.775+0000 7f519c42c040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-19T14:22:15.796 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:22:15.793+0000 7f5b0620e040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-19T14:22:15.847 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:22:15.844+0000 7f5b0620e040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-19T14:22:15.892 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:22:15.889+0000 7f5b0620e040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-19T14:22:15.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:15.952+0000 7f90920b9640 1 -- 172.21.15.57:0/2685503104 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f907006b100 msgr2=0x7f907006d5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/1573806521 2023-10-19T14:22:15.955 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:15.952+0000 7f90920b9640 1 --2- 172.21.15.57:0/2685503104 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f907006b100 0x7f907006d5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-19T14:22:16.091 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:22:16.088+0000 7f519c42c040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-19T14:22:16.221 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:22:16.218+0000 7f519c42c040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-19T14:22:16.234 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:22:16.231+0000 7f5b0620e040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-19T14:22:16.313 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:22:16.310+0000 7f519c42c040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-19T14:22:16.355 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:22:16.352+0000 7f519c42c040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-19T14:22:16.373 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:22:16.370+0000 7f5b0620e040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-19T14:22:16.451 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:22:16.448+0000 7f519c42c040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-19T14:22:16.471 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:22:16.468+0000 7f5b0620e040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-19T14:22:16.516 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:22:16.513+0000 7f5b0620e040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-19T14:22:16.541 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:22:16.539+0000 7f519c42c040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-19T14:22:16.616 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:22:16.614+0000 7f5b0620e040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-19T14:22:16.676 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:22:16.673+0000 7f519c42c040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-19T14:22:16.713 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:22:16.710+0000 7f5b0620e040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-19T14:22:16.718 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:22:16.715+0000 7f519c42c040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-19T14:22:16.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:16.753+0000 7f90920b9640 1 -- 172.21.15.57:0/2685503104 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f907006b100 msgr2=0x7f907006d5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/1573806521 2023-10-19T14:22:16.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:16.753+0000 7f90920b9640 1 --2- 172.21.15.57:0/2685503104 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f907006b100 0x7f907006d5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-19T14:22:16.821 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:22:16.818+0000 7f519c42c040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-19T14:22:16.858 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:22:16.855+0000 7f5b0620e040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-19T14:22:16.902 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:22:16.899+0000 7f5b0620e040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-19T14:22:17.013 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:22:17.010+0000 7f5b0620e040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-19T14:22:17.444 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:22:17.441+0000 7f519c42c040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-19T14:22:17.675 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:22:17.672+0000 7f5b0620e040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-19T14:22:17.705 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:22:17.702+0000 7f519c42c040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-19T14:22:17.754 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:22:17.751+0000 7f519c42c040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-19T14:22:17.854 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:22:17.851+0000 7f519c42c040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-19T14:22:17.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:17.880+0000 7f9082ffd640 1 -- 172.21.15.57:0/2685503104 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mgrmap(e 147) v1 ==== 88957+0+0 (secure 0 0 0) 0x7f90840546a0 con 0x7f908c15b180 2023-10-19T14:22:17.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:17.880+0000 7f9082ffd640 1 -- 172.21.15.57:0/2685503104 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f907006b100 msgr2=0x7f907006d5c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T14:22:17.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:17.880+0000 7f9082ffd640 1 --2- 172.21.15.57:0/2685503104 >> [v2:172.21.15.179:6832/1573806521,v1:172.21.15.179:6833/1573806521] conn(0x7f907006b100 0x7f907006d5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:17.965 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:22:17.963+0000 7f5b0620e040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-19T14:22:17.981 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:22:17.978+0000 7f511b864640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:22:17.981 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:22:17.978+0000 7f511b864640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:22:17.982 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:22:17.978+0000 7f511b864640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:22:17.982 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:22:17.978+0000 7f511b864640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:22:17.982 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:22:17.978+0000 7f511b864640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:22:17.982 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:22:17.978+0000 7f511881e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:22:17.982 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:22:17.978+0000 7f511881e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:22:17.983 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:22:17.978+0000 7f511881e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:22:17.983 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:22:17.978+0000 7f511881e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:22:17.983 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:22:17.978+0000 7f511881e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:22:18.018 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:22:18.015+0000 7f5b0620e040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-19T14:22:18.124 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:22:18.122+0000 7f5b0620e040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-19T14:22:18.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:18.879+0000 7f9082ffd640 1 -- 172.21.15.57:0/2685503104 <== mon.0 v2:172.21.15.57:3300/0 8 ==== mgrmap(e 148) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f9084076bc0 con 0x7f908c15b180 2023-10-19T14:22:18.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:18.880+0000 7f9082ffd640 1 --2- 172.21.15.57:0/2685503104 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f9070035860 0x7f9070075530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:18.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:18.880+0000 7f9082ffd640 1 -- 172.21.15.57:0/2685503104 --> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f908c0690e0 con 0x7f9070035860 2023-10-19T14:22:18.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:18.883+0000 7f90920b9640 1 --2- 172.21.15.57:0/2685503104 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f9070035860 0x7f9070075530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:18.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:18.885+0000 7f90920b9640 1 --2- 172.21.15.57:0/2685503104 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f9070035860 0x7f9070075530 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f908c0c9140 tx=0x7f907c005f90 comp rx=0 tx=0).ready entity=mgr.10394 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:18.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:18.938+0000 7f9082ffd640 1 -- 172.21.15.57:0/2685503104 <== mgr.10394 v2:172.21.15.179:6832/2005622276 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f908c0690e0 con 0x7f9070035860 2023-10-19T14:22:18.942 INFO:teuthology.orchestra.run.smithi057.stdout:Schedule set for path snap_test_dir1 2023-10-19T14:22:18.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:18.941+0000 7f909431d640 1 -- 172.21.15.57:0/2685503104 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f9070035860 msgr2=0x7f9070075530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:18.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:18.941+0000 7f909431d640 1 --2- 172.21.15.57:0/2685503104 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f9070035860 0x7f9070075530 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f908c0c9140 tx=0x7f907c005f90 comp rx=0 tx=0).stop 2023-10-19T14:22:18.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:18.941+0000 7f909431d640 1 -- 172.21.15.57:0/2685503104 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f908c15b180 msgr2=0x7f908c0b16b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:18.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:18.941+0000 7f909431d640 1 --2- 172.21.15.57:0/2685503104 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f908c15b180 0x7f908c0b16b0 secure :-1 s=READY pgs=2927 cs=0 l=1 rev1=1 crypto rx=0x7f9084001560 tx=0x7f908400cc70 comp rx=0 tx=0).stop 2023-10-19T14:22:18.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:18.941+0000 7f90928ba640 1 -- 172.21.15.57:0/2685503104 reap_dead start 2023-10-19T14:22:18.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:18.941+0000 7f909431d640 1 -- 172.21.15.57:0/2685503104 shutdown_connections 2023-10-19T14:22:18.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:18.941+0000 7f909431d640 1 -- 172.21.15.57:0/2685503104 >> 172.21.15.57:0/2685503104 conn(0x7f908c099ff0 msgr2=0x7f908c0cc3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:18.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:18.941+0000 7f909431d640 1 -- 172.21.15.57:0/2685503104 shutdown_connections 2023-10-19T14:22:18.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:18.941+0000 7f909431d640 1 -- 172.21.15.57:0/2685503104 wait complete. 2023-10-19T14:22:18.956 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2023-10-19T14:22:18.956 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention add snap_test_dir1 1m 2023-10-19T14:22:19.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.068+0000 7ff21a458640 1 -- 172.21.15.57:0/2368152417 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff1fc0039a0 msgr2=0x7ff1fc003d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:19.072 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.068+0000 7ff21a458640 1 --2- 172.21.15.57:0/2368152417 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff1fc0039a0 0x7ff1fc003d80 secure :-1 s=READY pgs=2929 cs=0 l=1 rev1=1 crypto rx=0x7ff208006f70 tx=0x7ff1fc0057f0 comp rx=0 tx=0).stop 2023-10-19T14:22:19.072 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.068+0000 7ff2189f5640 1 -- 172.21.15.57:0/2368152417 reap_dead start 2023-10-19T14:22:19.072 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.068+0000 7ff21a458640 1 -- 172.21.15.57:0/2368152417 shutdown_connections 2023-10-19T14:22:19.072 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.068+0000 7ff21a458640 1 -- 172.21.15.57:0/2368152417 >> 172.21.15.57:0/2368152417 conn(0x7ff214069930 msgr2=0x7ff214069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:19.073 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.068+0000 7ff21a458640 1 -- 172.21.15.57:0/2368152417 shutdown_connections 2023-10-19T14:22:19.073 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.068+0000 7ff21a458640 1 -- 172.21.15.57:0/2368152417 wait complete. 2023-10-19T14:22:19.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.069+0000 7ff21a458640 1 Processor -- start 2023-10-19T14:22:19.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.070+0000 7ff21a458640 1 -- start start 2023-10-19T14:22:19.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.070+0000 7ff21a458640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff1fc0039a0 0x7ff21413adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:19.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.070+0000 7ff21a458640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff21413e990 0x7ff21413cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:19.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.070+0000 7ff21a458640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff21413d0f0 0x7ff21412bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:19.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.070+0000 7ff21a458640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2140ab020 con 0x7ff21413e990 2023-10-19T14:22:19.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.070+0000 7ff21a458640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff214152db0 con 0x7ff1fc0039a0 2023-10-19T14:22:19.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.070+0000 7ff21a458640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff21413d670 con 0x7ff21413d0f0 2023-10-19T14:22:19.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.070+0000 7ff2137fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff21413e990 0x7ff21413cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:19.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.070+0000 7ff2137fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff21413e990 0x7ff21413cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46392/0 (socket says 172.21.15.57:46392) 2023-10-19T14:22:19.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.070+0000 7ff2137fe640 1 -- 172.21.15.57:0/3265463102 learned_addr learned my addr 172.21.15.57:0/3265463102 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:22:19.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.071+0000 7ff213fff640 1 --2- 172.21.15.57:0/3265463102 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff1fc0039a0 0x7ff21413adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:19.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.071+0000 7ff2189f5640 1 --2- 172.21.15.57:0/3265463102 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff21413d0f0 0x7ff21412bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:19.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.071+0000 7ff2137fe640 1 -- 172.21.15.57:0/3265463102 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff21413d0f0 msgr2=0x7ff21412bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:19.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.071+0000 7ff2137fe640 1 --2- 172.21.15.57:0/3265463102 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff21413d0f0 0x7ff21412bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:19.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.071+0000 7ff2137fe640 1 -- 172.21.15.57:0/3265463102 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff1fc0039a0 msgr2=0x7ff21413adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:19.078 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.071+0000 7ff2137fe640 1 --2- 172.21.15.57:0/3265463102 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff1fc0039a0 0x7ff21413adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:19.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.071+0000 7ff2137fe640 1 -- 172.21.15.57:0/3265463102 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2140aa960 con 0x7ff21413e990 2023-10-19T14:22:19.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.071+0000 7ff2137fe640 1 --2- 172.21.15.57:0/3265463102 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff21413e990 0x7ff21413cbb0 secure :-1 s=READY pgs=2930 cs=0 l=1 rev1=1 crypto rx=0x7ff204003ec0 tx=0x7ff204001760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:19.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.071+0000 7ff210ff9640 1 -- 172.21.15.57:0/3265463102 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff204011900 con 0x7ff21413e990 2023-10-19T14:22:19.079 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.071+0000 7ff210ff9640 1 -- 172.21.15.57:0/3265463102 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7ff204011a80 con 0x7ff21413e990 2023-10-19T14:22:19.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.071+0000 7ff21a458640 1 -- 172.21.15.57:0/3265463102 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2140aac40 con 0x7ff21413e990 2023-10-19T14:22:19.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.071+0000 7ff210ff9640 1 -- 172.21.15.57:0/3265463102 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff204011e90 con 0x7ff21413e990 2023-10-19T14:22:19.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.071+0000 7ff21a458640 1 -- 172.21.15.57:0/3265463102 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff21413d950 con 0x7ff21413e990 2023-10-19T14:22:19.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.072+0000 7ff21a458640 1 -- 172.21.15.57:0/3265463102 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2140ab160 con 0x7ff21413e990 2023-10-19T14:22:19.080 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.072+0000 7ff210ff9640 1 -- 172.21.15.57:0/3265463102 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 148) v1 ==== 89084+0+0 (secure 0 0 0) 0x7ff2040103c0 con 0x7ff21413e990 2023-10-19T14:22:19.082 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.073+0000 7ff210ff9640 1 --2- 172.21.15.57:0/3265463102 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7ff20006ad10 0x7ff20006d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:19.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.073+0000 7ff210ff9640 1 -- 172.21.15.57:0/3265463102 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(264..264 src has 1..264) v4 ==== 6487+0+0 (secure 0 0 0) 0x7ff20408a620 con 0x7ff21413e990 2023-10-19T14:22:19.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.073+0000 7ff213fff640 1 --2- 172.21.15.57:0/3265463102 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7ff20006ad10 0x7ff20006d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:19.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.074+0000 7ff213fff640 1 --2- 172.21.15.57:0/3265463102 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7ff20006ad10 0x7ff20006d1d0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff208034000 tx=0x7ff208039040 comp rx=0 tx=0).ready entity=mgr.10394 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:19.083 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.077+0000 7ff210ff9640 1 -- 172.21.15.57:0/3265463102 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff2140ab160 con 0x7ff21413e990 2023-10-19T14:22:19.276 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.274+0000 7ff21a458640 1 -- 172.21.15.57:0/3265463102 --> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "path": "snap_test_dir1", "retention_spec_or_period": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7ff2140b12b0 con 0x7ff20006ad10 2023-10-19T14:22:19.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.323+0000 7ff210ff9640 1 -- 172.21.15.57:0/3265463102 <== mgr.10394 v2:172.21.15.179:6832/2005622276 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+39 (secure 0 0 0) 0x7ff20803d040 con 0x7ff20006ad10 2023-10-19T14:22:19.326 INFO:teuthology.orchestra.run.smithi057.stdout:Retention added to path snap_test_dir1 2023-10-19T14:22:19.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.325+0000 7ff21a458640 1 -- 172.21.15.57:0/3265463102 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7ff20006ad10 msgr2=0x7ff20006d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:19.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.325+0000 7ff21a458640 1 --2- 172.21.15.57:0/3265463102 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7ff20006ad10 0x7ff20006d1d0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff208034000 tx=0x7ff208039040 comp rx=0 tx=0).stop 2023-10-19T14:22:19.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.325+0000 7ff21a458640 1 -- 172.21.15.57:0/3265463102 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff21413e990 msgr2=0x7ff21413cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:19.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.325+0000 7ff21a458640 1 --2- 172.21.15.57:0/3265463102 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff21413e990 0x7ff21413cbb0 secure :-1 s=READY pgs=2930 cs=0 l=1 rev1=1 crypto rx=0x7ff204003ec0 tx=0x7ff204001760 comp rx=0 tx=0).stop 2023-10-19T14:22:19.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.325+0000 7ff21a458640 1 -- 172.21.15.57:0/3265463102 shutdown_connections 2023-10-19T14:22:19.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.325+0000 7ff21a458640 1 --2- 172.21.15.57:0/3265463102 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7ff20006ad10 0x7ff20006d1d0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:19.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.325+0000 7ff21a458640 1 --2- 172.21.15.57:0/3265463102 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff21413d0f0 0x7ff21412bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:19.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.325+0000 7ff21a458640 1 --2- 172.21.15.57:0/3265463102 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff21413e990 0x7ff21413cbb0 unknown :-1 s=CLOSED pgs=2930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:19.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.325+0000 7ff21a458640 1 --2- 172.21.15.57:0/3265463102 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff1fc0039a0 0x7ff21413adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:19.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.325+0000 7ff21a458640 1 -- 172.21.15.57:0/3265463102 >> 172.21.15.57:0/3265463102 conn(0x7ff214069930 msgr2=0x7ff2140ac670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:19.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.325+0000 7ff21a458640 1 -- 172.21.15.57:0/3265463102 shutdown_connections 2023-10-19T14:22:19.330 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.325+0000 7ff21a458640 1 -- 172.21.15.57:0/3265463102 wait complete. 2023-10-19T14:22:19.339 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention added to path snap_test_dir1 2023-10-19T14:22:19.343 DEBUG:teuthology.orchestra.run.smithi057:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-19T14:22:19.346 DEBUG:teuthology.orchestra.run.smithi179:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-19T14:22:19.399 INFO:teuthology.orchestra.run.smithi179.stdout:[global] 2023-10-19T14:22:19.400 INFO:teuthology.orchestra.run.smithi179.stdout: chdir = "" 2023-10-19T14:22:19.400 INFO:teuthology.orchestra.run.smithi179.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-19T14:22:19.400 INFO:teuthology.orchestra.run.smithi179.stdout: auth supported = cephx 2023-10-19T14:22:19.400 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-10-19T14:22:19.401 INFO:teuthology.orchestra.run.smithi179.stdout: filestore xattr use omap = true 2023-10-19T14:22:19.401 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-10-19T14:22:19.401 INFO:teuthology.orchestra.run.smithi179.stdout: mon clock drift allowed = 1.000 2023-10-19T14:22:19.401 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-10-19T14:22:19.401 INFO:teuthology.orchestra.run.smithi179.stdout: osd crush chooseleaf type = 0 2023-10-19T14:22:19.402 INFO:teuthology.orchestra.run.smithi179.stdout: auth debug = true 2023-10-19T14:22:19.402 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-10-19T14:22:19.402 INFO:teuthology.orchestra.run.smithi179.stdout: ms die on old message = true 2023-10-19T14:22:19.402 INFO:teuthology.orchestra.run.smithi179.stdout: ms die on bug = true 2023-10-19T14:22:19.402 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-10-19T14:22:19.403 INFO:teuthology.orchestra.run.smithi179.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-19T14:22:19.403 INFO:teuthology.orchestra.run.smithi179.stdout: mon pg warn max object skew = 0 2023-10-19T14:22:19.403 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-10-19T14:22:19.403 INFO:teuthology.orchestra.run.smithi179.stdout: # disable pg_autoscaler by default for new pools 2023-10-19T14:22:19.403 INFO:teuthology.orchestra.run.smithi179.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-19T14:22:19.404 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-10-19T14:22:19.404 INFO:teuthology.orchestra.run.smithi179.stdout: osd pool default size = 2 2023-10-19T14:22:19.404 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-10-19T14:22:19.404 INFO:teuthology.orchestra.run.smithi179.stdout: mon osd allow primary affinity = true 2023-10-19T14:22:19.404 INFO:teuthology.orchestra.run.smithi179.stdout: mon osd allow pg remap = true 2023-10-19T14:22:19.405 INFO:teuthology.orchestra.run.smithi179.stdout: mon warn on legacy crush tunables = false 2023-10-19T14:22:19.405 INFO:teuthology.orchestra.run.smithi179.stdout: mon warn on crush straw calc version zero = false 2023-10-19T14:22:19.405 INFO:teuthology.orchestra.run.smithi179.stdout: mon warn on no sortbitwise = false 2023-10-19T14:22:19.405 INFO:teuthology.orchestra.run.smithi179.stdout: mon warn on osd down out interval zero = false 2023-10-19T14:22:19.405 INFO:teuthology.orchestra.run.smithi179.stdout: mon warn on too few osds = false 2023-10-19T14:22:19.406 INFO:teuthology.orchestra.run.smithi179.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-19T14:22:19.406 INFO:teuthology.orchestra.run.smithi179.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-19T14:22:19.406 INFO:teuthology.orchestra.run.smithi179.stdout: mon_allow_pool_size_one = true 2023-10-19T14:22:19.406 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-10-19T14:22:19.406 INFO:teuthology.orchestra.run.smithi179.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-19T14:22:19.406 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-10-19T14:22:19.407 INFO:teuthology.orchestra.run.smithi179.stdout: osd default data pool replay window = 5 2023-10-19T14:22:19.407 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-10-19T14:22:19.407 INFO:teuthology.orchestra.run.smithi179.stdout: mon allow pool delete = true 2023-10-19T14:22:19.407 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-10-19T14:22:19.407 INFO:teuthology.orchestra.run.smithi179.stdout: mon cluster log file level = debug 2023-10-19T14:22:19.408 INFO:teuthology.orchestra.run.smithi179.stdout: debug asserts on shutdown = true 2023-10-19T14:22:19.408 INFO:teuthology.orchestra.run.smithi179.stdout: mon health detail to clog = false 2023-10-19T14:22:19.408 INFO:teuthology.orchestra.run.smithi179.stdout: mon host = "172.21.15.57,172.21.15.179,[v2:172.21.15.179:3301,v1:172.21.15.179:6790]" 2023-10-19T14:22:19.408 INFO:teuthology.orchestra.run.smithi179.stdout: fsid = 1e9beb19-3260-4dd7-85ef-5b399ce774ac 2023-10-19T14:22:19.408 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-10-19T14:22:19.409 INFO:teuthology.orchestra.run.smithi179.stdout:[osd] 2023-10-19T14:22:19.409 INFO:teuthology.orchestra.run.smithi179.stdout: osd journal size = 100 2023-10-19T14:22:19.409 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-10-19T14:22:19.409 INFO:teuthology.orchestra.run.smithi179.stdout: osd scrub load threshold = 5.0 2023-10-19T14:22:19.409 INFO:teuthology.orchestra.run.smithi179.stdout: osd scrub max interval = 600 2023-10-19T14:22:19.409 INFO:teuthology.orchestra.run.smithi179.stdout: osd mclock profile = high_recovery_ops 2023-10-19T14:22:19.410 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-10-19T14:22:19.410 INFO:teuthology.orchestra.run.smithi179.stdout: osd recover clone overlap = true 2023-10-19T14:22:19.410 INFO:teuthology.orchestra.run.smithi179.stdout: osd recovery max chunk = 1048576 2023-10-19T14:22:19.410 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-10-19T14:22:19.410 INFO:teuthology.orchestra.run.smithi179.stdout: osd debug shutdown = true 2023-10-19T14:22:19.411 INFO:teuthology.orchestra.run.smithi179.stdout: osd debug op order = true 2023-10-19T14:22:19.411 INFO:teuthology.orchestra.run.smithi179.stdout: osd debug verify stray on activate = true 2023-10-19T14:22:19.411 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-10-19T14:22:19.411 INFO:teuthology.orchestra.run.smithi179.stdout: osd open classes on start = true 2023-10-19T14:22:19.411 INFO:teuthology.orchestra.run.smithi179.stdout: osd debug pg log writeout = true 2023-10-19T14:22:19.412 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-10-19T14:22:19.412 INFO:teuthology.orchestra.run.smithi179.stdout: osd deep scrub update digest min age = 30 2023-10-19T14:22:19.412 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-10-19T14:22:19.412 INFO:teuthology.orchestra.run.smithi179.stdout: osd map max advance = 10 2023-10-19T14:22:19.412 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-10-19T14:22:19.413 INFO:teuthology.orchestra.run.smithi179.stdout: journal zero on create = true 2023-10-19T14:22:19.413 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-10-19T14:22:19.413 INFO:teuthology.orchestra.run.smithi179.stdout: filestore ondisk finisher threads = 3 2023-10-19T14:22:19.413 INFO:teuthology.orchestra.run.smithi179.stdout: filestore apply finisher threads = 3 2023-10-19T14:22:19.413 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-10-19T14:22:19.413 INFO:teuthology.orchestra.run.smithi179.stdout: bdev debug aio = true 2023-10-19T14:22:19.414 INFO:teuthology.orchestra.run.smithi179.stdout: osd debug misdirected ops = true 2023-10-19T14:22:19.414 INFO:teuthology.orchestra.run.smithi179.stdout: bluestore block size = 96636764160 2023-10-19T14:22:19.414 INFO:teuthology.orchestra.run.smithi179.stdout: bluestore fsck on mount = True 2023-10-19T14:22:19.414 INFO:teuthology.orchestra.run.smithi179.stdout: debug bluefs = 20 2023-10-19T14:22:19.414 INFO:teuthology.orchestra.run.smithi179.stdout: debug bluestore = 20 2023-10-19T14:22:19.415 INFO:teuthology.orchestra.run.smithi179.stdout: debug ms = 1 2023-10-19T14:22:19.415 INFO:teuthology.orchestra.run.smithi179.stdout: debug osd = 20 2023-10-19T14:22:19.415 INFO:teuthology.orchestra.run.smithi179.stdout: debug rocksdb = 10 2023-10-19T14:22:19.415 INFO:teuthology.orchestra.run.smithi179.stdout: mon osd backfillfull_ratio = 0.85 2023-10-19T14:22:19.415 INFO:teuthology.orchestra.run.smithi179.stdout: mon osd full ratio = 0.9 2023-10-19T14:22:19.416 INFO:teuthology.orchestra.run.smithi179.stdout: mon osd nearfull ratio = 0.8 2023-10-19T14:22:19.416 INFO:teuthology.orchestra.run.smithi179.stdout: osd failsafe full ratio = 0.95 2023-10-19T14:22:19.416 INFO:teuthology.orchestra.run.smithi179.stdout: osd objectstore = bluestore 2023-10-19T14:22:19.416 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-10-19T14:22:19.416 INFO:teuthology.orchestra.run.smithi179.stdout:[mgr] 2023-10-19T14:22:19.416 INFO:teuthology.orchestra.run.smithi179.stdout: debug ms = 1 2023-10-19T14:22:19.417 INFO:teuthology.orchestra.run.smithi179.stdout: debug mgr = 20 2023-10-19T14:22:19.417 INFO:teuthology.orchestra.run.smithi179.stdout: debug mon = 20 2023-10-19T14:22:19.417 INFO:teuthology.orchestra.run.smithi179.stdout: debug auth = 20 2023-10-19T14:22:19.417 INFO:teuthology.orchestra.run.smithi179.stdout: mon reweight min pgs per osd = 4 2023-10-19T14:22:19.417 INFO:teuthology.orchestra.run.smithi179.stdout: mon reweight min bytes per osd = 10 2023-10-19T14:22:19.418 INFO:teuthology.orchestra.run.smithi179.stdout: mgr/telemetry/nag = false 2023-10-19T14:22:19.418 INFO:teuthology.orchestra.run.smithi179.stdout: debug client = 20 2023-10-19T14:22:19.418 INFO:teuthology.orchestra.run.smithi179.stdout: debug finisher = 20 2023-10-19T14:22:19.418 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-10-19T14:22:19.418 INFO:teuthology.orchestra.run.smithi179.stdout:[mon] 2023-10-19T14:22:19.419 INFO:teuthology.orchestra.run.smithi179.stdout: debug ms = 1 2023-10-19T14:22:19.419 INFO:teuthology.orchestra.run.smithi179.stdout: debug mon = 20 2023-10-19T14:22:19.419 INFO:teuthology.orchestra.run.smithi179.stdout: debug paxos = 20 2023-10-19T14:22:19.419 INFO:teuthology.orchestra.run.smithi179.stdout: debug auth = 20 2023-10-19T14:22:19.419 INFO:teuthology.orchestra.run.smithi179.stdout: mon data avail warn = 5 2023-10-19T14:22:19.419 INFO:teuthology.orchestra.run.smithi179.stdout: mon mgr mkfs grace = 240 2023-10-19T14:22:19.420 INFO:teuthology.orchestra.run.smithi179.stdout: mon reweight min pgs per osd = 4 2023-10-19T14:22:19.420 INFO:teuthology.orchestra.run.smithi179.stdout: mon osd reporter subtree level = osd 2023-10-19T14:22:19.420 INFO:teuthology.orchestra.run.smithi179.stdout: mon osd prime pg temp = true 2023-10-19T14:22:19.420 INFO:teuthology.orchestra.run.smithi179.stdout: mon reweight min bytes per osd = 10 2023-10-19T14:22:19.420 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-10-19T14:22:19.421 INFO:teuthology.orchestra.run.smithi179.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-19T14:22:19.421 INFO:teuthology.orchestra.run.smithi179.stdout: auth mon ticket ttl = 660 # 11m 2023-10-19T14:22:19.421 INFO:teuthology.orchestra.run.smithi179.stdout: auth service ticket ttl = 240 # 4m 2023-10-19T14:22:19.421 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-10-19T14:22:19.421 INFO:teuthology.orchestra.run.smithi179.stdout: # don't complain about insecure global_id in the test suite 2023-10-19T14:22:19.422 INFO:teuthology.orchestra.run.smithi179.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-19T14:22:19.422 INFO:teuthology.orchestra.run.smithi179.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-19T14:22:19.422 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-10-19T14:22:19.422 INFO:teuthology.orchestra.run.smithi179.stdout: # 1m isn't quite enough 2023-10-19T14:22:19.422 INFO:teuthology.orchestra.run.smithi179.stdout: mon_down_mkfs_grace = 2m 2023-10-19T14:22:19.423 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-10-19T14:22:19.423 INFO:teuthology.orchestra.run.smithi179.stdout: mon_warn_on_filestore_osds = false 2023-10-19T14:22:19.423 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-10-19T14:22:19.423 INFO:teuthology.orchestra.run.smithi179.stdout:[client] 2023-10-19T14:22:19.423 INFO:teuthology.orchestra.run.smithi179.stdout: rgw cache enabled = true 2023-10-19T14:22:19.423 INFO:teuthology.orchestra.run.smithi179.stdout: rgw enable ops log = true 2023-10-19T14:22:19.424 INFO:teuthology.orchestra.run.smithi179.stdout: rgw enable usage log = true 2023-10-19T14:22:19.424 INFO:teuthology.orchestra.run.smithi179.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-19T14:22:19.424 INFO:teuthology.orchestra.run.smithi179.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-19T14:22:19.424 INFO:teuthology.orchestra.run.smithi179.stdout: pid file = "" 2023-10-19T14:22:19.424 INFO:teuthology.orchestra.run.smithi179.stdout: client snapdir = .snap 2023-10-19T14:22:19.425 INFO:teuthology.orchestra.run.smithi179.stdout:[mon.a] 2023-10-19T14:22:19.425 INFO:teuthology.orchestra.run.smithi179.stdout:[mon.b] 2023-10-19T14:22:19.425 INFO:teuthology.orchestra.run.smithi179.stdout:[mon.c] 2023-10-19T14:22:19.425 INFO:teuthology.orchestra.run.smithi179.stdout:[mds.a] 2023-10-19T14:22:19.425 INFO:teuthology.orchestra.run.smithi179.stdout:[mds.c] 2023-10-19T14:22:19.425 INFO:teuthology.orchestra.run.smithi179.stdout:[mds.b] 2023-10-19T14:22:19.426 INFO:teuthology.orchestra.run.smithi179.stdout:[mds.d] 2023-10-19T14:22:19.427 INFO:teuthology.orchestra.run.smithi057.stdout:[global] 2023-10-19T14:22:19.428 INFO:teuthology.orchestra.run.smithi057.stdout: chdir = "" 2023-10-19T14:22:19.428 INFO:teuthology.orchestra.run.smithi057.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-19T14:22:19.428 INFO:teuthology.orchestra.run.smithi057.stdout: auth supported = cephx 2023-10-19T14:22:19.428 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:22:19.428 INFO:teuthology.orchestra.run.smithi057.stdout: filestore xattr use omap = true 2023-10-19T14:22:19.429 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:22:19.429 INFO:teuthology.orchestra.run.smithi057.stdout: mon clock drift allowed = 1.000 2023-10-19T14:22:19.429 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:22:19.429 INFO:teuthology.orchestra.run.smithi057.stdout: osd crush chooseleaf type = 0 2023-10-19T14:22:19.429 INFO:teuthology.orchestra.run.smithi057.stdout: auth debug = true 2023-10-19T14:22:19.429 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:22:19.430 INFO:teuthology.orchestra.run.smithi057.stdout: ms die on old message = true 2023-10-19T14:22:19.430 INFO:teuthology.orchestra.run.smithi057.stdout: ms die on bug = true 2023-10-19T14:22:19.430 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:22:19.430 INFO:teuthology.orchestra.run.smithi057.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-19T14:22:19.430 INFO:teuthology.orchestra.run.smithi057.stdout: mon pg warn max object skew = 0 2023-10-19T14:22:19.431 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:22:19.431 INFO:teuthology.orchestra.run.smithi057.stdout: # disable pg_autoscaler by default for new pools 2023-10-19T14:22:19.431 INFO:teuthology.orchestra.run.smithi057.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-19T14:22:19.431 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:22:19.431 INFO:teuthology.orchestra.run.smithi057.stdout: osd pool default size = 2 2023-10-19T14:22:19.432 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:22:19.432 INFO:teuthology.orchestra.run.smithi057.stdout: mon osd allow primary affinity = true 2023-10-19T14:22:19.432 INFO:teuthology.orchestra.run.smithi057.stdout: mon osd allow pg remap = true 2023-10-19T14:22:19.432 INFO:teuthology.orchestra.run.smithi057.stdout: mon warn on legacy crush tunables = false 2023-10-19T14:22:19.432 INFO:teuthology.orchestra.run.smithi057.stdout: mon warn on crush straw calc version zero = false 2023-10-19T14:22:19.433 INFO:teuthology.orchestra.run.smithi057.stdout: mon warn on no sortbitwise = false 2023-10-19T14:22:19.433 INFO:teuthology.orchestra.run.smithi057.stdout: mon warn on osd down out interval zero = false 2023-10-19T14:22:19.433 INFO:teuthology.orchestra.run.smithi057.stdout: mon warn on too few osds = false 2023-10-19T14:22:19.433 INFO:teuthology.orchestra.run.smithi057.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-19T14:22:19.433 INFO:teuthology.orchestra.run.smithi057.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-19T14:22:19.433 INFO:teuthology.orchestra.run.smithi057.stdout: mon_allow_pool_size_one = true 2023-10-19T14:22:19.434 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:22:19.434 INFO:teuthology.orchestra.run.smithi057.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-19T14:22:19.434 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:22:19.434 INFO:teuthology.orchestra.run.smithi057.stdout: osd default data pool replay window = 5 2023-10-19T14:22:19.434 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:22:19.435 INFO:teuthology.orchestra.run.smithi057.stdout: mon allow pool delete = true 2023-10-19T14:22:19.435 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:22:19.435 INFO:teuthology.orchestra.run.smithi057.stdout: mon cluster log file level = debug 2023-10-19T14:22:19.435 INFO:teuthology.orchestra.run.smithi057.stdout: debug asserts on shutdown = true 2023-10-19T14:22:19.435 INFO:teuthology.orchestra.run.smithi057.stdout: mon health detail to clog = false 2023-10-19T14:22:19.436 INFO:teuthology.orchestra.run.smithi057.stdout: mon host = "172.21.15.57,172.21.15.179,[v2:172.21.15.179:3301,v1:172.21.15.179:6790]" 2023-10-19T14:22:19.436 INFO:teuthology.orchestra.run.smithi057.stdout: fsid = 1e9beb19-3260-4dd7-85ef-5b399ce774ac 2023-10-19T14:22:19.436 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:22:19.436 INFO:teuthology.orchestra.run.smithi057.stdout:[osd] 2023-10-19T14:22:19.436 INFO:teuthology.orchestra.run.smithi057.stdout: osd journal size = 100 2023-10-19T14:22:19.437 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:22:19.437 INFO:teuthology.orchestra.run.smithi057.stdout: osd scrub load threshold = 5.0 2023-10-19T14:22:19.437 INFO:teuthology.orchestra.run.smithi057.stdout: osd scrub max interval = 600 2023-10-19T14:22:19.437 INFO:teuthology.orchestra.run.smithi057.stdout: osd mclock profile = high_recovery_ops 2023-10-19T14:22:19.437 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:22:19.438 INFO:teuthology.orchestra.run.smithi057.stdout: osd recover clone overlap = true 2023-10-19T14:22:19.438 INFO:teuthology.orchestra.run.smithi057.stdout: osd recovery max chunk = 1048576 2023-10-19T14:22:19.438 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:22:19.438 INFO:teuthology.orchestra.run.smithi057.stdout: osd debug shutdown = true 2023-10-19T14:22:19.438 INFO:teuthology.orchestra.run.smithi057.stdout: osd debug op order = true 2023-10-19T14:22:19.439 INFO:teuthology.orchestra.run.smithi057.stdout: osd debug verify stray on activate = true 2023-10-19T14:22:19.439 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:22:19.439 INFO:teuthology.orchestra.run.smithi057.stdout: osd open classes on start = true 2023-10-19T14:22:19.439 INFO:teuthology.orchestra.run.smithi057.stdout: osd debug pg log writeout = true 2023-10-19T14:22:19.439 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:22:19.439 INFO:teuthology.orchestra.run.smithi057.stdout: osd deep scrub update digest min age = 30 2023-10-19T14:22:19.440 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:22:19.440 INFO:teuthology.orchestra.run.smithi057.stdout: osd map max advance = 10 2023-10-19T14:22:19.440 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:22:19.440 INFO:teuthology.orchestra.run.smithi057.stdout: journal zero on create = true 2023-10-19T14:22:19.440 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:22:19.441 INFO:teuthology.orchestra.run.smithi057.stdout: filestore ondisk finisher threads = 3 2023-10-19T14:22:19.441 INFO:teuthology.orchestra.run.smithi057.stdout: filestore apply finisher threads = 3 2023-10-19T14:22:19.441 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:22:19.441 INFO:teuthology.orchestra.run.smithi057.stdout: bdev debug aio = true 2023-10-19T14:22:19.441 INFO:teuthology.orchestra.run.smithi057.stdout: osd debug misdirected ops = true 2023-10-19T14:22:19.441 INFO:teuthology.orchestra.run.smithi057.stdout: bluestore block size = 96636764160 2023-10-19T14:22:19.442 INFO:teuthology.orchestra.run.smithi057.stdout: bluestore fsck on mount = True 2023-10-19T14:22:19.442 INFO:teuthology.orchestra.run.smithi057.stdout: debug bluefs = 20 2023-10-19T14:22:19.442 INFO:teuthology.orchestra.run.smithi057.stdout: debug bluestore = 20 2023-10-19T14:22:19.442 INFO:teuthology.orchestra.run.smithi057.stdout: debug ms = 1 2023-10-19T14:22:19.442 INFO:teuthology.orchestra.run.smithi057.stdout: debug osd = 20 2023-10-19T14:22:19.443 INFO:teuthology.orchestra.run.smithi057.stdout: debug rocksdb = 10 2023-10-19T14:22:19.443 INFO:teuthology.orchestra.run.smithi057.stdout: mon osd backfillfull_ratio = 0.85 2023-10-19T14:22:19.443 INFO:teuthology.orchestra.run.smithi057.stdout: mon osd full ratio = 0.9 2023-10-19T14:22:19.443 INFO:teuthology.orchestra.run.smithi057.stdout: mon osd nearfull ratio = 0.8 2023-10-19T14:22:19.443 INFO:teuthology.orchestra.run.smithi057.stdout: osd failsafe full ratio = 0.95 2023-10-19T14:22:19.444 INFO:teuthology.orchestra.run.smithi057.stdout: osd objectstore = bluestore 2023-10-19T14:22:19.444 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:22:19.444 INFO:teuthology.orchestra.run.smithi057.stdout:[mgr] 2023-10-19T14:22:19.444 INFO:teuthology.orchestra.run.smithi057.stdout: debug ms = 1 2023-10-19T14:22:19.444 INFO:teuthology.orchestra.run.smithi057.stdout: debug mgr = 20 2023-10-19T14:22:19.444 INFO:teuthology.orchestra.run.smithi057.stdout: debug mon = 20 2023-10-19T14:22:19.445 INFO:teuthology.orchestra.run.smithi057.stdout: debug auth = 20 2023-10-19T14:22:19.445 INFO:teuthology.orchestra.run.smithi057.stdout: mon reweight min pgs per osd = 4 2023-10-19T14:22:19.445 INFO:teuthology.orchestra.run.smithi057.stdout: mon reweight min bytes per osd = 10 2023-10-19T14:22:19.445 INFO:teuthology.orchestra.run.smithi057.stdout: mgr/telemetry/nag = false 2023-10-19T14:22:19.445 INFO:teuthology.orchestra.run.smithi057.stdout: debug client = 20 2023-10-19T14:22:19.446 INFO:teuthology.orchestra.run.smithi057.stdout: debug finisher = 20 2023-10-19T14:22:19.446 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:22:19.446 INFO:teuthology.orchestra.run.smithi057.stdout:[mon] 2023-10-19T14:22:19.446 INFO:teuthology.orchestra.run.smithi057.stdout: debug ms = 1 2023-10-19T14:22:19.446 INFO:teuthology.orchestra.run.smithi057.stdout: debug mon = 20 2023-10-19T14:22:19.447 INFO:teuthology.orchestra.run.smithi057.stdout: debug paxos = 20 2023-10-19T14:22:19.447 INFO:teuthology.orchestra.run.smithi057.stdout: debug auth = 20 2023-10-19T14:22:19.447 INFO:teuthology.orchestra.run.smithi057.stdout: mon data avail warn = 5 2023-10-19T14:22:19.447 INFO:teuthology.orchestra.run.smithi057.stdout: mon mgr mkfs grace = 240 2023-10-19T14:22:19.447 INFO:teuthology.orchestra.run.smithi057.stdout: mon reweight min pgs per osd = 4 2023-10-19T14:22:19.447 INFO:teuthology.orchestra.run.smithi057.stdout: mon osd reporter subtree level = osd 2023-10-19T14:22:19.448 INFO:teuthology.orchestra.run.smithi057.stdout: mon osd prime pg temp = true 2023-10-19T14:22:19.448 INFO:teuthology.orchestra.run.smithi057.stdout: mon reweight min bytes per osd = 10 2023-10-19T14:22:19.448 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:22:19.448 INFO:teuthology.orchestra.run.smithi057.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-19T14:22:19.448 INFO:teuthology.orchestra.run.smithi057.stdout: auth mon ticket ttl = 660 # 11m 2023-10-19T14:22:19.449 INFO:teuthology.orchestra.run.smithi057.stdout: auth service ticket ttl = 240 # 4m 2023-10-19T14:22:19.449 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:22:19.449 INFO:teuthology.orchestra.run.smithi057.stdout: # don't complain about insecure global_id in the test suite 2023-10-19T14:22:19.449 INFO:teuthology.orchestra.run.smithi057.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-19T14:22:19.450 INFO:teuthology.orchestra.run.smithi057.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-19T14:22:19.450 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:22:19.450 INFO:teuthology.orchestra.run.smithi057.stdout: # 1m isn't quite enough 2023-10-19T14:22:19.450 INFO:teuthology.orchestra.run.smithi057.stdout: mon_down_mkfs_grace = 2m 2023-10-19T14:22:19.450 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:22:19.451 INFO:teuthology.orchestra.run.smithi057.stdout: mon_warn_on_filestore_osds = false 2023-10-19T14:22:19.451 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:22:19.451 INFO:teuthology.orchestra.run.smithi057.stdout:[client] 2023-10-19T14:22:19.451 INFO:teuthology.orchestra.run.smithi057.stdout: rgw cache enabled = true 2023-10-19T14:22:19.451 INFO:teuthology.orchestra.run.smithi057.stdout: rgw enable ops log = true 2023-10-19T14:22:19.451 INFO:teuthology.orchestra.run.smithi057.stdout: rgw enable usage log = true 2023-10-19T14:22:19.452 INFO:teuthology.orchestra.run.smithi057.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-19T14:22:19.452 INFO:teuthology.orchestra.run.smithi057.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-19T14:22:19.452 INFO:teuthology.orchestra.run.smithi057.stdout: pid file = "" 2023-10-19T14:22:19.452 INFO:teuthology.orchestra.run.smithi057.stdout: client snapdir = .snap 2023-10-19T14:22:19.452 INFO:teuthology.orchestra.run.smithi057.stdout:[mon.a] 2023-10-19T14:22:19.453 INFO:teuthology.orchestra.run.smithi057.stdout:[mon.b] 2023-10-19T14:22:19.453 INFO:teuthology.orchestra.run.smithi057.stdout:[mon.c] 2023-10-19T14:22:19.453 INFO:teuthology.orchestra.run.smithi057.stdout:[mds.a] 2023-10-19T14:22:19.453 INFO:teuthology.orchestra.run.smithi057.stdout:[mds.c] 2023-10-19T14:22:19.453 INFO:teuthology.orchestra.run.smithi057.stdout:[mds.b] 2023-10-19T14:22:19.453 INFO:teuthology.orchestra.run.smithi057.stdout:[mds.d] 2023-10-19T14:22:19.454 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T14:22:19.454 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T14:22:19.470 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T14:22:19.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.617+0000 7f666516e640 1 -- 172.21.15.57:0/2558237842 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f664c003970 msgr2=0x7f664c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:19.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.617+0000 7f666516e640 1 --2- 172.21.15.57:0/2558237842 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f664c003970 0x7f664c003d90 secure :-1 s=READY pgs=2932 cs=0 l=1 rev1=1 crypto rx=0x7f665000b9d0 tx=0x7f664c0057f0 comp rx=0 tx=0).stop 2023-10-19T14:22:19.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.617+0000 7f665f59e640 1 -- 172.21.15.57:0/2558237842 reap_dead start 2023-10-19T14:22:19.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.617+0000 7f666516e640 1 -- 172.21.15.57:0/2558237842 shutdown_connections 2023-10-19T14:22:19.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.617+0000 7f666516e640 1 -- 172.21.15.57:0/2558237842 >> 172.21.15.57:0/2558237842 conn(0x7f6660069810 msgr2=0x7f6660069c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:19.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.617+0000 7f666516e640 1 -- 172.21.15.57:0/2558237842 shutdown_connections 2023-10-19T14:22:19.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.617+0000 7f666516e640 1 -- 172.21.15.57:0/2558237842 wait complete. 2023-10-19T14:22:19.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.618+0000 7f666516e640 1 Processor -- start 2023-10-19T14:22:19.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.619+0000 7f666516e640 1 -- start start 2023-10-19T14:22:19.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.619+0000 7f666516e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f664c003970 0x7f666013b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:19.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.619+0000 7f666516e640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f66601578d0 0x7f6660155360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:19.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.619+0000 7f666516e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f666013bd50 0x7f66601e1f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:19.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.619+0000 7f666516e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f666013b3f0 con 0x7f666013bd50 2023-10-19T14:22:19.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.619+0000 7f666516e640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6660158170 con 0x7f664c003970 2023-10-19T14:22:19.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.619+0000 7f666516e640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f666013a770 con 0x7f66601578d0 2023-10-19T14:22:19.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.619+0000 7f665f59e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f666013bd50 0x7f66601e1f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:19.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.620+0000 7f665f59e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f666013bd50 0x7f66601e1f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46422/0 (socket says 172.21.15.57:46422) 2023-10-19T14:22:19.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.620+0000 7f665f59e640 1 -- 172.21.15.57:0/3316311681 learned_addr learned my addr 172.21.15.57:0/3316311681 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:22:19.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.620+0000 7f665ed9d640 1 --2- 172.21.15.57:0/3316311681 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f664c003970 0x7f666013b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:19.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.620+0000 7f665659c640 1 --2- 172.21.15.57:0/3316311681 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f66601578d0 0x7f6660155360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:19.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.620+0000 7f665f59e640 1 -- 172.21.15.57:0/3316311681 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f66601578d0 msgr2=0x7f6660155360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:19.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.620+0000 7f665f59e640 1 --2- 172.21.15.57:0/3316311681 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f66601578d0 0x7f6660155360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:19.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.620+0000 7f665f59e640 1 -- 172.21.15.57:0/3316311681 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f664c003970 msgr2=0x7f666013b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:19.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.620+0000 7f665f59e640 1 --2- 172.21.15.57:0/3316311681 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f664c003970 0x7f666013b810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:19.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.620+0000 7f665f59e640 1 -- 172.21.15.57:0/3316311681 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f666013aa50 con 0x7f666013bd50 2023-10-19T14:22:19.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.620+0000 7f665f59e640 1 --2- 172.21.15.57:0/3316311681 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f666013bd50 0x7f66601e1f10 secure :-1 s=READY pgs=2933 cs=0 l=1 rev1=1 crypto rx=0x7f6658005fd0 tx=0x7f6658004700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:19.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.620+0000 7f6657fff640 1 -- 172.21.15.57:0/3316311681 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6658004d20 con 0x7f666013bd50 2023-10-19T14:22:19.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.620+0000 7f6657fff640 1 -- 172.21.15.57:0/3316311681 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f6658004e80 con 0x7f666013bd50 2023-10-19T14:22:19.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.620+0000 7f666516e640 1 -- 172.21.15.57:0/3316311681 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f666013ad30 con 0x7f666013bd50 2023-10-19T14:22:19.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.620+0000 7f6657fff640 1 -- 172.21.15.57:0/3316311681 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6658010ce0 con 0x7f666013bd50 2023-10-19T14:22:19.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.620+0000 7f666516e640 1 -- 172.21.15.57:0/3316311681 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66601558a0 con 0x7f666013bd50 2023-10-19T14:22:19.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.621+0000 7f666516e640 1 -- 172.21.15.57:0/3316311681 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f666013b530 con 0x7f666013bd50 2023-10-19T14:22:19.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.622+0000 7f6657fff640 1 -- 172.21.15.57:0/3316311681 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 148) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f6658012450 con 0x7f666013bd50 2023-10-19T14:22:19.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.622+0000 7f6657fff640 1 --2- 172.21.15.57:0/3316311681 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f664006ace0 0x7f664006d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:19.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.622+0000 7f6657fff640 1 -- 172.21.15.57:0/3316311681 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(264..264 src has 1..264) v4 ==== 6487+0+0 (secure 0 0 0) 0x7f665808a3b0 con 0x7f666013bd50 2023-10-19T14:22:19.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.622+0000 7f665ed9d640 1 --2- 172.21.15.57:0/3316311681 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f664006ace0 0x7f664006d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:19.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.623+0000 7f665ed9d640 1 --2- 172.21.15.57:0/3316311681 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f664006ace0 0x7f664006d1a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f6650037000 tx=0x7f665002e040 comp rx=0 tx=0).ready entity=mgr.10394 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:19.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.626+0000 7f6657fff640 1 -- 172.21.15.57:0/3316311681 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f666013b530 con 0x7f666013bd50 2023-10-19T14:22:19.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.804+0000 7f666516e640 1 -- 172.21.15.57:0/3316311681 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f666013b530 con 0x7f666013bd50 2023-10-19T14:22:19.808 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.805+0000 7f6657fff640 1 -- 172.21.15.57:0/3316311681 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v264) v1 ==== 74+0+17407 (secure 0 0 0) 0x7f666013b530 con 0x7f666013bd50 2023-10-19T14:22:19.809 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:22:19.809 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":264,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T14:22:17.865478+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":36,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_metadata","create_time":"2023-10-19T14:22:00.047912+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"262","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_data","create_time":"2023-10-19T14:22:00.739698+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"263","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.179:0/1983646445":"2023-10-20T14:22:17.865415+0000","172.21.15.179:0/1840191723":"2023-10-20T14:22:17.865415+0000","172.21.15.179:0/4168916968":"2023-10-20T14:22:17.865415+0000","172.21.15.179:6833/1573806521":"2023-10-20T14:22:17.865415+0000","172.21.15.179:6832/1573806521":"2023-10-20T14:22:17.865415+0000","172.21.15.179:0/658875312":"2023-10-20T14:22:17.865415+0000","172.21.15.179:0/1799772260":"2023-10-20T14:22:17.865415+0000","172.21.15.179:0/3831670324":"2023-10-20T14:22:17.865415+0000","172.21.15.179:0/90219027":"2023-10-20T14:22:17.865415+0000","172.21.15.179:0/2739598080":"2023-10-20T14:22:17.865415+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T14:22:19.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.808+0000 7f666516e640 1 -- 172.21.15.57:0/3316311681 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f664006ace0 msgr2=0x7f664006d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:19.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.808+0000 7f666516e640 1 --2- 172.21.15.57:0/3316311681 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f664006ace0 0x7f664006d1a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f6650037000 tx=0x7f665002e040 comp rx=0 tx=0).stop 2023-10-19T14:22:19.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.808+0000 7f666516e640 1 -- 172.21.15.57:0/3316311681 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f666013bd50 msgr2=0x7f66601e1f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:19.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.808+0000 7f666516e640 1 --2- 172.21.15.57:0/3316311681 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f666013bd50 0x7f66601e1f10 secure :-1 s=READY pgs=2933 cs=0 l=1 rev1=1 crypto rx=0x7f6658005fd0 tx=0x7f6658004700 comp rx=0 tx=0).stop 2023-10-19T14:22:19.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.808+0000 7f666516e640 1 -- 172.21.15.57:0/3316311681 shutdown_connections 2023-10-19T14:22:19.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.808+0000 7f666516e640 1 --2- 172.21.15.57:0/3316311681 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f664006ace0 0x7f664006d1a0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:19.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.808+0000 7f666516e640 1 --2- 172.21.15.57:0/3316311681 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f666013bd50 0x7f66601e1f10 unknown :-1 s=CLOSED pgs=2933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:19.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.808+0000 7f666516e640 1 --2- 172.21.15.57:0/3316311681 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f66601578d0 0x7f6660155360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:19.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.808+0000 7f666516e640 1 --2- 172.21.15.57:0/3316311681 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f664c003970 0x7f666013b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:19.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.808+0000 7f666516e640 1 -- 172.21.15.57:0/3316311681 >> 172.21.15.57:0/3316311681 conn(0x7f6660069810 msgr2=0x7f66600acfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:19.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.808+0000 7f666516e640 1 -- 172.21.15.57:0/3316311681 shutdown_connections 2023-10-19T14:22:19.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.808+0000 7f666516e640 1 -- 172.21.15.57:0/3316311681 wait complete. 2023-10-19T14:22:19.822 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-19T12:44:07.371186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 35, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-19T14:22:00.047912+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '262', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 36, 'pool_name': 'cephfs_data', 'create_time': '2023-10-19T14:22:00.739698+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '263', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-19T14:22:19.823 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-19T14:22:19.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.934+0000 7f2ef89e2640 1 -- 172.21.15.57:0/2505556430 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2ef40d1250 msgr2=0x7f2ef40cea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:19.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.934+0000 7f2ef89e2640 1 --2- 172.21.15.57:0/2505556430 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2ef40d1250 0x7f2ef40cea30 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f2edc009830 tx=0x7f2edc02bd60 comp rx=0 tx=0).stop 2023-10-19T14:22:19.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.935+0000 7f2ef89e2640 1 -- 172.21.15.57:0/2505556430 shutdown_connections 2023-10-19T14:22:19.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.935+0000 7f2ef89e2640 1 --2- 172.21.15.57:0/2505556430 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2ef40cef70 0x7f2ef40cc310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:19.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.935+0000 7f2ef89e2640 1 --2- 172.21.15.57:0/2505556430 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2ef40d1250 0x7f2ef40cea30 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:19.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.935+0000 7f2ef89e2640 1 -- 172.21.15.57:0/2505556430 >> 172.21.15.57:0/2505556430 conn(0x7f2ef408a7e0 msgr2=0x7f2ef409a100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:19.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.935+0000 7f2ef89e2640 1 -- 172.21.15.57:0/2505556430 shutdown_connections 2023-10-19T14:22:19.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.935+0000 7f2ef89e2640 1 -- 172.21.15.57:0/2505556430 wait complete. 2023-10-19T14:22:19.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.936+0000 7f2ef89e2640 1 Processor -- start 2023-10-19T14:22:19.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.936+0000 7f2ef89e2640 1 -- start start 2023-10-19T14:22:19.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.937+0000 7f2ef89e2640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2ef40cef70 0x7f2ef41e62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:19.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.937+0000 7f2ef89e2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2ef40d1250 0x7f2ef41e42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:19.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.937+0000 7f2ef89e2640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2ef415b070 0x7f2ef40b6730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:19.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.937+0000 7f2ef89e2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef41e4800 con 0x7f2ef40d1250 2023-10-19T14:22:19.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.937+0000 7f2ef89e2640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef40b6c70 con 0x7f2ef415b070 2023-10-19T14:22:19.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.937+0000 7f2ef89e2640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef41e4a40 con 0x7f2ef40cef70 2023-10-19T14:22:19.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.937+0000 7f2eebfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2ef40d1250 0x7f2ef41e42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:19.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.937+0000 7f2eebfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2ef40d1250 0x7f2ef41e42c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46436/0 (socket says 172.21.15.57:46436) 2023-10-19T14:22:19.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.937+0000 7f2eebfff640 1 -- 172.21.15.57:0/4272446004 learned_addr learned my addr 172.21.15.57:0/4272446004 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:22:19.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.937+0000 7f2ef259c640 1 --2- 172.21.15.57:0/4272446004 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2ef40cef70 0x7f2ef41e62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:19.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.937+0000 7f2ef2d9d640 1 --2- 172.21.15.57:0/4272446004 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2ef415b070 0x7f2ef40b6730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:19.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.937+0000 7f2eebfff640 1 -- 172.21.15.57:0/4272446004 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2ef40cef70 msgr2=0x7f2ef41e62e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:19.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.937+0000 7f2eebfff640 1 --2- 172.21.15.57:0/4272446004 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2ef40cef70 0x7f2ef41e62e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:19.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.937+0000 7f2eebfff640 1 -- 172.21.15.57:0/4272446004 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2ef415b070 msgr2=0x7f2ef40b6730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:19.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.937+0000 7f2eebfff640 1 --2- 172.21.15.57:0/4272446004 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2ef415b070 0x7f2ef40b6730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:19.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.937+0000 7f2eebfff640 1 -- 172.21.15.57:0/4272446004 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2edc009490 con 0x7f2ef40d1250 2023-10-19T14:22:19.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.937+0000 7f2eebfff640 1 --2- 172.21.15.57:0/4272446004 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2ef40d1250 0x7f2ef41e42c0 secure :-1 s=READY pgs=2935 cs=0 l=1 rev1=1 crypto rx=0x7f2edc009780 tx=0x7f2edc0038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:19.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.938+0000 7f2eeaffd640 1 -- 172.21.15.57:0/4272446004 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2edc003ea0 con 0x7f2ef40d1250 2023-10-19T14:22:19.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.938+0000 7f2ef89e2640 1 -- 172.21.15.57:0/4272446004 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ef41e4d20 con 0x7f2ef40d1250 2023-10-19T14:22:19.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.938+0000 7f2eeaffd640 1 -- 172.21.15.57:0/4272446004 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f2edc034440 con 0x7f2ef40d1250 2023-10-19T14:22:19.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.938+0000 7f2eeaffd640 1 -- 172.21.15.57:0/4272446004 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2edc034850 con 0x7f2ef40d1250 2023-10-19T14:22:19.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.938+0000 7f2ef89e2640 1 -- 172.21.15.57:0/4272446004 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ef41e52e0 con 0x7f2ef40d1250 2023-10-19T14:22:19.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.939+0000 7f2ef89e2640 1 -- 172.21.15.57:0/4272446004 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ef40ce6a0 con 0x7f2ef40d1250 2023-10-19T14:22:19.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.939+0000 7f2eeaffd640 1 -- 172.21.15.57:0/4272446004 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 148) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f2edc03b040 con 0x7f2ef40d1250 2023-10-19T14:22:19.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.939+0000 7f2eeaffd640 1 --2- 172.21.15.57:0/4272446004 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f2ed406ace0 0x7f2ed406d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:19.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.940+0000 7f2eeaffd640 1 -- 172.21.15.57:0/4272446004 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(264..264 src has 1..264) v4 ==== 6487+0+0 (secure 0 0 0) 0x7f2edc0ab920 con 0x7f2ef40d1250 2023-10-19T14:22:19.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.940+0000 7f2ef259c640 1 --2- 172.21.15.57:0/4272446004 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f2ed406ace0 0x7f2ed406d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:19.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.940+0000 7f2ef259c640 1 --2- 172.21.15.57:0/4272446004 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f2ed406ace0 0x7f2ed406d1a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2ee40011e0 tx=0x7f2ee4007d20 comp rx=0 tx=0).ready entity=mgr.10394 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:19.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:19.944+0000 7f2eeaffd640 1 -- 172.21.15.57:0/4272446004 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2edc076a30 con 0x7f2ef40d1250 2023-10-19T14:22:20.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.120+0000 7f2ef89e2640 1 -- 172.21.15.57:0/4272446004 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2ef409ed00 con 0x7f2ef40d1250 2023-10-19T14:22:20.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.120+0000 7f2eeaffd640 1 -- 172.21.15.57:0/4272446004 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v264) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2edc07a050 con 0x7f2ef40d1250 2023-10-19T14:22:20.123 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-19T14:22:20.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.122+0000 7f2ef89e2640 1 -- 172.21.15.57:0/4272446004 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f2ed406ace0 msgr2=0x7f2ed406d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:20.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.122+0000 7f2ef89e2640 1 --2- 172.21.15.57:0/4272446004 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f2ed406ace0 0x7f2ed406d1a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2ee40011e0 tx=0x7f2ee4007d20 comp rx=0 tx=0).stop 2023-10-19T14:22:20.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.122+0000 7f2ef89e2640 1 -- 172.21.15.57:0/4272446004 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2ef40d1250 msgr2=0x7f2ef41e42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:20.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.122+0000 7f2ef89e2640 1 --2- 172.21.15.57:0/4272446004 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2ef40d1250 0x7f2ef41e42c0 secure :-1 s=READY pgs=2935 cs=0 l=1 rev1=1 crypto rx=0x7f2edc009780 tx=0x7f2edc0038a0 comp rx=0 tx=0).stop 2023-10-19T14:22:20.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.122+0000 7f2ef89e2640 1 -- 172.21.15.57:0/4272446004 shutdown_connections 2023-10-19T14:22:20.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.122+0000 7f2ef89e2640 1 --2- 172.21.15.57:0/4272446004 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f2ed406ace0 0x7f2ed406d1a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:20.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.122+0000 7f2ef89e2640 1 --2- 172.21.15.57:0/4272446004 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2ef415b070 0x7f2ef40b6730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:20.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.122+0000 7f2ef89e2640 1 --2- 172.21.15.57:0/4272446004 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2ef40d1250 0x7f2ef41e42c0 unknown :-1 s=CLOSED pgs=2935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:20.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.122+0000 7f2ef89e2640 1 --2- 172.21.15.57:0/4272446004 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2ef40cef70 0x7f2ef41e62e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:20.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.122+0000 7f2ef89e2640 1 -- 172.21.15.57:0/4272446004 >> 172.21.15.57:0/4272446004 conn(0x7f2ef408a7e0 msgr2=0x7f2ef40ada60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:20.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.122+0000 7f2ef89e2640 1 -- 172.21.15.57:0/4272446004 shutdown_connections 2023-10-19T14:22:20.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.122+0000 7f2ef89e2640 1 -- 172.21.15.57:0/4272446004 wait complete. 2023-10-19T14:22:20.136 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-19T14:22:20.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.246+0000 7f7de639e640 1 -- 172.21.15.57:0/1658403120 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7dc8003970 msgr2=0x7f7dc8003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:20.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.246+0000 7f7de639e640 1 --2- 172.21.15.57:0/1658403120 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7dc8003970 0x7f7dc8003d90 secure :-1 s=READY pgs=2937 cs=0 l=1 rev1=1 crypto rx=0x7f7dd400c290 tx=0x7f7dc80057f0 comp rx=0 tx=0).stop 2023-10-19T14:22:20.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.246+0000 7f7de493b640 1 -- 172.21.15.57:0/1658403120 reap_dead start 2023-10-19T14:22:20.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.246+0000 7f7de639e640 1 -- 172.21.15.57:0/1658403120 shutdown_connections 2023-10-19T14:22:20.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.246+0000 7f7de639e640 1 -- 172.21.15.57:0/1658403120 >> 172.21.15.57:0/1658403120 conn(0x7f7de00aebf0 msgr2=0x7f7de00af000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:20.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.246+0000 7f7de639e640 1 -- 172.21.15.57:0/1658403120 shutdown_connections 2023-10-19T14:22:20.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.246+0000 7f7de639e640 1 -- 172.21.15.57:0/1658403120 wait complete. 2023-10-19T14:22:20.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.247+0000 7f7de639e640 1 Processor -- start 2023-10-19T14:22:20.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.247+0000 7f7de639e640 1 -- start start 2023-10-19T14:22:20.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.248+0000 7f7de639e640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7dc8003970 0x7f7de013be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:20.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.248+0000 7f7de639e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7de013a600 0x7f7de013aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:20.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.248+0000 7f7de639e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7de0152500 0x7f7de01529b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:20.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.248+0000 7f7de639e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7de013afc0 con 0x7f7de013a600 2023-10-19T14:22:20.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.248+0000 7f7de639e640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7de0153210 con 0x7f7de0152500 2023-10-19T14:22:20.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.248+0000 7f7de639e640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7de01534f0 con 0x7f7dc8003970 2023-10-19T14:22:20.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.248+0000 7f7ddf7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7de013a600 0x7f7de013aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:20.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.248+0000 7f7ddf7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7de013a600 0x7f7de013aa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46444/0 (socket says 172.21.15.57:46444) 2023-10-19T14:22:20.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.248+0000 7f7ddf7fe640 1 -- 172.21.15.57:0/560053292 learned_addr learned my addr 172.21.15.57:0/560053292 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:22:20.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.248+0000 7f7de493b640 1 --2- 172.21.15.57:0/560053292 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7de0152500 0x7f7de01529b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:20.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.248+0000 7f7ddffff640 1 --2- 172.21.15.57:0/560053292 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7dc8003970 0x7f7de013be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:20.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.249+0000 7f7ddf7fe640 1 -- 172.21.15.57:0/560053292 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7dc8003970 msgr2=0x7f7de013be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:20.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.249+0000 7f7ddf7fe640 1 --2- 172.21.15.57:0/560053292 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7dc8003970 0x7f7de013be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:20.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.249+0000 7f7ddf7fe640 1 -- 172.21.15.57:0/560053292 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7de0152500 msgr2=0x7f7de01529b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:20.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.249+0000 7f7ddf7fe640 1 --2- 172.21.15.57:0/560053292 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7de0152500 0x7f7de01529b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:20.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.249+0000 7f7ddf7fe640 1 -- 172.21.15.57:0/560053292 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7de01537d0 con 0x7f7de013a600 2023-10-19T14:22:20.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.249+0000 7f7ddffff640 1 --2- 172.21.15.57:0/560053292 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7dc8003970 0x7f7de013be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:22:20.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.249+0000 7f7ddf7fe640 1 --2- 172.21.15.57:0/560053292 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7de013a600 0x7f7de013aa80 secure :-1 s=READY pgs=2938 cs=0 l=1 rev1=1 crypto rx=0x7f7dd0011b20 tx=0x7f7dd000f3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:20.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.249+0000 7f7ddcff9640 1 -- 172.21.15.57:0/560053292 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dd0018480 con 0x7f7de013a600 2023-10-19T14:22:20.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.249+0000 7f7ddcff9640 1 -- 172.21.15.57:0/560053292 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f7dd00185e0 con 0x7f7de013a600 2023-10-19T14:22:20.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.249+0000 7f7ddcff9640 1 -- 172.21.15.57:0/560053292 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dd0018990 con 0x7f7de013a600 2023-10-19T14:22:20.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.249+0000 7f7de639e640 1 -- 172.21.15.57:0/560053292 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7de0153ab0 con 0x7f7de013a600 2023-10-19T14:22:20.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.249+0000 7f7de639e640 1 -- 172.21.15.57:0/560053292 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7de0152ef0 con 0x7f7de013a600 2023-10-19T14:22:20.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.250+0000 7f7de639e640 1 -- 172.21.15.57:0/560053292 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7de006c710 con 0x7f7de013a600 2023-10-19T14:22:20.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.251+0000 7f7ddcff9640 1 -- 172.21.15.57:0/560053292 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 148) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f7dd0018af0 con 0x7f7de013a600 2023-10-19T14:22:20.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.251+0000 7f7ddcff9640 1 --2- 172.21.15.57:0/560053292 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f7dcc06ace0 0x7f7dcc06d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:20.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.251+0000 7f7ddcff9640 1 -- 172.21.15.57:0/560053292 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(264..264 src has 1..264) v4 ==== 6487+0+0 (secure 0 0 0) 0x7f7dd008d020 con 0x7f7de013a600 2023-10-19T14:22:20.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.251+0000 7f7ddffff640 1 --2- 172.21.15.57:0/560053292 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f7dcc06ace0 0x7f7dcc06d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:20.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.252+0000 7f7ddffff640 1 --2- 172.21.15.57:0/560053292 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f7dcc06ace0 0x7f7dcc06d1a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7dd400c260 tx=0x7f7dd4001c00 comp rx=0 tx=0).ready entity=mgr.10394 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:20.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.255+0000 7f7ddcff9640 1 -- 172.21.15.57:0/560053292 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7dd00581b0 con 0x7f7de013a600 2023-10-19T14:22:20.362 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.359+0000 7f7ddcff9640 1 -- 172.21.15.57:0/560053292 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mgrmap(e 149) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f7dd0058580 con 0x7f7de013a600 2023-10-19T14:22:20.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.434+0000 7f7de639e640 1 -- 172.21.15.57:0/560053292 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7de0141840 con 0x7f7de013a600 2023-10-19T14:22:20.437 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.434+0000 7f7ddcff9640 1 -- 172.21.15.57:0/560053292 <== mon.0 v2:172.21.15.57:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v264) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7dd0088d20 con 0x7f7de013a600 2023-10-19T14:22:20.437 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2023-10-19T14:22:20.439 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.436+0000 7f7de639e640 1 -- 172.21.15.57:0/560053292 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f7dcc06ace0 msgr2=0x7f7dcc06d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:20.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.436+0000 7f7de639e640 1 --2- 172.21.15.57:0/560053292 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f7dcc06ace0 0x7f7dcc06d1a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7dd400c260 tx=0x7f7dd4001c00 comp rx=0 tx=0).stop 2023-10-19T14:22:20.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.436+0000 7f7de639e640 1 -- 172.21.15.57:0/560053292 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7de013a600 msgr2=0x7f7de013aa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:20.440 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.436+0000 7f7de639e640 1 --2- 172.21.15.57:0/560053292 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7de013a600 0x7f7de013aa80 secure :-1 s=READY pgs=2938 cs=0 l=1 rev1=1 crypto rx=0x7f7dd0011b20 tx=0x7f7dd000f3c0 comp rx=0 tx=0).stop 2023-10-19T14:22:20.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.436+0000 7f7de639e640 1 -- 172.21.15.57:0/560053292 shutdown_connections 2023-10-19T14:22:20.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.436+0000 7f7de639e640 1 --2- 172.21.15.57:0/560053292 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f7dcc06ace0 0x7f7dcc06d1a0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:20.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.436+0000 7f7de639e640 1 --2- 172.21.15.57:0/560053292 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7de0152500 0x7f7de01529b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:20.441 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.436+0000 7f7de639e640 1 --2- 172.21.15.57:0/560053292 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7de013a600 0x7f7de013aa80 unknown :-1 s=CLOSED pgs=2938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:20.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.436+0000 7f7de639e640 1 --2- 172.21.15.57:0/560053292 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7dc8003970 0x7f7de013be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:20.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.436+0000 7f7de639e640 1 -- 172.21.15.57:0/560053292 >> 172.21.15.57:0/560053292 conn(0x7f7de00aebf0 msgr2=0x7f7de00b1cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:20.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.437+0000 7f7de639e640 1 -- 172.21.15.57:0/560053292 shutdown_connections 2023-10-19T14:22:20.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.437+0000 7f7de639e640 1 -- 172.21.15.57:0/560053292 wait complete. 2023-10-19T14:22:20.450 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-19T14:22:20.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.560+0000 7f86bf61a640 1 -- 172.21.15.57:0/4264884960 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f86b80d1080 msgr2=0x7f86b80cea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:20.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.560+0000 7f86bf61a640 1 --2- 172.21.15.57:0/4264884960 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f86b80d1080 0x7f86b80cea30 secure :-1 s=READY pgs=2940 cs=0 l=1 rev1=1 crypto rx=0x7f86b809ecd0 tx=0x7f86ac02be40 comp rx=0 tx=0).stop 2023-10-19T14:22:20.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.560+0000 7f86bf61a640 1 -- 172.21.15.57:0/4264884960 shutdown_connections 2023-10-19T14:22:20.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.560+0000 7f86bf61a640 1 --2- 172.21.15.57:0/4264884960 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f86b80d1080 0x7f86b80cea30 unknown :-1 s=CLOSED pgs=2940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:20.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.560+0000 7f86bf61a640 1 --2- 172.21.15.57:0/4264884960 >> v2:172.21.15.179:3300/0 conn(0x7f86b815b070 0x7f86b80d0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:20.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.560+0000 7f86bf61a640 1 -- 172.21.15.57:0/4264884960 >> 172.21.15.57:0/4264884960 conn(0x7f86b808a7e0 msgr2=0x7f86b809a100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:20.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.560+0000 7f86bf61a640 1 -- 172.21.15.57:0/4264884960 shutdown_connections 2023-10-19T14:22:20.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.560+0000 7f86bf61a640 1 -- 172.21.15.57:0/4264884960 wait complete. 2023-10-19T14:22:20.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.561+0000 7f86bf61a640 1 Processor -- start 2023-10-19T14:22:20.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.561+0000 7f86bf61a640 1 -- start start 2023-10-19T14:22:20.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.562+0000 7f86bf61a640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f86b80cef70 0x7f86b80ca830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:20.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.562+0000 7f86bf61a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f86b80d1080 0x7f86b8158a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:20.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.562+0000 7f86bf61a640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f86b815b070 0x7f86b815c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:20.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.562+0000 7f86bf61a640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86b80c8ad0 con 0x7f86b80d1080 2023-10-19T14:22:20.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.562+0000 7f86bf61a640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86b815c540 con 0x7f86b815b070 2023-10-19T14:22:20.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.562+0000 7f86bf61a640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86b80c7ec0 con 0x7f86b80cef70 2023-10-19T14:22:20.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.562+0000 7f86bcbb5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f86b80d1080 0x7f86b8158a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:20.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.562+0000 7f86bcbb5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f86b80d1080 0x7f86b8158a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46458/0 (socket says 172.21.15.57:46458) 2023-10-19T14:22:20.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.562+0000 7f86bcbb5640 1 -- 172.21.15.57:0/2129168977 learned_addr learned my addr 172.21.15.57:0/2129168977 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:22:20.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.562+0000 7f86bd3b6640 1 --2- 172.21.15.57:0/2129168977 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f86b80cef70 0x7f86b80ca830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:20.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.562+0000 7f86bdbb7640 1 --2- 172.21.15.57:0/2129168977 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f86b815b070 0x7f86b815c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:20.569 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.562+0000 7f86bcbb5640 1 -- 172.21.15.57:0/2129168977 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f86b80cef70 msgr2=0x7f86b80ca830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:20.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.562+0000 7f86bcbb5640 1 --2- 172.21.15.57:0/2129168977 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f86b80cef70 0x7f86b80ca830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:20.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.562+0000 7f86bcbb5640 1 -- 172.21.15.57:0/2129168977 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f86b815b070 msgr2=0x7f86b815c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:20.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.562+0000 7f86bcbb5640 1 --2- 172.21.15.57:0/2129168977 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f86b815b070 0x7f86b815c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:20.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.562+0000 7f86bcbb5640 1 -- 172.21.15.57:0/2129168977 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86ac009560 con 0x7f86b80d1080 2023-10-19T14:22:20.570 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.562+0000 7f86bd3b6640 1 --2- 172.21.15.57:0/2129168977 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f86b80cef70 0x7f86b80ca830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:22:20.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.562+0000 7f86bcbb5640 1 --2- 172.21.15.57:0/2129168977 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f86b80d1080 0x7f86b8158a40 secure :-1 s=READY pgs=2941 cs=0 l=1 rev1=1 crypto rx=0x7f86ac000c00 tx=0x7f86ac02de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:20.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.563+0000 7f86b5ffb640 1 -- 172.21.15.57:0/2129168977 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86ac005670 con 0x7f86b80d1080 2023-10-19T14:22:20.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.563+0000 7f86b5ffb640 1 -- 172.21.15.57:0/2129168977 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f86ac0038f0 con 0x7f86b80d1080 2023-10-19T14:22:20.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.563+0000 7f86bf61a640 1 -- 172.21.15.57:0/2129168977 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86b80c81a0 con 0x7f86b80d1080 2023-10-19T14:22:20.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.563+0000 7f86b5ffb640 1 -- 172.21.15.57:0/2129168977 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86ac003ce0 con 0x7f86b80d1080 2023-10-19T14:22:20.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.563+0000 7f86bf61a640 1 -- 172.21.15.57:0/2129168977 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86b80c8760 con 0x7f86b80d1080 2023-10-19T14:22:20.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.564+0000 7f86bf61a640 1 -- 172.21.15.57:0/2129168977 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86b80cede0 con 0x7f86b80d1080 2023-10-19T14:22:20.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.564+0000 7f86b5ffb640 1 -- 172.21.15.57:0/2129168977 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 149) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f86ac03e450 con 0x7f86b80d1080 2023-10-19T14:22:20.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.565+0000 7f86b5ffb640 1 --2- 172.21.15.57:0/2129168977 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f86a406afb0 0x7f86a406d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:20.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.565+0000 7f86b5ffb640 1 -- 172.21.15.57:0/2129168977 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(264..264 src has 1..264) v4 ==== 6487+0+0 (secure 0 0 0) 0x7f86ac0ac6e0 con 0x7f86b80d1080 2023-10-19T14:22:20.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.565+0000 7f86bd3b6640 1 --2- 172.21.15.57:0/2129168977 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f86a406afb0 0x7f86a406d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:20.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.566+0000 7f86bd3b6640 1 --2- 172.21.15.57:0/2129168977 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f86a406afb0 0x7f86a406d470 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f86b0009890 tx=0x7f86b0008040 comp rx=0 tx=0).ready entity=mgr.10394 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:20.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.569+0000 7f86b5ffb640 1 -- 172.21.15.57:0/2129168977 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f86ac0772d0 con 0x7f86b80d1080 2023-10-19T14:22:20.747 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.744+0000 7f86bf61a640 1 -- 172.21.15.57:0/2129168977 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f86b809ed00 con 0x7f86b80d1080 2023-10-19T14:22:20.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.745+0000 7f86b5ffb640 1 -- 172.21.15.57:0/2129168977 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v264) v1 ==== 100+0+11 (secure 0 0 0) 0x7f86ac07a8f0 con 0x7f86b80d1080 2023-10-19T14:22:20.748 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 64 2023-10-19T14:22:20.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.747+0000 7f86bf61a640 1 -- 172.21.15.57:0/2129168977 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f86a406afb0 msgr2=0x7f86a406d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:20.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.747+0000 7f86bf61a640 1 --2- 172.21.15.57:0/2129168977 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f86a406afb0 0x7f86a406d470 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f86b0009890 tx=0x7f86b0008040 comp rx=0 tx=0).stop 2023-10-19T14:22:20.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.747+0000 7f86bf61a640 1 -- 172.21.15.57:0/2129168977 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f86b80d1080 msgr2=0x7f86b8158a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:20.750 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.747+0000 7f86bf61a640 1 --2- 172.21.15.57:0/2129168977 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f86b80d1080 0x7f86b8158a40 secure :-1 s=READY pgs=2941 cs=0 l=1 rev1=1 crypto rx=0x7f86ac000c00 tx=0x7f86ac02de10 comp rx=0 tx=0).stop 2023-10-19T14:22:20.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.747+0000 7f86bf61a640 1 -- 172.21.15.57:0/2129168977 shutdown_connections 2023-10-19T14:22:20.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.747+0000 7f86bf61a640 1 --2- 172.21.15.57:0/2129168977 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f86a406afb0 0x7f86a406d470 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:20.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.747+0000 7f86bf61a640 1 --2- 172.21.15.57:0/2129168977 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f86b815b070 0x7f86b815c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:20.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.747+0000 7f86bf61a640 1 --2- 172.21.15.57:0/2129168977 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f86b80d1080 0x7f86b8158a40 unknown :-1 s=CLOSED pgs=2941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:20.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.747+0000 7f86bf61a640 1 --2- 172.21.15.57:0/2129168977 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f86b80cef70 0x7f86b80ca830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:20.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.747+0000 7f86bf61a640 1 -- 172.21.15.57:0/2129168977 >> 172.21.15.57:0/2129168977 conn(0x7f86b808a7e0 msgr2=0x7f86b80ada20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:20.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.747+0000 7f86bf61a640 1 -- 172.21.15.57:0/2129168977 shutdown_connections 2023-10-19T14:22:20.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.747+0000 7f86bf61a640 1 -- 172.21.15.57:0/2129168977 wait complete. 2023-10-19T14:22:20.761 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-19T14:22:20.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.868+0000 7f4eb6090640 1 -- 172.21.15.57:0/1259026393 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4eb0159400 msgr2=0x7f4eb00aa790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:20.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.868+0000 7f4eb6090640 1 --2- 172.21.15.57:0/1259026393 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4eb0159400 0x7f4eb00aa790 secure :-1 s=READY pgs=2943 cs=0 l=1 rev1=1 crypto rx=0x7f4eb006d330 tx=0x7f4ea402be40 comp rx=0 tx=0).stop 2023-10-19T14:22:20.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.868+0000 7f4eb6090640 1 -- 172.21.15.57:0/1259026393 shutdown_connections 2023-10-19T14:22:20.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.868+0000 7f4eb6090640 1 --2- 172.21.15.57:0/1259026393 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4eb0159400 0x7f4eb00aa790 unknown :-1 s=CLOSED pgs=2943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:20.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.868+0000 7f4eb6090640 1 --2- 172.21.15.57:0/1259026393 >> v2:172.21.15.179:3300/0 conn(0x7f4eb01534b0 0x7f4eb00b0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:20.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.868+0000 7f4eb6090640 1 -- 172.21.15.57:0/1259026393 >> 172.21.15.57:0/1259026393 conn(0x7f4eb0069810 msgr2=0x7f4eb0069c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:20.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.868+0000 7f4eb6090640 1 -- 172.21.15.57:0/1259026393 shutdown_connections 2023-10-19T14:22:20.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.868+0000 7f4eb6090640 1 -- 172.21.15.57:0/1259026393 wait complete. 2023-10-19T14:22:20.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.869+0000 7f4eb6090640 1 Processor -- start 2023-10-19T14:22:20.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.869+0000 7f4eb6090640 1 -- start start 2023-10-19T14:22:20.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.870+0000 7f4eb6090640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4eb00aacd0 0x7f4eb015b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:20.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.870+0000 7f4eb6090640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4eb01534b0 0x7f4eb0070cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:20.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.870+0000 7f4eb6090640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4eb0159400 0x7f4eb0158340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:20.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.870+0000 7f4eb6090640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb015ac40 con 0x7f4eb0159400 2023-10-19T14:22:20.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.870+0000 7f4eb6090640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb01f5ad0 con 0x7f4eb01534b0 2023-10-19T14:22:20.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.870+0000 7f4eb6090640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb015aee0 con 0x7f4eb00aacd0 2023-10-19T14:22:20.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.870+0000 7f4eaffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4eb0159400 0x7f4eb0158340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:20.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.870+0000 7f4eaffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4eb0159400 0x7f4eb0158340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46466/0 (socket says 172.21.15.57:46466) 2023-10-19T14:22:20.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.870+0000 7f4eaffff640 1 -- 172.21.15.57:0/2197212891 learned_addr learned my addr 172.21.15.57:0/2197212891 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:22:20.877 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.870+0000 7f4eaf7fe640 1 --2- 172.21.15.57:0/2197212891 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4eb00aacd0 0x7f4eb015b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:20.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.870+0000 7f4eaeffd640 1 --2- 172.21.15.57:0/2197212891 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4eb01534b0 0x7f4eb0070cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:20.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.870+0000 7f4eaffff640 1 -- 172.21.15.57:0/2197212891 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4eb00aacd0 msgr2=0x7f4eb015b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:20.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.870+0000 7f4eaffff640 1 --2- 172.21.15.57:0/2197212891 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4eb00aacd0 0x7f4eb015b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:20.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.870+0000 7f4eaffff640 1 -- 172.21.15.57:0/2197212891 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4eb01534b0 msgr2=0x7f4eb0070cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:20.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.870+0000 7f4eaffff640 1 --2- 172.21.15.57:0/2197212891 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4eb01534b0 0x7f4eb0070cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:20.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.870+0000 7f4eaffff640 1 -- 172.21.15.57:0/2197212891 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ea4009560 con 0x7f4eb0159400 2023-10-19T14:22:20.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.870+0000 7f4eaeffd640 1 --2- 172.21.15.57:0/2197212891 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4eb01534b0 0x7f4eb0070cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:22:20.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.871+0000 7f4eaffff640 1 --2- 172.21.15.57:0/2197212891 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4eb0159400 0x7f4eb0158340 secure :-1 s=READY pgs=2944 cs=0 l=1 rev1=1 crypto rx=0x7f4ea8001560 tx=0x7f4ea800cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:20.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.871+0000 7f4e97fff640 1 -- 172.21.15.57:0/2197212891 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ea8003da0 con 0x7f4eb0159400 2023-10-19T14:22:20.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.871+0000 7f4e97fff640 1 -- 172.21.15.57:0/2197212891 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f4ea8005070 con 0x7f4eb0159400 2023-10-19T14:22:20.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.871+0000 7f4eb6090640 1 -- 172.21.15.57:0/2197212891 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4eb015b1c0 con 0x7f4eb0159400 2023-10-19T14:22:20.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.871+0000 7f4e97fff640 1 -- 172.21.15.57:0/2197212891 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ea8005460 con 0x7f4eb0159400 2023-10-19T14:22:20.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.871+0000 7f4eb6090640 1 -- 172.21.15.57:0/2197212891 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4eb0071230 con 0x7f4eb0159400 2023-10-19T14:22:20.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.872+0000 7f4eb6090640 1 -- 172.21.15.57:0/2197212891 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4eb015b6e0 con 0x7f4eb0159400 2023-10-19T14:22:20.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.872+0000 7f4e97fff640 1 -- 172.21.15.57:0/2197212891 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 149) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f4ea80055c0 con 0x7f4eb0159400 2023-10-19T14:22:20.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.873+0000 7f4e97fff640 1 --2- 172.21.15.57:0/2197212891 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f4e9c06afb0 0x7f4e9c06d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:20.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.873+0000 7f4e97fff640 1 -- 172.21.15.57:0/2197212891 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(264..264 src has 1..264) v4 ==== 6487+0+0 (secure 0 0 0) 0x7f4ea8089610 con 0x7f4eb0159400 2023-10-19T14:22:20.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.873+0000 7f4eaf7fe640 1 --2- 172.21.15.57:0/2197212891 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f4e9c06afb0 0x7f4e9c06d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:20.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.873+0000 7f4eaf7fe640 1 --2- 172.21.15.57:0/2197212891 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f4e9c06afb0 0x7f4e9c06d470 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f4ea000bbe0 tx=0x7f4ea000b390 comp rx=0 tx=0).ready entity=mgr.10394 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:20.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:20.877+0000 7f4e97fff640 1 -- 172.21.15.57:0/2197212891 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4eb015b6e0 con 0x7f4eb0159400 2023-10-19T14:22:21.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.067+0000 7f4eb6090640 1 -- 172.21.15.57:0/2197212891 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f4eb015b6e0 con 0x7f4eb0159400 2023-10-19T14:22:21.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.068+0000 7f4e97fff640 1 -- 172.21.15.57:0/2197212891 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v264) v1 ==== 81+0+593 (secure 0 0 0) 0x7f4ea8017270 con 0x7f4eb0159400 2023-10-19T14:22:21.072 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/1983646445 2023-10-20T14:22:17.865415+0000 2023-10-19T14:22:21.072 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/1840191723 2023-10-20T14:22:17.865415+0000 2023-10-19T14:22:21.072 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/4168916968 2023-10-20T14:22:17.865415+0000 2023-10-19T14:22:21.072 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:6833/1573806521 2023-10-20T14:22:17.865415+0000 2023-10-19T14:22:21.073 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:6832/1573806521 2023-10-20T14:22:17.865415+0000 2023-10-19T14:22:21.073 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/658875312 2023-10-20T14:22:17.865415+0000 2023-10-19T14:22:21.073 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/1799772260 2023-10-20T14:22:17.865415+0000 2023-10-19T14:22:21.073 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/3831670324 2023-10-20T14:22:17.865415+0000 2023-10-19T14:22:21.073 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/90219027 2023-10-20T14:22:17.865415+0000 2023-10-19T14:22:21.074 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/2739598080 2023-10-20T14:22:17.865415+0000 2023-10-19T14:22:21.075 INFO:teuthology.orchestra.run.smithi057.stderr:listed 10 entries 2023-10-19T14:22:21.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.069+0000 7f4eb6090640 1 -- 172.21.15.57:0/2197212891 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f4e9c06afb0 msgr2=0x7f4e9c06d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:21.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.069+0000 7f4eb6090640 1 --2- 172.21.15.57:0/2197212891 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f4e9c06afb0 0x7f4e9c06d470 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f4ea000bbe0 tx=0x7f4ea000b390 comp rx=0 tx=0).stop 2023-10-19T14:22:21.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.069+0000 7f4eb6090640 1 -- 172.21.15.57:0/2197212891 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4eb0159400 msgr2=0x7f4eb0158340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:21.075 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.069+0000 7f4eb6090640 1 --2- 172.21.15.57:0/2197212891 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4eb0159400 0x7f4eb0158340 secure :-1 s=READY pgs=2944 cs=0 l=1 rev1=1 crypto rx=0x7f4ea8001560 tx=0x7f4ea800cc70 comp rx=0 tx=0).stop 2023-10-19T14:22:21.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.070+0000 7f4eb6090640 1 -- 172.21.15.57:0/2197212891 shutdown_connections 2023-10-19T14:22:21.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.070+0000 7f4eb6090640 1 --2- 172.21.15.57:0/2197212891 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f4e9c06afb0 0x7f4e9c06d470 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:21.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.070+0000 7f4eb6090640 1 --2- 172.21.15.57:0/2197212891 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4eb0159400 0x7f4eb0158340 unknown :-1 s=CLOSED pgs=2944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:21.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.070+0000 7f4eb6090640 1 --2- 172.21.15.57:0/2197212891 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4eb01534b0 0x7f4eb0070cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:21.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.070+0000 7f4eb6090640 1 --2- 172.21.15.57:0/2197212891 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4eb00aacd0 0x7f4eb015b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:21.076 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.070+0000 7f4eb6090640 1 -- 172.21.15.57:0/2197212891 >> 172.21.15.57:0/2197212891 conn(0x7f4eb0069810 msgr2=0x7f4eb01e51a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:21.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.070+0000 7f4eb6090640 1 -- 172.21.15.57:0/2197212891 shutdown_connections 2023-10-19T14:22:21.077 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.070+0000 7f4eb6090640 1 -- 172.21.15.57:0/2197212891 wait complete. 2023-10-19T14:22:21.083 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T14:22:21.084 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T14:22:21.100 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T14:22:21.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.246+0000 7f52c0c4b640 1 -- 172.21.15.57:0/1572758294 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f52bc159400 msgr2=0x7f52bc0aa790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:21.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.246+0000 7f52c0c4b640 1 --2- 172.21.15.57:0/1572758294 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f52bc159400 0x7f52bc0aa790 secure :-1 s=READY pgs=2945 cs=0 l=1 rev1=1 crypto rx=0x7f52bc06d330 tx=0x7f52a402bd90 comp rx=0 tx=0).stop 2023-10-19T14:22:21.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.246+0000 7f52c0c4b640 1 -- 172.21.15.57:0/1572758294 shutdown_connections 2023-10-19T14:22:21.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.246+0000 7f52c0c4b640 1 --2- 172.21.15.57:0/1572758294 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f52bc0aacd0 0x7f52bc15c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:21.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.246+0000 7f52c0c4b640 1 --2- 172.21.15.57:0/1572758294 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f52bc159400 0x7f52bc0aa790 unknown :-1 s=CLOSED pgs=2945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:21.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.246+0000 7f52c0c4b640 1 --2- 172.21.15.57:0/1572758294 >> v2:172.21.15.179:3300/0 conn(0x7f52bc1534b0 0x7f52bc0b0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:21.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.246+0000 7f52c0c4b640 1 -- 172.21.15.57:0/1572758294 >> 172.21.15.57:0/1572758294 conn(0x7f52bc069810 msgr2=0x7f52bc069c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:21.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.246+0000 7f52c0c4b640 1 -- 172.21.15.57:0/1572758294 shutdown_connections 2023-10-19T14:22:21.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.246+0000 7f52c0c4b640 1 -- 172.21.15.57:0/1572758294 wait complete. 2023-10-19T14:22:21.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.247+0000 7f52c0c4b640 1 Processor -- start 2023-10-19T14:22:21.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.247+0000 7f52c0c4b640 1 -- start start 2023-10-19T14:22:21.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.248+0000 7f52c0c4b640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f52bc0aacd0 0x7f52bc154e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:21.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.248+0000 7f52c0c4b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f52bc1534b0 0x7f52bc1564d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:21.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.248+0000 7f52c0c4b640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f52bc159400 0x7f52bc142760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:21.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.248+0000 7f52c0c4b640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52bc142ca0 con 0x7f52bc1534b0 2023-10-19T14:22:21.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.248+0000 7f52c0c4b640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52bc155430 con 0x7f52bc159400 2023-10-19T14:22:21.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.248+0000 7f52c0c4b640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52bc155710 con 0x7f52bc0aacd0 2023-10-19T14:22:21.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.248+0000 7f52b3fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f52bc1534b0 0x7f52bc1564d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:21.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.248+0000 7f52b3fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f52bc1534b0 0x7f52bc1564d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46478/0 (socket says 172.21.15.57:46478) 2023-10-19T14:22:21.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.248+0000 7f52b3fff640 1 -- 172.21.15.57:0/1968657471 learned_addr learned my addr 172.21.15.57:0/1968657471 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:22:21.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.248+0000 7f52ba59c640 1 --2- 172.21.15.57:0/1968657471 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f52bc0aacd0 0x7f52bc154e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:21.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.248+0000 7f52bad9d640 1 --2- 172.21.15.57:0/1968657471 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f52bc159400 0x7f52bc142760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:21.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.248+0000 7f52b3fff640 1 -- 172.21.15.57:0/1968657471 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f52bc0aacd0 msgr2=0x7f52bc154e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:21.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.248+0000 7f52b3fff640 1 --2- 172.21.15.57:0/1968657471 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f52bc0aacd0 0x7f52bc154e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:21.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.248+0000 7f52b3fff640 1 -- 172.21.15.57:0/1968657471 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f52bc159400 msgr2=0x7f52bc142760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:21.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.248+0000 7f52b3fff640 1 --2- 172.21.15.57:0/1968657471 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f52bc159400 0x7f52bc142760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:21.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.248+0000 7f52b3fff640 1 -- 172.21.15.57:0/1968657471 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52a4009560 con 0x7f52bc1534b0 2023-10-19T14:22:21.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.248+0000 7f52bad9d640 1 --2- 172.21.15.57:0/1968657471 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f52bc159400 0x7f52bc142760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:22:21.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.248+0000 7f52b3fff640 1 --2- 172.21.15.57:0/1968657471 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f52bc1534b0 0x7f52bc1564d0 secure :-1 s=READY pgs=2946 cs=0 l=1 rev1=1 crypto rx=0x7f52a4006010 tx=0x7f52a402dd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:21.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.249+0000 7f52b2ffd640 1 -- 172.21.15.57:0/1968657471 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52a4005650 con 0x7f52bc1534b0 2023-10-19T14:22:21.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.249+0000 7f52b2ffd640 1 -- 172.21.15.57:0/1968657471 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f52a40038f0 con 0x7f52bc1534b0 2023-10-19T14:22:21.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.249+0000 7f52b2ffd640 1 -- 172.21.15.57:0/1968657471 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52a4003d00 con 0x7f52bc1534b0 2023-10-19T14:22:21.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.249+0000 7f52c0c4b640 1 -- 172.21.15.57:0/1968657471 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52bc1559f0 con 0x7f52bc1534b0 2023-10-19T14:22:21.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.249+0000 7f52c0c4b640 1 -- 172.21.15.57:0/1968657471 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52bc155fb0 con 0x7f52bc1534b0 2023-10-19T14:22:21.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.250+0000 7f52c0c4b640 1 -- 172.21.15.57:0/1968657471 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52bc069100 con 0x7f52bc1534b0 2023-10-19T14:22:21.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.250+0000 7f52b2ffd640 1 -- 172.21.15.57:0/1968657471 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 149) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f52a403e450 con 0x7f52bc1534b0 2023-10-19T14:22:21.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.250+0000 7f52b2ffd640 1 --2- 172.21.15.57:0/1968657471 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f529c06afb0 0x7f529c06d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:21.261 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.251+0000 7f52b2ffd640 1 -- 172.21.15.57:0/1968657471 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(264..264 src has 1..264) v4 ==== 6487+0+0 (secure 0 0 0) 0x7f52a40ac760 con 0x7f52bc1534b0 2023-10-19T14:22:21.262 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.251+0000 7f52ba59c640 1 --2- 172.21.15.57:0/1968657471 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f529c06afb0 0x7f529c06d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:21.262 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.251+0000 7f52ba59c640 1 --2- 172.21.15.57:0/1968657471 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f529c06afb0 0x7f529c06d470 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f52ac005d90 tx=0x7f52ac005d20 comp rx=0 tx=0).ready entity=mgr.10394 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:21.263 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.255+0000 7f52b2ffd640 1 -- 172.21.15.57:0/1968657471 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f52a4077680 con 0x7f52bc1534b0 2023-10-19T14:22:21.438 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.435+0000 7f52c0c4b640 1 -- 172.21.15.57:0/1968657471 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f52bc0b06a0 con 0x7f52bc1534b0 2023-10-19T14:22:21.439 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.436+0000 7f52b2ffd640 1 -- 172.21.15.57:0/1968657471 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v264) v1 ==== 74+0+17407 (secure 0 0 0) 0x7f52a4003e60 con 0x7f52bc1534b0 2023-10-19T14:22:21.440 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:22:21.440 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":264,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T14:22:17.865478+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":36,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_metadata","create_time":"2023-10-19T14:22:00.047912+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"262","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_data","create_time":"2023-10-19T14:22:00.739698+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"263","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.179:0/1983646445":"2023-10-20T14:22:17.865415+0000","172.21.15.179:0/1840191723":"2023-10-20T14:22:17.865415+0000","172.21.15.179:0/4168916968":"2023-10-20T14:22:17.865415+0000","172.21.15.179:6833/1573806521":"2023-10-20T14:22:17.865415+0000","172.21.15.179:6832/1573806521":"2023-10-20T14:22:17.865415+0000","172.21.15.179:0/658875312":"2023-10-20T14:22:17.865415+0000","172.21.15.179:0/1799772260":"2023-10-20T14:22:17.865415+0000","172.21.15.179:0/3831670324":"2023-10-20T14:22:17.865415+0000","172.21.15.179:0/90219027":"2023-10-20T14:22:17.865415+0000","172.21.15.179:0/2739598080":"2023-10-20T14:22:17.865415+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T14:22:21.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.438+0000 7f52c0c4b640 1 -- 172.21.15.57:0/1968657471 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f529c06afb0 msgr2=0x7f529c06d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:21.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.438+0000 7f52c0c4b640 1 --2- 172.21.15.57:0/1968657471 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f529c06afb0 0x7f529c06d470 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f52ac005d90 tx=0x7f52ac005d20 comp rx=0 tx=0).stop 2023-10-19T14:22:21.443 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.438+0000 7f52c0c4b640 1 -- 172.21.15.57:0/1968657471 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f52bc1534b0 msgr2=0x7f52bc1564d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:21.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.438+0000 7f52c0c4b640 1 --2- 172.21.15.57:0/1968657471 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f52bc1534b0 0x7f52bc1564d0 secure :-1 s=READY pgs=2946 cs=0 l=1 rev1=1 crypto rx=0x7f52a4006010 tx=0x7f52a402dd60 comp rx=0 tx=0).stop 2023-10-19T14:22:21.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.438+0000 7f52c0c4b640 1 -- 172.21.15.57:0/1968657471 shutdown_connections 2023-10-19T14:22:21.444 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.438+0000 7f52c0c4b640 1 --2- 172.21.15.57:0/1968657471 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f529c06afb0 0x7f529c06d470 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:21.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.438+0000 7f52c0c4b640 1 --2- 172.21.15.57:0/1968657471 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f52bc159400 0x7f52bc142760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:21.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.438+0000 7f52c0c4b640 1 --2- 172.21.15.57:0/1968657471 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f52bc1534b0 0x7f52bc1564d0 unknown :-1 s=CLOSED pgs=2946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:21.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.438+0000 7f52c0c4b640 1 --2- 172.21.15.57:0/1968657471 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f52bc0aacd0 0x7f52bc154e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:21.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.438+0000 7f52c0c4b640 1 -- 172.21.15.57:0/1968657471 >> 172.21.15.57:0/1968657471 conn(0x7f52bc069810 msgr2=0x7f52bc0ac9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:21.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.438+0000 7f52c0c4b640 1 -- 172.21.15.57:0/1968657471 shutdown_connections 2023-10-19T14:22:21.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.438+0000 7f52c0c4b640 1 -- 172.21.15.57:0/1968657471 wait complete. 2023-10-19T14:22:21.453 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-19T12:44:07.371186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 35, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-19T14:22:00.047912+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '262', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 36, 'pool_name': 'cephfs_data', 'create_time': '2023-10-19T14:22:00.739698+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '263', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-19T14:22:21.455 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-19T14:22:21.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.566+0000 7f498fe90640 1 -- 172.21.15.57:0/3538807741 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4974003970 msgr2=0x7f4974003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:21.576 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.566+0000 7f498fe90640 1 --2- 172.21.15.57:0/3538807741 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4974003970 0x7f4974003d90 secure :-1 s=READY pgs=2948 cs=0 l=1 rev1=1 crypto rx=0x7f4984001680 tx=0x7f49740057f0 comp rx=0 tx=0).stop 2023-10-19T14:22:21.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.566+0000 7f498fe90640 1 -- 172.21.15.57:0/3538807741 shutdown_connections 2023-10-19T14:22:21.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.566+0000 7f498fe90640 1 --2- 172.21.15.57:0/3538807741 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4974003970 0x7f4974003d90 unknown :-1 s=CLOSED pgs=2948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:21.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.566+0000 7f498fe90640 1 -- 172.21.15.57:0/3538807741 >> 172.21.15.57:0/3538807741 conn(0x7f4988069a50 msgr2=0x7f4988069e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:21.578 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.566+0000 7f498fe90640 1 -- 172.21.15.57:0/3538807741 shutdown_connections 2023-10-19T14:22:21.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.566+0000 7f498fe90640 1 -- 172.21.15.57:0/3538807741 wait complete. 2023-10-19T14:22:21.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.567+0000 7f498fe90640 1 Processor -- start 2023-10-19T14:22:21.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.567+0000 7f498fe90640 1 -- start start 2023-10-19T14:22:21.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.568+0000 7f498fe90640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4974003970 0x7f49881f1ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:21.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.568+0000 7f498fe90640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4988070e70 0x7f49880aa8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:21.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.568+0000 7f498fe90640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49881ef150 0x7f49880aae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:21.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.568+0000 7f498fe90640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4988070350 con 0x7f49881ef150 2023-10-19T14:22:21.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.568+0000 7f498fe90640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49881f50a0 con 0x7f4974003970 2023-10-19T14:22:21.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.568+0000 7f498fe90640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49881f52f0 con 0x7f4988070e70 2023-10-19T14:22:21.582 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.568+0000 7f498e42d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49881ef150 0x7f49880aae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:21.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.568+0000 7f498e42d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49881ef150 0x7f49880aae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46502/0 (socket says 172.21.15.57:46502) 2023-10-19T14:22:21.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.568+0000 7f498e42d640 1 -- 172.21.15.57:0/75903762 learned_addr learned my addr 172.21.15.57:0/75903762 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:22:21.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.568+0000 7f498dc2c640 1 --2- 172.21.15.57:0/75903762 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4974003970 0x7f49881f1ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:21.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.568+0000 7f498d42b640 1 --2- 172.21.15.57:0/75903762 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4988070e70 0x7f49880aa8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:21.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.568+0000 7f498e42d640 1 -- 172.21.15.57:0/75903762 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4988070e70 msgr2=0x7f49880aa8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:21.584 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.568+0000 7f498e42d640 1 --2- 172.21.15.57:0/75903762 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4988070e70 0x7f49880aa8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:21.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.568+0000 7f498e42d640 1 -- 172.21.15.57:0/75903762 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4974003970 msgr2=0x7f49881f1ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:21.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.568+0000 7f498e42d640 1 --2- 172.21.15.57:0/75903762 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4974003970 0x7f49881f1ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:21.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.568+0000 7f498e42d640 1 -- 172.21.15.57:0/75903762 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49881f55d0 con 0x7f49881ef150 2023-10-19T14:22:21.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.568+0000 7f498dc2c640 1 --2- 172.21.15.57:0/75903762 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4974003970 0x7f49881f1ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:22:21.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.569+0000 7f498e42d640 1 --2- 172.21.15.57:0/75903762 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49881ef150 0x7f49880aae20 secure :-1 s=READY pgs=2949 cs=0 l=1 rev1=1 crypto rx=0x7f49880b0220 tx=0x7f498000eae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:21.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.569+0000 7f497e7fc640 1 -- 172.21.15.57:0/75903762 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f498000c8f0 con 0x7f49881ef150 2023-10-19T14:22:21.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.569+0000 7f497e7fc640 1 -- 172.21.15.57:0/75903762 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f498000cb40 con 0x7f49881ef150 2023-10-19T14:22:21.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.569+0000 7f498fe90640 1 -- 172.21.15.57:0/75903762 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49881f58b0 con 0x7f49881ef150 2023-10-19T14:22:21.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.569+0000 7f497e7fc640 1 -- 172.21.15.57:0/75903762 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4980011460 con 0x7f49881ef150 2023-10-19T14:22:21.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.569+0000 7f498fe90640 1 -- 172.21.15.57:0/75903762 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49881f5e70 con 0x7f49881ef150 2023-10-19T14:22:21.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.570+0000 7f498fe90640 1 -- 172.21.15.57:0/75903762 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4988138340 con 0x7f49881ef150 2023-10-19T14:22:21.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.570+0000 7f497e7fc640 1 -- 172.21.15.57:0/75903762 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 149) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f49800115c0 con 0x7f49881ef150 2023-10-19T14:22:21.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.571+0000 7f497e7fc640 1 --2- 172.21.15.57:0/75903762 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f496c06afb0 0x7f496c06d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:21.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.571+0000 7f497e7fc640 1 -- 172.21.15.57:0/75903762 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(264..264 src has 1..264) v4 ==== 6487+0+0 (secure 0 0 0) 0x7f498008a2b0 con 0x7f49881ef150 2023-10-19T14:22:21.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.571+0000 7f498dc2c640 1 --2- 172.21.15.57:0/75903762 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f496c06afb0 0x7f496c06d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:21.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.572+0000 7f498dc2c640 1 --2- 172.21.15.57:0/75903762 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f496c06afb0 0x7f496c06d470 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f4984034000 tx=0x7f498402e040 comp rx=0 tx=0).ready entity=mgr.10394 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:21.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.575+0000 7f497e7fc640 1 -- 172.21.15.57:0/75903762 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4980054d90 con 0x7f49881ef150 2023-10-19T14:22:21.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.751+0000 7f498fe90640 1 -- 172.21.15.57:0/75903762 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f498804db50 con 0x7f49881ef150 2023-10-19T14:22:21.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.751+0000 7f497e7fc640 1 -- 172.21.15.57:0/75903762 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v264) v1 ==== 93+0+10 (secure 0 0 0) 0x7f49800583b0 con 0x7f49881ef150 2023-10-19T14:22:21.755 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-19T14:22:21.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.753+0000 7f498fe90640 1 -- 172.21.15.57:0/75903762 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f496c06afb0 msgr2=0x7f496c06d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:21.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.753+0000 7f498fe90640 1 --2- 172.21.15.57:0/75903762 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f496c06afb0 0x7f496c06d470 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f4984034000 tx=0x7f498402e040 comp rx=0 tx=0).stop 2023-10-19T14:22:21.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.753+0000 7f498fe90640 1 -- 172.21.15.57:0/75903762 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49881ef150 msgr2=0x7f49880aae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:21.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.753+0000 7f498fe90640 1 --2- 172.21.15.57:0/75903762 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49881ef150 0x7f49880aae20 secure :-1 s=READY pgs=2949 cs=0 l=1 rev1=1 crypto rx=0x7f49880b0220 tx=0x7f498000eae0 comp rx=0 tx=0).stop 2023-10-19T14:22:21.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.753+0000 7f498fe90640 1 -- 172.21.15.57:0/75903762 shutdown_connections 2023-10-19T14:22:21.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.753+0000 7f498fe90640 1 --2- 172.21.15.57:0/75903762 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f496c06afb0 0x7f496c06d470 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:21.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.753+0000 7f498fe90640 1 --2- 172.21.15.57:0/75903762 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49881ef150 0x7f49880aae20 unknown :-1 s=CLOSED pgs=2949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:21.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.753+0000 7f498fe90640 1 --2- 172.21.15.57:0/75903762 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f4988070e70 0x7f49880aa8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:21.758 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.753+0000 7f498fe90640 1 --2- 172.21.15.57:0/75903762 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4974003970 0x7f49881f1ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:21.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.753+0000 7f498fe90640 1 -- 172.21.15.57:0/75903762 >> 172.21.15.57:0/75903762 conn(0x7f4988069a50 msgr2=0x7f4988140400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:21.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.753+0000 7f498fe90640 1 -- 172.21.15.57:0/75903762 shutdown_connections 2023-10-19T14:22:21.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.753+0000 7f498fe90640 1 -- 172.21.15.57:0/75903762 wait complete. 2023-10-19T14:22:21.767 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-19T14:22:21.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.876+0000 7f615816e640 1 -- 172.21.15.57:0/335650988 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f615015b770 msgr2=0x7f615015bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:21.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.876+0000 7f615816e640 1 --2- 172.21.15.57:0/335650988 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f615015b770 0x7f615015bbc0 secure :-1 s=READY pgs=2950 cs=0 l=1 rev1=1 crypto rx=0x7f6148005fd0 tx=0x7f614802c2e0 comp rx=0 tx=0).stop 2023-10-19T14:22:21.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.876+0000 7f615816e640 1 -- 172.21.15.57:0/335650988 shutdown_connections 2023-10-19T14:22:21.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.876+0000 7f615816e640 1 --2- 172.21.15.57:0/335650988 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f615015b770 0x7f615015bbc0 unknown :-1 s=CLOSED pgs=2950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:21.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.877+0000 7f615816e640 1 --2- 172.21.15.57:0/335650988 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f61500cbc00 0x7f615006f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:21.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.877+0000 7f615816e640 1 -- 172.21.15.57:0/335650988 >> 172.21.15.57:0/335650988 conn(0x7f6150069a50 msgr2=0x7f6150069e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:21.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.877+0000 7f615816e640 1 -- 172.21.15.57:0/335650988 shutdown_connections 2023-10-19T14:22:21.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.877+0000 7f615816e640 1 -- 172.21.15.57:0/335650988 wait complete. 2023-10-19T14:22:21.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.878+0000 7f615816e640 1 Processor -- start 2023-10-19T14:22:21.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.878+0000 7f615816e640 1 -- start start 2023-10-19T14:22:21.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.878+0000 7f615816e640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f61500cbc00 0x7f61501ebba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:21.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.878+0000 7f615816e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f61501415c0 0x7f61501f2560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:21.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.878+0000 7f615816e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f615015b770 0x7f615015a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:21.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.878+0000 7f615816e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6150138e80 con 0x7f61501415c0 2023-10-19T14:22:21.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.878+0000 7f615816e640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61501f2aa0 con 0x7f615015b770 2023-10-19T14:22:21.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.879+0000 7f615816e640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f615013e830 con 0x7f61500cbc00 2023-10-19T14:22:21.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.879+0000 7f6155709640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f61501415c0 0x7f61501f2560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:21.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.879+0000 7f6155709640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f61501415c0 0x7f61501f2560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46514/0 (socket says 172.21.15.57:46514) 2023-10-19T14:22:21.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.879+0000 7f6155709640 1 -- 172.21.15.57:0/2956586796 learned_addr learned my addr 172.21.15.57:0/2956586796 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:22:21.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.879+0000 7f615670b640 1 --2- 172.21.15.57:0/2956586796 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f615015b770 0x7f615015a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:21.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.879+0000 7f6155f0a640 1 --2- 172.21.15.57:0/2956586796 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f61500cbc00 0x7f61501ebba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:21.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.879+0000 7f6155709640 1 -- 172.21.15.57:0/2956586796 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f61500cbc00 msgr2=0x7f61501ebba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:21.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.879+0000 7f6155709640 1 --2- 172.21.15.57:0/2956586796 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f61500cbc00 0x7f61501ebba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:21.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.879+0000 7f6155709640 1 -- 172.21.15.57:0/2956586796 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f615015b770 msgr2=0x7f615015a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:21.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.879+0000 7f6155709640 1 --2- 172.21.15.57:0/2956586796 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f615015b770 0x7f615015a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:21.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.879+0000 7f6155709640 1 -- 172.21.15.57:0/2956586796 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6148009900 con 0x7f61501415c0 2023-10-19T14:22:21.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.879+0000 7f6155709640 1 --2- 172.21.15.57:0/2956586796 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f61501415c0 0x7f61501f2560 secure :-1 s=READY pgs=2951 cs=0 l=1 rev1=1 crypto rx=0x7f614000d950 tx=0x7f614000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:21.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.879+0000 7f61467fc640 1 -- 172.21.15.57:0/2956586796 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6140009c30 con 0x7f61501415c0 2023-10-19T14:22:21.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.879+0000 7f61467fc640 1 -- 172.21.15.57:0/2956586796 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f6140009e80 con 0x7f61501415c0 2023-10-19T14:22:21.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.879+0000 7f615816e640 1 -- 172.21.15.57:0/2956586796 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f615013dc20 con 0x7f61501415c0 2023-10-19T14:22:21.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.879+0000 7f61467fc640 1 -- 172.21.15.57:0/2956586796 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61400106c0 con 0x7f61501415c0 2023-10-19T14:22:21.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.879+0000 7f615816e640 1 -- 172.21.15.57:0/2956586796 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f615013e1e0 con 0x7f61501415c0 2023-10-19T14:22:21.888 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.880+0000 7f615816e640 1 -- 172.21.15.57:0/2956586796 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f615004db50 con 0x7f61501415c0 2023-10-19T14:22:21.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.881+0000 7f61467fc640 1 -- 172.21.15.57:0/2956586796 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 149) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f6140010820 con 0x7f61501415c0 2023-10-19T14:22:21.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.881+0000 7f61467fc640 1 --2- 172.21.15.57:0/2956586796 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f613406afe0 0x7f613406d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:21.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.881+0000 7f61467fc640 1 -- 172.21.15.57:0/2956586796 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(264..264 src has 1..264) v4 ==== 6487+0+0 (secure 0 0 0) 0x7f6140089fc0 con 0x7f61501415c0 2023-10-19T14:22:21.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.882+0000 7f6155f0a640 1 --2- 172.21.15.57:0/2956586796 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f613406afe0 0x7f613406d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:21.889 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.882+0000 7f6155f0a640 1 --2- 172.21.15.57:0/2956586796 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f613406afe0 0x7f613406d4a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f61500678f0 tx=0x7f614c009040 comp rx=0 tx=0).ready entity=mgr.10394 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:21.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:21.886+0000 7f61467fc640 1 -- 172.21.15.57:0/2956586796 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f61400bcca0 con 0x7f61501415c0 2023-10-19T14:22:22.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.060+0000 7f615816e640 1 -- 172.21.15.57:0/2956586796 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f61500cb290 con 0x7f61501415c0 2023-10-19T14:22:22.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.060+0000 7f61467fc640 1 -- 172.21.15.57:0/2956586796 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v264) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6140017020 con 0x7f61501415c0 2023-10-19T14:22:22.063 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2023-10-19T14:22:22.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.062+0000 7f615816e640 1 -- 172.21.15.57:0/2956586796 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f613406afe0 msgr2=0x7f613406d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:22.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.062+0000 7f615816e640 1 --2- 172.21.15.57:0/2956586796 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f613406afe0 0x7f613406d4a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f61500678f0 tx=0x7f614c009040 comp rx=0 tx=0).stop 2023-10-19T14:22:22.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.062+0000 7f615816e640 1 -- 172.21.15.57:0/2956586796 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f61501415c0 msgr2=0x7f61501f2560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:22.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.062+0000 7f615816e640 1 --2- 172.21.15.57:0/2956586796 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f61501415c0 0x7f61501f2560 secure :-1 s=READY pgs=2951 cs=0 l=1 rev1=1 crypto rx=0x7f614000d950 tx=0x7f614000de10 comp rx=0 tx=0).stop 2023-10-19T14:22:22.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.062+0000 7f615816e640 1 -- 172.21.15.57:0/2956586796 shutdown_connections 2023-10-19T14:22:22.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.062+0000 7f615816e640 1 --2- 172.21.15.57:0/2956586796 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f613406afe0 0x7f613406d4a0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:22.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.062+0000 7f615816e640 1 --2- 172.21.15.57:0/2956586796 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f615015b770 0x7f615015a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:22.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.062+0000 7f615816e640 1 --2- 172.21.15.57:0/2956586796 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f61501415c0 0x7f61501f2560 unknown :-1 s=CLOSED pgs=2951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:22.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.062+0000 7f615816e640 1 --2- 172.21.15.57:0/2956586796 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f61500cbc00 0x7f61501ebba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:22.067 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.062+0000 7f615816e640 1 -- 172.21.15.57:0/2956586796 >> 172.21.15.57:0/2956586796 conn(0x7f6150069a50 msgr2=0x7f61500ae890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:22.067 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.062+0000 7f615816e640 1 -- 172.21.15.57:0/2956586796 shutdown_connections 2023-10-19T14:22:22.067 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.062+0000 7f615816e640 1 -- 172.21.15.57:0/2956586796 wait complete. 2023-10-19T14:22:22.076 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-19T14:22:22.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.185+0000 7f464b261640 1 -- 172.21.15.57:0/620086136 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f464415c3e0 msgr2=0x7f46440cc7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:22.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.185+0000 7f464b261640 1 --2- 172.21.15.57:0/620086136 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f464415c3e0 0x7f46440cc7b0 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f463c005fd0 tx=0x7f463c02c200 comp rx=0 tx=0).stop 2023-10-19T14:22:22.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.186+0000 7f464b261640 1 -- 172.21.15.57:0/620086136 shutdown_connections 2023-10-19T14:22:22.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.186+0000 7f464b261640 1 --2- 172.21.15.57:0/620086136 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f464415c3e0 0x7f46440cc7b0 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:22.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.186+0000 7f464b261640 1 -- 172.21.15.57:0/620086136 >> 172.21.15.57:0/620086136 conn(0x7f46440cebe0 msgr2=0x7f46440ceff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:22.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.186+0000 7f464b261640 1 -- 172.21.15.57:0/620086136 shutdown_connections 2023-10-19T14:22:22.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.186+0000 7f464b261640 1 -- 172.21.15.57:0/620086136 wait complete. 2023-10-19T14:22:22.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.187+0000 7f464b261640 1 Processor -- start 2023-10-19T14:22:22.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.187+0000 7f464b261640 1 -- start start 2023-10-19T14:22:22.192 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.188+0000 7f464b261640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f46440cf700 0x7f4644156120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:22.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.188+0000 7f464b261640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f464415c3e0 0x7f4644156660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:22.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.188+0000 7f464b261640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f464415c7c0 0x7f4644155060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:22.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.188+0000 7f464b261640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4644156ba0 con 0x7f464415c3e0 2023-10-19T14:22:22.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.188+0000 7f464b261640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46440aba20 con 0x7f46440cf700 2023-10-19T14:22:22.193 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.188+0000 7f464b261640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4644152870 con 0x7f464415c7c0 2023-10-19T14:22:22.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.188+0000 7f4643fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f464415c3e0 0x7f4644156660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:22.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.188+0000 7f4643fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f464415c3e0 0x7f4644156660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46524/0 (socket says 172.21.15.57:46524) 2023-10-19T14:22:22.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.188+0000 7f4643fff640 1 -- 172.21.15.57:0/5322930 learned_addr learned my addr 172.21.15.57:0/5322930 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:22:22.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.188+0000 7f46497fe640 1 --2- 172.21.15.57:0/5322930 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f464415c7c0 0x7f4644155060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:22.194 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.188+0000 7f4648ffd640 1 --2- 172.21.15.57:0/5322930 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f46440cf700 0x7f4644156120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:22.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.188+0000 7f4643fff640 1 -- 172.21.15.57:0/5322930 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f464415c7c0 msgr2=0x7f4644155060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:22.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.188+0000 7f4643fff640 1 --2- 172.21.15.57:0/5322930 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f464415c7c0 0x7f4644155060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:22.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.188+0000 7f4643fff640 1 -- 172.21.15.57:0/5322930 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f46440cf700 msgr2=0x7f4644156120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:22.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.188+0000 7f4643fff640 1 --2- 172.21.15.57:0/5322930 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f46440cf700 0x7f4644156120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:22.195 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.188+0000 7f4643fff640 1 -- 172.21.15.57:0/5322930 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f463c009840 con 0x7f464415c3e0 2023-10-19T14:22:22.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.188+0000 7f4643fff640 1 --2- 172.21.15.57:0/5322930 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f464415c3e0 0x7f4644156660 secure :-1 s=READY pgs=2953 cs=0 l=1 rev1=1 crypto rx=0x7f463800c440 tx=0x7f463800c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:22.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.188+0000 7f46417fa640 1 -- 172.21.15.57:0/5322930 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46380052d0 con 0x7f464415c3e0 2023-10-19T14:22:22.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.188+0000 7f46417fa640 1 -- 172.21.15.57:0/5322930 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f4638007020 con 0x7f464415c3e0 2023-10-19T14:22:22.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.188+0000 7f464b261640 1 -- 172.21.15.57:0/5322930 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46441518a0 con 0x7f464415c3e0 2023-10-19T14:22:22.196 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.188+0000 7f46417fa640 1 -- 172.21.15.57:0/5322930 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4638007470 con 0x7f464415c3e0 2023-10-19T14:22:22.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.188+0000 7f464b261640 1 -- 172.21.15.57:0/5322930 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4644151f40 con 0x7f464415c3e0 2023-10-19T14:22:22.197 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.190+0000 7f464b261640 1 -- 172.21.15.57:0/5322930 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4644066980 con 0x7f464415c3e0 2023-10-19T14:22:22.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.190+0000 7f46417fa640 1 -- 172.21.15.57:0/5322930 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 149) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f46380067d0 con 0x7f464415c3e0 2023-10-19T14:22:22.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.190+0000 7f46417fa640 1 --2- 172.21.15.57:0/5322930 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f463006afe0 0x7f463006d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:22.198 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.190+0000 7f46417fa640 1 -- 172.21.15.57:0/5322930 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(264..264 src has 1..264) v4 ==== 6487+0+0 (secure 0 0 0) 0x7f4638089190 con 0x7f464415c3e0 2023-10-19T14:22:22.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.191+0000 7f4648ffd640 1 --2- 172.21.15.57:0/5322930 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f463006afe0 0x7f463006d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:22.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.191+0000 7f4648ffd640 1 --2- 172.21.15.57:0/5322930 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f463006afe0 0x7f463006d4a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f463c035000 tx=0x7f463c004ab0 comp rx=0 tx=0).ready entity=mgr.10394 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:22.199 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.195+0000 7f46417fa640 1 -- 172.21.15.57:0/5322930 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f4638053d10 con 0x7f464415c3e0 2023-10-19T14:22:22.371 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.368+0000 7f464b261640 1 -- 172.21.15.57:0/5322930 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f46440ca6d0 con 0x7f464415c3e0 2023-10-19T14:22:22.374 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.371+0000 7f46417fa640 1 -- 172.21.15.57:0/5322930 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v264) v1 ==== 100+0+11 (secure 0 0 0) 0x7f4638057330 con 0x7f464415c3e0 2023-10-19T14:22:22.374 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 64 2023-10-19T14:22:22.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.373+0000 7f464b261640 1 -- 172.21.15.57:0/5322930 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f463006afe0 msgr2=0x7f463006d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:22.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.373+0000 7f464b261640 1 --2- 172.21.15.57:0/5322930 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f463006afe0 0x7f463006d4a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f463c035000 tx=0x7f463c004ab0 comp rx=0 tx=0).stop 2023-10-19T14:22:22.376 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.373+0000 7f464b261640 1 -- 172.21.15.57:0/5322930 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f464415c3e0 msgr2=0x7f4644156660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:22.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.373+0000 7f464b261640 1 --2- 172.21.15.57:0/5322930 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f464415c3e0 0x7f4644156660 secure :-1 s=READY pgs=2953 cs=0 l=1 rev1=1 crypto rx=0x7f463800c440 tx=0x7f463800c900 comp rx=0 tx=0).stop 2023-10-19T14:22:22.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.373+0000 7f464b261640 1 -- 172.21.15.57:0/5322930 shutdown_connections 2023-10-19T14:22:22.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.373+0000 7f464b261640 1 --2- 172.21.15.57:0/5322930 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f463006afe0 0x7f463006d4a0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:22.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.373+0000 7f464b261640 1 --2- 172.21.15.57:0/5322930 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f464415c7c0 0x7f4644155060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:22.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.373+0000 7f464b261640 1 --2- 172.21.15.57:0/5322930 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f464415c3e0 0x7f4644156660 unknown :-1 s=CLOSED pgs=2953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:22.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.373+0000 7f464b261640 1 --2- 172.21.15.57:0/5322930 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f46440cf700 0x7f4644156120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:22.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.373+0000 7f464b261640 1 -- 172.21.15.57:0/5322930 >> 172.21.15.57:0/5322930 conn(0x7f46440cebe0 msgr2=0x7f4644159080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:22.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.373+0000 7f464b261640 1 -- 172.21.15.57:0/5322930 shutdown_connections 2023-10-19T14:22:22.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.373+0000 7f464b261640 1 -- 172.21.15.57:0/5322930 wait complete. 2023-10-19T14:22:22.389 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-19T14:22:22.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.497+0000 7f852927c640 1 -- 172.21.15.57:0/2856484643 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8524159400 msgr2=0x7f85240aa790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:22.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.497+0000 7f852927c640 1 --2- 172.21.15.57:0/2856484643 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8524159400 0x7f85240aa790 secure :-1 s=READY pgs=2954 cs=0 l=1 rev1=1 crypto rx=0x7f852406d330 tx=0x7f850c02bd90 comp rx=0 tx=0).stop 2023-10-19T14:22:22.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.497+0000 7f852927c640 1 -- 172.21.15.57:0/2856484643 shutdown_connections 2023-10-19T14:22:22.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.497+0000 7f852927c640 1 --2- 172.21.15.57:0/2856484643 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f85240aacd0 0x7f852415c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:22.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.497+0000 7f852927c640 1 --2- 172.21.15.57:0/2856484643 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8524159400 0x7f85240aa790 unknown :-1 s=CLOSED pgs=2954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:22.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.497+0000 7f852927c640 1 --2- 172.21.15.57:0/2856484643 >> v2:172.21.15.179:3300/0 conn(0x7f85241534b0 0x7f85240b0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:22.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.497+0000 7f852927c640 1 -- 172.21.15.57:0/2856484643 >> 172.21.15.57:0/2856484643 conn(0x7f8524069810 msgr2=0x7f8524069c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:22.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.497+0000 7f852927c640 1 -- 172.21.15.57:0/2856484643 shutdown_connections 2023-10-19T14:22:22.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.497+0000 7f852927c640 1 -- 172.21.15.57:0/2856484643 wait complete. 2023-10-19T14:22:22.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.499+0000 7f852927c640 1 Processor -- start 2023-10-19T14:22:22.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.499+0000 7f852927c640 1 -- start start 2023-10-19T14:22:22.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.499+0000 7f852927c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85240aacd0 0x7f852409a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:22.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.499+0000 7f852927c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f85241534b0 0x7f852413cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:22.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.499+0000 7f852927c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8524159400 0x7f85241559d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:22.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.499+0000 7f852927c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85240989f0 con 0x7f85240aacd0 2023-10-19T14:22:22.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.499+0000 7f852927c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8524098cd0 con 0x7f85241534b0 2023-10-19T14:22:22.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.499+0000 7f852927c640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8524098fb0 con 0x7f8524159400 2023-10-19T14:22:22.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.500+0000 7f8522ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85240aacd0 0x7f852409a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:22.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.500+0000 7f8522ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85240aacd0 0x7f852409a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46544/0 (socket says 172.21.15.57:46544) 2023-10-19T14:22:22.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.500+0000 7f8522ffd640 1 -- 172.21.15.57:0/1519389611 learned_addr learned my addr 172.21.15.57:0/1519389611 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:22:22.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.500+0000 7f85227fc640 1 --2- 172.21.15.57:0/1519389611 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f85241534b0 0x7f852413cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:22.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.500+0000 7f85237fe640 1 --2- 172.21.15.57:0/1519389611 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8524159400 0x7f85241559d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:22.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.500+0000 7f8522ffd640 1 -- 172.21.15.57:0/1519389611 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8524159400 msgr2=0x7f85241559d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:22.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.500+0000 7f8522ffd640 1 --2- 172.21.15.57:0/1519389611 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8524159400 0x7f85241559d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:22.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.500+0000 7f8522ffd640 1 -- 172.21.15.57:0/1519389611 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f85241534b0 msgr2=0x7f852413cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:22.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.500+0000 7f8522ffd640 1 --2- 172.21.15.57:0/1519389611 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f85241534b0 0x7f852413cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:22.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.500+0000 7f8522ffd640 1 -- 172.21.15.57:0/1519389611 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f850c009560 con 0x7f85240aacd0 2023-10-19T14:22:22.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.500+0000 7f85227fc640 1 --2- 172.21.15.57:0/1519389611 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f85241534b0 0x7f852413cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:22:22.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.500+0000 7f85237fe640 1 --2- 172.21.15.57:0/1519389611 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8524159400 0x7f85241559d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:22:22.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.500+0000 7f8522ffd640 1 --2- 172.21.15.57:0/1519389611 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85240aacd0 0x7f852409a320 secure :-1 s=READY pgs=2955 cs=0 l=1 rev1=1 crypto rx=0x7f851c00d950 tx=0x7f851c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:22.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.500+0000 7f851b7fe640 1 -- 172.21.15.57:0/1519389611 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f851c009c30 con 0x7f85240aacd0 2023-10-19T14:22:22.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.500+0000 7f851b7fe640 1 -- 172.21.15.57:0/1519389611 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f851c009e80 con 0x7f85240aacd0 2023-10-19T14:22:22.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.500+0000 7f852927c640 1 -- 172.21.15.57:0/1519389611 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8524099290 con 0x7f85240aacd0 2023-10-19T14:22:22.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.500+0000 7f852927c640 1 -- 172.21.15.57:0/1519389611 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f852409a860 con 0x7f85240aacd0 2023-10-19T14:22:22.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.500+0000 7f851b7fe640 1 -- 172.21.15.57:0/1519389611 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f851c0106c0 con 0x7f85240aacd0 2023-10-19T14:22:22.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.501+0000 7f852927c640 1 -- 172.21.15.57:0/1519389611 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8524069100 con 0x7f85240aacd0 2023-10-19T14:22:22.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.502+0000 7f851b7fe640 1 -- 172.21.15.57:0/1519389611 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 150) v1 ==== 89405+0+0 (secure 0 0 0) 0x7f851c010820 con 0x7f85240aacd0 2023-10-19T14:22:22.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.502+0000 7f851b7fe640 1 --2- 172.21.15.57:0/1519389611 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f850406b040 0x7f850406d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:22.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.502+0000 7f851b7fe640 1 -- 172.21.15.57:0/1519389611 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(264..264 src has 1..264) v4 ==== 6487+0+0 (secure 0 0 0) 0x7f851c08a2a0 con 0x7f85240aacd0 2023-10-19T14:22:22.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.502+0000 7f85227fc640 1 --2- 172.21.15.57:0/1519389611 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f850406b040 0x7f850406d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:22.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.503+0000 7f85227fc640 1 --2- 172.21.15.57:0/1519389611 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f850406b040 0x7f850406d500 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f850c035000 tx=0x7f850c004730 comp rx=0 tx=0).ready entity=mgr.10394 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:22.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.506+0000 7f851b7fe640 1 -- 172.21.15.57:0/1519389611 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f851c054c10 con 0x7f85240aacd0 2023-10-19T14:22:22.697 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.694+0000 7f852927c640 1 -- 172.21.15.57:0/1519389611 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f85240afd50 con 0x7f85240aacd0 2023-10-19T14:22:22.698 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.694+0000 7f851b7fe640 1 -- 172.21.15.57:0/1519389611 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v264) v1 ==== 81+0+593 (secure 0 0 0) 0x7f851c020270 con 0x7f85240aacd0 2023-10-19T14:22:22.698 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/1983646445 2023-10-20T14:22:17.865415+0000 2023-10-19T14:22:22.698 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/1840191723 2023-10-20T14:22:17.865415+0000 2023-10-19T14:22:22.699 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/4168916968 2023-10-20T14:22:17.865415+0000 2023-10-19T14:22:22.699 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:6833/1573806521 2023-10-20T14:22:17.865415+0000 2023-10-19T14:22:22.699 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:6832/1573806521 2023-10-20T14:22:17.865415+0000 2023-10-19T14:22:22.699 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/658875312 2023-10-20T14:22:17.865415+0000 2023-10-19T14:22:22.699 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/1799772260 2023-10-20T14:22:17.865415+0000 2023-10-19T14:22:22.700 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/3831670324 2023-10-20T14:22:17.865415+0000 2023-10-19T14:22:22.700 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/90219027 2023-10-20T14:22:17.865415+0000 2023-10-19T14:22:22.700 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/2739598080 2023-10-20T14:22:17.865415+0000 2023-10-19T14:22:22.701 INFO:teuthology.orchestra.run.smithi057.stderr:listed 10 entries 2023-10-19T14:22:22.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.696+0000 7f852927c640 1 -- 172.21.15.57:0/1519389611 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f850406b040 msgr2=0x7f850406d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:22.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.696+0000 7f852927c640 1 --2- 172.21.15.57:0/1519389611 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f850406b040 0x7f850406d500 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f850c035000 tx=0x7f850c004730 comp rx=0 tx=0).stop 2023-10-19T14:22:22.701 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.696+0000 7f852927c640 1 -- 172.21.15.57:0/1519389611 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85240aacd0 msgr2=0x7f852409a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:22.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.696+0000 7f852927c640 1 --2- 172.21.15.57:0/1519389611 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85240aacd0 0x7f852409a320 secure :-1 s=READY pgs=2955 cs=0 l=1 rev1=1 crypto rx=0x7f851c00d950 tx=0x7f851c00de10 comp rx=0 tx=0).stop 2023-10-19T14:22:22.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.696+0000 7f852927c640 1 -- 172.21.15.57:0/1519389611 shutdown_connections 2023-10-19T14:22:22.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.696+0000 7f852927c640 1 --2- 172.21.15.57:0/1519389611 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f850406b040 0x7f850406d500 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:22.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.696+0000 7f852927c640 1 --2- 172.21.15.57:0/1519389611 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f8524159400 0x7f85241559d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:22.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.696+0000 7f852927c640 1 --2- 172.21.15.57:0/1519389611 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f85241534b0 0x7f852413cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:22.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.696+0000 7f852927c640 1 --2- 172.21.15.57:0/1519389611 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f85240aacd0 0x7f852409a320 unknown :-1 s=CLOSED pgs=2955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:22.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.696+0000 7f852927c640 1 -- 172.21.15.57:0/1519389611 >> 172.21.15.57:0/1519389611 conn(0x7f8524069810 msgr2=0x7f8524152710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:22.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.696+0000 7f852927c640 1 -- 172.21.15.57:0/1519389611 shutdown_connections 2023-10-19T14:22:22.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:22.696+0000 7f852927c640 1 -- 172.21.15.57:0/1519389611 wait complete. 2023-10-19T14:22:22.710 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi057.front.sepia.ceph.com... 2023-10-19T14:22:22.710 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:22:22.711 DEBUG:teuthology.orchestra.run.smithi057:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-10-19T14:22:22.740 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi057.stderr:ceph-fuse[163529]: fuse finished with error 0 and tester_r 0 2023-10-19T14:22:22.742 INFO:teuthology.orchestra.run:waiting for 300 2023-10-19T14:22:22.838 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi057.front.sepia.ceph.com 2023-10-19T14:22:22.839 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:22:22.839 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-19T14:22:22.855 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T14:22:22.856 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T14:22:22.911 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055614,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=358 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248260k,nr_inodes=812065,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-19T14:22:22.911 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-19T14:22:23.060 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:23.057+0000 7f9acf490640 1 -- 172.21.15.57:0/3520931021 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9ac81534b0 msgr2=0x7f9ac80b0bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:23.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:23.057+0000 7f9acf490640 1 --2- 172.21.15.57:0/3520931021 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9ac81534b0 0x7f9ac80b0bd0 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f9ab8005fd0 tx=0x7f9ab802c170 comp rx=0 tx=0).stop 2023-10-19T14:22:23.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:23.057+0000 7f9acf490640 1 -- 172.21.15.57:0/3520931021 shutdown_connections 2023-10-19T14:22:23.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:23.057+0000 7f9acf490640 1 --2- 172.21.15.57:0/3520931021 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9ac81534b0 0x7f9ac80b0bd0 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:23.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:23.057+0000 7f9acf490640 1 -- 172.21.15.57:0/3520931021 >> 172.21.15.57:0/3520931021 conn(0x7f9ac8069810 msgr2=0x7f9ac8069c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:23.062 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:23.057+0000 7f9acf490640 1 -- 172.21.15.57:0/3520931021 shutdown_connections 2023-10-19T14:22:23.062 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:23.057+0000 7f9acf490640 1 -- 172.21.15.57:0/3520931021 wait complete. 2023-10-19T14:22:23.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:23.058+0000 7f9acf490640 1 Processor -- start 2023-10-19T14:22:23.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:23.059+0000 7f9acf490640 1 -- start start 2023-10-19T14:22:23.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:23.059+0000 7f9acf490640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9ac80aaea0 0x7f9ac81539b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:23.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:23.059+0000 7f9acf490640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9ac81534b0 0x7f9ac8154fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:23.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:23.059+0000 7f9acf490640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9ac8159400 0x7f9ac815bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:23.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:23.059+0000 7f9acf490640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac81efa40 con 0x7f9ac80aaea0 2023-10-19T14:22:23.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:23.059+0000 7f9acf490640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac8154940 con 0x7f9ac81534b0 2023-10-19T14:22:23.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:23.059+0000 7f9acf490640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac81f3410 con 0x7f9ac8159400 2023-10-19T14:22:23.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:23.059+0000 7f9acd22c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9ac80aaea0 0x7f9ac81539b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:23.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:23.059+0000 7f9acd22c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9ac80aaea0 0x7f9ac81539b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46550/0 (socket says 172.21.15.57:46550) 2023-10-19T14:22:23.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:23.059+0000 7f9acd22c640 1 -- 172.21.15.57:0/1131232150 learned_addr learned my addr 172.21.15.57:0/1131232150 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:22:23.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:23.060+0000 7f9acca2b640 1 --2- 172.21.15.57:0/1131232150 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9ac81534b0 0x7f9ac8154fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:23.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:23.060+0000 7f9acda2d640 1 --2- 172.21.15.57:0/1131232150 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9ac8159400 0x7f9ac815bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:23.067 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:23.060+0000 7f9acd22c640 1 -- 172.21.15.57:0/1131232150 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9ac8159400 msgr2=0x7f9ac815bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:23.067 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:23.060+0000 7f9acd22c640 1 --2- 172.21.15.57:0/1131232150 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9ac8159400 0x7f9ac815bf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:23.067 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:23.060+0000 7f9acd22c640 1 -- 172.21.15.57:0/1131232150 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9ac81534b0 msgr2=0x7f9ac8154fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:23.067 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:23.060+0000 7f9acd22c640 1 --2- 172.21.15.57:0/1131232150 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9ac81534b0 0x7f9ac8154fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:23.067 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:23.060+0000 7f9acd22c640 1 -- 172.21.15.57:0/1131232150 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ab8009840 con 0x7f9ac80aaea0 2023-10-19T14:22:23.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:23.060+0000 7f9acca2b640 1 --2- 172.21.15.57:0/1131232150 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9ac81534b0 0x7f9ac8154fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:22:23.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:23.060+0000 7f9acda2d640 1 --2- 172.21.15.57:0/1131232150 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9ac8159400 0x7f9ac815bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:22:23.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:23.060+0000 7f9acd22c640 1 --2- 172.21.15.57:0/1131232150 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9ac80aaea0 0x7f9ac81539b0 secure :-1 s=READY pgs=2957 cs=0 l=1 rev1=1 crypto rx=0x7f9ab8005fd0 tx=0x7f9ab80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:23.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:23.060+0000 7f9ac5ffb640 1 -- 172.21.15.57:0/1131232150 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab8003e80 con 0x7f9ac80aaea0 2023-10-19T14:22:23.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:23.060+0000 7f9ac5ffb640 1 -- 172.21.15.57:0/1131232150 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f9ab80047e0 con 0x7f9ac80aaea0 2023-10-19T14:22:23.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:23.060+0000 7f9acf490640 1 -- 172.21.15.57:0/1131232150 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ac81f36f0 con 0x7f9ac80aaea0 2023-10-19T14:22:23.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:23.060+0000 7f9ac5ffb640 1 -- 172.21.15.57:0/1131232150 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab8034800 con 0x7f9ac80aaea0 2023-10-19T14:22:23.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:23.060+0000 7f9acf490640 1 -- 172.21.15.57:0/1131232150 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ac81f3cb0 con 0x7f9ac80aaea0 2023-10-19T14:22:23.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:23.061+0000 7f9acf490640 1 -- 172.21.15.57:0/1131232150 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ac80abc40 con 0x7f9ac80aaea0 2023-10-19T14:22:23.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:23.062+0000 7f9ac5ffb640 1 -- 172.21.15.57:0/1131232150 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 150) v1 ==== 89405+0+0 (secure 0 0 0) 0x7f9ab8034960 con 0x7f9ac80aaea0 2023-10-19T14:22:23.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:23.062+0000 7f9ac5ffb640 1 --2- 172.21.15.57:0/1131232150 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f9ab406b040 0x7f9ab406d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:23.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:23.062+0000 7f9ac5ffb640 1 -- 172.21.15.57:0/1131232150 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(264..264 src has 1..264) v4 ==== 6487+0+0 (secure 0 0 0) 0x7f9ab80ace10 con 0x7f9ac80aaea0 2023-10-19T14:22:23.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:23.062+0000 7f9acca2b640 1 --2- 172.21.15.57:0/1131232150 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f9ab406b040 0x7f9ab406d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:23.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:23.063+0000 7f9acca2b640 1 --2- 172.21.15.57:0/1131232150 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f9ab406b040 0x7f9ab406d500 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f9ac815cb10 tx=0x7f9abc009040 comp rx=0 tx=0).ready entity=mgr.10394 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:23.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:23.066+0000 7f9ac5ffb640 1 -- 172.21.15.57:0/1131232150 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f9ab80778b0 con 0x7f9ac80aaea0 2023-10-19T14:22:23.280 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:23.277+0000 7f9acf490640 1 -- 172.21.15.57:0/1131232150 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f9ac8069100 con 0x7f9ac80aaea0 2023-10-19T14:22:23.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:23.278+0000 7f9ac5ffb640 1 -- 172.21.15.57:0/1131232150 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v116) v1 ==== 53+0+73 (secure 0 0 0) 0x7f9ab807aed0 con 0x7f9ac80aaea0 2023-10-19T14:22:23.281 INFO:teuthology.orchestra.run.smithi057.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-19T14:22:23.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:23.279+0000 7f9acf490640 1 -- 172.21.15.57:0/1131232150 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f9ab406b040 msgr2=0x7f9ab406d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:23.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:23.279+0000 7f9acf490640 1 --2- 172.21.15.57:0/1131232150 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f9ab406b040 0x7f9ab406d500 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f9ac815cb10 tx=0x7f9abc009040 comp rx=0 tx=0).stop 2023-10-19T14:22:23.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:23.279+0000 7f9acf490640 1 -- 172.21.15.57:0/1131232150 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9ac80aaea0 msgr2=0x7f9ac81539b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:23.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:23.279+0000 7f9acf490640 1 --2- 172.21.15.57:0/1131232150 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9ac80aaea0 0x7f9ac81539b0 secure :-1 s=READY pgs=2957 cs=0 l=1 rev1=1 crypto rx=0x7f9ab8005fd0 tx=0x7f9ab80042f0 comp rx=0 tx=0).stop 2023-10-19T14:22:23.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:23.280+0000 7f9acf490640 1 -- 172.21.15.57:0/1131232150 shutdown_connections 2023-10-19T14:22:23.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:23.280+0000 7f9acf490640 1 --2- 172.21.15.57:0/1131232150 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f9ab406b040 0x7f9ab406d500 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:23.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:23.280+0000 7f9acf490640 1 --2- 172.21.15.57:0/1131232150 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f9ac8159400 0x7f9ac815bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:23.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:23.280+0000 7f9acf490640 1 --2- 172.21.15.57:0/1131232150 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9ac81534b0 0x7f9ac8154fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:23.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:23.280+0000 7f9acf490640 1 --2- 172.21.15.57:0/1131232150 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9ac80aaea0 0x7f9ac81539b0 unknown :-1 s=CLOSED pgs=2957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:23.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:23.280+0000 7f9acf490640 1 -- 172.21.15.57:0/1131232150 >> 172.21.15.57:0/1131232150 conn(0x7f9ac8069810 msgr2=0x7f9ac8156ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:23.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:23.280+0000 7f9acf490640 1 -- 172.21.15.57:0/1131232150 shutdown_connections 2023-10-19T14:22:23.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:23.280+0000 7f9acf490640 1 -- 172.21.15.57:0/1131232150 wait complete. 2023-10-19T14:22:23.293 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-10-19T14:22:23.294 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-19T14:22:23.294 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi057.front.sepia.ceph.com 2023-10-19T14:22:23.295 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-19T14:22:23.295 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-19T14:22:23.295 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-10-19T14:22:23.296 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-10-19T14:22:23.296 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-19T14:22:23.296 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-19T14:22:23.297 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:22:23.297 DEBUG:teuthology.orchestra.run.smithi057:> ip addr 2023-10-19T14:22:23.313 INFO:teuthology.orchestra.run.smithi057.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-19T14:22:23.313 INFO:teuthology.orchestra.run.smithi057.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-19T14:22:23.313 INFO:teuthology.orchestra.run.smithi057.stdout: inet 127.0.0.1/8 scope host lo 2023-10-19T14:22:23.313 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T14:22:23.314 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 ::1/128 scope host 2023-10-19T14:22:23.314 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T14:22:23.314 INFO:teuthology.orchestra.run.smithi057.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-19T14:22:23.314 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:6b:ff:86 brd ff:ff:ff:ff:ff:ff 2023-10-19T14:22:23.314 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp7s0f0 2023-10-19T14:22:23.315 INFO:teuthology.orchestra.run.smithi057.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-19T14:22:23.315 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:6b:ff:87 brd ff:ff:ff:ff:ff:ff 2023-10-19T14:22:23.315 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp7s0f1 2023-10-19T14:22:23.315 INFO:teuthology.orchestra.run.smithi057.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-19T14:22:23.315 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:bd:15:62 brd ff:ff:ff:ff:ff:ff 2023-10-19T14:22:23.316 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp1s0f0 2023-10-19T14:22:23.316 INFO:teuthology.orchestra.run.smithi057.stdout: inet 172.21.15.57/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-19T14:22:23.316 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft 36767sec preferred_lft 36767sec 2023-10-19T14:22:23.316 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 fe80::ec4:7aff:febd:1562/64 scope link noprefixroute 2023-10-19T14:22:23.316 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T14:22:23.317 INFO:teuthology.orchestra.run.smithi057.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-19T14:22:23.317 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:bd:15:63 brd ff:ff:ff:ff:ff:ff 2023-10-19T14:22:23.317 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp1s0f1 2023-10-19T14:22:23.317 INFO:teuthology.orchestra.run.smithi057.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-19T14:22:23.317 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0e:fe:f1:b5:41:59 brd ff:ff:ff:ff:ff:ff 2023-10-19T14:22:23.318 INFO:teuthology.orchestra.run.smithi057.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-19T14:22:23.318 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T14:22:23.318 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 fe80::64b1:a2ff:feb3:b08/64 scope link 2023-10-19T14:22:23.318 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T14:22:23.318 INFO:teuthology.orchestra.run.smithi057.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-19T14:22:23.319 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0e:fe:f1:b5:41:59 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T14:22:23.319 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 fe80::cfe:f1ff:feb5:4159/64 scope link 2023-10-19T14:22:23.319 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T14:22:23.319 INFO:teuthology.orchestra.run.smithi057.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-19T14:22:23.319 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 62:53:72:9d:52:71 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-19T14:22:23.319 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 fe80::6053:72ff:fe9d:5271/64 scope link 2023-10-19T14:22:23.320 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2023-10-19T14:22:23.320 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:22:23.321 DEBUG:teuthology.orchestra.run.smithi057:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-19T14:22:23.382 INFO:teuthology.orchestra.run.smithi057.stdout:1 2023-10-19T14:22:23.383 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:22:23.384 DEBUG:teuthology.orchestra.run.smithi057:> route 2023-10-19T14:22:23.440 INFO:teuthology.orchestra.run.smithi057.stdout:Kernel IP routing table 2023-10-19T14:22:23.440 INFO:teuthology.orchestra.run.smithi057.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-19T14:22:23.440 INFO:teuthology.orchestra.run.smithi057.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-19T14:22:23.441 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-19T14:22:23.441 INFO:teuthology.orchestra.run.smithi057.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-19T14:22:23.441 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:22:23.442 DEBUG:teuthology.orchestra.run.smithi057:> (cd / && exec bash -c ' 2023-10-19T14:22:23.442 DEBUG:teuthology.orchestra.run.smithi057:> set -e 2023-10-19T14:22:23.442 DEBUG:teuthology.orchestra.run.smithi057:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-19T14:22:23.443 DEBUG:teuthology.orchestra.run.smithi057:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-19T14:22:23.443 DEBUG:teuthology.orchestra.run.smithi057:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-19T14:22:23.443 DEBUG:teuthology.orchestra.run.smithi057:> ') 2023-10-19T14:22:23.547 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:22:23.548 DEBUG:teuthology.orchestra.run.smithi057:> ip netns list 2023-10-19T14:22:23.563 INFO:teuthology.orchestra.run.smithi057.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-19T14:22:23.563 INFO:teuthology.orchestra.run.smithi057.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-19T14:22:23.564 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-19T14:22:23.564 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:22:23.564 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip link set brx.0 up 2023-10-19T14:22:23.634 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-10-19T14:22:23.635 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-19T14:22:23.635 DEBUG:teuthology.orchestra.run.smithi057:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-19T14:22:23.690 INFO:teuthology.orchestra.run.smithi057.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-19T14:22:23.690 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-19T14:22:23.691 DEBUG:teuthology.orchestra.run.smithi057:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-19T14:22:23.745 DEBUG:teuthology.orchestra.run.smithi057:> sudo modprobe fuse 2023-10-19T14:22:23.817 DEBUG:teuthology.orchestra.run.smithi057:> cat /proc/self/mounts | awk '{print $2}' 2023-10-19T14:22:23.874 INFO:teuthology.orchestra.run.smithi057.stdout:/proc 2023-10-19T14:22:23.874 INFO:teuthology.orchestra.run.smithi057.stdout:/sys 2023-10-19T14:22:23.874 INFO:teuthology.orchestra.run.smithi057.stdout:/dev 2023-10-19T14:22:23.875 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/security 2023-10-19T14:22:23.875 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/shm 2023-10-19T14:22:23.875 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/pts 2023-10-19T14:22:23.875 INFO:teuthology.orchestra.run.smithi057.stdout:/run 2023-10-19T14:22:23.875 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/cgroup 2023-10-19T14:22:23.876 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/pstore 2023-10-19T14:22:23.876 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/bpf 2023-10-19T14:22:23.876 INFO:teuthology.orchestra.run.smithi057.stdout:/ 2023-10-19T14:22:23.876 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/selinux 2023-10-19T14:22:23.876 INFO:teuthology.orchestra.run.smithi057.stdout:/proc/sys/fs/binfmt_misc 2023-10-19T14:22:23.877 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/hugepages 2023-10-19T14:22:23.877 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/mqueue 2023-10-19T14:22:23.877 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/debug 2023-10-19T14:22:23.877 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/tracing 2023-10-19T14:22:23.877 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/fuse/connections 2023-10-19T14:22:23.877 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/config 2023-10-19T14:22:23.878 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-sysctl.service 2023-10-19T14:22:23.878 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-10-19T14:22:23.878 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-10-19T14:22:23.878 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/nfs/rpc_pipefs 2023-10-19T14:22:23.879 INFO:teuthology.orchestra.run.smithi057.stdout:/run/user/1000 2023-10-19T14:22:23.879 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph 2023-10-19T14:22:23.879 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-0 2023-10-19T14:22:23.879 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-1 2023-10-19T14:22:23.879 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-2 2023-10-19T14:22:23.879 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-3 2023-10-19T14:22:23.880 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns 2023-10-19T14:22:23.880 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T14:22:23.880 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T14:22:23.880 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-19T14:22:23.880 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-19T14:22:23.881 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:22:23.881 DEBUG:teuthology.orchestra.run.smithi057:> ls /sys/fs/fuse/connections 2023-10-19T14:22:23.929 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-10-19T14:22:23.930 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-10-19T14:22:23.972 DEBUG:teuthology.orchestra.run.smithi057:> sudo modprobe fuse 2023-10-19T14:22:24.004 DEBUG:teuthology.orchestra.run.smithi057:> cat /proc/self/mounts | awk '{print $2}' 2023-10-19T14:22:24.059 INFO:teuthology.orchestra.run.smithi057.stdout:/proc 2023-10-19T14:22:24.059 INFO:teuthology.orchestra.run.smithi057.stdout:/sys 2023-10-19T14:22:24.059 INFO:teuthology.orchestra.run.smithi057.stdout:/dev 2023-10-19T14:22:24.060 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/security 2023-10-19T14:22:24.060 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/shm 2023-10-19T14:22:24.060 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/pts 2023-10-19T14:22:24.060 INFO:teuthology.orchestra.run.smithi057.stdout:/run 2023-10-19T14:22:24.060 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/cgroup 2023-10-19T14:22:24.060 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/pstore 2023-10-19T14:22:24.061 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/bpf 2023-10-19T14:22:24.061 INFO:teuthology.orchestra.run.smithi057.stdout:/ 2023-10-19T14:22:24.061 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/selinux 2023-10-19T14:22:24.061 INFO:teuthology.orchestra.run.smithi057.stdout:/proc/sys/fs/binfmt_misc 2023-10-19T14:22:24.061 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/hugepages 2023-10-19T14:22:24.062 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/mqueue 2023-10-19T14:22:24.062 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/debug 2023-10-19T14:22:24.062 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/tracing 2023-10-19T14:22:24.062 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/fuse/connections 2023-10-19T14:22:24.063 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/config 2023-10-19T14:22:24.063 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-sysctl.service 2023-10-19T14:22:24.063 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-10-19T14:22:24.063 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-10-19T14:22:24.063 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/nfs/rpc_pipefs 2023-10-19T14:22:24.064 INFO:teuthology.orchestra.run.smithi057.stdout:/run/user/1000 2023-10-19T14:22:24.064 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph 2023-10-19T14:22:24.064 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-0 2023-10-19T14:22:24.064 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-1 2023-10-19T14:22:24.064 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-2 2023-10-19T14:22:24.065 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph/osd/ceph-3 2023-10-19T14:22:24.065 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns 2023-10-19T14:22:24.065 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T14:22:24.065 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T14:22:24.065 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-19T14:22:24.066 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-19T14:22:24.067 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:22:24.067 DEBUG:teuthology.orchestra.run.smithi057:> ls /sys/fs/fuse/connections 2023-10-19T14:22:24.068 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi057.stderr:2023-10-19T14:22:24.061+0000 7fa748b5a480 -1 init, newargv = 0x55e785a03c40 newargc=15 2023-10-19T14:22:24.068 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi057.stderr:ceph-fuse[164782]: starting ceph client 2023-10-19T14:22:24.069 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi057.stderr:ceph-fuse[164782]: starting fuse 2023-10-19T14:22:24.083 INFO:teuthology.orchestra.run.smithi057.stdout:43 2023-10-19T14:22:24.084 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2023-10-19T14:22:24.084 DEBUG:teuthology.orchestra.run.smithi057:> sudo stdin-killer -- python3 -c ' 2023-10-19T14:22:24.085 DEBUG:teuthology.orchestra.run.smithi057:> import glob 2023-10-19T14:22:24.085 DEBUG:teuthology.orchestra.run.smithi057:> import re 2023-10-19T14:22:24.085 DEBUG:teuthology.orchestra.run.smithi057:> import os 2023-10-19T14:22:24.086 DEBUG:teuthology.orchestra.run.smithi057:> import subprocess 2023-10-19T14:22:24.086 DEBUG:teuthology.orchestra.run.smithi057:> 2023-10-19T14:22:24.087 DEBUG:teuthology.orchestra.run.smithi057:> def _find_admin_socket(client_name): 2023-10-19T14:22:24.087 DEBUG:teuthology.orchestra.run.smithi057:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-10-19T14:22:24.087 DEBUG:teuthology.orchestra.run.smithi057:> files = glob.glob(asok_path) 2023-10-19T14:22:24.088 DEBUG:teuthology.orchestra.run.smithi057:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-10-19T14:22:24.088 DEBUG:teuthology.orchestra.run.smithi057:> 2023-10-19T14:22:24.089 DEBUG:teuthology.orchestra.run.smithi057:> # Given a non-glob path, it better be there 2023-10-19T14:22:24.089 DEBUG:teuthology.orchestra.run.smithi057:> if "*" not in asok_path: 2023-10-19T14:22:24.089 DEBUG:teuthology.orchestra.run.smithi057:> assert(len(files) == 1) 2023-10-19T14:22:24.090 DEBUG:teuthology.orchestra.run.smithi057:> return files[0] 2023-10-19T14:22:24.090 DEBUG:teuthology.orchestra.run.smithi057:> 2023-10-19T14:22:24.091 DEBUG:teuthology.orchestra.run.smithi057:> for f in files: 2023-10-19T14:22:24.091 DEBUG:teuthology.orchestra.run.smithi057:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-10-19T14:22:24.091 DEBUG:teuthology.orchestra.run.smithi057:> if os.path.exists("/proc/{0}".format(pid)): 2023-10-19T14:22:24.092 DEBUG:teuthology.orchestra.run.smithi057:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-10-19T14:22:24.092 DEBUG:teuthology.orchestra.run.smithi057:> contents = proc_f.read() 2023-10-19T14:22:24.092 DEBUG:teuthology.orchestra.run.smithi057:> if mountpoint in contents: 2023-10-19T14:22:24.093 DEBUG:teuthology.orchestra.run.smithi057:> return f 2023-10-19T14:22:24.093 DEBUG:teuthology.orchestra.run.smithi057:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-10-19T14:22:24.093 DEBUG:teuthology.orchestra.run.smithi057:> 2023-10-19T14:22:24.094 DEBUG:teuthology.orchestra.run.smithi057:> print(_find_admin_socket("client.0")) 2023-10-19T14:22:24.094 DEBUG:teuthology.orchestra.run.smithi057:> ' 2023-10-19T14:22:24.209 INFO:teuthology.orchestra.run.smithi057.stdout:/var/run/ceph/ceph-client.0.164782.asok 2023-10-19T14:22:24.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2023-10-19T14:22:24.220 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.164782.asok 2023-10-19T14:22:24.221 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:22:24.222 DEBUG:teuthology.orchestra.run.smithi057:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.164782.asok status 2023-10-19T14:22:24.364 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2023-10-19T14:22:24.365 INFO:teuthology.orchestra.run.smithi057.stdout: "metadata": { 2023-10-19T14:22:24.365 INFO:teuthology.orchestra.run.smithi057.stdout: "ceph_sha1": "f7420d7f8e55ceea786dcbf536c338ce920c21eb", 2023-10-19T14:22:24.365 INFO:teuthology.orchestra.run.smithi057.stdout: "ceph_version": "ceph version 18.0.0-6786-gf7420d7f (f7420d7f8e55ceea786dcbf536c338ce920c21eb) reef (dev)", 2023-10-19T14:22:24.365 INFO:teuthology.orchestra.run.smithi057.stdout: "entity_id": "0", 2023-10-19T14:22:24.365 INFO:teuthology.orchestra.run.smithi057.stdout: "hostname": "smithi057", 2023-10-19T14:22:24.366 INFO:teuthology.orchestra.run.smithi057.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-10-19T14:22:24.366 INFO:teuthology.orchestra.run.smithi057.stdout: "pid": "164782", 2023-10-19T14:22:24.366 INFO:teuthology.orchestra.run.smithi057.stdout: "root": "/" 2023-10-19T14:22:24.366 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:22:24.367 INFO:teuthology.orchestra.run.smithi057.stdout: "dentry_count": 0, 2023-10-19T14:22:24.367 INFO:teuthology.orchestra.run.smithi057.stdout: "dentry_pinned_count": 0, 2023-10-19T14:22:24.367 INFO:teuthology.orchestra.run.smithi057.stdout: "id": 13701, 2023-10-19T14:22:24.367 INFO:teuthology.orchestra.run.smithi057.stdout: "inst": { 2023-10-19T14:22:24.367 INFO:teuthology.orchestra.run.smithi057.stdout: "name": { 2023-10-19T14:22:24.367 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "client", 2023-10-19T14:22:24.368 INFO:teuthology.orchestra.run.smithi057.stdout: "num": 13701 2023-10-19T14:22:24.368 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:22:24.368 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": { 2023-10-19T14:22:24.368 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "v1", 2023-10-19T14:22:24.369 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": "192.168.0.1:0", 2023-10-19T14:22:24.369 INFO:teuthology.orchestra.run.smithi057.stdout: "nonce": 2188203205 2023-10-19T14:22:24.369 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-10-19T14:22:24.369 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:22:24.369 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": { 2023-10-19T14:22:24.370 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "v1", 2023-10-19T14:22:24.370 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": "192.168.0.1:0", 2023-10-19T14:22:24.370 INFO:teuthology.orchestra.run.smithi057.stdout: "nonce": 2188203205 2023-10-19T14:22:24.370 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-10-19T14:22:24.370 INFO:teuthology.orchestra.run.smithi057.stdout: "inst_str": "client.13701 192.168.0.1:0/2188203205", 2023-10-19T14:22:24.371 INFO:teuthology.orchestra.run.smithi057.stdout: "addr_str": "192.168.0.1:0/2188203205", 2023-10-19T14:22:24.371 INFO:teuthology.orchestra.run.smithi057.stdout: "inode_count": 1, 2023-10-19T14:22:24.371 INFO:teuthology.orchestra.run.smithi057.stdout: "mds_epoch": 116, 2023-10-19T14:22:24.371 INFO:teuthology.orchestra.run.smithi057.stdout: "osd_epoch": 264, 2023-10-19T14:22:24.371 INFO:teuthology.orchestra.run.smithi057.stdout: "osd_epoch_barrier": 0, 2023-10-19T14:22:24.372 INFO:teuthology.orchestra.run.smithi057.stdout: "blocklisted": false, 2023-10-19T14:22:24.372 INFO:teuthology.orchestra.run.smithi057.stdout: "fs_name": "cephfs" 2023-10-19T14:22:24.372 INFO:teuthology.orchestra.run.smithi057.stdout:} 2023-10-19T14:22:24.373 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:22:24.374 DEBUG:teuthology.orchestra.run.smithi057:> stat --file-system '--printf=%T 2023-10-19T14:22:24.374 DEBUG:teuthology.orchestra.run.smithi057:> ' -- /home/ubuntu/cephtest/mnt.0 2023-10-19T14:22:24.390 INFO:teuthology.orchestra.run.smithi057.stdout:fuseblk 2023-10-19T14:22:24.391 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-10-19T14:22:24.391 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:22:24.392 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-19T14:22:24.461 INFO:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2023-10-19-14_23_00 in ~41s... 2023-10-19T14:22:24.462 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1m'], expected_retention: [{'m': 1}] 2023-10-19T14:22:24.462 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --format json snap_test_dir1 2023-10-19T14:22:24.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.611+0000 7ff6a0ba0640 1 -- 172.21.15.57:0/2005378367 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff6840047c0 msgr2=0x7ff684004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:24.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.611+0000 7ff6a0ba0640 1 --2- 172.21.15.57:0/2005378367 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff6840047c0 0x7ff684004c40 secure :-1 s=READY pgs=2962 cs=0 l=1 rev1=1 crypto rx=0x7ff690006fa0 tx=0x7ff6840057f0 comp rx=0 tx=0).stop 2023-10-19T14:22:24.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.611+0000 7ff69bfff640 1 -- 172.21.15.57:0/2005378367 reap_dead start 2023-10-19T14:22:24.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.611+0000 7ff6a0ba0640 1 -- 172.21.15.57:0/2005378367 shutdown_connections 2023-10-19T14:22:24.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.611+0000 7ff6a0ba0640 1 -- 172.21.15.57:0/2005378367 >> 172.21.15.57:0/2005378367 conn(0x7ff69c0718e0 msgr2=0x7ff69c097a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:24.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.611+0000 7ff6a0ba0640 1 -- 172.21.15.57:0/2005378367 shutdown_connections 2023-10-19T14:22:24.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.611+0000 7ff6a0ba0640 1 -- 172.21.15.57:0/2005378367 wait complete. 2023-10-19T14:22:24.618 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.612+0000 7ff6a0ba0640 1 Processor -- start 2023-10-19T14:22:24.618 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.612+0000 7ff6a0ba0640 1 -- start start 2023-10-19T14:22:24.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.613+0000 7ff6a0ba0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff6840047c0 0x7ff69c14c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:24.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.613+0000 7ff6a0ba0640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff69c148840 0x7ff69c13f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:24.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.613+0000 7ff6a0ba0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff69c14b5e0 0x7ff69c14ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:24.619 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.613+0000 7ff6a0ba0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff69c14f470 con 0x7ff69c14b5e0 2023-10-19T14:22:24.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.613+0000 7ff6a0ba0640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff69c14b1c0 con 0x7ff6840047c0 2023-10-19T14:22:24.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.613+0000 7ff6a0ba0640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff69c13fbc0 con 0x7ff69c148840 2023-10-19T14:22:24.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.613+0000 7ff69bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff69c14b5e0 0x7ff69c14ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:24.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.613+0000 7ff69bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff69c14b5e0 0x7ff69c14ba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46570/0 (socket says 172.21.15.57:46570) 2023-10-19T14:22:24.620 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.613+0000 7ff69bfff640 1 -- 172.21.15.57:0/3358517049 learned_addr learned my addr 172.21.15.57:0/3358517049 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:22:24.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.613+0000 7ff69b7fe640 1 --2- 172.21.15.57:0/3358517049 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff6840047c0 0x7ff69c14c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:24.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.613+0000 7ff69affd640 1 --2- 172.21.15.57:0/3358517049 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff69c148840 0x7ff69c13f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:24.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.613+0000 7ff69bfff640 1 -- 172.21.15.57:0/3358517049 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff69c148840 msgr2=0x7ff69c13f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:24.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.613+0000 7ff69bfff640 1 --2- 172.21.15.57:0/3358517049 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff69c148840 0x7ff69c13f680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:24.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.613+0000 7ff69bfff640 1 -- 172.21.15.57:0/3358517049 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff6840047c0 msgr2=0x7ff69c14c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:24.621 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.613+0000 7ff69bfff640 1 --2- 172.21.15.57:0/3358517049 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff6840047c0 0x7ff69c14c460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:24.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.613+0000 7ff69bfff640 1 -- 172.21.15.57:0/3358517049 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff69c14a540 con 0x7ff69c14b5e0 2023-10-19T14:22:24.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.613+0000 7ff69b7fe640 1 --2- 172.21.15.57:0/3358517049 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff6840047c0 0x7ff69c14c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:22:24.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.613+0000 7ff69bfff640 1 --2- 172.21.15.57:0/3358517049 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff69c14b5e0 0x7ff69c14ba60 secure :-1 s=READY pgs=2963 cs=0 l=1 rev1=1 crypto rx=0x7ff694010a70 tx=0x7ff694010f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:24.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.614+0000 7ff683fff640 1 -- 172.21.15.57:0/3358517049 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff69400ed80 con 0x7ff69c14b5e0 2023-10-19T14:22:24.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.614+0000 7ff683fff640 1 -- 172.21.15.57:0/3358517049 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7ff694004b20 con 0x7ff69c14b5e0 2023-10-19T14:22:24.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.614+0000 7ff6a0ba0640 1 -- 172.21.15.57:0/3358517049 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff69c14a820 con 0x7ff69c14b5e0 2023-10-19T14:22:24.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.614+0000 7ff683fff640 1 -- 172.21.15.57:0/3358517049 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff694003970 con 0x7ff69c14b5e0 2023-10-19T14:22:24.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.614+0000 7ff6a0ba0640 1 -- 172.21.15.57:0/3358517049 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff69c14ade0 con 0x7ff69c14b5e0 2023-10-19T14:22:24.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.615+0000 7ff6a0ba0640 1 -- 172.21.15.57:0/3358517049 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff69c14afa0 con 0x7ff69c14b5e0 2023-10-19T14:22:24.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.615+0000 7ff683fff640 1 -- 172.21.15.57:0/3358517049 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 150) v1 ==== 89405+0+0 (secure 0 0 0) 0x7ff694005020 con 0x7ff69c14b5e0 2023-10-19T14:22:24.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.616+0000 7ff683fff640 1 --2- 172.21.15.57:0/3358517049 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7ff68806b040 0x7ff68806d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:24.624 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.616+0000 7ff683fff640 1 -- 172.21.15.57:0/3358517049 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(264..264 src has 1..264) v4 ==== 6487+0+0 (secure 0 0 0) 0x7ff69408aad0 con 0x7ff69c14b5e0 2023-10-19T14:22:24.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.616+0000 7ff69b7fe640 1 --2- 172.21.15.57:0/3358517049 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7ff68806b040 0x7ff68806d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:24.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.617+0000 7ff69b7fe640 1 --2- 172.21.15.57:0/3358517049 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7ff68806b040 0x7ff68806d500 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff68c0065f0 tx=0x7ff68c006b30 comp rx=0 tx=0).ready entity=mgr.10394 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:24.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.620+0000 7ff683fff640 1 -- 172.21.15.57:0/3358517049 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7ff694055470 con 0x7ff69c14b5e0 2023-10-19T14:22:24.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.815+0000 7ff6a0ba0640 1 -- 172.21.15.57:0/3358517049 --> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff69c066980 con 0x7ff68806b040 2023-10-19T14:22:24.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.823+0000 7ff683fff640 1 -- 172.21.15.57:0/3358517049 <== mgr.10394 v2:172.21.15.179:6832/2005622276 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+72 (secure 0 0 0) 0x7ff69c066980 con 0x7ff68806b040 2023-10-19T14:22:24.826 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:22:24.826 INFO:teuthology.orchestra.run.smithi057.stdout:{"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{"m": 1}]} 2023-10-19T14:22:24.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.825+0000 7ff6a0ba0640 1 -- 172.21.15.57:0/3358517049 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7ff68806b040 msgr2=0x7ff68806d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:24.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.825+0000 7ff6a0ba0640 1 --2- 172.21.15.57:0/3358517049 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7ff68806b040 0x7ff68806d500 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff68c0065f0 tx=0x7ff68c006b30 comp rx=0 tx=0).stop 2023-10-19T14:22:24.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.825+0000 7ff6a0ba0640 1 -- 172.21.15.57:0/3358517049 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff69c14b5e0 msgr2=0x7ff69c14ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:24.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.825+0000 7ff6a0ba0640 1 --2- 172.21.15.57:0/3358517049 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff69c14b5e0 0x7ff69c14ba60 secure :-1 s=READY pgs=2963 cs=0 l=1 rev1=1 crypto rx=0x7ff694010a70 tx=0x7ff694010f30 comp rx=0 tx=0).stop 2023-10-19T14:22:24.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.825+0000 7ff6a0ba0640 1 -- 172.21.15.57:0/3358517049 shutdown_connections 2023-10-19T14:22:24.829 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.825+0000 7ff6a0ba0640 1 --2- 172.21.15.57:0/3358517049 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7ff68806b040 0x7ff68806d500 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:24.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.825+0000 7ff6a0ba0640 1 --2- 172.21.15.57:0/3358517049 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff69c14b5e0 0x7ff69c14ba60 unknown :-1 s=CLOSED pgs=2963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:24.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.825+0000 7ff6a0ba0640 1 --2- 172.21.15.57:0/3358517049 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7ff69c148840 0x7ff69c13f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:24.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.825+0000 7ff6a0ba0640 1 --2- 172.21.15.57:0/3358517049 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff6840047c0 0x7ff69c14c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:24.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.825+0000 7ff6a0ba0640 1 -- 172.21.15.57:0/3358517049 >> 172.21.15.57:0/3358517049 conn(0x7ff69c0718e0 msgr2=0x7ff69c0b07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:24.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.825+0000 7ff6a0ba0640 1 -- 172.21.15.57:0/3358517049 shutdown_connections 2023-10-19T14:22:24.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.825+0000 7ff6a0ba0640 1 -- 172.21.15.57:0/3358517049 wait complete. 2023-10-19T14:22:24.839 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{"m": 1}]} 2023-10-19T14:22:24.840 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': 'snap_test_dir1', 'schedule': ['1m'], 'retention': [{'m': 1}]} 2023-10-19T14:22:24.840 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove snap_test_dir1 2023-10-19T14:22:24.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.949+0000 7fb1ab61f640 1 -- 172.21.15.57:0/731845695 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb1a40cbf30 msgr2=0x7fb1a406bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:24.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.949+0000 7fb1ab61f640 1 --2- 172.21.15.57:0/731845695 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb1a40cbf30 0x7fb1a406bd30 secure :-1 s=READY pgs=2965 cs=0 l=1 rev1=1 crypto rx=0x7fb1a0005fd0 tx=0x7fb1a002bd90 comp rx=0 tx=0).stop 2023-10-19T14:22:24.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.949+0000 7fb1ab61f640 1 -- 172.21.15.57:0/731845695 shutdown_connections 2023-10-19T14:22:24.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.949+0000 7fb1ab61f640 1 --2- 172.21.15.57:0/731845695 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb1a40cbf30 0x7fb1a406bd30 unknown :-1 s=CLOSED pgs=2965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:24.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.949+0000 7fb1ab61f640 1 -- 172.21.15.57:0/731845695 >> 172.21.15.57:0/731845695 conn(0x7fb1a4069a50 msgr2=0x7fb1a4069e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:24.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.949+0000 7fb1ab61f640 1 -- 172.21.15.57:0/731845695 shutdown_connections 2023-10-19T14:22:24.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.949+0000 7fb1ab61f640 1 -- 172.21.15.57:0/731845695 wait complete. 2023-10-19T14:22:24.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.950+0000 7fb1ab61f640 1 Processor -- start 2023-10-19T14:22:24.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.950+0000 7fb1ab61f640 1 -- start start 2023-10-19T14:22:24.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.951+0000 7fb1ab61f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb1a40cbf30 0x7fb1a41f6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:24.956 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.951+0000 7fb1ab61f640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb1a40cc310 0x7fb1a415a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:24.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.951+0000 7fb1ab61f640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb1a415b770 0x7fb1a41f1f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:24.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.951+0000 7fb1ab61f640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1a4151590 con 0x7fb1a40cbf30 2023-10-19T14:22:24.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.951+0000 7fb1ab61f640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1a41f2e80 con 0x7fb1a415b770 2023-10-19T14:22:24.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.951+0000 7fb1ab61f640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1a41f2ba0 con 0x7fb1a40cc310 2023-10-19T14:22:24.957 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.951+0000 7fb1a93bb640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb1a40cbf30 0x7fb1a41f6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:24.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.951+0000 7fb1a93bb640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb1a40cbf30 0x7fb1a41f6130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46582/0 (socket says 172.21.15.57:46582) 2023-10-19T14:22:24.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.951+0000 7fb1a93bb640 1 -- 172.21.15.57:0/1521202305 learned_addr learned my addr 172.21.15.57:0/1521202305 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:22:24.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.951+0000 7fb1a93bb640 1 -- 172.21.15.57:0/1521202305 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb1a40cc310 msgr2=0x7fb1a415a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:24.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.951+0000 7fb1a93bb640 1 --2- 172.21.15.57:0/1521202305 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb1a40cc310 0x7fb1a415a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:24.958 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.951+0000 7fb1a93bb640 1 -- 172.21.15.57:0/1521202305 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb1a415b770 msgr2=0x7fb1a41f1f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:24.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.951+0000 7fb1a9bbc640 1 --2- 172.21.15.57:0/1521202305 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb1a415b770 0x7fb1a41f1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:24.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.951+0000 7fb1a93bb640 1 --2- 172.21.15.57:0/1521202305 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb1a415b770 0x7fb1a41f1f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:24.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.951+0000 7fb1a93bb640 1 -- 172.21.15.57:0/1521202305 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1a0009560 con 0x7fb1a40cbf30 2023-10-19T14:22:24.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.951+0000 7fb1a9bbc640 1 --2- 172.21.15.57:0/1521202305 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb1a415b770 0x7fb1a41f1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:22:24.959 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.951+0000 7fb1a93bb640 1 --2- 172.21.15.57:0/1521202305 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb1a40cbf30 0x7fb1a41f6130 secure :-1 s=READY pgs=2966 cs=0 l=1 rev1=1 crypto rx=0x7fb1a0006010 tx=0x7fb1a002dd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:24.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.951+0000 7fb199ffb640 1 -- 172.21.15.57:0/1521202305 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1a0005650 con 0x7fb1a40cbf30 2023-10-19T14:22:24.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.951+0000 7fb199ffb640 1 -- 172.21.15.57:0/1521202305 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fb1a00038f0 con 0x7fb1a40cbf30 2023-10-19T14:22:24.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.951+0000 7fb1ab61f640 1 -- 172.21.15.57:0/1521202305 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1a41f5650 con 0x7fb1a40cbf30 2023-10-19T14:22:24.960 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.951+0000 7fb199ffb640 1 -- 172.21.15.57:0/1521202305 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1a0003d00 con 0x7fb1a40cbf30 2023-10-19T14:22:24.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.952+0000 7fb1ab61f640 1 -- 172.21.15.57:0/1521202305 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1a41f5c10 con 0x7fb1a40cbf30 2023-10-19T14:22:24.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.953+0000 7fb1ab61f640 1 -- 172.21.15.57:0/1521202305 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1a41f5e00 con 0x7fb1a40cbf30 2023-10-19T14:22:24.961 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.953+0000 7fb199ffb640 1 -- 172.21.15.57:0/1521202305 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 150) v1 ==== 89405+0+0 (secure 0 0 0) 0x7fb1a003e450 con 0x7fb1a40cbf30 2023-10-19T14:22:24.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.954+0000 7fb199ffb640 1 --2- 172.21.15.57:0/1521202305 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7fb18806b040 0x7fb18806d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:24.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.954+0000 7fb199ffb640 1 -- 172.21.15.57:0/1521202305 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(264..264 src has 1..264) v4 ==== 6487+0+0 (secure 0 0 0) 0x7fb1a00ac9a0 con 0x7fb1a40cbf30 2023-10-19T14:22:24.962 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.954+0000 7fb1a8bba640 1 --2- 172.21.15.57:0/1521202305 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7fb18806b040 0x7fb18806d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:24.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.955+0000 7fb1a8bba640 1 --2- 172.21.15.57:0/1521202305 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7fb18806b040 0x7fb18806d500 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb1a40676e0 tx=0x7fb194007990 comp rx=0 tx=0).ready entity=mgr.10394 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:24.963 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:24.958+0000 7fb199ffb640 1 -- 172.21.15.57:0/1521202305 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb1a00776b0 con 0x7fb1a40cbf30 2023-10-19T14:22:25.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:25.157+0000 7fb1ab61f640 1 -- 172.21.15.57:0/1521202305 --> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7fb1a40c9af0 con 0x7fb18806b040 2023-10-19T14:22:25.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:25.209+0000 7fb199ffb640 1 -- 172.21.15.57:0/1521202305 <== mgr.10394 v2:172.21.15.179:6832/2005622276 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fb1a40c9af0 con 0x7fb18806b040 2023-10-19T14:22:25.213 INFO:teuthology.orchestra.run.smithi057.stdout:Schedule removed for path snap_test_dir1 2023-10-19T14:22:25.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:25.211+0000 7fb1ab61f640 1 -- 172.21.15.57:0/1521202305 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7fb18806b040 msgr2=0x7fb18806d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:25.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:25.211+0000 7fb1ab61f640 1 --2- 172.21.15.57:0/1521202305 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7fb18806b040 0x7fb18806d500 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb1a40676e0 tx=0x7fb194007990 comp rx=0 tx=0).stop 2023-10-19T14:22:25.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:25.211+0000 7fb1ab61f640 1 -- 172.21.15.57:0/1521202305 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb1a40cbf30 msgr2=0x7fb1a41f6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:25.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:25.211+0000 7fb1ab61f640 1 --2- 172.21.15.57:0/1521202305 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb1a40cbf30 0x7fb1a41f6130 secure :-1 s=READY pgs=2966 cs=0 l=1 rev1=1 crypto rx=0x7fb1a0006010 tx=0x7fb1a002dd60 comp rx=0 tx=0).stop 2023-10-19T14:22:25.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:25.211+0000 7fb1ab61f640 1 -- 172.21.15.57:0/1521202305 shutdown_connections 2023-10-19T14:22:25.216 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:25.212+0000 7fb1ab61f640 1 --2- 172.21.15.57:0/1521202305 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7fb18806b040 0x7fb18806d500 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:25.216 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:25.212+0000 7fb1ab61f640 1 --2- 172.21.15.57:0/1521202305 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb1a415b770 0x7fb1a41f1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:25.216 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:25.212+0000 7fb1ab61f640 1 --2- 172.21.15.57:0/1521202305 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb1a40cc310 0x7fb1a415a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:25.216 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:25.212+0000 7fb1ab61f640 1 --2- 172.21.15.57:0/1521202305 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb1a40cbf30 0x7fb1a41f6130 unknown :-1 s=CLOSED pgs=2966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:25.216 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:25.212+0000 7fb1ab61f640 1 -- 172.21.15.57:0/1521202305 >> 172.21.15.57:0/1521202305 conn(0x7fb1a4069a50 msgr2=0x7fb1a40af080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:25.217 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:25.212+0000 7fb1ab61f640 1 -- 172.21.15.57:0/1521202305 shutdown_connections 2023-10-19T14:22:25.217 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:25.212+0000 7fb1ab61f640 1 -- 172.21.15.57:0/1521202305 wait complete. 2023-10-19T14:22:25.225 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2023-10-19T14:22:25.226 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:22:25.226 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-19T14:22:25.247 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:22:25.247 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2023-10-19T14:22:25.306 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-19T14:22:25.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:25.456+0000 7f58a99d7640 1 -- 172.21.15.57:0/4288401378 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f58a40cbf30 msgr2=0x7f58a406bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:25.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:25.456+0000 7f58a99d7640 1 --2- 172.21.15.57:0/4288401378 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f58a40cbf30 0x7f58a406bd30 secure :-1 s=READY pgs=2968 cs=0 l=1 rev1=1 crypto rx=0x7f5894005fd0 tx=0x7f589402be20 comp rx=0 tx=0).stop 2023-10-19T14:22:25.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:25.456+0000 7f58a99d7640 1 -- 172.21.15.57:0/4288401378 shutdown_connections 2023-10-19T14:22:25.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:25.456+0000 7f58a99d7640 1 --2- 172.21.15.57:0/4288401378 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f58a40cbf30 0x7f58a406bd30 unknown :-1 s=CLOSED pgs=2968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:25.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:25.456+0000 7f58a99d7640 1 -- 172.21.15.57:0/4288401378 >> 172.21.15.57:0/4288401378 conn(0x7f58a4069a50 msgr2=0x7f58a4069e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:25.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:25.456+0000 7f58a99d7640 1 -- 172.21.15.57:0/4288401378 shutdown_connections 2023-10-19T14:22:25.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:25.456+0000 7f58a99d7640 1 -- 172.21.15.57:0/4288401378 wait complete. 2023-10-19T14:22:25.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:25.457+0000 7f58a99d7640 1 Processor -- start 2023-10-19T14:22:25.463 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:25.457+0000 7f58a99d7640 1 -- start start 2023-10-19T14:22:25.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:25.458+0000 7f58a99d7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f58a40cbf30 0x7f58a4144860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:25.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:25.458+0000 7f58a99d7640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f58a40cc310 0x7f58a4150480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:25.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:25.458+0000 7f58a99d7640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f58a415b770 0x7f58a4156440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:25.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:25.458+0000 7f58a99d7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58a4144da0 con 0x7f58a40cbf30 2023-10-19T14:22:25.464 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:25.458+0000 7f58a99d7640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58a41453f0 con 0x7f58a415b770 2023-10-19T14:22:25.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:25.458+0000 7f58a99d7640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58a41456d0 con 0x7f58a40cc310 2023-10-19T14:22:25.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:25.458+0000 7f58a2ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f58a40cbf30 0x7f58a4144860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:25.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:25.458+0000 7f58a2ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f58a40cbf30 0x7f58a4144860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46594/0 (socket says 172.21.15.57:46594) 2023-10-19T14:22:25.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:25.458+0000 7f58a2ffd640 1 -- 172.21.15.57:0/3941983964 learned_addr learned my addr 172.21.15.57:0/3941983964 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:22:25.465 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:25.458+0000 7f589bfff640 1 --2- 172.21.15.57:0/3941983964 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f58a40cc310 0x7f58a4150480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:25.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:25.458+0000 7f58a37fe640 1 --2- 172.21.15.57:0/3941983964 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f58a415b770 0x7f58a4156440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:25.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:25.458+0000 7f58a2ffd640 1 -- 172.21.15.57:0/3941983964 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f58a40cc310 msgr2=0x7f58a4150480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:25.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:25.458+0000 7f58a2ffd640 1 --2- 172.21.15.57:0/3941983964 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f58a40cc310 0x7f58a4150480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:25.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:25.458+0000 7f58a2ffd640 1 -- 172.21.15.57:0/3941983964 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f58a415b770 msgr2=0x7f58a4156440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:25.466 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:25.458+0000 7f58a2ffd640 1 --2- 172.21.15.57:0/3941983964 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f58a415b770 0x7f58a4156440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:25.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:25.458+0000 7f58a2ffd640 1 -- 172.21.15.57:0/3941983964 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5894009560 con 0x7f58a40cbf30 2023-10-19T14:22:25.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:25.458+0000 7f58a2ffd640 1 --2- 172.21.15.57:0/3941983964 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f58a40cbf30 0x7f58a4144860 secure :-1 s=READY pgs=2969 cs=0 l=1 rev1=1 crypto rx=0x7f5894006010 tx=0x7f589402ddf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:25.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:25.458+0000 7f58a0ff9640 1 -- 172.21.15.57:0/3941983964 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5894005650 con 0x7f58a40cbf30 2023-10-19T14:22:25.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:25.458+0000 7f58a0ff9640 1 -- 172.21.15.57:0/3941983964 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f58940038f0 con 0x7f58a40cbf30 2023-10-19T14:22:25.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:25.458+0000 7f58a99d7640 1 -- 172.21.15.57:0/3941983964 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58a41459b0 con 0x7f58a40cbf30 2023-10-19T14:22:25.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:25.458+0000 7f58a0ff9640 1 -- 172.21.15.57:0/3941983964 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5894003ca0 con 0x7f58a40cbf30 2023-10-19T14:22:25.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:25.459+0000 7f58a99d7640 1 -- 172.21.15.57:0/3941983964 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58a4144f60 con 0x7f58a40cbf30 2023-10-19T14:22:25.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:25.460+0000 7f58a99d7640 1 -- 172.21.15.57:0/3941983964 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58a4145ed0 con 0x7f58a40cbf30 2023-10-19T14:22:25.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:25.460+0000 7f58a0ff9640 1 -- 172.21.15.57:0/3941983964 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 150) v1 ==== 89405+0+0 (secure 0 0 0) 0x7f589403e450 con 0x7f58a40cbf30 2023-10-19T14:22:25.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:25.460+0000 7f58a0ff9640 1 --2- 172.21.15.57:0/3941983964 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f588406b040 0x7f588406d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:25.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:25.460+0000 7f58a0ff9640 1 -- 172.21.15.57:0/3941983964 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(264..264 src has 1..264) v4 ==== 6487+0+0 (secure 0 0 0) 0x7f58940ac980 con 0x7f58a40cbf30 2023-10-19T14:22:25.469 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:25.461+0000 7f589bfff640 1 --2- 172.21.15.57:0/3941983964 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f588406b040 0x7f588406d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:25.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:25.461+0000 7f589bfff640 1 --2- 172.21.15.57:0/3941983964 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f588406b040 0x7f588406d500 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f58a40b09c0 tx=0x7f588c009040 comp rx=0 tx=0).ready entity=mgr.10394 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:25.470 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:25.465+0000 7f58a0ff9640 1 -- 172.21.15.57:0/3941983964 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f58a4145ed0 con 0x7f58a40cbf30 2023-10-19T14:22:25.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:25.688+0000 7f58a99d7640 1 -- 172.21.15.57:0/3941983964 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f58a4145ed0 con 0x7f58a40cbf30 2023-10-19T14:22:26.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.386+0000 7f58a0ff9640 1 -- 172.21.15.57:0/3941983964 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v151) v1 ==== 93+0+0 (secure 0 0 0) 0x7f58a4145ed0 con 0x7f58a40cbf30 2023-10-19T14:22:26.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.390+0000 7f58a99d7640 1 -- 172.21.15.57:0/3941983964 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f588406b040 msgr2=0x7f588406d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:26.393 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.390+0000 7f58a99d7640 1 --2- 172.21.15.57:0/3941983964 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f588406b040 0x7f588406d500 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f58a40b09c0 tx=0x7f588c009040 comp rx=0 tx=0).stop 2023-10-19T14:22:26.394 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.390+0000 7f58a99d7640 1 -- 172.21.15.57:0/3941983964 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f58a40cbf30 msgr2=0x7f58a4144860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:26.394 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.390+0000 7f58a99d7640 1 --2- 172.21.15.57:0/3941983964 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f58a40cbf30 0x7f58a4144860 secure :-1 s=READY pgs=2969 cs=0 l=1 rev1=1 crypto rx=0x7f5894006010 tx=0x7f589402ddf0 comp rx=0 tx=0).stop 2023-10-19T14:22:26.394 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.390+0000 7f58a99d7640 1 -- 172.21.15.57:0/3941983964 shutdown_connections 2023-10-19T14:22:26.394 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.390+0000 7f58a99d7640 1 --2- 172.21.15.57:0/3941983964 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f588406b040 0x7f588406d500 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:26.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.390+0000 7f58a99d7640 1 --2- 172.21.15.57:0/3941983964 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f58a415b770 0x7f58a4156440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:26.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.390+0000 7f58a99d7640 1 --2- 172.21.15.57:0/3941983964 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f58a40cc310 0x7f58a4150480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:26.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.390+0000 7f58a99d7640 1 --2- 172.21.15.57:0/3941983964 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f58a40cbf30 0x7f58a4144860 secure :-1 s=CLOSED pgs=2969 cs=0 l=1 rev1=1 crypto rx=0x7f5894006010 tx=0x7f589402ddf0 comp rx=0 tx=0).stop 2023-10-19T14:22:26.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.390+0000 7f58a99d7640 1 -- 172.21.15.57:0/3941983964 >> 172.21.15.57:0/3941983964 conn(0x7f58a4069a50 msgr2=0x7f58a4158b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:26.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.390+0000 7f58a99d7640 1 -- 172.21.15.57:0/3941983964 shutdown_connections 2023-10-19T14:22:26.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.391+0000 7f58a99d7640 1 -- 172.21.15.57:0/3941983964 wait complete. 2023-10-19T14:22:26.408 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-19T14:22:26.409 DEBUG:teuthology.orchestra.run.smithi057:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-19T14:22:26.470 DEBUG:teuthology.orchestra.run.smithi179:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-19T14:22:26.514 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-19T14:22:26.514 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T14:22:26.515 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T14:22:26.531 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T14:22:26.682 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.679+0000 7f71de17d640 1 -- 172.21.15.57:0/89229973 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f71c4004350 msgr2=0x7f71c4004820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:26.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.679+0000 7f71de17d640 1 --2- 172.21.15.57:0/89229973 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f71c4004350 0x7f71c4004820 secure :-1 s=READY pgs=2974 cs=0 l=1 rev1=1 crypto rx=0x7f71c0001790 tx=0x7f71c40057f0 comp rx=0 tx=0).stop 2023-10-19T14:22:26.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.679+0000 7f71d7fff640 1 -- 172.21.15.57:0/89229973 reap_dead start 2023-10-19T14:22:26.683 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.679+0000 7f71de17d640 1 -- 172.21.15.57:0/89229973 shutdown_connections 2023-10-19T14:22:26.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.679+0000 7f71de17d640 1 -- 172.21.15.57:0/89229973 >> 172.21.15.57:0/89229973 conn(0x7f71d809a140 msgr2=0x7f71d80d0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:26.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.679+0000 7f71de17d640 1 -- 172.21.15.57:0/89229973 shutdown_connections 2023-10-19T14:22:26.684 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.680+0000 7f71de17d640 1 -- 172.21.15.57:0/89229973 wait complete. 2023-10-19T14:22:26.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.681+0000 7f71de17d640 1 Processor -- start 2023-10-19T14:22:26.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.681+0000 7f71de17d640 1 -- start start 2023-10-19T14:22:26.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.682+0000 7f71de17d640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f71c4004350 0x7f71d8153400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:26.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.682+0000 7f71de17d640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f71d80c6be0 0x7f71d80c7060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:26.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.682+0000 7f71de17d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f71d80ccf20 0x7f71d80cd3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:26.688 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.682+0000 7f71de17d640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71d80c75a0 con 0x7f71d80ccf20 2023-10-19T14:22:26.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.682+0000 7f71de17d640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71d80cd910 con 0x7f71d80c6be0 2023-10-19T14:22:26.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.682+0000 7f71de17d640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71d80c8960 con 0x7f71c4004350 2023-10-19T14:22:26.689 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.682+0000 7f71d7fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f71d80ccf20 0x7f71d80cd3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:26.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.682+0000 7f71d7fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f71d80ccf20 0x7f71d80cd3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38750/0 (socket says 172.21.15.57:38750) 2023-10-19T14:22:26.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.682+0000 7f71d7fff640 1 -- 172.21.15.57:0/4041349645 learned_addr learned my addr 172.21.15.57:0/4041349645 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:22:26.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.682+0000 7f71d6ffd640 1 --2- 172.21.15.57:0/4041349645 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f71d80c6be0 0x7f71d80c7060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:26.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.682+0000 7f71d77fe640 1 --2- 172.21.15.57:0/4041349645 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f71c4004350 0x7f71d8153400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:26.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.682+0000 7f71d7fff640 1 -- 172.21.15.57:0/4041349645 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f71c4004350 msgr2=0x7f71d8153400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:26.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.682+0000 7f71d7fff640 1 --2- 172.21.15.57:0/4041349645 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f71c4004350 0x7f71d8153400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:26.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.682+0000 7f71d7fff640 1 -- 172.21.15.57:0/4041349645 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f71d80c6be0 msgr2=0x7f71d80c7060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:26.691 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.682+0000 7f71d7fff640 1 --2- 172.21.15.57:0/4041349645 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f71d80c6be0 0x7f71d80c7060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:26.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.682+0000 7f71d7fff640 1 -- 172.21.15.57:0/4041349645 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71d8153940 con 0x7f71d80ccf20 2023-10-19T14:22:26.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.682+0000 7f71d6ffd640 1 --2- 172.21.15.57:0/4041349645 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f71d80c6be0 0x7f71d80c7060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T14:22:26.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.682+0000 7f71d77fe640 1 --2- 172.21.15.57:0/4041349645 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f71c4004350 0x7f71d8153400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T14:22:26.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.682+0000 7f71d7fff640 1 --2- 172.21.15.57:0/4041349645 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f71d80ccf20 0x7f71d80cd3d0 secure :-1 s=READY pgs=2975 cs=0 l=1 rev1=1 crypto rx=0x7f71d8069e50 tx=0x7f71d000f3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:26.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.683+0000 7f71cffff640 1 -- 172.21.15.57:0/4041349645 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71d0004b20 con 0x7f71d80ccf20 2023-10-19T14:22:26.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.683+0000 7f71cffff640 1 -- 172.21.15.57:0/4041349645 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f71d0004c80 con 0x7f71d80ccf20 2023-10-19T14:22:26.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.683+0000 7f71de17d640 1 -- 172.21.15.57:0/4041349645 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71d80c91c0 con 0x7f71d80ccf20 2023-10-19T14:22:26.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.683+0000 7f71cffff640 1 -- 172.21.15.57:0/4041349645 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71d0003bd0 con 0x7f71d80ccf20 2023-10-19T14:22:26.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.683+0000 7f71de17d640 1 -- 172.21.15.57:0/4041349645 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71d80c8b70 con 0x7f71d80ccf20 2023-10-19T14:22:26.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.684+0000 7f71de17d640 1 -- 172.21.15.57:0/4041349645 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71d80c9350 con 0x7f71d80ccf20 2023-10-19T14:22:26.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.684+0000 7f71cffff640 1 -- 172.21.15.57:0/4041349645 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 151) v1 ==== 89388+0+0 (secure 0 0 0) 0x7f71d0003d30 con 0x7f71d80ccf20 2023-10-19T14:22:26.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.685+0000 7f71cffff640 1 --2- 172.21.15.57:0/4041349645 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f71b806aff0 0x7f71b806d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:26.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.685+0000 7f71cffff640 1 -- 172.21.15.57:0/4041349645 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(264..264 src has 1..264) v4 ==== 6487+0+0 (secure 0 0 0) 0x7f71d0089f90 con 0x7f71d80ccf20 2023-10-19T14:22:26.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.685+0000 7f71d77fe640 1 -- 172.21.15.57:0/4041349645 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f71b806aff0 msgr2=0x7f71b806d4b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/2005622276 2023-10-19T14:22:26.695 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.685+0000 7f71d77fe640 1 --2- 172.21.15.57:0/4041349645 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f71b806aff0 0x7f71b806d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T14:22:26.696 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.689+0000 7f71cffff640 1 -- 172.21.15.57:0/4041349645 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f71d0054b00 con 0x7f71d80ccf20 2023-10-19T14:22:26.782 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:22:26.780+0000 7f29ada11040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-19T14:22:26.802 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:22:26.799+0000 7f0d028fa040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-19T14:22:26.827 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:22:26.824+0000 7f29ada11040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-19T14:22:26.849 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:22:26.847+0000 7f0d028fa040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-19T14:22:26.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.867+0000 7f71de17d640 1 -- 172.21.15.57:0/4041349645 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f71d8153940 con 0x7f71d80ccf20 2023-10-19T14:22:26.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.868+0000 7f71cffff640 1 -- 172.21.15.57:0/4041349645 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v264) v1 ==== 74+0+17407 (secure 0 0 0) 0x7f71d8153940 con 0x7f71d80ccf20 2023-10-19T14:22:26.871 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:22:26.872 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":264,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T14:22:17.865478+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":36,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_metadata","create_time":"2023-10-19T14:22:00.047912+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"262","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_data","create_time":"2023-10-19T14:22:00.739698+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"263","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.179:0/1983646445":"2023-10-20T14:22:17.865415+0000","172.21.15.179:0/1840191723":"2023-10-20T14:22:17.865415+0000","172.21.15.179:0/4168916968":"2023-10-20T14:22:17.865415+0000","172.21.15.179:6833/1573806521":"2023-10-20T14:22:17.865415+0000","172.21.15.179:6832/1573806521":"2023-10-20T14:22:17.865415+0000","172.21.15.179:0/658875312":"2023-10-20T14:22:17.865415+0000","172.21.15.179:0/1799772260":"2023-10-20T14:22:17.865415+0000","172.21.15.179:0/3831670324":"2023-10-20T14:22:17.865415+0000","172.21.15.179:0/90219027":"2023-10-20T14:22:17.865415+0000","172.21.15.179:0/2739598080":"2023-10-20T14:22:17.865415+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T14:22:26.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.870+0000 7f71de17d640 1 -- 172.21.15.57:0/4041349645 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f71b806aff0 msgr2=0x7f71b806d4b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T14:22:26.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.870+0000 7f71de17d640 1 --2- 172.21.15.57:0/4041349645 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f71b806aff0 0x7f71b806d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:26.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.870+0000 7f71de17d640 1 -- 172.21.15.57:0/4041349645 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f71d80ccf20 msgr2=0x7f71d80cd3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:26.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.870+0000 7f71de17d640 1 --2- 172.21.15.57:0/4041349645 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f71d80ccf20 0x7f71d80cd3d0 secure :-1 s=READY pgs=2975 cs=0 l=1 rev1=1 crypto rx=0x7f71d8069e50 tx=0x7f71d000f3c0 comp rx=0 tx=0).stop 2023-10-19T14:22:26.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.870+0000 7f71de17d640 1 -- 172.21.15.57:0/4041349645 shutdown_connections 2023-10-19T14:22:26.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.870+0000 7f71de17d640 1 --2- 172.21.15.57:0/4041349645 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f71b806aff0 0x7f71b806d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:26.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.870+0000 7f71de17d640 1 --2- 172.21.15.57:0/4041349645 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f71d80ccf20 0x7f71d80cd3d0 unknown :-1 s=CLOSED pgs=2975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:26.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.870+0000 7f71de17d640 1 --2- 172.21.15.57:0/4041349645 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f71d80c6be0 0x7f71d80c7060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:26.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.870+0000 7f71de17d640 1 --2- 172.21.15.57:0/4041349645 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f71c4004350 0x7f71d8153400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:26.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.870+0000 7f71de17d640 1 -- 172.21.15.57:0/4041349645 >> 172.21.15.57:0/4041349645 conn(0x7f71d809a140 msgr2=0x7f71d815d460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:26.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.870+0000 7f71de17d640 1 -- 172.21.15.57:0/4041349645 shutdown_connections 2023-10-19T14:22:26.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.870+0000 7f71de17d640 1 -- 172.21.15.57:0/4041349645 wait complete. 2023-10-19T14:22:26.885 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-19T12:44:07.371186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 35, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-19T14:22:00.047912+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '262', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 36, 'pool_name': 'cephfs_data', 'create_time': '2023-10-19T14:22:00.739698+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '263', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-19T14:22:26.885 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-19T14:22:26.998 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.995+0000 7f46c967c640 1 -- 172.21.15.57:0/2698319441 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f46c40cf140 msgr2=0x7f46c40cc310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:26.998 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.995+0000 7f46c967c640 1 --2- 172.21.15.57:0/2698319441 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f46c40cf140 0x7f46c40cc310 secure :-1 s=READY pgs=2976 cs=0 l=1 rev1=1 crypto rx=0x7f46bc005fd0 tx=0x7f46bc02c2e0 comp rx=0 tx=0).stop 2023-10-19T14:22:26.999 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.995+0000 7f46c967c640 1 -- 172.21.15.57:0/2698319441 shutdown_connections 2023-10-19T14:22:26.999 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.995+0000 7f46c967c640 1 --2- 172.21.15.57:0/2698319441 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f46c40cf140 0x7f46c40cc310 unknown :-1 s=CLOSED pgs=2976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:26.999 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.995+0000 7f46c967c640 1 --2- 172.21.15.57:0/2698319441 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f46c415b070 0x7f46c40d0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:26.999 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.995+0000 7f46c967c640 1 -- 172.21.15.57:0/2698319441 >> 172.21.15.57:0/2698319441 conn(0x7f46c408a7e0 msgr2=0x7f46c409a100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:26.999 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.995+0000 7f46c967c640 1 -- 172.21.15.57:0/2698319441 shutdown_connections 2023-10-19T14:22:27.000 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.995+0000 7f46c967c640 1 -- 172.21.15.57:0/2698319441 wait complete. 2023-10-19T14:22:27.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.997+0000 7f46c967c640 1 Processor -- start 2023-10-19T14:22:27.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.997+0000 7f46c967c640 1 -- start start 2023-10-19T14:22:27.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.997+0000 7f46c967c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f46c40cf140 0x7f46c40c7440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:27.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.997+0000 7f46c967c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f46c40d1080 0x7f46c4157e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:27.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.997+0000 7f46c967c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f46c415b070 0x7f46c40c5aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:27.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.997+0000 7f46c967c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46c40c6fb0 con 0x7f46c415b070 2023-10-19T14:22:27.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.997+0000 7f46c967c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46c41583d0 con 0x7f46c40cf140 2023-10-19T14:22:27.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.997+0000 7f46c967c640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46c40c63a0 con 0x7f46c40d1080 2023-10-19T14:22:27.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.997+0000 7f46c37fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f46c415b070 0x7f46c40c5aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:27.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.997+0000 7f46c37fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f46c415b070 0x7f46c40c5aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38766/0 (socket says 172.21.15.57:38766) 2023-10-19T14:22:27.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.997+0000 7f46c2ffd640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f46c40cf140 0x7f46c40c7440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:27.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.997+0000 7f46c37fe640 1 -- 172.21.15.57:0/671496020 learned_addr learned my addr 172.21.15.57:0/671496020 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:22:27.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.997+0000 7f46c27fc640 1 --2- 172.21.15.57:0/671496020 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f46c40d1080 0x7f46c4157e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:27.004 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.998+0000 7f46c2ffd640 1 -- 172.21.15.57:0/671496020 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f46c40d1080 msgr2=0x7f46c4157e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:27.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.998+0000 7f46c2ffd640 1 --2- 172.21.15.57:0/671496020 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f46c40d1080 0x7f46c4157e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:27.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.998+0000 7f46c2ffd640 1 -- 172.21.15.57:0/671496020 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f46c415b070 msgr2=0x7f46c40c5aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:27.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.998+0000 7f46c2ffd640 1 --2- 172.21.15.57:0/671496020 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f46c415b070 0x7f46c40c5aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:27.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.998+0000 7f46c2ffd640 1 -- 172.21.15.57:0/671496020 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46bc009900 con 0x7f46c40cf140 2023-10-19T14:22:27.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.998+0000 7f46c37fe640 1 --2- 172.21.15.57:0/671496020 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f46c415b070 0x7f46c40c5aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T14:22:27.006 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.998+0000 7f46c27fc640 1 --2- 172.21.15.57:0/671496020 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f46c40d1080 0x7f46c4157e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T14:22:27.006 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.998+0000 7f46c2ffd640 1 --2- 172.21.15.57:0/671496020 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f46c40cf140 0x7f46c40c7440 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f46b400e6d0 tx=0x7f46b400eb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:27.006 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.998+0000 7f46ab7fe640 1 -- 172.21.15.57:0/671496020 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b400cab0 con 0x7f46c40cf140 2023-10-19T14:22:27.006 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.998+0000 7f46ab7fe640 1 -- 172.21.15.57:0/671496020 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f46b400cd00 con 0x7f46c40cf140 2023-10-19T14:22:27.006 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.998+0000 7f46c967c640 1 -- 172.21.15.57:0/671496020 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46c40c6680 con 0x7f46c40cf140 2023-10-19T14:22:27.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.998+0000 7f46ab7fe640 1 -- 172.21.15.57:0/671496020 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b4010690 con 0x7f46c40cf140 2023-10-19T14:22:27.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.998+0000 7f46c967c640 1 -- 172.21.15.57:0/671496020 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46c40c6c40 con 0x7f46c40cf140 2023-10-19T14:22:27.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:26.999+0000 7f46c967c640 1 -- 172.21.15.57:0/671496020 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46c409ed00 con 0x7f46c40cf140 2023-10-19T14:22:27.008 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.000+0000 7f46ab7fe640 1 -- 172.21.15.57:0/671496020 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 151) v1 ==== 89388+0+0 (secure 0 0 0) 0x7f46b40107f0 con 0x7f46c40cf140 2023-10-19T14:22:27.008 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.001+0000 7f46ab7fe640 1 --2- 172.21.15.57:0/671496020 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f46b006aff0 0x7f46b006d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:27.008 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.001+0000 7f46c27fc640 1 -- 172.21.15.57:0/671496020 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f46b006aff0 msgr2=0x7f46b006d4b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/2005622276 2023-10-19T14:22:27.008 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.001+0000 7f46ab7fe640 1 -- 172.21.15.57:0/671496020 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(264..264 src has 1..264) v4 ==== 6487+0+0 (secure 0 0 0) 0x7f46b4014070 con 0x7f46c40cf140 2023-10-19T14:22:27.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.001+0000 7f46c27fc640 1 --2- 172.21.15.57:0/671496020 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f46b006aff0 0x7f46b006d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T14:22:27.009 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.005+0000 7f46ab7fe640 1 -- 172.21.15.57:0/671496020 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f46b4051ca0 con 0x7f46c40cf140 2023-10-19T14:22:27.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.181+0000 7f46c967c640 1 -- 172.21.15.57:0/671496020 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f46c404db50 con 0x7f46c40cf140 2023-10-19T14:22:27.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.182+0000 7f46ab7fe640 1 -- 172.21.15.57:0/671496020 <== mon.1 v2:172.21.15.179:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v264) v1 ==== 93+0+10 (secure 0 0 0) 0x7f46b4055150 con 0x7f46c40cf140 2023-10-19T14:22:27.185 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-19T14:22:27.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.184+0000 7f46c967c640 1 -- 172.21.15.57:0/671496020 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f46b006aff0 msgr2=0x7f46b006d4b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T14:22:27.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.184+0000 7f46c967c640 1 --2- 172.21.15.57:0/671496020 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f46b006aff0 0x7f46b006d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:27.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.184+0000 7f46c967c640 1 -- 172.21.15.57:0/671496020 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f46c40cf140 msgr2=0x7f46c40c7440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:27.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.184+0000 7f46c967c640 1 --2- 172.21.15.57:0/671496020 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f46c40cf140 0x7f46c40c7440 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f46b400e6d0 tx=0x7f46b400eb90 comp rx=0 tx=0).stop 2023-10-19T14:22:27.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.184+0000 7f46c967c640 1 -- 172.21.15.57:0/671496020 shutdown_connections 2023-10-19T14:22:27.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.184+0000 7f46c967c640 1 --2- 172.21.15.57:0/671496020 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f46b006aff0 0x7f46b006d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:27.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.184+0000 7f46c967c640 1 --2- 172.21.15.57:0/671496020 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f46c415b070 0x7f46c40c5aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:27.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.184+0000 7f46c967c640 1 --2- 172.21.15.57:0/671496020 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f46c40d1080 0x7f46c4157e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:27.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.184+0000 7f46c967c640 1 --2- 172.21.15.57:0/671496020 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f46c40cf140 0x7f46c40c7440 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:27.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.184+0000 7f46c967c640 1 -- 172.21.15.57:0/671496020 >> 172.21.15.57:0/671496020 conn(0x7f46c408a7e0 msgr2=0x7f46c4069ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:27.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.184+0000 7f46c967c640 1 -- 172.21.15.57:0/671496020 shutdown_connections 2023-10-19T14:22:27.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.184+0000 7f46c967c640 1 -- 172.21.15.57:0/671496020 wait complete. 2023-10-19T14:22:27.198 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-19T14:22:27.216 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:22:27.213+0000 7f29ada11040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-19T14:22:27.259 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:22:27.256+0000 7f29ada11040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-19T14:22:27.270 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:22:27.267+0000 7f0d028fa040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-19T14:22:27.301 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:22:27.299+0000 7f29ada11040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-19T14:22:27.313 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.310+0000 7f35fbe39640 1 -- 172.21.15.57:0/1672111554 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f35f415b070 msgr2=0x7f35f40d0b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:27.313 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.310+0000 7f35fbe39640 1 --2- 172.21.15.57:0/1672111554 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f35f415b070 0x7f35f40d0b40 secure :-1 s=READY pgs=2978 cs=0 l=1 rev1=1 crypto rx=0x7f35e4009830 tx=0x7f35e402bdf0 comp rx=0 tx=0).stop 2023-10-19T14:22:27.314 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.310+0000 7f35fbe39640 1 -- 172.21.15.57:0/1672111554 shutdown_connections 2023-10-19T14:22:27.314 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.310+0000 7f35fbe39640 1 --2- 172.21.15.57:0/1672111554 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f35f415b070 0x7f35f40d0b40 unknown :-1 s=CLOSED pgs=2978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:27.314 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.310+0000 7f35fbe39640 1 -- 172.21.15.57:0/1672111554 >> 172.21.15.57:0/1672111554 conn(0x7f35f408a7e0 msgr2=0x7f35f409a100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:27.314 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.310+0000 7f35fbe39640 1 -- 172.21.15.57:0/1672111554 shutdown_connections 2023-10-19T14:22:27.314 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.310+0000 7f35fbe39640 1 -- 172.21.15.57:0/1672111554 wait complete. 2023-10-19T14:22:27.316 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.312+0000 7f35fbe39640 1 Processor -- start 2023-10-19T14:22:27.317 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.312+0000 7f35fbe39640 1 -- start start 2023-10-19T14:22:27.317 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.312+0000 7f35fbe39640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f35f40cf140 0x7f35f415bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:27.317 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.312+0000 7f35fbe39640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f35f40d1080 0x7f35f41f0f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:27.317 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.312+0000 7f35fbe39640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f35f415b070 0x7f35f40b2560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:27.318 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.312+0000 7f35fbe39640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35f415c750 con 0x7f35f40d1080 2023-10-19T14:22:27.318 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.312+0000 7f35fbe39640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35f415ca30 con 0x7f35f415b070 2023-10-19T14:22:27.318 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.312+0000 7f35fbe39640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35f415cd10 con 0x7f35f40cf140 2023-10-19T14:22:27.318 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.312+0000 7f35f93d4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f35f40d1080 0x7f35f41f0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:27.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.313+0000 7f35f93d4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f35f40d1080 0x7f35f41f0f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38774/0 (socket says 172.21.15.57:38774) 2023-10-19T14:22:27.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.313+0000 7f35f93d4640 1 -- 172.21.15.57:0/4170148766 learned_addr learned my addr 172.21.15.57:0/4170148766 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:22:27.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.313+0000 7f35fa3d6640 1 --2- 172.21.15.57:0/4170148766 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f35f415b070 0x7f35f40b2560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:27.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.313+0000 7f35f9bd5640 1 --2- 172.21.15.57:0/4170148766 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f35f40cf140 0x7f35f415bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:27.319 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.313+0000 7f35f93d4640 1 -- 172.21.15.57:0/4170148766 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f35f40cf140 msgr2=0x7f35f415bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:27.320 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.313+0000 7f35f93d4640 1 --2- 172.21.15.57:0/4170148766 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f35f40cf140 0x7f35f415bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:27.320 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.313+0000 7f35f93d4640 1 -- 172.21.15.57:0/4170148766 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f35f415b070 msgr2=0x7f35f40b2560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:27.320 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.313+0000 7f35f93d4640 1 --2- 172.21.15.57:0/4170148766 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f35f415b070 0x7f35f40b2560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:27.320 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.313+0000 7f35f93d4640 1 -- 172.21.15.57:0/4170148766 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35e4009490 con 0x7f35f40d1080 2023-10-19T14:22:27.321 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.313+0000 7f35f93d4640 1 --2- 172.21.15.57:0/4170148766 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f35f40d1080 0x7f35f41f0f90 secure :-1 s=READY pgs=2979 cs=0 l=1 rev1=1 crypto rx=0x7f35f000c460 tx=0x7f35f000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:27.321 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.313+0000 7f35ea7fc640 1 -- 172.21.15.57:0/4170148766 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35f0005480 con 0x7f35f40d1080 2023-10-19T14:22:27.321 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.313+0000 7f35ea7fc640 1 -- 172.21.15.57:0/4170148766 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f35f00035f0 con 0x7f35f40d1080 2023-10-19T14:22:27.321 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.313+0000 7f35ea7fc640 1 -- 172.21.15.57:0/4170148766 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35f0003a00 con 0x7f35f40d1080 2023-10-19T14:22:27.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.313+0000 7f35fbe39640 1 -- 172.21.15.57:0/4170148766 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35f415cff0 con 0x7f35f40d1080 2023-10-19T14:22:27.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.313+0000 7f35fbe39640 1 -- 172.21.15.57:0/4170148766 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35f41f14d0 con 0x7f35f40d1080 2023-10-19T14:22:27.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.314+0000 7f35fbe39640 1 -- 172.21.15.57:0/4170148766 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35f415e700 con 0x7f35f40d1080 2023-10-19T14:22:27.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.315+0000 7f35ea7fc640 1 -- 172.21.15.57:0/4170148766 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 151) v1 ==== 89388+0+0 (secure 0 0 0) 0x7f35f0003b60 con 0x7f35f40d1080 2023-10-19T14:22:27.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.316+0000 7f35ea7fc640 1 --2- 172.21.15.57:0/4170148766 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f35d806aff0 0x7f35d806d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:27.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.316+0000 7f35ea7fc640 1 -- 172.21.15.57:0/4170148766 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(264..264 src has 1..264) v4 ==== 6487+0+0 (secure 0 0 0) 0x7f35f0089770 con 0x7f35f40d1080 2023-10-19T14:22:27.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.316+0000 7f35f9bd5640 1 -- 172.21.15.57:0/4170148766 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f35d806aff0 msgr2=0x7f35d806d4b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/2005622276 2023-10-19T14:22:27.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.316+0000 7f35f9bd5640 1 --2- 172.21.15.57:0/4170148766 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f35d806aff0 0x7f35d806d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T14:22:27.325 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:22:27.316+0000 7f0d028fa040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-19T14:22:27.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.320+0000 7f35ea7fc640 1 -- 172.21.15.57:0/4170148766 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f35f00543d0 con 0x7f35f40d1080 2023-10-19T14:22:27.353 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:22:27.351+0000 7f29ada11040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-19T14:22:27.364 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:22:27.362+0000 7f0d028fa040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-19T14:22:27.395 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:22:27.392+0000 7f29ada11040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-19T14:22:27.420 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:22:27.417+0000 7f0d028fa040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-19T14:22:27.463 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:22:27.461+0000 7f0d028fa040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-19T14:22:27.494 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:22:27.492+0000 7f29ada11040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-19T14:22:27.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.497+0000 7f35fbe39640 1 -- 172.21.15.57:0/4170148766 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f35f409ed00 con 0x7f35f40d1080 2023-10-19T14:22:27.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.497+0000 7f35ea7fc640 1 -- 172.21.15.57:0/4170148766 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v264) v1 ==== 104+0+11 (secure 0 0 0) 0x7f35f00579f0 con 0x7f35f40d1080 2023-10-19T14:22:27.501 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2023-10-19T14:22:27.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.499+0000 7f35fbe39640 1 -- 172.21.15.57:0/4170148766 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f35d806aff0 msgr2=0x7f35d806d4b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T14:22:27.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.499+0000 7f35fbe39640 1 --2- 172.21.15.57:0/4170148766 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f35d806aff0 0x7f35d806d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:27.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.499+0000 7f35fbe39640 1 -- 172.21.15.57:0/4170148766 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f35f40d1080 msgr2=0x7f35f41f0f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:27.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.499+0000 7f35fbe39640 1 --2- 172.21.15.57:0/4170148766 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f35f40d1080 0x7f35f41f0f90 secure :-1 s=READY pgs=2979 cs=0 l=1 rev1=1 crypto rx=0x7f35f000c460 tx=0x7f35f000c920 comp rx=0 tx=0).stop 2023-10-19T14:22:27.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.499+0000 7f35fbe39640 1 -- 172.21.15.57:0/4170148766 shutdown_connections 2023-10-19T14:22:27.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.499+0000 7f35fbe39640 1 --2- 172.21.15.57:0/4170148766 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f35d806aff0 0x7f35d806d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:27.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.499+0000 7f35fbe39640 1 --2- 172.21.15.57:0/4170148766 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f35f415b070 0x7f35f40b2560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:27.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.499+0000 7f35fbe39640 1 --2- 172.21.15.57:0/4170148766 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f35f40d1080 0x7f35f41f0f90 unknown :-1 s=CLOSED pgs=2979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:27.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.499+0000 7f35fbe39640 1 --2- 172.21.15.57:0/4170148766 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f35f40cf140 0x7f35f415bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:27.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.499+0000 7f35fbe39640 1 -- 172.21.15.57:0/4170148766 >> 172.21.15.57:0/4170148766 conn(0x7f35f408a7e0 msgr2=0x7f35f40ce940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:27.504 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.499+0000 7f35fbe39640 1 -- 172.21.15.57:0/4170148766 shutdown_connections 2023-10-19T14:22:27.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.499+0000 7f35fbe39640 1 -- 172.21.15.57:0/4170148766 wait complete. 2023-10-19T14:22:27.513 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-19T14:22:27.546 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:22:27.544+0000 7f29ada11040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-19T14:22:27.569 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:22:27.566+0000 7f0d028fa040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-19T14:22:27.587 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:22:27.585+0000 7f29ada11040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-19T14:22:27.626 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:22:27.623+0000 7f0d028fa040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-19T14:22:27.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.624+0000 7fc11e94c640 1 -- 172.21.15.57:0/4163936043 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc1040047c0 msgr2=0x7fc104004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:27.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.624+0000 7fc11e94c640 1 --2- 172.21.15.57:0/4163936043 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc1040047c0 0x7fc104004c40 secure :-1 s=READY pgs=2981 cs=0 l=1 rev1=1 crypto rx=0x7fc1000017d0 tx=0x7fc1040057f0 comp rx=0 tx=0).stop 2023-10-19T14:22:27.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.624+0000 7fc11cee9640 1 -- 172.21.15.57:0/4163936043 reap_dead start 2023-10-19T14:22:27.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.624+0000 7fc11e94c640 1 -- 172.21.15.57:0/4163936043 shutdown_connections 2023-10-19T14:22:27.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.624+0000 7fc11e94c640 1 -- 172.21.15.57:0/4163936043 >> 172.21.15.57:0/4163936043 conn(0x7fc11808a7e0 msgr2=0x7fc11809a100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:27.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.624+0000 7fc11e94c640 1 -- 172.21.15.57:0/4163936043 shutdown_connections 2023-10-19T14:22:27.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.624+0000 7fc11e94c640 1 -- 172.21.15.57:0/4163936043 wait complete. 2023-10-19T14:22:27.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.625+0000 7fc11e94c640 1 Processor -- start 2023-10-19T14:22:27.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.626+0000 7fc11e94c640 1 -- start start 2023-10-19T14:22:27.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.626+0000 7fc11e94c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc1040047c0 0x7fc118156120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:27.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.626+0000 7fc11e94c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc11815c750 0x7fc1180c9970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:27.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.626+0000 7fc11e94c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc11815d710 0x7fc11815db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:27.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.626+0000 7fc11e94c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc11815e0d0 con 0x7fc1040047c0 2023-10-19T14:22:27.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.626+0000 7fc11e94c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1180cadf0 con 0x7fc11815d710 2023-10-19T14:22:27.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.626+0000 7fc11e94c640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1180ca270 con 0x7fc11815c750 2023-10-19T14:22:27.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.626+0000 7fc117fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc1040047c0 0x7fc118156120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:27.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.626+0000 7fc117fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc1040047c0 0x7fc118156120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38786/0 (socket says 172.21.15.57:38786) 2023-10-19T14:22:27.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.626+0000 7fc117fff640 1 -- 172.21.15.57:0/2225752777 learned_addr learned my addr 172.21.15.57:0/2225752777 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:22:27.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.626+0000 7fc1177fe640 1 --2- 172.21.15.57:0/2225752777 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc11815c750 0x7fc1180c9970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:27.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.626+0000 7fc11cee9640 1 --2- 172.21.15.57:0/2225752777 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc11815d710 0x7fc11815db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:27.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.627+0000 7fc117fff640 1 -- 172.21.15.57:0/2225752777 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc11815c750 msgr2=0x7fc1180c9970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:27.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.627+0000 7fc117fff640 1 --2- 172.21.15.57:0/2225752777 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc11815c750 0x7fc1180c9970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:27.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.627+0000 7fc117fff640 1 -- 172.21.15.57:0/2225752777 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc11815d710 msgr2=0x7fc11815db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:27.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.627+0000 7fc117fff640 1 --2- 172.21.15.57:0/2225752777 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc11815d710 0x7fc11815db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:27.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.627+0000 7fc117fff640 1 -- 172.21.15.57:0/2225752777 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1180ca550 con 0x7fc1040047c0 2023-10-19T14:22:27.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.627+0000 7fc117fff640 1 --2- 172.21.15.57:0/2225752777 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc1040047c0 0x7fc118156120 secure :-1 s=READY pgs=2982 cs=0 l=1 rev1=1 crypto rx=0x7fc108003670 tx=0x7fc108003b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:27.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.627+0000 7fc114ff9640 1 -- 172.21.15.57:0/2225752777 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc10800fb30 con 0x7fc1040047c0 2023-10-19T14:22:27.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.627+0000 7fc114ff9640 1 -- 172.21.15.57:0/2225752777 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fc10800fc90 con 0x7fc1040047c0 2023-10-19T14:22:27.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.627+0000 7fc11e94c640 1 -- 172.21.15.57:0/2225752777 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1180ca830 con 0x7fc1040047c0 2023-10-19T14:22:27.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.627+0000 7fc114ff9640 1 -- 172.21.15.57:0/2225752777 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc108011430 con 0x7fc1040047c0 2023-10-19T14:22:27.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.627+0000 7fc11e94c640 1 -- 172.21.15.57:0/2225752777 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc118156660 con 0x7fc1040047c0 2023-10-19T14:22:27.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.628+0000 7fc11e94c640 1 -- 172.21.15.57:0/2225752777 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1180b4a00 con 0x7fc1040047c0 2023-10-19T14:22:27.636 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:22:27.633+0000 7f29ada11040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-19T14:22:27.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.629+0000 7fc114ff9640 1 -- 172.21.15.57:0/2225752777 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 151) v1 ==== 89388+0+0 (secure 0 0 0) 0x7fc108011590 con 0x7fc1040047c0 2023-10-19T14:22:27.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.629+0000 7fc114ff9640 1 --2- 172.21.15.57:0/2225752777 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7fc0f806b020 0x7fc0f806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:27.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.629+0000 7fc1177fe640 1 -- 172.21.15.57:0/2225752777 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7fc0f806b020 msgr2=0x7fc0f806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/2005622276 2023-10-19T14:22:27.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.629+0000 7fc114ff9640 1 -- 172.21.15.57:0/2225752777 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(264..264 src has 1..264) v4 ==== 6487+0+0 (secure 0 0 0) 0x7fc10808ab60 con 0x7fc1040047c0 2023-10-19T14:22:27.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.630+0000 7fc1177fe640 1 --2- 172.21.15.57:0/2225752777 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7fc0f806b020 0x7fc0f806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T14:22:27.638 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.634+0000 7fc114ff9640 1 -- 172.21.15.57:0/2225752777 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc1180b4a00 con 0x7fc1040047c0 2023-10-19T14:22:27.672 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:22:27.669+0000 7f0d028fa040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-19T14:22:27.678 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:22:27.676+0000 7f29ada11040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-19T14:22:27.723 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:22:27.720+0000 7f0d028fa040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-19T14:22:27.768 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:22:27.765+0000 7f0d028fa040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-19T14:22:27.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.810+0000 7fc11e94c640 1 -- 172.21.15.57:0/2225752777 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc1180b4a00 con 0x7fc1040047c0 2023-10-19T14:22:27.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.810+0000 7fc114ff9640 1 -- 172.21.15.57:0/2225752777 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v264) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc1180b4a00 con 0x7fc1040047c0 2023-10-19T14:22:27.814 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 64 2023-10-19T14:22:27.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.812+0000 7fc11e94c640 1 -- 172.21.15.57:0/2225752777 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7fc0f806b020 msgr2=0x7fc0f806d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T14:22:27.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.812+0000 7fc11e94c640 1 --2- 172.21.15.57:0/2225752777 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7fc0f806b020 0x7fc0f806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:27.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.812+0000 7fc11e94c640 1 -- 172.21.15.57:0/2225752777 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc1040047c0 msgr2=0x7fc118156120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:27.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.812+0000 7fc11e94c640 1 --2- 172.21.15.57:0/2225752777 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc1040047c0 0x7fc118156120 secure :-1 s=READY pgs=2982 cs=0 l=1 rev1=1 crypto rx=0x7fc108003670 tx=0x7fc108003b30 comp rx=0 tx=0).stop 2023-10-19T14:22:27.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.812+0000 7fc11e94c640 1 -- 172.21.15.57:0/2225752777 shutdown_connections 2023-10-19T14:22:27.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.812+0000 7fc11e94c640 1 --2- 172.21.15.57:0/2225752777 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7fc0f806b020 0x7fc0f806d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:27.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.812+0000 7fc11e94c640 1 --2- 172.21.15.57:0/2225752777 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc11815d710 0x7fc11815db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:27.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.812+0000 7fc11e94c640 1 --2- 172.21.15.57:0/2225752777 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc11815c750 0x7fc1180c9970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:27.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.812+0000 7fc11e94c640 1 --2- 172.21.15.57:0/2225752777 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc1040047c0 0x7fc118156120 unknown :-1 s=CLOSED pgs=2982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:27.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.812+0000 7fc11e94c640 1 -- 172.21.15.57:0/2225752777 >> 172.21.15.57:0/2225752777 conn(0x7fc11808a7e0 msgr2=0x7fc1180ada20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:27.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.812+0000 7fc11e94c640 1 -- 172.21.15.57:0/2225752777 shutdown_connections 2023-10-19T14:22:27.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.812+0000 7fc11e94c640 1 -- 172.21.15.57:0/2225752777 wait complete. 2023-10-19T14:22:27.826 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-19T14:22:27.937 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.934+0000 7f08227b0640 1 -- 172.21.15.57:0/4077665726 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f081c0cf8e0 msgr2=0x7f081c0b4840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:27.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.934+0000 7f08227b0640 1 --2- 172.21.15.57:0/4077665726 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f081c0cf8e0 0x7f081c0b4840 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f0814005fd0 tx=0x7f081402bfa0 comp rx=0 tx=0).stop 2023-10-19T14:22:27.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.934+0000 7f08227b0640 1 -- 172.21.15.57:0/4077665726 shutdown_connections 2023-10-19T14:22:27.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.934+0000 7f08227b0640 1 --2- 172.21.15.57:0/4077665726 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f081c0cf8e0 0x7f081c0b4840 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:27.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.934+0000 7f08227b0640 1 --2- 172.21.15.57:0/4077665726 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f081c15e5d0 0x7f081c0cdfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:27.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.934+0000 7f08227b0640 1 -- 172.21.15.57:0/4077665726 >> 172.21.15.57:0/4077665726 conn(0x7f081c09a140 msgr2=0x7f081c0d0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:27.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.934+0000 7f08227b0640 1 -- 172.21.15.57:0/4077665726 shutdown_connections 2023-10-19T14:22:27.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.934+0000 7f08227b0640 1 -- 172.21.15.57:0/4077665726 wait complete. 2023-10-19T14:22:27.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.935+0000 7f08227b0640 1 Processor -- start 2023-10-19T14:22:27.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.936+0000 7f08227b0640 1 -- start start 2023-10-19T14:22:27.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.936+0000 7f08227b0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f081c0ce980 0x7f081c158480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:27.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.936+0000 7f08227b0640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f081c0cf8e0 0x7f081c156010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:27.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.936+0000 7f08227b0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f081c15e5d0 0x7f081c0c71c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:27.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.936+0000 7f08227b0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f081c1589c0 con 0x7f081c15e5d0 2023-10-19T14:22:27.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.936+0000 7f08227b0640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f081c156550 con 0x7f081c0ce980 2023-10-19T14:22:27.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.936+0000 7f08227b0640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f081c158bb0 con 0x7f081c0cf8e0 2023-10-19T14:22:27.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.936+0000 7f0820d4d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f081c15e5d0 0x7f081c0c71c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:27.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.936+0000 7f0820d4d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f081c15e5d0 0x7f081c0c71c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38802/0 (socket says 172.21.15.57:38802) 2023-10-19T14:22:27.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.936+0000 7f0820d4d640 1 -- 172.21.15.57:0/3549858118 learned_addr learned my addr 172.21.15.57:0/3549858118 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:22:27.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.936+0000 7f081b7fe640 1 --2- 172.21.15.57:0/3549858118 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f081c0cf8e0 0x7f081c156010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:27.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.936+0000 7f081bfff640 1 --2- 172.21.15.57:0/3549858118 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f081c0ce980 0x7f081c158480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:27.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.937+0000 7f0820d4d640 1 -- 172.21.15.57:0/3549858118 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f081c0cf8e0 msgr2=0x7f081c156010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:27.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.937+0000 7f0820d4d640 1 --2- 172.21.15.57:0/3549858118 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f081c0cf8e0 0x7f081c156010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:27.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.937+0000 7f0820d4d640 1 -- 172.21.15.57:0/3549858118 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f081c0ce980 msgr2=0x7f081c158480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:27.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.937+0000 7f0820d4d640 1 --2- 172.21.15.57:0/3549858118 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f081c0ce980 0x7f081c158480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:27.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.937+0000 7f0820d4d640 1 -- 172.21.15.57:0/3549858118 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0810009490 con 0x7f081c15e5d0 2023-10-19T14:22:27.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.937+0000 7f0820d4d640 1 --2- 172.21.15.57:0/3549858118 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f081c15e5d0 0x7f081c0c71c0 secure :-1 s=READY pgs=2984 cs=0 l=1 rev1=1 crypto rx=0x7f0814036040 tx=0x7f0814003f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:27.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.937+0000 7f0818ff9640 1 -- 172.21.15.57:0/3549858118 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08140043d0 con 0x7f081c15e5d0 2023-10-19T14:22:27.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.937+0000 7f0818ff9640 1 -- 172.21.15.57:0/3549858118 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f0814004640 con 0x7f081c15e5d0 2023-10-19T14:22:27.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.937+0000 7f08227b0640 1 -- 172.21.15.57:0/3549858118 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0814009840 con 0x7f081c15e5d0 2023-10-19T14:22:27.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.937+0000 7f0818ff9640 1 -- 172.21.15.57:0/3549858118 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f081403e450 con 0x7f081c15e5d0 2023-10-19T14:22:27.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.937+0000 7f08227b0640 1 -- 172.21.15.57:0/3549858118 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f081c0c9050 con 0x7f081c15e5d0 2023-10-19T14:22:27.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.938+0000 7f08227b0640 1 -- 172.21.15.57:0/3549858118 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f081c1590d0 con 0x7f081c15e5d0 2023-10-19T14:22:27.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.939+0000 7f0818ff9640 1 -- 172.21.15.57:0/3549858118 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 151) v1 ==== 89388+0+0 (secure 0 0 0) 0x7f081403e5b0 con 0x7f081c15e5d0 2023-10-19T14:22:27.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.939+0000 7f0818ff9640 1 --2- 172.21.15.57:0/3549858118 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f080806b020 0x7f080806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:27.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.939+0000 7f0818ff9640 1 -- 172.21.15.57:0/3549858118 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(264..264 src has 1..264) v4 ==== 6487+0+0 (secure 0 0 0) 0x7f08140ac9d0 con 0x7f081c15e5d0 2023-10-19T14:22:27.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.939+0000 7f081bfff640 1 -- 172.21.15.57:0/3549858118 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f080806b020 msgr2=0x7f080806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/2005622276 2023-10-19T14:22:27.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.939+0000 7f081bfff640 1 --2- 172.21.15.57:0/3549858118 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f080806b020 0x7f080806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T14:22:27.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:27.943+0000 7f0818ff9640 1 -- 172.21.15.57:0/3549858118 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f0814077620 con 0x7f081c15e5d0 2023-10-19T14:22:27.997 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:22:27.994+0000 7f29ada11040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-19T14:22:28.108 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:22:28.106+0000 7f0d028fa040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-19T14:22:28.126 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:22:28.123+0000 7f29ada11040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-19T14:22:28.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.134+0000 7f08227b0640 1 -- 172.21.15.57:0/3549858118 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f081c069100 con 0x7f081c15e5d0 2023-10-19T14:22:28.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.134+0000 7f0818ff9640 1 -- 172.21.15.57:0/3549858118 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v264) v1 ==== 81+0+593 (secure 0 0 0) 0x7f0814043080 con 0x7f081c15e5d0 2023-10-19T14:22:28.137 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/1983646445 2023-10-20T14:22:17.865415+0000 2023-10-19T14:22:28.138 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/1840191723 2023-10-20T14:22:17.865415+0000 2023-10-19T14:22:28.138 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/4168916968 2023-10-20T14:22:17.865415+0000 2023-10-19T14:22:28.138 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:6833/1573806521 2023-10-20T14:22:17.865415+0000 2023-10-19T14:22:28.138 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:6832/1573806521 2023-10-20T14:22:17.865415+0000 2023-10-19T14:22:28.139 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/658875312 2023-10-20T14:22:17.865415+0000 2023-10-19T14:22:28.139 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/1799772260 2023-10-20T14:22:17.865415+0000 2023-10-19T14:22:28.139 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/3831670324 2023-10-20T14:22:17.865415+0000 2023-10-19T14:22:28.139 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/90219027 2023-10-20T14:22:17.865415+0000 2023-10-19T14:22:28.140 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.179:0/2739598080 2023-10-20T14:22:17.865415+0000 2023-10-19T14:22:28.141 INFO:teuthology.orchestra.run.smithi057.stderr:listed 10 entries 2023-10-19T14:22:28.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.136+0000 7f08227b0640 1 -- 172.21.15.57:0/3549858118 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f080806b020 msgr2=0x7f080806d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T14:22:28.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.136+0000 7f08227b0640 1 --2- 172.21.15.57:0/3549858118 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f080806b020 0x7f080806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:28.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.136+0000 7f08227b0640 1 -- 172.21.15.57:0/3549858118 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f081c15e5d0 msgr2=0x7f081c0c71c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:28.141 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.136+0000 7f08227b0640 1 --2- 172.21.15.57:0/3549858118 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f081c15e5d0 0x7f081c0c71c0 secure :-1 s=READY pgs=2984 cs=0 l=1 rev1=1 crypto rx=0x7f0814036040 tx=0x7f0814003f20 comp rx=0 tx=0).stop 2023-10-19T14:22:28.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.136+0000 7f08227b0640 1 -- 172.21.15.57:0/3549858118 shutdown_connections 2023-10-19T14:22:28.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.136+0000 7f08227b0640 1 --2- 172.21.15.57:0/3549858118 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f080806b020 0x7f080806d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:28.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.136+0000 7f08227b0640 1 --2- 172.21.15.57:0/3549858118 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f081c15e5d0 0x7f081c0c71c0 unknown :-1 s=CLOSED pgs=2984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:28.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.136+0000 7f08227b0640 1 --2- 172.21.15.57:0/3549858118 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f081c0cf8e0 0x7f081c156010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:28.142 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.136+0000 7f08227b0640 1 --2- 172.21.15.57:0/3549858118 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f081c0ce980 0x7f081c158480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:28.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.136+0000 7f08227b0640 1 -- 172.21.15.57:0/3549858118 >> 172.21.15.57:0/3549858118 conn(0x7f081c09a140 msgr2=0x7f081c06acc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:28.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.136+0000 7f08227b0640 1 -- 172.21.15.57:0/3549858118 shutdown_connections 2023-10-19T14:22:28.143 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.136+0000 7f08227b0640 1 -- 172.21.15.57:0/3549858118 wait complete. 2023-10-19T14:22:28.150 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi057.front.sepia.ceph.com... 2023-10-19T14:22:28.151 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:22:28.151 DEBUG:teuthology.orchestra.run.smithi057:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-10-19T14:22:28.180 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi057.stderr:ceph-fuse[164782]: fuse finished with error 0 and tester_r 0 2023-10-19T14:22:28.182 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi057.front.sepia.ceph.com 2023-10-19T14:22:28.183 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:22:28.183 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-19T14:22:28.218 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:22:28.216+0000 7f29ada11040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-19T14:22:28.250 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:22:28.247+0000 7f0d028fa040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-19T14:22:28.258 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-19T14:22:28.259 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T14:22:28.261 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:22:28.258+0000 7f29ada11040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-19T14:22:28.274 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054833,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=284 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3247632k,nr_inodes=811908,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-19T14:22:28.275 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi179.front.sepia.ceph.com 2023-10-19T14:22:28.275 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:22:28.275 DEBUG:teuthology.orchestra.run.smithi179:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-10-19T14:22:28.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-19T14:22:28.330 INFO:teuthology.orchestra.run.smithi179.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2023-10-19T14:22:28.330 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T14:22:28.331 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T14:22:28.347 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055614,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=358 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248260k,nr_inodes=812065,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-19T14:22:28.347 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi057.front.sepia.ceph.com 2023-10-19T14:22:28.348 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:22:28.348 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-10-19T14:22:28.349 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:22:28.344+0000 7f0d028fa040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-19T14:22:28.361 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:22:28.359+0000 7f29ada11040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-19T14:22:28.392 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:22:28.389+0000 7f0d028fa040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-19T14:22:28.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-19T14:22:28.403 INFO:teuthology.orchestra.run.smithi057.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2023-10-19T14:22:28.403 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-19T14:22:28.404 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T14:22:28.420 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054833,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=284 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3247632k,nr_inodes=811908,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-19T14:22:28.420 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi179.front.sepia.ceph.com 2023-10-19T14:22:28.420 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:22:28.421 DEBUG:teuthology.orchestra.run.smithi179:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-10-19T14:22:28.459 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:22:28.456+0000 7f29ada11040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-19T14:22:28.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-19T14:22:28.473 INFO:teuthology.orchestra.run.smithi179.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2023-10-19T14:22:28.474 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T14:22:28.493 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:22:28.490+0000 7f0d028fa040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-19T14:22:28.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.581+0000 7fe8c0f05640 1 -- 172.21.15.57:0/4117412348 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe8bc15e5d0 msgr2=0x7fe8bc0cdfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:28.585 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.581+0000 7fe8c0f05640 1 --2- 172.21.15.57:0/4117412348 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe8bc15e5d0 0x7fe8bc0cdfd0 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac009830 tx=0x7fe8ac02bc20 comp rx=0 tx=0).stop 2023-10-19T14:22:28.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.581+0000 7fe8c0f05640 1 -- 172.21.15.57:0/4117412348 shutdown_connections 2023-10-19T14:22:28.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.581+0000 7fe8c0f05640 1 --2- 172.21.15.57:0/4117412348 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe8bc0ce980 0x7fe8bc0cf1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:28.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.581+0000 7fe8c0f05640 1 --2- 172.21.15.57:0/4117412348 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe8bc15e5d0 0x7fe8bc0cdfd0 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:28.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.581+0000 7fe8c0f05640 1 -- 172.21.15.57:0/4117412348 >> 172.21.15.57:0/4117412348 conn(0x7fe8bc09a140 msgr2=0x7fe8bc0d0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:28.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.581+0000 7fe8c0f05640 1 -- 172.21.15.57:0/4117412348 shutdown_connections 2023-10-19T14:22:28.587 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.581+0000 7fe8c0f05640 1 -- 172.21.15.57:0/4117412348 wait complete. 2023-10-19T14:22:28.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.583+0000 7fe8c0f05640 1 Processor -- start 2023-10-19T14:22:28.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.583+0000 7fe8c0f05640 1 -- start start 2023-10-19T14:22:28.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.584+0000 7fe8c0f05640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe8bc0ce980 0x7fe8bc0b60b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:28.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.584+0000 7fe8c0f05640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe8bc0cf710 0x7fe8bc145d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:28.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.584+0000 7fe8c0f05640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe8bc15e5d0 0x7fe8bc1462d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:28.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.584+0000 7fe8c0f05640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8bc1e51e0 con 0x7fe8bc0ce980 2023-10-19T14:22:28.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.584+0000 7fe8c0f05640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8bc0c74d0 con 0x7fe8bc15e5d0 2023-10-19T14:22:28.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.584+0000 7fe8c0f05640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8bc1e48b0 con 0x7fe8bc0cf710 2023-10-19T14:22:28.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.584+0000 7fe8ba59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe8bc0ce980 0x7fe8bc0b60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:28.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.584+0000 7fe8ba59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe8bc0ce980 0x7fe8bc0b60b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38806/0 (socket says 172.21.15.57:38806) 2023-10-19T14:22:28.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.584+0000 7fe8ba59c640 1 -- 172.21.15.57:0/3239886534 learned_addr learned my addr 172.21.15.57:0/3239886534 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:22:28.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.584+0000 7fe8b1d9b640 1 --2- 172.21.15.57:0/3239886534 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe8bc0cf710 0x7fe8bc145d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:28.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.584+0000 7fe8bad9d640 1 --2- 172.21.15.57:0/3239886534 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe8bc15e5d0 0x7fe8bc1462d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:28.591 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.584+0000 7fe8ba59c640 1 -- 172.21.15.57:0/3239886534 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe8bc0cf710 msgr2=0x7fe8bc145d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:28.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.584+0000 7fe8ba59c640 1 --2- 172.21.15.57:0/3239886534 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe8bc0cf710 0x7fe8bc145d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:28.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.584+0000 7fe8ba59c640 1 -- 172.21.15.57:0/3239886534 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe8bc15e5d0 msgr2=0x7fe8bc1462d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:28.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.584+0000 7fe8ba59c640 1 --2- 172.21.15.57:0/3239886534 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe8bc15e5d0 0x7fe8bc1462d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:28.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.584+0000 7fe8ba59c640 1 -- 172.21.15.57:0/3239886534 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8a4009490 con 0x7fe8bc0ce980 2023-10-19T14:22:28.592 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.584+0000 7fe8ba59c640 1 --2- 172.21.15.57:0/3239886534 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe8bc0ce980 0x7fe8bc0b60b0 secure :-1 s=READY pgs=2986 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac009780 tx=0x7fe8ac02cd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:28.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.585+0000 7fe8b37fe640 1 -- 172.21.15.57:0/3239886534 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8ac003a00 con 0x7fe8bc0ce980 2023-10-19T14:22:28.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.585+0000 7fe8c0f05640 1 -- 172.21.15.57:0/3239886534 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8ac009490 con 0x7fe8bc0ce980 2023-10-19T14:22:28.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.585+0000 7fe8b37fe640 1 -- 172.21.15.57:0/3239886534 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fe8ac003c70 con 0x7fe8bc0ce980 2023-10-19T14:22:28.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.585+0000 7fe8b37fe640 1 -- 172.21.15.57:0/3239886534 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8ac005c90 con 0x7fe8bc0ce980 2023-10-19T14:22:28.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.585+0000 7fe8c0f05640 1 -- 172.21.15.57:0/3239886534 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8bc1e4e70 con 0x7fe8bc0ce980 2023-10-19T14:22:28.593 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.586+0000 7fe8c0f05640 1 -- 172.21.15.57:0/3239886534 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8bc069100 con 0x7fe8bc0ce980 2023-10-19T14:22:28.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.586+0000 7fe8b37fe640 1 -- 172.21.15.57:0/3239886534 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 151) v1 ==== 89388+0+0 (secure 0 0 0) 0x7fe8ac03b040 con 0x7fe8bc0ce980 2023-10-19T14:22:28.594 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.587+0000 7fe8b37fe640 1 --2- 172.21.15.57:0/3239886534 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7fe89c06aff0 0x7fe89c06d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:28.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.587+0000 7fe8b37fe640 1 -- 172.21.15.57:0/3239886534 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(264..264 src has 1..264) v4 ==== 6487+0+0 (secure 0 0 0) 0x7fe8ac0abbd0 con 0x7fe8bc0ce980 2023-10-19T14:22:28.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.587+0000 7fe8b1d9b640 1 -- 172.21.15.57:0/3239886534 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7fe89c06aff0 msgr2=0x7fe89c06d4b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/2005622276 2023-10-19T14:22:28.595 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.587+0000 7fe8b1d9b640 1 --2- 172.21.15.57:0/3239886534 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7fe89c06aff0 0x7fe89c06d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T14:22:28.595 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:22:28.591+0000 7f0d028fa040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-19T14:22:28.597 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:22:28.595+0000 7f29ada11040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-19T14:22:28.598 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.595+0000 7fe8b37fe640 1 -- 172.21.15.57:0/3239886534 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe8ac076690 con 0x7fe8bc0ce980 2023-10-19T14:22:28.639 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:22:28.636+0000 7f29ada11040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-19T14:22:28.741 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:22:28.738+0000 7f0d028fa040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-19T14:22:28.744 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:22:28.741+0000 7f29ada11040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-19T14:22:28.785 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:22:28.782+0000 7f0d028fa040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-19T14:22:28.790 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.788+0000 7fe8b1d9b640 1 -- 172.21.15.57:0/3239886534 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7fe89c06aff0 msgr2=0x7fe89c06d4b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/2005622276 2023-10-19T14:22:28.791 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.788+0000 7fe8b1d9b640 1 --2- 172.21.15.57:0/3239886534 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7fe89c06aff0 0x7fe89c06d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-19T14:22:28.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.809+0000 7fe8c0f05640 1 -- 172.21.15.57:0/3239886534 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe8bc0caeb0 con 0x7fe8bc0ce980 2023-10-19T14:22:28.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.810+0000 7fe8b37fe640 1 -- 172.21.15.57:0/3239886534 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+4935 (secure 0 0 0) 0x7fe8ac041020 con 0x7fe8bc0ce980 2023-10-19T14:22:28.813 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:22:28.814 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":116,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10280,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/1738328496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":1738328496},{"type":"v1","addr":"172.21.15.179:6837","nonce":1738328496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":113},{"gid":13344,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/3191801579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":3191801579},{"type":"v1","addr":"172.21.15.57:6834","nonce":3191801579}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":113},{"gid":13347,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/3876822488","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":3876822488},{"type":"v1","addr":"172.21.15.57:6835","nonce":3876822488}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T14:22:01.763128+0000","modified":"2023-10-19T14:22:03.758375+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":13359},"failed":[],"damaged":[],"stopped":[],"info":{"gid_13359":{"gid":13359,"name":"d","rank":0,"incarnation":114,"state":"up:active","state_seq":3,"addr":"172.21.15.179:6836/4133411706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":4133411706},{"type":"v1","addr":"172.21.15.179:6836","nonce":4133411706}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[36],"metadata_pool":35,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":17}]} 2023-10-19T14:22:28.814 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 116 2023-10-19T14:22:28.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.812+0000 7fe8c0f05640 1 -- 172.21.15.57:0/3239886534 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7fe89c06aff0 msgr2=0x7fe89c06d4b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T14:22:28.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.812+0000 7fe8c0f05640 1 --2- 172.21.15.57:0/3239886534 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7fe89c06aff0 0x7fe89c06d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:28.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.812+0000 7fe8c0f05640 1 -- 172.21.15.57:0/3239886534 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe8bc0ce980 msgr2=0x7fe8bc0b60b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:28.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.812+0000 7fe8c0f05640 1 --2- 172.21.15.57:0/3239886534 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe8bc0ce980 0x7fe8bc0b60b0 secure :-1 s=READY pgs=2986 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac009780 tx=0x7fe8ac02cd10 comp rx=0 tx=0).stop 2023-10-19T14:22:28.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.812+0000 7fe8c0f05640 1 -- 172.21.15.57:0/3239886534 shutdown_connections 2023-10-19T14:22:28.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.812+0000 7fe8c0f05640 1 --2- 172.21.15.57:0/3239886534 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7fe89c06aff0 0x7fe89c06d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:28.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.812+0000 7fe8c0f05640 1 --2- 172.21.15.57:0/3239886534 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe8bc15e5d0 0x7fe8bc1462d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:28.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.812+0000 7fe8c0f05640 1 --2- 172.21.15.57:0/3239886534 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe8bc0cf710 0x7fe8bc145d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:28.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.812+0000 7fe8c0f05640 1 --2- 172.21.15.57:0/3239886534 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe8bc0ce980 0x7fe8bc0b60b0 unknown :-1 s=CLOSED pgs=2986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:28.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.812+0000 7fe8c0f05640 1 -- 172.21.15.57:0/3239886534 >> 172.21.15.57:0/3239886534 conn(0x7fe8bc09a140 msgr2=0x7fe8bc0d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:28.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.812+0000 7fe8c0f05640 1 -- 172.21.15.57:0/3239886534 shutdown_connections 2023-10-19T14:22:28.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.812+0000 7fe8c0f05640 1 -- 172.21.15.57:0/3239886534 wait complete. 2023-10-19T14:22:28.826 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T14:22:28.897 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:22:28.895+0000 7f0d028fa040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-19T14:22:28.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.973+0000 7f7d91a44640 1 -- 172.21.15.57:0/1200739451 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7d8c157ca0 msgr2=0x7f7d8c0b1790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:28.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.973+0000 7f7d91a44640 1 --2- 172.21.15.57:0/1200739451 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7d8c157ca0 0x7f7d8c0b1790 secure :-1 s=READY pgs=2987 cs=0 l=1 rev1=1 crypto rx=0x7f7d7c005fd0 tx=0x7f7d7c02bd90 comp rx=0 tx=0).stop 2023-10-19T14:22:28.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.973+0000 7f7d91a44640 1 -- 172.21.15.57:0/1200739451 shutdown_connections 2023-10-19T14:22:28.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.973+0000 7f7d91a44640 1 --2- 172.21.15.57:0/1200739451 >> v2:172.21.15.179:3300/0 conn(0x7f7d8c065d90 0x7f7d8c0b1cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:28.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.973+0000 7f7d91a44640 1 --2- 172.21.15.57:0/1200739451 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7d8c157ca0 0x7f7d8c0b1790 unknown :-1 s=CLOSED pgs=2987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:28.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.973+0000 7f7d91a44640 1 -- 172.21.15.57:0/1200739451 >> 172.21.15.57:0/1200739451 conn(0x7f7d8c070980 msgr2=0x7f7d8c069630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:28.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.973+0000 7f7d91a44640 1 -- 172.21.15.57:0/1200739451 shutdown_connections 2023-10-19T14:22:28.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.973+0000 7f7d91a44640 1 -- 172.21.15.57:0/1200739451 wait complete. 2023-10-19T14:22:28.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.974+0000 7f7d91a44640 1 Processor -- start 2023-10-19T14:22:28.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.974+0000 7f7d91a44640 1 -- start start 2023-10-19T14:22:28.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.975+0000 7f7d91a44640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7d8c065d90 0x7f7d8c15a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:28.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.975+0000 7f7d91a44640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7d8c0c9ed0 0x7f7d8c1564b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:28.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.975+0000 7f7d91a44640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7d8c157ca0 0x7f7d8c149780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:28.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.975+0000 7f7d91a44640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d8c156020 con 0x7f7d8c157ca0 2023-10-19T14:22:28.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.975+0000 7f7d91a44640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d8c145ef0 con 0x7f7d8c065d90 2023-10-19T14:22:28.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.975+0000 7f7d91a44640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d8c155cf0 con 0x7f7d8c0c9ed0 2023-10-19T14:22:28.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.975+0000 7f7d8b7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7d8c157ca0 0x7f7d8c149780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:28.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.975+0000 7f7d8b7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7d8c157ca0 0x7f7d8c149780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38834/0 (socket says 172.21.15.57:38834) 2023-10-19T14:22:28.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.975+0000 7f7d8b7fe640 1 -- 172.21.15.57:0/4224348803 learned_addr learned my addr 172.21.15.57:0/4224348803 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:22:28.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.975+0000 7f7d8affd640 1 --2- 172.21.15.57:0/4224348803 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7d8c065d90 0x7f7d8c15a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:28.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.975+0000 7f7d8a7fc640 1 --2- 172.21.15.57:0/4224348803 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7d8c0c9ed0 0x7f7d8c1564b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:28.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.975+0000 7f7d8b7fe640 1 -- 172.21.15.57:0/4224348803 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7d8c0c9ed0 msgr2=0x7f7d8c1564b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:28.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.975+0000 7f7d8b7fe640 1 --2- 172.21.15.57:0/4224348803 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7d8c0c9ed0 0x7f7d8c1564b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:28.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.975+0000 7f7d8b7fe640 1 -- 172.21.15.57:0/4224348803 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7d8c065d90 msgr2=0x7f7d8c15a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:28.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.975+0000 7f7d8b7fe640 1 --2- 172.21.15.57:0/4224348803 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7d8c065d90 0x7f7d8c15a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:28.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.975+0000 7f7d8b7fe640 1 -- 172.21.15.57:0/4224348803 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d7c009560 con 0x7f7d8c157ca0 2023-10-19T14:22:28.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.976+0000 7f7d8b7fe640 1 --2- 172.21.15.57:0/4224348803 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7d8c157ca0 0x7f7d8c149780 secure :-1 s=READY pgs=2988 cs=0 l=1 rev1=1 crypto rx=0x7f7d8400c760 tx=0x7f7d8400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:28.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.976+0000 7f7d73fff640 1 -- 172.21.15.57:0/4224348803 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d84003b70 con 0x7f7d8c157ca0 2023-10-19T14:22:28.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.976+0000 7f7d73fff640 1 -- 172.21.15.57:0/4224348803 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f7d84003cd0 con 0x7f7d8c157ca0 2023-10-19T14:22:28.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.976+0000 7f7d73fff640 1 -- 172.21.15.57:0/4224348803 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d8400aab0 con 0x7f7d8c157ca0 2023-10-19T14:22:28.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.976+0000 7f7d91a44640 1 -- 172.21.15.57:0/4224348803 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d8c14a320 con 0x7f7d8c157ca0 2023-10-19T14:22:28.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.976+0000 7f7d91a44640 1 -- 172.21.15.57:0/4224348803 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d8c1569f0 con 0x7f7d8c157ca0 2023-10-19T14:22:28.985 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.977+0000 7f7d91a44640 1 -- 172.21.15.57:0/4224348803 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d8c066980 con 0x7f7d8c157ca0 2023-10-19T14:22:28.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.978+0000 7f7d73fff640 1 -- 172.21.15.57:0/4224348803 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 151) v1 ==== 89388+0+0 (secure 0 0 0) 0x7f7d8400ac10 con 0x7f7d8c157ca0 2023-10-19T14:22:28.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.978+0000 7f7d73fff640 1 --2- 172.21.15.57:0/4224348803 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f7d7806aff0 0x7f7d7806d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:28.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.978+0000 7f7d8affd640 1 -- 172.21.15.57:0/4224348803 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f7d7806aff0 msgr2=0x7f7d7806d4b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/2005622276 2023-10-19T14:22:28.986 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.978+0000 7f7d73fff640 1 -- 172.21.15.57:0/4224348803 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(264..264 src has 1..264) v4 ==== 6487+0+0 (secure 0 0 0) 0x7f7d84089b00 con 0x7f7d8c157ca0 2023-10-19T14:22:28.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.978+0000 7f7d8affd640 1 --2- 172.21.15.57:0/4224348803 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f7d7806aff0 0x7f7d7806d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T14:22:28.987 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:28.983+0000 7f7d73fff640 1 -- 172.21.15.57:0/4224348803 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7d8408be60 con 0x7f7d8c157ca0 2023-10-19T14:22:29.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.161+0000 7f7d91a44640 1 -- 172.21.15.57:0/4224348803 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7d8c069100 con 0x7f7d8c157ca0 2023-10-19T14:22:29.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.162+0000 7f7d73fff640 1 -- 172.21.15.57:0/4224348803 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v264) v1 ==== 74+0+17407 (secure 0 0 0) 0x7f7d84054640 con 0x7f7d8c157ca0 2023-10-19T14:22:29.165 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:22:29.166 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":264,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T14:22:17.865478+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":36,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_metadata","create_time":"2023-10-19T14:22:00.047912+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"262","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_data","create_time":"2023-10-19T14:22:00.739698+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"263","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.179:0/1983646445":"2023-10-20T14:22:17.865415+0000","172.21.15.179:0/1840191723":"2023-10-20T14:22:17.865415+0000","172.21.15.179:0/4168916968":"2023-10-20T14:22:17.865415+0000","172.21.15.179:6833/1573806521":"2023-10-20T14:22:17.865415+0000","172.21.15.179:6832/1573806521":"2023-10-20T14:22:17.865415+0000","172.21.15.179:0/658875312":"2023-10-20T14:22:17.865415+0000","172.21.15.179:0/1799772260":"2023-10-20T14:22:17.865415+0000","172.21.15.179:0/3831670324":"2023-10-20T14:22:17.865415+0000","172.21.15.179:0/90219027":"2023-10-20T14:22:17.865415+0000","172.21.15.179:0/2739598080":"2023-10-20T14:22:17.865415+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T14:22:29.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.164+0000 7f7d91a44640 1 -- 172.21.15.57:0/4224348803 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f7d7806aff0 msgr2=0x7f7d7806d4b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T14:22:29.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.164+0000 7f7d91a44640 1 --2- 172.21.15.57:0/4224348803 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f7d7806aff0 0x7f7d7806d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:29.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.164+0000 7f7d91a44640 1 -- 172.21.15.57:0/4224348803 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7d8c157ca0 msgr2=0x7f7d8c149780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:29.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.164+0000 7f7d91a44640 1 --2- 172.21.15.57:0/4224348803 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7d8c157ca0 0x7f7d8c149780 secure :-1 s=READY pgs=2988 cs=0 l=1 rev1=1 crypto rx=0x7f7d8400c760 tx=0x7f7d8400cc20 comp rx=0 tx=0).stop 2023-10-19T14:22:29.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.164+0000 7f7d91a44640 1 -- 172.21.15.57:0/4224348803 shutdown_connections 2023-10-19T14:22:29.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.164+0000 7f7d91a44640 1 --2- 172.21.15.57:0/4224348803 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7f7d7806aff0 0x7f7d7806d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:29.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.164+0000 7f7d91a44640 1 --2- 172.21.15.57:0/4224348803 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7d8c157ca0 0x7f7d8c149780 unknown :-1 s=CLOSED pgs=2988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:29.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.164+0000 7f7d91a44640 1 --2- 172.21.15.57:0/4224348803 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7d8c0c9ed0 0x7f7d8c1564b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:29.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.164+0000 7f7d91a44640 1 --2- 172.21.15.57:0/4224348803 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7d8c065d90 0x7f7d8c15a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:29.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.164+0000 7f7d91a44640 1 -- 172.21.15.57:0/4224348803 >> 172.21.15.57:0/4224348803 conn(0x7f7d8c070980 msgr2=0x7f7d8c0b14a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:29.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.164+0000 7f7d91a44640 1 -- 172.21.15.57:0/4224348803 shutdown_connections 2023-10-19T14:22:29.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.164+0000 7f7d91a44640 1 -- 172.21.15.57:0/4224348803 wait complete. 2023-10-19T14:22:29.178 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-19T12:44:07.371186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 35, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-19T14:22:00.047912+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '262', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 36, 'pool_name': 'cephfs_data', 'create_time': '2023-10-19T14:22:00.739698+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '263', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-19T14:22:29.179 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-19T14:22:29.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.331+0000 7fc1a28f2640 1 -- 172.21.15.57:0/1977603922 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc184003b30 msgr2=0x7fc184003f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:29.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.331+0000 7fc1a28f2640 1 --2- 172.21.15.57:0/1977603922 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc184003b30 0x7fc184003f30 secure :-1 s=READY pgs=2990 cs=0 l=1 rev1=1 crypto rx=0x7fc18c009b50 tx=0x7fc1840057f0 comp rx=0 tx=0).stop 2023-10-19T14:22:29.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.331+0000 7fc1a0e8f640 1 -- 172.21.15.57:0/1977603922 reap_dead start 2023-10-19T14:22:29.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.332+0000 7fc1a28f2640 1 -- 172.21.15.57:0/1977603922 shutdown_connections 2023-10-19T14:22:29.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.332+0000 7fc1a28f2640 1 -- 172.21.15.57:0/1977603922 >> 172.21.15.57:0/1977603922 conn(0x7fc19c0cebc0 msgr2=0x7fc19c0cefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:29.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.332+0000 7fc1a28f2640 1 -- 172.21.15.57:0/1977603922 shutdown_connections 2023-10-19T14:22:29.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.332+0000 7fc1a28f2640 1 -- 172.21.15.57:0/1977603922 wait complete. 2023-10-19T14:22:29.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.333+0000 7fc1a28f2640 1 Processor -- start 2023-10-19T14:22:29.336 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.333+0000 7fc1a28f2640 1 -- start start 2023-10-19T14:22:29.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.334+0000 7fc1a28f2640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc184003b30 0x7fc19c156140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:29.338 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.334+0000 7fc1a28f2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc19c0cae80 0x7fc19c0cb300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:29.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.334+0000 7fc1a28f2640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc19c1587e0 0x7fc19c158c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:29.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.334+0000 7fc1a28f2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc19c0cb840 con 0x7fc19c0cae80 2023-10-19T14:22:29.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.334+0000 7fc1a28f2640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc19c156680 con 0x7fc19c1587e0 2023-10-19T14:22:29.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.334+0000 7fc1a28f2640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc19c159580 con 0x7fc184003b30 2023-10-19T14:22:29.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.334+0000 7fc19b7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc19c0cae80 0x7fc19c0cb300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:29.339 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.335+0000 7fc19b7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc19c0cae80 0x7fc19c0cb300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38854/0 (socket says 172.21.15.57:38854) 2023-10-19T14:22:29.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.335+0000 7fc19b7fe640 1 -- 172.21.15.57:0/216720058 learned_addr learned my addr 172.21.15.57:0/216720058 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:22:29.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.335+0000 7fc1a0e8f640 1 --2- 172.21.15.57:0/216720058 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc19c1587e0 0x7fc19c158c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:29.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.335+0000 7fc19bfff640 1 --2- 172.21.15.57:0/216720058 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc184003b30 0x7fc19c156140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:29.340 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.335+0000 7fc19b7fe640 1 -- 172.21.15.57:0/216720058 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc184003b30 msgr2=0x7fc19c156140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:29.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.335+0000 7fc19b7fe640 1 --2- 172.21.15.57:0/216720058 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc184003b30 0x7fc19c156140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:29.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.335+0000 7fc19b7fe640 1 -- 172.21.15.57:0/216720058 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc19c1587e0 msgr2=0x7fc19c158c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:29.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.335+0000 7fc19b7fe640 1 --2- 172.21.15.57:0/216720058 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc19c1587e0 0x7fc19c158c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:29.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.335+0000 7fc19b7fe640 1 -- 172.21.15.57:0/216720058 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc19c0c38e0 con 0x7fc19c0cae80 2023-10-19T14:22:29.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.335+0000 7fc19b7fe640 1 --2- 172.21.15.57:0/216720058 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc19c0cae80 0x7fc19c0cb300 secure :-1 s=READY pgs=2991 cs=0 l=1 rev1=1 crypto rx=0x7fc190006480 tx=0x7fc1900039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:29.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.335+0000 7fc198ff9640 1 -- 172.21.15.57:0/216720058 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc190011900 con 0x7fc19c0cae80 2023-10-19T14:22:29.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.335+0000 7fc198ff9640 1 -- 172.21.15.57:0/216720058 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fc190011a60 con 0x7fc19c0cae80 2023-10-19T14:22:29.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.335+0000 7fc1a28f2640 1 -- 172.21.15.57:0/216720058 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc19c0c3bc0 con 0x7fc19c0cae80 2023-10-19T14:22:29.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.335+0000 7fc198ff9640 1 -- 172.21.15.57:0/216720058 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc190011e70 con 0x7fc19c0cae80 2023-10-19T14:22:29.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.335+0000 7fc1a28f2640 1 -- 172.21.15.57:0/216720058 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc19c0c4180 con 0x7fc19c0cae80 2023-10-19T14:22:29.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.336+0000 7fc1a28f2640 1 -- 172.21.15.57:0/216720058 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc19c09ccf0 con 0x7fc19c0cae80 2023-10-19T14:22:29.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.337+0000 7fc198ff9640 1 -- 172.21.15.57:0/216720058 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 151) v1 ==== 89388+0+0 (secure 0 0 0) 0x7fc1900103c0 con 0x7fc19c0cae80 2023-10-19T14:22:29.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.337+0000 7fc198ff9640 1 --2- 172.21.15.57:0/216720058 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7fc18806aff0 0x7fc18806d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:29.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.337+0000 7fc19bfff640 1 -- 172.21.15.57:0/216720058 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7fc18806aff0 msgr2=0x7fc18806d4b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/2005622276 2023-10-19T14:22:29.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.337+0000 7fc198ff9640 1 -- 172.21.15.57:0/216720058 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(264..264 src has 1..264) v4 ==== 6487+0+0 (secure 0 0 0) 0x7fc19008ad10 con 0x7fc19c0cae80 2023-10-19T14:22:29.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.337+0000 7fc19bfff640 1 --2- 172.21.15.57:0/216720058 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7fc18806aff0 0x7fc18806d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T14:22:29.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.341+0000 7fc198ff9640 1 -- 172.21.15.57:0/216720058 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fc190055890 con 0x7fc19c0cae80 2023-10-19T14:22:29.370 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:22:29.367+0000 7f29ada11040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-19T14:22:29.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.523+0000 7fc1a28f2640 1 -- 172.21.15.57:0/216720058 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc19c04db50 con 0x7fc19c0cae80 2023-10-19T14:22:29.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.524+0000 7fc198ff9640 1 -- 172.21.15.57:0/216720058 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v264) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc190015020 con 0x7fc19c0cae80 2023-10-19T14:22:29.527 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-10-19T14:22:29.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.526+0000 7fc1a28f2640 1 -- 172.21.15.57:0/216720058 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7fc18806aff0 msgr2=0x7fc18806d4b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T14:22:29.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.526+0000 7fc1a28f2640 1 --2- 172.21.15.57:0/216720058 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7fc18806aff0 0x7fc18806d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:29.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.526+0000 7fc1a28f2640 1 -- 172.21.15.57:0/216720058 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc19c0cae80 msgr2=0x7fc19c0cb300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:29.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.526+0000 7fc1a28f2640 1 --2- 172.21.15.57:0/216720058 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc19c0cae80 0x7fc19c0cb300 secure :-1 s=READY pgs=2991 cs=0 l=1 rev1=1 crypto rx=0x7fc190006480 tx=0x7fc1900039c0 comp rx=0 tx=0).stop 2023-10-19T14:22:29.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.526+0000 7fc1a28f2640 1 -- 172.21.15.57:0/216720058 shutdown_connections 2023-10-19T14:22:29.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.526+0000 7fc1a28f2640 1 --2- 172.21.15.57:0/216720058 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7fc18806aff0 0x7fc18806d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:29.530 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.526+0000 7fc1a28f2640 1 --2- 172.21.15.57:0/216720058 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc19c1587e0 0x7fc19c158c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:29.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.526+0000 7fc1a28f2640 1 --2- 172.21.15.57:0/216720058 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc19c0cae80 0x7fc19c0cb300 unknown :-1 s=CLOSED pgs=2991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:29.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.526+0000 7fc1a28f2640 1 --2- 172.21.15.57:0/216720058 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fc184003b30 0x7fc19c156140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:29.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.526+0000 7fc1a28f2640 1 -- 172.21.15.57:0/216720058 >> 172.21.15.57:0/216720058 conn(0x7fc19c0cebc0 msgr2=0x7fc19c06ebd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:29.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.526+0000 7fc1a28f2640 1 -- 172.21.15.57:0/216720058 shutdown_connections 2023-10-19T14:22:29.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.526+0000 7fc1a28f2640 1 -- 172.21.15.57:0/216720058 wait complete. 2023-10-19T14:22:29.540 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-19T14:22:29.575 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:22:29.572+0000 7f0d028fa040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-19T14:22:29.632 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:22:29.630+0000 7f29ada11040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-19T14:22:29.654 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.651+0000 7fb13f8c9640 1 -- 172.21.15.57:0/1343668581 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb1380cef70 msgr2=0x7fb1380cc310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:29.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.651+0000 7fb13f8c9640 1 --2- 172.21.15.57:0/1343668581 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb1380cef70 0x7fb1380cc310 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7fb130005fd0 tx=0x7fb13002c0e0 comp rx=0 tx=0).stop 2023-10-19T14:22:29.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.651+0000 7fb13f8c9640 1 -- 172.21.15.57:0/1343668581 shutdown_connections 2023-10-19T14:22:29.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.651+0000 7fb13f8c9640 1 --2- 172.21.15.57:0/1343668581 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb1380cef70 0x7fb1380cc310 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:29.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.651+0000 7fb13f8c9640 1 --2- 172.21.15.57:0/1343668581 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb1380d1250 0x7fb1380cea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:29.655 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.651+0000 7fb13f8c9640 1 -- 172.21.15.57:0/1343668581 >> 172.21.15.57:0/1343668581 conn(0x7fb13808a7e0 msgr2=0x7fb13809a100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:29.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.651+0000 7fb13f8c9640 1 -- 172.21.15.57:0/1343668581 shutdown_connections 2023-10-19T14:22:29.656 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.651+0000 7fb13f8c9640 1 -- 172.21.15.57:0/1343668581 wait complete. 2023-10-19T14:22:29.658 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.652+0000 7fb13f8c9640 1 Processor -- start 2023-10-19T14:22:29.658 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.652+0000 7fb13f8c9640 1 -- start start 2023-10-19T14:22:29.658 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.653+0000 7fb13f8c9640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb1380cef70 0x7fb1380b23d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:29.658 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.653+0000 7fb13f8c9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb1380d1250 0x7fb1380c9f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:29.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.653+0000 7fb13f8c9640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb13815b070 0x7fb1380c65d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:29.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.653+0000 7fb13f8c9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1380ca920 con 0x7fb1380d1250 2023-10-19T14:22:29.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.653+0000 7fb13f8c9640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1380cac00 con 0x7fb13815b070 2023-10-19T14:22:29.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.653+0000 7fb13f8c9640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1380caee0 con 0x7fb1380cef70 2023-10-19T14:22:29.659 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.653+0000 7fb13ce64640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb1380d1250 0x7fb1380c9f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:29.660 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.653+0000 7fb13ce64640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb1380d1250 0x7fb1380c9f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38858/0 (socket says 172.21.15.57:38858) 2023-10-19T14:22:29.660 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.653+0000 7fb13ce64640 1 -- 172.21.15.57:0/115928773 learned_addr learned my addr 172.21.15.57:0/115928773 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:22:29.660 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.653+0000 7fb13de66640 1 --2- 172.21.15.57:0/115928773 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb13815b070 0x7fb1380c65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:29.660 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.653+0000 7fb13d665640 1 --2- 172.21.15.57:0/115928773 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb1380cef70 0x7fb1380b23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:29.660 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.653+0000 7fb13de66640 1 -- 172.21.15.57:0/115928773 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb1380cef70 msgr2=0x7fb1380b23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:29.661 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.653+0000 7fb13de66640 1 --2- 172.21.15.57:0/115928773 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb1380cef70 0x7fb1380b23d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:29.661 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.653+0000 7fb13de66640 1 -- 172.21.15.57:0/115928773 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb1380d1250 msgr2=0x7fb1380c9f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:29.661 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.653+0000 7fb13de66640 1 --2- 172.21.15.57:0/115928773 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb1380d1250 0x7fb1380c9f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:29.661 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.653+0000 7fb13de66640 1 -- 172.21.15.57:0/115928773 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb130009840 con 0x7fb13815b070 2023-10-19T14:22:29.661 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.653+0000 7fb13ce64640 1 --2- 172.21.15.57:0/115928773 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb1380d1250 0x7fb1380c9f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-19T14:22:29.661 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.654+0000 7fb13de66640 1 --2- 172.21.15.57:0/115928773 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb13815b070 0x7fb1380c65d0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7fb130009000 tx=0x7fb130009330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:29.662 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.654+0000 7fb12dffb640 1 -- 172.21.15.57:0/115928773 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb130004600 con 0x7fb13815b070 2023-10-19T14:22:29.662 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.654+0000 7fb12dffb640 1 -- 172.21.15.57:0/115928773 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fb130004850 con 0x7fb13815b070 2023-10-19T14:22:29.662 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.654+0000 7fb13f8c9640 1 -- 172.21.15.57:0/115928773 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1380cb1c0 con 0x7fb13815b070 2023-10-19T14:22:29.662 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.654+0000 7fb12dffb640 1 -- 172.21.15.57:0/115928773 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1300347e0 con 0x7fb13815b070 2023-10-19T14:22:29.662 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.654+0000 7fb13f8c9640 1 -- 172.21.15.57:0/115928773 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1380b2910 con 0x7fb13815b070 2023-10-19T14:22:29.663 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.655+0000 7fb13f8c9640 1 -- 172.21.15.57:0/115928773 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1381480b0 con 0x7fb13815b070 2023-10-19T14:22:29.664 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.656+0000 7fb12dffb640 1 -- 172.21.15.57:0/115928773 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 151) v1 ==== 89388+0+0 (secure 0 0 0) 0x7fb130034940 con 0x7fb13815b070 2023-10-19T14:22:29.664 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.656+0000 7fb12dffb640 1 --2- 172.21.15.57:0/115928773 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7fb11c06aff0 0x7fb11c06d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:29.664 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.656+0000 7fb12dffb640 1 -- 172.21.15.57:0/115928773 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(264..264 src has 1..264) v4 ==== 6487+0+0 (secure 0 0 0) 0x7fb1300acbd0 con 0x7fb13815b070 2023-10-19T14:22:29.664 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.656+0000 7fb13d665640 1 -- 172.21.15.57:0/115928773 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7fb11c06aff0 msgr2=0x7fb11c06d4b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.179:6832/2005622276 2023-10-19T14:22:29.665 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.656+0000 7fb13d665640 1 --2- 172.21.15.57:0/115928773 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7fb11c06aff0 0x7fb11c06d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-19T14:22:29.665 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.660+0000 7fb12dffb640 1 -- 172.21.15.57:0/115928773 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fb1381480b0 con 0x7fb13815b070 2023-10-19T14:22:29.681 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:22:29.679+0000 7f29ada11040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-19T14:22:29.781 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:22:29.778+0000 7f29ada11040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-19T14:22:29.802 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.800+0000 7fb12dffb640 1 -- 172.21.15.57:0/115928773 <== mon.1 v2:172.21.15.179:3300/0 7 ==== mgrmap(e 152) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fb130077ad0 con 0x7fb13815b070 2023-10-19T14:22:29.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.800+0000 7fb12dffb640 1 -- 172.21.15.57:0/115928773 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7fb11c06aff0 msgr2=0x7fb11c06d4b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-19T14:22:29.803 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.800+0000 7fb12dffb640 1 --2- 172.21.15.57:0/115928773 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7fb11c06aff0 0x7fb11c06d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:29.843 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.841+0000 7fb13f8c9640 1 -- 172.21.15.57:0/115928773 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb1381480b0 con 0x7fb13815b070 2023-10-19T14:22:29.844 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.841+0000 7fb12dffb640 1 -- 172.21.15.57:0/115928773 <== mon.1 v2:172.21.15.179:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v265) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb1300ac020 con 0x7fb13815b070 2023-10-19T14:22:29.844 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2023-10-19T14:22:29.846 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.843+0000 7fb13f8c9640 1 -- 172.21.15.57:0/115928773 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb13815b070 msgr2=0x7fb1380c65d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:29.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.843+0000 7fb13f8c9640 1 --2- 172.21.15.57:0/115928773 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb13815b070 0x7fb1380c65d0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7fb130009000 tx=0x7fb130009330 comp rx=0 tx=0).stop 2023-10-19T14:22:29.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.844+0000 7fb13f8c9640 1 -- 172.21.15.57:0/115928773 shutdown_connections 2023-10-19T14:22:29.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.844+0000 7fb13f8c9640 1 --2- 172.21.15.57:0/115928773 >> [v2:172.21.15.179:6832/2005622276,v1:172.21.15.179:6833/2005622276] conn(0x7fb11c06aff0 0x7fb11c06d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:29.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.844+0000 7fb13f8c9640 1 --2- 172.21.15.57:0/115928773 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb13815b070 0x7fb1380c65d0 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:29.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.844+0000 7fb13f8c9640 1 --2- 172.21.15.57:0/115928773 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb1380d1250 0x7fb1380c9f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:29.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.844+0000 7fb13f8c9640 1 --2- 172.21.15.57:0/115928773 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb1380cef70 0x7fb1380b23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:29.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.844+0000 7fb13f8c9640 1 -- 172.21.15.57:0/115928773 >> 172.21.15.57:0/115928773 conn(0x7fb13808a7e0 msgr2=0x7fb1380af710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:29.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.844+0000 7fb13f8c9640 1 -- 172.21.15.57:0/115928773 shutdown_connections 2023-10-19T14:22:29.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.844+0000 7fb13f8c9640 1 -- 172.21.15.57:0/115928773 wait complete. 2023-10-19T14:22:29.858 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-19T14:22:29.859 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:22:29.857+0000 7f2932a4f640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:22:29.859 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:22:29.857+0000 7f2932a4f640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:22:29.859 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:22:29.857+0000 7f2932a4f640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:22:29.860 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:22:29.857+0000 7f2932a4f640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:22:29.860 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:22:29.857+0000 7f2932a4f640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:22:29.860 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:22:29.857+0000 7f292fa49640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:22:29.860 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:22:29.857+0000 7f292fa49640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:22:29.860 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:22:29.857+0000 7f292fa49640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:22:29.861 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:22:29.857+0000 7f292fa49640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:22:29.861 INFO:tasks.ceph.mgr.x.smithi179.stderr:2023-10-19T14:22:29.857+0000 7f292fa49640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-19T14:22:29.862 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:22:29.858+0000 7f0d028fa040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-19T14:22:29.914 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:22:29.911+0000 7f0d028fa040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-19T14:22:29.974 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.971+0000 7f29e5a9b640 1 -- 172.21.15.57:0/2598470877 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f29e015b070 msgr2=0x7f29e00d0b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:29.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.971+0000 7f29e5a9b640 1 --2- 172.21.15.57:0/2598470877 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f29e015b070 0x7f29e00d0b40 secure :-1 s=READY pgs=2993 cs=0 l=1 rev1=1 crypto rx=0x7f29d0005fd0 tx=0x7f29d002be10 comp rx=0 tx=0).stop 2023-10-19T14:22:29.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.971+0000 7f29e5a9b640 1 -- 172.21.15.57:0/2598470877 shutdown_connections 2023-10-19T14:22:29.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.971+0000 7f29e5a9b640 1 --2- 172.21.15.57:0/2598470877 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f29e00cf140 0x7f29e00cc310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:29.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.971+0000 7f29e5a9b640 1 --2- 172.21.15.57:0/2598470877 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f29e015b070 0x7f29e00d0b40 unknown :-1 s=CLOSED pgs=2993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:29.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.971+0000 7f29e5a9b640 1 -- 172.21.15.57:0/2598470877 >> 172.21.15.57:0/2598470877 conn(0x7f29e008a7e0 msgr2=0x7f29e009a100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:29.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.971+0000 7f29e5a9b640 1 -- 172.21.15.57:0/2598470877 shutdown_connections 2023-10-19T14:22:29.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.971+0000 7f29e5a9b640 1 -- 172.21.15.57:0/2598470877 wait complete. 2023-10-19T14:22:29.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.972+0000 7f29e5a9b640 1 Processor -- start 2023-10-19T14:22:29.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.973+0000 7f29e5a9b640 1 -- start start 2023-10-19T14:22:29.979 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.973+0000 7f29e5a9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f29e00cf140 0x7f29e00b2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:29.979 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.973+0000 7f29e5a9b640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f29e00d1080 0x7f29e015de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:29.979 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.973+0000 7f29e5a9b640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f29e015b070 0x7f29e00b39f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:29.979 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.973+0000 7f29e5a9b640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29e015da20 con 0x7f29e00cf140 2023-10-19T14:22:29.979 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.973+0000 7f29e5a9b640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29e01f1bc0 con 0x7f29e015b070 2023-10-19T14:22:29.979 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.973+0000 7f29e5a9b640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29e01f59b0 con 0x7f29e00d1080 2023-10-19T14:22:29.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.973+0000 7f29df7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f29e00cf140 0x7f29e00b2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:29.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.973+0000 7f29df7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f29e00cf140 0x7f29e00b2aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38864/0 (socket says 172.21.15.57:38864) 2023-10-19T14:22:29.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.973+0000 7f29df7fe640 1 -- 172.21.15.57:0/1447280145 learned_addr learned my addr 172.21.15.57:0/1447280145 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:22:29.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.974+0000 7f29dffff640 1 --2- 172.21.15.57:0/1447280145 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f29e015b070 0x7f29e00b39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:29.980 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.974+0000 7f29d6ffd640 1 --2- 172.21.15.57:0/1447280145 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f29e00d1080 0x7f29e015de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:29.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.974+0000 7f29df7fe640 1 -- 172.21.15.57:0/1447280145 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f29e00d1080 msgr2=0x7f29e015de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:29.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.974+0000 7f29df7fe640 1 --2- 172.21.15.57:0/1447280145 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f29e00d1080 0x7f29e015de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:29.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.974+0000 7f29df7fe640 1 -- 172.21.15.57:0/1447280145 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f29e015b070 msgr2=0x7f29e00b39f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:29.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.974+0000 7f29df7fe640 1 --2- 172.21.15.57:0/1447280145 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f29e015b070 0x7f29e00b39f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:29.981 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.974+0000 7f29df7fe640 1 -- 172.21.15.57:0/1447280145 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29d0009560 con 0x7f29e00cf140 2023-10-19T14:22:29.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.974+0000 7f29df7fe640 1 --2- 172.21.15.57:0/1447280145 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f29e00cf140 0x7f29e00b2aa0 secure :-1 s=READY pgs=2994 cs=0 l=1 rev1=1 crypto rx=0x7f29d0009850 tx=0x7f29d002dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:29.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.974+0000 7f29dcff9640 1 -- 172.21.15.57:0/1447280145 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29d00035f0 con 0x7f29e00cf140 2023-10-19T14:22:29.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.974+0000 7f29dcff9640 1 -- 172.21.15.57:0/1447280145 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f29d0003840 con 0x7f29e00cf140 2023-10-19T14:22:29.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.974+0000 7f29e5a9b640 1 -- 172.21.15.57:0/1447280145 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29e01f5c90 con 0x7f29e00cf140 2023-10-19T14:22:29.982 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.974+0000 7f29dcff9640 1 -- 172.21.15.57:0/1447280145 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29d0003bf0 con 0x7f29e00cf140 2023-10-19T14:22:29.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.974+0000 7f29e5a9b640 1 -- 172.21.15.57:0/1447280145 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29e015d640 con 0x7f29e00cf140 2023-10-19T14:22:29.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.975+0000 7f29e5a9b640 1 -- 172.21.15.57:0/1447280145 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29e015db60 con 0x7f29e00cf140 2023-10-19T14:22:29.983 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.976+0000 7f29dcff9640 1 -- 172.21.15.57:0/1447280145 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 152) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f29d0003d50 con 0x7f29e00cf140 2023-10-19T14:22:29.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.977+0000 7f29dcff9640 1 -- 172.21.15.57:0/1447280145 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(265..265 src has 1..265) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f29d00ac210 con 0x7f29e00cf140 2023-10-19T14:22:29.984 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:29.981+0000 7f29dcff9640 1 -- 172.21.15.57:0/1447280145 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f29e015db60 con 0x7f29e00cf140 2023-10-19T14:22:30.022 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-10-19T14:22:30.019+0000 7f0d028fa040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-19T14:22:30.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.155+0000 7f29e5a9b640 1 -- 172.21.15.57:0/1447280145 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f29e015db60 con 0x7f29e00cf140 2023-10-19T14:22:30.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.156+0000 7f29dcff9640 1 -- 172.21.15.57:0/1447280145 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v265) v1 ==== 100+0+11 (secure 0 0 0) 0x7f29d0043020 con 0x7f29e00cf140 2023-10-19T14:22:30.159 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 64 2023-10-19T14:22:30.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.158+0000 7f29e5a9b640 1 -- 172.21.15.57:0/1447280145 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f29e00cf140 msgr2=0x7f29e00b2aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:30.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.158+0000 7f29e5a9b640 1 --2- 172.21.15.57:0/1447280145 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f29e00cf140 0x7f29e00b2aa0 secure :-1 s=READY pgs=2994 cs=0 l=1 rev1=1 crypto rx=0x7f29d0009850 tx=0x7f29d002dae0 comp rx=0 tx=0).stop 2023-10-19T14:22:30.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.158+0000 7f29e5a9b640 1 -- 172.21.15.57:0/1447280145 shutdown_connections 2023-10-19T14:22:30.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.158+0000 7f29e5a9b640 1 --2- 172.21.15.57:0/1447280145 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f29e015b070 0x7f29e00b39f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:30.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.158+0000 7f29e5a9b640 1 --2- 172.21.15.57:0/1447280145 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f29e00d1080 0x7f29e015de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:30.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.158+0000 7f29e5a9b640 1 --2- 172.21.15.57:0/1447280145 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f29e00cf140 0x7f29e00b2aa0 unknown :-1 s=CLOSED pgs=2994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:30.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.158+0000 7f29e5a9b640 1 -- 172.21.15.57:0/1447280145 >> 172.21.15.57:0/1447280145 conn(0x7f29e008a7e0 msgr2=0x7f29e01587d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:30.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.158+0000 7f29e5a9b640 1 -- 172.21.15.57:0/1447280145 shutdown_connections 2023-10-19T14:22:30.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.158+0000 7f29e5a9b640 1 -- 172.21.15.57:0/1447280145 wait complete. 2023-10-19T14:22:30.172 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T14:22:30.282 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.278+0000 7fdc5f705640 1 -- 172.21.15.57:0/957245408 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdc5806f2a0 msgr2=0x7fdc580b0c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:30.282 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.279+0000 7fdc5f705640 1 --2- 172.21.15.57:0/957245408 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdc5806f2a0 0x7fdc580b0c10 secure :-1 s=READY pgs=2996 cs=0 l=1 rev1=1 crypto rx=0x7fdc48005fd0 tx=0x7fdc4802be10 comp rx=0 tx=0).stop 2023-10-19T14:22:30.282 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.279+0000 7fdc5f705640 1 -- 172.21.15.57:0/957245408 shutdown_connections 2023-10-19T14:22:30.282 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.279+0000 7fdc5f705640 1 --2- 172.21.15.57:0/957245408 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdc5806f2a0 0x7fdc580b0c10 unknown :-1 s=CLOSED pgs=2996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:30.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.279+0000 7fdc5f705640 1 --2- 172.21.15.57:0/957245408 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdc581417e0 0x7fdc58070a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:30.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.279+0000 7fdc5f705640 1 -- 172.21.15.57:0/957245408 >> 172.21.15.57:0/957245408 conn(0x7fdc58069810 msgr2=0x7fdc58069c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:30.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.279+0000 7fdc5f705640 1 -- 172.21.15.57:0/957245408 shutdown_connections 2023-10-19T14:22:30.283 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.279+0000 7fdc5f705640 1 -- 172.21.15.57:0/957245408 wait complete. 2023-10-19T14:22:30.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.280+0000 7fdc5f705640 1 Processor -- start 2023-10-19T14:22:30.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.280+0000 7fdc5f705640 1 -- start start 2023-10-19T14:22:30.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.280+0000 7fdc5f705640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdc5806f2a0 0x7fdc5813dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:30.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.280+0000 7fdc5f705640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdc580b1820 0x7fdc581576e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:30.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.281+0000 7fdc5f705640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdc581417e0 0x7fdc5813aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:30.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.281+0000 7fdc5f705640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc581f18e0 con 0x7fdc580b1820 2023-10-19T14:22:30.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.281+0000 7fdc5f705640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc581f1570 con 0x7fdc581417e0 2023-10-19T14:22:30.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.281+0000 7fdc5f705640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc58157c20 con 0x7fdc5806f2a0 2023-10-19T14:22:30.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.281+0000 7fdc5cca0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdc580b1820 0x7fdc581576e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:30.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.281+0000 7fdc5cca0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdc580b1820 0x7fdc581576e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38882/0 (socket says 172.21.15.57:38882) 2023-10-19T14:22:30.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.281+0000 7fdc5cca0640 1 -- 172.21.15.57:0/2897044283 learned_addr learned my addr 172.21.15.57:0/2897044283 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:22:30.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.281+0000 7fdc5d4a1640 1 --2- 172.21.15.57:0/2897044283 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdc5806f2a0 0x7fdc5813dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:30.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.281+0000 7fdc5dca2640 1 --2- 172.21.15.57:0/2897044283 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdc581417e0 0x7fdc5813aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:30.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.281+0000 7fdc5cca0640 1 -- 172.21.15.57:0/2897044283 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdc5806f2a0 msgr2=0x7fdc5813dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:30.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.281+0000 7fdc5cca0640 1 --2- 172.21.15.57:0/2897044283 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdc5806f2a0 0x7fdc5813dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:30.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.281+0000 7fdc5cca0640 1 -- 172.21.15.57:0/2897044283 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdc581417e0 msgr2=0x7fdc5813aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:30.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.281+0000 7fdc5cca0640 1 --2- 172.21.15.57:0/2897044283 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdc581417e0 0x7fdc5813aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:30.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.281+0000 7fdc5cca0640 1 -- 172.21.15.57:0/2897044283 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc48009560 con 0x7fdc580b1820 2023-10-19T14:22:30.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.281+0000 7fdc5dca2640 1 --2- 172.21.15.57:0/2897044283 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdc581417e0 0x7fdc5813aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:22:30.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.281+0000 7fdc5cca0640 1 --2- 172.21.15.57:0/2897044283 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdc580b1820 0x7fdc581576e0 secure :-1 s=READY pgs=2997 cs=0 l=1 rev1=1 crypto rx=0x7fdc480099f0 tx=0x7fdc4802dbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:30.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.281+0000 7fdc4dffb640 1 -- 172.21.15.57:0/2897044283 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc48003800 con 0x7fdc580b1820 2023-10-19T14:22:30.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.281+0000 7fdc4dffb640 1 -- 172.21.15.57:0/2897044283 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fdc48003a50 con 0x7fdc580b1820 2023-10-19T14:22:30.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.281+0000 7fdc5f705640 1 -- 172.21.15.57:0/2897044283 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc580b21c0 con 0x7fdc580b1820 2023-10-19T14:22:30.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.281+0000 7fdc4dffb640 1 -- 172.21.15.57:0/2897044283 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc48003e60 con 0x7fdc580b1820 2023-10-19T14:22:30.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.282+0000 7fdc5f705640 1 -- 172.21.15.57:0/2897044283 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc5813e650 con 0x7fdc580b1820 2023-10-19T14:22:30.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.283+0000 7fdc5f705640 1 -- 172.21.15.57:0/2897044283 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc581f1a20 con 0x7fdc580b1820 2023-10-19T14:22:30.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.283+0000 7fdc4dffb640 1 -- 172.21.15.57:0/2897044283 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 152) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fdc4803e450 con 0x7fdc580b1820 2023-10-19T14:22:30.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.284+0000 7fdc4dffb640 1 -- 172.21.15.57:0/2897044283 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(265..265 src has 1..265) v4 ==== 6874+0+0 (secure 0 0 0) 0x7fdc480abf10 con 0x7fdc580b1820 2023-10-19T14:22:30.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.288+0000 7fdc4dffb640 1 -- 172.21.15.57:0/2897044283 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fdc581f1a20 con 0x7fdc580b1820 2023-10-19T14:22:30.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.502+0000 7fdc5f705640 1 -- 172.21.15.57:0/2897044283 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdc581f1a20 con 0x7fdc580b1820 2023-10-19T14:22:30.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.502+0000 7fdc4dffb640 1 -- 172.21.15.57:0/2897044283 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+4935 (secure 0 0 0) 0x7fdc581f1a20 con 0x7fdc580b1820 2023-10-19T14:22:30.506 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:22:30.506 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":116,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10280,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/1738328496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":1738328496},{"type":"v1","addr":"172.21.15.179:6837","nonce":1738328496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":113},{"gid":13344,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/3191801579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":3191801579},{"type":"v1","addr":"172.21.15.57:6834","nonce":3191801579}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":113},{"gid":13347,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/3876822488","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":3876822488},{"type":"v1","addr":"172.21.15.57:6835","nonce":3876822488}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T14:22:01.763128+0000","modified":"2023-10-19T14:22:03.758375+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":13359},"failed":[],"damaged":[],"stopped":[],"info":{"gid_13359":{"gid":13359,"name":"d","rank":0,"incarnation":114,"state":"up:active","state_seq":3,"addr":"172.21.15.179:6836/4133411706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":4133411706},{"type":"v1","addr":"172.21.15.179:6836","nonce":4133411706}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[36],"metadata_pool":35,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":17}]} 2023-10-19T14:22:30.506 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 116 2023-10-19T14:22:30.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.504+0000 7fdc5f705640 1 -- 172.21.15.57:0/2897044283 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdc580b1820 msgr2=0x7fdc581576e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:30.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.504+0000 7fdc5f705640 1 --2- 172.21.15.57:0/2897044283 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdc580b1820 0x7fdc581576e0 secure :-1 s=READY pgs=2997 cs=0 l=1 rev1=1 crypto rx=0x7fdc480099f0 tx=0x7fdc4802dbb0 comp rx=0 tx=0).stop 2023-10-19T14:22:30.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.504+0000 7fdc5f705640 1 -- 172.21.15.57:0/2897044283 shutdown_connections 2023-10-19T14:22:30.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.504+0000 7fdc5f705640 1 --2- 172.21.15.57:0/2897044283 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdc581417e0 0x7fdc5813aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:30.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.504+0000 7fdc5f705640 1 --2- 172.21.15.57:0/2897044283 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdc580b1820 0x7fdc581576e0 unknown :-1 s=CLOSED pgs=2997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:30.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.505+0000 7fdc5f705640 1 --2- 172.21.15.57:0/2897044283 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdc5806f2a0 0x7fdc5813dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:30.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.505+0000 7fdc5f705640 1 -- 172.21.15.57:0/2897044283 >> 172.21.15.57:0/2897044283 conn(0x7fdc58069810 msgr2=0x7fdc58069bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:30.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.505+0000 7fdc5f705640 1 -- 172.21.15.57:0/2897044283 shutdown_connections 2023-10-19T14:22:30.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.505+0000 7fdc5f705640 1 -- 172.21.15.57:0/2897044283 wait complete. 2023-10-19T14:22:30.518 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T14:22:30.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.624+0000 7f41edc40640 1 -- 172.21.15.57:0/1222371353 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f41e81417e0 msgr2=0x7f41e8070a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:30.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.624+0000 7f41edc40640 1 --2- 172.21.15.57:0/1222371353 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f41e81417e0 0x7f41e8070a00 secure :-1 s=READY pgs=2998 cs=0 l=1 rev1=1 crypto rx=0x7f41d8005fd0 tx=0x7f41d802be10 comp rx=0 tx=0).stop 2023-10-19T14:22:30.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.624+0000 7f41edc40640 1 -- 172.21.15.57:0/1222371353 shutdown_connections 2023-10-19T14:22:30.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.624+0000 7f41edc40640 1 --2- 172.21.15.57:0/1222371353 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f41e80b1f30 0x7f41e80b0c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:30.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.624+0000 7f41edc40640 1 --2- 172.21.15.57:0/1222371353 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f41e81417e0 0x7f41e8070a00 unknown :-1 s=CLOSED pgs=2998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:30.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.624+0000 7f41edc40640 1 -- 172.21.15.57:0/1222371353 >> 172.21.15.57:0/1222371353 conn(0x7f41e8069810 msgr2=0x7f41e8069c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:30.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.624+0000 7f41edc40640 1 -- 172.21.15.57:0/1222371353 shutdown_connections 2023-10-19T14:22:30.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.624+0000 7f41edc40640 1 -- 172.21.15.57:0/1222371353 wait complete. 2023-10-19T14:22:30.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.625+0000 7f41edc40640 1 Processor -- start 2023-10-19T14:22:30.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.625+0000 7f41edc40640 1 -- start start 2023-10-19T14:22:30.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.626+0000 7f41edc40640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f41e806f2a0 0x7f41e8157340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:30.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.626+0000 7f41edc40640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f41e80b1f30 0x7f41e80b11f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:30.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.626+0000 7f41edc40640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f41e81417e0 0x7f41e80b1730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:30.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.626+0000 7f41edc40640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41e812e070 con 0x7f41e806f2a0 2023-10-19T14:22:30.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.626+0000 7f41edc40640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41e812e350 con 0x7f41e80b1f30 2023-10-19T14:22:30.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.626+0000 7f41edc40640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41e812e630 con 0x7f41e81417e0 2023-10-19T14:22:30.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.626+0000 7f41e77fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f41e806f2a0 0x7f41e8157340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:30.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.626+0000 7f41e77fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f41e806f2a0 0x7f41e8157340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38902/0 (socket says 172.21.15.57:38902) 2023-10-19T14:22:30.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.626+0000 7f41e77fe640 1 -- 172.21.15.57:0/1008223121 learned_addr learned my addr 172.21.15.57:0/1008223121 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:22:30.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.626+0000 7f41e7fff640 1 --2- 172.21.15.57:0/1008223121 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f41e81417e0 0x7f41e80b1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:30.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.626+0000 7f41e6ffd640 1 --2- 172.21.15.57:0/1008223121 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f41e80b1f30 0x7f41e80b11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:30.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.626+0000 7f41e77fe640 1 -- 172.21.15.57:0/1008223121 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f41e81417e0 msgr2=0x7f41e80b1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:30.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.626+0000 7f41e77fe640 1 --2- 172.21.15.57:0/1008223121 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f41e81417e0 0x7f41e80b1730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:30.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.626+0000 7f41e77fe640 1 -- 172.21.15.57:0/1008223121 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f41e80b1f30 msgr2=0x7f41e80b11f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:30.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.626+0000 7f41e77fe640 1 --2- 172.21.15.57:0/1008223121 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f41e80b1f30 0x7f41e80b11f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:30.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.626+0000 7f41e77fe640 1 -- 172.21.15.57:0/1008223121 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41d8009560 con 0x7f41e806f2a0 2023-10-19T14:22:30.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.626+0000 7f41e7fff640 1 --2- 172.21.15.57:0/1008223121 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f41e81417e0 0x7f41e80b1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:22:30.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.626+0000 7f41e6ffd640 1 --2- 172.21.15.57:0/1008223121 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f41e80b1f30 0x7f41e80b11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:22:30.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.627+0000 7f41e77fe640 1 --2- 172.21.15.57:0/1008223121 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f41e806f2a0 0x7f41e8157340 secure :-1 s=READY pgs=2999 cs=0 l=1 rev1=1 crypto rx=0x7f41d8009850 tx=0x7f41d802dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:30.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.627+0000 7f41cffff640 1 -- 172.21.15.57:0/1008223121 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41d80035f0 con 0x7f41e806f2a0 2023-10-19T14:22:30.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.627+0000 7f41cffff640 1 -- 172.21.15.57:0/1008223121 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f41d8003840 con 0x7f41e806f2a0 2023-10-19T14:22:30.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.627+0000 7f41edc40640 1 -- 172.21.15.57:0/1008223121 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41e812e910 con 0x7f41e806f2a0 2023-10-19T14:22:30.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.627+0000 7f41cffff640 1 -- 172.21.15.57:0/1008223121 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41d8003bf0 con 0x7f41e806f2a0 2023-10-19T14:22:30.636 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.627+0000 7f41edc40640 1 -- 172.21.15.57:0/1008223121 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41e8157880 con 0x7f41e806f2a0 2023-10-19T14:22:30.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.628+0000 7f41edc40640 1 -- 172.21.15.57:0/1008223121 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41e8069100 con 0x7f41e806f2a0 2023-10-19T14:22:30.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.628+0000 7f41cffff640 1 -- 172.21.15.57:0/1008223121 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 152) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f41d8003d50 con 0x7f41e806f2a0 2023-10-19T14:22:30.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.629+0000 7f41cffff640 1 -- 172.21.15.57:0/1008223121 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(265..265 src has 1..265) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f41d80ac370 con 0x7f41e806f2a0 2023-10-19T14:22:30.637 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.633+0000 7f41cffff640 1 -- 172.21.15.57:0/1008223121 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f41d8077500 con 0x7f41e806f2a0 2023-10-19T14:22:30.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.802+0000 7f41cffff640 1 -- 172.21.15.57:0/1008223121 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mgrmap(e 153) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f41d8076fb0 con 0x7f41e806f2a0 2023-10-19T14:22:30.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.802+0000 7f41cffff640 1 --2- 172.21.15.57:0/1008223121 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f41d4070c40 0x7f41d4073030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:30.805 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.803+0000 7f41e6ffd640 1 --2- 172.21.15.57:0/1008223121 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f41d4070c40 0x7f41d4073030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:30.806 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.804+0000 7f41e6ffd640 1 --2- 172.21.15.57:0/1008223121 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f41d4070c40 0x7f41d4073030 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f41e81310f0 tx=0x7f41dc009040 comp rx=0 tx=0).ready entity=mgr.10520 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:30.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.812+0000 7f41edc40640 1 -- 172.21.15.57:0/1008223121 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f41e8070bc0 con 0x7f41e806f2a0 2023-10-19T14:22:30.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.813+0000 7f41cffff640 1 -- 172.21.15.57:0/1008223121 <== mon.0 v2:172.21.15.57:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v265) v1 ==== 74+0+17980 (secure 0 0 0) 0x7f41d808f0e0 con 0x7f41e806f2a0 2023-10-19T14:22:30.817 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:22:30.817 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":265,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T14:22:29.793566+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":36,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_metadata","create_time":"2023-10-19T14:22:00.047912+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"262","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_data","create_time":"2023-10-19T14:22:00.739698+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"263","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.179:0/2632603637":"2023-10-20T14:22:29.793530+0000","172.21.15.179:0/2998005959":"2023-10-20T14:22:29.793530+0000","172.21.15.179:0/3483041475":"2023-10-20T14:22:29.793530+0000","172.21.15.179:0/3647691630":"2023-10-20T14:22:29.793530+0000","172.21.15.179:6833/2005622276":"2023-10-20T14:22:29.793530+0000","172.21.15.179:6832/2005622276":"2023-10-20T14:22:29.793530+0000","172.21.15.179:0/1983646445":"2023-10-20T14:22:17.865415+0000","172.21.15.179:0/1840191723":"2023-10-20T14:22:17.865415+0000","172.21.15.179:0/2852372821":"2023-10-20T14:22:29.793530+0000","172.21.15.179:0/4168916968":"2023-10-20T14:22:17.865415+0000","172.21.15.179:6833/1573806521":"2023-10-20T14:22:17.865415+0000","172.21.15.179:0/3886182326":"2023-10-20T14:22:29.793530+0000","172.21.15.179:6832/1573806521":"2023-10-20T14:22:17.865415+0000","172.21.15.179:0/658875312":"2023-10-20T14:22:17.865415+0000","172.21.15.179:0/1678121137":"2023-10-20T14:22:29.793530+0000","172.21.15.179:0/1799772260":"2023-10-20T14:22:17.865415+0000","172.21.15.179:0/3831670324":"2023-10-20T14:22:17.865415+0000","172.21.15.179:0/90219027":"2023-10-20T14:22:17.865415+0000","172.21.15.179:0/2739598080":"2023-10-20T14:22:17.865415+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T14:22:30.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.816+0000 7f41edc40640 1 -- 172.21.15.57:0/1008223121 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f41d4070c40 msgr2=0x7f41d4073030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:30.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.816+0000 7f41edc40640 1 --2- 172.21.15.57:0/1008223121 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f41d4070c40 0x7f41d4073030 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f41e81310f0 tx=0x7f41dc009040 comp rx=0 tx=0).stop 2023-10-19T14:22:30.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.816+0000 7f41edc40640 1 -- 172.21.15.57:0/1008223121 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f41e806f2a0 msgr2=0x7f41e8157340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:30.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.816+0000 7f41edc40640 1 --2- 172.21.15.57:0/1008223121 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f41e806f2a0 0x7f41e8157340 secure :-1 s=READY pgs=2999 cs=0 l=1 rev1=1 crypto rx=0x7f41d8009850 tx=0x7f41d802dae0 comp rx=0 tx=0).stop 2023-10-19T14:22:30.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.816+0000 7f41edc40640 1 -- 172.21.15.57:0/1008223121 shutdown_connections 2023-10-19T14:22:30.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.816+0000 7f41edc40640 1 --2- 172.21.15.57:0/1008223121 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f41d4070c40 0x7f41d4073030 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:30.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.816+0000 7f41edc40640 1 --2- 172.21.15.57:0/1008223121 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f41e81417e0 0x7f41e80b1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:30.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.816+0000 7f41edc40640 1 --2- 172.21.15.57:0/1008223121 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f41e80b1f30 0x7f41e80b11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:30.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.816+0000 7f41edc40640 1 --2- 172.21.15.57:0/1008223121 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f41e806f2a0 0x7f41e8157340 unknown :-1 s=CLOSED pgs=2999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:30.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.816+0000 7f41edc40640 1 -- 172.21.15.57:0/1008223121 >> 172.21.15.57:0/1008223121 conn(0x7f41e8069810 msgr2=0x7f41e80cb940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:30.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.816+0000 7f41edc40640 1 -- 172.21.15.57:0/1008223121 shutdown_connections 2023-10-19T14:22:30.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.816+0000 7f41edc40640 1 -- 172.21.15.57:0/1008223121 wait complete. 2023-10-19T14:22:30.829 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-19T14:22:30.830 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T14:22:30.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.936+0000 7fe91a705640 1 -- 172.21.15.57:0/1076241964 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe9140b1940 msgr2=0x7fe914070a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:30.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.936+0000 7fe91a705640 1 --2- 172.21.15.57:0/1076241964 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe9140b1940 0x7fe914070a00 secure :-1 s=READY pgs=3000 cs=0 l=1 rev1=1 crypto rx=0x7fe904005fd0 tx=0x7fe90402be10 comp rx=0 tx=0).stop 2023-10-19T14:22:30.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.936+0000 7fe91a705640 1 -- 172.21.15.57:0/1076241964 shutdown_connections 2023-10-19T14:22:30.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.936+0000 7fe91a705640 1 --2- 172.21.15.57:0/1076241964 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe9140b0d20 0x7fe914070f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:30.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.936+0000 7fe91a705640 1 --2- 172.21.15.57:0/1076241964 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe9140b1940 0x7fe914070a00 unknown :-1 s=CLOSED pgs=3000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:30.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.936+0000 7fe91a705640 1 -- 172.21.15.57:0/1076241964 >> 172.21.15.57:0/1076241964 conn(0x7fe914069810 msgr2=0x7fe914069c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:30.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.936+0000 7fe91a705640 1 -- 172.21.15.57:0/1076241964 shutdown_connections 2023-10-19T14:22:30.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.936+0000 7fe91a705640 1 -- 172.21.15.57:0/1076241964 wait complete. 2023-10-19T14:22:30.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.937+0000 7fe91a705640 1 Processor -- start 2023-10-19T14:22:30.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.937+0000 7fe91a705640 1 -- start start 2023-10-19T14:22:30.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.938+0000 7fe91a705640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe9140b0d20 0x7fe91415aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:30.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.938+0000 7fe91a705640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe9140b1940 0x7fe9140cb9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:30.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.938+0000 7fe91a705640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe91413d5f0 0x7fe9140cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:30.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.938+0000 7fe91a705640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe914129050 con 0x7fe91413d5f0 2023-10-19T14:22:30.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.938+0000 7fe91a705640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9140cc450 con 0x7fe9140b0d20 2023-10-19T14:22:30.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.938+0000 7fe91a705640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe91412bef0 con 0x7fe9140b1940 2023-10-19T14:22:30.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.938+0000 7fe918ca2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe91413d5f0 0x7fe9140cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:30.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.938+0000 7fe918ca2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe91413d5f0 0x7fe9140cbf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38920/0 (socket says 172.21.15.57:38920) 2023-10-19T14:22:30.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.938+0000 7fe918ca2640 1 -- 172.21.15.57:0/1893635630 learned_addr learned my addr 172.21.15.57:0/1893635630 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:22:30.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.938+0000 7fe913fff640 1 --2- 172.21.15.57:0/1893635630 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe9140b0d20 0x7fe91415aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:30.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.938+0000 7fe9137fe640 1 --2- 172.21.15.57:0/1893635630 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe9140b1940 0x7fe9140cb9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:30.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.938+0000 7fe918ca2640 1 -- 172.21.15.57:0/1893635630 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe9140b1940 msgr2=0x7fe9140cb9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:30.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.938+0000 7fe918ca2640 1 --2- 172.21.15.57:0/1893635630 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe9140b1940 0x7fe9140cb9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:30.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.938+0000 7fe918ca2640 1 -- 172.21.15.57:0/1893635630 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe9140b0d20 msgr2=0x7fe91415aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:30.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.938+0000 7fe918ca2640 1 --2- 172.21.15.57:0/1893635630 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe9140b0d20 0x7fe91415aef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:30.949 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.938+0000 7fe918ca2640 1 -- 172.21.15.57:0/1893635630 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe904009560 con 0x7fe91413d5f0 2023-10-19T14:22:30.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.938+0000 7fe918ca2640 1 --2- 172.21.15.57:0/1893635630 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe91413d5f0 0x7fe9140cbf10 secure :-1 s=READY pgs=3001 cs=0 l=1 rev1=1 crypto rx=0x7fe90c00dd70 tx=0x7fe90c00b730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:30.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.939+0000 7fe910ff9640 1 -- 172.21.15.57:0/1893635630 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe90c004d20 con 0x7fe91413d5f0 2023-10-19T14:22:30.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.939+0000 7fe910ff9640 1 -- 172.21.15.57:0/1893635630 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fe90c004e80 con 0x7fe91413d5f0 2023-10-19T14:22:30.950 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.939+0000 7fe910ff9640 1 -- 172.21.15.57:0/1893635630 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe90c00e8a0 con 0x7fe91413d5f0 2023-10-19T14:22:30.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.939+0000 7fe91a705640 1 -- 172.21.15.57:0/1893635630 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9141292f0 con 0x7fe91413d5f0 2023-10-19T14:22:30.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.939+0000 7fe91a705640 1 -- 172.21.15.57:0/1893635630 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9141298b0 con 0x7fe91413d5f0 2023-10-19T14:22:30.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.940+0000 7fe91a705640 1 -- 172.21.15.57:0/1893635630 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe914129aa0 con 0x7fe91413d5f0 2023-10-19T14:22:30.951 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.940+0000 7fe910ff9640 1 -- 172.21.15.57:0/1893635630 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 153) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fe90c00ea00 con 0x7fe91413d5f0 2023-10-19T14:22:30.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.940+0000 7fe910ff9640 1 --2- 172.21.15.57:0/1893635630 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fe90006ac90 0x7fe90006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:30.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.941+0000 7fe910ff9640 1 -- 172.21.15.57:0/1893635630 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(265..265 src has 1..265) v4 ==== 6874+0+0 (secure 0 0 0) 0x7fe90c08a4f0 con 0x7fe91413d5f0 2023-10-19T14:22:30.952 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.941+0000 7fe913fff640 1 --2- 172.21.15.57:0/1893635630 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fe90006ac90 0x7fe90006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:30.953 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.941+0000 7fe913fff640 1 --2- 172.21.15.57:0/1893635630 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fe90006ac90 0x7fe90006d150 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe908007fd0 tx=0x7fe908009040 comp rx=0 tx=0).ready entity=mgr.10520 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:30.954 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:30.945+0000 7fe910ff9640 1 -- 172.21.15.57:0/1893635630 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe90c055380 con 0x7fe91413d5f0 2023-10-19T14:22:31.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.155+0000 7fe91a705640 1 -- 172.21.15.57:0/1893635630 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe914069100 con 0x7fe91413d5f0 2023-10-19T14:22:31.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.155+0000 7fe910ff9640 1 -- 172.21.15.57:0/1893635630 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+4935 (secure 0 0 0) 0x7fe90c00ecb0 con 0x7fe91413d5f0 2023-10-19T14:22:31.159 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:22:31.159 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":116,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10280,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/1738328496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":1738328496},{"type":"v1","addr":"172.21.15.179:6837","nonce":1738328496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":113},{"gid":13344,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/3191801579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":3191801579},{"type":"v1","addr":"172.21.15.57:6834","nonce":3191801579}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":113},{"gid":13347,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/3876822488","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":3876822488},{"type":"v1","addr":"172.21.15.57:6835","nonce":3876822488}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T14:22:01.763128+0000","modified":"2023-10-19T14:22:03.758375+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":13359},"failed":[],"damaged":[],"stopped":[],"info":{"gid_13359":{"gid":13359,"name":"d","rank":0,"incarnation":114,"state":"up:active","state_seq":3,"addr":"172.21.15.179:6836/4133411706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":4133411706},{"type":"v1","addr":"172.21.15.179:6836","nonce":4133411706}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[36],"metadata_pool":35,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":17}]} 2023-10-19T14:22:31.159 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 116 2023-10-19T14:22:31.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.157+0000 7fe91a705640 1 -- 172.21.15.57:0/1893635630 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fe90006ac90 msgr2=0x7fe90006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:31.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.157+0000 7fe91a705640 1 --2- 172.21.15.57:0/1893635630 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fe90006ac90 0x7fe90006d150 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe908007fd0 tx=0x7fe908009040 comp rx=0 tx=0).stop 2023-10-19T14:22:31.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.157+0000 7fe91a705640 1 -- 172.21.15.57:0/1893635630 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe91413d5f0 msgr2=0x7fe9140cbf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:31.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.157+0000 7fe91a705640 1 --2- 172.21.15.57:0/1893635630 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe91413d5f0 0x7fe9140cbf10 secure :-1 s=READY pgs=3001 cs=0 l=1 rev1=1 crypto rx=0x7fe90c00dd70 tx=0x7fe90c00b730 comp rx=0 tx=0).stop 2023-10-19T14:22:31.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.158+0000 7fe91a705640 1 -- 172.21.15.57:0/1893635630 shutdown_connections 2023-10-19T14:22:31.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.158+0000 7fe91a705640 1 --2- 172.21.15.57:0/1893635630 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fe90006ac90 0x7fe90006d150 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:31.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.158+0000 7fe91a705640 1 --2- 172.21.15.57:0/1893635630 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe91413d5f0 0x7fe9140cbf10 unknown :-1 s=CLOSED pgs=3001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:31.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.158+0000 7fe91a705640 1 --2- 172.21.15.57:0/1893635630 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe9140b1940 0x7fe9140cb9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:31.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.158+0000 7fe91a705640 1 --2- 172.21.15.57:0/1893635630 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe9140b0d20 0x7fe91415aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:31.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.158+0000 7fe91a705640 1 -- 172.21.15.57:0/1893635630 >> 172.21.15.57:0/1893635630 conn(0x7fe914069810 msgr2=0x7fe91406be90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:31.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.158+0000 7fe91a705640 1 -- 172.21.15.57:0/1893635630 shutdown_connections 2023-10-19T14:22:31.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.158+0000 7fe91a705640 1 -- 172.21.15.57:0/1893635630 wait complete. 2023-10-19T14:22:31.171 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T14:22:31.280 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.277+0000 7f7f1bab4640 1 -- 172.21.15.57:0/3478593653 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7f140b1f30 msgr2=0x7f7f140b0c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:31.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.277+0000 7f7f1bab4640 1 --2- 172.21.15.57:0/3478593653 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7f140b1f30 0x7f7f140b0c10 secure :-1 s=READY pgs=3003 cs=0 l=1 rev1=1 crypto rx=0x7f7f140afaf0 tx=0x7f7f0402be40 comp rx=0 tx=0).stop 2023-10-19T14:22:31.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.277+0000 7f7f1bab4640 1 -- 172.21.15.57:0/3478593653 shutdown_connections 2023-10-19T14:22:31.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.277+0000 7f7f1bab4640 1 --2- 172.21.15.57:0/3478593653 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7f140b1f30 0x7f7f140b0c10 unknown :-1 s=CLOSED pgs=3003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:31.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.277+0000 7f7f1bab4640 1 --2- 172.21.15.57:0/3478593653 >> v2:172.21.15.179:3300/0 conn(0x7f7f141417e0 0x7f7f14070a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:31.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.277+0000 7f7f1bab4640 1 -- 172.21.15.57:0/3478593653 >> 172.21.15.57:0/3478593653 conn(0x7f7f14069810 msgr2=0x7f7f14069c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:31.282 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.277+0000 7f7f1bab4640 1 -- 172.21.15.57:0/3478593653 shutdown_connections 2023-10-19T14:22:31.282 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.277+0000 7f7f1bab4640 1 -- 172.21.15.57:0/3478593653 wait complete. 2023-10-19T14:22:31.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.278+0000 7f7f1bab4640 1 Processor -- start 2023-10-19T14:22:31.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.278+0000 7f7f1bab4640 1 -- start start 2023-10-19T14:22:31.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.279+0000 7f7f1bab4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7f1406f2a0 0x7f7f1409a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:31.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.279+0000 7f7f1bab4640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7f140b1f30 0x7f7f140ad870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:31.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.279+0000 7f7f1bab4640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7f141417e0 0x7f7f1414a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:31.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.279+0000 7f7f1bab4640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f14139df0 con 0x7f7f1406f2a0 2023-10-19T14:22:31.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.279+0000 7f7f1bab4640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f1413a090 con 0x7f7f141417e0 2023-10-19T14:22:31.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.279+0000 7f7f1bab4640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f1413a370 con 0x7f7f140b1f30 2023-10-19T14:22:31.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.279+0000 7f7f19850640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7f1406f2a0 0x7f7f1409a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:31.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.279+0000 7f7f19850640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7f1406f2a0 0x7f7f1409a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38944/0 (socket says 172.21.15.57:38944) 2023-10-19T14:22:31.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.279+0000 7f7f19850640 1 -- 172.21.15.57:0/3541209286 learned_addr learned my addr 172.21.15.57:0/3541209286 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:22:31.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.279+0000 7f7f1a051640 1 --2- 172.21.15.57:0/3541209286 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7f141417e0 0x7f7f1414a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:31.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.279+0000 7f7f1904f640 1 --2- 172.21.15.57:0/3541209286 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7f140b1f30 0x7f7f140ad870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:31.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.280+0000 7f7f19850640 1 -- 172.21.15.57:0/3541209286 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7f140b1f30 msgr2=0x7f7f140ad870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:31.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.280+0000 7f7f19850640 1 --2- 172.21.15.57:0/3541209286 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7f140b1f30 0x7f7f140ad870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:31.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.280+0000 7f7f19850640 1 -- 172.21.15.57:0/3541209286 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7f141417e0 msgr2=0x7f7f1414a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:31.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.280+0000 7f7f19850640 1 --2- 172.21.15.57:0/3541209286 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7f141417e0 0x7f7f1414a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:31.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.280+0000 7f7f19850640 1 -- 172.21.15.57:0/3541209286 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f04009560 con 0x7f7f1406f2a0 2023-10-19T14:22:31.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.280+0000 7f7f19850640 1 --2- 172.21.15.57:0/3541209286 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7f1406f2a0 0x7f7f1409a300 secure :-1 s=READY pgs=3004 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c00cd10 tx=0x7f7f0c00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:31.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.280+0000 7f7f127fc640 1 -- 172.21.15.57:0/3541209286 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f0c00f450 con 0x7f7f1406f2a0 2023-10-19T14:22:31.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.280+0000 7f7f127fc640 1 -- 172.21.15.57:0/3541209286 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f7f0c00f6a0 con 0x7f7f1406f2a0 2023-10-19T14:22:31.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.280+0000 7f7f1bab4640 1 -- 172.21.15.57:0/3541209286 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f1413a650 con 0x7f7f1406f2a0 2023-10-19T14:22:31.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.280+0000 7f7f127fc640 1 -- 172.21.15.57:0/3541209286 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f0c00fa90 con 0x7f7f1406f2a0 2023-10-19T14:22:31.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.280+0000 7f7f1bab4640 1 -- 172.21.15.57:0/3541209286 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f1409a840 con 0x7f7f1406f2a0 2023-10-19T14:22:31.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.281+0000 7f7f1bab4640 1 -- 172.21.15.57:0/3541209286 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f1413a1d0 con 0x7f7f1406f2a0 2023-10-19T14:22:31.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.281+0000 7f7f127fc640 1 -- 172.21.15.57:0/3541209286 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 153) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f7f0c00fbf0 con 0x7f7f1406f2a0 2023-10-19T14:22:31.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.282+0000 7f7f127fc640 1 --2- 172.21.15.57:0/3541209286 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f7f0006acc0 0x7f7f0006d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:31.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.282+0000 7f7f127fc640 1 -- 172.21.15.57:0/3541209286 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(265..265 src has 1..265) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f7f0c08a260 con 0x7f7f1406f2a0 2023-10-19T14:22:31.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.282+0000 7f7f1904f640 1 --2- 172.21.15.57:0/3541209286 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f7f0006acc0 0x7f7f0006d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:31.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.282+0000 7f7f1904f640 1 --2- 172.21.15.57:0/3541209286 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f7f0006acc0 0x7f7f0006d180 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f7f14157420 tx=0x7f7f08009040 comp rx=0 tx=0).ready entity=mgr.10520 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:31.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.286+0000 7f7f127fc640 1 -- 172.21.15.57:0/3541209286 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7f1413a1d0 con 0x7f7f1406f2a0 2023-10-19T14:22:31.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.499+0000 7f7f1bab4640 1 -- 172.21.15.57:0/3541209286 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7f1413ab70 con 0x7f7f1406f2a0 2023-10-19T14:22:31.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.500+0000 7f7f127fc640 1 -- 172.21.15.57:0/3541209286 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+4935 (secure 0 0 0) 0x7f7f1413ab70 con 0x7f7f1406f2a0 2023-10-19T14:22:31.503 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:22:31.503 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":116,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10280,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/1738328496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":1738328496},{"type":"v1","addr":"172.21.15.179:6837","nonce":1738328496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":113},{"gid":13344,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/3191801579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":3191801579},{"type":"v1","addr":"172.21.15.57:6834","nonce":3191801579}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":113},{"gid":13347,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/3876822488","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":3876822488},{"type":"v1","addr":"172.21.15.57:6835","nonce":3876822488}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-19T14:22:01.763128+0000","modified":"2023-10-19T14:22:03.758375+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":13359},"failed":[],"damaged":[],"stopped":[],"info":{"gid_13359":{"gid":13359,"name":"d","rank":0,"incarnation":114,"state":"up:active","state_seq":3,"addr":"172.21.15.179:6836/4133411706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":4133411706},{"type":"v1","addr":"172.21.15.179:6836","nonce":4133411706}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[36],"metadata_pool":35,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":17}]} 2023-10-19T14:22:31.504 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 116 2023-10-19T14:22:31.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.502+0000 7f7f1bab4640 1 -- 172.21.15.57:0/3541209286 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f7f0006acc0 msgr2=0x7f7f0006d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:31.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.502+0000 7f7f1bab4640 1 --2- 172.21.15.57:0/3541209286 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f7f0006acc0 0x7f7f0006d180 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f7f14157420 tx=0x7f7f08009040 comp rx=0 tx=0).stop 2023-10-19T14:22:31.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.502+0000 7f7f1bab4640 1 -- 172.21.15.57:0/3541209286 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7f1406f2a0 msgr2=0x7f7f1409a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:31.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.502+0000 7f7f1bab4640 1 --2- 172.21.15.57:0/3541209286 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7f1406f2a0 0x7f7f1409a300 secure :-1 s=READY pgs=3004 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c00cd10 tx=0x7f7f0c00a620 comp rx=0 tx=0).stop 2023-10-19T14:22:31.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.502+0000 7f7f1bab4640 1 -- 172.21.15.57:0/3541209286 shutdown_connections 2023-10-19T14:22:31.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.502+0000 7f7f1bab4640 1 --2- 172.21.15.57:0/3541209286 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f7f0006acc0 0x7f7f0006d180 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:31.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.502+0000 7f7f1bab4640 1 --2- 172.21.15.57:0/3541209286 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7f141417e0 0x7f7f1414a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:31.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.502+0000 7f7f1bab4640 1 --2- 172.21.15.57:0/3541209286 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7f140b1f30 0x7f7f140ad870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:31.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.502+0000 7f7f1bab4640 1 --2- 172.21.15.57:0/3541209286 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7f1406f2a0 0x7f7f1409a300 unknown :-1 s=CLOSED pgs=3004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:31.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.502+0000 7f7f1bab4640 1 -- 172.21.15.57:0/3541209286 >> 172.21.15.57:0/3541209286 conn(0x7f7f14069810 msgr2=0x7f7f14140880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:31.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.502+0000 7f7f1bab4640 1 -- 172.21.15.57:0/3541209286 shutdown_connections 2023-10-19T14:22:31.508 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.502+0000 7f7f1bab4640 1 -- 172.21.15.57:0/3541209286 wait complete. 2023-10-19T14:22:31.515 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T14:22:31.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.622+0000 7fdedca1a640 1 -- 172.21.15.57:0/1963439643 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdec4003b30 msgr2=0x7fdec4003f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:31.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.622+0000 7fdedca1a640 1 --2- 172.21.15.57:0/1963439643 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdec4003b30 0x7fdec4003f30 secure :-1 s=READY pgs=3006 cs=0 l=1 rev1=1 crypto rx=0x7fded0001800 tx=0x7fdec40057f0 comp rx=0 tx=0).stop 2023-10-19T14:22:31.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.622+0000 7fded6d9d640 1 -- 172.21.15.57:0/1963439643 reap_dead start 2023-10-19T14:22:31.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.622+0000 7fdedca1a640 1 -- 172.21.15.57:0/1963439643 shutdown_connections 2023-10-19T14:22:31.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.622+0000 7fdedca1a640 1 -- 172.21.15.57:0/1963439643 >> 172.21.15.57:0/1963439643 conn(0x7fded809a140 msgr2=0x7fded80d0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:31.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.622+0000 7fdedca1a640 1 -- 172.21.15.57:0/1963439643 shutdown_connections 2023-10-19T14:22:31.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.622+0000 7fdedca1a640 1 -- 172.21.15.57:0/1963439643 wait complete. 2023-10-19T14:22:31.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.623+0000 7fdedca1a640 1 Processor -- start 2023-10-19T14:22:31.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.623+0000 7fdedca1a640 1 -- start start 2023-10-19T14:22:31.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.624+0000 7fdedca1a640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdec4003b30 0x7fded81574b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:31.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.624+0000 7fdedca1a640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fded80ca7c0 0x7fded80c6850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:31.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.624+0000 7fdedca1a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fded80cac40 0x7fded80c6d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:31.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.624+0000 7fdedca1a640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fded80c8050 con 0x7fded80cac40 2023-10-19T14:22:31.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.624+0000 7fdedca1a640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fded80c8260 con 0x7fdec4003b30 2023-10-19T14:22:31.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.624+0000 7fdedca1a640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fded80c8540 con 0x7fded80ca7c0 2023-10-19T14:22:31.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.624+0000 7fded6d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fded80cac40 0x7fded80c6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:31.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.624+0000 7fded6d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fded80cac40 0x7fded80c6d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38974/0 (socket says 172.21.15.57:38974) 2023-10-19T14:22:31.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.624+0000 7fded6d9d640 1 -- 172.21.15.57:0/3926338991 learned_addr learned my addr 172.21.15.57:0/3926338991 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:22:31.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.624+0000 7fded6d9d640 1 -- 172.21.15.57:0/3926338991 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fded80ca7c0 msgr2=0x7fded80c6850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:31.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.624+0000 7fded6d9d640 1 --2- 172.21.15.57:0/3926338991 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fded80ca7c0 0x7fded80c6850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:31.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.625+0000 7fded6d9d640 1 -- 172.21.15.57:0/3926338991 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdec4003b30 msgr2=0x7fded81574b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:31.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.625+0000 7fded659c640 1 --2- 172.21.15.57:0/3926338991 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdec4003b30 0x7fded81574b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:31.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.625+0000 7fded6d9d640 1 --2- 172.21.15.57:0/3926338991 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdec4003b30 0x7fded81574b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:31.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.625+0000 7fded6d9d640 1 -- 172.21.15.57:0/3926338991 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fded80c8820 con 0x7fded80cac40 2023-10-19T14:22:31.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.625+0000 7fded659c640 1 --2- 172.21.15.57:0/3926338991 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdec4003b30 0x7fded81574b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:22:31.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.625+0000 7fded6d9d640 1 --2- 172.21.15.57:0/3926338991 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fded80cac40 0x7fded80c6d90 secure :-1 s=READY pgs=3007 cs=0 l=1 rev1=1 crypto rx=0x7fded8069e50 tx=0x7fdec8010f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:31.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.625+0000 7fdeceffd640 1 -- 172.21.15.57:0/3926338991 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdec800eeb0 con 0x7fded80cac40 2023-10-19T14:22:31.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.625+0000 7fdeceffd640 1 -- 172.21.15.57:0/3926338991 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fdec8003a60 con 0x7fded80cac40 2023-10-19T14:22:31.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.625+0000 7fdedca1a640 1 -- 172.21.15.57:0/3926338991 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fded80c8b00 con 0x7fded80cac40 2023-10-19T14:22:31.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.625+0000 7fdeceffd640 1 -- 172.21.15.57:0/3926338991 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdec8003e70 con 0x7fded80cac40 2023-10-19T14:22:31.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.625+0000 7fdedca1a640 1 -- 172.21.15.57:0/3926338991 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fded81579f0 con 0x7fded80cac40 2023-10-19T14:22:31.633 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.626+0000 7fdedca1a640 1 -- 172.21.15.57:0/3926338991 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fded815e130 con 0x7fded80cac40 2023-10-19T14:22:31.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.627+0000 7fdeceffd640 1 -- 172.21.15.57:0/3926338991 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 153) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fdec8019030 con 0x7fded80cac40 2023-10-19T14:22:31.634 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.627+0000 7fdeceffd640 1 --2- 172.21.15.57:0/3926338991 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fdeb806ac90 0x7fdeb806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:31.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.627+0000 7fdeceffd640 1 -- 172.21.15.57:0/3926338991 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(265..265 src has 1..265) v4 ==== 6874+0+0 (secure 0 0 0) 0x7fdec8017070 con 0x7fded80cac40 2023-10-19T14:22:31.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.627+0000 7fded659c640 1 --2- 172.21.15.57:0/3926338991 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fdeb806ac90 0x7fdeb806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:31.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.628+0000 7fded659c640 1 --2- 172.21.15.57:0/3926338991 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fdeb806ac90 0x7fdeb806d150 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fded0034000 tx=0x7fded002e070 comp rx=0 tx=0).ready entity=mgr.10520 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:31.635 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.631+0000 7fdeceffd640 1 -- 172.21.15.57:0/3926338991 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fdec80554f0 con 0x7fded80cac40 2023-10-19T14:22:31.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.807+0000 7fdedca1a640 1 -- 172.21.15.57:0/3926338991 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fded8069c10 con 0x7fded80cac40 2023-10-19T14:22:31.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.808+0000 7fdeceffd640 1 -- 172.21.15.57:0/3926338991 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v265) v1 ==== 74+0+17980 (secure 0 0 0) 0x7fdec8058b10 con 0x7fded80cac40 2023-10-19T14:22:31.811 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:22:31.812 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":265,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T14:22:29.793566+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":36,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_metadata","create_time":"2023-10-19T14:22:00.047912+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"262","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_data","create_time":"2023-10-19T14:22:00.739698+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"263","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.179:0/2632603637":"2023-10-20T14:22:29.793530+0000","172.21.15.179:0/2998005959":"2023-10-20T14:22:29.793530+0000","172.21.15.179:0/3483041475":"2023-10-20T14:22:29.793530+0000","172.21.15.179:0/3647691630":"2023-10-20T14:22:29.793530+0000","172.21.15.179:6833/2005622276":"2023-10-20T14:22:29.793530+0000","172.21.15.179:6832/2005622276":"2023-10-20T14:22:29.793530+0000","172.21.15.179:0/1983646445":"2023-10-20T14:22:17.865415+0000","172.21.15.179:0/1840191723":"2023-10-20T14:22:17.865415+0000","172.21.15.179:0/2852372821":"2023-10-20T14:22:29.793530+0000","172.21.15.179:0/4168916968":"2023-10-20T14:22:17.865415+0000","172.21.15.179:6833/1573806521":"2023-10-20T14:22:17.865415+0000","172.21.15.179:0/3886182326":"2023-10-20T14:22:29.793530+0000","172.21.15.179:6832/1573806521":"2023-10-20T14:22:17.865415+0000","172.21.15.179:0/658875312":"2023-10-20T14:22:17.865415+0000","172.21.15.179:0/1678121137":"2023-10-20T14:22:29.793530+0000","172.21.15.179:0/1799772260":"2023-10-20T14:22:17.865415+0000","172.21.15.179:0/3831670324":"2023-10-20T14:22:17.865415+0000","172.21.15.179:0/90219027":"2023-10-20T14:22:17.865415+0000","172.21.15.179:0/2739598080":"2023-10-20T14:22:17.865415+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T14:22:31.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.810+0000 7fdedca1a640 1 -- 172.21.15.57:0/3926338991 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fdeb806ac90 msgr2=0x7fdeb806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:31.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.810+0000 7fdedca1a640 1 --2- 172.21.15.57:0/3926338991 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fdeb806ac90 0x7fdeb806d150 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fded0034000 tx=0x7fded002e070 comp rx=0 tx=0).stop 2023-10-19T14:22:31.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.810+0000 7fdedca1a640 1 -- 172.21.15.57:0/3926338991 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fded80cac40 msgr2=0x7fded80c6d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:31.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.810+0000 7fdedca1a640 1 --2- 172.21.15.57:0/3926338991 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fded80cac40 0x7fded80c6d90 secure :-1 s=READY pgs=3007 cs=0 l=1 rev1=1 crypto rx=0x7fded8069e50 tx=0x7fdec8010f40 comp rx=0 tx=0).stop 2023-10-19T14:22:31.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.811+0000 7fdedca1a640 1 -- 172.21.15.57:0/3926338991 shutdown_connections 2023-10-19T14:22:31.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.811+0000 7fdedca1a640 1 --2- 172.21.15.57:0/3926338991 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fdeb806ac90 0x7fdeb806d150 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:31.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.811+0000 7fdedca1a640 1 --2- 172.21.15.57:0/3926338991 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fded80cac40 0x7fded80c6d90 unknown :-1 s=CLOSED pgs=3007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:31.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.811+0000 7fdedca1a640 1 --2- 172.21.15.57:0/3926338991 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fded80ca7c0 0x7fded80c6850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:31.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.811+0000 7fdedca1a640 1 --2- 172.21.15.57:0/3926338991 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdec4003b30 0x7fded81574b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:31.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.811+0000 7fdedca1a640 1 -- 172.21.15.57:0/3926338991 >> 172.21.15.57:0/3926338991 conn(0x7fded809a140 msgr2=0x7fded80cf1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:31.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.811+0000 7fdedca1a640 1 -- 172.21.15.57:0/3926338991 shutdown_connections 2023-10-19T14:22:31.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.811+0000 7fdedca1a640 1 -- 172.21.15.57:0/3926338991 wait complete. 2023-10-19T14:22:31.824 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-19T14:22:31.935 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.932+0000 7f6a227c5640 1 -- 172.21.15.57:0/2752084809 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6a1c06f2a0 msgr2=0x7f6a1c0b0c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:31.935 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.932+0000 7f6a227c5640 1 --2- 172.21.15.57:0/2752084809 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6a1c06f2a0 0x7f6a1c0b0c10 secure :-1 s=READY pgs=3008 cs=0 l=1 rev1=1 crypto rx=0x7f6a1c06b420 tx=0x7f6a1402c260 comp rx=0 tx=0).stop 2023-10-19T14:22:31.935 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.932+0000 7f6a227c5640 1 -- 172.21.15.57:0/2752084809 shutdown_connections 2023-10-19T14:22:31.936 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.932+0000 7f6a227c5640 1 --2- 172.21.15.57:0/2752084809 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6a1c06f2a0 0x7f6a1c0b0c10 unknown :-1 s=CLOSED pgs=3008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:31.936 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.932+0000 7f6a227c5640 1 --2- 172.21.15.57:0/2752084809 >> v2:172.21.15.179:3300/0 conn(0x7f6a1c0b1f30 0x7f6a1c070a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:31.936 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.932+0000 7f6a227c5640 1 -- 172.21.15.57:0/2752084809 >> 172.21.15.57:0/2752084809 conn(0x7f6a1c069810 msgr2=0x7f6a1c069c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:31.936 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.932+0000 7f6a227c5640 1 -- 172.21.15.57:0/2752084809 shutdown_connections 2023-10-19T14:22:31.936 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.932+0000 7f6a227c5640 1 -- 172.21.15.57:0/2752084809 wait complete. 2023-10-19T14:22:31.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.933+0000 7f6a227c5640 1 Processor -- start 2023-10-19T14:22:31.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.933+0000 7f6a227c5640 1 -- start start 2023-10-19T14:22:31.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.934+0000 7f6a227c5640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6a1c06f2a0 0x7f6a1c13b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:31.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.934+0000 7f6a227c5640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6a1c0b1f30 0x7f6a1c09a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:31.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.934+0000 7f6a227c5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6a1c1417e0 0x7f6a1c132b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:31.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.934+0000 7f6a227c5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a1c09a280 con 0x7f6a1c1417e0 2023-10-19T14:22:31.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.934+0000 7f6a227c5640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a1c0cbfb0 con 0x7f6a1c06f2a0 2023-10-19T14:22:31.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.934+0000 7f6a227c5640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a1c099c30 con 0x7f6a1c0b1f30 2023-10-19T14:22:31.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.934+0000 7f6a20d62640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6a1c1417e0 0x7f6a1c132b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:31.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.934+0000 7f6a20d62640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6a1c1417e0 0x7f6a1c132b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38986/0 (socket says 172.21.15.57:38986) 2023-10-19T14:22:31.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.934+0000 7f6a20d62640 1 -- 172.21.15.57:0/1123701378 learned_addr learned my addr 172.21.15.57:0/1123701378 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:22:31.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.934+0000 7f6a1bfff640 1 --2- 172.21.15.57:0/1123701378 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6a1c06f2a0 0x7f6a1c13b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:31.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.934+0000 7f6a1b7fe640 1 --2- 172.21.15.57:0/1123701378 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6a1c0b1f30 0x7f6a1c09a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:31.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.934+0000 7f6a20d62640 1 -- 172.21.15.57:0/1123701378 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6a1c0b1f30 msgr2=0x7f6a1c09a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:31.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.934+0000 7f6a20d62640 1 --2- 172.21.15.57:0/1123701378 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6a1c0b1f30 0x7f6a1c09a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:31.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.934+0000 7f6a20d62640 1 -- 172.21.15.57:0/1123701378 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6a1c06f2a0 msgr2=0x7f6a1c13b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:31.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.934+0000 7f6a20d62640 1 --2- 172.21.15.57:0/1123701378 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6a1c06f2a0 0x7f6a1c13b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:31.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.934+0000 7f6a20d62640 1 -- 172.21.15.57:0/1123701378 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a14009900 con 0x7f6a1c1417e0 2023-10-19T14:22:31.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.934+0000 7f6a1b7fe640 1 --2- 172.21.15.57:0/1123701378 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6a1c0b1f30 0x7f6a1c09a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:22:31.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.935+0000 7f6a20d62640 1 --2- 172.21.15.57:0/1123701378 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6a1c1417e0 0x7f6a1c132b60 secure :-1 s=READY pgs=3009 cs=0 l=1 rev1=1 crypto rx=0x7f6a14037040 tx=0x7f6a14003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:31.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.935+0000 7f6a18ff9640 1 -- 172.21.15.57:0/1123701378 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a14004250 con 0x7f6a1c1417e0 2023-10-19T14:22:31.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.935+0000 7f6a18ff9640 1 -- 172.21.15.57:0/1123701378 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f6a140044a0 con 0x7f6a1c1417e0 2023-10-19T14:22:31.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.935+0000 7f6a227c5640 1 -- 172.21.15.57:0/1123701378 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a1c099f10 con 0x7f6a1c1417e0 2023-10-19T14:22:31.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.935+0000 7f6a18ff9640 1 -- 172.21.15.57:0/1123701378 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a14004890 con 0x7f6a1c1417e0 2023-10-19T14:22:31.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.935+0000 7f6a227c5640 1 -- 172.21.15.57:0/1123701378 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a1c13beb0 con 0x7f6a1c1417e0 2023-10-19T14:22:31.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.936+0000 7f6a227c5640 1 -- 172.21.15.57:0/1123701378 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a1c06c710 con 0x7f6a1c1417e0 2023-10-19T14:22:31.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.936+0000 7f6a18ff9640 1 -- 172.21.15.57:0/1123701378 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 153) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f6a1403b040 con 0x7f6a1c1417e0 2023-10-19T14:22:31.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.937+0000 7f6a18ff9640 1 --2- 172.21.15.57:0/1123701378 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f6a0806ac90 0x7f6a0806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:31.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.937+0000 7f6a18ff9640 1 -- 172.21.15.57:0/1123701378 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(265..265 src has 1..265) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f6a140ac5e0 con 0x7f6a1c1417e0 2023-10-19T14:22:31.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.937+0000 7f6a1bfff640 1 --2- 172.21.15.57:0/1123701378 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f6a0806ac90 0x7f6a0806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:31.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.937+0000 7f6a1bfff640 1 --2- 172.21.15.57:0/1123701378 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f6a0806ac90 0x7f6a0806d150 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f6a10006fd0 tx=0x7f6a10008040 comp rx=0 tx=0).ready entity=mgr.10520 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:31.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:31.941+0000 7f6a18ff9640 1 -- 172.21.15.57:0/1123701378 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f6a1c06c710 con 0x7f6a1c1417e0 2023-10-19T14:22:32.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:32.154+0000 7f6a227c5640 1 -- 172.21.15.57:0/1123701378 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f6a1c06c710 con 0x7f6a1c1417e0 2023-10-19T14:22:32.158 INFO:tasks.ceph.mon.a.smithi057.stderr:2023-10-19T14:22:32.155+0000 7f1253ff3640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-19T14:22:32.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:32.156+0000 7f6a18ff9640 1 -- 172.21.15.57:0/1123701378 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v117) v1 ==== 161+0+0 (secure 0 0 0) 0x7f6a1c06c710 con 0x7f6a1c1417e0 2023-10-19T14:22:32.159 INFO:teuthology.orchestra.run.smithi057.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-19T14:22:32.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:32.159+0000 7f6a227c5640 1 -- 172.21.15.57:0/1123701378 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f6a0806ac90 msgr2=0x7f6a0806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:32.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:32.159+0000 7f6a227c5640 1 --2- 172.21.15.57:0/1123701378 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f6a0806ac90 0x7f6a0806d150 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f6a10006fd0 tx=0x7f6a10008040 comp rx=0 tx=0).stop 2023-10-19T14:22:32.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:32.159+0000 7f6a227c5640 1 -- 172.21.15.57:0/1123701378 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6a1c1417e0 msgr2=0x7f6a1c132b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:32.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:32.159+0000 7f6a227c5640 1 --2- 172.21.15.57:0/1123701378 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6a1c1417e0 0x7f6a1c132b60 secure :-1 s=READY pgs=3009 cs=0 l=1 rev1=1 crypto rx=0x7f6a14037040 tx=0x7f6a14003c10 comp rx=0 tx=0).stop 2023-10-19T14:22:32.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:32.159+0000 7f6a227c5640 1 -- 172.21.15.57:0/1123701378 shutdown_connections 2023-10-19T14:22:32.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:32.159+0000 7f6a227c5640 1 --2- 172.21.15.57:0/1123701378 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f6a0806ac90 0x7f6a0806d150 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:32.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:32.159+0000 7f6a227c5640 1 --2- 172.21.15.57:0/1123701378 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6a1c1417e0 0x7f6a1c132b60 unknown :-1 s=CLOSED pgs=3009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:32.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:32.159+0000 7f6a227c5640 1 --2- 172.21.15.57:0/1123701378 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f6a1c0b1f30 0x7f6a1c09a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:32.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:32.159+0000 7f6a227c5640 1 --2- 172.21.15.57:0/1123701378 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6a1c06f2a0 0x7f6a1c13b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:32.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:32.159+0000 7f6a227c5640 1 -- 172.21.15.57:0/1123701378 >> 172.21.15.57:0/1123701378 conn(0x7f6a1c069810 msgr2=0x7f6a1c0ac9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:32.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:32.160+0000 7f6a227c5640 1 -- 172.21.15.57:0/1123701378 shutdown_connections 2023-10-19T14:22:32.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:32.160+0000 7f6a227c5640 1 -- 172.21.15.57:0/1123701378 wait complete. 2023-10-19T14:22:32.165 INFO:tasks.ceph.mds.d.smithi179.stderr: -3859> 2023-10-19T14:22:04.373+0000 7f67eaf27640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-19T14:22:32.173 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-19T14:22:32.194 INFO:tasks.ceph.mds.d.smithi179.stdout:starting mds.d at 2023-10-19T14:22:32.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:32.281+0000 7fa846789640 1 -- 172.21.15.57:0/3901830029 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa8400d1080 msgr2=0x7fa8400cea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:32.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:32.281+0000 7fa846789640 1 --2- 172.21.15.57:0/3901830029 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa8400d1080 0x7fa8400cea30 secure :-1 s=READY pgs=3012 cs=0 l=1 rev1=1 crypto rx=0x7fa84009ecd0 tx=0x7fa83402bd70 comp rx=0 tx=0).stop 2023-10-19T14:22:32.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:32.281+0000 7fa846789640 1 -- 172.21.15.57:0/3901830029 shutdown_connections 2023-10-19T14:22:32.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:32.281+0000 7fa846789640 1 --2- 172.21.15.57:0/3901830029 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa8400d1080 0x7fa8400cea30 unknown :-1 s=CLOSED pgs=3012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:32.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:32.281+0000 7fa846789640 1 --2- 172.21.15.57:0/3901830029 >> v2:172.21.15.179:3300/0 conn(0x7fa84015b070 0x7fa8400d0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:32.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:32.281+0000 7fa846789640 1 -- 172.21.15.57:0/3901830029 >> 172.21.15.57:0/3901830029 conn(0x7fa84008a7e0 msgr2=0x7fa84009a100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:32.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:32.281+0000 7fa846789640 1 -- 172.21.15.57:0/3901830029 shutdown_connections 2023-10-19T14:22:32.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:32.281+0000 7fa846789640 1 -- 172.21.15.57:0/3901830029 wait complete. 2023-10-19T14:22:32.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:32.282+0000 7fa846789640 1 Processor -- start 2023-10-19T14:22:32.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:32.283+0000 7fa846789640 1 -- start start 2023-10-19T14:22:32.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:32.283+0000 7fa846789640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa8400cef70 0x7fa84015d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:32.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:32.283+0000 7fa846789640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa8400d1080 0x7fa8400b4400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:32.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:32.283+0000 7fa846789640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa84015b070 0x7fa8401f0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:32.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:32.283+0000 7fa846789640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8400b57b0 con 0x7fa84015b070 2023-10-19T14:22:32.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:32.283+0000 7fa846789640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8400b4940 con 0x7fa8400d1080 2023-10-19T14:22:32.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:32.283+0000 7fa846789640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8400b4ba0 con 0x7fa8400cef70 2023-10-19T14:22:32.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:32.284+0000 7fa844d26640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa84015b070 0x7fa8401f0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:32.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:32.284+0000 7fa844d26640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa84015b070 0x7fa8401f0ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39004/0 (socket says 172.21.15.57:39004) 2023-10-19T14:22:32.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:32.284+0000 7fa844d26640 1 -- 172.21.15.57:0/1887038997 learned_addr learned my addr 172.21.15.57:0/1887038997 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:22:32.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:32.284+0000 7fa83ffff640 1 --2- 172.21.15.57:0/1887038997 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa8400cef70 0x7fa84015d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:32.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:32.284+0000 7fa83f7fe640 1 --2- 172.21.15.57:0/1887038997 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa8400d1080 0x7fa8400b4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:32.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:32.284+0000 7fa844d26640 1 -- 172.21.15.57:0/1887038997 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa8400cef70 msgr2=0x7fa84015d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:32.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:32.284+0000 7fa844d26640 1 --2- 172.21.15.57:0/1887038997 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa8400cef70 0x7fa84015d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:32.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:32.284+0000 7fa844d26640 1 -- 172.21.15.57:0/1887038997 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa8400d1080 msgr2=0x7fa8400b4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:32.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:32.284+0000 7fa844d26640 1 --2- 172.21.15.57:0/1887038997 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa8400d1080 0x7fa8400b4400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:32.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:32.284+0000 7fa844d26640 1 -- 172.21.15.57:0/1887038997 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa834009490 con 0x7fa84015b070 2023-10-19T14:22:32.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:32.284+0000 7fa83ffff640 1 --2- 172.21.15.57:0/1887038997 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa8400cef70 0x7fa84015d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:22:32.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:32.284+0000 7fa844d26640 1 --2- 172.21.15.57:0/1887038997 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa84015b070 0x7fa8401f0ee0 secure :-1 s=READY pgs=3013 cs=0 l=1 rev1=1 crypto rx=0x7fa83800cb40 tx=0x7fa83800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:32.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:32.284+0000 7fa83cff9640 1 -- 172.21.15.57:0/1887038997 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa83800aca0 con 0x7fa84015b070 2023-10-19T14:22:32.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:32.284+0000 7fa83cff9640 1 -- 172.21.15.57:0/1887038997 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fa838012420 con 0x7fa84015b070 2023-10-19T14:22:32.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:32.284+0000 7fa83cff9640 1 -- 172.21.15.57:0/1887038997 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa838012850 con 0x7fa84015b070 2023-10-19T14:22:32.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:32.284+0000 7fa846789640 1 -- 172.21.15.57:0/1887038997 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8400b4e80 con 0x7fa84015b070 2023-10-19T14:22:32.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:32.284+0000 7fa846789640 1 -- 172.21.15.57:0/1887038997 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8400b5440 con 0x7fa84015b070 2023-10-19T14:22:32.293 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:32.285+0000 7fa846789640 1 -- 172.21.15.57:0/1887038997 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa84004db50 con 0x7fa84015b070 2023-10-19T14:22:32.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:32.286+0000 7fa83cff9640 1 -- 172.21.15.57:0/1887038997 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 153) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fa83801a3b0 con 0x7fa84015b070 2023-10-19T14:22:32.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:32.286+0000 7fa83cff9640 1 --2- 172.21.15.57:0/1887038997 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fa82c06ac90 0x7fa82c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:32.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:32.286+0000 7fa83cff9640 1 -- 172.21.15.57:0/1887038997 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(266..266 src has 1..266) v4 ==== 6960+0+0 (secure 0 0 0) 0x7fa838089ea0 con 0x7fa84015b070 2023-10-19T14:22:32.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:32.286+0000 7fa83ffff640 1 --2- 172.21.15.57:0/1887038997 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fa82c06ac90 0x7fa82c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:32.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:32.287+0000 7fa83ffff640 1 --2- 172.21.15.57:0/1887038997 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fa82c06ac90 0x7fa82c06d150 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa83000ab40 tx=0x7fa83000a520 comp rx=0 tx=0).ready entity=mgr.10520 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:32.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:32.290+0000 7fa83cff9640 1 -- 172.21.15.57:0/1887038997 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa838054db0 con 0x7fa84015b070 2023-10-19T14:22:32.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:32.504+0000 7fa846789640 1 -- 172.21.15.57:0/1887038997 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fa84015a740 con 0x7fa84015b070 2023-10-19T14:22:32.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:32.809+0000 7fa83cff9640 1 -- 172.21.15.57:0/1887038997 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mgrmap(e 154) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fa838055180 con 0x7fa84015b070 2023-10-19T14:22:33.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:33.160+0000 7fa83cff9640 1 -- 172.21.15.57:0/1887038997 <== mon.0 v2:172.21.15.57:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v118) v1 ==== 104+0+0 (secure 0 0 0) 0x7fa8380716e0 con 0x7fa84015b070 2023-10-19T14:22:33.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:33.163+0000 7fa846789640 1 -- 172.21.15.57:0/1887038997 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fa82c06ac90 msgr2=0x7fa82c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:33.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:33.163+0000 7fa846789640 1 --2- 172.21.15.57:0/1887038997 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fa82c06ac90 0x7fa82c06d150 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa83000ab40 tx=0x7fa83000a520 comp rx=0 tx=0).stop 2023-10-19T14:22:33.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:33.163+0000 7fa846789640 1 -- 172.21.15.57:0/1887038997 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa84015b070 msgr2=0x7fa8401f0ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:33.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:33.163+0000 7fa846789640 1 --2- 172.21.15.57:0/1887038997 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa84015b070 0x7fa8401f0ee0 secure :-1 s=READY pgs=3013 cs=0 l=1 rev1=1 crypto rx=0x7fa83800cb40 tx=0x7fa83800a060 comp rx=0 tx=0).stop 2023-10-19T14:22:33.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:33.163+0000 7fa846789640 1 -- 172.21.15.57:0/1887038997 shutdown_connections 2023-10-19T14:22:33.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:33.163+0000 7fa846789640 1 --2- 172.21.15.57:0/1887038997 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fa82c06ac90 0x7fa82c06d150 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:33.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:33.163+0000 7fa846789640 1 --2- 172.21.15.57:0/1887038997 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa84015b070 0x7fa8401f0ee0 unknown :-1 s=CLOSED pgs=3013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:33.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:33.163+0000 7fa846789640 1 --2- 172.21.15.57:0/1887038997 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa8400d1080 0x7fa8400b4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:33.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:33.163+0000 7fa846789640 1 --2- 172.21.15.57:0/1887038997 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa8400cef70 0x7fa84015d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:33.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:33.163+0000 7fa846789640 1 -- 172.21.15.57:0/1887038997 >> 172.21.15.57:0/1887038997 conn(0x7fa84008a7e0 msgr2=0x7fa8400b1120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:33.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:33.163+0000 7fa846789640 1 -- 172.21.15.57:0/1887038997 shutdown_connections 2023-10-19T14:22:33.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:33.163+0000 7fa846789640 1 -- 172.21.15.57:0/1887038997 wait complete. 2023-10-19T14:22:33.179 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-19T14:22:33.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:33.296+0000 7f3100f97640 1 -- 172.21.15.57:0/2650116946 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f30e80039a0 msgr2=0x7f30e8003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:33.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:33.296+0000 7f3100f97640 1 --2- 172.21.15.57:0/2650116946 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f30e80039a0 0x7f30e8003dc0 secure :-1 s=READY pgs=3015 cs=0 l=1 rev1=1 crypto rx=0x7f30ec006fa0 tx=0x7f30e80057f0 comp rx=0 tx=0).stop 2023-10-19T14:22:33.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:33.296+0000 7f30fad9d640 1 -- 172.21.15.57:0/2650116946 reap_dead start 2023-10-19T14:22:33.300 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:33.296+0000 7f3100f97640 1 -- 172.21.15.57:0/2650116946 shutdown_connections 2023-10-19T14:22:33.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:33.296+0000 7f3100f97640 1 -- 172.21.15.57:0/2650116946 >> 172.21.15.57:0/2650116946 conn(0x7f30fc09a1a0 msgr2=0x7f30fc0d0eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:33.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:33.296+0000 7f3100f97640 1 -- 172.21.15.57:0/2650116946 shutdown_connections 2023-10-19T14:22:33.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:33.296+0000 7f3100f97640 1 -- 172.21.15.57:0/2650116946 wait complete. 2023-10-19T14:22:33.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:33.297+0000 7f3100f97640 1 Processor -- start 2023-10-19T14:22:33.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:33.298+0000 7f3100f97640 1 -- start start 2023-10-19T14:22:33.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:33.298+0000 7f3100f97640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f30e80039a0 0x7f30fc152510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:33.304 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:33.298+0000 7f3100f97640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f30fc151450 0x7f30fc153760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:33.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:33.298+0000 7f3100f97640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f30fc1e5050 0x7f30fc1e54d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:33.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:33.298+0000 7f3100f97640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30fc1e5a10 con 0x7f30e80039a0 2023-10-19T14:22:33.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:33.298+0000 7f3100f97640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30fc1518d0 con 0x7f30fc151450 2023-10-19T14:22:33.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:33.298+0000 7f3100f97640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30fc152af0 con 0x7f30fc1e5050 2023-10-19T14:22:33.305 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:33.298+0000 7f30fa59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f30e80039a0 0x7f30fc152510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:33.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:33.298+0000 7f30fa59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f30e80039a0 0x7f30fc152510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39030/0 (socket says 172.21.15.57:39030) 2023-10-19T14:22:33.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:33.298+0000 7f30fa59c640 1 -- 172.21.15.57:0/2846783413 learned_addr learned my addr 172.21.15.57:0/2846783413 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:22:33.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:33.299+0000 7f30fad9d640 1 --2- 172.21.15.57:0/2846783413 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f30fc1e5050 0x7f30fc1e54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:33.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:33.299+0000 7f30f3fff640 1 --2- 172.21.15.57:0/2846783413 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f30fc151450 0x7f30fc153760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:33.306 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:33.299+0000 7f30fa59c640 1 -- 172.21.15.57:0/2846783413 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f30fc1e5050 msgr2=0x7f30fc1e54d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:33.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:33.299+0000 7f30fa59c640 1 --2- 172.21.15.57:0/2846783413 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f30fc1e5050 0x7f30fc1e54d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:33.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:33.299+0000 7f30fa59c640 1 -- 172.21.15.57:0/2846783413 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f30fc151450 msgr2=0x7f30fc153760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:33.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:33.299+0000 7f30fa59c640 1 --2- 172.21.15.57:0/2846783413 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f30fc151450 0x7f30fc153760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:33.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:33.299+0000 7f30fa59c640 1 -- 172.21.15.57:0/2846783413 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30fc152dd0 con 0x7f30e80039a0 2023-10-19T14:22:33.307 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:33.299+0000 7f30fad9d640 1 --2- 172.21.15.57:0/2846783413 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f30fc1e5050 0x7f30fc1e54d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:22:33.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:33.299+0000 7f30f3fff640 1 --2- 172.21.15.57:0/2846783413 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f30fc151450 0x7f30fc153760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:22:33.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:33.299+0000 7f30fa59c640 1 --2- 172.21.15.57:0/2846783413 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f30e80039a0 0x7f30fc152510 secure :-1 s=READY pgs=3016 cs=0 l=1 rev1=1 crypto rx=0x7f30ec001930 tx=0x7f30ec0014b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:33.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:33.299+0000 7f30f2ffd640 1 -- 172.21.15.57:0/2846783413 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30ec006250 con 0x7f30e80039a0 2023-10-19T14:22:33.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:33.299+0000 7f30f2ffd640 1 -- 172.21.15.57:0/2846783413 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f30ec0063b0 con 0x7f30e80039a0 2023-10-19T14:22:33.308 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:33.299+0000 7f3100f97640 1 -- 172.21.15.57:0/2846783413 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30fc1530b0 con 0x7f30e80039a0 2023-10-19T14:22:33.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:33.299+0000 7f30f2ffd640 1 -- 172.21.15.57:0/2846783413 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30ec003850 con 0x7f30e80039a0 2023-10-19T14:22:33.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:33.299+0000 7f3100f97640 1 -- 172.21.15.57:0/2846783413 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30fc153ca0 con 0x7f30e80039a0 2023-10-19T14:22:33.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:33.300+0000 7f3100f97640 1 -- 172.21.15.57:0/2846783413 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30fc153e60 con 0x7f30e80039a0 2023-10-19T14:22:33.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:33.300+0000 7f30f2ffd640 1 -- 172.21.15.57:0/2846783413 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 154) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f30ec031030 con 0x7f30e80039a0 2023-10-19T14:22:33.309 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:33.301+0000 7f30f2ffd640 1 --2- 172.21.15.57:0/2846783413 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f30dc06aed0 0x7f30dc06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:33.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:33.301+0000 7f30f2ffd640 1 -- 172.21.15.57:0/2846783413 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(266..266 src has 1..266) v4 ==== 6960+0+0 (secure 0 0 0) 0x7f30ec0aaf90 con 0x7f30e80039a0 2023-10-19T14:22:33.310 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:33.301+0000 7f30f3fff640 1 --2- 172.21.15.57:0/2846783413 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f30dc06aed0 0x7f30dc06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:33.313 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:33.302+0000 7f30f3fff640 1 --2- 172.21.15.57:0/2846783413 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f30dc06aed0 0x7f30dc06d390 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f30fc1f4b70 tx=0x7f30e4005f90 comp rx=0 tx=0).ready entity=mgr.10520 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:33.314 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:33.305+0000 7f30f2ffd640 1 -- 172.21.15.57:0/2846783413 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f30ec079510 con 0x7f30e80039a0 2023-10-19T14:22:33.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:33.484+0000 7f3100f97640 1 -- 172.21.15.57:0/2846783413 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f30fc069100 con 0x7f30e80039a0 2023-10-19T14:22:34.407 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:34.404+0000 7f30f2ffd640 1 -- 172.21.15.57:0/2846783413 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v267) v1 ==== 181+0+0 (secure 0 0 0) 0x7f30ec045020 con 0x7f30e80039a0 2023-10-19T14:22:34.407 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'cephfs_metadata' removed 2023-10-19T14:22:34.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:34.412+0000 7f3100f97640 1 -- 172.21.15.57:0/2846783413 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f30dc06aed0 msgr2=0x7f30dc06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:34.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:34.412+0000 7f3100f97640 1 --2- 172.21.15.57:0/2846783413 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f30dc06aed0 0x7f30dc06d390 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f30fc1f4b70 tx=0x7f30e4005f90 comp rx=0 tx=0).stop 2023-10-19T14:22:34.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:34.412+0000 7f3100f97640 1 -- 172.21.15.57:0/2846783413 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f30e80039a0 msgr2=0x7f30fc152510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:34.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:34.412+0000 7f3100f97640 1 --2- 172.21.15.57:0/2846783413 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f30e80039a0 0x7f30fc152510 secure :-1 s=READY pgs=3016 cs=0 l=1 rev1=1 crypto rx=0x7f30ec001930 tx=0x7f30ec0014b0 comp rx=0 tx=0).stop 2023-10-19T14:22:34.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:34.413+0000 7f3100f97640 1 -- 172.21.15.57:0/2846783413 shutdown_connections 2023-10-19T14:22:34.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:34.413+0000 7f3100f97640 1 --2- 172.21.15.57:0/2846783413 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f30dc06aed0 0x7f30dc06d390 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:34.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:34.413+0000 7f3100f97640 1 --2- 172.21.15.57:0/2846783413 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f30fc1e5050 0x7f30fc1e54d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:34.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:34.413+0000 7f3100f97640 1 --2- 172.21.15.57:0/2846783413 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f30fc151450 0x7f30fc153760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:34.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:34.413+0000 7f3100f97640 1 --2- 172.21.15.57:0/2846783413 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f30e80039a0 0x7f30fc152510 unknown :-1 s=CLOSED pgs=3016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:34.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:34.413+0000 7f3100f97640 1 -- 172.21.15.57:0/2846783413 >> 172.21.15.57:0/2846783413 conn(0x7f30fc09a1a0 msgr2=0x7f30fc0c75a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:34.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:34.414+0000 7f3100f97640 1 -- 172.21.15.57:0/2846783413 shutdown_connections 2023-10-19T14:22:34.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:34.414+0000 7f3100f97640 1 -- 172.21.15.57:0/2846783413 wait complete. 2023-10-19T14:22:34.432 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-19T14:22:34.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:34.553+0000 7f7e7dd61640 1 -- 172.21.15.57:0/3526354119 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e60003970 msgr2=0x7f7e60003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:34.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:34.553+0000 7f7e7dd61640 1 --2- 172.21.15.57:0/3526354119 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e60003970 0x7f7e60003d90 secure :-1 s=READY pgs=3018 cs=0 l=1 rev1=1 crypto rx=0x7f7e68009b50 tx=0x7f7e600057f0 comp rx=0 tx=0).stop 2023-10-19T14:22:34.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:34.553+0000 7f7e7dd61640 1 -- 172.21.15.57:0/3526354119 shutdown_connections 2023-10-19T14:22:34.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:34.553+0000 7f7e7dd61640 1 --2- 172.21.15.57:0/3526354119 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e60003970 0x7f7e60003d90 unknown :-1 s=CLOSED pgs=3018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:34.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:34.553+0000 7f7e7dd61640 1 -- 172.21.15.57:0/3526354119 >> 172.21.15.57:0/3526354119 conn(0x7f7e7809a1a0 msgr2=0x7f7e780d0eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:34.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:34.553+0000 7f7e7dd61640 1 -- 172.21.15.57:0/3526354119 shutdown_connections 2023-10-19T14:22:34.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:34.553+0000 7f7e7dd61640 1 -- 172.21.15.57:0/3526354119 wait complete. 2023-10-19T14:22:34.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:34.554+0000 7f7e7dd61640 1 Processor -- start 2023-10-19T14:22:34.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:34.554+0000 7f7e7dd61640 1 -- start start 2023-10-19T14:22:34.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:34.555+0000 7f7e7dd61640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7e60003970 0x7f7e7809e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:34.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:34.555+0000 7f7e7dd61640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e7809e7e0 0x7f7e7815d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:34.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:34.555+0000 7f7e7dd61640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7e7815d870 0x7f7e780c56a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:34.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:34.555+0000 7f7e7dd61640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7815dd00 con 0x7f7e7809e7e0 2023-10-19T14:22:34.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:34.555+0000 7f7e7dd61640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e780c6410 con 0x7f7e7815d870 2023-10-19T14:22:34.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:34.555+0000 7f7e7dd61640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7814a2a0 con 0x7f7e60003970 2023-10-19T14:22:34.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:34.555+0000 7f7e76ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e7809e7e0 0x7f7e7815d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:34.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:34.555+0000 7f7e76ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e7809e7e0 0x7f7e7815d330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39034/0 (socket says 172.21.15.57:39034) 2023-10-19T14:22:34.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:34.555+0000 7f7e76ffd640 1 -- 172.21.15.57:0/2514901752 learned_addr learned my addr 172.21.15.57:0/2514901752 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:22:34.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:34.555+0000 7f7e777fe640 1 --2- 172.21.15.57:0/2514901752 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7e60003970 0x7f7e7809e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:34.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:34.555+0000 7f7e77fff640 1 --2- 172.21.15.57:0/2514901752 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7e7815d870 0x7f7e780c56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:34.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:34.555+0000 7f7e76ffd640 1 -- 172.21.15.57:0/2514901752 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7e60003970 msgr2=0x7f7e7809e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:34.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:34.555+0000 7f7e76ffd640 1 --2- 172.21.15.57:0/2514901752 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7e60003970 0x7f7e7809e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:34.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:34.555+0000 7f7e76ffd640 1 -- 172.21.15.57:0/2514901752 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7e7815d870 msgr2=0x7f7e780c56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:34.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:34.555+0000 7f7e76ffd640 1 --2- 172.21.15.57:0/2514901752 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7e7815d870 0x7f7e780c56a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:34.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:34.556+0000 7f7e76ffd640 1 -- 172.21.15.57:0/2514901752 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e7814a580 con 0x7f7e7809e7e0 2023-10-19T14:22:34.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:34.556+0000 7f7e777fe640 1 --2- 172.21.15.57:0/2514901752 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7e60003970 0x7f7e7809e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:22:34.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:34.556+0000 7f7e77fff640 1 --2- 172.21.15.57:0/2514901752 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7e7815d870 0x7f7e780c56a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:22:34.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:34.556+0000 7f7e76ffd640 1 --2- 172.21.15.57:0/2514901752 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e7809e7e0 0x7f7e7815d330 secure :-1 s=READY pgs=3019 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c001400 tx=0x7f7e6c0039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:34.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:34.556+0000 7f7e5ffff640 1 -- 172.21.15.57:0/2514901752 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e6c011900 con 0x7f7e7809e7e0 2023-10-19T14:22:34.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:34.556+0000 7f7e5ffff640 1 -- 172.21.15.57:0/2514901752 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f7e6c011a80 con 0x7f7e7809e7e0 2023-10-19T14:22:34.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:34.556+0000 7f7e7dd61640 1 -- 172.21.15.57:0/2514901752 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e7814a860 con 0x7f7e7809e7e0 2023-10-19T14:22:34.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:34.556+0000 7f7e5ffff640 1 -- 172.21.15.57:0/2514901752 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e6c011e90 con 0x7f7e7809e7e0 2023-10-19T14:22:34.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:34.556+0000 7f7e7dd61640 1 -- 172.21.15.57:0/2514901752 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e780c5dc0 con 0x7f7e7809e7e0 2023-10-19T14:22:34.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:34.557+0000 7f7e7dd61640 1 -- 172.21.15.57:0/2514901752 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e780666a0 con 0x7f7e7809e7e0 2023-10-19T14:22:34.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:34.557+0000 7f7e5ffff640 1 -- 172.21.15.57:0/2514901752 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 155) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f7e6c010420 con 0x7f7e7809e7e0 2023-10-19T14:22:34.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:34.558+0000 7f7e5ffff640 1 --2- 172.21.15.57:0/2514901752 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f7e6406af60 0x7f7e6406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:34.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:34.558+0000 7f7e5ffff640 1 -- 172.21.15.57:0/2514901752 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(267..267 src has 1..267) v4 ==== 6522+0+0 (secure 0 0 0) 0x7f7e6c08adc0 con 0x7f7e7809e7e0 2023-10-19T14:22:34.566 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:34.558+0000 7f7e777fe640 1 --2- 172.21.15.57:0/2514901752 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f7e6406af60 0x7f7e6406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:34.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:34.559+0000 7f7e777fe640 1 --2- 172.21.15.57:0/2514901752 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f7e6406af60 0x7f7e6406d420 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7e68036000 tx=0x7f7e680046c0 comp rx=0 tx=0).ready entity=mgr.10520 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:34.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:34.562+0000 7f7e5ffff640 1 -- 172.21.15.57:0/2514901752 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7e6c055930 con 0x7f7e7809e7e0 2023-10-19T14:22:34.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:34.737+0000 7f7e7dd61640 1 -- 172.21.15.57:0/2514901752 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f7e78069100 con 0x7f7e7809e7e0 2023-10-19T14:22:35.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.407+0000 7f7e5ffff640 1 -- 172.21.15.57:0/2514901752 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v268) v1 ==== 169+0+0 (secure 0 0 0) 0x7f7e6c058f50 con 0x7f7e7809e7e0 2023-10-19T14:22:35.410 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'cephfs_data' removed 2023-10-19T14:22:35.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.410+0000 7f7e7dd61640 1 -- 172.21.15.57:0/2514901752 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f7e6406af60 msgr2=0x7f7e6406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:35.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.410+0000 7f7e7dd61640 1 --2- 172.21.15.57:0/2514901752 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f7e6406af60 0x7f7e6406d420 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7e68036000 tx=0x7f7e680046c0 comp rx=0 tx=0).stop 2023-10-19T14:22:35.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.410+0000 7f7e7dd61640 1 -- 172.21.15.57:0/2514901752 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e7809e7e0 msgr2=0x7f7e7815d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:35.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.410+0000 7f7e7dd61640 1 --2- 172.21.15.57:0/2514901752 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e7809e7e0 0x7f7e7815d330 secure :-1 s=READY pgs=3019 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c001400 tx=0x7f7e6c0039c0 comp rx=0 tx=0).stop 2023-10-19T14:22:35.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.411+0000 7f7e7dd61640 1 -- 172.21.15.57:0/2514901752 shutdown_connections 2023-10-19T14:22:35.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.411+0000 7f7e7dd61640 1 --2- 172.21.15.57:0/2514901752 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f7e6406af60 0x7f7e6406d420 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:35.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.411+0000 7f7e7dd61640 1 --2- 172.21.15.57:0/2514901752 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7e7815d870 0x7f7e780c56a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:35.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.411+0000 7f7e7dd61640 1 --2- 172.21.15.57:0/2514901752 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e7809e7e0 0x7f7e7815d330 unknown :-1 s=CLOSED pgs=3019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:35.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.411+0000 7f7e7dd61640 1 --2- 172.21.15.57:0/2514901752 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f7e60003970 0x7f7e7809e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:35.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.411+0000 7f7e7dd61640 1 -- 172.21.15.57:0/2514901752 >> 172.21.15.57:0/2514901752 conn(0x7f7e7809a1a0 msgr2=0x7f7e780c9f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:35.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.413+0000 7f7e7dd61640 1 -- 172.21.15.57:0/2514901752 shutdown_connections 2023-10-19T14:22:35.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.414+0000 7f7e7dd61640 1 -- 172.21.15.57:0/2514901752 wait complete. 2023-10-19T14:22:35.433 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 30 2023-10-19T14:22:35.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.547+0000 7fdb3d37d640 1 -- 172.21.15.57:0/4246307329 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdb240047c0 msgr2=0x7fdb24004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:35.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.547+0000 7fdb3d37d640 1 --2- 172.21.15.57:0/4246307329 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdb240047c0 0x7fdb24004c40 secure :-1 s=READY pgs=3021 cs=0 l=1 rev1=1 crypto rx=0x7fdb20006f00 tx=0x7fdb240057f0 comp rx=0 tx=0).stop 2023-10-19T14:22:35.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.547+0000 7fdb377fe640 1 -- 172.21.15.57:0/4246307329 reap_dead start 2023-10-19T14:22:35.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.547+0000 7fdb3d37d640 1 -- 172.21.15.57:0/4246307329 shutdown_connections 2023-10-19T14:22:35.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.547+0000 7fdb3d37d640 1 -- 172.21.15.57:0/4246307329 >> 172.21.15.57:0/4246307329 conn(0x7fdb3809a140 msgr2=0x7fdb380d0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:35.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.547+0000 7fdb3d37d640 1 -- 172.21.15.57:0/4246307329 shutdown_connections 2023-10-19T14:22:35.551 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.547+0000 7fdb3d37d640 1 -- 172.21.15.57:0/4246307329 wait complete. 2023-10-19T14:22:35.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.548+0000 7fdb3d37d640 1 Processor -- start 2023-10-19T14:22:35.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.548+0000 7fdb3d37d640 1 -- start start 2023-10-19T14:22:35.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.549+0000 7fdb3d37d640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdb240047c0 0x7fdb380c9320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:35.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.549+0000 7fdb3d37d640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdb3809abb0 0x7fdb3809b030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:35.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.549+0000 7fdb3d37d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdb380ca300 0x7fdb380ca780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:35.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.549+0000 7fdb3d37d640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb381f1530 con 0x7fdb380ca300 2023-10-19T14:22:35.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.549+0000 7fdb3d37d640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb380c8280 con 0x7fdb3809abb0 2023-10-19T14:22:35.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.549+0000 7fdb3d37d640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb380c8560 con 0x7fdb240047c0 2023-10-19T14:22:35.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.549+0000 7fdb377fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdb380ca300 0x7fdb380ca780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:35.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.549+0000 7fdb377fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdb380ca300 0x7fdb380ca780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39044/0 (socket says 172.21.15.57:39044) 2023-10-19T14:22:35.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.549+0000 7fdb377fe640 1 -- 172.21.15.57:0/860952166 learned_addr learned my addr 172.21.15.57:0/860952166 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:22:35.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.549+0000 7fdb367fc640 1 --2- 172.21.15.57:0/860952166 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdb3809abb0 0x7fdb3809b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:35.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.549+0000 7fdb36ffd640 1 --2- 172.21.15.57:0/860952166 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdb240047c0 0x7fdb380c9320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:35.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.549+0000 7fdb377fe640 1 -- 172.21.15.57:0/860952166 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdb240047c0 msgr2=0x7fdb380c9320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:35.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.549+0000 7fdb377fe640 1 --2- 172.21.15.57:0/860952166 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdb240047c0 0x7fdb380c9320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:35.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.549+0000 7fdb377fe640 1 -- 172.21.15.57:0/860952166 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdb3809abb0 msgr2=0x7fdb3809b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:35.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.549+0000 7fdb377fe640 1 --2- 172.21.15.57:0/860952166 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdb3809abb0 0x7fdb3809b030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:35.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.549+0000 7fdb377fe640 1 -- 172.21.15.57:0/860952166 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb380c8840 con 0x7fdb380ca300 2023-10-19T14:22:35.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.549+0000 7fdb36ffd640 1 --2- 172.21.15.57:0/860952166 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdb240047c0 0x7fdb380c9320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:22:35.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.550+0000 7fdb377fe640 1 --2- 172.21.15.57:0/860952166 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdb380ca300 0x7fdb380ca780 secure :-1 s=READY pgs=3022 cs=0 l=1 rev1=1 crypto rx=0x7fdb380cfcc0 tx=0x7fdb28001c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:35.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.550+0000 7fdb2f7fe640 1 -- 172.21.15.57:0/860952166 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb28011900 con 0x7fdb380ca300 2023-10-19T14:22:35.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.550+0000 7fdb2f7fe640 1 -- 172.21.15.57:0/860952166 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fdb28011a60 con 0x7fdb380ca300 2023-10-19T14:22:35.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.550+0000 7fdb2f7fe640 1 -- 172.21.15.57:0/860952166 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb28011e70 con 0x7fdb380ca300 2023-10-19T14:22:35.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.550+0000 7fdb3d37d640 1 -- 172.21.15.57:0/860952166 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb380c8b20 con 0x7fdb380ca300 2023-10-19T14:22:35.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.550+0000 7fdb3d37d640 1 -- 172.21.15.57:0/860952166 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb3809b570 con 0x7fdb380ca300 2023-10-19T14:22:35.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.551+0000 7fdb3d37d640 1 -- 172.21.15.57:0/860952166 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb38069100 con 0x7fdb380ca300 2023-10-19T14:22:35.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.551+0000 7fdb2f7fe640 1 -- 172.21.15.57:0/860952166 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 155) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fdb2801d460 con 0x7fdb380ca300 2023-10-19T14:22:35.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.552+0000 7fdb2f7fe640 1 --2- 172.21.15.57:0/860952166 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fdb1806af60 0x7fdb1806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:35.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.552+0000 7fdb2f7fe640 1 -- 172.21.15.57:0/860952166 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(268..268 src has 1..268) v4 ==== 6108+0+0 (secure 0 0 0) 0x7fdb2808a550 con 0x7fdb380ca300 2023-10-19T14:22:35.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.552+0000 7fdb36ffd640 1 --2- 172.21.15.57:0/860952166 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fdb1806af60 0x7fdb1806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:35.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.553+0000 7fdb36ffd640 1 --2- 172.21.15.57:0/860952166 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fdb1806af60 0x7fdb1806d420 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fdb30005d00 tx=0x7fdb3000e4c0 comp rx=0 tx=0).ready entity=mgr.10520 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:35.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.556+0000 7fdb2f7fe640 1 -- 172.21.15.57:0/860952166 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fdb28055250 con 0x7fdb380ca300 2023-10-19T14:22:35.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.741+0000 7fdb3d37d640 1 -- 172.21.15.57:0/860952166 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "config reset", "num": 30} v 0) v1 -- 0x7fdb3815dd80 con 0x7fdb380ca300 2023-10-19T14:22:35.748 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.745+0000 7fdb2f7fe640 1 -- 172.21.15.57:0/860952166 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 30}]=0 v86) v1 ==== 71+0+0 (secure 0 0 0) 0x7fdb28058870 con 0x7fdb380ca300 2023-10-19T14:22:35.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.748+0000 7fdb3d37d640 1 -- 172.21.15.57:0/860952166 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fdb1806af60 msgr2=0x7fdb1806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:35.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.748+0000 7fdb3d37d640 1 --2- 172.21.15.57:0/860952166 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fdb1806af60 0x7fdb1806d420 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fdb30005d00 tx=0x7fdb3000e4c0 comp rx=0 tx=0).stop 2023-10-19T14:22:35.751 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.748+0000 7fdb3d37d640 1 -- 172.21.15.57:0/860952166 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdb380ca300 msgr2=0x7fdb380ca780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:35.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.748+0000 7fdb3d37d640 1 --2- 172.21.15.57:0/860952166 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdb380ca300 0x7fdb380ca780 secure :-1 s=READY pgs=3022 cs=0 l=1 rev1=1 crypto rx=0x7fdb380cfcc0 tx=0x7fdb28001c30 comp rx=0 tx=0).stop 2023-10-19T14:22:35.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.748+0000 7fdb3d37d640 1 -- 172.21.15.57:0/860952166 shutdown_connections 2023-10-19T14:22:35.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.748+0000 7fdb3d37d640 1 --2- 172.21.15.57:0/860952166 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fdb1806af60 0x7fdb1806d420 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:35.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.748+0000 7fdb3d37d640 1 --2- 172.21.15.57:0/860952166 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdb380ca300 0x7fdb380ca780 unknown :-1 s=CLOSED pgs=3022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:35.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.748+0000 7fdb3d37d640 1 --2- 172.21.15.57:0/860952166 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdb3809abb0 0x7fdb3809b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:35.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.748+0000 7fdb3d37d640 1 --2- 172.21.15.57:0/860952166 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fdb240047c0 0x7fdb380c9320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:35.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.748+0000 7fdb3d37d640 1 -- 172.21.15.57:0/860952166 >> 172.21.15.57:0/860952166 conn(0x7fdb3809a140 msgr2=0x7fdb3806b470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:35.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.748+0000 7fdb3d37d640 1 -- 172.21.15.57:0/860952166 shutdown_connections 2023-10-19T14:22:35.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.748+0000 7fdb3d37d640 1 -- 172.21.15.57:0/860952166 wait complete. 2023-10-19T14:22:35.761 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name' 2023-10-19T14:22:35.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.868+0000 7f9534051640 1 -- 172.21.15.57:0/1967982834 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f952c15e5d0 msgr2=0x7f952c0cdfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:35.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.868+0000 7f9534051640 1 --2- 172.21.15.57:0/1967982834 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f952c15e5d0 0x7f952c0cdfd0 secure :-1 s=READY pgs=3023 cs=0 l=1 rev1=1 crypto rx=0x7f952c0ce510 tx=0x7f951c02c250 comp rx=0 tx=0).stop 2023-10-19T14:22:35.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.868+0000 7f9534051640 1 -- 172.21.15.57:0/1967982834 shutdown_connections 2023-10-19T14:22:35.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.868+0000 7f9534051640 1 --2- 172.21.15.57:0/1967982834 >> v2:172.21.15.179:3300/0 conn(0x7f952c0cf710 0x7f952c0cb950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:35.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.868+0000 7f9534051640 1 --2- 172.21.15.57:0/1967982834 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f952c0ce980 0x7f952c0cf1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:35.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.868+0000 7f9534051640 1 --2- 172.21.15.57:0/1967982834 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f952c15e5d0 0x7f952c0cdfd0 unknown :-1 s=CLOSED pgs=3023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:35.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.868+0000 7f9534051640 1 -- 172.21.15.57:0/1967982834 >> 172.21.15.57:0/1967982834 conn(0x7f952c09a140 msgr2=0x7f952c0d0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:35.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.868+0000 7f9534051640 1 -- 172.21.15.57:0/1967982834 shutdown_connections 2023-10-19T14:22:35.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.868+0000 7f9534051640 1 -- 172.21.15.57:0/1967982834 wait complete. 2023-10-19T14:22:35.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.869+0000 7f9534051640 1 Processor -- start 2023-10-19T14:22:35.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.870+0000 7f9534051640 1 -- start start 2023-10-19T14:22:35.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.870+0000 7f9534051640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f952c0ce980 0x7f952c09e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:35.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.870+0000 7f9534051640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f952c0cf710 0x7f952c0cabb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:35.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.870+0000 7f9534051640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f952c15e5d0 0x7f952c09c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:35.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.870+0000 7f9534051640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f952c0c7790 con 0x7f952c0cf710 2023-10-19T14:22:35.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.870+0000 7f9534051640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f952c09eaf0 con 0x7f952c15e5d0 2023-10-19T14:22:35.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.870+0000 7f9534051640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f952c09ab50 con 0x7f952c0ce980 2023-10-19T14:22:35.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.870+0000 7f95315ec640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f952c0cf710 0x7f952c0cabb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:35.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.870+0000 7f95315ec640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f952c0cf710 0x7f952c0cabb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39058/0 (socket says 172.21.15.57:39058) 2023-10-19T14:22:35.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.870+0000 7f95315ec640 1 -- 172.21.15.57:0/3823358586 learned_addr learned my addr 172.21.15.57:0/3823358586 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:22:35.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.871+0000 7f9531ded640 1 --2- 172.21.15.57:0/3823358586 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f952c0ce980 0x7f952c09e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:35.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.871+0000 7f95325ee640 1 --2- 172.21.15.57:0/3823358586 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f952c15e5d0 0x7f952c09c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:35.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.871+0000 7f95315ec640 1 -- 172.21.15.57:0/3823358586 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f952c0ce980 msgr2=0x7f952c09e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:35.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.871+0000 7f95315ec640 1 --2- 172.21.15.57:0/3823358586 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f952c0ce980 0x7f952c09e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:35.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.871+0000 7f95315ec640 1 -- 172.21.15.57:0/3823358586 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f952c15e5d0 msgr2=0x7f952c09c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:35.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.871+0000 7f95315ec640 1 --2- 172.21.15.57:0/3823358586 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f952c15e5d0 0x7f952c09c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:35.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.871+0000 7f95315ec640 1 -- 172.21.15.57:0/3823358586 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f951c009900 con 0x7f952c0cf710 2023-10-19T14:22:35.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.871+0000 7f9531ded640 1 --2- 172.21.15.57:0/3823358586 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f952c0ce980 0x7f952c09e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:22:35.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.871+0000 7f95325ee640 1 --2- 172.21.15.57:0/3823358586 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f952c15e5d0 0x7f952c09c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:22:35.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.871+0000 7f95315ec640 1 --2- 172.21.15.57:0/3823358586 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f952c0cf710 0x7f952c0cabb0 secure :-1 s=READY pgs=3024 cs=0 l=1 rev1=1 crypto rx=0x7f952000b4a0 tx=0x7f952000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:35.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.871+0000 7f952a7fc640 1 -- 172.21.15.57:0/3823358586 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9520010610 con 0x7f952c0cf710 2023-10-19T14:22:35.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.871+0000 7f952a7fc640 1 -- 172.21.15.57:0/3823358586 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f9520010860 con 0x7f952c0cf710 2023-10-19T14:22:35.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.871+0000 7f9534051640 1 -- 172.21.15.57:0/3823358586 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f952c09ae30 con 0x7f952c0cf710 2023-10-19T14:22:35.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.871+0000 7f952a7fc640 1 -- 172.21.15.57:0/3823358586 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9520010c70 con 0x7f952c0cf710 2023-10-19T14:22:35.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.871+0000 7f9534051640 1 -- 172.21.15.57:0/3823358586 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f952c0cb0f0 con 0x7f952c0cf710 2023-10-19T14:22:35.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.872+0000 7f9534051640 1 -- 172.21.15.57:0/3823358586 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f952c09b630 con 0x7f952c0cf710 2023-10-19T14:22:35.885 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.872+0000 7f952a7fc640 1 -- 172.21.15.57:0/3823358586 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 155) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f9520017360 con 0x7f952c0cf710 2023-10-19T14:22:35.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.873+0000 7f952a7fc640 1 --2- 172.21.15.57:0/3823358586 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f951806af60 0x7f951806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:35.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.873+0000 7f952a7fc640 1 -- 172.21.15.57:0/3823358586 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(268..268 src has 1..268) v4 ==== 6108+0+0 (secure 0 0 0) 0x7f9520089780 con 0x7f952c0cf710 2023-10-19T14:22:35.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.873+0000 7f9531ded640 1 --2- 172.21.15.57:0/3823358586 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f951806af60 0x7f951806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:35.886 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.874+0000 7f9531ded640 1 --2- 172.21.15.57:0/3823358586 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f951806af60 0x7f951806d420 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f951c036000 tx=0x7f951c004ab0 comp rx=0 tx=0).ready entity=mgr.10520 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:35.887 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:35.877+0000 7f952a7fc640 1 -- 172.21.15.57:0/3823358586 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f952c09b630 con 0x7f952c0cf710 2023-10-19T14:22:36.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:36.121+0000 7f9534051640 1 -- 172.21.15.57:0/3823358586 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name"]} v 0) v1 -- 0x7f952c09b630 con 0x7f952c0cf710 2023-10-19T14:22:36.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:36.415+0000 7f952a7fc640 1 -- 172.21.15.57:0/3823358586 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name"]}]=0 v3253) v1 ==== 155+0+0 (secure 0 0 0) 0x7f952c09b630 con 0x7f952c0cf710 2023-10-19T14:22:36.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:36.418+0000 7f9534051640 1 -- 172.21.15.57:0/3823358586 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f951806af60 msgr2=0x7f951806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:36.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:36.418+0000 7f9534051640 1 --2- 172.21.15.57:0/3823358586 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f951806af60 0x7f951806d420 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f951c036000 tx=0x7f951c004ab0 comp rx=0 tx=0).stop 2023-10-19T14:22:36.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:36.418+0000 7f9534051640 1 -- 172.21.15.57:0/3823358586 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f952c0cf710 msgr2=0x7f952c0cabb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:36.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:36.418+0000 7f9534051640 1 --2- 172.21.15.57:0/3823358586 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f952c0cf710 0x7f952c0cabb0 secure :-1 s=READY pgs=3024 cs=0 l=1 rev1=1 crypto rx=0x7f952000b4a0 tx=0x7f952000b960 comp rx=0 tx=0).stop 2023-10-19T14:22:36.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:36.418+0000 7f9534051640 1 -- 172.21.15.57:0/3823358586 shutdown_connections 2023-10-19T14:22:36.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:36.418+0000 7f9534051640 1 --2- 172.21.15.57:0/3823358586 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f951806af60 0x7f951806d420 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:36.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:36.418+0000 7f9534051640 1 --2- 172.21.15.57:0/3823358586 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f952c15e5d0 0x7f952c09c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:36.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:36.418+0000 7f9534051640 1 --2- 172.21.15.57:0/3823358586 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f952c0cf710 0x7f952c0cabb0 unknown :-1 s=CLOSED pgs=3024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:36.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:36.418+0000 7f9534051640 1 --2- 172.21.15.57:0/3823358586 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f952c0ce980 0x7f952c09e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:36.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:36.418+0000 7f9534051640 1 -- 172.21.15.57:0/3823358586 >> 172.21.15.57:0/3823358586 conn(0x7f952c09a140 msgr2=0x7f952c15aec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:36.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:36.418+0000 7f9534051640 1 -- 172.21.15.57:0/3823358586 shutdown_connections 2023-10-19T14:22:36.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:36.418+0000 7f9534051640 1 -- 172.21.15.57:0/3823358586 wait complete. 2023-10-19T14:22:36.432 INFO:tasks.cephfs_test_runner:Test the correctness of snap directory name ... ok 2023-10-19T14:22:36.432 INFO:tasks.cephfs_test_runner: 2023-10-19T14:22:36.432 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2023-10-19T14:22:36.433 INFO:tasks.cephfs_test_runner:Ran 14 tests in 5849.490s 2023-10-19T14:22:36.433 INFO:tasks.cephfs_test_runner: 2023-10-19T14:22:36.433 INFO:tasks.cephfs_test_runner:OK 2023-10-19T14:22:36.434 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2023-10-19T14:22:36.444 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2023-10-19T14:22:36.486 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2023-10-19T14:22:36.486 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T14:22:36.487 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T14:22:36.503 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055614,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=358 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248260k,nr_inodes=812065,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-19T14:22:36.504 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-10-19T14:22:36.504 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T14:22:36.561 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055614,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=358 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248260k,nr_inodes=812065,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6496528k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-19T14:22:36.561 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-19T14:22:36.562 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T14:22:36.580 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054833,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=284 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3247632k,nr_inodes=811908,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-19T14:22:36.581 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-19T14:22:36.581 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-19T14:22:36.638 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054833,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=284 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3247632k,nr_inodes=811908,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6495272k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-19T14:22:36.639 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:22:36.639 DEBUG:teuthology.orchestra.run.smithi179:> ip netns list 2023-10-19T14:22:36.649 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:22:36.651 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:22:36.696 INFO:teuthology.orchestra.run.smithi179.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2023-10-19T14:22:36.696 INFO:teuthology.orchestra.run.smithi179.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-10-19T14:22:36.697 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:22:36.697 DEBUG:teuthology.orchestra.run.smithi179:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2023-10-19T14:22:36.765 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:22:36.765 DEBUG:teuthology.orchestra.run.smithi179:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2023-10-19T14:22:36.833 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:22:36.834 DEBUG:teuthology.orchestra.run.smithi179:> sudo ip link delete ceph-brx 2023-10-19T14:22:36.939 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:22:36.939 DEBUG:teuthology.orchestra.run.smithi057:> ip netns list 2023-10-19T14:22:36.956 INFO:teuthology.orchestra.run.smithi057.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-19T14:22:36.957 INFO:teuthology.orchestra.run.smithi057.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-19T14:22:36.957 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:22:36.957 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-19T14:22:37.030 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:22:37.030 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-19T14:22:37.103 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-19T14:22:37.104 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip link delete ceph-brx 2023-10-19T14:22:37.195 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-10-19T14:22:37.204 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-19T14:22:37.205 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-19T14:22:37.315 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.311+0000 7fe37c697640 1 -- 172.21.15.57:0/1589283409 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3740b1820 msgr2=0x7fe3740b1cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:37.315 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.311+0000 7fe37c697640 1 --2- 172.21.15.57:0/1589283409 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3740b1820 0x7fe3740b1cb0 secure :-1 s=READY pgs=3025 cs=0 l=1 rev1=1 crypto rx=0x7fe3740afaf0 tx=0x7fe36c02c250 comp rx=0 tx=0).stop 2023-10-19T14:22:37.315 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.312+0000 7fe37c697640 1 -- 172.21.15.57:0/1589283409 shutdown_connections 2023-10-19T14:22:37.315 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.312+0000 7fe37c697640 1 --2- 172.21.15.57:0/1589283409 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3740b1820 0x7fe3740b1cb0 unknown :-1 s=CLOSED pgs=3025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:37.316 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.312+0000 7fe37c697640 1 --2- 172.21.15.57:0/1589283409 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe37406f2a0 0x7fe3740b0c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:37.316 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.312+0000 7fe37c697640 1 --2- 172.21.15.57:0/1589283409 >> v2:172.21.15.179:3300/0 conn(0x7fe3741417e0 0x7fe374070a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:37.316 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.312+0000 7fe37c697640 1 -- 172.21.15.57:0/1589283409 >> 172.21.15.57:0/1589283409 conn(0x7fe374069810 msgr2=0x7fe374069c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:37.316 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.312+0000 7fe37c697640 1 -- 172.21.15.57:0/1589283409 shutdown_connections 2023-10-19T14:22:37.316 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.312+0000 7fe37c697640 1 -- 172.21.15.57:0/1589283409 wait complete. 2023-10-19T14:22:37.321 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.313+0000 7fe37c697640 1 Processor -- start 2023-10-19T14:22:37.321 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.313+0000 7fe37c697640 1 -- start start 2023-10-19T14:22:37.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.314+0000 7fe37c697640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe37406f2a0 0x7fe3740cbbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:37.322 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.314+0000 7fe37c697640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe3740b1820 0x7fe3740c8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:37.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.314+0000 7fe37c697640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe3741417e0 0x7fe374132bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:37.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.314+0000 7fe37c697640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3741366b0 con 0x7fe37406f2a0 2023-10-19T14:22:37.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.314+0000 7fe37c697640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe374135a30 con 0x7fe3741417e0 2023-10-19T14:22:37.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.314+0000 7fe37c697640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe374135d10 con 0x7fe3740b1820 2023-10-19T14:22:37.323 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.314+0000 7fe37a433640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe37406f2a0 0x7fe3740cbbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:37.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.314+0000 7fe37a433640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe37406f2a0 0x7fe3740cbbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59884/0 (socket says 172.21.15.57:59884) 2023-10-19T14:22:37.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.314+0000 7fe37a433640 1 -- 172.21.15.57:0/1060420086 learned_addr learned my addr 172.21.15.57:0/1060420086 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:22:37.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.314+0000 7fe37ac34640 1 --2- 172.21.15.57:0/1060420086 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe3741417e0 0x7fe374132bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:37.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.314+0000 7fe379c32640 1 --2- 172.21.15.57:0/1060420086 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe3740b1820 0x7fe3740c8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:37.324 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.314+0000 7fe37a433640 1 -- 172.21.15.57:0/1060420086 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe3740b1820 msgr2=0x7fe3740c8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:37.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.314+0000 7fe37a433640 1 --2- 172.21.15.57:0/1060420086 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe3740b1820 0x7fe3740c8a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:37.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.314+0000 7fe37a433640 1 -- 172.21.15.57:0/1060420086 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe3741417e0 msgr2=0x7fe374132bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:37.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.314+0000 7fe37a433640 1 --2- 172.21.15.57:0/1060420086 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe3741417e0 0x7fe374132bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:37.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.314+0000 7fe37a433640 1 -- 172.21.15.57:0/1060420086 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe36c009900 con 0x7fe37406f2a0 2023-10-19T14:22:37.325 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.314+0000 7fe37ac34640 1 --2- 172.21.15.57:0/1060420086 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe3741417e0 0x7fe374132bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:22:37.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.314+0000 7fe37a433640 1 --2- 172.21.15.57:0/1060420086 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe37406f2a0 0x7fe3740cbbb0 secure :-1 s=READY pgs=3026 cs=0 l=1 rev1=1 crypto rx=0x7fe37000d740 tx=0x7fe37000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:37.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.314+0000 7fe36affd640 1 -- 172.21.15.57:0/1060420086 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe370009c30 con 0x7fe37406f2a0 2023-10-19T14:22:37.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.314+0000 7fe36affd640 1 -- 172.21.15.57:0/1060420086 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fe370009e80 con 0x7fe37406f2a0 2023-10-19T14:22:37.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.314+0000 7fe37c697640 1 -- 172.21.15.57:0/1060420086 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe374135ff0 con 0x7fe37406f2a0 2023-10-19T14:22:37.326 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.314+0000 7fe36affd640 1 -- 172.21.15.57:0/1060420086 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3700106c0 con 0x7fe37406f2a0 2023-10-19T14:22:37.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.314+0000 7fe37c697640 1 -- 172.21.15.57:0/1060420086 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3740cc0f0 con 0x7fe37406f2a0 2023-10-19T14:22:37.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.315+0000 7fe37c697640 1 -- 172.21.15.57:0/1060420086 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3740cc2b0 con 0x7fe37406f2a0 2023-10-19T14:22:37.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.316+0000 7fe36affd640 1 -- 172.21.15.57:0/1060420086 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 155) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe370010820 con 0x7fe37406f2a0 2023-10-19T14:22:37.327 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.316+0000 7fe36affd640 1 --2- 172.21.15.57:0/1060420086 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fe35806af60 0x7fe35806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:37.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.316+0000 7fe36affd640 1 -- 172.21.15.57:0/1060420086 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(268..268 src has 1..268) v4 ==== 6108+0+0 (secure 0 0 0) 0x7fe370089f30 con 0x7fe37406f2a0 2023-10-19T14:22:37.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.316+0000 7fe379c32640 1 --2- 172.21.15.57:0/1060420086 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fe35806af60 0x7fe35806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:37.328 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.317+0000 7fe379c32640 1 --2- 172.21.15.57:0/1060420086 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fe35806af60 0x7fe35806d420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe374070eb0 tx=0x7fe364005c90 comp rx=0 tx=0).ready entity=mgr.10520 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:37.329 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.320+0000 7fe36affd640 1 -- 172.21.15.57:0/1060420086 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe370054c30 con 0x7fe37406f2a0 2023-10-19T14:22:37.495 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.492+0000 7fe37c697640 1 -- 172.21.15.57:0/1060420086 --> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe374069100 con 0x7fe35806af60 2023-10-19T14:22:37.498 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.496+0000 7fe36affd640 1 -- 172.21.15.57:0/1060420086 <== mgr.10520 v2:172.21.15.179:6832/892689897 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+50897 (secure 0 0 0) 0x7fe374069100 con 0x7fe35806af60 2023-10-19T14:22:37.498 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:22:37.499 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2023-10-19T14:22:37.500 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.497+0000 7fe37c697640 1 -- 172.21.15.57:0/1060420086 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fe35806af60 msgr2=0x7fe35806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:37.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.497+0000 7fe37c697640 1 --2- 172.21.15.57:0/1060420086 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fe35806af60 0x7fe35806d420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe374070eb0 tx=0x7fe364005c90 comp rx=0 tx=0).stop 2023-10-19T14:22:37.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.497+0000 7fe37c697640 1 -- 172.21.15.57:0/1060420086 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe37406f2a0 msgr2=0x7fe3740cbbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:37.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.497+0000 7fe37c697640 1 --2- 172.21.15.57:0/1060420086 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe37406f2a0 0x7fe3740cbbb0 secure :-1 s=READY pgs=3026 cs=0 l=1 rev1=1 crypto rx=0x7fe37000d740 tx=0x7fe37000dc00 comp rx=0 tx=0).stop 2023-10-19T14:22:37.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.497+0000 7fe37c697640 1 -- 172.21.15.57:0/1060420086 shutdown_connections 2023-10-19T14:22:37.501 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.497+0000 7fe37c697640 1 --2- 172.21.15.57:0/1060420086 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fe35806af60 0x7fe35806d420 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:37.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.497+0000 7fe37c697640 1 --2- 172.21.15.57:0/1060420086 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe3741417e0 0x7fe374132bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:37.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.497+0000 7fe37c697640 1 --2- 172.21.15.57:0/1060420086 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe3740b1820 0x7fe3740c8a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:37.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.497+0000 7fe37c697640 1 --2- 172.21.15.57:0/1060420086 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe37406f2a0 0x7fe3740cbbb0 unknown :-1 s=CLOSED pgs=3026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:37.502 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.497+0000 7fe37c697640 1 -- 172.21.15.57:0/1060420086 >> 172.21.15.57:0/1060420086 conn(0x7fe374069810 msgr2=0x7fe3740af180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:37.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.497+0000 7fe37c697640 1 -- 172.21.15.57:0/1060420086 shutdown_connections 2023-10-19T14:22:37.503 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.497+0000 7fe37c697640 1 -- 172.21.15.57:0/1060420086 wait complete. 2023-10-19T14:22:37.510 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2023-10-19T14:22:35.810738+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2731,"num_read_kb":5751,"num_write":697,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3246,"ondisk_log_size":3246,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":188,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":458380,"kb_used_data":28460,"kb_used_omap":12,"kb_used_meta":429811,"kb_avail":754516340,"statfs":{"total":773094113280,"available":772624732160,"internally_reserved":0,"allocated":29143040,"data_stored":16093848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":440127033},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-667824,"num_objects":-25,"num_object_clones":0,"num_object_copies":-50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-25,"num_whiteouts":0,"num_read":-221,"num_read_kb":-75,"num_write":-264,"num_write_kb":-2056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.997342"},"pg_stats":[{"pgid":"1.0","version":"265'3246","reported_seq":4336,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-19T14:22:34.079745+0000","last_change":"2023-10-19T12:44:08.636613+0000","last_active":"2023-10-19T14:22:34.079745+0000","last_peered":"2023-10-19T14:22:34.079745+0000","last_clean":"2023-10-19T14:22:34.079745+0000","last_became_active":"2023-10-19T12:44:08.635697+0000","last_became_peered":"2023-10-19T12:44:08.635697+0000","last_unstale":"2023-10-19T14:22:34.079745+0000","last_undegraded":"2023-10-19T14:22:34.079745+0000","last_fullsized":"2023-10-19T14:22:34.079745+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_clean_scrub_stamp":"2023-10-19T12:44:07.399738+0000","objects_scrubbed":0,"log_size":3246,"log_dups_size":0,"ondisk_log_size":3246,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:15:23.685485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2731,"num_read_kb":5751,"num_write":697,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2731,"num_read_kb":5751,"num_write":697,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3246,"ondisk_log_size":3246,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576026,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66996,"kb_used_data":3880,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94304844,"statfs":{"total":96636764160,"available":96568160256,"internally_reserved":0,"allocated":3973120,"data_stored":2337324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":64616907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:22:11 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.53000000000000003,"15min":0.57299999999999995},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.86199999999999999},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.55600000000000005,"15min":0.59399999999999997},"min":{"1min":0.38300000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.77400000000000002}]},{"osd":1,"last update":"Thu Oct 19 14:22:11 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.57799999999999996,"15min":0.59399999999999997},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.56200000000000006,"15min":0.58899999999999997},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.754,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.70699999999999996}]},{"osd":2,"last update":"Thu Oct 19 14:22:11 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.61399999999999999,"15min":0.63800000000000001},"min":{"1min":0.42599999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.77300000000000002,"5min":0.84799999999999998,"15min":0.85499999999999998},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.623,"15min":0.65400000000000003},"min":{"1min":0.41599999999999998,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.024,"5min":1.024,"15min":1.024},"last":0.84399999999999997}]},{"osd":3,"last update":"Thu Oct 19 14:22:11 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.59499999999999997,"15min":0.628},"min":{"1min":0.36399999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.86799999999999999},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.66400000000000003,"15min":0.68300000000000005},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.86799999999999999,"5min":0.873,"15min":0.89600000000000002},"last":0.81299999999999994}]},{"osd":4,"last update":"Thu Oct 19 14:22:11 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.51600000000000001,"15min":0.498},"min":{"1min":0.34100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.89600000000000002,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.54400000000000004,"15min":0.51600000000000001},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":1.032,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.56799999999999995}]},{"osd":5,"last update":"Thu Oct 19 14:22:11 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.54000000000000004,"15min":0.51400000000000001},"min":{"1min":0.35299999999999998,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.58599999999999997,"15min":0.55600000000000005},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":1.069,"5min":1.069,"15min":1.069},"last":0.60699999999999998}]},{"osd":6,"last update":"Thu Oct 19 14:22:11 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.59899999999999998,"15min":0.57799999999999996},"min":{"1min":0.45600000000000002,"5min":0.38500000000000001,"15min":0.35999999999999999},"max":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.60899999999999999,"15min":0.57799999999999996},"min":{"1min":0.375,"5min":0.35599999999999998,"15min":0.33500000000000002},"max":{"1min":1.1100000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.68899999999999995}]}]},{"osd":6,"up_from":13,"seq":55834576026,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61156,"kb_used_data":3608,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310684,"statfs":{"total":96636764160,"available":96574140416,"internally_reserved":0,"allocated":3694592,"data_stored":2067569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58915271},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:22:04 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.57399999999999995,"15min":0.60299999999999998},"min":{"1min":0.32500000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.70799999999999996,"5min":0.81200000000000006,"15min":0.85799999999999998},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.55700000000000005,"15min":0.60299999999999998},"min":{"1min":0.371,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.80600000000000005,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.46400000000000002}]},{"osd":1,"last update":"Thu Oct 19 14:22:04 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.59099999999999997,"15min":0.61499999999999999},"min":{"1min":0.41899999999999998,"5min":0.373,"15min":0.35799999999999998},"max":{"1min":0.81100000000000005,"5min":0.82799999999999996,"15min":1.165},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.59499999999999997,"15min":0.622},"min":{"1min":0.435,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.83099999999999996,"5min":0.83699999999999997,"15min":0.91300000000000003},"last":0.51400000000000001}]},{"osd":2,"last update":"Thu Oct 19 14:22:04 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.55900000000000005,"15min":0.59699999999999998},"min":{"1min":0.38300000000000001,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.72399999999999998,"5min":0.80100000000000005,"15min":0.873},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59699999999999998,"15min":0.621},"min":{"1min":0.39400000000000002,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.93100000000000005},"last":0.57599999999999996}]},{"osd":3,"last update":"Thu Oct 19 14:22:04 2023","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.622,"15min":0.64800000000000002},"min":{"1min":0.377,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.83799999999999997,"5min":0.875,"15min":0.875},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.63,"15min":0.65300000000000002},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.78100000000000003,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.71799999999999997}]},{"osd":4,"last update":"Thu Oct 19 14:22:04 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.503,"15min":0.48799999999999999},"min":{"1min":0.31,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.89300000000000002,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51600000000000001,"15min":0.50900000000000001},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.78700000000000003,"5min":0.84899999999999998,"15min":0.85699999999999998},"last":0.69599999999999995}]},{"osd":5,"last update":"Thu Oct 19 14:22:04 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.52000000000000002,"15min":0.505},"min":{"1min":0.33200000000000002,"5min":0.32700000000000001,"15min":0.32500000000000001},"max":{"1min":0.86799999999999999,"5min":1.115,"15min":1.115},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.57799999999999996,"15min":0.54800000000000004},"min":{"1min":0.33900000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.76000000000000001}]},{"osd":7,"last update":"Thu Oct 19 14:22:04 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.57299999999999995,"15min":0.55100000000000005},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.82599999999999996,"5min":0.89300000000000002,"15min":0.90100000000000002},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.58199999999999996,"15min":0.55700000000000005},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.82899999999999996,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.78100000000000003}]}]},{"osd":5,"up_from":13,"seq":55834576025,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66396,"kb_used_data":3280,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305444,"statfs":{"total":96636764160,"available":96568774656,"internally_reserved":0,"allocated":3358720,"data_stored":1732598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:21:50 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.57399999999999995,"15min":0.60699999999999998},"min":{"1min":0.38,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.69399999999999995,"5min":0.79700000000000004,"15min":0.86399999999999999},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.59599999999999997,"15min":0.63300000000000001},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.70499999999999996,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.34599999999999997}]},{"osd":1,"last update":"Thu Oct 19 14:21:50 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.59099999999999997,"15min":0.622},"min":{"1min":0.34599999999999997,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.877},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.61199999999999999,"15min":0.63700000000000001},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.877,"5min":0.877,"15min":0.88500000000000001},"last":0.54900000000000004}]},{"osd":2,"last update":"Thu Oct 19 14:21:50 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.58199999999999996,"15min":0.61399999999999999},"min":{"1min":0.378,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.625,"15min":0.65100000000000002},"min":{"1min":0.441,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.78900000000000003,"5min":0.83299999999999996,"15min":0.90800000000000003},"last":0.40899999999999997}]},{"osd":3,"last update":"Thu Oct 19 14:21:50 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.60499999999999998,"15min":0.63600000000000001},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.81100000000000005,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.626,"15min":0.65900000000000003},"min":{"1min":0.40000000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.90900000000000003},"last":0.50600000000000001}]},{"osd":4,"last update":"Thu Oct 19 14:21:50 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.48799999999999999,"15min":0.48699999999999999},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.51800000000000002,"15min":0.505},"min":{"1min":0.309,"5min":0.308,"15min":0.28599999999999998},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.82099999999999995},"last":0.36499999999999999}]},{"osd":6,"last update":"Thu Oct 19 14:21:50 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.51700000000000002,"15min":0.502},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.88800000000000001},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.54100000000000004,"15min":0.53900000000000003},"min":{"1min":0.36899999999999999,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.48199999999999998}]},{"osd":7,"last update":"Thu Oct 19 14:21:50 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.52200000000000002,"15min":0.51800000000000002},"min":{"1min":0.32700000000000001,"5min":0.312,"15min":0.312},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.57899999999999996,"15min":0.57199999999999995},"min":{"1min":0.38,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.42099999999999999}]}]},{"osd":4,"up_from":13,"seq":55834576025,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48368,"kb_used_data":3300,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323472,"statfs":{"total":96636764160,"available":96587235328,"internally_reserved":0,"allocated":3379200,"data_stored":1743746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:22:16 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.53900000000000003,"15min":0.56499999999999995},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.28599999999999998},"max":{"1min":0.69799999999999995,"5min":0.83799999999999997,"15min":0.879},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.505,"5min":0.55700000000000005,"15min":0.58799999999999997},"min":{"1min":0.33400000000000002,"5min":0.31,"15min":0.31},"max":{"1min":0.71699999999999997,"5min":0.82599999999999996,"15min":0.873},"last":0.47299999999999998}]},{"osd":1,"last update":"Thu Oct 19 14:22:16 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.58499999999999996,"15min":0.626},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.32600000000000001},"max":{"1min":0.70299999999999996,"5min":0.80400000000000005,"15min":0.92700000000000005},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.59299999999999997,"15min":0.63200000000000001},"min":{"1min":0.374,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.68500000000000005,"5min":0.90000000000000002,"15min":1.0940000000000001},"last":0.432}]},{"osd":2,"last update":"Thu Oct 19 14:22:16 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.61099999999999999,"15min":0.63800000000000001},"min":{"1min":0.39000000000000001,"5min":0.372,"15min":0.32600000000000001},"max":{"1min":0.73699999999999999,"5min":1.008,"15min":1.008},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.625,"15min":0.65900000000000003},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.749,"5min":0.98799999999999999,"15min":1.008},"last":0.45100000000000001}]},{"osd":3,"last update":"Thu Oct 19 14:22:16 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.57999999999999996,"15min":0.621},"min":{"1min":0.39000000000000001,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.755,"5min":0.83399999999999996,"15min":0.91900000000000004},"last":0.442},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.64400000000000002,"15min":0.65900000000000003},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.76900000000000002,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.48599999999999999}]},{"osd":5,"last update":"Thu Oct 19 14:22:16 2023","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.48999999999999999,"15min":0.47899999999999998},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.755,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.51200000000000001,"15min":0.501},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29199999999999998},"max":{"1min":0.74399999999999999,"5min":0.873,"15min":0.873},"last":0.32500000000000001}]},{"osd":6,"last update":"Thu Oct 19 14:22:16 2023","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.50600000000000001,"15min":0.49399999999999999},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.28199999999999997},"max":{"1min":0.74099999999999999,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.377},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54700000000000004,"15min":0.54400000000000004},"min":{"1min":0.371,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.79700000000000004,"5min":0.88,"15min":1.2969999999999999},"last":0.34200000000000003}]},{"osd":7,"last update":"Thu Oct 19 14:22:16 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.54000000000000004,"15min":0.54400000000000004},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.72699999999999998,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60699999999999998,"15min":0.59999999999999998},"min":{"1min":0.42499999999999999,"5min":0.32100000000000001,"15min":0.318},"max":{"1min":0.77700000000000002,"5min":0.99399999999999999,"15min":1.032},"last":0.41999999999999998}]}]},{"osd":3,"up_from":13,"seq":55834576024,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56992,"kb_used_data":3604,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314848,"statfs":{"total":96636764160,"available":96578404352,"internally_reserved":0,"allocated":3690496,"data_stored":2060866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54655432},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:21:53 2023","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.39900000000000002,"15min":0.41699999999999998},"min":{"1min":0.23899999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.54500000000000004,"5min":0.60599999999999998,"15min":0.70699999999999996},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.39400000000000002,"15min":0.42199999999999999},"min":{"1min":0.23899999999999999,"5min":0.217,"15min":0.217},"max":{"1min":0.51900000000000002,"5min":0.61899999999999999,"15min":0.752},"last":0.313}]},{"osd":1,"last update":"Thu Oct 19 14:21:53 2023","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.40699999999999997,"15min":0.42799999999999999},"min":{"1min":0.28100000000000003,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.71599999999999997,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.45500000000000002,"15min":0.47199999999999998},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26400000000000001},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.38900000000000001}]},{"osd":2,"last update":"Thu Oct 19 14:21:53 2023","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.46200000000000002,"15min":0.48099999999999998},"min":{"1min":0.34300000000000003,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.73099999999999998},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48499999999999999,"15min":0.504},"min":{"1min":0.30599999999999999,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":1.212,"5min":1.212,"15min":1.212},"last":0.33300000000000002}]},{"osd":4,"last update":"Thu Oct 19 14:21:53 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.67400000000000004,"15min":0.70099999999999996},"min":{"1min":0.44600000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.94199999999999995,"5min":1.018,"15min":1.157},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.629,"5min":0.68700000000000006,"15min":0.72299999999999998},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.89500000000000002,"5min":0.996,"15min":1.173},"last":0.44600000000000001}]},{"osd":5,"last update":"Thu Oct 19 14:21:53 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.67500000000000004,"15min":0.68600000000000005},"min":{"1min":0.50900000000000001,"5min":0.46600000000000003,"15min":0.41999999999999998},"max":{"1min":0.96799999999999997,"5min":0.97299999999999998,"15min":1.0149999999999999},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.66800000000000004,"15min":0.68600000000000005},"min":{"1min":0.48899999999999999,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.97699999999999998,"5min":0.99399999999999999,"15min":1.105},"last":0.64200000000000002}]},{"osd":6,"last update":"Thu Oct 19 14:21:53 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.65000000000000002,"15min":0.68600000000000005},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.93000000000000005,"5min":1.0009999999999999,"15min":1.0669999999999999},"last":0.497},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.68899999999999995,"15min":0.70999999999999996},"min":{"1min":0.505,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.079},"last":0.54800000000000004}]},{"osd":7,"last update":"Thu Oct 19 14:21:53 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67500000000000004,"15min":0.69799999999999995},"min":{"1min":0.46600000000000003,"5min":0.434,"15min":0.434},"max":{"1min":0.94999999999999996,"5min":0.96899999999999997,"15min":1.085},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.69799999999999995,"15min":0.72799999999999998},"min":{"1min":0.48799999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.95899999999999996,"5min":0.97199999999999998,"15min":1.1739999999999999},"last":0.53400000000000003}]}]},{"osd":2,"up_from":13,"seq":55834576027,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48368,"kb_used_data":3300,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323472,"statfs":{"total":96636764160,"available":96587235328,"internally_reserved":0,"allocated":3379200,"data_stored":1746956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.374,"15min":0.39600000000000002},"min":{"1min":0.25700000000000001,"5min":0.218,"15min":0.218},"max":{"1min":0.58799999999999997,"5min":0.66600000000000004,"15min":0.69699999999999995},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.41499999999999998,"15min":0.432},"min":{"1min":0.248,"5min":0.248,"15min":0.223},"max":{"1min":0.53100000000000003,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.39800000000000002}]},{"osd":1,"last update":"Thu Oct 19 14:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.38300000000000001,"15min":0.40600000000000003},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.65400000000000003},"last":0.316},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.44400000000000001,"15min":0.46000000000000002},"min":{"1min":0.28699999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.45100000000000001,"5min":1.742,"15min":1.742},"last":0.28699999999999998}]},{"osd":3,"last update":"Thu Oct 19 14:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.45500000000000002,"15min":0.47599999999999998},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.63300000000000001,"5min":0.69299999999999995,"15min":0.74399999999999999},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.47299999999999998,"15min":0.501},"min":{"1min":0.32700000000000001,"5min":0.32000000000000001,"15min":0.315},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.83599999999999997},"last":0.48399999999999999}]},{"osd":4,"last update":"Thu Oct 19 14:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.63900000000000001,"15min":0.66200000000000003},"min":{"1min":0.40699999999999997,"5min":0.39600000000000002,"15min":0.39500000000000002},"max":{"1min":0.74399999999999999,"5min":0.89200000000000002,"15min":0.998},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.65400000000000003,"15min":0.67600000000000005},"min":{"1min":0.41899999999999998,"5min":0.38900000000000001,"15min":0.35199999999999998},"max":{"1min":0.79000000000000004,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.73799999999999999}]},{"osd":5,"last update":"Thu Oct 19 14:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.67800000000000005,"15min":0.69299999999999995},"min":{"1min":0.443,"5min":0.443,"15min":0.40600000000000003},"max":{"1min":0.79300000000000004,"5min":1.4330000000000001,"15min":1.4330000000000001},"last":0.752},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.66900000000000004,"15min":0.68000000000000005},"min":{"1min":0.46899999999999997,"5min":0.45100000000000001,"15min":0.39300000000000002},"max":{"1min":0.77400000000000002,"5min":1.595,"15min":1.595},"last":0.68000000000000005}]},{"osd":6,"last update":"Thu Oct 19 14:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.67700000000000005,"15min":0.70199999999999996},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.80900000000000005,"5min":0.996,"15min":1.0189999999999999},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.629,"5min":0.69499999999999995,"15min":0.71399999999999997},"min":{"1min":0.438,"5min":0.433,"15min":0.40600000000000003},"max":{"1min":0.83399999999999996,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":0.69299999999999995}]},{"osd":7,"last update":"Thu Oct 19 14:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.68400000000000005,"15min":0.71299999999999997},"min":{"1min":0.47199999999999998,"5min":0.44800000000000001,"15min":0.41799999999999998},"max":{"1min":0.78800000000000003,"5min":1.0129999999999999,"15min":1.0289999999999999},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.71199999999999997,"15min":0.72599999999999998},"min":{"1min":0.42199999999999999,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.86699999999999999,"5min":1.53,"15min":1.53},"last":0.63100000000000001}]}]},{"osd":1,"up_from":13,"seq":55834576026,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48368,"kb_used_data":3300,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323472,"statfs":{"total":96636764160,"available":96587235328,"internally_reserved":0,"allocated":3379200,"data_stored":1746956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.38400000000000001,"15min":0.40300000000000002},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.46899999999999997,"5min":0.61599999999999999,"15min":0.67400000000000004},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.378,"5min":0.41999999999999998,"15min":0.45000000000000001},"min":{"1min":0.27700000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.51500000000000001,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.36799999999999999}]},{"osd":2,"last update":"Thu Oct 19 14:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.42399999999999999,"15min":0.438},"min":{"1min":0.26100000000000001,"5min":0.251,"15min":0.251},"max":{"1min":0.505,"5min":0.63100000000000001,"15min":0.77500000000000002},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.434,"5min":0.47699999999999998,"15min":0.48799999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.56699999999999995,"5min":0.68700000000000006,"15min":0.82799999999999996},"last":0.48999999999999999}]},{"osd":3,"last update":"Thu Oct 19 14:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.495,"15min":0.501},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.29499999999999998},"max":{"1min":0.59399999999999997,"5min":0.76000000000000001,"15min":0.78900000000000003},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.42899999999999999,"15min":0.44600000000000001},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.26100000000000001},"max":{"1min":0.59899999999999998,"5min":0.65200000000000002,"15min":0.69499999999999995},"last":0.41799999999999998}]},{"osd":4,"last update":"Thu Oct 19 14:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.64200000000000002,"15min":0.66700000000000004},"min":{"1min":0.42299999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.80400000000000005,"5min":0.95999999999999996,"15min":1.042},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.65400000000000003,"15min":0.66600000000000004},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.32300000000000001},"max":{"1min":0.80500000000000005,"5min":0.90800000000000003,"15min":1.1659999999999999},"last":0.54100000000000004}]},{"osd":5,"last update":"Thu Oct 19 14:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.66700000000000004,"15min":0.68200000000000005},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.81200000000000006,"5min":0.96199999999999997,"15min":1.3500000000000001},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.62,"5min":0.68899999999999995,"15min":0.70299999999999996},"min":{"1min":0.442,"5min":0.44,"15min":0.40500000000000003},"max":{"1min":0.82899999999999996,"5min":1.0009999999999999,"15min":1.121},"last":0.64800000000000002}]},{"osd":6,"last update":"Thu Oct 19 14:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.65400000000000003,"15min":0.67600000000000005},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.84999999999999998,"5min":1.0129999999999999,"15min":1.2310000000000001},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.69699999999999995,"15min":0.71499999999999997},"min":{"1min":0.46700000000000003,"5min":0.434,"15min":0.434},"max":{"1min":0.85799999999999998,"5min":1.0009999999999999,"15min":1.274},"last":0.47099999999999997}]},{"osd":7,"last update":"Thu Oct 19 14:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.70499999999999996,"15min":0.72199999999999998},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.86899999999999999,"5min":0.99099999999999999,"15min":1.075},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.68200000000000005,"15min":0.69599999999999995},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.84599999999999997,"5min":1.0169999999999999,"15min":1.117},"last":0.51300000000000001}]}]},{"osd":0,"up_from":13,"seq":55834576027,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61736,"kb_used_data":4188,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310104,"statfs":{"total":96636764160,"available":96573546496,"internally_reserved":0,"allocated":4288512,"data_stored":2657833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Oct 19 14:22:13 2023","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.39000000000000001,"15min":0.40600000000000003},"min":{"1min":0.254,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.47699999999999998,"5min":0.59999999999999998,"15min":0.624},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.40699999999999997,"15min":0.44900000000000001},"min":{"1min":0.252,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.81100000000000005},"last":0.63200000000000001}]},{"osd":2,"last update":"Thu Oct 19 14:22:13 2023","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.45700000000000002,"15min":0.47699999999999998},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.58699999999999997,"5min":0.73099999999999998,"15min":0.754},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.41399999999999998,"15min":0.439},"min":{"1min":0.316,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.54000000000000004,"5min":0.60199999999999998,"15min":0.69099999999999995},"last":0.66200000000000003}]},{"osd":3,"last update":"Thu Oct 19 14:22:13 2023","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.40300000000000002,"15min":0.433},"min":{"1min":0.28399999999999997,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.56899999999999995,"5min":0.58799999999999997,"15min":0.66800000000000004},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.46800000000000003,"15min":0.503},"min":{"1min":0.314,"5min":0.254,"15min":0.254},"max":{"1min":0.55600000000000005,"5min":0.78100000000000003,"15min":0.84599999999999997},"last":0.68000000000000005}]},{"osd":4,"last update":"Thu Oct 19 14:22:13 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.627,"15min":0.66600000000000004},"min":{"1min":0.437,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":1.1890000000000001},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.626,"15min":0.67000000000000004},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.42599999999999999},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.996},"last":0.93600000000000005}]},{"osd":5,"last update":"Thu Oct 19 14:22:13 2023","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.66100000000000003,"15min":0.70999999999999996},"min":{"1min":0.46500000000000002,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.86599999999999999,"5min":0.98499999999999999,"15min":1.1579999999999999},"last":0.89600000000000002},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.67400000000000004,"15min":0.71099999999999997},"min":{"1min":0.52700000000000002,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.91700000000000004,"5min":0.98299999999999998,"15min":1.2310000000000001},"last":0.98999999999999999}]},{"osd":6,"last update":"Thu Oct 19 14:22:13 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.65400000000000003,"15min":0.69599999999999995},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.95499999999999996,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.91600000000000004},{"interface":"front","average":{"1min":0.628,"5min":0.64100000000000001,"15min":0.68999999999999995},"min":{"1min":0.44600000000000001,"5min":0.441,"15min":0.441},"max":{"1min":0.86399999999999999,"5min":0.89900000000000002,"15min":1.01},"last":1.036}]},{"osd":7,"last update":"Thu Oct 19 14:22:13 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.65500000000000003,"15min":0.70499999999999996},"min":{"1min":0.52300000000000002,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.89500000000000002,"5min":0.93400000000000005,"15min":1.014},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.65800000000000003,"15min":0.69299999999999995},"min":{"1min":0.498,"5min":0.434,"15min":0.434},"max":{"1min":0.88400000000000001,"5min":0.93799999999999994,"15min":1.0349999999999999},"last":0.96199999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-19T14:22:37.513 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-19T14:22:37.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.619+0000 7fe24e0c2640 1 -- 172.21.15.57:0/1261466228 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe230003970 msgr2=0x7fe230003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:37.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.619+0000 7fe24e0c2640 1 --2- 172.21.15.57:0/1261466228 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe230003970 0x7fe230003d90 secure :-1 s=READY pgs=3028 cs=0 l=1 rev1=1 crypto rx=0x7fe238009b80 tx=0x7fe2300057f0 comp rx=0 tx=0).stop 2023-10-19T14:22:37.622 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.619+0000 7fe247fff640 1 -- 172.21.15.57:0/1261466228 reap_dead start 2023-10-19T14:22:37.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.619+0000 7fe24e0c2640 1 -- 172.21.15.57:0/1261466228 shutdown_connections 2023-10-19T14:22:37.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.619+0000 7fe24e0c2640 1 -- 172.21.15.57:0/1261466228 >> 172.21.15.57:0/1261466228 conn(0x7fe248070980 msgr2=0x7fe248069630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:37.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.619+0000 7fe24e0c2640 1 -- 172.21.15.57:0/1261466228 shutdown_connections 2023-10-19T14:22:37.623 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.619+0000 7fe24e0c2640 1 -- 172.21.15.57:0/1261466228 wait complete. 2023-10-19T14:22:37.625 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.620+0000 7fe24e0c2640 1 Processor -- start 2023-10-19T14:22:37.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.620+0000 7fe24e0c2640 1 -- start start 2023-10-19T14:22:37.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.621+0000 7fe24e0c2640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe230003970 0x7fe2481e2280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:37.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.621+0000 7fe24e0c2640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe248149db0 0x7fe24815ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:37.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.621+0000 7fe24e0c2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe24813cc80 0x7fe248152d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:37.626 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.621+0000 7fe24e0c2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe248159820 con 0x7fe24813cc80 2023-10-19T14:22:37.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.621+0000 7fe24e0c2640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2481e27c0 con 0x7fe230003970 2023-10-19T14:22:37.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.621+0000 7fe24e0c2640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe248159ac0 con 0x7fe248149db0 2023-10-19T14:22:37.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.621+0000 7fe247fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe24813cc80 0x7fe248152d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:37.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.621+0000 7fe247fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe24813cc80 0x7fe248152d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59910/0 (socket says 172.21.15.57:59910) 2023-10-19T14:22:37.627 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.621+0000 7fe247fff640 1 -- 172.21.15.57:0/4059791904 learned_addr learned my addr 172.21.15.57:0/4059791904 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:22:37.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.621+0000 7fe2477fe640 1 --2- 172.21.15.57:0/4059791904 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe230003970 0x7fe2481e2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:37.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.621+0000 7fe246ffd640 1 --2- 172.21.15.57:0/4059791904 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe248149db0 0x7fe24815ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:37.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.621+0000 7fe247fff640 1 -- 172.21.15.57:0/4059791904 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe248149db0 msgr2=0x7fe24815ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:37.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.621+0000 7fe247fff640 1 --2- 172.21.15.57:0/4059791904 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe248149db0 0x7fe24815ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:37.628 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.621+0000 7fe247fff640 1 -- 172.21.15.57:0/4059791904 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe230003970 msgr2=0x7fe2481e2280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:37.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.621+0000 7fe247fff640 1 --2- 172.21.15.57:0/4059791904 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe230003970 0x7fe2481e2280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:37.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.621+0000 7fe247fff640 1 -- 172.21.15.57:0/4059791904 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe248159da0 con 0x7fe24813cc80 2023-10-19T14:22:37.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.621+0000 7fe2477fe640 1 --2- 172.21.15.57:0/4059791904 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe230003970 0x7fe2481e2280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:22:37.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.621+0000 7fe247fff640 1 --2- 172.21.15.57:0/4059791904 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe24813cc80 0x7fe248152d60 secure :-1 s=READY pgs=3029 cs=0 l=1 rev1=1 crypto rx=0x7fe2480ca300 tx=0x7fe240006720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:37.629 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.622+0000 7fe22ffff640 1 -- 172.21.15.57:0/4059791904 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe24000fc90 con 0x7fe24813cc80 2023-10-19T14:22:37.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.622+0000 7fe22ffff640 1 -- 172.21.15.57:0/4059791904 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fe24000fdf0 con 0x7fe24813cc80 2023-10-19T14:22:37.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.622+0000 7fe24e0c2640 1 -- 172.21.15.57:0/4059791904 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe24815a080 con 0x7fe24813cc80 2023-10-19T14:22:37.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.622+0000 7fe22ffff640 1 -- 172.21.15.57:0/4059791904 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe240011690 con 0x7fe24813cc80 2023-10-19T14:22:37.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.622+0000 7fe24e0c2640 1 -- 172.21.15.57:0/4059791904 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe24815a640 con 0x7fe24813cc80 2023-10-19T14:22:37.630 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.623+0000 7fe24e0c2640 1 -- 172.21.15.57:0/4059791904 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe248066980 con 0x7fe24813cc80 2023-10-19T14:22:37.631 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.623+0000 7fe22ffff640 1 -- 172.21.15.57:0/4059791904 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 155) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe2400117f0 con 0x7fe24813cc80 2023-10-19T14:22:37.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.624+0000 7fe22ffff640 1 --2- 172.21.15.57:0/4059791904 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fe23406af90 0x7fe23406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:37.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.624+0000 7fe22ffff640 1 -- 172.21.15.57:0/4059791904 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(268..268 src has 1..268) v4 ==== 6108+0+0 (secure 0 0 0) 0x7fe24008a560 con 0x7fe24813cc80 2023-10-19T14:22:37.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.624+0000 7fe2477fe640 1 --2- 172.21.15.57:0/4059791904 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fe23406af90 0x7fe23406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:37.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.624+0000 7fe2477fe640 1 --2- 172.21.15.57:0/4059791904 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fe23406af90 0x7fe23406d450 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe238036000 tx=0x7fe238035040 comp rx=0 tx=0).ready entity=mgr.10520 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:37.632 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.628+0000 7fe22ffff640 1 -- 172.21.15.57:0/4059791904 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fe2400551c0 con 0x7fe24813cc80 2023-10-19T14:22:37.804 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.801+0000 7fe24e0c2640 1 -- 172.21.15.57:0/4059791904 --> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe24806cca0 con 0x7fe23406af90 2023-10-19T14:22:37.807 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.804+0000 7fe22ffff640 1 -- 172.21.15.57:0/4059791904 <== mgr.10520 v2:172.21.15.179:6832/892689897 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+50897 (secure 0 0 0) 0x7fe238047450 con 0x7fe23406af90 2023-10-19T14:22:37.807 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:22:37.808 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2023-10-19T14:22:37.809 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.806+0000 7fe24e0c2640 1 -- 172.21.15.57:0/4059791904 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fe23406af90 msgr2=0x7fe23406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:37.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.806+0000 7fe24e0c2640 1 --2- 172.21.15.57:0/4059791904 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fe23406af90 0x7fe23406d450 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe238036000 tx=0x7fe238035040 comp rx=0 tx=0).stop 2023-10-19T14:22:37.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.806+0000 7fe24e0c2640 1 -- 172.21.15.57:0/4059791904 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe24813cc80 msgr2=0x7fe248152d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:37.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.806+0000 7fe24e0c2640 1 --2- 172.21.15.57:0/4059791904 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe24813cc80 0x7fe248152d60 secure :-1 s=READY pgs=3029 cs=0 l=1 rev1=1 crypto rx=0x7fe2480ca300 tx=0x7fe240006720 comp rx=0 tx=0).stop 2023-10-19T14:22:37.810 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.806+0000 7fe24e0c2640 1 -- 172.21.15.57:0/4059791904 shutdown_connections 2023-10-19T14:22:37.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.806+0000 7fe24e0c2640 1 --2- 172.21.15.57:0/4059791904 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fe23406af90 0x7fe23406d450 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:37.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.806+0000 7fe24e0c2640 1 --2- 172.21.15.57:0/4059791904 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe24813cc80 0x7fe248152d60 unknown :-1 s=CLOSED pgs=3029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:37.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.806+0000 7fe24e0c2640 1 --2- 172.21.15.57:0/4059791904 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fe248149db0 0x7fe24815ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:37.811 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.806+0000 7fe24e0c2640 1 --2- 172.21.15.57:0/4059791904 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe230003970 0x7fe2481e2280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:37.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.806+0000 7fe24e0c2640 1 -- 172.21.15.57:0/4059791904 >> 172.21.15.57:0/4059791904 conn(0x7fe248070980 msgr2=0x7fe2480b1ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:37.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.806+0000 7fe24e0c2640 1 -- 172.21.15.57:0/4059791904 shutdown_connections 2023-10-19T14:22:37.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.806+0000 7fe24e0c2640 1 -- 172.21.15.57:0/4059791904 wait complete. 2023-10-19T14:22:37.819 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2023-10-19T14:22:35.810738+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2731,"num_read_kb":5751,"num_write":697,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3246,"ondisk_log_size":3246,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":188,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":458380,"kb_used_data":28460,"kb_used_omap":12,"kb_used_meta":429811,"kb_avail":754516340,"statfs":{"total":773094113280,"available":772624732160,"internally_reserved":0,"allocated":29143040,"data_stored":16093848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":440127033},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-667824,"num_objects":-25,"num_object_clones":0,"num_object_copies":-50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-25,"num_whiteouts":0,"num_read":-221,"num_read_kb":-75,"num_write":-264,"num_write_kb":-2056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.997342"},"pg_stats":[{"pgid":"1.0","version":"265'3246","reported_seq":4336,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-19T14:22:34.079745+0000","last_change":"2023-10-19T12:44:08.636613+0000","last_active":"2023-10-19T14:22:34.079745+0000","last_peered":"2023-10-19T14:22:34.079745+0000","last_clean":"2023-10-19T14:22:34.079745+0000","last_became_active":"2023-10-19T12:44:08.635697+0000","last_became_peered":"2023-10-19T12:44:08.635697+0000","last_unstale":"2023-10-19T14:22:34.079745+0000","last_undegraded":"2023-10-19T14:22:34.079745+0000","last_fullsized":"2023-10-19T14:22:34.079745+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_clean_scrub_stamp":"2023-10-19T12:44:07.399738+0000","objects_scrubbed":0,"log_size":3246,"log_dups_size":0,"ondisk_log_size":3246,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:15:23.685485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2731,"num_read_kb":5751,"num_write":697,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2731,"num_read_kb":5751,"num_write":697,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3246,"ondisk_log_size":3246,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576026,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66996,"kb_used_data":3880,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94304844,"statfs":{"total":96636764160,"available":96568160256,"internally_reserved":0,"allocated":3973120,"data_stored":2337324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":64616907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:22:11 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.53000000000000003,"15min":0.57299999999999995},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.86199999999999999},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.55600000000000005,"15min":0.59399999999999997},"min":{"1min":0.38300000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.77400000000000002}]},{"osd":1,"last update":"Thu Oct 19 14:22:11 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.57799999999999996,"15min":0.59399999999999997},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.56200000000000006,"15min":0.58899999999999997},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.754,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.70699999999999996}]},{"osd":2,"last update":"Thu Oct 19 14:22:11 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.61399999999999999,"15min":0.63800000000000001},"min":{"1min":0.42599999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.77300000000000002,"5min":0.84799999999999998,"15min":0.85499999999999998},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.623,"15min":0.65400000000000003},"min":{"1min":0.41599999999999998,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.024,"5min":1.024,"15min":1.024},"last":0.84399999999999997}]},{"osd":3,"last update":"Thu Oct 19 14:22:11 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.59499999999999997,"15min":0.628},"min":{"1min":0.36399999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.86799999999999999},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.66400000000000003,"15min":0.68300000000000005},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.86799999999999999,"5min":0.873,"15min":0.89600000000000002},"last":0.81299999999999994}]},{"osd":4,"last update":"Thu Oct 19 14:22:11 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.51600000000000001,"15min":0.498},"min":{"1min":0.34100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.89600000000000002,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.54400000000000004,"15min":0.51600000000000001},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":1.032,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.56799999999999995}]},{"osd":5,"last update":"Thu Oct 19 14:22:11 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.54000000000000004,"15min":0.51400000000000001},"min":{"1min":0.35299999999999998,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.58599999999999997,"15min":0.55600000000000005},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":1.069,"5min":1.069,"15min":1.069},"last":0.60699999999999998}]},{"osd":6,"last update":"Thu Oct 19 14:22:11 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.59899999999999998,"15min":0.57799999999999996},"min":{"1min":0.45600000000000002,"5min":0.38500000000000001,"15min":0.35999999999999999},"max":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.60899999999999999,"15min":0.57799999999999996},"min":{"1min":0.375,"5min":0.35599999999999998,"15min":0.33500000000000002},"max":{"1min":1.1100000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.68899999999999995}]}]},{"osd":6,"up_from":13,"seq":55834576026,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61156,"kb_used_data":3608,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310684,"statfs":{"total":96636764160,"available":96574140416,"internally_reserved":0,"allocated":3694592,"data_stored":2067569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58915271},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:22:04 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.57399999999999995,"15min":0.60299999999999998},"min":{"1min":0.32500000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.70799999999999996,"5min":0.81200000000000006,"15min":0.85799999999999998},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.55700000000000005,"15min":0.60299999999999998},"min":{"1min":0.371,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.80600000000000005,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.46400000000000002}]},{"osd":1,"last update":"Thu Oct 19 14:22:04 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.59099999999999997,"15min":0.61499999999999999},"min":{"1min":0.41899999999999998,"5min":0.373,"15min":0.35799999999999998},"max":{"1min":0.81100000000000005,"5min":0.82799999999999996,"15min":1.165},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.59499999999999997,"15min":0.622},"min":{"1min":0.435,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.83099999999999996,"5min":0.83699999999999997,"15min":0.91300000000000003},"last":0.51400000000000001}]},{"osd":2,"last update":"Thu Oct 19 14:22:04 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.55900000000000005,"15min":0.59699999999999998},"min":{"1min":0.38300000000000001,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.72399999999999998,"5min":0.80100000000000005,"15min":0.873},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59699999999999998,"15min":0.621},"min":{"1min":0.39400000000000002,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.93100000000000005},"last":0.57599999999999996}]},{"osd":3,"last update":"Thu Oct 19 14:22:04 2023","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.622,"15min":0.64800000000000002},"min":{"1min":0.377,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.83799999999999997,"5min":0.875,"15min":0.875},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.63,"15min":0.65300000000000002},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.78100000000000003,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.71799999999999997}]},{"osd":4,"last update":"Thu Oct 19 14:22:04 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.503,"15min":0.48799999999999999},"min":{"1min":0.31,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.89300000000000002,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51600000000000001,"15min":0.50900000000000001},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.78700000000000003,"5min":0.84899999999999998,"15min":0.85699999999999998},"last":0.69599999999999995}]},{"osd":5,"last update":"Thu Oct 19 14:22:04 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.52000000000000002,"15min":0.505},"min":{"1min":0.33200000000000002,"5min":0.32700000000000001,"15min":0.32500000000000001},"max":{"1min":0.86799999999999999,"5min":1.115,"15min":1.115},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.57799999999999996,"15min":0.54800000000000004},"min":{"1min":0.33900000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.76000000000000001}]},{"osd":7,"last update":"Thu Oct 19 14:22:04 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.57299999999999995,"15min":0.55100000000000005},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.82599999999999996,"5min":0.89300000000000002,"15min":0.90100000000000002},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.58199999999999996,"15min":0.55700000000000005},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.82899999999999996,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.78100000000000003}]}]},{"osd":5,"up_from":13,"seq":55834576025,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66396,"kb_used_data":3280,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305444,"statfs":{"total":96636764160,"available":96568774656,"internally_reserved":0,"allocated":3358720,"data_stored":1732598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:21:50 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.57399999999999995,"15min":0.60699999999999998},"min":{"1min":0.38,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.69399999999999995,"5min":0.79700000000000004,"15min":0.86399999999999999},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.59599999999999997,"15min":0.63300000000000001},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.70499999999999996,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.34599999999999997}]},{"osd":1,"last update":"Thu Oct 19 14:21:50 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.59099999999999997,"15min":0.622},"min":{"1min":0.34599999999999997,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.877},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.61199999999999999,"15min":0.63700000000000001},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.877,"5min":0.877,"15min":0.88500000000000001},"last":0.54900000000000004}]},{"osd":2,"last update":"Thu Oct 19 14:21:50 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.58199999999999996,"15min":0.61399999999999999},"min":{"1min":0.378,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.625,"15min":0.65100000000000002},"min":{"1min":0.441,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.78900000000000003,"5min":0.83299999999999996,"15min":0.90800000000000003},"last":0.40899999999999997}]},{"osd":3,"last update":"Thu Oct 19 14:21:50 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.60499999999999998,"15min":0.63600000000000001},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.81100000000000005,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.626,"15min":0.65900000000000003},"min":{"1min":0.40000000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.90900000000000003},"last":0.50600000000000001}]},{"osd":4,"last update":"Thu Oct 19 14:21:50 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.48799999999999999,"15min":0.48699999999999999},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.51800000000000002,"15min":0.505},"min":{"1min":0.309,"5min":0.308,"15min":0.28599999999999998},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.82099999999999995},"last":0.36499999999999999}]},{"osd":6,"last update":"Thu Oct 19 14:21:50 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.51700000000000002,"15min":0.502},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.88800000000000001},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.54100000000000004,"15min":0.53900000000000003},"min":{"1min":0.36899999999999999,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.48199999999999998}]},{"osd":7,"last update":"Thu Oct 19 14:21:50 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.52200000000000002,"15min":0.51800000000000002},"min":{"1min":0.32700000000000001,"5min":0.312,"15min":0.312},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.57899999999999996,"15min":0.57199999999999995},"min":{"1min":0.38,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.42099999999999999}]}]},{"osd":4,"up_from":13,"seq":55834576025,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48368,"kb_used_data":3300,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323472,"statfs":{"total":96636764160,"available":96587235328,"internally_reserved":0,"allocated":3379200,"data_stored":1743746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:22:16 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.53900000000000003,"15min":0.56499999999999995},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.28599999999999998},"max":{"1min":0.69799999999999995,"5min":0.83799999999999997,"15min":0.879},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.505,"5min":0.55700000000000005,"15min":0.58799999999999997},"min":{"1min":0.33400000000000002,"5min":0.31,"15min":0.31},"max":{"1min":0.71699999999999997,"5min":0.82599999999999996,"15min":0.873},"last":0.47299999999999998}]},{"osd":1,"last update":"Thu Oct 19 14:22:16 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.58499999999999996,"15min":0.626},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.32600000000000001},"max":{"1min":0.70299999999999996,"5min":0.80400000000000005,"15min":0.92700000000000005},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.59299999999999997,"15min":0.63200000000000001},"min":{"1min":0.374,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.68500000000000005,"5min":0.90000000000000002,"15min":1.0940000000000001},"last":0.432}]},{"osd":2,"last update":"Thu Oct 19 14:22:16 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.61099999999999999,"15min":0.63800000000000001},"min":{"1min":0.39000000000000001,"5min":0.372,"15min":0.32600000000000001},"max":{"1min":0.73699999999999999,"5min":1.008,"15min":1.008},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.625,"15min":0.65900000000000003},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.749,"5min":0.98799999999999999,"15min":1.008},"last":0.45100000000000001}]},{"osd":3,"last update":"Thu Oct 19 14:22:16 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.57999999999999996,"15min":0.621},"min":{"1min":0.39000000000000001,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.755,"5min":0.83399999999999996,"15min":0.91900000000000004},"last":0.442},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.64400000000000002,"15min":0.65900000000000003},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.76900000000000002,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.48599999999999999}]},{"osd":5,"last update":"Thu Oct 19 14:22:16 2023","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.48999999999999999,"15min":0.47899999999999998},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.755,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.51200000000000001,"15min":0.501},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29199999999999998},"max":{"1min":0.74399999999999999,"5min":0.873,"15min":0.873},"last":0.32500000000000001}]},{"osd":6,"last update":"Thu Oct 19 14:22:16 2023","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.50600000000000001,"15min":0.49399999999999999},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.28199999999999997},"max":{"1min":0.74099999999999999,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.377},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54700000000000004,"15min":0.54400000000000004},"min":{"1min":0.371,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.79700000000000004,"5min":0.88,"15min":1.2969999999999999},"last":0.34200000000000003}]},{"osd":7,"last update":"Thu Oct 19 14:22:16 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.54000000000000004,"15min":0.54400000000000004},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.72699999999999998,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60699999999999998,"15min":0.59999999999999998},"min":{"1min":0.42499999999999999,"5min":0.32100000000000001,"15min":0.318},"max":{"1min":0.77700000000000002,"5min":0.99399999999999999,"15min":1.032},"last":0.41999999999999998}]}]},{"osd":3,"up_from":13,"seq":55834576024,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56992,"kb_used_data":3604,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314848,"statfs":{"total":96636764160,"available":96578404352,"internally_reserved":0,"allocated":3690496,"data_stored":2060866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54655432},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:21:53 2023","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.39900000000000002,"15min":0.41699999999999998},"min":{"1min":0.23899999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.54500000000000004,"5min":0.60599999999999998,"15min":0.70699999999999996},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.39400000000000002,"15min":0.42199999999999999},"min":{"1min":0.23899999999999999,"5min":0.217,"15min":0.217},"max":{"1min":0.51900000000000002,"5min":0.61899999999999999,"15min":0.752},"last":0.313}]},{"osd":1,"last update":"Thu Oct 19 14:21:53 2023","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.40699999999999997,"15min":0.42799999999999999},"min":{"1min":0.28100000000000003,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.71599999999999997,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.45500000000000002,"15min":0.47199999999999998},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26400000000000001},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.38900000000000001}]},{"osd":2,"last update":"Thu Oct 19 14:21:53 2023","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.46200000000000002,"15min":0.48099999999999998},"min":{"1min":0.34300000000000003,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.73099999999999998},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48499999999999999,"15min":0.504},"min":{"1min":0.30599999999999999,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":1.212,"5min":1.212,"15min":1.212},"last":0.33300000000000002}]},{"osd":4,"last update":"Thu Oct 19 14:21:53 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.67400000000000004,"15min":0.70099999999999996},"min":{"1min":0.44600000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.94199999999999995,"5min":1.018,"15min":1.157},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.629,"5min":0.68700000000000006,"15min":0.72299999999999998},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.89500000000000002,"5min":0.996,"15min":1.173},"last":0.44600000000000001}]},{"osd":5,"last update":"Thu Oct 19 14:21:53 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.67500000000000004,"15min":0.68600000000000005},"min":{"1min":0.50900000000000001,"5min":0.46600000000000003,"15min":0.41999999999999998},"max":{"1min":0.96799999999999997,"5min":0.97299999999999998,"15min":1.0149999999999999},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.66800000000000004,"15min":0.68600000000000005},"min":{"1min":0.48899999999999999,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.97699999999999998,"5min":0.99399999999999999,"15min":1.105},"last":0.64200000000000002}]},{"osd":6,"last update":"Thu Oct 19 14:21:53 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.65000000000000002,"15min":0.68600000000000005},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.93000000000000005,"5min":1.0009999999999999,"15min":1.0669999999999999},"last":0.497},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.68899999999999995,"15min":0.70999999999999996},"min":{"1min":0.505,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.079},"last":0.54800000000000004}]},{"osd":7,"last update":"Thu Oct 19 14:21:53 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67500000000000004,"15min":0.69799999999999995},"min":{"1min":0.46600000000000003,"5min":0.434,"15min":0.434},"max":{"1min":0.94999999999999996,"5min":0.96899999999999997,"15min":1.085},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.69799999999999995,"15min":0.72799999999999998},"min":{"1min":0.48799999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.95899999999999996,"5min":0.97199999999999998,"15min":1.1739999999999999},"last":0.53400000000000003}]}]},{"osd":2,"up_from":13,"seq":55834576027,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48368,"kb_used_data":3300,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323472,"statfs":{"total":96636764160,"available":96587235328,"internally_reserved":0,"allocated":3379200,"data_stored":1746956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.374,"15min":0.39600000000000002},"min":{"1min":0.25700000000000001,"5min":0.218,"15min":0.218},"max":{"1min":0.58799999999999997,"5min":0.66600000000000004,"15min":0.69699999999999995},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.41499999999999998,"15min":0.432},"min":{"1min":0.248,"5min":0.248,"15min":0.223},"max":{"1min":0.53100000000000003,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.39800000000000002}]},{"osd":1,"last update":"Thu Oct 19 14:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.38300000000000001,"15min":0.40600000000000003},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.65400000000000003},"last":0.316},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.44400000000000001,"15min":0.46000000000000002},"min":{"1min":0.28699999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.45100000000000001,"5min":1.742,"15min":1.742},"last":0.28699999999999998}]},{"osd":3,"last update":"Thu Oct 19 14:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.45500000000000002,"15min":0.47599999999999998},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.63300000000000001,"5min":0.69299999999999995,"15min":0.74399999999999999},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.47299999999999998,"15min":0.501},"min":{"1min":0.32700000000000001,"5min":0.32000000000000001,"15min":0.315},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.83599999999999997},"last":0.48399999999999999}]},{"osd":4,"last update":"Thu Oct 19 14:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.63900000000000001,"15min":0.66200000000000003},"min":{"1min":0.40699999999999997,"5min":0.39600000000000002,"15min":0.39500000000000002},"max":{"1min":0.74399999999999999,"5min":0.89200000000000002,"15min":0.998},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.65400000000000003,"15min":0.67600000000000005},"min":{"1min":0.41899999999999998,"5min":0.38900000000000001,"15min":0.35199999999999998},"max":{"1min":0.79000000000000004,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.73799999999999999}]},{"osd":5,"last update":"Thu Oct 19 14:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.67800000000000005,"15min":0.69299999999999995},"min":{"1min":0.443,"5min":0.443,"15min":0.40600000000000003},"max":{"1min":0.79300000000000004,"5min":1.4330000000000001,"15min":1.4330000000000001},"last":0.752},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.66900000000000004,"15min":0.68000000000000005},"min":{"1min":0.46899999999999997,"5min":0.45100000000000001,"15min":0.39300000000000002},"max":{"1min":0.77400000000000002,"5min":1.595,"15min":1.595},"last":0.68000000000000005}]},{"osd":6,"last update":"Thu Oct 19 14:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.67700000000000005,"15min":0.70199999999999996},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.80900000000000005,"5min":0.996,"15min":1.0189999999999999},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.629,"5min":0.69499999999999995,"15min":0.71399999999999997},"min":{"1min":0.438,"5min":0.433,"15min":0.40600000000000003},"max":{"1min":0.83399999999999996,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":0.69299999999999995}]},{"osd":7,"last update":"Thu Oct 19 14:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.68400000000000005,"15min":0.71299999999999997},"min":{"1min":0.47199999999999998,"5min":0.44800000000000001,"15min":0.41799999999999998},"max":{"1min":0.78800000000000003,"5min":1.0129999999999999,"15min":1.0289999999999999},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.71199999999999997,"15min":0.72599999999999998},"min":{"1min":0.42199999999999999,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.86699999999999999,"5min":1.53,"15min":1.53},"last":0.63100000000000001}]}]},{"osd":1,"up_from":13,"seq":55834576026,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48368,"kb_used_data":3300,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323472,"statfs":{"total":96636764160,"available":96587235328,"internally_reserved":0,"allocated":3379200,"data_stored":1746956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.38400000000000001,"15min":0.40300000000000002},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.46899999999999997,"5min":0.61599999999999999,"15min":0.67400000000000004},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.378,"5min":0.41999999999999998,"15min":0.45000000000000001},"min":{"1min":0.27700000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.51500000000000001,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.36799999999999999}]},{"osd":2,"last update":"Thu Oct 19 14:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.42399999999999999,"15min":0.438},"min":{"1min":0.26100000000000001,"5min":0.251,"15min":0.251},"max":{"1min":0.505,"5min":0.63100000000000001,"15min":0.77500000000000002},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.434,"5min":0.47699999999999998,"15min":0.48799999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.56699999999999995,"5min":0.68700000000000006,"15min":0.82799999999999996},"last":0.48999999999999999}]},{"osd":3,"last update":"Thu Oct 19 14:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.495,"15min":0.501},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.29499999999999998},"max":{"1min":0.59399999999999997,"5min":0.76000000000000001,"15min":0.78900000000000003},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.42899999999999999,"15min":0.44600000000000001},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.26100000000000001},"max":{"1min":0.59899999999999998,"5min":0.65200000000000002,"15min":0.69499999999999995},"last":0.41799999999999998}]},{"osd":4,"last update":"Thu Oct 19 14:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.64200000000000002,"15min":0.66700000000000004},"min":{"1min":0.42299999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.80400000000000005,"5min":0.95999999999999996,"15min":1.042},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.65400000000000003,"15min":0.66600000000000004},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.32300000000000001},"max":{"1min":0.80500000000000005,"5min":0.90800000000000003,"15min":1.1659999999999999},"last":0.54100000000000004}]},{"osd":5,"last update":"Thu Oct 19 14:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.66700000000000004,"15min":0.68200000000000005},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.81200000000000006,"5min":0.96199999999999997,"15min":1.3500000000000001},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.62,"5min":0.68899999999999995,"15min":0.70299999999999996},"min":{"1min":0.442,"5min":0.44,"15min":0.40500000000000003},"max":{"1min":0.82899999999999996,"5min":1.0009999999999999,"15min":1.121},"last":0.64800000000000002}]},{"osd":6,"last update":"Thu Oct 19 14:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.65400000000000003,"15min":0.67600000000000005},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.84999999999999998,"5min":1.0129999999999999,"15min":1.2310000000000001},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.69699999999999995,"15min":0.71499999999999997},"min":{"1min":0.46700000000000003,"5min":0.434,"15min":0.434},"max":{"1min":0.85799999999999998,"5min":1.0009999999999999,"15min":1.274},"last":0.47099999999999997}]},{"osd":7,"last update":"Thu Oct 19 14:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.70499999999999996,"15min":0.72199999999999998},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.86899999999999999,"5min":0.99099999999999999,"15min":1.075},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.68200000000000005,"15min":0.69599999999999995},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.84599999999999997,"5min":1.0169999999999999,"15min":1.117},"last":0.51300000000000001}]}]},{"osd":0,"up_from":13,"seq":55834576027,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61736,"kb_used_data":4188,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310104,"statfs":{"total":96636764160,"available":96573546496,"internally_reserved":0,"allocated":4288512,"data_stored":2657833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Oct 19 14:22:13 2023","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.39000000000000001,"15min":0.40600000000000003},"min":{"1min":0.254,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.47699999999999998,"5min":0.59999999999999998,"15min":0.624},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.40699999999999997,"15min":0.44900000000000001},"min":{"1min":0.252,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.81100000000000005},"last":0.63200000000000001}]},{"osd":2,"last update":"Thu Oct 19 14:22:13 2023","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.45700000000000002,"15min":0.47699999999999998},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.58699999999999997,"5min":0.73099999999999998,"15min":0.754},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.41399999999999998,"15min":0.439},"min":{"1min":0.316,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.54000000000000004,"5min":0.60199999999999998,"15min":0.69099999999999995},"last":0.66200000000000003}]},{"osd":3,"last update":"Thu Oct 19 14:22:13 2023","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.40300000000000002,"15min":0.433},"min":{"1min":0.28399999999999997,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.56899999999999995,"5min":0.58799999999999997,"15min":0.66800000000000004},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.46800000000000003,"15min":0.503},"min":{"1min":0.314,"5min":0.254,"15min":0.254},"max":{"1min":0.55600000000000005,"5min":0.78100000000000003,"15min":0.84599999999999997},"last":0.68000000000000005}]},{"osd":4,"last update":"Thu Oct 19 14:22:13 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.627,"15min":0.66600000000000004},"min":{"1min":0.437,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":1.1890000000000001},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.626,"15min":0.67000000000000004},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.42599999999999999},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.996},"last":0.93600000000000005}]},{"osd":5,"last update":"Thu Oct 19 14:22:13 2023","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.66100000000000003,"15min":0.70999999999999996},"min":{"1min":0.46500000000000002,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.86599999999999999,"5min":0.98499999999999999,"15min":1.1579999999999999},"last":0.89600000000000002},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.67400000000000004,"15min":0.71099999999999997},"min":{"1min":0.52700000000000002,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.91700000000000004,"5min":0.98299999999999998,"15min":1.2310000000000001},"last":0.98999999999999999}]},{"osd":6,"last update":"Thu Oct 19 14:22:13 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.65400000000000003,"15min":0.69599999999999995},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.95499999999999996,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.91600000000000004},{"interface":"front","average":{"1min":0.628,"5min":0.64100000000000001,"15min":0.68999999999999995},"min":{"1min":0.44600000000000001,"5min":0.441,"15min":0.441},"max":{"1min":0.86399999999999999,"5min":0.89900000000000002,"15min":1.01},"last":1.036}]},{"osd":7,"last update":"Thu Oct 19 14:22:13 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.65500000000000003,"15min":0.70499999999999996},"min":{"1min":0.52300000000000002,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.89500000000000002,"5min":0.93400000000000005,"15min":1.014},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.65800000000000003,"15min":0.69299999999999995},"min":{"1min":0.498,"5min":0.434,"15min":0.434},"max":{"1min":0.88400000000000001,"5min":0.93799999999999994,"15min":1.0349999999999999},"last":0.96199999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-19T14:22:37.821 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-19T14:22:37.822 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-19T14:22:37.932 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.929+0000 7fd48d226640 1 -- 172.21.15.57:0/1412225943 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd474003970 msgr2=0x7fd474003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:37.932 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.929+0000 7fd48d226640 1 --2- 172.21.15.57:0/1412225943 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd474003970 0x7fd474003d90 secure :-1 s=READY pgs=3031 cs=0 l=1 rev1=1 crypto rx=0x7fd480009b50 tx=0x7fd4740057f0 comp rx=0 tx=0).stop 2023-10-19T14:22:37.932 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.929+0000 7fd48d226640 1 -- 172.21.15.57:0/1412225943 shutdown_connections 2023-10-19T14:22:37.932 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.929+0000 7fd48d226640 1 --2- 172.21.15.57:0/1412225943 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd474003970 0x7fd474003d90 unknown :-1 s=CLOSED pgs=3031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:37.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.929+0000 7fd48d226640 1 -- 172.21.15.57:0/1412225943 >> 172.21.15.57:0/1412225943 conn(0x7fd488069810 msgr2=0x7fd488069c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:37.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.929+0000 7fd48d226640 1 -- 172.21.15.57:0/1412225943 shutdown_connections 2023-10-19T14:22:37.933 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.929+0000 7fd48d226640 1 -- 172.21.15.57:0/1412225943 wait complete. 2023-10-19T14:22:37.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.930+0000 7fd48d226640 1 Processor -- start 2023-10-19T14:22:37.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.930+0000 7fd48d226640 1 -- start start 2023-10-19T14:22:37.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.931+0000 7fd48d226640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd474003970 0x7fd4880cc1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:37.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.931+0000 7fd48d226640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd4881d12e0 0x7fd4881d1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:37.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.931+0000 7fd48d226640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd4880c64b0 0x7fd4880c6930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:37.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.931+0000 7fd48d226640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4881cfad0 con 0x7fd474003970 2023-10-19T14:22:37.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.931+0000 7fd48d226640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4881d2740 con 0x7fd4880c64b0 2023-10-19T14:22:37.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.931+0000 7fd48d226640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd48813eee0 con 0x7fd4881d12e0 2023-10-19T14:22:37.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.931+0000 7fd486d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd474003970 0x7fd4880cc1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:37.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.931+0000 7fd486d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd474003970 0x7fd4880cc1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59934/0 (socket says 172.21.15.57:59934) 2023-10-19T14:22:37.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.931+0000 7fd486d9d640 1 -- 172.21.15.57:0/202346907 learned_addr learned my addr 172.21.15.57:0/202346907 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:22:37.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.931+0000 7fd48759e640 1 --2- 172.21.15.57:0/202346907 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd4880c64b0 0x7fd4880c6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:37.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.931+0000 7fd48659c640 1 --2- 172.21.15.57:0/202346907 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd4881d12e0 0x7fd4881d1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:37.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.931+0000 7fd486d9d640 1 -- 172.21.15.57:0/202346907 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd4881d12e0 msgr2=0x7fd4881d1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:37.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.931+0000 7fd486d9d640 1 --2- 172.21.15.57:0/202346907 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd4881d12e0 0x7fd4881d1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:37.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.931+0000 7fd486d9d640 1 -- 172.21.15.57:0/202346907 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd4880c64b0 msgr2=0x7fd4880c6930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:37.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.931+0000 7fd486d9d640 1 --2- 172.21.15.57:0/202346907 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd4880c64b0 0x7fd4880c6930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:37.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.931+0000 7fd486d9d640 1 -- 172.21.15.57:0/202346907 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4881d2010 con 0x7fd474003970 2023-10-19T14:22:37.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.932+0000 7fd486d9d640 1 --2- 172.21.15.57:0/202346907 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd474003970 0x7fd4880cc1b0 secure :-1 s=READY pgs=3032 cs=0 l=1 rev1=1 crypto rx=0x7fd48002cb70 tx=0x7fd480033c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:37.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.932+0000 7fd47f7fe640 1 -- 172.21.15.57:0/202346907 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd480004240 con 0x7fd474003970 2023-10-19T14:22:37.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.932+0000 7fd47f7fe640 1 -- 172.21.15.57:0/202346907 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fd4800043a0 con 0x7fd474003970 2023-10-19T14:22:37.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.932+0000 7fd48d226640 1 -- 172.21.15.57:0/202346907 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4881cf410 con 0x7fd474003970 2023-10-19T14:22:37.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.932+0000 7fd47f7fe640 1 -- 172.21.15.57:0/202346907 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd480003850 con 0x7fd474003970 2023-10-19T14:22:37.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.932+0000 7fd48d226640 1 -- 172.21.15.57:0/202346907 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4881d1ca0 con 0x7fd474003970 2023-10-19T14:22:37.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.933+0000 7fd48d226640 1 -- 172.21.15.57:0/202346907 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4881cfd40 con 0x7fd474003970 2023-10-19T14:22:37.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.933+0000 7fd47f7fe640 1 -- 172.21.15.57:0/202346907 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 155) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd4800039b0 con 0x7fd474003970 2023-10-19T14:22:37.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.934+0000 7fd47f7fe640 1 --2- 172.21.15.57:0/202346907 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fd46806af90 0x7fd46806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:37.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.934+0000 7fd47f7fe640 1 -- 172.21.15.57:0/202346907 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(268..268 src has 1..268) v4 ==== 6108+0+0 (secure 0 0 0) 0x7fd4800ac4d0 con 0x7fd474003970 2023-10-19T14:22:37.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.934+0000 7fd48659c640 1 --2- 172.21.15.57:0/202346907 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fd46806af90 0x7fd46806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:37.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.935+0000 7fd48659c640 1 --2- 172.21.15.57:0/202346907 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fd46806af90 0x7fd46806d450 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fd4880ccbf0 tx=0x7fd470000ed0 comp rx=0 tx=0).ready entity=mgr.10520 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:37.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:37.938+0000 7fd47f7fe640 1 -- 172.21.15.57:0/202346907 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd480077280 con 0x7fd474003970 2023-10-19T14:22:38.113 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:38.110+0000 7fd48d226640 1 -- 172.21.15.57:0/202346907 --> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd4880723c0 con 0x7fd46806af90 2023-10-19T14:22:38.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:38.113+0000 7fd47f7fe640 1 -- 172.21.15.57:0/202346907 <== mgr.10520 v2:172.21.15.179:6832/892689897 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+50921 (secure 0 0 0) 0x7fd4880723c0 con 0x7fd46806af90 2023-10-19T14:22:38.116 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:22:38.117 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2023-10-19T14:22:38.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:38.115+0000 7fd48d226640 1 -- 172.21.15.57:0/202346907 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fd46806af90 msgr2=0x7fd46806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:38.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:38.115+0000 7fd48d226640 1 --2- 172.21.15.57:0/202346907 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fd46806af90 0x7fd46806d450 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fd4880ccbf0 tx=0x7fd470000ed0 comp rx=0 tx=0).stop 2023-10-19T14:22:38.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:38.115+0000 7fd48d226640 1 -- 172.21.15.57:0/202346907 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd474003970 msgr2=0x7fd4880cc1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:38.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:38.115+0000 7fd48d226640 1 --2- 172.21.15.57:0/202346907 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd474003970 0x7fd4880cc1b0 secure :-1 s=READY pgs=3032 cs=0 l=1 rev1=1 crypto rx=0x7fd48002cb70 tx=0x7fd480033c10 comp rx=0 tx=0).stop 2023-10-19T14:22:38.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:38.115+0000 7fd48d226640 1 -- 172.21.15.57:0/202346907 shutdown_connections 2023-10-19T14:22:38.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:38.115+0000 7fd48d226640 1 --2- 172.21.15.57:0/202346907 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fd46806af90 0x7fd46806d450 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:38.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:38.115+0000 7fd48d226640 1 --2- 172.21.15.57:0/202346907 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd4880c64b0 0x7fd4880c6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:38.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:38.115+0000 7fd48d226640 1 --2- 172.21.15.57:0/202346907 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd4881d12e0 0x7fd4881d1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:38.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:38.115+0000 7fd48d226640 1 --2- 172.21.15.57:0/202346907 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd474003970 0x7fd4880cc1b0 unknown :-1 s=CLOSED pgs=3032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:38.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:38.115+0000 7fd48d226640 1 -- 172.21.15.57:0/202346907 >> 172.21.15.57:0/202346907 conn(0x7fd488069810 msgr2=0x7fd48806c910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:38.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:38.115+0000 7fd48d226640 1 -- 172.21.15.57:0/202346907 shutdown_connections 2023-10-19T14:22:38.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:38.115+0000 7fd48d226640 1 -- 172.21.15.57:0/202346907 wait complete. 2023-10-19T14:22:38.128 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2023-10-19T14:22:37.811125+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2731,"num_read_kb":5751,"num_write":697,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3246,"ondisk_log_size":3246,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":180,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":5,"kb":754974720,"kb_used":458128,"kb_used_data":28204,"kb_used_omap":12,"kb_used_meta":429811,"kb_avail":754516592,"statfs":{"total":773094113280,"available":772624990208,"internally_reserved":0,"allocated":28880896,"data_stored":15808272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":440127033},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-667824,"num_objects":-25,"num_object_clones":0,"num_object_copies":-50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-25,"num_whiteouts":0,"num_read":-221,"num_read_kb":-75,"num_write":-264,"num_write_kb":-2056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001085"},"pg_stats":[{"pgid":"1.0","version":"265'3246","reported_seq":4336,"reported_epoch":266,"state":"active+clean","last_fresh":"2023-10-19T14:22:34.079745+0000","last_change":"2023-10-19T12:44:08.636613+0000","last_active":"2023-10-19T14:22:34.079745+0000","last_peered":"2023-10-19T14:22:34.079745+0000","last_clean":"2023-10-19T14:22:34.079745+0000","last_became_active":"2023-10-19T12:44:08.635697+0000","last_became_peered":"2023-10-19T12:44:08.635697+0000","last_unstale":"2023-10-19T14:22:34.079745+0000","last_undegraded":"2023-10-19T14:22:34.079745+0000","last_fullsized":"2023-10-19T14:22:34.079745+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_clean_scrub_stamp":"2023-10-19T12:44:07.399738+0000","objects_scrubbed":0,"log_size":3246,"log_dups_size":0,"ondisk_log_size":3246,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:15:23.685485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2731,"num_read_kb":5751,"num_write":697,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2731,"num_read_kb":5751,"num_write":697,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3246,"ondisk_log_size":3246,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576026,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66996,"kb_used_data":3880,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94304844,"statfs":{"total":96636764160,"available":96568160256,"internally_reserved":0,"allocated":3973120,"data_stored":2337324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":64616907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:22:11 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.53000000000000003,"15min":0.57299999999999995},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.86199999999999999},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.55600000000000005,"15min":0.59399999999999997},"min":{"1min":0.38300000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.77400000000000002}]},{"osd":1,"last update":"Thu Oct 19 14:22:11 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.57799999999999996,"15min":0.59399999999999997},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.56200000000000006,"15min":0.58899999999999997},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.754,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.70699999999999996}]},{"osd":2,"last update":"Thu Oct 19 14:22:11 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.61399999999999999,"15min":0.63800000000000001},"min":{"1min":0.42599999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.77300000000000002,"5min":0.84799999999999998,"15min":0.85499999999999998},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.623,"15min":0.65400000000000003},"min":{"1min":0.41599999999999998,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.024,"5min":1.024,"15min":1.024},"last":0.84399999999999997}]},{"osd":3,"last update":"Thu Oct 19 14:22:11 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.59499999999999997,"15min":0.628},"min":{"1min":0.36399999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.86799999999999999},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.66400000000000003,"15min":0.68300000000000005},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.86799999999999999,"5min":0.873,"15min":0.89600000000000002},"last":0.81299999999999994}]},{"osd":4,"last update":"Thu Oct 19 14:22:11 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.51600000000000001,"15min":0.498},"min":{"1min":0.34100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.89600000000000002,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.54400000000000004,"15min":0.51600000000000001},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":1.032,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.56799999999999995}]},{"osd":5,"last update":"Thu Oct 19 14:22:11 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.54000000000000004,"15min":0.51400000000000001},"min":{"1min":0.35299999999999998,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.58599999999999997,"15min":0.55600000000000005},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":1.069,"5min":1.069,"15min":1.069},"last":0.60699999999999998}]},{"osd":6,"last update":"Thu Oct 19 14:22:11 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.59899999999999998,"15min":0.57799999999999996},"min":{"1min":0.45600000000000002,"5min":0.38500000000000001,"15min":0.35999999999999999},"max":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.60899999999999999,"15min":0.57799999999999996},"min":{"1min":0.375,"5min":0.35599999999999998,"15min":0.33500000000000002},"max":{"1min":1.1100000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.68899999999999995}]}]},{"osd":6,"up_from":13,"seq":55834576026,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61156,"kb_used_data":3608,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310684,"statfs":{"total":96636764160,"available":96574140416,"internally_reserved":0,"allocated":3694592,"data_stored":2067569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58915271},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:22:04 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.57399999999999995,"15min":0.60299999999999998},"min":{"1min":0.32500000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.70799999999999996,"5min":0.81200000000000006,"15min":0.85799999999999998},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.55700000000000005,"15min":0.60299999999999998},"min":{"1min":0.371,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.80600000000000005,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.46400000000000002}]},{"osd":1,"last update":"Thu Oct 19 14:22:04 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.59099999999999997,"15min":0.61499999999999999},"min":{"1min":0.41899999999999998,"5min":0.373,"15min":0.35799999999999998},"max":{"1min":0.81100000000000005,"5min":0.82799999999999996,"15min":1.165},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.59499999999999997,"15min":0.622},"min":{"1min":0.435,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.83099999999999996,"5min":0.83699999999999997,"15min":0.91300000000000003},"last":0.51400000000000001}]},{"osd":2,"last update":"Thu Oct 19 14:22:04 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.55900000000000005,"15min":0.59699999999999998},"min":{"1min":0.38300000000000001,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.72399999999999998,"5min":0.80100000000000005,"15min":0.873},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59699999999999998,"15min":0.621},"min":{"1min":0.39400000000000002,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.93100000000000005},"last":0.57599999999999996}]},{"osd":3,"last update":"Thu Oct 19 14:22:04 2023","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.622,"15min":0.64800000000000002},"min":{"1min":0.377,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.83799999999999997,"5min":0.875,"15min":0.875},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.63,"15min":0.65300000000000002},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.78100000000000003,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.71799999999999997}]},{"osd":4,"last update":"Thu Oct 19 14:22:04 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.503,"15min":0.48799999999999999},"min":{"1min":0.31,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.89300000000000002,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51600000000000001,"15min":0.50900000000000001},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.78700000000000003,"5min":0.84899999999999998,"15min":0.85699999999999998},"last":0.69599999999999995}]},{"osd":5,"last update":"Thu Oct 19 14:22:04 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.52000000000000002,"15min":0.505},"min":{"1min":0.33200000000000002,"5min":0.32700000000000001,"15min":0.32500000000000001},"max":{"1min":0.86799999999999999,"5min":1.115,"15min":1.115},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.57799999999999996,"15min":0.54800000000000004},"min":{"1min":0.33900000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.76000000000000001}]},{"osd":7,"last update":"Thu Oct 19 14:22:04 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.57299999999999995,"15min":0.55100000000000005},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.82599999999999996,"5min":0.89300000000000002,"15min":0.90100000000000002},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.58199999999999996,"15min":0.55700000000000005},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.82899999999999996,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.78100000000000003}]}]},{"osd":5,"up_from":13,"seq":55834576026,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":66432,"kb_used_data":3316,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305408,"statfs":{"total":96636764160,"available":96568737792,"internally_reserved":0,"allocated":3395584,"data_stored":1753686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:21:50 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.57399999999999995,"15min":0.60699999999999998},"min":{"1min":0.38,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.69399999999999995,"5min":0.79700000000000004,"15min":0.86399999999999999},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.59599999999999997,"15min":0.63300000000000001},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.70499999999999996,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.45100000000000001}]},{"osd":1,"last update":"Thu Oct 19 14:21:50 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.59099999999999997,"15min":0.622},"min":{"1min":0.34599999999999997,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.877},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.61199999999999999,"15min":0.63700000000000001},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.877,"5min":0.877,"15min":0.88500000000000001},"last":0.70799999999999996}]},{"osd":2,"last update":"Thu Oct 19 14:21:50 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.58199999999999996,"15min":0.61399999999999999},"min":{"1min":0.378,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.625,"15min":0.65100000000000002},"min":{"1min":0.441,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.78900000000000003,"5min":0.83299999999999996,"15min":0.90800000000000003},"last":0.65500000000000003}]},{"osd":3,"last update":"Thu Oct 19 14:21:50 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.60499999999999998,"15min":0.63600000000000001},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.81100000000000005,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.626,"15min":0.65900000000000003},"min":{"1min":0.40000000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.90900000000000003},"last":0.623}]},{"osd":4,"last update":"Thu Oct 19 14:21:50 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.48799999999999999,"15min":0.48699999999999999},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.51800000000000002,"15min":0.505},"min":{"1min":0.309,"5min":0.308,"15min":0.28599999999999998},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.82099999999999995},"last":0.42199999999999999}]},{"osd":6,"last update":"Thu Oct 19 14:21:50 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.51700000000000002,"15min":0.502},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.88800000000000001},"last":0.75},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.54100000000000004,"15min":0.53900000000000003},"min":{"1min":0.36899999999999999,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.67100000000000004}]},{"osd":7,"last update":"Thu Oct 19 14:21:50 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.52200000000000002,"15min":0.51800000000000002},"min":{"1min":0.32700000000000001,"5min":0.312,"15min":0.312},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.57899999999999996,"15min":0.57199999999999995},"min":{"1min":0.38,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.55600000000000005}]}]},{"osd":4,"up_from":13,"seq":55834576026,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48380,"kb_used_data":3312,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323460,"statfs":{"total":96636764160,"available":96587223040,"internally_reserved":0,"allocated":3391488,"data_stored":1750992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:22:16 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.53900000000000003,"15min":0.56499999999999995},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.28599999999999998},"max":{"1min":0.69799999999999995,"5min":0.83799999999999997,"15min":0.879},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.505,"5min":0.55700000000000005,"15min":0.58799999999999997},"min":{"1min":0.33400000000000002,"5min":0.31,"15min":0.31},"max":{"1min":0.71699999999999997,"5min":0.82599999999999996,"15min":0.873},"last":0.64700000000000002}]},{"osd":1,"last update":"Thu Oct 19 14:22:16 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.58499999999999996,"15min":0.626},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.32600000000000001},"max":{"1min":0.70299999999999996,"5min":0.80400000000000005,"15min":0.92700000000000005},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.59299999999999997,"15min":0.63200000000000001},"min":{"1min":0.374,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.68500000000000005,"5min":0.90000000000000002,"15min":1.0940000000000001},"last":0.60599999999999998}]},{"osd":2,"last update":"Thu Oct 19 14:22:16 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.61099999999999999,"15min":0.63800000000000001},"min":{"1min":0.39000000000000001,"5min":0.372,"15min":0.32600000000000001},"max":{"1min":0.73699999999999999,"5min":1.008,"15min":1.008},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.625,"15min":0.65900000000000003},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.749,"5min":0.98799999999999999,"15min":1.008},"last":0.51900000000000002}]},{"osd":3,"last update":"Thu Oct 19 14:22:16 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.57999999999999996,"15min":0.621},"min":{"1min":0.39000000000000001,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.755,"5min":0.83399999999999996,"15min":0.91900000000000004},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.64400000000000002,"15min":0.65900000000000003},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.76900000000000002,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.629}]},{"osd":5,"last update":"Thu Oct 19 14:22:16 2023","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.48999999999999999,"15min":0.47899999999999998},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.755,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.51200000000000001,"15min":0.501},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29199999999999998},"max":{"1min":0.74399999999999999,"5min":0.873,"15min":0.873},"last":0.54500000000000004}]},{"osd":6,"last update":"Thu Oct 19 14:22:16 2023","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.50600000000000001,"15min":0.49399999999999999},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.28199999999999997},"max":{"1min":0.74099999999999999,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54700000000000004,"15min":0.54400000000000004},"min":{"1min":0.371,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.79700000000000004,"5min":0.88,"15min":1.2969999999999999},"last":0.66700000000000004}]},{"osd":7,"last update":"Thu Oct 19 14:22:16 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.54000000000000004,"15min":0.54400000000000004},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.72699999999999998,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60699999999999998,"15min":0.59999999999999998},"min":{"1min":0.42499999999999999,"5min":0.32100000000000001,"15min":0.318},"max":{"1min":0.77700000000000002,"5min":0.99399999999999999,"15min":1.032},"last":0.69199999999999995}]}]},{"osd":3,"up_from":13,"seq":55834576025,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":56692,"kb_used_data":3300,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315148,"statfs":{"total":96636764160,"available":96578711552,"internally_reserved":0,"allocated":3379200,"data_stored":1746956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54655432},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:21:53 2023","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.39900000000000002,"15min":0.41699999999999998},"min":{"1min":0.23899999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.54500000000000004,"5min":0.60599999999999998,"15min":0.70699999999999996},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.39400000000000002,"15min":0.42199999999999999},"min":{"1min":0.23899999999999999,"5min":0.217,"15min":0.217},"max":{"1min":0.51900000000000002,"5min":0.61899999999999999,"15min":0.752},"last":0.39800000000000002}]},{"osd":1,"last update":"Thu Oct 19 14:21:53 2023","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.40699999999999997,"15min":0.42799999999999999},"min":{"1min":0.28100000000000003,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.71599999999999997,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.45500000000000002,"15min":0.47199999999999998},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26400000000000001},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.379}]},{"osd":2,"last update":"Thu Oct 19 14:21:53 2023","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.46200000000000002,"15min":0.48099999999999998},"min":{"1min":0.34300000000000003,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.73099999999999998},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48499999999999999,"15min":0.504},"min":{"1min":0.30599999999999999,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":1.212,"5min":1.212,"15min":1.212},"last":0.44800000000000001}]},{"osd":4,"last update":"Thu Oct 19 14:21:53 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.67400000000000004,"15min":0.70099999999999996},"min":{"1min":0.44600000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.94199999999999995,"5min":1.018,"15min":1.157},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.629,"5min":0.68700000000000006,"15min":0.72299999999999998},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.89500000000000002,"5min":0.996,"15min":1.173},"last":0.64000000000000001}]},{"osd":5,"last update":"Thu Oct 19 14:21:53 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.67500000000000004,"15min":0.68600000000000005},"min":{"1min":0.50900000000000001,"5min":0.46600000000000003,"15min":0.41999999999999998},"max":{"1min":0.96799999999999997,"5min":0.97299999999999998,"15min":1.0149999999999999},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.66800000000000004,"15min":0.68600000000000005},"min":{"1min":0.48899999999999999,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.97699999999999998,"5min":0.99399999999999999,"15min":1.105},"last":0.70299999999999996}]},{"osd":6,"last update":"Thu Oct 19 14:21:53 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.65000000000000002,"15min":0.68600000000000005},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.93000000000000005,"5min":1.0009999999999999,"15min":1.0669999999999999},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.68899999999999995,"15min":0.70999999999999996},"min":{"1min":0.505,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.079},"last":0.68000000000000005}]},{"osd":7,"last update":"Thu Oct 19 14:21:53 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67500000000000004,"15min":0.69799999999999995},"min":{"1min":0.46600000000000003,"5min":0.434,"15min":0.434},"max":{"1min":0.94999999999999996,"5min":0.96899999999999997,"15min":1.085},"last":0.755},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.69799999999999995,"15min":0.72799999999999998},"min":{"1min":0.48799999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.95899999999999996,"5min":0.97199999999999998,"15min":1.1739999999999999},"last":0.71699999999999997}]}]},{"osd":2,"up_from":13,"seq":55834576027,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48368,"kb_used_data":3300,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323472,"statfs":{"total":96636764160,"available":96587235328,"internally_reserved":0,"allocated":3379200,"data_stored":1746956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.374,"15min":0.39600000000000002},"min":{"1min":0.25700000000000001,"5min":0.218,"15min":0.218},"max":{"1min":0.58799999999999997,"5min":0.66600000000000004,"15min":0.69699999999999995},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.41499999999999998,"15min":0.432},"min":{"1min":0.248,"5min":0.248,"15min":0.223},"max":{"1min":0.53100000000000003,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.39800000000000002}]},{"osd":1,"last update":"Thu Oct 19 14:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.38300000000000001,"15min":0.40600000000000003},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.65400000000000003},"last":0.316},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.44400000000000001,"15min":0.46000000000000002},"min":{"1min":0.28699999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.45100000000000001,"5min":1.742,"15min":1.742},"last":0.28699999999999998}]},{"osd":3,"last update":"Thu Oct 19 14:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.45500000000000002,"15min":0.47599999999999998},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.63300000000000001,"5min":0.69299999999999995,"15min":0.74399999999999999},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.47299999999999998,"15min":0.501},"min":{"1min":0.32700000000000001,"5min":0.32000000000000001,"15min":0.315},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.83599999999999997},"last":0.48399999999999999}]},{"osd":4,"last update":"Thu Oct 19 14:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.63900000000000001,"15min":0.66200000000000003},"min":{"1min":0.40699999999999997,"5min":0.39600000000000002,"15min":0.39500000000000002},"max":{"1min":0.74399999999999999,"5min":0.89200000000000002,"15min":0.998},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.65400000000000003,"15min":0.67600000000000005},"min":{"1min":0.41899999999999998,"5min":0.38900000000000001,"15min":0.35199999999999998},"max":{"1min":0.79000000000000004,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.73799999999999999}]},{"osd":5,"last update":"Thu Oct 19 14:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.67800000000000005,"15min":0.69299999999999995},"min":{"1min":0.443,"5min":0.443,"15min":0.40600000000000003},"max":{"1min":0.79300000000000004,"5min":1.4330000000000001,"15min":1.4330000000000001},"last":0.752},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.66900000000000004,"15min":0.68000000000000005},"min":{"1min":0.46899999999999997,"5min":0.45100000000000001,"15min":0.39300000000000002},"max":{"1min":0.77400000000000002,"5min":1.595,"15min":1.595},"last":0.68000000000000005}]},{"osd":6,"last update":"Thu Oct 19 14:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.67700000000000005,"15min":0.70199999999999996},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.80900000000000005,"5min":0.996,"15min":1.0189999999999999},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.629,"5min":0.69499999999999995,"15min":0.71399999999999997},"min":{"1min":0.438,"5min":0.433,"15min":0.40600000000000003},"max":{"1min":0.83399999999999996,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":0.69299999999999995}]},{"osd":7,"last update":"Thu Oct 19 14:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.68400000000000005,"15min":0.71299999999999997},"min":{"1min":0.47199999999999998,"5min":0.44800000000000001,"15min":0.41799999999999998},"max":{"1min":0.78800000000000003,"5min":1.0129999999999999,"15min":1.0289999999999999},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.71199999999999997,"15min":0.72599999999999998},"min":{"1min":0.42199999999999999,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.86699999999999999,"5min":1.53,"15min":1.53},"last":0.63100000000000001}]}]},{"osd":1,"up_from":13,"seq":55834576026,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48368,"kb_used_data":3300,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323472,"statfs":{"total":96636764160,"available":96587235328,"internally_reserved":0,"allocated":3379200,"data_stored":1746956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.38400000000000001,"15min":0.40300000000000002},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.46899999999999997,"5min":0.61599999999999999,"15min":0.67400000000000004},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.378,"5min":0.41999999999999998,"15min":0.45000000000000001},"min":{"1min":0.27700000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.51500000000000001,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.36799999999999999}]},{"osd":2,"last update":"Thu Oct 19 14:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.42399999999999999,"15min":0.438},"min":{"1min":0.26100000000000001,"5min":0.251,"15min":0.251},"max":{"1min":0.505,"5min":0.63100000000000001,"15min":0.77500000000000002},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.434,"5min":0.47699999999999998,"15min":0.48799999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.56699999999999995,"5min":0.68700000000000006,"15min":0.82799999999999996},"last":0.48999999999999999}]},{"osd":3,"last update":"Thu Oct 19 14:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.495,"15min":0.501},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.29499999999999998},"max":{"1min":0.59399999999999997,"5min":0.76000000000000001,"15min":0.78900000000000003},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.42899999999999999,"15min":0.44600000000000001},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.26100000000000001},"max":{"1min":0.59899999999999998,"5min":0.65200000000000002,"15min":0.69499999999999995},"last":0.41799999999999998}]},{"osd":4,"last update":"Thu Oct 19 14:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.64200000000000002,"15min":0.66700000000000004},"min":{"1min":0.42299999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.80400000000000005,"5min":0.95999999999999996,"15min":1.042},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.65400000000000003,"15min":0.66600000000000004},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.32300000000000001},"max":{"1min":0.80500000000000005,"5min":0.90800000000000003,"15min":1.1659999999999999},"last":0.54100000000000004}]},{"osd":5,"last update":"Thu Oct 19 14:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.66700000000000004,"15min":0.68200000000000005},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.81200000000000006,"5min":0.96199999999999997,"15min":1.3500000000000001},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.62,"5min":0.68899999999999995,"15min":0.70299999999999996},"min":{"1min":0.442,"5min":0.44,"15min":0.40500000000000003},"max":{"1min":0.82899999999999996,"5min":1.0009999999999999,"15min":1.121},"last":0.64800000000000002}]},{"osd":6,"last update":"Thu Oct 19 14:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.65400000000000003,"15min":0.67600000000000005},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.84999999999999998,"5min":1.0129999999999999,"15min":1.2310000000000001},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.69699999999999995,"15min":0.71499999999999997},"min":{"1min":0.46700000000000003,"5min":0.434,"15min":0.434},"max":{"1min":0.85799999999999998,"5min":1.0009999999999999,"15min":1.274},"last":0.47099999999999997}]},{"osd":7,"last update":"Thu Oct 19 14:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.70499999999999996,"15min":0.72199999999999998},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.86899999999999999,"5min":0.99099999999999999,"15min":1.075},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.68200000000000005,"15min":0.69599999999999995},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.84599999999999997,"5min":1.0169999999999999,"15min":1.117},"last":0.51300000000000001}]}]},{"osd":0,"up_from":13,"seq":55834576027,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61736,"kb_used_data":4188,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310104,"statfs":{"total":96636764160,"available":96573546496,"internally_reserved":0,"allocated":4288512,"data_stored":2657833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Oct 19 14:22:13 2023","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.39000000000000001,"15min":0.40600000000000003},"min":{"1min":0.254,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.47699999999999998,"5min":0.59999999999999998,"15min":0.624},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.40699999999999997,"15min":0.44900000000000001},"min":{"1min":0.252,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.81100000000000005},"last":0.63200000000000001}]},{"osd":2,"last update":"Thu Oct 19 14:22:13 2023","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.45700000000000002,"15min":0.47699999999999998},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.58699999999999997,"5min":0.73099999999999998,"15min":0.754},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.41399999999999998,"15min":0.439},"min":{"1min":0.316,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.54000000000000004,"5min":0.60199999999999998,"15min":0.69099999999999995},"last":0.66200000000000003}]},{"osd":3,"last update":"Thu Oct 19 14:22:13 2023","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.40300000000000002,"15min":0.433},"min":{"1min":0.28399999999999997,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.56899999999999995,"5min":0.58799999999999997,"15min":0.66800000000000004},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.46800000000000003,"15min":0.503},"min":{"1min":0.314,"5min":0.254,"15min":0.254},"max":{"1min":0.55600000000000005,"5min":0.78100000000000003,"15min":0.84599999999999997},"last":0.68000000000000005}]},{"osd":4,"last update":"Thu Oct 19 14:22:13 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.627,"15min":0.66600000000000004},"min":{"1min":0.437,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":1.1890000000000001},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.626,"15min":0.67000000000000004},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.42599999999999999},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.996},"last":0.93600000000000005}]},{"osd":5,"last update":"Thu Oct 19 14:22:13 2023","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.66100000000000003,"15min":0.70999999999999996},"min":{"1min":0.46500000000000002,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.86599999999999999,"5min":0.98499999999999999,"15min":1.1579999999999999},"last":0.89600000000000002},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.67400000000000004,"15min":0.71099999999999997},"min":{"1min":0.52700000000000002,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.91700000000000004,"5min":0.98299999999999998,"15min":1.2310000000000001},"last":0.98999999999999999}]},{"osd":6,"last update":"Thu Oct 19 14:22:13 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.65400000000000003,"15min":0.69599999999999995},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.95499999999999996,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.91600000000000004},{"interface":"front","average":{"1min":0.628,"5min":0.64100000000000001,"15min":0.68999999999999995},"min":{"1min":0.44600000000000001,"5min":0.441,"15min":0.441},"max":{"1min":0.86399999999999999,"5min":0.89900000000000002,"15min":1.01},"last":1.036}]},{"osd":7,"last update":"Thu Oct 19 14:22:13 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.65500000000000003,"15min":0.70499999999999996},"min":{"1min":0.52300000000000002,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.89500000000000002,"5min":0.93400000000000005,"15min":1.014},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.65800000000000003,"15min":0.69299999999999995},"min":{"1min":0.498,"5min":0.434,"15min":0.434},"max":{"1min":0.88400000000000001,"5min":0.93799999999999994,"15min":1.0349999999999999},"last":0.96199999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-19T14:22:38.130 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-19T14:22:38.239 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:38.236+0000 7f526ae39640 1 -- 172.21.15.57:0/697954359 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f52640ce980 msgr2=0x7f52640cf1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:38.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:38.236+0000 7f526ae39640 1 --2- 172.21.15.57:0/697954359 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f52640ce980 0x7f52640cf1d0 secure :-1 s=READY pgs=3034 cs=0 l=1 rev1=1 crypto rx=0x7f5258005fd0 tx=0x7f525802be10 comp rx=0 tx=0).stop 2023-10-19T14:22:38.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:38.236+0000 7f526ae39640 1 -- 172.21.15.57:0/697954359 shutdown_connections 2023-10-19T14:22:38.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:38.236+0000 7f526ae39640 1 --2- 172.21.15.57:0/697954359 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f52640ce980 0x7f52640cf1d0 unknown :-1 s=CLOSED pgs=3034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:38.240 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:38.236+0000 7f526ae39640 1 --2- 172.21.15.57:0/697954359 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f526415e5d0 0x7f52640cdfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:38.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:38.236+0000 7f526ae39640 1 -- 172.21.15.57:0/697954359 >> 172.21.15.57:0/697954359 conn(0x7f526409a140 msgr2=0x7f52640d0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:38.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:38.236+0000 7f526ae39640 1 -- 172.21.15.57:0/697954359 shutdown_connections 2023-10-19T14:22:38.241 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:38.236+0000 7f526ae39640 1 -- 172.21.15.57:0/697954359 wait complete. 2023-10-19T14:22:38.243 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:38.237+0000 7f526ae39640 1 Processor -- start 2023-10-19T14:22:38.243 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:38.237+0000 7f526ae39640 1 -- start start 2023-10-19T14:22:38.243 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:38.238+0000 7f526ae39640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f52640ce980 0x7f52640ca7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:38.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:38.238+0000 7f526ae39640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f52640cf710 0x7f5264152e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:38.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:38.238+0000 7f526ae39640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f526415e5d0 0x7f526415cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:38.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:38.238+0000 7f526ae39640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52640c62e0 con 0x7f52640ce980 2023-10-19T14:22:38.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:38.238+0000 7f526ae39640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52640c56d0 con 0x7f526415e5d0 2023-10-19T14:22:38.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:38.238+0000 7f526ae39640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52640c59b0 con 0x7f52640cf710 2023-10-19T14:22:38.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:38.238+0000 7f5268bd5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f52640ce980 0x7f52640ca7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:38.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:38.238+0000 7f5268bd5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f52640ce980 0x7f52640ca7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59950/0 (socket says 172.21.15.57:59950) 2023-10-19T14:22:38.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:38.238+0000 7f5268bd5640 1 -- 172.21.15.57:0/3240928626 learned_addr learned my addr 172.21.15.57:0/3240928626 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:22:38.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:38.239+0000 7f5263fff640 1 --2- 172.21.15.57:0/3240928626 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f52640cf710 0x7f5264152e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:38.245 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:38.239+0000 7f52693d6640 1 --2- 172.21.15.57:0/3240928626 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f526415e5d0 0x7f526415cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:38.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:38.239+0000 7f5268bd5640 1 -- 172.21.15.57:0/3240928626 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f52640cf710 msgr2=0x7f5264152e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:38.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:38.239+0000 7f5268bd5640 1 --2- 172.21.15.57:0/3240928626 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f52640cf710 0x7f5264152e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:38.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:38.239+0000 7f5268bd5640 1 -- 172.21.15.57:0/3240928626 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f526415e5d0 msgr2=0x7f526415cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:38.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:38.239+0000 7f5268bd5640 1 --2- 172.21.15.57:0/3240928626 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f526415e5d0 0x7f526415cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:38.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:38.239+0000 7f5268bd5640 1 -- 172.21.15.57:0/3240928626 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5258009560 con 0x7f52640ce980 2023-10-19T14:22:38.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:38.239+0000 7f5268bd5640 1 --2- 172.21.15.57:0/3240928626 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f52640ce980 0x7f52640ca7d0 secure :-1 s=READY pgs=3035 cs=0 l=1 rev1=1 crypto rx=0x7f525400c970 tx=0x7f525400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:38.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:38.239+0000 7f52617fa640 1 -- 172.21.15.57:0/3240928626 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5254007bb0 con 0x7f52640ce980 2023-10-19T14:22:38.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:38.239+0000 7f52617fa640 1 -- 172.21.15.57:0/3240928626 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f5254007e00 con 0x7f52640ce980 2023-10-19T14:22:38.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:38.239+0000 7f526ae39640 1 -- 172.21.15.57:0/3240928626 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52640c5c90 con 0x7f52640ce980 2023-10-19T14:22:38.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:38.239+0000 7f52617fa640 1 -- 172.21.15.57:0/3240928626 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5254005280 con 0x7f52640ce980 2023-10-19T14:22:38.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:38.239+0000 7f526ae39640 1 -- 172.21.15.57:0/3240928626 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52640cad10 con 0x7f52640ce980 2023-10-19T14:22:38.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:38.240+0000 7f526ae39640 1 -- 172.21.15.57:0/3240928626 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5264069100 con 0x7f52640ce980 2023-10-19T14:22:38.248 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:38.241+0000 7f52617fa640 1 -- 172.21.15.57:0/3240928626 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 155) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f52540053e0 con 0x7f52640ce980 2023-10-19T14:22:38.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:38.241+0000 7f52617fa640 1 --2- 172.21.15.57:0/3240928626 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f525006af60 0x7f525006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:38.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:38.241+0000 7f52617fa640 1 -- 172.21.15.57:0/3240928626 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(268..268 src has 1..268) v4 ==== 6108+0+0 (secure 0 0 0) 0x7f5254089d80 con 0x7f52640ce980 2023-10-19T14:22:38.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:38.241+0000 7f5263fff640 1 --2- 172.21.15.57:0/3240928626 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f525006af60 0x7f525006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:38.249 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:38.242+0000 7f5263fff640 1 --2- 172.21.15.57:0/3240928626 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f525006af60 0x7f525006d420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f5258035000 tx=0x7f5258004730 comp rx=0 tx=0).ready entity=mgr.10520 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:38.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:38.245+0000 7f52617fa640 1 -- 172.21.15.57:0/3240928626 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f5254054c80 con 0x7f52640ce980 2023-10-19T14:22:38.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:38.422+0000 7f526ae39640 1 -- 172.21.15.57:0/3240928626 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f52640d1a00 con 0x7f52640ce980 2023-10-19T14:22:38.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:38.424+0000 7f52617fa640 1 -- 172.21.15.57:0/3240928626 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v268) v1 ==== 74+0+14464 (secure 0 0 0) 0x7f525401d070 con 0x7f52640ce980 2023-10-19T14:22:38.427 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:22:38.427 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":268,"fsid":"1e9beb19-3260-4dd7-85ef-5b399ce774ac","created":"2023-10-19T12:43:53.174657+0000","modified":"2023-10-19T14:22:35.405372+0000","last_up_change":"2023-10-19T12:44:05.392368+0000","last_in_change":"2023-10-19T12:43:56.419827+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":36,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-19T12:44:07.371186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c0b0bea8-67a6-4909-9801-a10b4d135776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6825","nonce":1015489591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6827","nonce":1015489591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6831","nonce":1015489591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":1015489591},{"type":"v1","addr":"172.21.15.57:6829","nonce":1015489591}]},"public_addr":"172.21.15.57:6825/1015489591","cluster_addr":"172.21.15.57:6827/1015489591","heartbeat_back_addr":"172.21.15.57:6831/1015489591","heartbeat_front_addr":"172.21.15.57:6829/1015489591","state":["exists","up"]},{"osd":1,"uuid":"c8734f82-652c-4579-bf85-443fad32f442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6809","nonce":3729172421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6811","nonce":3729172421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6815","nonce":3729172421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3729172421},{"type":"v1","addr":"172.21.15.57:6813","nonce":3729172421}]},"public_addr":"172.21.15.57:6809/3729172421","cluster_addr":"172.21.15.57:6811/3729172421","heartbeat_back_addr":"172.21.15.57:6815/3729172421","heartbeat_front_addr":"172.21.15.57:6813/3729172421","state":["exists","up"]},{"osd":2,"uuid":"e9ff64fe-d989-42fe-97ee-1aed1b5d98d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6801","nonce":2805338151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6803","nonce":2805338151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6807","nonce":2805338151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":2805338151},{"type":"v1","addr":"172.21.15.57:6805","nonce":2805338151}]},"public_addr":"172.21.15.57:6801/2805338151","cluster_addr":"172.21.15.57:6803/2805338151","heartbeat_back_addr":"172.21.15.57:6807/2805338151","heartbeat_front_addr":"172.21.15.57:6805/2805338151","state":["exists","up"]},{"osd":3,"uuid":"19f034f4-5964-4284-a429-229d154a2130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6817","nonce":3386449871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6819","nonce":3386449871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6823","nonce":3386449871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":3386449871},{"type":"v1","addr":"172.21.15.57:6821","nonce":3386449871}]},"public_addr":"172.21.15.57:6817/3386449871","cluster_addr":"172.21.15.57:6819/3386449871","heartbeat_back_addr":"172.21.15.57:6823/3386449871","heartbeat_front_addr":"172.21.15.57:6821/3386449871","state":["exists","up"]},{"osd":4,"uuid":"082fed67-b8da-4207-bab9-27f01a29ab02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6805","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6807","nonce":4179727948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6809","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6811","nonce":4179727948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6815","nonce":4179727948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":4179727948},{"type":"v1","addr":"172.21.15.179:6813","nonce":4179727948}]},"public_addr":"172.21.15.179:6807/4179727948","cluster_addr":"172.21.15.179:6811/4179727948","heartbeat_back_addr":"172.21.15.179:6815/4179727948","heartbeat_front_addr":"172.21.15.179:6813/4179727948","state":["exists","up"]},{"osd":5,"uuid":"8ecee277-1594-48b1-89b6-01629c88636a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6801","nonce":1129921230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6803","nonce":1129921230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6810","nonce":1129921230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":1129921230},{"type":"v1","addr":"172.21.15.179:6806","nonce":1129921230}]},"public_addr":"172.21.15.179:6801/1129921230","cluster_addr":"172.21.15.179:6803/1129921230","heartbeat_back_addr":"172.21.15.179:6810/1129921230","heartbeat_front_addr":"172.21.15.179:6806/1129921230","state":["exists","up"]},{"osd":6,"uuid":"3b7b20a2-ea0e-40ca-be09-29a46bb1a997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6825","nonce":2898099196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6827","nonce":2898099196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6831","nonce":2898099196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":2898099196},{"type":"v1","addr":"172.21.15.179:6829","nonce":2898099196}]},"public_addr":"172.21.15.179:6825/2898099196","cluster_addr":"172.21.15.179:6827/2898099196","heartbeat_back_addr":"172.21.15.179:6831/2898099196","heartbeat_front_addr":"172.21.15.179:6829/2898099196","state":["exists","up"]},{"osd":7,"uuid":"2ec733c5-528d-4f91-b32f-42dc4ee0525f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":261,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6817","nonce":3758126853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6819","nonce":3758126853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6823","nonce":3758126853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":3758126853},{"type":"v1","addr":"172.21.15.179:6821","nonce":3758126853}]},"public_addr":"172.21.15.179:6817/3758126853","cluster_addr":"172.21.15.179:6819/3758126853","heartbeat_back_addr":"172.21.15.179:6823/3758126853","heartbeat_front_addr":"172.21.15.179:6821/3758126853","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.633251+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.557145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.582624+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.599530+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.577628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.560149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-19T12:44:01.568300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.179:6834/4133411706":"2023-10-20T14:22:32.156248+0000","172.21.15.179:0/2632603637":"2023-10-20T14:22:29.793530+0000","172.21.15.179:0/2998005959":"2023-10-20T14:22:29.793530+0000","172.21.15.179:0/3483041475":"2023-10-20T14:22:29.793530+0000","172.21.15.179:0/3647691630":"2023-10-20T14:22:29.793530+0000","172.21.15.179:6833/2005622276":"2023-10-20T14:22:29.793530+0000","172.21.15.179:6832/2005622276":"2023-10-20T14:22:29.793530+0000","172.21.15.179:0/1983646445":"2023-10-20T14:22:17.865415+0000","172.21.15.179:0/1840191723":"2023-10-20T14:22:17.865415+0000","172.21.15.179:0/2852372821":"2023-10-20T14:22:29.793530+0000","172.21.15.179:0/4168916968":"2023-10-20T14:22:17.865415+0000","172.21.15.179:6833/1573806521":"2023-10-20T14:22:17.865415+0000","172.21.15.179:0/3886182326":"2023-10-20T14:22:29.793530+0000","172.21.15.179:6832/1573806521":"2023-10-20T14:22:17.865415+0000","172.21.15.179:6836/4133411706":"2023-10-20T14:22:32.156248+0000","172.21.15.179:0/658875312":"2023-10-20T14:22:17.865415+0000","172.21.15.179:0/1678121137":"2023-10-20T14:22:29.793530+0000","172.21.15.179:0/1799772260":"2023-10-20T14:22:17.865415+0000","172.21.15.179:0/3831670324":"2023-10-20T14:22:17.865415+0000","172.21.15.179:0/90219027":"2023-10-20T14:22:17.865415+0000","172.21.15.179:0/2739598080":"2023-10-20T14:22:17.865415+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-19T14:22:38.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:38.426+0000 7f526ae39640 1 -- 172.21.15.57:0/3240928626 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f525006af60 msgr2=0x7f525006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:38.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:38.426+0000 7f526ae39640 1 --2- 172.21.15.57:0/3240928626 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f525006af60 0x7f525006d420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f5258035000 tx=0x7f5258004730 comp rx=0 tx=0).stop 2023-10-19T14:22:38.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:38.426+0000 7f526ae39640 1 -- 172.21.15.57:0/3240928626 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f52640ce980 msgr2=0x7f52640ca7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:38.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:38.426+0000 7f526ae39640 1 --2- 172.21.15.57:0/3240928626 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f52640ce980 0x7f52640ca7d0 secure :-1 s=READY pgs=3035 cs=0 l=1 rev1=1 crypto rx=0x7f525400c970 tx=0x7f525400ce30 comp rx=0 tx=0).stop 2023-10-19T14:22:38.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:38.426+0000 7f526ae39640 1 -- 172.21.15.57:0/3240928626 shutdown_connections 2023-10-19T14:22:38.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:38.426+0000 7f526ae39640 1 --2- 172.21.15.57:0/3240928626 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f525006af60 0x7f525006d420 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:38.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:38.426+0000 7f526ae39640 1 --2- 172.21.15.57:0/3240928626 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f526415e5d0 0x7f526415cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:38.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:38.426+0000 7f526ae39640 1 --2- 172.21.15.57:0/3240928626 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f52640cf710 0x7f5264152e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:38.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:38.426+0000 7f526ae39640 1 --2- 172.21.15.57:0/3240928626 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f52640ce980 0x7f52640ca7d0 unknown :-1 s=CLOSED pgs=3035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:38.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:38.426+0000 7f526ae39640 1 -- 172.21.15.57:0/3240928626 >> 172.21.15.57:0/3240928626 conn(0x7f526409a140 msgr2=0x7f5264155440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:38.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:38.426+0000 7f526ae39640 1 -- 172.21.15.57:0/3240928626 shutdown_connections 2023-10-19T14:22:38.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:38.426+0000 7f526ae39640 1 -- 172.21.15.57:0/3240928626 wait complete. 2023-10-19T14:22:39.440 INFO:tasks.ceph:Scrubbing osd.0 2023-10-19T14:22:39.441 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-10-19T14:22:39.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.564+0000 7fd600f97640 1 -- 172.21.15.57:0/1583059861 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5fc06f140 msgr2=0x7fd5fc06f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:39.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.564+0000 7fd600f97640 1 --2- 172.21.15.57:0/1583059861 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5fc06f140 0x7fd5fc06f590 secure :-1 s=READY pgs=3037 cs=0 l=1 rev1=1 crypto rx=0x7fd5f4005fd0 tx=0x7fd5f402c0e0 comp rx=0 tx=0).stop 2023-10-19T14:22:39.567 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.564+0000 7fd600f97640 1 -- 172.21.15.57:0/1583059861 shutdown_connections 2023-10-19T14:22:39.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.564+0000 7fd600f97640 1 --2- 172.21.15.57:0/1583059861 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd5fc0ccb00 0x7fd5fc0ccf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:39.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.564+0000 7fd600f97640 1 --2- 172.21.15.57:0/1583059861 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5fc06f140 0x7fd5fc06f590 unknown :-1 s=CLOSED pgs=3037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:39.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.564+0000 7fd600f97640 1 -- 172.21.15.57:0/1583059861 >> 172.21.15.57:0/1583059861 conn(0x7fd5fc069630 msgr2=0x7fd5fc069a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:39.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.564+0000 7fd600f97640 1 -- 172.21.15.57:0/1583059861 shutdown_connections 2023-10-19T14:22:39.568 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.564+0000 7fd600f97640 1 -- 172.21.15.57:0/1583059861 wait complete. 2023-10-19T14:22:39.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.565+0000 7fd600f97640 1 Processor -- start 2023-10-19T14:22:39.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.565+0000 7fd600f97640 1 -- start start 2023-10-19T14:22:39.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.566+0000 7fd600f97640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd5fc06f140 0x7fd5fc136c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:39.571 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.566+0000 7fd600f97640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5fc0ccb00 0x7fd5fc133b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:39.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.566+0000 7fd600f97640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd5fc1414b0 0x7fd5fc0c9940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:39.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.566+0000 7fd600f97640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5fc134080 con 0x7fd5fc0ccb00 2023-10-19T14:22:39.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.566+0000 7fd600f97640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5fc0c9e80 con 0x7fd5fc06f140 2023-10-19T14:22:39.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.566+0000 7fd600f97640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5fc0aa750 con 0x7fd5fc1414b0 2023-10-19T14:22:39.572 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.566+0000 7fd5f1d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5fc0ccb00 0x7fd5fc133b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:39.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.566+0000 7fd5f1d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5fc0ccb00 0x7fd5fc133b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59968/0 (socket says 172.21.15.57:59968) 2023-10-19T14:22:39.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.566+0000 7fd5f1d9b640 1 -- 172.21.15.57:0/1699298629 learned_addr learned my addr 172.21.15.57:0/1699298629 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:22:39.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.566+0000 7fd5fad9d640 1 --2- 172.21.15.57:0/1699298629 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd5fc1414b0 0x7fd5fc0c9940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:39.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.566+0000 7fd5fa59c640 1 --2- 172.21.15.57:0/1699298629 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd5fc06f140 0x7fd5fc136c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:39.573 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.567+0000 7fd5f1d9b640 1 -- 172.21.15.57:0/1699298629 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd5fc1414b0 msgr2=0x7fd5fc0c9940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:39.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.567+0000 7fd5f1d9b640 1 --2- 172.21.15.57:0/1699298629 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd5fc1414b0 0x7fd5fc0c9940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:39.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.567+0000 7fd5f1d9b640 1 -- 172.21.15.57:0/1699298629 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd5fc06f140 msgr2=0x7fd5fc136c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:39.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.567+0000 7fd5f1d9b640 1 --2- 172.21.15.57:0/1699298629 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd5fc06f140 0x7fd5fc136c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:39.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.567+0000 7fd5f1d9b640 1 -- 172.21.15.57:0/1699298629 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5f4009840 con 0x7fd5fc0ccb00 2023-10-19T14:22:39.574 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.567+0000 7fd5fad9d640 1 --2- 172.21.15.57:0/1699298629 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd5fc1414b0 0x7fd5fc0c9940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:22:39.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.567+0000 7fd5f1d9b640 1 --2- 172.21.15.57:0/1699298629 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5fc0ccb00 0x7fd5fc133b40 secure :-1 s=READY pgs=3038 cs=0 l=1 rev1=1 crypto rx=0x7fd5e400d980 tx=0x7fd5e400de40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:39.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.567+0000 7fd5f37fe640 1 -- 172.21.15.57:0/1699298629 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5e4009c30 con 0x7fd5fc0ccb00 2023-10-19T14:22:39.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.567+0000 7fd5f37fe640 1 -- 172.21.15.57:0/1699298629 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fd5e4009e80 con 0x7fd5fc0ccb00 2023-10-19T14:22:39.575 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.567+0000 7fd600f97640 1 -- 172.21.15.57:0/1699298629 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5fc0aaa30 con 0x7fd5fc0ccb00 2023-10-19T14:22:39.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.567+0000 7fd5f37fe640 1 -- 172.21.15.57:0/1699298629 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5e40106a0 con 0x7fd5fc0ccb00 2023-10-19T14:22:39.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.567+0000 7fd600f97640 1 -- 172.21.15.57:0/1699298629 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5fc0aaff0 con 0x7fd5fc0ccb00 2023-10-19T14:22:39.577 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.568+0000 7fd600f97640 1 -- 172.21.15.57:0/1699298629 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd5fc141890 con 0x7fd5fc0ccb00 2023-10-19T14:22:39.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.569+0000 7fd5f37fe640 1 -- 172.21.15.57:0/1699298629 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 155) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd5e4010800 con 0x7fd5fc0ccb00 2023-10-19T14:22:39.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.569+0000 7fd5f37fe640 1 --2- 172.21.15.57:0/1699298629 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fd5dc06af60 0x7fd5dc06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:39.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.569+0000 7fd5f37fe640 1 -- 172.21.15.57:0/1699298629 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(268..268 src has 1..268) v4 ==== 6108+0+0 (secure 0 0 0) 0x7fd5e4089ed0 con 0x7fd5fc0ccb00 2023-10-19T14:22:39.579 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.569+0000 7fd5f37fe640 1 --2- 172.21.15.57:0/1699298629 >> [v2:172.21.15.57:6824/1015489591,v1:172.21.15.57:6825/1015489591] conn(0x7fd5dc071940 0x7fd5dc073da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:39.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.569+0000 7fd5f37fe640 1 -- 172.21.15.57:0/1699298629 --> [v2:172.21.15.57:6824/1015489591,v1:172.21.15.57:6825/1015489591] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd5dc074470 con 0x7fd5dc071940 2023-10-19T14:22:39.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.569+0000 7fd5f37fe640 1 -- 172.21.15.57:0/1699298629 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_get_version_reply(handle=1 version=268) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd5e408c590 con 0x7fd5fc0ccb00 2023-10-19T14:22:39.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.569+0000 7fd5fa59c640 1 --2- 172.21.15.57:0/1699298629 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fd5dc06af60 0x7fd5dc06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:39.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.569+0000 7fd5fad9d640 1 --2- 172.21.15.57:0/1699298629 >> [v2:172.21.15.57:6824/1015489591,v1:172.21.15.57:6825/1015489591] conn(0x7fd5dc071940 0x7fd5dc073da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:39.580 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.570+0000 7fd5fad9d640 1 --2- 172.21.15.57:0/1699298629 >> [v2:172.21.15.57:6824/1015489591,v1:172.21.15.57:6825/1015489591] conn(0x7fd5dc071940 0x7fd5dc073da0 crc :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:39.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.570+0000 7fd5fa59c640 1 --2- 172.21.15.57:0/1699298629 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fd5dc06af60 0x7fd5dc06d420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec000f60 tx=0x7fd5ec009040 comp rx=0 tx=0).ready entity=mgr.10520 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:39.581 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.570+0000 7fd5f37fe640 1 -- 172.21.15.57:0/1699298629 <== osd.0 v2:172.21.15.57:6824/1015489591 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7fd5dc074470 con 0x7fd5dc071940 2023-10-19T14:22:39.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.583+0000 7fd600f97640 1 -- 172.21.15.57:0/1699298629 --> [v2:172.21.15.57:6824/1015489591,v1:172.21.15.57:6825/1015489591] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fd5fc069100 con 0x7fd5dc071940 2023-10-19T14:22:39.586 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.583+0000 7fd5f37fe640 1 -- 172.21.15.57:0/1699298629 <== osd.0 v2:172.21.15.57:6824/1015489591 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fd5fc069100 con 0x7fd5dc071940 2023-10-19T14:22:39.587 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2023-10-19T14:22:39.587 INFO:teuthology.orchestra.run.smithi057.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-10-19T14:22:39.587 INFO:teuthology.orchestra.run.smithi057.stdout:} 2023-10-19T14:22:39.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.584+0000 7fd600f97640 1 -- 172.21.15.57:0/1699298629 >> [v2:172.21.15.57:6824/1015489591,v1:172.21.15.57:6825/1015489591] conn(0x7fd5dc071940 msgr2=0x7fd5dc073da0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:39.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.584+0000 7fd600f97640 1 --2- 172.21.15.57:0/1699298629 >> [v2:172.21.15.57:6824/1015489591,v1:172.21.15.57:6825/1015489591] conn(0x7fd5dc071940 0x7fd5dc073da0 crc :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:39.588 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.584+0000 7fd600f97640 1 -- 172.21.15.57:0/1699298629 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fd5dc06af60 msgr2=0x7fd5dc06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:39.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.584+0000 7fd600f97640 1 --2- 172.21.15.57:0/1699298629 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fd5dc06af60 0x7fd5dc06d420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec000f60 tx=0x7fd5ec009040 comp rx=0 tx=0).stop 2023-10-19T14:22:39.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.584+0000 7fd600f97640 1 -- 172.21.15.57:0/1699298629 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5fc0ccb00 msgr2=0x7fd5fc133b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:39.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.584+0000 7fd600f97640 1 --2- 172.21.15.57:0/1699298629 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5fc0ccb00 0x7fd5fc133b40 secure :-1 s=READY pgs=3038 cs=0 l=1 rev1=1 crypto rx=0x7fd5e400d980 tx=0x7fd5e400de40 comp rx=0 tx=0).stop 2023-10-19T14:22:39.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.584+0000 7fd5fad9d640 1 -- 172.21.15.57:0/1699298629 reap_dead start 2023-10-19T14:22:39.589 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.584+0000 7fd600f97640 1 -- 172.21.15.57:0/1699298629 shutdown_connections 2023-10-19T14:22:39.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.584+0000 7fd600f97640 1 -- 172.21.15.57:0/1699298629 >> 172.21.15.57:0/1699298629 conn(0x7fd5fc069630 msgr2=0x7fd5fc0aeda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:39.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.584+0000 7fd600f97640 1 -- 172.21.15.57:0/1699298629 shutdown_connections 2023-10-19T14:22:39.590 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.584+0000 7fd600f97640 1 -- 172.21.15.57:0/1699298629 wait complete. 2023-10-19T14:22:39.597 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2023-10-19T14:22:39.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.705+0000 7f2f86c07640 1 -- 172.21.15.57:0/461324275 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2f8006f2a0 msgr2=0x7f2f800b0c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:39.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.705+0000 7f2f86c07640 1 --2- 172.21.15.57:0/461324275 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2f8006f2a0 0x7f2f800b0c10 secure :-1 s=READY pgs=3039 cs=0 l=1 rev1=1 crypto rx=0x7f2f800afaf0 tx=0x7f2f7402bdb0 comp rx=0 tx=0).stop 2023-10-19T14:22:39.708 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.705+0000 7f2f86c07640 1 -- 172.21.15.57:0/461324275 shutdown_connections 2023-10-19T14:22:39.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.705+0000 7f2f86c07640 1 --2- 172.21.15.57:0/461324275 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2f8006f2a0 0x7f2f800b0c10 unknown :-1 s=CLOSED pgs=3039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:39.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.705+0000 7f2f86c07640 1 --2- 172.21.15.57:0/461324275 >> v2:172.21.15.179:3300/0 conn(0x7f2f801417e0 0x7f2f80070a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:39.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.705+0000 7f2f86c07640 1 -- 172.21.15.57:0/461324275 >> 172.21.15.57:0/461324275 conn(0x7f2f80069810 msgr2=0x7f2f80069c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:39.709 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.705+0000 7f2f86c07640 1 -- 172.21.15.57:0/461324275 shutdown_connections 2023-10-19T14:22:39.710 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.705+0000 7f2f86c07640 1 -- 172.21.15.57:0/461324275 wait complete. 2023-10-19T14:22:39.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.706+0000 7f2f86c07640 1 Processor -- start 2023-10-19T14:22:39.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.706+0000 7f2f86c07640 1 -- start start 2023-10-19T14:22:39.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.707+0000 7f2f86c07640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2f8006f2a0 0x7f2f80130fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:39.712 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.707+0000 7f2f86c07640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2f800b1820 0x7f2f800c85f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:39.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.707+0000 7f2f86c07640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2f801417e0 0x7f2f800c6000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:39.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.707+0000 7f2f86c07640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f8012bff0 con 0x7f2f801417e0 2023-10-19T14:22:39.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.707+0000 7f2f86c07640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f8012c2d0 con 0x7f2f800b1820 2023-10-19T14:22:39.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.707+0000 7f2f86c07640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f8012c5b0 con 0x7f2f8006f2a0 2023-10-19T14:22:39.713 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.707+0000 7f2f851a4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2f801417e0 0x7f2f800c6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:39.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.707+0000 7f2f851a4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2f801417e0 0x7f2f800c6000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59990/0 (socket says 172.21.15.57:59990) 2023-10-19T14:22:39.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.707+0000 7f2f851a4640 1 -- 172.21.15.57:0/1914936717 learned_addr learned my addr 172.21.15.57:0/1914936717 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:22:39.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.707+0000 7f2f849a3640 1 --2- 172.21.15.57:0/1914936717 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2f8006f2a0 0x7f2f80130fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:39.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.707+0000 7f2f7ffff640 1 --2- 172.21.15.57:0/1914936717 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2f800b1820 0x7f2f800c85f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:39.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.707+0000 7f2f851a4640 1 -- 172.21.15.57:0/1914936717 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2f8006f2a0 msgr2=0x7f2f80130fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:39.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.707+0000 7f2f851a4640 1 --2- 172.21.15.57:0/1914936717 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2f8006f2a0 0x7f2f80130fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:39.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.707+0000 7f2f851a4640 1 -- 172.21.15.57:0/1914936717 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2f800b1820 msgr2=0x7f2f800c85f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:39.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.707+0000 7f2f851a4640 1 --2- 172.21.15.57:0/1914936717 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2f800b1820 0x7f2f800c85f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:39.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.707+0000 7f2f851a4640 1 -- 172.21.15.57:0/1914936717 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f74009560 con 0x7f2f801417e0 2023-10-19T14:22:39.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.707+0000 7f2f849a3640 1 --2- 172.21.15.57:0/1914936717 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2f8006f2a0 0x7f2f80130fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:22:39.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.707+0000 7f2f7ffff640 1 --2- 172.21.15.57:0/1914936717 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2f800b1820 0x7f2f800c85f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:22:39.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.708+0000 7f2f851a4640 1 --2- 172.21.15.57:0/1914936717 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2f801417e0 0x7f2f800c6000 secure :-1 s=READY pgs=3040 cs=0 l=1 rev1=1 crypto rx=0x7f2f7000c390 tx=0x7f2f7000c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:39.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.708+0000 7f2f7d7fa640 1 -- 172.21.15.57:0/1914936717 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f700035f0 con 0x7f2f801417e0 2023-10-19T14:22:39.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.708+0000 7f2f7d7fa640 1 -- 172.21.15.57:0/1914936717 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f2f70003750 con 0x7f2f801417e0 2023-10-19T14:22:39.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.708+0000 7f2f86c07640 1 -- 172.21.15.57:0/1914936717 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f800b21c0 con 0x7f2f801417e0 2023-10-19T14:22:39.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.708+0000 7f2f7d7fa640 1 -- 172.21.15.57:0/1914936717 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f70003b60 con 0x7f2f801417e0 2023-10-19T14:22:39.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.708+0000 7f2f86c07640 1 -- 172.21.15.57:0/1914936717 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f8012cb70 con 0x7f2f801417e0 2023-10-19T14:22:39.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.709+0000 7f2f86c07640 1 -- 172.21.15.57:0/1914936717 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f80069100 con 0x7f2f801417e0 2023-10-19T14:22:39.717 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.709+0000 7f2f7d7fa640 1 -- 172.21.15.57:0/1914936717 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 155) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f2f70003cc0 con 0x7f2f801417e0 2023-10-19T14:22:39.718 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.710+0000 7f2f7d7fa640 1 --2- 172.21.15.57:0/1914936717 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f2f6c06af60 0x7f2f6c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:39.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.710+0000 7f2f7d7fa640 1 -- 172.21.15.57:0/1914936717 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(268..268 src has 1..268) v4 ==== 6108+0+0 (secure 0 0 0) 0x7f2f700890d0 con 0x7f2f801417e0 2023-10-19T14:22:39.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.710+0000 7f2f849a3640 1 --2- 172.21.15.57:0/1914936717 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f2f6c06af60 0x7f2f6c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:39.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.710+0000 7f2f849a3640 1 --2- 172.21.15.57:0/1914936717 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f2f6c06af60 0x7f2f6c06d420 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2f801f2da0 tx=0x7f2f780073e0 comp rx=0 tx=0).ready entity=mgr.10520 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:39.719 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.714+0000 7f2f7d7fa640 1 -- 172.21.15.57:0/1914936717 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f2f70053ee0 con 0x7f2f801417e0 2023-10-19T14:22:39.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.889+0000 7f2f86c07640 1 -- 172.21.15.57:0/1914936717 --> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7f2f80069e50 con 0x7f2f6c06af60 2023-10-19T14:22:39.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.891+0000 7f2f7d7fa640 1 -- 172.21.15.57:0/1914936717 <== mgr.10520 v2:172.21.15.179:6832/892689897 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f2f80069e50 con 0x7f2f6c06af60 2023-10-19T14:22:39.893 INFO:teuthology.orchestra.run.smithi057.stderr:instructed osd(s) 0 to deep-scrub 2023-10-19T14:22:39.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.892+0000 7f2f86c07640 1 -- 172.21.15.57:0/1914936717 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f2f6c06af60 msgr2=0x7f2f6c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:39.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.892+0000 7f2f86c07640 1 --2- 172.21.15.57:0/1914936717 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f2f6c06af60 0x7f2f6c06d420 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2f801f2da0 tx=0x7f2f780073e0 comp rx=0 tx=0).stop 2023-10-19T14:22:39.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.892+0000 7f2f86c07640 1 -- 172.21.15.57:0/1914936717 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2f801417e0 msgr2=0x7f2f800c6000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:39.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.892+0000 7f2f86c07640 1 --2- 172.21.15.57:0/1914936717 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2f801417e0 0x7f2f800c6000 secure :-1 s=READY pgs=3040 cs=0 l=1 rev1=1 crypto rx=0x7f2f7000c390 tx=0x7f2f7000c850 comp rx=0 tx=0).stop 2023-10-19T14:22:39.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.892+0000 7f2f86c07640 1 -- 172.21.15.57:0/1914936717 shutdown_connections 2023-10-19T14:22:39.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.892+0000 7f2f86c07640 1 --2- 172.21.15.57:0/1914936717 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f2f6c06af60 0x7f2f6c06d420 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:39.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.892+0000 7f2f86c07640 1 --2- 172.21.15.57:0/1914936717 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2f801417e0 0x7f2f800c6000 unknown :-1 s=CLOSED pgs=3040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:39.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.892+0000 7f2f86c07640 1 --2- 172.21.15.57:0/1914936717 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2f800b1820 0x7f2f800c85f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:39.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.892+0000 7f2f86c07640 1 --2- 172.21.15.57:0/1914936717 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f2f8006f2a0 0x7f2f80130fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:39.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.892+0000 7f2f86c07640 1 -- 172.21.15.57:0/1914936717 >> 172.21.15.57:0/1914936717 conn(0x7f2f80069810 msgr2=0x7f2f800cb940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:39.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.892+0000 7f2f86c07640 1 -- 172.21.15.57:0/1914936717 shutdown_connections 2023-10-19T14:22:39.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:39.892+0000 7f2f86c07640 1 -- 172.21.15.57:0/1914936717 wait complete. 2023-10-19T14:22:39.905 INFO:tasks.ceph:Scrubbing osd.1 2023-10-19T14:22:39.906 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-10-19T14:22:40.018 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.015+0000 7fb2584e5640 1 -- 172.21.15.57:0/18447609 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb2500ccdc0 msgr2=0x7fb2500b2630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:40.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.015+0000 7fb2584e5640 1 --2- 172.21.15.57:0/18447609 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb2500ccdc0 0x7fb2500b2630 secure :-1 s=READY pgs=3042 cs=0 l=1 rev1=1 crypto rx=0x7fb24c005fd0 tx=0x7fb24c02c2e0 comp rx=0 tx=0).stop 2023-10-19T14:22:40.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.015+0000 7fb2584e5640 1 -- 172.21.15.57:0/18447609 shutdown_connections 2023-10-19T14:22:40.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.015+0000 7fb2584e5640 1 --2- 172.21.15.57:0/18447609 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb2500afa80 0x7fb2500b20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:40.019 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.015+0000 7fb2584e5640 1 --2- 172.21.15.57:0/18447609 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb2500ccdc0 0x7fb2500b2630 unknown :-1 s=CLOSED pgs=3042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:40.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.015+0000 7fb2584e5640 1 -- 172.21.15.57:0/18447609 >> 172.21.15.57:0/18447609 conn(0x7fb2500aef90 msgr2=0x7fb2500af370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:40.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.015+0000 7fb2584e5640 1 -- 172.21.15.57:0/18447609 shutdown_connections 2023-10-19T14:22:40.020 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.015+0000 7fb2584e5640 1 -- 172.21.15.57:0/18447609 wait complete. 2023-10-19T14:22:40.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.016+0000 7fb2584e5640 1 Processor -- start 2023-10-19T14:22:40.023 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.016+0000 7fb2584e5640 1 -- start start 2023-10-19T14:22:40.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.017+0000 7fb2584e5640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb25006a630 0x7fb250140a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:40.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.017+0000 7fb2584e5640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb2500afa80 0x7fb2501e60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:40.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.017+0000 7fb2584e5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb2500ccdc0 0x7fb25015a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:40.024 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.017+0000 7fb2584e5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb25015af60 con 0x7fb2500ccdc0 2023-10-19T14:22:40.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.017+0000 7fb2584e5640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb25013ffb0 con 0x7fb2500afa80 2023-10-19T14:22:40.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.017+0000 7fb2584e5640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb250140290 con 0x7fb25006a630 2023-10-19T14:22:40.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.017+0000 7fb256a82640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb2500ccdc0 0x7fb25015a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:40.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.017+0000 7fb256a82640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb2500ccdc0 0x7fb25015a4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:60000/0 (socket says 172.21.15.57:60000) 2023-10-19T14:22:40.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.017+0000 7fb256a82640 1 -- 172.21.15.57:0/4265159736 learned_addr learned my addr 172.21.15.57:0/4265159736 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:22:40.025 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.017+0000 7fb256281640 1 --2- 172.21.15.57:0/4265159736 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb25006a630 0x7fb250140a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:40.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.017+0000 7fb255a80640 1 --2- 172.21.15.57:0/4265159736 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb2500afa80 0x7fb2501e60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:40.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.017+0000 7fb256a82640 1 -- 172.21.15.57:0/4265159736 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb25006a630 msgr2=0x7fb250140a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:40.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.017+0000 7fb256a82640 1 --2- 172.21.15.57:0/4265159736 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fb25006a630 0x7fb250140a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:40.026 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.017+0000 7fb256a82640 1 -- 172.21.15.57:0/4265159736 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb2500afa80 msgr2=0x7fb2501e60e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:40.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.017+0000 7fb256a82640 1 --2- 172.21.15.57:0/4265159736 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb2500afa80 0x7fb2501e60e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:40.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.017+0000 7fb256a82640 1 -- 172.21.15.57:0/4265159736 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb24c009900 con 0x7fb2500ccdc0 2023-10-19T14:22:40.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.018+0000 7fb256a82640 1 --2- 172.21.15.57:0/4265159736 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb2500ccdc0 0x7fb25015a4a0 secure :-1 s=READY pgs=3043 cs=0 l=1 rev1=1 crypto rx=0x7fb24c009bf0 tx=0x7fb24c035b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:40.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.018+0000 7fb246ffd640 1 -- 172.21.15.57:0/4265159736 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb24c004240 con 0x7fb2500ccdc0 2023-10-19T14:22:40.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.018+0000 7fb2584e5640 1 -- 172.21.15.57:0/4265159736 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb250140570 con 0x7fb2500ccdc0 2023-10-19T14:22:40.027 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.018+0000 7fb246ffd640 1 -- 172.21.15.57:0/4265159736 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fb24c004490 con 0x7fb2500ccdc0 2023-10-19T14:22:40.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.018+0000 7fb246ffd640 1 -- 172.21.15.57:0/4265159736 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb24c004860 con 0x7fb2500ccdc0 2023-10-19T14:22:40.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.018+0000 7fb2584e5640 1 -- 172.21.15.57:0/4265159736 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb25015b240 con 0x7fb2500ccdc0 2023-10-19T14:22:40.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.019+0000 7fb2584e5640 1 -- 172.21.15.57:0/4265159736 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb250140120 con 0x7fb2500ccdc0 2023-10-19T14:22:40.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.019+0000 7fb246ffd640 1 -- 172.21.15.57:0/4265159736 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 155) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb24c03b050 con 0x7fb2500ccdc0 2023-10-19T14:22:40.028 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.020+0000 7fb246ffd640 1 --2- 172.21.15.57:0/4265159736 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fb23406af90 0x7fb23406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:40.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.020+0000 7fb246ffd640 1 -- 172.21.15.57:0/4265159736 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(268..268 src has 1..268) v4 ==== 6108+0+0 (secure 0 0 0) 0x7fb24c0ac490 con 0x7fb2500ccdc0 2023-10-19T14:22:40.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.020+0000 7fb246ffd640 1 --2- 172.21.15.57:0/4265159736 >> [v2:172.21.15.57:6808/3729172421,v1:172.21.15.57:6809/3729172421] conn(0x7fb234071970 0x7fb234073dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:40.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.020+0000 7fb246ffd640 1 -- 172.21.15.57:0/4265159736 --> [v2:172.21.15.57:6808/3729172421,v1:172.21.15.57:6809/3729172421] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb2340744a0 con 0x7fb234071970 2023-10-19T14:22:40.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.020+0000 7fb246ffd640 1 -- 172.21.15.57:0/4265159736 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_get_version_reply(handle=1 version=268) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb24c0aeb50 con 0x7fb2500ccdc0 2023-10-19T14:22:40.029 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.020+0000 7fb256281640 1 --2- 172.21.15.57:0/4265159736 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fb23406af90 0x7fb23406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:40.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.020+0000 7fb255a80640 1 --2- 172.21.15.57:0/4265159736 >> [v2:172.21.15.57:6808/3729172421,v1:172.21.15.57:6809/3729172421] conn(0x7fb234071970 0x7fb234073dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:40.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.020+0000 7fb255a80640 1 --2- 172.21.15.57:0/4265159736 >> [v2:172.21.15.57:6808/3729172421,v1:172.21.15.57:6809/3729172421] conn(0x7fb234071970 0x7fb234073dd0 crc :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:40.030 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.020+0000 7fb256281640 1 --2- 172.21.15.57:0/4265159736 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fb23406af90 0x7fb23406d450 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb248000f60 tx=0x7fb248009040 comp rx=0 tx=0).ready entity=mgr.10520 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:40.031 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.021+0000 7fb246ffd640 1 -- 172.21.15.57:0/4265159736 <== osd.1 v2:172.21.15.57:6808/3729172421 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7fb2340744a0 con 0x7fb234071970 2023-10-19T14:22:40.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.034+0000 7fb2584e5640 1 -- 172.21.15.57:0/4265159736 --> [v2:172.21.15.57:6808/3729172421,v1:172.21.15.57:6809/3729172421] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fb250071440 con 0x7fb234071970 2023-10-19T14:22:40.037 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.034+0000 7fb246ffd640 1 -- 172.21.15.57:0/4265159736 <== osd.1 v2:172.21.15.57:6808/3729172421 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fb24000ac20 con 0x7fb234071970 2023-10-19T14:22:40.037 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2023-10-19T14:22:40.037 INFO:teuthology.orchestra.run.smithi057.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-10-19T14:22:40.038 INFO:teuthology.orchestra.run.smithi057.stdout:} 2023-10-19T14:22:40.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.034+0000 7fb2584e5640 1 -- 172.21.15.57:0/4265159736 >> [v2:172.21.15.57:6808/3729172421,v1:172.21.15.57:6809/3729172421] conn(0x7fb234071970 msgr2=0x7fb234073dd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:40.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.034+0000 7fb2584e5640 1 --2- 172.21.15.57:0/4265159736 >> [v2:172.21.15.57:6808/3729172421,v1:172.21.15.57:6809/3729172421] conn(0x7fb234071970 0x7fb234073dd0 crc :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:40.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.034+0000 7fb2584e5640 1 -- 172.21.15.57:0/4265159736 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fb23406af90 msgr2=0x7fb23406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:40.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.034+0000 7fb2584e5640 1 --2- 172.21.15.57:0/4265159736 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fb23406af90 0x7fb23406d450 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb248000f60 tx=0x7fb248009040 comp rx=0 tx=0).stop 2023-10-19T14:22:40.039 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.034+0000 7fb2584e5640 1 -- 172.21.15.57:0/4265159736 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb2500ccdc0 msgr2=0x7fb25015a4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:40.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.034+0000 7fb2584e5640 1 --2- 172.21.15.57:0/4265159736 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb2500ccdc0 0x7fb25015a4a0 secure :-1 s=READY pgs=3043 cs=0 l=1 rev1=1 crypto rx=0x7fb24c009bf0 tx=0x7fb24c035b50 comp rx=0 tx=0).stop 2023-10-19T14:22:40.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.034+0000 7fb256a82640 1 -- 172.21.15.57:0/4265159736 reap_dead start 2023-10-19T14:22:40.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.034+0000 7fb2584e5640 1 -- 172.21.15.57:0/4265159736 shutdown_connections 2023-10-19T14:22:40.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.034+0000 7fb2584e5640 1 -- 172.21.15.57:0/4265159736 >> 172.21.15.57:0/4265159736 conn(0x7fb2500aef90 msgr2=0x7fb2500c70f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:40.040 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.034+0000 7fb2584e5640 1 -- 172.21.15.57:0/4265159736 shutdown_connections 2023-10-19T14:22:40.041 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.034+0000 7fb2584e5640 1 -- 172.21.15.57:0/4265159736 wait complete. 2023-10-19T14:22:40.047 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2023-10-19T14:22:40.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.154+0000 7fcf8fe7c640 1 -- 172.21.15.57:0/1159178760 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcf88158220 msgr2=0x7fcf880700d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:40.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.154+0000 7fcf8fe7c640 1 --2- 172.21.15.57:0/1159178760 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcf88158220 0x7fcf880700d0 secure :-1 s=READY pgs=3045 cs=0 l=1 rev1=1 crypto rx=0x7fcf8806a610 tx=0x7fcf7c02bcf0 comp rx=0 tx=0).stop 2023-10-19T14:22:40.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.154+0000 7fcf8fe7c640 1 -- 172.21.15.57:0/1159178760 shutdown_connections 2023-10-19T14:22:40.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.154+0000 7fcf8fe7c640 1 --2- 172.21.15.57:0/1159178760 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcf88158220 0x7fcf880700d0 unknown :-1 s=CLOSED pgs=3045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:40.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.154+0000 7fcf8fe7c640 1 --2- 172.21.15.57:0/1159178760 >> v2:172.21.15.179:3300/0 conn(0x7fcf88157e40 0x7fcf8806b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:40.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.154+0000 7fcf8fe7c640 1 -- 172.21.15.57:0/1159178760 >> 172.21.15.57:0/1159178760 conn(0x7fcf880aecf0 msgr2=0x7fcf880af0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:40.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.154+0000 7fcf8fe7c640 1 -- 172.21.15.57:0/1159178760 shutdown_connections 2023-10-19T14:22:40.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.154+0000 7fcf8fe7c640 1 -- 172.21.15.57:0/1159178760 wait complete. 2023-10-19T14:22:40.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.155+0000 7fcf8fe7c640 1 Processor -- start 2023-10-19T14:22:40.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.156+0000 7fcf8fe7c640 1 -- start start 2023-10-19T14:22:40.161 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.156+0000 7fcf8fe7c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcf880ac060 0x7fcf88158d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:40.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.156+0000 7fcf8fe7c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcf88157e40 0x7fcf88145a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:40.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.156+0000 7fcf8fe7c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcf88158220 0x7fcf8806bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:40.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.156+0000 7fcf8fe7c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf88161640 con 0x7fcf88157e40 2023-10-19T14:22:40.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.156+0000 7fcf8fe7c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf88159240 con 0x7fcf88158220 2023-10-19T14:22:40.162 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.156+0000 7fcf8fe7c640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf88163e30 con 0x7fcf880ac060 2023-10-19T14:22:40.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.156+0000 7fcf8d417640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcf88157e40 0x7fcf88145a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:40.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.156+0000 7fcf8d417640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcf88157e40 0x7fcf88145a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:60008/0 (socket says 172.21.15.57:60008) 2023-10-19T14:22:40.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.156+0000 7fcf8d417640 1 -- 172.21.15.57:0/1831727497 learned_addr learned my addr 172.21.15.57:0/1831727497 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:22:40.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.157+0000 7fcf8dc18640 1 --2- 172.21.15.57:0/1831727497 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcf880ac060 0x7fcf88158d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:40.163 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.157+0000 7fcf8e419640 1 --2- 172.21.15.57:0/1831727497 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcf88158220 0x7fcf8806bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:40.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.157+0000 7fcf8d417640 1 -- 172.21.15.57:0/1831727497 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcf880ac060 msgr2=0x7fcf88158d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:40.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.157+0000 7fcf8d417640 1 --2- 172.21.15.57:0/1831727497 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcf880ac060 0x7fcf88158d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:40.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.157+0000 7fcf8d417640 1 -- 172.21.15.57:0/1831727497 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcf88158220 msgr2=0x7fcf8806bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:40.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.157+0000 7fcf8d417640 1 --2- 172.21.15.57:0/1831727497 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcf88158220 0x7fcf8806bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:40.164 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.157+0000 7fcf8d417640 1 -- 172.21.15.57:0/1831727497 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf7c009490 con 0x7fcf88157e40 2023-10-19T14:22:40.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.157+0000 7fcf8d417640 1 --2- 172.21.15.57:0/1831727497 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcf88157e40 0x7fcf88145a90 secure :-1 s=READY pgs=3046 cs=0 l=1 rev1=1 crypto rx=0x7fcf7c000c00 tx=0x7fcf7c02c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:40.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.157+0000 7fcf867fc640 1 -- 172.21.15.57:0/1831727497 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf7c003a80 con 0x7fcf88157e40 2023-10-19T14:22:40.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.157+0000 7fcf867fc640 1 -- 172.21.15.57:0/1831727497 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fcf7c003cf0 con 0x7fcf88157e40 2023-10-19T14:22:40.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.157+0000 7fcf8fe7c640 1 -- 172.21.15.57:0/1831727497 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf88164110 con 0x7fcf88157e40 2023-10-19T14:22:40.165 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.157+0000 7fcf867fc640 1 -- 172.21.15.57:0/1831727497 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf7c0346a0 con 0x7fcf88157e40 2023-10-19T14:22:40.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.157+0000 7fcf8fe7c640 1 -- 172.21.15.57:0/1831727497 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf881612d0 con 0x7fcf88157e40 2023-10-19T14:22:40.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.158+0000 7fcf8fe7c640 1 -- 172.21.15.57:0/1831727497 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf881617f0 con 0x7fcf88157e40 2023-10-19T14:22:40.166 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.158+0000 7fcf867fc640 1 -- 172.21.15.57:0/1831727497 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 155) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fcf7c03b040 con 0x7fcf88157e40 2023-10-19T14:22:40.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.159+0000 7fcf867fc640 1 --2- 172.21.15.57:0/1831727497 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fcf7406af60 0x7fcf7406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:40.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.159+0000 7fcf867fc640 1 -- 172.21.15.57:0/1831727497 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(268..268 src has 1..268) v4 ==== 6108+0+0 (secure 0 0 0) 0x7fcf7c0aba80 con 0x7fcf88157e40 2023-10-19T14:22:40.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.159+0000 7fcf8dc18640 1 --2- 172.21.15.57:0/1831727497 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fcf7406af60 0x7fcf7406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:40.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.160+0000 7fcf8dc18640 1 --2- 172.21.15.57:0/1831727497 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fcf7406af60 0x7fcf7406d420 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fcf80005fd0 tx=0x7fcf80005850 comp rx=0 tx=0).ready entity=mgr.10520 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:40.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.163+0000 7fcf867fc640 1 -- 172.21.15.57:0/1831727497 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fcf881617f0 con 0x7fcf88157e40 2023-10-19T14:22:40.341 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.338+0000 7fcf8fe7c640 1 -- 172.21.15.57:0/1831727497 --> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7fcf8815b660 con 0x7fcf7406af60 2023-10-19T14:22:40.342 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.338+0000 7fcf867fc640 1 -- 172.21.15.57:0/1831727497 <== mgr.10520 v2:172.21.15.179:6832/892689897 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fcf8815b660 con 0x7fcf7406af60 2023-10-19T14:22:40.342 INFO:teuthology.orchestra.run.smithi057.stderr:instructed osd(s) 1 to deep-scrub 2023-10-19T14:22:40.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.340+0000 7fcf8fe7c640 1 -- 172.21.15.57:0/1831727497 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fcf7406af60 msgr2=0x7fcf7406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:40.343 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.340+0000 7fcf8fe7c640 1 --2- 172.21.15.57:0/1831727497 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fcf7406af60 0x7fcf7406d420 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fcf80005fd0 tx=0x7fcf80005850 comp rx=0 tx=0).stop 2023-10-19T14:22:40.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.340+0000 7fcf8fe7c640 1 -- 172.21.15.57:0/1831727497 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcf88157e40 msgr2=0x7fcf88145a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:40.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.340+0000 7fcf8fe7c640 1 --2- 172.21.15.57:0/1831727497 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcf88157e40 0x7fcf88145a90 secure :-1 s=READY pgs=3046 cs=0 l=1 rev1=1 crypto rx=0x7fcf7c000c00 tx=0x7fcf7c02c880 comp rx=0 tx=0).stop 2023-10-19T14:22:40.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.340+0000 7fcf8fe7c640 1 -- 172.21.15.57:0/1831727497 shutdown_connections 2023-10-19T14:22:40.344 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.340+0000 7fcf8fe7c640 1 --2- 172.21.15.57:0/1831727497 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fcf7406af60 0x7fcf7406d420 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:40.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.340+0000 7fcf8fe7c640 1 --2- 172.21.15.57:0/1831727497 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcf88158220 0x7fcf8806bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:40.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.340+0000 7fcf8fe7c640 1 --2- 172.21.15.57:0/1831727497 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcf88157e40 0x7fcf88145a90 unknown :-1 s=CLOSED pgs=3046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:40.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.340+0000 7fcf8fe7c640 1 --2- 172.21.15.57:0/1831727497 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcf880ac060 0x7fcf88158d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:40.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.340+0000 7fcf8fe7c640 1 -- 172.21.15.57:0/1831727497 >> 172.21.15.57:0/1831727497 conn(0x7fcf880aecf0 msgr2=0x7fcf880ade90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:40.345 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.340+0000 7fcf8fe7c640 1 -- 172.21.15.57:0/1831727497 shutdown_connections 2023-10-19T14:22:40.346 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.340+0000 7fcf8fe7c640 1 -- 172.21.15.57:0/1831727497 wait complete. 2023-10-19T14:22:40.353 INFO:tasks.ceph:Scrubbing osd.2 2023-10-19T14:22:40.354 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-10-19T14:22:40.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.463+0000 7f148c0f2640 1 -- 172.21.15.57:0/3300503468 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f14840cd200 msgr2=0x7f14840cd650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:40.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.463+0000 7f148c0f2640 1 --2- 172.21.15.57:0/3300503468 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f14840cd200 0x7f14840cd650 secure :-1 s=READY pgs=3048 cs=0 l=1 rev1=1 crypto rx=0x7f147c005fd0 tx=0x7f147c02c2e0 comp rx=0 tx=0).stop 2023-10-19T14:22:40.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.463+0000 7f148c0f2640 1 -- 172.21.15.57:0/3300503468 shutdown_connections 2023-10-19T14:22:40.467 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.463+0000 7f148c0f2640 1 --2- 172.21.15.57:0/3300503468 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f14840cd200 0x7f14840cd650 unknown :-1 s=CLOSED pgs=3048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:40.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.463+0000 7f148c0f2640 1 --2- 172.21.15.57:0/3300503468 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f14840cae30 0x7f14840cb4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:40.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.463+0000 7f148c0f2640 1 -- 172.21.15.57:0/3300503468 >> 172.21.15.57:0/3300503468 conn(0x7f14840b0880 msgr2=0x7f14840b0c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:40.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.464+0000 7f148c0f2640 1 -- 172.21.15.57:0/3300503468 shutdown_connections 2023-10-19T14:22:40.468 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.464+0000 7f148c0f2640 1 -- 172.21.15.57:0/3300503468 wait complete. 2023-10-19T14:22:40.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.465+0000 7f148c0f2640 1 Processor -- start 2023-10-19T14:22:40.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.465+0000 7f148c0f2640 1 -- start start 2023-10-19T14:22:40.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.465+0000 7f148c0f2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f148406d870 0x7f14841e31e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:40.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.466+0000 7f148c0f2640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f14840cae30 0x7f1484157160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:40.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.466+0000 7f148c0f2640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f14840cd200 0x7f14841576a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:40.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.466+0000 7f148c0f2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f148406c0b0 con 0x7f148406d870 2023-10-19T14:22:40.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.466+0000 7f148c0f2640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14841e3720 con 0x7f14840cd200 2023-10-19T14:22:40.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.466+0000 7f148c0f2640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f148406b430 con 0x7f14840cae30 2023-10-19T14:22:40.472 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.466+0000 7f1489e8e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f148406d870 0x7f14841e31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:40.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.466+0000 7f1489e8e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f148406d870 0x7f14841e31e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:60016/0 (socket says 172.21.15.57:60016) 2023-10-19T14:22:40.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.466+0000 7f1489e8e640 1 -- 172.21.15.57:0/1740176029 learned_addr learned my addr 172.21.15.57:0/1740176029 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:22:40.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.466+0000 7f1489e8e640 1 -- 172.21.15.57:0/1740176029 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f14840cae30 msgr2=0x7f1484157160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:40.473 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.466+0000 7f1489e8e640 1 --2- 172.21.15.57:0/1740176029 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f14840cae30 0x7f1484157160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:40.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.466+0000 7f1489e8e640 1 -- 172.21.15.57:0/1740176029 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f14840cd200 msgr2=0x7f14841576a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:40.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.466+0000 7f1489e8e640 1 --2- 172.21.15.57:0/1740176029 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f14840cd200 0x7f14841576a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:40.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.466+0000 7f1489e8e640 1 -- 172.21.15.57:0/1740176029 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f147c009900 con 0x7f148406d870 2023-10-19T14:22:40.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.466+0000 7f1489e8e640 1 --2- 172.21.15.57:0/1740176029 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f148406d870 0x7f14841e31e0 secure :-1 s=READY pgs=3049 cs=0 l=1 rev1=1 crypto rx=0x7f148000d950 tx=0x7f148000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:40.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.466+0000 7f147a7fc640 1 -- 172.21.15.57:0/1740176029 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1480009c30 con 0x7f148406d870 2023-10-19T14:22:40.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.466+0000 7f147a7fc640 1 -- 172.21.15.57:0/1740176029 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f1480009e80 con 0x7f148406d870 2023-10-19T14:22:40.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.466+0000 7f148c0f2640 1 -- 172.21.15.57:0/1740176029 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f148406b710 con 0x7f148406d870 2023-10-19T14:22:40.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.466+0000 7f147a7fc640 1 -- 172.21.15.57:0/1740176029 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14800106c0 con 0x7f148406d870 2023-10-19T14:22:40.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.466+0000 7f148c0f2640 1 -- 172.21.15.57:0/1740176029 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f148406bcd0 con 0x7f148406d870 2023-10-19T14:22:40.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.467+0000 7f148c0f2640 1 -- 172.21.15.57:0/1740176029 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f14840cdb90 con 0x7f148406d870 2023-10-19T14:22:40.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.468+0000 7f147a7fc640 1 -- 172.21.15.57:0/1740176029 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 155) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f1480010820 con 0x7f148406d870 2023-10-19T14:22:40.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.468+0000 7f147a7fc640 1 --2- 172.21.15.57:0/1740176029 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f146806af60 0x7f146806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:40.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.468+0000 7f147a7fc640 1 -- 172.21.15.57:0/1740176029 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(268..268 src has 1..268) v4 ==== 6108+0+0 (secure 0 0 0) 0x7f1480089f20 con 0x7f148406d870 2023-10-19T14:22:40.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.468+0000 7f147a7fc640 1 --2- 172.21.15.57:0/1740176029 >> [v2:172.21.15.57:6800/2805338151,v1:172.21.15.57:6801/2805338151] conn(0x7f1468071940 0x7f1468073da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:40.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.468+0000 7f147a7fc640 1 -- 172.21.15.57:0/1740176029 --> [v2:172.21.15.57:6800/2805338151,v1:172.21.15.57:6801/2805338151] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1468074470 con 0x7f1468071940 2023-10-19T14:22:40.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.468+0000 7f147a7fc640 1 -- 172.21.15.57:0/1740176029 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_get_version_reply(handle=1 version=268) v2 ==== 24+0+0 (secure 0 0 0) 0x7f148008c5e0 con 0x7f148406d870 2023-10-19T14:22:40.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.468+0000 7f148968d640 1 --2- 172.21.15.57:0/1740176029 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f146806af60 0x7f146806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:40.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.468+0000 7f148a68f640 1 --2- 172.21.15.57:0/1740176029 >> [v2:172.21.15.57:6800/2805338151,v1:172.21.15.57:6801/2805338151] conn(0x7f1468071940 0x7f1468073da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:40.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.469+0000 7f148a68f640 1 --2- 172.21.15.57:0/1740176029 >> [v2:172.21.15.57:6800/2805338151,v1:172.21.15.57:6801/2805338151] conn(0x7f1468071940 0x7f1468073da0 crc :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:40.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.469+0000 7f148968d640 1 --2- 172.21.15.57:0/1740176029 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f146806af60 0x7f146806d420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f1474000f60 tx=0x7f14740079b0 comp rx=0 tx=0).ready entity=mgr.10520 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:40.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.469+0000 7f147a7fc640 1 -- 172.21.15.57:0/1740176029 <== osd.2 v2:172.21.15.57:6800/2805338151 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f1468074470 con 0x7f1468071940 2023-10-19T14:22:40.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.483+0000 7f148c0f2640 1 -- 172.21.15.57:0/1740176029 --> [v2:172.21.15.57:6800/2805338151,v1:172.21.15.57:6801/2805338151] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f148406c320 con 0x7f1468071940 2023-10-19T14:22:40.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.483+0000 7f147a7fc640 1 -- 172.21.15.57:0/1740176029 <== osd.2 v2:172.21.15.57:6800/2805338151 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f148406c320 con 0x7f1468071940 2023-10-19T14:22:40.485 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2023-10-19T14:22:40.486 INFO:teuthology.orchestra.run.smithi057.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-10-19T14:22:40.486 INFO:teuthology.orchestra.run.smithi057.stdout:} 2023-10-19T14:22:40.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.483+0000 7f148c0f2640 1 -- 172.21.15.57:0/1740176029 >> [v2:172.21.15.57:6800/2805338151,v1:172.21.15.57:6801/2805338151] conn(0x7f1468071940 msgr2=0x7f1468073da0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:40.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.483+0000 7f148c0f2640 1 --2- 172.21.15.57:0/1740176029 >> [v2:172.21.15.57:6800/2805338151,v1:172.21.15.57:6801/2805338151] conn(0x7f1468071940 0x7f1468073da0 crc :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:40.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.483+0000 7f148c0f2640 1 -- 172.21.15.57:0/1740176029 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f146806af60 msgr2=0x7f146806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:40.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.483+0000 7f148c0f2640 1 --2- 172.21.15.57:0/1740176029 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f146806af60 0x7f146806d420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f1474000f60 tx=0x7f14740079b0 comp rx=0 tx=0).stop 2023-10-19T14:22:40.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.483+0000 7f148c0f2640 1 -- 172.21.15.57:0/1740176029 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f148406d870 msgr2=0x7f14841e31e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:40.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.483+0000 7f148c0f2640 1 --2- 172.21.15.57:0/1740176029 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f148406d870 0x7f14841e31e0 secure :-1 s=READY pgs=3049 cs=0 l=1 rev1=1 crypto rx=0x7f148000d950 tx=0x7f148000de10 comp rx=0 tx=0).stop 2023-10-19T14:22:40.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.483+0000 7f148a68f640 1 -- 172.21.15.57:0/1740176029 reap_dead start 2023-10-19T14:22:40.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.483+0000 7f148c0f2640 1 -- 172.21.15.57:0/1740176029 shutdown_connections 2023-10-19T14:22:40.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.483+0000 7f148c0f2640 1 -- 172.21.15.57:0/1740176029 >> 172.21.15.57:0/1740176029 conn(0x7f14840b0880 msgr2=0x7f14840c4c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:40.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.483+0000 7f148c0f2640 1 -- 172.21.15.57:0/1740176029 shutdown_connections 2023-10-19T14:22:40.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.483+0000 7f148c0f2640 1 -- 172.21.15.57:0/1740176029 wait complete. 2023-10-19T14:22:40.496 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2023-10-19T14:22:40.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.603+0000 7efe8cb73640 1 -- 172.21.15.57:0/2706313861 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7efe88159400 msgr2=0x7efe880aa790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:40.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.603+0000 7efe8cb73640 1 --2- 172.21.15.57:0/2706313861 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7efe88159400 0x7efe880aa790 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7efe7c005fd0 tx=0x7efe7c02be10 comp rx=0 tx=0).stop 2023-10-19T14:22:40.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.603+0000 7efe8cb73640 1 -- 172.21.15.57:0/2706313861 shutdown_connections 2023-10-19T14:22:40.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.603+0000 7efe8cb73640 1 --2- 172.21.15.57:0/2706313861 >> v2:172.21.15.179:3300/0 conn(0x7efe880aacd0 0x7efe8806b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:40.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.603+0000 7efe8cb73640 1 --2- 172.21.15.57:0/2706313861 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7efe88159400 0x7efe880aa790 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:40.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.603+0000 7efe8cb73640 1 -- 172.21.15.57:0/2706313861 >> 172.21.15.57:0/2706313861 conn(0x7efe88069810 msgr2=0x7efe88069c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:40.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.603+0000 7efe8cb73640 1 -- 172.21.15.57:0/2706313861 shutdown_connections 2023-10-19T14:22:40.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.603+0000 7efe8cb73640 1 -- 172.21.15.57:0/2706313861 wait complete. 2023-10-19T14:22:40.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.604+0000 7efe8cb73640 1 Processor -- start 2023-10-19T14:22:40.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.604+0000 7efe8cb73640 1 -- start start 2023-10-19T14:22:40.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.605+0000 7efe8cb73640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7efe880aacd0 0x7efe881f0e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:40.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.605+0000 7efe8cb73640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efe881534b0 0x7efe880aa030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:40.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.605+0000 7efe8cb73640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efe88159400 0x7efe880aa570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:40.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.605+0000 7efe8cb73640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe88149b10 con 0x7efe881534b0 2023-10-19T14:22:40.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.605+0000 7efe8cb73640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe8815b7b0 con 0x7efe88159400 2023-10-19T14:22:40.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.605+0000 7efe8cb73640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe88151c90 con 0x7efe880aacd0 2023-10-19T14:22:40.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.605+0000 7efe85d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efe881534b0 0x7efe880aa030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:40.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.605+0000 7efe85d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efe881534b0 0x7efe880aa030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:60024/0 (socket says 172.21.15.57:60024) 2023-10-19T14:22:40.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.605+0000 7efe85d9b640 1 -- 172.21.15.57:0/1363595007 learned_addr learned my addr 172.21.15.57:0/1363595007 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:22:40.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.605+0000 7efe8659c640 1 --2- 172.21.15.57:0/1363595007 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7efe880aacd0 0x7efe881f0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:40.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.605+0000 7efe86d9d640 1 --2- 172.21.15.57:0/1363595007 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efe88159400 0x7efe880aa570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:40.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.605+0000 7efe85d9b640 1 -- 172.21.15.57:0/1363595007 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7efe880aacd0 msgr2=0x7efe881f0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:40.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.605+0000 7efe85d9b640 1 --2- 172.21.15.57:0/1363595007 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7efe880aacd0 0x7efe881f0e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:40.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.605+0000 7efe85d9b640 1 -- 172.21.15.57:0/1363595007 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efe88159400 msgr2=0x7efe880aa570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:40.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.605+0000 7efe85d9b640 1 --2- 172.21.15.57:0/1363595007 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efe88159400 0x7efe880aa570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:40.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.605+0000 7efe85d9b640 1 -- 172.21.15.57:0/1363595007 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe7c009560 con 0x7efe881534b0 2023-10-19T14:22:40.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.606+0000 7efe85d9b640 1 --2- 172.21.15.57:0/1363595007 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efe881534b0 0x7efe880aa030 secure :-1 s=READY pgs=3050 cs=0 l=1 rev1=1 crypto rx=0x7efe7c009850 tx=0x7efe7c02da50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:40.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.606+0000 7efe76ffd640 1 -- 172.21.15.57:0/1363595007 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe7c02de50 con 0x7efe881534b0 2023-10-19T14:22:40.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.606+0000 7efe76ffd640 1 -- 172.21.15.57:0/1363595007 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7efe7c0036e0 con 0x7efe881534b0 2023-10-19T14:22:40.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.606+0000 7efe8cb73640 1 -- 172.21.15.57:0/1363595007 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe88149830 con 0x7efe881534b0 2023-10-19T14:22:40.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.606+0000 7efe76ffd640 1 -- 172.21.15.57:0/1363595007 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe7c003a90 con 0x7efe881534b0 2023-10-19T14:22:40.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.606+0000 7efe8cb73640 1 -- 172.21.15.57:0/1363595007 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe8815b3d0 con 0x7efe881534b0 2023-10-19T14:22:40.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.607+0000 7efe8cb73640 1 -- 172.21.15.57:0/1363595007 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe8815af50 con 0x7efe881534b0 2023-10-19T14:22:40.615 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.607+0000 7efe76ffd640 1 -- 172.21.15.57:0/1363595007 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 155) v1 ==== 89331+0+0 (secure 0 0 0) 0x7efe7c003bf0 con 0x7efe881534b0 2023-10-19T14:22:40.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.608+0000 7efe76ffd640 1 --2- 172.21.15.57:0/1363595007 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7efe6806af60 0x7efe6806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:40.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.608+0000 7efe76ffd640 1 -- 172.21.15.57:0/1363595007 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(268..268 src has 1..268) v4 ==== 6108+0+0 (secure 0 0 0) 0x7efe7c0ac320 con 0x7efe881534b0 2023-10-19T14:22:40.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.608+0000 7efe8659c640 1 --2- 172.21.15.57:0/1363595007 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7efe6806af60 0x7efe6806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:40.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.609+0000 7efe8659c640 1 --2- 172.21.15.57:0/1363595007 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7efe6806af60 0x7efe6806d420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7efe88152780 tx=0x7efe78009040 comp rx=0 tx=0).ready entity=mgr.10520 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:40.616 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.612+0000 7efe76ffd640 1 -- 172.21.15.57:0/1363595007 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7efe8815af50 con 0x7efe881534b0 2023-10-19T14:22:40.793 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.791+0000 7efe8cb73640 1 -- 172.21.15.57:0/1363595007 --> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7efe8815b8f0 con 0x7efe6806af60 2023-10-19T14:22:40.794 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.791+0000 7efe76ffd640 1 -- 172.21.15.57:0/1363595007 <== mgr.10520 v2:172.21.15.179:6832/892689897 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7efe8815b8f0 con 0x7efe6806af60 2023-10-19T14:22:40.794 INFO:teuthology.orchestra.run.smithi057.stderr:instructed osd(s) 2 to deep-scrub 2023-10-19T14:22:40.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.793+0000 7efe8cb73640 1 -- 172.21.15.57:0/1363595007 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7efe6806af60 msgr2=0x7efe6806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:40.796 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.793+0000 7efe8cb73640 1 --2- 172.21.15.57:0/1363595007 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7efe6806af60 0x7efe6806d420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7efe88152780 tx=0x7efe78009040 comp rx=0 tx=0).stop 2023-10-19T14:22:40.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.793+0000 7efe8cb73640 1 -- 172.21.15.57:0/1363595007 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efe881534b0 msgr2=0x7efe880aa030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:40.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.793+0000 7efe8cb73640 1 --2- 172.21.15.57:0/1363595007 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efe881534b0 0x7efe880aa030 secure :-1 s=READY pgs=3050 cs=0 l=1 rev1=1 crypto rx=0x7efe7c009850 tx=0x7efe7c02da50 comp rx=0 tx=0).stop 2023-10-19T14:22:40.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.793+0000 7efe8cb73640 1 -- 172.21.15.57:0/1363595007 shutdown_connections 2023-10-19T14:22:40.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.793+0000 7efe8cb73640 1 --2- 172.21.15.57:0/1363595007 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7efe6806af60 0x7efe6806d420 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:40.797 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.793+0000 7efe8cb73640 1 --2- 172.21.15.57:0/1363595007 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efe88159400 0x7efe880aa570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:40.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.793+0000 7efe8cb73640 1 --2- 172.21.15.57:0/1363595007 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efe881534b0 0x7efe880aa030 unknown :-1 s=CLOSED pgs=3050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:40.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.793+0000 7efe8cb73640 1 --2- 172.21.15.57:0/1363595007 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7efe880aacd0 0x7efe881f0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:40.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.793+0000 7efe8cb73640 1 -- 172.21.15.57:0/1363595007 >> 172.21.15.57:0/1363595007 conn(0x7efe88069810 msgr2=0x7efe88157750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:40.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.793+0000 7efe8cb73640 1 -- 172.21.15.57:0/1363595007 shutdown_connections 2023-10-19T14:22:40.798 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.793+0000 7efe8cb73640 1 -- 172.21.15.57:0/1363595007 wait complete. 2023-10-19T14:22:40.807 INFO:tasks.ceph:Scrubbing osd.3 2023-10-19T14:22:40.807 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2023-10-19T14:22:40.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.918+0000 7f2267c4f640 1 -- 172.21.15.57:0/4027839569 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2248003970 msgr2=0x7f2248003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:40.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.918+0000 7f2267c4f640 1 --2- 172.21.15.57:0/4027839569 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2248003970 0x7f2248003d90 secure :-1 s=READY pgs=3052 cs=0 l=1 rev1=1 crypto rx=0x7f2258006fd0 tx=0x7f22480057f0 comp rx=0 tx=0).stop 2023-10-19T14:22:40.921 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.918+0000 7f2267c4f640 1 -- 172.21.15.57:0/4027839569 shutdown_connections 2023-10-19T14:22:40.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.918+0000 7f2267c4f640 1 --2- 172.21.15.57:0/4027839569 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2248003970 0x7f2248003d90 unknown :-1 s=CLOSED pgs=3052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:40.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.918+0000 7f2267c4f640 1 -- 172.21.15.57:0/4027839569 >> 172.21.15.57:0/4027839569 conn(0x7f2260069630 msgr2=0x7f2260069a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:40.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.918+0000 7f2267c4f640 1 -- 172.21.15.57:0/4027839569 shutdown_connections 2023-10-19T14:22:40.922 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.918+0000 7f2267c4f640 1 -- 172.21.15.57:0/4027839569 wait complete. 2023-10-19T14:22:40.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.919+0000 7f2267c4f640 1 Processor -- start 2023-10-19T14:22:40.924 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.919+0000 7f2267c4f640 1 -- start start 2023-10-19T14:22:40.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.920+0000 7f2267c4f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2248003970 0x7f22601588c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:40.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.920+0000 7f2267c4f640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f226013fc60 0x7f226006ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:40.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.920+0000 7f2267c4f640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f22601400e0 0x7f2260070510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:40.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.920+0000 7f2267c4f640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2260140640 con 0x7f2248003970 2023-10-19T14:22:40.925 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.920+0000 7f2267c4f640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2260140920 con 0x7f226013fc60 2023-10-19T14:22:40.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.920+0000 7f2267c4f640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2260140c00 con 0x7f22601400e0 2023-10-19T14:22:40.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.920+0000 7f22659eb640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2248003970 0x7f22601588c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:40.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.920+0000 7f22659eb640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2248003970 0x7f22601588c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:60040/0 (socket says 172.21.15.57:60040) 2023-10-19T14:22:40.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.920+0000 7f22659eb640 1 -- 172.21.15.57:0/3550680507 learned_addr learned my addr 172.21.15.57:0/3550680507 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:22:40.926 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.920+0000 7f22659eb640 1 -- 172.21.15.57:0/3550680507 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f22601400e0 msgr2=0x7f2260070510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:40.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.920+0000 7f22661ec640 1 --2- 172.21.15.57:0/3550680507 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f22601400e0 0x7f2260070510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:40.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.920+0000 7f22651ea640 1 --2- 172.21.15.57:0/3550680507 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f226013fc60 0x7f226006ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:40.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.920+0000 7f22659eb640 1 --2- 172.21.15.57:0/3550680507 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f22601400e0 0x7f2260070510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:40.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.920+0000 7f22659eb640 1 -- 172.21.15.57:0/3550680507 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f226013fc60 msgr2=0x7f226006ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:40.927 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.920+0000 7f22659eb640 1 --2- 172.21.15.57:0/3550680507 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f226013fc60 0x7f226006ffd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:40.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.920+0000 7f22659eb640 1 -- 172.21.15.57:0/3550680507 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2260140ee0 con 0x7f2248003970 2023-10-19T14:22:40.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.920+0000 7f22659eb640 1 --2- 172.21.15.57:0/3550680507 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2248003970 0x7f22601588c0 secure :-1 s=READY pgs=3053 cs=0 l=1 rev1=1 crypto rx=0x7f225802c900 tx=0x7f2258001630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:40.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.921+0000 7f225e7fc640 1 -- 172.21.15.57:0/3550680507 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2258006230 con 0x7f2248003970 2023-10-19T14:22:40.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.921+0000 7f225e7fc640 1 -- 172.21.15.57:0/3550680507 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f2258006390 con 0x7f2248003970 2023-10-19T14:22:40.928 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.921+0000 7f2267c4f640 1 -- 172.21.15.57:0/3550680507 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22601411c0 con 0x7f2248003970 2023-10-19T14:22:40.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.921+0000 7f225e7fc640 1 -- 172.21.15.57:0/3550680507 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22580035f0 con 0x7f2248003970 2023-10-19T14:22:40.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.921+0000 7f2267c4f640 1 -- 172.21.15.57:0/3550680507 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f226006b360 con 0x7f2248003970 2023-10-19T14:22:40.929 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.921+0000 7f2267c4f640 1 -- 172.21.15.57:0/3550680507 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2260140a90 con 0x7f2248003970 2023-10-19T14:22:40.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.922+0000 7f225e7fc640 1 -- 172.21.15.57:0/3550680507 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 155) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f2258031030 con 0x7f2248003970 2023-10-19T14:22:40.930 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.922+0000 7f225e7fc640 1 --2- 172.21.15.57:0/3550680507 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f224c06af90 0x7f224c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:40.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.923+0000 7f225e7fc640 1 -- 172.21.15.57:0/3550680507 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(268..268 src has 1..268) v4 ==== 6108+0+0 (secure 0 0 0) 0x7f22580aa8f0 con 0x7f2248003970 2023-10-19T14:22:40.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.923+0000 7f225e7fc640 1 --2- 172.21.15.57:0/3550680507 >> [v2:172.21.15.57:6816/3386449871,v1:172.21.15.57:6817/3386449871] conn(0x7f224c071970 0x7f224c073dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:40.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.923+0000 7f225e7fc640 1 -- 172.21.15.57:0/3550680507 --> [v2:172.21.15.57:6816/3386449871,v1:172.21.15.57:6817/3386449871] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f224c0744a0 con 0x7f224c071970 2023-10-19T14:22:40.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.923+0000 7f225e7fc640 1 -- 172.21.15.57:0/3550680507 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_get_version_reply(handle=1 version=268) v2 ==== 24+0+0 (secure 0 0 0) 0x7f22580af2a0 con 0x7f2248003970 2023-10-19T14:22:40.931 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.923+0000 7f22661ec640 1 --2- 172.21.15.57:0/3550680507 >> [v2:172.21.15.57:6816/3386449871,v1:172.21.15.57:6817/3386449871] conn(0x7f224c071970 0x7f224c073dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:40.932 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.923+0000 7f22651ea640 1 --2- 172.21.15.57:0/3550680507 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f224c06af90 0x7f224c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:40.932 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.923+0000 7f22661ec640 1 --2- 172.21.15.57:0/3550680507 >> [v2:172.21.15.57:6816/3386449871,v1:172.21.15.57:6817/3386449871] conn(0x7f224c071970 0x7f224c073dd0 crc :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:40.932 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.923+0000 7f22651ea640 1 --2- 172.21.15.57:0/3550680507 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f224c06af90 0x7f224c06d450 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2260141f30 tx=0x7f2254005f90 comp rx=0 tx=0).ready entity=mgr.10520 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:40.932 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.924+0000 7f225e7fc640 1 -- 172.21.15.57:0/3550680507 <== osd.3 v2:172.21.15.57:6816/3386449871 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f224c0744a0 con 0x7f224c071970 2023-10-19T14:22:40.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.937+0000 7f2267c4f640 1 -- 172.21.15.57:0/3550680507 --> [v2:172.21.15.57:6816/3386449871,v1:172.21.15.57:6817/3386449871] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f22600cbb90 con 0x7f224c071970 2023-10-19T14:22:40.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.937+0000 7f225e7fc640 1 -- 172.21.15.57:0/3550680507 <== osd.3 v2:172.21.15.57:6816/3386449871 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f2250006120 con 0x7f224c071970 2023-10-19T14:22:40.940 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2023-10-19T14:22:40.940 INFO:teuthology.orchestra.run.smithi057.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-10-19T14:22:40.941 INFO:teuthology.orchestra.run.smithi057.stdout:} 2023-10-19T14:22:40.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.937+0000 7f2267c4f640 1 -- 172.21.15.57:0/3550680507 >> [v2:172.21.15.57:6816/3386449871,v1:172.21.15.57:6817/3386449871] conn(0x7f224c071970 msgr2=0x7f224c073dd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:40.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.937+0000 7f2267c4f640 1 --2- 172.21.15.57:0/3550680507 >> [v2:172.21.15.57:6816/3386449871,v1:172.21.15.57:6817/3386449871] conn(0x7f224c071970 0x7f224c073dd0 crc :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:40.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.937+0000 7f2267c4f640 1 -- 172.21.15.57:0/3550680507 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f224c06af90 msgr2=0x7f224c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:40.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.937+0000 7f2267c4f640 1 --2- 172.21.15.57:0/3550680507 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f224c06af90 0x7f224c06d450 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2260141f30 tx=0x7f2254005f90 comp rx=0 tx=0).stop 2023-10-19T14:22:40.942 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.937+0000 7f2267c4f640 1 -- 172.21.15.57:0/3550680507 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2248003970 msgr2=0x7f22601588c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:40.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.937+0000 7f2267c4f640 1 --2- 172.21.15.57:0/3550680507 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2248003970 0x7f22601588c0 secure :-1 s=READY pgs=3053 cs=0 l=1 rev1=1 crypto rx=0x7f225802c900 tx=0x7f2258001630 comp rx=0 tx=0).stop 2023-10-19T14:22:40.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.937+0000 7f22661ec640 1 -- 172.21.15.57:0/3550680507 reap_dead start 2023-10-19T14:22:40.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.937+0000 7f2267c4f640 1 -- 172.21.15.57:0/3550680507 shutdown_connections 2023-10-19T14:22:40.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.937+0000 7f2267c4f640 1 -- 172.21.15.57:0/3550680507 >> 172.21.15.57:0/3550680507 conn(0x7f2260069630 msgr2=0x7f226015bea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:40.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.937+0000 7f2267c4f640 1 -- 172.21.15.57:0/3550680507 shutdown_connections 2023-10-19T14:22:40.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:40.937+0000 7f2267c4f640 1 -- 172.21.15.57:0/3550680507 wait complete. 2023-10-19T14:22:40.950 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2023-10-19T14:22:41.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.060+0000 7f10424fc640 1 -- 172.21.15.57:0/1184952460 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f10240039a0 msgr2=0x7f1024003d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:41.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.060+0000 7f10424fc640 1 --2- 172.21.15.57:0/1184952460 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f10240039a0 0x7f1024003d80 secure :-1 s=READY pgs=3055 cs=0 l=1 rev1=1 crypto rx=0x7f102c009b50 tx=0x7f10240057f0 comp rx=0 tx=0).stop 2023-10-19T14:22:41.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.060+0000 7f1040a99640 1 -- 172.21.15.57:0/1184952460 reap_dead start 2023-10-19T14:22:41.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.060+0000 7f10424fc640 1 -- 172.21.15.57:0/1184952460 shutdown_connections 2023-10-19T14:22:41.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.060+0000 7f10424fc640 1 -- 172.21.15.57:0/1184952460 >> 172.21.15.57:0/1184952460 conn(0x7f103c069810 msgr2=0x7f103c069c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:41.064 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.060+0000 7f10424fc640 1 -- 172.21.15.57:0/1184952460 shutdown_connections 2023-10-19T14:22:41.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.060+0000 7f10424fc640 1 -- 172.21.15.57:0/1184952460 wait complete. 2023-10-19T14:22:41.067 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.061+0000 7f10424fc640 1 Processor -- start 2023-10-19T14:22:41.067 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.061+0000 7f10424fc640 1 -- start start 2023-10-19T14:22:41.067 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.062+0000 7f10424fc640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f10240039a0 0x7f103c0c8470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:41.067 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.062+0000 7f10424fc640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f103c1e7c50 0x7f103c1e80d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:41.067 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.062+0000 7f10424fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f103c1f19f0 0x7f103c1f1e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:41.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.062+0000 7f10424fc640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f103c099470 con 0x7f103c1f19f0 2023-10-19T14:22:41.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.062+0000 7f10424fc640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f103c1e8610 con 0x7f10240039a0 2023-10-19T14:22:41.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.062+0000 7f10424fc640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f103c099680 con 0x7f103c1e7c50 2023-10-19T14:22:41.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.062+0000 7f1040a99640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f103c1f19f0 0x7f103c1f1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:41.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.062+0000 7f1040a99640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f103c1f19f0 0x7f103c1f1e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:60056/0 (socket says 172.21.15.57:60056) 2023-10-19T14:22:41.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.062+0000 7f1040a99640 1 -- 172.21.15.57:0/2126046168 learned_addr learned my addr 172.21.15.57:0/2126046168 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:22:41.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.062+0000 7f103bfff640 1 --2- 172.21.15.57:0/2126046168 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f10240039a0 0x7f103c0c8470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:41.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.062+0000 7f103b7fe640 1 --2- 172.21.15.57:0/2126046168 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f103c1e7c50 0x7f103c1e80d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:41.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.062+0000 7f1040a99640 1 -- 172.21.15.57:0/2126046168 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f103c1e7c50 msgr2=0x7f103c1e80d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:41.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.062+0000 7f1040a99640 1 --2- 172.21.15.57:0/2126046168 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f103c1e7c50 0x7f103c1e80d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:41.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.062+0000 7f1040a99640 1 -- 172.21.15.57:0/2126046168 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f10240039a0 msgr2=0x7f103c0c8470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:41.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.062+0000 7f1040a99640 1 --2- 172.21.15.57:0/2126046168 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f10240039a0 0x7f103c0c8470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:41.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.062+0000 7f1040a99640 1 -- 172.21.15.57:0/2126046168 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f103c099960 con 0x7f103c1f19f0 2023-10-19T14:22:41.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.062+0000 7f103bfff640 1 --2- 172.21.15.57:0/2126046168 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f10240039a0 0x7f103c0c8470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:22:41.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.062+0000 7f103b7fe640 1 --2- 172.21.15.57:0/2126046168 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f103c1e7c50 0x7f103c1e80d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:22:41.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.063+0000 7f1040a99640 1 --2- 172.21.15.57:0/2126046168 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f103c1f19f0 0x7f103c1f1e70 secure :-1 s=READY pgs=3056 cs=0 l=1 rev1=1 crypto rx=0x7f103c0afa80 tx=0x7f1034004f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:41.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.063+0000 7f1038ff9640 1 -- 172.21.15.57:0/2126046168 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1034017070 con 0x7f103c1f19f0 2023-10-19T14:22:41.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.063+0000 7f1038ff9640 1 -- 172.21.15.57:0/2126046168 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f1034010690 con 0x7f103c1f19f0 2023-10-19T14:22:41.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.063+0000 7f10424fc640 1 -- 172.21.15.57:0/2126046168 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f103c099c40 con 0x7f103c1f19f0 2023-10-19T14:22:41.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.063+0000 7f1038ff9640 1 -- 172.21.15.57:0/2126046168 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1034010a80 con 0x7f103c1f19f0 2023-10-19T14:22:41.072 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.063+0000 7f10424fc640 1 -- 172.21.15.57:0/2126046168 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f103c09a440 con 0x7f103c1f19f0 2023-10-19T14:22:41.072 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.064+0000 7f10424fc640 1 -- 172.21.15.57:0/2126046168 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f103c09a160 con 0x7f103c1f19f0 2023-10-19T14:22:41.072 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.064+0000 7f1038ff9640 1 -- 172.21.15.57:0/2126046168 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 155) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f1034010be0 con 0x7f103c1f19f0 2023-10-19T14:22:41.073 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.065+0000 7f1038ff9640 1 --2- 172.21.15.57:0/2126046168 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f102806af90 0x7f102806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:41.073 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.065+0000 7f1038ff9640 1 -- 172.21.15.57:0/2126046168 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(268..268 src has 1..268) v4 ==== 6108+0+0 (secure 0 0 0) 0x7f1034089a40 con 0x7f103c1f19f0 2023-10-19T14:22:41.073 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.065+0000 7f103bfff640 1 --2- 172.21.15.57:0/2126046168 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f102806af90 0x7f102806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:41.074 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.065+0000 7f103bfff640 1 --2- 172.21.15.57:0/2126046168 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f102806af90 0x7f102806d450 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f102c009b50 tx=0x7f102c035040 comp rx=0 tx=0).ready entity=mgr.10520 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:41.074 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.070+0000 7f1038ff9640 1 -- 172.21.15.57:0/2126046168 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f103c09a160 con 0x7f103c1f19f0 2023-10-19T14:22:41.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.247+0000 7f10424fc640 1 -- 172.21.15.57:0/2126046168 --> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f103c071110 con 0x7f102806af90 2023-10-19T14:22:41.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.248+0000 7f1038ff9640 1 -- 172.21.15.57:0/2126046168 <== mgr.10520 v2:172.21.15.179:6832/892689897 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f102c004020 con 0x7f102806af90 2023-10-19T14:22:41.251 INFO:teuthology.orchestra.run.smithi057.stderr:instructed osd(s) 3 to deep-scrub 2023-10-19T14:22:41.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.250+0000 7f10424fc640 1 -- 172.21.15.57:0/2126046168 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f102806af90 msgr2=0x7f102806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:41.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.250+0000 7f10424fc640 1 --2- 172.21.15.57:0/2126046168 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f102806af90 0x7f102806d450 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f102c009b50 tx=0x7f102c035040 comp rx=0 tx=0).stop 2023-10-19T14:22:41.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.250+0000 7f10424fc640 1 -- 172.21.15.57:0/2126046168 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f103c1f19f0 msgr2=0x7f103c1f1e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:41.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.250+0000 7f10424fc640 1 --2- 172.21.15.57:0/2126046168 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f103c1f19f0 0x7f103c1f1e70 secure :-1 s=READY pgs=3056 cs=0 l=1 rev1=1 crypto rx=0x7f103c0afa80 tx=0x7f1034004f90 comp rx=0 tx=0).stop 2023-10-19T14:22:41.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.250+0000 7f10424fc640 1 -- 172.21.15.57:0/2126046168 shutdown_connections 2023-10-19T14:22:41.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.250+0000 7f10424fc640 1 --2- 172.21.15.57:0/2126046168 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f102806af90 0x7f102806d450 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:41.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.250+0000 7f10424fc640 1 --2- 172.21.15.57:0/2126046168 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f103c1f19f0 0x7f103c1f1e70 unknown :-1 s=CLOSED pgs=3056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:41.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.250+0000 7f10424fc640 1 --2- 172.21.15.57:0/2126046168 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f103c1e7c50 0x7f103c1e80d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:41.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.250+0000 7f10424fc640 1 --2- 172.21.15.57:0/2126046168 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f10240039a0 0x7f103c0c8470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:41.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.250+0000 7f10424fc640 1 -- 172.21.15.57:0/2126046168 >> 172.21.15.57:0/2126046168 conn(0x7f103c069810 msgr2=0x7f103c13c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:41.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.250+0000 7f10424fc640 1 -- 172.21.15.57:0/2126046168 shutdown_connections 2023-10-19T14:22:41.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.250+0000 7f10424fc640 1 -- 172.21.15.57:0/2126046168 wait complete. 2023-10-19T14:22:41.263 INFO:tasks.ceph:Scrubbing osd.4 2023-10-19T14:22:41.263 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2023-10-19T14:22:41.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.374+0000 7f3f8f516640 1 -- 172.21.15.57:0/3501083888 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3f880ac8c0 msgr2=0x7f3f8806a810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:41.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.374+0000 7f3f8f516640 1 --2- 172.21.15.57:0/3501083888 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3f880ac8c0 0x7f3f8806a810 secure :-1 s=READY pgs=3057 cs=0 l=1 rev1=1 crypto rx=0x7f3f8806b140 tx=0x7f3f7c02bd30 comp rx=0 tx=0).stop 2023-10-19T14:22:41.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.374+0000 7f3f8f516640 1 -- 172.21.15.57:0/3501083888 shutdown_connections 2023-10-19T14:22:41.377 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.374+0000 7f3f8f516640 1 --2- 172.21.15.57:0/3501083888 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3f880ac8c0 0x7f3f8806a810 unknown :-1 s=CLOSED pgs=3057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:41.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.374+0000 7f3f8f516640 1 --2- 172.21.15.57:0/3501083888 >> v2:172.21.15.179:3300/0 conn(0x7f3f8815b1a0 0x7f3f8806a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:41.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.374+0000 7f3f8f516640 1 -- 172.21.15.57:0/3501083888 >> 172.21.15.57:0/3501083888 conn(0x7f3f880aef90 msgr2=0x7f3f880af370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:41.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.375+0000 7f3f8f516640 1 -- 172.21.15.57:0/3501083888 shutdown_connections 2023-10-19T14:22:41.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.375+0000 7f3f8f516640 1 -- 172.21.15.57:0/3501083888 wait complete. 2023-10-19T14:22:41.381 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.376+0000 7f3f8f516640 1 Processor -- start 2023-10-19T14:22:41.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.376+0000 7f3f8f516640 1 -- start start 2023-10-19T14:22:41.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.376+0000 7f3f8f516640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3f88065d20 0x7f3f88147770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:41.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.376+0000 7f3f8f516640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3f880ac8c0 0x7f3f880ab340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:41.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.376+0000 7f3f8f516640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3f8815b1a0 0x7f3f88141dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:41.382 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.376+0000 7f3f8f516640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f88147cb0 con 0x7f3f88065d20 2023-10-19T14:22:41.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.376+0000 7f3f8f516640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f880aab80 con 0x7f3f880ac8c0 2023-10-19T14:22:41.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.376+0000 7f3f8f516640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f8815b7f0 con 0x7f3f8815b1a0 2023-10-19T14:22:41.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.377+0000 7f3f8d2b2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3f88065d20 0x7f3f88147770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:41.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.377+0000 7f3f8d2b2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3f88065d20 0x7f3f88147770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:60074/0 (socket says 172.21.15.57:60074) 2023-10-19T14:22:41.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.377+0000 7f3f8d2b2640 1 -- 172.21.15.57:0/1911260006 learned_addr learned my addr 172.21.15.57:0/1911260006 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:22:41.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.377+0000 7f3f8cab1640 1 --2- 172.21.15.57:0/1911260006 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3f880ac8c0 0x7f3f880ab340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:41.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.377+0000 7f3f8d2b2640 1 -- 172.21.15.57:0/1911260006 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3f8815b1a0 msgr2=0x7f3f88141dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:41.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.377+0000 7f3f8dab3640 1 --2- 172.21.15.57:0/1911260006 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3f8815b1a0 0x7f3f88141dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:41.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.377+0000 7f3f8d2b2640 1 --2- 172.21.15.57:0/1911260006 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f3f8815b1a0 0x7f3f88141dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:41.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.377+0000 7f3f8d2b2640 1 -- 172.21.15.57:0/1911260006 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3f880ac8c0 msgr2=0x7f3f880ab340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:41.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.377+0000 7f3f8d2b2640 1 --2- 172.21.15.57:0/1911260006 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3f880ac8c0 0x7f3f880ab340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:41.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.377+0000 7f3f8d2b2640 1 -- 172.21.15.57:0/1911260006 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f7c009560 con 0x7f3f88065d20 2023-10-19T14:22:41.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.377+0000 7f3f8cab1640 1 --2- 172.21.15.57:0/1911260006 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3f880ac8c0 0x7f3f880ab340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:22:41.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.377+0000 7f3f8d2b2640 1 --2- 172.21.15.57:0/1911260006 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3f88065d20 0x7f3f88147770 secure :-1 s=READY pgs=3058 cs=0 l=1 rev1=1 crypto rx=0x7f3f8000d6d0 tx=0x7f3f8000db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:41.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.377+0000 7f3f85ffb640 1 -- 172.21.15.57:0/1911260006 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f80005220 con 0x7f3f88065d20 2023-10-19T14:22:41.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.377+0000 7f3f85ffb640 1 -- 172.21.15.57:0/1911260006 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f3f80005490 con 0x7f3f88065d20 2023-10-19T14:22:41.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.377+0000 7f3f8f516640 1 -- 172.21.15.57:0/1911260006 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f8815bad0 con 0x7f3f88065d20 2023-10-19T14:22:41.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.377+0000 7f3f85ffb640 1 -- 172.21.15.57:0/1911260006 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f80009a80 con 0x7f3f88065d20 2023-10-19T14:22:41.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.377+0000 7f3f8f516640 1 -- 172.21.15.57:0/1911260006 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f880aae20 con 0x7f3f88065d20 2023-10-19T14:22:41.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.378+0000 7f3f8f516640 1 -- 172.21.15.57:0/1911260006 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3f880c6d20 con 0x7f3f88065d20 2023-10-19T14:22:41.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.379+0000 7f3f85ffb640 1 -- 172.21.15.57:0/1911260006 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 155) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f3f800047a0 con 0x7f3f88065d20 2023-10-19T14:22:41.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.379+0000 7f3f85ffb640 1 --2- 172.21.15.57:0/1911260006 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f3f7406af60 0x7f3f7406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:41.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.379+0000 7f3f85ffb640 1 -- 172.21.15.57:0/1911260006 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(268..268 src has 1..268) v4 ==== 6108+0+0 (secure 0 0 0) 0x7f3f80089e80 con 0x7f3f88065d20 2023-10-19T14:22:41.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.379+0000 7f3f85ffb640 1 --2- 172.21.15.57:0/1911260006 >> [v2:172.21.15.179:6805/4179727948,v1:172.21.15.179:6807/4179727948] conn(0x7f3f74071940 0x7f3f74073da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:41.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.379+0000 7f3f85ffb640 1 -- 172.21.15.57:0/1911260006 --> [v2:172.21.15.179:6805/4179727948,v1:172.21.15.179:6807/4179727948] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3f74074470 con 0x7f3f74071940 2023-10-19T14:22:41.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.379+0000 7f3f85ffb640 1 -- 172.21.15.57:0/1911260006 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_get_version_reply(handle=1 version=268) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3f8008c540 con 0x7f3f88065d20 2023-10-19T14:22:41.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.379+0000 7f3f8cab1640 1 --2- 172.21.15.57:0/1911260006 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f3f7406af60 0x7f3f7406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:41.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.379+0000 7f3f8dab3640 1 --2- 172.21.15.57:0/1911260006 >> [v2:172.21.15.179:6805/4179727948,v1:172.21.15.179:6807/4179727948] conn(0x7f3f74071940 0x7f3f74073da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:41.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.380+0000 7f3f8cab1640 1 --2- 172.21.15.57:0/1911260006 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f3f7406af60 0x7f3f7406d420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f3f7c035000 tx=0x7f3f7c004730 comp rx=0 tx=0).ready entity=mgr.10520 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:41.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.380+0000 7f3f8dab3640 1 --2- 172.21.15.57:0/1911260006 >> [v2:172.21.15.179:6805/4179727948,v1:172.21.15.179:6807/4179727948] conn(0x7f3f74071940 0x7f3f74073da0 crc :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:41.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.381+0000 7f3f85ffb640 1 -- 172.21.15.57:0/1911260006 <== osd.4 v2:172.21.15.179:6805/4179727948 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f3f74074470 con 0x7f3f74071940 2023-10-19T14:22:41.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.393+0000 7f3f8f516640 1 -- 172.21.15.57:0/1911260006 --> [v2:172.21.15.179:6805/4179727948,v1:172.21.15.179:6807/4179727948] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f3f88069100 con 0x7f3f74071940 2023-10-19T14:22:41.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.394+0000 7f3f85ffb640 1 -- 172.21.15.57:0/1911260006 <== osd.4 v2:172.21.15.179:6805/4179727948 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f3f88069100 con 0x7f3f74071940 2023-10-19T14:22:41.396 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2023-10-19T14:22:41.397 INFO:teuthology.orchestra.run.smithi057.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-10-19T14:22:41.397 INFO:teuthology.orchestra.run.smithi057.stdout:} 2023-10-19T14:22:41.398 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.394+0000 7f3f8f516640 1 -- 172.21.15.57:0/1911260006 >> [v2:172.21.15.179:6805/4179727948,v1:172.21.15.179:6807/4179727948] conn(0x7f3f74071940 msgr2=0x7f3f74073da0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:41.399 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.394+0000 7f3f8f516640 1 --2- 172.21.15.57:0/1911260006 >> [v2:172.21.15.179:6805/4179727948,v1:172.21.15.179:6807/4179727948] conn(0x7f3f74071940 0x7f3f74073da0 crc :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:41.399 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.394+0000 7f3f8f516640 1 -- 172.21.15.57:0/1911260006 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f3f7406af60 msgr2=0x7f3f7406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:41.399 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.394+0000 7f3f8f516640 1 --2- 172.21.15.57:0/1911260006 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f3f7406af60 0x7f3f7406d420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f3f7c035000 tx=0x7f3f7c004730 comp rx=0 tx=0).stop 2023-10-19T14:22:41.400 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.394+0000 7f3f8f516640 1 -- 172.21.15.57:0/1911260006 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3f88065d20 msgr2=0x7f3f88147770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:41.400 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.394+0000 7f3f8f516640 1 --2- 172.21.15.57:0/1911260006 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3f88065d20 0x7f3f88147770 secure :-1 s=READY pgs=3058 cs=0 l=1 rev1=1 crypto rx=0x7f3f8000d6d0 tx=0x7f3f8000db90 comp rx=0 tx=0).stop 2023-10-19T14:22:41.400 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.394+0000 7f3f8dab3640 1 -- 172.21.15.57:0/1911260006 reap_dead start 2023-10-19T14:22:41.400 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.394+0000 7f3f8f516640 1 -- 172.21.15.57:0/1911260006 shutdown_connections 2023-10-19T14:22:41.401 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.394+0000 7f3f8f516640 1 -- 172.21.15.57:0/1911260006 >> 172.21.15.57:0/1911260006 conn(0x7f3f880aef90 msgr2=0x7f3f880ae880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:41.401 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.395+0000 7f3f8f516640 1 -- 172.21.15.57:0/1911260006 shutdown_connections 2023-10-19T14:22:41.401 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.395+0000 7f3f8f516640 1 -- 172.21.15.57:0/1911260006 wait complete. 2023-10-19T14:22:41.406 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2023-10-19T14:22:41.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.514+0000 7f221f38e640 1 -- 172.21.15.57:0/3782238572 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f22181417e0 msgr2=0x7f2218070a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:41.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.514+0000 7f221f38e640 1 --2- 172.21.15.57:0/3782238572 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f22181417e0 0x7f2218070a00 secure :-1 s=READY pgs=3059 cs=0 l=1 rev1=1 crypto rx=0x7f2214005fd0 tx=0x7f221402be10 comp rx=0 tx=0).stop 2023-10-19T14:22:41.517 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.515+0000 7f221f38e640 1 -- 172.21.15.57:0/3782238572 shutdown_connections 2023-10-19T14:22:41.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.515+0000 7f221f38e640 1 --2- 172.21.15.57:0/3782238572 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f221806f2a0 0x7f22180b0c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:41.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.515+0000 7f221f38e640 1 --2- 172.21.15.57:0/3782238572 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f22181417e0 0x7f2218070a00 unknown :-1 s=CLOSED pgs=3059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:41.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.515+0000 7f221f38e640 1 -- 172.21.15.57:0/3782238572 >> 172.21.15.57:0/3782238572 conn(0x7f2218069810 msgr2=0x7f2218069c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:41.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.515+0000 7f221f38e640 1 -- 172.21.15.57:0/3782238572 shutdown_connections 2023-10-19T14:22:41.518 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.515+0000 7f221f38e640 1 -- 172.21.15.57:0/3782238572 wait complete. 2023-10-19T14:22:41.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.516+0000 7f221f38e640 1 Processor -- start 2023-10-19T14:22:41.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.516+0000 7f221f38e640 1 -- start start 2023-10-19T14:22:41.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.517+0000 7f221f38e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f221806f2a0 0x7f22180c8ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:41.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.517+0000 7f221f38e640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f22180b1820 0x7f2218157e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:41.521 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.517+0000 7f221f38e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f22181417e0 0x7f22180c72a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:41.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.517+0000 7f221f38e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22180c9400 con 0x7f221806f2a0 2023-10-19T14:22:41.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.517+0000 7f221f38e640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22180c6d50 con 0x7f22181417e0 2023-10-19T14:22:41.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.517+0000 7f221f38e640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22180c7e20 con 0x7f22180b1820 2023-10-19T14:22:41.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.517+0000 7f221d12a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f221806f2a0 0x7f22180c8ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:41.522 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.517+0000 7f221d12a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f221806f2a0 0x7f22180c8ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:60096/0 (socket says 172.21.15.57:60096) 2023-10-19T14:22:41.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.517+0000 7f221d12a640 1 -- 172.21.15.57:0/3179269114 learned_addr learned my addr 172.21.15.57:0/3179269114 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:22:41.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.517+0000 7f221c929640 1 --2- 172.21.15.57:0/3179269114 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f22180b1820 0x7f2218157e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:41.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.517+0000 7f221d92b640 1 --2- 172.21.15.57:0/3179269114 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f22181417e0 0x7f22180c72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:41.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.517+0000 7f221d12a640 1 -- 172.21.15.57:0/3179269114 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f22180b1820 msgr2=0x7f2218157e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:41.523 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.517+0000 7f221d12a640 1 --2- 172.21.15.57:0/3179269114 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f22180b1820 0x7f2218157e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:41.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.517+0000 7f221d12a640 1 -- 172.21.15.57:0/3179269114 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f22181417e0 msgr2=0x7f22180c72a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:41.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.517+0000 7f221d12a640 1 --2- 172.21.15.57:0/3179269114 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f22181417e0 0x7f22180c72a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:41.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.517+0000 7f221d12a640 1 -- 172.21.15.57:0/3179269114 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2214009560 con 0x7f221806f2a0 2023-10-19T14:22:41.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.517+0000 7f221d92b640 1 --2- 172.21.15.57:0/3179269114 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f22181417e0 0x7f22180c72a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:22:41.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.517+0000 7f221c929640 1 --2- 172.21.15.57:0/3179269114 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f22180b1820 0x7f2218157e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-19T14:22:41.524 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.517+0000 7f221d12a640 1 --2- 172.21.15.57:0/3179269114 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f221806f2a0 0x7f22180c8ec0 secure :-1 s=READY pgs=3060 cs=0 l=1 rev1=1 crypto rx=0x7f2214006010 tx=0x7f221402dbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:41.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.517+0000 7f220dffb640 1 -- 172.21.15.57:0/3179269114 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22140035f0 con 0x7f221806f2a0 2023-10-19T14:22:41.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.517+0000 7f220dffb640 1 -- 172.21.15.57:0/3179269114 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f2214003840 con 0x7f221806f2a0 2023-10-19T14:22:41.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.517+0000 7f221f38e640 1 -- 172.21.15.57:0/3179269114 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22180b21c0 con 0x7f221806f2a0 2023-10-19T14:22:41.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.517+0000 7f220dffb640 1 -- 172.21.15.57:0/3179269114 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2214003c50 con 0x7f221806f2a0 2023-10-19T14:22:41.525 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.518+0000 7f221f38e640 1 -- 172.21.15.57:0/3179269114 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22180c83e0 con 0x7f221806f2a0 2023-10-19T14:22:41.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.519+0000 7f221f38e640 1 -- 172.21.15.57:0/3179269114 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22180c95a0 con 0x7f221806f2a0 2023-10-19T14:22:41.526 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.519+0000 7f220dffb640 1 -- 172.21.15.57:0/3179269114 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 155) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f221403e450 con 0x7f221806f2a0 2023-10-19T14:22:41.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.519+0000 7f220dffb640 1 --2- 172.21.15.57:0/3179269114 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f21fc06af60 0x7f21fc06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:41.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.520+0000 7f220dffb640 1 -- 172.21.15.57:0/3179269114 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(268..268 src has 1..268) v4 ==== 6108+0+0 (secure 0 0 0) 0x7f22140ac5f0 con 0x7f221806f2a0 2023-10-19T14:22:41.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.520+0000 7f221c929640 1 --2- 172.21.15.57:0/3179269114 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f21fc06af60 0x7f21fc06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:41.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.520+0000 7f221c929640 1 --2- 172.21.15.57:0/3179269114 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f21fc06af60 0x7f21fc06d420 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2218133fb0 tx=0x7f2208009530 comp rx=0 tx=0).ready entity=mgr.10520 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:41.527 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.524+0000 7f220dffb640 1 -- 172.21.15.57:0/3179269114 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f22140774a0 con 0x7f221806f2a0 2023-10-19T14:22:41.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.698+0000 7f221f38e640 1 -- 172.21.15.57:0/3179269114 --> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f2218069100 con 0x7f21fc06af60 2023-10-19T14:22:41.700 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.698+0000 7f220dffb640 1 -- 172.21.15.57:0/3179269114 <== mgr.10520 v2:172.21.15.179:6832/892689897 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f2218069100 con 0x7f21fc06af60 2023-10-19T14:22:41.701 INFO:teuthology.orchestra.run.smithi057.stderr:instructed osd(s) 4 to deep-scrub 2023-10-19T14:22:41.702 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.700+0000 7f221f38e640 1 -- 172.21.15.57:0/3179269114 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f21fc06af60 msgr2=0x7f21fc06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:41.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.700+0000 7f221f38e640 1 --2- 172.21.15.57:0/3179269114 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f21fc06af60 0x7f21fc06d420 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2218133fb0 tx=0x7f2208009530 comp rx=0 tx=0).stop 2023-10-19T14:22:41.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.700+0000 7f221f38e640 1 -- 172.21.15.57:0/3179269114 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f221806f2a0 msgr2=0x7f22180c8ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:41.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.700+0000 7f221f38e640 1 --2- 172.21.15.57:0/3179269114 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f221806f2a0 0x7f22180c8ec0 secure :-1 s=READY pgs=3060 cs=0 l=1 rev1=1 crypto rx=0x7f2214006010 tx=0x7f221402dbb0 comp rx=0 tx=0).stop 2023-10-19T14:22:41.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.700+0000 7f221f38e640 1 -- 172.21.15.57:0/3179269114 shutdown_connections 2023-10-19T14:22:41.703 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.700+0000 7f221f38e640 1 --2- 172.21.15.57:0/3179269114 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f21fc06af60 0x7f21fc06d420 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:41.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.700+0000 7f221f38e640 1 --2- 172.21.15.57:0/3179269114 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f22181417e0 0x7f22180c72a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:41.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.700+0000 7f221f38e640 1 --2- 172.21.15.57:0/3179269114 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f22180b1820 0x7f2218157e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:41.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.700+0000 7f221f38e640 1 --2- 172.21.15.57:0/3179269114 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f221806f2a0 0x7f22180c8ec0 unknown :-1 s=CLOSED pgs=3060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:41.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.700+0000 7f221f38e640 1 -- 172.21.15.57:0/3179269114 >> 172.21.15.57:0/3179269114 conn(0x7f2218069810 msgr2=0x7f2218099a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:41.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.700+0000 7f221f38e640 1 -- 172.21.15.57:0/3179269114 shutdown_connections 2023-10-19T14:22:41.705 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.700+0000 7f221f38e640 1 -- 172.21.15.57:0/3179269114 wait complete. 2023-10-19T14:22:41.713 INFO:tasks.ceph:Scrubbing osd.5 2023-10-19T14:22:41.713 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2023-10-19T14:22:41.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.824+0000 7fd8fed39640 1 -- 172.21.15.57:0/3683990829 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd8f806c2d0 msgr2=0x7fd8f8157a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:41.826 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.824+0000 7fd8fed39640 1 --2- 172.21.15.57:0/3683990829 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd8f806c2d0 0x7fd8f8157a00 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec00a940 tx=0x7fd8ec02d090 comp rx=0 tx=0).stop 2023-10-19T14:22:41.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.824+0000 7fd8fed39640 1 -- 172.21.15.57:0/3683990829 shutdown_connections 2023-10-19T14:22:41.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.824+0000 7fd8fed39640 1 --2- 172.21.15.57:0/3683990829 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd8f80b1bb0 0x7fd8f815b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:41.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.824+0000 7fd8fed39640 1 --2- 172.21.15.57:0/3683990829 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd8f806c2d0 0x7fd8f8157a00 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:41.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.824+0000 7fd8fed39640 1 -- 172.21.15.57:0/3683990829 >> 172.21.15.57:0/3683990829 conn(0x7fd8f8069630 msgr2=0x7fd8f8069a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:41.827 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.824+0000 7fd8fed39640 1 -- 172.21.15.57:0/3683990829 shutdown_connections 2023-10-19T14:22:41.828 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.824+0000 7fd8fed39640 1 -- 172.21.15.57:0/3683990829 wait complete. 2023-10-19T14:22:41.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.825+0000 7fd8fed39640 1 Processor -- start 2023-10-19T14:22:41.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.825+0000 7fd8fed39640 1 -- start start 2023-10-19T14:22:41.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.826+0000 7fd8fed39640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd8f806c2d0 0x7fd8f814d100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:41.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.826+0000 7fd8fed39640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd8f80b1bb0 0x7fd8f814f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:41.830 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.826+0000 7fd8fed39640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd8f815bc40 0x7fd8f8150860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:41.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.826+0000 7fd8fed39640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f80ab760 con 0x7fd8f80b1bb0 2023-10-19T14:22:41.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.826+0000 7fd8fed39640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f8155830 con 0x7fd8f806c2d0 2023-10-19T14:22:41.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.826+0000 7fd8fed39640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f8155b10 con 0x7fd8f815bc40 2023-10-19T14:22:41.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.826+0000 7fd8f7fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd8f80b1bb0 0x7fd8f814f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:41.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.826+0000 7fd8f7fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd8f80b1bb0 0x7fd8f814f850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:60104/0 (socket says 172.21.15.57:60104) 2023-10-19T14:22:41.831 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.826+0000 7fd8f7fff640 1 -- 172.21.15.57:0/92935775 learned_addr learned my addr 172.21.15.57:0/92935775 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:22:41.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.826+0000 7fd8fd2d6640 1 --2- 172.21.15.57:0/92935775 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd8f815bc40 0x7fd8f8150860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:41.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.826+0000 7fd8fcad5640 1 --2- 172.21.15.57:0/92935775 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd8f806c2d0 0x7fd8f814d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:41.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.826+0000 7fd8f7fff640 1 -- 172.21.15.57:0/92935775 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd8f815bc40 msgr2=0x7fd8f8150860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:41.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.826+0000 7fd8f7fff640 1 --2- 172.21.15.57:0/92935775 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd8f815bc40 0x7fd8f8150860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:41.832 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.826+0000 7fd8f7fff640 1 -- 172.21.15.57:0/92935775 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd8f806c2d0 msgr2=0x7fd8f814d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:41.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.826+0000 7fd8f7fff640 1 --2- 172.21.15.57:0/92935775 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd8f806c2d0 0x7fd8f814d100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:41.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.826+0000 7fd8f7fff640 1 -- 172.21.15.57:0/92935775 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8e4009560 con 0x7fd8f80b1bb0 2023-10-19T14:22:41.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.827+0000 7fd8f7fff640 1 --2- 172.21.15.57:0/92935775 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd8f80b1bb0 0x7fd8f814f850 secure :-1 s=READY pgs=3061 cs=0 l=1 rev1=1 crypto rx=0x7fd8e400e9c0 tx=0x7fd8e400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:41.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.827+0000 7fd8f57fa640 1 -- 172.21.15.57:0/92935775 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8e400cde0 con 0x7fd8f80b1bb0 2023-10-19T14:22:41.833 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.827+0000 7fd8f57fa640 1 -- 172.21.15.57:0/92935775 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fd8e4004020 con 0x7fd8f80b1bb0 2023-10-19T14:22:41.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.827+0000 7fd8fed39640 1 -- 172.21.15.57:0/92935775 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8ec00a5a0 con 0x7fd8f80b1bb0 2023-10-19T14:22:41.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.827+0000 7fd8f57fa640 1 -- 172.21.15.57:0/92935775 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8e4004410 con 0x7fd8f80b1bb0 2023-10-19T14:22:41.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.827+0000 7fd8fed39640 1 -- 172.21.15.57:0/92935775 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8f81554e0 con 0x7fd8f80b1bb0 2023-10-19T14:22:41.834 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.828+0000 7fd8fed39640 1 -- 172.21.15.57:0/92935775 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd8f80ccaa0 con 0x7fd8f80b1bb0 2023-10-19T14:22:41.835 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.828+0000 7fd8f57fa640 1 -- 172.21.15.57:0/92935775 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 155) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd8e40035f0 con 0x7fd8f80b1bb0 2023-10-19T14:22:41.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.829+0000 7fd8f57fa640 1 --2- 172.21.15.57:0/92935775 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fd8dc06af60 0x7fd8dc06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:41.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.829+0000 7fd8f57fa640 1 -- 172.21.15.57:0/92935775 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(268..268 src has 1..268) v4 ==== 6108+0+0 (secure 0 0 0) 0x7fd8e4014070 con 0x7fd8f80b1bb0 2023-10-19T14:22:41.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.829+0000 7fd8f57fa640 1 --2- 172.21.15.57:0/92935775 >> [v2:172.21.15.179:6800/1129921230,v1:172.21.15.179:6801/1129921230] conn(0x7fd8dc071940 0x7fd8dc073da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:41.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.829+0000 7fd8f57fa640 1 -- 172.21.15.57:0/92935775 --> [v2:172.21.15.179:6800/1129921230,v1:172.21.15.179:6801/1129921230] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd8dc074470 con 0x7fd8dc071940 2023-10-19T14:22:41.836 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.829+0000 7fd8f57fa640 1 -- 172.21.15.57:0/92935775 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_get_version_reply(handle=1 version=268) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd8e408ca70 con 0x7fd8f80b1bb0 2023-10-19T14:22:41.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.829+0000 7fd8fcad5640 1 --2- 172.21.15.57:0/92935775 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fd8dc06af60 0x7fd8dc06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:41.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.829+0000 7fd8fd2d6640 1 --2- 172.21.15.57:0/92935775 >> [v2:172.21.15.179:6800/1129921230,v1:172.21.15.179:6801/1129921230] conn(0x7fd8dc071940 0x7fd8dc073da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:41.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.829+0000 7fd8fcad5640 1 --2- 172.21.15.57:0/92935775 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fd8dc06af60 0x7fd8dc06d420 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec037000 tx=0x7fd8ec004730 comp rx=0 tx=0).ready entity=mgr.10520 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:41.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.830+0000 7fd8fd2d6640 1 --2- 172.21.15.57:0/92935775 >> [v2:172.21.15.179:6800/1129921230,v1:172.21.15.179:6801/1129921230] conn(0x7fd8dc071940 0x7fd8dc073da0 crc :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:41.837 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.830+0000 7fd8f57fa640 1 -- 172.21.15.57:0/92935775 <== osd.5 v2:172.21.15.179:6800/1129921230 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7fd8dc074470 con 0x7fd8dc071940 2023-10-19T14:22:41.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.843+0000 7fd8fed39640 1 -- 172.21.15.57:0/92935775 --> [v2:172.21.15.179:6800/1129921230,v1:172.21.15.179:6801/1129921230] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fd8f8155c80 con 0x7fd8dc071940 2023-10-19T14:22:41.845 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.844+0000 7fd8f57fa640 1 -- 172.21.15.57:0/92935775 <== osd.5 v2:172.21.15.179:6800/1129921230 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fd8f8155c80 con 0x7fd8dc071940 2023-10-19T14:22:41.846 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2023-10-19T14:22:41.846 INFO:teuthology.orchestra.run.smithi057.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-10-19T14:22:41.846 INFO:teuthology.orchestra.run.smithi057.stdout:} 2023-10-19T14:22:41.847 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.844+0000 7fd8fed39640 1 -- 172.21.15.57:0/92935775 >> [v2:172.21.15.179:6800/1129921230,v1:172.21.15.179:6801/1129921230] conn(0x7fd8dc071940 msgr2=0x7fd8dc073da0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:41.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.844+0000 7fd8fed39640 1 --2- 172.21.15.57:0/92935775 >> [v2:172.21.15.179:6800/1129921230,v1:172.21.15.179:6801/1129921230] conn(0x7fd8dc071940 0x7fd8dc073da0 crc :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:41.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.844+0000 7fd8fed39640 1 -- 172.21.15.57:0/92935775 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fd8dc06af60 msgr2=0x7fd8dc06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:41.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.844+0000 7fd8fed39640 1 --2- 172.21.15.57:0/92935775 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fd8dc06af60 0x7fd8dc06d420 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec037000 tx=0x7fd8ec004730 comp rx=0 tx=0).stop 2023-10-19T14:22:41.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.844+0000 7fd8fed39640 1 -- 172.21.15.57:0/92935775 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd8f80b1bb0 msgr2=0x7fd8f814f850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:41.848 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.844+0000 7fd8fed39640 1 --2- 172.21.15.57:0/92935775 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd8f80b1bb0 0x7fd8f814f850 secure :-1 s=READY pgs=3061 cs=0 l=1 rev1=1 crypto rx=0x7fd8e400e9c0 tx=0x7fd8e400ee80 comp rx=0 tx=0).stop 2023-10-19T14:22:41.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.844+0000 7fd8fd2d6640 1 -- 172.21.15.57:0/92935775 reap_dead start 2023-10-19T14:22:41.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.844+0000 7fd8fed39640 1 -- 172.21.15.57:0/92935775 shutdown_connections 2023-10-19T14:22:41.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.844+0000 7fd8fed39640 1 -- 172.21.15.57:0/92935775 >> 172.21.15.57:0/92935775 conn(0x7fd8f8069630 msgr2=0x7fd8f8157830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:41.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.844+0000 7fd8fed39640 1 -- 172.21.15.57:0/92935775 shutdown_connections 2023-10-19T14:22:41.849 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.844+0000 7fd8fed39640 1 -- 172.21.15.57:0/92935775 wait complete. 2023-10-19T14:22:41.856 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2023-10-19T14:22:41.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.964+0000 7fcbc4a58640 1 -- 172.21.15.57:0/3383711189 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcbac004d00 msgr2=0x7fcbac005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:41.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.964+0000 7fcbc4a58640 1 --2- 172.21.15.57:0/3383711189 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcbac004d00 0x7fcbac005180 secure :-1 s=READY pgs=3063 cs=0 l=1 rev1=1 crypto rx=0x7fcbb0009ed0 tx=0x7fcbac0057f0 comp rx=0 tx=0).stop 2023-10-19T14:22:41.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.964+0000 7fcbbed9d640 1 -- 172.21.15.57:0/3383711189 reap_dead start 2023-10-19T14:22:41.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.964+0000 7fcbc4a58640 1 -- 172.21.15.57:0/3383711189 shutdown_connections 2023-10-19T14:22:41.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.964+0000 7fcbc4a58640 1 -- 172.21.15.57:0/3383711189 >> 172.21.15.57:0/3383711189 conn(0x7fcbc0069810 msgr2=0x7fcbc0069c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:41.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.965+0000 7fcbc4a58640 1 -- 172.21.15.57:0/3383711189 shutdown_connections 2023-10-19T14:22:41.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.965+0000 7fcbc4a58640 1 -- 172.21.15.57:0/3383711189 wait complete. 2023-10-19T14:22:41.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.966+0000 7fcbc4a58640 1 Processor -- start 2023-10-19T14:22:41.970 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.966+0000 7fcbc4a58640 1 -- start start 2023-10-19T14:22:41.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.967+0000 7fcbc4a58640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcbac004d00 0x7fcbc013c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:41.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.967+0000 7fcbc4a58640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcbc00ada10 0x7fcbc00af640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:41.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.967+0000 7fcbc4a58640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcbc0098380 0x7fcbc0098800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:41.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.967+0000 7fcbc4a58640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc0098d40 con 0x7fcbc0098380 2023-10-19T14:22:41.971 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.967+0000 7fcbc4a58640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc013cab0 con 0x7fcbc00ada10 2023-10-19T14:22:41.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.967+0000 7fcbc4a58640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc00aeae0 con 0x7fcbac004d00 2023-10-19T14:22:41.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.967+0000 7fcbbed9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcbc0098380 0x7fcbc0098800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:41.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.967+0000 7fcbbed9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcbc0098380 0x7fcbc0098800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:60116/0 (socket says 172.21.15.57:60116) 2023-10-19T14:22:41.972 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.967+0000 7fcbbed9d640 1 -- 172.21.15.57:0/1379668656 learned_addr learned my addr 172.21.15.57:0/1379668656 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:22:41.973 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.967+0000 7fcbbe59c640 1 --2- 172.21.15.57:0/1379668656 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcbac004d00 0x7fcbc013c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:41.973 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.967+0000 7fcbbdd9b640 1 --2- 172.21.15.57:0/1379668656 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcbc00ada10 0x7fcbc00af640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:41.973 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.967+0000 7fcbbed9d640 1 -- 172.21.15.57:0/1379668656 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcbac004d00 msgr2=0x7fcbc013c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:41.973 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.967+0000 7fcbbed9d640 1 --2- 172.21.15.57:0/1379668656 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcbac004d00 0x7fcbc013c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:41.973 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.967+0000 7fcbbed9d640 1 -- 172.21.15.57:0/1379668656 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcbc00ada10 msgr2=0x7fcbc00af640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:41.974 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.967+0000 7fcbbed9d640 1 --2- 172.21.15.57:0/1379668656 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcbc00ada10 0x7fcbc00af640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:41.974 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.967+0000 7fcbbed9d640 1 -- 172.21.15.57:0/1379668656 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbc00adf60 con 0x7fcbc0098380 2023-10-19T14:22:41.974 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.967+0000 7fcbbed9d640 1 --2- 172.21.15.57:0/1379668656 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcbc0098380 0x7fcbc0098800 secure :-1 s=READY pgs=3064 cs=0 l=1 rev1=1 crypto rx=0x7fcbb0009ed0 tx=0x7fcbb0031750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:41.974 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.967+0000 7fcbb6ffd640 1 -- 172.21.15.57:0/1379668656 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbb003f070 con 0x7fcbc0098380 2023-10-19T14:22:41.974 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.967+0000 7fcbb6ffd640 1 -- 172.21.15.57:0/1379668656 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fcbb0003970 con 0x7fcbc0098380 2023-10-19T14:22:41.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.967+0000 7fcbb6ffd640 1 -- 172.21.15.57:0/1379668656 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbb0003d20 con 0x7fcbc0098380 2023-10-19T14:22:41.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.967+0000 7fcbc4a58640 1 -- 172.21.15.57:0/1379668656 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbc00ae240 con 0x7fcbc0098380 2023-10-19T14:22:41.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.968+0000 7fcbc4a58640 1 -- 172.21.15.57:0/1379668656 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbc00ae800 con 0x7fcbc0098380 2023-10-19T14:22:41.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.968+0000 7fcbc4a58640 1 -- 172.21.15.57:0/1379668656 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbc0069100 con 0x7fcbc0098380 2023-10-19T14:22:41.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.969+0000 7fcbb6ffd640 1 -- 172.21.15.57:0/1379668656 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 155) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fcbb003a660 con 0x7fcbc0098380 2023-10-19T14:22:41.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.969+0000 7fcbb6ffd640 1 --2- 172.21.15.57:0/1379668656 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fcba006af60 0x7fcba006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:41.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.969+0000 7fcbb6ffd640 1 -- 172.21.15.57:0/1379668656 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(268..268 src has 1..268) v4 ==== 6108+0+0 (secure 0 0 0) 0x7fcbb00ac350 con 0x7fcbc0098380 2023-10-19T14:22:41.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.970+0000 7fcbbe59c640 1 --2- 172.21.15.57:0/1379668656 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fcba006af60 0x7fcba006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:41.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.970+0000 7fcbbe59c640 1 --2- 172.21.15.57:0/1379668656 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fcba006af60 0x7fcba006d420 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fcbb8006c20 tx=0x7fcbb8001e70 comp rx=0 tx=0).ready entity=mgr.10520 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:41.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:41.974+0000 7fcbb6ffd640 1 -- 172.21.15.57:0/1379668656 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fcbb0077050 con 0x7fcbc0098380 2023-10-19T14:22:42.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.152+0000 7fcbc4a58640 1 -- 172.21.15.57:0/1379668656 --> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7fcbc0098d40 con 0x7fcba006af60 2023-10-19T14:22:42.154 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.153+0000 7fcbb6ffd640 1 -- 172.21.15.57:0/1379668656 <== mgr.10520 v2:172.21.15.179:6832/892689897 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fcbc0098d40 con 0x7fcba006af60 2023-10-19T14:22:42.155 INFO:teuthology.orchestra.run.smithi057.stderr:instructed osd(s) 5 to deep-scrub 2023-10-19T14:22:42.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.154+0000 7fcbc4a58640 1 -- 172.21.15.57:0/1379668656 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fcba006af60 msgr2=0x7fcba006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:42.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.154+0000 7fcbc4a58640 1 --2- 172.21.15.57:0/1379668656 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fcba006af60 0x7fcba006d420 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fcbb8006c20 tx=0x7fcbb8001e70 comp rx=0 tx=0).stop 2023-10-19T14:22:42.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.154+0000 7fcbc4a58640 1 -- 172.21.15.57:0/1379668656 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcbc0098380 msgr2=0x7fcbc0098800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:42.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.154+0000 7fcbc4a58640 1 --2- 172.21.15.57:0/1379668656 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcbc0098380 0x7fcbc0098800 secure :-1 s=READY pgs=3064 cs=0 l=1 rev1=1 crypto rx=0x7fcbb0009ed0 tx=0x7fcbb0031750 comp rx=0 tx=0).stop 2023-10-19T14:22:42.157 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.154+0000 7fcbc4a58640 1 -- 172.21.15.57:0/1379668656 shutdown_connections 2023-10-19T14:22:42.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.154+0000 7fcbc4a58640 1 --2- 172.21.15.57:0/1379668656 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fcba006af60 0x7fcba006d420 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:42.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.154+0000 7fcbc4a58640 1 --2- 172.21.15.57:0/1379668656 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcbc0098380 0x7fcbc0098800 unknown :-1 s=CLOSED pgs=3064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:42.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.154+0000 7fcbc4a58640 1 --2- 172.21.15.57:0/1379668656 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcbc00ada10 0x7fcbc00af640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:42.158 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.154+0000 7fcbc4a58640 1 --2- 172.21.15.57:0/1379668656 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fcbac004d00 0x7fcbc013c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:42.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.154+0000 7fcbc4a58640 1 -- 172.21.15.57:0/1379668656 >> 172.21.15.57:0/1379668656 conn(0x7fcbc0069810 msgr2=0x7fcbc00991e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:42.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.155+0000 7fcbc4a58640 1 -- 172.21.15.57:0/1379668656 shutdown_connections 2023-10-19T14:22:42.159 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.155+0000 7fcbc4a58640 1 -- 172.21.15.57:0/1379668656 wait complete. 2023-10-19T14:22:42.167 INFO:tasks.ceph:Scrubbing osd.6 2023-10-19T14:22:42.167 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2023-10-19T14:22:42.280 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.278+0000 7f82fc097640 1 -- 172.21.15.57:0/1305317713 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f82e0005180 msgr2=0x7f82e000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:42.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.278+0000 7f82fc097640 1 --2- 172.21.15.57:0/1305317713 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f82e0005180 0x7f82e000b980 secure :-1 s=READY pgs=3066 cs=0 l=1 rev1=1 crypto rx=0x7f82ec001d80 tx=0x7f82e00057f0 comp rx=0 tx=0).stop 2023-10-19T14:22:42.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.278+0000 7f82fc097640 1 -- 172.21.15.57:0/1305317713 shutdown_connections 2023-10-19T14:22:42.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.278+0000 7f82fc097640 1 --2- 172.21.15.57:0/1305317713 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f82e0005180 0x7f82e000b980 unknown :-1 s=CLOSED pgs=3066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:42.281 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.278+0000 7f82fc097640 1 -- 172.21.15.57:0/1305317713 >> 172.21.15.57:0/1305317713 conn(0x7f82f40ccf20 msgr2=0x7f82f40cd300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:42.282 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.278+0000 7f82fc097640 1 -- 172.21.15.57:0/1305317713 shutdown_connections 2023-10-19T14:22:42.282 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.278+0000 7f82fc097640 1 -- 172.21.15.57:0/1305317713 wait complete. 2023-10-19T14:22:42.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.280+0000 7f82fc097640 1 Processor -- start 2023-10-19T14:22:42.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.280+0000 7f82fc097640 1 -- start start 2023-10-19T14:22:42.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.280+0000 7f82fc097640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f82e0005180 0x7f82f4150120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:42.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.280+0000 7f82fc097640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f82f415e080 0x7f82f415e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:42.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.280+0000 7f82fc097640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f82f40cad40 0x7f82f40cb1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:42.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.280+0000 7f82fc097640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f415ea40 con 0x7f82f415e080 2023-10-19T14:22:42.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.280+0000 7f82fc097640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f40cbad0 con 0x7f82f40cad40 2023-10-19T14:22:42.285 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.280+0000 7f82fc097640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82f40c9780 con 0x7f82e0005180 2023-10-19T14:22:42.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.280+0000 7f82f9632640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f82f415e080 0x7f82f415e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:42.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.280+0000 7f82f9632640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f82f415e080 0x7f82f415e500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:60136/0 (socket says 172.21.15.57:60136) 2023-10-19T14:22:42.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.280+0000 7f82f9632640 1 -- 172.21.15.57:0/1164356393 learned_addr learned my addr 172.21.15.57:0/1164356393 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:22:42.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.281+0000 7f82fa634640 1 --2- 172.21.15.57:0/1164356393 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f82f40cad40 0x7f82f40cb1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:42.286 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.281+0000 7f82f9e33640 1 --2- 172.21.15.57:0/1164356393 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f82e0005180 0x7f82f4150120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:42.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.281+0000 7f82f9632640 1 -- 172.21.15.57:0/1164356393 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f82e0005180 msgr2=0x7f82f4150120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:42.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.281+0000 7f82f9632640 1 --2- 172.21.15.57:0/1164356393 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f82e0005180 0x7f82f4150120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:42.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.281+0000 7f82f9632640 1 -- 172.21.15.57:0/1164356393 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f82f40cad40 msgr2=0x7f82f40cb1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:42.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.281+0000 7f82f9632640 1 --2- 172.21.15.57:0/1164356393 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f82f40cad40 0x7f82f40cb1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:42.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.281+0000 7f82f9632640 1 -- 172.21.15.57:0/1164356393 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82f40c9a60 con 0x7f82f415e080 2023-10-19T14:22:42.287 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.281+0000 7f82f9632640 1 --2- 172.21.15.57:0/1164356393 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f82f415e080 0x7f82f415e500 secure :-1 s=READY pgs=3067 cs=0 l=1 rev1=1 crypto rx=0x7f82e40117d0 tx=0x7f82e4011c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:42.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.281+0000 7f82ea7fc640 1 -- 172.21.15.57:0/1164356393 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e4011ee0 con 0x7f82f415e080 2023-10-19T14:22:42.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.281+0000 7f82ea7fc640 1 -- 172.21.15.57:0/1164356393 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f82e400f940 con 0x7f82f415e080 2023-10-19T14:22:42.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.281+0000 7f82fc097640 1 -- 172.21.15.57:0/1164356393 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82f40c9d40 con 0x7f82f415e080 2023-10-19T14:22:42.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.281+0000 7f82ea7fc640 1 -- 172.21.15.57:0/1164356393 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e400fcd0 con 0x7f82f415e080 2023-10-19T14:22:42.288 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.281+0000 7f82fc097640 1 -- 172.21.15.57:0/1164356393 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82f4150660 con 0x7f82f415e080 2023-10-19T14:22:42.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.282+0000 7f82fc097640 1 -- 172.21.15.57:0/1164356393 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f82f40cde60 con 0x7f82f415e080 2023-10-19T14:22:42.289 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.282+0000 7f82ea7fc640 1 -- 172.21.15.57:0/1164356393 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 155) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f82e40035f0 con 0x7f82f415e080 2023-10-19T14:22:42.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.283+0000 7f82ea7fc640 1 --2- 172.21.15.57:0/1164356393 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f82d806af60 0x7f82d806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:42.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.283+0000 7f82ea7fc640 1 -- 172.21.15.57:0/1164356393 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(268..268 src has 1..268) v4 ==== 6108+0+0 (secure 0 0 0) 0x7f82e408a6a0 con 0x7f82f415e080 2023-10-19T14:22:42.290 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.283+0000 7f82ea7fc640 1 --2- 172.21.15.57:0/1164356393 >> [v2:172.21.15.179:6824/2898099196,v1:172.21.15.179:6825/2898099196] conn(0x7f82d8071940 0x7f82d8073da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:42.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.283+0000 7f82ea7fc640 1 -- 172.21.15.57:0/1164356393 --> [v2:172.21.15.179:6824/2898099196,v1:172.21.15.179:6825/2898099196] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f82d8074470 con 0x7f82d8071940 2023-10-19T14:22:42.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.283+0000 7f82ea7fc640 1 -- 172.21.15.57:0/1164356393 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_get_version_reply(handle=1 version=268) v2 ==== 24+0+0 (secure 0 0 0) 0x7f82e408cd60 con 0x7f82f415e080 2023-10-19T14:22:42.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.283+0000 7f82f9e33640 1 --2- 172.21.15.57:0/1164356393 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f82d806af60 0x7f82d806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:42.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.283+0000 7f82fa634640 1 --2- 172.21.15.57:0/1164356393 >> [v2:172.21.15.179:6824/2898099196,v1:172.21.15.179:6825/2898099196] conn(0x7f82d8071940 0x7f82d8073da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:42.291 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.284+0000 7f82f9e33640 1 --2- 172.21.15.57:0/1164356393 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f82d806af60 0x7f82d806d420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f82f00017a0 tx=0x7f82f0001c70 comp rx=0 tx=0).ready entity=mgr.10520 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:42.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.284+0000 7f82fa634640 1 --2- 172.21.15.57:0/1164356393 >> [v2:172.21.15.179:6824/2898099196,v1:172.21.15.179:6825/2898099196] conn(0x7f82d8071940 0x7f82d8073da0 crc :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:42.292 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.285+0000 7f82ea7fc640 1 -- 172.21.15.57:0/1164356393 <== osd.6 v2:172.21.15.179:6824/2898099196 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f82d8074470 con 0x7f82d8071940 2023-10-19T14:22:42.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.297+0000 7f82fc097640 1 -- 172.21.15.57:0/1164356393 --> [v2:172.21.15.179:6824/2898099196,v1:172.21.15.179:6825/2898099196] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f82f4069100 con 0x7f82d8071940 2023-10-19T14:22:42.299 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.298+0000 7f82ea7fc640 1 -- 172.21.15.57:0/1164356393 <== osd.6 v2:172.21.15.179:6824/2898099196 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f82f4069100 con 0x7f82d8071940 2023-10-19T14:22:42.300 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2023-10-19T14:22:42.300 INFO:teuthology.orchestra.run.smithi057.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-10-19T14:22:42.300 INFO:teuthology.orchestra.run.smithi057.stdout:} 2023-10-19T14:22:42.301 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.298+0000 7f82fc097640 1 -- 172.21.15.57:0/1164356393 >> [v2:172.21.15.179:6824/2898099196,v1:172.21.15.179:6825/2898099196] conn(0x7f82d8071940 msgr2=0x7f82d8073da0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:42.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.298+0000 7f82fc097640 1 --2- 172.21.15.57:0/1164356393 >> [v2:172.21.15.179:6824/2898099196,v1:172.21.15.179:6825/2898099196] conn(0x7f82d8071940 0x7f82d8073da0 crc :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:42.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.298+0000 7f82fc097640 1 -- 172.21.15.57:0/1164356393 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f82d806af60 msgr2=0x7f82d806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:42.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.298+0000 7f82fc097640 1 --2- 172.21.15.57:0/1164356393 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f82d806af60 0x7f82d806d420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f82f00017a0 tx=0x7f82f0001c70 comp rx=0 tx=0).stop 2023-10-19T14:22:42.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.298+0000 7f82fc097640 1 -- 172.21.15.57:0/1164356393 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f82f415e080 msgr2=0x7f82f415e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:42.302 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.298+0000 7f82fc097640 1 --2- 172.21.15.57:0/1164356393 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f82f415e080 0x7f82f415e500 secure :-1 s=READY pgs=3067 cs=0 l=1 rev1=1 crypto rx=0x7f82e40117d0 tx=0x7f82e4011c90 comp rx=0 tx=0).stop 2023-10-19T14:22:42.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.298+0000 7f82fa634640 1 -- 172.21.15.57:0/1164356393 reap_dead start 2023-10-19T14:22:42.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.298+0000 7f82fc097640 1 -- 172.21.15.57:0/1164356393 shutdown_connections 2023-10-19T14:22:42.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.298+0000 7f82fc097640 1 -- 172.21.15.57:0/1164356393 >> 172.21.15.57:0/1164356393 conn(0x7f82f40ccf20 msgr2=0x7f82f40b1e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:42.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.298+0000 7f82fc097640 1 -- 172.21.15.57:0/1164356393 shutdown_connections 2023-10-19T14:22:42.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.298+0000 7f82fc097640 1 -- 172.21.15.57:0/1164356393 wait complete. 2023-10-19T14:22:42.310 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2023-10-19T14:22:42.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.418+0000 7fa0a87d7640 1 -- 172.21.15.57:0/3700546139 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0a015e5d0 msgr2=0x7fa0a00cdfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:42.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.418+0000 7fa0a87d7640 1 --2- 172.21.15.57:0/3700546139 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0a015e5d0 0x7fa0a00cdfd0 secure :-1 s=READY pgs=3069 cs=0 l=1 rev1=1 crypto rx=0x7fa0a00897d0 tx=0x7fa09802bd70 comp rx=0 tx=0).stop 2023-10-19T14:22:42.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.418+0000 7fa0a87d7640 1 -- 172.21.15.57:0/3700546139 shutdown_connections 2023-10-19T14:22:42.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.418+0000 7fa0a87d7640 1 --2- 172.21.15.57:0/3700546139 >> v2:172.21.15.179:3300/0 conn(0x7fa0a00ce980 0x7fa0a00cf1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:42.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.418+0000 7fa0a87d7640 1 --2- 172.21.15.57:0/3700546139 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0a015e5d0 0x7fa0a00cdfd0 unknown :-1 s=CLOSED pgs=3069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:42.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.418+0000 7fa0a87d7640 1 -- 172.21.15.57:0/3700546139 >> 172.21.15.57:0/3700546139 conn(0x7fa0a009a140 msgr2=0x7fa0a00d0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:42.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.418+0000 7fa0a87d7640 1 -- 172.21.15.57:0/3700546139 shutdown_connections 2023-10-19T14:22:42.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.418+0000 7fa0a87d7640 1 -- 172.21.15.57:0/3700546139 wait complete. 2023-10-19T14:22:42.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.420+0000 7fa0a87d7640 1 Processor -- start 2023-10-19T14:22:42.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.420+0000 7fa0a87d7640 1 -- start start 2023-10-19T14:22:42.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.420+0000 7fa0a87d7640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa0a00ce980 0x7fa0a009e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:42.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.420+0000 7fa0a87d7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0a00cf710 0x7fa0a01590c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:42.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.420+0000 7fa0a87d7640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa0a015e5d0 0x7fa0a00ca130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:42.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.420+0000 7fa0a87d7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a009e780 con 0x7fa0a00cf710 2023-10-19T14:22:42.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.420+0000 7fa0a87d7640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a00b0010 con 0x7fa0a015e5d0 2023-10-19T14:22:42.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.420+0000 7fa0a87d7640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a01598f0 con 0x7fa0a00ce980 2023-10-19T14:22:42.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.420+0000 7fa0a5d72640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0a00cf710 0x7fa0a01590c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:42.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.420+0000 7fa0a5d72640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0a00cf710 0x7fa0a01590c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:60146/0 (socket says 172.21.15.57:60146) 2023-10-19T14:22:42.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.420+0000 7fa0a5d72640 1 -- 172.21.15.57:0/1217614765 learned_addr learned my addr 172.21.15.57:0/1217614765 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:22:42.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.420+0000 7fa0a6573640 1 --2- 172.21.15.57:0/1217614765 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa0a00ce980 0x7fa0a009e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:42.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.421+0000 7fa0a6d74640 1 --2- 172.21.15.57:0/1217614765 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa0a015e5d0 0x7fa0a00ca130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:42.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.421+0000 7fa0a5d72640 1 -- 172.21.15.57:0/1217614765 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa0a00ce980 msgr2=0x7fa0a009e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:42.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.421+0000 7fa0a5d72640 1 --2- 172.21.15.57:0/1217614765 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa0a00ce980 0x7fa0a009e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:42.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.421+0000 7fa0a5d72640 1 -- 172.21.15.57:0/1217614765 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa0a015e5d0 msgr2=0x7fa0a00ca130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:42.427 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.421+0000 7fa0a5d72640 1 --2- 172.21.15.57:0/1217614765 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa0a015e5d0 0x7fa0a00ca130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:42.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.421+0000 7fa0a5d72640 1 -- 172.21.15.57:0/1217614765 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa098009490 con 0x7fa0a00cf710 2023-10-19T14:22:42.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.421+0000 7fa0a5d72640 1 --2- 172.21.15.57:0/1217614765 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0a00cf710 0x7fa0a01590c0 secure :-1 s=READY pgs=3070 cs=0 l=1 rev1=1 crypto rx=0x7fa09000e9f0 tx=0x7fa09000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:42.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.421+0000 7fa096ffd640 1 -- 172.21.15.57:0/1217614765 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa090009700 con 0x7fa0a00cf710 2023-10-19T14:22:42.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.421+0000 7fa096ffd640 1 -- 172.21.15.57:0/1217614765 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fa09000ce70 con 0x7fa0a00cf710 2023-10-19T14:22:42.428 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.421+0000 7fa0a87d7640 1 -- 172.21.15.57:0/1217614765 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0a0159bd0 con 0x7fa0a00cf710 2023-10-19T14:22:42.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.421+0000 7fa096ffd640 1 -- 172.21.15.57:0/1217614765 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa090010670 con 0x7fa0a00cf710 2023-10-19T14:22:42.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.421+0000 7fa0a87d7640 1 -- 172.21.15.57:0/1217614765 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0a015a190 con 0x7fa0a00cf710 2023-10-19T14:22:42.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.422+0000 7fa0a87d7640 1 -- 172.21.15.57:0/1217614765 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0a00b01b0 con 0x7fa0a00cf710 2023-10-19T14:22:42.429 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.423+0000 7fa096ffd640 1 -- 172.21.15.57:0/1217614765 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 155) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fa0900107d0 con 0x7fa0a00cf710 2023-10-19T14:22:42.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.423+0000 7fa096ffd640 1 --2- 172.21.15.57:0/1217614765 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fa08406af60 0x7fa08406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:42.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.423+0000 7fa096ffd640 1 -- 172.21.15.57:0/1217614765 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(268..268 src has 1..268) v4 ==== 6108+0+0 (secure 0 0 0) 0x7fa090014070 con 0x7fa0a00cf710 2023-10-19T14:22:42.430 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.423+0000 7fa0a6573640 1 --2- 172.21.15.57:0/1217614765 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fa08406af60 0x7fa08406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:42.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.424+0000 7fa0a6573640 1 --2- 172.21.15.57:0/1217614765 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fa08406af60 0x7fa08406d420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa098036000 tx=0x7fa098004730 comp rx=0 tx=0).ready entity=mgr.10520 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:42.431 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.427+0000 7fa096ffd640 1 -- 172.21.15.57:0/1217614765 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fa090051ca0 con 0x7fa0a00cf710 2023-10-19T14:22:42.606 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.604+0000 7fa0a87d7640 1 -- 172.21.15.57:0/1217614765 --> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7fa0a0069100 con 0x7fa08406af60 2023-10-19T14:22:42.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.605+0000 7fa096ffd640 1 -- 172.21.15.57:0/1217614765 <== mgr.10520 v2:172.21.15.179:6832/892689897 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fa0a0069100 con 0x7fa08406af60 2023-10-19T14:22:42.607 INFO:teuthology.orchestra.run.smithi057.stderr:instructed osd(s) 6 to deep-scrub 2023-10-19T14:22:42.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.607+0000 7fa0a87d7640 1 -- 172.21.15.57:0/1217614765 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fa08406af60 msgr2=0x7fa08406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:42.609 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.607+0000 7fa0a87d7640 1 --2- 172.21.15.57:0/1217614765 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fa08406af60 0x7fa08406d420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa098036000 tx=0x7fa098004730 comp rx=0 tx=0).stop 2023-10-19T14:22:42.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.607+0000 7fa0a87d7640 1 -- 172.21.15.57:0/1217614765 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0a00cf710 msgr2=0x7fa0a01590c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:42.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.607+0000 7fa0a87d7640 1 --2- 172.21.15.57:0/1217614765 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0a00cf710 0x7fa0a01590c0 secure :-1 s=READY pgs=3070 cs=0 l=1 rev1=1 crypto rx=0x7fa09000e9f0 tx=0x7fa09000eeb0 comp rx=0 tx=0).stop 2023-10-19T14:22:42.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.607+0000 7fa0a87d7640 1 -- 172.21.15.57:0/1217614765 shutdown_connections 2023-10-19T14:22:42.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.607+0000 7fa0a87d7640 1 --2- 172.21.15.57:0/1217614765 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fa08406af60 0x7fa08406d420 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:42.610 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.607+0000 7fa0a87d7640 1 --2- 172.21.15.57:0/1217614765 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa0a015e5d0 0x7fa0a00ca130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:42.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.607+0000 7fa0a87d7640 1 --2- 172.21.15.57:0/1217614765 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0a00cf710 0x7fa0a01590c0 unknown :-1 s=CLOSED pgs=3070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:42.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.607+0000 7fa0a87d7640 1 --2- 172.21.15.57:0/1217614765 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa0a00ce980 0x7fa0a009e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:42.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.607+0000 7fa0a87d7640 1 -- 172.21.15.57:0/1217614765 >> 172.21.15.57:0/1217614765 conn(0x7fa0a009a140 msgr2=0x7fa0a015b0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:42.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.607+0000 7fa0a87d7640 1 -- 172.21.15.57:0/1217614765 shutdown_connections 2023-10-19T14:22:42.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.607+0000 7fa0a87d7640 1 -- 172.21.15.57:0/1217614765 wait complete. 2023-10-19T14:22:42.619 INFO:tasks.ceph:Scrubbing osd.7 2023-10-19T14:22:42.620 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2023-10-19T14:22:42.733 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.731+0000 7fa436c1f640 1 -- 172.21.15.57:0/1134529412 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa430069d40 msgr2=0x7fa4300b2330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:42.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.731+0000 7fa436c1f640 1 --2- 172.21.15.57:0/1134529412 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa430069d40 0x7fa4300b2330 secure :-1 s=READY pgs=3072 cs=0 l=1 rev1=1 crypto rx=0x7fa430070890 tx=0x7fa41802bdb0 comp rx=0 tx=0).stop 2023-10-19T14:22:42.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.731+0000 7fa436c1f640 1 -- 172.21.15.57:0/1134529412 shutdown_connections 2023-10-19T14:22:42.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.731+0000 7fa436c1f640 1 --2- 172.21.15.57:0/1134529412 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa430069d40 0x7fa4300b2330 unknown :-1 s=CLOSED pgs=3072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:42.734 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.731+0000 7fa436c1f640 1 --2- 172.21.15.57:0/1134529412 >> v2:172.21.15.179:3300/0 conn(0x7fa430158d60 0x7fa43015d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:42.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.731+0000 7fa436c1f640 1 -- 172.21.15.57:0/1134529412 >> 172.21.15.57:0/1134529412 conn(0x7fa4300aef90 msgr2=0x7fa4300af370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:42.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.731+0000 7fa436c1f640 1 -- 172.21.15.57:0/1134529412 shutdown_connections 2023-10-19T14:22:42.735 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.731+0000 7fa436c1f640 1 -- 172.21.15.57:0/1134529412 wait complete. 2023-10-19T14:22:42.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.732+0000 7fa436c1f640 1 Processor -- start 2023-10-19T14:22:42.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.732+0000 7fa436c1f640 1 -- start start 2023-10-19T14:22:42.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.733+0000 7fa436c1f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa430069d40 0x7fa4301475d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:42.738 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.733+0000 7fa436c1f640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa4300cb2b0 0x7fa4301e37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:42.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.733+0000 7fa436c1f640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa430158d60 0x7fa43015b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:42.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.733+0000 7fa436c1f640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4301efd10 con 0x7fa430069d40 2023-10-19T14:22:42.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.733+0000 7fa436c1f640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4301ef3e0 con 0x7fa4300cb2b0 2023-10-19T14:22:42.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.733+0000 7fa436c1f640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4301ef650 con 0x7fa430158d60 2023-10-19T14:22:42.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.733+0000 7fa4349bb640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa430069d40 0x7fa4301475d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:42.739 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.733+0000 7fa4349bb640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa430069d40 0x7fa4301475d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:60158/0 (socket says 172.21.15.57:60158) 2023-10-19T14:22:42.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.733+0000 7fa4349bb640 1 -- 172.21.15.57:0/2921371620 learned_addr learned my addr 172.21.15.57:0/2921371620 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:22:42.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.733+0000 7fa4349bb640 1 -- 172.21.15.57:0/2921371620 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa430158d60 msgr2=0x7fa43015b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:42.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.733+0000 7fa4349bb640 1 --2- 172.21.15.57:0/2921371620 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fa430158d60 0x7fa43015b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:42.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.733+0000 7fa4349bb640 1 -- 172.21.15.57:0/2921371620 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa4300cb2b0 msgr2=0x7fa4301e37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:42.740 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.733+0000 7fa4349bb640 1 --2- 172.21.15.57:0/2921371620 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa4300cb2b0 0x7fa4301e37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:42.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.733+0000 7fa4349bb640 1 -- 172.21.15.57:0/2921371620 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa418009560 con 0x7fa430069d40 2023-10-19T14:22:42.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.734+0000 7fa4349bb640 1 --2- 172.21.15.57:0/2921371620 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa430069d40 0x7fa4301475d0 secure :-1 s=READY pgs=3073 cs=0 l=1 rev1=1 crypto rx=0x7fa42800daf0 tx=0x7fa42800b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:42.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.734+0000 7fa42d7fa640 1 -- 172.21.15.57:0/2921371620 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa42800be40 con 0x7fa430069d40 2023-10-19T14:22:42.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.734+0000 7fa42d7fa640 1 -- 172.21.15.57:0/2921371620 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fa428003a60 con 0x7fa430069d40 2023-10-19T14:22:42.741 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.734+0000 7fa436c1f640 1 -- 172.21.15.57:0/2921371620 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4301ef930 con 0x7fa430069d40 2023-10-19T14:22:42.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.734+0000 7fa42d7fa640 1 -- 172.21.15.57:0/2921371620 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa428003e70 con 0x7fa430069d40 2023-10-19T14:22:42.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.734+0000 7fa436c1f640 1 -- 172.21.15.57:0/2921371620 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4301e2b00 con 0x7fa430069d40 2023-10-19T14:22:42.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.735+0000 7fa436c1f640 1 -- 172.21.15.57:0/2921371620 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa430069100 con 0x7fa430069d40 2023-10-19T14:22:42.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.735+0000 7fa42d7fa640 1 -- 172.21.15.57:0/2921371620 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 155) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fa428004020 con 0x7fa430069d40 2023-10-19T14:22:42.742 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.736+0000 7fa42d7fa640 1 --2- 172.21.15.57:0/2921371620 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fa41006af60 0x7fa41006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:42.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.736+0000 7fa42d7fa640 1 -- 172.21.15.57:0/2921371620 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(268..268 src has 1..268) v4 ==== 6108+0+0 (secure 0 0 0) 0x7fa42808a710 con 0x7fa430069d40 2023-10-19T14:22:42.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.736+0000 7fa42d7fa640 1 --2- 172.21.15.57:0/2921371620 >> [v2:172.21.15.179:6816/3758126853,v1:172.21.15.179:6817/3758126853] conn(0x7fa410071940 0x7fa410073da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:42.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.736+0000 7fa42d7fa640 1 -- 172.21.15.57:0/2921371620 --> [v2:172.21.15.179:6816/3758126853,v1:172.21.15.179:6817/3758126853] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa410074470 con 0x7fa410071940 2023-10-19T14:22:42.743 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.736+0000 7fa42d7fa640 1 -- 172.21.15.57:0/2921371620 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_get_version_reply(handle=1 version=268) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa42808ab60 con 0x7fa430069d40 2023-10-19T14:22:42.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.736+0000 7fa42ffff640 1 --2- 172.21.15.57:0/2921371620 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fa41006af60 0x7fa41006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:42.744 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.736+0000 7fa4351bc640 1 --2- 172.21.15.57:0/2921371620 >> [v2:172.21.15.179:6816/3758126853,v1:172.21.15.179:6817/3758126853] conn(0x7fa410071940 0x7fa410073da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:42.745 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.737+0000 7fa4351bc640 1 --2- 172.21.15.57:0/2921371620 >> [v2:172.21.15.179:6816/3758126853,v1:172.21.15.179:6817/3758126853] conn(0x7fa410071940 0x7fa410073da0 crc :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:42.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.737+0000 7fa42ffff640 1 --2- 172.21.15.57:0/2921371620 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fa41006af60 0x7fa41006d420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa4300ae930 tx=0x7fa4200077c0 comp rx=0 tx=0).ready entity=mgr.10520 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:42.746 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.738+0000 7fa42d7fa640 1 -- 172.21.15.57:0/2921371620 <== osd.7 v2:172.21.15.179:6816/3758126853 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7fa410074470 con 0x7fa410071940 2023-10-19T14:22:42.752 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.750+0000 7fa436c1f640 1 -- 172.21.15.57:0/2921371620 --> [v2:172.21.15.179:6816/3758126853,v1:172.21.15.179:6817/3758126853] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fa4301eff80 con 0x7fa410071940 2023-10-19T14:22:42.753 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.751+0000 7fa42d7fa640 1 -- 172.21.15.57:0/2921371620 <== osd.7 v2:172.21.15.179:6816/3758126853 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fa4301eff80 con 0x7fa410071940 2023-10-19T14:22:42.753 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2023-10-19T14:22:42.754 INFO:teuthology.orchestra.run.smithi057.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-10-19T14:22:42.754 INFO:teuthology.orchestra.run.smithi057.stdout:} 2023-10-19T14:22:42.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.751+0000 7fa436c1f640 1 -- 172.21.15.57:0/2921371620 >> [v2:172.21.15.179:6816/3758126853,v1:172.21.15.179:6817/3758126853] conn(0x7fa410071940 msgr2=0x7fa410073da0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:42.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.751+0000 7fa436c1f640 1 --2- 172.21.15.57:0/2921371620 >> [v2:172.21.15.179:6816/3758126853,v1:172.21.15.179:6817/3758126853] conn(0x7fa410071940 0x7fa410073da0 crc :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:42.755 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.752+0000 7fa436c1f640 1 -- 172.21.15.57:0/2921371620 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fa41006af60 msgr2=0x7fa41006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:42.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.752+0000 7fa436c1f640 1 --2- 172.21.15.57:0/2921371620 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fa41006af60 0x7fa41006d420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa4300ae930 tx=0x7fa4200077c0 comp rx=0 tx=0).stop 2023-10-19T14:22:42.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.752+0000 7fa436c1f640 1 -- 172.21.15.57:0/2921371620 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa430069d40 msgr2=0x7fa4301475d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:42.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.752+0000 7fa436c1f640 1 --2- 172.21.15.57:0/2921371620 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa430069d40 0x7fa4301475d0 secure :-1 s=READY pgs=3073 cs=0 l=1 rev1=1 crypto rx=0x7fa42800daf0 tx=0x7fa42800b260 comp rx=0 tx=0).stop 2023-10-19T14:22:42.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.752+0000 7fa4351bc640 1 -- 172.21.15.57:0/2921371620 reap_dead start 2023-10-19T14:22:42.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.752+0000 7fa436c1f640 1 -- 172.21.15.57:0/2921371620 shutdown_connections 2023-10-19T14:22:42.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.752+0000 7fa436c1f640 1 -- 172.21.15.57:0/2921371620 >> 172.21.15.57:0/2921371620 conn(0x7fa4300aef90 msgr2=0x7fa4300cceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:42.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.752+0000 7fa436c1f640 1 -- 172.21.15.57:0/2921371620 shutdown_connections 2023-10-19T14:22:42.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.752+0000 7fa436c1f640 1 -- 172.21.15.57:0/2921371620 wait complete. 2023-10-19T14:22:42.764 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2023-10-19T14:22:42.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.872+0000 7f706be87640 1 -- 172.21.15.57:0/3085029570 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f70640ce980 msgr2=0x7f70640cdfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:42.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.872+0000 7f706be87640 1 --2- 172.21.15.57:0/3085029570 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f70640ce980 0x7f70640cdfd0 secure :-1 s=READY pgs=3075 cs=0 l=1 rev1=1 crypto rx=0x7f7058009830 tx=0x7f705802bdf0 comp rx=0 tx=0).stop 2023-10-19T14:22:42.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.872+0000 7f706be87640 1 -- 172.21.15.57:0/3085029570 shutdown_connections 2023-10-19T14:22:42.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.872+0000 7f706be87640 1 --2- 172.21.15.57:0/3085029570 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f70640ce980 0x7f70640cdfd0 unknown :-1 s=CLOSED pgs=3075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:42.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.872+0000 7f706be87640 1 -- 172.21.15.57:0/3085029570 >> 172.21.15.57:0/3085029570 conn(0x7f706409a140 msgr2=0x7f70640d0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:42.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.872+0000 7f706be87640 1 -- 172.21.15.57:0/3085029570 shutdown_connections 2023-10-19T14:22:42.876 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.872+0000 7f706be87640 1 -- 172.21.15.57:0/3085029570 wait complete. 2023-10-19T14:22:42.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.873+0000 7f706be87640 1 Processor -- start 2023-10-19T14:22:42.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.873+0000 7f706be87640 1 -- start start 2023-10-19T14:22:42.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.874+0000 7f706be87640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f70640ce980 0x7f7064154750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:42.878 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.874+0000 7f706be87640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f70640cf8e0 0x7f70640c74a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:42.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.874+0000 7f706be87640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f706415e5d0 0x7f7064155e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:42.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.874+0000 7f706be87640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7064155a60 con 0x7f706415e5d0 2023-10-19T14:22:42.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.874+0000 7f706be87640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7064151020 con 0x7f70640ce980 2023-10-19T14:22:42.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.874+0000 7f706be87640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f706409ec20 con 0x7f70640cf8e0 2023-10-19T14:22:42.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.874+0000 7f706a424640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f706415e5d0 0x7f7064155e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:42.879 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.874+0000 7f706a424640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f706415e5d0 0x7f7064155e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:60176/0 (socket says 172.21.15.57:60176) 2023-10-19T14:22:42.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.874+0000 7f706a424640 1 -- 172.21.15.57:0/1379930693 learned_addr learned my addr 172.21.15.57:0/1379930693 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:22:42.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.875+0000 7f7069422640 1 --2- 172.21.15.57:0/1379930693 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f70640cf8e0 0x7f70640c74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:42.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.875+0000 7f7069c23640 1 --2- 172.21.15.57:0/1379930693 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f70640ce980 0x7f7064154750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:42.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.875+0000 7f706a424640 1 -- 172.21.15.57:0/1379930693 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f70640cf8e0 msgr2=0x7f70640c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:42.880 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.875+0000 7f706a424640 1 --2- 172.21.15.57:0/1379930693 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f70640cf8e0 0x7f70640c74a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:42.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.875+0000 7f706a424640 1 -- 172.21.15.57:0/1379930693 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f70640ce980 msgr2=0x7f7064154750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:42.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.875+0000 7f706a424640 1 --2- 172.21.15.57:0/1379930693 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f70640ce980 0x7f7064154750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:42.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.875+0000 7f706a424640 1 -- 172.21.15.57:0/1379930693 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7058009490 con 0x7f706415e5d0 2023-10-19T14:22:42.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.875+0000 7f706a424640 1 --2- 172.21.15.57:0/1379930693 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f706415e5d0 0x7f7064155e70 secure :-1 s=READY pgs=3076 cs=0 l=1 rev1=1 crypto rx=0x7f705400c7d0 tx=0x7f705400cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:42.881 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.875+0000 7f70627fc640 1 -- 172.21.15.57:0/1379930693 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7054003b70 con 0x7f706415e5d0 2023-10-19T14:22:42.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.875+0000 7f70627fc640 1 -- 172.21.15.57:0/1379930693 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f7054003cf0 con 0x7f706415e5d0 2023-10-19T14:22:42.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.875+0000 7f706be87640 1 -- 172.21.15.57:0/1379930693 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7064154de0 con 0x7f706415e5d0 2023-10-19T14:22:42.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.875+0000 7f70627fc640 1 -- 172.21.15.57:0/1379930693 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7054003ea0 con 0x7f706415e5d0 2023-10-19T14:22:42.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.875+0000 7f706be87640 1 -- 172.21.15.57:0/1379930693 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70641553a0 con 0x7f706415e5d0 2023-10-19T14:22:42.882 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.876+0000 7f706be87640 1 -- 172.21.15.57:0/1379930693 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7064069100 con 0x7f706415e5d0 2023-10-19T14:22:42.883 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.877+0000 7f70627fc640 1 -- 172.21.15.57:0/1379930693 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 155) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f705400aab0 con 0x7f706415e5d0 2023-10-19T14:22:42.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.877+0000 7f70627fc640 1 --2- 172.21.15.57:0/1379930693 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f705006af60 0x7f705006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:42.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.877+0000 7f70627fc640 1 -- 172.21.15.57:0/1379930693 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(268..268 src has 1..268) v4 ==== 6108+0+0 (secure 0 0 0) 0x7f7054089890 con 0x7f706415e5d0 2023-10-19T14:22:42.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.877+0000 7f7069c23640 1 --2- 172.21.15.57:0/1379930693 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f705006af60 0x7f705006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:42.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.878+0000 7f7069c23640 1 --2- 172.21.15.57:0/1379930693 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f705006af60 0x7f705006d420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f705c000f60 tx=0x7f705c009040 comp rx=0 tx=0).ready entity=mgr.10520 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:42.884 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:42.881+0000 7f70627fc640 1 -- 172.21.15.57:0/1379930693 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f7054054730 con 0x7f706415e5d0 2023-10-19T14:22:43.058 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:43.056+0000 7f706be87640 1 -- 172.21.15.57:0/1379930693 --> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7f70640c8ed0 con 0x7f705006af60 2023-10-19T14:22:43.059 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:43.057+0000 7f70627fc640 1 -- 172.21.15.57:0/1379930693 <== mgr.10520 v2:172.21.15.179:6832/892689897 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f70640c8ed0 con 0x7f705006af60 2023-10-19T14:22:43.059 INFO:teuthology.orchestra.run.smithi057.stderr:instructed osd(s) 7 to deep-scrub 2023-10-19T14:22:43.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:43.059+0000 7f706be87640 1 -- 172.21.15.57:0/1379930693 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f705006af60 msgr2=0x7f705006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:43.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:43.059+0000 7f706be87640 1 --2- 172.21.15.57:0/1379930693 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f705006af60 0x7f705006d420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f705c000f60 tx=0x7f705c009040 comp rx=0 tx=0).stop 2023-10-19T14:22:43.061 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:43.059+0000 7f706be87640 1 -- 172.21.15.57:0/1379930693 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f706415e5d0 msgr2=0x7f7064155e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:43.062 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:43.059+0000 7f706be87640 1 --2- 172.21.15.57:0/1379930693 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f706415e5d0 0x7f7064155e70 secure :-1 s=READY pgs=3076 cs=0 l=1 rev1=1 crypto rx=0x7f705400c7d0 tx=0x7f705400cc90 comp rx=0 tx=0).stop 2023-10-19T14:22:43.062 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:43.059+0000 7f706be87640 1 -- 172.21.15.57:0/1379930693 shutdown_connections 2023-10-19T14:22:43.062 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:43.059+0000 7f706be87640 1 --2- 172.21.15.57:0/1379930693 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f705006af60 0x7f705006d420 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:43.062 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:43.059+0000 7f706be87640 1 --2- 172.21.15.57:0/1379930693 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f706415e5d0 0x7f7064155e70 unknown :-1 s=CLOSED pgs=3076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:43.062 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:43.059+0000 7f706be87640 1 --2- 172.21.15.57:0/1379930693 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f70640cf8e0 0x7f70640c74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:43.062 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:43.059+0000 7f706be87640 1 --2- 172.21.15.57:0/1379930693 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f70640ce980 0x7f7064154750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:43.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:43.059+0000 7f706be87640 1 -- 172.21.15.57:0/1379930693 >> 172.21.15.57:0/1379930693 conn(0x7f706409a140 msgr2=0x7f706415cbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:43.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:43.059+0000 7f706be87640 1 -- 172.21.15.57:0/1379930693 shutdown_connections 2023-10-19T14:22:43.063 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:43.059+0000 7f706be87640 1 -- 172.21.15.57:0/1379930693 wait complete. 2023-10-19T14:22:43.071 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-19T14:22:43.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:43.179+0000 7f52c164f640 1 -- 172.21.15.57:0/1739086887 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f52a8004d00 msgr2=0x7f52a8005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:43.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:43.179+0000 7f52c164f640 1 --2- 172.21.15.57:0/1739086887 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f52a8004d00 0x7f52a8005180 secure :-1 s=READY pgs=3078 cs=0 l=1 rev1=1 crypto rx=0x7f52ac005fd0 tx=0x7f52a80057f0 comp rx=0 tx=0).stop 2023-10-19T14:22:43.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:43.179+0000 7f52bb7fe640 1 -- 172.21.15.57:0/1739086887 reap_dead start 2023-10-19T14:22:43.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:43.179+0000 7f52c164f640 1 -- 172.21.15.57:0/1739086887 shutdown_connections 2023-10-19T14:22:43.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:43.179+0000 7f52c164f640 1 -- 172.21.15.57:0/1739086887 >> 172.21.15.57:0/1739086887 conn(0x7f52bc09a150 msgr2=0x7f52bc0d0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:43.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:43.179+0000 7f52c164f640 1 -- 172.21.15.57:0/1739086887 shutdown_connections 2023-10-19T14:22:43.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:43.179+0000 7f52c164f640 1 -- 172.21.15.57:0/1739086887 wait complete. 2023-10-19T14:22:43.184 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:43.180+0000 7f52c164f640 1 Processor -- start 2023-10-19T14:22:43.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:43.180+0000 7f52c164f640 1 -- start start 2023-10-19T14:22:43.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:43.181+0000 7f52c164f640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f52bc1536c0 0x7f52bc0b3a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:43.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:43.181+0000 7f52c164f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f52bc0b3fc0 0x7f52bc0b5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:43.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:43.181+0000 7f52c164f640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f52bc0b5850 0x7f52bc0ca290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:43.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:43.181+0000 7f52c164f640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52bc0c82b0 con 0x7f52bc0b3fc0 2023-10-19T14:22:43.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:43.181+0000 7f52c164f640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52bc0b5d00 con 0x7f52bc0b5850 2023-10-19T14:22:43.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:43.181+0000 7f52c164f640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52bc0c7f60 con 0x7f52bc1536c0 2023-10-19T14:22:43.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:43.181+0000 7f52ba7fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f52bc0b3fc0 0x7f52bc0b5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:43.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:43.181+0000 7f52ba7fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f52bc0b3fc0 0x7f52bc0b5310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:60206/0 (socket says 172.21.15.57:60206) 2023-10-19T14:22:43.186 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:43.181+0000 7f52ba7fc640 1 -- 172.21.15.57:0/521221149 learned_addr learned my addr 172.21.15.57:0/521221149 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:22:43.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:43.181+0000 7f52ba7fc640 1 -- 172.21.15.57:0/521221149 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f52bc1536c0 msgr2=0x7f52bc0b3a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:43.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:43.181+0000 7f52ba7fc640 1 --2- 172.21.15.57:0/521221149 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f52bc1536c0 0x7f52bc0b3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:43.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:43.181+0000 7f52ba7fc640 1 -- 172.21.15.57:0/521221149 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f52bc0b5850 msgr2=0x7f52bc0ca290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:43.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:43.181+0000 7f52ba7fc640 1 --2- 172.21.15.57:0/521221149 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f52bc0b5850 0x7f52bc0ca290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:43.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:43.181+0000 7f52ba7fc640 1 -- 172.21.15.57:0/521221149 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52bc0b4440 con 0x7f52bc0b3fc0 2023-10-19T14:22:43.187 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:43.181+0000 7f52ba7fc640 1 --2- 172.21.15.57:0/521221149 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f52bc0b3fc0 0x7f52bc0b5310 secure :-1 s=READY pgs=3079 cs=0 l=1 rev1=1 crypto rx=0x7f52a4003ea0 tx=0x7f52a4003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:43.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:43.182+0000 7f52b37fe640 1 -- 172.21.15.57:0/521221149 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52a4006110 con 0x7f52bc0b3fc0 2023-10-19T14:22:43.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:43.182+0000 7f52b37fe640 1 -- 172.21.15.57:0/521221149 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f52a4001d70 con 0x7f52bc0b3fc0 2023-10-19T14:22:43.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:43.182+0000 7f52b37fe640 1 -- 172.21.15.57:0/521221149 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52a4011ac0 con 0x7f52bc0b3fc0 2023-10-19T14:22:43.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:43.182+0000 7f52c164f640 1 -- 172.21.15.57:0/521221149 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52bc0b4720 con 0x7f52bc0b3fc0 2023-10-19T14:22:43.188 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:43.182+0000 7f52c164f640 1 -- 172.21.15.57:0/521221149 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52bc156fa0 con 0x7f52bc0b3fc0 2023-10-19T14:22:43.189 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:43.183+0000 7f52c164f640 1 -- 172.21.15.57:0/521221149 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52bc067610 con 0x7f52bc0b3fc0 2023-10-19T14:22:43.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:43.183+0000 7f52b37fe640 1 -- 172.21.15.57:0/521221149 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 155) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f52a4011c20 con 0x7f52bc0b3fc0 2023-10-19T14:22:43.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:43.184+0000 7f52b37fe640 1 --2- 172.21.15.57:0/521221149 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f529c06af60 0x7f529c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:22:43.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:43.184+0000 7f52b37fe640 1 -- 172.21.15.57:0/521221149 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(268..268 src has 1..268) v4 ==== 6108+0+0 (secure 0 0 0) 0x7f52a408a920 con 0x7f52bc0b3fc0 2023-10-19T14:22:43.190 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:43.184+0000 7f52baffd640 1 --2- 172.21.15.57:0/521221149 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f529c06af60 0x7f529c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:22:43.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:43.184+0000 7f52baffd640 1 --2- 172.21.15.57:0/521221149 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f529c06af60 0x7f529c06d420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f52b4001270 tx=0x7f52b4001d20 comp rx=0 tx=0).ready entity=mgr.10520 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:22:43.191 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:43.188+0000 7f52b37fe640 1 -- 172.21.15.57:0/521221149 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f52a40556b0 con 0x7f52bc0b3fc0 2023-10-19T14:22:43.360 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:43.358+0000 7f52c164f640 1 -- 172.21.15.57:0/521221149 --> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f52bc0ce290 con 0x7f529c06af60 2023-10-19T14:22:43.363 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:43.361+0000 7f52b37fe640 1 -- 172.21.15.57:0/521221149 <== mgr.10520 v2:172.21.15.179:6832/892689897 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+50732 (secure 0 0 0) 0x7f52b40036d0 con 0x7f529c06af60 2023-10-19T14:22:43.363 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:22:43.364 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2023-10-19T14:22:43.365 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:43.363+0000 7f52c164f640 1 -- 172.21.15.57:0/521221149 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f529c06af60 msgr2=0x7f529c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:43.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:43.363+0000 7f52c164f640 1 --2- 172.21.15.57:0/521221149 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f529c06af60 0x7f529c06d420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f52b4001270 tx=0x7f52b4001d20 comp rx=0 tx=0).stop 2023-10-19T14:22:43.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:43.363+0000 7f52c164f640 1 -- 172.21.15.57:0/521221149 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f52bc0b3fc0 msgr2=0x7f52bc0b5310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:22:43.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:43.363+0000 7f52c164f640 1 --2- 172.21.15.57:0/521221149 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f52bc0b3fc0 0x7f52bc0b5310 secure :-1 s=READY pgs=3079 cs=0 l=1 rev1=1 crypto rx=0x7f52a4003ea0 tx=0x7f52a4003f90 comp rx=0 tx=0).stop 2023-10-19T14:22:43.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:43.363+0000 7f52c164f640 1 -- 172.21.15.57:0/521221149 shutdown_connections 2023-10-19T14:22:43.366 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:43.363+0000 7f52c164f640 1 --2- 172.21.15.57:0/521221149 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f529c06af60 0x7f529c06d420 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:43.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:43.363+0000 7f52c164f640 1 --2- 172.21.15.57:0/521221149 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f52bc0b5850 0x7f52bc0ca290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:43.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:43.363+0000 7f52c164f640 1 --2- 172.21.15.57:0/521221149 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f52bc0b3fc0 0x7f52bc0b5310 unknown :-1 s=CLOSED pgs=3079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:43.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:43.363+0000 7f52c164f640 1 --2- 172.21.15.57:0/521221149 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f52bc1536c0 0x7f52bc0b3a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:22:43.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:43.363+0000 7f52c164f640 1 -- 172.21.15.57:0/521221149 >> 172.21.15.57:0/521221149 conn(0x7f52bc09a150 msgr2=0x7f52bc0b31f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:22:43.367 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:43.363+0000 7f52c164f640 1 -- 172.21.15.57:0/521221149 shutdown_connections 2023-10-19T14:22:43.368 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:22:43.363+0000 7f52c164f640 1 -- 172.21.15.57:0/521221149 wait complete. 2023-10-19T14:22:43.375 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2023-10-19T14:22:41.811893+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2731,"num_read_kb":5751,"num_write":697,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3246,"ondisk_log_size":3246,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":457624,"kb_used_data":27684,"kb_used_omap":12,"kb_used_meta":429811,"kb_avail":754517096,"statfs":{"total":773094113280,"available":772625506304,"internally_reserved":0,"allocated":28348416,"data_stored":15209605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":440127033},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-667824,"num_objects":-25,"num_object_clones":0,"num_object_copies":-50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-25,"num_whiteouts":0,"num_read":-221,"num_read_kb":-75,"num_write":-264,"num_write_kb":-2056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001489"},"pg_stats":[{"pgid":"1.0","version":"265'3246","reported_seq":4338,"reported_epoch":268,"state":"active+clean","last_fresh":"2023-10-19T14:22:36.234508+0000","last_change":"2023-10-19T12:44:08.636613+0000","last_active":"2023-10-19T14:22:36.234508+0000","last_peered":"2023-10-19T14:22:36.234508+0000","last_clean":"2023-10-19T14:22:36.234508+0000","last_became_active":"2023-10-19T12:44:08.635697+0000","last_became_peered":"2023-10-19T12:44:08.635697+0000","last_unstale":"2023-10-19T14:22:36.234508+0000","last_undegraded":"2023-10-19T14:22:36.234508+0000","last_fullsized":"2023-10-19T14:22:36.234508+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-19T12:44:07.399738+0000","last_clean_scrub_stamp":"2023-10-19T12:44:07.399738+0000","objects_scrubbed":0,"log_size":3246,"log_dups_size":0,"ondisk_log_size":3246,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T21:15:23.685485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2731,"num_read_kb":5751,"num_write":697,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2731,"num_read_kb":5751,"num_write":697,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3246,"ondisk_log_size":3246,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576027,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":67016,"kb_used_data":3896,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94304824,"statfs":{"total":96636764160,"available":96568139776,"internally_reserved":0,"allocated":3989504,"data_stored":2344054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":64616907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:22:11 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.53000000000000003,"15min":0.57299999999999995},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.86199999999999999},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.55600000000000005,"15min":0.59399999999999997},"min":{"1min":0.38300000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.62}]},{"osd":1,"last update":"Thu Oct 19 14:22:11 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.57799999999999996,"15min":0.59399999999999997},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.56200000000000006,"15min":0.58899999999999997},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.754,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.67900000000000005}]},{"osd":2,"last update":"Thu Oct 19 14:22:11 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.61399999999999999,"15min":0.63800000000000001},"min":{"1min":0.42599999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.77300000000000002,"5min":0.84799999999999998,"15min":0.85499999999999998},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.623,"15min":0.65400000000000003},"min":{"1min":0.41599999999999998,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.024,"5min":1.024,"15min":1.024},"last":0.66400000000000003}]},{"osd":3,"last update":"Thu Oct 19 14:22:11 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.59499999999999997,"15min":0.628},"min":{"1min":0.36399999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.66400000000000003,"15min":0.68300000000000005},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.86799999999999999,"5min":0.873,"15min":0.89600000000000002},"last":0.64900000000000002}]},{"osd":4,"last update":"Thu Oct 19 14:22:11 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.51600000000000001,"15min":0.498},"min":{"1min":0.34100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.89600000000000002,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.54400000000000004,"15min":0.51600000000000001},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":1.032,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.42499999999999999}]},{"osd":5,"last update":"Thu Oct 19 14:22:11 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.54000000000000004,"15min":0.51400000000000001},"min":{"1min":0.35299999999999998,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.58599999999999997,"15min":0.55600000000000005},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":1.069,"5min":1.069,"15min":1.069},"last":0.46700000000000003}]},{"osd":6,"last update":"Thu Oct 19 14:22:11 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.59899999999999998,"15min":0.57799999999999996},"min":{"1min":0.45600000000000002,"5min":0.38500000000000001,"15min":0.35999999999999999},"max":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.60899999999999999,"15min":0.57799999999999996},"min":{"1min":0.375,"5min":0.35599999999999998,"15min":0.33500000000000002},"max":{"1min":1.1100000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.51500000000000001}]}]},{"osd":6,"up_from":13,"seq":55834576027,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":60868,"kb_used_data":3316,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310972,"statfs":{"total":96636764160,"available":96574435328,"internally_reserved":0,"allocated":3395584,"data_stored":1753686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58915271},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:22:04 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.57399999999999995,"15min":0.60299999999999998},"min":{"1min":0.32500000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.70799999999999996,"5min":0.81200000000000006,"15min":0.85799999999999998},"last":0.62},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.55700000000000005,"15min":0.60299999999999998},"min":{"1min":0.371,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.80600000000000005,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.53900000000000003}]},{"osd":1,"last update":"Thu Oct 19 14:22:04 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.59099999999999997,"15min":0.61499999999999999},"min":{"1min":0.41899999999999998,"5min":0.373,"15min":0.35799999999999998},"max":{"1min":0.81100000000000005,"5min":0.82799999999999996,"15min":1.165},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.59499999999999997,"15min":0.622},"min":{"1min":0.435,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.83099999999999996,"5min":0.83699999999999997,"15min":0.91300000000000003},"last":0.49299999999999999}]},{"osd":2,"last update":"Thu Oct 19 14:22:04 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.55900000000000005,"15min":0.59699999999999998},"min":{"1min":0.38300000000000001,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.72399999999999998,"5min":0.80100000000000005,"15min":0.873},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59699999999999998,"15min":0.621},"min":{"1min":0.39400000000000002,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.93100000000000005},"last":0.72299999999999998}]},{"osd":3,"last update":"Thu Oct 19 14:22:04 2023","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.622,"15min":0.64800000000000002},"min":{"1min":0.377,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.83799999999999997,"5min":0.875,"15min":0.875},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.63,"15min":0.65300000000000002},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.78100000000000003,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.73399999999999999}]},{"osd":4,"last update":"Thu Oct 19 14:22:04 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.503,"15min":0.48799999999999999},"min":{"1min":0.31,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.89300000000000002,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.753},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51600000000000001,"15min":0.50900000000000001},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.78700000000000003,"5min":0.84899999999999998,"15min":0.85699999999999998},"last":0.64900000000000002}]},{"osd":5,"last update":"Thu Oct 19 14:22:04 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.52000000000000002,"15min":0.505},"min":{"1min":0.33200000000000002,"5min":0.32700000000000001,"15min":0.32500000000000001},"max":{"1min":0.86799999999999999,"5min":1.115,"15min":1.115},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.57799999999999996,"15min":0.54800000000000004},"min":{"1min":0.33900000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.78000000000000003}]},{"osd":7,"last update":"Thu Oct 19 14:22:04 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.57299999999999995,"15min":0.55100000000000005},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.82599999999999996,"5min":0.89300000000000002,"15min":0.90100000000000002},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.58199999999999996,"15min":0.55700000000000005},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.82899999999999996,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.76800000000000002}]}]},{"osd":5,"up_from":13,"seq":55834576027,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":66432,"kb_used_data":3316,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305408,"statfs":{"total":96636764160,"available":96568737792,"internally_reserved":0,"allocated":3395584,"data_stored":1753686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:21:50 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.57399999999999995,"15min":0.60699999999999998},"min":{"1min":0.38,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.69399999999999995,"5min":0.79700000000000004,"15min":0.86399999999999999},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.59599999999999997,"15min":0.63300000000000001},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.70499999999999996,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.75700000000000001}]},{"osd":1,"last update":"Thu Oct 19 14:21:50 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.59099999999999997,"15min":0.622},"min":{"1min":0.34599999999999997,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.877},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.61199999999999999,"15min":0.63700000000000001},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.877,"5min":0.877,"15min":0.88500000000000001},"last":0.48499999999999999}]},{"osd":2,"last update":"Thu Oct 19 14:21:50 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.58199999999999996,"15min":0.61399999999999999},"min":{"1min":0.378,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.623},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.625,"15min":0.65100000000000002},"min":{"1min":0.441,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.78900000000000003,"5min":0.83299999999999996,"15min":0.90800000000000003},"last":0.56499999999999995}]},{"osd":3,"last update":"Thu Oct 19 14:21:50 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.60499999999999998,"15min":0.63600000000000001},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.81100000000000005,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.626,"15min":0.65900000000000003},"min":{"1min":0.40000000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.90900000000000003},"last":0.53800000000000003}]},{"osd":4,"last update":"Thu Oct 19 14:21:50 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.48799999999999999,"15min":0.48699999999999999},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.51800000000000002,"15min":0.505},"min":{"1min":0.309,"5min":0.308,"15min":0.28599999999999998},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.82099999999999995},"last":0.73399999999999999}]},{"osd":6,"last update":"Thu Oct 19 14:21:50 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.51700000000000002,"15min":0.502},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.88800000000000001},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.54100000000000004,"15min":0.53900000000000003},"min":{"1min":0.36899999999999999,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.69399999999999995}]},{"osd":7,"last update":"Thu Oct 19 14:21:50 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.52200000000000002,"15min":0.51800000000000002},"min":{"1min":0.32700000000000001,"5min":0.312,"15min":0.312},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.57899999999999996,"15min":0.57199999999999995},"min":{"1min":0.38,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.77200000000000002}]}]},{"osd":4,"up_from":13,"seq":55834576027,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48388,"kb_used_data":3316,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323452,"statfs":{"total":96636764160,"available":96587214848,"internally_reserved":0,"allocated":3395584,"data_stored":1753686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:22:16 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.53900000000000003,"15min":0.56499999999999995},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.28599999999999998},"max":{"1min":0.69799999999999995,"5min":0.83799999999999997,"15min":0.879},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.505,"5min":0.55700000000000005,"15min":0.58799999999999997},"min":{"1min":0.33400000000000002,"5min":0.31,"15min":0.31},"max":{"1min":0.71699999999999997,"5min":0.82599999999999996,"15min":0.873},"last":0.58799999999999997}]},{"osd":1,"last update":"Thu Oct 19 14:22:16 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.58499999999999996,"15min":0.626},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.32600000000000001},"max":{"1min":0.70299999999999996,"5min":0.80400000000000005,"15min":0.92700000000000005},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.59299999999999997,"15min":0.63200000000000001},"min":{"1min":0.374,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.68500000000000005,"5min":0.90000000000000002,"15min":1.0940000000000001},"last":0.57299999999999995}]},{"osd":2,"last update":"Thu Oct 19 14:22:16 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.61099999999999999,"15min":0.63800000000000001},"min":{"1min":0.39000000000000001,"5min":0.372,"15min":0.32600000000000001},"max":{"1min":0.73699999999999999,"5min":1.008,"15min":1.008},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.625,"15min":0.65900000000000003},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.749,"5min":0.98799999999999999,"15min":1.008},"last":0.622}]},{"osd":3,"last update":"Thu Oct 19 14:22:16 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.57999999999999996,"15min":0.621},"min":{"1min":0.39000000000000001,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.755,"5min":0.83399999999999996,"15min":0.91900000000000004},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.64400000000000002,"15min":0.65900000000000003},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.76900000000000002,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.64200000000000002}]},{"osd":5,"last update":"Thu Oct 19 14:22:16 2023","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.48999999999999999,"15min":0.47899999999999998},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.755,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.51200000000000001,"15min":0.501},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29199999999999998},"max":{"1min":0.74399999999999999,"5min":0.873,"15min":0.873},"last":0.38300000000000001}]},{"osd":6,"last update":"Thu Oct 19 14:22:16 2023","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.50600000000000001,"15min":0.49399999999999999},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.28199999999999997},"max":{"1min":0.74099999999999999,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54700000000000004,"15min":0.54400000000000004},"min":{"1min":0.371,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.79700000000000004,"5min":0.88,"15min":1.2969999999999999},"last":0.44900000000000001}]},{"osd":7,"last update":"Thu Oct 19 14:22:16 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.54000000000000004,"15min":0.54400000000000004},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.72699999999999998,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.436},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60699999999999998,"15min":0.59999999999999998},"min":{"1min":0.42499999999999999,"5min":0.32100000000000001,"15min":0.318},"max":{"1min":0.77700000000000002,"5min":0.99399999999999999,"15min":1.032},"last":0.47499999999999998}]}]},{"osd":3,"up_from":13,"seq":55834576026,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":56704,"kb_used_data":3316,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315136,"statfs":{"total":96636764160,"available":96578699264,"internally_reserved":0,"allocated":3395584,"data_stored":1753686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54655432},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:21:53 2023","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.39900000000000002,"15min":0.41699999999999998},"min":{"1min":0.23899999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.54500000000000004,"5min":0.60599999999999998,"15min":0.70699999999999996},"last":0.245},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.39400000000000002,"15min":0.42199999999999999},"min":{"1min":0.23899999999999999,"5min":0.217,"15min":0.217},"max":{"1min":0.51900000000000002,"5min":0.61899999999999999,"15min":0.752},"last":0.32800000000000001}]},{"osd":1,"last update":"Thu Oct 19 14:21:53 2023","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.40699999999999997,"15min":0.42799999999999999},"min":{"1min":0.28100000000000003,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.71599999999999997,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.45500000000000002,"15min":0.47199999999999998},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26400000000000001},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.36599999999999999}]},{"osd":2,"last update":"Thu Oct 19 14:21:53 2023","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.46200000000000002,"15min":0.48099999999999998},"min":{"1min":0.34300000000000003,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.73099999999999998},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48499999999999999,"15min":0.504},"min":{"1min":0.30599999999999999,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":1.212,"5min":1.212,"15min":1.212},"last":0.29599999999999999}]},{"osd":4,"last update":"Thu Oct 19 14:21:53 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.67400000000000004,"15min":0.70099999999999996},"min":{"1min":0.44600000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.94199999999999995,"5min":1.018,"15min":1.157},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.629,"5min":0.68700000000000006,"15min":0.72299999999999998},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.89500000000000002,"5min":0.996,"15min":1.173},"last":0.55300000000000005}]},{"osd":5,"last update":"Thu Oct 19 14:21:53 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.67500000000000004,"15min":0.68600000000000005},"min":{"1min":0.50900000000000001,"5min":0.46600000000000003,"15min":0.41999999999999998},"max":{"1min":0.96799999999999997,"5min":0.97299999999999998,"15min":1.0149999999999999},"last":0.496},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.66800000000000004,"15min":0.68600000000000005},"min":{"1min":0.48899999999999999,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.97699999999999998,"5min":0.99399999999999999,"15min":1.105},"last":0.59999999999999998}]},{"osd":6,"last update":"Thu Oct 19 14:21:53 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.65000000000000002,"15min":0.68600000000000005},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.93000000000000005,"5min":1.0009999999999999,"15min":1.0669999999999999},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.68899999999999995,"15min":0.70999999999999996},"min":{"1min":0.505,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.079},"last":0.53600000000000003}]},{"osd":7,"last update":"Thu Oct 19 14:21:53 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67500000000000004,"15min":0.69799999999999995},"min":{"1min":0.46600000000000003,"5min":0.434,"15min":0.434},"max":{"1min":0.94999999999999996,"5min":0.96899999999999997,"15min":1.085},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.69799999999999995,"15min":0.72799999999999998},"min":{"1min":0.48799999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.95899999999999996,"5min":0.97199999999999998,"15min":1.1739999999999999},"last":0.58799999999999997}]}]},{"osd":2,"up_from":13,"seq":55834576028,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48388,"kb_used_data":3316,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323452,"statfs":{"total":96636764160,"available":96587214848,"internally_reserved":0,"allocated":3395584,"data_stored":1753686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:22:40 2023","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.36199999999999999,"15min":0.39100000000000001},"min":{"1min":0.26100000000000001,"5min":0.218,"15min":0.218},"max":{"1min":0.41599999999999998,"5min":0.66600000000000004,"15min":0.69699999999999995},"last":0.375},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.39400000000000002,"15min":0.42799999999999999},"min":{"1min":0.25800000000000001,"5min":0.248,"15min":0.223},"max":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.71399999999999997},"last":0.26800000000000002}]},{"osd":1,"last update":"Thu Oct 19 14:22:40 2023","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.37,"15min":0.40200000000000002},"min":{"1min":0.26300000000000001,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.501,"5min":0.55600000000000005,"15min":0.65400000000000003},"last":0.316},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.43099999999999999,"15min":0.45700000000000002},"min":{"1min":0.28399999999999997,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.59699999999999998,"5min":1.742,"15min":1.742},"last":0.29899999999999999}]},{"osd":3,"last update":"Thu Oct 19 14:22:40 2023","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.433,"15min":0.47399999999999998},"min":{"1min":0.315,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.56299999999999994,"5min":0.65900000000000003,"15min":0.74399999999999999},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.438,"5min":0.45600000000000002,"15min":0.496},"min":{"1min":0.34399999999999997,"5min":0.32000000000000001,"15min":0.315},"max":{"1min":0.66400000000000003,"5min":0.67700000000000005,"15min":0.83599999999999997},"last":0.434}]},{"osd":4,"last update":"Thu Oct 19 14:22:40 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.63200000000000001,"15min":0.65900000000000003},"min":{"1min":0.41399999999999998,"5min":0.39600000000000002,"15min":0.39500000000000002},"max":{"1min":0.84099999999999997,"5min":0.89200000000000002,"15min":0.98499999999999999},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.64300000000000002,"15min":0.67600000000000005},"min":{"1min":0.46300000000000002,"5min":0.38900000000000001,"15min":0.35199999999999998},"max":{"1min":0.90300000000000002,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.70799999999999996}]},{"osd":5,"last update":"Thu Oct 19 14:22:40 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.66800000000000004,"15min":0.69199999999999995},"min":{"1min":0.45800000000000002,"5min":0.443,"15min":0.40600000000000003},"max":{"1min":0.872,"5min":1.4330000000000001,"15min":1.4330000000000001},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.65900000000000003,"15min":0.67900000000000005},"min":{"1min":0.47499999999999998,"5min":0.45100000000000001,"15min":0.39300000000000002},"max":{"1min":0.83499999999999996,"5min":1.595,"15min":1.595},"last":0.65800000000000003}]},{"osd":6,"last update":"Thu Oct 19 14:22:40 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.66000000000000003,"15min":0.69799999999999995},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.879,"5min":0.996,"15min":1.0189999999999999},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.68300000000000005,"15min":0.71299999999999997},"min":{"1min":0.503,"5min":0.433,"15min":0.40600000000000003},"max":{"1min":0.83899999999999997,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":0.57799999999999996}]},{"osd":7,"last update":"Thu Oct 19 14:22:40 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.67000000000000004,"15min":0.71199999999999997},"min":{"1min":0.47099999999999997,"5min":0.44800000000000001,"15min":0.41799999999999998},"max":{"1min":0.84399999999999997,"5min":0.97999999999999998,"15min":1.0289999999999999},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.70299999999999996,"15min":0.72399999999999998},"min":{"1min":0.44,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.89100000000000001,"5min":1.53,"15min":1.53},"last":0.75}]}]},{"osd":1,"up_from":13,"seq":55834576027,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48384,"kb_used_data":3316,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323456,"statfs":{"total":96636764160,"available":96587218944,"internally_reserved":0,"allocated":3395584,"data_stored":1753686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.38400000000000001,"15min":0.40300000000000002},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.46899999999999997,"5min":0.61599999999999999,"15min":0.67400000000000004},"last":0.316},{"interface":"front","average":{"1min":0.378,"5min":0.41999999999999998,"15min":0.45000000000000001},"min":{"1min":0.27700000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.51500000000000001,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.36299999999999999}]},{"osd":2,"last update":"Thu Oct 19 14:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.42399999999999999,"15min":0.438},"min":{"1min":0.26100000000000001,"5min":0.251,"15min":0.251},"max":{"1min":0.505,"5min":0.63100000000000001,"15min":0.77500000000000002},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.434,"5min":0.47699999999999998,"15min":0.48799999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.56699999999999995,"5min":0.68700000000000006,"15min":0.82799999999999996},"last":0.39500000000000002}]},{"osd":3,"last update":"Thu Oct 19 14:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.495,"15min":0.501},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.29499999999999998},"max":{"1min":0.59399999999999997,"5min":0.76000000000000001,"15min":0.78900000000000003},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.42899999999999999,"15min":0.44600000000000001},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.26100000000000001},"max":{"1min":0.59899999999999998,"5min":0.65200000000000002,"15min":0.69499999999999995},"last":0.38100000000000001}]},{"osd":4,"last update":"Thu Oct 19 14:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.64200000000000002,"15min":0.66700000000000004},"min":{"1min":0.42299999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.80400000000000005,"5min":0.95999999999999996,"15min":1.042},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.65400000000000003,"15min":0.66600000000000004},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.32300000000000001},"max":{"1min":0.80500000000000005,"5min":0.90800000000000003,"15min":1.1659999999999999},"last":0.52200000000000002}]},{"osd":5,"last update":"Thu Oct 19 14:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.66700000000000004,"15min":0.68200000000000005},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.81200000000000006,"5min":0.96199999999999997,"15min":1.3500000000000001},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.62,"5min":0.68899999999999995,"15min":0.70299999999999996},"min":{"1min":0.442,"5min":0.44,"15min":0.40500000000000003},"max":{"1min":0.82899999999999996,"5min":1.0009999999999999,"15min":1.121},"last":0.59199999999999997}]},{"osd":6,"last update":"Thu Oct 19 14:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.65400000000000003,"15min":0.67600000000000005},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.84999999999999998,"5min":1.0129999999999999,"15min":1.2310000000000001},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.69699999999999995,"15min":0.71499999999999997},"min":{"1min":0.46700000000000003,"5min":0.434,"15min":0.434},"max":{"1min":0.85799999999999998,"5min":1.0009999999999999,"15min":1.274},"last":0.60399999999999998}]},{"osd":7,"last update":"Thu Oct 19 14:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.70499999999999996,"15min":0.72199999999999998},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.86899999999999999,"5min":0.99099999999999999,"15min":1.075},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.68200000000000005,"15min":0.69599999999999995},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.84599999999999997,"5min":1.0169999999999999,"15min":1.117},"last":0.64700000000000002}]}]},{"osd":0,"up_from":13,"seq":55834576028,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61444,"kb_used_data":3892,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310396,"statfs":{"total":96636764160,"available":96573845504,"internally_reserved":0,"allocated":3985408,"data_stored":2343435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Oct 19 14:22:13 2023","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.39000000000000001,"15min":0.40600000000000003},"min":{"1min":0.254,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.47699999999999998,"5min":0.59999999999999998,"15min":0.624},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.40699999999999997,"15min":0.44900000000000001},"min":{"1min":0.252,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.81100000000000005},"last":0.376}]},{"osd":2,"last update":"Thu Oct 19 14:22:13 2023","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.45700000000000002,"15min":0.47699999999999998},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.58699999999999997,"5min":0.73099999999999998,"15min":0.754},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.41399999999999998,"15min":0.439},"min":{"1min":0.316,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.54000000000000004,"5min":0.60199999999999998,"15min":0.69099999999999995},"last":0.36599999999999999}]},{"osd":3,"last update":"Thu Oct 19 14:22:13 2023","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.40300000000000002,"15min":0.433},"min":{"1min":0.28399999999999997,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.56899999999999995,"5min":0.58799999999999997,"15min":0.66800000000000004},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.46800000000000003,"15min":0.503},"min":{"1min":0.314,"5min":0.254,"15min":0.254},"max":{"1min":0.55600000000000005,"5min":0.78100000000000003,"15min":0.84599999999999997},"last":0.33900000000000002}]},{"osd":4,"last update":"Thu Oct 19 14:22:13 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.627,"15min":0.66600000000000004},"min":{"1min":0.437,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":1.1890000000000001},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.626,"15min":0.67000000000000004},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.42599999999999999},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.996},"last":0.51100000000000001}]},{"osd":5,"last update":"Thu Oct 19 14:22:13 2023","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.66100000000000003,"15min":0.70999999999999996},"min":{"1min":0.46500000000000002,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.86599999999999999,"5min":0.98499999999999999,"15min":1.1579999999999999},"last":0.434},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.67400000000000004,"15min":0.71099999999999997},"min":{"1min":0.52700000000000002,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.91700000000000004,"5min":0.98299999999999998,"15min":1.2310000000000001},"last":0.56799999999999995}]},{"osd":6,"last update":"Thu Oct 19 14:22:13 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.65400000000000003,"15min":0.69599999999999995},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.95499999999999996,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.442},{"interface":"front","average":{"1min":0.628,"5min":0.64100000000000001,"15min":0.68999999999999995},"min":{"1min":0.44600000000000001,"5min":0.441,"15min":0.441},"max":{"1min":0.86399999999999999,"5min":0.89900000000000002,"15min":1.01},"last":0.59799999999999998}]},{"osd":7,"last update":"Thu Oct 19 14:22:13 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.65500000000000003,"15min":0.70499999999999996},"min":{"1min":0.52300000000000002,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.89500000000000002,"5min":0.93400000000000005,"15min":1.014},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.65800000000000003,"15min":0.69299999999999995},"min":{"1min":0.498,"5min":0.434,"15min":0.434},"max":{"1min":0.88400000000000001,"5min":0.93799999999999994,"15min":1.0349999999999999},"last":0.55300000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-19T14:22:43.379 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-10-19T12:44:07.399738+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=19, tm_hour=12, tm_min=44, tm_sec=7, tm_wday=3, tm_yday=292, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=19, tm_hour=14, tm_min=22, tm_sec=38, tm_wday=3, tm_yday=292, tm_isdst=0) 2023-10-19T14:22:43.379 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-19T14:23:03.381 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-19T14:23:03.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.502+0000 7fab64db0640 1 -- 172.21.15.57:0/1827267979 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab600b0d20 msgr2=0x7fab60071110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:23:03.505 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.502+0000 7fab64db0640 1 --2- 172.21.15.57:0/1827267979 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab600b0d20 0x7fab60071110 secure :-1 s=READY pgs=3081 cs=0 l=1 rev1=1 crypto rx=0x7fab58005fd0 tx=0x7fab5802c0e0 comp rx=0 tx=0).stop 2023-10-19T14:23:03.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.502+0000 7fab64db0640 1 -- 172.21.15.57:0/1827267979 shutdown_connections 2023-10-19T14:23:03.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.502+0000 7fab64db0640 1 --2- 172.21.15.57:0/1827267979 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab600b0d20 0x7fab60071110 unknown :-1 s=CLOSED pgs=3081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:23:03.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.502+0000 7fab64db0640 1 -- 172.21.15.57:0/1827267979 >> 172.21.15.57:0/1827267979 conn(0x7fab60069810 msgr2=0x7fab60069c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:23:03.506 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.502+0000 7fab64db0640 1 -- 172.21.15.57:0/1827267979 shutdown_connections 2023-10-19T14:23:03.507 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.502+0000 7fab64db0640 1 -- 172.21.15.57:0/1827267979 wait complete. 2023-10-19T14:23:03.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.503+0000 7fab64db0640 1 Processor -- start 2023-10-19T14:23:03.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.503+0000 7fab64db0640 1 -- start start 2023-10-19T14:23:03.509 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.504+0000 7fab64db0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab600b0d20 0x7fab60155fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:23:03.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.504+0000 7fab64db0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fab600b1940 0x7fab600c1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:23:03.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.504+0000 7fab64db0640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fab6013d5f0 0x7fab600c5e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:23:03.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.504+0000 7fab64db0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab600c4de0 con 0x7fab600b0d20 2023-10-19T14:23:03.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.504+0000 7fab64db0640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab600c50c0 con 0x7fab600b1940 2023-10-19T14:23:03.510 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.504+0000 7fab64db0640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab600c53a0 con 0x7fab6013d5f0 2023-10-19T14:23:03.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.504+0000 7fab5e59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab600b0d20 0x7fab60155fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:23:03.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.504+0000 7fab5e59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab600b0d20 0x7fab60155fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46082/0 (socket says 172.21.15.57:46082) 2023-10-19T14:23:03.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.504+0000 7fab5e59c640 1 -- 172.21.15.57:0/3596782524 learned_addr learned my addr 172.21.15.57:0/3596782524 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:23:03.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.504+0000 7fab57fff640 1 --2- 172.21.15.57:0/3596782524 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fab600b1940 0x7fab600c1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:23:03.511 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.504+0000 7fab5ed9d640 1 --2- 172.21.15.57:0/3596782524 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fab6013d5f0 0x7fab600c5e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:23:03.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.505+0000 7fab5e59c640 1 -- 172.21.15.57:0/3596782524 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fab6013d5f0 msgr2=0x7fab600c5e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:23:03.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.505+0000 7fab5e59c640 1 --2- 172.21.15.57:0/3596782524 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fab6013d5f0 0x7fab600c5e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:23:03.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.505+0000 7fab5e59c640 1 -- 172.21.15.57:0/3596782524 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fab600b1940 msgr2=0x7fab600c1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:23:03.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.505+0000 7fab5e59c640 1 --2- 172.21.15.57:0/3596782524 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fab600b1940 0x7fab600c1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:23:03.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.505+0000 7fab5e59c640 1 -- 172.21.15.57:0/3596782524 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab58009840 con 0x7fab600b0d20 2023-10-19T14:23:03.512 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.505+0000 7fab5ed9d640 1 --2- 172.21.15.57:0/3596782524 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fab6013d5f0 0x7fab600c5e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-19T14:23:03.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.505+0000 7fab5e59c640 1 --2- 172.21.15.57:0/3596782524 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab600b0d20 0x7fab60155fa0 secure :-1 s=READY pgs=3082 cs=0 l=1 rev1=1 crypto rx=0x7fab5000c3c0 tx=0x7fab5000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:23:03.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.505+0000 7fab56ffd640 1 -- 172.21.15.57:0/3596782524 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab50011790 con 0x7fab600b0d20 2023-10-19T14:23:03.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.505+0000 7fab56ffd640 1 -- 172.21.15.57:0/3596782524 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7fab500119e0 con 0x7fab600b0d20 2023-10-19T14:23:03.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.505+0000 7fab56ffd640 1 -- 172.21.15.57:0/3596782524 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab50011e10 con 0x7fab600b0d20 2023-10-19T14:23:03.513 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.505+0000 7fab64db0640 1 -- 172.21.15.57:0/3596782524 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab600c5680 con 0x7fab600b0d20 2023-10-19T14:23:03.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.505+0000 7fab64db0640 1 -- 172.21.15.57:0/3596782524 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab601564e0 con 0x7fab600b0d20 2023-10-19T14:23:03.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.506+0000 7fab64db0640 1 -- 172.21.15.57:0/3596782524 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab600c5ba0 con 0x7fab600b0d20 2023-10-19T14:23:03.514 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.507+0000 7fab56ffd640 1 -- 172.21.15.57:0/3596782524 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 155) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fab5001c460 con 0x7fab600b0d20 2023-10-19T14:23:03.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.507+0000 7fab56ffd640 1 --2- 172.21.15.57:0/3596782524 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fab4006af60 0x7fab4006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:23:03.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.507+0000 7fab56ffd640 1 -- 172.21.15.57:0/3596782524 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(268..268 src has 1..268) v4 ==== 6108+0+0 (secure 0 0 0) 0x7fab500898b0 con 0x7fab600b0d20 2023-10-19T14:23:03.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.507+0000 7fab57fff640 1 --2- 172.21.15.57:0/3596782524 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fab4006af60 0x7fab4006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:23:03.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.508+0000 7fab57fff640 1 --2- 172.21.15.57:0/3596782524 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fab4006af60 0x7fab4006d420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fab6006ff70 tx=0x7fab48007e30 comp rx=0 tx=0).ready entity=mgr.10520 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:23:03.516 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.511+0000 7fab56ffd640 1 -- 172.21.15.57:0/3596782524 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fab600c5ba0 con 0x7fab600b0d20 2023-10-19T14:23:03.687 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.684+0000 7fab64db0640 1 -- 172.21.15.57:0/3596782524 --> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fab600c5ba0 con 0x7fab4006af60 2023-10-19T14:23:03.690 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.687+0000 7fab56ffd640 1 -- 172.21.15.57:0/3596782524 <== mgr.10520 v2:172.21.15.179:6832/892689897 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+50758 (secure 0 0 0) 0x7fab600c5ba0 con 0x7fab4006af60 2023-10-19T14:23:03.690 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:23:03.691 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2023-10-19T14:23:03.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.689+0000 7fab64db0640 1 -- 172.21.15.57:0/3596782524 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fab4006af60 msgr2=0x7fab4006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:23:03.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.689+0000 7fab64db0640 1 --2- 172.21.15.57:0/3596782524 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fab4006af60 0x7fab4006d420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fab6006ff70 tx=0x7fab48007e30 comp rx=0 tx=0).stop 2023-10-19T14:23:03.692 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.689+0000 7fab64db0640 1 -- 172.21.15.57:0/3596782524 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab600b0d20 msgr2=0x7fab60155fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:23:03.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.689+0000 7fab64db0640 1 --2- 172.21.15.57:0/3596782524 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab600b0d20 0x7fab60155fa0 secure :-1 s=READY pgs=3082 cs=0 l=1 rev1=1 crypto rx=0x7fab5000c3c0 tx=0x7fab5000c880 comp rx=0 tx=0).stop 2023-10-19T14:23:03.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.689+0000 7fab64db0640 1 -- 172.21.15.57:0/3596782524 shutdown_connections 2023-10-19T14:23:03.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.689+0000 7fab64db0640 1 --2- 172.21.15.57:0/3596782524 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fab4006af60 0x7fab4006d420 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:23:03.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.689+0000 7fab64db0640 1 --2- 172.21.15.57:0/3596782524 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fab6013d5f0 0x7fab600c5e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:23:03.693 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.689+0000 7fab64db0640 1 --2- 172.21.15.57:0/3596782524 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fab600b1940 0x7fab600c1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:23:03.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.689+0000 7fab64db0640 1 --2- 172.21.15.57:0/3596782524 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab600b0d20 0x7fab60155fa0 unknown :-1 s=CLOSED pgs=3082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:23:03.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.689+0000 7fab64db0640 1 -- 172.21.15.57:0/3596782524 >> 172.21.15.57:0/3596782524 conn(0x7fab60069810 msgr2=0x7fab60069bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:23:03.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.689+0000 7fab64db0640 1 -- 172.21.15.57:0/3596782524 shutdown_connections 2023-10-19T14:23:03.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.689+0000 7fab64db0640 1 -- 172.21.15.57:0/3596782524 wait complete. 2023-10-19T14:23:03.702 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2023-10-19T14:23:01.816334+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2731,"num_read_kb":5751,"num_write":697,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3247,"ondisk_log_size":3247,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":457608,"kb_used_data":27684,"kb_used_omap":12,"kb_used_meta":429811,"kb_avail":754517112,"statfs":{"total":773094113280,"available":772625522688,"internally_reserved":0,"allocated":28348416,"data_stored":15209605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":440127033},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002647"},"pg_stats":[{"pgid":"1.0","version":"268'3247","reported_seq":4348,"reported_epoch":268,"state":"active+clean","last_fresh":"2023-10-19T14:22:46.117663+0000","last_change":"2023-10-19T14:22:46.117752+0000","last_active":"2023-10-19T14:22:46.117663+0000","last_peered":"2023-10-19T14:22:46.117663+0000","last_clean":"2023-10-19T14:22:46.117663+0000","last_became_active":"2023-10-19T12:44:08.635697+0000","last_became_peered":"2023-10-19T12:44:08.635697+0000","last_unstale":"2023-10-19T14:22:46.117663+0000","last_undegraded":"2023-10-19T14:22:46.117663+0000","last_fullsized":"2023-10-19T14:22:46.117663+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'3247","last_scrub_stamp":"2023-10-19T14:22:46.117565+0000","last_deep_scrub":"268'3247","last_deep_scrub_stamp":"2023-10-19T14:22:46.117565+0000","last_clean_scrub_stamp":"2023-10-19T14:22:46.117565+0000","objects_scrubbed":2,"log_size":3247,"log_dups_size":0,"ondisk_log_size":3247,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T22:52:23.196246+0000","scrub_duration":0.006929045,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2731,"num_read_kb":5751,"num_write":697,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2731,"num_read_kb":5751,"num_write":697,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3247,"ondisk_log_size":3247,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576031,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":67008,"kb_used_data":3896,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94304832,"statfs":{"total":96636764160,"available":96568147968,"internally_reserved":0,"allocated":3989504,"data_stored":2344054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":64616907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:22:11 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.53000000000000003,"15min":0.57299999999999995},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.86199999999999999},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.55600000000000005,"15min":0.59399999999999997},"min":{"1min":0.38300000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.63200000000000001}]},{"osd":1,"last update":"Thu Oct 19 14:22:11 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.57799999999999996,"15min":0.59399999999999997},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.56200000000000006,"15min":0.58899999999999997},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.754,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.55000000000000004}]},{"osd":2,"last update":"Thu Oct 19 14:22:11 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.61399999999999999,"15min":0.63800000000000001},"min":{"1min":0.42599999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.77300000000000002,"5min":0.84799999999999998,"15min":0.85499999999999998},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.623,"15min":0.65400000000000003},"min":{"1min":0.41599999999999998,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.024,"5min":1.024,"15min":1.024},"last":0.67200000000000004}]},{"osd":3,"last update":"Thu Oct 19 14:22:11 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.59499999999999997,"15min":0.628},"min":{"1min":0.36399999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.66400000000000003,"15min":0.68300000000000005},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.86799999999999999,"5min":0.873,"15min":0.89600000000000002},"last":0.57199999999999995}]},{"osd":4,"last update":"Thu Oct 19 14:22:11 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.51600000000000001,"15min":0.498},"min":{"1min":0.34100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.89600000000000002,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.54400000000000004,"15min":0.51600000000000001},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":1.032,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.441}]},{"osd":5,"last update":"Thu Oct 19 14:22:11 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.54000000000000004,"15min":0.51400000000000001},"min":{"1min":0.35299999999999998,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.58599999999999997,"15min":0.55600000000000005},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":1.069,"5min":1.069,"15min":1.069},"last":0.46500000000000002}]},{"osd":6,"last update":"Thu Oct 19 14:22:11 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.59899999999999998,"15min":0.57799999999999996},"min":{"1min":0.45600000000000002,"5min":0.38500000000000001,"15min":0.35999999999999999},"max":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.502},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.60899999999999999,"15min":0.57799999999999996},"min":{"1min":0.375,"5min":0.35599999999999998,"15min":0.33500000000000002},"max":{"1min":1.1100000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.39500000000000002}]}]},{"osd":6,"up_from":13,"seq":55834576031,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":60868,"kb_used_data":3316,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310972,"statfs":{"total":96636764160,"available":96574435328,"internally_reserved":0,"allocated":3395584,"data_stored":1753686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58915271},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:22:04 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.57399999999999995,"15min":0.60299999999999998},"min":{"1min":0.32500000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.70799999999999996,"5min":0.81200000000000006,"15min":0.85799999999999998},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.55700000000000005,"15min":0.60299999999999998},"min":{"1min":0.371,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.80600000000000005,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.56499999999999995}]},{"osd":1,"last update":"Thu Oct 19 14:22:04 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.59099999999999997,"15min":0.61499999999999999},"min":{"1min":0.41899999999999998,"5min":0.373,"15min":0.35799999999999998},"max":{"1min":0.81100000000000005,"5min":0.82799999999999996,"15min":1.165},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.59499999999999997,"15min":0.622},"min":{"1min":0.435,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.83099999999999996,"5min":0.83699999999999997,"15min":0.91300000000000003},"last":0.60999999999999999}]},{"osd":2,"last update":"Thu Oct 19 14:22:04 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.55900000000000005,"15min":0.59699999999999998},"min":{"1min":0.38300000000000001,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.72399999999999998,"5min":0.80100000000000005,"15min":0.873},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59699999999999998,"15min":0.621},"min":{"1min":0.39400000000000002,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.93100000000000005},"last":0.63500000000000001}]},{"osd":3,"last update":"Thu Oct 19 14:22:04 2023","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.622,"15min":0.64800000000000002},"min":{"1min":0.377,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.83799999999999997,"5min":0.875,"15min":0.875},"last":0.504},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.63,"15min":0.65300000000000002},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.78100000000000003,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.53500000000000003}]},{"osd":4,"last update":"Thu Oct 19 14:22:04 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.503,"15min":0.48799999999999999},"min":{"1min":0.31,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.89300000000000002,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.443},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51600000000000001,"15min":0.50900000000000001},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.78700000000000003,"5min":0.84899999999999998,"15min":0.85699999999999998},"last":0.40300000000000002}]},{"osd":5,"last update":"Thu Oct 19 14:22:04 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.52000000000000002,"15min":0.505},"min":{"1min":0.33200000000000002,"5min":0.32700000000000001,"15min":0.32500000000000001},"max":{"1min":0.86799999999999999,"5min":1.115,"15min":1.115},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.57799999999999996,"15min":0.54800000000000004},"min":{"1min":0.33900000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.51600000000000001}]},{"osd":7,"last update":"Thu Oct 19 14:22:04 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.57299999999999995,"15min":0.55100000000000005},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.82599999999999996,"5min":0.89300000000000002,"15min":0.90100000000000002},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.58199999999999996,"15min":0.55700000000000005},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.82899999999999996,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.58299999999999996}]}]},{"osd":5,"up_from":13,"seq":55834576031,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":66432,"kb_used_data":3316,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305408,"statfs":{"total":96636764160,"available":96568737792,"internally_reserved":0,"allocated":3395584,"data_stored":1753686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:22:51 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55900000000000005,"15min":0.59899999999999998},"min":{"1min":0.34999999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.73299999999999998,"5min":0.79700000000000004,"15min":0.86399999999999999},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.58699999999999997,"15min":0.627},"min":{"1min":0.34599999999999997,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.75700000000000001,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.60699999999999998}]},{"osd":1,"last update":"Thu Oct 19 14:22:51 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.58499999999999996,"15min":0.61799999999999999},"min":{"1min":0.38900000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.76700000000000002,"5min":0.83299999999999996,"15min":0.877},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.60199999999999998,"15min":0.63100000000000001},"min":{"1min":0.39900000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.70799999999999996,"5min":0.877,"15min":0.88500000000000001},"last":0.47999999999999998}]},{"osd":2,"last update":"Thu Oct 19 14:22:51 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.57299999999999995,"15min":0.60899999999999999},"min":{"1min":0.374,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.83299999999999996,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.61599999999999999,"15min":0.64300000000000002},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.73799999999999999,"5min":0.83299999999999996,"15min":0.90800000000000003},"last":0.628}]},{"osd":3,"last update":"Thu Oct 19 14:22:51 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.59499999999999997,"15min":0.63100000000000001},"min":{"1min":0.44900000000000001,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.70999999999999996,"5min":0.84399999999999997,"15min":0.86099999999999999},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.61599999999999999,"15min":0.65600000000000003},"min":{"1min":0.50600000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.79400000000000004,"5min":0.89900000000000002,"15min":0.90900000000000003},"last":0.59499999999999997}]},{"osd":4,"last update":"Thu Oct 19 14:22:51 2023","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.501,"15min":0.48999999999999999},"min":{"1min":0.35599999999999998,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.71499999999999997,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.52700000000000002,"15min":0.503},"min":{"1min":0.36499999999999999,"5min":0.308,"15min":0.28599999999999998},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.53700000000000003}]},{"osd":6,"last update":"Thu Oct 19 14:22:51 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53200000000000003,"15min":0.504},"min":{"1min":0.37,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.78500000000000003,"5min":0.86599999999999999,"15min":0.88800000000000001},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.56000000000000005,"15min":0.54000000000000004},"min":{"1min":0.38100000000000001,"5min":0.36899999999999999,"15min":0.33400000000000002},"max":{"1min":0.753,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.373}]},{"osd":7,"last update":"Thu Oct 19 14:22:51 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52900000000000003,"15min":0.51700000000000002},"min":{"1min":0.34499999999999997,"5min":0.312,"15min":0.312},"max":{"1min":0.77000000000000002,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.437},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58999999999999997,"15min":0.57299999999999995},"min":{"1min":0.42099999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.80200000000000005,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.499}]}]},{"osd":4,"up_from":13,"seq":55834576031,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48388,"kb_used_data":3316,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323452,"statfs":{"total":96636764160,"available":96587214848,"internally_reserved":0,"allocated":3395584,"data_stored":1753686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:22:16 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.53900000000000003,"15min":0.56499999999999995},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.28599999999999998},"max":{"1min":0.69799999999999995,"5min":0.83799999999999997,"15min":0.879},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.505,"5min":0.55700000000000005,"15min":0.58799999999999997},"min":{"1min":0.33400000000000002,"5min":0.31,"15min":0.31},"max":{"1min":0.71699999999999997,"5min":0.82599999999999996,"15min":0.873},"last":0.60399999999999998}]},{"osd":1,"last update":"Thu Oct 19 14:22:16 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.58499999999999996,"15min":0.626},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.32600000000000001},"max":{"1min":0.70299999999999996,"5min":0.80400000000000005,"15min":0.92700000000000005},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.59299999999999997,"15min":0.63200000000000001},"min":{"1min":0.374,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.68500000000000005,"5min":0.90000000000000002,"15min":1.0940000000000001},"last":0.69599999999999995}]},{"osd":2,"last update":"Thu Oct 19 14:22:16 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.61099999999999999,"15min":0.63800000000000001},"min":{"1min":0.39000000000000001,"5min":0.372,"15min":0.32600000000000001},"max":{"1min":0.73699999999999999,"5min":1.008,"15min":1.008},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.625,"15min":0.65900000000000003},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.749,"5min":0.98799999999999999,"15min":1.008},"last":0.80000000000000004}]},{"osd":3,"last update":"Thu Oct 19 14:22:16 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.57999999999999996,"15min":0.621},"min":{"1min":0.39000000000000001,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.755,"5min":0.83399999999999996,"15min":0.91900000000000004},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.64400000000000002,"15min":0.65900000000000003},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.76900000000000002,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.81699999999999995}]},{"osd":5,"last update":"Thu Oct 19 14:22:16 2023","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.48999999999999999,"15min":0.47899999999999998},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.755,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.51200000000000001,"15min":0.501},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29199999999999998},"max":{"1min":0.74399999999999999,"5min":0.873,"15min":0.873},"last":0.46999999999999997}]},{"osd":6,"last update":"Thu Oct 19 14:22:16 2023","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.50600000000000001,"15min":0.49399999999999999},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.28199999999999997},"max":{"1min":0.74099999999999999,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54700000000000004,"15min":0.54400000000000004},"min":{"1min":0.371,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.79700000000000004,"5min":0.88,"15min":1.2969999999999999},"last":0.50800000000000001}]},{"osd":7,"last update":"Thu Oct 19 14:22:16 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.54000000000000004,"15min":0.54400000000000004},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.72699999999999998,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60699999999999998,"15min":0.59999999999999998},"min":{"1min":0.42499999999999999,"5min":0.32100000000000001,"15min":0.318},"max":{"1min":0.77700000000000002,"5min":0.99399999999999999,"15min":1.032},"last":0.55800000000000005}]}]},{"osd":3,"up_from":13,"seq":55834576030,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":56704,"kb_used_data":3316,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315136,"statfs":{"total":96636764160,"available":96578699264,"internally_reserved":0,"allocated":3395584,"data_stored":1753686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54655432},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:22:55 2023","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.376,"15min":0.41099999999999998},"min":{"1min":0.23499999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.57999999999999996,"5min":0.60599999999999998,"15min":0.70699999999999996},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.378,"15min":0.41599999999999998},"min":{"1min":0.29499999999999998,"5min":0.217,"15min":0.217},"max":{"1min":0.60699999999999998,"5min":0.61899999999999999,"15min":0.752},"last":0.39000000000000001}]},{"osd":1,"last update":"Thu Oct 19 14:22:55 2023","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":0.38100000000000001,"15min":0.42299999999999999},"min":{"1min":0.25900000000000001,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.51500000000000001,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.433,"15min":0.46700000000000003},"min":{"1min":0.30099999999999999,"5min":0.26600000000000001,"15min":0.26400000000000001},"max":{"1min":0.67000000000000004,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.41899999999999998}]},{"osd":2,"last update":"Thu Oct 19 14:22:55 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.44800000000000001,"15min":0.47699999999999998},"min":{"1min":0.33500000000000002,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.66700000000000004,"5min":0.67200000000000004,"15min":0.73099999999999998},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.45400000000000001,"15min":0.498},"min":{"1min":0.29599999999999999,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.58599999999999997,"5min":1.212,"15min":1.212},"last":0.34799999999999998}]},{"osd":4,"last update":"Thu Oct 19 14:22:55 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.65100000000000002,"15min":0.69399999999999995},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.84599999999999997,"5min":0.96099999999999997,"15min":1.157},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.65900000000000003,"15min":0.71399999999999997},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.81799999999999995,"5min":0.97699999999999998,"15min":1.173},"last":0.71299999999999997}]},{"osd":5,"last update":"Thu Oct 19 14:22:55 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.64600000000000002,"15min":0.68200000000000005},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.41999999999999998},"max":{"1min":0.88400000000000001,"5min":0.96799999999999997,"15min":1.0109999999999999},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.627,"5min":0.64500000000000002,"15min":0.68100000000000005},"min":{"1min":0.49299999999999999,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.80600000000000005,"5min":0.99399999999999999,"15min":1.105},"last":0.66400000000000003}]},{"osd":6,"last update":"Thu Oct 19 14:22:55 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.626,"15min":0.67800000000000005},"min":{"1min":0.435,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.80700000000000005,"5min":1.0009999999999999,"15min":1.0669999999999999},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.65600000000000003,"15min":0.70299999999999996},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.79100000000000004,"5min":1.0029999999999999,"15min":1.079},"last":0.63800000000000001}]},{"osd":7,"last update":"Thu Oct 19 14:22:55 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.65400000000000003,"15min":0.69499999999999995},"min":{"1min":0.44900000000000001,"5min":0.434,"15min":0.434},"max":{"1min":0.82799999999999996,"5min":0.94999999999999996,"15min":1.085},"last":0.751},{"interface":"front","average":{"1min":0.627,"5min":0.67300000000000004,"15min":0.71999999999999997},"min":{"1min":0.51500000000000001,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.84399999999999997,"5min":0.97199999999999998,"15min":1.1739999999999999},"last":0.79800000000000004}]}]},{"osd":2,"up_from":13,"seq":55834576032,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48388,"kb_used_data":3316,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323452,"statfs":{"total":96636764160,"available":96587214848,"internally_reserved":0,"allocated":3395584,"data_stored":1753686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:22:40 2023","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.36199999999999999,"15min":0.39100000000000001},"min":{"1min":0.26100000000000001,"5min":0.218,"15min":0.218},"max":{"1min":0.41599999999999998,"5min":0.66600000000000004,"15min":0.69699999999999995},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.39400000000000002,"15min":0.42799999999999999},"min":{"1min":0.25800000000000001,"5min":0.248,"15min":0.223},"max":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.71399999999999997},"last":0.50600000000000001}]},{"osd":1,"last update":"Thu Oct 19 14:22:40 2023","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.37,"15min":0.40200000000000002},"min":{"1min":0.26300000000000001,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.501,"5min":0.55600000000000005,"15min":0.65400000000000003},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.43099999999999999,"15min":0.45700000000000002},"min":{"1min":0.28399999999999997,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.59699999999999998,"5min":1.742,"15min":1.742},"last":0.58399999999999996}]},{"osd":3,"last update":"Thu Oct 19 14:22:40 2023","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.433,"15min":0.47399999999999998},"min":{"1min":0.315,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.56299999999999994,"5min":0.65900000000000003,"15min":0.74399999999999999},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.438,"5min":0.45600000000000002,"15min":0.496},"min":{"1min":0.34399999999999997,"5min":0.32000000000000001,"15min":0.315},"max":{"1min":0.66400000000000003,"5min":0.67700000000000005,"15min":0.83599999999999997},"last":0.56899999999999995}]},{"osd":4,"last update":"Thu Oct 19 14:22:40 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.63200000000000001,"15min":0.65900000000000003},"min":{"1min":0.41399999999999998,"5min":0.39600000000000002,"15min":0.39500000000000002},"max":{"1min":0.84099999999999997,"5min":0.89200000000000002,"15min":0.98499999999999999},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.64300000000000002,"15min":0.67600000000000005},"min":{"1min":0.46300000000000002,"5min":0.38900000000000001,"15min":0.35199999999999998},"max":{"1min":0.90300000000000002,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.70999999999999996}]},{"osd":5,"last update":"Thu Oct 19 14:22:40 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.66800000000000004,"15min":0.69199999999999995},"min":{"1min":0.45800000000000002,"5min":0.443,"15min":0.40600000000000003},"max":{"1min":0.872,"5min":1.4330000000000001,"15min":1.4330000000000001},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.65900000000000003,"15min":0.67900000000000005},"min":{"1min":0.47499999999999998,"5min":0.45100000000000001,"15min":0.39300000000000002},"max":{"1min":0.83499999999999996,"5min":1.595,"15min":1.595},"last":0.74199999999999999}]},{"osd":6,"last update":"Thu Oct 19 14:22:40 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.66000000000000003,"15min":0.69799999999999995},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.879,"5min":0.996,"15min":1.0189999999999999},"last":0.86699999999999999},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.68300000000000005,"15min":0.71299999999999997},"min":{"1min":0.503,"5min":0.433,"15min":0.40600000000000003},"max":{"1min":0.83899999999999997,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":0.84999999999999998}]},{"osd":7,"last update":"Thu Oct 19 14:22:40 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.67000000000000004,"15min":0.71199999999999997},"min":{"1min":0.47099999999999997,"5min":0.44800000000000001,"15min":0.41799999999999998},"max":{"1min":0.84399999999999997,"5min":0.97999999999999998,"15min":1.0289999999999999},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.70299999999999996,"15min":0.72399999999999998},"min":{"1min":0.44,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.89100000000000001,"5min":1.53,"15min":1.53},"last":0.81699999999999995}]}]},{"osd":1,"up_from":13,"seq":55834576031,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48384,"kb_used_data":3316,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323456,"statfs":{"total":96636764160,"available":96587218944,"internally_reserved":0,"allocated":3395584,"data_stored":1753686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 19 14:22:44 2023","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.36599999999999999,"15min":0.39700000000000002},"min":{"1min":0.22500000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.50900000000000001,"5min":0.61599999999999999,"15min":0.67400000000000004},"last":0.434},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.38300000000000001,"15min":0.442},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.38900000000000001,"5min":0.64200000000000002,"15min":0.82399999999999995},"last":0.40999999999999998}]},{"osd":2,"last update":"Thu Oct 19 14:22:44 2023","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.39700000000000002,"15min":0.432},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.51900000000000002,"5min":0.58599999999999997,"15min":0.77500000000000002},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.44900000000000001,"15min":0.48199999999999998},"min":{"1min":0.32100000000000001,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.52600000000000002,"5min":0.68700000000000006,"15min":0.82799999999999996},"last":0.45300000000000001}]},{"osd":3,"last update":"Thu Oct 19 14:22:44 2023","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.46400000000000002,"15min":0.495},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.61499999999999999,"5min":0.74399999999999999,"15min":0.78900000000000003},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.372,"5min":0.40899999999999997,"15min":0.441},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.54600000000000004,"5min":0.65200000000000002,"15min":0.69499999999999995},"last":0.46500000000000002}]},{"osd":4,"last update":"Thu Oct 19 14:22:44 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.60999999999999999,"15min":0.65900000000000003},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.76100000000000001,"5min":0.95999999999999996,"15min":1.042},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.628,"15min":0.66100000000000003},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.32300000000000001},"max":{"1min":0.85599999999999998,"5min":0.89400000000000002,"15min":1.133},"last":0.66900000000000004}]},{"osd":5,"last update":"Thu Oct 19 14:22:44 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.64400000000000002,"15min":0.67500000000000004},"min":{"1min":0.434,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.81200000000000006,"5min":0.88400000000000001,"15min":1.099},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.64800000000000002,"15min":0.69499999999999995},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40500000000000003},"max":{"1min":0.81299999999999994,"5min":1.0009999999999999,"15min":1.121},"last":0.77000000000000002}]},{"osd":6,"last update":"Thu Oct 19 14:22:44 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.628,"15min":0.66800000000000004},"min":{"1min":0.41599999999999998,"5min":0.373,"15min":0.373},"max":{"1min":0.84199999999999997,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.66200000000000003,"15min":0.70599999999999996},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.82199999999999995,"5min":0.93100000000000005,"15min":1.107},"last":0.79000000000000004}]},{"osd":7,"last update":"Thu Oct 19 14:22:44 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.67100000000000004,"15min":0.71399999999999997},"min":{"1min":0.47899999999999998,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.85799999999999998,"5min":0.99099999999999999,"15min":1.075},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.65200000000000002,"15min":0.68999999999999995},"min":{"1min":0.46200000000000002,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.79800000000000004,"5min":0.97599999999999998,"15min":1.0169999999999999},"last":0.755}]}]},{"osd":0,"up_from":13,"seq":55834576032,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61436,"kb_used_data":3892,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310404,"statfs":{"total":96636764160,"available":96573853696,"internally_reserved":0,"allocated":3985408,"data_stored":2343435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Oct 19 14:22:13 2023","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.39000000000000001,"15min":0.40600000000000003},"min":{"1min":0.254,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.47699999999999998,"5min":0.59999999999999998,"15min":0.624},"last":0.378},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.40699999999999997,"15min":0.44900000000000001},"min":{"1min":0.252,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.81100000000000005},"last":0.35799999999999998}]},{"osd":2,"last update":"Thu Oct 19 14:22:13 2023","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.45700000000000002,"15min":0.47699999999999998},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.58699999999999997,"5min":0.73099999999999998,"15min":0.754},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.41399999999999998,"15min":0.439},"min":{"1min":0.316,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.54000000000000004,"5min":0.60199999999999998,"15min":0.69099999999999995},"last":0.29899999999999999}]},{"osd":3,"last update":"Thu Oct 19 14:22:13 2023","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.40300000000000002,"15min":0.433},"min":{"1min":0.28399999999999997,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.56899999999999995,"5min":0.58799999999999997,"15min":0.66800000000000004},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.46800000000000003,"15min":0.503},"min":{"1min":0.314,"5min":0.254,"15min":0.254},"max":{"1min":0.55600000000000005,"5min":0.78100000000000003,"15min":0.84599999999999997},"last":0.441}]},{"osd":4,"last update":"Thu Oct 19 14:22:13 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.627,"15min":0.66600000000000004},"min":{"1min":0.437,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":1.1890000000000001},"last":0.745},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.626,"15min":0.67000000000000004},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.42599999999999999},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.996},"last":0.629}]},{"osd":5,"last update":"Thu Oct 19 14:22:13 2023","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.66100000000000003,"15min":0.70999999999999996},"min":{"1min":0.46500000000000002,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.86599999999999999,"5min":0.98499999999999999,"15min":1.1579999999999999},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.67400000000000004,"15min":0.71099999999999997},"min":{"1min":0.52700000000000002,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.91700000000000004,"5min":0.98299999999999998,"15min":1.2310000000000001},"last":0.68600000000000005}]},{"osd":6,"last update":"Thu Oct 19 14:22:13 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.65400000000000003,"15min":0.69599999999999995},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.95499999999999996,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.628,"5min":0.64100000000000001,"15min":0.68999999999999995},"min":{"1min":0.44600000000000001,"5min":0.441,"15min":0.441},"max":{"1min":0.86399999999999999,"5min":0.89900000000000002,"15min":1.01},"last":0.75800000000000001}]},{"osd":7,"last update":"Thu Oct 19 14:22:13 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.65500000000000003,"15min":0.70499999999999996},"min":{"1min":0.52300000000000002,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.89500000000000002,"5min":0.93400000000000005,"15min":1.014},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.65800000000000003,"15min":0.69299999999999995},"min":{"1min":0.498,"5min":0.434,"15min":0.434},"max":{"1min":0.88400000000000001,"5min":0.93799999999999994,"15min":1.0349999999999999},"last":0.65300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-19T14:23:03.705 DEBUG:teuthology.orchestra.run.smithi057:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-10-19T14:23:03.813 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.810+0000 7f9810b4c640 1 -- 172.21.15.57:0/3203609717 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f980c0cf140 msgr2=0x7f980c0cc310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:23:03.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.810+0000 7f9810b4c640 1 --2- 172.21.15.57:0/3203609717 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f980c0cf140 0x7f980c0cc310 secure :-1 s=READY pgs=3084 cs=0 l=1 rev1=1 crypto rx=0x7f980c09ecd0 tx=0x7f97fc02c110 comp rx=0 tx=0).stop 2023-10-19T14:23:03.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.810+0000 7f9810b4c640 1 -- 172.21.15.57:0/3203609717 shutdown_connections 2023-10-19T14:23:03.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.810+0000 7f9810b4c640 1 --2- 172.21.15.57:0/3203609717 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f980c0cf140 0x7f980c0cc310 unknown :-1 s=CLOSED pgs=3084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:23:03.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.810+0000 7f9810b4c640 1 --2- 172.21.15.57:0/3203609717 >> v2:172.21.15.179:3300/0 conn(0x7f980c15b070 0x7f980c0d0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:23:03.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.810+0000 7f9810b4c640 1 -- 172.21.15.57:0/3203609717 >> 172.21.15.57:0/3203609717 conn(0x7f980c08a7e0 msgr2=0x7f980c09a100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:23:03.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.810+0000 7f9810b4c640 1 -- 172.21.15.57:0/3203609717 shutdown_connections 2023-10-19T14:23:03.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.810+0000 7f9810b4c640 1 -- 172.21.15.57:0/3203609717 wait complete. 2023-10-19T14:23:03.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.811+0000 7f9810b4c640 1 Processor -- start 2023-10-19T14:23:03.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.812+0000 7f9810b4c640 1 -- start start 2023-10-19T14:23:03.817 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.812+0000 7f9810b4c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f980c0cf140 0x7f980c14a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:23:03.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.812+0000 7f9810b4c640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f980c0d1080 0x7f980c0c9a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:23:03.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.812+0000 7f9810b4c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f980c15b070 0x7f980c0cd050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:23:03.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.812+0000 7f9810b4c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f980c14aae0 con 0x7f980c15b070 2023-10-19T14:23:03.818 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.812+0000 7f9810b4c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f980c0ca440 con 0x7f980c0cf140 2023-10-19T14:23:03.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.812+0000 7f9810b4c640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f980c0ca720 con 0x7f980c0d1080 2023-10-19T14:23:03.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.812+0000 7f980ad9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f980c15b070 0x7f980c0cd050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:23:03.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.812+0000 7f980ad9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f980c15b070 0x7f980c0cd050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46096/0 (socket says 172.21.15.57:46096) 2023-10-19T14:23:03.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.812+0000 7f980ad9d640 1 -- 172.21.15.57:0/2912163153 learned_addr learned my addr 172.21.15.57:0/2912163153 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:23:03.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.813+0000 7f980a59c640 1 --2- 172.21.15.57:0/2912163153 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f980c0cf140 0x7f980c14a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:23:03.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.813+0000 7f9809d9b640 1 --2- 172.21.15.57:0/2912163153 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f980c0d1080 0x7f980c0c9a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:23:03.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.813+0000 7f980ad9d640 1 -- 172.21.15.57:0/2912163153 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f980c0d1080 msgr2=0x7f980c0c9a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:23:03.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.813+0000 7f980ad9d640 1 --2- 172.21.15.57:0/2912163153 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f980c0d1080 0x7f980c0c9a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:23:03.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.813+0000 7f980ad9d640 1 -- 172.21.15.57:0/2912163153 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f980c0cf140 msgr2=0x7f980c14a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:23:03.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.813+0000 7f980ad9d640 1 --2- 172.21.15.57:0/2912163153 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f980c0cf140 0x7f980c14a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:23:03.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.813+0000 7f980ad9d640 1 -- 172.21.15.57:0/2912163153 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97fc009840 con 0x7f980c15b070 2023-10-19T14:23:03.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.813+0000 7f980ad9d640 1 --2- 172.21.15.57:0/2912163153 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f980c15b070 0x7f980c0cd050 secure :-1 s=READY pgs=3085 cs=0 l=1 rev1=1 crypto rx=0x7f97fc036040 tx=0x7f97fc0042e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:23:03.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.813+0000 7f97faffd640 1 -- 172.21.15.57:0/2912163153 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97fc0046e0 con 0x7f980c15b070 2023-10-19T14:23:03.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.813+0000 7f97faffd640 1 -- 172.21.15.57:0/2912163153 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(5 keys) v1 ==== 141+0+0 (secure 0 0 0) 0x7f97fc004840 con 0x7f980c15b070 2023-10-19T14:23:03.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.813+0000 7f9810b4c640 1 -- 172.21.15.57:0/2912163153 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f980c0caa00 con 0x7f980c15b070 2023-10-19T14:23:03.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.813+0000 7f97faffd640 1 -- 172.21.15.57:0/2912163153 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97fc034800 con 0x7f980c15b070 2023-10-19T14:23:03.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.813+0000 7f9810b4c640 1 -- 172.21.15.57:0/2912163153 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f980c0cace0 con 0x7f980c15b070 2023-10-19T14:23:03.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.814+0000 7f9810b4c640 1 -- 172.21.15.57:0/2912163153 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f980c14ac50 con 0x7f980c15b070 2023-10-19T14:23:03.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.815+0000 7f97faffd640 1 -- 172.21.15.57:0/2912163153 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 155) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f97fc034960 con 0x7f980c15b070 2023-10-19T14:23:03.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.815+0000 7f97faffd640 1 --2- 172.21.15.57:0/2912163153 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f97ec06af60 0x7f97ec06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:23:03.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.815+0000 7f97faffd640 1 -- 172.21.15.57:0/2912163153 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(268..268 src has 1..268) v4 ==== 6108+0+0 (secure 0 0 0) 0x7f97fc0acb60 con 0x7f980c15b070 2023-10-19T14:23:03.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.815+0000 7f980a59c640 1 --2- 172.21.15.57:0/2912163153 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f97ec06af60 0x7f97ec06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:23:03.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.816+0000 7f980a59c640 1 --2- 172.21.15.57:0/2912163153 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f97ec06af60 0x7f97ec06d420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9804005fd0 tx=0x7f9804005850 comp rx=0 tx=0).ready entity=mgr.10520 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:23:03.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.819+0000 7f97faffd640 1 -- 172.21.15.57:0/2912163153 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7f97fc077a00 con 0x7f980c15b070 2023-10-19T14:23:03.999 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:03.996+0000 7f9810b4c640 1 -- 172.21.15.57:0/2912163153 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7f980c09ed00 con 0x7f980c15b070 2023-10-19T14:23:04.002 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:04.000+0000 7f97faffd640 1 -- 172.21.15.57:0/2912163153 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v87)=0 v87) v1 ==== 123+0+0 (secure 0 0 0) 0x7f97fc0430c0 con 0x7f980c15b070 2023-10-19T14:23:04.003 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:04.000+0000 7f97faffd640 1 -- 172.21.15.57:0/2912163153 <== mon.0 v2:172.21.15.57:3300/0 8 ==== config(6 keys) v1 ==== 172+0+0 (secure 0 0 0) 0x7f97fc077a00 con 0x7f980c15b070 2023-10-19T14:23:04.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:04.002+0000 7f9810b4c640 1 -- 172.21.15.57:0/2912163153 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f97ec06af60 msgr2=0x7f97ec06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:23:04.005 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:04.002+0000 7f9810b4c640 1 --2- 172.21.15.57:0/2912163153 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f97ec06af60 0x7f97ec06d420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9804005fd0 tx=0x7f9804005850 comp rx=0 tx=0).stop 2023-10-19T14:23:04.006 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:04.002+0000 7f9810b4c640 1 -- 172.21.15.57:0/2912163153 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f980c15b070 msgr2=0x7f980c0cd050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:23:04.006 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:04.002+0000 7f9810b4c640 1 --2- 172.21.15.57:0/2912163153 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f980c15b070 0x7f980c0cd050 secure :-1 s=READY pgs=3085 cs=0 l=1 rev1=1 crypto rx=0x7f97fc036040 tx=0x7f97fc0042e0 comp rx=0 tx=0).stop 2023-10-19T14:23:04.006 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:04.002+0000 7f9810b4c640 1 -- 172.21.15.57:0/2912163153 shutdown_connections 2023-10-19T14:23:04.006 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:04.002+0000 7f9810b4c640 1 --2- 172.21.15.57:0/2912163153 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7f97ec06af60 0x7f97ec06d420 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:23:04.006 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:04.002+0000 7f9810b4c640 1 --2- 172.21.15.57:0/2912163153 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f980c15b070 0x7f980c0cd050 unknown :-1 s=CLOSED pgs=3085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:23:04.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:04.002+0000 7f9810b4c640 1 --2- 172.21.15.57:0/2912163153 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7f980c0d1080 0x7f980c0c9a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:23:04.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:04.002+0000 7f9810b4c640 1 --2- 172.21.15.57:0/2912163153 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f980c0cf140 0x7f980c14a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:23:04.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:04.002+0000 7f9810b4c640 1 -- 172.21.15.57:0/2912163153 >> 172.21.15.57:0/2912163153 conn(0x7f980c08a7e0 msgr2=0x7f980c15d670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:23:04.007 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:04.002+0000 7f9810b4c640 1 -- 172.21.15.57:0/2912163153 shutdown_connections 2023-10-19T14:23:04.008 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:04.002+0000 7f9810b4c640 1 -- 172.21.15.57:0/2912163153 wait complete. 2023-10-19T14:23:04.015 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-19T14:23:04.016 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-19T14:23:04.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:04.123+0000 7fd84ab5e640 1 -- 172.21.15.57:0/1648314358 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd8300047c0 msgr2=0x7fd830004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:23:04.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:04.123+0000 7fd84ab5e640 1 --2- 172.21.15.57:0/1648314358 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd8300047c0 0x7fd830004c40 secure :-1 s=READY pgs=3087 cs=0 l=1 rev1=1 crypto rx=0x7fd834001f00 tx=0x7fd8300057f0 comp rx=0 tx=0).stop 2023-10-19T14:23:04.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:04.123+0000 7fd84ab5e640 1 -- 172.21.15.57:0/1648314358 shutdown_connections 2023-10-19T14:23:04.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:04.123+0000 7fd84ab5e640 1 --2- 172.21.15.57:0/1648314358 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd8300047c0 0x7fd830004c40 unknown :-1 s=CLOSED pgs=3087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:23:04.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:04.123+0000 7fd84ab5e640 1 -- 172.21.15.57:0/1648314358 >> 172.21.15.57:0/1648314358 conn(0x7fd844069810 msgr2=0x7fd844069c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:23:04.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:04.123+0000 7fd84ab5e640 1 -- 172.21.15.57:0/1648314358 shutdown_connections 2023-10-19T14:23:04.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:04.123+0000 7fd84ab5e640 1 -- 172.21.15.57:0/1648314358 wait complete. 2023-10-19T14:23:04.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:04.124+0000 7fd84ab5e640 1 Processor -- start 2023-10-19T14:23:04.130 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:04.124+0000 7fd84ab5e640 1 -- start start 2023-10-19T14:23:04.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:04.125+0000 7fd84ab5e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd8300047c0 0x7fd844152970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:23:04.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:04.125+0000 7fd84ab5e640 1 --2- >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd844155b80 0x7fd844156000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:23:04.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:04.125+0000 7fd84ab5e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd8440acc50 0x7fd8440ad0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:23:04.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:04.125+0000 7fd84ab5e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8440ad610 con 0x7fd8440acc50 2023-10-19T14:23:04.131 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:04.125+0000 7fd84ab5e640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8440ac200 con 0x7fd8300047c0 2023-10-19T14:23:04.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:04.125+0000 7fd84ab5e640 1 -- --> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8441539e0 con 0x7fd844155b80 2023-10-19T14:23:04.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:04.125+0000 7fd8490fb640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd8440acc50 0x7fd8440ad0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:23:04.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:04.125+0000 7fd8490fb640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd8440acc50 0x7fd8440ad0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46108/0 (socket says 172.21.15.57:46108) 2023-10-19T14:23:04.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:04.125+0000 7fd8490fb640 1 -- 172.21.15.57:0/46508950 learned_addr learned my addr 172.21.15.57:0/46508950 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-10-19T14:23:04.132 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:04.125+0000 7fd843fff640 1 --2- 172.21.15.57:0/46508950 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd844155b80 0x7fd844156000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:23:04.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:04.125+0000 7fd8488fa640 1 --2- 172.21.15.57:0/46508950 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd8300047c0 0x7fd844152970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:23:04.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:04.125+0000 7fd8490fb640 1 -- 172.21.15.57:0/46508950 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd844155b80 msgr2=0x7fd844156000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:23:04.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:04.125+0000 7fd8490fb640 1 --2- 172.21.15.57:0/46508950 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd844155b80 0x7fd844156000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:23:04.133 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:04.125+0000 7fd8490fb640 1 -- 172.21.15.57:0/46508950 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd8300047c0 msgr2=0x7fd844152970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:23:04.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:04.125+0000 7fd8490fb640 1 --2- 172.21.15.57:0/46508950 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd8300047c0 0x7fd844152970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:23:04.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:04.125+0000 7fd8490fb640 1 -- 172.21.15.57:0/46508950 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd844153c60 con 0x7fd8440acc50 2023-10-19T14:23:04.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:04.125+0000 7fd8490fb640 1 --2- 172.21.15.57:0/46508950 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd8440acc50 0x7fd8440ad0d0 secure :-1 s=READY pgs=3088 cs=0 l=1 rev1=1 crypto rx=0x7fd8440c93f0 tx=0x7fd82c00e9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:23:04.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:04.125+0000 7fd8417fa640 1 -- 172.21.15.57:0/46508950 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd82c011870 con 0x7fd8440acc50 2023-10-19T14:23:04.134 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:04.126+0000 7fd8417fa640 1 -- 172.21.15.57:0/46508950 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(6 keys) v1 ==== 172+0+0 (secure 0 0 0) 0x7fd82c011b00 con 0x7fd8440acc50 2023-10-19T14:23:04.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:04.126+0000 7fd84ab5e640 1 -- 172.21.15.57:0/46508950 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd844153f40 con 0x7fd8440acc50 2023-10-19T14:23:04.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:04.126+0000 7fd8417fa640 1 -- 172.21.15.57:0/46508950 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd82c0103c0 con 0x7fd8440acc50 2023-10-19T14:23:04.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:04.126+0000 7fd84ab5e640 1 -- 172.21.15.57:0/46508950 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd844154500 con 0x7fd8440acc50 2023-10-19T14:23:04.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:04.127+0000 7fd84ab5e640 1 -- 172.21.15.57:0/46508950 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd844069100 con 0x7fd8440acc50 2023-10-19T14:23:04.135 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:04.127+0000 7fd8417fa640 1 -- 172.21.15.57:0/46508950 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 155) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd82c010520 con 0x7fd8440acc50 2023-10-19T14:23:04.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:04.127+0000 7fd8417fa640 1 --2- 172.21.15.57:0/46508950 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fd82406af60 0x7fd82406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-19T14:23:04.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:04.128+0000 7fd8417fa640 1 -- 172.21.15.57:0/46508950 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(268..268 src has 1..268) v4 ==== 6108+0+0 (secure 0 0 0) 0x7fd82c089ff0 con 0x7fd8440acc50 2023-10-19T14:23:04.136 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:04.128+0000 7fd8488fa640 1 --2- 172.21.15.57:0/46508950 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fd82406af60 0x7fd82406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-19T14:23:04.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:04.128+0000 7fd8488fa640 1 --2- 172.21.15.57:0/46508950 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fd82406af60 0x7fd82406d420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd83c009b30 tx=0x7fd83c005f90 comp rx=0 tx=0).ready entity=mgr.10520 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-19T14:23:04.137 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:04.132+0000 7fd8417fa640 1 -- 172.21.15.57:0/46508950 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179335 (secure 0 0 0) 0x7fd82c054cb0 con 0x7fd8440acc50 2023-10-19T14:23:04.351 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:04.348+0000 7fd84ab5e640 1 -- 172.21.15.57:0/46508950 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd844070530 con 0x7fd8440acc50 2023-10-19T14:23:04.352 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:04.349+0000 7fd8417fa640 1 -- 172.21.15.57:0/46508950 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 118 v118) v1 ==== 95+0+3714 (secure 0 0 0) 0x7fd82c0582d0 con 0x7fd8440acc50 2023-10-19T14:23:04.352 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-10-19T14:23:04.353 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":118,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10280,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/1738328496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6835","nonce":1738328496},{"type":"v1","addr":"172.21.15.179:6837","nonce":1738328496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":113},{"gid":10637,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6836/2769261762","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2769261762},{"type":"v1","addr":"172.21.15.179:6836","nonce":2769261762}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":118},{"gid":13344,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6834/3191801579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":3191801579},{"type":"v1","addr":"172.21.15.57:6834","nonce":3191801579}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":113},{"gid":13347,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/3876822488","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6833","nonce":3876822488},{"type":"v1","addr":"172.21.15.57:6835","nonce":3876822488}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":112}],"filesystems":[]} 2023-10-19T14:23:04.353 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 118 2023-10-19T14:23:04.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:04.351+0000 7fd84ab5e640 1 -- 172.21.15.57:0/46508950 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fd82406af60 msgr2=0x7fd82406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:23:04.354 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:04.351+0000 7fd84ab5e640 1 --2- 172.21.15.57:0/46508950 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fd82406af60 0x7fd82406d420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd83c009b30 tx=0x7fd83c005f90 comp rx=0 tx=0).stop 2023-10-19T14:23:04.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:04.351+0000 7fd84ab5e640 1 -- 172.21.15.57:0/46508950 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd8440acc50 msgr2=0x7fd8440ad0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-19T14:23:04.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:04.351+0000 7fd84ab5e640 1 --2- 172.21.15.57:0/46508950 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd8440acc50 0x7fd8440ad0d0 secure :-1 s=READY pgs=3088 cs=0 l=1 rev1=1 crypto rx=0x7fd8440c93f0 tx=0x7fd82c00e9a0 comp rx=0 tx=0).stop 2023-10-19T14:23:04.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:04.351+0000 7fd84ab5e640 1 -- 172.21.15.57:0/46508950 shutdown_connections 2023-10-19T14:23:04.355 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:04.351+0000 7fd84ab5e640 1 --2- 172.21.15.57:0/46508950 >> [v2:172.21.15.179:6832/892689897,v1:172.21.15.179:6833/892689897] conn(0x7fd82406af60 0x7fd82406d420 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:23:04.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:04.351+0000 7fd84ab5e640 1 --2- 172.21.15.57:0/46508950 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd8440acc50 0x7fd8440ad0d0 unknown :-1 s=CLOSED pgs=3088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:23:04.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:04.351+0000 7fd84ab5e640 1 --2- 172.21.15.57:0/46508950 >> [v2:172.21.15.179:3301/0,v1:172.21.15.179:6790/0] conn(0x7fd844155b80 0x7fd844156000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:23:04.356 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:04.351+0000 7fd84ab5e640 1 --2- 172.21.15.57:0/46508950 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd8300047c0 0x7fd844152970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-19T14:23:04.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:04.351+0000 7fd84ab5e640 1 -- 172.21.15.57:0/46508950 >> 172.21.15.57:0/46508950 conn(0x7fd844069810 msgr2=0x7fd844159720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-19T14:23:04.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:04.351+0000 7fd84ab5e640 1 -- 172.21.15.57:0/46508950 shutdown_connections 2023-10-19T14:23:04.357 INFO:teuthology.orchestra.run.smithi057.stderr:2023-10-19T14:23:04.351+0000 7fd84ab5e640 1 -- 172.21.15.57:0/46508950 wait complete. 2023-10-19T14:23:04.365 DEBUG:tasks.cephfs.filesystem:already dead... 2023-10-19T14:23:04.366 INFO:teuthology.misc:Shutting down mds daemons... 2023-10-19T14:23:04.367 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-19T14:23:04.367 INFO:teuthology.orchestra.run:waiting for 300 2023-10-19T14:23:04.378 INFO:tasks.ceph.mds.a:Stopped 2023-10-19T14:23:04.379 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-19T14:23:04.380 INFO:teuthology.orchestra.run:waiting for 300 2023-10-19T14:23:04.440 INFO:tasks.ceph.mds.c:Stopped 2023-10-19T14:23:04.441 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-19T14:23:04.441 INFO:teuthology.orchestra.run:waiting for 300 2023-10-19T14:23:04.464 INFO:tasks.ceph.mds.b:Stopped 2023-10-19T14:23:04.466 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-19T14:23:04.467 INFO:teuthology.orchestra.run:waiting for 300 2023-10-19T14:23:04.516 INFO:tasks.ceph.mds.d:Stopped 2023-10-19T14:23:04.517 INFO:teuthology.misc:Shutting down osd daemons... 2023-10-19T14:23:04.517 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-10-19T14:23:04.518 INFO:teuthology.orchestra.run:waiting for 300 2023-10-19T14:23:04.547 INFO:tasks.ceph.osd.0:Stopped 2023-10-19T14:23:04.547 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-10-19T14:23:04.548 INFO:teuthology.orchestra.run:waiting for 300 2023-10-19T14:23:04.621 INFO:tasks.ceph.osd.1:Stopped 2023-10-19T14:23:04.622 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-10-19T14:23:04.622 INFO:teuthology.orchestra.run:waiting for 300 2023-10-19T14:23:04.695 INFO:tasks.ceph.osd.2:Stopped 2023-10-19T14:23:04.696 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-10-19T14:23:04.696 INFO:teuthology.orchestra.run:waiting for 300 2023-10-19T14:23:04.771 INFO:tasks.ceph.osd.3:Stopped 2023-10-19T14:23:04.772 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-10-19T14:23:04.772 INFO:teuthology.orchestra.run:waiting for 300 2023-10-19T14:23:04.802 INFO:tasks.ceph.osd.4:Stopped 2023-10-19T14:23:04.803 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2023-10-19T14:23:04.804 INFO:teuthology.orchestra.run:waiting for 300 2023-10-19T14:23:04.871 INFO:tasks.ceph.osd.5:Stopped 2023-10-19T14:23:04.872 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2023-10-19T14:23:04.872 INFO:teuthology.orchestra.run:waiting for 300 2023-10-19T14:23:04.945 INFO:tasks.ceph.osd.6:Stopped 2023-10-19T14:23:04.945 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2023-10-19T14:23:04.946 INFO:teuthology.orchestra.run:waiting for 300 2023-10-19T14:23:05.020 INFO:tasks.ceph.osd.7:Stopped 2023-10-19T14:23:05.021 INFO:teuthology.misc:Shutting down mgr daemons... 2023-10-19T14:23:05.022 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2023-10-19T14:23:05.023 INFO:teuthology.orchestra.run:waiting for 300 2023-10-19T14:23:05.050 INFO:tasks.ceph.mgr.y:Stopped 2023-10-19T14:23:05.051 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-10-19T14:23:05.051 INFO:teuthology.orchestra.run:waiting for 300 2023-10-19T14:23:05.093 INFO:tasks.ceph.mgr.x:Stopped 2023-10-19T14:23:05.094 INFO:teuthology.misc:Shutting down mon daemons... 2023-10-19T14:23:05.094 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-10-19T14:23:05.095 INFO:teuthology.orchestra.run:waiting for 300 2023-10-19T14:23:05.111 INFO:tasks.ceph.mon.a:Stopped 2023-10-19T14:23:05.111 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-10-19T14:23:05.112 INFO:teuthology.orchestra.run:waiting for 300 2023-10-19T14:23:05.148 INFO:tasks.ceph.mon.b:Stopped 2023-10-19T14:23:05.149 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-10-19T14:23:05.149 INFO:teuthology.orchestra.run:waiting for 300 2023-10-19T14:23:05.202 INFO:tasks.ceph.mon.c:Stopped 2023-10-19T14:23:05.203 INFO:tasks.ceph:Checking cluster log for badness... 2023-10-19T14:23:05.203 DEBUG:teuthology.orchestra.run.smithi057:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2023-10-19T14:23:05.242 INFO:teuthology.orchestra.run.smithi057.stdout:2023-10-19T12:46:58.974806+0000 mon.a (mon.0) 455 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2023-10-19T14:23:05.242 WARNING:tasks.ceph:Found errors (ERR|WRN|SEC) in cluster log 2023-10-19T14:23:05.243 DEBUG:teuthology.orchestra.run.smithi057:> sudo egrep '\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2023-10-19T14:23:05.318 DEBUG:teuthology.orchestra.run.smithi057:> sudo egrep '\[ERR\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2023-10-19T14:23:05.393 DEBUG:teuthology.orchestra.run.smithi057:> sudo egrep '\[WRN\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2023-10-19T14:23:05.468 INFO:teuthology.orchestra.run.smithi057.stdout:2023-10-19T12:46:58.974806+0000 mon.a (mon.0) 455 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2023-10-19T14:23:05.469 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi057.front.sepia.ceph.com 2023-10-19T14:23:05.469 DEBUG:teuthology.orchestra.run.smithi057:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-10-19T14:23:05.716 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi057.front.sepia.ceph.com 2023-10-19T14:23:05.716 DEBUG:teuthology.orchestra.run.smithi057:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-10-19T14:23:05.910 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi057.front.sepia.ceph.com 2023-10-19T14:23:05.911 DEBUG:teuthology.orchestra.run.smithi057:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-10-19T14:23:05.985 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi057.front.sepia.ceph.com 2023-10-19T14:23:05.986 DEBUG:teuthology.orchestra.run.smithi057:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-10-19T14:23:06.068 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi179.front.sepia.ceph.com 2023-10-19T14:23:06.068 DEBUG:teuthology.orchestra.run.smithi179:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2023-10-19T14:23:06.262 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi179.front.sepia.ceph.com 2023-10-19T14:23:06.262 DEBUG:teuthology.orchestra.run.smithi179:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2023-10-19T14:23:06.406 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi179.front.sepia.ceph.com 2023-10-19T14:23:06.406 DEBUG:teuthology.orchestra.run.smithi179:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2023-10-19T14:23:06.462 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi179.front.sepia.ceph.com 2023-10-19T14:23:06.462 DEBUG:teuthology.orchestra.run.smithi179:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2023-10-19T14:23:06.521 INFO:tasks.ceph:Archiving mon data... 2023-10-19T14:23:06.522 DEBUG:teuthology.misc:Transferring archived files from smithi057:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/pdonnell-2023-10-19_12:04:12-fs-wip-batrick-testing-20231016.203825-distro-default-smithi/7432290/data/mon.a.tgz 2023-10-19T14:23:06.523 DEBUG:teuthology.orchestra.run.smithi057:> mktemp 2023-10-19T14:23:06.539 INFO:teuthology.orchestra.run.smithi057.stdout:/tmp/tmp.O4ZkYrIleZ 2023-10-19T14:23:06.539 DEBUG:teuthology.orchestra.run.smithi057:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.O4ZkYrIleZ 2023-10-19T14:23:06.740 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:23:06.742 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:23:07.222 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod 0666 /tmp/tmp.O4ZkYrIleZ 2023-10-19T14:23:07.311 DEBUG:teuthology.orchestra.remote:smithi057:/tmp/tmp.O4ZkYrIleZ is 4MB 2023-10-19T14:23:07.477 DEBUG:teuthology.orchestra.run.smithi057:> rm -fr /tmp/tmp.O4ZkYrIleZ 2023-10-19T14:23:07.493 DEBUG:teuthology.misc:Transferring archived files from smithi179:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/pdonnell-2023-10-19_12:04:12-fs-wip-batrick-testing-20231016.203825-distro-default-smithi/7432290/data/mon.b.tgz 2023-10-19T14:23:07.494 DEBUG:teuthology.orchestra.run.smithi179:> mktemp 2023-10-19T14:23:07.509 INFO:teuthology.orchestra.run.smithi179.stdout:/tmp/tmp.InKJhV9vcD 2023-10-19T14:23:07.510 DEBUG:teuthology.orchestra.run.smithi179:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.InKJhV9vcD 2023-10-19T14:23:08.144 DEBUG:teuthology.orchestra.run.smithi179:> sudo chmod 0666 /tmp/tmp.InKJhV9vcD 2023-10-19T14:23:08.229 DEBUG:teuthology.orchestra.remote:smithi179:/tmp/tmp.InKJhV9vcD is 4MB 2023-10-19T14:23:08.387 DEBUG:teuthology.orchestra.run.smithi179:> rm -fr /tmp/tmp.InKJhV9vcD 2023-10-19T14:23:08.402 DEBUG:teuthology.misc:Transferring archived files from smithi179:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/pdonnell-2023-10-19_12:04:12-fs-wip-batrick-testing-20231016.203825-distro-default-smithi/7432290/data/mon.c.tgz 2023-10-19T14:23:08.402 DEBUG:teuthology.orchestra.run.smithi179:> mktemp 2023-10-19T14:23:08.456 INFO:teuthology.orchestra.run.smithi179.stdout:/tmp/tmp.d4X8PrIXnI 2023-10-19T14:23:08.456 DEBUG:teuthology.orchestra.run.smithi179:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.d4X8PrIXnI 2023-10-19T14:23:09.081 DEBUG:teuthology.orchestra.run.smithi179:> sudo chmod 0666 /tmp/tmp.d4X8PrIXnI 2023-10-19T14:23:09.165 DEBUG:teuthology.orchestra.remote:smithi179:/tmp/tmp.d4X8PrIXnI is 4MB 2023-10-19T14:23:09.317 DEBUG:teuthology.orchestra.run.smithi179:> rm -fr /tmp/tmp.d4X8PrIXnI 2023-10-19T14:23:09.333 INFO:tasks.ceph:Cleaning ceph cluster... 2023-10-19T14:23:09.333 DEBUG:teuthology.orchestra.run.smithi057:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-10-19T14:23:09.336 DEBUG:teuthology.orchestra.run.smithi179:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-10-19T14:23:09.399 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-10-19T14:23:09.399 DEBUG:teuthology.orchestra.run.smithi057:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-19T14:23:09.411 DEBUG:teuthology.orchestra.run.smithi179:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-19T14:23:09.446 INFO:teuthology.orchestra.run.smithi057.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-19T14:23:09.468 INFO:teuthology.orchestra.run.smithi179.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-19T14:23:09.472 INFO:tasks.ceph:Archiving crash dumps... 2023-10-19T14:23:09.482 DEBUG:teuthology.misc:Transferring archived files from smithi057:/var/lib/ceph/crash to /home/teuthworker/archive/pdonnell-2023-10-19_12:04:12-fs-wip-batrick-testing-20231016.203825-distro-default-smithi/7432290/remote/smithi057/crash 2023-10-19T14:23:09.485 DEBUG:teuthology.orchestra.run.smithi057:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-19T14:23:09.530 DEBUG:teuthology.misc:Transferring archived files from smithi179:/var/lib/ceph/crash to /home/teuthworker/archive/pdonnell-2023-10-19_12:04:12-fs-wip-batrick-testing-20231016.203825-distro-default-smithi/7432290/remote/smithi179/crash 2023-10-19T14:23:09.533 DEBUG:teuthology.orchestra.run.smithi179:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-19T14:23:09.560 INFO:tasks.ceph:Shutting down logrotate 2023-10-19T14:23:09.560 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:23:09.578 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-19T14:23:09.672 DEBUG:teuthology.orchestra.run.smithi057:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-10-19T14:23:09.704 DEBUG:teuthology.orchestra.run.smithi179:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-10-19T14:23:09.732 INFO:tasks.ceph:Compressing logs... 2023-10-19T14:23:09.732 DEBUG:teuthology.orchestra.run.smithi057:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-10-19T14:23:09.749 DEBUG:teuthology.orchestra.run.smithi179:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-10-19T14:23:09.778 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117969.log 2023-10-19T14:23:09.779 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97333.log 2023-10-19T14:23:09.779 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.117969.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99867.log 2023-10-19T14:23:09.779 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose 87.3% -- -- replaced with /var/log/ceph/ceph-client.admin.117969.log.gz /var/log/ceph/ceph-client.admin.166589.log 2023-10-19T14:23:09.779 INFO:teuthology.orchestra.run.smithi057.stderr: 2023-10-19T14:23:09.816 INFO:teuthology.orchestra.run.smithi179.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2023-10-19T14:23:09.816 INFO:teuthology.orchestra.run.smithi179.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88067.log 2023-10-19T14:23:09.816 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88187.log 2023-10-19T14:23:09.817 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ceph-client.admin.88067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88027.log 2023-10-19T14:23:09.817 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ceph-client.admin.88187.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.88067.log.gz 2023-10-19T14:23:09.817 INFO:teuthology.orchestra.run.smithi179.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-10-19T14:23:09.817 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ceph-client.admin.88027.log: gzip 86.8% -5 -- replaced with /var/log/ceph/ceph-client.admin.88187.log.gz --verbose 2023-10-19T14:23:09.817 INFO:teuthology.orchestra.run.smithi179.stderr: -- /var/log/ceph/ceph-mds.b.log 2023-10-19T14:23:09.818 INFO:teuthology.orchestra.run.smithi179.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.88027.log.gz 2023-10-19T14:23:09.818 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.3.94778.log 2023-10-19T14:23:09.818 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2023-10-19T14:23:09.818 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ceph-client.3.94778.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88227.log 2023-10-19T14:23:09.818 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88147.log 2023-10-19T14:23:09.819 INFO:teuthology.orchestra.run.smithi179.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.3.94778.log.gz 2023-10-19T14:23:09.819 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ceph-client.admin.88227.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2023-10-19T14:23:09.819 INFO:teuthology.orchestra.run.smithi179.stderr:gzip 86.9% -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2023-10-19T14:23:09.819 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ceph-client.admin.88147.log: -- replaced with /var/log/ceph/ceph-client.admin.88227.log.gz 2023-10-19T14:23:09.819 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.94341.log 2023-10-19T14:23:09.820 INFO:teuthology.orchestra.run.smithi179.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.88147.log.gz 2023-10-19T14:23:09.820 INFO:teuthology.orchestra.run.smithi179.stderr:gzip -5/var/log/ceph/ceph-osd.5.log: --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-10-19T14:23:09.820 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ceph-client.1.94341.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87947.log 2023-10-19T14:23:09.820 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ceph-mgr.x.log: 90.0% -- replaced with /var/log/ceph/ceph-client.1.94341.log.gz 2023-10-19T14:23:09.821 INFO:teuthology.orchestra.run.smithi179.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-10-19T14:23:09.821 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ceph-client.admin.87947.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.87947.log.gz 2023-10-19T14:23:09.821 INFO:teuthology.orchestra.run.smithi179.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87987.log 2023-10-19T14:23:09.829 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ceph.log: 93.7% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-10-19T14:23:09.830 INFO:teuthology.orchestra.run.smithi179.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88107.log 2023-10-19T14:23:09.830 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ceph-client.admin.87987.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.87987.log.gz 2023-10-19T14:23:09.830 INFO:teuthology.orchestra.run.smithi179.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2023-10-19T14:23:09.830 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ceph-client.admin.88107.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.88107.log.gz 2023-10-19T14:23:09.831 INFO:teuthology.orchestra.run.smithi179.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2023-10-19T14:23:09.831 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ceph-mon.b.log: 94.2% -- replaced with /var/log/ceph/ceph.log.gz 2023-10-19T14:23:09.831 INFO:teuthology.orchestra.run.smithi179.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2023-10-19T14:23:09.835 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ceph-mds.d.log: 93.7% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2023-10-19T14:23:10.193 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.97333.log: /var/log/ceph/ceph-client.admin.99867.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142702.log 2023-10-19T14:23:10.193 INFO:teuthology.orchestra.run.smithi057.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.97333.log.gz 2023-10-19T14:23:10.194 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.166589.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.99867.log.gz 2023-10-19T14:23:10.194 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162598.log 2023-10-19T14:23:10.194 INFO:teuthology.orchestra.run.smithi057.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.166589.log.gz 2023-10-19T14:23:10.194 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.142702.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134654.log 2023-10-19T14:23:10.194 INFO:teuthology.orchestra.run.smithi057.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.142702.log.gz 2023-10-19T14:23:10.195 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.162598.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113004.log 2023-10-19T14:23:10.195 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.134654.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.162598.log.gz 2023-10-19T14:23:10.195 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103266.log 2023-10-19T14:23:10.195 INFO:teuthology.orchestra.run.smithi057.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.134654.log.gz 2023-10-19T14:23:10.195 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.113004.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143202.log 2023-10-19T14:23:10.195 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.113004.log.gz 2023-10-19T14:23:10.196 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.103266.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123270.log 2023-10-19T14:23:10.196 INFO:teuthology.orchestra.run.smithi057.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.103266.log.gz 2023-10-19T14:23:10.196 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.143202.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131039.log 2023-10-19T14:23:10.196 INFO:teuthology.orchestra.run.smithi057.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.143202.log.gz 2023-10-19T14:23:10.196 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.167093.log 2023-10-19T14:23:10.197 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.123270.log: gzip/var/log/ceph/ceph-client.admin.131039.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.140467.log 2023-10-19T14:23:10.197 INFO:teuthology.orchestra.run.smithi057.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.123270.log.gz 2023-10-19T14:23:10.197 INFO:teuthology.orchestra.run.smithi057.stderr: 87.3%/var/log/ceph/ceph-client.admin.167093.log: -- replaced with /var/log/ceph/ceph-client.admin.131039.log.gz 2023-10-19T14:23:10.197 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113992.log 2023-10-19T14:23:10.197 INFO:teuthology.orchestra.run.smithi057.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.167093.log.gz 2023-10-19T14:23:10.197 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.140467.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165828.log 2023-10-19T14:23:10.198 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.113992.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.140467.log.gz 2023-10-19T14:23:10.198 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156749.log 2023-10-19T14:23:10.198 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.113992.log.gz 2023-10-19T14:23:10.198 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.165828.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127596.log 2023-10-19T14:23:10.198 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.156749.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119179.log 2023-10-19T14:23:10.199 INFO:teuthology.orchestra.run.smithi057.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.165828.log.gz 2023-10-19T14:23:10.199 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.127596.log: gzip 87.6% -- replaced with /var/log/ceph/ceph-client.admin.156749.log.gz 2023-10-19T14:23:10.199 INFO:teuthology.orchestra.run.smithi057.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.97675.log 2023-10-19T14:23:10.199 INFO:teuthology.orchestra.run.smithi057.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.127596.log.gz 2023-10-19T14:23:10.199 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.119179.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126621.log 2023-10-19T14:23:10.200 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.97675.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.119179.log.gz 2023-10-19T14:23:10.200 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140238.log 2023-10-19T14:23:10.200 INFO:teuthology.orchestra.run.smithi057.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.97675.log.gz 2023-10-19T14:23:10.200 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.126621.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124418.log 2023-10-19T14:23:10.200 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.126621.log.gz 2023-10-19T14:23:10.200 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.140238.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88667.log 2023-10-19T14:23:10.201 INFO:teuthology.orchestra.run.smithi057.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.140238.log.gz 2023-10-19T14:23:10.201 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.124418.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120230.log 2023-10-19T14:23:10.201 INFO:teuthology.orchestra.run.smithi057.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.124418.log.gz 2023-10-19T14:23:10.201 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.88667.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134162.log 2023-10-19T14:23:10.201 INFO:teuthology.orchestra.run.smithi057.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.88667.log.gz 2023-10-19T14:23:10.202 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.120230.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143067.log 2023-10-19T14:23:10.202 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.120230.log.gz 2023-10-19T14:23:10.202 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.134162.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120824.log 2023-10-19T14:23:10.202 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.134162.log.gz 2023-10-19T14:23:10.202 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.143067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129538.log 2023-10-19T14:23:10.202 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.120824.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.143067.log.gz 2023-10-19T14:23:10.203 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-10-19T14:23:10.203 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.120824.log.gz 2023-10-19T14:23:10.203 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.129538.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100371.log 2023-10-19T14:23:10.203 INFO:teuthology.orchestra.run.smithi057.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.129538.log.gz 2023-10-19T14:23:10.203 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88201.log 2023-10-19T14:23:10.204 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.100371.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160619.log 2023-10-19T14:23:10.204 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.100371.log.gz 2023-10-19T14:23:10.204 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.88201.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148000.log 2023-10-19T14:23:10.204 INFO:teuthology.orchestra.run.smithi057.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88201.log.gz 2023-10-19T14:23:10.204 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.160619.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97913.log 2023-10-19T14:23:10.204 INFO:teuthology.orchestra.run.smithi057.stderr: 86.8%/var/log/ceph/ceph-client.admin.148000.log: -- replaced with /var/log/ceph/ceph-client.admin.160619.log.gz 2023-10-19T14:23:10.205 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102132.log 2023-10-19T14:23:10.205 INFO:teuthology.orchestra.run.smithi057.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.148000.log.gz 2023-10-19T14:23:10.205 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.97913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155226.log 2023-10-19T14:23:10.205 INFO:teuthology.orchestra.run.smithi057.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.97913.log.gz 2023-10-19T14:23:10.205 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.102132.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97339.log 2023-10-19T14:23:10.206 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.155226.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.102132.log.gz 2023-10-19T14:23:10.206 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157760.log 2023-10-19T14:23:10.206 INFO:teuthology.orchestra.run.smithi057.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.155226.log.gz 2023-10-19T14:23:10.206 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.97339.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159010.log 2023-10-19T14:23:10.206 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.157760.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.97339.log.gz 2023-10-19T14:23:10.206 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101406.log 2023-10-19T14:23:10.207 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4%/var/log/ceph/ceph-client.admin.159010.log: -- replaced with /var/log/ceph/ceph-client.admin.157760.log.gz 2023-10-19T14:23:10.207 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156893.log 2023-10-19T14:23:10.207 INFO:teuthology.orchestra.run.smithi057.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.159010.log.gz 2023-10-19T14:23:10.207 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.101406.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163120.log 2023-10-19T14:23:10.207 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.101406.log.gz 2023-10-19T14:23:10.208 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.156893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.164782.log 2023-10-19T14:23:10.208 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.163120.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.156893.log.gz 2023-10-19T14:23:10.208 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166925.log 2023-10-19T14:23:10.208 INFO:teuthology.orchestra.run.smithi057.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.163120.log.gz 2023-10-19T14:23:10.208 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.0.164782.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125874.log 2023-10-19T14:23:10.208 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.166925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110842.log 2023-10-19T14:23:10.209 INFO:teuthology.orchestra.run.smithi057.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.166925.log.gz 2023-10-19T14:23:10.209 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.125874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102248.log 2023-10-19T14:23:10.209 INFO:teuthology.orchestra.run.smithi057.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.125874.log.gz 2023-10-19T14:23:10.209 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166631.log 2023-10-19T14:23:10.209 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.110842.log: /var/log/ceph/ceph-client.admin.102248.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.110842.log.gz 2023-10-19T14:23:10.210 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120375.log 2023-10-19T14:23:10.210 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.102248.log.gz 2023-10-19T14:23:10.210 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139612.log 2023-10-19T14:23:10.210 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.166631.log: 92.1%/var/log/ceph/ceph-client.admin.120375.log: -- replaced with /var/log/ceph/ceph-client.0.164782.log.gz 2023-10-19T14:23:10.210 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166883.log 2023-10-19T14:23:10.211 INFO:teuthology.orchestra.run.smithi057.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.166631.log.gz 2023-10-19T14:23:10.211 INFO:teuthology.orchestra.run.smithi057.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.120375.log.gz 2023-10-19T14:23:10.211 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.139612.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104650.log 2023-10-19T14:23:10.211 INFO:teuthology.orchestra.run.smithi057.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.139612.log.gz 2023-10-19T14:23:10.211 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.166883.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151783.log 2023-10-19T14:23:10.211 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.104650.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.166883.log.gz 2023-10-19T14:23:10.212 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161268.log 2023-10-19T14:23:10.212 INFO:teuthology.orchestra.run.smithi057.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.104650.log.gz 2023-10-19T14:23:10.212 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.151783.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151502.log 2023-10-19T14:23:10.212 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.151783.log.gz 2023-10-19T14:23:10.212 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.161268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154694.log 2023-10-19T14:23:10.213 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.151502.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.161268.log.gz 2023-10-19T14:23:10.213 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113174.log 2023-10-19T14:23:10.213 INFO:teuthology.orchestra.run.smithi057.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.151502.log.gz 2023-10-19T14:23:10.213 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.154694.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.167009.log 2023-10-19T14:23:10.213 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.154694.log.gz 2023-10-19T14:23:10.213 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.113174.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152306.log 2023-10-19T14:23:10.214 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.167009.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.113174.log.gz 2023-10-19T14:23:10.214 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148359.log 2023-10-19T14:23:10.214 INFO:teuthology.orchestra.run.smithi057.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.167009.log.gz 2023-10-19T14:23:10.214 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.152306.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98291.log 2023-10-19T14:23:10.214 INFO:teuthology.orchestra.run.smithi057.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.152306.log.gz 2023-10-19T14:23:10.215 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.148359.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146767.log 2023-10-19T14:23:10.215 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.98291.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.148359.log.gz 2023-10-19T14:23:10.215 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112021.log 2023-10-19T14:23:10.215 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.98291.log.gz 2023-10-19T14:23:10.215 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.146767.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121069.log 2023-10-19T14:23:10.216 INFO:teuthology.orchestra.run.smithi057.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.146767.log.gz 2023-10-19T14:23:10.216 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.112021.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106590.log 2023-10-19T14:23:10.216 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.121069.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.112021.log.gz 2023-10-19T14:23:10.216 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97003.log 2023-10-19T14:23:10.216 INFO:teuthology.orchestra.run.smithi057.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.121069.log.gz 2023-10-19T14:23:10.216 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.106590.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166253.log 2023-10-19T14:23:10.217 INFO:teuthology.orchestra.run.smithi057.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.106590.log.gz 2023-10-19T14:23:10.217 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.97003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102762.log 2023-10-19T14:23:10.217 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.166253.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.97003.log.gz 2023-10-19T14:23:10.217 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135164.log 2023-10-19T14:23:10.217 INFO:teuthology.orchestra.run.smithi057.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.166253.log.gz 2023-10-19T14:23:10.217 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.102762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114201.log 2023-10-19T14:23:10.218 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.135164.log: 87.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.102762.log.gz 2023-10-19T14:23:10.218 INFO:teuthology.orchestra.run.smithi057.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.105280.log 2023-10-19T14:23:10.218 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.135164.log.gz 2023-10-19T14:23:10.218 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.114201.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163204.log 2023-10-19T14:23:10.218 INFO:teuthology.orchestra.run.smithi057.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.114201.log.gz 2023-10-19T14:23:10.219 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.105280.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100581.log 2023-10-19T14:23:10.219 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.163204.log: gzip -5 --verbose -- 87.5% /var/log/ceph/ceph-client.admin.103434.log 2023-10-19T14:23:10.219 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/ceph-client.admin.105280.log.gz 2023-10-19T14:23:10.219 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.100581.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151544.log 2023-10-19T14:23:10.219 INFO:teuthology.orchestra.run.smithi057.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.163204.log.gz 2023-10-19T14:23:10.220 INFO:teuthology.orchestra.run.smithi057.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.100581.log.gz 2023-10-19T14:23:10.220 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.103434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151119.log 2023-10-19T14:23:10.220 INFO:teuthology.orchestra.run.smithi057.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.103434.log.gz 2023-10-19T14:23:10.220 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.151544.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149485.log 2023-10-19T14:23:10.220 INFO:teuthology.orchestra.run.smithi057.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.151544.log.gz 2023-10-19T14:23:10.220 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.151119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120477.log 2023-10-19T14:23:10.221 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.151119.log.gz 2023-10-19T14:23:10.221 INFO:teuthology.orchestra.run.smithi057.stderr:gzip/var/log/ceph/ceph-client.admin.149485.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.118571.log 2023-10-19T14:23:10.221 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.120477.log: gzip 87.1% -5 -- replaced with /var/log/ceph/ceph-client.admin.149485.log.gz 2023-10-19T14:23:10.221 INFO:teuthology.orchestra.run.smithi057.stderr: --verbose -- /var/log/ceph/ceph-client.admin.156791.log 2023-10-19T14:23:10.221 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5%/var/log/ceph/ceph-client.admin.118571.log: -- replaced with /var/log/ceph/ceph-client.admin.120477.log.gz 2023-10-19T14:23:10.221 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142386.log 2023-10-19T14:23:10.222 INFO:teuthology.orchestra.run.smithi057.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.118571.log.gz 2023-10-19T14:23:10.222 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.156791.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120062.log 2023-10-19T14:23:10.222 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.142386.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161520.log 2023-10-19T14:23:10.222 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.156791.log.gz 2023-10-19T14:23:10.222 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.120062.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.142386.log.gz 2023-10-19T14:23:10.223 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156100.log 2023-10-19T14:23:10.223 INFO:teuthology.orchestra.run.smithi057.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.120062.log.gz 2023-10-19T14:23:10.223 INFO:teuthology.orchestra.run.smithi057.stderr:gzip/var/log/ceph/ceph-client.admin.161520.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.120561.log 2023-10-19T14:23:10.223 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.156100.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.161520.log.gz 2023-10-19T14:23:10.223 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135122.log 2023-10-19T14:23:10.223 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.156100.log.gz 2023-10-19T14:23:10.224 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.120561.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105172.log 2023-10-19T14:23:10.224 INFO:teuthology.orchestra.run.smithi057.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.120561.log.gz 2023-10-19T14:23:10.224 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.135122.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120272.log 2023-10-19T14:23:10.224 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.105172.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.135122.log.gz 2023-10-19T14:23:10.224 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152954.log 2023-10-19T14:23:10.225 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.105172.log.gz 2023-10-19T14:23:10.225 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.120272.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102888.log 2023-10-19T14:23:10.225 INFO:teuthology.orchestra.run.smithi057.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.120272.log.gz 2023-10-19T14:23:10.225 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.152954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148751.log 2023-10-19T14:23:10.225 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.152954.log.gz 2023-10-19T14:23:10.225 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.102888.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117358.log 2023-10-19T14:23:10.226 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.102888.log.gz 2023-10-19T14:23:10.226 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.148751.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140070.log 2023-10-19T14:23:10.226 INFO:teuthology.orchestra.run.smithi057.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.148751.log.gz 2023-10-19T14:23:10.226 INFO:teuthology.orchestra.run.smithi057.stderr:gzip/var/log/ceph/ceph-client.admin.117358.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.161478.log 2023-10-19T14:23:10.226 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.140070.log: 87.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.117358.log.gz 2023-10-19T14:23:10.227 INFO:teuthology.orchestra.run.smithi057.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.164980.log 2023-10-19T14:23:10.227 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.161478.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.140070.log.gz 2023-10-19T14:23:10.227 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134899.log 2023-10-19T14:23:10.227 INFO:teuthology.orchestra.run.smithi057.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.161478.log.gz 2023-10-19T14:23:10.227 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.164980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100707.log 2023-10-19T14:23:10.228 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.134899.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144234.log 2023-10-19T14:23:10.228 INFO:teuthology.orchestra.run.smithi057.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.164980.log.gz 2023-10-19T14:23:10.228 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.100707.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.134899.log.gz 2023-10-19T14:23:10.228 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146996.log 2023-10-19T14:23:10.228 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.100707.log.gz 2023-10-19T14:23:10.228 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.144234.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118529.log 2023-10-19T14:23:10.229 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.146996.log: 87.4%gzip -- replaced with /var/log/ceph/ceph-client.admin.144234.log.gz 2023-10-19T14:23:10.229 INFO:teuthology.orchestra.run.smithi057.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.143496.log 2023-10-19T14:23:10.229 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.118529.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.146996.log.gz 2023-10-19T14:23:10.229 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88660.log 2023-10-19T14:23:10.229 INFO:teuthology.orchestra.run.smithi057.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.118529.log.gz 2023-10-19T14:23:10.230 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.143496.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124374.log 2023-10-19T14:23:10.230 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.88660.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.143496.log.gz 2023-10-19T14:23:10.230 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146377.log 2023-10-19T14:23:10.230 INFO:teuthology.orchestra.run.smithi057.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.88660.log.gz 2023-10-19T14:23:10.230 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.124374.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88500.log 2023-10-19T14:23:10.231 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.124374.log.gz 2023-10-19T14:23:10.231 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.146377.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105922.log 2023-10-19T14:23:10.231 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.88500.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.146377.log.gz 2023-10-19T14:23:10.231 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136288.log 2023-10-19T14:23:10.231 INFO:teuthology.orchestra.run.smithi057.stderr: 85.4% -- replaced with /var/log/ceph/ceph-client.admin.88500.log.gz 2023-10-19T14:23:10.231 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.105922.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155142.log 2023-10-19T14:23:10.232 INFO:teuthology.orchestra.run.smithi057.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.105922.log.gz 2023-10-19T14:23:10.232 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.122054.log 2023-10-19T14:23:10.232 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.136288.log: /var/log/ceph/ceph-client.admin.155142.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154862.log 2023-10-19T14:23:10.232 INFO:teuthology.orchestra.run.smithi057.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.136288.log.gz 2023-10-19T14:23:10.232 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.0.122054.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.155142.log.gz 2023-10-19T14:23:10.232 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120417.log 2023-10-19T14:23:10.233 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.154862.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117045.log 2023-10-19T14:23:10.233 INFO:teuthology.orchestra.run.smithi057.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.154862.log.gz 2023-10-19T14:23:10.233 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.120417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95911.log 2023-10-19T14:23:10.233 INFO:teuthology.orchestra.run.smithi057.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.0.122054.log.gz 2023-10-19T14:23:10.233 INFO:teuthology.orchestra.run.smithi057.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.120417.log.gz 2023-10-19T14:23:10.234 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.117045.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97672.log 2023-10-19T14:23:10.234 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.117045.log.gz 2023-10-19T14:23:10.234 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.95911.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101796.log 2023-10-19T14:23:10.234 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.97672.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.95911.log.gz 2023-10-19T14:23:10.234 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121714.log 2023-10-19T14:23:10.234 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.97672.log.gz 2023-10-19T14:23:10.235 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.101796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147706.log 2023-10-19T14:23:10.235 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.121714.log: gzip 87.3% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.101796.log.gz -- 2023-10-19T14:23:10.235 INFO:teuthology.orchestra.run.smithi057.stderr: /var/log/ceph/ceph-client.admin.159528.log 2023-10-19T14:23:10.235 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.147706.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152138.log 2023-10-19T14:23:10.235 INFO:teuthology.orchestra.run.smithi057.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.121714.log.gz 2023-10-19T14:23:10.236 INFO:teuthology.orchestra.run.smithi057.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.147706.log.gz 2023-10-19T14:23:10.236 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.159528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102374.log 2023-10-19T14:23:10.236 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.152138.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113216.log 2023-10-19T14:23:10.236 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.159528.log.gz 2023-10-19T14:23:10.236 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.152138.log.gz 2023-10-19T14:23:10.236 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.102374.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157230.log 2023-10-19T14:23:10.237 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5%/var/log/ceph/ceph-client.admin.113216.log: -- replaced with /var/log/ceph/ceph-client.admin.102374.log.gz 2023-10-19T14:23:10.237 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129776.log 2023-10-19T14:23:10.237 INFO:teuthology.orchestra.run.smithi057.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.113216.log.gz 2023-10-19T14:23:10.237 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.157230.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144106.log 2023-10-19T14:23:10.237 INFO:teuthology.orchestra.run.smithi057.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.157230.log.gz 2023-10-19T14:23:10.238 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.129776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159136.log 2023-10-19T14:23:10.238 INFO:teuthology.orchestra.run.smithi057.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.129776.log.gz 2023-10-19T14:23:10.238 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.144106.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131487.log 2023-10-19T14:23:10.238 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.159136.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.144106.log.gz 2023-10-19T14:23:10.238 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144533.log 2023-10-19T14:23:10.238 INFO:teuthology.orchestra.run.smithi057.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.159136.log.gz 2023-10-19T14:23:10.239 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.131487.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94571.log 2023-10-19T14:23:10.239 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.131487.log.gz 2023-10-19T14:23:10.239 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.144533.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119730.log 2023-10-19T14:23:10.239 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.94571.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.144533.log.gz 2023-10-19T14:23:10.239 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102972.log 2023-10-19T14:23:10.240 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.119730.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.94571.log.gz 2023-10-19T14:23:10.240 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107572.log 2023-10-19T14:23:10.240 INFO:teuthology.orchestra.run.smithi057.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.119730.log.gz 2023-10-19T14:23:10.240 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.102972.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160058.log 2023-10-19T14:23:10.240 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.107572.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.102972.log.gz 2023-10-19T14:23:10.241 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133434.log 2023-10-19T14:23:10.241 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.160058.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.107572.log.gz 2023-10-19T14:23:10.241 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101670.log 2023-10-19T14:23:10.241 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.160058.log.gz 2023-10-19T14:23:10.241 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.133434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98165.log 2023-10-19T14:23:10.241 INFO:teuthology.orchestra.run.smithi057.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.133434.log.gz 2023-10-19T14:23:10.242 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.101670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129631.log 2023-10-19T14:23:10.242 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.98165.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.101670.log.gz 2023-10-19T14:23:10.242 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159912.log 2023-10-19T14:23:10.242 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.98165.log.gz 2023-10-19T14:23:10.242 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.129631.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101448.log 2023-10-19T14:23:10.243 INFO:teuthology.orchestra.run.smithi057.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.129631.log.gz 2023-10-19T14:23:10.243 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.159912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120645.log 2023-10-19T14:23:10.243 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.101448.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151639.log 2023-10-19T14:23:10.243 INFO:teuthology.orchestra.run.smithi057.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.159912.log.gz 2023-10-19T14:23:10.243 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.101448.log.gz 2023-10-19T14:23:10.243 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.120645.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153334.log 2023-10-19T14:23:10.244 INFO:teuthology.orchestra.run.smithi057.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.120645.log.gz 2023-10-19T14:23:10.244 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.151639.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117884.log 2023-10-19T14:23:10.244 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.153334.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.151639.log.gz 2023-10-19T14:23:10.244 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152646.log 2023-10-19T14:23:10.244 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.153334.log.gz 2023-10-19T14:23:10.245 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.117884.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129818.log 2023-10-19T14:23:10.245 INFO:teuthology.orchestra.run.smithi057.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.117884.log.gz 2023-10-19T14:23:10.245 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.152646.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135292.log 2023-10-19T14:23:10.245 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.129818.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.152646.log.gz 2023-10-19T14:23:10.245 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165996.log 2023-10-19T14:23:10.246 INFO:teuthology.orchestra.run.smithi057.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.129818.log.gz 2023-10-19T14:23:10.246 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.135292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160203.log 2023-10-19T14:23:10.246 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.135292.log.gz 2023-10-19T14:23:10.246 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.165996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140551.log 2023-10-19T14:23:10.246 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.160203.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.165996.log.gz 2023-10-19T14:23:10.246 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112826.log 2023-10-19T14:23:10.247 INFO:teuthology.orchestra.run.smithi057.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.160203.log.gz 2023-10-19T14:23:10.247 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.140551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95995.log 2023-10-19T14:23:10.247 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.140551.log.gz/var/log/ceph/ceph-client.admin.112826.log: 2023-10-19T14:23:10.247 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161604.log 2023-10-19T14:23:10.247 INFO:teuthology.orchestra.run.smithi057.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.112826.log.gz 2023-10-19T14:23:10.248 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.95995.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166421.log 2023-10-19T14:23:10.248 INFO:teuthology.orchestra.run.smithi057.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.95995.log.gz 2023-10-19T14:23:10.248 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.161604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155642.log 2023-10-19T14:23:10.248 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.166421.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.161604.log.gz 2023-10-19T14:23:10.248 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122361.log 2023-10-19T14:23:10.248 INFO:teuthology.orchestra.run.smithi057.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.166421.log.gz 2023-10-19T14:23:10.249 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.155642.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126027.log 2023-10-19T14:23:10.249 INFO:teuthology.orchestra.run.smithi057.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.155642.log.gz 2023-10-19T14:23:10.249 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.122361.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127214.log 2023-10-19T14:23:10.249 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.126027.log: 86.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.122361.log.gz 2023-10-19T14:23:10.249 INFO:teuthology.orchestra.run.smithi057.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.107614.log 2023-10-19T14:23:10.250 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.127214.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.126027.log.gz 2023-10-19T14:23:10.250 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112782.log 2023-10-19T14:23:10.250 INFO:teuthology.orchestra.run.smithi057.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.127214.log.gz 2023-10-19T14:23:10.250 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.107614.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162556.log 2023-10-19T14:23:10.250 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.112782.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.107614.log.gz 2023-10-19T14:23:10.250 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160703.log 2023-10-19T14:23:10.251 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.162556.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.112782.log.gz 2023-10-19T14:23:10.251 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126968.log 2023-10-19T14:23:10.251 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.162556.log.gz 2023-10-19T14:23:10.251 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110579.log 2023-10-19T14:23:10.251 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.160703.log: /var/log/ceph/ceph-client.admin.126968.log: gzip 86.2% -- replaced with /var/log/ceph/ceph-client.admin.160703.log.gz -5 2023-10-19T14:23:10.252 INFO:teuthology.orchestra.run.smithi057.stderr: --verbose -- /var/log/ceph/ceph-client.admin.121112.log 2023-10-19T14:23:10.252 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4%/var/log/ceph/ceph-client.admin.110579.log: -- replaced with /var/log/ceph/ceph-client.admin.126968.log.gz 2023-10-19T14:23:10.252 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156328.log 2023-10-19T14:23:10.252 INFO:teuthology.orchestra.run.smithi057.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.110579.log.gz 2023-10-19T14:23:10.252 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.121112.log: -- /var/log/ceph/ceph-client.admin.88665.log 2023-10-19T14:23:10.252 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.156328.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.121112.log.gz 2023-10-19T14:23:10.253 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157062.log 2023-10-19T14:23:10.253 INFO:teuthology.orchestra.run.smithi057.stderr: 87.1%/var/log/ceph/ceph-client.admin.88665.log: -- replaced with /var/log/ceph/ceph-client.admin.156328.log.gz 2023-10-19T14:23:10.253 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132806.log 2023-10-19T14:23:10.253 INFO:teuthology.orchestra.run.smithi057.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.88665.log.gz 2023-10-19T14:23:10.253 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.157062.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107740.log 2023-10-19T14:23:10.254 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.132806.log: 87.1%gzip -- replaced with /var/log/ceph/ceph-client.admin.157062.log.gz -5 2023-10-19T14:23:10.254 INFO:teuthology.orchestra.run.smithi057.stderr: --verbose -- /var/log/ceph/ceph-client.admin.161039.log 2023-10-19T14:23:10.254 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.107740.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.132806.log.gz 2023-10-19T14:23:10.254 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154544.log 2023-10-19T14:23:10.254 INFO:teuthology.orchestra.run.smithi057.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.107740.log.gz 2023-10-19T14:23:10.254 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.161039.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154026.log 2023-10-19T14:23:10.255 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.154544.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.161039.log.gz 2023-10-19T14:23:10.255 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116742.log 2023-10-19T14:23:10.255 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.154026.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.154544.log.gz 2023-10-19T14:23:10.255 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144449.log 2023-10-19T14:23:10.255 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.154026.log.gz 2023-10-19T14:23:10.256 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.116742.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112231.log 2023-10-19T14:23:10.256 INFO:teuthology.orchestra.run.smithi057.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.116742.log.gz 2023-10-19T14:23:10.256 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.144449.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133047.log 2023-10-19T14:23:10.256 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.112231.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.144449.log.gz 2023-10-19T14:23:10.256 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156268.log 2023-10-19T14:23:10.256 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.112231.log.gz 2023-10-19T14:23:10.257 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.133047.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132391.log 2023-10-19T14:23:10.257 INFO:teuthology.orchestra.run.smithi057.stderr: 83.8% -- replaced with /var/log/ceph/ceph-client.admin.133047.log.gz 2023-10-19T14:23:10.257 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.156268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88651.log 2023-10-19T14:23:10.257 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.156268.log.gz 2023-10-19T14:23:10.257 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5/var/log/ceph/ceph-client.admin.132391.log: --verbose -- /var/log/ceph/ceph-client.admin.99715.log 2023-10-19T14:23:10.258 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.88651.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132629.log 87.5% 2023-10-19T14:23:10.258 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/ceph-client.admin.132391.log.gz 2023-10-19T14:23:10.258 INFO:teuthology.orchestra.run.smithi057.stderr: 85.0% -- replaced with /var/log/ceph/ceph-client.admin.88651.log.gz 2023-10-19T14:23:10.258 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.99715.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155600.log 2023-10-19T14:23:10.258 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.99715.log.gz 2023-10-19T14:23:10.258 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.132629.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166673.log 2023-10-19T14:23:10.259 INFO:teuthology.orchestra.run.smithi057.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.132629.log.gz 2023-10-19T14:23:10.259 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.155600.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144026.log 2023-10-19T14:23:10.259 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.166673.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.155600.log.gz 2023-10-19T14:23:10.259 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142870.log 2023-10-19T14:23:10.259 INFO:teuthology.orchestra.run.smithi057.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.166673.log.gz 2023-10-19T14:23:10.260 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.144026.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130055.log 2023-10-19T14:23:10.260 INFO:teuthology.orchestra.run.smithi057.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.144026.log.gz 2023-10-19T14:23:10.260 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.142870.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96993.log 2023-10-19T14:23:10.260 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.130055.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.142870.log.gz 2023-10-19T14:23:10.260 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96095.log 2023-10-19T14:23:10.260 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.96993.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.130055.log.gz 2023-10-19T14:23:10.261 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145535.log 2023-10-19T14:23:10.261 INFO:teuthology.orchestra.run.smithi057.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.96993.log.gz 2023-10-19T14:23:10.261 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.96095.log: /var/log/ceph/ceph-client.admin.97670.log 2023-10-19T14:23:10.261 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.145535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122643.log 2023-10-19T14:23:10.261 INFO:teuthology.orchestra.run.smithi057.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.96095.log.gz 2023-10-19T14:23:10.262 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.97670.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.145535.log.gz 2023-10-19T14:23:10.262 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96991.log 2023-10-19T14:23:10.262 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.97670.log.gz 2023-10-19T14:23:10.262 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.122643.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123330.log 2023-10-19T14:23:10.262 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.96991.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.122643.log.gz 2023-10-19T14:23:10.263 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96635.log 2023-10-19T14:23:10.263 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.123330.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.96991.log.gz 2023-10-19T14:23:10.263 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117927.log 2023-10-19T14:23:10.263 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.123330.log.gz 2023-10-19T14:23:10.263 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.96635.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134857.log 2023-10-19T14:23:10.263 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.117927.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.96635.log.gz 2023-10-19T14:23:10.264 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123517.log 2023-10-19T14:23:10.264 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.117927.log.gz 2023-10-19T14:23:10.264 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.134857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165244.log 2023-10-19T14:23:10.264 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.134857.log.gz 2023-10-19T14:23:10.264 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.123517.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112740.log 2023-10-19T14:23:10.265 INFO:teuthology.orchestra.run.smithi057.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.123517.log.gz 2023-10-19T14:23:10.265 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.165244.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151025.log 2023-10-19T14:23:10.265 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.112740.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.165244.log.gz 2023-10-19T14:23:10.265 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141248.log 2023-10-19T14:23:10.265 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.112740.log.gz 2023-10-19T14:23:10.265 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.151025.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144701.log 2023-10-19T14:23:10.266 INFO:teuthology.orchestra.run.smithi057.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.151025.log.gz 2023-10-19T14:23:10.266 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.141248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156016.log 2023-10-19T14:23:10.266 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.144701.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.128173.log 87.4% 2023-10-19T14:23:10.266 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/ceph-client.admin.141248.log.gz 2023-10-19T14:23:10.266 INFO:teuthology.orchestra.run.smithi057.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.144701.log.gz 2023-10-19T14:23:10.267 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.156016.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155058.log 2023-10-19T14:23:10.267 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.156016.log.gz 2023-10-19T14:23:10.267 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.0.128173.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133088.log 2023-10-19T14:23:10.267 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.155058.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108154.log 2023-10-19T14:23:10.267 INFO:teuthology.orchestra.run.smithi057.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.155058.log.gz 2023-10-19T14:23:10.267 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.133088.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129902.log 2023-10-19T14:23:10.268 INFO:teuthology.orchestra.run.smithi057.stderr:gzip 84.4% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.133088.log.gz -- 2023-10-19T14:23:10.268 INFO:teuthology.orchestra.run.smithi057.stderr: /var/log/ceph/ceph-client.admin.117484.log 2023-10-19T14:23:10.268 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.108154.log: /var/log/ceph/ceph-client.admin.129902.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.108154.log.gz 2023-10-19T14:23:10.268 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.167217.log 2023-10-19T14:23:10.268 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107488.log 2023-10-19T14:23:10.269 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.129902.log.gz 2023-10-19T14:23:10.269 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.117484.log: /var/log/ceph/ceph-client.admin.167217.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.117484.log.gz 2023-10-19T14:23:10.269 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118277.log 2023-10-19T14:23:10.269 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134475.log 2023-10-19T14:23:10.269 INFO:teuthology.orchestra.run.smithi057.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.167217.log.gz 2023-10-19T14:23:10.269 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.107488.log: /var/log/ceph/ceph-client.admin.118277.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.107488.log.gz 2023-10-19T14:23:10.270 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113090.log 2023-10-19T14:23:10.270 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162383.log 2023-10-19T14:23:10.270 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.134475.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.118277.log.gz 2023-10-19T14:23:10.270 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.113090.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.134475.log.gz 2023-10-19T14:23:10.270 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109213.log 2023-10-19T14:23:10.271 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153376.log 2023-10-19T14:23:10.271 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.113090.log.gz 2023-10-19T14:23:10.271 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.162383.log: /var/log/ceph/ceph-client.admin.109213.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.162383.log.gz 2023-10-19T14:23:10.271 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121588.log 2023-10-19T14:23:10.271 INFO:teuthology.orchestra.run.smithi057.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.109213.log.gz 2023-10-19T14:23:10.271 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.153376.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123685.log 2023-10-19T14:23:10.272 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.121588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119287.log 2023-10-19T14:23:10.272 INFO:teuthology.orchestra.run.smithi057.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.121588.log.gz 2023-10-19T14:23:10.272 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.123685.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133476.log 2023-10-19T14:23:10.272 INFO:teuthology.orchestra.run.smithi057.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.153376.log.gz 2023-10-19T14:23:10.272 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.123685.log.gz 2023-10-19T14:23:10.273 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94701.log 2023-10-19T14:23:10.273 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.119287.log: /var/log/ceph/ceph-client.admin.133476.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140280.log 2023-10-19T14:23:10.273 INFO:teuthology.orchestra.run.smithi057.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.119287.log.gz 2023-10-19T14:23:10.273 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.94701.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.133476.log.gz 2023-10-19T14:23:10.273 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132499.log 2023-10-19T14:23:10.273 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.94701.log.gz 2023-10-19T14:23:10.274 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.140280.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96999.log 2023-10-19T14:23:10.274 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.140280.log.gz 2023-10-19T14:23:10.274 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98713.log 2023-10-19T14:23:10.274 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.132499.log: /var/log/ceph/ceph-client.admin.96999.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.132499.log.gz 2023-10-19T14:23:10.274 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161099.log 2023-10-19T14:23:10.275 INFO:teuthology.orchestra.run.smithi057.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.96999.log.gz 2023-10-19T14:23:10.275 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144364.log 2023-10-19T14:23:10.275 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.98713.log: gzip/var/log/ceph/ceph-client.admin.161099.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.126248.log 2023-10-19T14:23:10.275 INFO:teuthology.orchestra.run.smithi057.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.98713.log.gz 2023-10-19T14:23:10.275 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.144364.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.161099.log.gz 2023-10-19T14:23:10.275 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100791.log 2023-10-19T14:23:10.276 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.144364.log.gz 2023-10-19T14:23:10.276 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.126248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98459.log 2023-10-19T14:23:10.276 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133392.log 2023-10-19T14:23:10.276 INFO:teuthology.orchestra.run.smithi057.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.126248.log.gz 2023-10-19T14:23:10.276 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.100791.log: /var/log/ceph/ceph-client.admin.98459.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.100791.log.gz 2023-10-19T14:23:10.277 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146419.log 2023-10-19T14:23:10.277 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.98459.log.gz 2023-10-19T14:23:10.277 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.133392.log: -- /var/log/ceph/ceph-client.admin.117621.log 2023-10-19T14:23:10.277 INFO:teuthology.orchestra.run.smithi057.stderr: 86.8%/var/log/ceph/ceph-client.admin.146419.log: -- replaced with /var/log/ceph/ceph-client.admin.133392.log.gz 2023-10-19T14:23:10.277 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94913.log 2023-10-19T14:23:10.277 INFO:teuthology.orchestra.run.smithi057.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.146419.log.gz 2023-10-19T14:23:10.278 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144407.log 2023-10-19T14:23:10.278 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.117621.log: /var/log/ceph/ceph-client.admin.94913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113258.log 2023-10-19T14:23:10.278 INFO:teuthology.orchestra.run.smithi057.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.117621.log.gz 2023-10-19T14:23:10.278 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.144407.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.94913.log.gz 2023-10-19T14:23:10.278 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164345.log 2023-10-19T14:23:10.279 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.144407.log.gz 2023-10-19T14:23:10.279 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.113258.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130276.log 2023-10-19T14:23:10.279 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145459.log 2023-10-19T14:23:10.279 INFO:teuthology.orchestra.run.smithi057.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.113258.log.gz 2023-10-19T14:23:10.279 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.164345.log: /var/log/ceph/ceph-client.admin.130276.log: 87.4%gzip -- replaced with /var/log/ceph/ceph-client.admin.164345.log.gz 2023-10-19T14:23:10.280 INFO:teuthology.orchestra.run.smithi057.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.133808.log 2023-10-19T14:23:10.280 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.130276.log.gz 2023-10-19T14:23:10.280 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157448.log 2023-10-19T14:23:10.280 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.145459.log: /var/log/ceph/ceph-client.admin.133808.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147328.log 2023-10-19T14:23:10.280 INFO:teuthology.orchestra.run.smithi057.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.145459.log.gz 2023-10-19T14:23:10.280 INFO:teuthology.orchestra.run.smithi057.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.133808.log.gz 2023-10-19T14:23:10.281 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.157448.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102090.log 2023-10-19T14:23:10.281 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.157448.log.gz 2023-10-19T14:23:10.281 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.147328.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143244.log 2023-10-19T14:23:10.281 INFO:teuthology.orchestra.run.smithi057.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.147328.log.gz 2023-10-19T14:23:10.281 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106716.log 2023-10-19T14:23:10.281 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.102090.log: /var/log/ceph/ceph-client.admin.143244.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.102090.log.gz 2023-10-19T14:23:10.282 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124332.log 2023-10-19T14:23:10.282 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.143244.log.gz 2023-10-19T14:23:10.282 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122559.log 2023-10-19T14:23:10.282 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.106716.log: /var/log/ceph/ceph-client.admin.124332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119458.log 2023-10-19T14:23:10.282 INFO:teuthology.orchestra.run.smithi057.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.106716.log.gz 2023-10-19T14:23:10.283 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.124332.log.gz 2023-10-19T14:23:10.283 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.122559.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120729.log 2023-10-19T14:23:10.283 INFO:teuthology.orchestra.run.smithi057.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.122559.log.gz 2023-10-19T14:23:10.283 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.119458.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112189.log 2023-10-19T14:23:10.283 INFO:teuthology.orchestra.run.smithi057.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.119458.log.gz 2023-10-19T14:23:10.284 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.120729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126747.log 2023-10-19T14:23:10.284 INFO:teuthology.orchestra.run.smithi057.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.120729.log.gz 2023-10-19T14:23:10.284 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.112189.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148443.log 2023-10-19T14:23:10.284 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131307.log 2023-10-19T14:23:10.284 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.126747.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.112189.log.gz 2023-10-19T14:23:10.284 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.148443.log: gzip -5 --verbose -- 87.3% /var/log/ceph/ceph-client.admin.133976.log -- replaced with /var/log/ceph/ceph-client.admin.126747.log.gz 2023-10-19T14:23:10.285 INFO:teuthology.orchestra.run.smithi057.stderr: 2023-10-19T14:23:10.285 INFO:teuthology.orchestra.run.smithi057.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.148443.log.gz 2023-10-19T14:23:10.285 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111793.log 2023-10-19T14:23:10.285 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.131307.log: /var/log/ceph/ceph-client.admin.133976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101180.log 2023-10-19T14:23:10.285 INFO:teuthology.orchestra.run.smithi057.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.131307.log.gz 2023-10-19T14:23:10.286 INFO:teuthology.orchestra.run.smithi057.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.133976.log.gz 2023-10-19T14:23:10.286 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.111793.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152054.log 2023-10-19T14:23:10.286 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.111793.log.gz 2023-10-19T14:23:10.286 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.101180.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142599.log 2023-10-19T14:23:10.286 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123186.log 2023-10-19T14:23:10.286 INFO:teuthology.orchestra.run.smithi057.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.101180.log.gz 2023-10-19T14:23:10.287 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.152054.log: /var/log/ceph/ceph-client.admin.142599.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.152054.log.gzgzip 2023-10-19T14:23:10.287 INFO:teuthology.orchestra.run.smithi057.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.136176.log 2023-10-19T14:23:10.287 INFO:teuthology.orchestra.run.smithi057.stderr: 87.1%/var/log/ceph/ceph-client.admin.123186.log: -- replaced with /var/log/ceph/ceph-client.admin.142599.log.gz 2023-10-19T14:23:10.287 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104352.log 2023-10-19T14:23:10.287 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.136176.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.123186.log.gz 2023-10-19T14:23:10.288 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149747.log 2023-10-19T14:23:10.288 INFO:teuthology.orchestra.run.smithi057.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.136176.log.gz 2023-10-19T14:23:10.288 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88173.log 2023-10-19T14:23:10.288 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.104352.log: /var/log/ceph/ceph-client.admin.149747.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.105539.log 2023-10-19T14:23:10.288 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.104352.log.gz 2023-10-19T14:23:10.288 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.88173.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.149747.log.gz 2023-10-19T14:23:10.289 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106842.log 2023-10-19T14:23:10.289 INFO:teuthology.orchestra.run.smithi057.stderr: 76.7% -- replaced with /var/log/ceph/ceph-client.admin.88173.log.gz 2023-10-19T14:23:10.289 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.0.105539.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135502.log 2023-10-19T14:23:10.289 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.106842.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152390.log 2023-10-19T14:23:10.289 INFO:teuthology.orchestra.run.smithi057.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.106842.log.gz 2023-10-19T14:23:10.290 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.135502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143538.log 2023-10-19T14:23:10.290 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134060.log 2023-10-19T14:23:10.290 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.152390.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.135502.log.gz 2023-10-19T14:23:10.290 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.143538.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164203.log 2023-10-19T14:23:10.290 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.152390.log.gz 2023-10-19T14:23:10.290 INFO:teuthology.orchestra.run.smithi057.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.143538.log.gz 2023-10-19T14:23:10.291 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.134060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97667.log 2023-10-19T14:23:10.291 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.164203.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.134060.log.gz 2023-10-19T14:23:10.291 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153984.log 2023-10-19T14:23:10.291 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.164203.log.gz 2023-10-19T14:23:10.291 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.97667.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104962.log 2023-10-19T14:23:10.292 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.153984.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.97667.log.gz 2023-10-19T14:23:10.292 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152904.log 2023-10-19T14:23:10.292 INFO:teuthology.orchestra.run.smithi057.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.153984.log.gz 2023-10-19T14:23:10.292 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.104962.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162299.log 2023-10-19T14:23:10.292 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.152904.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.104962.log.gz 2023-10-19T14:23:10.292 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104436.log 2023-10-19T14:23:10.293 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.152904.log.gz 2023-10-19T14:23:10.293 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.162299.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156936.log 2023-10-19T14:23:10.293 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144827.log 2023-10-19T14:23:10.293 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.162299.log.gz 2023-10-19T14:23:10.293 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.104436.log: /var/log/ceph/ceph-client.admin.156936.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131657.log 2023-10-19T14:23:10.294 INFO:teuthology.orchestra.run.smithi057.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.104436.log.gz 2023-10-19T14:23:10.294 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.144827.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128480.log 2023-10-19T14:23:10.294 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.156936.log.gz 2023-10-19T14:23:10.294 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.131657.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163836.log 2023-10-19T14:23:10.294 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6%/var/log/ceph/ceph-client.admin.128480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106548.log 2023-10-19T14:23:10.295 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/ceph-client.admin.131657.log.gz 2023-10-19T14:23:10.295 INFO:teuthology.orchestra.run.smithi057.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.144827.log.gz 2023-10-19T14:23:10.295 INFO:teuthology.orchestra.run.smithi057.stderr:gzip 87.5% -- replaced with /var/log/ceph/ceph-client.admin.128480.log.gz 2023-10-19T14:23:10.295 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.163836.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.140322.log 2023-10-19T14:23:10.295 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.106548.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.163836.log.gz 2023-10-19T14:23:10.295 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119604.log 2023-10-19T14:23:10.296 INFO:teuthology.orchestra.run.smithi057.stderr: 86.8%/var/log/ceph/ceph-client.admin.140322.log: -- replaced with /var/log/ceph/ceph-client.admin.106548.log.gz 2023-10-19T14:23:10.296 INFO:teuthology.orchestra.run.smithi057.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.140322.log.gz 2023-10-19T14:23:10.296 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157547.log 2023-10-19T14:23:10.296 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123373.log 2023-10-19T14:23:10.296 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.119604.log: /var/log/ceph/ceph-client.admin.157547.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116961.log 2023-10-19T14:23:10.297 INFO:teuthology.orchestra.run.smithi057.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.119604.log.gz 2023-10-19T14:23:10.297 INFO:teuthology.orchestra.run.smithi057.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.157547.log.gz 2023-10-19T14:23:10.297 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.123373.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165954.log 2023-10-19T14:23:10.297 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.116961.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139738.log 2023-10-19T14:23:10.297 INFO:teuthology.orchestra.run.smithi057.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.123373.log.gz 2023-10-19T14:23:10.297 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.165954.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.116961.log.gz 2023-10-19T14:23:10.298 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135418.log 2023-10-19T14:23:10.298 INFO:teuthology.orchestra.run.smithi057.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.165954.log.gz 2023-10-19T14:23:10.298 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.139738.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99951.log 2023-10-19T14:23:10.298 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100245.log 2023-10-19T14:23:10.298 INFO:teuthology.orchestra.run.smithi057.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.139738.log.gz 2023-10-19T14:23:10.299 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.135418.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123144.log 2023-10-19T14:23:10.299 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.99951.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.135418.log.gz 2023-10-19T14:23:10.299 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.100245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134942.log 2023-10-19T14:23:10.299 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.99951.log.gz 2023-10-19T14:23:10.299 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.100245.log.gz 2023-10-19T14:23:10.299 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.123144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133850.log 2023-10-19T14:23:10.300 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118613.log 2023-10-19T14:23:10.300 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.134942.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.123144.log.gz 2023-10-19T14:23:10.300 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.133850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165786.log 2023-10-19T14:23:10.300 INFO:teuthology.orchestra.run.smithi057.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.134942.log.gz 2023-10-19T14:23:10.300 INFO:teuthology.orchestra.run.smithi057.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.133850.log.gz 2023-10-19T14:23:10.301 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.118613.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152566.log 2023-10-19T14:23:10.301 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.165786.log: 87.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.0.149328.log 2023-10-19T14:23:10.301 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/ceph-client.admin.118613.log.gz 2023-10-19T14:23:10.301 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.165786.log.gz 2023-10-19T14:23:10.301 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.152566.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139933.log 2023-10-19T14:23:10.301 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.0.149328.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.152566.log.gz 2023-10-19T14:23:10.302 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136068.log 2023-10-19T14:23:10.302 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.139933.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121546.log 2023-10-19T14:23:10.302 INFO:teuthology.orchestra.run.smithi057.stderr:gzip/var/log/ceph/ceph-client.admin.136068.log: -5 87.2% -- replaced with /var/log/ceph/ceph-client.admin.139933.log.gz 2023-10-19T14:23:10.302 INFO:teuthology.orchestra.run.smithi057.stderr: --verbose -- /var/log/ceph/ceph-client.admin.157104.log 2023-10-19T14:23:10.302 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.121546.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.136068.log.gz 2023-10-19T14:23:10.303 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- 87.0% -- replaced with /var/log/ceph/ceph-client.admin.121546.log.gz 2023-10-19T14:23:10.303 INFO:teuthology.orchestra.run.smithi057.stderr: /var/log/ceph/ceph-client.admin.111835.log 2023-10-19T14:23:10.303 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.157104.log: gzip -5 --verbose 87.6% -- replaced with /var/log/ceph/ceph-client.admin.157104.log.gz 2023-10-19T14:23:10.303 INFO:teuthology.orchestra.run.smithi057.stderr: -- /var/log/ceph/ceph-client.admin.134391.log 2023-10-19T14:23:10.303 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.111835.log: gzip -5 --verbose -- 87.8% -- replaced with /var/log/ceph/ceph-client.admin.111835.log.gz 2023-10-19T14:23:10.303 INFO:teuthology.orchestra.run.smithi057.stderr: /var/log/ceph/ceph-client.admin.144617.log 2023-10-19T14:23:10.304 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.134391.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117274.log 2023-10-19T14:23:10.304 INFO:teuthology.orchestra.run.smithi057.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.134391.log.gz 2023-10-19T14:23:10.304 INFO:teuthology.orchestra.run.smithi057.stderr:gzip/var/log/ceph/ceph-client.admin.144617.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.156978.log 2023-10-19T14:23:10.304 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.117274.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.144617.log.gz 2023-10-19T14:23:10.304 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97669.log 2023-10-19T14:23:10.305 INFO:teuthology.orchestra.run.smithi057.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.117274.log.gz 2023-10-19T14:23:10.305 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.156978.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101838.log 2023-10-19T14:23:10.305 INFO:teuthology.orchestra.run.smithi057.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.156978.log.gz 2023-10-19T14:23:10.305 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.97669.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111877.log 2023-10-19T14:23:10.305 INFO:teuthology.orchestra.run.smithi057.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.97669.log.gz 2023-10-19T14:23:10.305 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.101838.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154248.log 2023-10-19T14:23:10.306 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.101838.log.gz 2023-10-19T14:23:10.306 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.111877.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155932.log 2023-10-19T14:23:10.306 INFO:teuthology.orchestra.run.smithi057.stderr: 87.8%/var/log/ceph/ceph-client.admin.154248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143622.log 2023-10-19T14:23:10.306 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/ceph-client.admin.111877.log.gz 2023-10-19T14:23:10.306 INFO:teuthology.orchestra.run.smithi057.stderr: 87.2%/var/log/ceph/ceph-client.admin.155932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126069.log 2023-10-19T14:23:10.307 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/ceph-client.admin.154248.log.gz 2023-10-19T14:23:10.307 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.143622.log: gzip -5 --verbose -- 86.3% /var/log/ceph/ceph-client.admin.134307.log -- replaced with /var/log/ceph/ceph-client.admin.155932.log.gz 2023-10-19T14:23:10.307 INFO:teuthology.orchestra.run.smithi057.stderr: 2023-10-19T14:23:10.307 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.126069.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165202.log 2023-10-19T14:23:10.307 INFO:teuthology.orchestra.run.smithi057.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.143622.log.gz 2023-10-19T14:23:10.307 INFO:teuthology.orchestra.run.smithi057.stderr:gzip 87.0% -5 --verbose -- /var/log/ceph/ceph-client.admin.144148.log 2023-10-19T14:23:10.308 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.134307.log: -- replaced with /var/log/ceph/ceph-client.admin.126069.log.gz 2023-10-19T14:23:10.308 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.165202.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159358.log 2023-10-19T14:23:10.308 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.134307.log.gz 2023-10-19T14:23:10.308 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.165202.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116698.log 2023-10-19T14:23:10.308 INFO:teuthology.orchestra.run.smithi057.stderr: 2023-10-19T14:23:10.309 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.144148.log: /var/log/ceph/ceph-client.admin.159358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122211.log 2023-10-19T14:23:10.309 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6% 87.1% -- replaced with /var/log/ceph/ceph-client.admin.159358.log.gz 2023-10-19T14:23:10.309 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.116698.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103392.log 2023-10-19T14:23:10.309 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/ceph-client.admin.144148.log.gz 2023-10-19T14:23:10.309 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.122211.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.116698.log.gz 2023-10-19T14:23:10.309 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164035.log 2023-10-19T14:23:10.310 INFO:teuthology.orchestra.run.smithi057.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.122211.log.gz 2023-10-19T14:23:10.310 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.103392.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142344.log 2023-10-19T14:23:10.310 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.103392.log.gz 2023-10-19T14:23:10.310 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.164035.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147664.log 2023-10-19T14:23:10.310 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.164035.log.gz 2023-10-19T14:23:10.311 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.142344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151867.log 2023-10-19T14:23:10.311 INFO:teuthology.orchestra.run.smithi057.stderr: 87.3%/var/log/ceph/ceph-client.admin.147664.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162892.log 2023-10-19T14:23:10.311 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/ceph-client.admin.142344.log.gz 2023-10-19T14:23:10.311 INFO:teuthology.orchestra.run.smithi057.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.147664.log.gz 2023-10-19T14:23:10.311 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107530.log 2023-10-19T14:23:10.312 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.151867.log: gzip 87.7% -- replaced with /var/log/ceph/ceph-client.admin.151867.log.gz 2023-10-19T14:23:10.312 INFO:teuthology.orchestra.run.smithi057.stderr: -5 --verbose/var/log/ceph/ceph-client.admin.162892.log: -- /var/log/ceph/ceph-client.admin.111578.log 2023-10-19T14:23:10.312 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.107530.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.162892.log.gz 2023-10-19T14:23:10.312 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160371.log 87.3% -- replaced with /var/log/ceph/ceph-client.admin.107530.log.gz 2023-10-19T14:23:10.312 INFO:teuthology.orchestra.run.smithi057.stderr: 2023-10-19T14:23:10.312 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.111578.log: 94.4% -- replaced with /var/log/ceph/ceph-client.0.128173.log.gz 2023-10-19T14:23:10.313 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131351.log 2023-10-19T14:23:10.313 INFO:teuthology.orchestra.run.smithi057.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.111578.log.gz 2023-10-19T14:23:10.313 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.160371.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152860.log 2023-10-19T14:23:10.313 INFO:teuthology.orchestra.run.smithi057.stderr: 87.0%/var/log/ceph/ceph-client.admin.131351.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159486.log 2023-10-19T14:23:10.313 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/ceph-client.admin.160371.log.gz 2023-10-19T14:23:10.314 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.152860.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112273.log 2023-10-19T14:23:10.314 INFO:teuthology.orchestra.run.smithi057.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.131351.log.gz 2023-10-19T14:23:10.314 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117824.log 2023-10-19T14:23:10.314 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.159486.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.152860.log.gz 2023-10-19T14:23:10.314 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.112273.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148581.log 2023-10-19T14:23:10.314 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.159486.log.gz 2023-10-19T14:23:10.315 INFO:teuthology.orchestra.run.smithi057.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.112273.log.gz 2023-10-19T14:23:10.315 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.117824.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161184.log 2023-10-19T14:23:10.315 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.148581.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.117824.log.gz 2023-10-19T14:23:10.315 INFO:teuthology.orchestra.run.smithi057.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.148581.log.gz 2023-10-19T14:23:10.315 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126873.log 2023-10-19T14:23:10.316 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.161184.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106110.log 2023-10-19T14:23:10.316 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.161184.log.gz 2023-10-19T14:23:10.316 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.126873.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.163529.log 2023-10-19T14:23:10.316 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.106110.log: gzip -5 87.7% --verbose -- /var/log/ceph/ceph-client.0.142079.log 2023-10-19T14:23:10.316 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/ceph-client.admin.126873.log.gz 2023-10-19T14:23:10.316 INFO:teuthology.orchestra.run.smithi057.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.106110.log.gz 2023-10-19T14:23:10.317 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.0.163529.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131223.log 2023-10-19T14:23:10.317 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.0.142079.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101922.log 2023-10-19T14:23:10.317 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.131223.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112315.log 2023-10-19T14:23:10.317 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124196.log 2023-10-19T14:23:10.317 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.101922.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.131223.log.gz 2023-10-19T14:23:10.318 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.101922.log.gz 2023-10-19T14:23:10.318 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.112315.log: gzip -5 91.6% -- replaced with /var/log/ceph/ceph-client.0.163529.log.gz 2023-10-19T14:23:10.318 INFO:teuthology.orchestra.run.smithi057.stderr: --verbose 87.3% -- /var/log/ceph/ceph-client.admin.98501.log 2023-10-19T14:23:10.318 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/ceph-client.admin.112315.log.gz 2023-10-19T14:23:10.318 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.124196.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112147.log 2023-10-19T14:23:10.318 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.124196.log.gz 2023-10-19T14:23:10.319 INFO:teuthology.orchestra.run.smithi057.stderr: 91.6% -- replaced with /var/log/ceph/ceph-client.0.142079.log.gz 2023-10-19T14:23:10.319 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.98501.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100077.log 2023-10-19T14:23:10.319 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160413.log 2023-10-19T14:23:10.319 INFO:teuthology.orchestra.run.smithi057.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.98501.log.gz 2023-10-19T14:23:10.319 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.112147.log: /var/log/ceph/ceph-client.admin.100077.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158024.log 2023-10-19T14:23:10.320 INFO:teuthology.orchestra.run.smithi057.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.112147.log.gz 2023-10-19T14:23:10.320 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.100077.log.gz 2023-10-19T14:23:10.320 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.160413.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112357.log 2023-10-19T14:23:10.320 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.160413.log.gz 2023-10-19T14:23:10.320 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.158024.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145402.log 2023-10-19T14:23:10.320 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88955.log 2023-10-19T14:23:10.321 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.158024.log.gz 2023-10-19T14:23:10.321 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.112357.log: /var/log/ceph/ceph-client.admin.145402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130097.log 2023-10-19T14:23:10.321 INFO:teuthology.orchestra.run.smithi057.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.112357.log.gz 2023-10-19T14:23:10.321 INFO:teuthology.orchestra.run.smithi057.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.145402.log.gz 2023-10-19T14:23:10.321 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.88955.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119646.log 2023-10-19T14:23:10.322 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112697.log 86.3% 2023-10-19T14:23:10.322 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/ceph-client.admin.88955.log.gz 2023-10-19T14:23:10.322 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.130097.log: /var/log/ceph/ceph-client.admin.119646.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140635.log 2023-10-19T14:23:10.322 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.130097.log.gz 2023-10-19T14:23:10.322 INFO:teuthology.orchestra.run.smithi057.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.119646.log.gz 2023-10-19T14:23:10.322 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119389.log 2023-10-19T14:23:10.323 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.112697.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98375.log 2023-10-19T14:23:10.323 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.140635.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.112697.log.gz 2023-10-19T14:23:10.323 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.119389.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103812.log 2023-10-19T14:23:10.323 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.140635.log.gz 2023-10-19T14:23:10.323 INFO:teuthology.orchestra.run.smithi057.stderr: 87.2%gzip -- replaced with /var/log/ceph/ceph-client.admin.119389.log.gz -5 2023-10-19T14:23:10.324 INFO:teuthology.orchestra.run.smithi057.stderr: --verbose -- /var/log/ceph/ceph-client.admin.114100.log 2023-10-19T14:23:10.324 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.98375.log: /var/log/ceph/ceph-client.admin.103812.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158192.log 2023-10-19T14:23:10.324 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.98375.log.gz 2023-10-19T14:23:10.324 INFO:teuthology.orchestra.run.smithi057.stderr:gzip/var/log/ceph/ceph-client.admin.114100.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.132865.log 2023-10-19T14:23:10.324 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.103812.log.gz 2023-10-19T14:23:10.324 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.158192.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.114100.log.gz 2023-10-19T14:23:10.325 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117087.log 2023-10-19T14:23:10.325 INFO:teuthology.orchestra.run.smithi057.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.158192.log.gz 2023-10-19T14:23:10.325 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.132865.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139780.log 2023-10-19T14:23:10.325 INFO:teuthology.orchestra.run.smithi057.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.132865.log.gz 2023-10-19T14:23:10.325 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.117087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97955.log 2023-10-19T14:23:10.326 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164261.log 2023-10-19T14:23:10.326 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.139780.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.117087.log.gz 2023-10-19T14:23:10.326 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.97955.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.139780.log.gz 2023-10-19T14:23:10.326 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88055.log 2023-10-19T14:23:10.326 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.97955.log.gz 2023-10-19T14:23:10.326 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.164261.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123102.log 2023-10-19T14:23:10.327 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.88055.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142236.log 2023-10-19T14:23:10.327 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.164261.log.gz 2023-10-19T14:23:10.327 INFO:teuthology.orchestra.run.smithi057.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88055.log.gz 2023-10-19T14:23:10.327 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.123102.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118361.log 2023-10-19T14:23:10.327 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.123102.log.gz 2023-10-19T14:23:10.328 INFO:teuthology.orchestra.run.smithi057.stderr:gzip/var/log/ceph/ceph-client.admin.142236.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.96425.log 2023-10-19T14:23:10.328 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.118361.log: 86.4%gzip -- replaced with /var/log/ceph/ceph-client.admin.142236.log.gz 2023-10-19T14:23:10.328 INFO:teuthology.orchestra.run.smithi057.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.130360.log 2023-10-19T14:23:10.328 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.96425.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.118361.log.gz 2023-10-19T14:23:10.328 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104776.log 2023-10-19T14:23:10.328 INFO:teuthology.orchestra.run.smithi057.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.96425.log.gz 2023-10-19T14:23:10.329 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.130360.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104099.log 2023-10-19T14:23:10.329 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97336.log 2023-10-19T14:23:10.329 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.104776.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.130360.log.gz 2023-10-19T14:23:10.329 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.104099.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143118.log 2023-10-19T14:23:10.329 INFO:teuthology.orchestra.run.smithi057.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.104776.log.gz 2023-10-19T14:23:10.330 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.104099.log.gz 2023-10-19T14:23:10.330 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.97336.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143766.log 2023-10-19T14:23:10.330 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105088.log 2023-10-19T14:23:10.330 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.143118.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.97336.log.gz 2023-10-19T14:23:10.330 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.143766.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123727.log 2023-10-19T14:23:10.333 INFO:teuthology.orchestra.run.smithi057.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.143118.log.gz 2023-10-19T14:23:10.333 INFO:teuthology.orchestra.run.smithi057.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.143766.log.gz 2023-10-19T14:23:10.334 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.105088.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102416.log 2023-10-19T14:23:10.334 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.123727.log: gzip 87.2% -- replaced with /var/log/ceph/ceph-client.admin.105088.log.gz 2023-10-19T14:23:10.334 INFO:teuthology.orchestra.run.smithi057.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.100455.log 2023-10-19T14:23:10.334 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.123727.log.gz 2023-10-19T14:23:10.334 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.102416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151265.log 2023-10-19T14:23:10.335 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.102416.log.gz 2023-10-19T14:23:10.335 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134265.log 2023-10-19T14:23:10.335 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.100455.log: /var/log/ceph/ceph-client.admin.151265.log: gzip -5 87.4% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.100455.log.gz /var/log/ceph/ceph-client.admin.101712.log 2023-10-19T14:23:10.335 INFO:teuthology.orchestra.run.smithi057.stderr: 2023-10-19T14:23:10.335 INFO:teuthology.orchestra.run.smithi057.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.151265.log.gz 2023-10-19T14:23:10.335 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.134265.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104141.log 2023-10-19T14:23:10.336 INFO:teuthology.orchestra.run.smithi057.stderr: 87.1%/var/log/ceph/ceph-client.admin.101712.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114296.log 2023-10-19T14:23:10.336 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/ceph-client.admin.134265.log.gz 2023-10-19T14:23:10.336 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5%/var/log/ceph/ceph-client.admin.104141.log: -- replaced with /var/log/ceph/ceph-client.admin.101712.log.gz 2023-10-19T14:23:10.336 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124544.log 2023-10-19T14:23:10.336 INFO:teuthology.orchestra.run.smithi057.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.104141.log.gz 2023-10-19T14:23:10.337 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.114296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121292.log 2023-10-19T14:23:10.337 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149789.log 2023-10-19T14:23:10.337 INFO:teuthology.orchestra.run.smithi057.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.114296.log.gz 2023-10-19T14:23:10.337 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.124544.log: /var/log/ceph/ceph-client.admin.121292.log: gzip 87.0% -- replaced with /var/log/ceph/ceph-client.admin.124544.log.gz 2023-10-19T14:23:10.337 INFO:teuthology.orchestra.run.smithi057.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.158360.log 2023-10-19T14:23:10.337 INFO:teuthology.orchestra.run.smithi057.stderr: 87.2%/var/log/ceph/ceph-client.admin.149789.log: -- replaced with /var/log/ceph/ceph-client.admin.121292.log.gz 2023-10-19T14:23:10.338 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95785.log 87.2% 2023-10-19T14:23:10.338 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/ceph-client.admin.149789.log.gz 2023-10-19T14:23:10.338 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124671.log 2023-10-19T14:23:10.338 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.158360.log: /var/log/ceph/ceph-client.admin.95785.log: gzip 87.2% -- replaced with /var/log/ceph/ceph-client.admin.158360.log.gz -5 2023-10-19T14:23:10.338 INFO:teuthology.orchestra.run.smithi057.stderr: --verbose -- /var/log/ceph/ceph-client.admin.159444.log 2023-10-19T14:23:10.339 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.95785.log.gz 2023-10-19T14:23:10.339 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151307.log 2023-10-19T14:23:10.339 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.124671.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96509.log 2023-10-19T14:23:10.339 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.159444.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.124671.log.gz 2023-10-19T14:23:10.339 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.151307.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106346.log 2023-10-19T14:23:10.339 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.159444.log.gz 2023-10-19T14:23:10.340 INFO:teuthology.orchestra.run.smithi057.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.151307.log.gz 2023-10-19T14:23:10.340 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.96509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141076.log 2023-10-19T14:23:10.340 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.106346.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.96509.log.gz 2023-10-19T14:23:10.340 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120104.log 2023-10-19T14:23:10.340 INFO:teuthology.orchestra.run.smithi057.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.106346.log.gz 2023-10-19T14:23:10.341 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.141076.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97671.log 2023-10-19T14:23:10.341 INFO:teuthology.orchestra.run.smithi057.stderr: 87.7%/var/log/ceph/ceph-client.admin.120104.log: -- replaced with /var/log/ceph/ceph-client.admin.141076.log.gz 2023-10-19T14:23:10.341 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100161.log 2023-10-19T14:23:10.341 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.120104.log.gz/var/log/ceph/ceph-client.admin.97671.log: 2023-10-19T14:23:10.341 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139696.log 2023-10-19T14:23:10.341 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.97671.log.gz 2023-10-19T14:23:10.342 INFO:teuthology.orchestra.run.smithi057.stderr:gzip/var/log/ceph/ceph-client.admin.100161.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.125985.log 2023-10-19T14:23:10.342 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161791.log 2023-10-19T14:23:10.342 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.139696.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.100161.log.gz 2023-10-19T14:23:10.342 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.125985.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134984.log 2023-10-19T14:23:10.342 INFO:teuthology.orchestra.run.smithi057.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.139696.log.gz 2023-10-19T14:23:10.343 INFO:teuthology.orchestra.run.smithi057.stderr:gzip/var/log/ceph/ceph-client.admin.161791.log: -5 87.5% -- replaced with /var/log/ceph/ceph-client.admin.125985.log.gz 2023-10-19T14:23:10.343 INFO:teuthology.orchestra.run.smithi057.stderr: --verbose -- /var/log/ceph/ceph-client.admin.165618.log 2023-10-19T14:23:10.343 INFO:teuthology.orchestra.run.smithi057.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.161791.log.gz 2023-10-19T14:23:10.343 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.134984.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140382.log 2023-10-19T14:23:10.343 INFO:teuthology.orchestra.run.smithi057.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.134984.log.gz 2023-10-19T14:23:10.344 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.165618.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-10-19T14:23:10.344 INFO:teuthology.orchestra.run.smithi057.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.165618.log.gz 2023-10-19T14:23:10.344 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.140382.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130192.log 2023-10-19T14:23:10.344 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph.audit.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.140382.log.gz 2023-10-19T14:23:10.344 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162766.log 2023-10-19T14:23:10.344 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.130192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155753.log 2023-10-19T14:23:10.345 INFO:teuthology.orchestra.run.smithi057.stderr:gzip/var/log/ceph/ceph-client.admin.162766.log: -5 86.7% -- replaced with /var/log/ceph/ceph-client.admin.130192.log.gz 2023-10-19T14:23:10.345 INFO:teuthology.orchestra.run.smithi057.stderr: --verbose -- /var/log/ceph/ceph-client.admin.121198.log 2023-10-19T14:23:10.345 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.155753.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.162766.log.gz 2023-10-19T14:23:10.345 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- 87.4% -- replaced with /var/log/ceph/ceph-client.admin.155753.log.gz 2023-10-19T14:23:10.345 INFO:teuthology.orchestra.run.smithi057.stderr: /var/log/ceph/ceph-client.admin.107918.log 2023-10-19T14:23:10.346 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.121198.log: gzip -5 --verbose 87.6% -- -- replaced with /var/log/ceph/ceph-client.admin.121198.log.gz 2023-10-19T14:23:10.346 INFO:teuthology.orchestra.run.smithi057.stderr: /var/log/ceph/ceph-client.admin.123228.log 2023-10-19T14:23:10.346 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.107918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153082.log 2023-10-19T14:23:10.346 INFO:teuthology.orchestra.run.smithi057.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.107918.log.gz 2023-10-19T14:23:10.346 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.123228.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161142.log 2023-10-19T14:23:10.346 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6%/var/log/ceph/ceph-client.admin.153082.log: -- replaced with /var/log/ceph/ceph-client.admin.123228.log.gz 2023-10-19T14:23:10.347 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161959.log 2023-10-19T14:23:10.347 INFO:teuthology.orchestra.run.smithi057.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.153082.log.gz 2023-10-19T14:23:10.347 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.161142.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160329.log 2023-10-19T14:23:10.347 INFO:teuthology.orchestra.run.smithi057.stderr: 87.2%/var/log/ceph/ceph-client.admin.161959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151349.log 2023-10-19T14:23:10.347 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/ceph-client.admin.161142.log.gz 2023-10-19T14:23:10.348 INFO:teuthology.orchestra.run.smithi057.stderr: 86.8%/var/log/ceph/ceph-client.admin.160329.log: gzip -- replaced with /var/log/ceph/ceph-client.admin.161959.log.gz 2023-10-19T14:23:10.348 INFO:teuthology.orchestra.run.smithi057.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.124779.log 2023-10-19T14:23:10.348 INFO:teuthology.orchestra.run.smithi057.stderr:gzip 87.2% -- replaced with /var/log/ceph/ceph-client.admin.160329.log.gz 2023-10-19T14:23:10.348 INFO:teuthology.orchestra.run.smithi057.stderr: -5 --verbose/var/log/ceph/ceph-client.admin.151349.log: -- /var/log/ceph/ceph-client.admin.140028.log 2023-10-19T14:23:10.348 INFO:teuthology.orchestra.run.smithi057.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.151349.log.gz 2023-10-19T14:23:10.349 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108653.log 2023-10-19T14:23:10.349 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.124779.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165870.log 2023-10-19T14:23:10.349 INFO:teuthology.orchestra.run.smithi057.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.124779.log.gz/var/log/ceph/ceph-client.admin.140028.log: 2023-10-19T14:23:10.349 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.108653.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105214.log 2023-10-19T14:23:10.349 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.165870.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154586.log 2023-10-19T14:23:10.349 INFO:teuthology.orchestra.run.smithi057.stderr: 87.0% 86.8% -- replaced with /var/log/ceph/ceph-client.admin.108653.log.gz 2023-10-19T14:23:10.350 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/ceph-client.admin.140028.log.gz 2023-10-19T14:23:10.350 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.105214.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.165870.log.gz 2023-10-19T14:23:10.350 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161226.log 2023-10-19T14:23:10.350 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.154586.log: gzip -5 --verbose 91.0% -- replaced with /var/log/ceph/ceph-client.admin.105214.log.gz 2023-10-19T14:23:10.350 INFO:teuthology.orchestra.run.smithi057.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.154586.log.gz 2023-10-19T14:23:10.351 INFO:teuthology.orchestra.run.smithi057.stderr: -- /var/log/ceph/ceph-client.admin.155474.log 2023-10-19T14:23:10.351 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.161226.log: gzip -5 --verbose 87.8% -- /var/log/ceph/ceph-client.admin.155558.log 2023-10-19T14:23:10.351 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/ceph-client.admin.161226.log.gz 2023-10-19T14:23:10.351 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.155474.log: 90.0% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-10-19T14:23:10.351 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.99099.log 2023-10-19T14:23:10.351 INFO:teuthology.orchestra.run.smithi057.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.155474.log.gz 2023-10-19T14:23:10.352 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134559.log 2023-10-19T14:23:10.352 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.155558.log: /var/log/ceph/ceph-client.0.99099.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140509.log 2023-10-19T14:23:10.352 INFO:teuthology.orchestra.run.smithi057.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.155558.log.gz 2023-10-19T14:23:10.352 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.134559.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88453.log 2023-10-19T14:23:10.352 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.134559.log.gz 2023-10-19T14:23:10.353 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112452.log 2023-10-19T14:23:10.353 INFO:teuthology.orchestra.run.smithi057.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.0.99099.log.gz 2023-10-19T14:23:10.353 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.140509.log: /var/log/ceph/ceph-client.admin.88453.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99909.log 2023-10-19T14:23:10.353 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% 85.6% -- replaced with /var/log/ceph/ceph-client.admin.140509.log.gz 2023-10-19T14:23:10.353 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/ceph-client.admin.88453.log.gz 2023-10-19T14:23:10.353 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.112452.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154502.log 2023-10-19T14:23:10.354 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.112452.log.gz 2023-10-19T14:23:10.354 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.99909.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143682.log 2023-10-19T14:23:10.354 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.154502.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.99909.log.gz 2023-10-19T14:23:10.354 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114034.log 2023-10-19T14:23:10.354 INFO:teuthology.orchestra.run.smithi057.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.154502.log.gz 2023-10-19T14:23:10.355 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.143682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96719.log 2023-10-19T14:23:10.355 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.143682.log.gz 2023-10-19T14:23:10.355 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.114034.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94657.log 2023-10-19T14:23:10.355 INFO:teuthology.orchestra.run.smithi057.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.114034.log.gz 2023-10-19T14:23:10.355 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.96719.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.96719.log.gz 2023-10-19T14:23:10.355 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127914.log 2023-10-19T14:23:10.356 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120332.log 2023-10-19T14:23:10.356 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.94657.log: /var/log/ceph/ceph-client.admin.127914.log: gzip -5 87.2% -- replaced with /var/log/ceph/ceph-client.admin.94657.log.gz --verbose 2023-10-19T14:23:10.356 INFO:teuthology.orchestra.run.smithi057.stderr: -- /var/log/ceph/ceph-client.admin.94785.log 2023-10-19T14:23:10.356 INFO:teuthology.orchestra.run.smithi057.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.127914.log.gz 2023-10-19T14:23:10.356 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.120332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136130.log 2023-10-19T14:23:10.357 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.120332.log.gz 2023-10-19T14:23:10.357 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135072.log 2023-10-19T14:23:10.357 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.94785.log: /var/log/ceph/ceph-client.admin.136130.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127764.log 2023-10-19T14:23:10.357 INFO:teuthology.orchestra.run.smithi057.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.94785.log.gz 2023-10-19T14:23:10.357 INFO:teuthology.orchestra.run.smithi057.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.136130.log.gz 2023-10-19T14:23:10.357 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.135072.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106405.log 2023-10-19T14:23:10.358 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125195.log 2023-10-19T14:23:10.358 INFO:teuthology.orchestra.run.smithi057.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.135072.log.gz 2023-10-19T14:23:10.358 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.127764.log: /var/log/ceph/ceph-client.admin.106405.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164161.log 2023-10-19T14:23:10.358 INFO:teuthology.orchestra.run.smithi057.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.127764.log.gz 2023-10-19T14:23:10.358 INFO:teuthology.orchestra.run.smithi057.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.106405.log.gz 2023-10-19T14:23:10.359 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.125195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97668.log 2023-10-19T14:23:10.359 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.164161.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.125195.log.gz 2023-10-19T14:23:10.359 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112870.log 2023-10-19T14:23:10.359 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.164161.log.gz 2023-10-19T14:23:10.359 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.97668.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127722.log 2023-10-19T14:23:10.359 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.112870.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105046.log 2023-10-19T14:23:10.360 INFO:teuthology.orchestra.run.smithi057.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.97668.log.gz 2023-10-19T14:23:10.360 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.127722.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.112870.log.gz 2023-10-19T14:23:10.360 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157938.log 2023-10-19T14:23:10.360 INFO:teuthology.orchestra.run.smithi057.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.127722.log.gz 2023-10-19T14:23:10.360 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.105046.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111377.log 2023-10-19T14:23:10.361 INFO:teuthology.orchestra.run.smithi057.stderr:gzip 86.9% -- replaced with /var/log/ceph/ceph-client.admin.105046.log.gz 2023-10-19T14:23:10.361 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.157938.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.95869.log 2023-10-19T14:23:10.361 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.111377.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.157938.log.gz 2023-10-19T14:23:10.361 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96593.log 2023-10-19T14:23:10.361 INFO:teuthology.orchestra.run.smithi057.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.111377.log.gz 2023-10-19T14:23:10.362 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.95869.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113576.log 2023-10-19T14:23:10.362 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158108.log 2023-10-19T14:23:10.362 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5%/var/log/ceph/ceph-client.admin.96593.log: -- replaced with /var/log/ceph/ceph-client.admin.95869.log.gz 2023-10-19T14:23:10.362 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.113576.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96179.log 2023-10-19T14:23:10.362 INFO:teuthology.orchestra.run.smithi057.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.96593.log.gz 2023-10-19T14:23:10.362 INFO:teuthology.orchestra.run.smithi057.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.113576.log.gz 2023-10-19T14:23:10.363 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.158108.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153250.log 2023-10-19T14:23:10.363 INFO:teuthology.orchestra.run.smithi057.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.158108.log.gz 2023-10-19T14:23:10.363 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.96179.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88109.log 2023-10-19T14:23:10.363 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.96179.log.gz 2023-10-19T14:23:10.363 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.153250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127428.log 2023-10-19T14:23:10.364 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.153250.log.gz 2023-10-19T14:23:10.364 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.88109.log: 94.3% -- replaced with /var/log/ceph/ceph-client.0.149328.log.gz 2023-10-19T14:23:10.364 INFO:teuthology.orchestra.run.smithi057.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88109.log.gz 2023-10-19T14:23:10.364 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116835.log 2023-10-19T14:23:10.364 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.127428.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.127428.log.gz 2023-10-19T14:23:10.364 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88673.log 2023-10-19T14:23:10.365 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.116835.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96383.log 2023-10-19T14:23:10.365 INFO:teuthology.orchestra.run.smithi057.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.116835.log.gz 2023-10-19T14:23:10.365 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.88673.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124713.log 2023-10-19T14:23:10.365 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119841.log 2023-10-19T14:23:10.365 INFO:teuthology.orchestra.run.smithi057.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.88673.log.gz 2023-10-19T14:23:10.366 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.96383.log: /var/log/ceph/ceph-client.admin.124713.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126831.log 2023-10-19T14:23:10.366 INFO:teuthology.orchestra.run.smithi057.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.96383.log.gz 2023-10-19T14:23:10.366 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98333.log 2023-10-19T14:23:10.366 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.119841.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.124713.log.gz 2023-10-19T14:23:10.366 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.126831.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161707.log 2023-10-19T14:23:10.367 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.119841.log.gz 2023-10-19T14:23:10.367 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153208.log 2023-10-19T14:23:10.367 INFO:teuthology.orchestra.run.smithi057.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.126831.log.gz 2023-10-19T14:23:10.367 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.98333.log: /var/log/ceph/ceph-client.admin.161707.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.98333.log.gz 2023-10-19T14:23:10.367 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116461.log 2023-10-19T14:23:10.367 INFO:teuthology.orchestra.run.smithi057.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.161707.log.gz 2023-10-19T14:23:10.368 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109773.log 2023-10-19T14:23:10.368 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.153208.log: /var/log/ceph/ceph-client.admin.116461.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102558.log 2023-10-19T14:23:10.368 INFO:teuthology.orchestra.run.smithi057.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.153208.log.gz 2023-10-19T14:23:10.368 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.109773.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.116461.log.gz 2023-10-19T14:23:10.368 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130822.log 2023-10-19T14:23:10.369 INFO:teuthology.orchestra.run.smithi057.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.109773.log.gz 2023-10-19T14:23:10.369 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.102558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148095.log 2023-10-19T14:23:10.369 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138454.log 2023-10-19T14:23:10.369 INFO:teuthology.orchestra.run.smithi057.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.102558.log.gz 2023-10-19T14:23:10.369 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.130822.log: /var/log/ceph/ceph-client.admin.148095.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.130822.log.gz 2023-10-19T14:23:10.369 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111251.log 2023-10-19T14:23:10.370 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.148095.log.gz 2023-10-19T14:23:10.370 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145309.log 2023-10-19T14:23:10.370 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.138454.log: /var/log/ceph/ceph-client.admin.111251.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125706.log 2023-10-19T14:23:10.370 INFO:teuthology.orchestra.run.smithi057.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.138454.log.gz 2023-10-19T14:23:10.370 INFO:teuthology.orchestra.run.smithi057.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.111251.log.gz 2023-10-19T14:23:10.371 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.145309.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94445.log 2023-10-19T14:23:10.371 INFO:teuthology.orchestra.run.smithi057.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.145309.log.gz 2023-10-19T14:23:10.371 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.125706.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164077.log 2023-10-19T14:23:10.371 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94955.log 2023-10-19T14:23:10.371 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.94445.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.125706.log.gz 2023-10-19T14:23:10.371 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.164077.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.94445.log.gz 2023-10-19T14:23:10.372 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.167051.log 2023-10-19T14:23:10.372 INFO:teuthology.orchestra.run.smithi057.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.164077.log.gz 2023-10-19T14:23:10.372 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160787.log 2023-10-19T14:23:10.372 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.94955.log: /var/log/ceph/ceph-client.admin.167051.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153442.log 2023-10-19T14:23:10.372 INFO:teuthology.orchestra.run.smithi057.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.94955.log.gz 2023-10-19T14:23:10.373 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.160787.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.167051.log.gz 2023-10-19T14:23:10.373 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160524.log 2023-10-19T14:23:10.373 INFO:teuthology.orchestra.run.smithi057.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.160787.log.gz 2023-10-19T14:23:10.373 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.153442.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110316.log 2023-10-19T14:23:10.373 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94487.log 2023-10-19T14:23:10.373 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.153442.log.gz 2023-10-19T14:23:10.374 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.160524.log: /var/log/ceph/ceph-client.admin.110316.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.160524.log.gz 2023-10-19T14:23:10.374 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.135869.log 2023-10-19T14:23:10.374 INFO:teuthology.orchestra.run.smithi057.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.110316.log.gz 2023-10-19T14:23:10.374 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94261.log 2023-10-19T14:23:10.374 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.94487.log: /var/log/ceph/ceph-client.0.135869.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107698.log 2023-10-19T14:23:10.375 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.94487.log.gz 2023-10-19T14:23:10.375 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.94261.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107800.log 2023-10-19T14:23:10.375 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.94261.log.gz 2023-10-19T14:23:10.375 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.107698.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147038.log 2023-10-19T14:23:10.375 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108272.log 2023-10-19T14:23:10.375 INFO:teuthology.orchestra.run.smithi057.stderr: 87.0%/var/log/ceph/ceph-client.admin.107800.log: -- replaced with /var/log/ceph/ceph-client.admin.107698.log.gz 2023-10-19T14:23:10.376 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.147038.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130013.log 2023-10-19T14:23:10.376 INFO:teuthology.orchestra.run.smithi057.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.107800.log.gz 2023-10-19T14:23:10.376 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.108272.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165702.log 2023-10-19T14:23:10.376 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.147038.log.gz 2023-10-19T14:23:10.376 INFO:teuthology.orchestra.run.smithi057.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.108272.log.gz 2023-10-19T14:23:10.377 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.130013.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160577.log 2023-10-19T14:23:10.377 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155390.log 2023-10-19T14:23:10.377 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.165702.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.130013.log.gz 2023-10-19T14:23:10.377 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.160577.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166757.log 2023-10-19T14:23:10.377 INFO:teuthology.orchestra.run.smithi057.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.165702.log.gz 2023-10-19T14:23:10.377 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.155390.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125237.log 2023-10-19T14:23:10.378 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.160577.log.gz 2023-10-19T14:23:10.378 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.166757.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117190.log 2023-10-19T14:23:10.378 INFO:teuthology.orchestra.run.smithi057.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.155390.log.gz 2023-10-19T14:23:10.378 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.125237.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133760.log 2023-10-19T14:23:10.378 INFO:teuthology.orchestra.run.smithi057.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.166757.log.gz 2023-10-19T14:23:10.379 INFO:teuthology.orchestra.run.smithi057.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.125237.log.gz 2023-10-19T14:23:10.379 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127130.log 2023-10-19T14:23:10.379 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.117190.log: gzip -5 --verbose/var/log/ceph/ceph-client.admin.133760.log: -- /var/log/ceph/ceph-client.admin.95953.log 2023-10-19T14:23:10.379 INFO:teuthology.orchestra.run.smithi057.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.117190.log.gz 2023-10-19T14:23:10.379 INFO:teuthology.orchestra.run.smithi057.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.133760.log.gz 2023-10-19T14:23:10.379 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.127130.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131741.log 2023-10-19T14:23:10.380 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135334.log 2023-10-19T14:23:10.380 INFO:teuthology.orchestra.run.smithi057.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.127130.log.gz 2023-10-19T14:23:10.380 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.95953.log: /var/log/ceph/ceph-client.admin.131741.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166295.log 2023-10-19T14:23:10.380 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.95953.log.gz 2023-10-19T14:23:10.380 INFO:teuthology.orchestra.run.smithi057.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.131741.log.gz 2023-10-19T14:23:10.381 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.135334.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163788.log 2023-10-19T14:23:10.381 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147412.log 2023-10-19T14:23:10.381 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.135334.log.gz/var/log/ceph/ceph-client.admin.166295.log: 2023-10-19T14:23:10.381 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.163788.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118191.log 2023-10-19T14:23:10.381 INFO:teuthology.orchestra.run.smithi057.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.166295.log.gz 2023-10-19T14:23:10.381 INFO:teuthology.orchestra.run.smithi057.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.163788.log.gz 2023-10-19T14:23:10.382 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.147412.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141544.log 2023-10-19T14:23:10.382 INFO:teuthology.orchestra.run.smithi057.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.147412.log.gzgzip 2023-10-19T14:23:10.382 INFO:teuthology.orchestra.run.smithi057.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.131933.log 2023-10-19T14:23:10.382 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.118191.log: /var/log/ceph/ceph-client.admin.141544.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161436.log 2023-10-19T14:23:10.382 INFO:teuthology.orchestra.run.smithi057.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.118191.log.gz 2023-10-19T14:23:10.383 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.141544.log.gz 2023-10-19T14:23:10.383 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.131933.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144659.log 2023-10-19T14:23:10.383 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88145.log 2023-10-19T14:23:10.383 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.131933.log.gz 2023-10-19T14:23:10.383 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.161436.log: /var/log/ceph/ceph-client.admin.144659.log: 87.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106228.log 2023-10-19T14:23:10.384 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/ceph-client.admin.161436.log.gz 2023-10-19T14:23:10.384 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.88145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154460.log 2023-10-19T14:23:10.384 INFO:teuthology.orchestra.run.smithi057.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.144659.log.gz 2023-10-19T14:23:10.384 INFO:teuthology.orchestra.run.smithi057.stderr: 76.0% -- replaced with /var/log/ceph/ceph-client.admin.88145.log.gz 2023-10-19T14:23:10.384 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.106228.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104608.log 2023-10-19T14:23:10.384 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105804.log 2023-10-19T14:23:10.385 INFO:teuthology.orchestra.run.smithi057.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.106228.log.gz 2023-10-19T14:23:10.385 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.154460.log: /var/log/ceph/ceph-client.admin.104608.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97338.log 2023-10-19T14:23:10.385 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.154460.log.gz 2023-10-19T14:23:10.385 INFO:teuthology.orchestra.run.smithi057.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.104608.log.gz 2023-10-19T14:23:10.385 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.105804.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130696.log 2023-10-19T14:23:10.386 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148919.log 2023-10-19T14:23:10.386 INFO:teuthology.orchestra.run.smithi057.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.105804.log.gz 2023-10-19T14:23:10.386 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.97338.log: /var/log/ceph/ceph-client.admin.130696.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147191.log 2023-10-19T14:23:10.386 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.97338.log.gz 2023-10-19T14:23:10.386 INFO:teuthology.orchestra.run.smithi057.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.130696.log.gz 2023-10-19T14:23:10.387 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.148919.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142828.log 2023-10-19T14:23:10.387 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88580.log 2023-10-19T14:23:10.387 INFO:teuthology.orchestra.run.smithi057.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.148919.log.gz 2023-10-19T14:23:10.387 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.147191.log: /var/log/ceph/ceph-client.admin.142828.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121630.log 2023-10-19T14:23:10.387 INFO:teuthology.orchestra.run.smithi057.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.147191.log.gz 2023-10-19T14:23:10.387 INFO:teuthology.orchestra.run.smithi057.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.142828.log.gz 2023-10-19T14:23:10.388 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.88580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161664.log 2023-10-19T14:23:10.388 INFO:teuthology.orchestra.run.smithi057.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.88580.log.gz 2023-10-19T14:23:10.388 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.121630.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157368.log 2023-10-19T14:23:10.388 INFO:teuthology.orchestra.run.smithi057.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.121630.log.gz 2023-10-19T14:23:10.388 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.161664.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112613.log 2023-10-19T14:23:10.389 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.161664.log.gz 2023-10-19T14:23:10.389 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.157368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158234.log 2023-10-19T14:23:10.389 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4%/var/log/ceph/ceph-client.admin.112613.log: -- replaced with /var/log/ceph/ceph-client.admin.157368.log.gz 2023-10-19T14:23:10.389 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132924.log 2023-10-19T14:23:10.389 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.112613.log.gz/var/log/ceph/ceph-client.admin.158234.log: 2023-10-19T14:23:10.389 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129860.log 2023-10-19T14:23:10.390 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.158234.log.gz 2023-10-19T14:23:10.390 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.132924.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104268.log 2023-10-19T14:23:10.390 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.129860.log: 84.9% -- replaced with /var/log/ceph/ceph-client.admin.132924.log.gz 2023-10-19T14:23:10.390 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121420.log 2023-10-19T14:23:10.390 INFO:teuthology.orchestra.run.smithi057.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.129860.log.gz 2023-10-19T14:23:10.391 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.104268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100917.log 2023-10-19T14:23:10.391 INFO:teuthology.orchestra.run.smithi057.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.104268.log.gz 2023-10-19T14:23:10.391 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.121420.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126206.log 2023-10-19T14:23:10.391 INFO:teuthology.orchestra.run.smithi057.stderr: 87.2%/var/log/ceph/ceph-client.admin.100917.log: -- replaced with /var/log/ceph/ceph-client.admin.121420.log.gz 2023-10-19T14:23:10.391 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104394.log 2023-10-19T14:23:10.391 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.100917.log.gz 2023-10-19T14:23:10.392 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.126206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157631.log 2023-10-19T14:23:10.392 INFO:teuthology.orchestra.run.smithi057.stderr:gzip 86.4% -- replaced with /var/log/ceph/ceph-client.admin.126206.log.gz 2023-10-19T14:23:10.392 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.104394.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.113048.log 2023-10-19T14:23:10.392 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.157631.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.104394.log.gz 2023-10-19T14:23:10.392 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132559.log 2023-10-19T14:23:10.393 INFO:teuthology.orchestra.run.smithi057.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.157631.log.gz 2023-10-19T14:23:10.393 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.113048.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163728.log 2023-10-19T14:23:10.393 INFO:teuthology.orchestra.run.smithi057.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.113048.log.gz 2023-10-19T14:23:10.393 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.132559.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103476.log 2023-10-19T14:23:10.393 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95081.log 2023-10-19T14:23:10.393 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.163728.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.132559.log.gz 2023-10-19T14:23:10.394 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.103476.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120188.log 2023-10-19T14:23:10.394 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.163728.log.gz 87.2% 2023-10-19T14:23:10.394 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/ceph-client.admin.103476.log.gz 2023-10-19T14:23:10.394 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.95081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162850.log 2023-10-19T14:23:10.394 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118107.log 2023-10-19T14:23:10.395 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.120188.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.95081.log.gz 2023-10-19T14:23:10.395 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.162850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108095.log 2023-10-19T14:23:10.395 INFO:teuthology.orchestra.run.smithi057.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.120188.log.gz 2023-10-19T14:23:10.395 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.118107.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139979.log 2023-10-19T14:23:10.395 INFO:teuthology.orchestra.run.smithi057.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.162850.log.gz 2023-10-19T14:23:10.395 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.108095.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125519.log 2023-10-19T14:23:10.396 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.118107.log.gz 2023-10-19T14:23:10.396 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152996.log 2023-10-19T14:23:10.396 INFO:teuthology.orchestra.run.smithi057.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.108095.log.gz/var/log/ceph/ceph-client.admin.139979.log: 2023-10-19T14:23:10.396 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.125519.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113342.log 2023-10-19T14:23:10.396 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.139979.log.gz 2023-10-19T14:23:10.397 INFO:teuthology.orchestra.run.smithi057.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.125519.log.gz 2023-10-19T14:23:10.397 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.152996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164523.log 2023-10-19T14:23:10.397 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.152996.log.gz 2023-10-19T14:23:10.397 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.113342.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104566.log 2023-10-19T14:23:10.397 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.113342.log.gz 2023-10-19T14:23:10.397 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.164523.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145351.log 2023-10-19T14:23:10.398 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120904.log 2023-10-19T14:23:10.398 INFO:teuthology.orchestra.run.smithi057.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.164523.log.gz 2023-10-19T14:23:10.398 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.104566.log: /var/log/ceph/ceph-client.admin.145351.log: gzip -5 --verbose 87.4% -- -- replaced with /var/log/ceph/ceph-client.admin.104566.log.gz /var/log/ceph/ceph-osd.1.log 2023-10-19T14:23:10.398 INFO:teuthology.orchestra.run.smithi057.stderr: 2023-10-19T14:23:10.398 INFO:teuthology.orchestra.run.smithi057.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.145351.log.gz 2023-10-19T14:23:10.399 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.120904.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153900.log 2023-10-19T14:23:10.399 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-osd.1.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.120904.log.gz 2023-10-19T14:23:10.399 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157982.log 2023-10-19T14:23:10.399 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.153900.log: -- /var/log/ceph/ceph-client.admin.112655.log 2023-10-19T14:23:10.399 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.157982.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117316.log 2023-10-19T14:23:10.400 INFO:teuthology.orchestra.run.smithi057.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.153900.log.gz 2023-10-19T14:23:10.400 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.112655.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157188.log 2023-10-19T14:23:10.400 INFO:teuthology.orchestra.run.smithi057.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.157982.log.gz 2023-10-19T14:23:10.400 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.117316.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126663.log 2023-10-19T14:23:10.400 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.112655.log.gz 2023-10-19T14:23:10.400 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.157188.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107859.log 2023-10-19T14:23:10.401 INFO:teuthology.orchestra.run.smithi057.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.117316.log.gz 2023-10-19T14:23:10.401 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.126663.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120687.log 2023-10-19T14:23:10.401 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.157188.log.gz 2023-10-19T14:23:10.401 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.107859.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98585.log 2023-10-19T14:23:10.401 INFO:teuthology.orchestra.run.smithi057.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.126663.log.gz 2023-10-19T14:23:10.402 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.120687.log: gzip 86.7% -5 --verbose -- /var/log/ceph/ceph-client.admin.141204.log 2023-10-19T14:23:10.402 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/ceph-client.admin.107859.log.gz 2023-10-19T14:23:10.402 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.98585.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.120687.log.gz 2023-10-19T14:23:10.402 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165576.log 2023-10-19T14:23:10.402 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.141204.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142744.log 2023-10-19T14:23:10.402 INFO:teuthology.orchestra.run.smithi057.stderr: 91.0% 87.3% -- replaced with /var/log/ceph/ceph-client.admin.141204.log.gz 2023-10-19T14:23:10.403 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.165576.log: -- replaced with /var/log/ceph/ceph-client.admin.98585.log.gz 2023-10-19T14:23:10.403 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121462.log 2023-10-19T14:23:10.403 INFO:teuthology.orchestra.run.smithi057.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.165576.log.gz 2023-10-19T14:23:10.403 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.142744.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127342.log 2023-10-19T14:23:10.403 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4%/var/log/ceph/ceph-client.admin.121462.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130738.log 2023-10-19T14:23:10.404 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/ceph-client.admin.142744.log.gz 2023-10-19T14:23:10.404 INFO:teuthology.orchestra.run.smithi057.stderr:gzip/var/log/ceph/ceph-client.admin.127342.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.121462.log.gz 2023-10-19T14:23:10.404 INFO:teuthology.orchestra.run.smithi057.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.100665.log 2023-10-19T14:23:10.404 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.130738.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.127342.log.gz 2023-10-19T14:23:10.404 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160745.log 2023-10-19T14:23:10.404 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.130738.log.gz 2023-10-19T14:23:10.405 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.100665.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97002.log 2023-10-19T14:23:10.405 INFO:teuthology.orchestra.run.smithi057.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.100665.log.gz 2023-10-19T14:23:10.405 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.160745.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146093.log 2023-10-19T14:23:10.405 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5%/var/log/ceph/ceph-client.admin.97002.log: -- replaced with /var/log/ceph/ceph-client.admin.160745.log.gz 2023-10-19T14:23:10.405 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96053.log 2023-10-19T14:23:10.405 INFO:teuthology.orchestra.run.smithi057.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.97002.log.gz 2023-10-19T14:23:10.406 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.146093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154921.log 2023-10-19T14:23:10.406 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.96053.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.132192.log 2023-10-19T14:23:10.406 INFO:teuthology.orchestra.run.smithi057.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.146093.log.gz 2023-10-19T14:23:10.406 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.154921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126416.log 2023-10-19T14:23:10.406 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.96053.log.gz 2023-10-19T14:23:10.407 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118235.log 2023-10-19T14:23:10.407 INFO:teuthology.orchestra.run.smithi057.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.154921.log.gz 2023-10-19T14:23:10.407 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.0.132192.log: /var/log/ceph/ceph-client.admin.126416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113468.log 2023-10-19T14:23:10.407 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.118235.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100035.log 2023-10-19T14:23:10.407 INFO:teuthology.orchestra.run.smithi057.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.126416.log.gz 2023-10-19T14:23:10.408 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5%/var/log/ceph/ceph-client.admin.113468.log: -- replaced with /var/log/ceph/ceph-client.admin.118235.log.gz 2023-10-19T14:23:10.408 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156226.log 2023-10-19T14:23:10.408 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.100035.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163949.log 2023-10-19T14:23:10.408 INFO:teuthology.orchestra.run.smithi057.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.113468.log.gz 2023-10-19T14:23:10.408 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.100035.log.gz/var/log/ceph/ceph-client.admin.156226.log: 2023-10-19T14:23:10.409 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166967.log 2023-10-19T14:23:10.409 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.156226.log.gz 2023-10-19T14:23:10.409 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132349.log 2023-10-19T14:23:10.409 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.163949.log: /var/log/ceph/ceph-client.admin.166967.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129734.log 2023-10-19T14:23:10.409 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.132349.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.163949.log.gz 2023-10-19T14:23:10.409 INFO:teuthology.orchestra.run.smithi057.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.166967.log.gz 2023-10-19T14:23:10.410 INFO:teuthology.orchestra.run.smithi057.stderr:gzip 87.3% -- replaced with /var/log/ceph/ceph-client.admin.132349.log.gz 2023-10-19T14:23:10.410 INFO:teuthology.orchestra.run.smithi057.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.155100.log 2023-10-19T14:23:10.410 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.129734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94743.log 2023-10-19T14:23:10.410 INFO:teuthology.orchestra.run.smithi057.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.129734.log.gz 2023-10-19T14:23:10.410 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.155100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106169.log 2023-10-19T14:23:10.411 INFO:teuthology.orchestra.run.smithi057.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.155100.log.gz 2023-10-19T14:23:10.411 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134102.log 2023-10-19T14:23:10.411 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.94743.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158150.log 2023-10-19T14:23:10.411 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.106169.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.94743.log.gz 2023-10-19T14:23:10.411 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.134102.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.106169.log.gz 2023-10-19T14:23:10.411 INFO:teuthology.orchestra.run.smithi057.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.134102.log.gz 2023-10-19T14:23:10.412 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144190.log 2023-10-19T14:23:10.412 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.158150.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148665.log 2023-10-19T14:23:10.412 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.144190.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161749.log 2023-10-19T14:23:10.412 INFO:teuthology.orchestra.run.smithi057.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.158150.log.gz 2023-10-19T14:23:10.412 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4%/var/log/ceph/ceph-client.admin.148665.log: -- replaced with /var/log/ceph/ceph-client.admin.144190.log.gz 2023-10-19T14:23:10.413 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151586.log 2023-10-19T14:23:10.413 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.161749.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154110.log 2023-10-19T14:23:10.413 INFO:teuthology.orchestra.run.smithi057.stderr: 93.7% -- replaced with /var/log/ceph/ceph-client.0.132192.log.gz 2023-10-19T14:23:10.413 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.148665.log.gz 2023-10-19T14:23:10.413 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.151586.log: 87.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140954.log 2023-10-19T14:23:10.413 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/ceph-client.admin.161749.log.gz 2023-10-19T14:23:10.414 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.151586.log.gz 2023-10-19T14:23:10.414 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.154110.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125603.log 2023-10-19T14:23:10.414 INFO:teuthology.orchestra.run.smithi057.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.154110.log.gz 2023-10-19T14:23:10.414 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94165.log 2023-10-19T14:23:10.414 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.140954.log: /var/log/ceph/ceph-client.admin.125603.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111209.log 2023-10-19T14:23:10.415 INFO:teuthology.orchestra.run.smithi057.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.140954.log.gz/var/log/ceph/ceph-client.admin.94165.log: 2023-10-19T14:23:10.415 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131395.log 2023-10-19T14:23:10.415 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.125603.log.gz 2023-10-19T14:23:10.415 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96821.log 2023-10-19T14:23:10.415 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.94165.log.gz/var/log/ceph/ceph-client.admin.111209.log: 2023-10-19T14:23:10.415 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.131395.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159612.log 2023-10-19T14:23:10.416 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.96821.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94829.log 2023-10-19T14:23:10.416 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% 87.1% -- replaced with /var/log/ceph/ceph-client.admin.131395.log.gz 2023-10-19T14:23:10.416 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/ceph-client.admin.111209.log.gz 2023-10-19T14:23:10.416 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.159612.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.96821.log.gz 2023-10-19T14:23:10.416 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162512.log 87.1% -- replaced with /var/log/ceph/ceph-client.admin.159612.log.gz 2023-10-19T14:23:10.417 INFO:teuthology.orchestra.run.smithi057.stderr: 2023-10-19T14:23:10.417 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.94829.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127048.log 2023-10-19T14:23:10.417 INFO:teuthology.orchestra.run.smithi057.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.94829.log.gz 2023-10-19T14:23:10.417 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.162512.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98081.log 2023-10-19T14:23:10.417 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.127048.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160913.log 2023-10-19T14:23:10.417 INFO:teuthology.orchestra.run.smithi057.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.162512.log.gz 2023-10-19T14:23:10.418 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.127048.log.gz 2023-10-19T14:23:10.418 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133713.log 2023-10-19T14:23:10.418 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.98081.log: /var/log/ceph/ceph-client.admin.160913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117400.log 2023-10-19T14:23:10.418 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.133713.log: gzip -5 --verbose 87.3% -- /var/log/ceph/ceph-client.admin.121672.log -- replaced with /var/log/ceph/ceph-client.admin.160913.log.gz 2023-10-19T14:23:10.418 INFO:teuthology.orchestra.run.smithi057.stderr: 2023-10-19T14:23:10.419 INFO:teuthology.orchestra.run.smithi057.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.98081.log.gz 2023-10-19T14:23:10.419 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.117400.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152180.log 2023-10-19T14:23:10.419 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.133713.log.gz 2023-10-19T14:23:10.419 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.121672.log: gzip 87.4% -- replaced with /var/log/ceph/ceph-client.admin.117400.log.gz 2023-10-19T14:23:10.419 INFO:teuthology.orchestra.run.smithi057.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.103728.log 2023-10-19T14:23:10.420 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.121672.log.gz 2023-10-19T14:23:10.420 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.152180.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96467.log 2023-10-19T14:23:10.420 INFO:teuthology.orchestra.run.smithi057.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.152180.log.gz 2023-10-19T14:23:10.420 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.103728.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128372.log 2023-10-19T14:23:10.420 INFO:teuthology.orchestra.run.smithi057.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.103728.log.gz 2023-10-19T14:23:10.420 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.96467.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.119022.log 2023-10-19T14:23:10.421 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.128372.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.96467.log.gz 2023-10-19T14:23:10.421 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155432.log 2023-10-19T14:23:10.421 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.128372.log.gz 2023-10-19T14:23:10.421 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.0.119022.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140112.log 2023-10-19T14:23:10.421 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.155432.log: gzip -5 --verbose -- 87.1% /var/log/ceph/ceph-client.admin.161352.log 2023-10-19T14:23:10.421 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/ceph-client.admin.155432.log.gz 2023-10-19T14:23:10.422 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.140112.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141754.log 2023-10-19T14:23:10.422 INFO:teuthology.orchestra.run.smithi057.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.140112.log.gz 2023-10-19T14:23:10.422 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.161352.log: gzip -5 --verbose -- 87.5% /var/log/ceph/ceph-client.admin.105992.log 2023-10-19T14:23:10.422 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/ceph-client.admin.161352.log.gz 2023-10-19T14:23:10.422 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.141754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118445.log 91.5% -- replaced with /var/log/ceph/ceph-client.0.119022.log.gz 2023-10-19T14:23:10.423 INFO:teuthology.orchestra.run.smithi057.stderr: 2023-10-19T14:23:10.423 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.105992.log: gzip -5 --verbose -- 91.0% -- replaced with /var/log/ceph/ceph-client.admin.141754.log.gz 2023-10-19T14:23:10.423 INFO:teuthology.orchestra.run.smithi057.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.105992.log.gz 2023-10-19T14:23:10.423 INFO:teuthology.orchestra.run.smithi057.stderr: /var/log/ceph/ceph-client.admin.149694.log 2023-10-19T14:23:10.423 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.118445.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101532.log 2023-10-19T14:23:10.424 INFO:teuthology.orchestra.run.smithi057.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.118445.log.gz 2023-10-19T14:23:10.424 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.149694.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106051.log 2023-10-19T14:23:10.424 INFO:teuthology.orchestra.run.smithi057.stderr:gzip 88.0% -- replaced with /var/log/ceph/ceph-client.admin.149694.log.gz 2023-10-19T14:23:10.424 INFO:teuthology.orchestra.run.smithi057.stderr: -5 --verbose/var/log/ceph/ceph-client.admin.101532.log: -- /var/log/ceph/ceph-client.admin.140196.log 2023-10-19T14:23:10.424 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.106051.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.101532.log.gz 2023-10-19T14:23:10.424 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157716.log 2023-10-19T14:23:10.425 INFO:teuthology.orchestra.run.smithi057.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.106051.log.gz 2023-10-19T14:23:10.425 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.140196.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108213.log 2023-10-19T14:23:10.425 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.140196.log.gz 2023-10-19T14:23:10.425 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158318.log 2023-10-19T14:23:10.425 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.157716.log: /var/log/ceph/ceph-client.admin.108213.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136235.log 2023-10-19T14:23:10.425 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.158318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119925.log 2023-10-19T14:23:10.426 INFO:teuthology.orchestra.run.smithi057.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.157716.log.gz 2023-10-19T14:23:10.426 INFO:teuthology.orchestra.run.smithi057.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.108213.log.gz 2023-10-19T14:23:10.426 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162724.log 2023-10-19T14:23:10.426 INFO:teuthology.orchestra.run.smithi057.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.158318.log.gz 2023-10-19T14:23:10.426 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.136235.log: /var/log/ceph/ceph-client.admin.119925.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.136235.log.gz 2023-10-19T14:23:10.427 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.119925.log.gz 2023-10-19T14:23:10.427 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138496.log 2023-10-19T14:23:10.427 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.162724.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110053.log 2023-10-19T14:23:10.427 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6%/var/log/ceph/ceph-client.admin.138496.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134349.log 2023-10-19T14:23:10.427 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/ceph-client.admin.162724.log.gz 2023-10-19T14:23:10.427 INFO:teuthology.orchestra.run.smithi057.stderr: 87.0%/var/log/ceph/ceph-client.admin.110053.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159400.log 2023-10-19T14:23:10.428 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/ceph-client.admin.138496.log.gz 2023-10-19T14:23:10.428 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.134349.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156707.log 2023-10-19T14:23:10.428 INFO:teuthology.orchestra.run.smithi057.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.110053.log.gz 2023-10-19T14:23:10.428 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130864.log 2023-10-19T14:23:10.428 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/ceph-client.admin.134349.log.gz 2023-10-19T14:23:10.429 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.159400.log: /var/log/ceph/ceph-client.admin.156707.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104184.log 2023-10-19T14:23:10.429 INFO:teuthology.orchestra.run.smithi057.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.159400.log.gz 2023-10-19T14:23:10.429 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.156707.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131615.log 2023-10-19T14:23:10.429 INFO:teuthology.orchestra.run.smithi057.stderr: 2023-10-19T14:23:10.429 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.130864.log: /var/log/ceph/ceph-client.admin.104184.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155516.log 2023-10-19T14:23:10.429 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% 87.4% -- replaced with /var/log/ceph/ceph-client.admin.130864.log.gz 2023-10-19T14:23:10.430 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.131615.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117232.log 2023-10-19T14:23:10.430 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/ceph-client.admin.104184.log.gz 2023-10-19T14:23:10.430 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.131615.log.gz 2023-10-19T14:23:10.430 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.155516.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113510.log 2023-10-19T14:23:10.430 INFO:teuthology.orchestra.run.smithi057.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.155516.log.gz 2023-10-19T14:23:10.431 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151681.log 2023-10-19T14:23:10.431 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.117232.log: /var/log/ceph/ceph-client.admin.113510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148487.log 2023-10-19T14:23:10.431 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135206.log/var/log/ceph/ceph-client.admin.151681.log: 2023-10-19T14:23:10.431 INFO:teuthology.orchestra.run.smithi057.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.117232.log.gz 2023-10-19T14:23:10.431 INFO:teuthology.orchestra.run.smithi057.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.151681.log.gz 2023-10-19T14:23:10.431 INFO:teuthology.orchestra.run.smithi057.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.113510.log.gz 2023-10-19T14:23:10.432 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.148487.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152486.log 2023-10-19T14:23:10.432 INFO:teuthology.orchestra.run.smithi057.stderr: 93.9% -- replaced with /var/log/ceph/ceph-client.0.105539.log.gz 2023-10-19T14:23:10.432 INFO:teuthology.orchestra.run.smithi057.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.148487.log.gz 2023-10-19T14:23:10.432 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146954.log 2023-10-19T14:23:10.432 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.135206.log: /var/log/ceph/ceph-client.admin.152486.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119329.log 2023-10-19T14:23:10.433 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.135206.log.gz 2023-10-19T14:23:10.433 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 87.3% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.152486.log.gz /var/log/ceph/ceph-client.admin.112962.log 2023-10-19T14:23:10.433 INFO:teuthology.orchestra.run.smithi057.stderr: 2023-10-19T14:23:10.433 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.146954.log: /var/log/ceph/ceph-client.admin.119329.log: gzip 87.5% -- replaced with /var/log/ceph/ceph-client.admin.146954.log.gz -5 2023-10-19T14:23:10.433 INFO:teuthology.orchestra.run.smithi057.stderr: --verbose -- /var/log/ceph/ceph-client.admin.118319.log 2023-10-19T14:23:10.434 INFO:teuthology.orchestra.run.smithi057.stderr: 87.1%/var/log/ceph/ceph-client.admin.112962.log: -- replaced with /var/log/ceph/ceph-client.admin.119329.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162426.log 2023-10-19T14:23:10.434 INFO:teuthology.orchestra.run.smithi057.stderr: 2023-10-19T14:23:10.434 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.112962.log.gz 2023-10-19T14:23:10.434 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.118319.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149527.log 2023-10-19T14:23:10.434 INFO:teuthology.orchestra.run.smithi057.stderr: 87.7%/var/log/ceph/ceph-client.admin.162426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124628.log 2023-10-19T14:23:10.434 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/ceph-client.admin.118319.log.gz 2023-10-19T14:23:10.435 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.149527.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.162426.log.gz 2023-10-19T14:23:10.435 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117526.log 2023-10-19T14:23:10.435 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.149527.log.gz 2023-10-19T14:23:10.435 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.124628.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140737.log 2023-10-19T14:23:10.435 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104478.log 2023-10-19T14:23:10.436 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.117526.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.124628.log.gz 2023-10-19T14:23:10.436 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.140737.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140677.log 2023-10-19T14:23:10.436 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.117526.log.gz 2023-10-19T14:23:10.436 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120519.log 2023-10-19T14:23:10.436 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/ceph-client.admin.140737.log.gz 2023-10-19T14:23:10.436 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.104478.log: /var/log/ceph/ceph-client.admin.140677.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142499.log 2023-10-19T14:23:10.437 INFO:teuthology.orchestra.run.smithi057.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.104478.log.gz 2023-10-19T14:23:10.437 INFO:teuthology.orchestra.run.smithi057.stderr: 87.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.140677.log.gz -5 2023-10-19T14:23:10.437 INFO:teuthology.orchestra.run.smithi057.stderr: --verbose -- /var/log/ceph/ceph-client.admin.131139.log 2023-10-19T14:23:10.437 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.120519.log: /var/log/ceph/ceph-client.admin.142499.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.120519.log.gz 2023-10-19T14:23:10.437 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154820.log 2023-10-19T14:23:10.438 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.131139.log: gzip -5 87.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.142499.log.gz 2023-10-19T14:23:10.438 INFO:teuthology.orchestra.run.smithi057.stderr: /var/log/ceph/ceph-client.admin.95123.log 2023-10-19T14:23:10.438 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.131139.log.gz 2023-10-19T14:23:10.438 INFO:teuthology.orchestra.run.smithi057.stderr:gzip/var/log/ceph/ceph-client.admin.154820.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.127256.log 2023-10-19T14:23:10.438 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108036.log 2023-10-19T14:23:10.438 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.95123.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.154820.log.gz 2023-10-19T14:23:10.439 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.127256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145588.log 2023-10-19T14:23:10.439 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6%/var/log/ceph/ceph-client.admin.108036.log: -- replaced with /var/log/ceph/ceph-client.admin.127256.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155184.log 2023-10-19T14:23:10.439 INFO:teuthology.orchestra.run.smithi057.stderr: 2023-10-19T14:23:10.439 INFO:teuthology.orchestra.run.smithi057.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.95123.log.gz 2023-10-19T14:23:10.439 INFO:teuthology.orchestra.run.smithi057.stderr: 86.8%/var/log/ceph/ceph-client.admin.145588.log: -- replaced with /var/log/ceph/ceph-client.admin.108036.log.gz 2023-10-19T14:23:10.440 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149069.log 2023-10-19T14:23:10.440 INFO:teuthology.orchestra.run.smithi057.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.145588.log.gz 2023-10-19T14:23:10.440 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100833.log 2023-10-19T14:23:10.440 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.155184.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94529.log 2023-10-19T14:23:10.440 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.149069.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.155184.log.gz 2023-10-19T14:23:10.441 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.100833.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157854.log 2023-10-19T14:23:10.441 INFO:teuthology.orchestra.run.smithi057.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.149069.log.gz 2023-10-19T14:23:10.441 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.94529.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.100833.log.gz 2023-10-19T14:23:10.441 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148257.log 2023-10-19T14:23:10.441 INFO:teuthology.orchestra.run.smithi057.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.94529.log.gz 2023-10-19T14:23:10.441 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.157854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162976.log 2023-10-19T14:23:10.442 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116877.log 2023-10-19T14:23:10.442 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.157854.log.gz/var/log/ceph/ceph-client.admin.148257.log: 2023-10-19T14:23:10.442 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.162976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131445.log 2023-10-19T14:23:10.442 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.148257.log.gz 2023-10-19T14:23:10.442 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.116877.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.162976.log.gzgzip 2023-10-19T14:23:10.443 INFO:teuthology.orchestra.run.smithi057.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.141118.log 2023-10-19T14:23:10.443 INFO:teuthology.orchestra.run.smithi057.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.116877.log.gz 2023-10-19T14:23:10.443 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.131445.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143370.log 2023-10-19T14:23:10.443 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165057.log 2023-10-19T14:23:10.443 INFO:teuthology.orchestra.run.smithi057.stderr: 87.8%/var/log/ceph/ceph-client.admin.141118.log: -- replaced with /var/log/ceph/ceph-client.admin.131445.log.gz 2023-10-19T14:23:10.443 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.143370.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141418.log 2023-10-19T14:23:10.444 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.141118.log.gz 2023-10-19T14:23:10.444 INFO:teuthology.orchestra.run.smithi057.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.143370.log.gz 2023-10-19T14:23:10.444 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.165057.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125790.log 2023-10-19T14:23:10.444 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.141418.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.165057.log.gz 2023-10-19T14:23:10.444 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162640.log 2023-10-19T14:23:10.445 INFO:teuthology.orchestra.run.smithi057.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.141418.log.gz 2023-10-19T14:23:10.445 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.125790.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103350.log 2023-10-19T14:23:10.445 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121334.log 2023-10-19T14:23:10.445 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.162640.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.125790.log.gz 2023-10-19T14:23:10.445 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.103350.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166080.log 2023-10-19T14:23:10.445 INFO:teuthology.orchestra.run.smithi057.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.162640.log.gz 2023-10-19T14:23:10.446 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6%/var/log/ceph/ceph-client.admin.121334.log: -- replaced with /var/log/ceph/ceph-client.admin.103350.log.gz 2023-10-19T14:23:10.446 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123559.log 2023-10-19T14:23:10.446 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.121334.log.gz 2023-10-19T14:23:10.446 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.166080.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158926.log 2023-10-19T14:23:10.446 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.123559.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.166080.log.gz 2023-10-19T14:23:10.447 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156142.log 2023-10-19T14:23:10.447 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.123559.log.gz 2023-10-19T14:23:10.447 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.158926.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141034.log 2023-10-19T14:23:10.447 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.158926.log.gz 2023-10-19T14:23:10.447 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.156142.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153124.log 2023-10-19T14:23:10.447 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98123.log 2023-10-19T14:23:10.448 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.141034.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.156142.log.gz 2023-10-19T14:23:10.448 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.153124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101264.log 2023-10-19T14:23:10.448 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.141034.log.gz 2023-10-19T14:23:10.448 INFO:teuthology.orchestra.run.smithi057.stderr: 87.3%/var/log/ceph/ceph-client.admin.98123.log: -- replaced with /var/log/ceph/ceph-client.admin.153124.log.gz 2023-10-19T14:23:10.448 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98543.log 2023-10-19T14:23:10.449 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.101264.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.98123.log.gz 2023-10-19T14:23:10.449 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165660.log 2023-10-19T14:23:10.449 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.101264.log.gz 2023-10-19T14:23:10.449 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.98543.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141460.log 2023-10-19T14:23:10.449 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.165660.log: gzip 87.3% -5 -- replaced with /var/log/ceph/ceph-client.admin.98543.log.gz --verbose 2023-10-19T14:23:10.449 INFO:teuthology.orchestra.run.smithi057.stderr: -- /var/log/ceph/ceph-client.admin.162468.log 2023-10-19T14:23:10.450 INFO:teuthology.orchestra.run.smithi057.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.165660.log.gz 2023-10-19T14:23:10.450 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.141460.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144785.log 2023-10-19T14:23:10.450 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.162468.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.141460.log.gz 2023-10-19T14:23:10.450 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133289.log 2023-10-19T14:23:10.450 INFO:teuthology.orchestra.run.smithi057.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.162468.log.gz 2023-10-19T14:23:10.451 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.144785.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141376.log 2023-10-19T14:23:10.451 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.144785.log.gz 2023-10-19T14:23:10.451 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.133289.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142278.log 2023-10-19T14:23:10.451 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148401.log 2023-10-19T14:23:10.451 INFO:teuthology.orchestra.run.smithi057.stderr: 87.7%/var/log/ceph/ceph-client.admin.141376.log: -- replaced with /var/log/ceph/ceph-client.admin.133289.log.gz 2023-10-19T14:23:10.451 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.142278.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156539.log 2023-10-19T14:23:10.452 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.141376.log.gz 2023-10-19T14:23:10.452 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.142278.log.gz 2023-10-19T14:23:10.452 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.148401.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104692.log 2023-10-19T14:23:10.452 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88380.log 2023-10-19T14:23:10.452 INFO:teuthology.orchestra.run.smithi057.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.148401.log.gz 2023-10-19T14:23:10.453 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.156539.log: /var/log/ceph/ceph-client.admin.104692.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113132.log 2023-10-19T14:23:10.453 INFO:teuthology.orchestra.run.smithi057.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.156539.log.gz 2023-10-19T14:23:10.453 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.88380.log: gzip -5 --verbose 87.5% -- /var/log/ceph/ceph-client.admin.118057.log -- replaced with /var/log/ceph/ceph-client.admin.104692.log.gz 2023-10-19T14:23:10.453 INFO:teuthology.orchestra.run.smithi057.stderr: 2023-10-19T14:23:10.453 INFO:teuthology.orchestra.run.smithi057.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.88380.log.gz 2023-10-19T14:23:10.453 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.113132.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119562.log 2023-10-19T14:23:10.454 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99221.log 2023-10-19T14:23:10.454 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.113132.log.gz 2023-10-19T14:23:10.454 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.118057.log: /var/log/ceph/ceph-client.admin.119562.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106800.log 2023-10-19T14:23:10.454 INFO:teuthology.orchestra.run.smithi057.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.118057.log.gz 2023-10-19T14:23:10.454 INFO:teuthology.orchestra.run.smithi057.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.119562.log.gz 2023-10-19T14:23:10.455 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.99221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132965.log 2023-10-19T14:23:10.455 INFO:teuthology.orchestra.run.smithi057.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.99221.log.gz 2023-10-19T14:23:10.455 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.106800.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94997.log 2023-10-19T14:23:10.455 INFO:teuthology.orchestra.run.smithi057.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.106800.log.gz 2023-10-19T14:23:10.455 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.132965.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156455.log 2023-10-19T14:23:10.455 INFO:teuthology.orchestra.run.smithi057.stderr: 84.0% -- replaced with /var/log/ceph/ceph-client.admin.132965.log.gzgzip 2023-10-19T14:23:10.456 INFO:teuthology.orchestra.run.smithi057.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.152816.log 2023-10-19T14:23:10.456 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.94997.log: /var/log/ceph/ceph-client.admin.156455.log: gzip -5 87.5% -- replaced with /var/log/ceph/ceph-client.admin.94997.log.gz 2023-10-19T14:23:10.456 INFO:teuthology.orchestra.run.smithi057.stderr: --verbose -- /var/log/ceph/ceph-client.admin.161833.log 2023-10-19T14:23:10.456 INFO:teuthology.orchestra.run.smithi057.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.156455.log.gz 2023-10-19T14:23:10.456 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.152816.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154198.log 2023-10-19T14:23:10.457 INFO:teuthology.orchestra.run.smithi057.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.152816.log.gz 2023-10-19T14:23:10.457 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96995.log 2023-10-19T14:23:10.457 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.161833.log: /var/log/ceph/ceph-client.admin.154198.log: gzip -5 --verbose 87.2% -- replaced with /var/log/ceph/ceph-client.admin.161833.log.gz 2023-10-19T14:23:10.457 INFO:teuthology.orchestra.run.smithi057.stderr: -- /var/log/ceph/ceph-osd.3.log 2023-10-19T14:23:10.457 INFO:teuthology.orchestra.run.smithi057.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.154198.log.gz 2023-10-19T14:23:10.457 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.96995.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95827.log 2023-10-19T14:23:10.458 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.96995.log.gz 2023-10-19T14:23:10.458 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156497.log 2023-10-19T14:23:10.458 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-osd.3.log: /var/log/ceph/ceph-client.admin.95827.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143580.log 2023-10-19T14:23:10.458 INFO:teuthology.orchestra.run.smithi057.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.95827.log.gz 2023-10-19T14:23:10.458 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.156497.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124240.log 2023-10-19T14:23:10.459 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.156497.log.gz 2023-10-19T14:23:10.459 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.143580.log: /var/log/ceph/ceph-client.admin.89259.log 2023-10-19T14:23:10.459 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.124240.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.143580.log.gz 2023-10-19T14:23:10.459 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 87.1% -- replaced with /var/log/ceph/ceph-client.admin.124240.log.gz 2023-10-19T14:23:10.459 INFO:teuthology.orchestra.run.smithi057.stderr: --verbose -- /var/log/ceph/ceph-client.admin.161917.log 2023-10-19T14:23:10.459 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.89259.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88351.log 2023-10-19T14:23:10.460 INFO:teuthology.orchestra.run.smithi057.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.89259.log.gz 2023-10-19T14:23:10.460 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.161917.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156184.log 2023-10-19T14:23:10.460 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.161917.log.gz 2023-10-19T14:23:10.460 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96551.log 2023-10-19T14:23:10.460 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.88351.log: 79.3%/var/log/ceph/ceph-client.admin.156184.log: -- replaced with /var/log/ceph/ceph-client.admin.88351.log.gz 2023-10-19T14:23:10.461 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152264.log 2023-10-19T14:23:10.461 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157589.log 2023-10-19T14:23:10.461 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/ceph-client.admin.156184.log.gz 2023-10-19T14:23:10.461 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.96551.log: /var/log/ceph/ceph-client.admin.152264.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156665.log 2023-10-19T14:23:10.461 INFO:teuthology.orchestra.run.smithi057.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.96551.log.gz 2023-10-19T14:23:10.461 INFO:teuthology.orchestra.run.smithi057.stderr: 87.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94084.log 2023-10-19T14:23:10.462 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/ceph-client.admin.152264.log.gz 2023-10-19T14:23:10.462 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.157589.log: gzip -5 --verbose/var/log/ceph/ceph-client.admin.156665.log: -- /var/log/ceph/ceph-client.admin.162341.log 2023-10-19T14:23:10.462 INFO:teuthology.orchestra.run.smithi057.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.157589.log.gz 2023-10-19T14:23:10.462 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.94084.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.156665.log.gz 2023-10-19T14:23:10.462 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148793.log 2023-10-19T14:23:10.463 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.94084.log.gz 2023-10-19T14:23:10.463 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.162341.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131783.log 2023-10-19T14:23:10.463 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.162341.log.gz 2023-10-19T14:23:10.463 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.148793.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143160.log 2023-10-19T14:23:10.463 INFO:teuthology.orchestra.run.smithi057.stderr:gzip 87.7% -- replaced with /var/log/ceph/ceph-client.admin.148793.log.gz 2023-10-19T14:23:10.463 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.131783.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.111625.log 2023-10-19T14:23:10.464 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.143160.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.131783.log.gz 2023-10-19T14:23:10.464 INFO:teuthology.orchestra.run.smithi057.stderr:gzip 86.5% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.143160.log.gz -- 2023-10-19T14:23:10.464 INFO:teuthology.orchestra.run.smithi057.stderr: /var/log/ceph/ceph-client.admin.156623.log 2023-10-19T14:23:10.464 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.111625.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147149.log 2023-10-19T14:23:10.464 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.156623.log: 86.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102206.log 2023-10-19T14:23:10.465 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/ceph-client.admin.111625.log.gz 2023-10-19T14:23:10.465 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4%/var/log/ceph/ceph-client.admin.147149.log: -- replaced with /var/log/ceph/ceph-client.admin.156623.log.gz 2023-10-19T14:23:10.465 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166799.log 2023-10-19T14:23:10.465 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.147149.log.gz 2023-10-19T14:23:10.465 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.102206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104522.log 2023-10-19T14:23:10.465 INFO:teuthology.orchestra.run.smithi057.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.102206.log.gz 2023-10-19T14:23:10.466 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.166799.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123601.log 2023-10-19T14:23:10.466 INFO:teuthology.orchestra.run.smithi057.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.166799.log.gz 2023-10-19T14:23:10.466 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.104522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102006.log 2023-10-19T14:23:10.466 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.123601.log: gzip 87.0% -- replaced with /var/log/ceph/ceph-client.admin.104522.log.gz 2023-10-19T14:23:10.466 INFO:teuthology.orchestra.run.smithi057.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.143023.log 2023-10-19T14:23:10.467 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.123601.log.gz 2023-10-19T14:23:10.467 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.102006.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127806.log 2023-10-19T14:23:10.467 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.102006.log.gz 2023-10-19T14:23:10.467 INFO:teuthology.orchestra.run.smithi057.stderr:gzip/var/log/ceph/ceph-client.admin.143023.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.123415.log 2023-10-19T14:23:10.467 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.127806.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.143023.log.gz 2023-10-19T14:23:10.468 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126705.log 2023-10-19T14:23:10.468 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.127806.log.gz 2023-10-19T14:23:10.468 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.123415.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103056.log 2023-10-19T14:23:10.468 INFO:teuthology.orchestra.run.smithi057.stderr:gzip 87.2% -5 --verbose -- /var/log/ceph/ceph-client.admin.102048.log 2023-10-19T14:23:10.468 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.126705.log: -- replaced with /var/log/ceph/ceph-client.admin.123415.log.gz 2023-10-19T14:23:10.468 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.103056.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88662.log 2023-10-19T14:23:10.469 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.126705.log.gz 2023-10-19T14:23:10.469 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.103056.log.gz 2023-10-19T14:23:10.469 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116919.log 2023-10-19T14:23:10.469 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.102048.log: gzip -5 87.4% -- replaced with /var/log/ceph/ceph-client.admin.102048.log.gz 2023-10-19T14:23:10.469 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.88662.log: --verbose -- /var/log/ceph/ceph-client.admin.127848.log 2023-10-19T14:23:10.470 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.116919.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.88662.log.gz 2023-10-19T14:23:10.470 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose 87.6% -- -- replaced with /var/log/ceph/ceph-client.admin.116919.log.gz 2023-10-19T14:23:10.470 INFO:teuthology.orchestra.run.smithi057.stderr: /var/log/ceph/ceph-client.admin.130528.log 2023-10-19T14:23:10.470 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.127848.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155806.log 2023-10-19T14:23:10.470 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.130528.log: gzip -5 90.9% -- replaced with /var/log/ceph/ceph-client.admin.127848.log.gz 2023-10-19T14:23:10.470 INFO:teuthology.orchestra.run.smithi057.stderr: --verbose -- /var/log/ceph/ceph-client.admin.116163.log 95.6% -- replaced with /var/log/ceph/ceph-client.0.135869.log.gz 2023-10-19T14:23:10.471 INFO:teuthology.orchestra.run.smithi057.stderr: 2023-10-19T14:23:10.471 INFO:teuthology.orchestra.run.smithi057.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.130528.log.gz 2023-10-19T14:23:10.471 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.155806.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130959.log 2023-10-19T14:23:10.471 INFO:teuthology.orchestra.run.smithi057.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.155806.log.gz 2023-10-19T14:23:10.471 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.116163.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126290.log 2023-10-19T14:23:10.472 INFO:teuthology.orchestra.run.smithi057.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.116163.log.gz 2023-10-19T14:23:10.472 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.130959.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.130959.log.gz 2023-10-19T14:23:10.472 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165523.log 2023-10-19T14:23:10.472 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.126290.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134734.log 2023-10-19T14:23:10.472 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.126290.log.gz 2023-10-19T14:23:10.472 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.165523.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131529.log 2023-10-19T14:23:10.473 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162808.log 2023-10-19T14:23:10.473 INFO:teuthology.orchestra.run.smithi057.stderr: 86.7%/var/log/ceph/ceph-client.admin.134734.log: -- replaced with /var/log/ceph/ceph-client.admin.165523.log.gz 2023-10-19T14:23:10.473 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.131529.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.134734.log.gz 2023-10-19T14:23:10.473 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97001.log 2023-10-19T14:23:10.473 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127470.log 2023-10-19T14:23:10.474 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.131529.log.gz 2023-10-19T14:23:10.474 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.162808.log: /var/log/ceph/ceph-client.admin.97001.log: 87.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148709.log 2023-10-19T14:23:10.474 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/ceph-client.admin.162808.log.gz 2023-10-19T14:23:10.474 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.97001.log.gz 2023-10-19T14:23:10.474 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.127470.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.167175.log 2023-10-19T14:23:10.474 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.148709.log: -- /var/log/ceph/ceph-client.admin.139891.log 2023-10-19T14:23:10.475 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.127470.log.gz 2023-10-19T14:23:10.475 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.167175.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.148709.log.gzgzip 2023-10-19T14:23:10.475 INFO:teuthology.orchestra.run.smithi057.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.121242.log 2023-10-19T14:23:10.475 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.167175.log.gz 2023-10-19T14:23:10.475 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.139891.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101964.log 2023-10-19T14:23:10.476 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.139891.log.gz 2023-10-19T14:23:10.476 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158402.log 2023-10-19T14:23:10.476 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.121242.log: /var/log/ceph/ceph-client.admin.101964.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112105.log 2023-10-19T14:23:10.476 INFO:teuthology.orchestra.run.smithi057.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.121242.log.gz 2023-10-19T14:23:10.476 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.101964.log.gz 2023-10-19T14:23:10.476 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.158402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106674.log 2023-10-19T14:23:10.477 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159094.log 2023-10-19T14:23:10.477 INFO:teuthology.orchestra.run.smithi057.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.158402.log.gz 2023-10-19T14:23:10.477 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.112105.log: /var/log/ceph/ceph-client.admin.106674.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.112105.log.gzgzip 2023-10-19T14:23:10.477 INFO:teuthology.orchestra.run.smithi057.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.147496.log 2023-10-19T14:23:10.477 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.159094.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.106674.log.gz 2023-10-19T14:23:10.478 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104734.log 2023-10-19T14:23:10.478 INFO:teuthology.orchestra.run.smithi057.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.159094.log.gz 2023-10-19T14:23:10.478 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.147496.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157896.log 2023-10-19T14:23:10.478 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.147496.log.gz 2023-10-19T14:23:10.478 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.113835.log 2023-10-19T14:23:10.478 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.104734.log: /var/log/ceph/ceph-client.admin.157896.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105696.log 2023-10-19T14:23:10.479 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.104734.log.gz 2023-10-19T14:23:10.479 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.0.113835.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.157896.log.gz 2023-10-19T14:23:10.479 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106506.log 2023-10-19T14:23:10.479 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.105696.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102332.log 2023-10-19T14:23:10.479 INFO:teuthology.orchestra.run.smithi057.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.105696.log.gz 2023-10-19T14:23:10.480 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.106506.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105130.log 2023-10-19T14:23:10.480 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.102332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96341.log 86.8% 2023-10-19T14:23:10.480 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/ceph-client.admin.106506.log.gz 2023-10-19T14:23:10.480 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141670.log 2023-10-19T14:23:10.480 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.105130.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.102332.log.gz 2023-10-19T14:23:10.480 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.105130.log.gz 2023-10-19T14:23:10.481 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130654.log 2023-10-19T14:23:10.481 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.96341.log: gzip -5/var/log/ceph/ceph-client.admin.141670.log: --verbose -- /var/log/ceph/ceph-client.admin.122253.log 2023-10-19T14:23:10.481 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.96341.log.gz 2023-10-19T14:23:10.481 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.130654.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.141670.log.gz 2023-10-19T14:23:10.481 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135610.log 2023-10-19T14:23:10.482 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.130654.log.gz 2023-10-19T14:23:10.482 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.122253.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130150.log 2023-10-19T14:23:10.482 INFO:teuthology.orchestra.run.smithi057.stderr:gzip 87.6% -- replaced with /var/log/ceph/ceph-client.admin.122253.log.gz 2023-10-19T14:23:10.482 INFO:teuthology.orchestra.run.smithi057.stderr: -5/var/log/ceph/ceph-client.admin.135610.log: --verbose -- /var/log/ceph/ceph-client.admin.120020.log 2023-10-19T14:23:10.482 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.130150.log: gzip 87.7% -- replaced with /var/log/ceph/ceph-client.admin.135610.log.gz 2023-10-19T14:23:10.482 INFO:teuthology.orchestra.run.smithi057.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.122403.log 87.1% -- replaced with /var/log/ceph/ceph-client.admin.130150.log.gz 2023-10-19T14:23:10.483 INFO:teuthology.orchestra.run.smithi057.stderr: 2023-10-19T14:23:10.483 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.120020.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117147.log 2023-10-19T14:23:10.483 INFO:teuthology.orchestra.run.smithi057.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.120020.log.gz 2023-10-19T14:23:10.483 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.122403.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106464.log 2023-10-19T14:23:10.483 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.117147.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147622.log 2023-10-19T14:23:10.484 INFO:teuthology.orchestra.run.smithi057.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.122403.log.gz 2023-10-19T14:23:10.484 INFO:teuthology.orchestra.run.smithi057.stderr: 87.3%/var/log/ceph/ceph-client.admin.106464.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154628.log 2023-10-19T14:23:10.484 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/ceph-client.admin.117147.log.gz 2023-10-19T14:23:10.484 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.147622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122455.log 2023-10-19T14:23:10.484 INFO:teuthology.orchestra.run.smithi057.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.106464.log.gz 2023-10-19T14:23:10.484 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.154628.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165328.log 2023-10-19T14:23:10.485 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.147622.log.gz 2023-10-19T14:23:10.485 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.122455.log: /var/log/ceph/ceph-client.admin.88229.log 2023-10-19T14:23:10.485 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.165328.log: 87.0% 91.1% -- replaced with /var/log/ceph/ceph-client.admin.154628.log.gz 2023-10-19T14:23:10.485 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97329.log 2023-10-19T14:23:10.485 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/ceph-client.admin.122455.log.gz 2023-10-19T14:23:10.486 INFO:teuthology.orchestra.run.smithi057.stderr:gzip 87.2% -5 -- replaced with /var/log/ceph/ceph-client.admin.165328.log.gz --verbose 2023-10-19T14:23:10.486 INFO:teuthology.orchestra.run.smithi057.stderr: -- /var/log/ceph/ceph-client.admin.111979.log 2023-10-19T14:23:10.486 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.88229.log: gzip 76.7% -- replaced with /var/log/ceph/ceph-client.admin.88229.log.gz 2023-10-19T14:23:10.486 INFO:teuthology.orchestra.run.smithi057.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.133934.log 2023-10-19T14:23:10.486 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.97329.log: /var/log/ceph/ceph-client.admin.111979.log: gzip 87.1% -- replaced with /var/log/ceph/ceph-client.admin.97329.log.gz 2023-10-19T14:23:10.487 INFO:teuthology.orchestra.run.smithi057.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.160871.log 2023-10-19T14:23:10.487 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.111979.log.gz 2023-10-19T14:23:10.487 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.133934.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111488.log 2023-10-19T14:23:10.487 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5%/var/log/ceph/ceph-client.admin.160871.log: -- replaced with /var/log/ceph/ceph-client.admin.133934.log.gz 2023-10-19T14:23:10.487 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103770.log 2023-10-19T14:23:10.487 INFO:teuthology.orchestra.run.smithi057.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.160871.log.gz 2023-10-19T14:23:10.488 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.111488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160997.log 2023-10-19T14:23:10.488 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6%/var/log/ceph/ceph-client.admin.103770.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146870.log 2023-10-19T14:23:10.488 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/ceph-client.admin.111488.log.gz 2023-10-19T14:23:10.488 INFO:teuthology.orchestra.run.smithi057.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.103770.log.gz 2023-10-19T14:23:10.488 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.160997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161875.log 2023-10-19T14:23:10.488 INFO:teuthology.orchestra.run.smithi057.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.160997.log.gz 2023-10-19T14:23:10.489 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130486.log 2023-10-19T14:23:10.489 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.146870.log: /var/log/ceph/ceph-client.admin.161875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149589.log 2023-10-19T14:23:10.489 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.161875.log.gz 87.1% -- replaced with /var/log/ceph/ceph-client.admin.146870.log.gz 2023-10-19T14:23:10.489 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.130486.log: 2023-10-19T14:23:10.489 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94384.log 2023-10-19T14:23:10.490 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.130486.log.gz 2023-10-19T14:23:10.490 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.149589.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109493.log 2023-10-19T14:23:10.490 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.94384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160161.log 2023-10-19T14:23:10.490 INFO:teuthology.orchestra.run.smithi057.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.149589.log.gz 2023-10-19T14:23:10.490 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.94384.log.gz 2023-10-19T14:23:10.491 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103140.log 2023-10-19T14:23:10.491 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.109493.log: gzip/var/log/ceph/ceph-client.admin.160161.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.159804.log 87.5% -- replaced with /var/log/ceph/ceph-client.admin.109493.log.gz 2023-10-19T14:23:10.491 INFO:teuthology.orchestra.run.smithi057.stderr: 2023-10-19T14:23:10.491 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.103140.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.160161.log.gz 2023-10-19T14:23:10.491 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 87.5% -- replaced with /var/log/ceph/ceph-client.admin.103140.log.gz 2023-10-19T14:23:10.491 INFO:teuthology.orchestra.run.smithi057.stderr: --verbose -- /var/log/ceph/ceph-client.admin.166337.log 2023-10-19T14:23:10.492 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.159804.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142786.log 2023-10-19T14:23:10.492 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.159804.log.gz 2023-10-19T14:23:10.492 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.166337.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133892.log 2023-10-19T14:23:10.492 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.142786.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147832.log 2023-10-19T14:23:10.492 INFO:teuthology.orchestra.run.smithi057.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.166337.log.gz 2023-10-19T14:23:10.493 INFO:teuthology.orchestra.run.smithi057.stderr:gzip/var/log/ceph/ceph-client.admin.133892.log: -5 --verbose 87.1% -- replaced with /var/log/ceph/ceph-client.admin.142786.log.gz 2023-10-19T14:23:10.493 INFO:teuthology.orchestra.run.smithi057.stderr: -- /var/log/ceph/ceph-client.admin.100287.log 2023-10-19T14:23:10.493 INFO:teuthology.orchestra.run.smithi057.stderr: 86.5%/var/log/ceph/ceph-client.admin.147832.log: -- replaced with /var/log/ceph/ceph-client.admin.133892.log.gz 2023-10-19T14:23:10.493 INFO:teuthology.orchestra.run.smithi057.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.147832.log.gz 2023-10-19T14:23:10.493 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103098.log 2023-10-19T14:23:10.493 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108331.log 2023-10-19T14:23:10.494 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.100287.log: /var/log/ceph/ceph-client.admin.103098.log: gzip -5 87.1% -- replaced with /var/log/ceph/ceph-client.admin.100287.log.gz 2023-10-19T14:23:10.494 INFO:teuthology.orchestra.run.smithi057.stderr: --verbose -- /var/log/ceph/ceph-client.admin.125299.log 2023-10-19T14:23:10.494 INFO:teuthology.orchestra.run.smithi057.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.103098.log.gz 2023-10-19T14:23:10.494 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.108331.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154154.log 2023-10-19T14:23:10.494 INFO:teuthology.orchestra.run.smithi057.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.108331.log.gz 2023-10-19T14:23:10.495 INFO:teuthology.orchestra.run.smithi057.stderr:gzip/var/log/ceph/ceph-client.admin.125299.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.165286.log 2023-10-19T14:23:10.495 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.154154.log: gzip 87.0% -- replaced with /var/log/ceph/ceph-client.admin.125299.log.gz 2023-10-19T14:23:10.495 INFO:teuthology.orchestra.run.smithi057.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.140874.log 2023-10-19T14:23:10.495 INFO:teuthology.orchestra.run.smithi057.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.154154.log.gz 2023-10-19T14:23:10.495 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.165286.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88666.log 2023-10-19T14:23:10.495 INFO:teuthology.orchestra.run.smithi057.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.165286.log.gz 2023-10-19T14:23:10.496 INFO:teuthology.orchestra.run.smithi057.stderr:gzip/var/log/ceph/ceph-client.admin.140874.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.100749.log 2023-10-19T14:23:10.496 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.88666.log: gzip -5 87.2% -- replaced with /var/log/ceph/ceph-client.admin.140874.log.gz 2023-10-19T14:23:10.496 INFO:teuthology.orchestra.run.smithi057.stderr: --verbose -- /var/log/ceph/ceph-client.admin.88257.log 2023-10-19T14:23:10.496 INFO:teuthology.orchestra.run.smithi057.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.88666.log.gz 2023-10-19T14:23:10.496 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.100749.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135376.log 2023-10-19T14:23:10.497 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.100749.log.gz 2023-10-19T14:23:10.497 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5/var/log/ceph/ceph-client.admin.88257.log: --verbose -- /var/log/ceph/ceph-client.admin.162043.log 2023-10-19T14:23:10.497 INFO:teuthology.orchestra.run.smithi057.stderr: 75.9% -- replaced with /var/log/ceph/ceph-client.admin.88257.log.gz 2023-10-19T14:23:10.497 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.135376.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155286.log 2023-10-19T14:23:10.497 INFO:teuthology.orchestra.run.smithi057.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.135376.log.gz 2023-10-19T14:23:10.497 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.162043.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136330.log 2023-10-19T14:23:10.498 INFO:teuthology.orchestra.run.smithi057.stderr:gzip 87.0% -- replaced with /var/log/ceph/ceph-client.admin.162043.log.gz 2023-10-19T14:23:10.498 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.155286.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.166841.log 2023-10-19T14:23:10.498 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.136330.log: gzip -5 87.4% -- replaced with /var/log/ceph/ceph-client.admin.155286.log.gz 2023-10-19T14:23:10.498 INFO:teuthology.orchestra.run.smithi057.stderr: --verbose -- /var/log/ceph/ceph-client.admin.98417.log 2023-10-19T14:23:10.498 INFO:teuthology.orchestra.run.smithi057.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.136330.log.gz 2023-10-19T14:23:10.499 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.166841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159178.log 2023-10-19T14:23:10.499 INFO:teuthology.orchestra.run.smithi057.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.166841.log.gz 2023-10-19T14:23:10.499 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.98417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148299.log 2023-10-19T14:23:10.499 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.98417.log.gz/var/log/ceph/ceph-client.admin.159178.log: 2023-10-19T14:23:10.499 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104920.log 2023-10-19T14:23:10.499 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.148299.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.159178.log.gzgzip 2023-10-19T14:23:10.500 INFO:teuthology.orchestra.run.smithi057.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.153942.log 2023-10-19T14:23:10.500 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.148299.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118655.log 2023-10-19T14:23:10.500 INFO:teuthology.orchestra.run.smithi057.stderr: 2023-10-19T14:23:10.500 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.104920.log: /var/log/ceph/ceph-client.admin.153942.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111751.log 2023-10-19T14:23:10.500 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4%/var/log/ceph/ceph-client.admin.118655.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111293.log 2023-10-19T14:23:10.501 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/ceph-client.admin.153942.log.gz 2023-10-19T14:23:10.501 INFO:teuthology.orchestra.run.smithi057.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.104920.log.gz 2023-10-19T14:23:10.501 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.111751.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.118655.log.gz 2023-10-19T14:23:10.501 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166547.log 2023-10-19T14:23:10.501 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.111751.log.gz 2023-10-19T14:23:10.502 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.111293.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153858.log 2023-10-19T14:23:10.502 INFO:teuthology.orchestra.run.smithi057.stderr:gzip 87.7% -- replaced with /var/log/ceph/ceph-client.admin.111293.log.gz 2023-10-19T14:23:10.502 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.166547.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.118149.log 2023-10-19T14:23:10.502 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.153858.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.166547.log.gz 2023-10-19T14:23:10.502 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132457.log 2023-10-19T14:23:10.502 INFO:teuthology.orchestra.run.smithi057.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.153858.log.gz 2023-10-19T14:23:10.503 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.118149.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98207.log 2023-10-19T14:23:10.503 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.118149.log.gz 2023-10-19T14:23:10.503 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.132457.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160245.log 2023-10-19T14:23:10.503 INFO:teuthology.orchestra.run.smithi057.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.132457.log.gz 2023-10-19T14:23:10.503 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.98207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165160.log 2023-10-19T14:23:10.504 INFO:teuthology.orchestra.run.smithi057.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.98207.log.gz 2023-10-19T14:23:10.504 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.160245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117782.log 2023-10-19T14:23:10.504 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5%/var/log/ceph/ceph-client.admin.165160.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123822.log 2023-10-19T14:23:10.504 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/ceph-client.admin.160245.log.gz 2023-10-19T14:23:10.504 INFO:teuthology.orchestra.run.smithi057.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.165160.log.gz 2023-10-19T14:23:10.504 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.117782.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151825.log 2023-10-19T14:23:10.505 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- 87.6% /var/log/ceph/ceph-client.admin.97335.log 2023-10-19T14:23:10.505 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/ceph-client.admin.117782.log.gz 2023-10-19T14:23:10.505 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.123822.log: /var/log/ceph/ceph-client.admin.151825.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2023-10-19T14:23:10.505 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.123822.log.gzgzip -5 --verbose -- 87.4% /var/log/ceph/ceph-client.admin.113300.log 2023-10-19T14:23:10.505 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/ceph-client.admin.151825.log.gz 2023-10-19T14:23:10.506 INFO:teuthology.orchestra.run.smithi057.stderr: 2023-10-19T14:23:10.506 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.97335.log: /var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122685.log 2023-10-19T14:23:10.506 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4%/var/log/ceph/ceph-client.admin.113300.log: -- replaced with /var/log/ceph/ceph-client.admin.97335.log.gz 2023-10-19T14:23:10.506 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120146.log 2023-10-19T14:23:10.506 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.122685.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.113300.log.gz 2023-10-19T14:23:10.506 INFO:teuthology.orchestra.run.smithi057.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.122685.log.gz 2023-10-19T14:23:10.507 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107446.log 2023-10-19T14:23:10.507 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.120146.log: gzip -5 --verbose -- 87.3% -- replaced with /var/log/ceph/ceph-client.admin.120146.log.gz 2023-10-19T14:23:10.507 INFO:teuthology.orchestra.run.smithi057.stderr: /var/log/ceph/ceph-client.admin.112532.log 2023-10-19T14:23:10.507 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.107446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159866.log 2023-10-19T14:23:10.507 INFO:teuthology.orchestra.run.smithi057.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.107446.log.gz 2023-10-19T14:23:10.508 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.112532.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164938.log 2023-10-19T14:23:10.508 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.112532.log.gz 2023-10-19T14:23:10.508 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148877.log 2023-10-19T14:23:10.508 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.159866.log: /var/log/ceph/ceph-client.admin.164938.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153038.log 2023-10-19T14:23:10.508 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.148877.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.159866.log.gz 2023-10-19T14:23:10.508 INFO:teuthology.orchestra.run.smithi057.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.164938.log.gz 2023-10-19T14:23:10.509 INFO:teuthology.orchestra.run.smithi057.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.148877.log.gz 2023-10-19T14:23:10.509 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118013.log 2023-10-19T14:23:10.509 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.153038.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162682.log 2023-10-19T14:23:10.509 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.153038.log.gz 2023-10-19T14:23:10.509 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.118013.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126122.log 2023-10-19T14:23:10.510 INFO:teuthology.orchestra.run.smithi057.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.118013.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156833.log 2023-10-19T14:23:10.510 INFO:teuthology.orchestra.run.smithi057.stderr: 2023-10-19T14:23:10.510 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.162682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102804.log 2023-10-19T14:23:10.510 INFO:teuthology.orchestra.run.smithi057.stderr: 87.1%/var/log/ceph/ceph-client.admin.126122.log: -- replaced with /var/log/ceph/ceph-client.admin.162682.log.gz 2023-10-19T14:23:10.510 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.156833.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.126122.log.gz 2023-10-19T14:23:10.510 INFO:teuthology.orchestra.run.smithi057.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.156833.log.gz 2023-10-19T14:23:10.511 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100119.log 2023-10-19T14:23:10.511 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.102804.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166505.log 2023-10-19T14:23:10.511 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.102804.log.gz 2023-10-19T14:23:10.511 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.100119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96677.log 2023-10-19T14:23:10.511 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.100119.log.gz 2023-10-19T14:23:10.512 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.166505.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157272.log 2023-10-19T14:23:10.512 INFO:teuthology.orchestra.run.smithi057.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.166505.log.gz 2023-10-19T14:23:10.512 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125561.log 2023-10-19T14:23:10.512 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.96677.log: /var/log/ceph/ceph-client.admin.157272.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161562.log 2023-10-19T14:23:10.512 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.125561.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.96677.log.gz 2023-10-19T14:23:10.512 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.157272.log.gz 2023-10-19T14:23:10.513 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127298.log 2023-10-19T14:23:10.513 INFO:teuthology.orchestra.run.smithi057.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.125561.log.gz 2023-10-19T14:23:10.513 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.161562.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.161562.log.gz 2023-10-19T14:23:10.513 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151741.log 2023-10-19T14:23:10.513 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.127298.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164303.log 2023-10-19T14:23:10.514 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4%/var/log/ceph/ceph-client.admin.151741.log: -- replaced with /var/log/ceph/ceph-client.admin.127298.log.gz 2023-10-19T14:23:10.514 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133006.log 2023-10-19T14:23:10.514 INFO:teuthology.orchestra.run.smithi057.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.151741.log.gz 2023-10-19T14:23:10.514 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130570.log 2023-10-19T14:23:10.514 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.164303.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163686.log 2023-10-19T14:23:10.515 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.133006.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.164303.log.gz 2023-10-19T14:23:10.515 INFO:teuthology.orchestra.run.smithi057.stderr: 83.9% -- replaced with /var/log/ceph/ceph-client.admin.133006.log.gz/var/log/ceph/ceph-client.admin.130570.log: 2023-10-19T14:23:10.515 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- 87.2% -- replaced with /var/log/ceph/ceph-client.admin.130570.log.gz 2023-10-19T14:23:10.515 INFO:teuthology.orchestra.run.smithi057.stderr: /var/log/ceph/ceph-client.admin.125832.log 2023-10-19T14:23:10.515 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.163686.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143412.log 2023-10-19T14:23:10.515 INFO:teuthology.orchestra.run.smithi057.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.163686.log.gz 2023-10-19T14:23:10.516 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.125832.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96263.log 2023-10-19T14:23:10.516 INFO:teuthology.orchestra.run.smithi057.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.125832.log.gz 2023-10-19T14:23:10.516 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.143412.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126476.log 2023-10-19T14:23:10.516 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.143412.log.gz 2023-10-19T14:23:10.516 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140425.log 2023-10-19T14:23:10.517 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.96263.log: /var/log/ceph/ceph-client.admin.126476.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143454.log 2023-10-19T14:23:10.517 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.140425.log: 87.6% 87.8% -- replaced with /var/log/ceph/ceph-client.admin.126476.log.gz 2023-10-19T14:23:10.517 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/ceph-client.admin.96263.log.gz 2023-10-19T14:23:10.517 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.140425.log.gz 2023-10-19T14:23:10.517 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103602.log 2023-10-19T14:23:10.517 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.143454.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133518.log 2023-10-19T14:23:10.518 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.143454.log.gz 2023-10-19T14:23:10.518 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.103602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152730.log 2023-10-19T14:23:10.518 INFO:teuthology.orchestra.run.smithi057.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.103602.log.gz 2023-10-19T14:23:10.518 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108373.log 2023-10-19T14:23:10.518 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.133518.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134205.log 2023-10-19T14:23:10.519 INFO:teuthology.orchestra.run.smithi057.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.133518.log.gz/var/log/ceph/ceph-client.admin.152730.log: 2023-10-19T14:23:10.519 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.108373.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.152730.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101880.log 2023-10-19T14:23:10.519 INFO:teuthology.orchestra.run.smithi057.stderr: 2023-10-19T14:23:10.519 INFO:teuthology.orchestra.run.smithi057.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.108373.log.gz 2023-10-19T14:23:10.519 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112920.log 2023-10-19T14:23:10.520 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.134205.log: /var/log/ceph/ceph-client.admin.101880.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166715.log 2023-10-19T14:23:10.520 INFO:teuthology.orchestra.run.smithi057.stderr: 87.1%/var/log/ceph/ceph-client.admin.112920.log: -- replaced with /var/log/ceph/ceph-client.admin.134205.log.gz 2023-10-19T14:23:10.520 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.101880.log.gz 2023-10-19T14:23:10.520 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.112920.log.gz 2023-10-19T14:23:10.520 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124110.log 2023-10-19T14:23:10.520 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.166715.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130402.log 87.1% -- replaced with /var/log/ceph/ceph-client.admin.166715.log.gz 2023-10-19T14:23:10.521 INFO:teuthology.orchestra.run.smithi057.stderr: 2023-10-19T14:23:10.521 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.124110.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158066.log 87.3% -- replaced with /var/log/ceph/ceph-client.admin.124110.log.gz 2023-10-19T14:23:10.521 INFO:teuthology.orchestra.run.smithi057.stderr: 2023-10-19T14:23:10.521 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.130402.log: gzip -5 --verbose -- 87.5% /var/log/ceph/ceph-client.admin.127512.log 2023-10-19T14:23:10.521 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/ceph-client.admin.130402.log.gz 2023-10-19T14:23:10.522 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.158066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152096.log 2023-10-19T14:23:10.522 INFO:teuthology.orchestra.run.smithi057.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.158066.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143946.log 2023-10-19T14:23:10.522 INFO:teuthology.orchestra.run.smithi057.stderr: 2023-10-19T14:23:10.522 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.127512.log: 87.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140779.log 2023-10-19T14:23:10.522 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/ceph-client.admin.127512.log.gz 2023-10-19T14:23:10.522 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.152096.log: /var/log/ceph/ceph-client.admin.143946.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.152096.log.gz 2023-10-19T14:23:10.523 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.143946.log.gz 2023-10-19T14:23:10.523 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118763.log 2023-10-19T14:23:10.523 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.140779.log: gzip -5 --verbose -- 87.0% -- replaced with /var/log/ceph/ceph-client.admin.140779.log.gz 2023-10-19T14:23:10.523 INFO:teuthology.orchestra.run.smithi057.stderr: /var/log/ceph/ceph-client.admin.121504.log 2023-10-19T14:23:10.523 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.118763.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122922.log 87.8% -- replaced with /var/log/ceph/ceph-client.admin.118763.log.gz 2023-10-19T14:23:10.524 INFO:teuthology.orchestra.run.smithi057.stderr: 2023-10-19T14:23:10.524 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.121504.log: gzip -5 --verbose -- 87.6% /var/log/ceph/ceph-client.admin.102684.log 2023-10-19T14:23:10.524 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/ceph-client.admin.121504.log.gz 2023-10-19T14:23:10.524 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.122922.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163896.log 2023-10-19T14:23:10.524 INFO:teuthology.orchestra.run.smithi057.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.122922.log.gz/var/log/ceph/ceph-client.admin.102684.log: 2023-10-19T14:23:10.524 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141628.log 2023-10-19T14:23:10.525 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.102684.log.gz 2023-10-19T14:23:10.525 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5/var/log/ceph/ceph-client.admin.163896.log: --verbose -- /var/log/ceph/ceph-client.admin.103686.log 2023-10-19T14:23:10.525 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.141628.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.163896.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116793.log 2023-10-19T14:23:10.525 INFO:teuthology.orchestra.run.smithi057.stderr: 2023-10-19T14:23:10.525 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.141628.log.gz 2023-10-19T14:23:10.526 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159052.log 2023-10-19T14:23:10.526 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.103686.log: /var/log/ceph/ceph-client.admin.116793.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158968.log 2023-10-19T14:23:10.526 INFO:teuthology.orchestra.run.smithi057.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.103686.log.gz/var/log/ceph/ceph-client.admin.159052.log: 2023-10-19T14:23:10.526 INFO:teuthology.orchestra.run.smithi057.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.116793.log.gz 2023-10-19T14:23:10.526 INFO:teuthology.orchestra.run.smithi057.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.159052.log.gz 2023-10-19T14:23:10.526 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121027.log 2023-10-19T14:23:10.527 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.158968.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131867.log 2023-10-19T14:23:10.527 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.158968.log.gz 2023-10-19T14:23:10.527 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.121027.log: gzip -5 --verbose -- 87.8% -- replaced with /var/log/ceph/ceph-client.admin.121027.log.gz 2023-10-19T14:23:10.527 INFO:teuthology.orchestra.run.smithi057.stderr: /var/log/ceph/ceph-client.admin.147370.log 2023-10-19T14:23:10.527 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.131867.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149003.log 2023-10-19T14:23:10.528 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.147370.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97871.log 2023-10-19T14:23:10.528 INFO:teuthology.orchestra.run.smithi057.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.131867.log.gz 2023-10-19T14:23:10.528 INFO:teuthology.orchestra.run.smithi057.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.147370.log.gz 2023-10-19T14:23:10.528 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103560.log 2023-10-19T14:23:10.528 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.149003.log: /var/log/ceph/ceph-client.admin.97871.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.97871.log.gz 2023-10-19T14:23:10.528 INFO:teuthology.orchestra.run.smithi057.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.149003.log.gz 2023-10-19T14:23:10.529 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148961.log 2023-10-19T14:23:10.529 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.103560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147748.log 2023-10-19T14:23:10.529 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5%/var/log/ceph/ceph-client.admin.148961.log: -- replaced with /var/log/ceph/ceph-client.admin.103560.log.gz 2023-10-19T14:23:10.529 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147874.log 2023-10-19T14:23:10.529 INFO:teuthology.orchestra.run.smithi057.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.148961.log.gz 2023-10-19T14:23:10.530 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119978.log 2023-10-19T14:23:10.530 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.147748.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161310.log 2023-10-19T14:23:10.530 INFO:teuthology.orchestra.run.smithi057.stderr: 87.3%/var/log/ceph/ceph-client.admin.147874.log: -- replaced with /var/log/ceph/ceph-client.admin.147748.log.gz 2023-10-19T14:23:10.530 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.119978.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.147874.log.gz 87.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100413.log 2023-10-19T14:23:10.530 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/ceph-client.admin.119978.log.gz 2023-10-19T14:23:10.530 INFO:teuthology.orchestra.run.smithi057.stderr: 2023-10-19T14:23:10.531 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.161310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.125038.log 87.5% 2023-10-19T14:23:10.531 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/ceph-client.admin.161310.log.gz 2023-10-19T14:23:10.531 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.100413.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113426.log 2023-10-19T14:23:10.531 INFO:teuthology.orchestra.run.smithi057.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.100413.log.gz 2023-10-19T14:23:10.531 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.0.125038.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116656.log 2023-10-19T14:23:10.532 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152688.log 2023-10-19T14:23:10.532 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.113426.log: /var/log/ceph/ceph-client.admin.116656.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.113426.log.gz 2023-10-19T14:23:10.532 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104226.log 2023-10-19T14:23:10.532 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/ceph-client.admin.116656.log.gz 2023-10-19T14:23:10.532 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.152688.log: gzip 94.1% -- replaced with /var/log/ceph/ceph-client.0.125038.log.gz 2023-10-19T14:23:10.532 INFO:teuthology.orchestra.run.smithi057.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.144276.log 2023-10-19T14:23:10.533 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.104226.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125440.log 2023-10-19T14:23:10.533 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.152688.log.gz 2023-10-19T14:23:10.533 INFO:teuthology.orchestra.run.smithi057.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.104226.log.gz 2023-10-19T14:23:10.533 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.144276.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96997.log 2023-10-19T14:23:10.533 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.125440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152011.log 2023-10-19T14:23:10.534 INFO:teuthology.orchestra.run.smithi057.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.144276.log.gz 2023-10-19T14:23:10.534 INFO:teuthology.orchestra.run.smithi057.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.125440.log.gz/var/log/ceph/ceph-client.admin.96997.log: 2023-10-19T14:23:10.534 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.153701.log 2023-10-19T14:23:10.534 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.152011.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111937.log 2023-10-19T14:23:10.534 INFO:teuthology.orchestra.run.smithi057.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.96997.log.gz 2023-10-19T14:23:10.535 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.0.153701.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.152011.log.gz 2023-10-19T14:23:10.535 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111709.log 2023-10-19T14:23:10.535 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.111937.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153292.log 2023-10-19T14:23:10.535 INFO:teuthology.orchestra.run.smithi057.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.111937.log.gz 2023-10-19T14:23:10.535 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.111709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126789.log 2023-10-19T14:23:10.537 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.153292.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.111709.log.gz 2023-10-19T14:23:10.537 INFO:teuthology.orchestra.run.smithi057.stderr: 91.9% -- replaced with /var/log/ceph/ceph-client.0.153701.log.gz 2023-10-19T14:23:10.538 INFO:teuthology.orchestra.run.smithi057.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.153292.log.gz 2023-10-19T14:23:10.538 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147790.log 2023-10-19T14:23:10.538 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.126789.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101754.log 2023-10-19T14:23:10.538 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.147790.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102516.log 2023-10-19T14:23:10.538 INFO:teuthology.orchestra.run.smithi057.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.126789.log.gz 2023-10-19T14:23:10.539 INFO:teuthology.orchestra.run.smithi057.stderr: 86.9%/var/log/ceph/ceph-client.admin.101754.log: -- replaced with /var/log/ceph/ceph-client.admin.147790.log.gz 2023-10-19T14:23:10.539 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94342.log 2023-10-19T14:23:10.539 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.102516.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130318.log 2023-10-19T14:23:10.539 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.101754.log.gz 2023-10-19T14:23:10.539 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.94342.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.102516.log.gz 2023-10-19T14:23:10.540 INFO:teuthology.orchestra.run.smithi057.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.94342.log.gz 2023-10-19T14:23:10.540 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166379.log 2023-10-19T14:23:10.540 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.130318.log: gzip -5 --verbose -- 87.3% -- replaced with /var/log/ceph/ceph-client.admin.130318.log.gz 2023-10-19T14:23:10.540 INFO:teuthology.orchestra.run.smithi057.stderr: /var/log/ceph/ceph-client.admin.152774.log 2023-10-19T14:23:10.540 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.166379.log: gzip -5 --verbose 87.5% -- replaced with /var/log/ceph/ceph-client.admin.166379.log.gz 2023-10-19T14:23:10.540 INFO:teuthology.orchestra.run.smithi057.stderr: -- /var/log/ceph/ceph-client.admin.164429.log 2023-10-19T14:23:10.541 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.152774.log: gzip -5 --verbose -- 87.8% -- replaced with /var/log/ceph/ceph-client.admin.152774.log.gz 2023-10-19T14:23:10.541 INFO:teuthology.orchestra.run.smithi057.stderr: /var/log/ceph/ceph-client.admin.146317.log 2023-10-19T14:23:10.541 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.164429.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103644.log 87.8% -- replaced with /var/log/ceph/ceph-client.admin.164429.log.gz 2023-10-19T14:23:10.541 INFO:teuthology.orchestra.run.smithi057.stderr: 2023-10-19T14:23:10.541 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.146317.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163162.log 87.0% -- replaced with /var/log/ceph/ceph-client.admin.146317.log.gz 2023-10-19T14:23:10.542 INFO:teuthology.orchestra.run.smithi057.stderr: 2023-10-19T14:23:10.542 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.103644.log: gzip -5 --verbose 87.5% -- replaced with /var/log/ceph/ceph-client.admin.103644.log.gz 2023-10-19T14:23:10.542 INFO:teuthology.orchestra.run.smithi057.stderr: -- /var/log/ceph/ceph-osd.0.log 2023-10-19T14:23:10.542 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.163162.log: gzip -5 --verbose -- 87.6% /var/log/ceph/ceph-client.admin.101364.log 2023-10-19T14:23:10.542 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/ceph-client.admin.163162.log.gz 2023-10-19T14:23:10.543 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95039.log 2023-10-19T14:23:10.543 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154736.log 2023-10-19T14:23:10.543 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.101364.log: /var/log/ceph/ceph-client.admin.95039.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.101364.log.gz 2023-10-19T14:23:10.543 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.95039.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164387.log 2023-10-19T14:23:10.543 INFO:teuthology.orchestra.run.smithi057.stderr: 2023-10-19T14:23:10.543 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.154736.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.154736.log.gz 2023-10-19T14:23:10.544 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158510.log 2023-10-19T14:23:10.544 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147916.log 2023-10-19T14:23:10.544 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.164387.log: /var/log/ceph/ceph-client.admin.158510.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.164387.log.gz 2023-10-19T14:23:10.544 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128330.log 2023-10-19T14:23:10.544 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/ceph-client.admin.158510.log.gz 2023-10-19T14:23:10.545 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.147916.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158276.log 2023-10-19T14:23:10.545 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.128330.log: 86.9% 87.2% -- replaced with /var/log/ceph/ceph-client.admin.128330.log.gz 2023-10-19T14:23:10.545 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/ceph-client.admin.147916.log.gz 2023-10-19T14:23:10.545 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116274.log 2023-10-19T14:23:10.545 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.158276.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123475.log 2023-10-19T14:23:10.545 INFO:teuthology.orchestra.run.smithi057.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.158276.log.gz 2023-10-19T14:23:10.546 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.116274.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122880.log 2023-10-19T14:23:10.546 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.123475.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.116274.log.gz 2023-10-19T14:23:10.546 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.123475.log.gz 2023-10-19T14:23:10.546 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165481.log 2023-10-19T14:23:10.546 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.122880.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96761.log 2023-10-19T14:23:10.547 INFO:teuthology.orchestra.run.smithi057.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.122880.log.gz 2023-10-19T14:23:10.547 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.165481.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95189.log 2023-10-19T14:23:10.547 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.96761.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.165481.log.gz 2023-10-19T14:23:10.547 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.96761.log.gz 2023-10-19T14:23:10.547 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122727.log 2023-10-19T14:23:10.547 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.95189.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130444.log 2023-10-19T14:23:10.548 INFO:teuthology.orchestra.run.smithi057.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.95189.log.gz 2023-10-19T14:23:10.548 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.122727.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127554.log 2023-10-19T14:23:10.548 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.122727.log.gz 2023-10-19T14:23:10.548 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.130444.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.130444.log.gz 2023-10-19T14:23:10.548 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151391.log 2023-10-19T14:23:10.549 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.127554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117442.log 2023-10-19T14:23:10.549 INFO:teuthology.orchestra.run.smithi057.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.127554.log.gz 2023-10-19T14:23:10.549 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.151391.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116419.log 2023-10-19T14:23:10.549 INFO:teuthology.orchestra.run.smithi057.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.151391.log.gz 2023-10-19T14:23:10.549 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.117442.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.117442.log.gz 2023-10-19T14:23:10.549 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154376.log 2023-10-19T14:23:10.550 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.116419.log: 94.4% -- replaced with /var/log/ceph/ceph-client.0.113835.log.gz 2023-10-19T14:23:10.550 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116545.log 2023-10-19T14:23:10.550 INFO:teuthology.orchestra.run.smithi057.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.116419.log.gz 2023-10-19T14:23:10.550 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.154376.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117003.log 87.6% -- replaced with /var/log/ceph/ceph-client.admin.154376.log.gz 2023-10-19T14:23:10.550 INFO:teuthology.orchestra.run.smithi057.stderr: 2023-10-19T14:23:10.551 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.116545.log: gzip -5 --verbose -- 87.5% -- replaced with /var/log/ceph/ceph-client.admin.116545.log.gz 2023-10-19T14:23:10.551 INFO:teuthology.orchestra.run.smithi057.stderr: /var/log/ceph/ceph-client.admin.124502.log 2023-10-19T14:23:10.551 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.117003.log: gzip 87.3% -- replaced with /var/log/ceph/ceph-client.admin.117003.log.gz 2023-10-19T14:23:10.551 INFO:teuthology.orchestra.run.smithi057.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.102290.log 2023-10-19T14:23:10.551 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.124502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139654.log 2023-10-19T14:23:10.551 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.124502.log.gz/var/log/ceph/ceph-client.admin.102290.log: 2023-10-19T14:23:10.552 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151223.log 2023-10-19T14:23:10.552 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.102290.log.gz 2023-10-19T14:23:10.552 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123643.log 2023-10-19T14:23:10.552 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.139654.log: /var/log/ceph/ceph-client.admin.151223.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105846.log 2023-10-19T14:23:10.552 INFO:teuthology.orchestra.run.smithi057.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.139654.log.gz 87.0% 2023-10-19T14:23:10.553 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/ceph-client.admin.151223.log.gz/var/log/ceph/ceph-client.admin.123643.log: 2023-10-19T14:23:10.553 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103182.log 2023-10-19T14:23:10.553 INFO:teuthology.orchestra.run.smithi057.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.123643.log.gz 2023-10-19T14:23:10.553 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.105846.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118487.log 2023-10-19T14:23:10.553 INFO:teuthology.orchestra.run.smithi057.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.105846.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101138.log 2023-10-19T14:23:10.554 INFO:teuthology.orchestra.run.smithi057.stderr: 2023-10-19T14:23:10.554 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.103182.log: /var/log/ceph/ceph-client.admin.118487.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98039.log 2023-10-19T14:23:10.554 INFO:teuthology.orchestra.run.smithi057.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.103182.log.gz 2023-10-19T14:23:10.554 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.101138.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.118487.log.gz 2023-10-19T14:23:10.554 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.101138.log.gz 2023-10-19T14:23:10.554 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104057.log 2023-10-19T14:23:10.555 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.98039.log: gzip -5 --verbose 87.8% -- replaced with /var/log/ceph/ceph-client.admin.98039.log.gz 2023-10-19T14:23:10.555 INFO:teuthology.orchestra.run.smithi057.stderr: -- /var/log/ceph/ceph-client.admin.125345.log 2023-10-19T14:23:10.555 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.104057.log: gzip -5 --verbose 87.4% -- replaced with /var/log/ceph/ceph-client.admin.104057.log.gz 2023-10-19T14:23:10.555 INFO:teuthology.orchestra.run.smithi057.stderr: -- /var/log/ceph/ceph-client.admin.162138.log 2023-10-19T14:23:10.555 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.125345.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165439.log 86.8% -- replaced with /var/log/ceph/ceph-client.admin.125345.log.gz 2023-10-19T14:23:10.556 INFO:teuthology.orchestra.run.smithi057.stderr: 2023-10-19T14:23:10.556 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.162138.log: gzip -5 --verbose 87.1% -- replaced with /var/log/ceph/ceph-client.admin.162138.log.gz 2023-10-19T14:23:10.556 INFO:teuthology.orchestra.run.smithi057.stderr: -- /var/log/ceph/ceph-client.0.158769.log 2023-10-19T14:23:10.556 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.165439.log: gzip -5 --verbose -- 87.5% -- replaced with /var/log/ceph/ceph-client.admin.165439.log.gz 2023-10-19T14:23:10.556 INFO:teuthology.orchestra.run.smithi057.stderr: /var/log/ceph/ceph-client.admin.121154.log 2023-10-19T14:23:10.556 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.0.158769.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133671.log 2023-10-19T14:23:10.557 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.121154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159316.log 2023-10-19T14:23:10.557 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.121154.log.gz 2023-10-19T14:23:10.557 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.133671.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126374.log 2023-10-19T14:23:10.557 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.159316.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.133671.log.gz 2023-10-19T14:23:10.557 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.159316.log.gz 2023-10-19T14:23:10.558 INFO:teuthology.orchestra.run.smithi057.stderr: 91.9% -- replaced with /var/log/ceph/ceph-client.0.158769.log.gz 2023-10-19T14:23:10.558 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147286.log 2023-10-19T14:23:10.558 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.126374.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120603.log 2023-10-19T14:23:10.558 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.147286.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.126374.log.gz 2023-10-19T14:23:10.558 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131265.log 2023-10-19T14:23:10.559 INFO:teuthology.orchestra.run.smithi057.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.147286.log.gz 2023-10-19T14:23:10.559 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124290.log 2023-10-19T14:23:10.559 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.120603.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141820.log 2023-10-19T14:23:10.559 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.131265.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.120603.log.gz 2023-10-19T14:23:10.559 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.124290.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.131265.log.gz 2023-10-19T14:23:10.559 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159654.log 2023-10-19T14:23:10.560 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.124290.log.gz 2023-10-19T14:23:10.560 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.141820.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159266.log 2023-10-19T14:23:10.560 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.159654.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-10-19T14:23:10.560 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.141820.log.gz 2023-10-19T14:23:10.560 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4%/var/log/ceph/ceph-client.admin.159266.log: -- replaced with /var/log/ceph/ceph-client.admin.159654.log.gz 2023-10-19T14:23:10.561 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126332.log 2023-10-19T14:23:10.561 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126579.log 2023-10-19T14:23:10.561 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.159266.log.gz 2023-10-19T14:23:10.561 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.126332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158444.log 2023-10-19T14:23:10.561 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.126579.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.126332.log.gz 2023-10-19T14:23:10.561 INFO:teuthology.orchestra.run.smithi057.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.126579.log.gz 2023-10-19T14:23:10.562 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143286.log 2023-10-19T14:23:10.562 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.158444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148835.log 2023-10-19T14:23:10.562 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.143286.log: 91.1% 87.4% -- replaced with /var/log/ceph/ceph-client.admin.143286.log.gz 2023-10-19T14:23:10.562 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/ceph-client.admin.158444.log.gz 2023-10-19T14:23:10.562 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117701.log 2023-10-19T14:23:10.563 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103308.log 2023-10-19T14:23:10.563 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.148835.log: /var/log/ceph/ceph-client.admin.117701.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.148835.log.gz 2023-10-19T14:23:10.563 INFO:teuthology.orchestra.run.smithi057.stderr: 87.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111105.log 2023-10-19T14:23:10.563 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/ceph-client.admin.117701.log.gz 2023-10-19T14:23:10.563 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.103308.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157804.log 2023-10-19T14:23:10.563 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.111105.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.103308.log.gz 87.2% -- replaced with /var/log/ceph/ceph-client.admin.111105.log.gz 2023-10-19T14:23:10.564 INFO:teuthology.orchestra.run.smithi057.stderr: 2023-10-19T14:23:10.564 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156581.log 2023-10-19T14:23:10.564 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.157804.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116377.log 2023-10-19T14:23:10.564 INFO:teuthology.orchestra.run.smithi057.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.157804.log.gz 2023-10-19T14:23:10.564 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.156581.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108933.log 2023-10-19T14:23:10.565 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.116377.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.156581.log.gz 2023-10-19T14:23:10.565 INFO:teuthology.orchestra.run.smithi057.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.116377.log.gz 2023-10-19T14:23:10.565 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157146.log 2023-10-19T14:23:10.565 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.108933.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127172.log 2023-10-19T14:23:10.565 INFO:teuthology.orchestra.run.smithi057.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.108933.log.gz 2023-10-19T14:23:10.565 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.157146.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162001.log 2023-10-19T14:23:10.566 INFO:teuthology.orchestra.run.smithi057.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.157146.log.gz 2023-10-19T14:23:10.566 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.127172.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159696.log 2023-10-19T14:23:10.566 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6%/var/log/ceph/ceph-client.admin.162001.log: -- replaced with /var/log/ceph/ceph-client.admin.127172.log.gz 2023-10-19T14:23:10.566 INFO:teuthology.orchestra.run.smithi057.stderr: 89.3% 87.5% -- replaced with /var/log/ceph/ceph-client.admin.162001.log.gz 2023-10-19T14:23:10.566 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/ceph.log.gz 2023-10-19T14:23:10.567 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104836.log 2023-10-19T14:23:10.567 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.159696.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154290.log 2023-10-19T14:23:10.567 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.104836.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102600.log 2023-10-19T14:23:10.567 INFO:teuthology.orchestra.run.smithi057.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.159696.log.gz 2023-10-19T14:23:10.567 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.104836.log.gz 2023-10-19T14:23:10.567 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.154290.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127386.log 2023-10-19T14:23:10.568 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.102600.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133129.log 2023-10-19T14:23:10.568 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.154290.log.gz 2023-10-19T14:23:10.568 INFO:teuthology.orchestra.run.smithi057.stderr: 87.2%/var/log/ceph/ceph-client.admin.127386.log: -- replaced with /var/log/ceph/ceph-client.admin.102600.log.gz 2023-10-19T14:23:10.568 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111335.log 2023-10-19T14:23:10.568 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.133129.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.127386.log.gz 2023-10-19T14:23:10.569 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 84.0% -- replaced with /var/log/ceph/ceph-client.admin.133129.log.gz 2023-10-19T14:23:10.569 INFO:teuthology.orchestra.run.smithi057.stderr: 2023-10-19T14:23:10.569 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.111335.log: gzip 87.1% -- replaced with /var/log/ceph/ceph-client.admin.111335.log.gz 2023-10-19T14:23:10.569 INFO:teuthology.orchestra.run.smithi057.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.141712.log 2023-10-19T14:23:10.569 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122838.log 2023-10-19T14:23:10.569 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.141712.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157020.log 2023-10-19T14:23:10.570 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.141712.log.gz 2023-10-19T14:23:10.570 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.122838.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128438.log 2023-10-19T14:23:10.570 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.157020.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.122838.log.gz 87.7% -- replaced with /var/log/ceph/ceph-client.admin.157020.log.gz 2023-10-19T14:23:10.570 INFO:teuthology.orchestra.run.smithi057.stderr: 2023-10-19T14:23:10.570 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163270.log 2023-10-19T14:23:10.571 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.128438.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147454.log 2023-10-19T14:23:10.571 INFO:teuthology.orchestra.run.smithi057.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.128438.log.gz 2023-10-19T14:23:10.571 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.163270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105738.log 2023-10-19T14:23:10.571 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.147454.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.163270.log.gz 2023-10-19T14:23:10.571 INFO:teuthology.orchestra.run.smithi057.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.147454.log.gz 2023-10-19T14:23:10.572 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101306.log 2023-10-19T14:23:10.572 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.105738.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128539.log 2023-10-19T14:23:10.572 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.105738.log.gz 2023-10-19T14:23:10.572 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.101306.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.82036.log 2023-10-19T14:23:10.572 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.128539.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.101306.log.gz 2023-10-19T14:23:10.572 INFO:teuthology.orchestra.run.smithi057.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.128539.log.gz 2023-10-19T14:23:10.573 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116503.log 2023-10-19T14:23:10.573 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph.tmp-client.admin.82036.log: 0.0%gzip -- replaced with /var/log/ceph/ceph.tmp-client.admin.82036.log.gz -5 --verbose -- /var/log/ceph/ceph-client.admin.156413.log 2023-10-19T14:23:10.573 INFO:teuthology.orchestra.run.smithi057.stderr: 2023-10-19T14:23:10.573 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102930.log 2023-10-19T14:23:10.573 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.116503.log: /var/log/ceph/ceph-client.admin.156413.log: 87.0% 87.6% -- replaced with /var/log/ceph/ceph-client.admin.156413.log.gz 2023-10-19T14:23:10.574 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/ceph-client.admin.116503.log.gz 2023-10-19T14:23:10.574 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126164.log 2023-10-19T14:23:10.574 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.102930.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2023-10-19T14:23:10.574 INFO:teuthology.orchestra.run.smithi057.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.102930.log.gz 2023-10-19T14:23:10.574 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.126164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122975.log 2023-10-19T14:23:10.574 INFO:teuthology.orchestra.run.smithi057.stderr: 86.6%/var/log/ceph/ceph-mds.c.log: -- replaced with /var/log/ceph/ceph-client.admin.126164.log.gz 2023-10-19T14:23:10.575 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100875.log 2023-10-19T14:23:10.575 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.122975.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.122975.log.gz 2023-10-19T14:23:10.575 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156371.log 2023-10-19T14:23:10.575 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.100875.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.100875.log.gz 2023-10-19T14:23:10.575 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132688.log 2023-10-19T14:23:10.576 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.156371.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.156371.log.gz 2023-10-19T14:23:10.576 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143724.log 2023-10-19T14:23:10.576 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.132688.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.132688.log.gz 2023-10-19T14:23:10.576 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144743.log 2023-10-19T14:23:10.576 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.143724.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.143724.log.gz 2023-10-19T14:23:10.576 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106884.log 2023-10-19T14:23:10.577 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.144743.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.144743.log.gz 2023-10-19T14:23:10.577 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159222.log 2023-10-19T14:23:10.577 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.106884.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.106884.log.gz 2023-10-19T14:23:10.577 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.167135.log 2023-10-19T14:23:10.577 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.159222.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.159222.log.gz 2023-10-19T14:23:10.578 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95612.log 2023-10-19T14:23:10.578 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.167135.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.167135.log.gz 2023-10-19T14:23:10.578 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105004.log 2023-10-19T14:23:10.578 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.95612.log: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.95612.log.gz 2023-10-19T14:23:10.578 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159762.log 2023-10-19T14:23:10.578 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.105004.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.105004.log.gz 2023-10-19T14:23:10.579 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131573.log 2023-10-19T14:23:10.579 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.159762.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.159762.log.gz 2023-10-19T14:23:10.579 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151951.log 2023-10-19T14:23:10.579 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.131573.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.131573.log.gz 2023-10-19T14:23:10.579 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162218.log 2023-10-19T14:23:10.580 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.151951.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.151951.log.gz 2023-10-19T14:23:10.580 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141502.log 2023-10-19T14:23:10.580 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.162218.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.162218.log.gz 2023-10-19T14:23:10.580 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121376.log 2023-10-19T14:23:10.580 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.141502.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.141502.log.gz 2023-10-19T14:23:10.580 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152222.log 2023-10-19T14:23:10.581 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.121376.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.121376.log.gz 2023-10-19T14:23:10.581 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155016.log 2023-10-19T14:23:10.581 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.152222.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.152222.log.gz 2023-10-19T14:23:10.581 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134815.log 2023-10-19T14:23:10.581 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.155016.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.155016.log.gz 2023-10-19T14:23:10.582 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151909.log 2023-10-19T14:23:10.582 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.134815.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.134815.log.gz 2023-10-19T14:23:10.582 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154332.log 2023-10-19T14:23:10.582 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.151909.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.151909.log.gz 2023-10-19T14:23:10.582 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135544.log 2023-10-19T14:23:10.582 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.154332.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.154332.log.gz 2023-10-19T14:23:10.583 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100203.log 2023-10-19T14:23:10.583 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.135544.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.135544.log.gz 2023-10-19T14:23:10.583 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131825.log 2023-10-19T14:23:10.583 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.100203.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.100203.log.gz 2023-10-19T14:23:10.583 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140593.log 2023-10-19T14:23:10.584 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.131825.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.131825.log.gz 2023-10-19T14:23:10.584 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103896.log 2023-10-19T14:23:10.584 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.140593.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.140593.log.gz 2023-10-19T14:23:10.584 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166038.log 2023-10-19T14:23:10.584 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.103896.log: 93.1% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2023-10-19T14:23:10.584 INFO:teuthology.orchestra.run.smithi057.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.103896.log.gz 2023-10-19T14:23:10.585 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160661.log 2023-10-19T14:23:10.585 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.166038.log: -- /var/log/ceph/ceph-client.admin.142446.log 2023-10-19T14:23:10.585 INFO:teuthology.orchestra.run.smithi057.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.166038.log.gz 2023-10-19T14:23:10.585 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.160661.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152348.log 2023-10-19T14:23:10.585 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.142446.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.160661.log.gz 2023-10-19T14:23:10.586 INFO:teuthology.orchestra.run.smithi057.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.142446.log.gz 2023-10-19T14:23:10.586 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124586.log 2023-10-19T14:23:10.586 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.152348.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101490.log 2023-10-19T14:23:10.586 INFO:teuthology.orchestra.run.smithi057.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.152348.log.gz 2023-10-19T14:23:10.586 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.124586.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101222.log 2023-10-19T14:23:10.586 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.101490.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.124586.log.gz 2023-10-19T14:23:10.587 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143808.log 2023-10-19T14:23:10.587 INFO:teuthology.orchestra.run.smithi057.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.101490.log.gz 2023-10-19T14:23:10.587 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.101222.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104878.log 2023-10-19T14:23:10.587 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.143808.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.101222.log.gz 2023-10-19T14:23:10.587 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.143808.log.gz 2023-10-19T14:23:10.588 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97334.log 2023-10-19T14:23:10.588 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134433.log 2023-10-19T14:23:10.588 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.97334.log: /var/log/ceph/ceph-client.admin.104878.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.97334.log.gz 86.8% -- replaced with /var/log/ceph/ceph-client.admin.104878.log.gz 2023-10-19T14:23:10.588 INFO:teuthology.orchestra.run.smithi057.stderr: 2023-10-19T14:23:10.588 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125387.log 2023-10-19T14:23:10.588 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100539.log 2023-10-19T14:23:10.589 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.134433.log: /var/log/ceph/ceph-client.admin.125387.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.134433.log.gz 2023-10-19T14:23:10.589 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162934.log 2023-10-19T14:23:10.589 INFO:teuthology.orchestra.run.smithi057.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.125387.log.gz 2023-10-19T14:23:10.589 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.100539.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.100539.log.gz 2023-10-19T14:23:10.589 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123983.log 2023-10-19T14:23:10.590 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149635.log 2023-10-19T14:23:10.590 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.162934.log: /var/log/ceph/ceph-client.admin.123983.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.162934.log.gz 2023-10-19T14:23:10.590 INFO:teuthology.orchestra.run.smithi057.stderr: 87.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107977.log 2023-10-19T14:23:10.590 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/ceph-client.admin.123983.log.gz 2023-10-19T14:23:10.590 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.149635.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.149635.log.gz 2023-10-19T14:23:10.591 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155890.log 2023-10-19T14:23:10.591 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135250.log 2023-10-19T14:23:10.591 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.107977.log: /var/log/ceph/ceph-client.admin.155890.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.107977.log.gz 2023-10-19T14:23:10.591 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119221.log 2023-10-19T14:23:10.591 INFO:teuthology.orchestra.run.smithi057.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.155890.log.gz 2023-10-19T14:23:10.591 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.135250.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.135250.log.gz 2023-10-19T14:23:10.592 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103854.log 2023-10-19T14:23:10.592 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131699.log 2023-10-19T14:23:10.592 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.119221.log: /var/log/ceph/ceph-client.admin.103854.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.119221.log.gz 2023-10-19T14:23:10.592 INFO:teuthology.orchestra.run.smithi057.stderr: 87.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101096.log 2023-10-19T14:23:10.592 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/ceph-client.admin.103854.log.gz 2023-10-19T14:23:10.593 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.131699.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.131699.log.gz 2023-10-19T14:23:10.593 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98249.log 2023-10-19T14:23:10.593 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148623.log 2023-10-19T14:23:10.593 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.101096.log: /var/log/ceph/ceph-client.admin.98249.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.101096.log.gz 2023-10-19T14:23:10.593 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118697.log 2023-10-19T14:23:10.593 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.98249.log.gz 2023-10-19T14:23:10.594 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.148623.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.148623.log.gz 2023-10-19T14:23:10.594 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147958.log 2023-10-19T14:23:10.594 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159570.log 2023-10-19T14:23:10.647 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.118697.log: /var/log/ceph/ceph-client.admin.147958.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.147958.log.gz 91.0% -- replaced with /var/log/ceph/ceph-client.admin.118697.log.gz 2023-10-19T14:23:10.648 INFO:teuthology.orchestra.run.smithi057.stderr: 2023-10-19T14:23:10.648 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144491.log 2023-10-19T14:23:10.648 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106758.log 2023-10-19T14:23:10.648 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.159570.log: /var/log/ceph/ceph-client.admin.144491.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.159570.log.gz 2023-10-19T14:23:10.649 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94871.log 2023-10-19T14:23:10.649 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/ceph-client.admin.144491.log.gz 2023-10-19T14:23:10.649 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.106758.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.106758.log.gz 2023-10-19T14:23:10.649 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111530.log 2023-10-19T14:23:10.649 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136026.log 2023-10-19T14:23:10.650 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.94871.log: /var/log/ceph/ceph-client.admin.111530.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.94871.log.gz 2023-10-19T14:23:10.650 INFO:teuthology.orchestra.run.smithi057.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.111530.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107362.log 2023-10-19T14:23:10.650 INFO:teuthology.orchestra.run.smithi057.stderr: 2023-10-19T14:23:10.650 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.136026.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.136026.log.gz 2023-10-19T14:23:10.650 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134018.log 2023-10-19T14:23:10.650 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160955.log 2023-10-19T14:23:10.651 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.107362.log: /var/log/ceph/ceph-client.admin.134018.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.107362.log.gz 2023-10-19T14:23:10.651 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.134018.log.gz 2023-10-19T14:23:10.651 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141162.log 2023-10-19T14:23:10.651 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.160955.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.160955.log.gz 2023-10-19T14:23:10.651 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103976.log 2023-10-19T14:23:10.652 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113384.log 2023-10-19T14:23:10.652 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.141162.log: /var/log/ceph/ceph-client.admin.103976.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.141162.log.gz 2023-10-19T14:23:10.652 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.103976.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130780.log 2023-10-19T14:23:10.652 INFO:teuthology.orchestra.run.smithi057.stderr: 2023-10-19T14:23:10.652 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.113384.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.113384.log.gz 2023-10-19T14:23:10.653 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.2.99593.log 2023-10-19T14:23:10.653 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164119.log 2023-10-19T14:23:10.653 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.130780.log: /var/log/ceph/ceph-client.2.99593.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.130780.log.gz 2023-10-19T14:23:10.653 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154068.log 2023-10-19T14:23:10.653 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.164119.log: 91.1% -- replaced with /var/log/ceph/ceph-client.2.99593.log.gz 2023-10-19T14:23:10.653 INFO:teuthology.orchestra.run.smithi057.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.164119.log.gz 2023-10-19T14:23:10.654 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163078.log 2023-10-19T14:23:10.654 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143850.log 2023-10-19T14:23:10.654 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.154068.log: /var/log/ceph/ceph-client.admin.163078.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.154068.log.gz 2023-10-19T14:23:10.654 INFO:teuthology.orchestra.run.smithi057.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.163078.log.gz 2023-10-19T14:23:10.654 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106632.log 2023-10-19T14:23:10.655 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.143850.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.143850.log.gz 2023-10-19T14:23:10.655 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139508.log 2023-10-19T14:23:10.655 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127638.log 2023-10-19T14:23:10.655 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.106632.log: /var/log/ceph/ceph-client.admin.139508.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.106632.log.gz 2023-10-19T14:23:10.655 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.139508.log.gz 2023-10-19T14:23:10.655 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114142.log 2023-10-19T14:23:10.656 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.127638.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.127638.log.gz 2023-10-19T14:23:10.656 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119688.log 2023-10-19T14:23:10.656 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97337.log 2023-10-19T14:23:10.656 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.114142.log: /var/log/ceph/ceph-client.admin.119688.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.114142.log.gz 2023-10-19T14:23:10.656 INFO:teuthology.orchestra.run.smithi057.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.119688.log.gz 2023-10-19T14:23:10.657 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157674.log 2023-10-19T14:23:10.657 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.97337.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.97337.log.gz 2023-10-19T14:23:10.657 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142541.log 2023-10-19T14:23:10.657 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132747.log 2023-10-19T14:23:10.657 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.157674.log: /var/log/ceph/ceph-client.admin.142541.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.157674.log.gz 2023-10-19T14:23:10.657 INFO:teuthology.orchestra.run.smithi057.stderr: 85.4% -- replaced with /var/log/ceph/ceph-client.admin.142541.log.gz 2023-10-19T14:23:10.658 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106287.log 2023-10-19T14:23:10.658 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.132747.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.132747.log.gz 2023-10-19T14:23:10.658 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155974.log 2023-10-19T14:23:10.658 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130612.log 2023-10-19T14:23:10.658 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.106287.log: /var/log/ceph/ceph-client.admin.155974.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.106287.log.gz 2023-10-19T14:23:10.659 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.155974.log.gz 2023-10-19T14:23:10.659 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141334.log 2023-10-19T14:23:10.659 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.130612.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.130612.log.gz 2023-10-19T14:23:10.659 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97997.log 2023-10-19T14:23:10.659 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118403.log 2023-10-19T14:23:10.659 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.141334.log: /var/log/ceph/ceph-client.admin.97997.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.141334.log.gz 2023-10-19T14:23:10.660 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.97997.log.gz 2023-10-19T14:23:10.660 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99993.log 2023-10-19T14:23:10.660 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.118403.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.118403.log.gz 2023-10-19T14:23:10.660 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161394.log 2023-10-19T14:23:10.660 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107656.log 2023-10-19T14:23:10.661 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.99993.log: /var/log/ceph/ceph-client.admin.161394.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.99993.log.gz 2023-10-19T14:23:10.661 INFO:teuthology.orchestra.run.smithi057.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.161394.log.gz 2023-10-19T14:23:10.661 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140154.log 2023-10-19T14:23:10.661 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.107656.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.107656.log.gz 2023-10-19T14:23:10.661 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100959.log 2023-10-19T14:23:10.661 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148175.log 2023-10-19T14:23:10.662 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.140154.log: /var/log/ceph/ceph-client.admin.100959.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.140154.log.gz 2023-10-19T14:23:10.662 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.100959.log.gz 2023-10-19T14:23:10.662 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146051.log 2023-10-19T14:23:10.662 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.148175.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.148175.log.gz 2023-10-19T14:23:10.662 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147233.log 2023-10-19T14:23:10.663 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159971.log 2023-10-19T14:23:10.663 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.146051.log: /var/log/ceph/ceph-client.admin.147233.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.146051.log.gz 2023-10-19T14:23:10.663 INFO:teuthology.orchestra.run.smithi057.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.147233.log.gz 2023-10-19T14:23:10.663 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124460.log 2023-10-19T14:23:10.663 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103518.log 2023-10-19T14:23:10.664 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.159971.log: /var/log/ceph/ceph-client.admin.124460.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.159971.log.gz 2023-10-19T14:23:10.664 INFO:teuthology.orchestra.run.smithi057.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.124460.log.gz 2023-10-19T14:23:10.664 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122601.log 2023-10-19T14:23:10.664 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.103518.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.103518.log.gz 2023-10-19T14:23:10.664 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102846.log 2023-10-19T14:23:10.664 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147580.log 2023-10-19T14:23:10.665 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.122601.log: /var/log/ceph/ceph-client.admin.102846.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.122601.log.gz 2023-10-19T14:23:10.665 INFO:teuthology.orchestra.run.smithi057.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.102846.log.gz 2023-10-19T14:23:10.665 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126519.log 2023-10-19T14:23:10.665 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.147580.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.147580.log.gz 2023-10-19T14:23:10.665 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89145.log 2023-10-19T14:23:10.666 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124152.log 2023-10-19T14:23:10.666 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.126519.log: /var/log/ceph/ceph-client.admin.89145.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.126519.log.gz 2023-10-19T14:23:10.666 INFO:teuthology.orchestra.run.smithi057.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.89145.log.gz 2023-10-19T14:23:10.666 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.145152.log 2023-10-19T14:23:10.666 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.124152.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.124152.log.gz 2023-10-19T14:23:10.666 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155848.log 2023-10-19T14:23:10.667 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125748.log 2023-10-19T14:23:10.667 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.0.145152.log: /var/log/ceph/ceph-client.admin.155848.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.155848.log.gz 2023-10-19T14:23:10.667 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147538.log 2023-10-19T14:23:10.667 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.125748.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.125748.log.gz 2023-10-19T14:23:10.667 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144575.log 2023-10-19T14:23:10.668 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.147538.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.147538.log.gz 2023-10-19T14:23:10.668 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124067.log 2023-10-19T14:23:10.668 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.144575.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.144575.log.gz 2023-10-19T14:23:10.668 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88663.log 2023-10-19T14:23:10.668 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.124067.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.124067.log.gz 2023-10-19T14:23:10.669 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153166.log 2023-10-19T14:23:10.669 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.88663.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.88663.log.gz 2023-10-19T14:23:10.669 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100623.log 2023-10-19T14:23:10.669 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.153166.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.153166.log.gz 2023-10-19T14:23:10.669 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111667.log 2023-10-19T14:23:10.669 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.100623.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.100623.log.gz 2023-10-19T14:23:10.670 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94613.log 2023-10-19T14:23:10.670 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.111667.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.111667.log.gz 2023-10-19T14:23:10.670 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135028.log 2023-10-19T14:23:10.670 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.94613.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.94613.log.gz 2023-10-19T14:23:10.670 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166463.log 2023-10-19T14:23:10.671 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.135028.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.135028.log.gz 2023-10-19T14:23:10.671 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135460.log 2023-10-19T14:23:10.671 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.166463.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.166463.log.gz 2023-10-19T14:23:10.671 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142981.log 2023-10-19T14:23:10.671 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.135460.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.135460.log.gz 2023-10-19T14:23:10.671 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154418.log 2023-10-19T14:23:10.672 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.142981.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.142981.log.gz 2023-10-19T14:23:10.672 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124025.log 2023-10-19T14:23:10.672 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.154418.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.154418.log.gz 2023-10-19T14:23:10.672 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119883.log 2023-10-19T14:23:10.672 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.124025.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.124025.log.gz 2023-10-19T14:23:10.673 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114254.log 2023-10-19T14:23:10.673 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.119883.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.119883.log.gz 2023-10-19T14:23:10.673 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144320.log 2023-10-19T14:23:10.673 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.114254.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.114254.log.gz 2023-10-19T14:23:10.673 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165912.log 2023-10-19T14:23:10.673 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.144320.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.144320.log.gz 2023-10-19T14:23:10.674 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103224.log 2023-10-19T14:23:10.674 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.165912.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.165912.log.gz 2023-10-19T14:23:10.674 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160287.log 2023-10-19T14:23:10.674 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.103224.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.103224.log.gz 2023-10-19T14:23:10.674 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100497.log 2023-10-19T14:23:10.675 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.160287.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.160287.log.gz 2023-10-19T14:23:10.675 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107404.log 2023-10-19T14:23:10.675 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.100497.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.100497.log.gz 2023-10-19T14:23:10.675 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120985.log 2023-10-19T14:23:10.675 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.107404.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.107404.log.gz 2023-10-19T14:23:10.675 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103014.log 2023-10-19T14:23:10.676 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.120985.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.120985.log.gz 2023-10-19T14:23:10.676 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144893.log 2023-10-19T14:23:10.676 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.103014.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.103014.log.gz 2023-10-19T14:23:10.676 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134517.log 2023-10-19T14:23:10.676 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.144893.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.144893.log.gz 2023-10-19T14:23:10.677 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101574.log 2023-10-19T14:23:10.677 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.134517.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.134517.log.gz 2023-10-19T14:23:10.677 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123060.log 2023-10-19T14:23:10.677 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.101574.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.101574.log.gz 2023-10-19T14:23:10.677 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100329.log 2023-10-19T14:23:10.677 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.123060.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.123060.log.gz 2023-10-19T14:23:10.678 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96137.log 2023-10-19T14:23:10.678 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.100329.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.100329.log.gz 2023-10-19T14:23:10.678 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2023-10-19T14:23:10.678 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.96137.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.96137.log.gz 2023-10-19T14:23:10.678 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133560.log 2023-10-19T14:23:10.679 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-mds.a.log: 95.1% -- replaced with /var/log/ceph/ceph-client.0.145152.log.gz 2023-10-19T14:23:10.679 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104310.log 2023-10-19T14:23:10.679 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.133560.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.133560.log.gz 2023-10-19T14:23:10.679 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101001.log 2023-10-19T14:23:10.679 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.104310.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.104310.log.gz 2023-10-19T14:23:10.679 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122315.log 2023-10-19T14:23:10.680 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.101001.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.101001.log.gz 2023-10-19T14:23:10.680 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88407.log 2023-10-19T14:23:10.680 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.122315.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.122315.log.gz 2023-10-19T14:23:10.680 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96221.log 2023-10-19T14:23:10.680 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.88407.log: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.88407.log.gz 2023-10-19T14:23:10.681 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112063.log 2023-10-19T14:23:10.681 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.96221.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.96221.log.gz 2023-10-19T14:23:10.681 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121780.log 2023-10-19T14:23:10.681 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.112063.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.112063.log.gz 2023-10-19T14:23:10.681 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143328.log 2023-10-19T14:23:10.681 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.121780.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.121780.log.gz 2023-10-19T14:23:10.682 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102474.log 2023-10-19T14:23:10.682 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.143328.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.143328.log.gz 2023-10-19T14:23:10.682 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160829.log 2023-10-19T14:23:10.682 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.102474.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.102474.log.gz 2023-10-19T14:23:10.683 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165744.log 2023-10-19T14:23:10.683 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.160829.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.160829.log.gz 2023-10-19T14:23:10.683 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148531.log 2023-10-19T14:23:10.683 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.165744.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.165744.log.gz 2023-10-19T14:23:10.683 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156058.log 2023-10-19T14:23:10.684 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.148531.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.148531.log.gz 2023-10-19T14:23:10.684 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127680.log 2023-10-19T14:23:10.684 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.156058.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.156058.log.gz 2023-10-19T14:23:10.684 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97535.log 2023-10-19T14:23:10.684 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.127680.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.127680.log.gz 2023-10-19T14:23:10.684 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102642.log 2023-10-19T14:23:10.685 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.97535.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.97535.log.gz 2023-10-19T14:23:10.685 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123017.log 2023-10-19T14:23:10.685 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.102642.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.102642.log.gz 2023-10-19T14:23:10.685 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141586.log 2023-10-19T14:23:10.685 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.123017.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.123017.log.gz 2023-10-19T14:23:10.686 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128592.log 2023-10-19T14:23:10.686 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.141586.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.141586.log.gz 2023-10-19T14:23:10.686 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163036.log 2023-10-19T14:23:10.686 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.128592.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.128592.log.gz 2023-10-19T14:23:10.686 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141292.log 2023-10-19T14:23:10.686 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.163036.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.163036.log.gz 2023-10-19T14:23:10.687 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154974.log 2023-10-19T14:23:10.687 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.141292.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.141292.log.gz 2023-10-19T14:23:10.687 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130234.log 2023-10-19T14:23:10.687 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.154974.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.154974.log.gz 2023-10-19T14:23:10.687 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123902.log 2023-10-19T14:23:10.688 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.130234.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.130234.log.gz 2023-10-19T14:23:10.688 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146912.log 2023-10-19T14:23:10.688 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.123902.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.123902.log.gz 2023-10-19T14:23:10.688 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131181.log 2023-10-19T14:23:10.688 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.146912.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.146912.log.gz 2023-10-19T14:23:10.688 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99757.log 2023-10-19T14:23:10.689 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.131181.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.131181.log.gz 2023-10-19T14:23:10.689 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.99757.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.99757.log.gz 2023-10-19T14:23:10.689 INFO:teuthology.orchestra.run.smithi057.stderr: 93.1% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2023-10-19T14:23:11.045 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ceph-mon.c.log: 93.6% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2023-10-19T14:23:11.071 INFO:teuthology.orchestra.run.smithi057.stderr: 93.9% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2023-10-19T14:23:11.481 INFO:teuthology.orchestra.run.smithi179.stderr: 92.4% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2023-10-19T14:23:12.004 INFO:teuthology.orchestra.run.smithi179.stderr: 92.3% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-10-19T14:23:12.143 INFO:teuthology.orchestra.run.smithi179.stderr: 92.2% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2023-10-19T14:23:13.450 INFO:teuthology.orchestra.run.smithi057.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-10-19T14:23:14.566 INFO:teuthology.orchestra.run.smithi179.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2023-10-19T14:23:14.746 INFO:teuthology.orchestra.run.smithi179.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2023-10-19T14:23:14.887 INFO:teuthology.orchestra.run.smithi179.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2023-10-19T14:23:15.071 INFO:teuthology.orchestra.run.smithi057.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2023-10-19T14:23:15.482 INFO:teuthology.orchestra.run.smithi057.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-10-19T14:23:15.592 INFO:teuthology.orchestra.run.smithi057.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-10-19T14:23:16.017 INFO:teuthology.orchestra.run.smithi179.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2023-10-19T14:23:16.019 INFO:teuthology.orchestra.run.smithi179.stderr: 2023-10-19T14:23:16.019 INFO:teuthology.orchestra.run.smithi179.stderr:real 0m6.226s 2023-10-19T14:23:16.019 INFO:teuthology.orchestra.run.smithi179.stderr:user 0m27.450s 2023-10-19T14:23:16.019 INFO:teuthology.orchestra.run.smithi179.stderr:sys 0m0.950s 2023-10-19T14:23:16.225 INFO:teuthology.orchestra.run.smithi057.stderr: 92.8% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-10-19T14:23:16.227 INFO:teuthology.orchestra.run.smithi057.stderr: 2023-10-19T14:23:16.227 INFO:teuthology.orchestra.run.smithi057.stderr:real 0m6.465s 2023-10-19T14:23:16.227 INFO:teuthology.orchestra.run.smithi057.stderr:user 0m27.315s 2023-10-19T14:23:16.227 INFO:teuthology.orchestra.run.smithi057.stderr:sys 0m1.472s 2023-10-19T14:23:16.228 INFO:tasks.ceph:Archiving logs... 2023-10-19T14:23:16.228 DEBUG:teuthology.misc:Transferring archived files from smithi057:/var/log/ceph to /home/teuthworker/archive/pdonnell-2023-10-19_12:04:12-fs-wip-batrick-testing-20231016.203825-distro-default-smithi/7432290/remote/smithi057/log 2023-10-19T14:23:16.230 DEBUG:teuthology.orchestra.run.smithi057:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-19T14:23:20.749 DEBUG:teuthology.misc:Transferring archived files from smithi179:/var/log/ceph to /home/teuthworker/archive/pdonnell-2023-10-19_12:04:12-fs-wip-batrick-testing-20231016.203825-distro-default-smithi/7432290/remote/smithi179/log 2023-10-19T14:23:20.751 DEBUG:teuthology.orchestra.run.smithi179:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-19T14:23:24.309 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-10-19T14:23:24.318 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-10-19T14:23:24.319 DEBUG:teuthology.orchestra.run.smithi057:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-19T14:23:24.322 DEBUG:teuthology.orchestra.run.smithi179:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-19T14:23:24.416 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-10-19T14:23:24.426 INFO:teuthology.task.clock:Checking final clock skew... 2023-10-19T14:23:24.426 DEBUG:teuthology.orchestra.run.smithi057:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-19T14:23:24.429 DEBUG:teuthology.orchestra.run.smithi179:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-19T14:23:24.444 INFO:teuthology.orchestra.run.smithi057.stderr:bash: line 1: ntpq: command not found 2023-10-19T14:23:24.446 INFO:teuthology.orchestra.run.smithi179.stderr:bash: line 1: ntpq: command not found 2023-10-19T14:23:24.448 INFO:teuthology.orchestra.run.smithi057.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-19T14:23:24.448 INFO:teuthology.orchestra.run.smithi057.stdout:=============================================================================== 2023-10-19T14:23:24.449 INFO:teuthology.orchestra.run.smithi057.stdout:^* hv01.front.sepia.ceph.com 3 8 377 171 +2567us[+2849us] +/- 49ms 2023-10-19T14:23:24.449 INFO:teuthology.orchestra.run.smithi057.stdout:^+ hv02.front.sepia.ceph.com 3 7 377 30 -2850us[-2850us] +/- 50ms 2023-10-19T14:23:24.449 INFO:teuthology.orchestra.run.smithi057.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 27 -338us[ -338us] +/- 73ms 2023-10-19T14:23:24.449 INFO:teuthology.orchestra.run.smithi057.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2023-10-19T14:23:24.450 INFO:teuthology.orchestra.run.smithi179.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-19T14:23:24.451 INFO:teuthology.orchestra.run.smithi179.stdout:=============================================================================== 2023-10-19T14:23:24.451 INFO:teuthology.orchestra.run.smithi179.stdout:^* hv01.front.sepia.ceph.com 3 7 377 32 +2704us[+2728us] +/- 51ms 2023-10-19T14:23:24.451 INFO:teuthology.orchestra.run.smithi179.stdout:^+ hv02.front.sepia.ceph.com 3 9 377 40 -2693us[-2669us] +/- 50ms 2023-10-19T14:23:24.451 INFO:teuthology.orchestra.run.smithi179.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 29 -177us[ -177us] +/- 73ms 2023-10-19T14:23:24.452 INFO:teuthology.orchestra.run.smithi179.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2023-10-19T14:23:24.453 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-10-19T14:23:24.462 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-10-19T14:23:24.463 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-10-19T14:23:24.481 DEBUG:teuthology.orchestra.run.smithi057:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-19T14:23:24.568 DEBUG:teuthology.orchestra.run.smithi179:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-19T14:23:24.674 DEBUG:teuthology.orchestra.run.smithi057:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-19T14:23:24.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-19T14:23:24.710 DEBUG:teuthology.orchestra.run.smithi179:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-19T14:23:24.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-19T14:23:24.745 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-10-19T14:23:24.755 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-10-19T14:23:24.806 INFO:teuthology.task.internal:Duration was 6450.084048 seconds 2023-10-19T14:23:24.807 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-10-19T14:23:24.832 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-10-19T14:23:24.832 DEBUG:teuthology.orchestra.run.smithi057:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-19T14:23:24.835 DEBUG:teuthology.orchestra.run.smithi179:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-19T14:23:24.879 INFO:teuthology.orchestra.run.smithi179.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-19T14:23:24.886 INFO:teuthology.orchestra.run.smithi057.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-19T14:23:25.217 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-10-19T14:23:25.218 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi057.front.sepia.ceph.com 2023-10-19T14:23:25.218 DEBUG:teuthology.orchestra.run.smithi057:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-19T14:23:25.240 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi179.front.sepia.ceph.com 2023-10-19T14:23:25.241 DEBUG:teuthology.orchestra.run.smithi179:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-19T14:23:25.264 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-10-19T14:23:25.264 DEBUG:teuthology.orchestra.run.smithi057:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-19T14:23:25.283 DEBUG:teuthology.orchestra.run.smithi179:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-19T14:23:25.333 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-10-19T14:23:25.333 DEBUG:teuthology.orchestra.run.smithi057:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-19T14:23:25.357 DEBUG:teuthology.orchestra.run.smithi179:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-19T14:23:25.552 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-10-19T14:23:25.563 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-10-19T14:23:25.563 DEBUG:teuthology.orchestra.run.smithi057:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-19T14:23:25.594 DEBUG:teuthology.orchestra.run.smithi179:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-19T14:23:25.622 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-10-19T14:23:25.633 DEBUG:teuthology.orchestra.run.smithi057:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-19T14:23:25.637 DEBUG:teuthology.orchestra.run.smithi179:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-19T14:23:25.663 INFO:teuthology.orchestra.run.smithi057.stdout:kernel.core_pattern = core 2023-10-19T14:23:25.688 INFO:teuthology.orchestra.run.smithi179.stdout:kernel.core_pattern = core 2023-10-19T14:23:25.703 DEBUG:teuthology.orchestra.run.smithi057:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-19T14:23:25.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-19T14:23:25.737 DEBUG:teuthology.orchestra.run.smithi179:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-19T14:23:25.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-19T14:23:25.756 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-10-19T14:23:25.767 INFO:teuthology.task.internal:Transferring archived files... 2023-10-19T14:23:25.768 DEBUG:teuthology.misc:Transferring archived files from smithi057:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2023-10-19_12:04:12-fs-wip-batrick-testing-20231016.203825-distro-default-smithi/7432290/remote/smithi057 2023-10-19T14:23:25.768 DEBUG:teuthology.orchestra.run.smithi057:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-19T14:23:25.816 DEBUG:teuthology.misc:Transferring archived files from smithi179:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2023-10-19_12:04:12-fs-wip-batrick-testing-20231016.203825-distro-default-smithi/7432290/remote/smithi179 2023-10-19T14:23:25.816 DEBUG:teuthology.orchestra.run.smithi179:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-19T14:23:25.853 INFO:teuthology.task.internal:Removing archive directory... 2023-10-19T14:23:25.853 DEBUG:teuthology.orchestra.run.smithi057:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-19T14:23:25.856 DEBUG:teuthology.orchestra.run.smithi179:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-19T14:23:25.901 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-10-19T14:23:25.913 INFO:teuthology.task.internal:Not uploading archives. 2023-10-19T14:23:25.913 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-10-19T14:23:25.932 INFO:teuthology.task.internal:Tidying up after the test... 2023-10-19T14:23:25.932 DEBUG:teuthology.orchestra.run.smithi057:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-19T14:23:25.935 DEBUG:teuthology.orchestra.run.smithi179:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-19T14:23:25.947 INFO:teuthology.orchestra.run.smithi057.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 19 14:23 /home/ubuntu/cephtest 2023-10-19T14:23:25.955 INFO:teuthology.orchestra.run.smithi179.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 19 14:23 /home/ubuntu/cephtest 2023-10-19T14:23:25.956 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-10-19T14:23:25.967 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-10-19T14:23:26.034 INFO:teuthology.nuke:Checking targets against current locks 2023-10-19T14:23:26.061 DEBUG:teuthology.nuke:shortname: smithi057 2023-10-19T14:23:26.061 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-19T14:23:26.081 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi057.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-10-19_12:04:12-fs-wip-batrick-testing-20231016.203825-distro-default-smithi/7432290', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-19 12:26:53.268577', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLIN80GzytgnsPmhQxnO98RJ6hrAia1j+6pBY3TcjQx0CYaH+EBbVFNcBgcBJT8k6V/pnOvlWsn8miLAG0PNdlU='} 2023-10-19T14:23:26.091 DEBUG:teuthology.nuke:shortname: smithi179 2023-10-19T14:23:26.091 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-19T14:23:26.114 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi179.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-10-19_12:04:12-fs-wip-batrick-testing-20231016.203825-distro-default-smithi/7432290', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-19 12:26:53.266642', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHzD4QfGTQMro3MfFhf2wmDWesRzz0Us4uWFCUG2uJYCokdZYdTLrJb2/y+R7emIU5CcjUKUzVhjYy6iyLxqsbk='} 2023-10-19T14:23:26.129 INFO:teuthology.orchestra.console.smithi057:Power off 2023-10-19T14:23:26.130 DEBUG:teuthology.orchestra.console.smithi057:pexpect command: ipmitool -H smithi057.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-19T14:23:26.155 DEBUG:teuthology.orchestra.console.smithi057:power off output: Chassis Power Control: Down/Off 2023-10-19T14:23:26.156 DEBUG:teuthology.orchestra.console.smithi057:pexpect command: ipmitool -H smithi057.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-19T14:23:26.171 INFO:teuthology.orchestra.console.smithi179:Power off 2023-10-19T14:23:26.172 DEBUG:teuthology.orchestra.console.smithi179:pexpect command: ipmitool -H smithi179.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-19T14:23:26.183 DEBUG:teuthology.orchestra.console.smithi057:check power output: Chassis Power is on 2023-10-19T14:23:26.196 DEBUG:teuthology.orchestra.console.smithi179:power off output: Chassis Power Control: Down/Off 2023-10-19T14:23:26.196 DEBUG:teuthology.orchestra.console.smithi179:pexpect command: ipmitool -H smithi179.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-19T14:23:26.228 DEBUG:teuthology.orchestra.console.smithi179:check power output: Chassis Power is on 2023-10-19T14:23:30.184 DEBUG:teuthology.orchestra.console.smithi057:pexpect command: ipmitool -H smithi057.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-19T14:23:30.229 DEBUG:teuthology.orchestra.console.smithi179:pexpect command: ipmitool -H smithi179.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-19T14:23:30.294 DEBUG:teuthology.orchestra.console.smithi057:check power output: Chassis Power is on 2023-10-19T14:23:30.340 DEBUG:teuthology.orchestra.console.smithi179:check power output: Chassis Power is on 2023-10-19T14:23:34.296 DEBUG:teuthology.orchestra.console.smithi057:pexpect command: ipmitool -H smithi057.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-19T14:23:34.342 DEBUG:teuthology.orchestra.console.smithi179:pexpect command: ipmitool -H smithi179.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-19T14:23:34.407 DEBUG:teuthology.orchestra.console.smithi057:check power output: Chassis Power is on 2023-10-19T14:23:34.452 DEBUG:teuthology.orchestra.console.smithi179:check power output: Chassis Power is on 2023-10-19T14:23:38.410 DEBUG:teuthology.orchestra.console.smithi057:pexpect command: ipmitool -H smithi057.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-19T14:23:38.453 DEBUG:teuthology.orchestra.console.smithi179:pexpect command: ipmitool -H smithi179.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-19T14:23:38.521 DEBUG:teuthology.orchestra.console.smithi057:check power output: Chassis Power is off 2023-10-19T14:23:38.565 DEBUG:teuthology.orchestra.console.smithi179:check power output: Chassis Power is off 2023-10-19T14:23:38.622 INFO:teuthology.orchestra.console.smithi057:Power off completed 2023-10-19T14:23:38.667 INFO:teuthology.orchestra.console.smithi179:Power off completed 2023-10-19T14:23:38.775 INFO:teuthology.run:Summary data: description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} duration: 6450.084048271179 failure_reason: '"2023-10-19T12:46:58.974806+0000 mon.a (mon.0) 455 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY)" in cluster log' flavor: default owner: scheduled_pdonnell@teuthology success: false 2023-10-19T14:23:38.776 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-19T14:23:38.876 INFO:teuthology.run:FAIL